2024-04-02T04:20:23.472 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T04:20:23.472 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi --verbose --description orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837 -- /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/orig.config.yaml 2024-04-02T04:20:23.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T04:20:23.591 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837 branch: squid description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7634837' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca workunit: branch: squid sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6530 sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca sleep_before_teardown: 0 subset: 52/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_6cec793d9073f0594e2364b9b687ecaf22424cca/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP+U30wtui0wY2oeBYbZqBievav6XXSANBn6lZOKYlIggCY/eZy+GPZhGjCfwjfGOgPq4IAllYWlG36iUPUivRk= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6C5vGUfmkJ7b9kYheJBNcdxX2++k3gI92WASO5w8vmpR7OEjJWL8hIsZMFJ9j6XHK4Fsp0bPaVnO5AZUPcOXU= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-01_21:08:08 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T04:20:23.592 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_6cec793d9073f0594e2364b9b687ecaf22424cca/qa; will attempt to use it 2024-04-02T04:20:23.592 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_6cec793d9073f0594e2364b9b687ecaf22424cca/qa/tasks 2024-04-02T04:20:23.593 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T04:20:23.594 INFO:teuthology.task.internal:Checking packages... 2024-04-02T04:20:23.618 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '6cec793d9073f0594e2364b9b687ecaf22424cca' 2024-04-02T04:20:23.619 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T04:20:23.619 INFO:teuthology.packaging:ref: None 2024-04-02T04:20:23.619 INFO:teuthology.packaging:tag: None 2024-04-02T04:20:23.619 INFO:teuthology.packaging:branch: squid 2024-04-02T04:20:23.619 INFO:teuthology.packaging:sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:20:23.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-02T04:20:23.784 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:20:23.786 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T04:20:23.794 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T04:20:23.794 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T04:20:23.832 INFO:teuthology.task.internal:Saving configuration 2024-04-02T04:20:23.845 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T04:20:23.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T04:20:23.879 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:14:49.115539', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC8F5c3r+NgbgFZJXqZHjRN6o1Aksq5RdV46zWnuWzo/'} 2024-04-02T04:20:23.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:14:49.113955', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILb0fs1n7cd0R1TnnVd7qW0MKYWSiaFjawgyfHWRJ0YL'} 2024-04-02T04:20:23.897 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T04:20:23.906 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-02T04:20:23.906 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-02T04:20:23.907 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T04:20:23.979 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6ff03e93a0>, signals=[15]) 2024-04-02T04:20:23.980 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T04:20:23.986 INFO:teuthology.task.internal:Opening connections... 2024-04-02T04:20:23.986 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T04:20:23.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:20:24.082 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-04-02T04:20:24.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:20:24.160 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T04:20:24.167 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-04-02T04:20:24.173 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-04-02T04:20:24.173 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-04-02T04:20:24.219 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-02T04:20:24.219 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Ubuntu" 2024-04-02T04:20:24.219 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="22.04" 2024-04-02T04:20:24.219 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-02T04:20:24.219 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_CODENAME=jammy 2024-04-02T04:20:24.219 INFO:teuthology.orchestra.run.smithi069.stdout:ID=ubuntu 2024-04-02T04:20:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE=debian 2024-04-02T04:20:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-02T04:20:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-02T04:20:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-02T04:20:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-02T04:20:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:UBUNTU_CODENAME=jammy 2024-04-02T04:20:24.220 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-04-02T04:20:24.241 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-04-02T04:20:24.247 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-04-02T04:20:24.247 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-04-02T04:20:24.295 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-02T04:20:24.295 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Ubuntu" 2024-04-02T04:20:24.295 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="22.04" 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_CODENAME=jammy 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:ID=ubuntu 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE=debian 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-02T04:20:24.296 INFO:teuthology.orchestra.run.smithi098.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-02T04:20:24.297 INFO:teuthology.orchestra.run.smithi098.stdout:UBUNTU_CODENAME=jammy 2024-04-02T04:20:24.297 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-04-02T04:20:24.316 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T04:20:24.326 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T04:20:24.341 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T04:20:24.341 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-04-02T04:20:24.344 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-04-02T04:20:24.348 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T04:20:24.357 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T04:20:24.357 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-04-02T04:20:24.389 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-04-02T04:20:24.398 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T04:20:24.450 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T04:20:24.464 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-02T04:20:24.465 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-02T04:20:24.465 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T04:20:24.465 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T04:20:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:20:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:20:24.471 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-02T04:20:24.471 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-02T04:20:24.518 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-56-generic 2024-04-02T04:20:24.519 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.15.0-56-generic 2024-04-02T04:20:24.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-04-02T04:20:24.520 INFO:teuthology.orchestra.run.smithi069.stdout:5.15.0-56-generic 2024-04-02T04:20:24.520 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.15.0-56-generic 2024-04-02T04:20:24.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2024-04-02T04:20:24.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-04-02T04:20:24.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2024-04-02T04:20:24.819 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-02T04:20:24.917 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-02T04:20:24.917 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-02T04:20:24.925 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-02T04:20:24.974 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-02T04:20:25.007 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-02T04:20:25.050 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-02T04:20:25.174 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-02T04:20:25.181 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-02T04:20:25.184 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-02T04:20:25.184 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-02T04:20:25.222 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-02T04:20:25.222 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-02T04:20:25.225 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-02T04:20:25.225 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-02T04:20:25.230 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-02T04:20:25.242 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-02T04:20:25.243 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-02T04:20:25.244 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-02T04:20:25.244 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-02T04:20:25.263 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-02T04:20:25.279 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-02T04:20:25.326 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-02T04:20:25.327 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-02T04:20:25.369 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-02T04:20:25.465 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-02T04:20:25.508 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-02T04:20:25.663 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-02T04:20:25.741 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-02T04:20:25.760 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-02T04:20:25.769 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-02T04:20:25.770 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-02T04:20:25.810 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-02T04:20:25.812 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-02T04:20:25.825 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-02T04:20:25.825 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-04-02T04:20:25.827 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-04-02T04:20:25.852 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-02T04:20:25.861 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-02T04:20:25.863 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-02T04:20:25.870 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-02T04:20:25.871 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-04-02T04:20:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-02T04:20:25.881 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-02T04:20:25.886 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-04-02T04:20:25.893 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-02T04:20:25.904 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-02T04:20:25.929 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-04-02T04:20:25.930 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-04-02T04:20:25.937 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-02T04:20:26.057 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-02T04:20:26.081 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-02T04:20:26.101 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-02T04:20:26.478 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-02T04:20:26.484 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-02T04:20:26.498 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-02T04:20:26.499 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-02T04:20:26.501 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-02T04:20:26.638 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-02T04:20:26.735 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-02T04:20:26.748 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-02T04:20:26.768 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-02T04:20:26.769 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-02T04:20:26.773 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-02T04:20:26.774 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-02T04:20:26.774 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-02T04:20:26.774 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-02T04:20:26.775 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-02T04:20:26.783 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-02T04:20:26.784 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-02T04:20:26.786 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-02T04:20:26.787 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-02T04:20:26.787 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-02T04:20:26.790 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-02T04:20:26.791 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-02T04:20:26.791 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-02T04:20:26.817 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-02T04:20:26.820 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-02T04:20:26.948 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-02T04:20:26.985 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-02T04:20:27.031 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-02T04:20:27.068 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-02T04:20:27.140 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-02T04:20:27.179 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-02T04:20:27.196 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-02T04:20:27.230 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-02T04:20:27.234 INFO:teuthology.orchestra.run.smithi069.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-02T04:20:27.268 INFO:teuthology.orchestra.run.smithi069.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-02T04:20:31.655 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 12.6 MB in 3s (4,991 kB/s) 2024-04-02T04:20:31.728 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 12.6 MB in 3s (4,980 kB/s) 2024-04-02T04:20:32.903 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-02T04:20:32.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-02T04:20:32.973 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-02T04:20:33.064 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2024-04-02T04:20:33.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-02T04:20:33.146 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-02T04:20:33.147 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-02T04:20:33.185 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2024-04-02T04:20:33.297 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-02T04:20:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-02T04:20:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-02T04:20:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-02T04:20:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-02T04:20:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: python2.7-minimal 2024-04-02T04:20:33.298 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-02T04:20:33.299 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-04-02T04:20:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-02T04:20:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-02T04:20:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-02T04:20:33.300 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-04-02T04:20:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-02T04:20:33.301 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-04-02T04:20:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: thermald 2024-04-02T04:20:33.348 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-04-02T04:20:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-02T04:20:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-02T04:20:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-02T04:20:33.350 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-04-02T04:20:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-02T04:20:33.385 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2024-04-02T04:20:33.386 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2024-04-02T04:20:33.407 INFO:teuthology.orchestra.run.smithi098.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-02T04:20:33.407 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 113 MB of archives. 2024-04-02T04:20:33.407 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-02T04:20:33.407 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-02T04:20:33.558 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2024-04-02T04:20:33.558 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-02T04:20:33.559 INFO:teuthology.orchestra.run.smithi069.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-02T04:20:33.559 INFO:teuthology.orchestra.run.smithi069.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-02T04:20:33.559 INFO:teuthology.orchestra.run.smithi069.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-02T04:20:33.559 INFO:teuthology.orchestra.run.smithi069.stdout: python2.7-minimal 2024-04-02T04:20:33.559 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-02T04:20:33.560 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2024-04-02T04:20:33.561 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-02T04:20:33.561 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-02T04:20:33.561 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-02T04:20:33.562 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2024-04-02T04:20:33.562 INFO:teuthology.orchestra.run.smithi069.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-02T04:20:33.562 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2024-04-02T04:20:33.563 INFO:teuthology.orchestra.run.smithi069.stdout: thermald 2024-04-02T04:20:33.609 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2024-04-02T04:20:33.609 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-02T04:20:33.609 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-02T04:20:33.610 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-02T04:20:33.611 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages will be upgraded: 2024-04-02T04:20:33.611 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-02T04:20:33.682 INFO:teuthology.orchestra.run.smithi069.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-02T04:20:33.683 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 113 MB of archives. 2024-04-02T04:20:33.683 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-02T04:20:33.683 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-02T04:20:33.942 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-02T04:20:34.284 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-02T04:20:34.812 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-02T04:20:35.096 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-02T04:20:36.467 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-02T04:20:36.468 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-02T04:20:36.468 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-02T04:20:36.542 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-02T04:20:36.543 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-02T04:20:36.543 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-02T04:20:36.657 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-02T04:20:36.710 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-02T04:20:36.712 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-02T04:20:36.788 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-02T04:20:37.108 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 113 MB in 3s (33.7 MB/s) 2024-04-02T04:20:37.216 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 113 MB in 3s (35.7 MB/s) 2024-04-02T04:20:37.277 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-02T04:20:37.378 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-02T04:20:39.263 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-02T04:20:39.267 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:39.351 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:39.397 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-02T04:20:39.401 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:39.508 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:41.478 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-02T04:20:41.499 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:41.597 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:41.684 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-02T04:20:41.706 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:41.796 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:42.039 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-02T04:20:42.060 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:42.097 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:42.237 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-02T04:20:42.257 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:42.287 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:48.737 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-02T04:20:48.861 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-02T04:20:49.165 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-02T04:20:49.211 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-02T04:20:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-02T04:20:49.332 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-02T04:20:49.593 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-02T04:20:49.600 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-02T04:20:49.615 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-02T04:20:49.651 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-02T04:20:49.720 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-02T04:20:50.007 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-02T04:20:50.029 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-02T04:20:50.073 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-02T04:20:55.525 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-02T04:20:55.562 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:55.600 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:56.054 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-02T04:20:56.091 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-02T04:20:56.129 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:58.121 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-02T04:20:58.227 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-02T04:20:58.561 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-02T04:20:58.682 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:58.766 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-02T04:20:58.808 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-02T04:20:58.892 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-02T04:20:58.912 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:59.226 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-02T04:20:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:20:59.481 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-02T04:20:59.602 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:00.305 INFO:teuthology.orchestra.run.smithi069.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-02T04:21:00.305 INFO:teuthology.orchestra.run.smithi069.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-02T04:21:00.305 INFO:teuthology.orchestra.run.smithi069.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-02T04:21:00.305 INFO:teuthology.orchestra.run.smithi069.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:00.986 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-02T04:21:00.987 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-02T04:21:00.987 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-02T04:21:00.987 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:01.007 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:01.190 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:02.457 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:02.605 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:03.850 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-02T04:21:03.951 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-02T04:21:04.007 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-02T04:21:04.052 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:04.110 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-02T04:21:04.110 INFO:teuthology.orchestra.run.smithi069.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:04.133 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-02T04:21:04.259 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-02T04:21:04.328 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-02T04:21:04.328 INFO:teuthology.orchestra.run.smithi098.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-02T04:21:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub' 2024-04-02T04:21:18.065 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-02T04:21:18.093 INFO:teuthology.orchestra.run.smithi069.stdout:Generating grub configuration file ... 2024-04-02T04:21:18.382 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-02T04:21:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:18.635 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-02T04:21:18.635 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub' 2024-04-02T04:21:18.660 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-02T04:21:18.678 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-02T04:21:18.681 INFO:teuthology.orchestra.run.smithi098.stdout:Generating grub configuration file ... 2024-04-02T04:21:18.684 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-02T04:21:18.753 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-02T04:21:18.759 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-02T04:21:18.919 INFO:teuthology.orchestra.run.smithi069.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-02T04:21:18.919 INFO:teuthology.orchestra.run.smithi069.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-02T04:21:18.919 INFO:teuthology.orchestra.run.smithi069.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-02T04:21:18.948 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-02T04:21:18.959 INFO:teuthology.orchestra.run.smithi069.stdout:done 2024-04-02T04:21:18.968 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:19.246 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-02T04:21:19.252 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-02T04:21:19.314 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-02T04:21:19.320 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-02T04:21:19.420 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2024-04-02T04:21:19.436 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2024-04-02T04:21:19.436 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2024-04-02T04:21:19.436 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2024-04-02T04:21:19.436 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2024-04-02T04:21:19.436 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 21 2024-04-02T04:21:19.436 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.15.0.101.98 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2024-04-02T04:21:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2024-04-02T04:21:19.437 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-02T04:21:19.437 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T04:21:19.438 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T04:21:19.438 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-02T04:21:19.438 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T04:21:19.438 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2024-04-02T04:21:19.501 INFO:teuthology.orchestra.run.smithi098.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-02T04:21:19.501 INFO:teuthology.orchestra.run.smithi098.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-02T04:21:19.501 INFO:teuthology.orchestra.run.smithi098.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-02T04:21:19.550 INFO:teuthology.orchestra.run.smithi098.stdout:done 2024-04-02T04:21:19.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2024-04-02T04:21:19.761 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-02T04:21:19.773 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-02T04:21:19.793 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-02T04:21:19.908 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-02T04:21:19.973 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 21 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.15.0.101.98 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-02T04:21:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-02T04:21:19.999 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-04-02T04:21:19.999 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-04-02T04:21:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-04-02T04:21:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-04-02T04:21:19.999 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-02T04:21:19.999 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T04:21:19.999 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T04:21:19.999 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-02T04:21:19.999 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T04:21:19.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-04-02T04:21:20.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-04-02T04:21:20.411 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-02T04:21:20.411 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-02T04:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-02T04:21:20.561 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-02T04:21:21.138 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2024-04-02T04:21:21.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-02T04:21:21.206 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2024-04-02T04:21:21.380 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2024-04-02T04:21:21.380 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2024-04-02T04:21:21.531 INFO:teuthology.orchestra.run.smithi069.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-02T04:21:21.532 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-02T04:21:21.568 INFO:teuthology.orchestra.run.smithi069.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-02T04:21:21.570 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2024-04-02T04:21:21.624 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 21 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.15.0.101.98 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2024-04-02T04:21:21.625 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2024-04-02T04:21:21.626 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2024-04-02T04:21:21.626 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-02T04:21:21.670 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.ptksfOTd3a 2024-04-02T04:21:21.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ptksfOTd3a 2024-04-02T04:21:21.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.ptksfOTd3a 2024-04-02T04:21:21.774 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-02T04:21:21.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-02T04:21:21.842 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-02T04:21:21.851 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.ptksfOTd3a is 10KB 2024-04-02T04:21:21.864 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.ptksfOTd3a 2024-04-02T04:21:21.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-02T04:21:21.922 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:21:21.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-02T04:21:21.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-02T04:21:21.990 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-02T04:21:21.990 DEBUG:teuthology.orchestra.run.smithi069:> sudo update-grub 2024-04-02T04:21:22.016 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-02T04:21:22.017 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-02T04:21:22.171 INFO:teuthology.orchestra.run.smithi098.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-02T04:21:22.171 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-02T04:21:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-02T04:21:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-02T04:21:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-02T04:21:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-02T04:21:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-02T04:21:22.172 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-02T04:21:22.210 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-02T04:21:22.212 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-04-02T04:21:22.272 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-04-02T04:21:22.272 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 21 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.15.0.101.98 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-04-02T04:21:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-04-02T04:21:22.274 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-04-02T04:21:22.318 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.4aoWDWQWGk 2024-04-02T04:21:22.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4aoWDWQWGk 2024-04-02T04:21:22.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.4aoWDWQWGk 2024-04-02T04:21:22.483 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.4aoWDWQWGk is 10KB 2024-04-02T04:21:22.496 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.4aoWDWQWGk 2024-04-02T04:21:22.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-02T04:21:22.553 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:21:22.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-02T04:21:22.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-02T04:21:22.624 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-02T04:21:22.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo update-grub 2024-04-02T04:21:23.164 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub' 2024-04-02T04:21:23.165 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-02T04:21:23.168 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-04-02T04:21:23.349 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-02T04:21:23.356 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:23.589 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-02T04:21:23.594 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-02T04:21:23.633 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-02T04:21:23.638 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-02T04:21:23.765 INFO:teuthology.orchestra.run.smithi069.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-02T04:21:23.766 INFO:teuthology.orchestra.run.smithi069.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-02T04:21:23.766 INFO:teuthology.orchestra.run.smithi069.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-02T04:21:23.778 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-04-02T04:21:23.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-04-02T04:21:23.827 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2024-04-02T04:21:23.829 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-02T04:21:23.832 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-04-02T04:21:24.016 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-02T04:21:24.021 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-02T04:21:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-02T04:21:24.263 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-02T04:21:24.305 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-02T04:21:24.310 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-02T04:21:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-02T04:21:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-02T04:21:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-02T04:21:24.460 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-04-02T04:21:24.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-04-02T04:21:53.826 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-02T04:21:53.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:21:53.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:21:54.479 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-02T04:21:54.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:21:54.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:22:12.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-04-02T04:22:13.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-04-02T04:22:21.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:22:21.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:22:22.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:22:22.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:22:24.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-04-02T04:22:25.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-04-02T04:22:36.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:22:36.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:22:37.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:22:37.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:22:54.755 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-04-02T04:22:55.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-04-02T04:23:09.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:23:09.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:23:10.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:23:10.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:23:10.624 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-02T04:23:10.871 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-02T04:23:11.343 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:23:11.343 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-04-02T04:23:11.343 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-02T04:23:11.392 INFO:teuthology.orchestra.run.smithi069.stdout:5.15.0-101-generic 2024-04-02T04:23:11.393 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-02T04:23:11.393 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T04:23:11.393 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-02T04:23:11.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:23:11.641 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-04-02T04:23:11.642 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-02T04:23:11.688 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-101-generic 2024-04-02T04:23:11.688 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-02T04:23:11.688 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T04:23:11.688 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-02T04:23:12.393 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-02T04:23:12.394 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T04:23:12.468 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-04-02T04:23:12.506 DEBUG:teuthology.parallel:result is None 2024-04-02T04:23:12.689 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-02T04:23:12.689 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T04:23:12.772 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-04-02T04:23:12.806 DEBUG:teuthology.parallel:result is None 2024-04-02T04:23:12.806 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T04:23:12.817 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T04:23:12.818 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T04:23:12.820 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T04:23:12.825 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T04:23:12.849 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T04:23:12.891 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T04:23:12.891 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T04:23:12.893 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T04:23:12.917 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T04:23:12.923 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T04:23:12.924 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T04:23:12.950 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T04:23:12.966 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:23:12.973 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:23:12.982 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:23:12.990 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:23:12.992 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T04:23:12.998 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T04:23:12.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T04:23:13.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T04:23:13.053 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T04:23:13.061 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T04:23:13.062 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T04:23:13.073 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T04:23:13.100 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T04:23:13.125 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T04:23:13.173 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:23:13.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T04:23:13.250 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T04:23:13.256 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T04:23:13.304 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:23:13.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T04:23:13.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-04-02T04:23:13.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-04-02T04:23:13.443 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T04:23:13.451 INFO:teuthology.task.internal:Starting timer... 2024-04-02T04:23:13.451 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T04:23:13.495 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T04:23:13.526 DEBUG:teuthology.task.selinux:Excluding smithi069: OS 'ubuntu' does not support SELinux 2024-04-02T04:23:13.527 DEBUG:teuthology.task.selinux:Excluding smithi098: OS 'ubuntu' does not support SELinux 2024-04-02T04:23:13.527 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T04:23:13.527 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-02T04:23:13.527 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T04:23:13.527 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T04:23:13.560 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T04:23:13.577 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T04:23:13.578 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T04:30:15.545 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-04-02T04:30:15.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:30:15.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:30:15.628 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-02T04:30:15.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-02T04:30:15.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:30:15.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:30:15.786 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-02T04:30:15.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-02T04:30:15.867 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T04:30:15.877 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T04:30:15.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T04:30:15.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T04:30:15.881 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T04:30:15.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T04:30:15.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-02T04:30:15.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Command line: ntpd -gq 2024-04-02T04:30:15.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: ---------------------------------------------------- 2024-04-02T04:30:15.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: ntp-4 is maintained by Network Time Foundation, 2024-04-02T04:30:15.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-02T04:30:15.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: corporation. Support and training for ntp-4 are 2024-04-02T04:30:15.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: available at https://www.nwtime.org/support 2024-04-02T04:30:15.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: ---------------------------------------------------- 2024-04-02T04:30:15.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: proto: precision = 0.044 usec (-24) 2024-04-02T04:30:15.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: basedate set to 2022-02-04 2024-04-02T04:30:15.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: gps base set to 2022-02-06 (week 2196) 2024-04-02T04:30:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-02T04:30:15.911 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-02T04:30:15.911 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: restrict ::: KOD does nothing without LIMITED. 2024-04-02T04:30:15.911 INFO:teuthology.orchestra.run.smithi069.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listen and drop on 0 v6wildcard [::]:123 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listen normally on 2 lo 127.0.0.1:123 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listen normally on 3 enp3s0f1 172.21.15.69:123 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listen normally on 4 lo [::1]:123 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d737%5]:123 2024-04-02T04:30:15.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:15 ntpd[17169]: Listening on routing socket on fd #22 for interface updates 2024-04-02T04:30:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-02T04:30:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Command line: ntpd -gq 2024-04-02T04:30:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: ---------------------------------------------------- 2024-04-02T04:30:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: ntp-4 is maintained by Network Time Foundation, 2024-04-02T04:30:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-02T04:30:15.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: corporation. Support and training for ntp-4 are 2024-04-02T04:30:15.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: available at https://www.nwtime.org/support 2024-04-02T04:30:15.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: ---------------------------------------------------- 2024-04-02T04:30:15.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: proto: precision = 0.058 usec (-24) 2024-04-02T04:30:15.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: basedate set to 2022-02-04 2024-04-02T04:30:15.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: gps base set to 2022-02-06 (week 2196) 2024-04-02T04:30:15.942 INFO:teuthology.orchestra.run.smithi098.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-02T04:30:15.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-02T04:30:15.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: restrict ::: KOD does nothing without LIMITED. 2024-04-02T04:30:15.943 INFO:teuthology.orchestra.run.smithi098.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-02T04:30:15.943 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listen and drop on 0 v6wildcard [::]:123 2024-04-02T04:30:15.943 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-02T04:30:15.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listen normally on 2 lo 127.0.0.1:123 2024-04-02T04:30:15.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listen normally on 3 enp3s0f1 172.21.15.98:123 2024-04-02T04:30:15.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listen normally on 4 lo [::1]:123 2024-04-02T04:30:15.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d26b%5]:123 2024-04-02T04:30:15.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:15 ntpd[17171]: Listening on routing socket on fd #22 for interface updates 2024-04-02T04:30:16.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:16 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:16.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:16 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:16.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:16 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:16.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:16 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:16.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:16 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:16.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:16 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:16.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:16.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:16.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:16.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:16 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:17.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:17 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:17.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:17 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:18.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:18.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:18.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:18 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:18.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:18.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:18 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:19.908 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:19 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:19.909 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:19 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:19.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:19 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:19.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:19 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:19.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:19 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:19.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:19 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:19.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:19 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:19.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:19 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:20.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:20.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:20 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:20.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:20.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:20.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:20 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:21.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:21 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:21.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:21 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:21.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:21 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:21.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:21 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:21.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:21 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:21.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:21 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:22.910 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:22.914 INFO:teuthology.orchestra.run.smithi069.stdout: 2 Apr 04:30:22 ntpd[17169]: ntpd: time slew -0.001358 s 2024-04-02T04:30:22.915 INFO:teuthology.orchestra.run.smithi069.stdout:ntpd: time slew -0.001358s 2024-04-02T04:30:22.915 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-02T04:30:22.915 INFO:teuthology.orchestra.run.smithi069.stderr: 2 Apr 04:30:22 ntpd[17169]: can't open /var/log/ntpstats/loopstats.20240402: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240402: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-02T04:30:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240402: Permission denied 2024-04-02T04:30:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2 Apr 04:30:22 ntpd[17171]: ntpd: time slew -0.004454 s 2024-04-02T04:30:22.942 INFO:teuthology.orchestra.run.smithi098.stdout:ntpd: time slew -0.004454s 2024-04-02T04:30:22.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-02T04:30:22.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2 Apr 04:30:22 ntpd[17171]: can't open /var/log/ntpstats/loopstats.20240402: Permission denied 2024-04-02T04:30:22.960 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2024-04-02T04:30:22.960 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2024-04-02T04:30:22.960 INFO:teuthology.orchestra.run.smithi069.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.960 INFO:teuthology.orchestra.run.smithi069.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.961 INFO:teuthology.orchestra.run.smithi069.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.961 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.997 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-04-02T04:30:22.997 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-04-02T04:30:22.997 INFO:teuthology.orchestra.run.smithi098.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.997 INFO:teuthology.orchestra.run.smithi098.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.997 INFO:teuthology.orchestra.run.smithi098.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.998 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-02T04:30:22.998 INFO:teuthology.run_tasks:Running task install... 2024-04-02T04:30:23.008 DEBUG:teuthology.task.install:project ceph 2024-04-02T04:30:23.008 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca'}} 2024-04-02T04:30:23.008 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca'} 2024-04-02T04:30:23.008 INFO:teuthology.task.install:Using flavor: default 2024-04-02T04:30:23.014 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T04:30:23.014 INFO:teuthology.task.install:extra packages: [] 2024-04-02T04:30:23.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-key list | grep Ceph 2024-04-02T04:30:23.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-key list | grep Ceph 2024-04-02T04:30:23.069 INFO:teuthology.orchestra.run.smithi069.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-02T04:30:23.096 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-02T04:30:23.096 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph.com (release key) 2024-04-02T04:30:23.096 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-02T04:30:23.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:30:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-02T04:30:23.131 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-02T04:30:23.132 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph.com (release key) 2024-04-02T04:30:23.132 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-02T04:30:23.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:30:23.249 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default/ 2024-04-02T04:30:23.249 INFO:teuthology.task.install.deb:Package version is 19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:30:23.269 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default/ 2024-04-02T04:30:23.269 INFO:teuthology.task.install.deb:Package version is 19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:30:23.381 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:30:23.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-02T04:30:23.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-04-02T04:30:23.467 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:30:23.467 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-02T04:30:23.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2024-04-02T04:30:23.584 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-02T04:30:23.584 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-02T04:30:23.611 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-02T04:30:23.618 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy InRelease 2024-04-02T04:30:23.633 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-02T04:30:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-02T04:30:23.657 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-02T04:30:23.657 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-02T04:30:23.663 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-02T04:30:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-02T04:30:23.684 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-04-02T04:30:23.692 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-02T04:30:23.706 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy InRelease 2024-04-02T04:30:23.711 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-02T04:30:23.727 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-02T04:30:23.748 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-02T04:30:23.769 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-04-02T04:30:23.797 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-02T04:30:23.921 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 28.9 kB in 0s (83.2 kB/s) 2024-04-02T04:30:24.014 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 28.9 kB in 0s (82.1 kB/s) 2024-04-02T04:30:24.844 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-02T04:30:24.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1979-g6cec793d-1jammy cephadm=19.0.0-1979-g6cec793d-1jammy ceph-mds=19.0.0-1979-g6cec793d-1jammy ceph-mgr=19.0.0-1979-g6cec793d-1jammy ceph-common=19.0.0-1979-g6cec793d-1jammy ceph-fuse=19.0.0-1979-g6cec793d-1jammy ceph-test=19.0.0-1979-g6cec793d-1jammy ceph-volume=19.0.0-1979-g6cec793d-1jammy radosgw=19.0.0-1979-g6cec793d-1jammy python3-rados=19.0.0-1979-g6cec793d-1jammy python3-rgw=19.0.0-1979-g6cec793d-1jammy python3-cephfs=19.0.0-1979-g6cec793d-1jammy python3-rbd=19.0.0-1979-g6cec793d-1jammy libcephfs2=19.0.0-1979-g6cec793d-1jammy libcephfs-dev=19.0.0-1979-g6cec793d-1jammy librados2=19.0.0-1979-g6cec793d-1jammy librbd1=19.0.0-1979-g6cec793d-1jammy rbd-fuse=19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:30:24.918 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-04-02T04:30:25.001 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2024-04-02T04:30:25.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1979-g6cec793d-1jammy cephadm=19.0.0-1979-g6cec793d-1jammy ceph-mds=19.0.0-1979-g6cec793d-1jammy ceph-mgr=19.0.0-1979-g6cec793d-1jammy ceph-common=19.0.0-1979-g6cec793d-1jammy ceph-fuse=19.0.0-1979-g6cec793d-1jammy ceph-test=19.0.0-1979-g6cec793d-1jammy ceph-volume=19.0.0-1979-g6cec793d-1jammy radosgw=19.0.0-1979-g6cec793d-1jammy python3-rados=19.0.0-1979-g6cec793d-1jammy python3-rgw=19.0.0-1979-g6cec793d-1jammy python3-cephfs=19.0.0-1979-g6cec793d-1jammy python3-rbd=19.0.0-1979-g6cec793d-1jammy libcephfs2=19.0.0-1979-g6cec793d-1jammy libcephfs-dev=19.0.0-1979-g6cec793d-1jammy librados2=19.0.0-1979-g6cec793d-1jammy librbd1=19.0.0-1979-g6cec793d-1jammy rbd-fuse=19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:30:25.079 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2024-04-02T04:30:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-04-02T04:30:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-04-02T04:30:25.276 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2024-04-02T04:30:25.277 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2024-04-02T04:30:25.298 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-04-02T04:30:25.298 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-02T04:30:25.298 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-02T04:30:25.298 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-02T04:30:25.298 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-02T04:30:25.300 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-02T04:30:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-02T04:30:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-04-02T04:30:25.300 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-02T04:30:25.300 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-02T04:30:25.301 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-02T04:30:25.302 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet zip 2024-04-02T04:30:25.303 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-04-02T04:30:25.304 INFO:teuthology.orchestra.run.smithi098.stdout: btrfs-tools 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-02T04:30:25.403 INFO:teuthology.orchestra.run.smithi098.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-02T04:30:25.404 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-02T04:30:25.405 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-02T04:30:25.405 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-02T04:30:25.405 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-02T04:30:25.405 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-02T04:30:25.405 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-02T04:30:25.405 INFO:teuthology.orchestra.run.smithi098.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-02T04:30:25.467 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 118 newly installed, 0 to remove and 299 not upgraded. 2024-04-02T04:30:25.467 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 178 MB of archives. 2024-04-02T04:30:25.467 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 815 MB of additional disk space will be used. 2024-04-02T04:30:25.467 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-02T04:30:25.470 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-02T04:30:25.471 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-02T04:30:25.472 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet zip 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2024-04-02T04:30:25.473 INFO:teuthology.orchestra.run.smithi069.stdout: btrfs-tools 2024-04-02T04:30:25.570 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-02T04:30:25.575 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2024-04-02T04:30:25.575 INFO:teuthology.orchestra.run.smithi069.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-02T04:30:25.575 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-02T04:30:25.575 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-02T04:30:25.576 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-02T04:30:25.577 INFO:teuthology.orchestra.run.smithi069.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-02T04:30:25.577 INFO:teuthology.orchestra.run.smithi069.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-02T04:30:25.578 INFO:teuthology.orchestra.run.smithi069.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-02T04:30:25.578 INFO:teuthology.orchestra.run.smithi069.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-02T04:30:25.578 INFO:teuthology.orchestra.run.smithi069.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-02T04:30:25.578 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-02T04:30:25.578 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-02T04:30:25.578 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-02T04:30:25.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-02T04:30:25.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-02T04:30:25.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-02T04:30:25.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-02T04:30:25.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-02T04:30:25.579 INFO:teuthology.orchestra.run.smithi069.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-02T04:30:25.580 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-02T04:30:25.581 INFO:teuthology.orchestra.run.smithi069.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-02T04:30:25.581 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-02T04:30:25.593 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-02T04:30:25.619 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1979-g6cec793d-1jammy [3,544 kB] 2024-04-02T04:30:25.660 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-02T04:30:25.665 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-02T04:30:25.678 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-02T04:30:25.682 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-02T04:30:25.682 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-02T04:30:25.684 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-02T04:30:25.692 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-02T04:30:25.694 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-02T04:30:25.697 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-02T04:30:25.698 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-02T04:30:25.704 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-02T04:30:25.706 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-02T04:30:25.709 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-02T04:30:25.715 INFO:teuthology.orchestra.run.smithi069.stdout:0 upgraded, 118 newly installed, 0 to remove and 299 not upgraded. 2024-04-02T04:30:25.715 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 178 MB of archives. 2024-04-02T04:30:25.715 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 815 MB of additional disk space will be used. 2024-04-02T04:30:25.715 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1979-g6cec793d-1jammy [3,544 kB] 2024-04-02T04:30:25.723 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-02T04:30:25.724 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-02T04:30:25.725 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-02T04:30:25.726 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-02T04:30:25.726 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-02T04:30:25.726 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-02T04:30:25.727 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-02T04:30:25.743 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-02T04:30:25.744 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-02T04:30:25.744 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-02T04:30:25.761 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-02T04:30:25.765 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-02T04:30:25.765 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-02T04:30:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-02T04:30:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-02T04:30:25.768 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-02T04:30:25.768 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-02T04:30:25.769 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-02T04:30:25.797 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-02T04:30:25.798 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-02T04:30:25.798 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-02T04:30:25.836 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-02T04:30:25.836 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-02T04:30:25.836 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-02T04:30:25.837 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-02T04:30:25.838 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-02T04:30:25.840 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-02T04:30:25.841 INFO:teuthology.orchestra.run.smithi098.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-02T04:30:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-02T04:30:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-02T04:30:25.882 INFO:teuthology.orchestra.run.smithi098.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-02T04:30:25.896 INFO:teuthology.orchestra.run.smithi098.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-02T04:30:25.897 INFO:teuthology.orchestra.run.smithi098.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-02T04:30:25.902 INFO:teuthology.orchestra.run.smithi098.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-02T04:30:25.980 INFO:teuthology.orchestra.run.smithi098.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-02T04:30:25.981 INFO:teuthology.orchestra.run.smithi098.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-02T04:30:25.982 INFO:teuthology.orchestra.run.smithi098.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-02T04:30:25.982 INFO:teuthology.orchestra.run.smithi098.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-02T04:30:25.983 INFO:teuthology.orchestra.run.smithi098.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-02T04:30:26.000 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1979-g6cec793d-1jammy [3,217 kB] 2024-04-02T04:30:26.023 INFO:teuthology.orchestra.run.smithi098.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-02T04:30:26.027 INFO:teuthology.orchestra.run.smithi098.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-02T04:30:26.028 INFO:teuthology.orchestra.run.smithi098.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-02T04:30:26.058 INFO:teuthology.orchestra.run.smithi098.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-02T04:30:26.058 INFO:teuthology.orchestra.run.smithi098.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-02T04:30:26.080 INFO:teuthology.orchestra.run.smithi098.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1979-g6cec793d-1jammy [3,217 kB] 2024-04-02T04:30:26.086 INFO:teuthology.orchestra.run.smithi098.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-02T04:30:26.087 INFO:teuthology.orchestra.run.smithi098.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-02T04:30:26.088 INFO:teuthology.orchestra.run.smithi098.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-02T04:30:26.089 INFO:teuthology.orchestra.run.smithi098.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-02T04:30:26.089 INFO:teuthology.orchestra.run.smithi098.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-02T04:30:26.090 INFO:teuthology.orchestra.run.smithi098.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-02T04:30:26.096 INFO:teuthology.orchestra.run.smithi098.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-02T04:30:26.126 INFO:teuthology.orchestra.run.smithi098.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-02T04:30:26.128 INFO:teuthology.orchestra.run.smithi098.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-02T04:30:26.129 INFO:teuthology.orchestra.run.smithi098.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-02T04:30:26.150 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1979-g6cec793d-1jammy [712 kB] 2024-04-02T04:30:26.162 INFO:teuthology.orchestra.run.smithi098.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-02T04:30:26.166 INFO:teuthology.orchestra.run.smithi098.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-02T04:30:26.168 INFO:teuthology.orchestra.run.smithi098.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-02T04:30:26.169 INFO:teuthology.orchestra.run.smithi098.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-02T04:30:26.177 INFO:teuthology.orchestra.run.smithi098.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-02T04:30:26.178 INFO:teuthology.orchestra.run.smithi098.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-02T04:30:26.180 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1979-g6cec793d-1jammy [356 kB] 2024-04-02T04:30:26.185 INFO:teuthology.orchestra.run.smithi098.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-02T04:30:26.191 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-02T04:30:26.202 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1979-g6cec793d-1jammy [32.8 kB] 2024-04-02T04:30:26.203 INFO:teuthology.orchestra.run.smithi098.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-02T04:30:26.203 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-02T04:30:26.203 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1979-g6cec793d-1jammy [180 kB] 2024-04-02T04:30:26.204 INFO:teuthology.orchestra.run.smithi098.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-02T04:30:26.204 INFO:teuthology.orchestra.run.smithi098.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-02T04:30:26.210 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1979-g6cec793d-1jammy [64.6 kB] 2024-04-02T04:30:26.213 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1979-g6cec793d-1jammy [330 kB] 2024-04-02T04:30:26.228 INFO:teuthology.orchestra.run.smithi098.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-02T04:30:26.230 INFO:teuthology.orchestra.run.smithi098.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-02T04:30:26.237 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1979-g6cec793d-1jammy [6,577 kB] 2024-04-02T04:30:26.242 INFO:teuthology.orchestra.run.smithi098.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-02T04:30:26.242 INFO:teuthology.orchestra.run.smithi098.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-02T04:30:26.243 INFO:teuthology.orchestra.run.smithi098.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-02T04:30:26.247 INFO:teuthology.orchestra.run.smithi098.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-02T04:30:26.248 INFO:teuthology.orchestra.run.smithi098.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-02T04:30:26.286 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-02T04:30:26.330 INFO:teuthology.orchestra.run.smithi098.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1979-g6cec793d-1jammy [712 kB] 2024-04-02T04:30:26.373 INFO:teuthology.orchestra.run.smithi098.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1979-g6cec793d-1jammy [356 kB] 2024-04-02T04:30:26.407 INFO:teuthology.orchestra.run.smithi098.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1979-g6cec793d-1jammy [32.8 kB] 2024-04-02T04:30:26.410 INFO:teuthology.orchestra.run.smithi098.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1979-g6cec793d-1jammy [180 kB] 2024-04-02T04:30:26.420 INFO:teuthology.orchestra.run.smithi098.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1979-g6cec793d-1jammy [64.6 kB] 2024-04-02T04:30:26.423 INFO:teuthology.orchestra.run.smithi098.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1979-g6cec793d-1jammy [330 kB] 2024-04-02T04:30:26.448 INFO:teuthology.orchestra.run.smithi098.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1979-g6cec793d-1jammy [6,577 kB] 2024-04-02T04:30:26.516 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-02T04:30:26.528 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-02T04:30:26.558 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-02T04:30:26.568 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-02T04:30:26.568 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-02T04:30:26.570 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-02T04:30:26.588 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-02T04:30:26.612 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1979-g6cec793d-1jammy [112 kB] 2024-04-02T04:30:26.615 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1979-g6cec793d-1jammy [466 kB] 2024-04-02T04:30:26.642 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1979-g6cec793d-1jammy [25.2 MB] 2024-04-02T04:30:26.756 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-02T04:30:27.008 INFO:teuthology.orchestra.run.smithi098.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1979-g6cec793d-1jammy [112 kB] 2024-04-02T04:30:27.011 INFO:teuthology.orchestra.run.smithi098.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1979-g6cec793d-1jammy [466 kB] 2024-04-02T04:30:27.021 INFO:teuthology.orchestra.run.smithi098.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1979-g6cec793d-1jammy [25.2 MB] 2024-04-02T04:30:27.163 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-02T04:30:27.198 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-02T04:30:27.200 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-02T04:30:27.279 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-02T04:30:27.331 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-02T04:30:27.353 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-02T04:30:27.358 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-02T04:30:27.360 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-02T04:30:27.371 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-02T04:30:27.372 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-02T04:30:27.538 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-02T04:30:27.621 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-02T04:30:27.649 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-02T04:30:27.666 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-02T04:30:27.711 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-02T04:30:27.926 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-02T04:30:27.941 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-02T04:30:27.947 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-02T04:30:27.953 INFO:teuthology.orchestra.run.smithi069.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-02T04:30:27.975 INFO:teuthology.orchestra.run.smithi069.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-02T04:30:27.978 INFO:teuthology.orchestra.run.smithi069.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-02T04:30:28.144 INFO:teuthology.orchestra.run.smithi069.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-02T04:30:28.207 INFO:teuthology.orchestra.run.smithi069.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1979-g6cec793d-1jammy [5,077 kB] 2024-04-02T04:30:28.433 INFO:teuthology.orchestra.run.smithi069.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1979-g6cec793d-1jammy [244 kB] 2024-04-02T04:30:28.446 INFO:teuthology.orchestra.run.smithi069.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1979-g6cec793d-1jammy [124 kB] 2024-04-02T04:30:28.455 INFO:teuthology.orchestra.run.smithi069.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1979-g6cec793d-1jammy [1,526 kB] 2024-04-02T04:30:28.460 INFO:teuthology.orchestra.run.smithi069.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-02T04:30:28.498 INFO:teuthology.orchestra.run.smithi069.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-02T04:30:28.509 INFO:teuthology.orchestra.run.smithi069.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-02T04:30:28.545 INFO:teuthology.orchestra.run.smithi069.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1979-g6cec793d-1jammy [6,247 kB] 2024-04-02T04:30:28.715 INFO:teuthology.orchestra.run.smithi069.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-02T04:30:28.716 INFO:teuthology.orchestra.run.smithi069.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-02T04:30:28.721 INFO:teuthology.orchestra.run.smithi069.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-02T04:30:28.727 INFO:teuthology.orchestra.run.smithi069.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-02T04:30:28.742 INFO:teuthology.orchestra.run.smithi069.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-02T04:30:28.763 INFO:teuthology.orchestra.run.smithi069.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-02T04:30:28.769 INFO:teuthology.orchestra.run.smithi069.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-02T04:30:28.907 INFO:teuthology.orchestra.run.smithi069.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1979-g6cec793d-1jammy [22.7 MB] 2024-04-02T04:30:28.924 INFO:teuthology.orchestra.run.smithi098.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1979-g6cec793d-1jammy [5,077 kB] 2024-04-02T04:30:28.944 INFO:teuthology.orchestra.run.smithi069.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-02T04:30:29.207 INFO:teuthology.orchestra.run.smithi069.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-02T04:30:29.208 INFO:teuthology.orchestra.run.smithi098.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1979-g6cec793d-1jammy [244 kB] 2024-04-02T04:30:29.228 INFO:teuthology.orchestra.run.smithi098.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1979-g6cec793d-1jammy [124 kB] 2024-04-02T04:30:29.236 INFO:teuthology.orchestra.run.smithi098.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1979-g6cec793d-1jammy [1,526 kB] 2024-04-02T04:30:29.345 INFO:teuthology.orchestra.run.smithi098.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1979-g6cec793d-1jammy [6,247 kB] 2024-04-02T04:30:29.385 INFO:teuthology.orchestra.run.smithi069.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-02T04:30:29.400 INFO:teuthology.orchestra.run.smithi069.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-02T04:30:29.419 INFO:teuthology.orchestra.run.smithi069.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-02T04:30:29.484 INFO:teuthology.orchestra.run.smithi069.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-02T04:30:29.774 INFO:teuthology.orchestra.run.smithi069.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-02T04:30:29.776 INFO:teuthology.orchestra.run.smithi069.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-02T04:30:29.779 INFO:teuthology.orchestra.run.smithi069.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-02T04:30:29.779 INFO:teuthology.orchestra.run.smithi069.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-02T04:30:29.782 INFO:teuthology.orchestra.run.smithi069.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-02T04:30:29.810 INFO:teuthology.orchestra.run.smithi098.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1979-g6cec793d-1jammy [22.7 MB] 2024-04-02T04:30:30.003 INFO:teuthology.orchestra.run.smithi069.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-02T04:30:30.337 INFO:teuthology.orchestra.run.smithi069.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1979-g6cec793d-1jammy [14.0 kB] 2024-04-02T04:30:30.338 INFO:teuthology.orchestra.run.smithi069.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1979-g6cec793d-1jammy [869 kB] 2024-04-02T04:30:30.355 INFO:teuthology.orchestra.run.smithi069.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1979-g6cec793d-1jammy [2,434 kB] 2024-04-02T04:30:30.439 INFO:teuthology.orchestra.run.smithi069.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-02T04:30:30.445 INFO:teuthology.orchestra.run.smithi069.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-02T04:30:30.525 INFO:teuthology.orchestra.run.smithi069.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1979-g6cec793d-1jammy [777 kB] 2024-04-02T04:30:30.540 INFO:teuthology.orchestra.run.smithi069.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1979-g6cec793d-1jammy [145 kB] 2024-04-02T04:30:30.543 INFO:teuthology.orchestra.run.smithi069.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1979-g6cec793d-1jammy [2,281 kB] 2024-04-02T04:30:30.651 INFO:teuthology.orchestra.run.smithi069.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1979-g6cec793d-1jammy [8,625 kB] 2024-04-02T04:30:30.736 INFO:teuthology.orchestra.run.smithi069.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-02T04:30:30.737 INFO:teuthology.orchestra.run.smithi069.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-02T04:30:30.739 INFO:teuthology.orchestra.run.smithi069.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-02T04:30:30.741 INFO:teuthology.orchestra.run.smithi069.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-02T04:30:30.742 INFO:teuthology.orchestra.run.smithi069.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-02T04:30:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-02T04:30:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-02T04:30:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-02T04:30:30.929 INFO:teuthology.orchestra.run.smithi069.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-02T04:30:31.153 INFO:teuthology.orchestra.run.smithi069.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1979-g6cec793d-1jammy [14.2 kB] 2024-04-02T04:30:31.153 INFO:teuthology.orchestra.run.smithi069.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1979-g6cec793d-1jammy [50.0 MB] 2024-04-02T04:30:31.340 INFO:teuthology.orchestra.run.smithi069.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-02T04:30:31.385 INFO:teuthology.orchestra.run.smithi069.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-02T04:30:31.422 INFO:teuthology.orchestra.run.smithi069.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-02T04:30:31.517 INFO:teuthology.orchestra.run.smithi069.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-02T04:30:31.557 INFO:teuthology.orchestra.run.smithi069.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-02T04:30:31.569 INFO:teuthology.orchestra.run.smithi069.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-02T04:30:31.574 INFO:teuthology.orchestra.run.smithi069.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-02T04:30:31.587 INFO:teuthology.orchestra.run.smithi098.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1979-g6cec793d-1jammy [14.0 kB] 2024-04-02T04:30:31.588 INFO:teuthology.orchestra.run.smithi098.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1979-g6cec793d-1jammy [869 kB] 2024-04-02T04:30:31.604 INFO:teuthology.orchestra.run.smithi098.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1979-g6cec793d-1jammy [2,434 kB] 2024-04-02T04:30:31.621 INFO:teuthology.orchestra.run.smithi069.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-02T04:30:31.624 INFO:teuthology.orchestra.run.smithi069.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-02T04:30:31.646 INFO:teuthology.orchestra.run.smithi069.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-02T04:30:31.742 INFO:teuthology.orchestra.run.smithi098.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1979-g6cec793d-1jammy [777 kB] 2024-04-02T04:30:31.802 INFO:teuthology.orchestra.run.smithi098.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1979-g6cec793d-1jammy [145 kB] 2024-04-02T04:30:31.810 INFO:teuthology.orchestra.run.smithi069.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-02T04:30:31.811 INFO:teuthology.orchestra.run.smithi098.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1979-g6cec793d-1jammy [2,281 kB] 2024-04-02T04:30:31.985 INFO:teuthology.orchestra.run.smithi098.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1979-g6cec793d-1jammy [8,625 kB] 2024-04-02T04:30:32.085 INFO:teuthology.orchestra.run.smithi069.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-02T04:30:32.102 INFO:teuthology.orchestra.run.smithi069.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-02T04:30:32.256 INFO:teuthology.orchestra.run.smithi069.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-02T04:30:32.286 INFO:teuthology.orchestra.run.smithi069.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-02T04:30:32.443 INFO:teuthology.orchestra.run.smithi069.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-02T04:30:32.447 INFO:teuthology.orchestra.run.smithi069.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-02T04:30:32.448 INFO:teuthology.orchestra.run.smithi069.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-02T04:30:32.471 INFO:teuthology.orchestra.run.smithi069.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-02T04:30:32.475 INFO:teuthology.orchestra.run.smithi069.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-02T04:30:32.629 INFO:teuthology.orchestra.run.smithi098.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1979-g6cec793d-1jammy [14.2 kB] 2024-04-02T04:30:32.630 INFO:teuthology.orchestra.run.smithi098.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1979-g6cec793d-1jammy [50.0 MB] 2024-04-02T04:30:34.498 INFO:teuthology.orchestra.run.smithi069.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1979-g6cec793d-1jammy [121 kB] 2024-04-02T04:30:34.501 INFO:teuthology.orchestra.run.smithi069.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1979-g6cec793d-1jammy [38.9 kB] 2024-04-02T04:30:34.502 INFO:teuthology.orchestra.run.smithi069.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1979-g6cec793d-1jammy [13.0 MB] 2024-04-02T04:30:35.049 INFO:teuthology.orchestra.run.smithi069.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1979-g6cec793d-1jammy [91.4 kB] 2024-04-02T04:30:35.602 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 178 MB in 9s (18.8 MB/s) 2024-04-02T04:30:35.768 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-02T04:30:35.802 INFO:teuthology.orchestra.run.smithi098.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1979-g6cec793d-1jammy [121 kB] 2024-04-02T04:30:35.804 INFO:teuthology.orchestra.run.smithi098.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1979-g6cec793d-1jammy [38.9 kB] 2024-04-02T04:30:35.805 INFO:teuthology.orchestra.run.smithi098.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1979-g6cec793d-1jammy [13.0 MB] 2024-04-02T04:30:35.867 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-02T04:30:35.873 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-02T04:30:35.958 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-02T04:30:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1979-g6cec793d-1jammy [91.4 kB] 2024-04-02T04:30:36.236 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-02T04:30:36.262 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-02T04:30:36.294 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-02T04:30:36.647 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-02T04:30:36.674 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-02T04:30:36.709 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 178 MB in 11s (16.6 MB/s) 2024-04-02T04:30:36.714 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-02T04:30:36.875 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-02T04:30:36.975 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-02T04:30:36.981 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-02T04:30:37.065 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-02T04:30:37.167 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-02T04:30:37.194 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-02T04:30:37.242 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:30:37.369 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-02T04:30:37.396 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-02T04:30:37.427 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-02T04:30:37.772 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-02T04:30:37.799 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-02T04:30:37.803 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-02T04:30:37.830 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-02T04:30:37.838 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-02T04:30:37.862 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:30:38.207 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-02T04:30:38.233 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-02T04:30:38.273 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:30:38.284 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-02T04:30:38.310 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-02T04:30:38.359 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:30:38.618 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-02T04:30:38.644 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-02T04:30:38.684 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-02T04:30:38.746 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-02T04:30:38.772 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-02T04:30:38.812 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:30:39.012 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librados2. 2024-04-02T04:30:39.039 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../007-librados2_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:39.079 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librados2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:39.157 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-02T04:30:39.184 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-02T04:30:39.224 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:30:39.523 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librbd1. 2024-04-02T04:30:39.549 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../008-librbd1_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:39.571 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-02T04:30:39.590 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librbd1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:39.597 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-02T04:30:39.637 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-02T04:30:39.949 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librados2. 2024-04-02T04:30:39.975 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../007-librados2_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:40.015 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librados2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:40.034 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs2. 2024-04-02T04:30:40.060 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:40.100 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:40.403 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rados. 2024-04-02T04:30:40.418 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:40.470 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rados (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:40.493 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librbd1. 2024-04-02T04:30:40.519 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../008-librbd1_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:40.560 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librbd1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:40.806 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-02T04:30:40.832 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:30:40.872 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-argparse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:40.954 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs2. 2024-04-02T04:30:40.981 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:41.021 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:41.133 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cephfs. 2024-04-02T04:30:41.156 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:41.200 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cephfs (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:41.290 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rados. 2024-04-02T04:30:41.306 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:41.348 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rados (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:41.544 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-02T04:30:41.570 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:30:41.619 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:41.693 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-02T04:30:41.719 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:30:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-argparse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:41.996 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cephfs. 2024-04-02T04:30:42.011 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:42.013 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-02T04:30:42.040 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-02T04:30:42.045 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cephfs (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:42.088 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-02T04:30:42.340 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-02T04:30:42.367 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:30:42.407 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:42.441 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-prettytable. 2024-04-02T04:30:42.467 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-02T04:30:42.508 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-02T04:30:42.735 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-02T04:30:42.761 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-02T04:30:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-02T04:30:42.927 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rbd. 2024-04-02T04:30:42.954 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:43.019 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rbd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:43.146 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-prettytable. 2024-04-02T04:30:43.173 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-02T04:30:43.213 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-02T04:30:43.380 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-02T04:30:43.406 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-02T04:30:43.440 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rbd. 2024-04-02T04:30:43.455 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:43.463 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-02T04:30:43.490 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rbd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:43.816 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-02T04:30:43.818 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-02T04:30:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-02T04:30:43.845 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-02T04:30:43.885 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-02T04:30:43.890 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-02T04:30:44.194 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-02T04:30:44.218 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-02T04:30:44.220 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-02T04:30:44.233 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-02T04:30:44.263 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-02T04:30:44.276 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-02T04:30:44.574 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-02T04:30:44.589 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-02T04:30:44.614 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package lua5.1. 2024-04-02T04:30:44.632 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-02T04:30:44.639 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-02T04:30:44.680 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-02T04:30:44.944 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua5.1. 2024-04-02T04:30:44.970 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-02T04:30:45.011 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-02T04:30:45.033 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package lua-any. 2024-04-02T04:30:45.059 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-02T04:30:45.108 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-02T04:30:45.322 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-any. 2024-04-02T04:30:45.338 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-02T04:30:45.381 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-02T04:30:45.410 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package zip. 2024-04-02T04:30:45.438 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-02T04:30:45.477 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking zip (3.0-12build2) ... 2024-04-02T04:30:45.608 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package zip. 2024-04-02T04:30:45.623 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-02T04:30:45.667 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking zip (3.0-12build2) ... 2024-04-02T04:30:45.796 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package unzip. 2024-04-02T04:30:45.822 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-02T04:30:45.871 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-02T04:30:45.987 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package unzip. 2024-04-02T04:30:46.013 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-02T04:30:46.053 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-02T04:30:46.275 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package luarocks. 2024-04-02T04:30:46.301 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-02T04:30:46.342 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-02T04:30:46.374 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package luarocks. 2024-04-02T04:30:46.400 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-02T04:30:46.432 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-02T04:30:46.741 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librgw2. 2024-04-02T04:30:46.767 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../025-librgw2_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:46.824 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librgw2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:46.840 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librgw2. 2024-04-02T04:30:46.867 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../025-librgw2_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:46.898 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librgw2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:47.385 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rgw. 2024-04-02T04:30:47.400 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:47.418 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rgw. 2024-04-02T04:30:47.433 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:47.434 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:47.476 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:47.746 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-02T04:30:47.772 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-02T04:30:47.821 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-02T04:30:47.848 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-02T04:30:47.887 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-02T04:30:47.955 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-02T04:30:48.165 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libradosstriper1. 2024-04-02T04:30:48.192 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:48.232 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libradosstriper1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:48.266 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libradosstriper1. 2024-04-02T04:30:48.291 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:48.332 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libradosstriper1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:48.627 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-common. 2024-04-02T04:30:48.654 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:48.693 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:48.777 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-common. 2024-04-02T04:30:48.803 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:48.843 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:50.158 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-base. 2024-04-02T04:30:50.184 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:50.215 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-base. 2024-04-02T04:30:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-base (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:50.242 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:30:50.291 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-base (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:30:50.804 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-02T04:30:50.830 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-02T04:30:50.870 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-02T04:30:50.911 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-02T04:30:50.937 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-02T04:30:50.977 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-02T04:30:51.123 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cheroot. 2024-04-02T04:30:51.139 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-02T04:30:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-02T04:30:51.213 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cheroot. 2024-04-02T04:30:51.228 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-02T04:30:51.280 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-02T04:30:51.501 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-02T04:30:51.528 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-02T04:30:51.585 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-02T04:30:51.599 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-02T04:30:51.626 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-02T04:30:51.666 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-02T04:30:51.896 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-02T04:30:51.923 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-02T04:30:51.963 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-02T04:30:51.969 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-02T04:30:51.996 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-02T04:30:52.035 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-02T04:30:52.224 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-02T04:30:52.250 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-02T04:30:52.282 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-02T04:30:52.313 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-02T04:30:52.340 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-02T04:30:52.379 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-02T04:30:52.569 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempora. 2024-04-02T04:30:52.595 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-02T04:30:52.635 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-02T04:30:52.649 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-tempora. 2024-04-02T04:30:52.676 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-02T04:30:52.716 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-02T04:30:52.972 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-portend. 2024-04-02T04:30:52.999 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-02T04:30:53.018 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-portend. 2024-04-02T04:30:53.038 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-02T04:30:53.045 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-02T04:30:53.085 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-02T04:30:53.341 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-02T04:30:53.368 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-02T04:30:53.388 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-02T04:30:53.408 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-02T04:30:53.414 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-02T04:30:53.454 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-02T04:30:53.711 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-02T04:30:53.724 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-02T04:30:53.737 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-02T04:30:53.738 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-02T04:30:53.765 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-02T04:30:53.777 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-02T04:30:54.106 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-natsort. 2024-04-02T04:30:54.133 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-02T04:30:54.135 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-natsort. 2024-04-02T04:30:54.162 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-02T04:30:54.164 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-02T04:30:54.210 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-02T04:30:54.434 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-logutils. 2024-04-02T04:30:54.461 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-02T04:30:54.501 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-02T04:30:54.580 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-logutils. 2024-04-02T04:30:54.607 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-02T04:30:54.655 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-02T04:30:54.804 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-02T04:30:54.831 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-02T04:30:54.870 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-02T04:30:54.958 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-02T04:30:54.984 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-02T04:30:55.024 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-02T04:30:55.174 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-mako. 2024-04-02T04:30:55.200 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-02T04:30:55.240 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-02T04:30:55.327 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-mako. 2024-04-02T04:30:55.353 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-02T04:30:55.394 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-02T04:30:55.569 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-02T04:30:55.595 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-02T04:30:55.635 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-02T04:30:55.763 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-02T04:30:55.790 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-02T04:30:55.838 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-02T04:30:55.938 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-02T04:30:55.964 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-02T04:30:56.005 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-02T04:30:56.100 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-02T04:30:56.126 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-02T04:30:56.158 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-02T04:30:56.266 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webob. 2024-04-02T04:30:56.292 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-02T04:30:56.324 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-02T04:30:56.452 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webob. 2024-04-02T04:30:56.478 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-02T04:30:56.510 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-02T04:30:56.652 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-waitress. 2024-04-02T04:30:56.679 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-02T04:30:56.722 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-02T04:30:56.855 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-waitress. 2024-04-02T04:30:56.881 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-02T04:30:56.924 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-02T04:30:57.056 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempita. 2024-04-02T04:30:57.082 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-02T04:30:57.122 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-02T04:30:57.268 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-tempita. 2024-04-02T04:30:57.294 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-02T04:30:57.334 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-02T04:30:57.425 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-paste. 2024-04-02T04:30:57.451 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-02T04:30:57.491 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-02T04:30:57.621 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-paste. 2024-04-02T04:30:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-02T04:30:57.687 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-02T04:30:57.811 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-02T04:30:57.827 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-02T04:30:57.870 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-02T04:30:58.007 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-02T04:30:58.034 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-02T04:30:58.090 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-02T04:30:58.173 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-02T04:30:58.199 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-02T04:30:58.239 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-02T04:30:58.385 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-02T04:30:58.411 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-02T04:30:58.451 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-02T04:30:58.554 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webtest. 2024-04-02T04:30:58.577 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-02T04:30:58.618 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-02T04:30:58.704 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webtest. 2024-04-02T04:30:58.717 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-02T04:30:58.746 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-02T04:30:59.705 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pecan. 2024-04-02T04:30:59.731 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-02T04:30:59.771 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-02T04:30:59.816 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pecan. 2024-04-02T04:30:59.843 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-02T04:30:59.891 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-02T04:31:00.246 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-certifi. 2024-04-02T04:31:00.273 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-02T04:31:00.312 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-02T04:31:00.326 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-certifi. 2024-04-02T04:31:00.351 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-02T04:31:00.392 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-02T04:31:00.648 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-idna. 2024-04-02T04:31:00.675 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-02T04:31:00.715 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-02T04:31:00.720 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-idna. 2024-04-02T04:31:00.747 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-02T04:31:00.779 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-02T04:31:01.018 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-urllib3. 2024-04-02T04:31:01.044 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-02T04:31:01.074 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-urllib3. 2024-04-02T04:31:01.076 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-02T04:31:01.100 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-02T04:31:01.140 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-02T04:31:01.387 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-requests. 2024-04-02T04:31:01.414 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-02T04:31:01.445 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-02T04:31:01.468 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests. 2024-04-02T04:31:01.495 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-02T04:31:01.535 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-02T04:31:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-02T04:31:01.764 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-02T04:31:01.798 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-02T04:31:01.846 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-02T04:31:01.873 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-02T04:31:01.913 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-02T04:31:02.093 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-02T04:31:02.108 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:02.151 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:02.208 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-02T04:31:02.234 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:02.274 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:02.539 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-02T04:31:02.566 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:02.604 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-02T04:31:02.606 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:02.619 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:02.654 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:02.934 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-02T04:31:02.960 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-02T04:31:02.974 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-02T04:31:03.000 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-02T04:31:03.001 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-02T04:31:03.041 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-02T04:31:03.327 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-openssl. 2024-04-02T04:31:03.353 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-02T04:31:03.393 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-02T04:31:03.528 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-openssl. 2024-04-02T04:31:03.555 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-02T04:31:03.586 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-02T04:31:03.705 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr. 2024-04-02T04:31:03.731 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:03.763 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:03.906 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr. 2024-04-02T04:31:03.932 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:03.972 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:04.150 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mon. 2024-04-02T04:31:04.176 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:04.216 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mon (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:04.384 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mon. 2024-04-02T04:31:04.411 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:04.450 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mon (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:04.837 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-osd. 2024-04-02T04:31:04.864 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:04.904 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-osd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:05.041 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-osd. 2024-04-02T04:31:05.064 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:05.107 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-osd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:05.998 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph. 2024-04-02T04:31:06.013 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../066-ceph_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:06.048 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:06.123 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph. 2024-04-02T04:31:06.139 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../066-ceph_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:06.182 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:06.343 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-fuse. 2024-04-02T04:31:06.369 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:06.409 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:06.493 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-fuse. 2024-04-02T04:31:06.520 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:06.559 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:06.788 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mds. 2024-04-02T04:31:06.814 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mds (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:06.946 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mds. 2024-04-02T04:31:06.973 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:07.012 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mds (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:07.249 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephadm. 2024-04-02T04:31:07.264 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../069-cephadm_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:07.299 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:07.390 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package cephadm. 2024-04-02T04:31:07.417 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../069-cephadm_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:07.448 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:07.594 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-02T04:31:07.620 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-02T04:31:07.660 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-02T04:31:07.776 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-02T04:31:07.803 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-02T04:31:07.843 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-02T04:31:07.980 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-02T04:31:08.006 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-02T04:31:08.038 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:08.137 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-02T04:31:08.152 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-02T04:31:08.187 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:09.077 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-babel. 2024-04-02T04:31:09.103 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-02T04:31:09.143 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:09.233 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-babel. 2024-04-02T04:31:09.261 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-02T04:31:09.300 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:09.463 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jinja2. 2024-04-02T04:31:09.489 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-02T04:31:09.529 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-02T04:31:09.636 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jinja2. 2024-04-02T04:31:09.663 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-02T04:31:09.703 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-02T04:31:09.924 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-02T04:31:09.951 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:10.081 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-02T04:31:10.108 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:10.147 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:10.328 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-02T04:31:10.354 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-02T04:31:10.394 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-02T04:31:10.500 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-02T04:31:10.527 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-02T04:31:10.567 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-02T04:31:10.697 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-routes. 2024-04-02T04:31:10.724 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-02T04:31:10.763 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-02T04:31:10.845 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-routes. 2024-04-02T04:31:10.860 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-02T04:31:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-02T04:31:11.092 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-02T04:31:11.119 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:11.158 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:11.231 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-02T04:31:11.258 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:11.297 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:12.372 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-02T04:31:12.388 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-02T04:31:12.422 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-02T04:31:12.553 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-02T04:31:12.580 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-02T04:31:12.619 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-02T04:31:12.917 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-joblib. 2024-04-02T04:31:12.944 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-02T04:31:12.975 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-02T04:31:13.106 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-joblib. 2024-04-02T04:31:13.133 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-02T04:31:13.172 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-02T04:31:13.287 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-02T04:31:13.303 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-02T04:31:13.345 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-02T04:31:13.484 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-02T04:31:13.500 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-02T04:31:13.542 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-02T04:31:13.657 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn. 2024-04-02T04:31:13.684 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-02T04:31:13.723 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-02T04:31:14.003 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn. 2024-04-02T04:31:14.030 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-02T04:31:14.070 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-02T04:31:14.424 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-02T04:31:14.452 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:14.584 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:14.728 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-02T04:31:14.744 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:14.871 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:15.498 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cachetools. 2024-04-02T04:31:15.524 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-02T04:31:15.555 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-02T04:31:15.893 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cachetools. 2024-04-02T04:31:15.920 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-02T04:31:15.976 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-02T04:31:16.033 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-02T04:31:16.060 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-02T04:31:16.099 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-02T04:31:16.270 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-02T04:31:16.297 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-02T04:31:16.336 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-02T04:31:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-02T04:31:16.438 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-02T04:31:16.469 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-02T04:31:16.639 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-02T04:31:16.667 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-02T04:31:16.714 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-02T04:31:16.731 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rsa. 2024-04-02T04:31:16.758 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-02T04:31:16.789 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-02T04:31:16.975 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rsa. 2024-04-02T04:31:16.989 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-02T04:31:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-02T04:31:17.112 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-google-auth. 2024-04-02T04:31:17.139 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-02T04:31:17.186 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-02T04:31:17.346 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-google-auth. 2024-04-02T04:31:17.373 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-02T04:31:17.412 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-02T04:31:17.573 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-02T04:31:17.600 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-02T04:31:17.648 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-02T04:31:17.732 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-02T04:31:17.760 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-02T04:31:17.799 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-02T04:31:17.968 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-websocket. 2024-04-02T04:31:17.995 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-02T04:31:18.034 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-02T04:31:18.110 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-websocket. 2024-04-02T04:31:18.137 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-02T04:31:18.177 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-02T04:31:18.363 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-02T04:31:18.390 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-02T04:31:18.429 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-02T04:31:18.480 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-02T04:31:18.508 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-02T04:31:18.546 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-02T04:31:19.052 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-02T04:31:19.080 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:19.118 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:19.194 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-02T04:31:19.222 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:19.260 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:19.446 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-02T04:31:19.473 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-02T04:31:19.513 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-02T04:31:19.571 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-02T04:31:19.599 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-02T04:31:19.638 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-02T04:31:19.841 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-02T04:31:19.868 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-02T04:31:19.907 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-02T04:31:19.940 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-02T04:31:19.968 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-02T04:31:20.007 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-02T04:31:20.177 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package jq. 2024-04-02T04:31:20.204 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-02T04:31:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-02T04:31:20.268 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package jq. 2024-04-02T04:31:20.296 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-02T04:31:20.351 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-02T04:31:20.587 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package socat. 2024-04-02T04:31:20.603 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-02T04:31:20.637 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-02T04:31:20.712 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package socat. 2024-04-02T04:31:20.712 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-02T04:31:20.712 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-02T04:31:20.923 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package xmlstarlet. 2024-04-02T04:31:20.939 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-02T04:31:20.973 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-02T04:31:21.008 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package xmlstarlet. 2024-04-02T04:31:21.023 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-02T04:31:21.230 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-02T04:31:21.303 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-test. 2024-04-02T04:31:21.319 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:21.361 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-test (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:21.579 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-test. 2024-04-02T04:31:21.603 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:21.645 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-test (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:23.557 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-volume. 2024-04-02T04:31:23.585 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:23.623 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-volume (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:23.775 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-volume. 2024-04-02T04:31:23.791 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1979-g6cec793d-1jammy_all.deb ... 2024-04-02T04:31:23.833 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-volume (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:23.877 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-02T04:31:23.890 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-02T04:31:24.143 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs-dev (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:24.153 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:24.187 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs-dev (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:24.463 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-02T04:31:24.481 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-02T04:31:24.491 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-02T04:31:24.509 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-02T04:31:24.529 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-02T04:31:24.548 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-02T04:31:24.833 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-02T04:31:24.861 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-02T04:31:24.878 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-02T04:31:24.900 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-02T04:31:24.906 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-02T04:31:24.946 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-02T04:31:25.212 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package nvme-cli. 2024-04-02T04:31:25.240 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-02T04:31:25.270 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-02T04:31:25.282 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package nvme-cli. 2024-04-02T04:31:25.310 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-02T04:31:25.349 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-02T04:31:25.642 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package pkg-config. 2024-04-02T04:31:25.670 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-02T04:31:25.700 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-02T04:31:25.720 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package pkg-config. 2024-04-02T04:31:25.748 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-02T04:31:25.778 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-02T04:31:25.953 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-02T04:31:25.981 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-02T04:31:25.998 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-02T04:31:26.020 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-02T04:31:26.025 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-02T04:31:26.064 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-02T04:31:26.423 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-02T04:31:26.443 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-02T04:31:26.451 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-02T04:31:26.470 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-02T04:31:26.489 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-02T04:31:26.509 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-02T04:31:26.817 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastescript. 2024-04-02T04:31:26.829 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastescript. 2024-04-02T04:31:26.844 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-02T04:31:26.856 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-02T04:31:26.883 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-02T04:31:26.895 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-02T04:31:27.203 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pluggy. 2024-04-02T04:31:27.216 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pluggy. 2024-04-02T04:31:27.231 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-02T04:31:27.243 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-02T04:31:27.270 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-02T04:31:27.282 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-02T04:31:27.573 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-psutil. 2024-04-02T04:31:27.601 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-02T04:31:27.610 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-psutil. 2024-04-02T04:31:27.638 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-02T04:31:27.639 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-02T04:31:27.677 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-02T04:31:27.983 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-py. 2024-04-02T04:31:27.996 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-py. 2024-04-02T04:31:28.012 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-02T04:31:28.024 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-02T04:31:28.050 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-02T04:31:28.071 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-02T04:31:28.387 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pygments. 2024-04-02T04:31:28.399 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pygments. 2024-04-02T04:31:28.415 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-02T04:31:28.427 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-02T04:31:28.453 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-02T04:31:28.466 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-02T04:31:28.929 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-02T04:31:28.956 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-02T04:31:28.957 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-02T04:31:28.985 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-02T04:31:28.995 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-02T04:31:29.032 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-02T04:31:29.306 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-toml. 2024-04-02T04:31:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-02T04:31:29.372 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-02T04:31:29.377 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-toml. 2024-04-02T04:31:29.405 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-02T04:31:29.443 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-02T04:31:29.675 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pytest. 2024-04-02T04:31:29.700 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-02T04:31:29.742 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-02T04:31:29.746 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pytest. 2024-04-02T04:31:29.774 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-02T04:31:29.812 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-02T04:31:30.104 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplejson. 2024-04-02T04:31:30.132 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-02T04:31:30.171 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-02T04:31:30.174 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplejson. 2024-04-02T04:31:30.193 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-02T04:31:30.232 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-02T04:31:30.440 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-02T04:31:30.468 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-02T04:31:30.506 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-02T04:31:30.544 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-02T04:31:30.572 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-02T04:31:30.610 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-02T04:31:31.319 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package radosgw. 2024-04-02T04:31:31.347 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../116-radosgw_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:31.347 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package radosgw. 2024-04-02T04:31:31.375 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../116-radosgw_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:31.385 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking radosgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:31.413 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking radosgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:32.163 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package rbd-fuse. 2024-04-02T04:31:32.175 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package rbd-fuse. 2024-04-02T04:31:32.179 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:32.203 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1979-g6cec793d-1jammy_amd64.deb ... 2024-04-02T04:31:32.221 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking rbd-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:32.233 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking rbd-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:32.500 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-02T04:31:32.521 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-02T04:31:32.752 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-02T04:31:32.779 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-02T04:31:32.878 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-02T04:31:32.905 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-02T04:31:33.133 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-02T04:31:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-02T04:31:33.446 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-02T04:31:33.465 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-02T04:31:33.907 INFO:teuthology.orchestra.run.smithi069.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-02T04:31:33.909 INFO:teuthology.orchestra.run.smithi098.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-02T04:31:33.989 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:34.108 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:34.531 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user cephadm....done 2024-04-02T04:31:34.721 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-02T04:31:34.891 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user cephadm....done 2024-04-02T04:31:35.047 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-02T04:31:35.318 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-02T04:31:35.341 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-02T04:31:35.557 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-02T04:31:35.567 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-02T04:31:35.683 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-02T04:31:35.701 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-02T04:31:35.943 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-02T04:31:35.952 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-02T04:31:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-02T04:31:36.203 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-02T04:31:36.329 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-02T04:31:36.329 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-02T04:31:36.622 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-02T04:31:36.639 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-02T04:31:36.966 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-02T04:31:37.000 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-02T04:31:37.218 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-02T04:31:37.268 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-02T04:31:37.494 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-02T04:31:37.535 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-02T04:31:37.745 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-02T04:31:37.778 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-02T04:31:38.004 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-02T04:31:38.038 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-02T04:31:38.297 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-02T04:31:38.331 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-02T04:31:38.548 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-argparse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:38.590 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-argparse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:38.783 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-02T04:31:38.866 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-02T04:31:38.894 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-02T04:31:39.025 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-02T04:31:39.028 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-02T04:31:39.152 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-02T04:31:39.154 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-02T04:31:39.278 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-02T04:31:39.413 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-02T04:31:39.529 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-02T04:31:39.780 INFO:teuthology.orchestra.run.smithi069.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-02T04:31:39.820 INFO:teuthology.orchestra.run.smithi069.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-02T04:31:39.886 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-02T04:31:39.899 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-02T04:31:39.927 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-02T04:31:39.990 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-02T04:31:40.009 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-02T04:31:40.116 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-02T04:31:40.377 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:40.485 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:40.503 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-02T04:31:40.610 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-02T04:31:40.771 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-02T04:31:40.903 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-02T04:31:41.030 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-02T04:31:41.171 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-02T04:31:41.340 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up zip (3.0-12build2) ... 2024-04-02T04:31:41.441 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-02T04:31:41.481 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up zip (3.0-12build2) ... 2024-04-02T04:31:41.607 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-02T04:31:42.102 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-02T04:31:42.317 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-02T04:31:42.361 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-02T04:31:42.487 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-02T04:31:42.577 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-02T04:31:42.613 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-02T04:31:42.703 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-02T04:31:42.846 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-02T04:31:42.905 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-02T04:31:43.138 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-02T04:31:43.265 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-02T04:31:43.491 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-02T04:31:43.506 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-02T04:31:43.741 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-02T04:31:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-02T04:31:44.117 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-02T04:31:44.135 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-02T04:31:44.402 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-02T04:31:44.478 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-02T04:31:44.729 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-02T04:31:44.745 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-02T04:31:44.847 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:44.996 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-02T04:31:45.097 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:45.165 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-idna (3.3-1) ... 2024-04-02T04:31:45.382 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-idna (3.3-1) ... 2024-04-02T04:31:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-02T04:31:45.674 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-02T04:31:46.597 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-02T04:31:46.764 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-02T04:31:46.832 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:31:46.941 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-02T04:31:47.016 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:31:47.151 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-02T04:31:47.183 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-02T04:31:47.309 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-02T04:31:47.409 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-02T04:31:47.435 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-02T04:31:47.552 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-02T04:31:47.679 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-02T04:31:47.737 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-02T04:31:47.971 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-02T04:31:47.996 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-02T04:31:48.222 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-02T04:31:48.264 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:31:48.365 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-02T04:31:48.465 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:31:48.591 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-02T04:31:48.649 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-02T04:31:48.901 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-02T04:31:48.901 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-02T04:31:49.168 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-02T04:31:49.177 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-02T04:31:49.428 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-02T04:31:49.437 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-02T04:31:49.679 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-02T04:31:49.696 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-02T04:31:49.947 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-02T04:31:49.963 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-02T04:31:50.089 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-02T04:31:50.207 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-02T04:31:50.308 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-02T04:31:50.374 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-02T04:31:50.475 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-02T04:31:50.567 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-02T04:31:50.694 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-02T04:31:50.759 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-02T04:31:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-02T04:31:51.068 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-02T04:31:51.271 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-02T04:31:51.328 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-02T04:31:51.454 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-02T04:31:51.555 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-02T04:31:51.580 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:51.669 INFO:teuthology.orchestra.run.smithi069.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-02T04:31:51.673 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-02T04:31:51.774 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-02T04:31:51.847 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-02T04:31:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-02T04:31:52.032 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-02T04:31:52.381 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-02T04:31:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-02T04:31:52.610 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-02T04:31:52.685 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-02T04:31:52.811 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-02T04:31:52.995 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-02T04:31:53.062 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:31:53.188 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-02T04:31:53.246 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-02T04:31:53.355 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-02T04:31:53.481 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-02T04:31:53.615 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-02T04:31:53.716 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-02T04:31:53.774 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-02T04:31:54.017 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-02T04:31:54.033 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-02T04:31:54.150 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-02T04:31:54.360 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-02T04:31:54.426 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-02T04:31:54.653 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-02T04:31:54.694 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-02T04:31:54.779 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-02T04:31:55.047 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-02T04:31:55.047 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-02T04:31:55.315 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-02T04:31:55.458 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-02T04:31:55.662 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-02T04:31:55.734 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-02T04:31:56.037 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-02T04:31:56.070 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-02T04:31:56.196 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librados2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:56.305 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:56.321 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-02T04:31:56.406 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-02T04:31:56.666 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-02T04:31:56.718 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:56.792 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librados2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:56.852 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libradosstriper1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:56.918 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:56.978 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-02T04:31:57.044 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-02T04:31:57.255 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librbd1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:57.323 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:57.364 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-modules-core (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:57.449 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libradosstriper1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:57.465 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:57.574 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-02T04:31:57.658 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-02T04:31:57.658 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-02T04:31:57.876 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librbd1 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.002 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-modules-core (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.117 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs-dev (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.128 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.234 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rados (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.360 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librgw2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.373 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-02T04:31:58.373 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-02T04:31:58.486 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-02T04:31:58.871 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs-dev (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:58.972 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rados (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:59.098 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librgw2 (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:59.224 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-02T04:31:59.689 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rbd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:59.790 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up rbd-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:31:59.916 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.042 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cephfs (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.168 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.461 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rbd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.587 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up rbd-fuse (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.713 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.736 INFO:teuthology.orchestra.run.smithi069.stdout:Adding group ceph....done 2024-04-02T04:32:00.831 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cephfs (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:00.949 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-common (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:01.104 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user ceph....done 2024-04-02T04:32:01.248 INFO:teuthology.orchestra.run.smithi069.stdout:Setting system user ceph properties....done 2024-04-02T04:32:01.266 INFO:teuthology.orchestra.run.smithi069.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-02T04:32:01.403 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-02T04:32:01.458 INFO:teuthology.orchestra.run.smithi098.stdout:Adding group ceph....done 2024-04-02T04:32:01.720 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-02T04:32:01.843 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user ceph....done 2024-04-02T04:32:01.962 INFO:teuthology.orchestra.run.smithi098.stdout:Setting system user ceph properties....done 2024-04-02T04:32:01.980 INFO:teuthology.orchestra.run.smithi098.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-02T04:32:02.119 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-02T04:32:02.211 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-test (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:02.337 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up radosgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:02.431 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-02T04:32:02.818 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-02T04:32:02.818 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-02T04:32:02.917 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-test (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:03.034 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up radosgw (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:03.264 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-base (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:03.514 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-02T04:32:03.515 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-02T04:32:03.586 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-02T04:32:04.022 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-base (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:04.032 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mds (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:04.253 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-02T04:32:04.253 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-02T04:32:04.373 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-02T04:32:04.742 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:04.849 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mds (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:04.970 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-02T04:32:04.970 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-02T04:32:05.066 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-02T04:32:05.066 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-02T04:32:05.469 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-osd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:05.593 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:05.755 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-02T04:32:05.755 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-02T04:32:05.801 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-02T04:32:05.801 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-02T04:32:06.254 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:06.335 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-osd (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:06.380 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:06.523 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mon (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:06.598 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-02T04:32:06.598 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-02T04:32:06.721 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-02T04:32:06.721 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-02T04:32:07.095 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:07.221 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:07.265 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:07.372 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mon (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:07.425 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-dashboard (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:07.581 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-02T04:32:07.581 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-02T04:32:07.711 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-volume (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:08.066 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-02T04:32:08.074 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-cephadm (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:08.225 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-02T04:32:08.343 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-dashboard (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:08.487 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-02T04:32:08.494 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-volume (19.0.0-1979-g6cec793d-1jammy) ... 2024-04-02T04:32:08.839 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-02T04:32:08.996 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-02T04:32:09.259 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-02T04:32:09.735 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-02T04:32:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-02T04:32:10.386 DEBUG:teuthology.parallel:result is None 2024-04-02T04:32:10.435 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-02T04:32:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-02T04:32:11.076 DEBUG:teuthology.parallel:result is None 2024-04-02T04:32:11.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:32:11.230 DEBUG:teuthology.orchestra.run.smithi069:> dpkg-query -W -f '${Version}' ceph 2024-04-02T04:32:11.250 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:32:11.251 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:32:11.251 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979-g6cec793d-1jammy is installed. 2024-04-02T04:32:11.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:32:11.472 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-query -W -f '${Version}' ceph 2024-04-02T04:32:11.492 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:32:11.492 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979-g6cec793d-1jammy 2024-04-02T04:32:11.492 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979-g6cec793d-1jammy is installed. 2024-04-02T04:32:11.494 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T04:32:11.494 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:32:11.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T04:32:11.508 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:32:11.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T04:32:11.548 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T04:32:11.548 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:32:11.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T04:32:11.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T04:32:11.618 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:32:11.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T04:32:11.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T04:32:11.685 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T04:32:11.685 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:32:11.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T04:32:11.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T04:32:11.754 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:32:11.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T04:32:11.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T04:32:11.821 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T04:32:11.821 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:32:11.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T04:32:11.835 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T04:32:11.890 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:32:11.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T04:32:11.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T04:32:11.957 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-02T04:32:12.060 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca', 'cephadm_mode': 'root'} 2024-04-02T04:32:12.060 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:32:12.061 INFO:tasks.cephadm:Cluster fsid is f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:32:12.061 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-02T04:32:12.061 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.69', 'mon.c': '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]', 'mon.b': '172.21.15.98'} 2024-04-02T04:32:12.061 INFO:tasks.cephadm:First mon is mon.a on smithi069 2024-04-02T04:32:12.061 INFO:tasks.cephadm:First mgr is y 2024-04-02T04:32:12.061 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-02T04:32:12.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo hostname $(hostname -s) 2024-04-02T04:32:12.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-04-02T04:32:12.090 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-02T04:32:12.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:32:12.241 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/54776/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g6cec793d', 'node_name': '172.21.2.2+braggi02', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979-g6cec793d-1jammy'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-04-01 18:21:29.623870', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-04-02T04:32:12.460 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 6cec793d9073f0594e2364b9b687ecaf22424cca from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:32:12.461 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-04-02T04:32:12.461 DEBUG:teuthology.orchestra.run.smithi069:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T04:32:12.973 INFO:teuthology.orchestra.run.smithi069.stdout:-rw-rw-r-- 1 ubuntu ubuntu 774691 Apr 2 04:32 /home/ubuntu/cephtest/cephadm 2024-04-02T04:32:12.973 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/6cec793d9073f0594e2364b9b687ecaf22424cca/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T04:32:13.669 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-rw-r-- 1 ubuntu ubuntu 774691 Apr 2 04:32 /home/ubuntu/cephtest/cephadm 2024-04-02T04:32:13.670 DEBUG:teuthology.orchestra.run.smithi069:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T04:32:13.678 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T04:32:13.700 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca on all hosts... 2024-04-02T04:32:13.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca pull 2024-04-02T04:32:13.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca pull 2024-04-02T04:32:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca... 2024-04-02T04:32:13.900 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca... 2024-04-02T04:33:26.409 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:33:26.409 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)", 2024-04-02T04:33:26.409 INFO:teuthology.orchestra.run.smithi069.stdout: "image_id": "78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808", 2024-04-02T04:33:26.410 INFO:teuthology.orchestra.run.smithi069.stdout: "repo_digests": [ 2024-04-02T04:33:26.410 INFO:teuthology.orchestra.run.smithi069.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae" 2024-04-02T04:33:26.410 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:33:26.410 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:33:31.860 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:33:31.860 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)", 2024-04-02T04:33:31.860 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808", 2024-04-02T04:33:31.861 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-04-02T04:33:31.861 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae" 2024-04-02T04:33:31.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:33:31.861 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:33:31.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph 2024-04-02T04:33:31.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-04-02T04:33:31.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /etc/ceph 2024-04-02T04:33:31.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-04-02T04:33:31.981 INFO:tasks.cephadm:Writing seed config... 2024-04-02T04:33:31.982 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-02T04:33:31.982 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-02T04:33:31.982 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-02T04:33:31.982 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-04-02T04:33:31.982 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-04-02T04:33:31.982 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-04-02T04:33:31.983 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-04-02T04:33:31.984 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-04-02T04:33:31.984 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-04-02T04:33:31.984 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-02T04:33:31.984 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-02T04:33:31.984 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-02T04:33:31.984 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-04-02T04:33:31.984 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:33:31.985 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-02T04:33:32.022 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f978400e-f0a9-11ee-b647-cb9ed24678a4 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-02T04:33:32.022 DEBUG:teuthology.orchestra.run.smithi069:mon.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service 2024-04-02T04:33:32.067 DEBUG:teuthology.orchestra.run.smithi069:mgr.y> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y.service 2024-04-02T04:33:32.110 INFO:tasks.cephadm:Bootstrapping... 2024-04-02T04:33:32.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca -v bootstrap --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.69 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:33:32.320 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-02T04:33:32.321 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca', '-v', 'bootstrap', '--fsid', 'f978400e-f0a9-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.69', '--skip-admin-label'] 2024-04-02T04:33:32.321 INFO:teuthology.orchestra.run.smithi069.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-02T04:33:32.321 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying podman|docker is present... 2024-04-02T04:33:32.322 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying lvm2 is present... 2024-04-02T04:33:32.322 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying time synchronization is in place... 2024-04-02T04:33:32.326 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-02T04:33:32.327 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-02T04:33:32.330 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-02T04:33:32.330 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.334 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-02T04:33:32.334 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-02T04:33:32.338 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-02T04:33:32.338 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.340 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-02T04:33:32.341 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout masked 2024-04-02T04:33:32.344 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-02T04:33:32.344 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.347 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-02T04:33:32.347 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-02T04:33:32.351 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-02T04:33:32.351 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.355 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout enabled 2024-04-02T04:33:32.359 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout active 2024-04-02T04:33:32.359 INFO:teuthology.orchestra.run.smithi069.stdout:Unit ntp.service is enabled and running 2024-04-02T04:33:32.359 INFO:teuthology.orchestra.run.smithi069.stdout:Repeating the final host check... 2024-04-02T04:33:32.359 INFO:teuthology.orchestra.run.smithi069.stdout:docker (/usr/bin/docker) is present 2024-04-02T04:33:32.360 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl is present 2024-04-02T04:33:32.360 INFO:teuthology.orchestra.run.smithi069.stdout:lvcreate is present 2024-04-02T04:33:32.363 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-02T04:33:32.363 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-02T04:33:32.366 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-02T04:33:32.366 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.370 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-02T04:33:32.370 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-02T04:33:32.373 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-02T04:33:32.373 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.377 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-02T04:33:32.377 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout masked 2024-04-02T04:33:32.381 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-02T04:33:32.381 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.385 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-02T04:33:32.385 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-02T04:33:32.389 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-02T04:33:32.389 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-04-02T04:33:32.393 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout enabled 2024-04-02T04:33:32.397 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout active 2024-04-02T04:33:32.397 INFO:teuthology.orchestra.run.smithi069.stdout:Unit ntp.service is enabled and running 2024-04-02T04:33:32.397 INFO:teuthology.orchestra.run.smithi069.stdout:Host looks OK 2024-04-02T04:33:32.398 INFO:teuthology.orchestra.run.smithi069.stdout:Cluster fsid: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:32.398 INFO:teuthology.orchestra.run.smithi069.stdout:Acquiring lock 140450161127152 on /run/cephadm/f978400e-f0a9-11ee-b647-cb9ed24678a4.lock 2024-04-02T04:33:32.398 INFO:teuthology.orchestra.run.smithi069.stdout:Lock 140450161127152 acquired on /run/cephadm/f978400e-f0a9-11ee-b647-cb9ed24678a4.lock 2024-04-02T04:33:32.398 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying IP 172.21.15.69 port 3300 ... 2024-04-02T04:33:32.398 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying IP 172.21.15.69 port 6789 ... 2024-04-02T04:33:32.398 INFO:teuthology.orchestra.run.smithi069.stdout:Base mon IP(s) is [172.21.15.69:3300, 172.21.15.69:6789], mon addrv is [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2024-04-02T04:33:32.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.69 metric 100 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 158.69.67.47 via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.69 metric 100 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.69 metric 100 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 172.21.0.1 dev enp3s0f1 proto dhcp scope link src 172.21.15.69 metric 100 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 172.21.0.2 dev enp3s0f1 proto dhcp scope link src 172.21.15.69 metric 100 2024-04-02T04:33:32.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.69 metric 100 2024-04-02T04:33:32.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-02T04:33:32.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-02T04:33:32.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1785sec pref medium 2024-04-02T04:33:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-02T04:33:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-02T04:33:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-02T04:33:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-02T04:33:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-02T04:33:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-02T04:33:32.406 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2024-04-02T04:33:32.407 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2024-04-02T04:33:32.407 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.1/32` 2024-04-02T04:33:32.407 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.1/32` 2024-04-02T04:33:32.407 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.2/32` 2024-04-02T04:33:32.407 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.2/32` 2024-04-02T04:33:32.407 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.15.254/32` 2024-04-02T04:33:32.408 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.15.254/32` 2024-04-02T04:33:32.408 INFO:teuthology.orchestra.run.smithi069.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20', '172.21.0.1/32', '172.21.0.1/32', '172.21.0.2/32', '172.21.0.2/32', '172.21.15.254/32', '172.21.15.254/32'] 2024-04-02T04:33:32.408 INFO:teuthology.orchestra.run.smithi069.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-02T04:33:32.408 INFO:teuthology.orchestra.run.smithi069.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca... 2024-04-02T04:33:32.622 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/docker: stdout 6cec793d9073f0594e2364b9b687ecaf22424cca: Pulling from ceph-ci/ceph 2024-04-02T04:33:32.622 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/docker: stdout Digest: sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:33:32.622 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:33:32.622 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:33:33.945 INFO:teuthology.orchestra.run.smithi069.stdout:ceph: stdout ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev) 2024-04-02T04:33:33.946 INFO:teuthology.orchestra.run.smithi069.stdout:Ceph version: ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev) 2024-04-02T04:33:33.946 INFO:teuthology.orchestra.run.smithi069.stdout:Extracting ceph user uid/gid from container image... 2024-04-02T04:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout:stat: stdout 167 167 2024-04-02T04:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial keys... 2024-04-02T04:33:36.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQCfigtmyaNGLRAAwXOLl2JQKHehjG8CLqh0mA== 2024-04-02T04:33:37.458 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQCgigtmqgyQOxAAuXplz3G/BG67LjHZI6C+CA== 2024-04-02T04:33:38.563 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQCiigtmaUk8BBAAd5wDeav3dEo903XxH9bvXA== 2024-04-02T04:33:38.564 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial monmap... 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool for a [v2:172.21.15.69:3300,v1:172.21.15.69:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = quincy 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: set fsid to f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:33:39.719 INFO:teuthology.orchestra.run.smithi069.stdout:Creating mon... 2024-04-02T04:33:41.037 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 1 imported monmap: 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-04-02T04:33:39.262476+0000 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 0 /usr/bin/ceph-mon: set fsid to f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.038 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Git sha 0 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Compile date 2024-04-01 15:23:26 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: DB SUMMARY 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: DB Session ID: 9VIU044BJTKAY9ARV699 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.error_if_exists: 0 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.create_if_missing: 1 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-02T04:33:41.039 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.env: 0x55f947f48c60 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.info_log: 0x55f948c3ed60 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.statistics: (nil) 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.use_fsync: 0 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-02T04:33:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.db_log_dir: 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.wal_dir: 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T04:33:41.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.write_buffer_manager: 0x55f948c13360 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.unordered_write: 0 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T04:33:41.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.row_cache: None 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.wal_filter: None 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.two_write_queues: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.wal_compression: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.atomic_flush: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T04:33:41.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T04:33:41.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_open_files: -1 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Compression algorithms supported: 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kZSTD supported: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kXpressCompression supported: 0 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kZlibCompression supported: 1 2024-04-02T04:33:41.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.454+0000 7f258c822c80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.merge_operator: 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_filter: None 2024-04-02T04:33:41.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f948c3ee60) 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-04-02T04:33:41.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x55f948c43090 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-04-02T04:33:41.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-02T04:33:41.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression: NoCompression 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.num_levels: 7 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T04:33:41.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T04:33:41.051 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T04:33:41.052 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T04:33:41.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.table_properties_collectors: 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.bloom_locality: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-02T04:33:41.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.ttl: 2592000 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.enable_blob_files: false 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.min_blob_size: 0 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-02T04:33:41.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c60e1548-2e29-45c3-8a17-5994d342fb89 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f948d0e000 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f258c822c80 4 rocksdb: DB pointer 0x55f948cfe000 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f257c98c700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.458+0000 7f257c98c700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-02T04:33:41.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-02T04:33:41.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:33:41.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x55f948c43090#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.462+0000 7f258c822c80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.462+0000 7f258c822c80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-02T04:33:40.462+0000 7f258c822c80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-04-02T04:33:41.059 INFO:teuthology.orchestra.run.smithi069.stdout:create mon.a on 2024-04-02T04:33:41.304 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-02T04:33:41.495 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-04-02T04:33:41.710 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4.target -> /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4.target. 2024-04-02T04:33:41.710 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4.target -> /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4.target. 2024-04-02T04:33:41.997 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a 2024-04-02T04:33:41.997 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-04-02T04:33:42.196 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4.target.wants/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service -> /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service. 2024-04-02T04:33:42.204 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld does not appear to be present 2024-04-02T04:33:42.204 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to enable service . firewalld.service is not available 2024-04-02T04:33:42.204 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mon to start... 2024-04-02T04:33:42.204 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mon... 2024-04-02T04:33:43.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:43 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.344271+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout cluster: 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout id: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout services: 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.189531s) 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout data: 2024-04-02T04:33:44.041 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout pgs: 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecfa214700 1 Processor -- start 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecfa214700 1 -- start start 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecfa214700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf41051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecfa214700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4105780 con 0x7fecf4104d90 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecf37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf41051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecf37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf41051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49756/0 (socket says 172.21.15.69:49756) 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecf37fe700 1 -- 172.21.15.69:0/2093334254 learned_addr learned my addr 172.21.15.69:0/2093334254 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.398+0000 7fecf37fe700 1 -- 172.21.15.69:0/2093334254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf4105f90 con 0x7fecf4104d90 2024-04-02T04:33:44.042 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf37fe700 1 --2- 172.21.15.69:0/2093334254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf41051b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fece40098a0 tx=0x7fece4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189d9d3545022cde server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf27fc700 1 -- 172.21.15.69:0/2093334254 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fece4004160 con 0x7fecf4104d90 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf27fc700 1 -- 172.21.15.69:0/2093334254 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece4004300 con 0x7fecf4104d90 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf27fc700 1 -- 172.21.15.69:0/2093334254 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fece40036a0 con 0x7fecf4104d90 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/2093334254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 msgr2=0x7fecf41051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 --2- 172.21.15.69:0/2093334254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf41051b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fece40098a0 tx=0x7fece4009bb0 comp rx=0 tx=0).stop 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/2093334254 shutdown_connections 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 --2- 172.21.15.69:0/2093334254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf41051b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/2093334254 >> 172.21.15.69:0/2093334254 conn(0x7fecf4100310 msgr2=0x7fecf4102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/2093334254 shutdown_connections 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/2093334254 wait complete. 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 librados: starting msgr at 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 librados: starting objecter 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 Processor -- start 2024-04-02T04:33:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- start start 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 librados: setting wanted keys 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 librados: calling monclient init 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf419bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fecf40746a0 con 0x7fecf4104d90 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf419bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf419bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49758/0 (socket says 172.21.15.69:49758) 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf37fe700 1 -- 172.21.15.69:0/260897128 learned_addr learned my addr 172.21.15.69:0/260897128 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf37fe700 1 -- 172.21.15.69:0/260897128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf419c4a0 con 0x7fecf4104d90 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf37fe700 1 --2- 172.21.15.69:0/260897128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf419bf60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 10 client.?.objecter ms_handle_connect 0x7fecf4104d90 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fece4003db0 con 0x7fecf4104d90 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece4011630 con 0x7fecf4104d90 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fece4011950 con 0x7fecf4104d90 2024-04-02T04:33:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf419c730 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf419f420 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 librados: init done 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fece401a460 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fece401b780 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 10 client.4098.objecter ms_dispatch 0x7fecf4001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecf0ff9700 20 client.4098.objecter dump_active .. 0 homeless 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.402+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecf4063870 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.406+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fece4031020 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fecf419f780 con 0x7fecf4104d90 2024-04-02T04:33:44.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecf0ff9700 1 -- 172.21.15.69:0/260897128 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fece401b9d0 con 0x7fecf4104d90 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 10 librados: watch_flush enter 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 10 librados: watch_flush exit 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 20 client.4098.objecter shutdown clearing up homeless session... 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 10 client.4098.objecter successfully canceled tick 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 msgr2=0x7fecf419bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 --2- 172.21.15.69:0/260897128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf419bf60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece4004750 comp rx=0 tx=0).stop 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 shutdown_connections 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 --2- 172.21.15.69:0/260897128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf4104d90 0x7fecf419bf60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 >> 172.21.15.69:0/260897128 conn(0x7fecf4100310 msgr2=0x7fecf418c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 shutdown_connections 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 -- 172.21.15.69:0/260897128 wait complete. 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:43.534+0000 7fecfa214700 1 librados: shutdown 2024-04-02T04:33:44.046 INFO:teuthology.orchestra.run.smithi069.stdout:mon is available 2024-04-02T04:33:44.047 INFO:teuthology.orchestra.run.smithi069.stdout:Assimilating anything we can from ceph.conf... 2024-04-02T04:33:44.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347569+0000 mon.a (mon.0) 2 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347754+0000 mon.a (mon.0) 3 : cluster 0 monmap epoch 1 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347768+0000 mon.a (mon.0) 4 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347777+0000 mon.a (mon.0) 5 : cluster 0 last_changed 2024-04-02T04:33:39.262476+0000 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347787+0000 mon.a (mon.0) 6 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347798+0000 mon.a (mon.0) 7 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347820+0000 mon.a (mon.0) 8 : cluster 0 election_strategy: 1 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.347832+0000 mon.a (mon.0) 9 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.349830+0000 mon.a (mon.0) 10 : cluster 0 fsmap 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.352329+0000 mon.a (mon.0) 11 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: cluster 2024-04-02T04:33:43.353005+0000 mon.a (mon.0) 12 : cluster 0 mgrmap e1: no daemons active 2024-04-02T04:33:44.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:44 smithi069 bash[21040]: audit 2024-04-02T04:33:43.537046+0000 mon.a (mon.0) 13 : audit 0 from='client.? 172.21.15.69:0/260897128' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [global] 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout fsid = f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-02T04:33:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [osd] 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 Processor -- start 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- start start 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f28105410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2807a6a0 con 0x7f2f28104ff0 2024-04-02T04:33:45.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f26ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f28105410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f26ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f28105410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49762/0 (socket says 172.21.15.69:49762) 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f26ffd700 1 -- 172.21.15.69:0/3462604116 learned_addr learned my addr 172.21.15.69:0/3462604116 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f26ffd700 1 -- 172.21.15.69:0/3462604116 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f28105950 con 0x7f2f28104ff0 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f26ffd700 1 --2- 172.21.15.69:0/3462604116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f28105410 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2f180098a0 tx=0x7f2f18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3b0e127f464d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f25ffb700 1 -- 172.21.15.69:0/3462604116 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f18004160 con 0x7f2f28104ff0 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f25ffb700 1 -- 172.21.15.69:0/3462604116 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f18004300 con 0x7f2f28104ff0 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f25ffb700 1 -- 172.21.15.69:0/3462604116 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f180036a0 con 0x7f2f28104ff0 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- 172.21.15.69:0/3462604116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 msgr2=0x7f2f28105410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 --2- 172.21.15.69:0/3462604116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f28105410 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2f180098a0 tx=0x7f2f18009bb0 comp rx=0 tx=0).stop 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- 172.21.15.69:0/3462604116 shutdown_connections 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 --2- 172.21.15.69:0/3462604116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f28105410 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- 172.21.15.69:0/3462604116 >> 172.21.15.69:0/3462604116 conn(0x7f2f28100b50 msgr2=0x7f2f28102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:45.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- 172.21.15.69:0/3462604116 shutdown_connections 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- 172.21.15.69:0/3462604116 wait complete. 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 librados: starting msgr at 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 librados: starting objecter 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 Processor -- start 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- start start 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 librados: setting wanted keys 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 librados: calling monclient init 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f2819bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.781+0000 7f2f2da57700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2f28105e80 con 0x7f2f28104ff0 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f26ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f2819bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f26ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f2819bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49778/0 (socket says 172.21.15.69:49778) 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f26ffd700 1 -- 172.21.15.69:0/1415590373 learned_addr learned my addr 172.21.15.69:0/1415590373 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f26ffd700 1 -- 172.21.15.69:0/1415590373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2819c410 con 0x7f2f28104ff0 2024-04-02T04:33:45.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f26ffd700 1 --2- 172.21.15.69:0/1415590373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f2819bed0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2f18000c00 tx=0x7f2f18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 10 client.?.objecter ms_handle_connect 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f18003db0 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f18011630 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f18011950 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2819c6a0 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2da57700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2819f390 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2da57700 1 librados: init done 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2f1801a460 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f1801b990 con 0x7f2f28104ff0 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 10 client.4100.objecter ms_dispatch 0x7f2f28001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:45.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2ca55700 20 client.4100.objecter dump_active .. 0 homeless 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.785+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f28063870 con 0x7f2f28104ff0 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.789+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2f1801bb90 con 0x7f2f28104ff0 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.909+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2f281a0930 con 0x7f2f28104ff0 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+338 (secure 0 0 0) 0x7f2f1801bd70 con 0x7f2f28104ff0 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2da57700 10 librados: watch_flush enter 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2da57700 10 librados: watch_flush exit 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2da57700 20 client.4100.objecter shutdown clearing up homeless session... 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2da57700 10 client.4100.objecter successfully canceled tick 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2ca55700 1 -- 172.21.15.69:0/1415590373 <== mon.0 v2:172.21.15.69:3300/0 8 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f2f1801abb0 con 0x7f2f28104ff0 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 msgr2=0x7f2f2819bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:45.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.913+0000 7f2f2da57700 1 --2- 172.21.15.69:0/1415590373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f2819bed0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2f18000c00 tx=0x7f2f18004750 comp rx=0 tx=0).stop 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.917+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 shutdown_connections 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.917+0000 7f2f2da57700 1 --2- 172.21.15.69:0/1415590373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f28104ff0 0x7f2f2819bed0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.917+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 >> 172.21.15.69:0/1415590373 conn(0x7f2f28100b50 msgr2=0x7f2f28192510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.917+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 shutdown_connections 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.917+0000 7f2f2da57700 1 -- 172.21.15.69:0/1415590373 wait complete. 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:44.917+0000 7f2f2da57700 1 librados: shutdown 2024-04-02T04:33:45.468 INFO:teuthology.orchestra.run.smithi069.stdout:Generating new minimal ceph.conf... 2024-04-02T04:33:45.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:45 smithi069 bash[21040]: audit 2024-04-02T04:33:44.913627+0000 mon.a (mon.0) 14 : audit 1 from='client.? 172.21.15.69:0/1415590373' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-02T04:33:45.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:45 smithi069 bash[21040]: audit 2024-04-02T04:33:44.918211+0000 mon.a (mon.0) 15 : audit 1 from='client.? 172.21.15.69:0/1415590373' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-02T04:33:46.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:46 smithi069 bash[21040]: audit 2024-04-02T04:33:46.408443+0000 mon.a (mon.0) 16 : audit 0 from='client.? 172.21.15.69:0/2918911876' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:33:46.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.273+0000 7f5192978700 1 Processor -- start 2024-04-02T04:33:46.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- start start 2024-04-02T04:33:46.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c109580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:46.902 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f518c07a740 con 0x7f518c107190 2024-04-02T04:33:46.902 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:46.902 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49782/0 (socket says 172.21.15.69:49782) 2024-04-02T04:33:46.902 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 -- 172.21.15.69:0/4025920124 learned_addr learned my addr 172.21.15.69:0/4025920124 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:46.908 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 -- 172.21.15.69:0/4025920124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c109ac0 con 0x7f518c107190 2024-04-02T04:33:46.908 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 --2- 172.21.15.69:0/4025920124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c109580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f51840098a0 tx=0x7f5184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06c2ea17b253367 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:46.908 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518affd700 1 -- 172.21.15.69:0/4025920124 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5184004160 con 0x7f518c107190 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518affd700 1 -- 172.21.15.69:0/4025920124 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f5184004300 con 0x7f518c107190 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518affd700 1 -- 172.21.15.69:0/4025920124 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5184011960 con 0x7f518c107190 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- 172.21.15.69:0/4025920124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 msgr2=0x7f518c109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 --2- 172.21.15.69:0/4025920124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c109580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f51840098a0 tx=0x7f5184009bb0 comp rx=0 tx=0).stop 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- 172.21.15.69:0/4025920124 shutdown_connections 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 --2- 172.21.15.69:0/4025920124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c109580 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- 172.21.15.69:0/4025920124 >> 172.21.15.69:0/4025920124 conn(0x7f518c100bc0 msgr2=0x7f518c103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- 172.21.15.69:0/4025920124 shutdown_connections 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- 172.21.15.69:0/4025920124 wait complete. 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 librados: starting msgr at 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 librados: starting objecter 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 Processor -- start 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- start start 2024-04-02T04:33:46.909 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 librados: setting wanted keys 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 librados: calling monclient init 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c106a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f518c10aa30 con 0x7f518c107190 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49786/0 (socket says 172.21.15.69:49786) 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 -- 172.21.15.69:0/2918911876 learned_addr learned my addr 172.21.15.69:0/2918911876 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 -- 172.21.15.69:0/2918911876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c106f50 con 0x7f518c107190 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f518bfff700 1 --2- 172.21.15.69:0/2918911876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c106a10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5184011740 tx=0x7f5184003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f51897fa700 10 client.?.objecter ms_handle_connect 0x7f518c107190 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f51897fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51840038b0 con 0x7f518c107190 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518c105140 con 0x7f518c107190 2024-04-02T04:33:46.910 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f51840248a0 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.277+0000 7f5192978700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f518401a6b0 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f518c105680 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f5192978700 1 librados: init done 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f51840193c0 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5184024490 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 10 client.4102.objecter ms_dispatch 0x7f518c001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 20 client.4102.objecter dump_active .. 0 homeless 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f518c063870 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.281+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f518400b3e0 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f518c105900 con 0x7f518c107190 2024-04-02T04:33:46.911 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f51897fa700 1 -- 172.21.15.69:0/2918911876 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f5184024690 con 0x7f518c107190 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 10 librados: watch_flush enter 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 10 librados: watch_flush exit 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 20 client.4102.objecter shutdown clearing up homeless session... 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 10 client.4102.objecter successfully canceled tick 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 msgr2=0x7f518c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 --2- 172.21.15.69:0/2918911876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c106a10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5184011740 tx=0x7f5184003dc0 comp rx=0 tx=0).stop 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 shutdown_connections 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 --2- 172.21.15.69:0/2918911876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f518c107190 0x7f518c106a10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 >> 172.21.15.69:0/2918911876 conn(0x7f518c100bc0 msgr2=0x7f518c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 shutdown_connections 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 -- 172.21.15.69:0/2918911876 wait complete. 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:46.405+0000 7f5192978700 1 librados: shutdown 2024-04-02T04:33:46.912 INFO:teuthology.orchestra.run.smithi069.stdout:Restarting the monitor... 2024-04-02T04:33:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:46 smithi069 systemd[1]: Stopping Ceph mon.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T04:33:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 bash[21040]: debug 2024-04-02T04:33:47.013+0000 7f2a976fe700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-02T04:33:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 bash[21040]: debug 2024-04-02T04:33:47.013+0000 7f2a976fe700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-02T04:33:47.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 bash[21424]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-mon-a 2024-04-02T04:33:47.643 INFO:teuthology.orchestra.run.smithi069.stdout:Setting public_network to 172.21.0.0/20,172.21.0.1/32,172.21.0.2/32,172.21.15.254/32 in mon config section 2024-04-02T04:33:48.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 bash[21486]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-mon-a 2024-04-02T04:33:48.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-02T04:33:48.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 systemd[1]: Stopped Ceph mon.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:33:48.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:47 smithi069 systemd[1]: Started Ceph mon.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:33:48.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.629+0000 7f0242c6fc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-02T04:33:48.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.629+0000 7f0242c6fc80 0 ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev), process ceph-mon, pid 7 2024-04-02T04:33:48.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.629+0000 7f0242c6fc80 0 pidfile_write: ignore empty --pid-file 2024-04-02T04:33:48.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 0 load: jerasure load: lrc 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: RocksDB version: 7.9.2 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Git sha 0 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Compile date 2024-04-01 15:23:26 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: DB SUMMARY 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: DB Session ID: T7FPW2UAS8G7L1TNFC5T 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: CURRENT file: CURRENT 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 96701 ; 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.error_if_exists: 0 2024-04-02T04:33:48.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.create_if_missing: 0 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.paranoid_checks: 1 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.env: 0x55d9413edc60 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.info_log: 0x55d942c35b20 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.statistics: (nil) 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.use_fsync: 0 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_log_file_size: 0 2024-04-02T04:33:48.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.allow_fallocate: 1 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.use_direct_reads: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.db_log_dir: 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.wal_dir: 2024-04-02T04:33:48.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.write_buffer_manager: 0x55d942c081e0 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T04:33:48.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.unordered_write: 0 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.row_cache: None 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.wal_filter: None 2024-04-02T04:33:48.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.two_write_queues: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.wal_compression: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.atomic_flush: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.log_readahead_size: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-02T04:33:48.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T04:33:48.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T04:33:48.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-02T04:33:48.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-02T04:33:48.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-02T04:33:48.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_background_jobs: 2 2024-04-02T04:33:48.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_background_compactions: -1 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_subcompactions: 1 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_open_files: -1 2024-04-02T04:33:48.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_background_flushes: -1 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Compression algorithms supported: 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kZSTD supported: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kXpressCompression supported: 0 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kZlibCompression supported: 1 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kSnappyCompression supported: 1 2024-04-02T04:33:48.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kLZ4Compression supported: 1 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: kBZip2Compression supported: 0 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.merge_operator: 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_filter: None 2024-04-02T04:33:48.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d942c35c40) 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cache_index_and_filter_blocks: 1 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: pin_top_level_index_and_filter: 1 2024-04-02T04:33:48.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: index_type: 0 2024-04-02T04:33:48.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: data_block_index_type: 0 2024-04-02T04:33:48.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: index_shortening: 1 2024-04-02T04:33:48.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: data_block_hash_table_util_ratio: 0.750000 2024-04-02T04:33:48.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: checksum: 4 2024-04-02T04:33:48.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: no_block_cache: 0 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_cache: 0x55d942c37090 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_cache_name: BinnedLRUCache 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_cache_options: 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: capacity : 536870912 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: num_shard_bits : 4 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: strict_capacity_limit : 0 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: high_pri_pool_ratio: 0.000 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_cache_compressed: (nil) 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: persistent_cache: (nil) 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_size: 4096 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_size_deviation: 10 2024-04-02T04:33:48.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_restart_interval: 16 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: index_block_restart_interval: 1 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: metadata_block_size: 4096 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: partition_filters: 0 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: use_delta_encoding: 1 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: filter_policy: bloomfilter 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: whole_key_filtering: 1 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: verify_compression: 0 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: read_amp_bytes_per_bit: 0 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: format_version: 5 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: enable_index_compression: 1 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: block_align: 0 2024-04-02T04:33:48.939 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: max_auto_readahead_size: 262144 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: prepopulate_block_cache: 0 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: initial_auto_readahead_size: 8192 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: num_file_reads_for_auto_readahead: 2 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression: NoCompression 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.num_levels: 7 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T04:33:48.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T04:33:48.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T04:33:48.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T04:33:48.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T04:33:48.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.table_properties_collectors: 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.inplace_update_support: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.bloom_locality: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.max_successive_merges: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T04:33:48.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.ttl: 2592000 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.enable_blob_files: false 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.min_blob_size: 0 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T04:33:48.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c60e1548-2e29-45c3-8a17-5994d342fb89 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032428642330, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032428643408, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92420, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 292, "table_properties": {"data_size": 90494, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12206, "raw_average_key_size": 48, "raw_value_size": 83837, "raw_average_value_size": 334, "num_data_blocks": 12, "num_entries": 251, "num_filter_entries": 251, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712032428, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c60e1548-2e29-45c3-8a17-5994d342fb89", "db_session_id": "T7FPW2UAS8G7L1TNFC5T", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032428643497, "job": 1, "event": "recovery_finished"} 2024-04-02T04:33:48.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.637+0000 7f0242c6fc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d942d2c000 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 4 rocksdb: DB pointer 0x55d942d16000 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f02313d1700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f02313d1700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: ** DB Stats ** 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:33:48.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: ** Compaction Stats [default] ** 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: L0 2/0 92.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Sum 2/0 92.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: ** Compaction Stats [default] ** 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:33:48.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Cumulative compaction: 0.00 GB write, 24.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Interval compaction: 0.00 GB write, 24.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Block cache BinnedLRUCache@0x55d942c37090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-02T04:33:48.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] at bind addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 1 mon.a@-1(???) e1 preinit fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 mon.a@-1(???).mds e1 new map 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 mon.a@-1(???).mds e1 print_map 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: e1 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: legacy client fscid: -1 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: No filesystems configured 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-02T04:33:48.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: debug 2024-04-02T04:33:48.641+0000 7f0242c6fc80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668394+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668499+0000 mon.a (mon.0) 2 : cluster 0 monmap epoch 1 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668516+0000 mon.a (mon.0) 3 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668525+0000 mon.a (mon.0) 4 : cluster 0 last_changed 2024-04-02T04:33:39.262476+0000 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668532+0000 mon.a (mon.0) 5 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668546+0000 mon.a (mon.0) 6 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668554+0000 mon.a (mon.0) 7 : cluster 0 election_strategy: 1 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668564+0000 mon.a (mon.0) 8 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:33:48.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668967+0000 mon.a (mon.0) 9 : cluster 0 fsmap 2024-04-02T04:33:48.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.668994+0000 mon.a (mon.0) 10 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-04-02T04:33:48.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:48 smithi069 bash[21528]: cluster 2024-04-02T04:33:48.669389+0000 mon.a (mon.0) 11 : cluster 0 mgrmap e1: no daemons active 2024-04-02T04:33:49.476 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.768+0000 7facc1b6d700 1 Processor -- start 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.768+0000 7facc1b6d700 1 -- start start 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc105a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7facbc106000 con 0x7facbc105610 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc105a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc105a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49356/0 (socket says 172.21.15.69:49356) 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 -- 172.21.15.69:0/240991757 learned_addr learned my addr 172.21.15.69:0/240991757 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:49.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 -- 172.21.15.69:0/240991757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facbc106810 con 0x7facbc105610 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 --2- 172.21.15.69:0/240991757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc105a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7facb4009e80 tx=0x7facb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd6c0f8b8d6b97b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facba7fc700 1 -- 172.21.15.69:0/240991757 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facb4017070 con 0x7facbc105610 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facba7fc700 1 -- 172.21.15.69:0/240991757 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7facb400b8c0 con 0x7facbc105610 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facba7fc700 1 -- 172.21.15.69:0/240991757 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facb4004340 con 0x7facbc105610 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/240991757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 msgr2=0x7facbc105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 --2- 172.21.15.69:0/240991757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc105a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7facb4009e80 tx=0x7facb400b330 comp rx=0 tx=0).stop 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/240991757 shutdown_connections 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 --2- 172.21.15.69:0/240991757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc105a30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/240991757 >> 172.21.15.69:0/240991757 conn(0x7facbc100bd0 msgr2=0x7facbc102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:49.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/240991757 shutdown_connections 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/240991757 wait complete. 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 librados: starting msgr at 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 librados: starting objecter 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 Processor -- start 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- start start 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 librados: setting wanted keys 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 librados: calling monclient init 2024-04-02T04:33:49.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7facbc109a40 con 0x7facbc105610 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc19a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49362/0 (socket says 172.21.15.69:49362) 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 -- 172.21.15.69:0/3590324417 learned_addr learned my addr 172.21.15.69:0/3590324417 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 -- 172.21.15.69:0/3590324417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facbc19a9d0 con 0x7facbc105610 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facbb7fe700 1 --2- 172.21.15.69:0/3590324417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc19a490 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7facb4009530 tx=0x7facb400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facb8ff9700 10 client.?.objecter ms_handle_connect 0x7facbc105610 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facb8ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facb4021020 con 0x7facbc105610 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7facb4003b50 con 0x7facbc105610 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facb4017040 con 0x7facbc105610 2024-04-02T04:33:49.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facbc19ac60 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facbc19d950 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.772+0000 7facc1b6d700 1 librados: init done 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7facb4024020 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7facb400f050 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 10 client.14098.objecter ms_dispatch 0x7facbc001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 20 client.14098.objecter dump_active .. 0 homeless 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facbc063870 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.776+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7facb402b020 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.896+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7facbc02dbd0 con 0x7facbc105610 2024-04-02T04:33:49.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facb8ff9700 1 -- 172.21.15.69:0/3590324417 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 169+0+0 (secure 0 0 0) 0x7facb400b860 con 0x7facbc105610 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 10 librados: watch_flush enter 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 10 librados: watch_flush exit 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 20 client.14098.objecter shutdown clearing up homeless session... 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 10 client.14098.objecter successfully canceled tick 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 msgr2=0x7facbc19a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 1 --2- 172.21.15.69:0/3590324417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc19a490 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7facb4009530 tx=0x7facb400bf10 comp rx=0 tx=0).stop 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 shutdown_connections 2024-04-02T04:33:49.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 1 --2- 172.21.15.69:0/3590324417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facbc105610 0x7facbc19a490 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:49.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.904+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 >> 172.21.15.69:0/3590324417 conn(0x7facbc100bd0 msgr2=0x7facbc191020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:49.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.908+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 shutdown_connections 2024-04-02T04:33:49.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.908+0000 7facc1b6d700 1 -- 172.21.15.69:0/3590324417 wait complete. 2024-04-02T04:33:49.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:48.908+0000 7facc1b6d700 1 librados: shutdown 2024-04-02T04:33:49.483 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-02T04:33:49.504 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:33:49.504 INFO:teuthology.orchestra.run.smithi069.stdout:Creating mgr... 2024-04-02T04:33:49.505 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-02T04:33:49.505 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-02T04:33:49.810 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y 2024-04-02T04:33:49.810 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y.service: Unit ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y.service not loaded. 2024-04-02T04:33:49.814 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:49 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:33:49.814 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:49 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4.target.wants/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y.service -> /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service. 2024-04-02T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld does not appear to be present 2024-04-02T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to enable service . firewalld.service is not available 2024-04-02T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld does not appear to be present 2024-04-02T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-02T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr to start... 2024-04-02T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr... 2024-04-02T04:33:50.083 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:49 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:33:50.083 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:50 smithi069 systemd[1]: Started Ceph mgr.y for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:33:50.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:49 smithi069 bash[21528]: audit 2024-04-02T04:33:48.909274+0000 mon.a (mon.0) 12 : audit 1 from='client.? 172.21.15.69:0/3590324417' entity='client.admin' 2024-04-02T04:33:50.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:49 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:33:51.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:51 smithi069 bash[21528]: audit 2024-04-02T04:33:51.350192+0000 mon.a (mon.0) 13 : audit 0 from='client.? 172.21.15.69:0/3221386943' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:33:51.509 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:51 smithi069 bash[21782]: debug 2024-04-02T04:33:51.232+0000 7fe51db43200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T04:33:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:33:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T04:33:43.348904+0000", 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:33:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.208+0000 7fc1276bb700 1 Processor -- start 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.208+0000 7fc1276bb700 1 -- start start 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.208+0000 7fc1276bb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12007ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.208+0000 7fc1276bb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc12007b160 con 0x7fc12007c730 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12007ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12007ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49398/0 (socket says 172.21.15.69:49398) 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 -- 172.21.15.69:0/1017602041 learned_addr learned my addr 172.21.15.69:0/1017602041 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 -- 172.21.15.69:0/1017602041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12007b2e0 con 0x7fc12007c730 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 --2- 172.21.15.69:0/1017602041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12007ab90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0098a0 tx=0x7fc11c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b20f13012f29b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc117fff700 1 -- 172.21.15.69:0/1017602041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc11c004160 con 0x7fc12007c730 2024-04-02T04:33:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc117fff700 1 -- 172.21.15.69:0/1017602041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fc11c004300 con 0x7fc12007c730 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc117fff700 1 -- 172.21.15.69:0/1017602041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc11c011960 con 0x7fc12007c730 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/1017602041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 msgr2=0x7fc12007ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 --2- 172.21.15.69:0/1017602041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12007ab90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0098a0 tx=0x7fc11c009bb0 comp rx=0 tx=0).stop 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/1017602041 shutdown_connections 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 --2- 172.21.15.69:0/1017602041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12007ab90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/1017602041 >> 172.21.15.69:0/1017602041 conn(0x7fc120101ca0 msgr2=0x7fc120104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/1017602041 shutdown_connections 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/1017602041 wait complete. 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 librados: starting msgr at 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 librados: starting objecter 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 Processor -- start 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- start start 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 librados: setting wanted keys 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 librados: calling monclient init 2024-04-02T04:33:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc12010cc70 con 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12019c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49410/0 (socket says 172.21.15.69:49410) 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 -- 172.21.15.69:0/3221386943 learned_addr learned my addr 172.21.15.69:0/3221386943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 -- 172.21.15.69:0/3221386943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12019ccb0 con 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc125457700 1 --2- 172.21.15.69:0/3221386943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12019c770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000c00 tx=0x7fc11c003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 10 client.?.objecter ms_handle_connect 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc11c0038b0 con 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/3221386943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc12019cf40 con 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fc11c0248a0 con 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc11c01a6a0 con 0x7fc12007c730 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 -- 172.21.15.69:0/3221386943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12019fc30 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1276bb700 1 librados: init done 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc11c02d3f0 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc11c019a90 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 10 client.14102.objecter ms_dispatch 0x7fc120001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.212+0000 7fc1167fc700 20 client.14102.objecter dump_active .. 0 homeless 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.216+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc100005120 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.216+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc11c043dd0 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc1000056e0 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc1167fc700 1 -- 172.21.15.69:0/3221386943 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc11c032020 con 0x7fc12007c730 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc10bfff700 10 librados: watch_flush enter 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc10bfff700 10 librados: watch_flush exit 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc10bfff700 20 client.14102.objecter shutdown clearing up homeless session... 2024-04-02T04:33:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc10bfff700 10 client.14102.objecter successfully canceled tick 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.344+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 msgr2=0x7fc12019c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 --2- 172.21.15.69:0/3221386943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12019c770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000c00 tx=0x7fc11c003dc0 comp rx=0 tx=0).stop 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 shutdown_connections 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 --2- 172.21.15.69:0/3221386943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc12007c730 0x7fc12019c770 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 >> 172.21.15.69:0/3221386943 conn(0x7fc120101ca0 msgr2=0x7fc120102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 shutdown_connections 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 -- 172.21.15.69:0/3221386943 wait complete. 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:51.348+0000 7fc10bfff700 1 librados: shutdown 2024-04-02T04:33:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (1/15)... 2024-04-02T04:33:53.138 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:52 smithi069 bash[21782]: debug 2024-04-02T04:33:52.792+0000 7fe51db43200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T04:33:53.139 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:52 smithi069 bash[21782]: debug 2024-04-02T04:33:52.888+0000 7fe51db43200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T04:33:53.139 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:53 smithi069 bash[21782]: debug 2024-04-02T04:33:53.132+0000 7fe51db43200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T04:33:54.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:53 smithi069 bash[21782]: debug 2024-04-02T04:33:53.920+0000 7fe51db43200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T04:33:54.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:54 smithi069 bash[21782]: debug 2024-04-02T04:33:54.024+0000 7fe51db43200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T04:33:54.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:54 smithi069 bash[21782]: debug 2024-04-02T04:33:54.128+0000 7fe51db43200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T04:33:54.781 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:54 smithi069 bash[21782]: debug 2024-04-02T04:33:54.480+0000 7fe51db43200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T04:33:54.782 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:54 smithi069 bash[21782]: debug 2024-04-02T04:33:54.600+0000 7fe51db43200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T04:33:54.782 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:54 smithi069 bash[21782]: debug 2024-04-02T04:33:54.708+0000 7fe51db43200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T04:33:55.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:54 smithi069 bash[21528]: audit 2024-04-02T04:33:54.842573+0000 mon.a (mon.0) 14 : audit 0 from='client.? 172.21.15.69:0/798468970' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:33:55.342 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:55 smithi069 bash[21782]: debug 2024-04-02T04:33:55.060+0000 7fe51db43200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T04:33:55.361 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T04:33:55.362 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T04:33:55.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T04:33:55.365 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T04:33:55.365 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T04:33:55.365 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.365 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T04:33:55.366 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T04:33:43.348904+0000", 2024-04-02T04:33:55.367 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a623ef700 1 Processor -- start 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a623ef700 1 -- start start 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a623ef700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c1095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a623ef700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c079e50 con 0x7f4a5c1071e0 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a5bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c1095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a5bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c1095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49412/0 (socket says 172.21.15.69:49412) 2024-04-02T04:33:55.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a5bfff700 1 -- 172.21.15.69:0/1108062563 learned_addr learned my addr 172.21.15.69:0/1108062563 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:55.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a5bfff700 1 -- 172.21.15.69:0/1108062563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c109b10 con 0x7f4a5c1071e0 2024-04-02T04:33:55.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.700+0000 7f4a5bfff700 1 --2- 172.21.15.69:0/1108062563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c1095d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098a0 tx=0x7f4a54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8220e7d8f4291f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:55.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5affd700 1 -- 172.21.15.69:0/1108062563 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a54004160 con 0x7f4a5c1071e0 2024-04-02T04:33:55.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5affd700 1 -- 172.21.15.69:0/1108062563 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f4a54004300 con 0x7f4a5c1071e0 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5affd700 1 -- 172.21.15.69:0/1108062563 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a54011960 con 0x7f4a5c1071e0 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/1108062563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 msgr2=0x7f4a5c1095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 --2- 172.21.15.69:0/1108062563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c1095d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4a540098a0 tx=0x7f4a54009bb0 comp rx=0 tx=0).stop 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/1108062563 shutdown_connections 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 --2- 172.21.15.69:0/1108062563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c1095d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/1108062563 >> 172.21.15.69:0/1108062563 conn(0x7f4a5c100b90 msgr2=0x7f4a5c102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:55.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/1108062563 shutdown_connections 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/1108062563 wait complete. 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 librados: starting msgr at 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 librados: starting objecter 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 Processor -- start 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- start start 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 librados: setting wanted keys 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 librados: calling monclient init 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c106a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c10aa20 con 0x7f4a5c1071e0 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49428/0 (socket says 172.21.15.69:49428) 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5bfff700 1 -- 172.21.15.69:0/798468970 learned_addr learned my addr 172.21.15.69:0/798468970 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:55.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5bfff700 1 -- 172.21.15.69:0/798468970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c106fa0 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a5bfff700 1 --2- 172.21.15.69:0/798468970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c106a60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006af0 tx=0x7f4a54003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 10 client.?.objecter ms_handle_connect 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a54011950 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f4a5401a890 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5c105190 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a540196e0 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5c1056d0 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 librados: init done 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4a54019c60 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4a54024730 con 0x7f4a5c1071e0 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 10 client.14104.objecter ms_dispatch 0x7f4a5c001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:55.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a597fa700 20 client.14104.objecter dump_active .. 0 homeless 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.704+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a5c063870 con 0x7f4a5c1071e0 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.708+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4a54043dd0 con 0x7f4a5c1071e0 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.836+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4a5c105bd0 con 0x7f4a5c1071e0 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.836+0000 7f4a597fa700 1 -- 172.21.15.69:0/798468970 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4a54032020 con 0x7f4a5c1071e0 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 10 librados: watch_flush enter 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 10 librados: watch_flush exit 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 20 client.14104.objecter shutdown clearing up homeless session... 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 10 client.14104.objecter successfully canceled tick 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 msgr2=0x7f4a5c106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 --2- 172.21.15.69:0/798468970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c106a60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006af0 tx=0x7f4a54003980 comp rx=0 tx=0).stop 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 shutdown_connections 2024-04-02T04:33:55.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 --2- 172.21.15.69:0/798468970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5c1071e0 0x7f4a5c106a60 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006af0 tx=0x7f4a54003980 comp rx=0 tx=0).stop 2024-04-02T04:33:55.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 >> 172.21.15.69:0/798468970 conn(0x7f4a5c100b90 msgr2=0x7f4a5c101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:55.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 shutdown_connections 2024-04-02T04:33:55.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 -- 172.21.15.69:0/798468970 wait complete. 2024-04-02T04:33:55.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:54.840+0000 7f4a623ef700 1 librados: shutdown 2024-04-02T04:33:55.374 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (2/15)... 2024-04-02T04:33:55.560 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:55 smithi069 bash[21782]: debug 2024-04-02T04:33:55.300+0000 7fe51db43200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T04:33:55.938 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:55 smithi069 bash[21782]: debug 2024-04-02T04:33:55.556+0000 7fe51db43200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T04:33:55.938 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:55 smithi069 bash[21782]: debug 2024-04-02T04:33:55.668+0000 7fe51db43200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T04:33:56.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:55 smithi069 bash[21782]: debug 2024-04-02T04:33:55.931+0000 7fe51db43200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T04:33:56.692 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:56 smithi069 bash[21782]: debug 2024-04-02T04:33:56.335+0000 7fe51db43200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:33:56.693 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:56 smithi069 bash[21782]: debug 2024-04-02T04:33:56.571+0000 7fe51db43200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T04:33:57.032 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:56 smithi069 bash[21782]: debug 2024-04-02T04:33:56.687+0000 7fe51db43200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T04:33:57.032 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:56 smithi069 bash[21782]: debug 2024-04-02T04:33:56.927+0000 7fe51db43200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:33:58.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:57 smithi069 bash[21782]: debug 2024-04-02T04:33:57.927+0000 7fe51db43200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T04:33:58.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:58 smithi069 bash[21782]: debug 2024-04-02T04:33:58.043+0000 7fe51db43200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T04:33:58.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:58 smithi069 bash[21528]: audit 2024-04-02T04:33:58.325838+0000 mon.a (mon.0) 15 : audit 0 from='client.? 172.21.15.69:0/1597418158' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:33:58.782 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:58 smithi069 bash[21782]: debug 2024-04-02T04:33:58.335+0000 7fe51db43200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T04:33:58.782 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:58 smithi069 bash[21782]: debug 2024-04-02T04:33:58.447+0000 7fe51db43200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.834 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T04:33:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T04:33:58.836 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T04:33:58.837 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T04:33:43.348904+0000", 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 Processor -- start 2024-04-02T04:33:58.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- start start 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a34106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34107110 con 0x7f9a34106720 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a34106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a34106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36140/0 (socket says 172.21.15.69:36140) 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a32ffd700 1 -- 172.21.15.69:0/3983923810 learned_addr learned my addr 172.21.15.69:0/3983923810 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a32ffd700 1 -- 172.21.15.69:0/3983923810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a34107920 con 0x7f9a34106720 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a32ffd700 1 --2- 172.21.15.69:0/3983923810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a34106b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9a240098a0 tx=0x7f9a24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a276bc3ab8a70c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a31ffb700 1 -- 172.21.15.69:0/3983923810 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a24004160 con 0x7f9a34106720 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a31ffb700 1 -- 172.21.15.69:0/3983923810 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f9a24004300 con 0x7f9a34106720 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a31ffb700 1 -- 172.21.15.69:0/3983923810 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a24011960 con 0x7f9a34106720 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- 172.21.15.69:0/3983923810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 msgr2=0x7f9a34106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 --2- 172.21.15.69:0/3983923810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a34106b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9a240098a0 tx=0x7f9a24009bb0 comp rx=0 tx=0).stop 2024-04-02T04:33:58.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- 172.21.15.69:0/3983923810 shutdown_connections 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 --2- 172.21.15.69:0/3983923810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a34106b40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- 172.21.15.69:0/3983923810 >> 172.21.15.69:0/3983923810 conn(0x7f9a34101ca0 msgr2=0x7f9a34104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- 172.21.15.69:0/3983923810 shutdown_connections 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.183+0000 7f9a398fb700 1 -- 172.21.15.69:0/3983923810 wait complete. 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 librados: starting msgr at 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 librados: starting objecter 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 Processor -- start 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 -- start start 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 librados: setting wanted keys 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 librados: calling monclient init 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a3419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3410ab50 con 0x7f9a34106720 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a3419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:33:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a3419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36156/0 (socket says 172.21.15.69:36156) 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a32ffd700 1 -- 172.21.15.69:0/1597418158 learned_addr learned my addr 172.21.15.69:0/1597418158 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a32ffd700 1 -- 172.21.15.69:0/1597418158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3419aab0 con 0x7f9a34106720 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a32ffd700 1 --2- 172.21.15.69:0/1597418158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a3419a570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9a24000c00 tx=0x7f9a24003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 10 client.?.objecter ms_handle_connect 0x7f9a34106720 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a240038b0 con 0x7f9a34106720 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3419ad40 con 0x7f9a34106720 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f9a240248a0 con 0x7f9a34106720 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a2401a6a0 con 0x7f9a34106720 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 10 client.14106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:33:58.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a3419da30 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 librados: init done 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9a2402d3f0 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9a24019700 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 10 client.14106.objecter ms_dispatch 0x7f9a34001550 osd_map(1..1 src has 1..1) 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 3 client.14106.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 3 client.14106.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a388f9700 20 client.14106.objecter dump_active .. 0 homeless 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.187+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a34063870 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.191+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9a24043dd0 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9a3419dd80 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a388f9700 1 -- 172.21.15.69:0/1597418158 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9a24032020 con 0x7f9a34106720 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 10 librados: watch_flush enter 2024-04-02T04:33:58.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 10 librados: watch_flush exit 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 20 client.14106.objecter shutdown clearing up homeless session... 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 10 client.14106.objecter successfully canceled tick 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 msgr2=0x7f9a3419a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 1 --2- 172.21.15.69:0/1597418158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a3419a570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9a24000c00 tx=0x7f9a24003dc0 comp rx=0 tx=0).stop 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 shutdown_connections 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.323+0000 7f9a398fb700 1 --2- 172.21.15.69:0/1597418158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a34106720 0x7f9a3419a570 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.327+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 >> 172.21.15.69:0/1597418158 conn(0x7f9a34101ca0 msgr2=0x7f9a34102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.327+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 shutdown_connections 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.327+0000 7f9a398fb700 1 -- 172.21.15.69:0/1597418158 wait complete. 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:33:58.327+0000 7f9a398fb700 1 librados: shutdown 2024-04-02T04:33:58.843 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (3/15)... 2024-04-02T04:33:59.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:58 smithi069 bash[21782]: debug 2024-04-02T04:33:58.899+0000 7fe51db43200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T04:33:59.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21782]: debug 2024-04-02T04:33:59.015+0000 7fe51db43200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: cluster 2024-04-02T04:33:59.037640+0000 mon.a (mon.0) 16 : cluster 1 Activating manager daemon y 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: cluster 2024-04-02T04:33:59.044609+0000 mon.a (mon.0) 17 : cluster 0 mgrmap e2: y(active, starting, since 0.00746462s) 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.046367+0000 mon.a (mon.0) 18 : audit 0 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.046929+0000 mon.a (mon.0) 19 : audit 0 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.047285+0000 mon.a (mon.0) 20 : audit 0 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.047670+0000 mon.a (mon.0) 21 : audit 0 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.048063+0000 mon.a (mon.0) 22 : audit 0 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: cluster 2024-04-02T04:33:59.073671+0000 mon.a (mon.0) 23 : cluster 1 Manager daemon y is now available 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.085938+0000 mon.a (mon.0) 24 : audit 1 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:33:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.088964+0000 mon.a (mon.0) 25 : audit 1 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:33:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.092087+0000 mon.a (mon.0) 26 : audit 1 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' 2024-04-02T04:33:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.097057+0000 mon.a (mon.0) 27 : audit 1 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' 2024-04-02T04:33:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:33:59 smithi069 bash[21528]: audit 2024-04-02T04:33:59.102278+0000 mon.a (mon.0) 28 : audit 1 from='mgr.14100 172.21.15.69:0/1189994694' entity='mgr.y' 2024-04-02T04:34:01.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:01 smithi069 bash[21528]: cluster 2024-04-02T04:34:00.052263+0000 mon.a (mon.0) 29 : cluster 0 mgrmap e3: y(active, since 1.01512s) 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T04:34:02.791 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.792 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T04:34:02.793 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T04:34:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T04:34:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T04:34:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T04:34:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T04:34:02.794 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.795 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T04:33:43.348904+0000", 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b087eb700 1 Processor -- start 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b087eb700 1 -- start start 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b087eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 0x7f7b00104df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:02.796 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b087eb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b001053c0 con 0x7f7b001049d0 2024-04-02T04:34:02.797 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b06587700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 0x7f7b00104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:02.797 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b06587700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 0x7f7b00104df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36250/0 (socket says 172.21.15.69:36250) 2024-04-02T04:34:02.797 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b06587700 1 -- 172.21.15.69:0/876643915 learned_addr learned my addr 172.21.15.69:0/876643915 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:02.797 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b06587700 1 -- 172.21.15.69:0/876643915 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b00105540 con 0x7f7b001049d0 2024-04-02T04:34:02.797 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b06587700 1 --2- 172.21.15.69:0/876643915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 0x7f7b00104df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0098a0 tx=0x7f7afc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d478d49f3fc04aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:02.797 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b05585700 1 -- 172.21.15.69:0/876643915 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7afc004160 con 0x7f7b001049d0 2024-04-02T04:34:02.798 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b05585700 1 -- 172.21.15.69:0/876643915 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f7afc004300 con 0x7f7b001049d0 2024-04-02T04:34:02.798 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.679+0000 7f7b05585700 1 -- 172.21.15.69:0/876643915 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7afc011960 con 0x7f7b001049d0 2024-04-02T04:34:02.798 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/876643915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 msgr2=0x7f7b00104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:02.798 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 --2- 172.21.15.69:0/876643915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 0x7f7b00104df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0098a0 tx=0x7f7afc009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:02.798 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/876643915 shutdown_connections 2024-04-02T04:34:02.798 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 --2- 172.21.15.69:0/876643915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b001049d0 0x7f7b00104df0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:02.799 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/876643915 >> 172.21.15.69:0/876643915 conn(0x7f7b001002d0 msgr2=0x7f7b00102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:02.799 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/876643915 shutdown_connections 2024-04-02T04:34:02.799 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/876643915 wait complete. 2024-04-02T04:34:02.799 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 librados: starting msgr at 2024-04-02T04:34:02.799 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 librados: starting objecter 2024-04-02T04:34:02.799 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 Processor -- start 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- start start 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 librados: setting wanted keys 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 librados: calling monclient init 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 0x7f7b0019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b001087d0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b06587700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 0x7f7b0019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b06587700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 0x7f7b0019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36252/0 (socket says 172.21.15.69:36252) 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b06587700 1 -- 172.21.15.69:0/618125479 learned_addr learned my addr 172.21.15.69:0/618125479 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b06587700 1 -- 172.21.15.69:0/618125479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0019b040 con 0x7f7b0019a6e0 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b06587700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 0x7f7b0019ab00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7b000795a0 tx=0x7f7afc012f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:02.800 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7af77fe700 10 client.?.objecter ms_handle_connect 0x7f7b0019a6e0 2024-04-02T04:34:02.801 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7af77fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:02.801 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7afc011950 con 0x7f7b0019a6e0 2024-04-02T04:34:02.801 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f7afc026480 con 0x7f7b0019a6e0 2024-04-02T04:34:02.801 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7afc01a5d0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.801 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0019b330 con 0x7f7b0019a6e0 2024-04-02T04:34:02.801 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0019de90 con 0x7f7b0019a6e0 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.683+0000 7f7b087eb700 1 librados: init done 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 3) ==== 45999+0+0 (secure 0 0 0) 0x7f7afc01a770 con 0x7f7b0019a6e0 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f7aec03a750 0x7f7aec03cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b0010ac80 con 0x7f7b0019a6e0 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7afc04d2f0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 10 client.14114.objecter ms_dispatch 0x7f7b00001550 osd_map(1..1 src has 1..1) 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:34:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7b05d86700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f7aec03a750 0x7f7aec03cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 20 client.14114.objecter dump_active .. 0 homeless 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7af77fe700 10 client.14114.objecter ms_handle_connect 0x7f7aec03a750 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.687+0000 7f7b05d86700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f7aec03a750 0x7f7aec03cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7af0006fd0 tx=0x7f7af0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:01.691+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7afc0247d0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.047+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f7afc0269a0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.207+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7b0019e4d0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.207+0000 7f7af77fe700 1 -- 172.21.15.69:0/618125479 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7afc0094f0 con 0x7f7b0019a6e0 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 10 librados: watch_flush enter 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 10 librados: watch_flush exit 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 20 client.14114.objecter shutdown clearing up homeless session... 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 10 client.14114.objecter successfully canceled tick 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f7aec03a750 msgr2=0x7f7aec03cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:02.803 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f7aec03a750 0x7f7aec03cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7af0006fd0 tx=0x7f7af0006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 msgr2=0x7f7b0019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 0x7f7b0019ab00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7b000795a0 tx=0x7f7afc012f00 comp rx=0 tx=0).stop 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 shutdown_connections 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f7aec03a750 0x7f7aec03cc10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 --2- 172.21.15.69:0/618125479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b0019a6e0 0x7f7b0019ab00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 >> 172.21.15.69:0/618125479 conn(0x7f7b001002d0 msgr2=0x7f7b00100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 shutdown_connections 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 -- 172.21.15.69:0/618125479 wait complete. 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:02.211+0000 7f7b087eb700 1 librados: shutdown 2024-04-02T04:34:02.804 INFO:teuthology.orchestra.run.smithi069.stdout:mgr is available 2024-04-02T04:34:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:03 smithi069 bash[21528]: cluster 2024-04-02T04:34:02.050207+0000 mon.a (mon.0) 30 : cluster 0 mgrmap e4: y(active, since 3s) 2024-04-02T04:34:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:03 smithi069 bash[21528]: audit 2024-04-02T04:34:02.210171+0000 mon.a (mon.0) 31 : audit 0 from='client.? 172.21.15.69:0/618125479' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:34:04.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:04 smithi069 bash[21528]: audit 2024-04-02T04:34:03.947905+0000 mon.a (mon.0) 32 : audit 1 from='client.? 172.21.15.69:0/616401832' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-02T04:34:04.523 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [global] 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout fsid = f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [osd] 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-02T04:34:04.524 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 Processor -- start 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- start start 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a4105a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f16a4105fe0 con 0x7f16a41055f0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a4105a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a4105a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36268/0 (socket says 172.21.15.69:36268) 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 -- 172.21.15.69:0/1627198893 learned_addr learned my addr 172.21.15.69:0/1627198893 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 -- 172.21.15.69:0/1627198893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a41067f0 con 0x7f16a41055f0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 --2- 172.21.15.69:0/1627198893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a4105a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f16940098a0 tx=0x7f1694009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7e06aeb7d9edf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a159a700 1 -- 172.21.15.69:0/1627198893 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1694004160 con 0x7f16a41055f0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a159a700 1 -- 172.21.15.69:0/1627198893 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f1694004300 con 0x7f16a41055f0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a159a700 1 -- 172.21.15.69:0/1627198893 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1694011960 con 0x7f16a41055f0 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- 172.21.15.69:0/1627198893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 msgr2=0x7f16a4105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 --2- 172.21.15.69:0/1627198893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a4105a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f16940098a0 tx=0x7f1694009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:04.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- 172.21.15.69:0/1627198893 shutdown_connections 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 --2- 172.21.15.69:0/1627198893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a4105a10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- 172.21.15.69:0/1627198893 >> 172.21.15.69:0/1627198893 conn(0x7f16a4100b50 msgr2=0x7f16a4102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- 172.21.15.69:0/1627198893 shutdown_connections 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- 172.21.15.69:0/1627198893 wait complete. 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 librados: starting msgr at 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 librados: starting objecter 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 Processor -- start 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- start start 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 librados: setting wanted keys 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 librados: calling monclient init 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a8ed8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f16a4109a20 con 0x7f16a41055f0 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:04.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36270/0 (socket says 172.21.15.69:36270) 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 -- 172.21.15.69:0/616401832 learned_addr learned my addr 172.21.15.69:0/616401832 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 -- 172.21.15.69:0/616401832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a419aab0 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16a259c700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a419a570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1694006af0 tx=0x7f1694003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16937fe700 10 client.?.objecter ms_handle_connect 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.587+0000 7f16937fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1694011950 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f169401a890 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16940196e0 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a419ad40 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a8ed8700 10 client.14116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a419da30 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a8ed8700 1 librados: init done 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f16940198d0 con 0x7f16a41055f0 2024-04-02T04:34:04.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a4063870 con 0x7f16a41055f0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f168c03a810 0x7f168c03ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f169404d280 con 0x7f16a41055f0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 10 client.14116.objecter ms_dispatch 0x7f16a4001550 osd_map(1..1 src has 1..1) 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 3 client.14116.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 3 client.14116.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 20 client.14116.objecter dump_active .. 0 homeless 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a1d9b700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f168c03a810 0x7f168c03ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16a1d9b700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f168c03a810 0x7f168c03ccd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1698006fd0 tx=0x7f1698006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.591+0000 7f16937fe700 10 client.14116.objecter ms_handle_connect 0x7f168c03a810 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.595+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f16940042c0 con 0x7f16a41055f0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.943+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f16a402dc00 con 0x7f16a41055f0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16937fe700 1 -- 172.21.15.69:0/616401832 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+282 (secure 0 0 0) 0x7f1694016df0 con 0x7f16a41055f0 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 10 librados: watch_flush enter 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 10 librados: watch_flush exit 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 20 client.14116.objecter shutdown clearing up homeless session... 2024-04-02T04:34:04.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 10 client.14116.objecter successfully canceled tick 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f168c03a810 msgr2=0x7f168c03ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f168c03a810 0x7f168c03ccd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1698006fd0 tx=0x7f1698006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 msgr2=0x7f16a419a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a419a570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1694006af0 tx=0x7f1694003980 comp rx=0 tx=0).stop 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 shutdown_connections 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f168c03a810 0x7f168c03ccd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 --2- 172.21.15.69:0/616401832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16a41055f0 0x7f16a419a570 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 >> 172.21.15.69:0/616401832 conn(0x7f16a4100b50 msgr2=0x7f16a4191120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 shutdown_connections 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.947+0000 7f16a8ed8700 1 -- 172.21.15.69:0/616401832 wait complete. 2024-04-02T04:34:04.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:03.951+0000 7f16a8ed8700 1 librados: shutdown 2024-04-02T04:34:04.530 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling cephadm module... 2024-04-02T04:34:06.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:05 smithi069 bash[21782]: ignoring --setuser ceph since I am not root 2024-04-02T04:34:06.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:05 smithi069 bash[21782]: ignoring --setgroup ceph since I am not root 2024-04-02T04:34:06.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:06 smithi069 bash[21782]: debug 2024-04-02T04:34:06.010+0000 7fee01707200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T04:34:06.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:05 smithi069 bash[21528]: audit 2024-04-02T04:34:05.780941+0000 mon.a (mon.0) 33 : audit 1 from='client.? 172.21.15.69:0/2952201940' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-02T04:34:06.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.310+0000 7f2387e3d700 1 Processor -- start 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.310+0000 7f2387e3d700 1 -- start start 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.310+0000 7f2387e3d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f2380106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.310+0000 7f2387e3d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2380107110 con 0x7f2380106720 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f2380106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f2380106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36282/0 (socket says 172.21.15.69:36282) 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 -- 172.21.15.69:0/1374504822 learned_addr learned my addr 172.21.15.69:0/1374504822 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 -- 172.21.15.69:0/1374504822 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2380107920 con 0x7f2380106720 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 --2- 172.21.15.69:0/1374504822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f2380106b40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f237c0098a0 tx=0x7f237c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34e92498995b174 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2384bd7700 1 -- 172.21.15.69:0/1374504822 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f237c004160 con 0x7f2380106720 2024-04-02T04:34:06.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2384bd7700 1 -- 172.21.15.69:0/1374504822 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f237c004300 con 0x7f2380106720 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2384bd7700 1 -- 172.21.15.69:0/1374504822 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f237c011960 con 0x7f2380106720 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/1374504822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 msgr2=0x7f2380106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 --2- 172.21.15.69:0/1374504822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f2380106b40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f237c0098a0 tx=0x7f237c009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/1374504822 shutdown_connections 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 --2- 172.21.15.69:0/1374504822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f2380106b40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/1374504822 >> 172.21.15.69:0/1374504822 conn(0x7f2380101d00 msgr2=0x7f2380104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/1374504822 shutdown_connections 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/1374504822 wait complete. 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 librados: starting msgr at 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 librados: starting objecter 2024-04-02T04:34:06.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 Processor -- start 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- start start 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 librados: setting wanted keys 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 librados: calling monclient init 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f238007c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f238010ab50 con 0x7f2380106720 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f238007c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f238007c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36290/0 (socket says 172.21.15.69:36290) 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 -- 172.21.15.69:0/2952201940 learned_addr learned my addr 172.21.15.69:0/2952201940 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 -- 172.21.15.69:0/2952201940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238007c940 con 0x7f2380106720 2024-04-02T04:34:06.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2385bd9700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f238007c400 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f237c006af0 tx=0x7f237c003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2376ffd700 10 client.?.objecter ms_handle_connect 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2376ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f237c011950 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f237c01a890 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f237c0196e0 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/2952201940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238007ab30 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 -- 172.21.15.69:0/2952201940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238007b070 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.314+0000 7f2387e3d700 1 librados: init done 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f237c01aa40 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2368005120 con 0x7f2380106720 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f236c03a810 0x7f236c03ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f237c04d660 con 0x7f2380106720 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 10 client.14118.objecter ms_dispatch 0x7f2380001550 osd_map(1..1 src has 1..1) 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 20 client.14118.objecter dump_active .. 0 homeless 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f23853d8700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f236c03a810 0x7f236c03ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f23853d8700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f236c03a810 0x7f236c03ccd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2370006fd0 tx=0x7f2370006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.318+0000 7f2376ffd700 10 client.14118.objecter ms_handle_connect 0x7f236c03a810 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.322+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f237c04d080 con 0x7f2380106720 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.778+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f2368005410 con 0x7f2380106720 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.830+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f237c019880 con 0x7f2380106720 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.830+0000 7f2374ff9700 10 librados: watch_flush enter 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 10 librados: watch_flush exit 2024-04-02T04:34:06.414 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 20 client.14118.objecter shutdown clearing up homeless session... 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 10 client.14118.objecter successfully canceled tick 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2376ffd700 1 -- 172.21.15.69:0/2952201940 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7f237c00b3e0 con 0x7f2380106720 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f236c03a810 msgr2=0x7f236c03ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f236c03a810 0x7f236c03ccd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2370006fd0 tx=0x7f2370006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 msgr2=0x7f238007c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f238007c400 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f237c006af0 tx=0x7f237c003980 comp rx=0 tx=0).stop 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 shutdown_connections 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f236c03a810 0x7f236c03ccd0 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2370006fd0 tx=0x7f2370006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 --2- 172.21.15.69:0/2952201940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2380106720 0x7f238007c400 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 >> 172.21.15.69:0/2952201940 conn(0x7f2380101d00 msgr2=0x7f2380102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 shutdown_connections 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 -- 172.21.15.69:0/2952201940 wait complete. 2024-04-02T04:34:06.415 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:05.834+0000 7f2374ff9700 1 librados: shutdown 2024-04-02T04:34:07.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:06 smithi069 bash[21528]: audit 2024-04-02T04:34:05.832761+0000 mon.a (mon.0) 34 : audit 1 from='client.? 172.21.15.69:0/2952201940' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-02T04:34:07.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:06 smithi069 bash[21528]: cluster 2024-04-02T04:34:05.837726+0000 mon.a (mon.0) 35 : cluster 0 mgrmap e5: y(active, since 6s) 2024-04-02T04:34:07.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:07 smithi069 bash[21528]: audit 2024-04-02T04:34:07.705244+0000 mon.a (mon.0) 36 : audit 0 from='client.? 172.21.15.69:0/443918566' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-02T04:34:07.907 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:07 smithi069 bash[21782]: debug 2024-04-02T04:34:07.586+0000 7fee01707200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T04:34:07.907 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:07 smithi069 bash[21782]: debug 2024-04-02T04:34:07.682+0000 7fee01707200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T04:34:08.274 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c61d16700 1 Processor -- start 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c61d16700 1 -- start start 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c61d16700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c1053d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c61d16700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5c079e70 con 0x7f1c5c104fb0 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c5b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c1053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c5b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c1053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36322/0 (socket says 172.21.15.69:36322) 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.214+0000 7f1c5b7fe700 1 -- 172.21.15.69:0/3262604295 learned_addr learned my addr 172.21.15.69:0/3262604295 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:08.275 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 -- 172.21.15.69:0/3262604295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5c105910 con 0x7f1c5c104fb0 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 --2- 172.21.15.69:0/3262604295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c1053d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009e80 tx=0x7f1c5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5accc97b0a4d924a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5a7fc700 1 -- 172.21.15.69:0/3262604295 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c54017070 con 0x7f1c5c104fb0 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5a7fc700 1 -- 172.21.15.69:0/3262604295 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f1c5400b8c0 con 0x7f1c5c104fb0 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5a7fc700 1 -- 172.21.15.69:0/3262604295 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c54004340 con 0x7f1c5c104fb0 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/3262604295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 msgr2=0x7f1c5c1053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 --2- 172.21.15.69:0/3262604295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c1053d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009e80 tx=0x7f1c5400b330 comp rx=0 tx=0).stop 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/3262604295 shutdown_connections 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 --2- 172.21.15.69:0/3262604295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c1053d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/3262604295 >> 172.21.15.69:0/3262604295 conn(0x7f1c5c100b90 msgr2=0x7f1c5c102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/3262604295 shutdown_connections 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/3262604295 wait complete. 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 librados: starting msgr at 2024-04-02T04:34:08.276 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 librados: starting objecter 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 Processor -- start 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- start start 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 librados: setting wanted keys 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 librados: calling monclient init 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5c1068e0 con 0x7f1c5c104fb0 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c19c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36330/0 (socket says 172.21.15.69:36330) 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 -- 172.21.15.69:0/443918566 learned_addr learned my addr 172.21.15.69:0/443918566 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 -- 172.21.15.69:0/443918566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5c19cb80 con 0x7f1c5c104fb0 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c5b7fe700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c19c640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400d010 tx=0x7f1c5400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c58ff9700 10 client.?.objecter ms_handle_connect 0x7f1c5c104fb0 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c58ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:08.277 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c54017040 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f1c54023070 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5c19ce10 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c5401e420 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5c19fb00 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.218+0000 7f1c61d16700 1 librados: init done 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c48005150 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7f1c54012430 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 0x7f1c4403cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1c5400f050 con 0x7f1c5c104fb0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 10 client.14122.objecter ms_dispatch 0x7f1c5c001550 osd_map(1..1 src has 1..1) 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 3 client.14122.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:34:08.278 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 3 client.14122.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c5affd700 1 -- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 msgr2=0x7f1c4403cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c58ff9700 20 client.14122.objecter dump_active .. 0 homeless 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.222+0000 7f1c5affd700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 0x7f1c4403cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.226+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1c54003b50 con 0x7f1c5c104fb0 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.426+0000 7f1c5affd700 1 -- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 msgr2=0x7f1c4403cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.426+0000 7f1c5affd700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 0x7f1c4403cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.702+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f1c48005c50 con 0x7f1c5c104fb0 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.702+0000 7f1c58ff9700 1 -- 172.21.15.69:0/443918566 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f1c5404c990 con 0x7f1c5c104fb0 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.702+0000 7f1c61d16700 10 librados: watch_flush enter 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 10 librados: watch_flush exit 2024-04-02T04:34:08.279 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 20 client.14122.objecter shutdown clearing up homeless session... 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 10 client.14122.objecter successfully canceled tick 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 msgr2=0x7f1c4403cd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 0x7f1c4403cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 msgr2=0x7f1c5c19c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c19c640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400d010 tx=0x7f1c5400bf10 comp rx=0 tx=0).stop 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 shutdown_connections 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7f1c4403a870 0x7f1c4403cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 --2- 172.21.15.69:0/443918566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c5c104fb0 0x7f1c5c19c640 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 >> 172.21.15.69:0/443918566 conn(0x7f1c5c100b90 msgr2=0x7f1c5c1908e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 shutdown_connections 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 -- 172.21.15.69:0/443918566 wait complete. 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:07.706+0000 7f1c61d16700 1 librados: shutdown 2024-04-02T04:34:08.280 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for the mgr to restart... 2024-04-02T04:34:08.281 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr epoch 5... 2024-04-02T04:34:08.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:07 smithi069 bash[21782]: debug 2024-04-02T04:34:07.930+0000 7fee01707200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T04:34:09.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:08 smithi069 bash[21782]: debug 2024-04-02T04:34:08.722+0000 7fee01707200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T04:34:09.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:08 smithi069 bash[21782]: debug 2024-04-02T04:34:08.834+0000 7fee01707200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T04:34:09.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:08 smithi069 bash[21782]: debug 2024-04-02T04:34:08.942+0000 7fee01707200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T04:34:09.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:09 smithi069 bash[21782]: debug 2024-04-02T04:34:09.306+0000 7fee01707200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T04:34:09.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:09 smithi069 bash[21782]: debug 2024-04-02T04:34:09.418+0000 7fee01707200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T04:34:09.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:09 smithi069 bash[21782]: debug 2024-04-02T04:34:09.518+0000 7fee01707200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T04:34:10.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:09 smithi069 bash[21782]: debug 2024-04-02T04:34:09.870+0000 7fee01707200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T04:34:10.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:10 smithi069 bash[21782]: debug 2024-04-02T04:34:10.106+0000 7fee01707200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T04:34:10.750 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:10 smithi069 bash[21782]: debug 2024-04-02T04:34:10.362+0000 7fee01707200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T04:34:10.750 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:10 smithi069 bash[21782]: debug 2024-04-02T04:34:10.474+0000 7fee01707200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T04:34:11.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:10 smithi069 bash[21782]: debug 2024-04-02T04:34:10.746+0000 7fee01707200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T04:34:11.506 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:11 smithi069 bash[21782]: debug 2024-04-02T04:34:11.150+0000 7fee01707200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:34:11.506 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:11 smithi069 bash[21782]: debug 2024-04-02T04:34:11.386+0000 7fee01707200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T04:34:11.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:11 smithi069 bash[21782]: debug 2024-04-02T04:34:11.502+0000 7fee01707200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T04:34:11.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:11 smithi069 bash[21782]: debug 2024-04-02T04:34:11.746+0000 7fee01707200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:34:13.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:12 smithi069 bash[21782]: debug 2024-04-02T04:34:12.746+0000 7fee01707200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T04:34:13.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:12 smithi069 bash[21782]: debug 2024-04-02T04:34:12.862+0000 7fee01707200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T04:34:13.533 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21782]: debug 2024-04-02T04:34:13.150+0000 7fee01707200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T04:34:13.533 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21782]: debug 2024-04-02T04:34:13.266+0000 7fee01707200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T04:34:14.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21782]: debug 2024-04-02T04:34:13.710+0000 7fee01707200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T04:34:14.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21782]: debug 2024-04-02T04:34:13.830+0000 7fee01707200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T04:34:14.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: cluster 2024-04-02T04:34:13.853976+0000 mon.a (mon.0) 37 : cluster 1 Active manager daemon y restarted 2024-04-02T04:34:14.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: cluster 2024-04-02T04:34:13.854211+0000 mon.a (mon.0) 38 : cluster 1 Activating manager daemon y 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: cluster 2024-04-02T04:34:13.861342+0000 mon.a (mon.0) 39 : cluster 0 osdmap e2: 0 total, 0 up, 0 in 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: cluster 2024-04-02T04:34:13.861526+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e6: y(active, starting, since 0.00739956s) 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: audit 2024-04-02T04:34:13.862564+0000 mon.a (mon.0) 41 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: audit 2024-04-02T04:34:13.864739+0000 mon.a (mon.0) 42 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: audit 2024-04-02T04:34:13.865734+0000 mon.a (mon.0) 43 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: audit 2024-04-02T04:34:13.865937+0000 mon.a (mon.0) 44 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: audit 2024-04-02T04:34:13.866142+0000 mon.a (mon.0) 45 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:34:14.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:13 smithi069 bash[21528]: cluster 2024-04-02T04:34:13.890076+0000 mon.a (mon.0) 46 : cluster 1 Manager daemon y is now available 2024-04-02T04:34:15.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: cephadm 2024-04-02T04:34:13.908547+0000 mgr.y (mgr.14120) 1 : cephadm 1 Found migration_current of "None". Setting to last migration. 2024-04-02T04:34:15.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:13.912827+0000 mon.a (mon.0) 47 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:15.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:13.915630+0000 mon.a (mon.0) 48 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:15.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:13.918845+0000 mon.a (mon.0) 49 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:13.922646+0000 mon.a (mon.0) 50 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:13.933286+0000 mon.a (mon.0) 51 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:13.936305+0000 mon.a (mon.0) 52 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:14.131935+0000 mon.a (mon.0) 53 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:14.137072+0000 mon.a (mon.0) 54 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: cephadm 2024-04-02T04:34:14.590728+0000 mgr.y (mgr.14120) 2 : cephadm 1 [02/Apr/2024:04:34:14] ENGINE Bus STARTING 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: cephadm 2024-04-02T04:34:14.707657+0000 mgr.y (mgr.14120) 3 : cephadm 1 [02/Apr/2024:04:34:14] ENGINE Serving on https://172.21.15.69:7150 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: cephadm 2024-04-02T04:34:14.808754+0000 mgr.y (mgr.14120) 4 : cephadm 1 [02/Apr/2024:04:34:14] ENGINE Serving on http://172.21.15.69:8765 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: cephadm 2024-04-02T04:34:14.808855+0000 mgr.y (mgr.14120) 5 : cephadm 1 [02/Apr/2024:04:34:14] ENGINE Bus STARTED 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:14.842496+0000 mon.a (mon.0) 55 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:14.864627+0000 mgr.y (mgr.14120) 6 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: cluster 2024-04-02T04:34:14.866253+0000 mon.a (mon.0) 56 : cluster 0 mgrmap e7: y(active, since 1.01212s) 2024-04-02T04:34:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:14 smithi069 bash[21528]: audit 2024-04-02T04:34:14.877819+0000 mgr.y (mgr.14120) 7 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 Processor -- start 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- start start 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd348109570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd348079eb0 con 0x7fd348107180 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd346ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd348109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd346ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd348109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55740/0 (socket says 172.21.15.69:55740) 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd346ffd700 1 -- 172.21.15.69:0/2749291039 learned_addr learned my addr 172.21.15.69:0/2749291039 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:15.483 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd346ffd700 1 -- 172.21.15.69:0/2749291039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd348109ab0 con 0x7fd348107180 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd346ffd700 1 --2- 172.21.15.69:0/2749291039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd348109570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3380098a0 tx=0x7fd338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d5d5aea62bcedb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd3467fc700 1 -- 172.21.15.69:0/2749291039 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd338004160 con 0x7fd348107180 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd3467fc700 1 -- 172.21.15.69:0/2749291039 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fd338004300 con 0x7fd348107180 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd3467fc700 1 -- 172.21.15.69:0/2749291039 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd338011960 con 0x7fd348107180 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- 172.21.15.69:0/2749291039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 msgr2=0x7fd348109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2749291039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd348109570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3380098a0 tx=0x7fd338009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- 172.21.15.69:0/2749291039 shutdown_connections 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2749291039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd348109570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- 172.21.15.69:0/2749291039 >> 172.21.15.69:0/2749291039 conn(0x7fd348100b90 msgr2=0x7fd348102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- 172.21.15.69:0/2749291039 shutdown_connections 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.102+0000 7fd34d66f700 1 -- 172.21.15.69:0/2749291039 wait complete. 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 librados: starting msgr at 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 librados: starting objecter 2024-04-02T04:34:15.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 Processor -- start 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 -- start start 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 librados: setting wanted keys 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 librados: calling monclient init 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd34819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd34810aa20 con 0x7fd348107180 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd346ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd34819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd346ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd34819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55756/0 (socket says 172.21.15.69:55756) 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd346ffd700 1 -- 172.21.15.69:0/2286418748 learned_addr learned my addr 172.21.15.69:0/2286418748 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd346ffd700 1 -- 172.21.15.69:0/2286418748 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34819aa20 con 0x7fd348107180 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd346ffd700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd34819a4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd338011740 tx=0x7fd338003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 10 client.?.objecter ms_handle_connect 0x7fd348107180 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3380038b0 con 0x7fd348107180 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fd3380248a0 con 0x7fd348107180 2024-04-02T04:34:15.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd33801a6b0 con 0x7fd348107180 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34819acb0 con 0x7fd348107180 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34819d9a0 con 0x7fd348107180 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 librados: init done 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7fd338024440 con 0x7fd348107180 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 --> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd348063870 con 0x7fd33403a8c0 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd33804d1e0 con 0x7fd348107180 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 10 client.14124.objecter ms_dispatch 0x7fd348001550 osd_map(1..1 src has 1..1) 2024-04-02T04:34:15.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 3 client.14124.objecter handle_osd_map got epochs [1,1] > 0 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 3 client.14124.objecter handle_osd_map decoding full epoch 1 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd33e7fc700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 msgr2=0x7fd33403cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.106+0000 7fd344ff9700 20 client.14124.objecter dump_active .. 0 homeless 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.310+0000 7fd33e7fc700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 msgr2=0x7fd33403cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.310+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.714+0000 7fd33e7fc700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 msgr2=0x7fd33403cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:09.714+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:10.518+0000 7fd33e7fc700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 msgr2=0x7fd33403cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:10.518+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:12.122+0000 7fd33e7fc700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 msgr2=0x7fd33403cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2442918057 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:12.122+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T04:34:15.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:13.854+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mgrmap(e 6) ==== 45883+0+0 (secure 0 0 0) 0x7fd33802d8c0 con 0x7fd348107180 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:13.854+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 msgr2=0x7fd33403cd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:13.854+0000 7fd344ff9700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.110+0000 7fd34ce6e700 10 client.14124.objecter tick 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.858+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 7) ==== 46010+0+0 (secure 0 0 0) 0x7fd33804e4d0 con 0x7fd348107180 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.858+0000 7fd344ff9700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd33403dfe0 0x7fd3340403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.858+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd33800af30 con 0x7fd33403dfe0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.858+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd33403dfe0 0x7fd3340403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.862+0000 7fd33e7fc700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd33403dfe0 0x7fd3340403d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd330003a10 tx=0x7fd3300092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.862+0000 7fd344ff9700 10 client.14124.objecter ms_handle_connect 0x7fd33403dfe0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.862+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd33800af30 con 0x7fd33403dfe0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd348101cf0 con 0x7fd33403dfe0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd344ff9700 1 -- 172.21.15.69:0/2286418748 <== mgr.14120 v2:172.21.15.69:6800/578732137 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd348101cf0 con 0x7fd33403dfe0 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 10 librados: watch_flush enter 2024-04-02T04:34:15.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 10 librados: watch_flush exit 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 20 client.14124.objecter shutdown clearing up homeless session... 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 10 client.14124.objecter successfully canceled tick 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd33403dfe0 msgr2=0x7fd3340403d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd33403dfe0 0x7fd3340403d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd330003a10 tx=0x7fd3300092b0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 msgr2=0x7fd34819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd34819a4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd338011740 tx=0x7fd338003dc0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 shutdown_connections 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd33403dfe0 0x7fd3340403d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd348107180 0x7fd34819a4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 --2- 172.21.15.69:0/2286418748 >> [v2:172.21.15.69:6800/2442918057,v1:172.21.15.69:6801/2442918057] conn(0x7fd33403a8c0 0x7fd33403cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 >> 172.21.15.69:0/2286418748 conn(0x7fd348100b90 msgr2=0x7fd3481015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 shutdown_connections 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 -- 172.21.15.69:0/2286418748 wait complete. 2024-04-02T04:34:15.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:14.874+0000 7fd34d66f700 1 librados: shutdown 2024-04-02T04:34:15.490 INFO:teuthology.orchestra.run.smithi069.stdout:mgr epoch 5 is available 2024-04-02T04:34:15.490 INFO:teuthology.orchestra.run.smithi069.stdout:Setting orchestrator backend to cephadm... 2024-04-02T04:34:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:16 smithi069 bash[21528]: cluster 2024-04-02T04:34:15.921278+0000 mon.a (mon.0) 57 : cluster 0 mgrmap e8: y(active, since 2s) 2024-04-02T04:34:17.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:16 smithi069 bash[21528]: audit 2024-04-02T04:34:16.713753+0000 mgr.y (mgr.14120) 8 : audit 0 from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:17.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:16 smithi069 bash[21528]: audit 2024-04-02T04:34:16.722829+0000 mon.a (mon.0) 58 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:17.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:16 smithi069 bash[21528]: audit 2024-04-02T04:34:16.768835+0000 mon.a (mon.0) 59 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:34:17.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.310+0000 7f7f9f838700 1 Processor -- start 2024-04-02T04:34:17.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.310+0000 7f7f9f838700 1 -- start start 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.310+0000 7f7f9f838700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f98108d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.310+0000 7f7f9f838700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f98109310 con 0x7f7f98108920 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f98108d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f98108d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55822/0 (socket says 172.21.15.69:55822) 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 -- 172.21.15.69:0/2224732687 learned_addr learned my addr 172.21.15.69:0/2224732687 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 -- 172.21.15.69:0/2224732687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f98109b20 con 0x7f7f98108920 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 --2- 172.21.15.69:0/2224732687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f98108d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0098a0 tx=0x7f7f8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8647759564869f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8bfff700 1 -- 172.21.15.69:0/2224732687 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f8c004160 con 0x7f7f98108920 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8bfff700 1 -- 172.21.15.69:0/2224732687 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f7f8c004300 con 0x7f7f98108920 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8bfff700 1 -- 172.21.15.69:0/2224732687 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f8c011960 con 0x7f7f98108920 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/2224732687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 msgr2=0x7f7f98108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:17.310 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 --2- 172.21.15.69:0/2224732687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f98108d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0098a0 tx=0x7f7f8c009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/2224732687 shutdown_connections 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 --2- 172.21.15.69:0/2224732687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f98108d40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/2224732687 >> 172.21.15.69:0/2224732687 conn(0x7f7f9807be20 msgr2=0x7f7f9807c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/2224732687 shutdown_connections 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/2224732687 wait complete. 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 librados: starting msgr at 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 librados: starting objecter 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 Processor -- start 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- start start 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 librados: setting wanted keys 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 librados: calling monclient init 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f9819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810cd50 con 0x7f7f98108920 2024-04-02T04:34:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f9819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f9819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55830/0 (socket says 172.21.15.69:55830) 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 -- 172.21.15.69:0/911343058 learned_addr learned my addr 172.21.15.69:0/911343058 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 -- 172.21.15.69:0/911343058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9819cc80 con 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9d5d4700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f9819c740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c000c00 tx=0x7f7f8c003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8a7fc700 10 client.?.objecter ms_handle_connect 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8a7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f8c0038b0 con 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f7f8c0248a0 con 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f8c01a6a0 con 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9819cf10 con 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9819fc00 con 0x7f7f98108920 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.314+0000 7f7f9f838700 1 librados: init done 2024-04-02T04:34:17.312 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f7f8c024440 con 0x7f7f98108920 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f98063870 con 0x7f7f98108920 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f7f8403a8c0 0x7f7f8403cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7f8c04d220 con 0x7f7f98108920 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 10 client.14132.objecter ms_dispatch 0x7f7f98001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 20 client.14132.objecter dump_active .. 0 homeless 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f9cdd3700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f7f8403a8c0 0x7f7f8403cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f9cdd3700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f7f8403a8c0 0x7f7f8403cd80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7f94006fd0 tx=0x7f7f94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.318+0000 7f7f8a7fc700 10 client.14132.objecter ms_handle_connect 0x7f7f8403a8c0 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.322+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7f8c00e260 con 0x7f7f98108920 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.706+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f7f98106410 con 0x7f7f8403a8c0 2024-04-02T04:34:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f8a7fc700 1 -- 172.21.15.69:0/911343058 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7f98106410 con 0x7f7f8403a8c0 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 10 librados: watch_flush enter 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 10 librados: watch_flush exit 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 20 client.14132.objecter shutdown clearing up homeless session... 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 10 client.14132.objecter successfully canceled tick 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f7f8403a8c0 msgr2=0x7f7f8403cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f7f8403a8c0 0x7f7f8403cd80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7f94006fd0 tx=0x7f7f94006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 msgr2=0x7f7f9819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f9819c740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c000c00 tx=0x7f7f8c003dc0 comp rx=0 tx=0).stop 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 shutdown_connections 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f7f8403a8c0 0x7f7f8403cd80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 --2- 172.21.15.69:0/911343058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f98108920 0x7f7f9819c740 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c000c00 tx=0x7f7f8c003dc0 comp rx=0 tx=0).stop 2024-04-02T04:34:17.314 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 >> 172.21.15.69:0/911343058 conn(0x7f7f9807be20 msgr2=0x7f7f98105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:17.315 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 shutdown_connections 2024-04-02T04:34:17.315 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 -- 172.21.15.69:0/911343058 wait complete. 2024-04-02T04:34:17.315 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:16.722+0000 7f7f9f838700 1 librados: shutdown 2024-04-02T04:34:19.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-02T04:34:19.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 Processor -- start 2024-04-02T04:34:19.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- start start 2024-04-02T04:34:19.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:19.058 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c107110 con 0x7fd71c106720 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45568/0 (socket says 172.21.15.69:45568) 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71b7fe700 1 -- 172.21.15.69:0/2180886278 learned_addr learned my addr 172.21.15.69:0/2180886278 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71b7fe700 1 -- 172.21.15.69:0/2180886278 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c107920 con 0x7fd71c106720 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71b7fe700 1 --2- 172.21.15.69:0/2180886278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c106b40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0098a0 tx=0x7fd70c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825695957b950ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71a7fc700 1 -- 172.21.15.69:0/2180886278 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c004160 con 0x7fd71c106720 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71a7fc700 1 -- 172.21.15.69:0/2180886278 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fd70c004300 con 0x7fd71c106720 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd71a7fc700 1 -- 172.21.15.69:0/2180886278 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c011960 con 0x7fd71c106720 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- 172.21.15.69:0/2180886278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 msgr2=0x7fd71c106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 --2- 172.21.15.69:0/2180886278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c106b40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0098a0 tx=0x7fd70c009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- 172.21.15.69:0/2180886278 shutdown_connections 2024-04-02T04:34:19.059 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 --2- 172.21.15.69:0/2180886278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c106b40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- 172.21.15.69:0/2180886278 >> 172.21.15.69:0/2180886278 conn(0x7fd71c101ca0 msgr2=0x7fd71c104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- 172.21.15.69:0/2180886278 shutdown_connections 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.093+0000 7fd722116700 1 -- 172.21.15.69:0/2180886278 wait complete. 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 librados: starting msgr at 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 librados: starting objecter 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 Processor -- start 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 -- start start 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 librados: setting wanted keys 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 librados: calling monclient init 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10ab50 con 0x7fd71c106720 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c19a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45578/0 (socket says 172.21.15.69:45578) 2024-04-02T04:34:19.060 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71b7fe700 1 -- 172.21.15.69:0/3204262162 learned_addr learned my addr 172.21.15.69:0/3204262162 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71b7fe700 1 -- 172.21.15.69:0/3204262162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c19aab0 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71b7fe700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c19a570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd70c000c00 tx=0x7fd70c003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 10 client.?.objecter ms_handle_connect 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c0038b0 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c19ad40 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fd70c0248a0 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c01a6a0 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c19da30 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 librados: init done 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fd70c01a950 con 0x7fd71c106720 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd70403a8c0 0x7fd70403cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:19.061 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd70c04d230 con 0x7fd71c106720 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 10 client.14134.objecter ms_dispatch 0x7fd71c001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71affd700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd70403a8c0 0x7fd70403cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 20 client.14134.objecter dump_active .. 0 homeless 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71c063870 con 0x7fd71c106720 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd71affd700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd70403a8c0 0x7fd70403cd80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd710006fd0 tx=0x7fd710006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.097+0000 7fd718ff9700 10 client.14134.objecter ms_handle_connect 0x7fd70403a8c0 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.105+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd70c02a2a0 con 0x7fd71c106720 2024-04-02T04:34:19.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.469+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fd71c102e50 con 0x7fd70403a8c0 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.469+0000 7fd718ff9700 1 -- 172.21.15.69:0/3204262162 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fd71c102e50 con 0x7fd70403a8c0 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.469+0000 7fd722116700 10 librados: watch_flush enter 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.469+0000 7fd722116700 10 librados: watch_flush exit 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.469+0000 7fd722116700 20 client.14134.objecter shutdown clearing up homeless session... 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.469+0000 7fd722116700 10 client.14134.objecter successfully canceled tick 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd70403a8c0 msgr2=0x7fd70403cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd70403a8c0 0x7fd70403cd80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd710006fd0 tx=0x7fd710006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 msgr2=0x7fd71c19a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c19a570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd70c000c00 tx=0x7fd70c003dc0 comp rx=0 tx=0).stop 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 shutdown_connections 2024-04-02T04:34:19.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd70403a8c0 0x7fd70403cd80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:19.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 --2- 172.21.15.69:0/3204262162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c106720 0x7fd71c19a570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:19.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 >> 172.21.15.69:0/3204262162 conn(0x7fd71c101ca0 msgr2=0x7fd71c102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:19.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 shutdown_connections 2024-04-02T04:34:19.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 -- 172.21.15.69:0/3204262162 wait complete. 2024-04-02T04:34:19.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:18.473+0000 7fd722116700 1 librados: shutdown 2024-04-02T04:34:19.064 INFO:teuthology.orchestra.run.smithi069.stdout:Generating ssh key... 2024-04-02T04:34:19.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:18 smithi069 bash[21528]: audit 2024-04-02T04:34:18.473029+0000 mgr.y (mgr.14120) 9 : audit 0 from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:20.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: Generating public/private rsa key pair. 2024-04-02T04:34:20.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: Your identification has been saved in /tmp/tmpan5o7173/key. 2024-04-02T04:34:20.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: Your public key has been saved in /tmp/tmpan5o7173/key.pub. 2024-04-02T04:34:20.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: The key fingerprint is: 2024-04-02T04:34:20.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: SHA256:8dZzLZhOFr3lAX/XjKC/z1lGBs+jvGVQHuHW4epoVsI ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:34:20.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: The key's randomart image is: 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: +---[RSA 3072]----+ 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | .. o | 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | . o=o=| 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | . . . +OB| 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | o.o +=O=| 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | S oEO++oB| 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | . +*+o+.| 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | +o+ oo| 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | o o++ | 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: | .+ | 2024-04-02T04:34:20.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:20 smithi069 bash[21782]: +----[SHA256]-----+ 2024-04-02T04:34:21.069 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860cb2f700 1 Processor -- start 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860cb2f700 1 -- start start 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860cb2f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f8608108d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860cb2f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8608109310 con 0x7f8608108920 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f8608108d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f8608108d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45590/0 (socket says 172.21.15.69:45590) 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860659c700 1 -- 172.21.15.69:0/3427577987 learned_addr learned my addr 172.21.15.69:0/3427577987 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860659c700 1 -- 172.21.15.69:0/3427577987 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8608109b20 con 0x7f8608108920 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860659c700 1 --2- 172.21.15.69:0/3427577987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f8608108d40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009e80 tx=0x7f85f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e59ea86e9029976 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860559a700 1 -- 172.21.15.69:0/3427577987 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85f8017070 con 0x7f8608108920 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860559a700 1 -- 172.21.15.69:0/3427577987 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f85f800b8c0 con 0x7f8608108920 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.889+0000 7f860559a700 1 -- 172.21.15.69:0/3427577987 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85f8004340 con 0x7f8608108920 2024-04-02T04:34:21.070 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/3427577987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 msgr2=0x7f8608108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 --2- 172.21.15.69:0/3427577987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f8608108d40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009e80 tx=0x7f85f800b330 comp rx=0 tx=0).stop 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/3427577987 shutdown_connections 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 --2- 172.21.15.69:0/3427577987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f8608108d40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/3427577987 >> 172.21.15.69:0/3427577987 conn(0x7f860807bde0 msgr2=0x7f860807c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/3427577987 shutdown_connections 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/3427577987 wait complete. 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 librados: starting msgr at 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 librados: starting objecter 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 Processor -- start 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- start start 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 librados: setting wanted keys 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 librados: calling monclient init 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f860819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f860810cd50 con 0x7f8608108920 2024-04-02T04:34:21.071 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f860819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f860819c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45594/0 (socket says 172.21.15.69:45594) 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860659c700 1 -- 172.21.15.69:0/1893347384 learned_addr learned my addr 172.21.15.69:0/1893347384 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860659c700 1 -- 172.21.15.69:0/1893347384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860819cba0 con 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860659c700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f860819c660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009530 tx=0x7f85f800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 10 client.?.objecter ms_handle_connect 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85f8021020 con 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860819ce30 con 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f85f8003b50 con 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85f8017040 con 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860819fb20 con 0x7f8608108920 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 librados: init done 2024-04-02T04:34:21.072 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f85f8024020 con 0x7f8608108920 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8608063870 con 0x7f8608108920 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f85f403a800 0x7f85f403ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f85f800f050 con 0x7f8608108920 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 10 client.14136.objecter ms_dispatch 0x7f8608001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f85ff7fe700 20 client.14136.objecter dump_active .. 0 homeless 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.893+0000 7f8605d9b700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f85f403a800 0x7f85f403ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.897+0000 7f85ff7fe700 10 client.14136.objecter ms_handle_connect 0x7f85f403a800 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.897+0000 7f8605d9b700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f85f403a800 0x7f85f403ccc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85f0006fd0 tx=0x7f85f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:19.901+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f85f802b020 con 0x7f8608108920 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.257+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f86081062c0 con 0x7f85f403a800 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.537+0000 7f85ff7fe700 1 -- 172.21.15.69:0/1893347384 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f86081062c0 con 0x7f85f403a800 2024-04-02T04:34:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 10 librados: watch_flush enter 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 10 librados: watch_flush exit 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 20 client.14136.objecter shutdown clearing up homeless session... 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 10 client.14136.objecter successfully canceled tick 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f85f403a800 msgr2=0x7f85f403ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f85f403a800 0x7f85f403ccc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85f0006fd0 tx=0x7f85f0006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 msgr2=0x7f860819c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f860819c660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009530 tx=0x7f85f800bf10 comp rx=0 tx=0).stop 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 shutdown_connections 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f85f403a800 0x7f85f403ccc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 --2- 172.21.15.69:0/1893347384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8608108920 0x7f860819c660 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 >> 172.21.15.69:0/1893347384 conn(0x7f860807bde0 msgr2=0x7f8608105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 shutdown_connections 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 -- 172.21.15.69:0/1893347384 wait complete. 2024-04-02T04:34:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:20.541+0000 7f860cb2f700 1 librados: shutdown 2024-04-02T04:34:22.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:21 smithi069 bash[21528]: audit 2024-04-02T04:34:20.260997+0000 mgr.y (mgr.14120) 10 : audit 0 from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:22.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:21 smithi069 bash[21528]: cephadm 2024-04-02T04:34:20.261294+0000 mgr.y (mgr.14120) 11 : cephadm 1 Generating ssh key... 2024-04-02T04:34:22.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:21 smithi069 bash[21528]: audit 2024-04-02T04:34:20.534614+0000 mon.a (mon.0) 60 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:22.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:21 smithi069 bash[21528]: audit 2024-04-02T04:34:20.539578+0000 mon.a (mon.0) 61 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:22.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS0flvN6atN0+P7osaPJSeC7Dj2GKtRjRWoWp7YtcsE7NUNhPIwFTUDq8PVfDQ3K1s0Vrbfuq3R2frgdSxFCigKVbH3YYHMz7SxmW6Z43q16O4UDO1rw5JwxfsuKZ6dHMffs5Hnzh409sLtpwix7Q0tnu4D8exlvUOEARyl6ioEp/pE2X4V2eoPfkNbFz4un4BoxImMSBfQTj4dq1sv7MEZI1Rb1H9XFVYm9fOUMo4hx+V3dCcAGxrFmRtHnrSA5VUCAAH9CUu/9gIazCEZ2vE4hA++X/f0BEo7WvmpNDVaIDfwE4V2ruc2cF3deyetW2Wzdi/KRvK4S8zEhmAJSO9nejSJqNZiqtWCDfsouBJz72e+aSVm4bsJ591AlbE/93F9SB6UikhneKQ0Y/IeOHlfzq/mk8MgXC/3vyaVmCByVzEP+syKGpWmX/VvsVHezupBorVY0yeX5fdbFA9P3UyazhWEqaHcq1+RyJWczM6EOiPct+LsZjXTRI5F50D2kc= ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:34:22.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 Processor -- start 2024-04-02T04:34:22.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- start start 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd94105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9407a6c0 con 0x7fcd94105000 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd94105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd94105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45604/0 (socket says 172.21.15.69:45604) 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 -- 172.21.15.69:0/1900906208 learned_addr learned my addr 172.21.15.69:0/1900906208 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 -- 172.21.15.69:0/1900906208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd94105960 con 0x7fcd94105000 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 --2- 172.21.15.69:0/1900906208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd94105420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0098a0 tx=0x7fcd7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33db489d338c4e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd937fe700 1 -- 172.21.15.69:0/1900906208 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd7c004160 con 0x7fcd94105000 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd937fe700 1 -- 172.21.15.69:0/1900906208 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fcd7c004300 con 0x7fcd94105000 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd937fe700 1 -- 172.21.15.69:0/1900906208 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd7c011960 con 0x7fcd94105000 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/1900906208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 msgr2=0x7fcd94105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 --2- 172.21.15.69:0/1900906208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd94105420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0098a0 tx=0x7fcd7c009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:22.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/1900906208 shutdown_connections 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 --2- 172.21.15.69:0/1900906208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd94105420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/1900906208 >> 172.21.15.69:0/1900906208 conn(0x7fcd94100bc0 msgr2=0x7fcd94103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/1900906208 shutdown_connections 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/1900906208 wait complete. 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 librados: starting msgr at 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 librados: starting objecter 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 Processor -- start 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- start start 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 librados: setting wanted keys 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 librados: calling monclient init 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd9419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd9ab4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd94106930 con 0x7fcd94105000 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd9419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd9419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45606/0 (socket says 172.21.15.69:45606) 2024-04-02T04:34:22.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.873+0000 7fcd988eb700 1 -- 172.21.15.69:0/959324558 learned_addr learned my addr 172.21.15.69:0/959324558 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd988eb700 1 -- 172.21.15.69:0/959324558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9419cc40 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd988eb700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd9419c700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c011740 tx=0x7fcd7c003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 10 client.?.objecter ms_handle_connect 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd7c0038b0 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7fcd7c0248a0 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/959324558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9419ced0 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd7c01a6b0 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd9ab4f700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd9ab4f700 1 -- 172.21.15.69:0/959324558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd9419fbc0 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd9ab4f700 1 librados: init done 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fcd7c01a9b0 con 0x7fcd94105000 2024-04-02T04:34:22.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd78005120 con 0x7fcd94105000 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fcd8003a870 0x7fcd8003cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcd7c04da10 con 0x7fcd94105000 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 10 client.14138.objecter ms_dispatch 0x7fcd94001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 20 client.14138.objecter dump_active .. 0 homeless 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd93fff700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fcd8003a870 0x7fcd8003cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd93fff700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fcd8003a870 0x7fcd8003cd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006fd0 tx=0x7fcd84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.877+0000 7fcd91ffb700 10 client.14138.objecter ms_handle_connect 0x7fcd8003a870 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:21.881+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd7c024a50 con 0x7fcd94105000 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.233+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fcd78000bc0 con 0x7fcd8003a870 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.233+0000 7fcd91ffb700 1 -- 172.21.15.69:0/959324558 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fcd78000bc0 con 0x7fcd8003a870 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 10 librados: watch_flush enter 2024-04-02T04:34:22.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 10 librados: watch_flush exit 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 20 client.14138.objecter shutdown clearing up homeless session... 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 10 client.14138.objecter successfully canceled tick 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fcd8003a870 msgr2=0x7fcd8003cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fcd8003a870 0x7fcd8003cd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006fd0 tx=0x7fcd84006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 msgr2=0x7fcd9419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd9419c700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c011740 tx=0x7fcd7c003dc0 comp rx=0 tx=0).stop 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 shutdown_connections 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fcd8003a870 0x7fcd8003cd30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 --2- 172.21.15.69:0/959324558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd94105000 0x7fcd9419c700 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 >> 172.21.15.69:0/959324558 conn(0x7fcd94100bc0 msgr2=0x7fcd941933c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 shutdown_connections 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 -- 172.21.15.69:0/959324558 wait complete. 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:22.237+0000 7fcd8b7fe700 1 librados: shutdown 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-02T04:34:22.871 INFO:teuthology.orchestra.run.smithi069.stdout:Adding key to root@localhost authorized_keys... 2024-04-02T04:34:22.872 INFO:teuthology.orchestra.run.smithi069.stdout:Adding host smithi069... 2024-04-02T04:34:23.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:22 smithi069 bash[21528]: audit 2024-04-02T04:34:22.237725+0000 mgr.y (mgr.14120) 12 : audit 0 from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:24.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:24 smithi069 bash[21528]: audit 2024-04-02T04:34:24.120666+0000 mgr.y (mgr.14120) 13 : audit 0 from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:24.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:24 smithi069 bash[21528]: cephadm 2024-04-02T04:34:24.698612+0000 mgr.y (mgr.14120) 14 : cephadm 1 Deploying cephadm binary to smithi069 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Added host 'smithi069' with addr '172.21.15.69' 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831ebc9700 1 Processor -- start 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831ebc9700 1 -- start start 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831ebc9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f8318108d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831ebc9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8318109310 con 0x7f8318108920 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831c965700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f8318108d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831c965700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f8318108d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45616/0 (socket says 172.21.15.69:45616) 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831c965700 1 -- 172.21.15.69:0/4068387487 learned_addr learned my addr 172.21.15.69:0/4068387487 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831c965700 1 -- 172.21.15.69:0/4068387487 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8318109b20 con 0x7f8318108920 2024-04-02T04:34:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831c965700 1 --2- 172.21.15.69:0/4068387487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f8318108d40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83000098a0 tx=0x7f8300009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae6d0d3e3953687 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f83177fe700 1 -- 172.21.15.69:0/4068387487 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8300004160 con 0x7f8318108920 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f83177fe700 1 -- 172.21.15.69:0/4068387487 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f8300004300 con 0x7f8318108920 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f83177fe700 1 -- 172.21.15.69:0/4068387487 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8300011960 con 0x7f8318108920 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831ebc9700 1 -- 172.21.15.69:0/4068387487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 msgr2=0x7f8318108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.741+0000 7f831ebc9700 1 --2- 172.21.15.69:0/4068387487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f8318108d40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83000098a0 tx=0x7f8300009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/4068387487 shutdown_connections 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 --2- 172.21.15.69:0/4068387487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f8318108d40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/4068387487 >> 172.21.15.69:0/4068387487 conn(0x7f831807be20 msgr2=0x7f831807c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/4068387487 shutdown_connections 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/4068387487 wait complete. 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 librados: starting msgr at 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 librados: starting objecter 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 Processor -- start 2024-04-02T04:34:30.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- start start 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 librados: setting wanted keys 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 librados: calling monclient init 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f831819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f831810cd50 con 0x7f8318108920 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831c965700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f831819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831c965700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f831819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45632/0 (socket says 172.21.15.69:45632) 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831c965700 1 -- 172.21.15.69:0/3521098594 learned_addr learned my addr 172.21.15.69:0/3521098594 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831c965700 1 -- 172.21.15.69:0/3521098594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831819cc80 con 0x7f8318108920 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831c965700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f831819c740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8300006af0 tx=0x7f8300003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 10 client.?.objecter ms_handle_connect 0x7f8318108920 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8300011950 con 0x7f8318108920 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f830001a890 con 0x7f8318108920 2024-04-02T04:34:30.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831819cf10 con 0x7f8318108920 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83000196e0 con 0x7f8318108920 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f831819fc00 con 0x7f8318108920 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 librados: init done 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f83000198d0 con 0x7f8318108920 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8318063870 con 0x7f8318108920 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f830403a870 0x7f830403cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f830004d540 con 0x7f8318108920 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 10 client.14140.objecter ms_dispatch 0x7f8318001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 20 client.14140.objecter dump_active .. 0 homeless 2024-04-02T04:34:30.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8317fff700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f830403a870 0x7f830403cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8317fff700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f830403a870 0x7f830403cd30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8308006fd0 tx=0x7f8308006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.745+0000 7f8315ffb700 10 client.14140.objecter ms_handle_connect 0x7f830403a870 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:23.753+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f83000042c0 con 0x7f8318108920 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:24.113+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}) -- 0x7f8318106410 con 0x7f830403a870 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:28.749+0000 7f831e3c8700 10 client.14140.objecter tick 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.429+0000 7f8315ffb700 1 -- 172.21.15.69:0/3521098594 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8318106410 con 0x7f830403a870 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 10 librados: watch_flush enter 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 10 librados: watch_flush exit 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 20 client.14140.objecter shutdown clearing up homeless session... 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 10 client.14140.objecter successfully canceled tick 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f830403a870 msgr2=0x7f830403cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f830403a870 0x7f830403cd30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8308006fd0 tx=0x7f8308006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 msgr2=0x7f831819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f831819c740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8300006af0 tx=0x7f8300003980 comp rx=0 tx=0).stop 2024-04-02T04:34:30.160 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 shutdown_connections 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f830403a870 0x7f830403cd30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 --2- 172.21.15.69:0/3521098594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8318108920 0x7f831819c740 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 >> 172.21.15.69:0/3521098594 conn(0x7f831807be20 msgr2=0x7f8318105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 shutdown_connections 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 -- 172.21.15.69:0/3521098594 wait complete. 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:29.433+0000 7f831ebc9700 1 librados: shutdown 2024-04-02T04:34:30.161 INFO:teuthology.orchestra.run.smithi069.stdout:Deploying unmanaged mon service... 2024-04-02T04:34:30.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:30 smithi069 bash[21528]: audit 2024-04-02T04:34:29.434559+0000 mon.a (mon.0) 62 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:30.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:30 smithi069 bash[21528]: cephadm 2024-04-02T04:34:29.434997+0000 mgr.y (mgr.14120) 15 : cephadm 1 Added host smithi069 2024-04-02T04:34:30.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:30 smithi069 bash[21528]: audit 2024-04-02T04:34:29.481960+0000 mon.a (mon.0) 63 : audit 0 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 Processor -- start 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- start start 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4109600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87a407a6c0 con 0x7f87a4107210 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37912/0 (socket says 172.21.15.69:37912) 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 -- 172.21.15.69:0/3832864697 learned_addr learned my addr 172.21.15.69:0/3832864697 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 -- 172.21.15.69:0/3832864697 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a4109b40 con 0x7f87a4107210 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 --2- 172.21.15.69:0/3832864697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4109600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f879c009e80 tx=0x7f879c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f74f315b0345158 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a1ffb700 1 -- 172.21.15.69:0/3832864697 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f879c017070 con 0x7f87a4107210 2024-04-02T04:34:32.304 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a1ffb700 1 -- 172.21.15.69:0/3832864697 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f879c00b8c0 con 0x7f87a4107210 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a1ffb700 1 -- 172.21.15.69:0/3832864697 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f879c004340 con 0x7f87a4107210 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- 172.21.15.69:0/3832864697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 msgr2=0x7f87a4109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 --2- 172.21.15.69:0/3832864697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4109600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f879c009e80 tx=0x7f879c00b330 comp rx=0 tx=0).stop 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- 172.21.15.69:0/3832864697 shutdown_connections 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 --2- 172.21.15.69:0/3832864697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4109600 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- 172.21.15.69:0/3832864697 >> 172.21.15.69:0/3832864697 conn(0x7f87a4100bc0 msgr2=0x7f87a4103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- 172.21.15.69:0/3832864697 shutdown_connections 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- 172.21.15.69:0/3832864697 wait complete. 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 librados: starting msgr at 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 librados: starting objecter 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 Processor -- start 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- start start 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 librados: setting wanted keys 2024-04-02T04:34:32.305 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 librados: calling monclient init 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4106a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a982b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87a410aa40 con 0x7f87a4107210 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4106a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4106a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37920/0 (socket says 172.21.15.69:37920) 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 -- 172.21.15.69:0/2136735207 learned_addr learned my addr 172.21.15.69:0/2136735207 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 -- 172.21.15.69:0/2136735207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a4106fd0 con 0x7f87a4107210 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a2ffd700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4106a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f879c00d010 tx=0x7f879c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a8829700 10 client.?.objecter ms_handle_connect 0x7f87a4107210 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.393+0000 7f87a8829700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f879c017040 con 0x7f87a4107210 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1024+0+0 (secure 0 0 0) 0x7f879c023070 con 0x7f87a4107210 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a41051c0 con 0x7f87a4107210 2024-04-02T04:34:32.306 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f879c01e420 con 0x7f87a4107210 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a982b700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a4105700 con 0x7f87a4107210 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a982b700 1 librados: init done 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f879c012430 con 0x7f87a4107210 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f879003a8c0 0x7f879003cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f879c00f050 con 0x7f87a4107210 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 10 client.14142.objecter ms_dispatch 0x7f87a4001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:32.307 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a4063870 con 0x7f87a4107210 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 20 client.14142.objecter dump_active .. 0 homeless 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a27fc700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f879003a8c0 0x7f879003cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a8829700 10 client.14142.objecter ms_handle_connect 0x7f879003a8c0 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.397+0000 7f87a27fc700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f879003a8c0 0x7f879003cd80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fd0 tx=0x7f878c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.401+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f879c019260 con 0x7f87a4107210 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.789+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f87a4105e20 con 0x7f879003a8c0 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.797+0000 7f87a8829700 1 -- 172.21.15.69:0/2136735207 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f87a4105e20 con 0x7f879003a8c0 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 10 librados: watch_flush enter 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 10 librados: watch_flush exit 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 20 client.14142.objecter shutdown clearing up homeless session... 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 10 client.14142.objecter successfully canceled tick 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f879003a8c0 msgr2=0x7f879003cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f879003a8c0 0x7f879003cd80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fd0 tx=0x7f878c006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:32.308 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 msgr2=0x7f87a4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4106a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f879c00d010 tx=0x7f879c00bf10 comp rx=0 tx=0).stop 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 shutdown_connections 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f879003a8c0 0x7f879003cd80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 --2- 172.21.15.69:0/2136735207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87a4107210 0x7f87a4106a90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 >> 172.21.15.69:0/2136735207 conn(0x7f87a4100bc0 msgr2=0x7f87a41014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 shutdown_connections 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 -- 172.21.15.69:0/2136735207 wait complete. 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:31.801+0000 7f87a982b700 1 librados: shutdown 2024-04-02T04:34:32.309 INFO:teuthology.orchestra.run.smithi069.stdout:Deploying unmanaged mgr service... 2024-04-02T04:34:33.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:32 smithi069 bash[21528]: audit 2024-04-02T04:34:31.796081+0000 mgr.y (mgr.14120) 16 : audit 0 from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:33.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:32 smithi069 bash[21528]: cephadm 2024-04-02T04:34:31.797901+0000 mgr.y (mgr.14120) 17 : cephadm 1 Saving service mon spec with placement count:5 2024-04-02T04:34:33.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:32 smithi069 bash[21528]: audit 2024-04-02T04:34:31.802489+0000 mon.a (mon.0) 64 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:33.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:32 smithi069 bash[21528]: audit 2024-04-02T04:34:31.904392+0000 mon.a (mon.0) 65 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:33.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:32 smithi069 bash[21528]: audit 2024-04-02T04:34:32.380444+0000 mon.a (mon.0) 66 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:34.030 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f9428be6700 1 Processor -- start 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f9428be6700 1 -- start start 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f9428be6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942407abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f9428be6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f942407b1a0 con 0x7f942407c770 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f942259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942407abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f942259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942407abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37926/0 (socket says 172.21.15.69:37926) 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.093+0000 7f942259c700 1 -- 172.21.15.69:0/2372246299 learned_addr learned my addr 172.21.15.69:0/2372246299 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 -- 172.21.15.69:0/2372246299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942407b320 con 0x7f942407c770 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 --2- 172.21.15.69:0/2372246299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942407abd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94140098a0 tx=0x7f9414009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79dbc0b2ad4b54d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942159a700 1 -- 172.21.15.69:0/2372246299 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9414004160 con 0x7f942407c770 2024-04-02T04:34:34.031 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942159a700 1 -- 172.21.15.69:0/2372246299 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9414004300 con 0x7f942407c770 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942159a700 1 -- 172.21.15.69:0/2372246299 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9414011930 con 0x7f942407c770 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/2372246299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 msgr2=0x7f942407abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 --2- 172.21.15.69:0/2372246299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942407abd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94140098a0 tx=0x7f9414009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/2372246299 shutdown_connections 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 --2- 172.21.15.69:0/2372246299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942407abd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/2372246299 >> 172.21.15.69:0/2372246299 conn(0x7f9424101d00 msgr2=0x7f9424104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/2372246299 shutdown_connections 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/2372246299 wait complete. 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 librados: starting msgr at 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 librados: starting objecter 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 Processor -- start 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- start start 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 librados: setting wanted keys 2024-04-02T04:34:34.032 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 librados: calling monclient init 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f942410ccd0 con 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37932/0 (socket says 172.21.15.69:37932) 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 -- 172.21.15.69:0/3276666861 learned_addr learned my addr 172.21.15.69:0/3276666861 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 -- 172.21.15.69:0/3276666861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942419ccd0 con 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f942259c700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942419c790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9414009fb0 tx=0x7f94140046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f941b7fe700 10 client.?.objecter ms_handle_connect 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f941b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94140037d0 con 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9414003970 con 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942419cf60 con 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9414024410 con 0x7f942407c770 2024-04-02T04:34:34.033 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942419fc50 con 0x7f942407c770 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.097+0000 7f9428be6700 1 librados: init done 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f9414022020 con 0x7f942407c770 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f941003a8c0 0x7f941003cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9424063870 con 0x7f942407c770 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f941404ce60 con 0x7f942407c770 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 10 client.14144.objecter ms_dispatch 0x7f9424001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 20 client.14144.objecter dump_active .. 0 homeless 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f9421d9b700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f941003a8c0 0x7f941003cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f9421d9b700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f941003a8c0 0x7f941003cd80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f940c00ad30 tx=0x7f940c0093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.101+0000 7f941b7fe700 10 client.14144.objecter ms_handle_connect 0x7f941003a8c0 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.105+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f941401a7b0 con 0x7f942407c770 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.477+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9424102e10 con 0x7f941003a8c0 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.485+0000 7f941b7fe700 1 -- 172.21.15.69:0/3276666861 <== mgr.14120 v2:172.21.15.69:6800/578732137 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9424102e10 con 0x7f941003a8c0 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 10 librados: watch_flush enter 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 10 librados: watch_flush exit 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 20 client.14144.objecter shutdown clearing up homeless session... 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 10 client.14144.objecter successfully canceled tick 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f941003a8c0 msgr2=0x7f941003cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f941003a8c0 0x7f941003cd80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f940c00ad30 tx=0x7f940c0093f0 comp rx=0 tx=0).stop 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 msgr2=0x7f942419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942419c790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9414009fb0 tx=0x7f94140046a0 comp rx=0 tx=0).stop 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 shutdown_connections 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f941003a8c0 0x7f941003cd80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:34.035 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 --2- 172.21.15.69:0/3276666861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f942407c770 0x7f942419c790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:34.036 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 >> 172.21.15.69:0/3276666861 conn(0x7f9424101d00 msgr2=0x7f9424102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:34.036 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 shutdown_connections 2024-04-02T04:34:34.036 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 -- 172.21.15.69:0/3276666861 wait complete. 2024-04-02T04:34:34.036 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:33.489+0000 7f9428be6700 1 librados: shutdown 2024-04-02T04:34:34.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:34 smithi069 bash[21528]: audit 2024-04-02T04:34:33.482614+0000 mgr.y (mgr.14120) 18 : audit 0 from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:34.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:34 smithi069 bash[21528]: cephadm 2024-04-02T04:34:33.485162+0000 mgr.y (mgr.14120) 19 : cephadm 1 Saving service mgr spec with placement count:2 2024-04-02T04:34:34.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:34 smithi069 bash[21528]: audit 2024-04-02T04:34:33.489909+0000 mon.a (mon.0) 67 : audit 1 from='mgr.14120 172.21.15.69:0/323735001' entity='mgr.y' 2024-04-02T04:34:34.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:34 smithi069 bash[21528]: cluster 2024-04-02T04:34:33.867482+0000 mgr.y (mgr.14120) 20 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:34:35.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2bcc5e700 1 Processor -- start 2024-04-02T04:34:35.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2bcc5e700 1 -- start start 2024-04-02T04:34:35.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2bcc5e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b8106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:35.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2bcc5e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b8107110 con 0x7fd2b8106720 2024-04-02T04:34:35.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2b659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b8106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:35.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2b659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b8106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37934/0 (socket says 172.21.15.69:37934) 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2b659c700 1 -- 172.21.15.69:0/1919907569 learned_addr learned my addr 172.21.15.69:0/1919907569 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.805+0000 7fd2b659c700 1 -- 172.21.15.69:0/1919907569 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b8107920 con 0x7fd2b8106720 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b659c700 1 --2- 172.21.15.69:0/1919907569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b8106b40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80098a0 tx=0x7fd2a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b5903c4d4b6475 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b559a700 1 -- 172.21.15.69:0/1919907569 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2a8004160 con 0x7fd2b8106720 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b559a700 1 -- 172.21.15.69:0/1919907569 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd2a8004300 con 0x7fd2b8106720 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b559a700 1 -- 172.21.15.69:0/1919907569 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2a8011930 con 0x7fd2b8106720 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/1919907569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 msgr2=0x7fd2b8106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 --2- 172.21.15.69:0/1919907569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b8106b40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80098a0 tx=0x7fd2a8009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/1919907569 shutdown_connections 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 --2- 172.21.15.69:0/1919907569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b8106b40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:35.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/1919907569 >> 172.21.15.69:0/1919907569 conn(0x7fd2b8101ca0 msgr2=0x7fd2b8104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/1919907569 shutdown_connections 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/1919907569 wait complete. 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 librados: starting msgr at 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 librados: starting objecter 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 Processor -- start 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- start start 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 librados: setting wanted keys 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 librados: calling monclient init 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b807c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b810ab50 con 0x7fd2b8106720 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b807c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b807c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37946/0 (socket says 172.21.15.69:37946) 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b659c700 1 -- 172.21.15.69:0/52703436 learned_addr learned my addr 172.21.15.69:0/52703436 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:35.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b659c700 1 -- 172.21.15.69:0/52703436 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b807c940 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2b659c700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b807c400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8003fa0 tx=0x7fd2a8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2a77fe700 10 client.?.objecter ms_handle_connect 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2a77fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2a80036a0 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b807ab30 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd2a801a430 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2a80193c0 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b807b070 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.809+0000 7fd2bcc5e700 1 librados: init done 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b810acd0 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fd2a801a930 con 0x7fd2b8106720 2024-04-02T04:34:35.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd2a003a8c0 0x7fd2a003cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd2a804d7f0 con 0x7fd2b8106720 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 10 client.14146.objecter ms_dispatch 0x7fd2b8001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 20 client.14146.objecter dump_active .. 0 homeless 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2b5d9b700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd2a003a8c0 0x7fd2a003cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2a77fe700 10 client.14146.objecter ms_handle_connect 0x7fd2a003a8c0 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.813+0000 7fd2b5d9b700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd2a003a8c0 0x7fd2a003cd80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac006fd0 tx=0x7fd2ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:34.817+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd2a8016260 con 0x7fd2b8106720 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.169+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fd2b807b760 con 0x7fd2b8106720 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.173+0000 7fd2a77fe700 1 -- 172.21.15.69:0/52703436 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fd2a8004300 con 0x7fd2b8106720 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 10 librados: watch_flush enter 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 10 librados: watch_flush exit 2024-04-02T04:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 20 client.14146.objecter shutdown clearing up homeless session... 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 10 client.14146.objecter successfully canceled tick 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd2a003a8c0 msgr2=0x7fd2a003cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd2a003a8c0 0x7fd2a003cd80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac006fd0 tx=0x7fd2ac006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 msgr2=0x7fd2b807c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b807c400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8003fa0 tx=0x7fd2a8011f30 comp rx=0 tx=0).stop 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 shutdown_connections 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fd2a003a8c0 0x7fd2a003cd80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 --2- 172.21.15.69:0/52703436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8106720 0x7fd2b807c400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 >> 172.21.15.69:0/52703436 conn(0x7fd2b8101ca0 msgr2=0x7fd2b8102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 shutdown_connections 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 -- 172.21.15.69:0/52703436 wait complete. 2024-04-02T04:34:35.760 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:35.177+0000 7fd2bcc5e700 1 librados: shutdown 2024-04-02T04:34:36.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:36 smithi069 bash[21528]: audit 2024-04-02T04:34:35.179479+0000 mon.a (mon.0) 68 : audit 1 from='client.? 172.21.15.69:0/52703436' entity='client.admin' 2024-04-02T04:34:36.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:36 smithi069 bash[21528]: cluster 2024-04-02T04:34:35.867994+0000 mgr.y (mgr.14120) 21 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:34:37.453 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 Processor -- start 2024-04-02T04:34:37.453 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- start start 2024-04-02T04:34:37.453 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c106b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:37.453 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c107100 con 0x7fea4c106710 2024-04-02T04:34:37.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c106b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:37.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c106b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37954/0 (socket says 172.21.15.69:37954) 2024-04-02T04:34:37.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 -- 172.21.15.69:0/1149880997 learned_addr learned my addr 172.21.15.69:0/1149880997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:37.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 -- 172.21.15.69:0/1149880997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c107910 con 0x7fea4c106710 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 --2- 172.21.15.69:0/1149880997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c106b30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea44009e80 tx=0x7fea4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de4bae1afc1123d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4a7fc700 1 -- 172.21.15.69:0/1149880997 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea44017070 con 0x7fea4c106710 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4a7fc700 1 -- 172.21.15.69:0/1149880997 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fea4400b8c0 con 0x7fea4c106710 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4a7fc700 1 -- 172.21.15.69:0/1149880997 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea44004400 con 0x7fea4c106710 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- 172.21.15.69:0/1149880997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 msgr2=0x7fea4c106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 --2- 172.21.15.69:0/1149880997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c106b30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea44009e80 tx=0x7fea4400b330 comp rx=0 tx=0).stop 2024-04-02T04:34:37.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- 172.21.15.69:0/1149880997 shutdown_connections 2024-04-02T04:34:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 --2- 172.21.15.69:0/1149880997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c106b30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- 172.21.15.69:0/1149880997 >> 172.21.15.69:0/1149880997 conn(0x7fea4c101cf0 msgr2=0x7fea4c104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- 172.21.15.69:0/1149880997 shutdown_connections 2024-04-02T04:34:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- 172.21.15.69:0/1149880997 wait complete. 2024-04-02T04:34:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 librados: starting msgr at 2024-04-02T04:34:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 librados: starting objecter 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 Processor -- start 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- start start 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 librados: setting wanted keys 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 librados: calling monclient init 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c07bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea51b55700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10ab40 con 0x7fea4c106710 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c07bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c07bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37968/0 (socket says 172.21.15.69:37968) 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.509+0000 7fea4b7fe700 1 -- 172.21.15.69:0/3706018868 learned_addr learned my addr 172.21.15.69:0/3706018868 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea4b7fe700 1 -- 172.21.15.69:0/3706018868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c07c0b0 con 0x7fea4c106710 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea4b7fe700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c07bb70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea44009680 tx=0x7fea440049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 10 client.?.objecter ms_handle_connect 0x7fea4c106710 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:37.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea44020020 con 0x7fea4c106710 2024-04-02T04:34:37.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fea44003c90 con 0x7fea4c106710 2024-04-02T04:34:37.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4c07a2a0 con 0x7fea4c106710 2024-04-02T04:34:37.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea44017040 con 0x7fea4c106710 2024-04-02T04:34:37.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea51b55700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:37.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4c07a7e0 con 0x7fea4c106710 2024-04-02T04:34:37.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea51b55700 1 librados: init done 2024-04-02T04:34:37.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fea44023020 con 0x7fea4c106710 2024-04-02T04:34:37.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fea3403a8c0 0x7fea3403cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fea4400f050 con 0x7fea4c106710 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 10 client.14148.objecter ms_dispatch 0x7fea4c001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 20 client.14148.objecter dump_active .. 0 homeless 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea4affd700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fea3403a8c0 0x7fea3403cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4c10acc0 con 0x7fea4c106710 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea4affd700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fea3403a8c0 0x7fea3403cd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fea40006fd0 tx=0x7fea40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.513+0000 7fea48ff9700 10 client.14148.objecter ms_handle_connect 0x7fea3403a8c0 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.517+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fea4402a020 con 0x7fea4c106710 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.877+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fea4c063870 con 0x7fea4c106710 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.881+0000 7fea48ff9700 1 -- 172.21.15.69:0/3706018868 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fea44012b20 con 0x7fea4c106710 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 10 librados: watch_flush enter 2024-04-02T04:34:37.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 10 librados: watch_flush exit 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 20 client.14148.objecter shutdown clearing up homeless session... 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 10 client.14148.objecter successfully canceled tick 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fea3403a8c0 msgr2=0x7fea3403cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fea3403a8c0 0x7fea3403cd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fea40006fd0 tx=0x7fea40006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 msgr2=0x7fea4c07bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c07bb70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea44009680 tx=0x7fea440049c0 comp rx=0 tx=0).stop 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 shutdown_connections 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fea3403a8c0 0x7fea3403cd80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 --2- 172.21.15.69:0/3706018868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea4c106710 0x7fea4c07bb70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 >> 172.21.15.69:0/3706018868 conn(0x7fea4c101cf0 msgr2=0x7fea4c102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 shutdown_connections 2024-04-02T04:34:37.466 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 -- 172.21.15.69:0/3706018868 wait complete. 2024-04-02T04:34:37.467 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:36.885+0000 7fea51b55700 1 librados: shutdown 2024-04-02T04:34:37.467 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling the dashboard module... 2024-04-02T04:34:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:37 smithi069 bash[21528]: audit 2024-04-02T04:34:36.886200+0000 mon.a (mon.0) 69 : audit 1 from='client.? 172.21.15.69:0/3706018868' entity='client.admin' 2024-04-02T04:34:39.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:38 smithi069 bash[21528]: cluster 2024-04-02T04:34:37.868334+0000 mgr.y (mgr.14120) 22 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:34:39.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:38 smithi069 bash[21528]: audit 2024-04-02T04:34:38.678373+0000 mon.a (mon.0) 70 : audit 1 from='client.? 172.21.15.69:0/1297814276' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-02T04:34:39.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:38 smithi069 bash[21782]: ignoring --setuser ceph since I am not root 2024-04-02T04:34:39.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:38 smithi069 bash[21782]: ignoring --setgroup ceph since I am not root 2024-04-02T04:34:39.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:39 smithi069 bash[21782]: debug 2024-04-02T04:34:39.100+0000 7feb9f84d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T04:34:39.541 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40fd7e700 1 Processor -- start 2024-04-02T04:34:39.541 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40fd7e700 1 -- start start 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40fd7e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb408108d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40fd7e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb408109310 con 0x7fb408108920 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40db1a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb408108d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40db1a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb408108d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39368/0 (socket says 172.21.15.69:39368) 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40db1a700 1 -- 172.21.15.69:0/2560002407 learned_addr learned my addr 172.21.15.69:0/2560002407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.200+0000 7fb40db1a700 1 -- 172.21.15.69:0/2560002407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb408109b20 con 0x7fb408108920 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40db1a700 1 --2- 172.21.15.69:0/2560002407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb408108d40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80098a0 tx=0x7fb3f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf8c66bcd76646e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40cb18700 1 -- 172.21.15.69:0/2560002407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3f8004160 con 0x7fb408108920 2024-04-02T04:34:39.542 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40cb18700 1 -- 172.21.15.69:0/2560002407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb3f8004300 con 0x7fb408108920 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40cb18700 1 -- 172.21.15.69:0/2560002407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3f8011930 con 0x7fb408108920 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/2560002407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 msgr2=0x7fb408108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 --2- 172.21.15.69:0/2560002407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb408108d40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80098a0 tx=0x7fb3f8009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/2560002407 shutdown_connections 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 --2- 172.21.15.69:0/2560002407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb408108d40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/2560002407 >> 172.21.15.69:0/2560002407 conn(0x7fb40807be20 msgr2=0x7fb40807c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/2560002407 shutdown_connections 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/2560002407 wait complete. 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 librados: starting msgr at 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 librados: starting objecter 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 Processor -- start 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- start start 2024-04-02T04:34:39.543 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 librados: setting wanted keys 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 librados: calling monclient init 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb40819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810cd50 con 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40db1a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb40819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40db1a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb40819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39378/0 (socket says 172.21.15.69:39378) 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40db1a700 1 -- 172.21.15.69:0/1297814276 learned_addr learned my addr 172.21.15.69:0/1297814276 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40db1a700 1 -- 172.21.15.69:0/1297814276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819cca0 con 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40db1a700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb40819c760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8003fa0 tx=0x7fb3f8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb406ffd700 10 client.?.objecter ms_handle_connect 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb406ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3f80036a0 con 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb3f801a430 con 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819cf30 con 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3f80193c0 con 0x7fb408108920 2024-04-02T04:34:39.544 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40819fc20 con 0x7fb408108920 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 librados: init done 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fb3f80198d0 con 0x7fb408108920 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.204+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40810ced0 con 0x7fb408108920 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fb3f003a870 0x7fb3f003cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb3f804d3c0 con 0x7fb408108920 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 10 client.14150.objecter ms_dispatch 0x7fb408001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 20 client.14150.objecter dump_active .. 0 homeless 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb40d319700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fb3f003a870 0x7fb3f003cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:39.545 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb406ffd700 10 client.14150.objecter ms_handle_connect 0x7fb3f003a870 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.208+0000 7fb40d319700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fb3f003a870 0x7fb3f003cd30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00ad30 tx=0x7fb3fc0093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.212+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb3f800f2e0 con 0x7fb408108920 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.672+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb40819ffd0 con 0x7fb408108920 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.892+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fb3f801a650 con 0x7fb408108920 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.892+0000 7fb406ffd700 1 -- 172.21.15.69:0/1297814276 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7fb3f801a930 con 0x7fb408108920 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 10 librados: watch_flush enter 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 10 librados: watch_flush exit 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 20 client.14150.objecter shutdown clearing up homeless session... 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 10 client.14150.objecter successfully canceled tick 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fb3f003a870 msgr2=0x7fb3f003cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fb3f003a870 0x7fb3f003cd30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00ad30 tx=0x7fb3fc0093f0 comp rx=0 tx=0).stop 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 msgr2=0x7fb40819c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:39.546 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb40819c760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8003fa0 tx=0x7fb3f8011f30 comp rx=0 tx=0).stop 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 shutdown_connections 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7fb3f003a870 0x7fb3f003cd30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 --2- 172.21.15.69:0/1297814276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408108920 0x7fb40819c760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 >> 172.21.15.69:0/1297814276 conn(0x7fb40807be20 msgr2=0x7fb408105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 shutdown_connections 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 -- 172.21.15.69:0/1297814276 wait complete. 2024-04-02T04:34:39.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:38.896+0000 7fb40fd7e700 1 librados: shutdown 2024-04-02T04:34:40.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:39 smithi069 bash[21528]: audit 2024-04-02T04:34:38.896045+0000 mon.a (mon.0) 71 : audit 1 from='client.? 172.21.15.69:0/1297814276' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-02T04:34:40.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:39 smithi069 bash[21528]: cluster 2024-04-02T04:34:38.900904+0000 mon.a (mon.0) 72 : cluster 0 mgrmap e9: y(active, since 25s) 2024-04-02T04:34:40.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:40 smithi069 bash[21528]: audit 2024-04-02T04:34:40.796740+0000 mon.a (mon.0) 73 : audit 0 from='client.? 172.21.15.69:0/3175170683' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-02T04:34:40.993 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:40 smithi069 bash[21782]: debug 2024-04-02T04:34:40.720+0000 7feb9f84d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T04:34:40.993 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:40 smithi069 bash[21782]: debug 2024-04-02T04:34:40.820+0000 7feb9f84d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T04:34:41.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:41 smithi069 bash[21782]: debug 2024-04-02T04:34:41.072+0000 7feb9f84d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d197a8700 1 Processor -- start 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d197a8700 1 -- start start 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d197a8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d14106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:41.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d197a8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d14107110 con 0x7f9d14106720 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d12ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d14106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d12ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d14106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39410/0 (socket says 172.21.15.69:39410) 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d12ffd700 1 -- 172.21.15.69:0/710678458 learned_addr learned my addr 172.21.15.69:0/710678458 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d12ffd700 1 -- 172.21.15.69:0/710678458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d14107920 con 0x7f9d14106720 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d12ffd700 1 --2- 172.21.15.69:0/710678458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d14106b40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9d080098a0 tx=0x7f9d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6f45198687b002 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d11ffb700 1 -- 172.21.15.69:0/710678458 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d08004160 con 0x7f9d14106720 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d11ffb700 1 -- 172.21.15.69:0/710678458 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9d08004300 con 0x7f9d14106720 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d11ffb700 1 -- 172.21.15.69:0/710678458 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d08011930 con 0x7f9d14106720 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d197a8700 1 -- 172.21.15.69:0/710678458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 msgr2=0x7f9d14106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.304+0000 7f9d197a8700 1 --2- 172.21.15.69:0/710678458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d14106b40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9d080098a0 tx=0x7f9d08009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/710678458 shutdown_connections 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 --2- 172.21.15.69:0/710678458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d14106b40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/710678458 >> 172.21.15.69:0/710678458 conn(0x7f9d14101cc0 msgr2=0x7f9d14104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/710678458 shutdown_connections 2024-04-02T04:34:41.371 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/710678458 wait complete. 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 librados: starting msgr at 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 librados: starting objecter 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 Processor -- start 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- start start 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 librados: setting wanted keys 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 librados: calling monclient init 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d1407c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1410ab50 con 0x7f9d14106720 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d12ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d1407c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d12ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d1407c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39420/0 (socket says 172.21.15.69:39420) 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d12ffd700 1 -- 172.21.15.69:0/3175170683 learned_addr learned my addr 172.21.15.69:0/3175170683 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d12ffd700 1 -- 172.21.15.69:0/3175170683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1407c940 con 0x7f9d14106720 2024-04-02T04:34:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d12ffd700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d1407c400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9d08003fa0 tx=0x7f9d08011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 10 client.?.objecter ms_handle_connect 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d080036a0 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9d0801a430 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1407ab30 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d080193c0 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1407b070 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 librados: init done 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7f9d080196c0 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 0x7f9cfc03cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9d0804e0e0 con 0x7f9d14106720 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 10 client.14154.objecter ms_dispatch 0x7f9d14001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:41.373 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 3 client.14154.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 3 client.14154.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9cfbfff700 20 client.14154.objecter dump_active .. 0 homeless 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.308+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1410acd0 con 0x7f9d14106720 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.316+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9d08020240 con 0x7f9d14106720 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.324+0000 7f9d127fc700 1 -- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 msgr2=0x7f9cfc03cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.324+0000 7f9d127fc700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 0x7f9cfc03cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.540+0000 7f9d127fc700 1 -- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 msgr2=0x7f9cfc03cd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.540+0000 7f9d127fc700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 0x7f9cfc03cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f9d14063870 con 0x7f9d14106720 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9cfbfff700 1 -- 172.21.15.69:0/3175170683 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f9d0800bc70 con 0x7f9d14106720 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 10 librados: watch_flush enter 2024-04-02T04:34:41.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 10 librados: watch_flush exit 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 20 client.14154.objecter shutdown clearing up homeless session... 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 10 client.14154.objecter successfully canceled tick 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 msgr2=0x7f9cfc03cd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 0x7f9cfc03cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 msgr2=0x7f9d1407c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.792+0000 7f9d197a8700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d1407c400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9d08003fa0 tx=0x7f9d08011f30 comp rx=0 tx=0).stop 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 shutdown_connections 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f9cfc03a8c0 0x7f9cfc03cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 --2- 172.21.15.69:0/3175170683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14106720 0x7f9d1407c400 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 >> 172.21.15.69:0/3175170683 conn(0x7f9d14101cc0 msgr2=0x7f9d14102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 shutdown_connections 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 -- 172.21.15.69:0/3175170683 wait complete. 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:40.796+0000 7f9d197a8700 1 librados: shutdown 2024-04-02T04:34:41.375 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for the mgr to restart... 2024-04-02T04:34:41.376 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr epoch 9... 2024-04-02T04:34:42.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:41 smithi069 bash[21782]: debug 2024-04-02T04:34:41.872+0000 7feb9f84d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T04:34:42.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:41 smithi069 bash[21782]: debug 2024-04-02T04:34:41.988+0000 7feb9f84d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T04:34:42.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:42 smithi069 bash[21782]: debug 2024-04-02T04:34:42.092+0000 7feb9f84d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T04:34:42.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:42 smithi069 bash[21782]: debug 2024-04-02T04:34:42.448+0000 7feb9f84d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T04:34:42.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:42 smithi069 bash[21782]: debug 2024-04-02T04:34:42.564+0000 7feb9f84d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T04:34:42.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:42 smithi069 bash[21782]: debug 2024-04-02T04:34:42.668+0000 7feb9f84d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T04:34:43.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:43 smithi069 bash[21782]: debug 2024-04-02T04:34:43.024+0000 7feb9f84d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T04:34:43.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:43 smithi069 bash[21782]: debug 2024-04-02T04:34:43.264+0000 7feb9f84d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T04:34:43.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:43 smithi069 bash[21782]: debug 2024-04-02T04:34:43.524+0000 7feb9f84d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T04:34:43.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:43 smithi069 bash[21782]: debug 2024-04-02T04:34:43.636+0000 7feb9f84d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T04:34:44.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:43 smithi069 bash[21782]: debug 2024-04-02T04:34:43.908+0000 7feb9f84d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T04:34:44.685 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:44 smithi069 bash[21782]: debug 2024-04-02T04:34:44.320+0000 7feb9f84d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:34:44.686 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:44 smithi069 bash[21782]: debug 2024-04-02T04:34:44.560+0000 7feb9f84d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T04:34:45.034 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:44 smithi069 bash[21782]: debug 2024-04-02T04:34:44.676+0000 7feb9f84d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T04:34:45.034 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:44 smithi069 bash[21782]: debug 2024-04-02T04:34:44.924+0000 7feb9f84d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:34:46.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:45 smithi069 bash[21782]: debug 2024-04-02T04:34:45.920+0000 7feb9f84d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T04:34:46.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:46 smithi069 bash[21782]: debug 2024-04-02T04:34:46.036+0000 7feb9f84d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T04:34:46.784 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:46 smithi069 bash[21782]: debug 2024-04-02T04:34:46.332+0000 7feb9f84d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T04:34:46.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:46 smithi069 bash[21782]: debug 2024-04-02T04:34:46.444+0000 7feb9f84d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T04:34:47.150 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:46 smithi069 bash[21782]: debug 2024-04-02T04:34:46.888+0000 7feb9f84d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21782]: debug 2024-04-02T04:34:47.008+0000 7feb9f84d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: cluster 2024-04-02T04:34:47.037749+0000 mon.a (mon.0) 74 : cluster 1 Active manager daemon y restarted 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: cluster 2024-04-02T04:34:47.037994+0000 mon.a (mon.0) 75 : cluster 1 Activating manager daemon y 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: cluster 2024-04-02T04:34:47.044409+0000 mon.a (mon.0) 76 : cluster 0 osdmap e3: 0 total, 0 up, 0 in 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: cluster 2024-04-02T04:34:47.044555+0000 mon.a (mon.0) 77 : cluster 0 mgrmap e10: y(active, starting, since 0.00667351s) 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: audit 2024-04-02T04:34:47.046885+0000 mon.a (mon.0) 78 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: audit 2024-04-02T04:34:47.048427+0000 mon.a (mon.0) 79 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: audit 2024-04-02T04:34:47.049513+0000 mon.a (mon.0) 80 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: audit 2024-04-02T04:34:47.049696+0000 mon.a (mon.0) 81 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:34:47.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: audit 2024-04-02T04:34:47.049850+0000 mon.a (mon.0) 82 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:34:47.152 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:47 smithi069 bash[21528]: cluster 2024-04-02T04:34:47.078548+0000 mon.a (mon.0) 83 : cluster 1 Manager daemon y is now available 2024-04-02T04:34:48.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:48 smithi069 bash[21528]: audit 2024-04-02T04:34:47.103985+0000 mon.a (mon.0) 84 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:34:48.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:48 smithi069 bash[21528]: audit 2024-04-02T04:34:47.122236+0000 mon.a (mon.0) 85 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:34:48.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:48 smithi069 bash[21528]: audit 2024-04-02T04:34:47.125380+0000 mon.a (mon.0) 86 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:34:48.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:48 smithi069 bash[21528]: cluster 2024-04-02T04:34:48.050441+0000 mon.a (mon.0) 87 : cluster 0 mgrmap e11: y(active, since 1.01255s) 2024-04-02T04:34:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-04-02T04:34:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-02T04:34:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-02T04:34:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-04-02T04:34:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19d568b700 1 Processor -- start 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19d568b700 1 -- start start 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19d568b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d0106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19d568b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f19d0107110 con 0x7f19d0106720 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19ceffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d0106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19ceffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d0106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39428/0 (socket says 172.21.15.69:39428) 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19ceffd700 1 -- 172.21.15.69:0/577573728 learned_addr learned my addr 172.21.15.69:0/577573728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19ceffd700 1 -- 172.21.15.69:0/577573728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d0107920 con 0x7f19d0106720 2024-04-02T04:34:48.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19ceffd700 1 --2- 172.21.15.69:0/577573728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d0106b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f19c40098a0 tx=0x7f19c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4353dd4360a10e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19cdffb700 1 -- 172.21.15.69:0/577573728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c4004160 con 0x7f19d0106720 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19cdffb700 1 -- 172.21.15.69:0/577573728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f19c4004300 con 0x7f19d0106720 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.140+0000 7f19cdffb700 1 -- 172.21.15.69:0/577573728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c4011930 con 0x7f19d0106720 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/577573728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 msgr2=0x7f19d0106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 --2- 172.21.15.69:0/577573728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d0106b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f19c40098a0 tx=0x7f19c4009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/577573728 shutdown_connections 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 --2- 172.21.15.69:0/577573728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d0106b40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/577573728 >> 172.21.15.69:0/577573728 conn(0x7f19d0101ca0 msgr2=0x7f19d0104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/577573728 shutdown_connections 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/577573728 wait complete. 2024-04-02T04:34:48.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 librados: starting msgr at 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 librados: starting objecter 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 Processor -- start 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- start start 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 librados: setting wanted keys 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 librados: calling monclient init 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d007c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f19d010ab50 con 0x7f19d0106720 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ceffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d007c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ceffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d007c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39434/0 (socket says 172.21.15.69:39434) 2024-04-02T04:34:48.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ceffd700 1 -- 172.21.15.69:0/651105353 learned_addr learned my addr 172.21.15.69:0/651105353 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ceffd700 1 -- 172.21.15.69:0/651105353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d007c940 con 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ceffd700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d007c400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f19c4006af0 tx=0x7f19c4011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 10 client.?.objecter ms_handle_connect 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c40039e0 con 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d007ab30 con 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f19c4003b80 con 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c4019620 con 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d007b070 con 0x7f19d0106720 2024-04-02T04:34:48.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19d568b700 1 librados: init done 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7f19c4019940 con 0x7f19d0106720 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f19c4052580 con 0x7f19d0106720 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 10 client.14156.objecter ms_dispatch 0x7f19d0001550 osd_map(2..2 src has 1..2) 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 3 client.14156.objecter handle_osd_map got epochs [2,2] > 0 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 3 client.14156.objecter handle_osd_map decoding full epoch 2 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ce7fc700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 msgr2=0x7f19b803c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.144+0000 7f19b7fff700 20 client.14156.objecter dump_active .. 0 homeless 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.148+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 --> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f19d0063870 con 0x7f19b803a460 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.348+0000 7f19ce7fc700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 msgr2=0x7f19b803c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:48.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.348+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.752+0000 7f19ce7fc700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 msgr2=0x7f19b803c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:42.752+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:43.556+0000 7f19ce7fc700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 msgr2=0x7f19b803c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:43.556+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:45.160+0000 7f19ce7fc700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 msgr2=0x7f19b803c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/578732137 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:45.160+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:47.036+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mgrmap(e 10) ==== 45896+0+0 (secure 0 0 0) 0x7f19c400bc80 con 0x7f19d0106720 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:47.036+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 msgr2=0x7f19b803c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:34:48.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:47.036+0000 7f19b7fff700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:47.148+0000 7f19d4e8a700 10 client.14156.objecter tick 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.040+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 11) ==== 46023+0+0 (secure 0 0 0) 0x7f19c4042860 con 0x7f19d0106720 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.040+0000 7f19b7fff700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19b803de40 0x7f19b8040230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.040+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f19c4022e70 con 0x7f19b803de40 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.040+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19b803de40 0x7f19b8040230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.040+0000 7f19b7fff700 10 client.14156.objecter ms_handle_connect 0x7f19b803de40 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.040+0000 7f19ce7fc700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19b803de40 0x7f19b8040230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f19c0003a10 tx=0x7f19c0009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.044+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f19c4022e70 con 0x7f19b803de40 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f19d010acd0 con 0x7f19b803de40 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19b7fff700 1 -- 172.21.15.69:0/651105353 <== mgr.14152 v2:172.21.15.69:6800/1560062353 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f19d010acd0 con 0x7f19b803de40 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 10 librados: watch_flush enter 2024-04-02T04:34:48.711 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 10 librados: watch_flush exit 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 20 client.14156.objecter shutdown clearing up homeless session... 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 10 client.14156.objecter successfully canceled tick 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19b803de40 msgr2=0x7f19b8040230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19b803de40 0x7f19b8040230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f19c0003a10 tx=0x7f19c0009270 comp rx=0 tx=0).stop 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 msgr2=0x7f19d007c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d007c400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f19c4006af0 tx=0x7f19c4011af0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 shutdown_connections 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d0106720 0x7f19d007c400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19b803de40 0x7f19b8040230 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 --2- 172.21.15.69:0/651105353 >> [v2:172.21.15.69:6800/578732137,v1:172.21.15.69:6801/578732137] conn(0x7f19b803a460 0x7f19b803c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 >> 172.21.15.69:0/651105353 conn(0x7f19d0101ca0 msgr2=0x7f19d0102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:48.712 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 shutdown_connections 2024-04-02T04:34:48.713 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 -- 172.21.15.69:0/651105353 wait complete. 2024-04-02T04:34:48.713 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:48.056+0000 7f19d568b700 1 librados: shutdown 2024-04-02T04:34:48.713 INFO:teuthology.orchestra.run.smithi069.stdout:mgr epoch 9 is available 2024-04-02T04:34:48.713 INFO:teuthology.orchestra.run.smithi069.stdout:Generating a dashboard self-signed certificate... 2024-04-02T04:34:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:49 smithi069 bash[21528]: cephadm 2024-04-02T04:34:48.159838+0000 mgr.y (mgr.14152) 3 : cephadm 1 [02/Apr/2024:04:34:48] ENGINE Bus STARTING 2024-04-02T04:34:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:49 smithi069 bash[21528]: cephadm 2024-04-02T04:34:48.274247+0000 mgr.y (mgr.14152) 4 : cephadm 1 [02/Apr/2024:04:34:48] ENGINE Serving on https://172.21.15.69:7150 2024-04-02T04:34:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:49 smithi069 bash[21528]: cephadm 2024-04-02T04:34:48.375561+0000 mgr.y (mgr.14152) 5 : cephadm 1 [02/Apr/2024:04:34:48] ENGINE Serving on http://172.21.15.69:8765 2024-04-02T04:34:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:49 smithi069 bash[21528]: cephadm 2024-04-02T04:34:48.375729+0000 mgr.y (mgr.14152) 6 : cephadm 1 [02/Apr/2024:04:34:48] ENGINE Bus STARTED 2024-04-02T04:34:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:50 smithi069 bash[21528]: cluster 2024-04-02T04:34:49.097843+0000 mon.a (mon.0) 88 : cluster 0 mgrmap e12: y(active, since 2s) 2024-04-02T04:34:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:50 smithi069 bash[21528]: audit 2024-04-02T04:34:49.884116+0000 mgr.y (mgr.14152) 7 : audit 0 from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:50 smithi069 bash[21528]: audit 2024-04-02T04:34:49.976682+0000 mon.a (mon.0) 89 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:34:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:50 smithi069 bash[21528]: audit 2024-04-02T04:34:49.980931+0000 mon.a (mon.0) 90 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:34:50.591 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-02T04:34:50.591 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 Processor -- start 2024-04-02T04:34:50.591 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- start start 2024-04-02T04:34:50.591 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb11010f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:50.591 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb11010fe90 con 0x7fb11010d560 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb114864700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb11010f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb114864700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb11010f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44354/0 (socket says 172.21.15.69:44354) 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb114864700 1 -- 172.21.15.69:0/3557523164 learned_addr learned my addr 172.21.15.69:0/3557523164 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb114864700 1 -- 172.21.15.69:0/3557523164 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb110110010 con 0x7fb11010d560 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb114864700 1 --2- 172.21.15.69:0/3557523164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb11010f950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb1000098a0 tx=0x7fb100009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78194343a4e2c0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb10f7fe700 1 -- 172.21.15.69:0/3557523164 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb100004160 con 0x7fb11010d560 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb10f7fe700 1 -- 172.21.15.69:0/3557523164 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb100004300 con 0x7fb11010d560 2024-04-02T04:34:50.594 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb10f7fe700 1 -- 172.21.15.69:0/3557523164 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb100011930 con 0x7fb11010d560 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- 172.21.15.69:0/3557523164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 msgr2=0x7fb11010f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 --2- 172.21.15.69:0/3557523164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb11010f950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb1000098a0 tx=0x7fb100009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- 172.21.15.69:0/3557523164 shutdown_connections 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 --2- 172.21.15.69:0/3557523164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb11010f950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- 172.21.15.69:0/3557523164 >> 172.21.15.69:0/3557523164 conn(0x7fb11007bde0 msgr2=0x7fb11007e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- 172.21.15.69:0/3557523164 shutdown_connections 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 -- 172.21.15.69:0/3557523164 wait complete. 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 librados: starting msgr at 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.496+0000 7fb116ac8700 1 librados: starting objecter 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 Processor -- start 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 -- start start 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 librados: setting wanted keys 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 librados: calling monclient init 2024-04-02T04:34:50.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb1101a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:50.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb110111070 con 0x7fb11010d560 2024-04-02T04:34:50.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb114864700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb1101a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:50.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb114864700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb1101a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44358/0 (socket says 172.21.15.69:44358) 2024-04-02T04:34:50.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb114864700 1 -- 172.21.15.69:0/3137106138 learned_addr learned my addr 172.21.15.69:0/3137106138 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb114864700 1 -- 172.21.15.69:0/3137106138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1101a1070 con 0x7fb11010d560 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb114864700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb1101a0b30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb100009870 tx=0x7fb100011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 10 client.?.objecter ms_handle_connect 0x7fb11010d560 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1000039a0 con 0x7fb11010d560 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb100003b40 con 0x7fb11010d560 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb100019640 con 0x7fb11010d560 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1101a1300 con 0x7fb11010d560 2024-04-02T04:34:50.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1101a3ff0 con 0x7fb11010d560 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 librados: init done 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7fb10001a430 con 0x7fb11010d560 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb110063870 con 0x7fb11010d560 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fb0f803a870 0x7fb0f803cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb10004d600 con 0x7fb11010d560 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 10 client.14164.objecter ms_dispatch 0x7fb110001550 osd_map(3..3 src has 1..3) 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 20 client.14164.objecter dump_active .. 0 homeless 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10ffff700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fb0f803a870 0x7fb0f803cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:50.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10ffff700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fb0f803a870 0x7fb0f803cd30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb104006fd0 tx=0x7fb104006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.500+0000 7fb10dffb700 10 client.14164.objecter ms_handle_connect 0x7fb0f803a870 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.508+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb10000fda0 con 0x7fb11010d560 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.876+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fb11007df90 con 0x7fb0f803a870 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.976+0000 7fb10dffb700 1 -- 172.21.15.69:0/3137106138 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fb11007df90 con 0x7fb0f803a870 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 10 librados: watch_flush enter 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 10 librados: watch_flush exit 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 20 client.14164.objecter shutdown clearing up homeless session... 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 10 client.14164.objecter successfully canceled tick 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fb0f803a870 msgr2=0x7fb0f803cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:50.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fb0f803a870 0x7fb0f803cd30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb104006fd0 tx=0x7fb104006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 msgr2=0x7fb1101a0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb1101a0b30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb100009870 tx=0x7fb100011af0 comp rx=0 tx=0).stop 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 shutdown_connections 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fb0f803a870 0x7fb0f803cd30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 --2- 172.21.15.69:0/3137106138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb11010d560 0x7fb1101a0b30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 >> 172.21.15.69:0/3137106138 conn(0x7fb11007bde0 msgr2=0x7fb11007d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 shutdown_connections 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 -- 172.21.15.69:0/3137106138 wait complete. 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:49.980+0000 7fb116ac8700 1 librados: shutdown 2024-04-02T04:34:50.603 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial admin user... 2024-04-02T04:34:52.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:52 smithi069 bash[21528]: audit 2024-04-02T04:34:51.806216+0000 mgr.y (mgr.14152) 8 : audit 0 from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$QPFf4RPlsQdquCzASPRgtefpvLIaNnoUdTi.3xJyt7dm/6G7xLzcG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712032492, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc2495f6700 1 Processor -- start 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc2495f6700 1 -- start start 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc2495f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24407ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc2495f6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc24407b160 con 0x7fc24407c730 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc242ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24407ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:52.547 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc242ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24407ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44370/0 (socket says 172.21.15.69:44370) 2024-04-02T04:34:52.548 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc242ffd700 1 -- 172.21.15.69:0/2773937323 learned_addr learned my addr 172.21.15.69:0/2773937323 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:52.548 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc242ffd700 1 -- 172.21.15.69:0/2773937323 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24407b2e0 con 0x7fc24407c730 2024-04-02T04:34:52.548 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc242ffd700 1 --2- 172.21.15.69:0/2773937323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24407ab90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2340098a0 tx=0x7fc234009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1581a3190def47f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:52.548 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc241ffb700 1 -- 172.21.15.69:0/2773937323 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc234004160 con 0x7fc24407c730 2024-04-02T04:34:52.548 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc241ffb700 1 -- 172.21.15.69:0/2773937323 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc234004300 con 0x7fc24407c730 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.404+0000 7fc241ffb700 1 -- 172.21.15.69:0/2773937323 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc234011930 con 0x7fc24407c730 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/2773937323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 msgr2=0x7fc24407ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 --2- 172.21.15.69:0/2773937323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24407ab90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2340098a0 tx=0x7fc234009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/2773937323 shutdown_connections 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 --2- 172.21.15.69:0/2773937323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24407ab90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/2773937323 >> 172.21.15.69:0/2773937323 conn(0x7fc244101c70 msgr2=0x7fc2441040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:52.554 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/2773937323 shutdown_connections 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/2773937323 wait complete. 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 librados: starting msgr at 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 librados: starting objecter 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 Processor -- start 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- start start 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 librados: setting wanted keys 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 librados: calling monclient init 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc24410cc90 con 0x7fc24407c730 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc242ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc242ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24419c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44384/0 (socket says 172.21.15.69:44384) 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc242ffd700 1 -- 172.21.15.69:0/3065376119 learned_addr learned my addr 172.21.15.69:0/3065376119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc242ffd700 1 -- 172.21.15.69:0/3065376119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24419cca0 con 0x7fc24407c730 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc242ffd700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24419c760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc234003fa0 tx=0x7fc234011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:52.555 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 10 client.?.objecter ms_handle_connect 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2340036a0 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc23401a430 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24419cf30 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2340193c0 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24419fc20 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 librados: init done 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7fc2340196c0 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc22c03a8c0 0x7fc22c03cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc23404d1e0 con 0x7fc24407c730 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 10 client.14166.objecter ms_dispatch 0x7fc244001550 osd_map(3..3 src has 1..3) 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:34:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc22bfff700 20 client.14166.objecter dump_active .. 0 homeless 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2427fc700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc22c03a8c0 0x7fc22c03cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.408+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc244063870 con 0x7fc24407c730 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.412+0000 7fc22bfff700 10 client.14166.objecter ms_handle_connect 0x7fc22c03a8c0 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.412+0000 7fc2427fc700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc22c03a8c0 0x7fc22c03cd80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc238006fd0 tx=0x7fc238006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.416+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc234004300 con 0x7fc24407c730 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:51.800+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc244102de0 con 0x7fc22c03a8c0 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc22bfff700 1 -- 172.21.15.69:0/3065376119 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fc244102de0 con 0x7fc22c03a8c0 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 10 librados: watch_flush enter 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 10 librados: watch_flush exit 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 20 client.14166.objecter shutdown clearing up homeless session... 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 10 client.14166.objecter successfully canceled tick 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc22c03a8c0 msgr2=0x7fc22c03cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc22c03a8c0 0x7fc22c03cd80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc238006fd0 tx=0x7fc238006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:52.557 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 msgr2=0x7fc24419c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24419c760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc234003fa0 tx=0x7fc234011f30 comp rx=0 tx=0).stop 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 shutdown_connections 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc22c03a8c0 0x7fc22c03cd80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 --2- 172.21.15.69:0/3065376119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc24407c730 0x7fc24419c760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 >> 172.21.15.69:0/3065376119 conn(0x7fc244101c70 msgr2=0x7fc2441026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 shutdown_connections 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 -- 172.21.15.69:0/3065376119 wait complete. 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:52.028+0000 7fc2495f6700 1 librados: shutdown 2024-04-02T04:34:52.558 INFO:teuthology.orchestra.run.smithi069.stdout:Fetching dashboard port number... 2024-04-02T04:34:53.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:53 smithi069 bash[21528]: audit 2024-04-02T04:34:52.031794+0000 mon.a (mon.0) 91 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:34:54.283 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 8443 2024-04-02T04:34:54.283 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bbbe2a700 1 Processor -- start 2024-04-02T04:34:54.283 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bbbe2a700 1 -- start start 2024-04-02T04:34:54.283 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bbbe2a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb4106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:54.283 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bbbe2a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb4107110 con 0x7f8bb4106720 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bb9bc6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb4106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bb9bc6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb4106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44386/0 (socket says 172.21.15.69:44386) 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.308+0000 7f8bb9bc6700 1 -- 172.21.15.69:0/10276286 learned_addr learned my addr 172.21.15.69:0/10276286 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 -- 172.21.15.69:0/10276286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb4107920 con 0x7f8bb4106720 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 --2- 172.21.15.69:0/10276286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb4106b40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00098a0 tx=0x7f8bb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d838d077931cd752 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb8bc4700 1 -- 172.21.15.69:0/10276286 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bb0004160 con 0x7f8bb4106720 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb8bc4700 1 -- 172.21.15.69:0/10276286 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8bb0004300 con 0x7f8bb4106720 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb8bc4700 1 -- 172.21.15.69:0/10276286 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bb0011930 con 0x7f8bb4106720 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/10276286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 msgr2=0x7f8bb4106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 --2- 172.21.15.69:0/10276286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb4106b40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00098a0 tx=0x7f8bb0009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/10276286 shutdown_connections 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 --2- 172.21.15.69:0/10276286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb4106b40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/10276286 >> 172.21.15.69:0/10276286 conn(0x7f8bb4101d00 msgr2=0x7f8bb4104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/10276286 shutdown_connections 2024-04-02T04:34:54.284 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/10276286 wait complete. 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 librados: starting msgr at 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 librados: starting objecter 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 Processor -- start 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- start start 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 librados: setting wanted keys 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 librados: calling monclient init 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb410ab50 con 0x7f8bb4106720 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44396/0 (socket says 172.21.15.69:44396) 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 -- 172.21.15.69:0/584879356 learned_addr learned my addr 172.21.15.69:0/584879356 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 -- 172.21.15.69:0/584879356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb419cd00 con 0x7f8bb4106720 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bb9bc6700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb419c7c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0006af0 tx=0x7f8bb0011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8baaffd700 10 client.?.objecter ms_handle_connect 0x7f8bb4106720 2024-04-02T04:34:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8baaffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bb00039e0 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb419cf90 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8bb0003b80 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bb0019620 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb419fc80 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.312+0000 7f8bbbe2a700 1 librados: init done 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7f8bb0019920 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ba003a870 0x7f8ba003cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8bb001a8b0 con 0x7f8bb4106720 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 10 client.14168.objecter ms_dispatch 0x7f8bb4001550 osd_map(3..3 src has 1..3) 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 20 client.14168.objecter dump_active .. 0 homeless 2024-04-02T04:34:54.286 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8bb93c5700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ba003a870 0x7f8ba003cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b98005120 con 0x7f8bb4106720 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8bb93c5700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ba003a870 0x7f8ba003cd30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4006fd0 tx=0x7f8ba4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.316+0000 7f8baaffd700 10 client.14168.objecter ms_handle_connect 0x7f8ba003a870 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.320+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8bb0029260 con 0x7f8bb4106720 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.672+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f8b98005410 con 0x7f8bb4106720 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.672+0000 7f8baaffd700 1 -- 172.21.15.69:0/584879356 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f8bb001ab80 con 0x7f8bb4106720 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 10 librados: watch_flush enter 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 10 librados: watch_flush exit 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 20 client.14168.objecter shutdown clearing up homeless session... 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 10 client.14168.objecter successfully canceled tick 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ba003a870 msgr2=0x7f8ba003cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ba003a870 0x7f8ba003cd30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4006fd0 tx=0x7f8ba4006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 msgr2=0x7f8bb419c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb419c7c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0006af0 tx=0x7f8bb0011af0 comp rx=0 tx=0).stop 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 shutdown_connections 2024-04-02T04:34:54.287 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ba003a870 0x7f8ba003cd30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 --2- 172.21.15.69:0/584879356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb4106720 0x7f8bb419c7c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 >> 172.21.15.69:0/584879356 conn(0x7f8bb4101d00 msgr2=0x7f8bb41026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 shutdown_connections 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 -- 172.21.15.69:0/584879356 wait complete. 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:53.676+0000 7f8bbbe2a700 1 librados: shutdown 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld does not appear to be present 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-02T04:34:54.288 INFO:teuthology.orchestra.run.smithi069.stdout:Ceph Dashboard is now available at: 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: URL: https://smithi069.front.sepia.ceph.com:8443/ 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: User: admin 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: Password: ik2763rfr9 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout:Saving cluster configuration to /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config directory 2024-04-02T04:34:54.289 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling autotune for osd_memory_target 2024-04-02T04:34:54.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:54 smithi069 bash[21528]: audit 2024-04-02T04:34:53.678861+0000 mon.a (mon.0) 92 : audit 0 from='client.? 172.21.15.69:0/584879356' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-02T04:34:55.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:55 smithi069 bash[21528]: cluster 2024-04-02T04:34:54.037170+0000 mon.a (mon.0) 93 : cluster 0 mgrmap e13: y(active, since 6s) 2024-04-02T04:34:55.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:55 smithi069 bash[21528]: audit 2024-04-02T04:34:54.963761+0000 mon.a (mon.0) 94 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:34:56.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 Processor -- start 2024-04-02T04:34:56.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- start start 2024-04-02T04:34:56.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c109600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:56.045 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f926c07a6c0 con 0x7f926c107210 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44402/0 (socket says 172.21.15.69:44402) 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 -- 172.21.15.69:0/2359656243 learned_addr learned my addr 172.21.15.69:0/2359656243 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 -- 172.21.15.69:0/2359656243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c109b40 con 0x7f926c107210 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 --2- 172.21.15.69:0/2359656243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c109600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f92680098a0 tx=0x7f9268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f31a70decd7dc4a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f9263fff700 1 -- 172.21.15.69:0/2359656243 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9268004160 con 0x7f926c107210 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f9263fff700 1 -- 172.21.15.69:0/2359656243 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9268004300 con 0x7f926c107210 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f9263fff700 1 -- 172.21.15.69:0/2359656243 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9268011930 con 0x7f926c107210 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- 172.21.15.69:0/2359656243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 msgr2=0x7f926c109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 --2- 172.21.15.69:0/2359656243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c109600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f92680098a0 tx=0x7f9268009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- 172.21.15.69:0/2359656243 shutdown_connections 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 --2- 172.21.15.69:0/2359656243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c109600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:56.046 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- 172.21.15.69:0/2359656243 >> 172.21.15.69:0/2359656243 conn(0x7f926c100bc0 msgr2=0x7f926c103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- 172.21.15.69:0/2359656243 shutdown_connections 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- 172.21.15.69:0/2359656243 wait complete. 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 librados: starting msgr at 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 librados: starting objecter 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 Processor -- start 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- start start 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 librados: setting wanted keys 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 librados: calling monclient init 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f927372d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f926c10aa80 con 0x7f926c107210 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c19a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44416/0 (socket says 172.21.15.69:44416) 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 -- 172.21.15.69:0/917956470 learned_addr learned my addr 172.21.15.69:0/917956470 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.080+0000 7f92714c9700 1 -- 172.21.15.69:0/917956470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c19aa80 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92714c9700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c19a540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9268006af0 tx=0x7f9268011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 10 client.?.objecter ms_handle_connect 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92680036a0 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9268024430 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f926801a5c0 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c19ad10 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f927372d700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c19b130 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f927372d700 1 librados: init done 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f9268003840 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926c063870 con 0x7f926c107210 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925803a9b0 0x7f925803ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:56.048 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f926804d2a0 con 0x7f926c107210 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 10 client.14170.objecter ms_dispatch 0x7f926c001550 osd_map(3..3 src has 1..3) 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 20 client.14170.objecter dump_active .. 0 homeless 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f9270cc8700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925803a9b0 0x7f925803ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f9270cc8700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925803a9b0 0x7f925803ce70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f925c006fd0 tx=0x7f925c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.084+0000 7f92627fc700 10 client.14170.objecter ms_handle_connect 0x7f925803a9b0 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.088+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f926800f2e0 con 0x7f926c107210 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.460+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f926c19dac0 con 0x7f926c107210 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.460+0000 7f92627fc700 1 -- 172.21.15.69:0/917956470 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f9268004300 con 0x7f926c107210 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 10 librados: watch_flush enter 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 10 librados: watch_flush exit 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 20 client.14170.objecter shutdown clearing up homeless session... 2024-04-02T04:34:56.049 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 10 client.14170.objecter successfully canceled tick 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925803a9b0 msgr2=0x7f925803ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925803a9b0 0x7f925803ce70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f925c006fd0 tx=0x7f925c006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 msgr2=0x7f926c19a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c19a540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9268006af0 tx=0x7f9268011ec0 comp rx=0 tx=0).stop 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 shutdown_connections 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925803a9b0 0x7f925803ce70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 --2- 172.21.15.69:0/917956470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c107210 0x7f926c19a540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 >> 172.21.15.69:0/917956470 conn(0x7f926c100bc0 msgr2=0x7f926c101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 shutdown_connections 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 -- 172.21.15.69:0/917956470 wait complete. 2024-04-02T04:34:56.050 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:55.464+0000 7f927372d700 1 librados: shutdown 2024-04-02T04:34:57.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:56 smithi069 bash[21528]: audit 2024-04-02T04:34:55.962663+0000 mon.a (mon.0) 95 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:34:57.937 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 Processor -- start 2024-04-02T04:34:57.938 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- start start 2024-04-02T04:34:57.938 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c106b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:57.938 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c107110 con 0x7f0f7c106720 2024-04-02T04:34:57.938 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:57.939 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44428/0 (socket says 172.21.15.69:44428) 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 -- 172.21.15.69:0/1857357610 learned_addr learned my addr 172.21.15.69:0/1857357610 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 -- 172.21.15.69:0/1857357610 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c107920 con 0x7f0f7c106720 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 --2- 172.21.15.69:0/1857357610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c106b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0098a0 tx=0x7f0f6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110c2d7a28cae667 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7affd700 1 -- 172.21.15.69:0/1857357610 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f6c004160 con 0x7f0f7c106720 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7affd700 1 -- 172.21.15.69:0/1857357610 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0f6c004300 con 0x7f0f7c106720 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7affd700 1 -- 172.21.15.69:0/1857357610 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f6c011930 con 0x7f0f7c106720 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/1857357610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 msgr2=0x7f0f7c106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 --2- 172.21.15.69:0/1857357610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c106b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0098a0 tx=0x7f0f6c009bb0 comp rx=0 tx=0).stop 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/1857357610 shutdown_connections 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 --2- 172.21.15.69:0/1857357610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c106b40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/1857357610 >> 172.21.15.69:0/1857357610 conn(0x7f0f7c101d00 msgr2=0x7f0f7c104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/1857357610 shutdown_connections 2024-04-02T04:34:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/1857357610 wait complete. 2024-04-02T04:34:57.946 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 librados: starting msgr at 2024-04-02T04:34:57.946 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 librados: starting objecter 2024-04-02T04:34:57.946 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 Processor -- start 2024-04-02T04:34:57.946 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- start start 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 librados: setting wanted keys 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 librados: calling monclient init 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c07c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10ab50 con 0x7f0f7c106720 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c07c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c07c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44436/0 (socket says 172.21.15.69:44436) 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 -- 172.21.15.69:0/3821218762 learned_addr learned my addr 172.21.15.69:0/3821218762 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 -- 172.21.15.69:0/3821218762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c07c940 con 0x7f0f7c106720 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f7bfff700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c07c400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c003fa0 tx=0x7f0f6c011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f797fa700 10 client.?.objecter ms_handle_connect 0x7f0f7c106720 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f797fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f6c0036a0 con 0x7f0f7c106720 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0f6c01a430 con 0x7f0f7c106720 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f6c0193c0 con 0x7f0f7c106720 2024-04-02T04:34:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c07ab30 con 0x7f0f7c106720 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7c07b070 con 0x7f0f7c106720 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.784+0000 7f0f822e1700 1 librados: init done 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f0f6c003840 con 0x7f0f7c106720 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7c10acd0 con 0x7f0f7c106720 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f0f6403a9b0 0x7f0f6403ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0f6c04d370 con 0x7f0f7c106720 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 10 client.14172.objecter ms_dispatch 0x7f0f7c001550 osd_map(3..3 src has 1..3) 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 20 client.14172.objecter dump_active .. 0 homeless 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f7b7fe700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f0f6403a9b0 0x7f0f6403ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f797fa700 10 client.14172.objecter ms_handle_connect 0x7f0f6403a9b0 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.788+0000 7f0f7b7fe700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f0f6403a9b0 0x7f0f6403ce70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0f70006fd0 tx=0x7f0f70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:34:57.948 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:56.792+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0f6c018350 con 0x7f0f7c106720 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.360+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f0f7c063870 con 0x7f0f7c106720 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f797fa700 1 -- 172.21.15.69:0/3821218762 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f0f6c029260 con 0x7f0f7c106720 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 10 librados: watch_flush enter 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 10 librados: watch_flush exit 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 20 client.14172.objecter shutdown clearing up homeless session... 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 10 client.14172.objecter successfully canceled tick 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f0f6403a9b0 msgr2=0x7f0f6403ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f0f6403a9b0 0x7f0f6403ce70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0f70006fd0 tx=0x7f0f70006e40 comp rx=0 tx=0).stop 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 msgr2=0x7f0f7c07c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c07c400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c003fa0 tx=0x7f0f6c011f30 comp rx=0 tx=0).stop 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 shutdown_connections 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f0f6403a9b0 0x7f0f6403ce70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 --2- 172.21.15.69:0/3821218762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f7c106720 0x7f0f7c07c400 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 >> 172.21.15.69:0/3821218762 conn(0x7f0f7c101d00 msgr2=0x7f0f7c102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:34:57.949 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 shutdown_connections 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 -- 172.21.15.69:0/3821218762 wait complete. 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-04-02T04:34:57.364+0000 7f0f822e1700 1 librados: shutdown 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:Or, if you are only running a single cluster on this host: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.950 INFO:teuthology.orchestra.run.smithi069.stdout: ceph telemetry on 2024-04-02T04:34:57.951 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.951 INFO:teuthology.orchestra.run.smithi069.stdout:For more information see: 2024-04-02T04:34:57.951 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.951 INFO:teuthology.orchestra.run.smithi069.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-02T04:34:57.951 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:34:57.951 INFO:teuthology.orchestra.run.smithi069.stdout:Bootstrap complete. 2024-04-02T04:34:58.004 INFO:tasks.cephadm:Fetching config... 2024-04-02T04:34:58.004 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:34:58.005 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-02T04:34:58.055 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-02T04:34:58.055 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:34:58.055 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-02T04:34:58.102 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-02T04:34:58.102 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:34:58.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-02T04:34:58.158 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-02T04:34:58.158 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:34:58.158 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-02T04:34:58.206 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-02T04:34:58.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS0flvN6atN0+P7osaPJSeC7Dj2GKtRjRWoWp7YtcsE7NUNhPIwFTUDq8PVfDQ3K1s0Vrbfuq3R2frgdSxFCigKVbH3YYHMz7SxmW6Z43q16O4UDO1rw5JwxfsuKZ6dHMffs5Hnzh409sLtpwix7Q0tnu4D8exlvUOEARyl6ioEp/pE2X4V2eoPfkNbFz4un4BoxImMSBfQTj4dq1sv7MEZI1Rb1H9XFVYm9fOUMo4hx+V3dCcAGxrFmRtHnrSA5VUCAAH9CUu/9gIazCEZ2vE4hA++X/f0BEo7WvmpNDVaIDfwE4V2ruc2cF3deyetW2Wzdi/KRvK4S8zEhmAJSO9nejSJqNZiqtWCDfsouBJz72e+aSVm4bsJ591AlbE/93F9SB6UikhneKQ0Y/IeOHlfzq/mk8MgXC/3vyaVmCByVzEP+syKGpWmX/VvsVHezupBorVY0yeX5fdbFA9P3UyazhWEqaHcq1+RyJWczM6EOiPct+LsZjXTRI5F50D2kc= ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T04:34:58.274 INFO:teuthology.orchestra.run.smithi069.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS0flvN6atN0+P7osaPJSeC7Dj2GKtRjRWoWp7YtcsE7NUNhPIwFTUDq8PVfDQ3K1s0Vrbfuq3R2frgdSxFCigKVbH3YYHMz7SxmW6Z43q16O4UDO1rw5JwxfsuKZ6dHMffs5Hnzh409sLtpwix7Q0tnu4D8exlvUOEARyl6ioEp/pE2X4V2eoPfkNbFz4un4BoxImMSBfQTj4dq1sv7MEZI1Rb1H9XFVYm9fOUMo4hx+V3dCcAGxrFmRtHnrSA5VUCAAH9CUu/9gIazCEZ2vE4hA++X/f0BEo7WvmpNDVaIDfwE4V2ruc2cF3deyetW2Wzdi/KRvK4S8zEhmAJSO9nejSJqNZiqtWCDfsouBJz72e+aSVm4bsJ591AlbE/93F9SB6UikhneKQ0Y/IeOHlfzq/mk8MgXC/3vyaVmCByVzEP+syKGpWmX/VvsVHezupBorVY0yeX5fdbFA9P3UyazhWEqaHcq1+RyJWczM6EOiPct+LsZjXTRI5F50D2kc= ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:34:58.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS0flvN6atN0+P7osaPJSeC7Dj2GKtRjRWoWp7YtcsE7NUNhPIwFTUDq8PVfDQ3K1s0Vrbfuq3R2frgdSxFCigKVbH3YYHMz7SxmW6Z43q16O4UDO1rw5JwxfsuKZ6dHMffs5Hnzh409sLtpwix7Q0tnu4D8exlvUOEARyl6ioEp/pE2X4V2eoPfkNbFz4un4BoxImMSBfQTj4dq1sv7MEZI1Rb1H9XFVYm9fOUMo4hx+V3dCcAGxrFmRtHnrSA5VUCAAH9CUu/9gIazCEZ2vE4hA++X/f0BEo7WvmpNDVaIDfwE4V2ruc2cF3deyetW2Wzdi/KRvK4S8zEhmAJSO9nejSJqNZiqtWCDfsouBJz72e+aSVm4bsJ591AlbE/93F9SB6UikhneKQ0Y/IeOHlfzq/mk8MgXC/3vyaVmCByVzEP+syKGpWmX/VvsVHezupBorVY0yeX5fdbFA9P3UyazhWEqaHcq1+RyJWczM6EOiPct+LsZjXTRI5F50D2kc= ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T04:34:58.308 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS0flvN6atN0+P7osaPJSeC7Dj2GKtRjRWoWp7YtcsE7NUNhPIwFTUDq8PVfDQ3K1s0Vrbfuq3R2frgdSxFCigKVbH3YYHMz7SxmW6Z43q16O4UDO1rw5JwxfsuKZ6dHMffs5Hnzh409sLtpwix7Q0tnu4D8exlvUOEARyl6ioEp/pE2X4V2eoPfkNbFz4un4BoxImMSBfQTj4dq1sv7MEZI1Rb1H9XFVYm9fOUMo4hx+V3dCcAGxrFmRtHnrSA5VUCAAH9CUu/9gIazCEZ2vE4hA++X/f0BEo7WvmpNDVaIDfwE4V2ruc2cF3deyetW2Wzdi/KRvK4S8zEhmAJSO9nejSJqNZiqtWCDfsouBJz72e+aSVm4bsJ591AlbE/93F9SB6UikhneKQ0Y/IeOHlfzq/mk8MgXC/3vyaVmCByVzEP+syKGpWmX/VvsVHezupBorVY0yeX5fdbFA9P3UyazhWEqaHcq1+RyJWczM6EOiPct+LsZjXTRI5F50D2kc= ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:34:58.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-02T04:34:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:34:58 smithi069 bash[21528]: audit 2024-04-02T04:34:57.367783+0000 mon.a (mon.0) 96 : audit 1 from='client.? 172.21.15.69:0/3821218762' entity='client.admin' 2024-04-02T04:35:02.897 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:35:05.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/4255589671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 msgr2=0x7f0280108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:05.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 --2- 172.21.15.69:0/4255589671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f0280108870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0270009910 tx=0x7f0270009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/4255589671 shutdown_connections 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 --2- 172.21.15.69:0/4255589671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f0280108870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/4255589671 >> 172.21.15.69:0/4255589671 conn(0x7f0280077330 msgr2=0x7f02800797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/4255589671 shutdown_connections 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/4255589671 wait complete. 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 librados: starting msgr at 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 librados: starting objecter 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 Processor -- start 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- start start 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 librados: setting wanted keys 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 librados: calling monclient init 2024-04-02T04:35:05.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f028019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f028010d780 con 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0286133700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f028019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0286133700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f028019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54548/0 (socket says 172.21.15.69:54548) 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0286133700 1 -- 172.21.15.69:0/470287083 learned_addr learned my addr 172.21.15.69:0/470287083 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0286133700 1 -- 172.21.15.69:0/470287083 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028019d520 con 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0286133700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f028019cfe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f02700050d0 tx=0x7f02700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f02777fe700 10 client.?.objecter ms_handle_connect 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f02777fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0270027070 con 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f027000f460 con 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028019d7b0 con 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0270017440 con 0x7f0280108450 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:05.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02801a04a0 con 0x7f0280108450 2024-04-02T04:35:05.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.427+0000 7f0288397700 1 librados: init done 2024-04-02T04:35:05.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f027002a070 con 0x7f0280108450 2024-04-02T04:35:05.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f028010df00 con 0x7f0280108450 2024-04-02T04:35:05.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f026c03a8a0 0x7f026c03cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f027004cfd0 con 0x7f0280108450 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 10 client.14174.objecter ms_dispatch 0x7f0280001550 osd_map(3..3 src has 1..3) 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 20 client.14174.objecter dump_active .. 0 homeless 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f0285932700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f026c03a8a0 0x7f026c03cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f02777fe700 10 client.14174.objecter ms_handle_connect 0x7f026c03a8a0 2024-04-02T04:35:05.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.431+0000 7f0285932700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f026c03a8a0 0x7f026c03cd60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f027c006fd0 tx=0x7f027c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:05.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.435+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0270013070 con 0x7f0280108450 2024-04-02T04:35:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.635+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f028002db30 con 0x7f0280108450 2024-04-02T04:35:05.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f02777fe700 1 -- 172.21.15.69:0/470287083 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f02700142b0 con 0x7f0280108450 2024-04-02T04:35:05.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 10 librados: watch_flush enter 2024-04-02T04:35:05.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 10 librados: watch_flush exit 2024-04-02T04:35:05.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 20 client.14174.objecter shutdown clearing up homeless session... 2024-04-02T04:35:05.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 10 client.14174.objecter successfully canceled tick 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f026c03a8a0 msgr2=0x7f026c03cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f026c03a8a0 0x7f026c03cd60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f027c006fd0 tx=0x7f027c006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 msgr2=0x7f028019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f028019cfe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f02700050d0 tx=0x7f02700057f0 comp rx=0 tx=0).stop 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 shutdown_connections 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f026c03a8a0 0x7f026c03cd60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 --2- 172.21.15.69:0/470287083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0280108450 0x7f028019cfe0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 >> 172.21.15.69:0/470287083 conn(0x7f0280077330 msgr2=0x7f0280077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 shutdown_connections 2024-04-02T04:35:05.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 -- 172.21.15.69:0/470287083 wait complete. 2024-04-02T04:35:05.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:05.643+0000 7f0288397700 1 librados: shutdown 2024-04-02T04:35:06.251 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-02T04:35:06.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.255498+0000 mon.a (mon.0) 97 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.262069+0000 mon.a (mon.0) 98 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.264013+0000 mon.a (mon.0) 99 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.270520+0000 mon.a (mon.0) 100 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.288102+0000 mon.a (mon.0) 101 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.292585+0000 mon.a (mon.0) 102 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.644367+0000 mon.a (mon.0) 103 : audit 1 from='client.? 172.21.15.69:0/470287083' entity='client.admin' 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.689917+0000 mon.a (mon.0) 104 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:06.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:06 smithi069 bash[21528]: audit 2024-04-02T04:35:05.693900+0000 mon.a (mon.0) 105 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:08.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:08 smithi069 bash[21528]: cluster 2024-04-02T04:35:07.051045+0000 mgr.y (mgr.14152) 9 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:10.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:10 smithi069 bash[21528]: cluster 2024-04-02T04:35:09.051472+0000 mgr.y (mgr.14152) 10 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:10.963 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:35:12.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:12 smithi069 bash[21528]: cluster 2024-04-02T04:35:11.051951+0000 mgr.y (mgr.14152) 11 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- 172.21.15.69:0/2392766753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 msgr2=0x7f99081036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 --2- 172.21.15.69:0/2392766753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081036a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009910 tx=0x7f98f8009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- 172.21.15.69:0/2392766753 shutdown_connections 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 --2- 172.21.15.69:0/2392766753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081036a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- 172.21.15.69:0/2392766753 >> 172.21.15.69:0/2392766753 conn(0x7f99080fb8a0 msgr2=0x7f99080fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- 172.21.15.69:0/2392766753 shutdown_connections 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- 172.21.15.69:0/2392766753 wait complete. 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 librados: starting msgr at 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 librados: starting objecter 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 Processor -- start 2024-04-02T04:35:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- start start 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 librados: setting wanted keys 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 librados: calling monclient init 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990ee3e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9908107170 con 0x7f9908103280 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990cbda700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990cbda700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081028a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39752/0 (socket says 172.21.15.69:39752) 2024-04-02T04:35:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.115+0000 7f990cbda700 1 -- 172.21.15.69:0/3826977499 learned_addr learned my addr 172.21.15.69:0/3826977499 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:35:13.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f990cbda700 1 -- 172.21.15.69:0/3826977499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9908102de0 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f990cbda700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081028a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005950 tx=0x7f98f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 10 client.?.objecter ms_handle_connect 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f8027070 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f98f800bde0 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f800fce0 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f990ee3e700 1 -- 172.21.15.69:0/3826977499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9908100fd0 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f990ee3e700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f990ee3e700 1 -- 172.21.15.69:0/3826977499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9908101490 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f990ee3e700 1 librados: init done 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ec005120 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f98f802a070 con 0x7f9908103280 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f98f403a850 0x7f98f403cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f98f804d160 con 0x7f9908103280 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 10 client.14176.objecter ms_dispatch 0x7f9908001550 osd_map(3..3 src has 1..3) 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 20 client.14176.objecter dump_active .. 0 homeless 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9907fff700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f98f403a850 0x7f98f403cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9905ffb700 10 client.14176.objecter ms_handle_connect 0x7f98f403a850 2024-04-02T04:35:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.119+0000 7f9907fff700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f98f403a850 0x7f98f403cd10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f98f0006fd0 tx=0x7f98f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:13.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.123+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f98f8013070 con 0x7f9908103280 2024-04-02T04:35:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.335+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f98ec000bc0 con 0x7f98f403a850 2024-04-02T04:35:13.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f9905ffb700 1 -- 172.21.15.69:0/3826977499 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f98ec000bc0 con 0x7f98f403a850 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 10 librados: watch_flush enter 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 10 librados: watch_flush exit 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 20 client.14176.objecter shutdown clearing up homeless session... 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 10 client.14176.objecter successfully canceled tick 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f98f403a850 msgr2=0x7f98f403cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f98f403a850 0x7f98f403cd10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f98f0006fd0 tx=0x7f98f0006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 msgr2=0x7f99081028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081028a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005950 tx=0x7f98f80057f0 comp rx=0 tx=0).stop 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 shutdown_connections 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f98f403a850 0x7f98f403cd10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 --2- 172.21.15.69:0/3826977499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9908103280 0x7f99081028a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 >> 172.21.15.69:0/3826977499 conn(0x7f99080fb8a0 msgr2=0x7f99080fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 shutdown_connections 2024-04-02T04:35:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 -- 172.21.15.69:0/3826977499 wait complete. 2024-04-02T04:35:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:13.343+0000 7f98ff7fe700 1 librados: shutdown 2024-04-02T04:35:13.928 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-04-02T04:35:13.929 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:35:13.929 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-04-02T04:35:13.935 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:35:13.935 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:35:13.983 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-04-02T04:35:13.984 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi098 2024-04-02T04:35:14.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: cluster 2024-04-02T04:35:13.052215+0000 mgr.y (mgr.14152) 12 : cluster 0 pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:14.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.338868+0000 mgr.y (mgr.14152) 13 : audit 0 from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:14.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.343195+0000 mon.a (mon.0) 106 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:14.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.393342+0000 mon.a (mon.0) 107 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.394833+0000 mon.a (mon.0) 108 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.395472+0000 mon.a (mon.0) 109 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: cephadm 2024-04-02T04:35:13.396474+0000 mgr.y (mgr.14152) 14 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: cephadm 2024-04-02T04:35:13.510673+0000 mgr.y (mgr.14152) 15 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: cephadm 2024-04-02T04:35:13.675308+0000 mgr.y (mgr.14152) 16 : cephadm 1 Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: cephadm 2024-04-02T04:35:13.789401+0000 mgr.y (mgr.14152) 17 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.906197+0000 mon.a (mon.0) 110 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.909382+0000 mon.a (mon.0) 111 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:14.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:14 smithi069 bash[21528]: audit 2024-04-02T04:35:13.912373+0000 mon.a (mon.0) 112 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:16.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:16 smithi069 bash[21528]: cluster 2024-04-02T04:35:15.052615+0000 mgr.y (mgr.14152) 18 : cluster 0 pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:18.677 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:35:18.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:18 smithi069 bash[21528]: cluster 2024-04-02T04:35:17.052999+0000 mgr.y (mgr.14152) 19 : cluster 0 pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:20.536 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:20 smithi069 bash[21528]: cluster 2024-04-02T04:35:19.053411+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/1598564677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 msgr2=0x7fc5041022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 --2- 172.21.15.69:0/1598564677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc5041022a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009910 tx=0x7fc4f4009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/1598564677 shutdown_connections 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 --2- 172.21.15.69:0/1598564677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc5041022a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/1598564677 >> 172.21.15.69:0/1598564677 conn(0x7fc5040fd770 msgr2=0x7fc5040ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/1598564677 shutdown_connections 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/1598564677 wait complete. 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 librados: starting msgr at 2024-04-02T04:35:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 librados: starting objecter 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 Processor -- start 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- start start 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 librados: setting wanted keys 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 librados: calling monclient init 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc504198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc50410b760 con 0x7fc504101ec0 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc502ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc504198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc502ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc504198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50444/0 (socket says 172.21.15.69:50444) 2024-04-02T04:35:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc502ffd700 1 -- 172.21.15.69:0/2798114458 learned_addr learned my addr 172.21.15.69:0/2798114458 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc502ffd700 1 -- 172.21.15.69:0/2798114458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc504199520 con 0x7fc504101ec0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc502ffd700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc504198fe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40098e0 tx=0x7fc4f4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc4ebfff700 10 client.?.objecter ms_handle_connect 0x7fc504101ec0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc4ebfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4027070 con 0x7fc504101ec0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc4f400bde0 con 0x7fc504101ec0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4022c90 con 0x7fc504101ec0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc504199810 con 0x7fc504101ec0 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50419d8b0 con 0x7fc504101ec0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc50941d700 1 librados: init done 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.867+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fc4f4022570 con 0x7fc504101ec0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc4ec03a8a0 0x7fc4ec03cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5041033a0 con 0x7fc504101ec0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc4f401e030 con 0x7fc504101ec0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 10 client.14178.objecter ms_dispatch 0x7fc504001550 osd_map(3..3 src has 1..3) 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 20 client.14178.objecter dump_active .. 0 homeless 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc5027fc700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc4ec03a8a0 0x7fc4ec03cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc5027fc700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc4ec03a8a0 0x7fc4ec03cd60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006fd0 tx=0x7fc4f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.871+0000 7fc4ebfff700 10 client.14178.objecter ms_handle_connect 0x7fc4ec03a8a0 2024-04-02T04:35:20.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:20.875+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc4f4013070 con 0x7fc504101ec0 2024-04-02T04:35:21.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:21.079+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}) -- 0x7fc504063870 con 0x7fc4ec03a8a0 2024-04-02T04:35:22.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:22 smithi069 bash[21528]: cluster 2024-04-02T04:35:21.053691+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:22.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:22 smithi069 bash[21528]: audit 2024-04-02T04:35:21.084869+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:22.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:22 smithi069 bash[21528]: cephadm 2024-04-02T04:35:21.746213+0000 mgr.y (mgr.14152) 23 : cephadm 1 Deploying cephadm binary to smithi098 2024-04-02T04:35:24.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:24 smithi069 bash[21528]: cluster 2024-04-02T04:35:23.054002+0000 mgr.y (mgr.14152) 24 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:25.870+0000 7fc508c1c700 10 client.14178.objecter tick 2024-04-02T04:35:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.774+0000 7fc4ebfff700 1 -- 172.21.15.69:0/2798114458 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fc504063870 con 0x7fc4ec03a8a0 2024-04-02T04:35:26.780 INFO:teuthology.orchestra.run.smithi069.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-04-02T04:35:26.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 10 librados: watch_flush enter 2024-04-02T04:35:26.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 10 librados: watch_flush exit 2024-04-02T04:35:26.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 20 client.14178.objecter shutdown clearing up homeless session... 2024-04-02T04:35:26.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 10 client.14178.objecter successfully canceled tick 2024-04-02T04:35:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc4ec03a8a0 msgr2=0x7fc4ec03cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc4ec03a8a0 0x7fc4ec03cd60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006fd0 tx=0x7fc4f8006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 msgr2=0x7fc504198fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc504198fe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40098e0 tx=0x7fc4f4005030 comp rx=0 tx=0).stop 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 shutdown_connections 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fc4ec03a8a0 0x7fc4ec03cd60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 --2- 172.21.15.69:0/2798114458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc504101ec0 0x7fc504198fe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 >> 172.21.15.69:0/2798114458 conn(0x7fc5040fd770 msgr2=0x7fc504109bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 shutdown_connections 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 -- 172.21.15.69:0/2798114458 wait complete. 2024-04-02T04:35:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:26.778+0000 7fc50941d700 1 librados: shutdown 2024-04-02T04:35:26.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:26 smithi069 bash[21528]: cluster 2024-04-02T04:35:25.054397+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:27.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-02T04:35:28.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:27 smithi069 bash[21528]: audit 2024-04-02T04:35:26.776079+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:28.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:27 smithi069 bash[21528]: cephadm 2024-04-02T04:35:26.776922+0000 mgr.y (mgr.14152) 26 : cephadm 1 Added host smithi098 2024-04-02T04:35:28.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:27 smithi069 bash[21528]: audit 2024-04-02T04:35:26.795639+0000 mon.a (mon.0) 114 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:28.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:27 smithi069 bash[21528]: audit 2024-04-02T04:35:27.220596+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:29.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:28 smithi069 bash[21528]: cluster 2024-04-02T04:35:27.054868+0000 mgr.y (mgr.14152) 27 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:29.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:28 smithi069 bash[21528]: audit 2024-04-02T04:35:28.696114+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:31.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:30 smithi069 bash[21528]: cluster 2024-04-02T04:35:29.055176+0000 mgr.y (mgr.14152) 28 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:31.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:30 smithi069 bash[21528]: audit 2024-04-02T04:35:30.104925+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:32.089 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:35:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:32 smithi069 bash[21528]: cluster 2024-04-02T04:35:31.055538+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 -- 172.21.15.69:0/2936486725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 msgr2=0x7f17f8102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 --2- 172.21.15.69:0/2936486725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8102260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009910 tx=0x7f17e8009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 -- 172.21.15.69:0/2936486725 shutdown_connections 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 --2- 172.21.15.69:0/2936486725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8102260 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 -- 172.21.15.69:0/2936486725 >> 172.21.15.69:0/2936486725 conn(0x7f17f80fd750 msgr2=0x7f17f80ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 -- 172.21.15.69:0/2936486725 shutdown_connections 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.346+0000 7f17fe894700 1 -- 172.21.15.69:0/2936486725 wait complete. 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 librados: starting msgr at 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 librados: starting objecter 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 Processor -- start 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 -- start start 2024-04-02T04:35:34.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 librados: setting wanted keys 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 librados: calling monclient init 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17f810b720 con 0x7f17f8101e80 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48382/0 (socket says 172.21.15.69:48382) 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f7fff700 1 -- 172.21.15.69:0/878341667 learned_addr learned my addr 172.21.15.69:0/878341667 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f7fff700 1 -- 172.21.15.69:0/878341667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f8076430 con 0x7f17f8101e80 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f7fff700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8074540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17e8004f80 tx=0x7f17e8005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 10 client.?.objecter ms_handle_connect 0x7f17f8101e80 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17e8004390 con 0x7f17f8101e80 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f17e8004530 con 0x7f17f8101e80 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f8074ae0 con 0x7f17f8101e80 2024-04-02T04:35:34.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17e8003aa0 con 0x7f17f8101e80 2024-04-02T04:35:34.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:34.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f8074fa0 con 0x7f17f8101e80 2024-04-02T04:35:34.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 librados: init done 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f17e8003d10 con 0x7f17f8101e80 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f8103360 con 0x7f17f8101e80 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f17e403a8a0 0x7f17e403cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f17e801e070 con 0x7f17f8101e80 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 10 client.14180.objecter ms_dispatch 0x7f17f8001550 osd_map(3..3 src has 1..3) 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 3 client.14180.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 3 client.14180.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 20 client.14180.objecter dump_active .. 0 homeless 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f77fe700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f17e403a8a0 0x7f17e403cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f77fe700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f17e403a8a0 0x7f17e403cd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006fd0 tx=0x7f17e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:34.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.350+0000 7f17f57fa700 10 client.14180.objecter ms_handle_connect 0x7f17e403a8a0 2024-04-02T04:35:34.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.354+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f17e8013070 con 0x7f17f8101e80 2024-04-02T04:35:34.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.562+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f17f8063870 con 0x7f17e403a8a0 2024-04-02T04:35:34.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.562+0000 7f17f57fa700 1 -- 172.21.15.69:0/878341667 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f17f8063870 con 0x7f17e403a8a0 2024-04-02T04:35:34.569 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:35:34.569 INFO:teuthology.orchestra.run.smithi069.stdout:[{"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-04-02T04:35:34.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 10 librados: watch_flush enter 2024-04-02T04:35:34.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 10 librados: watch_flush exit 2024-04-02T04:35:34.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 20 client.14180.objecter shutdown clearing up homeless session... 2024-04-02T04:35:34.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 10 client.14180.objecter successfully canceled tick 2024-04-02T04:35:34.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f17e403a8a0 msgr2=0x7f17e403cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:34.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f17e403a8a0 0x7f17e403cd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006fd0 tx=0x7f17e0006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 msgr2=0x7f17f8074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8074540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17e8004f80 tx=0x7f17e8005060 comp rx=0 tx=0).stop 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 shutdown_connections 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f17e403a8a0 0x7f17e403cd60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 --2- 172.21.15.69:0/878341667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8101e80 0x7f17f8074540 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 >> 172.21.15.69:0/878341667 conn(0x7f17f80fd750 msgr2=0x7f17f8108a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 shutdown_connections 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 -- 172.21.15.69:0/878341667 wait complete. 2024-04-02T04:35:34.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:34.566+0000 7f17fe894700 1 librados: shutdown 2024-04-02T04:35:35.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:34 smithi069 bash[21528]: cluster 2024-04-02T04:35:33.055980+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:35.134 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-04-02T04:35:35.134 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables jewel 2024-04-02T04:35:36.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:34.565411+0000 mgr.y (mgr.14152) 31 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:35:36.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.588876+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:36.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.595905+0000 mon.a (mon.0) 119 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:36.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.602153+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:36.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.607542+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:36.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.609087+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:35:36.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.611116+0000 mon.a (mon.0) 123 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:36.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:35 smithi069 bash[21528]: audit 2024-04-02T04:35:35.612957+0000 mon.a (mon.0) 124 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:35:37.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:36 smithi069 bash[21528]: cluster 2024-04-02T04:35:35.056338+0000 mgr.y (mgr.14152) 32 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:37.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:36 smithi069 bash[21528]: cephadm 2024-04-02T04:35:35.615596+0000 mgr.y (mgr.14152) 33 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:35:37.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:36 smithi069 bash[21528]: cephadm 2024-04-02T04:35:35.831670+0000 mgr.y (mgr.14152) 34 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:35:37.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:36 smithi069 bash[21528]: audit 2024-04-02T04:35:36.330841+0000 mon.a (mon.0) 125 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:37.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:36 smithi069 bash[21528]: audit 2024-04-02T04:35:36.335766+0000 mon.a (mon.0) 126 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:37.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:36 smithi069 bash[21528]: audit 2024-04-02T04:35:36.340939+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:38.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:37 smithi069 bash[21528]: cephadm 2024-04-02T04:35:35.997249+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:35:38.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:37 smithi069 bash[21528]: cephadm 2024-04-02T04:35:36.161507+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:35:39.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:38 smithi069 bash[21528]: cluster 2024-04-02T04:35:37.056768+0000 mgr.y (mgr.14152) 37 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:39.845 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:35:40.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:40 smithi069 bash[21528]: cluster 2024-04-02T04:35:39.057189+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/626035359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 msgr2=0x7efcd0102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 --2- 172.21.15.69:0/626035359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0102260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009910 tx=0x7efcb8009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/626035359 shutdown_connections 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 --2- 172.21.15.69:0/626035359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0102260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/626035359 >> 172.21.15.69:0/626035359 conn(0x7efcd00fd750 msgr2=0x7efcd00ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/626035359 shutdown_connections 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/626035359 wait complete. 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 librados: starting msgr at 2024-04-02T04:35:42.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 librados: starting objecter 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 Processor -- start 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- start start 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 librados: setting wanted keys 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 librados: calling monclient init 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efcd010b720 con 0x7efcd0101e80 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efccf7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efccf7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47068/0 (socket says 172.21.15.69:47068) 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efccf7fe700 1 -- 172.21.15.69:0/2882417833 learned_addr learned my addr 172.21.15.69:0/2882417833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:35:42.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efccf7fe700 1 -- 172.21.15.69:0/2882417833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd0199480 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efccf7fe700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0198f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efcb80098e0 tx=0x7efcb8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 10 client.?.objecter ms_handle_connect 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcb8027070 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7efcb800bde0 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd0199770 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcb8022c10 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd019d810 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 librados: init done 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd0103360 con 0x7efcd0101e80 2024-04-02T04:35:42.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7efcb802a070 con 0x7efcd0101e80 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efcbc03a8a0 0x7efcbc03cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efcb801e030 con 0x7efcd0101e80 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 10 client.14182.objecter ms_dispatch 0x7efcd0001550 osd_map(3..3 src has 1..3) 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 3 client.14182.objecter handle_osd_map got epochs [3,3] > 0 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 3 client.14182.objecter handle_osd_map decoding full epoch 3 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcccff9700 20 client.14182.objecter dump_active .. 0 homeless 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.062+0000 7efcceffd700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efcbc03a8a0 0x7efcbc03cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.066+0000 7efcccff9700 10 client.14182.objecter ms_handle_connect 0x7efcbc03a8a0 2024-04-02T04:35:42.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.066+0000 7efcceffd700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efcbc03a8a0 0x7efcbc03cd60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efcc0006fd0 tx=0x7efcc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:42.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.070+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efcb8013070 con 0x7efcd0101e80 2024-04-02T04:35:42.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.274+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) -- 0x7efcd0063870 con 0x7efcd0101e80 2024-04-02T04:35:42.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.802+0000 7efcccff9700 1 -- 172.21.15.69:0/2882417833 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) ==== 120+0+0 (secure 0 0 0) 0x7efcb8031070 con 0x7efcd0101e80 2024-04-02T04:35:42.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 10 librados: watch_flush enter 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 10 librados: watch_flush exit 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 20 client.14182.objecter shutdown clearing up homeless session... 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 10 client.14182.objecter successfully canceled tick 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efcbc03a8a0 msgr2=0x7efcbc03cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efcbc03a8a0 0x7efcbc03cd60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efcc0006fd0 tx=0x7efcc0006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 msgr2=0x7efcd0198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0198f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efcb80098e0 tx=0x7efcb8005030 comp rx=0 tx=0).stop 2024-04-02T04:35:42.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 shutdown_connections 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efcbc03a8a0 0x7efcbc03cd60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 --2- 172.21.15.69:0/2882417833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0101e80 0x7efcd0198f40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 >> 172.21.15.69:0/2882417833 conn(0x7efcd00fd750 msgr2=0x7efcd0109b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 shutdown_connections 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 -- 172.21.15.69:0/2882417833 wait complete. 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:35:42.806+0000 7efcd5c90700 1 librados: shutdown 2024-04-02T04:35:42.813 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to jewel 2024-04-02T04:35:43.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:42 smithi069 bash[21528]: cluster 2024-04-02T04:35:41.057605+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:43.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:42 smithi069 bash[21528]: audit 2024-04-02T04:35:42.277235+0000 mon.a (mon.0) 128 : audit 1 from='client.? 172.21.15.69:0/2882417833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-02T04:35:43.415 INFO:tasks.cephadm:Adding mon.a on smithi069 2024-04-02T04:35:43.415 INFO:tasks.cephadm:Adding mon.c on smithi069 2024-04-02T04:35:43.416 INFO:tasks.cephadm:Adding mon.b on smithi098 2024-04-02T04:35:43.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi098:172.21.15.98=b' 2024-04-02T04:35:44.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:43 smithi069 bash[21528]: audit 2024-04-02T04:35:42.804612+0000 mon.a (mon.0) 129 : audit 1 from='client.? 172.21.15.69:0/2882417833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-02T04:35:44.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:43 smithi069 bash[21528]: cluster 2024-04-02T04:35:42.809280+0000 mon.a (mon.0) 130 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:35:44.627 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:35:45.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:44 smithi069 bash[21528]: cluster 2024-04-02T04:35:43.058006+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:46.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- 172.21.15.98:0/1626696468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 msgr2=0x7f927c102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:46.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 --2- 172.21.15.98:0/1626696468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c102280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f926c009910 tx=0x7f926c009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- 172.21.15.98:0/1626696468 shutdown_connections 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 --2- 172.21.15.98:0/1626696468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c102280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- 172.21.15.98:0/1626696468 >> 172.21.15.98:0/1626696468 conn(0x7f927c0fd770 msgr2=0x7f927c0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- 172.21.15.98:0/1626696468 shutdown_connections 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- 172.21.15.98:0/1626696468 wait complete. 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 librados: starting msgr at 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 librados: starting objecter 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 Processor -- start 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- start start 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 librados: setting wanted keys 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 librados: calling monclient init 2024-04-02T04:35:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f928227c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f927c10b740 con 0x7f927c101ea0 2024-04-02T04:35:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f927bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f927bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.98:49390/0 (socket says 172.21.15.98:49390) 2024-04-02T04:35:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.985+0000 7f927bfff700 1 -- 172.21.15.98:0/3328270212 learned_addr learned my addr 172.21.15.98:0/3328270212 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:35:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f927bfff700 1 -- 172.21.15.98:0/3328270212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927c076430 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f927bfff700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c074540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f926c004f80 tx=0x7f926c005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 10 client.?.objecter ms_handle_connect 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f926c004390 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f927c074ae0 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f928227c700 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f927c074fa0 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f928227c700 1 librados: init done 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f926c004530 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f926c003990 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9260005120 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f926c0046e0 con 0x7f927c101ea0 2024-04-02T04:35:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925c03a850 0x7f925c03cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f926c01e070 con 0x7f927c101ea0 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 10 client.14184.objecter ms_dispatch 0x7f927c001550 osd_map(4..4 src has 1..4) 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 20 client.14184.objecter dump_active .. 0 homeless 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9273fff700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925c03a850 0x7f925c03cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9273fff700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925c03a850 0x7f925c03cd10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9264006fd0 tx=0x7f9264006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:46.989+0000 7f9279ffb700 10 client.14184.objecter ms_handle_connect 0x7f925c03a850 2024-04-02T04:35:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.005+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f926c013070 con 0x7f927c101ea0 2024-04-02T04:35:47.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.221+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi098:172.21.15.98=b", "target": ["mon-mgr", ""]}) -- 0x7f9260000c90 con 0x7f925c03a850 2024-04-02T04:35:47.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f9279ffb700 1 -- 172.21.15.98:0/3328270212 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9260000c90 con 0x7f925c03a850 2024-04-02T04:35:47.244 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mon update... 2024-04-02T04:35:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 10 librados: watch_flush enter 2024-04-02T04:35:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 10 librados: watch_flush exit 2024-04-02T04:35:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 20 client.14184.objecter shutdown clearing up homeless session... 2024-04-02T04:35:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 10 client.14184.objecter successfully canceled tick 2024-04-02T04:35:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925c03a850 msgr2=0x7f925c03cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925c03a850 0x7f925c03cd10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9264006fd0 tx=0x7f9264006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 msgr2=0x7f927c074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.237+0000 7f928227c700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c074540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f926c004f80 tx=0x7f926c005060 comp rx=0 tx=0).stop 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 shutdown_connections 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f925c03a850 0x7f925c03cd10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 --2- 172.21.15.98:0/3328270212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f927c101ea0 0x7f927c074540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 >> 172.21.15.98:0/3328270212 conn(0x7f927c0fd770 msgr2=0x7f927c109ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 shutdown_connections 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 -- 172.21.15.98:0/3328270212 wait complete. 2024-04-02T04:35:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:47.241+0000 7f928227c700 1 librados: shutdown 2024-04-02T04:35:47.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:46 smithi069 bash[21528]: cluster 2024-04-02T04:35:45.058352+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:47.996 DEBUG:teuthology.orchestra.run.smithi069:mon.c> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.c.service 2024-04-02T04:35:47.999 DEBUG:teuthology.orchestra.run.smithi098:mon.b> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.b.service 2024-04-02T04:35:48.001 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-02T04:35:48.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:35:48.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: cluster 2024-04-02T04:35:47.058764+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:48.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.226809+0000 mgr.y (mgr.14152) 43 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi098:172.21.15.98=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:48.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: cephadm 2024-04-02T04:35:47.232305+0000 mgr.y (mgr.14152) 44 : cephadm 1 Saving service mon spec with placement smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi098:172.21.15.98=b;count:3 2024-04-02T04:35:48.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.240167+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.248794+0000 mon.a (mon.0) 132 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.252666+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.254680+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.265441+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.269900+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: audit 2024-04-02T04:35:47.271684+0000 mon.a (mon.0) 137 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:48 smithi069 bash[21528]: cephadm 2024-04-02T04:35:47.273828+0000 mgr.y (mgr.14152) 45 : cephadm 1 Deploying daemon mon.b on smithi098 2024-04-02T04:35:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:35:50.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:50 smithi069 bash[21528]: cluster 2024-04-02T04:35:49.059222+0000 mgr.y (mgr.14152) 46 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:52.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:52 smithi069 bash[21528]: cluster 2024-04-02T04:35:51.059650+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:52.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:52 smithi098 systemd[1]: Started Ceph mon.b for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:35:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/1832727954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 msgr2=0x7f1a0c074940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 --2- 172.21.15.98:0/1832727954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c074940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009910 tx=0x7f19fc009c20 comp rx=0 tx=0).stop 2024-04-02T04:35:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/1832727954 shutdown_connections 2024-04-02T04:35:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 --2- 172.21.15.98:0/1832727954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c074940 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/1832727954 >> 172.21.15.98:0/1832727954 conn(0x7f1a0c0fd750 msgr2=0x7f1a0c0ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/1832727954 shutdown_connections 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/1832727954 wait complete. 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 librados: starting msgr at 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 librados: starting objecter 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 Processor -- start 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- start start 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 librados: setting wanted keys 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 librados: calling monclient init 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c10d980 con 0x7f1a0c074560 2024-04-02T04:35:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a0affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a0affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.98:47356/0 (socket says 172.21.15.98:47356) 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a0affd700 1 -- 172.21.15.98:0/3531283678 learned_addr learned my addr 172.21.15.98:0/3531283678 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a0affd700 1 -- 172.21.15.98:0/3531283678 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0c1994d0 con 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a0affd700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c198f90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f19fc004f80 tx=0x7f19fc005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f19f3fff700 10 client.?.objecter ms_handle_connect 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f19f3fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19fc026070 con 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a0c199760 con 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f19fc004370 con 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 10 client.14186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19fc0038f0 con 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a0c199bc0 con 0x7f1a0c074560 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.825+0000 7f1a113cf700 1 librados: init done 2024-04-02T04:35:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a0c075ca0 con 0x7f1a0c074560 2024-04-02T04:35:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f19fc029070 con 0x7f1a0c074560 2024-04-02T04:35:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19f403a850 0x7f19f403cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:35:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f19fc01e070 con 0x7f1a0c074560 2024-04-02T04:35:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 10 client.14186.objecter ms_dispatch 0x7f1a0c001550 osd_map(4..4 src has 1..4) 2024-04-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 3 client.14186.objecter handle_osd_map got epochs [4,4] > 0 2024-04-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 3 client.14186.objecter handle_osd_map decoding full epoch 4 2024-04-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 20 client.14186.objecter dump_active .. 0 homeless 2024-04-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f1a0a7fc700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19f403a850 0x7f19f403cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f19f3fff700 10 client.14186.objecter ms_handle_connect 0x7f19f403a850 2024-04-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.829+0000 7f1a0a7fc700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19f403a850 0x7f19f403cd10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:35:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:52.841+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f19fc030020 con 0x7f1a0c074560 2024-04-02T04:35:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1a0c063870 con 0x7f1a0c074560 2024-04-02T04:35:53.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f19f3fff700 1 -- 172.21.15.98:0/3531283678 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f19fc013070 con 0x7f1a0c074560 2024-04-02T04:35:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-02T04:35:53.140 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:33:39.262476Z","created":"2024-04-02T04:33:39.262476Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:35:53.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 10 librados: watch_flush enter 2024-04-02T04:35:53.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 10 librados: watch_flush exit 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 20 client.14186.objecter shutdown clearing up homeless session... 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 10 client.14186.objecter successfully canceled tick 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19f403a850 msgr2=0x7f19f403cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19f403a850 0x7f19f403cd10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00006e40 comp rx=0 tx=0).stop 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 msgr2=0x7f1a0c198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c198f90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f19fc004f80 tx=0x7f19fc005060 comp rx=0 tx=0).stop 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 shutdown_connections 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f19f403a850 0x7f19f403cd10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 --2- 172.21.15.98:0/3531283678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a0c074560 0x7f1a0c198f90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 >> 172.21.15.98:0/3531283678 conn(0x7f1a0c0fd750 msgr2=0x7f1a0c10b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 shutdown_connections 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 -- 172.21.15.98:0/3531283678 wait complete. 2024-04-02T04:35:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:35:53.133+0000 7f1a113cf700 1 librados: shutdown 2024-04-02T04:35:53.143 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-04-02T04:35:54.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: audit 2024-04-02T04:35:52.735514+0000 mon.a (mon.0) 138 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: audit 2024-04-02T04:35:52.743667+0000 mon.a (mon.0) 139 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: audit 2024-04-02T04:35:52.759093+0000 mon.a (mon.0) 140 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: audit 2024-04-02T04:35:52.760573+0000 mon.a (mon.0) 141 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:35:54.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: audit 2024-04-02T04:35:52.762273+0000 mon.a (mon.0) 142 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:54.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: cephadm 2024-04-02T04:35:52.764432+0000 mgr.y (mgr.14152) 48 : cephadm 1 Deploying daemon mon.c on smithi069 2024-04-02T04:35:54.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:53 smithi069 bash[21528]: audit 2024-04-02T04:35:53.136245+0000 mon.a (mon.0) 143 : audit 0 from='client.? 172.21.15.98:0/3531283678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:35:54.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.101+0000 7ffa393a0c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.101+0000 7ffa393a0c80 0 ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev), process ceph-mon, pid 7 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.101+0000 7ffa393a0c80 0 pidfile_write: ignore empty --pid-file 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.105+0000 7ffa393a0c80 0 load: jerasure load: lrc 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Git sha 0 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Compile date 2024-04-01 15:23:26 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: DB SUMMARY 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: DB Session ID: SMJ6TGU8LZXQNG0G7RSE 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: CURRENT file: CURRENT 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-02T04:35:54.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.error_if_exists: 0 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.create_if_missing: 0 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.env: 0x559732b6bc60 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.info_log: 0x559734d4bb20 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.statistics: (nil) 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.use_fsync: 0 2024-04-02T04:35:54.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-02T04:35:54.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T04:35:54.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T04:35:54.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-02T04:35:54.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-02T04:35:54.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-02T04:35:54.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.db_log_dir: 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.wal_dir: 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-02T04:35:54.480 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.write_buffer_manager: 0x559734d1e1e0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.unordered_write: 0 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T04:35:54.481 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.row_cache: None 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.wal_filter: None 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.two_write_queues: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.wal_compression: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.atomic_flush: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T04:35:54.482 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T04:35:54.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_open_files: -1 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Compression algorithms supported: 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kZSTD supported: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kXpressCompression supported: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kZlibCompression supported: 1 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-02T04:35:54.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T04:35:54.485 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-02T04:35:54.485 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-02T04:35:54.485 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-02T04:35:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T04:35:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.merge_operator: 2024-04-02T04:35:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_filter: None 2024-04-02T04:35:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-02T04:35:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-02T04:35:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559734d4bc40) 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cache_index_and_filter_blocks: 1 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: pin_top_level_index_and_filter: 1 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: index_type: 0 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: data_block_index_type: 0 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: index_shortening: 1 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: data_block_hash_table_util_ratio: 0.750000 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: checksum: 4 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: no_block_cache: 0 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_cache: 0x559734d4d090 2024-04-02T04:35:54.492 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_cache_name: BinnedLRUCache 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_cache_options: 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: capacity : 536870912 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: num_shard_bits : 4 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: strict_capacity_limit : 0 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: high_pri_pool_ratio: 0.000 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_cache_compressed: (nil) 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: persistent_cache: (nil) 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_size: 4096 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_size_deviation: 10 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_restart_interval: 16 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: index_block_restart_interval: 1 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: metadata_block_size: 4096 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: partition_filters: 0 2024-04-02T04:35:54.493 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: use_delta_encoding: 1 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: filter_policy: bloomfilter 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: whole_key_filtering: 1 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: verify_compression: 0 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: read_amp_bytes_per_bit: 0 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: format_version: 5 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: enable_index_compression: 1 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: block_align: 0 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: max_auto_readahead_size: 262144 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: prepopulate_block_cache: 0 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: initial_auto_readahead_size: 8192 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: num_file_reads_for_auto_readahead: 2 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-02T04:35:54.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression: NoCompression 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.num_levels: 7 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T04:35:54.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T04:35:54.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T04:35:54.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T04:35:54.498 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.table_properties_collectors: 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.bloom_locality: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.ttl: 2592000 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T04:35:54.499 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.enable_blob_files: false 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.min_blob_size: 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-02T04:35:54.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c18b39d6-9001-44a1-9570-372f43273add 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032554114341, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032554114909, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712032554, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c18b39d6-9001-44a1-9570-372f43273add", "db_session_id": "SMJ6TGU8LZXQNG0G7RSE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032554115025, "job": 1, "event": "recovery_finished"} 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.109+0000 7ffa393a0c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa393a0c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa393a0c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559734e42000 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa393a0c80 4 rocksdb: DB pointer 0x559734e2c000 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa27b02700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa27b02700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: ** DB Stats ** 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:35:54.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: ** Compaction Stats [default] ** 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: ** Compaction Stats [default] ** 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T04:35:54.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Cumulative compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Interval compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Block cache BinnedLRUCache@0x559734d4d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa393a0c80 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-02T04:35:54.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa393a0c80 0 using public_addr v2:172.21.15.98:0/0 -> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.113+0000 7ffa393a0c80 0 starting mon.b rank -1 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.117+0000 7ffa393a0c80 1 mon.b@-1(???) e0 preinit fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 0 mon.b@-1(synchronizing).mds e1 new map 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: e1 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: legacy client fscid: -1 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: No filesystems configured 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa262ff700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2f311700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.140389) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-02T04:35:54.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2f311700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2f311700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032554140448, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1109, "num_deletes": 24, "total_data_size": 2246911, "memory_usage": 2270048, "flush_reason": "Manual Compaction"} 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2f311700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:23.054002+0000 mgr.y (mgr.14152) 24 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:25.054397+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:26.776079+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:26.776922+0000 mgr.y (mgr.14152) 26 : cephadm 1 Added host smithi098 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:26.795639+0000 mon.a (mon.0) 114 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:27.220596+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:27.054868+0000 mgr.y (mgr.14152) 27 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:28.696114+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:29.055176+0000 mgr.y (mgr.14152) 28 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:30.104925+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:31.055538+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:33.055980+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:34.565411+0000 mgr.y (mgr.14152) 31 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.588876+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.595905+0000 mon.a (mon.0) 119 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.602153+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.607542+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.609087+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.611116+0000 mon.a (mon.0) 123 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:35.612957+0000 mon.a (mon.0) 124 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:35.056338+0000 mgr.y (mgr.14152) 32 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:35.615596+0000 mgr.y (mgr.14152) 33 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:35.831670+0000 mgr.y (mgr.14152) 34 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:36.330841+0000 mon.a (mon.0) 125 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:36.335766+0000 mon.a (mon.0) 126 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:36.340939+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:35.997249+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:36.161507+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:37.056768+0000 mgr.y (mgr.14152) 37 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:39.057189+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:41.057605+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:42.277235+0000 mon.a (mon.0) 128 : audit 1 from='client.? 172.21.15.69:0/2882417833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:42.804612+0000 mon.a (mon.0) 129 : audit 1 from='client.? 172.21.15.69:0/2882417833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:42.809280+0000 mon.a (mon.0) 130 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:43.058006+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:45.058352+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:47.058764+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.226809+0000 mgr.y (mgr.14152) 43 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi098:172.21.15.98=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:47.232305+0000 mgr.y (mgr.14152) 44 : cephadm 1 Saving service mon spec with placement smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi098:172.21.15.98=b;count:3 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.240167+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.248794+0000 mon.a (mon.0) 132 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.252666+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.254680+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:35:54.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.265441+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.269900+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:47.271684+0000 mon.a (mon.0) 137 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:47.273828+0000 mgr.y (mgr.14152) 45 : cephadm 1 Deploying daemon mon.b on smithi098 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:49.059222+0000 mgr.y (mgr.14152) 46 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cluster 2024-04-02T04:35:51.059650+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:52.735514+0000 mon.a (mon.0) 138 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:52.743667+0000 mon.a (mon.0) 139 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:52.759093+0000 mon.a (mon.0) 140 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:52.760573+0000 mon.a (mon.0) 141 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:52.762273+0000 mon.a (mon.0) 142 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: cephadm 2024-04-02T04:35:52.764432+0000 mgr.y (mgr.14152) 48 : cephadm 1 Deploying daemon mon.c on smithi069 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: audit 2024-04-02T04:35:53.136245+0000 mon.a (mon.0) 143 : audit 0 from='client.? 172.21.15.98:0/3531283678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.137+0000 7ffa2ab08700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-02T04:35:54.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032554144555, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1984040, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1114, "table_properties": {"data_size": 1978450, "index_size": 2436, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2117, "raw_key_size": 25246, "raw_average_key_size": 30, "raw_value_size": 1961335, "raw_average_value_size": 2383, "num_data_blocks": 114, "num_entries": 823, "num_filter_entries": 823, "num_deletions": 24, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712032554, "oldest_key_time": 1712032554, "file_creation_time": 1712032554, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c18b39d6-9001-44a1-9570-372f43273add", "db_session_id": "SMJ6TGU8LZXQNG0G7RSE", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 4189 microseconds, and 3481 cpu microseconds. 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.144582) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 1984040 bytes OK 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.144602) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.144730) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.144743) EVENT_LOG_v1 {"time_micros": 1712032554144739, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.144752) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2239184, prev total WAL file size 2239325, number of live WAL files 2. 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2f311700 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2fb12700 4 rocksdb: (Original Log Time 2024/04/02-04:35:54.145112) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003232' seq:0, type:0; will stop at (end) 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2fb12700 4 rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2fb12700 4 rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1937KB) 8(1622B)] 2024-04-02T04:35:54.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:35:54 smithi098 bash[23790]: debug 2024-04-02T04:35:54.141+0000 7ffa2fb12700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712032554145172, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 1985662, "oldest_snapshot_seqno": -1} 2024-04-02T04:35:55.046 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-02T04:35:55.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:35:56.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:56 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:35:56.251 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:35:56 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:35:56.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:35:56 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:35:56.535 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:35:56 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:35:57.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:35:57 smithi069 bash[28904]: debug 2024-04-02T04:35:57.234+0000 7fdc2f3aa700 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 4443 microseconds, and 3749 cpu microseconds. 2024-04-02T04:35:58.739 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:36:00.535 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:36:00 smithi069 bash[21782]: debug 2024-04-02T04:36:00.146+0000 7feb5dfb4700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-02T04:36:04.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- 172.21.15.98:0/3953273297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf140035c0 msgr2=0x7fdf14005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:04.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 --2- 172.21.15.98:0/3953273297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf140035c0 0x7fdf14005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdf28101c50 tx=0x7fdf18013e90 comp rx=0 tx=0).stop 2024-04-02T04:36:04.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- 172.21.15.98:0/3953273297 shutdown_connections 2024-04-02T04:36:04.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 --2- 172.21.15.98:0/3953273297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf140035c0 0x7fdf14005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- 172.21.15.98:0/3953273297 >> 172.21.15.98:0/3953273297 conn(0x7fdf280fd810 msgr2=0x7fdf280ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:04.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- 172.21.15.98:0/3953273297 shutdown_connections 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- 172.21.15.98:0/3953273297 wait complete. 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 librados: starting msgr at 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 librados: starting objecter 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 Processor -- start 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- start start 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 librados: setting wanted keys 2024-04-02T04:36:04.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 librados: calling monclient init 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 0x7fdf281975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2819bfd0 0x7fdf2819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdf281092b0 con 0x7fdf2819bfd0 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28109130 con 0x7fdf28197ae0 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf30516700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdf28109430 con 0x7fdf281971c0 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48330/0 (socket says 172.21.15.98:48330) 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 -- 172.21.15.98:0/1966773037 learned_addr learned my addr 172.21.15.98:0/1966773037 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2e2b2700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 0x7fdf281975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 msgr2=0x7fdf2819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-02T04:36:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 msgr2=0x7fdf2819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2eab3700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2819bfd0 0x7fdf2819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2dab1700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2e2b2700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 msgr2=0x7fdf281975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2e2b2700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 msgr2=0x7fdf281975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2e2b2700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 0x7fdf281975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.437+0000 7fdf2e2b2700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 0x7fdf281975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2eab3700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 msgr2=0x7fdf281975a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2eab3700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 0x7fdf281975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2eab3700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 msgr2=0x7fdf2819ba90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2eab3700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2eab3700 1 -- 172.21.15.98:0/1966773037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf281a4010 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2eab3700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2819bfd0 0x7fdf2819c460 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf2000bd50 tx=0x7fdf2000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 10 client.?.objecter ms_handle_connect 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20019070 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fdf2000cd10 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf281a4300 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf200125a0 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf30516700 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf281a4840 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf30516700 1 librados: init done 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf28063870 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fdf2001c070 con 0x7fdf2819bfd0 2024-04-02T04:36:04.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fdf0c03acf0 0x7fdf0c03d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdf2004b770 con 0x7fdf2819bfd0 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 10 client.14190.objecter ms_dispatch 0x7fdf28001550 osd_map(4..4 src has 1..4) 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 20 client.14190.objecter dump_active .. 0 homeless 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2e2b2700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fdf0c03acf0 0x7fdf0c03d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf1f7fe700 10 client.14190.objecter ms_handle_connect 0x7fdf0c03acf0 2024-04-02T04:36:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.441+0000 7fdf2e2b2700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fdf0c03acf0 0x7fdf0c03d1b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdf18000c00 tx=0x7fdf18005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:04.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.449+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf20014440 con 0x7fdf2819bfd0 2024-04-02T04:36:04.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:35:59.061527+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:04.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:35:59.250971+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:36:04.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:35:59.251028+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-04-02T04:36:04.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:35:59.251324+0000 mon.a (mon.0) 172 : cluster 1 mon.a calling monitor election 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:35:59.273916+0000 mon.a (mon.0) 173 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:35:59.274237+0000 mon.a (mon.0) 174 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:00.242971+0000 mon.a (mon.0) 175 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:01.061958+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:01.243094+0000 mon.a (mon.0) 176 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:02.243287+0000 mon.a (mon.0) 177 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:03.062353+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:03.243301+0000 mon.a (mon.0) 178 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:04.243455+0000 mon.a (mon.0) 179 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.273227+0000 mon.a (mon.0) 180 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279263+0000 mon.a (mon.0) 181 : cluster 0 monmap epoch 3 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279302+0000 mon.a (mon.0) 182 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279322+0000 mon.a (mon.0) 183 : cluster 0 last_changed 2024-04-02T04:35:59.243766+0000 2024-04-02T04:36:04.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279344+0000 mon.a (mon.0) 184 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279368+0000 mon.a (mon.0) 185 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279393+0000 mon.a (mon.0) 186 : cluster 0 election_strategy: 1 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279419+0000 mon.a (mon.0) 187 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279439+0000 mon.a (mon.0) 188 : cluster 0 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.279462+0000 mon.a (mon.0) 189 : cluster 0 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.300731+0000 mon.a (mon.0) 190 : cluster 0 fsmap 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.300816+0000 mon.a (mon.0) 191 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.301108+0000 mon.a (mon.0) 192 : cluster 0 mgrmap e13: y(active, since 77s) 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.301416+0000 mon.a (mon.0) 193 : cluster 3 Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.308436+0000 mon.a (mon.0) 194 : cluster 3 Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: cluster 2024-04-02T04:36:04.308536+0000 mon.a (mon.0) 196 : cluster 3 mon.c (rank 2) addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] is down (out of quorum) 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:04.313494+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:04.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:04 smithi098 bash[23790]: audit 2024-04-02T04:36:04.321344+0000 mon.a (mon.0) 198 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.733+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fdf281a4b00 con 0x7fdf2819bfd0 2024-04-02T04:36:04.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf1f7fe700 1 -- 172.21.15.98:0/1966773037 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7fdf200127a0 con 0x7fdf2819bfd0 2024-04-02T04:36:04.743 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-02T04:36:04.743 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:35:59.243766Z","created":"2024-04-02T04:33:39.262476Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3301","nonce":0},{"type":"v1","addr":"172.21.15.69:6790","nonce":0}]},"addr":"172.21.15.69:6790/0","public_addr":"172.21.15.69:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 10 librados: watch_flush enter 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 10 librados: watch_flush exit 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 20 client.14190.objecter shutdown clearing up homeless session... 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 10 client.14190.objecter successfully canceled tick 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fdf0c03acf0 msgr2=0x7fdf0c03d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fdf0c03acf0 0x7fdf0c03d1b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdf18000c00 tx=0x7fdf18005960 comp rx=0 tx=0).stop 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2819bfd0 msgr2=0x7fdf2819c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2819bfd0 0x7fdf2819c460 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf2000bd50 tx=0x7fdf2000be30 comp rx=0 tx=0).stop 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 shutdown_connections 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2819bfd0 0x7fdf2819c460 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fdf0c03acf0 0x7fdf0c03d1b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf281971c0 0x7fdf281975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 --2- 172.21.15.98:0/1966773037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28197ae0 0x7fdf2819ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 >> 172.21.15.98:0/1966773037 conn(0x7fdf280fd810 msgr2=0x7fdf28105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 shutdown_connections 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 -- 172.21.15.98:0/1966773037 wait complete. 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:04.737+0000 7fdf30516700 1 librados: shutdown 2024-04-02T04:36:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 3 2024-04-02T04:36:04.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:35:59.061527+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:04.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:35:59.250971+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:35:59.251028+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:35:59.251324+0000 mon.a (mon.0) 172 : cluster 1 mon.a calling monitor election 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:35:59.273916+0000 mon.a (mon.0) 173 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:35:59.274237+0000 mon.a (mon.0) 174 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:00.242971+0000 mon.a (mon.0) 175 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:01.061958+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:01.243094+0000 mon.a (mon.0) 176 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:02.243287+0000 mon.a (mon.0) 177 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:03.062353+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:03.243301+0000 mon.a (mon.0) 178 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:04.243455+0000 mon.a (mon.0) 179 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:04.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.273227+0000 mon.a (mon.0) 180 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279263+0000 mon.a (mon.0) 181 : cluster 0 monmap epoch 3 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279302+0000 mon.a (mon.0) 182 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279322+0000 mon.a (mon.0) 183 : cluster 0 last_changed 2024-04-02T04:35:59.243766+0000 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279344+0000 mon.a (mon.0) 184 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279368+0000 mon.a (mon.0) 185 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279393+0000 mon.a (mon.0) 186 : cluster 0 election_strategy: 1 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279419+0000 mon.a (mon.0) 187 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279439+0000 mon.a (mon.0) 188 : cluster 0 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.279462+0000 mon.a (mon.0) 189 : cluster 0 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.300731+0000 mon.a (mon.0) 190 : cluster 0 fsmap 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.300816+0000 mon.a (mon.0) 191 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.301108+0000 mon.a (mon.0) 192 : cluster 0 mgrmap e13: y(active, since 77s) 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.301416+0000 mon.a (mon.0) 193 : cluster 3 Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-02T04:36:04.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.308436+0000 mon.a (mon.0) 194 : cluster 3 Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-02T04:36:04.788 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-02T04:36:04.788 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: cluster 2024-04-02T04:36:04.308536+0000 mon.a (mon.0) 196 : cluster 3 mon.c (rank 2) addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] is down (out of quorum) 2024-04-02T04:36:04.788 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:04.313494+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:04.788 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:04 smithi069 bash[21528]: audit 2024-04-02T04:36:04.321344+0000 mon.a (mon.0) 198 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:05.467 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-02T04:36:05.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-02T04:36:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:05 smithi098 bash[23790]: audit 2024-04-02T04:36:04.389652+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:05 smithi098 bash[23790]: audit 2024-04-02T04:36:04.739941+0000 mon.a (mon.0) 200 : audit 0 from='client.? 172.21.15.98:0/1966773037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:05 smithi098 bash[23790]: audit 2024-04-02T04:36:05.243788+0000 mon.a (mon.0) 201 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:05.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:05 smithi069 bash[21528]: audit 2024-04-02T04:36:04.389652+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:05.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:05 smithi069 bash[21528]: audit 2024-04-02T04:36:04.739941+0000 mon.a (mon.0) 200 : audit 0 from='client.? 172.21.15.98:0/1966773037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:05.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:05 smithi069 bash[21528]: audit 2024-04-02T04:36:05.243788+0000 mon.a (mon.0) 201 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:01.245425+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.265898+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.266854+0000 mon.a (mon.0) 203 : cluster 1 mon.a calling monitor election 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.267426+0000 mon.b (mon.1) 3 : cluster 1 mon.b calling monitor election 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.290996+0000 mon.a (mon.0) 204 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298276+0000 mon.a (mon.0) 205 : cluster 0 monmap epoch 3 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298315+0000 mon.a (mon.0) 206 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:06.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298344+0000 mon.a (mon.0) 207 : cluster 0 last_changed 2024-04-02T04:35:59.243766+0000 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298367+0000 mon.a (mon.0) 208 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298408+0000 mon.a (mon.0) 209 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298444+0000 mon.a (mon.0) 210 : cluster 0 election_strategy: 1 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298483+0000 mon.a (mon.0) 211 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298512+0000 mon.a (mon.0) 212 : cluster 0 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.298542+0000 mon.a (mon.0) 213 : cluster 0 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.299635+0000 mon.a (mon.0) 214 : cluster 0 fsmap 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.299704+0000 mon.a (mon.0) 215 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.300042+0000 mon.a (mon.0) 216 : cluster 0 mgrmap e13: y(active, since 79s) 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.300335+0000 mon.a (mon.0) 217 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.300381+0000 mon.a (mon.0) 218 : cluster 1 Cluster is now healthy 2024-04-02T04:36:06.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:06 smithi098 bash[23790]: cluster 2024-04-02T04:36:06.308986+0000 mon.a (mon.0) 219 : cluster 1 overall HEALTH_OK 2024-04-02T04:36:06.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:01.245425+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-04-02T04:36:06.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.265898+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2024-04-02T04:36:06.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.266854+0000 mon.a (mon.0) 203 : cluster 1 mon.a calling monitor election 2024-04-02T04:36:06.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.267426+0000 mon.b (mon.1) 3 : cluster 1 mon.b calling monitor election 2024-04-02T04:36:06.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.290996+0000 mon.a (mon.0) 204 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298276+0000 mon.a (mon.0) 205 : cluster 0 monmap epoch 3 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298315+0000 mon.a (mon.0) 206 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298344+0000 mon.a (mon.0) 207 : cluster 0 last_changed 2024-04-02T04:35:59.243766+0000 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298367+0000 mon.a (mon.0) 208 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298408+0000 mon.a (mon.0) 209 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298444+0000 mon.a (mon.0) 210 : cluster 0 election_strategy: 1 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298483+0000 mon.a (mon.0) 211 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298512+0000 mon.a (mon.0) 212 : cluster 0 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.298542+0000 mon.a (mon.0) 213 : cluster 0 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.299635+0000 mon.a (mon.0) 214 : cluster 0 fsmap 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.299704+0000 mon.a (mon.0) 215 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.300042+0000 mon.a (mon.0) 216 : cluster 0 mgrmap e13: y(active, since 79s) 2024-04-02T04:36:06.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.300335+0000 mon.a (mon.0) 217 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.300381+0000 mon.a (mon.0) 218 : cluster 1 Cluster is now healthy 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[21528]: cluster 2024-04-02T04:36:06.308986+0000 mon.a (mon.0) 219 : cluster 1 overall HEALTH_OK 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:35:59.061527+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:35:59.250971+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:35:59.251028+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:35:59.251324+0000 mon.a (mon.0) 172 : cluster 1 mon.a calling monitor election 2024-04-02T04:36:06.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:35:59.273916+0000 mon.a (mon.0) 173 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:36:06.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:35:59.274237+0000 mon.a (mon.0) 174 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:00.242971+0000 mon.a (mon.0) 175 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:01.061958+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:06.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:01.243094+0000 mon.a (mon.0) 176 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:02.243287+0000 mon.a (mon.0) 177 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:03.062353+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:03.243301+0000 mon.a (mon.0) 178 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:04.243455+0000 mon.a (mon.0) 179 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.273227+0000 mon.a (mon.0) 180 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279263+0000 mon.a (mon.0) 181 : cluster 0 monmap epoch 3 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279302+0000 mon.a (mon.0) 182 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:06.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279322+0000 mon.a (mon.0) 183 : cluster 0 last_changed 2024-04-02T04:35:59.243766+0000 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279344+0000 mon.a (mon.0) 184 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279368+0000 mon.a (mon.0) 185 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279393+0000 mon.a (mon.0) 186 : cluster 0 election_strategy: 1 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279419+0000 mon.a (mon.0) 187 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279439+0000 mon.a (mon.0) 188 : cluster 0 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.279462+0000 mon.a (mon.0) 189 : cluster 0 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.300731+0000 mon.a (mon.0) 190 : cluster 0 fsmap 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.300816+0000 mon.a (mon.0) 191 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.301108+0000 mon.a (mon.0) 192 : cluster 0 mgrmap e13: y(active, since 77s) 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.301416+0000 mon.a (mon.0) 193 : cluster 3 Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.308436+0000 mon.a (mon.0) 194 : cluster 3 Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-02T04:36:06.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:04.308536+0000 mon.a (mon.0) 196 : cluster 3 mon.c (rank 2) addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] is down (out of quorum) 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:04.313494+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:04.321344+0000 mon.a (mon.0) 198 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:04.389652+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:04.739941+0000 mon.a (mon.0) 200 : audit 0 from='client.? 172.21.15.98:0/1966773037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: audit 2024-04-02T04:36:05.243788+0000 mon.a (mon.0) 201 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:01.245425+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.265898+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.266854+0000 mon.a (mon.0) 203 : cluster 1 mon.a calling monitor election 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.267426+0000 mon.b (mon.1) 3 : cluster 1 mon.b calling monitor election 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.290996+0000 mon.a (mon.0) 204 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298276+0000 mon.a (mon.0) 205 : cluster 0 monmap epoch 3 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298315+0000 mon.a (mon.0) 206 : cluster 0 fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298344+0000 mon.a (mon.0) 207 : cluster 0 last_changed 2024-04-02T04:35:59.243766+0000 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298367+0000 mon.a (mon.0) 208 : cluster 0 created 2024-04-02T04:33:39.262476+0000 2024-04-02T04:36:06.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298408+0000 mon.a (mon.0) 209 : cluster 0 min_mon_release 19 (squid) 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298444+0000 mon.a (mon.0) 210 : cluster 0 election_strategy: 1 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298483+0000 mon.a (mon.0) 211 : cluster 0 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298512+0000 mon.a (mon.0) 212 : cluster 0 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.298542+0000 mon.a (mon.0) 213 : cluster 0 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.299635+0000 mon.a (mon.0) 214 : cluster 0 fsmap 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.299704+0000 mon.a (mon.0) 215 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.300042+0000 mon.a (mon.0) 216 : cluster 0 mgrmap e13: y(active, since 79s) 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.300335+0000 mon.a (mon.0) 217 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.300381+0000 mon.a (mon.0) 218 : cluster 1 Cluster is now healthy 2024-04-02T04:36:06.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:06 smithi069 bash[28904]: cluster 2024-04-02T04:36:06.308986+0000 mon.a (mon.0) 219 : cluster 1 overall HEALTH_OK 2024-04-02T04:36:08.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:08 smithi098 bash[23790]: cluster 2024-04-02T04:36:07.063342+0000 mgr.y (mgr.14152) 56 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:08.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:08 smithi098 bash[23790]: audit 2024-04-02T04:36:07.125874+0000 mon.a (mon.0) 220 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:08.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:08 smithi098 bash[23790]: audit 2024-04-02T04:36:07.244166+0000 mon.a (mon.0) 221 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:08.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:08 smithi069 bash[21528]: cluster 2024-04-02T04:36:07.063342+0000 mgr.y (mgr.14152) 56 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:08.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:08 smithi069 bash[21528]: audit 2024-04-02T04:36:07.125874+0000 mon.a (mon.0) 220 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:08.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:08 smithi069 bash[21528]: audit 2024-04-02T04:36:07.244166+0000 mon.a (mon.0) 221 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:08.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:08 smithi069 bash[28904]: cluster 2024-04-02T04:36:07.063342+0000 mgr.y (mgr.14152) 56 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:08.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:08 smithi069 bash[28904]: audit 2024-04-02T04:36:07.125874+0000 mon.a (mon.0) 220 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:08.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:08 smithi069 bash[28904]: audit 2024-04-02T04:36:07.244166+0000 mon.a (mon.0) 221 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:36:09.380 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:36:09.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:09 smithi069 bash[21528]: audit 2024-04-02T04:36:08.628367+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:09.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:09 smithi069 bash[21528]: audit 2024-04-02T04:36:08.635789+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:09.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:09 smithi069 bash[28904]: audit 2024-04-02T04:36:08.628367+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:09.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:09 smithi069 bash[28904]: audit 2024-04-02T04:36:08.635789+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:09 smithi098 bash[23790]: audit 2024-04-02T04:36:08.628367+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:09 smithi098 bash[23790]: audit 2024-04-02T04:36:08.635789+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: cluster 2024-04-02T04:36:09.063790+0000 mgr.y (mgr.14152) 57 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:10.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:09.678968+0000 mon.a (mon.0) 224 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:09.682997+0000 mon.a (mon.0) 225 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.688 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.265235+0000 mon.a (mon.0) 226 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:10.688 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.265993+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:10.688 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.604377+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.688 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.612183+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: cluster 2024-04-02T04:36:09.063790+0000 mgr.y (mgr.14152) 57 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:09.678968+0000 mon.a (mon.0) 224 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:09.682997+0000 mon.a (mon.0) 225 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.265235+0000 mon.a (mon.0) 226 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.265993+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.604377+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.612183+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.653291+0000 mon.a (mon.0) 230 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.662055+0000 mon.a (mon.0) 231 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[21528]: audit 2024-04-02T04:36:10.668099+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: cluster 2024-04-02T04:36:09.063790+0000 mgr.y (mgr.14152) 57 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:10.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:09.678968+0000 mon.a (mon.0) 224 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:09.682997+0000 mon.a (mon.0) 225 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.265235+0000 mon.a (mon.0) 226 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.265993+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.604377+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.612183+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.653291+0000 mon.a (mon.0) 230 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.662055+0000 mon.a (mon.0) 231 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:10 smithi069 bash[28904]: audit 2024-04-02T04:36:10.668099+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.653291+0000 mon.a (mon.0) 230 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.662055+0000 mon.a (mon.0) 231 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:10 smithi098 bash[23790]: audit 2024-04-02T04:36:10.668099+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: cephadm 2024-04-02T04:36:10.267228+0000 mgr.y (mgr.14152) 58 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: cephadm 2024-04-02T04:36:10.267392+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: cephadm 2024-04-02T04:36:10.431986+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: cephadm 2024-04-02T04:36:10.432121+0000 mgr.y (mgr.14152) 61 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.720910+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.727120+0000 mon.a (mon.0) 234 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.733531+0000 mon.a (mon.0) 235 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.739536+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: cephadm 2024-04-02T04:36:10.740128+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.740667+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.742073+0000 mon.a (mon.0) 238 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: audit 2024-04-02T04:36:10.743347+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[21528]: cephadm 2024-04-02T04:36:10.744620+0000 mgr.y (mgr.14152) 63 : cephadm 1 Reconfiguring daemon mon.a on smithi069 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: cephadm 2024-04-02T04:36:10.267228+0000 mgr.y (mgr.14152) 58 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: cephadm 2024-04-02T04:36:10.267392+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: cephadm 2024-04-02T04:36:10.431986+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: cephadm 2024-04-02T04:36:10.432121+0000 mgr.y (mgr.14152) 61 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.720910+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.727120+0000 mon.a (mon.0) 234 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.733531+0000 mon.a (mon.0) 235 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.739536+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: cephadm 2024-04-02T04:36:10.740128+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.740667+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.742073+0000 mon.a (mon.0) 238 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: audit 2024-04-02T04:36:10.743347+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:12.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:11 smithi069 bash[28904]: cephadm 2024-04-02T04:36:10.744620+0000 mgr.y (mgr.14152) 63 : cephadm 1 Reconfiguring daemon mon.a on smithi069 2024-04-02T04:36:12.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/3547096711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e8100db0 msgr2=0x7fe6e8101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:12.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/3547096711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e8100db0 0x7fe6e8101190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000b0a0 tx=0x7fe6e000b3b0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/3547096711 shutdown_connections 2024-04-02T04:36:12.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/3547096711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e8100db0 0x7fe6e8101190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/3547096711 >> 172.21.15.69:0/3547096711 conn(0x7fe6e80fc680 msgr2=0x7fe6e80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/3547096711 shutdown_connections 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/3547096711 wait complete. 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 librados: starting msgr at 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 librados: starting objecter 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 Processor -- start 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- start start 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 librados: setting wanted keys 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 librados: calling monclient init 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e8100db0 0x7fe6e8196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 0x7fe6e819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e819bd60 0x7fe6e819c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e810a5f0 con 0x7fe6e819bd60 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e810a470 con 0x7fe6e8100db0 2024-04-02T04:36:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e810a770 con 0x7fe6e8197490 2024-04-02T04:36:12.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 0x7fe6e819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:12.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 0x7fe6e819b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:39428/0 (socket says 172.21.15.69:39428) 2024-04-02T04:36:12.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 -- 172.21.15.69:0/1944077637 learned_addr learned my addr 172.21.15.69:0/1944077637 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:36:12.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e6d9d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e819bd60 0x7fe6e819c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:12.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 -- 172.21.15.69:0/1944077637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e8100db0 msgr2=0x7fe6e8196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:12.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e8100db0 0x7fe6e8196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 -- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e819bd60 msgr2=0x7fe6e819c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e819bd60 0x7fe6e819c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 -- 172.21.15.69:0/1944077637 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e81a3ad0 con 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6e5d9b700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 0x7fe6e819b820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b780 tx=0x7fe6d000ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6df7fe700 10 client.?.objecter ms_handle_connect 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6df7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0004230 con 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe6d00043d0 con 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0011a60 con 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e81a3d00 con 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 10 client.24101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e81a4230 con 0x7fe6e8197490 2024-04-02T04:36:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.210+0000 7fe6ec94d700 1 librados: init done 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fe6d0011c00 con 0x7fe6e8197490 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6e8063870 con 0x7fe6e8197490 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fe6d403acf0 0x7fe6d403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe6d004ba40 con 0x7fe6e8197490 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 10 client.24101.objecter ms_dispatch 0x7fe6e8001550 osd_map(4..4 src has 1..4) 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 3 client.24101.objecter handle_osd_map got epochs [4,4] > 0 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 3 client.24101.objecter handle_osd_map decoding full epoch 4 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 20 client.24101.objecter dump_active .. 0 homeless 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6e659c700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fe6d403acf0 0x7fe6d403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6e659c700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fe6d403acf0 0x7fe6d403d1b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009e60 tx=0x7fe6e00074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:12.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.214+0000 7fe6df7fe700 10 client.24101.objecter ms_handle_connect 0x7fe6d403acf0 2024-04-02T04:36:12.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.218+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe6d00202a0 con 0x7fe6e8197490 2024-04-02T04:36:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: cephadm 2024-04-02T04:36:10.267228+0000 mgr.y (mgr.14152) 58 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:36:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: cephadm 2024-04-02T04:36:10.267392+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:36:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: cephadm 2024-04-02T04:36:10.431986+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:36:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: cephadm 2024-04-02T04:36:10.432121+0000 mgr.y (mgr.14152) 61 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:36:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.720910+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.727120+0000 mon.a (mon.0) 234 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.733531+0000 mon.a (mon.0) 235 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.739536+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: cephadm 2024-04-02T04:36:10.740128+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.740667+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.742073+0000 mon.a (mon.0) 238 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: audit 2024-04-02T04:36:10.743347+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:11 smithi098 bash[23790]: cephadm 2024-04-02T04:36:10.744620+0000 mgr.y (mgr.14152) 63 : cephadm 1 Reconfiguring daemon mon.a on smithi069 2024-04-02T04:36:12.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.418+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe6e81982b0 con 0x7fe6e8197490 2024-04-02T04:36:12.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.418+0000 7fe6df7fe700 1 -- 172.21.15.69:0/1944077637 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7fe6e81982b0 con 0x7fe6e8197490 2024-04-02T04:36:12.425 INFO:teuthology.orchestra.run.smithi069.stdout:# minimal ceph.conf for f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:12.425 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-02T04:36:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:36:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: mon_host = [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 10 librados: watch_flush enter 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 10 librados: watch_flush exit 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 20 client.24101.objecter shutdown clearing up homeless session... 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 10 client.24101.objecter successfully canceled tick 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fe6d403acf0 msgr2=0x7fe6d403d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fe6d403acf0 0x7fe6d403d1b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009e60 tx=0x7fe6e00074e0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 msgr2=0x7fe6e819b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 0x7fe6e819b820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b780 tx=0x7fe6d000ba90 comp rx=0 tx=0).stop 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 shutdown_connections 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6e819bd60 0x7fe6e819c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e8100db0 0x7fe6e8196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fe6d403acf0 0x7fe6d403d1b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 --2- 172.21.15.69:0/1944077637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6e8197490 0x7fe6e819b820 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:12.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 >> 172.21.15.69:0/1944077637 conn(0x7fe6e80fc680 msgr2=0x7fe6e806a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:12.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 shutdown_connections 2024-04-02T04:36:12.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 -- 172.21.15.69:0/1944077637 wait complete. 2024-04-02T04:36:12.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:12.422+0000 7fe6ec94d700 1 librados: shutdown 2024-04-02T04:36:13.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:12 smithi069 bash[21528]: cluster 2024-04-02T04:36:11.064181+0000 mgr.y (mgr.14152) 64 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:13.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:12 smithi069 bash[21528]: audit 2024-04-02T04:36:12.422704+0000 mon.c (mon.2) 3 : audit 0 from='client.? 172.21.15.69:0/1944077637' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:13.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:12 smithi069 bash[28904]: cluster 2024-04-02T04:36:11.064181+0000 mgr.y (mgr.14152) 64 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:13.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:12 smithi069 bash[28904]: audit 2024-04-02T04:36:12.422704+0000 mon.c (mon.2) 3 : audit 0 from='client.? 172.21.15.69:0/1944077637' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:12 smithi098 bash[23790]: cluster 2024-04-02T04:36:11.064181+0000 mgr.y (mgr.14152) 64 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:12 smithi098 bash[23790]: audit 2024-04-02T04:36:12.422704+0000 mon.c (mon.2) 3 : audit 0 from='client.? 172.21.15.69:0/1944077637' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:13.273 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-02T04:36:13.273 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:36:13.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T04:36:13.286 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:36:13.286 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:36:13.341 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:36:13.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T04:36:13.356 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:36:13.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:36:13.410 INFO:tasks.cephadm:Adding mgr.y on smithi069 2024-04-02T04:36:13.410 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-04-02T04:36:13.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '2;smithi069=y;smithi098=x' 2024-04-02T04:36:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: audit 2024-04-02T04:36:12.911723+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: audit 2024-04-02T04:36:12.920295+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: cephadm 2024-04-02T04:36:12.963437+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-04-02T04:36:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: audit 2024-04-02T04:36:12.964052+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: audit 2024-04-02T04:36:12.965351+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:14.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: audit 2024-04-02T04:36:12.966570+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:14.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:13 smithi098 bash[23790]: cephadm 2024-04-02T04:36:12.968039+0000 mgr.y (mgr.14152) 66 : cephadm 1 Reconfiguring daemon mon.c on smithi069 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: audit 2024-04-02T04:36:12.911723+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: audit 2024-04-02T04:36:12.920295+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: cephadm 2024-04-02T04:36:12.963437+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: audit 2024-04-02T04:36:12.964052+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: audit 2024-04-02T04:36:12.965351+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: audit 2024-04-02T04:36:12.966570+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[21528]: cephadm 2024-04-02T04:36:12.968039+0000 mgr.y (mgr.14152) 66 : cephadm 1 Reconfiguring daemon mon.c on smithi069 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: audit 2024-04-02T04:36:12.911723+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: audit 2024-04-02T04:36:12.920295+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: cephadm 2024-04-02T04:36:12.963437+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-04-02T04:36:14.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: audit 2024-04-02T04:36:12.964052+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:14.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: audit 2024-04-02T04:36:12.965351+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:14.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: audit 2024-04-02T04:36:12.966570+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:14.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:13 smithi069 bash[28904]: cephadm 2024-04-02T04:36:12.968039+0000 mgr.y (mgr.14152) 66 : cephadm 1 Reconfiguring daemon mon.c on smithi069 2024-04-02T04:36:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:14 smithi098 bash[23790]: cluster 2024-04-02T04:36:13.064594+0000 mgr.y (mgr.14152) 67 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:14 smithi098 bash[23790]: audit 2024-04-02T04:36:14.779222+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:14 smithi098 bash[23790]: audit 2024-04-02T04:36:14.788353+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:14 smithi098 bash[23790]: audit 2024-04-02T04:36:14.838443+0000 mon.a (mon.0) 247 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:14 smithi098 bash[23790]: audit 2024-04-02T04:36:14.840373+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:14 smithi098 bash[23790]: audit 2024-04-02T04:36:14.842119+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[21528]: cluster 2024-04-02T04:36:13.064594+0000 mgr.y (mgr.14152) 67 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[21528]: audit 2024-04-02T04:36:14.779222+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[21528]: audit 2024-04-02T04:36:14.788353+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[21528]: audit 2024-04-02T04:36:14.838443+0000 mon.a (mon.0) 247 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[21528]: audit 2024-04-02T04:36:14.840373+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[21528]: audit 2024-04-02T04:36:14.842119+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[28904]: cluster 2024-04-02T04:36:13.064594+0000 mgr.y (mgr.14152) 67 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:15.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[28904]: audit 2024-04-02T04:36:14.779222+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:15.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[28904]: audit 2024-04-02T04:36:14.788353+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:15.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[28904]: audit 2024-04-02T04:36:14.838443+0000 mon.a (mon.0) 247 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:36:15.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[28904]: audit 2024-04-02T04:36:14.840373+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:36:15.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:14 smithi069 bash[28904]: audit 2024-04-02T04:36:14.842119+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:16.218 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:15 smithi098 bash[23790]: cephadm 2024-04-02T04:36:14.837731+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-04-02T04:36:16.219 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:15 smithi098 bash[23790]: cephadm 2024-04-02T04:36:14.844182+0000 mgr.y (mgr.14152) 69 : cephadm 1 Reconfiguring daemon mon.b on smithi098 2024-04-02T04:36:16.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:15 smithi069 bash[21528]: cephadm 2024-04-02T04:36:14.837731+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-04-02T04:36:16.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:15 smithi069 bash[21528]: cephadm 2024-04-02T04:36:14.844182+0000 mgr.y (mgr.14152) 69 : cephadm 1 Reconfiguring daemon mon.b on smithi098 2024-04-02T04:36:16.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:15 smithi069 bash[28904]: cephadm 2024-04-02T04:36:14.837731+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-04-02T04:36:16.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:15 smithi069 bash[28904]: cephadm 2024-04-02T04:36:14.844182+0000 mgr.y (mgr.14152) 69 : cephadm 1 Reconfiguring daemon mon.b on smithi098 2024-04-02T04:36:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: cluster 2024-04-02T04:36:15.065020+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: audit 2024-04-02T04:36:16.851431+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: audit 2024-04-02T04:36:16.858717+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: audit 2024-04-02T04:36:16.864878+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:17.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: audit 2024-04-02T04:36:16.868609+0000 mon.a (mon.0) 253 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:17.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: audit 2024-04-02T04:36:16.870566+0000 mon.a (mon.0) 254 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:17.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:16 smithi098 bash[23790]: audit 2024-04-02T04:36:16.881653+0000 mon.a (mon.0) 255 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: cluster 2024-04-02T04:36:15.065020+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: audit 2024-04-02T04:36:16.851431+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: audit 2024-04-02T04:36:16.858717+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: audit 2024-04-02T04:36:16.864878+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: audit 2024-04-02T04:36:16.868609+0000 mon.a (mon.0) 253 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: audit 2024-04-02T04:36:16.870566+0000 mon.a (mon.0) 254 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:17.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[21528]: audit 2024-04-02T04:36:16.881653+0000 mon.a (mon.0) 255 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: cluster 2024-04-02T04:36:15.065020+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: audit 2024-04-02T04:36:16.851431+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: audit 2024-04-02T04:36:16.858717+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: audit 2024-04-02T04:36:16.864878+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: audit 2024-04-02T04:36:16.868609+0000 mon.a (mon.0) 253 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: audit 2024-04-02T04:36:16.870566+0000 mon.a (mon.0) 254 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:17.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:16 smithi069 bash[28904]: audit 2024-04-02T04:36:16.881653+0000 mon.a (mon.0) 255 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:36:19.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:18 smithi098 bash[23790]: cluster 2024-04-02T04:36:17.065434+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:19.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:18 smithi069 bash[21528]: cluster 2024-04-02T04:36:17.065434+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:19.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:18 smithi069 bash[28904]: cluster 2024-04-02T04:36:17.065434+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- 172.21.15.98:0/2275989465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04074ca0 msgr2=0x7ffa04075120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/2275989465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04074ca0 0x7ffa04075120 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009990 tx=0x7ff9f4009ca0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- 172.21.15.98:0/2275989465 shutdown_connections 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/2275989465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa0410a1a0 0x7ffa0410c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/2275989465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa04076270 0x7ffa040746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/2275989465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04074ca0 0x7ffa04075120 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- 172.21.15.98:0/2275989465 >> 172.21.15.98:0/2275989465 conn(0x7ffa040fd870 msgr2=0x7ffa040ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- 172.21.15.98:0/2275989465 shutdown_connections 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- 172.21.15.98:0/2275989465 wait complete. 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 librados: starting msgr at 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 librados: starting objecter 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 Processor -- start 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- start start 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 librados: setting wanted keys 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 librados: calling monclient init 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa04074ca0 0x7ffa04199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa04076270 0x7ffa0419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 0x7ffa041a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410f590 con 0x7ffa04074ca0 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410f410 con 0x7ffa0410a1a0 2024-04-02T04:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0adcd700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0410f710 con 0x7ffa04076270 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 0x7ffa041a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 0x7ffa041a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52224/0 (socket says 172.21.15.98:52224) 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 -- 172.21.15.98:0/1604192482 learned_addr learned my addr 172.21.15.98:0/1604192482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa03fff700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa04076270 0x7ffa0419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa08b69700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa04074ca0 0x7ffa04199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 -- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa04076270 msgr2=0x7ffa0419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa04076270 0x7ffa0419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 -- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa04074ca0 msgr2=0x7ffa04199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa04074ca0 0x7ffa04199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 -- 172.21.15.98:0/1604192482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa041a1990 con 0x7ffa0410a1a0 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa0936a700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 0x7ffa041a12c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00ef30 tx=0x7ff9fc00c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa01ffb700 10 client.?.objecter ms_handle_connect 0x7ffa0410a1a0 2024-04-02T04:36:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.645+0000 7ffa01ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0092e0 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff9fc004bb0 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa041a1c20 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc004280 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa0adcd700 10 client.24103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a2160 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa0adcd700 1 librados: init done 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7ff9fc01b030 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0404ff00 con 0x7ffa0410a1a0 2024-04-02T04:36:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7ff9ec03aa60 0x7ff9ec03cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff9fc04bc90 con 0x7ffa0410a1a0 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 10 client.24103.objecter ms_dispatch 0x7ffa04001550 osd_map(4..4 src has 1..4) 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 3 client.24103.objecter handle_osd_map got epochs [4,4] > 0 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 3 client.24103.objecter handle_osd_map decoding full epoch 4 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 20 client.24103.objecter dump_active .. 0 homeless 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa08b69700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7ff9ec03aa60 0x7ff9ec03cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa01ffb700 10 client.24103.objecter ms_handle_connect 0x7ff9ec03aa60 2024-04-02T04:36:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.649+0000 7ffa08b69700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7ff9ec03aa60 0x7ff9ec03cf20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffa0419b220 tx=0x7ff9f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:19.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.653+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff9fc022020 con 0x7ffa0410a1a0 2024-04-02T04:36:19.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.873+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi098=x", "target": ["mon-mgr", ""]}) -- 0x7ffa04062680 con 0x7ff9ec03aa60 2024-04-02T04:36:19.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.889+0000 7ffa01ffb700 1 -- 172.21.15.98:0/1604192482 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ffa04062680 con 0x7ff9ec03aa60 2024-04-02T04:36:19.897 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mgr update... 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 10 librados: watch_flush enter 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 10 librados: watch_flush exit 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 20 client.24103.objecter shutdown clearing up homeless session... 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 10 client.24103.objecter successfully canceled tick 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7ff9ec03aa60 msgr2=0x7ff9ec03cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7ff9ec03aa60 0x7ff9ec03cf20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffa0419b220 tx=0x7ff9f8009260 comp rx=0 tx=0).stop 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 msgr2=0x7ffa041a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:19.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 0x7ffa041a12c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00ef30 tx=0x7ff9fc00c450 comp rx=0 tx=0).stop 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 shutdown_connections 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa0410a1a0 0x7ffa041a12c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa04074ca0 0x7ffa04199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7ff9ec03aa60 0x7ff9ec03cf20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 --2- 172.21.15.98:0/1604192482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa04076270 0x7ffa0419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 >> 172.21.15.98:0/1604192482 conn(0x7ffa040fd870 msgr2=0x7ffa040ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 shutdown_connections 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 -- 172.21.15.98:0/1604192482 wait complete. 2024-04-02T04:36:19.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:36:19.893+0000 7ffa0adcd700 1 librados: shutdown 2024-04-02T04:36:20.781 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.x.service 2024-04-02T04:36:20.784 INFO:tasks.cephadm:Deploying OSDs... 2024-04-02T04:36:20.784 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:36:20.784 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T04:36:20.791 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T04:36:20.791 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 04:26:57.201133167 +0000 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 04:26:56.949135418 +0000 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 04:26:56.949135418 +0000 2024-04-02T04:36:20.843 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2024-04-02T04:36:20.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T04:36:20.898 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T04:36:20.898 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T04:36:20.898 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000304849 s, 1.7 MB/s 2024-04-02T04:36:20.899 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T04:36:20.948 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 04:26:57.669128986 +0000 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 04:26:57.425131165 +0000 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 04:26:57.425131165 +0000 2024-04-02T04:36:20.994 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2024-04-02T04:36:20.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T04:36:21.049 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T04:36:21.049 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T04:36:21.049 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.00042387 s, 1.2 MB/s 2024-04-02T04:36:21.050 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T04:36:21.100 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-04-02T04:36:21.146 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T04:36:21.146 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:21.146 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-04-02T04:36:21.146 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:21.146 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 04:26:58.153124663 +0000 2024-04-02T04:36:21.147 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 04:26:57.893126985 +0000 2024-04-02T04:36:21.147 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 04:26:57.893126985 +0000 2024-04-02T04:36:21.147 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2024-04-02T04:36:21.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T04:36:21.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: cluster 2024-04-02T04:36:19.065844+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:21.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.881114+0000 mgr.y (mgr.14152) 73 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi098=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:21.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: cephadm 2024-04-02T04:36:19.884750+0000 mgr.y (mgr.14152) 74 : cephadm 1 Saving service mgr spec with placement smithi069=y;smithi098=x;count:2 2024-04-02T04:36:21.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.893290+0000 mon.a (mon.0) 256 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:21.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.922808+0000 mon.a (mon.0) 257 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.926643+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.928683+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.938037+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.941362+0000 mon.a (mon.0) 261 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.944185+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.949453+0000 mon.a (mon.0) 263 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: audit 2024-04-02T04:36:19.955243+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[28904]: cephadm 2024-04-02T04:36:19.957222+0000 mgr.y (mgr.14152) 75 : cephadm 1 Deploying daemon mgr.x on smithi098 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: cluster 2024-04-02T04:36:19.065844+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.881114+0000 mgr.y (mgr.14152) 73 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi098=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: cephadm 2024-04-02T04:36:19.884750+0000 mgr.y (mgr.14152) 74 : cephadm 1 Saving service mgr spec with placement smithi069=y;smithi098=x;count:2 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.893290+0000 mon.a (mon.0) 256 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.922808+0000 mon.a (mon.0) 257 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.926643+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.928683+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.938037+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.941362+0000 mon.a (mon.0) 261 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.944185+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.949453+0000 mon.a (mon.0) 263 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: audit 2024-04-02T04:36:19.955243+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:21.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:20 smithi069 bash[21528]: cephadm 2024-04-02T04:36:19.957222+0000 mgr.y (mgr.14152) 75 : cephadm 1 Deploying daemon mgr.x on smithi098 2024-04-02T04:36:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T04:36:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T04:36:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000450354 s, 1.1 MB/s 2024-04-02T04:36:21.202 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: cluster 2024-04-02T04:36:19.065844+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.881114+0000 mgr.y (mgr.14152) 73 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi098=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: cephadm 2024-04-02T04:36:19.884750+0000 mgr.y (mgr.14152) 74 : cephadm 1 Saving service mgr spec with placement smithi069=y;smithi098=x;count:2 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.893290+0000 mon.a (mon.0) 256 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.922808+0000 mon.a (mon.0) 257 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.926643+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.928683+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:21.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.938037+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:21.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.941362+0000 mon.a (mon.0) 261 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:36:21.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.944185+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T04:36:21.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.949453+0000 mon.a (mon.0) 263 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:36:21.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: audit 2024-04-02T04:36:19.955243+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:21.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:20 smithi098 bash[23790]: cephadm 2024-04-02T04:36:19.957222+0000 mgr.y (mgr.14152) 75 : cephadm 1 Deploying daemon mgr.x on smithi098 2024-04-02T04:36:21.252 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-04-02T04:36:21.298 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 944 Links: 1 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-02 04:26:58.653120196 +0000 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-02 04:26:58.377122662 +0000 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-02 04:26:58.377122662 +0000 2024-04-02T04:36:21.299 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2024-04-02T04:36:21.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T04:36:21.353 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-02T04:36:21.353 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-02T04:36:21.353 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000316777 s, 1.6 MB/s 2024-04-02T04:36:21.354 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T04:36:21.404 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:36:21.404 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T04:36:21.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T04:36:21.409 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-04-02T04:36:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T04:36:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:21.457 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-04-02T04:36:21.458 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:21.458 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-02 04:26:57.165205862 +0000 2024-04-02T04:36:21.458 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-02 04:26:56.913208114 +0000 2024-04-02T04:36:21.458 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-02 04:26:56.913208114 +0000 2024-04-02T04:36:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-02T04:36:21.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T04:36:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-02T04:36:21.510 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-02T04:36:21.510 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000291826 s, 1.8 MB/s 2024-04-02T04:36:21.511 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T04:36:21.559 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-04-02T04:36:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T04:36:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:21.612 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-04-02T04:36:21.612 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:21.613 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-02 04:26:57.393203827 +0000 2024-04-02T04:36:21.613 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-02 04:26:57.389203863 +0000 2024-04-02T04:36:21.613 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-02 04:26:57.389203863 +0000 2024-04-02T04:36:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-02T04:36:21.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T04:36:21.666 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-02T04:36:21.666 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-02T04:36:21.666 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000336789 s, 1.5 MB/s 2024-04-02T04:36:21.667 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T04:36:21.715 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-04-02T04:36:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T04:36:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:21.763 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-02T04:36:21.763 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:21.763 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-02 04:26:58.129197252 +0000 2024-04-02T04:36:21.763 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-02 04:26:57.881199467 +0000 2024-04-02T04:36:21.763 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-02 04:26:57.881199467 +0000 2024-04-02T04:36:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-02T04:36:21.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T04:36:21.817 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-02T04:36:21.817 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-02T04:36:21.817 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000423421 s, 1.2 MB/s 2024-04-02T04:36:21.818 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T04:36:21.867 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-04-02T04:36:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T04:36:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:36:21.914 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 940 Links: 1 2024-04-02T04:36:21.914 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:36:21.915 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-02 04:26:58.633192751 +0000 2024-04-02T04:36:21.915 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-02 04:26:58.365195144 +0000 2024-04-02T04:36:21.915 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-02 04:26:58.365195144 +0000 2024-04-02T04:36:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-02T04:36:21.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T04:36:21.969 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-02T04:36:21.969 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-02T04:36:21.969 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000392397 s, 1.3 MB/s 2024-04-02T04:36:21.970 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T04:36:22.019 INFO:tasks.cephadm:Deploying osd.0 on smithi069 with /dev/vg_nvme/lv_4... 2024-04-02T04:36:22.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T04:36:22.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:36:22.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:36:22.784 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:36:22.785 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:36:22.785 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:36:22.785 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:36:22.785 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:22 smithi098 systemd[1]: Started Ceph mgr.x for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:36:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 bash[23790]: cluster 2024-04-02T04:36:21.066198+0000 mgr.y (mgr.14152) 76 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 bash[23790]: audit 2024-04-02T04:36:22.789474+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 bash[23790]: audit 2024-04-02T04:36:22.798926+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 bash[23790]: audit 2024-04-02T04:36:22.837193+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:22 smithi098 bash[23790]: audit 2024-04-02T04:36:22.846515+0000 mon.a (mon.0) 268 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[28904]: cluster 2024-04-02T04:36:21.066198+0000 mgr.y (mgr.14152) 76 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[28904]: audit 2024-04-02T04:36:22.789474+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[28904]: audit 2024-04-02T04:36:22.798926+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[28904]: audit 2024-04-02T04:36:22.837193+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[28904]: audit 2024-04-02T04:36:22.846515+0000 mon.a (mon.0) 268 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[21528]: cluster 2024-04-02T04:36:21.066198+0000 mgr.y (mgr.14152) 76 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[21528]: audit 2024-04-02T04:36:22.789474+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[21528]: audit 2024-04-02T04:36:22.798926+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[21528]: audit 2024-04-02T04:36:22.837193+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:22 smithi069 bash[21528]: audit 2024-04-02T04:36:22.846515+0000 mon.a (mon.0) 268 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:23.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:23 smithi098 bash[23790]: audit 2024-04-02T04:36:22.914808+0000 mon.a (mon.0) 269 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:23.956 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:23 smithi098 bash[24743]: debug 2024-04-02T04:36:23.801+0000 7fd00b379700 1 -- 172.21.15.98:0/2215591639 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5d3d845a0 con 0x55a5d4b04800 2024-04-02T04:36:24.225 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:23 smithi098 bash[24743]: debug 2024-04-02T04:36:23.949+0000 7fd015581200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T04:36:24.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:23 smithi069 bash[21528]: audit 2024-04-02T04:36:22.914808+0000 mon.a (mon.0) 269 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:24.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:23 smithi069 bash[28904]: audit 2024-04-02T04:36:22.914808+0000 mon.a (mon.0) 269 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:24 smithi098 bash[23790]: cluster 2024-04-02T04:36:23.066581+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:25.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:24 smithi069 bash[21528]: cluster 2024-04-02T04:36:23.066581+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:24 smithi069 bash[28904]: cluster 2024-04-02T04:36:23.066581+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:25.853 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:25 smithi098 bash[24743]: debug 2024-04-02T04:36:25.505+0000 7fd015581200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T04:36:25.853 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:25 smithi098 bash[24743]: debug 2024-04-02T04:36:25.601+0000 7fd015581200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T04:36:26.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:25 smithi098 bash[24743]: debug 2024-04-02T04:36:25.845+0000 7fd015581200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T04:36:26.741 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:36:26.913 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:26 smithi098 bash[24743]: debug 2024-04-02T04:36:26.637+0000 7fd015581200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T04:36:26.914 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:26 smithi098 bash[24743]: debug 2024-04-02T04:36:26.741+0000 7fd015581200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T04:36:26.914 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:26 smithi098 bash[24743]: debug 2024-04-02T04:36:26.841+0000 7fd015581200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T04:36:27.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:26 smithi069 bash[21528]: cluster 2024-04-02T04:36:25.066998+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:27.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:26 smithi069 bash[28904]: cluster 2024-04-02T04:36:25.066998+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:27.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:26 smithi098 bash[23790]: cluster 2024-04-02T04:36:25.066998+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:27.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:27 smithi098 bash[24743]: debug 2024-04-02T04:36:27.193+0000 7fd015581200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T04:36:27.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:27 smithi098 bash[24743]: debug 2024-04-02T04:36:27.305+0000 7fd015581200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T04:36:27.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:27 smithi098 bash[24743]: debug 2024-04-02T04:36:27.405+0000 7fd015581200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T04:36:28.136 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:27 smithi098 bash[24743]: debug 2024-04-02T04:36:27.757+0000 7fd015581200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T04:36:28.136 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[24743]: debug 2024-04-02T04:36:27.997+0000 7fd015581200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[24743]: debug 2024-04-02T04:36:28.261+0000 7fd015581200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[24743]: debug 2024-04-02T04:36:28.373+0000 7fd015581200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: cluster 2024-04-02T04:36:27.067377+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: audit 2024-04-02T04:36:27.132787+0000 mon.a (mon.0) 270 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: audit 2024-04-02T04:36:28.104677+0000 mon.a (mon.0) 271 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: audit 2024-04-02T04:36:28.111851+0000 mon.a (mon.0) 272 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: audit 2024-04-02T04:36:28.113086+0000 mon.a (mon.0) 273 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: audit 2024-04-02T04:36:28.114117+0000 mon.a (mon.0) 274 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[23790]: audit 2024-04-02T04:36:28.119949+0000 mon.a (mon.0) 275 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: cluster 2024-04-02T04:36:27.067377+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:28.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: audit 2024-04-02T04:36:27.132787+0000 mon.a (mon.0) 270 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: audit 2024-04-02T04:36:28.104677+0000 mon.a (mon.0) 271 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: audit 2024-04-02T04:36:28.111851+0000 mon.a (mon.0) 272 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: audit 2024-04-02T04:36:28.113086+0000 mon.a (mon.0) 273 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: audit 2024-04-02T04:36:28.114117+0000 mon.a (mon.0) 274 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[21528]: audit 2024-04-02T04:36:28.119949+0000 mon.a (mon.0) 275 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: cluster 2024-04-02T04:36:27.067377+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: audit 2024-04-02T04:36:27.132787+0000 mon.a (mon.0) 270 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: audit 2024-04-02T04:36:28.104677+0000 mon.a (mon.0) 271 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: audit 2024-04-02T04:36:28.111851+0000 mon.a (mon.0) 272 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: audit 2024-04-02T04:36:28.113086+0000 mon.a (mon.0) 273 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: audit 2024-04-02T04:36:28.114117+0000 mon.a (mon.0) 274 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:28 smithi069 bash[28904]: audit 2024-04-02T04:36:28.119949+0000 mon.a (mon.0) 275 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:28.975 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:28 smithi098 bash[24743]: debug 2024-04-02T04:36:28.641+0000 7fd015581200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T04:36:29.408 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[24743]: debug 2024-04-02T04:36:29.049+0000 7fd015581200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:36:29.408 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[24743]: debug 2024-04-02T04:36:29.285+0000 7fd015581200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T04:36:29.408 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[24743]: debug 2024-04-02T04:36:29.401+0000 7fd015581200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T04:36:29.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[23790]: cephadm 2024-04-02T04:36:28.170317+0000 mgr.y (mgr.14152) 80 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-04-02T04:36:29.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[23790]: audit 2024-04-02T04:36:28.170781+0000 mon.a (mon.0) 276 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:36:29.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[23790]: audit 2024-04-02T04:36:28.171780+0000 mon.a (mon.0) 277 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:36:29.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[23790]: audit 2024-04-02T04:36:28.172611+0000 mon.a (mon.0) 278 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:29.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[23790]: cephadm 2024-04-02T04:36:28.173496+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring daemon mgr.y on smithi069 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[21528]: cephadm 2024-04-02T04:36:28.170317+0000 mgr.y (mgr.14152) 80 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[21528]: audit 2024-04-02T04:36:28.170781+0000 mon.a (mon.0) 276 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[21528]: audit 2024-04-02T04:36:28.171780+0000 mon.a (mon.0) 277 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[21528]: audit 2024-04-02T04:36:28.172611+0000 mon.a (mon.0) 278 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[21528]: cephadm 2024-04-02T04:36:28.173496+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring daemon mgr.y on smithi069 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[28904]: cephadm 2024-04-02T04:36:28.170317+0000 mgr.y (mgr.14152) 80 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[28904]: audit 2024-04-02T04:36:28.170781+0000 mon.a (mon.0) 276 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:36:29.418 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[28904]: audit 2024-04-02T04:36:28.171780+0000 mon.a (mon.0) 277 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:36:29.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[28904]: audit 2024-04-02T04:36:28.172611+0000 mon.a (mon.0) 278 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:29.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:29 smithi069 bash[28904]: cephadm 2024-04-02T04:36:28.173496+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring daemon mgr.y on smithi069 2024-04-02T04:36:29.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:29 smithi098 bash[24743]: debug 2024-04-02T04:36:29.641+0000 7fd015581200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:36:29.869 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T04:36:29.869 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T04:36:29.869 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-02T04:36:29.869 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-02T04:36:29.869 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117172 s, 895 MB/s 2024-04-02T04:36:29.869 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-02T04:36:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:30 smithi098 bash[23790]: cluster 2024-04-02T04:36:29.067764+0000 mgr.y (mgr.14152) 82 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:30 smithi098 bash[23790]: audit 2024-04-02T04:36:30.129825+0000 mon.a (mon.0) 279 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:30 smithi098 bash[23790]: audit 2024-04-02T04:36:30.137907+0000 mon.a (mon.0) 280 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:30.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:30 smithi069 bash[21528]: cluster 2024-04-02T04:36:29.067764+0000 mgr.y (mgr.14152) 82 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:30.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:30 smithi069 bash[21528]: audit 2024-04-02T04:36:30.129825+0000 mon.a (mon.0) 279 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:30.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:30 smithi069 bash[21528]: audit 2024-04-02T04:36:30.137907+0000 mon.a (mon.0) 280 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:30 smithi069 bash[28904]: cluster 2024-04-02T04:36:29.067764+0000 mgr.y (mgr.14152) 82 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:30 smithi069 bash[28904]: audit 2024-04-02T04:36:30.129825+0000 mon.a (mon.0) 279 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:30 smithi069 bash[28904]: audit 2024-04-02T04:36:30.137907+0000 mon.a (mon.0) 280 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:30.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi069:vg_nvme/lv_4 2024-04-02T04:36:30.974 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:30 smithi098 bash[24743]: debug 2024-04-02T04:36:30.649+0000 7fd015581200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T04:36:30.975 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:30 smithi098 bash[24743]: debug 2024-04-02T04:36:30.765+0000 7fd015581200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T04:36:31.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[24743]: debug 2024-04-02T04:36:31.053+0000 7fd015581200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T04:36:31.475 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[24743]: debug 2024-04-02T04:36:31.165+0000 7fd015581200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T04:36:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[23790]: audit 2024-04-02T04:36:30.142366+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[23790]: audit 2024-04-02T04:36:30.766918+0000 mon.a (mon.0) 282 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[23790]: audit 2024-04-02T04:36:30.767828+0000 mon.a (mon.0) 283 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[23790]: audit 2024-04-02T04:36:30.772428+0000 mon.a (mon.0) 284 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:31.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[28904]: audit 2024-04-02T04:36:30.142366+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:31.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[28904]: audit 2024-04-02T04:36:30.766918+0000 mon.a (mon.0) 282 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:31.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[28904]: audit 2024-04-02T04:36:30.767828+0000 mon.a (mon.0) 283 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:31.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[28904]: audit 2024-04-02T04:36:30.772428+0000 mon.a (mon.0) 284 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:31.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[21528]: audit 2024-04-02T04:36:30.142366+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:31.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[21528]: audit 2024-04-02T04:36:30.766918+0000 mon.a (mon.0) 282 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:31.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[21528]: audit 2024-04-02T04:36:30.767828+0000 mon.a (mon.0) 283 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:31.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:31 smithi069 bash[21528]: audit 2024-04-02T04:36:30.772428+0000 mon.a (mon.0) 284 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:36:31.975 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[24743]: debug 2024-04-02T04:36:31.613+0000 7fd015581200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T04:36:31.975 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:36:31 smithi098 bash[24743]: debug 2024-04-02T04:36:31.729+0000 7fd015581200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T04:36:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:32 smithi098 bash[23790]: cluster 2024-04-02T04:36:31.068259+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:32 smithi098 bash[23790]: cluster 2024-04-02T04:36:31.756298+0000 mon.a (mon.0) 285 : cluster 0 Standby manager daemon x started 2024-04-02T04:36:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:32 smithi098 bash[23790]: audit 2024-04-02T04:36:31.757333+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-02T04:36:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:32 smithi098 bash[23790]: audit 2024-04-02T04:36:31.758270+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:36:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:32 smithi098 bash[23790]: audit 2024-04-02T04:36:31.759556+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-02T04:36:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:32 smithi098 bash[23790]: audit 2024-04-02T04:36:31.760101+0000 mon.b (mon.1) 7 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:36:32.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[21528]: cluster 2024-04-02T04:36:31.068259+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:32.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[21528]: cluster 2024-04-02T04:36:31.756298+0000 mon.a (mon.0) 285 : cluster 0 Standby manager daemon x started 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[21528]: audit 2024-04-02T04:36:31.757333+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[21528]: audit 2024-04-02T04:36:31.758270+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[21528]: audit 2024-04-02T04:36:31.759556+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[21528]: audit 2024-04-02T04:36:31.760101+0000 mon.b (mon.1) 7 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[28904]: cluster 2024-04-02T04:36:31.068259+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[28904]: cluster 2024-04-02T04:36:31.756298+0000 mon.a (mon.0) 285 : cluster 0 Standby manager daemon x started 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[28904]: audit 2024-04-02T04:36:31.757333+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[28904]: audit 2024-04-02T04:36:31.758270+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[28904]: audit 2024-04-02T04:36:31.759556+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-02T04:36:32.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:32 smithi069 bash[28904]: audit 2024-04-02T04:36:31.760101+0000 mon.b (mon.1) 7 : audit 0 from='mgr.? 172.21.15.98:0/1563381212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:36:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:33 smithi098 bash[23790]: cluster 2024-04-02T04:36:32.164621+0000 mon.a (mon.0) 286 : cluster 0 mgrmap e14: y(active, since 105s), standbys: x 2024-04-02T04:36:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:33 smithi098 bash[23790]: audit 2024-04-02T04:36:32.165371+0000 mon.a (mon.0) 287 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T04:36:33.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:33 smithi069 bash[21528]: cluster 2024-04-02T04:36:32.164621+0000 mon.a (mon.0) 286 : cluster 0 mgrmap e14: y(active, since 105s), standbys: x 2024-04-02T04:36:33.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:33 smithi069 bash[21528]: audit 2024-04-02T04:36:32.165371+0000 mon.a (mon.0) 287 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T04:36:33.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:33 smithi069 bash[28904]: cluster 2024-04-02T04:36:32.164621+0000 mon.a (mon.0) 286 : cluster 0 mgrmap e14: y(active, since 105s), standbys: x 2024-04-02T04:36:33.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:33 smithi069 bash[28904]: audit 2024-04-02T04:36:32.165371+0000 mon.a (mon.0) 287 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T04:36:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:34 smithi098 bash[23790]: cluster 2024-04-02T04:36:33.068647+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:34 smithi069 bash[21528]: cluster 2024-04-02T04:36:33.068647+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:34 smithi069 bash[28904]: cluster 2024-04-02T04:36:33.068647+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:35.252 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:36:36.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:36 smithi069 bash[28904]: cluster 2024-04-02T04:36:35.069107+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:36.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:36 smithi069 bash[21528]: cluster 2024-04-02T04:36:35.069107+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:36.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:36 smithi098 bash[23790]: cluster 2024-04-02T04:36:35.069107+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:37.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- 172.21.15.69:0/1579055300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c103620 msgr2=0x7f6f4c103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- 172.21.15.69:0/1579055300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c103620 0x7f6f4c103a80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009990 tx=0x7f6f40009ca0 comp rx=0 tx=0).stop 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- 172.21.15.69:0/1579055300 shutdown_connections 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- 172.21.15.69:0/1579055300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f4c103fc0 0x7f6f4c10a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- 172.21.15.69:0/1579055300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c100640 0x7f6f4c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- 172.21.15.69:0/1579055300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c103620 0x7f6f4c103a80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- 172.21.15.69:0/1579055300 >> 172.21.15.69:0/1579055300 conn(0x7f6f4c0fbfa0 msgr2=0x7f6f4c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- 172.21.15.69:0/1579055300 shutdown_connections 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- 172.21.15.69:0/1579055300 wait complete. 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 librados: starting msgr at 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 librados: starting objecter 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 Processor -- start 2024-04-02T04:36:37.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- start start 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 librados: setting wanted keys 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 librados: calling monclient init 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 0x7f6f4c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f4c103620 0x7f6f4c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c103fc0 0x7f6f4c1a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c10d4e0 con 0x7f6f4c103fc0 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c10d360 con 0x7f6f4c103620 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f511d8700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6f4c10d660 con 0x7f6f4c100640 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 0x7f6f4c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4b59e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c103fc0 0x7f6f4c1a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 0x7f6f4c199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52094/0 (socket says 172.21.15.69:52094) 2024-04-02T04:36:37.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 -- 172.21.15.69:0/3121279201 learned_addr learned my addr 172.21.15.69:0/3121279201 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4a59c700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f4c103620 0x7f6f4c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 -- 172.21.15.69:0/3121279201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f4c103620 msgr2=0x7f6f4c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f4c103620 0x7f6f4c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 -- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c103fc0 msgr2=0x7f6f4c1a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c103fc0 0x7f6f4c1a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 -- 172.21.15.69:0/3121279201 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4c1a1a20 con 0x7f6f4c100640 2024-04-02T04:36:37.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4b59e700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c103fc0 0x7f6f4c1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f4ad9d700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 0x7f6f4c199dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c00ea80 tx=0x7f6f3c00ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f33fff700 10 client.?.objecter ms_handle_connect 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.417+0000 7f6f33fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c00cbd0 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6f3c00cd70 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f4c1a1d10 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c018680 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f511d8700 10 client.24116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f4c1a21e0 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f511d8700 1 librados: init done 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f4c04ff00 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 14) ==== 91718+0+0 (secure 0 0 0) 0x7f6f3c01f020 con 0x7f6f4c100640 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f6f3406f450 0x7f6f34071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:36:37.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6f3c08c580 con 0x7f6f4c100640 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 10 client.24116.objecter ms_dispatch 0x7f6f4c001550 osd_map(4..4 src has 1..4) 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 3 client.24116.objecter handle_osd_map got epochs [4,4] > 0 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 3 client.24116.objecter handle_osd_map decoding full epoch 4 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f4a59c700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f6f3406f450 0x7f6f34071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 20 client.24116.objecter dump_active .. 0 homeless 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f33fff700 10 client.24116.objecter ms_handle_connect 0x7f6f3406f450 2024-04-02T04:36:37.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.421+0000 7f6f4a59c700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f6f3406f450 0x7f6f34071910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b2e0 tx=0x7f6f4001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:36:37.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.425+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6f3c05aec0 con 0x7f6f4c100640 2024-04-02T04:36:37.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:37.641+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6f4c102b30 con 0x7f6f3406f450 2024-04-02T04:36:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:38 smithi098 bash[23790]: cluster 2024-04-02T04:36:37.069569+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:38 smithi098 bash[23790]: audit 2024-04-02T04:36:37.648144+0000 mgr.y (mgr.14152) 87 : audit 0 from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:38 smithi098 bash[23790]: audit 2024-04-02T04:36:37.651174+0000 mon.a (mon.0) 288 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:36:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:38 smithi098 bash[23790]: audit 2024-04-02T04:36:37.656986+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:36:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:38 smithi098 bash[23790]: audit 2024-04-02T04:36:37.658076+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:38.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[21528]: cluster 2024-04-02T04:36:37.069569+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[21528]: audit 2024-04-02T04:36:37.648144+0000 mgr.y (mgr.14152) 87 : audit 0 from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[21528]: audit 2024-04-02T04:36:37.651174+0000 mon.a (mon.0) 288 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[21528]: audit 2024-04-02T04:36:37.656986+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[21528]: audit 2024-04-02T04:36:37.658076+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[28904]: cluster 2024-04-02T04:36:37.069569+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[28904]: audit 2024-04-02T04:36:37.648144+0000 mgr.y (mgr.14152) 87 : audit 0 from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[28904]: audit 2024-04-02T04:36:37.651174+0000 mon.a (mon.0) 288 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[28904]: audit 2024-04-02T04:36:37.656986+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:36:38.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:38 smithi069 bash[28904]: audit 2024-04-02T04:36:37.658076+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:40 smithi098 bash[23790]: cluster 2024-04-02T04:36:39.070007+0000 mgr.y (mgr.14152) 88 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:40.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:40 smithi069 bash[21528]: cluster 2024-04-02T04:36:39.070007+0000 mgr.y (mgr.14152) 88 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:40.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:40 smithi069 bash[28904]: cluster 2024-04-02T04:36:39.070007+0000 mgr.y (mgr.14152) 88 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:42.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:42.425+0000 7f6f4bfff700 10 client.24116.objecter tick 2024-04-02T04:36:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:42 smithi098 bash[23790]: cluster 2024-04-02T04:36:41.070449+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:42.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:42 smithi069 bash[21528]: cluster 2024-04-02T04:36:41.070449+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:42 smithi069 bash[28904]: cluster 2024-04-02T04:36:41.070449+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:44.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:44 smithi098 bash[23790]: cluster 2024-04-02T04:36:43.070866+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:45.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:44 smithi069 bash[21528]: cluster 2024-04-02T04:36:43.070866+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:45.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:44 smithi069 bash[28904]: cluster 2024-04-02T04:36:43.070866+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:45.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:45 smithi098 bash[23790]: audit 2024-04-02T04:36:45.614740+0000 mon.c (mon.2) 4 : audit 1 from='client.? 172.21.15.69:0/2426938800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]: dispatch 2024-04-02T04:36:45.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:45 smithi098 bash[23790]: audit 2024-04-02T04:36:45.615428+0000 mon.a (mon.0) 291 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]: dispatch 2024-04-02T04:36:45.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:45 smithi098 bash[23790]: audit 2024-04-02T04:36:45.617894+0000 mon.a (mon.0) 292 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]': finished 2024-04-02T04:36:45.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:45 smithi098 bash[23790]: cluster 2024-04-02T04:36:45.622608+0000 mon.a (mon.0) 293 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-04-02T04:36:45.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:45 smithi098 bash[23790]: audit 2024-04-02T04:36:45.623129+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[21528]: audit 2024-04-02T04:36:45.614740+0000 mon.c (mon.2) 4 : audit 1 from='client.? 172.21.15.69:0/2426938800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]: dispatch 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[21528]: audit 2024-04-02T04:36:45.615428+0000 mon.a (mon.0) 291 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]: dispatch 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[21528]: audit 2024-04-02T04:36:45.617894+0000 mon.a (mon.0) 292 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]': finished 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[21528]: cluster 2024-04-02T04:36:45.622608+0000 mon.a (mon.0) 293 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[21528]: audit 2024-04-02T04:36:45.623129+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[28904]: audit 2024-04-02T04:36:45.614740+0000 mon.c (mon.2) 4 : audit 1 from='client.? 172.21.15.69:0/2426938800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]: dispatch 2024-04-02T04:36:46.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[28904]: audit 2024-04-02T04:36:45.615428+0000 mon.a (mon.0) 291 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]: dispatch 2024-04-02T04:36:46.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[28904]: audit 2024-04-02T04:36:45.617894+0000 mon.a (mon.0) 292 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b"}]': finished 2024-04-02T04:36:46.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[28904]: cluster 2024-04-02T04:36:45.622608+0000 mon.a (mon.0) 293 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-04-02T04:36:46.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:45 smithi069 bash[28904]: audit 2024-04-02T04:36:45.623129+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:36:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:46 smithi098 bash[23790]: cluster 2024-04-02T04:36:45.071261+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:46 smithi098 bash[23790]: audit 2024-04-02T04:36:46.320461+0000 mon.c (mon.2) 5 : audit 0 from='client.? 172.21.15.69:0/3572937114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:36:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:46 smithi069 bash[21528]: cluster 2024-04-02T04:36:45.071261+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:46 smithi069 bash[21528]: audit 2024-04-02T04:36:46.320461+0000 mon.c (mon.2) 5 : audit 0 from='client.? 172.21.15.69:0/3572937114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:36:47.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:46 smithi069 bash[28904]: cluster 2024-04-02T04:36:45.071261+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:47.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:46 smithi069 bash[28904]: audit 2024-04-02T04:36:46.320461+0000 mon.c (mon.2) 5 : audit 0 from='client.? 172.21.15.69:0/3572937114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:36:47.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:47.429+0000 7f6f4bfff700 10 client.24116.objecter tick 2024-04-02T04:36:48.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:48 smithi098 bash[23790]: cluster 2024-04-02T04:36:47.071760+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:49.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:48 smithi069 bash[21528]: cluster 2024-04-02T04:36:47.071760+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:49.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:48 smithi069 bash[28904]: cluster 2024-04-02T04:36:47.071760+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:50.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:50 smithi098 bash[23790]: cluster 2024-04-02T04:36:49.072212+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:51.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:50 smithi069 bash[21528]: cluster 2024-04-02T04:36:49.072212+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:51.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:50 smithi069 bash[28904]: cluster 2024-04-02T04:36:49.072212+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:52.433+0000 7f6f4bfff700 10 client.24116.objecter tick 2024-04-02T04:36:52.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:52 smithi098 bash[23790]: cluster 2024-04-02T04:36:51.072590+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:53.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:52 smithi069 bash[21528]: cluster 2024-04-02T04:36:51.072590+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:53.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:52 smithi069 bash[28904]: cluster 2024-04-02T04:36:51.072590+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:54.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:54 smithi098 bash[23790]: cluster 2024-04-02T04:36:53.072956+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:55.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:54 smithi069 bash[21528]: cluster 2024-04-02T04:36:53.072956+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:55.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:54 smithi069 bash[28904]: cluster 2024-04-02T04:36:53.072956+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:56.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:56 smithi098 bash[23790]: cluster 2024-04-02T04:36:55.073400+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:57.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:56 smithi069 bash[21528]: cluster 2024-04-02T04:36:55.073400+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:57.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:56 smithi069 bash[28904]: cluster 2024-04-02T04:36:55.073400+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:57.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:36:57.437+0000 7f6f4bfff700 10 client.24116.objecter tick 2024-04-02T04:36:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:58 smithi098 bash[23790]: cluster 2024-04-02T04:36:57.073718+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:58 smithi098 bash[23790]: audit 2024-04-02T04:36:57.916148+0000 mon.a (mon.0) 295 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:36:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:58 smithi098 bash[23790]: audit 2024-04-02T04:36:57.917691+0000 mon.a (mon.0) 296 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:36:58 smithi098 bash[23790]: cephadm 2024-04-02T04:36:57.919135+0000 mgr.y (mgr.14152) 98 : cephadm 1 Deploying daemon osd.0 on smithi069 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[21528]: cluster 2024-04-02T04:36:57.073718+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[21528]: audit 2024-04-02T04:36:57.916148+0000 mon.a (mon.0) 295 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[21528]: audit 2024-04-02T04:36:57.917691+0000 mon.a (mon.0) 296 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[21528]: cephadm 2024-04-02T04:36:57.919135+0000 mgr.y (mgr.14152) 98 : cephadm 1 Deploying daemon osd.0 on smithi069 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[28904]: cluster 2024-04-02T04:36:57.073718+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[28904]: audit 2024-04-02T04:36:57.916148+0000 mon.a (mon.0) 295 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[28904]: audit 2024-04-02T04:36:57.917691+0000 mon.a (mon.0) 296 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:59.021 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:36:58 smithi069 bash[28904]: cephadm 2024-04-02T04:36:57.919135+0000 mgr.y (mgr.14152) 98 : cephadm 1 Deploying daemon osd.0 on smithi069 2024-04-02T04:37:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:00 smithi098 bash[23790]: cluster 2024-04-02T04:36:59.074045+0000 mgr.y (mgr.14152) 99 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:01.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:00 smithi069 bash[21528]: cluster 2024-04-02T04:36:59.074045+0000 mgr.y (mgr.14152) 99 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:00 smithi069 bash[28904]: cluster 2024-04-02T04:36:59.074045+0000 mgr.y (mgr.14152) 99 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:01.808 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:37:01 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:01.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:01 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:01.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:01 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:02.093 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:37:01 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:02.094 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:01 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:02.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:01 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:02.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:02.441+0000 7f6f4bfff700 10 client.24116.objecter tick 2024-04-02T04:37:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:02 smithi098 bash[23790]: cluster 2024-04-02T04:37:01.074461+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:02 smithi098 bash[23790]: audit 2024-04-02T04:37:02.088881+0000 mon.a (mon.0) 297 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:02 smithi098 bash[23790]: audit 2024-04-02T04:37:02.094376+0000 mon.a (mon.0) 298 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:02 smithi098 bash[23790]: audit 2024-04-02T04:37:02.115124+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:03.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[28904]: cluster 2024-04-02T04:37:01.074461+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[28904]: audit 2024-04-02T04:37:02.088881+0000 mon.a (mon.0) 297 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[28904]: audit 2024-04-02T04:37:02.094376+0000 mon.a (mon.0) 298 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[28904]: audit 2024-04-02T04:37:02.115124+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[21528]: cluster 2024-04-02T04:37:01.074461+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[21528]: audit 2024-04-02T04:37:02.088881+0000 mon.a (mon.0) 297 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[21528]: audit 2024-04-02T04:37:02.094376+0000 mon.a (mon.0) 298 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:03.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:02 smithi069 bash[21528]: audit 2024-04-02T04:37:02.115124+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:05.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:04 smithi069 bash[21528]: cluster 2024-04-02T04:37:03.074772+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:04 smithi069 bash[28904]: cluster 2024-04-02T04:37:03.074772+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:04 smithi098 bash[23790]: cluster 2024-04-02T04:37:03.074772+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:07.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:06 smithi069 bash[21528]: cluster 2024-04-02T04:37:05.075140+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:07.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:06 smithi069 bash[28904]: cluster 2024-04-02T04:37:05.075140+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:07.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:06 smithi098 bash[23790]: cluster 2024-04-02T04:37:05.075140+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:07.445+0000 7f6f4bfff700 10 client.24116.objecter tick 2024-04-02T04:37:09.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:08 smithi069 bash[28904]: cluster 2024-04-02T04:37:07.075596+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:08 smithi069 bash[28904]: audit 2024-04-02T04:37:08.352557+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:08 smithi069 bash[28904]: audit 2024-04-02T04:37:08.352914+0000 mon.a (mon.0) 300 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:09.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:08 smithi069 bash[21528]: cluster 2024-04-02T04:37:07.075596+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:09.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:08 smithi069 bash[21528]: audit 2024-04-02T04:37:08.352557+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:09.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:08 smithi069 bash[21528]: audit 2024-04-02T04:37:08.352914+0000 mon.a (mon.0) 300 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:09.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:08 smithi098 bash[23790]: cluster 2024-04-02T04:37:07.075596+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:09.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:08 smithi098 bash[23790]: audit 2024-04-02T04:37:08.352557+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:09.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:08 smithi098 bash[23790]: audit 2024-04-02T04:37:08.352914+0000 mon.a (mon.0) 300 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.731418+0000 mon.a (mon.0) 301 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: cluster 2024-04-02T04:37:08.735242+0000 mon.a (mon.0) 302 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.735705+0000 mon.a (mon.0) 303 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.737514+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.742377+0000 mon.a (mon.0) 304 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.951913+0000 mon.a (mon.0) 305 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.956658+0000 mon.a (mon.0) 306 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.957874+0000 mon.a (mon.0) 307 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.959245+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[21528]: audit 2024-04-02T04:37:08.969961+0000 mon.a (mon.0) 309 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.731418+0000 mon.a (mon.0) 301 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: cluster 2024-04-02T04:37:08.735242+0000 mon.a (mon.0) 302 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.735705+0000 mon.a (mon.0) 303 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.737514+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.742377+0000 mon.a (mon.0) 304 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.951913+0000 mon.a (mon.0) 305 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.956658+0000 mon.a (mon.0) 306 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.957874+0000 mon.a (mon.0) 307 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.959245+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:09 smithi069 bash[28904]: audit 2024-04-02T04:37:08.969961+0000 mon.a (mon.0) 309 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:10.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.731418+0000 mon.a (mon.0) 301 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:37:10.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: cluster 2024-04-02T04:37:08.735242+0000 mon.a (mon.0) 302 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-04-02T04:37:10.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.735705+0000 mon.a (mon.0) 303 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:10.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.737514+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:37:10.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.742377+0000 mon.a (mon.0) 304 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:37:10.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.951913+0000 mon.a (mon.0) 305 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:10.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.956658+0000 mon.a (mon.0) 306 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:10.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.957874+0000 mon.a (mon.0) 307 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:10.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.959245+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:10.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:09 smithi098 bash[23790]: audit 2024-04-02T04:37:08.969961+0000 mon.a (mon.0) 309 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:11.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[21528]: cluster 2024-04-02T04:37:09.076097+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[21528]: audit 2024-04-02T04:37:09.748347+0000 mon.a (mon.0) 310 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[21528]: cluster 2024-04-02T04:37:09.750802+0000 mon.a (mon.0) 311 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[21528]: audit 2024-04-02T04:37:09.751244+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[21528]: audit 2024-04-02T04:37:09.761450+0000 mon.a (mon.0) 313 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[28904]: cluster 2024-04-02T04:37:09.076097+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[28904]: audit 2024-04-02T04:37:09.748347+0000 mon.a (mon.0) 310 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[28904]: cluster 2024-04-02T04:37:09.750802+0000 mon.a (mon.0) 311 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[28904]: audit 2024-04-02T04:37:09.751244+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:10 smithi069 bash[28904]: audit 2024-04-02T04:37:09.761450+0000 mon.a (mon.0) 313 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:10 smithi098 bash[23790]: cluster 2024-04-02T04:37:09.076097+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:10 smithi098 bash[23790]: audit 2024-04-02T04:37:09.748347+0000 mon.a (mon.0) 310 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:37:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:10 smithi098 bash[23790]: cluster 2024-04-02T04:37:09.750802+0000 mon.a (mon.0) 311 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-04-02T04:37:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:10 smithi098 bash[23790]: audit 2024-04-02T04:37:09.751244+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:10 smithi098 bash[23790]: audit 2024-04-02T04:37:09.761450+0000 mon.a (mon.0) 313 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:11.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[21528]: cluster 2024-04-02T04:37:09.304453+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:37:11.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[21528]: cluster 2024-04-02T04:37:09.304558+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:37:11.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[21528]: cluster 2024-04-02T04:37:10.757874+0000 mon.a (mon.0) 314 : cluster 1 osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] boot 2024-04-02T04:37:11.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[21528]: cluster 2024-04-02T04:37:10.757947+0000 mon.a (mon.0) 315 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-04-02T04:37:11.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[21528]: audit 2024-04-02T04:37:10.758515+0000 mon.a (mon.0) 316 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:12.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[28904]: cluster 2024-04-02T04:37:09.304453+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:37:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[28904]: cluster 2024-04-02T04:37:09.304558+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:37:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[28904]: cluster 2024-04-02T04:37:10.757874+0000 mon.a (mon.0) 314 : cluster 1 osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] boot 2024-04-02T04:37:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[28904]: cluster 2024-04-02T04:37:10.757947+0000 mon.a (mon.0) 315 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-04-02T04:37:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:11 smithi069 bash[28904]: audit 2024-04-02T04:37:10.758515+0000 mon.a (mon.0) 316 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:12.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.197+0000 7f6f33fff700 1 -- 172.21.15.69:0/3121279201 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6f4c102b30 con 0x7f6f3406f450 2024-04-02T04:37:12.204 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 0 on host 'smithi069' 2024-04-02T04:37:12.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 10 librados: watch_flush enter 2024-04-02T04:37:12.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 10 librados: watch_flush exit 2024-04-02T04:37:12.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 20 client.24116.objecter shutdown clearing up homeless session... 2024-04-02T04:37:12.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 10 client.24116.objecter successfully canceled tick 2024-04-02T04:37:12.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f6f3406f450 msgr2=0x7f6f34071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f6f3406f450 0x7f6f34071910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b2e0 tx=0x7f6f4001a040 comp rx=0 tx=0).stop 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 msgr2=0x7f6f4c199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 0x7f6f4c199dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c00ea80 tx=0x7f6f3c00ed90 comp rx=0 tx=0).stop 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 shutdown_connections 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f4c103fc0 0x7f6f4c1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f6f3406f450 0x7f6f34071910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f4c100640 0x7f6f4c199dc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 --2- 172.21.15.69:0/3121279201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f4c103620 0x7f6f4c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 >> 172.21.15.69:0/3121279201 conn(0x7f6f4c0fbfa0 msgr2=0x7f6f4c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 shutdown_connections 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 -- 172.21.15.69:0/3121279201 wait complete. 2024-04-02T04:37:12.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:12.201+0000 7f6f511d8700 1 librados: shutdown 2024-04-02T04:37:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:11 smithi098 bash[23790]: cluster 2024-04-02T04:37:09.304453+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:37:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:11 smithi098 bash[23790]: cluster 2024-04-02T04:37:09.304558+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:37:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:11 smithi098 bash[23790]: cluster 2024-04-02T04:37:10.757874+0000 mon.a (mon.0) 314 : cluster 1 osd.0 [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] boot 2024-04-02T04:37:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:11 smithi098 bash[23790]: cluster 2024-04-02T04:37:10.757947+0000 mon.a (mon.0) 315 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-04-02T04:37:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:11 smithi098 bash[23790]: audit 2024-04-02T04:37:10.758515+0000 mon.a (mon.0) 316 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:12.826 DEBUG:teuthology.orchestra.run.smithi069:osd.0> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.0.service 2024-04-02T04:37:12.828 INFO:tasks.cephadm:Deploying osd.1 on smithi069 with /dev/vg_nvme/lv_3... 2024-04-02T04:37:12.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T04:37:13.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[28904]: cluster 2024-04-02T04:37:11.076428+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[28904]: cluster 2024-04-02T04:37:11.779631+0000 mon.a (mon.0) 317 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[28904]: audit 2024-04-02T04:37:12.189658+0000 mon.a (mon.0) 318 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[28904]: audit 2024-04-02T04:37:12.200693+0000 mon.a (mon.0) 319 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[28904]: audit 2024-04-02T04:37:12.212414+0000 mon.a (mon.0) 320 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[21528]: cluster 2024-04-02T04:37:11.076428+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[21528]: cluster 2024-04-02T04:37:11.779631+0000 mon.a (mon.0) 317 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[21528]: audit 2024-04-02T04:37:12.189658+0000 mon.a (mon.0) 318 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[21528]: audit 2024-04-02T04:37:12.200693+0000 mon.a (mon.0) 319 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:13.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:12 smithi069 bash[21528]: audit 2024-04-02T04:37:12.212414+0000 mon.a (mon.0) 320 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:12 smithi098 bash[23790]: cluster 2024-04-02T04:37:11.076428+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:12 smithi098 bash[23790]: cluster 2024-04-02T04:37:11.779631+0000 mon.a (mon.0) 317 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-04-02T04:37:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:12 smithi098 bash[23790]: audit 2024-04-02T04:37:12.189658+0000 mon.a (mon.0) 318 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:12 smithi098 bash[23790]: audit 2024-04-02T04:37:12.200693+0000 mon.a (mon.0) 319 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:12 smithi098 bash[23790]: audit 2024-04-02T04:37:12.212414+0000 mon.a (mon.0) 320 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:15.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:14 smithi069 bash[21528]: cluster 2024-04-02T04:37:13.076683+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:15.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:14 smithi069 bash[28904]: cluster 2024-04-02T04:37:13.076683+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:14 smithi098 bash[23790]: cluster 2024-04-02T04:37:13.076683+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:16 smithi069 bash[21528]: cluster 2024-04-02T04:37:15.077124+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:16 smithi069 bash[28904]: cluster 2024-04-02T04:37:15.077124+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:16 smithi098 bash[23790]: cluster 2024-04-02T04:37:15.077124+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:37:18.918 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:18 smithi069 bash[21528]: cluster 2024-04-02T04:37:17.077662+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:18.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:18 smithi069 bash[28904]: cluster 2024-04-02T04:37:17.077662+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:19.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:18 smithi098 bash[23790]: cluster 2024-04-02T04:37:17.077662+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:20 smithi069 bash[21528]: cluster 2024-04-02T04:37:19.078140+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:20 smithi069 bash[28904]: cluster 2024-04-02T04:37:19.078140+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:20 smithi098 bash[23790]: cluster 2024-04-02T04:37:19.078140+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:21.423 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T04:37:21.423 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T04:37:21.423 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-02T04:37:21.424 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-02T04:37:21.424 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134684 s, 779 MB/s 2024-04-02T04:37:21.424 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 2024-04-02T04:37:21.424 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-02T04:37:22.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi069:vg_nvme/lv_3 2024-04-02T04:37:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: cluster 2024-04-02T04:37:21.078495+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.580483+0000 mon.a (mon.0) 321 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.588731+0000 mon.a (mon.0) 322 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.590787+0000 mon.a (mon.0) 323 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.598709+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.603418+0000 mon.a (mon.0) 325 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.605266+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:23.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:22 smithi098 bash[23790]: audit 2024-04-02T04:37:22.614950+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: cluster 2024-04-02T04:37:21.078495+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.580483+0000 mon.a (mon.0) 321 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.588731+0000 mon.a (mon.0) 322 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.590787+0000 mon.a (mon.0) 323 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.598709+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.603418+0000 mon.a (mon.0) 325 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.605266+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[21528]: audit 2024-04-02T04:37:22.614950+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: cluster 2024-04-02T04:37:21.078495+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.580483+0000 mon.a (mon.0) 321 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.588731+0000 mon.a (mon.0) 322 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.590787+0000 mon.a (mon.0) 323 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.598709+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.603418+0000 mon.a (mon.0) 325 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.605266+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:23.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:22 smithi069 bash[28904]: audit 2024-04-02T04:37:22.614950+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:24.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:23 smithi098 bash[23790]: cephadm 2024-04-02T04:37:22.572621+0000 mgr.y (mgr.14152) 111 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:37:24.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:23 smithi098 bash[23790]: cephadm 2024-04-02T04:37:22.592139+0000 mgr.y (mgr.14152) 112 : cephadm 1 Adjusting osd_memory_target on smithi069 to 16241M 2024-04-02T04:37:24.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:23 smithi069 bash[21528]: cephadm 2024-04-02T04:37:22.572621+0000 mgr.y (mgr.14152) 111 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:37:24.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:23 smithi069 bash[21528]: cephadm 2024-04-02T04:37:22.592139+0000 mgr.y (mgr.14152) 112 : cephadm 1 Adjusting osd_memory_target on smithi069 to 16241M 2024-04-02T04:37:24.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:23 smithi069 bash[28904]: cephadm 2024-04-02T04:37:22.572621+0000 mgr.y (mgr.14152) 111 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:37:24.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:23 smithi069 bash[28904]: cephadm 2024-04-02T04:37:22.592139+0000 mgr.y (mgr.14152) 112 : cephadm 1 Adjusting osd_memory_target on smithi069 to 16241M 2024-04-02T04:37:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:24 smithi098 bash[23790]: cluster 2024-04-02T04:37:23.078911+0000 mgr.y (mgr.14152) 113 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:24 smithi069 bash[21528]: cluster 2024-04-02T04:37:23.078911+0000 mgr.y (mgr.14152) 113 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:24 smithi069 bash[28904]: cluster 2024-04-02T04:37:23.078911+0000 mgr.y (mgr.14152) 113 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:26.920 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:37:27.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:26 smithi098 bash[23790]: cluster 2024-04-02T04:37:25.079380+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:27.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:26 smithi069 bash[28904]: cluster 2024-04-02T04:37:25.079380+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:27.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:26 smithi069 bash[21528]: cluster 2024-04-02T04:37:25.079380+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:29.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:28 smithi069 bash[21528]: cluster 2024-04-02T04:37:27.079911+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:29.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:28 smithi069 bash[28904]: cluster 2024-04-02T04:37:27.079911+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:29.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:28 smithi098 bash[23790]: cluster 2024-04-02T04:37:27.079911+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:29.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/979554547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9800108010 msgr2=0x7f980010a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:37:29.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- 172.21.15.69:0/979554547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9800108010 0x7f980010a420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f97f800b0a0 tx=0x7f97f800b3b0 comp rx=0 tx=0).stop 2024-04-02T04:37:29.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/979554547 shutdown_connections 2024-04-02T04:37:29.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- 172.21.15.69:0/979554547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9800108010 0x7f980010a420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:29.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- 172.21.15.69:0/979554547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98000fff00 0x7f9800100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:29.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- 172.21.15.69:0/979554547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98001008f0 0x7f9800107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/979554547 >> 172.21.15.69:0/979554547 conn(0x7f98000fbaa0 msgr2=0x7f98000fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/979554547 shutdown_connections 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/979554547 wait complete. 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 librados: starting msgr at 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 librados: starting objecter 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 Processor -- start 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- start start 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 librados: setting wanted keys 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 librados: calling monclient init 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 0x7f9800197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98001008f0 0x7f9800197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9800108010 0x7f980019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:37:29.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f980010d420 con 0x7f98001008f0 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f980010d2a0 con 0x7f9800108010 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f980010d5a0 con 0x7f98000fff00 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 0x7f9800197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97feffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98001008f0 0x7f9800197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 0x7f9800197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52540/0 (socket says 172.21.15.69:52540) 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 -- 172.21.15.69:0/1727237349 learned_addr learned my addr 172.21.15.69:0/1727237349 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97fffff700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9800108010 0x7f980019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:37:29.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 -- 172.21.15.69:0/1727237349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9800108010 msgr2=0x7f980019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9800108010 0x7f980019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 -- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98001008f0 msgr2=0x7f9800197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98001008f0 0x7f9800197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 -- 172.21.15.69:0/1727237349 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980019f690 con 0x7f98000fff00 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97feffd700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98001008f0 0x7f9800197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97ff7fe700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 0x7f9800197a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c9a0 tx=0x7f97f000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97fcff9700 10 client.?.objecter ms_handle_connect 0x7f98000fff00 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97fcff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:37:29.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0004220 con 0x7f98000fff00 2024-04-02T04:37:29.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980019f980 con 0x7f98000fff00 2024-04-02T04:37:29.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f97f0007ce0 con 0x7f98000fff00 2024-04-02T04:37:29.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0003bb0 con 0x7f98000fff00 2024-04-02T04:37:29.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 10 client.24143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f980019fec0 con 0x7f98000fff00 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.329+0000 7f9805c1d700 1 librados: init done 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f980004ff00 con 0x7f98000fff00 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 14) ==== 91718+0+0 (secure 0 0 0) 0x7f97f0007860 con 0x7f98000fff00 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f97ec06f450 0x7f97ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97feffd700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f97ec06f450 0x7f97ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:37:29.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f97f008c2d0 con 0x7f98000fff00 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 10 client.24143.objecter ms_dispatch 0x7f9800001550 osd_map(9..9 src has 1..9) 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 3 client.24143.objecter handle_osd_map got epochs [9,9] > 0 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 3 client.24143.objecter handle_osd_map decoding full epoch 9 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 20 client.24143.objecter dump_active .. 0 homeless 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97fcff9700 10 client.24143.objecter ms_handle_connect 0x7f97ec06f450 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.333+0000 7f97feffd700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f97ec06f450 0x7f97ec071910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009680 tx=0x7f97e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:37:29.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.337+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f97f0014030 con 0x7f98000fff00 2024-04-02T04:37:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:29.545+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f98000008d0 con 0x7f97ec06f450 2024-04-02T04:37:30.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:29 smithi098 bash[23790]: audit 2024-04-02T04:37:29.555846+0000 mon.a (mon.0) 328 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:37:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:29 smithi098 bash[23790]: audit 2024-04-02T04:37:29.561246+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:37:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:29 smithi098 bash[23790]: audit 2024-04-02T04:37:29.562624+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:29 smithi069 bash[21528]: audit 2024-04-02T04:37:29.555846+0000 mon.a (mon.0) 328 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:37:30.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:29 smithi069 bash[21528]: audit 2024-04-02T04:37:29.561246+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:37:30.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:29 smithi069 bash[21528]: audit 2024-04-02T04:37:29.562624+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:30.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:29 smithi069 bash[28904]: audit 2024-04-02T04:37:29.555846+0000 mon.a (mon.0) 328 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:37:30.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:29 smithi069 bash[28904]: audit 2024-04-02T04:37:29.561246+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:37:30.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:29 smithi069 bash[28904]: audit 2024-04-02T04:37:29.562624+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:31.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:30 smithi098 bash[23790]: cluster 2024-04-02T04:37:29.080359+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:31.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:30 smithi098 bash[23790]: audit 2024-04-02T04:37:29.553084+0000 mgr.y (mgr.14152) 117 : audit 0 from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:37:31.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:30 smithi069 bash[21528]: cluster 2024-04-02T04:37:29.080359+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:31.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:30 smithi069 bash[21528]: audit 2024-04-02T04:37:29.553084+0000 mgr.y (mgr.14152) 117 : audit 0 from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:37:31.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:30 smithi069 bash[28904]: cluster 2024-04-02T04:37:29.080359+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:31.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:30 smithi069 bash[28904]: audit 2024-04-02T04:37:29.553084+0000 mgr.y (mgr.14152) 117 : audit 0 from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:37:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:32 smithi098 bash[23790]: cluster 2024-04-02T04:37:31.080835+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:32 smithi069 bash[21528]: cluster 2024-04-02T04:37:31.080835+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:33.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:32 smithi069 bash[28904]: cluster 2024-04-02T04:37:31.080835+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:34.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:34.333+0000 7f980541c700 10 client.24143.objecter tick 2024-04-02T04:37:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:34 smithi098 bash[23790]: cluster 2024-04-02T04:37:33.081267+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:35.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:34 smithi069 bash[28904]: cluster 2024-04-02T04:37:33.081267+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:35.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:34 smithi069 bash[21528]: cluster 2024-04-02T04:37:33.081267+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:36 smithi069 bash[21528]: cluster 2024-04-02T04:37:35.081747+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:37.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:36 smithi069 bash[28904]: cluster 2024-04-02T04:37:35.081747+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:36 smithi098 bash[23790]: cluster 2024-04-02T04:37:35.081747+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:38.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:37 smithi098 bash[23790]: audit 2024-04-02T04:37:37.596788+0000 mon.a (mon.0) 331 : audit 1 from='client.? 172.21.15.69:0/1967981140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a"}]: dispatch 2024-04-02T04:37:38.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:37 smithi098 bash[23790]: audit 2024-04-02T04:37:37.599506+0000 mon.a (mon.0) 332 : audit 1 from='client.? 172.21.15.69:0/1967981140' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a"}]': finished 2024-04-02T04:37:38.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:37 smithi098 bash[23790]: cluster 2024-04-02T04:37:37.603893+0000 mon.a (mon.0) 333 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-04-02T04:37:38.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:37 smithi098 bash[23790]: audit 2024-04-02T04:37:37.604149+0000 mon.a (mon.0) 334 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:37:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[28904]: audit 2024-04-02T04:37:37.596788+0000 mon.a (mon.0) 331 : audit 1 from='client.? 172.21.15.69:0/1967981140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a"}]: dispatch 2024-04-02T04:37:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[28904]: audit 2024-04-02T04:37:37.599506+0000 mon.a (mon.0) 332 : audit 1 from='client.? 172.21.15.69:0/1967981140' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a"}]': finished 2024-04-02T04:37:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[28904]: cluster 2024-04-02T04:37:37.603893+0000 mon.a (mon.0) 333 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-04-02T04:37:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[28904]: audit 2024-04-02T04:37:37.604149+0000 mon.a (mon.0) 334 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:37:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[21528]: audit 2024-04-02T04:37:37.596788+0000 mon.a (mon.0) 331 : audit 1 from='client.? 172.21.15.69:0/1967981140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a"}]: dispatch 2024-04-02T04:37:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[21528]: audit 2024-04-02T04:37:37.599506+0000 mon.a (mon.0) 332 : audit 1 from='client.? 172.21.15.69:0/1967981140' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a"}]': finished 2024-04-02T04:37:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[21528]: cluster 2024-04-02T04:37:37.603893+0000 mon.a (mon.0) 333 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-04-02T04:37:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:37 smithi069 bash[21528]: audit 2024-04-02T04:37:37.604149+0000 mon.a (mon.0) 334 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:37:39.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:38 smithi098 bash[23790]: cluster 2024-04-02T04:37:37.082216+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:38 smithi098 bash[23790]: audit 2024-04-02T04:37:38.301214+0000 mon.c (mon.2) 8 : audit 0 from='client.? 172.21.15.69:0/2626378649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:37:39.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:38 smithi069 bash[21528]: cluster 2024-04-02T04:37:37.082216+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:39.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:38 smithi069 bash[21528]: audit 2024-04-02T04:37:38.301214+0000 mon.c (mon.2) 8 : audit 0 from='client.? 172.21.15.69:0/2626378649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:37:39.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:38 smithi069 bash[28904]: cluster 2024-04-02T04:37:37.082216+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:39.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:38 smithi069 bash[28904]: audit 2024-04-02T04:37:38.301214+0000 mon.c (mon.2) 8 : audit 0 from='client.? 172.21.15.69:0/2626378649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:37:39.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:39.337+0000 7f980541c700 10 client.24143.objecter tick 2024-04-02T04:37:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:40 smithi098 bash[23790]: cluster 2024-04-02T04:37:39.082657+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:40 smithi069 bash[21528]: cluster 2024-04-02T04:37:39.082657+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:41.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:40 smithi069 bash[28904]: cluster 2024-04-02T04:37:39.082657+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:42 smithi098 bash[23790]: cluster 2024-04-02T04:37:41.083157+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:43.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:42 smithi069 bash[21528]: cluster 2024-04-02T04:37:41.083157+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:43.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:42 smithi069 bash[28904]: cluster 2024-04-02T04:37:41.083157+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:44.341+0000 7f980541c700 10 client.24143.objecter tick 2024-04-02T04:37:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:44 smithi098 bash[23790]: cluster 2024-04-02T04:37:43.083670+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:44 smithi069 bash[21528]: cluster 2024-04-02T04:37:43.083670+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:45.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:44 smithi069 bash[28904]: cluster 2024-04-02T04:37:43.083670+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:47.222 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:46 smithi069 bash[21528]: cluster 2024-04-02T04:37:45.084135+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:47.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:46 smithi069 bash[28904]: cluster 2024-04-02T04:37:45.084135+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:46 smithi098 bash[23790]: cluster 2024-04-02T04:37:45.084135+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:48 smithi098 bash[23790]: cluster 2024-04-02T04:37:47.084574+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:49.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:48 smithi069 bash[28904]: cluster 2024-04-02T04:37:47.084574+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:49.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:48 smithi069 bash[21528]: cluster 2024-04-02T04:37:47.084574+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:49.345+0000 7f980541c700 10 client.24143.objecter tick 2024-04-02T04:37:50.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:49 smithi069 bash[21528]: audit 2024-04-02T04:37:49.671572+0000 mon.a (mon.0) 335 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:37:50.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:49 smithi069 bash[21528]: audit 2024-04-02T04:37:49.673283+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:50.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:49 smithi069 bash[28904]: audit 2024-04-02T04:37:49.671572+0000 mon.a (mon.0) 335 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:37:50.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:49 smithi069 bash[28904]: audit 2024-04-02T04:37:49.673283+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:50.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:49 smithi098 bash[23790]: audit 2024-04-02T04:37:49.671572+0000 mon.a (mon.0) 335 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:37:50.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:49 smithi098 bash[23790]: audit 2024-04-02T04:37:49.673283+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:51.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:50 smithi069 bash[28904]: cluster 2024-04-02T04:37:49.084947+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:51.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:50 smithi069 bash[28904]: cephadm 2024-04-02T04:37:49.674974+0000 mgr.y (mgr.14152) 128 : cephadm 1 Deploying daemon osd.1 on smithi069 2024-04-02T04:37:51.142 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:50 smithi069 bash[21528]: cluster 2024-04-02T04:37:49.084947+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:51.142 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:50 smithi069 bash[21528]: cephadm 2024-04-02T04:37:49.674974+0000 mgr.y (mgr.14152) 128 : cephadm 1 Deploying daemon osd.1 on smithi069 2024-04-02T04:37:51.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:50 smithi098 bash[23790]: cluster 2024-04-02T04:37:49.084947+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:51.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:50 smithi098 bash[23790]: cephadm 2024-04-02T04:37:49.674974+0000 mgr.y (mgr.14152) 128 : cephadm 1 Deploying daemon osd.1 on smithi069 2024-04-02T04:37:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:52 smithi069 bash[21528]: cluster 2024-04-02T04:37:51.085406+0000 mgr.y (mgr.14152) 129 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:53.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:52 smithi069 bash[28904]: cluster 2024-04-02T04:37:51.085406+0000 mgr.y (mgr.14152) 129 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:52 smithi098 bash[23790]: cluster 2024-04-02T04:37:51.085406+0000 mgr.y (mgr.14152) 129 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:53.679 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:53.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:53.680 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:54.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:54.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:54.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:54.034 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:37:53 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:37:54.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:54.349+0000 7f980541c700 10 client.24143.objecter tick 2024-04-02T04:37:55.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:54 smithi098 bash[23790]: cluster 2024-04-02T04:37:53.085957+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:55.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:54 smithi098 bash[23790]: audit 2024-04-02T04:37:53.951486+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:55.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:54 smithi098 bash[23790]: audit 2024-04-02T04:37:53.956336+0000 mon.a (mon.0) 338 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:55.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:54 smithi098 bash[23790]: audit 2024-04-02T04:37:53.973509+0000 mon.a (mon.0) 339 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:55.271 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[21528]: cluster 2024-04-02T04:37:53.085957+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[21528]: audit 2024-04-02T04:37:53.951486+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[21528]: audit 2024-04-02T04:37:53.956336+0000 mon.a (mon.0) 338 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[21528]: audit 2024-04-02T04:37:53.973509+0000 mon.a (mon.0) 339 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[28904]: cluster 2024-04-02T04:37:53.085957+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[28904]: audit 2024-04-02T04:37:53.951486+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[28904]: audit 2024-04-02T04:37:53.956336+0000 mon.a (mon.0) 338 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:37:55.272 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:54 smithi069 bash[28904]: audit 2024-04-02T04:37:53.973509+0000 mon.a (mon.0) 339 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:57.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:56 smithi069 bash[21528]: cluster 2024-04-02T04:37:55.086292+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:57.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:56 smithi069 bash[28904]: cluster 2024-04-02T04:37:55.086292+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:57.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:56 smithi098 bash[23790]: cluster 2024-04-02T04:37:55.086292+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:37:58 smithi098 bash[23790]: cluster 2024-04-02T04:37:57.086836+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:59.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:37:58 smithi069 bash[21528]: cluster 2024-04-02T04:37:57.086836+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:37:58 smithi069 bash[28904]: cluster 2024-04-02T04:37:57.086836+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:37:59.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:37:59.353+0000 7f980541c700 10 client.24143.objecter tick 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: cluster 2024-04-02T04:37:59.087424+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.147995+0000 mon.c (mon.2) 9 : audit 1 from='osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.148377+0000 mon.a (mon.0) 340 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.818101+0000 mon.a (mon.0) 341 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.828010+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.830087+0000 mon.a (mon.0) 343 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.831859+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:01.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:00 smithi098 bash[23790]: audit 2024-04-02T04:38:00.842758+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: cluster 2024-04-02T04:37:59.087424+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.147995+0000 mon.c (mon.2) 9 : audit 1 from='osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.148377+0000 mon.a (mon.0) 340 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.818101+0000 mon.a (mon.0) 341 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.828010+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.830087+0000 mon.a (mon.0) 343 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.831859+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:01.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[21528]: audit 2024-04-02T04:38:00.842758+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: cluster 2024-04-02T04:37:59.087424+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.147995+0000 mon.c (mon.2) 9 : audit 1 from='osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.148377+0000 mon.a (mon.0) 340 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.818101+0000 mon.a (mon.0) 341 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.828010+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.830087+0000 mon.a (mon.0) 343 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.831859+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:01.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:00 smithi069 bash[28904]: audit 2024-04-02T04:38:00.842758+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:02.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[21528]: audit 2024-04-02T04:38:00.963115+0000 mon.a (mon.0) 346 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:38:02.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[21528]: cluster 2024-04-02T04:38:00.968530+0000 mon.a (mon.0) 347 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-04-02T04:38:02.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[21528]: audit 2024-04-02T04:38:00.969696+0000 mon.a (mon.0) 348 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:02.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[21528]: audit 2024-04-02T04:38:00.969912+0000 mon.c (mon.2) 10 : audit 1 from='osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:02.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[21528]: audit 2024-04-02T04:38:00.978287+0000 mon.a (mon.0) 349 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:02.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[28904]: audit 2024-04-02T04:38:00.963115+0000 mon.a (mon.0) 346 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:38:02.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[28904]: cluster 2024-04-02T04:38:00.968530+0000 mon.a (mon.0) 347 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-04-02T04:38:02.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[28904]: audit 2024-04-02T04:38:00.969696+0000 mon.a (mon.0) 348 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:02.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[28904]: audit 2024-04-02T04:38:00.969912+0000 mon.c (mon.2) 10 : audit 1 from='osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:02.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:01 smithi069 bash[28904]: audit 2024-04-02T04:38:00.978287+0000 mon.a (mon.0) 349 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:01 smithi098 bash[23790]: audit 2024-04-02T04:38:00.963115+0000 mon.a (mon.0) 346 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:38:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:01 smithi098 bash[23790]: cluster 2024-04-02T04:38:00.968530+0000 mon.a (mon.0) 347 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-04-02T04:38:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:01 smithi098 bash[23790]: audit 2024-04-02T04:38:00.969696+0000 mon.a (mon.0) 348 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:01 smithi098 bash[23790]: audit 2024-04-02T04:38:00.969912+0000 mon.c (mon.2) 10 : audit 1 from='osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:01 smithi098 bash[23790]: audit 2024-04-02T04:38:00.978287+0000 mon.a (mon.0) 349 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[21528]: cluster 2024-04-02T04:38:01.087910+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[21528]: audit 2024-04-02T04:38:01.979075+0000 mon.a (mon.0) 350 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:38:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[21528]: cluster 2024-04-02T04:38:01.986156+0000 mon.a (mon.0) 351 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-04-02T04:38:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[21528]: audit 2024-04-02T04:38:01.987004+0000 mon.a (mon.0) 352 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[21528]: audit 2024-04-02T04:38:01.994199+0000 mon.a (mon.0) 353 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:03.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[21528]: audit 2024-04-02T04:38:02.457582+0000 mon.a (mon.0) 354 : audit 1 from='osd.1 ' entity='osd.1' 2024-04-02T04:38:03.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[28904]: cluster 2024-04-02T04:38:01.087910+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:03.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[28904]: audit 2024-04-02T04:38:01.979075+0000 mon.a (mon.0) 350 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:38:03.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[28904]: cluster 2024-04-02T04:38:01.986156+0000 mon.a (mon.0) 351 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-04-02T04:38:03.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[28904]: audit 2024-04-02T04:38:01.987004+0000 mon.a (mon.0) 352 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:03.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[28904]: audit 2024-04-02T04:38:01.994199+0000 mon.a (mon.0) 353 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:03.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:02 smithi069 bash[28904]: audit 2024-04-02T04:38:02.457582+0000 mon.a (mon.0) 354 : audit 1 from='osd.1 ' entity='osd.1' 2024-04-02T04:38:03.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:02 smithi098 bash[23790]: cluster 2024-04-02T04:38:01.087910+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:03.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:02 smithi098 bash[23790]: audit 2024-04-02T04:38:01.979075+0000 mon.a (mon.0) 350 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:38:03.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:02 smithi098 bash[23790]: cluster 2024-04-02T04:38:01.986156+0000 mon.a (mon.0) 351 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-04-02T04:38:03.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:02 smithi098 bash[23790]: audit 2024-04-02T04:38:01.987004+0000 mon.a (mon.0) 352 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:03.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:02 smithi098 bash[23790]: audit 2024-04-02T04:38:01.994199+0000 mon.a (mon.0) 353 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:03.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:02 smithi098 bash[23790]: audit 2024-04-02T04:38:02.457582+0000 mon.a (mon.0) 354 : audit 1 from='osd.1 ' entity='osd.1' 2024-04-02T04:38:04.219 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 1 on host 'smithi069' 2024-04-02T04:38:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.213+0000 7f97fcff9700 1 -- 172.21.15.69:0/1727237349 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f98000008d0 con 0x7f97ec06f450 2024-04-02T04:38:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.213+0000 7f9805c1d700 10 librados: watch_flush enter 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 10 librados: watch_flush exit 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 20 client.24143.objecter shutdown clearing up homeless session... 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 10 client.24143.objecter successfully canceled tick 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f97ec06f450 msgr2=0x7f97ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f97ec06f450 0x7f97ec071910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009680 tx=0x7f97e8009260 comp rx=0 tx=0).stop 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 msgr2=0x7f9800197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:04.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 0x7f9800197a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c9a0 tx=0x7f97f000ccb0 comp rx=0 tx=0).stop 2024-04-02T04:38:04.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 shutdown_connections 2024-04-02T04:38:04.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9800108010 0x7f980019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:04.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f97ec06f450 0x7f97ec071910 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:04.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98000fff00 0x7f9800197a30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:04.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 --2- 172.21.15.69:0/1727237349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98001008f0 0x7f9800197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:04.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 >> 172.21.15.69:0/1727237349 conn(0x7f98000fbaa0 msgr2=0x7f98000fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:38:04.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 shutdown_connections 2024-04-02T04:38:04.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 -- 172.21.15.69:0/1727237349 wait complete. 2024-04-02T04:38:04.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:04.217+0000 7f9805c1d700 1 librados: shutdown 2024-04-02T04:38:04.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[21528]: cluster 2024-04-02T04:38:01.164148+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[21528]: cluster 2024-04-02T04:38:01.164266+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[21528]: audit 2024-04-02T04:38:02.991904+0000 mon.a (mon.0) 355 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[21528]: cluster 2024-04-02T04:38:03.458490+0000 mon.a (mon.0) 356 : cluster 1 osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] boot 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[21528]: cluster 2024-04-02T04:38:03.458541+0000 mon.a (mon.0) 357 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[21528]: audit 2024-04-02T04:38:03.458661+0000 mon.a (mon.0) 358 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[28904]: cluster 2024-04-02T04:38:01.164148+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[28904]: cluster 2024-04-02T04:38:01.164266+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[28904]: audit 2024-04-02T04:38:02.991904+0000 mon.a (mon.0) 355 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[28904]: cluster 2024-04-02T04:38:03.458490+0000 mon.a (mon.0) 356 : cluster 1 osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] boot 2024-04-02T04:38:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[28904]: cluster 2024-04-02T04:38:03.458541+0000 mon.a (mon.0) 357 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-04-02T04:38:04.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:03 smithi069 bash[28904]: audit 2024-04-02T04:38:03.458661+0000 mon.a (mon.0) 358 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:04.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:03 smithi098 bash[23790]: cluster 2024-04-02T04:38:01.164148+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:04.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:03 smithi098 bash[23790]: cluster 2024-04-02T04:38:01.164266+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:04.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:03 smithi098 bash[23790]: audit 2024-04-02T04:38:02.991904+0000 mon.a (mon.0) 355 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:04.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:03 smithi098 bash[23790]: cluster 2024-04-02T04:38:03.458490+0000 mon.a (mon.0) 356 : cluster 1 osd.1 [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] boot 2024-04-02T04:38:04.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:03 smithi098 bash[23790]: cluster 2024-04-02T04:38:03.458541+0000 mon.a (mon.0) 357 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-04-02T04:38:04.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:03 smithi098 bash[23790]: audit 2024-04-02T04:38:03.458661+0000 mon.a (mon.0) 358 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:04.910 DEBUG:teuthology.orchestra.run.smithi069:osd.1> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.1.service 2024-04-02T04:38:04.913 INFO:tasks.cephadm:Deploying osd.2 on smithi069 with /dev/vg_nvme/lv_2... 2024-04-02T04:38:04.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T04:38:05.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[21528]: cluster 2024-04-02T04:38:03.088317+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v100: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:05.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[21528]: audit 2024-04-02T04:38:04.201374+0000 mon.a (mon.0) 359 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:05.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[21528]: audit 2024-04-02T04:38:04.209387+0000 mon.a (mon.0) 360 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[21528]: audit 2024-04-02T04:38:04.235516+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:05.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[28904]: cluster 2024-04-02T04:38:03.088317+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v100: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:05.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[28904]: audit 2024-04-02T04:38:04.201374+0000 mon.a (mon.0) 359 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:05.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[28904]: audit 2024-04-02T04:38:04.209387+0000 mon.a (mon.0) 360 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:05.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:04 smithi069 bash[28904]: audit 2024-04-02T04:38:04.235516+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:04 smithi098 bash[23790]: cluster 2024-04-02T04:38:03.088317+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v100: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:04 smithi098 bash[23790]: audit 2024-04-02T04:38:04.201374+0000 mon.a (mon.0) 359 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:04 smithi098 bash[23790]: audit 2024-04-02T04:38:04.209387+0000 mon.a (mon.0) 360 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:04 smithi098 bash[23790]: audit 2024-04-02T04:38:04.235516+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:06.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:06 smithi069 bash[21528]: cluster 2024-04-02T04:38:05.007348+0000 mon.a (mon.0) 362 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-04-02T04:38:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:06 smithi069 bash[28904]: cluster 2024-04-02T04:38:05.007348+0000 mon.a (mon.0) 362 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-04-02T04:38:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:06 smithi098 bash[23790]: cluster 2024-04-02T04:38:05.007348+0000 mon.a (mon.0) 362 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-04-02T04:38:07.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:07 smithi069 bash[21528]: cluster 2024-04-02T04:38:05.088755+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:07.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:07 smithi069 bash[28904]: cluster 2024-04-02T04:38:05.088755+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:07 smithi098 bash[23790]: cluster 2024-04-02T04:38:05.088755+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:08.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:08 smithi069 bash[21528]: cluster 2024-04-02T04:38:07.089321+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:08.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:08 smithi069 bash[28904]: cluster 2024-04-02T04:38:07.089321+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:08.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:08 smithi098 bash[23790]: cluster 2024-04-02T04:38:07.089321+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:09.198 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:38:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:10 smithi069 bash[28904]: cluster 2024-04-02T04:38:09.089861+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:10 smithi069 bash[21528]: cluster 2024-04-02T04:38:09.089861+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:10 smithi098 bash[23790]: cluster 2024-04-02T04:38:09.089861+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:12.937 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T04:38:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T04:38:12.938 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-02T04:38:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-02T04:38:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136285 s, 769 MB/s 2024-04-02T04:38:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-02T04:38:12.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:12 smithi069 bash[21528]: cluster 2024-04-02T04:38:11.090380+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:12.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:12 smithi069 bash[28904]: cluster 2024-04-02T04:38:11.090380+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:12 smithi098 bash[23790]: cluster 2024-04-02T04:38:11.090380+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:13.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi069:vg_nvme/lv_2 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: cluster 2024-04-02T04:38:13.090703+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.133250+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.141283+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.143206+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.144388+0000 mon.a (mon.0) 366 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.151326+0000 mon.a (mon.0) 367 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.155501+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:14.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.156846+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:14.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:14 smithi098 bash[23790]: audit 2024-04-02T04:38:14.165905+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: cluster 2024-04-02T04:38:13.090703+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:15.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.133250+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.141283+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.143206+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.144388+0000 mon.a (mon.0) 366 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.151326+0000 mon.a (mon.0) 367 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.155501+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.156846+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[21528]: audit 2024-04-02T04:38:14.165905+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: cluster 2024-04-02T04:38:13.090703+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.133250+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.141283+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.143206+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:15.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.144388+0000 mon.a (mon.0) 366 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:15.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.151326+0000 mon.a (mon.0) 367 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.155501+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:15.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.156846+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:15.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:14 smithi069 bash[28904]: audit 2024-04-02T04:38:14.165905+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:15 smithi098 bash[23790]: cephadm 2024-04-02T04:38:14.125975+0000 mgr.y (mgr.14152) 141 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:38:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:15 smithi098 bash[23790]: cephadm 2024-04-02T04:38:14.145198+0000 mgr.y (mgr.14152) 142 : cephadm 1 Adjusting osd_memory_target on smithi069 to 8120M 2024-04-02T04:38:16.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:15 smithi069 bash[21528]: cephadm 2024-04-02T04:38:14.125975+0000 mgr.y (mgr.14152) 141 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:38:16.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:15 smithi069 bash[21528]: cephadm 2024-04-02T04:38:14.145198+0000 mgr.y (mgr.14152) 142 : cephadm 1 Adjusting osd_memory_target on smithi069 to 8120M 2024-04-02T04:38:16.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:15 smithi069 bash[28904]: cephadm 2024-04-02T04:38:14.125975+0000 mgr.y (mgr.14152) 141 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:38:16.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:15 smithi069 bash[28904]: cephadm 2024-04-02T04:38:14.145198+0000 mgr.y (mgr.14152) 142 : cephadm 1 Adjusting osd_memory_target on smithi069 to 8120M 2024-04-02T04:38:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:16 smithi069 bash[21528]: cluster 2024-04-02T04:38:15.091175+0000 mgr.y (mgr.14152) 143 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:17.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:16 smithi069 bash[28904]: cluster 2024-04-02T04:38:15.091175+0000 mgr.y (mgr.14152) 143 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:16 smithi098 bash[23790]: cluster 2024-04-02T04:38:15.091175+0000 mgr.y (mgr.14152) 143 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:18.456 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:38:19.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:18 smithi069 bash[21528]: cluster 2024-04-02T04:38:17.091643+0000 mgr.y (mgr.14152) 144 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:18 smithi069 bash[28904]: cluster 2024-04-02T04:38:17.091643+0000 mgr.y (mgr.14152) 144 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:19.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:18 smithi098 bash[23790]: cluster 2024-04-02T04:38:17.091643+0000 mgr.y (mgr.14152) 144 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- 172.21.15.69:0/3348733614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 msgr2=0x7fd65c105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3348733614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 0x7fd65c105b40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd64400b0a0 tx=0x7fd64400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- 172.21.15.69:0/3348733614 shutdown_connections 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3348733614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd65c106080 0x7fd65c10c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3348733614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c104900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3348733614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 0x7fd65c105b40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- 172.21.15.69:0/3348733614 >> 172.21.15.69:0/3348733614 conn(0x7fd65c0ffaa0 msgr2=0x7fd65c101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- 172.21.15.69:0/3348733614 shutdown_connections 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- 172.21.15.69:0/3348733614 wait complete. 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 librados: starting msgr at 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 librados: starting objecter 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 Processor -- start 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- start start 2024-04-02T04:38:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 librados: setting wanted keys 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 librados: calling monclient init 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 0x7fd65c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd65c106080 0x7fd65c1a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c10f580 con 0x7fd65c1044e0 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c10f400 con 0x7fd65c106080 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd661b4f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c10f700 con 0x7fd65c1056e0 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd65b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.900+0000 7fd65b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53860/0 (socket says 172.21.15.69:53860) 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 -- 172.21.15.69:0/3193309609 learned_addr learned my addr 172.21.15.69:0/3193309609 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65affd700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 0x7fd65c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65bfff700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd65c106080 0x7fd65c1a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:38:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 -- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 msgr2=0x7fd65c19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 0x7fd65c19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 -- 172.21.15.69:0/3193309609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd65c106080 msgr2=0x7fd65c1a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd65c106080 0x7fd65c1a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 -- 172.21.15.69:0/3193309609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c1a1940 con 0x7fd65c1044e0 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65b7fe700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c199cf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00e970 tx=0x7fd64c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 10 client.?.objecter ms_handle_connect 0x7fd65c1044e0 2024-04-02T04:38:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:38:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c004220 con 0x7fd65c1044e0 2024-04-02T04:38:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd64c0043c0 con 0x7fd65c1044e0 2024-04-02T04:38:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c003c30 con 0x7fd65c1044e0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c1a1c30 con 0x7fd65c1044e0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd661b4f700 10 client.14262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c1a2120 con 0x7fd65c1044e0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd661b4f700 1 librados: init done 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65c04ff00 con 0x7fd65c1044e0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 14) ==== 91718+0+0 (secure 0 0 0) 0x7fd64c00c7e0 con 0x7fd65c1044e0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fd64806f450 0x7fd648071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2199+0+0 (secure 0 0 0) 0x7fd64c08ca30 con 0x7fd65c1044e0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 10 client.14262.objecter ms_dispatch 0x7fd65c001550 osd_map(14..14 src has 1..14) 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 3 client.14262.objecter handle_osd_map got epochs [14,14] > 0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 3 client.14262.objecter handle_osd_map decoding full epoch 14 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65affd700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fd64806f450 0x7fd648071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:38:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 20 client.14262.objecter dump_active .. 0 homeless 2024-04-02T04:38:20.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd658ff9700 10 client.14262.objecter ms_handle_connect 0x7fd64806f450 2024-04-02T04:38:20.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.904+0000 7fd65affd700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fd64806f450 0x7fd648071910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd644006130 tx=0x7fd6440060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:38:20.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:20.908+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd64c05b650 con 0x7fd65c1044e0 2024-04-02T04:38:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:20 smithi069 bash[21528]: cluster 2024-04-02T04:38:19.092123+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:21.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:20 smithi069 bash[28904]: cluster 2024-04-02T04:38:19.092123+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:21.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:21.128+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd65c062680 con 0x7fd64806f450 2024-04-02T04:38:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:20 smithi098 bash[23790]: cluster 2024-04-02T04:38:19.092123+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:22.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:21 smithi069 bash[21528]: audit 2024-04-02T04:38:21.137760+0000 mon.a (mon.0) 371 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:22.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:21 smithi069 bash[21528]: audit 2024-04-02T04:38:21.140566+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:22.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:21 smithi069 bash[21528]: audit 2024-04-02T04:38:21.141555+0000 mon.a (mon.0) 373 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:22.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:21 smithi069 bash[28904]: audit 2024-04-02T04:38:21.137760+0000 mon.a (mon.0) 371 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:22.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:21 smithi069 bash[28904]: audit 2024-04-02T04:38:21.140566+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:22.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:21 smithi069 bash[28904]: audit 2024-04-02T04:38:21.141555+0000 mon.a (mon.0) 373 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:22.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:21 smithi098 bash[23790]: audit 2024-04-02T04:38:21.137760+0000 mon.a (mon.0) 371 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:22.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:21 smithi098 bash[23790]: audit 2024-04-02T04:38:21.140566+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:22.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:21 smithi098 bash[23790]: audit 2024-04-02T04:38:21.141555+0000 mon.a (mon.0) 373 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:22 smithi069 bash[21528]: cluster 2024-04-02T04:38:21.092594+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:22 smithi069 bash[21528]: audit 2024-04-02T04:38:21.134912+0000 mgr.y (mgr.14152) 147 : audit 0 from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:23.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:22 smithi069 bash[28904]: cluster 2024-04-02T04:38:21.092594+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:23.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:22 smithi069 bash[28904]: audit 2024-04-02T04:38:21.134912+0000 mgr.y (mgr.14152) 147 : audit 0 from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:22 smithi098 bash[23790]: cluster 2024-04-02T04:38:21.092594+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:22 smithi098 bash[23790]: audit 2024-04-02T04:38:21.134912+0000 mgr.y (mgr.14152) 147 : audit 0 from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:24 smithi069 bash[21528]: cluster 2024-04-02T04:38:23.093009+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:24 smithi069 bash[28904]: cluster 2024-04-02T04:38:23.093009+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:24 smithi098 bash[23790]: cluster 2024-04-02T04:38:23.093009+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:25.908+0000 7fd66134e700 10 client.14262.objecter tick 2024-04-02T04:38:27.017 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:26 smithi069 bash[21528]: cluster 2024-04-02T04:38:25.093494+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:27.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:26 smithi069 bash[28904]: cluster 2024-04-02T04:38:25.093494+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:27.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:26 smithi098 bash[23790]: cluster 2024-04-02T04:38:25.093494+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:28 smithi069 bash[21528]: cluster 2024-04-02T04:38:27.093904+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:29.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:28 smithi069 bash[28904]: cluster 2024-04-02T04:38:27.093904+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:29.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:28 smithi098 bash[23790]: cluster 2024-04-02T04:38:27.093904+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[21528]: audit 2024-04-02T04:38:29.392259+0000 mon.c (mon.2) 11 : audit 1 from='client.? 172.21.15.69:0/1946181983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]: dispatch 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[21528]: audit 2024-04-02T04:38:29.392744+0000 mon.a (mon.0) 374 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]: dispatch 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[21528]: audit 2024-04-02T04:38:29.395643+0000 mon.a (mon.0) 375 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]': finished 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[21528]: cluster 2024-04-02T04:38:29.402363+0000 mon.a (mon.0) 376 : cluster 0 osdmap e15: 3 total, 2 up, 3 in 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[21528]: audit 2024-04-02T04:38:29.402764+0000 mon.a (mon.0) 377 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[28904]: audit 2024-04-02T04:38:29.392259+0000 mon.c (mon.2) 11 : audit 1 from='client.? 172.21.15.69:0/1946181983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]: dispatch 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[28904]: audit 2024-04-02T04:38:29.392744+0000 mon.a (mon.0) 374 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]: dispatch 2024-04-02T04:38:30.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[28904]: audit 2024-04-02T04:38:29.395643+0000 mon.a (mon.0) 375 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]': finished 2024-04-02T04:38:30.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[28904]: cluster 2024-04-02T04:38:29.402363+0000 mon.a (mon.0) 376 : cluster 0 osdmap e15: 3 total, 2 up, 3 in 2024-04-02T04:38:30.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:29 smithi069 bash[28904]: audit 2024-04-02T04:38:29.402764+0000 mon.a (mon.0) 377 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:30.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:29 smithi098 bash[23790]: audit 2024-04-02T04:38:29.392259+0000 mon.c (mon.2) 11 : audit 1 from='client.? 172.21.15.69:0/1946181983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]: dispatch 2024-04-02T04:38:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:29 smithi098 bash[23790]: audit 2024-04-02T04:38:29.392744+0000 mon.a (mon.0) 374 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]: dispatch 2024-04-02T04:38:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:29 smithi098 bash[23790]: audit 2024-04-02T04:38:29.395643+0000 mon.a (mon.0) 375 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7"}]': finished 2024-04-02T04:38:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:29 smithi098 bash[23790]: cluster 2024-04-02T04:38:29.402363+0000 mon.a (mon.0) 376 : cluster 0 osdmap e15: 3 total, 2 up, 3 in 2024-04-02T04:38:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:29 smithi098 bash[23790]: audit 2024-04-02T04:38:29.402764+0000 mon.a (mon.0) 377 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:30.912+0000 7fd66134e700 10 client.14262.objecter tick 2024-04-02T04:38:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:30 smithi069 bash[21528]: cluster 2024-04-02T04:38:29.094354+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:30 smithi069 bash[21528]: audit 2024-04-02T04:38:30.147216+0000 mon.c (mon.2) 12 : audit 0 from='client.? 172.21.15.69:0/244415485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:30 smithi069 bash[28904]: cluster 2024-04-02T04:38:29.094354+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:30 smithi069 bash[28904]: audit 2024-04-02T04:38:30.147216+0000 mon.c (mon.2) 12 : audit 0 from='client.? 172.21.15.69:0/244415485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:31.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:30 smithi098 bash[23790]: cluster 2024-04-02T04:38:29.094354+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:31.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:30 smithi098 bash[23790]: audit 2024-04-02T04:38:30.147216+0000 mon.c (mon.2) 12 : audit 0 from='client.? 172.21.15.69:0/244415485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:32.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:31 smithi098 bash[23790]: cluster 2024-04-02T04:38:30.884741+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:32.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:31 smithi098 bash[23790]: cluster 2024-04-02T04:38:30.884834+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:32.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:31 smithi069 bash[21528]: cluster 2024-04-02T04:38:30.884741+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:32.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:31 smithi069 bash[21528]: cluster 2024-04-02T04:38:30.884834+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:32.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:31 smithi069 bash[28904]: cluster 2024-04-02T04:38:30.884741+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:32.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:31 smithi069 bash[28904]: cluster 2024-04-02T04:38:30.884834+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:32 smithi069 bash[21528]: cluster 2024-04-02T04:38:31.094741+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:32 smithi069 bash[21528]: cluster 2024-04-02T04:38:31.787511+0000 mon.a (mon.0) 378 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-04-02T04:38:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:32 smithi069 bash[21528]: audit 2024-04-02T04:38:31.787737+0000 mon.a (mon.0) 379 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:33.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:32 smithi069 bash[28904]: cluster 2024-04-02T04:38:31.094741+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:33.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:32 smithi069 bash[28904]: cluster 2024-04-02T04:38:31.787511+0000 mon.a (mon.0) 378 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-04-02T04:38:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:32 smithi069 bash[28904]: audit 2024-04-02T04:38:31.787737+0000 mon.a (mon.0) 379 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:32 smithi098 bash[23790]: cluster 2024-04-02T04:38:31.094741+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:33.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:32 smithi098 bash[23790]: cluster 2024-04-02T04:38:31.787511+0000 mon.a (mon.0) 378 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-04-02T04:38:33.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:32 smithi098 bash[23790]: audit 2024-04-02T04:38:31.787737+0000 mon.a (mon.0) 379 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:34 smithi098 bash[23790]: cluster 2024-04-02T04:38:33.095163+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:34 smithi069 bash[21528]: cluster 2024-04-02T04:38:33.095163+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:34 smithi069 bash[28904]: cluster 2024-04-02T04:38:33.095163+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:35.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:35.916+0000 7fd66134e700 10 client.14262.objecter tick 2024-04-02T04:38:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:36 smithi098 bash[23790]: cluster 2024-04-02T04:38:35.095667+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:36 smithi069 bash[21528]: cluster 2024-04-02T04:38:35.095667+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:36 smithi069 bash[28904]: cluster 2024-04-02T04:38:35.095667+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:39.062 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:38 smithi069 bash[21528]: cluster 2024-04-02T04:38:37.096146+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:39.063 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:38 smithi069 bash[28904]: cluster 2024-04-02T04:38:37.096146+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:39.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:38 smithi098 bash[23790]: cluster 2024-04-02T04:38:37.096146+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:40.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:40.920+0000 7fd66134e700 10 client.14262.objecter tick 2024-04-02T04:38:41.130 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:40 smithi069 bash[21528]: cluster 2024-04-02T04:38:39.096557+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:41.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:40 smithi069 bash[28904]: cluster 2024-04-02T04:38:39.096557+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:40 smithi098 bash[23790]: cluster 2024-04-02T04:38:39.096557+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:42.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:41 smithi069 bash[21528]: audit 2024-04-02T04:38:41.646455+0000 mon.a (mon.0) 380 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:38:42.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:41 smithi069 bash[21528]: audit 2024-04-02T04:38:41.648215+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:42.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:41 smithi069 bash[28904]: audit 2024-04-02T04:38:41.646455+0000 mon.a (mon.0) 380 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:38:42.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:41 smithi069 bash[28904]: audit 2024-04-02T04:38:41.648215+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:42.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:41 smithi098 bash[23790]: audit 2024-04-02T04:38:41.646455+0000 mon.a (mon.0) 380 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:38:42.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:41 smithi098 bash[23790]: audit 2024-04-02T04:38:41.648215+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:43.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:42 smithi069 bash[21528]: cluster 2024-04-02T04:38:41.097018+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:43.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:42 smithi069 bash[21528]: cephadm 2024-04-02T04:38:41.649729+0000 mgr.y (mgr.14152) 158 : cephadm 1 Deploying daemon osd.2 on smithi069 2024-04-02T04:38:43.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:42 smithi069 bash[28904]: cluster 2024-04-02T04:38:41.097018+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:43.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:42 smithi069 bash[28904]: cephadm 2024-04-02T04:38:41.649729+0000 mgr.y (mgr.14152) 158 : cephadm 1 Deploying daemon osd.2 on smithi069 2024-04-02T04:38:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:42 smithi098 bash[23790]: cluster 2024-04-02T04:38:41.097018+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:42 smithi098 bash[23790]: cephadm 2024-04-02T04:38:41.649729+0000 mgr.y (mgr.14152) 158 : cephadm 1 Deploying daemon osd.2 on smithi069 2024-04-02T04:38:45.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:44 smithi069 bash[21528]: cluster 2024-04-02T04:38:43.097460+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:45.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:44 smithi069 bash[28904]: cluster 2024-04-02T04:38:43.097460+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:44 smithi098 bash[23790]: cluster 2024-04-02T04:38:43.097460+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:45.919 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:45.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:45.920 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:45.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:45.920 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:45.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:45.924+0000 7fd66134e700 10 client.14262.objecter tick 2024-04-02T04:38:46.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:46.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:46.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:46.283 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:46.284 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:38:45 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:38:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:46 smithi098 bash[23790]: cluster 2024-04-02T04:38:45.097951+0000 mgr.y (mgr.14152) 160 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:47.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:46 smithi098 bash[23790]: audit 2024-04-02T04:38:46.062646+0000 mon.a (mon.0) 382 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:47.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:46 smithi098 bash[23790]: audit 2024-04-02T04:38:46.070659+0000 mon.a (mon.0) 383 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:47.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:46 smithi098 bash[23790]: audit 2024-04-02T04:38:46.078786+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[28904]: cluster 2024-04-02T04:38:45.097951+0000 mgr.y (mgr.14152) 160 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[28904]: audit 2024-04-02T04:38:46.062646+0000 mon.a (mon.0) 382 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[28904]: audit 2024-04-02T04:38:46.070659+0000 mon.a (mon.0) 383 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[28904]: audit 2024-04-02T04:38:46.078786+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[21528]: cluster 2024-04-02T04:38:45.097951+0000 mgr.y (mgr.14152) 160 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[21528]: audit 2024-04-02T04:38:46.062646+0000 mon.a (mon.0) 382 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[21528]: audit 2024-04-02T04:38:46.070659+0000 mon.a (mon.0) 383 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:46 smithi069 bash[21528]: audit 2024-04-02T04:38:46.078786+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:48 smithi098 bash[23790]: cluster 2024-04-02T04:38:47.098337+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:49.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:48 smithi069 bash[28904]: cluster 2024-04-02T04:38:47.098337+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:49.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:48 smithi069 bash[21528]: cluster 2024-04-02T04:38:47.098337+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:50.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:50.928+0000 7fd66134e700 10 client.14262.objecter tick 2024-04-02T04:38:51.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:50 smithi098 bash[23790]: cluster 2024-04-02T04:38:49.098792+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:51.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:50 smithi069 bash[21528]: cluster 2024-04-02T04:38:49.098792+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:51.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:50 smithi069 bash[28904]: cluster 2024-04-02T04:38:49.098792+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[21528]: cluster 2024-04-02T04:38:51.099109+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[21528]: audit 2024-04-02T04:38:51.361370+0000 mon.a (mon.0) 385 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[21528]: audit 2024-04-02T04:38:51.366480+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[21528]: audit 2024-04-02T04:38:51.933889+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[21528]: audit 2024-04-02T04:38:51.935261+0000 mon.a (mon.0) 388 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[21528]: audit 2024-04-02T04:38:51.944595+0000 mon.a (mon.0) 389 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[28904]: cluster 2024-04-02T04:38:51.099109+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:52.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[28904]: audit 2024-04-02T04:38:51.361370+0000 mon.a (mon.0) 385 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[28904]: audit 2024-04-02T04:38:51.366480+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[28904]: audit 2024-04-02T04:38:51.933889+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:52.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[28904]: audit 2024-04-02T04:38:51.935261+0000 mon.a (mon.0) 388 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:52.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:52 smithi069 bash[28904]: audit 2024-04-02T04:38:51.944595+0000 mon.a (mon.0) 389 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:52 smithi098 bash[23790]: cluster 2024-04-02T04:38:51.099109+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:52.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:52 smithi098 bash[23790]: audit 2024-04-02T04:38:51.361370+0000 mon.a (mon.0) 385 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:52 smithi098 bash[23790]: audit 2024-04-02T04:38:51.366480+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:52.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:52 smithi098 bash[23790]: audit 2024-04-02T04:38:51.933889+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:52.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:52 smithi098 bash[23790]: audit 2024-04-02T04:38:51.935261+0000 mon.a (mon.0) 388 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:52.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:52 smithi098 bash[23790]: audit 2024-04-02T04:38:51.944595+0000 mon.a (mon.0) 389 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:53.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:53 smithi098 bash[23790]: audit 2024-04-02T04:38:52.485316+0000 mon.c (mon.2) 13 : audit 1 from='osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:53.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:53 smithi098 bash[23790]: audit 2024-04-02T04:38:52.485721+0000 mon.a (mon.0) 390 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:53.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:53 smithi069 bash[21528]: audit 2024-04-02T04:38:52.485316+0000 mon.c (mon.2) 13 : audit 1 from='osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:53.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:53 smithi069 bash[21528]: audit 2024-04-02T04:38:52.485721+0000 mon.a (mon.0) 390 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:53.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:53 smithi069 bash[28904]: audit 2024-04-02T04:38:52.485316+0000 mon.c (mon.2) 13 : audit 1 from='osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:53.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:53 smithi069 bash[28904]: audit 2024-04-02T04:38:52.485721+0000 mon.a (mon.0) 390 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:54.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:54 smithi098 bash[23790]: cluster 2024-04-02T04:38:53.099519+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:54.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:54 smithi098 bash[23790]: audit 2024-04-02T04:38:53.373207+0000 mon.a (mon.0) 391 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:38:54.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:54 smithi098 bash[23790]: cluster 2024-04-02T04:38:53.378479+0000 mon.a (mon.0) 392 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-04-02T04:38:54.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:54 smithi098 bash[23790]: audit 2024-04-02T04:38:53.379147+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:54.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:54 smithi098 bash[23790]: audit 2024-04-02T04:38:53.379194+0000 mon.c (mon.2) 14 : audit 1 from='osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:54.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:54 smithi098 bash[23790]: audit 2024-04-02T04:38:53.380072+0000 mon.a (mon.0) 394 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:54.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.732+0000 7fd658ff9700 1 -- 172.21.15.69:0/3193309609 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd65c062680 con 0x7fd64806f450 2024-04-02T04:38:54.738 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 2 on host 'smithi069' 2024-04-02T04:38:54.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 10 librados: watch_flush enter 2024-04-02T04:38:54.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 10 librados: watch_flush exit 2024-04-02T04:38:54.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 20 client.14262.objecter shutdown clearing up homeless session... 2024-04-02T04:38:54.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 10 client.14262.objecter successfully canceled tick 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fd64806f450 msgr2=0x7fd648071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fd64806f450 0x7fd648071910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd644006130 tx=0x7fd6440060c0 comp rx=0 tx=0).stop 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 msgr2=0x7fd65c199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c199cf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00e970 tx=0x7fd64c00ec80 comp rx=0 tx=0).stop 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 shutdown_connections 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd65c106080 0x7fd65c1a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd65c1044e0 0x7fd65c199cf0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fd64806f450 0x7fd648071910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:54.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 --2- 172.21.15.69:0/3193309609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd65c1056e0 0x7fd65c19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:38:54.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 >> 172.21.15.69:0/3193309609 conn(0x7fd65c0ffaa0 msgr2=0x7fd65c102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:38:54.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 shutdown_connections 2024-04-02T04:38:54.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 -- 172.21.15.69:0/3193309609 wait complete. 2024-04-02T04:38:54.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:38:54.736+0000 7fd661b4f700 1 librados: shutdown 2024-04-02T04:38:54.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[21528]: cluster 2024-04-02T04:38:53.099519+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:54.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[21528]: audit 2024-04-02T04:38:53.373207+0000 mon.a (mon.0) 391 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:38:54.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[21528]: cluster 2024-04-02T04:38:53.378479+0000 mon.a (mon.0) 392 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-04-02T04:38:54.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[21528]: audit 2024-04-02T04:38:53.379147+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:54.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[21528]: audit 2024-04-02T04:38:53.379194+0000 mon.c (mon.2) 14 : audit 1 from='osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:54.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[21528]: audit 2024-04-02T04:38:53.380072+0000 mon.a (mon.0) 394 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:54.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[28904]: cluster 2024-04-02T04:38:53.099519+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:54.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[28904]: audit 2024-04-02T04:38:53.373207+0000 mon.a (mon.0) 391 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:38:54.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[28904]: cluster 2024-04-02T04:38:53.378479+0000 mon.a (mon.0) 392 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-04-02T04:38:54.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[28904]: audit 2024-04-02T04:38:53.379147+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:54.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[28904]: audit 2024-04-02T04:38:53.379194+0000 mon.c (mon.2) 14 : audit 1 from='osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:54.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:54 smithi069 bash[28904]: audit 2024-04-02T04:38:53.380072+0000 mon.a (mon.0) 394 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:38:55.418 DEBUG:teuthology.orchestra.run.smithi069:osd.2> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.2.service 2024-04-02T04:38:55.420 INFO:tasks.cephadm:Deploying osd.3 on smithi069 with /dev/vg_nvme/lv_1... 2024-04-02T04:38:55.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T04:38:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: audit 2024-04-02T04:38:54.375392+0000 mon.a (mon.0) 395 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:38:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: cluster 2024-04-02T04:38:54.381508+0000 mon.a (mon.0) 396 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-04-02T04:38:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: audit 2024-04-02T04:38:54.382465+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: audit 2024-04-02T04:38:54.391719+0000 mon.a (mon.0) 398 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: audit 2024-04-02T04:38:54.718764+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: audit 2024-04-02T04:38:54.719462+0000 mon.a (mon.0) 400 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:55 smithi098 bash[23790]: audit 2024-04-02T04:38:54.729680+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: audit 2024-04-02T04:38:54.375392+0000 mon.a (mon.0) 395 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: cluster 2024-04-02T04:38:54.381508+0000 mon.a (mon.0) 396 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: audit 2024-04-02T04:38:54.382465+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: audit 2024-04-02T04:38:54.391719+0000 mon.a (mon.0) 398 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: audit 2024-04-02T04:38:54.718764+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: audit 2024-04-02T04:38:54.719462+0000 mon.a (mon.0) 400 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[21528]: audit 2024-04-02T04:38:54.729680+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: audit 2024-04-02T04:38:54.375392+0000 mon.a (mon.0) 395 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: cluster 2024-04-02T04:38:54.381508+0000 mon.a (mon.0) 396 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: audit 2024-04-02T04:38:54.382465+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: audit 2024-04-02T04:38:54.391719+0000 mon.a (mon.0) 398 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: audit 2024-04-02T04:38:54.718764+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: audit 2024-04-02T04:38:54.719462+0000 mon.a (mon.0) 400 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:55 smithi069 bash[28904]: audit 2024-04-02T04:38:54.729680+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:38:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: cluster 2024-04-02T04:38:53.527683+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: cluster 2024-04-02T04:38:53.527765+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: cluster 2024-04-02T04:38:55.100075+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: audit 2024-04-02T04:38:55.390368+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: cluster 2024-04-02T04:38:55.398174+0000 mon.a (mon.0) 403 : cluster 1 osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] boot 2024-04-02T04:38:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: cluster 2024-04-02T04:38:55.398243+0000 mon.a (mon.0) 404 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-04-02T04:38:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:56 smithi098 bash[23790]: audit 2024-04-02T04:38:55.398436+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: cluster 2024-04-02T04:38:53.527683+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: cluster 2024-04-02T04:38:53.527765+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: cluster 2024-04-02T04:38:55.100075+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: audit 2024-04-02T04:38:55.390368+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: cluster 2024-04-02T04:38:55.398174+0000 mon.a (mon.0) 403 : cluster 1 osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] boot 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: cluster 2024-04-02T04:38:55.398243+0000 mon.a (mon.0) 404 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-04-02T04:38:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[21528]: audit 2024-04-02T04:38:55.398436+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: cluster 2024-04-02T04:38:53.527683+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: cluster 2024-04-02T04:38:53.527765+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: cluster 2024-04-02T04:38:55.100075+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: audit 2024-04-02T04:38:55.390368+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: cluster 2024-04-02T04:38:55.398174+0000 mon.a (mon.0) 403 : cluster 1 osd.2 [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] boot 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: cluster 2024-04-02T04:38:55.398243+0000 mon.a (mon.0) 404 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-04-02T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:56 smithi069 bash[28904]: audit 2024-04-02T04:38:55.398436+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:57.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:57 smithi098 bash[23790]: cluster 2024-04-02T04:38:56.401534+0000 mon.a (mon.0) 406 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-04-02T04:38:58.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:57 smithi069 bash[21528]: cluster 2024-04-02T04:38:56.401534+0000 mon.a (mon.0) 406 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-04-02T04:38:58.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:57 smithi069 bash[28904]: cluster 2024-04-02T04:38:56.401534+0000 mon.a (mon.0) 406 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-04-02T04:38:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:58 smithi098 bash[23790]: cluster 2024-04-02T04:38:57.100540+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v135: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:38:58.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:58 smithi098 bash[23790]: audit 2024-04-02T04:38:57.236375+0000 mon.a (mon.0) 407 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:58.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:58 smithi098 bash[23790]: audit 2024-04-02T04:38:57.723614+0000 mon.a (mon.0) 408 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-02T04:38:58.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:58 smithi098 bash[23790]: cluster 2024-04-02T04:38:57.726681+0000 mon.a (mon.0) 409 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-04-02T04:38:58.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:58 smithi098 bash[23790]: audit 2024-04-02T04:38:57.727471+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[21528]: cluster 2024-04-02T04:38:57.100540+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v135: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[21528]: audit 2024-04-02T04:38:57.236375+0000 mon.a (mon.0) 407 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[21528]: audit 2024-04-02T04:38:57.723614+0000 mon.a (mon.0) 408 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[21528]: cluster 2024-04-02T04:38:57.726681+0000 mon.a (mon.0) 409 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[21528]: audit 2024-04-02T04:38:57.727471+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[28904]: cluster 2024-04-02T04:38:57.100540+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v135: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[28904]: audit 2024-04-02T04:38:57.236375+0000 mon.a (mon.0) 407 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[28904]: audit 2024-04-02T04:38:57.723614+0000 mon.a (mon.0) 408 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[28904]: cluster 2024-04-02T04:38:57.726681+0000 mon.a (mon.0) 409 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-04-02T04:38:59.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:58 smithi069 bash[28904]: audit 2024-04-02T04:38:57.727471+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:39:00.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:59 smithi069 bash[28904]: audit 2024-04-02T04:38:58.726019+0000 mon.a (mon.0) 411 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:39:00.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:38:59 smithi069 bash[28904]: cluster 2024-04-02T04:38:58.731847+0000 mon.a (mon.0) 412 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-04-02T04:39:00.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:59 smithi069 bash[21528]: audit 2024-04-02T04:38:58.726019+0000 mon.a (mon.0) 411 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:39:00.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:38:59 smithi069 bash[21528]: cluster 2024-04-02T04:38:58.731847+0000 mon.a (mon.0) 412 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-04-02T04:39:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:59 smithi098 bash[23790]: audit 2024-04-02T04:38:58.726019+0000 mon.a (mon.0) 411 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:39:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:38:59 smithi098 bash[23790]: cluster 2024-04-02T04:38:58.731847+0000 mon.a (mon.0) 412 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-04-02T04:39:01.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:00 smithi069 bash[21528]: cluster 2024-04-02T04:38:59.101037+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v138: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:01.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:00 smithi069 bash[21528]: cluster 2024-04-02T04:38:59.739888+0000 mon.a (mon.0) 413 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-04-02T04:39:01.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:00 smithi069 bash[21528]: audit 2024-04-02T04:39:00.441808+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:00 smithi069 bash[28904]: cluster 2024-04-02T04:38:59.101037+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v138: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:00 smithi069 bash[28904]: cluster 2024-04-02T04:38:59.739888+0000 mon.a (mon.0) 413 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-04-02T04:39:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:00 smithi069 bash[28904]: audit 2024-04-02T04:39:00.441808+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:00 smithi098 bash[23790]: cluster 2024-04-02T04:38:59.101037+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v138: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:00 smithi098 bash[23790]: cluster 2024-04-02T04:38:59.739888+0000 mon.a (mon.0) 413 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-04-02T04:39:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:00 smithi098 bash[23790]: audit 2024-04-02T04:39:00.441808+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:03.436 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T04:39:03.436 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T04:39:03.437 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-02T04:39:03.437 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-02T04:39:03.437 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135029 s, 777 MB/s 2024-04-02T04:39:03.437 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-02T04:39:03.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:03 smithi069 bash[21528]: cluster 2024-04-02T04:39:01.101472+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v140: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:03.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:03 smithi069 bash[21528]: cluster 2024-04-02T04:39:01.754750+0000 mon.a (mon.0) 415 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-04-02T04:39:03.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:03 smithi069 bash[21528]: audit 2024-04-02T04:39:02.221407+0000 mon.a (mon.0) 416 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:03.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:03 smithi069 bash[28904]: cluster 2024-04-02T04:39:01.101472+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v140: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:03.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:03 smithi069 bash[28904]: cluster 2024-04-02T04:39:01.754750+0000 mon.a (mon.0) 415 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-04-02T04:39:03.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:03 smithi069 bash[28904]: audit 2024-04-02T04:39:02.221407+0000 mon.a (mon.0) 416 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:03.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:03 smithi098 bash[23790]: cluster 2024-04-02T04:39:01.101472+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v140: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:03.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:03 smithi098 bash[23790]: cluster 2024-04-02T04:39:01.754750+0000 mon.a (mon.0) 415 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-04-02T04:39:03.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:03 smithi098 bash[23790]: audit 2024-04-02T04:39:02.221407+0000 mon.a (mon.0) 416 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:04.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi069:vg_nvme/lv_1 2024-04-02T04:39:05.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: cluster 2024-04-02T04:39:03.101843+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v141: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.130361+0000 mon.a (mon.0) 417 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.131285+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.131525+0000 mon.a (mon.0) 419 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.131701+0000 mon.a (mon.0) 420 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.136645+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.138689+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.138855+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:03.138992+0000 mon.a (mon.0) 423 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:04.036488+0000 mon.b (mon.1) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:05.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:04.097490+0000 mon.a (mon.0) 424 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:04.103072+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:04.103629+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[21528]: audit 2024-04-02T04:39:04.104361+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: cluster 2024-04-02T04:39:03.101843+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v141: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.130361+0000 mon.a (mon.0) 417 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.131285+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.131525+0000 mon.a (mon.0) 419 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.131701+0000 mon.a (mon.0) 420 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.136645+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.138689+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.138855+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:03.138992+0000 mon.a (mon.0) 423 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:04.036488+0000 mon.b (mon.1) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:04.097490+0000 mon.a (mon.0) 424 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:04.103072+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:04.103629+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:05 smithi069 bash[28904]: audit 2024-04-02T04:39:04.104361+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: cluster 2024-04-02T04:39:03.101843+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v141: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.130361+0000 mon.a (mon.0) 417 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.131285+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.131525+0000 mon.a (mon.0) 419 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.131701+0000 mon.a (mon.0) 420 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.136645+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.138689+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.138855+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:03.138992+0000 mon.a (mon.0) 423 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:04.036488+0000 mon.b (mon.1) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:05.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:04.097490+0000 mon.a (mon.0) 424 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:05.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:04.103072+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:39:05.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:04.103629+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:39:05.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:05 smithi098 bash[23790]: audit 2024-04-02T04:39:04.104361+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:04.100550+0000 mon.c (mon.2) 15 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: cephadm 2024-04-02T04:39:04.570367+0000 mgr.y (mgr.14152) 170 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.020391+0000 mon.c (mon.2) 16 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.032281+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.041398+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.043050+0000 mon.a (mon.0) 430 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.044063+0000 mon.a (mon.0) 431 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.044995+0000 mon.a (mon.0) 432 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: cephadm 2024-04-02T04:39:05.045706+0000 mgr.y (mgr.14152) 171 : cephadm 1 Adjusting osd_memory_target on smithi069 to 5413M 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.050494+0000 mon.a (mon.0) 433 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.053337+0000 mon.a (mon.0) 434 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.054148+0000 mon.a (mon.0) 435 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: audit 2024-04-02T04:39:05.061644+0000 mon.a (mon.0) 436 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[28904]: cluster 2024-04-02T04:39:05.102231+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:04.100550+0000 mon.c (mon.2) 15 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: cephadm 2024-04-02T04:39:04.570367+0000 mgr.y (mgr.14152) 170 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.020391+0000 mon.c (mon.2) 16 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.032281+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.041398+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.043050+0000 mon.a (mon.0) 430 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.044063+0000 mon.a (mon.0) 431 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.044995+0000 mon.a (mon.0) 432 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: cephadm 2024-04-02T04:39:05.045706+0000 mgr.y (mgr.14152) 171 : cephadm 1 Adjusting osd_memory_target on smithi069 to 5413M 2024-04-02T04:39:06.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.050494+0000 mon.a (mon.0) 433 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.053337+0000 mon.a (mon.0) 434 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:06.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.054148+0000 mon.a (mon.0) 435 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:06.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: audit 2024-04-02T04:39:05.061644+0000 mon.a (mon.0) 436 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:06 smithi069 bash[21528]: cluster 2024-04-02T04:39:05.102231+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:04.100550+0000 mon.c (mon.2) 15 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: cephadm 2024-04-02T04:39:04.570367+0000 mgr.y (mgr.14152) 170 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.020391+0000 mon.c (mon.2) 16 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.032281+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.041398+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.043050+0000 mon.a (mon.0) 430 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.044063+0000 mon.a (mon.0) 431 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.044995+0000 mon.a (mon.0) 432 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: cephadm 2024-04-02T04:39:05.045706+0000 mgr.y (mgr.14152) 171 : cephadm 1 Adjusting osd_memory_target on smithi069 to 5413M 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.050494+0000 mon.a (mon.0) 433 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.053337+0000 mon.a (mon.0) 434 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:06.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.054148+0000 mon.a (mon.0) 435 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:06.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: audit 2024-04-02T04:39:05.061644+0000 mon.a (mon.0) 436 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:06.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:06 smithi098 bash[23790]: cluster 2024-04-02T04:39:05.102231+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:08.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:08 smithi098 bash[23790]: cluster 2024-04-02T04:39:07.102851+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:08.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:08 smithi069 bash[21528]: cluster 2024-04-02T04:39:07.102851+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:08.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:08 smithi069 bash[28904]: cluster 2024-04-02T04:39:07.102851+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:08.880 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:39:11.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:10 smithi069 bash[28904]: cluster 2024-04-02T04:39:09.103358+0000 mgr.y (mgr.14152) 174 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-02T04:39:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:10 smithi069 bash[21528]: cluster 2024-04-02T04:39:09.103358+0000 mgr.y (mgr.14152) 174 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-02T04:39:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:10 smithi098 bash[23790]: cluster 2024-04-02T04:39:09.103358+0000 mgr.y (mgr.14152) 174 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-02T04:39:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/3890909042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 msgr2=0x7f8780105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:39:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- 172.21.15.69:0/3890909042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f8780105150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8770009990 tx=0x7f8770009ca0 comp rx=0 tx=0).stop 2024-04-02T04:39:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/3890909042 shutdown_connections 2024-04-02T04:39:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- 172.21.15.69:0/3890909042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 0x7f878010a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- 172.21.15.69:0/3890909042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f8780105150 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- 172.21.15.69:0/3890909042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/3890909042 >> 172.21.15.69:0/3890909042 conn(0x7f87800fc770 msgr2=0x7f87800febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:39:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/3890909042 shutdown_connections 2024-04-02T04:39:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/3890909042 wait complete. 2024-04-02T04:39:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 librados: starting msgr at 2024-04-02T04:39:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 librados: starting objecter 2024-04-02T04:39:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 Processor -- start 2024-04-02T04:39:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- start start 2024-04-02T04:39:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 librados: setting wanted keys 2024-04-02T04:39:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 librados: calling monclient init 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f87801026c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 0x7f8780101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f878010d3c0 con 0x7f8780105690 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f878010d240 con 0x7f8780107fc0 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f878010d540 con 0x7f8780102d60 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48866/0 (socket says 172.21.15.69:48866) 2024-04-02T04:39:11.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 -- 172.21.15.69:0/2196319187 learned_addr learned my addr 172.21.15.69:0/2196319187 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:39:11.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8786066700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f87801026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:39:11.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8786867700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 0x7f8780101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:39:11.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 -- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 msgr2=0x7f87801026c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f87801026c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 -- 172.21.15.69:0/2196319187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 msgr2=0x7f8780101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 0x7f8780101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 -- 172.21.15.69:0/2196319187 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8780101b00 con 0x7f8780105690 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8786867700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 0x7f8780101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8786066700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f87801026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f8785865700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780100d10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f877c00eb30 tx=0x7f877c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87777fe700 10 client.?.objecter ms_handle_connect 0x7f8780105690 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87777fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c00cc90 con 0x7f8780105690 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f877c00ce30 con 0x7f8780105690 2024-04-02T04:39:11.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87801a1ca0 con 0x7f8780105690 2024-04-02T04:39:11.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c003df0 con 0x7f8780105690 2024-04-02T04:39:11.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 10 client.14298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:39:11.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87801a21b0 con 0x7f8780105690 2024-04-02T04:39:11.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.240+0000 7f87882ca700 1 librados: init done 2024-04-02T04:39:11.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f878010db20 con 0x7f8780105690 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f877c004000 con 0x7f8780105690 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f876c06f4f0 0x7f876c0719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 3017+0+0 (secure 0 0 0) 0x7f877c08d000 con 0x7f8780105690 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 10 client.14298.objecter ms_dispatch 0x7f8780001550 osd_map(24..24 src has 1..24) 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 3 client.14298.objecter handle_osd_map got epochs [24,24] > 0 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 3 client.14298.objecter handle_osd_map decoding full epoch 24 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f8786066700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f876c06f4f0 0x7f876c0719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 20 client.14298.objecter dump_active .. 0 homeless 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f8786066700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f876c06f4f0 0x7f876c0719b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f877000b2e0 tx=0x7f877001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:39:11.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.244+0000 7f87777fe700 10 client.14298.objecter ms_handle_connect 0x7f876c06f4f0 2024-04-02T04:39:11.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.248+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f877c05b790 con 0x7f8780105690 2024-04-02T04:39:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:11.468+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f878002df10 con 0x7f876c06f4f0 2024-04-02T04:39:12.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[28904]: cluster 2024-04-02T04:39:10.169043+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[28904]: cluster 2024-04-02T04:39:10.169169+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[28904]: cluster 2024-04-02T04:39:10.724962+0000 mon.a (mon.0) 437 : cluster 0 osdmap e24: 3 total, 3 up, 3 in 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[28904]: audit 2024-04-02T04:39:11.480301+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[28904]: audit 2024-04-02T04:39:11.486673+0000 mon.a (mon.0) 439 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[28904]: audit 2024-04-02T04:39:11.488021+0000 mon.a (mon.0) 440 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[21528]: cluster 2024-04-02T04:39:10.169043+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[21528]: cluster 2024-04-02T04:39:10.169169+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[21528]: cluster 2024-04-02T04:39:10.724962+0000 mon.a (mon.0) 437 : cluster 0 osdmap e24: 3 total, 3 up, 3 in 2024-04-02T04:39:12.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[21528]: audit 2024-04-02T04:39:11.480301+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:12.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[21528]: audit 2024-04-02T04:39:11.486673+0000 mon.a (mon.0) 439 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:12.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:11 smithi069 bash[21528]: audit 2024-04-02T04:39:11.488021+0000 mon.a (mon.0) 440 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:11 smithi098 bash[23790]: cluster 2024-04-02T04:39:10.169043+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:11 smithi098 bash[23790]: cluster 2024-04-02T04:39:10.169169+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:11 smithi098 bash[23790]: cluster 2024-04-02T04:39:10.724962+0000 mon.a (mon.0) 437 : cluster 0 osdmap e24: 3 total, 3 up, 3 in 2024-04-02T04:39:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:11 smithi098 bash[23790]: audit 2024-04-02T04:39:11.480301+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:11 smithi098 bash[23790]: audit 2024-04-02T04:39:11.486673+0000 mon.a (mon.0) 439 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:12.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:11 smithi098 bash[23790]: audit 2024-04-02T04:39:11.488021+0000 mon.a (mon.0) 440 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:13.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:12 smithi069 bash[21528]: cluster 2024-04-02T04:39:11.103749+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-02T04:39:13.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:12 smithi069 bash[21528]: audit 2024-04-02T04:39:11.473994+0000 mgr.y (mgr.14152) 176 : audit 0 from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:13.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:12 smithi069 bash[28904]: cluster 2024-04-02T04:39:11.103749+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-02T04:39:13.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:12 smithi069 bash[28904]: audit 2024-04-02T04:39:11.473994+0000 mgr.y (mgr.14152) 176 : audit 0 from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:12 smithi098 bash[23790]: cluster 2024-04-02T04:39:11.103749+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-02T04:39:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:12 smithi098 bash[23790]: audit 2024-04-02T04:39:11.473994+0000 mgr.y (mgr.14152) 176 : audit 0 from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:15.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:14 smithi069 bash[21528]: cluster 2024-04-02T04:39:13.104303+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-02T04:39:15.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:14 smithi069 bash[28904]: cluster 2024-04-02T04:39:13.104303+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-02T04:39:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:14 smithi098 bash[23790]: cluster 2024-04-02T04:39:13.104303+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-02T04:39:16.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:16.244+0000 7f8787ac9700 10 client.14298.objecter tick 2024-04-02T04:39:16.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:16 smithi069 bash[21528]: cluster 2024-04-02T04:39:15.104889+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-02T04:39:17.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:16 smithi069 bash[28904]: cluster 2024-04-02T04:39:15.104889+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-02T04:39:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:16 smithi098 bash[23790]: cluster 2024-04-02T04:39:15.104889+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-02T04:39:19.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:18 smithi069 bash[21528]: cluster 2024-04-02T04:39:17.105486+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-02T04:39:19.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:18 smithi069 bash[28904]: cluster 2024-04-02T04:39:17.105486+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-02T04:39:19.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:18 smithi098 bash[23790]: cluster 2024-04-02T04:39:17.105486+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-02T04:39:20.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[21528]: audit 2024-04-02T04:39:19.623946+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.69:0/3649226396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]: dispatch 2024-04-02T04:39:20.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[21528]: audit 2024-04-02T04:39:19.624506+0000 mon.a (mon.0) 441 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]: dispatch 2024-04-02T04:39:20.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[21528]: audit 2024-04-02T04:39:19.626921+0000 mon.a (mon.0) 442 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]': finished 2024-04-02T04:39:20.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[21528]: cluster 2024-04-02T04:39:19.629499+0000 mon.a (mon.0) 443 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-04-02T04:39:20.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[21528]: audit 2024-04-02T04:39:19.629670+0000 mon.a (mon.0) 444 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:20.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[28904]: audit 2024-04-02T04:39:19.623946+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.69:0/3649226396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]: dispatch 2024-04-02T04:39:20.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[28904]: audit 2024-04-02T04:39:19.624506+0000 mon.a (mon.0) 441 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]: dispatch 2024-04-02T04:39:20.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[28904]: audit 2024-04-02T04:39:19.626921+0000 mon.a (mon.0) 442 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]': finished 2024-04-02T04:39:20.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[28904]: cluster 2024-04-02T04:39:19.629499+0000 mon.a (mon.0) 443 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-04-02T04:39:20.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:19 smithi069 bash[28904]: audit 2024-04-02T04:39:19.629670+0000 mon.a (mon.0) 444 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:20.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:19 smithi098 bash[23790]: audit 2024-04-02T04:39:19.623946+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.69:0/3649226396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]: dispatch 2024-04-02T04:39:20.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:19 smithi098 bash[23790]: audit 2024-04-02T04:39:19.624506+0000 mon.a (mon.0) 441 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]: dispatch 2024-04-02T04:39:20.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:19 smithi098 bash[23790]: audit 2024-04-02T04:39:19.626921+0000 mon.a (mon.0) 442 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed"}]': finished 2024-04-02T04:39:20.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:19 smithi098 bash[23790]: cluster 2024-04-02T04:39:19.629499+0000 mon.a (mon.0) 443 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-04-02T04:39:20.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:19 smithi098 bash[23790]: audit 2024-04-02T04:39:19.629670+0000 mon.a (mon.0) 444 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:21.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:20 smithi069 bash[21528]: cluster 2024-04-02T04:39:19.105850+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:21.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:20 smithi069 bash[21528]: audit 2024-04-02T04:39:20.374806+0000 mon.c (mon.2) 18 : audit 0 from='client.? 172.21.15.69:0/4281750142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:20 smithi069 bash[28904]: cluster 2024-04-02T04:39:19.105850+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:20 smithi069 bash[28904]: audit 2024-04-02T04:39:20.374806+0000 mon.c (mon.2) 18 : audit 0 from='client.? 172.21.15.69:0/4281750142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:20 smithi098 bash[23790]: cluster 2024-04-02T04:39:19.105850+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:20 smithi098 bash[23790]: audit 2024-04-02T04:39:20.374806+0000 mon.c (mon.2) 18 : audit 0 from='client.? 172.21.15.69:0/4281750142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:21.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:21.248+0000 7f8787ac9700 10 client.14298.objecter tick 2024-04-02T04:39:23.015 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:22 smithi069 bash[21528]: cluster 2024-04-02T04:39:21.106342+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:23.016 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:22 smithi069 bash[28904]: cluster 2024-04-02T04:39:21.106342+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:22 smithi098 bash[23790]: cluster 2024-04-02T04:39:21.106342+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:25.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:24 smithi069 bash[21528]: cluster 2024-04-02T04:39:23.106917+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:25.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:24 smithi069 bash[28904]: cluster 2024-04-02T04:39:23.106917+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:24 smithi098 bash[23790]: cluster 2024-04-02T04:39:23.106917+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:26.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:26.252+0000 7f8787ac9700 10 client.14298.objecter tick 2024-04-02T04:39:27.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:26 smithi069 bash[21528]: cluster 2024-04-02T04:39:25.107608+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:27.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:26 smithi069 bash[28904]: cluster 2024-04-02T04:39:25.107608+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:27.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:26 smithi098 bash[23790]: cluster 2024-04-02T04:39:25.107608+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:29.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:28 smithi069 bash[21528]: cluster 2024-04-02T04:39:27.108232+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:29.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:28 smithi069 bash[28904]: cluster 2024-04-02T04:39:27.108232+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:29.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:28 smithi098 bash[23790]: cluster 2024-04-02T04:39:27.108232+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:31.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:30 smithi069 bash[21528]: cluster 2024-04-02T04:39:29.108722+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:31.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:30 smithi069 bash[28904]: cluster 2024-04-02T04:39:29.108722+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:31.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:30 smithi098 bash[23790]: cluster 2024-04-02T04:39:29.108722+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:31.256+0000 7f8787ac9700 10 client.14298.objecter tick 2024-04-02T04:39:33.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[21528]: cluster 2024-04-02T04:39:31.109221+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:33.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[21528]: audit 2024-04-02T04:39:31.886956+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:39:33.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[21528]: audit 2024-04-02T04:39:31.888516+0000 mon.a (mon.0) 446 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:33.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[21528]: cephadm 2024-04-02T04:39:31.889858+0000 mgr.y (mgr.14152) 187 : cephadm 1 Deploying daemon osd.3 on smithi069 2024-04-02T04:39:33.041 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[28904]: cluster 2024-04-02T04:39:31.109221+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:33.041 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[28904]: audit 2024-04-02T04:39:31.886956+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:39:33.041 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[28904]: audit 2024-04-02T04:39:31.888516+0000 mon.a (mon.0) 446 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:33.041 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:32 smithi069 bash[28904]: cephadm 2024-04-02T04:39:31.889858+0000 mgr.y (mgr.14152) 187 : cephadm 1 Deploying daemon osd.3 on smithi069 2024-04-02T04:39:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:32 smithi098 bash[23790]: cluster 2024-04-02T04:39:31.109221+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:32 smithi098 bash[23790]: audit 2024-04-02T04:39:31.886956+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:39:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:32 smithi098 bash[23790]: audit 2024-04-02T04:39:31.888516+0000 mon.a (mon.0) 446 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:32 smithi098 bash[23790]: cephadm 2024-04-02T04:39:31.889858+0000 mgr.y (mgr.14152) 187 : cephadm 1 Deploying daemon osd.3 on smithi069 2024-04-02T04:39:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:34 smithi098 bash[23790]: cluster 2024-04-02T04:39:33.109696+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:34 smithi069 bash[21528]: cluster 2024-04-02T04:39:33.109696+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:34 smithi069 bash[28904]: cluster 2024-04-02T04:39:33.109696+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:35.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:35 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:35.890 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:39:35 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:35.890 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:39:35 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:35.890 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:39:35 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:35.891 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:35 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:35.891 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:39:35 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:36.260+0000 7f8787ac9700 10 client.14298.objecter tick 2024-04-02T04:39:36.282 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:39:36 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:36.282 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:39:36 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:36.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:36 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:36.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:36 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:36.283 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:39:36 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:36.283 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:39:36 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:39:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:36 smithi098 bash[23790]: cluster 2024-04-02T04:39:35.110322+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:36 smithi098 bash[23790]: audit 2024-04-02T04:39:36.156252+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:36 smithi098 bash[23790]: audit 2024-04-02T04:39:36.158796+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:36 smithi098 bash[23790]: audit 2024-04-02T04:39:36.164568+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[21528]: cluster 2024-04-02T04:39:35.110322+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[21528]: audit 2024-04-02T04:39:36.156252+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[21528]: audit 2024-04-02T04:39:36.158796+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[21528]: audit 2024-04-02T04:39:36.164568+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[28904]: cluster 2024-04-02T04:39:35.110322+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[28904]: audit 2024-04-02T04:39:36.156252+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[28904]: audit 2024-04-02T04:39:36.158796+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:36 smithi069 bash[28904]: audit 2024-04-02T04:39:36.164568+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:39.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:38 smithi098 bash[23790]: cluster 2024-04-02T04:39:37.110856+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:39.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:38 smithi098 bash[23790]: cluster 2024-04-02T04:39:37.486433+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:38 smithi098 bash[23790]: cluster 2024-04-02T04:39:37.486504+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:38 smithi098 bash[23790]: cluster 2024-04-02T04:39:37.806879+0000 mon.a (mon.0) 450 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-04-02T04:39:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:38 smithi098 bash[23790]: audit 2024-04-02T04:39:37.807160+0000 mon.a (mon.0) 451 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[21528]: cluster 2024-04-02T04:39:37.110856+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[21528]: cluster 2024-04-02T04:39:37.486433+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[21528]: cluster 2024-04-02T04:39:37.486504+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[21528]: cluster 2024-04-02T04:39:37.806879+0000 mon.a (mon.0) 450 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-04-02T04:39:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[21528]: audit 2024-04-02T04:39:37.807160+0000 mon.a (mon.0) 451 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:39.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[28904]: cluster 2024-04-02T04:39:37.110856+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:39.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[28904]: cluster 2024-04-02T04:39:37.486433+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:39.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[28904]: cluster 2024-04-02T04:39:37.486504+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:39.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[28904]: cluster 2024-04-02T04:39:37.806879+0000 mon.a (mon.0) 450 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-04-02T04:39:39.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:38 smithi069 bash[28904]: audit 2024-04-02T04:39:37.807160+0000 mon.a (mon.0) 451 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:40 smithi098 bash[23790]: cluster 2024-04-02T04:39:39.111495+0000 mgr.y (mgr.14152) 191 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:41.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:41.263+0000 7f8787ac9700 10 client.14298.objecter tick 2024-04-02T04:39:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:40 smithi069 bash[21528]: cluster 2024-04-02T04:39:39.111495+0000 mgr.y (mgr.14152) 191 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:41.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:40 smithi069 bash[28904]: cluster 2024-04-02T04:39:39.111495+0000 mgr.y (mgr.14152) 191 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:42 smithi098 bash[23790]: cluster 2024-04-02T04:39:41.112209+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:42 smithi098 bash[23790]: audit 2024-04-02T04:39:42.496489+0000 mon.a (mon.0) 452 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:39:43.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:42 smithi069 bash[21528]: cluster 2024-04-02T04:39:41.112209+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:43.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:42 smithi069 bash[21528]: audit 2024-04-02T04:39:42.496489+0000 mon.a (mon.0) 452 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:39:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:42 smithi069 bash[28904]: cluster 2024-04-02T04:39:41.112209+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:42 smithi069 bash[28904]: audit 2024-04-02T04:39:42.496489+0000 mon.a (mon.0) 452 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:39:43.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:42.809115+0000 mon.a (mon.0) 453 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:39:43.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: cluster 2024-04-02T04:39:42.812481+0000 mon.a (mon.0) 454 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-04-02T04:39:43.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:42.813095+0000 mon.a (mon.0) 455 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:39:43.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:42.813520+0000 mon.a (mon.0) 456 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:43.198853+0000 mon.a (mon.0) 457 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:43.205765+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:43.206720+0000 mon.a (mon.0) 459 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:43.207505+0000 mon.a (mon.0) 460 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:43.214167+0000 mon.a (mon.0) 461 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[21528]: audit 2024-04-02T04:39:43.811364+0000 mon.a (mon.0) 462 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:42.809115+0000 mon.a (mon.0) 453 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: cluster 2024-04-02T04:39:42.812481+0000 mon.a (mon.0) 454 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:42.813095+0000 mon.a (mon.0) 455 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:42.813520+0000 mon.a (mon.0) 456 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:43.198853+0000 mon.a (mon.0) 457 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:43.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:43.205765+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:43.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:43.206720+0000 mon.a (mon.0) 459 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:43.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:43.207505+0000 mon.a (mon.0) 460 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:43.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:43.214167+0000 mon.a (mon.0) 461 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:43.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:43 smithi069 bash[28904]: audit 2024-04-02T04:39:43.811364+0000 mon.a (mon.0) 462 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:39:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:42.809115+0000 mon.a (mon.0) 453 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: cluster 2024-04-02T04:39:42.812481+0000 mon.a (mon.0) 454 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:42.813095+0000 mon.a (mon.0) 455 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:42.813520+0000 mon.a (mon.0) 456 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:43.198853+0000 mon.a (mon.0) 457 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:43.205765+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:43.206720+0000 mon.a (mon.0) 459 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:43.207505+0000 mon.a (mon.0) 460 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:43.214167+0000 mon.a (mon.0) 461 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:44.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:43 smithi098 bash[23790]: audit 2024-04-02T04:39:43.811364+0000 mon.a (mon.0) 462 : audit 1 from='osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-02T04:39:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:44 smithi098 bash[23790]: cluster 2024-04-02T04:39:43.112579+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:44 smithi098 bash[23790]: cluster 2024-04-02T04:39:43.816784+0000 mon.a (mon.0) 463 : cluster 0 osdmap e28: 4 total, 3 up, 4 in 2024-04-02T04:39:45.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:44 smithi098 bash[23790]: audit 2024-04-02T04:39:43.817823+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:44 smithi098 bash[23790]: audit 2024-04-02T04:39:43.826749+0000 mon.a (mon.0) 465 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:44 smithi098 bash[23790]: audit 2024-04-02T04:39:44.825072+0000 mon.a (mon.0) 466 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[28904]: cluster 2024-04-02T04:39:43.112579+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:45.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[28904]: cluster 2024-04-02T04:39:43.816784+0000 mon.a (mon.0) 463 : cluster 0 osdmap e28: 4 total, 3 up, 4 in 2024-04-02T04:39:45.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[28904]: audit 2024-04-02T04:39:43.817823+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[28904]: audit 2024-04-02T04:39:43.826749+0000 mon.a (mon.0) 465 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[28904]: audit 2024-04-02T04:39:44.825072+0000 mon.a (mon.0) 466 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[21528]: cluster 2024-04-02T04:39:43.112579+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[21528]: cluster 2024-04-02T04:39:43.816784+0000 mon.a (mon.0) 463 : cluster 0 osdmap e28: 4 total, 3 up, 4 in 2024-04-02T04:39:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[21528]: audit 2024-04-02T04:39:43.817823+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[21528]: audit 2024-04-02T04:39:43.826749+0000 mon.a (mon.0) 465 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:44 smithi069 bash[21528]: audit 2024-04-02T04:39:44.825072+0000 mon.a (mon.0) 466 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:46.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:45 smithi098 bash[23790]: cluster 2024-04-02T04:39:43.523355+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:46.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:45 smithi098 bash[23790]: cluster 2024-04-02T04:39:43.523495+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:46.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:45 smithi098 bash[23790]: cluster 2024-04-02T04:39:44.853079+0000 mon.a (mon.0) 467 : cluster 1 osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] boot 2024-04-02T04:39:46.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:45 smithi098 bash[23790]: cluster 2024-04-02T04:39:44.853178+0000 mon.a (mon.0) 468 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-04-02T04:39:46.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:45 smithi098 bash[23790]: audit 2024-04-02T04:39:44.854074+0000 mon.a (mon.0) 469 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:46.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.231+0000 7f87777fe700 1 -- 172.21.15.69:0/2196319187 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f878002df10 con 0x7f876c06f4f0 2024-04-02T04:39:46.239 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 3 on host 'smithi069' 2024-04-02T04:39:46.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 10 librados: watch_flush enter 2024-04-02T04:39:46.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 10 librados: watch_flush exit 2024-04-02T04:39:46.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 20 client.14298.objecter shutdown clearing up homeless session... 2024-04-02T04:39:46.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 10 client.14298.objecter successfully canceled tick 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f876c06f4f0 msgr2=0x7f876c0719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f876c06f4f0 0x7f876c0719b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f877000b2e0 tx=0x7f877001a040 comp rx=0 tx=0).stop 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 msgr2=0x7f8780100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780100d10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f877c00eb30 tx=0x7f877c00ee40 comp rx=0 tx=0).stop 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 shutdown_connections 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780107fc0 0x7f8780101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f876c06f4f0 0x7f876c0719b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8780102d60 0x7f87801026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:46.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 --2- 172.21.15.69:0/2196319187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8780105690 0x7f8780100d10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:39:46.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 >> 172.21.15.69:0/2196319187 conn(0x7f87800fc770 msgr2=0x7f87801039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:39:46.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 shutdown_connections 2024-04-02T04:39:46.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 -- 172.21.15.69:0/2196319187 wait complete. 2024-04-02T04:39:46.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:39:46.235+0000 7f87882ca700 1 librados: shutdown 2024-04-02T04:39:46.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[21528]: cluster 2024-04-02T04:39:43.523355+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:46.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[21528]: cluster 2024-04-02T04:39:43.523495+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:46.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[21528]: cluster 2024-04-02T04:39:44.853079+0000 mon.a (mon.0) 467 : cluster 1 osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] boot 2024-04-02T04:39:46.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[21528]: cluster 2024-04-02T04:39:44.853178+0000 mon.a (mon.0) 468 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-04-02T04:39:46.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[21528]: audit 2024-04-02T04:39:44.854074+0000 mon.a (mon.0) 469 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:46.259 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[28904]: cluster 2024-04-02T04:39:43.523355+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:39:46.259 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[28904]: cluster 2024-04-02T04:39:43.523495+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:39:46.259 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[28904]: cluster 2024-04-02T04:39:44.853079+0000 mon.a (mon.0) 467 : cluster 1 osd.3 [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] boot 2024-04-02T04:39:46.259 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[28904]: cluster 2024-04-02T04:39:44.853178+0000 mon.a (mon.0) 468 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-04-02T04:39:46.259 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:45 smithi069 bash[28904]: audit 2024-04-02T04:39:44.854074+0000 mon.a (mon.0) 469 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:46.965 DEBUG:teuthology.orchestra.run.smithi069:osd.3> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.3.service 2024-04-02T04:39:46.968 INFO:tasks.cephadm:Deploying osd.4 on smithi098 with /dev/vg_nvme/lv_4... 2024-04-02T04:39:46.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T04:39:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:46 smithi098 bash[23790]: cluster 2024-04-02T04:39:45.112872+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:46 smithi098 bash[23790]: cluster 2024-04-02T04:39:45.863106+0000 mon.a (mon.0) 470 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-04-02T04:39:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:46 smithi098 bash[23790]: audit 2024-04-02T04:39:46.189370+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:46 smithi098 bash[23790]: audit 2024-04-02T04:39:46.194887+0000 mon.a (mon.0) 472 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:47.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:46 smithi098 bash[23790]: audit 2024-04-02T04:39:46.204473+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[28904]: cluster 2024-04-02T04:39:45.112872+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[28904]: cluster 2024-04-02T04:39:45.863106+0000 mon.a (mon.0) 470 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-04-02T04:39:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[28904]: audit 2024-04-02T04:39:46.189370+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[28904]: audit 2024-04-02T04:39:46.194887+0000 mon.a (mon.0) 472 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[28904]: audit 2024-04-02T04:39:46.204473+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[21528]: cluster 2024-04-02T04:39:45.112872+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[21528]: cluster 2024-04-02T04:39:45.863106+0000 mon.a (mon.0) 470 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-04-02T04:39:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[21528]: audit 2024-04-02T04:39:46.189370+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[21528]: audit 2024-04-02T04:39:46.194887+0000 mon.a (mon.0) 472 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:47.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:46 smithi069 bash[21528]: audit 2024-04-02T04:39:46.204473+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:48 smithi098 bash[23790]: cluster 2024-04-02T04:39:47.113344+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:49.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:48 smithi069 bash[28904]: cluster 2024-04-02T04:39:47.113344+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:49.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:48 smithi069 bash[21528]: cluster 2024-04-02T04:39:47.113344+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:51.222 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:50 smithi069 bash[21528]: cluster 2024-04-02T04:39:49.113947+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:50 smithi069 bash[28904]: cluster 2024-04-02T04:39:49.113947+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:51.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:50 smithi098 bash[23790]: cluster 2024-04-02T04:39:49.113947+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:51.677 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:39:53.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:52 smithi069 bash[21528]: cluster 2024-04-02T04:39:51.114526+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:53.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:52 smithi069 bash[28904]: cluster 2024-04-02T04:39:51.114526+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:52 smithi098 bash[23790]: cluster 2024-04-02T04:39:51.114526+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T04:39:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T04:39:54.708 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-02T04:39:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-02T04:39:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138435 s, 757 MB/s 2024-04-02T04:39:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-02T04:39:55.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:54 smithi069 bash[21528]: cluster 2024-04-02T04:39:53.115055+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:55.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:54 smithi069 bash[28904]: cluster 2024-04-02T04:39:53.115055+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:54 smithi098 bash[23790]: cluster 2024-04-02T04:39:53.115055+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:55.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-04-02T04:39:56.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: cluster 2024-04-02T04:39:55.115611+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:56.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: cephadm 2024-04-02T04:39:55.478545+0000 mgr.y (mgr.14152) 200 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:39:56.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.487666+0000 mon.a (mon.0) 474 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.498487+0000 mon.a (mon.0) 475 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.501258+0000 mon.a (mon.0) 476 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.503123+0000 mon.a (mon.0) 477 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.504841+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.506559+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: cephadm 2024-04-02T04:39:55.507958+0000 mgr.y (mgr.14152) 201 : cephadm 1 Adjusting osd_memory_target on smithi069 to 4060M 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.514770+0000 mon.a (mon.0) 480 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.519914+0000 mon.a (mon.0) 481 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.521717+0000 mon.a (mon.0) 482 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[21528]: audit 2024-04-02T04:39:55.532515+0000 mon.a (mon.0) 483 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: cluster 2024-04-02T04:39:55.115611+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: cephadm 2024-04-02T04:39:55.478545+0000 mgr.y (mgr.14152) 200 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.487666+0000 mon.a (mon.0) 474 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.498487+0000 mon.a (mon.0) 475 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.501258+0000 mon.a (mon.0) 476 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.503123+0000 mon.a (mon.0) 477 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.504841+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.506559+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: cephadm 2024-04-02T04:39:55.507958+0000 mgr.y (mgr.14152) 201 : cephadm 1 Adjusting osd_memory_target on smithi069 to 4060M 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.514770+0000 mon.a (mon.0) 480 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.519914+0000 mon.a (mon.0) 481 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.521717+0000 mon.a (mon.0) 482 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:56 smithi069 bash[28904]: audit 2024-04-02T04:39:55.532515+0000 mon.a (mon.0) 483 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: cluster 2024-04-02T04:39:55.115611+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: cephadm 2024-04-02T04:39:55.478545+0000 mgr.y (mgr.14152) 200 : cephadm 1 Detected new or changed devices on smithi069 2024-04-02T04:39:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.487666+0000 mon.a (mon.0) 474 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.498487+0000 mon.a (mon.0) 475 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.501258+0000 mon.a (mon.0) 476 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.503123+0000 mon.a (mon.0) 477 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.504841+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.506559+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:56.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: cephadm 2024-04-02T04:39:55.507958+0000 mgr.y (mgr.14152) 201 : cephadm 1 Adjusting osd_memory_target on smithi069 to 4060M 2024-04-02T04:39:56.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.514770+0000 mon.a (mon.0) 480 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:56.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.519914+0000 mon.a (mon.0) 481 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:56.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.521717+0000 mon.a (mon.0) 482 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:56.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:56 smithi098 bash[23790]: audit 2024-04-02T04:39:55.532515+0000 mon.a (mon.0) 483 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:39:58.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:39:58 smithi069 bash[21528]: cluster 2024-04-02T04:39:57.116198+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:58.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:39:58 smithi069 bash[28904]: cluster 2024-04-02T04:39:57.116198+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:39:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:39:58 smithi098 bash[23790]: cluster 2024-04-02T04:39:57.116198+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[21528]: cluster 2024-04-02T04:39:59.116695+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[21528]: cluster 2024-04-02T04:39:59.334745+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[21528]: cluster 2024-04-02T04:39:59.334887+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[21528]: cluster 2024-04-02T04:39:59.503408+0000 mon.a (mon.0) 484 : cluster 0 osdmap e31: 4 total, 4 up, 4 in 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[21528]: cluster 2024-04-02T04:40:00.000141+0000 mon.a (mon.0) 485 : cluster 1 overall HEALTH_OK 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[28904]: cluster 2024-04-02T04:39:59.116695+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[28904]: cluster 2024-04-02T04:39:59.334745+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[28904]: cluster 2024-04-02T04:39:59.334887+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[28904]: cluster 2024-04-02T04:39:59.503408+0000 mon.a (mon.0) 484 : cluster 0 osdmap e31: 4 total, 4 up, 4 in 2024-04-02T04:40:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:00 smithi069 bash[28904]: cluster 2024-04-02T04:40:00.000141+0000 mon.a (mon.0) 485 : cluster 1 overall HEALTH_OK 2024-04-02T04:40:00.822 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:00 smithi098 bash[23790]: cluster 2024-04-02T04:39:59.116695+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:00.822 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:00 smithi098 bash[23790]: cluster 2024-04-02T04:39:59.334745+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:00.822 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:00 smithi098 bash[23790]: cluster 2024-04-02T04:39:59.334887+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:00.822 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:00 smithi098 bash[23790]: cluster 2024-04-02T04:39:59.503408+0000 mon.a (mon.0) 484 : cluster 0 osdmap e31: 4 total, 4 up, 4 in 2024-04-02T04:40:00.822 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:00 smithi098 bash[23790]: cluster 2024-04-02T04:40:00.000141+0000 mon.a (mon.0) 485 : cluster 1 overall HEALTH_OK 2024-04-02T04:40:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/3592335596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81022d0 msgr2=0x7fadf81026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- 172.21.15.98:0/3592335596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81022d0 0x7fadf81026f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fadf40099c0 tx=0x7fadf4009cd0 comp rx=0 tx=0).stop 2024-04-02T04:40:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/3592335596 shutdown_connections 2024-04-02T04:40:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- 172.21.15.98:0/3592335596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf8103e70 0x7fadf810a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- 172.21.15.98:0/3592335596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81022d0 0x7fadf81026f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- 172.21.15.98:0/3592335596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadf81034d0 0x7fadf8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/3592335596 >> 172.21.15.98:0/3592335596 conn(0x7fadf80fd870 msgr2=0x7fadf80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/3592335596 shutdown_connections 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/3592335596 wait complete. 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 librados: starting msgr at 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 librados: starting objecter 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 Processor -- start 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- start start 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 librados: setting wanted keys 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 librados: calling monclient init 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf81022d0 0x7fadf8075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 0x7fadf8074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:02.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadf8103e70 0x7fadf8074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fadf810d410 con 0x7fadf81022d0 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadf810d290 con 0x7fadf81034d0 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fadf810d590 con 0x7fadf8103e70 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 0x7fadf8074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 0x7fadf8074630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48476/0 (socket says 172.21.15.98:48476) 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 -- 172.21.15.98:0/2537146175 learned_addr learned my addr 172.21.15.98:0/2537146175 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfe1ec700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadf8103e70 0x7fadf8074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd9eb700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf81022d0 0x7fadf8075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 -- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadf8103e70 msgr2=0x7fadf8074b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadf8103e70 0x7fadf8074b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 -- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf81022d0 msgr2=0x7fadf8075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf81022d0 0x7fadf8075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:02.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 -- 172.21.15.98:0/2537146175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf8075420 con 0x7fadf81034d0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadfd1ea700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 0x7fadf8074630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fade800d980 tx=0x7fade800dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadeeffd700 10 client.?.objecter ms_handle_connect 0x7fadf81034d0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadeeffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8017070 con 0x7fadf81034d0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fade8004830 con 0x7fadf81034d0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf81a1ca0 con 0x7fadf81034d0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 10 client.24181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf81a2160 con 0x7fadf81034d0 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadffc4f700 1 librados: init done 2024-04-02T04:40:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.302+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade80053c0 con 0x7fadf81034d0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf804ff00 con 0x7fadf81034d0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7fade8005560 con 0x7fadf81034d0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fade406f4f0 0x7fade40719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 3449+0+0 (secure 0 0 0) 0x7fade808ce10 con 0x7fadf81034d0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 10 client.24181.objecter ms_dispatch 0x7fadf8001550 osd_map(31..31 src has 1..31) 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 3 client.24181.objecter handle_osd_map got epochs [31,31] > 0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 3 client.24181.objecter handle_osd_map decoding full epoch 31 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 20 client.24181.objecter dump_active .. 0 homeless 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadfd9eb700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fade406f4f0 0x7fade40719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadeeffd700 10 client.24181.objecter ms_handle_connect 0x7fade406f4f0 2024-04-02T04:40:02.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.306+0000 7fadfd9eb700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fade406f4f0 0x7fade40719b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadf4000c00 tx=0x7fadf4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:40:02.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.310+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fade8020020 con 0x7fadf81034d0 2024-04-02T04:40:02.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:02.522+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fadf802dea0 con 0x7fade406f4f0 2024-04-02T04:40:02.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:02 smithi069 bash[21528]: cluster 2024-04-02T04:40:01.117237+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:02.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:02 smithi069 bash[28904]: cluster 2024-04-02T04:40:01.117237+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:02 smithi098 bash[23790]: cluster 2024-04-02T04:40:01.117237+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:03.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[21528]: audit 2024-04-02T04:40:02.529481+0000 mgr.y (mgr.14152) 205 : audit 0 from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:03.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[21528]: audit 2024-04-02T04:40:02.536969+0000 mon.a (mon.0) 486 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:03.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[21528]: audit 2024-04-02T04:40:02.543360+0000 mon.a (mon.0) 487 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:03.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[21528]: audit 2024-04-02T04:40:02.545029+0000 mon.a (mon.0) 488 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:03.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[28904]: audit 2024-04-02T04:40:02.529481+0000 mgr.y (mgr.14152) 205 : audit 0 from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:03.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[28904]: audit 2024-04-02T04:40:02.536969+0000 mon.a (mon.0) 486 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:03.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[28904]: audit 2024-04-02T04:40:02.543360+0000 mon.a (mon.0) 487 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:03.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:03 smithi069 bash[28904]: audit 2024-04-02T04:40:02.545029+0000 mon.a (mon.0) 488 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:03 smithi098 bash[23790]: audit 2024-04-02T04:40:02.529481+0000 mgr.y (mgr.14152) 205 : audit 0 from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:03 smithi098 bash[23790]: audit 2024-04-02T04:40:02.536969+0000 mon.a (mon.0) 486 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:03 smithi098 bash[23790]: audit 2024-04-02T04:40:02.543360+0000 mon.a (mon.0) 487 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:03 smithi098 bash[23790]: audit 2024-04-02T04:40:02.545029+0000 mon.a (mon.0) 488 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:04.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:04 smithi069 bash[21528]: cluster 2024-04-02T04:40:03.117854+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:04 smithi069 bash[28904]: cluster 2024-04-02T04:40:03.117854+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:04.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:04 smithi098 bash[23790]: cluster 2024-04-02T04:40:03.117854+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:06.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:06 smithi069 bash[21528]: cluster 2024-04-02T04:40:05.118426+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:06.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:06 smithi069 bash[28904]: cluster 2024-04-02T04:40:05.118426+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:06 smithi098 bash[23790]: cluster 2024-04-02T04:40:05.118426+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:07.306+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:08.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:08 smithi098 bash[23790]: cluster 2024-04-02T04:40:07.119077+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:08.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:08 smithi069 bash[21528]: cluster 2024-04-02T04:40:07.119077+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:08.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:08 smithi069 bash[28904]: cluster 2024-04-02T04:40:07.119077+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[21528]: cluster 2024-04-02T04:40:09.119644+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[21528]: audit 2024-04-02T04:40:10.667151+0000 mon.b (mon.1) 10 : audit 1 from='client.? 172.21.15.98:0/4272593950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]: dispatch 2024-04-02T04:40:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[21528]: audit 2024-04-02T04:40:10.670622+0000 mon.a (mon.0) 489 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]: dispatch 2024-04-02T04:40:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[21528]: audit 2024-04-02T04:40:10.673847+0000 mon.a (mon.0) 490 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]': finished 2024-04-02T04:40:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[21528]: cluster 2024-04-02T04:40:10.681472+0000 mon.a (mon.0) 491 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-04-02T04:40:11.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[21528]: audit 2024-04-02T04:40:10.682185+0000 mon.a (mon.0) 492 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:11.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[28904]: cluster 2024-04-02T04:40:09.119644+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:11.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[28904]: audit 2024-04-02T04:40:10.667151+0000 mon.b (mon.1) 10 : audit 1 from='client.? 172.21.15.98:0/4272593950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]: dispatch 2024-04-02T04:40:11.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[28904]: audit 2024-04-02T04:40:10.670622+0000 mon.a (mon.0) 489 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]: dispatch 2024-04-02T04:40:11.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[28904]: audit 2024-04-02T04:40:10.673847+0000 mon.a (mon.0) 490 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]': finished 2024-04-02T04:40:11.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[28904]: cluster 2024-04-02T04:40:10.681472+0000 mon.a (mon.0) 491 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-04-02T04:40:11.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:10 smithi069 bash[28904]: audit 2024-04-02T04:40:10.682185+0000 mon.a (mon.0) 492 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:10 smithi098 bash[23790]: cluster 2024-04-02T04:40:09.119644+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:10 smithi098 bash[23790]: audit 2024-04-02T04:40:10.667151+0000 mon.b (mon.1) 10 : audit 1 from='client.? 172.21.15.98:0/4272593950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]: dispatch 2024-04-02T04:40:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:10 smithi098 bash[23790]: audit 2024-04-02T04:40:10.670622+0000 mon.a (mon.0) 489 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]: dispatch 2024-04-02T04:40:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:10 smithi098 bash[23790]: audit 2024-04-02T04:40:10.673847+0000 mon.a (mon.0) 490 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852"}]': finished 2024-04-02T04:40:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:10 smithi098 bash[23790]: cluster 2024-04-02T04:40:10.681472+0000 mon.a (mon.0) 491 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-04-02T04:40:11.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:10 smithi098 bash[23790]: audit 2024-04-02T04:40:10.682185+0000 mon.a (mon.0) 492 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:12.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:11 smithi069 bash[21528]: audit 2024-04-02T04:40:11.360425+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.98:0/2175246031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:12.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:11 smithi069 bash[28904]: audit 2024-04-02T04:40:11.360425+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.98:0/2175246031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:12.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:11 smithi098 bash[23790]: audit 2024-04-02T04:40:11.360425+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.98:0/2175246031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:12.310+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:13.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:12 smithi069 bash[21528]: cluster 2024-04-02T04:40:11.120198+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:13.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:12 smithi069 bash[28904]: cluster 2024-04-02T04:40:11.120198+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:12 smithi098 bash[23790]: cluster 2024-04-02T04:40:11.120198+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:15.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:14 smithi069 bash[21528]: cluster 2024-04-02T04:40:13.120685+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:15.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:14 smithi069 bash[28904]: cluster 2024-04-02T04:40:13.120685+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:14 smithi098 bash[23790]: cluster 2024-04-02T04:40:13.120685+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:17.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:16 smithi069 bash[21528]: cluster 2024-04-02T04:40:15.121235+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:17.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:16 smithi069 bash[28904]: cluster 2024-04-02T04:40:15.121235+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:16 smithi098 bash[23790]: cluster 2024-04-02T04:40:15.121235+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:17.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:17.314+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:19.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:18 smithi069 bash[21528]: cluster 2024-04-02T04:40:17.121789+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:19.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:18 smithi069 bash[28904]: cluster 2024-04-02T04:40:17.121789+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:19.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:18 smithi098 bash[23790]: cluster 2024-04-02T04:40:17.121789+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:20 smithi098 bash[23790]: cluster 2024-04-02T04:40:19.122377+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:20 smithi069 bash[21528]: cluster 2024-04-02T04:40:19.122377+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:20 smithi069 bash[28904]: cluster 2024-04-02T04:40:19.122377+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:22.318+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:23.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:22 smithi069 bash[21528]: cluster 2024-04-02T04:40:21.122902+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:23.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:22 smithi069 bash[28904]: cluster 2024-04-02T04:40:21.122902+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:22 smithi098 bash[23790]: cluster 2024-04-02T04:40:21.122902+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:24.027 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:23 smithi098 bash[23790]: audit 2024-04-02T04:40:22.857683+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:40:24.028 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:23 smithi098 bash[23790]: audit 2024-04-02T04:40:22.859767+0000 mon.a (mon.0) 494 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:24.028 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:23 smithi098 bash[23790]: cephadm 2024-04-02T04:40:22.861618+0000 mgr.y (mgr.14152) 216 : cephadm 1 Deploying daemon osd.4 on smithi098 2024-04-02T04:40:24.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:23 smithi069 bash[21528]: audit 2024-04-02T04:40:22.857683+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:40:24.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:23 smithi069 bash[21528]: audit 2024-04-02T04:40:22.859767+0000 mon.a (mon.0) 494 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:24.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:23 smithi069 bash[21528]: cephadm 2024-04-02T04:40:22.861618+0000 mgr.y (mgr.14152) 216 : cephadm 1 Deploying daemon osd.4 on smithi098 2024-04-02T04:40:24.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:23 smithi069 bash[28904]: audit 2024-04-02T04:40:22.857683+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:40:24.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:23 smithi069 bash[28904]: audit 2024-04-02T04:40:22.859767+0000 mon.a (mon.0) 494 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:24.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:23 smithi069 bash[28904]: cephadm 2024-04-02T04:40:22.861618+0000 mgr.y (mgr.14152) 216 : cephadm 1 Deploying daemon osd.4 on smithi098 2024-04-02T04:40:25.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:24 smithi069 bash[21528]: cluster 2024-04-02T04:40:23.123506+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:25.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:24 smithi069 bash[28904]: cluster 2024-04-02T04:40:23.123506+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:24 smithi098 bash[23790]: cluster 2024-04-02T04:40:23.123506+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:25.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:25 smithi098 bash[23790]: cluster 2024-04-02T04:40:24.894938+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-04-02T04:40:25.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:25 smithi098 bash[23790]: cluster 2024-04-02T04:40:24.898518+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-04-02T04:40:26.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:25 smithi069 bash[21528]: cluster 2024-04-02T04:40:24.894938+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-04-02T04:40:26.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:25 smithi069 bash[21528]: cluster 2024-04-02T04:40:24.898518+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-04-02T04:40:26.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:25 smithi069 bash[28904]: cluster 2024-04-02T04:40:24.894938+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-04-02T04:40:26.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:25 smithi069 bash[28904]: cluster 2024-04-02T04:40:24.898518+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-04-02T04:40:26.968 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:40:26 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:40:26.968 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:26 smithi098 bash[23790]: cluster 2024-04-02T04:40:25.124218+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:26.968 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:26 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:40:27.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:40:27.225 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:40:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:40:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:26 smithi069 bash[21528]: cluster 2024-04-02T04:40:25.124218+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:27.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:26 smithi069 bash[28904]: cluster 2024-04-02T04:40:25.124218+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:27.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:27.322+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[21528]: cluster 2024-04-02T04:40:27.124749+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[21528]: audit 2024-04-02T04:40:27.255270+0000 mon.a (mon.0) 495 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[21528]: audit 2024-04-02T04:40:27.256109+0000 mon.a (mon.0) 496 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[21528]: audit 2024-04-02T04:40:27.264221+0000 mon.a (mon.0) 497 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[28904]: cluster 2024-04-02T04:40:27.124749+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[28904]: audit 2024-04-02T04:40:27.255270+0000 mon.a (mon.0) 495 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[28904]: audit 2024-04-02T04:40:27.256109+0000 mon.a (mon.0) 496 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:28.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:28 smithi069 bash[28904]: audit 2024-04-02T04:40:27.264221+0000 mon.a (mon.0) 497 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:28.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:28 smithi098 bash[23790]: cluster 2024-04-02T04:40:27.124749+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:28.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:28 smithi098 bash[23790]: audit 2024-04-02T04:40:27.255270+0000 mon.a (mon.0) 495 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:28.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:28 smithi098 bash[23790]: audit 2024-04-02T04:40:27.256109+0000 mon.a (mon.0) 496 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:28.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:28 smithi098 bash[23790]: audit 2024-04-02T04:40:27.264221+0000 mon.a (mon.0) 497 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:30 smithi098 bash[23790]: cluster 2024-04-02T04:40:29.125308+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:31.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:30 smithi069 bash[28904]: cluster 2024-04-02T04:40:29.125308+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:31.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:30 smithi069 bash[21528]: cluster 2024-04-02T04:40:29.125308+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:32.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:32.326+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:33.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:32 smithi069 bash[21528]: cluster 2024-04-02T04:40:31.125791+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:33.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:32 smithi069 bash[21528]: cluster 2024-04-02T04:40:31.793210+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:33.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:32 smithi069 bash[21528]: cluster 2024-04-02T04:40:31.793297+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:33.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:32 smithi069 bash[28904]: cluster 2024-04-02T04:40:31.125791+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:33.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:32 smithi069 bash[28904]: cluster 2024-04-02T04:40:31.793210+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:33.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:32 smithi069 bash[28904]: cluster 2024-04-02T04:40:31.793297+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:32 smithi098 bash[23790]: cluster 2024-04-02T04:40:31.125791+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:32 smithi098 bash[23790]: cluster 2024-04-02T04:40:31.793210+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:33.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:32 smithi098 bash[23790]: cluster 2024-04-02T04:40:31.793297+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:34.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[28904]: cluster 2024-04-02T04:40:32.770248+0000 mon.a (mon.0) 498 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-04-02T04:40:34.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[28904]: audit 2024-04-02T04:40:32.770546+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:34.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[28904]: audit 2024-04-02T04:40:33.450963+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:40:34.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[28904]: audit 2024-04-02T04:40:33.454481+0000 mon.a (mon.0) 500 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:40:34.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[21528]: cluster 2024-04-02T04:40:32.770248+0000 mon.a (mon.0) 498 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-04-02T04:40:34.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[21528]: audit 2024-04-02T04:40:32.770546+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:34.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[21528]: audit 2024-04-02T04:40:33.450963+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:40:34.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:33 smithi069 bash[21528]: audit 2024-04-02T04:40:33.454481+0000 mon.a (mon.0) 500 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:40:34.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:33 smithi098 bash[23790]: cluster 2024-04-02T04:40:32.770248+0000 mon.a (mon.0) 498 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-04-02T04:40:34.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:33 smithi098 bash[23790]: audit 2024-04-02T04:40:32.770546+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:34.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:33 smithi098 bash[23790]: audit 2024-04-02T04:40:33.450963+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:40:34.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:33 smithi098 bash[23790]: audit 2024-04-02T04:40:33.454481+0000 mon.a (mon.0) 500 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:40:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: cluster 2024-04-02T04:40:33.126300+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:33.772428+0000 mon.a (mon.0) 501 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:33.776529+0000 mon.b (mon.1) 13 : audit 1 from='osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: cluster 2024-04-02T04:40:33.783030+0000 mon.a (mon.0) 502 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:33.784011+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:33.784638+0000 mon.a (mon.0) 504 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:34.052637+0000 mon.a (mon.0) 505 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:34.061775+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:34.738932+0000 mon.a (mon.0) 507 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:34.741012+0000 mon.a (mon.0) 508 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:34 smithi098 bash[23790]: audit 2024-04-02T04:40:34.751582+0000 mon.a (mon.0) 509 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: cluster 2024-04-02T04:40:33.126300+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:33.772428+0000 mon.a (mon.0) 501 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:33.776529+0000 mon.b (mon.1) 13 : audit 1 from='osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: cluster 2024-04-02T04:40:33.783030+0000 mon.a (mon.0) 502 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:33.784011+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:33.784638+0000 mon.a (mon.0) 504 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:34.052637+0000 mon.a (mon.0) 505 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:34.061775+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:34.738932+0000 mon.a (mon.0) 507 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:34.741012+0000 mon.a (mon.0) 508 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:35.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[21528]: audit 2024-04-02T04:40:34.751582+0000 mon.a (mon.0) 509 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: cluster 2024-04-02T04:40:33.126300+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:33.772428+0000 mon.a (mon.0) 501 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:33.776529+0000 mon.b (mon.1) 13 : audit 1 from='osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: cluster 2024-04-02T04:40:33.783030+0000 mon.a (mon.0) 502 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:33.784011+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:33.784638+0000 mon.a (mon.0) 504 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:34.052637+0000 mon.a (mon.0) 505 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:34.061775+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:34.738932+0000 mon.a (mon.0) 507 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:34.741012+0000 mon.a (mon.0) 508 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:35.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:34 smithi069 bash[28904]: audit 2024-04-02T04:40:34.751582+0000 mon.a (mon.0) 509 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:35 smithi098 bash[23790]: audit 2024-04-02T04:40:34.776372+0000 mon.a (mon.0) 510 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:40:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:35 smithi098 bash[23790]: cluster 2024-04-02T04:40:34.779864+0000 mon.a (mon.0) 511 : cluster 0 osdmap e35: 5 total, 4 up, 5 in 2024-04-02T04:40:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:35 smithi098 bash[23790]: audit 2024-04-02T04:40:34.780447+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:35 smithi098 bash[23790]: audit 2024-04-02T04:40:34.792801+0000 mon.a (mon.0) 513 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[21528]: audit 2024-04-02T04:40:34.776372+0000 mon.a (mon.0) 510 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[21528]: cluster 2024-04-02T04:40:34.779864+0000 mon.a (mon.0) 511 : cluster 0 osdmap e35: 5 total, 4 up, 5 in 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[21528]: audit 2024-04-02T04:40:34.780447+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[21528]: audit 2024-04-02T04:40:34.792801+0000 mon.a (mon.0) 513 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[28904]: audit 2024-04-02T04:40:34.776372+0000 mon.a (mon.0) 510 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[28904]: cluster 2024-04-02T04:40:34.779864+0000 mon.a (mon.0) 511 : cluster 0 osdmap e35: 5 total, 4 up, 5 in 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[28904]: audit 2024-04-02T04:40:34.780447+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:35 smithi069 bash[28904]: audit 2024-04-02T04:40:34.792801+0000 mon.a (mon.0) 513 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:36 smithi098 bash[23790]: cluster 2024-04-02T04:40:34.399502+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:36 smithi098 bash[23790]: cluster 2024-04-02T04:40:34.399628+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:36 smithi098 bash[23790]: cluster 2024-04-02T04:40:35.126820+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:36 smithi098 bash[23790]: cluster 2024-04-02T04:40:35.797814+0000 mon.a (mon.0) 514 : cluster 1 osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] boot 2024-04-02T04:40:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:36 smithi098 bash[23790]: cluster 2024-04-02T04:40:35.797921+0000 mon.a (mon.0) 515 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-04-02T04:40:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:36 smithi098 bash[23790]: audit 2024-04-02T04:40:35.798854+0000 mon.a (mon.0) 516 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[21528]: cluster 2024-04-02T04:40:34.399502+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[21528]: cluster 2024-04-02T04:40:34.399628+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[21528]: cluster 2024-04-02T04:40:35.126820+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[21528]: cluster 2024-04-02T04:40:35.797814+0000 mon.a (mon.0) 514 : cluster 1 osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] boot 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[21528]: cluster 2024-04-02T04:40:35.797921+0000 mon.a (mon.0) 515 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[21528]: audit 2024-04-02T04:40:35.798854+0000 mon.a (mon.0) 516 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[28904]: cluster 2024-04-02T04:40:34.399502+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[28904]: cluster 2024-04-02T04:40:34.399628+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[28904]: cluster 2024-04-02T04:40:35.126820+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-02T04:40:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[28904]: cluster 2024-04-02T04:40:35.797814+0000 mon.a (mon.0) 514 : cluster 1 osd.4 [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] boot 2024-04-02T04:40:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[28904]: cluster 2024-04-02T04:40:35.797921+0000 mon.a (mon.0) 515 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-04-02T04:40:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:36 smithi069 bash[28904]: audit 2024-04-02T04:40:35.798854+0000 mon.a (mon.0) 516 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:40:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.330+0000 7fadff44e700 10 client.24181.objecter tick 2024-04-02T04:40:37.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.402+0000 7fadeeffd700 1 -- 172.21.15.98:0/2537146175 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fadf802dea0 con 0x7fade406f4f0 2024-04-02T04:40:37.410 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 4 on host 'smithi098' 2024-04-02T04:40:37.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 10 librados: watch_flush enter 2024-04-02T04:40:37.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 10 librados: watch_flush exit 2024-04-02T04:40:37.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 20 client.24181.objecter shutdown clearing up homeless session... 2024-04-02T04:40:37.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 10 client.24181.objecter successfully canceled tick 2024-04-02T04:40:37.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fade406f4f0 msgr2=0x7fade40719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:37.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fade406f4f0 0x7fade40719b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadf4000c00 tx=0x7fadf4005fb0 comp rx=0 tx=0).stop 2024-04-02T04:40:37.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 msgr2=0x7fadf8074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:37.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 0x7fadf8074630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fade800d980 tx=0x7fade800dd40 comp rx=0 tx=0).stop 2024-04-02T04:40:37.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 shutdown_connections 2024-04-02T04:40:37.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7fade406f4f0 0x7fade40719b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadf8103e70 0x7fadf8074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf81022d0 0x7fadf8075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 --2- 172.21.15.98:0/2537146175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadf81034d0 0x7fadf8074630 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 >> 172.21.15.98:0/2537146175 conn(0x7fadf80fd870 msgr2=0x7fadf81068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 shutdown_connections 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 -- 172.21.15.98:0/2537146175 wait complete. 2024-04-02T04:40:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:37.406+0000 7fadffc4f700 1 librados: shutdown 2024-04-02T04:40:38.057 DEBUG:teuthology.orchestra.run.smithi098:osd.4> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.4.service 2024-04-02T04:40:38.060 INFO:tasks.cephadm:Deploying osd.5 on smithi098 with /dev/vg_nvme/lv_3... 2024-04-02T04:40:38.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T04:40:38.087 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:37 smithi098 bash[23790]: cluster 2024-04-02T04:40:36.816829+0000 mon.a (mon.0) 517 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-04-02T04:40:38.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:37 smithi098 bash[23790]: audit 2024-04-02T04:40:37.343484+0000 mon.a (mon.0) 518 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:38.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:37 smithi098 bash[23790]: audit 2024-04-02T04:40:37.349460+0000 mon.a (mon.0) 519 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:38.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:37 smithi098 bash[23790]: audit 2024-04-02T04:40:37.360300+0000 mon.a (mon.0) 520 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:38.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[21528]: cluster 2024-04-02T04:40:36.816829+0000 mon.a (mon.0) 517 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[21528]: audit 2024-04-02T04:40:37.343484+0000 mon.a (mon.0) 518 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[21528]: audit 2024-04-02T04:40:37.349460+0000 mon.a (mon.0) 519 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[21528]: audit 2024-04-02T04:40:37.360300+0000 mon.a (mon.0) 520 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[28904]: cluster 2024-04-02T04:40:36.816829+0000 mon.a (mon.0) 517 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[28904]: audit 2024-04-02T04:40:37.343484+0000 mon.a (mon.0) 518 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[28904]: audit 2024-04-02T04:40:37.349460+0000 mon.a (mon.0) 519 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:38.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:37 smithi069 bash[28904]: audit 2024-04-02T04:40:37.360300+0000 mon.a (mon.0) 520 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:38 smithi098 bash[23790]: cluster 2024-04-02T04:40:37.127345+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:38 smithi098 bash[23790]: cluster 2024-04-02T04:40:37.838365+0000 mon.a (mon.0) 521 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-04-02T04:40:39.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:38 smithi098 bash[23790]: cluster 2024-04-02T04:40:38.744104+0000 mon.a (mon.0) 522 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-04-02T04:40:39.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:38 smithi069 bash[21528]: cluster 2024-04-02T04:40:37.127345+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:38 smithi069 bash[21528]: cluster 2024-04-02T04:40:37.838365+0000 mon.a (mon.0) 521 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-04-02T04:40:39.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:38 smithi069 bash[21528]: cluster 2024-04-02T04:40:38.744104+0000 mon.a (mon.0) 522 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-04-02T04:40:39.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:38 smithi069 bash[28904]: cluster 2024-04-02T04:40:37.127345+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:39.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:38 smithi069 bash[28904]: cluster 2024-04-02T04:40:37.838365+0000 mon.a (mon.0) 521 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-04-02T04:40:39.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:38 smithi069 bash[28904]: cluster 2024-04-02T04:40:38.744104+0000 mon.a (mon.0) 522 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-04-02T04:40:41.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:40 smithi069 bash[21528]: cluster 2024-04-02T04:40:39.127968+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v205: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T04:40:41.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:40 smithi069 bash[21528]: cluster 2024-04-02T04:40:39.746417+0000 mon.a (mon.0) 523 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-04-02T04:40:41.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:40 smithi069 bash[28904]: cluster 2024-04-02T04:40:39.127968+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v205: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T04:40:41.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:40 smithi069 bash[28904]: cluster 2024-04-02T04:40:39.746417+0000 mon.a (mon.0) 523 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-04-02T04:40:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:40 smithi098 bash[23790]: cluster 2024-04-02T04:40:39.127968+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v205: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T04:40:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:40 smithi098 bash[23790]: cluster 2024-04-02T04:40:39.746417+0000 mon.a (mon.0) 523 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-04-02T04:40:42.330 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:40:43.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:42 smithi069 bash[21528]: cluster 2024-04-02T04:40:41.128487+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v207: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-02T04:40:43.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:42 smithi069 bash[28904]: cluster 2024-04-02T04:40:41.128487+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v207: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-02T04:40:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:42 smithi098 bash[23790]: cluster 2024-04-02T04:40:41.128487+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v207: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-02T04:40:45.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:44 smithi069 bash[21528]: cluster 2024-04-02T04:40:43.129081+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T04:40:45.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:44 smithi069 bash[28904]: cluster 2024-04-02T04:40:43.129081+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T04:40:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:44 smithi098 bash[23790]: cluster 2024-04-02T04:40:43.129081+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T04:40:46.292 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T04:40:46.292 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T04:40:46.292 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-02T04:40:46.292 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-02T04:40:46.292 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130163 s, 806 MB/s 2024-04-02T04:40:46.292 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-02T04:40:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:46 smithi098 bash[23790]: cluster 2024-04-02T04:40:45.129679+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:40:47.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-04-02T04:40:47.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:46 smithi069 bash[21528]: cluster 2024-04-02T04:40:45.129679+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:40:47.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:46 smithi069 bash[28904]: cluster 2024-04-02T04:40:45.129679+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:40:48.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: cluster 2024-04-02T04:40:47.130267+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: cephadm 2024-04-02T04:40:47.547734+0000 mgr.y (mgr.14152) 230 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.556657+0000 mon.a (mon.0) 524 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.566246+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.568401+0000 mon.a (mon.0) 526 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: cephadm 2024-04-02T04:40:47.569716+0000 mgr.y (mgr.14152) 231 : cephadm 1 Adjusting osd_memory_target on smithi098 to 17265M 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.576588+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.581307+0000 mon.a (mon.0) 528 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.583267+0000 mon.a (mon.0) 529 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:48.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:48 smithi098 bash[23790]: audit 2024-04-02T04:40:47.593813+0000 mon.a (mon.0) 530 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: cluster 2024-04-02T04:40:47.130267+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-02T04:40:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: cephadm 2024-04-02T04:40:47.547734+0000 mgr.y (mgr.14152) 230 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.556657+0000 mon.a (mon.0) 524 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.566246+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.568401+0000 mon.a (mon.0) 526 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: cephadm 2024-04-02T04:40:47.569716+0000 mgr.y (mgr.14152) 231 : cephadm 1 Adjusting osd_memory_target on smithi098 to 17265M 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.576588+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.581307+0000 mon.a (mon.0) 528 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.583267+0000 mon.a (mon.0) 529 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[28904]: audit 2024-04-02T04:40:47.593813+0000 mon.a (mon.0) 530 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: cluster 2024-04-02T04:40:47.130267+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: cephadm 2024-04-02T04:40:47.547734+0000 mgr.y (mgr.14152) 230 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:40:49.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.556657+0000 mon.a (mon.0) 524 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.566246+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.568401+0000 mon.a (mon.0) 526 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: cephadm 2024-04-02T04:40:47.569716+0000 mgr.y (mgr.14152) 231 : cephadm 1 Adjusting osd_memory_target on smithi098 to 17265M 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.576588+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.581307+0000 mon.a (mon.0) 528 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.583267+0000 mon.a (mon.0) 529 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:48 smithi069 bash[21528]: audit 2024-04-02T04:40:47.593813+0000 mon.a (mon.0) 530 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:40:51.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:50 smithi069 bash[28904]: cluster 2024-04-02T04:40:49.130768+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:51.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:50 smithi069 bash[21528]: cluster 2024-04-02T04:40:49.130768+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:51.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:50 smithi098 bash[23790]: cluster 2024-04-02T04:40:49.130768+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:40:52.836 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:52 smithi098 bash[23790]: cluster 2024-04-02T04:40:51.131347+0000 mgr.y (mgr.14152) 233 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:53.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:52 smithi069 bash[21528]: cluster 2024-04-02T04:40:51.131347+0000 mgr.y (mgr.14152) 233 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:52 smithi069 bash[28904]: cluster 2024-04-02T04:40:51.131347+0000 mgr.y (mgr.14152) 233 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/1175466450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 msgr2=0x7f888810a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- 172.21.15.98:0/1175466450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f888810a7d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8880009990 tx=0x7f8880009ca0 comp rx=0 tx=0).stop 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/1175466450 shutdown_connections 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- 172.21.15.98:0/1175466450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f888810a7d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- 172.21.15.98:0/1175466450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88881005a0 0x7f88881009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- 172.21.15.98:0/1175466450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8888103580 0x7f88881039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/1175466450 >> 172.21.15.98:0/1175466450 conn(0x7f88880fbf00 msgr2=0x7f88880fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/1175466450 shutdown_connections 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/1175466450 wait complete. 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 librados: starting msgr at 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 librados: starting objecter 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 Processor -- start 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- start start 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 librados: setting wanted keys 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 librados: calling monclient init 2024-04-02T04:40:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88881005a0 0x7f8888199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8888103580 0x7f888819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f88881a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f888810d450 con 0x7f8888103580 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f888810d2d0 con 0x7f8888103f20 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f888810d5d0 con 0x7f88881005a0 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f88881a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f88881a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46486/0 (socket says 172.21.15.98:46486) 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 -- 172.21.15.98:0/2369162817 learned_addr learned my addr 172.21.15.98:0/2369162817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f8886ffd700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88881005a0 0x7f8888199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88867fc700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8888103580 0x7f888819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 -- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88881005a0 msgr2=0x7f8888199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88881005a0 0x7f8888199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 -- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8888103580 msgr2=0x7f888819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8888103580 0x7f888819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 -- 172.21.15.98:0/2369162817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88881a1920 con 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f88877fe700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f88881a1250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8880006010 tx=0x7f88800047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f886ffff700 10 client.?.objecter ms_handle_connect 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f886ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8880027070 con 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8880004100 con 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88881a1bb0 con 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8880022d80 con 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 10 client.24208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88881a2000 con 0x7f8888103f20 2024-04-02T04:40:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.165+0000 7f888d501700 1 librados: init done 2024-04-02T04:40:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f888804ff00 con 0x7f8888103f20 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f888002a070 con 0x7f8888103f20 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f887006f4f0 0x7f88700719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 3991+0+0 (secure 0 0 0) 0x7f888008e170 con 0x7f8888103f20 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 10 client.24208.objecter ms_dispatch 0x7f8888001550 osd_map(40..40 src has 1..40) 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 3 client.24208.objecter handle_osd_map got epochs [40,40] > 0 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 3 client.24208.objecter handle_osd_map decoding full epoch 40 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 20 client.24208.objecter dump_active .. 0 homeless 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f8886ffd700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f887006f4f0 0x7f88700719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f886ffff700 10 client.24208.objecter ms_handle_connect 0x7f887006f4f0 2024-04-02T04:40:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.169+0000 7f8886ffd700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f887006f4f0 0x7f88700719b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f888819ab80 tx=0x7f8878009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:40:54.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.173+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8880013070 con 0x7f8888103f20 2024-04-02T04:40:54.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:54.385+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8888102a90 con 0x7f887006f4f0 2024-04-02T04:40:55.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[21528]: cluster 2024-04-02T04:40:53.131938+0000 mgr.y (mgr.14152) 234 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:55.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[21528]: audit 2024-04-02T04:40:54.403791+0000 mon.a (mon.0) 531 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:55.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[21528]: audit 2024-04-02T04:40:54.410089+0000 mon.a (mon.0) 532 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:55.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[21528]: audit 2024-04-02T04:40:54.411864+0000 mon.a (mon.0) 533 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:55.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[28904]: cluster 2024-04-02T04:40:53.131938+0000 mgr.y (mgr.14152) 234 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:55.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[28904]: audit 2024-04-02T04:40:54.403791+0000 mon.a (mon.0) 531 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:55.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[28904]: audit 2024-04-02T04:40:54.410089+0000 mon.a (mon.0) 532 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:55.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:54 smithi069 bash[28904]: audit 2024-04-02T04:40:54.411864+0000 mon.a (mon.0) 533 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:54 smithi098 bash[23790]: cluster 2024-04-02T04:40:53.131938+0000 mgr.y (mgr.14152) 234 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:54 smithi098 bash[23790]: audit 2024-04-02T04:40:54.403791+0000 mon.a (mon.0) 531 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:54 smithi098 bash[23790]: audit 2024-04-02T04:40:54.410089+0000 mon.a (mon.0) 532 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:54 smithi098 bash[23790]: audit 2024-04-02T04:40:54.411864+0000 mon.a (mon.0) 533 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:56.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:55 smithi069 bash[21528]: audit 2024-04-02T04:40:54.396449+0000 mgr.y (mgr.14152) 235 : audit 0 from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:56.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:55 smithi069 bash[21528]: cluster 2024-04-02T04:40:55.228545+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:56.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:55 smithi069 bash[21528]: cluster 2024-04-02T04:40:55.228639+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:56.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:55 smithi069 bash[28904]: audit 2024-04-02T04:40:54.396449+0000 mgr.y (mgr.14152) 235 : audit 0 from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:56.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:55 smithi069 bash[28904]: cluster 2024-04-02T04:40:55.228545+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:56.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:55 smithi069 bash[28904]: cluster 2024-04-02T04:40:55.228639+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:56.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:55 smithi098 bash[23790]: audit 2024-04-02T04:40:54.396449+0000 mgr.y (mgr.14152) 235 : audit 0 from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:56.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:55 smithi098 bash[23790]: cluster 2024-04-02T04:40:55.228545+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:40:56.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:55 smithi098 bash[23790]: cluster 2024-04-02T04:40:55.228639+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:40:57.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:56 smithi069 bash[28904]: cluster 2024-04-02T04:40:55.132442+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:57.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:56 smithi069 bash[28904]: cluster 2024-04-02T04:40:55.773555+0000 mon.a (mon.0) 534 : cluster 0 osdmap e41: 5 total, 5 up, 5 in 2024-04-02T04:40:57.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:56 smithi069 bash[21528]: cluster 2024-04-02T04:40:55.132442+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:57.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:56 smithi069 bash[21528]: cluster 2024-04-02T04:40:55.773555+0000 mon.a (mon.0) 534 : cluster 0 osdmap e41: 5 total, 5 up, 5 in 2024-04-02T04:40:57.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:56 smithi098 bash[23790]: cluster 2024-04-02T04:40:55.132442+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:57.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:56 smithi098 bash[23790]: cluster 2024-04-02T04:40:55.773555+0000 mon.a (mon.0) 534 : cluster 0 osdmap e41: 5 total, 5 up, 5 in 2024-04-02T04:40:59.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:40:58 smithi069 bash[21528]: cluster 2024-04-02T04:40:57.133021+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:59.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:40:58 smithi069 bash[28904]: cluster 2024-04-02T04:40:57.133021+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:59.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:40:59.169+0000 7f888cd00700 10 client.24208.objecter tick 2024-04-02T04:40:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:40:58 smithi098 bash[23790]: cluster 2024-04-02T04:40:57.133021+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:00 smithi098 bash[23790]: cluster 2024-04-02T04:40:59.133663+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:00 smithi069 bash[21528]: cluster 2024-04-02T04:40:59.133663+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:00 smithi069 bash[28904]: cluster 2024-04-02T04:40:59.133663+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:03.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[28904]: cluster 2024-04-02T04:41:01.134187+0000 mgr.y (mgr.14152) 239 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:03.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[28904]: audit 2024-04-02T04:41:02.666293+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.98:0/972061347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]: dispatch 2024-04-02T04:41:03.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[28904]: audit 2024-04-02T04:41:02.670051+0000 mon.a (mon.0) 535 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]: dispatch 2024-04-02T04:41:03.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[28904]: audit 2024-04-02T04:41:02.673621+0000 mon.a (mon.0) 536 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]': finished 2024-04-02T04:41:03.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[28904]: cluster 2024-04-02T04:41:02.681852+0000 mon.a (mon.0) 537 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-04-02T04:41:03.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[28904]: audit 2024-04-02T04:41:02.682448+0000 mon.a (mon.0) 538 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:03.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[21528]: cluster 2024-04-02T04:41:01.134187+0000 mgr.y (mgr.14152) 239 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:03.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[21528]: audit 2024-04-02T04:41:02.666293+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.98:0/972061347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]: dispatch 2024-04-02T04:41:03.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[21528]: audit 2024-04-02T04:41:02.670051+0000 mon.a (mon.0) 535 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]: dispatch 2024-04-02T04:41:03.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[21528]: audit 2024-04-02T04:41:02.673621+0000 mon.a (mon.0) 536 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]': finished 2024-04-02T04:41:03.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[21528]: cluster 2024-04-02T04:41:02.681852+0000 mon.a (mon.0) 537 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-04-02T04:41:03.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:02 smithi069 bash[21528]: audit 2024-04-02T04:41:02.682448+0000 mon.a (mon.0) 538 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:02 smithi098 bash[23790]: cluster 2024-04-02T04:41:01.134187+0000 mgr.y (mgr.14152) 239 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:02 smithi098 bash[23790]: audit 2024-04-02T04:41:02.666293+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.98:0/972061347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]: dispatch 2024-04-02T04:41:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:02 smithi098 bash[23790]: audit 2024-04-02T04:41:02.670051+0000 mon.a (mon.0) 535 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]: dispatch 2024-04-02T04:41:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:02 smithi098 bash[23790]: audit 2024-04-02T04:41:02.673621+0000 mon.a (mon.0) 536 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe"}]': finished 2024-04-02T04:41:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:02 smithi098 bash[23790]: cluster 2024-04-02T04:41:02.681852+0000 mon.a (mon.0) 537 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-04-02T04:41:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:02 smithi098 bash[23790]: audit 2024-04-02T04:41:02.682448+0000 mon.a (mon.0) 538 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:04.173+0000 7f888cd00700 10 client.24208.objecter tick 2024-04-02T04:41:04.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:03 smithi098 bash[23790]: audit 2024-04-02T04:41:03.427251+0000 mon.b (mon.1) 15 : audit 0 from='client.? 172.21.15.98:0/361200370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:41:04.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:03 smithi069 bash[28904]: audit 2024-04-02T04:41:03.427251+0000 mon.b (mon.1) 15 : audit 0 from='client.? 172.21.15.98:0/361200370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:41:04.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:03 smithi069 bash[21528]: audit 2024-04-02T04:41:03.427251+0000 mon.b (mon.1) 15 : audit 0 from='client.? 172.21.15.98:0/361200370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:41:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:04 smithi098 bash[23790]: cluster 2024-04-02T04:41:03.134798+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:04 smithi098 bash[23790]: cluster 2024-04-02T04:41:04.477002+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:04 smithi098 bash[23790]: cluster 2024-04-02T04:41:04.477095+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:05.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:04 smithi069 bash[28904]: cluster 2024-04-02T04:41:03.134798+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:05.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:04 smithi069 bash[28904]: cluster 2024-04-02T04:41:04.477002+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:05.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:04 smithi069 bash[28904]: cluster 2024-04-02T04:41:04.477095+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:05.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:04 smithi069 bash[21528]: cluster 2024-04-02T04:41:03.134798+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:05.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:04 smithi069 bash[21528]: cluster 2024-04-02T04:41:04.477002+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:05.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:04 smithi069 bash[21528]: cluster 2024-04-02T04:41:04.477095+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:06.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:05 smithi098 bash[23790]: cluster 2024-04-02T04:41:04.806837+0000 mon.a (mon.0) 539 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-04-02T04:41:06.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:05 smithi098 bash[23790]: audit 2024-04-02T04:41:04.807158+0000 mon.a (mon.0) 540 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:06.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:05 smithi069 bash[21528]: cluster 2024-04-02T04:41:04.806837+0000 mon.a (mon.0) 539 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-04-02T04:41:06.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:05 smithi069 bash[21528]: audit 2024-04-02T04:41:04.807158+0000 mon.a (mon.0) 540 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:05 smithi069 bash[28904]: cluster 2024-04-02T04:41:04.806837+0000 mon.a (mon.0) 539 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-04-02T04:41:06.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:05 smithi069 bash[28904]: audit 2024-04-02T04:41:04.807158+0000 mon.a (mon.0) 540 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:07.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:06 smithi098 bash[23790]: cluster 2024-04-02T04:41:05.135461+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:07.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:06 smithi069 bash[21528]: cluster 2024-04-02T04:41:05.135461+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:07.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:06 smithi069 bash[28904]: cluster 2024-04-02T04:41:05.135461+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:09.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:09.177+0000 7f888cd00700 10 client.24208.objecter tick 2024-04-02T04:41:09.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:08 smithi098 bash[23790]: cluster 2024-04-02T04:41:07.136118+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:09.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:08 smithi069 bash[21528]: cluster 2024-04-02T04:41:07.136118+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:09.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:08 smithi069 bash[28904]: cluster 2024-04-02T04:41:07.136118+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:10 smithi098 bash[23790]: cluster 2024-04-02T04:41:09.136689+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:10 smithi098 bash[23790]: cluster 2024-04-02T04:41:09.737675+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:10 smithi098 bash[23790]: cluster 2024-04-02T04:41:09.737769+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:10 smithi098 bash[23790]: cluster 2024-04-02T04:41:09.831910+0000 mon.a (mon.0) 541 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-04-02T04:41:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:10 smithi098 bash[23790]: audit 2024-04-02T04:41:09.832158+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:11.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[21528]: cluster 2024-04-02T04:41:09.136689+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:11.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[21528]: cluster 2024-04-02T04:41:09.737675+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:11.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[21528]: cluster 2024-04-02T04:41:09.737769+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:11.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[21528]: cluster 2024-04-02T04:41:09.831910+0000 mon.a (mon.0) 541 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-04-02T04:41:11.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[21528]: audit 2024-04-02T04:41:09.832158+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:11.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[28904]: cluster 2024-04-02T04:41:09.136689+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:11.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[28904]: cluster 2024-04-02T04:41:09.737675+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:11.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[28904]: cluster 2024-04-02T04:41:09.737769+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:11.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[28904]: cluster 2024-04-02T04:41:09.831910+0000 mon.a (mon.0) 541 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-04-02T04:41:11.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:10 smithi069 bash[28904]: audit 2024-04-02T04:41:09.832158+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:12 smithi098 bash[23790]: cluster 2024-04-02T04:41:11.137250+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:12 smithi069 bash[28904]: cluster 2024-04-02T04:41:11.137250+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:12 smithi069 bash[21528]: cluster 2024-04-02T04:41:11.137250+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:14.181+0000 7f888cd00700 10 client.24208.objecter tick 2024-04-02T04:41:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:14 smithi098 bash[23790]: cluster 2024-04-02T04:41:13.137781+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:15.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:14 smithi069 bash[28904]: cluster 2024-04-02T04:41:13.137781+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:15.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:14 smithi069 bash[21528]: cluster 2024-04-02T04:41:13.137781+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:16.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:15 smithi098 bash[23790]: audit 2024-04-02T04:41:15.166686+0000 mon.a (mon.0) 543 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:41:16.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:15 smithi098 bash[23790]: audit 2024-04-02T04:41:15.168685+0000 mon.a (mon.0) 544 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:16.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:15 smithi069 bash[28904]: audit 2024-04-02T04:41:15.166686+0000 mon.a (mon.0) 543 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:41:16.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:15 smithi069 bash[28904]: audit 2024-04-02T04:41:15.168685+0000 mon.a (mon.0) 544 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:16.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:15 smithi069 bash[21528]: audit 2024-04-02T04:41:15.166686+0000 mon.a (mon.0) 543 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:41:16.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:15 smithi069 bash[21528]: audit 2024-04-02T04:41:15.168685+0000 mon.a (mon.0) 544 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:16 smithi098 bash[23790]: cluster 2024-04-02T04:41:15.138374+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:17.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:16 smithi098 bash[23790]: cephadm 2024-04-02T04:41:15.170365+0000 mgr.y (mgr.14152) 247 : cephadm 1 Deploying daemon osd.5 on smithi098 2024-04-02T04:41:17.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:16 smithi069 bash[21528]: cluster 2024-04-02T04:41:15.138374+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:17.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:16 smithi069 bash[21528]: cephadm 2024-04-02T04:41:15.170365+0000 mgr.y (mgr.14152) 247 : cephadm 1 Deploying daemon osd.5 on smithi098 2024-04-02T04:41:17.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:16 smithi069 bash[28904]: cluster 2024-04-02T04:41:15.138374+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:17.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:16 smithi069 bash[28904]: cephadm 2024-04-02T04:41:15.170365+0000 mgr.y (mgr.14152) 247 : cephadm 1 Deploying daemon osd.5 on smithi098 2024-04-02T04:41:19.133 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:18 smithi098 bash[23790]: cluster 2024-04-02T04:41:17.138985+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:19.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:19.185+0000 7f888cd00700 10 client.24208.objecter tick 2024-04-02T04:41:19.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:18 smithi069 bash[21528]: cluster 2024-04-02T04:41:17.138985+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:19.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:18 smithi069 bash[28904]: cluster 2024-04-02T04:41:17.138985+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:19.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:41:19.393 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:41:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:41:19.394 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:41:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:41:19.674 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:41:19.674 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:41:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:41:19.675 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:41:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:41:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:20 smithi098 bash[23790]: cluster 2024-04-02T04:41:19.139631+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:20 smithi098 bash[23790]: audit 2024-04-02T04:41:19.680449+0000 mon.a (mon.0) 545 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:20 smithi098 bash[23790]: audit 2024-04-02T04:41:19.691788+0000 mon.a (mon.0) 546 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:20 smithi098 bash[23790]: audit 2024-04-02T04:41:19.720483+0000 mon.a (mon.0) 547 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:21.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[28904]: cluster 2024-04-02T04:41:19.139631+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[28904]: audit 2024-04-02T04:41:19.680449+0000 mon.a (mon.0) 545 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[28904]: audit 2024-04-02T04:41:19.691788+0000 mon.a (mon.0) 546 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[28904]: audit 2024-04-02T04:41:19.720483+0000 mon.a (mon.0) 547 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[21528]: cluster 2024-04-02T04:41:19.139631+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[21528]: audit 2024-04-02T04:41:19.680449+0000 mon.a (mon.0) 545 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[21528]: audit 2024-04-02T04:41:19.691788+0000 mon.a (mon.0) 546 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:20 smithi069 bash[21528]: audit 2024-04-02T04:41:19.720483+0000 mon.a (mon.0) 547 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:22.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:22 smithi098 bash[23790]: cluster 2024-04-02T04:41:21.140166+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:23.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:22 smithi069 bash[21528]: cluster 2024-04-02T04:41:21.140166+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:23.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:22 smithi069 bash[28904]: cluster 2024-04-02T04:41:21.140166+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:24.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:24.189+0000 7f888cd00700 10 client.24208.objecter tick 2024-04-02T04:41:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:24 smithi098 bash[23790]: cluster 2024-04-02T04:41:23.140859+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:25.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:24 smithi069 bash[21528]: cluster 2024-04-02T04:41:23.140859+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:25.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:24 smithi069 bash[28904]: cluster 2024-04-02T04:41:23.140859+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:26.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:25 smithi098 bash[23790]: audit 2024-04-02T04:41:24.966431+0000 mon.a (mon.0) 548 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:25 smithi098 bash[23790]: audit 2024-04-02T04:41:24.976367+0000 mon.a (mon.0) 549 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:25 smithi098 bash[23790]: audit 2024-04-02T04:41:24.978334+0000 mon.a (mon.0) 550 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:26.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:25 smithi098 bash[23790]: audit 2024-04-02T04:41:24.980111+0000 mon.a (mon.0) 551 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:26.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:25 smithi098 bash[23790]: audit 2024-04-02T04:41:24.990221+0000 mon.a (mon.0) 552 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[21528]: audit 2024-04-02T04:41:24.966431+0000 mon.a (mon.0) 548 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[21528]: audit 2024-04-02T04:41:24.976367+0000 mon.a (mon.0) 549 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[21528]: audit 2024-04-02T04:41:24.978334+0000 mon.a (mon.0) 550 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[21528]: audit 2024-04-02T04:41:24.980111+0000 mon.a (mon.0) 551 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[21528]: audit 2024-04-02T04:41:24.990221+0000 mon.a (mon.0) 552 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[28904]: audit 2024-04-02T04:41:24.966431+0000 mon.a (mon.0) 548 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[28904]: audit 2024-04-02T04:41:24.976367+0000 mon.a (mon.0) 549 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:26.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[28904]: audit 2024-04-02T04:41:24.978334+0000 mon.a (mon.0) 550 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:26.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[28904]: audit 2024-04-02T04:41:24.980111+0000 mon.a (mon.0) 551 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:26.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:25 smithi069 bash[28904]: audit 2024-04-02T04:41:24.990221+0000 mon.a (mon.0) 552 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:27.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:26 smithi098 bash[23790]: cluster 2024-04-02T04:41:25.141559+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:27.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:26 smithi098 bash[23790]: audit 2024-04-02T04:41:26.203564+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:41:27.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:26 smithi098 bash[23790]: audit 2024-04-02T04:41:26.207374+0000 mon.a (mon.0) 553 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:41:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:26 smithi069 bash[21528]: cluster 2024-04-02T04:41:25.141559+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:26 smithi069 bash[21528]: audit 2024-04-02T04:41:26.203564+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:41:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:26 smithi069 bash[21528]: audit 2024-04-02T04:41:26.207374+0000 mon.a (mon.0) 553 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:41:27.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:27 smithi069 bash[28904]: cluster 2024-04-02T04:41:25.141559+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:27.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:27 smithi069 bash[28904]: audit 2024-04-02T04:41:26.203564+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:41:27.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:27 smithi069 bash[28904]: audit 2024-04-02T04:41:26.207374+0000 mon.a (mon.0) 553 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:41:28.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:27 smithi098 bash[23790]: audit 2024-04-02T04:41:26.990699+0000 mon.a (mon.0) 554 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:41:28.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:27 smithi098 bash[23790]: audit 2024-04-02T04:41:26.994174+0000 mon.b (mon.1) 17 : audit 1 from='osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:41:28.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:27 smithi098 bash[23790]: cluster 2024-04-02T04:41:26.996194+0000 mon.a (mon.0) 555 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-04-02T04:41:28.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:27 smithi098 bash[23790]: audit 2024-04-02T04:41:26.997088+0000 mon.a (mon.0) 556 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:28.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:27 smithi098 bash[23790]: audit 2024-04-02T04:41:27.005495+0000 mon.a (mon.0) 557 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[21528]: audit 2024-04-02T04:41:26.990699+0000 mon.a (mon.0) 554 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[21528]: audit 2024-04-02T04:41:26.994174+0000 mon.b (mon.1) 17 : audit 1 from='osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[21528]: cluster 2024-04-02T04:41:26.996194+0000 mon.a (mon.0) 555 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[21528]: audit 2024-04-02T04:41:26.997088+0000 mon.a (mon.0) 556 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[21528]: audit 2024-04-02T04:41:27.005495+0000 mon.a (mon.0) 557 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[28904]: audit 2024-04-02T04:41:26.990699+0000 mon.a (mon.0) 554 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[28904]: audit 2024-04-02T04:41:26.994174+0000 mon.b (mon.1) 17 : audit 1 from='osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:41:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[28904]: cluster 2024-04-02T04:41:26.996194+0000 mon.a (mon.0) 555 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-04-02T04:41:28.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[28904]: audit 2024-04-02T04:41:26.997088+0000 mon.a (mon.0) 556 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:28.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:28 smithi069 bash[28904]: audit 2024-04-02T04:41:27.005495+0000 mon.a (mon.0) 557 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:41:28.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.509+0000 7f886ffff700 1 -- 172.21.15.98:0/2369162817 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8888102a90 con 0x7f887006f4f0 2024-04-02T04:41:28.514 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 5 on host 'smithi098' 2024-04-02T04:41:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 10 librados: watch_flush enter 2024-04-02T04:41:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 10 librados: watch_flush exit 2024-04-02T04:41:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 20 client.24208.objecter shutdown clearing up homeless session... 2024-04-02T04:41:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 10 client.24208.objecter successfully canceled tick 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f887006f4f0 msgr2=0x7f88700719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f887006f4f0 0x7f88700719b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f888819ab80 tx=0x7f8878009040 comp rx=0 tx=0).stop 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 msgr2=0x7f88881a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f88881a1250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8880006010 tx=0x7f88800047b0 comp rx=0 tx=0).stop 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 shutdown_connections 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8888103f20 0x7f88881a1250 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f887006f4f0 0x7f88700719b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88881005a0 0x7f8888199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:28.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 --2- 172.21.15.98:0/2369162817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8888103580 0x7f888819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:28.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 >> 172.21.15.98:0/2369162817 conn(0x7f88880fbf00 msgr2=0x7f8888101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:41:28.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 shutdown_connections 2024-04-02T04:41:28.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 -- 172.21.15.98:0/2369162817 wait complete. 2024-04-02T04:41:28.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:28.513+0000 7f888d501700 1 librados: shutdown 2024-04-02T04:41:29.223 DEBUG:teuthology.orchestra.run.smithi098:osd.5> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.5.service 2024-04-02T04:41:29.226 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-04-02T04:41:29.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: cluster 2024-04-02T04:41:27.142200+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.008674+0000 mon.a (mon.0) 558 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: cluster 2024-04-02T04:41:28.014752+0000 mon.a (mon.0) 559 : cluster 0 osdmap e46: 6 total, 5 up, 6 in 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.015439+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.021287+0000 mon.a (mon.0) 561 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.454099+0000 mon.a (mon.0) 562 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.463048+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.485844+0000 mon.a (mon.0) 564 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: cluster 2024-04-02T04:41:28.758397+0000 mon.a (mon.0) 565 : cluster 1 osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] boot 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: cluster 2024-04-02T04:41:28.758529+0000 mon.a (mon.0) 566 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[21528]: audit 2024-04-02T04:41:28.759261+0000 mon.a (mon.0) 567 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: cluster 2024-04-02T04:41:27.142200+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.008674+0000 mon.a (mon.0) 558 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: cluster 2024-04-02T04:41:28.014752+0000 mon.a (mon.0) 559 : cluster 0 osdmap e46: 6 total, 5 up, 6 in 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.015439+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.021287+0000 mon.a (mon.0) 561 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.454099+0000 mon.a (mon.0) 562 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.463048+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.485844+0000 mon.a (mon.0) 564 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: cluster 2024-04-02T04:41:28.758397+0000 mon.a (mon.0) 565 : cluster 1 osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] boot 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: cluster 2024-04-02T04:41:28.758529+0000 mon.a (mon.0) 566 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:29 smithi069 bash[28904]: audit 2024-04-02T04:41:28.759261+0000 mon.a (mon.0) 567 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: cluster 2024-04-02T04:41:27.142200+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.008674+0000 mon.a (mon.0) 558 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: cluster 2024-04-02T04:41:28.014752+0000 mon.a (mon.0) 559 : cluster 0 osdmap e46: 6 total, 5 up, 6 in 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.015439+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.021287+0000 mon.a (mon.0) 561 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.454099+0000 mon.a (mon.0) 562 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.463048+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:29.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.485844+0000 mon.a (mon.0) 564 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:29.284 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: cluster 2024-04-02T04:41:28.758397+0000 mon.a (mon.0) 565 : cluster 1 osd.5 [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] boot 2024-04-02T04:41:29.284 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: cluster 2024-04-02T04:41:28.758529+0000 mon.a (mon.0) 566 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-04-02T04:41:29.284 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:29 smithi098 bash[23790]: audit 2024-04-02T04:41:28.759261+0000 mon.a (mon.0) 567 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:41:30.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:30 smithi069 bash[21528]: cluster 2024-04-02T04:41:27.170382+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:30.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:30 smithi069 bash[21528]: cluster 2024-04-02T04:41:27.170475+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:30.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:30 smithi069 bash[21528]: cluster 2024-04-02T04:41:29.761028+0000 mon.a (mon.0) 568 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-04-02T04:41:30.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:30 smithi069 bash[28904]: cluster 2024-04-02T04:41:27.170382+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:30.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:30 smithi069 bash[28904]: cluster 2024-04-02T04:41:27.170475+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:30.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:30 smithi069 bash[28904]: cluster 2024-04-02T04:41:29.761028+0000 mon.a (mon.0) 568 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-04-02T04:41:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:30 smithi098 bash[23790]: cluster 2024-04-02T04:41:27.170382+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:30 smithi098 bash[23790]: cluster 2024-04-02T04:41:27.170475+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:30 smithi098 bash[23790]: cluster 2024-04-02T04:41:29.761028+0000 mon.a (mon.0) 568 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-04-02T04:41:31.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:31 smithi069 bash[21528]: cluster 2024-04-02T04:41:29.142708+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:31.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:31 smithi069 bash[21528]: cluster 2024-04-02T04:41:30.763974+0000 mon.a (mon.0) 569 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-04-02T04:41:31.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:31 smithi069 bash[28904]: cluster 2024-04-02T04:41:29.142708+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:31.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:31 smithi069 bash[28904]: cluster 2024-04-02T04:41:30.763974+0000 mon.a (mon.0) 569 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-04-02T04:41:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:31 smithi098 bash[23790]: cluster 2024-04-02T04:41:29.142708+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:31 smithi098 bash[23790]: cluster 2024-04-02T04:41:30.763974+0000 mon.a (mon.0) 569 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-04-02T04:41:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:32 smithi098 bash[23790]: cluster 2024-04-02T04:41:31.143368+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v241: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T04:41:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:32 smithi098 bash[23790]: cluster 2024-04-02T04:41:32.022559+0000 mon.a (mon.0) 570 : cluster 3 Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-02T04:41:32.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:32 smithi098 bash[23790]: cluster 2024-04-02T04:41:32.036840+0000 mon.a (mon.0) 571 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-04-02T04:41:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:32 smithi069 bash[21528]: cluster 2024-04-02T04:41:31.143368+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v241: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T04:41:32.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:32 smithi069 bash[21528]: cluster 2024-04-02T04:41:32.022559+0000 mon.a (mon.0) 570 : cluster 3 Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-02T04:41:32.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:32 smithi069 bash[21528]: cluster 2024-04-02T04:41:32.036840+0000 mon.a (mon.0) 571 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-04-02T04:41:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:32 smithi069 bash[28904]: cluster 2024-04-02T04:41:31.143368+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v241: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T04:41:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:32 smithi069 bash[28904]: cluster 2024-04-02T04:41:32.022559+0000 mon.a (mon.0) 570 : cluster 3 Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-02T04:41:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:32 smithi069 bash[28904]: cluster 2024-04-02T04:41:32.036840+0000 mon.a (mon.0) 571 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-04-02T04:41:33.481 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:41:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:34 smithi098 bash[23790]: cluster 2024-04-02T04:41:33.040013+0000 mon.a (mon.0) 572 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-04-02T04:41:34.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:34 smithi069 bash[21528]: cluster 2024-04-02T04:41:33.040013+0000 mon.a (mon.0) 572 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-04-02T04:41:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:34 smithi069 bash[28904]: cluster 2024-04-02T04:41:33.040013+0000 mon.a (mon.0) 572 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-04-02T04:41:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:35 smithi098 bash[23790]: cluster 2024-04-02T04:41:33.143983+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v244: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T04:41:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:35 smithi098 bash[23790]: cluster 2024-04-02T04:41:33.289269+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:41:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:35 smithi098 bash[23790]: cluster 2024-04-02T04:41:33.293097+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:41:35.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:35 smithi069 bash[28904]: cluster 2024-04-02T04:41:33.143983+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v244: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T04:41:35.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:35 smithi069 bash[28904]: cluster 2024-04-02T04:41:33.289269+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:41:35.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:35 smithi069 bash[28904]: cluster 2024-04-02T04:41:33.293097+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:41:35.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:35 smithi069 bash[21528]: cluster 2024-04-02T04:41:33.143983+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v244: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T04:41:35.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:35 smithi069 bash[21528]: cluster 2024-04-02T04:41:33.289269+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:41:35.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:35 smithi069 bash[21528]: cluster 2024-04-02T04:41:33.293097+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:41:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:36 smithi098 bash[23790]: cluster 2024-04-02T04:41:35.144558+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:36.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:36 smithi069 bash[28904]: cluster 2024-04-02T04:41:35.144558+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:36.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:36 smithi069 bash[21528]: cluster 2024-04-02T04:41:35.144558+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:37 smithi098 bash[23790]: cluster 2024-04-02T04:41:36.048243+0000 mon.a (mon.0) 573 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-02T04:41:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:37 smithi098 bash[23790]: cluster 2024-04-02T04:41:36.048334+0000 mon.a (mon.0) 574 : cluster 1 Cluster is now healthy 2024-04-02T04:41:37.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:37 smithi069 bash[21528]: cluster 2024-04-02T04:41:36.048243+0000 mon.a (mon.0) 573 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-02T04:41:37.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:37 smithi069 bash[21528]: cluster 2024-04-02T04:41:36.048334+0000 mon.a (mon.0) 574 : cluster 1 Cluster is now healthy 2024-04-02T04:41:37.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:37 smithi069 bash[28904]: cluster 2024-04-02T04:41:36.048243+0000 mon.a (mon.0) 573 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-02T04:41:37.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:37 smithi069 bash[28904]: cluster 2024-04-02T04:41:36.048334+0000 mon.a (mon.0) 574 : cluster 1 Cluster is now healthy 2024-04-02T04:41:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T04:41:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T04:41:37.624 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-02T04:41:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-02T04:41:37.624 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140494 s, 746 MB/s 2024-04-02T04:41:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-02T04:41:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:38 smithi098 bash[23790]: cluster 2024-04-02T04:41:37.145244+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:38.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:38 smithi069 bash[21528]: cluster 2024-04-02T04:41:37.145244+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:38.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:38 smithi069 bash[28904]: cluster 2024-04-02T04:41:37.145244+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:38.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-04-02T04:41:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:39 smithi098 bash[23790]: cephadm 2024-04-02T04:41:39.044503+0000 mgr.y (mgr.14152) 259 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:41:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:39 smithi098 bash[23790]: audit 2024-04-02T04:41:39.053611+0000 mon.a (mon.0) 575 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:39 smithi098 bash[23790]: audit 2024-04-02T04:41:39.063418+0000 mon.a (mon.0) 576 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:39 smithi098 bash[23790]: audit 2024-04-02T04:41:39.065872+0000 mon.a (mon.0) 577 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:41:39.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[21528]: cephadm 2024-04-02T04:41:39.044503+0000 mgr.y (mgr.14152) 259 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:41:39.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[21528]: audit 2024-04-02T04:41:39.053611+0000 mon.a (mon.0) 575 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:39.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[21528]: audit 2024-04-02T04:41:39.063418+0000 mon.a (mon.0) 576 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:39.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[21528]: audit 2024-04-02T04:41:39.065872+0000 mon.a (mon.0) 577 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:41:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[28904]: cephadm 2024-04-02T04:41:39.044503+0000 mgr.y (mgr.14152) 259 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:41:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[28904]: audit 2024-04-02T04:41:39.053611+0000 mon.a (mon.0) 575 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[28904]: audit 2024-04-02T04:41:39.063418+0000 mon.a (mon.0) 576 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:39 smithi069 bash[28904]: audit 2024-04-02T04:41:39.065872+0000 mon.a (mon.0) 577 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: cluster 2024-04-02T04:41:38.702357+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: cluster 2024-04-02T04:41:38.702477+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: audit 2024-04-02T04:41:39.067943+0000 mon.a (mon.0) 578 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: cephadm 2024-04-02T04:41:39.076582+0000 mgr.y (mgr.14152) 260 : cephadm 1 Adjusting osd_memory_target on smithi098 to 8632M 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: cluster 2024-04-02T04:41:39.082245+0000 mon.a (mon.0) 579 : cluster 0 osdmap e52: 6 total, 6 up, 6 in 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: audit 2024-04-02T04:41:39.088780+0000 mon.a (mon.0) 580 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: audit 2024-04-02T04:41:39.093341+0000 mon.a (mon.0) 581 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: audit 2024-04-02T04:41:39.095194+0000 mon.a (mon.0) 582 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: audit 2024-04-02T04:41:39.105269+0000 mon.a (mon.0) 583 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:40 smithi098 bash[23790]: cluster 2024-04-02T04:41:39.145967+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: cluster 2024-04-02T04:41:38.702357+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: cluster 2024-04-02T04:41:38.702477+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: audit 2024-04-02T04:41:39.067943+0000 mon.a (mon.0) 578 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: cephadm 2024-04-02T04:41:39.076582+0000 mgr.y (mgr.14152) 260 : cephadm 1 Adjusting osd_memory_target on smithi098 to 8632M 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: cluster 2024-04-02T04:41:39.082245+0000 mon.a (mon.0) 579 : cluster 0 osdmap e52: 6 total, 6 up, 6 in 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: audit 2024-04-02T04:41:39.088780+0000 mon.a (mon.0) 580 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: audit 2024-04-02T04:41:39.093341+0000 mon.a (mon.0) 581 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: audit 2024-04-02T04:41:39.095194+0000 mon.a (mon.0) 582 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: audit 2024-04-02T04:41:39.105269+0000 mon.a (mon.0) 583 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:40.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[28904]: cluster 2024-04-02T04:41:39.145967+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: cluster 2024-04-02T04:41:38.702357+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: cluster 2024-04-02T04:41:38.702477+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: audit 2024-04-02T04:41:39.067943+0000 mon.a (mon.0) 578 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: cephadm 2024-04-02T04:41:39.076582+0000 mgr.y (mgr.14152) 260 : cephadm 1 Adjusting osd_memory_target on smithi098 to 8632M 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: cluster 2024-04-02T04:41:39.082245+0000 mon.a (mon.0) 579 : cluster 0 osdmap e52: 6 total, 6 up, 6 in 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: audit 2024-04-02T04:41:39.088780+0000 mon.a (mon.0) 580 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: audit 2024-04-02T04:41:39.093341+0000 mon.a (mon.0) 581 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: audit 2024-04-02T04:41:39.095194+0000 mon.a (mon.0) 582 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: audit 2024-04-02T04:41:39.105269+0000 mon.a (mon.0) 583 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:41:40.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:40 smithi069 bash[21528]: cluster 2024-04-02T04:41:39.145967+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-02T04:41:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:42 smithi098 bash[23790]: cluster 2024-04-02T04:41:41.146610+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-02T04:41:42.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:42 smithi069 bash[21528]: cluster 2024-04-02T04:41:41.146610+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-02T04:41:42.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:42 smithi069 bash[28904]: cluster 2024-04-02T04:41:41.146610+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-02T04:41:43.373 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:41:44.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:44 smithi098 bash[23790]: cluster 2024-04-02T04:41:43.147177+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T04:41:45.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:44 smithi069 bash[21528]: cluster 2024-04-02T04:41:43.147177+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T04:41:45.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:44 smithi069 bash[28904]: cluster 2024-04-02T04:41:43.147177+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.993+0000 7f3dedef0700 1 -- 172.21.15.98:0/3949076493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 msgr2=0x7f3de8107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.993+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3949076493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8107a70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009990 tx=0x7f3dd0009ca0 comp rx=0 tx=0).stop 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.993+0000 7f3dedef0700 1 -- 172.21.15.98:0/3949076493 shutdown_connections 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.993+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3949076493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de8107fb0 0x7f3de810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3949076493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 0x7f3de8100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3949076493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8107a70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- 172.21.15.98:0/3949076493 >> 172.21.15.98:0/3949076493 conn(0x7f3de80fbaa0 msgr2=0x7f3de80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- 172.21.15.98:0/3949076493 shutdown_connections 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- 172.21.15.98:0/3949076493 wait complete. 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 librados: starting msgr at 2024-04-02T04:41:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 librados: starting objecter 2024-04-02T04:41:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 Processor -- start 2024-04-02T04:41:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- start start 2024-04-02T04:41:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 librados: setting wanted keys 2024-04-02T04:41:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 librados: calling monclient init 2024-04-02T04:41:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 0x7f3de8103360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de8107fb0 0x7f3de8101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810d430 con 0x7f3de8107fb0 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810d2b0 con 0x7f3de8100920 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3de810d5b0 con 0x7f3de80fff30 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50098/0 (socket says 172.21.15.98:50098) 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 -- 172.21.15.98:0/3386280263 learned_addr learned my addr 172.21.15.98:0/3386280263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de7fff700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de8107fb0 0x7f3de8101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de77fe700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 0x7f3de8103360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 -- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 msgr2=0x7f3de8103360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:41:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 0x7f3de8103360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 -- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de8107fb0 msgr2=0x7f3de8101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de8107fb0 0x7f3de8101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 -- 172.21.15.98:0/3386280263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de8102810 con 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de77fe700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 0x7f3de8103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3ddeffd700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8101a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0006010 tx=0x7f3dd00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de57fa700 10 client.?.objecter ms_handle_connect 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de57fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0027070 con 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3dd0004180 con 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de81a1dc0 con 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0022d90 con 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 10 client.24235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de81a2230 con 0x7f3de8100920 2024-04-02T04:41:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 librados: init done 2024-04-02T04:41:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de804ff00 con 0x7f3de8100920 2024-04-02T04:41:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:45.997+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f3dd002a070 con 0x7f3de8100920 2024-04-02T04:41:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3dd406f4f0 0x7f3dd40719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4423+0+0 (secure 0 0 0) 0x7f3dd008e3e0 con 0x7f3de8100920 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 10 client.24235.objecter ms_dispatch 0x7f3de8001550 osd_map(52..52 src has 1..52) 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 3 client.24235.objecter handle_osd_map got epochs [52,52] > 0 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 3 client.24235.objecter handle_osd_map decoding full epoch 52 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 20 client.24235.objecter dump_active .. 0 homeless 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de77fe700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3dd406f4f0 0x7f3dd40719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de77fe700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3dd406f4f0 0x7f3dd40719b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8007eb0 tx=0x7f3dd80073c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:41:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.001+0000 7f3de57fa700 10 client.24235.objecter ms_handle_connect 0x7f3dd406f4f0 2024-04-02T04:41:46.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.005+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3dd0013070 con 0x7f3de8100920 2024-04-02T04:41:46.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:46.221+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3de802de60 con 0x7f3dd406f4f0 2024-04-02T04:41:47.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[21528]: cluster 2024-04-02T04:41:45.147870+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:47.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[21528]: audit 2024-04-02T04:41:46.236987+0000 mon.a (mon.0) 584 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:41:47.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[21528]: audit 2024-04-02T04:41:46.243493+0000 mon.a (mon.0) 585 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:41:47.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[21528]: audit 2024-04-02T04:41:46.245360+0000 mon.a (mon.0) 586 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:47.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[28904]: cluster 2024-04-02T04:41:45.147870+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:47.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[28904]: audit 2024-04-02T04:41:46.236987+0000 mon.a (mon.0) 584 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:41:47.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[28904]: audit 2024-04-02T04:41:46.243493+0000 mon.a (mon.0) 585 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:41:47.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:46 smithi069 bash[28904]: audit 2024-04-02T04:41:46.245360+0000 mon.a (mon.0) 586 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:46 smithi098 bash[23790]: cluster 2024-04-02T04:41:45.147870+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:46 smithi098 bash[23790]: audit 2024-04-02T04:41:46.236987+0000 mon.a (mon.0) 584 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:41:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:46 smithi098 bash[23790]: audit 2024-04-02T04:41:46.243493+0000 mon.a (mon.0) 585 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:41:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:46 smithi098 bash[23790]: audit 2024-04-02T04:41:46.245360+0000 mon.a (mon.0) 586 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:48.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:47 smithi069 bash[21528]: audit 2024-04-02T04:41:46.230097+0000 mgr.y (mgr.14152) 265 : audit 0 from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:41:48.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:47 smithi069 bash[28904]: audit 2024-04-02T04:41:46.230097+0000 mgr.y (mgr.14152) 265 : audit 0 from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:41:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:47 smithi098 bash[23790]: audit 2024-04-02T04:41:46.230097+0000 mgr.y (mgr.14152) 265 : audit 0 from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:41:49.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:48 smithi069 bash[28904]: cluster 2024-04-02T04:41:47.148538+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:49.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:48 smithi069 bash[21528]: cluster 2024-04-02T04:41:47.148538+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:48 smithi098 bash[23790]: cluster 2024-04-02T04:41:47.148538+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:51.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:51.001+0000 7f3ded6ef700 10 client.24235.objecter tick 2024-04-02T04:41:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:50 smithi098 bash[23790]: cluster 2024-04-02T04:41:49.149127+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:51.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:50 smithi069 bash[28904]: cluster 2024-04-02T04:41:49.149127+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:50 smithi069 bash[21528]: cluster 2024-04-02T04:41:49.149127+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:52 smithi098 bash[23790]: cluster 2024-04-02T04:41:51.149685+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:52 smithi098 bash[23790]: cluster 2024-04-02T04:41:52.266894+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:52 smithi098 bash[23790]: cluster 2024-04-02T04:41:52.266983+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:52 smithi069 bash[21528]: cluster 2024-04-02T04:41:51.149685+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:52 smithi069 bash[21528]: cluster 2024-04-02T04:41:52.266894+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:52 smithi069 bash[21528]: cluster 2024-04-02T04:41:52.266983+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:52 smithi069 bash[28904]: cluster 2024-04-02T04:41:51.149685+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:52 smithi069 bash[28904]: cluster 2024-04-02T04:41:52.266894+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-04-02T04:41:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:52 smithi069 bash[28904]: cluster 2024-04-02T04:41:52.266983+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-04-02T04:41:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:53 smithi098 bash[23790]: cluster 2024-04-02T04:41:52.803380+0000 mon.a (mon.0) 587 : cluster 0 osdmap e53: 6 total, 6 up, 6 in 2024-04-02T04:41:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:53 smithi069 bash[21528]: cluster 2024-04-02T04:41:52.803380+0000 mon.a (mon.0) 587 : cluster 0 osdmap e53: 6 total, 6 up, 6 in 2024-04-02T04:41:54.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:53 smithi069 bash[28904]: cluster 2024-04-02T04:41:52.803380+0000 mon.a (mon.0) 587 : cluster 0 osdmap e53: 6 total, 6 up, 6 in 2024-04-02T04:41:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:54 smithi098 bash[23790]: cluster 2024-04-02T04:41:53.150240+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:54 smithi098 bash[23790]: audit 2024-04-02T04:41:54.569544+0000 mon.b (mon.1) 18 : audit 1 from='client.? 172.21.15.98:0/2520094827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]: dispatch 2024-04-02T04:41:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:54 smithi098 bash[23790]: audit 2024-04-02T04:41:54.574117+0000 mon.a (mon.0) 588 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]: dispatch 2024-04-02T04:41:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:54 smithi098 bash[23790]: audit 2024-04-02T04:41:54.578054+0000 mon.a (mon.0) 589 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]': finished 2024-04-02T04:41:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:54 smithi098 bash[23790]: cluster 2024-04-02T04:41:54.584031+0000 mon.a (mon.0) 590 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-04-02T04:41:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:54 smithi098 bash[23790]: audit 2024-04-02T04:41:54.584499+0000 mon.a (mon.0) 591 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:41:55.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[21528]: cluster 2024-04-02T04:41:53.150240+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[21528]: audit 2024-04-02T04:41:54.569544+0000 mon.b (mon.1) 18 : audit 1 from='client.? 172.21.15.98:0/2520094827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]: dispatch 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[21528]: audit 2024-04-02T04:41:54.574117+0000 mon.a (mon.0) 588 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]: dispatch 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[21528]: audit 2024-04-02T04:41:54.578054+0000 mon.a (mon.0) 589 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]': finished 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[21528]: cluster 2024-04-02T04:41:54.584031+0000 mon.a (mon.0) 590 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[21528]: audit 2024-04-02T04:41:54.584499+0000 mon.a (mon.0) 591 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[28904]: cluster 2024-04-02T04:41:53.150240+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[28904]: audit 2024-04-02T04:41:54.569544+0000 mon.b (mon.1) 18 : audit 1 from='client.? 172.21.15.98:0/2520094827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]: dispatch 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[28904]: audit 2024-04-02T04:41:54.574117+0000 mon.a (mon.0) 588 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]: dispatch 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[28904]: audit 2024-04-02T04:41:54.578054+0000 mon.a (mon.0) 589 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727"}]': finished 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[28904]: cluster 2024-04-02T04:41:54.584031+0000 mon.a (mon.0) 590 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-04-02T04:41:55.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:54 smithi069 bash[28904]: audit 2024-04-02T04:41:54.584499+0000 mon.a (mon.0) 591 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:41:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:41:56.005+0000 7f3ded6ef700 10 client.24235.objecter tick 2024-04-02T04:41:56.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:55 smithi098 bash[23790]: audit 2024-04-02T04:41:55.368402+0000 mon.b (mon.1) 19 : audit 0 from='client.? 172.21.15.98:0/1636098188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:41:56.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:55 smithi069 bash[21528]: audit 2024-04-02T04:41:55.368402+0000 mon.b (mon.1) 19 : audit 0 from='client.? 172.21.15.98:0/1636098188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:41:56.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:55 smithi069 bash[28904]: audit 2024-04-02T04:41:55.368402+0000 mon.b (mon.1) 19 : audit 0 from='client.? 172.21.15.98:0/1636098188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:41:57.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:56 smithi098 bash[23790]: cluster 2024-04-02T04:41:55.150775+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:57.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:56 smithi069 bash[21528]: cluster 2024-04-02T04:41:55.150775+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:57.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:56 smithi069 bash[28904]: cluster 2024-04-02T04:41:55.150775+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:41:58 smithi098 bash[23790]: cluster 2024-04-02T04:41:57.151349+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:59.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:41:58 smithi069 bash[21528]: cluster 2024-04-02T04:41:57.151349+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:41:59.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:41:58 smithi069 bash[28904]: cluster 2024-04-02T04:41:57.151349+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:01.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:01.009+0000 7f3ded6ef700 10 client.24235.objecter tick 2024-04-02T04:42:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:00 smithi098 bash[23790]: cluster 2024-04-02T04:41:59.151958+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:01.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:00 smithi069 bash[21528]: cluster 2024-04-02T04:41:59.151958+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:01.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:00 smithi069 bash[28904]: cluster 2024-04-02T04:41:59.151958+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:03.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:02 smithi098 bash[23790]: cluster 2024-04-02T04:42:01.152452+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:03.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:02 smithi069 bash[21528]: cluster 2024-04-02T04:42:01.152452+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:03.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:02 smithi069 bash[28904]: cluster 2024-04-02T04:42:01.152452+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:04.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:04 smithi098 bash[23790]: cluster 2024-04-02T04:42:03.152985+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:04 smithi069 bash[21528]: cluster 2024-04-02T04:42:03.152985+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:04 smithi069 bash[28904]: cluster 2024-04-02T04:42:03.152985+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:06.013+0000 7f3ded6ef700 10 client.24235.objecter tick 2024-04-02T04:42:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:06 smithi098 bash[23790]: cluster 2024-04-02T04:42:05.153573+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:07.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:06 smithi069 bash[21528]: cluster 2024-04-02T04:42:05.153573+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:07.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:06 smithi069 bash[28904]: cluster 2024-04-02T04:42:05.153573+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:08.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:07 smithi098 bash[23790]: audit 2024-04-02T04:42:07.071653+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:42:08.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:07 smithi098 bash[23790]: audit 2024-04-02T04:42:07.073470+0000 mon.a (mon.0) 593 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:08.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:07 smithi069 bash[21528]: audit 2024-04-02T04:42:07.071653+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:42:08.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:07 smithi069 bash[21528]: audit 2024-04-02T04:42:07.073470+0000 mon.a (mon.0) 593 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:08.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:07 smithi069 bash[28904]: audit 2024-04-02T04:42:07.071653+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:42:08.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:07 smithi069 bash[28904]: audit 2024-04-02T04:42:07.073470+0000 mon.a (mon.0) 593 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:08 smithi098 bash[23790]: cephadm 2024-04-02T04:42:07.075181+0000 mgr.y (mgr.14152) 276 : cephadm 1 Deploying daemon osd.6 on smithi098 2024-04-02T04:42:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:08 smithi098 bash[23790]: cluster 2024-04-02T04:42:07.154102+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:08 smithi069 bash[21528]: cephadm 2024-04-02T04:42:07.075181+0000 mgr.y (mgr.14152) 276 : cephadm 1 Deploying daemon osd.6 on smithi098 2024-04-02T04:42:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:08 smithi069 bash[21528]: cluster 2024-04-02T04:42:07.154102+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:09.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:08 smithi069 bash[28904]: cephadm 2024-04-02T04:42:07.075181+0000 mgr.y (mgr.14152) 276 : cephadm 1 Deploying daemon osd.6 on smithi098 2024-04-02T04:42:09.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:08 smithi069 bash[28904]: cluster 2024-04-02T04:42:07.154102+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:11.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:11.017+0000 7f3ded6ef700 10 client.24235.objecter tick 2024-04-02T04:42:11.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:10 smithi098 bash[23790]: cluster 2024-04-02T04:42:09.154682+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:11.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:10 smithi069 bash[21528]: cluster 2024-04-02T04:42:09.154682+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:11.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:10 smithi069 bash[28904]: cluster 2024-04-02T04:42:09.154682+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:11.474 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.475 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.475 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.979 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.980 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:11.980 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:42:11 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:42:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:12 smithi098 bash[23790]: cluster 2024-04-02T04:42:11.155213+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:12 smithi098 bash[23790]: audit 2024-04-02T04:42:11.662841+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:12 smithi098 bash[23790]: audit 2024-04-02T04:42:11.672377+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:12 smithi098 bash[23790]: audit 2024-04-02T04:42:11.676593+0000 mon.a (mon.0) 596 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:13.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[21528]: cluster 2024-04-02T04:42:11.155213+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:13.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[21528]: audit 2024-04-02T04:42:11.662841+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:13.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[21528]: audit 2024-04-02T04:42:11.672377+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:13.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[21528]: audit 2024-04-02T04:42:11.676593+0000 mon.a (mon.0) 596 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:13.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[28904]: cluster 2024-04-02T04:42:11.155213+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:13.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[28904]: audit 2024-04-02T04:42:11.662841+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:13.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[28904]: audit 2024-04-02T04:42:11.672377+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:13.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:12 smithi069 bash[28904]: audit 2024-04-02T04:42:11.676593+0000 mon.a (mon.0) 596 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:14 smithi098 bash[23790]: cluster 2024-04-02T04:42:13.155792+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:15.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:14 smithi069 bash[21528]: cluster 2024-04-02T04:42:13.155792+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:15.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:14 smithi069 bash[28904]: cluster 2024-04-02T04:42:13.155792+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:16.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:16.021+0000 7f3ded6ef700 10 client.24235.objecter tick 2024-04-02T04:42:16.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:15 smithi098 bash[23790]: cluster 2024-04-02T04:42:15.447980+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:16.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:15 smithi098 bash[23790]: cluster 2024-04-02T04:42:15.448072+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:15 smithi069 bash[21528]: cluster 2024-04-02T04:42:15.447980+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:15 smithi069 bash[21528]: cluster 2024-04-02T04:42:15.448072+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:16.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:15 smithi069 bash[28904]: cluster 2024-04-02T04:42:15.447980+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:16.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:15 smithi069 bash[28904]: cluster 2024-04-02T04:42:15.448072+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:16 smithi098 bash[23790]: cluster 2024-04-02T04:42:15.156343+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:16.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:16 smithi098 bash[23790]: cluster 2024-04-02T04:42:15.793286+0000 mon.a (mon.0) 597 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-04-02T04:42:16.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:16 smithi098 bash[23790]: audit 2024-04-02T04:42:15.793600+0000 mon.a (mon.0) 598 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:17.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:16 smithi069 bash[21528]: cluster 2024-04-02T04:42:15.156343+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:17.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:16 smithi069 bash[21528]: cluster 2024-04-02T04:42:15.793286+0000 mon.a (mon.0) 597 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-04-02T04:42:17.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:16 smithi069 bash[21528]: audit 2024-04-02T04:42:15.793600+0000 mon.a (mon.0) 598 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:17.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:16 smithi069 bash[28904]: cluster 2024-04-02T04:42:15.156343+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:17.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:16 smithi069 bash[28904]: cluster 2024-04-02T04:42:15.793286+0000 mon.a (mon.0) 597 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-04-02T04:42:17.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:16 smithi069 bash[28904]: audit 2024-04-02T04:42:15.793600+0000 mon.a (mon.0) 598 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:18.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:18 smithi098 bash[23790]: audit 2024-04-02T04:42:17.023222+0000 mon.a (mon.0) 599 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:18 smithi098 bash[23790]: audit 2024-04-02T04:42:17.034900+0000 mon.a (mon.0) 600 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:18 smithi098 bash[23790]: audit 2024-04-02T04:42:17.714319+0000 mon.a (mon.0) 601 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:18.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:18 smithi098 bash[23790]: audit 2024-04-02T04:42:17.716337+0000 mon.a (mon.0) 602 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:18.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:18 smithi098 bash[23790]: audit 2024-04-02T04:42:17.726699+0000 mon.a (mon.0) 603 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[21528]: audit 2024-04-02T04:42:17.023222+0000 mon.a (mon.0) 599 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[21528]: audit 2024-04-02T04:42:17.034900+0000 mon.a (mon.0) 600 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[21528]: audit 2024-04-02T04:42:17.714319+0000 mon.a (mon.0) 601 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[21528]: audit 2024-04-02T04:42:17.716337+0000 mon.a (mon.0) 602 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[21528]: audit 2024-04-02T04:42:17.726699+0000 mon.a (mon.0) 603 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[28904]: audit 2024-04-02T04:42:17.023222+0000 mon.a (mon.0) 599 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[28904]: audit 2024-04-02T04:42:17.034900+0000 mon.a (mon.0) 600 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:18.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[28904]: audit 2024-04-02T04:42:17.714319+0000 mon.a (mon.0) 601 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:18.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[28904]: audit 2024-04-02T04:42:17.716337+0000 mon.a (mon.0) 602 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:18.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:18 smithi069 bash[28904]: audit 2024-04-02T04:42:17.726699+0000 mon.a (mon.0) 603 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:19.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:19 smithi098 bash[23790]: cluster 2024-04-02T04:42:17.156895+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:19.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:19 smithi098 bash[23790]: audit 2024-04-02T04:42:18.271074+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:42:19.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:19 smithi098 bash[23790]: audit 2024-04-02T04:42:18.275379+0000 mon.a (mon.0) 604 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:42:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:19 smithi069 bash[21528]: cluster 2024-04-02T04:42:17.156895+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:19 smithi069 bash[21528]: audit 2024-04-02T04:42:18.271074+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:42:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:19 smithi069 bash[21528]: audit 2024-04-02T04:42:18.275379+0000 mon.a (mon.0) 604 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:42:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:19 smithi069 bash[28904]: cluster 2024-04-02T04:42:17.156895+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:19 smithi069 bash[28904]: audit 2024-04-02T04:42:18.271074+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:42:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:19 smithi069 bash[28904]: audit 2024-04-02T04:42:18.275379+0000 mon.a (mon.0) 604 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:42:20.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: audit 2024-04-02T04:42:19.036915+0000 mon.a (mon.0) 605 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:42:20.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: audit 2024-04-02T04:42:19.040427+0000 mon.b (mon.1) 21 : audit 1 from='osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:42:20.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: cluster 2024-04-02T04:42:19.043796+0000 mon.a (mon.0) 606 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-04-02T04:42:20.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: audit 2024-04-02T04:42:19.044672+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:20.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: audit 2024-04-02T04:42:19.045548+0000 mon.a (mon.0) 608 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:42:20.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: audit 2024-04-02T04:42:20.039528+0000 mon.a (mon.0) 609 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:42:20.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:20 smithi098 bash[23790]: cluster 2024-04-02T04:42:20.045684+0000 mon.a (mon.0) 610 : cluster 0 osdmap e57: 7 total, 6 up, 7 in 2024-04-02T04:42:20.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: audit 2024-04-02T04:42:19.036915+0000 mon.a (mon.0) 605 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:42:20.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: audit 2024-04-02T04:42:19.040427+0000 mon.b (mon.1) 21 : audit 1 from='osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:42:20.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: cluster 2024-04-02T04:42:19.043796+0000 mon.a (mon.0) 606 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-04-02T04:42:20.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: audit 2024-04-02T04:42:19.044672+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:20.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: audit 2024-04-02T04:42:19.045548+0000 mon.a (mon.0) 608 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:42:20.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: audit 2024-04-02T04:42:20.039528+0000 mon.a (mon.0) 609 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[21528]: cluster 2024-04-02T04:42:20.045684+0000 mon.a (mon.0) 610 : cluster 0 osdmap e57: 7 total, 6 up, 7 in 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: audit 2024-04-02T04:42:19.036915+0000 mon.a (mon.0) 605 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: audit 2024-04-02T04:42:19.040427+0000 mon.b (mon.1) 21 : audit 1 from='osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: cluster 2024-04-02T04:42:19.043796+0000 mon.a (mon.0) 606 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: audit 2024-04-02T04:42:19.044672+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: audit 2024-04-02T04:42:19.045548+0000 mon.a (mon.0) 608 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: audit 2024-04-02T04:42:20.039528+0000 mon.a (mon.0) 609 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:42:20.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:20 smithi069 bash[28904]: cluster 2024-04-02T04:42:20.045684+0000 mon.a (mon.0) 610 : cluster 0 osdmap e57: 7 total, 6 up, 7 in 2024-04-02T04:42:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.705+0000 7f3de57fa700 1 -- 172.21.15.98:0/3386280263 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3de802de60 con 0x7f3dd406f4f0 2024-04-02T04:42:20.713 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-04-02T04:42:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 10 librados: watch_flush enter 2024-04-02T04:42:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 10 librados: watch_flush exit 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 20 client.24235.objecter shutdown clearing up homeless session... 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 10 client.24235.objecter successfully canceled tick 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3dd406f4f0 msgr2=0x7f3dd40719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3dd406f4f0 0x7f3dd40719b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8007eb0 tx=0x7f3dd80073c0 comp rx=0 tx=0).stop 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 msgr2=0x7f3de8101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8101a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0006010 tx=0x7f3dd00057c0 comp rx=0 tx=0).stop 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 shutdown_connections 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3de8107fb0 0x7f3de8101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3dd406f4f0 0x7f3dd40719b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3de80fff30 0x7f3de8103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 --2- 172.21.15.98:0/3386280263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3de8100920 0x7f3de8101a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 >> 172.21.15.98:0/3386280263 conn(0x7f3de80fbaa0 msgr2=0x7f3de8103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:42:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 shutdown_connections 2024-04-02T04:42:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 -- 172.21.15.98:0/3386280263 wait complete. 2024-04-02T04:42:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:20.709+0000 7f3dedef0700 1 librados: shutdown 2024-04-02T04:42:21.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:21 smithi098 bash[23790]: cluster 2024-04-02T04:42:19.157430+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:21.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:21 smithi098 bash[23790]: audit 2024-04-02T04:42:20.046374+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:21.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:21 smithi098 bash[23790]: audit 2024-04-02T04:42:20.053608+0000 mon.a (mon.0) 612 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:21.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:21 smithi098 bash[23790]: audit 2024-04-02T04:42:20.687447+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:21.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:21 smithi098 bash[23790]: audit 2024-04-02T04:42:20.698439+0000 mon.a (mon.0) 614 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:21.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:21 smithi098 bash[23790]: audit 2024-04-02T04:42:20.721240+0000 mon.a (mon.0) 615 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:21.379 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.6.service 2024-04-02T04:42:21.382 INFO:tasks.cephadm:Deploying osd.7 on smithi098 with /dev/vg_nvme/lv_1... 2024-04-02T04:42:21.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T04:42:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[21528]: cluster 2024-04-02T04:42:19.157430+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[21528]: audit 2024-04-02T04:42:20.046374+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[21528]: audit 2024-04-02T04:42:20.053608+0000 mon.a (mon.0) 612 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[21528]: audit 2024-04-02T04:42:20.687447+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[21528]: audit 2024-04-02T04:42:20.698439+0000 mon.a (mon.0) 614 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[21528]: audit 2024-04-02T04:42:20.721240+0000 mon.a (mon.0) 615 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[28904]: cluster 2024-04-02T04:42:19.157430+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[28904]: audit 2024-04-02T04:42:20.046374+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[28904]: audit 2024-04-02T04:42:20.053608+0000 mon.a (mon.0) 612 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[28904]: audit 2024-04-02T04:42:20.687447+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[28904]: audit 2024-04-02T04:42:20.698439+0000 mon.a (mon.0) 614 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:21 smithi069 bash[28904]: audit 2024-04-02T04:42:20.721240+0000 mon.a (mon.0) 615 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:22.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:22 smithi098 bash[23790]: cluster 2024-04-02T04:42:19.279541+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:22.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:22 smithi098 bash[23790]: cluster 2024-04-02T04:42:19.279622+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:22.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:22 smithi098 bash[23790]: cluster 2024-04-02T04:42:21.062413+0000 mon.a (mon.0) 616 : cluster 1 osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] boot 2024-04-02T04:42:22.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:22 smithi098 bash[23790]: cluster 2024-04-02T04:42:21.062526+0000 mon.a (mon.0) 617 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-04-02T04:42:22.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:22 smithi098 bash[23790]: audit 2024-04-02T04:42:21.062796+0000 mon.a (mon.0) 618 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:22.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:22 smithi098 bash[23790]: cluster 2024-04-02T04:42:21.157912+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:22.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[21528]: cluster 2024-04-02T04:42:19.279541+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:22.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[21528]: cluster 2024-04-02T04:42:19.279622+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:22.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[21528]: cluster 2024-04-02T04:42:21.062413+0000 mon.a (mon.0) 616 : cluster 1 osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] boot 2024-04-02T04:42:22.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[21528]: cluster 2024-04-02T04:42:21.062526+0000 mon.a (mon.0) 617 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-04-02T04:42:22.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[21528]: audit 2024-04-02T04:42:21.062796+0000 mon.a (mon.0) 618 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:22.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[21528]: cluster 2024-04-02T04:42:21.157912+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:22.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[28904]: cluster 2024-04-02T04:42:19.279541+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:22.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[28904]: cluster 2024-04-02T04:42:19.279622+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:22.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[28904]: cluster 2024-04-02T04:42:21.062413+0000 mon.a (mon.0) 616 : cluster 1 osd.6 [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] boot 2024-04-02T04:42:22.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[28904]: cluster 2024-04-02T04:42:21.062526+0000 mon.a (mon.0) 617 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-04-02T04:42:22.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[28904]: audit 2024-04-02T04:42:21.062796+0000 mon.a (mon.0) 618 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:42:22.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:22 smithi069 bash[28904]: cluster 2024-04-02T04:42:21.157912+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:42:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:23 smithi098 bash[23790]: cluster 2024-04-02T04:42:22.091355+0000 mon.a (mon.0) 619 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-04-02T04:42:23.526 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:23 smithi098 bash[23790]: cluster 2024-04-02T04:42:22.395248+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:23.526 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:23 smithi098 bash[23790]: cluster 2024-04-02T04:42:22.395330+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:23.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:23 smithi069 bash[28904]: cluster 2024-04-02T04:42:22.091355+0000 mon.a (mon.0) 619 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-04-02T04:42:23.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:23 smithi069 bash[28904]: cluster 2024-04-02T04:42:22.395248+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:23.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:23 smithi069 bash[28904]: cluster 2024-04-02T04:42:22.395330+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:23.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:23 smithi069 bash[21528]: cluster 2024-04-02T04:42:22.091355+0000 mon.a (mon.0) 619 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-04-02T04:42:23.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:23 smithi069 bash[21528]: cluster 2024-04-02T04:42:22.395248+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:23.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:23 smithi069 bash[21528]: cluster 2024-04-02T04:42:22.395330+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:24 smithi098 bash[23790]: cluster 2024-04-02T04:42:23.087723+0000 mon.a (mon.0) 620 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-04-02T04:42:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:24 smithi098 bash[23790]: cluster 2024-04-02T04:42:23.158573+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:24 smithi098 bash[23790]: cluster 2024-04-02T04:42:23.774455+0000 mon.a (mon.0) 621 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-04-02T04:42:24.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:24 smithi069 bash[21528]: cluster 2024-04-02T04:42:23.087723+0000 mon.a (mon.0) 620 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-04-02T04:42:24.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:24 smithi069 bash[21528]: cluster 2024-04-02T04:42:23.158573+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:24.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:24 smithi069 bash[21528]: cluster 2024-04-02T04:42:23.774455+0000 mon.a (mon.0) 621 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-04-02T04:42:24.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:24 smithi069 bash[28904]: cluster 2024-04-02T04:42:23.087723+0000 mon.a (mon.0) 620 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-04-02T04:42:24.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:24 smithi069 bash[28904]: cluster 2024-04-02T04:42:23.158573+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:24.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:24 smithi069 bash[28904]: cluster 2024-04-02T04:42:23.774455+0000 mon.a (mon.0) 621 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-04-02T04:42:25.673 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:42:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:25 smithi098 bash[23790]: cluster 2024-04-02T04:42:24.776664+0000 mon.a (mon.0) 622 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-04-02T04:42:26.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:25 smithi069 bash[28904]: cluster 2024-04-02T04:42:24.776664+0000 mon.a (mon.0) 622 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-04-02T04:42:26.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:25 smithi069 bash[21528]: cluster 2024-04-02T04:42:24.776664+0000 mon.a (mon.0) 622 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-04-02T04:42:27.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:26 smithi069 bash[21528]: cluster 2024-04-02T04:42:25.159232+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:27.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:26 smithi069 bash[28904]: cluster 2024-04-02T04:42:25.159232+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:27.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:26 smithi098 bash[23790]: cluster 2024-04-02T04:42:25.159232+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:29.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:28 smithi098 bash[23790]: cluster 2024-04-02T04:42:27.159901+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-02T04:42:29.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:28 smithi069 bash[21528]: cluster 2024-04-02T04:42:27.159901+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-02T04:42:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:28 smithi069 bash[28904]: cluster 2024-04-02T04:42:27.159901+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-02T04:42:29.941 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T04:42:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T04:42:29.942 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-02T04:42:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-02T04:42:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133761 s, 784 MB/s 2024-04-02T04:42:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-02T04:42:30.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-04-02T04:42:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:30 smithi098 bash[23790]: cluster 2024-04-02T04:42:29.160618+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-02T04:42:31.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:30 smithi069 bash[21528]: cluster 2024-04-02T04:42:29.160618+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-02T04:42:31.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:30 smithi069 bash[28904]: cluster 2024-04-02T04:42:29.160618+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: cluster 2024-04-02T04:42:31.161368+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: cephadm 2024-04-02T04:42:31.417991+0000 mgr.y (mgr.14152) 290 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.427234+0000 mon.a (mon.0) 623 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.438217+0000 mon.a (mon.0) 624 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.440667+0000 mon.a (mon.0) 625 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.442394+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.444230+0000 mon.a (mon.0) 627 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: cephadm 2024-04-02T04:42:31.445650+0000 mgr.y (mgr.14152) 291 : cephadm 1 Adjusting osd_memory_target on smithi098 to 5755M 2024-04-02T04:42:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.452100+0000 mon.a (mon.0) 628 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.456417+0000 mon.a (mon.0) 629 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:32.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.458029+0000 mon.a (mon.0) 630 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:32.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:32 smithi098 bash[23790]: audit 2024-04-02T04:42:31.467775+0000 mon.a (mon.0) 631 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: cluster 2024-04-02T04:42:31.161368+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T04:42:32.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: cephadm 2024-04-02T04:42:31.417991+0000 mgr.y (mgr.14152) 290 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.427234+0000 mon.a (mon.0) 623 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.438217+0000 mon.a (mon.0) 624 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.440667+0000 mon.a (mon.0) 625 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.442394+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.444230+0000 mon.a (mon.0) 627 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: cephadm 2024-04-02T04:42:31.445650+0000 mgr.y (mgr.14152) 291 : cephadm 1 Adjusting osd_memory_target on smithi098 to 5755M 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.452100+0000 mon.a (mon.0) 628 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.456417+0000 mon.a (mon.0) 629 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:32.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.458029+0000 mon.a (mon.0) 630 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:32.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[28904]: audit 2024-04-02T04:42:31.467775+0000 mon.a (mon.0) 631 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: cluster 2024-04-02T04:42:31.161368+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T04:42:32.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: cephadm 2024-04-02T04:42:31.417991+0000 mgr.y (mgr.14152) 290 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:42:32.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.427234+0000 mon.a (mon.0) 623 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.438217+0000 mon.a (mon.0) 624 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.440667+0000 mon.a (mon.0) 625 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.442394+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.444230+0000 mon.a (mon.0) 627 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: cephadm 2024-04-02T04:42:31.445650+0000 mgr.y (mgr.14152) 291 : cephadm 1 Adjusting osd_memory_target on smithi098 to 5755M 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.452100+0000 mon.a (mon.0) 628 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.456417+0000 mon.a (mon.0) 629 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.458029+0000 mon.a (mon.0) 630 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:32.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:32 smithi069 bash[21528]: audit 2024-04-02T04:42:31.467775+0000 mon.a (mon.0) 631 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:42:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:34 smithi098 bash[23790]: cluster 2024-04-02T04:42:33.162024+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-02T04:42:34.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:34 smithi069 bash[21528]: cluster 2024-04-02T04:42:33.162024+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-02T04:42:34.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:34 smithi069 bash[28904]: cluster 2024-04-02T04:42:33.162024+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-02T04:42:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:42:36.694 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:36 smithi098 bash[23790]: cluster 2024-04-02T04:42:35.162602+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-02T04:42:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:36 smithi069 bash[21528]: cluster 2024-04-02T04:42:35.162602+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-02T04:42:36.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:36 smithi069 bash[28904]: cluster 2024-04-02T04:42:35.162602+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/930936191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 msgr2=0x7f3ba8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- 172.21.15.98:0/930936191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8103930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b980099c0 tx=0x7f3b98009cd0 comp rx=0 tx=0).stop 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/930936191 shutdown_connections 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- 172.21.15.98:0/930936191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ba8103e70 0x7f3ba810a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- 172.21.15.98:0/930936191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 0x7f3ba81026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- 172.21.15.98:0/930936191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8103930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/930936191 >> 172.21.15.98:0/930936191 conn(0x7f3ba80fd870 msgr2=0x7f3ba80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:42:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/930936191 shutdown_connections 2024-04-02T04:42:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/930936191 wait complete. 2024-04-02T04:42:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 librados: starting msgr at 2024-04-02T04:42:38.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 librados: starting objecter 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 Processor -- start 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- start start 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 librados: setting wanted keys 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 librados: calling monclient init 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 0x7f3ba8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ba8103e70 0x7f3ba819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba810d450 con 0x7f3ba81022d0 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba810d2d0 con 0x7f3ba81034d0 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba810d5d0 con 0x7f3ba8103e70 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59408/0 (socket says 172.21.15.98:59408) 2024-04-02T04:42:38.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 -- 172.21.15.98:0/2597364642 learned_addr learned my addr 172.21.15.98:0/2597364642 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba6ffd700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 0x7f3ba8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 -- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ba8103e70 msgr2=0x7f3ba819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ba8103e70 0x7f3ba819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 -- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 msgr2=0x7f3ba8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 0x7f3ba8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 -- 172.21.15.98:0/2597364642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba819f750 con 0x7f3ba81034d0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba6ffd700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 0x7f3ba8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3ba67fc700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8198030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b98005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3baca54700 10 client.?.objecter ms_handle_connect 0x7f3ba81034d0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3baca54700 10 client.?.objecter resend_mon_ops 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98027070 con 0x7f3ba81034d0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3b98004180 con 0x7f3ba81034d0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98022e60 con 0x7f3ba81034d0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba819f9e0 con 0x7f3ba81034d0 2024-04-02T04:42:38.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 10 client.24262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:42:38.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.348+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba819fe30 con 0x7f3ba81034d0 2024-04-02T04:42:38.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3bada56700 1 librados: init done 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba804ff00 con 0x7f3ba81034d0 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f3b9802a070 con 0x7f3ba81034d0 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3b8806f4f0 0x7f3b880719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(62..62 src has 12..62) ==== 4855+0+0 (secure 0 0 0) 0x7f3b9808e1f0 con 0x7f3ba81034d0 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 10 client.24262.objecter ms_dispatch 0x7f3ba8001550 osd_map(62..62 src has 12..62) 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 3 client.24262.objecter handle_osd_map got epochs [62,62] > 0 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 3 client.24262.objecter handle_osd_map decoding full epoch 62 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 20 client.24262.objecter dump_active .. 0 homeless 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3ba6ffd700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3b8806f4f0 0x7f3b880719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3baca54700 10 client.24262.objecter ms_handle_connect 0x7f3b8806f4f0 2024-04-02T04:42:38.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.352+0000 7f3ba6ffd700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3b8806f4f0 0x7f3b880719b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8103330 tx=0x7f3b90009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:42:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.356+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3b98013070 con 0x7f3ba81034d0 2024-04-02T04:42:38.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:38 smithi098 bash[23790]: cluster 2024-04-02T04:42:37.163193+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-02T04:42:38.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:38.564+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3ba802df40 con 0x7f3b8806f4f0 2024-04-02T04:42:38.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:38 smithi069 bash[21528]: cluster 2024-04-02T04:42:37.163193+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-02T04:42:38.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:38 smithi069 bash[28904]: cluster 2024-04-02T04:42:37.163193+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-02T04:42:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:39 smithi098 bash[23790]: audit 2024-04-02T04:42:38.575027+0000 mgr.y (mgr.14152) 295 : audit 0 from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:39 smithi098 bash[23790]: audit 2024-04-02T04:42:38.582048+0000 mon.a (mon.0) 632 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:42:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:39 smithi098 bash[23790]: audit 2024-04-02T04:42:38.588392+0000 mon.a (mon.0) 633 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:42:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:39 smithi098 bash[23790]: audit 2024-04-02T04:42:38.590037+0000 mon.a (mon.0) 634 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[21528]: audit 2024-04-02T04:42:38.575027+0000 mgr.y (mgr.14152) 295 : audit 0 from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[21528]: audit 2024-04-02T04:42:38.582048+0000 mon.a (mon.0) 632 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[21528]: audit 2024-04-02T04:42:38.588392+0000 mon.a (mon.0) 633 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[21528]: audit 2024-04-02T04:42:38.590037+0000 mon.a (mon.0) 634 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[28904]: audit 2024-04-02T04:42:38.575027+0000 mgr.y (mgr.14152) 295 : audit 0 from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[28904]: audit 2024-04-02T04:42:38.582048+0000 mon.a (mon.0) 632 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[28904]: audit 2024-04-02T04:42:38.588392+0000 mon.a (mon.0) 633 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:42:39.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:39 smithi069 bash[28904]: audit 2024-04-02T04:42:38.590037+0000 mon.a (mon.0) 634 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:40.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:40 smithi098 bash[23790]: cluster 2024-04-02T04:42:39.163759+0000 mgr.y (mgr.14152) 296 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:40.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:40 smithi098 bash[23790]: cluster 2024-04-02T04:42:39.969750+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:40.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:40 smithi098 bash[23790]: cluster 2024-04-02T04:42:39.969844+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:40.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:40 smithi069 bash[28904]: cluster 2024-04-02T04:42:39.163759+0000 mgr.y (mgr.14152) 296 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:40.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:40 smithi069 bash[28904]: cluster 2024-04-02T04:42:39.969750+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:40.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:40 smithi069 bash[28904]: cluster 2024-04-02T04:42:39.969844+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:40.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:40 smithi069 bash[21528]: cluster 2024-04-02T04:42:39.163759+0000 mgr.y (mgr.14152) 296 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:40.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:40 smithi069 bash[21528]: cluster 2024-04-02T04:42:39.969750+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:40.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:40 smithi069 bash[21528]: cluster 2024-04-02T04:42:39.969844+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:41.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:41 smithi098 bash[23790]: cluster 2024-04-02T04:42:39.689827+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:41.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:41 smithi098 bash[23790]: cluster 2024-04-02T04:42:39.689918+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:41.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:41 smithi098 bash[23790]: cluster 2024-04-02T04:42:40.455139+0000 mon.a (mon.0) 635 : cluster 0 osdmap e63: 7 total, 7 up, 7 in 2024-04-02T04:42:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:41 smithi069 bash[28904]: cluster 2024-04-02T04:42:39.689827+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:41.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:41 smithi069 bash[28904]: cluster 2024-04-02T04:42:39.689918+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:41.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:41 smithi069 bash[28904]: cluster 2024-04-02T04:42:40.455139+0000 mon.a (mon.0) 635 : cluster 0 osdmap e63: 7 total, 7 up, 7 in 2024-04-02T04:42:41.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:41 smithi069 bash[21528]: cluster 2024-04-02T04:42:39.689827+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:41.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:41 smithi069 bash[21528]: cluster 2024-04-02T04:42:39.689918+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:41.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:41 smithi069 bash[21528]: cluster 2024-04-02T04:42:40.455139+0000 mon.a (mon.0) 635 : cluster 0 osdmap e63: 7 total, 7 up, 7 in 2024-04-02T04:42:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:42 smithi098 bash[23790]: cluster 2024-04-02T04:42:41.164280+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:42.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:42 smithi069 bash[21528]: cluster 2024-04-02T04:42:41.164280+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:42.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:42 smithi069 bash[28904]: cluster 2024-04-02T04:42:41.164280+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:43.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:43.352+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:42:43.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:43 smithi098 bash[23790]: cluster 2024-04-02T04:42:42.377155+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:42:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:43 smithi098 bash[23790]: cluster 2024-04-02T04:42:42.380623+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:42:43.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:43 smithi069 bash[21528]: cluster 2024-04-02T04:42:42.377155+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:42:43.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:43 smithi069 bash[21528]: cluster 2024-04-02T04:42:42.380623+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:42:43.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:43 smithi069 bash[28904]: cluster 2024-04-02T04:42:42.377155+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:42:43.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:43 smithi069 bash[28904]: cluster 2024-04-02T04:42:42.380623+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:42:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:44 smithi098 bash[23790]: cluster 2024-04-02T04:42:43.164877+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:44.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:44 smithi069 bash[21528]: cluster 2024-04-02T04:42:43.164877+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:44.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:44 smithi069 bash[28904]: cluster 2024-04-02T04:42:43.164877+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:46.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:46 smithi069 bash[21528]: cluster 2024-04-02T04:42:45.165482+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:46.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:46 smithi069 bash[28904]: cluster 2024-04-02T04:42:45.165482+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:46 smithi098 bash[23790]: cluster 2024-04-02T04:42:45.165482+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[21528]: audit 2024-04-02T04:42:47.059110+0000 mon.b (mon.1) 22 : audit 1 from='client.? 172.21.15.98:0/3633423603' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]: dispatch 2024-04-02T04:42:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[21528]: audit 2024-04-02T04:42:47.063792+0000 mon.a (mon.0) 636 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]: dispatch 2024-04-02T04:42:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[21528]: audit 2024-04-02T04:42:47.067281+0000 mon.a (mon.0) 637 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]': finished 2024-04-02T04:42:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[21528]: cluster 2024-04-02T04:42:47.074722+0000 mon.a (mon.0) 638 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-04-02T04:42:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[21528]: audit 2024-04-02T04:42:47.075581+0000 mon.a (mon.0) 639 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:42:47.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[28904]: audit 2024-04-02T04:42:47.059110+0000 mon.b (mon.1) 22 : audit 1 from='client.? 172.21.15.98:0/3633423603' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]: dispatch 2024-04-02T04:42:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[28904]: audit 2024-04-02T04:42:47.063792+0000 mon.a (mon.0) 636 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]: dispatch 2024-04-02T04:42:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[28904]: audit 2024-04-02T04:42:47.067281+0000 mon.a (mon.0) 637 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]': finished 2024-04-02T04:42:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[28904]: cluster 2024-04-02T04:42:47.074722+0000 mon.a (mon.0) 638 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-04-02T04:42:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:47 smithi069 bash[28904]: audit 2024-04-02T04:42:47.075581+0000 mon.a (mon.0) 639 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:42:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:47 smithi098 bash[23790]: audit 2024-04-02T04:42:47.059110+0000 mon.b (mon.1) 22 : audit 1 from='client.? 172.21.15.98:0/3633423603' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]: dispatch 2024-04-02T04:42:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:47 smithi098 bash[23790]: audit 2024-04-02T04:42:47.063792+0000 mon.a (mon.0) 636 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]: dispatch 2024-04-02T04:42:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:47 smithi098 bash[23790]: audit 2024-04-02T04:42:47.067281+0000 mon.a (mon.0) 637 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3"}]': finished 2024-04-02T04:42:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:47 smithi098 bash[23790]: cluster 2024-04-02T04:42:47.074722+0000 mon.a (mon.0) 638 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-04-02T04:42:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:47 smithi098 bash[23790]: audit 2024-04-02T04:42:47.075581+0000 mon.a (mon.0) 639 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:42:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:48.356+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:42:48.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:48 smithi069 bash[21528]: cluster 2024-04-02T04:42:47.166006+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:48.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:48 smithi069 bash[21528]: audit 2024-04-02T04:42:47.822223+0000 mon.b (mon.1) 23 : audit 0 from='client.? 172.21.15.98:0/663018673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:42:48.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:48 smithi069 bash[28904]: cluster 2024-04-02T04:42:47.166006+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:48.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:48 smithi069 bash[28904]: audit 2024-04-02T04:42:47.822223+0000 mon.b (mon.1) 23 : audit 0 from='client.? 172.21.15.98:0/663018673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:42:48.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:48 smithi098 bash[23790]: cluster 2024-04-02T04:42:47.166006+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:48.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:48 smithi098 bash[23790]: audit 2024-04-02T04:42:47.822223+0000 mon.b (mon.1) 23 : audit 0 from='client.? 172.21.15.98:0/663018673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:42:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:50 smithi069 bash[21528]: cluster 2024-04-02T04:42:49.166571+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:50 smithi069 bash[28904]: cluster 2024-04-02T04:42:49.166571+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:50 smithi098 bash[23790]: cluster 2024-04-02T04:42:49.166571+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:53.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:52 smithi098 bash[23790]: cluster 2024-04-02T04:42:51.167098+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:52 smithi069 bash[21528]: cluster 2024-04-02T04:42:51.167098+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:53.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:52 smithi069 bash[28904]: cluster 2024-04-02T04:42:51.167098+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:53.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:53.360+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:42:55.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:54 smithi098 bash[23790]: cluster 2024-04-02T04:42:53.167721+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:54 smithi098 bash[23790]: cluster 2024-04-02T04:42:53.917465+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:55.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:54 smithi098 bash[23790]: cluster 2024-04-02T04:42:53.917578+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:55.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:54 smithi069 bash[21528]: cluster 2024-04-02T04:42:53.167721+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:55.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:54 smithi069 bash[21528]: cluster 2024-04-02T04:42:53.917465+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:55.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:54 smithi069 bash[21528]: cluster 2024-04-02T04:42:53.917578+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:55.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:54 smithi069 bash[28904]: cluster 2024-04-02T04:42:53.167721+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:55.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:54 smithi069 bash[28904]: cluster 2024-04-02T04:42:53.917465+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:42:55.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:54 smithi069 bash[28904]: cluster 2024-04-02T04:42:53.917578+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:42:56.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:55 smithi098 bash[23790]: cluster 2024-04-02T04:42:54.803672+0000 mon.a (mon.0) 640 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-04-02T04:42:56.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:55 smithi098 bash[23790]: audit 2024-04-02T04:42:54.804018+0000 mon.a (mon.0) 641 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:42:56.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:55 smithi069 bash[21528]: cluster 2024-04-02T04:42:54.803672+0000 mon.a (mon.0) 640 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-04-02T04:42:56.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:55 smithi069 bash[21528]: audit 2024-04-02T04:42:54.804018+0000 mon.a (mon.0) 641 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:42:56.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:55 smithi069 bash[28904]: cluster 2024-04-02T04:42:54.803672+0000 mon.a (mon.0) 640 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-04-02T04:42:56.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:55 smithi069 bash[28904]: audit 2024-04-02T04:42:54.804018+0000 mon.a (mon.0) 641 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:42:57.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:56 smithi098 bash[23790]: cluster 2024-04-02T04:42:55.168383+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:57.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:56 smithi069 bash[21528]: cluster 2024-04-02T04:42:55.168383+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:57.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:56 smithi069 bash[28904]: cluster 2024-04-02T04:42:55.168383+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:42:58.364+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:42:59.219 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:58 smithi098 bash[23790]: cluster 2024-04-02T04:42:57.168991+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:59.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:58 smithi069 bash[21528]: cluster 2024-04-02T04:42:57.168991+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:42:59.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:58 smithi069 bash[28904]: cluster 2024-04-02T04:42:57.168991+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:00.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:59 smithi098 bash[23790]: audit 2024-04-02T04:42:59.795041+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:43:00.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:42:59 smithi098 bash[23790]: audit 2024-04-02T04:42:59.796935+0000 mon.a (mon.0) 643 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:59 smithi069 bash[21528]: audit 2024-04-02T04:42:59.795041+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:43:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:42:59 smithi069 bash[21528]: audit 2024-04-02T04:42:59.796935+0000 mon.a (mon.0) 643 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:00.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:59 smithi069 bash[28904]: audit 2024-04-02T04:42:59.795041+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:43:00.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:42:59 smithi069 bash[28904]: audit 2024-04-02T04:42:59.796935+0000 mon.a (mon.0) 643 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:01.097 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:00 smithi098 bash[23790]: cluster 2024-04-02T04:42:59.169590+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:01.097 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:00 smithi098 bash[23790]: cephadm 2024-04-02T04:42:59.798819+0000 mgr.y (mgr.14152) 307 : cephadm 1 Deploying daemon osd.7 on smithi098 2024-04-02T04:43:01.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:00 smithi069 bash[21528]: cluster 2024-04-02T04:42:59.169590+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:01.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:00 smithi069 bash[21528]: cephadm 2024-04-02T04:42:59.798819+0000 mgr.y (mgr.14152) 307 : cephadm 1 Deploying daemon osd.7 on smithi098 2024-04-02T04:43:01.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:00 smithi069 bash[28904]: cluster 2024-04-02T04:42:59.169590+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:01.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:00 smithi069 bash[28904]: cephadm 2024-04-02T04:42:59.798819+0000 mgr.y (mgr.14152) 307 : cephadm 1 Deploying daemon osd.7 on smithi098 2024-04-02T04:43:02.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:02 smithi098 bash[23790]: cluster 2024-04-02T04:43:01.170279+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:03.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:02 smithi069 bash[21528]: cluster 2024-04-02T04:43:01.170279+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:03.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:02 smithi069 bash[28904]: cluster 2024-04-02T04:43:01.170279+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:03.368+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:43:04.135 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.136 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.136 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:03 smithi098 bash[23790]: cluster 2024-04-02T04:43:03.020441+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:04.136 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:03 smithi098 bash[23790]: cluster 2024-04-02T04:43:03.020491+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:04.136 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.136 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.137 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:03 smithi069 bash[21528]: cluster 2024-04-02T04:43:03.020441+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:04.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:03 smithi069 bash[21528]: cluster 2024-04-02T04:43:03.020491+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:04.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:03 smithi069 bash[28904]: cluster 2024-04-02T04:43:03.020441+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:04.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:03 smithi069 bash[28904]: cluster 2024-04-02T04:43:03.020491+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:04.389 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.389 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.389 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.390 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:04.390 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:43:04 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 bash[23790]: cluster 2024-04-02T04:43:03.170977+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 bash[23790]: cluster 2024-04-02T04:43:03.842499+0000 mon.a (mon.0) 644 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-04-02T04:43:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 bash[23790]: audit 2024-04-02T04:43:03.842810+0000 mon.a (mon.0) 645 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 bash[23790]: audit 2024-04-02T04:43:04.445839+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 bash[23790]: audit 2024-04-02T04:43:04.446803+0000 mon.a (mon.0) 647 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:04 smithi098 bash[23790]: audit 2024-04-02T04:43:04.457672+0000 mon.a (mon.0) 648 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:05.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[21528]: cluster 2024-04-02T04:43:03.170977+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[21528]: cluster 2024-04-02T04:43:03.842499+0000 mon.a (mon.0) 644 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[21528]: audit 2024-04-02T04:43:03.842810+0000 mon.a (mon.0) 645 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[21528]: audit 2024-04-02T04:43:04.445839+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[21528]: audit 2024-04-02T04:43:04.446803+0000 mon.a (mon.0) 647 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[21528]: audit 2024-04-02T04:43:04.457672+0000 mon.a (mon.0) 648 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[28904]: cluster 2024-04-02T04:43:03.170977+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[28904]: cluster 2024-04-02T04:43:03.842499+0000 mon.a (mon.0) 644 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[28904]: audit 2024-04-02T04:43:03.842810+0000 mon.a (mon.0) 645 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[28904]: audit 2024-04-02T04:43:04.445839+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[28904]: audit 2024-04-02T04:43:04.446803+0000 mon.a (mon.0) 647 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:05.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:04 smithi069 bash[28904]: audit 2024-04-02T04:43:04.457672+0000 mon.a (mon.0) 648 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:06 smithi098 bash[23790]: cluster 2024-04-02T04:43:05.171675+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:06 smithi069 bash[21528]: cluster 2024-04-02T04:43:05.171675+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:07.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:06 smithi069 bash[28904]: cluster 2024-04-02T04:43:05.171675+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:08.372+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:43:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:08 smithi098 bash[23790]: cluster 2024-04-02T04:43:07.172282+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:09.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:08 smithi069 bash[21528]: cluster 2024-04-02T04:43:07.172282+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:08 smithi069 bash[28904]: cluster 2024-04-02T04:43:07.172282+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:11.120 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:10 smithi098 bash[23790]: cluster 2024-04-02T04:43:09.172855+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:11.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:10 smithi069 bash[21528]: cluster 2024-04-02T04:43:09.172855+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:10 smithi069 bash[28904]: cluster 2024-04-02T04:43:09.172855+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:12.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:10.842797+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:43:12.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:10.847642+0000 mon.a (mon.0) 649 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:10.900594+0000 mon.a (mon.0) 650 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:10.902304+0000 mon.b (mon.1) 25 : audit 1 from='osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: cluster 2024-04-02T04:43:10.905887+0000 mon.a (mon.0) 651 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:10.906542+0000 mon.a (mon.0) 652 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:10.907711+0000 mon.a (mon.0) 653 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:11.452181+0000 mon.a (mon.0) 654 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:11.464929+0000 mon.a (mon.0) 655 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:11.467195+0000 mon.a (mon.0) 656 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:11.469086+0000 mon.a (mon.0) 657 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:12.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:11 smithi098 bash[23790]: audit 2024-04-02T04:43:11.480957+0000 mon.a (mon.0) 658 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:10.842797+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:10.847642+0000 mon.a (mon.0) 649 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:10.900594+0000 mon.a (mon.0) 650 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:10.902304+0000 mon.b (mon.1) 25 : audit 1 from='osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: cluster 2024-04-02T04:43:10.905887+0000 mon.a (mon.0) 651 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:10.906542+0000 mon.a (mon.0) 652 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:10.907711+0000 mon.a (mon.0) 653 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:11.452181+0000 mon.a (mon.0) 654 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:11.464929+0000 mon.a (mon.0) 655 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:11.467195+0000 mon.a (mon.0) 656 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:11.469086+0000 mon.a (mon.0) 657 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:12.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[21528]: audit 2024-04-02T04:43:11.480957+0000 mon.a (mon.0) 658 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:10.842797+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:10.847642+0000 mon.a (mon.0) 649 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:10.900594+0000 mon.a (mon.0) 650 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:10.902304+0000 mon.b (mon.1) 25 : audit 1 from='osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: cluster 2024-04-02T04:43:10.905887+0000 mon.a (mon.0) 651 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:10.906542+0000 mon.a (mon.0) 652 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:10.907711+0000 mon.a (mon.0) 653 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:11.452181+0000 mon.a (mon.0) 654 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:11.464929+0000 mon.a (mon.0) 655 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:11.467195+0000 mon.a (mon.0) 656 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:11.469086+0000 mon.a (mon.0) 657 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:12.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:11 smithi069 bash[28904]: audit 2024-04-02T04:43:11.480957+0000 mon.a (mon.0) 658 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:12 smithi098 bash[23790]: cluster 2024-04-02T04:43:11.173419+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:12 smithi098 bash[23790]: audit 2024-04-02T04:43:11.908616+0000 mon.a (mon.0) 659 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:43:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:12 smithi098 bash[23790]: cluster 2024-04-02T04:43:11.913869+0000 mon.a (mon.0) 660 : cluster 0 osdmap e68: 8 total, 7 up, 8 in 2024-04-02T04:43:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:12 smithi098 bash[23790]: audit 2024-04-02T04:43:11.914384+0000 mon.a (mon.0) 661 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:12 smithi098 bash[23790]: audit 2024-04-02T04:43:11.923445+0000 mon.a (mon.0) 662 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[21528]: cluster 2024-04-02T04:43:11.173419+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[21528]: audit 2024-04-02T04:43:11.908616+0000 mon.a (mon.0) 659 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[21528]: cluster 2024-04-02T04:43:11.913869+0000 mon.a (mon.0) 660 : cluster 0 osdmap e68: 8 total, 7 up, 8 in 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[21528]: audit 2024-04-02T04:43:11.914384+0000 mon.a (mon.0) 661 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[21528]: audit 2024-04-02T04:43:11.923445+0000 mon.a (mon.0) 662 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[28904]: cluster 2024-04-02T04:43:11.173419+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[28904]: audit 2024-04-02T04:43:11.908616+0000 mon.a (mon.0) 659 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[28904]: cluster 2024-04-02T04:43:11.913869+0000 mon.a (mon.0) 660 : cluster 0 osdmap e68: 8 total, 7 up, 8 in 2024-04-02T04:43:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[28904]: audit 2024-04-02T04:43:11.914384+0000 mon.a (mon.0) 661 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:12 smithi069 bash[28904]: audit 2024-04-02T04:43:11.923445+0000 mon.a (mon.0) 662 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:13.376+0000 7f3bad255700 10 client.24262.objecter tick 2024-04-02T04:43:13.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[21528]: cluster 2024-04-02T04:43:11.866343+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:13.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[21528]: cluster 2024-04-02T04:43:11.866445+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:13.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[21528]: cluster 2024-04-02T04:43:12.919898+0000 mon.a (mon.0) 663 : cluster 1 osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] boot 2024-04-02T04:43:13.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[21528]: cluster 2024-04-02T04:43:12.920008+0000 mon.a (mon.0) 664 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-04-02T04:43:13.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[21528]: audit 2024-04-02T04:43:12.920953+0000 mon.a (mon.0) 665 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:13.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[21528]: cluster 2024-04-02T04:43:13.790270+0000 mon.a (mon.0) 666 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-04-02T04:43:14.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:13 smithi098 bash[23790]: cluster 2024-04-02T04:43:11.866343+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:14.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:13 smithi098 bash[23790]: cluster 2024-04-02T04:43:11.866445+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:13 smithi098 bash[23790]: cluster 2024-04-02T04:43:12.919898+0000 mon.a (mon.0) 663 : cluster 1 osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] boot 2024-04-02T04:43:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:13 smithi098 bash[23790]: cluster 2024-04-02T04:43:12.920008+0000 mon.a (mon.0) 664 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-04-02T04:43:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:13 smithi098 bash[23790]: audit 2024-04-02T04:43:12.920953+0000 mon.a (mon.0) 665 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:14.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:13 smithi098 bash[23790]: cluster 2024-04-02T04:43:13.790270+0000 mon.a (mon.0) 666 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-04-02T04:43:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[28904]: cluster 2024-04-02T04:43:11.866343+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[28904]: cluster 2024-04-02T04:43:11.866445+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:14.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[28904]: cluster 2024-04-02T04:43:12.919898+0000 mon.a (mon.0) 663 : cluster 1 osd.7 [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] boot 2024-04-02T04:43:14.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[28904]: cluster 2024-04-02T04:43:12.920008+0000 mon.a (mon.0) 664 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-04-02T04:43:14.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[28904]: audit 2024-04-02T04:43:12.920953+0000 mon.a (mon.0) 665 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:43:14.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:13 smithi069 bash[28904]: cluster 2024-04-02T04:43:13.790270+0000 mon.a (mon.0) 666 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-04-02T04:43:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.156+0000 7f3baca54700 1 -- 172.21.15.98:0/2597364642 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ba802df40 con 0x7f3b8806f4f0 2024-04-02T04:43:15.164 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 7 on host 'smithi098' 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 10 librados: watch_flush enter 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 10 librados: watch_flush exit 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 20 client.24262.objecter shutdown clearing up homeless session... 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 10 client.24262.objecter successfully canceled tick 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3b8806f4f0 msgr2=0x7f3b880719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3b8806f4f0 0x7f3b880719b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8103330 tx=0x7f3b90009040 comp rx=0 tx=0).stop 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 msgr2=0x7f3ba8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8198030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b98005ef0 comp rx=0 tx=0).stop 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 shutdown_connections 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f3b8806f4f0 0x7f3b880719b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:15.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ba8103e70 0x7f3ba819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:15.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81022d0 0x7f3ba8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:15.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 --2- 172.21.15.98:0/2597364642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba81034d0 0x7f3ba8198030 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:15.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 >> 172.21.15.98:0/2597364642 conn(0x7f3ba80fd870 msgr2=0x7f3ba8106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:15.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 shutdown_connections 2024-04-02T04:43:15.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 -- 172.21.15.98:0/2597364642 wait complete. 2024-04-02T04:43:15.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:15.160+0000 7f3bada56700 1 librados: shutdown 2024-04-02T04:43:15.183 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:14 smithi098 bash[23790]: cluster 2024-04-02T04:43:13.173968+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:15.183 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:14 smithi098 bash[23790]: cluster 2024-04-02T04:43:14.791019+0000 mon.a (mon.0) 667 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:43:15.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:14 smithi069 bash[21528]: cluster 2024-04-02T04:43:13.173968+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:15.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:14 smithi069 bash[21528]: cluster 2024-04-02T04:43:14.791019+0000 mon.a (mon.0) 667 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:43:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:14 smithi069 bash[28904]: cluster 2024-04-02T04:43:13.173968+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:14 smithi069 bash[28904]: cluster 2024-04-02T04:43:14.791019+0000 mon.a (mon.0) 667 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:43:15.840 DEBUG:teuthology.orchestra.run.smithi098:osd.7> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7.service 2024-04-02T04:43:15.843 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-02T04:43:15.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-04-02T04:43:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:16 smithi098 bash[23790]: audit 2024-04-02T04:43:15.124032+0000 mon.a (mon.0) 668 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:16 smithi098 bash[23790]: audit 2024-04-02T04:43:15.128914+0000 mon.a (mon.0) 669 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:16 smithi098 bash[23790]: audit 2024-04-02T04:43:15.136859+0000 mon.a (mon.0) 670 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:16.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:16 smithi098 bash[23790]: cluster 2024-04-02T04:43:15.174469+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:16.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:16 smithi098 bash[23790]: cluster 2024-04-02T04:43:15.941101+0000 mon.a (mon.0) 671 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:43:16.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[21528]: audit 2024-04-02T04:43:15.124032+0000 mon.a (mon.0) 668 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:16.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[21528]: audit 2024-04-02T04:43:15.128914+0000 mon.a (mon.0) 669 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:16.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[21528]: audit 2024-04-02T04:43:15.136859+0000 mon.a (mon.0) 670 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[21528]: cluster 2024-04-02T04:43:15.174469+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[21528]: cluster 2024-04-02T04:43:15.941101+0000 mon.a (mon.0) 671 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[28904]: audit 2024-04-02T04:43:15.124032+0000 mon.a (mon.0) 668 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[28904]: audit 2024-04-02T04:43:15.128914+0000 mon.a (mon.0) 669 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[28904]: audit 2024-04-02T04:43:15.136859+0000 mon.a (mon.0) 670 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[28904]: cluster 2024-04-02T04:43:15.174469+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:16.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:16 smithi069 bash[28904]: cluster 2024-04-02T04:43:15.941101+0000 mon.a (mon.0) 671 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:43:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:17 smithi098 bash[23790]: cluster 2024-04-02T04:43:16.943761+0000 mon.a (mon.0) 672 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-04-02T04:43:18.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:17 smithi069 bash[21528]: cluster 2024-04-02T04:43:16.943761+0000 mon.a (mon.0) 672 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-04-02T04:43:18.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:17 smithi069 bash[28904]: cluster 2024-04-02T04:43:16.943761+0000 mon.a (mon.0) 672 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-04-02T04:43:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:18 smithi098 bash[23790]: cluster 2024-04-02T04:43:17.175008+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v318: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T04:43:19.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:18 smithi069 bash[21528]: cluster 2024-04-02T04:43:17.175008+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v318: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T04:43:19.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:18 smithi069 bash[28904]: cluster 2024-04-02T04:43:17.175008+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v318: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T04:43:20.600 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:43:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:20 smithi098 bash[23790]: cluster 2024-04-02T04:43:19.175640+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-02T04:43:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:20 smithi069 bash[21528]: cluster 2024-04-02T04:43:19.175640+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-02T04:43:21.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:20 smithi069 bash[28904]: cluster 2024-04-02T04:43:19.175640+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-02T04:43:23.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- 172.21.15.69:0/2233760356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c1011d0 msgr2=0x7efe0c1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- 172.21.15.69:0/2233760356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c1011d0 0x7efe0c1015f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009fc0 tx=0x7efdf400b330 comp rx=0 tx=0).stop 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- 172.21.15.69:0/2233760356 shutdown_connections 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- 172.21.15.69:0/2233760356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c102d70 0x7efe0c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- 172.21.15.69:0/2233760356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c1011d0 0x7efe0c1015f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- 172.21.15.69:0/2233760356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1023d0 0x7efe0c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- 172.21.15.69:0/2233760356 >> 172.21.15.69:0/2233760356 conn(0x7efe0c0fc790 msgr2=0x7efe0c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- 172.21.15.69:0/2233760356 shutdown_connections 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- 172.21.15.69:0/2233760356 wait complete. 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 librados: starting msgr at 2024-04-02T04:43:23.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 librados: starting objecter 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 Processor -- start 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- start start 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 librados: setting wanted keys 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 librados: calling monclient init 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 0x7efe0c105160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c1023d0 0x7efe0c1037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102d70 0x7efe0c103cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c10c2f0 con 0x7efe0c102d70 2024-04-02T04:43:23.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c10c170 con 0x7efe0c1023d0 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe13242700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efe0c10c470 con 0x7efe0c1011d0 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 0x7efe0c105160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 0x7efe0c105160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60292/0 (socket says 172.21.15.69:60292) 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 -- 172.21.15.69:0/456629906 learned_addr learned my addr 172.21.15.69:0/456629906 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe117df700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102d70 0x7efe0c103cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe0bfff700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c1023d0 0x7efe0c1037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 -- 172.21.15.69:0/456629906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c1023d0 msgr2=0x7efe0c1037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c1023d0 0x7efe0c1037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 -- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102d70 msgr2=0x7efe0c103cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102d70 0x7efe0c103cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 -- 172.21.15.69:0/456629906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c1045a0 con 0x7efe0c1011d0 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe117df700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102d70 0x7efe0c103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:43:23.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe10fde700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 0x7efe0c105160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000f80 tx=0x7efdf400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe09ffb700 10 client.?.objecter ms_handle_connect 0x7efe0c1011d0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.057+0000 7efe09ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf4021070 con 0x7efe0c1011d0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7efdf4007ac0 con 0x7efe0c1011d0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe13242700 1 -- 172.21.15.69:0/456629906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c1a1c10 con 0x7efe0c1011d0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf40052a0 con 0x7efe0c1011d0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe13242700 10 client.24290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe13242700 1 -- 172.21.15.69:0/456629906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0c1a2070 con 0x7efe0c1011d0 2024-04-02T04:43:23.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe13242700 1 librados: init done 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf0005120 con 0x7efe0c1011d0 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7efdf4024070 con 0x7efe0c1011d0 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efdf806f4f0 0x7efdf80719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(73..73 src has 23..73) ==== 5287+0+0 (secure 0 0 0) 0x7efdf400f020 con 0x7efe0c1011d0 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 10 client.24290.objecter ms_dispatch 0x7efe0c001550 osd_map(73..73 src has 23..73) 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 3 client.24290.objecter handle_osd_map got epochs [73,73] > 0 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 3 client.24290.objecter handle_osd_map decoding full epoch 73 2024-04-02T04:43:23.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe0bfff700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efdf806f4f0 0x7efdf80719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:23.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 20 client.24290.objecter dump_active .. 0 homeless 2024-04-02T04:43:23.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe0bfff700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efdf806f4f0 0x7efdf80719b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009750 tx=0x7efdfc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:23.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.061+0000 7efe09ffb700 10 client.24290.objecter ms_handle_connect 0x7efdf806f4f0 2024-04-02T04:43:23.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.065+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efdf402b020 con 0x7efe0c1011d0 2024-04-02T04:43:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:22 smithi098 bash[23790]: cluster 2024-04-02T04:43:21.176256+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-02T04:43:23.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:22 smithi069 bash[21528]: cluster 2024-04-02T04:43:21.176256+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-02T04:43:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:22 smithi069 bash[28904]: cluster 2024-04-02T04:43:21.176256+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-02T04:43:23.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.277+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7efdf00056e0 con 0x7efe0c1011d0 2024-04-02T04:43:23.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.277+0000 7efe09ffb700 1 -- 172.21.15.69:0/456629906 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+130 (secure 0 0 0) 0x7efdf402e350 con 0x7efe0c1011d0 2024-04-02T04:43:23.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:43:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 10 librados: watch_flush enter 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 10 librados: watch_flush exit 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 20 client.24290.objecter shutdown clearing up homeless session... 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 10 client.24290.objecter successfully canceled tick 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efdf806f4f0 msgr2=0x7efdf80719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efdf806f4f0 0x7efdf80719b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009750 tx=0x7efdfc009310 comp rx=0 tx=0).stop 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 msgr2=0x7efe0c105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 0x7efe0c105160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000f80 tx=0x7efdf400ccb0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 shutdown_connections 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102d70 0x7efe0c103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7efdf806f4f0 0x7efdf80719b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c1011d0 0x7efe0c105160 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 --2- 172.21.15.69:0/456629906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe0c1023d0 0x7efe0c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 >> 172.21.15.69:0/456629906 conn(0x7efe0c0fc790 msgr2=0x7efe0c06a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 shutdown_connections 2024-04-02T04:43:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 -- 172.21.15.69:0/456629906 wait complete. 2024-04-02T04:43:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:23.281+0000 7efe037fe700 1 librados: shutdown 2024-04-02T04:43:23.909 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":73,"num_osds":8,"num_up_osds":8,"osd_up_since":1712032992,"num_in_osds":8,"osd_in_since":1712032967,"num_remapped_pgs":0} 2024-04-02T04:43:23.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T04:43:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:23 smithi098 bash[23790]: audit 2024-04-02T04:43:23.282839+0000 mon.c (mon.2) 19 : audit 0 from='client.? 172.21.15.69:0/456629906' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T04:43:24.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:23 smithi069 bash[21528]: audit 2024-04-02T04:43:23.282839+0000 mon.c (mon.2) 19 : audit 0 from='client.? 172.21.15.69:0/456629906' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T04:43:24.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:23 smithi069 bash[28904]: audit 2024-04-02T04:43:23.282839+0000 mon.c (mon.2) 19 : audit 0 from='client.? 172.21.15.69:0/456629906' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: cluster 2024-04-02T04:43:23.176868+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.479877+0000 mon.a (mon.0) 673 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.490023+0000 mon.a (mon.0) 674 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.492675+0000 mon.a (mon.0) 675 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.494454+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.495989+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.497417+0000 mon.a (mon.0) 678 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.505176+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.509422+0000 mon.a (mon.0) 680 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.510992+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:24 smithi098 bash[23790]: audit 2024-04-02T04:43:24.520776+0000 mon.a (mon.0) 682 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: cluster 2024-04-02T04:43:23.176868+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T04:43:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.479877+0000 mon.a (mon.0) 673 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.490023+0000 mon.a (mon.0) 674 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.492675+0000 mon.a (mon.0) 675 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.494454+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.495989+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.497417+0000 mon.a (mon.0) 678 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.505176+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.509422+0000 mon.a (mon.0) 680 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.510992+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[21528]: audit 2024-04-02T04:43:24.520776+0000 mon.a (mon.0) 682 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: cluster 2024-04-02T04:43:23.176868+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.479877+0000 mon.a (mon.0) 673 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.490023+0000 mon.a (mon.0) 674 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.492675+0000 mon.a (mon.0) 675 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.494454+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.495989+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.497417+0000 mon.a (mon.0) 678 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.505176+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.509422+0000 mon.a (mon.0) 680 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.510992+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:24 smithi069 bash[28904]: audit 2024-04-02T04:43:24.520776+0000 mon.a (mon.0) 682 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:25 smithi098 bash[23790]: cephadm 2024-04-02T04:43:24.470480+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:43:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:25 smithi098 bash[23790]: cephadm 2024-04-02T04:43:24.498633+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi098 to 4316M 2024-04-02T04:43:26.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:25 smithi069 bash[21528]: cephadm 2024-04-02T04:43:24.470480+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:43:26.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:25 smithi069 bash[21528]: cephadm 2024-04-02T04:43:24.498633+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi098 to 4316M 2024-04-02T04:43:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:25 smithi069 bash[28904]: cephadm 2024-04-02T04:43:24.470480+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi098 2024-04-02T04:43:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:25 smithi069 bash[28904]: cephadm 2024-04-02T04:43:24.498633+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi098 to 4316M 2024-04-02T04:43:27.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:26 smithi069 bash[21528]: cluster 2024-04-02T04:43:25.177499+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:26 smithi069 bash[28904]: cluster 2024-04-02T04:43:25.177499+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:26 smithi098 bash[23790]: cluster 2024-04-02T04:43:25.177499+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:28.674 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:43:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:28 smithi069 bash[21528]: cluster 2024-04-02T04:43:27.178180+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:28 smithi069 bash[21528]: cluster 2024-04-02T04:43:28.269321+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:28 smithi069 bash[21528]: cluster 2024-04-02T04:43:28.269457+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:29.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:28 smithi069 bash[28904]: cluster 2024-04-02T04:43:27.178180+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:28 smithi069 bash[28904]: cluster 2024-04-02T04:43:28.269321+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:28 smithi069 bash[28904]: cluster 2024-04-02T04:43:28.269457+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:28 smithi098 bash[23790]: cluster 2024-04-02T04:43:27.178180+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:28 smithi098 bash[23790]: cluster 2024-04-02T04:43:28.269321+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:28 smithi098 bash[23790]: cluster 2024-04-02T04:43:28.269457+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:30.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:29 smithi069 bash[21528]: cluster 2024-04-02T04:43:28.430699+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:30.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:29 smithi069 bash[21528]: cluster 2024-04-02T04:43:28.430815+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:30.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:29 smithi069 bash[21528]: cluster 2024-04-02T04:43:28.996088+0000 mon.a (mon.0) 683 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-04-02T04:43:30.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:30 smithi069 bash[28904]: cluster 2024-04-02T04:43:28.430699+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:30.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:30 smithi069 bash[28904]: cluster 2024-04-02T04:43:28.430815+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:30.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:30 smithi069 bash[28904]: cluster 2024-04-02T04:43:28.996088+0000 mon.a (mon.0) 683 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-04-02T04:43:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:29 smithi098 bash[23790]: cluster 2024-04-02T04:43:28.430699+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:29 smithi098 bash[23790]: cluster 2024-04-02T04:43:28.430815+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:29 smithi098 bash[23790]: cluster 2024-04-02T04:43:28.996088+0000 mon.a (mon.0) 683 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- 172.21.15.69:0/3473621954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006b280 msgr2=0x7f489010f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- 172.21.15.69:0/3473621954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006b280 0x7f489010f030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f488800b0d0 tx=0x7f488800b3e0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- 172.21.15.69:0/3473621954 shutdown_connections 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- 172.21.15.69:0/3473621954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006b280 0x7f489010f030 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- 172.21.15.69:0/3473621954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48901073b0 0x7f4890107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- 172.21.15.69:0/3473621954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f489006a8c0 0x7f489006ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- 172.21.15.69:0/3473621954 >> 172.21.15.69:0/3473621954 conn(0x7f48900780f0 msgr2=0x7f4890078500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:31.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- 172.21.15.69:0/3473621954 shutdown_connections 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- 172.21.15.69:0/3473621954 wait complete. 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 librados: starting msgr at 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 librados: starting objecter 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 Processor -- start 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- start start 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 librados: setting wanted keys 2024-04-02T04:43:31.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 librados: calling monclient init 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006a8c0 0x7f4890197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 0x7f4890198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48901073b0 0x7f489019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4890111a70 con 0x7f489006b280 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48901118f0 con 0x7f48901073b0 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f489486f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4890111bf0 con 0x7f489006a8c0 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 0x7f4890198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488e59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006a8c0 0x7f4890197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:31.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 0x7f4890198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54936/0 (socket says 172.21.15.69:54936) 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488dd9b700 1 -- 172.21.15.69:0/862925622 learned_addr learned my addr 172.21.15.69:0/862925622 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488ed9d700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48901073b0 0x7f489019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488e59c700 1 -- 172.21.15.69:0/862925622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48901073b0 msgr2=0x7f489019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488e59c700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48901073b0 0x7f489019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488e59c700 1 -- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 msgr2=0x7f4890198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488e59c700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 0x7f4890198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488e59c700 1 -- 172.21.15.69:0/862925622 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489019cf30 con 0x7f489006a8c0 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.185+0000 7f488dd9b700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 0x7f4890198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f488e59c700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006a8c0 0x7f4890197f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f488000e9f0 tx=0x7f488000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 10 client.?.objecter ms_handle_connect 0x7f489006a8c0 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488000cb70 con 0x7f489006a8c0 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489019d220 con 0x7f489006a8c0 2024-04-02T04:43:31.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f48800044a0 con 0x7f489006a8c0 2024-04-02T04:43:31.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880003cb0 con 0x7f489006a8c0 2024-04-02T04:43:31.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f489486f700 10 client.24296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:43:31.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48901a4aa0 con 0x7f489006a8c0 2024-04-02T04:43:31.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f489486f700 1 librados: init done 2024-04-02T04:43:31.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48901039e0 con 0x7f489006a8c0 2024-04-02T04:43:31.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f4880003e90 con 0x7f489006a8c0 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f487806f4f0 0x7f48780719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(75..75 src has 23..75) ==== 5287+0+0 (secure 0 0 0) 0x7f488008d7b0 con 0x7f489006a8c0 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 10 client.24296.objecter ms_dispatch 0x7f4890001550 osd_map(75..75 src has 23..75) 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 3 client.24296.objecter handle_osd_map got epochs [75,75] > 0 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 3 client.24296.objecter handle_osd_map decoding full epoch 75 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f488dd9b700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f487806f4f0 0x7f48780719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 20 client.24296.objecter dump_active .. 0 homeless 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f487f7fe700 10 client.24296.objecter ms_handle_connect 0x7f487806f4f0 2024-04-02T04:43:31.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.189+0000 7f488dd9b700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f487806f4f0 0x7f48780719b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4890078830 tx=0x7f4884005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:31.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.193+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f488005b6e0 con 0x7f489006a8c0 2024-04-02T04:43:31.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[21528]: cluster 2024-04-02T04:43:29.178778+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:31.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[21528]: cluster 2024-04-02T04:43:29.825386+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:31.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[21528]: cluster 2024-04-02T04:43:29.825476+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:31.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[21528]: cluster 2024-04-02T04:43:30.006599+0000 mon.a (mon.0) 684 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-04-02T04:43:31.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[28904]: cluster 2024-04-02T04:43:29.178778+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:31.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[28904]: cluster 2024-04-02T04:43:29.825386+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:31.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[28904]: cluster 2024-04-02T04:43:29.825476+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:31.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:31 smithi069 bash[28904]: cluster 2024-04-02T04:43:30.006599+0000 mon.a (mon.0) 684 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-04-02T04:43:31.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.405+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48901a4d30 con 0x7f489006a8c0 2024-04-02T04:43:31.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f487f7fe700 1 -- 172.21.15.69:0/862925622 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+13480 (secure 0 0 0) 0x7f488005afe0 con 0x7f489006a8c0 2024-04-02T04:43:31.409 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:43:31.410 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":75,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:33:43.348385+0000","modified":"2024-04-02T04:43:29.990218+0000","last_up_change":"2024-04-02T04:43:12.908579+0000","last_in_change":"2024-04-02T04:42:47.064987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.671714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b13e248-930e-40ca-9bb0-e8c5ce16bd0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6803","nonce":1087500595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6805","nonce":1087500595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6809","nonce":1087500595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6807","nonce":1087500595}]},"public_addr":"172.21.15.69:6803/1087500595","cluster_addr":"172.21.15.69:6805/1087500595","heartbeat_back_addr":"172.21.15.69:6809/1087500595","heartbeat_front_addr":"172.21.15.69:6807/1087500595","state":["exists","up"]},{"osd":1,"uuid":"afb71182-d4f7-416d-ae44-90f28eab4b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6811","nonce":2651666015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6813","nonce":2651666015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6817","nonce":2651666015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6815","nonce":2651666015}]},"public_addr":"172.21.15.69:6811/2651666015","cluster_addr":"172.21.15.69:6813/2651666015","heartbeat_back_addr":"172.21.15.69:6817/2651666015","heartbeat_front_addr":"172.21.15.69:6815/2651666015","state":["exists","up"]},{"osd":2,"uuid":"523bcbd7-d389-4b8f-aca8-88b71ce61de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6819","nonce":619998764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6821","nonce":619998764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6825","nonce":619998764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6823","nonce":619998764}]},"public_addr":"172.21.15.69:6819/619998764","cluster_addr":"172.21.15.69:6821/619998764","heartbeat_back_addr":"172.21.15.69:6825/619998764","heartbeat_front_addr":"172.21.15.69:6823/619998764","state":["exists","up"]},{"osd":3,"uuid":"20e4526d-2707-44d0-b13a-af36019c42ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6827","nonce":207593208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6829","nonce":207593208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6833","nonce":207593208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6831","nonce":207593208}]},"public_addr":"172.21.15.69:6827/207593208","cluster_addr":"172.21.15.69:6829/207593208","heartbeat_back_addr":"172.21.15.69:6833/207593208","heartbeat_front_addr":"172.21.15.69:6831/207593208","state":["exists","up"]},{"osd":4,"uuid":"aa3bc4f3-4476-4ba1-b033-428e5edf5852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6801","nonce":2065947829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6803","nonce":2065947829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6807","nonce":2065947829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6805","nonce":2065947829}]},"public_addr":"172.21.15.98:6801/2065947829","cluster_addr":"172.21.15.98:6803/2065947829","heartbeat_back_addr":"172.21.15.98:6807/2065947829","heartbeat_front_addr":"172.21.15.98:6805/2065947829","state":["exists","up"]},{"osd":5,"uuid":"902326af-4a40-47c8-a3b8-d5d5536f90fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6809","nonce":219578185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6811","nonce":219578185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6815","nonce":219578185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6813","nonce":219578185}]},"public_addr":"172.21.15.98:6809/219578185","cluster_addr":"172.21.15.98:6811/219578185","heartbeat_back_addr":"172.21.15.98:6815/219578185","heartbeat_front_addr":"172.21.15.98:6813/219578185","state":["exists","up"]},{"osd":6,"uuid":"8c73e4b0-cb6b-486a-8b55-a0fe4d97a727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6817","nonce":1226172499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6819","nonce":1226172499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6823","nonce":1226172499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6821","nonce":1226172499}]},"public_addr":"172.21.15.98:6817/1226172499","cluster_addr":"172.21.15.98:6819/1226172499","heartbeat_back_addr":"172.21.15.98:6823/1226172499","heartbeat_front_addr":"172.21.15.98:6821/1226172499","state":["exists","up"]},{"osd":7,"uuid":"4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6825","nonce":148009079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6827","nonce":148009079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6831","nonce":148009079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6829","nonce":148009079}]},"public_addr":"172.21.15.98:6825/148009079","cluster_addr":"172.21.15.98:6827/148009079","heartbeat_back_addr":"172.21.15.98:6831/148009079","heartbeat_front_addr":"172.21.15.98:6829/148009079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:43:28.430822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:42:53.917585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:43:29.825481+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:42:39.689923+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:43:03.020493+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:42:39.969850+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:43:28.269463+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:43:11.866451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/3657409442":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/2763215770":"2024-04-03T04:34:47.037764+0000","172.21.15.69:6801/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:6800/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/2928651991":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/52352690":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/1798147282":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/2738867930":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6800/578732137":"2024-04-03T04:34:47.037764+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 10 librados: watch_flush enter 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 10 librados: watch_flush exit 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 20 client.24296.objecter shutdown clearing up homeless session... 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 10 client.24296.objecter successfully canceled tick 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f487806f4f0 msgr2=0x7f48780719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f487806f4f0 0x7f48780719b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4890078830 tx=0x7f4884005f90 comp rx=0 tx=0).stop 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006a8c0 msgr2=0x7f4890197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006a8c0 0x7f4890197f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f488000e9f0 tx=0x7f488000ed00 comp rx=0 tx=0).stop 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 shutdown_connections 2024-04-02T04:43:31.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48901073b0 0x7f489019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f487806f4f0 0x7f48780719b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489006a8c0 0x7f4890197f20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 --2- 172.21.15.69:0/862925622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489006b280 0x7f4890198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 >> 172.21.15.69:0/862925622 conn(0x7f48900780f0 msgr2=0x7f4890106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 shutdown_connections 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.409+0000 7f489486f700 1 -- 172.21.15.69:0/862925622 wait complete. 2024-04-02T04:43:31.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:31.413+0000 7f489486f700 1 librados: shutdown 2024-04-02T04:43:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:30 smithi098 bash[23790]: cluster 2024-04-02T04:43:29.178778+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:30 smithi098 bash[23790]: cluster 2024-04-02T04:43:29.825386+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:30 smithi098 bash[23790]: cluster 2024-04-02T04:43:29.825476+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:30 smithi098 bash[23790]: cluster 2024-04-02T04:43:30.006599+0000 mon.a (mon.0) 684 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-04-02T04:43:32.092 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T04:43:32.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-02T04:43:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:32 smithi069 bash[21528]: audit 2024-04-02T04:43:31.411155+0000 mon.c (mon.2) 20 : audit 0 from='client.? 172.21.15.69:0/862925622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:43:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:32 smithi069 bash[28904]: audit 2024-04-02T04:43:31.411155+0000 mon.c (mon.2) 20 : audit 0 from='client.? 172.21.15.69:0/862925622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:43:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:32 smithi098 bash[23790]: audit 2024-04-02T04:43:31.411155+0000 mon.c (mon.2) 20 : audit 0 from='client.? 172.21.15.69:0/862925622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:43:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:33 smithi069 bash[21528]: cluster 2024-04-02T04:43:31.179339+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:33.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:33 smithi069 bash[28904]: cluster 2024-04-02T04:43:31.179339+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:33.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:33 smithi098 bash[23790]: cluster 2024-04-02T04:43:31.179339+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:35.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:35 smithi069 bash[21528]: cluster 2024-04-02T04:43:33.179953+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:35 smithi069 bash[28904]: cluster 2024-04-02T04:43:33.179953+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:35 smithi098 bash[23790]: cluster 2024-04-02T04:43:33.179953+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:36.852 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:43:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:37 smithi069 bash[21528]: cluster 2024-04-02T04:43:35.180642+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:37.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:37 smithi069 bash[28904]: cluster 2024-04-02T04:43:35.180642+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:37 smithi098 bash[23790]: cluster 2024-04-02T04:43:35.180642+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:39.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:39 smithi069 bash[21528]: cluster 2024-04-02T04:43:37.181277+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:39.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:39 smithi069 bash[28904]: cluster 2024-04-02T04:43:37.181277+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:39.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- 172.21.15.69:0/4162193293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c1024b0 msgr2=0x7f8f0c102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- 172.21.15.69:0/4162193293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c1024b0 0x7f8f0c102890 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0099c0 tx=0x7f8efc009cd0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- 172.21.15.69:0/4162193293 shutdown_connections 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- 172.21.15.69:0/4162193293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c107590 0x7f8f0c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- 172.21.15.69:0/4162193293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c1024b0 0x7f8f0c102890 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- 172.21.15.69:0/4162193293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f0c102e60 0x7f8f0c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- 172.21.15.69:0/4162193293 >> 172.21.15.69:0/4162193293 conn(0x7f8f0c079610 msgr2=0x7f8f0c079a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- 172.21.15.69:0/4162193293 shutdown_connections 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- 172.21.15.69:0/4162193293 wait complete. 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 librados: starting msgr at 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 librados: starting objecter 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 Processor -- start 2024-04-02T04:43:39.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- start start 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 librados: setting wanted keys 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 librados: calling monclient init 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 0x7f8f0c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f0c102e60 0x7f8f0c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c107590 0x7f8f0c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c111a60 con 0x7f8f0c107590 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1118e0 con 0x7f8f0c102e60 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f120a9700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f0c111be0 con 0x7f8f0c1024b0 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 0x7f8f0c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 0x7f8f0c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42092/0 (socket says 172.21.15.69:42092) 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c107590 0x7f8f0c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:39.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 -- 172.21.15.69:0/3063664170 learned_addr learned my addr 172.21.15.69:0/3063664170 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 -- 172.21.15.69:0/3063664170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f0c102e60 msgr2=0x7f8f0c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f0c102e60 0x7f8f0c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 -- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c107590 msgr2=0x7f8f0c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c107590 0x7f8f0c19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 -- 172.21.15.69:0/3063664170 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0c19d050 con 0x7f8f0c1024b0 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f0b7fe700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 0x7f8f0c198040 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8efc005910 tx=0x7f8efc00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f08ff9700 10 client.?.objecter ms_handle_connect 0x7f8f0c1024b0 2024-04-02T04:43:39.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.421+0000 7f8f08ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc027070 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8efc00fc90 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0c19d2e0 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc022c90 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f120a9700 10 client.24302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0c1a4bc0 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f120a9700 1 librados: init done 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f0c04ff00 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f8efc02a070 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ef806f4f0 0x7f8ef80719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(75..75 src has 23..75) ==== 5287+0+0 (secure 0 0 0) 0x7f8efc08e390 con 0x7f8f0c1024b0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 10 client.24302.objecter ms_dispatch 0x7f8f0c001550 osd_map(75..75 src has 23..75) 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 3 client.24302.objecter handle_osd_map got epochs [75,75] > 0 2024-04-02T04:43:39.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 3 client.24302.objecter handle_osd_map decoding full epoch 75 2024-04-02T04:43:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f0affd700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ef806f4f0 0x7f8ef80719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 20 client.24302.objecter dump_active .. 0 homeless 2024-04-02T04:43:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f08ff9700 10 client.24302.objecter ms_handle_connect 0x7f8ef806f4f0 2024-04-02T04:43:39.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.425+0000 7f8f0affd700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ef806f4f0 0x7f8ef80719b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40060b0 tx=0x7f8ef4006040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.429+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8efc013070 con 0x7f8f0c1024b0 2024-04-02T04:43:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:39 smithi098 bash[23790]: cluster 2024-04-02T04:43:37.181277+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:39.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8f0c1a4d90 con 0x7f8f0c1024b0 2024-04-02T04:43:39.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f08ff9700 1 -- 172.21.15.69:0/3063664170 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f8efc05c2c0 con 0x7f8f0c1024b0 2024-04-02T04:43:39.638 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-02T04:43:39.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 10 librados: watch_flush enter 2024-04-02T04:43:39.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 10 librados: watch_flush exit 2024-04-02T04:43:39.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 20 client.24302.objecter shutdown clearing up homeless session... 2024-04-02T04:43:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 10 client.24302.objecter successfully canceled tick 2024-04-02T04:43:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ef806f4f0 msgr2=0x7f8ef80719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ef806f4f0 0x7f8ef80719b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40060b0 tx=0x7f8ef4006040 comp rx=0 tx=0).stop 2024-04-02T04:43:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 msgr2=0x7f8f0c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 0x7f8f0c198040 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8efc005910 tx=0x7f8efc00fa00 comp rx=0 tx=0).stop 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 shutdown_connections 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f0c107590 0x7f8f0c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f8ef806f4f0 0x7f8ef80719b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f0c1024b0 0x7f8f0c198040 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.637+0000 7f8f120a9700 1 --2- 172.21.15.69:0/3063664170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f0c102e60 0x7f8f0c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.641+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 >> 172.21.15.69:0/3063664170 conn(0x7f8f0c079610 msgr2=0x7f8f0c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.641+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 shutdown_connections 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.641+0000 7f8f120a9700 1 -- 172.21.15.69:0/3063664170 wait complete. 2024-04-02T04:43:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:43:39.641+0000 7f8f120a9700 1 librados: shutdown 2024-04-02T04:43:40.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:40 smithi069 bash[21528]: audit 2024-04-02T04:43:39.641043+0000 mon.c (mon.2) 21 : audit 0 from='client.? 172.21.15.69:0/3063664170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T04:43:40.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:40 smithi069 bash[28904]: audit 2024-04-02T04:43:39.641043+0000 mon.c (mon.2) 21 : audit 0 from='client.? 172.21.15.69:0/3063664170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T04:43:40.338 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi069 2024-04-02T04:43:40.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply rgw foo.a --placement '1;smithi069=foo.a' 2024-04-02T04:43:40.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:40 smithi098 bash[23790]: audit 2024-04-02T04:43:39.641043+0000 mon.c (mon.2) 21 : audit 0 from='client.? 172.21.15.69:0/3063664170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T04:43:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:41 smithi098 bash[23790]: cluster 2024-04-02T04:43:39.181900+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:41.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:41 smithi069 bash[21528]: cluster 2024-04-02T04:43:39.181900+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:41.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:41 smithi069 bash[28904]: cluster 2024-04-02T04:43:39.181900+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:43 smithi098 bash[23790]: cluster 2024-04-02T04:43:41.182563+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:43 smithi098 bash[23790]: cluster 2024-04-02T04:43:42.261051+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:43 smithi098 bash[23790]: cluster 2024-04-02T04:43:42.261146+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:43 smithi069 bash[21528]: cluster 2024-04-02T04:43:41.182563+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:43 smithi069 bash[21528]: cluster 2024-04-02T04:43:42.261051+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:43 smithi069 bash[21528]: cluster 2024-04-02T04:43:42.261146+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:43 smithi069 bash[28904]: cluster 2024-04-02T04:43:41.182563+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:43 smithi069 bash[28904]: cluster 2024-04-02T04:43:42.261051+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:43 smithi069 bash[28904]: cluster 2024-04-02T04:43:42.261146+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:44 smithi098 bash[23790]: cluster 2024-04-02T04:43:43.044028+0000 mon.a (mon.0) 685 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-04-02T04:43:44.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:44 smithi069 bash[21528]: cluster 2024-04-02T04:43:43.044028+0000 mon.a (mon.0) 685 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-04-02T04:43:44.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:44 smithi069 bash[28904]: cluster 2024-04-02T04:43:43.044028+0000 mon.a (mon.0) 685 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-04-02T04:43:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:43:45.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:45 smithi098 bash[23790]: cluster 2024-04-02T04:43:43.183105+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:45.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:45 smithi098 bash[23790]: cluster 2024-04-02T04:43:44.506738+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:43:45.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:45 smithi098 bash[23790]: cluster 2024-04-02T04:43:44.509954+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:43:45.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:45 smithi069 bash[21528]: cluster 2024-04-02T04:43:43.183105+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:45.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:45 smithi069 bash[21528]: cluster 2024-04-02T04:43:44.506738+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:43:45.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:45 smithi069 bash[21528]: cluster 2024-04-02T04:43:44.509954+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:43:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:45 smithi069 bash[28904]: cluster 2024-04-02T04:43:43.183105+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:45 smithi069 bash[28904]: cluster 2024-04-02T04:43:44.506738+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-04-02T04:43:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:45 smithi069 bash[28904]: cluster 2024-04-02T04:43:44.509954+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-04-02T04:43:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:47 smithi098 bash[23790]: cluster 2024-04-02T04:43:45.183714+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:47.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:47 smithi069 bash[21528]: cluster 2024-04-02T04:43:45.183714+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:47.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:47 smithi069 bash[28904]: cluster 2024-04-02T04:43:45.183714+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- 172.21.15.98:0/560266122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 msgr2=0x7f4bf0102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/560266122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf0102820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4be40099c0 tx=0x7f4be4009cd0 comp rx=0 tx=0).stop 2024-04-02T04:43:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- 172.21.15.98:0/560266122 shutdown_connections 2024-04-02T04:43:47.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/560266122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf0102d60 0x7f4bf0109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/560266122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf01011c0 0x7f4bf01015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/560266122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf0102820 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- 172.21.15.98:0/560266122 >> 172.21.15.98:0/560266122 conn(0x7f4bf00fc760 msgr2=0x7f4bf00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- 172.21.15.98:0/560266122 shutdown_connections 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- 172.21.15.98:0/560266122 wait complete. 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 librados: starting msgr at 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 librados: starting objecter 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 Processor -- start 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- start start 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 librados: setting wanted keys 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 librados: calling monclient init 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf01011c0 0x7f4bf006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf0102d60 0x7f4bf019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010c340 con 0x7f4bf01011c0 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010c1c0 con 0x7f4bf01023c0 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bf63b6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf010c4c0 con 0x7f4bf0102d60 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bef7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bef7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf019c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58812/0 (socket says 172.21.15.98:58812) 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.908+0000 7f4bef7fe700 1 -- 172.21.15.98:0/156161958 learned_addr learned my addr 172.21.15.98:0/156161958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf4953700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf0102d60 0x7f4bf019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4beffff700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf01011c0 0x7f4bf006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bef7fe700 1 -- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf0102d60 msgr2=0x7f4bf019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bef7fe700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf0102d60 0x7f4bf019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bef7fe700 1 -- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf01011c0 msgr2=0x7f4bf006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bef7fe700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf01011c0 0x7f4bf006b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bef7fe700 1 -- 172.21.15.98:0/156161958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf019f6f0 con 0x7f4bf01023c0 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4beffff700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf01011c0 0x7f4bf006b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf4953700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf0102d60 0x7f4bf019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bef7fe700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf019c7d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4be4005910 tx=0x7f4be4003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 10 client.?.objecter ms_handle_connect 0x7f4bf01023c0 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4027070 con 0x7f4bf01023c0 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4be4022470 con 0x7f4bf01023c0 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be400f6c0 con 0x7f4bf01023c0 2024-04-02T04:43:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf019f980 con 0x7f4bf01023c0 2024-04-02T04:43:47.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf63b6700 10 client.24301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:43:47.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf019fd80 con 0x7f4bf01023c0 2024-04-02T04:43:47.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf63b6700 1 librados: init done 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf004ff00 con 0x7f4bf01023c0 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f4be4022900 con 0x7f4bf01023c0 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f4bd806f4f0 0x7f4bd80719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 23..76) ==== 5287+0+0 (secure 0 0 0) 0x7f4be408e800 con 0x7f4bf01023c0 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 10 client.24301.objecter ms_dispatch 0x7f4bf0001550 osd_map(76..76 src has 23..76) 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 3 client.24301.objecter handle_osd_map got epochs [76,76] > 0 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 3 client.24301.objecter handle_osd_map decoding full epoch 76 2024-04-02T04:43:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4bed7fa700 20 client.24301.objecter dump_active .. 0 homeless 2024-04-02T04:43:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.912+0000 7f4beffff700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f4bd806f4f0 0x7f4bd80719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.916+0000 7f4bed7fa700 10 client.24301.objecter ms_handle_connect 0x7f4bd806f4f0 2024-04-02T04:43:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.916+0000 7f4beffff700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f4bd806f4f0 0x7f4bd80719b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0102220 tx=0x7f4be80076a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:47.920+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4be4013070 con 0x7f4bf01023c0 2024-04-02T04:43:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.124+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}) -- 0x7f4bf002dfb0 con 0x7f4bd806f4f0 2024-04-02T04:43:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.140+0000 7f4bed7fa700 1 -- 172.21.15.98:0/156161958 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f4bf002dfb0 con 0x7f4bd806f4f0 2024-04-02T04:43:48.147 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled rgw.foo.a update... 2024-04-02T04:43:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 10 librados: watch_flush enter 2024-04-02T04:43:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 10 librados: watch_flush exit 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 20 client.24301.objecter shutdown clearing up homeless session... 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 10 client.24301.objecter successfully canceled tick 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f4bd806f4f0 msgr2=0x7f4bd80719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f4bd806f4f0 0x7f4bd80719b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0102220 tx=0x7f4be80076a0 comp rx=0 tx=0).stop 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 msgr2=0x7f4bf019c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf019c7d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4be4005910 tx=0x7f4be4003970 comp rx=0 tx=0).stop 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 shutdown_connections 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f4bd806f4f0 0x7f4bd80719b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bf0102d60 0x7f4bf019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bf01011c0 0x7f4bf006b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 --2- 172.21.15.98:0/156161958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bf01023c0 0x7f4bf019c7d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 >> 172.21.15.98:0/156161958 conn(0x7f4bf00fc760 msgr2=0x7f4bf01056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 shutdown_connections 2024-04-02T04:43:48.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 -- 172.21.15.98:0/156161958 wait complete. 2024-04-02T04:43:48.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:48.144+0000 7f4bf63b6700 1 librados: shutdown 2024-04-02T04:43:48.897 DEBUG:teuthology.orchestra.run.smithi069:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@rgw.foo.a.service 2024-04-02T04:43:48.899 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi098 2024-04-02T04:43:48.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool create datapool 3 3 replicated 2024-04-02T04:43:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: cluster 2024-04-02T04:43:47.184234+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.149334+0000 mon.a (mon.0) 686 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.200545+0000 mon.a (mon.0) 687 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.886485+0000 mon.a (mon.0) 688 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.888706+0000 mon.a (mon.0) 689 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.901194+0000 mon.a (mon.0) 690 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.908105+0000 mon.a (mon.0) 691 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.913750+0000 mon.a (mon.0) 692 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.928083+0000 mon.a (mon.0) 693 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:49 smithi098 bash[23790]: audit 2024-04-02T04:43:48.931747+0000 mon.a (mon.0) 694 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:49.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: cluster 2024-04-02T04:43:47.184234+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:49.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.149334+0000 mon.a (mon.0) 686 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.200545+0000 mon.a (mon.0) 687 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:49.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.886485+0000 mon.a (mon.0) 688 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.888706+0000 mon.a (mon.0) 689 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.901194+0000 mon.a (mon.0) 690 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.908105+0000 mon.a (mon.0) 691 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.913750+0000 mon.a (mon.0) 692 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.928083+0000 mon.a (mon.0) 693 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[21528]: audit 2024-04-02T04:43:48.931747+0000 mon.a (mon.0) 694 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: cluster 2024-04-02T04:43:47.184234+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.149334+0000 mon.a (mon.0) 686 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.200545+0000 mon.a (mon.0) 687 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.886485+0000 mon.a (mon.0) 688 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.888706+0000 mon.a (mon.0) 689 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:49.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.901194+0000 mon.a (mon.0) 690 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.908105+0000 mon.a (mon.0) 691 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:43:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.913750+0000 mon.a (mon.0) 692 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:43:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.928083+0000 mon.a (mon.0) 693 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:49 smithi069 bash[28904]: audit 2024-04-02T04:43:48.931747+0000 mon.a (mon.0) 694 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:50.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[28904]: audit 2024-04-02T04:43:48.138111+0000 mgr.y (mgr.14152) 334 : audit 0 from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[28904]: cephadm 2024-04-02T04:43:48.142115+0000 mgr.y (mgr.14152) 335 : cephadm 1 Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[28904]: cephadm 2024-04-02T04:43:48.933561+0000 mgr.y (mgr.14152) 336 : cephadm 1 Deploying daemon rgw.foo.a on smithi069 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[28904]: cluster 2024-04-02T04:43:49.184801+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[21528]: audit 2024-04-02T04:43:48.138111+0000 mgr.y (mgr.14152) 334 : audit 0 from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[21528]: cephadm 2024-04-02T04:43:48.142115+0000 mgr.y (mgr.14152) 335 : cephadm 1 Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[21528]: cephadm 2024-04-02T04:43:48.933561+0000 mgr.y (mgr.14152) 336 : cephadm 1 Deploying daemon rgw.foo.a on smithi069 2024-04-02T04:43:50.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:50 smithi069 bash[21528]: cluster 2024-04-02T04:43:49.184801+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:50 smithi098 bash[23790]: audit 2024-04-02T04:43:48.138111+0000 mgr.y (mgr.14152) 334 : audit 0 from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:50 smithi098 bash[23790]: cephadm 2024-04-02T04:43:48.142115+0000 mgr.y (mgr.14152) 335 : cephadm 1 Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2024-04-02T04:43:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:50 smithi098 bash[23790]: cephadm 2024-04-02T04:43:48.933561+0000 mgr.y (mgr.14152) 336 : cephadm 1 Deploying daemon rgw.foo.a on smithi069 2024-04-02T04:43:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:50 smithi098 bash[23790]: cluster 2024-04-02T04:43:49.184801+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:51.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.279 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.280 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.280 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.280 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.281 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.779 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.779 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.780 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.780 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:51.780 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:43:51 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:43:53.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: cluster 2024-04-02T04:43:51.185220+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:53.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:51.448140+0000 mon.a (mon.0) 695 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:51.453399+0000 mon.a (mon.0) 696 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:51.507475+0000 mon.a (mon.0) 697 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: cephadm 2024-04-02T04:43:51.508355+0000 mgr.y (mgr.14152) 339 : cephadm 1 Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2024-04-02T04:43:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:51.512223+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:51.516814+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:51.581898+0000 mon.a (mon.0) 700 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:52 smithi098 bash[23790]: audit 2024-04-02T04:43:52.183895+0000 mon.a (mon.0) 701 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: cluster 2024-04-02T04:43:51.185220+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:51.448140+0000 mon.a (mon.0) 695 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:51.453399+0000 mon.a (mon.0) 696 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:51.507475+0000 mon.a (mon.0) 697 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: cephadm 2024-04-02T04:43:51.508355+0000 mgr.y (mgr.14152) 339 : cephadm 1 Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:51.512223+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:51.516814+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:51.581898+0000 mon.a (mon.0) 700 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[21528]: audit 2024-04-02T04:43:52.183895+0000 mon.a (mon.0) 701 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: cluster 2024-04-02T04:43:51.185220+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:53.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:51.448140+0000 mon.a (mon.0) 695 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:51.453399+0000 mon.a (mon.0) 696 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:51.507475+0000 mon.a (mon.0) 697 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: cephadm 2024-04-02T04:43:51.508355+0000 mgr.y (mgr.14152) 339 : cephadm 1 Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:51.512223+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:51.516814+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:51.581898+0000 mon.a (mon.0) 700 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:53.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:52 smithi069 bash[28904]: audit 2024-04-02T04:43:52.183895+0000 mon.a (mon.0) 701 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:43:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:54 smithi098 bash[23790]: cluster 2024-04-02T04:43:53.186191+0000 mgr.y (mgr.14152) 340 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:54 smithi098 bash[23790]: cluster 2024-04-02T04:43:53.191483+0000 mon.a (mon.0) 702 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-04-02T04:43:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:54 smithi098 bash[23790]: audit 2024-04-02T04:43:53.200805+0000 mon.c (mon.2) 22 : audit 1 from='client.? 172.21.15.69:0/680486136' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-02T04:43:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:54 smithi098 bash[23790]: audit 2024-04-02T04:43:53.201499+0000 mon.a (mon.0) 703 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-02T04:43:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[21528]: cluster 2024-04-02T04:43:53.186191+0000 mgr.y (mgr.14152) 340 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[21528]: cluster 2024-04-02T04:43:53.191483+0000 mon.a (mon.0) 702 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[21528]: audit 2024-04-02T04:43:53.200805+0000 mon.c (mon.2) 22 : audit 1 from='client.? 172.21.15.69:0/680486136' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[21528]: audit 2024-04-02T04:43:53.201499+0000 mon.a (mon.0) 703 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[28904]: cluster 2024-04-02T04:43:53.186191+0000 mgr.y (mgr.14152) 340 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[28904]: cluster 2024-04-02T04:43:53.191483+0000 mon.a (mon.0) 702 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[28904]: audit 2024-04-02T04:43:53.200805+0000 mon.c (mon.2) 22 : audit 1 from='client.? 172.21.15.69:0/680486136' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-02T04:43:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:54 smithi069 bash[28904]: audit 2024-04-02T04:43:53.201499+0000 mon.a (mon.0) 703 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-02T04:43:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:55 smithi098 bash[23790]: cluster 2024-04-02T04:43:54.184060+0000 mon.a (mon.0) 704 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:43:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:55 smithi098 bash[23790]: audit 2024-04-02T04:43:54.185639+0000 mon.a (mon.0) 705 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-02T04:43:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:55 smithi098 bash[23790]: cluster 2024-04-02T04:43:54.192813+0000 mon.a (mon.0) 706 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-04-02T04:43:55.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:55 smithi069 bash[21528]: cluster 2024-04-02T04:43:54.184060+0000 mon.a (mon.0) 704 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:43:55.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:55 smithi069 bash[21528]: audit 2024-04-02T04:43:54.185639+0000 mon.a (mon.0) 705 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-02T04:43:55.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:55 smithi069 bash[21528]: cluster 2024-04-02T04:43:54.192813+0000 mon.a (mon.0) 706 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-04-02T04:43:55.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:55 smithi069 bash[28904]: cluster 2024-04-02T04:43:54.184060+0000 mon.a (mon.0) 704 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:43:55.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:55 smithi069 bash[28904]: audit 2024-04-02T04:43:54.185639+0000 mon.a (mon.0) 705 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-02T04:43:55.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:55 smithi069 bash[28904]: cluster 2024-04-02T04:43:54.192813+0000 mon.a (mon.0) 706 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-04-02T04:43:56.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: cluster 2024-04-02T04:43:55.186788+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v342: 33 pgs: 11 creating+peering, 1 active+clean, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: cluster 2024-04-02T04:43:55.200884+0000 mon.a (mon.0) 707 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: cluster 2024-04-02T04:43:55.200990+0000 mon.a (mon.0) 708 : cluster 1 Cluster is now healthy 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: cluster 2024-04-02T04:43:55.213793+0000 mon.a (mon.0) 709 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: audit 2024-04-02T04:43:55.217571+0000 mon.a (mon.0) 710 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: audit 2024-04-02T04:43:56.205383+0000 mon.a (mon.0) 711 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[21528]: cluster 2024-04-02T04:43:56.214591+0000 mon.a (mon.0) 712 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: cluster 2024-04-02T04:43:55.186788+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v342: 33 pgs: 11 creating+peering, 1 active+clean, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: cluster 2024-04-02T04:43:55.200884+0000 mon.a (mon.0) 707 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: cluster 2024-04-02T04:43:55.200990+0000 mon.a (mon.0) 708 : cluster 1 Cluster is now healthy 2024-04-02T04:43:56.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: cluster 2024-04-02T04:43:55.213793+0000 mon.a (mon.0) 709 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-04-02T04:43:56.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: audit 2024-04-02T04:43:55.217571+0000 mon.a (mon.0) 710 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-02T04:43:56.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: audit 2024-04-02T04:43:56.205383+0000 mon.a (mon.0) 711 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-02T04:43:56.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:56 smithi069 bash[28904]: cluster 2024-04-02T04:43:56.214591+0000 mon.a (mon.0) 712 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/1720620660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820102d60 msgr2=0x7f2820105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- 172.21.15.98:0/1720620660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820102d60 0x7f2820105150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f281000b0a0 tx=0x7f281000b3b0 comp rx=0 tx=0).stop 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/1720620660 shutdown_connections 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- 172.21.15.98:0/1720620660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2820107fc0 0x7f282010a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- 172.21.15.98:0/1720620660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820102d60 0x7f2820105150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- 172.21.15.98:0/1720620660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2820105690 0x7f2820107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/1720620660 >> 172.21.15.98:0/1720620660 conn(0x7f28200fc770 msgr2=0x7f28200febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/1720620660 shutdown_connections 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/1720620660 wait complete. 2024-04-02T04:43:56.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 librados: starting msgr at 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 librados: starting objecter 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 Processor -- start 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- start start 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 librados: setting wanted keys 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 librados: calling monclient init 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2820102d60 0x7f28201026c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 0x7f2820100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2820107fc0 0x7f2820101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f282010d3c0 con 0x7f2820102d60 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f282010d240 con 0x7f2820105690 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f282010d540 con 0x7f2820107fc0 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 0x7f2820100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 0x7f2820100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58936/0 (socket says 172.21.15.98:58936) 2024-04-02T04:43:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 -- 172.21.15.98:0/4049066393 learned_addr learned my addr 172.21.15.98:0/4049066393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281e59c700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2820102d60 0x7f28201026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281ed9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2820107fc0 0x7f2820101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 -- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2820107fc0 msgr2=0x7f2820101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2820107fc0 0x7f2820101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 -- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2820102d60 msgr2=0x7f28201026c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2820102d60 0x7f28201026c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 -- 172.21.15.98:0/4049066393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2820101b00 con 0x7f2820105690 2024-04-02T04:43:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f281dd9b700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 0x7f2820100d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f281400d950 tx=0x7f281400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f280f7fe700 10 client.?.objecter ms_handle_connect 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f280f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2814016070 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f281400fd10 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28201a1c40 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 10 client.24319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281400e3b0 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28201a2150 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.584+0000 7f2824f9d700 1 librados: init done 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f282004ff00 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f2814019070 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f280806f4f0 0x7f28080719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 23..80) ==== 6011+0+0 (secure 0 0 0) 0x7f281408d760 con 0x7f2820105690 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 10 client.24319.objecter ms_dispatch 0x7f2820001550 osd_map(80..80 src has 23..80) 2024-04-02T04:43:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 3 client.24319.objecter handle_osd_map got epochs [80,80] > 0 2024-04-02T04:43:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 3 client.24319.objecter handle_osd_map decoding full epoch 80 2024-04-02T04:43:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f281e59c700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f280806f4f0 0x7f28080719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:43:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 20 client.24319.objecter dump_active .. 0 homeless 2024-04-02T04:43:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f280f7fe700 10 client.24319.objecter ms_handle_connect 0x7f280806f4f0 2024-04-02T04:43:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.588+0000 7f281e59c700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f280806f4f0 0x7f28080719b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2810009fd0 tx=0x7f281000b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:43:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.592+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f281405b340 con 0x7f2820105690 2024-04-02T04:43:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: cluster 2024-04-02T04:43:55.186788+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v342: 33 pgs: 11 creating+peering, 1 active+clean, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:43:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: cluster 2024-04-02T04:43:55.200884+0000 mon.a (mon.0) 707 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-02T04:43:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: cluster 2024-04-02T04:43:55.200990+0000 mon.a (mon.0) 708 : cluster 1 Cluster is now healthy 2024-04-02T04:43:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: cluster 2024-04-02T04:43:55.213793+0000 mon.a (mon.0) 709 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-04-02T04:43:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: audit 2024-04-02T04:43:55.217571+0000 mon.a (mon.0) 710 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-02T04:43:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: audit 2024-04-02T04:43:56.205383+0000 mon.a (mon.0) 711 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-02T04:43:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:56 smithi098 bash[23790]: cluster 2024-04-02T04:43:56.214591+0000 mon.a (mon.0) 712 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-04-02T04:43:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:56.804+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7f282002dea0 con 0x7f2820105690 2024-04-02T04:43:57.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.260+0000 7f280f7fe700 1 -- 172.21.15.98:0/4049066393 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v81) ==== 160+0+0 (secure 0 0 0) 0x7f281405ac40 con 0x7f2820105690 2024-04-02T04:43:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.268+0000 7f2824f9d700 10 librados: watch_flush enter 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.268+0000 7f2824f9d700 10 librados: watch_flush exit 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.268+0000 7f2824f9d700 20 client.24319.objecter shutdown clearing up homeless session... 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.268+0000 7f2824f9d700 10 client.24319.objecter successfully canceled tick 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f280806f4f0 msgr2=0x7f28080719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f280806f4f0 0x7f28080719b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2810009fd0 tx=0x7f281000b7b0 comp rx=0 tx=0).stop 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 msgr2=0x7f2820100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 0x7f2820100d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f281400d950 tx=0x7f281400dc60 comp rx=0 tx=0).stop 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 shutdown_connections 2024-04-02T04:43:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f280806f4f0 0x7f28080719b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2820107fc0 0x7f2820101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2820102d60 0x7f28201026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 --2- 172.21.15.98:0/4049066393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2820105690 0x7f2820100d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 >> 172.21.15.98:0/4049066393 conn(0x7f28200fc770 msgr2=0x7f28201039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 shutdown_connections 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 -- 172.21.15.98:0/4049066393 wait complete. 2024-04-02T04:43:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:43:57.272+0000 7f2824f9d700 1 librados: shutdown 2024-04-02T04:43:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'datapool' created 2024-04-02T04:43:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:56.810174+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.98:0/4049066393' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:56.815256+0000 mon.a (mon.0) 713 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: cluster 2024-04-02T04:43:56.948895+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: cluster 2024-04-02T04:43:56.948975+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:57.260125+0000 mon.a (mon.0) 714 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: cluster 2024-04-02T04:43:57.274959+0000 mon.a (mon.0) 715 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:57.282220+0000 mon.a (mon.0) 716 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:57.284145+0000 mon.a (mon.0) 717 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:57.292422+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:57.294012+0000 mon.a (mon.0) 719 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:57 smithi098 bash[23790]: audit 2024-04-02T04:43:57.294960+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:56.810174+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.98:0/4049066393' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:56.815256+0000 mon.a (mon.0) 713 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: cluster 2024-04-02T04:43:56.948895+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: cluster 2024-04-02T04:43:56.948975+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:57.260125+0000 mon.a (mon.0) 714 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: cluster 2024-04-02T04:43:57.274959+0000 mon.a (mon.0) 715 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:57.282220+0000 mon.a (mon.0) 716 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:57.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:57.284145+0000 mon.a (mon.0) 717 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:57.292422+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:57.294012+0000 mon.a (mon.0) 719 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[21528]: audit 2024-04-02T04:43:57.294960+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:56.810174+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.98:0/4049066393' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:56.815256+0000 mon.a (mon.0) 713 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: cluster 2024-04-02T04:43:56.948895+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: cluster 2024-04-02T04:43:56.948975+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:57.260125+0000 mon.a (mon.0) 714 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: cluster 2024-04-02T04:43:57.274959+0000 mon.a (mon.0) 715 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:57.282220+0000 mon.a (mon.0) 716 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:57.284145+0000 mon.a (mon.0) 717 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-02T04:43:57.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:57.292422+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:43:57.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:57.294012+0000 mon.a (mon.0) 719 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:57.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:57 smithi069 bash[28904]: audit 2024-04-02T04:43:57.294960+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:58.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- rbd pool init datapool 2024-04-02T04:43:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:58 smithi098 bash[23790]: cluster 2024-04-02T04:43:57.187536+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v345: 65 pgs: 23 creating+peering, 22 active+clean, 20 unknown; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s 2024-04-02T04:43:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:58 smithi098 bash[23790]: cephadm 2024-04-02T04:43:57.299357+0000 mgr.y (mgr.14152) 343 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-02T04:43:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:58 smithi098 bash[23790]: audit 2024-04-02T04:43:58.262848+0000 mon.a (mon.0) 721 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-02T04:43:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:43:58 smithi098 bash[23790]: cluster 2024-04-02T04:43:58.271378+0000 mon.a (mon.0) 722 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[21528]: cluster 2024-04-02T04:43:57.187536+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v345: 65 pgs: 23 creating+peering, 22 active+clean, 20 unknown; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[21528]: cephadm 2024-04-02T04:43:57.299357+0000 mgr.y (mgr.14152) 343 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[21528]: audit 2024-04-02T04:43:58.262848+0000 mon.a (mon.0) 721 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[21528]: cluster 2024-04-02T04:43:58.271378+0000 mon.a (mon.0) 722 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[28904]: cluster 2024-04-02T04:43:57.187536+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v345: 65 pgs: 23 creating+peering, 22 active+clean, 20 unknown; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[28904]: cephadm 2024-04-02T04:43:57.299357+0000 mgr.y (mgr.14152) 343 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[28904]: audit 2024-04-02T04:43:58.262848+0000 mon.a (mon.0) 721 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-02T04:43:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:43:58 smithi069 bash[28904]: cluster 2024-04-02T04:43:58.271378+0000 mon.a (mon.0) 722 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-04-02T04:44:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:00 smithi098 bash[23790]: cluster 2024-04-02T04:43:59.188477+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v348: 100 pgs: 12 creating+peering, 60 active+clean, 28 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-02T04:44:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:00 smithi098 bash[23790]: cluster 2024-04-02T04:43:59.301073+0000 mon.a (mon.0) 723 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-04-02T04:44:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:00 smithi098 bash[23790]: audit 2024-04-02T04:43:59.302611+0000 mon.a (mon.0) 724 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-02T04:44:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:00 smithi098 bash[23790]: audit 2024-04-02T04:43:59.303123+0000 mon.a (mon.0) 725 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-02T04:44:00.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:00 smithi098 bash[23790]: cluster 2024-04-02T04:44:00.290845+0000 mon.a (mon.0) 726 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[21528]: cluster 2024-04-02T04:43:59.188477+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v348: 100 pgs: 12 creating+peering, 60 active+clean, 28 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[21528]: cluster 2024-04-02T04:43:59.301073+0000 mon.a (mon.0) 723 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[21528]: audit 2024-04-02T04:43:59.302611+0000 mon.a (mon.0) 724 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[21528]: audit 2024-04-02T04:43:59.303123+0000 mon.a (mon.0) 725 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[21528]: cluster 2024-04-02T04:44:00.290845+0000 mon.a (mon.0) 726 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[28904]: cluster 2024-04-02T04:43:59.188477+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v348: 100 pgs: 12 creating+peering, 60 active+clean, 28 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[28904]: cluster 2024-04-02T04:43:59.301073+0000 mon.a (mon.0) 723 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[28904]: audit 2024-04-02T04:43:59.302611+0000 mon.a (mon.0) 724 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-02T04:44:00.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[28904]: audit 2024-04-02T04:43:59.303123+0000 mon.a (mon.0) 725 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-02T04:44:00.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:00 smithi069 bash[28904]: cluster 2024-04-02T04:44:00.290845+0000 mon.a (mon.0) 726 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: audit 2024-04-02T04:44:00.303378+0000 mon.a (mon.0) 727 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: audit 2024-04-02T04:44:00.303541+0000 mon.a (mon.0) 728 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:00.311302+0000 mon.a (mon.0) 729 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: audit 2024-04-02T04:44:00.314804+0000 mon.a (mon.0) 730 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: audit 2024-04-02T04:44:00.334238+0000 mon.a (mon.0) 731 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: audit 2024-04-02T04:44:01.306304+0000 mon.a (mon.0) 732 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: audit 2024-04-02T04:44:01.306663+0000 mon.a (mon.0) 733 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-02T04:44:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:01.314107+0000 mon.a (mon.0) 734 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-04-02T04:44:01.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: audit 2024-04-02T04:44:00.303378+0000 mon.a (mon.0) 727 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-02T04:44:01.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: audit 2024-04-02T04:44:00.303541+0000 mon.a (mon.0) 728 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:00.311302+0000 mon.a (mon.0) 729 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: audit 2024-04-02T04:44:00.314804+0000 mon.a (mon.0) 730 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: audit 2024-04-02T04:44:00.334238+0000 mon.a (mon.0) 731 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: audit 2024-04-02T04:44:01.306304+0000 mon.a (mon.0) 732 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: audit 2024-04-02T04:44:01.306663+0000 mon.a (mon.0) 733 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:01.314107+0000 mon.a (mon.0) 734 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: audit 2024-04-02T04:44:00.303378+0000 mon.a (mon.0) 727 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: audit 2024-04-02T04:44:00.303541+0000 mon.a (mon.0) 728 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:00.311302+0000 mon.a (mon.0) 729 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: audit 2024-04-02T04:44:00.314804+0000 mon.a (mon.0) 730 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-02T04:44:01.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: audit 2024-04-02T04:44:00.334238+0000 mon.a (mon.0) 731 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-02T04:44:01.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: audit 2024-04-02T04:44:01.306304+0000 mon.a (mon.0) 732 : audit 1 from='client.? 172.21.15.69:0/1159886337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-02T04:44:01.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: audit 2024-04-02T04:44:01.306663+0000 mon.a (mon.0) 733 : audit 1 from='client.? 172.21.15.69:0/1965820815' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-02T04:44:01.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:01.314107+0000 mon.a (mon.0) 734 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-04-02T04:44:02.884 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: cluster 2024-04-02T04:44:01.189473+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v351: 132 pgs: 97 active+clean, 35 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.628458+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.638723+0000 mon.a (mon.0) 736 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.653340+0000 mon.a (mon.0) 737 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.722356+0000 mon.a (mon.0) 738 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.724430+0000 mon.a (mon.0) 739 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.725209+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: cephadm 2024-04-02T04:44:01.729953+0000 mgr.y (mgr.14152) 346 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-02T04:44:02.906 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:02 smithi098 bash[23790]: audit 2024-04-02T04:44:01.984083+0000 mon.a (mon.0) 741 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: cluster 2024-04-02T04:44:01.189473+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v351: 132 pgs: 97 active+clean, 35 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.628458+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.638723+0000 mon.a (mon.0) 736 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.653340+0000 mon.a (mon.0) 737 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.722356+0000 mon.a (mon.0) 738 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.724430+0000 mon.a (mon.0) 739 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.725209+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: cephadm 2024-04-02T04:44:01.729953+0000 mgr.y (mgr.14152) 346 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-02T04:44:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[21528]: audit 2024-04-02T04:44:01.984083+0000 mon.a (mon.0) 741 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: cluster 2024-04-02T04:44:01.189473+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v351: 132 pgs: 97 active+clean, 35 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.628458+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.638723+0000 mon.a (mon.0) 736 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.653340+0000 mon.a (mon.0) 737 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.722356+0000 mon.a (mon.0) 738 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.724430+0000 mon.a (mon.0) 739 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.725209+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: cephadm 2024-04-02T04:44:01.729953+0000 mgr.y (mgr.14152) 346 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-02T04:44:03.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:02 smithi069 bash[28904]: audit 2024-04-02T04:44:01.984083+0000 mon.a (mon.0) 741 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:04.266 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:03 smithi098 bash[23790]: cluster 2024-04-02T04:44:02.561692+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:04.267 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:03 smithi098 bash[23790]: cluster 2024-04-02T04:44:02.561800+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:04.267 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:03 smithi098 bash[23790]: cluster 2024-04-02T04:44:02.993919+0000 mon.a (mon.0) 742 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-04-02T04:44:04.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:03 smithi069 bash[21528]: cluster 2024-04-02T04:44:02.561692+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:04.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:03 smithi069 bash[21528]: cluster 2024-04-02T04:44:02.561800+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:04.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:03 smithi069 bash[21528]: cluster 2024-04-02T04:44:02.993919+0000 mon.a (mon.0) 742 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-04-02T04:44:04.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:03 smithi069 bash[28904]: cluster 2024-04-02T04:44:02.561692+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:04.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:03 smithi069 bash[28904]: cluster 2024-04-02T04:44:02.561800+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:04.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:03 smithi069 bash[28904]: cluster 2024-04-02T04:44:02.993919+0000 mon.a (mon.0) 742 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-04-02T04:44:05.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:04 smithi069 bash[21528]: cluster 2024-04-02T04:44:03.190453+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v354: 132 pgs: 125 active+clean, 7 unknown; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.5 KiB/s wr, 149 op/s 2024-04-02T04:44:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:04 smithi069 bash[28904]: cluster 2024-04-02T04:44:03.190453+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v354: 132 pgs: 125 active+clean, 7 unknown; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.5 KiB/s wr, 149 op/s 2024-04-02T04:44:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:04 smithi098 bash[23790]: cluster 2024-04-02T04:44:03.190453+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v354: 132 pgs: 125 active+clean, 7 unknown; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.5 KiB/s wr, 149 op/s 2024-04-02T04:44:06.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:05 smithi098 bash[23790]: audit 2024-04-02T04:44:05.897032+0000 mon.b (mon.1) 27 : audit 1 from='client.? 172.21.15.98:0/2534211887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-02T04:44:06.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:05 smithi098 bash[23790]: audit 2024-04-02T04:44:05.902322+0000 mon.a (mon.0) 743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-02T04:44:06.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:05 smithi069 bash[21528]: audit 2024-04-02T04:44:05.897032+0000 mon.b (mon.1) 27 : audit 1 from='client.? 172.21.15.98:0/2534211887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-02T04:44:06.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:05 smithi069 bash[21528]: audit 2024-04-02T04:44:05.902322+0000 mon.a (mon.0) 743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-02T04:44:06.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:06 smithi069 bash[28904]: audit 2024-04-02T04:44:05.897032+0000 mon.b (mon.1) 27 : audit 1 from='client.? 172.21.15.98:0/2534211887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-02T04:44:06.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:06 smithi069 bash[28904]: audit 2024-04-02T04:44:05.902322+0000 mon.a (mon.0) 743 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-02T04:44:07.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:07 smithi069 bash[21528]: cluster 2024-04-02T04:44:05.191694+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.1 KiB/s wr, 184 op/s 2024-04-02T04:44:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:07 smithi069 bash[21528]: audit 2024-04-02T04:44:06.008439+0000 mon.a (mon.0) 744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-02T04:44:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:07 smithi069 bash[21528]: cluster 2024-04-02T04:44:06.015918+0000 mon.a (mon.0) 745 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-04-02T04:44:07.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:07 smithi069 bash[28904]: cluster 2024-04-02T04:44:05.191694+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.1 KiB/s wr, 184 op/s 2024-04-02T04:44:07.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:07 smithi069 bash[28904]: audit 2024-04-02T04:44:06.008439+0000 mon.a (mon.0) 744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-02T04:44:07.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:07 smithi069 bash[28904]: cluster 2024-04-02T04:44:06.015918+0000 mon.a (mon.0) 745 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-04-02T04:44:07.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:07 smithi098 bash[23790]: cluster 2024-04-02T04:44:05.191694+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.1 KiB/s wr, 184 op/s 2024-04-02T04:44:07.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:07 smithi098 bash[23790]: audit 2024-04-02T04:44:06.008439+0000 mon.a (mon.0) 744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-02T04:44:07.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:07 smithi098 bash[23790]: cluster 2024-04-02T04:44:06.015918+0000 mon.a (mon.0) 745 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-04-02T04:44:08.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:08 smithi098 bash[23790]: cluster 2024-04-02T04:44:07.018969+0000 mon.a (mon.0) 746 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-04-02T04:44:08.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:08 smithi069 bash[21528]: cluster 2024-04-02T04:44:07.018969+0000 mon.a (mon.0) 746 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-04-02T04:44:08.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:08 smithi069 bash[28904]: cluster 2024-04-02T04:44:07.018969+0000 mon.a (mon.0) 746 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-04-02T04:44:08.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.98 --placement '1;smithi098=iscsi.a' 2024-04-02T04:44:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:09 smithi098 bash[23790]: cluster 2024-04-02T04:44:07.192542+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 5.9 KiB/s wr, 274 op/s 2024-04-02T04:44:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:09 smithi098 bash[23790]: cluster 2024-04-02T04:44:08.024692+0000 mon.a (mon.0) 747 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-02T04:44:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:09 smithi098 bash[23790]: cluster 2024-04-02T04:44:08.024751+0000 mon.a (mon.0) 748 : cluster 1 Cluster is now healthy 2024-04-02T04:44:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:09 smithi098 bash[23790]: cluster 2024-04-02T04:44:08.047472+0000 mon.a (mon.0) 749 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-04-02T04:44:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:09 smithi098 bash[23790]: cluster 2024-04-02T04:44:08.858808+0000 mon.a (mon.0) 750 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-04-02T04:44:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[21528]: cluster 2024-04-02T04:44:07.192542+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 5.9 KiB/s wr, 274 op/s 2024-04-02T04:44:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[21528]: cluster 2024-04-02T04:44:08.024692+0000 mon.a (mon.0) 747 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[21528]: cluster 2024-04-02T04:44:08.024751+0000 mon.a (mon.0) 748 : cluster 1 Cluster is now healthy 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[21528]: cluster 2024-04-02T04:44:08.047472+0000 mon.a (mon.0) 749 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[21528]: cluster 2024-04-02T04:44:08.858808+0000 mon.a (mon.0) 750 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[28904]: cluster 2024-04-02T04:44:07.192542+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 5.9 KiB/s wr, 274 op/s 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[28904]: cluster 2024-04-02T04:44:08.024692+0000 mon.a (mon.0) 747 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[28904]: cluster 2024-04-02T04:44:08.024751+0000 mon.a (mon.0) 748 : cluster 1 Cluster is now healthy 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[28904]: cluster 2024-04-02T04:44:08.047472+0000 mon.a (mon.0) 749 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-04-02T04:44:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:09 smithi069 bash[28904]: cluster 2024-04-02T04:44:08.858808+0000 mon.a (mon.0) 750 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-04-02T04:44:11.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:11 smithi098 bash[23790]: cluster 2024-04-02T04:44:09.193511+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 2.7 KiB/s wr, 132 op/s 2024-04-02T04:44:11.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:11 smithi069 bash[21528]: cluster 2024-04-02T04:44:09.193511+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 2.7 KiB/s wr, 132 op/s 2024-04-02T04:44:11.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:11 smithi069 bash[28904]: cluster 2024-04-02T04:44:09.193511+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 2.7 KiB/s wr, 132 op/s 2024-04-02T04:44:13.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:13 smithi098 bash[23790]: cluster 2024-04-02T04:44:11.194558+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 197 B/s wr, 2 op/s 2024-04-02T04:44:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:13 smithi069 bash[21528]: cluster 2024-04-02T04:44:11.194558+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 197 B/s wr, 2 op/s 2024-04-02T04:44:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:13 smithi069 bash[28904]: cluster 2024-04-02T04:44:11.194558+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 197 B/s wr, 2 op/s 2024-04-02T04:44:13.584 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:44:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:14 smithi098 bash[23790]: cluster 2024-04-02T04:44:13.195688+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 331 B/s wr, 0 op/s 2024-04-02T04:44:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:14 smithi098 bash[23790]: cluster 2024-04-02T04:44:13.863452+0000 mon.a (mon.0) 751 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-04-02T04:44:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:14 smithi069 bash[21528]: cluster 2024-04-02T04:44:13.195688+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 331 B/s wr, 0 op/s 2024-04-02T04:44:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:14 smithi069 bash[21528]: cluster 2024-04-02T04:44:13.863452+0000 mon.a (mon.0) 751 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-04-02T04:44:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:14 smithi069 bash[28904]: cluster 2024-04-02T04:44:13.195688+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 331 B/s wr, 0 op/s 2024-04-02T04:44:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:14 smithi069 bash[28904]: cluster 2024-04-02T04:44:13.863452+0000 mon.a (mon.0) 751 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- 172.21.15.98:0/758381128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd4102d70 msgr2=0x7f5cd4109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- 172.21.15.98:0/758381128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd4102d70 0x7f5cd4109620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00b0d0 tx=0x7f5ccc00b3e0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- 172.21.15.98:0/758381128 shutdown_connections 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- 172.21.15.98:0/758381128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd4102d70 0x7f5cd4109620 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- 172.21.15.98:0/758381128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41011d0 0x7f5cd41015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- 172.21.15.98:0/758381128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd41023d0 0x7f5cd4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- 172.21.15.98:0/758381128 >> 172.21.15.98:0/758381128 conn(0x7f5cd40fc770 msgr2=0x7f5cd40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:16.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- 172.21.15.98:0/758381128 shutdown_connections 2024-04-02T04:44:16.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- 172.21.15.98:0/758381128 wait complete. 2024-04-02T04:44:16.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 librados: starting msgr at 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 librados: starting objecter 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 Processor -- start 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- start start 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 librados: setting wanted keys 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 librados: calling monclient init 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 0x7f5cd41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41023d0 0x7f5cd4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd4102d70 0x7f5cd419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd410c310 con 0x7f5cd41023d0 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd410c190 con 0x7f5cd41011d0 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cdbaee700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd410c490 con 0x7f5cd4102d70 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cd988a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 0x7f5cd41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:16.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cd988a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 0x7f5cd41979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43310/0 (socket says 172.21.15.98:43310) 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cd988a700 1 -- 172.21.15.98:0/2223592195 learned_addr learned my addr 172.21.15.98:0/2223592195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.364+0000 7f5cda08b700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd4102d70 0x7f5cd419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd988a700 1 -- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd4102d70 msgr2=0x7f5cd419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd988a700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd4102d70 0x7f5cd419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd988a700 1 -- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41023d0 msgr2=0x7f5cd4197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd9089700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41023d0 0x7f5cd4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd988a700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41023d0 0x7f5cd4197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd988a700 1 -- 172.21.15.98:0/2223592195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd419f650 con 0x7f5cd41011d0 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd9089700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41023d0 0x7f5cd4197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd988a700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 0x7f5cd41979f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000c990 tx=0x7f5cd000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 10 client.?.objecter ms_handle_connect 0x7f5cd41011d0 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0007b00 con 0x7f5cd41011d0 2024-04-02T04:44:16.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cdbaee700 1 -- 172.21.15.98:0/2223592195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd419f940 con 0x7f5cd41011d0 2024-04-02T04:44:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f5cd0007ca0 con 0x7f5cd41011d0 2024-04-02T04:44:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0018660 con 0x7f5cd41011d0 2024-04-02T04:44:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cdbaee700 10 client.24397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:44:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cdbaee700 1 -- 172.21.15.98:0/2223592195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd419fe80 con 0x7f5cd41011d0 2024-04-02T04:44:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cdbaee700 1 librados: init done 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cbc005120 con 0x7f5cd41011d0 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 91774+0+0 (secure 0 0 0) 0x7f5cd000f450 con 0x7f5cd41011d0 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f5cc006f4f0 0x7f5cc00719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 23..91) ==== 7148+0+0 (secure 0 0 0) 0x7f5cd008dd40 con 0x7f5cd41011d0 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 10 client.24397.objecter ms_dispatch 0x7f5cd4001550 osd_map(91..91 src has 23..91) 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 3 client.24397.objecter handle_osd_map got epochs [91,91] > 0 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 3 client.24397.objecter handle_osd_map decoding full epoch 91 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 20 client.24397.objecter dump_active .. 0 homeless 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd9089700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f5cc006f4f0 0x7f5cc00719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5ccaffd700 10 client.24397.objecter ms_handle_connect 0x7f5cc006f4f0 2024-04-02T04:44:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.368+0000 7f5cd9089700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f5cc006f4f0 0x7f5cc00719b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005fd0 tx=0x7f5cc4005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.372+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5cd0014030 con 0x7f5cd41011d0 2024-04-02T04:44:16.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.596+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.98", "placement": "1;smithi098=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7f5cbc000c90 con 0x7f5cc006f4f0 2024-04-02T04:44:16.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.612+0000 7f5ccaffd700 1 -- 172.21.15.98:0/2223592195 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (secure 0 0 0) 0x7f5cbc000c90 con 0x7f5cc006f4f0 2024-04-02T04:44:16.620 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled iscsi.datapool update... 2024-04-02T04:44:16.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 10 librados: watch_flush enter 2024-04-02T04:44:16.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 10 librados: watch_flush exit 2024-04-02T04:44:16.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 20 client.24397.objecter shutdown clearing up homeless session... 2024-04-02T04:44:16.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 10 client.24397.objecter successfully canceled tick 2024-04-02T04:44:16.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f5cc006f4f0 msgr2=0x7f5cc00719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f5cc006f4f0 0x7f5cc00719b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005fd0 tx=0x7f5cc4005dc0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 msgr2=0x7f5cd41979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 0x7f5cd41979f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000c990 tx=0x7f5cd000cd50 comp rx=0 tx=0).stop 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 shutdown_connections 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f5cc006f4f0 0x7f5cc00719b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5cd4102d70 0x7f5cd419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cd41011d0 0x7f5cd41979f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 --2- 172.21.15.98:0/2223592195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cd41023d0 0x7f5cd4197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 >> 172.21.15.98:0/2223592195 conn(0x7f5cd40fc770 msgr2=0x7f5cd40fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 shutdown_connections 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 -- 172.21.15.98:0/2223592195 wait complete. 2024-04-02T04:44:16.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:16.616+0000 7f5cc8ff9700 1 librados: shutdown 2024-04-02T04:44:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:16 smithi098 bash[23790]: cluster 2024-04-02T04:44:15.196676+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 143 B/s wr, 0 op/s 2024-04-02T04:44:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:16 smithi098 bash[23790]: audit 2024-04-02T04:44:16.622516+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:16 smithi098 bash[23790]: audit 2024-04-02T04:44:16.624603+0000 mon.a (mon.0) 753 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:16 smithi069 bash[21528]: cluster 2024-04-02T04:44:15.196676+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 143 B/s wr, 0 op/s 2024-04-02T04:44:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:16 smithi069 bash[21528]: audit 2024-04-02T04:44:16.622516+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:16 smithi069 bash[21528]: audit 2024-04-02T04:44:16.624603+0000 mon.a (mon.0) 753 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:16 smithi069 bash[28904]: cluster 2024-04-02T04:44:15.196676+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 143 B/s wr, 0 op/s 2024-04-02T04:44:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:16 smithi069 bash[28904]: audit 2024-04-02T04:44:16.622516+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:16 smithi069 bash[28904]: audit 2024-04-02T04:44:16.624603+0000 mon.a (mon.0) 753 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:17.361 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-04-02T04:44:17.361 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:44:17.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-02T04:44:17.375 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:44:17.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-02T04:44:17.389 DEBUG:teuthology.orchestra.run.smithi098:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service 2024-04-02T04:44:17.434 INFO:tasks.cephadm:Adding prometheus.a on smithi098 2024-04-02T04:44:17.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi098=a' 2024-04-02T04:44:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:16.607741+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.98", "placement": "1;smithi098=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: cephadm 2024-04-02T04:44:16.614204+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi098=iscsi.a;count:1 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:17.251063+0000 mon.a (mon.0) 754 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:17.252994+0000 mon.a (mon.0) 755 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:17.263133+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:17.270305+0000 mon.a (mon.0) 757 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:17.273817+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-02T04:44:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:17 smithi098 bash[23790]: audit 2024-04-02T04:44:17.298358+0000 mon.a (mon.0) 759 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:18.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:16.607741+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.98", "placement": "1;smithi098=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:18.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: cephadm 2024-04-02T04:44:16.614204+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi098=iscsi.a;count:1 2024-04-02T04:44:18.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:17.251063+0000 mon.a (mon.0) 754 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:17.252994+0000 mon.a (mon.0) 755 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:17.263133+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:17.270305+0000 mon.a (mon.0) 757 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:17.273817+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[21528]: audit 2024-04-02T04:44:17.298358+0000 mon.a (mon.0) 759 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:16.607741+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.98", "placement": "1;smithi098=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: cephadm 2024-04-02T04:44:16.614204+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi098=iscsi.a;count:1 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:17.251063+0000 mon.a (mon.0) 754 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:17.252994+0000 mon.a (mon.0) 755 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:17.263133+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:17.270305+0000 mon.a (mon.0) 757 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-02T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:17.273817+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-02T04:44:18.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:17 smithi069 bash[28904]: audit 2024-04-02T04:44:17.298358+0000 mon.a (mon.0) 759 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:19.076 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:18 smithi098 bash[23790]: cluster 2024-04-02T04:44:17.197396+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 122 B/s wr, 0 op/s 2024-04-02T04:44:19.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:18 smithi098 bash[23790]: cephadm 2024-04-02T04:44:17.300317+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi098 2024-04-02T04:44:19.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:18 smithi098 bash[23790]: cluster 2024-04-02T04:44:18.865555+0000 mon.a (mon.0) 760 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-04-02T04:44:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:18 smithi069 bash[21528]: cluster 2024-04-02T04:44:17.197396+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 122 B/s wr, 0 op/s 2024-04-02T04:44:19.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:18 smithi069 bash[21528]: cephadm 2024-04-02T04:44:17.300317+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi098 2024-04-02T04:44:19.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:18 smithi069 bash[21528]: cluster 2024-04-02T04:44:18.865555+0000 mon.a (mon.0) 760 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-04-02T04:44:19.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:18 smithi069 bash[28904]: cluster 2024-04-02T04:44:17.197396+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 122 B/s wr, 0 op/s 2024-04-02T04:44:19.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:18 smithi069 bash[28904]: cephadm 2024-04-02T04:44:17.300317+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi098 2024-04-02T04:44:19.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:18 smithi069 bash[28904]: cluster 2024-04-02T04:44:18.865555+0000 mon.a (mon.0) 760 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-04-02T04:44:19.974 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:19.975 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:44:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:19.975 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:44:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:19.975 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:44:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:19.976 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:44:19 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:20.246 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:44:20 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:20.247 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:44:20 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:20.247 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:20.247 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:20 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:20.248 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:44:20 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:20.248 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:44:20 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 bash[23790]: cluster 2024-04-02T04:44:19.198240+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-04-02T04:44:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 bash[23790]: audit 2024-04-02T04:44:20.307921+0000 mon.a (mon.0) 761 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 bash[23790]: audit 2024-04-02T04:44:20.318946+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 bash[23790]: audit 2024-04-02T04:44:20.328550+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 bash[23790]: audit 2024-04-02T04:44:20.345274+0000 mon.a (mon.0) 764 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:20 smithi098 bash[23790]: audit 2024-04-02T04:44:20.410608+0000 mon.a (mon.0) 765 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[21528]: cluster 2024-04-02T04:44:19.198240+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[21528]: audit 2024-04-02T04:44:20.307921+0000 mon.a (mon.0) 761 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[21528]: audit 2024-04-02T04:44:20.318946+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[21528]: audit 2024-04-02T04:44:20.328550+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[21528]: audit 2024-04-02T04:44:20.345274+0000 mon.a (mon.0) 764 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[21528]: audit 2024-04-02T04:44:20.410608+0000 mon.a (mon.0) 765 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[28904]: cluster 2024-04-02T04:44:19.198240+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[28904]: audit 2024-04-02T04:44:20.307921+0000 mon.a (mon.0) 761 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[28904]: audit 2024-04-02T04:44:20.318946+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[28904]: audit 2024-04-02T04:44:20.328550+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[28904]: audit 2024-04-02T04:44:20.345274+0000 mon.a (mon.0) 764 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:21.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:20 smithi069 bash[28904]: audit 2024-04-02T04:44:20.410608+0000 mon.a (mon.0) 765 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:22.161 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:21 smithi098 bash[23790]: cephadm 2024-04-02T04:44:20.330347+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-04-02T04:44:22.213 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:44:22.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:21 smithi069 bash[21528]: cephadm 2024-04-02T04:44:20.330347+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-04-02T04:44:22.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:21 smithi069 bash[28904]: cephadm 2024-04-02T04:44:20.330347+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-04-02T04:44:22.590 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[48594]: debug Processing osd blocklist entries for this node 2024-04-02T04:44:22.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[48594]: debug Reading the configuration object to update local LIO configuration 2024-04-02T04:44:22.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[48594]: debug Configuration does not have an entry for this host(smithi098.front.sepia.ceph.com) - nothing to define to LIO 2024-04-02T04:44:22.895 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[48594]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-04-02T04:44:23.211 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[23790]: cluster 2024-04-02T04:44:21.199285+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:23.211 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[23790]: audit 2024-04-02T04:44:22.195625+0000 mon.a (mon.0) 766 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:23.211 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:22 smithi098 bash[23790]: audit 2024-04-02T04:44:22.575842+0000 mon.b (mon.1) 28 : audit 0 from='client.? 172.21.15.98:0/708014349' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-02T04:44:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:22 smithi069 bash[21528]: cluster 2024-04-02T04:44:21.199285+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:22 smithi069 bash[21528]: audit 2024-04-02T04:44:22.195625+0000 mon.a (mon.0) 766 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:23.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:22 smithi069 bash[21528]: audit 2024-04-02T04:44:22.575842+0000 mon.b (mon.1) 28 : audit 0 from='client.? 172.21.15.98:0/708014349' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-02T04:44:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:22 smithi069 bash[28904]: cluster 2024-04-02T04:44:21.199285+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:22 smithi069 bash[28904]: audit 2024-04-02T04:44:22.195625+0000 mon.a (mon.0) 766 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:22 smithi069 bash[28904]: audit 2024-04-02T04:44:22.575842+0000 mon.b (mon.1) 28 : audit 0 from='client.? 172.21.15.98:0/708014349' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-02T04:44:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:24 smithi098 bash[23790]: cluster 2024-04-02T04:44:23.200723+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:24 smithi098 bash[23790]: cluster 2024-04-02T04:44:23.209769+0000 mon.a (mon.0) 767 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-04-02T04:44:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:24 smithi098 bash[23790]: cluster 2024-04-02T04:44:23.867157+0000 mon.a (mon.0) 768 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-04-02T04:44:24.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:24 smithi069 bash[21528]: cluster 2024-04-02T04:44:23.200723+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:24.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:24 smithi069 bash[21528]: cluster 2024-04-02T04:44:23.209769+0000 mon.a (mon.0) 767 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-04-02T04:44:24.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:24 smithi069 bash[21528]: cluster 2024-04-02T04:44:23.867157+0000 mon.a (mon.0) 768 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-04-02T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:24 smithi069 bash[28904]: cluster 2024-04-02T04:44:23.200723+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:24 smithi069 bash[28904]: cluster 2024-04-02T04:44:23.209769+0000 mon.a (mon.0) 767 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-04-02T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:24 smithi069 bash[28904]: cluster 2024-04-02T04:44:23.867157+0000 mon.a (mon.0) 768 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-04-02T04:44:25.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 -- 172.21.15.98:0/344019930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c103e80 msgr2=0x7f976c10a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:25.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 --2- 172.21.15.98:0/344019930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c103e80 0x7f976c10a730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f976400b0d0 tx=0x7f976400b3e0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 -- 172.21.15.98:0/344019930 shutdown_connections 2024-04-02T04:44:25.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 --2- 172.21.15.98:0/344019930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c103e80 0x7f976c10a730 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 --2- 172.21.15.98:0/344019930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976c1022e0 0x7f976c102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 --2- 172.21.15.98:0/344019930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f976c1034e0 0x7f976c103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 -- 172.21.15.98:0/344019930 >> 172.21.15.98:0/344019930 conn(0x7f976c0fd8a0 msgr2=0x7f976c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 -- 172.21.15.98:0/344019930 shutdown_connections 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.231+0000 7f9771ba3700 1 -- 172.21.15.98:0/344019930 wait complete. 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 librados: starting msgr at 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 librados: starting objecter 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 Processor -- start 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- start start 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 librados: setting wanted keys 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 librados: calling monclient init 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976c1022e0 0x7f976c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 0x7f976c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f976c103e80 0x7f976c19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f976c10d480 con 0x7f976c1022e0 2024-04-02T04:44:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f976c10d300 con 0x7f976c1034e0 2024-04-02T04:44:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f976c10d600 con 0x7f976c103e80 2024-04-02T04:44:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 0x7f976c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 0x7f976c198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34766/0 (socket says 172.21.15.98:34766) 2024-04-02T04:44:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 -- 172.21.15.98:0/1680076992 learned_addr learned my addr 172.21.15.98:0/1680076992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:44:25.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976bfff700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f976c103e80 0x7f976c19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:25.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976b7fe700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976c1022e0 0x7f976c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:25.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 -- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f976c103e80 msgr2=0x7f976c19f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:25.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f976c103e80 0x7f976c19f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 -- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976c1022e0 msgr2=0x7f976c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:25.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976c1022e0 0x7f976c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 -- 172.21.15.98:0/1680076992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c19f820 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f976affd700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 0x7f976c198100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f976000b530 tx=0x7f976000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9768ff9700 10 client.?.objecter ms_handle_connect 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9768ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760015020 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9760011cb0 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976c19fb10 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760010ce0 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 10 client.24433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976c1a0050 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 librados: init done 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976c04ff00 con 0x7f976c1034e0 2024-04-02T04:44:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.235+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 91825+0+0 (secure 0 0 0) 0x7f9760018020 con 0x7f976c1034e0 2024-04-02T04:44:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f975406f570 0x7f9754071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 23..93) ==== 7120+0+0 (secure 0 0 0) 0x7f976008da80 con 0x7f976c1034e0 2024-04-02T04:44:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 10 client.24433.objecter ms_dispatch 0x7f976c001550 osd_map(93..93 src has 23..93) 2024-04-02T04:44:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 3 client.24433.objecter handle_osd_map got epochs [93,93] > 0 2024-04-02T04:44:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 3 client.24433.objecter handle_osd_map decoding full epoch 93 2024-04-02T04:44:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 20 client.24433.objecter dump_active .. 0 homeless 2024-04-02T04:44:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f976b7fe700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f975406f570 0x7f9754071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f9768ff9700 10 client.24433.objecter ms_handle_connect 0x7f975406f570 2024-04-02T04:44:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.239+0000 7f976b7fe700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f975406f570 0x7f9754071a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f975c006fd0 tx=0x7f975c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.243+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f976005b340 con 0x7f976c1034e0 2024-04-02T04:44:25.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.463+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}) -- 0x7f976c062680 con 0x7f975406f570 2024-04-02T04:44:25.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.475+0000 7f9768ff9700 1 -- 172.21.15.98:0/1680076992 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (secure 0 0 0) 0x7f976c062680 con 0x7f975406f570 2024-04-02T04:44:25.480 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled prometheus update... 2024-04-02T04:44:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 10 librados: watch_flush enter 2024-04-02T04:44:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 10 librados: watch_flush exit 2024-04-02T04:44:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 20 client.24433.objecter shutdown clearing up homeless session... 2024-04-02T04:44:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 10 client.24433.objecter successfully canceled tick 2024-04-02T04:44:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f975406f570 msgr2=0x7f9754071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f975406f570 0x7f9754071a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f975c006fd0 tx=0x7f975c008040 comp rx=0 tx=0).stop 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 msgr2=0x7f976c198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 0x7f976c198100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f976000b530 tx=0x7f976000b8f0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 shutdown_connections 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f975406f570 0x7f9754071a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f976c103e80 0x7f976c19f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f976c1022e0 0x7f976c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 --2- 172.21.15.98:0/1680076992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f976c1034e0 0x7f976c198100 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 >> 172.21.15.98:0/1680076992 conn(0x7f976c0fd8a0 msgr2=0x7f976c100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 shutdown_connections 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 -- 172.21.15.98:0/1680076992 wait complete. 2024-04-02T04:44:25.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:25.479+0000 7f9771ba3700 1 librados: shutdown 2024-04-02T04:44:26.235 DEBUG:teuthology.orchestra.run.smithi098:prometheus.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@prometheus.a.service 2024-04-02T04:44:26.237 INFO:tasks.cephadm:Adding node-exporter.a on smithi069 2024-04-02T04:44:26.237 INFO:tasks.cephadm:Adding node-exporter.b on smithi098 2024-04-02T04:44:26.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi069=a;smithi098=b' 2024-04-02T04:44:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:26 smithi098 bash[23790]: cluster 2024-04-02T04:44:25.201725+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:26 smithi098 bash[23790]: audit 2024-04-02T04:44:25.471025+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:26 smithi098 bash[23790]: cephadm 2024-04-02T04:44:25.474612+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi098=a;count:1 2024-04-02T04:44:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:26 smithi098 bash[23790]: audit 2024-04-02T04:44:25.482518+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:26.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[21528]: cluster 2024-04-02T04:44:25.201725+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:26.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[21528]: audit 2024-04-02T04:44:25.471025+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:26.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[21528]: cephadm 2024-04-02T04:44:25.474612+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi098=a;count:1 2024-04-02T04:44:26.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[21528]: audit 2024-04-02T04:44:25.482518+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:26.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[28904]: cluster 2024-04-02T04:44:25.201725+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:26.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[28904]: audit 2024-04-02T04:44:25.471025+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:26.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[28904]: cephadm 2024-04-02T04:44:25.474612+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi098=a;count:1 2024-04-02T04:44:26.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:26 smithi069 bash[28904]: audit 2024-04-02T04:44:25.482518+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.209 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: cluster 2024-04-02T04:44:27.202862+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 490 B/s rd, 122 B/s wr, 1 op/s 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: audit 2024-04-02T04:44:27.212842+0000 mon.a (mon.0) 770 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: audit 2024-04-02T04:44:27.224914+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: audit 2024-04-02T04:44:27.227397+0000 mon.a (mon.0) 772 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: audit 2024-04-02T04:44:27.229225+0000 mon.a (mon.0) 773 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: audit 2024-04-02T04:44:27.239293+0000 mon.a (mon.0) 774 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:28 smithi098 bash[23790]: cephadm 2024-04-02T04:44:27.534884+0000 mgr.y (mgr.14152) 366 : cephadm 1 Deploying daemon prometheus.a on smithi098 2024-04-02T04:44:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: cluster 2024-04-02T04:44:27.202862+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 490 B/s rd, 122 B/s wr, 1 op/s 2024-04-02T04:44:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: audit 2024-04-02T04:44:27.212842+0000 mon.a (mon.0) 770 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: audit 2024-04-02T04:44:27.224914+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: audit 2024-04-02T04:44:27.227397+0000 mon.a (mon.0) 772 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: audit 2024-04-02T04:44:27.229225+0000 mon.a (mon.0) 773 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: audit 2024-04-02T04:44:27.239293+0000 mon.a (mon.0) 774 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[21528]: cephadm 2024-04-02T04:44:27.534884+0000 mgr.y (mgr.14152) 366 : cephadm 1 Deploying daemon prometheus.a on smithi098 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: cluster 2024-04-02T04:44:27.202862+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 490 B/s rd, 122 B/s wr, 1 op/s 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: audit 2024-04-02T04:44:27.212842+0000 mon.a (mon.0) 770 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: audit 2024-04-02T04:44:27.224914+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: audit 2024-04-02T04:44:27.227397+0000 mon.a (mon.0) 772 : audit 0 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: audit 2024-04-02T04:44:27.229225+0000 mon.a (mon.0) 773 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: audit 2024-04-02T04:44:27.239293+0000 mon.a (mon.0) 774 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:28 smithi069 bash[28904]: cephadm 2024-04-02T04:44:27.534884+0000 mgr.y (mgr.14152) 366 : cephadm 1 Deploying daemon prometheus.a on smithi098 2024-04-02T04:44:30.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:29 smithi098 bash[23790]: cluster 2024-04-02T04:44:28.871010+0000 mon.a (mon.0) 775 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-04-02T04:44:30.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:29 smithi069 bash[21528]: cluster 2024-04-02T04:44:28.871010+0000 mon.a (mon.0) 775 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-04-02T04:44:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:29 smithi069 bash[28904]: cluster 2024-04-02T04:44:28.871010+0000 mon.a (mon.0) 775 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-04-02T04:44:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:44:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:30 smithi098 bash[23790]: cluster 2024-04-02T04:44:29.203800+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-04-02T04:44:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:30 smithi069 bash[21528]: cluster 2024-04-02T04:44:29.203800+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-04-02T04:44:31.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:30 smithi069 bash[28904]: cluster 2024-04-02T04:44:29.203800+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-04-02T04:44:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:31 smithi098 bash[23790]: cluster 2024-04-02T04:44:31.136225+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:31 smithi098 bash[23790]: cluster 2024-04-02T04:44:31.136349+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:31 smithi069 bash[21528]: cluster 2024-04-02T04:44:31.136225+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:31 smithi069 bash[21528]: cluster 2024-04-02T04:44:31.136349+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:32.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:31 smithi069 bash[28904]: cluster 2024-04-02T04:44:31.136225+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:32.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:31 smithi069 bash[28904]: cluster 2024-04-02T04:44:31.136349+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:32.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:32 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:44:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:32 smithi098 bash[23790]: cluster 2024-04-02T04:44:31.204973+0000 mgr.y (mgr.14152) 368 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-02T04:44:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:32 smithi098 bash[23790]: cluster 2024-04-02T04:44:31.898548+0000 mon.a (mon.0) 776 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-04-02T04:44:33.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:32 smithi069 bash[21528]: cluster 2024-04-02T04:44:31.204973+0000 mgr.y (mgr.14152) 368 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-02T04:44:33.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:32 smithi069 bash[21528]: cluster 2024-04-02T04:44:31.898548+0000 mon.a (mon.0) 776 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-04-02T04:44:33.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:32 smithi069 bash[28904]: cluster 2024-04-02T04:44:31.204973+0000 mgr.y (mgr.14152) 368 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-02T04:44:33.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:32 smithi069 bash[28904]: cluster 2024-04-02T04:44:31.898548+0000 mon.a (mon.0) 776 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-04-02T04:44:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:33 smithi098 bash[23790]: audit 2024-04-02T04:44:32.270917+0000 mgr.y (mgr.14152) 369 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:33 smithi098 bash[23790]: cluster 2024-04-02T04:44:33.492816+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:33 smithi098 bash[23790]: cluster 2024-04-02T04:44:33.492962+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:33 smithi098 bash[23790]: cluster 2024-04-02T04:44:33.873528+0000 mon.a (mon.0) 777 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-04-02T04:44:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[21528]: audit 2024-04-02T04:44:32.270917+0000 mgr.y (mgr.14152) 369 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[21528]: cluster 2024-04-02T04:44:33.492816+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[21528]: cluster 2024-04-02T04:44:33.492962+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[21528]: cluster 2024-04-02T04:44:33.873528+0000 mon.a (mon.0) 777 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-04-02T04:44:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[28904]: audit 2024-04-02T04:44:32.270917+0000 mgr.y (mgr.14152) 369 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[28904]: cluster 2024-04-02T04:44:33.492816+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[28904]: cluster 2024-04-02T04:44:33.492962+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:33 smithi069 bash[28904]: cluster 2024-04-02T04:44:33.873528+0000 mon.a (mon.0) 777 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-04-02T04:44:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:34 smithi098 bash[23790]: cluster 2024-04-02T04:44:33.205932+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-02T04:44:35.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:34 smithi069 bash[21528]: cluster 2024-04-02T04:44:33.205932+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-02T04:44:35.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:34 smithi069 bash[28904]: cluster 2024-04-02T04:44:33.205932+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-04-02T04:44:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/1297687063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b410a480 msgr2=0x7f24b4110d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/1297687063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b410a480 0x7f24b4110d30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00b0d0 tx=0x7f24ac00b3e0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/1297687063 shutdown_connections 2024-04-02T04:44:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/1297687063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b410a480 0x7f24b4110d30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/1297687063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b41088e0 0x7f24b4108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/1297687063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b4109ae0 0x7f24b4109f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/1297687063 >> 172.21.15.98:0/1297687063 conn(0x7f24b40778d0 msgr2=0x7f24b4079cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/1297687063 shutdown_connections 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/1297687063 wait complete. 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 librados: starting msgr at 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 librados: starting objecter 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 Processor -- start 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- start start 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 librados: setting wanted keys 2024-04-02T04:44:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 librados: calling monclient init 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 0x7f24b419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b4109ae0 0x7f24b419e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b410a480 0x7f24b41a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24b41139c0 con 0x7f24b4109ae0 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24b4113840 con 0x7f24b41088e0 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f24b4113b40 con 0x7f24b410a480 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 0x7f24b419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 0x7f24b419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39434/0 (socket says 172.21.15.98:39434) 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 -- 172.21.15.98:0/3319995732 learned_addr learned my addr 172.21.15.98:0/3319995732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b2ffd700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b4109ae0 0x7f24b419e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b3fff700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b410a480 0x7f24b41a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 -- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b410a480 msgr2=0x7f24b41a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b410a480 0x7f24b41a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 -- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b4109ae0 msgr2=0x7f24b419e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b4109ae0 0x7f24b419e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 -- 172.21.15.98:0/3319995732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b41a5c40 con 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b37fe700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 0x7f24b419dfe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f24a400c990 tx=0x7f24a400cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b0ff9700 10 client.?.objecter ms_handle_connect 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b0ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4007b00 con 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f24a4007ca0 con 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b41a5f30 con 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4018660 con 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 10 client.24439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b41a6390 con 0x7f24b41088e0 2024-04-02T04:44:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 librados: init done 2024-04-02T04:44:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.079+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b404ff00 con 0x7f24b41088e0 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 91825+0+0 (secure 0 0 0) 0x7f24a400f450 con 0x7f24b41088e0 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f249c06f570 0x7f249c071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 23..97) ==== 7120+0+0 (secure 0 0 0) 0x7f24a408ddb0 con 0x7f24b41088e0 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 10 client.24439.objecter ms_dispatch 0x7f24b4001550 osd_map(97..97 src has 23..97) 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 3 client.24439.objecter handle_osd_map got epochs [97,97] > 0 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 3 client.24439.objecter handle_osd_map decoding full epoch 97 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 20 client.24439.objecter dump_active .. 0 homeless 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b2ffd700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f249c06f570 0x7f249c071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b0ff9700 10 client.24439.objecter ms_handle_connect 0x7f249c06f570 2024-04-02T04:44:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.083+0000 7f24b2ffd700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f249c06f570 0x7f249c071a30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f24a8005fd0 tx=0x7f24a800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.087+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f24a4014030 con 0x7f24b41088e0 2024-04-02T04:44:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:36 smithi098 bash[23790]: cluster 2024-04-02T04:44:35.206896+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 0 op/s 2024-04-02T04:44:37.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:36 smithi069 bash[21528]: cluster 2024-04-02T04:44:35.206896+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 0 op/s 2024-04-02T04:44:37.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:36 smithi069 bash[28904]: cluster 2024-04-02T04:44:35.206896+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 0 op/s 2024-04-02T04:44:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.311+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi098=b", "target": ["mon-mgr", ""]}) -- 0x7f24b4062680 con 0x7f249c06f570 2024-04-02T04:44:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.327+0000 7f24b0ff9700 1 -- 172.21.15.98:0/3319995732 <== mgr.14152 v2:172.21.15.69:6800/1560062353 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (secure 0 0 0) 0x7f24b4062680 con 0x7f249c06f570 2024-04-02T04:44:37.332 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled node-exporter update... 2024-04-02T04:44:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 10 librados: watch_flush enter 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 10 librados: watch_flush exit 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 20 client.24439.objecter shutdown clearing up homeless session... 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 10 client.24439.objecter successfully canceled tick 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f249c06f570 msgr2=0x7f249c071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f249c06f570 0x7f249c071a30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f24a8005fd0 tx=0x7f24a800a560 comp rx=0 tx=0).stop 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 msgr2=0x7f24b419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 0x7f24b419dfe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f24a400c990 tx=0x7f24a400cd50 comp rx=0 tx=0).stop 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 shutdown_connections 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f249c06f570 0x7f249c071a30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b410a480 0x7f24b41a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b41088e0 0x7f24b419dfe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 --2- 172.21.15.98:0/3319995732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b4109ae0 0x7f24b419e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 >> 172.21.15.98:0/3319995732 conn(0x7f24b40778d0 msgr2=0x7f24b40793d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 shutdown_connections 2024-04-02T04:44:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 -- 172.21.15.98:0/3319995732 wait complete. 2024-04-02T04:44:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:37.331+0000 7f24b9ac0700 1 librados: shutdown 2024-04-02T04:44:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:37 smithi098 bash[23790]: cluster 2024-04-02T04:44:36.598129+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:37 smithi098 bash[23790]: cluster 2024-04-02T04:44:36.598242+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:37 smithi098 bash[23790]: cluster 2024-04-02T04:44:36.923056+0000 mon.a (mon.0) 778 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-04-02T04:44:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:37 smithi098 bash[23790]: audit 2024-04-02T04:44:37.334566+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:38.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[21528]: cluster 2024-04-02T04:44:36.598129+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:38.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[21528]: cluster 2024-04-02T04:44:36.598242+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:38.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[21528]: cluster 2024-04-02T04:44:36.923056+0000 mon.a (mon.0) 778 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-04-02T04:44:38.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[21528]: audit 2024-04-02T04:44:37.334566+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:38.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[28904]: cluster 2024-04-02T04:44:36.598129+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:38.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[28904]: cluster 2024-04-02T04:44:36.598242+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:38.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[28904]: cluster 2024-04-02T04:44:36.923056+0000 mon.a (mon.0) 778 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-04-02T04:44:38.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:37 smithi069 bash[28904]: audit 2024-04-02T04:44:37.334566+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:38.966 DEBUG:teuthology.orchestra.run.smithi069:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@node-exporter.a.service 2024-04-02T04:44:38.968 DEBUG:teuthology.orchestra.run.smithi098:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@node-exporter.b.service 2024-04-02T04:44:38.971 INFO:tasks.cephadm:Adding alertmanager.a on smithi069 2024-04-02T04:44:38.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi069=a' 2024-04-02T04:44:39.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:38 smithi098 bash[23790]: cluster 2024-04-02T04:44:37.207662+0000 mgr.y (mgr.14152) 372 : cluster 0 pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:44:39.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:38 smithi098 bash[23790]: audit 2024-04-02T04:44:37.321111+0000 mgr.y (mgr.14152) 373 : audit 0 from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi098=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:39.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:38 smithi098 bash[23790]: cephadm 2024-04-02T04:44:37.324880+0000 mgr.y (mgr.14152) 374 : cephadm 1 Saving service node-exporter spec with placement smithi069=a;smithi098=b;count:2 2024-04-02T04:44:39.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:38 smithi098 bash[23790]: cluster 2024-04-02T04:44:38.874310+0000 mon.a (mon.0) 780 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-04-02T04:44:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[21528]: cluster 2024-04-02T04:44:37.207662+0000 mgr.y (mgr.14152) 372 : cluster 0 pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:44:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[21528]: audit 2024-04-02T04:44:37.321111+0000 mgr.y (mgr.14152) 373 : audit 0 from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi098=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[21528]: cephadm 2024-04-02T04:44:37.324880+0000 mgr.y (mgr.14152) 374 : cephadm 1 Saving service node-exporter spec with placement smithi069=a;smithi098=b;count:2 2024-04-02T04:44:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[21528]: cluster 2024-04-02T04:44:38.874310+0000 mon.a (mon.0) 780 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-04-02T04:44:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[28904]: cluster 2024-04-02T04:44:37.207662+0000 mgr.y (mgr.14152) 372 : cluster 0 pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:44:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[28904]: audit 2024-04-02T04:44:37.321111+0000 mgr.y (mgr.14152) 373 : audit 0 from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi098=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[28904]: cephadm 2024-04-02T04:44:37.324880+0000 mgr.y (mgr.14152) 374 : cephadm 1 Saving service node-exporter spec with placement smithi069=a;smithi098=b;count:2 2024-04-02T04:44:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:38 smithi069 bash[28904]: cluster 2024-04-02T04:44:38.874310+0000 mon.a (mon.0) 780 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-04-02T04:44:41.217 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:40 smithi069 bash[21528]: cluster 2024-04-02T04:44:39.208585+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:44:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:40 smithi098 bash[23790]: cluster 2024-04-02T04:44:39.208585+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:44:41.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:41 smithi069 bash[28904]: cluster 2024-04-02T04:44:39.208585+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:44:42.224 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.224 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.224 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.225 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.225 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.226 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.226 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.226 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.226 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.724 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.725 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.725 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.725 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:42 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:44:42.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.727 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:44:42.727 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:42 smithi098 systemd[1]: Started Ceph prometheus.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:44:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 bash[23790]: cluster 2024-04-02T04:44:41.209669+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T04:44:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 bash[23790]: audit 2024-04-02T04:44:42.499583+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 bash[23790]: audit 2024-04-02T04:44:42.510467+0000 mon.a (mon.0) 782 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 bash[23790]: audit 2024-04-02T04:44:42.549513+0000 mon.a (mon.0) 783 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:42 smithi098 bash[23790]: audit 2024-04-02T04:44:42.557898+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-02T04:44:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[21528]: cluster 2024-04-02T04:44:41.209669+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T04:44:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[21528]: audit 2024-04-02T04:44:42.499583+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[21528]: audit 2024-04-02T04:44:42.510467+0000 mon.a (mon.0) 782 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[21528]: audit 2024-04-02T04:44:42.549513+0000 mon.a (mon.0) 783 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[21528]: audit 2024-04-02T04:44:42.557898+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-02T04:44:43.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[28904]: cluster 2024-04-02T04:44:41.209669+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T04:44:43.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[28904]: audit 2024-04-02T04:44:42.499583+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[28904]: audit 2024-04-02T04:44:42.510467+0000 mon.a (mon.0) 782 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[28904]: audit 2024-04-02T04:44:42.549513+0000 mon.a (mon.0) 783 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' 2024-04-02T04:44:43.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:42 smithi069 bash[28904]: audit 2024-04-02T04:44:42.557898+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-02T04:44:43.760 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[24743]: ignoring --setuser ceph since I am not root 2024-04-02T04:44:43.760 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[24743]: ignoring --setgroup ceph since I am not root 2024-04-02T04:44:43.760 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[24743]: debug 2024-04-02T04:44:43.611+0000 7f7661b92700 1 -- 172.21.15.98:0/2574604376 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a6f3fb65a0 con 0x55a6f4d34c00 2024-04-02T04:44:43.761 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[24743]: debug 2024-04-02T04:44:43.755+0000 7f766b97c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T04:44:43.761 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.620Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-02T04:44:43.761 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.620Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-02T04:44:43.761 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.620Z caller=main.go:570 level=info host_details="(Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 smithi098 (none))" 2024-04-02T04:44:43.761 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.620Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-02T04:44:43.761 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.620Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-02T04:44:43.761 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.621Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.621Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.622Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.622Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.624Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.624Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.675µs 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.624Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.627Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.627Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.588µs wal_replay_duration=2.285336ms wbl_replay_duration=136ns total_replay_duration=2.392286ms 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.628Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.628Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.628Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-02T04:44:43.762 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.629Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-02T04:44:43.763 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.629Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-02T04:44:43.763 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.648Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.148954ms db_storage=862ns remote_storage=1.247µs web_handler=384ns query_engine=744ns scrape=251.259µs scrape_sd=76.923µs notify=1.298µs notify_sd=2.439µs rules=19.499822ms tracing=7.637µs 2024-04-02T04:44:43.763 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.648Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-02T04:44:43.763 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[49697]: ts=2024-04-02T04:44:43.648Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-02T04:44:43.947 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[21782]: ignoring --setuser ceph since I am not root 2024-04-02T04:44:43.947 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[21782]: ignoring --setgroup ceph since I am not root 2024-04-02T04:44:43.947 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[21782]: debug 2024-04-02T04:44:43.760+0000 7f431aff0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T04:44:44.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[23790]: audit 2024-04-02T04:44:42.281655+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:44.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[23790]: audit 2024-04-02T04:44:43.551415+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-02T04:44:44.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:43 smithi098 bash[23790]: cluster 2024-04-02T04:44:43.565296+0000 mon.a (mon.0) 786 : cluster 0 mgrmap e17: y(active, since 9m), standbys: x 2024-04-02T04:44:44.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[21528]: audit 2024-04-02T04:44:42.281655+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:44.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[21528]: audit 2024-04-02T04:44:43.551415+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-02T04:44:44.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[21528]: cluster 2024-04-02T04:44:43.565296+0000 mon.a (mon.0) 786 : cluster 0 mgrmap e17: y(active, since 9m), standbys: x 2024-04-02T04:44:44.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[28904]: audit 2024-04-02T04:44:42.281655+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:44.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[28904]: audit 2024-04-02T04:44:43.551415+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14152 172.21.15.69:0/2142001522' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-02T04:44:44.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:43 smithi069 bash[28904]: cluster 2024-04-02T04:44:43.565296+0000 mon.a (mon.0) 786 : cluster 0 mgrmap e17: y(active, since 9m), standbys: x 2024-04-02T04:44:45.663 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:45 smithi098 bash[24743]: debug 2024-04-02T04:44:45.315+0000 7f766b97c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T04:44:45.663 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:45 smithi098 bash[24743]: debug 2024-04-02T04:44:45.411+0000 7f766b97c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T04:44:45.680 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:45 smithi069 bash[21782]: debug 2024-04-02T04:44:45.332+0000 7f431aff0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T04:44:45.680 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:45 smithi069 bash[21782]: debug 2024-04-02T04:44:45.428+0000 7f431aff0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T04:44:45.936 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:44:46.029 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:45 smithi069 bash[21782]: debug 2024-04-02T04:44:45.676+0000 7f431aff0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T04:44:46.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:45 smithi098 bash[24743]: debug 2024-04-02T04:44:45.659+0000 7f766b97c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T04:44:46.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:46 smithi098 bash[24743]: debug 2024-04-02T04:44:46.455+0000 7f766b97c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T04:44:46.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:46 smithi098 bash[24743]: debug 2024-04-02T04:44:46.567+0000 7f766b97c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T04:44:46.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:46 smithi098 bash[24743]: debug 2024-04-02T04:44:46.671+0000 7f766b97c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T04:44:46.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:46 smithi069 bash[21782]: debug 2024-04-02T04:44:46.476+0000 7f431aff0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T04:44:46.779 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:46 smithi069 bash[21782]: debug 2024-04-02T04:44:46.588+0000 7f431aff0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T04:44:46.779 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:46 smithi069 bash[21782]: debug 2024-04-02T04:44:46.692+0000 7f431aff0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T04:44:47.255 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:47 smithi098 bash[24743]: debug 2024-04-02T04:44:47.031+0000 7f766b97c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T04:44:47.255 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:47 smithi098 bash[24743]: debug 2024-04-02T04:44:47.147+0000 7f766b97c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T04:44:47.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:47 smithi069 bash[21782]: debug 2024-04-02T04:44:47.048+0000 7f431aff0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T04:44:47.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:47 smithi069 bash[21782]: debug 2024-04-02T04:44:47.164+0000 7f431aff0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T04:44:47.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:47 smithi069 bash[21782]: debug 2024-04-02T04:44:47.268+0000 7f431aff0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T04:44:47.611 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:47 smithi098 bash[24743]: debug 2024-04-02T04:44:47.251+0000 7f766b97c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T04:44:47.933 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:47 smithi098 bash[24743]: debug 2024-04-02T04:44:47.607+0000 7f766b97c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T04:44:47.933 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:47 smithi098 bash[24743]: debug 2024-04-02T04:44:47.847+0000 7f766b97c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T04:44:48.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:47 smithi069 bash[21782]: debug 2024-04-02T04:44:47.628+0000 7f431aff0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T04:44:48.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:47 smithi069 bash[21782]: debug 2024-04-02T04:44:47.868+0000 7f431aff0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T04:44:48.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:48 smithi098 bash[24743]: debug 2024-04-02T04:44:48.107+0000 7f766b97c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T04:44:48.503 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:48 smithi098 bash[24743]: debug 2024-04-02T04:44:48.223+0000 7f766b97c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T04:44:48.525 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:48 smithi069 bash[21782]: debug 2024-04-02T04:44:48.128+0000 7f431aff0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T04:44:48.525 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:48 smithi069 bash[21782]: debug 2024-04-02T04:44:48.244+0000 7f431aff0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T04:44:48.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:48 smithi069 bash[21782]: debug 2024-04-02T04:44:48.524+0000 7f431aff0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T04:44:48.921 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:48 smithi098 bash[24743]: debug 2024-04-02T04:44:48.499+0000 7f766b97c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/2538135590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108010 msgr2=0x7f45d010a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- 172.21.15.98:0/2538135590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108010 0x7f45d010a420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f45c800b0a0 tx=0x7f45c800b3b0 comp rx=0 tx=0).stop 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/2538135590 shutdown_connections 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- 172.21.15.98:0/2538135590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108010 0x7f45d010a420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- 172.21.15.98:0/2538135590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45d0102db0 0x7f45d01051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- 172.21.15.98:0/2538135590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45d01056e0 0x7f45d0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/2538135590 >> 172.21.15.98:0/2538135590 conn(0x7f45d00fc760 msgr2=0x7f45d00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/2538135590 shutdown_connections 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/2538135590 wait complete. 2024-04-02T04:44:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 librados: starting msgr at 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 librados: starting objecter 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 Processor -- start 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- start start 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 librados: setting wanted keys 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 librados: calling monclient init 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45d0102db0 0x7f45d0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d01056e0 0x7f45d0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45d0108010 0x7f45d019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d390 con 0x7f45d0102db0 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d210 con 0x7f45d01056e0 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f45d010d510 con 0x7f45d0108010 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d01056e0 0x7f45d0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:49.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d01056e0 0x7f45d0197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46198/0 (socket says 172.21.15.98:46198) 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 -- 172.21.15.98:0/1456329458 learned_addr learned my addr 172.21.15.98:0/1456329458 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45d0102db0 0x7f45d0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d4ed1700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45d0108010 0x7f45d019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45d0108010 msgr2=0x7f45d019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45d0108010 0x7f45d019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45d0102db0 msgr2=0x7f45d0197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45d0102db0 0x7f45d0197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 -- 172.21.15.98:0/1456329458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d019f650 con 0x7f45d01056e0 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45c77fe700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d01056e0 0x7f45d0197f80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45b800d950 tx=0x7f45b800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cdffb700 10 client.?.objecter ms_handle_connect 0x7f45d01056e0 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cdffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:44:49.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8016070 con 0x7f45d01056e0 2024-04-02T04:44:49.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d019f940 con 0x7f45d01056e0 2024-04-02T04:44:49.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f45b8009dc0 con 0x7f45d01056e0 2024-04-02T04:44:49.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b800f3c0 con 0x7f45d01056e0 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 10 client.24454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d019fe80 con 0x7f45d01056e0 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 librados: init done 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d004ff00 con 0x7f45d01056e0 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.131+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 91839+0+0 (secure 0 0 0) 0x7f45b8019070 con 0x7f45d01056e0 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cdffb700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 0x7f45bc071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 23..98) ==== 7120+0+0 (secure 0 0 0) 0x7f45b808da90 con 0x7f45d01056e0 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cdffb700 10 client.24454.objecter ms_dispatch 0x7f45d0001550 osd_map(98..98 src has 23..98) 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cdffb700 3 client.24454.objecter handle_osd_map got epochs [98,98] > 0 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cdffb700 3 client.24454.objecter handle_osd_map decoding full epoch 98 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cffff700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 msgr2=0x7f45bc071a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/1560062353 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 0x7f45bc071a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T04:44:49.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.135+0000 7f45cdffb700 20 client.24454.objecter dump_active .. 0 homeless 2024-04-02T04:44:49.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.139+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f45b805b140 con 0x7f45d01056e0 2024-04-02T04:44:49.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:48 smithi098 bash[24743]: debug 2024-04-02T04:44:48.915+0000 7f766b97c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:44:49.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:49 smithi098 bash[24743]: debug 2024-04-02T04:44:49.163+0000 7f766b97c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T04:44:49.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:48 smithi069 bash[21782]: debug 2024-04-02T04:44:48.940+0000 7f431aff0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:44:49.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:49 smithi069 bash[21782]: debug 2024-04-02T04:44:49.180+0000 7f431aff0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T04:44:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.339+0000 7f45cffff700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 msgr2=0x7f45bc071a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/1560062353 2024-04-02T04:44:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.339+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 0x7f45bc071a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T04:44:49.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.359+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 --> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}) -- 0x7f45d0062680 con 0x7f45bc06f5c0 2024-04-02T04:44:49.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:49 smithi098 bash[24743]: debug 2024-04-02T04:44:49.283+0000 7f766b97c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T04:44:49.724 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:49 smithi098 bash[24743]: debug 2024-04-02T04:44:49.527+0000 7f766b97c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:44:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.743+0000 7f45cffff700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 msgr2=0x7f45bc071a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/1560062353 2024-04-02T04:44:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:49.743+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 0x7f45bc071a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T04:44:49.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:49 smithi069 bash[21782]: debug 2024-04-02T04:44:49.300+0000 7f431aff0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T04:44:49.779 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:49 smithi069 bash[21782]: debug 2024-04-02T04:44:49.548+0000 7f431aff0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T04:44:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:50.547+0000 7f45cffff700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 msgr2=0x7f45bc071a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/1560062353 2024-04-02T04:44:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:50.547+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 0x7f45bc071a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T04:44:50.936 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:50 smithi098 bash[24743]: debug 2024-04-02T04:44:50.527+0000 7f766b97c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T04:44:50.936 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:50 smithi098 bash[24743]: debug 2024-04-02T04:44:50.643+0000 7f766b97c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T04:44:50.974 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:50 smithi069 bash[21782]: debug 2024-04-02T04:44:50.560+0000 7f431aff0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T04:44:50.975 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:50 smithi069 bash[21782]: debug 2024-04-02T04:44:50.676+0000 7f431aff0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T04:44:51.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:50 smithi098 bash[24743]: debug 2024-04-02T04:44:50.931+0000 7f766b97c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T04:44:51.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: debug 2024-04-02T04:44:51.043+0000 7f766b97c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T04:44:51.277 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:50 smithi069 bash[21782]: debug 2024-04-02T04:44:50.972+0000 7f431aff0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T04:44:51.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: debug 2024-04-02T04:44:51.088+0000 7f431aff0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T04:44:51.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:51.687+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 18) ==== 91839+0+0 (secure 0 0 0) 0x7f45b80570b0 con 0x7f45d01056e0 2024-04-02T04:44:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:51.711+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 19) ==== 91456+0+0 (secure 0 0 0) 0x7f45b8058780 con 0x7f45d01056e0 2024-04-02T04:44:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:51.715+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 msgr2=0x7f45bc071a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:44:51.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:51.715+0000 7f45cdffb700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1560062353,v1:172.21.15.69:6801/1560062353] conn(0x7f45bc06f5c0 0x7f45bc071a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: debug 2024-04-02T04:44:51.487+0000 7f766b97c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: debug 2024-04-02T04:44:51.607+0000 7f766b97c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: [02/Apr/2024:04:44:51] ENGINE Bus STARTING 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.635145+0000 mon.a (mon.0) 787 : cluster 0 Standby manager daemon x restarted 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.635253+0000 mon.a (mon.0) 788 : cluster 0 Standby manager daemon x started 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: audit 2024-04-02T04:44:51.640309+0000 mon.a (mon.0) 789 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: audit 2024-04-02T04:44:51.641205+0000 mon.a (mon.0) 790 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: audit 2024-04-02T04:44:51.644269+0000 mon.a (mon.0) 791 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: audit 2024-04-02T04:44:51.646324+0000 mon.a (mon.0) 792 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:51.743 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.678519+0000 mon.a (mon.0) 793 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-04-02T04:44:51.823 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: debug 2024-04-02T04:44:51.540+0000 7f431aff0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: debug 2024-04-02T04:44:51.664+0000 7f431aff0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.635145+0000 mon.a (mon.0) 787 : cluster 0 Standby manager daemon x restarted 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.635253+0000 mon.a (mon.0) 788 : cluster 0 Standby manager daemon x started 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: audit 2024-04-02T04:44:51.640309+0000 mon.a (mon.0) 789 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: audit 2024-04-02T04:44:51.641205+0000 mon.a (mon.0) 790 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: audit 2024-04-02T04:44:51.644269+0000 mon.a (mon.0) 791 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: audit 2024-04-02T04:44:51.646324+0000 mon.a (mon.0) 792 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:51.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.678519+0000 mon.a (mon.0) 793 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.635145+0000 mon.a (mon.0) 787 : cluster 0 Standby manager daemon x restarted 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.635253+0000 mon.a (mon.0) 788 : cluster 0 Standby manager daemon x started 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: audit 2024-04-02T04:44:51.640309+0000 mon.a (mon.0) 789 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: audit 2024-04-02T04:44:51.641205+0000 mon.a (mon.0) 790 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: audit 2024-04-02T04:44:51.644269+0000 mon.a (mon.0) 791 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: audit 2024-04-02T04:44:51.646324+0000 mon.a (mon.0) 792 : audit 0 from='mgr.? 172.21.15.98:0/1542199577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:51.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.678519+0000 mon.a (mon.0) 793 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-04-02T04:44:52.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: CherryPy Checker: 2024-04-02T04:44:52.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: The Application mounted at '' has an empty config. 2024-04-02T04:44:52.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: [02/Apr/2024:04:44:51] ENGINE Serving on http://:::9283 2024-04-02T04:44:52.224 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:44:51 smithi098 bash[24743]: [02/Apr/2024:04:44:51] ENGINE Bus STARTED 2024-04-02T04:44:52.277 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: [02/Apr/2024:04:44:51] ENGINE Bus STARTING 2024-04-02T04:44:52.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: CherryPy Checker: 2024-04-02T04:44:52.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: The Application mounted at '' has an empty config. 2024-04-02T04:44:52.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: [02/Apr/2024:04:44:51] ENGINE Serving on http://:::9283 2024-04-02T04:44:52.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:44:51 smithi069 bash[21782]: [02/Apr/2024:04:44:51] ENGINE Bus STARTED 2024-04-02T04:44:52.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.727+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mon.1 v2:172.21.15.98:3300/0 9 ==== mgrmap(e 20) ==== 91583+0+0 (secure 0 0 0) 0x7f45b8057980 con 0x7f45d01056e0 2024-04-02T04:44:52.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.727+0000 7f45cdffb700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45bc078d40 0x7f45bc07b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:44:52.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.727+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}) -- 0x7f45b808b710 con 0x7f45bc078d40 2024-04-02T04:44:52.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.731+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45bc078d40 0x7f45bc07b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:44:52.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.731+0000 7f45cdffb700 10 client.24454.objecter ms_handle_connect 0x7f45bc078d40 2024-04-02T04:44:52.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.731+0000 7f45cffff700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45bc078d40 0x7f45bc07b130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45c000a6a0 tx=0x7f45c000a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:44:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.747+0000 7f45cdffb700 1 -- 172.21.15.98:0/1456329458 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (secure 0 0 0) 0x7f45b808b710 con 0x7f45bc078d40 2024-04-02T04:44:52.750 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled alertmanager update... 2024-04-02T04:44:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 10 librados: watch_flush enter 2024-04-02T04:44:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 10 librados: watch_flush exit 2024-04-02T04:44:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 20 client.24454.objecter shutdown clearing up homeless session... 2024-04-02T04:44:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 10 client.24454.objecter successfully canceled tick 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45bc078d40 msgr2=0x7f45bc07b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45bc078d40 0x7f45bc07b130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45c000a6a0 tx=0x7f45c000a380 comp rx=0 tx=0).stop 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d01056e0 msgr2=0x7f45d0197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 --2- 172.21.15.98:0/1456329458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d01056e0 0x7f45d0197f80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45b800d950 tx=0x7f45b800dc60 comp rx=0 tx=0).stop 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d4ed1700 1 -- 172.21.15.98:0/1456329458 reap_dead start 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 shutdown_connections 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 >> 172.21.15.98:0/1456329458 conn(0x7f45d00fc760 msgr2=0x7f45d0103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:44:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 shutdown_connections 2024-04-02T04:44:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 -- 172.21.15.98:0/1456329458 wait complete. 2024-04-02T04:44:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:44:52.751+0000 7f45d6934700 1 librados: shutdown 2024-04-02T04:44:52.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:44:52.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.617581+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:52.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.617709+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.698723+0000 mon.a (mon.0) 794 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.699242+0000 mon.a (mon.0) 795 : cluster 1 Active manager daemon y restarted 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.700053+0000 mon.a (mon.0) 796 : cluster 1 Activating manager daemon y 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.719123+0000 mon.b (mon.1) 29 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.719302+0000 mon.b (mon.1) 30 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.719478+0000 mon.b (mon.1) 31 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.721348+0000 mon.b (mon.1) 32 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.721486+0000 mon.b (mon.1) 33 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.723538+0000 mon.b (mon.1) 34 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.723741+0000 mon.b (mon.1) 35 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.724303+0000 mon.b (mon.1) 36 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:44:52.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.724508+0000 mon.b (mon.1) 37 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.724742+0000 mon.b (mon.1) 38 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.724957+0000 mon.b (mon.1) 39 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.725177+0000 mon.b (mon.1) 40 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.725399+0000 mon.b (mon.1) 41 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.725627+0000 mon.b (mon.1) 42 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.725839+0000 mon.b (mon.1) 43 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.726060+0000 mon.b (mon.1) 44 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.726637+0000 mon.a (mon.0) 797 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.727462+0000 mon.a (mon.0) 798 : cluster 0 mgrmap e19: y(active, starting, since 0.0276496s), standbys: x 2024-04-02T04:44:52.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:51.755177+0000 mon.a (mon.0) 799 : cluster 1 Manager daemon y is now available 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.784422+0000 mon.b (mon.1) 45 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.817704+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.825806+0000 mon.b (mon.1) 46 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.834527+0000 mon.b (mon.1) 47 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.840105+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.854430+0000 mon.b (mon.1) 48 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:51.860079+0000 mon.a (mon.0) 802 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cephadm 2024-04-02T04:44:52.282137+0000 mgr.y (mgr.14538) 1 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Bus STARTING 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cephadm 2024-04-02T04:44:52.398622+0000 mgr.y (mgr.14538) 2 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Serving on https://172.21.15.69:7150 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:52.466150+0000 osd.2 (osd.2) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cluster 2024-04-02T04:44:52.466232+0000 osd.2 (osd.2) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cephadm 2024-04-02T04:44:52.500082+0000 mgr.y (mgr.14538) 3 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Serving on http://172.21.15.69:8765 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: cephadm 2024-04-02T04:44:52.500291+0000 mgr.y (mgr.14538) 4 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Bus STARTED 2024-04-02T04:44:52.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:52.640292+0000 mon.a (mon.0) 803 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:52.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:52 smithi098 bash[23790]: audit 2024-04-02T04:44:52.649358+0000 mon.a (mon.0) 804 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.617581+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:53.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.617709+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:53.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.698723+0000 mon.a (mon.0) 794 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.699242+0000 mon.a (mon.0) 795 : cluster 1 Active manager daemon y restarted 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.700053+0000 mon.a (mon.0) 796 : cluster 1 Activating manager daemon y 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.719123+0000 mon.b (mon.1) 29 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.719302+0000 mon.b (mon.1) 30 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.719478+0000 mon.b (mon.1) 31 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.721348+0000 mon.b (mon.1) 32 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.721486+0000 mon.b (mon.1) 33 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.723538+0000 mon.b (mon.1) 34 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.723741+0000 mon.b (mon.1) 35 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.724303+0000 mon.b (mon.1) 36 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.724508+0000 mon.b (mon.1) 37 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:44:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.724742+0000 mon.b (mon.1) 38 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.724957+0000 mon.b (mon.1) 39 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.725177+0000 mon.b (mon.1) 40 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.725399+0000 mon.b (mon.1) 41 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.725627+0000 mon.b (mon.1) 42 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.725839+0000 mon.b (mon.1) 43 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.617581+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.617709+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.698723+0000 mon.a (mon.0) 794 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-04-02T04:44:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.699242+0000 mon.a (mon.0) 795 : cluster 1 Active manager daemon y restarted 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.700053+0000 mon.a (mon.0) 796 : cluster 1 Activating manager daemon y 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.719123+0000 mon.b (mon.1) 29 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.719302+0000 mon.b (mon.1) 30 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.719478+0000 mon.b (mon.1) 31 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.721348+0000 mon.b (mon.1) 32 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.721486+0000 mon.b (mon.1) 33 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.723538+0000 mon.b (mon.1) 34 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.723741+0000 mon.b (mon.1) 35 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.724303+0000 mon.b (mon.1) 36 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.724508+0000 mon.b (mon.1) 37 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.724742+0000 mon.b (mon.1) 38 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:44:53.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.724957+0000 mon.b (mon.1) 39 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.725177+0000 mon.b (mon.1) 40 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.725399+0000 mon.b (mon.1) 41 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.725627+0000 mon.b (mon.1) 42 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.725839+0000 mon.b (mon.1) 43 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.726060+0000 mon.b (mon.1) 44 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.726637+0000 mon.a (mon.0) 797 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.727462+0000 mon.a (mon.0) 798 : cluster 0 mgrmap e19: y(active, starting, since 0.0276496s), standbys: x 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:51.755177+0000 mon.a (mon.0) 799 : cluster 1 Manager daemon y is now available 2024-04-02T04:44:53.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.784422+0000 mon.b (mon.1) 45 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.817704+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.825806+0000 mon.b (mon.1) 46 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.834527+0000 mon.b (mon.1) 47 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.840105+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.854430+0000 mon.b (mon.1) 48 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:51.860079+0000 mon.a (mon.0) 802 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cephadm 2024-04-02T04:44:52.282137+0000 mgr.y (mgr.14538) 1 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Bus STARTING 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cephadm 2024-04-02T04:44:52.398622+0000 mgr.y (mgr.14538) 2 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Serving on https://172.21.15.69:7150 2024-04-02T04:44:53.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:52.466150+0000 osd.2 (osd.2) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:53.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cluster 2024-04-02T04:44:52.466232+0000 osd.2 (osd.2) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:53.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cephadm 2024-04-02T04:44:52.500082+0000 mgr.y (mgr.14538) 3 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Serving on http://172.21.15.69:8765 2024-04-02T04:44:53.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: cephadm 2024-04-02T04:44:52.500291+0000 mgr.y (mgr.14538) 4 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Bus STARTED 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:52.640292+0000 mon.a (mon.0) 803 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[21528]: audit 2024-04-02T04:44:52.649358+0000 mon.a (mon.0) 804 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.726060+0000 mon.b (mon.1) 44 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.726637+0000 mon.a (mon.0) 797 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.727462+0000 mon.a (mon.0) 798 : cluster 0 mgrmap e19: y(active, starting, since 0.0276496s), standbys: x 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:51.755177+0000 mon.a (mon.0) 799 : cluster 1 Manager daemon y is now available 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.784422+0000 mon.b (mon.1) 45 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:53.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.817704+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.825806+0000 mon.b (mon.1) 46 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.834527+0000 mon.b (mon.1) 47 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.840105+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.854430+0000 mon.b (mon.1) 48 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:51.860079+0000 mon.a (mon.0) 802 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cephadm 2024-04-02T04:44:52.282137+0000 mgr.y (mgr.14538) 1 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Bus STARTING 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cephadm 2024-04-02T04:44:52.398622+0000 mgr.y (mgr.14538) 2 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Serving on https://172.21.15.69:7150 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:52.466150+0000 osd.2 (osd.2) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cluster 2024-04-02T04:44:52.466232+0000 osd.2 (osd.2) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cephadm 2024-04-02T04:44:52.500082+0000 mgr.y (mgr.14538) 3 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Serving on http://172.21.15.69:8765 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: cephadm 2024-04-02T04:44:52.500291+0000 mgr.y (mgr.14538) 4 : cephadm 1 [02/Apr/2024:04:44:52] ENGINE Bus STARTED 2024-04-02T04:44:53.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:52.640292+0000 mon.a (mon.0) 803 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:52 smithi069 bash[28904]: audit 2024-04-02T04:44:52.649358+0000 mon.a (mon.0) 804 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.618 DEBUG:teuthology.orchestra.run.smithi069:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@alertmanager.a.service 2024-04-02T04:44:53.621 INFO:tasks.cephadm:Adding grafana.a on smithi098 2024-04-02T04:44:53.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph orch apply grafana '1;smithi098=a' 2024-04-02T04:44:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: cluster 2024-04-02T04:44:52.732044+0000 mon.a (mon.0) 805 : cluster 0 mgrmap e20: y(active, since 1.03222s), standbys: x 2024-04-02T04:44:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: audit 2024-04-02T04:44:52.737726+0000 mgr.y (mgr.14538) 5 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: audit 2024-04-02T04:44:52.740795+0000 mgr.y (mgr.14538) 6 : audit 0 from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:53.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: cluster 2024-04-02T04:44:52.741402+0000 mgr.y (mgr.14538) 7 : cluster 0 pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:53.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: cephadm 2024-04-02T04:44:52.745312+0000 mgr.y (mgr.14538) 8 : cephadm 1 Saving service alertmanager spec with placement smithi069=a;count:1 2024-04-02T04:44:53.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: audit 2024-04-02T04:44:52.751334+0000 mon.a (mon.0) 806 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:53.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:53 smithi098 bash[23790]: cluster 2024-04-02T04:44:52.824612+0000 mon.a (mon.0) 807 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-04-02T04:44:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: cluster 2024-04-02T04:44:52.732044+0000 mon.a (mon.0) 805 : cluster 0 mgrmap e20: y(active, since 1.03222s), standbys: x 2024-04-02T04:44:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: audit 2024-04-02T04:44:52.737726+0000 mgr.y (mgr.14538) 5 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: audit 2024-04-02T04:44:52.740795+0000 mgr.y (mgr.14538) 6 : audit 0 from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: cluster 2024-04-02T04:44:52.741402+0000 mgr.y (mgr.14538) 7 : cluster 0 pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: cephadm 2024-04-02T04:44:52.745312+0000 mgr.y (mgr.14538) 8 : cephadm 1 Saving service alertmanager spec with placement smithi069=a;count:1 2024-04-02T04:44:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: audit 2024-04-02T04:44:52.751334+0000 mon.a (mon.0) 806 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[21528]: cluster 2024-04-02T04:44:52.824612+0000 mon.a (mon.0) 807 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: cluster 2024-04-02T04:44:52.732044+0000 mon.a (mon.0) 805 : cluster 0 mgrmap e20: y(active, since 1.03222s), standbys: x 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: audit 2024-04-02T04:44:52.737726+0000 mgr.y (mgr.14538) 5 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: audit 2024-04-02T04:44:52.740795+0000 mgr.y (mgr.14538) 6 : audit 0 from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: cluster 2024-04-02T04:44:52.741402+0000 mgr.y (mgr.14538) 7 : cluster 0 pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: cephadm 2024-04-02T04:44:52.745312+0000 mgr.y (mgr.14538) 8 : cephadm 1 Saving service alertmanager spec with placement smithi069=a;count:1 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: audit 2024-04-02T04:44:52.751334+0000 mon.a (mon.0) 806 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:53 smithi069 bash[28904]: cluster 2024-04-02T04:44:52.824612+0000 mon.a (mon.0) 807 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-04-02T04:44:55.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:54 smithi098 bash[23790]: cluster 2024-04-02T04:44:53.729171+0000 mgr.y (mgr.14538) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:55.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:54 smithi098 bash[23790]: cluster 2024-04-02T04:44:53.836043+0000 mon.a (mon.0) 808 : cluster 0 mgrmap e21: y(active, since 2s), standbys: x 2024-04-02T04:44:55.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:54 smithi098 bash[23790]: cluster 2024-04-02T04:44:53.875837+0000 mon.a (mon.0) 809 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-04-02T04:44:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:54 smithi069 bash[21528]: cluster 2024-04-02T04:44:53.729171+0000 mgr.y (mgr.14538) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:54 smithi069 bash[21528]: cluster 2024-04-02T04:44:53.836043+0000 mon.a (mon.0) 808 : cluster 0 mgrmap e21: y(active, since 2s), standbys: x 2024-04-02T04:44:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:54 smithi069 bash[21528]: cluster 2024-04-02T04:44:53.875837+0000 mon.a (mon.0) 809 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-04-02T04:44:55.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:54 smithi069 bash[28904]: cluster 2024-04-02T04:44:53.729171+0000 mgr.y (mgr.14538) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:55.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:54 smithi069 bash[28904]: cluster 2024-04-02T04:44:53.836043+0000 mon.a (mon.0) 808 : cluster 0 mgrmap e21: y(active, since 2s), standbys: x 2024-04-02T04:44:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:54 smithi069 bash[28904]: cluster 2024-04-02T04:44:53.875837+0000 mon.a (mon.0) 809 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-04-02T04:44:56.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:55 smithi098 bash[23790]: cluster 2024-04-02T04:44:55.226408+0000 osd.5 (osd.5) 9 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:44:56.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:55 smithi098 bash[23790]: cluster 2024-04-02T04:44:55.228064+0000 osd.5 (osd.5) 10 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:44:56.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:55 smithi098 bash[23790]: cluster 2024-04-02T04:44:55.515645+0000 osd.3 (osd.3) 9 : cluster 0 2.1b scrub starts 2024-04-02T04:44:56.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:55 smithi098 bash[23790]: cluster 2024-04-02T04:44:55.518216+0000 osd.3 (osd.3) 10 : cluster 0 2.1b scrub ok 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[21528]: cluster 2024-04-02T04:44:55.226408+0000 osd.5 (osd.5) 9 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[21528]: cluster 2024-04-02T04:44:55.228064+0000 osd.5 (osd.5) 10 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[21528]: cluster 2024-04-02T04:44:55.515645+0000 osd.3 (osd.3) 9 : cluster 0 2.1b scrub starts 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[21528]: cluster 2024-04-02T04:44:55.518216+0000 osd.3 (osd.3) 10 : cluster 0 2.1b scrub ok 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[28904]: cluster 2024-04-02T04:44:55.226408+0000 osd.5 (osd.5) 9 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[28904]: cluster 2024-04-02T04:44:55.228064+0000 osd.5 (osd.5) 10 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:44:56.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[28904]: cluster 2024-04-02T04:44:55.515645+0000 osd.3 (osd.3) 9 : cluster 0 2.1b scrub starts 2024-04-02T04:44:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:55 smithi069 bash[28904]: cluster 2024-04-02T04:44:55.518216+0000 osd.3 (osd.3) 10 : cluster 0 2.1b scrub ok 2024-04-02T04:44:57.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:56 smithi098 bash[23790]: cluster 2024-04-02T04:44:55.730029+0000 mgr.y (mgr.14538) 10 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:57.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:56 smithi098 bash[23790]: cluster 2024-04-02T04:44:55.854837+0000 mon.a (mon.0) 810 : cluster 0 mgrmap e22: y(active, since 4s), standbys: x 2024-04-02T04:44:57.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:56 smithi069 bash[21528]: cluster 2024-04-02T04:44:55.730029+0000 mgr.y (mgr.14538) 10 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:57.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:56 smithi069 bash[21528]: cluster 2024-04-02T04:44:55.854837+0000 mon.a (mon.0) 810 : cluster 0 mgrmap e22: y(active, since 4s), standbys: x 2024-04-02T04:44:57.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:56 smithi069 bash[28904]: cluster 2024-04-02T04:44:55.730029+0000 mgr.y (mgr.14538) 10 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:44:57.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:56 smithi069 bash[28904]: cluster 2024-04-02T04:44:55.854837+0000 mon.a (mon.0) 810 : cluster 0 mgrmap e22: y(active, since 4s), standbys: x 2024-04-02T04:44:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:44:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: cluster 2024-04-02T04:44:57.730691+0000 mgr.y (mgr.14538) 11 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:44:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: audit 2024-04-02T04:44:57.862951+0000 mon.a (mon.0) 811 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: audit 2024-04-02T04:44:57.870348+0000 mon.a (mon.0) 812 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: cluster 2024-04-02T04:44:57.984198+0000 osd.2 (osd.2) 13 : cluster 0 2.1 deep-scrub starts 2024-04-02T04:44:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: cluster 2024-04-02T04:44:57.985231+0000 osd.2 (osd.2) 14 : cluster 0 2.1 deep-scrub ok 2024-04-02T04:44:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: audit 2024-04-02T04:44:58.035485+0000 mon.a (mon.0) 813 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:58 smithi098 bash[23790]: audit 2024-04-02T04:44:58.047950+0000 mon.a (mon.0) 814 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: cluster 2024-04-02T04:44:57.730691+0000 mgr.y (mgr.14538) 11 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: audit 2024-04-02T04:44:57.862951+0000 mon.a (mon.0) 811 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: audit 2024-04-02T04:44:57.870348+0000 mon.a (mon.0) 812 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: cluster 2024-04-02T04:44:57.984198+0000 osd.2 (osd.2) 13 : cluster 0 2.1 deep-scrub starts 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: cluster 2024-04-02T04:44:57.985231+0000 osd.2 (osd.2) 14 : cluster 0 2.1 deep-scrub ok 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: audit 2024-04-02T04:44:58.035485+0000 mon.a (mon.0) 813 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[21528]: audit 2024-04-02T04:44:58.047950+0000 mon.a (mon.0) 814 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: cluster 2024-04-02T04:44:57.730691+0000 mgr.y (mgr.14538) 11 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: audit 2024-04-02T04:44:57.862951+0000 mon.a (mon.0) 811 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: audit 2024-04-02T04:44:57.870348+0000 mon.a (mon.0) 812 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: cluster 2024-04-02T04:44:57.984198+0000 osd.2 (osd.2) 13 : cluster 0 2.1 deep-scrub starts 2024-04-02T04:44:59.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: cluster 2024-04-02T04:44:57.985231+0000 osd.2 (osd.2) 14 : cluster 0 2.1 deep-scrub ok 2024-04-02T04:44:59.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: audit 2024-04-02T04:44:58.035485+0000 mon.a (mon.0) 813 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:44:59.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:58 smithi069 bash[28904]: audit 2024-04-02T04:44:58.047950+0000 mon.a (mon.0) 814 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.349501+0000 osd.1 (osd.1) 15 : cluster 0 2.d scrub starts 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.351061+0000 osd.1 (osd.1) 16 : cluster 0 2.d scrub ok 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.698158+0000 osd.4 (osd.4) 9 : cluster 0 2.f scrub starts 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.700059+0000 osd.0 (osd.0) 15 : cluster 0 2.1f scrub starts 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.701316+0000 osd.4 (osd.4) 10 : cluster 0 2.f scrub ok 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.702079+0000 osd.0 (osd.0) 16 : cluster 0 2.1f scrub ok 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.793212+0000 osd.6 (osd.6) 9 : cluster 0 2.17 deep-scrub starts 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cluster 2024-04-02T04:44:58.795137+0000 osd.6 (osd.6) 10 : cluster 0 2.17 deep-scrub ok 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.925955+0000 mon.a (mon.0) 815 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.936229+0000 mon.b (mon.1) 49 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.938249+0000 mon.a (mon.0) 816 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.939302+0000 mon.b (mon.1) 50 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.941942+0000 mon.a (mon.0) 817 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.942230+0000 mon.b (mon.1) 51 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.944931+0000 mon.b (mon.1) 52 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.945303+0000 mon.a (mon.0) 818 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.948069+0000 mon.a (mon.0) 819 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.950785+0000 mon.a (mon.0) 820 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:58.952392+0000 mgr.y (mgr.14538) 12 : cephadm 1 Adjusting osd_memory_target on smithi098 to 3804M 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:58.960560+0000 mon.a (mon.0) 821 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.105050+0000 mon.a (mon.0) 822 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.119748+0000 mon.b (mon.1) 53 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.120215+0000 mon.a (mon.0) 823 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.122890+0000 mon.b (mon.1) 54 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.125725+0000 mon.a (mon.0) 824 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.125919+0000 mon.b (mon.1) 55 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.128808+0000 mon.a (mon.0) 825 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.128869+0000 mon.b (mon.1) 56 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.131891+0000 mon.a (mon.0) 826 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.134801+0000 mon.a (mon.0) 827 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.136579+0000 mgr.y (mgr.14538) 13 : cephadm 1 Adjusting osd_memory_target on smithi069 to 3804M 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.143943+0000 mon.a (mon.0) 828 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.145569+0000 mon.b (mon.1) 57 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.147548+0000 mon.b (mon.1) 58 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.155329+0000 mgr.y (mgr.14538) 14 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.155705+0000 mgr.y (mgr.14538) 15 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.370320+0000 mgr.y (mgr.14538) 16 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.373035+0000 mgr.y (mgr.14538) 17 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.534687+0000 mgr.y (mgr.14538) 18 : cephadm 1 Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.538692+0000 mgr.y (mgr.14538) 19 : cephadm 1 Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:00.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.871248+0000 mon.a (mon.0) 829 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.878180+0000 mon.a (mon.0) 830 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.883777+0000 mon.a (mon.0) 831 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.892684+0000 mon.a (mon.0) 832 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:44:59 smithi098 bash[23790]: audit 2024-04-02T04:44:59.900757+0000 mon.a (mon.0) 833 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.349501+0000 osd.1 (osd.1) 15 : cluster 0 2.d scrub starts 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.351061+0000 osd.1 (osd.1) 16 : cluster 0 2.d scrub ok 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.698158+0000 osd.4 (osd.4) 9 : cluster 0 2.f scrub starts 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.700059+0000 osd.0 (osd.0) 15 : cluster 0 2.1f scrub starts 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.701316+0000 osd.4 (osd.4) 10 : cluster 0 2.f scrub ok 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.702079+0000 osd.0 (osd.0) 16 : cluster 0 2.1f scrub ok 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.793212+0000 osd.6 (osd.6) 9 : cluster 0 2.17 deep-scrub starts 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cluster 2024-04-02T04:44:58.795137+0000 osd.6 (osd.6) 10 : cluster 0 2.17 deep-scrub ok 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.925955+0000 mon.a (mon.0) 815 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.936229+0000 mon.b (mon.1) 49 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.938249+0000 mon.a (mon.0) 816 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.939302+0000 mon.b (mon.1) 50 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.941942+0000 mon.a (mon.0) 817 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.942230+0000 mon.b (mon.1) 51 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.944931+0000 mon.b (mon.1) 52 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.945303+0000 mon.a (mon.0) 818 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.948069+0000 mon.a (mon.0) 819 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.950785+0000 mon.a (mon.0) 820 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:58.952392+0000 mgr.y (mgr.14538) 12 : cephadm 1 Adjusting osd_memory_target on smithi098 to 3804M 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:58.960560+0000 mon.a (mon.0) 821 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.105050+0000 mon.a (mon.0) 822 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.119748+0000 mon.b (mon.1) 53 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.120215+0000 mon.a (mon.0) 823 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.122890+0000 mon.b (mon.1) 54 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.125725+0000 mon.a (mon.0) 824 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.125919+0000 mon.b (mon.1) 55 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.128808+0000 mon.a (mon.0) 825 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.128869+0000 mon.b (mon.1) 56 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.131891+0000 mon.a (mon.0) 826 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.134801+0000 mon.a (mon.0) 827 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.136579+0000 mgr.y (mgr.14538) 13 : cephadm 1 Adjusting osd_memory_target on smithi069 to 3804M 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.143943+0000 mon.a (mon.0) 828 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.145569+0000 mon.b (mon.1) 57 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.147548+0000 mon.b (mon.1) 58 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.155329+0000 mgr.y (mgr.14538) 14 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:45:00.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.155705+0000 mgr.y (mgr.14538) 15 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.370320+0000 mgr.y (mgr.14538) 16 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.373035+0000 mgr.y (mgr.14538) 17 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.534687+0000 mgr.y (mgr.14538) 18 : cephadm 1 Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.538692+0000 mgr.y (mgr.14538) 19 : cephadm 1 Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.871248+0000 mon.a (mon.0) 829 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.878180+0000 mon.a (mon.0) 830 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.883777+0000 mon.a (mon.0) 831 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.892684+0000 mon.a (mon.0) 832 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[21528]: audit 2024-04-02T04:44:59.900757+0000 mon.a (mon.0) 833 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.349501+0000 osd.1 (osd.1) 15 : cluster 0 2.d scrub starts 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.351061+0000 osd.1 (osd.1) 16 : cluster 0 2.d scrub ok 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.698158+0000 osd.4 (osd.4) 9 : cluster 0 2.f scrub starts 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.700059+0000 osd.0 (osd.0) 15 : cluster 0 2.1f scrub starts 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.701316+0000 osd.4 (osd.4) 10 : cluster 0 2.f scrub ok 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.702079+0000 osd.0 (osd.0) 16 : cluster 0 2.1f scrub ok 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.793212+0000 osd.6 (osd.6) 9 : cluster 0 2.17 deep-scrub starts 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cluster 2024-04-02T04:44:58.795137+0000 osd.6 (osd.6) 10 : cluster 0 2.17 deep-scrub ok 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.925955+0000 mon.a (mon.0) 815 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.936229+0000 mon.b (mon.1) 49 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.938249+0000 mon.a (mon.0) 816 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.939302+0000 mon.b (mon.1) 50 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.941942+0000 mon.a (mon.0) 817 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.942230+0000 mon.b (mon.1) 51 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.944931+0000 mon.b (mon.1) 52 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.945303+0000 mon.a (mon.0) 818 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.948069+0000 mon.a (mon.0) 819 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.950785+0000 mon.a (mon.0) 820 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:58.952392+0000 mgr.y (mgr.14538) 12 : cephadm 1 Adjusting osd_memory_target on smithi098 to 3804M 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:58.960560+0000 mon.a (mon.0) 821 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.105050+0000 mon.a (mon.0) 822 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.119748+0000 mon.b (mon.1) 53 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.120215+0000 mon.a (mon.0) 823 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.122890+0000 mon.b (mon.1) 54 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.125725+0000 mon.a (mon.0) 824 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.125919+0000 mon.b (mon.1) 55 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.128808+0000 mon.a (mon.0) 825 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.128869+0000 mon.b (mon.1) 56 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.131891+0000 mon.a (mon.0) 826 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.134801+0000 mon.a (mon.0) 827 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.136579+0000 mgr.y (mgr.14538) 13 : cephadm 1 Adjusting osd_memory_target on smithi069 to 3804M 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.143943+0000 mon.a (mon.0) 828 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.145569+0000 mon.b (mon.1) 57 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.147548+0000 mon.b (mon.1) 58 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.155329+0000 mgr.y (mgr.14538) 14 : cephadm 1 Updating smithi069:/etc/ceph/ceph.conf 2024-04-02T04:45:00.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.155705+0000 mgr.y (mgr.14538) 15 : cephadm 1 Updating smithi098:/etc/ceph/ceph.conf 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.370320+0000 mgr.y (mgr.14538) 16 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.373035+0000 mgr.y (mgr.14538) 17 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.534687+0000 mgr.y (mgr.14538) 18 : cephadm 1 Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.538692+0000 mgr.y (mgr.14538) 19 : cephadm 1 Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.871248+0000 mon.a (mon.0) 829 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.878180+0000 mon.a (mon.0) 830 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.883777+0000 mon.a (mon.0) 831 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.892684+0000 mon.a (mon.0) 832 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:44:59 smithi069 bash[28904]: audit 2024-04-02T04:44:59.900757+0000 mon.a (mon.0) 833 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:00.614 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.615 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.615 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.615 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.616 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.616 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.878 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.878 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.879 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.879 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.879 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.880 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:00.880 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.149 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:00 smithi069 systemd[1]: Started Ceph node-exporter.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:01.149 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:00 smithi069 bash[57424]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 -- 172.21.15.98:0/1382681964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006ab00 msgr2=0x7fdaf006aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/1382681964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006ab00 0x7fdaf006aee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdae4009990 tx=0x7fdae4009ca0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 -- 172.21.15.98:0/1382681964 shutdown_connections 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/1382681964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 0x7fdaf010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/1382681964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006ab00 0x7fdaf006aee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/1382681964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf006b4b0 0x7fdaf0106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 -- 172.21.15.98:0/1382681964 >> 172.21.15.98:0/1382681964 conn(0x7fdaf0078240 msgr2=0x7fdaf0078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 -- 172.21.15.98:0/1382681964 shutdown_connections 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.355+0000 7fdaf60ad700 1 -- 172.21.15.98:0/1382681964 wait complete. 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 librados: starting msgr at 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 librados: starting objecter 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 Processor -- start 2024-04-02T04:45:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- start start 2024-04-02T04:45:01.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 librados: setting wanted keys 2024-04-02T04:45:01.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 librados: calling monclient init 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf006ab00 0x7fdaf0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 0x7fdaf0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 0x7fdaf019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf0111a90 con 0x7fdaf0107680 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf0111910 con 0x7fdaf006b4b0 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdaf0111c10 con 0x7fdaf006ab00 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 0x7fdaf0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 0x7fdaf0198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38540/0 (socket says 172.21.15.98:38540) 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 -- 172.21.15.98:0/2985583584 learned_addr learned my addr 172.21.15.98:0/2985583584 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaef7fe700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf006ab00 0x7fdaf0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeffff700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 0x7fdaf019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 -- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf006ab00 msgr2=0x7fdaf0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf006ab00 0x7fdaf0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 -- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 msgr2=0x7fdaf019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 0x7fdaf019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 -- 172.21.15.98:0/2985583584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf019d000 con 0x7fdaf006b4b0 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeffff700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 0x7fdaf019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaeeffd700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 0x7fdaf0198530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdae000c9e0 tx=0x7fdae000cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 10 client.?.objecter ms_handle_connect 0x7fdaf006b4b0 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0007aa0 con 0x7fdaf006b4b0 2024-04-02T04:45:01.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fdae0007c40 con 0x7fdaf006b4b0 2024-04-02T04:45:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf019d2f0 con 0x7fdaf006b4b0 2024-04-02T04:45:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0018660 con 0x7fdaf006b4b0 2024-04-02T04:45:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 10 client.24475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:45:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf01a4b70 con 0x7fdaf006b4b0 2024-04-02T04:45:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 librados: init done 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaf004ff00 con 0x7fdaf006b4b0 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fdae000f450 con 0x7fdaf006b4b0 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.359+0000 7fdaecff9700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdad806f5e0 0x7fdad8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 23..103) ==== 7421+0+0 (secure 0 0 0) 0x7fdae008dfb0 con 0x7fdaf006b4b0 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaecff9700 10 client.24475.objecter ms_dispatch 0x7fdaf0001550 osd_map(103..103 src has 23..103) 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaecff9700 3 client.24475.objecter handle_osd_map got epochs [103,103] > 0 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaecff9700 3 client.24475.objecter handle_osd_map decoding full epoch 103 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaecff9700 20 client.24475.objecter dump_active .. 0 homeless 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaef7fe700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdad806f5e0 0x7fdad8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:01.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaecff9700 10 client.24475.objecter ms_handle_connect 0x7fdad806f5e0 2024-04-02T04:45:01.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.363+0000 7fdaef7fe700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdad806f5e0 0x7fdad8071aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdae4009750 tx=0x7fdae400b540 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:01.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.367+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdae0014030 con 0x7fdaf006b4b0 2024-04-02T04:45:01.474 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.371731+0000 osd.1 (osd.1) 17 : cluster 0 2.a scrub starts 2024-04-02T04:45:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.372869+0000 osd.1 (osd.1) 18 : cluster 0 2.a scrub ok 2024-04-02T04:45:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.558638+0000 osd.3 (osd.3) 11 : cluster 0 3.b scrub starts 2024-04-02T04:45:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.560712+0000 osd.3 (osd.3) 12 : cluster 0 3.b scrub ok 2024-04-02T04:45:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.699561+0000 mgr.y (mgr.14538) 20 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.702946+0000 mgr.y (mgr.14538) 21 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.706987+0000 osd.4 (osd.4) 11 : cluster 0 3.1a scrub starts 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.709442+0000 osd.4 (osd.4) 12 : cluster 0 3.1a scrub ok 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.731558+0000 mgr.y (mgr.14538) 22 : cluster 0 pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.806542+0000 osd.6 (osd.6) 11 : cluster 0 4.5 scrub starts 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.808008+0000 osd.6 (osd.6) 12 : cluster 0 4.5 scrub ok 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.838283+0000 osd.3 (osd.3) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:44:59.838351+0000 osd.3 (osd.3) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cephadm 2024-04-02T04:44:59.908144+0000 mgr.y (mgr.14538) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi069 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.153873+0000 mon.a (mon.0) 834 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.275727+0000 osd.7 (osd.7) 7 : cluster 0 1.0 scrub starts 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.279553+0000 osd.7 (osd.7) 8 : cluster 0 1.0 scrub ok 2024-04-02T04:45:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: audit 2024-04-02T04:45:00.923466+0000 mon.a (mon.0) 835 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: audit 2024-04-02T04:45:00.936612+0000 mon.a (mon.0) 836 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 bash[23790]: audit 2024-04-02T04:45:00.981015+0000 mon.a (mon.0) 837 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.371731+0000 osd.1 (osd.1) 17 : cluster 0 2.a scrub starts 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.372869+0000 osd.1 (osd.1) 18 : cluster 0 2.a scrub ok 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.558638+0000 osd.3 (osd.3) 11 : cluster 0 3.b scrub starts 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.560712+0000 osd.3 (osd.3) 12 : cluster 0 3.b scrub ok 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.699561+0000 mgr.y (mgr.14538) 20 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.702946+0000 mgr.y (mgr.14538) 21 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.706987+0000 osd.4 (osd.4) 11 : cluster 0 3.1a scrub starts 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.709442+0000 osd.4 (osd.4) 12 : cluster 0 3.1a scrub ok 2024-04-02T04:45:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.731558+0000 mgr.y (mgr.14538) 22 : cluster 0 pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.806542+0000 osd.6 (osd.6) 11 : cluster 0 4.5 scrub starts 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.808008+0000 osd.6 (osd.6) 12 : cluster 0 4.5 scrub ok 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.838283+0000 osd.3 (osd.3) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:44:59.838351+0000 osd.3 (osd.3) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cephadm 2024-04-02T04:44:59.908144+0000 mgr.y (mgr.14538) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi069 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.153873+0000 mon.a (mon.0) 834 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.275727+0000 osd.7 (osd.7) 7 : cluster 0 1.0 scrub starts 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.279553+0000 osd.7 (osd.7) 8 : cluster 0 1.0 scrub ok 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: audit 2024-04-02T04:45:00.923466+0000 mon.a (mon.0) 835 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: audit 2024-04-02T04:45:00.936612+0000 mon.a (mon.0) 836 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[21528]: audit 2024-04-02T04:45:00.981015+0000 mon.a (mon.0) 837 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.371731+0000 osd.1 (osd.1) 17 : cluster 0 2.a scrub starts 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.372869+0000 osd.1 (osd.1) 18 : cluster 0 2.a scrub ok 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.558638+0000 osd.3 (osd.3) 11 : cluster 0 3.b scrub starts 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.560712+0000 osd.3 (osd.3) 12 : cluster 0 3.b scrub ok 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.699561+0000 mgr.y (mgr.14538) 20 : cephadm 1 Updating smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.702946+0000 mgr.y (mgr.14538) 21 : cephadm 1 Updating smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.706987+0000 osd.4 (osd.4) 11 : cluster 0 3.1a scrub starts 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.709442+0000 osd.4 (osd.4) 12 : cluster 0 3.1a scrub ok 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.731558+0000 mgr.y (mgr.14538) 22 : cluster 0 pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.806542+0000 osd.6 (osd.6) 11 : cluster 0 4.5 scrub starts 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.808008+0000 osd.6 (osd.6) 12 : cluster 0 4.5 scrub ok 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.838283+0000 osd.3 (osd.3) 13 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:01.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:44:59.838351+0000 osd.3 (osd.3) 14 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cephadm 2024-04-02T04:44:59.908144+0000 mgr.y (mgr.14538) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi069 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.153873+0000 mon.a (mon.0) 834 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.275727+0000 osd.7 (osd.7) 7 : cluster 0 1.0 scrub starts 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.279553+0000 osd.7 (osd.7) 8 : cluster 0 1.0 scrub ok 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: audit 2024-04-02T04:45:00.923466+0000 mon.a (mon.0) 835 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: audit 2024-04-02T04:45:00.936612+0000 mon.a (mon.0) 836 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:01 smithi069 bash[28904]: audit 2024-04-02T04:45:00.981015+0000 mon.a (mon.0) 837 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.595+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}) -- 0x7fdaf00ff6a0 con 0x7fdad806f5e0 2024-04-02T04:45:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.611+0000 7fdaecff9700 1 -- 172.21.15.98:0/2985583584 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fdaf00ff6a0 con 0x7fdad806f5e0 2024-04-02T04:45:01.617 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled grafana update... 2024-04-02T04:45:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 10 librados: watch_flush enter 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 10 librados: watch_flush exit 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 20 client.24475.objecter shutdown clearing up homeless session... 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 10 client.24475.objecter successfully canceled tick 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdad806f5e0 msgr2=0x7fdad8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdad806f5e0 0x7fdad8071aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdae4009750 tx=0x7fdae400b540 comp rx=0 tx=0).stop 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 msgr2=0x7fdaf0198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 0x7fdaf0198530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdae000c9e0 tx=0x7fdae000cda0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 shutdown_connections 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdad806f5e0 0x7fdad8071aa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0107680 0x7fdaf019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf006ab00 0x7fdaf0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 --2- 172.21.15.98:0/2985583584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf006b4b0 0x7fdaf0198530 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 >> 172.21.15.98:0/2985583584 conn(0x7fdaf0078240 msgr2=0x7fdaf00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 shutdown_connections 2024-04-02T04:45:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 -- 172.21.15.98:0/2985583584 wait complete. 2024-04-02T04:45:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:01.615+0000 7fdaf60ad700 1 librados: shutdown 2024-04-02T04:45:01.746 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.746 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.747 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.747 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.747 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.748 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.748 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.748 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.748 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.748 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.996 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.996 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.997 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.997 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.997 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.998 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.998 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.998 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.998 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:01.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:01 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:02.474 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:02 smithi098 systemd[1]: Started Ceph node-exporter.b for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:02.474 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[50713]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.348365+0000 osd.1 (osd.1) 19 : cluster 0 2.9 scrub starts 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.349998+0000 osd.1 (osd.1) 20 : cluster 0 2.9 scrub ok 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.563866+0000 osd.3 (osd.3) 15 : cluster 0 3.8 deep-scrub starts 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.578749+0000 osd.3 (osd.3) 16 : cluster 0 3.8 deep-scrub ok 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.642915+0000 osd.0 (osd.0) 17 : cluster 0 6.18 scrub starts 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cluster 2024-04-02T04:45:00.644305+0000 osd.0 (osd.0) 18 : cluster 0 6.18 scrub ok 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cephadm 2024-04-02T04:45:00.983695+0000 mgr.y (mgr.14538) 24 : cephadm 1 Deploying daemon node-exporter.b on smithi098 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:01.607212+0000 mgr.y (mgr.14538) 25 : audit 0 from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: cephadm 2024-04-02T04:45:01.611412+0000 mgr.y (mgr.14538) 26 : cephadm 1 Saving service grafana spec with placement smithi098=a;count:1 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:01.619663+0000 mon.a (mon.0) 838 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:02.054327+0000 mon.a (mon.0) 839 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:02.069276+0000 mon.a (mon.0) 840 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:02.109973+0000 mon.a (mon.0) 841 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:02.119382+0000 mon.a (mon.0) 842 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[23790]: audit 2024-04-02T04:45:02.137898+0000 mon.a (mon.0) 843 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.518 DEBUG:teuthology.orchestra.run.smithi098:grafana.a> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@grafana.a.service 2024-04-02T04:45:02.521 INFO:tasks.cephadm:Setting up client nodes... 2024-04-02T04:45:02.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-02T04:45:02.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.348365+0000 osd.1 (osd.1) 19 : cluster 0 2.9 scrub starts 2024-04-02T04:45:02.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.349998+0000 osd.1 (osd.1) 20 : cluster 0 2.9 scrub ok 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.563866+0000 osd.3 (osd.3) 15 : cluster 0 3.8 deep-scrub starts 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.578749+0000 osd.3 (osd.3) 16 : cluster 0 3.8 deep-scrub ok 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.642915+0000 osd.0 (osd.0) 17 : cluster 0 6.18 scrub starts 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cluster 2024-04-02T04:45:00.644305+0000 osd.0 (osd.0) 18 : cluster 0 6.18 scrub ok 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cephadm 2024-04-02T04:45:00.983695+0000 mgr.y (mgr.14538) 24 : cephadm 1 Deploying daemon node-exporter.b on smithi098 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:01.607212+0000 mgr.y (mgr.14538) 25 : audit 0 from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: cephadm 2024-04-02T04:45:01.611412+0000 mgr.y (mgr.14538) 26 : cephadm 1 Saving service grafana spec with placement smithi098=a;count:1 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:01.619663+0000 mon.a (mon.0) 838 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:02.054327+0000 mon.a (mon.0) 839 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:02.069276+0000 mon.a (mon.0) 840 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:02.109973+0000 mon.a (mon.0) 841 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:02.119382+0000 mon.a (mon.0) 842 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[21528]: audit 2024-04-02T04:45:02.137898+0000 mon.a (mon.0) 843 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.348365+0000 osd.1 (osd.1) 19 : cluster 0 2.9 scrub starts 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.349998+0000 osd.1 (osd.1) 20 : cluster 0 2.9 scrub ok 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.563866+0000 osd.3 (osd.3) 15 : cluster 0 3.8 deep-scrub starts 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.578749+0000 osd.3 (osd.3) 16 : cluster 0 3.8 deep-scrub ok 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.642915+0000 osd.0 (osd.0) 17 : cluster 0 6.18 scrub starts 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cluster 2024-04-02T04:45:00.644305+0000 osd.0 (osd.0) 18 : cluster 0 6.18 scrub ok 2024-04-02T04:45:02.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cephadm 2024-04-02T04:45:00.983695+0000 mgr.y (mgr.14538) 24 : cephadm 1 Deploying daemon node-exporter.b on smithi098 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:01.607212+0000 mgr.y (mgr.14538) 25 : audit 0 from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: cephadm 2024-04-02T04:45:01.611412+0000 mgr.y (mgr.14538) 26 : cephadm 1 Saving service grafana spec with placement smithi098=a;count:1 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:01.619663+0000 mon.a (mon.0) 838 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:02.054327+0000 mon.a (mon.0) 839 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:02.069276+0000 mon.a (mon.0) 840 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:02.109973+0000 mon.a (mon.0) 841 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:02.119382+0000 mon.a (mon.0) 842 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:02 smithi069 bash[28904]: audit 2024-04-02T04:45:02.137898+0000 mon.a (mon.0) 843 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:02.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:45:03.123 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:02 smithi098 bash[50713]: v1.5.0: Pulling from prometheus/node-exporter 2024-04-02T04:45:03.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[21528]: cluster 2024-04-02T04:45:01.592143+0000 osd.3 (osd.3) 17 : cluster 0 3.18 scrub starts 2024-04-02T04:45:03.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[21528]: cluster 2024-04-02T04:45:01.596526+0000 osd.3 (osd.3) 18 : cluster 0 3.18 scrub ok 2024-04-02T04:45:03.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[21528]: cluster 2024-04-02T04:45:01.732296+0000 mgr.y (mgr.14538) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:03.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[21528]: cephadm 2024-04-02T04:45:02.485680+0000 mgr.y (mgr.14538) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi069 2024-04-02T04:45:03.194 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[28904]: cluster 2024-04-02T04:45:01.592143+0000 osd.3 (osd.3) 17 : cluster 0 3.18 scrub starts 2024-04-02T04:45:03.194 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[28904]: cluster 2024-04-02T04:45:01.596526+0000 osd.3 (osd.3) 18 : cluster 0 3.18 scrub ok 2024-04-02T04:45:03.194 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[28904]: cluster 2024-04-02T04:45:01.732296+0000 mgr.y (mgr.14538) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:03.194 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[28904]: cephadm 2024-04-02T04:45:02.485680+0000 mgr.y (mgr.14538) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi069 2024-04-02T04:45:03.195 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:02 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:03.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[23790]: cluster 2024-04-02T04:45:01.592143+0000 osd.3 (osd.3) 17 : cluster 0 3.18 scrub starts 2024-04-02T04:45:03.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[23790]: cluster 2024-04-02T04:45:01.596526+0000 osd.3 (osd.3) 18 : cluster 0 3.18 scrub ok 2024-04-02T04:45:03.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[23790]: cluster 2024-04-02T04:45:01.732296+0000 mgr.y (mgr.14538) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:03.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[23790]: cephadm 2024-04-02T04:45:02.485680+0000 mgr.y (mgr.14538) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi069 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 22b70bddd3ac: Pulling fs layer 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 5c12815fee55: Pulling fs layer 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: c0e87333d380: Pulling fs layer 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 22b70bddd3ac: Verifying Checksum 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 22b70bddd3ac: Download complete 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 5c12815fee55: Verifying Checksum 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 5c12815fee55: Download complete 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: c0e87333d380: Verifying Checksum 2024-04-02T04:45:03.472 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: c0e87333d380: Download complete 2024-04-02T04:45:03.528 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: v1.5.0: Pulling from prometheus/node-exporter 2024-04-02T04:45:03.724 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 22b70bddd3ac: Pull complete 2024-04-02T04:45:03.792 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: 22b70bddd3ac: Pulling fs layer 2024-04-02T04:45:03.792 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: 5c12815fee55: Pulling fs layer 2024-04-02T04:45:03.792 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: c0e87333d380: Pulling fs layer 2024-04-02T04:45:03.792 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: 5c12815fee55: Verifying Checksum 2024-04-02T04:45:04.153 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: 5c12815fee55: Download complete 2024-04-02T04:45:04.153 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: c0e87333d380: Verifying Checksum 2024-04-02T04:45:04.153 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: c0e87333d380: Download complete 2024-04-02T04:45:04.153 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: 22b70bddd3ac: Verifying Checksum 2024-04-02T04:45:04.154 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:03 smithi069 bash[57424]: 22b70bddd3ac: Download complete 2024-04-02T04:45:04.154 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[57424]: 22b70bddd3ac: Pull complete 2024-04-02T04:45:04.317 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:03 smithi098 bash[50713]: 5c12815fee55: Pull complete 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:02.689881+0000 osd.0 (osd.0) 19 : cluster 0 4.1e scrub starts 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:02.691601+0000 osd.0 (osd.0) 20 : cluster 0 4.1e scrub ok 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: audit 2024-04-02T04:45:02.745783+0000 mgr.y (mgr.14538) 29 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:02.904174+0000 osd.5 (osd.5) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:02.904287+0000 osd.5 (osd.5) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.197775+0000 osd.5 (osd.5) 13 : cluster 0 3.5 scrub starts 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.202978+0000 osd.5 (osd.5) 14 : cluster 0 3.5 scrub ok 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.257809+0000 osd.7 (osd.7) 9 : cluster 0 2.8 scrub starts 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.259867+0000 osd.7 (osd.7) 10 : cluster 0 2.8 scrub ok 2024-04-02T04:45:04.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.628079+0000 mon.a (mon.0) 844 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:02.689881+0000 osd.0 (osd.0) 19 : cluster 0 4.1e scrub starts 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:02.691601+0000 osd.0 (osd.0) 20 : cluster 0 4.1e scrub ok 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: audit 2024-04-02T04:45:02.745783+0000 mgr.y (mgr.14538) 29 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:02.904174+0000 osd.5 (osd.5) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:02.904287+0000 osd.5 (osd.5) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.197775+0000 osd.5 (osd.5) 13 : cluster 0 3.5 scrub starts 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.202978+0000 osd.5 (osd.5) 14 : cluster 0 3.5 scrub ok 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.257809+0000 osd.7 (osd.7) 9 : cluster 0 2.8 scrub starts 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.259867+0000 osd.7 (osd.7) 10 : cluster 0 2.8 scrub ok 2024-04-02T04:45:04.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.628079+0000 mon.a (mon.0) 844 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:02.689881+0000 osd.0 (osd.0) 19 : cluster 0 4.1e scrub starts 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:02.691601+0000 osd.0 (osd.0) 20 : cluster 0 4.1e scrub ok 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: audit 2024-04-02T04:45:02.745783+0000 mgr.y (mgr.14538) 29 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:02.904174+0000 osd.5 (osd.5) 11 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:02.904287+0000 osd.5 (osd.5) 12 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.197775+0000 osd.5 (osd.5) 13 : cluster 0 3.5 scrub starts 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.202978+0000 osd.5 (osd.5) 14 : cluster 0 3.5 scrub ok 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.257809+0000 osd.7 (osd.7) 9 : cluster 0 2.8 scrub starts 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.259867+0000 osd.7 (osd.7) 10 : cluster 0 2.8 scrub ok 2024-04-02T04:45:04.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.628079+0000 mon.a (mon.0) 844 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-04-02T04:45:04.586 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[50713]: c0e87333d380: Pull complete 2024-04-02T04:45:04.586 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[50713]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-04-02T04:45:04.587 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:04 smithi098 bash[50713]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-04-02T04:45:04.777 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[57424]: 5c12815fee55: Pull complete 2024-04-02T04:45:05.154 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:04 smithi069 bash[57424]: c0e87333d380: Pull complete 2024-04-02T04:45:05.154 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[57424]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-04-02T04:45:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.598265+0000 osd.3 (osd.3) 19 : cluster 0 2.1e scrub starts 2024-04-02T04:45:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.600673+0000 osd.3 (osd.3) 20 : cluster 0 2.1e scrub ok 2024-04-02T04:45:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.666412+0000 osd.4 (osd.4) 13 : cluster 0 4.13 scrub starts 2024-04-02T04:45:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.668339+0000 osd.4 (osd.4) 14 : cluster 0 4.13 scrub ok 2024-04-02T04:45:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:03.733123+0000 mgr.y (mgr.14538) 30 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.161542+0000 osd.5 (osd.5) 15 : cluster 0 4.8 scrub starts 2024-04-02T04:45:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:05 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.162801+0000 osd.5 (osd.5) 16 : cluster 0 4.8 scrub ok 2024-04-02T04:45:05.528 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[57424]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-04-02T04:45:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.598265+0000 osd.3 (osd.3) 19 : cluster 0 2.1e scrub starts 2024-04-02T04:45:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.600673+0000 osd.3 (osd.3) 20 : cluster 0 2.1e scrub ok 2024-04-02T04:45:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.666412+0000 osd.4 (osd.4) 13 : cluster 0 4.13 scrub starts 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.668339+0000 osd.4 (osd.4) 14 : cluster 0 4.13 scrub ok 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:03.733123+0000 mgr.y (mgr.14538) 30 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.161542+0000 osd.5 (osd.5) 15 : cluster 0 4.8 scrub starts 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.162801+0000 osd.5 (osd.5) 16 : cluster 0 4.8 scrub ok 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.598265+0000 osd.3 (osd.3) 19 : cluster 0 2.1e scrub starts 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.600673+0000 osd.3 (osd.3) 20 : cluster 0 2.1e scrub ok 2024-04-02T04:45:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.666412+0000 osd.4 (osd.4) 13 : cluster 0 4.13 scrub starts 2024-04-02T04:45:05.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.668339+0000 osd.4 (osd.4) 14 : cluster 0 4.13 scrub ok 2024-04-02T04:45:05.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:03.733123+0000 mgr.y (mgr.14538) 30 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:05.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.161542+0000 osd.5 (osd.5) 15 : cluster 0 4.8 scrub starts 2024-04-02T04:45:05.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:05 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.162801+0000 osd.5 (osd.5) 16 : cluster 0 4.8 scrub ok 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.616316+0000 osd.3 (osd.3) 21 : cluster 0 4.10 scrub starts 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.617778+0000 osd.3 (osd.3) 22 : cluster 0 4.10 scrub ok 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.742012+0000 osd.0 (osd.0) 21 : cluster 0 4.b scrub starts 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.743595+0000 osd.0 (osd.0) 22 : cluster 0 4.b scrub ok 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.989238+0000 osd.2 (osd.2) 15 : cluster 0 6.14 scrub starts 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:04.991749+0000 osd.2 (osd.2) 16 : cluster 0 6.14 scrub ok 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.162221+0000 osd.5 (osd.5) 17 : cluster 0 3.16 scrub starts 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.165976+0000 osd.5 (osd.5) 18 : cluster 0 3.16 scrub ok 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.232084+0000 osd.7 (osd.7) 11 : cluster 0 6.3 deep-scrub starts 2024-04-02T04:45:06.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.233720+0000 osd.7 (osd.7) 12 : cluster 0 6.3 deep-scrub ok 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.027Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.027Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.027Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-02T04:45:06.226 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=arp 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=edac 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-02T04:45:06.227 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-02T04:45:06.228 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=os 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=stat 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-02T04:45:06.229 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.028Z caller=node_exporter.go:117 level=info collector=time 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=node_exporter.go:117 level=info collector=uname 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-02T04:45:06.230 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:06 smithi098 bash[50713]: ts=2024-04-02T04:45:06.029Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.616316+0000 osd.3 (osd.3) 21 : cluster 0 4.10 scrub starts 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.617778+0000 osd.3 (osd.3) 22 : cluster 0 4.10 scrub ok 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.742012+0000 osd.0 (osd.0) 21 : cluster 0 4.b scrub starts 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.743595+0000 osd.0 (osd.0) 22 : cluster 0 4.b scrub ok 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.989238+0000 osd.2 (osd.2) 15 : cluster 0 6.14 scrub starts 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:04.991749+0000 osd.2 (osd.2) 16 : cluster 0 6.14 scrub ok 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.162221+0000 osd.5 (osd.5) 17 : cluster 0 3.16 scrub starts 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.165976+0000 osd.5 (osd.5) 18 : cluster 0 3.16 scrub ok 2024-04-02T04:45:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.232084+0000 osd.7 (osd.7) 11 : cluster 0 6.3 deep-scrub starts 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.233720+0000 osd.7 (osd.7) 12 : cluster 0 6.3 deep-scrub ok 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.616316+0000 osd.3 (osd.3) 21 : cluster 0 4.10 scrub starts 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.617778+0000 osd.3 (osd.3) 22 : cluster 0 4.10 scrub ok 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.742012+0000 osd.0 (osd.0) 21 : cluster 0 4.b scrub starts 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.743595+0000 osd.0 (osd.0) 22 : cluster 0 4.b scrub ok 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.989238+0000 osd.2 (osd.2) 15 : cluster 0 6.14 scrub starts 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:04.991749+0000 osd.2 (osd.2) 16 : cluster 0 6.14 scrub ok 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.162221+0000 osd.5 (osd.5) 17 : cluster 0 3.16 scrub starts 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.165976+0000 osd.5 (osd.5) 18 : cluster 0 3.16 scrub ok 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.232084+0000 osd.7 (osd.7) 11 : cluster 0 6.3 deep-scrub starts 2024-04-02T04:45:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:06 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.233720+0000 osd.7 (osd.7) 12 : cluster 0 6.3 deep-scrub ok 2024-04-02T04:45:07.265 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:07.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.216504+0000 osd.1 (osd.1) 21 : cluster 0 4.2 scrub starts 2024-04-02T04:45:07.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.218491+0000 osd.1 (osd.1) 22 : cluster 0 4.2 scrub ok 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.733916+0000 mgr.y (mgr.14538) 31 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.861369+0000 osd.6 (osd.6) 13 : cluster 0 2.1a deep-scrub starts 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:05.863856+0000 osd.6 (osd.6) 14 : cluster 0 2.1a deep-scrub ok 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:06.189781+0000 osd.7 (osd.7) 13 : cluster 0 2.1d scrub starts 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:06.192580+0000 osd.7 (osd.7) 14 : cluster 0 2.1d scrub ok 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:06.193117+0000 osd.5 (osd.5) 19 : cluster 0 2.18 scrub starts 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: cluster 2024-04-02T04:45:06.195515+0000 osd.5 (osd.5) 20 : cluster 0 2.18 scrub ok 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: audit 2024-04-02T04:45:06.820090+0000 mon.a (mon.0) 845 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:07.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:07 smithi098 bash[23790]: audit 2024-04-02T04:45:06.853735+0000 mon.b (mon.1) 59 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.216504+0000 osd.1 (osd.1) 21 : cluster 0 4.2 scrub starts 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.218491+0000 osd.1 (osd.1) 22 : cluster 0 4.2 scrub ok 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.733916+0000 mgr.y (mgr.14538) 31 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.861369+0000 osd.6 (osd.6) 13 : cluster 0 2.1a deep-scrub starts 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:05.863856+0000 osd.6 (osd.6) 14 : cluster 0 2.1a deep-scrub ok 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:06.189781+0000 osd.7 (osd.7) 13 : cluster 0 2.1d scrub starts 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:06.192580+0000 osd.7 (osd.7) 14 : cluster 0 2.1d scrub ok 2024-04-02T04:45:07.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:06.193117+0000 osd.5 (osd.5) 19 : cluster 0 2.18 scrub starts 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: cluster 2024-04-02T04:45:06.195515+0000 osd.5 (osd.5) 20 : cluster 0 2.18 scrub ok 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: audit 2024-04-02T04:45:06.820090+0000 mon.a (mon.0) 845 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[28904]: audit 2024-04-02T04:45:06.853735+0000 mon.b (mon.1) 59 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.216504+0000 osd.1 (osd.1) 21 : cluster 0 4.2 scrub starts 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.218491+0000 osd.1 (osd.1) 22 : cluster 0 4.2 scrub ok 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.733916+0000 mgr.y (mgr.14538) 31 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.861369+0000 osd.6 (osd.6) 13 : cluster 0 2.1a deep-scrub starts 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:05.863856+0000 osd.6 (osd.6) 14 : cluster 0 2.1a deep-scrub ok 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:06.189781+0000 osd.7 (osd.7) 13 : cluster 0 2.1d scrub starts 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:06.192580+0000 osd.7 (osd.7) 14 : cluster 0 2.1d scrub ok 2024-04-02T04:45:07.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:06.193117+0000 osd.5 (osd.5) 19 : cluster 0 2.18 scrub starts 2024-04-02T04:45:07.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: cluster 2024-04-02T04:45:06.195515+0000 osd.5 (osd.5) 20 : cluster 0 2.18 scrub ok 2024-04-02T04:45:07.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: audit 2024-04-02T04:45:06.820090+0000 mon.a (mon.0) 845 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:07.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:07 smithi069 bash[21528]: audit 2024-04-02T04:45:06.853735+0000 mon.b (mon.1) 59 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:08.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:08 smithi098 bash[23790]: cluster 2024-04-02T04:45:06.675839+0000 osd.3 (osd.3) 23 : cluster 0 4.19 scrub starts 2024-04-02T04:45:08.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:08 smithi098 bash[23790]: cluster 2024-04-02T04:45:06.677349+0000 osd.3 (osd.3) 24 : cluster 0 4.19 scrub ok 2024-04-02T04:45:08.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[28904]: cluster 2024-04-02T04:45:06.675839+0000 osd.3 (osd.3) 23 : cluster 0 4.19 scrub starts 2024-04-02T04:45:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[28904]: cluster 2024-04-02T04:45:06.677349+0000 osd.3 (osd.3) 24 : cluster 0 4.19 scrub ok 2024-04-02T04:45:08.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[21528]: cluster 2024-04-02T04:45:06.675839+0000 osd.3 (osd.3) 23 : cluster 0 4.19 scrub starts 2024-04-02T04:45:08.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[21528]: cluster 2024-04-02T04:45:06.677349+0000 osd.3 (osd.3) 24 : cluster 0 4.19 scrub ok 2024-04-02T04:45:08.960 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.688Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.688Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.689Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.689Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.689Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.689Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=arp 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-02T04:45:08.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=edac 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-02T04:45:08.962 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=os 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-02T04:45:08.963 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=stat 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=time 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=uname 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-02T04:45:08.964 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:08 smithi069 bash[57424]: ts=2024-04-02T04:45:08.690Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-02T04:45:09.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.641272+0000 osd.4 (osd.4) 15 : cluster 0 3.9 scrub starts 2024-04-02T04:45:09.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.646159+0000 osd.4 (osd.4) 16 : cluster 0 3.9 scrub ok 2024-04-02T04:45:09.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.689105+0000 osd.3 (osd.3) 25 : cluster 0 4.1d scrub starts 2024-04-02T04:45:09.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.691398+0000 osd.3 (osd.3) 26 : cluster 0 4.1d scrub ok 2024-04-02T04:45:09.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.734926+0000 mgr.y (mgr.14538) 32 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.866738+0000 osd.6 (osd.6) 15 : cluster 0 2.14 scrub starts 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:07.869124+0000 osd.6 (osd.6) 16 : cluster 0 2.14 scrub ok 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.099333+0000 osd.5 (osd.5) 21 : cluster 0 6.7 scrub starts 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.101507+0000 osd.5 (osd.5) 22 : cluster 0 6.7 scrub ok 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.232317+0000 osd.7 (osd.7) 15 : cluster 0 3.e scrub starts 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.239123+0000 osd.7 (osd.7) 16 : cluster 0 3.e scrub ok 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.641272+0000 osd.4 (osd.4) 15 : cluster 0 3.9 scrub starts 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.646159+0000 osd.4 (osd.4) 16 : cluster 0 3.9 scrub ok 2024-04-02T04:45:09.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.689105+0000 osd.3 (osd.3) 25 : cluster 0 4.1d scrub starts 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.691398+0000 osd.3 (osd.3) 26 : cluster 0 4.1d scrub ok 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.734926+0000 mgr.y (mgr.14538) 32 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.866738+0000 osd.6 (osd.6) 15 : cluster 0 2.14 scrub starts 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:07.869124+0000 osd.6 (osd.6) 16 : cluster 0 2.14 scrub ok 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.099333+0000 osd.5 (osd.5) 21 : cluster 0 6.7 scrub starts 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.101507+0000 osd.5 (osd.5) 22 : cluster 0 6.7 scrub ok 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.232317+0000 osd.7 (osd.7) 15 : cluster 0 3.e scrub starts 2024-04-02T04:45:09.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:09 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.239123+0000 osd.7 (osd.7) 16 : cluster 0 3.e scrub ok 2024-04-02T04:45:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.641272+0000 osd.4 (osd.4) 15 : cluster 0 3.9 scrub starts 2024-04-02T04:45:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.646159+0000 osd.4 (osd.4) 16 : cluster 0 3.9 scrub ok 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.689105+0000 osd.3 (osd.3) 25 : cluster 0 4.1d scrub starts 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.691398+0000 osd.3 (osd.3) 26 : cluster 0 4.1d scrub ok 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.734926+0000 mgr.y (mgr.14538) 32 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.866738+0000 osd.6 (osd.6) 15 : cluster 0 2.14 scrub starts 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:07.869124+0000 osd.6 (osd.6) 16 : cluster 0 2.14 scrub ok 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.099333+0000 osd.5 (osd.5) 21 : cluster 0 6.7 scrub starts 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.101507+0000 osd.5 (osd.5) 22 : cluster 0 6.7 scrub ok 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.232317+0000 osd.7 (osd.7) 15 : cluster 0 3.e scrub starts 2024-04-02T04:45:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:09 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.239123+0000 osd.7 (osd.7) 16 : cluster 0 3.e scrub ok 2024-04-02T04:45:10.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.235568+0000 osd.1 (osd.1) 23 : cluster 0 2.4 scrub starts 2024-04-02T04:45:10.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.238832+0000 osd.1 (osd.1) 24 : cluster 0 2.4 scrub ok 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.685570+0000 osd.3 (osd.3) 27 : cluster 0 3.1e scrub starts 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.688058+0000 osd.3 (osd.3) 28 : cluster 0 3.1e scrub ok 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.728833+0000 osd.1 (osd.1) 25 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:08.728921+0000 osd.1 (osd.1) 26 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:09.187878+0000 mon.a (mon.0) 846 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:09.275311+0000 osd.7 (osd.7) 17 : cluster 0 3.13 scrub starts 2024-04-02T04:45:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:10 smithi098 bash[23790]: cluster 2024-04-02T04:45:09.281302+0000 osd.7 (osd.7) 18 : cluster 0 3.13 scrub ok 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.235568+0000 osd.1 (osd.1) 23 : cluster 0 2.4 scrub starts 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.238832+0000 osd.1 (osd.1) 24 : cluster 0 2.4 scrub ok 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.685570+0000 osd.3 (osd.3) 27 : cluster 0 3.1e scrub starts 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.688058+0000 osd.3 (osd.3) 28 : cluster 0 3.1e scrub ok 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.728833+0000 osd.1 (osd.1) 25 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:08.728921+0000 osd.1 (osd.1) 26 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:09.187878+0000 mon.a (mon.0) 846 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:09.275311+0000 osd.7 (osd.7) 17 : cluster 0 3.13 scrub starts 2024-04-02T04:45:10.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[21528]: cluster 2024-04-02T04:45:09.281302+0000 osd.7 (osd.7) 18 : cluster 0 3.13 scrub ok 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.235568+0000 osd.1 (osd.1) 23 : cluster 0 2.4 scrub starts 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.238832+0000 osd.1 (osd.1) 24 : cluster 0 2.4 scrub ok 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.685570+0000 osd.3 (osd.3) 27 : cluster 0 3.1e scrub starts 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.688058+0000 osd.3 (osd.3) 28 : cluster 0 3.1e scrub ok 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.728833+0000 osd.1 (osd.1) 25 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:08.728921+0000 osd.1 (osd.1) 26 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:09.187878+0000 mon.a (mon.0) 846 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:09.275311+0000 osd.7 (osd.7) 17 : cluster 0 3.13 scrub starts 2024-04-02T04:45:10.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:10 smithi069 bash[28904]: cluster 2024-04-02T04:45:09.281302+0000 osd.7 (osd.7) 18 : cluster 0 3.13 scrub ok 2024-04-02T04:45:11.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- 172.21.15.69:0/30833301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce81075b0 msgr2=0x7fbce810f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:11.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- 172.21.15.69:0/30833301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce81075b0 0x7fbce810f180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbce000b0d0 tx=0x7fbce000b3e0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- 172.21.15.69:0/30833301 shutdown_connections 2024-04-02T04:45:11.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- 172.21.15.69:0/30833301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce81075b0 0x7fbce810f180 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- 172.21.15.69:0/30833301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81024d0 0x7fbce81028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- 172.21.15.69:0/30833301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce8102e80 0x7fbce8106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- 172.21.15.69:0/30833301 >> 172.21.15.69:0/30833301 conn(0x7fbce80fdd40 msgr2=0x7fbce8100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- 172.21.15.69:0/30833301 shutdown_connections 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- 172.21.15.69:0/30833301 wait complete. 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 librados: starting msgr at 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 librados: starting objecter 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 Processor -- start 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- start start 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 librados: setting wanted keys 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 librados: calling monclient init 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 0x7fbce8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 0x7fbce81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81075b0 0x7fbce819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:11.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbce8111ab0 con 0x7fbce8102e80 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbce8111930 con 0x7fbce81075b0 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcee61f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbce8111c30 con 0x7fbce81024d0 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 0x7fbce81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 0x7fbce8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 0x7fbce81984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49894/0 (socket says 172.21.15.69:49894) 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 -- 172.21.15.69:0/3394819720 learned_addr learned my addr 172.21.15.69:0/3394819720 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 0x7fbce8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50054/0 (socket says 172.21.15.69:50054) 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcecbbc700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81075b0 0x7fbce819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 -- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 msgr2=0x7fbce8197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 0x7fbce8197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 -- 172.21.15.69:0/3394819720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81075b0 msgr2=0x7fbce819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:11.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81075b0 0x7fbce819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 -- 172.21.15.69:0/3394819720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce819cfa0 con 0x7fbce8102e80 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce7fff700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 0x7fbce8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbcecbbc700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81075b0 0x7fbce819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce77fe700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 0x7fbce81984d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000e970 tx=0x7fbcd000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce57fa700 10 client.?.objecter ms_handle_connect 0x7fbce8102e80 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce57fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.468+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd000cbd0 con 0x7fbce8102e80 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fbcd000cd70 con 0x7fbce8102e80 2024-04-02T04:45:11.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce819d290 con 0x7fbce8102e80 2024-04-02T04:45:11.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0005590 con 0x7fbce8102e80 2024-04-02T04:45:11.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbcee61f700 10 client.14577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:45:11.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce8074470 con 0x7fbce8102e80 2024-04-02T04:45:11.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbcee61f700 1 librados: init done 2024-04-02T04:45:11.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:11 smithi098 bash[23790]: cluster 2024-04-02T04:45:09.735688+0000 mgr.y (mgr.14538) 33 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:11.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:11 smithi098 bash[23790]: cluster 2024-04-02T04:45:09.799353+0000 osd.0 (osd.0) 23 : cluster 0 3.1 scrub starts 2024-04-02T04:45:11.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:11 smithi098 bash[23790]: cluster 2024-04-02T04:45:09.804747+0000 osd.0 (osd.0) 24 : cluster 0 3.1 scrub ok 2024-04-02T04:45:11.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:11 smithi098 bash[23790]: cluster 2024-04-02T04:45:10.088634+0000 osd.5 (osd.5) 23 : cluster 0 4.15 scrub starts 2024-04-02T04:45:11.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:11 smithi098 bash[23790]: cluster 2024-04-02T04:45:10.093149+0000 osd.5 (osd.5) 24 : cluster 0 4.15 scrub ok 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce804ff00 con 0x7fbce8102e80 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fbcd0004750 con 0x7fbce8102e80 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbcd406f5e0 0x7fbcd4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 54..105) ==== 7421+0+0 (secure 0 0 0) 0x7fbcd008e2e0 con 0x7fbce8102e80 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 10 client.14577.objecter ms_dispatch 0x7fbce8001550 osd_map(105..105 src has 54..105) 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 3 client.14577.objecter handle_osd_map got epochs [105,105] > 0 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 3 client.14577.objecter handle_osd_map decoding full epoch 105 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce7fff700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbcd406f5e0 0x7fbcd4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 20 client.14577.objecter dump_active .. 0 homeless 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce57fa700 10 client.14577.objecter ms_handle_connect 0x7fbcd406f5e0 2024-04-02T04:45:11.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.472+0000 7fbce7fff700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbcd406f5e0 0x7fbcd4071aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005950 tx=0x7fbcd800a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:11.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.476+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbcd005b9c0 con 0x7fbce8102e80 2024-04-02T04:45:11.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[28904]: cluster 2024-04-02T04:45:09.735688+0000 mgr.y (mgr.14538) 33 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:11.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[28904]: cluster 2024-04-02T04:45:09.799353+0000 osd.0 (osd.0) 23 : cluster 0 3.1 scrub starts 2024-04-02T04:45:11.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[28904]: cluster 2024-04-02T04:45:09.804747+0000 osd.0 (osd.0) 24 : cluster 0 3.1 scrub ok 2024-04-02T04:45:11.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[28904]: cluster 2024-04-02T04:45:10.088634+0000 osd.5 (osd.5) 23 : cluster 0 4.15 scrub starts 2024-04-02T04:45:11.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[28904]: cluster 2024-04-02T04:45:10.093149+0000 osd.5 (osd.5) 24 : cluster 0 4.15 scrub ok 2024-04-02T04:45:11.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[21528]: cluster 2024-04-02T04:45:09.735688+0000 mgr.y (mgr.14538) 33 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:11.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[21528]: cluster 2024-04-02T04:45:09.799353+0000 osd.0 (osd.0) 23 : cluster 0 3.1 scrub starts 2024-04-02T04:45:11.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[21528]: cluster 2024-04-02T04:45:09.804747+0000 osd.0 (osd.0) 24 : cluster 0 3.1 scrub ok 2024-04-02T04:45:11.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[21528]: cluster 2024-04-02T04:45:10.088634+0000 osd.5 (osd.5) 23 : cluster 0 4.15 scrub starts 2024-04-02T04:45:11.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:11 smithi069 bash[21528]: cluster 2024-04-02T04:45:10.093149+0000 osd.5 (osd.5) 24 : cluster 0 4.15 scrub ok 2024-04-02T04:45:11.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.776+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbce8074790 con 0x7fbce8102e80 2024-04-02T04:45:11.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbce57fa700 1 -- 172.21.15.69:0/3394819720 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7fbcd005f410 con 0x7fbce8102e80 2024-04-02T04:45:11.780 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-02T04:45:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBXjQtm70+iLhAAEws/ZJySG4gibQTqi2a1fQ== 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 10 librados: watch_flush enter 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 10 librados: watch_flush exit 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 20 client.14577.objecter shutdown clearing up homeless session... 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 10 client.14577.objecter successfully canceled tick 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbcd406f5e0 msgr2=0x7fbcd4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbcd406f5e0 0x7fbcd4071aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005950 tx=0x7fbcd800a300 comp rx=0 tx=0).stop 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 msgr2=0x7fbce81984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:11.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 0x7fbce81984d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000e970 tx=0x7fbcd000ec80 comp rx=0 tx=0).stop 2024-04-02T04:45:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 shutdown_connections 2024-04-02T04:45:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbcd406f5e0 0x7fbcd4071aa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbce81075b0 0x7fbce819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbce81024d0 0x7fbce8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 --2- 172.21.15.69:0/3394819720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbce8102e80 0x7fbce81984d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 >> 172.21.15.69:0/3394819720 conn(0x7fbce80fdd40 msgr2=0x7fbce80fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:11.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 shutdown_connections 2024-04-02T04:45:11.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 -- 172.21.15.69:0/3394819720 wait complete. 2024-04-02T04:45:11.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:11.780+0000 7fbcee61f700 1 librados: shutdown 2024-04-02T04:45:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: cluster 2024-04-02T04:45:10.304650+0000 osd.1 (osd.1) 27 : cluster 0 4.7 scrub starts 2024-04-02T04:45:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: cluster 2024-04-02T04:45:10.310061+0000 osd.1 (osd.1) 28 : cluster 0 4.7 scrub ok 2024-04-02T04:45:12.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: cluster 2024-04-02T04:45:10.926458+0000 osd.6 (osd.6) 17 : cluster 0 2.7 scrub starts 2024-04-02T04:45:12.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: cluster 2024-04-02T04:45:10.928664+0000 osd.6 (osd.6) 18 : cluster 0 2.7 scrub ok 2024-04-02T04:45:12.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: cluster 2024-04-02T04:45:11.249449+0000 osd.7 (osd.7) 19 : cluster 0 6.8 scrub starts 2024-04-02T04:45:12.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: cluster 2024-04-02T04:45:11.252183+0000 osd.7 (osd.7) 20 : cluster 0 6.8 scrub ok 2024-04-02T04:45:12.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: audit 2024-04-02T04:45:11.782205+0000 mon.a (mon.0) 847 : audit 1 from='client.? 172.21.15.69:0/3394819720' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:12.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[23790]: audit 2024-04-02T04:45:11.784272+0000 mon.a (mon.0) 848 : audit 1 from='client.? 172.21.15.69:0/3394819720' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:45:12.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: cluster 2024-04-02T04:45:10.304650+0000 osd.1 (osd.1) 27 : cluster 0 4.7 scrub starts 2024-04-02T04:45:12.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: cluster 2024-04-02T04:45:10.310061+0000 osd.1 (osd.1) 28 : cluster 0 4.7 scrub ok 2024-04-02T04:45:12.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: cluster 2024-04-02T04:45:10.926458+0000 osd.6 (osd.6) 17 : cluster 0 2.7 scrub starts 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: cluster 2024-04-02T04:45:10.928664+0000 osd.6 (osd.6) 18 : cluster 0 2.7 scrub ok 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: cluster 2024-04-02T04:45:11.249449+0000 osd.7 (osd.7) 19 : cluster 0 6.8 scrub starts 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: cluster 2024-04-02T04:45:11.252183+0000 osd.7 (osd.7) 20 : cluster 0 6.8 scrub ok 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: audit 2024-04-02T04:45:11.782205+0000 mon.a (mon.0) 847 : audit 1 from='client.? 172.21.15.69:0/3394819720' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[21528]: audit 2024-04-02T04:45:11.784272+0000 mon.a (mon.0) 848 : audit 1 from='client.? 172.21.15.69:0/3394819720' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: cluster 2024-04-02T04:45:10.304650+0000 osd.1 (osd.1) 27 : cluster 0 4.7 scrub starts 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: cluster 2024-04-02T04:45:10.310061+0000 osd.1 (osd.1) 28 : cluster 0 4.7 scrub ok 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: cluster 2024-04-02T04:45:10.926458+0000 osd.6 (osd.6) 17 : cluster 0 2.7 scrub starts 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: cluster 2024-04-02T04:45:10.928664+0000 osd.6 (osd.6) 18 : cluster 0 2.7 scrub ok 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: cluster 2024-04-02T04:45:11.249449+0000 osd.7 (osd.7) 19 : cluster 0 6.8 scrub starts 2024-04-02T04:45:12.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: cluster 2024-04-02T04:45:11.252183+0000 osd.7 (osd.7) 20 : cluster 0 6.8 scrub ok 2024-04-02T04:45:12.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: audit 2024-04-02T04:45:11.782205+0000 mon.a (mon.0) 847 : audit 1 from='client.? 172.21.15.69:0/3394819720' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:12.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:12 smithi069 bash[28904]: audit 2024-04-02T04:45:11.784272+0000 mon.a (mon.0) 848 : audit 1 from='client.? 172.21.15.69:0/3394819720' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:45:12.760 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:45:12.776 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-02T04:45:12.776 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T04:45:12.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-02T04:45:13.212 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:12 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:45:13.377 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:13 smithi069 bash[28904]: cluster 2024-04-02T04:45:11.736338+0000 mgr.y (mgr.14538) 34 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 630 B/s rd, 0 op/s 2024-04-02T04:45:13.378 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:13 smithi069 bash[28904]: cluster 2024-04-02T04:45:11.940267+0000 osd.6 (osd.6) 19 : cluster 0 3.a scrub starts 2024-04-02T04:45:13.378 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:13 smithi069 bash[28904]: cluster 2024-04-02T04:45:11.945605+0000 osd.6 (osd.6) 20 : cluster 0 3.a scrub ok 2024-04-02T04:45:13.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:13 smithi069 bash[21528]: cluster 2024-04-02T04:45:11.736338+0000 mgr.y (mgr.14538) 34 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 630 B/s rd, 0 op/s 2024-04-02T04:45:13.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:13 smithi069 bash[21528]: cluster 2024-04-02T04:45:11.940267+0000 osd.6 (osd.6) 19 : cluster 0 3.a scrub starts 2024-04-02T04:45:13.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:13 smithi069 bash[21528]: cluster 2024-04-02T04:45:11.945605+0000 osd.6 (osd.6) 20 : cluster 0 3.a scrub ok 2024-04-02T04:45:13.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:13 smithi098 bash[23790]: cluster 2024-04-02T04:45:11.736338+0000 mgr.y (mgr.14538) 34 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 630 B/s rd, 0 op/s 2024-04-02T04:45:13.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:13 smithi098 bash[23790]: cluster 2024-04-02T04:45:11.940267+0000 osd.6 (osd.6) 19 : cluster 0 3.a scrub starts 2024-04-02T04:45:13.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:13 smithi098 bash[23790]: cluster 2024-04-02T04:45:11.945605+0000 osd.6 (osd.6) 20 : cluster 0 3.a scrub ok 2024-04-02T04:45:14.277 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.278 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.278 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[21528]: cluster 2024-04-02T04:45:12.296666+0000 osd.1 (osd.1) 29 : cluster 0 3.4 scrub starts 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[21528]: cluster 2024-04-02T04:45:12.302292+0000 osd.1 (osd.1) 30 : cluster 0 3.4 scrub ok 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[21528]: audit 2024-04-02T04:45:12.757065+0000 mgr.y (mgr.14538) 35 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[21528]: cluster 2024-04-02T04:45:13.174547+0000 osd.7 (osd.7) 21 : cluster 0 6.1c scrub starts 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[21528]: cluster 2024-04-02T04:45:13.177288+0000 osd.7 (osd.7) 22 : cluster 0 6.1c scrub ok 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[28904]: cluster 2024-04-02T04:45:12.296666+0000 osd.1 (osd.1) 29 : cluster 0 3.4 scrub starts 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[28904]: cluster 2024-04-02T04:45:12.302292+0000 osd.1 (osd.1) 30 : cluster 0 3.4 scrub ok 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[28904]: audit 2024-04-02T04:45:12.757065+0000 mgr.y (mgr.14538) 35 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[28904]: cluster 2024-04-02T04:45:13.174547+0000 osd.7 (osd.7) 21 : cluster 0 6.1c scrub starts 2024-04-02T04:45:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 bash[28904]: cluster 2024-04-02T04:45:13.177288+0000 osd.7 (osd.7) 22 : cluster 0 6.1c scrub ok 2024-04-02T04:45:14.280 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.280 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.280 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:14 smithi098 bash[23790]: cluster 2024-04-02T04:45:12.296666+0000 osd.1 (osd.1) 29 : cluster 0 3.4 scrub starts 2024-04-02T04:45:14.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:14 smithi098 bash[23790]: cluster 2024-04-02T04:45:12.302292+0000 osd.1 (osd.1) 30 : cluster 0 3.4 scrub ok 2024-04-02T04:45:14.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:14 smithi098 bash[23790]: audit 2024-04-02T04:45:12.757065+0000 mgr.y (mgr.14538) 35 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:14.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:14 smithi098 bash[23790]: cluster 2024-04-02T04:45:13.174547+0000 osd.7 (osd.7) 21 : cluster 0 6.1c scrub starts 2024-04-02T04:45:14.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:14 smithi098 bash[23790]: cluster 2024-04-02T04:45:13.177288+0000 osd.7 (osd.7) 22 : cluster 0 6.1c scrub ok 2024-04-02T04:45:14.535 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.536 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.536 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.536 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.537 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.537 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.537 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.537 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.538 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.538 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.538 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.538 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:14.538 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:14 smithi069 systemd[1]: Started Ceph alertmanager.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:15.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: cluster 2024-04-02T04:45:13.291440+0000 osd.1 (osd.1) 31 : cluster 0 2.6 scrub starts 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: cluster 2024-04-02T04:45:13.293643+0000 osd.1 (osd.1) 32 : cluster 0 2.6 scrub ok 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: cluster 2024-04-02T04:45:13.737748+0000 mgr.y (mgr.14538) 36 : cluster 0 pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: cluster 2024-04-02T04:45:14.321465+0000 osd.1 (osd.1) 33 : cluster 0 4.4 scrub starts 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: cluster 2024-04-02T04:45:14.322519+0000 osd.1 (osd.1) 34 : cluster 0 4.4 scrub ok 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: audit 2024-04-02T04:45:14.584492+0000 mon.a (mon.0) 849 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: audit 2024-04-02T04:45:14.595911+0000 mon.a (mon.0) 850 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: audit 2024-04-02T04:45:14.631511+0000 mon.a (mon.0) 851 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: audit 2024-04-02T04:45:14.638586+0000 mon.a (mon.0) 852 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[21528]: audit 2024-04-02T04:45:14.684923+0000 mon.b (mon.1) 60 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: cluster 2024-04-02T04:45:13.291440+0000 osd.1 (osd.1) 31 : cluster 0 2.6 scrub starts 2024-04-02T04:45:15.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: cluster 2024-04-02T04:45:13.293643+0000 osd.1 (osd.1) 32 : cluster 0 2.6 scrub ok 2024-04-02T04:45:15.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: cluster 2024-04-02T04:45:13.737748+0000 mgr.y (mgr.14538) 36 : cluster 0 pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:15.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: cluster 2024-04-02T04:45:14.321465+0000 osd.1 (osd.1) 33 : cluster 0 4.4 scrub starts 2024-04-02T04:45:15.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: cluster 2024-04-02T04:45:14.322519+0000 osd.1 (osd.1) 34 : cluster 0 4.4 scrub ok 2024-04-02T04:45:15.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: audit 2024-04-02T04:45:14.584492+0000 mon.a (mon.0) 849 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: audit 2024-04-02T04:45:14.595911+0000 mon.a (mon.0) 850 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: audit 2024-04-02T04:45:14.631511+0000 mon.a (mon.0) 851 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: audit 2024-04-02T04:45:14.638586+0000 mon.a (mon.0) 852 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[28904]: audit 2024-04-02T04:45:14.684923+0000 mon.b (mon.1) 60 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: cluster 2024-04-02T04:45:13.291440+0000 osd.1 (osd.1) 31 : cluster 0 2.6 scrub starts 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: cluster 2024-04-02T04:45:13.293643+0000 osd.1 (osd.1) 32 : cluster 0 2.6 scrub ok 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: cluster 2024-04-02T04:45:13.737748+0000 mgr.y (mgr.14538) 36 : cluster 0 pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: cluster 2024-04-02T04:45:14.321465+0000 osd.1 (osd.1) 33 : cluster 0 4.4 scrub starts 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: cluster 2024-04-02T04:45:14.322519+0000 osd.1 (osd.1) 34 : cluster 0 4.4 scrub ok 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: audit 2024-04-02T04:45:14.584492+0000 mon.a (mon.0) 849 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: audit 2024-04-02T04:45:14.595911+0000 mon.a (mon.0) 850 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: audit 2024-04-02T04:45:14.631511+0000 mon.a (mon.0) 851 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: audit 2024-04-02T04:45:14.638586+0000 mon.a (mon.0) 852 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:15.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:15 smithi098 bash[23790]: audit 2024-04-02T04:45:14.684923+0000 mon.b (mon.1) 60 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.595Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.595Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.597Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.598Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.628Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.629Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.632Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-02T04:45:15.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:15 smithi069 bash[57978]: ts=2024-04-02T04:45:15.632Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-02T04:45:16.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[21528]: cluster 2024-04-02T04:45:14.608558+0000 osd.4 (osd.4) 17 : cluster 0 4.9 scrub starts 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[21528]: cluster 2024-04-02T04:45:14.611111+0000 osd.4 (osd.4) 18 : cluster 0 4.9 scrub ok 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.351591+0000 osd.1 (osd.1) 35 : cluster 0 3.0 scrub starts 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.356905+0000 osd.1 (osd.1) 36 : cluster 0 3.0 scrub ok 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[28904]: cluster 2024-04-02T04:45:14.608558+0000 osd.4 (osd.4) 17 : cluster 0 4.9 scrub starts 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[28904]: cluster 2024-04-02T04:45:14.611111+0000 osd.4 (osd.4) 18 : cluster 0 4.9 scrub ok 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.351591+0000 osd.1 (osd.1) 35 : cluster 0 3.0 scrub starts 2024-04-02T04:45:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:16 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.356905+0000 osd.1 (osd.1) 36 : cluster 0 3.0 scrub ok 2024-04-02T04:45:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:16 smithi098 bash[23790]: cluster 2024-04-02T04:45:14.608558+0000 osd.4 (osd.4) 17 : cluster 0 4.9 scrub starts 2024-04-02T04:45:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:16 smithi098 bash[23790]: cluster 2024-04-02T04:45:14.611111+0000 osd.4 (osd.4) 18 : cluster 0 4.9 scrub ok 2024-04-02T04:45:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:16 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.351591+0000 osd.1 (osd.1) 35 : cluster 0 3.0 scrub starts 2024-04-02T04:45:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:16 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.356905+0000 osd.1 (osd.1) 36 : cluster 0 3.0 scrub ok 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.738035+0000 osd.0 (osd.0) 25 : cluster 0 3.1f scrub starts 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.738433+0000 mgr.y (mgr.14538) 37 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.742778+0000 osd.0 (osd.0) 26 : cluster 0 3.1f scrub ok 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.921387+0000 osd.6 (osd.6) 21 : cluster 0 5.2 scrub starts 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:15.924473+0000 osd.6 (osd.6) 22 : cluster 0 5.2 scrub ok 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.369964+0000 osd.1 (osd.1) 37 : cluster 0 2.15 scrub starts 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.372517+0000 osd.1 (osd.1) 38 : cluster 0 2.15 scrub ok 2024-04-02T04:45:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[21528]: audit 2024-04-02T04:45:16.834584+0000 mon.a (mon.0) 853 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.738035+0000 osd.0 (osd.0) 25 : cluster 0 3.1f scrub starts 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.738433+0000 mgr.y (mgr.14538) 37 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.742778+0000 osd.0 (osd.0) 26 : cluster 0 3.1f scrub ok 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.921387+0000 osd.6 (osd.6) 21 : cluster 0 5.2 scrub starts 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:15.924473+0000 osd.6 (osd.6) 22 : cluster 0 5.2 scrub ok 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.369964+0000 osd.1 (osd.1) 37 : cluster 0 2.15 scrub starts 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.372517+0000 osd.1 (osd.1) 38 : cluster 0 2.15 scrub ok 2024-04-02T04:45:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[28904]: audit 2024-04-02T04:45:16.834584+0000 mon.a (mon.0) 853 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:17.605 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.738035+0000 osd.0 (osd.0) 25 : cluster 0 3.1f scrub starts 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.738433+0000 mgr.y (mgr.14538) 37 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.742778+0000 osd.0 (osd.0) 26 : cluster 0 3.1f scrub ok 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.921387+0000 osd.6 (osd.6) 21 : cluster 0 5.2 scrub starts 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:15.924473+0000 osd.6 (osd.6) 22 : cluster 0 5.2 scrub ok 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.369964+0000 osd.1 (osd.1) 37 : cluster 0 2.15 scrub starts 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.372517+0000 osd.1 (osd.1) 38 : cluster 0 2.15 scrub ok 2024-04-02T04:45:17.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:17 smithi098 bash[23790]: audit 2024-04-02T04:45:16.834584+0000 mon.a (mon.0) 853 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:18.027 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:17 smithi069 bash[57978]: ts=2024-04-02T04:45:17.599Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000830545s 2024-04-02T04:45:18.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.601645+0000 osd.4 (osd.4) 19 : cluster 0 4.e scrub starts 2024-04-02T04:45:18.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.604958+0000 osd.4 (osd.4) 20 : cluster 0 4.e scrub ok 2024-04-02T04:45:18.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.719749+0000 osd.0 (osd.0) 27 : cluster 0 2.13 scrub starts 2024-04-02T04:45:18.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.722235+0000 osd.0 (osd.0) 28 : cluster 0 2.13 scrub ok 2024-04-02T04:45:18.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.731962+0000 osd.3 (osd.3) 29 : cluster 0 4.14 scrub starts 2024-04-02T04:45:18.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:16.734889+0000 osd.3 (osd.3) 30 : cluster 0 4.14 scrub ok 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.029003+0000 osd.5 (osd.5) 25 : cluster 0 6.19 scrub starts 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.031428+0000 osd.5 (osd.5) 26 : cluster 0 6.19 scrub ok 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.226936+0000 osd.7 (osd.7) 23 : cluster 0 6.5 scrub starts 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.229171+0000 osd.7 (osd.7) 24 : cluster 0 6.5 scrub ok 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.601645+0000 osd.4 (osd.4) 19 : cluster 0 4.e scrub starts 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.604958+0000 osd.4 (osd.4) 20 : cluster 0 4.e scrub ok 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.719749+0000 osd.0 (osd.0) 27 : cluster 0 2.13 scrub starts 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.722235+0000 osd.0 (osd.0) 28 : cluster 0 2.13 scrub ok 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.731962+0000 osd.3 (osd.3) 29 : cluster 0 4.14 scrub starts 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:16.734889+0000 osd.3 (osd.3) 30 : cluster 0 4.14 scrub ok 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.029003+0000 osd.5 (osd.5) 25 : cluster 0 6.19 scrub starts 2024-04-02T04:45:18.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.031428+0000 osd.5 (osd.5) 26 : cluster 0 6.19 scrub ok 2024-04-02T04:45:18.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.226936+0000 osd.7 (osd.7) 23 : cluster 0 6.5 scrub starts 2024-04-02T04:45:18.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:18 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.229171+0000 osd.7 (osd.7) 24 : cluster 0 6.5 scrub ok 2024-04-02T04:45:18.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.601645+0000 osd.4 (osd.4) 19 : cluster 0 4.e scrub starts 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.604958+0000 osd.4 (osd.4) 20 : cluster 0 4.e scrub ok 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.719749+0000 osd.0 (osd.0) 27 : cluster 0 2.13 scrub starts 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.722235+0000 osd.0 (osd.0) 28 : cluster 0 2.13 scrub ok 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.731962+0000 osd.3 (osd.3) 29 : cluster 0 4.14 scrub starts 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:16.734889+0000 osd.3 (osd.3) 30 : cluster 0 4.14 scrub ok 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.029003+0000 osd.5 (osd.5) 25 : cluster 0 6.19 scrub starts 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.031428+0000 osd.5 (osd.5) 26 : cluster 0 6.19 scrub ok 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.226936+0000 osd.7 (osd.7) 23 : cluster 0 6.5 scrub starts 2024-04-02T04:45:18.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:18 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.229171+0000 osd.7 (osd.7) 24 : cluster 0 6.5 scrub ok 2024-04-02T04:45:19.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.604901+0000 osd.4 (osd.4) 21 : cluster 0 4.1b scrub starts 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.606603+0000 osd.4 (osd.4) 22 : cluster 0 4.1b scrub ok 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.739519+0000 mgr.y (mgr.14538) 38 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.968064+0000 osd.6 (osd.6) 23 : cluster 0 2.11 scrub starts 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:17.970465+0000 osd.6 (osd.6) 24 : cluster 0 2.11 scrub ok 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:18.096822+0000 osd.2 (osd.2) 17 : cluster 0 6.f scrub starts 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:18.099029+0000 osd.2 (osd.2) 18 : cluster 0 6.f scrub ok 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:18.946100+0000 osd.5 (osd.5) 27 : cluster 0 2.12 scrub starts 2024-04-02T04:45:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:19 smithi098 bash[23790]: cluster 2024-04-02T04:45:18.948503+0000 osd.5 (osd.5) 28 : cluster 0 2.12 scrub ok 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.604901+0000 osd.4 (osd.4) 21 : cluster 0 4.1b scrub starts 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.606603+0000 osd.4 (osd.4) 22 : cluster 0 4.1b scrub ok 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.739519+0000 mgr.y (mgr.14538) 38 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.968064+0000 osd.6 (osd.6) 23 : cluster 0 2.11 scrub starts 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:17.970465+0000 osd.6 (osd.6) 24 : cluster 0 2.11 scrub ok 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:18.096822+0000 osd.2 (osd.2) 17 : cluster 0 6.f scrub starts 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:18.099029+0000 osd.2 (osd.2) 18 : cluster 0 6.f scrub ok 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:18.946100+0000 osd.5 (osd.5) 27 : cluster 0 2.12 scrub starts 2024-04-02T04:45:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[21528]: cluster 2024-04-02T04:45:18.948503+0000 osd.5 (osd.5) 28 : cluster 0 2.12 scrub ok 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.604901+0000 osd.4 (osd.4) 21 : cluster 0 4.1b scrub starts 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.606603+0000 osd.4 (osd.4) 22 : cluster 0 4.1b scrub ok 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.739519+0000 mgr.y (mgr.14538) 38 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.968064+0000 osd.6 (osd.6) 23 : cluster 0 2.11 scrub starts 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:17.970465+0000 osd.6 (osd.6) 24 : cluster 0 2.11 scrub ok 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:18.096822+0000 osd.2 (osd.2) 17 : cluster 0 6.f scrub starts 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:18.099029+0000 osd.2 (osd.2) 18 : cluster 0 6.f scrub ok 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:18.946100+0000 osd.5 (osd.5) 27 : cluster 0 2.12 scrub starts 2024-04-02T04:45:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:19 smithi069 bash[28904]: cluster 2024-04-02T04:45:18.948503+0000 osd.5 (osd.5) 28 : cluster 0 2.12 scrub ok 2024-04-02T04:45:20.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:20 smithi069 bash[21528]: cluster 2024-04-02T04:45:18.706694+0000 osd.0 (osd.0) 29 : cluster 0 3.12 scrub starts 2024-04-02T04:45:20.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:20 smithi069 bash[21528]: cluster 2024-04-02T04:45:18.710894+0000 osd.0 (osd.0) 30 : cluster 0 3.12 scrub ok 2024-04-02T04:45:20.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:20 smithi069 bash[28904]: cluster 2024-04-02T04:45:18.706694+0000 osd.0 (osd.0) 29 : cluster 0 3.12 scrub starts 2024-04-02T04:45:20.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:20 smithi069 bash[28904]: cluster 2024-04-02T04:45:18.710894+0000 osd.0 (osd.0) 30 : cluster 0 3.12 scrub ok 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/1290260850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0102e80 msgr2=0x7f09b0106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- 172.21.15.98:0/1290260850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0102e80 0x7f09b0106ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f099c009990 tx=0x7f099c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/1290260850 shutdown_connections 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- 172.21.15.98:0/1290260850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09b01075b0 0x7f09b010f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- 172.21.15.98:0/1290260850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 0x7f09b01028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- 172.21.15.98:0/1290260850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0102e80 0x7f09b0106ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/1290260850 >> 172.21.15.98:0/1290260850 conn(0x7f09b00fdd40 msgr2=0x7f09b0100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/1290260850 shutdown_connections 2024-04-02T04:45:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/1290260850 wait complete. 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 librados: starting msgr at 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 librados: starting objecter 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 Processor -- start 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- start start 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 librados: setting wanted keys 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 librados: calling monclient init 2024-04-02T04:45:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 0x7f09b0074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09b0102e80 0x7f09b0074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 0x7f09b0074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09b0111a80 con 0x7f09b0102e80 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09b0111900 con 0x7f09b01075b0 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f09b0111c00 con 0x7f09b01024d0 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 0x7f09b0074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 0x7f09b0074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52036/0 (socket says 172.21.15.98:52036) 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 -- 172.21.15.98:0/3701905798 learned_addr learned my addr 172.21.15.98:0/3701905798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09affff700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09b0102e80 0x7f09b0074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b4f69700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 0x7f09b0074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 -- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 msgr2=0x7f09b0074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 0x7f09b0074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 -- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09b0102e80 msgr2=0x7f09b0074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09b0102e80 0x7f09b0074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 -- 172.21.15.98:0/3701905798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b01a6610 con 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b4f69700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 0x7f09b0074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b576a700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 0x7f09b0074fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f09a8007ee0 tx=0x7f09a800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09adffb700 10 client.?.objecter ms_handle_connect 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09adffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a800dbd0 con 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f09a800dd70 con 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b01a6900 con 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80058d0 con 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 10 client.24481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b01a6df0 con 0x7f09b01075b0 2024-04-02T04:45:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 librados: init done 2024-04-02T04:45:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.651+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09b004ff00 con 0x7f09b01075b0 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f09a801a020 con 0x7f09b01075b0 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f09a006f5e0 0x7f09a0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 54..105) ==== 7421+0+0 (secure 0 0 0) 0x7f09a808d7f0 con 0x7f09b01075b0 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 10 client.24481.objecter ms_dispatch 0x7f09b0001550 osd_map(105..105 src has 54..105) 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 3 client.24481.objecter handle_osd_map got epochs [105,105] > 0 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 3 client.24481.objecter handle_osd_map decoding full epoch 105 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 20 client.24481.objecter dump_active .. 0 homeless 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09b4f69700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f09a006f5e0 0x7f09a0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09adffb700 10 client.24481.objecter ms_handle_connect 0x7f09a006f5e0 2024-04-02T04:45:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.655+0000 7f09b4f69700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f09a006f5e0 0x7f09a0071aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09a400bcf0 tx=0x7f09a400b2c0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:20.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.659+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f09a8010070 con 0x7f09b01075b0 2024-04-02T04:45:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:20 smithi098 bash[23790]: cluster 2024-04-02T04:45:18.706694+0000 osd.0 (osd.0) 29 : cluster 0 3.12 scrub starts 2024-04-02T04:45:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:20 smithi098 bash[23790]: cluster 2024-04-02T04:45:18.710894+0000 osd.0 (osd.0) 30 : cluster 0 3.12 scrub ok 2024-04-02T04:45:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.959+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f09b0075a70 con 0x7f09b01075b0 2024-04-02T04:45:20.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.963+0000 7f09adffb700 1 -- 172.21.15.98:0/3701905798 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) ==== 170+0+59 (secure 0 0 0) 0x7f09b0075a70 con 0x7f09b01075b0 2024-04-02T04:45:20.967 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-04-02T04:45:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBgjQtmnLmlORAAlYuVdHgGmXkcDE+jpJrV+w== 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 10 librados: watch_flush enter 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 10 librados: watch_flush exit 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 20 client.24481.objecter shutdown clearing up homeless session... 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 10 client.24481.objecter successfully canceled tick 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f09a006f5e0 msgr2=0x7f09a0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f09a006f5e0 0x7f09a0071aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09a400bcf0 tx=0x7f09a400b2c0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 msgr2=0x7f09b0074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 0x7f09b0074fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f09a8007ee0 tx=0x7f09a800d3b0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 shutdown_connections 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f09a006f5e0 0x7f09a0071aa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b01075b0 0x7f09b0074fc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09b01024d0 0x7f09b0074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 --2- 172.21.15.98:0/3701905798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09b0102e80 0x7f09b0074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 >> 172.21.15.98:0/3701905798 conn(0x7f09b00fdd40 msgr2=0x7f09b00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 shutdown_connections 2024-04-02T04:45:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 -- 172.21.15.98:0/3701905798 wait complete. 2024-04-02T04:45:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-02T04:45:20.967+0000 7f09b71cd700 1 librados: shutdown 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21782]: [02/Apr/2024:04:45:21] ENGINE Bus STOPPING 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21782]: [02/Apr/2024:04:45:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21782]: [02/Apr/2024:04:45:21] ENGINE Bus STOPPED 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21782]: [02/Apr/2024:04:45:21] ENGINE Bus STARTING 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21782]: [02/Apr/2024:04:45:21] ENGINE Serving on http://:::9283 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21782]: [02/Apr/2024:04:45:21] ENGINE Bus STARTED 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:19.667127+0000 osd.0 (osd.0) 31 : cluster 0 4.3 scrub starts 2024-04-02T04:45:21.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:19.669786+0000 osd.0 (osd.0) 32 : cluster 0 4.3 scrub ok 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:19.740199+0000 mgr.y (mgr.14538) 39 : cluster 0 pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:19.778315+0000 osd.3 (osd.3) 31 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:19.782086+0000 osd.3 (osd.3) 32 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.282580+0000 mon.a (mon.0) 854 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.295529+0000 mon.a (mon.0) 855 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.893235+0000 mon.a (mon.0) 856 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.909895+0000 mon.b (mon.1) 61 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.911208+0000 mon.a (mon.0) 857 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.911781+0000 mon.b (mon.1) 62 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.928850+0000 mon.a (mon.0) 858 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.960612+0000 mon.b (mon.1) 63 : audit 1 from='client.? 172.21.15.98:0/3701905798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.966787+0000 mon.a (mon.0) 859 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:21.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:20.970532+0000 mon.a (mon.0) 860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:20.980862+0000 osd.5 (osd.5) 29 : cluster 0 3.c deep-scrub starts 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: cluster 2024-04-02T04:45:20.987640+0000 osd.5 (osd.5) 30 : cluster 0 3.c deep-scrub ok 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:21.097303+0000 mon.a (mon.0) 861 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:21.105138+0000 mon.b (mon.1) 64 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:21.105560+0000 mon.a (mon.0) 862 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[28904]: audit 2024-04-02T04:45:21.118728+0000 mon.a (mon.0) 863 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:19.667127+0000 osd.0 (osd.0) 31 : cluster 0 4.3 scrub starts 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:19.669786+0000 osd.0 (osd.0) 32 : cluster 0 4.3 scrub ok 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:19.740199+0000 mgr.y (mgr.14538) 39 : cluster 0 pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:19.778315+0000 osd.3 (osd.3) 31 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:19.782086+0000 osd.3 (osd.3) 32 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:45:21.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.282580+0000 mon.a (mon.0) 854 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.295529+0000 mon.a (mon.0) 855 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.893235+0000 mon.a (mon.0) 856 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.909895+0000 mon.b (mon.1) 61 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.911208+0000 mon.a (mon.0) 857 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.911781+0000 mon.b (mon.1) 62 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.928850+0000 mon.a (mon.0) 858 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.960612+0000 mon.b (mon.1) 63 : audit 1 from='client.? 172.21.15.98:0/3701905798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.966787+0000 mon.a (mon.0) 859 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:20.970532+0000 mon.a (mon.0) 860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:20.980862+0000 osd.5 (osd.5) 29 : cluster 0 3.c deep-scrub starts 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: cluster 2024-04-02T04:45:20.987640+0000 osd.5 (osd.5) 30 : cluster 0 3.c deep-scrub ok 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:21.097303+0000 mon.a (mon.0) 861 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:21.105138+0000 mon.b (mon.1) 64 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:21.105560+0000 mon.a (mon.0) 862 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:21 smithi069 bash[21528]: audit 2024-04-02T04:45:21.118728+0000 mon.a (mon.0) 863 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:19.667127+0000 osd.0 (osd.0) 31 : cluster 0 4.3 scrub starts 2024-04-02T04:45:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:19.669786+0000 osd.0 (osd.0) 32 : cluster 0 4.3 scrub ok 2024-04-02T04:45:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:19.740199+0000 mgr.y (mgr.14538) 39 : cluster 0 pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:19.778315+0000 osd.3 (osd.3) 31 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:19.782086+0000 osd.3 (osd.3) 32 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.282580+0000 mon.a (mon.0) 854 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.295529+0000 mon.a (mon.0) 855 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.893235+0000 mon.a (mon.0) 856 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.909895+0000 mon.b (mon.1) 61 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.911208+0000 mon.a (mon.0) 857 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.911781+0000 mon.b (mon.1) 62 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.928850+0000 mon.a (mon.0) 858 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.960612+0000 mon.b (mon.1) 63 : audit 1 from='client.? 172.21.15.98:0/3701905798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.966787+0000 mon.a (mon.0) 859 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:20.970532+0000 mon.a (mon.0) 860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:45:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:20.980862+0000 osd.5 (osd.5) 29 : cluster 0 3.c deep-scrub starts 2024-04-02T04:45:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: cluster 2024-04-02T04:45:20.987640+0000 osd.5 (osd.5) 30 : cluster 0 3.c deep-scrub ok 2024-04-02T04:45:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:21.097303+0000 mon.a (mon.0) 861 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:21.105138+0000 mon.b (mon.1) 64 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:21.105560+0000 mon.a (mon.0) 862 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:21 smithi098 bash[23790]: audit 2024-04-02T04:45:21.118728+0000 mon.a (mon.0) 863 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:21.747 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-02T04:45:21.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-02T04:45:21.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-02T04:45:21.769 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T04:45:21.769 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-02T04:45:21.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-02T04:45:22.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:21 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[23790]: cluster 2024-04-02T04:45:20.623512+0000 osd.4 (osd.4) 23 : cluster 0 4.1a scrub starts 2024-04-02T04:45:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[23790]: cluster 2024-04-02T04:45:20.625216+0000 osd.4 (osd.4) 24 : cluster 0 4.1a scrub ok 2024-04-02T04:45:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[23790]: cephadm 2024-04-02T04:45:20.970000+0000 mgr.y (mgr.14538) 40 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-04-02T04:45:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[23790]: audit 2024-04-02T04:45:21.111895+0000 mgr.y (mgr.14538) 41 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[23790]: cephadm 2024-04-02T04:45:21.145262+0000 mgr.y (mgr.14538) 42 : cephadm 1 Deploying daemon grafana.a on smithi098 2024-04-02T04:45:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[23790]: audit 2024-04-02T04:45:21.883041+0000 mon.b (mon.1) 65 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[21528]: cluster 2024-04-02T04:45:20.623512+0000 osd.4 (osd.4) 23 : cluster 0 4.1a scrub starts 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[21528]: cluster 2024-04-02T04:45:20.625216+0000 osd.4 (osd.4) 24 : cluster 0 4.1a scrub ok 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[21528]: cephadm 2024-04-02T04:45:20.970000+0000 mgr.y (mgr.14538) 40 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[21528]: audit 2024-04-02T04:45:21.111895+0000 mgr.y (mgr.14538) 41 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[21528]: cephadm 2024-04-02T04:45:21.145262+0000 mgr.y (mgr.14538) 42 : cephadm 1 Deploying daemon grafana.a on smithi098 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[21528]: audit 2024-04-02T04:45:21.883041+0000 mon.b (mon.1) 65 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[28904]: cluster 2024-04-02T04:45:20.623512+0000 osd.4 (osd.4) 23 : cluster 0 4.1a scrub starts 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[28904]: cluster 2024-04-02T04:45:20.625216+0000 osd.4 (osd.4) 24 : cluster 0 4.1a scrub ok 2024-04-02T04:45:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[28904]: cephadm 2024-04-02T04:45:20.970000+0000 mgr.y (mgr.14538) 40 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-04-02T04:45:22.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[28904]: audit 2024-04-02T04:45:21.111895+0000 mgr.y (mgr.14538) 41 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:22.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[28904]: cephadm 2024-04-02T04:45:21.145262+0000 mgr.y (mgr.14538) 42 : cephadm 1 Deploying daemon grafana.a on smithi098 2024-04-02T04:45:22.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:22 smithi069 bash[28904]: audit 2024-04-02T04:45:21.883041+0000 mon.b (mon.1) 65 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:23.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:22 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:45:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:23 smithi098 bash[23790]: cluster 2024-04-02T04:45:21.740812+0000 mgr.y (mgr.14538) 43 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:23 smithi098 bash[23790]: cluster 2024-04-02T04:45:21.746787+0000 osd.3 (osd.3) 33 : cluster 0 5.0 scrub starts 2024-04-02T04:45:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:23 smithi098 bash[23790]: cluster 2024-04-02T04:45:21.749113+0000 osd.3 (osd.3) 34 : cluster 0 5.0 scrub ok 2024-04-02T04:45:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:23 smithi098 bash[23790]: cluster 2024-04-02T04:45:22.349782+0000 osd.1 (osd.1) 39 : cluster 0 6.1d scrub starts 2024-04-02T04:45:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:23 smithi098 bash[23790]: cluster 2024-04-02T04:45:22.352110+0000 osd.1 (osd.1) 40 : cluster 0 6.1d scrub ok 2024-04-02T04:45:23.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[21528]: cluster 2024-04-02T04:45:21.740812+0000 mgr.y (mgr.14538) 43 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[21528]: cluster 2024-04-02T04:45:21.746787+0000 osd.3 (osd.3) 33 : cluster 0 5.0 scrub starts 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[21528]: cluster 2024-04-02T04:45:21.749113+0000 osd.3 (osd.3) 34 : cluster 0 5.0 scrub ok 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[21528]: cluster 2024-04-02T04:45:22.349782+0000 osd.1 (osd.1) 39 : cluster 0 6.1d scrub starts 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[21528]: cluster 2024-04-02T04:45:22.352110+0000 osd.1 (osd.1) 40 : cluster 0 6.1d scrub ok 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[28904]: cluster 2024-04-02T04:45:21.740812+0000 mgr.y (mgr.14538) 43 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[28904]: cluster 2024-04-02T04:45:21.746787+0000 osd.3 (osd.3) 33 : cluster 0 5.0 scrub starts 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[28904]: cluster 2024-04-02T04:45:21.749113+0000 osd.3 (osd.3) 34 : cluster 0 5.0 scrub ok 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[28904]: cluster 2024-04-02T04:45:22.349782+0000 osd.1 (osd.1) 39 : cluster 0 6.1d scrub starts 2024-04-02T04:45:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:23 smithi069 bash[28904]: cluster 2024-04-02T04:45:22.352110+0000 osd.1 (osd.1) 40 : cluster 0 6.1d scrub ok 2024-04-02T04:45:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:22.706994+0000 osd.3 (osd.3) 35 : cluster 0 2.19 deep-scrub starts 2024-04-02T04:45:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:22.711756+0000 osd.3 (osd.3) 36 : cluster 0 2.19 deep-scrub ok 2024-04-02T04:45:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: audit 2024-04-02T04:45:22.766092+0000 mgr.y (mgr.14538) 44 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.146946+0000 osd.2 (osd.2) 19 : cluster 0 2.c deep-scrub starts 2024-04-02T04:45:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.149585+0000 osd.2 (osd.2) 20 : cluster 0 2.c deep-scrub ok 2024-04-02T04:45:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.306407+0000 osd.1 (osd.1) 41 : cluster 0 3.19 scrub starts 2024-04-02T04:45:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.311851+0000 osd.1 (osd.1) 42 : cluster 0 3.19 scrub ok 2024-04-02T04:45:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.344313+0000 osd.7 (osd.7) 25 : cluster 0 3.15 scrub starts 2024-04-02T04:45:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:24 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.347588+0000 osd.7 (osd.7) 26 : cluster 0 3.15 scrub ok 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:22.706994+0000 osd.3 (osd.3) 35 : cluster 0 2.19 deep-scrub starts 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:22.711756+0000 osd.3 (osd.3) 36 : cluster 0 2.19 deep-scrub ok 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: audit 2024-04-02T04:45:22.766092+0000 mgr.y (mgr.14538) 44 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.146946+0000 osd.2 (osd.2) 19 : cluster 0 2.c deep-scrub starts 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.149585+0000 osd.2 (osd.2) 20 : cluster 0 2.c deep-scrub ok 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.306407+0000 osd.1 (osd.1) 41 : cluster 0 3.19 scrub starts 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.311851+0000 osd.1 (osd.1) 42 : cluster 0 3.19 scrub ok 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.344313+0000 osd.7 (osd.7) 25 : cluster 0 3.15 scrub starts 2024-04-02T04:45:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.347588+0000 osd.7 (osd.7) 26 : cluster 0 3.15 scrub ok 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:22.706994+0000 osd.3 (osd.3) 35 : cluster 0 2.19 deep-scrub starts 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:22.711756+0000 osd.3 (osd.3) 36 : cluster 0 2.19 deep-scrub ok 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: audit 2024-04-02T04:45:22.766092+0000 mgr.y (mgr.14538) 44 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.146946+0000 osd.2 (osd.2) 19 : cluster 0 2.c deep-scrub starts 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.149585+0000 osd.2 (osd.2) 20 : cluster 0 2.c deep-scrub ok 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.306407+0000 osd.1 (osd.1) 41 : cluster 0 3.19 scrub starts 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.311851+0000 osd.1 (osd.1) 42 : cluster 0 3.19 scrub ok 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.344313+0000 osd.7 (osd.7) 25 : cluster 0 3.15 scrub starts 2024-04-02T04:45:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:24 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.347588+0000 osd.7 (osd.7) 26 : cluster 0 3.15 scrub ok 2024-04-02T04:45:25.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.692679+0000 osd.4 (osd.4) 25 : cluster 0 4.d scrub starts 2024-04-02T04:45:25.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.695191+0000 osd.4 (osd.4) 26 : cluster 0 4.d scrub ok 2024-04-02T04:45:25.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:23.742261+0000 mgr.y (mgr.14538) 45 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:25.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:24.189511+0000 osd.2 (osd.2) 21 : cluster 0 4.1c scrub starts 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:24.191983+0000 osd.2 (osd.2) 22 : cluster 0 4.1c scrub ok 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:24.288321+0000 osd.1 (osd.1) 43 : cluster 0 4.f scrub starts 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[21528]: cluster 2024-04-02T04:45:24.290734+0000 osd.1 (osd.1) 44 : cluster 0 4.f scrub ok 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.692679+0000 osd.4 (osd.4) 25 : cluster 0 4.d scrub starts 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.695191+0000 osd.4 (osd.4) 26 : cluster 0 4.d scrub ok 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:23.742261+0000 mgr.y (mgr.14538) 45 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:24.189511+0000 osd.2 (osd.2) 21 : cluster 0 4.1c scrub starts 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:24.191983+0000 osd.2 (osd.2) 22 : cluster 0 4.1c scrub ok 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:24.288321+0000 osd.1 (osd.1) 43 : cluster 0 4.f scrub starts 2024-04-02T04:45:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[28904]: cluster 2024-04-02T04:45:24.290734+0000 osd.1 (osd.1) 44 : cluster 0 4.f scrub ok 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.692679+0000 osd.4 (osd.4) 25 : cluster 0 4.d scrub starts 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.695191+0000 osd.4 (osd.4) 26 : cluster 0 4.d scrub ok 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:23.742261+0000 mgr.y (mgr.14538) 45 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:24.189511+0000 osd.2 (osd.2) 21 : cluster 0 4.1c scrub starts 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:24.191983+0000 osd.2 (osd.2) 22 : cluster 0 4.1c scrub ok 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:24.288321+0000 osd.1 (osd.1) 43 : cluster 0 4.f scrub starts 2024-04-02T04:45:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:25 smithi098 bash[23790]: cluster 2024-04-02T04:45:24.290734+0000 osd.1 (osd.1) 44 : cluster 0 4.f scrub ok 2024-04-02T04:45:26.027 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:25 smithi069 bash[57978]: ts=2024-04-02T04:45:25.601Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003169286s 2024-04-02T04:45:26.542 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:26 smithi098 bash[23790]: cluster 2024-04-02T04:45:25.231556+0000 osd.2 (osd.2) 23 : cluster 0 2.e scrub starts 2024-04-02T04:45:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:26 smithi098 bash[23790]: cluster 2024-04-02T04:45:25.234013+0000 osd.2 (osd.2) 24 : cluster 0 2.e scrub ok 2024-04-02T04:45:26.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:26 smithi069 bash[21528]: cluster 2024-04-02T04:45:25.231556+0000 osd.2 (osd.2) 23 : cluster 0 2.e scrub starts 2024-04-02T04:45:26.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:26 smithi069 bash[21528]: cluster 2024-04-02T04:45:25.234013+0000 osd.2 (osd.2) 24 : cluster 0 2.e scrub ok 2024-04-02T04:45:26.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:26 smithi069 bash[28904]: cluster 2024-04-02T04:45:25.231556+0000 osd.2 (osd.2) 23 : cluster 0 2.e scrub starts 2024-04-02T04:45:26.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:26 smithi069 bash[28904]: cluster 2024-04-02T04:45:25.234013+0000 osd.2 (osd.2) 24 : cluster 0 2.e scrub ok 2024-04-02T04:45:27.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[21528]: cluster 2024-04-02T04:45:25.742956+0000 mgr.y (mgr.14538) 46 : cluster 0 pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:27.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.319001+0000 osd.1 (osd.1) 45 : cluster 0 4.12 scrub starts 2024-04-02T04:45:27.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.322094+0000 osd.1 (osd.1) 46 : cluster 0 4.12 scrub ok 2024-04-02T04:45:27.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.942877+0000 osd.6 (osd.6) 25 : cluster 0 3.10 scrub starts 2024-04-02T04:45:27.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.945679+0000 osd.6 (osd.6) 26 : cluster 0 3.10 scrub ok 2024-04-02T04:45:27.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[28904]: cluster 2024-04-02T04:45:25.742956+0000 mgr.y (mgr.14538) 46 : cluster 0 pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:27.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.319001+0000 osd.1 (osd.1) 45 : cluster 0 4.12 scrub starts 2024-04-02T04:45:27.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.322094+0000 osd.1 (osd.1) 46 : cluster 0 4.12 scrub ok 2024-04-02T04:45:27.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.942877+0000 osd.6 (osd.6) 25 : cluster 0 3.10 scrub starts 2024-04-02T04:45:27.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:27 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.945679+0000 osd.6 (osd.6) 26 : cluster 0 3.10 scrub ok 2024-04-02T04:45:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:27 smithi098 bash[23790]: cluster 2024-04-02T04:45:25.742956+0000 mgr.y (mgr.14538) 46 : cluster 0 pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:27 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.319001+0000 osd.1 (osd.1) 45 : cluster 0 4.12 scrub starts 2024-04-02T04:45:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:27 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.322094+0000 osd.1 (osd.1) 46 : cluster 0 4.12 scrub ok 2024-04-02T04:45:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:27 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.942877+0000 osd.6 (osd.6) 25 : cluster 0 3.10 scrub starts 2024-04-02T04:45:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:27 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.945679+0000 osd.6 (osd.6) 26 : cluster 0 3.10 scrub ok 2024-04-02T04:45:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:28 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.629010+0000 osd.4 (osd.4) 27 : cluster 0 6.2 scrub starts 2024-04-02T04:45:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:28 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.630928+0000 osd.4 (osd.4) 28 : cluster 0 6.2 scrub ok 2024-04-02T04:45:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:28 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.745056+0000 osd.3 (osd.3) 37 : cluster 0 6.c deep-scrub starts 2024-04-02T04:45:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:28 smithi098 bash[23790]: cluster 2024-04-02T04:45:26.746918+0000 osd.3 (osd.3) 38 : cluster 0 6.c deep-scrub ok 2024-04-02T04:45:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:28 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.898728+0000 osd.5 (osd.5) 31 : cluster 0 3.1d scrub starts 2024-04-02T04:45:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:28 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.901647+0000 osd.5 (osd.5) 32 : cluster 0 3.1d scrub ok 2024-04-02T04:45:28.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.629010+0000 osd.4 (osd.4) 27 : cluster 0 6.2 scrub starts 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.630928+0000 osd.4 (osd.4) 28 : cluster 0 6.2 scrub ok 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.745056+0000 osd.3 (osd.3) 37 : cluster 0 6.c deep-scrub starts 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[21528]: cluster 2024-04-02T04:45:26.746918+0000 osd.3 (osd.3) 38 : cluster 0 6.c deep-scrub ok 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.898728+0000 osd.5 (osd.5) 31 : cluster 0 3.1d scrub starts 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.901647+0000 osd.5 (osd.5) 32 : cluster 0 3.1d scrub ok 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.629010+0000 osd.4 (osd.4) 27 : cluster 0 6.2 scrub starts 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.630928+0000 osd.4 (osd.4) 28 : cluster 0 6.2 scrub ok 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.745056+0000 osd.3 (osd.3) 37 : cluster 0 6.c deep-scrub starts 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[28904]: cluster 2024-04-02T04:45:26.746918+0000 osd.3 (osd.3) 38 : cluster 0 6.c deep-scrub ok 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.898728+0000 osd.5 (osd.5) 31 : cluster 0 3.1d scrub starts 2024-04-02T04:45:28.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:28 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.901647+0000 osd.5 (osd.5) 32 : cluster 0 3.1d scrub ok 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/2131040487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 msgr2=0x7fd5a406b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/2131040487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a406b030 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5940098f0 tx=0x7fd594009c00 comp rx=0 tx=0).stop 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/2131040487 shutdown_connections 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/2131040487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 0x7fd5a410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/2131040487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a406b030 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/2131040487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 0x7fd5a41043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/2131040487 >> 172.21.15.69:0/2131040487 conn(0x7fd5a40771e0 msgr2=0x7fd5a40775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/2131040487 shutdown_connections 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/2131040487 wait complete. 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 librados: starting msgr at 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 librados: starting objecter 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 Processor -- start 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- start start 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 librados: setting wanted keys 2024-04-02T04:45:29.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 librados: calling monclient init 2024-04-02T04:45:29.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a4198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:29.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 0x7fd5a41987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 0x7fd5a419cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a4111c20 con 0x7fd5a4104920 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a4111aa0 con 0x7fd5a4103f60 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a4111da0 con 0x7fd5a406ac50 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a259c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a4198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 0x7fd5a419cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a259c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a4198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48512/0 (socket says 172.21.15.69:48512) 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a259c700 1 -- 172.21.15.69:0/1124163785 learned_addr learned my addr 172.21.15.69:0/1124163785 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a1d9b700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 0x7fd5a41987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 -- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 msgr2=0x7fd5a4198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a4198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 -- 172.21.15.69:0/1124163785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 msgr2=0x7fd5a41987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:29.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 0x7fd5a41987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 -- 172.21.15.69:0/1124163785 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a419d270 con 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a259c700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a4198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a1d9b700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 0x7fd5a41987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a2d9d700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 0x7fd5a419cb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009cd0 tx=0x7fd59c00c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd59b7fe700 10 client.?.objecter ms_handle_connect 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd59b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00cdc0 con 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a419d500 con 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd59c003a00 con 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0044f0 con 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 10 client.14589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a41a4de0 con 0x7fd5a4104920 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 librados: init done 2024-04-02T04:45:29.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.272+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a404ff00 con 0x7fd5a4104920 2024-04-02T04:45:29.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd59c01a020 con 0x7fd5a4104920 2024-04-02T04:45:29.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd59006f5e0 0x7fd590071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:29.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 54..105) ==== 7421+0+0 (secure 0 0 0) 0x7fd59c08d940 con 0x7fd5a4104920 2024-04-02T04:45:29.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 10 client.14589.objecter ms_dispatch 0x7fd5a4001550 osd_map(105..105 src has 54..105) 2024-04-02T04:45:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 3 client.14589.objecter handle_osd_map got epochs [105,105] > 0 2024-04-02T04:45:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 3 client.14589.objecter handle_osd_map decoding full epoch 105 2024-04-02T04:45:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd5a259c700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd59006f5e0 0x7fd590071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 20 client.14589.objecter dump_active .. 0 homeless 2024-04-02T04:45:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd59b7fe700 10 client.14589.objecter ms_handle_connect 0x7fd59006f5e0 2024-04-02T04:45:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.276+0000 7fd5a259c700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd59006f5e0 0x7fd590071aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd59400c010 tx=0x7fd594019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.280+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd59c010040 con 0x7fd5a4104920 2024-04-02T04:45:29.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.571571+0000 osd.0 (osd.0) 33 : cluster 0 4.16 scrub starts 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.572975+0000 osd.0 (osd.0) 34 : cluster 0 4.16 scrub ok 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.667174+0000 osd.4 (osd.4) 29 : cluster 0 3.14 scrub starts 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.671061+0000 osd.4 (osd.4) 30 : cluster 0 3.14 scrub ok 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.744247+0000 mgr.y (mgr.14538) 47 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.762438+0000 osd.3 (osd.3) 39 : cluster 0 3.7 scrub starts 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:27.767537+0000 osd.3 (osd.3) 40 : cluster 0 3.7 scrub ok 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:28.274979+0000 osd.2 (osd.2) 25 : cluster 0 2.10 scrub starts 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:28.276051+0000 osd.2 (osd.2) 26 : cluster 0 2.10 scrub ok 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:28.308809+0000 osd.1 (osd.1) 47 : cluster 0 6.1 scrub starts 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[28904]: cluster 2024-04-02T04:45:28.310005+0000 osd.1 (osd.1) 48 : cluster 0 6.1 scrub ok 2024-04-02T04:45:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.571571+0000 osd.0 (osd.0) 33 : cluster 0 4.16 scrub starts 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.572975+0000 osd.0 (osd.0) 34 : cluster 0 4.16 scrub ok 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.667174+0000 osd.4 (osd.4) 29 : cluster 0 3.14 scrub starts 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.671061+0000 osd.4 (osd.4) 30 : cluster 0 3.14 scrub ok 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.744247+0000 mgr.y (mgr.14538) 47 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.762438+0000 osd.3 (osd.3) 39 : cluster 0 3.7 scrub starts 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:27.767537+0000 osd.3 (osd.3) 40 : cluster 0 3.7 scrub ok 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:28.274979+0000 osd.2 (osd.2) 25 : cluster 0 2.10 scrub starts 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:28.276051+0000 osd.2 (osd.2) 26 : cluster 0 2.10 scrub ok 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:28.308809+0000 osd.1 (osd.1) 47 : cluster 0 6.1 scrub starts 2024-04-02T04:45:29.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:29 smithi069 bash[21528]: cluster 2024-04-02T04:45:28.310005+0000 osd.1 (osd.1) 48 : cluster 0 6.1 scrub ok 2024-04-02T04:45:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.548+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd5a41a4fb0 con 0x7fd5a4104920 2024-04-02T04:45:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.552+0000 7fd59b7fe700 1 -- 172.21.15.69:0/1124163785 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v22) ==== 74+0+175412 (secure 0 0 0) 0x7fd59c05afa0 con 0x7fd5a4104920 2024-04-02T04:45:29.555 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:45:29.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 10 librados: watch_flush enter 2024-04-02T04:45:29.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 10 librados: watch_flush exit 2024-04-02T04:45:29.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 20 client.14589.objecter shutdown clearing up homeless session... 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 10 client.14589.objecter successfully canceled tick 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd59006f5e0 msgr2=0x7fd590071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd59006f5e0 0x7fd590071aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd59400c010 tx=0x7fd594019040 comp rx=0 tx=0).stop 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 msgr2=0x7fd5a419cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 0x7fd5a419cb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009cd0 tx=0x7fd59c00c620 comp rx=0 tx=0).stop 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 shutdown_connections 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd59006f5e0 0x7fd590071aa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5a4104920 0x7fd5a419cb30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd5a406ac50 0x7fd5a4198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 --2- 172.21.15.69:0/1124163785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5a4103f60 0x7fd5a41987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 >> 172.21.15.69:0/1124163785 conn(0x7fd5a40771e0 msgr2=0x7fd5a4109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:29.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 shutdown_connections 2024-04-02T04:45:29.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 -- 172.21.15.69:0/1124163785 wait complete. 2024-04-02T04:45:29.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:29.556+0000 7fd5a8a76700 1 librados: shutdown 2024-04-02T04:45:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.571571+0000 osd.0 (osd.0) 33 : cluster 0 4.16 scrub starts 2024-04-02T04:45:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.572975+0000 osd.0 (osd.0) 34 : cluster 0 4.16 scrub ok 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.667174+0000 osd.4 (osd.4) 29 : cluster 0 3.14 scrub starts 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.671061+0000 osd.4 (osd.4) 30 : cluster 0 3.14 scrub ok 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.744247+0000 mgr.y (mgr.14538) 47 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.762438+0000 osd.3 (osd.3) 39 : cluster 0 3.7 scrub starts 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:27.767537+0000 osd.3 (osd.3) 40 : cluster 0 3.7 scrub ok 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:28.274979+0000 osd.2 (osd.2) 25 : cluster 0 2.10 scrub starts 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:28.276051+0000 osd.2 (osd.2) 26 : cluster 0 2.10 scrub ok 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:28.308809+0000 osd.1 (osd.1) 47 : cluster 0 6.1 scrub starts 2024-04-02T04:45:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:29 smithi098 bash[23790]: cluster 2024-04-02T04:45:28.310005+0000 osd.1 (osd.1) 48 : cluster 0 6.1 scrub ok 2024-04-02T04:45:30.283 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"flags":0,"active_gid":14538,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1004417273},{"type":"v1","addr":"172.21.15.69:6801","nonce":1004417273}]},"active_addr":"172.21.15.69:6801/1004417273","active_change":"2024-04-02T04:44:51.699797+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14541,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.69:8443/","prometheus":"http://172.21.15.69:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":100,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1362582507}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1578751521}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2252228867}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1043847261}]}]} 2024-04-02T04:45:30.287 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-02T04:45:30.287 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-02T04:45:30.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T04:45:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:30 smithi098 bash[23790]: cluster 2024-04-02T04:45:28.774675+0000 osd.3 (osd.3) 41 : cluster 0 6.11 scrub starts 2024-04-02T04:45:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:30 smithi098 bash[23790]: cluster 2024-04-02T04:45:28.776979+0000 osd.3 (osd.3) 42 : cluster 0 6.11 scrub ok 2024-04-02T04:45:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:30 smithi098 bash[23790]: audit 2024-04-02T04:45:29.555246+0000 mon.a (mon.0) 864 : audit 0 from='client.? 172.21.15.69:0/1124163785' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T04:45:30.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:30 smithi069 bash[21528]: cluster 2024-04-02T04:45:28.774675+0000 osd.3 (osd.3) 41 : cluster 0 6.11 scrub starts 2024-04-02T04:45:30.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:30 smithi069 bash[21528]: cluster 2024-04-02T04:45:28.776979+0000 osd.3 (osd.3) 42 : cluster 0 6.11 scrub ok 2024-04-02T04:45:30.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:30 smithi069 bash[21528]: audit 2024-04-02T04:45:29.555246+0000 mon.a (mon.0) 864 : audit 0 from='client.? 172.21.15.69:0/1124163785' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T04:45:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:30 smithi069 bash[28904]: cluster 2024-04-02T04:45:28.774675+0000 osd.3 (osd.3) 41 : cluster 0 6.11 scrub starts 2024-04-02T04:45:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:30 smithi069 bash[28904]: cluster 2024-04-02T04:45:28.776979+0000 osd.3 (osd.3) 42 : cluster 0 6.11 scrub ok 2024-04-02T04:45:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:30 smithi069 bash[28904]: audit 2024-04-02T04:45:29.555246+0000 mon.a (mon.0) 864 : audit 0 from='client.? 172.21.15.69:0/1124163785' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T04:45:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:29.627691+0000 osd.4 (osd.4) 31 : cluster 0 3.3 scrub starts 2024-04-02T04:45:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:29.633064+0000 osd.4 (osd.4) 32 : cluster 0 3.3 scrub ok 2024-04-02T04:45:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:29.745136+0000 mgr.y (mgr.14538) 48 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:29.801975+0000 osd.3 (osd.3) 43 : cluster 0 6.1b deep-scrub starts 2024-04-02T04:45:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:29.804203+0000 osd.3 (osd.3) 44 : cluster 0 6.1b deep-scrub ok 2024-04-02T04:45:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:30.243271+0000 osd.1 (osd.1) 49 : cluster 0 6.4 scrub starts 2024-04-02T04:45:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:30.245070+0000 osd.1 (osd.1) 50 : cluster 0 6.4 scrub ok 2024-04-02T04:45:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:30.358113+0000 osd.7 (osd.7) 27 : cluster 0 2.0 scrub starts 2024-04-02T04:45:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:31 smithi098 bash[23790]: cluster 2024-04-02T04:45:30.360362+0000 osd.7 (osd.7) 28 : cluster 0 2.0 scrub ok 2024-04-02T04:45:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:29.627691+0000 osd.4 (osd.4) 31 : cluster 0 3.3 scrub starts 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:29.633064+0000 osd.4 (osd.4) 32 : cluster 0 3.3 scrub ok 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:29.745136+0000 mgr.y (mgr.14538) 48 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:29.801975+0000 osd.3 (osd.3) 43 : cluster 0 6.1b deep-scrub starts 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:29.804203+0000 osd.3 (osd.3) 44 : cluster 0 6.1b deep-scrub ok 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:30.243271+0000 osd.1 (osd.1) 49 : cluster 0 6.4 scrub starts 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:30.245070+0000 osd.1 (osd.1) 50 : cluster 0 6.4 scrub ok 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:30.358113+0000 osd.7 (osd.7) 27 : cluster 0 2.0 scrub starts 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[21528]: cluster 2024-04-02T04:45:30.360362+0000 osd.7 (osd.7) 28 : cluster 0 2.0 scrub ok 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:29.627691+0000 osd.4 (osd.4) 31 : cluster 0 3.3 scrub starts 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:29.633064+0000 osd.4 (osd.4) 32 : cluster 0 3.3 scrub ok 2024-04-02T04:45:31.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:29.745136+0000 mgr.y (mgr.14538) 48 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:29.801975+0000 osd.3 (osd.3) 43 : cluster 0 6.1b deep-scrub starts 2024-04-02T04:45:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:29.804203+0000 osd.3 (osd.3) 44 : cluster 0 6.1b deep-scrub ok 2024-04-02T04:45:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:30.243271+0000 osd.1 (osd.1) 49 : cluster 0 6.4 scrub starts 2024-04-02T04:45:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:30.245070+0000 osd.1 (osd.1) 50 : cluster 0 6.4 scrub ok 2024-04-02T04:45:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:30.358113+0000 osd.7 (osd.7) 27 : cluster 0 2.0 scrub starts 2024-04-02T04:45:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:31 smithi069 bash[28904]: cluster 2024-04-02T04:45:30.360362+0000 osd.7 (osd.7) 28 : cluster 0 2.0 scrub ok 2024-04-02T04:45:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:32 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.344482+0000 osd.7 (osd.7) 29 : cluster 0 6.15 scrub starts 2024-04-02T04:45:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:32 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.347026+0000 osd.7 (osd.7) 30 : cluster 0 6.15 scrub ok 2024-04-02T04:45:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:32 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.930704+0000 osd.6 (osd.6) 27 : cluster 0 4.a scrub starts 2024-04-02T04:45:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:32 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.932902+0000 osd.6 (osd.6) 28 : cluster 0 4.a scrub ok 2024-04-02T04:45:32.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.344482+0000 osd.7 (osd.7) 29 : cluster 0 6.15 scrub starts 2024-04-02T04:45:32.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.347026+0000 osd.7 (osd.7) 30 : cluster 0 6.15 scrub ok 2024-04-02T04:45:32.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.930704+0000 osd.6 (osd.6) 27 : cluster 0 4.a scrub starts 2024-04-02T04:45:32.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.932902+0000 osd.6 (osd.6) 28 : cluster 0 4.a scrub ok 2024-04-02T04:45:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.344482+0000 osd.7 (osd.7) 29 : cluster 0 6.15 scrub starts 2024-04-02T04:45:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.347026+0000 osd.7 (osd.7) 30 : cluster 0 6.15 scrub ok 2024-04-02T04:45:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.930704+0000 osd.6 (osd.6) 27 : cluster 0 4.a scrub starts 2024-04-02T04:45:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:32 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.932902+0000 osd.6 (osd.6) 28 : cluster 0 4.a scrub ok 2024-04-02T04:45:33.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:32 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:45:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:33 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.745832+0000 mgr.y (mgr.14538) 49 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:33 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.756121+0000 osd.3 (osd.3) 45 : cluster 0 6.b scrub starts 2024-04-02T04:45:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:33 smithi098 bash[23790]: cluster 2024-04-02T04:45:31.758478+0000 osd.3 (osd.3) 46 : cluster 0 6.b scrub ok 2024-04-02T04:45:33.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:33 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.745832+0000 mgr.y (mgr.14538) 49 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:33.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:33 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.756121+0000 osd.3 (osd.3) 45 : cluster 0 6.b scrub starts 2024-04-02T04:45:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:33 smithi069 bash[21528]: cluster 2024-04-02T04:45:31.758478+0000 osd.3 (osd.3) 46 : cluster 0 6.b scrub ok 2024-04-02T04:45:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:33 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.745832+0000 mgr.y (mgr.14538) 49 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:33 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.756121+0000 osd.3 (osd.3) 45 : cluster 0 6.b scrub starts 2024-04-02T04:45:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:33 smithi069 bash[28904]: cluster 2024-04-02T04:45:31.758478+0000 osd.3 (osd.3) 46 : cluster 0 6.b scrub ok 2024-04-02T04:45:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:34 smithi098 bash[23790]: cluster 2024-04-02T04:45:32.672812+0000 osd.4 (osd.4) 33 : cluster 0 4.c deep-scrub starts 2024-04-02T04:45:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:34 smithi098 bash[23790]: cluster 2024-04-02T04:45:32.675290+0000 osd.4 (osd.4) 34 : cluster 0 4.c deep-scrub ok 2024-04-02T04:45:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:34 smithi098 bash[23790]: audit 2024-04-02T04:45:32.773147+0000 mgr.y (mgr.14538) 50 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:34 smithi098 bash[23790]: cluster 2024-04-02T04:45:34.003670+0000 osd.6 (osd.6) 29 : cluster 0 4.1f scrub starts 2024-04-02T04:45:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:34 smithi098 bash[23790]: cluster 2024-04-02T04:45:34.007097+0000 osd.6 (osd.6) 30 : cluster 0 4.1f scrub ok 2024-04-02T04:45:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[21528]: cluster 2024-04-02T04:45:32.672812+0000 osd.4 (osd.4) 33 : cluster 0 4.c deep-scrub starts 2024-04-02T04:45:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[21528]: cluster 2024-04-02T04:45:32.675290+0000 osd.4 (osd.4) 34 : cluster 0 4.c deep-scrub ok 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[21528]: audit 2024-04-02T04:45:32.773147+0000 mgr.y (mgr.14538) 50 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[21528]: cluster 2024-04-02T04:45:34.003670+0000 osd.6 (osd.6) 29 : cluster 0 4.1f scrub starts 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[21528]: cluster 2024-04-02T04:45:34.007097+0000 osd.6 (osd.6) 30 : cluster 0 4.1f scrub ok 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[28904]: cluster 2024-04-02T04:45:32.672812+0000 osd.4 (osd.4) 33 : cluster 0 4.c deep-scrub starts 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[28904]: cluster 2024-04-02T04:45:32.675290+0000 osd.4 (osd.4) 34 : cluster 0 4.c deep-scrub ok 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[28904]: audit 2024-04-02T04:45:32.773147+0000 mgr.y (mgr.14538) 50 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[28904]: cluster 2024-04-02T04:45:34.003670+0000 osd.6 (osd.6) 29 : cluster 0 4.1f scrub starts 2024-04-02T04:45:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:34 smithi069 bash[28904]: cluster 2024-04-02T04:45:34.007097+0000 osd.6 (osd.6) 30 : cluster 0 4.1f scrub ok 2024-04-02T04:45:35.065 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:35.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:35 smithi098 bash[23790]: cluster 2024-04-02T04:45:33.747344+0000 mgr.y (mgr.14538) 51 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:35.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:35 smithi098 bash[23790]: cluster 2024-04-02T04:45:34.752705+0000 osd.5 (osd.5) 33 : cluster 0 6.d deep-scrub starts 2024-04-02T04:45:35.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:35 smithi098 bash[23790]: cluster 2024-04-02T04:45:34.755040+0000 osd.5 (osd.5) 34 : cluster 0 6.d deep-scrub ok 2024-04-02T04:45:35.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:35 smithi069 bash[21528]: cluster 2024-04-02T04:45:33.747344+0000 mgr.y (mgr.14538) 51 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:35 smithi069 bash[21528]: cluster 2024-04-02T04:45:34.752705+0000 osd.5 (osd.5) 33 : cluster 0 6.d deep-scrub starts 2024-04-02T04:45:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:35 smithi069 bash[21528]: cluster 2024-04-02T04:45:34.755040+0000 osd.5 (osd.5) 34 : cluster 0 6.d deep-scrub ok 2024-04-02T04:45:35.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:35 smithi069 bash[28904]: cluster 2024-04-02T04:45:33.747344+0000 mgr.y (mgr.14538) 51 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:35.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:35 smithi069 bash[28904]: cluster 2024-04-02T04:45:34.752705+0000 osd.5 (osd.5) 33 : cluster 0 6.d deep-scrub starts 2024-04-02T04:45:35.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:35 smithi069 bash[28904]: cluster 2024-04-02T04:45:34.755040+0000 osd.5 (osd.5) 34 : cluster 0 6.d deep-scrub ok 2024-04-02T04:45:36.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 bash[23790]: cluster 2024-04-02T04:45:34.633587+0000 osd.0 (osd.0) 35 : cluster 0 6.0 scrub starts 2024-04-02T04:45:36.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 bash[23790]: cluster 2024-04-02T04:45:34.636332+0000 osd.0 (osd.0) 36 : cluster 0 6.0 scrub ok 2024-04-02T04:45:36.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.334962+0000 osd.7 (osd.7) 31 : cluster 0 2.1c scrub starts 2024-04-02T04:45:36.705 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.338021+0000 osd.7 (osd.7) 32 : cluster 0 2.1c scrub ok 2024-04-02T04:45:36.705 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.790499+0000 osd.5 (osd.5) 35 : cluster 0 2.3 scrub starts 2024-04-02T04:45:36.705 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.792257+0000 osd.5 (osd.5) 36 : cluster 0 2.3 scrub ok 2024-04-02T04:45:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[21528]: cluster 2024-04-02T04:45:34.633587+0000 osd.0 (osd.0) 35 : cluster 0 6.0 scrub starts 2024-04-02T04:45:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[21528]: cluster 2024-04-02T04:45:34.636332+0000 osd.0 (osd.0) 36 : cluster 0 6.0 scrub ok 2024-04-02T04:45:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.334962+0000 osd.7 (osd.7) 31 : cluster 0 2.1c scrub starts 2024-04-02T04:45:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.338021+0000 osd.7 (osd.7) 32 : cluster 0 2.1c scrub ok 2024-04-02T04:45:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.790499+0000 osd.5 (osd.5) 35 : cluster 0 2.3 scrub starts 2024-04-02T04:45:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.792257+0000 osd.5 (osd.5) 36 : cluster 0 2.3 scrub ok 2024-04-02T04:45:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[28904]: cluster 2024-04-02T04:45:34.633587+0000 osd.0 (osd.0) 35 : cluster 0 6.0 scrub starts 2024-04-02T04:45:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[28904]: cluster 2024-04-02T04:45:34.636332+0000 osd.0 (osd.0) 36 : cluster 0 6.0 scrub ok 2024-04-02T04:45:36.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.334962+0000 osd.7 (osd.7) 31 : cluster 0 2.1c scrub starts 2024-04-02T04:45:36.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.338021+0000 osd.7 (osd.7) 32 : cluster 0 2.1c scrub ok 2024-04-02T04:45:36.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.790499+0000 osd.5 (osd.5) 35 : cluster 0 2.3 scrub starts 2024-04-02T04:45:36.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:36 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.792257+0000 osd.5 (osd.5) 36 : cluster 0 2.3 scrub ok 2024-04-02T04:45:36.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.974 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.975 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.975 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.976 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.976 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.977 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:36.977 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:45:36 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.285 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.286 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.286 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.287 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.287 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.288 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.288 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.288 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.288 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.288 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.288 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: Started Ceph grafana.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:37.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.289 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:37 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- 172.21.15.69:0/1672218159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc101b60 msgr2=0x7fd3dc10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- 172.21.15.69:0/1672218159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc101b60 0x7fd3dc10dfd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009f90 tx=0x7fd3d400b330 comp rx=0 tx=0).stop 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- 172.21.15.69:0/1672218159 shutdown_connections 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- 172.21.15.69:0/1672218159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc101b60 0x7fd3dc10dfd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- 172.21.15.69:0/1672218159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3dc107220 0x7fd3dc107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- 172.21.15.69:0/1672218159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc1011a0 0x7fd3dc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- 172.21.15.69:0/1672218159 >> 172.21.15.69:0/1672218159 conn(0x7fd3dc0771e0 msgr2=0x7fd3dc0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:37.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- 172.21.15.69:0/1672218159 shutdown_connections 2024-04-02T04:45:37.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- 172.21.15.69:0/1672218159 wait complete. 2024-04-02T04:45:37.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 librados: starting msgr at 2024-04-02T04:45:37.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 librados: starting objecter 2024-04-02T04:45:37.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 Processor -- start 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- start start 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 librados: setting wanted keys 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 librados: calling monclient init 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3dc1011a0 0x7fd3dc196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 0x7fd3dc1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc107220 0x7fd3dc1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc110930 con 0x7fd3dc101b60 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1107b0 con 0x7fd3dc1011a0 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3e2083700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd3dc110ab0 con 0x7fd3dc107220 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3daffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 0x7fd3dc1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3daffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 0x7fd3dc1970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43446/0 (socket says 172.21.15.69:43446) 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3daffd700 1 -- 172.21.15.69:0/3317356165 learned_addr learned my addr 172.21.15.69:0/3317356165 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:45:37.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3dbfff700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc107220 0x7fd3dc1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.696+0000 7fd3db7fe700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3dc1011a0 0x7fd3dc196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3daffd700 1 -- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc107220 msgr2=0x7fd3dc1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3daffd700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc107220 0x7fd3dc1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3daffd700 1 -- 172.21.15.69:0/3317356165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3dc1011a0 msgr2=0x7fd3dc196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3daffd700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3dc1011a0 0x7fd3dc196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3daffd700 1 -- 172.21.15.69:0/3317356165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc1a63e0 con 0x7fd3dc101b60 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3daffd700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 0x7fd3dc1970b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00ec10 tx=0x7fd3cc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 10 client.?.objecter ms_handle_connect 0x7fd3dc101b60 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc00cde0 con 0x7fd3dc101b60 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd3cc0044a0 con 0x7fd3dc101b60 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc010430 con 0x7fd3dc101b60 2024-04-02T04:45:37.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3dc1a66d0 con 0x7fd3dc101b60 2024-04-02T04:45:37.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3e2083700 10 client.14595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:45:37.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3dc1a6bc0 con 0x7fd3dc101b60 2024-04-02T04:45:37.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3e2083700 1 librados: init done 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3dc04ff00 con 0x7fd3dc101b60 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd3cc004000 con 0x7fd3dc101b60 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd3c406f5e0 0x7fd3c4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 54..105) ==== 7421+0+0 (secure 0 0 0) 0x7fd3cc08e450 con 0x7fd3dc101b60 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 10 client.14595.objecter ms_dispatch 0x7fd3dc001550 osd_map(105..105 src has 54..105) 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 3 client.14595.objecter handle_osd_map got epochs [105,105] > 0 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 3 client.14595.objecter handle_osd_map decoding full epoch 105 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3db7fe700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd3c406f5e0 0x7fd3c4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 20 client.14595.objecter dump_active .. 0 homeless 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3db7fe700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd3c406f5e0 0x7fd3c4071aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0005950 tx=0x7fd3d0005eb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.700+0000 7fd3d8ff9700 10 client.14595.objecter ms_handle_connect 0x7fd3c406f5e0 2024-04-02T04:45:37.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.704+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd3cc016330 con 0x7fd3dc101b60 2024-04-02T04:45:37.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.690126+0000 osd.3 (osd.3) 47 : cluster 0 6.13 scrub starts 2024-04-02T04:45:37.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.692616+0000 osd.3 (osd.3) 48 : cluster 0 6.13 scrub ok 2024-04-02T04:45:37.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: cluster 2024-04-02T04:45:35.748085+0000 mgr.y (mgr.14538) 52 : cluster 0 pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:37.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: cluster 2024-04-02T04:45:36.352980+0000 osd.7 (osd.7) 33 : cluster 0 3.11 scrub starts 2024-04-02T04:45:37.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: cluster 2024-04-02T04:45:36.358871+0000 osd.7 (osd.7) 34 : cluster 0 3.11 scrub ok 2024-04-02T04:45:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: audit 2024-04-02T04:45:36.910264+0000 mon.b (mon.1) 66 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: audit 2024-04-02T04:45:37.295690+0000 mon.a (mon.0) 865 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: audit 2024-04-02T04:45:37.309403+0000 mon.a (mon.0) 866 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: audit 2024-04-02T04:45:37.328450+0000 mon.a (mon.0) 867 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:37 smithi098 bash[23790]: audit 2024-04-02T04:45:37.334277+0000 mon.a (mon.0) 868 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.690126+0000 osd.3 (osd.3) 47 : cluster 0 6.13 scrub starts 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.692616+0000 osd.3 (osd.3) 48 : cluster 0 6.13 scrub ok 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: cluster 2024-04-02T04:45:35.748085+0000 mgr.y (mgr.14538) 52 : cluster 0 pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: cluster 2024-04-02T04:45:36.352980+0000 osd.7 (osd.7) 33 : cluster 0 3.11 scrub starts 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: cluster 2024-04-02T04:45:36.358871+0000 osd.7 (osd.7) 34 : cluster 0 3.11 scrub ok 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: audit 2024-04-02T04:45:36.910264+0000 mon.b (mon.1) 66 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: audit 2024-04-02T04:45:37.295690+0000 mon.a (mon.0) 865 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: audit 2024-04-02T04:45:37.309403+0000 mon.a (mon.0) 866 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: audit 2024-04-02T04:45:37.328450+0000 mon.a (mon.0) 867 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[21528]: audit 2024-04-02T04:45:37.334277+0000 mon.a (mon.0) 868 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.690126+0000 osd.3 (osd.3) 47 : cluster 0 6.13 scrub starts 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.692616+0000 osd.3 (osd.3) 48 : cluster 0 6.13 scrub ok 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: cluster 2024-04-02T04:45:35.748085+0000 mgr.y (mgr.14538) 52 : cluster 0 pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: cluster 2024-04-02T04:45:36.352980+0000 osd.7 (osd.7) 33 : cluster 0 3.11 scrub starts 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: cluster 2024-04-02T04:45:36.358871+0000 osd.7 (osd.7) 34 : cluster 0 3.11 scrub ok 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: audit 2024-04-02T04:45:36.910264+0000 mon.b (mon.1) 66 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: audit 2024-04-02T04:45:37.295690+0000 mon.a (mon.0) 865 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: audit 2024-04-02T04:45:37.309403+0000 mon.a (mon.0) 866 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: audit 2024-04-02T04:45:37.328450+0000 mon.a (mon.0) 867 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:37 smithi069 bash[28904]: audit 2024-04-02T04:45:37.334277+0000 mon.a (mon.0) 868 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:37.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.924+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3dc1a6ee0 con 0x7fd3dc101b60 2024-04-02T04:45:37.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3d8ff9700 1 -- 172.21.15.69:0/3317356165 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+22947 (secure 0 0 0) 0x7fd3cc05bb30 con 0x7fd3dc101b60 2024-04-02T04:45:37.927 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:45:37.928 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":105,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:33:43.348385+0000","modified":"2024-04-02T04:45:09.177745+0000","last_up_change":"2024-04-02T04:43:12.908579+0000","last_in_change":"2024-04-02T04:42:47.064987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.671714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-02T04:43:52.801763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-02T04:43:54.772439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-04-02T04:43:56.696111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-04-02T04:43:57.254108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-02T04:43:58.846023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b13e248-930e-40ca-9bb0-e8c5ce16bd0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6803","nonce":1087500595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6805","nonce":1087500595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6809","nonce":1087500595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6807","nonce":1087500595}]},"public_addr":"172.21.15.69:6803/1087500595","cluster_addr":"172.21.15.69:6805/1087500595","heartbeat_back_addr":"172.21.15.69:6809/1087500595","heartbeat_front_addr":"172.21.15.69:6807/1087500595","state":["exists","up"]},{"osd":1,"uuid":"afb71182-d4f7-416d-ae44-90f28eab4b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6811","nonce":2651666015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6813","nonce":2651666015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6817","nonce":2651666015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6815","nonce":2651666015}]},"public_addr":"172.21.15.69:6811/2651666015","cluster_addr":"172.21.15.69:6813/2651666015","heartbeat_back_addr":"172.21.15.69:6817/2651666015","heartbeat_front_addr":"172.21.15.69:6815/2651666015","state":["exists","up"]},{"osd":2,"uuid":"523bcbd7-d389-4b8f-aca8-88b71ce61de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6819","nonce":619998764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6821","nonce":619998764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6825","nonce":619998764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6823","nonce":619998764}]},"public_addr":"172.21.15.69:6819/619998764","cluster_addr":"172.21.15.69:6821/619998764","heartbeat_back_addr":"172.21.15.69:6825/619998764","heartbeat_front_addr":"172.21.15.69:6823/619998764","state":["exists","up"]},{"osd":3,"uuid":"20e4526d-2707-44d0-b13a-af36019c42ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6827","nonce":207593208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6829","nonce":207593208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6833","nonce":207593208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6831","nonce":207593208}]},"public_addr":"172.21.15.69:6827/207593208","cluster_addr":"172.21.15.69:6829/207593208","heartbeat_back_addr":"172.21.15.69:6833/207593208","heartbeat_front_addr":"172.21.15.69:6831/207593208","state":["exists","up"]},{"osd":4,"uuid":"aa3bc4f3-4476-4ba1-b033-428e5edf5852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6801","nonce":2065947829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6803","nonce":2065947829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6807","nonce":2065947829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6805","nonce":2065947829}]},"public_addr":"172.21.15.98:6801/2065947829","cluster_addr":"172.21.15.98:6803/2065947829","heartbeat_back_addr":"172.21.15.98:6807/2065947829","heartbeat_front_addr":"172.21.15.98:6805/2065947829","state":["exists","up"]},{"osd":5,"uuid":"902326af-4a40-47c8-a3b8-d5d5536f90fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6809","nonce":219578185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6811","nonce":219578185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6815","nonce":219578185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6813","nonce":219578185}]},"public_addr":"172.21.15.98:6809/219578185","cluster_addr":"172.21.15.98:6811/219578185","heartbeat_back_addr":"172.21.15.98:6815/219578185","heartbeat_front_addr":"172.21.15.98:6813/219578185","state":["exists","up"]},{"osd":6,"uuid":"8c73e4b0-cb6b-486a-8b55-a0fe4d97a727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6817","nonce":1226172499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6819","nonce":1226172499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6823","nonce":1226172499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6821","nonce":1226172499}]},"public_addr":"172.21.15.98:6817/1226172499","cluster_addr":"172.21.15.98:6819/1226172499","heartbeat_back_addr":"172.21.15.98:6823/1226172499","heartbeat_front_addr":"172.21.15.98:6821/1226172499","state":["exists","up"]},{"osd":7,"uuid":"4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6825","nonce":148009079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6827","nonce":148009079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6831","nonce":148009079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6829","nonce":148009079}]},"public_addr":"172.21.15.98:6825/148009079","cluster_addr":"172.21.15.98:6827/148009079","heartbeat_back_addr":"172.21.15.98:6831/148009079","heartbeat_front_addr":"172.21.15.98:6829/148009079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:36.598249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:45:08.728923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:52.466235+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:59.838353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:31.136355+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:45:02.904291+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:51.617716+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:33.492968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/2738867930":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/1798147282":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/52352690":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/2928651991":"2024-04-03T04:34:47.037764+0000","172.21.15.69:6800/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/2763215770":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/3657409442":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/879855771":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1025808168":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1168952113":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/1892956550":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6801/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/54578327":"2024-04-03T04:44:51.699278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 10 librados: watch_flush enter 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 10 librados: watch_flush exit 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 20 client.14595.objecter shutdown clearing up homeless session... 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 10 client.14595.objecter successfully canceled tick 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd3c406f5e0 msgr2=0x7fd3c4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd3c406f5e0 0x7fd3c4071aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0005950 tx=0x7fd3d0005eb0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 msgr2=0x7fd3dc1970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 0x7fd3dc1970b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00ec10 tx=0x7fd3cc00c2d0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 shutdown_connections 2024-04-02T04:45:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd3c406f5e0 0x7fd3c4071aa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3dc107220 0x7fd3dc1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3dc1011a0 0x7fd3dc196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 --2- 172.21.15.69:0/3317356165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3dc101b60 0x7fd3dc1970b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 >> 172.21.15.69:0/3317356165 conn(0x7fd3dc0771e0 msgr2=0x7fd3dc107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 shutdown_connections 2024-04-02T04:45:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 -- 172.21.15.69:0/3317356165 wait complete. 2024-04-02T04:45:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:37.928+0000 7fd3e2083700 1 librados: shutdown 2024-04-02T04:45:38.622 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[23790]: audit 2024-04-02T04:45:37.393989+0000 mon.b (mon.1) 67 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:38.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[23790]: cluster 2024-04-02T04:45:37.841325+0000 osd.5 (osd.5) 37 : cluster 0 6.a scrub starts 2024-04-02T04:45:38.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[23790]: cluster 2024-04-02T04:45:37.842409+0000 osd.5 (osd.5) 38 : cluster 0 6.a scrub ok 2024-04-02T04:45:38.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[23790]: audit 2024-04-02T04:45:37.929141+0000 mon.a (mon.0) 869 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:38.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[23790]: audit 2024-04-02T04:45:37.930422+0000 mon.a (mon.0) 870 : audit 0 from='client.? 172.21.15.69:0/3317356165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:45:38.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[23790]: audit 2024-04-02T04:45:37.945878+0000 mon.a (mon.0) 871 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:38.623 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595554111Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-02T04:45:38.623 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595758823Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-02T04:45:38.623 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595777655Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595781622Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595784931Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595788067Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595790963Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595794799Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595797882Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595803172Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595808707Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595813893Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595818529Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595823225Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595828218Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595832982Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-02T04:45:38.624 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595838158Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.595843405Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=settings t=2024-04-02T04:45:38.59584873Z level=info msg="App mode production" 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=sqlstore t=2024-04-02T04:45:38.595912652Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=sqlstore t=2024-04-02T04:45:38.595943831Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.596970062Z level=info msg="Starting DB migrations" 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.597076437Z level=info msg="Executing migration" id="create migration_log table" 2024-04-02T04:45:38.625 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.621493123Z level=info msg="Executing migration" id="create user table" 2024-04-02T04:45:38.626 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-02T04:45:38.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T04:45:38.637 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[21528]: audit 2024-04-02T04:45:37.393989+0000 mon.b (mon.1) 67 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:38.637 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[21528]: cluster 2024-04-02T04:45:37.841325+0000 osd.5 (osd.5) 37 : cluster 0 6.a scrub starts 2024-04-02T04:45:38.637 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[21528]: cluster 2024-04-02T04:45:37.842409+0000 osd.5 (osd.5) 38 : cluster 0 6.a scrub ok 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[21528]: audit 2024-04-02T04:45:37.929141+0000 mon.a (mon.0) 869 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[21528]: audit 2024-04-02T04:45:37.930422+0000 mon.a (mon.0) 870 : audit 0 from='client.? 172.21.15.69:0/3317356165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[21528]: audit 2024-04-02T04:45:37.945878+0000 mon.a (mon.0) 871 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[28904]: audit 2024-04-02T04:45:37.393989+0000 mon.b (mon.1) 67 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[28904]: cluster 2024-04-02T04:45:37.841325+0000 osd.5 (osd.5) 37 : cluster 0 6.a scrub starts 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[28904]: cluster 2024-04-02T04:45:37.842409+0000 osd.5 (osd.5) 38 : cluster 0 6.a scrub ok 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[28904]: audit 2024-04-02T04:45:37.929141+0000 mon.a (mon.0) 869 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[28904]: audit 2024-04-02T04:45:37.930422+0000 mon.a (mon.0) 870 : audit 0 from='client.? 172.21.15.69:0/3317356165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:45:38.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:38 smithi069 bash[28904]: audit 2024-04-02T04:45:37.945878+0000 mon.a (mon.0) 871 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:38.875 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.671706925Z level=info msg="Executing migration" id="add unique index user.login" 2024-04-02T04:45:38.875 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.697312304Z level=info msg="Executing migration" id="add unique index user.email" 2024-04-02T04:45:38.875 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.739240968Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-04-02T04:45:38.875 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.781259563Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-04-02T04:45:38.876 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.806426063Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-04-02T04:45:38.876 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.831866047Z level=info msg="Executing migration" id="create user table v2" 2024-04-02T04:45:38.876 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.873838784Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-04-02T04:45:39.153 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.924461893Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-04-02T04:45:39.153 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:38 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:38.966485308Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-04-02T04:45:39.153 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.008383197Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-04-02T04:45:39.153 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.033718382Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-04-02T04:45:39.153 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.05896339Z level=info msg="Executing migration" id="Update user table charset" 2024-04-02T04:45:39.154 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.084255738Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-04-02T04:45:39.154 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.109447488Z level=info msg="Executing migration" id="Add missing user data" 2024-04-02T04:45:39.422 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[23790]: cluster 2024-04-02T04:45:37.748962+0000 mgr.y (mgr.14538) 53 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:39.423 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[23790]: cluster 2024-04-02T04:45:38.853251+0000 osd.5 (osd.5) 39 : cluster 0 2.2 scrub starts 2024-04-02T04:45:39.423 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[23790]: cluster 2024-04-02T04:45:38.854992+0000 osd.5 (osd.5) 40 : cluster 0 2.2 scrub ok 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.151494019Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.176640669Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.218783236Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.260722598Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.286177664Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.32822114Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.378825956Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-04-02T04:45:39.423 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.420986473Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.446203562Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.488294721Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.51365135Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.538908972Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.564232983Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.589434453Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.614804903Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.640161315Z level=info msg="Executing migration" id="create temp_user v2" 2024-04-02T04:45:39.675 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.67368516Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-04-02T04:45:39.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:39 smithi069 bash[28904]: cluster 2024-04-02T04:45:37.748962+0000 mgr.y (mgr.14538) 53 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:39.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:39 smithi069 bash[28904]: cluster 2024-04-02T04:45:38.853251+0000 osd.5 (osd.5) 39 : cluster 0 2.2 scrub starts 2024-04-02T04:45:39.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:39 smithi069 bash[28904]: cluster 2024-04-02T04:45:38.854992+0000 osd.5 (osd.5) 40 : cluster 0 2.2 scrub ok 2024-04-02T04:45:39.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:39 smithi069 bash[21528]: cluster 2024-04-02T04:45:37.748962+0000 mgr.y (mgr.14538) 53 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:39.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:39 smithi069 bash[21528]: cluster 2024-04-02T04:45:38.853251+0000 osd.5 (osd.5) 39 : cluster 0 2.2 scrub starts 2024-04-02T04:45:39.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:39 smithi069 bash[21528]: cluster 2024-04-02T04:45:38.854992+0000 osd.5 (osd.5) 40 : cluster 0 2.2 scrub ok 2024-04-02T04:45:39.927 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.723980299Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-04-02T04:45:39.927 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.774402014Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-04-02T04:45:39.927 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.808008516Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-04-02T04:45:39.927 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.858414998Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-04-02T04:45:39.927 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.892004227Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-04-02T04:45:39.927 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.925622436Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-04-02T04:45:40.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:39 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:39.976063684Z level=info msg="Executing migration" id="create star table" 2024-04-02T04:45:40.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.026345506Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-04-02T04:45:40.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.076820247Z level=info msg="Executing migration" id="create org table v1" 2024-04-02T04:45:40.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.127294632Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-04-02T04:45:40.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[23790]: cluster 2024-04-02T04:45:38.758734+0000 osd.3 (osd.3) 49 : cluster 0 4.11 scrub starts 2024-04-02T04:45:40.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[23790]: cluster 2024-04-02T04:45:38.761190+0000 osd.3 (osd.3) 50 : cluster 0 4.11 scrub ok 2024-04-02T04:45:40.431 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.177672614Z level=info msg="Executing migration" id="create org_user table v1" 2024-04-02T04:45:40.431 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.211300829Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-04-02T04:45:40.431 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.262120184Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-04-02T04:45:40.432 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.295775729Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-04-02T04:45:40.432 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.346040479Z level=info msg="Executing migration" id="Update org table charset" 2024-04-02T04:45:40.432 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.37136524Z level=info msg="Executing migration" id="Update org_user table charset" 2024-04-02T04:45:40.432 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.430107322Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.463765958Z level=info msg="Executing migration" id="create dashboard table" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.514164836Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.547848761Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.573042867Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.606757654Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.657088003Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-02T04:45:40.692 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.690665723Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-04-02T04:45:40.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:40 smithi069 bash[21528]: cluster 2024-04-02T04:45:38.758734+0000 osd.3 (osd.3) 49 : cluster 0 4.11 scrub starts 2024-04-02T04:45:40.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:40 smithi069 bash[21528]: cluster 2024-04-02T04:45:38.761190+0000 osd.3 (osd.3) 50 : cluster 0 4.11 scrub ok 2024-04-02T04:45:40.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:40 smithi069 bash[28904]: cluster 2024-04-02T04:45:38.758734+0000 osd.3 (osd.3) 49 : cluster 0 4.11 scrub starts 2024-04-02T04:45:40.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:40 smithi069 bash[28904]: cluster 2024-04-02T04:45:38.761190+0000 osd.3 (osd.3) 50 : cluster 0 4.11 scrub ok 2024-04-02T04:45:40.960 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.724380088Z level=info msg="Executing migration" id="create dashboard v2" 2024-04-02T04:45:40.960 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.791387145Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-04-02T04:45:40.960 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.841766347Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-04-02T04:45:40.960 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.875523372Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-04-02T04:45:40.960 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.909001114Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-04-02T04:45:40.960 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:40 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:40.95919372Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-04-02T04:45:41.213 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.009497291Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-04-02T04:45:41.213 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.043254521Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-04-02T04:45:41.213 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.076913091Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-04-02T04:45:41.213 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.110632959Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-04-02T04:45:41.213 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.160962819Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-04-02T04:45:41.213 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.211303235Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-04-02T04:45:41.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.261743584Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-04-02T04:45:41.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.320623562Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-04-02T04:45:41.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.354329366Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-04-02T04:45:41.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.387911193Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-04-02T04:45:41.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.438618395Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.555730+0000 osd.0 (osd.0) 37 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.555861+0000 osd.0 (osd.0) 38 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.711257+0000 osd.4 (osd.4) 35 : cluster 0 4.18 scrub starts 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.713584+0000 osd.4 (osd.4) 36 : cluster 0 4.18 scrub ok 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.749573+0000 mgr.y (mgr.14538) 54 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.777014+0000 osd.3 (osd.3) 51 : cluster 0 4.17 scrub starts 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:39.779283+0000 osd.3 (osd.3) 52 : cluster 0 4.17 scrub ok 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:40.404303+0000 mon.a (mon.0) 872 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:40.821895+0000 osd.5 (osd.5) 41 : cluster 0 3.1c scrub starts 2024-04-02T04:45:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[23790]: cluster 2024-04-02T04:45:40.828109+0000 osd.5 (osd.5) 42 : cluster 0 3.1c scrub ok 2024-04-02T04:45:41.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.555730+0000 osd.0 (osd.0) 37 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:41.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.555861+0000 osd.0 (osd.0) 38 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.711257+0000 osd.4 (osd.4) 35 : cluster 0 4.18 scrub starts 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.713584+0000 osd.4 (osd.4) 36 : cluster 0 4.18 scrub ok 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.749573+0000 mgr.y (mgr.14538) 54 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.777014+0000 osd.3 (osd.3) 51 : cluster 0 4.17 scrub starts 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:39.779283+0000 osd.3 (osd.3) 52 : cluster 0 4.17 scrub ok 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:40.404303+0000 mon.a (mon.0) 872 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:40.821895+0000 osd.5 (osd.5) 41 : cluster 0 3.1c scrub starts 2024-04-02T04:45:41.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[21528]: cluster 2024-04-02T04:45:40.828109+0000 osd.5 (osd.5) 42 : cluster 0 3.1c scrub ok 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.555730+0000 osd.0 (osd.0) 37 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.555861+0000 osd.0 (osd.0) 38 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.711257+0000 osd.4 (osd.4) 35 : cluster 0 4.18 scrub starts 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.713584+0000 osd.4 (osd.4) 36 : cluster 0 4.18 scrub ok 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.749573+0000 mgr.y (mgr.14538) 54 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.777014+0000 osd.3 (osd.3) 51 : cluster 0 4.17 scrub starts 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:39.779283+0000 osd.3 (osd.3) 52 : cluster 0 4.17 scrub ok 2024-04-02T04:45:41.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:40.404303+0000 mon.a (mon.0) 872 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-04-02T04:45:41.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:40.821895+0000 osd.5 (osd.5) 41 : cluster 0 3.1c scrub starts 2024-04-02T04:45:41.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:41 smithi069 bash[28904]: cluster 2024-04-02T04:45:40.828109+0000 osd.5 (osd.5) 42 : cluster 0 3.1c scrub ok 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.513887843Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.547524866Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.581065705Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.614756926Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.648327956Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.698641135Z level=info msg="Executing migration" id="Update dashboard title length" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.749183946Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-02T04:45:41.784 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.782787755Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-04-02T04:45:42.036 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.81629113Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-02T04:45:42.036 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.866738271Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-04-02T04:45:42.036 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.900521005Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-02T04:45:42.036 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:41 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:41.950800221Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-02T04:45:42.036 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.001169384Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-04-02T04:45:42.036 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.034682715Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.085141736Z level=info msg="Executing migration" id="Add check_sum column" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.135502725Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.177553825Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.20283103Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.244739303Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.270210227Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-02T04:45:42.314 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.312480997Z level=info msg="Executing migration" id="create data_source table" 2024-04-02T04:45:42.420 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:42.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.354681631Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-04-02T04:45:42.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.380068185Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-04-02T04:45:42.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.430433319Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-04-02T04:45:42.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.46389118Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-04-02T04:45:42.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.514348103Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-04-02T04:45:42.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.539659223Z level=info msg="Executing migration" id="create data_source table v2" 2024-04-02T04:45:42.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.564900832Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-04-02T04:45:42.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:45:42.819 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.590227901Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-04-02T04:45:42.819 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.615499533Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-04-02T04:45:42.819 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.657489535Z level=info msg="Executing migration" id="Add column with_credentials" 2024-04-02T04:45:42.819 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.682810101Z level=info msg="Executing migration" id="Add secure json data column" 2024-04-02T04:45:42.819 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.724698721Z level=info msg="Executing migration" id="Update data_source table charset" 2024-04-02T04:45:43.070 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.817288464Z level=info msg="Executing migration" id="Update initial version to 1" 2024-04-02T04:45:43.070 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.859325877Z level=info msg="Executing migration" id="Add read_only data column" 2024-04-02T04:45:43.070 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.901488378Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-04-02T04:45:43.071 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.92673765Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-04-02T04:45:43.071 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:42.968642247Z level=info msg="Executing migration" id="Add uid column" 2024-04-02T04:45:43.071 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.027380203Z level=info msg="Executing migration" id="Update uid value" 2024-04-02T04:45:43.071 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.0695065Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-04-02T04:45:43.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[23790]: cluster 2024-04-02T04:45:41.702006+0000 osd.3 (osd.3) 53 : cluster 0 6.6 deep-scrub starts 2024-04-02T04:45:43.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[23790]: cluster 2024-04-02T04:45:41.707726+0000 osd.3 (osd.3) 54 : cluster 0 6.6 deep-scrub ok 2024-04-02T04:45:43.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[23790]: cluster 2024-04-02T04:45:41.738808+0000 osd.4 (osd.4) 37 : cluster 0 5.1 scrub starts 2024-04-02T04:45:43.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[23790]: cluster 2024-04-02T04:45:41.740984+0000 osd.4 (osd.4) 38 : cluster 0 5.1 scrub ok 2024-04-02T04:45:43.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[23790]: cluster 2024-04-02T04:45:41.750240+0000 mgr.y (mgr.14538) 55 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:43.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:42 smithi098 bash[23790]: audit 2024-04-02T04:45:41.851731+0000 mon.a (mon.0) 873 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:43.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[21528]: cluster 2024-04-02T04:45:41.702006+0000 osd.3 (osd.3) 53 : cluster 0 6.6 deep-scrub starts 2024-04-02T04:45:43.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[21528]: cluster 2024-04-02T04:45:41.707726+0000 osd.3 (osd.3) 54 : cluster 0 6.6 deep-scrub ok 2024-04-02T04:45:43.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[21528]: cluster 2024-04-02T04:45:41.738808+0000 osd.4 (osd.4) 37 : cluster 0 5.1 scrub starts 2024-04-02T04:45:43.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[21528]: cluster 2024-04-02T04:45:41.740984+0000 osd.4 (osd.4) 38 : cluster 0 5.1 scrub ok 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[21528]: cluster 2024-04-02T04:45:41.750240+0000 mgr.y (mgr.14538) 55 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[21528]: audit 2024-04-02T04:45:41.851731+0000 mon.a (mon.0) 873 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[28904]: cluster 2024-04-02T04:45:41.702006+0000 osd.3 (osd.3) 53 : cluster 0 6.6 deep-scrub starts 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[28904]: cluster 2024-04-02T04:45:41.707726+0000 osd.3 (osd.3) 54 : cluster 0 6.6 deep-scrub ok 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[28904]: cluster 2024-04-02T04:45:41.738808+0000 osd.4 (osd.4) 37 : cluster 0 5.1 scrub starts 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[28904]: cluster 2024-04-02T04:45:41.740984+0000 osd.4 (osd.4) 38 : cluster 0 5.1 scrub ok 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[28904]: cluster 2024-04-02T04:45:41.750240+0000 mgr.y (mgr.14538) 55 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:43.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:42 smithi069 bash[28904]: audit 2024-04-02T04:45:41.851731+0000 mon.a (mon.0) 873 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:43.331 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.094965592Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-04-02T04:45:43.459 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.145727682Z level=info msg="Executing migration" id="create api_key table" 2024-04-02T04:45:43.459 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.171101495Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-04-02T04:45:43.459 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.196162289Z level=info msg="Executing migration" id="add index api_key.key" 2024-04-02T04:45:43.459 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.221401066Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-04-02T04:45:43.460 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.246949745Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-04-02T04:45:43.460 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.288687668Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-04-02T04:45:43.460 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.330711932Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-04-02T04:45:43.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.372978104Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-04-02T04:45:43.593 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.398299551Z level=info msg="Executing migration" id="create api_key table v2" 2024-04-02T04:45:43.593 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.440348137Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-04-02T04:45:43.593 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.482359073Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-04-02T04:45:43.593 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.507654677Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-04-02T04:45:43.593 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.549798576Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-04-02T04:45:43.593 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.591794249Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-04-02T04:45:43.846 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.633960456Z level=info msg="Executing migration" id="Update api_key table charset" 2024-04-02T04:45:43.846 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.675988688Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-04-02T04:45:43.846 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.718322369Z level=info msg="Executing migration" id="Add service account foreign key" 2024-04-02T04:45:43.846 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.760403198Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-02T04:45:43.846 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.802603306Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-02T04:45:43.846 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.828011812Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-02T04:45:44.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[21528]: cluster 2024-04-02T04:45:42.325531+0000 osd.7 (osd.7) 35 : cluster 0 3.f scrub starts 2024-04-02T04:45:44.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[21528]: cluster 2024-04-02T04:45:42.329948+0000 osd.7 (osd.7) 36 : cluster 0 3.f scrub ok 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[21528]: audit 2024-04-02T04:45:42.782783+0000 mgr.y (mgr.14538) 56 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[21528]: audit 2024-04-02T04:45:43.046159+0000 mon.a (mon.0) 874 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[21528]: audit 2024-04-02T04:45:43.056649+0000 mon.a (mon.0) 875 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[28904]: cluster 2024-04-02T04:45:42.325531+0000 osd.7 (osd.7) 35 : cluster 0 3.f scrub starts 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[28904]: cluster 2024-04-02T04:45:42.329948+0000 osd.7 (osd.7) 36 : cluster 0 3.f scrub ok 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[28904]: audit 2024-04-02T04:45:42.782783+0000 mgr.y (mgr.14538) 56 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[28904]: audit 2024-04-02T04:45:43.046159+0000 mon.a (mon.0) 874 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:43 smithi069 bash[28904]: audit 2024-04-02T04:45:43.056649+0000 mon.a (mon.0) 875 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:44.107 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.870245381Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-04-02T04:45:44.107 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.912485329Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-04-02T04:45:44.108 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.937762278Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-04-02T04:45:44.108 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:43.979993917Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-04-02T04:45:44.108 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.005175851Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-02T04:45:44.108 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.047291686Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-02T04:45:44.108 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.080952549Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-04-02T04:45:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[23790]: cluster 2024-04-02T04:45:42.325531+0000 osd.7 (osd.7) 35 : cluster 0 3.f scrub starts 2024-04-02T04:45:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[23790]: cluster 2024-04-02T04:45:42.329948+0000 osd.7 (osd.7) 36 : cluster 0 3.f scrub ok 2024-04-02T04:45:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[23790]: audit 2024-04-02T04:45:42.782783+0000 mgr.y (mgr.14538) 56 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[23790]: audit 2024-04-02T04:45:43.046159+0000 mon.a (mon.0) 874 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:43 smithi098 bash[23790]: audit 2024-04-02T04:45:43.056649+0000 mon.a (mon.0) 875 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.106280784Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.148360751Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.190305407Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.232363906Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.257700491Z level=info msg="Executing migration" id="create quota table v1" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.308002475Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.350004082Z level=info msg="Executing migration" id="Update quota table charset" 2024-04-02T04:45:44.394 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.392303625Z level=info msg="Executing migration" id="create plugin_setting table" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.434415537Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.459674686Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.484826511Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.510250051Z level=info msg="Executing migration" id="create session table" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.535546611Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.560757513Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.586122465Z level=info msg="Executing migration" id="create playlist table v2" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.611426287Z level=info msg="Executing migration" id="create playlist item table v2" 2024-04-02T04:45:44.655 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.653330168Z level=info msg="Executing migration" id="Update playlist table charset" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.678661737Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.720554961Z level=info msg="Executing migration" id="drop preferences table v2" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.771191885Z level=info msg="Executing migration" id="drop preferences table v3" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.813316957Z level=info msg="Executing migration" id="create preferences table v3" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.838606526Z level=info msg="Executing migration" id="Update preferences table charset" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.880515676Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-04-02T04:45:44.924 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.922579103Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-04-02T04:45:45.178 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cluster 2024-04-02T04:45:43.600638+0000 osd.0 (osd.0) 39 : cluster 0 3.17 scrub starts 2024-04-02T04:45:45.178 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cluster 2024-04-02T04:45:43.604312+0000 osd.0 (osd.0) 40 : cluster 0 3.17 scrub ok 2024-04-02T04:45:45.178 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cluster 2024-04-02T04:45:43.751727+0000 mgr.y (mgr.14538) 57 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: audit 2024-04-02T04:45:44.044192+0000 mon.a (mon.0) 876 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: audit 2024-04-02T04:45:44.061680+0000 mon.a (mon.0) 877 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: audit 2024-04-02T04:45:44.061959+0000 mon.b (mon.1) 68 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: audit 2024-04-02T04:45:44.065597+0000 mon.b (mon.1) 69 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: audit 2024-04-02T04:45:44.082002+0000 mon.a (mon.0) 878 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cephadm 2024-04-02T04:45:44.160673+0000 mgr.y (mgr.14538) 58 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cephadm 2024-04-02T04:45:44.169616+0000 mgr.y (mgr.14538) 59 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi069 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cluster 2024-04-02T04:45:44.285160+0000 osd.7 (osd.7) 37 : cluster 0 2.5 scrub starts 2024-04-02T04:45:45.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[23790]: cluster 2024-04-02T04:45:44.287397+0000 osd.7 (osd.7) 38 : cluster 0 2.5 scrub ok 2024-04-02T04:45:45.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.947904792Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-04-02T04:45:45.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:44 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:44.989865129Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-02T04:45:45.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.0151763Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-02T04:45:45.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.032177166Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-02T04:45:45.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.082755264Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-02T04:45:45.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.108139658Z level=info msg="Executing migration" id="create alert table v1" 2024-04-02T04:45:45.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.15003615Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-04-02T04:45:45.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.175446618Z level=info msg="Executing migration" id="add index alert state" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.200591365Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.22596311Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.267912527Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.343463581Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.368724157Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.402424613Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-04-02T04:45:45.429 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.427613946Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-02T04:45:45.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cluster 2024-04-02T04:45:43.600638+0000 osd.0 (osd.0) 39 : cluster 0 3.17 scrub starts 2024-04-02T04:45:45.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cluster 2024-04-02T04:45:43.604312+0000 osd.0 (osd.0) 40 : cluster 0 3.17 scrub ok 2024-04-02T04:45:45.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cluster 2024-04-02T04:45:43.751727+0000 mgr.y (mgr.14538) 57 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:45.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: audit 2024-04-02T04:45:44.044192+0000 mon.a (mon.0) 876 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: audit 2024-04-02T04:45:44.061680+0000 mon.a (mon.0) 877 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: audit 2024-04-02T04:45:44.061959+0000 mon.b (mon.1) 68 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: audit 2024-04-02T04:45:44.065597+0000 mon.b (mon.1) 69 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: audit 2024-04-02T04:45:44.082002+0000 mon.a (mon.0) 878 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cephadm 2024-04-02T04:45:44.160673+0000 mgr.y (mgr.14538) 58 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cephadm 2024-04-02T04:45:44.169616+0000 mgr.y (mgr.14538) 59 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi069 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cluster 2024-04-02T04:45:44.285160+0000 osd.7 (osd.7) 37 : cluster 0 2.5 scrub starts 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[28904]: cluster 2024-04-02T04:45:44.287397+0000 osd.7 (osd.7) 38 : cluster 0 2.5 scrub ok 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cluster 2024-04-02T04:45:43.600638+0000 osd.0 (osd.0) 39 : cluster 0 3.17 scrub starts 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cluster 2024-04-02T04:45:43.604312+0000 osd.0 (osd.0) 40 : cluster 0 3.17 scrub ok 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cluster 2024-04-02T04:45:43.751727+0000 mgr.y (mgr.14538) 57 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: audit 2024-04-02T04:45:44.044192+0000 mon.a (mon.0) 876 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: audit 2024-04-02T04:45:44.061680+0000 mon.a (mon.0) 877 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: audit 2024-04-02T04:45:44.061959+0000 mon.b (mon.1) 68 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: audit 2024-04-02T04:45:44.065597+0000 mon.b (mon.1) 69 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: audit 2024-04-02T04:45:44.082002+0000 mon.a (mon.0) 878 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cephadm 2024-04-02T04:45:44.160673+0000 mgr.y (mgr.14538) 58 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cephadm 2024-04-02T04:45:44.169616+0000 mgr.y (mgr.14538) 59 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi069 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cluster 2024-04-02T04:45:44.285160+0000 osd.7 (osd.7) 37 : cluster 0 2.5 scrub starts 2024-04-02T04:45:45.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:45 smithi069 bash[21528]: cluster 2024-04-02T04:45:44.287397+0000 osd.7 (osd.7) 38 : cluster 0 2.5 scrub ok 2024-04-02T04:45:45.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- 172.21.15.69:0/2675322040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 msgr2=0x7f0bf006aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:45.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- 172.21.15.69:0/2675322040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf006aee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009990 tx=0x7f0be0009ca0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- 172.21.15.69:0/2675322040 shutdown_connections 2024-04-02T04:45:45.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- 172.21.15.69:0/2675322040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bf0107680 0x7f0bf010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- 172.21.15.69:0/2675322040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf006aee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- 172.21.15.69:0/2675322040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- 172.21.15.69:0/2675322040 >> 172.21.15.69:0/2675322040 conn(0x7f0bf0078240 msgr2=0x7f0bf0078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- 172.21.15.69:0/2675322040 shutdown_connections 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- 172.21.15.69:0/2675322040 wait complete. 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 librados: starting msgr at 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 librados: starting objecter 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 Processor -- start 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- start start 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 librados: setting wanted keys 2024-04-02T04:45:45.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 librados: calling monclient init 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf01011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bf0107680 0x7f0bf0101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0111a90 con 0x7f0bf006b4b0 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0111910 con 0x7f0bf0107680 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bf5005700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf0111c10 con 0x7f0bf006ab00 2024-04-02T04:45:45.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf01011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf01011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57364/0 (socket says 172.21.15.69:57364) 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 -- 172.21.15.69:0/1498975866 learned_addr learned my addr 172.21.15.69:0/1498975866 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bee59c700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43456/0 (socket says 172.21.15.69:43456) 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bef59e700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bf0107680 0x7f0bf0101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 -- 172.21.15.69:0/1498975866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bf0107680 msgr2=0x7f0bf0101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bf0107680 0x7f0bf0101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 -- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 msgr2=0x7f0bf0101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0beed9d700 1 -- 172.21.15.69:0/1498975866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf01a6560 con 0x7f0bf006ab00 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.616+0000 7f0bee59c700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0beed9d700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf01011f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000c00 tx=0x7f0be00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:45.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 10 client.?.objecter ms_handle_connect 0x7f0bf006ab00 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0027070 con 0x7f0bf006ab00 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf01a67f0 con 0x7f0bf006ab00 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bf5005700 10 client.24488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf01a6ca0 con 0x7f0bf006ab00 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bf5005700 1 librados: init done 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0be0022470 con 0x7f0bf006ab00 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0031060 con 0x7f0bf006ab00 2024-04-02T04:45:45.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc005150 con 0x7f0bf006ab00 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0be002a070 con 0x7f0bf006ab00 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0bd806f5e0 0x7f0bd8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(106..106 src has 54..106) ==== 7421+0+0 (secure 0 0 0) 0x7f0be008ed70 con 0x7f0bf006ab00 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 10 client.24488.objecter ms_dispatch 0x7f0bf0001550 osd_map(106..106 src has 54..106) 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 3 client.24488.objecter handle_osd_map got epochs [106,106] > 0 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 3 client.24488.objecter handle_osd_map decoding full epoch 106 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 20 client.24488.objecter dump_active .. 0 homeless 2024-04-02T04:45:45.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bee59c700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0bd806f5e0 0x7f0bd8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:45:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bd7fff700 10 client.24488.objecter ms_handle_connect 0x7f0bd806f5e0 2024-04-02T04:45:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.620+0000 7f0bee59c700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0bd806f5e0 0x7f0bd8071aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009680 tx=0x7f0be4009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:45:45.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.624+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0be0013070 con 0x7f0bf006ab00 2024-04-02T04:45:45.698 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.452942822Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-04-02T04:45:45.698 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.478180219Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-04-02T04:45:45.698 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.503548357Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-04-02T04:45:45.698 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.528734461Z level=info msg="Executing migration" id="Add column is_default" 2024-04-02T04:45:45.698 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.562375923Z level=info msg="Executing migration" id="Add column frequency" 2024-04-02T04:45:45.699 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.596296504Z level=info msg="Executing migration" id="Add column send_reminder" 2024-04-02T04:45:45.699 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.646607178Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-04-02T04:45:45.699 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.696987051Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-04-02T04:45:45.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.836+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bdc005710 con 0x7f0bf006ab00 2024-04-02T04:45:45.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.836+0000 7f0bd7fff700 1 -- 172.21.15.69:0/1498975866 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+22947 (secure 0 0 0) 0x7f0be005fea0 con 0x7f0bf006ab00 2024-04-02T04:45:45.839 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:45:45.839 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":106,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:33:43.348385+0000","modified":"2024-04-02T04:45:40.389758+0000","last_up_change":"2024-04-02T04:43:12.908579+0000","last_in_change":"2024-04-02T04:42:47.064987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.671714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-02T04:43:52.801763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-02T04:43:54.772439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-04-02T04:43:56.696111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-04-02T04:43:57.254108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-02T04:43:58.846023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b13e248-930e-40ca-9bb0-e8c5ce16bd0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6803","nonce":1087500595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6805","nonce":1087500595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6809","nonce":1087500595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6807","nonce":1087500595}]},"public_addr":"172.21.15.69:6803/1087500595","cluster_addr":"172.21.15.69:6805/1087500595","heartbeat_back_addr":"172.21.15.69:6809/1087500595","heartbeat_front_addr":"172.21.15.69:6807/1087500595","state":["exists","up"]},{"osd":1,"uuid":"afb71182-d4f7-416d-ae44-90f28eab4b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6811","nonce":2651666015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6813","nonce":2651666015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6817","nonce":2651666015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6815","nonce":2651666015}]},"public_addr":"172.21.15.69:6811/2651666015","cluster_addr":"172.21.15.69:6813/2651666015","heartbeat_back_addr":"172.21.15.69:6817/2651666015","heartbeat_front_addr":"172.21.15.69:6815/2651666015","state":["exists","up"]},{"osd":2,"uuid":"523bcbd7-d389-4b8f-aca8-88b71ce61de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6819","nonce":619998764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6821","nonce":619998764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6825","nonce":619998764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6823","nonce":619998764}]},"public_addr":"172.21.15.69:6819/619998764","cluster_addr":"172.21.15.69:6821/619998764","heartbeat_back_addr":"172.21.15.69:6825/619998764","heartbeat_front_addr":"172.21.15.69:6823/619998764","state":["exists","up"]},{"osd":3,"uuid":"20e4526d-2707-44d0-b13a-af36019c42ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6827","nonce":207593208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6829","nonce":207593208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6833","nonce":207593208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6831","nonce":207593208}]},"public_addr":"172.21.15.69:6827/207593208","cluster_addr":"172.21.15.69:6829/207593208","heartbeat_back_addr":"172.21.15.69:6833/207593208","heartbeat_front_addr":"172.21.15.69:6831/207593208","state":["exists","up"]},{"osd":4,"uuid":"aa3bc4f3-4476-4ba1-b033-428e5edf5852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6801","nonce":2065947829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6803","nonce":2065947829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6807","nonce":2065947829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6805","nonce":2065947829}]},"public_addr":"172.21.15.98:6801/2065947829","cluster_addr":"172.21.15.98:6803/2065947829","heartbeat_back_addr":"172.21.15.98:6807/2065947829","heartbeat_front_addr":"172.21.15.98:6805/2065947829","state":["exists","up"]},{"osd":5,"uuid":"902326af-4a40-47c8-a3b8-d5d5536f90fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6809","nonce":219578185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6811","nonce":219578185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6815","nonce":219578185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6813","nonce":219578185}]},"public_addr":"172.21.15.98:6809/219578185","cluster_addr":"172.21.15.98:6811/219578185","heartbeat_back_addr":"172.21.15.98:6815/219578185","heartbeat_front_addr":"172.21.15.98:6813/219578185","state":["exists","up"]},{"osd":6,"uuid":"8c73e4b0-cb6b-486a-8b55-a0fe4d97a727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6817","nonce":1226172499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6819","nonce":1226172499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6823","nonce":1226172499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6821","nonce":1226172499}]},"public_addr":"172.21.15.98:6817/1226172499","cluster_addr":"172.21.15.98:6819/1226172499","heartbeat_back_addr":"172.21.15.98:6823/1226172499","heartbeat_front_addr":"172.21.15.98:6821/1226172499","state":["exists","up"]},{"osd":7,"uuid":"4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6825","nonce":148009079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6827","nonce":148009079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6831","nonce":148009079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":148009079},{"type":"v1","addr":"172.21.15.98:6829","nonce":148009079}]},"public_addr":"172.21.15.98:6825/148009079","cluster_addr":"172.21.15.98:6827/148009079","heartbeat_back_addr":"172.21.15.98:6831/148009079","heartbeat_front_addr":"172.21.15.98:6829/148009079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:45:39.555867+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:45:08.728923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:52.466235+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:59.838353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:31.136355+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:45:02.904291+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:51.617716+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:44:33.492968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/2738867930":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/1798147282":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/52352690":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/2928651991":"2024-04-03T04:34:47.037764+0000","172.21.15.69:6800/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/2763215770":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/3657409442":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/879855771":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1025808168":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1168952113":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/1892956550":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6801/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/54578327":"2024-04-03T04:44:51.699278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 10 librados: watch_flush enter 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 10 librados: watch_flush exit 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 20 client.24488.objecter shutdown clearing up homeless session... 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 10 client.24488.objecter successfully canceled tick 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0bd806f5e0 msgr2=0x7f0bd8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0bd806f5e0 0x7f0bd8071aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009680 tx=0x7f0be4009260 comp rx=0 tx=0).stop 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 msgr2=0x7f0bf01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf01011f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000c00 tx=0x7f0be00047b0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 shutdown_connections 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0bd806f5e0 0x7f0bd8071aa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bf0107680 0x7f0bf0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf006ab00 0x7f0bf01011f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 --2- 172.21.15.69:0/1498975866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf006b4b0 0x7f0bf0101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:45:45.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 >> 172.21.15.69:0/1498975866 conn(0x7f0bf0078240 msgr2=0x7f0bf00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:45:45.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 shutdown_connections 2024-04-02T04:45:45.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 -- 172.21.15.69:0/1498975866 wait complete. 2024-04-02T04:45:45.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:45:45.840+0000 7f0bf5005700 1 librados: shutdown 2024-04-02T04:45:45.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.747248063Z level=info msg="Executing migration" id="Update alert table charset" 2024-04-02T04:45:45.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.797697309Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-04-02T04:45:45.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.848116516Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-04-02T04:45:45.959 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.881701723Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-02T04:45:45.959 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.93205805Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-04-02T04:45:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[23790]: audit 2024-04-02T04:45:45.842328+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.69:0/1498975866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:45:46.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.957330443Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-04-02T04:45:46.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:45 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:45.999360726Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-02T04:45:46.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.024621146Z level=info msg="Executing migration" id="Add for to alert table" 2024-04-02T04:45:46.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.066729572Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-04-02T04:45:46.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.092022775Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-04-02T04:45:46.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.134028607Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-04-02T04:45:46.225 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.167734096Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-04-02T04:45:46.225 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.184699178Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-04-02T04:45:46.225 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.201658477Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-04-02T04:45:46.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:46 smithi069 bash[21528]: audit 2024-04-02T04:45:45.842328+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.69:0/1498975866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:45:46.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:46 smithi069 bash[28904]: audit 2024-04-02T04:45:45.842328+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.69:0/1498975866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.23541902Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.260796648Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.302770222Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.345065849Z level=info msg="Executing migration" id="create annotation table v5" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.378761878Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.412344302Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.446007515Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.479872953Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-04-02T04:45:46.515 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.513470873Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-04-02T04:45:46.800 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.547102273Z level=info msg="Executing migration" id="Update annotation table charset" 2024-04-02T04:45:46.800 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.580590751Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-04-02T04:45:46.800 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.614296924Z level=info msg="Executing migration" id="Drop category_id index" 2024-04-02T04:45:46.800 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.664598172Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-04-02T04:45:46.800 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.698267241Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-04-02T04:45:46.800 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.748609939Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-02T04:45:46.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-02T04:45:46.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-02T04:45:46.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-02T04:45:46.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-02T04:45:46.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-02T04:45:46.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-02T04:45:46.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-02T04:45:46.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-02T04:45:47.050 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.799009386Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-02T04:45:47.051 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.849293738Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-02T04:45:47.051 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.899680442Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-04-02T04:45:47.051 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.924932678Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-02T04:45:47.051 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.950278667Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-04-02T04:45:47.051 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:46 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:46.992239791Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-04-02T04:45:47.051 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.025941854Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-04-02T04:45:47.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:45.617543+0000 osd.0 (osd.0) 41 : cluster 0 3.6 scrub starts 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:45.620071+0000 osd.0 (osd.0) 42 : cluster 0 3.6 scrub ok 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:45.676932+0000 osd.3 (osd.3) 55 : cluster 0 3.2 scrub starts 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:45.679769+0000 osd.3 (osd.3) 56 : cluster 0 3.2 scrub ok 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:45.752162+0000 mgr.y (mgr.14538) 60 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:46.302266+0000 osd.7 (osd.7) 39 : cluster 0 2.b scrub starts 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[21528]: cluster 2024-04-02T04:45:46.305170+0000 osd.7 (osd.7) 40 : cluster 0 2.b scrub ok 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:45.617543+0000 osd.0 (osd.0) 41 : cluster 0 3.6 scrub starts 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:45.620071+0000 osd.0 (osd.0) 42 : cluster 0 3.6 scrub ok 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:45.676932+0000 osd.3 (osd.3) 55 : cluster 0 3.2 scrub starts 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:45.679769+0000 osd.3 (osd.3) 56 : cluster 0 3.2 scrub ok 2024-04-02T04:45:47.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:45.752162+0000 mgr.y (mgr.14538) 60 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:47.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:46.302266+0000 osd.7 (osd.7) 39 : cluster 0 2.b scrub starts 2024-04-02T04:45:47.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:47 smithi069 bash[28904]: cluster 2024-04-02T04:45:46.305170+0000 osd.7 (osd.7) 40 : cluster 0 2.b scrub ok 2024-04-02T04:45:47.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:45.617543+0000 osd.0 (osd.0) 41 : cluster 0 3.6 scrub starts 2024-04-02T04:45:47.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:45.620071+0000 osd.0 (osd.0) 42 : cluster 0 3.6 scrub ok 2024-04-02T04:45:47.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:45.676932+0000 osd.3 (osd.3) 55 : cluster 0 3.2 scrub starts 2024-04-02T04:45:47.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:45.679769+0000 osd.3 (osd.3) 56 : cluster 0 3.2 scrub ok 2024-04-02T04:45:47.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:45.752162+0000 mgr.y (mgr.14538) 60 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:47.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:46.302266+0000 osd.7 (osd.7) 39 : cluster 0 2.b scrub starts 2024-04-02T04:45:47.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[23790]: cluster 2024-04-02T04:45:46.305170+0000 osd.7 (osd.7) 40 : cluster 0 2.b scrub ok 2024-04-02T04:45:47.331 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.076496046Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-04-02T04:45:47.331 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.101803846Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-04-02T04:45:47.331 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.14375959Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-04-02T04:45:47.331 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.169069236Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-04-02T04:45:47.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.194417731Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-04-02T04:45:47.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.219636949Z level=info msg="Executing migration" id="Add epoch_end column" 2024-04-02T04:45:47.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.261641432Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-04-02T04:45:47.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.286994508Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-04-02T04:45:47.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.32882917Z level=info msg="Executing migration" id="Move region to single row" 2024-04-02T04:45:47.591 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.362716718Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.396601302Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.421862955Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.447267053Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.47255499Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.498846774Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.53991533Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-02T04:45:47.592 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.590423409Z level=info msg="Executing migration" id="create test_data table" 2024-04-02T04:45:47.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.624094629Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-04-02T04:45:47.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.67447644Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-04-02T04:45:47.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.724847701Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-02T04:45:47.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.758753998Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-04-02T04:45:47.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.784121147Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-04-02T04:45:47.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.826027688Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-04-02T04:45:48.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:48 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.326011+0000 osd.7 (osd.7) 41 : cluster 0 3.d scrub starts 2024-04-02T04:45:48.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:48 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.331021+0000 osd.7 (osd.7) 42 : cluster 0 3.d scrub ok 2024-04-02T04:45:48.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:48 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.326011+0000 osd.7 (osd.7) 41 : cluster 0 3.d scrub starts 2024-04-02T04:45:48.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:48 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.331021+0000 osd.7 (osd.7) 42 : cluster 0 3.d scrub ok 2024-04-02T04:45:48.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.326011+0000 osd.7 (osd.7) 41 : cluster 0 3.d scrub starts 2024-04-02T04:45:48.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.331021+0000 osd.7 (osd.7) 42 : cluster 0 3.d scrub ok 2024-04-02T04:45:48.380 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:47 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:47.985044572Z level=info msg="Executing migration" id="create team table" 2024-04-02T04:45:48.380 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.085667833Z level=info msg="Executing migration" id="add index team.org_id" 2024-04-02T04:45:48.380 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.186194561Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-04-02T04:45:48.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.37849772Z level=info msg="Executing migration" id="create team member table" 2024-04-02T04:45:48.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.437168856Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-04-02T04:45:48.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.462496855Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-04-02T04:45:48.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.487777762Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-04-02T04:45:48.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.513308502Z level=info msg="Executing migration" id="Add column email to team table" 2024-04-02T04:45:48.659 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.538574659Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-04-02T04:45:48.659 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.563872309Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-04-02T04:45:48.659 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.589208981Z level=info msg="Executing migration" id="create dashboard acl table" 2024-04-02T04:45:48.659 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.656285307Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-04-02T04:45:48.918 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.681634709Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-02T04:45:48.918 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.723708976Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-02T04:45:48.918 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.748956489Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-04-02T04:45:48.919 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.782672462Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-04-02T04:45:48.919 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.816197941Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-04-02T04:45:48.919 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.866531655Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-04-02T04:45:48.919 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.916839614Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-04-02T04:45:48.946 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:48 smithi069 systemd[1]: Stopping Ceph alertmanager.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T04:45:48.947 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:48 smithi069 bash[57978]: ts=2024-04-02T04:45:48.781Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-02T04:45:48.947 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:45:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:45:49.195 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:48 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:48.967322661Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-04-02T04:45:49.196 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.017650936Z level=info msg="Executing migration" id="create tag table" 2024-04-02T04:45:49.196 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.067974638Z level=info msg="Executing migration" id="add index tag.key_value" 2024-04-02T04:45:49.196 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.085064026Z level=info msg="Executing migration" id="create login attempt table" 2024-04-02T04:45:49.196 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.102045047Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-04-02T04:45:49.196 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.166632781Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-04-02T04:45:49.196 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.611999+0000 osd.0 (osd.0) 43 : cluster 0 4.6 scrub starts 2024-04-02T04:45:49.196 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.613396+0000 osd.0 (osd.0) 44 : cluster 0 4.6 scrub ok 2024-04-02T04:45:49.196 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.689142+0000 osd.3 (osd.3) 57 : cluster 0 6.1f scrub starts 2024-04-02T04:45:49.196 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.691974+0000 osd.3 (osd.3) 58 : cluster 0 6.1f scrub ok 2024-04-02T04:45:49.196 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[23790]: cluster 2024-04-02T04:45:47.753475+0000 mgr.y (mgr.14538) 61 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:49.227 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.611999+0000 osd.0 (osd.0) 43 : cluster 0 4.6 scrub starts 2024-04-02T04:45:49.227 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.613396+0000 osd.0 (osd.0) 44 : cluster 0 4.6 scrub ok 2024-04-02T04:45:49.227 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.689142+0000 osd.3 (osd.3) 57 : cluster 0 6.1f scrub starts 2024-04-02T04:45:49.227 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.691974+0000 osd.3 (osd.3) 58 : cluster 0 6.1f scrub ok 2024-04-02T04:45:49.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[21528]: cluster 2024-04-02T04:45:47.753475+0000 mgr.y (mgr.14538) 61 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:49.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.611999+0000 osd.0 (osd.0) 43 : cluster 0 4.6 scrub starts 2024-04-02T04:45:49.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.613396+0000 osd.0 (osd.0) 44 : cluster 0 4.6 scrub ok 2024-04-02T04:45:49.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.689142+0000 osd.3 (osd.3) 57 : cluster 0 6.1f scrub starts 2024-04-02T04:45:49.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.691974+0000 osd.3 (osd.3) 58 : cluster 0 6.1f scrub ok 2024-04-02T04:45:49.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[28904]: cluster 2024-04-02T04:45:47.753475+0000 mgr.y (mgr.14538) 61 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.194313122Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.227986162Z level=info msg="Executing migration" id="create login_attempt v2" 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.253255258Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.295423701Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.337398799Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.362722192Z level=info msg="Executing migration" id="create user auth table" 2024-04-02T04:45:49.457 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.387919199Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-02T04:45:49.458 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.430305477Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-04-02T04:45:49.458 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.455545407Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-04-02T04:45:49.527 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[59607]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-alertmanager-a 2024-04-02T04:45:49.527 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:49 smithi069 bash[59668]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-alertmanager-a 2024-04-02T04:45:49.527 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:49 smithi069 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-02T04:45:49.527 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:49 smithi069 systemd[1]: Stopped Ceph alertmanager.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:49.527 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:49 smithi069 systemd[1]: Started Ceph alertmanager.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:49.718 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.497531748Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-04-02T04:45:49.718 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.522883778Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-04-02T04:45:49.718 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.54819668Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-04-02T04:45:49.718 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.573459847Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-04-02T04:45:49.718 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.598764477Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-02T04:45:49.719 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.62408363Z level=info msg="Executing migration" id="create server_lock table" 2024-04-02T04:45:49.719 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.666027085Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-04-02T04:45:49.719 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.69138964Z level=info msg="Executing migration" id="create user auth token table" 2024-04-02T04:45:49.719 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.716545575Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-04-02T04:45:49.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.758672025Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-04-02T04:45:49.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.783957698Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-04-02T04:45:49.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.80930986Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-04-02T04:45:49.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.834554627Z level=info msg="Executing migration" id="create cache_data table" 2024-04-02T04:45:49.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.876561229Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-04-02T04:45:49.975 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.893506078Z level=info msg="Executing migration" id="create short_url table v1" 2024-04-02T04:45:49.975 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.910411347Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-04-02T04:45:49.975 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.927354236Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-02T04:45:49.975 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.944441726Z level=info msg="Executing migration" id="delete alert_definition table" 2024-04-02T04:45:50.404 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:49 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:49.986478991Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-04-02T04:45:50.404 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.011796414Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-04-02T04:45:50.404 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.037026201Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-04-02T04:45:50.404 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.079005476Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-04-02T04:45:50.404 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.104310852Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-04-02T04:45:50.404 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.146044815Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-04-02T04:45:50.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.402294434Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-04-02T04:45:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[23790]: audit 2024-04-02T04:45:49.474696+0000 mon.a (mon.0) 879 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[23790]: audit 2024-04-02T04:45:49.485433+0000 mon.a (mon.0) 880 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[23790]: cephadm 2024-04-02T04:45:49.495998+0000 mgr.y (mgr.14538) 62 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-04-02T04:45:50.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[28904]: audit 2024-04-02T04:45:49.474696+0000 mon.a (mon.0) 879 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:50.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[28904]: audit 2024-04-02T04:45:49.485433+0000 mon.a (mon.0) 880 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:50.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[28904]: cephadm 2024-04-02T04:45:49.495998+0000 mgr.y (mgr.14538) 62 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-04-02T04:45:50.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[21528]: audit 2024-04-02T04:45:49.474696+0000 mon.a (mon.0) 879 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:50.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[21528]: audit 2024-04-02T04:45:49.485433+0000 mon.a (mon.0) 880 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:50.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[21528]: cephadm 2024-04-02T04:45:49.495998+0000 mgr.y (mgr.14538) 62 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-04-02T04:45:50.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.375Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-02T04:45:50.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.375Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-02T04:45:50.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.377Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2024-04-02T04:45:50.778 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.378Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-02T04:45:50.779 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.408Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-02T04:45:50.779 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.409Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-02T04:45:50.779 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.412Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-02T04:45:50.779 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:50 smithi069 bash[59696]: ts=2024-04-02T04:45:50.412Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-02T04:45:51.214 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.807721232Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-04-02T04:45:51.214 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:50 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:50.925768135Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-04-02T04:45:51.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.213040343Z level=info msg="Executing migration" id="drop alert_definition table" 2024-04-02T04:45:51.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.377329854Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-04-02T04:45:51.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.452927094Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-04-02T04:45:51.870 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.553510922Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-02T04:45:51.871 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.662390642Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-02T04:45:51.871 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.712685679Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-02T04:45:51.871 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.771435418Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-04-02T04:45:51.871 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[23790]: cephadm 2024-04-02T04:45:49.731791+0000 mgr.y (mgr.14538) 63 : cephadm 1 Reconfiguring daemon prometheus.a on smithi098 2024-04-02T04:45:51.871 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[23790]: cluster 2024-04-02T04:45:49.753844+0000 mgr.y (mgr.14538) 64 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:51.871 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[23790]: cluster 2024-04-02T04:45:50.837981+0000 osd.4 (osd.4) 39 : cluster 0 6.17 scrub starts 2024-04-02T04:45:51.871 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[23790]: cluster 2024-04-02T04:45:50.839962+0000 osd.4 (osd.4) 40 : cluster 0 6.17 scrub ok 2024-04-02T04:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.946 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.947 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.959 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.961 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:45:51.978 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[21528]: cephadm 2024-04-02T04:45:49.731791+0000 mgr.y (mgr.14538) 63 : cephadm 1 Reconfiguring daemon prometheus.a on smithi098 2024-04-02T04:45:51.978 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[21528]: cluster 2024-04-02T04:45:49.753844+0000 mgr.y (mgr.14538) 64 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:51.978 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[21528]: cluster 2024-04-02T04:45:50.837981+0000 osd.4 (osd.4) 39 : cluster 0 6.17 scrub starts 2024-04-02T04:45:51.979 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[21528]: cluster 2024-04-02T04:45:50.839962+0000 osd.4 (osd.4) 40 : cluster 0 6.17 scrub ok 2024-04-02T04:45:51.979 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[28904]: cephadm 2024-04-02T04:45:49.731791+0000 mgr.y (mgr.14538) 63 : cephadm 1 Reconfiguring daemon prometheus.a on smithi098 2024-04-02T04:45:51.979 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[28904]: cluster 2024-04-02T04:45:49.753844+0000 mgr.y (mgr.14538) 64 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:45:51.979 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[28904]: cluster 2024-04-02T04:45:50.837981+0000 osd.4 (osd.4) 39 : cluster 0 6.17 scrub starts 2024-04-02T04:45:51.979 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:51 smithi069 bash[28904]: cluster 2024-04-02T04:45:50.839962+0000 osd.4 (osd.4) 40 : cluster 0 6.17 scrub ok 2024-04-02T04:45:52.220 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.869565151Z level=info msg="Executing migration" id="create alert_instance table" 2024-04-02T04:45:52.220 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.923078375Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-02T04:45:52.220 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:51 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:51.973350302Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-02T04:45:52.221 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:52.058025743Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-04-02T04:45:52.221 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:52.119591959Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-02T04:45:52.221 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:52.219105426Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-04-02T04:45:52.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:52.319768368Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-04-02T04:45:52.777 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[59696]: ts=2024-04-02T04:45:52.378Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000194704s 2024-04-02T04:45:52.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.765167+0000 mon.b (mon.1) 70 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-02T04:45:52.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.765879+0000 mon.b (mon.1) 71 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T04:45:52.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.766408+0000 mon.b (mon.1) 72 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.766901+0000 mon.b (mon.1) 73 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.771976+0000 mon.a (mon.0) 881 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-02T04:45:52.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.772895+0000 mon.a (mon.0) 882 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.773676+0000 mon.a (mon.0) 883 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.774154+0000 mon.a (mon.0) 884 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: cluster 2024-04-02T04:45:51.842176+0000 osd.7 (osd.7) 43 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: cluster 2024-04-02T04:45:51.842292+0000 osd.7 (osd.7) 44 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[21528]: audit 2024-04-02T04:45:51.925283+0000 mon.b (mon.1) 74 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.765167+0000 mon.b (mon.1) 70 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.765879+0000 mon.b (mon.1) 71 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.766408+0000 mon.b (mon.1) 72 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.766901+0000 mon.b (mon.1) 73 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.771976+0000 mon.a (mon.0) 881 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.772895+0000 mon.a (mon.0) 882 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.773676+0000 mon.a (mon.0) 883 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.774154+0000 mon.a (mon.0) 884 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: cluster 2024-04-02T04:45:51.842176+0000 osd.7 (osd.7) 43 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: cluster 2024-04-02T04:45:51.842292+0000 osd.7 (osd.7) 44 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:52.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:52 smithi069 bash[28904]: audit 2024-04-02T04:45:51.925283+0000 mon.b (mon.1) 74 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:52.824 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:52.528008951Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.765167+0000 mon.b (mon.1) 70 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.765879+0000 mon.b (mon.1) 71 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.766408+0000 mon.b (mon.1) 72 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.766901+0000 mon.b (mon.1) 73 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.771976+0000 mon.a (mon.0) 881 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.772895+0000 mon.a (mon.0) 882 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.773676+0000 mon.a (mon.0) 883 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.825 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.774154+0000 mon.a (mon.0) 884 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-02T04:45:52.826 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: cluster 2024-04-02T04:45:51.842176+0000 osd.7 (osd.7) 43 : cluster 0 purged_snaps scrub starts 2024-04-02T04:45:52.826 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: cluster 2024-04-02T04:45:51.842292+0000 osd.7 (osd.7) 44 : cluster 0 purged_snaps scrub ok 2024-04-02T04:45:52.826 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[23790]: audit 2024-04-02T04:45:51.925283+0000 mon.b (mon.1) 74 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:53.081 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:52.915805604Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-02T04:45:53.081 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:45:52 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:45:53.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:53.079223805Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-04-02T04:45:53.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:53.155417787Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-02T04:45:53.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:53.231620722Z level=info msg="Executing migration" id="create alert_rule table" 2024-04-02T04:45:53.710 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:53.399179347Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-04-02T04:45:53.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[23790]: cluster 2024-04-02T04:45:51.754445+0000 mgr.y (mgr.14538) 65 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-04-02T04:45:53.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[23790]: audit 2024-04-02T04:45:52.557725+0000 mon.a (mon.0) 885 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-02T04:45:53.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[23790]: audit 2024-04-02T04:45:52.558005+0000 mon.a (mon.0) 886 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-02T04:45:53.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[23790]: audit 2024-04-02T04:45:52.558232+0000 mon.a (mon.0) 887 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-02T04:45:53.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[23790]: audit 2024-04-02T04:45:52.558360+0000 mon.a (mon.0) 888 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-02T04:45:53.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[23790]: cluster 2024-04-02T04:45:52.571838+0000 mon.a (mon.0) 889 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-04-02T04:45:53.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:53.709176077Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-04-02T04:45:53.974 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:53 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:53.860455187Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-02T04:45:54.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[21528]: cluster 2024-04-02T04:45:51.754445+0000 mgr.y (mgr.14538) 65 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[21528]: audit 2024-04-02T04:45:52.557725+0000 mon.a (mon.0) 885 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[21528]: audit 2024-04-02T04:45:52.558005+0000 mon.a (mon.0) 886 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[21528]: audit 2024-04-02T04:45:52.558232+0000 mon.a (mon.0) 887 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[21528]: audit 2024-04-02T04:45:52.558360+0000 mon.a (mon.0) 888 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[21528]: cluster 2024-04-02T04:45:52.571838+0000 mon.a (mon.0) 889 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[28904]: cluster 2024-04-02T04:45:51.754445+0000 mgr.y (mgr.14538) 65 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[28904]: audit 2024-04-02T04:45:52.557725+0000 mon.a (mon.0) 885 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-02T04:45:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[28904]: audit 2024-04-02T04:45:52.558005+0000 mon.a (mon.0) 886 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-02T04:45:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[28904]: audit 2024-04-02T04:45:52.558232+0000 mon.a (mon.0) 887 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-02T04:45:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[28904]: audit 2024-04-02T04:45:52.558360+0000 mon.a (mon.0) 888 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-02T04:45:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:53 smithi069 bash[28904]: cluster 2024-04-02T04:45:52.571838+0000 mon.a (mon.0) 889 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-04-02T04:45:54.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.236302794Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-04-02T04:45:54.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.387214873Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-04-02T04:45:54.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.479950094Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-04-02T04:45:54.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.563893196Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-04-02T04:45:54.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[21528]: audit 2024-04-02T04:45:52.794314+0000 mgr.y (mgr.14538) 66 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:54.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[21528]: cluster 2024-04-02T04:45:53.373217+0000 osd.7 (osd.7) 45 : cluster 0 6.12 deep-scrub starts 2024-04-02T04:45:54.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[21528]: cluster 2024-04-02T04:45:53.375657+0000 osd.7 (osd.7) 46 : cluster 0 6.12 deep-scrub ok 2024-04-02T04:45:54.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[21528]: cluster 2024-04-02T04:45:53.568295+0000 mon.a (mon.0) 890 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-04-02T04:45:54.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[28904]: audit 2024-04-02T04:45:52.794314+0000 mgr.y (mgr.14538) 66 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:54.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[28904]: cluster 2024-04-02T04:45:53.373217+0000 osd.7 (osd.7) 45 : cluster 0 6.12 deep-scrub starts 2024-04-02T04:45:54.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[28904]: cluster 2024-04-02T04:45:53.375657+0000 osd.7 (osd.7) 46 : cluster 0 6.12 deep-scrub ok 2024-04-02T04:45:54.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:54 smithi069 bash[28904]: cluster 2024-04-02T04:45:53.568295+0000 mon.a (mon.0) 890 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-04-02T04:45:54.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[23790]: audit 2024-04-02T04:45:52.794314+0000 mgr.y (mgr.14538) 66 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:45:54.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[23790]: cluster 2024-04-02T04:45:53.373217+0000 osd.7 (osd.7) 45 : cluster 0 6.12 deep-scrub starts 2024-04-02T04:45:54.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[23790]: cluster 2024-04-02T04:45:53.375657+0000 osd.7 (osd.7) 46 : cluster 0 6.12 deep-scrub ok 2024-04-02T04:45:54.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[23790]: cluster 2024-04-02T04:45:53.568295+0000 mon.a (mon.0) 890 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-04-02T04:45:54.884 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.66055827Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-04-02T04:45:54.884 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.714768367Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-02T04:45:54.884 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.794322614Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-04-02T04:45:54.885 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.882485513Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-04-02T04:45:55.136 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.924406386Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-02T04:45:55.136 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:54 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:54.975236344Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-02T04:45:55.137 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:55.051229846Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-02T04:45:55.137 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:55.135125965Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-02T04:45:55.448 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:55.235761183Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-04-02T04:45:55.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:55.446754424Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-02T04:45:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[23790]: cluster 2024-04-02T04:45:53.755147+0000 mgr.y (mgr.14538) 67 : cluster 0 pgmap v42: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:55.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:55 smithi069 bash[21528]: cluster 2024-04-02T04:45:53.755147+0000 mgr.y (mgr.14538) 67 : cluster 0 pgmap v42: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:55.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:55 smithi069 bash[28904]: cluster 2024-04-02T04:45:53.755147+0000 mgr.y (mgr.14538) 67 : cluster 0 pgmap v42: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:56.095 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:55.792011923Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-02T04:45:56.095 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:55 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:55.930176476Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-02T04:45:56.095 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.006229133Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-04-02T04:45:56.412 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.093562647Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-04-02T04:45:56.412 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.157805643Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-04-02T04:45:56.412 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.208419169Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-02T04:45:56.412 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.267738237Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-02T04:45:56.412 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.318528999Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-02T04:45:56.412 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.410938853Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 systemd[1]: Stopping Ceph prometheus.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.239Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.240Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-02T04:45:56.413 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.240Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-02T04:45:56.414 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.240Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-02T04:45:56.414 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.240Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-02T04:45:56.414 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[49697]: ts=2024-04-02T04:45:56.240Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-02T04:45:56.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.513836008Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-04-02T04:45:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[23790]: cluster 2024-04-02T04:45:55.848832+0000 osd.4 (osd.4) 41 : cluster 0 6.1e scrub starts 2024-04-02T04:45:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[23790]: cluster 2024-04-02T04:45:55.851282+0000 osd.4 (osd.4) 42 : cluster 0 6.1e scrub ok 2024-04-02T04:45:56.847 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:56 smithi069 bash[28904]: cluster 2024-04-02T04:45:55.848832+0000 osd.4 (osd.4) 41 : cluster 0 6.1e scrub starts 2024-04-02T04:45:56.847 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:56 smithi069 bash[28904]: cluster 2024-04-02T04:45:55.851282+0000 osd.4 (osd.4) 42 : cluster 0 6.1e scrub ok 2024-04-02T04:45:56.847 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:56 smithi069 bash[21528]: cluster 2024-04-02T04:45:55.848832+0000 osd.4 (osd.4) 41 : cluster 0 6.1e scrub starts 2024-04-02T04:45:56.848 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:56 smithi069 bash[21528]: cluster 2024-04-02T04:45:55.851282+0000 osd.4 (osd.4) 42 : cluster 0 6.1e scrub ok 2024-04-02T04:45:57.132 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:56 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:56.745616114Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-02T04:45:57.392 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[52256]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-prometheus-a 2024-04-02T04:45:57.392 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.130835173Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-04-02T04:45:57.392 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.297965393Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-04-02T04:45:57.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.390408943Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-02T04:45:57.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.466789097Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-04-02T04:45:57.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.483873343Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-04-02T04:45:57.724 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.630510244Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-02T04:45:57.724 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[52314]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-prometheus-a 2024-04-02T04:45:57.725 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:57 smithi098 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-02T04:45:57.725 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:57 smithi098 systemd[1]: Stopped Ceph prometheus.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:57.725 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:57 smithi098 systemd[1]: Started Ceph prometheus.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: cluster 2024-04-02T04:45:55.755548+0000 mgr.y (mgr.14538) 68 : cluster 0 pgmap v43: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: cluster 2024-04-02T04:45:56.826708+0000 osd.4 (osd.4) 43 : cluster 0 6.1a scrub starts 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: cluster 2024-04-02T04:45:56.829138+0000 osd.4 (osd.4) 44 : cluster 0 6.1a scrub ok 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: audit 2024-04-02T04:45:57.514000+0000 mon.a (mon.0) 891 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: audit 2024-04-02T04:45:57.530592+0000 mon.a (mon.0) 892 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: audit 2024-04-02T04:45:57.573295+0000 mon.b (mon.1) 75 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:45:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: audit 2024-04-02T04:45:57.576363+0000 mon.b (mon.1) 76 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:45:57.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[23790]: audit 2024-04-02T04:45:57.592673+0000 mon.a (mon.0) 893 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: cluster 2024-04-02T04:45:55.755548+0000 mgr.y (mgr.14538) 68 : cluster 0 pgmap v43: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:45:58.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: cluster 2024-04-02T04:45:56.826708+0000 osd.4 (osd.4) 43 : cluster 0 6.1a scrub starts 2024-04-02T04:45:58.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: cluster 2024-04-02T04:45:56.829138+0000 osd.4 (osd.4) 44 : cluster 0 6.1a scrub ok 2024-04-02T04:45:58.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: audit 2024-04-02T04:45:57.514000+0000 mon.a (mon.0) 891 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: audit 2024-04-02T04:45:57.530592+0000 mon.a (mon.0) 892 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: audit 2024-04-02T04:45:57.573295+0000 mon.b (mon.1) 75 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:45:58.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: audit 2024-04-02T04:45:57.576363+0000 mon.b (mon.1) 76 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:45:58.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21528]: audit 2024-04-02T04:45:57.592673+0000 mon.a (mon.0) 893 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.048 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STOPPING 2024-04-02T04:45:58.048 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STOPPED 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STARTING 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Serving on http://:::9283 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STARTED 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STOPPING 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STOPPED 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STARTING 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Serving on http://:::9283 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STARTED 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STOPPING 2024-04-02T04:45:58.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: cluster 2024-04-02T04:45:55.755548+0000 mgr.y (mgr.14538) 68 : cluster 0 pgmap v43: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: cluster 2024-04-02T04:45:56.826708+0000 osd.4 (osd.4) 43 : cluster 0 6.1a scrub starts 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: cluster 2024-04-02T04:45:56.829138+0000 osd.4 (osd.4) 44 : cluster 0 6.1a scrub ok 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: audit 2024-04-02T04:45:57.514000+0000 mon.a (mon.0) 891 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: audit 2024-04-02T04:45:57.530592+0000 mon.a (mon.0) 892 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: audit 2024-04-02T04:45:57.573295+0000 mon.b (mon.1) 75 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: audit 2024-04-02T04:45:57.576363+0000 mon.b (mon.1) 76 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:45:58.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:57 smithi069 bash[28904]: audit 2024-04-02T04:45:57.592673+0000 mon.a (mon.0) 893 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.341 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:57 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:57.997673144Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-02T04:45:58.370 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-02T04:45:58.370 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STOPPED 2024-04-02T04:45:58.370 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STARTING 2024-04-02T04:45:58.370 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Serving on http://:::9283 2024-04-02T04:45:58.370 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21782]: [02/Apr/2024:04:45:57] ENGINE Bus STARTED 2024-04-02T04:45:58.600 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.339767507Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-02T04:45:58.600 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.531168987Z level=info msg="Executing migration" id="create alert_image table" 2024-04-02T04:45:58.600 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.598275134Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-02T04:45:58.719 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: cluster 2024-04-02T04:45:57.515391+0000 osd.0 (osd.0) 45 : cluster 0 3.1b scrub starts 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: cluster 2024-04-02T04:45:57.518338+0000 osd.0 (osd.0) 46 : cluster 0 3.1b scrub ok 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.580127+0000 mgr.y (mgr.14538) 69 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.582993+0000 mgr.y (mgr.14538) 70 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.612263+0000 mon.b (mon.1) 77 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.618742+0000 mon.b (mon.1) 78 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.618786+0000 mgr.y (mgr.14538) 71 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.625207+0000 mgr.y (mgr.14538) 72 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.636997+0000 mon.a (mon.0) 894 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.641316+0000 mon.b (mon.1) 79 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.645500+0000 mon.b (mon.1) 80 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.658371+0000 mon.a (mon.0) 895 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: audit 2024-04-02T04:45:57.672180+0000 mon.b (mon.1) 81 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:58.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: cluster 2024-04-02T04:45:57.719149+0000 osd.3 (osd.3) 59 : cluster 0 2.1b scrub starts 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[28904]: cluster 2024-04-02T04:45:57.720714+0000 osd.3 (osd.3) 60 : cluster 0 2.1b scrub ok 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: cluster 2024-04-02T04:45:57.515391+0000 osd.0 (osd.0) 45 : cluster 0 3.1b scrub starts 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: cluster 2024-04-02T04:45:57.518338+0000 osd.0 (osd.0) 46 : cluster 0 3.1b scrub ok 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.580127+0000 mgr.y (mgr.14538) 69 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.582993+0000 mgr.y (mgr.14538) 70 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.612263+0000 mon.b (mon.1) 77 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.618742+0000 mon.b (mon.1) 78 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.618786+0000 mgr.y (mgr.14538) 71 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.625207+0000 mgr.y (mgr.14538) 72 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:45:58.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.636997+0000 mon.a (mon.0) 894 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.641316+0000 mon.b (mon.1) 79 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:45:58.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.645500+0000 mon.b (mon.1) 80 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:45:58.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.658371+0000 mon.a (mon.0) 895 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: audit 2024-04-02T04:45:57.672180+0000 mon.b (mon.1) 81 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:58.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: cluster 2024-04-02T04:45:57.719149+0000 osd.3 (osd.3) 59 : cluster 0 2.1b scrub starts 2024-04-02T04:45:58.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:58 smithi069 bash[21528]: cluster 2024-04-02T04:45:57.720714+0000 osd.3 (osd.3) 60 : cluster 0 2.1b scrub ok 2024-04-02T04:45:58.853 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.668424336Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-02T04:45:58.853 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.770605408Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-02T04:45:58.853 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.851674446Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-02T04:45:58.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: cluster 2024-04-02T04:45:57.515391+0000 osd.0 (osd.0) 45 : cluster 0 3.1b scrub starts 2024-04-02T04:45:58.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: cluster 2024-04-02T04:45:57.518338+0000 osd.0 (osd.0) 46 : cluster 0 3.1b scrub ok 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.580127+0000 mgr.y (mgr.14538) 69 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.582993+0000 mgr.y (mgr.14538) 70 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.612263+0000 mon.b (mon.1) 77 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.618742+0000 mon.b (mon.1) 78 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.618786+0000 mgr.y (mgr.14538) 71 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.625207+0000 mgr.y (mgr.14538) 72 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.636997+0000 mon.a (mon.0) 894 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.641316+0000 mon.b (mon.1) 79 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.645500+0000 mon.b (mon.1) 80 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.658371+0000 mon.a (mon.0) 895 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: audit 2024-04-02T04:45:57.672180+0000 mon.b (mon.1) 81 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: cluster 2024-04-02T04:45:57.719149+0000 osd.3 (osd.3) 59 : cluster 0 2.1b scrub starts 2024-04-02T04:45:58.854 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[23790]: cluster 2024-04-02T04:45:57.720714+0000 osd.3 (osd.3) 60 : cluster 0 2.1b scrub ok 2024-04-02T04:45:59.111 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.934485447Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-02T04:45:59.111 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:58 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:58.934940146Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-02T04:45:59.111 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.006290892Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-02T04:45:59.111 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.06827739Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-02T04:45:59.399 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.127649493Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-04-02T04:45:59.399 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.12790986Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.12815432Z level=info msg="alerts found to migrate" alerts=0 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.179557978Z level=info msg="Executing migration" id="create library_element table v1" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.237731936Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.263019079Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.288433906Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.330418183Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.356096404Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.398019218Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.184Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.184Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-02T04:45:59.400 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.184Z caller=main.go:570 level=info host_details="(Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 smithi098 (none))" 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.184Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.184Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.186Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.186Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.187Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.187Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.190Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.190Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.786µs 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.190Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.191Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.191Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.191Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.47µs wal_replay_duration=803.168µs wbl_replay_duration=134ns total_replay_duration=863.978µs 2024-04-02T04:45:59.401 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.192Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-02T04:45:59.402 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.192Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-02T04:45:59.402 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.192Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-02T04:45:59.402 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.214Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.365576ms db_storage=962ns remote_storage=1.466µs web_handler=458ns query_engine=577ns scrape=262.136µs scrape_sd=79.193µs notify=10.27µs notify_sd=15.169µs rules=20.751484ms tracing=6.607µs 2024-04-02T04:45:59.402 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.214Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-02T04:45:59.402 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[52341]: ts=2024-04-02T04:45:59.214Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-02T04:45:59.685 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.440162304Z level=info msg="Executing migration" id="create data_keys table" 2024-04-02T04:45:59.685 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.48217807Z level=info msg="Executing migration" id="create secrets table" 2024-04-02T04:45:59.685 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.507570034Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-02T04:45:59.685 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.549469522Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-02T04:45:59.685 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.574713077Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-02T04:45:59.686 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.600074427Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-02T04:45:59.686 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.642150764Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-02T04:45:59.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[23790]: audit 2024-04-02T04:45:57.647689+0000 mgr.y (mgr.14538) 73 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:45:59.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[23790]: audit 2024-04-02T04:45:57.651730+0000 mgr.y (mgr.14538) 74 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:45:59.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[23790]: cluster 2024-04-02T04:45:57.756239+0000 mgr.y (mgr.14538) 75 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:45:59.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[23790]: cluster 2024-04-02T04:45:58.495690+0000 osd.0 (osd.0) 47 : cluster 0 6.9 scrub starts 2024-04-02T04:45:59.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[23790]: cluster 2024-04-02T04:45:58.497434+0000 osd.0 (osd.0) 48 : cluster 0 6.9 scrub ok 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.684141301Z level=info msg="Executing migration" id="create kv_store table v1" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.726144171Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.751599758Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.793554568Z level=info msg="Executing migration" id="create permission table" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.818940379Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.844249896Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.886261929Z level=info msg="Executing migration" id="create role table" 2024-04-02T04:45:59.938 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.936611878Z level=info msg="Executing migration" id="add column display_name" 2024-04-02T04:46:00.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[21528]: audit 2024-04-02T04:45:57.647689+0000 mgr.y (mgr.14538) 73 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[21528]: audit 2024-04-02T04:45:57.651730+0000 mgr.y (mgr.14538) 74 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[21528]: cluster 2024-04-02T04:45:57.756239+0000 mgr.y (mgr.14538) 75 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[21528]: cluster 2024-04-02T04:45:58.495690+0000 osd.0 (osd.0) 47 : cluster 0 6.9 scrub starts 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[21528]: cluster 2024-04-02T04:45:58.497434+0000 osd.0 (osd.0) 48 : cluster 0 6.9 scrub ok 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[28904]: audit 2024-04-02T04:45:57.647689+0000 mgr.y (mgr.14538) 73 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[28904]: audit 2024-04-02T04:45:57.651730+0000 mgr.y (mgr.14538) 74 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[28904]: cluster 2024-04-02T04:45:57.756239+0000 mgr.y (mgr.14538) 75 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:00.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[28904]: cluster 2024-04-02T04:45:58.495690+0000 osd.0 (osd.0) 47 : cluster 0 6.9 scrub starts 2024-04-02T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:45:59 smithi069 bash[28904]: cluster 2024-04-02T04:45:58.497434+0000 osd.0 (osd.0) 48 : cluster 0 6.9 scrub ok 2024-04-02T04:46:00.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.961825943Z level=info msg="Executing migration" id="add column group_name" 2024-04-02T04:46:00.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:45:59 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:45:59.987188061Z level=info msg="Executing migration" id="add index role.org_id" 2024-04-02T04:46:00.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.012517338Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-04-02T04:46:00.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.037778913Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-04-02T04:46:00.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.096372784Z level=info msg="Executing migration" id="create team role table" 2024-04-02T04:46:00.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.138356341Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-04-02T04:46:00.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.163548531Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-04-02T04:46:00.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.189376273Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-04-02T04:46:00.451 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.231445233Z level=info msg="Executing migration" id="create user role table" 2024-04-02T04:46:00.451 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.273478126Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-04-02T04:46:00.452 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.315519136Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-04-02T04:46:00.452 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.357756748Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-04-02T04:46:00.452 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.383078482Z level=info msg="Executing migration" id="create builtin role table" 2024-04-02T04:46:00.452 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.408335244Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-04-02T04:46:00.647 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[59696]: ts=2024-04-02T04:46:00.381Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003044467s 2024-04-02T04:46:00.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[23790]: cluster 2024-04-02T04:45:59.474419+0000 osd.0 (osd.0) 49 : cluster 0 6.10 scrub starts 2024-04-02T04:46:00.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[23790]: cluster 2024-04-02T04:45:59.476629+0000 osd.0 (osd.0) 50 : cluster 0 6.10 scrub ok 2024-04-02T04:46:00.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[23790]: cluster 2024-04-02T04:46:00.355175+0000 osd.4 (osd.4) 45 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:00.704 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[23790]: cluster 2024-04-02T04:46:00.355291+0000 osd.4 (osd.4) 46 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.450369691Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.475675896Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.517743792Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.54310537Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.568313252Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.593608816Z level=info msg="Executing migration" id="add unique index role.uid" 2024-04-02T04:46:00.705 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.635826445Z level=info msg="Executing migration" id="create seed assignment table" 2024-04-02T04:46:00.957 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.702956409Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-04-02T04:46:00.957 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.745042525Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-02T04:46:00.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.770368233Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-02T04:46:00.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.795695696Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-02T04:46:00.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.820944126Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-02T04:46:00.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.862957956Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-02T04:46:00.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.888462822Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-02T04:46:00.958 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.913721737Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-02T04:46:01.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[21528]: cluster 2024-04-02T04:45:59.474419+0000 osd.0 (osd.0) 49 : cluster 0 6.10 scrub starts 2024-04-02T04:46:01.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[21528]: cluster 2024-04-02T04:45:59.476629+0000 osd.0 (osd.0) 50 : cluster 0 6.10 scrub ok 2024-04-02T04:46:01.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[21528]: cluster 2024-04-02T04:46:00.355175+0000 osd.4 (osd.4) 45 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:01.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[21528]: cluster 2024-04-02T04:46:00.355291+0000 osd.4 (osd.4) 46 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:01.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[28904]: cluster 2024-04-02T04:45:59.474419+0000 osd.0 (osd.0) 49 : cluster 0 6.10 scrub starts 2024-04-02T04:46:01.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[28904]: cluster 2024-04-02T04:45:59.476629+0000 osd.0 (osd.0) 50 : cluster 0 6.10 scrub ok 2024-04-02T04:46:01.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[28904]: cluster 2024-04-02T04:46:00.355175+0000 osd.4 (osd.4) 45 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:01.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:00 smithi069 bash[28904]: cluster 2024-04-02T04:46:00.355291+0000 osd.4 (osd.4) 46 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:00 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:00.955775757Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.006629974Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.031980002Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.074108414Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.116148788Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.149899947Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.166829606Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-02T04:46:01.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.192140322Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-02T04:46:01.522 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.23410137Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-02T04:46:01.522 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.25936139Z level=info msg="Executing migration" id="add correlation config column" 2024-04-02T04:46:01.522 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.284640246Z level=info msg="Executing migration" id="create entity_events table" 2024-04-02T04:46:01.522 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.310020938Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.3354456Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.335692672Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.360705751Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.360916321Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.385925914Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.411295983Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.453526674Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.478836383Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-02T04:46:01.523 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.520742218Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-02T04:46:01.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[21528]: cluster 2024-04-02T04:45:59.756850+0000 mgr.y (mgr.14538) 76 : cluster 0 pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:01.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[21528]: cluster 2024-04-02T04:46:00.625581+0000 mon.a (mon.0) 896 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-04-02T04:46:01.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[21528]: cluster 2024-04-02T04:46:00.846837+0000 osd.4 (osd.4) 47 : cluster 0 4.1 scrub starts 2024-04-02T04:46:01.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[21528]: cluster 2024-04-02T04:46:00.849143+0000 osd.4 (osd.4) 48 : cluster 0 4.1 scrub ok 2024-04-02T04:46:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[28904]: cluster 2024-04-02T04:45:59.756850+0000 mgr.y (mgr.14538) 76 : cluster 0 pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[28904]: cluster 2024-04-02T04:46:00.625581+0000 mon.a (mon.0) 896 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-04-02T04:46:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[28904]: cluster 2024-04-02T04:46:00.846837+0000 osd.4 (osd.4) 47 : cluster 0 4.1 scrub starts 2024-04-02T04:46:01.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:01 smithi069 bash[28904]: cluster 2024-04-02T04:46:00.849143+0000 osd.4 (osd.4) 48 : cluster 0 4.1 scrub ok 2024-04-02T04:46:01.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[23790]: cluster 2024-04-02T04:45:59.756850+0000 mgr.y (mgr.14538) 76 : cluster 0 pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:01.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[23790]: cluster 2024-04-02T04:46:00.625581+0000 mon.a (mon.0) 896 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-04-02T04:46:01.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[23790]: cluster 2024-04-02T04:46:00.846837+0000 osd.4 (osd.4) 47 : cluster 0 4.1 scrub starts 2024-04-02T04:46:01.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[23790]: cluster 2024-04-02T04:46:00.849143+0000 osd.4 (osd.4) 48 : cluster 0 4.1 scrub ok 2024-04-02T04:46:01.793 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.537758467Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-02T04:46:01.793 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.554765373Z level=info msg="Executing migration" id="Drop public config table" 2024-04-02T04:46:01.793 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.571769653Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-02T04:46:01.793 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.605422229Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-02T04:46:01.794 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.639148136Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-02T04:46:01.794 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.673051265Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-02T04:46:01.794 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.698274393Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-02T04:46:01.794 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.723713796Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-02T04:46:01.794 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.765588929Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-02T04:46:01.794 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.791151298Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-02T04:46:02.046 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.816439126Z level=info msg="Executing migration" id="add share column" 2024-04-02T04:46:02.046 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.858398428Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-02T04:46:02.046 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.883860927Z level=info msg="Executing migration" id="create file table" 2024-04-02T04:46:02.046 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.909177659Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-02T04:46:02.047 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.934532393Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-02T04:46:02.047 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.968229425Z level=info msg="Executing migration" id="create file_meta table" 2024-04-02T04:46:02.047 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:01 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:01.993529927Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-02T04:46:02.047 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.018806155Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-02T04:46:02.047 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.043996425Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.086072445Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.128110025Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.153395057Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.178781131Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.220703266Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.262932325Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.288364692Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-02T04:46:02.332 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.330269451Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-02T04:46:02.587 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.355719861Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-02T04:46:02.587 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.380967993Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-02T04:46:02.587 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.423297577Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-02T04:46:02.587 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.448659868Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-02T04:46:02.587 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.532364264Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-02T04:46:02.588 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.557641966Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-02T04:46:02.588 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.582965Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-02T04:46:02.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:46:02.863 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.642205179Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-02T04:46:02.863 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.683912836Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-02T04:46:02.863 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.70922468Z level=info msg="Executing migration" id="create folder table" 2024-04-02T04:46:02.863 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.751308445Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-02T04:46:02.863 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.776684478Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-02T04:46:02.864 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.818800065Z level=info msg="Executing migration" id="Update folder title length" 2024-04-02T04:46:02.864 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.86072274Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-02T04:46:02.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.452696+0000 osd.0 (osd.0) 51 : cluster 0 6.16 deep-scrub starts 2024-04-02T04:46:02.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.454798+0000 osd.0 (osd.0) 52 : cluster 0 6.16 deep-scrub ok 2024-04-02T04:46:02.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.648968+0000 osd.3 (osd.3) 61 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:02.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.649085+0000 osd.3 (osd.3) 62 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:02.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.854406+0000 osd.4 (osd.4) 49 : cluster 0 6.e scrub starts 2024-04-02T04:46:02.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.856645+0000 osd.4 (osd.4) 50 : cluster 0 6.e scrub ok 2024-04-02T04:46:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3558837105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc1018c0 msgr2=0x7f78fc101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- 172.21.15.69:0/3558837105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc1018c0 0x7f78fc101d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0098f0 tx=0x7f78ec009c00 comp rx=0 tx=0).stop 2024-04-02T04:46:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3558837105 shutdown_connections 2024-04-02T04:46:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- 172.21.15.69:0/3558837105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78fc102280 0x7f78fc10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:02.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- 172.21.15.69:0/3558837105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc106190 0x7f78fc106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- 172.21.15.69:0/3558837105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc1018c0 0x7f78fc101d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3558837105 >> 172.21.15.69:0/3558837105 conn(0x7f78fc077740 msgr2=0x7f78fc0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3558837105 shutdown_connections 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3558837105 wait complete. 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 librados: starting msgr at 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 librados: starting objecter 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 Processor -- start 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- start start 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 librados: setting wanted keys 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 librados: calling monclient init 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78fc1018c0 0x7f78fc19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:02.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc102280 0x7f78fc19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc106190 0x7f78fc19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78fc110960 con 0x7f78fc106190 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78fc1107e0 con 0x7f78fc1018c0 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f78fc110ae0 con 0x7f78fc102280 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7901477700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc102280 0x7f78fc19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc106190 0x7f78fc19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7901477700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc102280 0x7f78fc19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:51538/0 (socket says 172.21.15.69:51538) 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7901477700 1 -- 172.21.15.69:0/3928890846 learned_addr learned my addr 172.21.15.69:0/3928890846 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7901c78700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78fc1018c0 0x7f78fc19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 -- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc102280 msgr2=0x7f78fc19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc102280 0x7f78fc19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 -- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78fc1018c0 msgr2=0x7f78fc19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:02.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78fc1018c0 0x7f78fc19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 -- 172.21.15.69:0/3928890846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78fc19f1f0 con 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7901477700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc102280 0x7f78fc19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7902479700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc106190 0x7f78fc19eab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f78f400ab40 tx=0x7f78f400ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f78f2ffd700 10 client.?.objecter ms_handle_connect 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f78f2ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4018070 con 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f78f400c640 con 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4004270 con 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78fc19f4e0 con 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 10 client.14607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78fc1a6d60 con 0x7f78fc106190 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.879+0000 7f7903edc700 1 librados: init done 2024-04-02T04:46:02.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7903edc700 10 client.14607.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7903edc700 15 client.14607.objecter _session_command_op_assign -1 1 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7903edc700 20 client.14607.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7903edc700 10 client.14607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78fc1028d0 con 0x7f78fc106190 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f78f401b070 con 0x7f78fc106190 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78e806f5e0 0x7f78e8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7f78f408d030 con 0x7f78fc106190 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 10 client.14607.objecter ms_dispatch 0x7f78fc001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 3 client.14607.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 3 client.14607.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7901c78700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78e806f5e0 0x7f78e8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 10 client.14607.objecter checking command 1 2024-04-02T04:46:02.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e8079570 0x7f78e807ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 20 client.14607.objecter _get_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 15 client.14607.objecter _session_command_op_remove -1 1 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 20 client.14607.objecter _get_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 20 client.14607.objecter get_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 15 client.14607.objecter _session_command_op_assign 6 1 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 4 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 10 client.14607.objecter _send_command 1 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78f400eec0 con 0x7f78e8079570 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 20 client.14607.objecter dump_active .. 0 homeless 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f78f4094050 con 0x7f78fc106190 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 10 client.14607.objecter ms_handle_connect 0x7f78e806f5e0 2024-04-02T04:46:02.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7901c78700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78e806f5e0 0x7f78e8071aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005fd0 tx=0x7f78f8009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7901477700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e8079570 0x7f78e807ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f78f2ffd700 10 client.14607.objecter ms_handle_connect 0x7f78e8079570 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.883+0000 7f7901477700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e8079570 0x7f78e807ba10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.887+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78f400eec0 con 0x7f78e8079570 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.887+0000 7f78f2ffd700 10 client.14607.objecter ms_dispatch 0x7f78fc001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.887+0000 7f78f2ffd700 10 client.14607.objecter _finish_command 1 = system:0 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.887+0000 7f78f2ffd700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 4 2024-04-02T04:46:02.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.887+0000 7f78f2ffd700 15 client.14607.objecter _session_command_op_remove 6 1 2024-04-02T04:46:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 10 client.14607.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 15 client.14607.objecter _session_command_op_assign -1 2 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter _get_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter _get_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 15 client.14607.objecter _session_command_op_remove -1 2 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter get_session s=0x7f78e8078ea0 osd=6 3 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 15 client.14607.objecter _session_command_op_assign 6 2 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 4 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 10 client.14607.objecter _send_command 2 2024-04-02T04:46:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78fc04ff00 con 0x7f78e8079570 2024-04-02T04:46:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f78f2ffd700 1 -- 172.21.15.69:0/3928890846 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f78fc04ff00 con 0x7f78e8079570 2024-04-02T04:46:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f78f2ffd700 10 client.14607.objecter ms_dispatch 0x7f78fc001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f78f2ffd700 10 client.14607.objecter _finish_command 2 = system:0 2024-04-02T04:46:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f78f2ffd700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 4 2024-04-02T04:46:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f78f2ffd700 15 client.14607.objecter _session_command_op_remove 6 2 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 10 librados: watch_flush enter 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 10 librados: watch_flush exit 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 10 client.14607.objecter close_session for osd.6 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e8079570 msgr2=0x7f78e807ba10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e8079570 0x7f78e807ba10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter put_session s=0x7f78e8078ea0 osd=6 1 2024-04-02T04:46:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 20 client.14607.objecter shutdown clearing up homeless session... 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.903+0000 7f7903edc700 10 client.14607.objecter successfully canceled tick 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78e806f5e0 msgr2=0x7f78e8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78e806f5e0 0x7f78e8071aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005fd0 tx=0x7f78f8009310 comp rx=0 tx=0).stop 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc106190 msgr2=0x7f78fc19eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 --2- 172.21.15.69:0/3928890846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc106190 0x7f78fc19eab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f78f400ab40 tx=0x7f78f400ae50 comp rx=0 tx=0).stop 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7902479700 1 -- 172.21.15.69:0/3928890846 reap_dead start 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 shutdown_connections 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 >> 172.21.15.69:0/3928890846 conn(0x7f78fc077740 msgr2=0x7f78fc103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 shutdown_connections 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 -- 172.21.15.69:0/3928890846 wait complete. 2024-04-02T04:46:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:02.907+0000 7f7903edc700 1 librados: shutdown 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.452696+0000 osd.0 (osd.0) 51 : cluster 0 6.16 deep-scrub starts 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.454798+0000 osd.0 (osd.0) 52 : cluster 0 6.16 deep-scrub ok 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.648968+0000 osd.3 (osd.3) 61 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.649085+0000 osd.3 (osd.3) 62 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.854406+0000 osd.4 (osd.4) 49 : cluster 0 6.e scrub starts 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.856645+0000 osd.4 (osd.4) 50 : cluster 0 6.e scrub ok 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.452696+0000 osd.0 (osd.0) 51 : cluster 0 6.16 deep-scrub starts 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.454798+0000 osd.0 (osd.0) 52 : cluster 0 6.16 deep-scrub ok 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.648968+0000 osd.3 (osd.3) 61 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.649085+0000 osd.3 (osd.3) 62 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:02.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.854406+0000 osd.4 (osd.4) 49 : cluster 0 6.e scrub starts 2024-04-02T04:46:02.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:02 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.856645+0000 osd.4 (osd.4) 50 : cluster 0 6.e scrub ok 2024-04-02T04:46:03.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=migrator t=2024-04-02T04:46:02.903052894Z level=info msg="migrations completed" performed=485 skipped=0 duration=24.305979986s 2024-04-02T04:46:03.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:02 smithi098 bash[51598]: logger=sqlstore t=2024-04-02T04:46:02.904768351Z level=info msg="Created default organization" 2024-04-02T04:46:03.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[51598]: logger=plugin.finder t=2024-04-02T04:46:03.035335977Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-02T04:46:03.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[51598]: logger=secrets t=2024-04-02T04:46:03.035521993Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-02T04:46:03.224 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[51598]: logger=query_data t=2024-04-02T04:46:03.040121373Z level=info msg="Query Service initialization" 2024-04-02T04:46:03.225 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[51598]: logger=live.push_http t=2024-04-02T04:46:03.046585056Z level=info msg="Live Push Gateway initialization" 2024-04-02T04:46:03.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- 172.21.15.69:0/3237461318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c1044a0 msgr2=0x7ffa6c104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:03.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- 172.21.15.69:0/3237461318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c1044a0 0x7ffa6c104920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009920 tx=0x7ffa60009c30 comp rx=0 tx=0).stop 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- 172.21.15.69:0/3237461318 shutdown_connections 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- 172.21.15.69:0/3237461318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa6c104e60 0x7ffa6c111320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- 172.21.15.69:0/3237461318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa6c10a550 0x7ffa6c10a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- 172.21.15.69:0/3237461318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c1044a0 0x7ffa6c104920 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- 172.21.15.69:0/3237461318 >> 172.21.15.69:0/3237461318 conn(0x7ffa6c0fff70 msgr2=0x7ffa6c102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- 172.21.15.69:0/3237461318 shutdown_connections 2024-04-02T04:46:03.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- 172.21.15.69:0/3237461318 wait complete. 2024-04-02T04:46:03.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 librados: starting msgr at 2024-04-02T04:46:03.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 librados: starting objecter 2024-04-02T04:46:03.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 Processor -- start 2024-04-02T04:46:03.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- start start 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 librados: setting wanted keys 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 librados: calling monclient init 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa6c1044a0 0x7ffa6c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c104e60 0x7ffa6c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa6c10a550 0x7ffa6c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c113c80 con 0x7ffa6c104e60 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c113b00 con 0x7ffa6c1044a0 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.615+0000 7ffa720cf700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c113e00 con 0x7ffa6c10a550 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c104e60 0x7ffa6c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa6c10a550 0x7ffa6c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:03.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c104e60 0x7ffa6c19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38094/0 (socket says 172.21.15.69:38094) 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 -- 172.21.15.69:0/31205009 learned_addr learned my addr 172.21.15.69:0/31205009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6b7fe700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa6c1044a0 0x7ffa6c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 -- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa6c10a550 msgr2=0x7ffa6c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa6c10a550 0x7ffa6c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 -- 172.21.15.69:0/31205009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa6c1044a0 msgr2=0x7ffa6c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa6c1044a0 0x7ffa6c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 -- 172.21.15.69:0/31205009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c19f2f0 con 0x7ffa6c104e60 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6bfff700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa6c10a550 0x7ffa6c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6affd700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c104e60 0x7ffa6c19a820 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009230 tx=0x7ffa6000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.?.objecter ms_handle_connect 0x7ffa6c104e60 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:03.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60027070 con 0x7ffa6c104e60 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ffa6000bb60 con 0x7ffa6c104e60 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c19f580 con 0x7ffa6c104e60 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60004930 con 0x7ffa6c104e60 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 10 client.14613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c1a6e60 con 0x7ffa6c104e60 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 1 librados: init done 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 10 client.14613.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 15 client.14613.objecter _session_command_op_assign -1 1 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 20 client.14613.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 10 client.14613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:03.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa6c105bc0 con 0x7ffa6c104e60 2024-04-02T04:46:03.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ffa6002a070 con 0x7ffa6c104e60 2024-04-02T04:46:03.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa5406f5e0 0x7ffa54071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7ffa6008ec60 con 0x7ffa6c104e60 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.14613.objecter ms_dispatch 0x7ffa6c001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 3 client.14613.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 3 client.14613.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6b7fe700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa5406f5e0 0x7ffa54071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.14613.objecter checking command 1 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ffa540793c0 0x7ffa5407b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 20 client.14613.objecter _get_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 15 client.14613.objecter _session_command_op_remove -1 1 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 20 client.14613.objecter _get_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 20 client.14613.objecter get_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 15 client.14613.objecter _session_command_op_assign 2 1 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 4 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.14613.objecter _send_command 1 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa6000bd80 con 0x7ffa540793c0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 20 client.14613.objecter dump_active .. 0 homeless 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ffa60094070 con 0x7ffa6c104e60 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6bfff700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ffa540793c0 0x7ffa5407b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6b7fe700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa5406f5e0 0x7ffa54071aa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005fd0 tx=0x7ffa5c009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.14613.objecter ms_handle_connect 0x7ffa5406f5e0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa68ff9700 10 client.14613.objecter ms_handle_connect 0x7ffa540793c0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.619+0000 7ffa6bfff700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ffa540793c0 0x7ffa5407b860 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.623+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffa6000bd80 con 0x7ffa540793c0 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.623+0000 7ffa68ff9700 10 client.14613.objecter ms_dispatch 0x7ffa6c001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.623+0000 7ffa68ff9700 10 client.14613.objecter _finish_command 1 = system:0 2024-04-02T04:46:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.623+0000 7ffa68ff9700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 4 2024-04-02T04:46:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.623+0000 7ffa68ff9700 15 client.14613.objecter _session_command_op_remove 2 1 2024-04-02T04:46:03.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 10 client.14613.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:03.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 15 client.14613.objecter _session_command_op_assign -1 2 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter _get_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter _get_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 15 client.14613.objecter _session_command_op_remove -1 2 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter get_session s=0x7ffa54078cf0 osd=2 3 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 15 client.14613.objecter _session_command_op_assign 2 2 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 4 2024-04-02T04:46:03.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 10 client.14613.objecter _send_command 2 2024-04-02T04:46:03.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffa6c04ff00 con 0x7ffa540793c0 2024-04-02T04:46:03.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa68ff9700 1 -- 172.21.15.69:0/31205009 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ffa6c04ff00 con 0x7ffa540793c0 2024-04-02T04:46:03.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa68ff9700 10 client.14613.objecter ms_dispatch 0x7ffa6c001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:03.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa68ff9700 10 client.14613.objecter _finish_command 2 = system:0 2024-04-02T04:46:03.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa68ff9700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 4 2024-04-02T04:46:03.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa68ff9700 15 client.14613.objecter _session_command_op_remove 2 2 2024-04-02T04:46:03.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 10 librados: watch_flush enter 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 10 librados: watch_flush exit 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 10 client.14613.objecter close_session for osd.2 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ffa540793c0 msgr2=0x7ffa5407b860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ffa540793c0 0x7ffa5407b860 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter put_session s=0x7ffa54078cf0 osd=2 1 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 20 client.14613.objecter shutdown clearing up homeless session... 2024-04-02T04:46:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 10 client.14613.objecter successfully canceled tick 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa5406f5e0 msgr2=0x7ffa54071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa5406f5e0 0x7ffa54071aa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005fd0 tx=0x7ffa5c009310 comp rx=0 tx=0).stop 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c104e60 msgr2=0x7ffa6c19a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 --2- 172.21.15.69:0/31205009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa6c104e60 0x7ffa6c19a820 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009230 tx=0x7ffa6000fa70 comp rx=0 tx=0).stop 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa6bfff700 1 -- 172.21.15.69:0/31205009 reap_dead start 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 shutdown_connections 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 >> 172.21.15.69:0/31205009 conn(0x7ffa6c0fff70 msgr2=0x7ffa6c10b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 shutdown_connections 2024-04-02T04:46:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 -- 172.21.15.69:0/31205009 wait complete. 2024-04-02T04:46:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:03.643+0000 7ffa720cf700 1 librados: shutdown 2024-04-02T04:46:03.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[21528]: cluster 2024-04-02T04:46:01.757680+0000 mgr.y (mgr.14538) 77 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-02T04:46:03.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[21528]: cluster 2024-04-02T04:46:02.649938+0000 mon.a (mon.0) 897 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-04-02T04:46:03.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[21528]: cluster 2024-04-02T04:46:02.902097+0000 osd.4 (osd.4) 51 : cluster 0 2.f scrub starts 2024-04-02T04:46:03.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[21528]: cluster 2024-04-02T04:46:02.905864+0000 osd.4 (osd.4) 52 : cluster 0 2.f scrub ok 2024-04-02T04:46:03.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[28904]: cluster 2024-04-02T04:46:01.757680+0000 mgr.y (mgr.14538) 77 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-02T04:46:03.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[28904]: cluster 2024-04-02T04:46:02.649938+0000 mon.a (mon.0) 897 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-04-02T04:46:03.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[28904]: cluster 2024-04-02T04:46:02.902097+0000 osd.4 (osd.4) 51 : cluster 0 2.f scrub starts 2024-04-02T04:46:03.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:03 smithi069 bash[28904]: cluster 2024-04-02T04:46:02.905864+0000 osd.4 (osd.4) 52 : cluster 0 2.f scrub ok 2024-04-02T04:46:03.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[23790]: cluster 2024-04-02T04:46:01.757680+0000 mgr.y (mgr.14538) 77 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-02T04:46:03.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[23790]: cluster 2024-04-02T04:46:02.649938+0000 mon.a (mon.0) 897 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-04-02T04:46:03.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[23790]: cluster 2024-04-02T04:46:02.902097+0000 osd.4 (osd.4) 51 : cluster 0 2.f scrub starts 2024-04-02T04:46:03.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:03 smithi098 bash[23790]: cluster 2024-04-02T04:46:02.905864+0000 osd.4 (osd.4) 52 : cluster 0 2.f scrub ok 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=infra.usagestats.collector t=2024-04-02T04:46:04.051538996Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=provisioning.datasources t=2024-04-02T04:46:04.052666214Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=provisioning.datasources t=2024-04-02T04:46:04.113123922Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=provisioning.alerting t=2024-04-02T04:46:04.163554551Z level=info msg="starting to provision alerting" 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=provisioning.alerting t=2024-04-02T04:46:04.16357511Z level=info msg="finished to provision alerting" 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=grafanaStorageLogger t=2024-04-02T04:46:04.16392801Z level=info msg="storage starting" 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=ngalert.state.manager t=2024-04-02T04:46:04.164922908Z level=info msg="Warming state cache for startup" 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=http.server t=2024-04-02T04:46:04.16687006Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-02T04:46:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=ngalert.state.manager t=2024-04-02T04:46:04.281736244Z level=info msg="State cache has been initialized" states=0 duration=116.809788ms 2024-04-02T04:46:04.475 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=ticker t=2024-04-02T04:46:04.281828975Z level=info msg=starting first_tick=2024-04-02T04:46:10Z 2024-04-02T04:46:04.475 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[51598]: logger=ngalert.multiorg.alertmanager t=2024-04-02T04:46:04.281849324Z level=info msg="starting MultiOrg Alertmanager" 2024-04-02T04:46:04.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:04 smithi069 bash[21528]: audit 2024-04-02T04:46:02.805348+0000 mgr.y (mgr.14538) 78 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:04.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:04 smithi069 bash[28904]: audit 2024-04-02T04:46:02.805348+0000 mgr.y (mgr.14538) 78 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:04.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:04 smithi098 bash[23790]: audit 2024-04-02T04:46:02.805348+0000 mgr.y (mgr.14538) 78 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/3482339928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0108390 msgr2=0x7f85a0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- 172.21.15.69:0/3482339928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0108390 0x7f85a0108770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f859000b0a0 tx=0x7f859000b3b0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/3482339928 shutdown_connections 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- 172.21.15.69:0/3482339928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85a0102ce0 0x7f85a010f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- 172.21.15.69:0/3482339928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0108390 0x7f85a0108770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- 172.21.15.69:0/3482339928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102320 0x7f85a01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/3482339928 >> 172.21.15.69:0/3482339928 conn(0x7f85a0079310 msgr2=0x7f85a0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:05.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/3482339928 shutdown_connections 2024-04-02T04:46:05.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/3482339928 wait complete. 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 librados: starting msgr at 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 librados: starting objecter 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 Processor -- start 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- start start 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 librados: setting wanted keys 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 librados: calling monclient init 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102320 0x7f85a0074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0102ce0 0x7f85a0074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85a0108390 0x7f85a0074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0111a80 con 0x7f85a0108390 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0111900 con 0x7f85a0102320 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f85a0111c00 con 0x7f85a0102ce0 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0102ce0 0x7f85a0074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a62f5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85a0108390 0x7f85a0074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a62f5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85a0108390 0x7f85a0074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38106/0 (socket says 172.21.15.69:38106) 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a62f5700 1 -- 172.21.15.69:0/524471846 learned_addr learned my addr 172.21.15.69:0/524471846 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a5af4700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102320 0x7f85a0074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 -- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102320 msgr2=0x7f85a0074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102320 0x7f85a0074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 -- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85a0108390 msgr2=0x7f85a0074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85a0108390 0x7f85a0074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 -- 172.21.15.69:0/524471846 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a01a6580 con 0x7f85a0102ce0 2024-04-02T04:46:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a5af4700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102320 0x7f85a0074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:46:05.859 INFO:teuthology.orchestra.run.smithi069.stdout:249108103215 2024-04-02T04:46:05.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a52f3700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0102ce0 0x7f85a0074a80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f859400c970 tx=0x7f859400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f859effd700 10 client.?.objecter ms_handle_connect 0x7f85a0102ce0 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f859effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859400ce60 con 0x7f85a0102ce0 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8594004830 con 0x7f85a0102ce0 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a01a6870 con 0x7f85a0102ce0 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85940053a0 con 0x7f85a0102ce0 2024-04-02T04:46:05.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 10 client.24506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a01a6e00 con 0x7f85a0102ce0 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.855+0000 7f85a7d58700 1 librados: init done 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a7d58700 10 client.24506.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a7d58700 15 client.24506.objecter _session_command_op_assign -1 1 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a7d58700 20 client.24506.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a7d58700 10 client.24506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f85a01039b0 con 0x7f85a0102ce0 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8594005540 con 0x7f85a0102ce0 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858806f5e0 0x7f8588071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7f859408de40 con 0x7f85a0102ce0 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 10 client.24506.objecter ms_dispatch 0x7f85a0001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 3 client.24506.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:05.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 3 client.24506.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a5af4700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858806f5e0 0x7f8588071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 10 client.24506.objecter checking command 1 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8588079570 0x7f858807ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 20 client.24506.objecter _get_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 15 client.24506.objecter _session_command_op_remove -1 1 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 20 client.24506.objecter _get_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 20 client.24506.objecter get_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 15 client.24506.objecter _session_command_op_assign 4 1 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 4 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 10 client.24506.objecter _send_command 1 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8594012ec0 con 0x7f8588079570 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 20 client.24506.objecter dump_active .. 0 homeless 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f8594090540 con 0x7f85a0102ce0 2024-04-02T04:46:05.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a5af4700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858806f5e0 0x7f8588071aa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f859000b780 tx=0x7f8590009c60 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 10 client.24506.objecter ms_handle_connect 0x7f858806f5e0 2024-04-02T04:46:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a62f5700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8588079570 0x7f858807ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f859effd700 10 client.24506.objecter ms_handle_connect 0x7f8588079570 2024-04-02T04:46:05.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.859+0000 7f85a62f5700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8588079570 0x7f858807ba10 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:05.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.863+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8594012ec0 con 0x7f8588079570 2024-04-02T04:46:05.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.863+0000 7f859effd700 10 client.24506.objecter ms_dispatch 0x7f85a0001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:05.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.863+0000 7f859effd700 10 client.24506.objecter _finish_command 1 = system:0 2024-04-02T04:46:05.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.863+0000 7f859effd700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 4 2024-04-02T04:46:05.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.863+0000 7f859effd700 15 client.24506.objecter _session_command_op_remove 4 1 2024-04-02T04:46:05.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[28904]: cluster 2024-04-02T04:46:03.758730+0000 mgr.y (mgr.14538) 79 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:05.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[28904]: cluster 2024-04-02T04:46:04.463554+0000 osd.0 (osd.0) 53 : cluster 0 4.1e scrub starts 2024-04-02T04:46:05.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[28904]: cluster 2024-04-02T04:46:04.465952+0000 osd.0 (osd.0) 54 : cluster 0 4.1e scrub ok 2024-04-02T04:46:05.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[28904]: audit 2024-04-02T04:46:04.774115+0000 mon.a (mon.0) 898 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:05.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[28904]: audit 2024-04-02T04:46:04.788003+0000 mon.a (mon.0) 899 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:05.894 INFO:teuthology.orchestra.run.smithi069.stdout:81604378712 2024-04-02T04:46:05.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-02T04:46:05.895 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[21528]: cluster 2024-04-02T04:46:03.758730+0000 mgr.y (mgr.14538) 79 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:05.895 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[21528]: cluster 2024-04-02T04:46:04.463554+0000 osd.0 (osd.0) 53 : cluster 0 4.1e scrub starts 2024-04-02T04:46:05.895 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[21528]: cluster 2024-04-02T04:46:04.465952+0000 osd.0 (osd.0) 54 : cluster 0 4.1e scrub ok 2024-04-02T04:46:05.895 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[21528]: audit 2024-04-02T04:46:04.774115+0000 mon.a (mon.0) 898 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:05.895 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:05 smithi069 bash[21528]: audit 2024-04-02T04:46:04.788003+0000 mon.a (mon.0) 899 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 10 client.24506.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 15 client.24506.objecter _session_command_op_assign -1 2 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 20 client.24506.objecter _get_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 20 client.24506.objecter _get_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 15 client.24506.objecter _session_command_op_remove -1 2 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 20 client.24506.objecter get_session s=0x7f8588078ea0 osd=4 3 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 15 client.24506.objecter _session_command_op_assign 4 2 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 4 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 10 client.24506.objecter _send_command 2 2024-04-02T04:46:05.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f85a004ff00 con 0x7f8588079570 2024-04-02T04:46:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f859effd700 1 -- 172.21.15.69:0/524471846 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f85a004ff00 con 0x7f8588079570 2024-04-02T04:46:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f859effd700 10 client.24506.objecter ms_dispatch 0x7f85a0001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f859effd700 10 client.24506.objecter _finish_command 2 = system:0 2024-04-02T04:46:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f859effd700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 4 2024-04-02T04:46:05.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f859effd700 15 client.24506.objecter _session_command_op_remove 4 2 2024-04-02T04:46:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.895+0000 7f85a7d58700 10 librados: watch_flush enter 2024-04-02T04:46:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 10 librados: watch_flush exit 2024-04-02T04:46:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 10 client.24506.objecter close_session for osd.4 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8588079570 msgr2=0x7f858807ba10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8588079570 0x7f858807ba10 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 20 client.24506.objecter put_session s=0x7f8588078ea0 osd=4 1 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 20 client.24506.objecter shutdown clearing up homeless session... 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 10 client.24506.objecter successfully canceled tick 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858806f5e0 msgr2=0x7f8588071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858806f5e0 0x7f8588071aa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f859000b780 tx=0x7f8590009c60 comp rx=0 tx=0).stop 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0102ce0 msgr2=0x7f85a0074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 --2- 172.21.15.69:0/524471846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85a0102ce0 0x7f85a0074a80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f859400c970 tx=0x7f859400cc80 comp rx=0 tx=0).stop 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a62f5700 1 -- 172.21.15.69:0/524471846 reap_dead start 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 shutdown_connections 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 >> 172.21.15.69:0/524471846 conn(0x7f85a0079310 msgr2=0x7f85a0105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:05.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 shutdown_connections 2024-04-02T04:46:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 -- 172.21.15.69:0/524471846 wait complete. 2024-04-02T04:46:05.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:05.899+0000 7f85a7d58700 1 librados: shutdown 2024-04-02T04:46:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:05 smithi098 bash[23790]: cluster 2024-04-02T04:46:03.758730+0000 mgr.y (mgr.14538) 79 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:05 smithi098 bash[23790]: cluster 2024-04-02T04:46:04.463554+0000 osd.0 (osd.0) 53 : cluster 0 4.1e scrub starts 2024-04-02T04:46:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:05 smithi098 bash[23790]: cluster 2024-04-02T04:46:04.465952+0000 osd.0 (osd.0) 54 : cluster 0 4.1e scrub ok 2024-04-02T04:46:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:05 smithi098 bash[23790]: audit 2024-04-02T04:46:04.774115+0000 mon.a (mon.0) 898 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:05 smithi098 bash[23790]: audit 2024-04-02T04:46:04.788003+0000 mon.a (mon.0) 899 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/3057270883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c107260 msgr2=0x7f067c107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- 172.21.15.69:0/3057270883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c107260 0x7f067c107640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f066c0099c0 tx=0x7f066c009cd0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/3057270883 shutdown_connections 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- 172.21.15.69:0/3057270883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f067c101150 0x7f067c10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- 172.21.15.69:0/3057270883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c107260 0x7f067c107640 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- 172.21.15.69:0/3057270883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f067c100790 0x7f067c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/3057270883 >> 172.21.15.69:0/3057270883 conn(0x7f067c077740 msgr2=0x7f067c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/3057270883 shutdown_connections 2024-04-02T04:46:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/3057270883 wait complete. 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 librados: starting msgr at 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 librados: starting objecter 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 Processor -- start 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- start start 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 librados: setting wanted keys 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 librados: calling monclient init 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c100790 0x7f067c198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f067c101150 0x7f067c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c100790 0x7f067c198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c100790 0x7f067c198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38110/0 (socket says 172.21.15.69:38110) 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f067c107260 0x7f067c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f067c111b80 con 0x7f067c100790 2024-04-02T04:46:06.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 -- 172.21.15.69:0/2165166518 learned_addr learned my addr 172.21.15.69:0/2165166518 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:06.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f067c111a00 con 0x7f067c107260 2024-04-02T04:46:06.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f067c111d00 con 0x7f067c101150 2024-04-02T04:46:06.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067affd700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f067c101150 0x7f067c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 -- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f067c101150 msgr2=0x7f067c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f067c101150 0x7f067c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 -- 172.21.15.69:0/2165166518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f067c107260 msgr2=0x7f067c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f067c107260 0x7f067c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c19d160 con 0x7f067c100790 2024-04-02T04:46:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067affd700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f067c101150 0x7f067c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f067b7fe700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c100790 0x7f067c198150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f066c006010 tx=0x7f066c003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0678ff9700 10 client.?.objecter ms_handle_connect 0x7f067c100790 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0678ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c009e60 con 0x7f067c100790 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f066c00f460 con 0x7f067c100790 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c017650 con 0x7f067c100790 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067c19d3f0 con 0x7f067c100790 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 10 client.14625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067c1a4cd0 con 0x7f067c100790 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.075+0000 7f0681e77700 1 librados: init done 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f06627fc700 10 client.14625.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f06627fc700 15 client.14625.objecter _session_command_op_assign -1 1 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f06627fc700 20 client.14625.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f06627fc700 10 client.14625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f065c002920 con 0x7f067c100790 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f066c00f980 con 0x7f067c100790 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f066406f5e0 0x7f0664071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7f066c08efb0 con 0x7f067c100790 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 10 client.14625.objecter ms_dispatch 0x7f067c001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 3 client.14625.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 3 client.14625.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 10 client.14625.objecter checking command 1 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0664079440 0x7f066407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter _get_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 15 client.14625.objecter _session_command_op_remove -1 1 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter _get_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter get_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 15 client.14625.objecter _session_command_op_assign 0 1 2024-04-02T04:46:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 4 2024-04-02T04:46:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 10 client.14625.objecter _send_command 1 2024-04-02T04:46:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f066c001d30 con 0x7f0664079440 2024-04-02T04:46:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter dump_active .. 0 homeless 2024-04-02T04:46:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f066c08f440 con 0x7f067c100790 2024-04-02T04:46:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f067bfff700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0664079440 0x7f066407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f067bfff700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0664079440 0x7f066407b8e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 10 client.14625.objecter ms_handle_connect 0x7f0664079440 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f066c001d30 con 0x7f0664079440 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 10 client.14625.objecter ms_dispatch 0x7f067c001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 10 client.14625.objecter _finish_command 1 = system:0 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 4 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.079+0000 7f0678ff9700 15 client.14625.objecter _session_command_op_remove 0 1 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.083+0000 7f067affd700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f066406f5e0 0x7f0664071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.083+0000 7f067affd700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f066406f5e0 0x7f0664071aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f067c100030 tx=0x7f0670009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.083+0000 7f0678ff9700 10 client.14625.objecter ms_handle_connect 0x7f066406f5e0 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 10 client.14625.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 15 client.14625.objecter _session_command_op_assign -1 2 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter _get_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter _get_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 15 client.14625.objecter _session_command_op_remove -1 2 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter get_session s=0x7f0664078d70 osd=0 3 2024-04-02T04:46:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 15 client.14625.objecter _session_command_op_assign 0 2 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 4 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 10 client.14625.objecter _send_command 2 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f065c001240 con 0x7f0664079440 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f0678ff9700 1 -- 172.21.15.69:0/2165166518 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f065c001240 con 0x7f0664079440 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f0678ff9700 10 client.14625.objecter ms_dispatch 0x7f067c001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f0678ff9700 10 client.14625.objecter _finish_command 2 = system:0 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f0678ff9700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 4 2024-04-02T04:46:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f0678ff9700 15 client.14625.objecter _session_command_op_remove 0 2 2024-04-02T04:46:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 10 librados: watch_flush enter 2024-04-02T04:46:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 10 librados: watch_flush exit 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 10 client.14625.objecter close_session for osd.0 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0664079440 msgr2=0x7f066407b8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0664079440 0x7f066407b8e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter put_session s=0x7f0664078d70 osd=0 1 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 20 client.14625.objecter shutdown clearing up homeless session... 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 10 client.14625.objecter successfully canceled tick 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f066406f5e0 msgr2=0x7f0664071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f066406f5e0 0x7f0664071aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f067c100030 tx=0x7f0670009190 comp rx=0 tx=0).stop 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c100790 msgr2=0x7f067c198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 --2- 172.21.15.69:0/2165166518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f067c100790 0x7f067c198150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f066c006010 tx=0x7f066c003da0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f067bfff700 1 -- 172.21.15.69:0/2165166518 reap_dead start 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 shutdown_connections 2024-04-02T04:46:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 >> 172.21.15.69:0/2165166518 conn(0x7f067c077740 msgr2=0x7f067c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 shutdown_connections 2024-04-02T04:46:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 -- 172.21.15.69:0/2165166518 wait complete. 2024-04-02T04:46:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.099+0000 7f06627fc700 1 librados: shutdown 2024-04-02T04:46:06.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- 172.21.15.69:0/1004664458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4574102520 msgr2=0x7f4574102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- 172.21.15.69:0/1004664458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4574102520 0x7f4574102900 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45640099c0 tx=0x7f4564009cd0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- 172.21.15.69:0/1004664458 shutdown_connections 2024-04-02T04:46:06.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- 172.21.15.69:0/1004664458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45741075a0 0x7f457410f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- 172.21.15.69:0/1004664458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4574102520 0x7f4574102900 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- 172.21.15.69:0/1004664458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102ed0 0x7f4574106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- 172.21.15.69:0/1004664458 >> 172.21.15.69:0/1004664458 conn(0x7f4574079360 msgr2=0x7f4574079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- 172.21.15.69:0/1004664458 shutdown_connections 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- 172.21.15.69:0/1004664458 wait complete. 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 librados: starting msgr at 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 librados: starting objecter 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 Processor -- start 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- start start 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 librados: setting wanted keys 2024-04-02T04:46:06.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 librados: calling monclient init 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102520 0x7f4574074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4574102ed0 0x7f4574074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45741075a0 0x7f4574074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4574111ad0 con 0x7f4574102520 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4574111950 con 0x7f4574102ed0 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4574111c50 con 0x7f45741075a0 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102520 0x7f4574074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4578e1e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45741075a0 0x7f4574074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102520 0x7f4574074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38136/0 (socket says 172.21.15.69:38136) 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 -- 172.21.15.69:0/2753261030 learned_addr learned my addr 172.21.15.69:0/2753261030 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f45737fe700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4574102ed0 0x7f4574074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 -- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45741075a0 msgr2=0x7f4574074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45741075a0 0x7f4574074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 -- 172.21.15.69:0/2753261030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4574102ed0 msgr2=0x7f4574074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4574102ed0 0x7f4574074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 -- 172.21.15.69:0/2753261030 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45741a6620 con 0x7f4574102520 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4578e1e700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45741075a0 0x7f4574074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f4573fff700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102520 0x7f4574074540 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4564005910 tx=0x7f456400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f45717fa700 10 client.?.objecter ms_handle_connect 0x7f4574102520 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f45717fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564027070 con 0x7f4574102520 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f456400fc90 con 0x7f4574102520 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45741a68b0 con 0x7f4574102520 2024-04-02T04:46:06.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.447+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564022c90 con 0x7f4574102520 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 10 client.14631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45741a6d00 con 0x7f4574102520 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 1 librados: init done 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 10 client.14631.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 15 client.14631.objecter _session_command_op_assign -1 1 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 20 client.14631.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 10 client.14631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4560002920 con 0x7f4574102520 2024-04-02T04:46:06.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f456402a070 con 0x7f4574102520 2024-04-02T04:46:06.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f455c06f5e0 0x7f455c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7f456408ed90 con 0x7f4574102520 2024-04-02T04:46:06.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter ms_dispatch 0x7f4574001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:06.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 3 client.14631.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 3 client.14631.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45737fe700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f455c06f5e0 0x7f455c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter checking command 1 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f455c0794f0 0x7f455c07b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter _get_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 15 client.14631.objecter _session_command_op_remove -1 1 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter _get_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter get_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 15 client.14631.objecter _session_command_op_assign 1 1 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 4 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter _send_command 1 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f456400b0b0 con 0x7f455c0794f0 2024-04-02T04:46:06.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter dump_active .. 0 homeless 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f4564091490 con 0x7f4574102520 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f4578e1e700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f455c0794f0 0x7f455c07b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter ms_handle_connect 0x7f455c06f5e0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45737fe700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f455c06f5e0 0x7f455c071aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4574108b40 tx=0x7f4568009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter ms_handle_connect 0x7f455c0794f0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f4578e1e700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f455c0794f0 0x7f455c07b990 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f456400b0b0 con 0x7f455c0794f0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter ms_dispatch 0x7f4574001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 10 client.14631.objecter _finish_command 1 = system:0 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 4 2024-04-02T04:46:06.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.451+0000 7f45717fa700 15 client.14631.objecter _session_command_op_remove 1 1 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 10 client.14631.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 15 client.14631.objecter _session_command_op_assign -1 2 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter _get_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter _get_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 15 client.14631.objecter _session_command_op_remove -1 2 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter get_session s=0x7f455c078e20 osd=1 3 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 15 client.14631.objecter _session_command_op_assign 1 2 2024-04-02T04:46:06.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 4 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 10 client.14631.objecter _send_command 2 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4560001220 con 0x7f455c0794f0 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f45717fa700 1 -- 172.21.15.69:0/2753261030 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4560001220 con 0x7f455c0794f0 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f45717fa700 10 client.14631.objecter ms_dispatch 0x7f4574001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f45717fa700 10 client.14631.objecter _finish_command 2 = system:0 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f45717fa700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 4 2024-04-02T04:46:06.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f45717fa700 15 client.14631.objecter _session_command_op_remove 1 2 2024-04-02T04:46:06.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 10 librados: watch_flush enter 2024-04-02T04:46:06.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 10 librados: watch_flush exit 2024-04-02T04:46:06.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 10 client.14631.objecter close_session for osd.1 2024-04-02T04:46:06.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f455c0794f0 msgr2=0x7f455c07b990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f455c0794f0 0x7f455c07b990 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter put_session s=0x7f455c078e20 osd=1 1 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 20 client.14631.objecter shutdown clearing up homeless session... 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 10 client.14631.objecter successfully canceled tick 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f455c06f5e0 msgr2=0x7f455c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f455c06f5e0 0x7f455c071aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4574108b40 tx=0x7f4568009260 comp rx=0 tx=0).stop 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102520 msgr2=0x7f4574074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 --2- 172.21.15.69:0/2753261030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4574102520 0x7f4574074540 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4564005910 tx=0x7f456400fa00 comp rx=0 tx=0).stop 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f4578e1e700 1 -- 172.21.15.69:0/2753261030 reap_dead start 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 shutdown_connections 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 >> 172.21.15.69:0/2753261030 conn(0x7f4574079360 msgr2=0x7f4574100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 shutdown_connections 2024-04-02T04:46:06.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 -- 172.21.15.69:0/2753261030 wait complete. 2024-04-02T04:46:06.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.471+0000 7f457a881700 1 librados: shutdown 2024-04-02T04:46:06.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- 172.21.15.69:0/811781431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c101dd0 msgr2=0x7f4d4c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- 172.21.15.69:0/811781431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c101dd0 0x7f4d4c105e20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009920 tx=0x7f4d40009c30 comp rx=0 tx=0).stop 2024-04-02T04:46:06.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- 172.21.15.69:0/811781431 shutdown_connections 2024-04-02T04:46:06.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- 172.21.15.69:0/811781431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d4c106500 0x7f4d4c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- 172.21.15.69:0/811781431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 0x7f4d4c101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- 172.21.15.69:0/811781431 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c101dd0 0x7f4d4c105e20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- 172.21.15.69:0/811781431 >> 172.21.15.69:0/811781431 conn(0x7f4d4c078240 msgr2=0x7f4d4c078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:06.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- 172.21.15.69:0/811781431 shutdown_connections 2024-04-02T04:46:06.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- 172.21.15.69:0/811781431 wait complete. 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 librados: starting msgr at 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 librados: starting objecter 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 Processor -- start 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- start start 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 librados: setting wanted keys 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 librados: calling monclient init 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 0x7f4d4c196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d4c101dd0 0x7f4d4c197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c106500 0x7f4d4c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c1109c0 con 0x7f4d4c101420 2024-04-02T04:46:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c110840 con 0x7f4d4c101dd0 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d5091e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4c110b40 con 0x7f4d4c106500 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d4ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c106500 0x7f4d4c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d4a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 0x7f4d4c196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d4a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 0x7f4d4c196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38166/0 (socket says 172.21.15.69:38166) 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.835+0000 7f4d4a59c700 1 -- 172.21.15.69:0/4224303953 learned_addr learned my addr 172.21.15.69:0/4224303953 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d49d9b700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d4c101dd0 0x7f4d4c197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4a59c700 1 -- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c106500 msgr2=0x7f4d4c1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4a59c700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c106500 0x7f4d4c1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4a59c700 1 -- 172.21.15.69:0/4224303953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d4c101dd0 msgr2=0x7f4d4c197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4a59c700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d4c101dd0 0x7f4d4c197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4a59c700 1 -- 172.21.15.69:0/4224303953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c1a6590 con 0x7f4d4c101420 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4ad9d700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d4c106500 0x7f4d4c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d49d9b700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d4c101dd0 0x7f4d4c197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4a59c700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 0x7f4d4c196d20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00ec10 tx=0x7f4d3c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.?.objecter ms_handle_connect 0x7f4d4c101420 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00cd40 con 0x7f4d4c101420 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4c1a6880 con 0x7f4d4c101420 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4d3c0044a0 con 0x7f4d4c101420 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c010430 con 0x7f4d4c101420 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 10 client.14637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4c1a6dc0 con 0x7f4d4c101420 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 1 librados: init done 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 10 client.14637.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 15 client.14637.objecter _session_command_op_assign -1 1 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 20 client.14637.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:06.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 10 client.14637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:06.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d4c102900 con 0x7f4d4c101420 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4d3c004020 con 0x7f4d4c101420 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d3406f5e0 0x7f4d34071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7f4d3c08e960 con 0x7f4d4c101420 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.14637.objecter ms_dispatch 0x7f4d4c001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 3 client.14637.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 3 client.14637.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d49d9b700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d3406f5e0 0x7f4d34071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.14637.objecter checking command 1 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4d340794f0 0x7f4d3407b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 20 client.14637.objecter _get_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 15 client.14637.objecter _session_command_op_remove -1 1 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 20 client.14637.objecter _get_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 20 client.14637.objecter get_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 15 client.14637.objecter _session_command_op_assign 3 1 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 4 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.14637.objecter _send_command 1 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d3c022e80 con 0x7f4d340794f0 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 20 client.14637.objecter dump_active .. 0 homeless 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f4d3c08edf0 con 0x7f4d4c101420 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4ad9d700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4d340794f0 0x7f4d3407b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d49d9b700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d3406f5e0 0x7f4d34071aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009230 tx=0x7f4d40005b40 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.14637.objecter ms_handle_connect 0x7f4d3406f5e0 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d4ad9d700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4d340794f0 0x7f4d3407b990 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.839+0000 7f4d3b7fe700 10 client.14637.objecter ms_handle_connect 0x7f4d340794f0 2024-04-02T04:46:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.843+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d3c022e80 con 0x7f4d340794f0 2024-04-02T04:46:06.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.843+0000 7f4d3b7fe700 10 client.14637.objecter ms_dispatch 0x7f4d4c001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:06.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.843+0000 7f4d3b7fe700 10 client.14637.objecter _finish_command 1 = system:0 2024-04-02T04:46:06.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.843+0000 7f4d3b7fe700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 4 2024-04-02T04:46:06.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.843+0000 7f4d3b7fe700 15 client.14637.objecter _session_command_op_remove 3 1 2024-04-02T04:46:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 10 client.14637.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 15 client.14637.objecter _session_command_op_assign -1 2 2024-04-02T04:46:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter _get_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter _get_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 15 client.14637.objecter _session_command_op_remove -1 2 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter get_session s=0x7f4d34078e20 osd=3 3 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 15 client.14637.objecter _session_command_op_assign 3 2 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 4 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 10 client.14637.objecter _send_command 2 2024-04-02T04:46:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d4c04ff00 con 0x7f4d340794f0 2024-04-02T04:46:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d3b7fe700 1 -- 172.21.15.69:0/4224303953 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4d4c04ff00 con 0x7f4d340794f0 2024-04-02T04:46:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d3b7fe700 10 client.14637.objecter ms_dispatch 0x7f4d4c001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d3b7fe700 10 client.14637.objecter _finish_command 2 = system:0 2024-04-02T04:46:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d3b7fe700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 4 2024-04-02T04:46:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d3b7fe700 15 client.14637.objecter _session_command_op_remove 3 2 2024-04-02T04:46:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 10 librados: watch_flush enter 2024-04-02T04:46:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 10 librados: watch_flush exit 2024-04-02T04:46:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 10 client.14637.objecter close_session for osd.3 2024-04-02T04:46:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4d340794f0 msgr2=0x7f4d3407b990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4d340794f0 0x7f4d3407b990 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter put_session s=0x7f4d34078e20 osd=3 1 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 20 client.14637.objecter shutdown clearing up homeless session... 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 10 client.14637.objecter successfully canceled tick 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d3406f5e0 msgr2=0x7f4d34071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d3406f5e0 0x7f4d34071aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009230 tx=0x7f4d40005b40 comp rx=0 tx=0).stop 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 msgr2=0x7f4d4c196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 --2- 172.21.15.69:0/4224303953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d4c101420 0x7f4d4c196d20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00ec10 tx=0x7f4d3c00ef20 comp rx=0 tx=0).stop 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d4ad9d700 1 -- 172.21.15.69:0/4224303953 reap_dead start 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 shutdown_connections 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 >> 172.21.15.69:0/4224303953 conn(0x7f4d4c078240 msgr2=0x7f4d4c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 shutdown_connections 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 -- 172.21.15.69:0/4224303953 wait complete. 2024-04-02T04:46:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:06.859+0000 7f4d5091e700 1 librados: shutdown 2024-04-02T04:46:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 -- 172.21.15.69:0/536204834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0101460 msgr2=0x7fbee01018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 --2- 172.21.15.69:0/536204834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0101460 0x7fbee01018e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbed00099c0 tx=0x7fbed0009cd0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 -- 172.21.15.69:0/536204834 shutdown_connections 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 --2- 172.21.15.69:0/536204834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbee0101e20 0x7fbee010dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 --2- 172.21.15.69:0/536204834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0107350 0x7fbee0107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 --2- 172.21.15.69:0/536204834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0101460 0x7fbee01018e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 -- 172.21.15.69:0/536204834 >> 172.21.15.69:0/536204834 conn(0x7fbee0077ea0 msgr2=0x7fbee00782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 -- 172.21.15.69:0/536204834 shutdown_connections 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.219+0000 7fbee8656700 1 -- 172.21.15.69:0/536204834 wait complete. 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 librados: starting msgr at 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 librados: starting objecter 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 Processor -- start 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- start start 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 librados: setting wanted keys 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 librados: calling monclient init 2024-04-02T04:46:07.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbee0101460 0x7fbee019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0101e20 0x7fbee019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0107350 0x7fbee019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0110a60 con 0x7fbee0107350 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbee01108e0 con 0x7fbee0101460 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbee0110be0 con 0x7fbee0101e20 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0101e20 0x7fbee019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0101e20 0x7fbee019a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:51654/0 (socket says 172.21.15.69:51654) 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 -- 172.21.15.69:0/3391043538 learned_addr learned my addr 172.21.15.69:0/3391043538 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:07.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee6bf3700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0107350 0x7fbee019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee63f2700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbee0101460 0x7fbee019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 -- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbee0101460 msgr2=0x7fbee019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbee0101460 0x7fbee019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 -- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0107350 msgr2=0x7fbee019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbee0107350 0x7fbee019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 -- 172.21.15.69:0/3391043538 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee019f280 con 0x7fbee0101e20 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee63f2700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbee0101460 0x7fbee019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee5bf1700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0101e20 0x7fbee019a7b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbed0005910 tx=0x7fbed000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 10 client.?.objecter ms_handle_connect 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0027070 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fbed000fc60 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee019f510 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0022d40 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 10 client.24530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01a6df0 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 librados: init done 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 10 client.24530.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 15 client.24530.objecter _session_command_op_assign -1 1 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 20 client.24530.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 10 client.24530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbee01028d0 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fbed002a070 con 0x7fbee0101e20 2024-04-02T04:46:07.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbecc06f5e0 0x7fbecc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7fbed008e9f0 con 0x7fbee0101e20 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 10 client.24530.objecter ms_dispatch 0x7fbee0001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 3 client.24530.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbed77fe700 3 client.24530.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.223+0000 7fbee63f2700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbecc06f5e0 0x7fbecc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 10 client.24530.objecter checking command 1 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbecc0794f0 0x7fbecc07b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter _get_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 15 client.24530.objecter _session_command_op_remove -1 1 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter _get_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter get_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 15 client.24530.objecter _session_command_op_assign 7 1 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 4 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 10 client.24530.objecter _send_command 1 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbecc07c060 con 0x7fbecc0794f0 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter dump_active .. 0 homeless 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7fbed00932c0 con 0x7fbee0101e20 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 10 client.24530.objecter ms_handle_connect 0x7fbecc06f5e0 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbee63f2700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbecc06f5e0 0x7fbecc071aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009be0 tx=0x7fbedc009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:07.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbee6bf3700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbecc0794f0 0x7fbecc07b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbee6bf3700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbecc0794f0 0x7fbecc07b990 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 10 client.24530.objecter ms_handle_connect 0x7fbecc0794f0 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbecc07c060 con 0x7fbecc0794f0 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 10 client.24530.objecter ms_dispatch 0x7fbee0001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 10 client.24530.objecter _finish_command 1 = system:0 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 4 2024-04-02T04:46:07.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.227+0000 7fbed77fe700 15 client.24530.objecter _session_command_op_remove 7 1 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 10 client.24530.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 15 client.24530.objecter _session_command_op_assign -1 2 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 20 client.24530.objecter _get_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 20 client.24530.objecter _get_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 15 client.24530.objecter _session_command_op_remove -1 2 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 20 client.24530.objecter get_session s=0x7fbecc078e20 osd=7 3 2024-04-02T04:46:07.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 15 client.24530.objecter _session_command_op_assign 7 2 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 4 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 10 client.24530.objecter _send_command 2 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.247+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbee004ff00 con 0x7fbecc0794f0 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbed77fe700 1 -- 172.21.15.69:0/3391043538 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbee004ff00 con 0x7fbecc0794f0 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbed77fe700 10 client.24530.objecter ms_dispatch 0x7fbee0001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbed77fe700 10 client.24530.objecter _finish_command 2 = system:0 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbed77fe700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 4 2024-04-02T04:46:07.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbed77fe700 15 client.24530.objecter _session_command_op_remove 7 2 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 10 librados: watch_flush enter 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 10 librados: watch_flush exit 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 10 client.24530.objecter close_session for osd.7 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbecc0794f0 msgr2=0x7fbecc07b990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbecc0794f0 0x7fbecc07b990 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 20 client.24530.objecter put_session s=0x7fbecc078e20 osd=7 1 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 20 client.24530.objecter shutdown clearing up homeless session... 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 10 client.24530.objecter successfully canceled tick 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbecc06f5e0 msgr2=0x7fbecc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbecc06f5e0 0x7fbecc071aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009be0 tx=0x7fbedc009260 comp rx=0 tx=0).stop 2024-04-02T04:46:07.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0101e20 msgr2=0x7fbee019a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 --2- 172.21.15.69:0/3391043538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbee0101e20 0x7fbee019a7b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbed0005910 tx=0x7fbed000fa00 comp rx=0 tx=0).stop 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee6bf3700 1 -- 172.21.15.69:0/3391043538 reap_dead start 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 shutdown_connections 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 >> 172.21.15.69:0/3391043538 conn(0x7fbee0077ea0 msgr2=0x7fbee006a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 shutdown_connections 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 -- 172.21.15.69:0/3391043538 wait complete. 2024-04-02T04:46:07.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.251+0000 7fbee8656700 1 librados: shutdown 2024-04-02T04:46:07.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/3311352757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c107220 msgr2=0x7f7a0c107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- 172.21.15.69:0/3311352757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c107220 0x7f7a0c107600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009990 tx=0x7f7a08009ca0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/3311352757 shutdown_connections 2024-04-02T04:46:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- 172.21.15.69:0/3311352757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a0c101b60 0x7f7a0c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- 172.21.15.69:0/3311352757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c107220 0x7f7a0c107600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- 172.21.15.69:0/3311352757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a0c1011a0 0x7f7a0c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/3311352757 >> 172.21.15.69:0/3311352757 conn(0x7f7a0c0771e0 msgr2=0x7f7a0c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/3311352757 shutdown_connections 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/3311352757 wait complete. 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 librados: starting msgr at 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 librados: starting objecter 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 Processor -- start 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- start start 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 librados: setting wanted keys 2024-04-02T04:46:07.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 librados: calling monclient init 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c1011a0 0x7f7a0c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a0c101b60 0x7f7a0c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a0c107220 0x7f7a0c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c110970 con 0x7f7a0c1011a0 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c1107f0 con 0x7f7a0c101b60 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7a0c110af0 con 0x7f7a0c107220 2024-04-02T04:46:07.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c1011a0 0x7f7a0c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c1011a0 0x7f7a0c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40026/0 (socket says 172.21.15.69:40026) 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 -- 172.21.15.69:0/592538122 learned_addr learned my addr 172.21.15.69:0/592538122 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a12681700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a0c107220 0x7f7a0c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a1167f700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a0c101b60 0x7f7a0c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 -- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a0c107220 msgr2=0x7f7a0c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a0c107220 0x7f7a0c19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 -- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a0c101b60 msgr2=0x7f7a0c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a0c101b60 0x7f7a0c198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 -- 172.21.15.69:0/592538122 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0c19d050 con 0x7f7a0c1011a0 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a11e80700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c1011a0 0x7f7a0c198040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a08000c00 tx=0x7f7a080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a02ffd700 10 client.?.objecter ms_handle_connect 0x7f7a0c1011a0 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a02ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08027070 con 0x7f7a0c1011a0 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a0c19d2e0 con 0x7f7a0c1011a0 2024-04-02T04:46:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7a08022470 con 0x7f7a0c1011a0 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08017650 con 0x7f7a0c1011a0 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a0c1a4bc0 con 0x7f7a0c1011a0 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 librados: init done 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 10 client.14649.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 15 client.14649.objecter _session_command_op_assign -1 1 2024-04-02T04:46:07.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 20 client.14649.objecter _get_session osd=-1 returning homeless 2024-04-02T04:46:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:07.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.395+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7a0c102860 con 0x7f7a0c1011a0 2024-04-02T04:46:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7a0802a070 con 0x7f7a0c1011a0 2024-04-02T04:46:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79f806f5e0 0x7f79f8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 54..110) ==== 7537+0+0 (secure 0 0 0) 0x7f7a0808f920 con 0x7f7a0c1011a0 2024-04-02T04:46:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 10 client.14649.objecter ms_dispatch 0x7f7a0c001550 osd_map(110..110 src has 54..110) 2024-04-02T04:46:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a1167f700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79f806f5e0 0x7f79f8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 3 client.14649.objecter handle_osd_map got epochs [110,110] > 0 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 3 client.14649.objecter handle_osd_map decoding full epoch 110 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 10 client.14649.objecter checking command 1 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f79f8079570 0x7f79f807ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 20 client.14649.objecter _get_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 15 client.14649.objecter _session_command_op_remove -1 1 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 20 client.14649.objecter _get_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 20 client.14649.objecter get_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 15 client.14649.objecter _session_command_op_assign 5 1 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 4 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 10 client.14649.objecter _send_command 1 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a080095d0 con 0x7f79f8079570 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 20 client.14649.objecter dump_active .. 0 homeless 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f7a0808fdb0 con 0x7f7a0c1011a0 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a1167f700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79f806f5e0 0x7f79f8071aa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c077900 tx=0x7f79fc009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:07.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 10 client.14649.objecter ms_handle_connect 0x7f79f806f5e0 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a12681700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f79f8079570 0x7f79f807ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a12681700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f79f8079570 0x7f79f807ba10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.399+0000 7f7a02ffd700 10 client.14649.objecter ms_handle_connect 0x7f79f8079570 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.403+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7a080095d0 con 0x7f79f8079570 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.403+0000 7f7a02ffd700 10 client.14649.objecter ms_dispatch 0x7f7a0c001550 command_reply(tid 1: 0 ) 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.403+0000 7f7a02ffd700 10 client.14649.objecter _finish_command 1 = system:0 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.403+0000 7f7a02ffd700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 4 2024-04-02T04:46:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.403+0000 7f7a02ffd700 15 client.14649.objecter _session_command_op_remove 5 1 2024-04-02T04:46:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 10 client.14649.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T04:46:07.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 15 client.14649.objecter _session_command_op_assign -1 2 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter _get_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter _get_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 15 client.14649.objecter _session_command_op_remove -1 2 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter get_session s=0x7f79f8078ea0 osd=5 3 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 15 client.14649.objecter _session_command_op_assign 5 2 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 4 2024-04-02T04:46:07.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 10 client.14649.objecter _send_command 2 2024-04-02T04:46:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7a0c04ff00 con 0x7f79f8079570 2024-04-02T04:46:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a02ffd700 1 -- 172.21.15.69:0/592538122 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7a0c04ff00 con 0x7f79f8079570 2024-04-02T04:46:07.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a02ffd700 10 client.14649.objecter ms_dispatch 0x7f7a0c001550 command_reply(tid 2: 0 ) 2024-04-02T04:46:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a02ffd700 10 client.14649.objecter _finish_command 2 = system:0 2024-04-02T04:46:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a02ffd700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 4 2024-04-02T04:46:07.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a02ffd700 15 client.14649.objecter _session_command_op_remove 5 2 2024-04-02T04:46:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 10 librados: watch_flush enter 2024-04-02T04:46:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 10 librados: watch_flush exit 2024-04-02T04:46:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 10 client.14649.objecter close_session for osd.5 2024-04-02T04:46:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f79f8079570 msgr2=0x7f79f807ba10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f79f8079570 0x7f79f807ba10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter put_session s=0x7f79f8078ea0 osd=5 1 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 20 client.14649.objecter shutdown clearing up homeless session... 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 10 client.14649.objecter successfully canceled tick 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79f806f5e0 msgr2=0x7f79f8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79f806f5e0 0x7f79f8071aa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c077900 tx=0x7f79fc009260 comp rx=0 tx=0).stop 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c1011a0 msgr2=0x7f7a0c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 --2- 172.21.15.69:0/592538122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a0c1011a0 0x7f7a0c198040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a08000c00 tx=0x7f7a080047b0 comp rx=0 tx=0).stop 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a12681700 1 -- 172.21.15.69:0/592538122 reap_dead start 2024-04-02T04:46:07.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 shutdown_connections 2024-04-02T04:46:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 >> 172.21.15.69:0/592538122 conn(0x7f7a0c0771e0 msgr2=0x7f7a0c069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 shutdown_connections 2024-04-02T04:46:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 -- 172.21.15.69:0/592538122 wait complete. 2024-04-02T04:46:07.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:07.423+0000 7f7a140e4700 1 librados: shutdown 2024-04-02T04:46:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: cluster 2024-04-02T04:46:05.759254+0000 mgr.y (mgr.14538) 80 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:06.891477+0000 mon.a (mon.0) 900 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:06.903124+0000 mon.a (mon.0) 901 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:06.952278+0000 mon.b (mon.1) 82 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: cluster 2024-04-02T04:46:07.377560+0000 osd.2 (osd.2) 27 : cluster 0 2.1 scrub starts 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: cluster 2024-04-02T04:46:07.378515+0000 osd.2 (osd.2) 28 : cluster 0 2.1 scrub ok 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.477136+0000 mon.b (mon.1) 83 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.478653+0000 mon.b (mon.1) 84 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.490015+0000 mon.a (mon.0) 902 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.548352+0000 mon.b (mon.1) 85 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.550950+0000 mon.b (mon.1) 86 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.552198+0000 mon.b (mon.1) 87 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:07.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:07 smithi098 bash[23790]: audit 2024-04-02T04:46:07.563469+0000 mon.a (mon.0) 903 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: cluster 2024-04-02T04:46:05.759254+0000 mgr.y (mgr.14538) 80 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:06.891477+0000 mon.a (mon.0) 900 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:06.903124+0000 mon.a (mon.0) 901 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:06.952278+0000 mon.b (mon.1) 82 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: cluster 2024-04-02T04:46:07.377560+0000 osd.2 (osd.2) 27 : cluster 0 2.1 scrub starts 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: cluster 2024-04-02T04:46:07.378515+0000 osd.2 (osd.2) 28 : cluster 0 2.1 scrub ok 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.477136+0000 mon.b (mon.1) 83 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.478653+0000 mon.b (mon.1) 84 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.490015+0000 mon.a (mon.0) 902 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.548352+0000 mon.b (mon.1) 85 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:08.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.550950+0000 mon.b (mon.1) 86 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.552198+0000 mon.b (mon.1) 87 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[21528]: audit 2024-04-02T04:46:07.563469+0000 mon.a (mon.0) 903 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: cluster 2024-04-02T04:46:05.759254+0000 mgr.y (mgr.14538) 80 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:06.891477+0000 mon.a (mon.0) 900 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:06.903124+0000 mon.a (mon.0) 901 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:06.952278+0000 mon.b (mon.1) 82 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: cluster 2024-04-02T04:46:07.377560+0000 osd.2 (osd.2) 27 : cluster 0 2.1 scrub starts 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: cluster 2024-04-02T04:46:07.378515+0000 osd.2 (osd.2) 28 : cluster 0 2.1 scrub ok 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.477136+0000 mon.b (mon.1) 83 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.478653+0000 mon.b (mon.1) 84 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:08.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.490015+0000 mon.a (mon.0) 902 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.548352+0000 mon.b (mon.1) 85 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:08.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.550950+0000 mon.b (mon.1) 86 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:08.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.552198+0000 mon.b (mon.1) 87 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:08.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:07 smithi069 bash[28904]: audit 2024-04-02T04:46:07.563469+0000 mon.a (mon.0) 903 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:46:08.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:46:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:46:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:46:08.944 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:08 smithi069 bash[28904]: cluster 2024-04-02T04:46:07.760151+0000 mgr.y (mgr.14538) 81 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:08.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:08 smithi069 bash[21528]: cluster 2024-04-02T04:46:07.760151+0000 mgr.y (mgr.14538) 81 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:08.945 INFO:teuthology.orchestra.run.smithi069.stdout:154618822724 2024-04-02T04:46:08.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-02T04:46:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:08 smithi098 bash[23790]: cluster 2024-04-02T04:46:07.760151+0000 mgr.y (mgr.14538) 81 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:08.979 INFO:teuthology.orchestra.run.smithi069.stdout:34359738476 2024-04-02T04:46:08.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-02T04:46:08.987 INFO:teuthology.orchestra.run.smithi069.stdout:124554051662 2024-04-02T04:46:08.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-02T04:46:08.993 INFO:teuthology.orchestra.run.smithi069.stdout:55834574946 2024-04-02T04:46:08.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-02T04:46:09.856 INFO:teuthology.orchestra.run.smithi069.stdout:201863462968 2024-04-02T04:46:09.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-02T04:46:09.869 INFO:teuthology.orchestra.run.smithi069.stdout:296352743460 2024-04-02T04:46:09.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-02T04:46:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:10 smithi098 bash[23790]: cluster 2024-04-02T04:46:09.357595+0000 osd.1 (osd.1) 51 : cluster 0 2.a scrub starts 2024-04-02T04:46:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:10 smithi098 bash[23790]: cluster 2024-04-02T04:46:09.359729+0000 osd.1 (osd.1) 52 : cluster 0 2.a scrub ok 2024-04-02T04:46:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:10 smithi098 bash[23790]: cluster 2024-04-02T04:46:09.760752+0000 mgr.y (mgr.14538) 82 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:46:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:10 smithi098 bash[23790]: cluster 2024-04-02T04:46:10.055427+0000 osd.6 (osd.6) 31 : cluster 0 4.5 scrub starts 2024-04-02T04:46:11.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:10 smithi098 bash[23790]: cluster 2024-04-02T04:46:10.057866+0000 osd.6 (osd.6) 32 : cluster 0 4.5 scrub ok 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[21528]: cluster 2024-04-02T04:46:09.357595+0000 osd.1 (osd.1) 51 : cluster 0 2.a scrub starts 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[21528]: cluster 2024-04-02T04:46:09.359729+0000 osd.1 (osd.1) 52 : cluster 0 2.a scrub ok 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[21528]: cluster 2024-04-02T04:46:09.760752+0000 mgr.y (mgr.14538) 82 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[21528]: cluster 2024-04-02T04:46:10.055427+0000 osd.6 (osd.6) 31 : cluster 0 4.5 scrub starts 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[21528]: cluster 2024-04-02T04:46:10.057866+0000 osd.6 (osd.6) 32 : cluster 0 4.5 scrub ok 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[28904]: cluster 2024-04-02T04:46:09.357595+0000 osd.1 (osd.1) 51 : cluster 0 2.a scrub starts 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[28904]: cluster 2024-04-02T04:46:09.359729+0000 osd.1 (osd.1) 52 : cluster 0 2.a scrub ok 2024-04-02T04:46:11.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[28904]: cluster 2024-04-02T04:46:09.760752+0000 mgr.y (mgr.14538) 82 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:46:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[28904]: cluster 2024-04-02T04:46:10.055427+0000 osd.6 (osd.6) 31 : cluster 0 4.5 scrub starts 2024-04-02T04:46:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:10 smithi069 bash[28904]: cluster 2024-04-02T04:46:10.057866+0000 osd.6 (osd.6) 32 : cluster 0 4.5 scrub ok 2024-04-02T04:46:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:11 smithi098 bash[23790]: cluster 2024-04-02T04:46:10.722072+0000 osd.3 (osd.3) 63 : cluster 0 4.10 scrub starts 2024-04-02T04:46:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:11 smithi098 bash[23790]: cluster 2024-04-02T04:46:10.724353+0000 osd.3 (osd.3) 64 : cluster 0 4.10 scrub ok 2024-04-02T04:46:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:11 smithi098 bash[23790]: cluster 2024-04-02T04:46:10.892966+0000 osd.4 (osd.4) 53 : cluster 0 4.13 scrub starts 2024-04-02T04:46:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:11 smithi098 bash[23790]: cluster 2024-04-02T04:46:10.895569+0000 osd.4 (osd.4) 54 : cluster 0 4.13 scrub ok 2024-04-02T04:46:12.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[21528]: cluster 2024-04-02T04:46:10.722072+0000 osd.3 (osd.3) 63 : cluster 0 4.10 scrub starts 2024-04-02T04:46:12.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[21528]: cluster 2024-04-02T04:46:10.724353+0000 osd.3 (osd.3) 64 : cluster 0 4.10 scrub ok 2024-04-02T04:46:12.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[21528]: cluster 2024-04-02T04:46:10.892966+0000 osd.4 (osd.4) 53 : cluster 0 4.13 scrub starts 2024-04-02T04:46:12.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[21528]: cluster 2024-04-02T04:46:10.895569+0000 osd.4 (osd.4) 54 : cluster 0 4.13 scrub ok 2024-04-02T04:46:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[28904]: cluster 2024-04-02T04:46:10.722072+0000 osd.3 (osd.3) 63 : cluster 0 4.10 scrub starts 2024-04-02T04:46:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[28904]: cluster 2024-04-02T04:46:10.724353+0000 osd.3 (osd.3) 64 : cluster 0 4.10 scrub ok 2024-04-02T04:46:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[28904]: cluster 2024-04-02T04:46:10.892966+0000 osd.4 (osd.4) 53 : cluster 0 4.13 scrub starts 2024-04-02T04:46:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:11 smithi069 bash[28904]: cluster 2024-04-02T04:46:10.895569+0000 osd.4 (osd.4) 54 : cluster 0 4.13 scrub ok 2024-04-02T04:46:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:13.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[21528]: cluster 2024-04-02T04:46:11.721447+0000 osd.6 (osd.6) 33 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:13.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[21528]: cluster 2024-04-02T04:46:11.721570+0000 osd.6 (osd.6) 34 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[21528]: cluster 2024-04-02T04:46:11.761317+0000 mgr.y (mgr.14538) 83 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[21528]: cluster 2024-04-02T04:46:11.915226+0000 mon.a (mon.0) 904 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[21528]: cluster 2024-04-02T04:46:12.518835+0000 osd.2 (osd.2) 29 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[21528]: cluster 2024-04-02T04:46:12.518954+0000 osd.2 (osd.2) 30 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[28904]: cluster 2024-04-02T04:46:11.721447+0000 osd.6 (osd.6) 33 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[28904]: cluster 2024-04-02T04:46:11.721570+0000 osd.6 (osd.6) 34 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[28904]: cluster 2024-04-02T04:46:11.761317+0000 mgr.y (mgr.14538) 83 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[28904]: cluster 2024-04-02T04:46:11.915226+0000 mon.a (mon.0) 904 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[28904]: cluster 2024-04-02T04:46:12.518835+0000 osd.2 (osd.2) 29 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:12 smithi069 bash[28904]: cluster 2024-04-02T04:46:12.518954+0000 osd.2 (osd.2) 30 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:13.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:46:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[23790]: cluster 2024-04-02T04:46:11.721447+0000 osd.6 (osd.6) 33 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[23790]: cluster 2024-04-02T04:46:11.721570+0000 osd.6 (osd.6) 34 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[23790]: cluster 2024-04-02T04:46:11.761317+0000 mgr.y (mgr.14538) 83 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[23790]: cluster 2024-04-02T04:46:11.915226+0000 mon.a (mon.0) 904 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-04-02T04:46:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[23790]: cluster 2024-04-02T04:46:12.518835+0000 osd.2 (osd.2) 29 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:12 smithi098 bash[23790]: cluster 2024-04-02T04:46:12.518954+0000 osd.2 (osd.2) 30 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:13.716 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:13.716 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:13.717 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:13.720 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:13.720 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:13.722 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:14.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:13 smithi069 bash[21528]: audit 2024-04-02T04:46:12.812119+0000 mgr.y (mgr.14538) 84 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:13 smithi069 bash[21528]: cluster 2024-04-02T04:46:12.916262+0000 mon.a (mon.0) 905 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-04-02T04:46:14.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:13 smithi069 bash[28904]: audit 2024-04-02T04:46:12.812119+0000 mgr.y (mgr.14538) 84 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:14.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:13 smithi069 bash[28904]: cluster 2024-04-02T04:46:12.916262+0000 mon.a (mon.0) 905 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-04-02T04:46:14.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:13 smithi098 bash[23790]: audit 2024-04-02T04:46:12.812119+0000 mgr.y (mgr.14538) 84 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:14.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:13 smithi098 bash[23790]: cluster 2024-04-02T04:46:12.916262+0000 mon.a (mon.0) 905 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-04-02T04:46:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:13.708333+0000 osd.3 (osd.3) 65 : cluster 0 2.1e scrub starts 2024-04-02T04:46:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:13.710334+0000 osd.3 (osd.3) 66 : cluster 0 2.1e scrub ok 2024-04-02T04:46:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:13.761765+0000 mgr.y (mgr.14538) 85 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:14.101157+0000 osd.5 (osd.5) 43 : cluster 0 3.5 scrub starts 2024-04-02T04:46:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:14.106198+0000 osd.5 (osd.5) 44 : cluster 0 3.5 scrub ok 2024-04-02T04:46:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:14.461429+0000 osd.7 (osd.7) 47 : cluster 0 2.8 scrub starts 2024-04-02T04:46:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:14 smithi098 bash[23790]: cluster 2024-04-02T04:46:14.463948+0000 osd.7 (osd.7) 48 : cluster 0 2.8 scrub ok 2024-04-02T04:46:15.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:13.708333+0000 osd.3 (osd.3) 65 : cluster 0 2.1e scrub starts 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:13.710334+0000 osd.3 (osd.3) 66 : cluster 0 2.1e scrub ok 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:13.761765+0000 mgr.y (mgr.14538) 85 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:14.101157+0000 osd.5 (osd.5) 43 : cluster 0 3.5 scrub starts 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:14.106198+0000 osd.5 (osd.5) 44 : cluster 0 3.5 scrub ok 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:14.461429+0000 osd.7 (osd.7) 47 : cluster 0 2.8 scrub starts 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[21528]: cluster 2024-04-02T04:46:14.463948+0000 osd.7 (osd.7) 48 : cluster 0 2.8 scrub ok 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:13.708333+0000 osd.3 (osd.3) 65 : cluster 0 2.1e scrub starts 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:13.710334+0000 osd.3 (osd.3) 66 : cluster 0 2.1e scrub ok 2024-04-02T04:46:15.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:13.761765+0000 mgr.y (mgr.14538) 85 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:14.101157+0000 osd.5 (osd.5) 43 : cluster 0 3.5 scrub starts 2024-04-02T04:46:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:14.106198+0000 osd.5 (osd.5) 44 : cluster 0 3.5 scrub ok 2024-04-02T04:46:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:14.461429+0000 osd.7 (osd.7) 47 : cluster 0 2.8 scrub starts 2024-04-02T04:46:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:14 smithi069 bash[28904]: cluster 2024-04-02T04:46:14.463948+0000 osd.7 (osd.7) 48 : cluster 0 2.8 scrub ok 2024-04-02T04:46:17.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[21528]: cluster 2024-04-02T04:46:15.762225+0000 mgr.y (mgr.14538) 86 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:17.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[21528]: cluster 2024-04-02T04:46:15.986883+0000 osd.6 (osd.6) 35 : cluster 0 2.17 scrub starts 2024-04-02T04:46:17.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[21528]: cluster 2024-04-02T04:46:15.989116+0000 osd.6 (osd.6) 36 : cluster 0 2.17 scrub ok 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[21528]: cluster 2024-04-02T04:46:16.161153+0000 osd.5 (osd.5) 45 : cluster 0 2.18 scrub starts 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[21528]: cluster 2024-04-02T04:46:16.163272+0000 osd.5 (osd.5) 46 : cluster 0 2.18 scrub ok 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[28904]: cluster 2024-04-02T04:46:15.762225+0000 mgr.y (mgr.14538) 86 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[28904]: cluster 2024-04-02T04:46:15.986883+0000 osd.6 (osd.6) 35 : cluster 0 2.17 scrub starts 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[28904]: cluster 2024-04-02T04:46:15.989116+0000 osd.6 (osd.6) 36 : cluster 0 2.17 scrub ok 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[28904]: cluster 2024-04-02T04:46:16.161153+0000 osd.5 (osd.5) 45 : cluster 0 2.18 scrub starts 2024-04-02T04:46:17.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:16 smithi069 bash[28904]: cluster 2024-04-02T04:46:16.163272+0000 osd.5 (osd.5) 46 : cluster 0 2.18 scrub ok 2024-04-02T04:46:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:16 smithi098 bash[23790]: cluster 2024-04-02T04:46:15.762225+0000 mgr.y (mgr.14538) 86 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:16 smithi098 bash[23790]: cluster 2024-04-02T04:46:15.986883+0000 osd.6 (osd.6) 35 : cluster 0 2.17 scrub starts 2024-04-02T04:46:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:16 smithi098 bash[23790]: cluster 2024-04-02T04:46:15.989116+0000 osd.6 (osd.6) 36 : cluster 0 2.17 scrub ok 2024-04-02T04:46:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:16 smithi098 bash[23790]: cluster 2024-04-02T04:46:16.161153+0000 osd.5 (osd.5) 45 : cluster 0 2.18 scrub starts 2024-04-02T04:46:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:16 smithi098 bash[23790]: cluster 2024-04-02T04:46:16.163272+0000 osd.5 (osd.5) 46 : cluster 0 2.18 scrub ok 2024-04-02T04:46:18.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[21528]: cluster 2024-04-02T04:46:16.439944+0000 osd.1 (osd.1) 53 : cluster 0 2.d deep-scrub starts 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[21528]: cluster 2024-04-02T04:46:16.442223+0000 osd.1 (osd.1) 54 : cluster 0 2.d deep-scrub ok 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[21528]: cluster 2024-04-02T04:46:16.672295+0000 osd.3 (osd.3) 67 : cluster 0 3.8 scrub starts 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[21528]: cluster 2024-04-02T04:46:16.675521+0000 osd.3 (osd.3) 68 : cluster 0 3.8 scrub ok 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[21528]: cluster 2024-04-02T04:46:17.430118+0000 osd.7 (osd.7) 49 : cluster 0 1.0 scrub starts 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[21528]: cluster 2024-04-02T04:46:17.433551+0000 osd.7 (osd.7) 50 : cluster 0 1.0 scrub ok 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[28904]: cluster 2024-04-02T04:46:16.439944+0000 osd.1 (osd.1) 53 : cluster 0 2.d deep-scrub starts 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[28904]: cluster 2024-04-02T04:46:16.442223+0000 osd.1 (osd.1) 54 : cluster 0 2.d deep-scrub ok 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[28904]: cluster 2024-04-02T04:46:16.672295+0000 osd.3 (osd.3) 67 : cluster 0 3.8 scrub starts 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[28904]: cluster 2024-04-02T04:46:16.675521+0000 osd.3 (osd.3) 68 : cluster 0 3.8 scrub ok 2024-04-02T04:46:18.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[28904]: cluster 2024-04-02T04:46:17.430118+0000 osd.7 (osd.7) 49 : cluster 0 1.0 scrub starts 2024-04-02T04:46:18.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:17 smithi069 bash[28904]: cluster 2024-04-02T04:46:17.433551+0000 osd.7 (osd.7) 50 : cluster 0 1.0 scrub ok 2024-04-02T04:46:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:17 smithi098 bash[23790]: cluster 2024-04-02T04:46:16.439944+0000 osd.1 (osd.1) 53 : cluster 0 2.d deep-scrub starts 2024-04-02T04:46:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:17 smithi098 bash[23790]: cluster 2024-04-02T04:46:16.442223+0000 osd.1 (osd.1) 54 : cluster 0 2.d deep-scrub ok 2024-04-02T04:46:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:17 smithi098 bash[23790]: cluster 2024-04-02T04:46:16.672295+0000 osd.3 (osd.3) 67 : cluster 0 3.8 scrub starts 2024-04-02T04:46:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:17 smithi098 bash[23790]: cluster 2024-04-02T04:46:16.675521+0000 osd.3 (osd.3) 68 : cluster 0 3.8 scrub ok 2024-04-02T04:46:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:17 smithi098 bash[23790]: cluster 2024-04-02T04:46:17.430118+0000 osd.7 (osd.7) 49 : cluster 0 1.0 scrub starts 2024-04-02T04:46:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:17 smithi098 bash[23790]: cluster 2024-04-02T04:46:17.433551+0000 osd.7 (osd.7) 50 : cluster 0 1.0 scrub ok 2024-04-02T04:46:18.940 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:46:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:46:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:18 smithi098 bash[23790]: cluster 2024-04-02T04:46:17.763117+0000 mgr.y (mgr.14538) 87 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:18 smithi098 bash[23790]: cluster 2024-04-02T04:46:18.274871+0000 osd.0 (osd.0) 55 : cluster 0 4.b scrub starts 2024-04-02T04:46:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:18 smithi098 bash[23790]: cluster 2024-04-02T04:46:18.276502+0000 osd.0 (osd.0) 56 : cluster 0 4.b scrub ok 2024-04-02T04:46:19.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[21528]: cluster 2024-04-02T04:46:17.763117+0000 mgr.y (mgr.14538) 87 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:19.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[21528]: cluster 2024-04-02T04:46:18.274871+0000 osd.0 (osd.0) 55 : cluster 0 4.b scrub starts 2024-04-02T04:46:19.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[21528]: cluster 2024-04-02T04:46:18.276502+0000 osd.0 (osd.0) 56 : cluster 0 4.b scrub ok 2024-04-02T04:46:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[28904]: cluster 2024-04-02T04:46:17.763117+0000 mgr.y (mgr.14538) 87 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:19.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[28904]: cluster 2024-04-02T04:46:18.274871+0000 osd.0 (osd.0) 55 : cluster 0 4.b scrub starts 2024-04-02T04:46:19.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:18 smithi069 bash[28904]: cluster 2024-04-02T04:46:18.276502+0000 osd.0 (osd.0) 56 : cluster 0 4.b scrub ok 2024-04-02T04:46:19.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:19 smithi069 bash[21528]: cluster 2024-04-02T04:46:18.666841+0000 osd.3 (osd.3) 69 : cluster 0 3.b deep-scrub starts 2024-04-02T04:46:19.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:19 smithi069 bash[21528]: cluster 2024-04-02T04:46:18.673541+0000 osd.3 (osd.3) 70 : cluster 0 3.b deep-scrub ok 2024-04-02T04:46:19.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:19 smithi069 bash[21528]: cluster 2024-04-02T04:46:19.416528+0000 osd.7 (osd.7) 51 : cluster 0 3.e scrub starts 2024-04-02T04:46:19.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:19 smithi069 bash[21528]: cluster 2024-04-02T04:46:19.420728+0000 osd.7 (osd.7) 52 : cluster 0 3.e scrub ok 2024-04-02T04:46:20.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:19 smithi098 bash[23790]: cluster 2024-04-02T04:46:18.666841+0000 osd.3 (osd.3) 69 : cluster 0 3.b deep-scrub starts 2024-04-02T04:46:20.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:19 smithi098 bash[23790]: cluster 2024-04-02T04:46:18.673541+0000 osd.3 (osd.3) 70 : cluster 0 3.b deep-scrub ok 2024-04-02T04:46:20.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:19 smithi098 bash[23790]: cluster 2024-04-02T04:46:19.416528+0000 osd.7 (osd.7) 51 : cluster 0 3.e scrub starts 2024-04-02T04:46:20.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:19 smithi098 bash[23790]: cluster 2024-04-02T04:46:19.420728+0000 osd.7 (osd.7) 52 : cluster 0 3.e scrub ok 2024-04-02T04:46:20.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:20 smithi069 bash[28904]: cluster 2024-04-02T04:46:18.666841+0000 osd.3 (osd.3) 69 : cluster 0 3.b deep-scrub starts 2024-04-02T04:46:20.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:20 smithi069 bash[28904]: cluster 2024-04-02T04:46:18.673541+0000 osd.3 (osd.3) 70 : cluster 0 3.b deep-scrub ok 2024-04-02T04:46:20.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:20 smithi069 bash[28904]: cluster 2024-04-02T04:46:19.416528+0000 osd.7 (osd.7) 51 : cluster 0 3.e scrub starts 2024-04-02T04:46:20.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:20 smithi069 bash[28904]: cluster 2024-04-02T04:46:19.420728+0000 osd.7 (osd.7) 52 : cluster 0 3.e scrub ok 2024-04-02T04:46:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:20 smithi098 bash[23790]: cluster 2024-04-02T04:46:19.763597+0000 mgr.y (mgr.14538) 88 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:21.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:20 smithi069 bash[21528]: cluster 2024-04-02T04:46:19.763597+0000 mgr.y (mgr.14538) 88 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:21.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:20 smithi069 bash[28904]: cluster 2024-04-02T04:46:19.763597+0000 mgr.y (mgr.14538) 88 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:22.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.006986+0000 osd.6 (osd.6) 37 : cluster 0 2.1a scrub starts 2024-04-02T04:46:22.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.009408+0000 osd.6 (osd.6) 38 : cluster 0 2.1a scrub ok 2024-04-02T04:46:22.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.242570+0000 osd.5 (osd.5) 47 : cluster 0 4.8 scrub starts 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.244824+0000 osd.5 (osd.5) 48 : cluster 0 4.8 scrub ok 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.288542+0000 osd.0 (osd.0) 57 : cluster 0 2.1f scrub starts 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.290189+0000 osd.0 (osd.0) 58 : cluster 0 2.1f scrub ok 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.006986+0000 osd.6 (osd.6) 37 : cluster 0 2.1a scrub starts 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.009408+0000 osd.6 (osd.6) 38 : cluster 0 2.1a scrub ok 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.242570+0000 osd.5 (osd.5) 47 : cluster 0 4.8 scrub starts 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.244824+0000 osd.5 (osd.5) 48 : cluster 0 4.8 scrub ok 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.288542+0000 osd.0 (osd.0) 57 : cluster 0 2.1f scrub starts 2024-04-02T04:46:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:21 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.290189+0000 osd.0 (osd.0) 58 : cluster 0 2.1f scrub ok 2024-04-02T04:46:22.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2099918027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 msgr2=0x7f6c2c10f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2099918027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c10f2d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009f90 tx=0x7f6c2400b330 comp rx=0 tx=0).stop 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2099918027 shutdown_connections 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2099918027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c10f2d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2099918027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c06ac50 0x7f6c2c06b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2099918027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c2c103f60 0x7f6c2c1043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2099918027 >> 172.21.15.69:0/2099918027 conn(0x7f6c2c0771e0 msgr2=0x7f6c2c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2099918027 shutdown_connections 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.111+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2099918027 wait complete. 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 librados: starting msgr at 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 librados: starting objecter 2024-04-02T04:46:22.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 Processor -- start 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- start start 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 librados: setting wanted keys 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 librados: calling monclient init 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c2c06ac50 0x7f6c2c101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c103f60 0x7f6c2c101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c111c00 con 0x7f6c2c104920 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c111a80 con 0x7f6c2c06ac50 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2c111d80 con 0x7f6c2c103f60 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c103f60 0x7f6c2c101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:22.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c101d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55986/0 (socket says 172.21.15.69:55986) 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 -- 172.21.15.69:0/2330570009 learned_addr learned my addr 172.21.15.69:0/2330570009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c30d0a700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c2c06ac50 0x7f6c2c101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 -- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c103f60 msgr2=0x7f6c2c101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c103f60 0x7f6c2c101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 -- 172.21.15.69:0/2330570009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c2c06ac50 msgr2=0x7f6c2c101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c2c06ac50 0x7f6c2c101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 -- 172.21.15.69:0/2330570009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c1a66b0 con 0x7f6c2c104920 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c2bfff700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c103f60 0x7f6c2c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c3150b700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c101d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009f60 tx=0x7f6c2400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c29ffb700 10 client.?.objecter ms_handle_connect 0x7f6c2c104920 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c29ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:22.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24021070 con 0x7f6c2c104920 2024-04-02T04:46:22.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6c2401d450 con 0x7f6c2c104920 2024-04-02T04:46:22.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2c1a6940 con 0x7f6c2c104920 2024-04-02T04:46:22.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240052a0 con 0x7f6c2c104920 2024-04-02T04:46:22.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 10 client.14655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:22.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2c1a6df0 con 0x7f6c2c104920 2024-04-02T04:46:22.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 librados: init done 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c2c04ff00 con 0x7f6c2c104920 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.115+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6c24024070 con 0x7f6c2c104920 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c1c06f5e0 0x7f6c1c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c30d0a700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c1c06f5e0 0x7f6c1c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 54..112) ==== 7537+0+0 (secure 0 0 0) 0x7f6c2400f020 con 0x7f6c2c104920 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 10 client.14655.objecter ms_dispatch 0x7f6c2c001550 osd_map(112..112 src has 54..112) 2024-04-02T04:46:22.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 3 client.14655.objecter handle_osd_map got epochs [112,112] > 0 2024-04-02T04:46:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 3 client.14655.objecter handle_osd_map decoding full epoch 112 2024-04-02T04:46:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 20 client.14655.objecter dump_active .. 0 homeless 2024-04-02T04:46:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c29ffb700 10 client.14655.objecter ms_handle_connect 0x7f6c1c06f5e0 2024-04-02T04:46:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.119+0000 7f6c30d0a700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c1c06f5e0 0x7f6c1c071aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009750 tx=0x7f6c20009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:22.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.123+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c2402b020 con 0x7f6c2c104920 2024-04-02T04:46:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:21 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.006986+0000 osd.6 (osd.6) 37 : cluster 0 2.1a scrub starts 2024-04-02T04:46:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:21 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.009408+0000 osd.6 (osd.6) 38 : cluster 0 2.1a scrub ok 2024-04-02T04:46:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:21 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.242570+0000 osd.5 (osd.5) 47 : cluster 0 4.8 scrub starts 2024-04-02T04:46:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:21 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.244824+0000 osd.5 (osd.5) 48 : cluster 0 4.8 scrub ok 2024-04-02T04:46:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:21 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.288542+0000 osd.0 (osd.0) 57 : cluster 0 2.1f scrub starts 2024-04-02T04:46:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:21 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.290189+0000 osd.0 (osd.0) 58 : cluster 0 2.1f scrub ok 2024-04-02T04:46:22.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.335+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f6c2c1a7110 con 0x7f6c2c104920 2024-04-02T04:46:22.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.335+0000 7f6c29ffb700 1 -- 172.21.15.69:0/2330570009 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6c2407ae20 con 0x7f6c2c104920 2024-04-02T04:46:22.333 INFO:teuthology.orchestra.run.smithi069.stdout:249108103218 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.335+0000 7f6c32f6e700 10 librados: watch_flush enter 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.335+0000 7f6c32f6e700 10 librados: watch_flush exit 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.335+0000 7f6c32f6e700 20 client.14655.objecter shutdown clearing up homeless session... 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.335+0000 7f6c32f6e700 10 client.14655.objecter successfully canceled tick 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c1c06f5e0 msgr2=0x7f6c1c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c1c06f5e0 0x7f6c1c071aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009750 tx=0x7f6c20009310 comp rx=0 tx=0).stop 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 msgr2=0x7f6c2c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:22.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c101d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009f60 tx=0x7f6c2400cd00 comp rx=0 tx=0).stop 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 shutdown_connections 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c1c06f5e0 0x7f6c1c071aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c104920 0x7f6c2c101d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c2c06ac50 0x7f6c2c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 --2- 172.21.15.69:0/2330570009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c2c103f60 0x7f6c2c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 >> 172.21.15.69:0/2330570009 conn(0x7f6c2c0771e0 msgr2=0x7f6c2c107080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 shutdown_connections 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 -- 172.21.15.69:0/2330570009 wait complete. 2024-04-02T04:46:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:22.339+0000 7f6c32f6e700 1 librados: shutdown 2024-04-02T04:46:23.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:46:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.764153+0000 mgr.y (mgr.14538) 89 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:46:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.846018+0000 osd.4 (osd.4) 55 : cluster 0 3.1a scrub starts 2024-04-02T04:46:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: cluster 2024-04-02T04:46:21.850359+0000 osd.4 (osd.4) 56 : cluster 0 3.1a scrub ok 2024-04-02T04:46:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: audit 2024-04-02T04:46:21.967796+0000 mon.b (mon.1) 88 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: audit 2024-04-02T04:46:22.337601+0000 mon.a (mon.0) 906 : audit 0 from='client.? 172.21.15.69:0/2330570009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T04:46:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: cluster 2024-04-02T04:46:22.436092+0000 osd.7 (osd.7) 53 : cluster 0 6.3 deep-scrub starts 2024-04-02T04:46:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:22 smithi098 bash[23790]: cluster 2024-04-02T04:46:22.438102+0000 osd.7 (osd.7) 54 : cluster 0 6.3 deep-scrub ok 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.764153+0000 mgr.y (mgr.14538) 89 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.846018+0000 osd.4 (osd.4) 55 : cluster 0 3.1a scrub starts 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: cluster 2024-04-02T04:46:21.850359+0000 osd.4 (osd.4) 56 : cluster 0 3.1a scrub ok 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: audit 2024-04-02T04:46:21.967796+0000 mon.b (mon.1) 88 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: audit 2024-04-02T04:46:22.337601+0000 mon.a (mon.0) 906 : audit 0 from='client.? 172.21.15.69:0/2330570009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: cluster 2024-04-02T04:46:22.436092+0000 osd.7 (osd.7) 53 : cluster 0 6.3 deep-scrub starts 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[21528]: cluster 2024-04-02T04:46:22.438102+0000 osd.7 (osd.7) 54 : cluster 0 6.3 deep-scrub ok 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.764153+0000 mgr.y (mgr.14538) 89 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:46:23.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.846018+0000 osd.4 (osd.4) 55 : cluster 0 3.1a scrub starts 2024-04-02T04:46:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: cluster 2024-04-02T04:46:21.850359+0000 osd.4 (osd.4) 56 : cluster 0 3.1a scrub ok 2024-04-02T04:46:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: audit 2024-04-02T04:46:21.967796+0000 mon.b (mon.1) 88 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: audit 2024-04-02T04:46:22.337601+0000 mon.a (mon.0) 906 : audit 0 from='client.? 172.21.15.69:0/2330570009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T04:46:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: cluster 2024-04-02T04:46:22.436092+0000 osd.7 (osd.7) 53 : cluster 0 6.3 deep-scrub starts 2024-04-02T04:46:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:22 smithi069 bash[28904]: cluster 2024-04-02T04:46:22.438102+0000 osd.7 (osd.7) 54 : cluster 0 6.3 deep-scrub ok 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: cluster 2024-04-02T04:46:22.409300+0000 osd.1 (osd.1) 55 : cluster 0 2.4 scrub starts 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: cluster 2024-04-02T04:46:22.411005+0000 osd.1 (osd.1) 56 : cluster 0 2.4 scrub ok 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: cluster 2024-04-02T04:46:22.809298+0000 osd.4 (osd.4) 57 : cluster 0 4.1b scrub starts 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: cluster 2024-04-02T04:46:22.811812+0000 osd.4 (osd.4) 58 : cluster 0 4.1b scrub ok 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: audit 2024-04-02T04:46:22.813979+0000 mgr.y (mgr.14538) 90 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: cluster 2024-04-02T04:46:23.268683+0000 osd.0 (osd.0) 59 : cluster 0 4.3 scrub starts 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[21528]: cluster 2024-04-02T04:46:23.271918+0000 osd.0 (osd.0) 60 : cluster 0 4.3 scrub ok 2024-04-02T04:46:24.052 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: cluster 2024-04-02T04:46:22.409300+0000 osd.1 (osd.1) 55 : cluster 0 2.4 scrub starts 2024-04-02T04:46:24.053 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: cluster 2024-04-02T04:46:22.411005+0000 osd.1 (osd.1) 56 : cluster 0 2.4 scrub ok 2024-04-02T04:46:24.053 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: cluster 2024-04-02T04:46:22.809298+0000 osd.4 (osd.4) 57 : cluster 0 4.1b scrub starts 2024-04-02T04:46:24.053 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: cluster 2024-04-02T04:46:22.811812+0000 osd.4 (osd.4) 58 : cluster 0 4.1b scrub ok 2024-04-02T04:46:24.053 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: audit 2024-04-02T04:46:22.813979+0000 mgr.y (mgr.14538) 90 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:24.053 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: cluster 2024-04-02T04:46:23.268683+0000 osd.0 (osd.0) 59 : cluster 0 4.3 scrub starts 2024-04-02T04:46:24.053 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:23 smithi069 bash[28904]: cluster 2024-04-02T04:46:23.271918+0000 osd.0 (osd.0) 60 : cluster 0 4.3 scrub ok 2024-04-02T04:46:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: cluster 2024-04-02T04:46:22.409300+0000 osd.1 (osd.1) 55 : cluster 0 2.4 scrub starts 2024-04-02T04:46:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: cluster 2024-04-02T04:46:22.411005+0000 osd.1 (osd.1) 56 : cluster 0 2.4 scrub ok 2024-04-02T04:46:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: cluster 2024-04-02T04:46:22.809298+0000 osd.4 (osd.4) 57 : cluster 0 4.1b scrub starts 2024-04-02T04:46:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: cluster 2024-04-02T04:46:22.811812+0000 osd.4 (osd.4) 58 : cluster 0 4.1b scrub ok 2024-04-02T04:46:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: audit 2024-04-02T04:46:22.813979+0000 mgr.y (mgr.14538) 90 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:24.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: cluster 2024-04-02T04:46:23.268683+0000 osd.0 (osd.0) 59 : cluster 0 4.3 scrub starts 2024-04-02T04:46:24.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:23 smithi098 bash[23790]: cluster 2024-04-02T04:46:23.271918+0000 osd.0 (osd.0) 60 : cluster 0 4.3 scrub ok 2024-04-02T04:46:25.064 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[21528]: cluster 2024-04-02T04:46:23.765605+0000 mgr.y (mgr.14538) 91 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-04-02T04:46:25.064 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[21528]: cluster 2024-04-02T04:46:23.960561+0000 osd.5 (osd.5) 49 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:25.064 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[21528]: cluster 2024-04-02T04:46:23.960670+0000 osd.5 (osd.5) 50 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:25.064 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[21528]: cluster 2024-04-02T04:46:24.026371+0000 mon.a (mon.0) 907 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-04-02T04:46:25.064 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[21528]: cluster 2024-04-02T04:46:24.362327+0000 osd.7 (osd.7) 55 : cluster 0 6.8 scrub starts 2024-04-02T04:46:25.064 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[21528]: cluster 2024-04-02T04:46:24.364992+0000 osd.7 (osd.7) 56 : cluster 0 6.8 scrub ok 2024-04-02T04:46:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:25 smithi098 bash[23790]: cluster 2024-04-02T04:46:23.765605+0000 mgr.y (mgr.14538) 91 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-04-02T04:46:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:25 smithi098 bash[23790]: cluster 2024-04-02T04:46:23.960561+0000 osd.5 (osd.5) 49 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:25 smithi098 bash[23790]: cluster 2024-04-02T04:46:23.960670+0000 osd.5 (osd.5) 50 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:25 smithi098 bash[23790]: cluster 2024-04-02T04:46:24.026371+0000 mon.a (mon.0) 907 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-04-02T04:46:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:25 smithi098 bash[23790]: cluster 2024-04-02T04:46:24.362327+0000 osd.7 (osd.7) 55 : cluster 0 6.8 scrub starts 2024-04-02T04:46:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:25 smithi098 bash[23790]: cluster 2024-04-02T04:46:24.364992+0000 osd.7 (osd.7) 56 : cluster 0 6.8 scrub ok 2024-04-02T04:46:25.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[28904]: cluster 2024-04-02T04:46:23.765605+0000 mgr.y (mgr.14538) 91 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-04-02T04:46:25.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[28904]: cluster 2024-04-02T04:46:23.960561+0000 osd.5 (osd.5) 49 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:25.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[28904]: cluster 2024-04-02T04:46:23.960670+0000 osd.5 (osd.5) 50 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:25.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[28904]: cluster 2024-04-02T04:46:24.026371+0000 mon.a (mon.0) 907 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-04-02T04:46:25.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[28904]: cluster 2024-04-02T04:46:24.362327+0000 osd.7 (osd.7) 55 : cluster 0 6.8 scrub starts 2024-04-02T04:46:25.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:25 smithi069 bash[28904]: cluster 2024-04-02T04:46:24.364992+0000 osd.7 (osd.7) 56 : cluster 0 6.8 scrub ok 2024-04-02T04:46:26.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:26 smithi069 bash[28904]: cluster 2024-04-02T04:46:24.721722+0000 osd.3 (osd.3) 71 : cluster 0 3.18 scrub starts 2024-04-02T04:46:26.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:26 smithi069 bash[28904]: cluster 2024-04-02T04:46:24.725723+0000 osd.3 (osd.3) 72 : cluster 0 3.18 scrub ok 2024-04-02T04:46:26.152 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:26 smithi069 bash[21528]: cluster 2024-04-02T04:46:24.721722+0000 osd.3 (osd.3) 71 : cluster 0 3.18 scrub starts 2024-04-02T04:46:26.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:26 smithi069 bash[21528]: cluster 2024-04-02T04:46:24.725723+0000 osd.3 (osd.3) 72 : cluster 0 3.18 scrub ok 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2614759202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c101150 msgr2=0x7fc80c10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- 172.21.15.69:0/2614759202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c101150 0x7fc80c10f120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc80400b0d0 tx=0x7fc80400b3e0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2614759202 shutdown_connections 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- 172.21.15.69:0/2614759202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c101150 0x7fc80c10f120 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- 172.21.15.69:0/2614759202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c107260 0x7fc80c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- 172.21.15.69:0/2614759202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c100790 0x7fc80c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2614759202 >> 172.21.15.69:0/2614759202 conn(0x7fc80c077740 msgr2=0x7fc80c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:26.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2614759202 shutdown_connections 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2614759202 wait complete. 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 librados: starting msgr at 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 librados: starting objecter 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 Processor -- start 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- start start 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 librados: setting wanted keys 2024-04-02T04:46:26.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 librados: calling monclient init 2024-04-02T04:46:26.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c100790 0x7fc80c101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c101150 0x7fc80c101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 0x7fc80c1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c111be0 con 0x7fc80c100790 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c111a60 con 0x7fc80c101150 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc80c111d60 con 0x7fc80c107260 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 0x7fc80c1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc80bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c100790 0x7fc80c101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 0x7fc80c1023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36848/0 (socket says 172.21.15.69:36848) 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 -- 172.21.15.69:0/2100194829 learned_addr learned my addr 172.21.15.69:0/2100194829 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc80b7fe700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c101150 0x7fc80c101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 -- 172.21.15.69:0/2100194829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c101150 msgr2=0x7fc80c101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c101150 0x7fc80c101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 -- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c100790 msgr2=0x7fc80c101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c100790 0x7fc80c101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 -- 172.21.15.69:0/2100194829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c1a65a0 con 0x7fc80c107260 2024-04-02T04:46:26.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc80bfff700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c100790 0x7fc80c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc80b7fe700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c101150 0x7fc80c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc810f17700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 0x7fc80c1023f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc804006070 tx=0x7fc804007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc8097fa700 10 client.?.objecter ms_handle_connect 0x7fc80c107260 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc8097fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804021020 con 0x7fc80c107260 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc804004c50 con 0x7fc80c107260 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c1a6830 con 0x7fc80c107260 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804005050 con 0x7fc80c107260 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 10 client.24548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c1a6d30 con 0x7fc80c107260 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 librados: init done 2024-04-02T04:46:26.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.335+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80c102e50 con 0x7fc80c107260 2024-04-02T04:46:26.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc804024020 con 0x7fc80c107260 2024-04-02T04:46:26.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc7f406f5e0 0x7fc7f4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(113..113 src has 54..113) ==== 7537+0+0 (secure 0 0 0) 0x7fc80400f020 con 0x7fc80c107260 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 10 client.24548.objecter ms_dispatch 0x7fc80c001550 osd_map(113..113 src has 54..113) 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 3 client.24548.objecter handle_osd_map got epochs [113,113] > 0 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 3 client.24548.objecter handle_osd_map decoding full epoch 113 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc80bfff700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc7f406f5e0 0x7fc7f4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 20 client.24548.objecter dump_active .. 0 homeless 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc80bfff700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc7f406f5e0 0x7fc7f4071aa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009680 tx=0x7fc7fc009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:26.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.339+0000 7fc8097fa700 10 client.24548.objecter ms_handle_connect 0x7fc7f406f5e0 2024-04-02T04:46:26.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.343+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc80405c710 con 0x7fc80c107260 2024-04-02T04:46:26.385 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103215 got 249108103218 for osd.6 2024-04-02T04:46:26.386 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:26.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:26 smithi098 bash[23790]: cluster 2024-04-02T04:46:24.721722+0000 osd.3 (osd.3) 71 : cluster 0 3.18 scrub starts 2024-04-02T04:46:26.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:26 smithi098 bash[23790]: cluster 2024-04-02T04:46:24.725723+0000 osd.3 (osd.3) 72 : cluster 0 3.18 scrub ok 2024-04-02T04:46:26.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc80c1a6f90 con 0x7fc80c107260 2024-04-02T04:46:26.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc8097fa700 1 -- 172.21.15.69:0/2100194829 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc804060160 con 0x7fc80c107260 2024-04-02T04:46:26.568 INFO:teuthology.orchestra.run.smithi069.stdout:81604378716 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 10 librados: watch_flush enter 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 10 librados: watch_flush exit 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 20 client.24548.objecter shutdown clearing up homeless session... 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 10 client.24548.objecter successfully canceled tick 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc7f406f5e0 msgr2=0x7fc7f4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc7f406f5e0 0x7fc7f4071aa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009680 tx=0x7fc7fc009260 comp rx=0 tx=0).stop 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 msgr2=0x7fc80c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 0x7fc80c1023f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc804006070 tx=0x7fc804007a20 comp rx=0 tx=0).stop 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 shutdown_connections 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc7f406f5e0 0x7fc7f4071aa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc80c107260 0x7fc80c1023f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc80c100790 0x7fc80c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 --2- 172.21.15.69:0/2100194829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc80c101150 0x7fc80c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 >> 172.21.15.69:0/2100194829 conn(0x7fc80c077740 msgr2=0x7fc80c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 shutdown_connections 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 -- 172.21.15.69:0/2100194829 wait complete. 2024-04-02T04:46:26.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.571+0000 7fc81297a700 1 librados: shutdown 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- 172.21.15.69:0/3509834163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c107230 msgr2=0x7f567c107610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- 172.21.15.69:0/3509834163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c107230 0x7f567c107610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5678009990 tx=0x7f5678009ca0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- 172.21.15.69:0/3509834163 shutdown_connections 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- 172.21.15.69:0/3509834163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 0x7f567c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- 172.21.15.69:0/3509834163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c107230 0x7f567c107610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- 172.21.15.69:0/3509834163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c101180 0x7f567c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- 172.21.15.69:0/3509834163 >> 172.21.15.69:0/3509834163 conn(0x7f567c078140 msgr2=0x7f567c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- 172.21.15.69:0/3509834163 shutdown_connections 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- 172.21.15.69:0/3509834163 wait complete. 2024-04-02T04:46:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 librados: starting msgr at 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 librados: starting objecter 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 Processor -- start 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- start start 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 librados: setting wanted keys 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 librados: calling monclient init 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c101180 0x7f567c196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 0x7f567c1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 0x7f567c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f567c110980 con 0x7f567c107230 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f567c110800 con 0x7f567c101b40 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f568376c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f567c110b00 con 0x7f567c101180 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 0x7f567c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 0x7f567c197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56030/0 (socket says 172.21.15.69:56030) 2024-04-02T04:46:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 -- 172.21.15.69:0/3306925524 learned_addr learned my addr 172.21.15.69:0/3306925524 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681508700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c101180 0x7f567c196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5680d07700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 0x7f567c1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 -- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c101180 msgr2=0x7f567c196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c101180 0x7f567c196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 -- 172.21.15.69:0/3306925524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 msgr2=0x7f567c1971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 0x7f567c1971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681d09700 1 -- 172.21.15.69:0/3306925524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c1a6500 con 0x7f567c107230 2024-04-02T04:46:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5681508700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c101180 0x7f567c196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.743+0000 7f5680d07700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 0x7f567c1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f5681d09700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 0x7f567c197710 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f567400b970 tx=0x7f567400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 10 client.?.objecter ms_handle_connect 0x7f567c107230 2024-04-02T04:46:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5674019070 con 0x7f567c107230 2024-04-02T04:46:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f567400cdb0 con 0x7f567c107230 2024-04-02T04:46:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567c1a67f0 con 0x7f567c107230 2024-04-02T04:46:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5674014e40 con 0x7f567c107230 2024-04-02T04:46:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f568376c700 10 client.14667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f567c1a6d60 con 0x7f567c107230 2024-04-02T04:46:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f568376c700 1 librados: init done 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f567c102870 con 0x7f567c107230 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f567401c070 con 0x7f567c107230 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566806f5e0 0x7f5668071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 54..113) ==== 7537+0+0 (secure 0 0 0) 0x7f567408d440 con 0x7f567c107230 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 10 client.14667.objecter ms_dispatch 0x7f567c001550 osd_map(113..113 src has 54..113) 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 3 client.14667.objecter handle_osd_map got epochs [113,113] > 0 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 3 client.14667.objecter handle_osd_map decoding full epoch 113 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f5681508700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566806f5e0 0x7f5668071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 20 client.14667.objecter dump_active .. 0 homeless 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f5681508700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566806f5e0 0x7f5668071aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f56780098e0 tx=0x7f5678005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.747+0000 7f56727fc700 10 client.14667.objecter ms_handle_connect 0x7f566806f5e0 2024-04-02T04:46:26.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.751+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f567405e470 con 0x7f567c107230 2024-04-02T04:46:26.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- 172.21.15.69:0/1434960400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0102c80 msgr2=0x7fafe010f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- 172.21.15.69:0/1434960400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0102c80 0x7fafe010f0f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fafd800b0a0 tx=0x7fafd800b3b0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- 172.21.15.69:0/1434960400 shutdown_connections 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- 172.21.15.69:0/1434960400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0102c80 0x7fafe010f0f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- 172.21.15.69:0/1434960400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0108340 0x7fafe0108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- 172.21.15.69:0/1434960400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafe01022c0 0x7fafe0102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- 172.21.15.69:0/1434960400 >> 172.21.15.69:0/1434960400 conn(0x7fafe00793f0 msgr2=0x7fafe0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- 172.21.15.69:0/1434960400 shutdown_connections 2024-04-02T04:46:26.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- 172.21.15.69:0/1434960400 wait complete. 2024-04-02T04:46:26.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 librados: starting msgr at 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 librados: starting objecter 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 Processor -- start 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- start start 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 librados: setting wanted keys 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 librados: calling monclient init 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafe01022c0 0x7fafe0074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 0x7fafe0074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 0x7fafe0074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fafe0111b40 con 0x7fafe0102c80 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafe01119c0 con 0x7fafe01022c0 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafe5ada700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fafe0111cc0 con 0x7fafe0108340 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafdffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 0x7fafe0074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 0x7fafe0074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafdffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 0x7fafe0074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36906/0 (socket says 172.21.15.69:36906) 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafdffff700 1 -- 172.21.15.69:0/1379459469 learned_addr learned my addr 172.21.15.69:0/1379459469 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.883+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 0x7fafe0074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56052/0 (socket says 172.21.15.69:56052) 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdf7fe700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafe01022c0 0x7fafe0074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdffff700 1 -- 172.21.15.69:0/1379459469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafe01022c0 msgr2=0x7fafe0074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdffff700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafe01022c0 0x7fafe0074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdffff700 1 -- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 msgr2=0x7fafe0074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdffff700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 0x7fafe0074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdffff700 1 -- 172.21.15.69:0/1379459469 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe01a6570 con 0x7fafe0108340 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafd6ffd700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 0x7fafe0074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:26.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdffff700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 0x7fafe0074fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009850 tx=0x7fafd80079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 10 client.?.objecter ms_handle_connect 0x7fafe0108340 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd8003c00 con 0x7fafe0108340 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fafd8003da0 con 0x7fafe0108340 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafe5ada700 1 -- 172.21.15.69:0/1379459469 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe01a6800 con 0x7fafe0108340 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd801b6a0 con 0x7fafe0108340 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafe5ada700 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafe5ada700 1 -- 172.21.15.69:0/1379459469 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe01a6cb0 con 0x7fafe0108340 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafe5ada700 1 librados: init done 2024-04-02T04:46:26.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc4005120 con 0x7fafe0108340 2024-04-02T04:46:26.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fafd8026450 con 0x7fafe0108340 2024-04-02T04:46:26.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fafcc06f5e0 0x7fafcc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:26.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(113..113 src has 54..113) ==== 7537+0+0 (secure 0 0 0) 0x7fafd800f020 con 0x7fafe0108340 2024-04-02T04:46:26.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 10 client.24560.objecter ms_dispatch 0x7fafe0001550 osd_map(113..113 src has 54..113) 2024-04-02T04:46:26.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 3 client.24560.objecter handle_osd_map got epochs [113,113] > 0 2024-04-02T04:46:26.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 3 client.24560.objecter handle_osd_map decoding full epoch 113 2024-04-02T04:46:26.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdf7fe700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fafcc06f5e0 0x7fafcc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:26.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 20 client.24560.objecter dump_active .. 0 homeless 2024-04-02T04:46:26.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdf7fe700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fafcc06f5e0 0x7fafcc071aa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009680 tx=0x7fafd0009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:26.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.887+0000 7fafdd7fa700 10 client.24560.objecter ms_handle_connect 0x7fafcc06f5e0 2024-04-02T04:46:26.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.895+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fafd805c7d0 con 0x7fafe0108340 2024-04-02T04:46:26.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.963+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f567c1a7020 con 0x7f567c107230 2024-04-02T04:46:26.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.963+0000 7f56727fc700 1 -- 172.21.15.69:0/3306925524 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5674062ca0 con 0x7f567c107230 2024-04-02T04:46:26.962 INFO:teuthology.orchestra.run.smithi069.stdout:34359738480 2024-04-02T04:46:26.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 10 librados: watch_flush enter 2024-04-02T04:46:26.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 10 librados: watch_flush exit 2024-04-02T04:46:26.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 20 client.14667.objecter shutdown clearing up homeless session... 2024-04-02T04:46:26.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 10 client.14667.objecter successfully canceled tick 2024-04-02T04:46:26.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566806f5e0 msgr2=0x7f5668071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566806f5e0 0x7f5668071aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f56780098e0 tx=0x7f5678005960 comp rx=0 tx=0).stop 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 msgr2=0x7f567c197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 0x7f567c197710 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f567400b970 tx=0x7f567400bc80 comp rx=0 tx=0).stop 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 shutdown_connections 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566806f5e0 0x7f5668071aa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f567c107230 0x7f567c197710 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f567c101180 0x7f567c196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 --2- 172.21.15.69:0/3306925524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f567c101b40 0x7f567c1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 >> 172.21.15.69:0/3306925524 conn(0x7f567c078140 msgr2=0x7f567c107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 shutdown_connections 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 -- 172.21.15.69:0/3306925524 wait complete. 2024-04-02T04:46:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:26.967+0000 7f568376c700 1 librados: shutdown 2024-04-02T04:46:27.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.099+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fafc40056e0 con 0x7fafe0108340 2024-04-02T04:46:27.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafdd7fa700 1 -- 172.21.15.69:0/1379459469 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fafd8060220 con 0x7fafe0108340 2024-04-02T04:46:27.099 INFO:teuthology.orchestra.run.smithi069.stdout:201863462972 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 10 librados: watch_flush enter 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 10 librados: watch_flush exit 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 20 client.24560.objecter shutdown clearing up homeless session... 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 10 client.24560.objecter successfully canceled tick 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fafcc06f5e0 msgr2=0x7fafcc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fafcc06f5e0 0x7fafcc071aa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009680 tx=0x7fafd0009260 comp rx=0 tx=0).stop 2024-04-02T04:46:27.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 msgr2=0x7fafe0074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 0x7fafe0074fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009850 tx=0x7fafd80079a0 comp rx=0 tx=0).stop 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 shutdown_connections 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fafcc06f5e0 0x7fafcc071aa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafe0108340 0x7fafe0074fc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafe01022c0 0x7fafe0074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 --2- 172.21.15.69:0/1379459469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafe0102c80 0x7fafe0074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 >> 172.21.15.69:0/1379459469 conn(0x7fafe00793f0 msgr2=0x7fafe0109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 shutdown_connections 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 -- 172.21.15.69:0/1379459469 wait complete. 2024-04-02T04:46:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:27.103+0000 7fafd67fc700 1 librados: shutdown 2024-04-02T04:46:27.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: cluster 2024-04-02T04:46:25.690370+0000 osd.3 (osd.3) 73 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:46:27.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: cluster 2024-04-02T04:46:25.694314+0000 osd.3 (osd.3) 74 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:46:27.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: cluster 2024-04-02T04:46:25.766323+0000 mgr.y (mgr.14538) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:27.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: cluster 2024-04-02T04:46:26.066106+0000 osd.6 (osd.6) 39 : cluster 0 3.a scrub starts 2024-04-02T04:46:27.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: cluster 2024-04-02T04:46:26.071235+0000 osd.6 (osd.6) 40 : cluster 0 3.a scrub ok 2024-04-02T04:46:27.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: audit 2024-04-02T04:46:26.572263+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.69:0/2100194829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[21528]: audit 2024-04-02T04:46:26.966573+0000 mon.a (mon.0) 908 : audit 0 from='client.? 172.21.15.69:0/3306925524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: cluster 2024-04-02T04:46:25.690370+0000 osd.3 (osd.3) 73 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: cluster 2024-04-02T04:46:25.694314+0000 osd.3 (osd.3) 74 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: cluster 2024-04-02T04:46:25.766323+0000 mgr.y (mgr.14538) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: cluster 2024-04-02T04:46:26.066106+0000 osd.6 (osd.6) 39 : cluster 0 3.a scrub starts 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: cluster 2024-04-02T04:46:26.071235+0000 osd.6 (osd.6) 40 : cluster 0 3.a scrub ok 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: audit 2024-04-02T04:46:26.572263+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.69:0/2100194829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T04:46:27.275 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:27 smithi069 bash[28904]: audit 2024-04-02T04:46:26.966573+0000 mon.a (mon.0) 908 : audit 0 from='client.? 172.21.15.69:0/3306925524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T04:46:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: cluster 2024-04-02T04:46:25.690370+0000 osd.3 (osd.3) 73 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:46:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: cluster 2024-04-02T04:46:25.694314+0000 osd.3 (osd.3) 74 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:46:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: cluster 2024-04-02T04:46:25.766323+0000 mgr.y (mgr.14538) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: cluster 2024-04-02T04:46:26.066106+0000 osd.6 (osd.6) 39 : cluster 0 3.a scrub starts 2024-04-02T04:46:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: cluster 2024-04-02T04:46:26.071235+0000 osd.6 (osd.6) 40 : cluster 0 3.a scrub ok 2024-04-02T04:46:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: audit 2024-04-02T04:46:26.572263+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.69:0/2100194829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T04:46:27.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:27 smithi098 bash[23790]: audit 2024-04-02T04:46:26.966573+0000 mon.a (mon.0) 908 : audit 0 from='client.? 172.21.15.69:0/3306925524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T04:46:28.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:26.735668+0000 osd.3 (osd.3) 75 : cluster 0 4.19 scrub starts 2024-04-02T04:46:28.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:26.736957+0000 osd.3 (osd.3) 76 : cluster 0 4.19 scrub ok 2024-04-02T04:46:28.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:26.815590+0000 osd.4 (osd.4) 59 : cluster 0 3.9 scrub starts 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:26.820366+0000 osd.4 (osd.4) 60 : cluster 0 3.9 scrub ok 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.074418+0000 osd.6 (osd.6) 41 : cluster 0 2.7 scrub starts 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.076873+0000 osd.6 (osd.6) 42 : cluster 0 2.7 scrub ok 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: audit 2024-04-02T04:46:27.103826+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.69:0/1379459469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.231678+0000 osd.0 (osd.0) 61 : cluster 0 6.18 scrub starts 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.233995+0000 osd.0 (osd.0) 62 : cluster 0 6.18 scrub ok 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.349328+0000 osd.7 (osd.7) 57 : cluster 0 3.15 scrub starts 2024-04-02T04:46:28.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:28 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.353492+0000 osd.7 (osd.7) 58 : cluster 0 3.15 scrub ok 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:26.735668+0000 osd.3 (osd.3) 75 : cluster 0 4.19 scrub starts 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:26.736957+0000 osd.3 (osd.3) 76 : cluster 0 4.19 scrub ok 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:26.815590+0000 osd.4 (osd.4) 59 : cluster 0 3.9 scrub starts 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:26.820366+0000 osd.4 (osd.4) 60 : cluster 0 3.9 scrub ok 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.074418+0000 osd.6 (osd.6) 41 : cluster 0 2.7 scrub starts 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.076873+0000 osd.6 (osd.6) 42 : cluster 0 2.7 scrub ok 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: audit 2024-04-02T04:46:27.103826+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.69:0/1379459469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.231678+0000 osd.0 (osd.0) 61 : cluster 0 6.18 scrub starts 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.233995+0000 osd.0 (osd.0) 62 : cluster 0 6.18 scrub ok 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.349328+0000 osd.7 (osd.7) 57 : cluster 0 3.15 scrub starts 2024-04-02T04:46:28.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.353492+0000 osd.7 (osd.7) 58 : cluster 0 3.15 scrub ok 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:26.735668+0000 osd.3 (osd.3) 75 : cluster 0 4.19 scrub starts 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:26.736957+0000 osd.3 (osd.3) 76 : cluster 0 4.19 scrub ok 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:26.815590+0000 osd.4 (osd.4) 59 : cluster 0 3.9 scrub starts 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:26.820366+0000 osd.4 (osd.4) 60 : cluster 0 3.9 scrub ok 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.074418+0000 osd.6 (osd.6) 41 : cluster 0 2.7 scrub starts 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.076873+0000 osd.6 (osd.6) 42 : cluster 0 2.7 scrub ok 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: audit 2024-04-02T04:46:27.103826+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.69:0/1379459469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.231678+0000 osd.0 (osd.0) 61 : cluster 0 6.18 scrub starts 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.233995+0000 osd.0 (osd.0) 62 : cluster 0 6.18 scrub ok 2024-04-02T04:46:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.349328+0000 osd.7 (osd.7) 57 : cluster 0 3.15 scrub starts 2024-04-02T04:46:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.353492+0000 osd.7 (osd.7) 58 : cluster 0 3.15 scrub ok 2024-04-02T04:46:29.052 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:46:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:46:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:46:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.767719+0000 mgr.y (mgr.14538) 93 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.774600+0000 osd.3 (osd.3) 77 : cluster 0 2.19 scrub starts 2024-04-02T04:46:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.777666+0000 osd.3 (osd.3) 78 : cluster 0 2.19 scrub ok 2024-04-02T04:46:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.783547+0000 osd.4 (osd.4) 61 : cluster 0 4.9 scrub starts 2024-04-02T04:46:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:27.787326+0000 osd.4 (osd.4) 62 : cluster 0 4.9 scrub ok 2024-04-02T04:46:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.036371+0000 osd.6 (osd.6) 43 : cluster 0 2.11 scrub starts 2024-04-02T04:46:29.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.039556+0000 osd.6 (osd.6) 44 : cluster 0 2.11 scrub ok 2024-04-02T04:46:29.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.200852+0000 osd.0 (osd.0) 63 : cluster 0 3.12 scrub starts 2024-04-02T04:46:29.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:29 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.203150+0000 osd.0 (osd.0) 64 : cluster 0 3.12 scrub ok 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.767719+0000 mgr.y (mgr.14538) 93 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.774600+0000 osd.3 (osd.3) 77 : cluster 0 2.19 scrub starts 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.777666+0000 osd.3 (osd.3) 78 : cluster 0 2.19 scrub ok 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.783547+0000 osd.4 (osd.4) 61 : cluster 0 4.9 scrub starts 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:27.787326+0000 osd.4 (osd.4) 62 : cluster 0 4.9 scrub ok 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.036371+0000 osd.6 (osd.6) 43 : cluster 0 2.11 scrub starts 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.039556+0000 osd.6 (osd.6) 44 : cluster 0 2.11 scrub ok 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.200852+0000 osd.0 (osd.0) 63 : cluster 0 3.12 scrub starts 2024-04-02T04:46:29.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.203150+0000 osd.0 (osd.0) 64 : cluster 0 3.12 scrub ok 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.767719+0000 mgr.y (mgr.14538) 93 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.774600+0000 osd.3 (osd.3) 77 : cluster 0 2.19 scrub starts 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.777666+0000 osd.3 (osd.3) 78 : cluster 0 2.19 scrub ok 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.783547+0000 osd.4 (osd.4) 61 : cluster 0 4.9 scrub starts 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:27.787326+0000 osd.4 (osd.4) 62 : cluster 0 4.9 scrub ok 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.036371+0000 osd.6 (osd.6) 43 : cluster 0 2.11 scrub starts 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.039556+0000 osd.6 (osd.6) 44 : cluster 0 2.11 scrub ok 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.200852+0000 osd.0 (osd.0) 63 : cluster 0 3.12 scrub starts 2024-04-02T04:46:29.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:29 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.203150+0000 osd.0 (osd.0) 64 : cluster 0 3.12 scrub ok 2024-04-02T04:46:29.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3988608256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 msgr2=0x7f5048111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:29.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- 172.21.15.69:0/3988608256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f5048111300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5040009f90 tx=0x7f504000b330 comp rx=0 tx=0).stop 2024-04-02T04:46:29.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3988608256 shutdown_connections 2024-04-02T04:46:29.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- 172.21.15.69:0/3988608256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f5048111300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:29.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- 172.21.15.69:0/3988608256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048103380 0x7f5048100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:29.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- 172.21.15.69:0/3988608256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048100cd0 0x7f5048101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:29.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3988608256 >> 172.21.15.69:0/3988608256 conn(0x7f5048077740 msgr2=0x7f50480feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:29.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3988608256 shutdown_connections 2024-04-02T04:46:29.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3988608256 wait complete. 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 librados: starting msgr at 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 librados: starting objecter 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 Processor -- start 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- start start 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 librados: setting wanted keys 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 librados: calling monclient init 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048100cd0 0x7f504819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048103380 0x7f504819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f504819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5048113f40 con 0x7f504810ef10 2024-04-02T04:46:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5048113dc0 con 0x7f5048103380 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f50481140c0 con 0x7f5048100cd0 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f504819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f504819ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34866/0 (socket says 172.21.15.69:34866) 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 -- 172.21.15.69:0/3301587068 learned_addr learned my addr 172.21.15.69:0/3301587068 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504d9e1700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048100cd0 0x7f504819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504d1e0700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048103380 0x7f504819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 -- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048100cd0 msgr2=0x7f504819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048100cd0 0x7f504819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 -- 172.21.15.69:0/3301587068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048103380 msgr2=0x7f504819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048103380 0x7f504819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 -- 172.21.15.69:0/3301587068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504819f480 con 0x7f504810ef10 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504d9e1700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048100cd0 0x7f504819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504d1e0700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048103380 0x7f504819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504e1e2700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f504819ed40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f504000b6c0 tx=0x7f5040009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f503effd700 10 client.?.objecter ms_handle_connect 0x7f504810ef10 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f503effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040021070 con 0x7f504810ef10 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f504001d450 con 0x7f504810ef10 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504819f710 con 0x7f504810ef10 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50400052a0 con 0x7f504810ef10 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 10 client.14679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50481a6ff0 con 0x7f504810ef10 2024-04-02T04:46:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.815+0000 7f504fc45700 1 librados: init done 2024-04-02T04:46:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f504804ff00 con 0x7f504810ef10 2024-04-02T04:46:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5040024070 con 0x7f504810ef10 2024-04-02T04:46:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503406f5e0 0x7f5034071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 54..114) ==== 7537+0+0 (secure 0 0 0) 0x7f504008eff0 con 0x7f504810ef10 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 10 client.14679.objecter ms_dispatch 0x7f5048001550 osd_map(114..114 src has 54..114) 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 3 client.14679.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 3 client.14679.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f504d9e1700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503406f5e0 0x7f5034071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 20 client.14679.objecter dump_active .. 0 homeless 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f503effd700 10 client.14679.objecter ms_handle_connect 0x7f503406f5e0 2024-04-02T04:46:29.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.819+0000 7f504d9e1700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503406f5e0 0x7f5034071aa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5044005fd0 tx=0x7f5044009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:29.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:29.823+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f504005c650 con 0x7f504810ef10 2024-04-02T04:46:30.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5048069d50 con 0x7f504810ef10 2024-04-02T04:46:30.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f503effd700 1 -- 172.21.15.69:0/3301587068 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f50400600a0 con 0x7f504810ef10 2024-04-02T04:46:30.037 INFO:teuthology.orchestra.run.smithi069.stdout:55834574951 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 10 librados: watch_flush enter 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 10 librados: watch_flush exit 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 20 client.14679.objecter shutdown clearing up homeless session... 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 10 client.14679.objecter successfully canceled tick 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503406f5e0 msgr2=0x7f5034071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503406f5e0 0x7f5034071aa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5044005fd0 tx=0x7f5044009310 comp rx=0 tx=0).stop 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 msgr2=0x7f504819ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.039+0000 7f504fc45700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f504819ed40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f504000b6c0 tx=0x7f5040009510 comp rx=0 tx=0).stop 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 shutdown_connections 2024-04-02T04:46:30.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503406f5e0 0x7f5034071aa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f504810ef10 0x7f504819ed40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5048100cd0 0x7f504819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 --2- 172.21.15.69:0/3301587068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5048103380 0x7f504819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 >> 172.21.15.69:0/3301587068 conn(0x7f5048077740 msgr2=0x7f50481104b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 shutdown_connections 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 -- 172.21.15.69:0/3301587068 wait complete. 2024-04-02T04:46:30.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.043+0000 7f504fc45700 1 librados: shutdown 2024-04-02T04:46:30.065 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378712 got 81604378716 for osd.2 2024-04-02T04:46:30.065 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.647885+0000 osd.1 (osd.1) 57 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.647990+0000 osd.1 (osd.1) 58 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.771676+0000 osd.3 (osd.3) 79 : cluster 0 4.14 scrub starts 2024-04-02T04:46:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[21528]: cluster 2024-04-02T04:46:28.775060+0000 osd.3 (osd.3) 80 : cluster 0 4.14 scrub ok 2024-04-02T04:46:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[21528]: cluster 2024-04-02T04:46:29.058614+0000 mon.a (mon.0) 909 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-04-02T04:46:30.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[21528]: audit 2024-04-02T04:46:30.041140+0000 mon.a (mon.0) 910 : audit 0 from='client.? 172.21.15.69:0/3301587068' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T04:46:30.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738476 got 34359738480 for osd.0 2024-04-02T04:46:30.451 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:30 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.647885+0000 osd.1 (osd.1) 57 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:30 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.647990+0000 osd.1 (osd.1) 58 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:30 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.771676+0000 osd.3 (osd.3) 79 : cluster 0 4.14 scrub starts 2024-04-02T04:46:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:30 smithi098 bash[23790]: cluster 2024-04-02T04:46:28.775060+0000 osd.3 (osd.3) 80 : cluster 0 4.14 scrub ok 2024-04-02T04:46:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:30 smithi098 bash[23790]: cluster 2024-04-02T04:46:29.058614+0000 mon.a (mon.0) 909 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-04-02T04:46:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:30 smithi098 bash[23790]: audit 2024-04-02T04:46:30.041140+0000 mon.a (mon.0) 910 : audit 0 from='client.? 172.21.15.69:0/3301587068' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T04:46:30.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.647885+0000 osd.1 (osd.1) 57 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:30.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.647990+0000 osd.1 (osd.1) 58 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:30.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.771676+0000 osd.3 (osd.3) 79 : cluster 0 4.14 scrub starts 2024-04-02T04:46:30.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[28904]: cluster 2024-04-02T04:46:28.775060+0000 osd.3 (osd.3) 80 : cluster 0 4.14 scrub ok 2024-04-02T04:46:30.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[28904]: cluster 2024-04-02T04:46:29.058614+0000 mon.a (mon.0) 909 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-04-02T04:46:30.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:30 smithi069 bash[28904]: audit 2024-04-02T04:46:30.041140+0000 mon.a (mon.0) 910 : audit 0 from='client.? 172.21.15.69:0/3301587068' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T04:46:30.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 -- 172.21.15.69:0/736205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104490 msgr2=0x7fd82c104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 --2- 172.21.15.69:0/736205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104490 0x7fd82c104910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd8140099c0 tx=0x7fd814009cd0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 -- 172.21.15.69:0/736205502 shutdown_connections 2024-04-02T04:46:30.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 --2- 172.21.15.69:0/736205502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c104e50 0x7fd82c111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 --2- 172.21.15.69:0/736205502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c10a540 0x7fd82c10a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 --2- 172.21.15.69:0/736205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104490 0x7fd82c104910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.739+0000 7fd831998700 1 -- 172.21.15.69:0/736205502 >> 172.21.15.69:0/736205502 conn(0x7fd82c0fff20 msgr2=0x7fd82c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- 172.21.15.69:0/736205502 shutdown_connections 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- 172.21.15.69:0/736205502 wait complete. 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 librados: starting msgr at 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 librados: starting objecter 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 Processor -- start 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- start start 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 librados: setting wanted keys 2024-04-02T04:46:30.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 librados: calling monclient init 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 0x7fd82c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104e50 0x7fd82c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c10a540 0x7fd82c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c113cc0 con 0x7fd82c104e50 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c113b40 con 0x7fd82c10a540 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd82c113e40 con 0x7fd82c104490 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 0x7fd82c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 0x7fd82c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56040/0 (socket says 172.21.15.69:56040) 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82affd700 1 -- 172.21.15.69:0/2065861386 learned_addr learned my addr 172.21.15.69:0/2065861386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104e50 0x7fd82c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82b7fe700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c10a540 0x7fd82c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:30.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 -- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 msgr2=0x7fd82c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 0x7fd82c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 -- 172.21.15.69:0/2065861386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c10a540 msgr2=0x7fd82c19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c10a540 0x7fd82c19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 -- 172.21.15.69:0/2065861386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82c19f210 con 0x7fd82c104e50 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82affd700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 0x7fd82c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82b7fe700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c10a540 0x7fd82c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd82a7fc700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104e50 0x7fd82c19a740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd814009f50 tx=0x7fd8140057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 10 client.?.objecter ms_handle_connect 0x7fd82c104e50 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814027070 con 0x7fd82c104e50 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd814003f50 con 0x7fd82c104e50 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82c19f4a0 con 0x7fd82c104e50 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814022c20 con 0x7fd82c104e50 2024-04-02T04:46:30.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 10 client.14685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:30.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd82c1a6d80 con 0x7fd82c104e50 2024-04-02T04:46:30.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 librados: init done 2024-04-02T04:46:30.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd82c105bb0 con 0x7fd82c104e50 2024-04-02T04:46:30.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd81402a070 con 0x7fd82c104e50 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.743+0000 7fd830996700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd81806f5e0 0x7fd818071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 54..114) ==== 7537+0+0 (secure 0 0 0) 0x7fd81408ee90 con 0x7fd82c104e50 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd830996700 10 client.14685.objecter ms_dispatch 0x7fd82c001550 osd_map(114..114 src has 54..114) 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd830996700 3 client.14685.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd82affd700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd81806f5e0 0x7fd818071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd830996700 3 client.14685.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd830996700 20 client.14685.objecter dump_active .. 0 homeless 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd830996700 10 client.14685.objecter ms_handle_connect 0x7fd81806f5e0 2024-04-02T04:46:30.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.747+0000 7fd82affd700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd81806f5e0 0x7fd818071aa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd81c005950 tx=0x7fd81c00b500 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:30.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.751+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd814013070 con 0x7fd82c104e50 2024-04-02T04:46:30.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.962+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd82c19b560 con 0x7fd82c104e50 2024-04-02T04:46:30.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.962+0000 7fd830996700 1 -- 172.21.15.69:0/2065861386 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd81405c470 con 0x7fd82c104e50 2024-04-02T04:46:30.961 INFO:teuthology.orchestra.run.smithi069.stdout:296352743465 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.962+0000 7fd831998700 10 librados: watch_flush enter 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.962+0000 7fd831998700 10 librados: watch_flush exit 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.962+0000 7fd831998700 20 client.14685.objecter shutdown clearing up homeless session... 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.962+0000 7fd831998700 10 client.14685.objecter successfully canceled tick 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd81806f5e0 msgr2=0x7fd818071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd81806f5e0 0x7fd818071aa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd81c005950 tx=0x7fd81c00b500 comp rx=0 tx=0).stop 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104e50 msgr2=0x7fd82c19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104e50 0x7fd82c19a740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd814009f50 tx=0x7fd8140057c0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 shutdown_connections 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd81806f5e0 0x7fd818071aa0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd82c10a540 0x7fd82c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c104490 0x7fd82c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 --2- 172.21.15.69:0/2065861386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c104e50 0x7fd82c19a740 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 >> 172.21.15.69:0/2065861386 conn(0x7fd82c0fff20 msgr2=0x7fd82c10b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 shutdown_connections 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 -- 172.21.15.69:0/2065861386 wait complete. 2024-04-02T04:46:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:30.966+0000 7fd831998700 1 librados: shutdown 2024-04-02T04:46:31.000 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462968 got 201863462972 for osd.5 2024-04-02T04:46:31.000 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/3372957964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 msgr2=0x7ff4ec107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/3372957964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec107600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009990 tx=0x7ff4dc009ca0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/3372957964 shutdown_connections 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/3372957964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/3372957964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec107600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/3372957964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 0x7ff4ec101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/3372957964 >> 172.21.15.69:0/3372957964 conn(0x7ff4ec0771e0 msgr2=0x7ff4ec0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/3372957964 shutdown_connections 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/3372957964 wait complete. 2024-04-02T04:46:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 librados: starting msgr at 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 librados: starting objecter 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 Processor -- start 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- start start 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 librados: setting wanted keys 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 librados: calling monclient init 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 0x7ff4ec198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec110970 con 0x7ff4ec107220 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1107f0 con 0x7ff4ec1011a0 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff4ec110af0 con 0x7ff4ec101b60 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56070/0 (socket says 172.21.15.69:56070) 2024-04-02T04:46:31.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 -- 172.21.15.69:0/2348661445 learned_addr learned my addr 172.21.15.69:0/2348661445 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4ead9d700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4ea59c700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 0x7ff4ec198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 -- 172.21.15.69:0/2348661445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 msgr2=0x7ff4ec198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 0x7ff4ec198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 -- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 msgr2=0x7ff4ec19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 -- 172.21.15.69:0/2348661445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ec19d050 con 0x7ff4ec101b60 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4ea59c700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 0x7ff4ec198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4ead9d700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4e9d9b700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec198580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000b530 tx=0x7ff4e000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4db7fe700 10 client.?.objecter ms_handle_connect 0x7ff4ec101b60 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4db7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0015020 con 0x7ff4ec101b60 2024-04-02T04:46:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ec19d340 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff4e0011cb0 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0010d90 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 10 client.24578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ec102d50 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 librados: init done 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.098+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ec04ff00 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff4e0018020 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff4d406f5e0 0x7ff4d4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(114..114 src has 54..114) ==== 7537+0+0 (secure 0 0 0) 0x7ff4e008d880 con 0x7ff4ec101b60 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 10 client.24578.objecter ms_dispatch 0x7ff4ec001550 osd_map(114..114 src has 54..114) 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 3 client.24578.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 3 client.24578.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4ea59c700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff4d406f5e0 0x7ff4d4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:31.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 20 client.24578.objecter dump_active .. 0 homeless 2024-04-02T04:46:31.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4ea59c700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff4d406f5e0 0x7ff4d4071aa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00b2e0 tx=0x7ff4dc005e80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:31.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.102+0000 7ff4db7fe700 10 client.24578.objecter ms_handle_connect 0x7ff4d406f5e0 2024-04-02T04:46:31.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.106+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff4e001f020 con 0x7ff4ec101b60 2024-04-02T04:46:31.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[21528]: cluster 2024-04-02T04:46:29.363247+0000 osd.1 (osd.1) 59 : cluster 0 3.4 scrub starts 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[21528]: cluster 2024-04-02T04:46:29.368728+0000 osd.1 (osd.1) 60 : cluster 0 3.4 scrub ok 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[21528]: cluster 2024-04-02T04:46:29.768168+0000 mgr.y (mgr.14538) 94 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[21528]: cluster 2024-04-02T04:46:30.522619+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[21528]: cluster 2024-04-02T04:46:30.524731+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[21528]: audit 2024-04-02T04:46:30.965490+0000 mon.a (mon.0) 911 : audit 0 from='client.? 172.21.15.69:0/2065861386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[28904]: cluster 2024-04-02T04:46:29.363247+0000 osd.1 (osd.1) 59 : cluster 0 3.4 scrub starts 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[28904]: cluster 2024-04-02T04:46:29.368728+0000 osd.1 (osd.1) 60 : cluster 0 3.4 scrub ok 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[28904]: cluster 2024-04-02T04:46:29.768168+0000 mgr.y (mgr.14538) 94 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:31.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[28904]: cluster 2024-04-02T04:46:30.522619+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-04-02T04:46:31.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[28904]: cluster 2024-04-02T04:46:30.524731+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-04-02T04:46:31.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:31 smithi069 bash[28904]: audit 2024-04-02T04:46:30.965490+0000 mon.a (mon.0) 911 : audit 0 from='client.? 172.21.15.69:0/2065861386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T04:46:31.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.310+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff4ec103010 con 0x7ff4ec101b60 2024-04-02T04:46:31.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.310+0000 7ff4db7fe700 1 -- 172.21.15.69:0/2348661445 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff4e005aee0 con 0x7ff4ec101b60 2024-04-02T04:46:31.309 INFO:teuthology.orchestra.run.smithi069.stdout:124554051667 2024-04-02T04:46:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 10 librados: watch_flush enter 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 10 librados: watch_flush exit 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 20 client.24578.objecter shutdown clearing up homeless session... 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 10 client.24578.objecter successfully canceled tick 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff4d406f5e0 msgr2=0x7ff4d4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff4d406f5e0 0x7ff4d4071aa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00b2e0 tx=0x7ff4dc005e80 comp rx=0 tx=0).stop 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 msgr2=0x7ff4ec198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec198580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000b530 tx=0x7ff4e000b840 comp rx=0 tx=0).stop 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 shutdown_connections 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff4d406f5e0 0x7ff4d4071aa0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4ec107220 0x7ff4ec19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4ec1011a0 0x7ff4ec198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 --2- 172.21.15.69:0/2348661445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4ec101b60 0x7ff4ec198580 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:31.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 >> 172.21.15.69:0/2348661445 conn(0x7ff4ec0771e0 msgr2=0x7ff4ec069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:31.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 shutdown_connections 2024-04-02T04:46:31.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 -- 172.21.15.69:0/2348661445 wait complete. 2024-04-02T04:46:31.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:31.314+0000 7ff4f0ee1700 1 librados: shutdown 2024-04-02T04:46:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:31 smithi098 bash[23790]: cluster 2024-04-02T04:46:29.363247+0000 osd.1 (osd.1) 59 : cluster 0 3.4 scrub starts 2024-04-02T04:46:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:31 smithi098 bash[23790]: cluster 2024-04-02T04:46:29.368728+0000 osd.1 (osd.1) 60 : cluster 0 3.4 scrub ok 2024-04-02T04:46:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:31 smithi098 bash[23790]: cluster 2024-04-02T04:46:29.768168+0000 mgr.y (mgr.14538) 94 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:31 smithi098 bash[23790]: cluster 2024-04-02T04:46:30.522619+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-04-02T04:46:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:31 smithi098 bash[23790]: cluster 2024-04-02T04:46:30.524731+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-04-02T04:46:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:31 smithi098 bash[23790]: audit 2024-04-02T04:46:30.965490+0000 mon.a (mon.0) 911 : audit 0 from='client.? 172.21.15.69:0/2065861386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T04:46:32.118 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574946 got 55834574951 for osd.1 2024-04-02T04:46:32.118 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:32 smithi098 bash[23790]: audit 2024-04-02T04:46:31.313916+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.69:0/2348661445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T04:46:32.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:32 smithi069 bash[21528]: audit 2024-04-02T04:46:31.313916+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.69:0/2348661445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T04:46:32.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:32 smithi069 bash[28904]: audit 2024-04-02T04:46:31.313916+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.69:0/2348661445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T04:46:32.847 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051662 got 124554051667 for osd.3 2024-04-02T04:46:32.848 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:32.863 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743460 got 296352743465 for osd.7 2024-04-02T04:46:32.863 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 -- 172.21.15.69:0/2217266866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102ce0 msgr2=0x7f7b4c10f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 --2- 172.21.15.69:0/2217266866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102ce0 0x7f7b4c10f0d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400b0a0 tx=0x7f7b4400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 -- 172.21.15.69:0/2217266866 shutdown_connections 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 --2- 172.21.15.69:0/2217266866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102ce0 0x7f7b4c10f0d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 --2- 172.21.15.69:0/2217266866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c108390 0x7f7b4c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 --2- 172.21.15.69:0/2217266866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c102320 0x7f7b4c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.946+0000 7f7b529b4700 1 -- 172.21.15.69:0/2217266866 >> 172.21.15.69:0/2217266866 conn(0x7f7b4c079310 msgr2=0x7f7b4c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- 172.21.15.69:0/2217266866 shutdown_connections 2024-04-02T04:46:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- 172.21.15.69:0/2217266866 wait complete. 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 librados: starting msgr at 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 librados: starting objecter 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 Processor -- start 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- start start 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 librados: setting wanted keys 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 librados: calling monclient init 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 0x7f7b4c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 0x7f7b4c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c108390 0x7f7b4c19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111b00 con 0x7f7b4c102320 2024-04-02T04:46:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111980 con 0x7f7b4c108390 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7b4c111c80 con 0x7f7b4c102ce0 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 0x7f7b4c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 0x7f7b4c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 0x7f7b4c198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34938/0 (socket says 172.21.15.69:34938) 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 0x7f7b4c198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56094/0 (socket says 172.21.15.69:56094) 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 -- 172.21.15.69:0/442503370 learned_addr learned my addr 172.21.15.69:0/442503370 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b50f51700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c108390 0x7f7b4c19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 -- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 msgr2=0x7f7b4c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 0x7f7b4c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 -- 172.21.15.69:0/442503370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c108390 msgr2=0x7f7b4c19c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c108390 0x7f7b4c19c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 -- 172.21.15.69:0/442503370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4c19d130 con 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4b7fe700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 0x7f7b4c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b4bfff700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 0x7f7b4c198120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00e9a0 tx=0x7f7b3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b497fa700 10 client.?.objecter ms_handle_connect 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b497fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00cbd0 con 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4c19d420 con 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7b3c00cd70 con 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c010e60 con 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 10 client.14697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4c074470 con 0x7f7b4c102320 2024-04-02T04:46:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 librados: init done 2024-04-02T04:46:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b4c04ff00 con 0x7f7b4c102320 2024-04-02T04:46:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.950+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7b3c010430 con 0x7f7b4c102320 2024-04-02T04:46:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7b3806f5e0 0x7f7b38071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 54..114) ==== 7537+0+0 (secure 0 0 0) 0x7f7b3c08e250 con 0x7f7b4c102320 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b4b7fe700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7b3806f5e0 0x7f7b38071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 10 client.14697.objecter ms_dispatch 0x7f7b4c001550 osd_map(114..114 src has 54..114) 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 3 client.14697.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 3 client.14697.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 20 client.14697.objecter dump_active .. 0 homeless 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b497fa700 10 client.14697.objecter ms_handle_connect 0x7f7b3806f5e0 2024-04-02T04:46:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.954+0000 7f7b4b7fe700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7b3806f5e0 0x7f7b38071aa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c079c10 tx=0x7f7b3400a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:32.958+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7b3c05b830 con 0x7f7b4c102320 2024-04-02T04:46:33.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7b4c0746d0 con 0x7f7b4c102320 2024-04-02T04:46:33.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b497fa700 1 -- 172.21.15.69:0/442503370 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7b3c05f280 con 0x7f7b4c102320 2024-04-02T04:46:33.164 INFO:teuthology.orchestra.run.smithi069.stdout:154618822729 2024-04-02T04:46:33.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 10 librados: watch_flush enter 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 10 librados: watch_flush exit 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 20 client.14697.objecter shutdown clearing up homeless session... 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 10 client.14697.objecter successfully canceled tick 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7b3806f5e0 msgr2=0x7f7b38071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7b3806f5e0 0x7f7b38071aa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c079c10 tx=0x7f7b3400a300 comp rx=0 tx=0).stop 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 msgr2=0x7f7b4c198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 0x7f7b4c198120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00e9a0 tx=0x7f7b3c00ecb0 comp rx=0 tx=0).stop 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 shutdown_connections 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7b3806f5e0 0x7f7b38071aa0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b4c108390 0x7f7b4c19c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b4c102320 0x7f7b4c198120 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 --2- 172.21.15.69:0/442503370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b4c102ce0 0x7f7b4c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 >> 172.21.15.69:0/442503370 conn(0x7f7b4c079310 msgr2=0x7f7b4c109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:33.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 shutdown_connections 2024-04-02T04:46:33.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 -- 172.21.15.69:0/442503370 wait complete. 2024-04-02T04:46:33.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:33.166+0000 7f7b529b4700 1 librados: shutdown 2024-04-02T04:46:33.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:33 smithi069 bash[28904]: cluster 2024-04-02T04:46:31.768854+0000 mgr.y (mgr.14538) 95 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:33.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:33 smithi069 bash[21528]: cluster 2024-04-02T04:46:31.768854+0000 mgr.y (mgr.14538) 95 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:33.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:46:32 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:46:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:33 smithi098 bash[23790]: cluster 2024-04-02T04:46:31.768854+0000 mgr.y (mgr.14538) 95 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:46:33.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822724 got 154618822729 for osd.4 2024-04-02T04:46:33.953 DEBUG:teuthology.parallel:result is None 2024-04-02T04:46:33.953 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-02T04:46:33.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:32.333672+0000 osd.1 (osd.1) 61 : cluster 0 4.4 scrub starts 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:32.335535+0000 osd.1 (osd.1) 62 : cluster 0 4.4 scrub ok 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:32.596105+0000 osd.2 (osd.2) 33 : cluster 0 4.1c scrub starts 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:32.598411+0000 osd.2 (osd.2) 34 : cluster 0 4.1c scrub ok 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: audit 2024-04-02T04:46:32.825199+0000 mgr.y (mgr.14538) 96 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:32.833991+0000 osd.4 (osd.4) 63 : cluster 0 4.d scrub starts 2024-04-02T04:46:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:32.836957+0000 osd.4 (osd.4) 64 : cluster 0 4.d scrub ok 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: audit 2024-04-02T04:46:33.168764+0000 mon.a (mon.0) 912 : audit 0 from='client.? 172.21.15.69:0/442503370' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.249038+0000 osd.0 (osd.0) 65 : cluster 0 3.1 scrub starts 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.251768+0000 osd.0 (osd.0) 66 : cluster 0 3.1 scrub ok 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.342866+0000 osd.7 (osd.7) 59 : cluster 0 3.13 scrub starts 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.347043+0000 osd.7 (osd.7) 60 : cluster 0 3.13 scrub ok 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:32.333672+0000 osd.1 (osd.1) 61 : cluster 0 4.4 scrub starts 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:32.335535+0000 osd.1 (osd.1) 62 : cluster 0 4.4 scrub ok 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:32.596105+0000 osd.2 (osd.2) 33 : cluster 0 4.1c scrub starts 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:32.598411+0000 osd.2 (osd.2) 34 : cluster 0 4.1c scrub ok 2024-04-02T04:46:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: audit 2024-04-02T04:46:32.825199+0000 mgr.y (mgr.14538) 96 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:32.833991+0000 osd.4 (osd.4) 63 : cluster 0 4.d scrub starts 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:32.836957+0000 osd.4 (osd.4) 64 : cluster 0 4.d scrub ok 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: audit 2024-04-02T04:46:33.168764+0000 mon.a (mon.0) 912 : audit 0 from='client.? 172.21.15.69:0/442503370' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.249038+0000 osd.0 (osd.0) 65 : cluster 0 3.1 scrub starts 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.251768+0000 osd.0 (osd.0) 66 : cluster 0 3.1 scrub ok 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.342866+0000 osd.7 (osd.7) 59 : cluster 0 3.13 scrub starts 2024-04-02T04:46:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:34 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.347043+0000 osd.7 (osd.7) 60 : cluster 0 3.13 scrub ok 2024-04-02T04:46:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:32.333672+0000 osd.1 (osd.1) 61 : cluster 0 4.4 scrub starts 2024-04-02T04:46:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:32.335535+0000 osd.1 (osd.1) 62 : cluster 0 4.4 scrub ok 2024-04-02T04:46:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:32.596105+0000 osd.2 (osd.2) 33 : cluster 0 4.1c scrub starts 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:32.598411+0000 osd.2 (osd.2) 34 : cluster 0 4.1c scrub ok 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: audit 2024-04-02T04:46:32.825199+0000 mgr.y (mgr.14538) 96 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:32.833991+0000 osd.4 (osd.4) 63 : cluster 0 4.d scrub starts 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:32.836957+0000 osd.4 (osd.4) 64 : cluster 0 4.d scrub ok 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: audit 2024-04-02T04:46:33.168764+0000 mon.a (mon.0) 912 : audit 0 from='client.? 172.21.15.69:0/442503370' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.249038+0000 osd.0 (osd.0) 65 : cluster 0 3.1 scrub starts 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.251768+0000 osd.0 (osd.0) 66 : cluster 0 3.1 scrub ok 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.342866+0000 osd.7 (osd.7) 59 : cluster 0 3.13 scrub starts 2024-04-02T04:46:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:34 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.347043+0000 osd.7 (osd.7) 60 : cluster 0 3.13 scrub ok 2024-04-02T04:46:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:35 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.613754+0000 osd.2 (osd.2) 35 : cluster 0 2.e scrub starts 2024-04-02T04:46:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:35 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.616287+0000 osd.2 (osd.2) 36 : cluster 0 2.e scrub ok 2024-04-02T04:46:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:35 smithi098 bash[23790]: cluster 2024-04-02T04:46:33.770324+0000 mgr.y (mgr.14538) 97 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:46:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:35 smithi098 bash[23790]: cluster 2024-04-02T04:46:34.220108+0000 osd.6 (osd.6) 45 : cluster 0 2.14 scrub starts 2024-04-02T04:46:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:35 smithi098 bash[23790]: cluster 2024-04-02T04:46:34.223336+0000 osd.6 (osd.6) 46 : cluster 0 2.14 scrub ok 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.613754+0000 osd.2 (osd.2) 35 : cluster 0 2.e scrub starts 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.616287+0000 osd.2 (osd.2) 36 : cluster 0 2.e scrub ok 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[21528]: cluster 2024-04-02T04:46:33.770324+0000 mgr.y (mgr.14538) 97 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[21528]: cluster 2024-04-02T04:46:34.220108+0000 osd.6 (osd.6) 45 : cluster 0 2.14 scrub starts 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[21528]: cluster 2024-04-02T04:46:34.223336+0000 osd.6 (osd.6) 46 : cluster 0 2.14 scrub ok 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.613754+0000 osd.2 (osd.2) 35 : cluster 0 2.e scrub starts 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.616287+0000 osd.2 (osd.2) 36 : cluster 0 2.e scrub ok 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[28904]: cluster 2024-04-02T04:46:33.770324+0000 mgr.y (mgr.14538) 97 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[28904]: cluster 2024-04-02T04:46:34.220108+0000 osd.6 (osd.6) 45 : cluster 0 2.14 scrub starts 2024-04-02T04:46:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:35 smithi069 bash[28904]: cluster 2024-04-02T04:46:34.223336+0000 osd.6 (osd.6) 46 : cluster 0 2.14 scrub ok 2024-04-02T04:46:36.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:36 smithi098 bash[23790]: cluster 2024-04-02T04:46:35.290124+0000 osd.7 (osd.7) 61 : cluster 0 2.1d scrub starts 2024-04-02T04:46:36.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:36 smithi098 bash[23790]: cluster 2024-04-02T04:46:35.292326+0000 osd.7 (osd.7) 62 : cluster 0 2.1d scrub ok 2024-04-02T04:46:36.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:36 smithi069 bash[21528]: cluster 2024-04-02T04:46:35.290124+0000 osd.7 (osd.7) 61 : cluster 0 2.1d scrub starts 2024-04-02T04:46:36.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:36 smithi069 bash[21528]: cluster 2024-04-02T04:46:35.292326+0000 osd.7 (osd.7) 62 : cluster 0 2.1d scrub ok 2024-04-02T04:46:36.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:36 smithi069 bash[28904]: cluster 2024-04-02T04:46:35.290124+0000 osd.7 (osd.7) 61 : cluster 0 2.1d scrub starts 2024-04-02T04:46:36.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:36 smithi069 bash[28904]: cluster 2024-04-02T04:46:35.292326+0000 osd.7 (osd.7) 62 : cluster 0 2.1d scrub ok 2024-04-02T04:46:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:37 smithi098 bash[23790]: cluster 2024-04-02T04:46:35.274098+0000 osd.5 (osd.5) 51 : cluster 0 6.7 scrub starts 2024-04-02T04:46:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:37 smithi098 bash[23790]: cluster 2024-04-02T04:46:35.276489+0000 osd.5 (osd.5) 52 : cluster 0 6.7 scrub ok 2024-04-02T04:46:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:37 smithi098 bash[23790]: cluster 2024-04-02T04:46:35.771025+0000 mgr.y (mgr.14538) 98 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:37 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.241972+0000 osd.0 (osd.0) 67 : cluster 0 2.13 scrub starts 2024-04-02T04:46:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:37 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.244504+0000 osd.0 (osd.0) 68 : cluster 0 2.13 scrub ok 2024-04-02T04:46:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:37 smithi098 bash[23790]: audit 2024-04-02T04:46:36.983106+0000 mon.b (mon.1) 89 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[21528]: cluster 2024-04-02T04:46:35.274098+0000 osd.5 (osd.5) 51 : cluster 0 6.7 scrub starts 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[21528]: cluster 2024-04-02T04:46:35.276489+0000 osd.5 (osd.5) 52 : cluster 0 6.7 scrub ok 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[21528]: cluster 2024-04-02T04:46:35.771025+0000 mgr.y (mgr.14538) 98 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.241972+0000 osd.0 (osd.0) 67 : cluster 0 2.13 scrub starts 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.244504+0000 osd.0 (osd.0) 68 : cluster 0 2.13 scrub ok 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[21528]: audit 2024-04-02T04:46:36.983106+0000 mon.b (mon.1) 89 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[28904]: cluster 2024-04-02T04:46:35.274098+0000 osd.5 (osd.5) 51 : cluster 0 6.7 scrub starts 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[28904]: cluster 2024-04-02T04:46:35.276489+0000 osd.5 (osd.5) 52 : cluster 0 6.7 scrub ok 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[28904]: cluster 2024-04-02T04:46:35.771025+0000 mgr.y (mgr.14538) 98 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:37.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.241972+0000 osd.0 (osd.0) 67 : cluster 0 2.13 scrub starts 2024-04-02T04:46:37.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.244504+0000 osd.0 (osd.0) 68 : cluster 0 2.13 scrub ok 2024-04-02T04:46:37.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:37 smithi069 bash[28904]: audit 2024-04-02T04:46:36.983106+0000 mon.b (mon.1) 89 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:38.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.283044+0000 osd.5 (osd.5) 53 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:46:38.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.285628+0000 osd.5 (osd.5) 54 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:46:38.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.315044+0000 osd.1 (osd.1) 63 : cluster 0 4.2 scrub starts 2024-04-02T04:46:38.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.317526+0000 osd.1 (osd.1) 64 : cluster 0 4.2 scrub ok 2024-04-02T04:46:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.770285+0000 osd.4 (osd.4) 65 : cluster 0 3.3 scrub starts 2024-04-02T04:46:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.775165+0000 osd.4 (osd.4) 66 : cluster 0 3.3 scrub ok 2024-04-02T04:46:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.939853+0000 osd.3 (osd.3) 81 : cluster 0 4.1d scrub starts 2024-04-02T04:46:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:38 smithi098 bash[23790]: cluster 2024-04-02T04:46:36.942130+0000 osd.3 (osd.3) 82 : cluster 0 4.1d scrub ok 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.283044+0000 osd.5 (osd.5) 53 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.285628+0000 osd.5 (osd.5) 54 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.315044+0000 osd.1 (osd.1) 63 : cluster 0 4.2 scrub starts 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.317526+0000 osd.1 (osd.1) 64 : cluster 0 4.2 scrub ok 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.770285+0000 osd.4 (osd.4) 65 : cluster 0 3.3 scrub starts 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.775165+0000 osd.4 (osd.4) 66 : cluster 0 3.3 scrub ok 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.939853+0000 osd.3 (osd.3) 81 : cluster 0 4.1d scrub starts 2024-04-02T04:46:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21528]: cluster 2024-04-02T04:46:36.942130+0000 osd.3 (osd.3) 82 : cluster 0 4.1d scrub ok 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.283044+0000 osd.5 (osd.5) 53 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.285628+0000 osd.5 (osd.5) 54 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.315044+0000 osd.1 (osd.1) 63 : cluster 0 4.2 scrub starts 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.317526+0000 osd.1 (osd.1) 64 : cluster 0 4.2 scrub ok 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.770285+0000 osd.4 (osd.4) 65 : cluster 0 3.3 scrub starts 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.775165+0000 osd.4 (osd.4) 66 : cluster 0 3.3 scrub ok 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.939853+0000 osd.3 (osd.3) 81 : cluster 0 4.1d scrub starts 2024-04-02T04:46:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[28904]: cluster 2024-04-02T04:46:36.942130+0000 osd.3 (osd.3) 82 : cluster 0 4.1d scrub ok 2024-04-02T04:46:38.735 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:39.027 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:46:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:46:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:46:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.278135+0000 osd.1 (osd.1) 65 : cluster 0 2.9 scrub starts 2024-04-02T04:46:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.280344+0000 osd.1 (osd.1) 66 : cluster 0 2.9 scrub ok 2024-04-02T04:46:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.297661+0000 osd.5 (osd.5) 55 : cluster 0 3.c deep-scrub starts 2024-04-02T04:46:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.302240+0000 osd.5 (osd.5) 56 : cluster 0 3.c deep-scrub ok 2024-04-02T04:46:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.772408+0000 mgr.y (mgr.14538) 99 : cluster 0 pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.803326+0000 osd.4 (osd.4) 67 : cluster 0 3.14 scrub starts 2024-04-02T04:46:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:39 smithi098 bash[23790]: cluster 2024-04-02T04:46:37.808080+0000 osd.4 (osd.4) 68 : cluster 0 3.14 scrub ok 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.278135+0000 osd.1 (osd.1) 65 : cluster 0 2.9 scrub starts 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.280344+0000 osd.1 (osd.1) 66 : cluster 0 2.9 scrub ok 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.297661+0000 osd.5 (osd.5) 55 : cluster 0 3.c deep-scrub starts 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.302240+0000 osd.5 (osd.5) 56 : cluster 0 3.c deep-scrub ok 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.772408+0000 mgr.y (mgr.14538) 99 : cluster 0 pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.803326+0000 osd.4 (osd.4) 67 : cluster 0 3.14 scrub starts 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[21528]: cluster 2024-04-02T04:46:37.808080+0000 osd.4 (osd.4) 68 : cluster 0 3.14 scrub ok 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.278135+0000 osd.1 (osd.1) 65 : cluster 0 2.9 scrub starts 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.280344+0000 osd.1 (osd.1) 66 : cluster 0 2.9 scrub ok 2024-04-02T04:46:39.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.297661+0000 osd.5 (osd.5) 55 : cluster 0 3.c deep-scrub starts 2024-04-02T04:46:39.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.302240+0000 osd.5 (osd.5) 56 : cluster 0 3.c deep-scrub ok 2024-04-02T04:46:39.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.772408+0000 mgr.y (mgr.14538) 99 : cluster 0 pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:46:39.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.803326+0000 osd.4 (osd.4) 67 : cluster 0 3.14 scrub starts 2024-04-02T04:46:39.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:39 smithi069 bash[28904]: cluster 2024-04-02T04:46:37.808080+0000 osd.4 (osd.4) 68 : cluster 0 3.14 scrub ok 2024-04-02T04:46:40.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[28904]: cluster 2024-04-02T04:46:38.619666+0000 osd.2 (osd.2) 37 : cluster 0 6.14 scrub starts 2024-04-02T04:46:40.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[28904]: cluster 2024-04-02T04:46:38.622210+0000 osd.2 (osd.2) 38 : cluster 0 6.14 scrub ok 2024-04-02T04:46:40.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[28904]: cluster 2024-04-02T04:46:38.890369+0000 osd.3 (osd.3) 83 : cluster 0 6.11 scrub starts 2024-04-02T04:46:40.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[28904]: cluster 2024-04-02T04:46:38.892763+0000 osd.3 (osd.3) 84 : cluster 0 6.11 scrub ok 2024-04-02T04:46:40.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[21528]: cluster 2024-04-02T04:46:38.619666+0000 osd.2 (osd.2) 37 : cluster 0 6.14 scrub starts 2024-04-02T04:46:40.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[21528]: cluster 2024-04-02T04:46:38.622210+0000 osd.2 (osd.2) 38 : cluster 0 6.14 scrub ok 2024-04-02T04:46:40.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[21528]: cluster 2024-04-02T04:46:38.890369+0000 osd.3 (osd.3) 83 : cluster 0 6.11 scrub starts 2024-04-02T04:46:40.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:40 smithi069 bash[21528]: cluster 2024-04-02T04:46:38.892763+0000 osd.3 (osd.3) 84 : cluster 0 6.11 scrub ok 2024-04-02T04:46:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:40 smithi098 bash[23790]: cluster 2024-04-02T04:46:38.619666+0000 osd.2 (osd.2) 37 : cluster 0 6.14 scrub starts 2024-04-02T04:46:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:40 smithi098 bash[23790]: cluster 2024-04-02T04:46:38.622210+0000 osd.2 (osd.2) 38 : cluster 0 6.14 scrub ok 2024-04-02T04:46:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:40 smithi098 bash[23790]: cluster 2024-04-02T04:46:38.890369+0000 osd.3 (osd.3) 83 : cluster 0 6.11 scrub starts 2024-04-02T04:46:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:40 smithi098 bash[23790]: cluster 2024-04-02T04:46:38.892763+0000 osd.3 (osd.3) 84 : cluster 0 6.11 scrub ok 2024-04-02T04:46:41.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1492502931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c84100f60 msgr2=0x7f8c8410a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:41.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1492502931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c84100f60 0x7f8c8410a3a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009990 tx=0x7f8c74009ca0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1492502931 shutdown_connections 2024-04-02T04:46:41.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1492502931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c8410aad0 0x7f8c84111450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1492502931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c841005b0 0x7f8c84100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1492502931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c84100f60 0x7f8c8410a3a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1492502931 >> 172.21.15.69:0/1492502931 conn(0x7f8c840782b0 msgr2=0x7f8c840fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1492502931 shutdown_connections 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1492502931 wait complete. 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 librados: starting msgr at 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 librados: starting objecter 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 Processor -- start 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- start start 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 librados: setting wanted keys 2024-04-02T04:46:41.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 librados: calling monclient init 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c841005b0 0x7f8c8419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c84100f60 0x7f8c8419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 0x7f8c8419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84113c60 con 0x7f8c84100f60 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84113ae0 con 0x7f8c841005b0 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8c84113de0 con 0x7f8c8410aad0 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 0x7f8c8419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 0x7f8c8419ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:51492/0 (socket says 172.21.15.69:51492) 2024-04-02T04:46:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 -- 172.21.15.69:0/1023945842 learned_addr learned my addr 172.21.15.69:0/1023945842 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c81d9b700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c84100f60 0x7f8c8419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8259c700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c841005b0 0x7f8c8419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 -- 172.21.15.69:0/1023945842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c841005b0 msgr2=0x7f8c8419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c841005b0 0x7f8c8419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 -- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c84100f60 msgr2=0x7f8c8419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c84100f60 0x7f8c8419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 -- 172.21.15.69:0/1023945842 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8419f1c0 con 0x7f8c8410aad0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c82d9d700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 0x7f8c8419ea80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00ada0 tx=0x7f8c7c00c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c7b7fe700 10 client.?.objecter ms_handle_connect 0x7f8c8410aad0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c7b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c017020 con 0x7f8c8410aad0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8c7c00ce00 con 0x7f8c8410aad0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c005260 con 0x7f8c8410aad0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8419f4b0 con 0x7f8c8410aad0 2024-04-02T04:46:41.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 10 client.24590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:41.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c841a6d30 con 0x7f8c8410aad0 2024-04-02T04:46:41.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.382+0000 7f8c8884a700 1 librados: init done 2024-04-02T04:46:41.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c84101790 con 0x7f8c8410aad0 2024-04-02T04:46:41.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8c7c00c980 con 0x7f8c8410aad0 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7006f5e0 0x7f8c70071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(114..114 src has 64..114) ==== 7537+0+0 (secure 0 0 0) 0x7f8c7c08d7b0 con 0x7f8c8410aad0 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 10 client.24590.objecter ms_dispatch 0x7f8c84001550 osd_map(114..114 src has 64..114) 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 3 client.24590.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 3 client.24590.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c8259c700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7006f5e0 0x7f8c70071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 20 client.24590.objecter dump_active .. 0 homeless 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c7b7fe700 10 client.24590.objecter ms_handle_connect 0x7f8c7006f5e0 2024-04-02T04:46:41.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.386+0000 7f8c8259c700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7006f5e0 0x7f8c70071aa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009bf0 tx=0x7f8c6c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:41.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.390+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8c7c021020 con 0x7f8c8410aad0 2024-04-02T04:46:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.340952+0000 osd.5 (osd.5) 57 : cluster 0 6.19 scrub starts 2024-04-02T04:46:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.343574+0000 osd.5 (osd.5) 58 : cluster 0 6.19 scrub ok 2024-04-02T04:46:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.347284+0000 osd.1 (osd.1) 67 : cluster 0 2.6 scrub starts 2024-04-02T04:46:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.351140+0000 osd.1 (osd.1) 68 : cluster 0 2.6 scrub ok 2024-04-02T04:46:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.772872+0000 mgr.y (mgr.14538) 100 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-02T04:46:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.855858+0000 osd.3 (osd.3) 85 : cluster 0 3.1e scrub starts 2024-04-02T04:46:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:41 smithi098 bash[23790]: cluster 2024-04-02T04:46:39.861244+0000 osd.3 (osd.3) 86 : cluster 0 3.1e scrub ok 2024-04-02T04:46:41.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.340952+0000 osd.5 (osd.5) 57 : cluster 0 6.19 scrub starts 2024-04-02T04:46:41.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.343574+0000 osd.5 (osd.5) 58 : cluster 0 6.19 scrub ok 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.347284+0000 osd.1 (osd.1) 67 : cluster 0 2.6 scrub starts 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.351140+0000 osd.1 (osd.1) 68 : cluster 0 2.6 scrub ok 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.772872+0000 mgr.y (mgr.14538) 100 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.855858+0000 osd.3 (osd.3) 85 : cluster 0 3.1e scrub starts 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[28904]: cluster 2024-04-02T04:46:39.861244+0000 osd.3 (osd.3) 86 : cluster 0 3.1e scrub ok 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.340952+0000 osd.5 (osd.5) 57 : cluster 0 6.19 scrub starts 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.343574+0000 osd.5 (osd.5) 58 : cluster 0 6.19 scrub ok 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.347284+0000 osd.1 (osd.1) 67 : cluster 0 2.6 scrub starts 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.351140+0000 osd.1 (osd.1) 68 : cluster 0 2.6 scrub ok 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.772872+0000 mgr.y (mgr.14538) 100 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-02T04:46:41.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.855858+0000 osd.3 (osd.3) 85 : cluster 0 3.1e scrub starts 2024-04-02T04:46:41.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:41 smithi069 bash[21528]: cluster 2024-04-02T04:46:39.861244+0000 osd.3 (osd.3) 86 : cluster 0 3.1e scrub ok 2024-04-02T04:46:41.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.594+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8c8419b510 con 0x7f8c7006f5e0 2024-04-02T04:46:41.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.606+0000 7f8c7b7fe700 1 -- 172.21.15.69:0/1023945842 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346825 (secure 0 0 0) 0x7f8c8419b510 con 0x7f8c7006f5e0 2024-04-02T04:46:41.609 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:46:41.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 10 librados: watch_flush enter 2024-04-02T04:46:41.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 10 librados: watch_flush exit 2024-04-02T04:46:41.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 20 client.24590.objecter shutdown clearing up homeless session... 2024-04-02T04:46:41.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 10 client.24590.objecter successfully canceled tick 2024-04-02T04:46:41.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7006f5e0 msgr2=0x7f8c70071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:41.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7006f5e0 0x7f8c70071aa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009bf0 tx=0x7f8c6c009260 comp rx=0 tx=0).stop 2024-04-02T04:46:41.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 msgr2=0x7f8c8419ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:41.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 0x7f8c8419ea80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00ada0 tx=0x7f8c7c00c330 comp rx=0 tx=0).stop 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 shutdown_connections 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7006f5e0 0x7f8c70071aa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8410aad0 0x7f8c8419ea80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c841005b0 0x7f8c8419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 --2- 172.21.15.69:0/1023945842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c84100f60 0x7f8c8419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 >> 172.21.15.69:0/1023945842 conn(0x7f8c840782b0 msgr2=0x7f8c84108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 shutdown_connections 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 -- 172.21.15.69:0/1023945842 wait complete. 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:41.610+0000 7f8c8884a700 1 librados: shutdown 2024-04-02T04:46:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:46:42.282 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-02T04:46:39.772630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1379,"num_read_kb":1193,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228476,"kb_used_data":8188,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749732740,"statfs":{"total":767960285184,"available":767726325760,"internally_reserved":0,"allocated":8384512,"data_stored":5073650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.720783"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055945+0000","last_change":"2024-04-02T04:45:29.804443+0000","last_active":"2024-04-02T04:46:29.055945+0000","last_peered":"2024-04-02T04:46:29.055945+0000","last_clean":"2024-04-02T04:46:29.055945+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:46:29.055945+0000","last_undegraded":"2024-04-02T04:46:29.055945+0000","last_fullsized":"2024-04-02T04:46:29.055945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:29.804217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:29.804217+0000","last_clean_scrub_stamp":"2024-04-02T04:45:29.804217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:55.947849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058715+0000","last_change":"2024-04-02T04:46:26.737065+0000","last_active":"2024-04-02T04:46:29.058715+0000","last_peered":"2024-04-02T04:46:29.058715+0000","last_clean":"2024-04-02T04:46:29.058715+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:46:29.058715+0000","last_undegraded":"2024-04-02T04:46:29.058715+0000","last_fullsized":"2024-04-02T04:46:29.058715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:26.736961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.736961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.907303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057857+0000","last_change":"2024-04-02T04:46:21.290286+0000","last_active":"2024-04-02T04:46:29.057857+0000","last_peered":"2024-04-02T04:46:29.057857+0000","last_clean":"2024-04-02T04:46:29.057857+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:46:29.057857+0000","last_undegraded":"2024-04-02T04:46:29.057857+0000","last_fullsized":"2024-04-02T04:46:29.057857+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.290194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.290194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.622357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058660+0000","last_change":"2024-04-02T04:45:08.688185+0000","last_active":"2024-04-02T04:46:29.058660+0000","last_peered":"2024-04-02T04:46:29.058660+0000","last_clean":"2024-04-02T04:46:29.058660+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:46:29.058660+0000","last_undegraded":"2024-04-02T04:46:29.058660+0000","last_fullsized":"2024-04-02T04:46:29.058660+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:45:08.688063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:08.688063+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:37.923901+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048868+0000","last_change":"2024-04-02T04:45:56.829369+0000","last_active":"2024-04-02T04:46:29.048868+0000","last_peered":"2024-04-02T04:46:29.048868+0000","last_clean":"2024-04-02T04:46:29.048868+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:46:29.048868+0000","last_undegraded":"2024-04-02T04:46:29.048868+0000","last_fullsized":"2024-04-02T04:46:29.048868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:56.829149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:56.829149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:11.291730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048580+0000","last_change":"2024-04-02T04:45:39.713804+0000","last_active":"2024-04-02T04:46:29.048580+0000","last_peered":"2024-04-02T04:46:29.048580+0000","last_clean":"2024-04-02T04:46:29.048580+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:46:29.048580+0000","last_undegraded":"2024-04-02T04:46:29.048580+0000","last_fullsized":"2024-04-02T04:46:29.048580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:39.713597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:39.713597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:05.446000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058249+0000","last_change":"2024-04-02T04:46:13.710516+0000","last_active":"2024-04-02T04:46:29.058249+0000","last_peered":"2024-04-02T04:46:29.058249+0000","last_clean":"2024-04-02T04:46:29.058249+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:46:29.058249+0000","last_undegraded":"2024-04-02T04:46:29.058249+0000","last_fullsized":"2024-04-02T04:46:29.058249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:13.710340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:13.710340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.036057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057617+0000","last_change":"2024-04-02T04:45:15.743054+0000","last_active":"2024-04-02T04:46:29.057617+0000","last_peered":"2024-04-02T04:46:29.057617+0000","last_clean":"2024-04-02T04:46:29.057617+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:46:29.057617+0000","last_undegraded":"2024-04-02T04:46:29.057617+0000","last_fullsized":"2024-04-02T04:46:29.057617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:15.742789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:15.742789+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:44.224254+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":78,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427277+0000","last_change":"2024-04-02T04:45:17.031670+0000","last_active":"2024-04-02T04:46:29.427277+0000","last_peered":"2024-04-02T04:46:29.427277+0000","last_clean":"2024-04-02T04:46:29.427277+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:46:29.427277+0000","last_undegraded":"2024-04-02T04:46:29.427277+0000","last_fullsized":"2024-04-02T04:46:29.427277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:17.031440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:17.031440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049495+0000","last_change":"2024-04-02T04:46:22.812121+0000","last_active":"2024-04-02T04:46:29.049495+0000","last_peered":"2024-04-02T04:46:29.049495+0000","last_clean":"2024-04-02T04:46:29.049495+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:46:29.049495+0000","last_undegraded":"2024-04-02T04:46:29.049495+0000","last_fullsized":"2024-04-02T04:46:29.049495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.811836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.811836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.362979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:35.292487+0000","last_change":"2024-04-02T04:46:35.292548+0000","last_active":"2024-04-02T04:46:35.292487+0000","last_peered":"2024-04-02T04:46:35.292487+0000","last_clean":"2024-04-02T04:46:35.292487+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:46:35.292487+0000","last_undegraded":"2024-04-02T04:46:35.292487+0000","last_fullsized":"2024-04-02T04:46:35.292487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.292336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.292336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.961526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.428915+0000","last_change":"2024-04-02T04:45:40.828366+0000","last_active":"2024-04-02T04:46:29.428915+0000","last_peered":"2024-04-02T04:46:29.428915+0000","last_clean":"2024-04-02T04:46:29.428915+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:46:29.428915+0000","last_undegraded":"2024-04-02T04:46:29.428915+0000","last_fullsized":"2024-04-02T04:46:29.428915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:45:40.828122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:40.828122+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:59.185115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":88,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054279+0000","last_change":"2024-04-02T04:46:27.234204+0000","last_active":"2024-04-02T04:46:29.054279+0000","last_peered":"2024-04-02T04:46:29.054279+0000","last_clean":"2024-04-02T04:46:29.054279+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:46:29.054279+0000","last_undegraded":"2024-04-02T04:46:29.054279+0000","last_fullsized":"2024-04-02T04:46:29.054279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.234006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.234006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.363415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049525+0000","last_change":"2024-04-02T04:45:20.625340+0000","last_active":"2024-04-02T04:46:29.049525+0000","last_peered":"2024-04-02T04:46:29.049525+0000","last_clean":"2024-04-02T04:46:29.049525+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:46:29.049525+0000","last_undegraded":"2024-04-02T04:46:29.049525+0000","last_fullsized":"2024-04-02T04:46:29.049525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:20.625222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:20.625222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:44.448253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223471+0000","last_change":"2024-04-02T04:45:35.338383+0000","last_active":"2024-04-02T04:46:30.223471+0000","last_peered":"2024-04-02T04:46:30.223471+0000","last_clean":"2024-04-02T04:46:30.223471+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:46:30.223471+0000","last_undegraded":"2024-04-02T04:46:30.223471+0000","last_fullsized":"2024-04-02T04:46:30.223471+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:45:35.338035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.338035+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:55.625980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.428924+0000","last_change":"2024-04-02T04:45:27.901815+0000","last_active":"2024-04-02T04:46:29.428924+0000","last_peered":"2024-04-02T04:46:29.428924+0000","last_clean":"2024-04-02T04:46:29.428924+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:46:29.428924+0000","last_undegraded":"2024-04-02T04:46:29.428924+0000","last_fullsized":"2024-04-02T04:46:29.428924+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:45:27.901655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.901655+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:44.054719+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059371+0000","last_change":"2024-04-02T04:45:47.692378+0000","last_active":"2024-04-02T04:46:29.059371+0000","last_peered":"2024-04-02T04:46:29.059371+0000","last_clean":"2024-04-02T04:46:29.059371+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:46:29.059371+0000","last_undegraded":"2024-04-02T04:46:29.059371+0000","last_fullsized":"2024-04-02T04:46:29.059371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:47.691992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.691992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:51.815637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.942301+0000","last_change":"2024-04-02T04:46:36.942358+0000","last_active":"2024-04-02T04:46:36.942301+0000","last_peered":"2024-04-02T04:46:36.942301+0000","last_clean":"2024-04-02T04:46:36.942301+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:46:36.942301+0000","last_undegraded":"2024-04-02T04:46:36.942301+0000","last_fullsized":"2024-04-02T04:46:36.942301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.942139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.942139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.526854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056204+0000","last_change":"2024-04-02T04:45:57.720821+0000","last_active":"2024-04-02T04:46:29.056204+0000","last_peered":"2024-04-02T04:46:29.056204+0000","last_clean":"2024-04-02T04:46:29.056204+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:46:29.056204+0000","last_undegraded":"2024-04-02T04:46:29.056204+0000","last_fullsized":"2024-04-02T04:46:29.056204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:57.720720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.720720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:17.288812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"85'23","reported_seq":137,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048946+0000","last_change":"2024-04-02T04:46:21.850698+0000","last_active":"2024-04-02T04:46:29.048946+0000","last_peered":"2024-04-02T04:46:29.048946+0000","last_clean":"2024-04-02T04:46:29.048946+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:46:29.048946+0000","last_undegraded":"2024-04-02T04:46:29.048946+0000","last_fullsized":"2024-04-02T04:46:29.048946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-02T04:46:21.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.850373+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:35.100552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050836+0000","last_change":"2024-04-02T04:45:55.851523+0000","last_active":"2024-04-02T04:46:29.050836+0000","last_peered":"2024-04-02T04:46:29.050836+0000","last_clean":"2024-04-02T04:46:29.050836+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:46:29.050836+0000","last_undegraded":"2024-04-02T04:46:29.050836+0000","last_fullsized":"2024-04-02T04:46:29.050836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:55.851292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:55.851292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:58.449590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:32.598626+0000","last_change":"2024-04-02T04:46:32.598626+0000","last_active":"2024-04-02T04:46:32.598626+0000","last_peered":"2024-04-02T04:46:32.598626+0000","last_clean":"2024-04-02T04:46:32.598626+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:46:32.598626+0000","last_undegraded":"2024-04-02T04:46:32.598626+0000","last_fullsized":"2024-04-02T04:46:32.598626+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.598421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.598421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.059173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427426+0000","last_change":"2024-04-02T04:46:21.009651+0000","last_active":"2024-04-02T04:46:29.427426+0000","last_peered":"2024-04-02T04:46:29.427426+0000","last_clean":"2024-04-02T04:46:29.427426+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:46:29.427426+0000","last_undegraded":"2024-04-02T04:46:29.427426+0000","last_fullsized":"2024-04-02T04:46:29.427426+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.009424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.009424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.532492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059949+0000","last_change":"2024-04-02T04:45:57.518634+0000","last_active":"2024-04-02T04:46:29.059949+0000","last_peered":"2024-04-02T04:46:29.059949+0000","last_clean":"2024-04-02T04:46:29.059949+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:46:29.059949+0000","last_undegraded":"2024-04-02T04:46:29.059949+0000","last_fullsized":"2024-04-02T04:46:29.059949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:45:57.518350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.518350+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:12.947455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432435+0000","last_change":"2024-04-02T04:45:22.352334+0000","last_active":"2024-04-02T04:46:29.432435+0000","last_peered":"2024-04-02T04:46:29.432435+0000","last_clean":"2024-04-02T04:46:29.432435+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:46:29.432435+0000","last_undegraded":"2024-04-02T04:46:29.432435+0000","last_fullsized":"2024-04-02T04:46:29.432435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:22.352122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:22.352122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:47.681591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.430433+0000","last_change":"2024-04-02T04:45:34.007364+0000","last_active":"2024-04-02T04:46:29.430433+0000","last_peered":"2024-04-02T04:46:29.430433+0000","last_clean":"2024-04-02T04:46:29.430433+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:46:29.430433+0000","last_undegraded":"2024-04-02T04:46:29.430433+0000","last_fullsized":"2024-04-02T04:46:29.430433+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:45:34.007109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:34.007109+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:46.587296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":102,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.060543+0000","last_change":"2024-04-02T04:46:27.777917+0000","last_active":"2024-04-02T04:46:29.060543+0000","last_peered":"2024-04-02T04:46:29.060543+0000","last_clean":"2024-04-02T04:46:29.060543+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:46:29.060543+0000","last_undegraded":"2024-04-02T04:46:29.060543+0000","last_fullsized":"2024-04-02T04:46:29.060543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:46:27.777679+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.777679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.754491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":124,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054971+0000","last_change":"2024-04-02T04:46:24.726007+0000","last_active":"2024-04-02T04:46:29.054971+0000","last_peered":"2024-04-02T04:46:29.054971+0000","last_clean":"2024-04-02T04:46:29.054971+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:46:29.054971+0000","last_undegraded":"2024-04-02T04:46:29.054971+0000","last_fullsized":"2024-04-02T04:46:29.054971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:24.725734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.725734+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.190873+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":81,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223292+0000","last_change":"2024-04-02T04:45:13.177531+0000","last_active":"2024-04-02T04:46:30.223292+0000","last_peered":"2024-04-02T04:46:30.223292+0000","last_clean":"2024-04-02T04:46:30.223292+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:46:30.223292+0000","last_undegraded":"2024-04-02T04:46:30.223292+0000","last_fullsized":"2024-04-02T04:46:30.223292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:45:13.177302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:13.177302+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:41.353846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059292+0000","last_change":"2024-04-02T04:46:04.466247+0000","last_active":"2024-04-02T04:46:29.059292+0000","last_peered":"2024-04-02T04:46:29.059292+0000","last_clean":"2024-04-02T04:46:29.059292+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:46:29.059292+0000","last_undegraded":"2024-04-02T04:46:29.059292+0000","last_fullsized":"2024-04-02T04:46:29.059292+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:04.465964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:04.465964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.060111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.426191+0000","last_change":"2024-04-02T04:46:16.163503+0000","last_active":"2024-04-02T04:46:29.426191+0000","last_peered":"2024-04-02T04:46:29.426191+0000","last_clean":"2024-04-02T04:46:29.426191+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:46:29.426191+0000","last_undegraded":"2024-04-02T04:46:29.426191+0000","last_fullsized":"2024-04-02T04:46:29.426191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.163286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.163286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:21.454867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":160,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431917+0000","last_change":"2024-04-02T04:45:23.312146+0000","last_active":"2024-04-02T04:46:29.431917+0000","last_peered":"2024-04-02T04:46:29.431917+0000","last_clean":"2024-04-02T04:46:29.431917+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:46:29.431917+0000","last_undegraded":"2024-04-02T04:46:29.431917+0000","last_fullsized":"2024-04-02T04:46:29.431917+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:45:23.311862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:23.311862+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:53.017930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057066+0000","last_change":"2024-04-02T04:45:35.692872+0000","last_active":"2024-04-02T04:46:29.057066+0000","last_peered":"2024-04-02T04:46:29.057066+0000","last_clean":"2024-04-02T04:46:29.057066+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:46:29.057066+0000","last_undegraded":"2024-04-02T04:46:29.057066+0000","last_fullsized":"2024-04-02T04:46:29.057066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:35.692628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.692628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:54.735437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":84,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.061009+0000","last_change":"2024-04-02T04:45:38.761482+0000","last_active":"2024-04-02T04:46:29.061009+0000","last_peered":"2024-04-02T04:46:29.061009+0000","last_clean":"2024-04-02T04:46:29.061009+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:46:29.061009+0000","last_undegraded":"2024-04-02T04:46:29.061009+0000","last_fullsized":"2024-04-02T04:46:29.061009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:38.761200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:38.761200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:54.439691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429773+0000","last_change":"2024-04-02T04:46:15.989347+0000","last_active":"2024-04-02T04:46:29.429773+0000","last_peered":"2024-04-02T04:46:29.429773+0000","last_clean":"2024-04-02T04:46:29.429773+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:46:29.429773+0000","last_undegraded":"2024-04-02T04:46:29.429773+0000","last_fullsized":"2024-04-02T04:46:29.429773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:15.989126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:46:15.989126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.092291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":126,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.428837+0000","last_change":"2024-04-02T04:45:05.166240+0000","last_active":"2024-04-02T04:46:29.428837+0000","last_peered":"2024-04-02T04:46:29.428837+0000","last_clean":"2024-04-02T04:46:29.428837+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:46:29.428837+0000","last_undegraded":"2024-04-02T04:46:29.428837+0000","last_fullsized":"2024-04-02T04:46:29.428837+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:45:05.165991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:05.165991+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:31.440107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222483+0000","last_change":"2024-04-02T04:45:53.375887+0000","last_active":"2024-04-02T04:46:30.222483+0000","last_peered":"2024-04-02T04:46:30.222483+0000","last_clean":"2024-04-02T04:46:30.222483+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:46:30.222483+0000","last_undegraded":"2024-04-02T04:46:30.222483+0000","last_fullsized":"2024-04-02T04:46:30.222483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:45:53.375671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:03.905102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055260+0000","last_change":"2024-04-02T04:46:10.724569+0000","last_active":"2024-04-02T04:46:29.055260+0000","last_peered":"2024-04-02T04:46:29.055260+0000","last_clean":"2024-04-02T04:46:29.055260+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:46:29.055260+0000","last_undegraded":"2024-04-02T04:46:29.055260+0000","last_fullsized":"2024-04-02T04:46:29.055260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.724363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.724363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:18.641739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.285798+0000","last_change":"2024-04-02T04:46:36.285859+0000","last_active":"2024-04-02T04:46:36.285798+0000","last_peered":"2024-04-02T04:46:36.285798+0000","last_clean":"2024-04-02T04:46:36.285798+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:46:36.285798+0000","last_undegraded":"2024-04-02T04:46:36.285798+0000","last_fullsized":"2024-04-02T04:46:36.285798+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.285639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.484787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057448+0000","last_change":"2024-04-02T04:45:43.604563+0000","last_active":"2024-04-02T04:46:29.057448+0000","last_peered":"2024-04-02T04:46:29.057448+0000","last_clean":"2024-04-02T04:46:29.057448+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:46:29.057448+0000","last_undegraded":"2024-04-02T04:46:29.057448+0000","last_fullsized":"2024-04-02T04:46:29.057448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:45:43.604325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:43.604325+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:03.047811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055477+0000","last_change":"2024-04-02T04:45:28.777238+0000","last_active":"2024-04-02T04:46:29.055477+0000","last_peered":"2024-04-02T04:46:29.055477+0000","last_clean":"2024-04-02T04:46:29.055477+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:46:29.055477+0000","last_undegraded":"2024-04-02T04:46:29.055477+0000","last_fullsized":"2024-04-02T04:46:29.055477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:28.776988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:28.776988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:32.824236+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049082+0000","last_change":"2024-04-02T04:46:10.895794+0000","last_active":"2024-04-02T04:46:29.049082+0000","last_peered":"2024-04-02T04:46:29.049082+0000","last_clean":"2024-04-02T04:46:29.049082+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:46:29.049082+0000","last_undegraded":"2024-04-02T04:46:29.049082+0000","last_fullsized":"2024-04-02T04:46:29.049082+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.895579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.895579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.482340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.433107+0000","last_change":"2024-04-02T04:45:16.372742+0000","last_active":"2024-04-02T04:46:29.433107+0000","last_peered":"2024-04-02T04:46:29.433107+0000","last_clean":"2024-04-02T04:46:29.433107+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:46:29.433107+0000","last_undegraded":"2024-04-02T04:46:29.433107+0000","last_fullsized":"2024-04-02T04:46:29.433107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:16.372527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:16.372527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:41.922002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":126,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049684+0000","last_change":"2024-04-02T04:45:27.671251+0000","last_active":"2024-04-02T04:46:29.049684+0000","last_peered":"2024-04-02T04:46:29.049684+0000","last_clean":"2024-04-02T04:46:29.049684+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:46:29.049684+0000","last_undegraded":"2024-04-02T04:46:29.049684+0000","last_fullsized":"2024-04-02T04:46:29.049684+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:45:27.671069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.671069+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:37.167390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054319+0000","last_change":"2024-04-02T04:45:59.476847+0000","last_active":"2024-04-02T04:46:29.054319+0000","last_peered":"2024-04-02T04:46:29.054319+0000","last_clean":"2024-04-02T04:46:29.054319+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:46:29.054319+0000","last_undegraded":"2024-04-02T04:46:29.054319+0000","last_fullsized":"2024-04-02T04:46:29.054319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:59.476642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:59.476642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:16.549126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431859+0000","last_change":"2024-04-02T04:45:26.322458+0000","last_active":"2024-04-02T04:46:29.431859+0000","last_peered":"2024-04-02T04:46:29.431859+0000","last_clean":"2024-04-02T04:46:29.431859+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:46:29.431859+0000","last_undegraded":"2024-04-02T04:46:29.431859+0000","last_fullsized":"2024-04-02T04:46:29.431859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:26.322114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:26.322114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:36.598502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:34.223544+0000","last_change":"2024-04-02T04:46:34.223691+0000","last_active":"2024-04-02T04:46:34.223544+0000","last_peered":"2024-04-02T04:46:34.223544+0000","last_clean":"2024-04-02T04:46:34.223544+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:46:34.223544+0000","last_undegraded":"2024-04-02T04:46:34.223544+0000","last_fullsized":"2024-04-02T04:46:34.223544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:34.223346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:34.223346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.085223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.224172+0000","last_change":"2024-04-02T04:46:27.353748+0000","last_active":"2024-04-02T04:46:30.224172+0000","last_peered":"2024-04-02T04:46:30.224172+0000","last_clean":"2024-04-02T04:46:30.224172+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:46:30.224172+0000","last_undegraded":"2024-04-02T04:46:30.224172+0000","last_fullsized":"2024-04-02T04:46:30.224172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:27.353503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.353503+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.633132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050841+0000","last_change":"2024-04-02T04:45:50.840290+0000","last_active":"2024-04-02T04:46:29.050841+0000","last_peered":"2024-04-02T04:46:29.050841+0000","last_clean":"2024-04-02T04:46:29.050841+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:46:29.050841+0000","last_undegraded":"2024-04-02T04:46:29.050841+0000","last_fullsized":"2024-04-02T04:46:29.050841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:50.839972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:50.839972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:01.685776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":128,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429978+0000","last_change":"2024-04-02T04:45:10.093384+0000","last_active":"2024-04-02T04:46:29.429978+0000","last_peered":"2024-04-02T04:46:29.429978+0000","last_clean":"2024-04-02T04:46:29.429978+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:46:29.429978+0000","last_undegraded":"2024-04-02T04:46:29.429978+0000","last_fullsized":"2024-04-02T04:46:29.429978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:45:10.093160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:10.093160+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:35.206923+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.244665+0000","last_change":"2024-04-02T04:46:36.244725+0000","last_active":"2024-04-02T04:46:36.244665+0000","last_peered":"2024-04-02T04:46:36.244665+0000","last_clean":"2024-04-02T04:46:36.244665+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:46:36.244665+0000","last_undegraded":"2024-04-02T04:46:36.244665+0000","last_fullsized":"2024-04-02T04:46:36.244665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.244514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.244514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.017937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":112,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055883+0000","last_change":"2024-04-02T04:46:28.203298+0000","last_active":"2024-04-02T04:46:29.055883+0000","last_peered":"2024-04-02T04:46:29.055883+0000","last_clean":"2024-04-02T04:46:29.055883+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:46:29.055883+0000","last_undegraded":"2024-04-02T04:46:29.055883+0000","last_fullsized":"2024-04-02T04:46:29.055883+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:46:28.203156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.203156+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.777551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058203+0000","last_change":"2024-04-02T04:46:01.454913+0000","last_active":"2024-04-02T04:46:29.058203+0000","last_peered":"2024-04-02T04:46:29.058203+0000","last_clean":"2024-04-02T04:46:29.058203+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:46:29.058203+0000","last_undegraded":"2024-04-02T04:46:29.058203+0000","last_fullsized":"2024-04-02T04:46:29.058203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:46:01.454803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:09.150138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"85'11","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429338+0000","last_change":"2024-04-02T04:46:28.775353+0000","last_active":"2024-04-02T04:46:29.429338+0000","last_peered":"2024-04-02T04:46:29.429338+0000","last_clean":"2024-04-02T04:46:29.429338+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:46:29.429338+0000","last_undegraded":"2024-04-02T04:46:29.429338+0000","last_fullsized":"2024-04-02T04:46:29.429338+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:28.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.775071+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:55.875528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427268+0000","last_change":"2024-04-02T04:45:18.948742+0000","last_active":"2024-04-02T04:46:29.427268+0000","last_peered":"2024-04-02T04:46:29.427268+0000","last_clean":"2024-04-02T04:46:29.427268+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:46:29.427268+0000","last_undegraded":"2024-04-02T04:46:29.427268+0000","last_fullsized":"2024-04-02T04:46:29.427268+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:18.948514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:18.948514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:43.158274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:33.347205+0000","last_change":"2024-04-02T04:46:33.347283+0000","last_active":"2024-04-02T04:46:33.347205+0000","last_peered":"2024-04-02T04:46:33.347205+0000","last_clean":"2024-04-02T04:46:33.347205+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:46:33.347205+0000","last_undegraded":"2024-04-02T04:46:33.347205+0000","last_fullsized":"2024-04-02T04:46:33.347205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:33.347053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.347053+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.542332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223274+0000","last_change":"2024-04-02T04:45:31.347266+0000","last_active":"2024-04-02T04:46:30.223274+0000","last_peered":"2024-04-02T04:46:30.223274+0000","last_clean":"2024-04-02T04:46:30.223274+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:46:30.223274+0000","last_undegraded":"2024-04-02T04:46:30.223274+0000","last_fullsized":"2024-04-02T04:46:30.223274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:31.347039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:31.347039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.504167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059507+0000","last_change":"2024-04-02T04:45:53.588113+0000","last_active":"2024-04-02T04:46:29.059507+0000","last_peered":"2024-04-02T04:46:29.059507+0000","last_clean":"2024-04-02T04:46:29.059507+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:46:29.059507+0000","last_undegraded":"2024-04-02T04:46:29.059507+0000","last_fullsized":"2024-04-02T04:46:29.059507+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:39.779316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:39.779316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:40.912003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.426783+0000","last_change":"2024-04-02T04:46:28.039798+0000","last_active":"2024-04-02T04:46:29.426783+0000","last_peered":"2024-04-02T04:46:29.426783+0000","last_clean":"2024-04-02T04:46:29.426783+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:46:29.426783+0000","last_undegraded":"2024-04-02T04:46:29.426783+0000","last_fullsized":"2024-04-02T04:46:29.426783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:28.039566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.039566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.163582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":94,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.426658+0000","last_change":"2024-04-02T04:45:26.945858+0000","last_active":"2024-04-02T04:46:29.426658+0000","last_peered":"2024-04-02T04:46:29.426658+0000","last_clean":"2024-04-02T04:46:29.426658+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:46:29.426658+0000","last_undegraded":"2024-04-02T04:46:29.426658+0000","last_fullsized":"2024-04-02T04:46:29.426658+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:45:26.945691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:26.945691+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:40.964869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":78,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.228362+0000","last_change":"2024-04-02T04:45:04.992006+0000","last_active":"2024-04-02T04:46:30.228362+0000","last_peered":"2024-04-02T04:46:30.228362+0000","last_clean":"2024-04-02T04:46:30.228362+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:46:30.228362+0000","last_undegraded":"2024-04-02T04:46:30.228362+0000","last_fullsized":"2024-04-02T04:46:30.228362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:04.991762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:04.991762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:34.868982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":84,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058073+0000","last_change":"2024-04-02T04:45:27.573088+0000","last_active":"2024-04-02T04:46:29.058073+0000","last_peered":"2024-04-02T04:46:29.058073+0000","last_clean":"2024-04-02T04:46:29.058073+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:46:29.058073+0000","last_undegraded":"2024-04-02T04:46:29.058073+0000","last_fullsized":"2024-04-02T04:46:29.058073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:27.572981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.572981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:54.743792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.227200+0000","last_change":"2024-04-02T04:45:28.276186+0000","last_active":"2024-04-02T04:46:30.227200+0000","last_peered":"2024-04-02T04:46:30.227200+0000","last_clean":"2024-04-02T04:46:30.227200+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:46:30.227200+0000","last_undegraded":"2024-04-02T04:46:30.227200+0000","last_fullsized":"2024-04-02T04:46:30.227200+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:28.276056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:28.276056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:43.977535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222640+0000","last_change":"2024-04-02T04:45:36.359126+0000","last_active":"2024-04-02T04:46:30.222640+0000","last_peered":"2024-04-02T04:46:30.222640+0000","last_clean":"2024-04-02T04:46:30.222640+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:46:30.222640+0000","last_undegraded":"2024-04-02T04:46:30.222640+0000","last_fullsized":"2024-04-02T04:46:30.222640+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:36.358885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:36.358885+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:47.833512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056780+0000","last_change":"2024-04-02T04:45:31.758736+0000","last_active":"2024-04-02T04:46:29.056780+0000","last_peered":"2024-04-02T04:46:29.056780+0000","last_clean":"2024-04-02T04:46:29.056780+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:46:29.056780+0000","last_undegraded":"2024-04-02T04:46:29.056780+0000","last_fullsized":"2024-04-02T04:46:29.056780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:31.758488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:31.758488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.166829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050530+0000","last_change":"2024-04-02T04:46:02.906156+0000","last_active":"2024-04-02T04:46:29.050530+0000","last_peered":"2024-04-02T04:46:29.050530+0000","last_clean":"2024-04-02T04:46:29.050530+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:46:29.050530+0000","last_undegraded":"2024-04-02T04:46:29.050530+0000","last_fullsized":"2024-04-02T04:46:29.050530+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:46:02.905900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:02.905900+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.976578+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":126,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223855+0000","last_change":"2024-04-02T04:46:19.421011+0000","last_active":"2024-04-02T04:46:30.223855+0000","last_peered":"2024-04-02T04:46:30.223855+0000","last_clean":"2024-04-02T04:46:30.223855+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:46:30.223855+0000","last_undegraded":"2024-04-02T04:46:30.223855+0000","last_fullsized":"2024-04-02T04:46:30.223855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:19.420739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:19.420739+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.952287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"85'11","reported_seq":134,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.423924+0000","last_change":"2024-04-02T04:46:27.787700+0000","last_active":"2024-04-02T04:46:29.423924+0000","last_peered":"2024-04-02T04:46:29.423924+0000","last_clean":"2024-04-02T04:46:29.423924+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:46:29.423924+0000","last_undegraded":"2024-04-02T04:46:29.423924+0000","last_fullsized":"2024-04-02T04:46:29.423924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:27.787353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.787353+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:41.371457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054634+0000","last_change":"2024-04-02T04:45:34.636728+0000","last_active":"2024-04-02T04:46:29.054634+0000","last_peered":"2024-04-02T04:46:29.054634+0000","last_clean":"2024-04-02T04:46:29.054634+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:46:29.054634+0000","last_undegraded":"2024-04-02T04:46:29.054634+0000","last_fullsized":"2024-04-02T04:46:29.054634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:34.636349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:34.636349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:37.774714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":107,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432666+0000","last_change":"2024-04-02T04:46:22.411099+0000","last_active":"2024-04-02T04:46:29.432666+0000","last_peered":"2024-04-02T04:46:29.432666+0000","last_clean":"2024-04-02T04:46:29.432666+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:46:29.432666+0000","last_undegraded":"2024-04-02T04:46:29.432666+0000","last_fullsized":"2024-04-02T04:46:29.432666+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:22.411010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.411010+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:32.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":149,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.430505+0000","last_change":"2024-04-02T04:46:14.106481+0000","last_active":"2024-04-02T04:46:29.430505+0000","last_peered":"2024-04-02T04:46:29.430505+0000","last_clean":"2024-04-02T04:46:29.430505+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:46:29.430505+0000","last_undegraded":"2024-04-02T04:46:29.430505+0000","last_fullsized":"2024-04-02T04:46:29.430505+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:14.106232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.106232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.525489+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.317696+0000","last_change":"2024-04-02T04:46:36.317753+0000","last_active":"2024-04-02T04:46:36.317696+0000","last_peered":"2024-04-02T04:46:36.317696+0000","last_clean":"2024-04-02T04:46:36.317696+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:46:36.317696+0000","last_undegraded":"2024-04-02T04:46:36.317696+0000","last_fullsized":"2024-04-02T04:46:36.317696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.317542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.317542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.283515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":82,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058401+0000","last_change":"2024-04-02T04:45:41.708071+0000","last_active":"2024-04-02T04:46:29.058401+0000","last_peered":"2024-04-02T04:46:29.058401+0000","last_clean":"2024-04-02T04:46:29.058401+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:46:29.058401+0000","last_undegraded":"2024-04-02T04:46:29.058401+0000","last_fullsized":"2024-04-02T04:46:29.058401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:45:41.707741+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:50.082235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427333+0000","last_change":"2024-04-02T04:45:38.855217+0000","last_active":"2024-04-02T04:46:29.427333+0000","last_peered":"2024-04-02T04:46:29.427333+0000","last_clean":"2024-04-02T04:46:29.427333+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:46:29.427333+0000","last_undegraded":"2024-04-02T04:46:29.427333+0000","last_fullsized":"2024-04-02T04:46:29.427333+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:38.855005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:38.855005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:53.569786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.775337+0000","last_change":"2024-04-02T04:46:36.775425+0000","last_active":"2024-04-02T04:46:36.775337+0000","last_peered":"2024-04-02T04:46:36.775337+0000","last_clean":"2024-04-02T04:46:36.775337+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:46:36.775337+0000","last_undegraded":"2024-04-02T04:46:36.775337+0000","last_fullsized":"2024-04-02T04:46:36.775337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:46:36.775178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.775178+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.278965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:32.335622+0000","last_change":"2024-04-02T04:46:32.335647+0000","last_active":"2024-04-02T04:46:32.335622+0000","last_peered":"2024-04-02T04:46:32.335622+0000","last_clean":"2024-04-02T04:46:32.335622+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:46:32.335622+0000","last_undegraded":"2024-04-02T04:46:32.335622+0000","last_fullsized":"2024-04-02T04:46:32.335622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.335539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.335539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.500350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223825+0000","last_change":"2024-04-02T04:45:17.229471+0000","last_active":"2024-04-02T04:46:30.223825+0000","last_peered":"2024-04-02T04:46:30.223825+0000","last_clean":"2024-04-02T04:46:30.223825+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:46:30.223825+0000","last_undegraded":"2024-04-02T04:46:30.223825+0000","last_fullsized":"2024-04-02T04:46:30.223825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:17.229181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:17.229181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:41.376732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431510+0000","last_change":"2024-04-02T04:45:15.357170+0000","last_active":"2024-04-02T04:46:29.431510+0000","last_peered":"2024-04-02T04:46:29.431510+0000","last_clean":"2024-04-02T04:46:29.431510+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:46:29.431510+0000","last_undegraded":"2024-04-02T04:46:29.431510+0000","last_fullsized":"2024-04-02T04:46:29.431510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:45:15.356917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:15.356917+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:45.114912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.227474+0000","last_change":"2024-04-02T04:46:07.378595+0000","last_active":"2024-04-02T04:46:30.227474+0000","last_peered":"2024-04-02T04:46:30.227474+0000","last_clean":"2024-04-02T04:46:30.227474+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:46:30.227474+0000","last_undegraded":"2024-04-02T04:46:30.227474+0000","last_fullsized":"2024-04-02T04:46:30.227474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:07.378519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:46:07.378519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:12.389433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.433465+0000","last_change":"2024-04-02T04:45:10.310286+0000","last_active":"2024-04-02T04:46:29.433465+0000","last_peered":"2024-04-02T04:46:29.433465+0000","last_clean":"2024-04-02T04:46:29.433465+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:46:29.433465+0000","last_undegraded":"2024-04-02T04:46:29.433465+0000","last_fullsized":"2024-04-02T04:46:29.433465+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:10.310072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:10.310072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:33.612581+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:35.276656+0000","last_change":"2024-04-02T04:46:35.276714+0000","last_active":"2024-04-02T04:46:35.276656+0000","last_peered":"2024-04-02T04:46:35.276656+0000","last_clean":"2024-04-02T04:46:35.276656+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:46:35.276656+0000","last_undegraded":"2024-04-02T04:46:35.276656+0000","last_fullsized":"2024-04-02T04:46:35.276656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.276502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.276502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.541614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055955+0000","last_change":"2024-04-02T04:45:45.679903+0000","last_active":"2024-04-02T04:46:29.055955+0000","last_peered":"2024-04-02T04:46:29.055955+0000","last_clean":"2024-04-02T04:46:29.055955+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:46:29.055955+0000","last_undegraded":"2024-04-02T04:46:29.055955+0000","last_fullsized":"2024-04-02T04:46:29.055955+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:45:45.679774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:45.679774+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:45.816952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427650+0000","last_change":"2024-04-02T04:45:35.792402+0000","last_active":"2024-04-02T04:46:29.427650+0000","last_peered":"2024-04-02T04:46:29.427650+0000","last_clean":"2024-04-02T04:46:29.427650+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:46:29.427650+0000","last_undegraded":"2024-04-02T04:46:29.427650+0000","last_fullsized":"2024-04-02T04:46:29.427650+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:35.792264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.792264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:50.312208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":356,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223025+0000","last_change":"2024-04-02T04:46:17.433834+0000","last_active":"2024-04-02T04:46:30.223025+0000","last_peered":"2024-04-02T04:46:30.223025+0000","last_clean":"2024-04-02T04:46:30.223025+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:46:30.223025+0000","last_undegraded":"2024-04-02T04:46:30.223025+0000","last_fullsized":"2024-04-02T04:46:30.223025+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:46:17.433563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:46:17.433563+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.028333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429295+0000","last_change":"2024-04-02T04:46:10.058200+0000","last_active":"2024-04-02T04:46:29.429295+0000","last_peered":"2024-04-02T04:46:29.429295+0000","last_clean":"2024-04-02T04:46:29.429295+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:46:29.429295+0000","last_undegraded":"2024-04-02T04:46:29.429295+0000","last_fullsized":"2024-04-02T04:46:29.429295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.057875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.057875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.984328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432225+0000","last_change":"2024-04-02T04:45:30.245248+0000","last_active":"2024-04-02T04:46:29.432225+0000","last_peered":"2024-04-02T04:46:29.432225+0000","last_clean":"2024-04-02T04:46:29.432225+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:46:29.432225+0000","last_undegraded":"2024-04-02T04:46:29.432225+0000","last_fullsized":"2024-04-02T04:46:29.432225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:30.245093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:30.245093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:32.255063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":129,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:33.251855+0000","last_change":"2024-04-02T04:46:33.251890+0000","last_active":"2024-04-02T04:46:33.251855+0000","last_peered":"2024-04-02T04:46:33.251855+0000","last_clean":"2024-04-02T04:46:33.251855+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:46:33.251855+0000","last_undegraded":"2024-04-02T04:46:33.251855+0000","last_fullsized":"2024-04-02T04:46:33.251855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:46:33.251772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.251772+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.560770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222331+0000","last_change":"2024-04-02T04:45:30.360575+0000","last_active":"2024-04-02T04:46:30.222331+0000","last_peered":"2024-04-02T04:46:30.222331+0000","last_clean":"2024-04-02T04:46:30.222331+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:46:30.222331+0000","last_undegraded":"2024-04-02T04:46:30.222331+0000","last_fullsized":"2024-04-02T04:46:30.222331+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:30.360372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:30.360372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.774203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059056+0000","last_change":"2024-04-02T04:45:53.944039+0000","last_active":"2024-04-02T04:46:29.059056+0000","last_peered":"2024-04-02T04:46:29.059056+0000","last_clean":"2024-04-02T04:46:29.059056+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:46:29.059056+0000","last_undegraded":"2024-04-02T04:46:29.059056+0000","last_fullsized":"2024-04-02T04:46:29.059056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:47.613401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.613401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:48.889382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431379+0000","last_change":"2024-04-02T04:45:28.310099+0000","last_active":"2024-04-02T04:46:29.431379+0000","last_peered":"2024-04-02T04:46:29.431379+0000","last_clean":"2024-04-02T04:46:29.431379+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:46:29.431379+0000","last_undegraded":"2024-04-02T04:46:29.431379+0000","last_fullsized":"2024-04-02T04:46:29.431379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:28.310009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:28.310009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:36.401092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":93,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427255+0000","last_change":"2024-04-02T04:45:15.924734+0000","last_active":"2024-04-02T04:46:29.427255+0000","last_peered":"2024-04-02T04:46:29.427255+0000","last_clean":"2024-04-02T04:46:29.427255+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:46:29.427255+0000","last_undegraded":"2024-04-02T04:46:29.427255+0000","last_fullsized":"2024-04-02T04:46:29.427255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:45:15.924484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:15.924484+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:41.678061+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223761+0000","last_change":"2024-04-02T04:45:44.287628+0000","last_active":"2024-04-02T04:46:30.223761+0000","last_peered":"2024-04-02T04:46:30.223761+0000","last_clean":"2024-04-02T04:46:30.223761+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:46:30.223761+0000","last_undegraded":"2024-04-02T04:46:30.223761+0000","last_fullsized":"2024-04-02T04:46:30.223761+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:44.287411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:44.287411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:02.404385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'36","reported_seq":190,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.433631+0000","last_change":"2024-04-02T04:46:29.368990+0000","last_active":"2024-04-02T04:46:29.433631+0000","last_peered":"2024-04-02T04:46:29.433631+0000","last_clean":"2024-04-02T04:46:29.433631+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:46:29.433631+0000","last_undegraded":"2024-04-02T04:46:29.433631+0000","last_fullsized":"2024-04-02T04:46:29.433631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'36","last_scrub_stamp":"2024-04-02T04:46:29.368745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:29.368745+0000","objects_scrubbed":11,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.702277+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"85'11","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429015+0000","last_change":"2024-04-02T04:46:23.272159+0000","last_active":"2024-04-02T04:46:29.429015+0000","last_peered":"2024-04-02T04:46:29.429015+0000","last_clean":"2024-04-02T04:46:29.429015+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:46:29.429015+0000","last_undegraded":"2024-04-02T04:46:29.429015+0000","last_fullsized":"2024-04-02T04:46:29.429015+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:23.271930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:23.271930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.461094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049026+0000","last_change":"2024-04-02T04:45:26.631108+0000","last_active":"2024-04-02T04:46:29.049026+0000","last_peered":"2024-04-02T04:46:29.049026+0000","last_clean":"2024-04-02T04:46:29.049026+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:46:29.049026+0000","last_undegraded":"2024-04-02T04:46:29.049026+0000","last_fullsized":"2024-04-02T04:46:29.049026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:26.630936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:26.630936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:55.932555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048435+0000","last_change":"2024-04-02T04:45:41.741296+0000","last_active":"2024-04-02T04:46:29.048435+0000","last_peered":"2024-04-02T04:46:29.048435+0000","last_clean":"2024-04-02T04:46:29.048435+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:46:29.048435+0000","last_undegraded":"2024-04-02T04:46:29.048435+0000","last_fullsized":"2024-04-02T04:46:29.048435+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:41.740994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:41.740994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.452513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.433425+0000","last_change":"2024-04-02T04:45:13.293773+0000","last_active":"2024-04-02T04:46:29.433425+0000","last_peered":"2024-04-02T04:46:29.433425+0000","last_clean":"2024-04-02T04:46:29.433425+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:46:29.433425+0000","last_undegraded":"2024-04-02T04:46:29.433425+0000","last_fullsized":"2024-04-02T04:46:29.433425+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:45:13.293650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:13.293650+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:31.693664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":127,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059139+0000","last_change":"2024-04-02T04:45:27.767794+0000","last_active":"2024-04-02T04:46:29.059139+0000","last_peered":"2024-04-02T04:46:29.059139+0000","last_clean":"2024-04-02T04:46:29.059139+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:46:29.059139+0000","last_undegraded":"2024-04-02T04:46:29.059139+0000","last_fullsized":"2024-04-02T04:46:29.059139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:45:27.767550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.767550+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:53.113655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":136,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429517+0000","last_change":"2024-04-02T04:46:25.694818+0000","last_active":"2024-04-02T04:46:29.429517+0000","last_peered":"2024-04-02T04:46:29.429517+0000","last_clean":"2024-04-02T04:46:29.429517+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:46:29.429517+0000","last_undegraded":"2024-04-02T04:46:29.429517+0000","last_fullsized":"2024-04-02T04:46:29.429517+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:46:25.694340+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:28.430989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222801+0000","last_change":"2024-04-02T04:46:22.438361+0000","last_active":"2024-04-02T04:46:30.222801+0000","last_peered":"2024-04-02T04:46:30.222801+0000","last_clean":"2024-04-02T04:46:30.222801+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:46:30.222801+0000","last_undegraded":"2024-04-02T04:46:30.222801+0000","last_fullsized":"2024-04-02T04:46:30.222801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.438115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.444291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":235,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.552855+0000","last_change":"2024-04-02T04:45:21.749332+0000","last_active":"2024-04-02T04:46:36.552855+0000","last_peered":"2024-04-02T04:46:36.552855+0000","last_clean":"2024-04-02T04:46:36.552855+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:46:36.552855+0000","last_undegraded":"2024-04-02T04:46:36.552855+0000","last_fullsized":"2024-04-02T04:46:36.552855+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:45:21.749123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:21.749123+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:40.022070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":141,"num_read_kb":136,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429516+0000","last_change":"2024-04-02T04:46:27.077097+0000","last_active":"2024-04-02T04:46:29.429516+0000","last_peered":"2024-04-02T04:46:29.429516+0000","last_clean":"2024-04-02T04:46:29.429516+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:46:29.429516+0000","last_undegraded":"2024-04-02T04:46:29.429516+0000","last_fullsized":"2024-04-02T04:46:29.429516+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.353345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":106,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059425+0000","last_change":"2024-04-02T04:45:45.620204+0000","last_active":"2024-04-02T04:46:29.059425+0000","last_peered":"2024-04-02T04:46:29.059425+0000","last_clean":"2024-04-02T04:46:29.059425+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:46:29.059425+0000","last_undegraded":"2024-04-02T04:46:29.059425+0000","last_fullsized":"2024-04-02T04:46:29.059425+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:45:45.620076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:45.620076+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:48.466905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049046+0000","last_change":"2024-04-02T04:46:00.849416+0000","last_active":"2024-04-02T04:46:29.049046+0000","last_peered":"2024-04-02T04:46:29.049046+0000","last_clean":"2024-04-02T04:46:29.049046+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:46:29.049046+0000","last_undegraded":"2024-04-02T04:46:29.049046+0000","last_fullsized":"2024-04-02T04:46:29.049046+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.666708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056969+0000","last_change":"2024-04-02T04:45:26.747150+0000","last_active":"2024-04-02T04:46:29.056969+0000","last_peered":"2024-04-02T04:46:29.056969+0000","last_clean":"2024-04-02T04:46:29.056969+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:46:29.056969+0000","last_undegraded":"2024-04-02T04:46:29.056969+0000","last_fullsized":"2024-04-02T04:46:29.056969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:45:26.746930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:45.270160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":126,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.423842+0000","last_change":"2024-04-02T04:45:16.605196+0000","last_active":"2024-04-02T04:46:29.423842+0000","last_peered":"2024-04-02T04:46:29.423842+0000","last_clean":"2024-04-02T04:46:29.423842+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:46:29.423842+0000","last_undegraded":"2024-04-02T04:46:29.423842+0000","last_fullsized":"2024-04-02T04:46:29.423842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:45:16.604968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:16.604968+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:40.262351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222601+0000","last_change":"2024-04-02T04:46:14.464223+0000","last_active":"2024-04-02T04:46:30.222601+0000","last_peered":"2024-04-02T04:46:30.222601+0000","last_clean":"2024-04-02T04:46:30.222601+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:46:30.222601+0000","last_undegraded":"2024-04-02T04:46:30.222601+0000","last_fullsized":"2024-04-02T04:46:30.222601+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:14.463957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.463957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:20.901600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":142,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048739+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:46:29.048739+0000","last_peered":"2024-04-02T04:46:29.048739+0000","last_clean":"2024-04-02T04:46:29.048739+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:46:29.048739+0000","last_undegraded":"2024-04-02T04:46:29.048739+0000","last_fullsized":"2024-04-02T04:46:29.048739+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.759911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":78,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429320+0000","last_change":"2024-04-02T04:45:34.755361+0000","last_active":"2024-04-02T04:46:29.429320+0000","last_peered":"2024-04-02T04:46:29.429320+0000","last_clean":"2024-04-02T04:46:29.429320+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:46:29.429320+0000","last_undegraded":"2024-04-02T04:46:29.429320+0000","last_fullsized":"2024-04-02T04:46:29.429320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:45:34.755051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:40.165503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.228315+0000","last_change":"2024-04-02T04:45:53.850495+0000","last_active":"2024-04-02T04:46:30.228315+0000","last_peered":"2024-04-02T04:46:30.228315+0000","last_clean":"2024-04-02T04:46:30.228315+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:46:30.228315+0000","last_undegraded":"2024-04-02T04:46:30.228315+0000","last_fullsized":"2024-04-02T04:46:30.228315+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:24.290744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:24.290744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:34.966712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":105,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:37.280527+0000","last_change":"2024-04-02T04:46:37.280588+0000","last_active":"2024-04-02T04:46:37.280527+0000","last_peered":"2024-04-02T04:46:37.280527+0000","last_clean":"2024-04-02T04:46:37.280527+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:46:37.280527+0000","last_undegraded":"2024-04-02T04:46:37.280527+0000","last_fullsized":"2024-04-02T04:46:37.280527+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:37.280354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.280354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.558805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":154,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055532+0000","last_change":"2024-04-02T04:46:16.675668+0000","last_active":"2024-04-02T04:46:29.055532+0000","last_peered":"2024-04-02T04:46:29.055532+0000","last_clean":"2024-04-02T04:46:29.055532+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:46:29.055532+0000","last_undegraded":"2024-04-02T04:46:29.055532+0000","last_fullsized":"2024-04-02T04:46:29.055532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:46:16.675527+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.675527+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:25.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050982+0000","last_change":"2024-04-02T04:46:01.856899+0000","last_active":"2024-04-02T04:46:29.050982+0000","last_peered":"2024-04-02T04:46:29.050982+0000","last_clean":"2024-04-02T04:46:29.050982+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:46:29.050982+0000","last_undegraded":"2024-04-02T04:46:29.050982+0000","last_fullsized":"2024-04-02T04:46:29.050982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:01.856656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:01.856656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:07.776394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432144+0000","last_change":"2024-04-02T04:46:09.359995+0000","last_active":"2024-04-02T04:46:29.432144+0000","last_peered":"2024-04-02T04:46:29.432144+0000","last_clean":"2024-04-02T04:46:29.432144+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:46:29.432144+0000","last_undegraded":"2024-04-02T04:46:29.432144+0000","last_fullsized":"2024-04-02T04:46:29.432144+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:09.359739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:09.359739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.174067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":129,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056205+0000","last_change":"2024-04-02T04:46:18.673713+0000","last_active":"2024-04-02T04:46:29.056205+0000","last_peered":"2024-04-02T04:46:29.056205+0000","last_clean":"2024-04-02T04:46:29.056205+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:46:29.056205+0000","last_undegraded":"2024-04-02T04:46:29.056205+0000","last_fullsized":"2024-04-02T04:46:29.056205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.673548+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.102258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049445+0000","last_change":"2024-04-02T04:45:32.675544+0000","last_active":"2024-04-02T04:46:29.049445+0000","last_peered":"2024-04-02T04:46:29.049445+0000","last_clean":"2024-04-02T04:46:29.049445+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:46:29.049445+0000","last_undegraded":"2024-04-02T04:46:29.049445+0000","last_fullsized":"2024-04-02T04:46:29.049445+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:45:32.675301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:51.488983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.524890+0000","last_change":"2024-04-02T04:46:30.524944+0000","last_active":"2024-04-02T04:46:30.524890+0000","last_peered":"2024-04-02T04:46:30.524890+0000","last_clean":"2024-04-02T04:46:30.524890+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:46:30.524890+0000","last_undegraded":"2024-04-02T04:46:30.524890+0000","last_fullsized":"2024-04-02T04:46:30.524890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:30.524741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:30.524741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:37.418739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"85'11","reported_seq":134,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:32.837238+0000","last_change":"2024-04-02T04:46:32.837238+0000","last_active":"2024-04-02T04:46:32.837238+0000","last_peered":"2024-04-02T04:46:32.837238+0000","last_clean":"2024-04-02T04:46:32.837238+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:46:32.837238+0000","last_undegraded":"2024-04-02T04:46:32.837238+0000","last_fullsized":"2024-04-02T04:46:32.837238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:32.836976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.836976+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.523209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222879+0000","last_change":"2024-04-02T04:45:46.305411+0000","last_active":"2024-04-02T04:46:30.222879+0000","last_peered":"2024-04-02T04:46:30.222879+0000","last_clean":"2024-04-02T04:46:30.222879+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:46:30.222879+0000","last_undegraded":"2024-04-02T04:46:30.222879+0000","last_fullsized":"2024-04-02T04:46:30.222879+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:46.305180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:46.305180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.447717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":140,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427547+0000","last_change":"2024-04-02T04:46:26.071505+0000","last_active":"2024-04-02T04:46:29.427547+0000","last_peered":"2024-04-02T04:46:29.427547+0000","last_clean":"2024-04-02T04:46:29.427547+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:46:29.427547+0000","last_undegraded":"2024-04-02T04:46:29.427547+0000","last_fullsized":"2024-04-02T04:46:29.427547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:46:26.071249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.071249+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.710130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222056+0000","last_change":"2024-04-02T04:46:24.365206+0000","last_active":"2024-04-02T04:46:30.222056+0000","last_peered":"2024-04-02T04:46:30.222056+0000","last_clean":"2024-04-02T04:46:30.222056+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:46:30.222056+0000","last_undegraded":"2024-04-02T04:46:30.222056+0000","last_fullsized":"2024-04-02T04:46:30.222056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:24.365001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.365001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:27.300770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.227696+0000","last_change":"2024-04-02T04:45:23.149819+0000","last_active":"2024-04-02T04:46:30.227696+0000","last_peered":"2024-04-02T04:46:30.227696+0000","last_clean":"2024-04-02T04:46:30.227696+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:46:30.227696+0000","last_undegraded":"2024-04-02T04:46:30.227696+0000","last_fullsized":"2024-04-02T04:46:30.227696+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:45:23.149595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:35.359437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.224153+0000","last_change":"2024-04-02T04:45:47.331295+0000","last_active":"2024-04-02T04:46:30.224153+0000","last_peered":"2024-04-02T04:46:30.224153+0000","last_clean":"2024-04-02T04:46:30.224153+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:46:30.224153+0000","last_undegraded":"2024-04-02T04:46:30.224153+0000","last_fullsized":"2024-04-02T04:46:30.224153+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:47.331035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.331035+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.790268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":84,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427163+0000","last_change":"2024-04-02T04:45:31.933201+0000","last_active":"2024-04-02T04:46:29.427163+0000","last_peered":"2024-04-02T04:46:29.427163+0000","last_clean":"2024-04-02T04:46:29.427163+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:46:29.427163+0000","last_undegraded":"2024-04-02T04:46:29.427163+0000","last_fullsized":"2024-04-02T04:46:29.427163+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:31.932911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:31.932911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:48.094398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054503+0000","last_change":"2024-04-02T04:45:58.497534+0000","last_active":"2024-04-02T04:46:29.054503+0000","last_peered":"2024-04-02T04:46:29.054503+0000","last_clean":"2024-04-02T04:46:29.054503+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:46:29.054503+0000","last_undegraded":"2024-04-02T04:46:29.054503+0000","last_fullsized":"2024-04-02T04:46:29.054503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:58.497439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:58.497439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.018970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432910+0000","last_change":"2024-04-02T04:46:16.442450+0000","last_active":"2024-04-02T04:46:29.432910+0000","last_peered":"2024-04-02T04:46:29.432910+0000","last_clean":"2024-04-02T04:46:29.432910+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:46:29.432910+0000","last_undegraded":"2024-04-02T04:46:29.432910+0000","last_fullsized":"2024-04-02T04:46:29.432910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.442234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:22.146814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:37.302423+0000","last_change":"2024-04-02T04:46:37.302510+0000","last_active":"2024-04-02T04:46:37.302423+0000","last_peered":"2024-04-02T04:46:37.302423+0000","last_clean":"2024-04-02T04:46:37.302423+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:46:37.302423+0000","last_undegraded":"2024-04-02T04:46:37.302423+0000","last_fullsized":"2024-04-02T04:46:37.302423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.302252+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.744491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.060336+0000","last_change":"2024-04-02T04:46:18.276638+0000","last_active":"2024-04-02T04:46:29.060336+0000","last_peered":"2024-04-02T04:46:29.060336+0000","last_clean":"2024-04-02T04:46:29.060336+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:46:29.060336+0000","last_undegraded":"2024-04-02T04:46:29.060336+0000","last_fullsized":"2024-04-02T04:46:29.060336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:18.276507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.276507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.306028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":78,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427988+0000","last_change":"2024-04-02T04:45:37.842572+0000","last_active":"2024-04-02T04:46:29.427988+0000","last_peered":"2024-04-02T04:46:29.427988+0000","last_clean":"2024-04-02T04:46:29.427988+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:46:29.427988+0000","last_undegraded":"2024-04-02T04:46:29.427988+0000","last_fullsized":"2024-04-02T04:46:29.427988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:37.842415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:37.842415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:01.493276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:33.616449+0000","last_change":"2024-04-02T04:46:33.616502+0000","last_active":"2024-04-02T04:46:33.616449+0000","last_peered":"2024-04-02T04:46:33.616449+0000","last_clean":"2024-04-02T04:46:33.616449+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:46:33.616449+0000","last_undegraded":"2024-04-02T04:46:33.616449+0000","last_fullsized":"2024-04-02T04:46:33.616449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:33.616300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.616300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.778531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":112,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223884+0000","last_change":"2024-04-02T04:45:42.330243+0000","last_active":"2024-04-02T04:46:30.223884+0000","last_peered":"2024-04-02T04:46:30.223884+0000","last_clean":"2024-04-02T04:46:30.223884+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:46:30.223884+0000","last_undegraded":"2024-04-02T04:46:30.223884+0000","last_fullsized":"2024-04-02T04:46:30.223884+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:45:42.329958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:42.329958+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:42.971250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.425992+0000","last_change":"2024-04-02T04:46:21.245136+0000","last_active":"2024-04-02T04:46:29.425992+0000","last_peered":"2024-04-02T04:46:29.425992+0000","last_clean":"2024-04-02T04:46:29.425992+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:46:29.425992+0000","last_undegraded":"2024-04-02T04:46:29.425992+0000","last_fullsized":"2024-04-02T04:46:29.425992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.244838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.244838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.980660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":136,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743467,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":1004353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103222,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28388,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716764,"statfs":{"total":95995035648,"available":95965966336,"internally_reserved":0,"allocated":1404928,"data_stored":1002166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":412878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28404,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716748,"statfs":{"total":95995035648,"available":95965949952,"internally_reserved":0,"allocated":1421312,"data_stored":1002830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378719,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":413114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051669,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":412179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822731,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":413107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462975,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":413023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:46:42.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:46:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: cluster 2024-04-02T04:46:40.316569+0000 osd.1 (osd.1) 69 : cluster 0 6.4 scrub starts 2024-04-02T04:46:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: cluster 2024-04-02T04:46:40.318298+0000 osd.1 (osd.1) 70 : cluster 0 6.4 scrub ok 2024-04-02T04:46:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: cluster 2024-04-02T04:46:40.634395+0000 osd.2 (osd.2) 39 : cluster 0 4.f scrub starts 2024-04-02T04:46:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: cluster 2024-04-02T04:46:40.636000+0000 osd.2 (osd.2) 40 : cluster 0 4.f scrub ok 2024-04-02T04:46:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.251417+0000 osd.6 (osd.6) 47 : cluster 0 3.10 scrub starts 2024-04-02T04:46:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.254622+0000 osd.6 (osd.6) 48 : cluster 0 3.10 scrub ok 2024-04-02T04:46:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[23790]: audit 2024-04-02T04:46:41.596954+0000 mgr.y (mgr.14538) 101 : audit 0 from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:46:42.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: cluster 2024-04-02T04:46:40.316569+0000 osd.1 (osd.1) 69 : cluster 0 6.4 scrub starts 2024-04-02T04:46:42.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: cluster 2024-04-02T04:46:40.318298+0000 osd.1 (osd.1) 70 : cluster 0 6.4 scrub ok 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: cluster 2024-04-02T04:46:40.634395+0000 osd.2 (osd.2) 39 : cluster 0 4.f scrub starts 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: cluster 2024-04-02T04:46:40.636000+0000 osd.2 (osd.2) 40 : cluster 0 4.f scrub ok 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.251417+0000 osd.6 (osd.6) 47 : cluster 0 3.10 scrub starts 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.254622+0000 osd.6 (osd.6) 48 : cluster 0 3.10 scrub ok 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[21528]: audit 2024-04-02T04:46:41.596954+0000 mgr.y (mgr.14538) 101 : audit 0 from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: cluster 2024-04-02T04:46:40.316569+0000 osd.1 (osd.1) 69 : cluster 0 6.4 scrub starts 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: cluster 2024-04-02T04:46:40.318298+0000 osd.1 (osd.1) 70 : cluster 0 6.4 scrub ok 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: cluster 2024-04-02T04:46:40.634395+0000 osd.2 (osd.2) 39 : cluster 0 4.f scrub starts 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: cluster 2024-04-02T04:46:40.636000+0000 osd.2 (osd.2) 40 : cluster 0 4.f scrub ok 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.251417+0000 osd.6 (osd.6) 47 : cluster 0 3.10 scrub starts 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.254622+0000 osd.6 (osd.6) 48 : cluster 0 3.10 scrub ok 2024-04-02T04:46:42.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:42 smithi069 bash[28904]: audit 2024-04-02T04:46:41.596954+0000 mgr.y (mgr.14538) 101 : audit 0 from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:46:43.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:46:42 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:46:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.321797+0000 osd.5 (osd.5) 59 : cluster 0 3.16 scrub starts 2024-04-02T04:46:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.326142+0000 osd.5 (osd.5) 60 : cluster 0 3.16 scrub ok 2024-04-02T04:46:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.672100+0000 osd.2 (osd.2) 41 : cluster 0 2.c scrub starts 2024-04-02T04:46:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.673444+0000 osd.2 (osd.2) 42 : cluster 0 2.c scrub ok 2024-04-02T04:46:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:41.773481+0000 mgr.y (mgr.14538) 102 : cluster 0 pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:42.287206+0000 osd.7 (osd.7) 63 : cluster 0 6.1c scrub starts 2024-04-02T04:46:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:43 smithi098 bash[23790]: cluster 2024-04-02T04:46:42.290061+0000 osd.7 (osd.7) 64 : cluster 0 6.1c scrub ok 2024-04-02T04:46:43.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.321797+0000 osd.5 (osd.5) 59 : cluster 0 3.16 scrub starts 2024-04-02T04:46:43.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.326142+0000 osd.5 (osd.5) 60 : cluster 0 3.16 scrub ok 2024-04-02T04:46:43.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.672100+0000 osd.2 (osd.2) 41 : cluster 0 2.c scrub starts 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.673444+0000 osd.2 (osd.2) 42 : cluster 0 2.c scrub ok 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:41.773481+0000 mgr.y (mgr.14538) 102 : cluster 0 pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:42.287206+0000 osd.7 (osd.7) 63 : cluster 0 6.1c scrub starts 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[21528]: cluster 2024-04-02T04:46:42.290061+0000 osd.7 (osd.7) 64 : cluster 0 6.1c scrub ok 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.321797+0000 osd.5 (osd.5) 59 : cluster 0 3.16 scrub starts 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.326142+0000 osd.5 (osd.5) 60 : cluster 0 3.16 scrub ok 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.672100+0000 osd.2 (osd.2) 41 : cluster 0 2.c scrub starts 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.673444+0000 osd.2 (osd.2) 42 : cluster 0 2.c scrub ok 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:41.773481+0000 mgr.y (mgr.14538) 102 : cluster 0 pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:42.287206+0000 osd.7 (osd.7) 63 : cluster 0 6.1c scrub starts 2024-04-02T04:46:43.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:43 smithi069 bash[28904]: cluster 2024-04-02T04:46:42.290061+0000 osd.7 (osd.7) 64 : cluster 0 6.1c scrub ok 2024-04-02T04:46:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:44 smithi098 bash[23790]: cluster 2024-04-02T04:46:42.341496+0000 osd.1 (osd.1) 71 : cluster 0 4.7 scrub starts 2024-04-02T04:46:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:44 smithi098 bash[23790]: cluster 2024-04-02T04:46:42.343073+0000 osd.1 (osd.1) 72 : cluster 0 4.7 scrub ok 2024-04-02T04:46:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:44 smithi098 bash[23790]: audit 2024-04-02T04:46:42.836439+0000 mgr.y (mgr.14538) 103 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:44 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.412059+0000 osd.0 (osd.0) 69 : cluster 0 6.0 scrub starts 2024-04-02T04:46:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:44 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.414498+0000 osd.0 (osd.0) 70 : cluster 0 6.0 scrub ok 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[21528]: cluster 2024-04-02T04:46:42.341496+0000 osd.1 (osd.1) 71 : cluster 0 4.7 scrub starts 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[21528]: cluster 2024-04-02T04:46:42.343073+0000 osd.1 (osd.1) 72 : cluster 0 4.7 scrub ok 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[21528]: audit 2024-04-02T04:46:42.836439+0000 mgr.y (mgr.14538) 103 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.412059+0000 osd.0 (osd.0) 69 : cluster 0 6.0 scrub starts 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.414498+0000 osd.0 (osd.0) 70 : cluster 0 6.0 scrub ok 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[28904]: cluster 2024-04-02T04:46:42.341496+0000 osd.1 (osd.1) 71 : cluster 0 4.7 scrub starts 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[28904]: cluster 2024-04-02T04:46:42.343073+0000 osd.1 (osd.1) 72 : cluster 0 4.7 scrub ok 2024-04-02T04:46:44.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[28904]: audit 2024-04-02T04:46:42.836439+0000 mgr.y (mgr.14538) 103 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:44.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.412059+0000 osd.0 (osd.0) 69 : cluster 0 6.0 scrub starts 2024-04-02T04:46:44.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:44 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.414498+0000 osd.0 (osd.0) 70 : cluster 0 6.0 scrub ok 2024-04-02T04:46:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.731222+0000 osd.4 (osd.4) 69 : cluster 0 4.e scrub starts 2024-04-02T04:46:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.734410+0000 osd.4 (osd.4) 70 : cluster 0 4.e scrub ok 2024-04-02T04:46:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.774932+0000 mgr.y (mgr.14538) 104 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.869872+0000 osd.3 (osd.3) 87 : cluster 0 5.0 scrub starts 2024-04-02T04:46:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:43.872995+0000 osd.3 (osd.3) 88 : cluster 0 5.0 scrub ok 2024-04-02T04:46:45.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.175510+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-04-02T04:46:45.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.178314+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-04-02T04:46:45.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.343424+0000 osd.7 (osd.7) 65 : cluster 0 6.5 scrub starts 2024-04-02T04:46:45.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:45 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.346714+0000 osd.7 (osd.7) 66 : cluster 0 6.5 scrub ok 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.731222+0000 osd.4 (osd.4) 69 : cluster 0 4.e scrub starts 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.734410+0000 osd.4 (osd.4) 70 : cluster 0 4.e scrub ok 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.774932+0000 mgr.y (mgr.14538) 104 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.869872+0000 osd.3 (osd.3) 87 : cluster 0 5.0 scrub starts 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:43.872995+0000 osd.3 (osd.3) 88 : cluster 0 5.0 scrub ok 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.175510+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.178314+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.343424+0000 osd.7 (osd.7) 65 : cluster 0 6.5 scrub starts 2024-04-02T04:46:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.346714+0000 osd.7 (osd.7) 66 : cluster 0 6.5 scrub ok 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.731222+0000 osd.4 (osd.4) 69 : cluster 0 4.e scrub starts 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.734410+0000 osd.4 (osd.4) 70 : cluster 0 4.e scrub ok 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.774932+0000 mgr.y (mgr.14538) 104 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.869872+0000 osd.3 (osd.3) 87 : cluster 0 5.0 scrub starts 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:43.872995+0000 osd.3 (osd.3) 88 : cluster 0 5.0 scrub ok 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.175510+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.178314+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.343424+0000 osd.7 (osd.7) 65 : cluster 0 6.5 scrub starts 2024-04-02T04:46:45.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:45 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.346714+0000 osd.7 (osd.7) 66 : cluster 0 6.5 scrub ok 2024-04-02T04:46:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.342361+0000 osd.5 (osd.5) 61 : cluster 0 4.15 scrub starts 2024-04-02T04:46:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.346294+0000 osd.5 (osd.5) 62 : cluster 0 4.15 scrub ok 2024-04-02T04:46:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.731959+0000 osd.2 (osd.2) 43 : cluster 0 2.10 scrub starts 2024-04-02T04:46:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.734498+0000 osd.2 (osd.2) 44 : cluster 0 2.10 scrub ok 2024-04-02T04:46:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.772422+0000 osd.4 (osd.4) 71 : cluster 0 4.1a scrub starts 2024-04-02T04:46:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:44.775059+0000 osd.4 (osd.4) 72 : cluster 0 4.1a scrub ok 2024-04-02T04:46:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:45.392725+0000 osd.0 (osd.0) 71 : cluster 0 3.1f scrub starts 2024-04-02T04:46:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:46 smithi098 bash[23790]: cluster 2024-04-02T04:46:45.396750+0000 osd.0 (osd.0) 72 : cluster 0 3.1f scrub ok 2024-04-02T04:46:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.342361+0000 osd.5 (osd.5) 61 : cluster 0 4.15 scrub starts 2024-04-02T04:46:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.346294+0000 osd.5 (osd.5) 62 : cluster 0 4.15 scrub ok 2024-04-02T04:46:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.731959+0000 osd.2 (osd.2) 43 : cluster 0 2.10 scrub starts 2024-04-02T04:46:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.734498+0000 osd.2 (osd.2) 44 : cluster 0 2.10 scrub ok 2024-04-02T04:46:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.772422+0000 osd.4 (osd.4) 71 : cluster 0 4.1a scrub starts 2024-04-02T04:46:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:44.775059+0000 osd.4 (osd.4) 72 : cluster 0 4.1a scrub ok 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:45.392725+0000 osd.0 (osd.0) 71 : cluster 0 3.1f scrub starts 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[21528]: cluster 2024-04-02T04:46:45.396750+0000 osd.0 (osd.0) 72 : cluster 0 3.1f scrub ok 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.342361+0000 osd.5 (osd.5) 61 : cluster 0 4.15 scrub starts 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.346294+0000 osd.5 (osd.5) 62 : cluster 0 4.15 scrub ok 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.731959+0000 osd.2 (osd.2) 43 : cluster 0 2.10 scrub starts 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.734498+0000 osd.2 (osd.2) 44 : cluster 0 2.10 scrub ok 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.772422+0000 osd.4 (osd.4) 71 : cluster 0 4.1a scrub starts 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:44.775059+0000 osd.4 (osd.4) 72 : cluster 0 4.1a scrub ok 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:45.392725+0000 osd.0 (osd.0) 71 : cluster 0 3.1f scrub starts 2024-04-02T04:46:46.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:46 smithi069 bash[28904]: cluster 2024-04-02T04:46:45.396750+0000 osd.0 (osd.0) 72 : cluster 0 3.1f scrub ok 2024-04-02T04:46:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:47 smithi098 bash[23790]: cluster 2024-04-02T04:46:45.775642+0000 mgr.y (mgr.14538) 105 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:47 smithi098 bash[23790]: cluster 2024-04-02T04:46:45.918881+0000 osd.3 (osd.3) 89 : cluster 0 4.17 deep-scrub starts 2024-04-02T04:46:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:47 smithi098 bash[23790]: cluster 2024-04-02T04:46:45.921153+0000 osd.3 (osd.3) 90 : cluster 0 4.17 deep-scrub ok 2024-04-02T04:46:47.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:47 smithi069 bash[21528]: cluster 2024-04-02T04:46:45.775642+0000 mgr.y (mgr.14538) 105 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:47.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:47 smithi069 bash[21528]: cluster 2024-04-02T04:46:45.918881+0000 osd.3 (osd.3) 89 : cluster 0 4.17 deep-scrub starts 2024-04-02T04:46:47.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:47 smithi069 bash[21528]: cluster 2024-04-02T04:46:45.921153+0000 osd.3 (osd.3) 90 : cluster 0 4.17 deep-scrub ok 2024-04-02T04:46:47.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:47 smithi069 bash[28904]: cluster 2024-04-02T04:46:45.775642+0000 mgr.y (mgr.14538) 105 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:47.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:47 smithi069 bash[28904]: cluster 2024-04-02T04:46:45.918881+0000 osd.3 (osd.3) 89 : cluster 0 4.17 deep-scrub starts 2024-04-02T04:46:47.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:47 smithi069 bash[28904]: cluster 2024-04-02T04:46:45.921153+0000 osd.3 (osd.3) 90 : cluster 0 4.17 deep-scrub ok 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21528]: cluster 2024-04-02T04:46:46.304070+0000 osd.1 (osd.1) 73 : cluster 0 6.1 scrub starts 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21528]: cluster 2024-04-02T04:46:46.306845+0000 osd.1 (osd.1) 74 : cluster 0 6.1 scrub ok 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21528]: cluster 2024-04-02T04:46:46.966431+0000 osd.3 (osd.3) 91 : cluster 0 6.c scrub starts 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21528]: cluster 2024-04-02T04:46:46.968887+0000 osd.3 (osd.3) 92 : cluster 0 6.c scrub ok 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21528]: cluster 2024-04-02T04:46:47.151312+0000 osd.6 (osd.6) 51 : cluster 0 4.1f scrub starts 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21528]: cluster 2024-04-02T04:46:47.154072+0000 osd.6 (osd.6) 52 : cluster 0 4.1f scrub ok 2024-04-02T04:46:48.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[28904]: cluster 2024-04-02T04:46:46.304070+0000 osd.1 (osd.1) 73 : cluster 0 6.1 scrub starts 2024-04-02T04:46:48.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[28904]: cluster 2024-04-02T04:46:46.306845+0000 osd.1 (osd.1) 74 : cluster 0 6.1 scrub ok 2024-04-02T04:46:48.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[28904]: cluster 2024-04-02T04:46:46.966431+0000 osd.3 (osd.3) 91 : cluster 0 6.c scrub starts 2024-04-02T04:46:48.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[28904]: cluster 2024-04-02T04:46:46.968887+0000 osd.3 (osd.3) 92 : cluster 0 6.c scrub ok 2024-04-02T04:46:48.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[28904]: cluster 2024-04-02T04:46:47.151312+0000 osd.6 (osd.6) 51 : cluster 0 4.1f scrub starts 2024-04-02T04:46:48.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[28904]: cluster 2024-04-02T04:46:47.154072+0000 osd.6 (osd.6) 52 : cluster 0 4.1f scrub ok 2024-04-02T04:46:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:48 smithi098 bash[23790]: cluster 2024-04-02T04:46:46.304070+0000 osd.1 (osd.1) 73 : cluster 0 6.1 scrub starts 2024-04-02T04:46:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:48 smithi098 bash[23790]: cluster 2024-04-02T04:46:46.306845+0000 osd.1 (osd.1) 74 : cluster 0 6.1 scrub ok 2024-04-02T04:46:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:48 smithi098 bash[23790]: cluster 2024-04-02T04:46:46.966431+0000 osd.3 (osd.3) 91 : cluster 0 6.c scrub starts 2024-04-02T04:46:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:48 smithi098 bash[23790]: cluster 2024-04-02T04:46:46.968887+0000 osd.3 (osd.3) 92 : cluster 0 6.c scrub ok 2024-04-02T04:46:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:48 smithi098 bash[23790]: cluster 2024-04-02T04:46:47.151312+0000 osd.6 (osd.6) 51 : cluster 0 4.1f scrub starts 2024-04-02T04:46:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:48 smithi098 bash[23790]: cluster 2024-04-02T04:46:47.154072+0000 osd.6 (osd.6) 52 : cluster 0 4.1f scrub ok 2024-04-02T04:46:49.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:46:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:46:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:46:49.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:49 smithi098 bash[23790]: cluster 2024-04-02T04:46:47.397507+0000 osd.5 (osd.5) 63 : cluster 0 6.d scrub starts 2024-04-02T04:46:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:49 smithi098 bash[23790]: cluster 2024-04-02T04:46:47.400057+0000 osd.5 (osd.5) 64 : cluster 0 6.d scrub ok 2024-04-02T04:46:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:49 smithi098 bash[23790]: cluster 2024-04-02T04:46:47.777330+0000 mgr.y (mgr.14538) 106 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:49.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:49 smithi069 bash[28904]: cluster 2024-04-02T04:46:47.397507+0000 osd.5 (osd.5) 63 : cluster 0 6.d scrub starts 2024-04-02T04:46:49.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:49 smithi069 bash[28904]: cluster 2024-04-02T04:46:47.400057+0000 osd.5 (osd.5) 64 : cluster 0 6.d scrub ok 2024-04-02T04:46:49.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:49 smithi069 bash[28904]: cluster 2024-04-02T04:46:47.777330+0000 mgr.y (mgr.14538) 106 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:49.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:49 smithi069 bash[21528]: cluster 2024-04-02T04:46:47.397507+0000 osd.5 (osd.5) 63 : cluster 0 6.d scrub starts 2024-04-02T04:46:49.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:49 smithi069 bash[21528]: cluster 2024-04-02T04:46:47.400057+0000 osd.5 (osd.5) 64 : cluster 0 6.d scrub ok 2024-04-02T04:46:49.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:49 smithi069 bash[21528]: cluster 2024-04-02T04:46:47.777330+0000 mgr.y (mgr.14538) 106 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/3210532090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c10ecd0 msgr2=0x7f7c6c1110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- 172.21.15.69:0/3210532090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c10ecd0 0x7f7c6c1110c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400b0a0 tx=0x7f7c6400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/3210532090 shutdown_connections 2024-04-02T04:46:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- 172.21.15.69:0/3210532090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c10ecd0 0x7f7c6c1110c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- 172.21.15.69:0/3210532090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6c074470 0x7f7c6c074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- 172.21.15.69:0/3210532090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c074d90 0x7f7c6c10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/3210532090 >> 172.21.15.69:0/3210532090 conn(0x7f7c6c079310 msgr2=0x7f7c6c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/3210532090 shutdown_connections 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/3210532090 wait complete. 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 librados: starting msgr at 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 librados: starting objecter 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 Processor -- start 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- start start 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 librados: setting wanted keys 2024-04-02T04:46:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 librados: calling monclient init 2024-04-02T04:46:49.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 0x7f7c6c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:49.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6c074d90 0x7f7c6c19a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:49.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c10ecd0 0x7f7c6c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c113ca0 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c113b20 con 0x7f7c6c074d90 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6c113e20 con 0x7f7c6c074470 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 0x7f7c6c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 0x7f7c6c19a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60398/0 (socket says 172.21.15.69:60398) 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6affd700 1 -- 172.21.15.69:0/2131820245 learned_addr learned my addr 172.21.15.69:0/2131820245 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c10ecd0 0x7f7c6c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c63fff700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6c074d90 0x7f7c6c19a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 -- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 msgr2=0x7f7c6c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 0x7f7c6c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 -- 172.21.15.69:0/2131820245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6c074d90 msgr2=0x7f7c6c19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6c074d90 0x7f7c6c19a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 -- 172.21.15.69:0/2131820245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c19f310 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6affd700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 0x7f7c6c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c6b7fe700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c10ecd0 0x7f7c6c19ebd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400bfd0 tx=0x7f7c64007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c68ff9700 10 client.?.objecter ms_handle_connect 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c68ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64021020 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6c19f5a0 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 10 client.14709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c6c1a6e80 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 librados: init done 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7c64004c50 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64005050 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.730+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c005120 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7c64024020 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5806f5e0 0x7f7c58071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:49.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c6affd700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5806f5e0 0x7f7c58071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:49.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 64..114) ==== 7537+0+0 (secure 0 0 0) 0x7f7c6400f020 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 10 client.14709.objecter ms_dispatch 0x7f7c6c001550 osd_map(114..114 src has 64..114) 2024-04-02T04:46:49.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 3 client.14709.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:49.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 3 client.14709.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:49.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 20 client.14709.objecter dump_active .. 0 homeless 2024-04-02T04:46:49.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c68ff9700 10 client.14709.objecter ms_handle_connect 0x7f7c5806f5e0 2024-04-02T04:46:49.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.734+0000 7f7c6affd700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5806f5e0 0x7f7c58071aa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009ea0 tx=0x7f7c5c009290 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:49.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.738+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c6405c760 con 0x7f7c6c10ecd0 2024-04-02T04:46:49.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.942+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c4c000bc0 con 0x7f7c5806f5e0 2024-04-02T04:46:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c68ff9700 1 -- 172.21.15.69:0/2131820245 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346870 (secure 0 0 0) 0x7f7c4c000bc0 con 0x7f7c5806f5e0 2024-04-02T04:46:49.959 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:46:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 10 librados: watch_flush enter 2024-04-02T04:46:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 10 librados: watch_flush exit 2024-04-02T04:46:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 20 client.14709.objecter shutdown clearing up homeless session... 2024-04-02T04:46:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 10 client.14709.objecter successfully canceled tick 2024-04-02T04:46:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5806f5e0 msgr2=0x7f7c58071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5806f5e0 0x7f7c58071aa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009ea0 tx=0x7f7c5c009290 comp rx=0 tx=0).stop 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c10ecd0 msgr2=0x7f7c6c19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c10ecd0 0x7f7c6c19ebd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400bfd0 tx=0x7f7c64007a20 comp rx=0 tx=0).stop 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 shutdown_connections 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5806f5e0 0x7f7c58071aa0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c6c10ecd0 0x7f7c6c19ebd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c6c074470 0x7f7c6c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 --2- 172.21.15.69:0/2131820245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c6c074d90 0x7f7c6c19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 >> 172.21.15.69:0/2131820245 conn(0x7f7c6c079310 msgr2=0x7f7c6c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 shutdown_connections 2024-04-02T04:46:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.958+0000 7f7c71332700 1 -- 172.21.15.69:0/2131820245 wait complete. 2024-04-02T04:46:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:49.962+0000 7f7c71332700 1 librados: shutdown 2024-04-02T04:46:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[21528]: cluster 2024-04-02T04:46:48.352426+0000 osd.5 (osd.5) 65 : cluster 0 2.12 scrub starts 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[21528]: cluster 2024-04-02T04:46:48.354858+0000 osd.5 (osd.5) 66 : cluster 0 2.12 scrub ok 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[21528]: cluster 2024-04-02T04:46:49.349764+0000 osd.0 (osd.0) 73 : cluster 0 3.6 scrub starts 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[21528]: cluster 2024-04-02T04:46:49.354282+0000 osd.0 (osd.0) 74 : cluster 0 3.6 scrub ok 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[28904]: cluster 2024-04-02T04:46:48.352426+0000 osd.5 (osd.5) 65 : cluster 0 2.12 scrub starts 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[28904]: cluster 2024-04-02T04:46:48.354858+0000 osd.5 (osd.5) 66 : cluster 0 2.12 scrub ok 2024-04-02T04:46:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[28904]: cluster 2024-04-02T04:46:49.349764+0000 osd.0 (osd.0) 73 : cluster 0 3.6 scrub starts 2024-04-02T04:46:50.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:50 smithi069 bash[28904]: cluster 2024-04-02T04:46:49.354282+0000 osd.0 (osd.0) 74 : cluster 0 3.6 scrub ok 2024-04-02T04:46:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:50 smithi098 bash[23790]: cluster 2024-04-02T04:46:48.352426+0000 osd.5 (osd.5) 65 : cluster 0 2.12 scrub starts 2024-04-02T04:46:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:50 smithi098 bash[23790]: cluster 2024-04-02T04:46:48.354858+0000 osd.5 (osd.5) 66 : cluster 0 2.12 scrub ok 2024-04-02T04:46:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:50 smithi098 bash[23790]: cluster 2024-04-02T04:46:49.349764+0000 osd.0 (osd.0) 73 : cluster 0 3.6 scrub starts 2024-04-02T04:46:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:50 smithi098 bash[23790]: cluster 2024-04-02T04:46:49.354282+0000 osd.0 (osd.0) 74 : cluster 0 3.6 scrub ok 2024-04-02T04:46:50.668 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-02T04:46:49.777586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1389,"num_read_kb":1203,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228476,"kb_used_data":8188,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749732740,"statfs":{"total":767960285184,"available":767726325760,"internally_reserved":0,"allocated":8384512,"data_stored":5073650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006327"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055945+0000","last_change":"2024-04-02T04:45:29.804443+0000","last_active":"2024-04-02T04:46:29.055945+0000","last_peered":"2024-04-02T04:46:29.055945+0000","last_clean":"2024-04-02T04:46:29.055945+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:46:29.055945+0000","last_undegraded":"2024-04-02T04:46:29.055945+0000","last_fullsized":"2024-04-02T04:46:29.055945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:29.804217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:29.804217+0000","last_clean_scrub_stamp":"2024-04-02T04:45:29.804217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:55.947849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058715+0000","last_change":"2024-04-02T04:46:26.737065+0000","last_active":"2024-04-02T04:46:29.058715+0000","last_peered":"2024-04-02T04:46:29.058715+0000","last_clean":"2024-04-02T04:46:29.058715+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:46:29.058715+0000","last_undegraded":"2024-04-02T04:46:29.058715+0000","last_fullsized":"2024-04-02T04:46:29.058715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:26.736961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.736961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.907303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057857+0000","last_change":"2024-04-02T04:46:21.290286+0000","last_active":"2024-04-02T04:46:29.057857+0000","last_peered":"2024-04-02T04:46:29.057857+0000","last_clean":"2024-04-02T04:46:29.057857+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:46:29.057857+0000","last_undegraded":"2024-04-02T04:46:29.057857+0000","last_fullsized":"2024-04-02T04:46:29.057857+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.290194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.290194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.622357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.861417+0000","last_change":"2024-04-02T04:46:39.861495+0000","last_active":"2024-04-02T04:46:39.861417+0000","last_peered":"2024-04-02T04:46:39.861417+0000","last_clean":"2024-04-02T04:46:39.861417+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:46:39.861417+0000","last_undegraded":"2024-04-02T04:46:39.861417+0000","last_fullsized":"2024-04-02T04:46:39.861417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:39.861258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.861258+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.553184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048868+0000","last_change":"2024-04-02T04:45:56.829369+0000","last_active":"2024-04-02T04:46:29.048868+0000","last_peered":"2024-04-02T04:46:29.048868+0000","last_clean":"2024-04-02T04:46:29.048868+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:46:29.048868+0000","last_undegraded":"2024-04-02T04:46:29.048868+0000","last_fullsized":"2024-04-02T04:46:29.048868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:56.829149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:56.829149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:11.291730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048580+0000","last_change":"2024-04-02T04:45:39.713804+0000","last_active":"2024-04-02T04:46:29.048580+0000","last_peered":"2024-04-02T04:46:29.048580+0000","last_clean":"2024-04-02T04:46:29.048580+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:46:29.048580+0000","last_undegraded":"2024-04-02T04:46:29.048580+0000","last_fullsized":"2024-04-02T04:46:29.048580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:39.713597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:39.713597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:05.446000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058249+0000","last_change":"2024-04-02T04:46:13.710516+0000","last_active":"2024-04-02T04:46:29.058249+0000","last_peered":"2024-04-02T04:46:29.058249+0000","last_clean":"2024-04-02T04:46:29.058249+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:46:29.058249+0000","last_undegraded":"2024-04-02T04:46:29.058249+0000","last_fullsized":"2024-04-02T04:46:29.058249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:13.710340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:13.710340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.036057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":127,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:45.396915+0000","last_change":"2024-04-02T04:46:45.396999+0000","last_active":"2024-04-02T04:46:45.396915+0000","last_peered":"2024-04-02T04:46:45.396915+0000","last_clean":"2024-04-02T04:46:45.396915+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:46:45.396915+0000","last_undegraded":"2024-04-02T04:46:45.396915+0000","last_fullsized":"2024-04-02T04:46:45.396915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:45.396760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.396760+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.943901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.343851+0000","last_change":"2024-04-02T04:46:39.343919+0000","last_active":"2024-04-02T04:46:39.343851+0000","last_peered":"2024-04-02T04:46:39.343851+0000","last_clean":"2024-04-02T04:46:39.343851+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:46:39.343851+0000","last_undegraded":"2024-04-02T04:46:39.343851+0000","last_fullsized":"2024-04-02T04:46:39.343851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:39.343592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.343592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.312506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049495+0000","last_change":"2024-04-02T04:46:22.812121+0000","last_active":"2024-04-02T04:46:29.049495+0000","last_peered":"2024-04-02T04:46:29.049495+0000","last_clean":"2024-04-02T04:46:29.049495+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:46:29.049495+0000","last_undegraded":"2024-04-02T04:46:29.049495+0000","last_fullsized":"2024-04-02T04:46:29.049495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.811836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.811836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.362979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:35.292487+0000","last_change":"2024-04-02T04:46:35.292548+0000","last_active":"2024-04-02T04:46:35.292487+0000","last_peered":"2024-04-02T04:46:35.292487+0000","last_clean":"2024-04-02T04:46:35.292487+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:46:35.292487+0000","last_undegraded":"2024-04-02T04:46:35.292487+0000","last_fullsized":"2024-04-02T04:46:35.292487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.292336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.292336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.961526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.428915+0000","last_change":"2024-04-02T04:45:40.828366+0000","last_active":"2024-04-02T04:46:29.428915+0000","last_peered":"2024-04-02T04:46:29.428915+0000","last_clean":"2024-04-02T04:46:29.428915+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:46:29.428915+0000","last_undegraded":"2024-04-02T04:46:29.428915+0000","last_fullsized":"2024-04-02T04:46:29.428915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:45:40.828122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:40.828122+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:59.185115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":88,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054279+0000","last_change":"2024-04-02T04:46:27.234204+0000","last_active":"2024-04-02T04:46:29.054279+0000","last_peered":"2024-04-02T04:46:29.054279+0000","last_clean":"2024-04-02T04:46:29.054279+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:46:29.054279+0000","last_undegraded":"2024-04-02T04:46:29.054279+0000","last_fullsized":"2024-04-02T04:46:29.054279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.234006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.234006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.363415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:44.775227+0000","last_change":"2024-04-02T04:46:44.775289+0000","last_active":"2024-04-02T04:46:44.775227+0000","last_peered":"2024-04-02T04:46:44.775227+0000","last_clean":"2024-04-02T04:46:44.775227+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:46:44.775227+0000","last_undegraded":"2024-04-02T04:46:44.775227+0000","last_fullsized":"2024-04-02T04:46:44.775227+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.775071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.207519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223471+0000","last_change":"2024-04-02T04:45:35.338383+0000","last_active":"2024-04-02T04:46:30.223471+0000","last_peered":"2024-04-02T04:46:30.223471+0000","last_clean":"2024-04-02T04:46:30.223471+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:46:30.223471+0000","last_undegraded":"2024-04-02T04:46:30.223471+0000","last_fullsized":"2024-04-02T04:46:30.223471+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:45:35.338035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.338035+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:55.625980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.428924+0000","last_change":"2024-04-02T04:45:27.901815+0000","last_active":"2024-04-02T04:46:29.428924+0000","last_peered":"2024-04-02T04:46:29.428924+0000","last_clean":"2024-04-02T04:46:29.428924+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:46:29.428924+0000","last_undegraded":"2024-04-02T04:46:29.428924+0000","last_fullsized":"2024-04-02T04:46:29.428924+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:45:27.901655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.901655+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:44.054719+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059371+0000","last_change":"2024-04-02T04:45:47.692378+0000","last_active":"2024-04-02T04:46:29.059371+0000","last_peered":"2024-04-02T04:46:29.059371+0000","last_clean":"2024-04-02T04:46:29.059371+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:46:29.059371+0000","last_undegraded":"2024-04-02T04:46:29.059371+0000","last_fullsized":"2024-04-02T04:46:29.059371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:47.691992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.691992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:51.815637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.942301+0000","last_change":"2024-04-02T04:46:36.942358+0000","last_active":"2024-04-02T04:46:36.942301+0000","last_peered":"2024-04-02T04:46:36.942301+0000","last_clean":"2024-04-02T04:46:36.942301+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:46:36.942301+0000","last_undegraded":"2024-04-02T04:46:36.942301+0000","last_fullsized":"2024-04-02T04:46:36.942301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.942139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.942139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.526854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056204+0000","last_change":"2024-04-02T04:45:57.720821+0000","last_active":"2024-04-02T04:46:29.056204+0000","last_peered":"2024-04-02T04:46:29.056204+0000","last_clean":"2024-04-02T04:46:29.056204+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:46:29.056204+0000","last_undegraded":"2024-04-02T04:46:29.056204+0000","last_fullsized":"2024-04-02T04:46:29.056204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:57.720720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.720720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:17.288812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"85'23","reported_seq":137,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048946+0000","last_change":"2024-04-02T04:46:21.850698+0000","last_active":"2024-04-02T04:46:29.048946+0000","last_peered":"2024-04-02T04:46:29.048946+0000","last_clean":"2024-04-02T04:46:29.048946+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:46:29.048946+0000","last_undegraded":"2024-04-02T04:46:29.048946+0000","last_fullsized":"2024-04-02T04:46:29.048946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-02T04:46:21.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.850373+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:35.100552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050836+0000","last_change":"2024-04-02T04:45:55.851523+0000","last_active":"2024-04-02T04:46:29.050836+0000","last_peered":"2024-04-02T04:46:29.050836+0000","last_clean":"2024-04-02T04:46:29.050836+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:46:29.050836+0000","last_undegraded":"2024-04-02T04:46:29.050836+0000","last_fullsized":"2024-04-02T04:46:29.050836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:55.851292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:55.851292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:58.449590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:32.598626+0000","last_change":"2024-04-02T04:46:32.598626+0000","last_active":"2024-04-02T04:46:32.598626+0000","last_peered":"2024-04-02T04:46:32.598626+0000","last_clean":"2024-04-02T04:46:32.598626+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:46:32.598626+0000","last_undegraded":"2024-04-02T04:46:32.598626+0000","last_fullsized":"2024-04-02T04:46:32.598626+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.598421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.598421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.059173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427426+0000","last_change":"2024-04-02T04:46:21.009651+0000","last_active":"2024-04-02T04:46:29.427426+0000","last_peered":"2024-04-02T04:46:29.427426+0000","last_clean":"2024-04-02T04:46:29.427426+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:46:29.427426+0000","last_undegraded":"2024-04-02T04:46:29.427426+0000","last_fullsized":"2024-04-02T04:46:29.427426+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.009424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.009424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.532492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059949+0000","last_change":"2024-04-02T04:45:57.518634+0000","last_active":"2024-04-02T04:46:29.059949+0000","last_peered":"2024-04-02T04:46:29.059949+0000","last_clean":"2024-04-02T04:46:29.059949+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:46:29.059949+0000","last_undegraded":"2024-04-02T04:46:29.059949+0000","last_fullsized":"2024-04-02T04:46:29.059949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:45:57.518350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.518350+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:12.947455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432435+0000","last_change":"2024-04-02T04:45:22.352334+0000","last_active":"2024-04-02T04:46:29.432435+0000","last_peered":"2024-04-02T04:46:29.432435+0000","last_clean":"2024-04-02T04:46:29.432435+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:46:29.432435+0000","last_undegraded":"2024-04-02T04:46:29.432435+0000","last_fullsized":"2024-04-02T04:46:29.432435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:22.352122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:22.352122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:47.681591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":140,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:47.154243+0000","last_change":"2024-04-02T04:46:47.154319+0000","last_active":"2024-04-02T04:46:47.154243+0000","last_peered":"2024-04-02T04:46:47.154243+0000","last_clean":"2024-04-02T04:46:47.154243+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:46:47.154243+0000","last_undegraded":"2024-04-02T04:46:47.154243+0000","last_fullsized":"2024-04-02T04:46:47.154243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:47.154082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.154082+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.920192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":102,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.060543+0000","last_change":"2024-04-02T04:46:27.777917+0000","last_active":"2024-04-02T04:46:29.060543+0000","last_peered":"2024-04-02T04:46:29.060543+0000","last_clean":"2024-04-02T04:46:29.060543+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:46:29.060543+0000","last_undegraded":"2024-04-02T04:46:29.060543+0000","last_fullsized":"2024-04-02T04:46:29.060543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:46:27.777679+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.777679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.754491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":124,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054971+0000","last_change":"2024-04-02T04:46:24.726007+0000","last_active":"2024-04-02T04:46:29.054971+0000","last_peered":"2024-04-02T04:46:29.054971+0000","last_clean":"2024-04-02T04:46:29.054971+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:46:29.054971+0000","last_undegraded":"2024-04-02T04:46:29.054971+0000","last_fullsized":"2024-04-02T04:46:29.054971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:24.725734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.725734+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.190873+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":89,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:42.290255+0000","last_change":"2024-04-02T04:46:42.290326+0000","last_active":"2024-04-02T04:46:42.290255+0000","last_peered":"2024-04-02T04:46:42.290255+0000","last_clean":"2024-04-02T04:46:42.290255+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:46:42.290255+0000","last_undegraded":"2024-04-02T04:46:42.290255+0000","last_fullsized":"2024-04-02T04:46:42.290255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:46:42.290071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.290071+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.053573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059292+0000","last_change":"2024-04-02T04:46:04.466247+0000","last_active":"2024-04-02T04:46:29.059292+0000","last_peered":"2024-04-02T04:46:29.059292+0000","last_clean":"2024-04-02T04:46:29.059292+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:46:29.059292+0000","last_undegraded":"2024-04-02T04:46:29.059292+0000","last_fullsized":"2024-04-02T04:46:29.059292+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:04.465964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:04.465964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.060111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.426191+0000","last_change":"2024-04-02T04:46:16.163503+0000","last_active":"2024-04-02T04:46:29.426191+0000","last_peered":"2024-04-02T04:46:29.426191+0000","last_clean":"2024-04-02T04:46:29.426191+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:46:29.426191+0000","last_undegraded":"2024-04-02T04:46:29.426191+0000","last_fullsized":"2024-04-02T04:46:29.426191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.163286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.163286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:21.454867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":160,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431917+0000","last_change":"2024-04-02T04:45:23.312146+0000","last_active":"2024-04-02T04:46:29.431917+0000","last_peered":"2024-04-02T04:46:29.431917+0000","last_clean":"2024-04-02T04:46:29.431917+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:46:29.431917+0000","last_undegraded":"2024-04-02T04:46:29.431917+0000","last_fullsized":"2024-04-02T04:46:29.431917+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:45:23.311862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:23.311862+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:53.017930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057066+0000","last_change":"2024-04-02T04:45:35.692872+0000","last_active":"2024-04-02T04:46:29.057066+0000","last_peered":"2024-04-02T04:46:29.057066+0000","last_clean":"2024-04-02T04:46:29.057066+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:46:29.057066+0000","last_undegraded":"2024-04-02T04:46:29.057066+0000","last_fullsized":"2024-04-02T04:46:29.057066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:35.692628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.692628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:54.735437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":84,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.061009+0000","last_change":"2024-04-02T04:45:38.761482+0000","last_active":"2024-04-02T04:46:29.061009+0000","last_peered":"2024-04-02T04:46:29.061009+0000","last_clean":"2024-04-02T04:46:29.061009+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:46:29.061009+0000","last_undegraded":"2024-04-02T04:46:29.061009+0000","last_fullsized":"2024-04-02T04:46:29.061009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:38.761200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:38.761200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:54.439691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429773+0000","last_change":"2024-04-02T04:46:15.989347+0000","last_active":"2024-04-02T04:46:29.429773+0000","last_peered":"2024-04-02T04:46:29.429773+0000","last_clean":"2024-04-02T04:46:29.429773+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:46:29.429773+0000","last_undegraded":"2024-04-02T04:46:29.429773+0000","last_fullsized":"2024-04-02T04:46:29.429773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:15.989126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:46:15.989126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.092291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":134,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:41.326315+0000","last_change":"2024-04-02T04:46:41.326411+0000","last_active":"2024-04-02T04:46:41.326315+0000","last_peered":"2024-04-02T04:46:41.326315+0000","last_clean":"2024-04-02T04:46:41.326315+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:46:41.326315+0000","last_undegraded":"2024-04-02T04:46:41.326315+0000","last_fullsized":"2024-04-02T04:46:41.326315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:46:41.326158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.326158+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.505788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222483+0000","last_change":"2024-04-02T04:45:53.375887+0000","last_active":"2024-04-02T04:46:30.222483+0000","last_peered":"2024-04-02T04:46:30.222483+0000","last_clean":"2024-04-02T04:46:30.222483+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:46:30.222483+0000","last_undegraded":"2024-04-02T04:46:30.222483+0000","last_fullsized":"2024-04-02T04:46:30.222483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:45:53.375671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:03.905102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055260+0000","last_change":"2024-04-02T04:46:10.724569+0000","last_active":"2024-04-02T04:46:29.055260+0000","last_peered":"2024-04-02T04:46:29.055260+0000","last_clean":"2024-04-02T04:46:29.055260+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:46:29.055260+0000","last_undegraded":"2024-04-02T04:46:29.055260+0000","last_fullsized":"2024-04-02T04:46:29.055260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.724363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.724363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:18.641739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.285798+0000","last_change":"2024-04-02T04:46:36.285859+0000","last_active":"2024-04-02T04:46:36.285798+0000","last_peered":"2024-04-02T04:46:36.285798+0000","last_clean":"2024-04-02T04:46:36.285798+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:46:36.285798+0000","last_undegraded":"2024-04-02T04:46:36.285798+0000","last_fullsized":"2024-04-02T04:46:36.285798+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.285639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.484787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.057448+0000","last_change":"2024-04-02T04:45:43.604563+0000","last_active":"2024-04-02T04:46:29.057448+0000","last_peered":"2024-04-02T04:46:29.057448+0000","last_clean":"2024-04-02T04:46:29.057448+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:46:29.057448+0000","last_undegraded":"2024-04-02T04:46:29.057448+0000","last_fullsized":"2024-04-02T04:46:29.057448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:45:43.604325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:43.604325+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:03.047811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":88,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:38.892954+0000","last_change":"2024-04-02T04:46:38.893014+0000","last_active":"2024-04-02T04:46:38.892954+0000","last_peered":"2024-04-02T04:46:38.892954+0000","last_clean":"2024-04-02T04:46:38.892954+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:46:38.892954+0000","last_undegraded":"2024-04-02T04:46:38.892954+0000","last_fullsized":"2024-04-02T04:46:38.892954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.892773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.892773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.901685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049082+0000","last_change":"2024-04-02T04:46:10.895794+0000","last_active":"2024-04-02T04:46:29.049082+0000","last_peered":"2024-04-02T04:46:29.049082+0000","last_clean":"2024-04-02T04:46:29.049082+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:46:29.049082+0000","last_undegraded":"2024-04-02T04:46:29.049082+0000","last_fullsized":"2024-04-02T04:46:29.049082+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.895579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.895579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.482340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.433107+0000","last_change":"2024-04-02T04:45:16.372742+0000","last_active":"2024-04-02T04:46:29.433107+0000","last_peered":"2024-04-02T04:46:29.433107+0000","last_clean":"2024-04-02T04:46:29.433107+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:46:29.433107+0000","last_undegraded":"2024-04-02T04:46:29.433107+0000","last_fullsized":"2024-04-02T04:46:29.433107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:16.372527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:16.372527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:41.922002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":134,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:37.808297+0000","last_change":"2024-04-02T04:46:37.808452+0000","last_active":"2024-04-02T04:46:37.808297+0000","last_peered":"2024-04-02T04:46:37.808297+0000","last_clean":"2024-04-02T04:46:37.808297+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:46:37.808297+0000","last_undegraded":"2024-04-02T04:46:37.808297+0000","last_fullsized":"2024-04-02T04:46:37.808297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:46:37.808090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.808090+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.632664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054319+0000","last_change":"2024-04-02T04:45:59.476847+0000","last_active":"2024-04-02T04:46:29.054319+0000","last_peered":"2024-04-02T04:46:29.054319+0000","last_clean":"2024-04-02T04:46:29.054319+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:46:29.054319+0000","last_undegraded":"2024-04-02T04:46:29.054319+0000","last_fullsized":"2024-04-02T04:46:29.054319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:59.476642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:59.476642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:16.549126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431859+0000","last_change":"2024-04-02T04:45:26.322458+0000","last_active":"2024-04-02T04:46:29.431859+0000","last_peered":"2024-04-02T04:46:29.431859+0000","last_clean":"2024-04-02T04:46:29.431859+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:46:29.431859+0000","last_undegraded":"2024-04-02T04:46:29.431859+0000","last_fullsized":"2024-04-02T04:46:29.431859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:26.322114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:26.322114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:36.598502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:34.223544+0000","last_change":"2024-04-02T04:46:34.223691+0000","last_active":"2024-04-02T04:46:34.223544+0000","last_peered":"2024-04-02T04:46:34.223544+0000","last_clean":"2024-04-02T04:46:34.223544+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:46:34.223544+0000","last_undegraded":"2024-04-02T04:46:34.223544+0000","last_fullsized":"2024-04-02T04:46:34.223544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:34.223346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:34.223346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.085223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.224172+0000","last_change":"2024-04-02T04:46:27.353748+0000","last_active":"2024-04-02T04:46:30.224172+0000","last_peered":"2024-04-02T04:46:30.224172+0000","last_clean":"2024-04-02T04:46:30.224172+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:46:30.224172+0000","last_undegraded":"2024-04-02T04:46:30.224172+0000","last_fullsized":"2024-04-02T04:46:30.224172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:27.353503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.353503+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.633132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050841+0000","last_change":"2024-04-02T04:45:50.840290+0000","last_active":"2024-04-02T04:46:29.050841+0000","last_peered":"2024-04-02T04:46:29.050841+0000","last_clean":"2024-04-02T04:46:29.050841+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:46:29.050841+0000","last_undegraded":"2024-04-02T04:46:29.050841+0000","last_fullsized":"2024-04-02T04:46:29.050841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:50.839972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:50.839972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:01.685776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":138,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:44.346692+0000","last_change":"2024-04-02T04:46:44.346692+0000","last_active":"2024-04-02T04:46:44.346692+0000","last_peered":"2024-04-02T04:46:44.346692+0000","last_clean":"2024-04-02T04:46:44.346692+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:46:44.346692+0000","last_undegraded":"2024-04-02T04:46:44.346692+0000","last_fullsized":"2024-04-02T04:46:44.346692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:44.346318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.092878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.244665+0000","last_change":"2024-04-02T04:46:36.244725+0000","last_active":"2024-04-02T04:46:36.244665+0000","last_peered":"2024-04-02T04:46:36.244665+0000","last_clean":"2024-04-02T04:46:36.244665+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:46:36.244665+0000","last_undegraded":"2024-04-02T04:46:36.244665+0000","last_fullsized":"2024-04-02T04:46:36.244665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.244514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.244514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.017937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":112,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055883+0000","last_change":"2024-04-02T04:46:28.203298+0000","last_active":"2024-04-02T04:46:29.055883+0000","last_peered":"2024-04-02T04:46:29.055883+0000","last_clean":"2024-04-02T04:46:29.055883+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:46:29.055883+0000","last_undegraded":"2024-04-02T04:46:29.055883+0000","last_fullsized":"2024-04-02T04:46:29.055883+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:46:28.203156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.203156+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.777551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058203+0000","last_change":"2024-04-02T04:46:01.454913+0000","last_active":"2024-04-02T04:46:29.058203+0000","last_peered":"2024-04-02T04:46:29.058203+0000","last_clean":"2024-04-02T04:46:29.058203+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:46:29.058203+0000","last_undegraded":"2024-04-02T04:46:29.058203+0000","last_fullsized":"2024-04-02T04:46:29.058203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:46:01.454803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:09.150138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"85'11","reported_seq":137,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.437098+0000","last_change":"2024-04-02T04:46:28.775353+0000","last_active":"2024-04-02T04:46:39.437098+0000","last_peered":"2024-04-02T04:46:39.437098+0000","last_clean":"2024-04-02T04:46:39.437098+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:46:39.437098+0000","last_undegraded":"2024-04-02T04:46:39.437098+0000","last_fullsized":"2024-04-02T04:46:39.437098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:28.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.775071+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:55.875528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427268+0000","last_change":"2024-04-02T04:45:18.948742+0000","last_active":"2024-04-02T04:46:29.427268+0000","last_peered":"2024-04-02T04:46:29.427268+0000","last_clean":"2024-04-02T04:46:29.427268+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:46:29.427268+0000","last_undegraded":"2024-04-02T04:46:29.427268+0000","last_fullsized":"2024-04-02T04:46:29.427268+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:18.948514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:18.948514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:43.158274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:33.347205+0000","last_change":"2024-04-02T04:46:33.347283+0000","last_active":"2024-04-02T04:46:33.347205+0000","last_peered":"2024-04-02T04:46:33.347205+0000","last_clean":"2024-04-02T04:46:33.347205+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:46:33.347205+0000","last_undegraded":"2024-04-02T04:46:33.347205+0000","last_fullsized":"2024-04-02T04:46:33.347205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:33.347053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.347053+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.542332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223274+0000","last_change":"2024-04-02T04:45:31.347266+0000","last_active":"2024-04-02T04:46:30.223274+0000","last_peered":"2024-04-02T04:46:30.223274+0000","last_clean":"2024-04-02T04:46:30.223274+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:46:30.223274+0000","last_undegraded":"2024-04-02T04:46:30.223274+0000","last_fullsized":"2024-04-02T04:46:30.223274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:31.347039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:31.347039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.504167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:45.921334+0000","last_change":"2024-04-02T04:46:45.921390+0000","last_active":"2024-04-02T04:46:45.921334+0000","last_peered":"2024-04-02T04:46:45.921334+0000","last_clean":"2024-04-02T04:46:45.921334+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:46:45.921334+0000","last_undegraded":"2024-04-02T04:46:45.921334+0000","last_fullsized":"2024-04-02T04:46:45.921334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.369948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.426783+0000","last_change":"2024-04-02T04:46:28.039798+0000","last_active":"2024-04-02T04:46:29.426783+0000","last_peered":"2024-04-02T04:46:29.426783+0000","last_clean":"2024-04-02T04:46:29.426783+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:46:29.426783+0000","last_undegraded":"2024-04-02T04:46:29.426783+0000","last_fullsized":"2024-04-02T04:46:29.426783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:28.039566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.039566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.163582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":102,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:41.254794+0000","last_change":"2024-04-02T04:46:41.254878+0000","last_active":"2024-04-02T04:46:41.254794+0000","last_peered":"2024-04-02T04:46:41.254794+0000","last_clean":"2024-04-02T04:46:41.254794+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:46:41.254794+0000","last_undegraded":"2024-04-02T04:46:41.254794+0000","last_fullsized":"2024-04-02T04:46:41.254794+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:46:41.254633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.254633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.695806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:38.622455+0000","last_change":"2024-04-02T04:46:38.622583+0000","last_active":"2024-04-02T04:46:38.622455+0000","last_peered":"2024-04-02T04:46:38.622455+0000","last_clean":"2024-04-02T04:46:38.622455+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:46:38.622455+0000","last_undegraded":"2024-04-02T04:46:38.622455+0000","last_fullsized":"2024-04-02T04:46:38.622455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.622220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.622220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.379107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":84,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058073+0000","last_change":"2024-04-02T04:45:27.573088+0000","last_active":"2024-04-02T04:46:29.058073+0000","last_peered":"2024-04-02T04:46:29.058073+0000","last_clean":"2024-04-02T04:46:29.058073+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:46:29.058073+0000","last_undegraded":"2024-04-02T04:46:29.058073+0000","last_fullsized":"2024-04-02T04:46:29.058073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:27.572981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.572981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:54.743792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:44.734683+0000","last_change":"2024-04-02T04:46:44.734741+0000","last_active":"2024-04-02T04:46:44.734683+0000","last_peered":"2024-04-02T04:46:44.734683+0000","last_clean":"2024-04-02T04:46:44.734683+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:46:44.734683+0000","last_undegraded":"2024-04-02T04:46:44.734683+0000","last_fullsized":"2024-04-02T04:46:44.734683+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.734508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.734508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.434405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222640+0000","last_change":"2024-04-02T04:45:36.359126+0000","last_active":"2024-04-02T04:46:30.222640+0000","last_peered":"2024-04-02T04:46:30.222640+0000","last_clean":"2024-04-02T04:46:30.222640+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:46:30.222640+0000","last_undegraded":"2024-04-02T04:46:30.222640+0000","last_fullsized":"2024-04-02T04:46:30.222640+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:36.358885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:36.358885+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:47.833512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056780+0000","last_change":"2024-04-02T04:45:31.758736+0000","last_active":"2024-04-02T04:46:29.056780+0000","last_peered":"2024-04-02T04:46:29.056780+0000","last_clean":"2024-04-02T04:46:29.056780+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:46:29.056780+0000","last_undegraded":"2024-04-02T04:46:29.056780+0000","last_fullsized":"2024-04-02T04:46:29.056780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:31.758488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:31.758488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.166829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050530+0000","last_change":"2024-04-02T04:46:02.906156+0000","last_active":"2024-04-02T04:46:29.050530+0000","last_peered":"2024-04-02T04:46:29.050530+0000","last_clean":"2024-04-02T04:46:29.050530+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:46:29.050530+0000","last_undegraded":"2024-04-02T04:46:29.050530+0000","last_fullsized":"2024-04-02T04:46:29.050530+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:46:02.905900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:02.905900+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.976578+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":126,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223855+0000","last_change":"2024-04-02T04:46:19.421011+0000","last_active":"2024-04-02T04:46:30.223855+0000","last_peered":"2024-04-02T04:46:30.223855+0000","last_clean":"2024-04-02T04:46:30.223855+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:46:30.223855+0000","last_undegraded":"2024-04-02T04:46:30.223855+0000","last_fullsized":"2024-04-02T04:46:30.223855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:19.420739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:19.420739+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.952287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"85'11","reported_seq":136,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.432195+0000","last_change":"2024-04-02T04:46:27.787700+0000","last_active":"2024-04-02T04:46:39.432195+0000","last_peered":"2024-04-02T04:46:39.432195+0000","last_clean":"2024-04-02T04:46:39.432195+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:46:39.432195+0000","last_undegraded":"2024-04-02T04:46:39.432195+0000","last_fullsized":"2024-04-02T04:46:39.432195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:27.787353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.787353+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:41.371457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":88,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:43.414661+0000","last_change":"2024-04-02T04:46:43.414718+0000","last_active":"2024-04-02T04:46:43.414661+0000","last_peered":"2024-04-02T04:46:43.414661+0000","last_clean":"2024-04-02T04:46:43.414661+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:46:43.414661+0000","last_undegraded":"2024-04-02T04:46:43.414661+0000","last_fullsized":"2024-04-02T04:46:43.414661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:43.414511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.414511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:43.745696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":107,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432666+0000","last_change":"2024-04-02T04:46:22.411099+0000","last_active":"2024-04-02T04:46:29.432666+0000","last_peered":"2024-04-02T04:46:29.432666+0000","last_clean":"2024-04-02T04:46:29.432666+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:46:29.432666+0000","last_undegraded":"2024-04-02T04:46:29.432666+0000","last_fullsized":"2024-04-02T04:46:29.432666+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:22.411010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.411010+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:32.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":151,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.432264+0000","last_change":"2024-04-02T04:46:14.106481+0000","last_active":"2024-04-02T04:46:39.432264+0000","last_peered":"2024-04-02T04:46:39.432264+0000","last_clean":"2024-04-02T04:46:39.432264+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:46:39.432264+0000","last_undegraded":"2024-04-02T04:46:39.432264+0000","last_fullsized":"2024-04-02T04:46:39.432264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:14.106232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.106232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.525489+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.317696+0000","last_change":"2024-04-02T04:46:36.317753+0000","last_active":"2024-04-02T04:46:36.317696+0000","last_peered":"2024-04-02T04:46:36.317696+0000","last_clean":"2024-04-02T04:46:36.317696+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:46:36.317696+0000","last_undegraded":"2024-04-02T04:46:36.317696+0000","last_fullsized":"2024-04-02T04:46:36.317696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.317542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.317542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.283515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":82,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.058401+0000","last_change":"2024-04-02T04:45:41.708071+0000","last_active":"2024-04-02T04:46:29.058401+0000","last_peered":"2024-04-02T04:46:29.058401+0000","last_clean":"2024-04-02T04:46:29.058401+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:46:29.058401+0000","last_undegraded":"2024-04-02T04:46:29.058401+0000","last_fullsized":"2024-04-02T04:46:29.058401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:45:41.707741+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:50.082235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427333+0000","last_change":"2024-04-02T04:45:38.855217+0000","last_active":"2024-04-02T04:46:29.427333+0000","last_peered":"2024-04-02T04:46:29.427333+0000","last_clean":"2024-04-02T04:46:29.427333+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:46:29.427333+0000","last_undegraded":"2024-04-02T04:46:29.427333+0000","last_fullsized":"2024-04-02T04:46:29.427333+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:38.855005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:38.855005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:53.569786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:36.775337+0000","last_change":"2024-04-02T04:46:36.775425+0000","last_active":"2024-04-02T04:46:36.775337+0000","last_peered":"2024-04-02T04:46:36.775337+0000","last_clean":"2024-04-02T04:46:36.775337+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:46:36.775337+0000","last_undegraded":"2024-04-02T04:46:36.775337+0000","last_fullsized":"2024-04-02T04:46:36.775337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:46:36.775178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.775178+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.278965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:32.335622+0000","last_change":"2024-04-02T04:46:32.335647+0000","last_active":"2024-04-02T04:46:32.335622+0000","last_peered":"2024-04-02T04:46:32.335622+0000","last_clean":"2024-04-02T04:46:32.335622+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:46:32.335622+0000","last_undegraded":"2024-04-02T04:46:32.335622+0000","last_fullsized":"2024-04-02T04:46:32.335622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.335539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.335539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.500350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:44.346919+0000","last_change":"2024-04-02T04:46:44.347022+0000","last_active":"2024-04-02T04:46:44.346919+0000","last_peered":"2024-04-02T04:46:44.346919+0000","last_clean":"2024-04-02T04:46:44.346919+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:46:44.346919+0000","last_undegraded":"2024-04-02T04:46:44.346919+0000","last_fullsized":"2024-04-02T04:46:44.346919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.346726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:53.151319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.431510+0000","last_change":"2024-04-02T04:45:15.357170+0000","last_active":"2024-04-02T04:46:29.431510+0000","last_peered":"2024-04-02T04:46:29.431510+0000","last_clean":"2024-04-02T04:46:29.431510+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:46:29.431510+0000","last_undegraded":"2024-04-02T04:46:29.431510+0000","last_fullsized":"2024-04-02T04:46:29.431510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:45:15.356917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:15.356917+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:45.114912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.227474+0000","last_change":"2024-04-02T04:46:07.378595+0000","last_active":"2024-04-02T04:46:30.227474+0000","last_peered":"2024-04-02T04:46:30.227474+0000","last_clean":"2024-04-02T04:46:30.227474+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:46:30.227474+0000","last_undegraded":"2024-04-02T04:46:30.227474+0000","last_fullsized":"2024-04-02T04:46:30.227474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:07.378519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:46:07.378519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:12.389433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:42.343198+0000","last_change":"2024-04-02T04:46:42.343240+0000","last_active":"2024-04-02T04:46:42.343198+0000","last_peered":"2024-04-02T04:46:42.343198+0000","last_clean":"2024-04-02T04:46:42.343198+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:46:42.343198+0000","last_undegraded":"2024-04-02T04:46:42.343198+0000","last_fullsized":"2024-04-02T04:46:42.343198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:42.343078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.343078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.356250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:35.276656+0000","last_change":"2024-04-02T04:46:35.276714+0000","last_active":"2024-04-02T04:46:35.276656+0000","last_peered":"2024-04-02T04:46:35.276656+0000","last_clean":"2024-04-02T04:46:35.276656+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:46:35.276656+0000","last_undegraded":"2024-04-02T04:46:35.276656+0000","last_fullsized":"2024-04-02T04:46:35.276656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.276502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.276502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.541614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055955+0000","last_change":"2024-04-02T04:45:45.679903+0000","last_active":"2024-04-02T04:46:29.055955+0000","last_peered":"2024-04-02T04:46:29.055955+0000","last_clean":"2024-04-02T04:46:29.055955+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:46:29.055955+0000","last_undegraded":"2024-04-02T04:46:29.055955+0000","last_fullsized":"2024-04-02T04:46:29.055955+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:45:45.679774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:45.679774+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:45.816952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427650+0000","last_change":"2024-04-02T04:45:35.792402+0000","last_active":"2024-04-02T04:46:29.427650+0000","last_peered":"2024-04-02T04:46:29.427650+0000","last_clean":"2024-04-02T04:46:29.427650+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:46:29.427650+0000","last_undegraded":"2024-04-02T04:46:29.427650+0000","last_fullsized":"2024-04-02T04:46:29.427650+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:35.792264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.792264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:50.312208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":356,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223025+0000","last_change":"2024-04-02T04:46:17.433834+0000","last_active":"2024-04-02T04:46:30.223025+0000","last_peered":"2024-04-02T04:46:30.223025+0000","last_clean":"2024-04-02T04:46:30.223025+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:46:30.223025+0000","last_undegraded":"2024-04-02T04:46:30.223025+0000","last_fullsized":"2024-04-02T04:46:30.223025+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:46:17.433563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:46:17.433563+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.028333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429295+0000","last_change":"2024-04-02T04:46:10.058200+0000","last_active":"2024-04-02T04:46:29.429295+0000","last_peered":"2024-04-02T04:46:29.429295+0000","last_clean":"2024-04-02T04:46:29.429295+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:46:29.429295+0000","last_undegraded":"2024-04-02T04:46:29.429295+0000","last_fullsized":"2024-04-02T04:46:29.429295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.057875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.057875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.984328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:40.318414+0000","last_change":"2024-04-02T04:46:40.318480+0000","last_active":"2024-04-02T04:46:40.318414+0000","last_peered":"2024-04-02T04:46:40.318414+0000","last_clean":"2024-04-02T04:46:40.318414+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:46:40.318414+0000","last_undegraded":"2024-04-02T04:46:40.318414+0000","last_fullsized":"2024-04-02T04:46:40.318414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.318304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.318304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.789228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":129,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:33.251855+0000","last_change":"2024-04-02T04:46:33.251890+0000","last_active":"2024-04-02T04:46:33.251855+0000","last_peered":"2024-04-02T04:46:33.251855+0000","last_clean":"2024-04-02T04:46:33.251855+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:46:33.251855+0000","last_undegraded":"2024-04-02T04:46:33.251855+0000","last_fullsized":"2024-04-02T04:46:33.251855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:46:33.251772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.251772+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.560770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222331+0000","last_change":"2024-04-02T04:45:30.360575+0000","last_active":"2024-04-02T04:46:30.222331+0000","last_peered":"2024-04-02T04:46:30.222331+0000","last_clean":"2024-04-02T04:46:30.222331+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:46:30.222331+0000","last_undegraded":"2024-04-02T04:46:30.222331+0000","last_fullsized":"2024-04-02T04:46:30.222331+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:30.360372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:30.360372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.774203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059056+0000","last_change":"2024-04-02T04:45:53.944039+0000","last_active":"2024-04-02T04:46:29.059056+0000","last_peered":"2024-04-02T04:46:29.059056+0000","last_clean":"2024-04-02T04:46:29.059056+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:46:29.059056+0000","last_undegraded":"2024-04-02T04:46:29.059056+0000","last_fullsized":"2024-04-02T04:46:29.059056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:47.613401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.613401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:48.889382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:46.307142+0000","last_change":"2024-04-02T04:46:46.307232+0000","last_active":"2024-04-02T04:46:46.307142+0000","last_peered":"2024-04-02T04:46:46.307142+0000","last_clean":"2024-04-02T04:46:46.307142+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:46:46.307142+0000","last_undegraded":"2024-04-02T04:46:46.307142+0000","last_fullsized":"2024-04-02T04:46:46.307142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.306862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.306862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.499577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":101,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:44.178501+0000","last_change":"2024-04-02T04:46:44.178572+0000","last_active":"2024-04-02T04:46:44.178501+0000","last_peered":"2024-04-02T04:46:44.178501+0000","last_clean":"2024-04-02T04:46:44.178501+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:46:44.178501+0000","last_undegraded":"2024-04-02T04:46:44.178501+0000","last_fullsized":"2024-04-02T04:46:44.178501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:46:44.178327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.178327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.305669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223761+0000","last_change":"2024-04-02T04:45:44.287628+0000","last_active":"2024-04-02T04:46:30.223761+0000","last_peered":"2024-04-02T04:46:30.223761+0000","last_clean":"2024-04-02T04:46:30.223761+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:46:30.223761+0000","last_undegraded":"2024-04-02T04:46:30.223761+0000","last_fullsized":"2024-04-02T04:46:30.223761+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:44.287411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:44.287411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:02.404385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'36","reported_seq":192,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.437231+0000","last_change":"2024-04-02T04:46:29.368990+0000","last_active":"2024-04-02T04:46:39.437231+0000","last_peered":"2024-04-02T04:46:39.437231+0000","last_clean":"2024-04-02T04:46:39.437231+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:46:39.437231+0000","last_undegraded":"2024-04-02T04:46:39.437231+0000","last_fullsized":"2024-04-02T04:46:39.437231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'36","last_scrub_stamp":"2024-04-02T04:46:29.368745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:29.368745+0000","objects_scrubbed":11,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.702277+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"85'11","reported_seq":137,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.436810+0000","last_change":"2024-04-02T04:46:23.272159+0000","last_active":"2024-04-02T04:46:39.436810+0000","last_peered":"2024-04-02T04:46:39.436810+0000","last_clean":"2024-04-02T04:46:39.436810+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:46:39.436810+0000","last_undegraded":"2024-04-02T04:46:39.436810+0000","last_fullsized":"2024-04-02T04:46:39.436810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:23.271930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:23.271930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.461094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049026+0000","last_change":"2024-04-02T04:45:26.631108+0000","last_active":"2024-04-02T04:46:29.049026+0000","last_peered":"2024-04-02T04:46:29.049026+0000","last_clean":"2024-04-02T04:46:29.049026+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:46:29.049026+0000","last_undegraded":"2024-04-02T04:46:29.049026+0000","last_fullsized":"2024-04-02T04:46:29.049026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:26.630936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:26.630936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:55.932555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048435+0000","last_change":"2024-04-02T04:45:41.741296+0000","last_active":"2024-04-02T04:46:29.048435+0000","last_peered":"2024-04-02T04:46:29.048435+0000","last_clean":"2024-04-02T04:46:29.048435+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:46:29.048435+0000","last_undegraded":"2024-04-02T04:46:29.048435+0000","last_fullsized":"2024-04-02T04:46:29.048435+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:41.740994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:41.740994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.452513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.351300+0000","last_change":"2024-04-02T04:46:39.351367+0000","last_active":"2024-04-02T04:46:39.351300+0000","last_peered":"2024-04-02T04:46:39.351300+0000","last_clean":"2024-04-02T04:46:39.351300+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:46:39.351300+0000","last_undegraded":"2024-04-02T04:46:39.351300+0000","last_fullsized":"2024-04-02T04:46:39.351300+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:39.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.351151+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.995435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":127,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059139+0000","last_change":"2024-04-02T04:45:27.767794+0000","last_active":"2024-04-02T04:46:29.059139+0000","last_peered":"2024-04-02T04:46:29.059139+0000","last_clean":"2024-04-02T04:46:29.059139+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:46:29.059139+0000","last_undegraded":"2024-04-02T04:46:29.059139+0000","last_fullsized":"2024-04-02T04:46:29.059139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:45:27.767550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:27.767550+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:53.113655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":138,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.437256+0000","last_change":"2024-04-02T04:46:25.694818+0000","last_active":"2024-04-02T04:46:39.437256+0000","last_peered":"2024-04-02T04:46:39.437256+0000","last_clean":"2024-04-02T04:46:39.437256+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:46:39.437256+0000","last_undegraded":"2024-04-02T04:46:39.437256+0000","last_fullsized":"2024-04-02T04:46:39.437256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:46:25.694340+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:28.430989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222801+0000","last_change":"2024-04-02T04:46:22.438361+0000","last_active":"2024-04-02T04:46:30.222801+0000","last_peered":"2024-04-02T04:46:30.222801+0000","last_clean":"2024-04-02T04:46:30.222801+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:46:30.222801+0000","last_undegraded":"2024-04-02T04:46:30.222801+0000","last_fullsized":"2024-04-02T04:46:30.222801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.438115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.444291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":253,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:46.575958+0000","last_change":"2024-04-02T04:46:43.873401+0000","last_active":"2024-04-02T04:46:46.575958+0000","last_peered":"2024-04-02T04:46:46.575958+0000","last_clean":"2024-04-02T04:46:46.575958+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:46:46.575958+0000","last_undegraded":"2024-04-02T04:46:46.575958+0000","last_fullsized":"2024-04-02T04:46:46.575958+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:46:43.873013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.873013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:08.482316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":151,"num_read_kb":146,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.429516+0000","last_change":"2024-04-02T04:46:27.077097+0000","last_active":"2024-04-02T04:46:29.429516+0000","last_peered":"2024-04-02T04:46:29.429516+0000","last_clean":"2024-04-02T04:46:29.429516+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:46:29.429516+0000","last_undegraded":"2024-04-02T04:46:29.429516+0000","last_fullsized":"2024-04-02T04:46:29.429516+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.353345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":106,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.059425+0000","last_change":"2024-04-02T04:45:45.620204+0000","last_active":"2024-04-02T04:46:29.059425+0000","last_peered":"2024-04-02T04:46:29.059425+0000","last_clean":"2024-04-02T04:46:29.059425+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:46:29.059425+0000","last_undegraded":"2024-04-02T04:46:29.059425+0000","last_fullsized":"2024-04-02T04:46:29.059425+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:45:45.620076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:45.620076+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:48.466905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049046+0000","last_change":"2024-04-02T04:46:00.849416+0000","last_active":"2024-04-02T04:46:29.049046+0000","last_peered":"2024-04-02T04:46:29.049046+0000","last_clean":"2024-04-02T04:46:29.049046+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:46:29.049046+0000","last_undegraded":"2024-04-02T04:46:29.049046+0000","last_fullsized":"2024-04-02T04:46:29.049046+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.666708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":88,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:46.969077+0000","last_change":"2024-04-02T04:46:46.969136+0000","last_active":"2024-04-02T04:46:46.969077+0000","last_peered":"2024-04-02T04:46:46.969077+0000","last_clean":"2024-04-02T04:46:46.969077+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:46:46.969077+0000","last_undegraded":"2024-04-02T04:46:46.969077+0000","last_fullsized":"2024-04-02T04:46:46.969077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.968921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.968921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.377963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":136,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:43.734784+0000","last_change":"2024-04-02T04:46:43.734784+0000","last_active":"2024-04-02T04:46:43.734784+0000","last_peered":"2024-04-02T04:46:43.734784+0000","last_clean":"2024-04-02T04:46:43.734784+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:46:43.734784+0000","last_undegraded":"2024-04-02T04:46:43.734784+0000","last_fullsized":"2024-04-02T04:46:43.734784+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:43.734423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.734423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:07.446921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222601+0000","last_change":"2024-04-02T04:46:14.464223+0000","last_active":"2024-04-02T04:46:30.222601+0000","last_peered":"2024-04-02T04:46:30.222601+0000","last_clean":"2024-04-02T04:46:30.222601+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:46:30.222601+0000","last_undegraded":"2024-04-02T04:46:30.222601+0000","last_fullsized":"2024-04-02T04:46:30.222601+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:14.463957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.463957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:20.901600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":142,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.048739+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:46:29.048739+0000","last_peered":"2024-04-02T04:46:29.048739+0000","last_clean":"2024-04-02T04:46:29.048739+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:46:29.048739+0000","last_undegraded":"2024-04-02T04:46:29.048739+0000","last_fullsized":"2024-04-02T04:46:29.048739+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.759911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:47.400234+0000","last_change":"2024-04-02T04:46:47.400298+0000","last_active":"2024-04-02T04:46:47.400234+0000","last_peered":"2024-04-02T04:46:47.400234+0000","last_clean":"2024-04-02T04:46:47.400234+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:46:47.400234+0000","last_undegraded":"2024-04-02T04:46:47.400234+0000","last_fullsized":"2024-04-02T04:46:47.400234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:47.400068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.400068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.923320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:40.636097+0000","last_change":"2024-04-02T04:46:40.636137+0000","last_active":"2024-04-02T04:46:40.636097+0000","last_peered":"2024-04-02T04:46:40.636097+0000","last_clean":"2024-04-02T04:46:40.636097+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:46:40.636097+0000","last_undegraded":"2024-04-02T04:46:40.636097+0000","last_fullsized":"2024-04-02T04:46:40.636097+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.636006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.636006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:54.838665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":105,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:37.280527+0000","last_change":"2024-04-02T04:46:37.280588+0000","last_active":"2024-04-02T04:46:37.280527+0000","last_peered":"2024-04-02T04:46:37.280527+0000","last_clean":"2024-04-02T04:46:37.280527+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:46:37.280527+0000","last_undegraded":"2024-04-02T04:46:37.280527+0000","last_fullsized":"2024-04-02T04:46:37.280527+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:37.280354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.280354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.558805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":154,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.055532+0000","last_change":"2024-04-02T04:46:16.675668+0000","last_active":"2024-04-02T04:46:29.055532+0000","last_peered":"2024-04-02T04:46:29.055532+0000","last_clean":"2024-04-02T04:46:29.055532+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:46:29.055532+0000","last_undegraded":"2024-04-02T04:46:29.055532+0000","last_fullsized":"2024-04-02T04:46:29.055532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:46:16.675527+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.675527+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:25.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":79,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.050982+0000","last_change":"2024-04-02T04:46:01.856899+0000","last_active":"2024-04-02T04:46:29.050982+0000","last_peered":"2024-04-02T04:46:29.050982+0000","last_clean":"2024-04-02T04:46:29.050982+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:46:29.050982+0000","last_undegraded":"2024-04-02T04:46:29.050982+0000","last_fullsized":"2024-04-02T04:46:29.050982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:01.856656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:01.856656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:07.776394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432144+0000","last_change":"2024-04-02T04:46:09.359995+0000","last_active":"2024-04-02T04:46:29.432144+0000","last_peered":"2024-04-02T04:46:29.432144+0000","last_clean":"2024-04-02T04:46:29.432144+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:46:29.432144+0000","last_undegraded":"2024-04-02T04:46:29.432144+0000","last_fullsized":"2024-04-02T04:46:29.432144+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:09.359739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:09.359739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.174067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":129,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.056205+0000","last_change":"2024-04-02T04:46:18.673713+0000","last_active":"2024-04-02T04:46:29.056205+0000","last_peered":"2024-04-02T04:46:29.056205+0000","last_clean":"2024-04-02T04:46:29.056205+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:46:29.056205+0000","last_undegraded":"2024-04-02T04:46:29.056205+0000","last_fullsized":"2024-04-02T04:46:29.056205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.673548+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.102258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.049445+0000","last_change":"2024-04-02T04:45:32.675544+0000","last_active":"2024-04-02T04:46:29.049445+0000","last_peered":"2024-04-02T04:46:29.049445+0000","last_clean":"2024-04-02T04:46:29.049445+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:46:29.049445+0000","last_undegraded":"2024-04-02T04:46:29.049445+0000","last_fullsized":"2024-04-02T04:46:29.049445+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:45:32.675301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:51.488983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.524890+0000","last_change":"2024-04-02T04:46:30.524944+0000","last_active":"2024-04-02T04:46:30.524890+0000","last_peered":"2024-04-02T04:46:30.524890+0000","last_clean":"2024-04-02T04:46:30.524890+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:46:30.524890+0000","last_undegraded":"2024-04-02T04:46:30.524890+0000","last_fullsized":"2024-04-02T04:46:30.524890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:30.524741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:30.524741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:37.418739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"85'11","reported_seq":136,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:39.432175+0000","last_change":"2024-04-02T04:46:32.837238+0000","last_active":"2024-04-02T04:46:39.432175+0000","last_peered":"2024-04-02T04:46:39.432175+0000","last_clean":"2024-04-02T04:46:39.432175+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:46:39.432175+0000","last_undegraded":"2024-04-02T04:46:39.432175+0000","last_fullsized":"2024-04-02T04:46:39.432175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:32.836976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.836976+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.523209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":91,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222879+0000","last_change":"2024-04-02T04:45:46.305411+0000","last_active":"2024-04-02T04:46:30.222879+0000","last_peered":"2024-04-02T04:46:30.222879+0000","last_clean":"2024-04-02T04:46:30.222879+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:46:30.222879+0000","last_undegraded":"2024-04-02T04:46:30.222879+0000","last_fullsized":"2024-04-02T04:46:30.222879+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:46.305180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:46.305180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:49.447717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":140,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427547+0000","last_change":"2024-04-02T04:46:26.071505+0000","last_active":"2024-04-02T04:46:29.427547+0000","last_peered":"2024-04-02T04:46:29.427547+0000","last_clean":"2024-04-02T04:46:29.427547+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:46:29.427547+0000","last_undegraded":"2024-04-02T04:46:29.427547+0000","last_fullsized":"2024-04-02T04:46:29.427547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:46:26.071249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.071249+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.710130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":87,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.222056+0000","last_change":"2024-04-02T04:46:24.365206+0000","last_active":"2024-04-02T04:46:30.222056+0000","last_peered":"2024-04-02T04:46:30.222056+0000","last_clean":"2024-04-02T04:46:30.222056+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:46:30.222056+0000","last_undegraded":"2024-04-02T04:46:30.222056+0000","last_fullsized":"2024-04-02T04:46:30.222056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:24.365001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.365001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:27.300770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:41.673503+0000","last_change":"2024-04-02T04:46:41.673523+0000","last_active":"2024-04-02T04:46:41.673503+0000","last_peered":"2024-04-02T04:46:41.673503+0000","last_clean":"2024-04-02T04:46:41.673503+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:46:41.673503+0000","last_undegraded":"2024-04-02T04:46:41.673503+0000","last_fullsized":"2024-04-02T04:46:41.673503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:41.673448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.673448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.552075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.224153+0000","last_change":"2024-04-02T04:45:47.331295+0000","last_active":"2024-04-02T04:46:30.224153+0000","last_peered":"2024-04-02T04:46:30.224153+0000","last_clean":"2024-04-02T04:46:30.224153+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:46:30.224153+0000","last_undegraded":"2024-04-02T04:46:30.224153+0000","last_fullsized":"2024-04-02T04:46:30.224153+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:47.331035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.331035+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.790268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":84,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427163+0000","last_change":"2024-04-02T04:45:31.933201+0000","last_active":"2024-04-02T04:46:29.427163+0000","last_peered":"2024-04-02T04:46:29.427163+0000","last_clean":"2024-04-02T04:46:29.427163+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:46:29.427163+0000","last_undegraded":"2024-04-02T04:46:29.427163+0000","last_fullsized":"2024-04-02T04:46:29.427163+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:31.932911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:45:31.932911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:48.094398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":80,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.054503+0000","last_change":"2024-04-02T04:45:58.497534+0000","last_active":"2024-04-02T04:46:29.054503+0000","last_peered":"2024-04-02T04:46:29.054503+0000","last_clean":"2024-04-02T04:46:29.054503+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:46:29.054503+0000","last_undegraded":"2024-04-02T04:46:29.054503+0000","last_fullsized":"2024-04-02T04:46:29.054503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:58.497439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:58.497439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.018970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":99,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.432910+0000","last_change":"2024-04-02T04:46:16.442450+0000","last_active":"2024-04-02T04:46:29.432910+0000","last_peered":"2024-04-02T04:46:29.432910+0000","last_clean":"2024-04-02T04:46:29.432910+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:46:29.432910+0000","last_undegraded":"2024-04-02T04:46:29.432910+0000","last_fullsized":"2024-04-02T04:46:29.432910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.442234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:22.146814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:37.302423+0000","last_change":"2024-04-02T04:46:37.302510+0000","last_active":"2024-04-02T04:46:37.302423+0000","last_peered":"2024-04-02T04:46:37.302423+0000","last_clean":"2024-04-02T04:46:37.302423+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:46:37.302423+0000","last_undegraded":"2024-04-02T04:46:37.302423+0000","last_fullsized":"2024-04-02T04:46:37.302423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.302252+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.744491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":92,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.060336+0000","last_change":"2024-04-02T04:46:18.276638+0000","last_active":"2024-04-02T04:46:29.060336+0000","last_peered":"2024-04-02T04:46:29.060336+0000","last_clean":"2024-04-02T04:46:29.060336+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:46:29.060336+0000","last_undegraded":"2024-04-02T04:46:29.060336+0000","last_fullsized":"2024-04-02T04:46:29.060336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:18.276507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.276507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.306028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":78,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.427988+0000","last_change":"2024-04-02T04:45:37.842572+0000","last_active":"2024-04-02T04:46:29.427988+0000","last_peered":"2024-04-02T04:46:29.427988+0000","last_clean":"2024-04-02T04:46:29.427988+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:46:29.427988+0000","last_undegraded":"2024-04-02T04:46:29.427988+0000","last_fullsized":"2024-04-02T04:46:29.427988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:37.842415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:37.842415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:01.493276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":98,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:33.616449+0000","last_change":"2024-04-02T04:46:33.616502+0000","last_active":"2024-04-02T04:46:33.616449+0000","last_peered":"2024-04-02T04:46:33.616449+0000","last_clean":"2024-04-02T04:46:33.616449+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:46:33.616449+0000","last_undegraded":"2024-04-02T04:46:33.616449+0000","last_fullsized":"2024-04-02T04:46:33.616449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:33.616300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.616300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.778531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":112,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:30.223884+0000","last_change":"2024-04-02T04:45:42.330243+0000","last_active":"2024-04-02T04:46:30.223884+0000","last_peered":"2024-04-02T04:46:30.223884+0000","last_clean":"2024-04-02T04:46:30.223884+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:46:30.223884+0000","last_undegraded":"2024-04-02T04:46:30.223884+0000","last_fullsized":"2024-04-02T04:46:30.223884+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:45:42.329958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:42.329958+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:46:42.971250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":90,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-02T04:46:29.425992+0000","last_change":"2024-04-02T04:46:21.245136+0000","last_active":"2024-04-02T04:46:29.425992+0000","last_peered":"2024-04-02T04:46:29.425992+0000","last_clean":"2024-04-02T04:46:29.425992+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:46:29.425992+0000","last_undegraded":"2024-04-02T04:46:29.425992+0000","last_fullsized":"2024-04-02T04:46:29.425992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.244838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.244838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.980660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":151,"num_read_kb":146,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743469,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":1004353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103224,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28388,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716764,"statfs":{"total":95995035648,"available":95965966336,"internally_reserved":0,"allocated":1404928,"data_stored":1002166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":412878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28404,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716748,"statfs":{"total":95995035648,"available":95965949952,"internally_reserved":0,"allocated":1421312,"data_stored":1002830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378721,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":413114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051671,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":412179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822733,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":413107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462977,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":413023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:46:50.673 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-02T04:46:50.673 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T04:46:50.673 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-02T04:46:50.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-04-02T04:46:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:51 smithi098 bash[23790]: cluster 2024-04-02T04:46:49.777779+0000 mgr.y (mgr.14538) 107 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:51 smithi098 bash[23790]: cluster 2024-04-02T04:46:49.852322+0000 osd.4 (osd.4) 73 : cluster 0 5.1 scrub starts 2024-04-02T04:46:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:51 smithi098 bash[23790]: cluster 2024-04-02T04:46:49.854609+0000 osd.4 (osd.4) 74 : cluster 0 5.1 scrub ok 2024-04-02T04:46:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:51 smithi098 bash[23790]: audit 2024-04-02T04:46:49.944540+0000 mgr.y (mgr.14538) 108 : audit 0 from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:46:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:51 smithi098 bash[23790]: cluster 2024-04-02T04:46:50.397924+0000 osd.0 (osd.0) 75 : cluster 0 4.6 deep-scrub starts 2024-04-02T04:46:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:51 smithi098 bash[23790]: cluster 2024-04-02T04:46:50.400380+0000 osd.0 (osd.0) 76 : cluster 0 4.6 deep-scrub ok 2024-04-02T04:46:51.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[21528]: cluster 2024-04-02T04:46:49.777779+0000 mgr.y (mgr.14538) 107 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:51.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[21528]: cluster 2024-04-02T04:46:49.852322+0000 osd.4 (osd.4) 73 : cluster 0 5.1 scrub starts 2024-04-02T04:46:51.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[21528]: cluster 2024-04-02T04:46:49.854609+0000 osd.4 (osd.4) 74 : cluster 0 5.1 scrub ok 2024-04-02T04:46:51.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[21528]: audit 2024-04-02T04:46:49.944540+0000 mgr.y (mgr.14538) 108 : audit 0 from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:46:51.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[21528]: cluster 2024-04-02T04:46:50.397924+0000 osd.0 (osd.0) 75 : cluster 0 4.6 deep-scrub starts 2024-04-02T04:46:51.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[21528]: cluster 2024-04-02T04:46:50.400380+0000 osd.0 (osd.0) 76 : cluster 0 4.6 deep-scrub ok 2024-04-02T04:46:51.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[28904]: cluster 2024-04-02T04:46:49.777779+0000 mgr.y (mgr.14538) 107 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:51.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[28904]: cluster 2024-04-02T04:46:49.852322+0000 osd.4 (osd.4) 73 : cluster 0 5.1 scrub starts 2024-04-02T04:46:51.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[28904]: cluster 2024-04-02T04:46:49.854609+0000 osd.4 (osd.4) 74 : cluster 0 5.1 scrub ok 2024-04-02T04:46:51.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[28904]: audit 2024-04-02T04:46:49.944540+0000 mgr.y (mgr.14538) 108 : audit 0 from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:46:51.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[28904]: cluster 2024-04-02T04:46:50.397924+0000 osd.0 (osd.0) 75 : cluster 0 4.6 deep-scrub starts 2024-04-02T04:46:51.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:51 smithi069 bash[28904]: cluster 2024-04-02T04:46:50.400380+0000 osd.0 (osd.0) 76 : cluster 0 4.6 deep-scrub ok 2024-04-02T04:46:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:52 smithi098 bash[23790]: cluster 2024-04-02T04:46:51.144872+0000 osd.6 (osd.6) 53 : cluster 0 4.a scrub starts 2024-04-02T04:46:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:52 smithi098 bash[23790]: cluster 2024-04-02T04:46:51.147142+0000 osd.6 (osd.6) 54 : cluster 0 4.a scrub ok 2024-04-02T04:46:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:52 smithi098 bash[23790]: audit 2024-04-02T04:46:52.014945+0000 mon.b (mon.1) 90 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:52.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:52 smithi069 bash[28904]: cluster 2024-04-02T04:46:51.144872+0000 osd.6 (osd.6) 53 : cluster 0 4.a scrub starts 2024-04-02T04:46:52.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:52 smithi069 bash[28904]: cluster 2024-04-02T04:46:51.147142+0000 osd.6 (osd.6) 54 : cluster 0 4.a scrub ok 2024-04-02T04:46:52.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:52 smithi069 bash[28904]: audit 2024-04-02T04:46:52.014945+0000 mon.b (mon.1) 90 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:52.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:52 smithi069 bash[21528]: cluster 2024-04-02T04:46:51.144872+0000 osd.6 (osd.6) 53 : cluster 0 4.a scrub starts 2024-04-02T04:46:52.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:52 smithi069 bash[21528]: cluster 2024-04-02T04:46:51.147142+0000 osd.6 (osd.6) 54 : cluster 0 4.a scrub ok 2024-04-02T04:46:52.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:52 smithi069 bash[21528]: audit 2024-04-02T04:46:52.014945+0000 mon.b (mon.1) 90 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:53.189 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:46:52 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:46:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:53 smithi098 bash[23790]: cluster 2024-04-02T04:46:51.345875+0000 osd.5 (osd.5) 67 : cluster 0 3.1d scrub starts 2024-04-02T04:46:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:53 smithi098 bash[23790]: cluster 2024-04-02T04:46:51.350190+0000 osd.5 (osd.5) 68 : cluster 0 3.1d scrub ok 2024-04-02T04:46:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:53 smithi098 bash[23790]: cluster 2024-04-02T04:46:51.778432+0000 mgr.y (mgr.14538) 109 : cluster 0 pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:53 smithi098 bash[23790]: cluster 2024-04-02T04:46:52.338902+0000 osd.7 (osd.7) 67 : cluster 0 3.f scrub starts 2024-04-02T04:46:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:53 smithi098 bash[23790]: cluster 2024-04-02T04:46:52.343562+0000 osd.7 (osd.7) 68 : cluster 0 3.f scrub ok 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[21528]: cluster 2024-04-02T04:46:51.345875+0000 osd.5 (osd.5) 67 : cluster 0 3.1d scrub starts 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[21528]: cluster 2024-04-02T04:46:51.350190+0000 osd.5 (osd.5) 68 : cluster 0 3.1d scrub ok 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[21528]: cluster 2024-04-02T04:46:51.778432+0000 mgr.y (mgr.14538) 109 : cluster 0 pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[21528]: cluster 2024-04-02T04:46:52.338902+0000 osd.7 (osd.7) 67 : cluster 0 3.f scrub starts 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[21528]: cluster 2024-04-02T04:46:52.343562+0000 osd.7 (osd.7) 68 : cluster 0 3.f scrub ok 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[28904]: cluster 2024-04-02T04:46:51.345875+0000 osd.5 (osd.5) 67 : cluster 0 3.1d scrub starts 2024-04-02T04:46:53.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[28904]: cluster 2024-04-02T04:46:51.350190+0000 osd.5 (osd.5) 68 : cluster 0 3.1d scrub ok 2024-04-02T04:46:53.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[28904]: cluster 2024-04-02T04:46:51.778432+0000 mgr.y (mgr.14538) 109 : cluster 0 pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:53.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[28904]: cluster 2024-04-02T04:46:52.338902+0000 osd.7 (osd.7) 67 : cluster 0 3.f scrub starts 2024-04-02T04:46:53.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:53 smithi069 bash[28904]: cluster 2024-04-02T04:46:52.343562+0000 osd.7 (osd.7) 68 : cluster 0 3.f scrub ok 2024-04-02T04:46:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:54 smithi098 bash[23790]: audit 2024-04-02T04:46:52.845598+0000 mgr.y (mgr.14538) 110 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:54 smithi098 bash[23790]: cluster 2024-04-02T04:46:52.980560+0000 osd.3 (osd.3) 93 : cluster 0 3.2 scrub starts 2024-04-02T04:46:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:54 smithi098 bash[23790]: cluster 2024-04-02T04:46:52.984983+0000 osd.3 (osd.3) 94 : cluster 0 3.2 scrub ok 2024-04-02T04:46:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:54 smithi069 bash[21528]: audit 2024-04-02T04:46:52.845598+0000 mgr.y (mgr.14538) 110 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:54 smithi069 bash[21528]: cluster 2024-04-02T04:46:52.980560+0000 osd.3 (osd.3) 93 : cluster 0 3.2 scrub starts 2024-04-02T04:46:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:54 smithi069 bash[21528]: cluster 2024-04-02T04:46:52.984983+0000 osd.3 (osd.3) 94 : cluster 0 3.2 scrub ok 2024-04-02T04:46:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:54 smithi069 bash[28904]: audit 2024-04-02T04:46:52.845598+0000 mgr.y (mgr.14538) 110 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:46:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:54 smithi069 bash[28904]: cluster 2024-04-02T04:46:52.980560+0000 osd.3 (osd.3) 93 : cluster 0 3.2 scrub starts 2024-04-02T04:46:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:54 smithi069 bash[28904]: cluster 2024-04-02T04:46:52.984983+0000 osd.3 (osd.3) 94 : cluster 0 3.2 scrub ok 2024-04-02T04:46:55.458 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:46:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:55 smithi098 bash[23790]: cluster 2024-04-02T04:46:53.294456+0000 osd.1 (osd.1) 75 : cluster 0 4.12 scrub starts 2024-04-02T04:46:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:55 smithi098 bash[23790]: cluster 2024-04-02T04:46:53.297048+0000 osd.1 (osd.1) 76 : cluster 0 4.12 scrub ok 2024-04-02T04:46:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:55 smithi098 bash[23790]: cluster 2024-04-02T04:46:53.780008+0000 mgr.y (mgr.14538) 111 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:55.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:55 smithi069 bash[21528]: cluster 2024-04-02T04:46:53.294456+0000 osd.1 (osd.1) 75 : cluster 0 4.12 scrub starts 2024-04-02T04:46:55.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:55 smithi069 bash[21528]: cluster 2024-04-02T04:46:53.297048+0000 osd.1 (osd.1) 76 : cluster 0 4.12 scrub ok 2024-04-02T04:46:55.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:55 smithi069 bash[21528]: cluster 2024-04-02T04:46:53.780008+0000 mgr.y (mgr.14538) 111 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:55.479 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:55 smithi069 bash[28904]: cluster 2024-04-02T04:46:53.294456+0000 osd.1 (osd.1) 75 : cluster 0 4.12 scrub starts 2024-04-02T04:46:55.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:55 smithi069 bash[28904]: cluster 2024-04-02T04:46:53.297048+0000 osd.1 (osd.1) 76 : cluster 0 4.12 scrub ok 2024-04-02T04:46:55.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:55 smithi069 bash[28904]: cluster 2024-04-02T04:46:53.780008+0000 mgr.y (mgr.14538) 111 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:56.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:56 smithi098 bash[23790]: cluster 2024-04-02T04:46:54.267010+0000 osd.1 (osd.1) 77 : cluster 0 2.15 scrub starts 2024-04-02T04:46:56.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:56 smithi098 bash[23790]: cluster 2024-04-02T04:46:54.269332+0000 osd.1 (osd.1) 78 : cluster 0 2.15 scrub ok 2024-04-02T04:46:56.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:56 smithi069 bash[21528]: cluster 2024-04-02T04:46:54.267010+0000 osd.1 (osd.1) 77 : cluster 0 2.15 scrub starts 2024-04-02T04:46:56.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:56 smithi069 bash[21528]: cluster 2024-04-02T04:46:54.269332+0000 osd.1 (osd.1) 78 : cluster 0 2.15 scrub ok 2024-04-02T04:46:56.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:56 smithi069 bash[28904]: cluster 2024-04-02T04:46:54.267010+0000 osd.1 (osd.1) 77 : cluster 0 2.15 scrub starts 2024-04-02T04:46:56.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:56 smithi069 bash[28904]: cluster 2024-04-02T04:46:54.269332+0000 osd.1 (osd.1) 78 : cluster 0 2.15 scrub ok 2024-04-02T04:46:57.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.302319+0000 osd.1 (osd.1) 79 : cluster 0 3.0 scrub starts 2024-04-02T04:46:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.302802+0000 osd.5 (osd.5) 69 : cluster 0 2.3 deep-scrub starts 2024-04-02T04:46:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.305282+0000 osd.5 (osd.5) 70 : cluster 0 2.3 deep-scrub ok 2024-04-02T04:46:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.308505+0000 osd.1 (osd.1) 80 : cluster 0 3.0 scrub ok 2024-04-02T04:46:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.780817+0000 mgr.y (mgr.14538) 112 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.942445+0000 osd.4 (osd.4) 75 : cluster 0 4.c scrub starts 2024-04-02T04:46:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:57 smithi098 bash[23790]: cluster 2024-04-02T04:46:55.944856+0000 osd.4 (osd.4) 76 : cluster 0 4.c scrub ok 2024-04-02T04:46:57.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.302319+0000 osd.1 (osd.1) 79 : cluster 0 3.0 scrub starts 2024-04-02T04:46:57.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.302802+0000 osd.5 (osd.5) 69 : cluster 0 2.3 deep-scrub starts 2024-04-02T04:46:57.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.305282+0000 osd.5 (osd.5) 70 : cluster 0 2.3 deep-scrub ok 2024-04-02T04:46:57.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.308505+0000 osd.1 (osd.1) 80 : cluster 0 3.0 scrub ok 2024-04-02T04:46:57.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.780817+0000 mgr.y (mgr.14538) 112 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.942445+0000 osd.4 (osd.4) 75 : cluster 0 4.c scrub starts 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[21528]: cluster 2024-04-02T04:46:55.944856+0000 osd.4 (osd.4) 76 : cluster 0 4.c scrub ok 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.302319+0000 osd.1 (osd.1) 79 : cluster 0 3.0 scrub starts 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.302802+0000 osd.5 (osd.5) 69 : cluster 0 2.3 deep-scrub starts 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.305282+0000 osd.5 (osd.5) 70 : cluster 0 2.3 deep-scrub ok 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.308505+0000 osd.1 (osd.1) 80 : cluster 0 3.0 scrub ok 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.780817+0000 mgr.y (mgr.14538) 112 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.942445+0000 osd.4 (osd.4) 75 : cluster 0 4.c scrub starts 2024-04-02T04:46:57.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:57 smithi069 bash[28904]: cluster 2024-04-02T04:46:55.944856+0000 osd.4 (osd.4) 76 : cluster 0 4.c scrub ok 2024-04-02T04:46:57.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- 172.21.15.69:0/1913417792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104e10 msgr2=0x7fa534111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- 172.21.15.69:0/1913417792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104e10 0x7fa534111320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009990 tx=0x7fa52c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- 172.21.15.69:0/1913417792 shutdown_connections 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- 172.21.15.69:0/1913417792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104e10 0x7fa534111320 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- 172.21.15.69:0/1913417792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa53410a4e0 0x7fa53410a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- 172.21.15.69:0/1913417792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534104450 0x7fa5341048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- 172.21.15.69:0/1913417792 >> 172.21.15.69:0/1913417792 conn(0x7fa5340fff20 msgr2=0x7fa534102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- 172.21.15.69:0/1913417792 shutdown_connections 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- 172.21.15.69:0/1913417792 wait complete. 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 librados: starting msgr at 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 librados: starting objecter 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 Processor -- start 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- start start 2024-04-02T04:46:57.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 librados: setting wanted keys 2024-04-02T04:46:57.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 librados: calling monclient init 2024-04-02T04:46:57.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 0x7fa53419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:57.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa534104e10 0x7fa53419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:57.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa53410a4e0 0x7fa53419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa534113c20 con 0x7fa53410a4e0 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa534113aa0 con 0x7fa534104e10 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53b601700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa534113da0 con 0x7fa534104450 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.786+0000 7fa53939d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 0x7fa53419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa53410a4e0 0x7fa53419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53939d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 0x7fa53419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41452/0 (socket says 172.21.15.69:41452) 2024-04-02T04:46:57.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53939d700 1 -- 172.21.15.69:0/1897922943 learned_addr learned my addr 172.21.15.69:0/1897922943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa538b9c700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa534104e10 0x7fa53419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 -- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 msgr2=0x7fa53419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 0x7fa53419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 -- 172.21.15.69:0/1897922943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa534104e10 msgr2=0x7fa53419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa534104e10 0x7fa53419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 -- 172.21.15.69:0/1897922943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53419f170 con 0x7fa53410a4e0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53939d700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 0x7fa53419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa539b9e700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa53410a4e0 0x7fa53419ea30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0059d0 tx=0x7fa52c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 10 client.?.objecter ms_handle_connect 0x7fa53410a4e0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c026070 con 0x7fa53410a4e0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa52c004c90 con 0x7fa53410a4e0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c003b90 con 0x7fa53410a4e0 2024-04-02T04:46:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53419f400 con 0x7fa53410a4e0 2024-04-02T04:46:57.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53b601700 10 client.14715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:46:57.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5341a6ce0 con 0x7fa53410a4e0 2024-04-02T04:46:57.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53b601700 1 librados: init done 2024-04-02T04:46:57.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa53404ff00 con 0x7fa53410a4e0 2024-04-02T04:46:57.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa52c029070 con 0x7fa53410a4e0 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.790+0000 7fa5327fc700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa52006f5e0 0x7fa520071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa53939d700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa52006f5e0 0x7fa520071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 64..114) ==== 7537+0+0 (secure 0 0 0) 0x7fa52c08ece0 con 0x7fa53410a4e0 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 10 client.14715.objecter ms_dispatch 0x7fa534001550 osd_map(114..114 src has 64..114) 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 3 client.14715.objecter handle_osd_map got epochs [114,114] > 0 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 3 client.14715.objecter handle_osd_map decoding full epoch 114 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 20 client.14715.objecter dump_active .. 0 homeless 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 10 client.14715.objecter ms_handle_connect 0x7fa52006f5e0 2024-04-02T04:46:57.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa53939d700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa52006f5e0 0x7fa520071aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa51c007fd0 tx=0x7fa51c00c040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:46:57.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:57.794+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa52c030020 con 0x7fa53410a4e0 2024-04-02T04:46:58.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.046+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa5341a6eb0 con 0x7fa53410a4e0 2024-04-02T04:46:58.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.046+0000 7fa5327fc700 1 -- 172.21.15.69:0/1897922943 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa52c013070 con 0x7fa53410a4e0 2024-04-02T04:46:58.046 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:46:58.046 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 10 librados: watch_flush enter 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 10 librados: watch_flush exit 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 20 client.14715.objecter shutdown clearing up homeless session... 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 10 client.14715.objecter successfully canceled tick 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa52006f5e0 msgr2=0x7fa520071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa52006f5e0 0x7fa520071aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa51c007fd0 tx=0x7fa51c00c040 comp rx=0 tx=0).stop 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa53410a4e0 msgr2=0x7fa53419ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:46:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa53410a4e0 0x7fa53419ea30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0059d0 tx=0x7fa52c009580 comp rx=0 tx=0).stop 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 shutdown_connections 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa52006f5e0 0x7fa520071aa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa53410a4e0 0x7fa53419ea30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa534104450 0x7fa53419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 --2- 172.21.15.69:0/1897922943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa534104e10 0x7fa53419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 >> 172.21.15.69:0/1897922943 conn(0x7fa5340fff20 msgr2=0x7fa5341018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 shutdown_connections 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 -- 172.21.15.69:0/1897922943 wait complete. 2024-04-02T04:46:58.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:46:58.050+0000 7fa53b601700 1 librados: shutdown 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: cluster 2024-04-02T04:46:56.261791+0000 osd.1 (osd.1) 81 : cluster 0 6.1d scrub starts 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: cluster 2024-04-02T04:46:56.263734+0000 osd.1 (osd.1) 82 : cluster 0 6.1d scrub ok 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.427082+0000 osd.0 (osd.0) 77 : cluster 0 4.16 scrub starts 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.428891+0000 osd.0 (osd.0) 78 : cluster 0 4.16 scrub ok 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.613252+0000 osd.7 (osd.7) 69 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.613344+0000 osd.7 (osd.7) 70 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21528]: audit 2024-04-02T04:46:58.050265+0000 mon.a (mon.0) 913 : audit 0 from='client.? 172.21.15.69:0/1897922943' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T04:46:58.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: cluster 2024-04-02T04:46:56.261791+0000 osd.1 (osd.1) 81 : cluster 0 6.1d scrub starts 2024-04-02T04:46:58.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: cluster 2024-04-02T04:46:56.263734+0000 osd.1 (osd.1) 82 : cluster 0 6.1d scrub ok 2024-04-02T04:46:58.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.427082+0000 osd.0 (osd.0) 77 : cluster 0 4.16 scrub starts 2024-04-02T04:46:58.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.428891+0000 osd.0 (osd.0) 78 : cluster 0 4.16 scrub ok 2024-04-02T04:46:58.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.613252+0000 osd.7 (osd.7) 69 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:58.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.613344+0000 osd.7 (osd.7) 70 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:58.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[28904]: audit 2024-04-02T04:46:58.050265+0000 mon.a (mon.0) 913 : audit 0 from='client.? 172.21.15.69:0/1897922943' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: cluster 2024-04-02T04:46:56.261791+0000 osd.1 (osd.1) 81 : cluster 0 6.1d scrub starts 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: cluster 2024-04-02T04:46:56.263734+0000 osd.1 (osd.1) 82 : cluster 0 6.1d scrub ok 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.427082+0000 osd.0 (osd.0) 77 : cluster 0 4.16 scrub starts 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.428891+0000 osd.0 (osd.0) 78 : cluster 0 4.16 scrub ok 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.613252+0000 osd.7 (osd.7) 69 : cluster 0 purged_snaps scrub starts 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.613344+0000 osd.7 (osd.7) 70 : cluster 0 purged_snaps scrub ok 2024-04-02T04:46:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:58 smithi098 bash[23790]: audit 2024-04-02T04:46:58.050265+0000 mon.a (mon.0) 913 : audit 0 from='client.? 172.21.15.69:0/1897922943' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T04:46:58.754 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-02T04:46:58.755 INFO:tasks.cephadm:Setup complete, yielding 2024-04-02T04:46:58.755 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-02T04:46:58.789 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-02T04:46:58.790 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca'}}, 'workunit': {'branch': 'squid', 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca'}} 2024-04-02T04:46:58.790 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-02T04:46:58.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-02T04:46:59.027 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:46:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:46:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:46:59.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.317215+0000 osd.5 (osd.5) 71 : cluster 0 2.2 scrub starts 2024-04-02T04:46:59.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.319254+0000 osd.5 (osd.5) 72 : cluster 0 2.2 scrub ok 2024-04-02T04:46:59.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[28904]: cluster 2024-04-02T04:46:57.782316+0000 mgr.y (mgr.14538) 113 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:59.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[28904]: cluster 2024-04-02T04:46:58.236943+0000 mon.a (mon.0) 914 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-04-02T04:46:59.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.317215+0000 osd.5 (osd.5) 71 : cluster 0 2.2 scrub starts 2024-04-02T04:46:59.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.319254+0000 osd.5 (osd.5) 72 : cluster 0 2.2 scrub ok 2024-04-02T04:46:59.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[21528]: cluster 2024-04-02T04:46:57.782316+0000 mgr.y (mgr.14538) 113 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:59.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:46:59 smithi069 bash[21528]: cluster 2024-04-02T04:46:58.236943+0000 mon.a (mon.0) 914 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-04-02T04:46:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:59 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.317215+0000 osd.5 (osd.5) 71 : cluster 0 2.2 scrub starts 2024-04-02T04:46:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:59 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.319254+0000 osd.5 (osd.5) 72 : cluster 0 2.2 scrub ok 2024-04-02T04:46:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:59 smithi098 bash[23790]: cluster 2024-04-02T04:46:57.782316+0000 mgr.y (mgr.14538) 113 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:46:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:46:59 smithi098 bash[23790]: cluster 2024-04-02T04:46:58.236943+0000 mon.a (mon.0) 914 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-04-02T04:47:01.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:46:59.259399+0000 osd.1 (osd.1) 83 : cluster 0 3.19 scrub starts 2024-04-02T04:47:01.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:46:59.265093+0000 osd.1 (osd.1) 84 : cluster 0 3.19 scrub ok 2024-04-02T04:47:01.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:46:59.783027+0000 mgr.y (mgr.14538) 114 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:01.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:47:00.095917+0000 osd.3 (osd.3) 95 : cluster 0 6.b scrub starts 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:47:00.098259+0000 osd.3 (osd.3) 96 : cluster 0 6.b scrub ok 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:47:00.336791+0000 osd.7 (osd.7) 71 : cluster 0 3.11 scrub starts 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[21528]: cluster 2024-04-02T04:47:00.341460+0000 osd.7 (osd.7) 72 : cluster 0 3.11 scrub ok 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:46:59.259399+0000 osd.1 (osd.1) 83 : cluster 0 3.19 scrub starts 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:46:59.265093+0000 osd.1 (osd.1) 84 : cluster 0 3.19 scrub ok 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:46:59.783027+0000 mgr.y (mgr.14538) 114 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:47:00.095917+0000 osd.3 (osd.3) 95 : cluster 0 6.b scrub starts 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:47:00.098259+0000 osd.3 (osd.3) 96 : cluster 0 6.b scrub ok 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:47:00.336791+0000 osd.7 (osd.7) 71 : cluster 0 3.11 scrub starts 2024-04-02T04:47:01.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:01 smithi069 bash[28904]: cluster 2024-04-02T04:47:00.341460+0000 osd.7 (osd.7) 72 : cluster 0 3.11 scrub ok 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:46:59.259399+0000 osd.1 (osd.1) 83 : cluster 0 3.19 scrub starts 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:46:59.265093+0000 osd.1 (osd.1) 84 : cluster 0 3.19 scrub ok 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:46:59.783027+0000 mgr.y (mgr.14538) 114 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:47:00.095917+0000 osd.3 (osd.3) 95 : cluster 0 6.b scrub starts 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:47:00.098259+0000 osd.3 (osd.3) 96 : cluster 0 6.b scrub ok 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:47:00.336791+0000 osd.7 (osd.7) 71 : cluster 0 3.11 scrub starts 2024-04-02T04:47:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:01 smithi098 bash[23790]: cluster 2024-04-02T04:47:00.341460+0000 osd.7 (osd.7) 72 : cluster 0 3.11 scrub ok 2024-04-02T04:47:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:02 smithi069 bash[21528]: cluster 2024-04-02T04:47:00.936816+0000 osd.4 (osd.4) 77 : cluster 0 6.2 scrub starts 2024-04-02T04:47:02.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:02 smithi069 bash[21528]: cluster 2024-04-02T04:47:00.939342+0000 osd.4 (osd.4) 78 : cluster 0 6.2 scrub ok 2024-04-02T04:47:02.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:02 smithi069 bash[28904]: cluster 2024-04-02T04:47:00.936816+0000 osd.4 (osd.4) 77 : cluster 0 6.2 scrub starts 2024-04-02T04:47:02.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:02 smithi069 bash[28904]: cluster 2024-04-02T04:47:00.939342+0000 osd.4 (osd.4) 78 : cluster 0 6.2 scrub ok 2024-04-02T04:47:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:02 smithi098 bash[23790]: cluster 2024-04-02T04:47:00.936816+0000 osd.4 (osd.4) 77 : cluster 0 6.2 scrub starts 2024-04-02T04:47:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:02 smithi098 bash[23790]: cluster 2024-04-02T04:47:00.939342+0000 osd.4 (osd.4) 78 : cluster 0 6.2 scrub ok 2024-04-02T04:47:03.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:47:02 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:01.783682+0000 mgr.y (mgr.14538) 115 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:01.934078+0000 osd.4 (osd.4) 79 : cluster 0 6.1e scrub starts 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:01.936265+0000 osd.4 (osd.4) 80 : cluster 0 6.1e scrub ok 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:02.150388+0000 osd.3 (osd.3) 97 : cluster 0 6.6 scrub starts 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:02.153470+0000 osd.3 (osd.3) 98 : cluster 0 6.6 scrub ok 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:02.313646+0000 osd.7 (osd.7) 73 : cluster 0 2.b scrub starts 2024-04-02T04:47:03.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[21528]: cluster 2024-04-02T04:47:02.316315+0000 osd.7 (osd.7) 74 : cluster 0 2.b scrub ok 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:01.783682+0000 mgr.y (mgr.14538) 115 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:01.934078+0000 osd.4 (osd.4) 79 : cluster 0 6.1e scrub starts 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:01.936265+0000 osd.4 (osd.4) 80 : cluster 0 6.1e scrub ok 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:02.150388+0000 osd.3 (osd.3) 97 : cluster 0 6.6 scrub starts 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:02.153470+0000 osd.3 (osd.3) 98 : cluster 0 6.6 scrub ok 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:02.313646+0000 osd.7 (osd.7) 73 : cluster 0 2.b scrub starts 2024-04-02T04:47:03.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:03 smithi069 bash[28904]: cluster 2024-04-02T04:47:02.316315+0000 osd.7 (osd.7) 74 : cluster 0 2.b scrub ok 2024-04-02T04:47:03.574 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:01.783682+0000 mgr.y (mgr.14538) 115 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:01.934078+0000 osd.4 (osd.4) 79 : cluster 0 6.1e scrub starts 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:01.936265+0000 osd.4 (osd.4) 80 : cluster 0 6.1e scrub ok 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:02.150388+0000 osd.3 (osd.3) 97 : cluster 0 6.6 scrub starts 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:02.153470+0000 osd.3 (osd.3) 98 : cluster 0 6.6 scrub ok 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:02.313646+0000 osd.7 (osd.7) 73 : cluster 0 2.b scrub starts 2024-04-02T04:47:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:03 smithi098 bash[23790]: cluster 2024-04-02T04:47:02.316315+0000 osd.7 (osd.7) 74 : cluster 0 2.b scrub ok 2024-04-02T04:47:04.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:04 smithi069 bash[21528]: audit 2024-04-02T04:47:02.855127+0000 mgr.y (mgr.14538) 116 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:04.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:04 smithi069 bash[21528]: cluster 2024-04-02T04:47:03.155975+0000 osd.3 (osd.3) 99 : cluster 0 6.1f scrub starts 2024-04-02T04:47:04.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:04 smithi069 bash[21528]: cluster 2024-04-02T04:47:03.158531+0000 osd.3 (osd.3) 100 : cluster 0 6.1f scrub ok 2024-04-02T04:47:04.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:04 smithi069 bash[28904]: audit 2024-04-02T04:47:02.855127+0000 mgr.y (mgr.14538) 116 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:04.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:04 smithi069 bash[28904]: cluster 2024-04-02T04:47:03.155975+0000 osd.3 (osd.3) 99 : cluster 0 6.1f scrub starts 2024-04-02T04:47:04.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:04 smithi069 bash[28904]: cluster 2024-04-02T04:47:03.158531+0000 osd.3 (osd.3) 100 : cluster 0 6.1f scrub ok 2024-04-02T04:47:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:04 smithi098 bash[23790]: audit 2024-04-02T04:47:02.855127+0000 mgr.y (mgr.14538) 116 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:04 smithi098 bash[23790]: cluster 2024-04-02T04:47:03.155975+0000 osd.3 (osd.3) 99 : cluster 0 6.1f scrub starts 2024-04-02T04:47:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:04 smithi098 bash[23790]: cluster 2024-04-02T04:47:03.158531+0000 osd.3 (osd.3) 100 : cluster 0 6.1f scrub ok 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[21528]: cluster 2024-04-02T04:47:03.785120+0000 mgr.y (mgr.14538) 117 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[21528]: cluster 2024-04-02T04:47:04.157879+0000 osd.3 (osd.3) 101 : cluster 0 3.7 scrub starts 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[21528]: cluster 2024-04-02T04:47:04.162133+0000 osd.3 (osd.3) 102 : cluster 0 3.7 scrub ok 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[21528]: cluster 2024-04-02T04:47:04.413229+0000 osd.7 (osd.7) 75 : cluster 0 6.15 scrub starts 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[21528]: cluster 2024-04-02T04:47:04.415725+0000 osd.7 (osd.7) 76 : cluster 0 6.15 scrub ok 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[28904]: cluster 2024-04-02T04:47:03.785120+0000 mgr.y (mgr.14538) 117 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[28904]: cluster 2024-04-02T04:47:04.157879+0000 osd.3 (osd.3) 101 : cluster 0 3.7 scrub starts 2024-04-02T04:47:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[28904]: cluster 2024-04-02T04:47:04.162133+0000 osd.3 (osd.3) 102 : cluster 0 3.7 scrub ok 2024-04-02T04:47:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[28904]: cluster 2024-04-02T04:47:04.413229+0000 osd.7 (osd.7) 75 : cluster 0 6.15 scrub starts 2024-04-02T04:47:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:05 smithi069 bash[28904]: cluster 2024-04-02T04:47:04.415725+0000 osd.7 (osd.7) 76 : cluster 0 6.15 scrub ok 2024-04-02T04:47:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:05 smithi098 bash[23790]: cluster 2024-04-02T04:47:03.785120+0000 mgr.y (mgr.14538) 117 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:05 smithi098 bash[23790]: cluster 2024-04-02T04:47:04.157879+0000 osd.3 (osd.3) 101 : cluster 0 3.7 scrub starts 2024-04-02T04:47:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:05 smithi098 bash[23790]: cluster 2024-04-02T04:47:04.162133+0000 osd.3 (osd.3) 102 : cluster 0 3.7 scrub ok 2024-04-02T04:47:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:05 smithi098 bash[23790]: cluster 2024-04-02T04:47:04.413229+0000 osd.7 (osd.7) 75 : cluster 0 6.15 scrub starts 2024-04-02T04:47:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:05 smithi098 bash[23790]: cluster 2024-04-02T04:47:04.415725+0000 osd.7 (osd.7) 76 : cluster 0 6.15 scrub ok 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- 172.21.15.69:0/3451010311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f200410aa60 msgr2=0x7f2004111430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451010311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f200410aa60 0x7f2004111430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400b0a0 tx=0x7f1ff400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- 172.21.15.69:0/3451010311 shutdown_connections 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451010311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f200410aa60 0x7f2004111430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451010311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f2004100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451010311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2004100eb0 0x7f200410a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- 172.21.15.69:0/3451010311 >> 172.21.15.69:0/3451010311 conn(0x7f20040782b0 msgr2=0x7f20040fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- 172.21.15.69:0/3451010311 shutdown_connections 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- 172.21.15.69:0/3451010311 wait complete. 2024-04-02T04:47:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 librados: starting msgr at 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 librados: starting objecter 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 Processor -- start 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- start start 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 librados: setting wanted keys 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 librados: calling monclient init 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f200419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2004100eb0 0x7f200419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200410aa60 0x7f200419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2004113d40 con 0x7f2004100590 2024-04-02T04:47:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2004113bc0 con 0x7f200410aa60 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f200b80c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2004113ec0 con 0x7f2004100eb0 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f20095a8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f200419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2008da7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2004100eb0 0x7f200419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f20095a8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f200419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36936/0 (socket says 172.21.15.69:36936) 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f20095a8700 1 -- 172.21.15.69:0/3451987603 learned_addr learned my addr 172.21.15.69:0/3451987603 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2009da9700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200410aa60 0x7f200419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2008da7700 1 -- 172.21.15.69:0/3451987603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200410aa60 msgr2=0x7f200419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2008da7700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200410aa60 0x7f200419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2008da7700 1 -- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 msgr2=0x7f200419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2008da7700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f200419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f2008da7700 1 -- 172.21.15.69:0/3451987603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200419f210 con 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.089+0000 7f20095a8700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f200419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f2008da7700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2004100eb0 0x7f200419a740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800b450 tx=0x7f1ff800b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 10 client.?.objecter ms_handle_connect 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8015020 con 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f1ff8011cb0 con 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200419f500 con 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8010c40 con 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f200b80c700 10 client.24608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20041a6d80 con 0x7f2004100eb0 2024-04-02T04:47:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f200b80c700 1 librados: init done 2024-04-02T04:47:06.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200404ff00 con 0x7f2004100eb0 2024-04-02T04:47:06.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f1ff8018020 con 0x7f2004100eb0 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fec06f5e0 0x7f1fec071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(115..115 src has 64..115) ==== 7537+0+0 (secure 0 0 0) 0x7f1ff808d6f0 con 0x7f2004100eb0 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 10 client.24608.objecter ms_dispatch 0x7f2004001550 osd_map(115..115 src has 64..115) 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 3 client.24608.objecter handle_osd_map got epochs [115,115] > 0 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 3 client.24608.objecter handle_osd_map decoding full epoch 115 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20095a8700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fec06f5e0 0x7f1fec071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 20 client.24608.objecter dump_active .. 0 homeless 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20027fc700 10 client.24608.objecter ms_handle_connect 0x7f1fec06f5e0 2024-04-02T04:47:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.093+0000 7f20095a8700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fec06f5e0 0x7f1fec071aa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc005950 tx=0x7f1ffc009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.097+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1ff801f020 con 0x7f2004100eb0 2024-04-02T04:47:06.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.305+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) -- 0x7f20041a6fb0 con 0x7f2004100eb0 2024-04-02T04:47:06.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.313+0000 7f20027fc700 1 -- 172.21.15.69:0/3451987603 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v28)=0 v28) ==== 136+0+0 (secure 0 0 0) 0x7f1ff805acd0 con 0x7f2004100eb0 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 10 librados: watch_flush enter 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 10 librados: watch_flush exit 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 20 client.24608.objecter shutdown clearing up homeless session... 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 10 client.24608.objecter successfully canceled tick 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fec06f5e0 msgr2=0x7f1fec071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fec06f5e0 0x7f1fec071aa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc005950 tx=0x7f1ffc009260 comp rx=0 tx=0).stop 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2004100eb0 msgr2=0x7f200419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2004100eb0 0x7f200419a740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800b450 tx=0x7f1ff800b760 comp rx=0 tx=0).stop 2024-04-02T04:47:06.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 shutdown_connections 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fec06f5e0 0x7f1fec071aa0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f200410aa60 0x7f200419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2004100590 0x7f200419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 --2- 172.21.15.69:0/3451987603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2004100eb0 0x7f200419a740 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 >> 172.21.15.69:0/3451987603 conn(0x7f20040782b0 msgr2=0x7f2004108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 shutdown_connections 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 -- 172.21.15.69:0/3451987603 wait complete. 2024-04-02T04:47:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:06.317+0000 7f200b80c700 1 librados: shutdown 2024-04-02T04:47:06.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.280519+0000 osd.5 (osd.5) 73 : cluster 0 3.1c scrub starts 2024-04-02T04:47:06.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.285507+0000 osd.5 (osd.5) 74 : cluster 0 3.1c scrub ok 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.386299+0000 osd.7 (osd.7) 77 : cluster 0 2.0 scrub starts 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.388758+0000 osd.7 (osd.7) 78 : cluster 0 2.0 scrub ok 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21782]: [02/Apr/2024:04:47:06] ENGINE Bus STOPPING 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21782]: [02/Apr/2024:04:47:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21782]: [02/Apr/2024:04:47:06] ENGINE Bus STOPPED 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21782]: [02/Apr/2024:04:47:06] ENGINE Bus STARTING 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21782]: [02/Apr/2024:04:47:06] ENGINE Serving on http://:::9283 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[21782]: [02/Apr/2024:04:47:06] ENGINE Bus STARTED 2024-04-02T04:47:06.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.280519+0000 osd.5 (osd.5) 73 : cluster 0 3.1c scrub starts 2024-04-02T04:47:06.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.285507+0000 osd.5 (osd.5) 74 : cluster 0 3.1c scrub ok 2024-04-02T04:47:06.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.386299+0000 osd.7 (osd.7) 77 : cluster 0 2.0 scrub starts 2024-04-02T04:47:06.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:06 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.388758+0000 osd.7 (osd.7) 78 : cluster 0 2.0 scrub ok 2024-04-02T04:47:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:06 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.280519+0000 osd.5 (osd.5) 73 : cluster 0 3.1c scrub starts 2024-04-02T04:47:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:06 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.285507+0000 osd.5 (osd.5) 74 : cluster 0 3.1c scrub ok 2024-04-02T04:47:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:06 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.386299+0000 osd.7 (osd.7) 77 : cluster 0 2.0 scrub starts 2024-04-02T04:47:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:06 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.388758+0000 osd.7 (osd.7) 78 : cluster 0 2.0 scrub ok 2024-04-02T04:47:06.923 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-02T04:47:06.924 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-02T04:47:06.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:47:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.785747+0000 mgr.y (mgr.14538) 118 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.882641+0000 osd.4 (osd.4) 81 : cluster 0 6.17 scrub starts 2024-04-02T04:47:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: cluster 2024-04-02T04:47:05.884833+0000 osd.4 (osd.4) 82 : cluster 0 6.17 scrub ok 2024-04-02T04:47:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: audit 2024-04-02T04:47:06.315525+0000 mon.a (mon.0) 915 : audit 1 from='client.? ' entity='client.admin' 2024-04-02T04:47:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: audit 2024-04-02T04:47:06.344905+0000 mon.b (mon.1) 91 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: cluster 2024-04-02T04:47:06.423803+0000 osd.0 (osd.0) 79 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: cluster 2024-04-02T04:47:06.423925+0000 osd.0 (osd.0) 80 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: audit 2024-04-02T04:47:06.974082+0000 mon.b (mon.1) 92 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: audit 2024-04-02T04:47:06.975890+0000 mon.b (mon.1) 93 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: audit 2024-04-02T04:47:06.988855+0000 mon.a (mon.0) 916 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:47:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:07 smithi098 bash[23790]: audit 2024-04-02T04:47:07.044893+0000 mon.b (mon.1) 94 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.785747+0000 mgr.y (mgr.14538) 118 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.882641+0000 osd.4 (osd.4) 81 : cluster 0 6.17 scrub starts 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: cluster 2024-04-02T04:47:05.884833+0000 osd.4 (osd.4) 82 : cluster 0 6.17 scrub ok 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: audit 2024-04-02T04:47:06.315525+0000 mon.a (mon.0) 915 : audit 1 from='client.? ' entity='client.admin' 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: audit 2024-04-02T04:47:06.344905+0000 mon.b (mon.1) 91 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: cluster 2024-04-02T04:47:06.423803+0000 osd.0 (osd.0) 79 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: cluster 2024-04-02T04:47:06.423925+0000 osd.0 (osd.0) 80 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: audit 2024-04-02T04:47:06.974082+0000 mon.b (mon.1) 92 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: audit 2024-04-02T04:47:06.975890+0000 mon.b (mon.1) 93 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:07.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: audit 2024-04-02T04:47:06.988855+0000 mon.a (mon.0) 916 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[21528]: audit 2024-04-02T04:47:07.044893+0000 mon.b (mon.1) 94 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.785747+0000 mgr.y (mgr.14538) 118 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.882641+0000 osd.4 (osd.4) 81 : cluster 0 6.17 scrub starts 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: cluster 2024-04-02T04:47:05.884833+0000 osd.4 (osd.4) 82 : cluster 0 6.17 scrub ok 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: audit 2024-04-02T04:47:06.315525+0000 mon.a (mon.0) 915 : audit 1 from='client.? ' entity='client.admin' 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: audit 2024-04-02T04:47:06.344905+0000 mon.b (mon.1) 91 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: cluster 2024-04-02T04:47:06.423803+0000 osd.0 (osd.0) 79 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: cluster 2024-04-02T04:47:06.423925+0000 osd.0 (osd.0) 80 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: audit 2024-04-02T04:47:06.974082+0000 mon.b (mon.1) 92 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: audit 2024-04-02T04:47:06.975890+0000 mon.b (mon.1) 93 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:07.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: audit 2024-04-02T04:47:06.988855+0000 mon.a (mon.0) 916 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:47:07.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:07 smithi069 bash[28904]: audit 2024-04-02T04:47:07.044893+0000 mon.b (mon.1) 94 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:08 smithi069 bash[21528]: cluster 2024-04-02T04:47:07.340442+0000 mon.a (mon.0) 917 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-04-02T04:47:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:08 smithi069 bash[28904]: cluster 2024-04-02T04:47:07.340442+0000 mon.a (mon.0) 917 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-04-02T04:47:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:08 smithi098 bash[23790]: cluster 2024-04-02T04:47:07.340442+0000 mon.a (mon.0) 917 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-04-02T04:47:09.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:47:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:47:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:09 smithi098 bash[23790]: cluster 2024-04-02T04:47:07.787147+0000 mgr.y (mgr.14538) 119 : cluster 0 pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:47:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:09 smithi098 bash[23790]: cluster 2024-04-02T04:47:07.885700+0000 osd.4 (osd.4) 83 : cluster 0 4.18 scrub starts 2024-04-02T04:47:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:09 smithi098 bash[23790]: cluster 2024-04-02T04:47:07.888822+0000 osd.4 (osd.4) 84 : cluster 0 4.18 scrub ok 2024-04-02T04:47:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:09 smithi098 bash[23790]: cluster 2024-04-02T04:47:08.329238+0000 osd.0 (osd.0) 81 : cluster 0 3.17 scrub starts 2024-04-02T04:47:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:09 smithi098 bash[23790]: cluster 2024-04-02T04:47:08.333505+0000 osd.0 (osd.0) 82 : cluster 0 3.17 scrub ok 2024-04-02T04:47:09.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[21528]: cluster 2024-04-02T04:47:07.787147+0000 mgr.y (mgr.14538) 119 : cluster 0 pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:47:09.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[21528]: cluster 2024-04-02T04:47:07.885700+0000 osd.4 (osd.4) 83 : cluster 0 4.18 scrub starts 2024-04-02T04:47:09.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[21528]: cluster 2024-04-02T04:47:07.888822+0000 osd.4 (osd.4) 84 : cluster 0 4.18 scrub ok 2024-04-02T04:47:09.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[21528]: cluster 2024-04-02T04:47:08.329238+0000 osd.0 (osd.0) 81 : cluster 0 3.17 scrub starts 2024-04-02T04:47:09.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[21528]: cluster 2024-04-02T04:47:08.333505+0000 osd.0 (osd.0) 82 : cluster 0 3.17 scrub ok 2024-04-02T04:47:09.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[28904]: cluster 2024-04-02T04:47:07.787147+0000 mgr.y (mgr.14538) 119 : cluster 0 pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:47:09.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[28904]: cluster 2024-04-02T04:47:07.885700+0000 osd.4 (osd.4) 83 : cluster 0 4.18 scrub starts 2024-04-02T04:47:09.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[28904]: cluster 2024-04-02T04:47:07.888822+0000 osd.4 (osd.4) 84 : cluster 0 4.18 scrub ok 2024-04-02T04:47:09.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[28904]: cluster 2024-04-02T04:47:08.329238+0000 osd.0 (osd.0) 81 : cluster 0 3.17 scrub starts 2024-04-02T04:47:09.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:09 smithi069 bash[28904]: cluster 2024-04-02T04:47:08.333505+0000 osd.0 (osd.0) 82 : cluster 0 3.17 scrub ok 2024-04-02T04:47:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:10 smithi098 bash[23790]: cluster 2024-04-02T04:47:08.859532+0000 osd.4 (osd.4) 85 : cluster 0 6.e scrub starts 2024-04-02T04:47:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:10 smithi098 bash[23790]: cluster 2024-04-02T04:47:08.861967+0000 osd.4 (osd.4) 86 : cluster 0 6.e scrub ok 2024-04-02T04:47:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:10 smithi098 bash[23790]: cluster 2024-04-02T04:47:09.292972+0000 osd.0 (osd.0) 83 : cluster 0 6.16 scrub starts 2024-04-02T04:47:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:10 smithi098 bash[23790]: cluster 2024-04-02T04:47:09.295488+0000 osd.0 (osd.0) 84 : cluster 0 6.16 scrub ok 2024-04-02T04:47:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:10 smithi098 bash[23790]: cluster 2024-04-02T04:47:09.384289+0000 osd.7 (osd.7) 79 : cluster 0 2.1c scrub starts 2024-04-02T04:47:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:10 smithi098 bash[23790]: cluster 2024-04-02T04:47:09.387358+0000 osd.7 (osd.7) 80 : cluster 0 2.1c scrub ok 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[21528]: cluster 2024-04-02T04:47:08.859532+0000 osd.4 (osd.4) 85 : cluster 0 6.e scrub starts 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[21528]: cluster 2024-04-02T04:47:08.861967+0000 osd.4 (osd.4) 86 : cluster 0 6.e scrub ok 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[21528]: cluster 2024-04-02T04:47:09.292972+0000 osd.0 (osd.0) 83 : cluster 0 6.16 scrub starts 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[21528]: cluster 2024-04-02T04:47:09.295488+0000 osd.0 (osd.0) 84 : cluster 0 6.16 scrub ok 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[21528]: cluster 2024-04-02T04:47:09.384289+0000 osd.7 (osd.7) 79 : cluster 0 2.1c scrub starts 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[21528]: cluster 2024-04-02T04:47:09.387358+0000 osd.7 (osd.7) 80 : cluster 0 2.1c scrub ok 2024-04-02T04:47:10.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[28904]: cluster 2024-04-02T04:47:08.859532+0000 osd.4 (osd.4) 85 : cluster 0 6.e scrub starts 2024-04-02T04:47:10.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[28904]: cluster 2024-04-02T04:47:08.861967+0000 osd.4 (osd.4) 86 : cluster 0 6.e scrub ok 2024-04-02T04:47:10.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[28904]: cluster 2024-04-02T04:47:09.292972+0000 osd.0 (osd.0) 83 : cluster 0 6.16 scrub starts 2024-04-02T04:47:10.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[28904]: cluster 2024-04-02T04:47:09.295488+0000 osd.0 (osd.0) 84 : cluster 0 6.16 scrub ok 2024-04-02T04:47:10.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[28904]: cluster 2024-04-02T04:47:09.384289+0000 osd.7 (osd.7) 79 : cluster 0 2.1c scrub starts 2024-04-02T04:47:10.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:10 smithi069 bash[28904]: cluster 2024-04-02T04:47:09.387358+0000 osd.7 (osd.7) 80 : cluster 0 2.1c scrub ok 2024-04-02T04:47:11.711 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:11 smithi098 bash[23790]: cluster 2024-04-02T04:47:09.787860+0000 mgr.y (mgr.14538) 120 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:11.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:11 smithi069 bash[21528]: cluster 2024-04-02T04:47:09.787860+0000 mgr.y (mgr.14538) 120 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:11.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:11 smithi069 bash[28904]: cluster 2024-04-02T04:47:09.787860+0000 mgr.y (mgr.14538) 120 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:13.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:47:12 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:11.788287+0000 mgr.y (mgr.14538) 121 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:11.923091+0000 osd.4 (osd.4) 87 : cluster 0 6.1a scrub starts 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:11.925417+0000 osd.4 (osd.4) 88 : cluster 0 6.1a scrub ok 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:12.269184+0000 osd.3 (osd.3) 103 : cluster 0 4.11 scrub starts 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:12.271670+0000 osd.3 (osd.3) 104 : cluster 0 4.11 scrub ok 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:12.342407+0000 osd.5 (osd.5) 75 : cluster 0 6.a scrub starts 2024-04-02T04:47:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:13 smithi098 bash[23790]: cluster 2024-04-02T04:47:12.345221+0000 osd.5 (osd.5) 76 : cluster 0 6.a scrub ok 2024-04-02T04:47:13.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:11.788287+0000 mgr.y (mgr.14538) 121 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:13.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:11.923091+0000 osd.4 (osd.4) 87 : cluster 0 6.1a scrub starts 2024-04-02T04:47:13.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:11.925417+0000 osd.4 (osd.4) 88 : cluster 0 6.1a scrub ok 2024-04-02T04:47:13.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:12.269184+0000 osd.3 (osd.3) 103 : cluster 0 4.11 scrub starts 2024-04-02T04:47:13.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:12.271670+0000 osd.3 (osd.3) 104 : cluster 0 4.11 scrub ok 2024-04-02T04:47:13.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:12.342407+0000 osd.5 (osd.5) 75 : cluster 0 6.a scrub starts 2024-04-02T04:47:13.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[21528]: cluster 2024-04-02T04:47:12.345221+0000 osd.5 (osd.5) 76 : cluster 0 6.a scrub ok 2024-04-02T04:47:13.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:11.788287+0000 mgr.y (mgr.14538) 121 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:13.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:11.923091+0000 osd.4 (osd.4) 87 : cluster 0 6.1a scrub starts 2024-04-02T04:47:13.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:11.925417+0000 osd.4 (osd.4) 88 : cluster 0 6.1a scrub ok 2024-04-02T04:47:13.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:12.269184+0000 osd.3 (osd.3) 103 : cluster 0 4.11 scrub starts 2024-04-02T04:47:13.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:12.271670+0000 osd.3 (osd.3) 104 : cluster 0 4.11 scrub ok 2024-04-02T04:47:13.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:12.342407+0000 osd.5 (osd.5) 75 : cluster 0 6.a scrub starts 2024-04-02T04:47:13.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:13 smithi069 bash[28904]: cluster 2024-04-02T04:47:12.345221+0000 osd.5 (osd.5) 76 : cluster 0 6.a scrub ok 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/1590923093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101b30 msgr2=0x7f8a0410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- 172.21.15.69:0/1590923093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101b30 0x7f8a0410e020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00b0a0 tx=0x7f89fc00b3b0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/1590923093 shutdown_connections 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- 172.21.15.69:0/1590923093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101b30 0x7f8a0410e020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- 172.21.15.69:0/1590923093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a041075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- 172.21.15.69:0/1590923093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a04101170 0x7f8a041015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/1590923093 >> 172.21.15.69:0/1590923093 conn(0x7f8a04078110 msgr2=0x7f8a04078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/1590923093 shutdown_connections 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/1590923093 wait complete. 2024-04-02T04:47:14.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 librados: starting msgr at 2024-04-02T04:47:14.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 librados: starting objecter 2024-04-02T04:47:14.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 Processor -- start 2024-04-02T04:47:14.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- start start 2024-04-02T04:47:14.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 librados: setting wanted keys 2024-04-02T04:47:14.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 librados: calling monclient init 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 0x7f8a04197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a04101b30 0x7f8a04198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a0419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a04110930 con 0x7f8a04101170 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a041107b0 con 0x7f8a04101b30 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8a04110ab0 con 0x7f8a041071e0 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a02ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 0x7f8a04197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a0419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a02ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 0x7f8a04197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44204/0 (socket says 172.21.15.69:44204) 2024-04-02T04:47:14.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a0419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33676/0 (socket says 172.21.15.69:33676) 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a02ffd700 1 -- 172.21.15.69:0/2066148280 learned_addr learned my addr 172.21.15.69:0/2066148280 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a027fc700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a04101b30 0x7f8a04198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 -- 172.21.15.69:0/2066148280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a04101b30 msgr2=0x7f8a04198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a04101b30 0x7f8a04198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 -- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 msgr2=0x7f8a04197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 0x7f8a04197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 -- 172.21.15.69:0/2066148280 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0419cf00 con 0x7f8a041071e0 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a02ffd700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 0x7f8a04197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a037fe700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a0419c7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00bfd0 tx=0x7f89fc007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a08829700 10 client.?.objecter ms_handle_connect 0x7f8a041071e0 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a08829700 10 client.?.objecter resend_mon_ops 2024-04-02T04:47:14.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc021020 con 0x7f8a041071e0 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0419d190 con 0x7f8a041071e0 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f89fc004c50 con 0x7f8a041071e0 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc005050 con 0x7f8a041071e0 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 10 client.24614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a041a4a70 con 0x7f8a041071e0 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.241+0000 7f8a0982b700 1 librados: init done 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a04102890 con 0x7f8a041071e0 2024-04-02T04:47:14.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f89fc024020 con 0x7f8a041071e0 2024-04-02T04:47:14.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f89ec06f5e0 0x7f89ec071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:14.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(116..116 src has 64..116) ==== 7537+0+0 (secure 0 0 0) 0x7f89fc00f020 con 0x7f8a041071e0 2024-04-02T04:47:14.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 10 client.24614.objecter ms_dispatch 0x7f8a04001550 osd_map(116..116 src has 64..116) 2024-04-02T04:47:14.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 3 client.24614.objecter handle_osd_map got epochs [116,116] > 0 2024-04-02T04:47:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a02ffd700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f89ec06f5e0 0x7f89ec071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 3 client.24614.objecter handle_osd_map decoding full epoch 116 2024-04-02T04:47:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 20 client.24614.objecter dump_active .. 0 homeless 2024-04-02T04:47:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a02ffd700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f89ec06f5e0 0x7f89ec071aa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009680 tx=0x7f89f4009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.245+0000 7f8a08829700 10 client.24614.objecter ms_handle_connect 0x7f89ec06f5e0 2024-04-02T04:47:14.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.249+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89fc05c810 con 0x7f8a041071e0 2024-04-02T04:47:14.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.457+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8a04199250 con 0x7f89ec06f5e0 2024-04-02T04:47:14.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a08829700 1 -- 172.21.15.69:0/2066148280 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346719 (secure 0 0 0) 0x7f8a04199250 con 0x7f89ec06f5e0 2024-04-02T04:47:14.480 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:47:14.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 10 librados: watch_flush enter 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 10 librados: watch_flush exit 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 20 client.24614.objecter shutdown clearing up homeless session... 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 10 client.24614.objecter successfully canceled tick 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f89ec06f5e0 msgr2=0x7f89ec071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f89ec06f5e0 0x7f89ec071aa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009680 tx=0x7f89f4009260 comp rx=0 tx=0).stop 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 msgr2=0x7f8a0419c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.477+0000 7f8a0982b700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a0419c7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00bfd0 tx=0x7f89fc007a20 comp rx=0 tx=0).stop 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 shutdown_connections 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f89ec06f5e0 0x7f89ec071aa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a041071e0 0x7f8a0419c7c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a04101170 0x7f8a04197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 --2- 172.21.15.69:0/2066148280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a04101b30 0x7f8a04198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 >> 172.21.15.69:0/2066148280 conn(0x7f8a04078110 msgr2=0x7f8a040ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:14.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 shutdown_connections 2024-04-02T04:47:14.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 -- 172.21.15.69:0/2066148280 wait complete. 2024-04-02T04:47:14.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:14.481+0000 7f8a0982b700 1 librados: shutdown 2024-04-02T04:47:14.484 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:47:14.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[21528]: audit 2024-04-02T04:47:12.866402+0000 mgr.y (mgr.14538) 122 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:14.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[21528]: cluster 2024-04-02T04:47:13.286757+0000 osd.3 (osd.3) 105 : cluster 0 6.13 scrub starts 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[21528]: cluster 2024-04-02T04:47:13.288358+0000 osd.3 (osd.3) 106 : cluster 0 6.13 scrub ok 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[21528]: cluster 2024-04-02T04:47:13.342436+0000 osd.0 (osd.0) 85 : cluster 0 3.1b deep-scrub starts 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[21528]: cluster 2024-04-02T04:47:13.347664+0000 osd.0 (osd.0) 86 : cluster 0 3.1b deep-scrub ok 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[28904]: audit 2024-04-02T04:47:12.866402+0000 mgr.y (mgr.14538) 122 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[28904]: cluster 2024-04-02T04:47:13.286757+0000 osd.3 (osd.3) 105 : cluster 0 6.13 scrub starts 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[28904]: cluster 2024-04-02T04:47:13.288358+0000 osd.3 (osd.3) 106 : cluster 0 6.13 scrub ok 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[28904]: cluster 2024-04-02T04:47:13.342436+0000 osd.0 (osd.0) 85 : cluster 0 3.1b deep-scrub starts 2024-04-02T04:47:14.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:14 smithi069 bash[28904]: cluster 2024-04-02T04:47:13.347664+0000 osd.0 (osd.0) 86 : cluster 0 3.1b deep-scrub ok 2024-04-02T04:47:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:14 smithi098 bash[23790]: audit 2024-04-02T04:47:12.866402+0000 mgr.y (mgr.14538) 122 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:14 smithi098 bash[23790]: cluster 2024-04-02T04:47:13.286757+0000 osd.3 (osd.3) 105 : cluster 0 6.13 scrub starts 2024-04-02T04:47:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:14 smithi098 bash[23790]: cluster 2024-04-02T04:47:13.288358+0000 osd.3 (osd.3) 106 : cluster 0 6.13 scrub ok 2024-04-02T04:47:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:14 smithi098 bash[23790]: cluster 2024-04-02T04:47:13.342436+0000 osd.0 (osd.0) 85 : cluster 0 3.1b deep-scrub starts 2024-04-02T04:47:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:14 smithi098 bash[23790]: cluster 2024-04-02T04:47:13.347664+0000 osd.0 (osd.0) 86 : cluster 0 3.1b deep-scrub ok 2024-04-02T04:47:15.075 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-04-02T04:47:13.788551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1414,"num_read_kb":1228,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231884,"kb_used_data":7420,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729332,"statfs":{"total":767960285184,"available":767722835968,"internally_reserved":0,"allocated":7598080,"data_stored":4781946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004599"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337864+0000","last_change":"2024-04-02T04:45:29.804443+0000","last_active":"2024-04-02T04:47:07.337864+0000","last_peered":"2024-04-02T04:47:07.337864+0000","last_clean":"2024-04-02T04:47:07.337864+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:47:07.337864+0000","last_undegraded":"2024-04-02T04:47:07.337864+0000","last_fullsized":"2024-04-02T04:47:07.337864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:29.804217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:29.804217+0000","last_clean_scrub_stamp":"2024-04-02T04:45:29.804217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336603+0000","last_change":"2024-04-02T04:46:26.737065+0000","last_active":"2024-04-02T04:47:07.336603+0000","last_peered":"2024-04-02T04:47:07.336603+0000","last_clean":"2024-04-02T04:47:07.336603+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:47:07.336603+0000","last_undegraded":"2024-04-02T04:47:07.336603+0000","last_fullsized":"2024-04-02T04:47:07.336603+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:26.736961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.736961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.907303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329660+0000","last_change":"2024-04-02T04:46:21.290286+0000","last_active":"2024-04-02T04:47:08.329660+0000","last_peered":"2024-04-02T04:47:08.329660+0000","last_clean":"2024-04-02T04:47:08.329660+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:47:08.329660+0000","last_undegraded":"2024-04-02T04:47:08.329660+0000","last_fullsized":"2024-04-02T04:47:08.329660+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.290194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.290194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.622357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342725+0000","last_change":"2024-04-02T04:46:39.861495+0000","last_active":"2024-04-02T04:47:07.342725+0000","last_peered":"2024-04-02T04:47:07.342725+0000","last_clean":"2024-04-02T04:47:07.342725+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:47:07.342725+0000","last_undegraded":"2024-04-02T04:47:07.342725+0000","last_fullsized":"2024-04-02T04:47:07.342725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:39.861258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.861258+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.553184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:11.925614+0000","last_change":"2024-04-02T04:47:11.925670+0000","last_active":"2024-04-02T04:47:11.925614+0000","last_peered":"2024-04-02T04:47:11.925614+0000","last_clean":"2024-04-02T04:47:11.925614+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:47:11.925614+0000","last_undegraded":"2024-04-02T04:47:11.925614+0000","last_fullsized":"2024-04-02T04:47:11.925614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:11.925452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:11.925452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.710115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.888993+0000","last_change":"2024-04-02T04:47:07.889053+0000","last_active":"2024-04-02T04:47:07.888993+0000","last_peered":"2024-04-02T04:47:07.888993+0000","last_clean":"2024-04-02T04:47:07.888993+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:47:07.888993+0000","last_undegraded":"2024-04-02T04:47:07.888993+0000","last_fullsized":"2024-04-02T04:47:07.888993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:07.888835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:07.888835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:10.239127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342401+0000","last_change":"2024-04-02T04:46:13.710516+0000","last_active":"2024-04-02T04:47:07.342401+0000","last_peered":"2024-04-02T04:47:07.342401+0000","last_clean":"2024-04-02T04:47:07.342401+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:47:07.342401+0000","last_undegraded":"2024-04-02T04:47:07.342401+0000","last_fullsized":"2024-04-02T04:47:07.342401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:13.710340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:13.710340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.036057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":131,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327382+0000","last_change":"2024-04-02T04:46:45.396999+0000","last_active":"2024-04-02T04:47:08.327382+0000","last_peered":"2024-04-02T04:47:08.327382+0000","last_clean":"2024-04-02T04:47:08.327382+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:47:08.327382+0000","last_undegraded":"2024-04-02T04:47:08.327382+0000","last_fullsized":"2024-04-02T04:47:08.327382+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:45.396760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.396760+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.943901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327368+0000","last_change":"2024-04-02T04:46:39.343919+0000","last_active":"2024-04-02T04:47:08.327368+0000","last_peered":"2024-04-02T04:47:08.327368+0000","last_clean":"2024-04-02T04:47:08.327368+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:47:08.327368+0000","last_undegraded":"2024-04-02T04:47:08.327368+0000","last_fullsized":"2024-04-02T04:47:08.327368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:39.343592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.343592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.312506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330459+0000","last_change":"2024-04-02T04:46:22.812121+0000","last_active":"2024-04-02T04:47:07.330459+0000","last_peered":"2024-04-02T04:47:07.330459+0000","last_clean":"2024-04-02T04:47:07.330459+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:47:07.330459+0000","last_undegraded":"2024-04-02T04:47:07.330459+0000","last_fullsized":"2024-04-02T04:47:07.330459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.811836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.811836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.362979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.378376+0000","last_change":"2024-04-02T04:46:35.292548+0000","last_active":"2024-04-02T04:47:08.378376+0000","last_peered":"2024-04-02T04:47:08.378376+0000","last_clean":"2024-04-02T04:47:08.378376+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:47:08.378376+0000","last_undegraded":"2024-04-02T04:47:08.378376+0000","last_fullsized":"2024-04-02T04:47:08.378376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.292336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.292336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.961526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":123,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326418+0000","last_change":"2024-04-02T04:47:05.285775+0000","last_active":"2024-04-02T04:47:08.326418+0000","last_peered":"2024-04-02T04:47:08.326418+0000","last_clean":"2024-04-02T04:47:08.326418+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:47:08.326418+0000","last_undegraded":"2024-04-02T04:47:08.326418+0000","last_fullsized":"2024-04-02T04:47:08.326418+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:05.285521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.285521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:28.240436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326768+0000","last_change":"2024-04-02T04:46:27.234204+0000","last_active":"2024-04-02T04:47:08.326768+0000","last_peered":"2024-04-02T04:47:08.326768+0000","last_clean":"2024-04-02T04:47:08.326768+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:47:08.326768+0000","last_undegraded":"2024-04-02T04:47:08.326768+0000","last_fullsized":"2024-04-02T04:47:08.326768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.234006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.234006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.363415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330948+0000","last_change":"2024-04-02T04:46:44.775289+0000","last_active":"2024-04-02T04:47:07.330948+0000","last_peered":"2024-04-02T04:47:07.330948+0000","last_clean":"2024-04-02T04:47:07.330948+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:47:07.330948+0000","last_undegraded":"2024-04-02T04:47:07.330948+0000","last_fullsized":"2024-04-02T04:47:07.330948+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.775071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.207519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.387532+0000","last_change":"2024-04-02T04:47:09.387603+0000","last_active":"2024-04-02T04:47:09.387532+0000","last_peered":"2024-04-02T04:47:09.387532+0000","last_clean":"2024-04-02T04:47:09.387532+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:47:09.387532+0000","last_undegraded":"2024-04-02T04:47:09.387532+0000","last_fullsized":"2024-04-02T04:47:09.387532+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:09.387375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.387375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.279958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326488+0000","last_change":"2024-04-02T04:46:51.350464+0000","last_active":"2024-04-02T04:47:08.326488+0000","last_peered":"2024-04-02T04:47:08.326488+0000","last_clean":"2024-04-02T04:47:08.326488+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:47:08.326488+0000","last_undegraded":"2024-04-02T04:47:08.326488+0000","last_fullsized":"2024-04-02T04:47:08.326488+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:46:51.350201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:51.350201+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.520062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342833+0000","last_change":"2024-04-02T04:47:03.158759+0000","last_active":"2024-04-02T04:47:07.342833+0000","last_peered":"2024-04-02T04:47:07.342833+0000","last_clean":"2024-04-02T04:47:07.342833+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:47:07.342833+0000","last_undegraded":"2024-04-02T04:47:07.342833+0000","last_fullsized":"2024-04-02T04:47:07.342833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:03.158541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:03.158541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.522158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342147+0000","last_change":"2024-04-02T04:46:36.942358+0000","last_active":"2024-04-02T04:47:07.342147+0000","last_peered":"2024-04-02T04:47:07.342147+0000","last_clean":"2024-04-02T04:47:07.342147+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:47:07.342147+0000","last_undegraded":"2024-04-02T04:47:07.342147+0000","last_fullsized":"2024-04-02T04:47:07.342147+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.942139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.942139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.526854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338428+0000","last_change":"2024-04-02T04:45:57.720821+0000","last_active":"2024-04-02T04:47:07.338428+0000","last_peered":"2024-04-02T04:47:07.338428+0000","last_clean":"2024-04-02T04:47:07.338428+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:47:07.338428+0000","last_undegraded":"2024-04-02T04:47:07.338428+0000","last_fullsized":"2024-04-02T04:47:07.338428+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:57.720720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.720720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:17.288812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"85'23","reported_seq":141,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330226+0000","last_change":"2024-04-02T04:46:21.850698+0000","last_active":"2024-04-02T04:47:07.330226+0000","last_peered":"2024-04-02T04:47:07.330226+0000","last_clean":"2024-04-02T04:47:07.330226+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:47:07.330226+0000","last_undegraded":"2024-04-02T04:47:07.330226+0000","last_fullsized":"2024-04-02T04:47:07.330226+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-02T04:46:21.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.850373+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:35.100552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.332050+0000","last_change":"2024-04-02T04:47:01.936520+0000","last_active":"2024-04-02T04:47:07.332050+0000","last_peered":"2024-04-02T04:47:07.332050+0000","last_clean":"2024-04-02T04:47:07.332050+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:47:07.332050+0000","last_undegraded":"2024-04-02T04:47:07.332050+0000","last_fullsized":"2024-04-02T04:47:07.332050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:01.936275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:01.936275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.709309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380137+0000","last_change":"2024-04-02T04:46:32.598626+0000","last_active":"2024-04-02T04:47:07.380137+0000","last_peered":"2024-04-02T04:47:07.380137+0000","last_clean":"2024-04-02T04:47:07.380137+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:47:07.380137+0000","last_undegraded":"2024-04-02T04:47:07.380137+0000","last_fullsized":"2024-04-02T04:47:07.380137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.598421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.598421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.059173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885523+0000","last_change":"2024-04-02T04:46:21.009651+0000","last_active":"2024-04-02T04:47:07.885523+0000","last_peered":"2024-04-02T04:47:07.885523+0000","last_clean":"2024-04-02T04:47:07.885523+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:47:07.885523+0000","last_undegraded":"2024-04-02T04:47:07.885523+0000","last_fullsized":"2024-04-02T04:47:07.885523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.009424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.009424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.532492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326601+0000","last_change":"2024-04-02T04:45:57.518634+0000","last_active":"2024-04-02T04:47:08.326601+0000","last_peered":"2024-04-02T04:47:08.326601+0000","last_clean":"2024-04-02T04:47:08.326601+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:47:08.326601+0000","last_undegraded":"2024-04-02T04:47:08.326601+0000","last_fullsized":"2024-04-02T04:47:08.326601+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:45:57.518350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.518350+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:12.947455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.340619+0000","last_change":"2024-04-02T04:46:56.263880+0000","last_active":"2024-04-02T04:47:07.340619+0000","last_peered":"2024-04-02T04:47:07.340619+0000","last_clean":"2024-04-02T04:47:07.340619+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:47:07.340619+0000","last_undegraded":"2024-04-02T04:47:07.340619+0000","last_fullsized":"2024-04-02T04:47:07.340619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:56.263740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:56.263740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.910160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":149,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.886894+0000","last_change":"2024-04-02T04:46:47.154319+0000","last_active":"2024-04-02T04:47:07.886894+0000","last_peered":"2024-04-02T04:47:07.886894+0000","last_clean":"2024-04-02T04:47:07.886894+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:47:07.886894+0000","last_undegraded":"2024-04-02T04:47:07.886894+0000","last_fullsized":"2024-04-02T04:47:07.886894+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:47.154082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.154082+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.920192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337028+0000","last_change":"2024-04-02T04:46:27.777917+0000","last_active":"2024-04-02T04:47:07.337028+0000","last_peered":"2024-04-02T04:47:07.337028+0000","last_clean":"2024-04-02T04:47:07.337028+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:47:07.337028+0000","last_undegraded":"2024-04-02T04:47:07.337028+0000","last_fullsized":"2024-04-02T04:47:07.337028+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:46:27.777679+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.777679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.754491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":128,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336098+0000","last_change":"2024-04-02T04:46:24.726007+0000","last_active":"2024-04-02T04:47:07.336098+0000","last_peered":"2024-04-02T04:47:07.336098+0000","last_clean":"2024-04-02T04:47:07.336098+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:47:07.336098+0000","last_undegraded":"2024-04-02T04:47:07.336098+0000","last_fullsized":"2024-04-02T04:47:07.336098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:24.725734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.725734+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.190873+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377528+0000","last_change":"2024-04-02T04:46:42.290326+0000","last_active":"2024-04-02T04:47:08.377528+0000","last_peered":"2024-04-02T04:47:08.377528+0000","last_clean":"2024-04-02T04:47:08.377528+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:47:08.377528+0000","last_undegraded":"2024-04-02T04:47:08.377528+0000","last_fullsized":"2024-04-02T04:47:08.377528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:46:42.290071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.290071+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.053573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329111+0000","last_change":"2024-04-02T04:46:04.466247+0000","last_active":"2024-04-02T04:47:08.329111+0000","last_peered":"2024-04-02T04:47:08.329111+0000","last_clean":"2024-04-02T04:47:08.329111+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:47:08.329111+0000","last_undegraded":"2024-04-02T04:47:08.329111+0000","last_fullsized":"2024-04-02T04:47:08.329111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:04.465964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:04.465964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.060111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327481+0000","last_change":"2024-04-02T04:46:16.163503+0000","last_active":"2024-04-02T04:47:08.327481+0000","last_peered":"2024-04-02T04:47:08.327481+0000","last_clean":"2024-04-02T04:47:08.327481+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:47:08.327481+0000","last_undegraded":"2024-04-02T04:47:08.327481+0000","last_fullsized":"2024-04-02T04:47:08.327481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.163286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.163286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:21.454867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":172,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338096+0000","last_change":"2024-04-02T04:46:59.265375+0000","last_active":"2024-04-02T04:47:07.338096+0000","last_peered":"2024-04-02T04:47:07.338096+0000","last_clean":"2024-04-02T04:47:07.338096+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:47:07.338096+0000","last_undegraded":"2024-04-02T04:47:07.338096+0000","last_fullsized":"2024-04-02T04:47:07.338096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:46:59.265104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:59.265104+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.709345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339214+0000","last_change":"2024-04-02T04:45:35.692872+0000","last_active":"2024-04-02T04:47:07.339214+0000","last_peered":"2024-04-02T04:47:07.339214+0000","last_clean":"2024-04-02T04:47:07.339214+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:47:07.339214+0000","last_undegraded":"2024-04-02T04:47:07.339214+0000","last_fullsized":"2024-04-02T04:47:07.339214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:35.692628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:35.692628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:12.271889+0000","last_change":"2024-04-02T04:47:12.271950+0000","last_active":"2024-04-02T04:47:12.271889+0000","last_peered":"2024-04-02T04:47:12.271889+0000","last_clean":"2024-04-02T04:47:12.271889+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:47:12.271889+0000","last_undegraded":"2024-04-02T04:47:12.271889+0000","last_fullsized":"2024-04-02T04:47:12.271889+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.271680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.271680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.801349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.887583+0000","last_change":"2024-04-02T04:46:15.989347+0000","last_active":"2024-04-02T04:47:07.887583+0000","last_peered":"2024-04-02T04:47:07.887583+0000","last_clean":"2024-04-02T04:47:07.887583+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:47:07.887583+0000","last_undegraded":"2024-04-02T04:47:07.887583+0000","last_fullsized":"2024-04-02T04:47:07.887583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:15.989126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:46:15.989126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.092291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326516+0000","last_change":"2024-04-02T04:46:41.326411+0000","last_active":"2024-04-02T04:47:08.326516+0000","last_peered":"2024-04-02T04:47:08.326516+0000","last_clean":"2024-04-02T04:47:08.326516+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:47:08.326516+0000","last_undegraded":"2024-04-02T04:47:08.326516+0000","last_fullsized":"2024-04-02T04:47:08.326516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:46:41.326158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.326158+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.505788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376864+0000","last_change":"2024-04-02T04:45:53.375887+0000","last_active":"2024-04-02T04:47:08.376864+0000","last_peered":"2024-04-02T04:47:08.376864+0000","last_clean":"2024-04-02T04:47:08.376864+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:47:08.376864+0000","last_undegraded":"2024-04-02T04:47:08.376864+0000","last_fullsized":"2024-04-02T04:47:08.376864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:45:53.375671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336100+0000","last_change":"2024-04-02T04:46:10.724569+0000","last_active":"2024-04-02T04:47:07.336100+0000","last_peered":"2024-04-02T04:47:07.336100+0000","last_clean":"2024-04-02T04:47:07.336100+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:47:07.336100+0000","last_undegraded":"2024-04-02T04:47:07.336100+0000","last_fullsized":"2024-04-02T04:47:07.336100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.724363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.724363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:18.641739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329075+0000","last_change":"2024-04-02T04:46:36.285859+0000","last_active":"2024-04-02T04:47:08.329075+0000","last_peered":"2024-04-02T04:47:08.329075+0000","last_clean":"2024-04-02T04:47:08.329075+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:47:08.329075+0000","last_undegraded":"2024-04-02T04:47:08.329075+0000","last_fullsized":"2024-04-02T04:47:08.329075+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.285639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.484787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.333777+0000","last_change":"2024-04-02T04:47:08.333929+0000","last_active":"2024-04-02T04:47:08.333777+0000","last_peered":"2024-04-02T04:47:08.333777+0000","last_clean":"2024-04-02T04:47:08.333777+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:47:08.333777+0000","last_undegraded":"2024-04-02T04:47:08.333777+0000","last_fullsized":"2024-04-02T04:47:08.333777+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:47:08.333516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.333516+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.726461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336174+0000","last_change":"2024-04-02T04:46:38.893014+0000","last_active":"2024-04-02T04:47:07.336174+0000","last_peered":"2024-04-02T04:47:07.336174+0000","last_clean":"2024-04-02T04:47:07.336174+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:47:07.336174+0000","last_undegraded":"2024-04-02T04:47:07.336174+0000","last_fullsized":"2024-04-02T04:47:07.336174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.892773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.892773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.901685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330743+0000","last_change":"2024-04-02T04:46:10.895794+0000","last_active":"2024-04-02T04:47:07.330743+0000","last_peered":"2024-04-02T04:47:07.330743+0000","last_clean":"2024-04-02T04:47:07.330743+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:47:07.330743+0000","last_undegraded":"2024-04-02T04:47:07.330743+0000","last_fullsized":"2024-04-02T04:47:07.330743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.895579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.895579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.482340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341608+0000","last_change":"2024-04-02T04:46:54.269554+0000","last_active":"2024-04-02T04:47:07.341608+0000","last_peered":"2024-04-02T04:47:07.341608+0000","last_clean":"2024-04-02T04:47:07.341608+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:47:07.341608+0000","last_undegraded":"2024-04-02T04:47:07.341608+0000","last_fullsized":"2024-04-02T04:47:07.341608+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:54.269343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:54.269343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.989031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.331176+0000","last_change":"2024-04-02T04:46:37.808452+0000","last_active":"2024-04-02T04:47:07.331176+0000","last_peered":"2024-04-02T04:47:07.331176+0000","last_clean":"2024-04-02T04:47:07.331176+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:47:07.331176+0000","last_undegraded":"2024-04-02T04:47:07.331176+0000","last_fullsized":"2024-04-02T04:47:07.331176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:46:37.808090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.808090+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.632664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328397+0000","last_change":"2024-04-02T04:45:59.476847+0000","last_active":"2024-04-02T04:47:08.328397+0000","last_peered":"2024-04-02T04:47:08.328397+0000","last_clean":"2024-04-02T04:47:08.328397+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:47:08.328397+0000","last_undegraded":"2024-04-02T04:47:08.328397+0000","last_fullsized":"2024-04-02T04:47:08.328397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:59.476642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:59.476642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:16.549126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338577+0000","last_change":"2024-04-02T04:46:53.297425+0000","last_active":"2024-04-02T04:47:07.338577+0000","last_peered":"2024-04-02T04:47:07.338577+0000","last_clean":"2024-04-02T04:47:07.338577+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:47:07.338577+0000","last_undegraded":"2024-04-02T04:47:07.338577+0000","last_fullsized":"2024-04-02T04:47:07.338577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:53.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:53.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.497586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.886217+0000","last_change":"2024-04-02T04:46:34.223691+0000","last_active":"2024-04-02T04:47:07.886217+0000","last_peered":"2024-04-02T04:47:07.886217+0000","last_clean":"2024-04-02T04:47:07.886217+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:47:07.886217+0000","last_undegraded":"2024-04-02T04:47:07.886217+0000","last_fullsized":"2024-04-02T04:47:07.886217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:34.223346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:34.223346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.085223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375455+0000","last_change":"2024-04-02T04:46:27.353748+0000","last_active":"2024-04-02T04:47:08.375455+0000","last_peered":"2024-04-02T04:47:08.375455+0000","last_clean":"2024-04-02T04:47:08.375455+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:47:08.375455+0000","last_undegraded":"2024-04-02T04:47:08.375455+0000","last_fullsized":"2024-04-02T04:47:08.375455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:27.353503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.353503+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.633132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.332501+0000","last_change":"2024-04-02T04:47:05.885138+0000","last_active":"2024-04-02T04:47:07.332501+0000","last_peered":"2024-04-02T04:47:07.332501+0000","last_clean":"2024-04-02T04:47:07.332501+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:47:07.332501+0000","last_undegraded":"2024-04-02T04:47:07.332501+0000","last_fullsized":"2024-04-02T04:47:07.332501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:30.330301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328719+0000","last_change":"2024-04-02T04:46:44.346692+0000","last_active":"2024-04-02T04:47:08.328719+0000","last_peered":"2024-04-02T04:47:08.328719+0000","last_clean":"2024-04-02T04:47:08.328719+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:47:08.328719+0000","last_undegraded":"2024-04-02T04:47:08.328719+0000","last_fullsized":"2024-04-02T04:47:08.328719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:44.346318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.092878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327637+0000","last_change":"2024-04-02T04:46:36.244725+0000","last_active":"2024-04-02T04:47:08.327637+0000","last_peered":"2024-04-02T04:47:08.327637+0000","last_clean":"2024-04-02T04:47:08.327637+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:47:08.327637+0000","last_undegraded":"2024-04-02T04:47:08.327637+0000","last_fullsized":"2024-04-02T04:47:08.327637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.244514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.244514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.017937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":116,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326566+0000","last_change":"2024-04-02T04:46:28.203298+0000","last_active":"2024-04-02T04:47:08.326566+0000","last_peered":"2024-04-02T04:47:08.326566+0000","last_clean":"2024-04-02T04:47:08.326566+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:47:08.326566+0000","last_undegraded":"2024-04-02T04:47:08.326566+0000","last_fullsized":"2024-04-02T04:47:08.326566+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:46:28.203156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.203156+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.777551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.295656+0000","last_change":"2024-04-02T04:47:09.295716+0000","last_active":"2024-04-02T04:47:09.295656+0000","last_peered":"2024-04-02T04:47:09.295656+0000","last_clean":"2024-04-02T04:47:09.295656+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:47:09.295656+0000","last_undegraded":"2024-04-02T04:47:09.295656+0000","last_fullsized":"2024-04-02T04:47:09.295656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:09.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.295498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.587769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"85'11","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341801+0000","last_change":"2024-04-02T04:46:28.775353+0000","last_active":"2024-04-02T04:47:07.341801+0000","last_peered":"2024-04-02T04:47:07.341801+0000","last_clean":"2024-04-02T04:47:07.341801+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:47:07.341801+0000","last_undegraded":"2024-04-02T04:47:07.341801+0000","last_fullsized":"2024-04-02T04:47:07.341801+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:28.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.775071+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:55.875528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327304+0000","last_change":"2024-04-02T04:46:48.355080+0000","last_active":"2024-04-02T04:47:08.327304+0000","last_peered":"2024-04-02T04:47:08.327304+0000","last_clean":"2024-04-02T04:47:08.327304+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:47:08.327304+0000","last_undegraded":"2024-04-02T04:47:08.327304+0000","last_fullsized":"2024-04-02T04:47:08.327304+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:48.354869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:48.354869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.638786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375332+0000","last_change":"2024-04-02T04:46:33.347283+0000","last_active":"2024-04-02T04:47:08.375332+0000","last_peered":"2024-04-02T04:47:08.375332+0000","last_clean":"2024-04-02T04:47:08.375332+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:47:08.375332+0000","last_undegraded":"2024-04-02T04:47:08.375332+0000","last_fullsized":"2024-04-02T04:47:08.375332+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:33.347053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.347053+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.542332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377319+0000","last_change":"2024-04-02T04:47:04.415949+0000","last_active":"2024-04-02T04:47:08.377319+0000","last_peered":"2024-04-02T04:47:08.377319+0000","last_clean":"2024-04-02T04:47:08.377319+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:47:08.377319+0000","last_undegraded":"2024-04-02T04:47:08.377319+0000","last_fullsized":"2024-04-02T04:47:08.377319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:04.415735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.415735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:34.243533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342938+0000","last_change":"2024-04-02T04:46:45.921390+0000","last_active":"2024-04-02T04:47:07.342938+0000","last_peered":"2024-04-02T04:47:07.342938+0000","last_clean":"2024-04-02T04:47:07.342938+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:47:07.342938+0000","last_undegraded":"2024-04-02T04:47:07.342938+0000","last_fullsized":"2024-04-02T04:47:07.342938+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.369948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885087+0000","last_change":"2024-04-02T04:46:28.039798+0000","last_active":"2024-04-02T04:47:07.885087+0000","last_peered":"2024-04-02T04:47:07.885087+0000","last_clean":"2024-04-02T04:47:07.885087+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:47:07.885087+0000","last_undegraded":"2024-04-02T04:47:07.885087+0000","last_fullsized":"2024-04-02T04:47:07.885087+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:28.039566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.039566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.163582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.884895+0000","last_change":"2024-04-02T04:46:41.254878+0000","last_active":"2024-04-02T04:47:07.884895+0000","last_peered":"2024-04-02T04:47:07.884895+0000","last_clean":"2024-04-02T04:47:07.884895+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:47:07.884895+0000","last_undegraded":"2024-04-02T04:47:07.884895+0000","last_fullsized":"2024-04-02T04:47:07.884895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:46:41.254633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.254633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.695806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380222+0000","last_change":"2024-04-02T04:46:38.622583+0000","last_active":"2024-04-02T04:47:07.380222+0000","last_peered":"2024-04-02T04:47:07.380222+0000","last_clean":"2024-04-02T04:47:07.380222+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:47:07.380222+0000","last_undegraded":"2024-04-02T04:47:07.380222+0000","last_fullsized":"2024-04-02T04:47:07.380222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.622220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.622220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.379107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328365+0000","last_change":"2024-04-02T04:46:57.429067+0000","last_active":"2024-04-02T04:47:08.328365+0000","last_peered":"2024-04-02T04:47:08.328365+0000","last_clean":"2024-04-02T04:47:08.328365+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:47:08.328365+0000","last_undegraded":"2024-04-02T04:47:08.328365+0000","last_fullsized":"2024-04-02T04:47:08.328365+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.428901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.428901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.996741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.379012+0000","last_change":"2024-04-02T04:46:44.734741+0000","last_active":"2024-04-02T04:47:07.379012+0000","last_peered":"2024-04-02T04:47:07.379012+0000","last_clean":"2024-04-02T04:47:07.379012+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:47:07.379012+0000","last_undegraded":"2024-04-02T04:47:07.379012+0000","last_fullsized":"2024-04-02T04:47:07.379012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.734508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.734508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.434405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375304+0000","last_change":"2024-04-02T04:47:00.341837+0000","last_active":"2024-04-02T04:47:08.375304+0000","last_peered":"2024-04-02T04:47:08.375304+0000","last_clean":"2024-04-02T04:47:08.375304+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:47:08.375304+0000","last_undegraded":"2024-04-02T04:47:08.375304+0000","last_fullsized":"2024-04-02T04:47:08.375304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:00.341473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.341473+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:24.505569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338960+0000","last_change":"2024-04-02T04:47:00.098587+0000","last_active":"2024-04-02T04:47:07.338960+0000","last_peered":"2024-04-02T04:47:07.338960+0000","last_clean":"2024-04-02T04:47:07.338960+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:47:07.338960+0000","last_undegraded":"2024-04-02T04:47:07.338960+0000","last_fullsized":"2024-04-02T04:47:07.338960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.098270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.098270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.905960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.332151+0000","last_change":"2024-04-02T04:46:02.906156+0000","last_active":"2024-04-02T04:47:07.332151+0000","last_peered":"2024-04-02T04:47:07.332151+0000","last_clean":"2024-04-02T04:47:07.332151+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:47:07.332151+0000","last_undegraded":"2024-04-02T04:47:07.332151+0000","last_fullsized":"2024-04-02T04:47:07.332151+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:46:02.905900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:02.905900+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.976578+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375477+0000","last_change":"2024-04-02T04:46:19.421011+0000","last_active":"2024-04-02T04:47:08.375477+0000","last_peered":"2024-04-02T04:47:08.375477+0000","last_clean":"2024-04-02T04:47:08.375477+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:47:08.375477+0000","last_undegraded":"2024-04-02T04:47:08.375477+0000","last_fullsized":"2024-04-02T04:47:08.375477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:19.420739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:19.420739+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.952287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"85'11","reported_seq":145,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330027+0000","last_change":"2024-04-02T04:46:27.787700+0000","last_active":"2024-04-02T04:47:07.330027+0000","last_peered":"2024-04-02T04:47:07.330027+0000","last_clean":"2024-04-02T04:47:07.330027+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:47:07.330027+0000","last_undegraded":"2024-04-02T04:47:07.330027+0000","last_fullsized":"2024-04-02T04:47:07.330027+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:27.787353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.787353+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:41.371457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328415+0000","last_change":"2024-04-02T04:46:43.414718+0000","last_active":"2024-04-02T04:47:08.328415+0000","last_peered":"2024-04-02T04:47:08.328415+0000","last_clean":"2024-04-02T04:47:08.328415+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:47:08.328415+0000","last_undegraded":"2024-04-02T04:47:08.328415+0000","last_fullsized":"2024-04-02T04:47:08.328415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:43.414511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.414511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:43.745696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338916+0000","last_change":"2024-04-02T04:46:22.411099+0000","last_active":"2024-04-02T04:47:07.338916+0000","last_peered":"2024-04-02T04:47:07.338916+0000","last_clean":"2024-04-02T04:47:07.338916+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:47:07.338916+0000","last_undegraded":"2024-04-02T04:47:07.338916+0000","last_fullsized":"2024-04-02T04:47:07.338916+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:22.411010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.411010+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:32.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":160,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327147+0000","last_change":"2024-04-02T04:46:14.106481+0000","last_active":"2024-04-02T04:47:08.327147+0000","last_peered":"2024-04-02T04:47:08.327147+0000","last_clean":"2024-04-02T04:47:08.327147+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:47:08.327147+0000","last_undegraded":"2024-04-02T04:47:08.327147+0000","last_fullsized":"2024-04-02T04:47:08.327147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:14.106232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.106232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.525489+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339469+0000","last_change":"2024-04-02T04:46:36.317753+0000","last_active":"2024-04-02T04:47:07.339469+0000","last_peered":"2024-04-02T04:47:07.339469+0000","last_clean":"2024-04-02T04:47:07.339469+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:47:07.339469+0000","last_undegraded":"2024-04-02T04:47:07.339469+0000","last_fullsized":"2024-04-02T04:47:07.339469+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.317542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.317542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.283515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342414+0000","last_change":"2024-04-02T04:47:02.153712+0000","last_active":"2024-04-02T04:47:07.342414+0000","last_peered":"2024-04-02T04:47:07.342414+0000","last_clean":"2024-04-02T04:47:07.342414+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:47:07.342414+0000","last_undegraded":"2024-04-02T04:47:07.342414+0000","last_fullsized":"2024-04-02T04:47:07.342414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:47:02.153481+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.153481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.645312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327385+0000","last_change":"2024-04-02T04:46:57.319477+0000","last_active":"2024-04-02T04:47:08.327385+0000","last_peered":"2024-04-02T04:47:08.327385+0000","last_clean":"2024-04-02T04:47:08.327385+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:47:08.327385+0000","last_undegraded":"2024-04-02T04:47:08.327385+0000","last_fullsized":"2024-04-02T04:47:08.327385+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.319266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.319266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.104997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.331515+0000","last_change":"2024-04-02T04:46:36.775425+0000","last_active":"2024-04-02T04:47:07.331515+0000","last_peered":"2024-04-02T04:47:07.331515+0000","last_clean":"2024-04-02T04:47:07.331515+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:47:07.331515+0000","last_undegraded":"2024-04-02T04:47:07.331515+0000","last_fullsized":"2024-04-02T04:47:07.331515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:46:36.775178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.775178+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.278965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.340002+0000","last_change":"2024-04-02T04:46:32.335647+0000","last_active":"2024-04-02T04:47:07.340002+0000","last_peered":"2024-04-02T04:47:07.340002+0000","last_clean":"2024-04-02T04:47:07.340002+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:47:07.340002+0000","last_undegraded":"2024-04-02T04:47:07.340002+0000","last_fullsized":"2024-04-02T04:47:07.340002+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.335539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.335539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.500350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377644+0000","last_change":"2024-04-02T04:46:44.347022+0000","last_active":"2024-04-02T04:47:08.377644+0000","last_peered":"2024-04-02T04:47:08.377644+0000","last_clean":"2024-04-02T04:47:08.377644+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:47:08.377644+0000","last_undegraded":"2024-04-02T04:47:08.377644+0000","last_fullsized":"2024-04-02T04:47:08.377644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.346726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:53.151319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":155,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337064+0000","last_change":"2024-04-02T04:46:55.308858+0000","last_active":"2024-04-02T04:47:07.337064+0000","last_peered":"2024-04-02T04:47:07.337064+0000","last_clean":"2024-04-02T04:47:07.337064+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:47:07.337064+0000","last_undegraded":"2024-04-02T04:47:07.337064+0000","last_fullsized":"2024-04-02T04:47:07.337064+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:46:55.308528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.308528+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.148543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.379641+0000","last_change":"2024-04-02T04:46:07.378595+0000","last_active":"2024-04-02T04:47:07.379641+0000","last_peered":"2024-04-02T04:47:07.379641+0000","last_clean":"2024-04-02T04:47:07.379641+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:47:07.379641+0000","last_undegraded":"2024-04-02T04:47:07.379641+0000","last_fullsized":"2024-04-02T04:47:07.379641+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:07.378519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:46:07.378519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:12.389433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.340353+0000","last_change":"2024-04-02T04:46:42.343240+0000","last_active":"2024-04-02T04:47:07.340353+0000","last_peered":"2024-04-02T04:47:07.340353+0000","last_clean":"2024-04-02T04:47:07.340353+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:47:07.340353+0000","last_undegraded":"2024-04-02T04:47:07.340353+0000","last_fullsized":"2024-04-02T04:47:07.340353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:42.343078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.343078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.356250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329563+0000","last_change":"2024-04-02T04:46:35.276714+0000","last_active":"2024-04-02T04:47:08.329563+0000","last_peered":"2024-04-02T04:47:08.329563+0000","last_clean":"2024-04-02T04:47:08.329563+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:47:08.329563+0000","last_undegraded":"2024-04-02T04:47:08.329563+0000","last_fullsized":"2024-04-02T04:47:08.329563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.276502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.276502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.541614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341446+0000","last_change":"2024-04-02T04:46:52.985248+0000","last_active":"2024-04-02T04:47:07.341446+0000","last_peered":"2024-04-02T04:47:07.341446+0000","last_clean":"2024-04-02T04:47:07.341446+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:47:07.341446+0000","last_undegraded":"2024-04-02T04:47:07.341446+0000","last_fullsized":"2024-04-02T04:47:07.341446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:52.984994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.984994+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.008710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327978+0000","last_change":"2024-04-02T04:46:55.305525+0000","last_active":"2024-04-02T04:47:08.327978+0000","last_peered":"2024-04-02T04:47:08.327978+0000","last_clean":"2024-04-02T04:47:08.327978+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:47:08.327978+0000","last_undegraded":"2024-04-02T04:47:08.327978+0000","last_fullsized":"2024-04-02T04:47:08.327978+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.305293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.165521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":360,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376481+0000","last_change":"2024-04-02T04:46:17.433834+0000","last_active":"2024-04-02T04:47:08.376481+0000","last_peered":"2024-04-02T04:47:08.376481+0000","last_clean":"2024-04-02T04:47:08.376481+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:47:08.376481+0000","last_undegraded":"2024-04-02T04:47:08.376481+0000","last_fullsized":"2024-04-02T04:47:08.376481+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:46:17.433563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:46:17.433563+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.028333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.886341+0000","last_change":"2024-04-02T04:46:10.058200+0000","last_active":"2024-04-02T04:47:07.886341+0000","last_peered":"2024-04-02T04:47:07.886341+0000","last_clean":"2024-04-02T04:47:07.886341+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:47:07.886341+0000","last_undegraded":"2024-04-02T04:47:07.886341+0000","last_fullsized":"2024-04-02T04:47:07.886341+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.057875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.057875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.984328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338554+0000","last_change":"2024-04-02T04:46:40.318480+0000","last_active":"2024-04-02T04:47:07.338554+0000","last_peered":"2024-04-02T04:47:07.338554+0000","last_clean":"2024-04-02T04:47:07.338554+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:47:07.338554+0000","last_undegraded":"2024-04-02T04:47:07.338554+0000","last_fullsized":"2024-04-02T04:47:07.338554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.318304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.318304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.789228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326440+0000","last_change":"2024-04-02T04:46:33.251890+0000","last_active":"2024-04-02T04:47:08.326440+0000","last_peered":"2024-04-02T04:47:08.326440+0000","last_clean":"2024-04-02T04:47:08.326440+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:47:08.326440+0000","last_undegraded":"2024-04-02T04:47:08.326440+0000","last_fullsized":"2024-04-02T04:47:08.326440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:46:33.251772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.251772+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.560770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376463+0000","last_change":"2024-04-02T04:47:05.388983+0000","last_active":"2024-04-02T04:47:08.376463+0000","last_peered":"2024-04-02T04:47:08.376463+0000","last_clean":"2024-04-02T04:47:08.376463+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:47:08.376463+0000","last_undegraded":"2024-04-02T04:47:08.376463+0000","last_fullsized":"2024-04-02T04:47:08.376463+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.388768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.388768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.419043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328929+0000","last_change":"2024-04-02T04:46:50.400686+0000","last_active":"2024-04-02T04:47:08.328929+0000","last_peered":"2024-04-02T04:47:08.328929+0000","last_clean":"2024-04-02T04:47:08.328929+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:47:08.328929+0000","last_undegraded":"2024-04-02T04:47:08.328929+0000","last_fullsized":"2024-04-02T04:47:08.328929+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:46:50.400390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.328220+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336782+0000","last_change":"2024-04-02T04:46:46.307232+0000","last_active":"2024-04-02T04:47:07.336782+0000","last_peered":"2024-04-02T04:47:07.336782+0000","last_clean":"2024-04-02T04:47:07.336782+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:47:07.336782+0000","last_undegraded":"2024-04-02T04:47:07.336782+0000","last_fullsized":"2024-04-02T04:47:07.336782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.306862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.306862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.499577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885450+0000","last_change":"2024-04-02T04:46:44.178572+0000","last_active":"2024-04-02T04:47:07.885450+0000","last_peered":"2024-04-02T04:47:07.885450+0000","last_clean":"2024-04-02T04:47:07.885450+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:47:07.885450+0000","last_undegraded":"2024-04-02T04:47:07.885450+0000","last_fullsized":"2024-04-02T04:47:07.885450+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:46:44.178327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.178327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.305669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.378113+0000","last_change":"2024-04-02T04:45:44.287628+0000","last_active":"2024-04-02T04:47:08.378113+0000","last_peered":"2024-04-02T04:47:08.378113+0000","last_clean":"2024-04-02T04:47:08.378113+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:47:08.378113+0000","last_undegraded":"2024-04-02T04:47:08.378113+0000","last_fullsized":"2024-04-02T04:47:08.378113+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:44.287411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:44.287411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'36","reported_seq":201,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339529+0000","last_change":"2024-04-02T04:46:29.368990+0000","last_active":"2024-04-02T04:47:07.339529+0000","last_peered":"2024-04-02T04:47:07.339529+0000","last_clean":"2024-04-02T04:47:07.339529+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:47:07.339529+0000","last_undegraded":"2024-04-02T04:47:07.339529+0000","last_fullsized":"2024-04-02T04:47:07.339529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'36","last_scrub_stamp":"2024-04-02T04:46:29.368745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:29.368745+0000","objects_scrubbed":11,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.702277+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"85'11","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328130+0000","last_change":"2024-04-02T04:46:23.272159+0000","last_active":"2024-04-02T04:47:08.328130+0000","last_peered":"2024-04-02T04:47:08.328130+0000","last_clean":"2024-04-02T04:47:08.328130+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:47:08.328130+0000","last_undegraded":"2024-04-02T04:47:08.328130+0000","last_fullsized":"2024-04-02T04:47:08.328130+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:23.271930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:23.271930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.461094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330415+0000","last_change":"2024-04-02T04:47:00.939593+0000","last_active":"2024-04-02T04:47:07.330415+0000","last_peered":"2024-04-02T04:47:07.330415+0000","last_clean":"2024-04-02T04:47:07.330415+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:47:07.330415+0000","last_undegraded":"2024-04-02T04:47:07.330415+0000","last_fullsized":"2024-04-02T04:47:07.330415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.939352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.939352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.488733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.329906+0000","last_change":"2024-04-02T04:46:49.854837+0000","last_active":"2024-04-02T04:47:07.329906+0000","last_peered":"2024-04-02T04:47:07.329906+0000","last_clean":"2024-04-02T04:47:07.329906+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:47:07.329906+0000","last_undegraded":"2024-04-02T04:47:07.329906+0000","last_fullsized":"2024-04-02T04:47:07.329906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:49.854618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.854618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.738084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339496+0000","last_change":"2024-04-02T04:46:39.351367+0000","last_active":"2024-04-02T04:47:07.339496+0000","last_peered":"2024-04-02T04:47:07.339496+0000","last_clean":"2024-04-02T04:47:07.339496+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:47:07.339496+0000","last_undegraded":"2024-04-02T04:47:07.339496+0000","last_fullsized":"2024-04-02T04:47:07.339496+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:39.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.351151+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.995435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":139,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342662+0000","last_change":"2024-04-02T04:47:04.162419+0000","last_active":"2024-04-02T04:47:07.342662+0000","last_peered":"2024-04-02T04:47:07.342662+0000","last_clean":"2024-04-02T04:47:07.342662+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:47:07.342662+0000","last_undegraded":"2024-04-02T04:47:07.342662+0000","last_fullsized":"2024-04-02T04:47:07.342662+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:47:04.162146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.162146+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.370811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336594+0000","last_change":"2024-04-02T04:46:25.694818+0000","last_active":"2024-04-02T04:47:07.336594+0000","last_peered":"2024-04-02T04:47:07.336594+0000","last_clean":"2024-04-02T04:47:07.336594+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:47:07.336594+0000","last_undegraded":"2024-04-02T04:47:07.336594+0000","last_fullsized":"2024-04-02T04:47:07.336594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:46:25.694340+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:28.430989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377234+0000","last_change":"2024-04-02T04:46:22.438361+0000","last_active":"2024-04-02T04:47:08.377234+0000","last_peered":"2024-04-02T04:47:08.377234+0000","last_clean":"2024-04-02T04:47:08.377234+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:47:08.377234+0000","last_undegraded":"2024-04-02T04:47:08.377234+0000","last_fullsized":"2024-04-02T04:47:08.377234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.438115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.444291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":282,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:11.636101+0000","last_change":"2024-04-02T04:46:43.873401+0000","last_active":"2024-04-02T04:47:11.636101+0000","last_peered":"2024-04-02T04:47:11.636101+0000","last_clean":"2024-04-02T04:47:11.636101+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:47:11.636101+0000","last_undegraded":"2024-04-02T04:47:11.636101+0000","last_fullsized":"2024-04-02T04:47:11.636101+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:46:43.873013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.873013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:08.482316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":171,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.887308+0000","last_change":"2024-04-02T04:46:27.077097+0000","last_active":"2024-04-02T04:47:07.887308+0000","last_peered":"2024-04-02T04:47:07.887308+0000","last_clean":"2024-04-02T04:47:07.887308+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:47:07.887308+0000","last_undegraded":"2024-04-02T04:47:07.887308+0000","last_fullsized":"2024-04-02T04:47:07.887308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.353345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326677+0000","last_change":"2024-04-02T04:46:49.354576+0000","last_active":"2024-04-02T04:47:08.326677+0000","last_peered":"2024-04-02T04:47:08.326677+0000","last_clean":"2024-04-02T04:47:08.326677+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:47:08.326677+0000","last_undegraded":"2024-04-02T04:47:08.326677+0000","last_fullsized":"2024-04-02T04:47:08.326677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:46:49.354293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.354293+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.930955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330447+0000","last_change":"2024-04-02T04:46:00.849416+0000","last_active":"2024-04-02T04:47:07.330447+0000","last_peered":"2024-04-02T04:47:07.330447+0000","last_clean":"2024-04-02T04:47:07.330447+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:47:07.330447+0000","last_undegraded":"2024-04-02T04:47:07.330447+0000","last_fullsized":"2024-04-02T04:47:07.330447+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.666708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339644+0000","last_change":"2024-04-02T04:46:46.969136+0000","last_active":"2024-04-02T04:47:07.339644+0000","last_peered":"2024-04-02T04:47:07.339644+0000","last_clean":"2024-04-02T04:47:07.339644+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:47:07.339644+0000","last_undegraded":"2024-04-02T04:47:07.339644+0000","last_fullsized":"2024-04-02T04:47:07.339644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.968921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.968921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.377963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":145,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.331996+0000","last_change":"2024-04-02T04:46:43.734784+0000","last_active":"2024-04-02T04:47:07.331996+0000","last_peered":"2024-04-02T04:47:07.331996+0000","last_clean":"2024-04-02T04:47:07.331996+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:47:07.331996+0000","last_undegraded":"2024-04-02T04:47:07.331996+0000","last_fullsized":"2024-04-02T04:47:07.331996+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:43.734423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.734423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:07.446921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376701+0000","last_change":"2024-04-02T04:46:14.464223+0000","last_active":"2024-04-02T04:47:08.376701+0000","last_peered":"2024-04-02T04:47:08.376701+0000","last_clean":"2024-04-02T04:47:08.376701+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:47:08.376701+0000","last_undegraded":"2024-04-02T04:47:08.376701+0000","last_fullsized":"2024-04-02T04:47:08.376701+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:14.463957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.463957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:20.901600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330247+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:47:07.330247+0000","last_peered":"2024-04-02T04:47:07.330247+0000","last_clean":"2024-04-02T04:47:07.330247+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:47:07.330247+0000","last_undegraded":"2024-04-02T04:47:07.330247+0000","last_fullsized":"2024-04-02T04:47:07.330247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.759911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328691+0000","last_change":"2024-04-02T04:46:47.400298+0000","last_active":"2024-04-02T04:47:08.328691+0000","last_peered":"2024-04-02T04:47:08.328691+0000","last_clean":"2024-04-02T04:47:08.328691+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:47:08.328691+0000","last_undegraded":"2024-04-02T04:47:08.328691+0000","last_fullsized":"2024-04-02T04:47:08.328691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:47.400068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.400068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.923320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":35,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380526+0000","last_change":"2024-04-02T04:46:40.636137+0000","last_active":"2024-04-02T04:47:07.380526+0000","last_peered":"2024-04-02T04:47:07.380526+0000","last_clean":"2024-04-02T04:47:07.380526+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:47:07.380526+0000","last_undegraded":"2024-04-02T04:47:07.380526+0000","last_fullsized":"2024-04-02T04:47:07.380526+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.636006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.636006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:54.838665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337828+0000","last_change":"2024-04-02T04:46:37.280588+0000","last_active":"2024-04-02T04:47:07.337828+0000","last_peered":"2024-04-02T04:47:07.337828+0000","last_clean":"2024-04-02T04:47:07.337828+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:47:07.337828+0000","last_undegraded":"2024-04-02T04:47:07.337828+0000","last_fullsized":"2024-04-02T04:47:07.337828+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:37.280354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.280354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.558805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":158,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336998+0000","last_change":"2024-04-02T04:46:16.675668+0000","last_active":"2024-04-02T04:47:07.336998+0000","last_peered":"2024-04-02T04:47:07.336998+0000","last_clean":"2024-04-02T04:47:07.336998+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:47:07.336998+0000","last_undegraded":"2024-04-02T04:47:07.336998+0000","last_fullsized":"2024-04-02T04:47:07.336998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:46:16.675527+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.675527+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:25.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.862137+0000","last_change":"2024-04-02T04:47:08.862194+0000","last_active":"2024-04-02T04:47:08.862137+0000","last_peered":"2024-04-02T04:47:08.862137+0000","last_clean":"2024-04-02T04:47:08.862137+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:47:08.862137+0000","last_undegraded":"2024-04-02T04:47:08.862137+0000","last_fullsized":"2024-04-02T04:47:08.862137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:08.861978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.861978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.953269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338856+0000","last_change":"2024-04-02T04:46:09.359995+0000","last_active":"2024-04-02T04:47:07.338856+0000","last_peered":"2024-04-02T04:47:07.338856+0000","last_clean":"2024-04-02T04:47:07.338856+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:47:07.338856+0000","last_undegraded":"2024-04-02T04:47:07.338856+0000","last_fullsized":"2024-04-02T04:47:07.338856+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:09.359739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:09.359739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.174067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338687+0000","last_change":"2024-04-02T04:46:18.673713+0000","last_active":"2024-04-02T04:47:07.338687+0000","last_peered":"2024-04-02T04:47:07.338687+0000","last_clean":"2024-04-02T04:47:07.338687+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:47:07.338687+0000","last_undegraded":"2024-04-02T04:47:07.338687+0000","last_fullsized":"2024-04-02T04:47:07.338687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.673548+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.102258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330963+0000","last_change":"2024-04-02T04:46:55.945141+0000","last_active":"2024-04-02T04:47:07.330963+0000","last_peered":"2024-04-02T04:47:07.330963+0000","last_clean":"2024-04-02T04:47:07.330963+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:47:07.330963+0000","last_undegraded":"2024-04-02T04:47:07.330963+0000","last_fullsized":"2024-04-02T04:47:07.330963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.944866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.944866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.910143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380266+0000","last_change":"2024-04-02T04:46:30.524944+0000","last_active":"2024-04-02T04:47:07.380266+0000","last_peered":"2024-04-02T04:47:07.380266+0000","last_clean":"2024-04-02T04:47:07.380266+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:47:07.380266+0000","last_undegraded":"2024-04-02T04:47:07.380266+0000","last_fullsized":"2024-04-02T04:47:07.380266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:30.524741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:30.524741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:37.418739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"85'11","reported_seq":145,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.331649+0000","last_change":"2024-04-02T04:46:32.837238+0000","last_active":"2024-04-02T04:47:07.331649+0000","last_peered":"2024-04-02T04:47:07.331649+0000","last_clean":"2024-04-02T04:47:07.331649+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:47:07.331649+0000","last_undegraded":"2024-04-02T04:47:07.331649+0000","last_fullsized":"2024-04-02T04:47:07.331649+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:32.836976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.836976+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.523209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377238+0000","last_change":"2024-04-02T04:47:02.316549+0000","last_active":"2024-04-02T04:47:08.377238+0000","last_peered":"2024-04-02T04:47:08.377238+0000","last_clean":"2024-04-02T04:47:08.377238+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:47:08.377238+0000","last_undegraded":"2024-04-02T04:47:08.377238+0000","last_fullsized":"2024-04-02T04:47:08.377238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:02.316325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.316325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.699710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":144,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885725+0000","last_change":"2024-04-02T04:46:26.071505+0000","last_active":"2024-04-02T04:47:07.885725+0000","last_peered":"2024-04-02T04:47:07.885725+0000","last_clean":"2024-04-02T04:47:07.885725+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:47:07.885725+0000","last_undegraded":"2024-04-02T04:47:07.885725+0000","last_fullsized":"2024-04-02T04:47:07.885725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:46:26.071249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.071249+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.710130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375731+0000","last_change":"2024-04-02T04:46:24.365206+0000","last_active":"2024-04-02T04:47:08.375731+0000","last_peered":"2024-04-02T04:47:08.375731+0000","last_clean":"2024-04-02T04:47:08.375731+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:47:08.375731+0000","last_undegraded":"2024-04-02T04:47:08.375731+0000","last_fullsized":"2024-04-02T04:47:08.375731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:24.365001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.365001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:27.300770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.379737+0000","last_change":"2024-04-02T04:46:41.673523+0000","last_active":"2024-04-02T04:47:07.379737+0000","last_peered":"2024-04-02T04:47:07.379737+0000","last_clean":"2024-04-02T04:47:07.379737+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:47:07.379737+0000","last_undegraded":"2024-04-02T04:47:07.379737+0000","last_fullsized":"2024-04-02T04:47:07.379737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:41.673448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.673448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.552075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376227+0000","last_change":"2024-04-02T04:45:47.331295+0000","last_active":"2024-04-02T04:47:08.376227+0000","last_peered":"2024-04-02T04:47:08.376227+0000","last_clean":"2024-04-02T04:47:08.376227+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:47:08.376227+0000","last_undegraded":"2024-04-02T04:47:08.376227+0000","last_fullsized":"2024-04-02T04:47:08.376227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:47.331035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.331035+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.790268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885047+0000","last_change":"2024-04-02T04:46:51.147376+0000","last_active":"2024-04-02T04:47:07.885047+0000","last_peered":"2024-04-02T04:47:07.885047+0000","last_clean":"2024-04-02T04:47:07.885047+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:47:07.885047+0000","last_undegraded":"2024-04-02T04:47:07.885047+0000","last_fullsized":"2024-04-02T04:47:07.885047+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:51.147155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:51.147155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.727249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326990+0000","last_change":"2024-04-02T04:45:58.497534+0000","last_active":"2024-04-02T04:47:08.326990+0000","last_peered":"2024-04-02T04:47:08.326990+0000","last_clean":"2024-04-02T04:47:08.326990+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:47:08.326990+0000","last_undegraded":"2024-04-02T04:47:08.326990+0000","last_fullsized":"2024-04-02T04:47:08.326990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:58.497439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:58.497439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.018970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341175+0000","last_change":"2024-04-02T04:46:16.442450+0000","last_active":"2024-04-02T04:47:07.341175+0000","last_peered":"2024-04-02T04:47:07.341175+0000","last_clean":"2024-04-02T04:47:07.341175+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:47:07.341175+0000","last_undegraded":"2024-04-02T04:47:07.341175+0000","last_fullsized":"2024-04-02T04:47:07.341175+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.442234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:22.146814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326874+0000","last_change":"2024-04-02T04:46:37.302510+0000","last_active":"2024-04-02T04:47:08.326874+0000","last_peered":"2024-04-02T04:47:08.326874+0000","last_clean":"2024-04-02T04:47:08.326874+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:47:08.326874+0000","last_undegraded":"2024-04-02T04:47:08.326874+0000","last_fullsized":"2024-04-02T04:47:08.326874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.302252+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.744491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329291+0000","last_change":"2024-04-02T04:46:18.276638+0000","last_active":"2024-04-02T04:47:08.329291+0000","last_peered":"2024-04-02T04:47:08.329291+0000","last_clean":"2024-04-02T04:47:08.329291+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:47:08.329291+0000","last_undegraded":"2024-04-02T04:47:08.329291+0000","last_fullsized":"2024-04-02T04:47:08.329291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:18.276507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.276507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.306028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:12.345448+0000","last_change":"2024-04-02T04:47:12.345553+0000","last_active":"2024-04-02T04:47:12.345448+0000","last_peered":"2024-04-02T04:47:12.345448+0000","last_clean":"2024-04-02T04:47:12.345448+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:47:12.345448+0000","last_undegraded":"2024-04-02T04:47:12.345448+0000","last_fullsized":"2024-04-02T04:47:12.345448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.345237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.345237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.300398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380228+0000","last_change":"2024-04-02T04:46:33.616502+0000","last_active":"2024-04-02T04:47:07.380228+0000","last_peered":"2024-04-02T04:47:07.380228+0000","last_clean":"2024-04-02T04:47:07.380228+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:47:07.380228+0000","last_undegraded":"2024-04-02T04:47:07.380228+0000","last_fullsized":"2024-04-02T04:47:07.380228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:33.616300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.616300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.778531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":124,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375653+0000","last_change":"2024-04-02T04:46:52.343834+0000","last_active":"2024-04-02T04:47:08.375653+0000","last_peered":"2024-04-02T04:47:08.375653+0000","last_clean":"2024-04-02T04:47:08.375653+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:47:08.375653+0000","last_undegraded":"2024-04-02T04:47:08.375653+0000","last_fullsized":"2024-04-02T04:47:08.375653+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:52.343572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.343572+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.554866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327168+0000","last_change":"2024-04-02T04:46:21.245136+0000","last_active":"2024-04-02T04:47:08.327168+0000","last_peered":"2024-04-02T04:47:08.327168+0000","last_clean":"2024-04-02T04:47:08.327168+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:47:08.327168+0000","last_undegraded":"2024-04-02T04:47:08.327168+0000","last_fullsized":"2024-04-02T04:47:08.327168+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.244838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.244838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.980660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":176,"num_read_kb":171,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743474,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28324,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716828,"statfs":{"total":95995035648,"available":95966031872,"internally_reserved":0,"allocated":1339392,"data_stored":967890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103229,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28296,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716856,"statfs":{"total":95995035648,"available":95966060544,"internally_reserved":0,"allocated":1306624,"data_stored":965703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":376415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712680,"statfs":{"total":95995035648,"available":95961784320,"internally_reserved":0,"allocated":1323008,"data_stored":966367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378726,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":716800,"data_stored":376651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051676,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":712704,"data_stored":375716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822738,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":753664,"data_stored":376644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462982,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":716800,"data_stored":376560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:47:15.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:47:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:15 smithi098 bash[23790]: cluster 2024-04-02T04:47:13.789743+0000 mgr.y (mgr.14538) 123 : cluster 0 pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:15 smithi098 bash[23790]: cluster 2024-04-02T04:47:14.334859+0000 osd.3 (osd.3) 107 : cluster 0 6.1b deep-scrub starts 2024-04-02T04:47:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:15 smithi098 bash[23790]: cluster 2024-04-02T04:47:14.337168+0000 osd.3 (osd.3) 108 : cluster 0 6.1b deep-scrub ok 2024-04-02T04:47:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:15 smithi098 bash[23790]: audit 2024-04-02T04:47:14.462019+0000 mgr.y (mgr.14538) 124 : audit 0 from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:47:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[21528]: cluster 2024-04-02T04:47:13.789743+0000 mgr.y (mgr.14538) 123 : cluster 0 pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[21528]: cluster 2024-04-02T04:47:14.334859+0000 osd.3 (osd.3) 107 : cluster 0 6.1b deep-scrub starts 2024-04-02T04:47:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[21528]: cluster 2024-04-02T04:47:14.337168+0000 osd.3 (osd.3) 108 : cluster 0 6.1b deep-scrub ok 2024-04-02T04:47:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[21528]: audit 2024-04-02T04:47:14.462019+0000 mgr.y (mgr.14538) 124 : audit 0 from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:47:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[28904]: cluster 2024-04-02T04:47:13.789743+0000 mgr.y (mgr.14538) 123 : cluster 0 pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[28904]: cluster 2024-04-02T04:47:14.334859+0000 osd.3 (osd.3) 107 : cluster 0 6.1b deep-scrub starts 2024-04-02T04:47:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[28904]: cluster 2024-04-02T04:47:14.337168+0000 osd.3 (osd.3) 108 : cluster 0 6.1b deep-scrub ok 2024-04-02T04:47:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:15 smithi069 bash[28904]: audit 2024-04-02T04:47:14.462019+0000 mgr.y (mgr.14538) 124 : audit 0 from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:47:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:16 smithi098 bash[23790]: cluster 2024-04-02T04:47:14.879109+0000 osd.2 (osd.2) 45 : cluster 0 2.1 scrub starts 2024-04-02T04:47:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:16 smithi098 bash[23790]: cluster 2024-04-02T04:47:14.881015+0000 osd.2 (osd.2) 46 : cluster 0 2.1 scrub ok 2024-04-02T04:47:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:16 smithi098 bash[23790]: cluster 2024-04-02T04:47:14.892592+0000 osd.4 (osd.4) 89 : cluster 0 2.f scrub starts 2024-04-02T04:47:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:16 smithi098 bash[23790]: cluster 2024-04-02T04:47:14.895801+0000 osd.4 (osd.4) 90 : cluster 0 2.f scrub ok 2024-04-02T04:47:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:16 smithi098 bash[23790]: cluster 2024-04-02T04:47:15.263955+0000 osd.1 (osd.1) 85 : cluster 0 2.a deep-scrub starts 2024-04-02T04:47:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:16 smithi098 bash[23790]: cluster 2024-04-02T04:47:15.265486+0000 osd.1 (osd.1) 86 : cluster 0 2.a deep-scrub ok 2024-04-02T04:47:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[21528]: cluster 2024-04-02T04:47:14.879109+0000 osd.2 (osd.2) 45 : cluster 0 2.1 scrub starts 2024-04-02T04:47:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[21528]: cluster 2024-04-02T04:47:14.881015+0000 osd.2 (osd.2) 46 : cluster 0 2.1 scrub ok 2024-04-02T04:47:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[21528]: cluster 2024-04-02T04:47:14.892592+0000 osd.4 (osd.4) 89 : cluster 0 2.f scrub starts 2024-04-02T04:47:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[21528]: cluster 2024-04-02T04:47:14.895801+0000 osd.4 (osd.4) 90 : cluster 0 2.f scrub ok 2024-04-02T04:47:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[21528]: cluster 2024-04-02T04:47:15.263955+0000 osd.1 (osd.1) 85 : cluster 0 2.a deep-scrub starts 2024-04-02T04:47:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[21528]: cluster 2024-04-02T04:47:15.265486+0000 osd.1 (osd.1) 86 : cluster 0 2.a deep-scrub ok 2024-04-02T04:47:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[28904]: cluster 2024-04-02T04:47:14.879109+0000 osd.2 (osd.2) 45 : cluster 0 2.1 scrub starts 2024-04-02T04:47:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[28904]: cluster 2024-04-02T04:47:14.881015+0000 osd.2 (osd.2) 46 : cluster 0 2.1 scrub ok 2024-04-02T04:47:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[28904]: cluster 2024-04-02T04:47:14.892592+0000 osd.4 (osd.4) 89 : cluster 0 2.f scrub starts 2024-04-02T04:47:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[28904]: cluster 2024-04-02T04:47:14.895801+0000 osd.4 (osd.4) 90 : cluster 0 2.f scrub ok 2024-04-02T04:47:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[28904]: cluster 2024-04-02T04:47:15.263955+0000 osd.1 (osd.1) 85 : cluster 0 2.a deep-scrub starts 2024-04-02T04:47:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:16 smithi069 bash[28904]: cluster 2024-04-02T04:47:15.265486+0000 osd.1 (osd.1) 86 : cluster 0 2.a deep-scrub ok 2024-04-02T04:47:17.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:17 smithi098 bash[23790]: cluster 2024-04-02T04:47:15.790440+0000 mgr.y (mgr.14538) 125 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:17.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:17 smithi069 bash[21528]: cluster 2024-04-02T04:47:15.790440+0000 mgr.y (mgr.14538) 125 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:17.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:17 smithi069 bash[28904]: cluster 2024-04-02T04:47:15.790440+0000 mgr.y (mgr.14538) 125 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:18 smithi069 bash[21528]: cluster 2024-04-02T04:47:17.468436+0000 osd.7 (osd.7) 81 : cluster 0 2.5 scrub starts 2024-04-02T04:47:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:18 smithi069 bash[21528]: cluster 2024-04-02T04:47:17.471087+0000 osd.7 (osd.7) 82 : cluster 0 2.5 scrub ok 2024-04-02T04:47:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:18 smithi069 bash[28904]: cluster 2024-04-02T04:47:17.468436+0000 osd.7 (osd.7) 81 : cluster 0 2.5 scrub starts 2024-04-02T04:47:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:18 smithi069 bash[28904]: cluster 2024-04-02T04:47:17.471087+0000 osd.7 (osd.7) 82 : cluster 0 2.5 scrub ok 2024-04-02T04:47:18.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:18 smithi098 bash[23790]: cluster 2024-04-02T04:47:17.468436+0000 osd.7 (osd.7) 81 : cluster 0 2.5 scrub starts 2024-04-02T04:47:18.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:18 smithi098 bash[23790]: cluster 2024-04-02T04:47:17.471087+0000 osd.7 (osd.7) 82 : cluster 0 2.5 scrub ok 2024-04-02T04:47:19.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:47:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:47:19.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:19 smithi098 bash[23790]: cluster 2024-04-02T04:47:17.791852+0000 mgr.y (mgr.14538) 126 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T04:47:19.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:19 smithi098 bash[23790]: cluster 2024-04-02T04:47:18.289026+0000 osd.0 (osd.0) 87 : cluster 0 6.10 scrub starts 2024-04-02T04:47:19.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:19 smithi098 bash[23790]: cluster 2024-04-02T04:47:18.291418+0000 osd.0 (osd.0) 88 : cluster 0 6.10 scrub ok 2024-04-02T04:47:19.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:19 smithi098 bash[23790]: cluster 2024-04-02T04:47:18.356723+0000 osd.3 (osd.3) 109 : cluster 0 2.1b scrub starts 2024-04-02T04:47:19.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:19 smithi098 bash[23790]: cluster 2024-04-02T04:47:18.359088+0000 osd.3 (osd.3) 110 : cluster 0 2.1b scrub ok 2024-04-02T04:47:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[21528]: cluster 2024-04-02T04:47:17.791852+0000 mgr.y (mgr.14538) 126 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T04:47:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[21528]: cluster 2024-04-02T04:47:18.289026+0000 osd.0 (osd.0) 87 : cluster 0 6.10 scrub starts 2024-04-02T04:47:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[21528]: cluster 2024-04-02T04:47:18.291418+0000 osd.0 (osd.0) 88 : cluster 0 6.10 scrub ok 2024-04-02T04:47:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[21528]: cluster 2024-04-02T04:47:18.356723+0000 osd.3 (osd.3) 109 : cluster 0 2.1b scrub starts 2024-04-02T04:47:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[21528]: cluster 2024-04-02T04:47:18.359088+0000 osd.3 (osd.3) 110 : cluster 0 2.1b scrub ok 2024-04-02T04:47:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[28904]: cluster 2024-04-02T04:47:17.791852+0000 mgr.y (mgr.14538) 126 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T04:47:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[28904]: cluster 2024-04-02T04:47:18.289026+0000 osd.0 (osd.0) 87 : cluster 0 6.10 scrub starts 2024-04-02T04:47:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[28904]: cluster 2024-04-02T04:47:18.291418+0000 osd.0 (osd.0) 88 : cluster 0 6.10 scrub ok 2024-04-02T04:47:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[28904]: cluster 2024-04-02T04:47:18.356723+0000 osd.3 (osd.3) 109 : cluster 0 2.1b scrub starts 2024-04-02T04:47:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:19 smithi069 bash[28904]: cluster 2024-04-02T04:47:18.359088+0000 osd.3 (osd.3) 110 : cluster 0 2.1b scrub ok 2024-04-02T04:47:19.867 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:20 smithi098 bash[23790]: cluster 2024-04-02T04:47:19.320618+0000 osd.6 (osd.6) 55 : cluster 0 4.5 scrub starts 2024-04-02T04:47:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:20 smithi098 bash[23790]: cluster 2024-04-02T04:47:19.323083+0000 osd.6 (osd.6) 56 : cluster 0 4.5 scrub ok 2024-04-02T04:47:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:20 smithi098 bash[23790]: cluster 2024-04-02T04:47:19.395926+0000 osd.3 (osd.3) 111 : cluster 0 4.10 scrub starts 2024-04-02T04:47:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:20 smithi098 bash[23790]: cluster 2024-04-02T04:47:19.398322+0000 osd.3 (osd.3) 112 : cluster 0 4.10 scrub ok 2024-04-02T04:47:20.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[21528]: cluster 2024-04-02T04:47:19.320618+0000 osd.6 (osd.6) 55 : cluster 0 4.5 scrub starts 2024-04-02T04:47:20.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[21528]: cluster 2024-04-02T04:47:19.323083+0000 osd.6 (osd.6) 56 : cluster 0 4.5 scrub ok 2024-04-02T04:47:20.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[21528]: cluster 2024-04-02T04:47:19.395926+0000 osd.3 (osd.3) 111 : cluster 0 4.10 scrub starts 2024-04-02T04:47:20.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[21528]: cluster 2024-04-02T04:47:19.398322+0000 osd.3 (osd.3) 112 : cluster 0 4.10 scrub ok 2024-04-02T04:47:20.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[28904]: cluster 2024-04-02T04:47:19.320618+0000 osd.6 (osd.6) 55 : cluster 0 4.5 scrub starts 2024-04-02T04:47:20.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[28904]: cluster 2024-04-02T04:47:19.323083+0000 osd.6 (osd.6) 56 : cluster 0 4.5 scrub ok 2024-04-02T04:47:20.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[28904]: cluster 2024-04-02T04:47:19.395926+0000 osd.3 (osd.3) 111 : cluster 0 4.10 scrub starts 2024-04-02T04:47:20.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:20 smithi069 bash[28904]: cluster 2024-04-02T04:47:19.398322+0000 osd.3 (osd.3) 112 : cluster 0 4.10 scrub ok 2024-04-02T04:47:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:21 smithi098 bash[23790]: cluster 2024-04-02T04:47:19.792482+0000 mgr.y (mgr.14538) 127 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:21 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.289783+0000 osd.4 (osd.4) 91 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:21 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.289921+0000 osd.4 (osd.4) 92 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:21 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.416737+0000 mon.a (mon.0) 918 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-04-02T04:47:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:21 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.422949+0000 osd.7 (osd.7) 83 : cluster 0 6.12 scrub starts 2024-04-02T04:47:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:21 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.426965+0000 osd.7 (osd.7) 84 : cluster 0 6.12 scrub ok 2024-04-02T04:47:21.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[21528]: cluster 2024-04-02T04:47:19.792482+0000 mgr.y (mgr.14538) 127 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:21.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.289783+0000 osd.4 (osd.4) 91 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:21.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.289921+0000 osd.4 (osd.4) 92 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:21.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.416737+0000 mon.a (mon.0) 918 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-04-02T04:47:21.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.422949+0000 osd.7 (osd.7) 83 : cluster 0 6.12 scrub starts 2024-04-02T04:47:21.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.426965+0000 osd.7 (osd.7) 84 : cluster 0 6.12 scrub ok 2024-04-02T04:47:21.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[28904]: cluster 2024-04-02T04:47:19.792482+0000 mgr.y (mgr.14538) 127 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:21.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.289783+0000 osd.4 (osd.4) 91 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:21.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.289921+0000 osd.4 (osd.4) 92 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:21.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.416737+0000 mon.a (mon.0) 918 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-04-02T04:47:21.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.422949+0000 osd.7 (osd.7) 83 : cluster 0 6.12 scrub starts 2024-04-02T04:47:21.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:21 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.426965+0000 osd.7 (osd.7) 84 : cluster 0 6.12 scrub ok 2024-04-02T04:47:22.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/1385899130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c06a900 msgr2=0x7ff07c06ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:22.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- 172.21.15.69:0/1385899130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c06a900 0x7ff07c06ad80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009990 tx=0x7ff06c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/1385899130 shutdown_connections 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- 172.21.15.69:0/1385899130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 0x7ff07c10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- 172.21.15.69:0/1385899130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff07c108400 0x7ff07c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- 172.21.15.69:0/1385899130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c06a900 0x7ff07c06ad80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/1385899130 >> 172.21.15.69:0/1385899130 conn(0x7ff07c078030 msgr2=0x7ff07c078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/1385899130 shutdown_connections 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/1385899130 wait complete. 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 librados: starting msgr at 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 librados: starting objecter 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 Processor -- start 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- start start 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 librados: setting wanted keys 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 librados: calling monclient init 2024-04-02T04:47:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff07c06a900 0x7ff07c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 0x7ff07c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 0x7ff07c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c111b30 con 0x7ff07c108400 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1119b0 con 0x7ff07c06b2c0 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff07c111cb0 con 0x7ff07c06a900 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 0x7ff07c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff07bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff07c06a900 0x7ff07c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 0x7ff07c19c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53382/0 (socket says 172.21.15.69:53382) 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 -- 172.21.15.69:0/257342215 learned_addr learned my addr 172.21.15.69:0/257342215 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff07b7fe700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 0x7ff07c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 -- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff07c06a900 msgr2=0x7ff07c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff07c06a900 0x7ff07c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 -- 172.21.15.69:0/257342215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 msgr2=0x7ff07c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 0x7ff07c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 -- 172.21.15.69:0/257342215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c19d0a0 con 0x7ff07c108400 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff07b7fe700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 0x7ff07c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff080a32700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 0x7ff07c19c960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff07400b9e0 tx=0x7ff07400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff0797fa700 10 client.?.objecter ms_handle_connect 0x7ff07c108400 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff0797fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07400d8e0 con 0x7ff07c108400 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff074015420 con 0x7ff07c108400 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07c19d390 con 0x7ff07c108400 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074014630 con 0x7ff07c108400 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 10 client.14733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:47:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07c1a4c10 con 0x7ff07c108400 2024-04-02T04:47:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.609+0000 7ff082495700 1 librados: init done 2024-04-02T04:47:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07c04ff00 con 0x7ff07c108400 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff0740147d0 con 0x7ff07c108400 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff06406f5e0 0x7ff064071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(118..118 src has 64..118) ==== 7537+0+0 (secure 0 0 0) 0x7ff07408d910 con 0x7ff07c108400 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 10 client.14733.objecter ms_dispatch 0x7ff07c001550 osd_map(118..118 src has 64..118) 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 3 client.14733.objecter handle_osd_map got epochs [118,118] > 0 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 3 client.14733.objecter handle_osd_map decoding full epoch 118 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff07bfff700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff06406f5e0 0x7ff064071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 20 client.14733.objecter dump_active .. 0 homeless 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff07bfff700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff06406f5e0 0x7ff064071aa0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff070009680 tx=0x7ff070009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.613+0000 7ff0797fa700 10 client.14733.objecter ms_handle_connect 0x7ff06406f5e0 2024-04-02T04:47:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.617+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff07405aef0 con 0x7ff07c108400 2024-04-02T04:47:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:22 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.995619+0000 osd.3 (osd.3) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:22 smithi098 bash[23790]: cluster 2024-04-02T04:47:20.995744+0000 osd.3 (osd.3) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:22 smithi098 bash[23790]: cluster 2024-04-02T04:47:21.414824+0000 mon.a (mon.0) 919 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-04-02T04:47:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:22 smithi098 bash[23790]: audit 2024-04-02T04:47:22.078789+0000 mon.b (mon.1) 95 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.995619+0000 osd.3 (osd.3) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[28904]: cluster 2024-04-02T04:47:20.995744+0000 osd.3 (osd.3) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[28904]: cluster 2024-04-02T04:47:21.414824+0000 mon.a (mon.0) 919 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-04-02T04:47:22.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[28904]: audit 2024-04-02T04:47:22.078789+0000 mon.b (mon.1) 95 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.995619+0000 osd.3 (osd.3) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:22.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[21528]: cluster 2024-04-02T04:47:20.995744+0000 osd.3 (osd.3) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:22.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[21528]: cluster 2024-04-02T04:47:21.414824+0000 mon.a (mon.0) 919 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-04-02T04:47:22.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:22 smithi069 bash[21528]: audit 2024-04-02T04:47:22.078789+0000 mon.b (mon.1) 95 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.821+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff07c107640 con 0x7ff06406f5e0 2024-04-02T04:47:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.857+0000 7ff0797fa700 1 -- 172.21.15.69:0/257342215 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346847 (secure 0 0 0) 0x7ff07c107640 con 0x7ff06406f5e0 2024-04-02T04:47:22.860 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:47:22.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 10 librados: watch_flush enter 2024-04-02T04:47:22.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 10 librados: watch_flush exit 2024-04-02T04:47:22.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 20 client.14733.objecter shutdown clearing up homeless session... 2024-04-02T04:47:22.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 10 client.14733.objecter successfully canceled tick 2024-04-02T04:47:22.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff06406f5e0 msgr2=0x7ff064071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:22.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff06406f5e0 0x7ff064071aa0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff070009680 tx=0x7ff070009260 comp rx=0 tx=0).stop 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 msgr2=0x7ff07c19c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 0x7ff07c19c960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff07400b9e0 tx=0x7ff07400bcf0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 shutdown_connections 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff06406f5e0 0x7ff064071aa0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff07c108400 0x7ff07c19c960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff07c06a900 0x7ff07c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 --2- 172.21.15.69:0/257342215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07c06b2c0 0x7ff07c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 >> 172.21.15.69:0/257342215 conn(0x7ff07c078030 msgr2=0x7ff07c105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 shutdown_connections 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 -- 172.21.15.69:0/257342215 wait complete. 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:22.861+0000 7ff082495700 1 librados: shutdown 2024-04-02T04:47:22.864 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:47:23.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:47:22 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:47:23.516 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-04-02T04:47:21.792788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1419,"num_read_kb":1233,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":846,"ondisk_log_size":846,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236028,"kb_used_data":7420,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725188,"statfs":{"total":767960285184,"available":767718592512,"internally_reserved":0,"allocated":7598080,"data_stored":4781946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004237"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:14.337370+0000","last_change":"2024-04-02T04:47:14.337462+0000","last_active":"2024-04-02T04:47:14.337370+0000","last_peered":"2024-04-02T04:47:14.337370+0000","last_clean":"2024-04-02T04:47:14.337370+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:47:14.337370+0000","last_undegraded":"2024-04-02T04:47:14.337370+0000","last_fullsized":"2024-04-02T04:47:14.337370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.337178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.950297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336603+0000","last_change":"2024-04-02T04:46:26.737065+0000","last_active":"2024-04-02T04:47:07.336603+0000","last_peered":"2024-04-02T04:47:07.336603+0000","last_clean":"2024-04-02T04:47:07.336603+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:47:07.336603+0000","last_undegraded":"2024-04-02T04:47:07.336603+0000","last_fullsized":"2024-04-02T04:47:07.336603+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:26.736961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.736961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.907303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329660+0000","last_change":"2024-04-02T04:46:21.290286+0000","last_active":"2024-04-02T04:47:08.329660+0000","last_peered":"2024-04-02T04:47:08.329660+0000","last_clean":"2024-04-02T04:47:08.329660+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:47:08.329660+0000","last_undegraded":"2024-04-02T04:47:08.329660+0000","last_fullsized":"2024-04-02T04:47:08.329660+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.290194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.290194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.622357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342725+0000","last_change":"2024-04-02T04:46:39.861495+0000","last_active":"2024-04-02T04:47:07.342725+0000","last_peered":"2024-04-02T04:47:07.342725+0000","last_clean":"2024-04-02T04:47:07.342725+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:47:07.342725+0000","last_undegraded":"2024-04-02T04:47:07.342725+0000","last_fullsized":"2024-04-02T04:47:07.342725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:39.861258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.861258+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.553184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:11.925614+0000","last_change":"2024-04-02T04:47:11.925670+0000","last_active":"2024-04-02T04:47:11.925614+0000","last_peered":"2024-04-02T04:47:11.925614+0000","last_clean":"2024-04-02T04:47:11.925614+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:47:11.925614+0000","last_undegraded":"2024-04-02T04:47:11.925614+0000","last_fullsized":"2024-04-02T04:47:11.925614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:11.925452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:11.925452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.710115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.888993+0000","last_change":"2024-04-02T04:47:07.889053+0000","last_active":"2024-04-02T04:47:07.888993+0000","last_peered":"2024-04-02T04:47:07.888993+0000","last_clean":"2024-04-02T04:47:07.888993+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:47:07.888993+0000","last_undegraded":"2024-04-02T04:47:07.888993+0000","last_fullsized":"2024-04-02T04:47:07.888993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:07.888835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:07.888835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:10.239127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342401+0000","last_change":"2024-04-02T04:46:13.710516+0000","last_active":"2024-04-02T04:47:07.342401+0000","last_peered":"2024-04-02T04:47:07.342401+0000","last_clean":"2024-04-02T04:47:07.342401+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:47:07.342401+0000","last_undegraded":"2024-04-02T04:47:07.342401+0000","last_fullsized":"2024-04-02T04:47:07.342401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:13.710340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:13.710340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.036057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":131,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327382+0000","last_change":"2024-04-02T04:46:45.396999+0000","last_active":"2024-04-02T04:47:08.327382+0000","last_peered":"2024-04-02T04:47:08.327382+0000","last_clean":"2024-04-02T04:47:08.327382+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:47:08.327382+0000","last_undegraded":"2024-04-02T04:47:08.327382+0000","last_fullsized":"2024-04-02T04:47:08.327382+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:45.396760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.396760+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.943901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327368+0000","last_change":"2024-04-02T04:46:39.343919+0000","last_active":"2024-04-02T04:47:08.327368+0000","last_peered":"2024-04-02T04:47:08.327368+0000","last_clean":"2024-04-02T04:47:08.327368+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:47:08.327368+0000","last_undegraded":"2024-04-02T04:47:08.327368+0000","last_fullsized":"2024-04-02T04:47:08.327368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:39.343592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.343592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.312506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330459+0000","last_change":"2024-04-02T04:46:22.812121+0000","last_active":"2024-04-02T04:47:07.330459+0000","last_peered":"2024-04-02T04:47:07.330459+0000","last_clean":"2024-04-02T04:47:07.330459+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:47:07.330459+0000","last_undegraded":"2024-04-02T04:47:07.330459+0000","last_fullsized":"2024-04-02T04:47:07.330459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.811836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.811836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.362979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.378376+0000","last_change":"2024-04-02T04:46:35.292548+0000","last_active":"2024-04-02T04:47:08.378376+0000","last_peered":"2024-04-02T04:47:08.378376+0000","last_clean":"2024-04-02T04:47:08.378376+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:47:08.378376+0000","last_undegraded":"2024-04-02T04:47:08.378376+0000","last_fullsized":"2024-04-02T04:47:08.378376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.292336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.292336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.961526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":123,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326418+0000","last_change":"2024-04-02T04:47:05.285775+0000","last_active":"2024-04-02T04:47:08.326418+0000","last_peered":"2024-04-02T04:47:08.326418+0000","last_clean":"2024-04-02T04:47:08.326418+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:47:08.326418+0000","last_undegraded":"2024-04-02T04:47:08.326418+0000","last_fullsized":"2024-04-02T04:47:08.326418+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:05.285521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.285521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:28.240436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326768+0000","last_change":"2024-04-02T04:46:27.234204+0000","last_active":"2024-04-02T04:47:08.326768+0000","last_peered":"2024-04-02T04:47:08.326768+0000","last_clean":"2024-04-02T04:47:08.326768+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:47:08.326768+0000","last_undegraded":"2024-04-02T04:47:08.326768+0000","last_fullsized":"2024-04-02T04:47:08.326768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.234006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.234006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.363415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330948+0000","last_change":"2024-04-02T04:46:44.775289+0000","last_active":"2024-04-02T04:47:07.330948+0000","last_peered":"2024-04-02T04:47:07.330948+0000","last_clean":"2024-04-02T04:47:07.330948+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:47:07.330948+0000","last_undegraded":"2024-04-02T04:47:07.330948+0000","last_fullsized":"2024-04-02T04:47:07.330948+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.775071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.207519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.387532+0000","last_change":"2024-04-02T04:47:09.387603+0000","last_active":"2024-04-02T04:47:09.387532+0000","last_peered":"2024-04-02T04:47:09.387532+0000","last_clean":"2024-04-02T04:47:09.387532+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:47:09.387532+0000","last_undegraded":"2024-04-02T04:47:09.387532+0000","last_fullsized":"2024-04-02T04:47:09.387532+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:09.387375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.387375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.279958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326488+0000","last_change":"2024-04-02T04:46:51.350464+0000","last_active":"2024-04-02T04:47:08.326488+0000","last_peered":"2024-04-02T04:47:08.326488+0000","last_clean":"2024-04-02T04:47:08.326488+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:47:08.326488+0000","last_undegraded":"2024-04-02T04:47:08.326488+0000","last_fullsized":"2024-04-02T04:47:08.326488+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:46:51.350201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:51.350201+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.520062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342833+0000","last_change":"2024-04-02T04:47:03.158759+0000","last_active":"2024-04-02T04:47:07.342833+0000","last_peered":"2024-04-02T04:47:07.342833+0000","last_clean":"2024-04-02T04:47:07.342833+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:47:07.342833+0000","last_undegraded":"2024-04-02T04:47:07.342833+0000","last_fullsized":"2024-04-02T04:47:07.342833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:03.158541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:03.158541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.522158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342147+0000","last_change":"2024-04-02T04:46:36.942358+0000","last_active":"2024-04-02T04:47:07.342147+0000","last_peered":"2024-04-02T04:47:07.342147+0000","last_clean":"2024-04-02T04:47:07.342147+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:47:07.342147+0000","last_undegraded":"2024-04-02T04:47:07.342147+0000","last_fullsized":"2024-04-02T04:47:07.342147+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.942139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.942139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.526854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338428+0000","last_change":"2024-04-02T04:45:57.720821+0000","last_active":"2024-04-02T04:47:07.338428+0000","last_peered":"2024-04-02T04:47:07.338428+0000","last_clean":"2024-04-02T04:47:07.338428+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:47:07.338428+0000","last_undegraded":"2024-04-02T04:47:07.338428+0000","last_fullsized":"2024-04-02T04:47:07.338428+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:57.720720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:45:57.720720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:17.288812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"85'23","reported_seq":141,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330226+0000","last_change":"2024-04-02T04:46:21.850698+0000","last_active":"2024-04-02T04:47:07.330226+0000","last_peered":"2024-04-02T04:47:07.330226+0000","last_clean":"2024-04-02T04:47:07.330226+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:47:07.330226+0000","last_undegraded":"2024-04-02T04:47:07.330226+0000","last_fullsized":"2024-04-02T04:47:07.330226+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-02T04:46:21.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.850373+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:35.100552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.332050+0000","last_change":"2024-04-02T04:47:01.936520+0000","last_active":"2024-04-02T04:47:07.332050+0000","last_peered":"2024-04-02T04:47:07.332050+0000","last_clean":"2024-04-02T04:47:07.332050+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:47:07.332050+0000","last_undegraded":"2024-04-02T04:47:07.332050+0000","last_fullsized":"2024-04-02T04:47:07.332050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:01.936275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:01.936275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.709309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380137+0000","last_change":"2024-04-02T04:46:32.598626+0000","last_active":"2024-04-02T04:47:07.380137+0000","last_peered":"2024-04-02T04:47:07.380137+0000","last_clean":"2024-04-02T04:47:07.380137+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:47:07.380137+0000","last_undegraded":"2024-04-02T04:47:07.380137+0000","last_fullsized":"2024-04-02T04:47:07.380137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.598421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.598421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.059173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885523+0000","last_change":"2024-04-02T04:46:21.009651+0000","last_active":"2024-04-02T04:47:07.885523+0000","last_peered":"2024-04-02T04:47:07.885523+0000","last_clean":"2024-04-02T04:47:07.885523+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:47:07.885523+0000","last_undegraded":"2024-04-02T04:47:07.885523+0000","last_fullsized":"2024-04-02T04:47:07.885523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.009424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.009424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.532492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":125,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:13.347739+0000","last_change":"2024-04-02T04:47:13.347780+0000","last_active":"2024-04-02T04:47:13.347739+0000","last_peered":"2024-04-02T04:47:13.347739+0000","last_clean":"2024-04-02T04:47:13.347739+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:47:13.347739+0000","last_undegraded":"2024-04-02T04:47:13.347739+0000","last_fullsized":"2024-04-02T04:47:13.347739+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.347670+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.686984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.340619+0000","last_change":"2024-04-02T04:46:56.263880+0000","last_active":"2024-04-02T04:47:07.340619+0000","last_peered":"2024-04-02T04:47:07.340619+0000","last_clean":"2024-04-02T04:47:07.340619+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:47:07.340619+0000","last_undegraded":"2024-04-02T04:47:07.340619+0000","last_fullsized":"2024-04-02T04:47:07.340619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:56.263740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:56.263740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.910160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":150,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.456785+0000","last_change":"2024-04-02T04:46:47.154319+0000","last_active":"2024-04-02T04:47:09.456785+0000","last_peered":"2024-04-02T04:47:09.456785+0000","last_clean":"2024-04-02T04:47:09.456785+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:47:09.456785+0000","last_undegraded":"2024-04-02T04:47:09.456785+0000","last_fullsized":"2024-04-02T04:47:09.456785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:47.154082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.154082+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.920192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337028+0000","last_change":"2024-04-02T04:46:27.777917+0000","last_active":"2024-04-02T04:47:07.337028+0000","last_peered":"2024-04-02T04:47:07.337028+0000","last_clean":"2024-04-02T04:47:07.337028+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:47:07.337028+0000","last_undegraded":"2024-04-02T04:47:07.337028+0000","last_fullsized":"2024-04-02T04:47:07.337028+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:46:27.777679+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.777679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.754491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":128,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336098+0000","last_change":"2024-04-02T04:46:24.726007+0000","last_active":"2024-04-02T04:47:07.336098+0000","last_peered":"2024-04-02T04:47:07.336098+0000","last_clean":"2024-04-02T04:47:07.336098+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:47:07.336098+0000","last_undegraded":"2024-04-02T04:47:07.336098+0000","last_fullsized":"2024-04-02T04:47:07.336098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:24.725734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.725734+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:39.190873+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377528+0000","last_change":"2024-04-02T04:46:42.290326+0000","last_active":"2024-04-02T04:47:08.377528+0000","last_peered":"2024-04-02T04:47:08.377528+0000","last_clean":"2024-04-02T04:47:08.377528+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:47:08.377528+0000","last_undegraded":"2024-04-02T04:47:08.377528+0000","last_fullsized":"2024-04-02T04:47:08.377528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:46:42.290071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.290071+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.053573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329111+0000","last_change":"2024-04-02T04:46:04.466247+0000","last_active":"2024-04-02T04:47:08.329111+0000","last_peered":"2024-04-02T04:47:08.329111+0000","last_clean":"2024-04-02T04:47:08.329111+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:47:08.329111+0000","last_undegraded":"2024-04-02T04:47:08.329111+0000","last_fullsized":"2024-04-02T04:47:08.329111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:04.465964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:04.465964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.060111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327481+0000","last_change":"2024-04-02T04:46:16.163503+0000","last_active":"2024-04-02T04:47:08.327481+0000","last_peered":"2024-04-02T04:47:08.327481+0000","last_clean":"2024-04-02T04:47:08.327481+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:47:08.327481+0000","last_undegraded":"2024-04-02T04:47:08.327481+0000","last_fullsized":"2024-04-02T04:47:08.327481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.163286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.163286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:21.454867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":172,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338096+0000","last_change":"2024-04-02T04:46:59.265375+0000","last_active":"2024-04-02T04:47:07.338096+0000","last_peered":"2024-04-02T04:47:07.338096+0000","last_clean":"2024-04-02T04:47:07.338096+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:47:07.338096+0000","last_undegraded":"2024-04-02T04:47:07.338096+0000","last_fullsized":"2024-04-02T04:47:07.338096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:46:59.265104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:59.265104+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.709345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:13.288474+0000","last_change":"2024-04-02T04:47:13.288506+0000","last_active":"2024-04-02T04:47:13.288474+0000","last_peered":"2024-04-02T04:47:13.288474+0000","last_clean":"2024-04-02T04:47:13.288474+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:47:13.288474+0000","last_undegraded":"2024-04-02T04:47:13.288474+0000","last_fullsized":"2024-04-02T04:47:13.288474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:13.288364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.288364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.927428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:12.271889+0000","last_change":"2024-04-02T04:47:12.271950+0000","last_active":"2024-04-02T04:47:12.271889+0000","last_peered":"2024-04-02T04:47:12.271889+0000","last_clean":"2024-04-02T04:47:12.271889+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:47:12.271889+0000","last_undegraded":"2024-04-02T04:47:12.271889+0000","last_fullsized":"2024-04-02T04:47:12.271889+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.271680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.271680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.801349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.887583+0000","last_change":"2024-04-02T04:46:15.989347+0000","last_active":"2024-04-02T04:47:07.887583+0000","last_peered":"2024-04-02T04:47:07.887583+0000","last_clean":"2024-04-02T04:47:07.887583+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:47:07.887583+0000","last_undegraded":"2024-04-02T04:47:07.887583+0000","last_fullsized":"2024-04-02T04:47:07.887583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:15.989126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:46:15.989126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.092291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326516+0000","last_change":"2024-04-02T04:46:41.326411+0000","last_active":"2024-04-02T04:47:08.326516+0000","last_peered":"2024-04-02T04:47:08.326516+0000","last_clean":"2024-04-02T04:47:08.326516+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:47:08.326516+0000","last_undegraded":"2024-04-02T04:47:08.326516+0000","last_fullsized":"2024-04-02T04:47:08.326516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:46:41.326158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.326158+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.505788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376864+0000","last_change":"2024-04-02T04:45:53.375887+0000","last_active":"2024-04-02T04:47:08.376864+0000","last_peered":"2024-04-02T04:47:08.376864+0000","last_clean":"2024-04-02T04:47:08.376864+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:47:08.376864+0000","last_undegraded":"2024-04-02T04:47:08.376864+0000","last_fullsized":"2024-04-02T04:47:08.376864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:45:53.375671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336100+0000","last_change":"2024-04-02T04:46:10.724569+0000","last_active":"2024-04-02T04:47:07.336100+0000","last_peered":"2024-04-02T04:47:07.336100+0000","last_clean":"2024-04-02T04:47:07.336100+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:47:07.336100+0000","last_undegraded":"2024-04-02T04:47:07.336100+0000","last_fullsized":"2024-04-02T04:47:07.336100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.724363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.724363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:18.641739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329075+0000","last_change":"2024-04-02T04:46:36.285859+0000","last_active":"2024-04-02T04:47:08.329075+0000","last_peered":"2024-04-02T04:47:08.329075+0000","last_clean":"2024-04-02T04:47:08.329075+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:47:08.329075+0000","last_undegraded":"2024-04-02T04:47:08.329075+0000","last_fullsized":"2024-04-02T04:47:08.329075+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.285639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.484787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.333777+0000","last_change":"2024-04-02T04:47:08.333929+0000","last_active":"2024-04-02T04:47:08.333777+0000","last_peered":"2024-04-02T04:47:08.333777+0000","last_clean":"2024-04-02T04:47:08.333777+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:47:08.333777+0000","last_undegraded":"2024-04-02T04:47:08.333777+0000","last_fullsized":"2024-04-02T04:47:08.333777+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:47:08.333516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.333516+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.726461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336174+0000","last_change":"2024-04-02T04:46:38.893014+0000","last_active":"2024-04-02T04:47:07.336174+0000","last_peered":"2024-04-02T04:47:07.336174+0000","last_clean":"2024-04-02T04:47:07.336174+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:47:07.336174+0000","last_undegraded":"2024-04-02T04:47:07.336174+0000","last_fullsized":"2024-04-02T04:47:07.336174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.892773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.892773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.901685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330743+0000","last_change":"2024-04-02T04:46:10.895794+0000","last_active":"2024-04-02T04:47:07.330743+0000","last_peered":"2024-04-02T04:47:07.330743+0000","last_clean":"2024-04-02T04:47:07.330743+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:47:07.330743+0000","last_undegraded":"2024-04-02T04:47:07.330743+0000","last_fullsized":"2024-04-02T04:47:07.330743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.895579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.895579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.482340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341608+0000","last_change":"2024-04-02T04:46:54.269554+0000","last_active":"2024-04-02T04:47:07.341608+0000","last_peered":"2024-04-02T04:47:07.341608+0000","last_clean":"2024-04-02T04:47:07.341608+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:47:07.341608+0000","last_undegraded":"2024-04-02T04:47:07.341608+0000","last_fullsized":"2024-04-02T04:47:07.341608+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:54.269343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:54.269343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.989031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.331176+0000","last_change":"2024-04-02T04:46:37.808452+0000","last_active":"2024-04-02T04:47:07.331176+0000","last_peered":"2024-04-02T04:47:07.331176+0000","last_clean":"2024-04-02T04:47:07.331176+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:47:07.331176+0000","last_undegraded":"2024-04-02T04:47:07.331176+0000","last_fullsized":"2024-04-02T04:47:07.331176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:46:37.808090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.808090+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.632664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328397+0000","last_change":"2024-04-02T04:45:59.476847+0000","last_active":"2024-04-02T04:47:08.328397+0000","last_peered":"2024-04-02T04:47:08.328397+0000","last_clean":"2024-04-02T04:47:08.328397+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:47:08.328397+0000","last_undegraded":"2024-04-02T04:47:08.328397+0000","last_fullsized":"2024-04-02T04:47:08.328397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:59.476642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:59.476642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:16.549126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338577+0000","last_change":"2024-04-02T04:46:53.297425+0000","last_active":"2024-04-02T04:47:07.338577+0000","last_peered":"2024-04-02T04:47:07.338577+0000","last_clean":"2024-04-02T04:47:07.338577+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:47:07.338577+0000","last_undegraded":"2024-04-02T04:47:07.338577+0000","last_fullsized":"2024-04-02T04:47:07.338577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:53.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:53.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.497586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.886217+0000","last_change":"2024-04-02T04:46:34.223691+0000","last_active":"2024-04-02T04:47:07.886217+0000","last_peered":"2024-04-02T04:47:07.886217+0000","last_clean":"2024-04-02T04:47:07.886217+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:47:07.886217+0000","last_undegraded":"2024-04-02T04:47:07.886217+0000","last_fullsized":"2024-04-02T04:47:07.886217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:34.223346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:34.223346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.085223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375455+0000","last_change":"2024-04-02T04:46:27.353748+0000","last_active":"2024-04-02T04:47:08.375455+0000","last_peered":"2024-04-02T04:47:08.375455+0000","last_clean":"2024-04-02T04:47:08.375455+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:47:08.375455+0000","last_undegraded":"2024-04-02T04:47:08.375455+0000","last_fullsized":"2024-04-02T04:47:08.375455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:27.353503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.353503+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.633132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.332501+0000","last_change":"2024-04-02T04:47:05.885138+0000","last_active":"2024-04-02T04:47:07.332501+0000","last_peered":"2024-04-02T04:47:07.332501+0000","last_clean":"2024-04-02T04:47:07.332501+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:47:07.332501+0000","last_undegraded":"2024-04-02T04:47:07.332501+0000","last_fullsized":"2024-04-02T04:47:07.332501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:30.330301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":148,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.456664+0000","last_change":"2024-04-02T04:46:44.346692+0000","last_active":"2024-04-02T04:47:09.456664+0000","last_peered":"2024-04-02T04:47:09.456664+0000","last_clean":"2024-04-02T04:47:09.456664+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:47:09.456664+0000","last_undegraded":"2024-04-02T04:47:09.456664+0000","last_fullsized":"2024-04-02T04:47:09.456664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:44.346318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.092878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327637+0000","last_change":"2024-04-02T04:46:36.244725+0000","last_active":"2024-04-02T04:47:08.327637+0000","last_peered":"2024-04-02T04:47:08.327637+0000","last_clean":"2024-04-02T04:47:08.327637+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:47:08.327637+0000","last_undegraded":"2024-04-02T04:47:08.327637+0000","last_fullsized":"2024-04-02T04:47:08.327637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.244514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.244514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.017937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":116,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326566+0000","last_change":"2024-04-02T04:46:28.203298+0000","last_active":"2024-04-02T04:47:08.326566+0000","last_peered":"2024-04-02T04:47:08.326566+0000","last_clean":"2024-04-02T04:47:08.326566+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:47:08.326566+0000","last_undegraded":"2024-04-02T04:47:08.326566+0000","last_fullsized":"2024-04-02T04:47:08.326566+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:46:28.203156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.203156+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.777551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.295656+0000","last_change":"2024-04-02T04:47:09.295716+0000","last_active":"2024-04-02T04:47:09.295656+0000","last_peered":"2024-04-02T04:47:09.295656+0000","last_clean":"2024-04-02T04:47:09.295656+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:47:09.295656+0000","last_undegraded":"2024-04-02T04:47:09.295656+0000","last_fullsized":"2024-04-02T04:47:09.295656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:09.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.295498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.587769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"85'11","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.461279+0000","last_change":"2024-04-02T04:46:28.775353+0000","last_active":"2024-04-02T04:47:09.461279+0000","last_peered":"2024-04-02T04:47:09.461279+0000","last_clean":"2024-04-02T04:47:09.461279+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:47:09.461279+0000","last_undegraded":"2024-04-02T04:47:09.461279+0000","last_fullsized":"2024-04-02T04:47:09.461279+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:28.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.775071+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:55.875528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327304+0000","last_change":"2024-04-02T04:46:48.355080+0000","last_active":"2024-04-02T04:47:08.327304+0000","last_peered":"2024-04-02T04:47:08.327304+0000","last_clean":"2024-04-02T04:47:08.327304+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:47:08.327304+0000","last_undegraded":"2024-04-02T04:47:08.327304+0000","last_fullsized":"2024-04-02T04:47:08.327304+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:48.354869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:48.354869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.638786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375332+0000","last_change":"2024-04-02T04:46:33.347283+0000","last_active":"2024-04-02T04:47:08.375332+0000","last_peered":"2024-04-02T04:47:08.375332+0000","last_clean":"2024-04-02T04:47:08.375332+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:47:08.375332+0000","last_undegraded":"2024-04-02T04:47:08.375332+0000","last_fullsized":"2024-04-02T04:47:08.375332+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:33.347053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.347053+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.542332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377319+0000","last_change":"2024-04-02T04:47:04.415949+0000","last_active":"2024-04-02T04:47:08.377319+0000","last_peered":"2024-04-02T04:47:08.377319+0000","last_clean":"2024-04-02T04:47:08.377319+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:47:08.377319+0000","last_undegraded":"2024-04-02T04:47:08.377319+0000","last_fullsized":"2024-04-02T04:47:08.377319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:04.415735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.415735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:34.243533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342938+0000","last_change":"2024-04-02T04:46:45.921390+0000","last_active":"2024-04-02T04:47:07.342938+0000","last_peered":"2024-04-02T04:47:07.342938+0000","last_clean":"2024-04-02T04:47:07.342938+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:47:07.342938+0000","last_undegraded":"2024-04-02T04:47:07.342938+0000","last_fullsized":"2024-04-02T04:47:07.342938+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.369948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885087+0000","last_change":"2024-04-02T04:46:28.039798+0000","last_active":"2024-04-02T04:47:07.885087+0000","last_peered":"2024-04-02T04:47:07.885087+0000","last_clean":"2024-04-02T04:47:07.885087+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:47:07.885087+0000","last_undegraded":"2024-04-02T04:47:07.885087+0000","last_fullsized":"2024-04-02T04:47:07.885087+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:28.039566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.039566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:34.163582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.884895+0000","last_change":"2024-04-02T04:46:41.254878+0000","last_active":"2024-04-02T04:47:07.884895+0000","last_peered":"2024-04-02T04:47:07.884895+0000","last_clean":"2024-04-02T04:47:07.884895+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:47:07.884895+0000","last_undegraded":"2024-04-02T04:47:07.884895+0000","last_fullsized":"2024-04-02T04:47:07.884895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:46:41.254633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.254633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.695806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380222+0000","last_change":"2024-04-02T04:46:38.622583+0000","last_active":"2024-04-02T04:47:07.380222+0000","last_peered":"2024-04-02T04:47:07.380222+0000","last_clean":"2024-04-02T04:47:07.380222+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:47:07.380222+0000","last_undegraded":"2024-04-02T04:47:07.380222+0000","last_fullsized":"2024-04-02T04:47:07.380222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.622220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.622220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.379107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328365+0000","last_change":"2024-04-02T04:46:57.429067+0000","last_active":"2024-04-02T04:47:08.328365+0000","last_peered":"2024-04-02T04:47:08.328365+0000","last_clean":"2024-04-02T04:47:08.328365+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:47:08.328365+0000","last_undegraded":"2024-04-02T04:47:08.328365+0000","last_fullsized":"2024-04-02T04:47:08.328365+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.428901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.428901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.996741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.379012+0000","last_change":"2024-04-02T04:46:44.734741+0000","last_active":"2024-04-02T04:47:07.379012+0000","last_peered":"2024-04-02T04:47:07.379012+0000","last_clean":"2024-04-02T04:47:07.379012+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:47:07.379012+0000","last_undegraded":"2024-04-02T04:47:07.379012+0000","last_fullsized":"2024-04-02T04:47:07.379012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.734508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.734508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:47.434405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375304+0000","last_change":"2024-04-02T04:47:00.341837+0000","last_active":"2024-04-02T04:47:08.375304+0000","last_peered":"2024-04-02T04:47:08.375304+0000","last_clean":"2024-04-02T04:47:08.375304+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:47:08.375304+0000","last_undegraded":"2024-04-02T04:47:08.375304+0000","last_fullsized":"2024-04-02T04:47:08.375304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:00.341473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.341473+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:24.505569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338960+0000","last_change":"2024-04-02T04:47:00.098587+0000","last_active":"2024-04-02T04:47:07.338960+0000","last_peered":"2024-04-02T04:47:07.338960+0000","last_clean":"2024-04-02T04:47:07.338960+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:47:07.338960+0000","last_undegraded":"2024-04-02T04:47:07.338960+0000","last_fullsized":"2024-04-02T04:47:07.338960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.098270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.098270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.905960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:14.896017+0000","last_change":"2024-04-02T04:47:14.896155+0000","last_active":"2024-04-02T04:47:14.896017+0000","last_peered":"2024-04-02T04:47:14.896017+0000","last_clean":"2024-04-02T04:47:14.896017+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:47:14.896017+0000","last_undegraded":"2024-04-02T04:47:14.896017+0000","last_fullsized":"2024-04-02T04:47:14.896017+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:47:14.895813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.895813+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.683265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375477+0000","last_change":"2024-04-02T04:46:19.421011+0000","last_active":"2024-04-02T04:47:08.375477+0000","last_peered":"2024-04-02T04:47:08.375477+0000","last_clean":"2024-04-02T04:47:08.375477+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:47:08.375477+0000","last_undegraded":"2024-04-02T04:47:08.375477+0000","last_fullsized":"2024-04-02T04:47:08.375477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:19.420739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:19.420739+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:30.952287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"85'11","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.456487+0000","last_change":"2024-04-02T04:46:27.787700+0000","last_active":"2024-04-02T04:47:09.456487+0000","last_peered":"2024-04-02T04:47:09.456487+0000","last_clean":"2024-04-02T04:47:09.456487+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:47:09.456487+0000","last_undegraded":"2024-04-02T04:47:09.456487+0000","last_fullsized":"2024-04-02T04:47:09.456487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:27.787353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.787353+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:41.371457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328415+0000","last_change":"2024-04-02T04:46:43.414718+0000","last_active":"2024-04-02T04:47:08.328415+0000","last_peered":"2024-04-02T04:47:08.328415+0000","last_clean":"2024-04-02T04:47:08.328415+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:47:08.328415+0000","last_undegraded":"2024-04-02T04:47:08.328415+0000","last_fullsized":"2024-04-02T04:47:08.328415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:43.414511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.414511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:43.745696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338916+0000","last_change":"2024-04-02T04:46:22.411099+0000","last_active":"2024-04-02T04:47:07.338916+0000","last_peered":"2024-04-02T04:47:07.338916+0000","last_clean":"2024-04-02T04:47:07.338916+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:47:07.338916+0000","last_undegraded":"2024-04-02T04:47:07.338916+0000","last_fullsized":"2024-04-02T04:47:07.338916+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:22.411010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.411010+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:32.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":161,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.457171+0000","last_change":"2024-04-02T04:46:14.106481+0000","last_active":"2024-04-02T04:47:09.457171+0000","last_peered":"2024-04-02T04:47:09.457171+0000","last_clean":"2024-04-02T04:47:09.457171+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:47:09.457171+0000","last_undegraded":"2024-04-02T04:47:09.457171+0000","last_fullsized":"2024-04-02T04:47:09.457171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:14.106232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.106232+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.525489+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339469+0000","last_change":"2024-04-02T04:46:36.317753+0000","last_active":"2024-04-02T04:47:07.339469+0000","last_peered":"2024-04-02T04:47:07.339469+0000","last_clean":"2024-04-02T04:47:07.339469+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:47:07.339469+0000","last_undegraded":"2024-04-02T04:47:07.339469+0000","last_fullsized":"2024-04-02T04:47:07.339469+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.317542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.317542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:44.283515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342414+0000","last_change":"2024-04-02T04:47:02.153712+0000","last_active":"2024-04-02T04:47:07.342414+0000","last_peered":"2024-04-02T04:47:07.342414+0000","last_clean":"2024-04-02T04:47:07.342414+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:47:07.342414+0000","last_undegraded":"2024-04-02T04:47:07.342414+0000","last_fullsized":"2024-04-02T04:47:07.342414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:47:02.153481+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.153481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.645312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327385+0000","last_change":"2024-04-02T04:46:57.319477+0000","last_active":"2024-04-02T04:47:08.327385+0000","last_peered":"2024-04-02T04:47:08.327385+0000","last_clean":"2024-04-02T04:47:08.327385+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:47:08.327385+0000","last_undegraded":"2024-04-02T04:47:08.327385+0000","last_fullsized":"2024-04-02T04:47:08.327385+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.319266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.319266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.104997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.331515+0000","last_change":"2024-04-02T04:46:36.775425+0000","last_active":"2024-04-02T04:47:07.331515+0000","last_peered":"2024-04-02T04:47:07.331515+0000","last_clean":"2024-04-02T04:47:07.331515+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:47:07.331515+0000","last_undegraded":"2024-04-02T04:47:07.331515+0000","last_fullsized":"2024-04-02T04:47:07.331515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:46:36.775178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.775178+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.278965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.340002+0000","last_change":"2024-04-02T04:46:32.335647+0000","last_active":"2024-04-02T04:47:07.340002+0000","last_peered":"2024-04-02T04:47:07.340002+0000","last_clean":"2024-04-02T04:47:07.340002+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:47:07.340002+0000","last_undegraded":"2024-04-02T04:47:07.340002+0000","last_fullsized":"2024-04-02T04:47:07.340002+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:32.335539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.335539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.500350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377644+0000","last_change":"2024-04-02T04:46:44.347022+0000","last_active":"2024-04-02T04:47:08.377644+0000","last_peered":"2024-04-02T04:47:08.377644+0000","last_clean":"2024-04-02T04:47:08.377644+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:47:08.377644+0000","last_undegraded":"2024-04-02T04:47:08.377644+0000","last_fullsized":"2024-04-02T04:47:08.377644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.346726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:53.151319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":155,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337064+0000","last_change":"2024-04-02T04:46:55.308858+0000","last_active":"2024-04-02T04:47:07.337064+0000","last_peered":"2024-04-02T04:47:07.337064+0000","last_clean":"2024-04-02T04:47:07.337064+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:47:07.337064+0000","last_undegraded":"2024-04-02T04:47:07.337064+0000","last_fullsized":"2024-04-02T04:47:07.337064+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:46:55.308528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.308528+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.148543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:14.881125+0000","last_change":"2024-04-02T04:47:14.881150+0000","last_active":"2024-04-02T04:47:14.881125+0000","last_peered":"2024-04-02T04:47:14.881125+0000","last_clean":"2024-04-02T04:47:14.881125+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:47:14.881125+0000","last_undegraded":"2024-04-02T04:47:14.881125+0000","last_fullsized":"2024-04-02T04:47:14.881125+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.881031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.881031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:23.207112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.340353+0000","last_change":"2024-04-02T04:46:42.343240+0000","last_active":"2024-04-02T04:47:07.340353+0000","last_peered":"2024-04-02T04:47:07.340353+0000","last_clean":"2024-04-02T04:47:07.340353+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:47:07.340353+0000","last_undegraded":"2024-04-02T04:47:07.340353+0000","last_fullsized":"2024-04-02T04:47:07.340353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:42.343078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.343078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.356250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329563+0000","last_change":"2024-04-02T04:46:35.276714+0000","last_active":"2024-04-02T04:47:08.329563+0000","last_peered":"2024-04-02T04:47:08.329563+0000","last_clean":"2024-04-02T04:47:08.329563+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:47:08.329563+0000","last_undegraded":"2024-04-02T04:47:08.329563+0000","last_fullsized":"2024-04-02T04:47:08.329563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.276502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.276502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.541614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341446+0000","last_change":"2024-04-02T04:46:52.985248+0000","last_active":"2024-04-02T04:47:07.341446+0000","last_peered":"2024-04-02T04:47:07.341446+0000","last_clean":"2024-04-02T04:47:07.341446+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:47:07.341446+0000","last_undegraded":"2024-04-02T04:47:07.341446+0000","last_fullsized":"2024-04-02T04:47:07.341446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:52.984994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.984994+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.008710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327978+0000","last_change":"2024-04-02T04:46:55.305525+0000","last_active":"2024-04-02T04:47:08.327978+0000","last_peered":"2024-04-02T04:47:08.327978+0000","last_clean":"2024-04-02T04:47:08.327978+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:47:08.327978+0000","last_undegraded":"2024-04-02T04:47:08.327978+0000","last_fullsized":"2024-04-02T04:47:08.327978+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.305293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.165521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":360,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376481+0000","last_change":"2024-04-02T04:46:17.433834+0000","last_active":"2024-04-02T04:47:08.376481+0000","last_peered":"2024-04-02T04:47:08.376481+0000","last_clean":"2024-04-02T04:47:08.376481+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:47:08.376481+0000","last_undegraded":"2024-04-02T04:47:08.376481+0000","last_fullsized":"2024-04-02T04:47:08.376481+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:46:17.433563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:46:17.433563+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.028333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.886341+0000","last_change":"2024-04-02T04:46:10.058200+0000","last_active":"2024-04-02T04:47:07.886341+0000","last_peered":"2024-04-02T04:47:07.886341+0000","last_clean":"2024-04-02T04:47:07.886341+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:47:07.886341+0000","last_undegraded":"2024-04-02T04:47:07.886341+0000","last_fullsized":"2024-04-02T04:47:07.886341+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:10.057875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:10.057875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:15.984328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338554+0000","last_change":"2024-04-02T04:46:40.318480+0000","last_active":"2024-04-02T04:47:07.338554+0000","last_peered":"2024-04-02T04:47:07.338554+0000","last_clean":"2024-04-02T04:47:07.338554+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:47:07.338554+0000","last_undegraded":"2024-04-02T04:47:07.338554+0000","last_fullsized":"2024-04-02T04:47:07.338554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.318304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.318304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.789228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326440+0000","last_change":"2024-04-02T04:46:33.251890+0000","last_active":"2024-04-02T04:47:08.326440+0000","last_peered":"2024-04-02T04:47:08.326440+0000","last_clean":"2024-04-02T04:47:08.326440+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:47:08.326440+0000","last_undegraded":"2024-04-02T04:47:08.326440+0000","last_fullsized":"2024-04-02T04:47:08.326440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:46:33.251772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.251772+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.560770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376463+0000","last_change":"2024-04-02T04:47:05.388983+0000","last_active":"2024-04-02T04:47:08.376463+0000","last_peered":"2024-04-02T04:47:08.376463+0000","last_clean":"2024-04-02T04:47:08.376463+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:47:08.376463+0000","last_undegraded":"2024-04-02T04:47:08.376463+0000","last_fullsized":"2024-04-02T04:47:08.376463+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.388768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.388768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.419043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328929+0000","last_change":"2024-04-02T04:46:50.400686+0000","last_active":"2024-04-02T04:47:08.328929+0000","last_peered":"2024-04-02T04:47:08.328929+0000","last_clean":"2024-04-02T04:47:08.328929+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:47:08.328929+0000","last_undegraded":"2024-04-02T04:47:08.328929+0000","last_fullsized":"2024-04-02T04:47:08.328929+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:46:50.400390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.328220+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336782+0000","last_change":"2024-04-02T04:46:46.307232+0000","last_active":"2024-04-02T04:47:07.336782+0000","last_peered":"2024-04-02T04:47:07.336782+0000","last_clean":"2024-04-02T04:47:07.336782+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:47:07.336782+0000","last_undegraded":"2024-04-02T04:47:07.336782+0000","last_fullsized":"2024-04-02T04:47:07.336782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.306862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.306862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.499577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885450+0000","last_change":"2024-04-02T04:46:44.178572+0000","last_active":"2024-04-02T04:47:07.885450+0000","last_peered":"2024-04-02T04:47:07.885450+0000","last_clean":"2024-04-02T04:47:07.885450+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:47:07.885450+0000","last_undegraded":"2024-04-02T04:47:07.885450+0000","last_fullsized":"2024-04-02T04:47:07.885450+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:46:44.178327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.178327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.305669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:17.471344+0000","last_change":"2024-04-02T04:47:17.471458+0000","last_active":"2024-04-02T04:47:17.471344+0000","last_peered":"2024-04-02T04:47:17.471344+0000","last_clean":"2024-04-02T04:47:17.471344+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:47:17.471344+0000","last_undegraded":"2024-04-02T04:47:17.471344+0000","last_fullsized":"2024-04-02T04:47:17.471344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:17.471105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:17.471105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:23.064426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'36","reported_seq":202,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.461096+0000","last_change":"2024-04-02T04:46:29.368990+0000","last_active":"2024-04-02T04:47:09.461096+0000","last_peered":"2024-04-02T04:47:09.461096+0000","last_clean":"2024-04-02T04:47:09.461096+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:47:09.461096+0000","last_undegraded":"2024-04-02T04:47:09.461096+0000","last_fullsized":"2024-04-02T04:47:09.461096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'36","last_scrub_stamp":"2024-04-02T04:46:29.368745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:29.368745+0000","objects_scrubbed":11,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.702277+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"85'11","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.461086+0000","last_change":"2024-04-02T04:46:23.272159+0000","last_active":"2024-04-02T04:47:09.461086+0000","last_peered":"2024-04-02T04:47:09.461086+0000","last_clean":"2024-04-02T04:47:09.461086+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:47:09.461086+0000","last_undegraded":"2024-04-02T04:47:09.461086+0000","last_fullsized":"2024-04-02T04:47:09.461086+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:23.271930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:23.271930+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:29.461094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330415+0000","last_change":"2024-04-02T04:47:00.939593+0000","last_active":"2024-04-02T04:47:07.330415+0000","last_peered":"2024-04-02T04:47:07.330415+0000","last_clean":"2024-04-02T04:47:07.330415+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:47:07.330415+0000","last_undegraded":"2024-04-02T04:47:07.330415+0000","last_fullsized":"2024-04-02T04:47:07.330415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.939352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.939352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.488733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.329906+0000","last_change":"2024-04-02T04:46:49.854837+0000","last_active":"2024-04-02T04:47:07.329906+0000","last_peered":"2024-04-02T04:47:07.329906+0000","last_clean":"2024-04-02T04:47:07.329906+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:47:07.329906+0000","last_undegraded":"2024-04-02T04:47:07.329906+0000","last_fullsized":"2024-04-02T04:47:07.329906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:49.854618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.854618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.738084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339496+0000","last_change":"2024-04-02T04:46:39.351367+0000","last_active":"2024-04-02T04:47:07.339496+0000","last_peered":"2024-04-02T04:47:07.339496+0000","last_clean":"2024-04-02T04:47:07.339496+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:47:07.339496+0000","last_undegraded":"2024-04-02T04:47:07.339496+0000","last_fullsized":"2024-04-02T04:47:07.339496+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:39.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.351151+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.995435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":139,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.342662+0000","last_change":"2024-04-02T04:47:04.162419+0000","last_active":"2024-04-02T04:47:07.342662+0000","last_peered":"2024-04-02T04:47:07.342662+0000","last_clean":"2024-04-02T04:47:07.342662+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:47:07.342662+0000","last_undegraded":"2024-04-02T04:47:07.342662+0000","last_fullsized":"2024-04-02T04:47:07.342662+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:47:04.162146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.162146+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.370811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":148,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.461129+0000","last_change":"2024-04-02T04:46:25.694818+0000","last_active":"2024-04-02T04:47:09.461129+0000","last_peered":"2024-04-02T04:47:09.461129+0000","last_clean":"2024-04-02T04:47:09.461129+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:47:09.461129+0000","last_undegraded":"2024-04-02T04:47:09.461129+0000","last_fullsized":"2024-04-02T04:47:09.461129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:46:25.694340+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:28.430989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377234+0000","last_change":"2024-04-02T04:46:22.438361+0000","last_active":"2024-04-02T04:47:08.377234+0000","last_peered":"2024-04-02T04:47:08.377234+0000","last_clean":"2024-04-02T04:47:08.377234+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:47:08.377234+0000","last_undegraded":"2024-04-02T04:47:08.377234+0000","last_fullsized":"2024-04-02T04:47:08.377234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:46:22.438115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:45.444291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":287,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:16.647756+0000","last_change":"2024-04-02T04:46:43.873401+0000","last_active":"2024-04-02T04:47:16.647756+0000","last_peered":"2024-04-02T04:47:16.647756+0000","last_clean":"2024-04-02T04:47:16.647756+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:47:16.647756+0000","last_undegraded":"2024-04-02T04:47:16.647756+0000","last_fullsized":"2024-04-02T04:47:16.647756+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:46:43.873013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.873013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:08.482316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":181,"num_read_kb":176,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.887308+0000","last_change":"2024-04-02T04:46:27.077097+0000","last_active":"2024-04-02T04:47:07.887308+0000","last_peered":"2024-04-02T04:47:07.887308+0000","last_clean":"2024-04-02T04:47:07.887308+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:47:07.887308+0000","last_undegraded":"2024-04-02T04:47:07.887308+0000","last_fullsized":"2024-04-02T04:47:07.887308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.076884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.076884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.353345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326677+0000","last_change":"2024-04-02T04:46:49.354576+0000","last_active":"2024-04-02T04:47:08.326677+0000","last_peered":"2024-04-02T04:47:08.326677+0000","last_clean":"2024-04-02T04:47:08.326677+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:47:08.326677+0000","last_undegraded":"2024-04-02T04:47:08.326677+0000","last_fullsized":"2024-04-02T04:47:08.326677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:46:49.354293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.354293+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.930955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330447+0000","last_change":"2024-04-02T04:46:00.849416+0000","last_active":"2024-04-02T04:47:07.330447+0000","last_peered":"2024-04-02T04:47:07.330447+0000","last_clean":"2024-04-02T04:47:07.330447+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:47:07.330447+0000","last_undegraded":"2024-04-02T04:47:07.330447+0000","last_fullsized":"2024-04-02T04:47:07.330447+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.666708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.339644+0000","last_change":"2024-04-02T04:46:46.969136+0000","last_active":"2024-04-02T04:47:07.339644+0000","last_peered":"2024-04-02T04:47:07.339644+0000","last_clean":"2024-04-02T04:47:07.339644+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:47:07.339644+0000","last_undegraded":"2024-04-02T04:47:07.339644+0000","last_fullsized":"2024-04-02T04:47:07.339644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.968921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.968921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.377963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.456370+0000","last_change":"2024-04-02T04:46:43.734784+0000","last_active":"2024-04-02T04:47:09.456370+0000","last_peered":"2024-04-02T04:47:09.456370+0000","last_clean":"2024-04-02T04:47:09.456370+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:47:09.456370+0000","last_undegraded":"2024-04-02T04:47:09.456370+0000","last_fullsized":"2024-04-02T04:47:09.456370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:43.734423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.734423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:07.446921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376701+0000","last_change":"2024-04-02T04:46:14.464223+0000","last_active":"2024-04-02T04:47:08.376701+0000","last_peered":"2024-04-02T04:47:08.376701+0000","last_clean":"2024-04-02T04:47:08.376701+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:47:08.376701+0000","last_undegraded":"2024-04-02T04:47:08.376701+0000","last_fullsized":"2024-04-02T04:47:08.376701+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:14.463957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:14.463957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:20.901600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330247+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:47:07.330247+0000","last_peered":"2024-04-02T04:47:07.330247+0000","last_clean":"2024-04-02T04:47:07.330247+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:47:07.330247+0000","last_undegraded":"2024-04-02T04:47:07.330247+0000","last_fullsized":"2024-04-02T04:47:07.330247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.759911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.328691+0000","last_change":"2024-04-02T04:46:47.400298+0000","last_active":"2024-04-02T04:47:08.328691+0000","last_peered":"2024-04-02T04:47:08.328691+0000","last_clean":"2024-04-02T04:47:08.328691+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:47:08.328691+0000","last_undegraded":"2024-04-02T04:47:08.328691+0000","last_fullsized":"2024-04-02T04:47:08.328691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:47.400068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.400068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.923320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":35,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380526+0000","last_change":"2024-04-02T04:46:40.636137+0000","last_active":"2024-04-02T04:47:07.380526+0000","last_peered":"2024-04-02T04:47:07.380526+0000","last_clean":"2024-04-02T04:47:07.380526+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:47:07.380526+0000","last_undegraded":"2024-04-02T04:47:07.380526+0000","last_fullsized":"2024-04-02T04:47:07.380526+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.636006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.636006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:54.838665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.337828+0000","last_change":"2024-04-02T04:46:37.280588+0000","last_active":"2024-04-02T04:47:07.337828+0000","last_peered":"2024-04-02T04:47:07.337828+0000","last_clean":"2024-04-02T04:47:07.337828+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:47:07.337828+0000","last_undegraded":"2024-04-02T04:47:07.337828+0000","last_fullsized":"2024-04-02T04:47:07.337828+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:37.280354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.280354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:40.558805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":158,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.336998+0000","last_change":"2024-04-02T04:46:16.675668+0000","last_active":"2024-04-02T04:47:07.336998+0000","last_peered":"2024-04-02T04:47:07.336998+0000","last_clean":"2024-04-02T04:47:07.336998+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:47:07.336998+0000","last_undegraded":"2024-04-02T04:47:07.336998+0000","last_fullsized":"2024-04-02T04:47:07.336998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:46:16.675527+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.675527+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:25.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.862137+0000","last_change":"2024-04-02T04:47:08.862194+0000","last_active":"2024-04-02T04:47:08.862137+0000","last_peered":"2024-04-02T04:47:08.862137+0000","last_clean":"2024-04-02T04:47:08.862137+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:47:08.862137+0000","last_undegraded":"2024-04-02T04:47:08.862137+0000","last_fullsized":"2024-04-02T04:47:08.862137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:08.861978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.861978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.953269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:15.265556+0000","last_change":"2024-04-02T04:47:15.265577+0000","last_active":"2024-04-02T04:47:15.265556+0000","last_peered":"2024-04-02T04:47:15.265556+0000","last_clean":"2024-04-02T04:47:15.265556+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:47:15.265556+0000","last_undegraded":"2024-04-02T04:47:15.265556+0000","last_fullsized":"2024-04-02T04:47:15.265556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_clean_scrub_stamp":"2024-04-02T04:47:15.265490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:39.021422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.338687+0000","last_change":"2024-04-02T04:46:18.673713+0000","last_active":"2024-04-02T04:47:07.338687+0000","last_peered":"2024-04-02T04:47:07.338687+0000","last_clean":"2024-04-02T04:47:07.338687+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:47:07.338687+0000","last_undegraded":"2024-04-02T04:47:07.338687+0000","last_fullsized":"2024-04-02T04:47:07.338687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.673548+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:38.102258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.330963+0000","last_change":"2024-04-02T04:46:55.945141+0000","last_active":"2024-04-02T04:47:07.330963+0000","last_peered":"2024-04-02T04:47:07.330963+0000","last_clean":"2024-04-02T04:47:07.330963+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:47:07.330963+0000","last_undegraded":"2024-04-02T04:47:07.330963+0000","last_fullsized":"2024-04-02T04:47:07.330963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.944866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.944866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.910143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380266+0000","last_change":"2024-04-02T04:46:30.524944+0000","last_active":"2024-04-02T04:47:07.380266+0000","last_peered":"2024-04-02T04:47:07.380266+0000","last_clean":"2024-04-02T04:47:07.380266+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:47:07.380266+0000","last_undegraded":"2024-04-02T04:47:07.380266+0000","last_fullsized":"2024-04-02T04:47:07.380266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:30.524741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:30.524741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:37.418739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"85'11","reported_seq":146,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:09.456438+0000","last_change":"2024-04-02T04:46:32.837238+0000","last_active":"2024-04-02T04:47:09.456438+0000","last_peered":"2024-04-02T04:47:09.456438+0000","last_clean":"2024-04-02T04:47:09.456438+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:47:09.456438+0000","last_undegraded":"2024-04-02T04:47:09.456438+0000","last_fullsized":"2024-04-02T04:47:09.456438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:32.836976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:32.836976+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.523209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.377238+0000","last_change":"2024-04-02T04:47:02.316549+0000","last_active":"2024-04-02T04:47:08.377238+0000","last_peered":"2024-04-02T04:47:08.377238+0000","last_clean":"2024-04-02T04:47:08.377238+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:47:08.377238+0000","last_undegraded":"2024-04-02T04:47:08.377238+0000","last_fullsized":"2024-04-02T04:47:08.377238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:02.316325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.316325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.699710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":144,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885725+0000","last_change":"2024-04-02T04:46:26.071505+0000","last_active":"2024-04-02T04:47:07.885725+0000","last_peered":"2024-04-02T04:47:07.885725+0000","last_clean":"2024-04-02T04:47:07.885725+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:47:07.885725+0000","last_undegraded":"2024-04-02T04:47:07.885725+0000","last_fullsized":"2024-04-02T04:47:07.885725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:46:26.071249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.071249+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.710130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375731+0000","last_change":"2024-04-02T04:46:24.365206+0000","last_active":"2024-04-02T04:47:08.375731+0000","last_peered":"2024-04-02T04:47:08.375731+0000","last_clean":"2024-04-02T04:47:08.375731+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:47:08.375731+0000","last_undegraded":"2024-04-02T04:47:08.375731+0000","last_fullsized":"2024-04-02T04:47:08.375731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:24.365001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:24.365001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:27.300770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.379737+0000","last_change":"2024-04-02T04:46:41.673523+0000","last_active":"2024-04-02T04:47:07.379737+0000","last_peered":"2024-04-02T04:47:07.379737+0000","last_clean":"2024-04-02T04:47:07.379737+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:47:07.379737+0000","last_undegraded":"2024-04-02T04:47:07.379737+0000","last_fullsized":"2024-04-02T04:47:07.379737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:41.673448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.673448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.552075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.376227+0000","last_change":"2024-04-02T04:45:47.331295+0000","last_active":"2024-04-02T04:47:08.376227+0000","last_peered":"2024-04-02T04:47:08.376227+0000","last_clean":"2024-04-02T04:47:08.376227+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:47:08.376227+0000","last_undegraded":"2024-04-02T04:47:08.376227+0000","last_fullsized":"2024-04-02T04:47:08.376227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:45:47.331035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:45:47.331035+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:13.790268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.885047+0000","last_change":"2024-04-02T04:46:51.147376+0000","last_active":"2024-04-02T04:47:07.885047+0000","last_peered":"2024-04-02T04:47:07.885047+0000","last_clean":"2024-04-02T04:47:07.885047+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:47:07.885047+0000","last_undegraded":"2024-04-02T04:47:07.885047+0000","last_fullsized":"2024-04-02T04:47:07.885047+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:51.147155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:51.147155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.727249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326990+0000","last_change":"2024-04-02T04:45:58.497534+0000","last_active":"2024-04-02T04:47:08.326990+0000","last_peered":"2024-04-02T04:47:08.326990+0000","last_clean":"2024-04-02T04:47:08.326990+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:47:08.326990+0000","last_undegraded":"2024-04-02T04:47:08.326990+0000","last_fullsized":"2024-04-02T04:47:08.326990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:45:58.497439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:45:58.497439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:24.018970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.341175+0000","last_change":"2024-04-02T04:46:16.442450+0000","last_active":"2024-04-02T04:47:07.341175+0000","last_peered":"2024-04-02T04:47:07.341175+0000","last_clean":"2024-04-02T04:47:07.341175+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:47:07.341175+0000","last_undegraded":"2024-04-02T04:47:07.341175+0000","last_fullsized":"2024-04-02T04:47:07.341175+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:46:16.442234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:22.146814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.326874+0000","last_change":"2024-04-02T04:46:37.302510+0000","last_active":"2024-04-02T04:47:08.326874+0000","last_peered":"2024-04-02T04:47:08.326874+0000","last_clean":"2024-04-02T04:47:08.326874+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:47:08.326874+0000","last_undegraded":"2024-04-02T04:47:08.326874+0000","last_fullsized":"2024-04-02T04:47:08.326874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.302252+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:42.744491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":96,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.329291+0000","last_change":"2024-04-02T04:46:18.276638+0000","last_active":"2024-04-02T04:47:08.329291+0000","last_peered":"2024-04-02T04:47:08.329291+0000","last_clean":"2024-04-02T04:47:08.329291+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:47:08.329291+0000","last_undegraded":"2024-04-02T04:47:08.329291+0000","last_fullsized":"2024-04-02T04:47:08.329291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:18.276507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:18.276507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:23.306028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:12.345448+0000","last_change":"2024-04-02T04:47:12.345553+0000","last_active":"2024-04-02T04:47:12.345448+0000","last_peered":"2024-04-02T04:47:12.345448+0000","last_clean":"2024-04-02T04:47:12.345448+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:47:12.345448+0000","last_undegraded":"2024-04-02T04:47:12.345448+0000","last_fullsized":"2024-04-02T04:47:12.345448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.345237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.345237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.300398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:07.380228+0000","last_change":"2024-04-02T04:46:33.616502+0000","last_active":"2024-04-02T04:47:07.380228+0000","last_peered":"2024-04-02T04:47:07.380228+0000","last_clean":"2024-04-02T04:47:07.380228+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:47:07.380228+0000","last_undegraded":"2024-04-02T04:47:07.380228+0000","last_fullsized":"2024-04-02T04:47:07.380228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:33.616300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.616300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.778531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":124,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.375653+0000","last_change":"2024-04-02T04:46:52.343834+0000","last_active":"2024-04-02T04:47:08.375653+0000","last_peered":"2024-04-02T04:47:08.375653+0000","last_clean":"2024-04-02T04:47:08.375653+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:47:08.375653+0000","last_undegraded":"2024-04-02T04:47:08.375653+0000","last_fullsized":"2024-04-02T04:47:08.375653+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:52.343572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.343572+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.554866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-02T04:47:08.327168+0000","last_change":"2024-04-02T04:46:21.245136+0000","last_active":"2024-04-02T04:47:08.327168+0000","last_peered":"2024-04-02T04:47:08.327168+0000","last_clean":"2024-04-02T04:47:08.327168+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:47:08.327168+0000","last_undegraded":"2024-04-02T04:47:08.327168+0000","last_fullsized":"2024-04-02T04:47:08.327168+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:21.244838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:21.244838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:46.980660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":181,"num_read_kb":176,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":612,"ondisk_log_size":612,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743475,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28324,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716828,"statfs":{"total":95995035648,"available":95966031872,"internally_reserved":0,"allocated":1339392,"data_stored":967890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103230,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1306624,"data_stored":965703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31888,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713264,"statfs":{"total":95995035648,"available":95962382336,"internally_reserved":0,"allocated":729088,"data_stored":376415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":966367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378727,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":716800,"data_stored":376651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051677,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":712704,"data_stored":375716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822739,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":753664,"data_stored":376644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462983,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":716800,"data_stored":376560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:47:23.521 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-02T04:47:23.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump 2024-04-02T04:47:23.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:23 smithi098 bash[23790]: cluster 2024-04-02T04:47:21.793183+0000 mgr.y (mgr.14538) 128 : cluster 0 pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:23.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:23 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.174015+0000 osd.1 (osd.1) 87 : cluster 0 2.d scrub starts 2024-04-02T04:47:23.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:23 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.176356+0000 osd.1 (osd.1) 88 : cluster 0 2.d scrub ok 2024-04-02T04:47:23.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:23 smithi069 bash[28904]: cluster 2024-04-02T04:47:21.793183+0000 mgr.y (mgr.14538) 128 : cluster 0 pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:23 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.174015+0000 osd.1 (osd.1) 87 : cluster 0 2.d scrub starts 2024-04-02T04:47:23.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:23 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.176356+0000 osd.1 (osd.1) 88 : cluster 0 2.d scrub ok 2024-04-02T04:47:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:23 smithi069 bash[21528]: cluster 2024-04-02T04:47:21.793183+0000 mgr.y (mgr.14538) 128 : cluster 0 pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:23 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.174015+0000 osd.1 (osd.1) 87 : cluster 0 2.d scrub starts 2024-04-02T04:47:23.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:23 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.176356+0000 osd.1 (osd.1) 88 : cluster 0 2.d scrub ok 2024-04-02T04:47:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:24 smithi098 bash[23790]: audit 2024-04-02T04:47:22.827567+0000 mgr.y (mgr.14538) 129 : audit 0 from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:47:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:24 smithi098 bash[23790]: audit 2024-04-02T04:47:22.872724+0000 mgr.y (mgr.14538) 130 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:24 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.321727+0000 osd.6 (osd.6) 57 : cluster 0 2.17 scrub starts 2024-04-02T04:47:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:24 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.323909+0000 osd.6 (osd.6) 58 : cluster 0 2.17 scrub ok 2024-04-02T04:47:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:24 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.459340+0000 osd.7 (osd.7) 85 : cluster 0 3.d scrub starts 2024-04-02T04:47:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:24 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.464006+0000 osd.7 (osd.7) 86 : cluster 0 3.d scrub ok 2024-04-02T04:47:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[28904]: audit 2024-04-02T04:47:22.827567+0000 mgr.y (mgr.14538) 129 : audit 0 from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:47:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[28904]: audit 2024-04-02T04:47:22.872724+0000 mgr.y (mgr.14538) 130 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.321727+0000 osd.6 (osd.6) 57 : cluster 0 2.17 scrub starts 2024-04-02T04:47:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.323909+0000 osd.6 (osd.6) 58 : cluster 0 2.17 scrub ok 2024-04-02T04:47:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.459340+0000 osd.7 (osd.7) 85 : cluster 0 3.d scrub starts 2024-04-02T04:47:24.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.464006+0000 osd.7 (osd.7) 86 : cluster 0 3.d scrub ok 2024-04-02T04:47:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[21528]: audit 2024-04-02T04:47:22.827567+0000 mgr.y (mgr.14538) 129 : audit 0 from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:47:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[21528]: audit 2024-04-02T04:47:22.872724+0000 mgr.y (mgr.14538) 130 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.321727+0000 osd.6 (osd.6) 57 : cluster 0 2.17 scrub starts 2024-04-02T04:47:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.323909+0000 osd.6 (osd.6) 58 : cluster 0 2.17 scrub ok 2024-04-02T04:47:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.459340+0000 osd.7 (osd.7) 85 : cluster 0 3.d scrub starts 2024-04-02T04:47:24.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:24 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.464006+0000 osd.7 (osd.7) 86 : cluster 0 3.d scrub ok 2024-04-02T04:47:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:25 smithi098 bash[23790]: cluster 2024-04-02T04:47:23.793966+0000 mgr.y (mgr.14538) 131 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:25.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:25 smithi069 bash[21528]: cluster 2024-04-02T04:47:23.793966+0000 mgr.y (mgr.14538) 131 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:25.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:25 smithi069 bash[28904]: cluster 2024-04-02T04:47:23.793966+0000 mgr.y (mgr.14538) 131 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:27 smithi098 bash[23790]: cluster 2024-04-02T04:47:25.794615+0000 mgr.y (mgr.14538) 132 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:27 smithi098 bash[23790]: cluster 2024-04-02T04:47:25.812337+0000 osd.4 (osd.4) 93 : cluster 0 4.1 scrub starts 2024-04-02T04:47:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:27 smithi098 bash[23790]: cluster 2024-04-02T04:47:25.814712+0000 osd.4 (osd.4) 94 : cluster 0 4.1 scrub ok 2024-04-02T04:47:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:27 smithi098 bash[23790]: cluster 2024-04-02T04:47:26.445628+0000 osd.3 (osd.3) 115 : cluster 0 3.8 scrub starts 2024-04-02T04:47:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:27 smithi098 bash[23790]: cluster 2024-04-02T04:47:26.453313+0000 osd.3 (osd.3) 116 : cluster 0 3.8 scrub ok 2024-04-02T04:47:27.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[21528]: cluster 2024-04-02T04:47:25.794615+0000 mgr.y (mgr.14538) 132 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:27.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[21528]: cluster 2024-04-02T04:47:25.812337+0000 osd.4 (osd.4) 93 : cluster 0 4.1 scrub starts 2024-04-02T04:47:27.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[21528]: cluster 2024-04-02T04:47:25.814712+0000 osd.4 (osd.4) 94 : cluster 0 4.1 scrub ok 2024-04-02T04:47:27.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[21528]: cluster 2024-04-02T04:47:26.445628+0000 osd.3 (osd.3) 115 : cluster 0 3.8 scrub starts 2024-04-02T04:47:27.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[21528]: cluster 2024-04-02T04:47:26.453313+0000 osd.3 (osd.3) 116 : cluster 0 3.8 scrub ok 2024-04-02T04:47:27.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[28904]: cluster 2024-04-02T04:47:25.794615+0000 mgr.y (mgr.14538) 132 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:27.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[28904]: cluster 2024-04-02T04:47:25.812337+0000 osd.4 (osd.4) 93 : cluster 0 4.1 scrub starts 2024-04-02T04:47:27.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[28904]: cluster 2024-04-02T04:47:25.814712+0000 osd.4 (osd.4) 94 : cluster 0 4.1 scrub ok 2024-04-02T04:47:27.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[28904]: cluster 2024-04-02T04:47:26.445628+0000 osd.3 (osd.3) 115 : cluster 0 3.8 scrub starts 2024-04-02T04:47:27.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:27 smithi069 bash[28904]: cluster 2024-04-02T04:47:26.453313+0000 osd.3 (osd.3) 116 : cluster 0 3.8 scrub ok 2024-04-02T04:47:28.297 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[21528]: cluster 2024-04-02T04:47:27.377469+0000 osd.5 (osd.5) 77 : cluster 0 2.18 scrub starts 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[21528]: cluster 2024-04-02T04:47:27.379786+0000 osd.5 (osd.5) 78 : cluster 0 2.18 scrub ok 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[21528]: cluster 2024-04-02T04:47:27.780123+0000 osd.6 (osd.6) 59 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[21528]: cluster 2024-04-02T04:47:27.780231+0000 osd.6 (osd.6) 60 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[28904]: cluster 2024-04-02T04:47:27.377469+0000 osd.5 (osd.5) 77 : cluster 0 2.18 scrub starts 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[28904]: cluster 2024-04-02T04:47:27.379786+0000 osd.5 (osd.5) 78 : cluster 0 2.18 scrub ok 2024-04-02T04:47:28.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[28904]: cluster 2024-04-02T04:47:27.780123+0000 osd.6 (osd.6) 59 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:28.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[28904]: cluster 2024-04-02T04:47:27.780231+0000 osd.6 (osd.6) 60 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:28 smithi098 bash[23790]: cluster 2024-04-02T04:47:27.377469+0000 osd.5 (osd.5) 77 : cluster 0 2.18 scrub starts 2024-04-02T04:47:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:28 smithi098 bash[23790]: cluster 2024-04-02T04:47:27.379786+0000 osd.5 (osd.5) 78 : cluster 0 2.18 scrub ok 2024-04-02T04:47:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:28 smithi098 bash[23790]: cluster 2024-04-02T04:47:27.780123+0000 osd.6 (osd.6) 59 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:28 smithi098 bash[23790]: cluster 2024-04-02T04:47:27.780231+0000 osd.6 (osd.6) 60 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:29.027 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:47:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:47:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:29 smithi098 bash[23790]: cluster 2024-04-02T04:47:27.795374+0000 mgr.y (mgr.14538) 133 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:29 smithi098 bash[23790]: cluster 2024-04-02T04:47:28.450813+0000 osd.7 (osd.7) 87 : cluster 0 2.8 scrub starts 2024-04-02T04:47:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:29 smithi098 bash[23790]: cluster 2024-04-02T04:47:28.453432+0000 osd.7 (osd.7) 88 : cluster 0 2.8 scrub ok 2024-04-02T04:47:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:29 smithi098 bash[23790]: cluster 2024-04-02T04:47:28.468337+0000 mon.a (mon.0) 920 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-04-02T04:47:29.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[21528]: cluster 2024-04-02T04:47:27.795374+0000 mgr.y (mgr.14538) 133 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:29.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[21528]: cluster 2024-04-02T04:47:28.450813+0000 osd.7 (osd.7) 87 : cluster 0 2.8 scrub starts 2024-04-02T04:47:29.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[21528]: cluster 2024-04-02T04:47:28.453432+0000 osd.7 (osd.7) 88 : cluster 0 2.8 scrub ok 2024-04-02T04:47:29.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[21528]: cluster 2024-04-02T04:47:28.468337+0000 mon.a (mon.0) 920 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-04-02T04:47:29.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[28904]: cluster 2024-04-02T04:47:27.795374+0000 mgr.y (mgr.14538) 133 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:29.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[28904]: cluster 2024-04-02T04:47:28.450813+0000 osd.7 (osd.7) 87 : cluster 0 2.8 scrub starts 2024-04-02T04:47:29.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[28904]: cluster 2024-04-02T04:47:28.453432+0000 osd.7 (osd.7) 88 : cluster 0 2.8 scrub ok 2024-04-02T04:47:29.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:29 smithi069 bash[28904]: cluster 2024-04-02T04:47:28.468337+0000 mon.a (mon.0) 920 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/255620853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 msgr2=0x7fd94c06ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- 172.21.15.69:0/255620853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c06ad80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd948009990 tx=0x7fd948009ca0 comp rx=0 tx=0).stop 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/255620853 shutdown_connections 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- 172.21.15.69:0/255620853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 0x7fd94c10f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- 172.21.15.69:0/255620853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 0x7fd94c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- 172.21.15.69:0/255620853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c06ad80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:30.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/255620853 >> 172.21.15.69:0/255620853 conn(0x7fd94c078140 msgr2=0x7fd94c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/255620853 shutdown_connections 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/255620853 wait complete. 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 librados: starting msgr at 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 librados: starting objecter 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 Processor -- start 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- start start 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 librados: setting wanted keys 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 librados: calling monclient init 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 0x7fd94c101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 0x7fd94c101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c111ae0 con 0x7fd94c1082f0 2024-04-02T04:47:30.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c111960 con 0x7fd94c06b2c0 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd94c111c60 con 0x7fd94c06a920 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53460/0 (socket says 172.21.15.69:53460) 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 -- 172.21.15.69:0/1506668036 learned_addr learned my addr 172.21.15.69:0/1506668036 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9519e4700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 0x7fd94c101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9509e2700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 0x7fd94c101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 -- 172.21.15.69:0/1506668036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 msgr2=0x7fd94c101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 0x7fd94c101750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 -- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 msgr2=0x7fd94c101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 0x7fd94c101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 -- 172.21.15.69:0/1506668036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94c1a65e0 con 0x7fd94c06a920 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9509e2700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 0x7fd94c101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9519e4700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 0x7fd94c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9511e3700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c101210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00eb30 tx=0x7fd93c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:30.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9427fc700 10 client.?.objecter ms_handle_connect 0x7fd94c06a920 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9427fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00cc90 con 0x7fd94c06a920 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd93c00ce30 con 0x7fd94c06a920 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94c1a68d0 con 0x7fd94c06a920 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c003df0 con 0x7fd94c06a920 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 10 client.24626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94c1a6e10 con 0x7fd94c06a920 2024-04-02T04:47:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.909+0000 7fd953447700 1 librados: init done 2024-04-02T04:47:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd94c04ff00 con 0x7fd94c06a920 2024-04-02T04:47:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd93c004000 con 0x7fd94c06a920 2024-04-02T04:47:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd93806f5e0 0x7fd938071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(119..119 src has 64..119) ==== 7537+0+0 (secure 0 0 0) 0x7fd93c08e2a0 con 0x7fd94c06a920 2024-04-02T04:47:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 10 client.24626.objecter ms_dispatch 0x7fd94c001550 osd_map(119..119 src has 64..119) 2024-04-02T04:47:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 3 client.24626.objecter handle_osd_map got epochs [119,119] > 0 2024-04-02T04:47:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 3 client.24626.objecter handle_osd_map decoding full epoch 119 2024-04-02T04:47:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9509e2700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd93806f5e0 0x7fd938071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 20 client.24626.objecter dump_active .. 0 homeless 2024-04-02T04:47:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9509e2700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd93806f5e0 0x7fd938071aa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd94800b2e0 tx=0x7fd948005e80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.913+0000 7fd9427fc700 10 client.24626.objecter ms_handle_connect 0x7fd93806f5e0 2024-04-02T04:47:30.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:30.917+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd93c05b880 con 0x7fd94c06a920 2024-04-02T04:47:31.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.133+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fd94c102740 con 0x7fd94c06a920 2024-04-02T04:47:31.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.133+0000 7fd9427fc700 1 -- 172.21.15.69:0/1506668036 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v119) ==== 56+0+5212 (secure 0 0 0) 0x7fd93c05f2d0 con 0x7fd94c06a920 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 119 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:created 2024-04-02T04:33:43.348385+0000 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:modified 2024-04-02T04:47:28.451118+0000 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:crush_version 18 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:full_ratio 0.95 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:backfillfull_ratio 0.9 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:nearfull_ratio 0.85 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:require_min_compat_client luminous 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:min_compat_client luminous 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:require_osd_release squid 2024-04-02T04:47:31.134 INFO:teuthology.orchestra.run.smithi069.stdout:stretch_mode_enabled false 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:pool 5 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 89 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:max_osd 8 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 up in weight 1 up_from 8 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] [v2:172.21.15.69:6804/1087500595,v1:172.21.15.69:6805/1087500595] exists,up 6b13e248-930e-40ca-9bb0-e8c5ce16bd0b 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 up in weight 1 up_from 13 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] [v2:172.21.15.69:6812/2651666015,v1:172.21.15.69:6813/2651666015] exists,up afb71182-d4f7-416d-ae44-90f28eab4b7a 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 up in weight 1 up_from 19 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] [v2:172.21.15.69:6820/619998764,v1:172.21.15.69:6821/619998764] exists,up 523bcbd7-d389-4b8f-aca8-88b71ce61de7 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 up in weight 1 up_from 29 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] [v2:172.21.15.69:6828/207593208,v1:172.21.15.69:6829/207593208] exists,up 20e4526d-2707-44d0-b13a-af36019c42ed 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 up in weight 1 up_from 36 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] [v2:172.21.15.98:6802/2065947829,v1:172.21.15.98:6803/2065947829] exists,up aa3bc4f3-4476-4ba1-b033-428e5edf5852 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 up in weight 1 up_from 47 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] [v2:172.21.15.98:6810/219578185,v1:172.21.15.98:6811/219578185] exists,up 902326af-4a40-47c8-a3b8-d5d5536f90fe 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 up in weight 1 up_from 58 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] [v2:172.21.15.98:6818/1226172499,v1:172.21.15.98:6819/1226172499] exists,up 8c73e4b0-cb6b-486a-8b55-a0fe4d97a727 2024-04-02T04:47:31.135 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 up in weight 1 up_from 69 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] [v2:172.21.15.98:6826/148009079,v1:172.21.15.98:6827/148009079] exists,up 4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 2.9 [7,2] 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.6 [1,5] 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.f [1,2] 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.17 [1,2] 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/2738867930 expires 2024-04-03T04:34:13.853990+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6801/2442918057 expires 2024-04-03T04:34:13.853990+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1798147282 expires 2024-04-03T04:34:13.853990+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/52352690 expires 2024-04-03T04:34:13.853990+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/2928651991 expires 2024-04-03T04:34:47.037764+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6800/2442918057 expires 2024-04-03T04:34:13.853990+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6801/578732137 expires 2024-04-03T04:34:47.037764+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/2763215770 expires 2024-04-03T04:34:47.037764+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/3657409442 expires 2024-04-03T04:34:47.037764+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/879855771 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.136 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1025808168 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.137 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1168952113 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.137 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6800/1560062353 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.137 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6800/578732137 expires 2024-04-03T04:34:47.037764+0000 2024-04-02T04:47:31.137 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1892956550 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.137 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6801/1560062353 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.137 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/54578327 expires 2024-04-03T04:44:51.699278+0000 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 10 librados: watch_flush enter 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 10 librados: watch_flush exit 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 20 client.24626.objecter shutdown clearing up homeless session... 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 10 client.24626.objecter successfully canceled tick 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd93806f5e0 msgr2=0x7fd938071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd93806f5e0 0x7fd938071aa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd94800b2e0 tx=0x7fd948005e80 comp rx=0 tx=0).stop 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 msgr2=0x7fd94c101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c101210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00eb30 tx=0x7fd93c00ee40 comp rx=0 tx=0).stop 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 shutdown_connections 2024-04-02T04:47:31.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd93806f5e0 0x7fd938071aa0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd94c1082f0 0x7fd94c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd94c06a920 0x7fd94c101210 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 --2- 172.21.15.69:0/1506668036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd94c06b2c0 0x7fd94c101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 >> 172.21.15.69:0/1506668036 conn(0x7fd94c078140 msgr2=0x7fd94c105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 shutdown_connections 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 -- 172.21.15.69:0/1506668036 wait complete. 2024-04-02T04:47:31.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:31.137+0000 7fd953447700 1 librados: shutdown 2024-04-02T04:47:31.754 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[21528]: cluster 2024-04-02T04:47:29.796032+0000 mgr.y (mgr.14538) 134 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:31.754 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[21528]: cluster 2024-04-02T04:47:30.466308+0000 osd.3 (osd.3) 117 : cluster 0 4.0 scrub starts 2024-04-02T04:47:31.754 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[21528]: cluster 2024-04-02T04:47:30.469958+0000 osd.3 (osd.3) 118 : cluster 0 4.0 scrub ok 2024-04-02T04:47:31.754 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[21528]: audit 2024-04-02T04:47:31.136583+0000 mon.c (mon.2) 27 : audit 0 from='client.? 172.21.15.69:0/1506668036' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T04:47:31.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[28904]: cluster 2024-04-02T04:47:29.796032+0000 mgr.y (mgr.14538) 134 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:31.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[28904]: cluster 2024-04-02T04:47:30.466308+0000 osd.3 (osd.3) 117 : cluster 0 4.0 scrub starts 2024-04-02T04:47:31.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[28904]: cluster 2024-04-02T04:47:30.469958+0000 osd.3 (osd.3) 118 : cluster 0 4.0 scrub ok 2024-04-02T04:47:31.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:31 smithi069 bash[28904]: audit 2024-04-02T04:47:31.136583+0000 mon.c (mon.2) 27 : audit 0 from='client.? 172.21.15.69:0/1506668036' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T04:47:31.755 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] [v2:172.21.15.69:6804/1087500595,v1:172.21.15.69:6805/1087500595] exists,up 6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'osd.1 up in weight 1 up_from 13 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] [v2:172.21.15.69:6812/2651666015,v1:172.21.15.69:6813/2651666015] exists,up afb71182-d4f7-416d-ae44-90f28eab4b7a', 'osd.2 up in weight 1 up_from 19 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] [v2:172.21.15.69:6820/619998764,v1:172.21.15.69:6821/619998764] exists,up 523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'osd.3 up in weight 1 up_from 29 up_thru 107 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] [v2:172.21.15.69:6828/207593208,v1:172.21.15.69:6829/207593208] exists,up 20e4526d-2707-44d0-b13a-af36019c42ed', 'osd.4 up in weight 1 up_from 36 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] [v2:172.21.15.98:6802/2065947829,v1:172.21.15.98:6803/2065947829] exists,up aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'osd.5 up in weight 1 up_from 47 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] [v2:172.21.15.98:6810/219578185,v1:172.21.15.98:6811/219578185] exists,up 902326af-4a40-47c8-a3b8-d5d5536f90fe', 'osd.6 up in weight 1 up_from 58 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] [v2:172.21.15.98:6818/1226172499,v1:172.21.15.98:6819/1226172499] exists,up 8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'osd.7 up in weight 1 up_from 69 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] [v2:172.21.15.98:6826/148009079,v1:172.21.15.98:6827/148009079] exists,up 4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3'] 2024-04-02T04:47:31.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon mon.a config show 2024-04-02T04:47:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:31 smithi098 bash[23790]: cluster 2024-04-02T04:47:29.796032+0000 mgr.y (mgr.14538) 134 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:31 smithi098 bash[23790]: cluster 2024-04-02T04:47:30.466308+0000 osd.3 (osd.3) 117 : cluster 0 4.0 scrub starts 2024-04-02T04:47:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:31 smithi098 bash[23790]: cluster 2024-04-02T04:47:30.469958+0000 osd.3 (osd.3) 118 : cluster 0 4.0 scrub ok 2024-04-02T04:47:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:31 smithi098 bash[23790]: audit 2024-04-02T04:47:31.136583+0000 mon.c (mon.2) 27 : audit 0 from='client.? 172.21.15.69:0/1506668036' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T04:47:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.436413+0000 osd.7 (osd.7) 89 : cluster 0 6.8 scrub starts 2024-04-02T04:47:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.438807+0000 osd.7 (osd.7) 90 : cluster 0 6.8 scrub ok 2024-04-02T04:47:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.449385+0000 osd.2 (osd.2) 47 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.449478+0000 osd.2 (osd.2) 48 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:32.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.473587+0000 mon.a (mon.0) 921 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-04-02T04:47:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.436413+0000 osd.7 (osd.7) 89 : cluster 0 6.8 scrub starts 2024-04-02T04:47:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.438807+0000 osd.7 (osd.7) 90 : cluster 0 6.8 scrub ok 2024-04-02T04:47:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.449385+0000 osd.2 (osd.2) 47 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.449478+0000 osd.2 (osd.2) 48 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:32 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.473587+0000 mon.a (mon.0) 921 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-04-02T04:47:32.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:32 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.436413+0000 osd.7 (osd.7) 89 : cluster 0 6.8 scrub starts 2024-04-02T04:47:32.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:32 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.438807+0000 osd.7 (osd.7) 90 : cluster 0 6.8 scrub ok 2024-04-02T04:47:32.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:32 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.449385+0000 osd.2 (osd.2) 47 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:32.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:32 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.449478+0000 osd.2 (osd.2) 48 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:32.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:32 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.473587+0000 mon.a (mon.0) 921 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-04-02T04:47:33.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:47:32 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:47:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:33 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.777624+0000 osd.4 (osd.4) 95 : cluster 0 4.13 scrub starts 2024-04-02T04:47:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:33 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.779683+0000 osd.4 (osd.4) 96 : cluster 0 4.13 scrub ok 2024-04-02T04:47:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:33 smithi069 bash[21528]: cluster 2024-04-02T04:47:31.796705+0000 mgr.y (mgr.14538) 135 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:33 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.777624+0000 osd.4 (osd.4) 95 : cluster 0 4.13 scrub starts 2024-04-02T04:47:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:33 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.779683+0000 osd.4 (osd.4) 96 : cluster 0 4.13 scrub ok 2024-04-02T04:47:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:33 smithi069 bash[28904]: cluster 2024-04-02T04:47:31.796705+0000 mgr.y (mgr.14538) 135 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:33 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.777624+0000 osd.4 (osd.4) 95 : cluster 0 4.13 scrub starts 2024-04-02T04:47:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:33 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.779683+0000 osd.4 (osd.4) 96 : cluster 0 4.13 scrub ok 2024-04-02T04:47:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:33 smithi098 bash[23790]: cluster 2024-04-02T04:47:31.796705+0000 mgr.y (mgr.14538) 135 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:47:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:34 smithi069 bash[21528]: audit 2024-04-02T04:47:32.883993+0000 mgr.y (mgr.14538) 136 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:34 smithi069 bash[28904]: audit 2024-04-02T04:47:32.883993+0000 mgr.y (mgr.14538) 136 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:34 smithi098 bash[23790]: audit 2024-04-02T04:47:32.883993+0000 mgr.y (mgr.14538) 136 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:33.798202+0000 mgr.y (mgr.14538) 137 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:34.388986+0000 osd.7 (osd.7) 91 : cluster 0 3.e scrub starts 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:34.393311+0000 osd.7 (osd.7) 92 : cluster 0 3.e scrub ok 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.005389+0000 osd.1 (osd.1) 89 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.005511+0000 osd.1 (osd.1) 90 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.157997+0000 osd.6 (osd.6) 61 : cluster 0 2.11 scrub starts 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.160163+0000 osd.6 (osd.6) 62 : cluster 0 2.11 scrub ok 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.205572+0000 osd.5 (osd.5) 79 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.205662+0000 osd.5 (osd.5) 80 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.258179+0000 osd.0 (osd.0) 89 : cluster 0 4.b scrub starts 2024-04-02T04:47:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.260643+0000 osd.0 (osd.0) 90 : cluster 0 4.b scrub ok 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:33.798202+0000 mgr.y (mgr.14538) 137 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:34.388986+0000 osd.7 (osd.7) 91 : cluster 0 3.e scrub starts 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:34.393311+0000 osd.7 (osd.7) 92 : cluster 0 3.e scrub ok 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.005389+0000 osd.1 (osd.1) 89 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.005511+0000 osd.1 (osd.1) 90 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.157997+0000 osd.6 (osd.6) 61 : cluster 0 2.11 scrub starts 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.160163+0000 osd.6 (osd.6) 62 : cluster 0 2.11 scrub ok 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.205572+0000 osd.5 (osd.5) 79 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.205662+0000 osd.5 (osd.5) 80 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.258179+0000 osd.0 (osd.0) 89 : cluster 0 4.b scrub starts 2024-04-02T04:47:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:35 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.260643+0000 osd.0 (osd.0) 90 : cluster 0 4.b scrub ok 2024-04-02T04:47:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:33.798202+0000 mgr.y (mgr.14538) 137 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:34.388986+0000 osd.7 (osd.7) 91 : cluster 0 3.e scrub starts 2024-04-02T04:47:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:34.393311+0000 osd.7 (osd.7) 92 : cluster 0 3.e scrub ok 2024-04-02T04:47:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.005389+0000 osd.1 (osd.1) 89 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.005511+0000 osd.1 (osd.1) 90 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.157997+0000 osd.6 (osd.6) 61 : cluster 0 2.11 scrub starts 2024-04-02T04:47:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.160163+0000 osd.6 (osd.6) 62 : cluster 0 2.11 scrub ok 2024-04-02T04:47:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.205572+0000 osd.5 (osd.5) 79 : cluster 0 purged_snaps scrub starts 2024-04-02T04:47:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.205662+0000 osd.5 (osd.5) 80 : cluster 0 purged_snaps scrub ok 2024-04-02T04:47:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.258179+0000 osd.0 (osd.0) 89 : cluster 0 4.b scrub starts 2024-04-02T04:47:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:35 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.260643+0000 osd.0 (osd.0) 90 : cluster 0 4.b scrub ok 2024-04-02T04:47:36.543 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:36.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:36 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.509260+0000 mon.a (mon.0) 922 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-04-02T04:47:36.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:36 smithi069 bash[28904]: cluster 2024-04-02T04:47:36.240230+0000 osd.0 (osd.0) 91 : cluster 0 6.9 scrub starts 2024-04-02T04:47:36.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:36 smithi069 bash[28904]: cluster 2024-04-02T04:47:36.242773+0000 osd.0 (osd.0) 92 : cluster 0 6.9 scrub ok 2024-04-02T04:47:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:36 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.509260+0000 mon.a (mon.0) 922 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-04-02T04:47:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:36 smithi069 bash[21528]: cluster 2024-04-02T04:47:36.240230+0000 osd.0 (osd.0) 91 : cluster 0 6.9 scrub starts 2024-04-02T04:47:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:36 smithi069 bash[21528]: cluster 2024-04-02T04:47:36.242773+0000 osd.0 (osd.0) 92 : cluster 0 6.9 scrub ok 2024-04-02T04:47:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:36 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.509260+0000 mon.a (mon.0) 922 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-04-02T04:47:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:36 smithi098 bash[23790]: cluster 2024-04-02T04:47:36.240230+0000 osd.0 (osd.0) 91 : cluster 0 6.9 scrub starts 2024-04-02T04:47:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:36 smithi098 bash[23790]: cluster 2024-04-02T04:47:36.242773+0000 osd.0 (osd.0) 92 : cluster 0 6.9 scrub ok 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[21528]: cluster 2024-04-02T04:47:35.798879+0000 mgr.y (mgr.14538) 138 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[21528]: cluster 2024-04-02T04:47:36.366396+0000 osd.7 (osd.7) 93 : cluster 0 3.15 scrub starts 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[21528]: cluster 2024-04-02T04:47:36.370481+0000 osd.7 (osd.7) 94 : cluster 0 3.15 scrub ok 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[21528]: audit 2024-04-02T04:47:37.109658+0000 mon.b (mon.1) 96 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[28904]: cluster 2024-04-02T04:47:35.798879+0000 mgr.y (mgr.14538) 138 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[28904]: cluster 2024-04-02T04:47:36.366396+0000 osd.7 (osd.7) 93 : cluster 0 3.15 scrub starts 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[28904]: cluster 2024-04-02T04:47:36.370481+0000 osd.7 (osd.7) 94 : cluster 0 3.15 scrub ok 2024-04-02T04:47:37.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:37 smithi069 bash[28904]: audit 2024-04-02T04:47:37.109658+0000 mon.b (mon.1) 96 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:37 smithi098 bash[23790]: cluster 2024-04-02T04:47:35.798879+0000 mgr.y (mgr.14538) 138 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-04-02T04:47:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:37 smithi098 bash[23790]: cluster 2024-04-02T04:47:36.366396+0000 osd.7 (osd.7) 93 : cluster 0 3.15 scrub starts 2024-04-02T04:47:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:37 smithi098 bash[23790]: cluster 2024-04-02T04:47:36.370481+0000 osd.7 (osd.7) 94 : cluster 0 3.15 scrub ok 2024-04-02T04:47:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:37 smithi098 bash[23790]: audit 2024-04-02T04:47:37.109658+0000 mon.b (mon.1) 96 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:38.896 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:47:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:47:38.896 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:38 smithi069 bash[21528]: cluster 2024-04-02T04:47:37.874081+0000 osd.4 (osd.4) 97 : cluster 0 4.1b scrub starts 2024-04-02T04:47:38.896 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:38 smithi069 bash[21528]: cluster 2024-04-02T04:47:37.876635+0000 osd.4 (osd.4) 98 : cluster 0 4.1b scrub ok 2024-04-02T04:47:38.897 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:38 smithi069 bash[28904]: cluster 2024-04-02T04:47:37.874081+0000 osd.4 (osd.4) 97 : cluster 0 4.1b scrub starts 2024-04-02T04:47:38.897 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:38 smithi069 bash[28904]: cluster 2024-04-02T04:47:37.876635+0000 osd.4 (osd.4) 98 : cluster 0 4.1b scrub ok 2024-04-02T04:47:38.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:38 smithi098 bash[23790]: cluster 2024-04-02T04:47:37.874081+0000 osd.4 (osd.4) 97 : cluster 0 4.1b scrub starts 2024-04-02T04:47:38.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:38 smithi098 bash[23790]: cluster 2024-04-02T04:47:37.876635+0000 osd.4 (osd.4) 98 : cluster 0 4.1b scrub ok 2024-04-02T04:47:39.054 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:47:39.054 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mon.a", 2024-04-02T04:47:39.054 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster": "ceph", 2024-04-02T04:47:39.054 INFO:teuthology.orchestra.run.smithi069.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "admin_socket_mode": "", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_client_required": "cephx, none", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_cluster_required": "cephx", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_debug": "true", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_service_required": "cephx", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_supported": "", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_aio": "true", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_aio_poll_ms": "250", 2024-04-02T04:47:39.055 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_aio_reap_max": "16", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_async_discard": "false", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_block_size": "4096", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_debug_aio": "false", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_debug_inflight_ios": "false", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_enable_discard": "false", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_flock_retry": "3", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_inject_crash": "0", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_ioring": "false", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_ioring_hipri": "false", 2024-04-02T04:47:39.056 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bdev_type": "", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_alloc_size": "1048576", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_allocator": "hybrid", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_buffered_io": "true", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_check_for_zeros": "false", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_compact_log_sync": "false", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-02T04:47:39.057 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_max_log_runway": "4194304", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_max_prefetch": "1048576", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_min_flush_size": "524288", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_min_log_runway": "1048576", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_replay_recovery": "false", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_sync_write": "false", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_allocation_from_file": "true", 2024-04-02T04:47:39.058 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_allocator": "hybrid", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_create": "true", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_db_create": "false", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_db_path": "", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_db_size": "0", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_path": "", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_preallocate_file": "false", 2024-04-02T04:47:39.059 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_size": "107374182400", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_wal_create": "false", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_wal_path": "", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_block_wal_size": "100663296", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_bluefs": "true", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-02T04:47:39.060 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_autotune": "true", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_size": "0", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_type": "2q", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_clone_cow": "true", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-02T04:47:39.061 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_mode": "none", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_csum_type": "crc32c", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_freelist": "false", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-02T04:47:39.062 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_small_allocations": "0", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_default_buffered_read": "true", 2024-04-02T04:47:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_default_buffered_write": "false", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fail_eio": "false", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-02T04:47:39.064 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_on_mount": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_on_umount": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_ignore_data_csum": "false", 2024-04-02T04:47:39.065 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_log_op_age": "5.000000", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_max_alloc_size": "0", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_max_blob_size": "0", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_max_deferred_txc": "32", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_min_alloc_size": "0", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-02T04:47:39.066 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_nid_prealloc": "1024", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_retry_disk_reads": "3", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_rocksdb_cf": "true", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-02T04:47:39.067 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_spdk_mem": "512", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_tracing": "false", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-02T04:47:39.068 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_zero_block_detection": "false", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-02T04:47:39.069 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_cluster_require_signatures": "false", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_cluster_require_version": "2", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_require_signatures": "false", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_require_version": "2", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_service_require_signatures": "false", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_service_require_version": "2", 2024-04-02T04:47:39.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cephx_sign_messages": "true", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "chdir": "", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_acl_type": "", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_asio_thread_count": "2", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_cache_mid": "0.750000", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_cache_size": "16384", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_caps_release_delay": "5", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_check_pool_perm": "true", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_debug_force_sync_read": "false", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_debug_getattr_caps": "false", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_debug_inject_tick_delay": "0", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_die_on_failed_remount": "false", 2024-04-02T04:47:39.071 INFO:teuthology.orchestra.run.smithi069.stdout: "client_dirsize_rbytes": "true", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_force_lazyio": "false", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_fs": "", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_inject_release_failure": "false", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_max_inline_size": "4096", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_mds_namespace": "", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_metadata": "", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_mount_gid": "-1", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_mount_timeout": "300", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_mount_uid": "-1", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_mountpoint": "/", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_notify_timeout": "10", 2024-04-02T04:47:39.072 INFO:teuthology.orchestra.run.smithi069.stdout: "client_oc": "true", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_oc_max_dirty": "104857600", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_oc_max_objects": "1000", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_oc_size": "209715200", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_oc_target_dirty": "8388608", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_permissions": "true", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_quota": "true", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_quota_df": "true", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_readahead_max_bytes": "0", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_readahead_max_periods": "4", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_readahead_min": "131072", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_reconnect_stale": "false", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_shutdown_timeout": "30", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_snapdir": ".snap", 2024-04-02T04:47:39.073 INFO:teuthology.orchestra.run.smithi069.stdout: "client_tick_interval": "1", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "client_trace": "", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "client_try_dentry_invalidate": "false", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "client_use_faked_inos": "false", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "client_use_random_mds": "false", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_graylog": "false", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_graylog_port": "12201", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_monitors": "default=true", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_syslog": "false", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "clog_to_syslog_level": "info", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "-", 2024-04-02T04:47:39.074 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_network": "", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_network_interface": "", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "compressor_zlib_isal": "false", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "compressor_zlib_level": "5", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "compressor_zlib_winsize": "-15", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "compressor_zstd_level": "1", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crimson_alien_op_num_threads": "6", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crimson_seastar_cpu_cores": "", 2024-04-02T04:47:39.075 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_location": "", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_location_hook": "", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_location_hook_timeout": "10", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "daemonize": "false", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "daos_pool": "tank", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_alienstore": "0/5", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_allow_any_pool_priority": "false", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_asok": "1/5", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_asok_assert_abort": "false", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_asserts_on_shutdown": "true", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_auth": "1/5", 2024-04-02T04:47:39.076 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_bdev": "1/3", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_bluefs": "1/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_bluestore": "1/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_buffer": "0/1", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_ceph_exporter": "1/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_cephfs_mirror": "0/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_cephsqlite": "0/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_client": "0/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_compressor": "1/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_context": "0/1", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_crimson_interrupt": "0/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_crush": "1/1", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_crypto": "1/5", 2024-04-02T04:47:39.077 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_cyanstore": "0/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_deliberately_leak_memory": "false", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_disable_randomized_ping": "false", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_dpdk": "1/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_eventtrace": "1/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_filer": "0/1", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_filestore": "1/3", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_finisher": "1/1", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_fuse": "1/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_heartbeat_testing_span": "0", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_heartbeatmap": "1/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_javaclient": "1/5", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_journal": "1/3", 2024-04-02T04:47:39.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_journaler": "0/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_kstore": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_lockdep": "0/1", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mclock": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds_balancer": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds_locker": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds_log": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds_log_expire": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds_migrator": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mds_quiesce": "3/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_memstore": "1/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mgr": "2/5", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mgrc": "20/20", 2024-04-02T04:47:39.079 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_mon": "20/20", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_monc": "0/10", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_ms": "1/1", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_none": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_objclass": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_objectcacher": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_objecter": "0/1", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_optracker": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_osd": "1/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_paxos": "20/20", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_perfcounter": "1/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_prioritycache": "1/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rados": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rbd": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rbd_mirror": "0/5", 2024-04-02T04:47:39.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rbd_pwl": "0/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rbd_replay": "0/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_refs": "0/0", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_reserver": "1/1", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw_access": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw_datacache": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw_dbstore": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw_flight": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rgw_sync": "1/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_rocksdb": "4/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore": "0/5", 2024-04-02T04:47:39.081 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_backref": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_cache": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_cleaner": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_device": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_epm": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_journal": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_lba": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_odata": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_omap": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_onode": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_t": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_seastore_tm": "0/5", 2024-04-02T04:47:39.082 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_striper": "0/1", 2024-04-02T04:47:39.085 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_test": "0/5", 2024-04-02T04:47:39.085 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_throttle": "1/1", 2024-04-02T04:47:39.085 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_timer": "0/1", 2024-04-02T04:47:39.085 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_tp": "0/5", 2024-04-02T04:47:39.085 INFO:teuthology.orchestra.run.smithi069.stdout: "debug_trace": "1/5", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "device_failure_prediction_mode": "none", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "err_to_graylog": "false", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "err_to_journald": "false", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "err_to_stderr": "true", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "err_to_syslog": "false", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "event_tracing": "false", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "exporter_addr": "0.0.0.0", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "exporter_http_port": "9926", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "exporter_prio_limit": "5", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-02T04:47:39.086 INFO:teuthology.orchestra.run.smithi069.stdout: "exporter_sort_metrics": "true", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "exporter_stats_period": "5", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "fake_statfs_for_testing": "0", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "fatal_signal_handlers": "true", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filer_max_purge_ops": "10", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filer_max_truncate_ops": "128", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_apply_finisher_threads": "1", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_blackhole": "false", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_btrfs_clone_range": "true", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_btrfs_snap": "true", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_caller_concurrency": "10", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_collect_device_partition_information": "true", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_commit_timeout": "600.000000", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_debug_inject_read_err": "false", 2024-04-02T04:47:39.087 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_debug_omap_check": "false", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_debug_verify_split": "false", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_dump_file": "", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fadvise": "true", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fail_eio": "true", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fd_cache_shards": "16", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fd_cache_size": "128", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fiemap": "false", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fiemap_threshold": "4096", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-02T04:47:39.088 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_inject_stall": "0", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_journal_parallel": "false", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_journal_trailing": "false", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_journal_writeahead": "false", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_kill_at": "0", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattrs": "0", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-02T04:47:39.089 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_xattr_value_size": "0", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_merge_threshold": "-10", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_odsync_write": "false", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_omap_backend": "rocksdb", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_omap_backend_path": "", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-02T04:47:39.090 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_op_thread_timeout": "60", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_op_threads": "2", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_punch_hole": "false", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_queue_max_ops": "50", 2024-04-02T04:47:39.091 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_seek_data_hole": "false", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_sloppy_crc": "false", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_splice": "false", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_split_multiple": "2", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_split_rand_factor": "20", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_update_to": "1000", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-02T04:47:39.092 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_enable": "true", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_xfs_extsize": "false", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "filestore_zfs_snap": "false", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "fio_dir": "/tmp/fio", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_allow_other": "true", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_atomic_o_trunc": "true", 2024-04-02T04:47:39.093 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_big_writes": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_debug": "false", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_default_permissions": "false", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_disable_pagecache": "false", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_max_write": "0", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_multithreaded": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_require_active_mds": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_set_user_groups": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_splice_move": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_splice_read": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_splice_write": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "fuse_use_invalidate_cb": "true", 2024-04-02T04:47:39.094 INFO:teuthology.orchestra.run.smithi069.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "gss_target_name": "ceph", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_file": "", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_inject_failure": "0", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_interval": "5", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "host": "smithi069", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-02T04:47:39.095 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "inject_early_sigterm": "false", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "jaeger_agent_port": "6799", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "jaeger_tracing_enable": "false", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_aio": "true", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_align_min_size": "65536", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_block_align": "true", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_block_size": "4096", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_dio": "true", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_discard": "false", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_force_aio": "false", 2024-04-02T04:47:39.096 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_ignore_corruption": "false", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_max_write_bytes": "10485760", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_max_write_entries": "100", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_replay_from": "0", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_write_header_frequency": "0", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journal_zero_on_create": "false", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journaler_prefetch_periods": "10", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journaler_prezero_periods": "5", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "journaler_write_head_interval": "15", 2024-04-02T04:47:39.097 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "keyfile": "", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_backend": "rocksdb", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_default_stripe_size": "65536", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_fsck_on_mount": "false", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_max_bytes": "67108864", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_max_ops": "512", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_nid_prealloc": "1024", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_onode_map_size": "1024", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_sync_submit_transaction": "false", 2024-04-02T04:47:39.098 INFO:teuthology.orchestra.run.smithi069.stdout: "kstore_sync_transaction": "false", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "librados_thread_count": "2", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "lockdep": "false", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "lockdep_force_backtrace": "false", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_coarse_timestamps": "true", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_flush_on_exit": "false", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_graylog_host": "127.0.0.1", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_graylog_port": "12201", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_max_new": "1000", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_max_recent": "10000", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_stderr_prefix": "debug ", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_stop_at_utilization": "0.970000", 2024-04-02T04:47:39.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": "true", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_graylog": "false", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_journald": "false", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_stderr": "false", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_syslog": "false", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "max_rotating_auth_attempts": "10", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_action_on_write_error": "1", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_alternate_name_max": "8192", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_asio_thread_count": "2", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_export_pin": "true", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_fragment_dirs": "true", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_fragment_interval": "5", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-02T04:47:39.100 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_interval": "10", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_max": "-1", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_max_until": "-1", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_merge_size": "50", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_midchunk": "0.300000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_min_start": "0.200000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_minchunk": "0.001000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_mode": "0", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_need_max": "1.200000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_need_min": "0.800000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_overload_epochs": "2", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-02T04:47:39.101 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_split_bits": "3", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_split_size": "10000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_target_decay": "10.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_beacon_grace": "15.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_beacon_interval": "4.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_mid": "0.700000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_quiesce_delay": "0", 2024-04-02T04:47:39.102 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_release_free_interval": "10", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_reservation": "0.050000", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_trim_interval": "1", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache_trim_threshold": "262144", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_client_prealloc_inos": "1000", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-02T04:47:39.103 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_connect_bootstrapping": "false", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_damage_table_max_entries": "10000", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_debug_auth_pins": "false", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_debug_frag": "false", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_debug_scatterstat": "false", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_debug_subtrees": "false", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_decay_halflife": "5.000000", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_default_dir_hash": "2", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_defer_session_stale": "true", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_deny_all_reconnect": "false", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dir_keys_per_op": "16384", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dir_max_commit_size": "10", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dir_max_entries": "0", 2024-04-02T04:47:39.104 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dir_prefetch": "true", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dump_cache_on_map": "false", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_early_reply": "true", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_enable_op_tracker": "true", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_enforce_unique_name": "true", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_export_ephemeral_random": "true", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-02T04:47:39.105 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_health_summarize_threshold": "10", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_inject_health_dummy": "false", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_inject_migrator_session_race": "false", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-02T04:47:39.106 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_join_fs": "", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_journal_format": "1", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_create_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_export_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_import_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_journal_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_journal_expire_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_journal_replay_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_link_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_mdstable_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_openc_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_rename_at": "0", 2024-04-02T04:47:39.107 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_shutdown_at": "0", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_event_large_threshold": "524288", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_events_per_segment": "1024", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_max_events": "-1", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_max_segments": "128", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_pause": "false", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_segment_size": "0", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_trim_threshold": "128", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-02T04:47:39.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log_warn_factor": "2.000000", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_caps_per_client": "1048576", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_completed_flushes": "100000", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_completed_requests": "100000", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_export_size": "20971520", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_file_recover": "32", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_purge_files": "64", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_purge_ops": "8192", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_max_snaps_per_dir": "100", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_metrics_update_interval": "2", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_min_caps_per_client": "100", 2024-04-02T04:47:39.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_min_caps_working_set": "10000", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_numa_node": "-1", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_op_complaint_time": "30.000000", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_op_history_duration": "600", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_op_history_size": "20", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_op_history_slow_op_size": "20", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_op_log_threshold": "5", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_ping_grace": "15", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_ping_interval": "5", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-02T04:47:39.110 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_recall_max_caps": "30000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_recall_warning_threshold": "262144", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_replay_interval": "1.000000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_root_ino_gid": "0", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_root_ino_uid": "0", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-02T04:47:39.111 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_shutdown_check": "0", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_skip_ino": "0", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_snap_max_uid": "4294967294", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_snap_min_uid": "0", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_snap_rstat": "false", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_standby_replay_damaged": "false", 2024-04-02T04:47:39.112 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_symlink_recovery": "true", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_thrash_exports": "0", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_thrash_fragments": "0", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_tick_interval": "5.000000", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_valgrind_exit": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_verify_backtrace": "1", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_verify_scatter": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_wipe_sessions": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool_debug": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "memstore_device_bytes": "1073741824", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "memstore_page_set": "false", 2024-04-02T04:47:39.113 INFO:teuthology.orchestra.run.smithi069.stdout: "memstore_page_size": "65536", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_client_bytes": "134217728", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_client_messages": "512", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_disabled_modules": "", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_max_pg_num_change": "128", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_mds_bytes": "134217728", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_mds_messages": "128", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-02T04:47:39.114 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_mon_bytes": "134217728", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_mon_messages": "128", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_osd_bytes": "536870912", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_osd_messages": "8192", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_pool": "true", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_standby_modules": "true", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_stats_period": "5", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_stats_threshold": "5", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_tick_period": "2", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_allow_pool_delete": "true", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_allow_pool_size_one": "false", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_auth_validate_all_caps": "true", 2024-04-02T04:47:39.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_bytes": "104857600", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_directed_command_retry": "2", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_hunt_parallel": "3", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_log_interval": "1.000000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_ping_interval": "10.000000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_client_target_rank": "-1", 2024-04-02T04:47:39.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_file_level": "debug", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_file": "true", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_journald": "false", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_stderr": "true", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-02T04:47:39.117 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_compact_on_bootstrap": "false", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_compact_on_start": "false", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_compact_on_trim": "true", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_cpu_threads": "4", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_crush_min_required_version": "hammer", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_daemon_bytes": "419430400", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_data_avail_crit": "5", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_data_avail_warn": "5", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_data_size_warn": "16106127360", 2024-04-02T04:47:39.118 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_dump_json": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_dump_transactions": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_extra_checks": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_no_require_reef": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_no_require_squid": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_down_mkfs_grace": "60", 2024-04-02T04:47:39.119 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_down_uptime_grace": "60", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_election_default_strategy": "1", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_election_timeout": "5.000000", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_elector_ping_divisor": "2", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_enable_op_tracker": "true", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_fake_pool_delete": "false", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_force_quorum_join": "false", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_globalid_prealloc": "10000", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_health_detail_to_clog": "true", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_health_log_update_period": "5", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_health_max_detail": "50", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_health_to_clog": "true", 2024-04-02T04:47:39.120 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_health_to_clog_interval": "600", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_host": "[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_host_override": "", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_initial_members": "", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_lease": "5.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_log_full_interval": "50", 2024-04-02T04:47:39.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_log_max": "10000", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_log_max_summary": "50", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_log_epochs": "500", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_osd": "10000", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_pg_per_osd": "10000", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_pool_pg_num": "65536", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mds_force_trim_to": "0", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mds_skip_sanity": "false", 2024-04-02T04:47:39.122 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_memory_autotune": "true", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_memory_target": "2147483648", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mgr_beacon_grace": "30", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mgr_digest_period": "5", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mgr_inactive_grace": "60", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_min_osdmap_epochs": "50", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_op_complaint_time": "30", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_op_history_duration": "600", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_op_history_size": "20", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_op_history_slow_op_size": "20", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-02T04:47:39.123 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_op_log_threshold": "5", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_auto_mark_in": "false", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_cache_size": "500", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_down_out_interval": "600", 2024-04-02T04:47:39.124 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_force_trim_to": "0", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_min_down_reporters": "2", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-02T04:47:39.125 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_report_timeout": "900", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pg_stuck_threshold": "60", 2024-04-02T04:47:39.126 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_probe_timeout": "2.000000", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_reweight_max_change": "0.050000", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_reweight_max_osds": "4", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-02T04:47:39.127 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_scrub_interval": "86400", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_scrub_max_keys": "100", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_scrub_timeout": "300", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_session_timeout": "300", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_smart_report_timeout": "5", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_stat_smooth_intervals": "6", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_stretch_pool_min_size": "2", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_stretch_pool_size": "4", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_sync_debug": "false", 2024-04-02T04:47:39.128 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_sync_provider_kill_at": "0", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_sync_requester_kill_at": "0", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_sync_timeout": "60.000000", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_target_pg_per_osd": "100", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_tick_interval": "5", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_timecheck_interval": "300.000000", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_use_min_delay_socket": "false", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_older_version_delay": "604800", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-02T04:47:39.129 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_misplaced": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_older_version": "true", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-02T04:47:39.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "monmap": "", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "motr_tracing_enabled": "false", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_op_threads": "3", 2024-04-02T04:47:39.131 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_cm": "false", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_device_name": "", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_dscp": "96", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_local_gid": "", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_port_num": "1", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-02T04:47:39.132 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_sl": "3", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_support_srq": "true", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_rdma_type": "ib", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_async_reap_threshold": "5", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_before_connect": "false", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_ipv4": "true", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_ipv6": "false", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_msgr1": "true", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_msgr2": "true", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_port_max": "7568", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_port_min": "6800", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_retry_count": "3", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_bind_retry_delay": "5", 2024-04-02T04:47:39.133 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_blackhole_client": "false", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_blackhole_mds": "false", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_blackhole_mgr": "false", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_blackhole_mon": "false", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_blackhole_osd": "false", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_client_mode": "crc secure", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_cluster_mode": "crc secure", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_cluster_type": "", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_compress_secure": "false", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_connection_idle_timeout": "900", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_connection_ready_timeout": "10", 2024-04-02T04:47:39.134 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_crc_data": "true", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_crc_header": "true", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_die_on_bad_msg": "false", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_die_on_bug": "true", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_die_on_old_message": "true", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_die_on_skipped_message": "false", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_coremask": "0xF", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_enable_tso": "true", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-02T04:47:39.135 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_hugepages": "", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_lro": "true", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_memory_channel": "4", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_pmd": "", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_port_id": "0", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_dump_on_send": "false", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_initial_backoff": "0.200000", 2024-04-02T04:47:39.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_inject_delay_max": "1.000000", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_inject_delay_type": "", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_inject_network_congestion": "0", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_inject_socket_failures": "0", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_learn_addr_from_peer": "true", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_max_accept_failures": "4", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_max_backoff": "15.000000", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_mon_client_mode": "secure crc", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_mon_service_mode": "secure crc", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_osd_compress_min_size": "1024", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_osd_compress_mode": "none", 2024-04-02T04:47:39.137 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_pq_min_cost": "65536", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_public_type": "", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_service_mode": "crc secure", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_tcp_listen_backlog": "512", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_tcp_nodelay": "true", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_tcp_rcvbuf": "0", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "ms_type": "async+posix", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "no_config_file": "false", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_completion_locks_per_session": "32", 2024-04-02T04:47:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_inflight_ops": "1024", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_tick_interval": "5.000000", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter_timeout": "10.000000", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objectstore_blackhole": "false", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "openssl_engine_opts": "", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_delay_time": "5.000000", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_hist_halflife": "1000", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_max_low_ops": "2", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_max_ops": "4", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-02T04:47:39.139 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_agent_slop": "0.020000", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_asio_thread_count": "2", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_async_recovery_min_cost": "100", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_backfill_scan_max": "512", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_backfill_scan_min": "64", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_backoff_on_degraded": "false", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_backoff_on_peering": "false", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_backoff_on_unfound": "true", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_beacon_report_interval": "300", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bench_duration": "30", 2024-04-02T04:47:39.140 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bench_max_block_size": "67108864", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_blkin_trace_all": "false", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_check_for_log_corruption": "false", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_class_update_on_start": "true", 2024-04-02T04:47:39.141 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_client_message_cap": "256", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_client_message_size_cap": "524288000", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_client_op_priority": "63", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_client_watch_timeout": "30", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_command_max_records": "256", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_compact_on_start": "false", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_crush_update_on_start": "true", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_crush_update_weight_set": "true", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-02T04:47:39.142 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_feed_pullee": "-1", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_misdirected_ops": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_no_acting_change": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_no_purge_strays": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_op_order": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-02T04:47:39.143 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_shutdown": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_trim_objects": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_verify_snaps": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_keys": "1024", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-02T04:47:39.144 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_stride": "524288", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_default_notify_timeout": "30", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_delete_sleep": "0.000000", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_discard_disconnected_ops": "true", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_enable_op_tracker": "true", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-02T04:47:39.145 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_fast_info": "true", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_fast_shutdown": "true", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_function_tracing": "false", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_grace": "20", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_interval": "6", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_min_peers": "10", 2024-04-02T04:47:39.146 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_min_size": "2000", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_stale": "600", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_hit_set_max_size": "100000", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_hit_set_min_size": "1000", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_journal_size": "5120", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_kill_backfill_at": "0", 2024-04-02T04:47:39.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_map_cache_size": "50", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_map_dedup": "true", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_map_message_max": "40", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_map_share_max_epochs": "40", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_attr_name_len": "100", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_attr_size": "0", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_backfills": "1", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_markdown_count": "5", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_markdown_period": "600", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_object_name_len": "2048", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_object_namespace_len": "256", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_object_size": "134217728", 2024-04-02T04:47:39.148 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_pg_blocked_by": "16", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_pg_log_entries": "10000", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_pgls": "1024", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_push_cost": "8388608", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_push_objects": "10", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_scrubs": "3", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_trimming_pgs": "2", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_write_op_reply_len": "64", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_max_write_size": "90", 2024-04-02T04:47:39.149 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_profile": "balanced", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-02T04:47:39.150 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_base": "805306368", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_cache_min": "134217728", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_target": "4294967296", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_target_autotune": "false", 2024-04-02T04:47:39.151 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_min_pg_log_entries": "250", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_min_recovery_priority": "0", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mon_report_interval": "5", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_num_cache_shards": "32", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_num_op_tracker_shard": "32", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_numa_auto_affinity": "true", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_numa_node": "-1", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_numa_prefer_iface": "true", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-02T04:47:39.152 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_objecter_finishers": "1", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_objectstore": "bluestore", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_objectstore_fuse": "false", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_objectstore_tracing": "false", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_complaint_time": "30.000000", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_history_duration": "600", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_history_size": "20", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_history_slow_op_size": "20", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_log_threshold": "5", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_num_shards": "0", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_num_shards_hdd": "5", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_num_shards_ssd": "8", 2024-04-02T04:47:39.153 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_pq_min_cost": "65536", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_queue_cut_off": "high", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_op_thread_timeout": "15", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_open_classes_on_start": "true", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_os_flags": "0", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_peering_op_priority": "255", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_delete_cost": "1048576", 2024-04-02T04:47:39.154 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_delete_priority": "5", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_log_trim_max": "10000", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_log_trim_min": "100", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-02T04:47:39.155 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_crimson": "false", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_flags": "0", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-02T04:47:39.156 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_min_size": "0", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_pg_num": "32", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_pgp_num": "0", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_read_ratio": "70", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_size": "3", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_default_type": "replicated", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_push_per_object_cost": "1000", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recover_clone_overlap": "true", 2024-04-02T04:47:39.157 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_cost": "20971520", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_max_active": "0", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_max_single_start": "1", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_op_priority": "3", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_priority": "5", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_sleep": "0.000000", 2024-04-02T04:47:39.158 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_repair_during_recovery": "false", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_requested_scrub_priority": "120", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_auto_repair": "false", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_begin_hour": "0", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_begin_week_day": "0", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_chunk_max": "25", 2024-04-02T04:47:39.159 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_chunk_min": "5", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_cost": "52428800", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_during_recovery": "false", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_end_hour": "0", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_end_week_day": "0", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_event_cost": "4096", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_invalid_stats": "true", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_max_preemptions": "5", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_priority": "5", 2024-04-02T04:47:39.160 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_sleep": "0.000000", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_skip_data_digest": "false", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_smart_report_timeout": "5", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_snap_trim_cost": "1048576", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_snap_trim_priority": "5", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-02T04:47:39.161 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_target_transaction_size": "30", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-02T04:47:39.162 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_tracing": "false", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_use_stale_snap": "false", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "osdc_blkin_trace_all": "false", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_kill_at": "0", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_max_join_drift": "10", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_min": "500", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_min_wait": "0.050000", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_propose_interval": "1.000000", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_service_trim_max": "500", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_service_trim_min": "10", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_stash_full_interval": "25", 2024-04-02T04:47:39.163 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_trim_max": "500", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "paxos_trim_min": "250", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "perf": "true", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "pid_file": "", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "-", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrv": "", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "public_bind_addr": "-", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "public_network": "", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "public_network_interface": "", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "qat_compressor_enabled": "false", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "qat_compressor_session_max_number": "256", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "rados_mon_op_timeout": "0", 2024-04-02T04:47:39.164 INFO:teuthology.orchestra.run.smithi069.stdout: "rados_osd_op_timeout": "0", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rados_tracing": "false", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_atime_update_interval": "60", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_balance_parent_reads": "false", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_balance_snap_reads": "false", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_blkin_trace_all": "false", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache": "true", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-02T04:47:39.165 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_policy": "writearound", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_size": "33554432", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_clone_copy_on_read": "false", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_compression_hint": "none", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_concurrent_management_ops": "10", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_clone_format": "auto", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_data_pool": "", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_features": "61", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_format": "2", 2024-04-02T04:47:39.166 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_map_options": "", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_order": "22", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_pool": "rbd", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_stripe_count": "0", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_default_stripe_unit": "0", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_enable_alloc_hint": "true", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-02T04:47:39.167 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_io_scheduler": "simple", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_order": "24", 2024-04-02T04:47:39.168 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_pool": "", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_journal_splay_width": "4", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_localize_parent_reads": "false", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_localize_snap_reads": "false", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-02T04:47:39.169 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-02T04:47:39.170 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_mtime_update_interval": "60", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_non_blocking_aio": "true", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_op_thread_timeout": "60", 2024-04-02T04:47:39.171 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_op_threads": "1", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_parent_cache_enabled": "false", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_plugins": "", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_bps_burst": "0", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_bps_limit": "0", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_exclude_ops": "0", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_iops_burst": "0", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_iops_limit": "0", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-02T04:47:39.172 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_read_from_replica_policy": "default", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-02T04:47:39.173 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_skip_partial_discard": "true", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_tracing": "false", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_validate_names": "true", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_validate_pool": "true", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "restapi_base_url": "", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "restapi_log_level": "", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_acl_grants_max_num": "100", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_admin_entry": "admin", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-02T04:47:39.174 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_backend_store": "rados", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_barbican_url": "", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_beast_enable_async": "true", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_counters_cache": "false", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_cache_enabled": "true", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_cache_expiry_interval": "900", 2024-04-02T04:47:39.175 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_cache_lru_size": "10000", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_config_store": "rados", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_content_length_compat": "false", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_copy_obj_progress": "true", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_cors_rules_max_num": "100", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_cross_domain_policy": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_addr": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_password": "", 2024-04-02T04:47:39.176 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_kmip_username": "", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_require_ssl": "true", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-02T04:47:39.177 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_addr": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_auth": "token", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_namespace": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_prefix": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-02T04:47:39.178 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_token_file": "", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_curl_buffersize": "524288", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_curl_low_speed_time": "300", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-02T04:47:39.179 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_d4n_port": "6379", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_log_changes_size": "1000", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_log_num_shards": "128", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_log_window": "30", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-02T04:47:39.180 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-02T04:47:39.181 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dns_name": "", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dns_s3website_name": "", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_dynamic_resharding": "true", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_gc_threads": "true", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_lc_threads": "true", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_ops_log": "false", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_quota_threads": "true", 2024-04-02T04:47:39.182 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_static_website": "false", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enable_usage_log": "false", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_enforce_swift_acls": "true", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_exit_timeout_secs": "120", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_expose_bucket": "false", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_extended_http_attrs": "", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_filter": "none", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_frontends": "beast port=7480", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_max_deferred": "50", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_max_objs": "32", 2024-04-02T04:47:39.183 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_gc_processor_period": "3600", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_init_timeout": "300", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_kafka_connection_idle": "300", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-02T04:47:39.184 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_domain": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_password": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_password_path": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_project": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_tenant": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_token": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_token_path": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_admin_user": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_api_version": "2", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_barbican_domain": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_barbican_password": "", 2024-04-02T04:47:39.185 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_barbican_project": "", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_barbican_user": "", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_url": "", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_debug_interval": "-1", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_lock_max_time": "90", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_max_objs": "32", 2024-04-02T04:47:39.186 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_max_rules": "1000", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_max_worker": "3", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lc_thread_delay": "0", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ldap_dnattr": "uid", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ldap_searchfilter": "", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ldap_uri": "ldaps://", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-02T04:47:39.187 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_log_http_headers": "", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_log_object_name_utc": "false", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_attr_name_len": "0", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_attr_size": "0", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_chunk_size": "4194304", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_listing_results": "1000", 2024-04-02T04:47:39.188 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_notify_retries": "10", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_put_param_size": "1048576", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_put_size": "5368709120", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_max_slo_entries": "1000", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_md_log_max_shards": "64", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_mp_lock_max_time": "600", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-02T04:47:39.189 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_max_gc": "300", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_num_async_rados_threads": "32", 2024-04-02T04:47:39.190 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_num_control_oids": "8", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_numa_node": "-1", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_objexp_chunk_size": "100", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_objexp_gc_interval": "600", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_op_thread_timeout": "600", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_op_tracing": "false", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_opa_token": "", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_opa_url": "", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_opa_verify_ssl": "true", 2024-04-02T04:47:39.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ops_log_rados": "false", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_ops_log_socket_path": "", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_period_push_interval": "2.000000", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_posix_cache_lanes": "3", 2024-04-02T04:47:39.192 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_posix_cache_partitions": "3", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_print_continue": "true", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_rados_tracing": "false", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_realm": "", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_realm_id": "", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-02T04:47:39.193 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_region": "", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_request_uri": "", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_reshard_batch_size": "64", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_reshard_max_aio": "128", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_reshard_num_logs": "16", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_reshard_thread_interval": "600", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_resolve_cname": "false", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_run_sync_thread": "true", 2024-04-02T04:47:39.194 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_scheduler_type": "throttler", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_script_uri": "", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_service_provider_name": "", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_client_id": "", 2024-04-02T04:47:39.195 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_client_secret": "", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_entry": "sts", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_key": "sts", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_min_session_duration": "900", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sts_token_introspection_url": "", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_account_in_url": "false", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_auth_entry": "auth", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_auth_url": "", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_custom_header": "", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_need_stats": "true", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_tenant_name": "", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_token_expiration": "86400", 2024-04-02T04:47:39.196 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_url": "", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_url_prefix": "swift", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_lease_period": "120", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-02T04:47:39.197 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_thread_pool_size": "512", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_comment": "", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_createby": "", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_encoding": "", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_flag": "false", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_origin": "", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_torrent_tracker": "", 2024-04-02T04:47:39.198 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_trust_forwarded_https": "false", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_usage_max_shards": "32", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_usage_max_user_shards": "1", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_use_opa_authz": "false", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_counters_cache": "false", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_max_buckets": "1000", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_policies_max_num": "100", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-02T04:47:39.199 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_user_unique_email": "true", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_verify_ssl": "true", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_zone": "", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_zone_id": "", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_zonegroup": "", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_zonegroup_id": "", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgwlc_auto_session_clear": "true", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_block_size": "4096", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-02T04:47:39.200 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cache_size": "536870912", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_index_type": "binary_search", 2024-04-02T04:47:39.201 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_partition_filters": "false", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_perf": "false", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "run_dir": "/var/run/ceph", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_block_create": "true", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_cache_lru_size": "67108864", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_default_max_object_size": "16777216", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-02T04:47:39.202 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_device_size": "53687091200", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_journal_batch_capacity": "16", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_main_device_type": "SSD", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "seastore_segment_size": "67108864", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "set_keepcaps": "false", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "setgroup": "ceph", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "setuser": "ceph", 2024-04-02T04:47:39.203 INFO:teuthology.orchestra.run.smithi069.stdout: "setuser_match_path": "", 2024-04-02T04:47:39.204 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-02T04:47:39.204 INFO:teuthology.orchestra.run.smithi069.stdout: "thp": "false", 2024-04-02T04:47:39.204 INFO:teuthology.orchestra.run.smithi069.stdout: "threadpool_default_timeout": "60", 2024-04-02T04:47:39.204 INFO:teuthology.orchestra.run.smithi069.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-02T04:47:39.204 INFO:teuthology.orchestra.run.smithi069.stdout: "throttler_perf_counter": "true" 2024-04-02T04:47:39.204 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:47:39.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:37.800336+0000 mgr.y (mgr.14538) 139 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:37.861425+0000 osd.2 (osd.2) 49 : cluster 0 6.f scrub starts 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:37.864028+0000 osd.2 (osd.2) 50 : cluster 0 6.f scrub ok 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:39.170281+0000 osd.0 (osd.0) 93 : cluster 0 4.1e scrub starts 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:39.171950+0000 osd.0 (osd.0) 94 : cluster 0 4.1e scrub ok 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:39.217314+0000 osd.5 (osd.5) 81 : cluster 0 3.5 scrub starts 2024-04-02T04:47:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:39 smithi098 bash[23790]: cluster 2024-04-02T04:47:39.221726+0000 osd.5 (osd.5) 82 : cluster 0 3.5 scrub ok 2024-04-02T04:47:40.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:37.800336+0000 mgr.y (mgr.14538) 139 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:40.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:37.861425+0000 osd.2 (osd.2) 49 : cluster 0 6.f scrub starts 2024-04-02T04:47:40.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:37.864028+0000 osd.2 (osd.2) 50 : cluster 0 6.f scrub ok 2024-04-02T04:47:40.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:39.170281+0000 osd.0 (osd.0) 93 : cluster 0 4.1e scrub starts 2024-04-02T04:47:40.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:39.171950+0000 osd.0 (osd.0) 94 : cluster 0 4.1e scrub ok 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:39.217314+0000 osd.5 (osd.5) 81 : cluster 0 3.5 scrub starts 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[21528]: cluster 2024-04-02T04:47:39.221726+0000 osd.5 (osd.5) 82 : cluster 0 3.5 scrub ok 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:37.800336+0000 mgr.y (mgr.14538) 139 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:37.861425+0000 osd.2 (osd.2) 49 : cluster 0 6.f scrub starts 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:37.864028+0000 osd.2 (osd.2) 50 : cluster 0 6.f scrub ok 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:39.170281+0000 osd.0 (osd.0) 93 : cluster 0 4.1e scrub starts 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:39.171950+0000 osd.0 (osd.0) 94 : cluster 0 4.1e scrub ok 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:39.217314+0000 osd.5 (osd.5) 81 : cluster 0 3.5 scrub starts 2024-04-02T04:47:40.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:39 smithi069 bash[28904]: cluster 2024-04-02T04:47:39.221726+0000 osd.5 (osd.5) 82 : cluster 0 3.5 scrub ok 2024-04-02T04:47:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:40 smithi098 bash[23790]: cluster 2024-04-02T04:47:40.131785+0000 osd.0 (osd.0) 95 : cluster 0 4.3 deep-scrub starts 2024-04-02T04:47:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:40 smithi098 bash[23790]: cluster 2024-04-02T04:47:40.137115+0000 osd.0 (osd.0) 96 : cluster 0 4.3 deep-scrub ok 2024-04-02T04:47:41.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:40 smithi069 bash[21528]: cluster 2024-04-02T04:47:40.131785+0000 osd.0 (osd.0) 95 : cluster 0 4.3 deep-scrub starts 2024-04-02T04:47:41.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:40 smithi069 bash[21528]: cluster 2024-04-02T04:47:40.137115+0000 osd.0 (osd.0) 96 : cluster 0 4.3 deep-scrub ok 2024-04-02T04:47:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:40 smithi069 bash[28904]: cluster 2024-04-02T04:47:40.131785+0000 osd.0 (osd.0) 95 : cluster 0 4.3 deep-scrub starts 2024-04-02T04:47:41.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:40 smithi069 bash[28904]: cluster 2024-04-02T04:47:40.137115+0000 osd.0 (osd.0) 96 : cluster 0 4.3 deep-scrub ok 2024-04-02T04:47:41.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:41 smithi098 bash[23790]: cluster 2024-04-02T04:47:39.800796+0000 mgr.y (mgr.14538) 140 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:41.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:41 smithi098 bash[23790]: cluster 2024-04-02T04:47:40.823778+0000 osd.4 (osd.4) 99 : cluster 0 3.1a deep-scrub starts 2024-04-02T04:47:41.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:41 smithi098 bash[23790]: cluster 2024-04-02T04:47:40.838605+0000 osd.4 (osd.4) 100 : cluster 0 3.1a deep-scrub ok 2024-04-02T04:47:41.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:41 smithi098 bash[23790]: cluster 2024-04-02T04:47:41.041088+0000 osd.6 (osd.6) 63 : cluster 0 2.7 scrub starts 2024-04-02T04:47:41.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:41 smithi098 bash[23790]: cluster 2024-04-02T04:47:41.043399+0000 osd.6 (osd.6) 64 : cluster 0 2.7 scrub ok 2024-04-02T04:47:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[21528]: cluster 2024-04-02T04:47:39.800796+0000 mgr.y (mgr.14538) 140 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[21528]: cluster 2024-04-02T04:47:40.823778+0000 osd.4 (osd.4) 99 : cluster 0 3.1a deep-scrub starts 2024-04-02T04:47:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[21528]: cluster 2024-04-02T04:47:40.838605+0000 osd.4 (osd.4) 100 : cluster 0 3.1a deep-scrub ok 2024-04-02T04:47:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[21528]: cluster 2024-04-02T04:47:41.041088+0000 osd.6 (osd.6) 63 : cluster 0 2.7 scrub starts 2024-04-02T04:47:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[21528]: cluster 2024-04-02T04:47:41.043399+0000 osd.6 (osd.6) 64 : cluster 0 2.7 scrub ok 2024-04-02T04:47:42.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[28904]: cluster 2024-04-02T04:47:39.800796+0000 mgr.y (mgr.14538) 140 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:42.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[28904]: cluster 2024-04-02T04:47:40.823778+0000 osd.4 (osd.4) 99 : cluster 0 3.1a deep-scrub starts 2024-04-02T04:47:42.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[28904]: cluster 2024-04-02T04:47:40.838605+0000 osd.4 (osd.4) 100 : cluster 0 3.1a deep-scrub ok 2024-04-02T04:47:42.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[28904]: cluster 2024-04-02T04:47:41.041088+0000 osd.6 (osd.6) 63 : cluster 0 2.7 scrub starts 2024-04-02T04:47:42.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:41 smithi069 bash[28904]: cluster 2024-04-02T04:47:41.043399+0000 osd.6 (osd.6) 64 : cluster 0 2.7 scrub ok 2024-04-02T04:47:42.892 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:42 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.154841+0000 osd.1 (osd.1) 91 : cluster 0 3.4 deep-scrub starts 2024-04-02T04:47:42.892 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:42 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.172881+0000 osd.1 (osd.1) 92 : cluster 0 3.4 deep-scrub ok 2024-04-02T04:47:42.893 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:42 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.183660+0000 osd.5 (osd.5) 83 : cluster 0 6.19 scrub starts 2024-04-02T04:47:42.893 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:42 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.185978+0000 osd.5 (osd.5) 84 : cluster 0 6.19 scrub ok 2024-04-02T04:47:43.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.154841+0000 osd.1 (osd.1) 91 : cluster 0 3.4 deep-scrub starts 2024-04-02T04:47:43.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.172881+0000 osd.1 (osd.1) 92 : cluster 0 3.4 deep-scrub ok 2024-04-02T04:47:43.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.183660+0000 osd.5 (osd.5) 83 : cluster 0 6.19 scrub starts 2024-04-02T04:47:43.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.185978+0000 osd.5 (osd.5) 84 : cluster 0 6.19 scrub ok 2024-04-02T04:47:43.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.154841+0000 osd.1 (osd.1) 91 : cluster 0 3.4 deep-scrub starts 2024-04-02T04:47:43.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.172881+0000 osd.1 (osd.1) 92 : cluster 0 3.4 deep-scrub ok 2024-04-02T04:47:43.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.183660+0000 osd.5 (osd.5) 83 : cluster 0 6.19 scrub starts 2024-04-02T04:47:43.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:42 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.185978+0000 osd.5 (osd.5) 84 : cluster 0 6.19 scrub ok 2024-04-02T04:47:43.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:47:42 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:47:43.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:41.572700+0000 osd.3 (osd.3) 119 : cluster 0 3.b scrub starts 2024-04-02T04:47:43.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:41.576629+0000 osd.3 (osd.3) 120 : cluster 0 3.b scrub ok 2024-04-02T04:47:43.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:41.801447+0000 mgr.y (mgr.14538) 141 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:43.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.549544+0000 osd.3 (osd.3) 121 : cluster 0 2.1e scrub starts 2024-04-02T04:47:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.551238+0000 osd.3 (osd.3) 122 : cluster 0 2.1e scrub ok 2024-04-02T04:47:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.826252+0000 osd.4 (osd.4) 101 : cluster 0 4.9 deep-scrub starts 2024-04-02T04:47:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:42.831434+0000 osd.4 (osd.4) 102 : cluster 0 4.9 deep-scrub ok 2024-04-02T04:47:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:43.148301+0000 osd.1 (osd.1) 93 : cluster 0 2.4 scrub starts 2024-04-02T04:47:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:43 smithi098 bash[23790]: cluster 2024-04-02T04:47:43.151409+0000 osd.1 (osd.1) 94 : cluster 0 2.4 scrub ok 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:41.572700+0000 osd.3 (osd.3) 119 : cluster 0 3.b scrub starts 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:41.576629+0000 osd.3 (osd.3) 120 : cluster 0 3.b scrub ok 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:41.801447+0000 mgr.y (mgr.14538) 141 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.549544+0000 osd.3 (osd.3) 121 : cluster 0 2.1e scrub starts 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.551238+0000 osd.3 (osd.3) 122 : cluster 0 2.1e scrub ok 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.826252+0000 osd.4 (osd.4) 101 : cluster 0 4.9 deep-scrub starts 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:42.831434+0000 osd.4 (osd.4) 102 : cluster 0 4.9 deep-scrub ok 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:43.148301+0000 osd.1 (osd.1) 93 : cluster 0 2.4 scrub starts 2024-04-02T04:47:44.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[21528]: cluster 2024-04-02T04:47:43.151409+0000 osd.1 (osd.1) 94 : cluster 0 2.4 scrub ok 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:41.572700+0000 osd.3 (osd.3) 119 : cluster 0 3.b scrub starts 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:41.576629+0000 osd.3 (osd.3) 120 : cluster 0 3.b scrub ok 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:41.801447+0000 mgr.y (mgr.14538) 141 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.549544+0000 osd.3 (osd.3) 121 : cluster 0 2.1e scrub starts 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.551238+0000 osd.3 (osd.3) 122 : cluster 0 2.1e scrub ok 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.826252+0000 osd.4 (osd.4) 101 : cluster 0 4.9 deep-scrub starts 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:42.831434+0000 osd.4 (osd.4) 102 : cluster 0 4.9 deep-scrub ok 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:43.148301+0000 osd.1 (osd.1) 93 : cluster 0 2.4 scrub starts 2024-04-02T04:47:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:43 smithi069 bash[28904]: cluster 2024-04-02T04:47:43.151409+0000 osd.1 (osd.1) 94 : cluster 0 2.4 scrub ok 2024-04-02T04:47:44.543 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:44.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:44 smithi098 bash[23790]: audit 2024-04-02T04:47:42.895423+0000 mgr.y (mgr.14538) 142 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:44.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:44 smithi098 bash[23790]: cluster 2024-04-02T04:47:44.110074+0000 osd.1 (osd.1) 95 : cluster 0 2.9 scrub starts 2024-04-02T04:47:44.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:44 smithi098 bash[23790]: cluster 2024-04-02T04:47:44.112260+0000 osd.1 (osd.1) 96 : cluster 0 2.9 scrub ok 2024-04-02T04:47:44.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:44 smithi098 bash[23790]: cluster 2024-04-02T04:47:44.209558+0000 osd.5 (osd.5) 85 : cluster 0 3.c scrub starts 2024-04-02T04:47:44.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:44 smithi098 bash[23790]: cluster 2024-04-02T04:47:44.215271+0000 osd.5 (osd.5) 86 : cluster 0 3.c scrub ok 2024-04-02T04:47:45.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[21528]: audit 2024-04-02T04:47:42.895423+0000 mgr.y (mgr.14538) 142 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:45.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[21528]: cluster 2024-04-02T04:47:44.110074+0000 osd.1 (osd.1) 95 : cluster 0 2.9 scrub starts 2024-04-02T04:47:45.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[21528]: cluster 2024-04-02T04:47:44.112260+0000 osd.1 (osd.1) 96 : cluster 0 2.9 scrub ok 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[21528]: cluster 2024-04-02T04:47:44.209558+0000 osd.5 (osd.5) 85 : cluster 0 3.c scrub starts 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[21528]: cluster 2024-04-02T04:47:44.215271+0000 osd.5 (osd.5) 86 : cluster 0 3.c scrub ok 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[28904]: audit 2024-04-02T04:47:42.895423+0000 mgr.y (mgr.14538) 142 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[28904]: cluster 2024-04-02T04:47:44.110074+0000 osd.1 (osd.1) 95 : cluster 0 2.9 scrub starts 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[28904]: cluster 2024-04-02T04:47:44.112260+0000 osd.1 (osd.1) 96 : cluster 0 2.9 scrub ok 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[28904]: cluster 2024-04-02T04:47:44.209558+0000 osd.5 (osd.5) 85 : cluster 0 3.c scrub starts 2024-04-02T04:47:45.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:44 smithi069 bash[28904]: cluster 2024-04-02T04:47:44.215271+0000 osd.5 (osd.5) 86 : cluster 0 3.c scrub ok 2024-04-02T04:47:45.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[21528]: cluster 2024-04-02T04:47:43.802927+0000 mgr.y (mgr.14538) 143 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:45.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[21528]: cluster 2024-04-02T04:47:44.210388+0000 osd.7 (osd.7) 95 : cluster 0 1.0 scrub starts 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[21528]: cluster 2024-04-02T04:47:44.214037+0000 osd.7 (osd.7) 96 : cluster 0 1.0 scrub ok 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.106528+0000 osd.1 (osd.1) 97 : cluster 0 4.2 scrub starts 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.109159+0000 osd.1 (osd.1) 98 : cluster 0 4.2 scrub ok 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[28904]: cluster 2024-04-02T04:47:43.802927+0000 mgr.y (mgr.14538) 143 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[28904]: cluster 2024-04-02T04:47:44.210388+0000 osd.7 (osd.7) 95 : cluster 0 1.0 scrub starts 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[28904]: cluster 2024-04-02T04:47:44.214037+0000 osd.7 (osd.7) 96 : cluster 0 1.0 scrub ok 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.106528+0000 osd.1 (osd.1) 97 : cluster 0 4.2 scrub starts 2024-04-02T04:47:45.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:45 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.109159+0000 osd.1 (osd.1) 98 : cluster 0 4.2 scrub ok 2024-04-02T04:47:45.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:45 smithi098 bash[23790]: cluster 2024-04-02T04:47:43.802927+0000 mgr.y (mgr.14538) 143 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:47:45.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:45 smithi098 bash[23790]: cluster 2024-04-02T04:47:44.210388+0000 osd.7 (osd.7) 95 : cluster 0 1.0 scrub starts 2024-04-02T04:47:45.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:45 smithi098 bash[23790]: cluster 2024-04-02T04:47:44.214037+0000 osd.7 (osd.7) 96 : cluster 0 1.0 scrub ok 2024-04-02T04:47:45.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:45 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.106528+0000 osd.1 (osd.1) 97 : cluster 0 4.2 scrub starts 2024-04-02T04:47:45.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:45 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.109159+0000 osd.1 (osd.1) 98 : cluster 0 4.2 scrub ok 2024-04-02T04:47:46.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.252270+0000 osd.7 (osd.7) 97 : cluster 0 2.1d scrub starts 2024-04-02T04:47:46.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.254439+0000 osd.7 (osd.7) 98 : cluster 0 2.1d scrub ok 2024-04-02T04:47:46.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.514661+0000 osd.3 (osd.3) 123 : cluster 0 3.18 scrub starts 2024-04-02T04:47:46.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.516401+0000 osd.3 (osd.3) 124 : cluster 0 3.18 scrub ok 2024-04-02T04:47:46.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.828027+0000 osd.2 (osd.2) 51 : cluster 0 4.1c scrub starts 2024-04-02T04:47:46.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.830359+0000 osd.2 (osd.2) 52 : cluster 0 4.1c scrub ok 2024-04-02T04:47:46.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.252270+0000 osd.7 (osd.7) 97 : cluster 0 2.1d scrub starts 2024-04-02T04:47:46.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.254439+0000 osd.7 (osd.7) 98 : cluster 0 2.1d scrub ok 2024-04-02T04:47:46.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.514661+0000 osd.3 (osd.3) 123 : cluster 0 3.18 scrub starts 2024-04-02T04:47:46.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.516401+0000 osd.3 (osd.3) 124 : cluster 0 3.18 scrub ok 2024-04-02T04:47:46.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.828027+0000 osd.2 (osd.2) 51 : cluster 0 4.1c scrub starts 2024-04-02T04:47:46.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:46 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.830359+0000 osd.2 (osd.2) 52 : cluster 0 4.1c scrub ok 2024-04-02T04:47:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:46 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.252270+0000 osd.7 (osd.7) 97 : cluster 0 2.1d scrub starts 2024-04-02T04:47:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:46 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.254439+0000 osd.7 (osd.7) 98 : cluster 0 2.1d scrub ok 2024-04-02T04:47:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:46 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.514661+0000 osd.3 (osd.3) 123 : cluster 0 3.18 scrub starts 2024-04-02T04:47:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:46 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.516401+0000 osd.3 (osd.3) 124 : cluster 0 3.18 scrub ok 2024-04-02T04:47:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:46 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.828027+0000 osd.2 (osd.2) 51 : cluster 0 4.1c scrub starts 2024-04-02T04:47:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:46 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.830359+0000 osd.2 (osd.2) 52 : cluster 0 4.1c scrub ok 2024-04-02T04:47:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/2980678460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 msgr2=0x7fc01c10d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- 172.21.15.69:0/2980678460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 0x7fc01c10d7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc01400b0a0 tx=0x7fc01400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/2980678460 shutdown_connections 2024-04-02T04:47:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- 172.21.15.69:0/2980678460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 0x7fc01c10d7b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- 172.21.15.69:0/2980678460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1068e0 0x7fc01c106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- 172.21.15.69:0/2980678460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c1008e0 0x7fc01c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/2980678460 >> 172.21.15.69:0/2980678460 conn(0x7fc01c077740 msgr2=0x7fc01c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/2980678460 shutdown_connections 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/2980678460 wait complete. 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 librados: starting msgr at 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 librados: starting objecter 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 Processor -- start 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- start start 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 librados: setting wanted keys 2024-04-02T04:47:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 librados: calling monclient init 2024-04-02T04:47:47.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1008e0 0x7fc01c102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 0x7fc01c102db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c1068e0 0x7fc01c1032f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c110190 con 0x7fc01c1012a0 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c110010 con 0x7fc01c1068e0 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c110310 con 0x7fc01c1008e0 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0217f7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1008e0 0x7fc01c102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 0x7fc01c102db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0217f7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1008e0 0x7fc01c102870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36352/0 (socket says 172.21.15.69:36352) 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0217f7700 1 -- 172.21.15.69:0/1180302012 learned_addr learned my addr 172.21.15.69:0/1180302012 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1008e0 msgr2=0x7fc01c102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c1068e0 0x7fc01c1032f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1008e0 0x7fc01c102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c1068e0 msgr2=0x7fc01c1032f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c1068e0 0x7fc01c1032f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c1a6590 con 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0217f7700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c1008e0 0x7fc01c102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc020ff6700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 0x7fc01c102db0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00b450 tx=0x7fc00c00b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0127fc700 10 client.?.objecter ms_handle_connect 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0127fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c015020 con 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc00c011cb0 con 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01c1a6880 con 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c010c40 con 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 10 client.14745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c1a6e10 con 0x7fc01c1012a0 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 librados: init done 2024-04-02T04:47:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.120+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc01c04ff00 con 0x7fc01c1012a0 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc00c018020 con 0x7fc01c1012a0 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc00806f5e0 0x7fc008071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(121..121 src has 64..121) ==== 7537+0+0 (secure 0 0 0) 0x7fc00c08d680 con 0x7fc01c1012a0 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 10 client.14745.objecter ms_dispatch 0x7fc01c001550 osd_map(121..121 src has 64..121) 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 3 client.14745.objecter handle_osd_map got epochs [121,121] > 0 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 3 client.14745.objecter handle_osd_map decoding full epoch 121 2024-04-02T04:47:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0217f7700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc00806f5e0 0x7fc008071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 20 client.14745.objecter dump_active .. 0 homeless 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1277 (secure 0 0 0) 0x7fc00c01f020 con 0x7fc01c1012a0 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 0x7fc01c062e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c1a6a50 con 0x7fc01c02db30 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc00806f5e0 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0217f7700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc00806f5e0 0x7fc008071aa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc018011630 tx=0x7fc018009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 0x7fc01c062e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fc01c109f30 con 0x7fc01c02db30 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc01c02db30 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 10 client.14745.objecter resend_mon_ops 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 0x7fc01c062e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc01400b070 tx=0x7fc014004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014020070 con 0x7fc01c02db30 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc0140078f0 con 0x7fc01c02db30 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 msgr2=0x7fc01c062e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 0x7fc01c062e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc01400b070 tx=0x7fc014004d10 comp rx=0 tx=0).stop 2024-04-02T04:47:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 msgr2=0x7fc01c062e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T04:47:47.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.124+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c02db30 0x7fc01c062e50 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc01400b070 tx=0x7fc014004d10 comp rx=0 tx=0).stop 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 0x7fc01c1ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c103da0 con 0x7fc01c063390 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 0x7fc01c1ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7fc01c02df10 con 0x7fc01c063390 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc01c063390 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc0127fc700 10 client.14745.objecter resend_mon_ops 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 0x7fc01c1ab0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc01c106e00 tx=0x7fc0140099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.132+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0140340a0 con 0x7fc01c063390 2024-04-02T04:47:47.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.0 v2:172.21.15.69:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7fc01402f470 con 0x7fc01c063390 2024-04-02T04:47:47.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 msgr2=0x7fc01c1ab0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 0x7fc01c1ab0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc01c106e00 tx=0x7fc0140099e0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 msgr2=0x7fc01c1ab0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T04:47:47.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c063390 0x7fc01c1ab0c0 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc01c106e00 tx=0x7fc0140099e0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.136 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a: {} 2024-04-02T04:47:47.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 0x7fc01c108d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c109260 con 0x7fc01c108910 2024-04-02T04:47:47.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 0x7fc01c108d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fc01c109460 con 0x7fc01c108910 2024-04-02T04:47:47.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 0x7fc01c108d20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc0140062a0 tx=0x7fc014006210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc01c108910 2024-04-02T04:47:47.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 10 client.14745.objecter resend_mon_ops 2024-04-02T04:47:47.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01401c930 con 0x7fc01c108910 2024-04-02T04:47:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.1 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc01401c930 con 0x7fc01c108910 2024-04-02T04:47:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 msgr2=0x7fc01c108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 0x7fc01c108d20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc0140062a0 tx=0x7fc014006210 comp rx=0 tx=0).stop 2024-04-02T04:47:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 msgr2=0x7fc01c108d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T04:47:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c108910 0x7fc01c108d20 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc0140062a0 tx=0x7fc014006210 comp rx=0 tx=0).stop 2024-04-02T04:47:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.136+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 reap_dead start 2024-04-02T04:47:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 0x7fc01c1032f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c1a59b0 con 0x7fc01c102990 2024-04-02T04:47:47.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 0x7fc01c1032f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7fc01c1a5b80 con 0x7fc01c102990 2024-04-02T04:47:47.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 0x7fc01c1032f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc01c109430 tx=0x7fc01402c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc01c102990 2024-04-02T04:47:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc0127fc700 10 client.14745.objecter resend_mon_ops 2024-04-02T04:47:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.144+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014003c00 con 0x7fc01c102990 2024-04-02T04:47:47.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.1 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7fc014003c00 con 0x7fc01c102990 2024-04-02T04:47:47.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 msgr2=0x7fc01c1032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 0x7fc01c1032f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc01c109430 tx=0x7fc01402c000 comp rx=0 tx=0).stop 2024-04-02T04:47:47.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 msgr2=0x7fc01c1032f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T04:47:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c102990 0x7fc01c1032f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:47.151 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b: {} 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 0x7fc01c1a4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c1a5eb0 con 0x7fc01c109670 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 0x7fc01c1a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fc01c1a6060 con 0x7fc01c109670 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 0x7fc01c1a4210 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0140062a0 tx=0x7fc014032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc01c109670 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.148+0000 7fc0127fc700 10 client.14745.objecter resend_mon_ops 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.152+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014050070 con 0x7fc01c109670 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.152+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.2 v2:172.21.15.69:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc014043880 con 0x7fc01c109670 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.152+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 msgr2=0x7fc01c1a4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.152+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 0x7fc01c1a4210 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0140062a0 tx=0x7fc014032000 comp rx=0 tx=0).stop 2024-04-02T04:47:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.152+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 msgr2=0x7fc01c1a4210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T04:47:47.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.152+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109670 0x7fc01c1a4210 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0140062a0 tx=0x7fc014032000 comp rx=0 tx=0).stop 2024-04-02T04:47:47.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.156+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 0x7fc01c1a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:47.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.156+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c1b2dd0 con 0x7fc01c109a50 2024-04-02T04:47:47.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.156+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 0x7fc01c1a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:47.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7fc01c1b2f80 con 0x7fc01c109a50 2024-04-02T04:47:47.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc021ff8700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 0x7fc01c1a4750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc014026fd0 tx=0x7fc014013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:47.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 10 client.14745.objecter ms_handle_connect 0x7fc01c109a50 2024-04-02T04:47:47.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 10 client.14745.objecter resend_mon_ops 2024-04-02T04:47:47.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01405b070 con 0x7fc01c109a50 2024-04-02T04:47:47.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 <== mon.2 v2:172.21.15.69:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7fc01404c4f0 con 0x7fc01c109a50 2024-04-02T04:47:47.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 msgr2=0x7fc01c1a4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 0x7fc01c1a4750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc014026fd0 tx=0x7fc014013000 comp rx=0 tx=0).stop 2024-04-02T04:47:47.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 msgr2=0x7fc01c1a4750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T04:47:47.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc0127fc700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc01c109a50 0x7fc01c1a4750 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc014026fd0 tx=0x7fc014013000 comp rx=0 tx=0).stop 2024-04-02T04:47:47.161 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c: {} 2024-04-02T04:47:47.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 10 librados: watch_flush enter 2024-04-02T04:47:47.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 10 librados: watch_flush exit 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 20 client.14745.objecter shutdown clearing up homeless session... 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 10 client.14745.objecter successfully canceled tick 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc00806f5e0 msgr2=0x7fc008071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc00806f5e0 0x7fc008071aa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc018011630 tx=0x7fc018009260 comp rx=0 tx=0).stop 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 msgr2=0x7fc01c102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 --2- 172.21.15.69:0/1180302012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc01c1012a0 0x7fc01c102db0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00b450 tx=0x7fc00c00b760 comp rx=0 tx=0).stop 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc021ff8700 1 -- 172.21.15.69:0/1180302012 reap_dead start 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 shutdown_connections 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 >> 172.21.15.69:0/1180302012 conn(0x7fc01c077740 msgr2=0x7fc01c0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 shutdown_connections 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 -- 172.21.15.69:0/1180302012 wait complete. 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:47.160+0000 7fc023a5b700 1 librados: shutdown 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-02T04:47:47.163 INFO:teuthology.orchestra.run.smithi069.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-02T04:47:47.164 INFO:teuthology.orchestra.run.smithi069.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-02T04:47:47.793 INFO:teuthology.run_tasks:Running task workunit... 2024-04-02T04:47:47.805 INFO:tasks.workunit:Pulling workunits from ref 6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:47:47.806 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-02T04:47:47.806 DEBUG:teuthology.orchestra.run.smithi069:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-02T04:47:47.807 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-02T04:47:47.807 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:47:47.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:47:47.807 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-02T04:47:47.807 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-02T04:47:47.807 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:47:47.808 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-02T04:47:47.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:47.808 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-02T04:47:47.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:47:47.809 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-02T04:47:47.809 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-02T04:47:47.809 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-02T04:47:47.810 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:47:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:47.817 INFO:teuthology.orchestra.run.smithi069.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-02T04:47:47.818 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-02T04:47:47.837 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:47.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:45.803603+0000 mgr.y (mgr.14538) 144 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-02T04:47:47.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:46.275516+0000 osd.5 (osd.5) 87 : cluster 0 2.16 scrub starts 2024-04-02T04:47:47.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:46.277879+0000 osd.5 (osd.5) 88 : cluster 0 2.16 scrub ok 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:47.104942+0000 osd.1 (osd.1) 99 : cluster 0 4.4 deep-scrub starts 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:47.106493+0000 osd.1 (osd.1) 100 : cluster 0 4.4 deep-scrub ok 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:47.121192+0000 osd.0 (osd.0) 97 : cluster 0 6.0 scrub starts 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: cluster 2024-04-02T04:47:47.122208+0000 osd.0 (osd.0) 98 : cluster 0 6.0 scrub ok 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[21528]: audit 2024-04-02T04:47:47.128146+0000 mon.a (mon.0) 923 : audit 0 from='client.? 172.21.15.69:0/1180302012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:45.803603+0000 mgr.y (mgr.14538) 144 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:46.275516+0000 osd.5 (osd.5) 87 : cluster 0 2.16 scrub starts 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:46.277879+0000 osd.5 (osd.5) 88 : cluster 0 2.16 scrub ok 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:47.104942+0000 osd.1 (osd.1) 99 : cluster 0 4.4 deep-scrub starts 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:47.106493+0000 osd.1 (osd.1) 100 : cluster 0 4.4 deep-scrub ok 2024-04-02T04:47:47.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:47.121192+0000 osd.0 (osd.0) 97 : cluster 0 6.0 scrub starts 2024-04-02T04:47:47.841 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: cluster 2024-04-02T04:47:47.122208+0000 osd.0 (osd.0) 98 : cluster 0 6.0 scrub ok 2024-04-02T04:47:47.841 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:47 smithi069 bash[28904]: audit 2024-04-02T04:47:47.128146+0000 mon.a (mon.0) 923 : audit 0 from='client.? 172.21.15.69:0/1180302012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:47:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:47.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:47.842 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-02T04:47:47.843 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-02T04:47:47.890 INFO:tasks.workunit:timeout=3h 2024-04-02T04:47:47.890 INFO:tasks.workunit:cleanup=True 2024-04-02T04:47:47.891 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:47:47.935 INFO:tasks.workunit.client.0.smithi069.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-02T04:47:47.943 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:45.803603+0000 mgr.y (mgr.14538) 144 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:46.275516+0000 osd.5 (osd.5) 87 : cluster 0 2.16 scrub starts 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:46.277879+0000 osd.5 (osd.5) 88 : cluster 0 2.16 scrub ok 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:47.104942+0000 osd.1 (osd.1) 99 : cluster 0 4.4 deep-scrub starts 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:47.106493+0000 osd.1 (osd.1) 100 : cluster 0 4.4 deep-scrub ok 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:47.121192+0000 osd.0 (osd.0) 97 : cluster 0 6.0 scrub starts 2024-04-02T04:47:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: cluster 2024-04-02T04:47:47.122208+0000 osd.0 (osd.0) 98 : cluster 0 6.0 scrub ok 2024-04-02T04:47:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:47 smithi098 bash[23790]: audit 2024-04-02T04:47:47.128146+0000 mon.a (mon.0) 923 : audit 0 from='client.? 172.21.15.69:0/1180302012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:47:47.976 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:47.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:48.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:48.099 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:48.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:48.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:48.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:48.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:48.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:48.750 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:48.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:47:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:47:48.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:48 smithi069 bash[21528]: cluster 2024-04-02T04:47:48.243842+0000 osd.5 (osd.5) 89 : cluster 0 4.8 scrub starts 2024-04-02T04:47:48.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:48 smithi069 bash[21528]: cluster 2024-04-02T04:47:48.245174+0000 osd.5 (osd.5) 90 : cluster 0 4.8 scrub ok 2024-04-02T04:47:48.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:48 smithi069 bash[28904]: cluster 2024-04-02T04:47:48.243842+0000 osd.5 (osd.5) 89 : cluster 0 4.8 scrub starts 2024-04-02T04:47:48.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:48 smithi069 bash[28904]: cluster 2024-04-02T04:47:48.245174+0000 osd.5 (osd.5) 90 : cluster 0 4.8 scrub ok 2024-04-02T04:47:48.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:48.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:48 smithi098 bash[23790]: cluster 2024-04-02T04:47:48.243842+0000 osd.5 (osd.5) 89 : cluster 0 4.8 scrub starts 2024-04-02T04:47:48.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:48 smithi098 bash[23790]: cluster 2024-04-02T04:47:48.245174+0000 osd.5 (osd.5) 90 : cluster 0 4.8 scrub ok 2024-04-02T04:47:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:48.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:48.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:49.009 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:49.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:49.138 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:49.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:49.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:49.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:49.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:49.526 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:49.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:49.640 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:49 smithi098 bash[23790]: cluster 2024-04-02T04:47:47.804629+0000 mgr.y (mgr.14538) 145 : cluster 0 pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:49.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:49.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:49 smithi069 bash[21528]: cluster 2024-04-02T04:47:47.804629+0000 mgr.y (mgr.14538) 145 : cluster 0 pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:49.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:49 smithi069 bash[28904]: cluster 2024-04-02T04:47:47.804629+0000 mgr.y (mgr.14538) 145 : cluster 0 pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:49.786 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:49.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:49.913 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:49.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:50.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:50.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:50.171 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:50.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:50.301 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:50.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:50.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:50.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:50.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:50 smithi098 bash[23790]: cluster 2024-04-02T04:47:49.222193+0000 osd.7 (osd.7) 99 : cluster 0 6.3 scrub starts 2024-04-02T04:47:50.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:50 smithi098 bash[23790]: cluster 2024-04-02T04:47:49.224495+0000 osd.7 (osd.7) 100 : cluster 0 6.3 scrub ok 2024-04-02T04:47:50.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:50 smithi098 bash[23790]: cluster 2024-04-02T04:47:49.836395+0000 osd.2 (osd.2) 53 : cluster 0 2.10 deep-scrub starts 2024-04-02T04:47:50.679 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:50 smithi098 bash[23790]: cluster 2024-04-02T04:47:49.838989+0000 osd.2 (osd.2) 54 : cluster 0 2.10 deep-scrub ok 2024-04-02T04:47:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:50.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:50.824 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:50.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:50.953 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:50.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:51.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[21528]: cluster 2024-04-02T04:47:49.222193+0000 osd.7 (osd.7) 99 : cluster 0 6.3 scrub starts 2024-04-02T04:47:51.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[21528]: cluster 2024-04-02T04:47:49.224495+0000 osd.7 (osd.7) 100 : cluster 0 6.3 scrub ok 2024-04-02T04:47:51.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[21528]: cluster 2024-04-02T04:47:49.836395+0000 osd.2 (osd.2) 53 : cluster 0 2.10 deep-scrub starts 2024-04-02T04:47:51.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[21528]: cluster 2024-04-02T04:47:49.838989+0000 osd.2 (osd.2) 54 : cluster 0 2.10 deep-scrub ok 2024-04-02T04:47:51.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[28904]: cluster 2024-04-02T04:47:49.222193+0000 osd.7 (osd.7) 99 : cluster 0 6.3 scrub starts 2024-04-02T04:47:51.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[28904]: cluster 2024-04-02T04:47:49.224495+0000 osd.7 (osd.7) 100 : cluster 0 6.3 scrub ok 2024-04-02T04:47:51.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[28904]: cluster 2024-04-02T04:47:49.836395+0000 osd.2 (osd.2) 53 : cluster 0 2.10 deep-scrub starts 2024-04-02T04:47:51.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:50 smithi069 bash[28904]: cluster 2024-04-02T04:47:49.838989+0000 osd.2 (osd.2) 54 : cluster 0 2.10 deep-scrub ok 2024-04-02T04:47:51.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:51.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:51.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:51.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:51.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:51.705 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:51.732 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:51.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:51.846 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:51 smithi098 bash[23790]: cluster 2024-04-02T04:47:49.805300+0000 mgr.y (mgr.14538) 146 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:51.846 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:51 smithi098 bash[23790]: cluster 2024-04-02T04:47:50.753271+0000 osd.4 (osd.4) 103 : cluster 0 3.3 scrub starts 2024-04-02T04:47:51.846 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:51 smithi098 bash[23790]: cluster 2024-04-02T04:47:50.758301+0000 osd.4 (osd.4) 104 : cluster 0 3.3 scrub ok 2024-04-02T04:47:51.860 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:51.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:51.991 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:51.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:52.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:51 smithi069 bash[21528]: cluster 2024-04-02T04:47:49.805300+0000 mgr.y (mgr.14538) 146 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:52.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:51 smithi069 bash[21528]: cluster 2024-04-02T04:47:50.753271+0000 osd.4 (osd.4) 103 : cluster 0 3.3 scrub starts 2024-04-02T04:47:52.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:51 smithi069 bash[21528]: cluster 2024-04-02T04:47:50.758301+0000 osd.4 (osd.4) 104 : cluster 0 3.3 scrub ok 2024-04-02T04:47:52.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:51 smithi069 bash[28904]: cluster 2024-04-02T04:47:49.805300+0000 mgr.y (mgr.14538) 146 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:52.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:51 smithi069 bash[28904]: cluster 2024-04-02T04:47:50.753271+0000 osd.4 (osd.4) 103 : cluster 0 3.3 scrub starts 2024-04-02T04:47:52.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:51 smithi069 bash[28904]: cluster 2024-04-02T04:47:50.758301+0000 osd.4 (osd.4) 104 : cluster 0 3.3 scrub ok 2024-04-02T04:47:52.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:52.121 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:52.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:52.253 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:52.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:52.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:47:52.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:52.639 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:52 smithi098 bash[23790]: audit 2024-04-02T04:47:52.141173+0000 mon.b (mon.1) 97 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:52 smithi098 bash[23790]: cluster 2024-04-02T04:47:52.447880+0000 osd.3 (osd.3) 125 : cluster 0 4.19 scrub starts 2024-04-02T04:47:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:52 smithi098 bash[23790]: cluster 2024-04-02T04:47:52.450172+0000 osd.3 (osd.3) 126 : cluster 0 4.19 scrub ok 2024-04-02T04:47:52.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:52.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:52.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:52 smithi069 bash[21528]: audit 2024-04-02T04:47:52.141173+0000 mon.b (mon.1) 97 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:52.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:52 smithi069 bash[21528]: cluster 2024-04-02T04:47:52.447880+0000 osd.3 (osd.3) 125 : cluster 0 4.19 scrub starts 2024-04-02T04:47:52.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:52 smithi069 bash[21528]: cluster 2024-04-02T04:47:52.450172+0000 osd.3 (osd.3) 126 : cluster 0 4.19 scrub ok 2024-04-02T04:47:52.882 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:52 smithi069 bash[28904]: audit 2024-04-02T04:47:52.141173+0000 mon.b (mon.1) 97 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:52.882 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:52 smithi069 bash[28904]: cluster 2024-04-02T04:47:52.447880+0000 osd.3 (osd.3) 125 : cluster 0 4.19 scrub starts 2024-04-02T04:47:52.882 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:52 smithi069 bash[28904]: cluster 2024-04-02T04:47:52.450172+0000 osd.3 (osd.3) 126 : cluster 0 4.19 scrub ok 2024-04-02T04:47:52.898 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:52.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:53.000 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:53.028 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:53.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:53.158 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:53.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:47:52 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:47:53.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:53.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:53.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:53.546 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:53.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:53.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[21528]: cluster 2024-04-02T04:47:51.806000+0000 mgr.y (mgr.14538) 147 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.170566+0000 osd.5 (osd.5) 91 : cluster 0 6.d scrub starts 2024-04-02T04:47:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.172781+0000 osd.5 (osd.5) 92 : cluster 0 6.d scrub ok 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.177213+0000 osd.0 (osd.0) 99 : cluster 0 3.12 scrub starts 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.179306+0000 osd.0 (osd.0) 100 : cluster 0 3.12 scrub ok 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[28904]: cluster 2024-04-02T04:47:51.806000+0000 mgr.y (mgr.14538) 147 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.170566+0000 osd.5 (osd.5) 91 : cluster 0 6.d scrub starts 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.172781+0000 osd.5 (osd.5) 92 : cluster 0 6.d scrub ok 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.177213+0000 osd.0 (osd.0) 99 : cluster 0 3.12 scrub starts 2024-04-02T04:47:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:53 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.179306+0000 osd.0 (osd.0) 100 : cluster 0 3.12 scrub ok 2024-04-02T04:47:53.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:53.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:53 smithi098 bash[23790]: cluster 2024-04-02T04:47:51.806000+0000 mgr.y (mgr.14538) 147 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:53.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:53 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.170566+0000 osd.5 (osd.5) 91 : cluster 0 6.d scrub starts 2024-04-02T04:47:53.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:53 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.172781+0000 osd.5 (osd.5) 92 : cluster 0 6.d scrub ok 2024-04-02T04:47:53.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:53 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.177213+0000 osd.0 (osd.0) 99 : cluster 0 3.12 scrub starts 2024-04-02T04:47:53.791 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:53 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.179306+0000 osd.0 (osd.0) 100 : cluster 0 3.12 scrub ok 2024-04-02T04:47:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:53.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:53.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:54.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:54.066 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:54.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:54.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:54.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:54.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:54.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:54.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:54.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:54.704 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: audit 2024-04-02T04:47:52.901155+0000 mgr.y (mgr.14538) 148 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.268526+0000 osd.7 (osd.7) 101 : cluster 0 6.5 scrub starts 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.270708+0000 osd.7 (osd.7) 102 : cluster 0 6.5 scrub ok 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.937208+0000 osd.6 (osd.6) 65 : cluster 0 2.1a scrub starts 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.938532+0000 osd.6 (osd.6) 66 : cluster 0 2.1a scrub ok 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: cluster 2024-04-02T04:47:54.212380+0000 osd.0 (osd.0) 101 : cluster 0 2.1f scrub starts 2024-04-02T04:47:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:54 smithi098 bash[23790]: cluster 2024-04-02T04:47:54.214536+0000 osd.0 (osd.0) 102 : cluster 0 2.1f scrub ok 2024-04-02T04:47:54.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: audit 2024-04-02T04:47:52.901155+0000 mgr.y (mgr.14538) 148 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.268526+0000 osd.7 (osd.7) 101 : cluster 0 6.5 scrub starts 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.270708+0000 osd.7 (osd.7) 102 : cluster 0 6.5 scrub ok 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.937208+0000 osd.6 (osd.6) 65 : cluster 0 2.1a scrub starts 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.938532+0000 osd.6 (osd.6) 66 : cluster 0 2.1a scrub ok 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: cluster 2024-04-02T04:47:54.212380+0000 osd.0 (osd.0) 101 : cluster 0 2.1f scrub starts 2024-04-02T04:47:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[21528]: cluster 2024-04-02T04:47:54.214536+0000 osd.0 (osd.0) 102 : cluster 0 2.1f scrub ok 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: audit 2024-04-02T04:47:52.901155+0000 mgr.y (mgr.14538) 148 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.268526+0000 osd.7 (osd.7) 101 : cluster 0 6.5 scrub starts 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.270708+0000 osd.7 (osd.7) 102 : cluster 0 6.5 scrub ok 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.937208+0000 osd.6 (osd.6) 65 : cluster 0 2.1a scrub starts 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.938532+0000 osd.6 (osd.6) 66 : cluster 0 2.1a scrub ok 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: cluster 2024-04-02T04:47:54.212380+0000 osd.0 (osd.0) 101 : cluster 0 2.1f scrub starts 2024-04-02T04:47:54.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:54 smithi069 bash[28904]: cluster 2024-04-02T04:47:54.214536+0000 osd.0 (osd.0) 102 : cluster 0 2.1f scrub ok 2024-04-02T04:47:54.829 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:54.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:54.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:55.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:55.088 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:55.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:55.218 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:55.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:55.342 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:55.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:55.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:55.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:55.733 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:55.836 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:55.864 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:55.952 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:55 smithi069 bash[28904]: cluster 2024-04-02T04:47:53.807452+0000 mgr.y (mgr.14538) 149 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:55.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:55 smithi069 bash[21528]: cluster 2024-04-02T04:47:53.807452+0000 mgr.y (mgr.14538) 149 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:55.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:55.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:55 smithi098 bash[23790]: cluster 2024-04-02T04:47:53.807452+0000 mgr.y (mgr.14538) 149 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:55.994 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:55.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:56.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:56.125 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:56.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:56.256 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:56.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:56.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:56.516 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:56.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:56.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[21528]: cluster 2024-04-02T04:47:55.853752+0000 osd.2 (osd.2) 55 : cluster 0 4.f scrub starts 2024-04-02T04:47:56.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[21528]: cluster 2024-04-02T04:47:55.855504+0000 osd.2 (osd.2) 56 : cluster 0 4.f scrub ok 2024-04-02T04:47:56.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[21528]: cluster 2024-04-02T04:47:55.937830+0000 osd.6 (osd.6) 67 : cluster 0 4.a scrub starts 2024-04-02T04:47:56.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[21528]: cluster 2024-04-02T04:47:55.939974+0000 osd.6 (osd.6) 68 : cluster 0 4.a scrub ok 2024-04-02T04:47:56.647 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:56.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[28904]: cluster 2024-04-02T04:47:55.853752+0000 osd.2 (osd.2) 55 : cluster 0 4.f scrub starts 2024-04-02T04:47:56.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[28904]: cluster 2024-04-02T04:47:55.855504+0000 osd.2 (osd.2) 56 : cluster 0 4.f scrub ok 2024-04-02T04:47:56.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[28904]: cluster 2024-04-02T04:47:55.937830+0000 osd.6 (osd.6) 67 : cluster 0 4.a scrub starts 2024-04-02T04:47:56.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:56 smithi069 bash[28904]: cluster 2024-04-02T04:47:55.939974+0000 osd.6 (osd.6) 68 : cluster 0 4.a scrub ok 2024-04-02T04:47:56.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:56.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:56.892 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:56 smithi098 bash[23790]: cluster 2024-04-02T04:47:55.853752+0000 osd.2 (osd.2) 55 : cluster 0 4.f scrub starts 2024-04-02T04:47:56.892 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:56 smithi098 bash[23790]: cluster 2024-04-02T04:47:55.855504+0000 osd.2 (osd.2) 56 : cluster 0 4.f scrub ok 2024-04-02T04:47:56.892 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:56 smithi098 bash[23790]: cluster 2024-04-02T04:47:55.937830+0000 osd.6 (osd.6) 67 : cluster 0 4.a scrub starts 2024-04-02T04:47:56.892 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:56 smithi098 bash[23790]: cluster 2024-04-02T04:47:55.939974+0000 osd.6 (osd.6) 68 : cluster 0 4.a scrub ok 2024-04-02T04:47:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:56.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:57.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:57.041 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:57.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:47:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:47:57.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:57.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:57.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:57.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:57.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:47:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:47:57.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:57.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:57.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:57.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:57.929 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:57 smithi098 bash[23790]: cluster 2024-04-02T04:47:55.808166+0000 mgr.y (mgr.14538) 150 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:57.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:57 smithi098 bash[23790]: cluster 2024-04-02T04:47:56.700018+0000 osd.4 (osd.4) 105 : cluster 0 4.d deep-scrub starts 2024-04-02T04:47:57.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:57 smithi098 bash[23790]: cluster 2024-04-02T04:47:56.704368+0000 osd.4 (osd.4) 106 : cluster 0 4.d deep-scrub ok 2024-04-02T04:47:57.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:57 smithi098 bash[23790]: cluster 2024-04-02T04:47:57.457268+0000 osd.3 (osd.3) 127 : cluster 0 4.1d scrub starts 2024-04-02T04:47:57.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:57 smithi098 bash[23790]: cluster 2024-04-02T04:47:57.459035+0000 osd.3 (osd.3) 128 : cluster 0 4.1d scrub ok 2024-04-02T04:47:57.945 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:57.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[21528]: cluster 2024-04-02T04:47:55.808166+0000 mgr.y (mgr.14538) 150 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[21528]: cluster 2024-04-02T04:47:56.700018+0000 osd.4 (osd.4) 105 : cluster 0 4.d deep-scrub starts 2024-04-02T04:47:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[21528]: cluster 2024-04-02T04:47:56.704368+0000 osd.4 (osd.4) 106 : cluster 0 4.d deep-scrub ok 2024-04-02T04:47:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[21528]: cluster 2024-04-02T04:47:57.457268+0000 osd.3 (osd.3) 127 : cluster 0 4.1d scrub starts 2024-04-02T04:47:58.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[21528]: cluster 2024-04-02T04:47:57.459035+0000 osd.3 (osd.3) 128 : cluster 0 4.1d scrub ok 2024-04-02T04:47:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[28904]: cluster 2024-04-02T04:47:55.808166+0000 mgr.y (mgr.14538) 150 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:47:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[28904]: cluster 2024-04-02T04:47:56.700018+0000 osd.4 (osd.4) 105 : cluster 0 4.d deep-scrub starts 2024-04-02T04:47:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[28904]: cluster 2024-04-02T04:47:56.704368+0000 osd.4 (osd.4) 106 : cluster 0 4.d deep-scrub ok 2024-04-02T04:47:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[28904]: cluster 2024-04-02T04:47:57.457268+0000 osd.3 (osd.3) 127 : cluster 0 4.1d scrub starts 2024-04-02T04:47:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:57 smithi069 bash[28904]: cluster 2024-04-02T04:47:57.459035+0000 osd.3 (osd.3) 128 : cluster 0 4.1d scrub ok 2024-04-02T04:47:58.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:58.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:58.200 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:58.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.325 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:58.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:58.456 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:58.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:58.586 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:58.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.717 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:58.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.845 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:58.948 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:47:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:57.881770+0000 osd.6 (osd.6) 69 : cluster 0 3.a scrub starts 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:57.887055+0000 osd.6 (osd.6) 70 : cluster 0 3.a scrub ok 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:58.107461+0000 osd.1 (osd.1) 101 : cluster 0 2.6 deep-scrub starts 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:58.111555+0000 osd.1 (osd.1) 102 : cluster 0 2.6 deep-scrub ok 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:58.183825+0000 osd.5 (osd.5) 93 : cluster 0 6.7 deep-scrub starts 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:58.185857+0000 osd.5 (osd.5) 94 : cluster 0 6.7 deep-scrub ok 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:58.470716+0000 osd.3 (osd.3) 129 : cluster 0 2.19 scrub starts 2024-04-02T04:47:58.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[28904]: cluster 2024-04-02T04:47:58.472698+0000 osd.3 (osd.3) 130 : cluster 0 2.19 scrub ok 2024-04-02T04:47:58.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:57.881770+0000 osd.6 (osd.6) 69 : cluster 0 3.a scrub starts 2024-04-02T04:47:58.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:57.887055+0000 osd.6 (osd.6) 70 : cluster 0 3.a scrub ok 2024-04-02T04:47:58.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:58.107461+0000 osd.1 (osd.1) 101 : cluster 0 2.6 deep-scrub starts 2024-04-02T04:47:58.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:58.111555+0000 osd.1 (osd.1) 102 : cluster 0 2.6 deep-scrub ok 2024-04-02T04:47:58.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:58.183825+0000 osd.5 (osd.5) 93 : cluster 0 6.7 deep-scrub starts 2024-04-02T04:47:58.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:58.185857+0000 osd.5 (osd.5) 94 : cluster 0 6.7 deep-scrub ok 2024-04-02T04:47:58.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:58.470716+0000 osd.3 (osd.3) 129 : cluster 0 2.19 scrub starts 2024-04-02T04:47:58.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:58 smithi069 bash[21528]: cluster 2024-04-02T04:47:58.472698+0000 osd.3 (osd.3) 130 : cluster 0 2.19 scrub ok 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:57.881770+0000 osd.6 (osd.6) 69 : cluster 0 3.a scrub starts 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:57.887055+0000 osd.6 (osd.6) 70 : cluster 0 3.a scrub ok 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:58.107461+0000 osd.1 (osd.1) 101 : cluster 0 2.6 deep-scrub starts 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:58.111555+0000 osd.1 (osd.1) 102 : cluster 0 2.6 deep-scrub ok 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:58.183825+0000 osd.5 (osd.5) 93 : cluster 0 6.7 deep-scrub starts 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:58.185857+0000 osd.5 (osd.5) 94 : cluster 0 6.7 deep-scrub ok 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:58.470716+0000 osd.3 (osd.3) 129 : cluster 0 2.19 scrub starts 2024-04-02T04:47:58.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:58 smithi098 bash[23790]: cluster 2024-04-02T04:47:58.472698+0000 osd.3 (osd.3) 130 : cluster 0 2.19 scrub ok 2024-04-02T04:47:58.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:58.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:59.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:59.103 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:47:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:47:59.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:59.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:59.363 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:47:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:47:59.466 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:59.493 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:47:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:47:59.596 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:59.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/903748906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588107220 msgr2=0x7f1588107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- 172.21.15.69:0/903748906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588107220 0x7f1588107600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f15780098f0 tx=0x7f1578009c00 comp rx=0 tx=0).stop 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/903748906 shutdown_connections 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- 172.21.15.69:0/903748906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1588101b60 0x7f158810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- 172.21.15.69:0/903748906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588107220 0x7f1588107600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- 172.21.15.69:0/903748906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 0x7f1588101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/903748906 >> 172.21.15.69:0/903748906 conn(0x7f15880771e0 msgr2=0x7f15880775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/903748906 shutdown_connections 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/903748906 wait complete. 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 librados: starting msgr at 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 librados: starting objecter 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 Processor -- start 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- start start 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 librados: setting wanted keys 2024-04-02T04:47:59.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 librados: calling monclient init 2024-04-02T04:47:59.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 0x7f1588102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588101b60 0x7f1588103360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1588107220 0x7f15881038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1588110950 con 0x7f15881011a0 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15881107d0 con 0x7f1588107220 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1588110ad0 con 0x7f1588101b60 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 0x7f1588102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 0x7f1588102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39680/0 (socket says 172.21.15.69:39680) 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 -- 172.21.15.69:0/3748401737 learned_addr learned my addr 172.21.15.69:0/3748401737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588101b60 0x7f1588103360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588101b60 msgr2=0x7f1588103360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1588107220 0x7f15881038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588101b60 0x7f1588103360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1588107220 msgr2=0x7f15881038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1588107220 0x7f15881038a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15881a6500 con 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1588101b60 0x7f1588103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 0x7f1588102e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1578005950 tx=0x7f157800fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15857fa700 10 client.?.objecter ms_handle_connect 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15857fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157800b7d0 con 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f15780054c0 con 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578003aa0 con 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15881a6790 con 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15881a6c40 con 0x7f15881011a0 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 librados: init done 2024-04-02T04:47:59.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f158804ff00 con 0x7f15881011a0 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.692+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f157800b970 con 0x7f15881011a0 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f157006f5e0 0x7f1570071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(121..121 src has 64..121) ==== 7537+0+0 (secure 0 0 0) 0x7f157808f510 con 0x7f15881011a0 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 osd_map(121..121 src has 64..121) 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 3 client.14757.objecter handle_osd_map got epochs [121,121] > 0 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 3 client.14757.objecter handle_osd_map decoding full epoch 121 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f157006f5e0 0x7f1570071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 20 client.14757.objecter dump_active .. 0 homeless 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v121) ==== 54+0+16 (secure 0 0 0) 0x7f1578013030 con 0x7f15881011a0 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 10 client.14757.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 1 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f1588062e50 0x7f1588063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 1 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 0 1 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 4 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 10 client.14757.objecter _send_command 1 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15881aa030 con 0x7f1588062e50 2024-04-02T04:47:59.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f157006f5e0 0x7f1570071aa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f157c009750 tx=0x7f157c009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f157006f5e0 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f1588062e50 0x7f1588063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f1588062e50 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f1588062e50 0x7f1588063230 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881aa030 con 0x7f1588062e50 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 1: 0 ) 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 10 client.14757.objecter _finish_command 1 = system:0 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 4 2024-04-02T04:47:59.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.696+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 0 1 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 10 client.14757.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 2 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 2 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f1588104300 osd=0 3 2024-04-02T04:47:59.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 0 2 2024-04-02T04:47:59.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 4 2024-04-02T04:47:59.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 10 client.14757.objecter _send_command 2 2024-04-02T04:47:59.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.712+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f15880628d0 con 0x7f1588062e50 2024-04-02T04:47:59.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f15880628d0 con 0x7f1588062e50 2024-04-02T04:47:59.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 10 client.14757.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 4 2024-04-02T04:47:59.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 0 2 2024-04-02T04:47:59.717 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 10 client.14757.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 3 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f158802c9b0 0x7f158802cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 3 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 1 3 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 4 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 10 client.14757.objecter _send_command 3 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f158810ad70 con 0x7f158802c9b0 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f158802c9b0 0x7f158802cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f158802c9b0 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f158802c9b0 0x7f158802cd90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f158810ad70 con 0x7f158802c9b0 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 3: 0 ) 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 10 client.14757.objecter _finish_command 3 = system:0 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 4 2024-04-02T04:47:59.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.716+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 1 3 2024-04-02T04:47:59.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 10 client.14757.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 4 2024-04-02T04:47:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 4 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f158810a6f0 osd=1 3 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 1 4 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 4 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 10 client.14757.objecter _send_command 4 2024-04-02T04:47:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f158810a170 con 0x7f158802c9b0 2024-04-02T04:47:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f158810a170 con 0x7f158802c9b0 2024-04-02T04:47:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 10 client.14757.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 4 2024-04-02T04:47:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 1 4 2024-04-02T04:47:59.738 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:47:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 10 client.14757.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 5 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f15881ac1f0 0x7f15881ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 5 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 2 5 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 4 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 10 client.14757.objecter _send_command 5 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f15881b1440 con 0x7f15881ac1f0 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f15881ac1f0 0x7f15881ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f15881ac1f0 0x7f15881ae5e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f15881ac1f0 2024-04-02T04:47:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881b1440 con 0x7f15881ac1f0 2024-04-02T04:47:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 5: 0 ) 2024-04-02T04:47:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 10 client.14757.objecter _finish_command 5 = system:0 2024-04-02T04:47:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 4 2024-04-02T04:47:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.736+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 2 5 2024-04-02T04:47:59.745 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:47:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:47:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 10 client.14757.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 6 2024-04-02T04:47:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 6 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f158802daa0 osd=2 3 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 2 6 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 4 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 10 client.14757.objecter _send_command 6 2024-04-02T04:47:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f158802d520 con 0x7f15881ac1f0 2024-04-02T04:47:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f158802d520 con 0x7f15881ac1f0 2024-04-02T04:47:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 10 client.14757.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 4 2024-04-02T04:47:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 2 6 2024-04-02T04:47:59.758 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:47:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 10 client.14757.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 7 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f15881a9b90 0x7f15881afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 7 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 3 7 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 4 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 10 client.14757.objecter _send_command 7 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f15881b0350 con 0x7f15881a9b90 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f15881a9b90 0x7f15881afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f15881a9b90 2024-04-02T04:47:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f15881a9b90 0x7f15881afca0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881b0350 con 0x7f15881a9b90 2024-04-02T04:47:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 7: 0 ) 2024-04-02T04:47:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 10 client.14757.objecter _finish_command 7 = system:0 2024-04-02T04:47:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 4 2024-04-02T04:47:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.756+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 3 7 2024-04-02T04:47:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 10 client.14757.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 8 2024-04-02T04:47:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 8 2024-04-02T04:47:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f158802d920 osd=3 3 2024-04-02T04:47:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 3 8 2024-04-02T04:47:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 4 2024-04-02T04:47:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 10 client.14757.objecter _send_command 8 2024-04-02T04:47:59.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f15881a9100 con 0x7f15881a9b90 2024-04-02T04:47:59.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:47:59 smithi069 bash[21528]: cluster 2024-04-02T04:47:57.809287+0000 mgr.y (mgr.14538) 151 : cluster 0 pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:59.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:47:59 smithi069 bash[28904]: cluster 2024-04-02T04:47:57.809287+0000 mgr.y (mgr.14538) 151 : cluster 0 pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:59.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f15881a9100 con 0x7f15881a9b90 2024-04-02T04:47:59.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f15857fa700 10 client.14757.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 4 2024-04-02T04:47:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 3 8 2024-04-02T04:47:59.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:47:59.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 10 client.14757.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 9 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f15881b0d30 0x7f15881b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 9 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 4 9 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 4 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 10 client.14757.objecter _send_command 9 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f15881b8470 con 0x7f15881b0d30 2024-04-02T04:47:59.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.776+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f15881b0d30 0x7f15881b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f15881b0d30 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f15881b0d30 0x7f15881b5610 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881b8470 con 0x7f15881b0d30 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 9: 0 ) 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f15857fa700 10 client.14757.objecter _finish_command 9 = system:0 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 4 2024-04-02T04:47:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.780+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 4 9 2024-04-02T04:47:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 10 client.14757.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 10 2024-04-02T04:47:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 10 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881b0350 osd=4 3 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 4 10 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 4 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 10 client.14757.objecter _send_command 10 2024-04-02T04:47:59.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.796+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f15881a8f10 con 0x7f15881b0d30 2024-04-02T04:47:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f15881a8f10 con 0x7f15881b0d30 2024-04-02T04:47:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 10 client.14757.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 4 2024-04-02T04:47:59.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 4 10 2024-04-02T04:47:59.800 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 10 client.14757.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 11 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f15881b61d0 0x7f15881b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 11 2024-04-02T04:47:59.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 5 11 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 4 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 10 client.14757.objecter _send_command 11 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f15881bb680 con 0x7f15881b61d0 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f15881b61d0 0x7f15881b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15877fe700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f15881b61d0 0x7f15881b85c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f15881b61d0 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881bb680 con 0x7f15881b61d0 2024-04-02T04:47:59.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 11: 0 ) 2024-04-02T04:47:59.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 10 client.14757.objecter _finish_command 11 = system:0 2024-04-02T04:47:59.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 4 2024-04-02T04:47:59.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.800+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 5 11 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 10 client.14757.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 12 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 12 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881b5b50 osd=5 3 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 5 12 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 4 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 10 client.14757.objecter _send_command 12 2024-04-02T04:47:59.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f15881ba410 con 0x7f15881b61d0 2024-04-02T04:47:59.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f15881ba410 con 0x7f15881b61d0 2024-04-02T04:47:59.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f15857fa700 10 client.14757.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 4 2024-04-02T04:47:59.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 5 12 2024-04-02T04:47:59.822 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:47:59.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 10 client.14757.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 13 2024-04-02T04:47:59.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f15881bab30 0x7f15881baf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 13 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 6 13 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 4 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 10 client.14757.objecter _send_command 13 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f15881bea90 con 0x7f15881bab30 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.820+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f15881bab30 0x7f15881baf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f15881bab30 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f158caac700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f15881bab30 0x7f15881baf70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881bea90 con 0x7f15881bab30 2024-04-02T04:47:59.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 13: 0 ) 2024-04-02T04:47:59.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f15857fa700 10 client.14757.objecter _finish_command 13 = system:0 2024-04-02T04:47:59.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 4 2024-04-02T04:47:59.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.824+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 6 13 2024-04-02T04:47:59.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 10 client.14757.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 14 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 14 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881a8f70 osd=6 3 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 6 14 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 4 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 10 client.14757.objecter _send_command 14 2024-04-02T04:47:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.840+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f15881bd8d0 con 0x7f15881bab30 2024-04-02T04:47:59.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f15881bd8d0 con 0x7f15881bab30 2024-04-02T04:47:59.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f15857fa700 10 client.14757.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 4 2024-04-02T04:47:59.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 6 14 2024-04-02T04:47:59.846 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:47:59.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 10 client.14757.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:47:59.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 15 2024-04-02T04:47:59.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f15881bdff0 0x7f15881be410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:47:59.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 15 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 7 15 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 4 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 10 client.14757.objecter _send_command 15 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f15881c1e30 con 0x7f15881bdff0 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f15881bdff0 0x7f15881be410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f15857fa700 10 client.14757.objecter ms_handle_connect 0x7f15881bdff0 2024-04-02T04:47:59.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.844+0000 7f1587fff700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f15881bdff0 0x7f15881be410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:47:59.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:59.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.848+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15881c1e30 con 0x7f15881bdff0 2024-04-02T04:47:59.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.848+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 15: 0 ) 2024-04-02T04:47:59.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.848+0000 7f15857fa700 10 client.14757.objecter _finish_command 15 = system:0 2024-04-02T04:47:59.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.848+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 4 2024-04-02T04:47:59.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.848+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 7 15 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 10 client.14757.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign -1 16 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 20 client.14757.objecter _get_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 15 client.14757.objecter _session_command_op_remove -1 16 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 20 client.14757.objecter get_session s=0x7f15881bd8d0 osd=7 3 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 15 client.14757.objecter _session_command_op_assign 7 16 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 4 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 10 client.14757.objecter _send_command 16 2024-04-02T04:47:59.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.868+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f15881b8c40 con 0x7f15881bdff0 2024-04-02T04:47:59.869 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:47:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:47:59.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f15857fa700 1 -- 172.21.15.69:0/3748401737 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f15881b8c40 con 0x7f15881bdff0 2024-04-02T04:47:59.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f15857fa700 10 client.14757.objecter ms_dispatch 0x7f1588001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:47:59.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f15857fa700 10 client.14757.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:47:59.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f15857fa700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 4 2024-04-02T04:47:59.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f15857fa700 15 client.14757.objecter _session_command_op_remove 7 16 2024-04-02T04:47:59.872 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 librados: watch_flush enter 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 librados: watch_flush exit 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.0 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f1588062e50 msgr2=0x7f1588063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f1588062e50 0x7f1588063230 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f1588104300 osd=0 1 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.1 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f158802c9b0 msgr2=0x7f158802cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f158802c9b0 0x7f158802cd90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158810a6f0 osd=1 1 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.2 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f15881ac1f0 msgr2=0x7f15881ae5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f15881ac1f0 0x7f15881ae5e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802daa0 osd=2 1 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.3 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f15881a9b90 msgr2=0x7f15881afca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f15881a9b90 0x7f15881afca0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f158802d920 osd=3 1 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.4 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f15881b0d30 msgr2=0x7f15881b5610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f15881b0d30 0x7f15881b5610 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b0350 osd=4 1 2024-04-02T04:47:59.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.5 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f15881b61d0 msgr2=0x7f15881b85c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f15881b61d0 0x7f15881b85c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881b5b50 osd=5 1 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.6 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f15881bab30 msgr2=0x7f15881baf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f15881bab30 0x7f15881baf70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881a8f70 osd=6 1 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter close_session for osd.7 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f15881bdff0 msgr2=0x7f15881be410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f15881bdff0 0x7f15881be410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter put_session s=0x7f15881bd8d0 osd=7 1 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 20 client.14757.objecter shutdown clearing up homeless session... 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 10 client.14757.objecter successfully canceled tick 2024-04-02T04:47:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158caac700 1 -- 172.21.15.69:0/3748401737 reap_dead start 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158caac700 1 -- 172.21.15.69:0/3748401737 reap_dead start 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f157006f5e0 msgr2=0x7f1570071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f157006f5e0 0x7f1570071aa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f157c009750 tx=0x7f157c009310 comp rx=0 tx=0).stop 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 msgr2=0x7f1588102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 --2- 172.21.15.69:0/3748401737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881011a0 0x7f1588102e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1578005950 tx=0x7f157800fb40 comp rx=0 tx=0).stop 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158caac700 1 -- 172.21.15.69:0/3748401737 reap_dead start 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158caac700 1 -- 172.21.15.69:0/3748401737 reap_dead start 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158caac700 1 -- 172.21.15.69:0/3748401737 reap_dead start 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 shutdown_connections 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 >> 172.21.15.69:0/3748401737 conn(0x7f15880771e0 msgr2=0x7f1588069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 shutdown_connections 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 -- 172.21.15.69:0/3748401737 wait complete. 2024-04-02T04:47:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:47:59.872+0000 7f158e50f700 1 librados: shutdown 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:47:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:47:59.971 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:59.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:47:59 smithi098 bash[23790]: cluster 2024-04-02T04:47:57.809287+0000 mgr.y (mgr.14538) 151 : cluster 0 pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:47:59.992 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:47:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:47:59.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:00.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:00.120 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:00.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:00.249 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:00.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:00.380 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/1444135841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c107230 msgr2=0x7f361c107610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- 172.21.15.69:0/1444135841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c107230 0x7f361c107610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3610009990 tx=0x7f3610009ca0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/1444135841 shutdown_connections 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- 172.21.15.69:0/1444135841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f361c101b40 0x7f361c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- 172.21.15.69:0/1444135841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c107230 0x7f361c107610 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- 172.21.15.69:0/1444135841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101180 0x7f361c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/1444135841 >> 172.21.15.69:0/1444135841 conn(0x7f361c078190 msgr2=0x7f361c0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:00.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/1444135841 shutdown_connections 2024-04-02T04:48:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/1444135841 wait complete. 2024-04-02T04:48:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 librados: starting msgr at 2024-04-02T04:48:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 librados: starting objecter 2024-04-02T04:48:00.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 Processor -- start 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- start start 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 librados: setting wanted keys 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 librados: calling monclient init 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 0x7f361c102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101b40 0x7f361c103370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f361c107230 0x7f361c1038b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f361c110980 con 0x7f361c101180 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f361c110800 con 0x7f361c101b40 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f361c110b00 con 0x7f361c107230 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 0x7f361c102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f362164c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f361c107230 0x7f361c1038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:00.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 0x7f361c102e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39692/0 (socket says 172.21.15.69:39692) 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 -- 172.21.15.69:0/3504161550 learned_addr learned my addr 172.21.15.69:0/3504161550 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 -- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f361c107230 msgr2=0x7f361c1038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f361c107230 0x7f361c1038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f361bfff700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101b40 0x7f361c103370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 -- 172.21.15.69:0/3504161550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101b40 msgr2=0x7f361c103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101b40 0x7f361c103370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 -- 172.21.15.69:0/3504161550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c1a6620 con 0x7f361c101180 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f361bfff700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101b40 0x7f361c103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3620e4b700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 0x7f361c102e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f36100057f0 tx=0x7f361000bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3619ffb700 10 client.?.objecter ms_handle_connect 0x7f361c101180 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3619ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610026070 con 0x7f361c101180 2024-04-02T04:48:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f361000fbc0 con 0x7f361c101180 2024-04-02T04:48:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361c1a68b0 con 0x7f361c101180 2024-04-02T04:48:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610017650 con 0x7f361c101180 2024-04-02T04:48:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 10 client.14763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361c1a6d90 con 0x7f361c101180 2024-04-02T04:48:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.392+0000 7f36230af700 1 librados: init done 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361c04ff00 con 0x7f361c101180 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3610029070 con 0x7f361c101180 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f360c06f5e0 0x7f360c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(121..121 src has 64..121) ==== 7537+0+0 (secure 0 0 0) 0x7f361008f360 con 0x7f361c101180 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 10 client.14763.objecter ms_dispatch 0x7f361c001550 osd_map(121..121 src has 64..121) 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 3 client.14763.objecter handle_osd_map got epochs [121,121] > 0 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 3 client.14763.objecter handle_osd_map decoding full epoch 121 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f361bfff700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f360c06f5e0 0x7f360c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 20 client.14763.objecter dump_active .. 0 homeless 2024-04-02T04:48:00.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f3619ffb700 10 client.14763.objecter ms_handle_connect 0x7f360c06f5e0 2024-04-02T04:48:00.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.396+0000 7f361bfff700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f360c06f5e0 0x7f360c071aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f361c078a20 tx=0x7f3608008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:00.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.400+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3610030080 con 0x7f361c101180 2024-04-02T04:48:00.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:00.505 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:00.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.596+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f361c100580 con 0x7f360c06f5e0 2024-04-02T04:48:00.608 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.608+0000 7f3619ffb700 1 -- 172.21.15.69:0/3504161550 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346935 (secure 0 0 0) 0x7f361c100580 con 0x7f360c06f5e0 2024-04-02T04:48:00.613 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:48:00.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 10 librados: watch_flush enter 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 10 librados: watch_flush exit 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 20 client.14763.objecter shutdown clearing up homeless session... 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 10 client.14763.objecter successfully canceled tick 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f360c06f5e0 msgr2=0x7f360c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f360c06f5e0 0x7f360c071aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f361c078a20 tx=0x7f3608008040 comp rx=0 tx=0).stop 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 msgr2=0x7f361c102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 0x7f361c102e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f36100057f0 tx=0x7f361000bfa0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 shutdown_connections 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f360c06f5e0 0x7f360c071aa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f361c107230 0x7f361c1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f361c101180 0x7f361c102e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 --2- 172.21.15.69:0/3504161550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c101b40 0x7f361c103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 >> 172.21.15.69:0/3504161550 conn(0x7f361c078190 msgr2=0x7f361c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:00.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 shutdown_connections 2024-04-02T04:48:00.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 -- 172.21.15.69:0/3504161550 wait complete. 2024-04-02T04:48:00.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:00.612+0000 7f36230af700 1 librados: shutdown 2024-04-02T04:48:00.618 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:48:00.630 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:00.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:00.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:00 smithi069 bash[21528]: audit 2024-04-02T04:47:59.699177+0000 mon.a (mon.0) 924 : audit 0 from='client.? 172.21.15.69:0/3748401737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:00.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:00 smithi069 bash[28904]: audit 2024-04-02T04:47:59.699177+0000 mon.a (mon.0) 924 : audit 0 from='client.? 172.21.15.69:0/3748401737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:00.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:00.888 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:00.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:00.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:00 smithi098 bash[23790]: audit 2024-04-02T04:47:59.699177+0000 mon.a (mon.0) 924 : audit 0 from='client.? 172.21.15.69:0/3748401737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:00.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:01.123 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:01.148 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:01.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:01.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:01.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:01.538 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:01.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:01.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:01.781 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:47:59.809663+0000 mgr.y (mgr.14538) 152 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:48:01.781 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:00.299613+0000 osd.7 (osd.7) 103 : cluster 0 3.13 scrub starts 2024-04-02T04:48:01.781 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:00.302294+0000 osd.7 (osd.7) 104 : cluster 0 3.13 scrub ok 2024-04-02T04:48:01.781 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: audit 2024-04-02T04:48:00.601296+0000 mgr.y (mgr.14538) 153 : audit 0 from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:01.781 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:00.684991+0000 osd.4 (osd.4) 107 : cluster 0 3.14 scrub starts 2024-04-02T04:48:01.781 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:00.691364+0000 osd.4 (osd.4) 108 : cluster 0 3.14 scrub ok 2024-04-02T04:48:01.782 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:00.877010+0000 osd.6 (osd.6) 71 : cluster 0 5.2 scrub starts 2024-04-02T04:48:01.782 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:00.879710+0000 osd.6 (osd.6) 72 : cluster 0 5.2 scrub ok 2024-04-02T04:48:01.782 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:01.025882+0000 osd.1 (osd.1) 103 : cluster 0 6.4 scrub starts 2024-04-02T04:48:01.782 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:01.027846+0000 osd.1 (osd.1) 104 : cluster 0 6.4 scrub ok 2024-04-02T04:48:01.782 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:01.252361+0000 osd.5 (osd.5) 95 : cluster 0 3.16 scrub starts 2024-04-02T04:48:01.782 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:01.256887+0000 osd.5 (osd.5) 96 : cluster 0 3.16 scrub ok 2024-04-02T04:48:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:01.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:47:59.809663+0000 mgr.y (mgr.14538) 152 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:00.299613+0000 osd.7 (osd.7) 103 : cluster 0 3.13 scrub starts 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:00.302294+0000 osd.7 (osd.7) 104 : cluster 0 3.13 scrub ok 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: audit 2024-04-02T04:48:00.601296+0000 mgr.y (mgr.14538) 153 : audit 0 from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:00.684991+0000 osd.4 (osd.4) 107 : cluster 0 3.14 scrub starts 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:00.691364+0000 osd.4 (osd.4) 108 : cluster 0 3.14 scrub ok 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:00.877010+0000 osd.6 (osd.6) 71 : cluster 0 5.2 scrub starts 2024-04-02T04:48:01.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:00.879710+0000 osd.6 (osd.6) 72 : cluster 0 5.2 scrub ok 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:01.025882+0000 osd.1 (osd.1) 103 : cluster 0 6.4 scrub starts 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:01.027846+0000 osd.1 (osd.1) 104 : cluster 0 6.4 scrub ok 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:01.252361+0000 osd.5 (osd.5) 95 : cluster 0 3.16 scrub starts 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:01.256887+0000 osd.5 (osd.5) 96 : cluster 0 3.16 scrub ok 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:47:59.809663+0000 mgr.y (mgr.14538) 152 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:00.299613+0000 osd.7 (osd.7) 103 : cluster 0 3.13 scrub starts 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:00.302294+0000 osd.7 (osd.7) 104 : cluster 0 3.13 scrub ok 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: audit 2024-04-02T04:48:00.601296+0000 mgr.y (mgr.14538) 153 : audit 0 from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:00.684991+0000 osd.4 (osd.4) 107 : cluster 0 3.14 scrub starts 2024-04-02T04:48:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:00.691364+0000 osd.4 (osd.4) 108 : cluster 0 3.14 scrub ok 2024-04-02T04:48:01.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:00.877010+0000 osd.6 (osd.6) 71 : cluster 0 5.2 scrub starts 2024-04-02T04:48:01.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:00.879710+0000 osd.6 (osd.6) 72 : cluster 0 5.2 scrub ok 2024-04-02T04:48:01.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:01.025882+0000 osd.1 (osd.1) 103 : cluster 0 6.4 scrub starts 2024-04-02T04:48:01.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:01.027846+0000 osd.1 (osd.1) 104 : cluster 0 6.4 scrub ok 2024-04-02T04:48:01.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:01.252361+0000 osd.5 (osd.5) 95 : cluster 0 3.16 scrub starts 2024-04-02T04:48:01.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:01.256887+0000 osd.5 (osd.5) 96 : cluster 0 3.16 scrub ok 2024-04-02T04:48:01.915 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-02T04:47:59.809474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1459,"num_read_kb":1273,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240668,"kb_used_data":7900,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720548,"statfs":{"total":767960285184,"available":767713841152,"internally_reserved":0,"allocated":8089600,"data_stored":5093826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005529"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.506819+0000","last_change":"2024-04-02T04:47:14.337462+0000","last_active":"2024-04-02T04:47:35.506819+0000","last_peered":"2024-04-02T04:47:35.506819+0000","last_clean":"2024-04-02T04:47:35.506819+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:47:35.506819+0000","last_undegraded":"2024-04-02T04:47:35.506819+0000","last_fullsized":"2024-04-02T04:47:35.506819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.337178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.950297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:52.450357+0000","last_change":"2024-04-02T04:47:52.450415+0000","last_active":"2024-04-02T04:47:52.450357+0000","last_peered":"2024-04-02T04:47:52.450357+0000","last_clean":"2024-04-02T04:47:52.450357+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:47:52.450357+0000","last_undegraded":"2024-04-02T04:47:52.450357+0000","last_fullsized":"2024-04-02T04:47:52.450357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:52.450181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:52.450181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:59.502057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:54.214799+0000","last_change":"2024-04-02T04:47:54.214859+0000","last_active":"2024-04-02T04:47:54.214799+0000","last_peered":"2024-04-02T04:47:54.214799+0000","last_clean":"2024-04-02T04:47:54.214799+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:47:54.214799+0000","last_undegraded":"2024-04-02T04:47:54.214799+0000","last_fullsized":"2024-04-02T04:47:54.214799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.434714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":138,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.509857+0000","last_change":"2024-04-02T04:46:39.861495+0000","last_active":"2024-04-02T04:47:35.509857+0000","last_peered":"2024-04-02T04:47:35.509857+0000","last_clean":"2024-04-02T04:47:35.509857+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:47:35.509857+0000","last_undegraded":"2024-04-02T04:47:35.509857+0000","last_fullsized":"2024-04-02T04:47:35.509857+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:39.861258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.861258+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501083+0000","last_change":"2024-04-02T04:47:11.925670+0000","last_active":"2024-04-02T04:47:35.501083+0000","last_peered":"2024-04-02T04:47:35.501083+0000","last_clean":"2024-04-02T04:47:35.501083+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:47:35.501083+0000","last_undegraded":"2024-04-02T04:47:35.501083+0000","last_fullsized":"2024-04-02T04:47:35.501083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:11.925452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:11.925452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.710115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.500860+0000","last_change":"2024-04-02T04:47:07.889053+0000","last_active":"2024-04-02T04:47:35.500860+0000","last_peered":"2024-04-02T04:47:35.500860+0000","last_clean":"2024-04-02T04:47:35.500860+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:47:35.500860+0000","last_undegraded":"2024-04-02T04:47:35.500860+0000","last_fullsized":"2024-04-02T04:47:35.500860+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:07.888835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:07.888835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:10.239127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":123,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:42.551341+0000","last_change":"2024-04-02T04:47:42.551341+0000","last_active":"2024-04-02T04:47:42.551341+0000","last_peered":"2024-04-02T04:47:42.551341+0000","last_clean":"2024-04-02T04:47:42.551341+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:47:42.551341+0000","last_undegraded":"2024-04-02T04:47:42.551341+0000","last_fullsized":"2024-04-02T04:47:42.551341+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.551243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.551243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:59.906138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":141,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.767306+0000","last_change":"2024-04-02T04:46:45.396999+0000","last_active":"2024-04-02T04:47:35.767306+0000","last_peered":"2024-04-02T04:47:35.767306+0000","last_clean":"2024-04-02T04:47:35.767306+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:47:35.767306+0000","last_undegraded":"2024-04-02T04:47:35.767306+0000","last_fullsized":"2024-04-02T04:47:35.767306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:45.396760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.396760+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.943901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:42.186145+0000","last_change":"2024-04-02T04:47:42.186212+0000","last_active":"2024-04-02T04:47:42.186145+0000","last_peered":"2024-04-02T04:47:42.186145+0000","last_clean":"2024-04-02T04:47:42.186145+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:47:42.186145+0000","last_undegraded":"2024-04-02T04:47:42.186145+0000","last_fullsized":"2024-04-02T04:47:42.186145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.185988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.185988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.015773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:37.876794+0000","last_change":"2024-04-02T04:47:37.876855+0000","last_active":"2024-04-02T04:47:37.876794+0000","last_peered":"2024-04-02T04:47:37.876794+0000","last_clean":"2024-04-02T04:47:37.876794+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:47:37.876794+0000","last_undegraded":"2024-04-02T04:47:37.876794+0000","last_fullsized":"2024-04-02T04:47:37.876794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.876644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.876644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.719969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:45.254606+0000","last_change":"2024-04-02T04:47:45.254673+0000","last_active":"2024-04-02T04:47:45.254606+0000","last_peered":"2024-04-02T04:47:45.254606+0000","last_clean":"2024-04-02T04:47:45.254606+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:47:45.254606+0000","last_undegraded":"2024-04-02T04:47:45.254606+0000","last_fullsized":"2024-04-02T04:47:45.254606+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.254450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.254450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:58.071917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":133,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.283338+0000","last_change":"2024-04-02T04:47:05.285775+0000","last_active":"2024-04-02T04:47:36.283338+0000","last_peered":"2024-04-02T04:47:36.283338+0000","last_clean":"2024-04-02T04:47:36.283338+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:47:36.283338+0000","last_undegraded":"2024-04-02T04:47:36.283338+0000","last_fullsized":"2024-04-02T04:47:36.283338+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:05.285521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.285521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:28.240436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.765383+0000","last_change":"2024-04-02T04:46:27.234204+0000","last_active":"2024-04-02T04:47:35.765383+0000","last_peered":"2024-04-02T04:47:35.765383+0000","last_clean":"2024-04-02T04:47:35.765383+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:47:35.765383+0000","last_undegraded":"2024-04-02T04:47:35.765383+0000","last_fullsized":"2024-04-02T04:47:35.765383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:27.234006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.234006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:48.363415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501910+0000","last_change":"2024-04-02T04:46:44.775289+0000","last_active":"2024-04-02T04:47:35.501910+0000","last_peered":"2024-04-02T04:47:35.501910+0000","last_clean":"2024-04-02T04:47:35.501910+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:47:35.501910+0000","last_undegraded":"2024-04-02T04:47:35.501910+0000","last_fullsized":"2024-04-02T04:47:35.501910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.775071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.207519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098601+0000","last_change":"2024-04-02T04:47:09.387603+0000","last_active":"2024-04-02T04:47:36.098601+0000","last_peered":"2024-04-02T04:47:36.098601+0000","last_clean":"2024-04-02T04:47:36.098601+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:47:36.098601+0000","last_undegraded":"2024-04-02T04:47:36.098601+0000","last_fullsized":"2024-04-02T04:47:36.098601+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:09.387375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.387375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.279958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":143,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.283571+0000","last_change":"2024-04-02T04:46:51.350464+0000","last_active":"2024-04-02T04:47:36.283571+0000","last_peered":"2024-04-02T04:47:36.283571+0000","last_clean":"2024-04-02T04:47:36.283571+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:47:36.283571+0000","last_undegraded":"2024-04-02T04:47:36.283571+0000","last_fullsized":"2024-04-02T04:47:36.283571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:46:51.350201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:51.350201+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.520062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.510473+0000","last_change":"2024-04-02T04:47:03.158759+0000","last_active":"2024-04-02T04:47:35.510473+0000","last_peered":"2024-04-02T04:47:35.510473+0000","last_clean":"2024-04-02T04:47:35.510473+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:47:35.510473+0000","last_undegraded":"2024-04-02T04:47:35.510473+0000","last_fullsized":"2024-04-02T04:47:35.510473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:03.158541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:03.158541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.522158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:57.459149+0000","last_change":"2024-04-02T04:47:57.459178+0000","last_active":"2024-04-02T04:47:57.459149+0000","last_peered":"2024-04-02T04:47:57.459149+0000","last_clean":"2024-04-02T04:47:57.459149+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:47:57.459149+0000","last_undegraded":"2024-04-02T04:47:57.459149+0000","last_fullsized":"2024-04-02T04:47:57.459149+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:57.459040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.459040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:17.124034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507219+0000","last_change":"2024-04-02T04:47:18.359335+0000","last_active":"2024-04-02T04:47:35.507219+0000","last_peered":"2024-04-02T04:47:35.507219+0000","last_clean":"2024-04-02T04:47:35.507219+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:47:35.507219+0000","last_undegraded":"2024-04-02T04:47:35.507219+0000","last_fullsized":"2024-04-02T04:47:35.507219+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:18.359101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:18.359101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:38.654960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":165,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:40.838775+0000","last_change":"2024-04-02T04:47:40.838867+0000","last_active":"2024-04-02T04:47:40.838775+0000","last_peered":"2024-04-02T04:47:40.838775+0000","last_clean":"2024-04-02T04:47:40.838775+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:47:40.838775+0000","last_undegraded":"2024-04-02T04:47:40.838775+0000","last_fullsized":"2024-04-02T04:47:40.838775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.838617+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.066994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.503133+0000","last_change":"2024-04-02T04:47:01.936520+0000","last_active":"2024-04-02T04:47:35.503133+0000","last_peered":"2024-04-02T04:47:35.503133+0000","last_clean":"2024-04-02T04:47:35.503133+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:47:35.503133+0000","last_undegraded":"2024-04-02T04:47:35.503133+0000","last_fullsized":"2024-04-02T04:47:35.503133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:01.936275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:01.936275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.709309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:45.830555+0000","last_change":"2024-04-02T04:47:45.830668+0000","last_active":"2024-04-02T04:47:45.830555+0000","last_peered":"2024-04-02T04:47:45.830555+0000","last_clean":"2024-04-02T04:47:45.830555+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:47:45.830555+0000","last_undegraded":"2024-04-02T04:47:45.830555+0000","last_fullsized":"2024-04-02T04:47:45.830555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.830369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.830369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.027610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:53.938642+0000","last_change":"2024-04-02T04:47:53.938679+0000","last_active":"2024-04-02T04:47:53.938642+0000","last_peered":"2024-04-02T04:47:53.938642+0000","last_clean":"2024-04-02T04:47:53.938642+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:47:53.938642+0000","last_undegraded":"2024-04-02T04:47:53.938642+0000","last_fullsized":"2024-04-02T04:47:53.938642+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.938537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.938537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:12.383253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":135,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.766714+0000","last_change":"2024-04-02T04:47:13.347780+0000","last_active":"2024-04-02T04:47:35.766714+0000","last_peered":"2024-04-02T04:47:35.766714+0000","last_clean":"2024-04-02T04:47:35.766714+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:47:35.766714+0000","last_undegraded":"2024-04-02T04:47:35.766714+0000","last_fullsized":"2024-04-02T04:47:35.766714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.347670+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.686984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507660+0000","last_change":"2024-04-02T04:46:56.263880+0000","last_active":"2024-04-02T04:47:35.507660+0000","last_peered":"2024-04-02T04:47:35.507660+0000","last_clean":"2024-04-02T04:47:35.507660+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:47:35.507660+0000","last_undegraded":"2024-04-02T04:47:35.507660+0000","last_fullsized":"2024-04-02T04:47:35.507660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:56.263740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:56.263740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.910160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":168,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.489234+0000","last_change":"2024-04-02T04:46:47.154319+0000","last_active":"2024-04-02T04:47:49.489234+0000","last_peered":"2024-04-02T04:47:49.489234+0000","last_clean":"2024-04-02T04:47:49.489234+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:47:49.489234+0000","last_undegraded":"2024-04-02T04:47:49.489234+0000","last_fullsized":"2024-04-02T04:47:49.489234+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:47.154082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.154082+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.920192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.505695+0000","last_change":"2024-04-02T04:46:27.777917+0000","last_active":"2024-04-02T04:47:35.505695+0000","last_peered":"2024-04-02T04:47:35.505695+0000","last_clean":"2024-04-02T04:47:35.505695+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:47:35.505695+0000","last_undegraded":"2024-04-02T04:47:35.505695+0000","last_fullsized":"2024-04-02T04:47:35.505695+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:46:27.777679+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:46:27.777679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:50.754491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":146,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:45.516493+0000","last_change":"2024-04-02T04:47:45.516535+0000","last_active":"2024-04-02T04:47:45.516493+0000","last_peered":"2024-04-02T04:47:45.516493+0000","last_clean":"2024-04-02T04:47:45.516493+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:47:45.516493+0000","last_undegraded":"2024-04-02T04:47:45.516493+0000","last_fullsized":"2024-04-02T04:47:45.516493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:45.516407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.516407+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.285568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":103,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.099111+0000","last_change":"2024-04-02T04:46:42.290326+0000","last_active":"2024-04-02T04:47:36.099111+0000","last_peered":"2024-04-02T04:47:36.099111+0000","last_clean":"2024-04-02T04:47:36.099111+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:47:36.099111+0000","last_undegraded":"2024-04-02T04:47:36.099111+0000","last_fullsized":"2024-04-02T04:47:36.099111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:46:42.290071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.290071+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.053573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:39.172062+0000","last_change":"2024-04-02T04:47:39.172102+0000","last_active":"2024-04-02T04:47:39.172062+0000","last_peered":"2024-04-02T04:47:39.172062+0000","last_clean":"2024-04-02T04:47:39.172062+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:47:39.172062+0000","last_undegraded":"2024-04-02T04:47:39.172062+0000","last_fullsized":"2024-04-02T04:47:39.172062+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:39.171955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.171955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:54.119072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":120,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.282255+0000","last_change":"2024-04-02T04:47:27.380112+0000","last_active":"2024-04-02T04:47:36.282255+0000","last_peered":"2024-04-02T04:47:36.282255+0000","last_clean":"2024-04-02T04:47:36.282255+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:47:36.282255+0000","last_undegraded":"2024-04-02T04:47:36.282255+0000","last_fullsized":"2024-04-02T04:47:36.282255+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:27.379800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:27.379800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.831508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":182,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.512714+0000","last_change":"2024-04-02T04:46:59.265375+0000","last_active":"2024-04-02T04:47:35.512714+0000","last_peered":"2024-04-02T04:47:35.512714+0000","last_clean":"2024-04-02T04:47:35.512714+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:47:35.512714+0000","last_undegraded":"2024-04-02T04:47:35.512714+0000","last_fullsized":"2024-04-02T04:47:35.512714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:46:59.265104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:59.265104+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.709345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507534+0000","last_change":"2024-04-02T04:47:13.288506+0000","last_active":"2024-04-02T04:47:35.507534+0000","last_peered":"2024-04-02T04:47:35.507534+0000","last_clean":"2024-04-02T04:47:35.507534+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:47:35.507534+0000","last_undegraded":"2024-04-02T04:47:35.507534+0000","last_fullsized":"2024-04-02T04:47:35.507534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:13.288364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.288364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.927428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":106,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.505431+0000","last_change":"2024-04-02T04:47:12.271950+0000","last_active":"2024-04-02T04:47:35.505431+0000","last_peered":"2024-04-02T04:47:35.505431+0000","last_clean":"2024-04-02T04:47:35.505431+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:47:35.505431+0000","last_undegraded":"2024-04-02T04:47:35.505431+0000","last_fullsized":"2024-04-02T04:47:35.505431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.271680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.271680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.801349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.840538+0000","last_change":"2024-04-02T04:47:23.324160+0000","last_active":"2024-04-02T04:47:35.840538+0000","last_peered":"2024-04-02T04:47:35.840538+0000","last_clean":"2024-04-02T04:47:35.840538+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:47:35.840538+0000","last_undegraded":"2024-04-02T04:47:35.840538+0000","last_fullsized":"2024-04-02T04:47:35.840538+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:23.323919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.323919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.280221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":148,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.283623+0000","last_change":"2024-04-02T04:46:41.326411+0000","last_active":"2024-04-02T04:47:36.283623+0000","last_peered":"2024-04-02T04:47:36.283623+0000","last_clean":"2024-04-02T04:47:36.283623+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:47:36.283623+0000","last_undegraded":"2024-04-02T04:47:36.283623+0000","last_fullsized":"2024-04-02T04:47:36.283623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:46:41.326158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.326158+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.505788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.097554+0000","last_change":"2024-04-02T04:47:20.427195+0000","last_active":"2024-04-02T04:47:36.097554+0000","last_peered":"2024-04-02T04:47:36.097554+0000","last_clean":"2024-04-02T04:47:36.097554+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:47:36.097554+0000","last_undegraded":"2024-04-02T04:47:36.097554+0000","last_fullsized":"2024-04-02T04:47:36.097554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:20.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:47:20.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:48.706473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.504710+0000","last_change":"2024-04-02T04:47:19.398500+0000","last_active":"2024-04-02T04:47:35.504710+0000","last_peered":"2024-04-02T04:47:35.504710+0000","last_clean":"2024-04-02T04:47:35.504710+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:47:35.504710+0000","last_undegraded":"2024-04-02T04:47:35.504710+0000","last_fullsized":"2024-04-02T04:47:35.504710+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:19.398328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:19.398328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.197933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":120,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:46.278053+0000","last_change":"2024-04-02T04:47:46.278114+0000","last_active":"2024-04-02T04:47:46.278053+0000","last_peered":"2024-04-02T04:47:46.278053+0000","last_clean":"2024-04-02T04:47:46.278053+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:47:46.278053+0000","last_undegraded":"2024-04-02T04:47:46.278053+0000","last_fullsized":"2024-04-02T04:47:46.278053+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:46.277889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:47:46.277889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.753634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.767249+0000","last_change":"2024-04-02T04:47:08.333929+0000","last_active":"2024-04-02T04:47:35.767249+0000","last_peered":"2024-04-02T04:47:35.767249+0000","last_clean":"2024-04-02T04:47:35.767249+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:47:35.767249+0000","last_undegraded":"2024-04-02T04:47:35.767249+0000","last_fullsized":"2024-04-02T04:47:35.767249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:47:08.333516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.333516+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.726461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.504958+0000","last_change":"2024-04-02T04:46:38.893014+0000","last_active":"2024-04-02T04:47:35.504958+0000","last_peered":"2024-04-02T04:47:35.504958+0000","last_clean":"2024-04-02T04:47:35.504958+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:47:35.504958+0000","last_undegraded":"2024-04-02T04:47:35.504958+0000","last_fullsized":"2024-04-02T04:47:35.504958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.892773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.892773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.901685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501974+0000","last_change":"2024-04-02T04:47:31.779975+0000","last_active":"2024-04-02T04:47:35.501974+0000","last_peered":"2024-04-02T04:47:35.501974+0000","last_clean":"2024-04-02T04:47:35.501974+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:47:35.501974+0000","last_undegraded":"2024-04-02T04:47:35.501974+0000","last_fullsized":"2024-04-02T04:47:35.501974+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:31.779694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:31.779694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:40.720115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.510851+0000","last_change":"2024-04-02T04:46:54.269554+0000","last_active":"2024-04-02T04:47:35.510851+0000","last_peered":"2024-04-02T04:47:35.510851+0000","last_clean":"2024-04-02T04:47:35.510851+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:47:35.510851+0000","last_undegraded":"2024-04-02T04:47:35.510851+0000","last_fullsized":"2024-04-02T04:47:35.510851+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:54.269343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:54.269343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.989031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":148,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501647+0000","last_change":"2024-04-02T04:46:37.808452+0000","last_active":"2024-04-02T04:47:35.501647+0000","last_peered":"2024-04-02T04:47:35.501647+0000","last_clean":"2024-04-02T04:47:35.501647+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:47:35.501647+0000","last_undegraded":"2024-04-02T04:47:35.501647+0000","last_fullsized":"2024-04-02T04:47:35.501647+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:46:37.808090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:37.808090+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.632664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.765395+0000","last_change":"2024-04-02T04:47:18.291662+0000","last_active":"2024-04-02T04:47:35.765395+0000","last_peered":"2024-04-02T04:47:35.765395+0000","last_clean":"2024-04-02T04:47:35.765395+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:47:35.765395+0000","last_undegraded":"2024-04-02T04:47:35.765395+0000","last_fullsized":"2024-04-02T04:47:35.765395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:18.291429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:18.291429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:39.162464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507361+0000","last_change":"2024-04-02T04:46:53.297425+0000","last_active":"2024-04-02T04:47:35.507361+0000","last_peered":"2024-04-02T04:47:35.507361+0000","last_clean":"2024-04-02T04:47:35.507361+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:47:35.507361+0000","last_undegraded":"2024-04-02T04:47:35.507361+0000","last_fullsized":"2024-04-02T04:47:35.507361+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:53.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:53.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.497586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.840002+0000","last_change":"2024-04-02T04:46:34.223691+0000","last_active":"2024-04-02T04:47:35.840002+0000","last_peered":"2024-04-02T04:47:35.840002+0000","last_clean":"2024-04-02T04:47:35.840002+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:47:35.840002+0000","last_undegraded":"2024-04-02T04:47:35.840002+0000","last_fullsized":"2024-04-02T04:47:35.840002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:34.223346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:34.223346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.085223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":145,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.370650+0000","last_change":"2024-04-02T04:47:36.370736+0000","last_active":"2024-04-02T04:47:36.370650+0000","last_peered":"2024-04-02T04:47:36.370650+0000","last_clean":"2024-04-02T04:47:36.370650+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:47:36.370650+0000","last_undegraded":"2024-04-02T04:47:36.370650+0000","last_fullsized":"2024-04-02T04:47:36.370650+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:36.370493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.370493+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.329887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.503076+0000","last_change":"2024-04-02T04:47:05.885138+0000","last_active":"2024-04-02T04:47:35.503076+0000","last_peered":"2024-04-02T04:47:35.503076+0000","last_clean":"2024-04-02T04:47:35.503076+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:47:35.503076+0000","last_undegraded":"2024-04-02T04:47:35.503076+0000","last_fullsized":"2024-04-02T04:47:35.503076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:30.330301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":166,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.488807+0000","last_change":"2024-04-02T04:46:44.346692+0000","last_active":"2024-04-02T04:47:49.488807+0000","last_peered":"2024-04-02T04:47:49.488807+0000","last_clean":"2024-04-02T04:47:49.488807+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:47:49.488807+0000","last_undegraded":"2024-04-02T04:47:49.488807+0000","last_fullsized":"2024-04-02T04:47:49.488807+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:44.346318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.092878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.766386+0000","last_change":"2024-04-02T04:46:36.244725+0000","last_active":"2024-04-02T04:47:35.766386+0000","last_peered":"2024-04-02T04:47:35.766386+0000","last_clean":"2024-04-02T04:47:35.766386+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:47:35.766386+0000","last_undegraded":"2024-04-02T04:47:35.766386+0000","last_fullsized":"2024-04-02T04:47:35.766386+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.244514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.244514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.017937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":134,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:53.179394+0000","last_change":"2024-04-02T04:47:53.179435+0000","last_active":"2024-04-02T04:47:53.179394+0000","last_peered":"2024-04-02T04:47:53.179394+0000","last_clean":"2024-04-02T04:47:53.179394+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:47:53.179394+0000","last_undegraded":"2024-04-02T04:47:53.179394+0000","last_fullsized":"2024-04-02T04:47:53.179394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:47:53.179316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.179316+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.558109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.766890+0000","last_change":"2024-04-02T04:47:09.295716+0000","last_active":"2024-04-02T04:47:35.766890+0000","last_peered":"2024-04-02T04:47:35.766890+0000","last_clean":"2024-04-02T04:47:35.766890+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:47:35.766890+0000","last_undegraded":"2024-04-02T04:47:35.766890+0000","last_fullsized":"2024-04-02T04:47:35.766890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:09.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.295498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.587769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"85'11","reported_seq":165,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.493019+0000","last_change":"2024-04-02T04:46:28.775353+0000","last_active":"2024-04-02T04:47:49.493019+0000","last_peered":"2024-04-02T04:47:49.493019+0000","last_clean":"2024-04-02T04:47:49.493019+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:47:49.493019+0000","last_undegraded":"2024-04-02T04:47:49.493019+0000","last_fullsized":"2024-04-02T04:47:49.493019+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:28.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:28.775071+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:55.875528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.282263+0000","last_change":"2024-04-02T04:46:48.355080+0000","last_active":"2024-04-02T04:47:36.282263+0000","last_peered":"2024-04-02T04:47:36.282263+0000","last_clean":"2024-04-02T04:47:36.282263+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:47:36.282263+0000","last_undegraded":"2024-04-02T04:47:36.282263+0000","last_fullsized":"2024-04-02T04:47:36.282263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:48.354869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:48.354869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.638786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":128,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098567+0000","last_change":"2024-04-02T04:46:33.347283+0000","last_active":"2024-04-02T04:47:36.098567+0000","last_peered":"2024-04-02T04:47:36.098567+0000","last_clean":"2024-04-02T04:47:36.098567+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:47:36.098567+0000","last_undegraded":"2024-04-02T04:47:36.098567+0000","last_fullsized":"2024-04-02T04:47:36.098567+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:33.347053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.347053+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:57.542332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098955+0000","last_change":"2024-04-02T04:47:04.415949+0000","last_active":"2024-04-02T04:47:36.098955+0000","last_peered":"2024-04-02T04:47:36.098955+0000","last_clean":"2024-04-02T04:47:36.098955+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:47:36.098955+0000","last_undegraded":"2024-04-02T04:47:36.098955+0000","last_fullsized":"2024-04-02T04:47:36.098955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:04.415735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.415735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:34.243533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.510754+0000","last_change":"2024-04-02T04:46:45.921390+0000","last_active":"2024-04-02T04:47:35.510754+0000","last_peered":"2024-04-02T04:47:35.510754+0000","last_clean":"2024-04-02T04:47:35.510754+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:47:35.510754+0000","last_undegraded":"2024-04-02T04:47:35.510754+0000","last_fullsized":"2024-04-02T04:47:35.510754+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.369948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.839206+0000","last_change":"2024-04-02T04:47:35.160389+0000","last_active":"2024-04-02T04:47:35.839206+0000","last_peered":"2024-04-02T04:47:35.839206+0000","last_clean":"2024-04-02T04:47:35.839206+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:47:35.839206+0000","last_undegraded":"2024-04-02T04:47:35.839206+0000","last_fullsized":"2024-04-02T04:47:35.839206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.160172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.160172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.333854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.839648+0000","last_change":"2024-04-02T04:46:41.254878+0000","last_active":"2024-04-02T04:47:35.839648+0000","last_peered":"2024-04-02T04:47:35.839648+0000","last_clean":"2024-04-02T04:47:35.839648+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:47:35.839648+0000","last_undegraded":"2024-04-02T04:47:35.839648+0000","last_fullsized":"2024-04-02T04:47:35.839648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:46:41.254633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.254633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.695806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.238922+0000","last_change":"2024-04-02T04:46:38.622583+0000","last_active":"2024-04-02T04:47:36.238922+0000","last_peered":"2024-04-02T04:47:36.238922+0000","last_clean":"2024-04-02T04:47:36.238922+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:47:36.238922+0000","last_undegraded":"2024-04-02T04:47:36.238922+0000","last_fullsized":"2024-04-02T04:47:36.238922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.622220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.622220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.379107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":106,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.766791+0000","last_change":"2024-04-02T04:46:57.429067+0000","last_active":"2024-04-02T04:47:35.766791+0000","last_peered":"2024-04-02T04:47:35.766791+0000","last_clean":"2024-04-02T04:47:35.766791+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:47:35.766791+0000","last_undegraded":"2024-04-02T04:47:35.766791+0000","last_fullsized":"2024-04-02T04:47:35.766791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.428901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.428901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.996741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":120,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.839156+0000","last_change":"2024-04-02T04:47:49.839212+0000","last_active":"2024-04-02T04:47:49.839156+0000","last_peered":"2024-04-02T04:47:49.839156+0000","last_clean":"2024-04-02T04:47:49.839156+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:47:49.839156+0000","last_undegraded":"2024-04-02T04:47:49.839156+0000","last_fullsized":"2024-04-02T04:47:49.839156+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.839000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:09.914868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":140,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.097791+0000","last_change":"2024-04-02T04:47:00.341837+0000","last_active":"2024-04-02T04:47:36.097791+0000","last_peered":"2024-04-02T04:47:36.097791+0000","last_clean":"2024-04-02T04:47:36.097791+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:47:36.097791+0000","last_undegraded":"2024-04-02T04:47:36.097791+0000","last_fullsized":"2024-04-02T04:47:36.097791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:00.341473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.341473+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:24.505569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507524+0000","last_change":"2024-04-02T04:47:00.098587+0000","last_active":"2024-04-02T04:47:35.507524+0000","last_peered":"2024-04-02T04:47:35.507524+0000","last_clean":"2024-04-02T04:47:35.507524+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:47:35.507524+0000","last_undegraded":"2024-04-02T04:47:35.507524+0000","last_fullsized":"2024-04-02T04:47:35.507524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.098270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.098270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.905960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.502711+0000","last_change":"2024-04-02T04:47:14.896155+0000","last_active":"2024-04-02T04:47:35.502711+0000","last_peered":"2024-04-02T04:47:35.502711+0000","last_clean":"2024-04-02T04:47:35.502711+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:47:35.502711+0000","last_undegraded":"2024-04-02T04:47:35.502711+0000","last_fullsized":"2024-04-02T04:47:35.502711+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:47:14.895813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.895813+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.683265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":148,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098816+0000","last_change":"2024-04-02T04:47:34.393560+0000","last_active":"2024-04-02T04:47:36.098816+0000","last_peered":"2024-04-02T04:47:36.098816+0000","last_clean":"2024-04-02T04:47:36.098816+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:47:36.098816+0000","last_undegraded":"2024-04-02T04:47:36.098816+0000","last_fullsized":"2024-04-02T04:47:36.098816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:34.393321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:34.393321+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.086104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":173,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.488398+0000","last_change":"2024-04-02T04:47:42.831672+0000","last_active":"2024-04-02T04:47:49.488398+0000","last_peered":"2024-04-02T04:47:49.488398+0000","last_clean":"2024-04-02T04:47:49.488398+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:47:49.488398+0000","last_undegraded":"2024-04-02T04:47:49.488398+0000","last_fullsized":"2024-04-02T04:47:49.488398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.831447+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:46.338365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:47.122292+0000","last_change":"2024-04-02T04:47:47.122324+0000","last_active":"2024-04-02T04:47:47.122292+0000","last_peered":"2024-04-02T04:47:47.122292+0000","last_clean":"2024-04-02T04:47:47.122292+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:47:47.122292+0000","last_undegraded":"2024-04-02T04:47:47.122292+0000","last_fullsized":"2024-04-02T04:47:47.122292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.122214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.122214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.799063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":129,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:43.151583+0000","last_change":"2024-04-02T04:47:43.151655+0000","last_active":"2024-04-02T04:47:43.151583+0000","last_peered":"2024-04-02T04:47:43.151583+0000","last_clean":"2024-04-02T04:47:43.151583+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:47:43.151583+0000","last_undegraded":"2024-04-02T04:47:43.151583+0000","last_fullsized":"2024-04-02T04:47:43.151583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:43.151427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:43.151427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.872479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":187,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.488868+0000","last_change":"2024-04-02T04:47:39.221972+0000","last_active":"2024-04-02T04:47:49.488868+0000","last_peered":"2024-04-02T04:47:49.488868+0000","last_clean":"2024-04-02T04:47:49.488868+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:47:49.488868+0000","last_undegraded":"2024-04-02T04:47:49.488868+0000","last_fullsized":"2024-04-02T04:47:49.488868+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:47:39.221741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.221741+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:05.865282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:45.109328+0000","last_change":"2024-04-02T04:47:45.109386+0000","last_active":"2024-04-02T04:47:45.109328+0000","last_peered":"2024-04-02T04:47:45.109328+0000","last_clean":"2024-04-02T04:47:45.109328+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:47:45.109328+0000","last_undegraded":"2024-04-02T04:47:45.109328+0000","last_fullsized":"2024-04-02T04:47:45.109328+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.109170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.109170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.076716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.509233+0000","last_change":"2024-04-02T04:47:02.153712+0000","last_active":"2024-04-02T04:47:35.509233+0000","last_peered":"2024-04-02T04:47:35.509233+0000","last_clean":"2024-04-02T04:47:35.509233+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:47:35.509233+0000","last_undegraded":"2024-04-02T04:47:35.509233+0000","last_fullsized":"2024-04-02T04:47:35.509233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:47:02.153481+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.153481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.645312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.282430+0000","last_change":"2024-04-02T04:46:57.319477+0000","last_active":"2024-04-02T04:47:36.282430+0000","last_peered":"2024-04-02T04:47:36.282430+0000","last_clean":"2024-04-02T04:47:36.282430+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:47:36.282430+0000","last_undegraded":"2024-04-02T04:47:36.282430+0000","last_fullsized":"2024-04-02T04:47:36.282430+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.319266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.319266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.104997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":165,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:50.758508+0000","last_change":"2024-04-02T04:47:50.758701+0000","last_active":"2024-04-02T04:47:50.758508+0000","last_peered":"2024-04-02T04:47:50.758508+0000","last_clean":"2024-04-02T04:47:50.758508+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:47:50.758508+0000","last_undegraded":"2024-04-02T04:47:50.758508+0000","last_fullsized":"2024-04-02T04:47:50.758508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:47:50.758312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:50.758312+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.203712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:47.106580+0000","last_change":"2024-04-02T04:47:47.106602+0000","last_active":"2024-04-02T04:47:47.106580+0000","last_peered":"2024-04-02T04:47:47.106580+0000","last_clean":"2024-04-02T04:47:47.106580+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:47:47.106580+0000","last_undegraded":"2024-04-02T04:47:47.106580+0000","last_fullsized":"2024-04-02T04:47:47.106580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.106497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.032879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":109,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:53.270929+0000","last_change":"2024-04-02T04:47:53.271021+0000","last_active":"2024-04-02T04:47:53.270929+0000","last_peered":"2024-04-02T04:47:53.270929+0000","last_clean":"2024-04-02T04:47:53.270929+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:47:53.270929+0000","last_undegraded":"2024-04-02T04:47:53.270929+0000","last_fullsized":"2024-04-02T04:47:53.270929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.902292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":165,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.505745+0000","last_change":"2024-04-02T04:46:55.308858+0000","last_active":"2024-04-02T04:47:35.505745+0000","last_peered":"2024-04-02T04:47:35.505745+0000","last_clean":"2024-04-02T04:47:35.505745+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:47:35.505745+0000","last_undegraded":"2024-04-02T04:47:35.505745+0000","last_fullsized":"2024-04-02T04:47:35.505745+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:46:55.308528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.308528+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.148543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":120,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.238175+0000","last_change":"2024-04-02T04:47:14.881150+0000","last_active":"2024-04-02T04:47:36.238175+0000","last_peered":"2024-04-02T04:47:36.238175+0000","last_clean":"2024-04-02T04:47:36.238175+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:47:36.238175+0000","last_undegraded":"2024-04-02T04:47:36.238175+0000","last_fullsized":"2024-04-02T04:47:36.238175+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.881031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.881031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:23.207112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.506362+0000","last_change":"2024-04-02T04:46:42.343240+0000","last_active":"2024-04-02T04:47:35.506362+0000","last_peered":"2024-04-02T04:47:35.506362+0000","last_clean":"2024-04-02T04:47:35.506362+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:47:35.506362+0000","last_undegraded":"2024-04-02T04:47:35.506362+0000","last_fullsized":"2024-04-02T04:47:35.506362+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:42.343078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.343078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.356250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.284146+0000","last_change":"2024-04-02T04:46:35.276714+0000","last_active":"2024-04-02T04:47:36.284146+0000","last_peered":"2024-04-02T04:47:36.284146+0000","last_clean":"2024-04-02T04:47:36.284146+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:47:36.284146+0000","last_undegraded":"2024-04-02T04:47:36.284146+0000","last_fullsized":"2024-04-02T04:47:36.284146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:35.276502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:35.276502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.541614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507100+0000","last_change":"2024-04-02T04:46:52.985248+0000","last_active":"2024-04-02T04:47:35.507100+0000","last_peered":"2024-04-02T04:47:35.507100+0000","last_clean":"2024-04-02T04:47:35.507100+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:47:35.507100+0000","last_undegraded":"2024-04-02T04:47:35.507100+0000","last_fullsized":"2024-04-02T04:47:35.507100+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:52.984994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.984994+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.008710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.282503+0000","last_change":"2024-04-02T04:46:55.305525+0000","last_active":"2024-04-02T04:47:36.282503+0000","last_peered":"2024-04-02T04:47:36.282503+0000","last_clean":"2024-04-02T04:47:36.282503+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:47:36.282503+0000","last_undegraded":"2024-04-02T04:47:36.282503+0000","last_fullsized":"2024-04-02T04:47:36.282503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.305293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.165521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":379,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:44.214219+0000","last_change":"2024-04-02T04:47:44.214306+0000","last_active":"2024-04-02T04:47:44.214219+0000","last_peered":"2024-04-02T04:47:44.214219+0000","last_clean":"2024-04-02T04:47:44.214219+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:47:44.214219+0000","last_undegraded":"2024-04-02T04:47:44.214219+0000","last_fullsized":"2024-04-02T04:47:44.214219+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:47:44.214047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.214047+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.404997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.839873+0000","last_change":"2024-04-02T04:47:19.323314+0000","last_active":"2024-04-02T04:47:35.839873+0000","last_peered":"2024-04-02T04:47:35.839873+0000","last_clean":"2024-04-02T04:47:35.839873+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:47:35.839873+0000","last_undegraded":"2024-04-02T04:47:35.839873+0000","last_fullsized":"2024-04-02T04:47:35.839873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:19.323097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:19.323097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:20.126618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.509614+0000","last_change":"2024-04-02T04:46:40.318480+0000","last_active":"2024-04-02T04:47:35.509614+0000","last_peered":"2024-04-02T04:47:35.509614+0000","last_clean":"2024-04-02T04:47:35.509614+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:47:35.509614+0000","last_undegraded":"2024-04-02T04:47:35.509614+0000","last_fullsized":"2024-04-02T04:47:35.509614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:40.318304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:40.318304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.789228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":143,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.766006+0000","last_change":"2024-04-02T04:46:33.251890+0000","last_active":"2024-04-02T04:47:35.766006+0000","last_peered":"2024-04-02T04:47:35.766006+0000","last_clean":"2024-04-02T04:47:35.766006+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:47:35.766006+0000","last_undegraded":"2024-04-02T04:47:35.766006+0000","last_fullsized":"2024-04-02T04:47:35.766006+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:46:33.251772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.251772+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:00.560770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.097423+0000","last_change":"2024-04-02T04:47:05.388983+0000","last_active":"2024-04-02T04:47:36.097423+0000","last_peered":"2024-04-02T04:47:36.097423+0000","last_clean":"2024-04-02T04:47:36.097423+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:47:36.097423+0000","last_undegraded":"2024-04-02T04:47:36.097423+0000","last_fullsized":"2024-04-02T04:47:36.097423+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.388768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.388768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.419043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.767072+0000","last_change":"2024-04-02T04:46:50.400686+0000","last_active":"2024-04-02T04:47:35.767072+0000","last_peered":"2024-04-02T04:47:35.767072+0000","last_clean":"2024-04-02T04:47:35.767072+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:47:35.767072+0000","last_undegraded":"2024-04-02T04:47:35.767072+0000","last_fullsized":"2024-04-02T04:47:35.767072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:46:50.400390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.328220+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.506179+0000","last_change":"2024-04-02T04:46:46.307232+0000","last_active":"2024-04-02T04:47:35.506179+0000","last_peered":"2024-04-02T04:47:35.506179+0000","last_clean":"2024-04-02T04:47:35.506179+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:47:35.506179+0000","last_undegraded":"2024-04-02T04:47:35.506179+0000","last_fullsized":"2024-04-02T04:47:35.506179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.306862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.306862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:59.499577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.839356+0000","last_change":"2024-04-02T04:46:44.178572+0000","last_active":"2024-04-02T04:47:35.839356+0000","last_peered":"2024-04-02T04:47:35.839356+0000","last_clean":"2024-04-02T04:47:35.839356+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:47:35.839356+0000","last_undegraded":"2024-04-02T04:47:35.839356+0000","last_fullsized":"2024-04-02T04:47:35.839356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:46:44.178327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.178327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:56.305669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098611+0000","last_change":"2024-04-02T04:47:17.471458+0000","last_active":"2024-04-02T04:47:36.098611+0000","last_peered":"2024-04-02T04:47:36.098611+0000","last_clean":"2024-04-02T04:47:36.098611+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:47:36.098611+0000","last_undegraded":"2024-04-02T04:47:36.098611+0000","last_fullsized":"2024-04-02T04:47:36.098611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:17.471105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:17.471105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:23.064426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":239,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.493057+0000","last_change":"2024-04-02T04:47:42.173169+0000","last_active":"2024-04-02T04:47:49.493057+0000","last_peered":"2024-04-02T04:47:49.493057+0000","last_clean":"2024-04-02T04:47:49.493057+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:47:49.493057+0000","last_undegraded":"2024-04-02T04:47:49.493057+0000","last_fullsized":"2024-04-02T04:47:49.493057+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.172894+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.893986+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"121'12","reported_seq":174,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.492935+0000","last_change":"2024-04-02T04:47:40.137383+0000","last_active":"2024-04-02T04:47:49.492935+0000","last_peered":"2024-04-02T04:47:49.492935+0000","last_clean":"2024-04-02T04:47:49.492935+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:47:49.492935+0000","last_undegraded":"2024-04-02T04:47:49.492935+0000","last_fullsized":"2024-04-02T04:47:49.492935+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.137127+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:52.797250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501407+0000","last_change":"2024-04-02T04:47:00.939593+0000","last_active":"2024-04-02T04:47:35.501407+0000","last_peered":"2024-04-02T04:47:35.501407+0000","last_clean":"2024-04-02T04:47:35.501407+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:47:35.501407+0000","last_undegraded":"2024-04-02T04:47:35.501407+0000","last_fullsized":"2024-04-02T04:47:35.501407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.939352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.939352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.488733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.500840+0000","last_change":"2024-04-02T04:46:49.854837+0000","last_active":"2024-04-02T04:47:35.500840+0000","last_peered":"2024-04-02T04:47:35.500840+0000","last_clean":"2024-04-02T04:47:35.500840+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:47:35.500840+0000","last_undegraded":"2024-04-02T04:47:35.500840+0000","last_fullsized":"2024-04-02T04:47:35.500840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:49.854618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.854618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.738084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.508974+0000","last_change":"2024-04-02T04:46:39.351367+0000","last_active":"2024-04-02T04:47:35.508974+0000","last_peered":"2024-04-02T04:47:35.508974+0000","last_clean":"2024-04-02T04:47:35.508974+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:47:35.508974+0000","last_undegraded":"2024-04-02T04:47:35.508974+0000","last_fullsized":"2024-04-02T04:47:35.508974+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:46:39.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.351151+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.995435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":149,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.510017+0000","last_change":"2024-04-02T04:47:04.162419+0000","last_active":"2024-04-02T04:47:35.510017+0000","last_peered":"2024-04-02T04:47:35.510017+0000","last_clean":"2024-04-02T04:47:35.510017+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:47:35.510017+0000","last_undegraded":"2024-04-02T04:47:35.510017+0000","last_fullsized":"2024-04-02T04:47:35.510017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:47:04.162146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.162146+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.370811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":174,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.492982+0000","last_change":"2024-04-02T04:47:30.470228+0000","last_active":"2024-04-02T04:47:49.492982+0000","last_peered":"2024-04-02T04:47:49.492982+0000","last_clean":"2024-04-02T04:47:49.492982+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:47:49.492982+0000","last_undegraded":"2024-04-02T04:47:49.492982+0000","last_fullsized":"2024-04-02T04:47:49.492982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:47:30.469973+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:47:30.469973+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:37.580189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":109,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.224676+0000","last_change":"2024-04-02T04:47:49.224735+0000","last_active":"2024-04-02T04:47:49.224676+0000","last_peered":"2024-04-02T04:47:49.224676+0000","last_clean":"2024-04-02T04:47:49.224676+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:47:49.224676+0000","last_undegraded":"2024-04-02T04:47:49.224676+0000","last_fullsized":"2024-04-02T04:47:49.224676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.916792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":337,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:56.741999+0000","last_change":"2024-04-02T04:46:43.873401+0000","last_active":"2024-04-02T04:47:56.741999+0000","last_peered":"2024-04-02T04:47:56.741999+0000","last_clean":"2024-04-02T04:47:56.741999+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:47:56.741999+0000","last_undegraded":"2024-04-02T04:47:56.741999+0000","last_fullsized":"2024-04-02T04:47:56.741999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:46:43.873013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.873013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:08.482316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":221,"num_read_kb":216,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:41.043572+0000","last_change":"2024-04-02T04:47:41.043637+0000","last_active":"2024-04-02T04:47:41.043572+0000","last_peered":"2024-04-02T04:47:41.043572+0000","last_clean":"2024-04-02T04:47:41.043572+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:47:41.043572+0000","last_undegraded":"2024-04-02T04:47:41.043572+0000","last_fullsized":"2024-04-02T04:47:41.043572+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:41.043411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.043411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.404918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":128,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.767417+0000","last_change":"2024-04-02T04:46:49.354576+0000","last_active":"2024-04-02T04:47:35.767417+0000","last_peered":"2024-04-02T04:47:35.767417+0000","last_clean":"2024-04-02T04:47:35.767417+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:47:35.767417+0000","last_undegraded":"2024-04-02T04:47:35.767417+0000","last_fullsized":"2024-04-02T04:47:35.767417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:46:49.354293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.354293+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:51.930955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501383+0000","last_change":"2024-04-02T04:47:25.814952+0000","last_active":"2024-04-02T04:47:35.501383+0000","last_peered":"2024-04-02T04:47:35.501383+0000","last_clean":"2024-04-02T04:47:35.501383+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:47:35.501383+0000","last_undegraded":"2024-04-02T04:47:35.501383+0000","last_fullsized":"2024-04-02T04:47:35.501383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:25.814722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:25.814722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.783076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.507269+0000","last_change":"2024-04-02T04:46:46.969136+0000","last_active":"2024-04-02T04:47:35.507269+0000","last_peered":"2024-04-02T04:47:35.507269+0000","last_clean":"2024-04-02T04:47:35.507269+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:47:35.507269+0000","last_undegraded":"2024-04-02T04:47:35.507269+0000","last_fullsized":"2024-04-02T04:47:35.507269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.968921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.968921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.377963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":164,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:49.488446+0000","last_change":"2024-04-02T04:46:43.734784+0000","last_active":"2024-04-02T04:47:49.488446+0000","last_peered":"2024-04-02T04:47:49.488446+0000","last_clean":"2024-04-02T04:47:49.488446+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:47:49.488446+0000","last_undegraded":"2024-04-02T04:47:49.488446+0000","last_fullsized":"2024-04-02T04:47:49.488446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:43.734423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.734423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:07.446921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.097695+0000","last_change":"2024-04-02T04:47:28.453678+0000","last_active":"2024-04-02T04:47:36.097695+0000","last_peered":"2024-04-02T04:47:36.097695+0000","last_clean":"2024-04-02T04:47:36.097695+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:47:36.097695+0000","last_undegraded":"2024-04-02T04:47:36.097695+0000","last_fullsized":"2024-04-02T04:47:36.097695+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:28.453442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:28.453442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.301035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":156,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501127+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:47:35.501127+0000","last_peered":"2024-04-02T04:47:35.501127+0000","last_clean":"2024-04-02T04:47:35.501127+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:47:35.501127+0000","last_undegraded":"2024-04-02T04:47:35.501127+0000","last_fullsized":"2024-04-02T04:47:35.501127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.759911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:53.172953+0000","last_change":"2024-04-02T04:47:53.173015+0000","last_active":"2024-04-02T04:47:53.172953+0000","last_peered":"2024-04-02T04:47:53.172953+0000","last_clean":"2024-04-02T04:47:53.172953+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:47:53.172953+0000","last_undegraded":"2024-04-02T04:47:53.172953+0000","last_fullsized":"2024-04-02T04:47:53.172953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.172792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.172792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.046175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":53,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:55.855627+0000","last_change":"2024-04-02T04:47:55.855659+0000","last_active":"2024-04-02T04:47:55.855627+0000","last_peered":"2024-04-02T04:47:55.855627+0000","last_clean":"2024-04-02T04:47:55.855627+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:47:55.855627+0000","last_undegraded":"2024-04-02T04:47:55.855627+0000","last_fullsized":"2024-04-02T04:47:55.855627+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.855510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.855510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:18.631275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":127,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:44.112423+0000","last_change":"2024-04-02T04:47:44.112479+0000","last_active":"2024-04-02T04:47:44.112423+0000","last_peered":"2024-04-02T04:47:44.112423+0000","last_clean":"2024-04-02T04:47:44.112423+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:47:44.112423+0000","last_undegraded":"2024-04-02T04:47:44.112423+0000","last_fullsized":"2024-04-02T04:47:44.112423+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:44.112270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.112270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:58.773698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":176,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.505056+0000","last_change":"2024-04-02T04:47:26.453777+0000","last_active":"2024-04-02T04:47:35.505056+0000","last_peered":"2024-04-02T04:47:35.505056+0000","last_clean":"2024-04-02T04:47:35.505056+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:47:35.505056+0000","last_undegraded":"2024-04-02T04:47:35.505056+0000","last_fullsized":"2024-04-02T04:47:35.505056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:47:26.453340+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:47:26.453340+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.433920+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":101,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.503141+0000","last_change":"2024-04-02T04:47:08.862194+0000","last_active":"2024-04-02T04:47:35.503141+0000","last_peered":"2024-04-02T04:47:35.503141+0000","last_clean":"2024-04-02T04:47:35.503141+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:47:35.503141+0000","last_undegraded":"2024-04-02T04:47:35.503141+0000","last_fullsized":"2024-04-02T04:47:35.503141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:08.861978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.861978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.953269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.511034+0000","last_change":"2024-04-02T04:47:15.265577+0000","last_active":"2024-04-02T04:47:35.511034+0000","last_peered":"2024-04-02T04:47:35.511034+0000","last_clean":"2024-04-02T04:47:35.511034+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:47:35.511034+0000","last_undegraded":"2024-04-02T04:47:35.511034+0000","last_fullsized":"2024-04-02T04:47:35.511034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_clean_scrub_stamp":"2024-04-02T04:47:15.265490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:39.021422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":151,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:41.576801+0000","last_change":"2024-04-02T04:47:41.576889+0000","last_active":"2024-04-02T04:47:41.576801+0000","last_peered":"2024-04-02T04:47:41.576801+0000","last_clean":"2024-04-02T04:47:41.576801+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:47:41.576801+0000","last_undegraded":"2024-04-02T04:47:41.576801+0000","last_fullsized":"2024-04-02T04:47:41.576801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:47:41.576640+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.576640+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.199333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.501571+0000","last_change":"2024-04-02T04:46:55.945141+0000","last_active":"2024-04-02T04:47:35.501571+0000","last_peered":"2024-04-02T04:47:35.501571+0000","last_clean":"2024-04-02T04:47:35.501571+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:47:35.501571+0000","last_undegraded":"2024-04-02T04:47:35.501571+0000","last_fullsized":"2024-04-02T04:47:35.501571+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.944866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.944866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:58.910143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":108,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:37.864219+0000","last_change":"2024-04-02T04:47:37.864283+0000","last_active":"2024-04-02T04:47:37.864219+0000","last_peered":"2024-04-02T04:47:37.864219+0000","last_clean":"2024-04-02T04:47:37.864219+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:47:37.864219+0000","last_undegraded":"2024-04-02T04:47:37.864219+0000","last_fullsized":"2024-04-02T04:47:37.864219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.864039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.864039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.035813+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":173,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:56.704527+0000","last_change":"2024-04-02T04:47:56.704599+0000","last_active":"2024-04-02T04:47:56.704527+0000","last_peered":"2024-04-02T04:47:56.704527+0000","last_clean":"2024-04-02T04:47:56.704527+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:47:56.704527+0000","last_undegraded":"2024-04-02T04:47:56.704527+0000","last_fullsized":"2024-04-02T04:47:56.704527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:47:56.704379+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.874702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":113,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098246+0000","last_change":"2024-04-02T04:47:02.316549+0000","last_active":"2024-04-02T04:47:36.098246+0000","last_peered":"2024-04-02T04:47:36.098246+0000","last_clean":"2024-04-02T04:47:36.098246+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:47:36.098246+0000","last_undegraded":"2024-04-02T04:47:36.098246+0000","last_fullsized":"2024-04-02T04:47:36.098246+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:02.316325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.316325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.699710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":154,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.839671+0000","last_change":"2024-04-02T04:46:26.071505+0000","last_active":"2024-04-02T04:47:35.839671+0000","last_peered":"2024-04-02T04:47:35.839671+0000","last_clean":"2024-04-02T04:47:35.839671+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:47:35.839671+0000","last_undegraded":"2024-04-02T04:47:35.839671+0000","last_fullsized":"2024-04-02T04:47:35.839671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:46:26.071249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.071249+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:47:52.710130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":109,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.097208+0000","last_change":"2024-04-02T04:47:31.439024+0000","last_active":"2024-04-02T04:47:36.097208+0000","last_peered":"2024-04-02T04:47:36.097208+0000","last_clean":"2024-04-02T04:47:36.097208+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:47:36.097208+0000","last_undegraded":"2024-04-02T04:47:36.097208+0000","last_fullsized":"2024-04-02T04:47:36.097208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:31.438817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:31.438817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:45.747704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.238519+0000","last_change":"2024-04-02T04:46:41.673523+0000","last_active":"2024-04-02T04:47:36.238519+0000","last_peered":"2024-04-02T04:47:36.238519+0000","last_clean":"2024-04-02T04:47:36.238519+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:47:36.238519+0000","last_undegraded":"2024-04-02T04:47:36.238519+0000","last_fullsized":"2024-04-02T04:47:36.238519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:41.673448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.673448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.552075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.099189+0000","last_change":"2024-04-02T04:47:23.464318+0000","last_active":"2024-04-02T04:47:36.099189+0000","last_peered":"2024-04-02T04:47:36.099189+0000","last_clean":"2024-04-02T04:47:36.099189+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:47:36.099189+0000","last_undegraded":"2024-04-02T04:47:36.099189+0000","last_fullsized":"2024-04-02T04:47:36.099189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:23.464017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.464017+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.507017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:55.940165+0000","last_change":"2024-04-02T04:47:55.940230+0000","last_active":"2024-04-02T04:47:55.940165+0000","last_peered":"2024-04-02T04:47:55.940165+0000","last_clean":"2024-04-02T04:47:55.940165+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:47:55.940165+0000","last_undegraded":"2024-04-02T04:47:55.940165+0000","last_fullsized":"2024-04-02T04:47:55.940165+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.939983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.939983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.949757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.242967+0000","last_change":"2024-04-02T04:47:36.243027+0000","last_active":"2024-04-02T04:47:36.242967+0000","last_peered":"2024-04-02T04:47:36.242967+0000","last_clean":"2024-04-02T04:47:36.242967+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:47:36.242967+0000","last_undegraded":"2024-04-02T04:47:36.242967+0000","last_fullsized":"2024-04-02T04:47:36.242967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.633289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.511852+0000","last_change":"2024-04-02T04:47:23.176576+0000","last_active":"2024-04-02T04:47:35.511852+0000","last_peered":"2024-04-02T04:47:35.511852+0000","last_clean":"2024-04-02T04:47:35.511852+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:47:35.511852+0000","last_undegraded":"2024-04-02T04:47:35.511852+0000","last_fullsized":"2024-04-02T04:47:35.511852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:23.176366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.176366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:30.576582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":136,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:44.215508+0000","last_change":"2024-04-02T04:47:44.215606+0000","last_active":"2024-04-02T04:47:44.215508+0000","last_peered":"2024-04-02T04:47:44.215508+0000","last_clean":"2024-04-02T04:47:44.215508+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:47:44.215508+0000","last_undegraded":"2024-04-02T04:47:44.215508+0000","last_fullsized":"2024-04-02T04:47:44.215508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:47:44.215294+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.215294+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.905308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:35.767099+0000","last_change":"2024-04-02T04:47:35.260903+0000","last_active":"2024-04-02T04:47:35.767099+0000","last_peered":"2024-04-02T04:47:35.767099+0000","last_clean":"2024-04-02T04:47:35.767099+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:47:35.767099+0000","last_undegraded":"2024-04-02T04:47:35.767099+0000","last_fullsized":"2024-04-02T04:47:35.767099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.260653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.260653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.706903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.282922+0000","last_change":"2024-04-02T04:47:12.345553+0000","last_active":"2024-04-02T04:47:36.282922+0000","last_peered":"2024-04-02T04:47:36.282922+0000","last_clean":"2024-04-02T04:47:36.282922+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:47:36.282922+0000","last_undegraded":"2024-04-02T04:47:36.282922+0000","last_fullsized":"2024-04-02T04:47:36.282922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.345237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.345237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.300398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.239221+0000","last_change":"2024-04-02T04:46:33.616502+0000","last_active":"2024-04-02T04:47:36.239221+0000","last_peered":"2024-04-02T04:47:36.239221+0000","last_clean":"2024-04-02T04:47:36.239221+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:47:36.239221+0000","last_undegraded":"2024-04-02T04:47:36.239221+0000","last_fullsized":"2024-04-02T04:47:36.239221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:33.616300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.616300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:01.778531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":134,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:36.098590+0000","last_change":"2024-04-02T04:46:52.343834+0000","last_active":"2024-04-02T04:47:36.098590+0000","last_peered":"2024-04-02T04:47:36.098590+0000","last_clean":"2024-04-02T04:47:36.098590+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:47:36.098590+0000","last_undegraded":"2024-04-02T04:47:36.098590+0000","last_fullsized":"2024-04-02T04:47:36.098590+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:52.343572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.343572+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.554866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-02T04:47:48.245254+0000","last_change":"2024-04-02T04:47:48.245279+0000","last_active":"2024-04-02T04:47:48.245254+0000","last_peered":"2024-04-02T04:47:48.245254+0000","last_clean":"2024-04-02T04:47:48.245254+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:47:48.245254+0000","last_undegraded":"2024-04-02T04:47:48.245254+0000","last_fullsized":"2024-04-02T04:47:48.245254+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:48.245180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:48.245180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.877121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":629,"ondisk_log_size":629,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743483,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1400832,"data_stored":1006875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103238,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1368064,"data_stored":1004688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":790528,"data_stored":415400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32528,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712624,"statfs":{"total":95995035648,"available":95961726976,"internally_reserved":0,"allocated":1384448,"data_stored":1005352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378735,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":778240,"data_stored":415636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051685,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":414701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822747,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":415629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462991,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":778240,"data_stored":415545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:48:01.920 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-04-02T04:48:01.924 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:01.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/2134798476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab01022f0 msgr2=0x7f4ab0102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/2134798476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab01022f0 0x7f4ab0102750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40098f0 tx=0x7f4aa4009c00 comp rx=0 tx=0).stop 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/2134798476 shutdown_connections 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/2134798476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ab0102c90 0x7f4ab010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/2134798476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab0108300 0x7f4ab01086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/2134798476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab01022f0 0x7f4ab0102750 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/2134798476 >> 172.21.15.69:0/2134798476 conn(0x7f4ab0079310 msgr2=0x7f4ab0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:02.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/2134798476 shutdown_connections 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/2134798476 wait complete. 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 librados: starting msgr at 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 librados: starting objecter 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 Processor -- start 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- start start 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 librados: setting wanted keys 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 librados: calling monclient init 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 0x7f4ab01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ab0102c90 0x7f4ab01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0108300 0x7f4ab019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0111a90 con 0x7f4ab0108300 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0111910 con 0x7f4ab0102c90 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab0111c10 con 0x7f4ab01022f0 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 0x7f4ab01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:02.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 0x7f4ab01980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48512/0 (socket says 172.21.15.69:48512) 2024-04-02T04:48:02.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 -- 172.21.15.69:0/4225791188 learned_addr learned my addr 172.21.15.69:0/4225791188 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:02.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab4fa4700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0108300 0x7f4ab019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:02.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaf7fe700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ab0102c90 0x7f4ab01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:02.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 -- 172.21.15.69:0/4225791188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ab0102c90 msgr2=0x7f4ab01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:02.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ab0102c90 0x7f4ab01985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 -- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0108300 msgr2=0x7f4ab019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:02.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0108300 0x7f4ab019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 -- 172.21.15.69:0/4225791188 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab019d0b0 con 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab4fa4700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0108300 0x7f4ab019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aaffff700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 0x7f4ab01980a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000ec10 tx=0x7f4aa000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aad7fa700 10 client.?.objecter ms_handle_connect 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aad7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa000cd40 con 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab019d3a0 con 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4aa00044a0 con 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 10 client.24656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab01a4c20 con 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4ab6a07700 1 librados: init done 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.012+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0010430 con 0x7f4ab01022f0 2024-04-02T04:48:02.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab004ff00 con 0x7f4ab01022f0 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4aa0004000 con 0x7f4ab01022f0 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a9806f5e0 0x7f4a98071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(121..121 src has 64..121) ==== 7537+0+0 (secure 0 0 0) 0x7f4aa008e3d0 con 0x7f4ab01022f0 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 10 client.24656.objecter ms_dispatch 0x7f4ab0001550 osd_map(121..121 src has 64..121) 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 3 client.24656.objecter handle_osd_map got epochs [121,121] > 0 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 3 client.24656.objecter handle_osd_map decoding full epoch 121 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aaf7fe700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a9806f5e0 0x7f4a98071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 20 client.24656.objecter dump_active .. 0 homeless 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aaf7fe700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a9806f5e0 0x7f4a98071aa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009230 tx=0x7f4aa4005b40 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:02.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.016+0000 7f4aad7fa700 10 client.24656.objecter ms_handle_connect 0x7f4a9806f5e0 2024-04-02T04:48:02.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.020+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4aa0016330 con 0x7f4ab01022f0 2024-04-02T04:48:02.027 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:02.047 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:02.066 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:48:02.066 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:48:02.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:02.168 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:02.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.252+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f4ab01a4f10 con 0x7f4ab01022f0 2024-04-02T04:48:02.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:02.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:02.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:02.553 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:02.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:02.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.672+0000 7f4aad7fa700 1 -- 172.21.15.69:0/4225791188 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v122) ==== 87+0+0 (secure 0 0 0) 0x7f4aa0055e20 con 0x7f4ab01022f0 2024-04-02T04:48:02.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 10 librados: watch_flush enter 2024-04-02T04:48:02.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 10 librados: watch_flush exit 2024-04-02T04:48:02.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 20 client.24656.objecter shutdown clearing up homeless session... 2024-04-02T04:48:02.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 10 client.24656.objecter successfully canceled tick 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a9806f5e0 msgr2=0x7f4a98071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a9806f5e0 0x7f4a98071aa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009230 tx=0x7f4aa4005b40 comp rx=0 tx=0).stop 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 msgr2=0x7f4ab01980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 0x7f4ab01980a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000ec10 tx=0x7f4aa000ef20 comp rx=0 tx=0).stop 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 shutdown_connections 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a9806f5e0 0x7f4a98071aa0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0108300 0x7f4ab019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ab01022f0 0x7f4ab01980a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 --2- 172.21.15.69:0/4225791188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ab0102c90 0x7f4ab01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:02.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 >> 172.21.15.69:0/4225791188 conn(0x7f4ab0079310 msgr2=0x7f4ab0105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:02.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 shutdown_connections 2024-04-02T04:48:02.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 -- 172.21.15.69:0/4225791188 wait complete. 2024-04-02T04:48:02.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:02.680+0000 7f4ab6a07700 1 librados: shutdown 2024-04-02T04:48:02.682 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:48:02.686 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:02.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:02.813 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:02.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-02T04:48:02.909 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[23790]: cluster 2024-04-02T04:48:02.192263+0000 osd.0 (osd.0) 103 : cluster 0 6.18 scrub starts 2024-04-02T04:48:02.909 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[23790]: cluster 2024-04-02T04:48:02.193625+0000 osd.0 (osd.0) 104 : cluster 0 6.18 scrub ok 2024-04-02T04:48:02.909 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[23790]: audit 2024-04-02T04:48:02.259118+0000 mon.c (mon.2) 28 : audit 1 from='client.? 172.21.15.69:0/4225791188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:48:02.910 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[23790]: audit 2024-04-02T04:48:02.259540+0000 mon.a (mon.0) 925 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:48:02.910 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[23790]: cluster 2024-04-02T04:48:02.443423+0000 osd.3 (osd.3) 131 : cluster 0 4.14 deep-scrub starts 2024-04-02T04:48:02.910 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[23790]: cluster 2024-04-02T04:48:02.448111+0000 osd.3 (osd.3) 132 : cluster 0 4.14 deep-scrub ok 2024-04-02T04:48:02.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:02.927 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:48:02 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:48:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:02.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[21528]: cluster 2024-04-02T04:48:02.192263+0000 osd.0 (osd.0) 103 : cluster 0 6.18 scrub starts 2024-04-02T04:48:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[21528]: cluster 2024-04-02T04:48:02.193625+0000 osd.0 (osd.0) 104 : cluster 0 6.18 scrub ok 2024-04-02T04:48:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[21528]: audit 2024-04-02T04:48:02.259118+0000 mon.c (mon.2) 28 : audit 1 from='client.? 172.21.15.69:0/4225791188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:48:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[21528]: audit 2024-04-02T04:48:02.259540+0000 mon.a (mon.0) 925 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:48:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[21528]: cluster 2024-04-02T04:48:02.443423+0000 osd.3 (osd.3) 131 : cluster 0 4.14 deep-scrub starts 2024-04-02T04:48:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[21528]: cluster 2024-04-02T04:48:02.448111+0000 osd.3 (osd.3) 132 : cluster 0 4.14 deep-scrub ok 2024-04-02T04:48:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[28904]: cluster 2024-04-02T04:48:02.192263+0000 osd.0 (osd.0) 103 : cluster 0 6.18 scrub starts 2024-04-02T04:48:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[28904]: cluster 2024-04-02T04:48:02.193625+0000 osd.0 (osd.0) 104 : cluster 0 6.18 scrub ok 2024-04-02T04:48:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[28904]: audit 2024-04-02T04:48:02.259118+0000 mon.c (mon.2) 28 : audit 1 from='client.? 172.21.15.69:0/4225791188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:48:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[28904]: audit 2024-04-02T04:48:02.259540+0000 mon.a (mon.0) 925 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:48:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[28904]: cluster 2024-04-02T04:48:02.443423+0000 osd.3 (osd.3) 131 : cluster 0 4.14 deep-scrub starts 2024-04-02T04:48:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:02 smithi069 bash[28904]: cluster 2024-04-02T04:48:02.448111+0000 osd.3 (osd.3) 132 : cluster 0 4.14 deep-scrub ok 2024-04-02T04:48:03.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:03.073 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:03.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.200 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:03.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:03.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:48:03.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.457 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:03.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:03.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:03.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[28904]: cluster 2024-04-02T04:48:01.810302+0000 mgr.y (mgr.14538) 154 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:48:03.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[28904]: cluster 2024-04-02T04:48:02.662769+0000 mon.a (mon.0) 926 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:03.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[28904]: audit 2024-04-02T04:48:02.672722+0000 mon.a (mon.0) 927 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:48:03.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[28904]: cluster 2024-04-02T04:48:02.680787+0000 mon.a (mon.0) 928 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-04-02T04:48:03.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[21528]: cluster 2024-04-02T04:48:01.810302+0000 mgr.y (mgr.14538) 154 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:48:03.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[21528]: cluster 2024-04-02T04:48:02.662769+0000 mon.a (mon.0) 926 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:03.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[21528]: audit 2024-04-02T04:48:02.672722+0000 mon.a (mon.0) 927 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:48:03.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:03 smithi069 bash[21528]: cluster 2024-04-02T04:48:02.680787+0000 mon.a (mon.0) 928 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-04-02T04:48:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:03.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:03.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:03.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:03.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:03 smithi098 bash[23790]: cluster 2024-04-02T04:48:01.810302+0000 mgr.y (mgr.14538) 154 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:48:03.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:03 smithi098 bash[23790]: cluster 2024-04-02T04:48:02.662769+0000 mon.a (mon.0) 926 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:03.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:03 smithi098 bash[23790]: audit 2024-04-02T04:48:02.672722+0000 mon.a (mon.0) 927 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:48:03.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:03 smithi098 bash[23790]: cluster 2024-04-02T04:48:02.680787+0000 mon.a (mon.0) 928 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-04-02T04:48:04.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:04.097 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:04.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:04.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:04.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:04.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:04.487 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:04.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:04.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:04.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:04 smithi098 bash[23790]: audit 2024-04-02T04:48:02.912424+0000 mgr.y (mgr.14538) 155 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:04 smithi098 bash[23790]: cluster 2024-04-02T04:48:03.811766+0000 mgr.y (mgr.14538) 156 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:04 smithi098 bash[23790]: cluster 2024-04-02T04:48:03.998993+0000 osd.1 (osd.1) 105 : cluster 0 6.1 deep-scrub starts 2024-04-02T04:48:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:04 smithi098 bash[23790]: cluster 2024-04-02T04:48:04.001081+0000 osd.1 (osd.1) 106 : cluster 0 6.1 deep-scrub ok 2024-04-02T04:48:04.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:04.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:04.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:04.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[21528]: audit 2024-04-02T04:48:02.912424+0000 mgr.y (mgr.14538) 155 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[21528]: cluster 2024-04-02T04:48:03.811766+0000 mgr.y (mgr.14538) 156 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[21528]: cluster 2024-04-02T04:48:03.998993+0000 osd.1 (osd.1) 105 : cluster 0 6.1 deep-scrub starts 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[21528]: cluster 2024-04-02T04:48:04.001081+0000 osd.1 (osd.1) 106 : cluster 0 6.1 deep-scrub ok 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[28904]: audit 2024-04-02T04:48:02.912424+0000 mgr.y (mgr.14538) 155 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[28904]: cluster 2024-04-02T04:48:03.811766+0000 mgr.y (mgr.14538) 156 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[28904]: cluster 2024-04-02T04:48:03.998993+0000 osd.1 (osd.1) 105 : cluster 0 6.1 deep-scrub starts 2024-04-02T04:48:04.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:04 smithi069 bash[28904]: cluster 2024-04-02T04:48:04.001081+0000 osd.1 (osd.1) 106 : cluster 0 6.1 deep-scrub ok 2024-04-02T04:48:05.008 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:05.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:05.133 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:05.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:05.264 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:05.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:05.394 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:05.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:48:05.497 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:05.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:05.673 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:05.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:05.803 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:05.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:05.933 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:05.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:06.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:06.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:06.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:06.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:06.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:06.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:06.449 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:06.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:06.578 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:06.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:06.708 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:06.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:06.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:06.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:48:06.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:48:06.921 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-02T04:48:06.921 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-02T04:48:06.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:48:06.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:06 smithi098 bash[23790]: cluster 2024-04-02T04:48:05.812443+0000 mgr.y (mgr.14538) 157 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:06.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:07.026 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:07.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:07.157 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:07.260 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:07.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:06 smithi069 bash[28904]: cluster 2024-04-02T04:48:05.812443+0000 mgr.y (mgr.14538) 157 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:07.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:06 smithi069 bash[21528]: cluster 2024-04-02T04:48:05.812443+0000 mgr.y (mgr.14538) 157 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:07.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:07.410 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:07.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:07.533 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:07.636 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:07.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:07.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:07.788 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:07.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:07.914 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:07.915 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[28904]: audit 2024-04-02T04:48:07.106446+0000 mon.b (mon.1) 98 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:07.915 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[28904]: audit 2024-04-02T04:48:07.178052+0000 mon.b (mon.1) 99 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[28904]: audit 2024-04-02T04:48:07.180434+0000 mon.a (mon.0) 929 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[28904]: audit 2024-04-02T04:48:07.732977+0000 mon.b (mon.1) 100 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[28904]: audit 2024-04-02T04:48:07.734695+0000 mon.b (mon.1) 101 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[28904]: audit 2024-04-02T04:48:07.747766+0000 mon.a (mon.0) 930 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[21528]: audit 2024-04-02T04:48:07.106446+0000 mon.b (mon.1) 98 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[21528]: audit 2024-04-02T04:48:07.178052+0000 mon.b (mon.1) 99 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[21528]: audit 2024-04-02T04:48:07.180434+0000 mon.a (mon.0) 929 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[21528]: audit 2024-04-02T04:48:07.732977+0000 mon.b (mon.1) 100 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[21528]: audit 2024-04-02T04:48:07.734695+0000 mon.b (mon.1) 101 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:07.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:07 smithi069 bash[21528]: audit 2024-04-02T04:48:07.747766+0000 mon.a (mon.0) 930 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:07.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:08.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:08.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:07 smithi098 bash[23790]: audit 2024-04-02T04:48:07.106446+0000 mon.b (mon.1) 98 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:08.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:07 smithi098 bash[23790]: audit 2024-04-02T04:48:07.178052+0000 mon.b (mon.1) 99 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:08.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:07 smithi098 bash[23790]: audit 2024-04-02T04:48:07.180434+0000 mon.a (mon.0) 929 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:08.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:07 smithi098 bash[23790]: audit 2024-04-02T04:48:07.732977+0000 mon.b (mon.1) 100 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:08.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:07 smithi098 bash[23790]: audit 2024-04-02T04:48:07.734695+0000 mon.b (mon.1) 101 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:08.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:07 smithi098 bash[23790]: audit 2024-04-02T04:48:07.747766+0000 mon.a (mon.0) 930 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:08.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:08.175 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:08.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:08.305 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:08.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:08.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:08.565 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:08.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:08.697 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:08.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:08.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:08.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:08.955 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:08.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:08.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:08 smithi069 bash[21528]: cluster 2024-04-02T04:48:07.813093+0000 mgr.y (mgr.14538) 158 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:08.965 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:48:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:48:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:48:08.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:08 smithi069 bash[28904]: cluster 2024-04-02T04:48:07.813093+0000 mgr.y (mgr.14538) 158 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:09.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:09.078 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:09.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:09.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:08 smithi098 bash[23790]: cluster 2024-04-02T04:48:07.813093+0000 mgr.y (mgr.14538) 158 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:09.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:09.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:09.462 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:09.565 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:09.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:09.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:09.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:09.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:09.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:09.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:09.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:10.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:10.215 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.240 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:10.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.369 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:10.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:10.498 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:10.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.627 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:10.687 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:10.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.754 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:10.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:10.883 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:10.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:10.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:11.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:10 smithi069 bash[21528]: cluster 2024-04-02T04:48:09.813765+0000 mgr.y (mgr.14538) 159 : cluster 0 pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:11.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:10 smithi069 bash[28904]: cluster 2024-04-02T04:48:09.813765+0000 mgr.y (mgr.14538) 159 : cluster 0 pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:11.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:11.142 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:11.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:10 smithi098 bash[23790]: cluster 2024-04-02T04:48:09.813765+0000 mgr.y (mgr.14538) 159 : cluster 0 pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:11.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:11.272 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:11.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:11.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:11.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:11.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:11.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:11.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:11.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:12.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:12.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:12.177 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:12.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:12.302 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:12.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:12.427 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:12.454 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:48:12.454 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:48:12.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:12.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:12.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:12.682 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:12.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 -- 172.21.15.69:0/3018533014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c102520 msgr2=0x7fa50c102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 --2- 172.21.15.69:0/3018533014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c102520 0x7fa50c102900 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009920 tx=0x7fa4fc009c30 comp rx=0 tx=0).stop 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 -- 172.21.15.69:0/3018533014 shutdown_connections 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 --2- 172.21.15.69:0/3018533014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c1075a0 0x7fa50c10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 --2- 172.21.15.69:0/3018533014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c102520 0x7fa50c102900 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 --2- 172.21.15.69:0/3018533014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c102ed0 0x7fa50c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.688+0000 7fa513967700 1 -- 172.21.15.69:0/3018533014 >> 172.21.15.69:0/3018533014 conn(0x7fa50c079360 msgr2=0x7fa50c079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:12.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- 172.21.15.69:0/3018533014 shutdown_connections 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- 172.21.15.69:0/3018533014 wait complete. 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 librados: starting msgr at 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 librados: starting objecter 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 Processor -- start 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- start start 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 librados: setting wanted keys 2024-04-02T04:48:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 librados: calling monclient init 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c102520 0x7fa50c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c102ed0 0x7fa50c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c1075a0 0x7fa50c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c111ac0 con 0x7fa50c102ed0 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c111940 con 0x7fa50c102520 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa50c111c40 con 0x7fa50c1075a0 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa511f04700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c1075a0 0x7fa50c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c102ed0 0x7fa50c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa511f04700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c1075a0 0x7fa50c19ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47596/0 (socket says 172.21.15.69:47596) 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa511f04700 1 -- 172.21.15.69:0/2718145031 learned_addr learned my addr 172.21.15.69:0/2718145031 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c102520 0x7fa50c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c1075a0 msgr2=0x7fa50c19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c1075a0 0x7fa50c19ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c102520 msgr2=0x7fa50c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c102520 0x7fa50c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c19d1b0 con 0x7fa50c102ed0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa50c1075a0 0x7fa50c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c102ed0 0x7fa50c1986e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa50c069e80 tx=0x7fa50800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa5027fc700 10 client.?.objecter ms_handle_connect 0x7fa50c102ed0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa5027fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508015020 con 0x7fa50c102ed0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa508011cb0 con 0x7fa50c102ed0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50c19d4a0 con 0x7fa50c102ed0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508010bb0 con 0x7fa50c102ed0 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 10 client.14775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50c1a4d20 con 0x7fa50c102ed0 2024-04-02T04:48:12.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 librados: init done 2024-04-02T04:48:12.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa50c04ff00 con 0x7fa50c102ed0 2024-04-02T04:48:12.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.692+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa508018020 con 0x7fa50c102ed0 2024-04-02T04:48:12.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa4f806f5e0 0x7fa4f8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(122..122 src has 64..122) ==== 7537+0+0 (secure 0 0 0) 0x7fa50808d6e0 con 0x7fa50c102ed0 2024-04-02T04:48:12.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 osd_map(122..122 src has 64..122) 2024-04-02T04:48:12.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 3 client.14775.objecter handle_osd_map got epochs [122,122] > 0 2024-04-02T04:48:12.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 3 client.14775.objecter handle_osd_map decoding full epoch 122 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa4f806f5e0 0x7fa4f8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 20 client.14775.objecter dump_active .. 0 homeless 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v122) ==== 54+0+16 (secure 0 0 0) 0x7fa50801f020 con 0x7fa50c102ed0 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 10 client.14775.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 1 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa50c062e50 0x7fa50c063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 1 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 0 1 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 4 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 10 client.14775.objecter _send_command 1 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa50c1a80e0 con 0x7fa50c062e50 2024-04-02T04:48:12.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa4f806f5e0 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa4f806f5e0 0x7fa4f8071aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009680 tx=0x7fa4fc019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa50c062e50 0x7fa50c063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c062e50 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa50c062e50 0x7fa50c063230 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c1a80e0 con 0x7fa50c062e50 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 1: 0 ) 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 10 client.14775.objecter _finish_command 1 = system:0 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 4 2024-04-02T04:48:12.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.696+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 0 1 2024-04-02T04:48:12.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 10 client.14775.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 2 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 2 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c199500 osd=0 3 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 0 2 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 4 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 10 client.14775.objecter _send_command 2 2024-04-02T04:48:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.712+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c0628d0 con 0x7fa50c062e50 2024-04-02T04:48:12.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c0628d0 con 0x7fa50c062e50 2024-04-02T04:48:12.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 10 client.14775.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 4 2024-04-02T04:48:12.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 0 2 2024-04-02T04:48:12.717 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:48:12.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 10 client.14775.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa50c02c9b0 0x7fa50c02cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 1 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 4 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 10 client.14775.objecter _send_command 3 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa50c109f10 con 0x7fa50c02c9b0 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa50c02c9b0 0x7fa50c02cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c02c9b0 2024-04-02T04:48:12.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa50c02c9b0 0x7fa50c02cd90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c109f10 con 0x7fa50c02c9b0 2024-04-02T04:48:12.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 3: 0 ) 2024-04-02T04:48:12.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 10 client.14775.objecter _finish_command 3 = system:0 2024-04-02T04:48:12.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 4 2024-04-02T04:48:12.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.716+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 1 3 2024-04-02T04:48:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 10 client.14775.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 4 2024-04-02T04:48:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 4 2024-04-02T04:48:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c109890 osd=1 3 2024-04-02T04:48:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 1 4 2024-04-02T04:48:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 4 2024-04-02T04:48:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 10 client.14775.objecter _send_command 4 2024-04-02T04:48:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.732+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c109310 con 0x7fa50c02c9b0 2024-04-02T04:48:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c109310 con 0x7fa50c02c9b0 2024-04-02T04:48:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 10 client.14775.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 4 2024-04-02T04:48:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 1 4 2024-04-02T04:48:12.737 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:48:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 10 client.14775.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 5 2024-04-02T04:48:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa50c1aa2a0 0x7fa50c1ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 5 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 2 5 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 4 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 10 client.14775.objecter _send_command 5 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa50c02dfb0 con 0x7fa50c1aa2a0 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa50c1aa2a0 0x7fa50c1ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa50c1aa2a0 0x7fa50c1ac690 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c1aa2a0 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c02dfb0 con 0x7fa50c1aa2a0 2024-04-02T04:48:12.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 5: 0 ) 2024-04-02T04:48:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 10 client.14775.objecter _finish_command 5 = system:0 2024-04-02T04:48:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 4 2024-04-02T04:48:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.736+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 2 5 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 10 client.14775.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 6 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 6 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c109710 osd=2 3 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 2 6 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 4 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 10 client.14775.objecter _send_command 6 2024-04-02T04:48:12.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c02d5e0 con 0x7fa50c1aa2a0 2024-04-02T04:48:12.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c02d5e0 con 0x7fa50c1aa2a0 2024-04-02T04:48:12.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 10 client.14775.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 4 2024-04-02T04:48:12.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 2 6 2024-04-02T04:48:12.758 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:48:12.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 10 client.14775.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 7 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa50c1add50 0x7fa50c1b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 7 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 3 7 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 4 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 10 client.14775.objecter _send_command 7 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa50c1b2fa0 con 0x7fa50c1add50 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa50c1add50 0x7fa50c1b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c1add50 2024-04-02T04:48:12.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa50c1add50 0x7fa50c1b0140 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c1b2fa0 con 0x7fa50c1add50 2024-04-02T04:48:12.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 7: 0 ) 2024-04-02T04:48:12.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 10 client.14775.objecter _finish_command 7 = system:0 2024-04-02T04:48:12.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 4 2024-04-02T04:48:12.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.756+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 3 7 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 10 client.14775.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 8 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 8 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1a7730 osd=3 3 2024-04-02T04:48:12.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 3 8 2024-04-02T04:48:12.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 4 2024-04-02T04:48:12.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 10 client.14775.objecter _send_command 8 2024-04-02T04:48:12.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c1a71b0 con 0x7fa50c1add50 2024-04-02T04:48:12.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c1a71b0 con 0x7fa50c1add50 2024-04-02T04:48:12.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa5027fc700 10 client.14775.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 4 2024-04-02T04:48:12.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 3 8 2024-04-02T04:48:12.778 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:48:12.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 10 client.14775.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 9 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa50c1ad8b0 0x7fa50c1b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 9 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 4 9 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 4 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 10 client.14775.objecter _send_command 9 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa50c1b6900 con 0x7fa50c1ad8b0 2024-04-02T04:48:12.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa50c1ad8b0 0x7fa50c1b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c1ad8b0 2024-04-02T04:48:12.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.776+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa50c1ad8b0 0x7fa50c1b1800 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.780+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c1b6900 con 0x7fa50c1ad8b0 2024-04-02T04:48:12.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.780+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 9: 0 ) 2024-04-02T04:48:12.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.780+0000 7fa5027fc700 10 client.14775.objecter _finish_command 9 = system:0 2024-04-02T04:48:12.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.780+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 4 2024-04-02T04:48:12.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.780+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 4 9 2024-04-02T04:48:12.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 10 client.14775.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 10 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 10 2024-04-02T04:48:12.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1a71b0 osd=4 3 2024-04-02T04:48:12.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 4 10 2024-04-02T04:48:12.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 4 2024-04-02T04:48:12.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 10 client.14775.objecter _send_command 10 2024-04-02T04:48:12.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.796+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c1a74a0 con 0x7fa50c1ad8b0 2024-04-02T04:48:12.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c1a74a0 con 0x7fa50c1ad8b0 2024-04-02T04:48:12.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa5027fc700 10 client.14775.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 4 2024-04-02T04:48:12.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 4 10 2024-04-02T04:48:12.802 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 10 client.14775.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 11 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa50c1b0e50 0x7fa50c1b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 11 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 5 11 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 4 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 10 client.14775.objecter _send_command 11 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa50c076480 con 0x7fa50c1b0e50 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa50c1b0e50 0x7fa50c1b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c1b0e50 2024-04-02T04:48:12.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.800+0000 7fa510f02700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa50c1b0e50 0x7fa50c1b1250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.804+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c076480 con 0x7fa50c1b0e50 2024-04-02T04:48:12.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.804+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 11: 0 ) 2024-04-02T04:48:12.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.804+0000 7fa5027fc700 10 client.14775.objecter _finish_command 11 = system:0 2024-04-02T04:48:12.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.804+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 4 2024-04-02T04:48:12.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.804+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 5 11 2024-04-02T04:48:12.811 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 10 client.14775.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 12 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 12 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1b6950 osd=5 3 2024-04-02T04:48:12.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 5 12 2024-04-02T04:48:12.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 4 2024-04-02T04:48:12.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 10 client.14775.objecter _send_command 12 2024-04-02T04:48:12.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c1baf20 con 0x7fa50c1b0e50 2024-04-02T04:48:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c1baf20 con 0x7fa50c1b0e50 2024-04-02T04:48:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa5027fc700 10 client.14775.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 4 2024-04-02T04:48:12.827 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:48:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 5 12 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 10 client.14775.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 13 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa50c1bb640 0x7fa50c1bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 13 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 6 13 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 4 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 10 client.14775.objecter _send_command 13 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa50c1bf260 con 0x7fa50c1bb640 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.824+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa50c1bb640 0x7fa50c1bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c1bb640 2024-04-02T04:48:12.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa511f04700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa50c1bb640 0x7fa50c1bbaa0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c1bf260 con 0x7fa50c1bb640 2024-04-02T04:48:12.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 13: 0 ) 2024-04-02T04:48:12.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa5027fc700 10 client.14775.objecter _finish_command 13 = system:0 2024-04-02T04:48:12.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 4 2024-04-02T04:48:12.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.828+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 6 13 2024-04-02T04:48:12.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 10 client.14775.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 14 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 14 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1baf20 osd=6 3 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 6 14 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 4 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 10 client.14775.objecter _send_command 14 2024-04-02T04:48:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.844+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c1b5160 con 0x7fa50c1bb640 2024-04-02T04:48:12.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c1b5160 con 0x7fa50c1bb640 2024-04-02T04:48:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa5027fc700 10 client.14775.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 4 2024-04-02T04:48:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 6 14 2024-04-02T04:48:12.849 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 10 client.14775.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 15 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fa50c1b5880 0x7fa50c1b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 15 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 7 15 2024-04-02T04:48:12.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 4 2024-04-02T04:48:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 10 client.14775.objecter _send_command 15 2024-04-02T04:48:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa50c1b63e0 con 0x7fa50c1b5880 2024-04-02T04:48:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fa50c1b5880 0x7fa50c1b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa5027fc700 10 client.14775.objecter ms_handle_connect 0x7fa50c1b5880 2024-04-02T04:48:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.848+0000 7fa511703700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fa50c1b5880 0x7fa50c1b5ce0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.852+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa50c1b63e0 con 0x7fa50c1b5880 2024-04-02T04:48:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.852+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 15: 0 ) 2024-04-02T04:48:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.852+0000 7fa5027fc700 10 client.14775.objecter _finish_command 15 = system:0 2024-04-02T04:48:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.852+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 4 2024-04-02T04:48:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.852+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 7 15 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 10 client.14775.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign -1 16 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 20 client.14775.objecter _get_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 15 client.14775.objecter _session_command_op_remove -1 16 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 20 client.14775.objecter get_session s=0x7fa50c1b5160 osd=7 3 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 15 client.14775.objecter _session_command_op_assign 7 16 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 4 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 10 client.14775.objecter _send_command 16 2024-04-02T04:48:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.868+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa50c1b63e0 con 0x7fa50c1b5880 2024-04-02T04:48:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa5027fc700 1 -- 172.21.15.69:0/2718145031 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa50c1b63e0 con 0x7fa50c1b5880 2024-04-02T04:48:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa5027fc700 10 client.14775.objecter ms_dispatch 0x7fa50c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa5027fc700 10 client.14775.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa5027fc700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 4 2024-04-02T04:48:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa5027fc700 15 client.14775.objecter _session_command_op_remove 7 16 2024-04-02T04:48:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 librados: watch_flush enter 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 librados: watch_flush exit 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.0 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa50c062e50 msgr2=0x7fa50c063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa50c062e50 0x7fa50c063230 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c199500 osd=0 1 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.1 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa50c02c9b0 msgr2=0x7fa50c02cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa50c02c9b0 0x7fa50c02cd90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109890 osd=1 1 2024-04-02T04:48:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.2 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa50c1aa2a0 msgr2=0x7fa50c1ac690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa50c1aa2a0 0x7fa50c1ac690 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c109710 osd=2 1 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.3 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa50c1add50 msgr2=0x7fa50c1b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa50c1add50 0x7fa50c1b0140 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a7730 osd=3 1 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.4 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa50c1ad8b0 msgr2=0x7fa50c1b1800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa50c1ad8b0 0x7fa50c1b1800 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1a71b0 osd=4 1 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.5 2024-04-02T04:48:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa50c1b0e50 msgr2=0x7fa50c1b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa50c1b0e50 0x7fa50c1b1250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b6950 osd=5 1 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.6 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa50c1bb640 msgr2=0x7fa50c1bbaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa50c1bb640 0x7fa50c1bbaa0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1baf20 osd=6 1 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter close_session for osd.7 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fa50c1b5880 msgr2=0x7fa50c1b5ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fa50c1b5880 0x7fa50c1b5ce0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter put_session s=0x7fa50c1b5160 osd=7 1 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 20 client.14775.objecter shutdown clearing up homeless session... 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 10 client.14775.objecter successfully canceled tick 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa511f04700 1 -- 172.21.15.69:0/2718145031 reap_dead start 2024-04-02T04:48:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa511f04700 1 -- 172.21.15.69:0/2718145031 reap_dead start 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa511f04700 1 -- 172.21.15.69:0/2718145031 reap_dead start 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa4f806f5e0 msgr2=0x7fa4f8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa511f04700 1 -- 172.21.15.69:0/2718145031 reap_dead start 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa4f806f5e0 0x7fa4f8071aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009680 tx=0x7fa4fc019040 comp rx=0 tx=0).stop 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa511f04700 1 -- 172.21.15.69:0/2718145031 reap_dead start 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c102ed0 msgr2=0x7fa50c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c102ed0 0x7fa50c1986e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa50c069e80 tx=0x7fa50800b730 comp rx=0 tx=0).stop 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 shutdown_connections 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 --2- 172.21.15.69:0/2718145031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa50c102ed0 0x7fa50c1986e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 >> 172.21.15.69:0/2718145031 conn(0x7fa50c079360 msgr2=0x7fa50c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 shutdown_connections 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 -- 172.21.15.69:0/2718145031 wait complete. 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:12.872+0000 7fa513967700 1 librados: shutdown 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:48:12.879 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:48:12.880 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:48:12.880 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:48:12.880 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:48:12.880 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:48:12.880 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:48:12.880 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:48:12.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:12.940 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:12.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:13.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:12 smithi069 bash[21528]: cluster 2024-04-02T04:48:11.814459+0000 mgr.y (mgr.14538) 160 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:13.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:12 smithi069 bash[21528]: audit 2024-04-02T04:48:12.699792+0000 mon.a (mon.0) 931 : audit 0 from='client.? 172.21.15.69:0/2718145031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:13.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:12 smithi069 bash[28904]: cluster 2024-04-02T04:48:11.814459+0000 mgr.y (mgr.14538) 160 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:13.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:12 smithi069 bash[28904]: audit 2024-04-02T04:48:12.699792+0000 mon.a (mon.0) 931 : audit 0 from='client.? 172.21.15.69:0/2718145031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:13.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:13.062 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:13.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:13.176 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:48:12 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:48:13.176 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:12 smithi098 bash[23790]: cluster 2024-04-02T04:48:11.814459+0000 mgr.y (mgr.14538) 160 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-02T04:48:13.176 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:12 smithi098 bash[23790]: audit 2024-04-02T04:48:12.699792+0000 mon.a (mon.0) 931 : audit 0 from='client.? 172.21.15.69:0/2718145031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:13.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:13.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:13.426 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:13.452 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:13.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:13.583 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:13.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:13.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:13.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:13.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:13.974 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:13.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:14.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:13 smithi069 bash[21528]: audit 2024-04-02T04:48:12.920879+0000 mgr.y (mgr.14538) 161 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:14.040 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:13 smithi069 bash[28904]: audit 2024-04-02T04:48:12.920879+0000 mgr.y (mgr.14538) 161 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:14.077 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:14.103 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:14.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:14.217 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:13 smithi098 bash[23790]: audit 2024-04-02T04:48:12.920879+0000 mgr.y (mgr.14538) 161 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:14.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:14.364 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:14.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-02T04:48:14.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:14.494 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:14.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:14.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:14.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:14.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:14.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:14.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:15.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[21528]: cluster 2024-04-02T04:48:13.738401+0000 osd.7 (osd.7) 105 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:15.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[21528]: cluster 2024-04-02T04:48:13.738519+0000 osd.7 (osd.7) 106 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:15.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[21528]: cluster 2024-04-02T04:48:13.815913+0000 mgr.y (mgr.14538) 162 : cluster 0 pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-02T04:48:15.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[21528]: cluster 2024-04-02T04:48:13.887203+0000 mon.a (mon.0) 932 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-04-02T04:48:15.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[28904]: cluster 2024-04-02T04:48:13.738401+0000 osd.7 (osd.7) 105 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:15.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[28904]: cluster 2024-04-02T04:48:13.738519+0000 osd.7 (osd.7) 106 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:15.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[28904]: cluster 2024-04-02T04:48:13.815913+0000 mgr.y (mgr.14538) 162 : cluster 0 pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-02T04:48:15.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:14 smithi069 bash[28904]: cluster 2024-04-02T04:48:13.887203+0000 mon.a (mon.0) 932 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-04-02T04:48:15.112 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:15.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:15.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:14 smithi098 bash[23790]: cluster 2024-04-02T04:48:13.738401+0000 osd.7 (osd.7) 105 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:15.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:14 smithi098 bash[23790]: cluster 2024-04-02T04:48:13.738519+0000 osd.7 (osd.7) 106 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:14 smithi098 bash[23790]: cluster 2024-04-02T04:48:13.815913+0000 mgr.y (mgr.14538) 162 : cluster 0 pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-02T04:48:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:14 smithi098 bash[23790]: cluster 2024-04-02T04:48:13.887203+0000 mon.a (mon.0) 932 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-04-02T04:48:15.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:15.270 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:15.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:15.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:15.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:15.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:15.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:15.660 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:15.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:15.788 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:15.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:15.918 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:15.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:16.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:16.048 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:16.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:16.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:16.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:16.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:16.434 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:16.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:16.564 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:16.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:16.693 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:16.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:16.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:48:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:16.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:16.936 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:16 smithi098 bash[23790]: cluster 2024-04-02T04:48:15.816449+0000 mgr.y (mgr.14538) 163 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:16.953 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:16.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:17.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 -- 172.21.15.69:0/719432119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea44101b60 msgr2=0x7fea4410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 --2- 172.21.15.69:0/719432119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea44101b60 0x7fea4410dfd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00b0a0 tx=0x7fea3c00b3b0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 -- 172.21.15.69:0/719432119 shutdown_connections 2024-04-02T04:48:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 --2- 172.21.15.69:0/719432119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea44101b60 0x7fea4410dfd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 --2- 172.21.15.69:0/719432119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea44107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 --2- 172.21.15.69:0/719432119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea441011a0 0x7fea44101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 -- 172.21.15.69:0/719432119 >> 172.21.15.69:0/719432119 conn(0x7fea440771e0 msgr2=0x7fea440775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 -- 172.21.15.69:0/719432119 shutdown_connections 2024-04-02T04:48:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.011+0000 7fea4adc4700 1 -- 172.21.15.69:0/719432119 wait complete. 2024-04-02T04:48:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 librados: starting msgr at 2024-04-02T04:48:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 librados: starting objecter 2024-04-02T04:48:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 Processor -- start 2024-04-02T04:48:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 -- start start 2024-04-02T04:48:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 librados: setting wanted keys 2024-04-02T04:48:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 librados: calling monclient init 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea441011a0 0x7fea44195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea44101b60 0x7fea441962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea4419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea4419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea4419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47612/0 (socket says 172.21.15.69:47612) 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 -- 172.21.15.69:0/3201193281 learned_addr learned my addr 172.21.15.69:0/3201193281 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea43fff700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea44101b60 0x7fea441962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea4410f8d0 con 0x7fea441011a0 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea48b60700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea441011a0 0x7fea44195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fea4410f750 con 0x7fea44101b60 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fea4410fa50 con 0x7fea44107220 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 -- 172.21.15.69:0/3201193281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea44101b60 msgr2=0x7fea441962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea44101b60 0x7fea441962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 -- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea441011a0 msgr2=0x7fea44195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea441011a0 0x7fea44195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4419ad90 con 0x7fea44107220 2024-04-02T04:48:17.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea43fff700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea44101b60 0x7fea441962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:48:17.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea49361700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea4419a650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009850 tx=0x7fea3c007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea41ffb700 10 client.?.objecter ms_handle_connect 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea41ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c003c00 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fea3c003da0 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4419b020 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c01b7d0 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 10 client.24668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea441a2900 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.015+0000 7fea4adc4700 1 librados: init done 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4404ff00 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fea3c005610 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fea340739d0 0x7fea34075e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(123..123 src has 64..123) ==== 7537+0+0 (secure 0 0 0) 0x7fea3c00f020 con 0x7fea44107220 2024-04-02T04:48:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 10 client.24668.objecter ms_dispatch 0x7fea44001550 osd_map(123..123 src has 64..123) 2024-04-02T04:48:17.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 3 client.24668.objecter handle_osd_map got epochs [123,123] > 0 2024-04-02T04:48:17.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 3 client.24668.objecter handle_osd_map decoding full epoch 123 2024-04-02T04:48:17.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.019+0000 7fea41ffb700 20 client.24668.objecter dump_active .. 0 homeless 2024-04-02T04:48:17.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 -- 172.21.15.69:0/1484171180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580101dd0 msgr2=0x7fb580105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 --2- 172.21.15.69:0/1484171180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580101dd0 0x7fb580105e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb5740099c0 tx=0x7fb574009cd0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 -- 172.21.15.69:0/1484171180 shutdown_connections 2024-04-02T04:48:17.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 --2- 172.21.15.69:0/1484171180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580106500 0x7fb58010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 --2- 172.21.15.69:0/1484171180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb580101420 0x7fb580101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 --2- 172.21.15.69:0/1484171180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580101dd0 0x7fb580105e20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fb584d00700 1 -- 172.21.15.69:0/1484171180 >> 172.21.15.69:0/1484171180 conn(0x7fb580078240 msgr2=0x7fb580078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:17.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- 172.21.15.69:0/1484171180 shutdown_connections 2024-04-02T04:48:17.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- 172.21.15.69:0/1484171180 wait complete. 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 librados: starting msgr at 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 librados: starting objecter 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 Processor -- start 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- start start 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 librados: setting wanted keys 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 librados: calling monclient init 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb580101420 0x7fb580198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 0x7fb580199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 0x7fb580199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 0x7fb580199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58738/0 (socket says 172.21.15.69:58738) 2024-04-02T04:48:17.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 -- 172.21.15.69:0/2054321550 learned_addr learned my addr 172.21.15.69:0/2054321550 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580106500 0x7fb5801a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57ed9d700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580106500 0x7fb5801a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5801109c0 con 0x7fb580101dd0 2024-04-02T04:48:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57e59c700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb580101420 0x7fb580198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb580110840 con 0x7fb580101420 2024-04-02T04:48:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fea48b60700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fea340739d0 0x7fea34075e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fea48b60700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fea340739d0 0x7fea34075e90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fea38009680 tx=0x7fea38009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fea41ffb700 10 client.24668.objecter ms_handle_connect 0x7fea340739d0 2024-04-02T04:48:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.027+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fea3c05c270 con 0x7fea44107220 2024-04-02T04:48:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb580110b40 con 0x7fb580106500 2024-04-02T04:48:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 -- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580106500 msgr2=0x7fb5801a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580106500 0x7fb5801a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 -- 172.21.15.69:0/2054321550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb580101420 msgr2=0x7fb580198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb580101420 0x7fb580198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5801a44e0 con 0x7fb580101dd0 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb57dd9b700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 0x7fb580199460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb57401be20 tx=0x7fb57401be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb56f7fe700 10 client.?.objecter ms_handle_connect 0x7fb580101dd0 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb56f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57402b070 con 0x7fb580101dd0 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb57400f8e0 con 0x7fb580101dd0 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb580196d10 con 0x7fb580101dd0 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.031+0000 7fb584d00700 10 client.14787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574026680 con 0x7fb580101dd0 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb584d00700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb580197180 con 0x7fb580101dd0 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb57402e070 con 0x7fb580101dd0 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb584d00700 1 librados: init done 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb56806f5e0 0x7fb568071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(123..123 src has 64..123) ==== 7537+0+0 (secure 0 0 0) 0x7fb574093260 con 0x7fb580101dd0 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 10 client.14787.objecter ms_dispatch 0x7fb580001550 osd_map(123..123 src has 64..123) 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 3 client.14787.objecter handle_osd_map got epochs [123,123] > 0 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 3 client.14787.objecter handle_osd_map decoding full epoch 123 2024-04-02T04:48:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56f7fe700 20 client.14787.objecter dump_active .. 0 homeless 2024-04-02T04:48:17.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb57e59c700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb56806f5e0 0x7fb568071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:17.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.035+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb564005120 con 0x7fb580101dd0 2024-04-02T04:48:17.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.039+0000 7fb57e59c700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb56806f5e0 0x7fb568071aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb570009c60 tx=0x7fb570009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:17.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.039+0000 7fb56f7fe700 10 client.14787.objecter ms_handle_connect 0x7fb56806f5e0 2024-04-02T04:48:17.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.043+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb574013070 con 0x7fb580101dd0 2024-04-02T04:48:17.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:17.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:17.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:16 smithi069 bash[21528]: cluster 2024-04-02T04:48:15.816449+0000 mgr.y (mgr.14538) 163 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:17.190 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:16 smithi069 bash[28904]: cluster 2024-04-02T04:48:15.816449+0000 mgr.y (mgr.14538) 163 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:17.204 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:17.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.283+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb564000bc0 con 0x7fb56806f5e0 2024-04-02T04:48:17.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.291+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fea4402db30 con 0x7fea44107220 2024-04-02T04:48:17.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:17.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.303+0000 7fb56f7fe700 1 -- 172.21.15.69:0/2054321550 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346356 (secure 0 0 0) 0x7fb564000bc0 con 0x7fb56806f5e0 2024-04-02T04:48:17.311 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:48:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 10 librados: watch_flush enter 2024-04-02T04:48:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 10 librados: watch_flush exit 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 20 client.14787.objecter shutdown clearing up homeless session... 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 10 client.14787.objecter successfully canceled tick 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb56806f5e0 msgr2=0x7fb568071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb56806f5e0 0x7fb568071aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb570009c60 tx=0x7fb570009260 comp rx=0 tx=0).stop 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 msgr2=0x7fb580199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 0x7fb580199460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb57401be20 tx=0x7fb57401be50 comp rx=0 tx=0).stop 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 shutdown_connections 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb56806f5e0 0x7fb568071aa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb580106500 0x7fb5801a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb580101420 0x7fb580198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 --2- 172.21.15.69:0/2054321550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb580101dd0 0x7fb580199460 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 >> 172.21.15.69:0/2054321550 conn(0x7fb580078240 msgr2=0x7fb5800fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:17.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 shutdown_connections 2024-04-02T04:48:17.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 -- 172.21.15.69:0/2054321550 wait complete. 2024-04-02T04:48:17.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.307+0000 7fb56d7fa700 1 librados: shutdown 2024-04-02T04:48:17.315 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:48:17.333 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:17.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:17.456 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:17.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:17.585 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:17.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:17.817 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:17.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea41ffb700 1 -- 172.21.15.69:0/3201193281 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v124) ==== 97+0+0 (secure 0 0 0) 0x7fea3c05fcc0 con 0x7fea44107220 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 10 librados: watch_flush enter 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 10 librados: watch_flush exit 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 20 client.24668.objecter shutdown clearing up homeless session... 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 10 client.24668.objecter successfully canceled tick 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fea340739d0 msgr2=0x7fea34075e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fea340739d0 0x7fea34075e90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fea38009680 tx=0x7fea38009260 comp rx=0 tx=0).stop 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 msgr2=0x7fea4419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea4419a650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009850 tx=0x7fea3c007a50 comp rx=0 tx=0).stop 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.907+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 shutdown_connections 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fea340739d0 0x7fea34075e90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea44107220 0x7fea4419a650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea441011a0 0x7fea44195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 --2- 172.21.15.69:0/3201193281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea44101b60 0x7fea441962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 >> 172.21.15.69:0/3201193281 conn(0x7fea440771e0 msgr2=0x7fea440ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 shutdown_connections 2024-04-02T04:48:17.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 -- 172.21.15.69:0/3201193281 wait complete. 2024-04-02T04:48:17.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:17.911+0000 7fea4adc4700 1 librados: shutdown 2024-04-02T04:48:17.914 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:48:17.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:17.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:17 smithi069 bash[21528]: audit 2024-04-02T04:48:17.291676+0000 mgr.y (mgr.14538) 164 : audit 0 from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:17.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:17 smithi069 bash[21528]: audit 2024-04-02T04:48:17.300076+0000 mon.c (mon.2) 29 : audit 1 from='client.? 172.21.15.69:0/3201193281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:17.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:17 smithi069 bash[21528]: audit 2024-04-02T04:48:17.300443+0000 mon.a (mon.0) 933 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:17.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:17 smithi069 bash[28904]: audit 2024-04-02T04:48:17.291676+0000 mgr.y (mgr.14538) 164 : audit 0 from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:17.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:17 smithi069 bash[28904]: audit 2024-04-02T04:48:17.300076+0000 mon.c (mon.2) 29 : audit 1 from='client.? 172.21.15.69:0/3201193281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:17.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:17 smithi069 bash[28904]: audit 2024-04-02T04:48:17.300443+0000 mon.a (mon.0) 933 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:17.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:18.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:18.096 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:18.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:18.221 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:17 smithi098 bash[23790]: audit 2024-04-02T04:48:17.291676+0000 mgr.y (mgr.14538) 164 : audit 0 from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:17 smithi098 bash[23790]: audit 2024-04-02T04:48:17.300076+0000 mon.c (mon.2) 29 : audit 1 from='client.? 172.21.15.69:0/3201193281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:18.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:17 smithi098 bash[23790]: audit 2024-04-02T04:48:17.300443+0000 mon.a (mon.0) 933 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:18.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:18.419 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-04-02T04:48:15.816194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1473,"num_read_kb":1287,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240796,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720420,"statfs":{"total":767960285184,"available":767713710080,"internally_reserved":0,"allocated":8187904,"data_stored":5155722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004137"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679150+0000","last_change":"2024-04-02T04:47:14.337462+0000","last_active":"2024-04-02T04:48:02.679150+0000","last_peered":"2024-04-02T04:48:02.679150+0000","last_clean":"2024-04-02T04:48:02.679150+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:48:02.679150+0000","last_undegraded":"2024-04-02T04:48:02.679150+0000","last_fullsized":"2024-04-02T04:48:02.679150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.337178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.950297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677808+0000","last_change":"2024-04-02T04:47:52.450415+0000","last_active":"2024-04-02T04:48:02.677808+0000","last_peered":"2024-04-02T04:48:02.677808+0000","last_clean":"2024-04-02T04:48:02.677808+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:48:02.677808+0000","last_undegraded":"2024-04-02T04:48:02.677808+0000","last_fullsized":"2024-04-02T04:48:02.677808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:52.450181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:52.450181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:59.502057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.678311+0000","last_change":"2024-04-02T04:47:54.214859+0000","last_active":"2024-04-02T04:48:02.678311+0000","last_peered":"2024-04-02T04:48:02.678311+0000","last_clean":"2024-04-02T04:48:02.678311+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:48:02.678311+0000","last_undegraded":"2024-04-02T04:48:02.678311+0000","last_fullsized":"2024-04-02T04:48:02.678311+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.434714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":140,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679554+0000","last_change":"2024-04-02T04:46:39.861495+0000","last_active":"2024-04-02T04:48:02.679554+0000","last_peered":"2024-04-02T04:48:02.679554+0000","last_clean":"2024-04-02T04:48:02.679554+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:48:02.679554+0000","last_undegraded":"2024-04-02T04:48:02.679554+0000","last_fullsized":"2024-04-02T04:48:02.679554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:39.861258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:39.861258+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":103,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.547672+0000","last_change":"2024-04-02T04:47:11.925670+0000","last_active":"2024-04-02T04:48:03.547672+0000","last_peered":"2024-04-02T04:48:03.547672+0000","last_clean":"2024-04-02T04:48:03.547672+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:48:03.547672+0000","last_undegraded":"2024-04-02T04:48:03.547672+0000","last_fullsized":"2024-04-02T04:48:03.547672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:11.925452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:11.925452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.710115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":107,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.547644+0000","last_change":"2024-04-02T04:47:07.889053+0000","last_active":"2024-04-02T04:48:03.547644+0000","last_peered":"2024-04-02T04:48:03.547644+0000","last_clean":"2024-04-02T04:48:03.547644+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:48:03.547644+0000","last_undegraded":"2024-04-02T04:48:03.547644+0000","last_fullsized":"2024-04-02T04:48:03.547644+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:07.888835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:07.888835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:10.239127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679206+0000","last_change":"2024-04-02T04:47:42.551341+0000","last_active":"2024-04-02T04:48:02.679206+0000","last_peered":"2024-04-02T04:48:02.679206+0000","last_clean":"2024-04-02T04:48:02.679206+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:48:02.679206+0000","last_undegraded":"2024-04-02T04:48:02.679206+0000","last_fullsized":"2024-04-02T04:48:02.679206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.551243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.551243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:59.906138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":144,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.678112+0000","last_change":"2024-04-02T04:46:45.396999+0000","last_active":"2024-04-02T04:48:02.678112+0000","last_peered":"2024-04-02T04:48:02.678112+0000","last_clean":"2024-04-02T04:48:02.678112+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:48:02.678112+0000","last_undegraded":"2024-04-02T04:48:02.678112+0000","last_fullsized":"2024-04-02T04:48:02.678112+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:45.396760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.396760+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.943901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":110,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.663426+0000","last_change":"2024-04-02T04:47:42.186212+0000","last_active":"2024-04-02T04:48:03.663426+0000","last_peered":"2024-04-02T04:48:03.663426+0000","last_clean":"2024-04-02T04:48:03.663426+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:48:03.663426+0000","last_undegraded":"2024-04-02T04:48:03.663426+0000","last_fullsized":"2024-04-02T04:48:03.663426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.185988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.185988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.015773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548137+0000","last_change":"2024-04-02T04:47:37.876855+0000","last_active":"2024-04-02T04:48:03.548137+0000","last_peered":"2024-04-02T04:48:03.548137+0000","last_clean":"2024-04-02T04:48:03.548137+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:48:03.548137+0000","last_undegraded":"2024-04-02T04:48:03.548137+0000","last_fullsized":"2024-04-02T04:48:03.548137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.876644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.876644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.719969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676669+0000","last_change":"2024-04-02T04:47:45.254673+0000","last_active":"2024-04-02T04:48:02.676669+0000","last_peered":"2024-04-02T04:48:02.676669+0000","last_clean":"2024-04-02T04:48:02.676669+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:48:02.676669+0000","last_undegraded":"2024-04-02T04:48:02.676669+0000","last_fullsized":"2024-04-02T04:48:02.676669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.254450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.254450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:58.071917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":135,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.665049+0000","last_change":"2024-04-02T04:47:05.285775+0000","last_active":"2024-04-02T04:48:03.665049+0000","last_peered":"2024-04-02T04:48:03.665049+0000","last_clean":"2024-04-02T04:48:03.665049+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:48:03.665049+0000","last_undegraded":"2024-04-02T04:48:03.665049+0000","last_fullsized":"2024-04-02T04:48:03.665049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:05.285521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.285521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:28.240436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":112,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677917+0000","last_change":"2024-04-02T04:48:02.193707+0000","last_active":"2024-04-02T04:48:02.677917+0000","last_peered":"2024-04-02T04:48:02.677917+0000","last_clean":"2024-04-02T04:48:02.677917+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:48:02.677917+0000","last_undegraded":"2024-04-02T04:48:02.677917+0000","last_fullsized":"2024-04-02T04:48:02.677917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:31.172290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":107,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548500+0000","last_change":"2024-04-02T04:46:44.775289+0000","last_active":"2024-04-02T04:48:03.548500+0000","last_peered":"2024-04-02T04:48:03.548500+0000","last_clean":"2024-04-02T04:48:03.548500+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:48:03.548500+0000","last_undegraded":"2024-04-02T04:48:03.548500+0000","last_fullsized":"2024-04-02T04:48:03.548500+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.775071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.207519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676573+0000","last_change":"2024-04-02T04:47:09.387603+0000","last_active":"2024-04-02T04:48:02.676573+0000","last_peered":"2024-04-02T04:48:02.676573+0000","last_clean":"2024-04-02T04:48:02.676573+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:48:02.676573+0000","last_undegraded":"2024-04-02T04:48:02.676573+0000","last_fullsized":"2024-04-02T04:48:02.676573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:09.387375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.387375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.279958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":145,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.664789+0000","last_change":"2024-04-02T04:46:51.350464+0000","last_active":"2024-04-02T04:48:03.664789+0000","last_peered":"2024-04-02T04:48:03.664789+0000","last_clean":"2024-04-02T04:48:03.664789+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:48:03.664789+0000","last_undegraded":"2024-04-02T04:48:03.664789+0000","last_fullsized":"2024-04-02T04:48:03.664789+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:46:51.350201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:51.350201+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.520062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":105,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679075+0000","last_change":"2024-04-02T04:47:03.158759+0000","last_active":"2024-04-02T04:48:02.679075+0000","last_peered":"2024-04-02T04:48:02.679075+0000","last_clean":"2024-04-02T04:48:02.679075+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:48:02.679075+0000","last_undegraded":"2024-04-02T04:48:02.679075+0000","last_fullsized":"2024-04-02T04:48:02.679075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:03.158541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:03.158541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.522158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.680144+0000","last_change":"2024-04-02T04:47:57.459178+0000","last_active":"2024-04-02T04:48:02.680144+0000","last_peered":"2024-04-02T04:48:02.680144+0000","last_clean":"2024-04-02T04:48:02.680144+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:48:02.680144+0000","last_undegraded":"2024-04-02T04:48:02.680144+0000","last_fullsized":"2024-04-02T04:48:02.680144+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:57.459040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.459040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:17.124034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679686+0000","last_change":"2024-04-02T04:47:18.359335+0000","last_active":"2024-04-02T04:48:02.679686+0000","last_peered":"2024-04-02T04:48:02.679686+0000","last_clean":"2024-04-02T04:48:02.679686+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:48:02.679686+0000","last_undegraded":"2024-04-02T04:48:02.679686+0000","last_fullsized":"2024-04-02T04:48:02.679686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:18.359101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:18.359101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:38.654960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":167,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.547829+0000","last_change":"2024-04-02T04:47:40.838867+0000","last_active":"2024-04-02T04:48:03.547829+0000","last_peered":"2024-04-02T04:48:03.547829+0000","last_clean":"2024-04-02T04:48:03.547829+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:48:03.547829+0000","last_undegraded":"2024-04-02T04:48:03.547829+0000","last_fullsized":"2024-04-02T04:48:03.547829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.838617+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.066994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.549470+0000","last_change":"2024-04-02T04:47:01.936520+0000","last_active":"2024-04-02T04:48:03.549470+0000","last_peered":"2024-04-02T04:48:03.549470+0000","last_clean":"2024-04-02T04:48:03.549470+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:48:03.549470+0000","last_undegraded":"2024-04-02T04:48:03.549470+0000","last_fullsized":"2024-04-02T04:48:03.549470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:01.936275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:01.936275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.709309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.002297+0000","last_change":"2024-04-02T04:47:45.830668+0000","last_active":"2024-04-02T04:48:04.002297+0000","last_peered":"2024-04-02T04:48:04.002297+0000","last_clean":"2024-04-02T04:48:04.002297+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:48:04.002297+0000","last_undegraded":"2024-04-02T04:48:04.002297+0000","last_fullsized":"2024-04-02T04:48:04.002297+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.830369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.830369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.027610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.145932+0000","last_change":"2024-04-02T04:47:53.938679+0000","last_active":"2024-04-02T04:48:03.145932+0000","last_peered":"2024-04-02T04:48:03.145932+0000","last_clean":"2024-04-02T04:48:03.145932+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:48:03.145932+0000","last_undegraded":"2024-04-02T04:48:03.145932+0000","last_fullsized":"2024-04-02T04:48:03.145932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.938537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.938537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:12.383253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677078+0000","last_change":"2024-04-02T04:47:13.347780+0000","last_active":"2024-04-02T04:48:02.677078+0000","last_peered":"2024-04-02T04:48:02.677078+0000","last_clean":"2024-04-02T04:48:02.677078+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:48:02.677078+0000","last_undegraded":"2024-04-02T04:48:02.677078+0000","last_fullsized":"2024-04-02T04:48:02.677078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.347670+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.686984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":103,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000483+0000","last_change":"2024-04-02T04:46:56.263880+0000","last_active":"2024-04-02T04:48:04.000483+0000","last_peered":"2024-04-02T04:48:04.000483+0000","last_clean":"2024-04-02T04:48:04.000483+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:48:04.000483+0000","last_undegraded":"2024-04-02T04:48:04.000483+0000","last_fullsized":"2024-04-02T04:48:04.000483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:56.263740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:56.263740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":174,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:09.505402+0000","last_change":"2024-04-02T04:46:47.154319+0000","last_active":"2024-04-02T04:48:09.505402+0000","last_peered":"2024-04-02T04:48:09.505402+0000","last_clean":"2024-04-02T04:48:09.505402+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:48:09.505402+0000","last_undegraded":"2024-04-02T04:48:09.505402+0000","last_fullsized":"2024-04-02T04:48:09.505402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:47.154082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:47.154082+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.844937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.678975+0000","last_change":"2024-04-02T04:47:58.472809+0000","last_active":"2024-04-02T04:48:02.678975+0000","last_peered":"2024-04-02T04:48:02.678975+0000","last_clean":"2024-04-02T04:48:02.678975+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:48:02.678975+0000","last_undegraded":"2024-04-02T04:48:02.678975+0000","last_fullsized":"2024-04-02T04:48:02.678975+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:47:58.472703+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.472703+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:08.433557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":148,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676406+0000","last_change":"2024-04-02T04:47:45.516535+0000","last_active":"2024-04-02T04:48:02.676406+0000","last_peered":"2024-04-02T04:48:02.676406+0000","last_clean":"2024-04-02T04:48:02.676406+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:48:02.676406+0000","last_undegraded":"2024-04-02T04:48:02.676406+0000","last_fullsized":"2024-04-02T04:48:02.676406+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:45.516407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.516407+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.285568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":105,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675837+0000","last_change":"2024-04-02T04:46:42.290326+0000","last_active":"2024-04-02T04:48:02.675837+0000","last_peered":"2024-04-02T04:48:02.675837+0000","last_clean":"2024-04-02T04:48:02.675837+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:48:02.675837+0000","last_undegraded":"2024-04-02T04:48:02.675837+0000","last_fullsized":"2024-04-02T04:48:02.675837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:46:42.290071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.290071+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.053573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677541+0000","last_change":"2024-04-02T04:47:39.172102+0000","last_active":"2024-04-02T04:48:02.677541+0000","last_peered":"2024-04-02T04:48:02.677541+0000","last_clean":"2024-04-02T04:48:02.677541+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:48:02.677541+0000","last_undegraded":"2024-04-02T04:48:02.677541+0000","last_fullsized":"2024-04-02T04:48:02.677541+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:39.171955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.171955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:54.119072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.663156+0000","last_change":"2024-04-02T04:47:27.380112+0000","last_active":"2024-04-02T04:48:03.663156+0000","last_peered":"2024-04-02T04:48:03.663156+0000","last_clean":"2024-04-02T04:48:03.663156+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:48:03.663156+0000","last_undegraded":"2024-04-02T04:48:03.663156+0000","last_fullsized":"2024-04-02T04:48:03.663156+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:27.379800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:27.379800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.831508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":184,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000207+0000","last_change":"2024-04-02T04:46:59.265375+0000","last_active":"2024-04-02T04:48:04.000207+0000","last_peered":"2024-04-02T04:48:04.000207+0000","last_clean":"2024-04-02T04:48:04.000207+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:48:04.000207+0000","last_undegraded":"2024-04-02T04:48:04.000207+0000","last_fullsized":"2024-04-02T04:48:04.000207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:46:59.265104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:59.265104+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.709345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.680529+0000","last_change":"2024-04-02T04:47:13.288506+0000","last_active":"2024-04-02T04:48:02.680529+0000","last_peered":"2024-04-02T04:48:02.680529+0000","last_clean":"2024-04-02T04:48:02.680529+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:48:02.680529+0000","last_undegraded":"2024-04-02T04:48:02.680529+0000","last_fullsized":"2024-04-02T04:48:02.680529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:13.288364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.288364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.927428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":108,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.678838+0000","last_change":"2024-04-02T04:47:12.271950+0000","last_active":"2024-04-02T04:48:02.678838+0000","last_peered":"2024-04-02T04:48:02.678838+0000","last_clean":"2024-04-02T04:48:02.678838+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:48:02.678838+0000","last_undegraded":"2024-04-02T04:48:02.678838+0000","last_fullsized":"2024-04-02T04:48:02.678838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.271680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.271680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.801349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.146747+0000","last_change":"2024-04-02T04:47:23.324160+0000","last_active":"2024-04-02T04:48:03.146747+0000","last_peered":"2024-04-02T04:48:03.146747+0000","last_clean":"2024-04-02T04:48:03.146747+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:48:03.146747+0000","last_undegraded":"2024-04-02T04:48:03.146747+0000","last_fullsized":"2024-04-02T04:48:03.146747+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:23.323919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.323919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.280221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":158,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.665370+0000","last_change":"2024-04-02T04:48:01.257149+0000","last_active":"2024-04-02T04:48:03.665370+0000","last_peered":"2024-04-02T04:48:03.665370+0000","last_clean":"2024-04-02T04:48:03.665370+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:48:03.665370+0000","last_undegraded":"2024-04-02T04:48:03.665370+0000","last_fullsized":"2024-04-02T04:48:03.665370+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:48:01.256898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:01.256898+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.641480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":103,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675316+0000","last_change":"2024-04-02T04:47:20.427195+0000","last_active":"2024-04-02T04:48:02.675316+0000","last_peered":"2024-04-02T04:48:02.675316+0000","last_clean":"2024-04-02T04:48:02.675316+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:48:02.675316+0000","last_undegraded":"2024-04-02T04:48:02.675316+0000","last_fullsized":"2024-04-02T04:48:02.675316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:20.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:47:20.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:48.706473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676717+0000","last_change":"2024-04-02T04:47:19.398500+0000","last_active":"2024-04-02T04:48:02.676717+0000","last_peered":"2024-04-02T04:48:02.676717+0000","last_clean":"2024-04-02T04:48:02.676717+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:48:02.676717+0000","last_undegraded":"2024-04-02T04:48:02.676717+0000","last_fullsized":"2024-04-02T04:48:02.676717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:19.398328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:19.398328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.197933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.665264+0000","last_change":"2024-04-02T04:47:46.278114+0000","last_active":"2024-04-02T04:48:03.665264+0000","last_peered":"2024-04-02T04:48:03.665264+0000","last_clean":"2024-04-02T04:48:03.665264+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:48:03.665264+0000","last_undegraded":"2024-04-02T04:48:03.665264+0000","last_fullsized":"2024-04-02T04:48:03.665264+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:46.277889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:47:46.277889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.753634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":133,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677917+0000","last_change":"2024-04-02T04:47:08.333929+0000","last_active":"2024-04-02T04:48:02.677917+0000","last_peered":"2024-04-02T04:48:02.677917+0000","last_clean":"2024-04-02T04:48:02.677917+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:48:02.677917+0000","last_undegraded":"2024-04-02T04:48:02.677917+0000","last_fullsized":"2024-04-02T04:48:02.677917+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:47:08.333516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.333516+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.726461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677374+0000","last_change":"2024-04-02T04:46:38.893014+0000","last_active":"2024-04-02T04:48:02.677374+0000","last_peered":"2024-04-02T04:48:02.677374+0000","last_clean":"2024-04-02T04:48:02.677374+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:48:02.677374+0000","last_undegraded":"2024-04-02T04:48:02.677374+0000","last_fullsized":"2024-04-02T04:48:02.677374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.892773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.892773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548074+0000","last_change":"2024-04-02T04:47:31.779975+0000","last_active":"2024-04-02T04:48:03.548074+0000","last_peered":"2024-04-02T04:48:03.548074+0000","last_clean":"2024-04-02T04:48:03.548074+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:48:03.548074+0000","last_undegraded":"2024-04-02T04:48:03.548074+0000","last_fullsized":"2024-04-02T04:48:03.548074+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:31.779694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:31.779694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:40.720115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001050+0000","last_change":"2024-04-02T04:46:54.269554+0000","last_active":"2024-04-02T04:48:04.001050+0000","last_peered":"2024-04-02T04:48:04.001050+0000","last_clean":"2024-04-02T04:48:04.001050+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:48:04.001050+0000","last_undegraded":"2024-04-02T04:48:04.001050+0000","last_fullsized":"2024-04-02T04:48:04.001050+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:54.269343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:54.269343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.989031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":158,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548600+0000","last_change":"2024-04-02T04:48:00.691627+0000","last_active":"2024-04-02T04:48:03.548600+0000","last_peered":"2024-04-02T04:48:03.548600+0000","last_clean":"2024-04-02T04:48:03.548600+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:48:03.548600+0000","last_undegraded":"2024-04-02T04:48:03.548600+0000","last_fullsized":"2024-04-02T04:48:03.548600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:26.639529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677971+0000","last_change":"2024-04-02T04:47:18.291662+0000","last_active":"2024-04-02T04:48:02.677971+0000","last_peered":"2024-04-02T04:48:02.677971+0000","last_clean":"2024-04-02T04:48:02.677971+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:48:02.677971+0000","last_undegraded":"2024-04-02T04:48:02.677971+0000","last_fullsized":"2024-04-02T04:48:02.677971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:18.291429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:18.291429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:39.162464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":107,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000135+0000","last_change":"2024-04-02T04:46:53.297425+0000","last_active":"2024-04-02T04:48:04.000135+0000","last_peered":"2024-04-02T04:48:04.000135+0000","last_clean":"2024-04-02T04:48:04.000135+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:48:04.000135+0000","last_undegraded":"2024-04-02T04:48:04.000135+0000","last_fullsized":"2024-04-02T04:48:04.000135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:53.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:53.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.497586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.145753+0000","last_change":"2024-04-02T04:46:34.223691+0000","last_active":"2024-04-02T04:48:03.145753+0000","last_peered":"2024-04-02T04:48:03.145753+0000","last_clean":"2024-04-02T04:48:03.145753+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:48:03.145753+0000","last_undegraded":"2024-04-02T04:48:03.145753+0000","last_fullsized":"2024-04-02T04:48:03.145753+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:34.223346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:34.223346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.085223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":147,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676938+0000","last_change":"2024-04-02T04:47:36.370736+0000","last_active":"2024-04-02T04:48:02.676938+0000","last_peered":"2024-04-02T04:48:02.676938+0000","last_clean":"2024-04-02T04:48:02.676938+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:48:02.676938+0000","last_undegraded":"2024-04-02T04:48:02.676938+0000","last_fullsized":"2024-04-02T04:48:02.676938+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:36.370493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.370493+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.329887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":103,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.549469+0000","last_change":"2024-04-02T04:47:05.885138+0000","last_active":"2024-04-02T04:48:03.549469+0000","last_peered":"2024-04-02T04:48:03.549469+0000","last_clean":"2024-04-02T04:48:03.549469+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:48:03.549469+0000","last_undegraded":"2024-04-02T04:48:03.549469+0000","last_fullsized":"2024-04-02T04:48:03.549469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:30.330301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":171,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.500644+0000","last_change":"2024-04-02T04:46:44.346692+0000","last_active":"2024-04-02T04:48:04.500644+0000","last_peered":"2024-04-02T04:48:04.500644+0000","last_clean":"2024-04-02T04:48:04.500644+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:48:04.500644+0000","last_undegraded":"2024-04-02T04:48:04.500644+0000","last_fullsized":"2024-04-02T04:48:04.500644+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:44.346318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.346318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.092878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676885+0000","last_change":"2024-04-02T04:46:36.244725+0000","last_active":"2024-04-02T04:48:02.676885+0000","last_peered":"2024-04-02T04:48:02.676885+0000","last_clean":"2024-04-02T04:48:02.676885+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:48:02.676885+0000","last_undegraded":"2024-04-02T04:48:02.676885+0000","last_fullsized":"2024-04-02T04:48:02.676885+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:36.244514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:36.244514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":136,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677683+0000","last_change":"2024-04-02T04:47:53.179435+0000","last_active":"2024-04-02T04:48:02.677683+0000","last_peered":"2024-04-02T04:48:02.677683+0000","last_clean":"2024-04-02T04:48:02.677683+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:48:02.677683+0000","last_undegraded":"2024-04-02T04:48:02.677683+0000","last_fullsized":"2024-04-02T04:48:02.677683+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:47:53.179316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.179316+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.558109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677196+0000","last_change":"2024-04-02T04:47:09.295716+0000","last_active":"2024-04-02T04:48:02.677196+0000","last_peered":"2024-04-02T04:48:02.677196+0000","last_clean":"2024-04-02T04:48:02.677196+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:48:02.677196+0000","last_undegraded":"2024-04-02T04:48:02.677196+0000","last_fullsized":"2024-04-02T04:48:02.677196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:09.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.295498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:13.587769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"121'12","reported_seq":179,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.505790+0000","last_change":"2024-04-02T04:48:02.448348+0000","last_active":"2024-04-02T04:48:04.505790+0000","last_peered":"2024-04-02T04:48:04.505790+0000","last_clean":"2024-04-02T04:48:04.505790+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:48:04.505790+0000","last_undegraded":"2024-04-02T04:48:04.505790+0000","last_fullsized":"2024-04-02T04:48:04.505790+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:07.814627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.663476+0000","last_change":"2024-04-02T04:46:48.355080+0000","last_active":"2024-04-02T04:48:03.663476+0000","last_peered":"2024-04-02T04:48:03.663476+0000","last_clean":"2024-04-02T04:48:03.663476+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:48:03.663476+0000","last_undegraded":"2024-04-02T04:48:03.663476+0000","last_fullsized":"2024-04-02T04:48:03.663476+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:48.354869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:48.354869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.638786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676713+0000","last_change":"2024-04-02T04:48:00.302558+0000","last_active":"2024-04-02T04:48:02.676713+0000","last_peered":"2024-04-02T04:48:02.676713+0000","last_clean":"2024-04-02T04:48:02.676713+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:48:02.676713+0000","last_undegraded":"2024-04-02T04:48:02.676713+0000","last_fullsized":"2024-04-02T04:48:02.676713+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:26.019165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":103,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676130+0000","last_change":"2024-04-02T04:47:04.415949+0000","last_active":"2024-04-02T04:48:02.676130+0000","last_peered":"2024-04-02T04:48:02.676130+0000","last_clean":"2024-04-02T04:48:02.676130+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:48:02.676130+0000","last_undegraded":"2024-04-02T04:48:02.676130+0000","last_fullsized":"2024-04-02T04:48:02.676130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:04.415735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.415735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:34.243533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679208+0000","last_change":"2024-04-02T04:46:45.921390+0000","last_active":"2024-04-02T04:48:02.679208+0000","last_peered":"2024-04-02T04:48:02.679208+0000","last_clean":"2024-04-02T04:48:02.679208+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:48:02.679208+0000","last_undegraded":"2024-04-02T04:48:02.679208+0000","last_fullsized":"2024-04-02T04:48:02.679208+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:14.369948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.146240+0000","last_change":"2024-04-02T04:47:35.160389+0000","last_active":"2024-04-02T04:48:03.146240+0000","last_peered":"2024-04-02T04:48:03.146240+0000","last_clean":"2024-04-02T04:48:03.146240+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:48:03.146240+0000","last_undegraded":"2024-04-02T04:48:03.146240+0000","last_fullsized":"2024-04-02T04:48:03.146240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.160172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.160172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.333854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.145970+0000","last_change":"2024-04-02T04:46:41.254878+0000","last_active":"2024-04-02T04:48:03.145970+0000","last_peered":"2024-04-02T04:48:03.145970+0000","last_clean":"2024-04-02T04:48:03.145970+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:48:03.145970+0000","last_undegraded":"2024-04-02T04:48:03.145970+0000","last_fullsized":"2024-04-02T04:48:03.145970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:46:41.254633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.254633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":102,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.002355+0000","last_change":"2024-04-02T04:46:38.622583+0000","last_active":"2024-04-02T04:48:04.002355+0000","last_peered":"2024-04-02T04:48:04.002355+0000","last_clean":"2024-04-02T04:48:04.002355+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:48:04.002355+0000","last_undegraded":"2024-04-02T04:48:04.002355+0000","last_fullsized":"2024-04-02T04:48:04.002355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:38.622220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:38.622220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.379107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":108,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677085+0000","last_change":"2024-04-02T04:46:57.429067+0000","last_active":"2024-04-02T04:48:02.677085+0000","last_peered":"2024-04-02T04:48:02.677085+0000","last_clean":"2024-04-02T04:48:02.677085+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:48:02.677085+0000","last_undegraded":"2024-04-02T04:48:02.677085+0000","last_fullsized":"2024-04-02T04:48:02.677085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.428901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.428901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.996741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001762+0000","last_change":"2024-04-02T04:47:49.839212+0000","last_active":"2024-04-02T04:48:04.001762+0000","last_peered":"2024-04-02T04:48:04.001762+0000","last_clean":"2024-04-02T04:48:04.001762+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:48:04.001762+0000","last_undegraded":"2024-04-02T04:48:04.001762+0000","last_fullsized":"2024-04-02T04:48:04.001762+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.839000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:09.914868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":142,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675478+0000","last_change":"2024-04-02T04:47:00.341837+0000","last_active":"2024-04-02T04:48:02.675478+0000","last_peered":"2024-04-02T04:48:02.675478+0000","last_clean":"2024-04-02T04:48:02.675478+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:48:02.675478+0000","last_undegraded":"2024-04-02T04:48:02.675478+0000","last_fullsized":"2024-04-02T04:48:02.675478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:00.341473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.341473+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:24.505569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.680480+0000","last_change":"2024-04-02T04:47:00.098587+0000","last_active":"2024-04-02T04:48:02.680480+0000","last_peered":"2024-04-02T04:48:02.680480+0000","last_clean":"2024-04-02T04:48:02.680480+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:48:02.680480+0000","last_undegraded":"2024-04-02T04:48:02.680480+0000","last_fullsized":"2024-04-02T04:48:02.680480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.098270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.098270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.905960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":139,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.549070+0000","last_change":"2024-04-02T04:47:14.896155+0000","last_active":"2024-04-02T04:48:03.549070+0000","last_peered":"2024-04-02T04:48:03.549070+0000","last_clean":"2024-04-02T04:48:03.549070+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:48:03.549070+0000","last_undegraded":"2024-04-02T04:48:03.549070+0000","last_fullsized":"2024-04-02T04:48:03.549070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:47:14.895813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.895813+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.683265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":150,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677460+0000","last_change":"2024-04-02T04:47:34.393560+0000","last_active":"2024-04-02T04:48:02.677460+0000","last_peered":"2024-04-02T04:48:02.677460+0000","last_clean":"2024-04-02T04:48:02.677460+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:48:02.677460+0000","last_undegraded":"2024-04-02T04:48:02.677460+0000","last_fullsized":"2024-04-02T04:48:02.677460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:34.393321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:34.393321+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.086104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":178,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.500729+0000","last_change":"2024-04-02T04:47:42.831672+0000","last_active":"2024-04-02T04:48:04.500729+0000","last_peered":"2024-04-02T04:48:04.500729+0000","last_clean":"2024-04-02T04:48:04.500729+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:48:04.500729+0000","last_undegraded":"2024-04-02T04:48:04.500729+0000","last_fullsized":"2024-04-02T04:48:04.500729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.831447+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:46.338365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":113,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.678140+0000","last_change":"2024-04-02T04:47:47.122324+0000","last_active":"2024-04-02T04:48:02.678140+0000","last_peered":"2024-04-02T04:48:02.678140+0000","last_clean":"2024-04-02T04:48:02.678140+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:48:02.678140+0000","last_undegraded":"2024-04-02T04:48:02.678140+0000","last_fullsized":"2024-04-02T04:48:02.678140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.122214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.122214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.799063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001088+0000","last_change":"2024-04-02T04:47:43.151655+0000","last_active":"2024-04-02T04:48:04.001088+0000","last_peered":"2024-04-02T04:48:04.001088+0000","last_clean":"2024-04-02T04:48:04.001088+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:48:04.001088+0000","last_undegraded":"2024-04-02T04:48:04.001088+0000","last_fullsized":"2024-04-02T04:48:04.001088+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:43.151427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:43.151427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.872479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":193,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:09.505261+0000","last_change":"2024-04-02T04:47:39.221972+0000","last_active":"2024-04-02T04:48:09.505261+0000","last_peered":"2024-04-02T04:48:09.505261+0000","last_clean":"2024-04-02T04:48:09.505261+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:48:09.505261+0000","last_undegraded":"2024-04-02T04:48:09.505261+0000","last_fullsized":"2024-04-02T04:48:09.505261+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:47:39.221741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.221741+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:05.865282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000618+0000","last_change":"2024-04-02T04:47:45.109386+0000","last_active":"2024-04-02T04:48:04.000618+0000","last_peered":"2024-04-02T04:48:04.000618+0000","last_clean":"2024-04-02T04:48:04.000618+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:48:04.000618+0000","last_undegraded":"2024-04-02T04:48:04.000618+0000","last_fullsized":"2024-04-02T04:48:04.000618+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.109170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.109170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.076716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":106,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679338+0000","last_change":"2024-04-02T04:47:02.153712+0000","last_active":"2024-04-02T04:48:02.679338+0000","last_peered":"2024-04-02T04:48:02.679338+0000","last_clean":"2024-04-02T04:48:02.679338+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:48:02.679338+0000","last_undegraded":"2024-04-02T04:48:02.679338+0000","last_fullsized":"2024-04-02T04:48:02.679338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:47:02.153481+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.153481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.645312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.663740+0000","last_change":"2024-04-02T04:46:57.319477+0000","last_active":"2024-04-02T04:48:03.663740+0000","last_peered":"2024-04-02T04:48:03.663740+0000","last_clean":"2024-04-02T04:48:03.663740+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:48:03.663740+0000","last_undegraded":"2024-04-02T04:48:03.663740+0000","last_fullsized":"2024-04-02T04:48:03.663740+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.319266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.319266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:25.104997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":168,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548867+0000","last_change":"2024-04-02T04:47:50.758701+0000","last_active":"2024-04-02T04:48:03.548867+0000","last_peered":"2024-04-02T04:48:03.548867+0000","last_clean":"2024-04-02T04:48:03.548867+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:48:03.548867+0000","last_undegraded":"2024-04-02T04:48:03.548867+0000","last_fullsized":"2024-04-02T04:48:03.548867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:47:50.758312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:50.758312+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.203712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001083+0000","last_change":"2024-04-02T04:47:47.106602+0000","last_active":"2024-04-02T04:48:04.001083+0000","last_peered":"2024-04-02T04:48:04.001083+0000","last_clean":"2024-04-02T04:48:04.001083+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:48:04.001083+0000","last_undegraded":"2024-04-02T04:48:04.001083+0000","last_fullsized":"2024-04-02T04:48:04.001083+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.106497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.032879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":111,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676254+0000","last_change":"2024-04-02T04:47:53.271021+0000","last_active":"2024-04-02T04:48:02.676254+0000","last_peered":"2024-04-02T04:48:02.676254+0000","last_clean":"2024-04-02T04:48:02.676254+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:48:02.676254+0000","last_undegraded":"2024-04-02T04:48:02.676254+0000","last_fullsized":"2024-04-02T04:48:02.676254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.902292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":168,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000022+0000","last_change":"2024-04-02T04:46:55.308858+0000","last_active":"2024-04-02T04:48:04.000022+0000","last_peered":"2024-04-02T04:48:04.000022+0000","last_clean":"2024-04-02T04:48:04.000022+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:48:04.000022+0000","last_undegraded":"2024-04-02T04:48:04.000022+0000","last_fullsized":"2024-04-02T04:48:04.000022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:46:55.308528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.308528+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:12.148543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001953+0000","last_change":"2024-04-02T04:47:14.881150+0000","last_active":"2024-04-02T04:48:04.001953+0000","last_peered":"2024-04-02T04:48:04.001953+0000","last_clean":"2024-04-02T04:48:04.001953+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:48:04.001953+0000","last_undegraded":"2024-04-02T04:48:04.001953+0000","last_fullsized":"2024-04-02T04:48:04.001953+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.881031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.881031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:23.207112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":107,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001020+0000","last_change":"2024-04-02T04:46:42.343240+0000","last_active":"2024-04-02T04:48:04.001020+0000","last_peered":"2024-04-02T04:48:04.001020+0000","last_clean":"2024-04-02T04:48:04.001020+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:48:04.001020+0000","last_undegraded":"2024-04-02T04:48:04.001020+0000","last_fullsized":"2024-04-02T04:48:04.001020+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:42.343078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:42.343078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":110,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.665783+0000","last_change":"2024-04-02T04:47:58.186088+0000","last_active":"2024-04-02T04:48:03.665783+0000","last_peered":"2024-04-02T04:48:03.665783+0000","last_clean":"2024-04-02T04:48:03.665783+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:48:03.665783+0000","last_undegraded":"2024-04-02T04:48:03.665783+0000","last_fullsized":"2024-04-02T04:48:03.665783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:58.185868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:58.185868+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.185868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:23.742931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":140,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.680429+0000","last_change":"2024-04-02T04:46:52.985248+0000","last_active":"2024-04-02T04:48:02.680429+0000","last_peered":"2024-04-02T04:48:02.680429+0000","last_clean":"2024-04-02T04:48:02.680429+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:48:02.680429+0000","last_undegraded":"2024-04-02T04:48:02.680429+0000","last_fullsized":"2024-04-02T04:48:02.680429+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:52.984994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.984994+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.008710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.663573+0000","last_change":"2024-04-02T04:46:55.305525+0000","last_active":"2024-04-02T04:48:03.663573+0000","last_peered":"2024-04-02T04:48:03.663573+0000","last_clean":"2024-04-02T04:48:03.663573+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:48:03.663573+0000","last_undegraded":"2024-04-02T04:48:03.663573+0000","last_fullsized":"2024-04-02T04:48:03.663573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.305293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":381,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675404+0000","last_change":"2024-04-02T04:47:44.214306+0000","last_active":"2024-04-02T04:48:02.675404+0000","last_peered":"2024-04-02T04:48:02.675404+0000","last_clean":"2024-04-02T04:48:02.675404+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:48:02.675404+0000","last_undegraded":"2024-04-02T04:48:02.675404+0000","last_fullsized":"2024-04-02T04:48:02.675404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:47:44.214047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.214047+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.404997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.145945+0000","last_change":"2024-04-02T04:47:19.323314+0000","last_active":"2024-04-02T04:48:03.145945+0000","last_peered":"2024-04-02T04:48:03.145945+0000","last_clean":"2024-04-02T04:48:03.145945+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:48:03.145945+0000","last_undegraded":"2024-04-02T04:48:03.145945+0000","last_fullsized":"2024-04-02T04:48:03.145945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:19.323097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:19.323097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:20.126618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":111,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000908+0000","last_change":"2024-04-02T04:48:01.028154+0000","last_active":"2024-04-02T04:48:04.000908+0000","last_peered":"2024-04-02T04:48:04.000908+0000","last_clean":"2024-04-02T04:48:04.000908+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:48:04.000908+0000","last_undegraded":"2024-04-02T04:48:04.000908+0000","last_fullsized":"2024-04-02T04:48:04.000908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:01.027858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:01.027858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:06.895866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":146,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676855+0000","last_change":"2024-04-02T04:46:33.251890+0000","last_active":"2024-04-02T04:48:02.676855+0000","last_peered":"2024-04-02T04:48:02.676855+0000","last_clean":"2024-04-02T04:48:02.676855+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:48:02.676855+0000","last_undegraded":"2024-04-02T04:48:02.676855+0000","last_fullsized":"2024-04-02T04:48:02.676855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:46:33.251772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.251772+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675510+0000","last_change":"2024-04-02T04:47:05.388983+0000","last_active":"2024-04-02T04:48:02.675510+0000","last_peered":"2024-04-02T04:48:02.675510+0000","last_clean":"2024-04-02T04:48:02.675510+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:48:02.675510+0000","last_undegraded":"2024-04-02T04:48:02.675510+0000","last_fullsized":"2024-04-02T04:48:02.675510+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.388768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.388768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:22.419043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677313+0000","last_change":"2024-04-02T04:46:50.400686+0000","last_active":"2024-04-02T04:48:02.677313+0000","last_peered":"2024-04-02T04:48:02.677313+0000","last_clean":"2024-04-02T04:48:02.677313+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:48:02.677313+0000","last_undegraded":"2024-04-02T04:48:02.677313+0000","last_fullsized":"2024-04-02T04:48:02.677313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:46:50.400390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":112,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.001245+0000","last_change":"2024-04-02T04:48:04.001245+0000","last_active":"2024-04-02T04:48:04.001245+0000","last_peered":"2024-04-02T04:48:04.001245+0000","last_clean":"2024-04-02T04:48:04.001245+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:48:04.001245+0000","last_undegraded":"2024-04-02T04:48:04.001245+0000","last_fullsized":"2024-04-02T04:48:04.001245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:48:04.001090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:30.589675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.146031+0000","last_change":"2024-04-02T04:48:00.879991+0000","last_active":"2024-04-02T04:48:03.146031+0000","last_peered":"2024-04-02T04:48:03.146031+0000","last_clean":"2024-04-02T04:48:03.146031+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:48:03.146031+0000","last_undegraded":"2024-04-02T04:48:03.146031+0000","last_fullsized":"2024-04-02T04:48:03.146031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:48:00.879724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.879724+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.988600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676685+0000","last_change":"2024-04-02T04:47:17.471458+0000","last_active":"2024-04-02T04:48:02.676685+0000","last_peered":"2024-04-02T04:48:02.676685+0000","last_clean":"2024-04-02T04:48:02.676685+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:48:02.676685+0000","last_undegraded":"2024-04-02T04:48:02.676685+0000","last_fullsized":"2024-04-02T04:48:02.676685+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:17.471105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:17.471105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:23.064426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":245,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:09.509277+0000","last_change":"2024-04-02T04:47:42.173169+0000","last_active":"2024-04-02T04:48:09.509277+0000","last_peered":"2024-04-02T04:48:09.509277+0000","last_clean":"2024-04-02T04:48:09.509277+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:48:09.509277+0000","last_undegraded":"2024-04-02T04:48:09.509277+0000","last_fullsized":"2024-04-02T04:48:09.509277+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.172894+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.893986+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"121'12","reported_seq":180,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:09.509188+0000","last_change":"2024-04-02T04:47:40.137383+0000","last_active":"2024-04-02T04:48:09.509188+0000","last_peered":"2024-04-02T04:48:09.509188+0000","last_clean":"2024-04-02T04:48:09.509188+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:48:09.509188+0000","last_undegraded":"2024-04-02T04:48:09.509188+0000","last_fullsized":"2024-04-02T04:48:09.509188+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.137127+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:52.797250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548125+0000","last_change":"2024-04-02T04:47:00.939593+0000","last_active":"2024-04-02T04:48:03.548125+0000","last_peered":"2024-04-02T04:48:03.548125+0000","last_clean":"2024-04-02T04:48:03.548125+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:48:03.548125+0000","last_undegraded":"2024-04-02T04:48:03.548125+0000","last_fullsized":"2024-04-02T04:48:03.548125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.939352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.939352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:17.488733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.547575+0000","last_change":"2024-04-02T04:46:49.854837+0000","last_active":"2024-04-02T04:48:03.547575+0000","last_peered":"2024-04-02T04:48:03.547575+0000","last_clean":"2024-04-02T04:48:03.547575+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:48:03.547575+0000","last_undegraded":"2024-04-02T04:48:03.547575+0000","last_fullsized":"2024-04-02T04:48:03.547575+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:49.854618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.854618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000784+0000","last_change":"2024-04-02T04:47:58.111681+0000","last_active":"2024-04-02T04:48:04.000784+0000","last_peered":"2024-04-02T04:48:04.000784+0000","last_clean":"2024-04-02T04:48:04.000784+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:48:04.000784+0000","last_undegraded":"2024-04-02T04:48:04.000784+0000","last_fullsized":"2024-04-02T04:48:04.000784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.111561+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:08.316506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":151,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.678404+0000","last_change":"2024-04-02T04:47:04.162419+0000","last_active":"2024-04-02T04:48:02.678404+0000","last_peered":"2024-04-02T04:48:02.678404+0000","last_clean":"2024-04-02T04:48:02.678404+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:48:02.678404+0000","last_undegraded":"2024-04-02T04:48:02.678404+0000","last_fullsized":"2024-04-02T04:48:02.678404+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:47:04.162146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.162146+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:19.370811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":180,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.505110+0000","last_change":"2024-04-02T04:47:30.470228+0000","last_active":"2024-04-02T04:48:04.505110+0000","last_peered":"2024-04-02T04:48:04.505110+0000","last_clean":"2024-04-02T04:48:04.505110+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:48:04.505110+0000","last_undegraded":"2024-04-02T04:48:04.505110+0000","last_fullsized":"2024-04-02T04:48:04.505110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:47:30.469973+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:47:30.469973+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:37.580189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":112,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677402+0000","last_change":"2024-04-02T04:47:49.224735+0000","last_active":"2024-04-02T04:48:02.677402+0000","last_peered":"2024-04-02T04:48:02.677402+0000","last_clean":"2024-04-02T04:48:02.677402+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:48:02.677402+0000","last_undegraded":"2024-04-02T04:48:02.677402+0000","last_fullsized":"2024-04-02T04:48:02.677402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.916792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":354,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:10.771986+0000","last_change":"2024-04-02T04:46:43.873401+0000","last_active":"2024-04-02T04:48:10.771986+0000","last_peered":"2024-04-02T04:48:10.771986+0000","last_clean":"2024-04-02T04:48:10.771986+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:48:10.771986+0000","last_undegraded":"2024-04-02T04:48:10.771986+0000","last_fullsized":"2024-04-02T04:48:10.771986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:46:43.873013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.873013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":235,"num_read_kb":230,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.146640+0000","last_change":"2024-04-02T04:47:41.043637+0000","last_active":"2024-04-02T04:48:03.146640+0000","last_peered":"2024-04-02T04:48:03.146640+0000","last_clean":"2024-04-02T04:48:03.146640+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:48:03.146640+0000","last_undegraded":"2024-04-02T04:48:03.146640+0000","last_fullsized":"2024-04-02T04:48:03.146640+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:41.043411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.043411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.404918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":131,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677665+0000","last_change":"2024-04-02T04:46:49.354576+0000","last_active":"2024-04-02T04:48:02.677665+0000","last_peered":"2024-04-02T04:48:02.677665+0000","last_clean":"2024-04-02T04:48:02.677665+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:48:02.677665+0000","last_undegraded":"2024-04-02T04:48:02.677665+0000","last_fullsized":"2024-04-02T04:48:02.677665+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:46:49.354293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:49.354293+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:16.146965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":108,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548003+0000","last_change":"2024-04-02T04:47:25.814952+0000","last_active":"2024-04-02T04:48:03.548003+0000","last_peered":"2024-04-02T04:48:03.548003+0000","last_clean":"2024-04-02T04:48:03.548003+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:48:03.548003+0000","last_undegraded":"2024-04-02T04:48:03.548003+0000","last_fullsized":"2024-04-02T04:48:03.548003+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:25.814722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:25.814722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.783076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.680624+0000","last_change":"2024-04-02T04:46:46.969136+0000","last_active":"2024-04-02T04:48:02.680624+0000","last_peered":"2024-04-02T04:48:02.680624+0000","last_clean":"2024-04-02T04:48:02.680624+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:48:02.680624+0000","last_undegraded":"2024-04-02T04:48:02.680624+0000","last_fullsized":"2024-04-02T04:48:02.680624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:46.968921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:46:46.968921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:04.377963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":169,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.500644+0000","last_change":"2024-04-02T04:46:43.734784+0000","last_active":"2024-04-02T04:48:04.500644+0000","last_peered":"2024-04-02T04:48:04.500644+0000","last_clean":"2024-04-02T04:48:04.500644+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:48:04.500644+0000","last_undegraded":"2024-04-02T04:48:04.500644+0000","last_fullsized":"2024-04-02T04:48:04.500644+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:43.734423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.734423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675675+0000","last_change":"2024-04-02T04:47:28.453678+0000","last_active":"2024-04-02T04:48:02.675675+0000","last_peered":"2024-04-02T04:48:02.675675+0000","last_clean":"2024-04-02T04:48:02.675675+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:48:02.675675+0000","last_undegraded":"2024-04-02T04:48:02.675675+0000","last_fullsized":"2024-04-02T04:48:02.675675+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:28.453442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:28.453442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.301035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":158,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.547752+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:48:03.547752+0000","last_peered":"2024-04-02T04:48:03.547752+0000","last_clean":"2024-04-02T04:48:03.547752+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:48:03.547752+0000","last_undegraded":"2024-04-02T04:48:03.547752+0000","last_fullsized":"2024-04-02T04:48:03.547752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":110,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.665170+0000","last_change":"2024-04-02T04:47:53.173015+0000","last_active":"2024-04-02T04:48:03.665170+0000","last_peered":"2024-04-02T04:48:03.665170+0000","last_clean":"2024-04-02T04:48:03.665170+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:48:03.665170+0000","last_undegraded":"2024-04-02T04:48:03.665170+0000","last_fullsized":"2024-04-02T04:48:03.665170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.172792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.172792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.046175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":56,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.002547+0000","last_change":"2024-04-02T04:47:55.855659+0000","last_active":"2024-04-02T04:48:04.002547+0000","last_peered":"2024-04-02T04:48:04.002547+0000","last_clean":"2024-04-02T04:48:04.002547+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:48:04.002547+0000","last_undegraded":"2024-04-02T04:48:04.002547+0000","last_fullsized":"2024-04-02T04:48:04.002547+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.855510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.855510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:18.631275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000243+0000","last_change":"2024-04-02T04:47:44.112479+0000","last_active":"2024-04-02T04:48:04.000243+0000","last_peered":"2024-04-02T04:48:04.000243+0000","last_clean":"2024-04-02T04:48:04.000243+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:48:04.000243+0000","last_undegraded":"2024-04-02T04:48:04.000243+0000","last_fullsized":"2024-04-02T04:48:04.000243+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:44.112270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.112270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:58.773698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":178,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676974+0000","last_change":"2024-04-02T04:47:26.453777+0000","last_active":"2024-04-02T04:48:02.676974+0000","last_peered":"2024-04-02T04:48:02.676974+0000","last_clean":"2024-04-02T04:48:02.676974+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:48:02.676974+0000","last_undegraded":"2024-04-02T04:48:02.676974+0000","last_fullsized":"2024-04-02T04:48:02.676974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:47:26.453340+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:47:26.453340+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:41.433920+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":103,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.549587+0000","last_change":"2024-04-02T04:47:08.862194+0000","last_active":"2024-04-02T04:48:03.549587+0000","last_peered":"2024-04-02T04:48:03.549587+0000","last_clean":"2024-04-02T04:48:03.549587+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:48:03.549587+0000","last_undegraded":"2024-04-02T04:48:03.549587+0000","last_fullsized":"2024-04-02T04:48:03.549587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:08.861978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.861978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:29.953269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000207+0000","last_change":"2024-04-02T04:47:15.265577+0000","last_active":"2024-04-02T04:48:04.000207+0000","last_peered":"2024-04-02T04:48:04.000207+0000","last_clean":"2024-04-02T04:48:04.000207+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:48:04.000207+0000","last_undegraded":"2024-04-02T04:48:04.000207+0000","last_fullsized":"2024-04-02T04:48:04.000207+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_clean_scrub_stamp":"2024-04-02T04:47:15.265490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:39.021422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":153,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.679971+0000","last_change":"2024-04-02T04:47:41.576889+0000","last_active":"2024-04-02T04:48:02.679971+0000","last_peered":"2024-04-02T04:48:02.679971+0000","last_clean":"2024-04-02T04:48:02.679971+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:48:02.679971+0000","last_undegraded":"2024-04-02T04:48:02.679971+0000","last_fullsized":"2024-04-02T04:48:02.679971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:47:41.576640+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.576640+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.199333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":107,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.548463+0000","last_change":"2024-04-02T04:46:55.945141+0000","last_active":"2024-04-02T04:48:03.548463+0000","last_peered":"2024-04-02T04:48:03.548463+0000","last_clean":"2024-04-02T04:48:03.548463+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:48:03.548463+0000","last_undegraded":"2024-04-02T04:48:03.548463+0000","last_fullsized":"2024-04-02T04:48:03.548463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.944866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.944866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":110,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.002334+0000","last_change":"2024-04-02T04:47:37.864283+0000","last_active":"2024-04-02T04:48:04.002334+0000","last_peered":"2024-04-02T04:48:04.002334+0000","last_clean":"2024-04-02T04:48:04.002334+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:48:04.002334+0000","last_undegraded":"2024-04-02T04:48:04.002334+0000","last_fullsized":"2024-04-02T04:48:04.002334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.864039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.864039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.035813+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":178,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.500649+0000","last_change":"2024-04-02T04:47:56.704599+0000","last_active":"2024-04-02T04:48:04.500649+0000","last_peered":"2024-04-02T04:48:04.500649+0000","last_clean":"2024-04-02T04:48:04.500649+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:48:04.500649+0000","last_undegraded":"2024-04-02T04:48:04.500649+0000","last_fullsized":"2024-04-02T04:48:04.500649+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:47:56.704379+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.874702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":115,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676539+0000","last_change":"2024-04-02T04:47:02.316549+0000","last_active":"2024-04-02T04:48:02.676539+0000","last_peered":"2024-04-02T04:48:02.676539+0000","last_clean":"2024-04-02T04:48:02.676539+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:48:02.676539+0000","last_undegraded":"2024-04-02T04:48:02.676539+0000","last_fullsized":"2024-04-02T04:48:02.676539+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:02.316325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.316325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:03.699710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":164,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.146060+0000","last_change":"2024-04-02T04:47:57.887340+0000","last_active":"2024-04-02T04:48:03.146060+0000","last_peered":"2024-04-02T04:48:03.146060+0000","last_clean":"2024-04-02T04:48:03.146060+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:48:03.146060+0000","last_undegraded":"2024-04-02T04:48:03.146060+0000","last_fullsized":"2024-04-02T04:48:03.146060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:47:57.887065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.887065+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:04.529466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":111,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.675020+0000","last_change":"2024-04-02T04:47:31.439024+0000","last_active":"2024-04-02T04:48:02.675020+0000","last_peered":"2024-04-02T04:48:02.675020+0000","last_clean":"2024-04-02T04:48:02.675020+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:48:02.675020+0000","last_undegraded":"2024-04-02T04:48:02.675020+0000","last_fullsized":"2024-04-02T04:48:02.675020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:31.438817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:31.438817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:45.747704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.002207+0000","last_change":"2024-04-02T04:46:41.673523+0000","last_active":"2024-04-02T04:48:04.002207+0000","last_peered":"2024-04-02T04:48:04.002207+0000","last_clean":"2024-04-02T04:48:04.002207+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:48:04.002207+0000","last_undegraded":"2024-04-02T04:48:04.002207+0000","last_fullsized":"2024-04-02T04:48:04.002207+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:41.673448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.673448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:06.552075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":139,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676833+0000","last_change":"2024-04-02T04:47:23.464318+0000","last_active":"2024-04-02T04:48:02.676833+0000","last_peered":"2024-04-02T04:48:02.676833+0000","last_clean":"2024-04-02T04:48:02.676833+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:48:02.676833+0000","last_undegraded":"2024-04-02T04:48:02.676833+0000","last_fullsized":"2024-04-02T04:48:02.676833+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:23.464017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.464017+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:42.507017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.145781+0000","last_change":"2024-04-02T04:47:55.940230+0000","last_active":"2024-04-02T04:48:03.145781+0000","last_peered":"2024-04-02T04:48:03.145781+0000","last_clean":"2024-04-02T04:48:03.145781+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:48:03.145781+0000","last_undegraded":"2024-04-02T04:48:03.145781+0000","last_fullsized":"2024-04-02T04:48:03.145781+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.939983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.939983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.949757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":104,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676490+0000","last_change":"2024-04-02T04:47:36.243027+0000","last_active":"2024-04-02T04:48:02.676490+0000","last_peered":"2024-04-02T04:48:02.676490+0000","last_clean":"2024-04-02T04:48:02.676490+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:48:02.676490+0000","last_undegraded":"2024-04-02T04:48:02.676490+0000","last_fullsized":"2024-04-02T04:48:02.676490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.633289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.000898+0000","last_change":"2024-04-02T04:47:23.176576+0000","last_active":"2024-04-02T04:48:04.000898+0000","last_peered":"2024-04-02T04:48:04.000898+0000","last_clean":"2024-04-02T04:48:04.000898+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:48:04.000898+0000","last_undegraded":"2024-04-02T04:48:04.000898+0000","last_fullsized":"2024-04-02T04:48:04.000898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:23.176366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.176366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:30.576582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.664722+0000","last_change":"2024-04-02T04:47:44.215606+0000","last_active":"2024-04-02T04:48:03.664722+0000","last_peered":"2024-04-02T04:48:03.664722+0000","last_clean":"2024-04-02T04:48:03.664722+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:48:03.664722+0000","last_undegraded":"2024-04-02T04:48:03.664722+0000","last_fullsized":"2024-04-02T04:48:03.664722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:47:44.215294+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.215294+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.905308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.677395+0000","last_change":"2024-04-02T04:47:35.260903+0000","last_active":"2024-04-02T04:48:02.677395+0000","last_peered":"2024-04-02T04:48:02.677395+0000","last_clean":"2024-04-02T04:48:02.677395+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:48:02.677395+0000","last_undegraded":"2024-04-02T04:48:02.677395+0000","last_fullsized":"2024-04-02T04:48:02.677395+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.260653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.260653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.706903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":102,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.664233+0000","last_change":"2024-04-02T04:47:12.345553+0000","last_active":"2024-04-02T04:48:03.664233+0000","last_peered":"2024-04-02T04:48:03.664233+0000","last_clean":"2024-04-02T04:48:03.664233+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:48:03.664233+0000","last_undegraded":"2024-04-02T04:48:03.664233+0000","last_fullsized":"2024-04-02T04:48:03.664233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.345237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.345237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:26.300398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:04.002469+0000","last_change":"2024-04-02T04:46:33.616502+0000","last_active":"2024-04-02T04:48:04.002469+0000","last_peered":"2024-04-02T04:48:04.002469+0000","last_clean":"2024-04-02T04:48:04.002469+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:48:04.002469+0000","last_undegraded":"2024-04-02T04:48:04.002469+0000","last_fullsized":"2024-04-02T04:48:04.002469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:33.616300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:33.616300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":136,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:02.676552+0000","last_change":"2024-04-02T04:46:52.343834+0000","last_active":"2024-04-02T04:48:02.676552+0000","last_peered":"2024-04-02T04:48:02.676552+0000","last_clean":"2024-04-02T04:48:02.676552+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:48:02.676552+0000","last_undegraded":"2024-04-02T04:48:02.676552+0000","last_fullsized":"2024-04-02T04:48:02.676552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:46:52.343572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.343572+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:09.554866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":114,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-02T04:48:03.662925+0000","last_change":"2024-04-02T04:47:48.245279+0000","last_active":"2024-04-02T04:48:03.662925+0000","last_peered":"2024-04-02T04:48:03.662925+0000","last_clean":"2024-04-02T04:48:03.662925+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:48:03.662925+0000","last_undegraded":"2024-04-02T04:48:03.662925+0000","last_fullsized":"2024-04-02T04:48:03.662925+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:48.245180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:48.245180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.877121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":235,"num_read_kb":230,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":629,"ondisk_log_size":629,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743486,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1413120,"data_stored":1014612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103241,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1380352,"data_stored":1012425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":802816,"data_stored":423137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712608,"statfs":{"total":95995035648,"available":95961710592,"internally_reserved":0,"allocated":1396736,"data_stored":1013089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378738,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":790528,"data_stored":423373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051688,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":786432,"data_stored":422438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822750,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":827392,"data_stored":423366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462994,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":790528,"data_stored":423282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:48:18.424 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-04-02T04:48:18.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:18.477 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:18.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:18.599 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:18.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:18.711 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:48:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:48:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:18.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:18.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:18.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[21528]: cluster 2024-04-02T04:48:17.817767+0000 mgr.y (mgr.14538) 165 : cluster 0 pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:18.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[21528]: cluster 2024-04-02T04:48:17.900891+0000 mon.a (mon.0) 934 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:18.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[21528]: audit 2024-04-02T04:48:17.908873+0000 mon.a (mon.0) 935 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:48:18.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[21528]: cluster 2024-04-02T04:48:17.912019+0000 mon.a (mon.0) 936 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-04-02T04:48:18.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[28904]: cluster 2024-04-02T04:48:17.817767+0000 mgr.y (mgr.14538) 165 : cluster 0 pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:18.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[28904]: cluster 2024-04-02T04:48:17.900891+0000 mon.a (mon.0) 934 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:18.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[28904]: audit 2024-04-02T04:48:17.908873+0000 mon.a (mon.0) 935 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:48:18.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:18 smithi069 bash[28904]: cluster 2024-04-02T04:48:17.912019+0000 mon.a (mon.0) 936 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-04-02T04:48:18.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:18 smithi098 bash[23790]: cluster 2024-04-02T04:48:17.817767+0000 mgr.y (mgr.14538) 165 : cluster 0 pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:18.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:18 smithi098 bash[23790]: cluster 2024-04-02T04:48:17.900891+0000 mon.a (mon.0) 934 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:18 smithi098 bash[23790]: audit 2024-04-02T04:48:17.908873+0000 mon.a (mon.0) 935 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:48:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:18 smithi098 bash[23790]: cluster 2024-04-02T04:48:17.912019+0000 mon.a (mon.0) 936 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-04-02T04:48:18.978 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:18.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:19.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:19.211 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:19.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:19.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:19.366 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:19.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:19.496 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:19.599 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:19.622 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:19.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:19.752 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:19.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:19.883 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:19.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:19.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:20.012 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:20.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:20.140 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:20.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:20.271 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:20.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:20.397 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:20.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:20.518 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:20.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:20.635 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:20.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:20.769 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:20.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:20.893 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:20.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:20.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:21.022 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:21.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:20 smithi069 bash[28904]: cluster 2024-04-02T04:48:19.818458+0000 mgr.y (mgr.14538) 166 : cluster 0 pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:21.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:20 smithi069 bash[28904]: cluster 2024-04-02T04:48:20.072047+0000 osd.0 (osd.0) 105 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:21.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:20 smithi069 bash[28904]: cluster 2024-04-02T04:48:20.072118+0000 osd.0 (osd.0) 106 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:21.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:20 smithi069 bash[21528]: cluster 2024-04-02T04:48:19.818458+0000 mgr.y (mgr.14538) 166 : cluster 0 pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:21.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:20 smithi069 bash[21528]: cluster 2024-04-02T04:48:20.072047+0000 osd.0 (osd.0) 105 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:21.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:20 smithi069 bash[21528]: cluster 2024-04-02T04:48:20.072118+0000 osd.0 (osd.0) 106 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:21.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:21.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:21.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:48:21.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:20 smithi098 bash[23790]: cluster 2024-04-02T04:48:19.818458+0000 mgr.y (mgr.14538) 166 : cluster 0 pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:21.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:20 smithi098 bash[23790]: cluster 2024-04-02T04:48:20.072047+0000 osd.0 (osd.0) 105 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:21.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:20 smithi098 bash[23790]: cluster 2024-04-02T04:48:20.072118+0000 osd.0 (osd.0) 106 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:21.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:21.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:21.404 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:21.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:21.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:21.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:21.661 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:21.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:21.794 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:21.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:21.924 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:21.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:22.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:21 smithi069 bash[21528]: cluster 2024-04-02T04:48:20.949505+0000 mon.a (mon.0) 937 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-04-02T04:48:22.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:21 smithi069 bash[28904]: cluster 2024-04-02T04:48:20.949505+0000 mon.a (mon.0) 937 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-04-02T04:48:22.050 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:22.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:22.216 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:22.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:21 smithi098 bash[23790]: cluster 2024-04-02T04:48:20.949505+0000 mon.a (mon.0) 937 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-04-02T04:48:22.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:22.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:22.474 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:22.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:22.605 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:22.663 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:22.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:22.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:22.864 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:22.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:22.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:22.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:23.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:23.106 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:48:22 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:48:23.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:22 smithi098 bash[23790]: cluster 2024-04-02T04:48:21.819122+0000 mgr.y (mgr.14538) 167 : cluster 0 pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-04-02T04:48:23.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:22 smithi098 bash[23790]: audit 2024-04-02T04:48:22.191856+0000 mon.b (mon.1) 102 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:23.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:23.247 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:23.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:22 smithi069 bash[28904]: cluster 2024-04-02T04:48:21.819122+0000 mgr.y (mgr.14538) 167 : cluster 0 pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-04-02T04:48:23.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:22 smithi069 bash[28904]: audit 2024-04-02T04:48:22.191856+0000 mon.b (mon.1) 102 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:22 smithi069 bash[21528]: cluster 2024-04-02T04:48:21.819122+0000 mgr.y (mgr.14538) 167 : cluster 0 pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-04-02T04:48:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:22 smithi069 bash[21528]: audit 2024-04-02T04:48:22.191856+0000 mon.b (mon.1) 102 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:23.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:23.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:48:23.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:48:23.425 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-02T04:48:23.425 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-02T04:48:23.480 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:23.500 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:23.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:23.622 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:23.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:23.742 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:23.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:23.870 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:23.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:23.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:24.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:24.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:23 smithi069 bash[21528]: audit 2024-04-02T04:48:22.928870+0000 mgr.y (mgr.14538) 168 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:24.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:23 smithi069 bash[28904]: audit 2024-04-02T04:48:22.928870+0000 mgr.y (mgr.14538) 168 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:24.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:24.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:23 smithi098 bash[23790]: audit 2024-04-02T04:48:22.928870+0000 mgr.y (mgr.14538) 168 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:24.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:24.249 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:24.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:24.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:24.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:24.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:24.628 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:24.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:24.751 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:24.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:24.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:24.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:24.996 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:24.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:24 smithi069 bash[21528]: cluster 2024-04-02T04:48:23.820500+0000 mgr.y (mgr.14538) 169 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:24.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:24 smithi069 bash[28904]: cluster 2024-04-02T04:48:23.820500+0000 mgr.y (mgr.14538) 169 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:24.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:25.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:25.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:24 smithi098 bash[23790]: cluster 2024-04-02T04:48:23.820500+0000 mgr.y (mgr.14538) 169 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:25.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:25.376 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:25.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:25.506 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:25.609 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:25.634 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:25.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:25.765 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:25.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:25.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:25.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:26.024 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:26.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:26.148 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:26.251 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:26.271 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:26.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:26.397 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:26.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:26.521 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:26.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:26.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:26.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:26.780 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:26.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:26.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:27.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:27.032 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:27.138 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:26 smithi098 bash[23790]: cluster 2024-04-02T04:48:25.821196+0000 mgr.y (mgr.14538) 170 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:27.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:26 smithi069 bash[21528]: cluster 2024-04-02T04:48:25.821196+0000 mgr.y (mgr.14538) 170 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:27.272 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:26 smithi069 bash[28904]: cluster 2024-04-02T04:48:25.821196+0000 mgr.y (mgr.14538) 170 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:27.284 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:27.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:27.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:27.418 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:27.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:27.538 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:27.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.665 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:27.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.790 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:27.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:27.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:27.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:28.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:28.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:28.283 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:28.304 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:28.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:28.430 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:28.532 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:28.554 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:28.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:28.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:28.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:28.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:28.938 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:28.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:28.976 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:48:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:48:29.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:29.062 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:29.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:29.176 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:28 smithi098 bash[23790]: cluster 2024-04-02T04:48:27.822099+0000 mgr.y (mgr.14538) 171 : cluster 0 pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:48:29.176 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:28 smithi098 bash[23790]: cluster 2024-04-02T04:48:28.047645+0000 osd.6 (osd.6) 73 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:29.176 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:28 smithi098 bash[23790]: cluster 2024-04-02T04:48:28.047745+0000 osd.6 (osd.6) 74 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:29.191 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:29.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:29.273 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[21528]: cluster 2024-04-02T04:48:27.822099+0000 mgr.y (mgr.14538) 171 : cluster 0 pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:48:29.273 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[21528]: cluster 2024-04-02T04:48:28.047645+0000 osd.6 (osd.6) 73 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:29.273 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[21528]: cluster 2024-04-02T04:48:28.047745+0000 osd.6 (osd.6) 74 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:29.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[28904]: cluster 2024-04-02T04:48:27.822099+0000 mgr.y (mgr.14538) 171 : cluster 0 pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:48:29.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[28904]: cluster 2024-04-02T04:48:28.047645+0000 osd.6 (osd.6) 73 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:29.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:28 smithi069 bash[28904]: cluster 2024-04-02T04:48:28.047745+0000 osd.6 (osd.6) 74 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:29.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:29.317 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:29.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:29.444 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:29.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:29.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:29.807 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:29.830 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:29.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:29.954 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:30.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:30.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:30.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:29 smithi069 bash[21528]: cluster 2024-04-02T04:48:28.986963+0000 mon.a (mon.0) 938 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-04-02T04:48:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:29 smithi069 bash[28904]: cluster 2024-04-02T04:48:28.986963+0000 mon.a (mon.0) 938 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-04-02T04:48:30.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:30.340 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:30.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:30.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:30.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:29 smithi098 bash[23790]: cluster 2024-04-02T04:48:28.986963+0000 mon.a (mon.0) 938 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-04-02T04:48:30.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:30.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:30.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:30.843 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:30.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:30.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:31.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:31.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:31.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:31.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:30 smithi098 bash[23790]: cluster 2024-04-02T04:48:29.822512+0000 mgr.y (mgr.14538) 172 : cluster 0 pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:31.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:30 smithi069 bash[21528]: cluster 2024-04-02T04:48:29.822512+0000 mgr.y (mgr.14538) 172 : cluster 0 pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:31.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:30 smithi069 bash[28904]: cluster 2024-04-02T04:48:29.822512+0000 mgr.y (mgr.14538) 172 : cluster 0 pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:31.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:31.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:31.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:31.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:31.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:31.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:31.873 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:31.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:31.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:32.104 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:32.125 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:32.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:32.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:32.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:32.508 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:32.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:32.638 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:32.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:32.756 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:48:32.756 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.14158.0:144 1.0 e23/21 1:30306672:devicehealth::main.db.0000000000000000:head v 23'102, mlcod=23'98)", 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:39:04.077725+0000", 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 568.67119419999995, 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0023017810000000001, 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.757 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077725+0000", 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077725+0000", 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077727+0000", 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.567e-06 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.758 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077784+0000", 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7079e-05 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077811+0000", 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7364000000000001e-05 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077831+0000", 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9366000000000001e-05 2024-04-02T04:48:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.077958+0000", 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000127028 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.078005+0000", 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6907000000000003e-05 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.079923+0000", 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0019177910000000001 2024-04-02T04:48:32.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:39:04.080027+0000", 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010467900000000001 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.14457.0:46 4.0 e82/81 4:05bf5b68:::notify.1:head v 82'2, mlcod=0'0)", 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:43:58.325197+0000", 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 274.42372266400002, 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034093999999999999, 2024-04-02T04:48:32.761 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325197+0000", 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325197+0000", 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325199+0000", 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7439999999999999e-06 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325213+0000", 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4746000000000001e-05 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325243+0000", 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9437e-05 2024-04-02T04:48:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325263+0000", 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9689000000000001e-05 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325347+0000", 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4850000000000002e-05 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.764 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.325376+0000", 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8462e-05 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.328504+0000", 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0031283050000000001 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:58.328606+0000", 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000102167 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.765 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:197 4.3 4:c609908c:::notify.5:head [watch ping cookie 94452570221952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:43:59.313180+0000", 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 273.43573930100001, 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0029600770000000002, 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.766 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 197 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.313180+0000", 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.313180+0000", 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.767 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.313183+0000", 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.706e-06 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.313196+0000", 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2737e-05 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.313227+0000", 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1276999999999999e-05 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.768 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.313255+0000", 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8245000000000001e-05 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.316003+0000", 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0027472460000000001 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.316073+0000", 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0580000000000005e-05 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.769 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:43:59.316140+0000", 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.7286000000000006e-05 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:214 3.12 3:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.358486+0000", 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.39043318099999, 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0029897499999999998, 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.770 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 214 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358486+0000", 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358486+0000", 2024-04-02T04:48:32.771 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358489+0000", 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0429999999999999e-06 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358504+0000", 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.456e-05 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358535+0000", 2024-04-02T04:48:32.772 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1069999999999999e-05 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358553+0000", 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7995999999999998e-05 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358641+0000", 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7558000000000004e-05 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.773 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.358875+0000", 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023401300000000001 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,7", 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359630+0000", 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000754826 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.360579+0000", 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00094981099999999995 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.774 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.360606+0000", 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6251e-05 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361319+0000", 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00071308000000000005 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361403+0000", 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4362000000000003e-05 2024-04-02T04:48:32.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361476+0000", 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3180000000000001e-05 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14457.0:201 4.3 4:c609908c:::notify.5:head [watch unwatch cookie 94002687224976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.359277+0000", 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.38964308200002, 2024-04-02T04:48:32.776 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024728699999999998, 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14457", 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1965820815", 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 201 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359277+0000", 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359277+0000", 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359278+0000", 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7150000000000001e-06 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359287+0000", 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6449999999999994e-06 2024-04-02T04:48:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359313+0000", 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6154000000000001e-05 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359344+0000", 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0571999999999998e-05 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359427+0000", 2024-04-02T04:48:32.779 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3415e-05 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359611+0000", 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018380800000000001 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 5,7", 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.359993+0000", 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00038236500000000002 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.360793+0000", 2024-04-02T04:48:32.780 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079952199999999999 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361603+0000", 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080985499999999995 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361631+0000", 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8736999999999999e-05 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361686+0000", 2024-04-02T04:48:32.781 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4802000000000001e-05 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.361749+0000", 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3280000000000004e-05 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:244 3.17 3:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.424077+0000", 2024-04-02T04:48:32.782 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.324842563, 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0025462079999999999, 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 244 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424077+0000", 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.783 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424077+0000", 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424078+0000", 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0699999999999999e-06 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424085+0000", 2024-04-02T04:48:32.784 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.7499999999999997e-06 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424099+0000", 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4286e-05 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424109+0000", 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0192e-05 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424185+0000", 2024-04-02T04:48:32.785 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5457999999999995e-05 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424284+0000", 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9532999999999997e-05 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,5", 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.424480+0000", 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019552600000000001 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.786 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.425403+0000", 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00092307999999999995 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.425882+0000", 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047911999999999998 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.426216+0000", 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00033387899999999998 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.787 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.426271+0000", 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5231e-05 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.426623+0000", 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000352083 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:276 3.1b 3:dabbc408:lc::lc.3:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.788 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.432338+0000", 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.31658166599999, 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003348059, 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 276 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432338+0000", 2024-04-02T04:48:32.789 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432338+0000", 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432339+0000", 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.065e-06 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.790 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432344+0000", 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7539999999999997e-06 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432353+0000", 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.738e-06 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432359+0000", 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.562e-06 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.791 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432398+0000", 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9119999999999998e-05 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432479+0000", 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1101999999999997e-05 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4,7", 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.432650+0000", 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017065 2024-04-02T04:48:32.792 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.433009+0000", 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00035857199999999998 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.434170+0000", 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0011619040000000001 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.435545+0000", 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001374629 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.793 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.435626+0000", 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0788e-05 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.435686+0000", 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0174999999999999e-05 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:290 3.1b 3:dabbc408:lc::lc.3:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.436668+0000", 2024-04-02T04:48:32.794 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.31225167700001, 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0030043940000000001, 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 290 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436668+0000", 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.795 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436668+0000", 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436669+0000", 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.61e-07 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436674+0000", 2024-04-02T04:48:32.796 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5600000000000001e-06 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436687+0000", 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2408999999999999e-05 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436692+0000", 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7989999999999999e-06 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.797 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436736+0000", 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3878999999999998e-05 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436786+0000", 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9196000000000001e-05 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4,7", 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.436944+0000", 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015884199999999999 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.798 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.437305+0000", 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00036093900000000001 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.439324+0000", 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002018855 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.439562+0000", 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023794699999999999 2024-04-02T04:48:32.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.439625+0000", 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2966999999999995e-05 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.439672+0000", 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7240000000000002e-05 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.800 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:306 3.1 3:856b9b26:reshard::reshard.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.443561+0000", 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.30535849500001, 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003722665, 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.801 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 306 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443561+0000", 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443561+0000", 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443562+0000", 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5499999999999997e-07 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443568+0000", 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.251e-06 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443577+0000", 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9339999999999995e-06 2024-04-02T04:48:32.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443585+0000", 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6170000000000001e-06 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443667+0000", 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2773000000000002e-05 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.443881+0000", 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000213332 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,4", 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.444270+0000", 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00038897699999999999 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.444977+0000", 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00070758300000000002 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.445822+0000", 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00084503599999999996 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.805 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.447198+0000", 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001375413 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.447244+0000", 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6165999999999998e-05 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.447284+0000", 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9827999999999999e-05 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.806 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:359 3.17 3:ee62d5d2:lc::lc.1:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.462492+0000", 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.28642757599999, 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022757509999999999, 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 359 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.807 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462492+0000", 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462492+0000", 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462494+0000", 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.652e-06 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462501+0000", 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7950000000000008e-06 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462517+0000", 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5628999999999999e-05 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462526+0000", 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2080000000000006e-06 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462577+0000", 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0293000000000001e-05 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462733+0000", 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000156391 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,5", 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.462966+0000", 2024-04-02T04:48:32.810 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023261899999999999 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.463855+0000", 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00088936799999999997 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.464124+0000", 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026940800000000001 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.464683+0000", 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00055812799999999997 2024-04-02T04:48:32.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.464720+0000", 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7697999999999997e-05 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.464768+0000", 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7562000000000003e-05 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:366 3.6 3:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.465376+0000", 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.28354392799997, 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002583278, 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 366 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.813 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465376+0000", 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465376+0000", 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.814 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465377+0000", 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1200000000000001e-07 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465382+0000", 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6470000000000003e-06 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465395+0000", 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3016e-05 2024-04-02T04:48:32.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465406+0000", 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0283e-05 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465469+0000", 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3177999999999997e-05 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.816 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465624+0000", 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015542600000000001 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,4", 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.465814+0000", 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018947500000000001 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.466346+0000", 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000532666 2024-04-02T04:48:32.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.466933+0000", 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058621699999999999 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.467888+0000", 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00095529999999999996 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.467917+0000", 2024-04-02T04:48:32.818 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9611000000000002e-05 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.467959+0000", 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1547e-05 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:630 3.1 3:8386e9da:lc::lc.16:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.551097+0000", 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.19782221499997, 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0029710560000000001, 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 630 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.820 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551097+0000", 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551097+0000", 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551099+0000", 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9910000000000001e-06 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.821 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551108+0000", 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9670000000000007e-06 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551126+0000", 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7224e-05 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551135+0000", 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7399999999999999e-06 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.822 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551202+0000", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6497999999999999e-05 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551314+0000", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000111808 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,4", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551504+0000", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019009100000000001 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.551995+0000", 2024-04-02T04:48:32.823 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00049117200000000001 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.552504+0000", 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00050874700000000004 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.553957+0000", 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001452853 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.554020+0000", 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3634999999999996e-05 2024-04-02T04:48:32.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.554068+0000", 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.833e-05 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:867 3.17 3:ee62d5d2:lc::lc.1:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.614714+0000", 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.13420516000002, 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024661459999999998, 2024-04-02T04:48:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 867 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614714+0000", 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.826 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614714+0000", 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614716+0000", 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1379999999999999e-06 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614722+0000", 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9699999999999996e-06 2024-04-02T04:48:32.827 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614734+0000", 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2568999999999999e-05 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614742+0000", 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3280000000000006e-06 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614886+0000", 2024-04-02T04:48:32.828 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014331399999999999 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.614973+0000", 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7480999999999998e-05 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,5", 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.615132+0000", 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000159161 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.615525+0000", 2024-04-02T04:48:32.829 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00039299499999999999 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.615792+0000", 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026647900000000002 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.617085+0000", 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0012930890000000001 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.830 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.617139+0000", 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4502000000000001e-05 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.617181+0000", 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1119999999999999e-05 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.831 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:906 3.1f 3:ffe20264:lc::lc.0:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.624633+0000", 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.12428616300002, 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034323819999999999, 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 906 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624633+0000", 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624633+0000", 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624634+0000", 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6000000000000003e-07 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.833 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624639+0000", 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3819999999999997e-06 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624647+0000", 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7029999999999995e-06 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624655+0000", 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9699999999999999e-06 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.834 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624734+0000", 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8891000000000001e-05 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.624828+0000", 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3821999999999997e-05 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 2,5", 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625290+0000", 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00046237600000000002 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.835 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625663+0000", 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00037285999999999998 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.626664+0000", 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010007390000000001 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.628002+0000", 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001338028 2024-04-02T04:48:32.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.628037+0000", 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5228999999999998e-05 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.628066+0000", 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8622000000000001e-05 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.837 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:911 3.1b 3:dabbc408:lc::lc.3:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.625969+0000", 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.12295083499998, 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0026750680000000001, 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.838 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 911 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625969+0000", 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625969+0000", 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.839 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625970+0000", 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0699999999999996e-07 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625982+0000", 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2254e-05 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.840 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.625995+0000", 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3216e-05 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.626003+0000", 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0490000000000004e-06 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.626068+0000", 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4678000000000007e-05 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.626158+0000", 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0064999999999999e-05 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4,7", 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.626344+0000", 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018595499999999999 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.627078+0000", 2024-04-02T04:48:32.842 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00073438599999999996 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.627128+0000", 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9529e-05 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.628557+0000", 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0014292949999999999 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.843 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.628603+0000", 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6272e-05 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.628644+0000", 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0462e-05 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.844 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:927 3.1 3:84d19476:::obj_delete_at_hint.0000000038:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:44:01.629747+0000", 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 271.11917232899998, 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0037391619999999999, 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 927 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.845 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.629747+0000", 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.629747+0000", 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.629748+0000", 2024-04-02T04:48:32.846 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8800000000000002e-07 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.629752+0000", 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3869999999999997e-06 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.629762+0000", 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8400000000000007e-06 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.847 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.629774+0000", 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1270000000000001e-05 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.630187+0000", 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000413416 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.630266+0000", 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9464999999999999e-05 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,4", 2024-04-02T04:48:32.848 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.630367+0000", 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000100246 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.630997+0000", 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00063013200000000004 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.631616+0000", 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00061920699999999998 2024-04-02T04:48:32.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.633264+0000", 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016481219999999999 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.633330+0000", 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5374000000000001e-05 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.850 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:44:01.633486+0000", 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015691500000000001 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(2.f,QREQUEST,e:110,from:4,reservation_nonce:4)", 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:46:02.898379+0000", 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 149.85054033899999, 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024813639999999998, 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.851 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898379+0000", 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898379+0000", 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898382+0000", 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3549999999999999e-06 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898391+0000", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2019999999999993e-06 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898411+0000", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0497e-05 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898426+0000", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.483e-05 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898520+0000", 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3326999999999996e-05 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.898530+0000", 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0164e-05 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.900814+0000", 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002284763 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.854 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.900826+0000", 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1959e-05 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:46:02.900861+0000", 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4267000000000002e-05 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(3.17,GRANT,e:115,from:5,reservation_nonce:0)", 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:47:08.327196+0000", 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 84.421723424999996, 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002308713, 2024-04-02T04:48:32.855 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.327196+0000", 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.327196+0000", 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.856 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.327198+0000", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.59e-06 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.327208+0000", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0485999999999999e-05 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.327230+0000", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1259000000000001e-05 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.857 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.327248+0000", 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8162999999999999e-05 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.329106+0000", 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00185859 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.329151+0000", 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4681999999999997e-05 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.858 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:47:08.329505+0000", 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00035394300000000001 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:1792 4.3 4:c609908c:::notify.5:head [watch ping cookie 94452570221952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:48:14.512092+0000", 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 18.236827443999999, 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061788939999999999, 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.859 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1792 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512092+0000", 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512092+0000", 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.860 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512095+0000", 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.729e-06 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512108+0000", 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3475999999999999e-05 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.861 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512142+0000", 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3608000000000001e-05 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512158+0000", 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6455000000000001e-05 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512290+0000", 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000131452 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.512300+0000", 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0344e-05 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.517996+0000", 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056957270000000003 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.518114+0000", 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011829599999999999 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:14.518271+0000", 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015680699999999999 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:1822 4.3 4:c609908c:::notify.5:head [watch ping cookie 94452570221952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:48:29.520044+0000", 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 3.2288760889999999, 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0026486109999999999, 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:48:32.864 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1822 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520044+0000", 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520044+0000", 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.865 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520046+0000", 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.525e-06 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520058+0000", 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1708e-05 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520084+0000", 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6196000000000001e-05 2024-04-02T04:48:32.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520098+0000", 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4396000000000001e-05 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520182+0000", 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3997000000000002e-05 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.520190+0000", 2024-04-02T04:48:32.867 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2320000000000004e-06 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.522546+0000", 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0023566500000000001 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.522625+0000", 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9212e-05 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:48:32.868 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:48:29.522692+0000", 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6694999999999998e-05 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:32.869 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:48:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:32.870 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:32.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:32.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- 172.21.15.69:0/1628706861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7768107220 msgr2=0x7f7768107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:32.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- 172.21.15.69:0/1628706861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7768107220 0x7f7768107600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7764009990 tx=0x7f7764009ca0 comp rx=0 tx=0).stop 2024-04-02T04:48:32.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- 172.21.15.69:0/1628706861 shutdown_connections 2024-04-02T04:48:32.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- 172.21.15.69:0/1628706861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7768101b60 0x7f776810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:32.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- 172.21.15.69:0/1628706861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7768107220 0x7f7768107600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:32.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- 172.21.15.69:0/1628706861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77681011a0 0x7f7768101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- 172.21.15.69:0/1628706861 >> 172.21.15.69:0/1628706861 conn(0x7f77680771e0 msgr2=0x7f77680775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- 172.21.15.69:0/1628706861 shutdown_connections 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- 172.21.15.69:0/1628706861 wait complete. 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 librados: starting msgr at 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 librados: starting objecter 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 Processor -- start 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- start start 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 librados: setting wanted keys 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 librados: calling monclient init 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 0x7f7768196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7768101b60 0x7f77681973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7768107220 0x7f776819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f77681109c0 con 0x7f77681011a0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776d360700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 0x7f7768196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7768110840 con 0x7f7768107220 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776d360700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 0x7f7768196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56840/0 (socket says 172.21.15.69:56840) 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776d360700 1 -- 172.21.15.69:0/2529583147 learned_addr learned my addr 172.21.15.69:0/2529583147 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776f5c4700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7768110b40 con 0x7f7768101b60 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.815+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7768101b60 0x7f77681973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7768107220 0x7f776819b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776d360700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7768101b60 msgr2=0x7f77681973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7768101b60 0x7f77681973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776d360700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7768107220 msgr2=0x7f776819b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7768107220 0x7f776819b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776d360700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776819beb0 con 0x7f77681011a0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 0x7f7768196ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7764000c00 tx=0x7f7764009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 10 client.?.objecter ms_handle_connect 0x7f77681011a0 2024-04-02T04:48:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764027070 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7764003fc0 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776f5c4700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776819c140 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764022c50 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776f5c4700 10 client.14793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776f5c4700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7768069f70 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776f5c4700 1 librados: init done 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f776402a070 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f776804ff00 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f77540717f0 0x7f7754073cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(126..126 src has 74..126) ==== 7537+0+0 (secure 0 0 0) 0x7f776408fec0 con 0x7f77681011a0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 osd_map(126..126 src has 74..126) 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 3 client.14793.objecter handle_osd_map got epochs [126,126] > 0 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 3 client.14793.objecter handle_osd_map decoding full epoch 126 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 20 client.14793.objecter dump_active .. 0 homeless 2024-04-02T04:48:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v126) ==== 54+0+16 (secure 0 0 0) 0x7f7764093020 con 0x7f77681011a0 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f77540717f0 0x7f7754073cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 10 client.14793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 1 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f776806abf0 0x7f7768108420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 1 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 0 1 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 4 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 10 client.14793.objecter _send_command 1 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.819+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7768108960 con 0x7f776806abf0 2024-04-02T04:48:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f776806abf0 0x7f7768108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f77540717f0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f77540717f0 0x7f7754073cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77680778c0 tx=0x7f775800a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f776806abf0 0x7f7768108420 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f776806abf0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7768108960 con 0x7f776806abf0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 1: 0 ) 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 10 client.14793.objecter _finish_command 1 = system:0 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 4 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.823+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 0 1 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 10 client.14793.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 2 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 2 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f7768198200 osd=0 3 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 0 2 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 4 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 10 client.14793.objecter _send_command 2 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.839+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7768108dc0 con 0x7f776806abf0 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7768108dc0 con 0x7f776806abf0 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 10 client.14793.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 4 2024-04-02T04:48:32.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 0 2 2024-04-02T04:48:32.880 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:48:32.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 10 client.14793.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f776806a7e0 0x7f77681a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 1 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 4 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 10 client.14793.objecter _send_command 3 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7768063360 con 0x7f776806a7e0 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f776806a7e0 0x7f77681a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f776806a7e0 0x7f77681a8bc0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f776806a7e0 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7768063360 con 0x7f776806a7e0 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 3: 0 ) 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 10 client.14793.objecter _finish_command 3 = system:0 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 4 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.843+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 1 3 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 10 client.14793.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 4 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 4 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f776806a670 osd=1 3 2024-04-02T04:48:32.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 1 4 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 4 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 10 client.14793.objecter _send_command 4 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681a9100 con 0x7f776806a7e0 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681a9100 con 0x7f776806a7e0 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 10 client.14793.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 4 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 1 4 2024-04-02T04:48:32.883 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:48:32.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 10 client.14793.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:32.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 5 2024-04-02T04:48:32.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7768062a40 0x7f77681ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 5 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 2 5 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 4 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 10 client.14793.objecter _send_command 5 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f77681a6710 con 0x7f7768062a40 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7768062a40 0x7f77681ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f7768062a40 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7768062a40 0x7f77681ac4a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77681a6710 con 0x7f7768062a40 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 5: 0 ) 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 10 client.14793.objecter _finish_command 5 = system:0 2024-04-02T04:48:32.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 4 2024-04-02T04:48:32.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.863+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 2 5 2024-04-02T04:48:32.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 10 client.14793.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 6 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 6 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77680628d0 osd=2 3 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 2 6 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 4 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 10 client.14793.objecter _send_command 6 2024-04-02T04:48:32.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681ac9e0 con 0x7f7768062a40 2024-04-02T04:48:32.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681ac9e0 con 0x7f7768062a40 2024-04-02T04:48:32.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:32.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f775e7fc700 10 client.14793.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:32.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 4 2024-04-02T04:48:32.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 2 6 2024-04-02T04:48:32.899 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:48:32.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 10 client.14793.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:32.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 7 2024-04-02T04:48:32.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f77681a9ff0 0x7f77681afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 7 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 3 7 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 4 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 10 client.14793.objecter _send_command 7 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f77681a5ff0 con 0x7f77681a9ff0 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f77681a9ff0 0x7f77681afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f77681a9ff0 2024-04-02T04:48:32.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.899+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f77681a9ff0 0x7f77681afd80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.903+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77681a5ff0 con 0x7f77681a9ff0 2024-04-02T04:48:32.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.903+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 7: 0 ) 2024-04-02T04:48:32.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.903+0000 7f775e7fc700 10 client.14793.objecter _finish_command 7 = system:0 2024-04-02T04:48:32.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.903+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 4 2024-04-02T04:48:32.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.903+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 3 7 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 10 client.14793.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 8 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 8 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681a5c80 osd=3 3 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 3 8 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 4 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 10 client.14793.objecter _send_command 8 2024-04-02T04:48:32.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681a9560 con 0x7f77681a9ff0 2024-04-02T04:48:32.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681a9560 con 0x7f77681a9ff0 2024-04-02T04:48:32.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:32.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f775e7fc700 10 client.14793.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:32.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 4 2024-04-02T04:48:32.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 3 8 2024-04-02T04:48:32.927 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 10 client.14793.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 9 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f77681ad8d0 0x7f77681b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 9 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 4 9 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 4 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 10 client.14793.objecter _send_command 9 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f77681b63c0 con 0x7f77681ad8d0 2024-04-02T04:48:32.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.927+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f77681ad8d0 0x7f77681b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f77681ad8d0 2024-04-02T04:48:32.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f77681ad8d0 0x7f77681b1440 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77681b63c0 con 0x7f77681ad8d0 2024-04-02T04:48:32.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 9: 0 ) 2024-04-02T04:48:32.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f775e7fc700 10 client.14793.objecter _finish_command 9 = system:0 2024-04-02T04:48:32.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 4 2024-04-02T04:48:32.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.931+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 4 9 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 10 client.14793.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 10 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 10 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681a9560 osd=4 3 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 4 10 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 4 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 10 client.14793.objecter _send_command 10 2024-04-02T04:48:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.959+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681a97d0 con 0x7f77681ad8d0 2024-04-02T04:48:32.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681a97d0 con 0x7f77681ad8d0 2024-04-02T04:48:32.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:32.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 10 client.14793.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:32.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 4 2024-04-02T04:48:32.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 4 10 2024-04-02T04:48:32.960 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 10 client.14793.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 11 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f77681b0a90 0x7f77681b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 11 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 5 11 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 4 2024-04-02T04:48:32.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 10 client.14793.objecter _send_command 11 2024-04-02T04:48:32.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f77681b95f0 con 0x7f77681b0a90 2024-04-02T04:48:32.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f77681b0a90 0x7f77681b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:32.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f77681b0a90 2024-04-02T04:48:32.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f776cb5f700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f77681b0a90 0x7f77681b0ef0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:32.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77681b95f0 con 0x7f77681b0a90 2024-04-02T04:48:32.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 11: 0 ) 2024-04-02T04:48:32.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 10 client.14793.objecter _finish_command 11 = system:0 2024-04-02T04:48:32.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 4 2024-04-02T04:48:32.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.963+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 5 11 2024-04-02T04:48:32.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 10 client.14793.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 12 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 12 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681a97d0 osd=5 3 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 5 12 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 4 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 10 client.14793.objecter _send_command 12 2024-04-02T04:48:32.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:32.995+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681b6cd0 con 0x7f77681b0a90 2024-04-02T04:48:32.997 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681b6cd0 con 0x7f77681b0a90 2024-04-02T04:48:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f775e7fc700 10 client.14793.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 4 2024-04-02T04:48:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 5 12 2024-04-02T04:48:33.001 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 10 client.14793.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 13 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f77681b8bc0 0x7f77681b9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 13 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 6 13 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 4 2024-04-02T04:48:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 10 client.14793.objecter _send_command 13 2024-04-02T04:48:33.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f77681bc990 con 0x7f77681b8bc0 2024-04-02T04:48:33.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f77681b8bc0 0x7f77681b9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:33.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f77681b8bc0 2024-04-02T04:48:33.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.003+0000 7f776db61700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f77681b8bc0 0x7f77681b9020 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:33.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.007+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77681bc990 con 0x7f77681b8bc0 2024-04-02T04:48:33.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.007+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 13: 0 ) 2024-04-02T04:48:33.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.007+0000 7f775e7fc700 10 client.14793.objecter _finish_command 13 = system:0 2024-04-02T04:48:33.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.007+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 4 2024-04-02T04:48:33.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.007+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 6 13 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 10 client.14793.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 14 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 14 2024-04-02T04:48:33.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681b6cd0 osd=6 3 2024-04-02T04:48:33.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 6 14 2024-04-02T04:48:33.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 4 2024-04-02T04:48:33.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 10 client.14793.objecter _send_command 14 2024-04-02T04:48:33.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.023+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681bb810 con 0x7f77681b8bc0 2024-04-02T04:48:33.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681bb810 con 0x7f77681b8bc0 2024-04-02T04:48:33.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:33.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f775e7fc700 10 client.14793.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:33.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 4 2024-04-02T04:48:33.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 6 14 2024-04-02T04:48:33.027 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 10 client.14793.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 15 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f77681bbee0 0x7f77681bc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 15 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 7 15 2024-04-02T04:48:33.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 4 2024-04-02T04:48:33.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 10 client.14793.objecter _send_command 15 2024-04-02T04:48:33.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f77681bfd50 con 0x7f77681bbee0 2024-04-02T04:48:33.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.027+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f77681bbee0 0x7f77681bc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:33.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[28904]: cluster 2024-04-02T04:48:31.822847+0000 mgr.y (mgr.14538) 173 : cluster 0 pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:33.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[28904]: cluster 2024-04-02T04:48:32.478203+0000 osd.2 (osd.2) 57 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:33.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[28904]: cluster 2024-04-02T04:48:32.478279+0000 osd.2 (osd.2) 58 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:33.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[28904]: audit 2024-04-02T04:48:32.822924+0000 mon.a (mon.0) 939 : audit 0 from='client.? 172.21.15.69:0/2529583147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:33.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[21528]: cluster 2024-04-02T04:48:31.822847+0000 mgr.y (mgr.14538) 173 : cluster 0 pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:33.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[21528]: cluster 2024-04-02T04:48:32.478203+0000 osd.2 (osd.2) 57 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:33.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[21528]: cluster 2024-04-02T04:48:32.478279+0000 osd.2 (osd.2) 58 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:33.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:33 smithi069 bash[21528]: audit 2024-04-02T04:48:32.822924+0000 mon.a (mon.0) 939 : audit 0 from='client.? 172.21.15.69:0/2529583147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:33.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f775e7fc700 10 client.14793.objecter ms_handle_connect 0x7f77681bbee0 2024-04-02T04:48:33.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f776d360700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f77681bbee0 0x7f77681bc300 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:33.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77681bfd50 con 0x7f77681bbee0 2024-04-02T04:48:33.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 15: 0 ) 2024-04-02T04:48:33.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f775e7fc700 10 client.14793.objecter _finish_command 15 = system:0 2024-04-02T04:48:33.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 4 2024-04-02T04:48:33.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.031+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 7 15 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 10 client.14793.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign -1 16 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 20 client.14793.objecter _get_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 15 client.14793.objecter _session_command_op_remove -1 16 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 20 client.14793.objecter get_session s=0x7f77681bb810 osd=7 3 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 15 client.14793.objecter _session_command_op_assign 7 16 2024-04-02T04:48:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 4 2024-04-02T04:48:33.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 10 client.14793.objecter _send_command 16 2024-04-02T04:48:33.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.063+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77681b6e80 con 0x7f77681bbee0 2024-04-02T04:48:33.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.067+0000 7f775e7fc700 1 -- 172.21.15.69:0/2529583147 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77681b6e80 con 0x7f77681bbee0 2024-04-02T04:48:33.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.067+0000 7f775e7fc700 10 client.14793.objecter ms_dispatch 0x7f7768001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:33.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.067+0000 7f775e7fc700 10 client.14793.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:33.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.067+0000 7f775e7fc700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 4 2024-04-02T04:48:33.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.067+0000 7f775e7fc700 15 client.14793.objecter _session_command_op_remove 7 16 2024-04-02T04:48:33.068 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 librados: watch_flush enter 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 librados: watch_flush exit 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.0 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f776806abf0 msgr2=0x7f7768108420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f776806abf0 0x7f7768108420 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f7768198200 osd=0 1 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.1 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f776806a7e0 msgr2=0x7f77681a8bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f776806a7e0 0x7f77681a8bc0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f776806a670 osd=1 1 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.2 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7768062a40 msgr2=0x7f77681ac4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7768062a40 0x7f77681ac4a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77680628d0 osd=2 1 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.3 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f77681a9ff0 msgr2=0x7f77681afd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f77681a9ff0 0x7f77681afd80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a5c80 osd=3 1 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.4 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f77681ad8d0 msgr2=0x7f77681b1440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f77681ad8d0 0x7f77681b1440 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a9560 osd=4 1 2024-04-02T04:48:33.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.5 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f77681b0a90 msgr2=0x7f77681b0ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f776db61700 1 -- 172.21.15.69:0/2529583147 reap_dead start 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f77681b0a90 0x7f77681b0ef0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681a97d0 osd=5 1 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.6 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f77681b8bc0 msgr2=0x7f77681b9020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f77681b8bc0 0x7f77681b9020 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681b6cd0 osd=6 1 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter close_session for osd.7 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f776db61700 1 -- 172.21.15.69:0/2529583147 reap_dead start 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f77681bbee0 msgr2=0x7f77681bc300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f77681bbee0 0x7f77681bc300 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter put_session s=0x7f77681bb810 osd=7 1 2024-04-02T04:48:33.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 20 client.14793.objecter shutdown clearing up homeless session... 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 10 client.14793.objecter successfully canceled tick 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f776db61700 1 -- 172.21.15.69:0/2529583147 reap_dead start 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f77540717f0 msgr2=0x7f7754073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f77540717f0 0x7f7754073cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77680778c0 tx=0x7f775800a380 comp rx=0 tx=0).stop 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 msgr2=0x7f7768196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 0x7f7768196ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7764000c00 tx=0x7f7764009f50 comp rx=0 tx=0).stop 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 shutdown_connections 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77681011a0 0x7f7768196ea0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 --2- 172.21.15.69:0/2529583147 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f77540717f0 0x7f7754073cb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 >> 172.21.15.69:0/2529583147 conn(0x7f77680771e0 msgr2=0x7f7768104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 shutdown_connections 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 -- 172.21.15.69:0/2529583147 wait complete. 2024-04-02T04:48:33.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:33.071+0000 7f7753fff700 1 librados: shutdown 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:48:33.073 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:48:33.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:33.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:33.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:48:32 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:48:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:33 smithi098 bash[23790]: cluster 2024-04-02T04:48:31.822847+0000 mgr.y (mgr.14538) 173 : cluster 0 pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:48:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:33 smithi098 bash[23790]: cluster 2024-04-02T04:48:32.478203+0000 osd.2 (osd.2) 57 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:33 smithi098 bash[23790]: cluster 2024-04-02T04:48:32.478279+0000 osd.2 (osd.2) 58 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:33 smithi098 bash[23790]: audit 2024-04-02T04:48:32.822924+0000 mon.a (mon.0) 939 : audit 0 from='client.? 172.21.15.69:0/2529583147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:33.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:33.263 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:33.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:33.385 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:33.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:33.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:33.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:33.771 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:33.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:33.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:33.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:34.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:34.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:34.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:34 smithi098 bash[23790]: audit 2024-04-02T04:48:32.939987+0000 mgr.y (mgr.14538) 174 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:34 smithi098 bash[23790]: cluster 2024-04-02T04:48:32.999911+0000 mon.a (mon.0) 940 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-04-02T04:48:34.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:34.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:34 smithi069 bash[21528]: audit 2024-04-02T04:48:32.939987+0000 mgr.y (mgr.14538) 174 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:34.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:34 smithi069 bash[21528]: cluster 2024-04-02T04:48:32.999911+0000 mon.a (mon.0) 940 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-04-02T04:48:34.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:34 smithi069 bash[28904]: audit 2024-04-02T04:48:32.939987+0000 mgr.y (mgr.14538) 174 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:34.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:34 smithi069 bash[28904]: cluster 2024-04-02T04:48:32.999911+0000 mon.a (mon.0) 940 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-04-02T04:48:34.275 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:34.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:34.401 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:34.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:34.528 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:34.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:34.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:34.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:34.787 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:34.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:34.911 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:34.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:35.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:35.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:35.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:35 smithi069 bash[21528]: cluster 2024-04-02T04:48:33.823197+0000 mgr.y (mgr.14538) 175 : cluster 0 pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:35.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:35 smithi069 bash[28904]: cluster 2024-04-02T04:48:33.823197+0000 mgr.y (mgr.14538) 175 : cluster 0 pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:35.167 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:35.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:35.281 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:35 smithi098 bash[23790]: cluster 2024-04-02T04:48:33.823197+0000 mgr.y (mgr.14538) 175 : cluster 0 pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:35.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:35.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:35.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:35.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:35.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-02T04:48:35.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:35.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:35.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:35.945 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:35.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:36.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:36.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:36.195 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:36.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:36.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:36.456 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:36.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:36.589 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:36.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:36.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9559a700 1 -- 172.21.15.69:0/2526902384 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88003aa0 con 0x7fdb981071e0 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 -- 172.21.15.69:0/2526902384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb981071e0 msgr2=0x7fdb981075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/2526902384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb981071e0 0x7fdb981075c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009990 tx=0x7fdb88009ca0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 -- 172.21.15.69:0/2526902384 shutdown_connections 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/2526902384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb98101b30 0x7fdb9810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/2526902384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb981071e0 0x7fdb981075c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/2526902384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb98101170 0x7fdb981015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.783+0000 7fdb9cbae700 1 -- 172.21.15.69:0/2526902384 >> 172.21.15.69:0/2526902384 conn(0x7fdb98078110 msgr2=0x7fdb98078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:36.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 -- 172.21.15.69:0/2526902384 shutdown_connections 2024-04-02T04:48:36.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 -- 172.21.15.69:0/2526902384 wait complete. 2024-04-02T04:48:36.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 librados: starting msgr at 2024-04-02T04:48:36.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 librados: starting objecter 2024-04-02T04:48:36.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 Processor -- start 2024-04-02T04:48:36.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 -- start start 2024-04-02T04:48:36.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 librados: setting wanted keys 2024-04-02T04:48:36.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 librados: calling monclient init 2024-04-02T04:48:36.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 0x7fdb9819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb98101b30 0x7fdb9819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb981071e0 0x7fdb9819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb98111b00 con 0x7fdb98101b30 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb98111980 con 0x7fdb981071e0 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9cbae700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdb98111c80 con 0x7fdb98101170 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 0x7fdb9819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 0x7fdb9819a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47866/0 (socket says 172.21.15.69:47866) 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 -- 172.21.15.69:0/3090750948 learned_addr learned my addr 172.21.15.69:0/3090750948 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb96d9d700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb981071e0 0x7fdb9819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 -- 172.21.15.69:0/3090750948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb981071e0 msgr2=0x7fdb9819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:36.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb981071e0 0x7fdb9819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 -- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb98101b30 msgr2=0x7fdb9819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:36.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb98101b30 0x7fdb9819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.787+0000 7fdb9659c700 1 -- 172.21.15.69:0/3090750948 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9819f2d0 con 0x7fdb98101170 2024-04-02T04:48:36.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb9659c700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 0x7fdb9819a2c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800b5c0 tx=0x7fdb880047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:36.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb877fe700 10 client.?.objecter ms_handle_connect 0x7fdb98101170 2024-04-02T04:48:36.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb877fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88027070 con 0x7fdb98101170 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fdb8800fbb0 con 0x7fdb98101170 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88022be0 con 0x7fdb98101170 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9819f560 con 0x7fdb98101170 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb9cbae700 10 client.24683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb981a6e40 con 0x7fdb98101170 2024-04-02T04:48:36.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb9cbae700 1 librados: init done 2024-04-02T04:48:36.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.791+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb9806a150 con 0x7fdb98101170 2024-04-02T04:48:36.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fdb8800f730 con 0x7fdb98101170 2024-04-02T04:48:36.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdb8006f5e0 0x7fdb80071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(127..127 src has 74..127) ==== 7537+0+0 (secure 0 0 0) 0x7fdb8808fc30 con 0x7fdb98101170 2024-04-02T04:48:36.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 10 client.24683.objecter ms_dispatch 0x7fdb98001550 osd_map(127..127 src has 74..127) 2024-04-02T04:48:36.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 3 client.24683.objecter handle_osd_map got epochs [127,127] > 0 2024-04-02T04:48:36.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 3 client.24683.objecter handle_osd_map decoding full epoch 127 2024-04-02T04:48:36.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.795+0000 7fdb877fe700 20 client.24683.objecter dump_active .. 0 homeless 2024-04-02T04:48:36.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.799+0000 7fdb95d9b700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdb8006f5e0 0x7fdb80071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.799+0000 7fdb95d9b700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdb8006f5e0 0x7fdb80071aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdb9810e460 tx=0x7fdb8c006d20 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:36.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.799+0000 7fdb877fe700 10 client.24683.objecter ms_handle_connect 0x7fdb8006f5e0 2024-04-02T04:48:36.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.799+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdb88030030 con 0x7fdb98101170 2024-04-02T04:48:36.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3144035320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101420 msgr2=0x7f25b4101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 --2- 172.21.15.69:0/3144035320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101420 0x7f25b4101800 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f25b00099c0 tx=0x7f25b0009cd0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3144035320 shutdown_connections 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 --2- 172.21.15.69:0/3144035320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 --2- 172.21.15.69:0/3144035320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101420 0x7f25b4101800 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 --2- 172.21.15.69:0/3144035320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101dd0 0x7f25b4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3144035320 >> 172.21.15.69:0/3144035320 conn(0x7f25b4078240 msgr2=0x7f25b4078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3144035320 shutdown_connections 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3144035320 wait complete. 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 librados: starting msgr at 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 librados: starting objecter 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 Processor -- start 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 -- start start 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 librados: setting wanted keys 2024-04-02T04:48:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.915+0000 7f25bbbbe700 1 librados: calling monclient init 2024-04-02T04:48:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101420 0x7f25b41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 0x7f25b41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25b9159700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 0x7f25b41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25b9159700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 0x7f25b41986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47880/0 (socket says 172.21.15.69:47880) 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25b995a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101420 0x7f25b41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56878/0 (socket says 172.21.15.69:56878) 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25b9159700 1 -- 172.21.15.69:0/3363135494 learned_addr learned my addr 172.21.15.69:0/3363135494 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25b4111bd0 con 0x7f25b4106500 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b4111a50 con 0x7f25b4101420 2024-04-02T04:48:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f25b4111d50 con 0x7f25b4101dd0 2024-04-02T04:48:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 -- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 msgr2=0x7f25b41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 0x7f25b41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 -- 172.21.15.69:0/3363135494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101420 msgr2=0x7f25b41981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101420 0x7f25b41981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b419d1c0 con 0x7f25b4106500 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25b995a700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101420 0x7f25b41981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25b9159700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 0x7f25b41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25ba15b700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b419ca80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00c340 tx=0x7f25ac00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25aaffd700 10 client.?.objecter ms_handle_connect 0x7f25b4106500 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25aaffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac0090d0 con 0x7f25b4106500 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f25ac009270 con 0x7f25b4106500 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac004840 con 0x7f25b4106500 2024-04-02T04:48:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.919+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b419d4b0 con 0x7f25b4106500 2024-04-02T04:48:36.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25bbbbe700 10 client.14802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:36.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25bbbbe700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b41a4d30 con 0x7f25b4106500 2024-04-02T04:48:36.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25bbbbe700 1 librados: init done 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f25ac012050 con 0x7f25b4106500 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f259c005120 con 0x7f25b4106500 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25a006f630 0x7f25a0071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25b995a700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25a006f630 0x7f25a0071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(127..127 src has 74..127) ==== 7537+0+0 (secure 0 0 0) 0x7f25ac0587e0 con 0x7f25b4106500 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 10 client.14802.objecter ms_dispatch 0x7f25b4001550 osd_map(127..127 src has 74..127) 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 3 client.14802.objecter handle_osd_map got epochs [127,127] > 0 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 3 client.14802.objecter handle_osd_map decoding full epoch 127 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 20 client.14802.objecter dump_active .. 0 homeless 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25aaffd700 10 client.14802.objecter ms_handle_connect 0x7f25a006f630 2024-04-02T04:48:36.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.923+0000 7f25b995a700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25a006f630 0x7f25a0071af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25b00037a0 tx=0x7f25b0019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:36.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:36.935+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f25ac010040 con 0x7f25b4106500 2024-04-02T04:48:36.952 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:36.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:36.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:37.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:37.096 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:37.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:37.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:37.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.211+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fdb98102890 con 0x7fdb98101170 2024-04-02T04:48:37.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.215+0000 7fdb877fe700 1 -- 172.21.15.69:0/3090750948 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v127) ==== 81+0+44946 (secure 0 0 0) 0x7fdb88013070 con 0x7fdb98101170 2024-04-02T04:48:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:48:37.216 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:48:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 127, 2024-04-02T04:48:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:48:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: "created": "2024-04-02T04:33:43.348385+0000", 2024-04-02T04:48:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: "modified": "2024-04-02T04:48:32.994640+0000", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "last_up_change": "2024-04-02T04:43:12.908579+0000", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "last_in_change": "2024-04-02T04:42:47.064987+0000", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_num": 5806080, 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_set": [ 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "nodeep-scrub", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "noscrub", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "pglog_hardlimit", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "purged_snapdirs", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_deletes", 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "sortbitwise" 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:48:37.217 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_version": 18, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "full_ratio": 0.94999998807907104, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_snapshot": "", 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_max": 6, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd": 8, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "require_min_compat_client": "luminous", 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "min_compat_client": "luminous", 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "require_osd_release": "squid", 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_crimson": false, 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: "pools": [ 2024-04-02T04:48:37.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 1, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".mgr", 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:38:57.671714+0000", 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:48:37.219 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 1, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 1, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 1, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 1, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 1, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:48:37.220 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "23", 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:48:37.221 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:48:37.222 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:48:37.223 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_max": 32, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_min": 1 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": {} 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 7.8899998664855957, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 7.8899998664855957, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.37999999523162842, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 3, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 3, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:48:37.224 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 2, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".rgw.root", 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:52.801763+0000", 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:48:37.225 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:48:37.226 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "79", 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:48:37.227 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:48:37.228 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:48:37.229 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.5, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.5, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 1, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:48:37.230 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 3, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.log", 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:54.772439+0000", 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:48:37.231 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:48:37.232 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "81", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:48:37.233 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:48:37.234 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:48:37.235 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.5, 2024-04-02T04:48:37.236 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.5, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 1, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 4, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.control", 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:56.696111+0000", 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:48:37.237 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:48:37.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "83", 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:48:37.239 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:48:37.240 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:48:37.241 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.25, 2024-04-02T04:48:37.242 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.25, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 1, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.25, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.25, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 5, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "datapool", 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:57.254108+0000", 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:48:37.243 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 3, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 3, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 3, 2024-04-02T04:48:37.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 3, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 3, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "89", 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:48:37.245 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 2, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 89, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:48:37.246 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:48:37.247 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd": {} 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.6500000953674316, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.6500000953674316, 2024-04-02T04:48:37.248 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 6, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.meta", 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:58.846023+0000", 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:48:37.249 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:48:37.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "85", 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:48:37.251 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:48:37.252 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:48:37.253 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_bias": 4 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:48:37.254 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.75, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.75, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 1, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.255 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b", 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 8, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 107, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6802", 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6803", 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.257 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6804", 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6805", 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.258 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6808", 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6809", 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.259 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6806", 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6807", 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6803/1087500595", 2024-04-02T04:48:37.260 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6805/1087500595", 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6809/1087500595", 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6807/1087500595", 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a", 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.261 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 13, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 107, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6810", 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6811", 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6812", 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6813", 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6816", 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.264 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6817", 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6814", 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6815", 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6811/2651666015", 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6813/2651666015", 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6817/2651666015", 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6815/2651666015", 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7", 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 19, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 107, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.267 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6818", 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6819", 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6820", 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6821", 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6824", 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6825", 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6822", 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6823", 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:48:37.271 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6819/619998764", 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6821/619998764", 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6825/619998764", 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6823/619998764", 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.272 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed", 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 29, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 107, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.273 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6826", 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6827", 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.274 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6828", 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6829", 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.275 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6832", 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6833", 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6830", 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6831", 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6827/207593208", 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6829/207593208", 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6833/207593208", 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6831/207593208", 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852", 2024-04-02T04:48:37.278 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 36, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 83, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.279 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6800", 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6801", 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6802", 2024-04-02T04:48:37.280 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6803", 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6806", 2024-04-02T04:48:37.281 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6807", 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.282 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6804", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6805", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6801/2065947829", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6803/2065947829", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6807/2065947829", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6805/2065947829", 2024-04-02T04:48:37.283 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe", 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 47, 2024-04-02T04:48:37.284 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 83, 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6808", 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6809", 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.285 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6810", 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6811", 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.286 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6814", 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6815", 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.287 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6812", 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6813", 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:48:37.288 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6809/219578185", 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6811/219578185", 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6815/219578185", 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6813/219578185", 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:48:37.289 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727", 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 58, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 81, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.290 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6816", 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6817", 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.291 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6818", 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6819", 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6822", 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6823", 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6820", 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6821", 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6817/1226172499", 2024-04-02T04:48:37.294 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6819/1226172499", 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6823/1226172499", 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6821/1226172499", 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3", 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:48:37.295 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 69, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 83, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6824", 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6825", 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6826", 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6827", 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6830", 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6831", 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6828", 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.299 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6829", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6825/148009079", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6827/148009079", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6831/148009079", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6829/148009079", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.300 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_xinfo": [ 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:48:20.072120+0000", 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:48:37.301 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:47:35.005517+0000", 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.302 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:48:32.478283+0000", 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:47:20.995748+0000", 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.303 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:47:20.289928+0000", 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.304 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:47:35.205666+0000", 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:48:28.047749+0000", 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:48:13.738528+0000", 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap": [], 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_items": [ 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.306 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "2.9", 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 7, 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.6", 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 5 2024-04-02T04:48:37.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.f", 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.17", 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_primaries": [], 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_temp": [], 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_temp": [], 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklist": { 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2738867930": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:48:37.309 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1798147282": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/52352690": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2928651991": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2763215770": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/3657409442": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/879855771": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1025808168": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1168952113": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1892956550": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:48:37.310 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/54578327": "2024-04-03T04:44:51.699278+0000" 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "range_blocklist": {}, 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profiles": { 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "default": { 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van" 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps_queue": [], 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "new_removed_snaps": [], 2024-04-02T04:48:37.311 INFO:teuthology.orchestra.run.smithi069.stdout: "new_purged_snaps": [], 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_node_flags": {}, 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "device_class_flags": {}, 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode": { 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_enabled": false, 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_bucket_count": 0, 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "degraded_stretch_mode": 0, 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "recovering_stretch_mode": 0, 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_bucket": 0 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:48:37.312 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:48:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:37.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 10 librados: watch_flush enter 2024-04-02T04:48:37.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 10 librados: watch_flush exit 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 20 client.24683.objecter shutdown clearing up homeless session... 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 10 client.24683.objecter successfully canceled tick 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdb8006f5e0 msgr2=0x7fdb80071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdb8006f5e0 0x7fdb80071aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdb9810e460 tx=0x7fdb8c006d20 comp rx=0 tx=0).stop 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 msgr2=0x7fdb9819a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 0x7fdb9819a2c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800b5c0 tx=0x7fdb880047b0 comp rx=0 tx=0).stop 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 shutdown_connections 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdb8006f5e0 0x7fdb80071aa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb981071e0 0x7fdb9819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb98101170 0x7fdb9819a2c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 --2- 172.21.15.69:0/3090750948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb98101b30 0x7fdb9819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 >> 172.21.15.69:0/3090750948 conn(0x7fdb98078110 msgr2=0x7fdb9810e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 shutdown_connections 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 -- 172.21.15.69:0/3090750948 wait complete. 2024-04-02T04:48:37.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.223+0000 7fdb9cbae700 1 librados: shutdown 2024-04-02T04:48:37.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:37 smithi069 bash[21528]: cluster 2024-04-02T04:48:35.823620+0000 mgr.y (mgr.14538) 176 : cluster 0 pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:37.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:37 smithi069 bash[28904]: cluster 2024-04-02T04:48:35.823620+0000 mgr.y (mgr.14538) 176 : cluster 0 pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:37.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:37.335+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f259c0056e0 con 0x7f25b4106500 2024-04-02T04:48:37.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:37.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:37 smithi098 bash[23790]: cluster 2024-04-02T04:48:35.823620+0000 mgr.y (mgr.14538) 176 : cluster 0 pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:37.488 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:37.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:37.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:37.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:37.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:37.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:48:37.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:37.746 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:37.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:37.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:37.874 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:37.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:37.918 INFO:tasks.thrashosds.thrasher:j is {'epoch': 127, 'fsid': 'f978400e-f0a9-11ee-b647-cb9ed24678a4', 'created': '2024-04-02T04:33:43.348385+0000', 'modified': '2024-04-02T04:48:32.994640+0000', 'last_up_change': '2024-04-02T04:43:12.908579+0000', 'last_in_change': '2024-04-02T04:42:47.064987+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-02T04:43:52.801763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-02T04:43:54.772439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-02T04:43:56.696111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-04-02T04:43:57.254108+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-02T04:43:58.846023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1087500595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1087500595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 1087500595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1087500595}]}, 'public_addr': '172.21.15.69:6803/1087500595', 'cluster_addr': '172.21.15.69:6805/1087500595', 'heartbeat_back_addr': '172.21.15.69:6809/1087500595', 'heartbeat_front_addr': '172.21.15.69:6807/1087500595', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'afb71182-d4f7-416d-ae44-90f28eab4b7a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6811', 'nonce': 2651666015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6812', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2651666015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6816', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6817', 'nonce': 2651666015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6815', 'nonce': 2651666015}]}, 'public_addr': '172.21.15.69:6811/2651666015', 'cluster_addr': '172.21.15.69:6813/2651666015', 'heartbeat_back_addr': '172.21.15.69:6817/2651666015', 'heartbeat_front_addr': '172.21.15.69:6815/2651666015', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6818', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6819', 'nonce': 619998764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6820', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6821', 'nonce': 619998764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6825', 'nonce': 619998764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6822', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 619998764}]}, 'public_addr': '172.21.15.69:6819/619998764', 'cluster_addr': '172.21.15.69:6821/619998764', 'heartbeat_back_addr': '172.21.15.69:6825/619998764', 'heartbeat_front_addr': '172.21.15.69:6823/619998764', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '20e4526d-2707-44d0-b13a-af36019c42ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6826', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 207593208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 207593208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 207593208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 207593208}]}, 'public_addr': '172.21.15.69:6827/207593208', 'cluster_addr': '172.21.15.69:6829/207593208', 'heartbeat_back_addr': '172.21.15.69:6833/207593208', 'heartbeat_front_addr': '172.21.15.69:6831/207593208', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 83, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 2065947829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 2065947829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 2065947829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 2065947829}]}, 'public_addr': '172.21.15.98:6801/2065947829', 'cluster_addr': '172.21.15.98:6803/2065947829', 'heartbeat_back_addr': '172.21.15.98:6807/2065947829', 'heartbeat_front_addr': '172.21.15.98:6805/2065947829', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '902326af-4a40-47c8-a3b8-d5d5536f90fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 83, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 219578185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 219578185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6814', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 219578185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6813', 'nonce': 219578185}]}, 'public_addr': '172.21.15.98:6809/219578185', 'cluster_addr': '172.21.15.98:6811/219578185', 'heartbeat_back_addr': '172.21.15.98:6815/219578185', 'heartbeat_front_addr': '172.21.15.98:6813/219578185', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 1226172499}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 1226172499}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 1226172499}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 1226172499}]}, 'public_addr': '172.21.15.98:6817/1226172499', 'cluster_addr': '172.21.15.98:6819/1226172499', 'heartbeat_back_addr': '172.21.15.98:6823/1226172499', 'heartbeat_front_addr': '172.21.15.98:6821/1226172499', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 83, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 148009079}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 148009079}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 148009079}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 148009079}]}, 'public_addr': '172.21.15.98:6825/148009079', 'cluster_addr': '172.21.15.98:6827/148009079', 'heartbeat_back_addr': '172.21.15.98:6831/148009079', 'heartbeat_front_addr': '172.21.15.98:6829/148009079', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:48:20.072120+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:47:35.005517+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:48:32.478283+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:47:20.995748+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:47:20.289928+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:47:35.205666+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:48:28.047749+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:48:13.738528+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.69:0/2738867930': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/1798147282': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/52352690': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/2928651991': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:6800/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/2763215770': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/3657409442': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/879855771': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1025808168': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1168952113': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/1892956550': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6801/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/54578327': '2024-04-03T04:44:51.699278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-02T04:48:37.918 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-02T04:48:37.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:38.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.027+0000 7f25aaffd700 1 -- 172.21.15.69:0/3363135494 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v128) ==== 91+0+0 (secure 0 0 0) 0x7f25ac05ae40 con 0x7f25b4106500 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 10 librados: watch_flush enter 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 10 librados: watch_flush exit 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 20 client.14802.objecter shutdown clearing up homeless session... 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 10 client.14802.objecter successfully canceled tick 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25a006f630 msgr2=0x7f25a0071af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25a006f630 0x7f25a0071af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25b00037a0 tx=0x7f25b0019040 comp rx=0 tx=0).stop 2024-04-02T04:48:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 msgr2=0x7f25b419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b419ca80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00c340 tx=0x7f25ac00ef90 comp rx=0 tx=0).stop 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 shutdown_connections 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25a006f630 0x7f25a0071af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4106500 0x7f25b419ca80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4101420 0x7f25b41981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 --2- 172.21.15.69:0/3363135494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4101dd0 0x7f25b41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.031+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 >> 172.21.15.69:0/3363135494 conn(0x7f25b4078240 msgr2=0x7f25b40fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:38.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.039+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 shutdown_connections 2024-04-02T04:48:38.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.039+0000 7f25a8ff9700 1 -- 172.21.15.69:0/3363135494 wait complete. 2024-04-02T04:48:38.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:38.039+0000 7f25a8ff9700 1 librados: shutdown 2024-04-02T04:48:38.043 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:48:38.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:38.202 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[21528]: audit 2024-04-02T04:48:37.213258+0000 mon.b (mon.1) 103 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:38.202 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[21528]: audit 2024-04-02T04:48:37.215095+0000 mon.c (mon.2) 30 : audit 0 from='client.? 172.21.15.69:0/3090750948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:48:38.202 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[21528]: audit 2024-04-02T04:48:37.339174+0000 mon.a (mon.0) 941 : audit 1 from='client.? 172.21.15.69:0/3363135494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:48:38.202 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[28904]: audit 2024-04-02T04:48:37.213258+0000 mon.b (mon.1) 103 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:38.202 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[28904]: audit 2024-04-02T04:48:37.215095+0000 mon.c (mon.2) 30 : audit 0 from='client.? 172.21.15.69:0/3090750948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:48:38.202 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[28904]: audit 2024-04-02T04:48:37.339174+0000 mon.a (mon.0) 941 : audit 1 from='client.? 172.21.15.69:0/3363135494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:48:38.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:38.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:38 smithi098 bash[23790]: audit 2024-04-02T04:48:37.213258+0000 mon.b (mon.1) 103 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:38.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:38 smithi098 bash[23790]: audit 2024-04-02T04:48:37.215095+0000 mon.c (mon.2) 30 : audit 0 from='client.? 172.21.15.69:0/3090750948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:48:38.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:38 smithi098 bash[23790]: audit 2024-04-02T04:48:37.339174+0000 mon.a (mon.0) 941 : audit 1 from='client.? 172.21.15.69:0/3363135494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:48:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:38.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:38.396 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:38.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:38.524 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:38.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:38.653 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:38.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:38.765 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:48:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:48:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:48:38.784 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:38.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:38.912 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:38.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:39.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[21528]: cluster 2024-04-02T04:48:37.824443+0000 mgr.y (mgr.14538) 177 : cluster 0 pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[21528]: cluster 2024-04-02T04:48:38.022521+0000 mon.a (mon.0) 942 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[21528]: audit 2024-04-02T04:48:38.028943+0000 mon.a (mon.0) 943 : audit 1 from='client.? 172.21.15.69:0/3363135494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:48:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[21528]: cluster 2024-04-02T04:48:38.045433+0000 mon.a (mon.0) 944 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-04-02T04:48:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[21528]: cluster 2024-04-02T04:48:38.476209+0000 osd.3 (osd.3) 133 : cluster 0 4.0 scrub starts 2024-04-02T04:48:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[21528]: cluster 2024-04-02T04:48:38.477819+0000 osd.3 (osd.3) 134 : cluster 0 4.0 scrub ok 2024-04-02T04:48:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:39.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:39.173 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:39.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:39.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[28904]: cluster 2024-04-02T04:48:37.824443+0000 mgr.y (mgr.14538) 177 : cluster 0 pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:39.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[28904]: cluster 2024-04-02T04:48:38.022521+0000 mon.a (mon.0) 942 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:39.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[28904]: audit 2024-04-02T04:48:38.028943+0000 mon.a (mon.0) 943 : audit 1 from='client.? 172.21.15.69:0/3363135494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:48:39.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[28904]: cluster 2024-04-02T04:48:38.045433+0000 mon.a (mon.0) 944 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-04-02T04:48:39.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[28904]: cluster 2024-04-02T04:48:38.476209+0000 osd.3 (osd.3) 133 : cluster 0 4.0 scrub starts 2024-04-02T04:48:39.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:39 smithi069 bash[28904]: cluster 2024-04-02T04:48:38.477819+0000 osd.3 (osd.3) 134 : cluster 0 4.0 scrub ok 2024-04-02T04:48:39.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:39 smithi098 bash[23790]: cluster 2024-04-02T04:48:37.824443+0000 mgr.y (mgr.14538) 177 : cluster 0 pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:39.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:39 smithi098 bash[23790]: cluster 2024-04-02T04:48:38.022521+0000 mon.a (mon.0) 942 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:48:39.288 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:39 smithi098 bash[23790]: audit 2024-04-02T04:48:38.028943+0000 mon.a (mon.0) 943 : audit 1 from='client.? 172.21.15.69:0/3363135494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:48:39.288 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:39 smithi098 bash[23790]: cluster 2024-04-02T04:48:38.045433+0000 mon.a (mon.0) 944 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-04-02T04:48:39.288 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:39 smithi098 bash[23790]: cluster 2024-04-02T04:48:38.476209+0000 osd.3 (osd.3) 133 : cluster 0 4.0 scrub starts 2024-04-02T04:48:39.288 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:39 smithi098 bash[23790]: cluster 2024-04-02T04:48:38.477819+0000 osd.3 (osd.3) 134 : cluster 0 4.0 scrub ok 2024-04-02T04:48:39.304 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:39.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:39.536 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:39.560 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:39.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:39.689 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:39.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:39.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:39.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:40.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:40.180 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:40.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[21528]: cluster 2024-04-02T04:48:38.624761+0000 osd.4 (osd.4) 109 : cluster 0 6.1e scrub starts 2024-04-02T04:48:40.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[21528]: cluster 2024-04-02T04:48:38.627229+0000 osd.4 (osd.4) 110 : cluster 0 6.1e scrub ok 2024-04-02T04:48:40.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.369279+0000 osd.4 (osd.4) 111 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:40.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.369438+0000 osd.4 (osd.4) 112 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:40.191 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[28904]: cluster 2024-04-02T04:48:38.624761+0000 osd.4 (osd.4) 109 : cluster 0 6.1e scrub starts 2024-04-02T04:48:40.191 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[28904]: cluster 2024-04-02T04:48:38.627229+0000 osd.4 (osd.4) 110 : cluster 0 6.1e scrub ok 2024-04-02T04:48:40.191 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.369279+0000 osd.4 (osd.4) 111 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:40.191 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:40 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.369438+0000 osd.4 (osd.4) 112 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:40.205 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:40.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:40.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:40.379 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:40.437 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:40.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:40 smithi098 bash[23790]: cluster 2024-04-02T04:48:38.624761+0000 osd.4 (osd.4) 109 : cluster 0 6.1e scrub starts 2024-04-02T04:48:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:40 smithi098 bash[23790]: cluster 2024-04-02T04:48:38.627229+0000 osd.4 (osd.4) 110 : cluster 0 6.1e scrub ok 2024-04-02T04:48:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:40 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.369279+0000 osd.4 (osd.4) 111 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:40 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.369438+0000 osd.4 (osd.4) 112 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:40.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:40.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:48:40.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:40.722 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:40.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:40.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:40.978 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:40.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:41.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:41.107 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:41.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.605327+0000 osd.4 (osd.4) 113 : cluster 0 2.f scrub starts 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.608241+0000 osd.4 (osd.4) 114 : cluster 0 2.f scrub ok 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.824865+0000 mgr.y (mgr.14538) 178 : cluster 0 pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.902375+0000 osd.5 (osd.5) 97 : cluster 0 2.3 scrub starts 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:39.903678+0000 osd.5 (osd.5) 98 : cluster 0 2.3 scrub ok 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.046495+0000 mon.a (mon.0) 945 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.064231+0000 osd.0 (osd.0) 107 : cluster 0 6.10 scrub starts 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.066257+0000 osd.0 (osd.0) 108 : cluster 0 6.10 scrub ok 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.605327+0000 osd.4 (osd.4) 113 : cluster 0 2.f scrub starts 2024-04-02T04:48:41.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.608241+0000 osd.4 (osd.4) 114 : cluster 0 2.f scrub ok 2024-04-02T04:48:41.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.824865+0000 mgr.y (mgr.14538) 178 : cluster 0 pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:41.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.902375+0000 osd.5 (osd.5) 97 : cluster 0 2.3 scrub starts 2024-04-02T04:48:41.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:39.903678+0000 osd.5 (osd.5) 98 : cluster 0 2.3 scrub ok 2024-04-02T04:48:41.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.046495+0000 mon.a (mon.0) 945 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-04-02T04:48:41.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.064231+0000 osd.0 (osd.0) 107 : cluster 0 6.10 scrub starts 2024-04-02T04:48:41.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:41 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.066257+0000 osd.0 (osd.0) 108 : cluster 0 6.10 scrub ok 2024-04-02T04:48:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.605327+0000 osd.4 (osd.4) 113 : cluster 0 2.f scrub starts 2024-04-02T04:48:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.608241+0000 osd.4 (osd.4) 114 : cluster 0 2.f scrub ok 2024-04-02T04:48:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.824865+0000 mgr.y (mgr.14538) 178 : cluster 0 pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.902375+0000 osd.5 (osd.5) 97 : cluster 0 2.3 scrub starts 2024-04-02T04:48:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:39.903678+0000 osd.5 (osd.5) 98 : cluster 0 2.3 scrub ok 2024-04-02T04:48:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.046495+0000 mon.a (mon.0) 945 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-04-02T04:48:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.064231+0000 osd.0 (osd.0) 107 : cluster 0 6.10 scrub starts 2024-04-02T04:48:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:41 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.066257+0000 osd.0 (osd.0) 108 : cluster 0 6.10 scrub ok 2024-04-02T04:48:41.234 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:41.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:41.364 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:41.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:41.499 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:41.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:41.628 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:41.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:41.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:41.889 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:41.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:41.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:42.017 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:42.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:42.145 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:42.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:42.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.602932+0000 osd.4 (osd.4) 115 : cluster 0 6.e scrub starts 2024-04-02T04:48:42.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.604287+0000 osd.4 (osd.4) 116 : cluster 0 6.e scrub ok 2024-04-02T04:48:42.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.898124+0000 osd.5 (osd.5) 99 : cluster 0 2.12 scrub starts 2024-04-02T04:48:42.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[23790]: cluster 2024-04-02T04:48:40.899350+0000 osd.5 (osd.5) 100 : cluster 0 2.12 scrub ok 2024-04-02T04:48:42.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.481047+0000 osd.3 (osd.3) 135 : cluster 0 3.1e scrub starts 2024-04-02T04:48:42.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.484039+0000 osd.3 (osd.3) 136 : cluster 0 3.1e scrub ok 2024-04-02T04:48:42.280 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:42.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:42.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.602932+0000 osd.4 (osd.4) 115 : cluster 0 6.e scrub starts 2024-04-02T04:48:42.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.604287+0000 osd.4 (osd.4) 116 : cluster 0 6.e scrub ok 2024-04-02T04:48:42.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.898124+0000 osd.5 (osd.5) 99 : cluster 0 2.12 scrub starts 2024-04-02T04:48:42.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[21528]: cluster 2024-04-02T04:48:40.899350+0000 osd.5 (osd.5) 100 : cluster 0 2.12 scrub ok 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.481047+0000 osd.3 (osd.3) 135 : cluster 0 3.1e scrub starts 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.484039+0000 osd.3 (osd.3) 136 : cluster 0 3.1e scrub ok 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.602932+0000 osd.4 (osd.4) 115 : cluster 0 6.e scrub starts 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.604287+0000 osd.4 (osd.4) 116 : cluster 0 6.e scrub ok 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.898124+0000 osd.5 (osd.5) 99 : cluster 0 2.12 scrub starts 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[28904]: cluster 2024-04-02T04:48:40.899350+0000 osd.5 (osd.5) 100 : cluster 0 2.12 scrub ok 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.481047+0000 osd.3 (osd.3) 135 : cluster 0 3.1e scrub starts 2024-04-02T04:48:42.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:42 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.484039+0000 osd.3 (osd.3) 136 : cluster 0 3.1e scrub ok 2024-04-02T04:48:42.408 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:42.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:42.630 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:42.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:42.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:42.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:42.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:42.918 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:48:42.919 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:48:42.919 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-02T04:48:42.919 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-02T04:48:42.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:48:42.931 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:42.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:43.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:43.062 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:43.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:43.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:43.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:48:42 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.676249+0000 osd.6 (osd.6) 75 : cluster 0 3.10 scrub starts 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.679473+0000 osd.6 (osd.6) 76 : cluster 0 3.10 scrub ok 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.825288+0000 mgr.y (mgr.14538) 179 : cluster 0 pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.930728+0000 osd.5 (osd.5) 101 : cluster 0 4.15 scrub starts 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:41.932301+0000 osd.5 (osd.5) 102 : cluster 0 4.15 scrub ok 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:42.391317+0000 osd.7 (osd.7) 107 : cluster 0 6.1c scrub starts 2024-04-02T04:48:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:43 smithi098 bash[23790]: cluster 2024-04-02T04:48:42.394214+0000 osd.7 (osd.7) 108 : cluster 0 6.1c scrub ok 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.676249+0000 osd.6 (osd.6) 75 : cluster 0 3.10 scrub starts 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.679473+0000 osd.6 (osd.6) 76 : cluster 0 3.10 scrub ok 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.825288+0000 mgr.y (mgr.14538) 179 : cluster 0 pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.930728+0000 osd.5 (osd.5) 101 : cluster 0 4.15 scrub starts 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:41.932301+0000 osd.5 (osd.5) 102 : cluster 0 4.15 scrub ok 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:42.391317+0000 osd.7 (osd.7) 107 : cluster 0 6.1c scrub starts 2024-04-02T04:48:43.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[21528]: cluster 2024-04-02T04:48:42.394214+0000 osd.7 (osd.7) 108 : cluster 0 6.1c scrub ok 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.676249+0000 osd.6 (osd.6) 75 : cluster 0 3.10 scrub starts 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.679473+0000 osd.6 (osd.6) 76 : cluster 0 3.10 scrub ok 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.825288+0000 mgr.y (mgr.14538) 179 : cluster 0 pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.930728+0000 osd.5 (osd.5) 101 : cluster 0 4.15 scrub starts 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:41.932301+0000 osd.5 (osd.5) 102 : cluster 0 4.15 scrub ok 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:42.391317+0000 osd.7 (osd.7) 107 : cluster 0 6.1c scrub starts 2024-04-02T04:48:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:43 smithi069 bash[28904]: cluster 2024-04-02T04:48:42.394214+0000 osd.7 (osd.7) 108 : cluster 0 6.1c scrub ok 2024-04-02T04:48:43.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:43.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:43.457 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:43.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:43.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:43.717 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:43.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:43.949 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:43.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:43.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:44.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:44.105 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:44.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:42.657317+0000 osd.2 (osd.2) 59 : cluster 0 2.e scrub starts 2024-04-02T04:48:44.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:42.658682+0000 osd.2 (osd.2) 60 : cluster 0 2.e scrub ok 2024-04-02T04:48:44.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:42.658872+0000 osd.6 (osd.6) 77 : cluster 0 2.14 scrub starts 2024-04-02T04:48:44.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:42.660242+0000 osd.6 (osd.6) 78 : cluster 0 2.14 scrub ok 2024-04-02T04:48:44.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: audit 2024-04-02T04:48:42.948775+0000 mgr.y (mgr.14538) 180 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.039780+0000 osd.0 (osd.0) 109 : cluster 0 3.6 scrub starts 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.043148+0000 osd.0 (osd.0) 110 : cluster 0 3.6 scrub ok 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.454610+0000 osd.3 (osd.3) 137 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.454697+0000 osd.3 (osd.3) 138 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:42.657317+0000 osd.2 (osd.2) 59 : cluster 0 2.e scrub starts 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:42.658682+0000 osd.2 (osd.2) 60 : cluster 0 2.e scrub ok 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:42.658872+0000 osd.6 (osd.6) 77 : cluster 0 2.14 scrub starts 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:42.660242+0000 osd.6 (osd.6) 78 : cluster 0 2.14 scrub ok 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: audit 2024-04-02T04:48:42.948775+0000 mgr.y (mgr.14538) 180 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:44.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.039780+0000 osd.0 (osd.0) 109 : cluster 0 3.6 scrub starts 2024-04-02T04:48:44.119 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.043148+0000 osd.0 (osd.0) 110 : cluster 0 3.6 scrub ok 2024-04-02T04:48:44.119 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.454610+0000 osd.3 (osd.3) 137 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:44.119 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:44 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.454697+0000 osd.3 (osd.3) 138 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:44.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:44.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:44.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:44.348 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:48:44.348 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:48:44.372 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:42.657317+0000 osd.2 (osd.2) 59 : cluster 0 2.e scrub starts 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:42.658682+0000 osd.2 (osd.2) 60 : cluster 0 2.e scrub ok 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:42.658872+0000 osd.6 (osd.6) 77 : cluster 0 2.14 scrub starts 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:42.660242+0000 osd.6 (osd.6) 78 : cluster 0 2.14 scrub ok 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: audit 2024-04-02T04:48:42.948775+0000 mgr.y (mgr.14538) 180 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.039780+0000 osd.0 (osd.0) 109 : cluster 0 3.6 scrub starts 2024-04-02T04:48:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.043148+0000 osd.0 (osd.0) 110 : cluster 0 3.6 scrub ok 2024-04-02T04:48:44.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.454610+0000 osd.3 (osd.3) 137 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:44.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:44 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.454697+0000 osd.3 (osd.3) 138 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:44.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:44.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:44.629 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:44.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:44.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:44.891 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:44.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:44.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:45.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.626825+0000 osd.6 (osd.6) 79 : cluster 0 4.1f scrub starts 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.629785+0000 osd.6 (osd.6) 80 : cluster 0 4.1f scrub ok 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.826124+0000 mgr.y (mgr.14538) 181 : cluster 0 pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.957072+0000 osd.5 (osd.5) 103 : cluster 0 3.1d scrub starts 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:43.963220+0000 osd.5 (osd.5) 104 : cluster 0 3.1d scrub ok 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.067231+0000 mon.a (mon.0) 946 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.526494+0000 osd.3 (osd.3) 139 : cluster 0 6.11 scrub starts 2024-04-02T04:48:45.132 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.527668+0000 osd.3 (osd.3) 140 : cluster 0 6.11 scrub ok 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.626825+0000 osd.6 (osd.6) 79 : cluster 0 4.1f scrub starts 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.629785+0000 osd.6 (osd.6) 80 : cluster 0 4.1f scrub ok 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.826124+0000 mgr.y (mgr.14538) 181 : cluster 0 pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.957072+0000 osd.5 (osd.5) 103 : cluster 0 3.1d scrub starts 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:43.963220+0000 osd.5 (osd.5) 104 : cluster 0 3.1d scrub ok 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.067231+0000 mon.a (mon.0) 946 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.526494+0000 osd.3 (osd.3) 139 : cluster 0 6.11 scrub starts 2024-04-02T04:48:45.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:45 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.527668+0000 osd.3 (osd.3) 140 : cluster 0 6.11 scrub ok 2024-04-02T04:48:45.146 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:45.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:45.274 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:45.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:45.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.626825+0000 osd.6 (osd.6) 79 : cluster 0 4.1f scrub starts 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.629785+0000 osd.6 (osd.6) 80 : cluster 0 4.1f scrub ok 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.826124+0000 mgr.y (mgr.14538) 181 : cluster 0 pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.957072+0000 osd.5 (osd.5) 103 : cluster 0 3.1d scrub starts 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:43.963220+0000 osd.5 (osd.5) 104 : cluster 0 3.1d scrub ok 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.067231+0000 mon.a (mon.0) 946 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.526494+0000 osd.3 (osd.3) 139 : cluster 0 6.11 scrub starts 2024-04-02T04:48:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:45 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.527668+0000 osd.3 (osd.3) 140 : cluster 0 6.11 scrub ok 2024-04-02T04:48:45.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:45.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:45.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:45.688 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:45.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:45.821 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:45.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:45.927 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:45.950 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:45.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:46.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:46.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:46.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.554698+0000 osd.4 (osd.4) 117 : cluster 0 5.1 scrub starts 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.555848+0000 osd.4 (osd.4) 118 : cluster 0 5.1 scrub ok 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.645061+0000 osd.2 (osd.2) 61 : cluster 0 6.14 scrub starts 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.646701+0000 osd.2 (osd.2) 62 : cluster 0 6.14 scrub ok 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.910219+0000 osd.5 (osd.5) 105 : cluster 0 2.2 scrub starts 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:44.911598+0000 osd.5 (osd.5) 106 : cluster 0 2.2 scrub ok 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.437129+0000 osd.7 (osd.7) 109 : cluster 0 2.b scrub starts 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.439444+0000 osd.7 (osd.7) 110 : cluster 0 2.b scrub ok 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.483903+0000 osd.3 (osd.3) 141 : cluster 0 6.c scrub starts 2024-04-02T04:48:46.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:46 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.485435+0000 osd.3 (osd.3) 142 : cluster 0 6.c scrub ok 2024-04-02T04:48:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:46.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.554698+0000 osd.4 (osd.4) 117 : cluster 0 5.1 scrub starts 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.555848+0000 osd.4 (osd.4) 118 : cluster 0 5.1 scrub ok 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.645061+0000 osd.2 (osd.2) 61 : cluster 0 6.14 scrub starts 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.646701+0000 osd.2 (osd.2) 62 : cluster 0 6.14 scrub ok 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.910219+0000 osd.5 (osd.5) 105 : cluster 0 2.2 scrub starts 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:44.911598+0000 osd.5 (osd.5) 106 : cluster 0 2.2 scrub ok 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.437129+0000 osd.7 (osd.7) 109 : cluster 0 2.b scrub starts 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.439444+0000 osd.7 (osd.7) 110 : cluster 0 2.b scrub ok 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.483903+0000 osd.3 (osd.3) 141 : cluster 0 6.c scrub starts 2024-04-02T04:48:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.485435+0000 osd.3 (osd.3) 142 : cluster 0 6.c scrub ok 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.554698+0000 osd.4 (osd.4) 117 : cluster 0 5.1 scrub starts 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.555848+0000 osd.4 (osd.4) 118 : cluster 0 5.1 scrub ok 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.645061+0000 osd.2 (osd.2) 61 : cluster 0 6.14 scrub starts 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.646701+0000 osd.2 (osd.2) 62 : cluster 0 6.14 scrub ok 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.910219+0000 osd.5 (osd.5) 105 : cluster 0 2.2 scrub starts 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:44.911598+0000 osd.5 (osd.5) 106 : cluster 0 2.2 scrub ok 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.437129+0000 osd.7 (osd.7) 109 : cluster 0 2.b scrub starts 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.439444+0000 osd.7 (osd.7) 110 : cluster 0 2.b scrub ok 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.483903+0000 osd.3 (osd.3) 141 : cluster 0 6.c scrub starts 2024-04-02T04:48:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:46 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.485435+0000 osd.3 (osd.3) 142 : cluster 0 6.c scrub ok 2024-04-02T04:48:46.467 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:46.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:46.597 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:46.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:46.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:46.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:46.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:46.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:47.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:47.118 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:47.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f735959a700 1 -- 172.21.15.69:0/3862517185 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c003a70 con 0x7f735c101420 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/3862517185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c101420 msgr2=0x7f735c101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- 172.21.15.69:0/3862517185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c101420 0x7f735c101800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f734c0099c0 tx=0x7f734c009cd0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/3862517185 shutdown_connections 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- 172.21.15.69:0/3862517185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c106500 0x7f735c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- 172.21.15.69:0/3862517185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c101420 0x7f735c101800 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- 172.21.15.69:0/3862517185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/3862517185 >> 172.21.15.69:0/3862517185 conn(0x7f735c078240 msgr2=0x7f735c078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:47.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/3862517185 shutdown_connections 2024-04-02T04:48:47.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/3862517185 wait complete. 2024-04-02T04:48:47.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 librados: starting msgr at 2024-04-02T04:48:47.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 librados: starting objecter 2024-04-02T04:48:47.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 Processor -- start 2024-04-02T04:48:47.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- start start 2024-04-02T04:48:47.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 librados: setting wanted keys 2024-04-02T04:48:47.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 librados: calling monclient init 2024-04-02T04:48:47.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c101420 0x7f735c102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c106500 0x7f735c103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c103400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:59922/0 (socket says 172.21.15.69:59922) 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 -- 172.21.15.69:0/4270301083 learned_addr learned my addr 172.21.15.69:0/4270301083 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f735c110a50 con 0x7f735c106500 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1108d0 con 0x7f735c101420 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f735c110bd0 con 0x7f735c101dd0 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c101420 0x7f735c102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c106500 0x7f735c103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c101420 msgr2=0x7f735c102ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c101420 0x7f735c102ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c106500 msgr2=0x7f735c103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f735c106500 0x7f735c103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c1a6600 con 0x7f735c101dd0 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c101420 0x7f735c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c103400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f735000e970 tx=0x7f735000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f734b7fe700 10 client.?.objecter ms_handle_connect 0x7f735c101dd0 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f734b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.207+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735000cbd0 con 0x7f735c101dd0 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735c1a68f0 con 0x7f735c101dd0 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f7360cbd700 10 client.24695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f735c1a6e80 con 0x7f735c101dd0 2024-04-02T04:48:47.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f7360cbd700 1 librados: init done 2024-04-02T04:48:47.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f735000cd70 con 0x7f735c101dd0 2024-04-02T04:48:47.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350005640 con 0x7f735c101dd0 2024-04-02T04:48:47.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7350004750 con 0x7f735c101dd0 2024-04-02T04:48:47.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f733c005130 con 0x7f735c101dd0 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f734406f5e0 0x7f7344071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(130..130 src has 74..130) ==== 7537+0+0 (secure 0 0 0) 0x7f735008e1f0 con 0x7f735c101dd0 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 osd_map(130..130 src has 74..130) 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 3 client.24695.objecter handle_osd_map got epochs [130,130] > 0 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 3 client.24695.objecter handle_osd_map decoding full epoch 130 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f734406f5e0 0x7f7344071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.211+0000 7f734b7fe700 20 client.24695.objecter dump_active .. 0 homeless 2024-04-02T04:48:47.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v130) ==== 54+0+16 (secure 0 0 0) 0x7f735005b850 con 0x7f735c101dd0 2024-04-02T04:48:47.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f734406f5e0 2024-04-02T04:48:47.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f734406f5e0 0x7f7344071aa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f734c000c00 tx=0x7f734c005fb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 10 client.24695.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 1 2024-04-02T04:48:47.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f733c004760 0x7f733c004ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 1 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 0 1 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 4 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 10 client.24695.objecter _send_command 1 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f733c008ba0 con 0x7f733c004760 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f733c004760 0x7f733c004ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c004760 2024-04-02T04:48:47.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f733c004760 0x7f733c004ba0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c008ba0 con 0x7f733c004760 2024-04-02T04:48:47.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 1: 0 ) 2024-04-02T04:48:47.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 10 client.24695.objecter _finish_command 1 = system:0 2024-04-02T04:48:47.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 4 2024-04-02T04:48:47.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.215+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 0 1 2024-04-02T04:48:47.222 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:47.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.566696+0000 osd.6 (osd.6) 81 : cluster 0 4.5 scrub starts 2024-04-02T04:48:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.569048+0000 osd.6 (osd.6) 82 : cluster 0 4.5 scrub ok 2024-04-02T04:48:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.826550+0000 mgr.y (mgr.14538) 182 : cluster 0 pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-04-02T04:48:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.908837+0000 osd.5 (osd.5) 107 : cluster 0 6.a scrub starts 2024-04-02T04:48:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:45.911416+0000 osd.5 (osd.5) 108 : cluster 0 6.a scrub ok 2024-04-02T04:48:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:46.072632+0000 osd.0 (osd.0) 111 : cluster 0 2.13 scrub starts 2024-04-02T04:48:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:47 smithi098 bash[23790]: cluster 2024-04-02T04:48:46.073938+0000 osd.0 (osd.0) 112 : cluster 0 2.13 scrub ok 2024-04-02T04:48:47.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 10 client.24695.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 2 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 2 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c005410 osd=0 3 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 0 2 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 4 2024-04-02T04:48:47.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 10 client.24695.objecter _send_command 2 2024-04-02T04:48:47.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.251+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c0042d0 con 0x7f733c004760 2024-04-02T04:48:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:47.254 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c0042d0 con 0x7f733c004760 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f734b7fe700 10 client.24695.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 4 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 0 2 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 10 client.24695.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 3 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f733c002780 0x7f733c002c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 3 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 1 3 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 4 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 10 client.24695.objecter _send_command 3 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f733c003170 con 0x7f733c002780 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f733c002780 0x7f733c002c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f733c002780 0x7f733c002c30 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.255+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c002780 2024-04-02T04:48:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.259+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c003170 con 0x7f733c002780 2024-04-02T04:48:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.259+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 3: 0 ) 2024-04-02T04:48:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.259+0000 7f734b7fe700 10 client.24695.objecter _finish_command 3 = system:0 2024-04-02T04:48:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.259+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 4 2024-04-02T04:48:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.259+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 1 3 2024-04-02T04:48:47.259 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.566696+0000 osd.6 (osd.6) 81 : cluster 0 4.5 scrub starts 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.569048+0000 osd.6 (osd.6) 82 : cluster 0 4.5 scrub ok 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.826550+0000 mgr.y (mgr.14538) 182 : cluster 0 pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.908837+0000 osd.5 (osd.5) 107 : cluster 0 6.a scrub starts 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:45.911416+0000 osd.5 (osd.5) 108 : cluster 0 6.a scrub ok 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:46.072632+0000 osd.0 (osd.0) 111 : cluster 0 2.13 scrub starts 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[28904]: cluster 2024-04-02T04:48:46.073938+0000 osd.0 (osd.0) 112 : cluster 0 2.13 scrub ok 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.566696+0000 osd.6 (osd.6) 81 : cluster 0 4.5 scrub starts 2024-04-02T04:48:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.569048+0000 osd.6 (osd.6) 82 : cluster 0 4.5 scrub ok 2024-04-02T04:48:47.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.826550+0000 mgr.y (mgr.14538) 182 : cluster 0 pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-04-02T04:48:47.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.908837+0000 osd.5 (osd.5) 107 : cluster 0 6.a scrub starts 2024-04-02T04:48:47.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:45.911416+0000 osd.5 (osd.5) 108 : cluster 0 6.a scrub ok 2024-04-02T04:48:47.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:46.072632+0000 osd.0 (osd.0) 111 : cluster 0 2.13 scrub starts 2024-04-02T04:48:47.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:47 smithi069 bash[21528]: cluster 2024-04-02T04:48:46.073938+0000 osd.0 (osd.0) 112 : cluster 0 2.13 scrub ok 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 10 client.24695.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 4 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 4 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c004560 osd=1 3 2024-04-02T04:48:47.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 1 4 2024-04-02T04:48:47.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 4 2024-04-02T04:48:47.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 10 client.24695.objecter _send_command 4 2024-04-02T04:48:47.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c003bc0 con 0x7f733c002780 2024-04-02T04:48:47.294 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:48:47.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c003bc0 con 0x7f733c002780 2024-04-02T04:48:47.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f734b7fe700 10 client.24695.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 4 2024-04-02T04:48:47.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 1 4 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 10 client.24695.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 5 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f733c00b270 0x7f733c00d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 5 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 2 5 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 4 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 10 client.24695.objecter _send_command 5 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f733c0105a0 con 0x7f733c00b270 2024-04-02T04:48:47.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.295+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f733c00b270 0x7f733c00d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c00b270 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f733c00b270 0x7f733c00d660 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c0105a0 con 0x7f733c00b270 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 5: 0 ) 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f734b7fe700 10 client.24695.objecter _finish_command 5 = system:0 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 4 2024-04-02T04:48:47.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.299+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 2 5 2024-04-02T04:48:47.334 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 10 client.24695.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 6 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 6 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c003d70 osd=2 3 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 2 6 2024-04-02T04:48:47.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 4 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 10 client.24695.objecter _send_command 6 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.331+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c0036c0 con 0x7f733c00b270 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c0036c0 con 0x7f733c00b270 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 10 client.24695.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 4 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 2 6 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 10 client.24695.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 7 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f733c00ed20 0x7f733c00f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 7 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f733c00ed20 0x7f733c00f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 3 7 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 4 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 10 client.24695.objecter _send_command 7 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f733c00f6f0 con 0x7f733c00ed20 2024-04-02T04:48:47.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f733c00ed20 0x7f733c00f1b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c00ed20 2024-04-02T04:48:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c00f6f0 con 0x7f733c00ed20 2024-04-02T04:48:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 7: 0 ) 2024-04-02T04:48:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 10 client.24695.objecter _finish_command 7 = system:0 2024-04-02T04:48:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 4 2024-04-02T04:48:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.335+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 3 7 2024-04-02T04:48:47.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:47.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 10 client.24695.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 8 2024-04-02T04:48:47.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 8 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c0081f0 osd=3 3 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 3 8 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 4 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 10 client.24695.objecter _send_command 8 2024-04-02T04:48:47.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.371+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c010100 con 0x7f733c00ed20 2024-04-02T04:48:47.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c010100 con 0x7f733c00ed20 2024-04-02T04:48:47.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f734b7fe700 10 client.24695.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 4 2024-04-02T04:48:47.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 3 8 2024-04-02T04:48:47.372 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 10 client.24695.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 9 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f733c012760 0x7f733c014b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 9 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 4 9 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 4 2024-04-02T04:48:47.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 10 client.24695.objecter _send_command 9 2024-04-02T04:48:47.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f733c0179b0 con 0x7f733c012760 2024-04-02T04:48:47.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.375+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f733c012760 0x7f733c014b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.378 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:47.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f733c012760 0x7f733c014b50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c012760 2024-04-02T04:48:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c0179b0 con 0x7f733c012760 2024-04-02T04:48:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 9: 0 ) 2024-04-02T04:48:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f734b7fe700 10 client.24695.objecter _finish_command 9 = system:0 2024-04-02T04:48:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 4 2024-04-02T04:48:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.383+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 4 9 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 10 client.24695.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 10 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 10 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c010100 osd=4 3 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 4 10 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 4 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 10 client.24695.objecter _send_command 10 2024-04-02T04:48:47.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.415+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c007a80 con 0x7f733c012760 2024-04-02T04:48:47.419 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c007a80 con 0x7f733c012760 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f734b7fe700 10 client.24695.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 4 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 4 10 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 10 client.24695.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 11 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f733c00e370 0x7f733c00e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 11 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 5 11 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 4 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 10 client.24695.objecter _send_command 11 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.419+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f733c01ac70 con 0x7f733c00e370 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f733c00e370 0x7f733c00e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c00e370 2024-04-02T04:48:47.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f7359d9b700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f733c00e370 0x7f733c00e7d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c01ac70 con 0x7f733c00e370 2024-04-02T04:48:47.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 11: 0 ) 2024-04-02T04:48:47.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f734b7fe700 10 client.24695.objecter _finish_command 11 = system:0 2024-04-02T04:48:47.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 4 2024-04-02T04:48:47.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.423+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 5 11 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 10 client.24695.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 12 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 12 2024-04-02T04:48:47.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c007a80 osd=5 3 2024-04-02T04:48:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 5 12 2024-04-02T04:48:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 4 2024-04-02T04:48:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 10 client.24695.objecter _send_command 12 2024-04-02T04:48:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.447+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c01ac70 con 0x7f733c00e370 2024-04-02T04:48:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c01ac70 con 0x7f733c00e370 2024-04-02T04:48:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f734b7fe700 10 client.24695.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 4 2024-04-02T04:48:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 5 12 2024-04-02T04:48:47.449 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:48:47.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 10 client.24695.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 13 2024-04-02T04:48:47.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f733c019f50 0x7f733c01a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 13 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 6 13 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 4 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 10 client.24695.objecter _send_command 13 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f733c01e090 con 0x7f733c019f50 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f733c019f50 0x7f733c01a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c019f50 2024-04-02T04:48:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.451+0000 7f735ad9d700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f733c019f50 0x7f733c01a3b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.455+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c01e090 con 0x7f733c019f50 2024-04-02T04:48:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.455+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 13: 0 ) 2024-04-02T04:48:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.455+0000 7f734b7fe700 10 client.24695.objecter _finish_command 13 = system:0 2024-04-02T04:48:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.455+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 4 2024-04-02T04:48:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.455+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 6 13 2024-04-02T04:48:47.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 10 client.24695.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 14 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 14 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c01ac70 osd=6 3 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 6 14 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 4 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 10 client.24695.objecter _send_command 14 2024-04-02T04:48:47.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.471+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c018270 con 0x7f733c019f50 2024-04-02T04:48:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c018270 con 0x7f733c019f50 2024-04-02T04:48:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f734b7fe700 10 client.24695.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.474 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 4 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 6 14 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 10 client.24695.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 15 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f733c018780 0x7f733c018bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 15 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 7 15 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 4 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 10 client.24695.objecter _send_command 15 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f733c019100 con 0x7f733c018780 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f733c018780 0x7f733c018bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f735a59c700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f733c018780 0x7f733c018bc0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.475+0000 7f734b7fe700 10 client.24695.objecter ms_handle_connect 0x7f733c018780 2024-04-02T04:48:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.479+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f733c019100 con 0x7f733c018780 2024-04-02T04:48:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.479+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 15: 0 ) 2024-04-02T04:48:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.479+0000 7f734b7fe700 10 client.24695.objecter _finish_command 15 = system:0 2024-04-02T04:48:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.479+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 4 2024-04-02T04:48:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.479+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 7 15 2024-04-02T04:48:47.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 10 client.24695.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:48:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign -1 16 2024-04-02T04:48:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 20 client.24695.objecter _get_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_remove -1 16 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 20 client.24695.objecter get_session s=0x7f733c0195f0 osd=7 3 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 15 client.24695.objecter _session_command_op_assign 7 16 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 4 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 10 client.24695.objecter _send_command 16 2024-04-02T04:48:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.495+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f733c019100 con 0x7f733c018780 2024-04-02T04:48:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f734b7fe700 1 -- 172.21.15.69:0/4270301083 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f733c019100 con 0x7f733c018780 2024-04-02T04:48:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f734b7fe700 10 client.24695.objecter ms_dispatch 0x7f735c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:48:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f734b7fe700 10 client.24695.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:48:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f734b7fe700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 4 2024-04-02T04:48:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f734b7fe700 15 client.24695.objecter _session_command_op_remove 7 16 2024-04-02T04:48:47.499 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 librados: watch_flush enter 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 librados: watch_flush exit 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.0 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f733c004760 msgr2=0x7f733c004ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f733c004760 0x7f733c004ba0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c005410 osd=0 1 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.1 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f733c002780 msgr2=0x7f733c002c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f733c002780 0x7f733c002c30 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c004560 osd=1 1 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.2 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f733c00b270 msgr2=0x7f733c00d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f733c00b270 0x7f733c00d660 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c003d70 osd=2 1 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f735ad9d700 1 -- 172.21.15.69:0/4270301083 reap_dead start 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.3 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f733c00ed20 msgr2=0x7f733c00f1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f733c00ed20 0x7f733c00f1b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0081f0 osd=3 1 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.4 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f733c012760 msgr2=0x7f733c014b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f733c012760 0x7f733c014b50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c010100 osd=4 1 2024-04-02T04:48:47.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.5 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f733c00e370 msgr2=0x7f733c00e7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f733c00e370 0x7f733c00e7d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c007a80 osd=5 1 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.6 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f733c019f50 msgr2=0x7f733c01a3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f733c019f50 0x7f733c01a3b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c01ac70 osd=6 1 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter close_session for osd.7 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f733c018780 msgr2=0x7f733c018bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f733c018780 0x7f733c018bc0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter put_session s=0x7f733c0195f0 osd=7 1 2024-04-02T04:48:47.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 20 client.24695.objecter shutdown clearing up homeless session... 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 10 client.24695.objecter successfully canceled tick 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f735ad9d700 1 -- 172.21.15.69:0/4270301083 reap_dead start 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f734406f5e0 msgr2=0x7f7344071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f734406f5e0 0x7f7344071aa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f734c000c00 tx=0x7f734c005fb0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 msgr2=0x7f735c103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.499+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c103400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f735000e970 tx=0x7f735000ed30 comp rx=0 tx=0).stop 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 shutdown_connections 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f735c101dd0 0x7f735c103400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 --2- 172.21.15.69:0/4270301083 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f734406f5e0 0x7f7344071aa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 >> 172.21.15.69:0/4270301083 conn(0x7f735c078240 msgr2=0x7f735c06a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 shutdown_connections 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 -- 172.21.15.69:0/4270301083 wait complete. 2024-04-02T04:48:47.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:47.503+0000 7f7360cbd700 1 librados: shutdown 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:48:47.504 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:48:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:47.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:47.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:47.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:47.896 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:47.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:47.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:48.024 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:48.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:46.601745+0000 osd.6 (osd.6) 83 : cluster 0 2.17 scrub starts 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:46.602844+0000 osd.6 (osd.6) 84 : cluster 0 2.17 scrub ok 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:46.885581+0000 osd.5 (osd.5) 109 : cluster 0 3.1c scrub starts 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:46.887898+0000 osd.5 (osd.5) 110 : cluster 0 3.1c scrub ok 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.090443+0000 osd.0 (osd.0) 113 : cluster 0 3.1 scrub starts 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.093751+0000 osd.0 (osd.0) 114 : cluster 0 3.1 scrub ok 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.124671+0000 osd.1 (osd.1) 107 : cluster 0 4.7 scrub starts 2024-04-02T04:48:48.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.126356+0000 osd.1 (osd.1) 108 : cluster 0 4.7 scrub ok 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: audit 2024-04-02T04:48:47.215088+0000 mon.c (mon.2) 31 : audit 0 from='client.? 172.21.15.69:0/4270301083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.424921+0000 osd.7 (osd.7) 111 : cluster 0 3.f scrub starts 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.429007+0000 osd.7 (osd.7) 112 : cluster 0 3.f scrub ok 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:46.601745+0000 osd.6 (osd.6) 83 : cluster 0 2.17 scrub starts 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:46.602844+0000 osd.6 (osd.6) 84 : cluster 0 2.17 scrub ok 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:46.885581+0000 osd.5 (osd.5) 109 : cluster 0 3.1c scrub starts 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:46.887898+0000 osd.5 (osd.5) 110 : cluster 0 3.1c scrub ok 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.090443+0000 osd.0 (osd.0) 113 : cluster 0 3.1 scrub starts 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.093751+0000 osd.0 (osd.0) 114 : cluster 0 3.1 scrub ok 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.124671+0000 osd.1 (osd.1) 107 : cluster 0 4.7 scrub starts 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.126356+0000 osd.1 (osd.1) 108 : cluster 0 4.7 scrub ok 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: audit 2024-04-02T04:48:47.215088+0000 mon.c (mon.2) 31 : audit 0 from='client.? 172.21.15.69:0/4270301083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:48.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.424921+0000 osd.7 (osd.7) 111 : cluster 0 3.f scrub starts 2024-04-02T04:48:48.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.429007+0000 osd.7 (osd.7) 112 : cluster 0 3.f scrub ok 2024-04-02T04:48:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:48.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:48.283 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:48.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:48.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:46.601745+0000 osd.6 (osd.6) 83 : cluster 0 2.17 scrub starts 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:46.602844+0000 osd.6 (osd.6) 84 : cluster 0 2.17 scrub ok 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:46.885581+0000 osd.5 (osd.5) 109 : cluster 0 3.1c scrub starts 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:46.887898+0000 osd.5 (osd.5) 110 : cluster 0 3.1c scrub ok 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.090443+0000 osd.0 (osd.0) 113 : cluster 0 3.1 scrub starts 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.093751+0000 osd.0 (osd.0) 114 : cluster 0 3.1 scrub ok 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.124671+0000 osd.1 (osd.1) 107 : cluster 0 4.7 scrub starts 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.126356+0000 osd.1 (osd.1) 108 : cluster 0 4.7 scrub ok 2024-04-02T04:48:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: audit 2024-04-02T04:48:47.215088+0000 mon.c (mon.2) 31 : audit 0 from='client.? 172.21.15.69:0/4270301083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:48:48.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.424921+0000 osd.7 (osd.7) 111 : cluster 0 3.f scrub starts 2024-04-02T04:48:48.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:48 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.429007+0000 osd.7 (osd.7) 112 : cluster 0 3.f scrub ok 2024-04-02T04:48:48.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:48.539 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:48.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:48.670 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:48.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:48.798 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:48.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:48.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:48.969 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:48:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:48:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:48:49.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:49.055 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:49.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.184 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:49.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:49 smithi069 bash[21528]: cluster 2024-04-02T04:48:47.827371+0000 mgr.y (mgr.14538) 183 : cluster 0 pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:49.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:49 smithi069 bash[21528]: cluster 2024-04-02T04:48:48.434003+0000 osd.7 (osd.7) 113 : cluster 0 2.1c scrub starts 2024-04-02T04:48:49.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:49 smithi069 bash[21528]: cluster 2024-04-02T04:48:48.436642+0000 osd.7 (osd.7) 114 : cluster 0 2.1c scrub ok 2024-04-02T04:48:49.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:49 smithi069 bash[28904]: cluster 2024-04-02T04:48:47.827371+0000 mgr.y (mgr.14538) 183 : cluster 0 pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:49.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:49 smithi069 bash[28904]: cluster 2024-04-02T04:48:48.434003+0000 osd.7 (osd.7) 113 : cluster 0 2.1c scrub starts 2024-04-02T04:48:49.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:49 smithi069 bash[28904]: cluster 2024-04-02T04:48:48.436642+0000 osd.7 (osd.7) 114 : cluster 0 2.1c scrub ok 2024-04-02T04:48:49.313 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:49.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:49.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:49 smithi098 bash[23790]: cluster 2024-04-02T04:48:47.827371+0000 mgr.y (mgr.14538) 183 : cluster 0 pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:49.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:49 smithi098 bash[23790]: cluster 2024-04-02T04:48:48.434003+0000 osd.7 (osd.7) 113 : cluster 0 2.1c scrub starts 2024-04-02T04:48:49.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:49 smithi098 bash[23790]: cluster 2024-04-02T04:48:48.436642+0000 osd.7 (osd.7) 114 : cluster 0 2.1c scrub ok 2024-04-02T04:48:49.442 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:49.467 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:49.544 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.571 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/1856517491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc107350 msgr2=0x7faafc107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- 172.21.15.69:0/1856517491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc107350 0x7faafc107730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faaf80099c0 tx=0x7faaf8009cd0 comp rx=0 tx=0).stop 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/1856517491 shutdown_connections 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- 172.21.15.69:0/1856517491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc101e20 0x7faafc10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- 172.21.15.69:0/1856517491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc107350 0x7faafc107730 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- 172.21.15.69:0/1856517491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faafc101460 0x7faafc1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/1856517491 >> 172.21.15.69:0/1856517491 conn(0x7faafc077ea0 msgr2=0x7faafc0782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:49.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/1856517491 shutdown_connections 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/1856517491 wait complete. 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 librados: starting msgr at 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 librados: starting objecter 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 Processor -- start 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- start start 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 librados: setting wanted keys 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 librados: calling monclient init 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc101460 0x7faafc196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:49.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faafc101e20 0x7faafc1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:49.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc107350 0x7faafc197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:49.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab0145a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc101460 0x7faafc196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab0145a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc101460 0x7faafc196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52052/0 (socket says 172.21.15.69:52052) 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab0145a700 1 -- 172.21.15.69:0/3561671308 learned_addr learned my addr 172.21.15.69:0/3561671308 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faafc110990 con 0x7faafc107350 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc107350 0x7faafc197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faafc110810 con 0x7faafc101e20 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faafc110b10 con 0x7faafc101460 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 -- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc101460 msgr2=0x7faafc196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc101460 0x7faafc196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 -- 172.21.15.69:0/3561671308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faafc101e20 msgr2=0x7faafc1971c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faafc101e20 0x7faafc1971c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc1a64f0 con 0x7faafc107350 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab01c5b700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc107350 0x7faafc197700 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faaf400efb0 tx=0x7faaf400c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:49.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7faaf27fc700 10 client.?.objecter ms_handle_connect 0x7faafc107350 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7faaf27fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf400cb00 con 0x7faafc107350 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faafc1a67e0 con 0x7faafc107350 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7faaf400cca0 con 0x7faafc107350 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 10 client.14814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faafc1a6d20 con 0x7faafc107350 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7fab036be700 1 librados: init done 2024-04-02T04:48:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.635+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004dd0 con 0x7faafc107350 2024-04-02T04:48:49.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faae0005120 con 0x7faafc107350 2024-04-02T04:48:49.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7faaf4012050 con 0x7faafc107350 2024-04-02T04:48:49.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faae806f5e0 0x7faae8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:49.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(131..131 src has 74..131) ==== 7537+0+0 (secure 0 0 0) 0x7faaf408dcb0 con 0x7faafc107350 2024-04-02T04:48:49.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 10 client.14814.objecter ms_dispatch 0x7faafc001550 osd_map(131..131 src has 74..131) 2024-04-02T04:48:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 3 client.14814.objecter handle_osd_map got epochs [131,131] > 0 2024-04-02T04:48:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 3 client.14814.objecter handle_osd_map decoding full epoch 131 2024-04-02T04:48:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7fab0145a700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faae806f5e0 0x7faae8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 20 client.14814.objecter dump_active .. 0 homeless 2024-04-02T04:48:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7fab0145a700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faae806f5e0 0x7faae8071aa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faaf80037a0 tx=0x7faaf8005ad0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.639+0000 7faaf27fc700 10 client.14814.objecter ms_handle_connect 0x7faae806f5e0 2024-04-02T04:48:49.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.647+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faaf4022020 con 0x7faafc107350 2024-04-02T04:48:49.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:49.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:49.867+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7faae0005410 con 0x7faafc107350 2024-04-02T04:48:49.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:49.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:50.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:50.085 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.091+0000 7faaf27fc700 1 -- 172.21.15.69:0/3561671308 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v132) ==== 101+0+0 (secure 0 0 0) 0x7faaf4010040 con 0x7faafc107350 2024-04-02T04:48:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.099+0000 7fab036be700 10 librados: watch_flush enter 2024-04-02T04:48:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.099+0000 7fab036be700 10 librados: watch_flush exit 2024-04-02T04:48:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.099+0000 7fab036be700 20 client.14814.objecter shutdown clearing up homeless session... 2024-04-02T04:48:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.099+0000 7fab036be700 10 client.14814.objecter successfully canceled tick 2024-04-02T04:48:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.099+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faae806f5e0 msgr2=0x7faae8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.099+0000 7fab036be700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faae806f5e0 0x7faae8071aa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faaf80037a0 tx=0x7faaf8005ad0 comp rx=0 tx=0).stop 2024-04-02T04:48:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc107350 msgr2=0x7faafc197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc107350 0x7faafc197700 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faaf400efb0 tx=0x7faaf400c450 comp rx=0 tx=0).stop 2024-04-02T04:48:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 shutdown_connections 2024-04-02T04:48:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faae806f5e0 0x7faae8071aa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc107350 0x7faafc197700 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc101460 0x7faafc196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 --2- 172.21.15.69:0/3561671308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faafc101e20 0x7faafc1971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.103+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 >> 172.21.15.69:0/3561671308 conn(0x7faafc077ea0 msgr2=0x7faafc104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:50.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.107+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 shutdown_connections 2024-04-02T04:48:50.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.107+0000 7fab036be700 1 -- 172.21.15.69:0/3561671308 wait complete. 2024-04-02T04:48:50.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:50.107+0000 7fab036be700 1 librados: shutdown 2024-04-02T04:48:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:48:50.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:50.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:48.279069+0000 osd.5 (osd.5) 111 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:50.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:48.279183+0000 osd.5 (osd.5) 112 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:50.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.031423+0000 osd.0 (osd.0) 115 : cluster 0 4.6 scrub starts 2024-04-02T04:48:50.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.033093+0000 osd.0 (osd.0) 116 : cluster 0 4.6 scrub ok 2024-04-02T04:48:50.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.085625+0000 mon.a (mon.0) 947 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.439997+0000 osd.7 (osd.7) 115 : cluster 0 2.0 scrub starts 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.441315+0000 osd.7 (osd.7) 116 : cluster 0 2.0 scrub ok 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.481191+0000 osd.3 (osd.3) 143 : cluster 0 6.6 scrub starts 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.483154+0000 osd.3 (osd.3) 144 : cluster 0 6.6 scrub ok 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.567087+0000 osd.2 (osd.2) 63 : cluster 0 2.c scrub starts 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.568669+0000 osd.2 (osd.2) 64 : cluster 0 2.c scrub ok 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[21528]: audit 2024-04-02T04:48:49.869571+0000 mon.a (mon.0) 948 : audit 1 from='client.? 172.21.15.69:0/3561671308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:48.279069+0000 osd.5 (osd.5) 111 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:48.279183+0000 osd.5 (osd.5) 112 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.031423+0000 osd.0 (osd.0) 115 : cluster 0 4.6 scrub starts 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.033093+0000 osd.0 (osd.0) 116 : cluster 0 4.6 scrub ok 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.085625+0000 mon.a (mon.0) 947 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-04-02T04:48:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.439997+0000 osd.7 (osd.7) 115 : cluster 0 2.0 scrub starts 2024-04-02T04:48:50.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.441315+0000 osd.7 (osd.7) 116 : cluster 0 2.0 scrub ok 2024-04-02T04:48:50.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.481191+0000 osd.3 (osd.3) 143 : cluster 0 6.6 scrub starts 2024-04-02T04:48:50.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.483154+0000 osd.3 (osd.3) 144 : cluster 0 6.6 scrub ok 2024-04-02T04:48:50.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.567087+0000 osd.2 (osd.2) 63 : cluster 0 2.c scrub starts 2024-04-02T04:48:50.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.568669+0000 osd.2 (osd.2) 64 : cluster 0 2.c scrub ok 2024-04-02T04:48:50.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:50 smithi069 bash[28904]: audit 2024-04-02T04:48:49.869571+0000 mon.a (mon.0) 948 : audit 1 from='client.? 172.21.15.69:0/3561671308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:50.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:50.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:50.337 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:50.440 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:48.279069+0000 osd.5 (osd.5) 111 : cluster 0 purged_snaps scrub starts 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:48.279183+0000 osd.5 (osd.5) 112 : cluster 0 purged_snaps scrub ok 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.031423+0000 osd.0 (osd.0) 115 : cluster 0 4.6 scrub starts 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.033093+0000 osd.0 (osd.0) 116 : cluster 0 4.6 scrub ok 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.085625+0000 mon.a (mon.0) 947 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.439997+0000 osd.7 (osd.7) 115 : cluster 0 2.0 scrub starts 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.441315+0000 osd.7 (osd.7) 116 : cluster 0 2.0 scrub ok 2024-04-02T04:48:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.481191+0000 osd.3 (osd.3) 143 : cluster 0 6.6 scrub starts 2024-04-02T04:48:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.483154+0000 osd.3 (osd.3) 144 : cluster 0 6.6 scrub ok 2024-04-02T04:48:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.567087+0000 osd.2 (osd.2) 63 : cluster 0 2.c scrub starts 2024-04-02T04:48:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.568669+0000 osd.2 (osd.2) 64 : cluster 0 2.c scrub ok 2024-04-02T04:48:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:50 smithi098 bash[23790]: audit 2024-04-02T04:48:49.869571+0000 mon.a (mon.0) 948 : audit 1 from='client.? 172.21.15.69:0/3561671308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:48:50.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:50.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:50.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:50.845 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:50.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:50.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:51.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:51.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:51.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:51.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.529627+0000 osd.4 (osd.4) 119 : cluster 0 3.9 scrub starts 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.534745+0000 osd.4 (osd.4) 120 : cluster 0 3.9 scrub ok 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:49.827790+0000 mgr.y (mgr.14538) 184 : cluster 0 pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:50.090120+0000 mon.a (mon.0) 949 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:50.090151+0000 mon.a (mon.0) 950 : cluster 1 Cluster is now healthy 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: audit 2024-04-02T04:48:50.094784+0000 mon.a (mon.0) 951 : audit 1 from='client.? 172.21.15.69:0/3561671308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:50.108236+0000 mon.a (mon.0) 952 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:50.121070+0000 osd.1 (osd.1) 109 : cluster 0 6.1d scrub starts 2024-04-02T04:48:51.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:51 smithi098 bash[23790]: cluster 2024-04-02T04:48:50.123324+0000 osd.1 (osd.1) 110 : cluster 0 6.1d scrub ok 2024-04-02T04:48:51.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:48:51.234 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:51.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.529627+0000 osd.4 (osd.4) 119 : cluster 0 3.9 scrub starts 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.534745+0000 osd.4 (osd.4) 120 : cluster 0 3.9 scrub ok 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:49.827790+0000 mgr.y (mgr.14538) 184 : cluster 0 pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:50.090120+0000 mon.a (mon.0) 949 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:50.090151+0000 mon.a (mon.0) 950 : cluster 1 Cluster is now healthy 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: audit 2024-04-02T04:48:50.094784+0000 mon.a (mon.0) 951 : audit 1 from='client.? 172.21.15.69:0/3561671308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:48:51.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:50.108236+0000 mon.a (mon.0) 952 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:50.121070+0000 osd.1 (osd.1) 109 : cluster 0 6.1d scrub starts 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[21528]: cluster 2024-04-02T04:48:50.123324+0000 osd.1 (osd.1) 110 : cluster 0 6.1d scrub ok 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.529627+0000 osd.4 (osd.4) 119 : cluster 0 3.9 scrub starts 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.534745+0000 osd.4 (osd.4) 120 : cluster 0 3.9 scrub ok 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:49.827790+0000 mgr.y (mgr.14538) 184 : cluster 0 pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:50.090120+0000 mon.a (mon.0) 949 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:50.090151+0000 mon.a (mon.0) 950 : cluster 1 Cluster is now healthy 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: audit 2024-04-02T04:48:50.094784+0000 mon.a (mon.0) 951 : audit 1 from='client.? 172.21.15.69:0/3561671308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:50.108236+0000 mon.a (mon.0) 952 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:50.121070+0000 osd.1 (osd.1) 109 : cluster 0 6.1d scrub starts 2024-04-02T04:48:51.364 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:51 smithi069 bash[28904]: cluster 2024-04-02T04:48:50.123324+0000 osd.1 (osd.1) 110 : cluster 0 6.1d scrub ok 2024-04-02T04:48:51.387 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:51.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:51.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:51.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:51.774 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:51.877 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:51.904 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:51.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:52.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:52.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:52.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:52.164 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:52.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:52.293 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:52.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:52.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:52.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:52.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:52.808 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:52.812 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:52.915 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:52.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:53.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:53.068 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:53.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:53.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:53.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- 172.21.15.69:0/3490676520 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 msgr2=0x7fa96c10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- 172.21.15.69:0/3490676520 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c10e020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa96400b3e0 tx=0x7fa96400b6f0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- 172.21.15.69:0/3490676520 shutdown_connections 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- 172.21.15.69:0/3490676520 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c10e020 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- 172.21.15.69:0/3490676520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c1071e0 0x7fa96c1075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- 172.21.15.69:0/3490676520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c101170 0x7fa96c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- 172.21.15.69:0/3490676520 >> 172.21.15.69:0/3490676520 conn(0x7fa96c078110 msgr2=0x7fa96c078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- 172.21.15.69:0/3490676520 shutdown_connections 2024-04-02T04:48:53.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- 172.21.15.69:0/3490676520 wait complete. 2024-04-02T04:48:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 librados: starting msgr at 2024-04-02T04:48:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 librados: starting objecter 2024-04-02T04:48:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 Processor -- start 2024-04-02T04:48:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- start start 2024-04-02T04:48:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 librados: setting wanted keys 2024-04-02T04:48:53.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 librados: calling monclient init 2024-04-02T04:48:53.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 0x7fa96c1080c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c108600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1071e0 0x7fa96c108b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c110a40 con 0x7fa96c101170 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1108c0 con 0x7fa96c1071e0 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa970c5f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa96c110bc0 con 0x7fa96c101b30 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 0x7fa96c1080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 0x7fa96c1080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33620/0 (socket says 172.21.15.69:33620) 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 -- 172.21.15.69:0/2757369213 learned_addr learned my addr 172.21.15.69:0/2757369213 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 -- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 msgr2=0x7fa96c108600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96ad9d700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1071e0 0x7fa96c108b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:53.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa969d9b700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c108600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c108600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 -- 172.21.15.69:0/2757369213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1071e0 msgr2=0x7fa96c108b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.183+0000 7fa96a59c700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1071e0 0x7fa96c108b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa96a59c700 1 -- 172.21.15.69:0/2757369213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96c1a6600 con 0x7fa96c101170 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa96ad9d700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1071e0 0x7fa96c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa969d9b700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:48:53.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa96a59c700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 0x7fa96c1080c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa95c012410 tx=0x7fa95c0127d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa95b7fe700 10 client.?.objecter ms_handle_connect 0x7fa96c101170 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa95b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c022070 con 0x7fa96c101170 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa95c015ce0 con 0x7fa96c101170 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c01dd70 con 0x7fa96c101170 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96c197e70 con 0x7fa96c101170 2024-04-02T04:48:53.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa970c5f700 10 client.14817.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:48:53.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96c198360 con 0x7fa96c101170 2024-04-02T04:48:53.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa970c5f700 1 librados: init done 2024-04-02T04:48:53.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.187+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96c102890 con 0x7fa96c101170 2024-04-02T04:48:53.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa95c015860 con 0x7fa96c101170 2024-04-02T04:48:53.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa95406f6b0 0x7fa954071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:48:53.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(132..132 src has 74..132) ==== 7537+0+0 (secure 0 0 0) 0x7fa95c096350 con 0x7fa96c101170 2024-04-02T04:48:53.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 10 client.14817.objecter ms_dispatch 0x7fa96c001550 osd_map(132..132 src has 74..132) 2024-04-02T04:48:53.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 3 client.14817.objecter handle_osd_map got epochs [132,132] > 0 2024-04-02T04:48:53.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 3 client.14817.objecter handle_osd_map decoding full epoch 132 2024-04-02T04:48:53.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 20 client.14817.objecter dump_active .. 0 homeless 2024-04-02T04:48:53.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa969d9b700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa95406f6b0 0x7fa954071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:48:53.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa969d9b700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa95406f6b0 0x7fa954071b70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0788c0 tx=0x7fa960009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:48:53.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.191+0000 7fa95b7fe700 10 client.14817.objecter ms_handle_connect 0x7fa95406f6b0 2024-04-02T04:48:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.199+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa95c063a50 con 0x7fa96c101170 2024-04-02T04:48:53.202 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:53.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:48:52 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:48:53.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:53.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:48:53.331 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:53.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:53.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:51.828181+0000 mgr.y (mgr.14538) 185 : cluster 0 pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-04-02T04:48:53.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:51.867767+0000 osd.5 (osd.5) 113 : cluster 0 2.18 scrub starts 2024-04-02T04:48:53.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:51.870028+0000 osd.5 (osd.5) 114 : cluster 0 2.18 scrub ok 2024-04-02T04:48:53.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:52.071407+0000 osd.0 (osd.0) 117 : cluster 0 3.17 scrub starts 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:52.073754+0000 osd.0 (osd.0) 118 : cluster 0 3.17 scrub ok 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: audit 2024-04-02T04:48:52.238354+0000 mon.a (mon.0) 953 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: audit 2024-04-02T04:48:52.239641+0000 mon.b (mon.1) 104 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:52.388242+0000 osd.7 (osd.7) 117 : cluster 0 2.5 scrub starts 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[21528]: cluster 2024-04-02T04:48:52.389758+0000 osd.7 (osd.7) 118 : cluster 0 2.5 scrub ok 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:51.828181+0000 mgr.y (mgr.14538) 185 : cluster 0 pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:51.867767+0000 osd.5 (osd.5) 113 : cluster 0 2.18 scrub starts 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:51.870028+0000 osd.5 (osd.5) 114 : cluster 0 2.18 scrub ok 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:52.071407+0000 osd.0 (osd.0) 117 : cluster 0 3.17 scrub starts 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:52.073754+0000 osd.0 (osd.0) 118 : cluster 0 3.17 scrub ok 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: audit 2024-04-02T04:48:52.238354+0000 mon.a (mon.0) 953 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:53.447 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: audit 2024-04-02T04:48:52.239641+0000 mon.b (mon.1) 104 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:53.448 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:52.388242+0000 osd.7 (osd.7) 117 : cluster 0 2.5 scrub starts 2024-04-02T04:48:53.448 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:53 smithi069 bash[28904]: cluster 2024-04-02T04:48:52.389758+0000 osd.7 (osd.7) 118 : cluster 0 2.5 scrub ok 2024-04-02T04:48:53.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:53.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:51.828181+0000 mgr.y (mgr.14538) 185 : cluster 0 pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:51.867767+0000 osd.5 (osd.5) 113 : cluster 0 2.18 scrub starts 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:51.870028+0000 osd.5 (osd.5) 114 : cluster 0 2.18 scrub ok 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:52.071407+0000 osd.0 (osd.0) 117 : cluster 0 3.17 scrub starts 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:52.073754+0000 osd.0 (osd.0) 118 : cluster 0 3.17 scrub ok 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: audit 2024-04-02T04:48:52.238354+0000 mon.a (mon.0) 953 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: audit 2024-04-02T04:48:52.239641+0000 mon.b (mon.1) 104 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:52.388242+0000 osd.7 (osd.7) 117 : cluster 0 2.5 scrub starts 2024-04-02T04:48:53.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:53 smithi098 bash[23790]: cluster 2024-04-02T04:48:52.389758+0000 osd.7 (osd.7) 118 : cluster 0 2.5 scrub ok 2024-04-02T04:48:53.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.595+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa96c100420 con 0x7fa95406f6b0 2024-04-02T04:48:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.607+0000 7fa95b7fe700 1 -- 172.21.15.69:0/2757369213 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+345093 (secure 0 0 0) 0x7fa96c100420 con 0x7fa95406f6b0 2024-04-02T04:48:53.607 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:48:53.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.611+0000 7fa970c5f700 10 librados: watch_flush enter 2024-04-02T04:48:53.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.611+0000 7fa970c5f700 10 librados: watch_flush exit 2024-04-02T04:48:53.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.611+0000 7fa970c5f700 20 client.14817.objecter shutdown clearing up homeless session... 2024-04-02T04:48:53.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.611+0000 7fa970c5f700 10 client.14817.objecter successfully canceled tick 2024-04-02T04:48:53.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa95406f6b0 msgr2=0x7fa954071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:53.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa95406f6b0 0x7fa954071b70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0788c0 tx=0x7fa960009260 comp rx=0 tx=0).stop 2024-04-02T04:48:53.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 msgr2=0x7fa96c1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 0x7fa96c1080c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa95c012410 tx=0x7fa95c0127d0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 shutdown_connections 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa95406f6b0 0x7fa954071b70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa96c1071e0 0x7fa96c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa96c101170 0x7fa96c1080c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 --2- 172.21.15.69:0/2757369213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa96c101b30 0x7fa96c108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.615+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 >> 172.21.15.69:0/2757369213 conn(0x7fa96c078110 msgr2=0x7fa96c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.619+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 shutdown_connections 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.619+0000 7fa970c5f700 1 -- 172.21.15.69:0/2757369213 wait complete. 2024-04-02T04:48:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:48:53.619+0000 7fa970c5f700 1 librados: shutdown 2024-04-02T04:48:53.626 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:48:53.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:53.731 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:53.834 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:53.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:53.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:53.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:54.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:54.115 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:54.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:54.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:54.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:54.490 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:54 smithi098 bash[23790]: cluster 2024-04-02T04:48:52.545177+0000 osd.4 (osd.4) 121 : cluster 0 4.c scrub starts 2024-04-02T04:48:54.490 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:54 smithi098 bash[23790]: cluster 2024-04-02T04:48:52.546505+0000 osd.4 (osd.4) 122 : cluster 0 4.c scrub ok 2024-04-02T04:48:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:54 smithi098 bash[23790]: audit 2024-04-02T04:48:52.958443+0000 mgr.y (mgr.14538) 186 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:54 smithi098 bash[23790]: cluster 2024-04-02T04:48:53.451558+0000 osd.3 (osd.3) 145 : cluster 0 5.0 scrub starts 2024-04-02T04:48:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:54 smithi098 bash[23790]: cluster 2024-04-02T04:48:53.454455+0000 osd.3 (osd.3) 146 : cluster 0 5.0 scrub ok 2024-04-02T04:48:54.491 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:54 smithi098 bash[23790]: audit 2024-04-02T04:48:53.597782+0000 mgr.y (mgr.14538) 187 : audit 0 from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[21528]: cluster 2024-04-02T04:48:52.545177+0000 osd.4 (osd.4) 121 : cluster 0 4.c scrub starts 2024-04-02T04:48:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[21528]: cluster 2024-04-02T04:48:52.546505+0000 osd.4 (osd.4) 122 : cluster 0 4.c scrub ok 2024-04-02T04:48:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[21528]: audit 2024-04-02T04:48:52.958443+0000 mgr.y (mgr.14538) 186 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[21528]: cluster 2024-04-02T04:48:53.451558+0000 osd.3 (osd.3) 145 : cluster 0 5.0 scrub starts 2024-04-02T04:48:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[21528]: cluster 2024-04-02T04:48:53.454455+0000 osd.3 (osd.3) 146 : cluster 0 5.0 scrub ok 2024-04-02T04:48:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[21528]: audit 2024-04-02T04:48:53.597782+0000 mgr.y (mgr.14538) 187 : audit 0 from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[28904]: cluster 2024-04-02T04:48:52.545177+0000 osd.4 (osd.4) 121 : cluster 0 4.c scrub starts 2024-04-02T04:48:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[28904]: cluster 2024-04-02T04:48:52.546505+0000 osd.4 (osd.4) 122 : cluster 0 4.c scrub ok 2024-04-02T04:48:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[28904]: audit 2024-04-02T04:48:52.958443+0000 mgr.y (mgr.14538) 186 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:48:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[28904]: cluster 2024-04-02T04:48:53.451558+0000 osd.3 (osd.3) 145 : cluster 0 5.0 scrub starts 2024-04-02T04:48:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[28904]: cluster 2024-04-02T04:48:53.454455+0000 osd.3 (osd.3) 146 : cluster 0 5.0 scrub ok 2024-04-02T04:48:54.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:54 smithi069 bash[28904]: audit 2024-04-02T04:48:53.597782+0000 mgr.y (mgr.14538) 187 : audit 0 from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:48:54.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:54.707 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-04-02T04:48:51.827974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1508,"num_read_kb":1322,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240588,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720628,"statfs":{"total":767960285184,"available":767713923072,"internally_reserved":0,"allocated":7991296,"data_stored":5205178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.766116"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":120,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061913+0000","last_change":"2024-04-02T04:47:14.337462+0000","last_active":"2024-04-02T04:48:44.061913+0000","last_peered":"2024-04-02T04:48:44.061913+0000","last_clean":"2024-04-02T04:48:44.061913+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:48:44.061913+0000","last_undegraded":"2024-04-02T04:48:44.061913+0000","last_fullsized":"2024-04-02T04:48:44.061913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.337178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061739+0000","last_change":"2024-04-02T04:47:52.450415+0000","last_active":"2024-04-02T04:48:44.061739+0000","last_peered":"2024-04-02T04:48:44.061739+0000","last_clean":"2024-04-02T04:48:44.061739+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:48:44.061739+0000","last_undegraded":"2024-04-02T04:48:44.061739+0000","last_fullsized":"2024-04-02T04:48:44.061739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:52.450181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:52.450181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:59.502057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.066732+0000","last_change":"2024-04-02T04:47:54.214859+0000","last_active":"2024-04-02T04:48:44.066732+0000","last_peered":"2024-04-02T04:48:44.066732+0000","last_clean":"2024-04-02T04:48:44.066732+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:48:44.066732+0000","last_undegraded":"2024-04-02T04:48:44.066732+0000","last_fullsized":"2024-04-02T04:48:44.066732+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.434714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":164,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.063270+0000","last_change":"2024-04-02T04:48:41.484207+0000","last_active":"2024-04-02T04:48:44.063270+0000","last_peered":"2024-04-02T04:48:44.063270+0000","last_clean":"2024-04-02T04:48:44.063270+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:48:44.063270+0000","last_undegraded":"2024-04-02T04:48:44.063270+0000","last_fullsized":"2024-04-02T04:48:44.063270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:48:41.484047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.484047+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:47.548311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060461+0000","last_change":"2024-04-02T04:47:11.925670+0000","last_active":"2024-04-02T04:48:44.060461+0000","last_peered":"2024-04-02T04:48:44.060461+0000","last_clean":"2024-04-02T04:48:44.060461+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:48:44.060461+0000","last_undegraded":"2024-04-02T04:48:44.060461+0000","last_fullsized":"2024-04-02T04:48:44.060461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:11.925452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:11.925452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":123,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060223+0000","last_change":"2024-04-02T04:47:07.889053+0000","last_active":"2024-04-02T04:48:44.060223+0000","last_peered":"2024-04-02T04:48:44.060223+0000","last_clean":"2024-04-02T04:48:44.060223+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:48:44.060223+0000","last_undegraded":"2024-04-02T04:48:44.060223+0000","last_fullsized":"2024-04-02T04:48:44.060223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:07.888835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:07.888835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.063031+0000","last_change":"2024-04-02T04:47:42.551341+0000","last_active":"2024-04-02T04:48:44.063031+0000","last_peered":"2024-04-02T04:48:44.063031+0000","last_clean":"2024-04-02T04:48:44.063031+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:48:44.063031+0000","last_undegraded":"2024-04-02T04:48:44.063031+0000","last_fullsized":"2024-04-02T04:48:44.063031+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.551243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.551243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:59.906138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":160,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.066580+0000","last_change":"2024-04-02T04:46:45.396999+0000","last_active":"2024-04-02T04:48:44.066580+0000","last_peered":"2024-04-02T04:48:44.066580+0000","last_clean":"2024-04-02T04:48:44.066580+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:48:44.066580+0000","last_undegraded":"2024-04-02T04:48:44.066580+0000","last_fullsized":"2024-04-02T04:48:44.066580+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:46:45.396760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.396760+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":126,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.521860+0000","last_change":"2024-04-02T04:47:42.186212+0000","last_active":"2024-04-02T04:48:44.521860+0000","last_peered":"2024-04-02T04:48:44.521860+0000","last_clean":"2024-04-02T04:48:44.521860+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:48:44.521860+0000","last_undegraded":"2024-04-02T04:48:44.521860+0000","last_fullsized":"2024-04-02T04:48:44.521860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.185988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.185988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.015773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":131,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060881+0000","last_change":"2024-04-02T04:47:37.876855+0000","last_active":"2024-04-02T04:48:44.060881+0000","last_peered":"2024-04-02T04:48:44.060881+0000","last_clean":"2024-04-02T04:48:44.060881+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:48:44.060881+0000","last_undegraded":"2024-04-02T04:48:44.060881+0000","last_fullsized":"2024-04-02T04:48:44.060881+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.876644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.876644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.719969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.425088+0000","last_change":"2024-04-02T04:47:45.254673+0000","last_active":"2024-04-02T04:48:44.425088+0000","last_peered":"2024-04-02T04:48:44.425088+0000","last_clean":"2024-04-02T04:48:44.425088+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:48:44.425088+0000","last_undegraded":"2024-04-02T04:48:44.425088+0000","last_fullsized":"2024-04-02T04:48:44.425088+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.254450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.254450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:58.071917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":159,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:46.888003+0000","last_change":"2024-04-02T04:48:46.888055+0000","last_active":"2024-04-02T04:48:46.888003+0000","last_peered":"2024-04-02T04:48:46.888003+0000","last_clean":"2024-04-02T04:48:46.888003+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:48:46.888003+0000","last_undegraded":"2024-04-02T04:48:46.888003+0000","last_fullsized":"2024-04-02T04:48:46.888003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:46.887903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.887903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:14.156917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.067158+0000","last_change":"2024-04-02T04:48:02.193707+0000","last_active":"2024-04-02T04:48:44.067158+0000","last_peered":"2024-04-02T04:48:44.067158+0000","last_clean":"2024-04-02T04:48:44.067158+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:48:44.067158+0000","last_undegraded":"2024-04-02T04:48:44.067158+0000","last_fullsized":"2024-04-02T04:48:44.067158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:31.172290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":123,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061213+0000","last_change":"2024-04-02T04:46:44.775289+0000","last_active":"2024-04-02T04:48:44.061213+0000","last_peered":"2024-04-02T04:48:44.061213+0000","last_clean":"2024-04-02T04:48:44.061213+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:48:44.061213+0000","last_undegraded":"2024-04-02T04:48:44.061213+0000","last_fullsized":"2024-04-02T04:48:44.061213+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:44.775071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:44.775071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.424753+0000","last_change":"2024-04-02T04:47:09.387603+0000","last_active":"2024-04-02T04:48:44.424753+0000","last_peered":"2024-04-02T04:48:44.424753+0000","last_clean":"2024-04-02T04:48:44.424753+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:48:44.424753+0000","last_undegraded":"2024-04-02T04:48:44.424753+0000","last_fullsized":"2024-04-02T04:48:44.424753+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:09.387375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.387375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":169,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.521533+0000","last_change":"2024-04-02T04:48:43.963496+0000","last_active":"2024-04-02T04:48:44.521533+0000","last_peered":"2024-04-02T04:48:44.521533+0000","last_clean":"2024-04-02T04:48:44.521533+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:48:44.521533+0000","last_undegraded":"2024-04-02T04:48:44.521533+0000","last_fullsized":"2024-04-02T04:48:44.521533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:48:43.963234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.963234+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:56.321677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":121,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062474+0000","last_change":"2024-04-02T04:47:03.158759+0000","last_active":"2024-04-02T04:48:44.062474+0000","last_peered":"2024-04-02T04:48:44.062474+0000","last_clean":"2024-04-02T04:48:44.062474+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:48:44.062474+0000","last_undegraded":"2024-04-02T04:48:44.062474+0000","last_fullsized":"2024-04-02T04:48:44.062474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:03.158541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:03.158541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062688+0000","last_change":"2024-04-02T04:47:57.459178+0000","last_active":"2024-04-02T04:48:44.062688+0000","last_peered":"2024-04-02T04:48:44.062688+0000","last_clean":"2024-04-02T04:48:44.062688+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:48:44.062688+0000","last_undegraded":"2024-04-02T04:48:44.062688+0000","last_fullsized":"2024-04-02T04:48:44.062688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:57.459040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.459040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:17.124034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":140,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062124+0000","last_change":"2024-04-02T04:47:18.359335+0000","last_active":"2024-04-02T04:48:44.062124+0000","last_peered":"2024-04-02T04:48:44.062124+0000","last_clean":"2024-04-02T04:48:44.062124+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:48:44.062124+0000","last_undegraded":"2024-04-02T04:48:44.062124+0000","last_fullsized":"2024-04-02T04:48:44.062124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:18.359101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:18.359101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":183,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060665+0000","last_change":"2024-04-02T04:47:40.838867+0000","last_active":"2024-04-02T04:48:44.060665+0000","last_peered":"2024-04-02T04:48:44.060665+0000","last_clean":"2024-04-02T04:48:44.060665+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:48:44.060665+0000","last_undegraded":"2024-04-02T04:48:44.060665+0000","last_fullsized":"2024-04-02T04:48:44.060665+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.838617+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.066994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061769+0000","last_change":"2024-04-02T04:48:38.627470+0000","last_active":"2024-04-02T04:48:44.061769+0000","last_peered":"2024-04-02T04:48:44.061769+0000","last_clean":"2024-04-02T04:48:44.061769+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:48:44.061769+0000","last_undegraded":"2024-04-02T04:48:44.061769+0000","last_fullsized":"2024-04-02T04:48:44.061769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:38.627239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.627239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.164715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":130,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.643843+0000","last_change":"2024-04-02T04:47:45.830668+0000","last_active":"2024-04-02T04:48:44.643843+0000","last_peered":"2024-04-02T04:48:44.643843+0000","last_clean":"2024-04-02T04:48:44.643843+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:48:44.643843+0000","last_undegraded":"2024-04-02T04:48:44.643843+0000","last_fullsized":"2024-04-02T04:48:44.643843+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.830369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.830369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.027610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":140,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.606038+0000","last_change":"2024-04-02T04:47:53.938679+0000","last_active":"2024-04-02T04:48:44.606038+0000","last_peered":"2024-04-02T04:48:44.606038+0000","last_clean":"2024-04-02T04:48:44.606038+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:48:44.606038+0000","last_undegraded":"2024-04-02T04:48:44.606038+0000","last_fullsized":"2024-04-02T04:48:44.606038+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.938537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.938537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:12.383253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.065692+0000","last_change":"2024-04-02T04:47:13.347780+0000","last_active":"2024-04-02T04:48:44.065692+0000","last_peered":"2024-04-02T04:48:44.065692+0000","last_clean":"2024-04-02T04:48:44.065692+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:48:44.065692+0000","last_undegraded":"2024-04-02T04:48:44.065692+0000","last_fullsized":"2024-04-02T04:48:44.065692+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.347670+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912596+0000","last_change":"2024-04-02T04:46:56.263880+0000","last_active":"2024-04-02T04:48:44.912596+0000","last_peered":"2024-04-02T04:48:44.912596+0000","last_clean":"2024-04-02T04:48:44.912596+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:48:44.912596+0000","last_undegraded":"2024-04-02T04:48:44.912596+0000","last_fullsized":"2024-04-02T04:48:44.912596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:56.263740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:46:56.263740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":205,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.606008+0000","last_change":"2024-04-02T04:48:43.630108+0000","last_active":"2024-04-02T04:48:44.606008+0000","last_peered":"2024-04-02T04:48:44.606008+0000","last_clean":"2024-04-02T04:48:44.606008+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:48:44.606008+0000","last_undegraded":"2024-04-02T04:48:44.606008+0000","last_fullsized":"2024-04-02T04:48:44.606008+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:43.629796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.629796+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:44.617590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":143,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061482+0000","last_change":"2024-04-02T04:47:58.472809+0000","last_active":"2024-04-02T04:48:44.061482+0000","last_peered":"2024-04-02T04:48:44.061482+0000","last_clean":"2024-04-02T04:48:44.061482+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:48:44.061482+0000","last_undegraded":"2024-04-02T04:48:44.061482+0000","last_fullsized":"2024-04-02T04:48:44.061482+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:47:58.472703+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.472703+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:08.433557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061343+0000","last_change":"2024-04-02T04:47:45.516535+0000","last_active":"2024-04-02T04:48:44.061343+0000","last_peered":"2024-04-02T04:48:44.061343+0000","last_clean":"2024-04-02T04:48:44.061343+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:48:44.061343+0000","last_undegraded":"2024-04-02T04:48:44.061343+0000","last_fullsized":"2024-04-02T04:48:44.061343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:45.516407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.516407+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.285568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":129,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.424351+0000","last_change":"2024-04-02T04:48:42.394472+0000","last_active":"2024-04-02T04:48:44.424351+0000","last_peered":"2024-04-02T04:48:44.424351+0000","last_clean":"2024-04-02T04:48:44.424351+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:48:44.424351+0000","last_undegraded":"2024-04-02T04:48:44.424351+0000","last_fullsized":"2024-04-02T04:48:44.424351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:48:42.394228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.394228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:10.561833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":133,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.068040+0000","last_change":"2024-04-02T04:47:39.172102+0000","last_active":"2024-04-02T04:48:44.068040+0000","last_peered":"2024-04-02T04:48:44.068040+0000","last_clean":"2024-04-02T04:48:44.068040+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:48:44.068040+0000","last_undegraded":"2024-04-02T04:48:44.068040+0000","last_fullsized":"2024-04-02T04:48:44.068040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:39.171955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.171955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:54.119072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.521783+0000","last_change":"2024-04-02T04:47:27.380112+0000","last_active":"2024-04-02T04:48:44.521783+0000","last_peered":"2024-04-02T04:48:44.521783+0000","last_clean":"2024-04-02T04:48:44.521783+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:48:44.521783+0000","last_undegraded":"2024-04-02T04:48:44.521783+0000","last_fullsized":"2024-04-02T04:48:44.521783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:27.379800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:27.379800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'39","reported_seq":200,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.911476+0000","last_change":"2024-04-02T04:46:59.265375+0000","last_active":"2024-04-02T04:48:44.911476+0000","last_peered":"2024-04-02T04:48:44.911476+0000","last_clean":"2024-04-02T04:48:44.911476+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:48:44.911476+0000","last_undegraded":"2024-04-02T04:48:44.911476+0000","last_fullsized":"2024-04-02T04:48:44.911476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'39","last_scrub_stamp":"2024-04-02T04:46:59.265104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:59.265104+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":121,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062413+0000","last_change":"2024-04-02T04:47:13.288506+0000","last_active":"2024-04-02T04:48:44.062413+0000","last_peered":"2024-04-02T04:48:44.062413+0000","last_clean":"2024-04-02T04:48:44.062413+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:48:44.062413+0000","last_undegraded":"2024-04-02T04:48:44.062413+0000","last_fullsized":"2024-04-02T04:48:44.062413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:13.288364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.288364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061348+0000","last_change":"2024-04-02T04:47:12.271950+0000","last_active":"2024-04-02T04:48:44.061348+0000","last_peered":"2024-04-02T04:48:44.061348+0000","last_clean":"2024-04-02T04:48:44.061348+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:48:44.061348+0000","last_undegraded":"2024-04-02T04:48:44.061348+0000","last_fullsized":"2024-04-02T04:48:44.061348+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:12.271680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:12.271680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:46.602957+0000","last_change":"2024-04-02T04:48:46.602957+0000","last_active":"2024-04-02T04:48:46.602957+0000","last_peered":"2024-04-02T04:48:46.602957+0000","last_clean":"2024-04-02T04:48:46.602957+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:48:46.602957+0000","last_undegraded":"2024-04-02T04:48:46.602957+0000","last_fullsized":"2024-04-02T04:48:46.602957+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:46.602849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.602849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:07.952150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":175,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.522686+0000","last_change":"2024-04-02T04:48:01.257149+0000","last_active":"2024-04-02T04:48:44.522686+0000","last_peered":"2024-04-02T04:48:44.522686+0000","last_clean":"2024-04-02T04:48:44.522686+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:48:44.522686+0000","last_undegraded":"2024-04-02T04:48:44.522686+0000","last_fullsized":"2024-04-02T04:48:44.522686+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:48:01.256898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:01.256898+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.641480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":120,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.423737+0000","last_change":"2024-04-02T04:47:20.427195+0000","last_active":"2024-04-02T04:48:44.423737+0000","last_peered":"2024-04-02T04:48:44.423737+0000","last_clean":"2024-04-02T04:48:44.423737+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:48:44.423737+0000","last_undegraded":"2024-04-02T04:48:44.423737+0000","last_fullsized":"2024-04-02T04:48:44.423737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:20.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:47:20.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:48.706473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":133,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061392+0000","last_change":"2024-04-02T04:47:19.398500+0000","last_active":"2024-04-02T04:48:44.061392+0000","last_peered":"2024-04-02T04:48:44.061392+0000","last_clean":"2024-04-02T04:48:44.061392+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:48:44.061392+0000","last_undegraded":"2024-04-02T04:48:44.061392+0000","last_fullsized":"2024-04-02T04:48:44.061392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:19.398328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:19.398328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.522570+0000","last_change":"2024-04-02T04:47:46.278114+0000","last_active":"2024-04-02T04:48:44.522570+0000","last_peered":"2024-04-02T04:48:44.522570+0000","last_clean":"2024-04-02T04:48:44.522570+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:48:44.522570+0000","last_undegraded":"2024-04-02T04:48:44.522570+0000","last_fullsized":"2024-04-02T04:48:44.522570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:46.277889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:47:46.277889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.753634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":150,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.066525+0000","last_change":"2024-04-02T04:47:08.333929+0000","last_active":"2024-04-02T04:48:44.066525+0000","last_peered":"2024-04-02T04:48:44.066525+0000","last_clean":"2024-04-02T04:48:44.066525+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:48:44.066525+0000","last_undegraded":"2024-04-02T04:48:44.066525+0000","last_fullsized":"2024-04-02T04:48:44.066525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:47:08.333516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:08.333516+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":129,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.527762+0000","last_change":"2024-04-02T04:48:44.527799+0000","last_active":"2024-04-02T04:48:44.527762+0000","last_peered":"2024-04-02T04:48:44.527762+0000","last_clean":"2024-04-02T04:48:44.527762+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:48:44.527762+0000","last_undegraded":"2024-04-02T04:48:44.527762+0000","last_fullsized":"2024-04-02T04:48:44.527762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.527676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.527676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:00.396302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061128+0000","last_change":"2024-04-02T04:47:31.779975+0000","last_active":"2024-04-02T04:48:44.061128+0000","last_peered":"2024-04-02T04:48:44.061128+0000","last_clean":"2024-04-02T04:48:44.061128+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:48:44.061128+0000","last_undegraded":"2024-04-02T04:48:44.061128+0000","last_fullsized":"2024-04-02T04:48:44.061128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:31.779694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:31.779694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912967+0000","last_change":"2024-04-02T04:46:54.269554+0000","last_active":"2024-04-02T04:48:44.912967+0000","last_peered":"2024-04-02T04:48:44.912967+0000","last_clean":"2024-04-02T04:48:44.912967+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:48:44.912967+0000","last_undegraded":"2024-04-02T04:48:44.912967+0000","last_fullsized":"2024-04-02T04:48:44.912967+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:54.269343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:46:54.269343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":175,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061616+0000","last_change":"2024-04-02T04:48:00.691627+0000","last_active":"2024-04-02T04:48:44.061616+0000","last_peered":"2024-04-02T04:48:44.061616+0000","last_clean":"2024-04-02T04:48:44.061616+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:48:44.061616+0000","last_undegraded":"2024-04-02T04:48:44.061616+0000","last_fullsized":"2024-04-02T04:48:44.061616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:26.639529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":129,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.067275+0000","last_change":"2024-04-02T04:48:40.066400+0000","last_active":"2024-04-02T04:48:44.067275+0000","last_peered":"2024-04-02T04:48:44.067275+0000","last_clean":"2024-04-02T04:48:44.067275+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:48:44.067275+0000","last_undegraded":"2024-04-02T04:48:44.067275+0000","last_fullsized":"2024-04-02T04:48:44.067275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.066265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.066265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.667647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":124,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.913316+0000","last_change":"2024-04-02T04:46:53.297425+0000","last_active":"2024-04-02T04:48:44.913316+0000","last_peered":"2024-04-02T04:48:44.913316+0000","last_clean":"2024-04-02T04:48:44.913316+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:48:44.913316+0000","last_undegraded":"2024-04-02T04:48:44.913316+0000","last_fullsized":"2024-04-02T04:48:44.913316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:53.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:53.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.606048+0000","last_change":"2024-04-02T04:48:42.660369+0000","last_active":"2024-04-02T04:48:44.606048+0000","last_peered":"2024-04-02T04:48:44.606048+0000","last_clean":"2024-04-02T04:48:44.606048+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:48:44.606048+0000","last_undegraded":"2024-04-02T04:48:44.606048+0000","last_fullsized":"2024-04-02T04:48:44.606048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.660246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.660246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:02.575601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":164,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.422715+0000","last_change":"2024-04-02T04:47:36.370736+0000","last_active":"2024-04-02T04:48:44.422715+0000","last_peered":"2024-04-02T04:48:44.422715+0000","last_clean":"2024-04-02T04:48:44.422715+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:48:44.422715+0000","last_undegraded":"2024-04-02T04:48:44.422715+0000","last_fullsized":"2024-04-02T04:48:44.422715+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:36.370493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.370493+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.329887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062508+0000","last_change":"2024-04-02T04:47:05.885138+0000","last_active":"2024-04-02T04:48:44.062508+0000","last_peered":"2024-04-02T04:48:44.062508+0000","last_clean":"2024-04-02T04:48:44.062508+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:48:44.062508+0000","last_undegraded":"2024-04-02T04:48:44.062508+0000","last_fullsized":"2024-04-02T04:48:44.062508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":203,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.528545+0000","last_change":"2024-04-02T04:48:41.932434+0000","last_active":"2024-04-02T04:48:44.528545+0000","last_peered":"2024-04-02T04:48:44.528545+0000","last_clean":"2024-04-02T04:48:44.528545+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:48:44.528545+0000","last_undegraded":"2024-04-02T04:48:44.528545+0000","last_fullsized":"2024-04-02T04:48:44.528545+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:41.932321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.932321+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:07.046064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:46.074033+0000","last_change":"2024-04-02T04:48:46.074077+0000","last_active":"2024-04-02T04:48:46.074033+0000","last_peered":"2024-04-02T04:48:46.074033+0000","last_clean":"2024-04-02T04:48:46.074033+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:48:46.074033+0000","last_undegraded":"2024-04-02T04:48:46.074033+0000","last_fullsized":"2024-04-02T04:48:46.074033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:46.073944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.073944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:00.740924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.065333+0000","last_change":"2024-04-02T04:47:53.179435+0000","last_active":"2024-04-02T04:48:44.065333+0000","last_peered":"2024-04-02T04:48:44.065333+0000","last_clean":"2024-04-02T04:48:44.065333+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:48:44.065333+0000","last_undegraded":"2024-04-02T04:48:44.065333+0000","last_fullsized":"2024-04-02T04:48:44.065333+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:47:53.179316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.179316+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.558109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":120,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.067871+0000","last_change":"2024-04-02T04:47:09.295716+0000","last_active":"2024-04-02T04:48:44.067871+0000","last_peered":"2024-04-02T04:48:44.067871+0000","last_clean":"2024-04-02T04:48:44.067871+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:48:44.067871+0000","last_undegraded":"2024-04-02T04:48:44.067871+0000","last_fullsized":"2024-04-02T04:48:44.067871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:09.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:47:09.295498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"121'12","reported_seq":203,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062487+0000","last_change":"2024-04-02T04:48:02.448348+0000","last_active":"2024-04-02T04:48:44.062487+0000","last_peered":"2024-04-02T04:48:44.062487+0000","last_clean":"2024-04-02T04:48:44.062487+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:48:44.062487+0000","last_undegraded":"2024-04-02T04:48:44.062487+0000","last_fullsized":"2024-04-02T04:48:44.062487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:07.814627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.522007+0000","last_change":"2024-04-02T04:48:40.899464+0000","last_active":"2024-04-02T04:48:44.522007+0000","last_peered":"2024-04-02T04:48:44.522007+0000","last_clean":"2024-04-02T04:48:44.522007+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:48:44.522007+0000","last_undegraded":"2024-04-02T04:48:44.522007+0000","last_fullsized":"2024-04-02T04:48:44.522007+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.899356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.899356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.480969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":155,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.422301+0000","last_change":"2024-04-02T04:48:00.302558+0000","last_active":"2024-04-02T04:48:44.422301+0000","last_peered":"2024-04-02T04:48:44.422301+0000","last_clean":"2024-04-02T04:48:44.422301+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:48:44.422301+0000","last_undegraded":"2024-04-02T04:48:44.422301+0000","last_fullsized":"2024-04-02T04:48:44.422301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:26.019165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.424598+0000","last_change":"2024-04-02T04:47:04.415949+0000","last_active":"2024-04-02T04:48:44.424598+0000","last_peered":"2024-04-02T04:48:44.424598+0000","last_clean":"2024-04-02T04:48:44.424598+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:48:44.424598+0000","last_undegraded":"2024-04-02T04:48:44.424598+0000","last_fullsized":"2024-04-02T04:48:44.424598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:04.415735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.415735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":130,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062568+0000","last_change":"2024-04-02T04:46:45.921390+0000","last_active":"2024-04-02T04:48:44.062568+0000","last_peered":"2024-04-02T04:48:44.062568+0000","last_clean":"2024-04-02T04:48:44.062568+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:48:44.062568+0000","last_undegraded":"2024-04-02T04:48:44.062568+0000","last_fullsized":"2024-04-02T04:48:44.062568+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:46:45.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.605172+0000","last_change":"2024-04-02T04:47:35.160389+0000","last_active":"2024-04-02T04:48:44.605172+0000","last_peered":"2024-04-02T04:48:44.605172+0000","last_clean":"2024-04-02T04:48:44.605172+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:48:44.605172+0000","last_undegraded":"2024-04-02T04:48:44.605172+0000","last_fullsized":"2024-04-02T04:48:44.605172+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.160172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.160172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.333854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":143,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.604537+0000","last_change":"2024-04-02T04:48:41.679887+0000","last_active":"2024-04-02T04:48:44.604537+0000","last_peered":"2024-04-02T04:48:44.604537+0000","last_clean":"2024-04-02T04:48:44.604537+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:48:44.604537+0000","last_undegraded":"2024-04-02T04:48:44.604537+0000","last_fullsized":"2024-04-02T04:48:44.604537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:48:41.679498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.679498+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:59.901463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.646796+0000","last_change":"2024-04-02T04:48:44.646833+0000","last_active":"2024-04-02T04:48:44.646796+0000","last_peered":"2024-04-02T04:48:44.646796+0000","last_clean":"2024-04-02T04:48:44.646796+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:48:44.646796+0000","last_undegraded":"2024-04-02T04:48:44.646796+0000","last_fullsized":"2024-04-02T04:48:44.646796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.646709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.646709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:52.485095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":124,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.066910+0000","last_change":"2024-04-02T04:46:57.429067+0000","last_active":"2024-04-02T04:48:44.066910+0000","last_peered":"2024-04-02T04:48:44.066910+0000","last_clean":"2024-04-02T04:48:44.066910+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:48:44.066910+0000","last_undegraded":"2024-04-02T04:48:44.066910+0000","last_fullsized":"2024-04-02T04:48:44.066910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:57.428901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:57.428901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.643196+0000","last_change":"2024-04-02T04:47:49.839212+0000","last_active":"2024-04-02T04:48:44.643196+0000","last_peered":"2024-04-02T04:48:44.643196+0000","last_clean":"2024-04-02T04:48:44.643196+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:48:44.643196+0000","last_undegraded":"2024-04-02T04:48:44.643196+0000","last_fullsized":"2024-04-02T04:48:44.643196+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.839000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:09.914868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":159,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.422448+0000","last_change":"2024-04-02T04:47:00.341837+0000","last_active":"2024-04-02T04:48:44.422448+0000","last_peered":"2024-04-02T04:48:44.422448+0000","last_clean":"2024-04-02T04:48:44.422448+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:48:44.422448+0000","last_undegraded":"2024-04-02T04:48:44.422448+0000","last_fullsized":"2024-04-02T04:48:44.422448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:00.341473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.341473+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":121,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062321+0000","last_change":"2024-04-02T04:47:00.098587+0000","last_active":"2024-04-02T04:48:44.062321+0000","last_peered":"2024-04-02T04:48:44.062321+0000","last_clean":"2024-04-02T04:48:44.062321+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:48:44.062321+0000","last_undegraded":"2024-04-02T04:48:44.062321+0000","last_fullsized":"2024-04-02T04:48:44.062321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.098270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.098270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":164,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062066+0000","last_change":"2024-04-02T04:48:39.608521+0000","last_active":"2024-04-02T04:48:44.062066+0000","last_peered":"2024-04-02T04:48:44.062066+0000","last_clean":"2024-04-02T04:48:44.062066+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:48:44.062066+0000","last_undegraded":"2024-04-02T04:48:44.062066+0000","last_fullsized":"2024-04-02T04:48:44.062066+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:48:39.608255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.608255+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:41.887929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":167,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.422690+0000","last_change":"2024-04-02T04:47:34.393560+0000","last_active":"2024-04-02T04:48:44.422690+0000","last_peered":"2024-04-02T04:48:44.422690+0000","last_clean":"2024-04-02T04:48:44.422690+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:48:44.422690+0000","last_undegraded":"2024-04-02T04:48:44.422690+0000","last_fullsized":"2024-04-02T04:48:44.422690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:34.393321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:34.393321+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.086104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":202,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060383+0000","last_change":"2024-04-02T04:47:42.831672+0000","last_active":"2024-04-02T04:48:44.060383+0000","last_peered":"2024-04-02T04:48:44.060383+0000","last_clean":"2024-04-02T04:48:44.060383+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:48:44.060383+0000","last_undegraded":"2024-04-02T04:48:44.060383+0000","last_fullsized":"2024-04-02T04:48:44.060383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.831447+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:46.338365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":129,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.067360+0000","last_change":"2024-04-02T04:47:47.122324+0000","last_active":"2024-04-02T04:48:44.067360+0000","last_peered":"2024-04-02T04:48:44.067360+0000","last_clean":"2024-04-02T04:48:44.067360+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:48:44.067360+0000","last_undegraded":"2024-04-02T04:48:44.067360+0000","last_fullsized":"2024-04-02T04:48:44.067360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.122214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.122214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.799063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":148,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912799+0000","last_change":"2024-04-02T04:47:43.151655+0000","last_active":"2024-04-02T04:48:44.912799+0000","last_peered":"2024-04-02T04:48:44.912799+0000","last_clean":"2024-04-02T04:48:44.912799+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:48:44.912799+0000","last_undegraded":"2024-04-02T04:48:44.912799+0000","last_fullsized":"2024-04-02T04:48:44.912799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:43.151427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:43.151427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:55.872479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":216,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.521598+0000","last_change":"2024-04-02T04:47:39.221972+0000","last_active":"2024-04-02T04:48:44.521598+0000","last_peered":"2024-04-02T04:48:44.521598+0000","last_clean":"2024-04-02T04:48:44.521598+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:48:44.521598+0000","last_undegraded":"2024-04-02T04:48:44.521598+0000","last_fullsized":"2024-04-02T04:48:44.521598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:47:39.221741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.221741+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:05.865282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.913520+0000","last_change":"2024-04-02T04:47:45.109386+0000","last_active":"2024-04-02T04:48:44.913520+0000","last_peered":"2024-04-02T04:48:44.913520+0000","last_clean":"2024-04-02T04:48:44.913520+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:48:44.913520+0000","last_undegraded":"2024-04-02T04:48:44.913520+0000","last_fullsized":"2024-04-02T04:48:44.913520+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.109170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.109170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.076716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":123,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.063119+0000","last_change":"2024-04-02T04:47:02.153712+0000","last_active":"2024-04-02T04:48:44.063119+0000","last_peered":"2024-04-02T04:48:44.063119+0000","last_clean":"2024-04-02T04:48:44.063119+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:48:44.063119+0000","last_undegraded":"2024-04-02T04:48:44.063119+0000","last_fullsized":"2024-04-02T04:48:44.063119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:47:02.153481+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:47:02.153481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.911700+0000","last_change":"2024-04-02T04:48:44.911734+0000","last_active":"2024-04-02T04:48:44.911700+0000","last_peered":"2024-04-02T04:48:44.911700+0000","last_clean":"2024-04-02T04:48:44.911700+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:48:44.911700+0000","last_undegraded":"2024-04-02T04:48:44.911700+0000","last_fullsized":"2024-04-02T04:48:44.911700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.911605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.911605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:04.418347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":184,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061613+0000","last_change":"2024-04-02T04:47:50.758701+0000","last_active":"2024-04-02T04:48:44.061613+0000","last_peered":"2024-04-02T04:48:44.061613+0000","last_clean":"2024-04-02T04:48:44.061613+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:48:44.061613+0000","last_undegraded":"2024-04-02T04:48:44.061613+0000","last_fullsized":"2024-04-02T04:48:44.061613+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:47:50.758312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:50.758312+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.203712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912706+0000","last_change":"2024-04-02T04:47:47.106602+0000","last_active":"2024-04-02T04:48:44.912706+0000","last_peered":"2024-04-02T04:48:44.912706+0000","last_clean":"2024-04-02T04:48:44.912706+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:48:44.912706+0000","last_undegraded":"2024-04-02T04:48:44.912706+0000","last_fullsized":"2024-04-02T04:48:44.912706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.106497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.032879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.424663+0000","last_change":"2024-04-02T04:47:53.271021+0000","last_active":"2024-04-02T04:48:44.424663+0000","last_peered":"2024-04-02T04:48:44.424663+0000","last_clean":"2024-04-02T04:48:44.424663+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:48:44.424663+0000","last_undegraded":"2024-04-02T04:48:44.424663+0000","last_fullsized":"2024-04-02T04:48:44.424663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.902292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":184,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912226+0000","last_change":"2024-04-02T04:46:55.308858+0000","last_active":"2024-04-02T04:48:44.912226+0000","last_peered":"2024-04-02T04:48:44.912226+0000","last_clean":"2024-04-02T04:48:44.912226+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:48:44.912226+0000","last_undegraded":"2024-04-02T04:48:44.912226+0000","last_fullsized":"2024-04-02T04:48:44.912226+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:46:55.308528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.308528+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.642984+0000","last_change":"2024-04-02T04:47:14.881150+0000","last_active":"2024-04-02T04:48:44.642984+0000","last_peered":"2024-04-02T04:48:44.642984+0000","last_clean":"2024-04-02T04:48:44.642984+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:48:44.642984+0000","last_undegraded":"2024-04-02T04:48:44.642984+0000","last_fullsized":"2024-04-02T04:48:44.642984+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:14.881031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:47:14.881031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:47.126455+0000","last_change":"2024-04-02T04:48:47.126491+0000","last_active":"2024-04-02T04:48:47.126455+0000","last_peered":"2024-04-02T04:48:47.126455+0000","last_clean":"2024-04-02T04:48:47.126455+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:48:47.126455+0000","last_undegraded":"2024-04-02T04:48:47.126455+0000","last_fullsized":"2024-04-02T04:48:47.126455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:47.126364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.126364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:00.049150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.522719+0000","last_change":"2024-04-02T04:47:58.186088+0000","last_active":"2024-04-02T04:48:44.522719+0000","last_peered":"2024-04-02T04:48:44.522719+0000","last_clean":"2024-04-02T04:48:44.522719+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:48:44.522719+0000","last_undegraded":"2024-04-02T04:48:44.522719+0000","last_fullsized":"2024-04-02T04:48:44.522719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:58.185868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:58.185868+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.185868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:23.742931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.063452+0000","last_change":"2024-04-02T04:46:52.985248+0000","last_active":"2024-04-02T04:48:44.063452+0000","last_peered":"2024-04-02T04:48:44.063452+0000","last_clean":"2024-04-02T04:48:44.063452+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:48:44.063452+0000","last_undegraded":"2024-04-02T04:48:44.063452+0000","last_fullsized":"2024-04-02T04:48:44.063452+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:46:52.984994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:52.984994+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.522075+0000","last_change":"2024-04-02T04:48:39.903784+0000","last_active":"2024-04-02T04:48:44.522075+0000","last_peered":"2024-04-02T04:48:44.522075+0000","last_clean":"2024-04-02T04:48:44.522075+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:48:44.522075+0000","last_undegraded":"2024-04-02T04:48:44.522075+0000","last_fullsized":"2024-04-02T04:48:44.522075+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:39.903683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.903683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:53.451472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":397,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.423469+0000","last_change":"2024-04-02T04:47:44.214306+0000","last_active":"2024-04-02T04:48:44.423469+0000","last_peered":"2024-04-02T04:48:44.423469+0000","last_clean":"2024-04-02T04:48:44.423469+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:48:44.423469+0000","last_undegraded":"2024-04-02T04:48:44.423469+0000","last_fullsized":"2024-04-02T04:48:44.423469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:47:44.214047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.214047+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.404997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:45.569206+0000","last_change":"2024-04-02T04:48:45.569264+0000","last_active":"2024-04-02T04:48:45.569206+0000","last_peered":"2024-04-02T04:48:45.569206+0000","last_clean":"2024-04-02T04:48:45.569206+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:48:45.569206+0000","last_undegraded":"2024-04-02T04:48:45.569206+0000","last_fullsized":"2024-04-02T04:48:45.569206+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.569058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.569058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.165593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912606+0000","last_change":"2024-04-02T04:48:01.028154+0000","last_active":"2024-04-02T04:48:44.912606+0000","last_peered":"2024-04-02T04:48:44.912606+0000","last_clean":"2024-04-02T04:48:44.912606+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:48:44.912606+0000","last_undegraded":"2024-04-02T04:48:44.912606+0000","last_fullsized":"2024-04-02T04:48:44.912606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:01.027858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:01.027858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:06.895866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":170,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:47.093879+0000","last_change":"2024-04-02T04:48:47.093951+0000","last_active":"2024-04-02T04:48:47.093879+0000","last_peered":"2024-04-02T04:48:47.093879+0000","last_clean":"2024-04-02T04:48:47.093879+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:48:47.093879+0000","last_undegraded":"2024-04-02T04:48:47.093879+0000","last_fullsized":"2024-04-02T04:48:47.093879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:48:47.093759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.093759+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:56.401562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.423461+0000","last_change":"2024-04-02T04:47:05.388983+0000","last_active":"2024-04-02T04:48:44.423461+0000","last_peered":"2024-04-02T04:48:44.423461+0000","last_clean":"2024-04-02T04:48:44.423461+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:48:44.423461+0000","last_undegraded":"2024-04-02T04:48:44.423461+0000","last_fullsized":"2024-04-02T04:48:44.423461+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:05.388768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:05.388768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":131,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.067872+0000","last_change":"2024-04-02T04:46:50.400686+0000","last_active":"2024-04-02T04:48:44.067872+0000","last_peered":"2024-04-02T04:48:44.067872+0000","last_clean":"2024-04-02T04:48:44.067872+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:48:44.067872+0000","last_undegraded":"2024-04-02T04:48:44.067872+0000","last_fullsized":"2024-04-02T04:48:44.067872+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:46:50.400390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.911908+0000","last_change":"2024-04-02T04:48:04.001245+0000","last_active":"2024-04-02T04:48:44.911908+0000","last_peered":"2024-04-02T04:48:44.911908+0000","last_clean":"2024-04-02T04:48:44.911908+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:48:44.911908+0000","last_undegraded":"2024-04-02T04:48:44.911908+0000","last_fullsized":"2024-04-02T04:48:44.911908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:48:04.001090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:30.589675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":142,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.605468+0000","last_change":"2024-04-02T04:48:00.879991+0000","last_active":"2024-04-02T04:48:44.605468+0000","last_peered":"2024-04-02T04:48:44.605468+0000","last_clean":"2024-04-02T04:48:44.605468+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:48:44.605468+0000","last_undegraded":"2024-04-02T04:48:44.605468+0000","last_fullsized":"2024-04-02T04:48:44.605468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:48:00.879724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.879724+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.988600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.424975+0000","last_change":"2024-04-02T04:47:17.471458+0000","last_active":"2024-04-02T04:48:44.424975+0000","last_peered":"2024-04-02T04:48:44.424975+0000","last_clean":"2024-04-02T04:48:44.424975+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:48:44.424975+0000","last_undegraded":"2024-04-02T04:48:44.424975+0000","last_fullsized":"2024-04-02T04:48:44.424975+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:17.471105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:17.471105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":268,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912952+0000","last_change":"2024-04-02T04:47:42.173169+0000","last_active":"2024-04-02T04:48:44.912952+0000","last_peered":"2024-04-02T04:48:44.912952+0000","last_clean":"2024-04-02T04:48:44.912952+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:48:44.912952+0000","last_undegraded":"2024-04-02T04:48:44.912952+0000","last_fullsized":"2024-04-02T04:48:44.912952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.172894+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.893986+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"121'12","reported_seq":203,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.065828+0000","last_change":"2024-04-02T04:47:40.137383+0000","last_active":"2024-04-02T04:48:44.065828+0000","last_peered":"2024-04-02T04:48:44.065828+0000","last_clean":"2024-04-02T04:48:44.065828+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:48:44.065828+0000","last_undegraded":"2024-04-02T04:48:44.065828+0000","last_fullsized":"2024-04-02T04:48:44.065828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.137127+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:52.797250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":120,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060761+0000","last_change":"2024-04-02T04:47:00.939593+0000","last_active":"2024-04-02T04:48:44.060761+0000","last_peered":"2024-04-02T04:48:44.060761+0000","last_clean":"2024-04-02T04:48:44.060761+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:48:44.060761+0000","last_undegraded":"2024-04-02T04:48:44.060761+0000","last_fullsized":"2024-04-02T04:48:44.060761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:00.939352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:00.939352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.555949+0000","last_change":"2024-04-02T04:48:44.556005+0000","last_active":"2024-04-02T04:48:44.555949+0000","last_peered":"2024-04-02T04:48:44.555949+0000","last_clean":"2024-04-02T04:48:44.555949+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:48:44.555949+0000","last_undegraded":"2024-04-02T04:48:44.555949+0000","last_fullsized":"2024-04-02T04:48:44.555949+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.555854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.555854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:10.873849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":142,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912985+0000","last_change":"2024-04-02T04:47:58.111681+0000","last_active":"2024-04-02T04:48:44.912985+0000","last_peered":"2024-04-02T04:48:44.912985+0000","last_clean":"2024-04-02T04:48:44.912985+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:48:44.912985+0000","last_undegraded":"2024-04-02T04:48:44.912985+0000","last_fullsized":"2024-04-02T04:48:44.912985+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.111561+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:08.316506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":168,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062330+0000","last_change":"2024-04-02T04:47:04.162419+0000","last_active":"2024-04-02T04:48:44.062330+0000","last_peered":"2024-04-02T04:48:44.062330+0000","last_clean":"2024-04-02T04:48:44.062330+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:48:44.062330+0000","last_undegraded":"2024-04-02T04:48:44.062330+0000","last_fullsized":"2024-04-02T04:48:44.062330+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:47:04.162146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:04.162146+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":211,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061611+0000","last_change":"2024-04-02T04:48:38.477973+0000","last_active":"2024-04-02T04:48:44.061611+0000","last_peered":"2024-04-02T04:48:44.061611+0000","last_clean":"2024-04-02T04:48:44.061611+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:48:44.061611+0000","last_undegraded":"2024-04-02T04:48:44.061611+0000","last_fullsized":"2024-04-02T04:48:44.061611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:48:38.477825+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.477825+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:04.284920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.424159+0000","last_change":"2024-04-02T04:47:49.224735+0000","last_active":"2024-04-02T04:48:44.424159+0000","last_peered":"2024-04-02T04:48:44.424159+0000","last_clean":"2024-04-02T04:48:44.424159+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:48:44.424159+0000","last_undegraded":"2024-04-02T04:48:44.424159+0000","last_fullsized":"2024-04-02T04:48:44.424159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.916792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":405,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:45.844550+0000","last_change":"2024-04-02T04:46:43.873401+0000","last_active":"2024-04-02T04:48:45.844550+0000","last_peered":"2024-04-02T04:48:45.844550+0000","last_clean":"2024-04-02T04:48:45.844550+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:48:45.844550+0000","last_undegraded":"2024-04-02T04:48:45.844550+0000","last_fullsized":"2024-04-02T04:48:45.844550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:46:43.873013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.873013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.606201+0000","last_change":"2024-04-02T04:47:41.043637+0000","last_active":"2024-04-02T04:48:44.606201+0000","last_peered":"2024-04-02T04:48:44.606201+0000","last_clean":"2024-04-02T04:48:44.606201+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:48:44.606201+0000","last_undegraded":"2024-04-02T04:48:44.606201+0000","last_fullsized":"2024-04-02T04:48:44.606201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:41.043411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.043411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.404918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":155,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.068130+0000","last_change":"2024-04-02T04:48:43.043307+0000","last_active":"2024-04-02T04:48:44.068130+0000","last_peered":"2024-04-02T04:48:44.068130+0000","last_clean":"2024-04-02T04:48:44.068130+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:48:44.068130+0000","last_undegraded":"2024-04-02T04:48:44.068130+0000","last_fullsized":"2024-04-02T04:48:44.068130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:48:43.043155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.043155+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:09.690835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":124,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060713+0000","last_change":"2024-04-02T04:47:25.814952+0000","last_active":"2024-04-02T04:48:44.060713+0000","last_peered":"2024-04-02T04:48:44.060713+0000","last_clean":"2024-04-02T04:48:44.060713+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:48:44.060713+0000","last_undegraded":"2024-04-02T04:48:44.060713+0000","last_fullsized":"2024-04-02T04:48:44.060713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:25.814722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:25.814722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":129,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:45.485538+0000","last_change":"2024-04-02T04:48:45.485574+0000","last_active":"2024-04-02T04:48:45.485538+0000","last_peered":"2024-04-02T04:48:45.485538+0000","last_clean":"2024-04-02T04:48:45.485538+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:48:45.485538+0000","last_undegraded":"2024-04-02T04:48:45.485538+0000","last_fullsized":"2024-04-02T04:48:45.485538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.485441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.485441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:01.318866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":193,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061902+0000","last_change":"2024-04-02T04:46:43.734784+0000","last_active":"2024-04-02T04:48:44.061902+0000","last_peered":"2024-04-02T04:48:44.061902+0000","last_clean":"2024-04-02T04:48:44.061902+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:48:44.061902+0000","last_undegraded":"2024-04-02T04:48:44.061902+0000","last_fullsized":"2024-04-02T04:48:44.061902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:46:43.734423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:46:43.734423+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":140,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.423749+0000","last_change":"2024-04-02T04:47:28.453678+0000","last_active":"2024-04-02T04:48:44.423749+0000","last_peered":"2024-04-02T04:48:44.423749+0000","last_clean":"2024-04-02T04:48:44.423749+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:48:44.423749+0000","last_undegraded":"2024-04-02T04:48:44.423749+0000","last_fullsized":"2024-04-02T04:48:44.423749+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:28.453442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:28.453442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.301035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":175,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.060366+0000","last_change":"2024-04-02T04:46:26.820637+0000","last_active":"2024-04-02T04:48:44.060366+0000","last_peered":"2024-04-02T04:48:44.060366+0000","last_clean":"2024-04-02T04:48:44.060366+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:48:44.060366+0000","last_undegraded":"2024-04-02T04:48:44.060366+0000","last_fullsized":"2024-04-02T04:48:44.060366+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:46:26.820380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:46:26.820380+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.522560+0000","last_change":"2024-04-02T04:47:53.173015+0000","last_active":"2024-04-02T04:48:44.522560+0000","last_peered":"2024-04-02T04:48:44.522560+0000","last_clean":"2024-04-02T04:48:44.522560+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:48:44.522560+0000","last_undegraded":"2024-04-02T04:48:44.522560+0000","last_fullsized":"2024-04-02T04:48:44.522560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.172792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.172792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:56.046175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":72,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.643004+0000","last_change":"2024-04-02T04:47:55.855659+0000","last_active":"2024-04-02T04:48:44.643004+0000","last_peered":"2024-04-02T04:48:44.643004+0000","last_clean":"2024-04-02T04:48:44.643004+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:48:44.643004+0000","last_undegraded":"2024-04-02T04:48:44.643004+0000","last_fullsized":"2024-04-02T04:48:44.643004+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.855510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.855510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:18.631275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":146,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912118+0000","last_change":"2024-04-02T04:47:44.112479+0000","last_active":"2024-04-02T04:48:44.912118+0000","last_peered":"2024-04-02T04:48:44.912118+0000","last_clean":"2024-04-02T04:48:44.912118+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:48:44.912118+0000","last_undegraded":"2024-04-02T04:48:44.912118+0000","last_fullsized":"2024-04-02T04:48:44.912118+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:44.112270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.112270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:58.773698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":195,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061476+0000","last_change":"2024-04-02T04:47:26.453777+0000","last_active":"2024-04-02T04:48:44.061476+0000","last_peered":"2024-04-02T04:48:44.061476+0000","last_clean":"2024-04-02T04:48:44.061476+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:48:44.061476+0000","last_undegraded":"2024-04-02T04:48:44.061476+0000","last_fullsized":"2024-04-02T04:48:44.061476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:47:26.453340+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:47:26.453340+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061940+0000","last_change":"2024-04-02T04:48:40.604446+0000","last_active":"2024-04-02T04:48:44.061940+0000","last_peered":"2024-04-02T04:48:44.061940+0000","last_clean":"2024-04-02T04:48:44.061940+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:48:44.061940+0000","last_undegraded":"2024-04-02T04:48:44.061940+0000","last_fullsized":"2024-04-02T04:48:44.061940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.604291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.604291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.578827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":140,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.913428+0000","last_change":"2024-04-02T04:47:15.265577+0000","last_active":"2024-04-02T04:48:44.913428+0000","last_peered":"2024-04-02T04:48:44.913428+0000","last_clean":"2024-04-02T04:48:44.913428+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:48:44.913428+0000","last_undegraded":"2024-04-02T04:48:44.913428+0000","last_fullsized":"2024-04-02T04:48:44.913428+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:15.265490+0000","last_clean_scrub_stamp":"2024-04-02T04:47:15.265490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":170,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062223+0000","last_change":"2024-04-02T04:47:41.576889+0000","last_active":"2024-04-02T04:48:44.062223+0000","last_peered":"2024-04-02T04:48:44.062223+0000","last_clean":"2024-04-02T04:48:44.062223+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:48:44.062223+0000","last_undegraded":"2024-04-02T04:48:44.062223+0000","last_fullsized":"2024-04-02T04:48:44.062223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:47:41.576640+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.576640+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:57.199333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":124,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.061318+0000","last_change":"2024-04-02T04:46:55.945141+0000","last_active":"2024-04-02T04:48:44.061318+0000","last_peered":"2024-04-02T04:48:44.061318+0000","last_clean":"2024-04-02T04:48:44.061318+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:48:44.061318+0000","last_undegraded":"2024-04-02T04:48:44.061318+0000","last_fullsized":"2024-04-02T04:48:44.061318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:55.944866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:46:55.944866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.642837+0000","last_change":"2024-04-02T04:47:37.864283+0000","last_active":"2024-04-02T04:48:44.642837+0000","last_peered":"2024-04-02T04:48:44.642837+0000","last_clean":"2024-04-02T04:48:44.642837+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:48:44.642837+0000","last_undegraded":"2024-04-02T04:48:44.642837+0000","last_fullsized":"2024-04-02T04:48:44.642837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.864039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.864039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:47.035813+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":202,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.062567+0000","last_change":"2024-04-02T04:47:56.704599+0000","last_active":"2024-04-02T04:48:44.062567+0000","last_peered":"2024-04-02T04:48:44.062567+0000","last_clean":"2024-04-02T04:48:44.062567+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:48:44.062567+0000","last_undegraded":"2024-04-02T04:48:44.062567+0000","last_fullsized":"2024-04-02T04:48:44.062567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:47:56.704379+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.874702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":140,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:45.439619+0000","last_change":"2024-04-02T04:48:45.439677+0000","last_active":"2024-04-02T04:48:45.439619+0000","last_peered":"2024-04-02T04:48:45.439619+0000","last_clean":"2024-04-02T04:48:45.439619+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:48:45.439619+0000","last_undegraded":"2024-04-02T04:48:45.439619+0000","last_fullsized":"2024-04-02T04:48:45.439619+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.439457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.439457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.918301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":181,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.604520+0000","last_change":"2024-04-02T04:47:57.887340+0000","last_active":"2024-04-02T04:48:44.604520+0000","last_peered":"2024-04-02T04:48:44.604520+0000","last_clean":"2024-04-02T04:48:44.604520+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:48:44.604520+0000","last_undegraded":"2024-04-02T04:48:44.604520+0000","last_fullsized":"2024-04-02T04:48:44.604520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:47:57.887065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.887065+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:04.529466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.423181+0000","last_change":"2024-04-02T04:47:31.439024+0000","last_active":"2024-04-02T04:48:44.423181+0000","last_peered":"2024-04-02T04:48:44.423181+0000","last_clean":"2024-04-02T04:48:44.423181+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:48:44.423181+0000","last_undegraded":"2024-04-02T04:48:44.423181+0000","last_fullsized":"2024-04-02T04:48:44.423181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:31.438817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:31.438817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:48:45.747704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":131,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.643638+0000","last_change":"2024-04-02T04:46:41.673523+0000","last_active":"2024-04-02T04:48:44.643638+0000","last_peered":"2024-04-02T04:48:44.643638+0000","last_clean":"2024-04-02T04:48:44.643638+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:48:44.643638+0000","last_undegraded":"2024-04-02T04:48:44.643638+0000","last_fullsized":"2024-04-02T04:48:44.643638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:46:41.673448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:46:41.673448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.425153+0000","last_change":"2024-04-02T04:47:23.464318+0000","last_active":"2024-04-02T04:48:44.425153+0000","last_peered":"2024-04-02T04:48:44.425153+0000","last_clean":"2024-04-02T04:48:44.425153+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:48:44.425153+0000","last_undegraded":"2024-04-02T04:48:44.425153+0000","last_fullsized":"2024-04-02T04:48:44.425153+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:47:23.464017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.464017+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":133,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.605457+0000","last_change":"2024-04-02T04:47:55.940230+0000","last_active":"2024-04-02T04:48:44.605457+0000","last_peered":"2024-04-02T04:48:44.605457+0000","last_clean":"2024-04-02T04:48:44.605457+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:48:44.605457+0000","last_undegraded":"2024-04-02T04:48:44.605457+0000","last_fullsized":"2024-04-02T04:48:44.605457+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.939983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.939983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.949757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":121,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.064789+0000","last_change":"2024-04-02T04:47:36.243027+0000","last_active":"2024-04-02T04:48:44.064789+0000","last_peered":"2024-04-02T04:48:44.064789+0000","last_clean":"2024-04-02T04:48:44.064789+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:48:44.064789+0000","last_undegraded":"2024-04-02T04:48:44.064789+0000","last_fullsized":"2024-04-02T04:48:44.064789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:02.633289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":140,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.912814+0000","last_change":"2024-04-02T04:47:23.176576+0000","last_active":"2024-04-02T04:48:44.912814+0000","last_peered":"2024-04-02T04:48:44.912814+0000","last_clean":"2024-04-02T04:48:44.912814+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:48:44.912814+0000","last_undegraded":"2024-04-02T04:48:44.912814+0000","last_fullsized":"2024-04-02T04:48:44.912814+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:23.176366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:47:23.176366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":155,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.521561+0000","last_change":"2024-04-02T04:47:44.215606+0000","last_active":"2024-04-02T04:48:44.521561+0000","last_peered":"2024-04-02T04:48:44.521561+0000","last_clean":"2024-04-02T04:48:44.521561+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:48:44.521561+0000","last_undegraded":"2024-04-02T04:48:44.521561+0000","last_fullsized":"2024-04-02T04:48:44.521561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:47:44.215294+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.215294+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.905308+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":133,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.065934+0000","last_change":"2024-04-02T04:47:35.260903+0000","last_active":"2024-04-02T04:48:44.065934+0000","last_peered":"2024-04-02T04:48:44.065934+0000","last_clean":"2024-04-02T04:48:44.065934+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:48:44.065934+0000","last_undegraded":"2024-04-02T04:48:44.065934+0000","last_fullsized":"2024-04-02T04:48:44.065934+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.260653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.260653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:01.706903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:45.911603+0000","last_change":"2024-04-02T04:48:45.911666+0000","last_active":"2024-04-02T04:48:45.911603+0000","last_peered":"2024-04-02T04:48:45.911603+0000","last_clean":"2024-04-02T04:48:45.911603+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:48:45.911603+0000","last_undegraded":"2024-04-02T04:48:45.911603+0000","last_fullsized":"2024-04-02T04:48:45.911603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.911425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.911425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:02.624974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":139,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.644590+0000","last_change":"2024-04-02T04:48:42.658846+0000","last_active":"2024-04-02T04:48:44.644590+0000","last_peered":"2024-04-02T04:48:44.644590+0000","last_clean":"2024-04-02T04:48:44.644590+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:48:44.644590+0000","last_undegraded":"2024-04-02T04:48:44.644590+0000","last_fullsized":"2024-04-02T04:48:44.644590+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.658688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.658688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:45.858271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":161,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:47.429221+0000","last_change":"2024-04-02T04:48:47.429314+0000","last_active":"2024-04-02T04:48:47.429221+0000","last_peered":"2024-04-02T04:48:47.429221+0000","last_clean":"2024-04-02T04:48:47.429221+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:48:47.429221+0000","last_undegraded":"2024-04-02T04:48:47.429221+0000","last_fullsized":"2024-04-02T04:48:47.429221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.133536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":131,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-02T04:48:44.521721+0000","last_change":"2024-04-02T04:47:48.245279+0000","last_active":"2024-04-02T04:48:44.521721+0000","last_peered":"2024-04-02T04:48:44.521721+0000","last_clean":"2024-04-02T04:48:44.521721+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:48:44.521721+0000","last_undegraded":"2024-04-02T04:48:44.521721+0000","last_fullsized":"2024-04-02T04:48:44.521721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:48.245180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:48.245180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:03.877121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":629,"ondisk_log_size":629,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743493,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1388544,"data_stored":1020794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103248,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28340,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716812,"statfs":{"total":95995035648,"available":95966015488,"internally_reserved":0,"allocated":1355776,"data_stored":1018607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574979,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":778240,"data_stored":429319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712632,"statfs":{"total":95995035648,"available":95961735168,"internally_reserved":0,"allocated":1372160,"data_stored":1019271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378745,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":765952,"data_stored":429555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051695,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":761856,"data_stored":428620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822757,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":802816,"data_stored":429548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463001,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":765952,"data_stored":429464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:48:54.713 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-04-02T04:48:54.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:54.759 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:54.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:54.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:54.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:55.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:55.149 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:55.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:55.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:55 smithi098 bash[23790]: cluster 2024-04-02T04:48:53.828959+0000 mgr.y (mgr.14538) 188 : cluster 0 pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:55.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:55 smithi098 bash[23790]: cluster 2024-04-02T04:48:54.068450+0000 osd.1 (osd.1) 111 : cluster 0 4.12 scrub starts 2024-04-02T04:48:55.263 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:55 smithi098 bash[23790]: cluster 2024-04-02T04:48:54.070451+0000 osd.1 (osd.1) 112 : cluster 0 4.12 scrub ok 2024-04-02T04:48:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:55.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:55.406 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:55.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:55.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:55 smithi069 bash[21528]: cluster 2024-04-02T04:48:53.828959+0000 mgr.y (mgr.14538) 188 : cluster 0 pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:55.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:55 smithi069 bash[21528]: cluster 2024-04-02T04:48:54.068450+0000 osd.1 (osd.1) 111 : cluster 0 4.12 scrub starts 2024-04-02T04:48:55.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:55 smithi069 bash[21528]: cluster 2024-04-02T04:48:54.070451+0000 osd.1 (osd.1) 112 : cluster 0 4.12 scrub ok 2024-04-02T04:48:55.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:55 smithi069 bash[28904]: cluster 2024-04-02T04:48:53.828959+0000 mgr.y (mgr.14538) 188 : cluster 0 pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:48:55.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:55 smithi069 bash[28904]: cluster 2024-04-02T04:48:54.068450+0000 osd.1 (osd.1) 111 : cluster 0 4.12 scrub starts 2024-04-02T04:48:55.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:55 smithi069 bash[28904]: cluster 2024-04-02T04:48:54.070451+0000 osd.1 (osd.1) 112 : cluster 0 4.12 scrub ok 2024-04-02T04:48:55.536 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:55.639 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:55.658 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:55.724 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:48:55.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:48:55.724 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:48:55.724 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:48:55.725 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:48:55.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:55.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:55.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:55.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:56.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:56.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:56.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:56.182 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:56.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:56.313 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:56.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:48:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:48:56.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:56.572 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:56.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:56.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-02T04:48:56.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:56.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:56.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:56.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:57.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:57.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:57.212 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:57.220 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:57.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:57.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[28904]: cluster 2024-04-02T04:48:55.734674+0000 osd.2 (osd.2) 65 : cluster 0 2.1 scrub starts 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[28904]: cluster 2024-04-02T04:48:55.735627+0000 osd.2 (osd.2) 66 : cluster 0 2.1 scrub ok 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[28904]: cluster 2024-04-02T04:48:55.829297+0000 mgr.y (mgr.14538) 189 : cluster 0 pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[28904]: cluster 2024-04-02T04:48:56.337726+0000 osd.7 (osd.7) 119 : cluster 0 3.11 scrub starts 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[28904]: cluster 2024-04-02T04:48:56.339795+0000 osd.7 (osd.7) 120 : cluster 0 3.11 scrub ok 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[21528]: cluster 2024-04-02T04:48:55.734674+0000 osd.2 (osd.2) 65 : cluster 0 2.1 scrub starts 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[21528]: cluster 2024-04-02T04:48:55.735627+0000 osd.2 (osd.2) 66 : cluster 0 2.1 scrub ok 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[21528]: cluster 2024-04-02T04:48:55.829297+0000 mgr.y (mgr.14538) 189 : cluster 0 pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[21528]: cluster 2024-04-02T04:48:56.337726+0000 osd.7 (osd.7) 119 : cluster 0 3.11 scrub starts 2024-04-02T04:48:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:57 smithi069 bash[21528]: cluster 2024-04-02T04:48:56.339795+0000 osd.7 (osd.7) 120 : cluster 0 3.11 scrub ok 2024-04-02T04:48:57.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:57.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:57.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:57 smithi098 bash[23790]: cluster 2024-04-02T04:48:55.734674+0000 osd.2 (osd.2) 65 : cluster 0 2.1 scrub starts 2024-04-02T04:48:57.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:57 smithi098 bash[23790]: cluster 2024-04-02T04:48:55.735627+0000 osd.2 (osd.2) 66 : cluster 0 2.1 scrub ok 2024-04-02T04:48:57.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:57 smithi098 bash[23790]: cluster 2024-04-02T04:48:55.829297+0000 mgr.y (mgr.14538) 189 : cluster 0 pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:48:57.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:57 smithi098 bash[23790]: cluster 2024-04-02T04:48:56.337726+0000 osd.7 (osd.7) 119 : cluster 0 3.11 scrub starts 2024-04-02T04:48:57.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:57 smithi098 bash[23790]: cluster 2024-04-02T04:48:56.339795+0000 osd.7 (osd.7) 120 : cluster 0 3.11 scrub ok 2024-04-02T04:48:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:48:57.569 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:57.591 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:57.612 INFO:tasks.workunit.client.0.smithi069.stderr:Updating files: 91% (11357/12435) Updating files: 92% (11441/12435) Updating files: 93% (11565/12435) Updating files: 94% (11689/12435) Updating files: 95% (11814/12435) Updating files: 96% (11938/12435) Updating files: 97% (12062/12435) Updating files: 98% (12187/12435) Updating files: 99% (12311/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-04-02T04:48:57.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:57.721 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:57.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:57.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:48:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:48:57.950 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:57.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:57.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:58.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:58.103 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:58.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:58.230 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:58.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:58.377 INFO:tasks.workunit.client.0.smithi069.stderr:Note: switching to '6cec793d9073f0594e2364b9b687ecaf22424cca'. 2024-04-02T04:48:58.377 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.377 INFO:tasks.workunit.client.0.smithi069.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr:state without impacting any branches by switching back to a branch. 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: git switch -c 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr:Or undo this operation with: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: git switch - 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-02T04:48:58.378 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-04-02T04:48:58.379 INFO:tasks.workunit.client.0.smithi069.stderr:HEAD is now at 6cec793d907 Merge pull request #56601 from zdover23/wip-doc-2024-04-01-backport-56600-to-squid 2024-04-02T04:48:58.382 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-02T04:48:58.389 INFO:tasks.workunit.client.0.smithi069.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-02T04:48:58.392 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-02T04:48:58.392 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-02T04:48:58.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:58.488 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:48:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:48:58.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:58.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:58.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:58.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:48:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:48:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:48:58.749 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:58.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:58.880 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:58.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:58.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:59.009 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:59.112 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:59.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:59.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:48:59.273 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:48:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:48:59.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:59.387 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:48:59 smithi069 bash[21528]: cluster 2024-04-02T04:48:57.830669+0000 mgr.y (mgr.14538) 190 : cluster 0 pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:59.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:48:59 smithi069 bash[28904]: cluster 2024-04-02T04:48:57.830669+0000 mgr.y (mgr.14538) 190 : cluster 0 pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:59.403 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:48:59.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:48:59.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:59.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:48:59 smithi098 bash[23790]: cluster 2024-04-02T04:48:57.830669+0000 mgr.y (mgr.14538) 190 : cluster 0 pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:48:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:59.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:59.664 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:48:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:48:59.714 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:48:59.714 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:48:59.715 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-02T04:48:59.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:48:59.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:59.791 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:48:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:48:59.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:59.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:48:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:48:59.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:00.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:00.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:00.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:00.305 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:00.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:00.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:00.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:00 smithi098 bash[23790]: cluster 2024-04-02T04:48:58.872323+0000 osd.1 (osd.1) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:00 smithi098 bash[23790]: cluster 2024-04-02T04:48:58.872440+0000 osd.1 (osd.1) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:00 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.266554+0000 mon.a (mon.0) 954 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-04-02T04:49:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:00 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.302543+0000 osd.7 (osd.7) 121 : cluster 0 6.15 scrub starts 2024-04-02T04:49:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:00 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.304147+0000 osd.7 (osd.7) 122 : cluster 0 6.15 scrub ok 2024-04-02T04:49:00.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[21528]: cluster 2024-04-02T04:48:58.872323+0000 osd.1 (osd.1) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:00.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[21528]: cluster 2024-04-02T04:48:58.872440+0000 osd.1 (osd.1) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:00.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.266554+0000 mon.a (mon.0) 954 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-04-02T04:49:00.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.302543+0000 osd.7 (osd.7) 121 : cluster 0 6.15 scrub starts 2024-04-02T04:49:00.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.304147+0000 osd.7 (osd.7) 122 : cluster 0 6.15 scrub ok 2024-04-02T04:49:00.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[28904]: cluster 2024-04-02T04:48:58.872323+0000 osd.1 (osd.1) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:00.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[28904]: cluster 2024-04-02T04:48:58.872440+0000 osd.1 (osd.1) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:00.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.266554+0000 mon.a (mon.0) 954 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-04-02T04:49:00.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.302543+0000 osd.7 (osd.7) 121 : cluster 0 6.15 scrub starts 2024-04-02T04:49:00.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:00 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.304147+0000 osd.7 (osd.7) 122 : cluster 0 6.15 scrub ok 2024-04-02T04:49:00.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:00.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:00.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:00.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:00.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:00.945 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:00.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:01.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:01.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:01.209 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:01.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:01.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.822130+0000 osd.2 (osd.2) 67 : cluster 0 6.f scrub starts 2024-04-02T04:49:01.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.823598+0000 osd.2 (osd.2) 68 : cluster 0 6.f scrub ok 2024-04-02T04:49:01.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.831070+0000 mgr.y (mgr.14538) 191 : cluster 0 pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:49:01.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.851636+0000 osd.5 (osd.5) 115 : cluster 0 6.19 scrub starts 2024-04-02T04:49:01.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:48:59.855574+0000 osd.5 (osd.5) 116 : cluster 0 6.19 scrub ok 2024-04-02T04:49:01.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:49:00.101028+0000 osd.0 (osd.0) 119 : cluster 0 4.16 deep-scrub starts 2024-04-02T04:49:01.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:49:00.103136+0000 osd.0 (osd.0) 120 : cluster 0 4.16 deep-scrub ok 2024-04-02T04:49:01.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:49:00.351229+0000 osd.7 (osd.7) 123 : cluster 0 3.d scrub starts 2024-04-02T04:49:01.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:01 smithi098 bash[23790]: cluster 2024-04-02T04:49:00.356097+0000 osd.7 (osd.7) 124 : cluster 0 3.d scrub ok 2024-04-02T04:49:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:01.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:01.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.822130+0000 osd.2 (osd.2) 67 : cluster 0 6.f scrub starts 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.823598+0000 osd.2 (osd.2) 68 : cluster 0 6.f scrub ok 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.831070+0000 mgr.y (mgr.14538) 191 : cluster 0 pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.851636+0000 osd.5 (osd.5) 115 : cluster 0 6.19 scrub starts 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:48:59.855574+0000 osd.5 (osd.5) 116 : cluster 0 6.19 scrub ok 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:49:00.101028+0000 osd.0 (osd.0) 119 : cluster 0 4.16 deep-scrub starts 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:49:00.103136+0000 osd.0 (osd.0) 120 : cluster 0 4.16 deep-scrub ok 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:49:00.351229+0000 osd.7 (osd.7) 123 : cluster 0 3.d scrub starts 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[21528]: cluster 2024-04-02T04:49:00.356097+0000 osd.7 (osd.7) 124 : cluster 0 3.d scrub ok 2024-04-02T04:49:01.458 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.822130+0000 osd.2 (osd.2) 67 : cluster 0 6.f scrub starts 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.823598+0000 osd.2 (osd.2) 68 : cluster 0 6.f scrub ok 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.831070+0000 mgr.y (mgr.14538) 191 : cluster 0 pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.851636+0000 osd.5 (osd.5) 115 : cluster 0 6.19 scrub starts 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:48:59.855574+0000 osd.5 (osd.5) 116 : cluster 0 6.19 scrub ok 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:49:00.101028+0000 osd.0 (osd.0) 119 : cluster 0 4.16 deep-scrub starts 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:49:00.103136+0000 osd.0 (osd.0) 120 : cluster 0 4.16 deep-scrub ok 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:49:00.351229+0000 osd.7 (osd.7) 123 : cluster 0 3.d scrub starts 2024-04-02T04:49:01.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:01 smithi069 bash[28904]: cluster 2024-04-02T04:49:00.356097+0000 osd.7 (osd.7) 124 : cluster 0 3.d scrub ok 2024-04-02T04:49:01.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:01.572 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:01.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:01.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:01.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:01.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:01.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:01.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:02.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:02.124 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:02.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:02.254 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:02.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:02.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:00.818380+0000 osd.5 (osd.5) 117 : cluster 0 6.d scrub starts 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:00.819871+0000 osd.5 (osd.5) 118 : cluster 0 6.d scrub ok 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.070659+0000 osd.0 (osd.0) 121 : cluster 0 3.1f scrub starts 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.074715+0000 osd.0 (osd.0) 122 : cluster 0 3.1f scrub ok 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.486928+0000 osd.3 (osd.3) 147 : cluster 0 6.b scrub starts 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.488392+0000 osd.3 (osd.3) 148 : cluster 0 6.b scrub ok 2024-04-02T04:49:02.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.944473+0000 osd.1 (osd.1) 115 : cluster 0 3.0 scrub starts 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.949352+0000 osd.1 (osd.1) 116 : cluster 0 3.0 scrub ok 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:00.818380+0000 osd.5 (osd.5) 117 : cluster 0 6.d scrub starts 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:00.819871+0000 osd.5 (osd.5) 118 : cluster 0 6.d scrub ok 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.070659+0000 osd.0 (osd.0) 121 : cluster 0 3.1f scrub starts 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.074715+0000 osd.0 (osd.0) 122 : cluster 0 3.1f scrub ok 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.486928+0000 osd.3 (osd.3) 147 : cluster 0 6.b scrub starts 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.488392+0000 osd.3 (osd.3) 148 : cluster 0 6.b scrub ok 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.944473+0000 osd.1 (osd.1) 115 : cluster 0 3.0 scrub starts 2024-04-02T04:49:02.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.949352+0000 osd.1 (osd.1) 116 : cluster 0 3.0 scrub ok 2024-04-02T04:49:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:02.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:02.511 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:02.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:02.640 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:00.818380+0000 osd.5 (osd.5) 117 : cluster 0 6.d scrub starts 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:00.819871+0000 osd.5 (osd.5) 118 : cluster 0 6.d scrub ok 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.070659+0000 osd.0 (osd.0) 121 : cluster 0 3.1f scrub starts 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.074715+0000 osd.0 (osd.0) 122 : cluster 0 3.1f scrub ok 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.486928+0000 osd.3 (osd.3) 147 : cluster 0 6.b scrub starts 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.488392+0000 osd.3 (osd.3) 148 : cluster 0 6.b scrub ok 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.944473+0000 osd.1 (osd.1) 115 : cluster 0 3.0 scrub starts 2024-04-02T04:49:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.949352+0000 osd.1 (osd.1) 116 : cluster 0 3.0 scrub ok 2024-04-02T04:49:02.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:02.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:02.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:03.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:03.015 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:49:02 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:49:03.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:03.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:03.162 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:03.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:03.294 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:03.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:03.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.624808+0000 osd.4 (osd.4) 123 : cluster 0 4.1a scrub starts 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.626737+0000 osd.4 (osd.4) 124 : cluster 0 4.1a scrub ok 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.796100+0000 osd.5 (osd.5) 119 : cluster 0 2.16 scrub starts 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.797575+0000 osd.5 (osd.5) 120 : cluster 0 2.16 scrub ok 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:01.831613+0000 mgr.y (mgr.14538) 192 : cluster 0 pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.436522+0000 osd.7 (osd.7) 125 : cluster 0 6.8 scrub starts 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.437944+0000 osd.7 (osd.7) 126 : cluster 0 6.8 scrub ok 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.483744+0000 osd.3 (osd.3) 149 : cluster 0 4.17 scrub starts 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.485159+0000 osd.3 (osd.3) 150 : cluster 0 4.17 scrub ok 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.783412+0000 osd.5 (osd.5) 121 : cluster 0 3.c scrub starts 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.785294+0000 osd.5 (osd.5) 122 : cluster 0 3.c scrub ok 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.966638+0000 osd.1 (osd.1) 117 : cluster 0 3.19 deep-scrub starts 2024-04-02T04:49:03.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[21528]: cluster 2024-04-02T04:49:02.986688+0000 osd.1 (osd.1) 118 : cluster 0 3.19 deep-scrub ok 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.624808+0000 osd.4 (osd.4) 123 : cluster 0 4.1a scrub starts 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.626737+0000 osd.4 (osd.4) 124 : cluster 0 4.1a scrub ok 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.796100+0000 osd.5 (osd.5) 119 : cluster 0 2.16 scrub starts 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.797575+0000 osd.5 (osd.5) 120 : cluster 0 2.16 scrub ok 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:01.831613+0000 mgr.y (mgr.14538) 192 : cluster 0 pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.436522+0000 osd.7 (osd.7) 125 : cluster 0 6.8 scrub starts 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.437944+0000 osd.7 (osd.7) 126 : cluster 0 6.8 scrub ok 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.483744+0000 osd.3 (osd.3) 149 : cluster 0 4.17 scrub starts 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.485159+0000 osd.3 (osd.3) 150 : cluster 0 4.17 scrub ok 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.783412+0000 osd.5 (osd.5) 121 : cluster 0 3.c scrub starts 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.785294+0000 osd.5 (osd.5) 122 : cluster 0 3.c scrub ok 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.966638+0000 osd.1 (osd.1) 117 : cluster 0 3.19 deep-scrub starts 2024-04-02T04:49:03.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:03 smithi069 bash[28904]: cluster 2024-04-02T04:49:02.986688+0000 osd.1 (osd.1) 118 : cluster 0 3.19 deep-scrub ok 2024-04-02T04:49:03.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:03.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.624808+0000 osd.4 (osd.4) 123 : cluster 0 4.1a scrub starts 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.626737+0000 osd.4 (osd.4) 124 : cluster 0 4.1a scrub ok 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.796100+0000 osd.5 (osd.5) 119 : cluster 0 2.16 scrub starts 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.797575+0000 osd.5 (osd.5) 120 : cluster 0 2.16 scrub ok 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:01.831613+0000 mgr.y (mgr.14538) 192 : cluster 0 pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.436522+0000 osd.7 (osd.7) 125 : cluster 0 6.8 scrub starts 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.437944+0000 osd.7 (osd.7) 126 : cluster 0 6.8 scrub ok 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.483744+0000 osd.3 (osd.3) 149 : cluster 0 4.17 scrub starts 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.485159+0000 osd.3 (osd.3) 150 : cluster 0 4.17 scrub ok 2024-04-02T04:49:03.533 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.783412+0000 osd.5 (osd.5) 121 : cluster 0 3.c scrub starts 2024-04-02T04:49:03.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.785294+0000 osd.5 (osd.5) 122 : cluster 0 3.c scrub ok 2024-04-02T04:49:03.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.966638+0000 osd.1 (osd.1) 117 : cluster 0 3.19 deep-scrub starts 2024-04-02T04:49:03.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:03 smithi098 bash[23790]: cluster 2024-04-02T04:49:02.986688+0000 osd.1 (osd.1) 118 : cluster 0 3.19 deep-scrub ok 2024-04-02T04:49:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:03.652 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:03.677 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:03.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:03.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:03.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:04.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.062 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:04.166 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:04.193 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:04.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.320 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:04.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:04.450 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:04.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:04.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:04 smithi098 bash[23790]: audit 2024-04-02T04:49:02.963493+0000 mgr.y (mgr.14538) 193 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:04.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:04 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.784351+0000 osd.5 (osd.5) 123 : cluster 0 3.16 scrub starts 2024-04-02T04:49:04.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:04 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.788826+0000 osd.5 (osd.5) 124 : cluster 0 3.16 scrub ok 2024-04-02T04:49:04.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:04 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.972988+0000 osd.1 (osd.1) 119 : cluster 0 2.15 scrub starts 2024-04-02T04:49:04.565 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:04 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.975388+0000 osd.1 (osd.1) 120 : cluster 0 2.15 scrub ok 2024-04-02T04:49:04.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:04.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[21528]: audit 2024-04-02T04:49:02.963493+0000 mgr.y (mgr.14538) 193 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:04.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.784351+0000 osd.5 (osd.5) 123 : cluster 0 3.16 scrub starts 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.788826+0000 osd.5 (osd.5) 124 : cluster 0 3.16 scrub ok 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.972988+0000 osd.1 (osd.1) 119 : cluster 0 2.15 scrub starts 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.975388+0000 osd.1 (osd.1) 120 : cluster 0 2.15 scrub ok 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[28904]: audit 2024-04-02T04:49:02.963493+0000 mgr.y (mgr.14538) 193 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.784351+0000 osd.5 (osd.5) 123 : cluster 0 3.16 scrub starts 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.788826+0000 osd.5 (osd.5) 124 : cluster 0 3.16 scrub ok 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.972988+0000 osd.1 (osd.1) 119 : cluster 0 2.15 scrub starts 2024-04-02T04:49:04.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:04 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.975388+0000 osd.1 (osd.1) 120 : cluster 0 2.15 scrub ok 2024-04-02T04:49:04.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:04.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.841 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:04.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:04.970 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:04.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:05.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:05.099 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:05.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:05.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:05.360 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:05.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.832594+0000 mgr.y (mgr.14538) 194 : cluster 0 pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.984173+0000 osd.0 (osd.0) 123 : cluster 0 6.16 scrub starts 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:03.986297+0000 osd.0 (osd.0) 124 : cluster 0 6.16 scrub ok 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:04.425033+0000 osd.7 (osd.7) 127 : cluster 0 3.e scrub starts 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:04.429304+0000 osd.7 (osd.7) 128 : cluster 0 3.e scrub ok 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:04.485089+0000 osd.3 (osd.3) 151 : cluster 0 3.2 scrub starts 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:04.488024+0000 osd.3 (osd.3) 152 : cluster 0 3.2 scrub ok 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.018434+0000 osd.1 (osd.1) 121 : cluster 0 2.d scrub starts 2024-04-02T04:49:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:05 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.020973+0000 osd.1 (osd.1) 122 : cluster 0 2.d scrub ok 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.832594+0000 mgr.y (mgr.14538) 194 : cluster 0 pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.984173+0000 osd.0 (osd.0) 123 : cluster 0 6.16 scrub starts 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:03.986297+0000 osd.0 (osd.0) 124 : cluster 0 6.16 scrub ok 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:04.425033+0000 osd.7 (osd.7) 127 : cluster 0 3.e scrub starts 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:04.429304+0000 osd.7 (osd.7) 128 : cluster 0 3.e scrub ok 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:04.485089+0000 osd.3 (osd.3) 151 : cluster 0 3.2 scrub starts 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:04.488024+0000 osd.3 (osd.3) 152 : cluster 0 3.2 scrub ok 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.018434+0000 osd.1 (osd.1) 121 : cluster 0 2.d scrub starts 2024-04-02T04:49:05.475 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.020973+0000 osd.1 (osd.1) 122 : cluster 0 2.d scrub ok 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.832594+0000 mgr.y (mgr.14538) 194 : cluster 0 pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.984173+0000 osd.0 (osd.0) 123 : cluster 0 6.16 scrub starts 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:03.986297+0000 osd.0 (osd.0) 124 : cluster 0 6.16 scrub ok 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:04.425033+0000 osd.7 (osd.7) 127 : cluster 0 3.e scrub starts 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:04.429304+0000 osd.7 (osd.7) 128 : cluster 0 3.e scrub ok 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:04.485089+0000 osd.3 (osd.3) 151 : cluster 0 3.2 scrub starts 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:04.488024+0000 osd.3 (osd.3) 152 : cluster 0 3.2 scrub ok 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.018434+0000 osd.1 (osd.1) 121 : cluster 0 2.d scrub starts 2024-04-02T04:49:05.476 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:05 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.020973+0000 osd.1 (osd.1) 122 : cluster 0 2.d scrub ok 2024-04-02T04:49:05.490 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:05.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:05.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:05.750 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:05.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:05.880 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:05.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:05.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:06.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:06.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:06.271 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:06.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:06.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:06.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:06.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[21528]: cluster 2024-04-02T04:49:04.645974+0000 osd.4 (osd.4) 125 : cluster 0 4.e scrub starts 2024-04-02T04:49:06.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[21528]: cluster 2024-04-02T04:49:04.647943+0000 osd.4 (osd.4) 126 : cluster 0 4.e scrub ok 2024-04-02T04:49:06.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.016048+0000 osd.0 (osd.0) 125 : cluster 0 3.1b scrub starts 2024-04-02T04:49:06.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.019928+0000 osd.0 (osd.0) 126 : cluster 0 3.1b scrub ok 2024-04-02T04:49:06.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.444773+0000 osd.3 (osd.3) 153 : cluster 0 6.1f scrub starts 2024-04-02T04:49:06.516 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.447011+0000 osd.3 (osd.3) 154 : cluster 0 6.1f scrub ok 2024-04-02T04:49:06.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[28904]: cluster 2024-04-02T04:49:04.645974+0000 osd.4 (osd.4) 125 : cluster 0 4.e scrub starts 2024-04-02T04:49:06.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[28904]: cluster 2024-04-02T04:49:04.647943+0000 osd.4 (osd.4) 126 : cluster 0 4.e scrub ok 2024-04-02T04:49:06.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.016048+0000 osd.0 (osd.0) 125 : cluster 0 3.1b scrub starts 2024-04-02T04:49:06.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.019928+0000 osd.0 (osd.0) 126 : cluster 0 3.1b scrub ok 2024-04-02T04:49:06.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.444773+0000 osd.3 (osd.3) 153 : cluster 0 6.1f scrub starts 2024-04-02T04:49:06.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:06 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.447011+0000 osd.3 (osd.3) 154 : cluster 0 6.1f scrub ok 2024-04-02T04:49:06.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:06.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:06.659 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:06.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:06 smithi098 bash[23790]: cluster 2024-04-02T04:49:04.645974+0000 osd.4 (osd.4) 125 : cluster 0 4.e scrub starts 2024-04-02T04:49:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:06 smithi098 bash[23790]: cluster 2024-04-02T04:49:04.647943+0000 osd.4 (osd.4) 126 : cluster 0 4.e scrub ok 2024-04-02T04:49:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:06 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.016048+0000 osd.0 (osd.0) 125 : cluster 0 3.1b scrub starts 2024-04-02T04:49:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:06 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.019928+0000 osd.0 (osd.0) 126 : cluster 0 3.1b scrub ok 2024-04-02T04:49:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:06 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.444773+0000 osd.3 (osd.3) 153 : cluster 0 6.1f scrub starts 2024-04-02T04:49:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:06 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.447011+0000 osd.3 (osd.3) 154 : cluster 0 6.1f scrub ok 2024-04-02T04:49:06.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:06.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:06.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:06.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:07.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:07.049 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:07.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:07.179 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:07.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:07.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:07.440 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:07.544 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.647605+0000 osd.4 (osd.4) 127 : cluster 0 4.18 deep-scrub starts 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.649916+0000 osd.4 (osd.4) 128 : cluster 0 4.18 deep-scrub ok 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[21528]: cluster 2024-04-02T04:49:05.833283+0000 mgr.y (mgr.14538) 195 : cluster 0 pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[21528]: cluster 2024-04-02T04:49:06.796575+0000 osd.5 (osd.5) 125 : cluster 0 4.8 scrub starts 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[21528]: cluster 2024-04-02T04:49:06.797650+0000 osd.5 (osd.5) 126 : cluster 0 4.8 scrub ok 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[21528]: audit 2024-04-02T04:49:07.258673+0000 mon.b (mon.1) 105 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.647605+0000 osd.4 (osd.4) 127 : cluster 0 4.18 deep-scrub starts 2024-04-02T04:49:07.555 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.649916+0000 osd.4 (osd.4) 128 : cluster 0 4.18 deep-scrub ok 2024-04-02T04:49:07.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[28904]: cluster 2024-04-02T04:49:05.833283+0000 mgr.y (mgr.14538) 195 : cluster 0 pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:07.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[28904]: cluster 2024-04-02T04:49:06.796575+0000 osd.5 (osd.5) 125 : cluster 0 4.8 scrub starts 2024-04-02T04:49:07.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[28904]: cluster 2024-04-02T04:49:06.797650+0000 osd.5 (osd.5) 126 : cluster 0 4.8 scrub ok 2024-04-02T04:49:07.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:07 smithi069 bash[28904]: audit 2024-04-02T04:49:07.258673+0000 mon.b (mon.1) 105 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:07.570 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:07.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:07.701 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:07 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.647605+0000 osd.4 (osd.4) 127 : cluster 0 4.18 deep-scrub starts 2024-04-02T04:49:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:07 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.649916+0000 osd.4 (osd.4) 128 : cluster 0 4.18 deep-scrub ok 2024-04-02T04:49:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:07 smithi098 bash[23790]: cluster 2024-04-02T04:49:05.833283+0000 mgr.y (mgr.14538) 195 : cluster 0 pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:07 smithi098 bash[23790]: cluster 2024-04-02T04:49:06.796575+0000 osd.5 (osd.5) 125 : cluster 0 4.8 scrub starts 2024-04-02T04:49:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:07 smithi098 bash[23790]: cluster 2024-04-02T04:49:06.797650+0000 osd.5 (osd.5) 126 : cluster 0 4.8 scrub ok 2024-04-02T04:49:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:07 smithi098 bash[23790]: audit 2024-04-02T04:49:07.258673+0000 mon.b (mon.1) 105 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:07.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:07.831 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:07.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:07.961 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:07.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:08.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:08.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:08.211 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:08.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:08.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: cluster 2024-04-02T04:49:06.649869+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-04-02T04:49:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: cluster 2024-04-02T04:49:06.651911+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-04-02T04:49:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.342394+0000 osd.7 (osd.7) 129 : cluster 0 1.0 scrub starts 2024-04-02T04:49:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.344602+0000 osd.7 (osd.7) 130 : cluster 0 1.0 scrub ok 2024-04-02T04:49:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.370725+0000 osd.3 (osd.3) 155 : cluster 0 3.7 scrub starts 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.374865+0000 osd.3 (osd.3) 156 : cluster 0 3.7 scrub ok 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[28904]: audit 2024-04-02T04:49:07.793221+0000 mon.b (mon.1) 106 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: cluster 2024-04-02T04:49:06.649869+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: cluster 2024-04-02T04:49:06.651911+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.342394+0000 osd.7 (osd.7) 129 : cluster 0 1.0 scrub starts 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.344602+0000 osd.7 (osd.7) 130 : cluster 0 1.0 scrub ok 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.370725+0000 osd.3 (osd.3) 155 : cluster 0 3.7 scrub starts 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.374865+0000 osd.3 (osd.3) 156 : cluster 0 3.7 scrub ok 2024-04-02T04:49:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21528]: audit 2024-04-02T04:49:07.793221+0000 mon.b (mon.1) 106 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:08.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:08.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: cluster 2024-04-02T04:49:06.649869+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-04-02T04:49:08.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: cluster 2024-04-02T04:49:06.651911+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-04-02T04:49:08.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.342394+0000 osd.7 (osd.7) 129 : cluster 0 1.0 scrub starts 2024-04-02T04:49:08.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.344602+0000 osd.7 (osd.7) 130 : cluster 0 1.0 scrub ok 2024-04-02T04:49:08.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.370725+0000 osd.3 (osd.3) 155 : cluster 0 3.7 scrub starts 2024-04-02T04:49:08.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.374865+0000 osd.3 (osd.3) 156 : cluster 0 3.7 scrub ok 2024-04-02T04:49:08.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:08 smithi098 bash[23790]: audit 2024-04-02T04:49:07.793221+0000 mon.b (mon.1) 106 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:08.582 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-02T04:49:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:08.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:08.718 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:08.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:08.844 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:08.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:08.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:09.005 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:49:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:49:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:49:09.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.101 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:09.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:09.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/71840108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc06a930 msgr2=0x7f62bc10f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- 172.21.15.69:0/71840108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc06a930 0x7f62bc10f0f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f62b400b0a0 tx=0x7f62b400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/71840108 shutdown_connections 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- 172.21.15.69:0/71840108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc06a930 0x7f62bc10f0f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- 172.21.15.69:0/71840108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc1082d0 0x7f62bc1086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- 172.21.15.69:0/71840108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc069f70 0x7f62bc06a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/71840108 >> 172.21.15.69:0/71840108 conn(0x7f62bc0771e0 msgr2=0x7f62bc0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/71840108 shutdown_connections 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/71840108 wait complete. 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 librados: starting msgr at 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 librados: starting objecter 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 Processor -- start 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- start start 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 librados: setting wanted keys 2024-04-02T04:49:09.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 librados: calling monclient init 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc069f70 0x7f62bc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 0x7f62bc1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc1082d0 0x7f62bc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc111ac0 con 0x7f62bc06a930 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc111940 con 0x7f62bc069f70 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f62bc111c40 con 0x7f62bc1082d0 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 0x7f62bc1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 0x7f62bc1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47708/0 (socket says 172.21.15.69:47708) 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 -- 172.21.15.69:0/2520800861 learned_addr learned my addr 172.21.15.69:0/2520800861 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:49:09.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc1082d0 0x7f62bc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc069f70 0x7f62bc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc1082d0 msgr2=0x7f62bc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62bc1082d0 0x7f62bc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc069f70 msgr2=0x7f62bc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc069f70 0x7f62bc197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc19cf70 con 0x7f62bc06a930 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 0x7f62bc1984f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00c970 tx=0x7f62ac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62b2ffd700 10 client.?.objecter ms_handle_connect 0x7f62bc06a930 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62b2ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac007aa0 con 0x7f62bc06a930 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62bc19d260 con 0x7f62bc06a930 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f62ac004cb0 con 0x7f62bc06a930 2024-04-02T04:49:09.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac005470 con 0x7f62bc06a930 2024-04-02T04:49:09.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 10 client.14823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:49:09.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62bc1a4ae0 con 0x7f62bc06a930 2024-04-02T04:49:09.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.298+0000 7f62c3d01700 1 librados: init done 2024-04-02T04:49:09.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f62bc04ff00 con 0x7f62bc06a930 2024-04-02T04:49:09.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f62ac004750 con 0x7f62bc06a930 2024-04-02T04:49:09.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f62a806f5e0 0x7f62a8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(133..133 src has 74..133) ==== 7537+0+0 (secure 0 0 0) 0x7f62ac08e4e0 con 0x7f62bc06a930 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 osd_map(133..133 src has 74..133) 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 3 client.14823.objecter handle_osd_map got epochs [133,133] > 0 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 3 client.14823.objecter handle_osd_map decoding full epoch 133 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f62a806f5e0 0x7f62a8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 20 client.14823.objecter dump_active .. 0 homeless 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v133) ==== 54+0+16 (secure 0 0 0) 0x7f62ac025020 con 0x7f62bc06a930 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 10 client.14823.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 1 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f62bc062e50 0x7f62bc063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 1 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 0 1 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 4 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 10 client.14823.objecter _send_command 1 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f62a806f5e0 0x7f62a8071aa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009680 tx=0x7f62b8009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62a806f5e0 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1a7ea0 con 0x7f62bc062e50 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f62bc062e50 0x7f62bc063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f62bc062e50 0x7f62bc063230 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc062e50 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1a7ea0 con 0x7f62bc062e50 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 1: 0 ) 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 1 = system:0 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 4 2024-04-02T04:49:09.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.302+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 0 1 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 10 client.14823.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 2 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 2 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1992c0 osd=0 3 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 0 2 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 4 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 10 client.14823.objecter _send_command 2 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.322+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc0628d0 con 0x7f62bc062e50 2024-04-02T04:49:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc0628d0 con 0x7f62bc062e50 2024-04-02T04:49:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 4 2024-04-02T04:49:09.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 0 2 2024-04-02T04:49:09.325 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:49:09.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 10 client.14823.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f62bc1a8050 0x7f62bc1a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 1 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 4 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 10 client.14823.objecter _send_command 3 2024-04-02T04:49:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1031b0 con 0x7f62bc1a8050 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f62bc1a8050 0x7f62bc1a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f62bc1a8050 0x7f62bc1a8450 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc1a8050 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1031b0 con 0x7f62bc1a8050 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 3: 0 ) 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 3 = system:0 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 4 2024-04-02T04:49:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.326+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 1 3 2024-04-02T04:49:09.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:09.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 10 client.14823.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 4 2024-04-02T04:49:09.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 4 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1a74f0 osd=1 3 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 1 4 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 4 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 10 client.14823.objecter _send_command 4 2024-04-02T04:49:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc1a8990 con 0x7f62bc1a8050 2024-04-02T04:49:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc1a8990 con 0x7f62bc1a8050 2024-04-02T04:49:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 4 2024-04-02T04:49:09.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 1 4 2024-04-02T04:49:09.347 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:49:09.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 10 client.14823.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 5 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f62bc02cec0 0x7f62bc02d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 5 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 2 5 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 4 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 10 client.14823.objecter _send_command 5 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1a8f60 con 0x7f62bc02cec0 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f62bc02cec0 0x7f62bc02d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc02cec0 2024-04-02T04:49:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.346+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f62bc02cec0 0x7f62bc02d330 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.350+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1a8f60 con 0x7f62bc02cec0 2024-04-02T04:49:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.350+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 5: 0 ) 2024-04-02T04:49:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.350+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 5 = system:0 2024-04-02T04:49:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.350+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 4 2024-04-02T04:49:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.350+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 2 5 2024-04-02T04:49:09.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 10 client.14823.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 6 2024-04-02T04:49:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 6 2024-04-02T04:49:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1a8df0 osd=2 3 2024-04-02T04:49:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 2 6 2024-04-02T04:49:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 4 2024-04-02T04:49:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 10 client.14823.objecter _send_command 6 2024-04-02T04:49:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.366+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc1a6f70 con 0x7f62bc02cec0 2024-04-02T04:49:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc1a6f70 con 0x7f62bc02cec0 2024-04-02T04:49:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 4 2024-04-02T04:49:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 2 6 2024-04-02T04:49:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:49:09.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 10 client.14823.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 7 2024-04-02T04:49:09.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f62bc1afb00 0x7f62bc1b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 7 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 3 7 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 4 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 10 client.14823.objecter _send_command 7 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1b4d50 con 0x7f62bc1afb00 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f62bc1afb00 0x7f62bc1b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc1afb00 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f62bc1afb00 0x7f62bc1b1ef0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1b4d50 con 0x7f62bc1afb00 2024-04-02T04:49:09.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 7: 0 ) 2024-04-02T04:49:09.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 7 = system:0 2024-04-02T04:49:09.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 4 2024-04-02T04:49:09.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.370+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 3 7 2024-04-02T04:49:09.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 10 client.14823.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 8 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 8 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1a7370 osd=3 3 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 3 8 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 4 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 10 client.14823.objecter _send_command 8 2024-04-02T04:49:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc02db80 con 0x7f62bc1afb00 2024-04-02T04:49:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc02db80 con 0x7f62bc1afb00 2024-04-02T04:49:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 4 2024-04-02T04:49:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 3 8 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 10 client.14823.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 9 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f62bc10adb0 0x7f62bc1b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 9 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 4 9 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 4 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 10 client.14823.objecter _send_command 9 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1b4c70 con 0x7f62bc10adb0 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f62bc10adb0 0x7f62bc1b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc10adb0 2024-04-02T04:49:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.390+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f62bc10adb0 0x7f62bc1b35b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.394+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1b4c70 con 0x7f62bc10adb0 2024-04-02T04:49:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.394+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 9: 0 ) 2024-04-02T04:49:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.394+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 9 = system:0 2024-04-02T04:49:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.394+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 4 2024-04-02T04:49:09.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.394+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 4 9 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 10 client.14823.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 10 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 10 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc02d930 osd=4 3 2024-04-02T04:49:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 4 10 2024-04-02T04:49:09.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 4 2024-04-02T04:49:09.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 10 client.14823.objecter _send_command 10 2024-04-02T04:49:09.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.410+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc1b3af0 con 0x7f62bc10adb0 2024-04-02T04:49:09.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc1b3af0 con 0x7f62bc10adb0 2024-04-02T04:49:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 4 2024-04-02T04:49:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 4 10 2024-04-02T04:49:09.414 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 10 client.14823.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 11 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f62bc1b4170 0x7f62bc1b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 11 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 5 11 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 4 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 10 client.14823.objecter _send_command 11 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1b4ba0 con 0x7f62bc1b4170 2024-04-02T04:49:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f62bc1b4170 0x7f62bc1b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62c129c700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f62bc1b4170 0x7f62bc1b45d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.414+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc1b4170 2024-04-02T04:49:09.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.418+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1b4ba0 con 0x7f62bc1b4170 2024-04-02T04:49:09.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.418+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 11: 0 ) 2024-04-02T04:49:09.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.418+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 11 = system:0 2024-04-02T04:49:09.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.418+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 4 2024-04-02T04:49:09.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.418+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 5 11 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 10 client.14823.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 12 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 12 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1b3af0 osd=5 3 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 5 12 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 4 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 10 client.14823.objecter _send_command 12 2024-04-02T04:49:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.434+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc1b4ba0 con 0x7f62bc1b4170 2024-04-02T04:49:09.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc1b4ba0 con 0x7f62bc1b4170 2024-04-02T04:49:09.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 4 2024-04-02T04:49:09.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 5 12 2024-04-02T04:49:09.438 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:49:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 10 client.14823.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 13 2024-04-02T04:49:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f62bc1b94c0 0x7f62bc1bb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 13 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 6 13 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 4 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 10 client.14823.objecter _send_command 13 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1be8b0 con 0x7f62bc1b94c0 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f62bc1b94c0 0x7f62bc1bb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc1b94c0 2024-04-02T04:49:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.438+0000 7f62c229e700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f62bc1b94c0 0x7f62bc1bb8b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.442+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1be8b0 con 0x7f62bc1b94c0 2024-04-02T04:49:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.442+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 13: 0 ) 2024-04-02T04:49:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.442+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 13 = system:0 2024-04-02T04:49:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.442+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 4 2024-04-02T04:49:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.442+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 6 13 2024-04-02T04:49:09.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 10 client.14823.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 14 2024-04-02T04:49:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 14 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc02dd00 osd=6 3 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 6 14 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 4 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 10 client.14823.objecter _send_command 14 2024-04-02T04:49:09.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.458+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc1bbee0 con 0x7f62bc1b94c0 2024-04-02T04:49:09.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.462+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc1bbee0 con 0x7f62bc1b94c0 2024-04-02T04:49:09.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 4 2024-04-02T04:49:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 6 14 2024-04-02T04:49:09.464 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 10 client.14823.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 15 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f62bc1bc600 0x7f62bc1bca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 15 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 7 15 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 4 2024-04-02T04:49:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 10 client.14823.objecter _send_command 15 2024-04-02T04:49:09.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f62bc1c1c80 con 0x7f62bc1bc600 2024-04-02T04:49:09.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f62bc1bc600 0x7f62bc1bca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:09.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 10 client.14823.objecter ms_handle_connect 0x7f62bc1bc600 2024-04-02T04:49:09.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62c1a9d700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f62bc1bc600 0x7f62bc1bca60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:09.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.673028+0000 osd.6 (osd.6) 85 : cluster 0 2.11 deep-scrub starts 2024-04-02T04:49:09.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.675117+0000 osd.4 (osd.4) 131 : cluster 0 6.1a deep-scrub starts 2024-04-02T04:49:09.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.675596+0000 osd.6 (osd.6) 86 : cluster 0 2.11 deep-scrub ok 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.677728+0000 osd.4 (osd.4) 132 : cluster 0 6.1a deep-scrub ok 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:07.834174+0000 mgr.y (mgr.14538) 196 : cluster 0 pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:08.350104+0000 osd.3 (osd.3) 157 : cluster 0 4.11 scrub starts 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: cluster 2024-04-02T04:49:08.351205+0000 osd.3 (osd.3) 158 : cluster 0 4.11 scrub ok 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: audit 2024-04-02T04:49:08.472399+0000 mon.b (mon.1) 107 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: audit 2024-04-02T04:49:08.474159+0000 mon.b (mon.1) 108 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: audit 2024-04-02T04:49:08.488043+0000 mon.a (mon.0) 955 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[21528]: audit 2024-04-02T04:49:09.304228+0000 mon.a (mon.0) 956 : audit 0 from='client.? 172.21.15.69:0/2520800861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.673028+0000 osd.6 (osd.6) 85 : cluster 0 2.11 deep-scrub starts 2024-04-02T04:49:09.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.675117+0000 osd.4 (osd.4) 131 : cluster 0 6.1a deep-scrub starts 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.675596+0000 osd.6 (osd.6) 86 : cluster 0 2.11 deep-scrub ok 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.677728+0000 osd.4 (osd.4) 132 : cluster 0 6.1a deep-scrub ok 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:07.834174+0000 mgr.y (mgr.14538) 196 : cluster 0 pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:08.350104+0000 osd.3 (osd.3) 157 : cluster 0 4.11 scrub starts 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: cluster 2024-04-02T04:49:08.351205+0000 osd.3 (osd.3) 158 : cluster 0 4.11 scrub ok 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: audit 2024-04-02T04:49:08.472399+0000 mon.b (mon.1) 107 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: audit 2024-04-02T04:49:08.474159+0000 mon.b (mon.1) 108 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: audit 2024-04-02T04:49:08.488043+0000 mon.a (mon.0) 955 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:09.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:09 smithi069 bash[28904]: audit 2024-04-02T04:49:09.304228+0000 mon.a (mon.0) 956 : audit 0 from='client.? 172.21.15.69:0/2520800861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:09.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62bc1c1c80 con 0x7f62bc1bc600 2024-04-02T04:49:09.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 15: 0 ) 2024-04-02T04:49:09.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 15 = system:0 2024-04-02T04:49:09.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 4 2024-04-02T04:49:09.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.466+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 7 15 2024-04-02T04:49:09.469 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.673028+0000 osd.6 (osd.6) 85 : cluster 0 2.11 deep-scrub starts 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.675117+0000 osd.4 (osd.4) 131 : cluster 0 6.1a deep-scrub starts 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.675596+0000 osd.6 (osd.6) 86 : cluster 0 2.11 deep-scrub ok 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.677728+0000 osd.4 (osd.4) 132 : cluster 0 6.1a deep-scrub ok 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:07.834174+0000 mgr.y (mgr.14538) 196 : cluster 0 pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:08.350104+0000 osd.3 (osd.3) 157 : cluster 0 4.11 scrub starts 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: cluster 2024-04-02T04:49:08.351205+0000 osd.3 (osd.3) 158 : cluster 0 4.11 scrub ok 2024-04-02T04:49:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: audit 2024-04-02T04:49:08.472399+0000 mon.b (mon.1) 107 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: audit 2024-04-02T04:49:08.474159+0000 mon.b (mon.1) 108 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:49:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: audit 2024-04-02T04:49:08.488043+0000 mon.a (mon.0) 955 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:09 smithi098 bash[23790]: audit 2024-04-02T04:49:09.304228+0000 mon.a (mon.0) 956 : audit 0 from='client.? 172.21.15.69:0/2520800861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:09.477 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:09.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 10 client.14823.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:09.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign -1 16 2024-04-02T04:49:09.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 20 client.14823.objecter _get_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_remove -1 16 2024-04-02T04:49:09.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 20 client.14823.objecter get_session s=0x7f62bc1bbee0 osd=7 3 2024-04-02T04:49:09.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 15 client.14823.objecter _session_command_op_assign 7 16 2024-04-02T04:49:09.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 4 2024-04-02T04:49:09.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 10 client.14823.objecter _send_command 16 2024-04-02T04:49:09.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.486+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f62bc1c0b00 con 0x7f62bc1bc600 2024-04-02T04:49:09.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62b2ffd700 1 -- 172.21.15.69:0/2520800861 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62bc1c0b00 con 0x7f62bc1bc600 2024-04-02T04:49:09.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62b2ffd700 10 client.14823.objecter ms_dispatch 0x7f62bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:09.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62b2ffd700 10 client.14823.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:09.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62b2ffd700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 4 2024-04-02T04:49:09.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62b2ffd700 15 client.14823.objecter _session_command_op_remove 7 16 2024-04-02T04:49:09.490 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 librados: watch_flush enter 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 librados: watch_flush exit 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.0 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f62bc062e50 msgr2=0x7f62bc063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f62bc062e50 0x7f62bc063230 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1992c0 osd=0 1 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.1 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f62bc1a8050 msgr2=0x7f62bc1a8450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f62bc1a8050 0x7f62bc1a8450 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a74f0 osd=1 1 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.2 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f62bc02cec0 msgr2=0x7f62bc02d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f62bc02cec0 0x7f62bc02d330 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a8df0 osd=2 1 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.3 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f62bc1afb00 msgr2=0x7f62bc1b1ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f62bc1afb00 0x7f62bc1b1ef0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1a7370 osd=3 1 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.4 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f62bc10adb0 msgr2=0x7f62bc1b35b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f62bc10adb0 0x7f62bc1b35b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02d930 osd=4 1 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.5 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f62bc1b4170 msgr2=0x7f62bc1b45d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f62bc1b4170 0x7f62bc1b45d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1b3af0 osd=5 1 2024-04-02T04:49:09.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.6 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f62bc1b94c0 msgr2=0x7f62bc1bb8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f62bc1b94c0 0x7f62bc1bb8b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc02dd00 osd=6 1 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter close_session for osd.7 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f62bc1bc600 msgr2=0x7f62bc1bca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7f62bc1bc600 0x7f62bc1bca60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter put_session s=0x7f62bc1bbee0 osd=7 1 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 20 client.14823.objecter shutdown clearing up homeless session... 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 10 client.14823.objecter successfully canceled tick 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c229e700 1 -- 172.21.15.69:0/2520800861 reap_dead start 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c229e700 1 -- 172.21.15.69:0/2520800861 reap_dead start 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c229e700 1 -- 172.21.15.69:0/2520800861 reap_dead start 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c229e700 1 -- 172.21.15.69:0/2520800861 reap_dead start 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f62a806f5e0 msgr2=0x7f62a8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f62a806f5e0 0x7f62a8071aa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009680 tx=0x7f62b8009260 comp rx=0 tx=0).stop 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c229e700 1 -- 172.21.15.69:0/2520800861 reap_dead start 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 msgr2=0x7f62bc1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 0x7f62bc1984f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00c970 tx=0x7f62ac00cc80 comp rx=0 tx=0).stop 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 shutdown_connections 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62bc06a930 0x7f62bc1984f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 --2- 172.21.15.69:0/2520800861 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f62a806f5e0 0x7f62a8071aa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 >> 172.21.15.69:0/2520800861 conn(0x7f62bc0771e0 msgr2=0x7f62bc109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 shutdown_connections 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 -- 172.21.15.69:0/2520800861 wait complete. 2024-04-02T04:49:09.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:09.490+0000 7f62c3d01700 1 librados: shutdown 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:49:09.495 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:49:09.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:09.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.722 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-02T04:49:09.724 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-02T04:49:09.724 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-02T04:49:09.729 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:09.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:09.858 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:09.858 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-02T04:49:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:09.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:09.862 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-02T04:49:09.862 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-02T04:49:09.868 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-02T04:49:09.869 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-02T04:49:09.869 DEBUG:teuthology.orchestra.run.smithi069:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-02T04:49:09.925 INFO:tasks.workunit.client.0.smithi069.stderr:+ parallel=1 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --serial ']' 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ crimson=0 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --crimson ']' 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ color= 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -t 1 ']' 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-02T04:49:09.926 INFO:tasks.workunit.client.0.smithi069.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-02T04:49:09.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ declare -A pids 2024-04-02T04:49:09.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.927 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio 2024-04-02T04:49:09.927 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio' 2024-04-02T04:49:09.928 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio 2024-04-02T04:49:09.928 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.930 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio 2024-04-02T04:49:09.930 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72140 2024-04-02T04:49:09.930 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio on pid 72140' 2024-04-02T04:49:09.930 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio on pid 72140 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72140 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio_pp 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio_pp' 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio_pp 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.933 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-02T04:49:09.933 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-02T04:49:09.933 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio: /' 2024-04-02T04:49:09.933 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio_pp 2024-04-02T04:49:09.933 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72148 2024-04-02T04:49:09.933 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio_pp on pid 72148' 2024-04-02T04:49:09.934 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio_pp on pid 72148 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72148 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io' 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-02T04:49:09.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-02T04:49:09.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-02T04:49:09.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io 2024-04-02T04:49:09.937 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72156 2024-04-02T04:49:09.937 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io on pid 72156' 2024-04-02T04:49:09.937 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72156 2024-04-02T04:49:09.937 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io on pid 72156 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io_pp 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io_pp' 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io_pp 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-02T04:49:09.939 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io.log 2024-04-02T04:49:09.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io: /' 2024-04-02T04:49:09.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io_pp 2024-04-02T04:49:09.940 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72164 2024-04-02T04:49:09.940 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io_pp on pid 72164 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io_pp on pid 72164' 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72164 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_asio 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_asio' 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_asio 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.943 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.943 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-02T04:49:09.943 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-02T04:49:09.943 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-02T04:49:09.944 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_asio 2024-04-02T04:49:09.944 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72172 2024-04-02T04:49:09.944 INFO:tasks.workunit.client.0.smithi069.stdout:test api_asio on pid 72172 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_asio on pid 72172' 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72172 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_list 2024-04-02T04:49:09.945 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_list' 2024-04-02T04:49:09.946 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_list 2024-04-02T04:49:09.946 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_asio: /' 2024-04-02T04:49:09.947 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-02T04:49:09.948 INFO:tasks.workunit.client.0.smithi069.stdout:test api_list on pid 72180 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_list 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72180 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_list on pid 72180' 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72180 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.949 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock' 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.950 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.951 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-02T04:49:09.951 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_list: /' 2024-04-02T04:49:09.951 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_list.log 2024-04-02T04:49:09.952 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock on pid 72188 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72188 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock on pid 72188' 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72188 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.953 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-02T04:49:09.954 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.954 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.954 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.954 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.955 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock_pp 2024-04-02T04:49:09.955 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock: /' 2024-04-02T04:49:09.955 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock_pp' 2024-04-02T04:49:09.956 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-02T04:49:09.956 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock_pp 2024-04-02T04:49:09.956 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-02T04:49:09.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:09.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.965 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock_pp 2024-04-02T04:49:09.966 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock_pp on pid 72201 2024-04-02T04:49:09.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72201 2024-04-02T04:49:09.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock_pp on pid 72201' 2024-04-02T04:49:09.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72201 2024-04-02T04:49:09.967 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.967 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc 2024-04-02T04:49:09.967 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc' 2024-04-02T04:49:09.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-02T04:49:09.972 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:09.973 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.974 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.974 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.974 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-02T04:49:09.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-02T04:49:09.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-02T04:49:09.983 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc on pid 72245 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72245 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc on pid 72245' 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72245 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:09.987 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-02T04:49:09.989 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc_pp 2024-04-02T04:49:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc_pp' 2024-04-02T04:49:09.992 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:09.992 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.992 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:09.992 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:09.993 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-02T04:49:09.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-02T04:49:09.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.004 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc_pp 2024-04-02T04:49:10.004 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc: /' 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc_pp on pid 72299 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc_pp 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72299 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc_pp on pid 72299' 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72299 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.008 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_tier_pp 2024-04-02T04:49:10.008 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_tier_pp' 2024-04-02T04:49:10.008 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_tier_pp 2024-04-02T04:49:10.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-02T04:49:10.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-02T04:49:10.015 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:10.015 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-02T04:49:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:10.018 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.024 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_tier_pp 2024-04-02T04:49:10.024 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-02T04:49:10.025 INFO:tasks.workunit.client.0.smithi069.stdout:test api_tier_pp on pid 72344 2024-04-02T04:49:10.025 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72344 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_tier_pp on pid 72344' 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72344 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_pool 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.027 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-02T04:49:10.027 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-02T04:49:10.027 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-02T04:49:10.028 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_pool' 2024-04-02T04:49:10.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_pool 2024-04-02T04:49:10.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.031 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_pool 2024-04-02T04:49:10.031 INFO:tasks.workunit.client.0.smithi069.stdout:test api_pool on pid 72356 2024-04-02T04:49:10.031 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72356 2024-04-02T04:49:10.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_pool on pid 72356' 2024-04-02T04:49:10.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72356 2024-04-02T04:49:10.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.032 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-02T04:49:10.033 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.033 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.033 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.034 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.034 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-02T04:49:10.035 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-02T04:49:10.035 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_pool: /' 2024-04-02T04:49:10.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots 2024-04-02T04:49:10.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots' 2024-04-02T04:49:10.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots 2024-04-02T04:49:10.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.042 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots 2024-04-02T04:49:10.042 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots on pid 72373 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72373 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots on pid 72373' 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72373 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots_pp 2024-04-02T04:49:10.043 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots_pp' 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots_pp 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-02T04:49:10.044 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-02T04:49:10.045 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-02T04:49:10.045 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots_pp on pid 72383 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots_pp 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72383 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots_pp on pid 72383' 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72383 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-02T04:49:10.047 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.047 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.048 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.048 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.048 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-02T04:49:10.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat 2024-04-02T04:49:10.048 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat' 2024-04-02T04:49:10.049 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-02T04:49:10.049 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-02T04:49:10.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat 2024-04-02T04:49:10.053 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat on pid 72405 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72405 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat on pid 72405' 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72405 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.057 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.058 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-02T04:49:10.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat_pp 2024-04-02T04:49:10.059 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat_pp' 2024-04-02T04:49:10.060 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.060 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.060 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.060 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.060 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.061 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-02T04:49:10.062 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-02T04:49:10.062 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat: /' 2024-04-02T04:49:10.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat_pp 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat_pp on pid 72413 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat_pp 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72413 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat_pp on pid 72413' 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72413 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify 2024-04-02T04:49:10.066 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify' 2024-04-02T04:49:10.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.067 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-02T04:49:10.068 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify 2024-04-02T04:49:10.069 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify on pid 72422 2024-04-02T04:49:10.069 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify 2024-04-02T04:49:10.069 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72422 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify on pid 72422' 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72422 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-02T04:49:10.070 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify_pp 2024-04-02T04:49:10.071 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify_pp' 2024-04-02T04:49:10.071 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.072 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-02T04:49:10.072 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify_pp 2024-04-02T04:49:10.073 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify_pp on pid 72434 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify_pp 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72434 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify_pp on pid 72434' 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72434 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-02T04:49:10.075 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.075 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.075 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.075 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd 2024-04-02T04:49:10.076 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd' 2024-04-02T04:49:10.077 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd 2024-04-02T04:49:10.077 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-02T04:49:10.077 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-02T04:49:10.079 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-02T04:49:10.083 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-02T04:49:10.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.085 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.085 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-02T04:49:10.086 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-02T04:49:10.087 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd 2024-04-02T04:49:10.087 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-02T04:49:10.087 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72468 2024-04-02T04:49:10.088 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cmd on pid 72468 2024-04-02T04:49:10.088 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd on pid 72468' 2024-04-02T04:49:10.088 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72468 2024-04-02T04:49:10.088 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.089 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.089 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd_pp 2024-04-02T04:49:10.089 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd_pp' 2024-04-02T04:49:10.089 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-02T04:49:10.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd: /' 2024-04-02T04:49:10.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-02T04:49:10.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-02T04:49:10.096 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.097 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd_pp 2024-04-02T04:49:10.102 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cmd_pp on pid 72490 2024-04-02T04:49:10.102 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd_pp 2024-04-02T04:49:10.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72490 2024-04-02T04:49:10.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd_pp on pid 72490' 2024-04-02T04:49:10.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72490 2024-04-02T04:49:10.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.103 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-02T04:49:10.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-02T04:49:10.107 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-02T04:49:10.108 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-02T04:49:10.111 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service 2024-04-02T04:49:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service' 2024-04-02T04:49:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:10.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service 2024-04-02T04:49:10.123 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service 2024-04-02T04:49:10.123 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service on pid 72548 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72548 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service on pid 72548' 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72548 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service_pp 2024-04-02T04:49:10.125 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service_pp' 2024-04-02T04:49:10.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.126 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service: /' 2024-04-02T04:49:10.127 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service.log 2024-04-02T04:49:10.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service_pp 2024-04-02T04:49:10.127 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-02T04:49:10.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.129 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service_pp 2024-04-02T04:49:10.129 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72563 2024-04-02T04:49:10.129 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-02T04:49:10.130 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service_pp on pid 72563 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service_pp on pid 72563' 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72563 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-02T04:49:10.132 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_write_operations 2024-04-02T04:49:10.132 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_write_operations' 2024-04-02T04:49:10.133 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-02T04:49:10.134 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-02T04:49:10.135 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_write_operations 2024-04-02T04:49:10.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_write_operations 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_write_operations on pid 72585 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72585 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_write_operations on pid 72585' 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72585 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.138 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_read_operations 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_read_operations' 2024-04-02T04:49:10.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-02T04:49:10.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-02T04:49:10.145 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-02T04:49:10.148 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.149 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_read_operations 2024-04-02T04:49:10.163 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_read_operations 2024-04-02T04:49:10.163 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_read_operations on pid 72634 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72634 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_read_operations on pid 72634' 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72634 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cls_remote_reads 2024-04-02T04:49:10.164 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cls_remote_reads' 2024-04-02T04:49:10.168 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:10.168 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-02T04:49:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cls_remote_reads 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-02T04:49:10.172 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-02T04:49:10.174 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cls_remote_reads 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cls_remote_reads on pid 72647 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72647 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cls_remote_reads on pid 72647' 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72647 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-02T04:49:10.175 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list_parallel 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list_parallel' 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-02T04:49:10.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-02T04:49:10.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list_parallel 2024-04-02T04:49:10.178 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list_parallel 2024-04-02T04:49:10.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72657 2024-04-02T04:49:10.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list_parallel on pid 72657' 2024-04-02T04:49:10.180 INFO:tasks.workunit.client.0.smithi069.stdout:test list_parallel on pid 72657 2024-04-02T04:49:10.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72657 2024-04-02T04:49:10.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s open_pools_parallel 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' open_pools_parallel' 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-02T04:49:10.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-02T04:49:10.183 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.188 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo open_pools_parallel 2024-04-02T04:49:10.188 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list_parallel: /' 2024-04-02T04:49:10.188 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=open_pools_parallel 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stdout:test open_pools_parallel on pid 72669 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72669 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test open_pools_parallel on pid 72669' 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72669 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.189 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-02T04:49:10.190 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.191 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.191 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.191 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.191 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-02T04:49:10.192 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-02T04:49:10.193 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-02T04:49:10.199 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s delete_pools_parallel 2024-04-02T04:49:10.199 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' delete_pools_parallel' 2024-04-02T04:49:10.200 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo delete_pools_parallel 2024-04-02T04:49:10.210 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stdout:test delete_pools_parallel on pid 72716 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=delete_pools_parallel 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72716 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test delete_pools_parallel on pid 72716' 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72716 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.213 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.221 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-02T04:49:10.223 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s cls 2024-04-02T04:49:10.223 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' cls' 2024-04-02T04:49:10.224 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo cls 2024-04-02T04:49:10.225 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.228 INFO:tasks.workunit.client.0.smithi069.stdout:test cls on pid 72764 2024-04-02T04:49:10.228 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=cls 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72764 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test cls on pid 72764' 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72764 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-02T04:49:10.229 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-02T04:49:10.231 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s cmd 2024-04-02T04:49:10.231 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' cmd' 2024-04-02T04:49:10.231 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-02T04:49:10.235 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-02T04:49:10.237 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.237 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.237 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.237 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.237 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_cls 2024-04-02T04:49:10.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_cls.log 2024-04-02T04:49:10.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ cls: /' 2024-04-02T04:49:10.245 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo cmd 2024-04-02T04:49:10.248 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=cmd 2024-04-02T04:49:10.249 INFO:tasks.workunit.client.0.smithi069.stdout:test cmd on pid 72821 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72821 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test cmd on pid 72821' 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72821 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s handler_error 2024-04-02T04:49:10.250 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' handler_error' 2024-04-02T04:49:10.256 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.256 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.256 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.256 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.256 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.259 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_cmd 2024-04-02T04:49:10.260 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-02T04:49:10.260 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo handler_error 2024-04-02T04:49:10.262 INFO:tasks.workunit.client.0.smithi069.stdout:test handler_error on pid 72844 2024-04-02T04:49:10.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=handler_error 2024-04-02T04:49:10.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72844 2024-04-02T04:49:10.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test handler_error on pid 72844' 2024-04-02T04:49:10.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72844 2024-04-02T04:49:10.262 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.263 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.263 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-02T04:49:10.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_handler_error 2024-04-02T04:49:10.265 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-02T04:49:10.267 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ cmd: /' 2024-04-02T04:49:10.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:10.272 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ handler_error: /' 2024-04-02T04:49:10.273 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s io 2024-04-02T04:49:10.273 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' io' 2024-04-02T04:49:10.274 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo io 2024-04-02T04:49:10.275 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.277 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=io 2024-04-02T04:49:10.277 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72876 2024-04-02T04:49:10.277 INFO:tasks.workunit.client.0.smithi069.stdout:test io on pid 72876 2024-04-02T04:49:10.278 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test io on pid 72876' 2024-04-02T04:49:10.278 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72876 2024-04-02T04:49:10.278 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.278 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.278 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-02T04:49:10.278 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s ec_io 2024-04-02T04:49:10.279 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' ec_io' 2024-04-02T04:49:10.280 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.280 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.280 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.280 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.280 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ io: /' 2024-04-02T04:49:10.282 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_io.log 2024-04-02T04:49:10.282 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_io 2024-04-02T04:49:10.284 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo ec_io 2024-04-02T04:49:10.286 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.290 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=ec_io 2024-04-02T04:49:10.290 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72914 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stdout:test ec_io on pid 72914 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test ec_io on pid 72914' 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72914 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-02T04:49:10.291 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list 2024-04-02T04:49:10.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list' 2024-04-02T04:49:10.292 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.293 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_ec_io 2024-04-02T04:49:10.293 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ ec_io: /' 2024-04-02T04:49:10.293 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list 2024-04-02T04:49:10.295 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-02T04:49:10.297 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.299 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:10.300 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list 2024-04-02T04:49:10.300 INFO:tasks.workunit.client.0.smithi069.stdout:test list on pid 72936 2024-04-02T04:49:10.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72936 2024-04-02T04:49:10.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list on pid 72936' 2024-04-02T04:49:10.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72936 2024-04-02T04:49:10.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s ec_list 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' ec_list' 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo ec_list 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_list 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_list.log 2024-04-02T04:49:10.302 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list: /' 2024-04-02T04:49:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.303 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=ec_list 2024-04-02T04:49:10.303 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72944 2024-04-02T04:49:10.303 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test ec_list on pid 72944' 2024-04-02T04:49:10.304 INFO:tasks.workunit.client.0.smithi069.stdout:test ec_list on pid 72944 2024-04-02T04:49:10.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:10.305 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72944 2024-04-02T04:49:10.305 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.305 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.305 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-02T04:49:10.305 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s misc 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' misc' 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_ec_list 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ ec_list: /' 2024-04-02T04:49:10.312 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo misc 2024-04-02T04:49:10.313 INFO:tasks.workunit.client.0.smithi069.stdout:test misc on pid 72958 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=misc 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72958 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test misc on pid 72958' 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72958 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.314 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_misc 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_misc.log 2024-04-02T04:49:10.315 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ misc: /' 2024-04-02T04:49:10.319 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s pool 2024-04-02T04:49:10.320 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' pool' 2024-04-02T04:49:10.320 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo pool 2024-04-02T04:49:10.323 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.326 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=pool 2024-04-02T04:49:10.326 INFO:tasks.workunit.client.0.smithi069.stdout:test pool on pid 72992 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72992 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test pool on pid 72992' 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=72992 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.327 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s read_operations 2024-04-02T04:49:10.328 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' read_operations' 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_pool 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo read_operations 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_pool.log 2024-04-02T04:49:10.329 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ pool: /' 2024-04-02T04:49:10.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=read_operations 2024-04-02T04:49:10.330 INFO:tasks.workunit.client.0.smithi069.stdout:test read_operations on pid 73008 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73008 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test read_operations on pid 73008' 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=73008 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s snapshots 2024-04-02T04:49:10.331 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' snapshots' 2024-04-02T04:49:10.332 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo snapshots 2024-04-02T04:49:10.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.332 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.333 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.333 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.333 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_read_operations 2024-04-02T04:49:10.333 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ read_operations: /' 2024-04-02T04:49:10.334 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=snapshots 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stdout:test snapshots on pid 73018 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73018 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test snapshots on pid 73018' 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=73018 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.335 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s watch_notify 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' watch_notify' 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo watch_notify 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.337 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_snapshots 2024-04-02T04:49:10.339 INFO:tasks.workunit.client.0.smithi069.stdout:test watch_notify on pid 73029 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=watch_notify 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73029 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test watch_notify on pid 73029' 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=73029 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s write_operations 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-02T04:49:10.340 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' write_operations' 2024-04-02T04:49:10.342 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-02T04:49:10.342 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-02T04:49:10.343 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.343 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.343 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.343 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.344 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-02T04:49:10.346 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo write_operations 2024-04-02T04:49:10.346 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ snapshots: /' 2024-04-02T04:49:10.347 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-04-02T04:49:10.349 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=write_operations 2024-04-02T04:49:10.350 INFO:tasks.workunit.client.0.smithi069.stdout:test write_operations on pid 73043 2024-04-02T04:49:10.350 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73043 2024-04-02T04:49:10.350 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test write_operations on pid 73043' 2024-04-02T04:49:10.350 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=73043 2024-04-02T04:49:10.350 INFO:tasks.workunit.client.0.smithi069.stderr:+ ret=0 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72344 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72344 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -z '' ']' 2024-04-02T04:49:10.351 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.352 INFO:tasks.workunit.client.0.smithi069.stderr:+ case $- in 2024-04-02T04:49:10.352 INFO:tasks.workunit.client.0.smithi069.stderr:+ return 2024-04-02T04:49:10.352 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_write_operations 2024-04-02T04:49:10.354 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_watch_notify 2024-04-02T04:49:10.354 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ watch_notify: /' 2024-04-02T04:49:10.356 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-02T04:49:10.362 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ write_operations: /' 2024-04-02T04:49:10.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:10.422 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:10.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:10.553 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:10.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:10.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:10.809 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:10.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:10.950 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:10.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:11.054 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:11.062 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[21528]: cluster 2024-04-02T04:49:08.688899+0000 osd.6 (osd.6) 87 : cluster 0 2.7 scrub starts 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[21528]: cluster 2024-04-02T04:49:08.690357+0000 osd.6 (osd.6) 88 : cluster 0 2.7 scrub ok 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[21528]: cluster 2024-04-02T04:49:08.960677+0000 osd.0 (osd.0) 127 : cluster 0 4.3 scrub starts 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[21528]: cluster 2024-04-02T04:49:08.963547+0000 osd.0 (osd.0) 128 : cluster 0 4.3 scrub ok 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[21528]: cluster 2024-04-02T04:49:09.387974+0000 osd.3 (osd.3) 159 : cluster 0 6.1b scrub starts 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[21528]: cluster 2024-04-02T04:49:09.389573+0000 osd.3 (osd.3) 160 : cluster 0 6.1b scrub ok 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[28904]: cluster 2024-04-02T04:49:08.688899+0000 osd.6 (osd.6) 87 : cluster 0 2.7 scrub starts 2024-04-02T04:49:11.063 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[28904]: cluster 2024-04-02T04:49:08.690357+0000 osd.6 (osd.6) 88 : cluster 0 2.7 scrub ok 2024-04-02T04:49:11.064 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[28904]: cluster 2024-04-02T04:49:08.960677+0000 osd.0 (osd.0) 127 : cluster 0 4.3 scrub starts 2024-04-02T04:49:11.064 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[28904]: cluster 2024-04-02T04:49:08.963547+0000 osd.0 (osd.0) 128 : cluster 0 4.3 scrub ok 2024-04-02T04:49:11.064 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[28904]: cluster 2024-04-02T04:49:09.387974+0000 osd.3 (osd.3) 159 : cluster 0 6.1b scrub starts 2024-04-02T04:49:11.064 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:10 smithi069 bash[28904]: cluster 2024-04-02T04:49:09.389573+0000 osd.3 (osd.3) 160 : cluster 0 6.1b scrub ok 2024-04-02T04:49:11.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:11.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:11.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:10 smithi098 bash[23790]: cluster 2024-04-02T04:49:08.688899+0000 osd.6 (osd.6) 87 : cluster 0 2.7 scrub starts 2024-04-02T04:49:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:10 smithi098 bash[23790]: cluster 2024-04-02T04:49:08.690357+0000 osd.6 (osd.6) 88 : cluster 0 2.7 scrub ok 2024-04-02T04:49:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:10 smithi098 bash[23790]: cluster 2024-04-02T04:49:08.960677+0000 osd.0 (osd.0) 127 : cluster 0 4.3 scrub starts 2024-04-02T04:49:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:10 smithi098 bash[23790]: cluster 2024-04-02T04:49:08.963547+0000 osd.0 (osd.0) 128 : cluster 0 4.3 scrub ok 2024-04-02T04:49:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:10 smithi098 bash[23790]: cluster 2024-04-02T04:49:09.387974+0000 osd.3 (osd.3) 159 : cluster 0 6.1b scrub starts 2024-04-02T04:49:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:10 smithi098 bash[23790]: cluster 2024-04-02T04:49:09.389573+0000 osd.3 (osd.3) 160 : cluster 0 6.1b scrub ok 2024-04-02T04:49:11.302 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:11.325 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:11.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:11.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:11.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:11.720 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:11.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:11.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:11.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:12.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:12.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:12.237 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:12.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:12.367 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:12.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:12.487 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:12.490 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:49:12.490 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:49:12.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:12.616 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:12.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:12.745 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:12.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:12.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:12.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:13.003 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:13.106 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:13.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:49:12 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:49:13.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:13.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:13.589 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:13.610 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:13.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:13.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-02T04:49:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:13.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:13.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:13.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:49:13.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:14.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:14.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:14.256 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:14.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:14.381 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:14.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:14.506 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:14.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:14.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:14.759 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:14.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3421474475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4102c90 msgr2=0x7fa0f410f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3421474475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4102c90 0x7fa0f410f170 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00b0a0 tx=0x7fa0ec00b3b0 comp rx=0 tx=0).stop 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3421474475 shutdown_connections 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3421474475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4102c90 0x7fa0f410f170 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3421474475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f4108330 0x7fa0f4108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3421474475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f41022f0 0x7fa0f4102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3421474475 >> 172.21.15.69:0/3421474475 conn(0x7fa0f40793f0 msgr2=0x7fa0f4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3421474475 shutdown_connections 2024-04-02T04:49:14.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3421474475 wait complete. 2024-04-02T04:49:14.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 librados: starting msgr at 2024-04-02T04:49:14.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 librados: starting objecter 2024-04-02T04:49:14.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 Processor -- start 2024-04-02T04:49:14.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- start start 2024-04-02T04:49:14.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 librados: setting wanted keys 2024-04-02T04:49:14.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 librados: calling monclient init 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f41022f0 0x7fa0f4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f4102c90 0x7fa0f41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 0x7fa0f419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4111b00 con 0x7fa0f4102c90 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4111980 con 0x7fa0f41022f0 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f4111c80 con 0x7fa0f4108330 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 0x7fa0f419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 0x7fa0f419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33244/0 (socket says 172.21.15.69:33244) 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 -- 172.21.15.69:0/3335571505 learned_addr learned my addr 172.21.15.69:0/3335571505 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0ebfff700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f4102c90 0x7fa0f41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f3fff700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f41022f0 0x7fa0f4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 -- 172.21.15.69:0/3335571505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f41022f0 msgr2=0x7fa0f4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:14.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f41022f0 0x7fa0f4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 -- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f4102c90 msgr2=0x7fa0f41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f4102c90 0x7fa0f41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 -- 172.21.15.69:0/3335571505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f419d080 con 0x7fa0f4108330 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f3fff700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f41022f0 0x7fa0f4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f8b87700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 0x7fa0f419c940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec006070 tx=0x7fa0ec009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f1ffb700 10 client.?.objecter ms_handle_connect 0x7fa0f4108330 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f1ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec003c00 con 0x7fa0f4108330 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa0ec022070 con 0x7fa0f4108330 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec01b460 con 0x7fa0f4108330 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f419d310 con 0x7fa0f4108330 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 10 client.24950.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:49:14.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f41a4bf0 con 0x7fa0f4108330 2024-04-02T04:49:14.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.866+0000 7fa0fa5ea700 1 librados: init done 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f41039e0 con 0x7fa0f4108330 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa0ec01d470 con 0x7fa0f4108330 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0e006f5e0 0x7fa0e0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(134..134 src has 74..134) ==== 8265+0+0 (secure 0 0 0) 0x7fa0ec00f020 con 0x7fa0f4108330 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 10 client.24950.objecter ms_dispatch 0x7fa0f4001550 osd_map(134..134 src has 74..134) 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 3 client.24950.objecter handle_osd_map got epochs [134,134] > 0 2024-04-02T04:49:14.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 3 client.24950.objecter handle_osd_map decoding full epoch 134 2024-04-02T04:49:14.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f3fff700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0e006f5e0 0x7fa0e0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:14.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 20 client.24950.objecter dump_active .. 0 homeless 2024-04-02T04:49:14.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f1ffb700 10 client.24950.objecter ms_handle_connect 0x7fa0e006f5e0 2024-04-02T04:49:14.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.870+0000 7fa0f3fff700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0e006f5e0 0x7fa0e0071aa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4005fd0 tx=0x7fa0e4005d40 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:14.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:14.874+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa0ec05c410 con 0x7fa0f4108330 2024-04-02T04:49:14.882 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:14.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:14.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:14.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:15.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.074+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0f41993d0 con 0x7fa0e006f5e0 2024-04-02T04:49:15.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:15.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.122+0000 7fa0f1ffb700 1 -- 172.21.15.69:0/3335571505 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+500929 (secure 0 0 0) 0x7fa0f41993d0 con 0x7fa0e006f5e0 2024-04-02T04:49:15.124 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 10 librados: watch_flush enter 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 10 librados: watch_flush exit 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 20 client.24950.objecter shutdown clearing up homeless session... 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 10 client.24950.objecter successfully canceled tick 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0e006f5e0 msgr2=0x7fa0e0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0e006f5e0 0x7fa0e0071aa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4005fd0 tx=0x7fa0e4005d40 comp rx=0 tx=0).stop 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 msgr2=0x7fa0f419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 0x7fa0f419c940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec006070 tx=0x7fa0ec009c90 comp rx=0 tx=0).stop 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 shutdown_connections 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0e006f5e0 0x7fa0e0071aa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f4108330 0x7fa0f419c940 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0f41022f0 0x7fa0f4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 --2- 172.21.15.69:0/3335571505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f4102c90 0x7fa0f41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 >> 172.21.15.69:0/3335571505 conn(0x7fa0f40793f0 msgr2=0x7fa0f4100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 shutdown_connections 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 -- 172.21.15.69:0/3335571505 wait complete. 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.130+0000 7fa0fa5ea700 1 librados: shutdown 2024-04-02T04:49:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:49:15.144 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:15.247 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:15.265 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:15.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:15.387 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:15.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:15.515 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:15.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:15.640 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:15.743 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:15.773 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:15.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:15.896 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:15.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:15.923 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-02T04:49:13.835850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1533,"num_read_kb":1347,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":555,"acting":555,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":413,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245176,"kb_used_data":8220,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716040,"statfs":{"total":767960285184,"available":767709224960,"internally_reserved":0,"allocated":8417280,"data_stored":5465530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003900"},"pg_stats":[{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.921999+0000","last_change":"2024-04-02T04:49:10.921999+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.921999+0000","last_undegraded":"2024-04-02T04:49:10.921999+0000","last_fullsized":"2024-04-02T04:49:10.921999+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.926531+0000","last_change":"2024-04-02T04:49:06.797768+0000","last_active":"2024-04-02T04:49:10.926531+0000","last_peered":"2024-04-02T04:49:10.926531+0000","last_clean":"2024-04-02T04:49:10.926531+0000","last_became_active":"2024-04-02T04:43:58.314454+0000","last_became_peered":"2024-04-02T04:43:58.314454+0000","last_unstale":"2024-04-02T04:49:10.926531+0000","last_undegraded":"2024-04-02T04:49:10.926531+0000","last_fullsized":"2024-04-02T04:49:10.926531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.797658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.797658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:24.630032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":169,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.947855+0000","last_change":"2024-04-02T04:48:47.429314+0000","last_active":"2024-04-02T04:49:10.947855+0000","last_peered":"2024-04-02T04:49:10.947855+0000","last_clean":"2024-04-02T04:49:10.947855+0000","last_became_active":"2024-04-02T04:43:56.240439+0000","last_became_peered":"2024-04-02T04:43:56.240439+0000","last_unstale":"2024-04-02T04:49:10.947855+0000","last_undegraded":"2024-04-02T04:49:10.947855+0000","last_fullsized":"2024-04-02T04:49:10.947855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.133536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.924785+0000","last_change":"2024-04-02T04:48:42.658846+0000","last_active":"2024-04-02T04:49:10.924785+0000","last_peered":"2024-04-02T04:49:10.924785+0000","last_clean":"2024-04-02T04:49:10.924785+0000","last_became_active":"2024-04-02T04:43:54.207717+0000","last_became_peered":"2024-04-02T04:43:54.207717+0000","last_unstale":"2024-04-02T04:49:10.924785+0000","last_undegraded":"2024-04-02T04:49:10.924785+0000","last_fullsized":"2024-04-02T04:49:10.924785+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.658688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.658688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:45.858271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.929272+0000","last_change":"2024-04-02T04:48:45.911666+0000","last_active":"2024-04-02T04:49:10.929272+0000","last_peered":"2024-04-02T04:49:10.929272+0000","last_clean":"2024-04-02T04:49:10.929272+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:49:10.929272+0000","last_undegraded":"2024-04-02T04:49:10.929272+0000","last_fullsized":"2024-04-02T04:49:10.929272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.911425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.911425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:02.624974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.926781+0000","last_change":"2024-04-02T04:49:10.926781+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.926781+0000","last_undegraded":"2024-04-02T04:49:10.926781+0000","last_fullsized":"2024-04-02T04:49:10.926781+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.926591+0000","last_change":"2024-04-02T04:49:10.926591+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.926591+0000","last_undegraded":"2024-04-02T04:49:10.926591+0000","last_fullsized":"2024-04-02T04:49:10.926591+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921004+0000","last_change":"2024-04-02T04:47:35.260903+0000","last_active":"2024-04-02T04:49:10.921004+0000","last_peered":"2024-04-02T04:49:10.921004+0000","last_clean":"2024-04-02T04:49:10.921004+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:49:10.921004+0000","last_undegraded":"2024-04-02T04:49:10.921004+0000","last_fullsized":"2024-04-02T04:49:10.921004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:35.260653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:35.260653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":171,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930374+0000","last_change":"2024-04-02T04:49:02.785412+0000","last_active":"2024-04-02T04:49:10.930374+0000","last_peered":"2024-04-02T04:49:10.930374+0000","last_clean":"2024-04-02T04:49:10.930374+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:49:10.930374+0000","last_undegraded":"2024-04-02T04:49:10.930374+0000","last_fullsized":"2024-04-02T04:49:10.930374+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:49:02.785301+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.785301+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:13.055239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":156,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.929634+0000","last_change":"2024-04-02T04:49:05.021205+0000","last_active":"2024-04-02T04:49:10.929634+0000","last_peered":"2024-04-02T04:49:10.929634+0000","last_clean":"2024-04-02T04:49:10.929634+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:49:10.929634+0000","last_undegraded":"2024-04-02T04:49:10.929634+0000","last_fullsized":"2024-04-02T04:49:10.929634+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.020988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.020988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:23.382709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":129,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919552+0000","last_change":"2024-04-02T04:47:36.243027+0000","last_active":"2024-04-02T04:49:10.919552+0000","last_peered":"2024-04-02T04:49:10.919552+0000","last_clean":"2024-04-02T04:49:10.919552+0000","last_became_active":"2024-04-02T04:44:00.348011+0000","last_became_peered":"2024-04-02T04:44:00.348011+0000","last_unstale":"2024-04-02T04:49:10.919552+0000","last_undegraded":"2024-04-02T04:49:10.919552+0000","last_fullsized":"2024-04-02T04:49:10.919552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.923512+0000","last_change":"2024-04-02T04:49:10.923512+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.923512+0000","last_undegraded":"2024-04-02T04:49:10.923512+0000","last_fullsized":"2024-04-02T04:49:10.923512+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.925755+0000","last_change":"2024-04-02T04:49:10.925755+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.925755+0000","last_undegraded":"2024-04-02T04:49:10.925755+0000","last_fullsized":"2024-04-02T04:49:10.925755+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.943836+0000","last_change":"2024-04-02T04:49:09.719360+0000","last_active":"2024-04-02T04:49:10.943836+0000","last_peered":"2024-04-02T04:49:10.943836+0000","last_clean":"2024-04-02T04:49:10.943836+0000","last_became_active":"2024-04-02T04:43:58.313173+0000","last_became_peered":"2024-04-02T04:43:58.313173+0000","last_unstale":"2024-04-02T04:49:10.943836+0000","last_undegraded":"2024-04-02T04:49:10.943836+0000","last_fullsized":"2024-04-02T04:49:10.943836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.719146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.719146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:26.727727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":172,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.946520+0000","last_change":"2024-04-02T04:49:00.356489+0000","last_active":"2024-04-02T04:49:10.946520+0000","last_peered":"2024-04-02T04:49:10.946520+0000","last_clean":"2024-04-02T04:49:10.946520+0000","last_became_active":"2024-04-02T04:43:56.247902+0000","last_became_peered":"2024-04-02T04:43:56.247902+0000","last_unstale":"2024-04-02T04:49:10.946520+0000","last_undegraded":"2024-04-02T04:49:10.946520+0000","last_fullsized":"2024-04-02T04:49:10.946520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:00.356122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.356122+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:03.322715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.924847+0000","last_change":"2024-04-02T04:48:49.568810+0000","last_active":"2024-04-02T04:49:10.924847+0000","last_peered":"2024-04-02T04:49:10.924847+0000","last_clean":"2024-04-02T04:49:10.924847+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:49:10.924847+0000","last_undegraded":"2024-04-02T04:49:10.924847+0000","last_fullsized":"2024-04-02T04:49:10.924847+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.568676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.568676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:09.601856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":144,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.939491+0000","last_change":"2024-04-02T04:49:02.438070+0000","last_active":"2024-04-02T04:49:10.939491+0000","last_peered":"2024-04-02T04:49:10.939491+0000","last_clean":"2024-04-02T04:49:10.939491+0000","last_became_active":"2024-04-02T04:44:00.339664+0000","last_became_peered":"2024-04-02T04:44:00.339664+0000","last_unstale":"2024-04-02T04:49:10.939491+0000","last_undegraded":"2024-04-02T04:49:10.939491+0000","last_fullsized":"2024-04-02T04:49:10.939491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.437949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.437949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:11.358616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.922291+0000","last_change":"2024-04-02T04:49:10.922291+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.922291+0000","last_undegraded":"2024-04-02T04:49:10.922291+0000","last_fullsized":"2024-04-02T04:49:10.922291+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":197,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.944553+0000","last_change":"2024-04-02T04:49:10.763649+0000","last_active":"2024-04-02T04:49:10.944553+0000","last_peered":"2024-04-02T04:49:10.944553+0000","last_clean":"2024-04-02T04:49:10.944553+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:49:10.944553+0000","last_undegraded":"2024-04-02T04:49:10.944553+0000","last_fullsized":"2024-04-02T04:49:10.944553+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:49:10.763404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.763404+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.679681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":148,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.942947+0000","last_change":"2024-04-02T04:48:45.439677+0000","last_active":"2024-04-02T04:49:10.942947+0000","last_peered":"2024-04-02T04:49:10.942947+0000","last_clean":"2024-04-02T04:49:10.942947+0000","last_became_active":"2024-04-02T04:43:54.212202+0000","last_became_peered":"2024-04-02T04:43:54.212202+0000","last_unstale":"2024-04-02T04:49:10.942947+0000","last_undegraded":"2024-04-02T04:49:10.942947+0000","last_fullsized":"2024-04-02T04:49:10.942947+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.439457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.439457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.918301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":215,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920578+0000","last_change":"2024-04-02T04:47:56.704599+0000","last_active":"2024-04-02T04:49:10.920578+0000","last_peered":"2024-04-02T04:49:10.920578+0000","last_clean":"2024-04-02T04:49:10.920578+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:49:10.920578+0000","last_undegraded":"2024-04-02T04:49:10.920578+0000","last_fullsized":"2024-04-02T04:49:10.920578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:47:56.704379+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.874702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":143,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.925043+0000","last_change":"2024-04-02T04:48:59.823688+0000","last_active":"2024-04-02T04:49:10.925043+0000","last_peered":"2024-04-02T04:49:10.925043+0000","last_clean":"2024-04-02T04:49:10.925043+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:49:10.925043+0000","last_undegraded":"2024-04-02T04:49:10.925043+0000","last_fullsized":"2024-04-02T04:49:10.925043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.823603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.823603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:11.719130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.923633+0000","last_change":"2024-04-02T04:49:10.923633+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.923633+0000","last_undegraded":"2024-04-02T04:49:10.923633+0000","last_fullsized":"2024-04-02T04:49:10.923633+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.928496+0000","last_change":"2024-04-02T04:49:10.928496+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.928496+0000","last_undegraded":"2024-04-02T04:49:10.928496+0000","last_fullsized":"2024-04-02T04:49:10.928496+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919008+0000","last_change":"2024-04-02T04:48:52.546612+0000","last_active":"2024-04-02T04:49:10.919008+0000","last_peered":"2024-04-02T04:49:10.919008+0000","last_clean":"2024-04-02T04:49:10.919008+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:49:10.919008+0000","last_undegraded":"2024-04-02T04:49:10.919008+0000","last_fullsized":"2024-04-02T04:49:10.919008+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.546510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.546510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:16.969630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":178,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923334+0000","last_change":"2024-04-02T04:47:41.576889+0000","last_active":"2024-04-02T04:49:10.923334+0000","last_peered":"2024-04-02T04:49:10.923334+0000","last_clean":"2024-04-02T04:49:10.923334+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:49:10.923334+0000","last_undegraded":"2024-04-02T04:49:10.923334+0000","last_fullsized":"2024-04-02T04:49:10.923334+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:47:41.576640+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:47:41.576640+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":156,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:11.071304+0000","last_change":"2024-04-02T04:49:11.071343+0000","last_active":"2024-04-02T04:49:11.071304+0000","last_peered":"2024-04-02T04:49:11.071304+0000","last_clean":"2024-04-02T04:49:11.071304+0000","last_became_active":"2024-04-02T04:43:54.237698+0000","last_became_peered":"2024-04-02T04:43:54.237698+0000","last_unstale":"2024-04-02T04:49:11.071304+0000","last_undegraded":"2024-04-02T04:49:11.071304+0000","last_fullsized":"2024-04-02T04:49:11.071304+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.071225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.408261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922347+0000","last_change":"2024-04-02T04:48:40.604446+0000","last_active":"2024-04-02T04:49:10.922347+0000","last_peered":"2024-04-02T04:49:10.922347+0000","last_clean":"2024-04-02T04:49:10.922347+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:49:10.922347+0000","last_undegraded":"2024-04-02T04:49:10.922347+0000","last_fullsized":"2024-04-02T04:49:10.922347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.604291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.604291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.578827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.922985+0000","last_change":"2024-04-02T04:49:10.922985+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.922985+0000","last_undegraded":"2024-04-02T04:49:10.922985+0000","last_fullsized":"2024-04-02T04:49:10.922985+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.927266+0000","last_change":"2024-04-02T04:49:10.927266+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.927266+0000","last_undegraded":"2024-04-02T04:49:10.927266+0000","last_fullsized":"2024-04-02T04:49:10.927266+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":203,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919045+0000","last_change":"2024-04-02T04:47:26.453777+0000","last_active":"2024-04-02T04:49:10.919045+0000","last_peered":"2024-04-02T04:49:10.919045+0000","last_clean":"2024-04-02T04:49:10.919045+0000","last_became_active":"2024-04-02T04:43:56.230502+0000","last_became_peered":"2024-04-02T04:43:56.230502+0000","last_unstale":"2024-04-02T04:49:10.919045+0000","last_undegraded":"2024-04-02T04:49:10.919045+0000","last_fullsized":"2024-04-02T04:49:10.919045+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:47:26.453340+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:47:26.453340+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":154,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.928814+0000","last_change":"2024-04-02T04:47:44.112479+0000","last_active":"2024-04-02T04:49:10.928814+0000","last_peered":"2024-04-02T04:49:10.928814+0000","last_clean":"2024-04-02T04:49:10.928814+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:49:10.928814+0000","last_undegraded":"2024-04-02T04:49:10.928814+0000","last_fullsized":"2024-04-02T04:49:10.928814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:44.112270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:44.112270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":80,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.925149+0000","last_change":"2024-04-02T04:47:55.855659+0000","last_active":"2024-04-02T04:49:10.925149+0000","last_peered":"2024-04-02T04:49:10.925149+0000","last_clean":"2024-04-02T04:49:10.925149+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:49:10.925149+0000","last_undegraded":"2024-04-02T04:49:10.925149+0000","last_fullsized":"2024-04-02T04:49:10.925149+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:55.855510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:55.855510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:18.631275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":143,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.934777+0000","last_change":"2024-04-02T04:49:00.820022+0000","last_active":"2024-04-02T04:49:10.934777+0000","last_peered":"2024-04-02T04:49:10.934777+0000","last_clean":"2024-04-02T04:49:10.934777+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:49:10.934777+0000","last_undegraded":"2024-04-02T04:49:10.934777+0000","last_fullsized":"2024-04-02T04:49:10.934777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.819876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.819876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:18.330461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.928802+0000","last_change":"2024-04-02T04:49:10.928802+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.928802+0000","last_undegraded":"2024-04-02T04:49:10.928802+0000","last_fullsized":"2024-04-02T04:49:10.928802+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.917336+0000","last_change":"2024-04-02T04:49:10.917336+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.917336+0000","last_undegraded":"2024-04-02T04:49:10.917336+0000","last_fullsized":"2024-04-02T04:49:10.917336+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":191,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920968+0000","last_change":"2024-04-02T04:48:49.535107+0000","last_active":"2024-04-02T04:49:10.920968+0000","last_peered":"2024-04-02T04:49:10.920968+0000","last_clean":"2024-04-02T04:49:10.920968+0000","last_became_active":"2024-04-02T04:43:56.224929+0000","last_became_peered":"2024-04-02T04:43:56.224929+0000","last_unstale":"2024-04-02T04:49:10.920968+0000","last_undegraded":"2024-04-02T04:49:10.920968+0000","last_fullsized":"2024-04-02T04:49:10.920968+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:48:49.534769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.534769+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:59.394972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":156,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:11.344711+0000","last_change":"2024-04-02T04:49:11.344773+0000","last_active":"2024-04-02T04:49:11.344711+0000","last_peered":"2024-04-02T04:49:11.344711+0000","last_clean":"2024-04-02T04:49:11.344711+0000","last_became_active":"2024-04-02T04:43:54.230296+0000","last_became_peered":"2024-04-02T04:43:54.230296+0000","last_unstale":"2024-04-02T04:49:11.344711+0000","last_undegraded":"2024-04-02T04:49:11.344711+0000","last_fullsized":"2024-04-02T04:49:11.344711+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.344546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:19.084518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":214,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921750+0000","last_change":"2024-04-02T04:49:04.648126+0000","last_active":"2024-04-02T04:49:10.921750+0000","last_peered":"2024-04-02T04:49:10.921750+0000","last_clean":"2024-04-02T04:49:10.921750+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:49:10.921750+0000","last_undegraded":"2024-04-02T04:49:10.921750+0000","last_fullsized":"2024-04-02T04:49:10.921750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:49:04.647949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.647949+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:13.909196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921864+0000","last_change":"2024-04-02T04:48:45.485574+0000","last_active":"2024-04-02T04:49:10.921864+0000","last_peered":"2024-04-02T04:49:10.921864+0000","last_clean":"2024-04-02T04:49:10.921864+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:49:10.921864+0000","last_undegraded":"2024-04-02T04:49:10.921864+0000","last_fullsized":"2024-04-02T04:49:10.921864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.485441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.485441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:01.318866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.929724+0000","last_change":"2024-04-02T04:49:10.929724+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.929724+0000","last_undegraded":"2024-04-02T04:49:10.929724+0000","last_fullsized":"2024-04-02T04:49:10.929724+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.924406+0000","last_change":"2024-04-02T04:49:10.924406+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.924406+0000","last_undegraded":"2024-04-02T04:49:10.924406+0000","last_fullsized":"2024-04-02T04:49:10.924406+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":132,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921415+0000","last_change":"2024-04-02T04:47:25.814952+0000","last_active":"2024-04-02T04:49:10.921415+0000","last_peered":"2024-04-02T04:49:10.921415+0000","last_clean":"2024-04-02T04:49:10.921415+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:49:10.921415+0000","last_undegraded":"2024-04-02T04:49:10.921415+0000","last_fullsized":"2024-04-02T04:49:10.921415+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:25.814722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:25.814722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":163,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921054+0000","last_change":"2024-04-02T04:48:43.043307+0000","last_active":"2024-04-02T04:49:10.921054+0000","last_peered":"2024-04-02T04:49:10.921054+0000","last_clean":"2024-04-02T04:49:10.921054+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:49:10.921054+0000","last_undegraded":"2024-04-02T04:49:10.921054+0000","last_fullsized":"2024-04-02T04:49:10.921054+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:48:43.043155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.043155+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:09.690835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.947689+0000","last_change":"2024-04-02T04:49:08.690467+0000","last_active":"2024-04-02T04:49:10.947689+0000","last_peered":"2024-04-02T04:49:10.947689+0000","last_clean":"2024-04-02T04:49:10.947689+0000","last_became_active":"2024-04-02T04:43:54.210379+0000","last_became_peered":"2024-04-02T04:43:54.210379+0000","last_unstale":"2024-04-02T04:49:10.947689+0000","last_undegraded":"2024-04-02T04:49:10.947689+0000","last_fullsized":"2024-04-02T04:49:10.947689+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.690362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.690362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:08.988678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":446,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919253+0000","last_change":"2024-04-02T04:48:53.454617+0000","last_active":"2024-04-02T04:49:10.919253+0000","last_peered":"2024-04-02T04:49:10.919253+0000","last_clean":"2024-04-02T04:49:10.919253+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:49:10.919253+0000","last_undegraded":"2024-04-02T04:49:10.919253+0000","last_fullsized":"2024-04-02T04:49:10.919253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:48:53.454462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:53.454462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:05.524795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.942548+0000","last_change":"2024-04-02T04:47:49.224735+0000","last_active":"2024-04-02T04:49:10.942548+0000","last_peered":"2024-04-02T04:49:10.942548+0000","last_clean":"2024-04-02T04:49:10.942548+0000","last_became_active":"2024-04-02T04:44:00.352016+0000","last_became_peered":"2024-04-02T04:44:00.352016+0000","last_unstale":"2024-04-02T04:49:10.942548+0000","last_undegraded":"2024-04-02T04:49:10.942548+0000","last_fullsized":"2024-04-02T04:49:10.942548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.916792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.926021+0000","last_change":"2024-04-02T04:49:10.926021+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.926021+0000","last_undegraded":"2024-04-02T04:49:10.926021+0000","last_fullsized":"2024-04-02T04:49:10.926021+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"105'12","reported_seq":224,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919123+0000","last_change":"2024-04-02T04:48:38.477973+0000","last_active":"2024-04-02T04:49:10.919123+0000","last_peered":"2024-04-02T04:49:10.919123+0000","last_clean":"2024-04-02T04:49:10.919123+0000","last_became_active":"2024-04-02T04:43:58.322032+0000","last_became_peered":"2024-04-02T04:43:58.322032+0000","last_unstale":"2024-04-02T04:49:10.919123+0000","last_undegraded":"2024-04-02T04:49:10.919123+0000","last_fullsized":"2024-04-02T04:49:10.919123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:48:38.477825+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.477825+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:04.284920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":184,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922593+0000","last_change":"2024-04-02T04:49:07.375115+0000","last_active":"2024-04-02T04:49:10.922593+0000","last_peered":"2024-04-02T04:49:10.922593+0000","last_clean":"2024-04-02T04:49:10.922593+0000","last_became_active":"2024-04-02T04:43:56.242810+0000","last_became_peered":"2024-04-02T04:43:56.242810+0000","last_unstale":"2024-04-02T04:49:10.922593+0000","last_undegraded":"2024-04-02T04:49:10.922593+0000","last_fullsized":"2024-04-02T04:49:10.922593+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:49:07.374876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.374876+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.264584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":150,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.929687+0000","last_change":"2024-04-02T04:47:58.111681+0000","last_active":"2024-04-02T04:49:10.929687+0000","last_peered":"2024-04-02T04:49:10.929687+0000","last_clean":"2024-04-02T04:49:10.929687+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:49:10.929687+0000","last_undegraded":"2024-04-02T04:49:10.929687+0000","last_fullsized":"2024-04-02T04:49:10.929687+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.111561+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918656+0000","last_change":"2024-04-02T04:48:44.556005+0000","last_active":"2024-04-02T04:49:10.918656+0000","last_peered":"2024-04-02T04:49:10.918656+0000","last_clean":"2024-04-02T04:49:10.918656+0000","last_became_active":"2024-04-02T04:43:58.316126+0000","last_became_peered":"2024-04-02T04:43:58.316126+0000","last_unstale":"2024-04-02T04:49:10.918656+0000","last_undegraded":"2024-04-02T04:49:10.918656+0000","last_fullsized":"2024-04-02T04:49:10.918656+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.555854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.555854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:10.873849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923428+0000","last_change":"2024-04-02T04:49:06.652181+0000","last_active":"2024-04-02T04:49:10.923428+0000","last_peered":"2024-04-02T04:49:10.923428+0000","last_clean":"2024-04-02T04:49:10.923428+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:49:10.923428+0000","last_undegraded":"2024-04-02T04:49:10.923428+0000","last_fullsized":"2024-04-02T04:49:10.923428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.651924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.651924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:09.519161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"121'12","reported_seq":224,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920867+0000","last_change":"2024-04-02T04:49:08.963853+0000","last_active":"2024-04-02T04:49:10.920867+0000","last_peered":"2024-04-02T04:49:10.920867+0000","last_clean":"2024-04-02T04:49:10.920867+0000","last_became_active":"2024-04-02T04:43:58.317578+0000","last_became_peered":"2024-04-02T04:43:58.317578+0000","last_unstale":"2024-04-02T04:49:10.920867+0000","last_undegraded":"2024-04-02T04:49:10.920867+0000","last_fullsized":"2024-04-02T04:49:10.920867+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.856812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":281,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.928037+0000","last_change":"2024-04-02T04:47:42.173169+0000","last_active":"2024-04-02T04:49:10.928037+0000","last_peered":"2024-04-02T04:49:10.928037+0000","last_clean":"2024-04-02T04:49:10.928037+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:49:10.928037+0000","last_undegraded":"2024-04-02T04:49:10.928037+0000","last_fullsized":"2024-04-02T04:49:10.928037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:47:42.172894+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.172894+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":148,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.946007+0000","last_change":"2024-04-02T04:48:52.389879+0000","last_active":"2024-04-02T04:49:10.946007+0000","last_peered":"2024-04-02T04:49:10.946007+0000","last_clean":"2024-04-02T04:49:10.946007+0000","last_became_active":"2024-04-02T04:43:54.222964+0000","last_became_peered":"2024-04-02T04:43:54.222964+0000","last_unstale":"2024-04-02T04:49:10.946007+0000","last_undegraded":"2024-04-02T04:49:10.946007+0000","last_fullsized":"2024-04-02T04:49:10.946007+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:53.347386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":150,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.944011+0000","last_change":"2024-04-02T04:48:00.879991+0000","last_active":"2024-04-02T04:49:10.944011+0000","last_peered":"2024-04-02T04:49:10.944011+0000","last_clean":"2024-04-02T04:49:10.944011+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:49:10.944011+0000","last_undegraded":"2024-04-02T04:49:10.944011+0000","last_fullsized":"2024-04-02T04:49:10.944011+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:48:00.879724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.879724+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:13.988600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930415+0000","last_change":"2024-04-02T04:48:04.001245+0000","last_active":"2024-04-02T04:49:10.930415+0000","last_peered":"2024-04-02T04:49:10.930415+0000","last_clean":"2024-04-02T04:49:10.930415+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:49:10.930415+0000","last_undegraded":"2024-04-02T04:49:10.930415+0000","last_fullsized":"2024-04-02T04:49:10.930415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:48:04.001090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:30.589675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.921695+0000","last_change":"2024-04-02T04:49:10.921695+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.921695+0000","last_undegraded":"2024-04-02T04:49:10.921695+0000","last_fullsized":"2024-04-02T04:49:10.921695+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.930133+0000","last_change":"2024-04-02T04:49:10.930133+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.930133+0000","last_undegraded":"2024-04-02T04:49:10.930133+0000","last_fullsized":"2024-04-02T04:49:10.930133+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920777+0000","last_change":"2024-04-02T04:48:49.033259+0000","last_active":"2024-04-02T04:49:10.920777+0000","last_peered":"2024-04-02T04:49:10.920777+0000","last_clean":"2024-04-02T04:49:10.920777+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:49:10.920777+0000","last_undegraded":"2024-04-02T04:49:10.920777+0000","last_fullsized":"2024-04-02T04:49:10.920777+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.033101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.033101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:52.865505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.939880+0000","last_change":"2024-04-02T04:48:49.441490+0000","last_active":"2024-04-02T04:49:10.939880+0000","last_peered":"2024-04-02T04:49:10.939880+0000","last_clean":"2024-04-02T04:49:10.939880+0000","last_became_active":"2024-04-02T04:43:54.231591+0000","last_became_peered":"2024-04-02T04:43:54.231591+0000","last_unstale":"2024-04-02T04:49:10.939880+0000","last_undegraded":"2024-04-02T04:49:10.939880+0000","last_fullsized":"2024-04-02T04:49:10.939880+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.441320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.441320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:54.336788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":178,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919770+0000","last_change":"2024-04-02T04:48:47.093951+0000","last_active":"2024-04-02T04:49:10.919770+0000","last_peered":"2024-04-02T04:49:10.919770+0000","last_clean":"2024-04-02T04:49:10.919770+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:49:10.919770+0000","last_undegraded":"2024-04-02T04:49:10.919770+0000","last_fullsized":"2024-04-02T04:49:10.919770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:48:47.093759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.093759+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:56.401562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.927705+0000","last_change":"2024-04-02T04:48:01.028154+0000","last_active":"2024-04-02T04:49:10.927705+0000","last_peered":"2024-04-02T04:49:10.927705+0000","last_clean":"2024-04-02T04:49:10.927705+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:49:10.927705+0000","last_undegraded":"2024-04-02T04:49:10.927705+0000","last_fullsized":"2024-04-02T04:49:10.927705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:01.027858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:01.027858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.938148+0000","last_change":"2024-04-02T04:49:10.938148+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.938148+0000","last_undegraded":"2024-04-02T04:49:10.938148+0000","last_fullsized":"2024-04-02T04:49:10.938148+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.927810+0000","last_change":"2024-04-02T04:49:10.927810+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.927810+0000","last_undegraded":"2024-04-02T04:49:10.927810+0000","last_fullsized":"2024-04-02T04:49:10.927810+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.945343+0000","last_change":"2024-04-02T04:48:45.569264+0000","last_active":"2024-04-02T04:49:10.945343+0000","last_peered":"2024-04-02T04:49:10.945343+0000","last_clean":"2024-04-02T04:49:10.945343+0000","last_became_active":"2024-04-02T04:43:58.314103+0000","last_became_peered":"2024-04-02T04:43:58.314103+0000","last_unstale":"2024-04-02T04:49:10.945343+0000","last_undegraded":"2024-04-02T04:49:10.945343+0000","last_fullsized":"2024-04-02T04:49:10.945343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.569058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.569058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.165593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":413,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.940086+0000","last_change":"2024-04-02T04:49:07.344739+0000","last_active":"2024-04-02T04:49:10.940086+0000","last_peered":"2024-04-02T04:49:10.940086+0000","last_clean":"2024-04-02T04:49:10.940086+0000","last_became_active":"2024-04-02T04:43:16.942773+0000","last_became_peered":"2024-04-02T04:43:16.942773+0000","last_unstale":"2024-04-02T04:49:10.940086+0000","last_undegraded":"2024-04-02T04:49:10.940086+0000","last_fullsized":"2024-04-02T04:49:10.940086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:22.789974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.928358+0000","last_change":"2024-04-02T04:48:39.903784+0000","last_active":"2024-04-02T04:49:10.928358+0000","last_peered":"2024-04-02T04:49:10.928358+0000","last_clean":"2024-04-02T04:49:10.928358+0000","last_became_active":"2024-04-02T04:43:54.211128+0000","last_became_peered":"2024-04-02T04:43:54.211128+0000","last_unstale":"2024-04-02T04:49:10.928358+0000","last_undegraded":"2024-04-02T04:49:10.928358+0000","last_fullsized":"2024-04-02T04:49:10.928358+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:39.903683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.903683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:53.451472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":172,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922860+0000","last_change":"2024-04-02T04:49:04.488216+0000","last_active":"2024-04-02T04:49:10.922860+0000","last_peered":"2024-04-02T04:49:10.922860+0000","last_clean":"2024-04-02T04:49:10.922860+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:49:10.922860+0000","last_undegraded":"2024-04-02T04:49:10.922860+0000","last_fullsized":"2024-04-02T04:49:10.922860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:49:04.488030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.488030+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.470177+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.935889+0000","last_change":"2024-04-02T04:47:58.186088+0000","last_active":"2024-04-02T04:49:10.935889+0000","last_peered":"2024-04-02T04:49:10.935889+0000","last_clean":"2024-04-02T04:49:10.935889+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:49:10.935889+0000","last_undegraded":"2024-04-02T04:49:10.935889+0000","last_fullsized":"2024-04-02T04:49:10.935889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:58.185868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:58.185868+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.185868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:23.742931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.922301+0000","last_change":"2024-04-02T04:49:10.922301+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.922301+0000","last_undegraded":"2024-04-02T04:49:10.922301+0000","last_fullsized":"2024-04-02T04:49:10.922301+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930708+0000","last_change":"2024-04-02T04:48:47.126491+0000","last_active":"2024-04-02T04:49:10.930708+0000","last_peered":"2024-04-02T04:49:10.930708+0000","last_clean":"2024-04-02T04:49:10.930708+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:49:10.930708+0000","last_undegraded":"2024-04-02T04:49:10.930708+0000","last_fullsized":"2024-04-02T04:49:10.930708+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:47.126364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.126364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:00.049150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.924390+0000","last_change":"2024-04-02T04:48:55.735754+0000","last_active":"2024-04-02T04:49:10.924390+0000","last_peered":"2024-04-02T04:49:10.924390+0000","last_clean":"2024-04-02T04:49:10.924390+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:49:10.924390+0000","last_undegraded":"2024-04-02T04:49:10.924390+0000","last_fullsized":"2024-04-02T04:49:10.924390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:55.735631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:48:55.735631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:12.433917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":200,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930025+0000","last_change":"2024-04-02T04:49:01.949693+0000","last_active":"2024-04-02T04:49:10.930025+0000","last_peered":"2024-04-02T04:49:10.930025+0000","last_clean":"2024-04-02T04:49:10.930025+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:49:10.930025+0000","last_undegraded":"2024-04-02T04:49:10.930025+0000","last_fullsized":"2024-04-02T04:49:10.930025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:49:01.949391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.949391+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:10.169019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.945554+0000","last_change":"2024-04-02T04:47:53.271021+0000","last_active":"2024-04-02T04:49:10.945554+0000","last_peered":"2024-04-02T04:49:10.945554+0000","last_clean":"2024-04-02T04:49:10.945554+0000","last_became_active":"2024-04-02T04:44:00.353848+0000","last_became_peered":"2024-04-02T04:44:00.353848+0000","last_unstale":"2024-04-02T04:49:10.945554+0000","last_undegraded":"2024-04-02T04:49:10.945554+0000","last_fullsized":"2024-04-02T04:49:10.945554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.902292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.921486+0000","last_change":"2024-04-02T04:49:10.921486+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.921486+0000","last_undegraded":"2024-04-02T04:49:10.921486+0000","last_fullsized":"2024-04-02T04:49:10.921486+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.920064+0000","last_change":"2024-04-02T04:49:10.920064+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.920064+0000","last_undegraded":"2024-04-02T04:49:10.920064+0000","last_fullsized":"2024-04-02T04:49:10.920064+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.927789+0000","last_change":"2024-04-02T04:47:47.106602+0000","last_active":"2024-04-02T04:49:10.927789+0000","last_peered":"2024-04-02T04:49:10.927789+0000","last_clean":"2024-04-02T04:49:10.927789+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:49:10.927789+0000","last_undegraded":"2024-04-02T04:49:10.927789+0000","last_fullsized":"2024-04-02T04:49:10.927789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:47:47.106497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":192,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920400+0000","last_change":"2024-04-02T04:47:50.758701+0000","last_active":"2024-04-02T04:49:10.920400+0000","last_peered":"2024-04-02T04:49:10.920400+0000","last_clean":"2024-04-02T04:49:10.920400+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:49:10.920400+0000","last_undegraded":"2024-04-02T04:49:10.920400+0000","last_fullsized":"2024-04-02T04:49:10.920400+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:47:50.758312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:50.758312+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.928694+0000","last_change":"2024-04-02T04:48:44.911734+0000","last_active":"2024-04-02T04:49:10.928694+0000","last_peered":"2024-04-02T04:49:10.928694+0000","last_clean":"2024-04-02T04:49:10.928694+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:49:10.928694+0000","last_undegraded":"2024-04-02T04:49:10.928694+0000","last_fullsized":"2024-04-02T04:49:10.928694+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.911605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.911605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:04.418347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":139,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922074+0000","last_change":"2024-04-02T04:48:49.483259+0000","last_active":"2024-04-02T04:49:10.922074+0000","last_peered":"2024-04-02T04:49:10.922074+0000","last_clean":"2024-04-02T04:49:10.922074+0000","last_became_active":"2024-04-02T04:44:00.360775+0000","last_became_peered":"2024-04-02T04:44:00.360775+0000","last_unstale":"2024-04-02T04:49:10.922074+0000","last_undegraded":"2024-04-02T04:49:10.922074+0000","last_fullsized":"2024-04-02T04:49:10.922074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:48:49.483159+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.483159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:58.239326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.930553+0000","last_change":"2024-04-02T04:49:10.930553+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.930553+0000","last_undegraded":"2024-04-02T04:49:10.930553+0000","last_fullsized":"2024-04-02T04:49:10.930553+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.926508+0000","last_change":"2024-04-02T04:49:10.926508+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.926508+0000","last_undegraded":"2024-04-02T04:49:10.926508+0000","last_fullsized":"2024-04-02T04:49:10.926508+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930732+0000","last_change":"2024-04-02T04:47:45.109386+0000","last_active":"2024-04-02T04:49:10.930732+0000","last_peered":"2024-04-02T04:49:10.930732+0000","last_clean":"2024-04-02T04:49:10.930732+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:49:10.930732+0000","last_undegraded":"2024-04-02T04:49:10.930732+0000","last_fullsized":"2024-04-02T04:49:10.930732+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.109170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.109170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":229,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930422+0000","last_change":"2024-04-02T04:47:39.221972+0000","last_active":"2024-04-02T04:49:10.930422+0000","last_peered":"2024-04-02T04:49:10.930422+0000","last_clean":"2024-04-02T04:49:10.930422+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:49:10.930422+0000","last_undegraded":"2024-04-02T04:49:10.930422+0000","last_fullsized":"2024-04-02T04:49:10.930422+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:47:39.221741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:39.221741+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":156,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.927874+0000","last_change":"2024-04-02T04:47:43.151655+0000","last_active":"2024-04-02T04:49:10.927874+0000","last_peered":"2024-04-02T04:49:10.927874+0000","last_clean":"2024-04-02T04:49:10.927874+0000","last_became_active":"2024-04-02T04:43:54.234878+0000","last_became_peered":"2024-04-02T04:43:54.234878+0000","last_unstale":"2024-04-02T04:49:10.927874+0000","last_undegraded":"2024-04-02T04:49:10.927874+0000","last_fullsized":"2024-04-02T04:49:10.927874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:47:43.151427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:43.151427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":145,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:11.991308+0000","last_change":"2024-04-02T04:49:11.991346+0000","last_active":"2024-04-02T04:49:11.991308+0000","last_peered":"2024-04-02T04:49:11.991308+0000","last_clean":"2024-04-02T04:49:11.991308+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:49:11.991308+0000","last_undegraded":"2024-04-02T04:49:11.991308+0000","last_fullsized":"2024-04-02T04:49:11.991308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.991232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.991232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.193266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.915950+0000","last_change":"2024-04-02T04:49:10.915950+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.915950+0000","last_undegraded":"2024-04-02T04:49:10.915950+0000","last_fullsized":"2024-04-02T04:49:10.915950+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.928093+0000","last_change":"2024-04-02T04:49:10.928093+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.928093+0000","last_undegraded":"2024-04-02T04:49:10.928093+0000","last_fullsized":"2024-04-02T04:49:10.928093+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":215,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918916+0000","last_change":"2024-04-02T04:47:42.831672+0000","last_active":"2024-04-02T04:49:10.918916+0000","last_peered":"2024-04-02T04:49:10.918916+0000","last_clean":"2024-04-02T04:49:10.918916+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:49:10.918916+0000","last_undegraded":"2024-04-02T04:49:10.918916+0000","last_fullsized":"2024-04-02T04:49:10.918916+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.831447+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":183,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.946444+0000","last_change":"2024-04-02T04:49:04.429559+0000","last_active":"2024-04-02T04:49:10.946444+0000","last_peered":"2024-04-02T04:49:10.946444+0000","last_clean":"2024-04-02T04:49:10.946444+0000","last_became_active":"2024-04-02T04:43:56.232648+0000","last_became_peered":"2024-04-02T04:43:56.232648+0000","last_unstale":"2024-04-02T04:49:10.946444+0000","last_undegraded":"2024-04-02T04:49:10.946444+0000","last_fullsized":"2024-04-02T04:49:10.946444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:17.151651+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":172,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921811+0000","last_change":"2024-04-02T04:48:39.608521+0000","last_active":"2024-04-02T04:49:10.921811+0000","last_peered":"2024-04-02T04:49:10.921811+0000","last_clean":"2024-04-02T04:49:10.921811+0000","last_became_active":"2024-04-02T04:43:54.211851+0000","last_became_peered":"2024-04-02T04:43:54.211851+0000","last_unstale":"2024-04-02T04:49:10.921811+0000","last_undegraded":"2024-04-02T04:49:10.921811+0000","last_fullsized":"2024-04-02T04:49:10.921811+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:48:39.608255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.608255+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:41.887929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923429+0000","last_change":"2024-04-02T04:49:01.488534+0000","last_active":"2024-04-02T04:49:10.923429+0000","last_peered":"2024-04-02T04:49:10.923429+0000","last_clean":"2024-04-02T04:49:10.923429+0000","last_became_active":"2024-04-02T04:44:00.356592+0000","last_became_peered":"2024-04-02T04:44:00.356592+0000","last_unstale":"2024-04-02T04:49:10.923429+0000","last_undegraded":"2024-04-02T04:49:10.923429+0000","last_fullsized":"2024-04-02T04:49:10.923429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.488397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.488397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:22.068285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.926952+0000","last_change":"2024-04-02T04:49:10.926952+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.926952+0000","last_undegraded":"2024-04-02T04:49:10.926952+0000","last_fullsized":"2024-04-02T04:49:10.926952+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":175,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.940809+0000","last_change":"2024-04-02T04:48:56.339921+0000","last_active":"2024-04-02T04:49:10.940809+0000","last_peered":"2024-04-02T04:49:10.940809+0000","last_clean":"2024-04-02T04:49:10.940809+0000","last_became_active":"2024-04-02T04:43:56.244092+0000","last_became_peered":"2024-04-02T04:43:56.244092+0000","last_unstale":"2024-04-02T04:49:10.940809+0000","last_undegraded":"2024-04-02T04:49:10.940809+0000","last_fullsized":"2024-04-02T04:49:10.940809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:09.362226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":155,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.924142+0000","last_change":"2024-04-02T04:49:10.653724+0000","last_active":"2024-04-02T04:49:10.924142+0000","last_peered":"2024-04-02T04:49:10.924142+0000","last_clean":"2024-04-02T04:49:10.924142+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:49:10.924142+0000","last_undegraded":"2024-04-02T04:49:10.924142+0000","last_fullsized":"2024-04-02T04:49:10.924142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.653529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.653529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:19.066671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920549+0000","last_change":"2024-04-02T04:49:00.103361+0000","last_active":"2024-04-02T04:49:10.920549+0000","last_peered":"2024-04-02T04:49:10.920549+0000","last_clean":"2024-04-02T04:49:10.920549+0000","last_became_active":"2024-04-02T04:43:58.317797+0000","last_became_peered":"2024-04-02T04:43:58.317797+0000","last_unstale":"2024-04-02T04:49:10.920549+0000","last_undegraded":"2024-04-02T04:49:10.920549+0000","last_fullsized":"2024-04-02T04:49:10.920549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.103147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.864579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.925155+0000","last_change":"2024-04-02T04:48:44.646833+0000","last_active":"2024-04-02T04:49:10.925155+0000","last_peered":"2024-04-02T04:49:10.925155+0000","last_clean":"2024-04-02T04:49:10.925155+0000","last_became_active":"2024-04-02T04:44:00.351593+0000","last_became_peered":"2024-04-02T04:44:00.351593+0000","last_unstale":"2024-04-02T04:49:10.925155+0000","last_undegraded":"2024-04-02T04:49:10.925155+0000","last_fullsized":"2024-04-02T04:49:10.925155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.646709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.646709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:52.485095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.936972+0000","last_change":"2024-04-02T04:49:10.936972+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.936972+0000","last_undegraded":"2024-04-02T04:49:10.936972+0000","last_fullsized":"2024-04-02T04:49:10.936972+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.916963+0000","last_change":"2024-04-02T04:49:10.916963+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.916963+0000","last_undegraded":"2024-04-02T04:49:10.916963+0000","last_fullsized":"2024-04-02T04:49:10.916963+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":151,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.941699+0000","last_change":"2024-04-02T04:48:41.679887+0000","last_active":"2024-04-02T04:49:10.941699+0000","last_peered":"2024-04-02T04:49:10.941699+0000","last_clean":"2024-04-02T04:49:10.941699+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:49:10.941699+0000","last_undegraded":"2024-04-02T04:49:10.941699+0000","last_fullsized":"2024-04-02T04:49:10.941699+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:48:41.679498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.679498+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:59.901463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":157,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.939821+0000","last_change":"2024-04-02T04:49:07.675897+0000","last_active":"2024-04-02T04:49:10.939821+0000","last_peered":"2024-04-02T04:49:10.939821+0000","last_clean":"2024-04-02T04:49:10.939821+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:49:10.939821+0000","last_undegraded":"2024-04-02T04:49:10.939821+0000","last_fullsized":"2024-04-02T04:49:10.939821+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.675606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.311827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922821+0000","last_change":"2024-04-02T04:49:02.485262+0000","last_active":"2024-04-02T04:49:10.922821+0000","last_peered":"2024-04-02T04:49:10.922821+0000","last_clean":"2024-04-02T04:49:10.922821+0000","last_became_active":"2024-04-02T04:45:53.587090+0000","last_became_peered":"2024-04-02T04:45:53.587090+0000","last_unstale":"2024-04-02T04:49:10.922821+0000","last_undegraded":"2024-04-02T04:49:10.922821+0000","last_fullsized":"2024-04-02T04:49:10.922821+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.485165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.485165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.349345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.945262+0000","last_change":"2024-04-02T04:48:59.304330+0000","last_active":"2024-04-02T04:49:10.945262+0000","last_peered":"2024-04-02T04:49:10.945262+0000","last_clean":"2024-04-02T04:49:10.945262+0000","last_became_active":"2024-04-02T04:44:00.351842+0000","last_became_peered":"2024-04-02T04:44:00.351842+0000","last_unstale":"2024-04-02T04:49:10.945262+0000","last_undegraded":"2024-04-02T04:49:10.945262+0000","last_fullsized":"2024-04-02T04:49:10.945262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.304153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.304153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:59.984699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.917157+0000","last_change":"2024-04-02T04:49:10.917157+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.917157+0000","last_undegraded":"2024-04-02T04:49:10.917157+0000","last_fullsized":"2024-04-02T04:49:10.917157+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":163,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.943217+0000","last_change":"2024-04-02T04:48:00.302558+0000","last_active":"2024-04-02T04:49:10.943217+0000","last_peered":"2024-04-02T04:49:10.943217+0000","last_clean":"2024-04-02T04:49:10.943217+0000","last_became_active":"2024-04-02T04:43:56.237320+0000","last_became_peered":"2024-04-02T04:43:56.237320+0000","last_unstale":"2024-04-02T04:49:10.943217+0000","last_undegraded":"2024-04-02T04:49:10.943217+0000","last_fullsized":"2024-04-02T04:49:10.943217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:26.019165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":147,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.927639+0000","last_change":"2024-04-02T04:48:40.899464+0000","last_active":"2024-04-02T04:49:10.927639+0000","last_peered":"2024-04-02T04:49:10.927639+0000","last_clean":"2024-04-02T04:49:10.927639+0000","last_became_active":"2024-04-02T04:43:54.212047+0000","last_became_peered":"2024-04-02T04:43:54.212047+0000","last_unstale":"2024-04-02T04:49:10.927639+0000","last_undegraded":"2024-04-02T04:49:10.927639+0000","last_fullsized":"2024-04-02T04:49:10.927639+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.899356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.899356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.480969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"121'12","reported_seq":216,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922136+0000","last_change":"2024-04-02T04:48:02.448348+0000","last_active":"2024-04-02T04:49:10.922136+0000","last_peered":"2024-04-02T04:49:10.922136+0000","last_clean":"2024-04-02T04:49:10.922136+0000","last_became_active":"2024-04-02T04:43:58.320754+0000","last_became_peered":"2024-04-02T04:43:58.320754+0000","last_unstale":"2024-04-02T04:49:10.922136+0000","last_undegraded":"2024-04-02T04:49:10.922136+0000","last_fullsized":"2024-04-02T04:49:10.922136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920676+0000","last_change":"2024-04-02T04:49:03.986508+0000","last_active":"2024-04-02T04:49:10.920676+0000","last_peered":"2024-04-02T04:49:10.920676+0000","last_clean":"2024-04-02T04:49:10.920676+0000","last_became_active":"2024-04-02T04:44:00.349058+0000","last_became_peered":"2024-04-02T04:44:00.349058+0000","last_unstale":"2024-04-02T04:49:10.920676+0000","last_undegraded":"2024-04-02T04:49:10.920676+0000","last_fullsized":"2024-04-02T04:49:10.920676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.986307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.986307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:05.197992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.935785+0000","last_change":"2024-04-02T04:49:10.935785+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.935785+0000","last_undegraded":"2024-04-02T04:49:10.935785+0000","last_fullsized":"2024-04-02T04:49:10.935785+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.919956+0000","last_change":"2024-04-02T04:49:10.919956+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.919956+0000","last_undegraded":"2024-04-02T04:49:10.919956+0000","last_fullsized":"2024-04-02T04:49:10.919956+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":161,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920100+0000","last_change":"2024-04-02T04:47:53.179435+0000","last_active":"2024-04-02T04:49:10.920100+0000","last_peered":"2024-04-02T04:49:10.920100+0000","last_clean":"2024-04-02T04:49:10.920100+0000","last_became_active":"2024-04-02T04:43:56.229079+0000","last_became_peered":"2024-04-02T04:43:56.229079+0000","last_unstale":"2024-04-02T04:49:10.920100+0000","last_undegraded":"2024-04-02T04:49:10.920100+0000","last_fullsized":"2024-04-02T04:49:10.920100+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:47:53.179316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.179316+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920478+0000","last_change":"2024-04-02T04:48:46.074077+0000","last_active":"2024-04-02T04:49:10.920478+0000","last_peered":"2024-04-02T04:49:10.920478+0000","last_clean":"2024-04-02T04:49:10.920478+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:49:10.920478+0000","last_undegraded":"2024-04-02T04:49:10.920478+0000","last_fullsized":"2024-04-02T04:49:10.920478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:46.073944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.073944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:00.740924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"85'11","reported_seq":216,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.934476+0000","last_change":"2024-04-02T04:48:41.932434+0000","last_active":"2024-04-02T04:49:10.934476+0000","last_peered":"2024-04-02T04:49:10.934476+0000","last_clean":"2024-04-02T04:49:10.934476+0000","last_became_active":"2024-04-02T04:43:58.313317+0000","last_became_peered":"2024-04-02T04:43:58.313317+0000","last_unstale":"2024-04-02T04:49:10.934476+0000","last_undegraded":"2024-04-02T04:49:10.934476+0000","last_fullsized":"2024-04-02T04:49:10.934476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:41.932321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.932321+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:07.046064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:11.708127+0000","last_change":"2024-04-02T04:49:11.708233+0000","last_active":"2024-04-02T04:49:11.708127+0000","last_peered":"2024-04-02T04:49:11.708127+0000","last_clean":"2024-04-02T04:49:11.708127+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:49:11.708127+0000","last_undegraded":"2024-04-02T04:49:11.708127+0000","last_fullsized":"2024-04-02T04:49:11.708127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.707910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.707910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:14.874080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.916833+0000","last_change":"2024-04-02T04:49:10.916833+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.916833+0000","last_undegraded":"2024-04-02T04:49:10.916833+0000","last_fullsized":"2024-04-02T04:49:10.916833+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":172,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.947066+0000","last_change":"2024-04-02T04:47:36.370736+0000","last_active":"2024-04-02T04:49:10.947066+0000","last_peered":"2024-04-02T04:49:10.947066+0000","last_clean":"2024-04-02T04:49:10.947066+0000","last_became_active":"2024-04-02T04:43:56.225222+0000","last_became_peered":"2024-04-02T04:43:56.225222+0000","last_unstale":"2024-04-02T04:49:10.947066+0000","last_undegraded":"2024-04-02T04:49:10.947066+0000","last_fullsized":"2024-04-02T04:49:10.947066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:36.370493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.370493+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.944429+0000","last_change":"2024-04-02T04:48:42.660369+0000","last_active":"2024-04-02T04:49:10.944429+0000","last_peered":"2024-04-02T04:49:10.944429+0000","last_clean":"2024-04-02T04:49:10.944429+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:49:10.944429+0000","last_undegraded":"2024-04-02T04:49:10.944429+0000","last_fullsized":"2024-04-02T04:49:10.944429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.660246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.660246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:02.575601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930452+0000","last_change":"2024-04-02T04:48:54.070585+0000","last_active":"2024-04-02T04:49:10.930452+0000","last_peered":"2024-04-02T04:49:10.930452+0000","last_clean":"2024-04-02T04:49:10.930452+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:49:10.930452+0000","last_undegraded":"2024-04-02T04:49:10.930452+0000","last_fullsized":"2024-04-02T04:49:10.930452+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:54.070457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:54.070457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:59.482523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918839+0000","last_change":"2024-04-02T04:48:40.066400+0000","last_active":"2024-04-02T04:49:10.918839+0000","last_peered":"2024-04-02T04:49:10.918839+0000","last_clean":"2024-04-02T04:49:10.918839+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:49:10.918839+0000","last_undegraded":"2024-04-02T04:49:10.918839+0000","last_fullsized":"2024-04-02T04:49:10.918839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.066265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.066265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:49.667647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.936162+0000","last_change":"2024-04-02T04:49:10.936162+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.936162+0000","last_undegraded":"2024-04-02T04:49:10.936162+0000","last_fullsized":"2024-04-02T04:49:10.936162+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.927472+0000","last_change":"2024-04-02T04:49:10.927472+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.927472+0000","last_undegraded":"2024-04-02T04:49:10.927472+0000","last_fullsized":"2024-04-02T04:49:10.927472+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":183,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919284+0000","last_change":"2024-04-02T04:48:00.691627+0000","last_active":"2024-04-02T04:49:10.919284+0000","last_peered":"2024-04-02T04:49:10.919284+0000","last_clean":"2024-04-02T04:49:10.919284+0000","last_became_active":"2024-04-02T04:43:56.240686+0000","last_became_peered":"2024-04-02T04:43:56.240686+0000","last_unstale":"2024-04-02T04:49:10.919284+0000","last_undegraded":"2024-04-02T04:49:10.919284+0000","last_fullsized":"2024-04-02T04:49:10.919284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:26.639529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":148,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.929489+0000","last_change":"2024-04-02T04:49:03.975681+0000","last_active":"2024-04-02T04:49:10.929489+0000","last_peered":"2024-04-02T04:49:10.929489+0000","last_clean":"2024-04-02T04:49:10.929489+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:49:10.929489+0000","last_undegraded":"2024-04-02T04:49:10.929489+0000","last_fullsized":"2024-04-02T04:49:10.929489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.975398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.975398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:16.751604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":148,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:12.685084+0000","last_change":"2024-04-02T04:49:12.685154+0000","last_active":"2024-04-02T04:49:12.685084+0000","last_peered":"2024-04-02T04:49:12.685084+0000","last_clean":"2024-04-02T04:49:12.685084+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:49:12.685084+0000","last_undegraded":"2024-04-02T04:49:12.685084+0000","last_fullsized":"2024-04-02T04:49:12.685084+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.684916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.684916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.981975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918834+0000","last_change":"2024-04-02T04:48:44.527799+0000","last_active":"2024-04-02T04:49:10.918834+0000","last_peered":"2024-04-02T04:49:10.918834+0000","last_clean":"2024-04-02T04:49:10.918834+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:49:10.918834+0000","last_undegraded":"2024-04-02T04:49:10.918834+0000","last_fullsized":"2024-04-02T04:49:10.918834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.527676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.527676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:00.396302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.923013+0000","last_change":"2024-04-02T04:49:10.923013+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.923013+0000","last_undegraded":"2024-04-02T04:49:10.923013+0000","last_fullsized":"2024-04-02T04:49:10.923013+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":166,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920951+0000","last_change":"2024-04-02T04:48:52.073913+0000","last_active":"2024-04-02T04:49:10.920951+0000","last_peered":"2024-04-02T04:49:10.920951+0000","last_clean":"2024-04-02T04:49:10.920951+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:49:10.920951+0000","last_undegraded":"2024-04-02T04:49:10.920951+0000","last_fullsized":"2024-04-02T04:49:10.920951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:48:52.073763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.073763+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.325220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":155,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.934516+0000","last_change":"2024-04-02T04:49:01.797690+0000","last_active":"2024-04-02T04:49:10.934516+0000","last_peered":"2024-04-02T04:49:10.934516+0000","last_clean":"2024-04-02T04:49:10.934516+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:49:10.934516+0000","last_undegraded":"2024-04-02T04:49:10.934516+0000","last_fullsized":"2024-04-02T04:49:10.934516+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.797580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.797580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:19.794386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918975+0000","last_change":"2024-04-02T04:47:19.398500+0000","last_active":"2024-04-02T04:49:10.918975+0000","last_peered":"2024-04-02T04:49:10.918975+0000","last_clean":"2024-04-02T04:49:10.918975+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:49:10.918975+0000","last_undegraded":"2024-04-02T04:49:10.918975+0000","last_fullsized":"2024-04-02T04:49:10.918975+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:19.398328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:19.398328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.940844+0000","last_change":"2024-04-02T04:49:10.372399+0000","last_active":"2024-04-02T04:49:10.940844+0000","last_peered":"2024-04-02T04:49:10.940844+0000","last_clean":"2024-04-02T04:49:10.940844+0000","last_became_active":"2024-04-02T04:44:00.344450+0000","last_became_peered":"2024-04-02T04:44:00.344450+0000","last_unstale":"2024-04-02T04:49:10.940844+0000","last_undegraded":"2024-04-02T04:49:10.940844+0000","last_fullsized":"2024-04-02T04:49:10.940844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.372299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.372299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.285338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.923627+0000","last_change":"2024-04-02T04:49:10.923627+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.923627+0000","last_undegraded":"2024-04-02T04:49:10.923627+0000","last_fullsized":"2024-04-02T04:49:10.923627+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.927021+0000","last_change":"2024-04-02T04:49:10.927021+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.927021+0000","last_undegraded":"2024-04-02T04:49:10.927021+0000","last_fullsized":"2024-04-02T04:49:10.927021+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":191,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.934761+0000","last_change":"2024-04-02T04:49:03.789090+0000","last_active":"2024-04-02T04:49:10.934761+0000","last_peered":"2024-04-02T04:49:10.934761+0000","last_clean":"2024-04-02T04:49:10.934761+0000","last_became_active":"2024-04-02T04:43:56.248560+0000","last_became_peered":"2024-04-02T04:43:56.248560+0000","last_unstale":"2024-04-02T04:49:10.934761+0000","last_undegraded":"2024-04-02T04:49:10.934761+0000","last_fullsized":"2024-04-02T04:49:10.934761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:49:03.788837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.788837+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:23.183446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":157,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.947305+0000","last_change":"2024-04-02T04:48:46.602957+0000","last_active":"2024-04-02T04:49:10.947305+0000","last_peered":"2024-04-02T04:49:10.947305+0000","last_clean":"2024-04-02T04:49:10.947305+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:49:10.947305+0000","last_undegraded":"2024-04-02T04:49:10.947305+0000","last_fullsized":"2024-04-02T04:49:10.947305+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:46.602849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.602849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:07.952150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921523+0000","last_change":"2024-04-02T04:49:08.351306+0000","last_active":"2024-04-02T04:49:10.921523+0000","last_peered":"2024-04-02T04:49:10.921523+0000","last_clean":"2024-04-02T04:49:10.921523+0000","last_became_active":"2024-04-02T04:43:58.318302+0000","last_became_peered":"2024-04-02T04:43:58.318302+0000","last_unstale":"2024-04-02T04:49:10.921523+0000","last_undegraded":"2024-04-02T04:49:10.921523+0000","last_fullsized":"2024-04-02T04:49:10.921523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.351209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:19.264476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":129,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923519+0000","last_change":"2024-04-02T04:47:13.288506+0000","last_active":"2024-04-02T04:49:10.923519+0000","last_peered":"2024-04-02T04:49:10.923519+0000","last_clean":"2024-04-02T04:49:10.923519+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:49:10.923519+0000","last_undegraded":"2024-04-02T04:49:10.923519+0000","last_fullsized":"2024-04-02T04:49:10.923519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:13.288364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:13.288364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.933798+0000","last_change":"2024-04-02T04:49:10.933798+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.933798+0000","last_undegraded":"2024-04-02T04:49:10.933798+0000","last_fullsized":"2024-04-02T04:49:10.933798+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.932602+0000","last_change":"2024-04-02T04:49:10.932602+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.932602+0000","last_undegraded":"2024-04-02T04:49:10.932602+0000","last_fullsized":"2024-04-02T04:49:10.932602+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"133'50","reported_seq":228,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.928714+0000","last_change":"2024-04-02T04:49:02.986990+0000","last_active":"2024-04-02T04:49:10.928714+0000","last_peered":"2024-04-02T04:49:10.928714+0000","last_clean":"2024-04-02T04:49:10.928714+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:49:10.928714+0000","last_undegraded":"2024-04-02T04:49:10.928714+0000","last_fullsized":"2024-04-02T04:49:10.928714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.986701+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:21.660310+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":155,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.926655+0000","last_change":"2024-04-02T04:48:51.870265+0000","last_active":"2024-04-02T04:49:10.926655+0000","last_peered":"2024-04-02T04:49:10.926655+0000","last_clean":"2024-04-02T04:49:10.926655+0000","last_became_active":"2024-04-02T04:43:54.209525+0000","last_became_peered":"2024-04-02T04:43:54.209525+0000","last_unstale":"2024-04-02T04:49:10.926655+0000","last_undegraded":"2024-04-02T04:49:10.926655+0000","last_fullsized":"2024-04-02T04:49:10.926655+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:51.870038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:51.870038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:03.817644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:11.011076+0000","last_change":"2024-04-02T04:49:11.011096+0000","last_active":"2024-04-02T04:49:11.011076+0000","last_peered":"2024-04-02T04:49:11.011076+0000","last_clean":"2024-04-02T04:49:11.011076+0000","last_became_active":"2024-04-02T04:43:58.317986+0000","last_became_peered":"2024-04-02T04:43:58.317986+0000","last_unstale":"2024-04-02T04:49:11.011076+0000","last_undegraded":"2024-04-02T04:49:11.011076+0000","last_fullsized":"2024-04-02T04:49:11.011076+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.322463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":138,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.945578+0000","last_change":"2024-04-02T04:48:42.394472+0000","last_active":"2024-04-02T04:49:10.945578+0000","last_peered":"2024-04-02T04:49:10.945578+0000","last_clean":"2024-04-02T04:49:10.945578+0000","last_became_active":"2024-04-02T04:44:00.339845+0000","last_became_peered":"2024-04-02T04:44:00.339845+0000","last_unstale":"2024-04-02T04:49:10.945578+0000","last_undegraded":"2024-04-02T04:49:10.945578+0000","last_fullsized":"2024-04-02T04:49:10.945578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:48:42.394228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.394228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:10.561833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.929333+0000","last_change":"2024-04-02T04:49:10.929333+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.929333+0000","last_undegraded":"2024-04-02T04:49:10.929333+0000","last_fullsized":"2024-04-02T04:49:10.929333+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":173,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918880+0000","last_change":"2024-04-02T04:47:45.516535+0000","last_active":"2024-04-02T04:49:10.918880+0000","last_peered":"2024-04-02T04:49:10.918880+0000","last_clean":"2024-04-02T04:49:10.918880+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:49:10.918880+0000","last_undegraded":"2024-04-02T04:49:10.918880+0000","last_fullsized":"2024-04-02T04:49:10.918880+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:47:45.516407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.516407+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":151,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921569+0000","last_change":"2024-04-02T04:47:58.472809+0000","last_active":"2024-04-02T04:49:10.921569+0000","last_peered":"2024-04-02T04:49:10.921569+0000","last_clean":"2024-04-02T04:49:10.921569+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:49:10.921569+0000","last_undegraded":"2024-04-02T04:49:10.921569+0000","last_fullsized":"2024-04-02T04:49:10.921569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:47:58.472703+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.472703+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":218,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.947751+0000","last_change":"2024-04-02T04:48:43.630108+0000","last_active":"2024-04-02T04:49:10.947751+0000","last_peered":"2024-04-02T04:49:10.947751+0000","last_clean":"2024-04-02T04:49:10.947751+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:49:10.947751+0000","last_undegraded":"2024-04-02T04:49:10.947751+0000","last_fullsized":"2024-04-02T04:49:10.947751+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:43.629796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.629796+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:44.617590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930043+0000","last_change":"2024-04-02T04:48:50.123453+0000","last_active":"2024-04-02T04:49:10.930043+0000","last_peered":"2024-04-02T04:49:10.930043+0000","last_clean":"2024-04-02T04:49:10.930043+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:49:10.930043+0000","last_undegraded":"2024-04-02T04:49:10.930043+0000","last_fullsized":"2024-04-02T04:49:10.930043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:50.123331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:50.123331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:09.567851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.920917+0000","last_change":"2024-04-02T04:49:10.920917+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.920917+0000","last_undegraded":"2024-04-02T04:49:10.920917+0000","last_fullsized":"2024-04-02T04:49:10.920917+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.935648+0000","last_change":"2024-04-02T04:49:10.935648+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.935648+0000","last_undegraded":"2024-04-02T04:49:10.935648+0000","last_fullsized":"2024-04-02T04:49:10.935648+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":170,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.920775+0000","last_change":"2024-04-02T04:49:05.020238+0000","last_active":"2024-04-02T04:49:10.920775+0000","last_peered":"2024-04-02T04:49:10.920775+0000","last_clean":"2024-04-02T04:49:10.920775+0000","last_became_active":"2024-04-02T04:43:56.241235+0000","last_became_peered":"2024-04-02T04:43:56.241235+0000","last_unstale":"2024-04-02T04:49:10.920775+0000","last_undegraded":"2024-04-02T04:49:10.920775+0000","last_fullsized":"2024-04-02T04:49:10.920775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:49:05.019939+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.019939+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:14.702289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.944279+0000","last_change":"2024-04-02T04:47:53.938679+0000","last_active":"2024-04-02T04:49:10.944279+0000","last_peered":"2024-04-02T04:49:10.944279+0000","last_clean":"2024-04-02T04:49:10.944279+0000","last_became_active":"2024-04-02T04:43:54.218901+0000","last_became_peered":"2024-04-02T04:43:54.218901+0000","last_unstale":"2024-04-02T04:49:10.944279+0000","last_undegraded":"2024-04-02T04:49:10.944279+0000","last_fullsized":"2024-04-02T04:49:10.944279+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.938537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.938537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:12.383253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":139,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.925042+0000","last_change":"2024-04-02T04:47:45.830668+0000","last_active":"2024-04-02T04:49:10.925042+0000","last_peered":"2024-04-02T04:49:10.925042+0000","last_clean":"2024-04-02T04:49:10.925042+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:49:10.925042+0000","last_undegraded":"2024-04-02T04:49:10.925042+0000","last_fullsized":"2024-04-02T04:49:10.925042+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.830369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.830369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:11.027610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922058+0000","last_change":"2024-04-02T04:48:38.627470+0000","last_active":"2024-04-02T04:49:10.922058+0000","last_peered":"2024-04-02T04:49:10.922058+0000","last_clean":"2024-04-02T04:49:10.922058+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:49:10.922058+0000","last_undegraded":"2024-04-02T04:49:10.922058+0000","last_fullsized":"2024-04-02T04:49:10.922058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:38.627239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.627239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:06.164715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.928276+0000","last_change":"2024-04-02T04:49:10.928276+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.928276+0000","last_undegraded":"2024-04-02T04:49:10.928276+0000","last_fullsized":"2024-04-02T04:49:10.928276+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.925646+0000","last_change":"2024-04-02T04:49:10.925646+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.925646+0000","last_undegraded":"2024-04-02T04:49:10.925646+0000","last_fullsized":"2024-04-02T04:49:10.925646+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":192,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923217+0000","last_change":"2024-04-02T04:47:40.838867+0000","last_active":"2024-04-02T04:49:10.923217+0000","last_peered":"2024-04-02T04:49:10.923217+0000","last_clean":"2024-04-02T04:49:10.923217+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:49:10.923217+0000","last_undegraded":"2024-04-02T04:49:10.923217+0000","last_fullsized":"2024-04-02T04:49:10.923217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:47:40.838617+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":157,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:12.329487+0000","last_change":"2024-04-02T04:49:12.329514+0000","last_active":"2024-04-02T04:49:12.329487+0000","last_peered":"2024-04-02T04:49:12.329487+0000","last_clean":"2024-04-02T04:49:12.329487+0000","last_became_active":"2024-04-02T04:43:54.226006+0000","last_became_peered":"2024-04-02T04:43:54.226006+0000","last_unstale":"2024-04-02T04:49:12.329487+0000","last_undegraded":"2024-04-02T04:49:12.329487+0000","last_fullsized":"2024-04-02T04:49:12.329487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.329420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.329420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:12.939887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922459+0000","last_change":"2024-04-02T04:47:57.459178+0000","last_active":"2024-04-02T04:49:10.922459+0000","last_peered":"2024-04-02T04:49:10.922459+0000","last_clean":"2024-04-02T04:49:10.922459+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:49:10.922459+0000","last_undegraded":"2024-04-02T04:49:10.922459+0000","last_fullsized":"2024-04-02T04:49:10.922459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:57.459040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.459040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:17.124034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922726+0000","last_change":"2024-04-02T04:49:05.447259+0000","last_active":"2024-04-02T04:49:10.922726+0000","last_peered":"2024-04-02T04:49:10.922726+0000","last_clean":"2024-04-02T04:49:10.922726+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:49:10.922726+0000","last_undegraded":"2024-04-02T04:49:10.922726+0000","last_fullsized":"2024-04-02T04:49:10.922726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.447020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.447020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:16.594500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.929378+0000","last_change":"2024-04-02T04:49:10.929378+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.929378+0000","last_undegraded":"2024-04-02T04:49:10.929378+0000","last_fullsized":"2024-04-02T04:49:10.929378+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:49:10.894037+0000","last_change":"2024-04-02T04:49:10.894037+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.894037+0000","last_undegraded":"2024-04-02T04:49:10.894037+0000","last_fullsized":"2024-04-02T04:49:10.894037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":178,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.930711+0000","last_change":"2024-04-02T04:48:43.963496+0000","last_active":"2024-04-02T04:49:10.930711+0000","last_peered":"2024-04-02T04:49:10.930711+0000","last_clean":"2024-04-02T04:49:10.930711+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:49:10.930711+0000","last_undegraded":"2024-04-02T04:49:10.930711+0000","last_fullsized":"2024-04-02T04:49:10.930711+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:48:43.963234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.963234+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:56.321677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":156,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.944720+0000","last_change":"2024-04-02T04:48:48.436877+0000","last_active":"2024-04-02T04:49:10.944720+0000","last_peered":"2024-04-02T04:49:10.944720+0000","last_clean":"2024-04-02T04:49:10.944720+0000","last_became_active":"2024-04-02T04:43:54.214496+0000","last_became_peered":"2024-04-02T04:43:54.214496+0000","last_unstale":"2024-04-02T04:49:10.944720+0000","last_undegraded":"2024-04-02T04:49:10.944720+0000","last_fullsized":"2024-04-02T04:49:10.944720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:48:48.436655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:48.436655+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:58.099345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923995+0000","last_change":"2024-04-02T04:49:01.626968+0000","last_active":"2024-04-02T04:49:10.923995+0000","last_peered":"2024-04-02T04:49:10.923995+0000","last_clean":"2024-04-02T04:49:10.923995+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:49:10.923995+0000","last_undegraded":"2024-04-02T04:49:10.923995+0000","last_fullsized":"2024-04-02T04:49:10.923995+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.626750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.626750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.603064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918719+0000","last_change":"2024-04-02T04:48:02.193707+0000","last_active":"2024-04-02T04:49:10.918719+0000","last_peered":"2024-04-02T04:49:10.918719+0000","last_clean":"2024-04-02T04:49:10.918719+0000","last_became_active":"2024-04-02T04:44:00.348135+0000","last_became_peered":"2024-04-02T04:44:00.348135+0000","last_unstale":"2024-04-02T04:49:10.918719+0000","last_undegraded":"2024-04-02T04:49:10.918719+0000","last_fullsized":"2024-04-02T04:49:10.918719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:31.172290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.928977+0000","last_change":"2024-04-02T04:49:10.928977+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.928977+0000","last_undegraded":"2024-04-02T04:49:10.928977+0000","last_fullsized":"2024-04-02T04:49:10.928977+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.936807+0000","last_change":"2024-04-02T04:49:10.936807+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.936807+0000","last_undegraded":"2024-04-02T04:49:10.936807+0000","last_fullsized":"2024-04-02T04:49:10.936807+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":168,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.933907+0000","last_change":"2024-04-02T04:48:46.888055+0000","last_active":"2024-04-02T04:49:10.933907+0000","last_peered":"2024-04-02T04:49:10.933907+0000","last_clean":"2024-04-02T04:49:10.933907+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:49:10.933907+0000","last_undegraded":"2024-04-02T04:49:10.933907+0000","last_fullsized":"2024-04-02T04:49:10.933907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:46.887903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.887903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:14.156917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":148,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.946258+0000","last_change":"2024-04-02T04:47:45.254673+0000","last_active":"2024-04-02T04:49:10.946258+0000","last_peered":"2024-04-02T04:49:10.946258+0000","last_clean":"2024-04-02T04:49:10.946258+0000","last_became_active":"2024-04-02T04:43:54.231048+0000","last_became_peered":"2024-04-02T04:43:54.231048+0000","last_unstale":"2024-04-02T04:49:10.946258+0000","last_undegraded":"2024-04-02T04:49:10.946258+0000","last_fullsized":"2024-04-02T04:49:10.946258+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:45.254450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:45.254450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.919049+0000","last_change":"2024-04-02T04:47:37.876855+0000","last_active":"2024-04-02T04:49:10.919049+0000","last_peered":"2024-04-02T04:49:10.919049+0000","last_clean":"2024-04-02T04:49:10.919049+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:49:10.919049+0000","last_undegraded":"2024-04-02T04:49:10.919049+0000","last_fullsized":"2024-04-02T04:49:10.919049+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:37.876644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:37.876644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":143,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.927010+0000","last_change":"2024-04-02T04:48:59.855804+0000","last_active":"2024-04-02T04:49:10.927010+0000","last_peered":"2024-04-02T04:49:10.927010+0000","last_clean":"2024-04-02T04:49:10.927010+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:49:10.927010+0000","last_undegraded":"2024-04-02T04:49:10.927010+0000","last_fullsized":"2024-04-02T04:49:10.927010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.855585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.855585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:02.473482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.924053+0000","last_change":"2024-04-02T04:49:10.924053+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.924053+0000","last_undegraded":"2024-04-02T04:49:10.924053+0000","last_fullsized":"2024-04-02T04:49:10.924053+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.916214+0000","last_change":"2024-04-02T04:49:10.916214+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.916214+0000","last_undegraded":"2024-04-02T04:49:10.916214+0000","last_fullsized":"2024-04-02T04:49:10.916214+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":176,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921046+0000","last_change":"2024-04-02T04:49:01.074976+0000","last_active":"2024-04-02T04:49:10.921046+0000","last_peered":"2024-04-02T04:49:10.921046+0000","last_clean":"2024-04-02T04:49:10.921046+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:49:10.921046+0000","last_undegraded":"2024-04-02T04:49:10.921046+0000","last_fullsized":"2024-04-02T04:49:10.921046+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:01.074728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.074728+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:18.216149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921977+0000","last_change":"2024-04-02T04:47:42.551341+0000","last_active":"2024-04-02T04:49:10.921977+0000","last_peered":"2024-04-02T04:49:10.921977+0000","last_clean":"2024-04-02T04:49:10.921977+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:49:10.921977+0000","last_undegraded":"2024-04-02T04:49:10.921977+0000","last_fullsized":"2024-04-02T04:49:10.921977+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:42.551243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:42.551243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":140,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.918437+0000","last_change":"2024-04-02T04:49:05.650145+0000","last_active":"2024-04-02T04:49:10.918437+0000","last_peered":"2024-04-02T04:49:10.918437+0000","last_clean":"2024-04-02T04:49:10.918437+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:49:10.918437+0000","last_undegraded":"2024-04-02T04:49:10.918437+0000","last_fullsized":"2024-04-02T04:49:10.918437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.649928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.752553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":136,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922982+0000","last_change":"2024-04-02T04:49:07.677962+0000","last_active":"2024-04-02T04:49:10.922982+0000","last_peered":"2024-04-02T04:49:10.922982+0000","last_clean":"2024-04-02T04:49:10.922982+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:49:10.922982+0000","last_undegraded":"2024-04-02T04:49:10.922982+0000","last_fullsized":"2024-04-02T04:49:10.922982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.677740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.970457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.925859+0000","last_change":"2024-04-02T04:49:10.925859+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.925859+0000","last_undegraded":"2024-04-02T04:49:10.925859+0000","last_fullsized":"2024-04-02T04:49:10.925859+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.938945+0000","last_change":"2024-04-02T04:49:10.938945+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.938945+0000","last_undegraded":"2024-04-02T04:49:10.938945+0000","last_fullsized":"2024-04-02T04:49:10.938945+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":172,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.922230+0000","last_change":"2024-04-02T04:48:41.484207+0000","last_active":"2024-04-02T04:49:10.922230+0000","last_peered":"2024-04-02T04:49:10.922230+0000","last_clean":"2024-04-02T04:49:10.922230+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:49:10.922230+0000","last_undegraded":"2024-04-02T04:49:10.922230+0000","last_fullsized":"2024-04-02T04:49:10.922230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:48:41.484047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.484047+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:47.548311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":149,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921195+0000","last_change":"2024-04-02T04:47:54.214859+0000","last_active":"2024-04-02T04:49:10.921195+0000","last_peered":"2024-04-02T04:49:10.921195+0000","last_clean":"2024-04-02T04:49:10.921195+0000","last_became_active":"2024-04-02T04:43:54.216319+0000","last_became_peered":"2024-04-02T04:43:54.216319+0000","last_unstale":"2024-04-02T04:49:10.921195+0000","last_undegraded":"2024-04-02T04:49:10.921195+0000","last_fullsized":"2024-04-02T04:49:10.921195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:19.434714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.921420+0000","last_change":"2024-04-02T04:47:52.450415+0000","last_active":"2024-04-02T04:49:10.921420+0000","last_peered":"2024-04-02T04:49:10.921420+0000","last_clean":"2024-04-02T04:49:10.921420+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:49:10.921420+0000","last_undegraded":"2024-04-02T04:49:10.921420+0000","last_fullsized":"2024-04-02T04:49:10.921420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:52.450181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:52.450181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":137,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-02T04:49:10.923047+0000","last_change":"2024-04-02T04:49:09.389664+0000","last_active":"2024-04-02T04:49:10.923047+0000","last_peered":"2024-04-02T04:49:10.923047+0000","last_clean":"2024-04-02T04:49:10.923047+0000","last_became_active":"2024-04-02T04:44:00.355360+0000","last_became_peered":"2024-04-02T04:44:00.355360+0000","last_unstale":"2024-04-02T04:49:10.923047+0000","last_undegraded":"2024-04-02T04:49:10.923047+0000","last_fullsized":"2024-04-02T04:49:10.923047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.389577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.389577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.278664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"creating+peering","last_fresh":"2024-04-02T04:49:10.916110+0000","last_change":"2024-04-02T04:49:10.916110+0000","last_active":"2024-04-02T04:49:10.894037+0000","last_peered":"2024-04-02T04:49:10.894037+0000","last_clean":"2024-04-02T04:49:10.894037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:49:10.916110+0000","last_undegraded":"2024-04-02T04:49:10.916110+0000","last_fullsized":"2024-04-02T04:49:10.916110+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:10.894037+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.894037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":81,"acting":81,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743498,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712564,"statfs":{"total":95995035648,"available":95961665536,"internally_reserved":0,"allocated":1441792,"data_stored":1053338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103253,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1409024,"data_stored":1051151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":831488,"data_stored":461863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738514,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1425408,"data_stored":1051815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378750,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":819200,"data_stored":462099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051700,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":461164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822762,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":856064,"data_stored":462092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463006,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":819200,"data_stored":462008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- 172.21.15.69:0/3432459120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c100790 msgr2=0x7efc1c100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- 172.21.15.69:0/3432459120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c100790 0x7efc1c100c10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc100099c0 tx=0x7efc10009cd0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- 172.21.15.69:0/3432459120 shutdown_connections 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- 172.21.15.69:0/3432459120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 0x7efc1c10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- 172.21.15.69:0/3432459120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c107260 0x7efc1c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- 172.21.15.69:0/3432459120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c100790 0x7efc1c100c10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- 172.21.15.69:0/3432459120 >> 172.21.15.69:0/3432459120 conn(0x7efc1c077740 msgr2=0x7efc1c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:15.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- 172.21.15.69:0/3432459120 shutdown_connections 2024-04-02T04:49:15.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- 172.21.15.69:0/3432459120 wait complete. 2024-04-02T04:49:15.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 librados: starting msgr at 2024-04-02T04:49:15.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 librados: starting objecter 2024-04-02T04:49:15.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 Processor -- start 2024-04-02T04:49:15.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- start start 2024-04-02T04:49:15.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 librados: setting wanted keys 2024-04-02T04:49:15.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 librados: calling monclient init 2024-04-02T04:49:15.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c100790 0x7efc1c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:15.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 0x7efc1c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:15.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c107260 0x7efc1c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:15.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc1c111b40 con 0x7efc1c107260 2024-04-02T04:49:15.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc1c1119c0 con 0x7efc1c100790 2024-04-02T04:49:15.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc2149e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efc1c111cc0 con 0x7efc1c101150 2024-04-02T04:49:15.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c100790 0x7efc1c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:15.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c100790 0x7efc1c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.69:34336/0 (socket says 172.21.15.69:34336) 2024-04-02T04:49:15.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.982+0000 7efc1affd700 1 -- 172.21.15.69:0/1573734000 learned_addr learned my addr 172.21.15.69:0/1573734000 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 0x7efc1c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 -- 172.21.15.69:0/1573734000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c100790 msgr2=0x7efc1c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c100790 0x7efc1c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 -- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c107260 msgr2=0x7efc1c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c107260 0x7efc1c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 -- 172.21.15.69:0/1573734000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1c19d0e0 con 0x7efc1c101150 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc1a7fc700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 0x7efc1c198610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc100098e0 tx=0x7efc10005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:15.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc03fff700 10 client.?.objecter ms_handle_connect 0x7efc1c101150 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc03fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10009e60 con 0x7efc1c101150 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7efc100054c0 con 0x7efc1c101150 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc1c19d370 con 0x7efc1c101150 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc2149e700 10 client.24971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc1c1018a0 con 0x7efc1c101150 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc2149e700 1 librados: init done 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1001fcd0 con 0x7efc1c101150 2024-04-02T04:49:15.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.986+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc08001fa0 con 0x7efc1c101150 2024-04-02T04:49:15.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7efc10005040 con 0x7efc1c101150 2024-04-02T04:49:15.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efc0406f5e0 0x7efc04071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:15.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(134..134 src has 74..134) ==== 8265+0+0 (secure 0 0 0) 0x7efc10093dd0 con 0x7efc1c101150 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 10 client.24971.objecter ms_dispatch 0x7efc1c001550 osd_map(134..134 src has 74..134) 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 3 client.24971.objecter handle_osd_map got epochs [134,134] > 0 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 3 client.24971.objecter handle_osd_map decoding full epoch 134 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc1affd700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efc0406f5e0 0x7efc04071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 20 client.24971.objecter dump_active .. 0 homeless 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc03fff700 10 client.24971.objecter ms_handle_connect 0x7efc0406f5e0 2024-04-02T04:49:15.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.990+0000 7efc1affd700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efc0406f5e0 0x7efc04071aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc0c005950 tx=0x7efc0c00b500 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:15.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:15.994+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efc10013070 con 0x7efc1c101150 2024-04-02T04:49:15.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: cluster 2024-04-02T04:49:09.716872+0000 osd.6 (osd.6) 89 : cluster 0 4.a scrub starts 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: cluster 2024-04-02T04:49:09.719133+0000 osd.6 (osd.6) 90 : cluster 0 4.a scrub ok 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: cluster 2024-04-02T04:49:09.834829+0000 mgr.y (mgr.14538) 197 : cluster 0 pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.159988+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.183358+0000 mon.c (mon.2) 33 : audit 0 from='client.? 172.21.15.69:0/3653090868' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.333277+0000 mon.c (mon.2) 34 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.358830+0000 mon.b (mon.1) 109 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: cluster 2024-04-02T04:49:10.371163+0000 osd.7 (osd.7) 131 : cluster 0 6.12 scrub starts 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: cluster 2024-04-02T04:49:10.372294+0000 osd.7 (osd.7) 132 : cluster 0 6.12 scrub ok 2024-04-02T04:49:16.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.910845+0000 mon.a (mon.0) 957 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.911056+0000 mon.a (mon.0) 958 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[21528]: audit 2024-04-02T04:49:10.912855+0000 mon.a (mon.0) 959 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: cluster 2024-04-02T04:49:09.716872+0000 osd.6 (osd.6) 89 : cluster 0 4.a scrub starts 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: cluster 2024-04-02T04:49:09.719133+0000 osd.6 (osd.6) 90 : cluster 0 4.a scrub ok 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: cluster 2024-04-02T04:49:09.834829+0000 mgr.y (mgr.14538) 197 : cluster 0 pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.159988+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.183358+0000 mon.c (mon.2) 33 : audit 0 from='client.? 172.21.15.69:0/3653090868' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.333277+0000 mon.c (mon.2) 34 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.358830+0000 mon.b (mon.1) 109 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: cluster 2024-04-02T04:49:10.371163+0000 osd.7 (osd.7) 131 : cluster 0 6.12 scrub starts 2024-04-02T04:49:16.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: cluster 2024-04-02T04:49:10.372294+0000 osd.7 (osd.7) 132 : cluster 0 6.12 scrub ok 2024-04-02T04:49:16.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.910845+0000 mon.a (mon.0) 957 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:16.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.911056+0000 mon.a (mon.0) 958 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:15 smithi069 bash[28904]: audit 2024-04-02T04:49:10.912855+0000 mon.a (mon.0) 959 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.048 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:16.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:16.173 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:16.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:16.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:16.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:16.306+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7efc08001530 con 0x7efc1c101150 2024-04-02T04:49:16.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:16.427 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: cluster 2024-04-02T04:49:09.716872+0000 osd.6 (osd.6) 89 : cluster 0 4.a scrub starts 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: cluster 2024-04-02T04:49:09.719133+0000 osd.6 (osd.6) 90 : cluster 0 4.a scrub ok 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: cluster 2024-04-02T04:49:09.834829+0000 mgr.y (mgr.14538) 197 : cluster 0 pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.159988+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.183358+0000 mon.c (mon.2) 33 : audit 0 from='client.? 172.21.15.69:0/3653090868' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.333277+0000 mon.c (mon.2) 34 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.358830+0000 mon.b (mon.1) 109 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: cluster 2024-04-02T04:49:10.371163+0000 osd.7 (osd.7) 131 : cluster 0 6.12 scrub starts 2024-04-02T04:49:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: cluster 2024-04-02T04:49:10.372294+0000 osd.7 (osd.7) 132 : cluster 0 6.12 scrub ok 2024-04-02T04:49:16.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.910845+0000 mon.a (mon.0) 957 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:16.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.911056+0000 mon.a (mon.0) 958 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:15 smithi098 bash[23790]: audit 2024-04-02T04:49:10.912855+0000 mon.a (mon.0) 959 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:16.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:16.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:16.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:16.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:16.941 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:16.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:17.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:17.067 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:17.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:17.194 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:17.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:17.319 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:17.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:17.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:17.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:17.575 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:17.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:17.705 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:17.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:17.832 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:17.935 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:17.959 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:17.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:18.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:18.089 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:18.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:18.216 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:18.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.343 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:18.446 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.471 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.472 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:18.472 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:18.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:18.600 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:18.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:18.727 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:18.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:49:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:49:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:49:18.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:18.951 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:18.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:19.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:19.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:19.238 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:19.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:19.372 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:19.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:19.502 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:19.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:19.629 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:19.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:19.756 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:19.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:19.889 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:19.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:19.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:20.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:20.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:20.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:20.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:20.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:20.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:20.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:20.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:20.654 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:20.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:20.783 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:20.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:20.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:20.932 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:49:20.932 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:49:20.933 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-02T04:49:20.933 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-02T04:49:20.933 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-02T04:49:20.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-04-02T04:49:20.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:20.990+0000 7efc20c9d700 10 client.24971.objecter tick 2024-04-02T04:49:21.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:21.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:21.189 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:21.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:10.651284+0000 osd.2 (osd.2) 69 : cluster 0 2.10 scrub starts 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:10.653516+0000 osd.2 (osd.2) 70 : cluster 0 2.10 scrub ok 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:10.758135+0000 osd.6 (osd.6) 91 : cluster 0 3.a scrub starts 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:10.763390+0000 osd.6 (osd.6) 92 : cluster 0 3.a scrub ok 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: audit 2024-04-02T04:49:10.936496+0000 mon.b (mon.1) 110 : audit 1 from='client.? 172.21.15.69:0/3394014510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.009516+0000 osd.0 (osd.0) 129 : cluster 0 4.1e scrub starts 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.011015+0000 osd.0 (osd.0) 130 : cluster 0 4.1e scrub ok 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.069513+0000 osd.1 (osd.1) 123 : cluster 0 2.a deep-scrub starts 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.071221+0000 osd.1 (osd.1) 124 : cluster 0 2.a deep-scrub ok 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.342616+0000 osd.7 (osd.7) 133 : cluster 0 2.8 deep-scrub starts 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.344530+0000 osd.7 (osd.7) 134 : cluster 0 2.8 deep-scrub ok 2024-04-02T04:49:21.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.705656+0000 osd.4 (osd.4) 133 : cluster 0 6.17 scrub starts 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.707901+0000 osd.4 (osd.4) 134 : cluster 0 6.17 scrub ok 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.835601+0000 mgr.y (mgr.14538) 198 : cluster 0 pgmap v167: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.990200+0000 osd.0 (osd.0) 131 : cluster 0 6.0 scrub starts 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:11.991227+0000 osd.0 (osd.0) 132 : cluster 0 6.0 scrub ok 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:12.327485+0000 osd.3 (osd.3) 161 : cluster 0 2.1b scrub starts 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:12.329416+0000 osd.3 (osd.3) 162 : cluster 0 2.1b scrub ok 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.971407+0000 mon.a (mon.0) 960 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[21528]: audit 2024-04-02T04:49:15.985287+0000 mon.a (mon.0) 961 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:10.651284+0000 osd.2 (osd.2) 69 : cluster 0 2.10 scrub starts 2024-04-02T04:49:21.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:10.653516+0000 osd.2 (osd.2) 70 : cluster 0 2.10 scrub ok 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:10.758135+0000 osd.6 (osd.6) 91 : cluster 0 3.a scrub starts 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:10.763390+0000 osd.6 (osd.6) 92 : cluster 0 3.a scrub ok 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: audit 2024-04-02T04:49:10.936496+0000 mon.b (mon.1) 110 : audit 1 from='client.? 172.21.15.69:0/3394014510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.009516+0000 osd.0 (osd.0) 129 : cluster 0 4.1e scrub starts 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.011015+0000 osd.0 (osd.0) 130 : cluster 0 4.1e scrub ok 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.069513+0000 osd.1 (osd.1) 123 : cluster 0 2.a deep-scrub starts 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.071221+0000 osd.1 (osd.1) 124 : cluster 0 2.a deep-scrub ok 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.342616+0000 osd.7 (osd.7) 133 : cluster 0 2.8 deep-scrub starts 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.344530+0000 osd.7 (osd.7) 134 : cluster 0 2.8 deep-scrub ok 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.705656+0000 osd.4 (osd.4) 133 : cluster 0 6.17 scrub starts 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.707901+0000 osd.4 (osd.4) 134 : cluster 0 6.17 scrub ok 2024-04-02T04:49:21.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.835601+0000 mgr.y (mgr.14538) 198 : cluster 0 pgmap v167: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:21.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.990200+0000 osd.0 (osd.0) 131 : cluster 0 6.0 scrub starts 2024-04-02T04:49:21.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:11.991227+0000 osd.0 (osd.0) 132 : cluster 0 6.0 scrub ok 2024-04-02T04:49:21.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:12.327485+0000 osd.3 (osd.3) 161 : cluster 0 2.1b scrub starts 2024-04-02T04:49:21.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:12.329416+0000 osd.3 (osd.3) 162 : cluster 0 2.1b scrub ok 2024-04-02T04:49:21.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.971407+0000 mon.a (mon.0) 960 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-04-02T04:49:21.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:21 smithi069 bash[28904]: audit 2024-04-02T04:49:15.985287+0000 mon.a (mon.0) 961 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:21.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:21.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:21.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:10.651284+0000 osd.2 (osd.2) 69 : cluster 0 2.10 scrub starts 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:10.653516+0000 osd.2 (osd.2) 70 : cluster 0 2.10 scrub ok 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:10.758135+0000 osd.6 (osd.6) 91 : cluster 0 3.a scrub starts 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:10.763390+0000 osd.6 (osd.6) 92 : cluster 0 3.a scrub ok 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: audit 2024-04-02T04:49:10.936496+0000 mon.b (mon.1) 110 : audit 1 from='client.? 172.21.15.69:0/3394014510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.009516+0000 osd.0 (osd.0) 129 : cluster 0 4.1e scrub starts 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.011015+0000 osd.0 (osd.0) 130 : cluster 0 4.1e scrub ok 2024-04-02T04:49:21.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.069513+0000 osd.1 (osd.1) 123 : cluster 0 2.a deep-scrub starts 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.071221+0000 osd.1 (osd.1) 124 : cluster 0 2.a deep-scrub ok 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.342616+0000 osd.7 (osd.7) 133 : cluster 0 2.8 deep-scrub starts 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.344530+0000 osd.7 (osd.7) 134 : cluster 0 2.8 deep-scrub ok 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.705656+0000 osd.4 (osd.4) 133 : cluster 0 6.17 scrub starts 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.707901+0000 osd.4 (osd.4) 134 : cluster 0 6.17 scrub ok 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.835601+0000 mgr.y (mgr.14538) 198 : cluster 0 pgmap v167: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.990200+0000 osd.0 (osd.0) 131 : cluster 0 6.0 scrub starts 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:11.991227+0000 osd.0 (osd.0) 132 : cluster 0 6.0 scrub ok 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:12.327485+0000 osd.3 (osd.3) 161 : cluster 0 2.1b scrub starts 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:12.329416+0000 osd.3 (osd.3) 162 : cluster 0 2.1b scrub ok 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.971407+0000 mon.a (mon.0) 960 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-04-02T04:49:21.438 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:21 smithi098 bash[23790]: audit 2024-04-02T04:49:15.985287+0000 mon.a (mon.0) 961 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:21.452 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:21.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:21.580 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:21.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:21.704 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:21.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:21.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:21.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:21.955 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:21.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:22.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:22.086 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:22.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:22.216 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:22.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:22.343 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:22.447 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:22.471 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:22.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:22.601 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:22.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- 172.21.15.69:0/3662872340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 msgr2=0x7fbc84106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- 172.21.15.69:0/3662872340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84106f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009990 tx=0x7fbc78009ca0 comp rx=0 tx=0).stop 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- 172.21.15.69:0/3662872340 shutdown_connections 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- 172.21.15.69:0/3662872340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc84107650 0x7fbc8410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- 172.21.15.69:0/3662872340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc84100a30 0x7fbc84100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- 172.21.15.69:0/3662872340 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84106f20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- 172.21.15.69:0/3662872340 >> 172.21.15.69:0/3662872340 conn(0x7fbc84077740 msgr2=0x7fbc840feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- 172.21.15.69:0/3662872340 shutdown_connections 2024-04-02T04:49:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- 172.21.15.69:0/3662872340 wait complete. 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 librados: starting msgr at 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 librados: starting objecter 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 Processor -- start 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- start start 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 librados: setting wanted keys 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 librados: calling monclient init 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc84100a30 0x7fbc841019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc84107650 0x7fbc84102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbc84111b20 con 0x7fbc84100a30 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc841119a0 con 0x7fbc84107650 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc8980b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbc84111ca0 con 0x7fbc841013e0 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc82ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc84100a30 0x7fbc841019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84101ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50866/0 (socket says 172.21.15.69:50866) 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 -- 172.21.15.69:0/1293974878 learned_addr learned my addr 172.21.15.69:0/1293974878 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc84107650 0x7fbc84102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc84107650 msgr2=0x7fbc84102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc84107650 0x7fbc84102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc84100a30 msgr2=0x7fbc841019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc84100a30 0x7fbc841019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc841a6530 con 0x7fbc841013e0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84101ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800bff0 tx=0x7fbc78009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc88809700 10 client.?.objecter ms_handle_connect 0x7fbc841013e0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc88809700 10 client.?.objecter resend_mon_ops 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc78027070 con 0x7fbc841013e0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.818+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fbc78003fc0 con 0x7fbc841013e0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc841a67c0 con 0x7fbc841013e0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc78022a80 con 0x7fbc841013e0 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 10 client.25055.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:49:22.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc841a6ca0 con 0x7fbc841013e0 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 1 librados: init done 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbc84102e90 con 0x7fbc841013e0 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fbc7802a070 con 0x7fbc841013e0 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbc6c06f5e0 0x7fbc6c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(136..136 src has 74..136) ==== 17791+0+0 (secure 0 0 0) 0x7fbc78091860 con 0x7fbc841013e0 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 osd_map(136..136 src has 74..136) 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 3 client.25055.objecter handle_osd_map got epochs [136,136] > 0 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 3 client.25055.objecter handle_osd_map decoding full epoch 136 2024-04-02T04:49:22.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbc6c06f5e0 0x7fbc6c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 20 client.25055.objecter dump_active .. 0 homeless 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v136) ==== 54+0+16 (secure 0 0 0) 0x7fbc78013070 con 0x7fbc841013e0 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 10 client.25055.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 1 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbc840630a0 0x7fbc841090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbc6c06f5e0 0x7fbc6c071aa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc74005950 tx=0x7fbc7400b410 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 1 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc6c06f5e0 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 0 1 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 4 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 10 client.25055.objecter _send_command 1 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc8410a790 con 0x7fbc840630a0 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbc840630a0 0x7fbc841090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc840630a0 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.822+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbc840630a0 0x7fbc841090d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.826+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc8410a790 con 0x7fbc840630a0 2024-04-02T04:49:22.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.826+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 1: 0 ) 2024-04-02T04:49:22.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.826+0000 7fbc88809700 10 client.25055.objecter _finish_command 1 = system:0 2024-04-02T04:49:22.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.826+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 4 2024-04-02T04:49:22.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.826+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 0 1 2024-04-02T04:49:22.832 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:49:22.832 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:49:22.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:22.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 10 client.25055.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:22.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 2 2024-04-02T04:49:22.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 2 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc84103ad0 osd=0 3 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 0 2 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 4 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 10 client.25055.objecter _send_command 2 2024-04-02T04:49:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.854+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc84109610 con 0x7fbc840630a0 2024-04-02T04:49:22.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc84109610 con 0x7fbc840630a0 2024-04-02T04:49:22.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:22.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 10 client.25055.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:22.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 4 2024-04-02T04:49:22.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 0 2 2024-04-02T04:49:22.858 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:49:22.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 10 client.25055.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:22.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fbc84062c90 0x7fbc841aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 1 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 4 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 10 client.25055.objecter _send_command 3 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbc8410a500 con 0x7fbc84062c90 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fbc84062c90 0x7fbc841aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc84062c90 2024-04-02T04:49:22.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fbc84062c90 0x7fbc841aaf20 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc8410a500 con 0x7fbc84062c90 2024-04-02T04:49:22.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 3: 0 ) 2024-04-02T04:49:22.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 10 client.25055.objecter _finish_command 3 = system:0 2024-04-02T04:49:22.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 4 2024-04-02T04:49:22.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.858+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 1 3 2024-04-02T04:49:22.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 10 client.25055.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 4 2024-04-02T04:49:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 4 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc84062b20 osd=1 3 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 1 4 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 4 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 10 client.25055.objecter _send_command 4 2024-04-02T04:49:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc84109a70 con 0x7fbc84062c90 2024-04-02T04:49:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc84109a70 con 0x7fbc84062c90 2024-04-02T04:49:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc88809700 10 client.25055.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 4 2024-04-02T04:49:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 1 4 2024-04-02T04:49:22.879 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:49:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 10 client.25055.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 5 2024-04-02T04:49:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fbc8402d690 0x7fbc841ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 5 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 2 5 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 4 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 10 client.25055.objecter _send_command 5 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbc841b1450 con 0x7fbc8402d690 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fbc8402d690 0x7fbc841ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fbc8402d690 0x7fbc841ae5f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.878+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc8402d690 2024-04-02T04:49:22.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.882+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc841b1450 con 0x7fbc8402d690 2024-04-02T04:49:22.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.882+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 5: 0 ) 2024-04-02T04:49:22.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.882+0000 7fbc88809700 10 client.25055.objecter _finish_command 5 = system:0 2024-04-02T04:49:22.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.882+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 4 2024-04-02T04:49:22.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.882+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 2 5 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 10 client.25055.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 6 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 6 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc84109e70 osd=2 3 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 2 6 2024-04-02T04:49:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 4 2024-04-02T04:49:22.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 10 client.25055.objecter _send_command 6 2024-04-02T04:49:22.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.898+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc8402cc60 con 0x7fbc8402d690 2024-04-02T04:49:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc8402cc60 con 0x7fbc8402d690 2024-04-02T04:49:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc88809700 10 client.25055.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 4 2024-04-02T04:49:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 2 6 2024-04-02T04:49:22.903 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:49:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 10 client.25055.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 7 2024-04-02T04:49:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fbc841afcb0 0x7fbc841b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 7 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 3 7 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 4 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 10 client.25055.objecter _send_command 7 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbc841b0630 con 0x7fbc841afcb0 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fbc841afcb0 0x7fbc841b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc841afcb0 2024-04-02T04:49:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.902+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fbc841afcb0 0x7fbc841b00f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.906+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc841b0630 con 0x7fbc841afcb0 2024-04-02T04:49:22.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.906+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 7: 0 ) 2024-04-02T04:49:22.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.906+0000 7fbc88809700 10 client.25055.objecter _finish_command 7 = system:0 2024-04-02T04:49:22.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.906+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 4 2024-04-02T04:49:22.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.906+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 3 7 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 10 client.25055.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 8 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 8 2024-04-02T04:49:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841abc30 osd=3 3 2024-04-02T04:49:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 3 8 2024-04-02T04:49:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 4 2024-04-02T04:49:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 10 client.25055.objecter _send_command 8 2024-04-02T04:49:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc841b0630 con 0x7fbc841afcb0 2024-04-02T04:49:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc841b0630 con 0x7fbc841afcb0 2024-04-02T04:49:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc88809700 10 client.25055.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 4 2024-04-02T04:49:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 3 8 2024-04-02T04:49:22.927 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:49:22.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 10 client.25055.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:22.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 9 2024-04-02T04:49:22.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fbc841b1010 0x7fbc841b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 9 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 4 9 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 4 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 10 client.25055.objecter _send_command 9 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.922+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbc841b84a0 con 0x7fbc841b1010 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fbc841b1010 0x7fbc841b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc841b1010 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fbc841b1010 0x7fbc841b5620 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc841b84a0 con 0x7fbc841b1010 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 9: 0 ) 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc88809700 10 client.25055.objecter _finish_command 9 = system:0 2024-04-02T04:49:22.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 4 2024-04-02T04:49:22.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.926+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 4 9 2024-04-02T04:49:22.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:22.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 10 client.25055.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:22.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 10 2024-04-02T04:49:22.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 10 2024-04-02T04:49:22.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841b0630 osd=4 3 2024-04-02T04:49:22.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 4 10 2024-04-02T04:49:22.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 4 2024-04-02T04:49:22.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 10 client.25055.objecter _send_command 10 2024-04-02T04:49:22.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.974+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc841b5b60 con 0x7fbc841b1010 2024-04-02T04:49:22.976 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc841b5b60 con 0x7fbc841b1010 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 10 client.25055.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 4 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 4 10 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 10 client.25055.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 11 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fbc841b6280 0x7fbc841b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 11 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 5 11 2024-04-02T04:49:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 4 2024-04-02T04:49:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 10 client.25055.objecter _send_command 11 2024-04-02T04:49:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbc841bb8c0 con 0x7fbc841b6280 2024-04-02T04:49:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fbc841b6280 0x7fbc841b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc841b6280 2024-04-02T04:49:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc827fc700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fbc841b6280 0x7fbc841b66e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:22.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc841bb8c0 con 0x7fbc841b6280 2024-04-02T04:49:22.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 11: 0 ) 2024-04-02T04:49:22.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 10 client.25055.objecter _finish_command 11 = system:0 2024-04-02T04:49:22.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 4 2024-04-02T04:49:22.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.978+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 5 11 2024-04-02T04:49:22.991 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:22.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 10 client.25055.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 12 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 12 2024-04-02T04:49:22.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841b5b60 osd=5 3 2024-04-02T04:49:22.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 5 12 2024-04-02T04:49:22.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 4 2024-04-02T04:49:22.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 10 client.25055.objecter _send_command 12 2024-04-02T04:49:22.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:22.998+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc841bb8c0 con 0x7fbc841b6280 2024-04-02T04:49:23.001 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc841bb8c0 con 0x7fbc841b6280 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc88809700 10 client.25055.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 4 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 5 12 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 10 client.25055.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 13 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fbc841bac00 0x7fbc841bb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:23.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 13 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 6 13 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 4 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 10 client.25055.objecter _send_command 13 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbc841bec80 con 0x7fbc841bac00 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fbc841bac00 0x7fbc841bb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc841bac00 2024-04-02T04:49:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.002+0000 7fbc837fe700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fbc841bac00 0x7fbc841bb060 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.006+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc841bec80 con 0x7fbc841bac00 2024-04-02T04:49:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.006+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 13: 0 ) 2024-04-02T04:49:23.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.006+0000 7fbc88809700 10 client.25055.objecter _finish_command 13 = system:0 2024-04-02T04:49:23.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.006+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 4 2024-04-02T04:49:23.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.006+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 6 13 2024-04-02T04:49:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 10 client.25055.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 14 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 14 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841bb8c0 osd=6 3 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 6 14 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 4 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 10 client.25055.objecter _send_command 14 2024-04-02T04:49:23.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.022+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc841b6c70 con 0x7fbc841bac00 2024-04-02T04:49:23.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc841b6c70 con 0x7fbc841bac00 2024-04-02T04:49:23.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:23.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc88809700 10 client.25055.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:23.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 4 2024-04-02T04:49:23.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 6 14 2024-04-02T04:49:23.027 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:49:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 10 client.25055.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 15 2024-04-02T04:49:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbc841b8050 0x7fbc841b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 15 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 7 15 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 4 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 10 client.25055.objecter _send_command 15 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbc841b7ac0 con 0x7fbc841b8050 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbc841b8050 0x7fbc841b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc88809700 10 client.25055.objecter ms_handle_connect 0x7fbc841b8050 2024-04-02T04:49:23.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.026+0000 7fbc82ffd700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbc841b8050 0x7fbc841b73d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.030+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc841b7ac0 con 0x7fbc841b8050 2024-04-02T04:49:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.030+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 15: 0 ) 2024-04-02T04:49:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.030+0000 7fbc88809700 10 client.25055.objecter _finish_command 15 = system:0 2024-04-02T04:49:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.030+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 4 2024-04-02T04:49:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.030+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 7 15 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 10 client.25055.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign -1 16 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 20 client.25055.objecter _get_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_remove -1 16 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 20 client.25055.objecter get_session s=0x7fbc841b6c70 osd=7 3 2024-04-02T04:49:23.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 15 client.25055.objecter _session_command_op_assign 7 16 2024-04-02T04:49:23.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 4 2024-04-02T04:49:23.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 10 client.25055.objecter _send_command 16 2024-04-02T04:49:23.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.046+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbc841b7ac0 con 0x7fbc841b8050 2024-04-02T04:49:23.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc88809700 1 -- 172.21.15.69:0/1293974878 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc841b7ac0 con 0x7fbc841b8050 2024-04-02T04:49:23.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc88809700 10 client.25055.objecter ms_dispatch 0x7fbc84001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:23.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc88809700 10 client.25055.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:23.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc88809700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 4 2024-04-02T04:49:23.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc88809700 15 client.25055.objecter _session_command_op_remove 7 16 2024-04-02T04:49:23.051 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:49:23.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 librados: watch_flush enter 2024-04-02T04:49:23.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 librados: watch_flush exit 2024-04-02T04:49:23.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.0 2024-04-02T04:49:23.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbc840630a0 msgr2=0x7fbc841090d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbc840630a0 0x7fbc841090d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84103ad0 osd=0 1 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.1 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fbc84062c90 msgr2=0x7fbc841aaf20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fbc84062c90 0x7fbc841aaf20 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84062b20 osd=1 1 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.2 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fbc8402d690 msgr2=0x7fbc841ae5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fbc8402d690 0x7fbc841ae5f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc84109e70 osd=2 1 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.3 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fbc841afcb0 msgr2=0x7fbc841b00f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc837fe700 1 -- 172.21.15.69:0/1293974878 reap_dead start 2024-04-02T04:49:23.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fbc841afcb0 0x7fbc841b00f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841abc30 osd=3 1 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.4 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fbc841b1010 msgr2=0x7fbc841b5620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fbc841b1010 0x7fbc841b5620 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b0630 osd=4 1 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.5 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fbc841b6280 msgr2=0x7fbc841b66e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fbc841b6280 0x7fbc841b66e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b5b60 osd=5 1 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.6 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fbc841bac00 msgr2=0x7fbc841bb060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fbc841bac00 0x7fbc841bb060 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841bb8c0 osd=6 1 2024-04-02T04:49:23.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter close_session for osd.7 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbc841b8050 msgr2=0x7fbc841b73d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fbc841b8050 0x7fbc841b73d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter put_session s=0x7fbc841b6c70 osd=7 1 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 20 client.25055.objecter shutdown clearing up homeless session... 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 10 client.25055.objecter successfully canceled tick 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc837fe700 1 -- 172.21.15.69:0/1293974878 reap_dead start 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbc6c06f5e0 msgr2=0x7fbc6c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbc6c06f5e0 0x7fbc6c071aa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc74005950 tx=0x7fbc7400b410 comp rx=0 tx=0).stop 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 msgr2=0x7fbc84101ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84101ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800bff0 tx=0x7fbc78009f50 comp rx=0 tx=0).stop 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 shutdown_connections 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc841013e0 0x7fbc84101ef0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 --2- 172.21.15.69:0/1293974878 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbc6c06f5e0 0x7fbc6c071aa0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 >> 172.21.15.69:0/1293974878 conn(0x7fbc84077740 msgr2=0x7fbc840feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:23.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 shutdown_connections 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.050+0000 7fbc8980b700 1 -- 172.21.15.69:0/1293974878 wait complete. 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:23.054+0000 7fbc8980b700 1 librados: shutdown 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:49:23.056 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:49:23.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:23.105 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:49:22 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:49:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:23.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:23.249 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:23.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:23.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:23.509 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:23.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:23.635 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:23.738 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:23.759 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:23.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:23.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:23.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:24.017 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:24.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:24.139 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:24.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:24.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-02T04:49:24.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:24.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:24.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:24.549 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:24.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:24.681 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:24.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:24.809 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:24.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:24.932 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:24.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:25.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:25.083 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:25.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:12.682838+0000 osd.4 (osd.4) 135 : cluster 0 4.13 scrub starts 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:12.684903+0000 osd.4 (osd.4) 136 : cluster 0 4.13 scrub ok 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:12.972461+0000 mgr.y (mgr.14538) 199 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:12.991854+0000 osd.0 (osd.0) 133 : cluster 0 4.b deep-scrub starts 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:12.993515+0000 osd.0 (osd.0) 134 : cluster 0 4.b deep-scrub ok 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.056518+0000 osd.1 (osd.1) 125 : cluster 0 2.4 scrub starts 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.057817+0000 osd.1 (osd.1) 126 : cluster 0 2.4 scrub ok 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.635168+0000 osd.4 (osd.4) 137 : cluster 0 4.1 deep-scrub starts 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.637509+0000 osd.4 (osd.4) 138 : cluster 0 4.1 deep-scrub ok 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.721553+0000 osd.2 (osd.2) 71 : cluster 0 4.1c scrub starts 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.722595+0000 osd.2 (osd.2) 72 : cluster 0 4.1c scrub ok 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:13.837297+0000 mgr.y (mgr.14538) 200 : cluster 0 pgmap v168: 196 pgs: 53 creating+peering, 132 active+clean, 11 unknown; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:25.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:14.776282+0000 osd.6 (osd.6) 93 : cluster 0 2.1a scrub starts 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:14.778559+0000 osd.6 (osd.6) 94 : cluster 0 2.1a scrub ok 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:14.986268+0000 osd.0 (osd.0) 135 : cluster 0 3.12 scrub starts 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:14.989369+0000 osd.0 (osd.0) 136 : cluster 0 3.12 scrub ok 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.012207+0000 osd.1 (osd.1) 127 : cluster 0 2.9 scrub starts 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.013753+0000 osd.1 (osd.1) 128 : cluster 0 2.9 scrub ok 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.075765+0000 osd.7 (osd.7) 135 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.075898+0000 osd.7 (osd.7) 136 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:15.078441+0000 mgr.y (mgr.14538) 201 : audit 0 from='client.24950 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.006689+0000 mon.b (mon.1) 111 : audit 1 from='client.? 172.21.15.69:0/2510353129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.008440+0000 mon.b (mon.1) 112 : audit 1 from='client.? 172.21.15.69:0/3321437036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.012509+0000 mon.b (mon.1) 113 : audit 1 from='client.? 172.21.15.69:0/775377320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.027063+0000 mon.c (mon.2) 35 : audit 1 from='client.? 172.21.15.69:0/3050184937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.028307+0000 mon.c (mon.2) 36 : audit 1 from='client.? 172.21.15.69:0/3467044255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.028502+0000 mon.c (mon.2) 37 : audit 1 from='client.? 172.21.15.69:0/2668378935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.032608+0000 mon.c (mon.2) 38 : audit 1 from='client.? 172.21.15.69:0/3075602668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.032838+0000 mon.c (mon.2) 39 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.033016+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:16.308183+0000 mon.c (mon.2) 41 : audit 1 from='client.? 172.21.15.69:0/1573734000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:18.344462+0000 osd.3 (osd.3) 163 : cluster 0 4.10 scrub starts 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:18.346032+0000 osd.3 (osd.3) 164 : cluster 0 4.10 scrub ok 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:18.553586+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.120255+0000 mon.a (mon.0) 962 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.121092+0000 mon.a (mon.0) 963 : cluster 3 Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:21.144354+0000 mon.a (mon.0) 964 : audit 1 from='client.? 172.21.15.69:0/1415417424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-72145-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:21.149391+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:24 smithi098 bash[23790]: audit 2024-04-02T04:49:21.149730+0000 mon.a (mon.0) 966 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:12.682838+0000 osd.4 (osd.4) 135 : cluster 0 4.13 scrub starts 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:12.684903+0000 osd.4 (osd.4) 136 : cluster 0 4.13 scrub ok 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:12.972461+0000 mgr.y (mgr.14538) 199 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:12.991854+0000 osd.0 (osd.0) 133 : cluster 0 4.b deep-scrub starts 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:12.993515+0000 osd.0 (osd.0) 134 : cluster 0 4.b deep-scrub ok 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.056518+0000 osd.1 (osd.1) 125 : cluster 0 2.4 scrub starts 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.057817+0000 osd.1 (osd.1) 126 : cluster 0 2.4 scrub ok 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.635168+0000 osd.4 (osd.4) 137 : cluster 0 4.1 deep-scrub starts 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.637509+0000 osd.4 (osd.4) 138 : cluster 0 4.1 deep-scrub ok 2024-04-02T04:49:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.721553+0000 osd.2 (osd.2) 71 : cluster 0 4.1c scrub starts 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.722595+0000 osd.2 (osd.2) 72 : cluster 0 4.1c scrub ok 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:13.837297+0000 mgr.y (mgr.14538) 200 : cluster 0 pgmap v168: 196 pgs: 53 creating+peering, 132 active+clean, 11 unknown; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:14.776282+0000 osd.6 (osd.6) 93 : cluster 0 2.1a scrub starts 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:14.778559+0000 osd.6 (osd.6) 94 : cluster 0 2.1a scrub ok 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:14.986268+0000 osd.0 (osd.0) 135 : cluster 0 3.12 scrub starts 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:14.989369+0000 osd.0 (osd.0) 136 : cluster 0 3.12 scrub ok 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.012207+0000 osd.1 (osd.1) 127 : cluster 0 2.9 scrub starts 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.013753+0000 osd.1 (osd.1) 128 : cluster 0 2.9 scrub ok 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.075765+0000 osd.7 (osd.7) 135 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.075898+0000 osd.7 (osd.7) 136 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:15.078441+0000 mgr.y (mgr.14538) 201 : audit 0 from='client.24950 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.006689+0000 mon.b (mon.1) 111 : audit 1 from='client.? 172.21.15.69:0/2510353129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.008440+0000 mon.b (mon.1) 112 : audit 1 from='client.? 172.21.15.69:0/3321437036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.012509+0000 mon.b (mon.1) 113 : audit 1 from='client.? 172.21.15.69:0/775377320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:12.682838+0000 osd.4 (osd.4) 135 : cluster 0 4.13 scrub starts 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:12.684903+0000 osd.4 (osd.4) 136 : cluster 0 4.13 scrub ok 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:12.972461+0000 mgr.y (mgr.14538) 199 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:12.991854+0000 osd.0 (osd.0) 133 : cluster 0 4.b deep-scrub starts 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:12.993515+0000 osd.0 (osd.0) 134 : cluster 0 4.b deep-scrub ok 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.056518+0000 osd.1 (osd.1) 125 : cluster 0 2.4 scrub starts 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.057817+0000 osd.1 (osd.1) 126 : cluster 0 2.4 scrub ok 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.635168+0000 osd.4 (osd.4) 137 : cluster 0 4.1 deep-scrub starts 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.637509+0000 osd.4 (osd.4) 138 : cluster 0 4.1 deep-scrub ok 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.721553+0000 osd.2 (osd.2) 71 : cluster 0 4.1c scrub starts 2024-04-02T04:49:25.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.722595+0000 osd.2 (osd.2) 72 : cluster 0 4.1c scrub ok 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:13.837297+0000 mgr.y (mgr.14538) 200 : cluster 0 pgmap v168: 196 pgs: 53 creating+peering, 132 active+clean, 11 unknown; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:14.776282+0000 osd.6 (osd.6) 93 : cluster 0 2.1a scrub starts 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:14.778559+0000 osd.6 (osd.6) 94 : cluster 0 2.1a scrub ok 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:14.986268+0000 osd.0 (osd.0) 135 : cluster 0 3.12 scrub starts 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:14.989369+0000 osd.0 (osd.0) 136 : cluster 0 3.12 scrub ok 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.012207+0000 osd.1 (osd.1) 127 : cluster 0 2.9 scrub starts 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.013753+0000 osd.1 (osd.1) 128 : cluster 0 2.9 scrub ok 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.075765+0000 osd.7 (osd.7) 135 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.075898+0000 osd.7 (osd.7) 136 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:15.078441+0000 mgr.y (mgr.14538) 201 : audit 0 from='client.24950 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.006689+0000 mon.b (mon.1) 111 : audit 1 from='client.? 172.21.15.69:0/2510353129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.008440+0000 mon.b (mon.1) 112 : audit 1 from='client.? 172.21.15.69:0/3321437036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.012509+0000 mon.b (mon.1) 113 : audit 1 from='client.? 172.21.15.69:0/775377320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.027063+0000 mon.c (mon.2) 35 : audit 1 from='client.? 172.21.15.69:0/3050184937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.028307+0000 mon.c (mon.2) 36 : audit 1 from='client.? 172.21.15.69:0/3467044255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.028502+0000 mon.c (mon.2) 37 : audit 1 from='client.? 172.21.15.69:0/2668378935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.032608+0000 mon.c (mon.2) 38 : audit 1 from='client.? 172.21.15.69:0/3075602668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.032838+0000 mon.c (mon.2) 39 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.033016+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:16.308183+0000 mon.c (mon.2) 41 : audit 1 from='client.? 172.21.15.69:0/1573734000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:18.344462+0000 osd.3 (osd.3) 163 : cluster 0 4.10 scrub starts 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:18.346032+0000 osd.3 (osd.3) 164 : cluster 0 4.10 scrub ok 2024-04-02T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:18.553586+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.120255+0000 mon.a (mon.0) 962 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.121092+0000 mon.a (mon.0) 963 : cluster 3 Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:21.144354+0000 mon.a (mon.0) 964 : audit 1 from='client.? 172.21.15.69:0/1415417424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-72145-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:21.149391+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[21528]: audit 2024-04-02T04:49:21.149730+0000 mon.a (mon.0) 966 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.027063+0000 mon.c (mon.2) 35 : audit 1 from='client.? 172.21.15.69:0/3050184937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.028307+0000 mon.c (mon.2) 36 : audit 1 from='client.? 172.21.15.69:0/3467044255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.028502+0000 mon.c (mon.2) 37 : audit 1 from='client.? 172.21.15.69:0/2668378935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.032608+0000 mon.c (mon.2) 38 : audit 1 from='client.? 172.21.15.69:0/3075602668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.032838+0000 mon.c (mon.2) 39 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.033016+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:16.308183+0000 mon.c (mon.2) 41 : audit 1 from='client.? 172.21.15.69:0/1573734000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:18.344462+0000 osd.3 (osd.3) 163 : cluster 0 4.10 scrub starts 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:18.346032+0000 osd.3 (osd.3) 164 : cluster 0 4.10 scrub ok 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:18.553586+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.120255+0000 mon.a (mon.0) 962 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.121092+0000 mon.a (mon.0) 963 : cluster 3 Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:21.144354+0000 mon.a (mon.0) 964 : audit 1 from='client.? 172.21.15.69:0/1415417424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-72145-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:21.149391+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:24 smithi069 bash[28904]: audit 2024-04-02T04:49:21.149730+0000 mon.a (mon.0) 966 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:25.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:25.339 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:25.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:25.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:25.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:25.591 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:25.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:25.717 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:25.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:25.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:25.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:25.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:25.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:25.994+0000 7efc20c9d700 10 client.24971.objecter tick 2024-04-02T04:49:26.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.022+0000 7efc03fff700 1 -- 172.21.15.69:0/1573734000 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v138) ==== 87+0+0 (secure 0 0 0) 0x7efc100610b0 con 0x7efc1c101150 2024-04-02T04:49:26.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.026+0000 7efc2149e700 10 librados: watch_flush enter 2024-04-02T04:49:26.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.030+0000 7efc2149e700 10 librados: watch_flush exit 2024-04-02T04:49:26.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.030+0000 7efc2149e700 20 client.24971.objecter shutdown clearing up homeless session... 2024-04-02T04:49:26.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.030+0000 7efc2149e700 10 client.24971.objecter successfully canceled tick 2024-04-02T04:49:26.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efc0406f5e0 msgr2=0x7efc04071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:26.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efc0406f5e0 0x7efc04071aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc0c005950 tx=0x7efc0c00b500 comp rx=0 tx=0).stop 2024-04-02T04:49:26.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 msgr2=0x7efc1c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:26.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 0x7efc1c198610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc100098e0 tx=0x7efc10005e80 comp rx=0 tx=0).stop 2024-04-02T04:49:26.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 shutdown_connections 2024-04-02T04:49:26.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efc0406f5e0 0x7efc04071aa0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc0c005950 tx=0x7efc0c00b500 comp rx=0 tx=0).stop 2024-04-02T04:49:26.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc1c107260 0x7efc1c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:26.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c100790 0x7efc1c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:26.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 --2- 172.21.15.69:0/1573734000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc1c101150 0x7efc1c198610 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:26.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 >> 172.21.15.69:0/1573734000 conn(0x7efc1c077740 msgr2=0x7efc1c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:26.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 shutdown_connections 2024-04-02T04:49:26.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 -- 172.21.15.69:0/1573734000 wait complete. 2024-04-02T04:49:26.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:26.034+0000 7efc2149e700 1 librados: shutdown 2024-04-02T04:49:26.039 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:49:26.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:26.085 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.753807+0000 osd.6 (osd.6) 95 : cluster 0 5.2 scrub starts 2024-04-02T04:49:26.085 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.756704+0000 osd.6 (osd.6) 96 : cluster 0 5.2 scrub ok 2024-04-02T04:49:26.085 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.787281+0000 osd.5 (osd.5) 127 : cluster 0 3.5 scrub starts 2024-04-02T04:49:26.085 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.791270+0000 osd.5 (osd.5) 128 : cluster 0 3.5 scrub ok 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:15.837769+0000 mgr.y (mgr.14538) 202 : cluster 0 pgmap v169: 196 pgs: 53 creating+peering, 132 active+clean, 11 unknown; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:16.088529+0000 osd.1 (osd.1) 129 : cluster 0 4.2 scrub starts 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:16.090218+0000 osd.1 (osd.1) 130 : cluster 0 4.2 scrub ok 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:16.601336+0000 osd.4 (osd.4) 139 : cluster 0 4.9 scrub starts 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:16.603440+0000 osd.4 (osd.4) 140 : cluster 0 4.9 scrub ok 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:17.092588+0000 osd.1 (osd.1) 131 : cluster 0 4.4 scrub starts 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:17.093480+0000 osd.1 (osd.1) 132 : cluster 0 4.4 scrub ok 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:17.841550+0000 mgr.y (mgr.14538) 203 : cluster 0 pgmap v171: 580 pgs: 401 creating+peering, 132 active+clean, 47 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:18.088701+0000 osd.1 (osd.1) 133 : cluster 0 3.4 deep-scrub starts 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:18.095879+0000 osd.1 (osd.1) 134 : cluster 0 3.4 deep-scrub ok 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:19.090248+0000 osd.1 (osd.1) 135 : cluster 0 6.4 scrub starts 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:19.092558+0000 osd.1 (osd.1) 136 : cluster 0 6.4 scrub ok 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:19.585832+0000 osd.2 (osd.2) 73 : cluster 0 4.f scrub starts 2024-04-02T04:49:26.086 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:19.587984+0000 osd.2 (osd.2) 74 : cluster 0 4.f scrub ok 2024-04-02T04:49:26.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:19.842242+0000 mgr.y (mgr.14538) 204 : cluster 0 pgmap v172: 580 pgs: 401 creating+peering, 132 active+clean, 47 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:49:26.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.054456+0000 osd.1 (osd.1) 137 : cluster 0 2.6 scrub starts 2024-04-02T04:49:26.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.056547+0000 osd.1 (osd.1) 138 : cluster 0 2.6 scrub ok 2024-04-02T04:49:26.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.150306+0000 mon.a (mon.0) 967 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:26.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.150387+0000 mon.a (mon.0) 968 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:26.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.161912+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.69:0/28671705' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.163771+0000 mon.c (mon.2) 44 : audit 1 from='client.? 172.21.15.69:0/2202278946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.164744+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.69:0/271762309' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.166184+0000 mon.c (mon.2) 46 : audit 1 from='client.? 172.21.15.69:0/242164296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.167111+0000 mon.c (mon.2) 47 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.167846+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.69:0/3334667875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:21.169630+0000 mon.b (mon.1) 114 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.843261+0000 mgr.y (mgr.14538) 205 : cluster 0 pgmap v174: 964 pgs: 401 creating+peering, 132 active+clean, 431 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.847494+0000 osd.0 (osd.0) 137 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:21.847571+0000 osd.0 (osd.0) 138 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:22.450273+0000 osd.7 (osd.7) 137 : cluster 0 2.1d scrub starts 2024-04-02T04:49:26.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:22.452690+0000 osd.7 (osd.7) 138 : cluster 0 2.1d scrub ok 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:22.825813+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.69:0/1293974878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:22.982777+0000 mgr.y (mgr.14538) 206 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:23.487469+0000 osd.7 (osd.7) 139 : cluster 0 3.15 scrub starts 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:23.491057+0000 osd.7 (osd.7) 140 : cluster 0 3.15 scrub ok 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:23.851021+0000 mgr.y (mgr.14538) 207 : cluster 0 pgmap v175: 964 pgs: 774 creating+peering, 132 active+clean, 58 unknown; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:24.267198+0000 osd.3 (osd.3) 165 : cluster 0 3.8 scrub starts 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:24.269990+0000 osd.3 (osd.3) 166 : cluster 0 3.8 scrub ok 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:24.947754+0000 mon.a (mon.0) 969 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.959751+0000 mon.a (mon.0) 970 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.962944+0000 mon.a (mon.0) 971 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.971716+0000 mon.a (mon.0) 972 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.972074+0000 mon.a (mon.0) 973 : audit 1 from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.972622+0000 mon.a (mon.0) 974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.089 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.972787+0000 mon.a (mon.0) 975 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.974025+0000 mon.a (mon.0) 976 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.974284+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.975008+0000 mon.a (mon.0) 978 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.975680+0000 mon.a (mon.0) 979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.976378+0000 mon.a (mon.0) 980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.976584+0000 mon.a (mon.0) 981 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.976841+0000 mon.a (mon.0) 982 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.976926+0000 mon.a (mon.0) 983 : audit 1 from='client.? 172.21.15.69:0/1415417424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-72145-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.976991+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.977059+0000 mon.a (mon.0) 985 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:24.988043+0000 mon.a (mon.0) 986 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.991222+0000 mon.a (mon.0) 987 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.991687+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:26.090 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.992131+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.992493+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:24.992973+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.008498+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.018329+0000 mon.a (mon.0) 992 : audit 1 from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.020243+0000 mon.c (mon.2) 51 : audit 1 from='client.? 172.21.15.69:0/1522194686' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.025633+0000 mon.a (mon.0) 993 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.032745+0000 mon.b (mon.1) 115 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:25.282928+0000 osd.3 (osd.3) 167 : cluster 0 6.13 scrub starts 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: cluster 2024-04-02T04:49:25.284910+0000 osd.3 (osd.3) 168 : cluster 0 6.13 scrub ok 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.468877+0000 mon.c (mon.2) 52 : audit 0 from='client.? 172.21.15.69:0/3577111494' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.877044+0000 mon.a (mon.0) 994 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:26.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[21528]: audit 2024-04-02T04:49:25.877396+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.753807+0000 osd.6 (osd.6) 95 : cluster 0 5.2 scrub starts 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.756704+0000 osd.6 (osd.6) 96 : cluster 0 5.2 scrub ok 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.787281+0000 osd.5 (osd.5) 127 : cluster 0 3.5 scrub starts 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.791270+0000 osd.5 (osd.5) 128 : cluster 0 3.5 scrub ok 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:15.837769+0000 mgr.y (mgr.14538) 202 : cluster 0 pgmap v169: 196 pgs: 53 creating+peering, 132 active+clean, 11 unknown; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:16.088529+0000 osd.1 (osd.1) 129 : cluster 0 4.2 scrub starts 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:16.090218+0000 osd.1 (osd.1) 130 : cluster 0 4.2 scrub ok 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:16.601336+0000 osd.4 (osd.4) 139 : cluster 0 4.9 scrub starts 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:16.603440+0000 osd.4 (osd.4) 140 : cluster 0 4.9 scrub ok 2024-04-02T04:49:26.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:17.092588+0000 osd.1 (osd.1) 131 : cluster 0 4.4 scrub starts 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:17.093480+0000 osd.1 (osd.1) 132 : cluster 0 4.4 scrub ok 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:17.841550+0000 mgr.y (mgr.14538) 203 : cluster 0 pgmap v171: 580 pgs: 401 creating+peering, 132 active+clean, 47 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:18.088701+0000 osd.1 (osd.1) 133 : cluster 0 3.4 deep-scrub starts 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:18.095879+0000 osd.1 (osd.1) 134 : cluster 0 3.4 deep-scrub ok 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:19.090248+0000 osd.1 (osd.1) 135 : cluster 0 6.4 scrub starts 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:19.092558+0000 osd.1 (osd.1) 136 : cluster 0 6.4 scrub ok 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:19.585832+0000 osd.2 (osd.2) 73 : cluster 0 4.f scrub starts 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:19.587984+0000 osd.2 (osd.2) 74 : cluster 0 4.f scrub ok 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:19.842242+0000 mgr.y (mgr.14538) 204 : cluster 0 pgmap v172: 580 pgs: 401 creating+peering, 132 active+clean, 47 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.054456+0000 osd.1 (osd.1) 137 : cluster 0 2.6 scrub starts 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.056547+0000 osd.1 (osd.1) 138 : cluster 0 2.6 scrub ok 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.150306+0000 mon.a (mon.0) 967 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.150387+0000 mon.a (mon.0) 968 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:26.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.161912+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.69:0/28671705' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.163771+0000 mon.c (mon.2) 44 : audit 1 from='client.? 172.21.15.69:0/2202278946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.164744+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.69:0/271762309' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.166184+0000 mon.c (mon.2) 46 : audit 1 from='client.? 172.21.15.69:0/242164296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.167111+0000 mon.c (mon.2) 47 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.167846+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.69:0/3334667875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:21.169630+0000 mon.b (mon.1) 114 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.843261+0000 mgr.y (mgr.14538) 205 : cluster 0 pgmap v174: 964 pgs: 401 creating+peering, 132 active+clean, 431 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.847494+0000 osd.0 (osd.0) 137 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:21.847571+0000 osd.0 (osd.0) 138 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:22.450273+0000 osd.7 (osd.7) 137 : cluster 0 2.1d scrub starts 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:22.452690+0000 osd.7 (osd.7) 138 : cluster 0 2.1d scrub ok 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:22.825813+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.69:0/1293974878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:22.982777+0000 mgr.y (mgr.14538) 206 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:26.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:23.487469+0000 osd.7 (osd.7) 139 : cluster 0 3.15 scrub starts 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:23.491057+0000 osd.7 (osd.7) 140 : cluster 0 3.15 scrub ok 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:23.851021+0000 mgr.y (mgr.14538) 207 : cluster 0 pgmap v175: 964 pgs: 774 creating+peering, 132 active+clean, 58 unknown; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:24.267198+0000 osd.3 (osd.3) 165 : cluster 0 3.8 scrub starts 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:24.269990+0000 osd.3 (osd.3) 166 : cluster 0 3.8 scrub ok 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:24.947754+0000 mon.a (mon.0) 969 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.959751+0000 mon.a (mon.0) 970 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.962944+0000 mon.a (mon.0) 971 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.971716+0000 mon.a (mon.0) 972 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.972074+0000 mon.a (mon.0) 973 : audit 1 from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.972622+0000 mon.a (mon.0) 974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.972787+0000 mon.a (mon.0) 975 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.974025+0000 mon.a (mon.0) 976 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.974284+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.975008+0000 mon.a (mon.0) 978 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.975680+0000 mon.a (mon.0) 979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.976378+0000 mon.a (mon.0) 980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.976584+0000 mon.a (mon.0) 981 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.976841+0000 mon.a (mon.0) 982 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.976926+0000 mon.a (mon.0) 983 : audit 1 from='client.? 172.21.15.69:0/1415417424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-72145-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.976991+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.977059+0000 mon.a (mon.0) 985 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:24.988043+0000 mon.a (mon.0) 986 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.991222+0000 mon.a (mon.0) 987 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.991687+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.992131+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.992493+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:24.992973+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.008498+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.018329+0000 mon.a (mon.0) 992 : audit 1 from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.020243+0000 mon.c (mon.2) 51 : audit 1 from='client.? 172.21.15.69:0/1522194686' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.025633+0000 mon.a (mon.0) 993 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.032745+0000 mon.b (mon.1) 115 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:25.282928+0000 osd.3 (osd.3) 167 : cluster 0 6.13 scrub starts 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: cluster 2024-04-02T04:49:25.284910+0000 osd.3 (osd.3) 168 : cluster 0 6.13 scrub ok 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.468877+0000 mon.c (mon.2) 52 : audit 0 from='client.? 172.21.15.69:0/3577111494' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.877044+0000 mon.a (mon.0) 994 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:26.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:25 smithi069 bash[28904]: audit 2024-04-02T04:49:25.877396+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.112 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:26.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:26.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.753807+0000 osd.6 (osd.6) 95 : cluster 0 5.2 scrub starts 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.756704+0000 osd.6 (osd.6) 96 : cluster 0 5.2 scrub ok 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.787281+0000 osd.5 (osd.5) 127 : cluster 0 3.5 scrub starts 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.791270+0000 osd.5 (osd.5) 128 : cluster 0 3.5 scrub ok 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:15.837769+0000 mgr.y (mgr.14538) 202 : cluster 0 pgmap v169: 196 pgs: 53 creating+peering, 132 active+clean, 11 unknown; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:16.088529+0000 osd.1 (osd.1) 129 : cluster 0 4.2 scrub starts 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:16.090218+0000 osd.1 (osd.1) 130 : cluster 0 4.2 scrub ok 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:16.601336+0000 osd.4 (osd.4) 139 : cluster 0 4.9 scrub starts 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:16.603440+0000 osd.4 (osd.4) 140 : cluster 0 4.9 scrub ok 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:17.092588+0000 osd.1 (osd.1) 131 : cluster 0 4.4 scrub starts 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:17.093480+0000 osd.1 (osd.1) 132 : cluster 0 4.4 scrub ok 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:17.841550+0000 mgr.y (mgr.14538) 203 : cluster 0 pgmap v171: 580 pgs: 401 creating+peering, 132 active+clean, 47 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:18.088701+0000 osd.1 (osd.1) 133 : cluster 0 3.4 deep-scrub starts 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:18.095879+0000 osd.1 (osd.1) 134 : cluster 0 3.4 deep-scrub ok 2024-04-02T04:49:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:19.090248+0000 osd.1 (osd.1) 135 : cluster 0 6.4 scrub starts 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:19.092558+0000 osd.1 (osd.1) 136 : cluster 0 6.4 scrub ok 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:19.585832+0000 osd.2 (osd.2) 73 : cluster 0 4.f scrub starts 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:19.587984+0000 osd.2 (osd.2) 74 : cluster 0 4.f scrub ok 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:19.842242+0000 mgr.y (mgr.14538) 204 : cluster 0 pgmap v172: 580 pgs: 401 creating+peering, 132 active+clean, 47 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.054456+0000 osd.1 (osd.1) 137 : cluster 0 2.6 scrub starts 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.056547+0000 osd.1 (osd.1) 138 : cluster 0 2.6 scrub ok 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.150306+0000 mon.a (mon.0) 967 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi069-72950-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.150387+0000 mon.a (mon.0) 968 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi069-72924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.161912+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.69:0/28671705' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.163771+0000 mon.c (mon.2) 44 : audit 1 from='client.? 172.21.15.69:0/2202278946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.164744+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.69:0/271762309' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.166184+0000 mon.c (mon.2) 46 : audit 1 from='client.? 172.21.15.69:0/242164296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.167111+0000 mon.c (mon.2) 47 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.167846+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.69:0/3334667875' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:21.169630+0000 mon.b (mon.1) 114 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.843261+0000 mgr.y (mgr.14538) 205 : cluster 0 pgmap v174: 964 pgs: 401 creating+peering, 132 active+clean, 431 unknown; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.847494+0000 osd.0 (osd.0) 137 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:26.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:21.847571+0000 osd.0 (osd.0) 138 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:22.450273+0000 osd.7 (osd.7) 137 : cluster 0 2.1d scrub starts 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:22.452690+0000 osd.7 (osd.7) 138 : cluster 0 2.1d scrub ok 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:22.825813+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.69:0/1293974878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:22.982777+0000 mgr.y (mgr.14538) 206 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:23.487469+0000 osd.7 (osd.7) 139 : cluster 0 3.15 scrub starts 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:23.491057+0000 osd.7 (osd.7) 140 : cluster 0 3.15 scrub ok 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:23.851021+0000 mgr.y (mgr.14538) 207 : cluster 0 pgmap v175: 964 pgs: 774 creating+peering, 132 active+clean, 58 unknown; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:24.267198+0000 osd.3 (osd.3) 165 : cluster 0 3.8 scrub starts 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:24.269990+0000 osd.3 (osd.3) 166 : cluster 0 3.8 scrub ok 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:24.947754+0000 mon.a (mon.0) 969 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.959751+0000 mon.a (mon.0) 970 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.962944+0000 mon.a (mon.0) 971 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.971716+0000 mon.a (mon.0) 972 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.972074+0000 mon.a (mon.0) 973 : audit 1 from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.972622+0000 mon.a (mon.0) 974 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.972787+0000 mon.a (mon.0) 975 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.974025+0000 mon.a (mon.0) 976 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.974284+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.975008+0000 mon.a (mon.0) 978 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.975680+0000 mon.a (mon.0) 979 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.976378+0000 mon.a (mon.0) 980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.976584+0000 mon.a (mon.0) 981 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.976841+0000 mon.a (mon.0) 982 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-72153-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.976926+0000 mon.a (mon.0) 983 : audit 1 from='client.? 172.21.15.69:0/1415417424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-72145-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.976991+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-72349-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.977059+0000 mon.a (mon.0) 985 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-72161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:24.988043+0000 mon.a (mon.0) 986 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-04-02T04:49:26.228 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.991222+0000 mon.a (mon.0) 987 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.991687+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.992131+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.992493+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:24.992973+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.008498+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.018329+0000 mon.a (mon.0) 992 : audit 1 from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.020243+0000 mon.c (mon.2) 51 : audit 1 from='client.? 172.21.15.69:0/1522194686' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.025633+0000 mon.a (mon.0) 993 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.032745+0000 mon.b (mon.1) 115 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:25.282928+0000 osd.3 (osd.3) 167 : cluster 0 6.13 scrub starts 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: cluster 2024-04-02T04:49:25.284910+0000 osd.3 (osd.3) 168 : cluster 0 6.13 scrub ok 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.468877+0000 mon.c (mon.2) 52 : audit 0 from='client.? 172.21.15.69:0/3577111494' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.877044+0000 mon.a (mon.0) 994 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:26.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:25 smithi098 bash[23790]: audit 2024-04-02T04:49:25.877396+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:26.284 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:26.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:26.431 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment set-up. 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (0 ms) 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (25 ms) 2024-04-02T04:49:26.439 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (16 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (11 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (6 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (11 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (3 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (25 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (15 ms) 2024-04-02T04:49:26.440 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-02T04:49:26.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (4 ms) 2024-04-02T04:49:26.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados (118 ms total) 2024-04-02T04:49:26.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-04-02T04:49:26.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment tear-down 2024-04-02T04:49:26.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (16437 ms total) 2024-04-02T04:49:26.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-02T04:49:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:26.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:26.617 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:26.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:26.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:26.867 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:26.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:26.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment set-up. 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjects (577 ms) 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (43 ms) 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2024-04-02T04:49:26.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2024-04-02T04:49:26.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (155 ms) 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2024-04-02T04:49:26.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (85 ms) 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: x cursor=MIN 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-04-02T04:49:26.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-04-02T04:49:26.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:566253c9:::13:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-04-02T04:49:26.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-04-02T04:49:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:26.994 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:26.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:27.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=14 cursor=13:62a1935d:::14:head 2024-04-02T04:49:27.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-04-02T04:49:27.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:863748b0:::15:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-04-02T04:49:27.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-04-02T04:49:27.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:e960b815:::9:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=10 expected=10 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:e960b815:::9:head 2024-04-02T04:49:27.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=9 expected=9 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=2 expected=2 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=1 expected=1 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-04-02T04:49:27.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=8 expected=8 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=5 expected=5 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=11 expected=11 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:863748b0:::15:head 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-04-02T04:49:27.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-04-02T04:49:27.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:27.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:27.251 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:27.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:27.376 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:27.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:27 smithi069 bash[21528]: cluster 2024-04-02T04:49:25.995379+0000 mon.a (mon.0) 996 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:49:27.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:27 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002012+0000 mon.a (mon.0) 997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:27.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:27 smithi069 bash[28904]: cluster 2024-04-02T04:49:25.995379+0000 mon.a (mon.0) 996 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:49:27.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:27 smithi098 bash[23790]: cluster 2024-04-02T04:49:25.995379+0000 mon.a (mon.0) 996 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:49:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:27 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002012+0000 mon.a (mon.0) 997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:27.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:27.508 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:27.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:27.633 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:27.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:27.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:27 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002012+0000 mon.a (mon.0) 997 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-72169-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:27.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : en list_parallel: process_1_[72677]: starting. 2024-04-02T04:49:27.784 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[72677]: creating pool ceph_test_rados_list_parallel.smithi069-72664 2024-04-02T04:49:27.784 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[72677]: created object 0... 2024-04-02T04:49:27.784 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[72677]: created object 25... 2024-04-02T04:49:27.784 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[72677]: created object 49... 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[72677]: finishing. 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[72677]: shutting down. 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[72678]: starting. 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[72678]: listing objects. 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[72678]: listed object 0... 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[72678]: listed object 25... 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[72678]: saw 50 objects 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[72678]: shutting down. 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: starting. 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: creating pool ceph_test_rados_list_parallel.smithi069-72664 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: created object 0... 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: created object 25... 2024-04-02T04:49:27.785 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: created object 49... 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: finishing. 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[75179]: shutting down. 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[75180]: starting. 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[75180]: listing objects. 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[75180]: listed object 0... 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[75180]: listed object 25... 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[75180]: saw 42 objects 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[75180]: shutting down. 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[75181]: starting. 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[75181]: removed 25 objects... 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[75181]: removed half of the objects 2024-04-02T04:49:27.786 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[75181]: removed 50 objects... 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[75181]: removed 50 objects 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[75181]: shutting down. 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: starting. 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: creating pool ceph_test_rados_list_parallel.smithi069-72664 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: created object 0... 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: created object 25... 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: created object 49... 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: finishing. 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[75227]: shutting down. 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: starting. 2024-04-02T04:49:27.787 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: listing objects. 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: listed object 0... 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: listed object 25... 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: listed object 50... 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: listed object 75... 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: saw 78 objects 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[75228]: shutting down. 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[75229]: starting. 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[75229]: added 25 objects... 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[75229]: added half of the objects 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[75229]: added 50 objects... 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[75229]: added 50 objects 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[75229]: shutting down. 2024-04-02T04:49:27.788 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:27.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:27.882 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:27.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:27.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:27.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:28.005 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:28.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:28.137 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:28.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:28.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:28.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:28.390 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:28.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:28.514 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:28.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:28.673 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:28.701 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:28.778 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:28.802 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:28.906 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:28.914 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:49:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:49:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:49:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:28.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: starting. 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: creating pool ceph_test_rados_list_parallel.smithi069-72664 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: created object 0... 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: created object 25... 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: created object 49... 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: finishing. 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[75407]: shutting down. 2024-04-02T04:49:29.003 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: starting. 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: listing objects. 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: listed object 0... 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: listed object 25... 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: listed object 50... 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: listed object 75... 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: saw 99 objects 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[75408]: shutting down. 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.004 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[75415]: starting. 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[75415]: removed 25 objects... 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[75415]: removed half of the objects 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[75415]: removed 50 objects... 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[75415]: removed 50 objects 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[75415]: shutting down. 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[75411]: starting. 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[75411]: added 25 objects... 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[75411]: added half of the objects 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[75411]: added 50 objects... 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[75411]: added 50 objects 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[75411]: shutting down. 2024-04-02T04:49:29.005 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[75409]: starting. 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[75409]: added 25 objects... 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[75409]: added half of the objects 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[75409]: added 50 objects... 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[75409]: added 50 objects 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[75409]: shutting down. 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.006 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: starting. 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: creating pool ceph_test_rados_list_parallel.smithi069-72664 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: created object 0... 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: created object 25... 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: created object 49... 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: finishing. 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[75532]: shutting down. 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: starting. 2024-04-02T04:49:29.007 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listing objects. 2024-04-02T04:49:29.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:29.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:29.181 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:29.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002184+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002313+0000 mon.a (mon.0) 999 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002446+0000 mon.a (mon.0) 1000 : audit 1 from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002556+0000 mon.a (mon.0) 1001 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002721+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002820+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.002942+0000 mon.a (mon.0) 1004 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003054+0000 mon.a (mon.0) 1005 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003174+0000 mon.a (mon.0) 1006 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003330+0000 mon.a (mon.0) 1007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003458+0000 mon.a (mon.0) 1008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003595+0000 mon.a (mon.0) 1009 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003725+0000 mon.a (mon.0) 1010 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003877+0000 mon.a (mon.0) 1011 : audit 1 from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002184+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002313+0000 mon.a (mon.0) 999 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002446+0000 mon.a (mon.0) 1000 : audit 1 from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002556+0000 mon.a (mon.0) 1001 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002721+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002820+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.002942+0000 mon.a (mon.0) 1004 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003054+0000 mon.a (mon.0) 1005 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003174+0000 mon.a (mon.0) 1006 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003330+0000 mon.a (mon.0) 1007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003458+0000 mon.a (mon.0) 1008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003595+0000 mon.a (mon.0) 1009 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003725+0000 mon.a (mon.0) 1010 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003877+0000 mon.a (mon.0) 1011 : audit 1 from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.003988+0000 mon.a (mon.0) 1012 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.004093+0000 mon.a (mon.0) 1013 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:26.026921+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: cluster 2024-04-02T04:49:27.107829+0000 mon.a (mon.0) 1014 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:27.119366+0000 mon.c (mon.2) 54 : audit 1 from='client.? 172.21.15.69:0/1546287619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:27.123253+0000 mon.a (mon.0) 1015 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:27.138830+0000 mon.a (mon.0) 1016 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:27.138952+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:29.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: cluster 2024-04-02T04:49:27.151067+0000 mon.a (mon.0) 1018 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:27.495026+0000 mon.c (mon.2) 55 : audit 0 from='client.? 172.21.15.69:0/3495572229' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[21528]: audit 2024-04-02T04:49:27.590901+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.003988+0000 mon.a (mon.0) 1012 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.004093+0000 mon.a (mon.0) 1013 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:26.026921+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: cluster 2024-04-02T04:49:27.107829+0000 mon.a (mon.0) 1014 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:27.119366+0000 mon.c (mon.2) 54 : audit 1 from='client.? 172.21.15.69:0/1546287619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:27.123253+0000 mon.a (mon.0) 1015 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:27.138830+0000 mon.a (mon.0) 1016 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:27.138952+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: cluster 2024-04-02T04:49:27.151067+0000 mon.a (mon.0) 1018 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:27.495026+0000 mon.c (mon.2) 55 : audit 0 from='client.? 172.21.15.69:0/3495572229' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-02T04:49:29.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:29 smithi069 bash[28904]: audit 2024-04-02T04:49:27.590901+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:29.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:29.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002184+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002313+0000 mon.a (mon.0) 999 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002446+0000 mon.a (mon.0) 1000 : audit 1 from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002556+0000 mon.a (mon.0) 1001 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002721+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002820+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.002942+0000 mon.a (mon.0) 1004 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003054+0000 mon.a (mon.0) 1005 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003174+0000 mon.a (mon.0) 1006 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003330+0000 mon.a (mon.0) 1007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003458+0000 mon.a (mon.0) 1008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003595+0000 mon.a (mon.0) 1009 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003725+0000 mon.a (mon.0) 1010 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003877+0000 mon.a (mon.0) 1011 : audit 1 from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.003988+0000 mon.a (mon.0) 1012 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.004093+0000 mon.a (mon.0) 1013 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:26.026921+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: cluster 2024-04-02T04:49:27.107829+0000 mon.a (mon.0) 1014 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:27.119366+0000 mon.c (mon.2) 54 : audit 1 from='client.? 172.21.15.69:0/1546287619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:27.123253+0000 mon.a (mon.0) 1015 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:27.138830+0000 mon.a (mon.0) 1016 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:27.138952+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: cluster 2024-04-02T04:49:27.151067+0000 mon.a (mon.0) 1018 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:27.495026+0000 mon.c (mon.2) 55 : audit 0 from='client.? 172.21.15.69:0/3495572229' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-02T04:49:29.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:29 smithi098 bash[23790]: audit 2024-04-02T04:49:27.590901+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:29.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:29.433 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:29.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:29.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:29.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listed object 0 api_c_read_operations: Running main() from gmock_main.cc 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (519 ms) 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-02T04:49:29.695 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (11 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (10 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (29 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (12 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (16 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-04-02T04:49:29.696 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (10 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (20 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (19 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (16 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (25 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (7 ms) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (724 ms total) 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-02T04:49:29.697 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (19506 ms total) 2024-04-02T04:49:29.698 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-02T04:49:29.703 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:29.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-02T04:49:29.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-02T04:49:29.718 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (19500 ms) 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (19500 ms total) 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (19500 ms total) 2024-04-02T04:49:29.719 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-02T04:49:29.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (61 ms) 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-02T04:49:29.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (62 ms) 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (19475 ms) 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002184+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-72410-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002313+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-72389-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002446+0000 mon.a [INF] from='client.? 172.21.15.69:0/3685817168' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-72190-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002556+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-72465-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002721+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-72185-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002820+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-72210-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.002942+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003054+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003174+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72472-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003330+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-72423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003458+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003595+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-72642-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003725+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:29.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:29.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:29.986 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:29.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:30.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:30.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:30.222 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:49:30.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:30.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:30.373 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:30.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:30.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:30.631 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:30.730 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:30.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:30.756 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:30.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:30.880 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:30.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:30.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:31.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.136 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:31.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:31.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.392 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:31.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.563 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:31.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:31.693 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:31.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: cluster 2024-04-02T04:49:25.571062+0000 osd.5 (osd.5) 129 : cluster 0 6.7 deep-scrub starts 2024-04-02T04:49:31.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: cluster 2024-04-02T04:49:25.573272+0000 osd.5 (osd.5) 130 : cluster 0 6.7 deep-scrub ok 2024-04-02T04:49:31.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: audit 2024-04-02T04:49:28.592340+0000 mon.c (mon.2) 57 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:31.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: cluster 2024-04-02T04:49:29.108889+0000 mon.a (mon.0) 1019 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-04-02T04:49:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: audit 2024-04-02T04:49:29.132543+0000 mon.a (mon.0) 1020 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: audit 2024-04-02T04:49:29.164364+0000 mon.a (mon.0) 1021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:31 smithi098 bash[23790]: audit 2024-04-02T04:49:29.594254+0000 mon.c (mon.2) 58 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: cluster 2024-04-02T04:49:25.571062+0000 osd.5 (osd.5) 129 : cluster 0 6.7 deep-scrub starts 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: cluster 2024-04-02T04:49:25.573272+0000 osd.5 (osd.5) 130 : cluster 0 6.7 deep-scrub ok 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: audit 2024-04-02T04:49:28.592340+0000 mon.c (mon.2) 57 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: cluster 2024-04-02T04:49:29.108889+0000 mon.a (mon.0) 1019 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: audit 2024-04-02T04:49:29.132543+0000 mon.a (mon.0) 1020 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: audit 2024-04-02T04:49:29.164364+0000 mon.a (mon.0) 1021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[28904]: audit 2024-04-02T04:49:29.594254+0000 mon.c (mon.2) 58 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: cluster 2024-04-02T04:49:25.571062+0000 osd.5 (osd.5) 129 : cluster 0 6.7 deep-scrub starts 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: cluster 2024-04-02T04:49:25.573272+0000 osd.5 (osd.5) 130 : cluster 0 6.7 deep-scrub ok 2024-04-02T04:49:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: audit 2024-04-02T04:49:28.592340+0000 mon.c (mon.2) 57 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:31.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: cluster 2024-04-02T04:49:29.108889+0000 mon.a (mon.0) 1019 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-04-02T04:49:31.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: audit 2024-04-02T04:49:29.132543+0000 mon.a (mon.0) 1020 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:31.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: audit 2024-04-02T04:49:29.164364+0000 mon.a (mon.0) 1021 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:31.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:31 smithi069 bash[21528]: audit 2024-04-02T04:49:29.594254+0000 mon.c (mon.2) 58 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:31.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:31.927 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.947 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: go... 2024-04-02T04:49:31.947 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listed object 25... 2024-04-02T04:49:31.947 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listed object 50... 2024-04-02T04:49:31.947 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listed object 75... 2024-04-02T04:49:31.947 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listed object 100... 2024-04-02T04:49:31.947 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: listed object 125... 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: saw 150 objects 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[75533]: shutting down. 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[75534]: starting. 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[75534]: added 25 objects... 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[75534]: added half of the objects 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[75534]: added 50 objects... 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[75534]: added 50 objects 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[75534]: shutting down. 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.948 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.949 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.949 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-04-02T04:49:31.949 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******* SUCCESS ********** 2024-04-02T04:49:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:31.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:32.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:32.089 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:32.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:32.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:32.351 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:32.392 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-04-02T04:49:32.392 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-02T04:49:32.392 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-02T04:49:32.392 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-02T04:49:32.392 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-02T04:49:32.392 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (5873 ms) 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (5173 ms) 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (11219 ms) 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (22265 ms total) 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (22265 ms total) 2024-04-02T04:49:32.393 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-02T04:49:32.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:32.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:32.605 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:32.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:32.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:32.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:32.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:33.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:33.113 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:33.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:33.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:49:32 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:49:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:33.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:33.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:33.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:33.628 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:33.674 INFO:tasks.workunit.client.0.smithi069.stdout:t: 2024-04-02T04:49:26.003877+0000 mon.a [INF] from='client.? 172.21.15.69:0/1506071398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.003988+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:33.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.004093+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-72378-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:26.026921+0000 mon.c [INF] from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:33.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:27.119366+0000 mon.c [INF] from='client.? 172.21.15.69:0/1546287619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:27.123253+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:27.138830+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi069-72950-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:27.138952+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi069-72924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.132543+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.164364+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.682806+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.682907+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.682983+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.769564+0000 mon.c [INF] from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.779724+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.780588+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.781676+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.783288+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:33.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.784351+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.787335+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.789185+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.790087+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.791053+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.792036+0000 mon.c [INF] from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.794010+0000 mon.c [INF] from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.823417+0000 mon.b [INF] from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.474932+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.475345+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.501549+0000 mon.c [INF] from='client.? 172.21.15.69:0/3024114509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.512591+0000 mon.c [INF] from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.946871+0000 mon.c [INF] from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.947599+0000 mon.c [INF] from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.682806+0000 mon.a (mon.0) 1022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.682907+0000 mon.a (mon.0) 1023 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.682983+0000 mon.a (mon.0) 1024 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.769564+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.779724+0000 mon.c (mon.2) 60 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.780588+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.781676+0000 mon.c (mon.2) 62 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:33.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.783288+0000 mon.c (mon.2) 63 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.784351+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.787335+0000 mon.c (mon.2) 65 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.789185+0000 mon.c (mon.2) 66 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.790087+0000 mon.c (mon.2) 67 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.791053+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.792036+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.794010+0000 mon.c (mon.2) 70 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:29.823417+0000 mon.b (mon.1) 116 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:29.879490+0000 osd.0 (osd.0) 139 : cluster 0 15.6 deep-scrub starts 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:29.882728+0000 osd.0 (osd.0) 140 : cluster 0 15.6 deep-scrub ok 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.101131+0000 osd.1 (osd.1) 139 : cluster 0 15.7 deep-scrub starts 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.209094+0000 osd.1 (osd.1) 140 : cluster 0 15.7 deep-scrub ok 2024-04-02T04:49:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:30.597345+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.145797+0000 osd.1 (osd.1) 141 : cluster 0 15.4 deep-scrub starts 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.189793+0000 osd.1 (osd.1) 142 : cluster 0 15.4 deep-scrub ok 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.460027+0000 mon.a (mon.0) 1025 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.474932+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.475345+0000 mon.a (mon.0) 1027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.501549+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.69:0/3024114509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.512591+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.600286+0000 mon.c (mon.2) 74 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.946786+0000 mon.a (mon.0) 1028 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.946871+0000 mon.c (mon.2) 75 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.947599+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:31.975320+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:32.373008+0000 mon.a (mon.0) 1029 : cluster 3 Health check update: 31 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: audit 2024-04-02T04:49:32.376894+0000 mon.a (mon.0) 1030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[28904]: cluster 2024-04-02T04:49:32.386553+0000 mon.a (mon.0) 1031 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.682806+0000 mon.a (mon.0) 1022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.682907+0000 mon.a (mon.0) 1023 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.682983+0000 mon.a (mon.0) 1024 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.769564+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.779724+0000 mon.c (mon.2) 60 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.780588+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.781676+0000 mon.c (mon.2) 62 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.783288+0000 mon.c (mon.2) 63 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.784351+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.787335+0000 mon.c (mon.2) 65 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.789185+0000 mon.c (mon.2) 66 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.790087+0000 mon.c (mon.2) 67 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.791053+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.792036+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.794010+0000 mon.c (mon.2) 70 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:29.823417+0000 mon.b (mon.1) 116 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:29.879490+0000 osd.0 (osd.0) 139 : cluster 0 15.6 deep-scrub starts 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:29.882728+0000 osd.0 (osd.0) 140 : cluster 0 15.6 deep-scrub ok 2024-04-02T04:49:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.101131+0000 osd.1 (osd.1) 139 : cluster 0 15.7 deep-scrub starts 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.209094+0000 osd.1 (osd.1) 140 : cluster 0 15.7 deep-scrub ok 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:30.597345+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.145797+0000 osd.1 (osd.1) 141 : cluster 0 15.4 deep-scrub starts 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.189793+0000 osd.1 (osd.1) 142 : cluster 0 15.4 deep-scrub ok 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.460027+0000 mon.a (mon.0) 1025 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.474932+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.475345+0000 mon.a (mon.0) 1027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.501549+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.69:0/3024114509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.512591+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.600286+0000 mon.c (mon.2) 74 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.946786+0000 mon.a (mon.0) 1028 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.946871+0000 mon.c (mon.2) 75 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.947599+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:31.975320+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:32.373008+0000 mon.a (mon.0) 1029 : cluster 3 Health check update: 31 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:33.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: audit 2024-04-02T04:49:32.376894+0000 mon.a (mon.0) 1030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:33 smithi069 bash[21528]: cluster 2024-04-02T04:49:32.386553+0000 mon.a (mon.0) 1031 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-04-02T04:49:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:33.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:33.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:33.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.682806+0000 mon.a (mon.0) 1022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-72439-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.682907+0000 mon.a (mon.0) 1023 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.682983+0000 mon.a (mon.0) 1024 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72504-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.769564+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.779724+0000 mon.c (mon.2) 60 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.780588+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.781676+0000 mon.c (mon.2) 62 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.783288+0000 mon.c (mon.2) 63 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.784351+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.787335+0000 mon.c (mon.2) 65 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:33.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.789185+0000 mon.c (mon.2) 66 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.790087+0000 mon.c (mon.2) 67 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.791053+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.792036+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.794010+0000 mon.c (mon.2) 70 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:29.823417+0000 mon.b (mon.1) 116 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:29.879490+0000 osd.0 (osd.0) 139 : cluster 0 15.6 deep-scrub starts 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:29.882728+0000 osd.0 (osd.0) 140 : cluster 0 15.6 deep-scrub ok 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.101131+0000 osd.1 (osd.1) 139 : cluster 0 15.7 deep-scrub starts 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.209094+0000 osd.1 (osd.1) 140 : cluster 0 15.7 deep-scrub ok 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:30.597345+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.145797+0000 osd.1 (osd.1) 141 : cluster 0 15.4 deep-scrub starts 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.189793+0000 osd.1 (osd.1) 142 : cluster 0 15.4 deep-scrub ok 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.460027+0000 mon.a (mon.0) 1025 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.474932+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.475345+0000 mon.a (mon.0) 1027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.501549+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.69:0/3024114509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.512591+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.600286+0000 mon.c (mon.2) 74 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.946786+0000 mon.a (mon.0) 1028 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.946871+0000 mon.c (mon.2) 75 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.947599+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:31.975320+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:32.373008+0000 mon.a (mon.0) 1029 : cluster 3 Health check update: 31 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: audit 2024-04-02T04:49:32.376894+0000 mon.a (mon.0) 1030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:33.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:33 smithi098 bash[23790]: cluster 2024-04-02T04:49:32.386553+0000 mon.a (mon.0) 1031 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-04-02T04:49:34.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:34.039 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:34.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:34.271 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:34.294 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:34.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:34.421 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:34.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:34.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:34.776 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:34.798 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:34.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:34.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:34.973 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:31.975320+0000 mon.c [INF] from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:32.373008+0000 mon.a [WRN] Health check update: 31 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:32.376894+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:32.397633+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:32.398271+0000 mon.a [INF] from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:32.399724+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:33.683799+0000 mon.a [INF] from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:33.685501+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:33.685626+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:33.685695+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:33.694750+0000 mon.c [INF] from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.144006+0000 mon.a [INF] from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.151677+0000 mon.c [INF] from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.573494+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.579519+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.579790+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:35.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.580100+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.032 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:35.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:35.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:25.852165+0000 mgr.y (mgr.14538) 208 : cluster 0 pgmap v177: 1252 pgs: 774 creating+peering, 132 active+clean, 346 unknown; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-04-02T04:49:35.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:27.856369+0000 mgr.y (mgr.14538) 209 : cluster 0 pgmap v180: 1140 pgs: 144 creating+peering, 996 active+clean; 641 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 89 KiB/s wr, 308 op/s 2024-04-02T04:49:35.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.646468+0000 osd.2 (osd.2) 75 : cluster 0 15.8 deep-scrub starts 2024-04-02T04:49:35.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.659365+0000 osd.2 (osd.2) 76 : cluster 0 15.8 deep-scrub ok 2024-04-02T04:49:35.197 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.681114+0000 osd.2 (osd.2) 77 : cluster 0 15.0 deep-scrub starts 2024-04-02T04:49:35.197 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.683157+0000 osd.2 (osd.2) 78 : cluster 0 15.0 deep-scrub ok 2024-04-02T04:49:35.197 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:32.397633+0000 mon.a (mon.0) 1032 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:32.398271+0000 mon.a (mon.0) 1033 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:32.399724+0000 mon.a (mon.0) 1034 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:32.602234+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:32.692956+0000 osd.2 (osd.2) 79 : cluster 0 15.2 deep-scrub starts 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:32.696760+0000 osd.2 (osd.2) 80 : cluster 0 15.2 deep-scrub ok 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:33.604423+0000 mon.c (mon.2) 79 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:33.683799+0000 mon.a (mon.0) 1035 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:33.685501+0000 mon.a (mon.0) 1036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:33.685626+0000 mon.a (mon.0) 1037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:33.685695+0000 mon.a (mon.0) 1038 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:33.694750+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: cluster 2024-04-02T04:49:34.142599+0000 mon.a (mon.0) 1039 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.144006+0000 mon.a (mon.0) 1040 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.151677+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.573494+0000 mon.a (mon.0) 1041 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.579519+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.579790+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:35.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.580100+0000 mon.a (mon.0) 1044 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.580165+0000 mon.c (mon.2) 82 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.580484+0000 mon.c (mon.2) 83 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.580849+0000 mon.c (mon.2) 84 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:35 smithi098 bash[23790]: audit 2024-04-02T04:49:34.585448+0000 mon.a (mon.0) 1045 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:25.852165+0000 mgr.y (mgr.14538) 208 : cluster 0 pgmap v177: 1252 pgs: 774 creating+peering, 132 active+clean, 346 unknown; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:27.856369+0000 mgr.y (mgr.14538) 209 : cluster 0 pgmap v180: 1140 pgs: 144 creating+peering, 996 active+clean; 641 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 89 KiB/s wr, 308 op/s 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.646468+0000 osd.2 (osd.2) 75 : cluster 0 15.8 deep-scrub starts 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.659365+0000 osd.2 (osd.2) 76 : cluster 0 15.8 deep-scrub ok 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.681114+0000 osd.2 (osd.2) 77 : cluster 0 15.0 deep-scrub starts 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.683157+0000 osd.2 (osd.2) 78 : cluster 0 15.0 deep-scrub ok 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:32.397633+0000 mon.a (mon.0) 1032 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:32.398271+0000 mon.a (mon.0) 1033 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:32.399724+0000 mon.a (mon.0) 1034 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:32.602234+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:32.692956+0000 osd.2 (osd.2) 79 : cluster 0 15.2 deep-scrub starts 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:32.696760+0000 osd.2 (osd.2) 80 : cluster 0 15.2 deep-scrub ok 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:33.604423+0000 mon.c (mon.2) 79 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:33.683799+0000 mon.a (mon.0) 1035 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:33.685501+0000 mon.a (mon.0) 1036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:33.685626+0000 mon.a (mon.0) 1037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:33.685695+0000 mon.a (mon.0) 1038 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:33.694750+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:25.852165+0000 mgr.y (mgr.14538) 208 : cluster 0 pgmap v177: 1252 pgs: 774 creating+peering, 132 active+clean, 346 unknown; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:27.856369+0000 mgr.y (mgr.14538) 209 : cluster 0 pgmap v180: 1140 pgs: 144 creating+peering, 996 active+clean; 641 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 89 KiB/s wr, 308 op/s 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.646468+0000 osd.2 (osd.2) 75 : cluster 0 15.8 deep-scrub starts 2024-04-02T04:49:35.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.659365+0000 osd.2 (osd.2) 76 : cluster 0 15.8 deep-scrub ok 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.681114+0000 osd.2 (osd.2) 77 : cluster 0 15.0 deep-scrub starts 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.683157+0000 osd.2 (osd.2) 78 : cluster 0 15.0 deep-scrub ok 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:32.397633+0000 mon.a (mon.0) 1032 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:32.398271+0000 mon.a (mon.0) 1033 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:32.399724+0000 mon.a (mon.0) 1034 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:32.602234+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:32.692956+0000 osd.2 (osd.2) 79 : cluster 0 15.2 deep-scrub starts 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:32.696760+0000 osd.2 (osd.2) 80 : cluster 0 15.2 deep-scrub ok 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:33.604423+0000 mon.c (mon.2) 79 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:33.683799+0000 mon.a (mon.0) 1035 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:33.685501+0000 mon.a (mon.0) 1036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-72145-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:33.685626+0000 mon.a (mon.0) 1037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:33.685695+0000 mon.a (mon.0) 1038 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:33.694750+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.69:0/2193715244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: cluster 2024-04-02T04:49:34.142599+0000 mon.a (mon.0) 1039 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.144006+0000 mon.a (mon.0) 1040 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.151677+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.573494+0000 mon.a (mon.0) 1041 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.579519+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.579790+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.580100+0000 mon.a (mon.0) 1044 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.580165+0000 mon.c (mon.2) 82 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.580484+0000 mon.c (mon.2) 83 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.580849+0000 mon.c (mon.2) 84 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[28904]: audit 2024-04-02T04:49:34.585448+0000 mon.a (mon.0) 1045 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: cluster 2024-04-02T04:49:34.142599+0000 mon.a (mon.0) 1039 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.144006+0000 mon.a (mon.0) 1040 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.151677+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.573494+0000 mon.a (mon.0) 1041 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.579519+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.579790+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.580100+0000 mon.a (mon.0) 1044 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.580165+0000 mon.c (mon.2) 82 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.580484+0000 mon.c (mon.2) 83 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.580849+0000 mon.c (mon.2) 84 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:35.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:35 smithi069 bash[21528]: audit 2024-04-02T04:49:34.585448+0000 mon.a (mon.0) 1045 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:35.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:35.327 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:35.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:35.454 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:35.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:35.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:35.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:35.840 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:35.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:35.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:35.965 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:35.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:36.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:36.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:36.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:36.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.780056+0000 mgr.y (mgr.14538) 210 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.780737+0000 mgr.y (mgr.14538) 211 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.781907+0000 mgr.y (mgr.14538) 212 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.783605+0000 mgr.y (mgr.14538) 213 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.784622+0000 mgr.y (mgr.14538) 214 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.787744+0000 mgr.y (mgr.14538) 215 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.789409+0000 mgr.y (mgr.14538) 216 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.790299+0000 mgr.y (mgr.14538) 217 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.791383+0000 mgr.y (mgr.14538) 218 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:29.792262+0000 mgr.y (mgr.14538) 219 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:29.827839+0000 client.admin (client.?) 0 : cluster 1 onexx 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:29.857182+0000 mgr.y (mgr.14538) 220 : cluster 0 pgmap v183: 764 pgs: 104 unknown, 144 creating+peering, 516 active+clean; 600 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 86 KiB/s wr, 374 op/s 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.544935+0000 osd.4 (osd.4) 141 : cluster 0 15.3 deep-scrub starts 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.546540+0000 osd.4 (osd.4) 142 : cluster 0 15.3 deep-scrub ok 2024-04-02T04:49:36.346 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.757011+0000 osd.6 (osd.6) 97 : cluster 0 15.1 deep-scrub starts 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:30.760383+0000 osd.6 (osd.6) 98 : cluster 0 15.1 deep-scrub ok 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.779777+0000 osd.6 (osd.6) 99 : cluster 0 15.9 deep-scrub starts 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.824951+0000 osd.6 (osd.6) 100 : cluster 0 15.9 deep-scrub ok 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:31.858495+0000 mgr.y (mgr.14538) 221 : cluster 0 pgmap v185: 820 pgs: 232 unknown, 136 creating+peering, 452 active+clean; 587 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 20 KiB/s wr, 253 op/s 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:32.778736+0000 osd.6 (osd.6) 101 : cluster 0 15.5 deep-scrub starts 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:32.781535+0000 osd.6 (osd.6) 102 : cluster 0 15.5 deep-scrub ok 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:32.992506+0000 mgr.y (mgr.14538) 222 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: cluster 2024-04-02T04:49:33.859626+0000 mgr.y (mgr.14538) 223 : cluster 0 pgmap v188: 884 pgs: 96 unknown, 1 active+clean+scrubbing+deep, 214 creating+peering, 573 active+clean; 145 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 36 MiB/s wr, 177 op/s 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:34.606931+0000 mon.c (mon.2) 85 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:35.014099+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:35.014292+0000 mon.a (mon.0) 1047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:35.014474+0000 mon.a (mon.0) 1048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:35.014659+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:36.347 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:35.015166+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:36.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:36 smithi098 bash[23790]: audit 2024-04-02T04:49:35.018717+0000 mon.a (mon.0) 1051 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:36.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:36.511 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:36.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:36.623 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.780056+0000 mgr.y (mgr.14538) 210 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:36.623 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.780737+0000 mgr.y (mgr.14538) 211 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:36.623 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.781907+0000 mgr.y (mgr.14538) 212 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:36.623 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.783605+0000 mgr.y (mgr.14538) 213 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:36.623 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.784622+0000 mgr.y (mgr.14538) 214 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.787744+0000 mgr.y (mgr.14538) 215 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.789409+0000 mgr.y (mgr.14538) 216 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.790299+0000 mgr.y (mgr.14538) 217 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.791383+0000 mgr.y (mgr.14538) 218 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:29.792262+0000 mgr.y (mgr.14538) 219 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:29.827839+0000 client.admin (client.?) 0 : cluster 1 onexx 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:29.857182+0000 mgr.y (mgr.14538) 220 : cluster 0 pgmap v183: 764 pgs: 104 unknown, 144 creating+peering, 516 active+clean; 600 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 86 KiB/s wr, 374 op/s 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.544935+0000 osd.4 (osd.4) 141 : cluster 0 15.3 deep-scrub starts 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.546540+0000 osd.4 (osd.4) 142 : cluster 0 15.3 deep-scrub ok 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.757011+0000 osd.6 (osd.6) 97 : cluster 0 15.1 deep-scrub starts 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:30.760383+0000 osd.6 (osd.6) 98 : cluster 0 15.1 deep-scrub ok 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.779777+0000 osd.6 (osd.6) 99 : cluster 0 15.9 deep-scrub starts 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.824951+0000 osd.6 (osd.6) 100 : cluster 0 15.9 deep-scrub ok 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:31.858495+0000 mgr.y (mgr.14538) 221 : cluster 0 pgmap v185: 820 pgs: 232 unknown, 136 creating+peering, 452 active+clean; 587 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 20 KiB/s wr, 253 op/s 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:32.778736+0000 osd.6 (osd.6) 101 : cluster 0 15.5 deep-scrub starts 2024-04-02T04:49:36.624 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:32.781535+0000 osd.6 (osd.6) 102 : cluster 0 15.5 deep-scrub ok 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:32.992506+0000 mgr.y (mgr.14538) 222 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.780056+0000 mgr.y (mgr.14538) 210 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.780737+0000 mgr.y (mgr.14538) 211 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.781907+0000 mgr.y (mgr.14538) 212 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.783605+0000 mgr.y (mgr.14538) 213 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.784622+0000 mgr.y (mgr.14538) 214 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.787744+0000 mgr.y (mgr.14538) 215 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.789409+0000 mgr.y (mgr.14538) 216 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:49:36.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.790299+0000 mgr.y (mgr.14538) 217 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.791383+0000 mgr.y (mgr.14538) 218 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:29.792262+0000 mgr.y (mgr.14538) 219 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:29.827839+0000 client.admin (client.?) 0 : cluster 1 onexx 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:29.857182+0000 mgr.y (mgr.14538) 220 : cluster 0 pgmap v183: 764 pgs: 104 unknown, 144 creating+peering, 516 active+clean; 600 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 86 KiB/s wr, 374 op/s 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.544935+0000 osd.4 (osd.4) 141 : cluster 0 15.3 deep-scrub starts 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.546540+0000 osd.4 (osd.4) 142 : cluster 0 15.3 deep-scrub ok 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.757011+0000 osd.6 (osd.6) 97 : cluster 0 15.1 deep-scrub starts 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:30.760383+0000 osd.6 (osd.6) 98 : cluster 0 15.1 deep-scrub ok 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.779777+0000 osd.6 (osd.6) 99 : cluster 0 15.9 deep-scrub starts 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.824951+0000 osd.6 (osd.6) 100 : cluster 0 15.9 deep-scrub ok 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:31.858495+0000 mgr.y (mgr.14538) 221 : cluster 0 pgmap v185: 820 pgs: 232 unknown, 136 creating+peering, 452 active+clean; 587 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 20 KiB/s wr, 253 op/s 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:32.778736+0000 osd.6 (osd.6) 101 : cluster 0 15.5 deep-scrub starts 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:32.781535+0000 osd.6 (osd.6) 102 : cluster 0 15.5 deep-scrub ok 2024-04-02T04:49:36.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:32.992506+0000 mgr.y (mgr.14538) 222 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: cluster 2024-04-02T04:49:33.859626+0000 mgr.y (mgr.14538) 223 : cluster 0 pgmap v188: 884 pgs: 96 unknown, 1 active+clean+scrubbing+deep, 214 creating+peering, 573 active+clean; 145 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 36 MiB/s wr, 177 op/s 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:34.606931+0000 mon.c (mon.2) 85 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:35.014099+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:35.014292+0000 mon.a (mon.0) 1047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:35.014474+0000 mon.a (mon.0) 1048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:35.014659+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:35.015166+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[21528]: audit 2024-04-02T04:49:35.018717+0000 mon.a (mon.0) 1051 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:36.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: cluster 2024-04-02T04:49:33.859626+0000 mgr.y (mgr.14538) 223 : cluster 0 pgmap v188: 884 pgs: 96 unknown, 1 active+clean+scrubbing+deep, 214 creating+peering, 573 active+clean; 145 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 36 MiB/s wr, 177 op/s 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:34.606931+0000 mon.c (mon.2) 85 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:35.014099+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:35.014292+0000 mon.a (mon.0) 1047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:35.014474+0000 mon.a (mon.0) 1048 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:35.014659+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:35.015166+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:36.628 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:36 smithi069 bash[28904]: audit 2024-04-02T04:49:35.018717+0000 mon.a (mon.0) 1051 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:36.634 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:36.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:36.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:36.878 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:36.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:36.916 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:49:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-02T04:49:36.916 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:49:36.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:37.001 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:37.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:37.124 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:49:37.124 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:49:37.124 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:49:37.124 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:49:37.124 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:49:37.132 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:37.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:37.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:37.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:37.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.018789+0000 mon.a (mon.0) 1052 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:37.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.018861+0000 mon.a (mon.0) 1053 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:37.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.041648+0000 mon.c (mon.2) 86 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:37.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.042553+0000 mon.b (mon.1) 117 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.378 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.048236+0000 mon.c (mon.2) 87 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.048515+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.094746+0000 mon.b (mon.1) 118 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:35.608636+0000 mon.c (mon.2) 89 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: cluster 2024-04-02T04:49:35.861062+0000 mgr.y (mgr.14538) 224 : cluster 0 pgmap v190: 908 pgs: 192 unknown, 1 active+clean+scrubbing+deep, 210 creating+peering, 505 active+clean; 145 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 229 op/s 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: cluster 2024-04-02T04:49:36.315036+0000 mon.a (mon.0) 1054 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.325806+0000 mon.a (mon.0) 1055 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.326462+0000 mon.a (mon.0) 1056 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.331405+0000 mon.b (mon.1) 119 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.334956+0000 mon.a (mon.0) 1057 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.335438+0000 mon.a (mon.0) 1058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: cluster 2024-04-02T04:49:36.335748+0000 client.admin (client.?) 0 : cluster 1 twoxx 2024-04-02T04:49:37.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.336655+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.336879+0000 mon.a (mon.0) 1060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.337023+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.337222+0000 mon.a (mon.0) 1062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.338481+0000 mon.a (mon.0) 1063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.338676+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.357522+0000 mon.c (mon.2) 90 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.357709+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.375646+0000 mon.b (mon.1) 120 : audit 1 from='client.? 172.21.15.69:0/1435587795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.376120+0000 mon.b (mon.1) 121 : audit 1 from='client.? 172.21.15.69:0/1321485616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.610227+0000 mon.c (mon.2) 92 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.816232+0000 mon.b (mon.1) 122 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: cluster 2024-04-02T04:49:36.819926+0000 mon.a (mon.0) 1065 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.819946+0000 mon.c (mon.2) 93 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.822250+0000 mon.a (mon.0) 1066 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.826028+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.826471+0000 mon.a (mon.0) 1068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.829919+0000 mon.a (mon.0) 1069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.830259+0000 mon.a (mon.0) 1070 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.830605+0000 mon.a (mon.0) 1071 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.835219+0000 mon.a (mon.0) 1072 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.837338+0000 mon.b (mon.1) 123 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.840414+0000 mon.a (mon.0) 1073 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:37 smithi098 bash[23790]: audit 2024-04-02T04:49:36.841775+0000 mon.a (mon.0) 1074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.403 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:37.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:37.517 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.018789+0000 mon.a (mon.0) 1052 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:37.517 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.018861+0000 mon.a (mon.0) 1053 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:37.517 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.041648+0000 mon.c (mon.2) 86 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:37.517 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.042553+0000 mon.b (mon.1) 117 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.517 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.048236+0000 mon.c (mon.2) 87 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.048515+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.094746+0000 mon.b (mon.1) 118 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:35.608636+0000 mon.c (mon.2) 89 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: cluster 2024-04-02T04:49:35.861062+0000 mgr.y (mgr.14538) 224 : cluster 0 pgmap v190: 908 pgs: 192 unknown, 1 active+clean+scrubbing+deep, 210 creating+peering, 505 active+clean; 145 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 229 op/s 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: cluster 2024-04-02T04:49:36.315036+0000 mon.a (mon.0) 1054 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.325806+0000 mon.a (mon.0) 1055 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.326462+0000 mon.a (mon.0) 1056 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.331405+0000 mon.b (mon.1) 119 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.334956+0000 mon.a (mon.0) 1057 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:37.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.335438+0000 mon.a (mon.0) 1058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: cluster 2024-04-02T04:49:36.335748+0000 client.admin (client.?) 0 : cluster 1 twoxx 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.336655+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.336879+0000 mon.a (mon.0) 1060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.337023+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.337222+0000 mon.a (mon.0) 1062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.338481+0000 mon.a (mon.0) 1063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.338676+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.357522+0000 mon.c (mon.2) 90 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.357709+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.375646+0000 mon.b (mon.1) 120 : audit 1 from='client.? 172.21.15.69:0/1435587795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.376120+0000 mon.b (mon.1) 121 : audit 1 from='client.? 172.21.15.69:0/1321485616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.610227+0000 mon.c (mon.2) 92 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:37.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.816232+0000 mon.b (mon.1) 122 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: cluster 2024-04-02T04:49:36.819926+0000 mon.a (mon.0) 1065 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.819946+0000 mon.c (mon.2) 93 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.822250+0000 mon.a (mon.0) 1066 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.826028+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.826471+0000 mon.a (mon.0) 1068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.829919+0000 mon.a (mon.0) 1069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.830259+0000 mon.a (mon.0) 1070 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.830605+0000 mon.a (mon.0) 1071 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.835219+0000 mon.a (mon.0) 1072 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.837338+0000 mon.b (mon.1) 123 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.840414+0000 mon.a (mon.0) 1073 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[21528]: audit 2024-04-02T04:49:36.841775+0000 mon.a (mon.0) 1074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.018789+0000 mon.a (mon.0) 1052 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.018861+0000 mon.a (mon.0) 1053 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.041648+0000 mon.c (mon.2) 86 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.042553+0000 mon.b (mon.1) 117 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.048236+0000 mon.c (mon.2) 87 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.048515+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.094746+0000 mon.b (mon.1) 118 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:35.608636+0000 mon.c (mon.2) 89 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: cluster 2024-04-02T04:49:35.861062+0000 mgr.y (mgr.14538) 224 : cluster 0 pgmap v190: 908 pgs: 192 unknown, 1 active+clean+scrubbing+deep, 210 creating+peering, 505 active+clean; 145 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 229 op/s 2024-04-02T04:49:37.521 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: cluster 2024-04-02T04:49:36.315036+0000 mon.a (mon.0) 1054 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.325806+0000 mon.a (mon.0) 1055 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.326462+0000 mon.a (mon.0) 1056 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.331405+0000 mon.b (mon.1) 119 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.334956+0000 mon.a (mon.0) 1057 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.335438+0000 mon.a (mon.0) 1058 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: cluster 2024-04-02T04:49:36.335748+0000 client.admin (client.?) 0 : cluster 1 twoxx 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.336655+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.336879+0000 mon.a (mon.0) 1060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.337023+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.337222+0000 mon.a (mon.0) 1062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.338481+0000 mon.a (mon.0) 1063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.338676+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:37.522 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.357522+0000 mon.c (mon.2) 90 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.357709+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.375646+0000 mon.b (mon.1) 120 : audit 1 from='client.? 172.21.15.69:0/1435587795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.376120+0000 mon.b (mon.1) 121 : audit 1 from='client.? 172.21.15.69:0/1321485616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.610227+0000 mon.c (mon.2) 92 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.816232+0000 mon.b (mon.1) 122 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: cluster 2024-04-02T04:49:36.819926+0000 mon.a (mon.0) 1065 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.819946+0000 mon.c (mon.2) 93 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.822250+0000 mon.a (mon.0) 1066 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.826028+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.826471+0000 mon.a (mon.0) 1068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.829919+0000 mon.a (mon.0) 1069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.830259+0000 mon.a (mon.0) 1070 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.830605+0000 mon.a (mon.0) 1071 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.835219+0000 mon.a (mon.0) 1072 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.837338+0000 mon.b (mon.1) 123 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.840414+0000 mon.a (mon.0) 1073 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:37.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:37 smithi069 bash[28904]: audit 2024-04-02T04:49:36.841775+0000 mon.a (mon.0) 1074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:37.548 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:37.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:37.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:37.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:37.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:38.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:38.059 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:38.163 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:38.187 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:38.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:38.317 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:38.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:38.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:38.576 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:38.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:38.699 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.580165+0000 mon.c [INF] from='client.? 172. delete_pools_parallel: process_1_[72808]: starting. 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[72808]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-72765 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[72808]: created object 0... 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[72808]: created object 25... 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[72808]: created object 49... 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[72808]: finishing. 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[72808]: shutting down. 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[72809]: starting. 2024-04-02T04:49:38.722 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[72809]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-72765 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[72809]: shutting down. 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: starting. 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-72765 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: created object 0... 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: created object 25... 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: created object 49... 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: finishing. 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[75673]: shutting down. 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[75675]: starting. 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[75675]: listing objects. 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[75675]: listed object 0... 2024-04-02T04:49:38.723 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[75675]: listed object 25... 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[75675]: saw 50 objects 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[75675]: shutting down. 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[75674]: starting. 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[75674]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-72765 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[75674]: shutting down. 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-04-02T04:49:38.724 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: starting. 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: creating pool ceph_test_rados_open_pools_parallel.smithi069-72674 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: created object 0... 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: created object 25... 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: created object 49... 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: finishing. 2024-04-02T04:49:38.734 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[72698]: shutting down. 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[72703]: starting. 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[72703]: rados_pool_create. 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[72703]: rados_ioctx_create. 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[72703]: shutting down. 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: starting. 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: creating pool ceph_test_rados_open_pools_parallel.smithi069-72674 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: created object 0... 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: created object 25... 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: created object 49... 2024-04-02T04:49:38.735 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: finishing. 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[75662]: shutting down. 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[75663]: starting. 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[75663]: rados_pool_create. 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[75663]: rados_ioctx_create. 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[75663]: shutting down. 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-04-02T04:49:38.736 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-02T04:49:38.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:38.826 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:38.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:38.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:39.049 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:49:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:49:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:49:39.076 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:39.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:39.199 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:39.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:39.335 INFO:tasks.workunit.client.0.smithi069.stdout:21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.580484+0000 mon.c [INF] from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.580849+0000 mon.c [INF] from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:39.335 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:34.585448+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:29.827839+0000 client.admin [INF] onexx 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.014099+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.014292+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.014474+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.014659+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.015166+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.018717+0000 mon.a [INF] from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.018789+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.018861+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi069-72924-1"}]': finished 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.041648+0000 mon.c [INF] from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.042553+0000 mon.b [INF] from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:39.336 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.048236+0000 mon.c [INF] from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.048515+0000 mon.c [INF] from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:35.094746+0000 mon.b [INF] from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.325806+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.326462+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.331405+0000 mon.b [INF] from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.334956+0000 mon.a [INF] from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.335438+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.335748+0000 client.admin [INF] twoxx 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.336655+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.336879+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:39.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.337023+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.337222+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:39.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.338481+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-72161-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:39.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.338676+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:39.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.357522+0000 mon.c [INF] from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:39.338 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.357709+0000 mon.c [INF] from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:39.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:39.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:39.590 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:39.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.375646+0000 mon.b [INF] from='client.? 172.21.15.69:0/1435587795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.376120+0000 mon.b [INF] from='client.? 172.21.15.69:0/1321485616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.816232+0000 mon.b [INF] from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.819946+0000 mon.c [INF] from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.822250+0000 mon.a [INF] from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.826028+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.826471+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.829919+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.830259+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.830605+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.835219+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.837338+0000 mon.b [INF] from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.840414+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:39.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:36.841775+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:39.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.328397+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-02T04:49:39.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.362060+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:39.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-02T04:49:39.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:39.720 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:39.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:39.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:39.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[21528]: cluster 2024-04-02T04:49:37.025579+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[21528]: cluster 2024-04-02T04:49:37.025690+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[21528]: audit 2024-04-02T04:49:37.328397+0000 mon.a (mon.0) 1075 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[21528]: audit 2024-04-02T04:49:37.362060+0000 mon.a (mon.0) 1076 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[21528]: audit 2024-04-02T04:49:37.362205+0000 mon.a (mon.0) 1077 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[28904]: cluster 2024-04-02T04:49:37.025579+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[28904]: cluster 2024-04-02T04:49:37.025690+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[28904]: audit 2024-04-02T04:49:37.328397+0000 mon.a (mon.0) 1075 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[28904]: audit 2024-04-02T04:49:37.362060+0000 mon.a (mon.0) 1076 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:39.953 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:39 smithi069 bash[28904]: audit 2024-04-02T04:49:37.362205+0000 mon.a (mon.0) 1077 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:39.966 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:39.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:39.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:39 smithi098 bash[23790]: cluster 2024-04-02T04:49:37.025579+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:39.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:39 smithi098 bash[23790]: cluster 2024-04-02T04:49:37.025690+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:39.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:39 smithi098 bash[23790]: audit 2024-04-02T04:49:37.328397+0000 mon.a (mon.0) 1075 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-02T04:49:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:39 smithi098 bash[23790]: audit 2024-04-02T04:49:37.362060+0000 mon.a (mon.0) 1076 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-4", "overlaypool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:39 smithi098 bash[23790]: audit 2024-04-02T04:49:37.362205+0000 mon.a (mon.0) 1077 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:40.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:40.114 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:40.129 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:40.216 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:40.234 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:40.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:40.358 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:40.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:40.488 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:40.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:40.615 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:40.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:40.745 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:40.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:40.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:40.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:40.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:40.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:41.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-02T04:49:41.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5727 ms) 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (47 ms) 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20039 ms) 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: attempt 0 of 20 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5071 ms) 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30884 ms total) 2024-04-02T04:49:41.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: 2024-04-02T04:49:41.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-02T04:49:41.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30884 ms total) 2024-04-02T04:49:41.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-02T04:49:41.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:41.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:41.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:41.369 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:41.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:41.502 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:41.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:41.632 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:41.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:41.759 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:41.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:41.888 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:41.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 -- 172.21.15.69:0/4100666809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f0101b30 msgr2=0x7fe8f010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/4100666809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f0101b30 0x7fe8f010e020 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800b0a0 tx=0x7fe8e800b3b0 comp rx=0 tx=0).stop 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 -- 172.21.15.69:0/4100666809 shutdown_connections 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/4100666809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f0101b30 0x7fe8f010e020 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/4100666809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f01071e0 0x7fe8f01075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/4100666809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0101170 0x7fe8f01015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 -- 172.21.15.69:0/4100666809 >> 172.21.15.69:0/4100666809 conn(0x7fe8f0078110 msgr2=0x7fe8f0078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:41.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 -- 172.21.15.69:0/4100666809 shutdown_connections 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.921+0000 7fe8f64ac700 1 -- 172.21.15.69:0/4100666809 wait complete. 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 librados: starting msgr at 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 librados: starting objecter 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 Processor -- start 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- start start 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 librados: setting wanted keys 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 librados: calling monclient init 2024-04-02T04:49:41.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0101170 0x7fe8f019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f0101b30 0x7fe8f019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f01071e0 0x7fe8f0195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ef7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f0101b30 0x7fe8f019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ef7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f0101b30 0x7fe8f019bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39312/0 (socket says 172.21.15.69:39312) 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ef7fe700 1 -- 172.21.15.69:0/872957453 learned_addr learned my addr 172.21.15.69:0/872957453 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f010f940 con 0x7fe8f0101b30 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f010f7c0 con 0x7fe8f0101170 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe8f010fac0 con 0x7fe8f01071e0 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0101170 0x7fe8f019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f01071e0 0x7fe8f0195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0101170 msgr2=0x7fe8f019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:41.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0101170 0x7fe8f019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f0101b30 msgr2=0x7fe8f019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f0101b30 0x7fe8f019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f0196180 con 0x7fe8f01071e0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8f0101170 0x7fe8f019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8f0101b30 0x7fe8f019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f01071e0 0x7fe8f0195a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009fd0 tx=0x7fe8e8007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ed7fa700 10 client.?.objecter ms_handle_connect 0x7fe8f01071e0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ed7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8021020 con 0x7fe8f01071e0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe8e801d8f0 con 0x7fe8f01071e0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f0196410 con 0x7fe8f01071e0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e80052b0 con 0x7fe8f01071e0 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 10 client.25355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:49:41.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f01a26e0 con 0x7fe8f01071e0 2024-04-02T04:49:41.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 librados: init done 2024-04-02T04:49:41.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.925+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe8f0102890 con 0x7fe8f01071e0 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe8e8024020 con 0x7fe8f01071e0 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe8d806f5e0 0x7fe8d8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(149..149 src has 74..149) ==== 19121+0+0 (secure 0 0 0) 0x7fe8e80926b0 con 0x7fe8f01071e0 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe8d806f5e0 0x7fe8d8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 osd_map(149..149 src has 74..149) 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 3 client.25355.objecter handle_osd_map got epochs [149,149] > 0 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 3 client.25355.objecter handle_osd_map decoding full epoch 149 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 20 client.25355.objecter dump_active .. 0 homeless 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v149) ==== 54+0+16 (secure 0 0 0) 0x7fe8e802b070 con 0x7fe8f01071e0 2024-04-02T04:49:41.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe8d806f5e0 0x7fe8d8071aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00ff850 tx=0x7fe8e0011040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 1 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8d806f5e0 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe8f00630a0 0x7fe8f0107f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 1 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 0 1 2024-04-02T04:49:41.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 4 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 10 client.25355.objecter _send_command 1 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8f004ff00 con 0x7fe8f00630a0 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe8f00630a0 0x7fe8f0107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe8f00630a0 0x7fe8f0107f60 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f00630a0 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f004ff00 con 0x7fe8f00630a0 2024-04-02T04:49:41.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 1: 0 ) 2024-04-02T04:49:41.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 1 = system:0 2024-04-02T04:49:41.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 4 2024-04-02T04:49:41.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.929+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 0 1 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 2 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 2 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f0196f00 osd=0 3 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 0 2 2024-04-02T04:49:41.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 4 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 10 client.25355.objecter _send_command 2 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01084a0 con 0x7fe8f00630a0 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01084a0 con 0x7fe8f00630a0 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 4 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 0 2 2024-04-02T04:49:41.951 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 3 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe8f0062c90 0x7fe8f01a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 3 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 1 3 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 4 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 10 client.25355.objecter _send_command 3 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe8f0062c90 0x7fe8f01a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe8f002d690 con 0x7fe8f0062c90 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f0062c90 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.949+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe8f0062c90 0x7fe8f01a68d0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.953+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f002d690 con 0x7fe8f0062c90 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.953+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 3: 0 ) 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.953+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 3 = system:0 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.953+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 4 2024-04-02T04:49:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.953+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 1 3 2024-04-02T04:49:41.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:41.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 4 2024-04-02T04:49:41.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 4 2024-04-02T04:49:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f0062b20 osd=1 3 2024-04-02T04:49:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 1 4 2024-04-02T04:49:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 4 2024-04-02T04:49:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 10 client.25355.objecter _send_command 4 2024-04-02T04:49:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.969+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01a6e10 con 0x7fe8f0062c90 2024-04-02T04:49:41.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01a6e10 con 0x7fe8f0062c90 2024-04-02T04:49:41.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:41.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:41.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 4 2024-04-02T04:49:41.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 1 4 2024-04-02T04:49:41.973 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:49:41.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:41.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 5 2024-04-02T04:49:41.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe8f002cd70 0x7fe8f01aa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 5 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 2 5 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 4 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 10 client.25355.objecter _send_command 5 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe8f01093c0 con 0x7fe8f002cd70 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe8f002cd70 0x7fe8f01aa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f002cd70 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe8f002cd70 0x7fe8f01aa1b0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f01093c0 con 0x7fe8f002cd70 2024-04-02T04:49:41.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 5: 0 ) 2024-04-02T04:49:41.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 5 = system:0 2024-04-02T04:49:41.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 4 2024-04-02T04:49:41.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.973+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 2 5 2024-04-02T04:49:41.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:41.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 6 2024-04-02T04:49:41.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 6 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f002cc00 osd=2 3 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 2 6 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 4 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 10 client.25355.objecter _send_command 6 2024-04-02T04:49:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.989+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01088a0 con 0x7fe8f002cd70 2024-04-02T04:49:41.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:41.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01088a0 con 0x7fe8f002cd70 2024-04-02T04:49:41.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:41.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:41.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 4 2024-04-02T04:49:41.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 2 6 2024-04-02T04:49:41.993 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:49:41.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:41.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 7 2024-04-02T04:49:41.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe8f01a7ca0 0x7fe8f01ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 7 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 3 7 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 4 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 10 client.25355.objecter _send_command 7 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe8f01b0730 con 0x7fe8f01a7ca0 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe8f01a7ca0 0x7fe8f01ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f01a7ca0 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe8f01a7ca0 0x7fe8f01ad880 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f01b0730 con 0x7fe8f01a7ca0 2024-04-02T04:49:41.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 7: 0 ) 2024-04-02T04:49:41.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 7 = system:0 2024-04-02T04:49:41.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 4 2024-04-02T04:49:41.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:41.993+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 3 7 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 8 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 8 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f0108ca0 osd=3 3 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 3 8 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 4 2024-04-02T04:49:42.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 10 client.25355.objecter _send_command 8 2024-04-02T04:49:42.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.009+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01a72d0 con 0x7fe8f01a7ca0 2024-04-02T04:49:42.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01a72d0 con 0x7fe8f01a7ca0 2024-04-02T04:49:42.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:42.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:42.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 4 2024-04-02T04:49:42.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 3 8 2024-04-02T04:49:42.013 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 9 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe8f01aef40 0x7fe8f01af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 9 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 4 9 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 4 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 10 client.25355.objecter _send_command 9 2024-04-02T04:49:42.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe8f01ab620 con 0x7fe8f01aef40 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe8f01aef40 0x7fe8f01af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f01aef40 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe8f01aef40 0x7fe8f01af3a0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f01ab620 con 0x7fe8f01aef40 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 9: 0 ) 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 9 = system:0 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 4 2024-04-02T04:49:42.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.013+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 4 9 2024-04-02T04:49:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 10 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 10 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01a72d0 osd=4 3 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 4 10 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 4 2024-04-02T04:49:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 10 client.25355.objecter _send_command 10 2024-04-02T04:49:42.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.033+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01af8e0 con 0x7fe8f01aef40 2024-04-02T04:49:42.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01af8e0 con 0x7fe8f01aef40 2024-04-02T04:49:42.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:42.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:42.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 4 2024-04-02T04:49:42.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 4 10 2024-04-02T04:49:42.036 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 11 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe8f01afdf0 0x7fe8f01b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 11 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 5 11 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 4 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 10 client.25355.objecter _send_command 11 2024-04-02T04:49:42.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe8f01b03c0 con 0x7fe8f01afdf0 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe8f01afdf0 0x7fe8f01b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f01afdf0 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ef7fe700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe8f01afdf0 0x7fe8f01b4990 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f01b03c0 con 0x7fe8f01afdf0 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 11: 0 ) 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 11 = system:0 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 4 2024-04-02T04:49:42.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.037+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 5 11 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 12 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 12 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01ab6d0 osd=5 3 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 5 12 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 4 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 10 client.25355.objecter _send_command 12 2024-04-02T04:49:42.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.053+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01b03c0 con 0x7fe8f01afdf0 2024-04-02T04:49:42.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:49:42.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01b03c0 con 0x7fe8f01afdf0 2024-04-02T04:49:42.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 4 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 5 12 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 13 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe8f01b53e0 0x7fe8f01b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 13 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 6 13 2024-04-02T04:49:42.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 4 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 10 client.25355.objecter _send_command 13 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe8f01ba770 con 0x7fe8f01b53e0 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe8f01b53e0 0x7fe8f01b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f01b53e0 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8f4a49700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe8f01b53e0 0x7fe8f01b77d0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.057+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f01ba770 con 0x7fe8f01b53e0 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.061+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 13: 0 ) 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.061+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 13 = system:0 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.061+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 4 2024-04-02T04:49:42.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.061+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 6 13 2024-04-02T04:49:42.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:42.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 14 2024-04-02T04:49:42.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 14 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01aaec0 osd=6 3 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 6 14 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 4 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 10 client.25355.objecter _send_command 14 2024-04-02T04:49:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.077+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01b95b0 con 0x7fe8f01b53e0 2024-04-02T04:49:42.080 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01b95b0 con 0x7fe8f01b53e0 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 4 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 6 14 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 15 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fe8f01b9cd0 0x7fe8f01ba110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 15 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 7 15 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 4 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 10 client.25355.objecter _send_command 15 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe8f01bdb90 con 0x7fe8f01b9cd0 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fe8f01b9cd0 0x7fe8f01ba110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 10 client.25355.objecter ms_handle_connect 0x7fe8f01b9cd0 2024-04-02T04:49:42.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8effff700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fe8f01b9cd0 0x7fe8f01ba110 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:42.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.7 v2:172.21.15.98:6824/148009079 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8f01bdb90 con 0x7fe8f01b9cd0 2024-04-02T04:49:42.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 15: 0 ) 2024-04-02T04:49:42.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 15 = system:0 2024-04-02T04:49:42.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 4 2024-04-02T04:49:42.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.081+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 7 15 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign -1 16 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter _get_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_remove -1 16 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter get_session s=0x7fe8f01b95b0 osd=7 3 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 15 client.25355.objecter _session_command_op_assign 7 16 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 4 2024-04-02T04:49:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter _send_command 16 2024-04-02T04:49:42.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 --> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8f01bdb90 con 0x7fe8f01b9cd0 2024-04-02T04:49:42.103 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:49:42.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8ed7fa700 1 -- 172.21.15.69:0/872957453 <== osd.7 v2:172.21.15.98:6824/148009079 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8f01bdb90 con 0x7fe8f01b9cd0 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8ed7fa700 10 client.25355.objecter ms_dispatch 0x7fe8f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8ed7fa700 10 client.25355.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8ed7fa700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 4 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8ed7fa700 15 client.25355.objecter _session_command_op_remove 7 16 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 librados: watch_flush enter 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 librados: watch_flush exit 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.0 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe8f00630a0 msgr2=0x7fe8f0107f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe8f00630a0 0x7fe8f0107f60 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0196f00 osd=0 1 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.1 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe8f0062c90 msgr2=0x7fe8f01a68d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe8f0062c90 0x7fe8f01a68d0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0062b20 osd=1 1 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.2 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe8f002cd70 msgr2=0x7fe8f01aa1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe8f002cd70 0x7fe8f01aa1b0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f002cc00 osd=2 1 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.3 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe8f01a7ca0 msgr2=0x7fe8f01ad880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe8f01a7ca0 0x7fe8f01ad880 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f0108ca0 osd=3 1 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f4a49700 1 -- 172.21.15.69:0/872957453 reap_dead start 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.4 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe8f01aef40 msgr2=0x7fe8f01af3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe8f01aef40 0x7fe8f01af3a0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01a72d0 osd=4 1 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.5 2024-04-02T04:49:42.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe8f01afdf0 msgr2=0x7fe8f01b4990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f4a49700 1 -- 172.21.15.69:0/872957453 reap_dead start 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe8f01afdf0 0x7fe8f01b4990 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01ab6d0 osd=5 1 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.6 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe8f01b53e0 msgr2=0x7fe8f01b77d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe8f01b53e0 0x7fe8f01b77d0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01aaec0 osd=6 1 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter close_session for osd.7 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fe8f01b9cd0 msgr2=0x7fe8f01ba110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.98:6824/148009079,v1:172.21.15.98:6825/148009079] conn(0x7fe8f01b9cd0 0x7fe8f01ba110 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter put_session s=0x7fe8f01b95b0 osd=7 1 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 20 client.25355.objecter shutdown clearing up homeless session... 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 10 client.25355.objecter successfully canceled tick 2024-04-02T04:49:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe8d806f5e0 msgr2=0x7fe8d8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe8d806f5e0 0x7fe8d8071aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00ff850 tx=0x7fe8e0011040 comp rx=0 tx=0).stop 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f01071e0 msgr2=0x7fe8f0195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f64ac700 1 --2- 172.21.15.69:0/872957453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8f01071e0 0x7fe8f0195a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009fd0 tx=0x7fe8e8007d10 comp rx=0 tx=0).stop 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.101+0000 7fe8f4a49700 1 -- 172.21.15.69:0/872957453 reap_dead start 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.105+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 shutdown_connections 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.105+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 >> 172.21.15.69:0/872957453 conn(0x7fe8f0078110 msgr2=0x7fe8f00ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.105+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 shutdown_connections 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.105+0000 7fe8f64ac700 1 -- 172.21.15.69:0/872957453 wait complete. 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:42.105+0000 7fe8f64ac700 1 librados: shutdown 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:49:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:49:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:49:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:49:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:49:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:49:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:49:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:49:42.124 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:42.142 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:42.244 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-02T04:49:42.244 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-02T04:49:42.244 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-02T04:49:42.244 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-02T04:49:42.244 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: handle_notify cookie 94695671546448 notify_id 592705486849 notifier_gid 24911 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (19317 ms) 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Trying... 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: handle_notify cookie 94695684277168 notify_id 622770257924 notifier_gid 15258 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Waiting for 3.000000000s 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Timed out. 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Flushing... 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Flushed... 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (12551 ms) 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (31868 ms total) 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (31868 ms total) 2024-04-02T04:49:42.245 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-02T04:49:42.246 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:42.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.362205+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.362385+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.362523+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.362697+0000 mon.a [INF] from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.362856+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.363050+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.363183+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.363417+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.371178+0000 mon.b [INF] from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.371329+0000 mon.c [INF] from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.371523+0000 mon.c [INF] from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:37.371687+0000 mon.c [INF] from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.713620+0000 mon.b [INF] from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:42.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.716179+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:42.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.362385+0000 mon.a (mon.0) 1078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.362523+0000 mon.a (mon.0) 1079 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.362697+0000 mon.a (mon.0) 1080 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.362856+0000 mon.a (mon.0) 1081 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.363050+0000 mon.a (mon.0) 1082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.363183+0000 mon.a (mon.0) 1083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.363417+0000 mon.a (mon.0) 1084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.371178+0000 mon.b (mon.1) 124 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.371329+0000 mon.c (mon.2) 94 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.371523+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.371687+0000 mon.c (mon.2) 96 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:37.611479+0000 mon.c (mon.2) 97 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.614079+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.713620+0000 mon.b (mon.1) 125 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:42.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: cluster 2024-04-02T04:49:38.715539+0000 mon.a (mon.0) 1085 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: cluster 2024-04-02T04:49:38.716179+0000 mon.a (mon.0) 1086 : cluster 3 Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.722699+0000 mon.a (mon.0) 1087 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.723114+0000 mon.a (mon.0) 1088 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.723420+0000 mon.a (mon.0) 1089 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.724565+0000 mon.a (mon.0) 1090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.726167+0000 mon.a (mon.0) 1091 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.727204+0000 mon.a (mon.0) 1092 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.732637+0000 mon.a (mon.0) 1093 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.732919+0000 mon.a (mon.0) 1094 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.733026+0000 mon.a (mon.0) 1095 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:38.767498+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:39.335512+0000 mon.b (mon.1) 126 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: cluster 2024-04-02T04:49:39.595563+0000 mon.a (mon.0) 1096 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: audit 2024-04-02T04:49:39.615554+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[28904]: cluster 2024-04-02T04:49:40.064037+0000 mon.a (mon.0) 1097 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.362385+0000 mon.a (mon.0) 1078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.362523+0000 mon.a (mon.0) 1079 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.362697+0000 mon.a (mon.0) 1080 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.362856+0000 mon.a (mon.0) 1081 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.363050+0000 mon.a (mon.0) 1082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.363183+0000 mon.a (mon.0) 1083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.363417+0000 mon.a (mon.0) 1084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.371178+0000 mon.b (mon.1) 124 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.371329+0000 mon.c (mon.2) 94 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.371523+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.371687+0000 mon.c (mon.2) 96 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:37.611479+0000 mon.c (mon.2) 97 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.614079+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.713620+0000 mon.b (mon.1) 125 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: cluster 2024-04-02T04:49:38.715539+0000 mon.a (mon.0) 1085 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: cluster 2024-04-02T04:49:38.716179+0000 mon.a (mon.0) 1086 : cluster 3 Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.722699+0000 mon.a (mon.0) 1087 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.723114+0000 mon.a (mon.0) 1088 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.723420+0000 mon.a (mon.0) 1089 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.724565+0000 mon.a (mon.0) 1090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.726167+0000 mon.a (mon.0) 1091 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.727204+0000 mon.a (mon.0) 1092 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.732637+0000 mon.a (mon.0) 1093 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.732919+0000 mon.a (mon.0) 1094 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.733026+0000 mon.a (mon.0) 1095 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:38.767498+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:39.335512+0000 mon.b (mon.1) 126 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: cluster 2024-04-02T04:49:39.595563+0000 mon.a (mon.0) 1096 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: audit 2024-04-02T04:49:39.615554+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:42 smithi069 bash[21528]: cluster 2024-04-02T04:49:40.064037+0000 mon.a (mon.0) 1097 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:42.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:42.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:42.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.362385+0000 mon.a (mon.0) 1078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.362523+0000 mon.a (mon.0) 1079 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-72190-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.362697+0000 mon.a (mon.0) 1080 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi069-72924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.362856+0000 mon.a (mon.0) 1081 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.363050+0000 mon.a (mon.0) 1082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.363183+0000 mon.a (mon.0) 1083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-72410-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.363417+0000 mon.a (mon.0) 1084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.371178+0000 mon.b (mon.1) 124 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.371329+0000 mon.c (mon.2) 94 : audit 1 from='client.? 172.21.15.69:0/2962558957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.371523+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.371687+0000 mon.c (mon.2) 96 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:37.611479+0000 mon.c (mon.2) 97 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.614079+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.713620+0000 mon.b (mon.1) 125 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: cluster 2024-04-02T04:49:38.715539+0000 mon.a (mon.0) 1085 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: cluster 2024-04-02T04:49:38.716179+0000 mon.a (mon.0) 1086 : cluster 3 Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.722699+0000 mon.a (mon.0) 1087 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.723114+0000 mon.a (mon.0) 1088 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:42.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.723420+0000 mon.a (mon.0) 1089 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.724565+0000 mon.a (mon.0) 1090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.726167+0000 mon.a (mon.0) 1091 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.727204+0000 mon.a (mon.0) 1092 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.732637+0000 mon.a (mon.0) 1093 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.732919+0000 mon.a (mon.0) 1094 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.733026+0000 mon.a (mon.0) 1095 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:38.767498+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:39.335512+0000 mon.b (mon.1) 126 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: cluster 2024-04-02T04:49:39.595563+0000 mon.a (mon.0) 1096 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: audit 2024-04-02T04:49:39.615554+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:42.536 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[23790]: cluster 2024-04-02T04:49:40.064037+0000 mon.a (mon.0) 1097 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:42.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:42.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:42.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:42.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:42.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:42.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:43.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:43.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:43.178 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:49:42 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:49:43.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:43.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:43.320 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:43.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:43.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:43.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:43.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:43.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:43.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:43.842 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:43.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:43.970 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:43.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:44.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:44.202 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:44.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:44.229 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:44.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:44.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.489 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:44.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:49:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:49:44.658 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:49:44.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:49:44.658 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:49:44.658 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-02T04:49:44.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7 2024-04-02T04:49:44.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:44.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:44.863 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:44.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:44.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:49:44 smithi098 systemd[1]: Stopping Ceph osd.7 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T04:49:44.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:49:44 smithi098 bash[44175]: debug 2024-04-02T04:49:44.796+0000 7f2b10b12700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T04:49:44.974 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:49:44 smithi098 bash[44175]: debug 2024-04-02T04:49:44.796+0000 7f2b10b12700 -1 osd.7 150 *** Got signal Terminated *** 2024-04-02T04:49:44.974 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:49:44 smithi098 bash[44175]: debug 2024-04-02T04:49:44.796+0000 7f2b10b12700 -1 osd.7 150 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:49:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:44.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:45.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:45.120 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:45.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:45.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.722699+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.723114+0000 mon.a [INF] from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.723420+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.724565+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.726167+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.727204+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.732637+0000 mon.a [INF] from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-72169-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.732919+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-72161-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.733026+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-72423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:38.767498+0000 mon.c [INF] from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:39.335512+0000 mon.b [INF] from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:40.064037+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:39.339812+0000 client.admin [INF] threexx 2024-04-02T04:49:45.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:40.066329+0000 mon.a [INF] from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]': finished 2024-04-02T04:49:45.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:40.066383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:45.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:41.355419+0000 mon.b [INF] from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.337 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2 cls: Running main() from gmock_main.cc 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] Global test environment set-up. 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ OK ] NeoRadosCls.DNE (18847 ms) 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (16225 ms) 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] 2 tests from NeoRadosCls (35072 ms total) 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] Global test environment tear-down 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [==========] 2 tests from 1 test suite ran. (35072 ms total) 2024-04-02T04:49:45.338 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ PASSED ] 2 tests. 2024-04-02T04:49:45.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: cluster 2024-04-02T04:49:37.863279+0000 mgr.y (mgr.14538) 225 : cluster 0 pgmap v193: 676 pgs: 30 creating+peering, 2 unknown, 644 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: cluster 2024-04-02T04:49:39.339812+0000 client.admin (client.?) 0 : cluster 1 threexx 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: cluster 2024-04-02T04:49:39.864108+0000 mgr.y (mgr.14538) 226 : cluster 0 pgmap v195: 700 pgs: 30 creating+peering, 26 unknown, 644 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 1 op/s 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:40.066329+0000 mon.a (mon.0) 1098 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]': finished 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:40.066383+0000 mon.a (mon.0) 1099 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:40.616840+0000 mon.c (mon.2) 101 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:41.062594+0000 mon.c (mon.2) 102 : audit 0 from='client.? 172.21.15.69:0/1526268698' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:41.062801+0000 mgr.y (mgr.14538) 227 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:41.355419+0000 mon.b (mon.1) 127 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:41.618360+0000 mon.c (mon.2) 103 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:41.931321+0000 mon.c (mon.2) 104 : audit 0 from='client.? 172.21.15.69:0/872957453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:45.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: cluster 2024-04-02T04:49:42.237504+0000 mon.a (mon.0) 1100 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:42.242867+0000 mon.a (mon.0) 1101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:42.253058+0000 mon.a (mon.0) 1102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:42.253860+0000 mon.a (mon.0) 1103 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:42.258176+0000 mon.a (mon.0) 1104 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[21528]: audit 2024-04-02T04:49:42.260267+0000 mon.a (mon.0) 1105 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: cluster 2024-04-02T04:49:37.863279+0000 mgr.y (mgr.14538) 225 : cluster 0 pgmap v193: 676 pgs: 30 creating+peering, 2 unknown, 644 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: cluster 2024-04-02T04:49:39.339812+0000 client.admin (client.?) 0 : cluster 1 threexx 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: cluster 2024-04-02T04:49:39.864108+0000 mgr.y (mgr.14538) 226 : cluster 0 pgmap v195: 700 pgs: 30 creating+peering, 26 unknown, 644 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 1 op/s 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:40.066329+0000 mon.a (mon.0) 1098 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]': finished 2024-04-02T04:49:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:40.066383+0000 mon.a (mon.0) 1099 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:40.616840+0000 mon.c (mon.2) 101 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:41.062594+0000 mon.c (mon.2) 102 : audit 0 from='client.? 172.21.15.69:0/1526268698' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:41.062801+0000 mgr.y (mgr.14538) 227 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:41.355419+0000 mon.b (mon.1) 127 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:41.618360+0000 mon.c (mon.2) 103 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:41.931321+0000 mon.c (mon.2) 104 : audit 0 from='client.? 172.21.15.69:0/872957453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: cluster 2024-04-02T04:49:42.237504+0000 mon.a (mon.0) 1100 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:42.242867+0000 mon.a (mon.0) 1101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:42.253058+0000 mon.a (mon.0) 1102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:42.253860+0000 mon.a (mon.0) 1103 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:42.258176+0000 mon.a (mon.0) 1104 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:45 smithi069 bash[28904]: audit 2024-04-02T04:49:42.260267+0000 mon.a (mon.0) 1105 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:45.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:45.490 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: Running main() from gmock_main.cc 2024-04-02T04:49:45.490 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-04-02T04:49:45.490 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-02T04:49:45.490 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-04-02T04:49:45.490 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: seed 72169 2024-04-02T04:49:45.490 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (520 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (22 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (7 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (7 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (7 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-04-02T04:49:45.491 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (9 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (9 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (8 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (7 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-04-02T04:49:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (30 ms) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (12 ms) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (9 ms) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (8 ms) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (686 ms total) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (5838 ms) 2024-04-02T04:49:45.493 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (21 ms) 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (11 ms) 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (6 ms) 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-02T04:49:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (6 ms) 2024-04-02T04:49:45.518 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:45.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/2608328725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01069b0 msgr2=0x7f6ab0106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:45.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/2608328725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01069b0 0x7f6ab0106d90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000aaa0 tx=0x7f6aa000adb0 comp rx=0 tx=0).stop 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/2608328725 shutdown_connections 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/2608328725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab01015c0 0x7f6ab010d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/2608328725 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01069b0 0x7f6ab0106d90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/2608328725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab0101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/2608328725 >> 172.21.15.69:0/2608328725 conn(0x7f6ab0076460 msgr2=0x7f6ab0076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/2608328725 shutdown_connections 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/2608328725 wait complete. 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 librados: starting msgr at 2024-04-02T04:49:45.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 librados: starting objecter 2024-04-02T04:49:45.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 Processor -- start 2024-04-02T04:49:45.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- start start 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 librados: setting wanted keys 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 librados: calling monclient init 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 0x7f6ab019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6aae59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6aae59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39358/0 (socket says 172.21.15.69:39358) 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6aadd9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 0x7f6ab019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6aadd9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 0x7f6ab019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:54180/0 (socket says 172.21.15.69:54180) 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab01069b0 0x7f6ab019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6aae59c700 1 -- 172.21.15.69:0/714095609 learned_addr learned my addr 172.21.15.69:0/714095609 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab010f920 con 0x7f6ab0100c00 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab010f7a0 con 0x7f6ab01069b0 2024-04-02T04:49:45.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab010faa0 con 0x7f6ab01015c0 2024-04-02T04:49:45.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.557+0000 7f6aaed9d700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab01069b0 0x7f6ab019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:45.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aadd9b700 1 -- 172.21.15.69:0/714095609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab01069b0 msgr2=0x7f6ab019ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:45.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aadd9b700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab01069b0 0x7f6ab019ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:45.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aadd9b700 1 -- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 msgr2=0x7f6ab019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aadd9b700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aadd9b700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab019f170 con 0x7f6ab01015c0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aae59c700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6aadd9b700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 0x7f6ab019a6a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9f0 tx=0x7f6aa400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6a9f7fe700 10 client.?.objecter ms_handle_connect 0x7f6ab01015c0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6a9f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400c9c0 con 0x7f6ab01015c0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6aa4010430 con 0x7f6ab01015c0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4018680 con 0x7f6ab01015c0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab019f460 con 0x7f6ab01015c0 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6ab4dbe700 10 client.25364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:49:45.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab0107710 con 0x7f6ab01015c0 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6ab4dbe700 1 librados: init done 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6aa4018820 con 0x7f6ab01015c0 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.561+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab004ff00 con 0x7f6ab01015c0 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a98077df0 0x7f6a9807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(151..151 src has 74..151) ==== 22444+0+0 (secure 0 0 0) 0x7f6aa4091b80 con 0x7f6ab01015c0 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 10 client.25364.objecter ms_dispatch 0x7f6ab0001550 osd_map(151..151 src has 74..151) 2024-04-02T04:49:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 3 client.25364.objecter handle_osd_map got epochs [151,151] > 0 2024-04-02T04:49:45.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 3 client.25364.objecter handle_osd_map decoding full epoch 151 2024-04-02T04:49:45.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6aae59c700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a98077df0 0x7f6a9807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:49:45.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 20 client.25364.objecter dump_active .. 0 homeless 2024-04-02T04:49:45.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6a9f7fe700 10 client.25364.objecter ms_handle_connect 0x7f6a98077df0 2024-04-02T04:49:45.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.565+0000 7f6aae59c700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a98077df0 0x7f6a9807a2b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0006010 tx=0x7f6aa000c650 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:49:45.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.569+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6aa405b700 con 0x7f6ab01015c0 2024-04-02T04:49:45.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:45.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:49:45.640 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: cluster 2024-04-02T04:49:37.863279+0000 mgr.y (mgr.14538) 225 : cluster 0 pgmap v193: 676 pgs: 30 creating+peering, 2 unknown, 644 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: cluster 2024-04-02T04:49:39.339812+0000 client.admin (client.?) 0 : cluster 1 threexx 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: cluster 2024-04-02T04:49:39.864108+0000 mgr.y (mgr.14538) 226 : cluster 0 pgmap v195: 700 pgs: 30 creating+peering, 26 unknown, 644 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 1 op/s 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:40.066329+0000 mon.a (mon.0) 1098 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-4-cache", "mode": "writeback"}]': finished 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:40.066383+0000 mon.a (mon.0) 1099 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-72439-1"}]': finished 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:40.616840+0000 mon.c (mon.2) 101 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:41.062594+0000 mon.c (mon.2) 102 : audit 0 from='client.? 172.21.15.69:0/1526268698' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-02T04:49:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:41.062801+0000 mgr.y (mgr.14538) 227 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:41.355419+0000 mon.b (mon.1) 127 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:41.618360+0000 mon.c (mon.2) 103 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:41.931321+0000 mon.c (mon.2) 104 : audit 0 from='client.? 172.21.15.69:0/872957453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: cluster 2024-04-02T04:49:42.237504+0000 mon.a (mon.0) 1100 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:42.242867+0000 mon.a (mon.0) 1101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:42.253058+0000 mon.a (mon.0) 1102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:42.253860+0000 mon.a (mon.0) 1103 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:42.258176+0000 mon.a (mon.0) 1104 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:45 smithi098 bash[23790]: audit 2024-04-02T04:49:42.260267+0000 mon.a (mon.0) 1105 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:45.743 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:45.762 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:45.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:45.801+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f6ab0107970 con 0x7f6ab01015c0 2024-04-02T04:49:45.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:45.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:45.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:46.021 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:46.123 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:46.145 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:46.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:46.266 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:46.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.305+0000 7f6a9f7fe700 1 -- 172.21.15.69:0/714095609 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v152) ==== 97+0+0 (secure 0 0 0) 0x7f6aa405f150 con 0x7f6ab01015c0 2024-04-02T04:49:46.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.313+0000 7f6ab4dbe700 10 librados: watch_flush enter 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 10 librados: watch_flush exit 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 20 client.25364.objecter shutdown clearing up homeless session... 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 10 client.25364.objecter successfully canceled tick 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a98077df0 msgr2=0x7f6a9807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a98077df0 0x7f6a9807a2b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0006010 tx=0x7f6aa000c650 comp rx=0 tx=0).stop 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 msgr2=0x7f6ab019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.321+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 0x7f6ab019a6a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9f0 tx=0x7f6aa400ed00 comp rx=0 tx=0).stop 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.325+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 shutdown_connections 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.325+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a98077df0 0x7f6a9807a2b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.325+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab01069b0 0x7f6ab019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.325+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0100c00 0x7f6ab019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:46.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.325+0000 7f6ab4dbe700 1 --2- 172.21.15.69:0/714095609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab01015c0 0x7f6ab019a6a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:49:46.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.325+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 >> 172.21.15.69:0/714095609 conn(0x7f6ab0076460 msgr2=0x7f6ab00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:49:46.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.329+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 shutdown_connections 2024-04-02T04:49:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.329+0000 7f6ab4dbe700 1 -- 172.21.15.69:0/714095609 wait complete. 2024-04-02T04:49:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:49:46.329+0000 7f6ab4dbe700 1 librados: shutdown 2024-04-02T04:49:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:49:46.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:41.743774+0000 osd.2 (osd.2) 81 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:41.743896+0000 osd.2 (osd.2) 82 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:41.865410+0000 mgr.y (mgr.14538) 228 : cluster 0 pgmap v197: 732 pgs: 30 creating+peering, 122 unknown, 580 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1 op/s 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:42.260383+0000 mon.a (mon.0) 1106 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:42.260503+0000 mon.a (mon.0) 1107 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:42.260628+0000 mon.a (mon.0) 1108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:42.620052+0000 mon.c (mon.2) 105 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:42.708389+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:43.000632+0000 mgr.y (mgr.14538) 229 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:43.621582+0000 mon.c (mon.2) 107 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:43.718425+0000 osd.3 (osd.3) 169 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:43.718544+0000 osd.3 (osd.3) 170 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:43.871153+0000 mgr.y (mgr.14538) 230 : cluster 0 pgmap v199: 764 pgs: 172 creating+peering, 11 unknown, 581 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 1 op/s 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:44.622772+0000 mon.c (mon.2) 108 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:45.278291+0000 mon.a (mon.0) 1109 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:45.279489+0000 mon.a (mon.0) 1110 : cluster 3 Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.289103+0000 mon.a (mon.0) 1111 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-4"}]: dispatch 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.292113+0000 mon.a (mon.0) 1112 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.294084+0000 mon.a (mon.0) 1113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.298065+0000 mon.a (mon.0) 1114 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.298252+0000 mon.a (mon.0) 1115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:46.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.300881+0000 mon.a (mon.0) 1116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:45.310775+0000 mon.a (mon.0) 1117 : cluster 1 osd.7 marked itself down and dead 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:45.310826+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.319319+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.322736+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.69:0/137483179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.323241+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.325334+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.69:0/859329830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.328570+0000 mon.b (mon.1) 128 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.329337+0000 mon.b (mon.1) 129 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.329970+0000 mon.b (mon.1) 130 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.333317+0000 mon.a (mon.0) 1119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.334479+0000 mon.b (mon.1) 131 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.335310+0000 mon.a (mon.0) 1120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.335704+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.336105+0000 mon.a (mon.0) 1122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: cluster 2024-04-02T04:49:45.336626+0000 client.admin (client.?) 0 : cluster 1 fourxx 2024-04-02T04:49:46.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.336707+0000 mon.a (mon.0) 1123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.337205+0000 mon.a (mon.0) 1124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.623892+0000 mon.c (mon.2) 113 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.807801+0000 mon.c (mon.2) 114 : audit 1 from='client.? 172.21.15.69:0/714095609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:45.835657+0000 mon.a (mon.0) 1125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[21528]: audit 2024-04-02T04:49:46.249772+0000 mon.a (mon.0) 1126 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:41.743774+0000 osd.2 (osd.2) 81 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:41.743896+0000 osd.2 (osd.2) 82 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:41.865410+0000 mgr.y (mgr.14538) 228 : cluster 0 pgmap v197: 732 pgs: 30 creating+peering, 122 unknown, 580 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1 op/s 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:42.260383+0000 mon.a (mon.0) 1106 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:42.260503+0000 mon.a (mon.0) 1107 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:42.260628+0000 mon.a (mon.0) 1108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:42.620052+0000 mon.c (mon.2) 105 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:42.708389+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:43.000632+0000 mgr.y (mgr.14538) 229 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:43.621582+0000 mon.c (mon.2) 107 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:43.718425+0000 osd.3 (osd.3) 169 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:43.718544+0000 osd.3 (osd.3) 170 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:43.871153+0000 mgr.y (mgr.14538) 230 : cluster 0 pgmap v199: 764 pgs: 172 creating+peering, 11 unknown, 581 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 1 op/s 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:44.622772+0000 mon.c (mon.2) 108 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:45.278291+0000 mon.a (mon.0) 1109 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:45.279489+0000 mon.a (mon.0) 1110 : cluster 3 Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.289103+0000 mon.a (mon.0) 1111 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-4"}]: dispatch 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.292113+0000 mon.a (mon.0) 1112 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.294084+0000 mon.a (mon.0) 1113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.298065+0000 mon.a (mon.0) 1114 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.298252+0000 mon.a (mon.0) 1115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.300881+0000 mon.a (mon.0) 1116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-02T04:49:46.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:45.310775+0000 mon.a (mon.0) 1117 : cluster 1 osd.7 marked itself down and dead 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:45.310826+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.319319+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.322736+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.69:0/137483179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.323241+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.325334+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.69:0/859329830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.328570+0000 mon.b (mon.1) 128 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.329337+0000 mon.b (mon.1) 129 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.329970+0000 mon.b (mon.1) 130 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.333317+0000 mon.a (mon.0) 1119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.334479+0000 mon.b (mon.1) 131 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.335310+0000 mon.a (mon.0) 1120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.335704+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.336105+0000 mon.a (mon.0) 1122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: cluster 2024-04-02T04:49:45.336626+0000 client.admin (client.?) 0 : cluster 1 fourxx 2024-04-02T04:49:46.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.336707+0000 mon.a (mon.0) 1123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.337205+0000 mon.a (mon.0) 1124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-02T04:49:46.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.623892+0000 mon.c (mon.2) 113 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.807801+0000 mon.c (mon.2) 114 : audit 1 from='client.? 172.21.15.69:0/714095609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:49:46.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:45.835657+0000 mon.a (mon.0) 1125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-02T04:49:46.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:46 smithi069 bash[28904]: audit 2024-04-02T04:49:46.249772+0000 mon.a (mon.0) 1126 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:49:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:46.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:46.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:41.743774+0000 osd.2 (osd.2) 81 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:41.743896+0000 osd.2 (osd.2) 82 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:41.865410+0000 mgr.y (mgr.14538) 228 : cluster 0 pgmap v197: 732 pgs: 30 creating+peering, 122 unknown, 580 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1 op/s 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:42.260383+0000 mon.a (mon.0) 1106 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-72210-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:42.260503+0000 mon.a (mon.0) 1107 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-72190-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:42.260628+0000 mon.a (mon.0) 1108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-72410-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:42.620052+0000 mon.c (mon.2) 105 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:42.708389+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:43.000632+0000 mgr.y (mgr.14538) 229 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:46.642 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:43.621582+0000 mon.c (mon.2) 107 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:43.718425+0000 osd.3 (osd.3) 169 : cluster 0 purged_snaps scrub starts 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:43.718544+0000 osd.3 (osd.3) 170 : cluster 0 purged_snaps scrub ok 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:43.871153+0000 mgr.y (mgr.14538) 230 : cluster 0 pgmap v199: 764 pgs: 172 creating+peering, 11 unknown, 581 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 1 op/s 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:44.622772+0000 mon.c (mon.2) 108 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:45.278291+0000 mon.a (mon.0) 1109 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:45.279489+0000 mon.a (mon.0) 1110 : cluster 3 Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.289103+0000 mon.a (mon.0) 1111 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-4"}]: dispatch 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.292113+0000 mon.a (mon.0) 1112 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.294084+0000 mon.a (mon.0) 1113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.298065+0000 mon.a (mon.0) 1114 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.298252+0000 mon.a (mon.0) 1115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.300881+0000 mon.a (mon.0) 1116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:45.310775+0000 mon.a (mon.0) 1117 : cluster 1 osd.7 marked itself down and dead 2024-04-02T04:49:46.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:45.310826+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.319319+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.322736+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.69:0/137483179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.323241+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.325334+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.69:0/859329830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.328570+0000 mon.b (mon.1) 128 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.329337+0000 mon.b (mon.1) 129 : audit 1 from='client.? 172.21.15.69:0/3153801708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.329970+0000 mon.b (mon.1) 130 : audit 1 from='client.? 172.21.15.69:0/1355238607' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.333317+0000 mon.a (mon.0) 1119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.334479+0000 mon.b (mon.1) 131 : audit 1 from='client.? 172.21.15.69:0/4011117274' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.335310+0000 mon.a (mon.0) 1120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.335704+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.336105+0000 mon.a (mon.0) 1122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: cluster 2024-04-02T04:49:45.336626+0000 client.admin (client.?) 0 : cluster 1 fourxx 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.336707+0000 mon.a (mon.0) 1123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.337205+0000 mon.a (mon.0) 1124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.623892+0000 mon.c (mon.2) 113 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.807801+0000 mon.c (mon.2) 114 : audit 1 from='client.? 172.21.15.69:0/714095609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:45.835657+0000 mon.a (mon.0) 1125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-02T04:49:46.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:46 smithi098 bash[23790]: audit 2024-04-02T04:49:46.249772+0000 mon.a (mon.0) 1126 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:49:46.655 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:46.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:46.784 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:46.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:46.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:46.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:47.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:47.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:47.162 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:47.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:47.292 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:47.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:47.423 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:47.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:47.545 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:47.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN api_service: [==========] Running 4 tests from 1 test suite. 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment set-up. 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5065 ms) 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-02T04:49:47.671 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterLate (791 ms) 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2 pool(s) do not have an application enabled 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 8 pools, 196 pgs 2024-04-02T04:49:47.672 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 230 objects, 582 KiB 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 239 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 5.612% pgs unknown 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 27.041% pgs not active 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 132 active+clean 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 53 creating+peering 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 11 unknown 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: io: 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T04:49:47.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: noscrub flag(s) set 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 31 pool(s) do not have an application enabled 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: flags noscrub 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 41 pools, 1252 pgs 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 230 objects, 582 KiB 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 256 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 27.636% pgs unknown 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 61.821% pgs not active 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 774 creating+peering 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 346 unknown 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 132 active+clean 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: io: 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: client: 638 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.StatusFormat (11510 ms) 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-02T04:49:47.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.Status (20144 ms) 2024-04-02T04:49:47.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService (37510 ms total) 2024-04-02T04:49:47.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-04-02T04:49:47.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment tear-down 2024-04-02T04:49:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:47.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==========] 4 tests from 1 test suite ran. (37510 ms total) 2024-04-02T04:49:47.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ PASSED ] 4 tests. 2024-04-02T04:49:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:47.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:47.804 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:47.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:47.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:47.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: cluster 2024-04-02T04:49:45.871910+0000 mgr.y (mgr.14538) 231 : cluster 0 pgmap v201: 924 pgs: 140 creating+peering, 235 unknown, 549 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: cluster 2024-04-02T04:49:46.301825+0000 mon.a (mon.0) 1127 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: cluster 2024-04-02T04:49:46.301854+0000 mon.a (mon.0) 1128 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304509+0000 mon.a (mon.0) 1129 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-4"}]': finished 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304594+0000 mon.a (mon.0) 1130 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304684+0000 mon.a (mon.0) 1131 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304760+0000 mon.a (mon.0) 1132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:47.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304834+0000 mon.a (mon.0) 1133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304885+0000 mon.a (mon.0) 1134 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304943+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.304990+0000 mon.a (mon.0) 1136 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: cluster 2024-04-02T04:49:46.309672+0000 mon.a (mon.0) 1137 : cluster 0 osdmap e152: 8 total, 7 up, 8 in 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.313063+0000 mon.a (mon.0) 1138 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.313799+0000 mon.c (mon.2) 115 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.313945+0000 mon.c (mon.2) 116 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.316645+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.326429+0000 mon.b (mon.1) 132 : audit 1 from='client.? 172.21.15.69:0/3364586528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.331182+0000 mon.a (mon.0) 1139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.345162+0000 mon.a (mon.0) 1140 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.346396+0000 mon.a (mon.0) 1141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:47.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.346747+0000 mon.a (mon.0) 1142 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.347025+0000 mon.a (mon.0) 1143 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.347379+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.377094+0000 mon.a (mon.0) 1145 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi069-72950-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.529118+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.624671+0000 mon.c (mon.2) 119 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:46.856140+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]: dispatch 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: cluster 2024-04-02T04:49:47.325895+0000 mon.a (mon.0) 1147 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:49:47.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:47 smithi098 bash[23790]: audit 2024-04-02T04:49:47.329836+0000 mon.a (mon.0) 1148 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:48.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:48.166 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:48.177 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: cluster 2024-04-02T04:49:45.871910+0000 mgr.y (mgr.14538) 231 : cluster 0 pgmap v201: 924 pgs: 140 creating+peering, 235 unknown, 549 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail 2024-04-02T04:49:48.177 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: cluster 2024-04-02T04:49:46.301825+0000 mon.a (mon.0) 1127 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:49:48.177 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: cluster 2024-04-02T04:49:46.301854+0000 mon.a (mon.0) 1128 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304509+0000 mon.a (mon.0) 1129 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-4"}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304594+0000 mon.a (mon.0) 1130 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304684+0000 mon.a (mon.0) 1131 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304760+0000 mon.a (mon.0) 1132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304834+0000 mon.a (mon.0) 1133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304885+0000 mon.a (mon.0) 1134 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304943+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.304990+0000 mon.a (mon.0) 1136 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: cluster 2024-04-02T04:49:46.309672+0000 mon.a (mon.0) 1137 : cluster 0 osdmap e152: 8 total, 7 up, 8 in 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.313063+0000 mon.a (mon.0) 1138 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.313799+0000 mon.c (mon.2) 115 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:48.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.313945+0000 mon.c (mon.2) 116 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: cluster 2024-04-02T04:49:45.871910+0000 mgr.y (mgr.14538) 231 : cluster 0 pgmap v201: 924 pgs: 140 creating+peering, 235 unknown, 549 active+clean; 145 MiB data, 738 MiB used, 714 GiB / 715 GiB avail 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: cluster 2024-04-02T04:49:46.301825+0000 mon.a (mon.0) 1127 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: cluster 2024-04-02T04:49:46.301854+0000 mon.a (mon.0) 1128 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304509+0000 mon.a (mon.0) 1129 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-4"}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304594+0000 mon.a (mon.0) 1130 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304684+0000 mon.a (mon.0) 1131 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi069-72950-1"}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304760+0000 mon.a (mon.0) 1132 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-72145-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304834+0000 mon.a (mon.0) 1133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304885+0000 mon.a (mon.0) 1134 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304943+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-72153-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.304990+0000 mon.a (mon.0) 1136 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:49:48.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: cluster 2024-04-02T04:49:46.309672+0000 mon.a (mon.0) 1137 : cluster 0 osdmap e152: 8 total, 7 up, 8 in 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.313063+0000 mon.a (mon.0) 1138 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.313799+0000 mon.c (mon.2) 115 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.313945+0000 mon.c (mon.2) 116 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.316645+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.326429+0000 mon.b (mon.1) 132 : audit 1 from='client.? 172.21.15.69:0/3364586528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.331182+0000 mon.a (mon.0) 1139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.345162+0000 mon.a (mon.0) 1140 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.346396+0000 mon.a (mon.0) 1141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.346747+0000 mon.a (mon.0) 1142 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.347025+0000 mon.a (mon.0) 1143 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.347379+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:48.180 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.377094+0000 mon.a (mon.0) 1145 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi069-72950-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.529118+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.624671+0000 mon.c (mon.2) 119 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:46.856140+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: cluster 2024-04-02T04:49:47.325895+0000 mon.a (mon.0) 1147 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[28904]: audit 2024-04-02T04:49:47.329836+0000 mon.a (mon.0) 1148 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.316645+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.326429+0000 mon.b (mon.1) 132 : audit 1 from='client.? 172.21.15.69:0/3364586528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.331182+0000 mon.a (mon.0) 1139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.345162+0000 mon.a (mon.0) 1140 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.346396+0000 mon.a (mon.0) 1141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:48.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.346747+0000 mon.a (mon.0) 1142 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.347025+0000 mon.a (mon.0) 1143 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.347379+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.377094+0000 mon.a (mon.0) 1145 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi069-72950-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.529118+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.624671+0000 mon.c (mon.2) 119 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:46.856140+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]: dispatch 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: cluster 2024-04-02T04:49:47.325895+0000 mon.a (mon.0) 1147 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:49:48.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:47 smithi069 bash[21528]: audit 2024-04-02T04:49:47.329836+0000 mon.a (mon.0) 1148 : audit 1 from='client.? 172.21.15.69:0/3055876148' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-4", "tierpool": "test-rados-api-smithi069-72349-4-cache"}]': finished 2024-04-02T04:49:48.190 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:48.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:48.316 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout:024-04-02T04:49:42.242867+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:42.253058+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi069-72950-1"}]: dispatch 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:42.253860+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:42.258176+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-72153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-04-02T04:49:42.260267+0000 mon.a [INF] from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi069-72924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (18624 ms) 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (38222 ms total) 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (38222 ms total) 2024-04-02T04:49:48.345 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-02T04:49:48.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:48.438 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:48.542 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:48.565 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:48.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: seed 72423 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (402 ms) 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (99 ms) 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-02T04:49:48.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (13 ms) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (27 ms) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (541 ms total) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (6673 ms) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (6 ms) 2024-04-02T04:49:48.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (29 ms) 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (6709 ms total) 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (38673 ms total) 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: Running main() from gmock_main.cc 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment set-up. 2024-04-02T04:49:48.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (140 ms) 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (23 ms) 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-02T04:49:48.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (1249 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (121 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (16 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (14 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (4 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TruncTest (13 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (16 ms) 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-02T04:49:48.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (9 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RmXattr (20 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrIter (7 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo (1636 ms total) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (6698 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (9 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (11 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (7 ms) 2024-04-02T04:49:48.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (9 ms) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (7 ms) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (8 ms) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (13 ms) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (7 ms) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC (6772 ms total) 2024-04-02T04:49:48.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2024-04-02T04:49:48.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment tear-down 2024-04-02T04:49:48.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] 24 tests from 2 test suites ran. (38806 ms total) 2024-04-02T04:49:48.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ PASSED ] 24 tests. 2024-04-02T04:49:48.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:48.819 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:48.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:48.932 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:49:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:49:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:49:48.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:48.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:49.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:49.134 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:49:48 smithi098 bash[57367]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T04:49:49.134 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:49:49 smithi098 bash[57562]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T04:49:49.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:49.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:49.221 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: Running main() from gmock_main.cc 2024-04-02T04:49:49.221 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-02T04:49:49.221 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] Global test environment set-up. 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (16848 ms) 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (10733 ms) 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (11362 ms) 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] 3 tests from NeoRadosCmd (38943 ms total) 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] Global test environment tear-down 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [==========] 3 tests from 1 test suite ran. (38944 ms total) 2024-04-02T04:49:49.222 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ PASSED ] 3 tests. 2024-04-02T04:49:49.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:49.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:49.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:49.453 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:49.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:49.569 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:49:49.569 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:49:49.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (7 ms) 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (6 ms) 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (8 ms) 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-04-02T04:49:49.644 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (5 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (10 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (9 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (6 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (8 ms) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (5963 ms total) 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2024-04-02T04:49:49.645 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-02T04:49:49.646 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (39682 ms total) 2024-04-02T04:49:49.646 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-04-02T04:49:49.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:49.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:49.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:49.968 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:49.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:50.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: Running main() from gmock_main.cc 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] Global test environment set-up. 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (24728 ms) 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (15032 ms) 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (39760 ms total) 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] Global test environment tear-down 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (39760 ms total) 2024-04-02T04:49:50.072 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-02T04:49:50.087 INFO:tasks.workunit.client.0.smithi069.stdout: list: Running main() from gmock_main.cc 2024-04-02T04:49:50.087 INFO:tasks.workunit.client.0.smithi069.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-02T04:49:50.087 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] Global test environment set-up. 2024-04-02T04:49:50.087 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] 3 tests from NeoradosList 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ OK ] NeoradosList.ListObjects (19396 ms) 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ OK ] NeoradosList.ListObjectsNS (9012 ms) 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ OK ] NeoradosList.ListObjectsMany (11360 ms) 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] 3 tests from NeoradosList (39768 ms total) 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] Global test environment tear-down 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [==========] 3 tests from 1 test suite ran. (39768 ms total) 2024-04-02T04:49:50.088 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ PASSED ] 3 tests. 2024-04-02T04:49:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:50.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:50.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.330014+0000 mon.a (mon.0) 1149 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.330154+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.330325+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.330500+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.330677+0000 mon.a (mon.0) 1153 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi069-72950-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.330806+0000 mon.a (mon.0) 1154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]': finished 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.343099+0000 mon.c (mon.2) 120 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.343471+0000 mon.c (mon.2) 121 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.343728+0000 mon.c (mon.2) 122 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.625725+0000 mon.c (mon.2) 123 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: cluster 2024-04-02T04:49:47.848091+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e153: 8 total, 7 up, 8 in 2024-04-02T04:49:50.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.850223+0000 mon.a (mon.0) 1156 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.857841+0000 mon.a (mon.0) 1157 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi069-72950-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.858256+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.858618+0000 mon.a (mon.0) 1159 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.858912+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:47.859270+0000 mon.a (mon.0) 1161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: cluster 2024-04-02T04:49:47.878565+0000 mgr.y (mgr.14538) 232 : cluster 0 pgmap v204: 864 pgs: 1 active+clean+snaptrim, 249 active+undersized, 527 active+clean, 31 creating+peering, 5 unknown, 51 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 547 B/s wr, 5 op/s; 148/1485 objects degraded (9.966%) 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:48.626571+0000 mon.c (mon.2) 124 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[28904]: audit 2024-04-02T04:49:48.750066+0000 mon.a (mon.0) 1162 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.330014+0000 mon.a (mon.0) 1149 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:50.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.330154+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.330325+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.330500+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.330677+0000 mon.a (mon.0) 1153 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi069-72950-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.330806+0000 mon.a (mon.0) 1154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]': finished 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.343099+0000 mon.c (mon.2) 120 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.343471+0000 mon.c (mon.2) 121 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.343728+0000 mon.c (mon.2) 122 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.625725+0000 mon.c (mon.2) 123 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: cluster 2024-04-02T04:49:47.848091+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e153: 8 total, 7 up, 8 in 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.850223+0000 mon.a (mon.0) 1156 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.857841+0000 mon.a (mon.0) 1157 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi069-72950-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.858256+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.858618+0000 mon.a (mon.0) 1159 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:50.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.858912+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:50.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:47.859270+0000 mon.a (mon.0) 1161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:49:50.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: cluster 2024-04-02T04:49:47.878565+0000 mgr.y (mgr.14538) 232 : cluster 0 pgmap v204: 864 pgs: 1 active+clean+snaptrim, 249 active+undersized, 527 active+clean, 31 creating+peering, 5 unknown, 51 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 547 B/s wr, 5 op/s; 148/1485 objects degraded (9.966%) 2024-04-02T04:49:50.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:48.626571+0000 mon.c (mon.2) 124 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:50.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:50 smithi069 bash[21528]: audit 2024-04-02T04:49:48.750066+0000 mon.a (mon.0) 1162 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:50.239 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:50.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.330014+0000 mon.a (mon.0) 1149 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.330154+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.330325+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.330500+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.330677+0000 mon.a (mon.0) 1153 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi069-72950-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.330806+0000 mon.a (mon.0) 1154 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72439-6", "pg_num": 4}]': finished 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.343099+0000 mon.c (mon.2) 120 : audit 1 from='client.? 172.21.15.69:0/371289681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.343471+0000 mon.c (mon.2) 121 : audit 1 from='client.? 172.21.15.69:0/39118168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.343728+0000 mon.c (mon.2) 122 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.625725+0000 mon.c (mon.2) 123 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: cluster 2024-04-02T04:49:47.848091+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e153: 8 total, 7 up, 8 in 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.850223+0000 mon.a (mon.0) 1156 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]: dispatch 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.857841+0000 mon.a (mon.0) 1157 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi069-72950-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:50.351 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.858256+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]: dispatch 2024-04-02T04:49:50.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.858618+0000 mon.a (mon.0) 1159 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:50.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.858912+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]: dispatch 2024-04-02T04:49:50.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:47.859270+0000 mon.a (mon.0) 1161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:49:50.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: cluster 2024-04-02T04:49:47.878565+0000 mgr.y (mgr.14538) 232 : cluster 0 pgmap v204: 864 pgs: 1 active+clean+snaptrim, 249 active+undersized, 527 active+clean, 31 creating+peering, 5 unknown, 51 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 547 B/s wr, 5 op/s; 148/1485 objects degraded (9.966%) 2024-04-02T04:49:50.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:48.626571+0000 mon.c (mon.2) 124 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:50.352 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:50 smithi098 bash[23790]: audit 2024-04-02T04:49:48.750066+0000 mon.a (mon.0) 1162 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-72465-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:50.361 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:50.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:50.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-02T04:49:50.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:50.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:50.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:50.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:50.981 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:50.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:50.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:49:51.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:51.109 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:51.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:51.231 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:51.333 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:51.356 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:51.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:51.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7.service' 2024-04-02T04:49:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:51.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:49:51.517 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-02T04:49:51.517 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-04-02T04:49:51.517 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi098.front.sepia.ceph.com 2024-04-02T04:49:51.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -n osd.7 --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-04-02T04:49:51.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:51.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/osd.7/config 2024-04-02T04:49:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:51.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:51.857 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:48.750237+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:51.857 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:48.750306+0000 mon.a (mon.0) 1164 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:48.750392+0000 mon.a (mon.0) 1165 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:48.750469+0000 mon.a (mon.0) 1166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:48.750571+0000 mon.a (mon.0) 1167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:48.758260+0000 mon.c (mon.2) 125 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:49.220650+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:49.627528+0000 mon.c (mon.2) 127 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: cluster 2024-04-02T04:49:49.635479+0000 mon.a (mon.0) 1168 : cluster 0 osdmap e154: 8 total, 7 up, 8 in 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: cluster 2024-04-02T04:49:49.635796+0000 mon.a (mon.0) 1169 : cluster 3 Health check failed: Degraded data redundancy: 148/1485 objects degraded (9.966%), 51 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:49.637040+0000 mon.a (mon.0) 1170 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:49.641932+0000 mon.a (mon.0) 1171 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:50.068347+0000 mon.a (mon.0) 1172 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:51.858 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[21528]: audit 2024-04-02T04:49:50.072411+0000 mon.a (mon.0) 1173 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi069-72950-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:48.750237+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:48.750306+0000 mon.a (mon.0) 1164 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:48.750392+0000 mon.a (mon.0) 1165 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:48.750469+0000 mon.a (mon.0) 1166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:48.750571+0000 mon.a (mon.0) 1167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:48.758260+0000 mon.c (mon.2) 125 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:49.220650+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:49.627528+0000 mon.c (mon.2) 127 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: cluster 2024-04-02T04:49:49.635479+0000 mon.a (mon.0) 1168 : cluster 0 osdmap e154: 8 total, 7 up, 8 in 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: cluster 2024-04-02T04:49:49.635796+0000 mon.a (mon.0) 1169 : cluster 3 Health check failed: Degraded data redundancy: 148/1485 objects degraded (9.966%), 51 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:49.637040+0000 mon.a (mon.0) 1170 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:49.641932+0000 mon.a (mon.0) 1171 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:49:51.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:50.068347+0000 mon.a (mon.0) 1172 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:51.860 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:51 smithi069 bash[28904]: audit 2024-04-02T04:49:50.072411+0000 mon.a (mon.0) 1173 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi069-72950-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:51.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:51.979 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:52.006 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:52.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:52.130 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:48.750237+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.69:0/233777584' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-72169-22"}]': finished 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:48.750306+0000 mon.a (mon.0) 1164 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-72161-16"}]': finished 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:48.750392+0000 mon.a (mon.0) 1165 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:48.750469+0000 mon.a (mon.0) 1166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-72423-7"}]': finished 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:48.750571+0000 mon.a (mon.0) 1167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:48.758260+0000 mon.c (mon.2) 125 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:49.220650+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:49.627528+0000 mon.c (mon.2) 127 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: cluster 2024-04-02T04:49:49.635479+0000 mon.a (mon.0) 1168 : cluster 0 osdmap e154: 8 total, 7 up, 8 in 2024-04-02T04:49:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: cluster 2024-04-02T04:49:49.635796+0000 mon.a (mon.0) 1169 : cluster 3 Health check failed: Degraded data redundancy: 148/1485 objects degraded (9.966%), 51 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:49.637040+0000 mon.a (mon.0) 1170 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi069-72924-2"}]: dispatch 2024-04-02T04:49:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:49.641932+0000 mon.a (mon.0) 1171 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:49:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:50.068347+0000 mon.a (mon.0) 1172 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:51 smithi098 bash[23790]: audit 2024-04-02T04:49:50.072411+0000 mon.a (mon.0) 1173 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi069-72950-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:52.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:52.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:52.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:52.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:52.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:52.637 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:52.740 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: Running main() from gmock_main.cc 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment set-up. 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat (413 ms) 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (19 ms) 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.StatNS (94 ms) 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-02T04:49:52.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat (533 ms total) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (4247 ms) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (998 ms) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (5248 ms total) 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2024-04-02T04:49:52.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment tear-down 2024-04-02T04:49:52.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (42729 ms total) 2024-04-02T04:49:52.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-02T04:49:52.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:52.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:49.880327+0000 mgr.y (mgr.14538) 233 : cluster 0 pgmap v206: 704 pgs: 1 active+clean+snaptrim, 168 active+undersized, 380 active+clean, 31 creating+peering, 77 unknown, 47 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s wr, 2 op/s; 144/1449 objects degraded (9.938%) 2024-04-02T04:49:52.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:50.072508+0000 mon.a (mon.0) 1174 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:52.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:50.072576+0000 mon.a (mon.0) 1175 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]': finished 2024-04-02T04:49:52.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:50.081599+0000 mon.b (mon.1) 133 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:50.085067+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]: dispatch 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:50.085473+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.69:0/86834543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:50.628693+0000 mon.c (mon.2) 130 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.630095+0000 mon.c (mon.2) 131 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:51.778859+0000 mon.a (mon.0) 1176 : cluster 0 osdmap e155: 8 total, 7 up, 8 in 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:51.779582+0000 mon.a (mon.0) 1177 : cluster 3 Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:51.781173+0000 mon.a (mon.0) 1178 : cluster 3 pool 'PoolQuotaPP_smithi069-72153-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:51.787195+0000 mon.a (mon.0) 1179 : cluster 3 Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.792468+0000 mon.a (mon.0) 1180 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.793073+0000 mon.a (mon.0) 1181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.800638+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:52.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:51.806428+0000 mon.a (mon.0) 1182 : cluster 0 osdmap e156: 8 total, 7 up, 8 in 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.812986+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.816272+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.69:0/1518543021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.817286+0000 mon.a (mon.0) 1183 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.818796+0000 mon.a (mon.0) 1184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.820214+0000 mon.a (mon.0) 1185 : audit 1 from='client.? 172.21.15.69:0/1813457633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.826019+0000 mon.a (mon.0) 1186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.826880+0000 mon.a (mon.0) 1187 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:51.827370+0000 mon.a (mon.0) 1188 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:51.881309+0000 mgr.y (mgr.14538) 234 : cluster 0 pgmap v209: 760 pgs: 1 active+clean+snaptrim, 149 active+undersized, 327 active+clean, 3 creating+peering, 241 unknown, 39 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 900 B/s wr, 0 op/s; 111/864 objects degraded (12.847%) 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.631352+0000 mon.c (mon.2) 135 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:52.794772+0000 mon.a (mon.0) 1189 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797003+0000 mon.a (mon.0) 1190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797170+0000 mon.a (mon.0) 1191 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797360+0000 mon.a (mon.0) 1192 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797559+0000 mon.a (mon.0) 1193 : audit 1 from='client.? 172.21.15.69:0/1813457633' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797646+0000 mon.a (mon.0) 1194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797719+0000 mon.a (mon.0) 1195 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.797805+0000 mon.a (mon.0) 1196 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: cluster 2024-04-02T04:49:52.804119+0000 mon.a (mon.0) 1197 : cluster 0 osdmap e157: 8 total, 7 up, 8 in 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.807694+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.808060+0000 mon.c (mon.2) 137 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.808593+0000 mon.b (mon.1) 134 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.812472+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:52.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.812889+0000 mon.a (mon.0) 1199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[21528]: audit 2024-04-02T04:49:52.813581+0000 mon.a (mon.0) 1200 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:49.880327+0000 mgr.y (mgr.14538) 233 : cluster 0 pgmap v206: 704 pgs: 1 active+clean+snaptrim, 168 active+undersized, 380 active+clean, 31 creating+peering, 77 unknown, 47 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s wr, 2 op/s; 144/1449 objects degraded (9.938%) 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:50.072508+0000 mon.a (mon.0) 1174 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:50.072576+0000 mon.a (mon.0) 1175 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]': finished 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:50.081599+0000 mon.b (mon.1) 133 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:50.085067+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]: dispatch 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:50.085473+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.69:0/86834543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:50.628693+0000 mon.c (mon.2) 130 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.630095+0000 mon.c (mon.2) 131 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:51.778859+0000 mon.a (mon.0) 1176 : cluster 0 osdmap e155: 8 total, 7 up, 8 in 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:51.779582+0000 mon.a (mon.0) 1177 : cluster 3 Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:51.781173+0000 mon.a (mon.0) 1178 : cluster 3 pool 'PoolQuotaPP_smithi069-72153-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:51.787195+0000 mon.a (mon.0) 1179 : cluster 3 Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.792468+0000 mon.a (mon.0) 1180 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.793073+0000 mon.a (mon.0) 1181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.800638+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:51.806428+0000 mon.a (mon.0) 1182 : cluster 0 osdmap e156: 8 total, 7 up, 8 in 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.812986+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.816272+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.69:0/1518543021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.817286+0000 mon.a (mon.0) 1183 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.818796+0000 mon.a (mon.0) 1184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.820214+0000 mon.a (mon.0) 1185 : audit 1 from='client.? 172.21.15.69:0/1813457633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.826019+0000 mon.a (mon.0) 1186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:52.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.826880+0000 mon.a (mon.0) 1187 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:51.827370+0000 mon.a (mon.0) 1188 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:51.881309+0000 mgr.y (mgr.14538) 234 : cluster 0 pgmap v209: 760 pgs: 1 active+clean+snaptrim, 149 active+undersized, 327 active+clean, 3 creating+peering, 241 unknown, 39 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 900 B/s wr, 0 op/s; 111/864 objects degraded (12.847%) 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.631352+0000 mon.c (mon.2) 135 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:52.794772+0000 mon.a (mon.0) 1189 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797003+0000 mon.a (mon.0) 1190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797170+0000 mon.a (mon.0) 1191 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797360+0000 mon.a (mon.0) 1192 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797559+0000 mon.a (mon.0) 1193 : audit 1 from='client.? 172.21.15.69:0/1813457633' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797646+0000 mon.a (mon.0) 1194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797719+0000 mon.a (mon.0) 1195 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.797805+0000 mon.a (mon.0) 1196 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: cluster 2024-04-02T04:49:52.804119+0000 mon.a (mon.0) 1197 : cluster 0 osdmap e157: 8 total, 7 up, 8 in 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.807694+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:52.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.808060+0000 mon.c (mon.2) 137 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.808593+0000 mon.b (mon.1) 134 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.812472+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:52.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.812889+0000 mon.a (mon.0) 1199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:52.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:52 smithi069 bash[28904]: audit 2024-04-02T04:49:52.813581+0000 mon.a (mon.0) 1200 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:52.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:52.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:52.989 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:53.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:53.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:53.142 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:49:53.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:49.880327+0000 mgr.y (mgr.14538) 233 : cluster 0 pgmap v206: 704 pgs: 1 active+clean+snaptrim, 168 active+undersized, 380 active+clean, 31 creating+peering, 77 unknown, 47 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s wr, 2 op/s; 144/1449 objects degraded (9.938%) 2024-04-02T04:49:53.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:50.072508+0000 mon.a (mon.0) 1174 : audit 1 from='client.? 172.21.15.69:0/2085595483' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi069-72924-2"}]': finished 2024-04-02T04:49:53.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:50.072576+0000 mon.a (mon.0) 1175 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "overlaypool": "test-rados-api-smithi069-72439-6"}]': finished 2024-04-02T04:49:53.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:50.081599+0000 mon.b (mon.1) 133 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:53.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:50.085067+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]: dispatch 2024-04-02T04:49:53.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:50.085473+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.69:0/86834543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:50.628693+0000 mon.c (mon.2) 130 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.630095+0000 mon.c (mon.2) 131 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:51.778859+0000 mon.a (mon.0) 1176 : cluster 0 osdmap e155: 8 total, 7 up, 8 in 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:51.779582+0000 mon.a (mon.0) 1177 : cluster 3 Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:51.781173+0000 mon.a (mon.0) 1178 : cluster 3 pool 'PoolQuotaPP_smithi069-72153-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:51.787195+0000 mon.a (mon.0) 1179 : cluster 3 Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.792468+0000 mon.a (mon.0) 1180 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.793073+0000 mon.a (mon.0) 1181 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.800638+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.69:0/2940842263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:51.806428+0000 mon.a (mon.0) 1182 : cluster 0 osdmap e156: 8 total, 7 up, 8 in 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.812986+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.816272+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.69:0/1518543021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.817286+0000 mon.a (mon.0) 1183 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.818796+0000 mon.a (mon.0) 1184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.820214+0000 mon.a (mon.0) 1185 : audit 1 from='client.? 172.21.15.69:0/1813457633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.826019+0000 mon.a (mon.0) 1186 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]: dispatch 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.826880+0000 mon.a (mon.0) 1187 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:51.827370+0000 mon.a (mon.0) 1188 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:51.881309+0000 mgr.y (mgr.14538) 234 : cluster 0 pgmap v209: 760 pgs: 1 active+clean+snaptrim, 149 active+undersized, 327 active+clean, 3 creating+peering, 241 unknown, 39 active+undersized+degraded; 145 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 900 B/s wr, 0 op/s; 111/864 objects degraded (12.847%) 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.631352+0000 mon.c (mon.2) 135 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:52.794772+0000 mon.a (mon.0) 1189 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797003+0000 mon.a (mon.0) 1190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797170+0000 mon.a (mon.0) 1191 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72439-6", "mode": "writeback"}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797360+0000 mon.a (mon.0) 1192 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797559+0000 mon.a (mon.0) 1193 : audit 1 from='client.? 172.21.15.69:0/1813457633' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797646+0000 mon.a (mon.0) 1194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-72410-7"}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797719+0000 mon.a (mon.0) 1195 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.797805+0000 mon.a (mon.0) 1196 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-72145-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: cluster 2024-04-02T04:49:52.804119+0000 mon.a (mon.0) 1197 : cluster 0 osdmap e157: 8 total, 7 up, 8 in 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.807694+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:53.145 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.808060+0000 mon.c (mon.2) 137 : audit 1 from='client.? 172.21.15.69:0/3761943508' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:53.146 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.808593+0000 mon.b (mon.1) 134 : audit 1 from='client.? 172.21.15.69:0/1655066126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:53.146 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.812472+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:53.146 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.812889+0000 mon.a (mon.0) 1199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]: dispatch 2024-04-02T04:49:53.146 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:52 smithi098 bash[23790]: audit 2024-04-02T04:49:52.813581+0000 mon.a (mon.0) 1200 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]: dispatch 2024-04-02T04:49:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:53.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:53.280 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:53.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:53.409 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:53.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:53.534 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:53.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:53.664 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:53.767 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:53.785 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:53.828 INFO:tasks.workunit.client.0.smithi069.stdout: pool: Running main() from gmock_main.cc 2024-04-02T04:49:53.828 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-02T04:49:53.828 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] Global test environment set-up. 2024-04-02T04:49:53.828 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-02T04:49:53.828 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolList (16781 ms) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (5261 ms) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (5471 ms) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (9999 ms) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (3952 ms) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2014 ms) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] 6 tests from NeoRadosPools (43478 ms total) 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] Global test environment tear-down 2024-04-02T04:49:53.829 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [==========] 6 tests from 1 test suite ran. (43478 ms total) 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ PASSED ] 6 tests. 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: seed 72210 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (410 ms) 2024-04-02T04:49:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (12 ms) 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1009 ms) 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1007 ms) 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (6 ms) 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (6 ms) 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (9 ms) 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-02T04:49:53.831 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4 ms) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2463 ms total) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (4184 ms) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (7 ms) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1103 ms) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1006 ms) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (9 ms) 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-02T04:49:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (9 ms) 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (10 ms) 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (7 ms) 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (6335 ms total) 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (43828 ms total) 2024-04-02T04:49:53.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: Running main() from gmock_main.cc 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment set-up. 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (81 ms) 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3 ms) 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-02T04:49:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1006 ms) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1018 ms) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (21 ms) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (14 ms) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (6 ms) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock (2153 ms total) 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-02T04:49:53.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (4193 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (1027 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1017 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1009 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (8 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (5 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (6 ms) 2024-04-02T04:49:53.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-02T04:49:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-04-02T04:49:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (7269 ms total) 2024-04-02T04:49:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2024-04-02T04:49:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment tear-down 2024-04-02T04:49:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (43857 ms total) 2024-04-02T04:49:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-02T04:49:53.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.008330+0000 mgr.y (mgr.14538) 235 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.632652+0000 mon.c (mon.2) 138 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.800457+0000 mon.a (mon.0) 1201 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.800532+0000 mon.a (mon.0) 1202 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.800610+0000 mon.a (mon.0) 1203 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: cluster 2024-04-02T04:49:53.810882+0000 mon.a (mon.0) 1204 : cluster 0 osdmap e158: 8 total, 7 up, 8 in 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.820381+0000 mon.c (mon.2) 139 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.821160+0000 mon.c (mon.2) 140 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.835488+0000 mon.a (mon.0) 1205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.836854+0000 mon.a (mon.0) 1206 : audit 1 from='client.? 172.21.15.69:0/1011361584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-72378-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.837198+0000 mon.a (mon.0) 1207 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:53.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:53 smithi098 bash[23790]: audit 2024-04-02T04:49:53.837769+0000 mon.a (mon.0) 1208 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:53.916 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:53.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:54.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.008330+0000 mgr.y (mgr.14538) 235 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.632652+0000 mon.c (mon.2) 138 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:54.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.800457+0000 mon.a (mon.0) 1201 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.800532+0000 mon.a (mon.0) 1202 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.800610+0000 mon.a (mon.0) 1203 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: cluster 2024-04-02T04:49:53.810882+0000 mon.a (mon.0) 1204 : cluster 0 osdmap e158: 8 total, 7 up, 8 in 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.820381+0000 mon.c (mon.2) 139 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.821160+0000 mon.c (mon.2) 140 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.835488+0000 mon.a (mon.0) 1205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.836854+0000 mon.a (mon.0) 1206 : audit 1 from='client.? 172.21.15.69:0/1011361584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-72378-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.837198+0000 mon.a (mon.0) 1207 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[28904]: audit 2024-04-02T04:49:53.837769+0000 mon.a (mon.0) 1208 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:54.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.008330+0000 mgr.y (mgr.14538) 235 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.632652+0000 mon.c (mon.2) 138 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.800457+0000 mon.a (mon.0) 1201 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.800532+0000 mon.a (mon.0) 1202 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-72190-10"}]': finished 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.800610+0000 mon.a (mon.0) 1203 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-72210-10"}]': finished 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: cluster 2024-04-02T04:49:53.810882+0000 mon.a (mon.0) 1204 : cluster 0 osdmap e158: 8 total, 7 up, 8 in 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.820381+0000 mon.c (mon.2) 139 : audit 1 from='client.? 172.21.15.69:0/665354487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.821160+0000 mon.c (mon.2) 140 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.835488+0000 mon.a (mon.0) 1205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.836854+0000 mon.a (mon.0) 1206 : audit 1 from='client.? 172.21.15.69:0/1011361584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-72378-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.837198+0000 mon.a (mon.0) 1207 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:54.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:53 smithi069 bash[21528]: audit 2024-04-02T04:49:53.837769+0000 mon.a (mon.0) 1208 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-02T04:49:54.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:54.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:54.320 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:54.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:54.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:54.574 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:54.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:54.700 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:54.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:54.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:54.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:55.028 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:55.029 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:49:55.056 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:55.073 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:55.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:55.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (436 ms) 2024-04-02T04:49:55.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94769173471696 err -107 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (36 ms) 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94769173485984 err -107 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (15 ms) 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 592705486848 cookie 94769173483280 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (16 ms) 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 592705486849 cookie 94769173483280 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (10 ms) 2024-04-02T04:49:55.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 592705486848 cookie 94769174901232 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (22 ms) 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 592705486850 cookie 94769174880912 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 592705486850 cookie 94769174901232 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (30 ms) 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 592705486848 cookie 94769174901232 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 597000454145 cookie 94769174901232 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3061 ms) 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94769174901232 err -107 2024-04-02T04:49:55.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 24637 notify_id 618475290627 cookie 94769174901232 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5051 ms) 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94769174882112 err -107 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1009 ms) 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9686 ms total) 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1430 ms) 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1430 ms total) 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-02T04:49:55.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (45112 ms total) 2024-04-02T04:49:55.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-02T04:49:55.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:55.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:55.454 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:55.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:55.575 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:55.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:84.1a 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:83.1d 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:83.11 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:82.c 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:81.d 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:81.b 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:81.3 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:81.1f 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:80.14 2024-04-02T04:49:55.698 INFO:teuthology.orchestra.run.smithi098.stdout:80.12 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:79.7 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:79.1e 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:79.19 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:79.18 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:78.4 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:78.1b 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:77.11 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:74.5s0 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:74.1s0 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:84.5 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:72.a 2024-04-02T04:49:55.699 INFO:teuthology.orchestra.run.smithi098.stdout:72.1c 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:72.1b 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:72.1a 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:72.19 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:72.0 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.7 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:84.f 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.1b 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.1 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.18 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.14 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.13 2024-04-02T04:49:55.700 INFO:teuthology.orchestra.run.smithi098.stdout:70.e 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:70.0 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:68.6s2 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:68.5s0 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:68.4s1 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:67.7s0 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:67.4s2 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:67.0s1 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:66.b 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:66.9 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:66.7 2024-04-02T04:49:55.701 INFO:teuthology.orchestra.run.smithi098.stdout:64.2 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:72.6 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:60.8 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:60.5 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:72.2 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:60.17 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:60.16 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:72.f 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:60.10 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:6.b 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:6.9 2024-04-02T04:49:55.702 INFO:teuthology.orchestra.run.smithi098.stdout:66.1b 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.5 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.3 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.1b 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.18 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:60.b 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.16 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:72.9 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.15 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.14 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.12 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:3.12 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:6.1c 2024-04-02T04:49:55.703 INFO:teuthology.orchestra.run.smithi098.stdout:2.f 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.b 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.8 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.5 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:17.a 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:84.12 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.4 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:4.5 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.1f 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.1d 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:66.15 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:13.f 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:60.19 2024-04-02T04:49:55.704 INFO:teuthology.orchestra.run.smithi098.stdout:2.1b 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:2.1a 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:36.6 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:19.2 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:70.6 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:19.1c 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:19.15 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:66.c 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:19.1 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:4.17 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:17.c 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:60.2 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:2.18 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:19.c 2024-04-02T04:49:55.705 INFO:teuthology.orchestra.run.smithi098.stdout:17.b 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:70.15 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:40.19 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:74.2s0 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:19.6 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:13.19 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:43.5 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:15.f 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:79.5 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:36.8 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:17.1d 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:55.4 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:15.1 2024-04-02T04:49:55.706 INFO:teuthology.orchestra.run.smithi098.stdout:19.12 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:10.1b 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:80.4 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:17.13 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:15.9 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:2.1c 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:4.0 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:68.2s0 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:55.1f 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:77.9 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:13.1d 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:2.a 2024-04-02T04:49:55.707 INFO:teuthology.orchestra.run.smithi098.stdout:3.7 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:17.7 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:41.2 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:15.18 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:4.8 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:40.10 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:10.9 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:46.1f 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:67.2s1 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:3.13 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:40.1a 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:1.0 2024-04-02T04:49:55.708 INFO:teuthology.orchestra.run.smithi098.stdout:19.1d 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:13.0 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:15.1e 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:70.5 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:4.a 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:19.3 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:55.14 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:19.14 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:66.0 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:60.4 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:10.13 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:70.9 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:17.0 2024-04-02T04:49:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:2.e 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:43.13 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:13.13 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:10.1f 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:66.4 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:15.d 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:2.12 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:80.15 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:41.c 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:19.b 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:15.15 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:3.9 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:10.11 2024-04-02T04:49:55.710 INFO:teuthology.orchestra.run.smithi098.stdout:81.a 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:75.1s0 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:10.10 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:40.1c 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:60.12 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:10.8 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:13.7 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:17.8 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:13.e 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:6.8 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:55.7 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:2.7 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:15.12 2024-04-02T04:49:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:43.17 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:3.8 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:2.3 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:80.d 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:17.15 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:15.1b 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:36.19 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:17.d 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:69.3s2 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:3.14 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:17.10 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:19.4 2024-04-02T04:49:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:19.9 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:55.f 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:36.1b 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:17.1e 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:55.18 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:17.18 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:79.0 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:55.a 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:41.1c 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:3.15 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:3.16 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:36.5 2024-04-02T04:49:55.713 INFO:teuthology.orchestra.run.smithi098.stdout:3.1b 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:46.19 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:3.d 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:2.0 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:41.1d 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:3.e 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:41.16 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:3.f 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:3.11 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:36.13 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:70.10 2024-04-02T04:49:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:40.0 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:36.3 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:36.4 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:40.b 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:13.d 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:4.11 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:41.6 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:72.4 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:10.c 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:4.15 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:46.1c 2024-04-02T04:49:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:4.1e 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:4.3 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:79.1b 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:40.15 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:40.18 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:46.4 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:43.8 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:55.10 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:41.0 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:41.11 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:66.1 2024-04-02T04:49:55.716 INFO:teuthology.orchestra.run.smithi098.stdout:41.15 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:41.18 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:41.b 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:13.14 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:41.d 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:43.16 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:43.1d 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:43.2 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:43.c 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:66.2 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:43.d 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:43.f 2024-04-02T04:49:55.717 INFO:teuthology.orchestra.run.smithi098.stdout:46.17 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:43.a 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:46.12 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:66.d 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:60.9 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:17.1f 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:4.16 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:46.d 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:6.6 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:10.1 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:46.1e 2024-04-02T04:49:55.718 INFO:teuthology.orchestra.run.smithi098.stdout:46.a 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:46.c 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:46.e 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:5.1 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:55.0 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:40.1f 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:55.16 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:55.1 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:4.14 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:55.3 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:55.6 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:55.b 2024-04-02T04:49:55.719 INFO:teuthology.orchestra.run.smithi098.stdout:80.13 2024-04-02T04:49:55.720 INFO:teuthology.orchestra.run.smithi098.stdout:13.18 2024-04-02T04:49:55.720 INFO:teuthology.orchestra.run.smithi098.stdout:55.c 2024-04-02T04:49:55.720 INFO:teuthology.orchestra.run.smithi098.stdout:72.d 2024-04-02T04:49:55.720 INFO:teuthology.orchestra.run.smithi098.stdout:43.1f 2024-04-02T04:49:55.720 INFO:teuthology.orchestra.run.smithi098.stdout:55.d 2024-04-02T04:49:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:55.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:55.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:55.844 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:55.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:55.973 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:55.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:56.050 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2777 ms) 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2782 ms total) 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-02T04:49:56.051 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (185 ms) 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (6440 ms) 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (17 ms) 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-02T04:49:56.052 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865477 notifier_gid 25337 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (12 ms) 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865474 notifier_gid 25337 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (18 ms) 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865478 notifier_gid 25337 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (13 ms) 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865479 notifier_gid 25337 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865475 notifier_gid 25337 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-04-02T04:49:56.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865480 notifier_gid 25337 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (6 ms) 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646879120 notify_id 674309865476 notifier_gid 25337 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2024-04-02T04:49:56.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-04-02T04:49:56.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:56.102 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:56.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:56.227 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:56.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:56.353 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:56.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:56.478 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:56.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:56.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:56.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:56 smithi098 bash[23790]: audit 2024-04-02T04:49:54.633939+0000 mon.c (mon.2) 141 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:56.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:56.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:56.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:56 smithi069 bash[28904]: audit 2024-04-02T04:49:54.633939+0000 mon.c (mon.2) 141 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:56.860 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:56 smithi069 bash[21528]: audit 2024-04-02T04:49:54.633939+0000 mon.c (mon.2) 141 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:56.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:56.976 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:56.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:57.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:57.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:57.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -n osd.7 --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 66.c --file /var/log/ceph/exp.66.c.7 2024-04-02T04:49:57.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:57.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:57.396 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/osd.7/config 2024-04-02T04:49:57.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:57.489 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:57.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:57.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:57.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:57.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:57.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:57.998 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:58.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:58.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: cluster 2024-04-02T04:49:53.882286+0000 mgr.y (mgr.14538) 236 : cluster 0 pgmap v212: 552 pgs: 126 active+undersized, 289 active+clean, 96 unknown, 41 active+undersized+degraded; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 114/867 objects degraded (13.149%) 2024-04-02T04:49:58.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: cluster 2024-04-02T04:49:55.192365+0000 mon.a (mon.0) 1209 : cluster 3 Health check update: Degraded data redundancy: 114/867 objects degraded (13.149%), 41 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:58.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:55.196049+0000 mon.a (mon.0) 1210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:58.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:55.196183+0000 mon.a (mon.0) 1211 : audit 1 from='client.? 172.21.15.69:0/1011361584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-72378-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:55.196299+0000 mon.a (mon.0) 1212 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:55.196376+0000 mon.a (mon.0) 1213 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:55.212929+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.69:0/3834080330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:55.635069+0000 mon.c (mon.2) 143 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: cluster 2024-04-02T04:49:56.496602+0000 mon.a (mon.0) 1214 : cluster 0 osdmap e159: 8 total, 7 up, 8 in 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:56.506684+0000 mon.a (mon.0) 1215 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:56.508213+0000 mon.a (mon.0) 1216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:56.520959+0000 mon.c (mon.2) 144 : audit 1 from='client.? 172.21.15.69:0/4223955538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:56.636287+0000 mon.c (mon.2) 145 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: cluster 2024-04-02T04:49:57.373611+0000 mon.a (mon.0) 1217 : cluster 0 osdmap e160: 8 total, 7 up, 8 in 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: cluster 2024-04-02T04:49:57.373948+0000 mon.a (mon.0) 1218 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:57.379880+0000 mon.a (mon.0) 1219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:57.380097+0000 mon.a (mon.0) 1220 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:58.132 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:57.380173+0000 mon.a (mon.0) 1221 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.132 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: cluster 2024-04-02T04:49:57.385913+0000 mon.a (mon.0) 1222 : cluster 0 osdmap e161: 8 total, 7 up, 8 in 2024-04-02T04:49:58.132 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:57.391563+0000 mon.c (mon.2) 146 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.132 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:57 smithi098 bash[23790]: audit 2024-04-02T04:49:57.394459+0000 mon.c (mon.2) 147 : audit 1 from='client.? 172.21.15.69:0/2377368795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:58.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:58.259 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: cluster 2024-04-02T04:49:53.882286+0000 mgr.y (mgr.14538) 236 : cluster 0 pgmap v212: 552 pgs: 126 active+undersized, 289 active+clean, 96 unknown, 41 active+undersized+degraded; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 114/867 objects degraded (13.149%) 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: cluster 2024-04-02T04:49:55.192365+0000 mon.a (mon.0) 1209 : cluster 3 Health check update: Degraded data redundancy: 114/867 objects degraded (13.149%), 41 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:55.196049+0000 mon.a (mon.0) 1210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:55.196183+0000 mon.a (mon.0) 1211 : audit 1 from='client.? 172.21.15.69:0/1011361584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-72378-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:55.196299+0000 mon.a (mon.0) 1212 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:55.196376+0000 mon.a (mon.0) 1213 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:55.212929+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.69:0/3834080330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:55.635069+0000 mon.c (mon.2) 143 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: cluster 2024-04-02T04:49:56.496602+0000 mon.a (mon.0) 1214 : cluster 0 osdmap e159: 8 total, 7 up, 8 in 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:56.506684+0000 mon.a (mon.0) 1215 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:58.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:56.508213+0000 mon.a (mon.0) 1216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:56.520959+0000 mon.c (mon.2) 144 : audit 1 from='client.? 172.21.15.69:0/4223955538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:56.636287+0000 mon.c (mon.2) 145 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: cluster 2024-04-02T04:49:57.373611+0000 mon.a (mon.0) 1217 : cluster 0 osdmap e160: 8 total, 7 up, 8 in 2024-04-02T04:49:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: cluster 2024-04-02T04:49:57.373948+0000 mon.a (mon.0) 1218 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:57.379880+0000 mon.a (mon.0) 1219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: cluster 2024-04-02T04:49:53.882286+0000 mgr.y (mgr.14538) 236 : cluster 0 pgmap v212: 552 pgs: 126 active+undersized, 289 active+clean, 96 unknown, 41 active+undersized+degraded; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 114/867 objects degraded (13.149%) 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: cluster 2024-04-02T04:49:55.192365+0000 mon.a (mon.0) 1209 : cluster 3 Health check update: Degraded data redundancy: 114/867 objects degraded (13.149%), 41 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:55.196049+0000 mon.a (mon.0) 1210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-72465-12"}]': finished 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:55.196183+0000 mon.a (mon.0) 1211 : audit 1 from='client.? 172.21.15.69:0/1011361584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-72378-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:55.196299+0000 mon.a (mon.0) 1212 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:55.196376+0000 mon.a (mon.0) 1213 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:55.212929+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.69:0/3834080330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:55.635069+0000 mon.c (mon.2) 143 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: cluster 2024-04-02T04:49:56.496602+0000 mon.a (mon.0) 1214 : cluster 0 osdmap e159: 8 total, 7 up, 8 in 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:56.506684+0000 mon.a (mon.0) 1215 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi069-72950-2"}]: dispatch 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:56.508213+0000 mon.a (mon.0) 1216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:56.520959+0000 mon.c (mon.2) 144 : audit 1 from='client.? 172.21.15.69:0/4223955538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:56.636287+0000 mon.c (mon.2) 145 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: cluster 2024-04-02T04:49:57.373611+0000 mon.a (mon.0) 1217 : cluster 0 osdmap e160: 8 total, 7 up, 8 in 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: cluster 2024-04-02T04:49:57.373948+0000 mon.a (mon.0) 1218 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:57.379880+0000 mon.a (mon.0) 1219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:57.380097+0000 mon.a (mon.0) 1220 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:57.380173+0000 mon.a (mon.0) 1221 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: cluster 2024-04-02T04:49:57.385913+0000 mon.a (mon.0) 1222 : cluster 0 osdmap e161: 8 total, 7 up, 8 in 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:57.391563+0000 mon.c (mon.2) 146 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[21528]: audit 2024-04-02T04:49:57.394459+0000 mon.c (mon.2) 147 : audit 1 from='client.? 172.21.15.69:0/2377368795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:57.380097+0000 mon.a (mon.0) 1220 : audit 1 from='client.? 172.21.15.69:0/625148147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi069-72950-2"}]': finished 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:57.380173+0000 mon.a (mon.0) 1221 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: cluster 2024-04-02T04:49:57.385913+0000 mon.a (mon.0) 1222 : cluster 0 osdmap e161: 8 total, 7 up, 8 in 2024-04-02T04:49:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:57.391563+0000 mon.c (mon.2) 146 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:57 smithi069 bash[28904]: audit 2024-04-02T04:49:57.394459+0000 mon.c (mon.2) 147 : audit 1 from='client.? 172.21.15.69:0/2377368795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:58.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:58.380 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:58.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:58.505 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:58.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:49:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:49:58.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:58.759 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:49:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:49:58.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:49:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:58.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:49:58.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:59.005 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:49:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:49:59.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.112 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:49:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:49:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:49:59.128 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:59.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:59.254 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:59.357 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.377 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:59.480 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.502 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:59.582 INFO:teuthology.orchestra.run.smithi098.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-02T04:49:59.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:59.625 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:49:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:49:59.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: cluster 2024-04-02T04:49:55.883036+0000 mgr.y (mgr.14538) 237 : cluster 0 pgmap v214: 648 pgs: 126 active+undersized, 289 active+clean, 192 unknown, 41 active+undersized+degraded; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s; 114/867 objects degraded (13.149%) 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:57.637402+0000 mon.c (mon.2) 148 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:57.878002+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi069-72950-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:57.879138+0000 mon.a (mon.0) 1224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:57.879488+0000 mon.a (mon.0) 1225 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: cluster 2024-04-02T04:49:58.381344+0000 mon.a (mon.0) 1226 : cluster 1 Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:58.387462+0000 mon.a (mon.0) 1227 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:58.387563+0000 mon.a (mon.0) 1228 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi069-72950-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:58.387625+0000 mon.a (mon.0) 1229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:58.387682+0000 mon.a (mon.0) 1230 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: cluster 2024-04-02T04:49:58.395395+0000 mon.a (mon.0) 1231 : cluster 0 osdmap e162: 8 total, 7 up, 8 in 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:58.396258+0000 mon.a (mon.0) 1232 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi069-72950-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[21528]: audit 2024-04-02T04:49:58.400692+0000 mon.b (mon.1) 135 : audit 1 from='client.? 172.21.15.69:0/2140839855' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: cluster 2024-04-02T04:49:55.883036+0000 mgr.y (mgr.14538) 237 : cluster 0 pgmap v214: 648 pgs: 126 active+undersized, 289 active+clean, 192 unknown, 41 active+undersized+degraded; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s; 114/867 objects degraded (13.149%) 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:57.637402+0000 mon.c (mon.2) 148 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:57.878002+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi069-72950-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:57.879138+0000 mon.a (mon.0) 1224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:57.879488+0000 mon.a (mon.0) 1225 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: cluster 2024-04-02T04:49:58.381344+0000 mon.a (mon.0) 1226 : cluster 1 Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:58.387462+0000 mon.a (mon.0) 1227 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:58.387563+0000 mon.a (mon.0) 1228 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi069-72950-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:58.387625+0000 mon.a (mon.0) 1229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:58.387682+0000 mon.a (mon.0) 1230 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: cluster 2024-04-02T04:49:58.395395+0000 mon.a (mon.0) 1231 : cluster 0 osdmap e162: 8 total, 7 up, 8 in 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:58.396258+0000 mon.a (mon.0) 1232 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi069-72950-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:49:59.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:49:59 smithi069 bash[28904]: audit 2024-04-02T04:49:58.400692+0000 mon.b (mon.1) 135 : audit 1 from='client.? 172.21.15.69:0/2140839855' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.750 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:49:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:49:59.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:49:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:49:59.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:49:59.941 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: cluster 2024-04-02T04:49:55.883036+0000 mgr.y (mgr.14538) 237 : cluster 0 pgmap v214: 648 pgs: 126 active+undersized, 289 active+clean, 192 unknown, 41 active+undersized+degraded; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s; 114/867 objects degraded (13.149%) 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:57.637402+0000 mon.c (mon.2) 148 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:57.878002+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi069-72950-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:57.879138+0000 mon.a (mon.0) 1224 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:57.879488+0000 mon.a (mon.0) 1225 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: cluster 2024-04-02T04:49:58.381344+0000 mon.a (mon.0) 1226 : cluster 1 Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:58.387462+0000 mon.a (mon.0) 1227 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-72145-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:58.387563+0000 mon.a (mon.0) 1228 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi069-72950-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:58.387625+0000 mon.a (mon.0) 1229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:58.387682+0000 mon.a (mon.0) 1230 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:49:59.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: cluster 2024-04-02T04:49:58.395395+0000 mon.a (mon.0) 1231 : cluster 0 osdmap e162: 8 total, 7 up, 8 in 2024-04-02T04:49:59.943 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:58.396258+0000 mon.a (mon.0) 1232 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi069-72950-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:49:59.943 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:49:59 smithi098 bash[23790]: audit 2024-04-02T04:49:58.400692+0000 mon.b (mon.1) 135 : audit 1 from='client.? 172.21.15.69:0/2140839855' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:49:59.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:00.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:00.129 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:00.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:00.257 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:00.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:00.387 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:00.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:00.515 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:00.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:00.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:00.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:00.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:00.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:49:57.885314+0000 mgr.y (mgr.14538) 238 : cluster 0 pgmap v217: 744 pgs: 165 active+undersized, 378 active+clean, 160 creating+peering, 41 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 111/891 objects degraded (12.458%) 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:58.454159+0000 mon.a (mon.0) 1233 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:58.638917+0000 mon.c (mon.2) 149 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:58.689656+0000 mon.c (mon.2) 150 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:59.408768+0000 mon.a (mon.0) 1234 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:59.639944+0000 mon.c (mon.2) 151 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:59.868391+0000 mon.c (mon.2) 152 : audit 1 from='client.? 172.21.15.69:0/3749132568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:49:59.869537+0000 mon.a (mon.0) 1235 : cluster 0 osdmap e163: 8 total, 7 up, 8 in 2024-04-02T04:50:00.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:59.880094+0000 mon.a (mon.0) 1236 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:00.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:49:59.881692+0000 mon.a (mon.0) 1237 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:00.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.112621+0000 mon.c (mon.2) 153 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:50:00.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.113636+0000 mon.c (mon.2) 154 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:50:00.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.114450+0000 mon.c (mon.2) 155 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:50:00.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.115267+0000 mon.c (mon.2) 156 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:50:00.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.116000+0000 mon.c (mon.2) 157 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.116872+0000 mon.c (mon.2) 158 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.117684+0000 mon.c (mon.2) 159 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.118606+0000 mon.c (mon.2) 160 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.119610+0000 mon.c (mon.2) 161 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: audit 2024-04-02T04:50:00.120609+0000 mon.c (mon.2) 162 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.337889+0000 mon.a (mon.0) 1238 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 111/891 objects degraded (12.458%), 41 pgs degraded; 11 pool(s) do not have an application enabled 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.337969+0000 mon.a (mon.0) 1239 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338022+0000 mon.a (mon.0) 1240 : cluster 3 pool 'test-rados-api-smithi069-72439-6' with cache_mode writeback needs hit_set_type to be set but it is not 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338065+0000 mon.a (mon.0) 1241 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338094+0000 mon.a (mon.0) 1242 : cluster 3 [WRN] OSD_DOWN: 1 osds down 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338147+0000 mon.a (mon.0) 1243 : cluster 3 osd.7 (root=default,host=smithi098) is down 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338170+0000 mon.a (mon.0) 1244 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 111/891 objects degraded (12.458%), 41 pgs degraded 2024-04-02T04:50:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338193+0000 mon.a (mon.0) 1245 : cluster 3 pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338216+0000 mon.a (mon.0) 1246 : cluster 3 pg 2.4 is active+undersized+degraded, acting [1,0] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338234+0000 mon.a (mon.0) 1247 : cluster 3 pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338259+0000 mon.a (mon.0) 1248 : cluster 3 pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338279+0000 mon.a (mon.0) 1249 : cluster 3 pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338301+0000 mon.a (mon.0) 1250 : cluster 3 pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338323+0000 mon.a (mon.0) 1251 : cluster 3 pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338347+0000 mon.a (mon.0) 1252 : cluster 3 pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338370+0000 mon.a (mon.0) 1253 : cluster 3 pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338392+0000 mon.a (mon.0) 1254 : cluster 3 pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338417+0000 mon.a (mon.0) 1255 : cluster 3 pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338451+0000 mon.a (mon.0) 1256 : cluster 3 pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338470+0000 mon.a (mon.0) 1257 : cluster 3 pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:00.977 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338492+0000 mon.a (mon.0) 1258 : cluster 3 pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338515+0000 mon.a (mon.0) 1259 : cluster 3 pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338533+0000 mon.a (mon.0) 1260 : cluster 3 pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338557+0000 mon.a (mon.0) 1261 : cluster 3 pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338582+0000 mon.a (mon.0) 1262 : cluster 3 pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338603+0000 mon.a (mon.0) 1263 : cluster 3 pg 4.3 is active+undersized+degraded, acting [0,5] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338624+0000 mon.a (mon.0) 1264 : cluster 3 pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338646+0000 mon.a (mon.0) 1265 : cluster 3 pg 4.15 is active+undersized+degraded, acting [5,3] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338668+0000 mon.a (mon.0) 1266 : cluster 3 pg 6.6 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338688+0000 mon.a (mon.0) 1267 : cluster 3 pg 6.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338708+0000 mon.a (mon.0) 1268 : cluster 3 pg 10.1 is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338743+0000 mon.a (mon.0) 1269 : cluster 3 pg 13.0 is active+undersized+degraded, acting [1,2] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338767+0000 mon.a (mon.0) 1270 : cluster 3 pg 13.d is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338792+0000 mon.a (mon.0) 1271 : cluster 3 pg 13.13 is active+undersized+degraded, acting [0,2] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338814+0000 mon.a (mon.0) 1272 : cluster 3 pg 13.1d is active+undersized+degraded, acting [2,1] 2024-04-02T04:50:00.978 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338836+0000 mon.a (mon.0) 1273 : cluster 3 pg 15.1 is active+undersized+degraded, acting [6,3] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338858+0000 mon.a (mon.0) 1274 : cluster 3 pg 15.9 is active+undersized+degraded, acting [6,1] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338879+0000 mon.a (mon.0) 1275 : cluster 3 pg 15.d is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338897+0000 mon.a (mon.0) 1276 : cluster 3 pg 15.12 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338915+0000 mon.a (mon.0) 1277 : cluster 3 pg 15.15 is active+undersized+degraded, acting [0,1] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338940+0000 mon.a (mon.0) 1278 : cluster 3 pg 15.1b is active+undersized+degraded, acting [6,4] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338960+0000 mon.a (mon.0) 1279 : cluster 3 pg 15.1e is active+undersized+degraded, acting [2,4] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.338984+0000 mon.a (mon.0) 1280 : cluster 3 pg 70.1 is active+undersized+degraded, acting [2,6] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339004+0000 mon.a (mon.0) 1281 : cluster 3 pg 70.5 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339024+0000 mon.a (mon.0) 1282 : cluster 3 pg 70.13 is active+undersized+degraded, acting [0,5] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339047+0000 mon.a (mon.0) 1283 : cluster 3 pg 101.8 is active+undersized+degraded, acting [6,2] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339075+0000 mon.a (mon.0) 1284 : cluster 3 pg 103.0 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339095+0000 mon.a (mon.0) 1285 : cluster 3 pg 104.15 is active+undersized+degraded, acting [3,5] 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339119+0000 mon.a (mon.0) 1286 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 11 pool(s) do not have an application enabled 2024-04-02T04:50:00.979 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339141+0000 mon.a (mon.0) 1287 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339161+0000 mon.a (mon.0) 1288 : cluster 3 application not enabled on pool 'poolrace.1804289383' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339182+0000 mon.a (mon.0) 1289 : cluster 3 application not enabled on pool 'test_handler_errorsmithi069-72850-1' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339202+0000 mon.a (mon.0) 1290 : cluster 3 application not enabled on pool 'poolrace2.846930886' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339222+0000 mon.a (mon.0) 1291 : cluster 3 application not enabled on pool 'SnapGetNamesmithi069-73023-4' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339242+0000 mon.a (mon.0) 1292 : cluster 3 application not enabled on pool 'LongAttrNamesmithi069-72966-6' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339260+0000 mon.a (mon.0) 1293 : cluster 3 application not enabled on pool 'Execsmithi069-73044-5' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339282+0000 mon.a (mon.0) 1294 : cluster 3 application not enabled on pool 'Readsmithi069-73015-5' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339303+0000 mon.a (mon.0) 1295 : cluster 3 application not enabled on pool 'RoundTrip2_smithi069-72145-5' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339323+0000 mon.a (mon.0) 1296 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72366-5' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339345+0000 mon.a (mon.0) 1297 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72349-7' 2024-04-02T04:50:00.980 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.339371+0000 mon.a (mon.0) 1298 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-02T04:50:01.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:49:57.885314+0000 mgr.y (mgr.14538) 238 : cluster 0 pgmap v217: 744 pgs: 165 active+undersized, 378 active+clean, 160 creating+peering, 41 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 111/891 objects degraded (12.458%) 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:58.454159+0000 mon.a (mon.0) 1233 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:58.638917+0000 mon.c (mon.2) 149 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:58.689656+0000 mon.c (mon.2) 150 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:59.408768+0000 mon.a (mon.0) 1234 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:59.639944+0000 mon.c (mon.2) 151 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:59.868391+0000 mon.c (mon.2) 152 : audit 1 from='client.? 172.21.15.69:0/3749132568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:49:59.869537+0000 mon.a (mon.0) 1235 : cluster 0 osdmap e163: 8 total, 7 up, 8 in 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:59.880094+0000 mon.a (mon.0) 1236 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:49:59.881692+0000 mon.a (mon.0) 1237 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.112621+0000 mon.c (mon.2) 153 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.113636+0000 mon.c (mon.2) 154 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:50:01.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.114450+0000 mon.c (mon.2) 155 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.115267+0000 mon.c (mon.2) 156 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.116000+0000 mon.c (mon.2) 157 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:49:57.885314+0000 mgr.y (mgr.14538) 238 : cluster 0 pgmap v217: 744 pgs: 165 active+undersized, 378 active+clean, 160 creating+peering, 41 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 111/891 objects degraded (12.458%) 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:58.454159+0000 mon.a (mon.0) 1233 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:58.638917+0000 mon.c (mon.2) 149 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:58.689656+0000 mon.c (mon.2) 150 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:59.408768+0000 mon.a (mon.0) 1234 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-72389-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:01.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:59.639944+0000 mon.c (mon.2) 151 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:59.868391+0000 mon.c (mon.2) 152 : audit 1 from='client.? 172.21.15.69:0/3749132568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:49:59.869537+0000 mon.a (mon.0) 1235 : cluster 0 osdmap e163: 8 total, 7 up, 8 in 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:59.880094+0000 mon.a (mon.0) 1236 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:49:59.881692+0000 mon.a (mon.0) 1237 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.112621+0000 mon.c (mon.2) 153 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.113636+0000 mon.c (mon.2) 154 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.114450+0000 mon.c (mon.2) 155 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.115267+0000 mon.c (mon.2) 156 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:50:01.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.116000+0000 mon.c (mon.2) 157 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.116872+0000 mon.c (mon.2) 158 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.117684+0000 mon.c (mon.2) 159 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.118606+0000 mon.c (mon.2) 160 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.119610+0000 mon.c (mon.2) 161 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: audit 2024-04-02T04:50:00.120609+0000 mon.c (mon.2) 162 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.337889+0000 mon.a (mon.0) 1238 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 111/891 objects degraded (12.458%), 41 pgs degraded; 11 pool(s) do not have an application enabled 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.337969+0000 mon.a (mon.0) 1239 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338022+0000 mon.a (mon.0) 1240 : cluster 3 pool 'test-rados-api-smithi069-72439-6' with cache_mode writeback needs hit_set_type to be set but it is not 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338065+0000 mon.a (mon.0) 1241 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338094+0000 mon.a (mon.0) 1242 : cluster 3 [WRN] OSD_DOWN: 1 osds down 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338147+0000 mon.a (mon.0) 1243 : cluster 3 osd.7 (root=default,host=smithi098) is down 2024-04-02T04:50:01.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338170+0000 mon.a (mon.0) 1244 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 111/891 objects degraded (12.458%), 41 pgs degraded 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338193+0000 mon.a (mon.0) 1245 : cluster 3 pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338216+0000 mon.a (mon.0) 1246 : cluster 3 pg 2.4 is active+undersized+degraded, acting [1,0] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338234+0000 mon.a (mon.0) 1247 : cluster 3 pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338259+0000 mon.a (mon.0) 1248 : cluster 3 pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338279+0000 mon.a (mon.0) 1249 : cluster 3 pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338301+0000 mon.a (mon.0) 1250 : cluster 3 pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338323+0000 mon.a (mon.0) 1251 : cluster 3 pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338347+0000 mon.a (mon.0) 1252 : cluster 3 pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338370+0000 mon.a (mon.0) 1253 : cluster 3 pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338392+0000 mon.a (mon.0) 1254 : cluster 3 pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338417+0000 mon.a (mon.0) 1255 : cluster 3 pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338451+0000 mon.a (mon.0) 1256 : cluster 3 pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338470+0000 mon.a (mon.0) 1257 : cluster 3 pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:01.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338492+0000 mon.a (mon.0) 1258 : cluster 3 pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338515+0000 mon.a (mon.0) 1259 : cluster 3 pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338533+0000 mon.a (mon.0) 1260 : cluster 3 pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338557+0000 mon.a (mon.0) 1261 : cluster 3 pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338582+0000 mon.a (mon.0) 1262 : cluster 3 pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338603+0000 mon.a (mon.0) 1263 : cluster 3 pg 4.3 is active+undersized+degraded, acting [0,5] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338624+0000 mon.a (mon.0) 1264 : cluster 3 pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338646+0000 mon.a (mon.0) 1265 : cluster 3 pg 4.15 is active+undersized+degraded, acting [5,3] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338668+0000 mon.a (mon.0) 1266 : cluster 3 pg 6.6 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338688+0000 mon.a (mon.0) 1267 : cluster 3 pg 6.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338708+0000 mon.a (mon.0) 1268 : cluster 3 pg 10.1 is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338743+0000 mon.a (mon.0) 1269 : cluster 3 pg 13.0 is active+undersized+degraded, acting [1,2] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338767+0000 mon.a (mon.0) 1270 : cluster 3 pg 13.d is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338792+0000 mon.a (mon.0) 1271 : cluster 3 pg 13.13 is active+undersized+degraded, acting [0,2] 2024-04-02T04:50:01.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338814+0000 mon.a (mon.0) 1272 : cluster 3 pg 13.1d is active+undersized+degraded, acting [2,1] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338836+0000 mon.a (mon.0) 1273 : cluster 3 pg 15.1 is active+undersized+degraded, acting [6,3] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338858+0000 mon.a (mon.0) 1274 : cluster 3 pg 15.9 is active+undersized+degraded, acting [6,1] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338879+0000 mon.a (mon.0) 1275 : cluster 3 pg 15.d is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338897+0000 mon.a (mon.0) 1276 : cluster 3 pg 15.12 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338915+0000 mon.a (mon.0) 1277 : cluster 3 pg 15.15 is active+undersized+degraded, acting [0,1] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338940+0000 mon.a (mon.0) 1278 : cluster 3 pg 15.1b is active+undersized+degraded, acting [6,4] 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.116872+0000 mon.c (mon.2) 158 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.117684+0000 mon.c (mon.2) 159 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.118606+0000 mon.c (mon.2) 160 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.119610+0000 mon.c (mon.2) 161 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: audit 2024-04-02T04:50:00.120609+0000 mon.c (mon.2) 162 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:50:01.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.337889+0000 mon.a (mon.0) 1238 : cluster 3 Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 111/891 objects degraded (12.458%), 41 pgs degraded; 11 pool(s) do not have an application enabled 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.337969+0000 mon.a (mon.0) 1239 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338022+0000 mon.a (mon.0) 1240 : cluster 3 pool 'test-rados-api-smithi069-72439-6' with cache_mode writeback needs hit_set_type to be set but it is not 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338065+0000 mon.a (mon.0) 1241 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338094+0000 mon.a (mon.0) 1242 : cluster 3 [WRN] OSD_DOWN: 1 osds down 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338147+0000 mon.a (mon.0) 1243 : cluster 3 osd.7 (root=default,host=smithi098) is down 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338170+0000 mon.a (mon.0) 1244 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 111/891 objects degraded (12.458%), 41 pgs degraded 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338193+0000 mon.a (mon.0) 1245 : cluster 3 pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338216+0000 mon.a (mon.0) 1246 : cluster 3 pg 2.4 is active+undersized+degraded, acting [1,0] 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338234+0000 mon.a (mon.0) 1247 : cluster 3 pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338259+0000 mon.a (mon.0) 1248 : cluster 3 pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338279+0000 mon.a (mon.0) 1249 : cluster 3 pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338301+0000 mon.a (mon.0) 1250 : cluster 3 pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:01.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338323+0000 mon.a (mon.0) 1251 : cluster 3 pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338347+0000 mon.a (mon.0) 1252 : cluster 3 pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338370+0000 mon.a (mon.0) 1253 : cluster 3 pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338392+0000 mon.a (mon.0) 1254 : cluster 3 pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338960+0000 mon.a (mon.0) 1279 : cluster 3 pg 15.1e is active+undersized+degraded, acting [2,4] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.338984+0000 mon.a (mon.0) 1280 : cluster 3 pg 70.1 is active+undersized+degraded, acting [2,6] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339004+0000 mon.a (mon.0) 1281 : cluster 3 pg 70.5 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339024+0000 mon.a (mon.0) 1282 : cluster 3 pg 70.13 is active+undersized+degraded, acting [0,5] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339047+0000 mon.a (mon.0) 1283 : cluster 3 pg 101.8 is active+undersized+degraded, acting [6,2] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339075+0000 mon.a (mon.0) 1284 : cluster 3 pg 103.0 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:01.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339095+0000 mon.a (mon.0) 1285 : cluster 3 pg 104.15 is active+undersized+degraded, acting [3,5] 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339119+0000 mon.a (mon.0) 1286 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 11 pool(s) do not have an application enabled 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339141+0000 mon.a (mon.0) 1287 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339161+0000 mon.a (mon.0) 1288 : cluster 3 application not enabled on pool 'poolrace.1804289383' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339182+0000 mon.a (mon.0) 1289 : cluster 3 application not enabled on pool 'test_handler_errorsmithi069-72850-1' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339202+0000 mon.a (mon.0) 1290 : cluster 3 application not enabled on pool 'poolrace2.846930886' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339222+0000 mon.a (mon.0) 1291 : cluster 3 application not enabled on pool 'SnapGetNamesmithi069-73023-4' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339242+0000 mon.a (mon.0) 1292 : cluster 3 application not enabled on pool 'LongAttrNamesmithi069-72966-6' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339260+0000 mon.a (mon.0) 1293 : cluster 3 application not enabled on pool 'Execsmithi069-73044-5' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339282+0000 mon.a (mon.0) 1294 : cluster 3 application not enabled on pool 'Readsmithi069-73015-5' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339303+0000 mon.a (mon.0) 1295 : cluster 3 application not enabled on pool 'RoundTrip2_smithi069-72145-5' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339323+0000 mon.a (mon.0) 1296 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72366-5' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339345+0000 mon.a (mon.0) 1297 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72349-7' 2024-04-02T04:50:01.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.339371+0000 mon.a (mon.0) 1298 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338417+0000 mon.a (mon.0) 1255 : cluster 3 pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338451+0000 mon.a (mon.0) 1256 : cluster 3 pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338470+0000 mon.a (mon.0) 1257 : cluster 3 pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338492+0000 mon.a (mon.0) 1258 : cluster 3 pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338515+0000 mon.a (mon.0) 1259 : cluster 3 pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338533+0000 mon.a (mon.0) 1260 : cluster 3 pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338557+0000 mon.a (mon.0) 1261 : cluster 3 pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338582+0000 mon.a (mon.0) 1262 : cluster 3 pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338603+0000 mon.a (mon.0) 1263 : cluster 3 pg 4.3 is active+undersized+degraded, acting [0,5] 2024-04-02T04:50:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338624+0000 mon.a (mon.0) 1264 : cluster 3 pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338646+0000 mon.a (mon.0) 1265 : cluster 3 pg 4.15 is active+undersized+degraded, acting [5,3] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338668+0000 mon.a (mon.0) 1266 : cluster 3 pg 6.6 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338688+0000 mon.a (mon.0) 1267 : cluster 3 pg 6.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338708+0000 mon.a (mon.0) 1268 : cluster 3 pg 10.1 is active+undersized+degraded, acting [4,0] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338743+0000 mon.a (mon.0) 1269 : cluster 3 pg 13.0 is active+undersized+degraded, acting [1,2] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338767+0000 mon.a (mon.0) 1270 : cluster 3 pg 13.d is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338792+0000 mon.a (mon.0) 1271 : cluster 3 pg 13.13 is active+undersized+degraded, acting [0,2] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338814+0000 mon.a (mon.0) 1272 : cluster 3 pg 13.1d is active+undersized+degraded, acting [2,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338836+0000 mon.a (mon.0) 1273 : cluster 3 pg 15.1 is active+undersized+degraded, acting [6,3] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338858+0000 mon.a (mon.0) 1274 : cluster 3 pg 15.9 is active+undersized+degraded, acting [6,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338879+0000 mon.a (mon.0) 1275 : cluster 3 pg 15.d is active+undersized+degraded, acting [4,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338897+0000 mon.a (mon.0) 1276 : cluster 3 pg 15.12 is active+undersized+degraded, acting [3,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338915+0000 mon.a (mon.0) 1277 : cluster 3 pg 15.15 is active+undersized+degraded, acting [0,1] 2024-04-02T04:50:01.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338940+0000 mon.a (mon.0) 1278 : cluster 3 pg 15.1b is active+undersized+degraded, acting [6,4] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338960+0000 mon.a (mon.0) 1279 : cluster 3 pg 15.1e is active+undersized+degraded, acting [2,4] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.338984+0000 mon.a (mon.0) 1280 : cluster 3 pg 70.1 is active+undersized+degraded, acting [2,6] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339004+0000 mon.a (mon.0) 1281 : cluster 3 pg 70.5 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339024+0000 mon.a (mon.0) 1282 : cluster 3 pg 70.13 is active+undersized+degraded, acting [0,5] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339047+0000 mon.a (mon.0) 1283 : cluster 3 pg 101.8 is active+undersized+degraded, acting [6,2] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339075+0000 mon.a (mon.0) 1284 : cluster 3 pg 103.0 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339095+0000 mon.a (mon.0) 1285 : cluster 3 pg 104.15 is active+undersized+degraded, acting [3,5] 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339119+0000 mon.a (mon.0) 1286 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 11 pool(s) do not have an application enabled 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339141+0000 mon.a (mon.0) 1287 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339161+0000 mon.a (mon.0) 1288 : cluster 3 application not enabled on pool 'poolrace.1804289383' 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339182+0000 mon.a (mon.0) 1289 : cluster 3 application not enabled on pool 'test_handler_errorsmithi069-72850-1' 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339202+0000 mon.a (mon.0) 1290 : cluster 3 application not enabled on pool 'poolrace2.846930886' 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339222+0000 mon.a (mon.0) 1291 : cluster 3 application not enabled on pool 'SnapGetNamesmithi069-73023-4' 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339242+0000 mon.a (mon.0) 1292 : cluster 3 application not enabled on pool 'LongAttrNamesmithi069-72966-6' 2024-04-02T04:50:01.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339260+0000 mon.a (mon.0) 1293 : cluster 3 application not enabled on pool 'Execsmithi069-73044-5' 2024-04-02T04:50:01.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339282+0000 mon.a (mon.0) 1294 : cluster 3 application not enabled on pool 'Readsmithi069-73015-5' 2024-04-02T04:50:01.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339303+0000 mon.a (mon.0) 1295 : cluster 3 application not enabled on pool 'RoundTrip2_smithi069-72145-5' 2024-04-02T04:50:01.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339323+0000 mon.a (mon.0) 1296 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72366-5' 2024-04-02T04:50:01.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339345+0000 mon.a (mon.0) 1297 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72349-7' 2024-04-02T04:50:01.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.339371+0000 mon.a (mon.0) 1298 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-02T04:50:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:01.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:Exporting 66.c info 66.c( empty local-lis/les=147/149 n=0 ec=147/147 lis/c=147/147 les/c/f=149/149/0 sis=147) 2024-04-02T04:50:01.078 INFO:teuthology.orchestra.run.smithi098.stderr:Export successful 2024-04-02T04:50:01.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:01.168 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:01.271 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:01.298 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:01.402 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:01.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:01.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:01.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:01.685 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:01.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:01.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:01.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:02.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:02.072 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:02.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:02.218 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:02.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:49:59.886283+0000 mgr.y (mgr.14538) 239 : cluster 0 pgmap v220: 712 pgs: 126 active+undersized, 293 active+clean, 128 unknown, 128 creating+peering, 37 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3 op/s; 106/870 objects degraded (12.184%) 2024-04-02T04:50:02.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.112892+0000 mgr.y (mgr.14538) 240 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.113857+0000 mgr.y (mgr.14538) 241 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.114660+0000 mgr.y (mgr.14538) 242 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.115430+0000 mgr.y (mgr.14538) 243 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.116262+0000 mgr.y (mgr.14538) 244 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.117096+0000 mgr.y (mgr.14538) 245 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.117928+0000 mgr.y (mgr.14538) 246 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.118888+0000 mgr.y (mgr.14538) 247 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.119874+0000 mgr.y (mgr.14538) 248 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.120862+0000 mgr.y (mgr.14538) 249 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.188941+0000 osd.1 (osd.1) 143 : cluster 0 15.7 deep-scrub starts 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.295552+0000 osd.1 (osd.1) 144 : cluster 0 15.7 deep-scrub ok 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.508613+0000 osd.4 (osd.4) 143 : cluster 0 15.3 deep-scrub starts 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.510231+0000 osd.4 (osd.4) 144 : cluster 0 15.3 deep-scrub ok 2024-04-02T04:50:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.641032+0000 mon.c (mon.2) 163 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.756037+0000 mon.a (mon.0) 1299 : cluster 3 Health check update: Degraded data redundancy: 106/870 objects degraded (12.184%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.778260+0000 mon.a (mon.0) 1300 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi069-72950-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.778450+0000 mon.a (mon.0) 1301 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.778676+0000 mon.a (mon.0) 1302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.789862+0000 mon.c (mon.2) 164 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.792116+0000 mon.a (mon.0) 1303 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: cluster 2024-04-02T04:50:00.796398+0000 mon.a (mon.0) 1304 : cluster 0 osdmap e164: 8 total, 7 up, 8 in 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.797011+0000 mon.b (mon.1) 136 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.798875+0000 mon.c (mon.2) 165 : audit 1 from='client.? 172.21.15.69:0/1173474357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.800879+0000 mon.a (mon.0) 1305 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.802961+0000 mon.a (mon.0) 1306 : audit 1 from='client.? 172.21.15.69:0/638232212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:00.808027+0000 mon.a (mon.0) 1307 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[21528]: audit 2024-04-02T04:50:01.642631+0000 mon.c (mon.2) 166 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:49:59.886283+0000 mgr.y (mgr.14538) 239 : cluster 0 pgmap v220: 712 pgs: 126 active+undersized, 293 active+clean, 128 unknown, 128 creating+peering, 37 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3 op/s; 106/870 objects degraded (12.184%) 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.112892+0000 mgr.y (mgr.14538) 240 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.113857+0000 mgr.y (mgr.14538) 241 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.114660+0000 mgr.y (mgr.14538) 242 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.115430+0000 mgr.y (mgr.14538) 243 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.116262+0000 mgr.y (mgr.14538) 244 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.117096+0000 mgr.y (mgr.14538) 245 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.117928+0000 mgr.y (mgr.14538) 246 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:50:02.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.118888+0000 mgr.y (mgr.14538) 247 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.119874+0000 mgr.y (mgr.14538) 248 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.120862+0000 mgr.y (mgr.14538) 249 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.188941+0000 osd.1 (osd.1) 143 : cluster 0 15.7 deep-scrub starts 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.295552+0000 osd.1 (osd.1) 144 : cluster 0 15.7 deep-scrub ok 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.508613+0000 osd.4 (osd.4) 143 : cluster 0 15.3 deep-scrub starts 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.510231+0000 osd.4 (osd.4) 144 : cluster 0 15.3 deep-scrub ok 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.641032+0000 mon.c (mon.2) 163 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.756037+0000 mon.a (mon.0) 1299 : cluster 3 Health check update: Degraded data redundancy: 106/870 objects degraded (12.184%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.778260+0000 mon.a (mon.0) 1300 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi069-72950-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.778450+0000 mon.a (mon.0) 1301 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.778676+0000 mon.a (mon.0) 1302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:02.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.789862+0000 mon.c (mon.2) 164 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.792116+0000 mon.a (mon.0) 1303 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: cluster 2024-04-02T04:50:00.796398+0000 mon.a (mon.0) 1304 : cluster 0 osdmap e164: 8 total, 7 up, 8 in 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.797011+0000 mon.b (mon.1) 136 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.798875+0000 mon.c (mon.2) 165 : audit 1 from='client.? 172.21.15.69:0/1173474357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.800879+0000 mon.a (mon.0) 1305 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.802961+0000 mon.a (mon.0) 1306 : audit 1 from='client.? 172.21.15.69:0/638232212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:00.808027+0000 mon.a (mon.0) 1307 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:02 smithi069 bash[28904]: audit 2024-04-02T04:50:01.642631+0000 mon.c (mon.2) 166 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:02.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:02.340 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:02.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:49:59.886283+0000 mgr.y (mgr.14538) 239 : cluster 0 pgmap v220: 712 pgs: 126 active+undersized, 293 active+clean, 128 unknown, 128 creating+peering, 37 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3 op/s; 106/870 objects degraded (12.184%) 2024-04-02T04:50:02.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.112892+0000 mgr.y (mgr.14538) 240 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-04-02T04:50:02.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.113857+0000 mgr.y (mgr.14538) 241 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.114660+0000 mgr.y (mgr.14538) 242 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.115430+0000 mgr.y (mgr.14538) 243 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.116262+0000 mgr.y (mgr.14538) 244 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.117096+0000 mgr.y (mgr.14538) 245 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.117928+0000 mgr.y (mgr.14538) 246 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.118888+0000 mgr.y (mgr.14538) 247 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.119874+0000 mgr.y (mgr.14538) 248 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.120862+0000 mgr.y (mgr.14538) 249 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.188941+0000 osd.1 (osd.1) 143 : cluster 0 15.7 deep-scrub starts 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.295552+0000 osd.1 (osd.1) 144 : cluster 0 15.7 deep-scrub ok 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.508613+0000 osd.4 (osd.4) 143 : cluster 0 15.3 deep-scrub starts 2024-04-02T04:50:02.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.510231+0000 osd.4 (osd.4) 144 : cluster 0 15.3 deep-scrub ok 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.641032+0000 mon.c (mon.2) 163 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.756037+0000 mon.a (mon.0) 1299 : cluster 3 Health check update: Degraded data redundancy: 106/870 objects degraded (12.184%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.778260+0000 mon.a (mon.0) 1300 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi069-72950-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.778450+0000 mon.a (mon.0) 1301 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.778676+0000 mon.a (mon.0) 1302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.789862+0000 mon.c (mon.2) 164 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.792116+0000 mon.a (mon.0) 1303 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: cluster 2024-04-02T04:50:00.796398+0000 mon.a (mon.0) 1304 : cluster 0 osdmap e164: 8 total, 7 up, 8 in 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.797011+0000 mon.b (mon.1) 136 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.798875+0000 mon.c (mon.2) 165 : audit 1 from='client.? 172.21.15.69:0/1173474357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.800879+0000 mon.a (mon.0) 1305 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.802961+0000 mon.a (mon.0) 1306 : audit 1 from='client.? 172.21.15.69:0/638232212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:00.808027+0000 mon.a (mon.0) 1307 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:02.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:02 smithi098 bash[23790]: audit 2024-04-02T04:50:01.642631+0000 mon.c (mon.2) 166 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:02.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:02.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:02.593 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:02.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:02.718 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:02.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -n osd.7 --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 66.c 2024-04-02T04:50:02.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:02.846 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/osd.7/config 2024-04-02T04:50:02.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:02.969 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:02.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:03.072 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:03.093 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:03.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:50:03.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:03.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:01.026622+0000 osd.0 (osd.0) 141 : cluster 0 15.6 deep-scrub starts 2024-04-02T04:50:03.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:01.028322+0000 osd.0 (osd.0) 142 : cluster 0 15.6 deep-scrub ok 2024-04-02T04:50:03.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:01.193460+0000 osd.1 (osd.1) 145 : cluster 0 15.4 deep-scrub starts 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:01.296020+0000 osd.1 (osd.1) 146 : cluster 0 15.4 deep-scrub ok 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:01.887223+0000 mgr.y (mgr.14538) 250 : cluster 0 pgmap v222: 624 pgs: 113 active+undersized, 275 active+clean, 168 unknown, 32 creating+peering, 36 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 4 op/s; 105/867 objects degraded (12.111%) 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:02.180068+0000 mon.a (mon.0) 1308 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]': finished 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:02.180140+0000 mon.a (mon.0) 1309 : audit 1 from='client.? 172.21.15.69:0/638232212' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:02.180197+0000 mon.a (mon.0) 1310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:02.207196+0000 mon.c (mon.2) 167 : audit 1 from='client.? 172.21.15.69:0/1980413310' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:02.279208+0000 mon.c (mon.2) 168 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:02.644260+0000 mon.c (mon.2) 169 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:03.066349+0000 mon.a (mon.0) 1311 : cluster 0 osdmap e165: 8 total, 7 up, 8 in 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: cluster 2024-04-02T04:50:03.067022+0000 mon.a (mon.0) 1312 : cluster 3 Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:03.072189+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:03 smithi098 bash[23790]: audit 2024-04-02T04:50:03.099145+0000 mon.a (mon.0) 1314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:03.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:03.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:01.026622+0000 osd.0 (osd.0) 141 : cluster 0 15.6 deep-scrub starts 2024-04-02T04:50:03.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:01.028322+0000 osd.0 (osd.0) 142 : cluster 0 15.6 deep-scrub ok 2024-04-02T04:50:03.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:01.193460+0000 osd.1 (osd.1) 145 : cluster 0 15.4 deep-scrub starts 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:01.296020+0000 osd.1 (osd.1) 146 : cluster 0 15.4 deep-scrub ok 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:01.887223+0000 mgr.y (mgr.14538) 250 : cluster 0 pgmap v222: 624 pgs: 113 active+undersized, 275 active+clean, 168 unknown, 32 creating+peering, 36 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 4 op/s; 105/867 objects degraded (12.111%) 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:02.180068+0000 mon.a (mon.0) 1308 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]': finished 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:02.180140+0000 mon.a (mon.0) 1309 : audit 1 from='client.? 172.21.15.69:0/638232212' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:02.180197+0000 mon.a (mon.0) 1310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:02.207196+0000 mon.c (mon.2) 167 : audit 1 from='client.? 172.21.15.69:0/1980413310' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:02.279208+0000 mon.c (mon.2) 168 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:02.644260+0000 mon.c (mon.2) 169 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:03.066349+0000 mon.a (mon.0) 1311 : cluster 0 osdmap e165: 8 total, 7 up, 8 in 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: cluster 2024-04-02T04:50:03.067022+0000 mon.a (mon.0) 1312 : cluster 3 Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:03.072189+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:03.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[21528]: audit 2024-04-02T04:50:03.099145+0000 mon.a (mon.0) 1314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:01.026622+0000 osd.0 (osd.0) 141 : cluster 0 15.6 deep-scrub starts 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:01.028322+0000 osd.0 (osd.0) 142 : cluster 0 15.6 deep-scrub ok 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:01.193460+0000 osd.1 (osd.1) 145 : cluster 0 15.4 deep-scrub starts 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:01.296020+0000 osd.1 (osd.1) 146 : cluster 0 15.4 deep-scrub ok 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:01.887223+0000 mgr.y (mgr.14538) 250 : cluster 0 pgmap v222: 624 pgs: 113 active+undersized, 275 active+clean, 168 unknown, 32 creating+peering, 36 active+undersized+degraded; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 4 op/s; 105/867 objects degraded (12.111%) 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:02.180068+0000 mon.a (mon.0) 1308 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-7"}]': finished 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:02.180140+0000 mon.a (mon.0) 1309 : audit 1 from='client.? 172.21.15.69:0/638232212' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:02.180197+0000 mon.a (mon.0) 1310 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:02.207196+0000 mon.c (mon.2) 167 : audit 1 from='client.? 172.21.15.69:0/1980413310' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:02.279208+0000 mon.c (mon.2) 168 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:02.644260+0000 mon.c (mon.2) 169 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:03.422 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:03.066349+0000 mon.a (mon.0) 1311 : cluster 0 osdmap e165: 8 total, 7 up, 8 in 2024-04-02T04:50:03.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: cluster 2024-04-02T04:50:03.067022+0000 mon.a (mon.0) 1312 : cluster 3 Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:03.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:03.072189+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:03.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:03 smithi069 bash[28904]: audit 2024-04-02T04:50:03.099145+0000 mon.a (mon.0) 1314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:03.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:03.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:03.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/1839893554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406b510 msgr2=0x7ff814106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:03.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- 172.21.15.69:0/1839893554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406b510 0x7ff814106ee0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff8000099c0 tx=0x7ff800009cd0 comp rx=0 tx=0).stop 2024-04-02T04:50:03.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/1839893554 shutdown_connections 2024-04-02T04:50:03.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- 172.21.15.69:0/1839893554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff814107610 0x7ff81410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:03.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- 172.21.15.69:0/1839893554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81406ab60 0x7ff81406af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:03.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- 172.21.15.69:0/1839893554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406b510 0x7ff814106ee0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/1839893554 >> 172.21.15.69:0/1839893554 conn(0x7ff814077ea0 msgr2=0x7ff8140782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/1839893554 shutdown_connections 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/1839893554 wait complete. 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 librados: starting msgr at 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 librados: starting objecter 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 Processor -- start 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- start start 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 librados: setting wanted keys 2024-04-02T04:50:03.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 librados: calling monclient init 2024-04-02T04:50:03.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406ab60 0x7ff814101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81406b510 0x7ff814101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff814107610 0x7ff814101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111b90 con 0x7ff81406b510 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111a10 con 0x7ff814107610 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff814111d10 con 0x7ff81406ab60 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406ab60 0x7ff814101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406ab60 0x7ff814101530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:39084/0 (socket says 172.21.15.69:39084) 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 -- 172.21.15.69:0/2498649038 learned_addr learned my addr 172.21.15.69:0/2498649038 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81406b510 0x7ff814101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff814107610 0x7ff814101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 -- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff814107610 msgr2=0x7ff814101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff814107610 0x7ff814101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 -- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81406b510 msgr2=0x7ff814101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81406b510 0x7ff814101a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:03.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8141a6670 con 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff814107610 0x7ff814101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff81406ab60 0x7ff814101530 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff80800eb30 tx=0x7ff80800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff811ffb700 10 client.?.objecter ms_handle_connect 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff811ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80800cc90 con 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8141a6960 con 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff80800ce30 con 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808003df0 con 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 10 client.25550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141a6e70 con 0x7ff81406ab60 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 librados: init done 2024-04-02T04:50:03.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff81404ff00 con 0x7ff81406ab60 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.597+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff808003f90 con 0x7ff81406ab60 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff80406f5e0 0x7ff804071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(166..166 src has 74..166) ==== 15109+0+0 (secure 0 0 0) 0x7ff8080900d0 con 0x7ff81406ab60 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 osd_map(166..166 src has 74..166) 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 3 client.25550.objecter handle_osd_map got epochs [166,166] > 0 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 3 client.25550.objecter handle_osd_map decoding full epoch 166 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 20 client.25550.objecter dump_active .. 0 homeless 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v166) ==== 54+0+16 (secure 0 0 0) 0x7ff808092670 con 0x7ff81406ab60 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff80406f5e0 0x7ff804071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 10 client.25550.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 1 2024-04-02T04:50:03.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7ff814062e50 0x7ff814063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 1 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 0 1 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814102a60 osd=0 4 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 10 client.25550.objecter _send_command 1 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8141aa290 con 0x7ff814062e50 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7ff814062e50 0x7ff814063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff80406f5e0 0x7ff804071aa0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff800000c00 tx=0x7ff800019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff80406f5e0 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff814062e50 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7ff814062e50 0x7ff814063230 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8141aa290 con 0x7ff814062e50 2024-04-02T04:50:03.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 1: 0 ) 2024-04-02T04:50:03.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 10 client.25550.objecter _finish_command 1 = system:0 2024-04-02T04:50:03.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff814102a60 osd=0 4 2024-04-02T04:50:03.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.601+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 0 1 2024-04-02T04:50:03.615 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 10 client.25550.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 2 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 2 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff814102a60 osd=0 3 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 0 2 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814102a60 osd=0 4 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 10 client.25550.objecter _send_command 2 2024-04-02T04:50:03.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.625+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8140628d0 con 0x7ff814062e50 2024-04-02T04:50:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8140628d0 con 0x7ff814062e50 2024-04-02T04:50:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 10 client.25550.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff814102a60 osd=0 4 2024-04-02T04:50:03.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 0 2 2024-04-02T04:50:03.628 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 10 client.25550.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7ff81402c9b0 0x7ff81402cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 1 3 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141098a0 osd=1 4 2024-04-02T04:50:03.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 10 client.25550.objecter _send_command 3 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff814109f20 con 0x7ff81402c9b0 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7ff81402c9b0 0x7ff81402cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff81402c9b0 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7ff81402c9b0 0x7ff81402cdd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff814109f20 con 0x7ff81402c9b0 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 3: 0 ) 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 10 client.25550.objecter _finish_command 3 = system:0 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141098a0 osd=1 4 2024-04-02T04:50:03.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.629+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 1 3 2024-04-02T04:50:03.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 10 client.25550.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 4 2024-04-02T04:50:03.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 4 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141098a0 osd=1 3 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 1 4 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141098a0 osd=1 4 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 10 client.25550.objecter _send_command 4 2024-04-02T04:50:03.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.645+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff814109320 con 0x7ff81402c9b0 2024-04-02T04:50:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff814109320 con 0x7ff81402c9b0 2024-04-02T04:50:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff811ffb700 10 client.25550.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141098a0 osd=1 4 2024-04-02T04:50:03.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 1 4 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 10 client.25550.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 5 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ff8141ac450 0x7ff8141ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 5 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 2 5 2024-04-02T04:50:03.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814109720 osd=2 4 2024-04-02T04:50:03.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 10 client.25550.objecter _send_command 5 2024-04-02T04:50:03.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.649+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff8141b16a0 con 0x7ff8141ac450 2024-04-02T04:50:03.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.673+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ff8141ac450 0x7ff8141ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.673+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff8141ac450 2024-04-02T04:50:03.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.673+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7ff8141ac450 0x7ff8141ae840 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.677+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8141b16a0 con 0x7ff8141ac450 2024-04-02T04:50:03.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.677+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 5: 0 ) 2024-04-02T04:50:03.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.677+0000 7ff811ffb700 10 client.25550.objecter _finish_command 5 = system:0 2024-04-02T04:50:03.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.677+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff814109720 osd=2 4 2024-04-02T04:50:03.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.677+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 2 5 2024-04-02T04:50:03.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 10 client.25550.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 6 2024-04-02T04:50:03.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 6 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff814109720 osd=2 3 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 2 6 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff814109720 osd=2 4 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 10 client.25550.objecter _send_command 6 2024-04-02T04:50:03.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff81402d560 con 0x7ff8141ac450 2024-04-02T04:50:03.708 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff81402d560 con 0x7ff8141ac450 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff811ffb700 10 client.25550.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff814109720 osd=2 4 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.705+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 2 6 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 10 client.25550.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 7 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7ff8141aff00 0x7ff8141b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 7 2024-04-02T04:50:03.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 3 7 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a98e0 osd=3 4 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 10 client.25550.objecter _send_command 7 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff8141a9fd0 con 0x7ff8141aff00 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7ff8141aff00 0x7ff8141b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff8141aff00 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7ff8141aff00 0x7ff8141b0340 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8141a9fd0 con 0x7ff8141aff00 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 7: 0 ) 2024-04-02T04:50:03.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff811ffb700 10 client.25550.objecter _finish_command 7 = system:0 2024-04-02T04:50:03.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141a98e0 osd=3 4 2024-04-02T04:50:03.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.709+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 3 7 2024-04-02T04:50:03.718 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:03.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 10 client.25550.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 8 2024-04-02T04:50:03.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 8 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141a98e0 osd=3 3 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 3 8 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a98e0 osd=3 4 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 10 client.25550.objecter _send_command 8 2024-04-02T04:50:03.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.729+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8141a9360 con 0x7ff8141aff00 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8141a9360 con 0x7ff8141aff00 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff811ffb700 10 client.25550.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141a98e0 osd=3 4 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 3 8 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:50:03.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 10 client.25550.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 9 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7ff8141b3860 0x7ff8141b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 9 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 4 9 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a9110 osd=4 4 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 10 client.25550.objecter _send_command 9 2024-04-02T04:50:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.733+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff8141a9310 con 0x7ff8141b3860 2024-04-02T04:50:03.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7ff8141b3860 0x7ff8141b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff818f27700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7ff8141b3860 0x7ff8141b5c50 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff8141b3860 2024-04-02T04:50:03.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8141a9310 con 0x7ff8141b3860 2024-04-02T04:50:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 9: 0 ) 2024-04-02T04:50:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff811ffb700 10 client.25550.objecter _finish_command 9 = system:0 2024-04-02T04:50:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141a9110 osd=4 4 2024-04-02T04:50:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.737+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 4 9 2024-04-02T04:50:03.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 10 client.25550.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 10 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 10 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141a9110 osd=4 3 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 4 10 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141a9110 osd=4 4 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 10 client.25550.objecter _send_command 10 2024-04-02T04:50:03.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.765+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8141b6190 con 0x7ff8141b3860 2024-04-02T04:50:03.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8141b6190 con 0x7ff8141b3860 2024-04-02T04:50:03.770 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff811ffb700 10 client.25550.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141a9110 osd=4 4 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 4 10 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 10 client.25550.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 11 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7ff8141b68b0 0x7ff8141b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 11 2024-04-02T04:50:03.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 5 11 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141b6190 osd=5 4 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 10 client.25550.objecter _send_command 11 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff8141bbbb0 con 0x7ff8141b68b0 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7ff8141b68b0 0x7ff8141b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff8141b68b0 2024-04-02T04:50:03.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.769+0000 7ff813fff700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7ff8141b68b0 0x7ff8141b8ca0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.773+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8141bbbb0 con 0x7ff8141b68b0 2024-04-02T04:50:03.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.773+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 11: 0 ) 2024-04-02T04:50:03.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.773+0000 7ff811ffb700 10 client.25550.objecter _finish_command 11 = system:0 2024-04-02T04:50:03.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.773+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141b6190 osd=5 4 2024-04-02T04:50:03.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.773+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 5 11 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 10 client.25550.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 12 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 12 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141b6190 osd=5 3 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 5 12 2024-04-02T04:50:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141b6190 osd=5 4 2024-04-02T04:50:03.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 10 client.25550.objecter _send_command 12 2024-04-02T04:50:03.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.797+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8141bbbb0 con 0x7ff8141b68b0 2024-04-02T04:50:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8141bbbb0 con 0x7ff8141b68b0 2024-04-02T04:50:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff811ffb700 10 client.25550.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141b6190 osd=5 4 2024-04-02T04:50:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 5 12 2024-04-02T04:50:03.804 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:50:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 10 client.25550.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 13 2024-04-02T04:50:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7ff8141b9750 0x7ff8141b9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 13 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 6 13 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141bbbb0 osd=6 4 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 10 client.25550.objecter _send_command 13 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.801+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff8141bef50 con 0x7ff8141b9750 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7ff8141b9750 0x7ff8141b9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff811ffb700 10 client.25550.objecter ms_handle_connect 0x7ff8141b9750 2024-04-02T04:50:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff819728700 1 --2- 172.21.15.69:0/2498649038 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7ff8141b9750 0x7ff8141b9bd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:03.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8141bef50 con 0x7ff8141b9750 2024-04-02T04:50:03.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 13: 0 ) 2024-04-02T04:50:03.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff811ffb700 10 client.25550.objecter _finish_command 13 = system:0 2024-04-02T04:50:03.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141bbbb0 osd=6 4 2024-04-02T04:50:03.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.805+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 6 13 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 10 client.25550.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 14 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 20 client.25550.objecter _get_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_remove -1 14 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 20 client.25550.objecter get_session s=0x7ff8141bbbb0 osd=6 3 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign 6 14 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 20 client.25550.objecter put_session s=0x7ff8141bbbb0 osd=6 4 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 10 client.25550.objecter _send_command 14 2024-04-02T04:50:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.829+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8141ba220 con 0x7ff8141b9750 2024-04-02T04:50:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.833+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8141ba220 con 0x7ff8141b9750 2024-04-02T04:50:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.833+0000 7ff811ffb700 10 client.25550.objecter ms_dispatch 0x7ff814001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.833+0000 7ff811ffb700 10 client.25550.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.833+0000 7ff811ffb700 20 client.25550.objecter put_session s=0x7ff8141bbbb0 osd=6 4 2024-04-02T04:50:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.833+0000 7ff811ffb700 15 client.25550.objecter _session_command_op_remove 6 14 2024-04-02T04:50:03.836 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:50:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff81b18b700 10 client.25550.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff81b18b700 15 client.25550.objecter _session_command_op_assign -1 15 2024-04-02T04:50:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff81b18b700 20 client.25550.objecter _get_session osd=-1 returning homeless 2024-04-02T04:50:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff81b18b700 10 client.25550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=167}) -- 0x7ff8141bb600 con 0x7ff81406ab60 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff81b18b700 1 -- 172.21.15.69:0/2498649038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff8141ba220 con 0x7ff81406ab60 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff811ffb700 1 -- 172.21.15.69:0/2498649038 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_get_version_reply(handle=1 version=166) ==== 24+0+0 (secure 0 0 0) 0x7ff808097070 con 0x7ff81406ab60 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff812ffd700 10 client.25550.objecter _check_command_map_dne tid 15 current 166 map_dne_bound 166 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff812ffd700 10 client.25550.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:03.837+0000 7ff812ffd700 15 client.25550.objecter _session_command_op_remove -1 15 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:50:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:50:03.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:03.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:03.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:03.998 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:04.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:04.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:04.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:04.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:04.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:04.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:04.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:04.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:04.747 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:04.768 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:04.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:04.892 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:04.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:04.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:05.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:05.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: cluster 2024-04-02T04:50:02.984567+0000 osd.1 (osd.1) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: cluster 2024-04-02T04:50:02.984657+0000 osd.1 (osd.1) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.016413+0000 mgr.y (mgr.14538) 251 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.185222+0000 mon.a (mon.0) 1315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.185395+0000 mon.a (mon.0) 1316 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: cluster 2024-04-02T04:50:03.196133+0000 mon.a (mon.0) 1317 : cluster 0 osdmap e166: 8 total, 7 up, 8 in 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.212282+0000 mon.c (mon.2) 170 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.603891+0000 mon.c (mon.2) 171 : audit 0 from='client.? 172.21.15.69:0/2498649038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.645386+0000 mon.c (mon.2) 172 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:05.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:05 smithi098 bash[23790]: audit 2024-04-02T04:50:03.699775+0000 mon.a (mon.0) 1318 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:05.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:05.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: cluster 2024-04-02T04:50:02.984567+0000 osd.1 (osd.1) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:05.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: cluster 2024-04-02T04:50:02.984657+0000 osd.1 (osd.1) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:05.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.016413+0000 mgr.y (mgr.14538) 251 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.185222+0000 mon.a (mon.0) 1315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.185395+0000 mon.a (mon.0) 1316 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: cluster 2024-04-02T04:50:03.196133+0000 mon.a (mon.0) 1317 : cluster 0 osdmap e166: 8 total, 7 up, 8 in 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.212282+0000 mon.c (mon.2) 170 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.603891+0000 mon.c (mon.2) 171 : audit 0 from='client.? 172.21.15.69:0/2498649038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.645386+0000 mon.c (mon.2) 172 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[28904]: audit 2024-04-02T04:50:03.699775+0000 mon.a (mon.0) 1318 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: cluster 2024-04-02T04:50:02.984567+0000 osd.1 (osd.1) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:05.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: cluster 2024-04-02T04:50:02.984657+0000 osd.1 (osd.1) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.016413+0000 mgr.y (mgr.14538) 251 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.185222+0000 mon.a (mon.0) 1315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-72145-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.185395+0000 mon.a (mon.0) 1316 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: cluster 2024-04-02T04:50:03.196133+0000 mon.a (mon.0) 1317 : cluster 0 osdmap e166: 8 total, 7 up, 8 in 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.212282+0000 mon.c (mon.2) 170 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.603891+0000 mon.c (mon.2) 171 : audit 0 from='client.? 172.21.15.69:0/2498649038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.645386+0000 mon.c (mon.2) 172 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:05.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:05 smithi069 bash[21528]: audit 2024-04-02T04:50:03.699775+0000 mon.a (mon.0) 1318 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]: dispatch 2024-04-02T04:50:05.280 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:05.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:05.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:05.536 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-02T04:50:05.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:05.659 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:05.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:05.788 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:05.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:05.918 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:05.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:06.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:06.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:06.196 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:06.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:06.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:06.462 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: cluster 2024-04-02T04:50:03.888228+0000 mgr.y (mgr.14538) 252 : cluster 0 pgmap v225: 656 pgs: 137 active+undersized, 320 active+clean, 82 creating+peering, 80 unknown, 37 active+undersized+degraded; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 16 MiB/s wr, 5 op/s; 109/906 objects degraded (12.031%) 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:04.592072+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]': finished 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:04.614150+0000 mon.c (mon.2) 173 : audit 1 from='client.? 172.21.15.69:0/2131827544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:04.646212+0000 mon.c (mon.2) 174 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.074305+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e167: 8 total, 7 up, 8 in 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.075345+0000 osd.5 (osd.5) 131 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.075517+0000 osd.5 (osd.5) 132 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.084590+0000 mon.c (mon.2) 175 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]: dispatch 2024-04-02T04:50:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.090137+0000 mon.a (mon.0) 1321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.099010+0000 mon.a (mon.0) 1322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.647436+0000 mon.c (mon.2) 176 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.928706+0000 mon.a (mon.0) 1323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.928957+0000 mon.a (mon.0) 1324 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]': finished 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.935778+0000 mon.c (mon.2) 177 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.946800+0000 mon.a (mon.0) 1325 : cluster 0 osdmap e168: 8 total, 7 up, 8 in 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.949075+0000 mon.c (mon.2) 178 : audit 1 from='client.? 172.21.15.69:0/1307684559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.959016+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.959948+0000 mon.a (mon.0) 1327 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:06 smithi098 bash[23790]: audit 2024-04-02T04:50:05.961485+0000 mon.a (mon.0) 1328 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:50:06.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:06.589 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:06.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:06.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: cluster 2024-04-02T04:50:03.888228+0000 mgr.y (mgr.14538) 252 : cluster 0 pgmap v225: 656 pgs: 137 active+undersized, 320 active+clean, 82 creating+peering, 80 unknown, 37 active+undersized+degraded; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 16 MiB/s wr, 5 op/s; 109/906 objects degraded (12.031%) 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:04.592072+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]': finished 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:04.614150+0000 mon.c (mon.2) 173 : audit 1 from='client.? 172.21.15.69:0/2131827544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:04.646212+0000 mon.c (mon.2) 174 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.074305+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e167: 8 total, 7 up, 8 in 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.075345+0000 osd.5 (osd.5) 131 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.075517+0000 osd.5 (osd.5) 132 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.084590+0000 mon.c (mon.2) 175 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]: dispatch 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.090137+0000 mon.a (mon.0) 1321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.099010+0000 mon.a (mon.0) 1322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]: dispatch 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.647436+0000 mon.c (mon.2) 176 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.928706+0000 mon.a (mon.0) 1323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.928957+0000 mon.a (mon.0) 1324 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]': finished 2024-04-02T04:50:06.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.935778+0000 mon.c (mon.2) 177 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.946800+0000 mon.a (mon.0) 1325 : cluster 0 osdmap e168: 8 total, 7 up, 8 in 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.949075+0000 mon.c (mon.2) 178 : audit 1 from='client.? 172.21.15.69:0/1307684559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.959016+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.959948+0000 mon.a (mon.0) 1327 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[21528]: audit 2024-04-02T04:50:05.961485+0000 mon.a (mon.0) 1328 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: cluster 2024-04-02T04:50:03.888228+0000 mgr.y (mgr.14538) 252 : cluster 0 pgmap v225: 656 pgs: 137 active+undersized, 320 active+clean, 82 creating+peering, 80 unknown, 37 active+undersized+degraded; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 16 MiB/s wr, 5 op/s; 109/906 objects degraded (12.031%) 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:04.592072+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-7"}]': finished 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:04.614150+0000 mon.c (mon.2) 173 : audit 1 from='client.? 172.21.15.69:0/2131827544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:04.646212+0000 mon.c (mon.2) 174 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:06.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.074305+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e167: 8 total, 7 up, 8 in 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.075345+0000 osd.5 (osd.5) 131 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.075517+0000 osd.5 (osd.5) 132 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.084590+0000 mon.c (mon.2) 175 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]: dispatch 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.090137+0000 mon.a (mon.0) 1321 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.099010+0000 mon.a (mon.0) 1322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]: dispatch 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.647436+0000 mon.c (mon.2) 176 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.928706+0000 mon.a (mon.0) 1323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72587-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.928957+0000 mon.a (mon.0) 1324 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4"}]': finished 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.935778+0000 mon.c (mon.2) 177 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.946800+0000 mon.a (mon.0) 1325 : cluster 0 osdmap e168: 8 total, 7 up, 8 in 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.949075+0000 mon.c (mon.2) 178 : audit 1 from='client.? 172.21.15.69:0/1307684559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.959016+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]: dispatch 2024-04-02T04:50:06.705 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.959948+0000 mon.a (mon.0) 1327 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:06.705 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:06 smithi069 bash[28904]: audit 2024-04-02T04:50:05.961485+0000 mon.a (mon.0) 1328 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:50:06.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:06.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:06.838 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:06.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:06.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:50:07.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:50:07.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:07.088 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/402298237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c102c90 msgr2=0x7f8f2c10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- 172.21.15.69:0/402298237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c102c90 0x7f8f2c10f170 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400b0a0 tx=0x7f8f2400b3b0 comp rx=0 tx=0).stop 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/402298237 shutdown_connections 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- 172.21.15.69:0/402298237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c102c90 0x7f8f2c10f170 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- 172.21.15.69:0/402298237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f2c108330 0x7f8f2c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- 172.21.15.69:0/402298237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c1022f0 0x7f8f2c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/402298237 >> 172.21.15.69:0/402298237 conn(0x7f8f2c0793f0 msgr2=0x7f8f2c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/402298237 shutdown_connections 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/402298237 wait complete. 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 librados: starting msgr at 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 librados: starting objecter 2024-04-02T04:50:07.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 Processor -- start 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- start start 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 librados: setting wanted keys 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 librados: calling monclient init 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 0x7f8f2c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f2c102c90 0x7f8f2c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c108330 0x7f8f2c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c111a90 con 0x7f8f2c108330 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c111910 con 0x7f8f2c102c90 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f2c111c10 con 0x7f8f2c1022f0 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 0x7f8f2c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 0x7f8f2c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:39254/0 (socket says 172.21.15.69:39254) 2024-04-02T04:50:07.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 -- 172.21.15.69:0/2644139754 learned_addr learned my addr 172.21.15.69:0/2644139754 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:07.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f3080f700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c108330 0x7f8f2c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f23fff700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f2c102c90 0x7f8f2c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 -- 172.21.15.69:0/2644139754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f2c102c90 msgr2=0x7f8f2c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f2c102c90 0x7f8f2c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 -- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c108330 msgr2=0x7f8f2c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c108330 0x7f8f2c19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 -- 172.21.15.69:0/2644139754 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c19cf60 con 0x7f8f2c1022f0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f3080f700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c108330 0x7f8f2c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f2bfff700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 0x7f8f2c197f50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00c9b0 tx=0x7f8f1c00cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f29ffb700 10 client.?.objecter ms_handle_connect 0x7f8f2c1022f0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f29ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c007b00 con 0x7f8f2c1022f0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/2644139754 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2c19d250 con 0x7f8f2c1022f0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8f1c007ca0 con 0x7f8f2c1022f0 2024-04-02T04:50:07.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c018660 con 0x7f8f2c1022f0 2024-04-02T04:50:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 10 client.25592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 -- 172.21.15.69:0/2644139754 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2c074470 con 0x7f8f2c1022f0 2024-04-02T04:50:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.141+0000 7f8f32272700 1 librados: init done 2024-04-02T04:50:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f04005120 con 0x7f8f2c1022f0 2024-04-02T04:50:07.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8f1c004830 con 0x7f8f2c1022f0 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f0c06f5e0 0x7f8f0c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(168..168 src has 74..168) ==== 14746+0+0 (secure 0 0 0) 0x7f8f1c08fcc0 con 0x7f8f2c1022f0 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 10 client.25592.objecter ms_dispatch 0x7f8f2c001550 osd_map(168..168 src has 74..168) 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 3 client.25592.objecter handle_osd_map got epochs [168,168] > 0 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 3 client.25592.objecter handle_osd_map decoding full epoch 168 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f23fff700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f0c06f5e0 0x7f8f0c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 20 client.25592.objecter dump_active .. 0 homeless 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f23fff700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f0c06f5e0 0x7f8f0c071aa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c079d40 tx=0x7f8f140094f0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:07.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.145+0000 7f8f29ffb700 10 client.25592.objecter ms_handle_connect 0x7f8f0c06f5e0 2024-04-02T04:50:07.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.153+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8f1c014030 con 0x7f8f2c1022f0 2024-04-02T04:50:07.177 INFO:teuthology.orchestra.run.smithi098.stdout: marking collection for removal 2024-04-02T04:50:07.177 INFO:teuthology.orchestra.run.smithi098.stdout:setting '_remove' omap key 2024-04-02T04:50:07.177 INFO:teuthology.orchestra.run.smithi098.stdout:finish_remove_pgs 66.c_head removing 66.c 2024-04-02T04:50:07.178 INFO:teuthology.orchestra.run.smithi098.stdout:Remove successful 2024-04-02T04:50:07.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.212 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:07.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:07.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:07.382 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:50:07.382 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:50:07.382 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2024-04-02T04:50:07.382 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:50:07.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:07.477+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8f04005410 con 0x7f8f2c1022f0 2024-04-02T04:50:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:07.583 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:07.605 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:07.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.734 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:07.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.859 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:07.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:07.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:05.972475+0000 mon.c (mon.2) 179 : audit 1 from='client.? 172.21.15.69:0/3782423706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:07.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:06.415463+0000 mon.a (mon.0) 1329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:07.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:06.648920+0000 mon.c (mon.2) 180 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:07.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: cluster 2024-04-02T04:50:07.360675+0000 mon.a (mon.0) 1330 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:07.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.362841+0000 mon.a (mon.0) 1331 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]': finished 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.362935+0000 mon.a (mon.0) 1332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.363068+0000 mon.a (mon.0) 1333 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.363141+0000 mon.a (mon.0) 1334 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.371848+0000 mon.c (mon.2) 181 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.372836+0000 mon.c (mon.2) 182 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: cluster 2024-04-02T04:50:07.379756+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e169: 8 total, 7 up, 8 in 2024-04-02T04:50:07.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:07 smithi098 bash[23790]: audit 2024-04-02T04:50:07.385498+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:50:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:07.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:08.086 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:08.114 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:08.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:08.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:05.972475+0000 mon.c (mon.2) 179 : audit 1 from='client.? 172.21.15.69:0/3782423706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:06.415463+0000 mon.a (mon.0) 1329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:06.648920+0000 mon.c (mon.2) 180 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: cluster 2024-04-02T04:50:07.360675+0000 mon.a (mon.0) 1330 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.362841+0000 mon.a (mon.0) 1331 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]': finished 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.362935+0000 mon.a (mon.0) 1332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.363068+0000 mon.a (mon.0) 1333 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.363141+0000 mon.a (mon.0) 1334 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.371848+0000 mon.c (mon.2) 181 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.372836+0000 mon.c (mon.2) 182 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: cluster 2024-04-02T04:50:07.379756+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e169: 8 total, 7 up, 8 in 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[21528]: audit 2024-04-02T04:50:07.385498+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:05.972475+0000 mon.c (mon.2) 179 : audit 1 from='client.? 172.21.15.69:0/3782423706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:06.415463+0000 mon.a (mon.0) 1329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:06.648920+0000 mon.c (mon.2) 180 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: cluster 2024-04-02T04:50:07.360675+0000 mon.a (mon.0) 1330 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.362841+0000 mon.a (mon.0) 1331 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-72439-4", "tierpool": "test-rados-api-smithi069-72439-6"}]': finished 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.362935+0000 mon.a (mon.0) 1332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-72153-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.363068+0000 mon.a (mon.0) 1333 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.363141+0000 mon.a (mon.0) 1334 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-72145-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.371848+0000 mon.c (mon.2) 181 : audit 1 from='client.? 172.21.15.69:0/1791105380' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.372836+0000 mon.c (mon.2) 182 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:08.229 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: cluster 2024-04-02T04:50:07.379756+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e169: 8 total, 7 up, 8 in 2024-04-02T04:50:08.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:07 smithi069 bash[28904]: audit 2024-04-02T04:50:07.385498+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi069-72950-3"}]: dispatch 2024-04-02T04:50:08.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:08.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:08.376 INFO:tasks.workunit.client.0.smithi069.stdout: api_ ec_list: Running main() from gmock_main.cc 2024-04-02T04:50:08.376 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] Global test environment set-up. 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (35996 ms) 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (11064 ms) 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (10995 ms) 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] 3 tests from NeoradosECList (58055 ms total) 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] Global test environment tear-down 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (58055 ms total) 2024-04-02T04:50:08.377 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-02T04:50:08.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.377+0000 7f8f29ffb700 1 -- 172.21.15.69:0/2644139754 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v170) ==== 91+0+0 (secure 0 0 0) 0x7f8f1c01f300 con 0x7f8f2c1022f0 2024-04-02T04:50:08.384 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2024-04-02T04:50:08.385 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (19536 ms) 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (12599 ms) 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (6928 ms) 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (4591 ms) 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4584 ms) 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4812 ms) 2024-04-02T04:50:08.386 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-02T04:50:08.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (5176 ms) 2024-04-02T04:50:08.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (58226 ms total) 2024-04-02T04:50:08.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2024-04-02T04:50:08.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-02T04:50:08.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (58226 ms total) 2024-04-02T04:50:08.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-02T04:50:08.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 10 librados: watch_flush enter 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 10 librados: watch_flush exit 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 20 client.25592.objecter shutdown clearing up homeless session... 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 10 client.25592.objecter successfully canceled tick 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f0c06f5e0 msgr2=0x7f8f0c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f0c06f5e0 0x7f8f0c071aa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c079d40 tx=0x7f8f140094f0 comp rx=0 tx=0).stop 2024-04-02T04:50:08.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 msgr2=0x7f8f2c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:08.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.385+0000 7f8f22ffd700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 0x7f8f2c197f50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00c9b0 tx=0x7f8f1c00cd70 comp rx=0 tx=0).stop 2024-04-02T04:50:08.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:08.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.389+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 shutdown_connections 2024-04-02T04:50:08.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.389+0000 7f8f22ffd700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f0c06f5e0 0x7f8f0c071aa0 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c079d40 tx=0x7f8f140094f0 comp rx=0 tx=0).stop 2024-04-02T04:50:08.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.389+0000 7f8f22ffd700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f2c108330 0x7f8f2c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:08.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.389+0000 7f8f22ffd700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f2c1022f0 0x7f8f2c197f50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:08.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.389+0000 7f8f22ffd700 1 --2- 172.21.15.69:0/2644139754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f2c102c90 0x7f8f2c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:08.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.389+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 >> 172.21.15.69:0/2644139754 conn(0x7f8f2c0793f0 msgr2=0x7f8f2c105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:08.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.401+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 shutdown_connections 2024-04-02T04:50:08.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.401+0000 7f8f22ffd700 1 -- 172.21.15.69:0/2644139754 wait complete. 2024-04-02T04:50:08.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:08.401+0000 7f8f22ffd700 1 librados: shutdown 2024-04-02T04:50:08.408 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:50:08.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:08.613 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:08.639 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:08.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:08.773 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:08.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-02T04:50:08.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:08.899 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:08.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:08.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell -n osd.7 --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.66.c.7 2024-04-02T04:50:09.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:09.010 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:50:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:50:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:50:09.011 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:50:09.028 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:09.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/osd.7/config 2024-04-02T04:50:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:09.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:09.274 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:09.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout:watch_notify_pp: flushed 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646886544 notify_id 682899800069 notifier_gid 25337 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2024-04-02T04:50:09.387 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3008 ms) 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646886544 notify_id 695784701955 notifier_gid 25337 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3010 ms) 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2024-04-02T04:50:09.388 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94500646886544 notify_id 704374636553 notifier_gid 25337 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18741 ms total) 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (59288 ms total) 2024-04-02T04:50:09.389 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-02T04:50:09.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.889791+0000 mgr.y (mgr.14538) 253 : cluster 0 pgmap v227: 688 pgs: 137 active+undersized, 320 active+clean, 24 creating+peering, 170 unknown, 37 active+undersized+degraded; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 14 MiB/s wr, 4 op/s; 109/906 objects degraded (12.031%) 2024-04-02T04:50:09.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.901978+0000 osd.2 (osd.2) 83 : cluster 0 15.8 deep-scrub starts 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:05.915172+0000 osd.2 (osd.2) 84 : cluster 0 15.8 deep-scrub ok 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:06.431809+0000 osd.4 (osd.4) 145 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:06.431905+0000 osd.4 (osd.4) 146 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:06.779989+0000 osd.6 (osd.6) 105 : cluster 0 15.5 deep-scrub starts 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:06.842736+0000 osd.6 (osd.6) 106 : cluster 0 15.5 deep-scrub ok 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:06.926637+0000 osd.2 (osd.2) 85 : cluster 0 15.0 deep-scrub starts 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:06.927763+0000 osd.2 (osd.2) 86 : cluster 0 15.0 deep-scrub ok 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:07.483739+0000 mon.c (mon.2) 183 : audit 1 from='client.? 172.21.15.69:0/2644139754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:07.649998+0000 mon.c (mon.2) 184 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:07.899731+0000 mon.a (mon.0) 1337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:07.900031+0000 mon.a (mon.0) 1338 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:07.905801+0000 mon.a (mon.0) 1339 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.329704+0000 mon.a (mon.0) 1340 : cluster 3 Health check update: Degraded data redundancy: 109/906 objects degraded (12.031%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:09.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.329736+0000 mon.a (mon.0) 1341 : cluster 3 Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.364469+0000 mon.a (mon.0) 1342 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:08.375520+0000 mon.a (mon.0) 1343 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:08.375609+0000 mon.a (mon.0) 1344 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:08.375701+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:08.375775+0000 mon.a (mon.0) 1346 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.383091+0000 mon.a (mon.0) 1347 : cluster 0 osdmap e170: 8 total, 7 up, 8 in 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:08.387486+0000 mon.b (mon.1) 137 : audit 1 from='client.? 172.21.15.69:0/783798498' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:09 smithi098 bash[23790]: audit 2024-04-02T04:50:08.406629+0000 mon.a (mon.0) 1348 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:09.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.889791+0000 mgr.y (mgr.14538) 253 : cluster 0 pgmap v227: 688 pgs: 137 active+undersized, 320 active+clean, 24 creating+peering, 170 unknown, 37 active+undersized+degraded; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 14 MiB/s wr, 4 op/s; 109/906 objects degraded (12.031%) 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.901978+0000 osd.2 (osd.2) 83 : cluster 0 15.8 deep-scrub starts 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:05.915172+0000 osd.2 (osd.2) 84 : cluster 0 15.8 deep-scrub ok 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:06.431809+0000 osd.4 (osd.4) 145 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:06.431905+0000 osd.4 (osd.4) 146 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:06.779989+0000 osd.6 (osd.6) 105 : cluster 0 15.5 deep-scrub starts 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:06.842736+0000 osd.6 (osd.6) 106 : cluster 0 15.5 deep-scrub ok 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:06.926637+0000 osd.2 (osd.2) 85 : cluster 0 15.0 deep-scrub starts 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:06.927763+0000 osd.2 (osd.2) 86 : cluster 0 15.0 deep-scrub ok 2024-04-02T04:50:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:07.483739+0000 mon.c (mon.2) 183 : audit 1 from='client.? 172.21.15.69:0/2644139754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:07.649998+0000 mon.c (mon.2) 184 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:07.899731+0000 mon.a (mon.0) 1337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:07.900031+0000 mon.a (mon.0) 1338 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:07.905801+0000 mon.a (mon.0) 1339 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.329704+0000 mon.a (mon.0) 1340 : cluster 3 Health check update: Degraded data redundancy: 109/906 objects degraded (12.031%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.329736+0000 mon.a (mon.0) 1341 : cluster 3 Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.364469+0000 mon.a (mon.0) 1342 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:08.375520+0000 mon.a (mon.0) 1343 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:08.375609+0000 mon.a (mon.0) 1344 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:08.375701+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:08.375775+0000 mon.a (mon.0) 1346 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.383091+0000 mon.a (mon.0) 1347 : cluster 0 osdmap e170: 8 total, 7 up, 8 in 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:08.387486+0000 mon.b (mon.1) 137 : audit 1 from='client.? 172.21.15.69:0/783798498' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[21528]: audit 2024-04-02T04:50:08.406629+0000 mon.a (mon.0) 1348 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.889791+0000 mgr.y (mgr.14538) 253 : cluster 0 pgmap v227: 688 pgs: 137 active+undersized, 320 active+clean, 24 creating+peering, 170 unknown, 37 active+undersized+degraded; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 14 MiB/s wr, 4 op/s; 109/906 objects degraded (12.031%) 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.901978+0000 osd.2 (osd.2) 83 : cluster 0 15.8 deep-scrub starts 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:05.915172+0000 osd.2 (osd.2) 84 : cluster 0 15.8 deep-scrub ok 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:06.431809+0000 osd.4 (osd.4) 145 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:06.431905+0000 osd.4 (osd.4) 146 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:06.779989+0000 osd.6 (osd.6) 105 : cluster 0 15.5 deep-scrub starts 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:06.842736+0000 osd.6 (osd.6) 106 : cluster 0 15.5 deep-scrub ok 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:06.926637+0000 osd.2 (osd.2) 85 : cluster 0 15.0 deep-scrub starts 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:06.927763+0000 osd.2 (osd.2) 86 : cluster 0 15.0 deep-scrub ok 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:07.483739+0000 mon.c (mon.2) 183 : audit 1 from='client.? 172.21.15.69:0/2644139754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:07.649998+0000 mon.c (mon.2) 184 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:07.899731+0000 mon.a (mon.0) 1337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:07.900031+0000 mon.a (mon.0) 1338 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:07.905801+0000 mon.a (mon.0) 1339 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.329704+0000 mon.a (mon.0) 1340 : cluster 3 Health check update: Degraded data redundancy: 109/906 objects degraded (12.031%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.329736+0000 mon.a (mon.0) 1341 : cluster 3 Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.364469+0000 mon.a (mon.0) 1342 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:08.375520+0000 mon.a (mon.0) 1343 : audit 1 from='client.? 172.21.15.69:0/1621702341' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi069-72950-3"}]': finished 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:08.375609+0000 mon.a (mon.0) 1344 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-72439-6", "pool2": "test-rados-api-smithi069-72439-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:08.375701+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:08.375775+0000 mon.a (mon.0) 1346 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.383091+0000 mon.a (mon.0) 1347 : cluster 0 osdmap e170: 8 total, 7 up, 8 in 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:08.387486+0000 mon.b (mon.1) 137 : audit 1 from='client.? 172.21.15.69:0/783798498' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:09 smithi069 bash[28904]: audit 2024-04-02T04:50:08.406629+0000 mon.a (mon.0) 1348 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:09.537 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:09.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:09.704 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:09.807 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:09.828 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:09.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:09.955 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:09.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:10.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:10.082 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:10.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:10.212 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:10.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:10.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: Running main() from gmock_main.cc 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] Global test environment set-up. 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (19349 ms) 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (9013 ms) 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (10028 ms) 2024-04-02T04:50:10.408 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (5064 ms) 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (7002 ms) 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3788 ms) 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (5790 ms) 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (60034 ms total) 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] Global test environment tear-down 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (60034 ms total) 2024-04-02T04:50:10.409 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-02T04:50:10.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:07.892202+0000 mgr.y (mgr.14538) 254 : cluster 0 pgmap v230: 712 pgs: 191 active+undersized, 387 active+clean, 96 creating+peering, 38 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s; 111/921 objects degraded (12.052%) 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.264485+0000 osd.2 (osd.2) 87 : cluster 0 15.2 deep-scrub starts 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.268396+0000 osd.2 (osd.2) 88 : cluster 0 15.2 deep-scrub ok 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:08.476176+0000 mon.c (mon.2) 185 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:08.568695+0000 mon.b (mon.1) 138 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:08.651056+0000 mon.c (mon.2) 186 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.817590+0000 osd.6 (osd.6) 107 : cluster 0 2.17 scrub starts 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:08.818814+0000 osd.6 (osd.6) 108 : cluster 0 2.17 scrub ok 2024-04-02T04:50:10.457 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:09.279018+0000 osd.1 (osd.1) 149 : cluster 0 6.1 scrub starts 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:09.280332+0000 osd.1 (osd.1) 150 : cluster 0 6.1 scrub ok 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.299961+0000 mon.b (mon.1) 139 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.370477+0000 mon.a (mon.0) 1349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.373303+0000 mon.b (mon.1) 140 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.378427+0000 mon.a (mon.0) 1350 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.378522+0000 mon.a (mon.0) 1351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.384076+0000 mon.c (mon.2) 187 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.385053+0000 mon.b (mon.1) 141 : audit 1 from='client.? 172.21.15.69:0/3180693281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.385860+0000 mon.c (mon.2) 188 : audit 1 from='client.? 172.21.15.69:0/1233792253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:09.388664+0000 mon.a (mon.0) 1352 : cluster 0 osdmap e171: 8 total, 7 up, 8 in 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.389768+0000 mon.a (mon.0) 1353 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:10.458 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.390673+0000 mon.a (mon.0) 1354 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.391370+0000 mon.a (mon.0) 1355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:09.463194+0000 osd.4 (osd.4) 147 : cluster 0 4.1b scrub starts 2024-04-02T04:50:10.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: cluster 2024-04-02T04:50:09.464218+0000 osd.4 (osd.4) 148 : cluster 0 4.1b scrub ok 2024-04-02T04:50:10.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.652332+0000 mon.c (mon.2) 189 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:10.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:10 smithi098 bash[23790]: audit 2024-04-02T04:50:09.823042+0000 mon.a (mon.0) 1356 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:10.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:10.590 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:07.892202+0000 mgr.y (mgr.14538) 254 : cluster 0 pgmap v230: 712 pgs: 191 active+undersized, 387 active+clean, 96 creating+peering, 38 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s; 111/921 objects degraded (12.052%) 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.264485+0000 osd.2 (osd.2) 87 : cluster 0 15.2 deep-scrub starts 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.268396+0000 osd.2 (osd.2) 88 : cluster 0 15.2 deep-scrub ok 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:08.476176+0000 mon.c (mon.2) 185 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:08.568695+0000 mon.b (mon.1) 138 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:08.651056+0000 mon.c (mon.2) 186 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.817590+0000 osd.6 (osd.6) 107 : cluster 0 2.17 scrub starts 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:08.818814+0000 osd.6 (osd.6) 108 : cluster 0 2.17 scrub ok 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:09.279018+0000 osd.1 (osd.1) 149 : cluster 0 6.1 scrub starts 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:09.280332+0000 osd.1 (osd.1) 150 : cluster 0 6.1 scrub ok 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.299961+0000 mon.b (mon.1) 139 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.370477+0000 mon.a (mon.0) 1349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:10.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.373303+0000 mon.b (mon.1) 140 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.378427+0000 mon.a (mon.0) 1350 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.378522+0000 mon.a (mon.0) 1351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.384076+0000 mon.c (mon.2) 187 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.385053+0000 mon.b (mon.1) 141 : audit 1 from='client.? 172.21.15.69:0/3180693281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:07.892202+0000 mgr.y (mgr.14538) 254 : cluster 0 pgmap v230: 712 pgs: 191 active+undersized, 387 active+clean, 96 creating+peering, 38 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s; 111/921 objects degraded (12.052%) 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.264485+0000 osd.2 (osd.2) 87 : cluster 0 15.2 deep-scrub starts 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.268396+0000 osd.2 (osd.2) 88 : cluster 0 15.2 deep-scrub ok 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:08.476176+0000 mon.c (mon.2) 185 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:08.568695+0000 mon.b (mon.1) 138 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:08.651056+0000 mon.c (mon.2) 186 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:10.592 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.817590+0000 osd.6 (osd.6) 107 : cluster 0 2.17 scrub starts 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:08.818814+0000 osd.6 (osd.6) 108 : cluster 0 2.17 scrub ok 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:09.279018+0000 osd.1 (osd.1) 149 : cluster 0 6.1 scrub starts 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:09.280332+0000 osd.1 (osd.1) 150 : cluster 0 6.1 scrub ok 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.299961+0000 mon.b (mon.1) 139 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.370477+0000 mon.a (mon.0) 1349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.373303+0000 mon.b (mon.1) 140 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.378427+0000 mon.a (mon.0) 1350 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.378522+0000 mon.a (mon.0) 1351 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.384076+0000 mon.c (mon.2) 187 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:10.593 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.385053+0000 mon.b (mon.1) 141 : audit 1 from='client.? 172.21.15.69:0/3180693281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.385860+0000 mon.c (mon.2) 188 : audit 1 from='client.? 172.21.15.69:0/1233792253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:09.388664+0000 mon.a (mon.0) 1352 : cluster 0 osdmap e171: 8 total, 7 up, 8 in 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.389768+0000 mon.a (mon.0) 1353 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.390673+0000 mon.a (mon.0) 1354 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.391370+0000 mon.a (mon.0) 1355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:09.463194+0000 osd.4 (osd.4) 147 : cluster 0 4.1b scrub starts 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: cluster 2024-04-02T04:50:09.464218+0000 osd.4 (osd.4) 148 : cluster 0 4.1b scrub ok 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.652332+0000 mon.c (mon.2) 189 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[28904]: audit 2024-04-02T04:50:09.823042+0000 mon.a (mon.0) 1356 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.385860+0000 mon.c (mon.2) 188 : audit 1 from='client.? 172.21.15.69:0/1233792253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.594 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:09.388664+0000 mon.a (mon.0) 1352 : cluster 0 osdmap e171: 8 total, 7 up, 8 in 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.389768+0000 mon.a (mon.0) 1353 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.390673+0000 mon.a (mon.0) 1354 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.391370+0000 mon.a (mon.0) 1355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:09.463194+0000 osd.4 (osd.4) 147 : cluster 0 4.1b scrub starts 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: cluster 2024-04-02T04:50:09.464218+0000 osd.4 (osd.4) 148 : cluster 0 4.1b scrub ok 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.652332+0000 mon.c (mon.2) 189 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:10.595 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:10 smithi069 bash[21528]: audit 2024-04-02T04:50:09.823042+0000 mon.a (mon.0) 1356 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:10.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:10.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:10.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:10.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:10.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:10.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:10.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:11.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:11.110 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:11.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:11.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:11.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:11.496 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:11.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-02T04:50:11.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:50:11.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:11.756 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:11.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:11.879 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:11.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:11.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:12.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:12.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:12.133 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:12.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:12.259 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:12.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:12.371 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: audit 2024-04-02T04:50:10.381000+0000 mon.a (mon.0) 1357 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]': finished 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: audit 2024-04-02T04:50:10.381096+0000 mon.a (mon.0) 1358 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: audit 2024-04-02T04:50:10.381164+0000 mon.a (mon.0) 1359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: cluster 2024-04-02T04:50:10.384590+0000 mon.a (mon.0) 1360 : cluster 0 osdmap e172: 8 total, 7 up, 8 in 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: audit 2024-04-02T04:50:10.388598+0000 mon.c (mon.2) 190 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]: dispatch 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: audit 2024-04-02T04:50:10.390392+0000 mon.a (mon.0) 1361 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]: dispatch 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[21528]: audit 2024-04-02T04:50:10.653395+0000 mon.c (mon.2) 191 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: audit 2024-04-02T04:50:10.381000+0000 mon.a (mon.0) 1357 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]': finished 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: audit 2024-04-02T04:50:10.381096+0000 mon.a (mon.0) 1358 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: audit 2024-04-02T04:50:10.381164+0000 mon.a (mon.0) 1359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: cluster 2024-04-02T04:50:10.384590+0000 mon.a (mon.0) 1360 : cluster 0 osdmap e172: 8 total, 7 up, 8 in 2024-04-02T04:50:12.372 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: audit 2024-04-02T04:50:10.388598+0000 mon.c (mon.2) 190 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]: dispatch 2024-04-02T04:50:12.373 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: audit 2024-04-02T04:50:10.390392+0000 mon.a (mon.0) 1361 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]: dispatch 2024-04-02T04:50:12.373 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:12 smithi069 bash[28904]: audit 2024-04-02T04:50:10.653395+0000 mon.c (mon.2) 191 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:12.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: audit 2024-04-02T04:50:10.381000+0000 mon.a (mon.0) 1357 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-9"}]': finished 2024-04-02T04:50:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: audit 2024-04-02T04:50:10.381096+0000 mon.a (mon.0) 1358 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-72153-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: audit 2024-04-02T04:50:10.381164+0000 mon.a (mon.0) 1359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-72145-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: cluster 2024-04-02T04:50:10.384590+0000 mon.a (mon.0) 1360 : cluster 0 osdmap e172: 8 total, 7 up, 8 in 2024-04-02T04:50:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: audit 2024-04-02T04:50:10.388598+0000 mon.c (mon.2) 190 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]: dispatch 2024-04-02T04:50:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: audit 2024-04-02T04:50:10.390392+0000 mon.a (mon.0) 1361 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]: dispatch 2024-04-02T04:50:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:12 smithi098 bash[23790]: audit 2024-04-02T04:50:10.653395+0000 mon.c (mon.2) 191 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:12.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:12.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:12.644 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:12.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:12.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:12.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:13.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:13.026 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:13.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:13.153 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:13.158 INFO:teuthology.orchestra.run.smithi098.stdout:get_pg_num_history pg_num_history pg_num_history(e152 pg_nums {1={21=1},2={77=32},3={79=32},4={81=32},5={81=3},6={83=32},7={134=32},8={134=32},9={135=32},10={135=32},11={135=32},12={135=32},13={135=32},14={135=32},15={135=32},16={135=32},17={135=32},18={135=32},19={135=32},20={135=32},21={136=32},22={136=32},23={136=32},24={136=32},25={136=32},26={136=32},27={136=32},28={136=32},29={136=32},30={136=32},31={136=32},32={136=32},33={137=32},34={137=32},35={137=32},36={137=32},37={137=32},38={137=32},39={137=32},40={137=32},41={137=32},42={138=32},43={138=32},44={139=8},45={139=8},46={140=32},47={140=32},48={140=32},49={141=8},50={141=32},51={142=32},52={142=32},53={142=32},54={142=32},55={143=32},56={143=32},57={144=32},58={144=32},59={144=32},60={145=32},61={145=32},62={145=32},63={146=32},64={146=32},65={146=32},66={147=32},67={148=8},68={148=8},69={148=8},70={149=32},71={149=32},72={149=32},73={150=8},74={150=8},75={150=8},76={150=8},77={150=32},78={151=32},79={151=32},80={151=32},81={151=32},82={151=32},83={151=32},84={151=32},85={152=32},86={152=32}} deleted_pools 138,7,138,8,138,20,138,25,138,35,138,39,140,18,140,23,140,24,140,34,140,37,141,9,141,11,141,26,141,27,141,28,141,29,141,30,141,31,141,32,141,33,141,47,142,21,142,22,142,45,143,42,143,50,144,14,145,12,145,38,145,49,147,16,147,44,147,48,147,51,147,52,147,53,147,54,147,56,147,57,147,59,147,61,147,62,149,58,149,63,150,65,151,64,151,71,152,67,152,68,152,69,152,72) 2024-04-02T04:50:13.158 INFO:teuthology.orchestra.run.smithi098.stdout:Importing pgid 66.c 2024-04-02T04:50:13.159 INFO:teuthology.orchestra.run.smithi098.stdout:write_pg epoch 149 info 66.c( empty local-lis/les=147/149 n=0 ec=147/147 lis/c=147/147 les/c/f=149/149/0 sis=147) 2024-04-02T04:50:13.159 INFO:teuthology.orchestra.run.smithi098.stdout:Import successful 2024-04-02T04:50:13.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:13.278 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:13.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:13.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:13.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:13.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:50:13.509 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:13.531 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:13.622 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:13.623 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:13.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:13.662 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:13.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:13.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:09.892955+0000 mgr.y (mgr.14538) 255 : cluster 0 pgmap v233: 708 pgs: 139 active+undersized, 307 active+clean, 128 unknown, 96 creating+peering, 38 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 111/870 objects degraded (12.759%) 2024-04-02T04:50:13.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:10.314227+0000 osd.1 (osd.1) 151 : cluster 0 4.12 scrub starts 2024-04-02T04:50:13.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:10.316402+0000 osd.1 (osd.1) 152 : cluster 0 4.12 scrub ok 2024-04-02T04:50:13.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:10.506259+0000 osd.5 (osd.5) 133 : cluster 0 6.19 scrub starts 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:10.507414+0000 osd.5 (osd.5) 134 : cluster 0 6.19 scrub ok 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.027581+0000 osd.0 (osd.0) 143 : cluster 0 6.10 scrub starts 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.028922+0000 osd.0 (osd.0) 144 : cluster 0 6.10 scrub ok 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.327708+0000 osd.1 (osd.1) 153 : cluster 0 4.7 scrub starts 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.329635+0000 osd.1 (osd.1) 154 : cluster 0 4.7 scrub ok 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.464920+0000 osd.5 (osd.5) 135 : cluster 0 6.a scrub starts 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.467172+0000 osd.5 (osd.5) 136 : cluster 0 6.a scrub ok 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:11.654381+0000 mon.c (mon.2) 192 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.711702+0000 mon.a (mon.0) 1362 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:11.714037+0000 mon.a (mon.0) 1363 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]': finished 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:11.820422+0000 mon.c (mon.2) 193 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:11.893695+0000 mgr.y (mgr.14538) 256 : cluster 0 pgmap v236: 644 pgs: 111 active+undersized, 272 active+clean, 192 unknown, 32 creating+peering, 37 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 109/861 objects degraded (12.660%) 2024-04-02T04:50:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.165246+0000 mon.a (mon.0) 1364 : cluster 0 osdmap e173: 8 total, 7 up, 8 in 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.177239+0000 mon.a (mon.0) 1365 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.310608+0000 osd.2 (osd.2) 89 : cluster 0 2.c scrub starts 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.312140+0000 osd.2 (osd.2) 90 : cluster 0 2.c scrub ok 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.655404+0000 mon.c (mon.2) 194 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.720573+0000 mon.a (mon.0) 1366 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.725257+0000 mon.a (mon.0) 1367 : cluster 0 osdmap e174: 8 total, 7 up, 8 in 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.726937+0000 mon.c (mon.2) 195 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.728999+0000 mon.a (mon.0) 1368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.806371+0000 mon.a (mon.0) 1369 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.808518+0000 mon.a (mon.0) 1370 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:13.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[28904]: audit 2024-04-02T04:50:12.810388+0000 mon.a (mon.0) 1371 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:09.892955+0000 mgr.y (mgr.14538) 255 : cluster 0 pgmap v233: 708 pgs: 139 active+undersized, 307 active+clean, 128 unknown, 96 creating+peering, 38 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 111/870 objects degraded (12.759%) 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:10.314227+0000 osd.1 (osd.1) 151 : cluster 0 4.12 scrub starts 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:10.316402+0000 osd.1 (osd.1) 152 : cluster 0 4.12 scrub ok 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:10.506259+0000 osd.5 (osd.5) 133 : cluster 0 6.19 scrub starts 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:10.507414+0000 osd.5 (osd.5) 134 : cluster 0 6.19 scrub ok 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.027581+0000 osd.0 (osd.0) 143 : cluster 0 6.10 scrub starts 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.028922+0000 osd.0 (osd.0) 144 : cluster 0 6.10 scrub ok 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.327708+0000 osd.1 (osd.1) 153 : cluster 0 4.7 scrub starts 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.329635+0000 osd.1 (osd.1) 154 : cluster 0 4.7 scrub ok 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.464920+0000 osd.5 (osd.5) 135 : cluster 0 6.a scrub starts 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.467172+0000 osd.5 (osd.5) 136 : cluster 0 6.a scrub ok 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:11.654381+0000 mon.c (mon.2) 192 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:13.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.711702+0000 mon.a (mon.0) 1362 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:11.714037+0000 mon.a (mon.0) 1363 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]': finished 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:11.820422+0000 mon.c (mon.2) 193 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:11.893695+0000 mgr.y (mgr.14538) 256 : cluster 0 pgmap v236: 644 pgs: 111 active+undersized, 272 active+clean, 192 unknown, 32 creating+peering, 37 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 109/861 objects degraded (12.660%) 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.165246+0000 mon.a (mon.0) 1364 : cluster 0 osdmap e173: 8 total, 7 up, 8 in 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.177239+0000 mon.a (mon.0) 1365 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.310608+0000 osd.2 (osd.2) 89 : cluster 0 2.c scrub starts 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.312140+0000 osd.2 (osd.2) 90 : cluster 0 2.c scrub ok 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.655404+0000 mon.c (mon.2) 194 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.720573+0000 mon.a (mon.0) 1366 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.725257+0000 mon.a (mon.0) 1367 : cluster 0 osdmap e174: 8 total, 7 up, 8 in 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.726937+0000 mon.c (mon.2) 195 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.728999+0000 mon.a (mon.0) 1368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.806371+0000 mon.a (mon.0) 1369 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:13.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.808518+0000 mon.a (mon.0) 1370 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:13.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:13 smithi069 bash[21528]: audit 2024-04-02T04:50:12.810388+0000 mon.a (mon.0) 1371 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:13.783 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:13.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:13.899 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:09.892955+0000 mgr.y (mgr.14538) 255 : cluster 0 pgmap v233: 708 pgs: 139 active+undersized, 307 active+clean, 128 unknown, 96 creating+peering, 38 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 111/870 objects degraded (12.759%) 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:10.314227+0000 osd.1 (osd.1) 151 : cluster 0 4.12 scrub starts 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:10.316402+0000 osd.1 (osd.1) 152 : cluster 0 4.12 scrub ok 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:10.506259+0000 osd.5 (osd.5) 133 : cluster 0 6.19 scrub starts 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:10.507414+0000 osd.5 (osd.5) 134 : cluster 0 6.19 scrub ok 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.027581+0000 osd.0 (osd.0) 143 : cluster 0 6.10 scrub starts 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.028922+0000 osd.0 (osd.0) 144 : cluster 0 6.10 scrub ok 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.327708+0000 osd.1 (osd.1) 153 : cluster 0 4.7 scrub starts 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.329635+0000 osd.1 (osd.1) 154 : cluster 0 4.7 scrub ok 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.464920+0000 osd.5 (osd.5) 135 : cluster 0 6.a scrub starts 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.467172+0000 osd.5 (osd.5) 136 : cluster 0 6.a scrub ok 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:11.654381+0000 mon.c (mon.2) 192 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.711702+0000 mon.a (mon.0) 1362 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:11.714037+0000 mon.a (mon.0) 1363 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-9", "mode": "writeback"}]': finished 2024-04-02T04:50:13.900 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:11.820422+0000 mon.c (mon.2) 193 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:11.893695+0000 mgr.y (mgr.14538) 256 : cluster 0 pgmap v236: 644 pgs: 111 active+undersized, 272 active+clean, 192 unknown, 32 creating+peering, 37 active+undersized+degraded; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 109/861 objects degraded (12.660%) 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.165246+0000 mon.a (mon.0) 1364 : cluster 0 osdmap e173: 8 total, 7 up, 8 in 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.177239+0000 mon.a (mon.0) 1365 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.310608+0000 osd.2 (osd.2) 89 : cluster 0 2.c scrub starts 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.312140+0000 osd.2 (osd.2) 90 : cluster 0 2.c scrub ok 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.655404+0000 mon.c (mon.2) 194 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.720573+0000 mon.a (mon.0) 1366 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.725257+0000 mon.a (mon.0) 1367 : cluster 0 osdmap e174: 8 total, 7 up, 8 in 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.726937+0000 mon.c (mon.2) 195 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.728999+0000 mon.a (mon.0) 1368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.806371+0000 mon.a (mon.0) 1369 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.808518+0000 mon.a (mon.0) 1370 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:13.901 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:13 smithi098 bash[23790]: audit 2024-04-02T04:50:12.810388+0000 mon.a (mon.0) 1371 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:13.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:13.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:14.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:14.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:14.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:14.164 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:14.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:14.289 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:14.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:14.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:14.546 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:14.649 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:14.674 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:14.776 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:14.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/exp.66.c.7 2024-04-02T04:50:14.797 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:14.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:14.922 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:14.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:15.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:15.046 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:15.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:15.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.542447+0000 osd.4 (osd.4) 149 : cluster 0 3.3 scrub starts 2024-04-02T04:50:15.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.546676+0000 osd.4 (osd.4) 150 : cluster 0 3.3 scrub ok 2024-04-02T04:50:15.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.839369+0000 osd.6 (osd.6) 109 : cluster 0 4.1f scrub starts 2024-04-02T04:50:15.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:12.840780+0000 osd.6 (osd.6) 110 : cluster 0 4.1f scrub ok 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:13.027496+0000 mgr.y (mgr.14538) 257 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.056287+0000 osd.0 (osd.0) 145 : cluster 0 4.6 scrub starts 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.057650+0000 osd.0 (osd.0) 146 : cluster 0 4.6 scrub ok 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.286799+0000 osd.2 (osd.2) 91 : cluster 0 6.f scrub starts 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.288131+0000 osd.2 (osd.2) 92 : cluster 0 6.f scrub ok 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.549293+0000 osd.5 (osd.5) 137 : cluster 0 3.1d scrub starts 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.553791+0000 osd.5 (osd.5) 138 : cluster 0 3.1d scrub ok 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:13.656331+0000 mon.c (mon.2) 196 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.662360+0000 mon.a (mon.0) 1372 : cluster 3 Health check update: Degraded data redundancy: 111/870 objects degraded (12.759%), 38 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.663793+0000 mon.a (mon.0) 1373 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:13.669716+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]': finished 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:13.669793+0000 mon.a (mon.0) 1375 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:13.696697+0000 mon.b (mon.1) 142 : audit 1 from='client.? 172.21.15.69:0/769728975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.111561+0000 mon.a (mon.0) 1376 : cluster 0 osdmap e175: 8 total, 7 up, 8 in 2024-04-02T04:50:15.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:14.113088+0000 mon.a (mon.0) 1377 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-72378-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:14.114517+0000 mon.a (mon.0) 1378 : audit 1 from='client.? 172.21.15.69:0/3019856671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-72153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:14.115687+0000 mon.a (mon.0) 1379 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.220098+0000 osd.3 (osd.3) 171 : cluster 0 3.18 scrub starts 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.221679+0000 osd.3 (osd.3) 172 : cluster 0 3.18 scrub ok 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.532359+0000 osd.4 (osd.4) 151 : cluster 0 3.1a scrub starts 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.537013+0000 osd.4 (osd.4) 152 : cluster 0 3.1a scrub ok 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[21528]: audit 2024-04-02T04:50:14.657448+0000 mon.c (mon.2) 197 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.542447+0000 osd.4 (osd.4) 149 : cluster 0 3.3 scrub starts 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.546676+0000 osd.4 (osd.4) 150 : cluster 0 3.3 scrub ok 2024-04-02T04:50:15.173 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.839369+0000 osd.6 (osd.6) 109 : cluster 0 4.1f scrub starts 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:12.840780+0000 osd.6 (osd.6) 110 : cluster 0 4.1f scrub ok 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:13.027496+0000 mgr.y (mgr.14538) 257 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.056287+0000 osd.0 (osd.0) 145 : cluster 0 4.6 scrub starts 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.057650+0000 osd.0 (osd.0) 146 : cluster 0 4.6 scrub ok 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.286799+0000 osd.2 (osd.2) 91 : cluster 0 6.f scrub starts 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.288131+0000 osd.2 (osd.2) 92 : cluster 0 6.f scrub ok 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.549293+0000 osd.5 (osd.5) 137 : cluster 0 3.1d scrub starts 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.553791+0000 osd.5 (osd.5) 138 : cluster 0 3.1d scrub ok 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:13.656331+0000 mon.c (mon.2) 196 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.662360+0000 mon.a (mon.0) 1372 : cluster 3 Health check update: Degraded data redundancy: 111/870 objects degraded (12.759%), 38 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.663793+0000 mon.a (mon.0) 1373 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:13.669716+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]': finished 2024-04-02T04:50:15.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:13.669793+0000 mon.a (mon.0) 1375 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:13.696697+0000 mon.b (mon.1) 142 : audit 1 from='client.? 172.21.15.69:0/769728975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.111561+0000 mon.a (mon.0) 1376 : cluster 0 osdmap e175: 8 total, 7 up, 8 in 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:14.113088+0000 mon.a (mon.0) 1377 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-72378-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:14.114517+0000 mon.a (mon.0) 1378 : audit 1 from='client.? 172.21.15.69:0/3019856671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-72153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:14.115687+0000 mon.a (mon.0) 1379 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.220098+0000 osd.3 (osd.3) 171 : cluster 0 3.18 scrub starts 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.221679+0000 osd.3 (osd.3) 172 : cluster 0 3.18 scrub ok 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.532359+0000 osd.4 (osd.4) 151 : cluster 0 3.1a scrub starts 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.537013+0000 osd.4 (osd.4) 152 : cluster 0 3.1a scrub ok 2024-04-02T04:50:15.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:15 smithi069 bash[28904]: audit 2024-04-02T04:50:14.657448+0000 mon.c (mon.2) 197 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:15.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:15.292 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:15.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:15.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:15.407 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.542447+0000 osd.4 (osd.4) 149 : cluster 0 3.3 scrub starts 2024-04-02T04:50:15.407 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.546676+0000 osd.4 (osd.4) 150 : cluster 0 3.3 scrub ok 2024-04-02T04:50:15.407 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.839369+0000 osd.6 (osd.6) 109 : cluster 0 4.1f scrub starts 2024-04-02T04:50:15.407 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:12.840780+0000 osd.6 (osd.6) 110 : cluster 0 4.1f scrub ok 2024-04-02T04:50:15.407 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:13.027496+0000 mgr.y (mgr.14538) 257 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:15.407 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.056287+0000 osd.0 (osd.0) 145 : cluster 0 4.6 scrub starts 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.057650+0000 osd.0 (osd.0) 146 : cluster 0 4.6 scrub ok 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.286799+0000 osd.2 (osd.2) 91 : cluster 0 6.f scrub starts 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.288131+0000 osd.2 (osd.2) 92 : cluster 0 6.f scrub ok 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.549293+0000 osd.5 (osd.5) 137 : cluster 0 3.1d scrub starts 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.553791+0000 osd.5 (osd.5) 138 : cluster 0 3.1d scrub ok 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:13.656331+0000 mon.c (mon.2) 196 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.662360+0000 mon.a (mon.0) 1372 : cluster 3 Health check update: Degraded data redundancy: 111/870 objects degraded (12.759%), 38 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.663793+0000 mon.a (mon.0) 1373 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:13.669716+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-9"}]': finished 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:13.669793+0000 mon.a (mon.0) 1375 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:15.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:13.696697+0000 mon.b (mon.1) 142 : audit 1 from='client.? 172.21.15.69:0/769728975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.111561+0000 mon.a (mon.0) 1376 : cluster 0 osdmap e175: 8 total, 7 up, 8 in 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:14.113088+0000 mon.a (mon.0) 1377 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-72378-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:14.114517+0000 mon.a (mon.0) 1378 : audit 1 from='client.? 172.21.15.69:0/3019856671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-72153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:14.115687+0000 mon.a (mon.0) 1379 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.220098+0000 osd.3 (osd.3) 171 : cluster 0 3.18 scrub starts 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.221679+0000 osd.3 (osd.3) 172 : cluster 0 3.18 scrub ok 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.532359+0000 osd.4 (osd.4) 151 : cluster 0 3.1a scrub starts 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.537013+0000 osd.4 (osd.4) 152 : cluster 0 3.1a scrub ok 2024-04-02T04:50:15.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:15 smithi098 bash[23790]: audit 2024-04-02T04:50:14.657448+0000 mon.c (mon.2) 197 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:15.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:15.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:15.657 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:15.677 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:15.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:15.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:15.911 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:15.935 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:15.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:16.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:16.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:16.214 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:16.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:16.338 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:16.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:16.469 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:16.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:16.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:16.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:16.723 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:16.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:16.853 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:16.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:13.895144+0000 mgr.y (mgr.14538) 258 : cluster 0 pgmap v239: 580 pgs: 13 creating+activating, 136 active+undersized, 298 active+clean, 32 creating+peering, 64 unknown, 37 active+undersized+degraded; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s; 109/861 objects degraded (12.660%) 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.306423+0000 osd.2 (osd.2) 93 : cluster 0 2.1 scrub starts 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.307474+0000 osd.2 (osd.2) 94 : cluster 0 2.1 scrub ok 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:14.673061+0000 mon.a (mon.0) 1380 : cluster 3 Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:14.674574+0000 mon.a (mon.0) 1381 : audit 1 from='client.? 172.21.15.69:0/3019856671' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-72153-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:14.674691+0000 mon.a (mon.0) 1382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:16.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:14.687206+0000 mon.b (mon.1) 143 : audit 1 from='client.? 172.21.15.69:0/1995744562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.135297+0000 mon.a (mon.0) 1383 : cluster 0 osdmap e176: 8 total, 7 up, 8 in 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:15.137719+0000 mon.a (mon.0) 1384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.297527+0000 osd.1 (osd.1) 155 : cluster 0 6.1d scrub starts 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.299309+0000 osd.1 (osd.1) 156 : cluster 0 6.1d scrub ok 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.513495+0000 osd.4 (osd.4) 153 : cluster 0 4.d scrub starts 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.516528+0000 osd.4 (osd.4) 154 : cluster 0 4.d scrub ok 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:15.659015+0000 mon.c (mon.2) 198 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:16.019161+0000 mon.a (mon.0) 1385 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-72378-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[28904]: audit 2024-04-02T04:50:16.019252+0000 mon.a (mon.0) 1386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:13.895144+0000 mgr.y (mgr.14538) 258 : cluster 0 pgmap v239: 580 pgs: 13 creating+activating, 136 active+undersized, 298 active+clean, 32 creating+peering, 64 unknown, 37 active+undersized+degraded; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s; 109/861 objects degraded (12.660%) 2024-04-02T04:50:16.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.306423+0000 osd.2 (osd.2) 93 : cluster 0 2.1 scrub starts 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.307474+0000 osd.2 (osd.2) 94 : cluster 0 2.1 scrub ok 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:14.673061+0000 mon.a (mon.0) 1380 : cluster 3 Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:14.674574+0000 mon.a (mon.0) 1381 : audit 1 from='client.? 172.21.15.69:0/3019856671' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-72153-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:14.674691+0000 mon.a (mon.0) 1382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:14.687206+0000 mon.b (mon.1) 143 : audit 1 from='client.? 172.21.15.69:0/1995744562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.135297+0000 mon.a (mon.0) 1383 : cluster 0 osdmap e176: 8 total, 7 up, 8 in 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:15.137719+0000 mon.a (mon.0) 1384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.297527+0000 osd.1 (osd.1) 155 : cluster 0 6.1d scrub starts 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.299309+0000 osd.1 (osd.1) 156 : cluster 0 6.1d scrub ok 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.513495+0000 osd.4 (osd.4) 153 : cluster 0 4.d scrub starts 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.516528+0000 osd.4 (osd.4) 154 : cluster 0 4.d scrub ok 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:15.659015+0000 mon.c (mon.2) 198 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:16.019161+0000 mon.a (mon.0) 1385 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-72378-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:16.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:16 smithi069 bash[21528]: audit 2024-04-02T04:50:16.019252+0000 mon.a (mon.0) 1386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:16.974 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:16.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:17.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:17.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:13.895144+0000 mgr.y (mgr.14538) 258 : cluster 0 pgmap v239: 580 pgs: 13 creating+activating, 136 active+undersized, 298 active+clean, 32 creating+peering, 64 unknown, 37 active+undersized+degraded; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s; 109/861 objects degraded (12.660%) 2024-04-02T04:50:17.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.306423+0000 osd.2 (osd.2) 93 : cluster 0 2.1 scrub starts 2024-04-02T04:50:17.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.307474+0000 osd.2 (osd.2) 94 : cluster 0 2.1 scrub ok 2024-04-02T04:50:17.088 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:14.673061+0000 mon.a (mon.0) 1380 : cluster 3 Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:14.674574+0000 mon.a (mon.0) 1381 : audit 1 from='client.? 172.21.15.69:0/3019856671' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-72153-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:14.674691+0000 mon.a (mon.0) 1382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-72145-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:14.687206+0000 mon.b (mon.1) 143 : audit 1 from='client.? 172.21.15.69:0/1995744562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.135297+0000 mon.a (mon.0) 1383 : cluster 0 osdmap e176: 8 total, 7 up, 8 in 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:15.137719+0000 mon.a (mon.0) 1384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.297527+0000 osd.1 (osd.1) 155 : cluster 0 6.1d scrub starts 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.299309+0000 osd.1 (osd.1) 156 : cluster 0 6.1d scrub ok 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.513495+0000 osd.4 (osd.4) 153 : cluster 0 4.d scrub starts 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.516528+0000 osd.4 (osd.4) 154 : cluster 0 4.d scrub ok 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:15.659015+0000 mon.c (mon.2) 198 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:16.019161+0000 mon.a (mon.0) 1385 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-72378-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:17.089 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:16 smithi098 bash[23790]: audit 2024-04-02T04:50:16.019252+0000 mon.a (mon.0) 1386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:17.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:17.230 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:17.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:17.368 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:17.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:17.496 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:17.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:17.623 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:17.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:17.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:17.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:17.876 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:17.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:17.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:18.000 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:18.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:18.130 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:18.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:18.254 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:18.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: cluster 2024-04-02T04:50:16.024084+0000 mon.a (mon.0) 1387 : cluster 0 osdmap e177: 8 total, 7 up, 8 in 2024-04-02T04:50:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:16.045588+0000 mon.c (mon.2) 199 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:16.477005+0000 mon.b (mon.1) 144 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:16.477802+0000 mon.a (mon.0) 1388 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:16.660374+0000 mon.c (mon.2) 200 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:16.918374+0000 mon.a (mon.0) 1389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:17.354251+0000 mon.a (mon.0) 1390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: audit 2024-04-02T04:50:17.661886+0000 mon.c (mon.2) 201 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:18 smithi098 bash[23790]: cluster 2024-04-02T04:50:17.805498+0000 mon.a (mon.0) 1391 : cluster 0 osdmap e178: 8 total, 7 up, 8 in 2024-04-02T04:50:18.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:18.525 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: cluster 2024-04-02T04:50:16.024084+0000 mon.a (mon.0) 1387 : cluster 0 osdmap e177: 8 total, 7 up, 8 in 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:16.045588+0000 mon.c (mon.2) 199 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:16.477005+0000 mon.b (mon.1) 144 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:16.477802+0000 mon.a (mon.0) 1388 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:16.660374+0000 mon.c (mon.2) 200 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:16.918374+0000 mon.a (mon.0) 1389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:17.354251+0000 mon.a (mon.0) 1390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: audit 2024-04-02T04:50:17.661886+0000 mon.c (mon.2) 201 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21528]: cluster 2024-04-02T04:50:17.805498+0000 mon.a (mon.0) 1391 : cluster 0 osdmap e178: 8 total, 7 up, 8 in 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: cluster 2024-04-02T04:50:16.024084+0000 mon.a (mon.0) 1387 : cluster 0 osdmap e177: 8 total, 7 up, 8 in 2024-04-02T04:50:18.526 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:16.045588+0000 mon.c (mon.2) 199 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:16.477005+0000 mon.b (mon.1) 144 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:16.477802+0000 mon.a (mon.0) 1388 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:16.660374+0000 mon.c (mon.2) 200 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:16.918374+0000 mon.a (mon.0) 1389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:17.354251+0000 mon.a (mon.0) 1390 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: audit 2024-04-02T04:50:17.661886+0000 mon.c (mon.2) 201 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:18.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[28904]: cluster 2024-04-02T04:50:17.805498+0000 mon.a (mon.0) 1391 : cluster 0 osdmap e178: 8 total, 7 up, 8 in 2024-04-02T04:50:18.536 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:18.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:18.662 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:18.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:18.791 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:18.815 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:50:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:50:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:50:18.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:18.920 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:18.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:19.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:19.044 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:19.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:19.128 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-02T04:50:19.128 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment set-up. 2024-04-02T04:50:19.128 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-02T04:50:19.128 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-04-02T04:49:10.926+0000 7f5600ba1880 -1 failed for service _ceph-mon._tcp 2024-04-02T04:50:19.129 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-04-02T04:49:10.926+0000 7f5600ba1880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (895 ms) 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2904 ms total) 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99f970300 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fabfdc0 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fac07b0 2024-04-02T04:50:19.130 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fac0db0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fac13b0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fac19f0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fac1540 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca81f0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca86c0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca87d0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fac12a0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca88e0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca89f0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8fb0 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8b00 2024-04-02T04:50:19.131 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8c80 2024-04-02T04:50:19.132 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8e00 2024-04-02T04:50:19.132 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca90c0 2024-04-02T04:50:19.132 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9220 2024-04-02T04:50:19.132 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9bf0 2024-04-02T04:50:19.132 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9800 2024-04-02T04:50:19.133 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9950 2024-04-02T04:50:19.133 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9330 2024-04-02T04:50:19.133 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8320 2024-04-02T04:50:19.133 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8430 2024-04-02T04:50:19.133 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca8540 2024-04-02T04:50:19.133 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa1b0 2024-04-02T04:50:19.134 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9d00 2024-04-02T04:50:19.134 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9440 2024-04-02T04:50:19.134 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9550 2024-04-02T04:50:19.134 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9660 2024-04-02T04:50:19.134 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa770 2024-04-02T04:50:19.134 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa2c0 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9e10 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9f20 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa030 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaad30 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa880 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa3d0 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fca9ab0 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa4e0 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa5f0 2024-04-02T04:50:19.135 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaae40 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaaf50 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab510 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab060 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaa990 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaaaa0 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcaabb0 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcabad0 2024-04-02T04:50:19.136 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab620 2024-04-02T04:50:19.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:19.173 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:19.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:19.299 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:19.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:19.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: starte misc: Running main() from gmock_main.cc 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] Global test environment set-up. 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Version (16777 ms) 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4348 ms) 2024-04-02T04:50:19.452 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongName (7244 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (10920 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (4182 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4576 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Exec (4812 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (4178 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3012 ms) 2024-04-02T04:50:19.453 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3717 ms) 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1917 ms) 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3413 ms) 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] 12 tests from NeoRadosMisc (69097 ms total) 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] Global test environment tear-down 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [==========] 12 tests from 1 test suite ran. (69097 ms total) 2024-04-02T04:50:19.454 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ PASSED ] 12 tests. 2024-04-02T04:50:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.548739+0000 osd.5 (osd.5) 139 : cluster 0 2.2 scrub starts 2024-04-02T04:50:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.550128+0000 osd.5 (osd.5) 140 : cluster 0 2.2 scrub ok 2024-04-02T04:50:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:15.896588+0000 mgr.y (mgr.14538) 259 : cluster 0 pgmap v241: 548 pgs: 10 creating+activating, 110 active+undersized, 263 active+clean, 128 unknown, 37 active+undersized+degraded; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 735 B/s wr, 2 op/s; 108/858 objects degraded (12.587%) 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:16.316691+0000 osd.1 (osd.1) 157 : cluster 0 3.0 scrub starts 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:16.319552+0000 osd.1 (osd.1) 158 : cluster 0 3.0 scrub ok 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:17.226083+0000 osd.3 (osd.3) 173 : cluster 0 3.b scrub starts 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:17.228646+0000 osd.3 (osd.3) 174 : cluster 0 3.b scrub ok 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:17.743833+0000 osd.6 (osd.6) 111 : cluster 0 3.10 scrub starts 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:17.746573+0000 osd.6 (osd.6) 112 : cluster 0 3.10 scrub ok 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:17.898789+0000 mgr.y (mgr.14538) 260 : cluster 0 pgmap v244: 620 pgs: 2 active+clean+snaptrim, 150 active+undersized, 311 active+clean, 120 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.8 KiB/s wr, 5 op/s; 110/876 objects degraded (12.557%) 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:18.187936+0000 osd.2 (osd.2) 95 : cluster 0 15.b scrub starts 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:18.189962+0000 osd.2 (osd.2) 96 : cluster 0 15.b scrub ok 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:18.372413+0000 mon.a (mon.0) 1392 : cluster 0 osdmap e179: 8 total, 7 up, 8 in 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: audit 2024-04-02T04:50:18.393725+0000 mon.a (mon.0) 1393 : audit 1 from='client.? 172.21.15.69:0/2758212486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-72145-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: audit 2024-04-02T04:50:18.394192+0000 mon.a (mon.0) 1394 : audit 1 from='client.? 172.21.15.69:0/2631156749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-72153-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: audit 2024-04-02T04:50:18.477317+0000 mon.c (mon.2) 202 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:18.500523+0000 osd.4 (osd.4) 155 : cluster 0 6.e scrub starts 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:18.501523+0000 osd.4 (osd.4) 156 : cluster 0 6.e scrub ok 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: audit 2024-04-02T04:50:18.663026+0000 mon.c (mon.2) 203 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: cluster 2024-04-02T04:50:18.863788+0000 mon.a (mon.0) 1395 : cluster 3 Health check update: Degraded data redundancy: 108/858 objects degraded (12.587%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[21528]: audit 2024-04-02T04:50:18.969968+0000 mon.a (mon.0) 1396 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.548739+0000 osd.5 (osd.5) 139 : cluster 0 2.2 scrub starts 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.550128+0000 osd.5 (osd.5) 140 : cluster 0 2.2 scrub ok 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:15.896588+0000 mgr.y (mgr.14538) 259 : cluster 0 pgmap v241: 548 pgs: 10 creating+activating, 110 active+undersized, 263 active+clean, 128 unknown, 37 active+undersized+degraded; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 735 B/s wr, 2 op/s; 108/858 objects degraded (12.587%) 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:16.316691+0000 osd.1 (osd.1) 157 : cluster 0 3.0 scrub starts 2024-04-02T04:50:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:16.319552+0000 osd.1 (osd.1) 158 : cluster 0 3.0 scrub ok 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:17.226083+0000 osd.3 (osd.3) 173 : cluster 0 3.b scrub starts 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:17.228646+0000 osd.3 (osd.3) 174 : cluster 0 3.b scrub ok 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:17.743833+0000 osd.6 (osd.6) 111 : cluster 0 3.10 scrub starts 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:17.746573+0000 osd.6 (osd.6) 112 : cluster 0 3.10 scrub ok 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:17.898789+0000 mgr.y (mgr.14538) 260 : cluster 0 pgmap v244: 620 pgs: 2 active+clean+snaptrim, 150 active+undersized, 311 active+clean, 120 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.8 KiB/s wr, 5 op/s; 110/876 objects degraded (12.557%) 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:18.187936+0000 osd.2 (osd.2) 95 : cluster 0 15.b scrub starts 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:18.189962+0000 osd.2 (osd.2) 96 : cluster 0 15.b scrub ok 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:18.372413+0000 mon.a (mon.0) 1392 : cluster 0 osdmap e179: 8 total, 7 up, 8 in 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: audit 2024-04-02T04:50:18.393725+0000 mon.a (mon.0) 1393 : audit 1 from='client.? 172.21.15.69:0/2758212486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-72145-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: audit 2024-04-02T04:50:18.394192+0000 mon.a (mon.0) 1394 : audit 1 from='client.? 172.21.15.69:0/2631156749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-72153-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:19.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: audit 2024-04-02T04:50:18.477317+0000 mon.c (mon.2) 202 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:18.500523+0000 osd.4 (osd.4) 155 : cluster 0 6.e scrub starts 2024-04-02T04:50:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:18.501523+0000 osd.4 (osd.4) 156 : cluster 0 6.e scrub ok 2024-04-02T04:50:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: audit 2024-04-02T04:50:18.663026+0000 mon.c (mon.2) 203 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: cluster 2024-04-02T04:50:18.863788+0000 mon.a (mon.0) 1395 : cluster 3 Health check update: Degraded data redundancy: 108/858 objects degraded (12.587%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:19 smithi069 bash[28904]: audit 2024-04-02T04:50:18.969968+0000 mon.a (mon.0) 1396 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:19.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:19.561 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:19.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.548739+0000 osd.5 (osd.5) 139 : cluster 0 2.2 scrub starts 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.550128+0000 osd.5 (osd.5) 140 : cluster 0 2.2 scrub ok 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:15.896588+0000 mgr.y (mgr.14538) 259 : cluster 0 pgmap v241: 548 pgs: 10 creating+activating, 110 active+undersized, 263 active+clean, 128 unknown, 37 active+undersized+degraded; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 735 B/s wr, 2 op/s; 108/858 objects degraded (12.587%) 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:16.316691+0000 osd.1 (osd.1) 157 : cluster 0 3.0 scrub starts 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:16.319552+0000 osd.1 (osd.1) 158 : cluster 0 3.0 scrub ok 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:17.226083+0000 osd.3 (osd.3) 173 : cluster 0 3.b scrub starts 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:17.228646+0000 osd.3 (osd.3) 174 : cluster 0 3.b scrub ok 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:17.743833+0000 osd.6 (osd.6) 111 : cluster 0 3.10 scrub starts 2024-04-02T04:50:19.675 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:17.746573+0000 osd.6 (osd.6) 112 : cluster 0 3.10 scrub ok 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:17.898789+0000 mgr.y (mgr.14538) 260 : cluster 0 pgmap v244: 620 pgs: 2 active+clean+snaptrim, 150 active+undersized, 311 active+clean, 120 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.8 KiB/s wr, 5 op/s; 110/876 objects degraded (12.557%) 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:18.187936+0000 osd.2 (osd.2) 95 : cluster 0 15.b scrub starts 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:18.189962+0000 osd.2 (osd.2) 96 : cluster 0 15.b scrub ok 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:18.372413+0000 mon.a (mon.0) 1392 : cluster 0 osdmap e179: 8 total, 7 up, 8 in 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: audit 2024-04-02T04:50:18.393725+0000 mon.a (mon.0) 1393 : audit 1 from='client.? 172.21.15.69:0/2758212486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-72145-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: audit 2024-04-02T04:50:18.394192+0000 mon.a (mon.0) 1394 : audit 1 from='client.? 172.21.15.69:0/2631156749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-72153-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: audit 2024-04-02T04:50:18.477317+0000 mon.c (mon.2) 202 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:18.500523+0000 osd.4 (osd.4) 155 : cluster 0 6.e scrub starts 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:18.501523+0000 osd.4 (osd.4) 156 : cluster 0 6.e scrub ok 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: audit 2024-04-02T04:50:18.663026+0000 mon.c (mon.2) 203 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: cluster 2024-04-02T04:50:18.863788+0000 mon.a (mon.0) 1395 : cluster 3 Health check update: Degraded data redundancy: 108/858 objects degraded (12.587%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:19.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:19 smithi098 bash[23790]: audit 2024-04-02T04:50:18.969968+0000 mon.a (mon.0) 1396 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:19.689 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:19.792 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:19.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-04-02T04:50:19.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:50:19.801 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-02T04:50:19.801 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-02T04:50:19.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:50:19.814 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:19.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:19.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:19.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:20.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:20.066 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:20.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:20.190 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:20.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:20.318 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:20.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.461 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:20.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:20.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:20.581 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:20.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:20.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.052670+0000 osd.0 (osd.0) 147 : cluster 0 3.1 scrub starts 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.055246+0000 osd.0 (osd.0) 148 : cluster 0 3.1 scrub ok 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.173573+0000 osd.2 (osd.2) 97 : cluster 0 15.a scrub starts 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.176457+0000 osd.2 (osd.2) 98 : cluster 0 15.a scrub ok 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.339502+0000 osd.1 (osd.1) 159 : cluster 0 2.15 scrub starts 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.340949+0000 osd.1 (osd.1) 160 : cluster 0 2.15 scrub ok 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: audit 2024-04-02T04:50:19.419045+0000 mon.a (mon.0) 1397 : audit 1 from='client.? 172.21.15.69:0/2758212486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-72145-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: audit 2024-04-02T04:50:19.419216+0000 mon.a (mon.0) 1398 : audit 1 from='client.? 172.21.15.69:0/2631156749' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-72153-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: audit 2024-04-02T04:50:19.419386+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.428551+0000 mon.a (mon.0) 1400 : cluster 0 osdmap e180: 8 total, 7 up, 8 in 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: audit 2024-04-02T04:50:19.431279+0000 mon.c (mon.2) 204 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: audit 2024-04-02T04:50:19.444042+0000 mon.a (mon.0) 1401 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:20.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:20 smithi098 bash[23790]: audit 2024-04-02T04:50:19.664254+0000 mon.c (mon.2) 205 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.052670+0000 osd.0 (osd.0) 147 : cluster 0 3.1 scrub starts 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.055246+0000 osd.0 (osd.0) 148 : cluster 0 3.1 scrub ok 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.173573+0000 osd.2 (osd.2) 97 : cluster 0 15.a scrub starts 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.176457+0000 osd.2 (osd.2) 98 : cluster 0 15.a scrub ok 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.339502+0000 osd.1 (osd.1) 159 : cluster 0 2.15 scrub starts 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.340949+0000 osd.1 (osd.1) 160 : cluster 0 2.15 scrub ok 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: audit 2024-04-02T04:50:19.419045+0000 mon.a (mon.0) 1397 : audit 1 from='client.? 172.21.15.69:0/2758212486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-72145-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: audit 2024-04-02T04:50:19.419216+0000 mon.a (mon.0) 1398 : audit 1 from='client.? 172.21.15.69:0/2631156749' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-72153-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: audit 2024-04-02T04:50:19.419386+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.428551+0000 mon.a (mon.0) 1400 : cluster 0 osdmap e180: 8 total, 7 up, 8 in 2024-04-02T04:50:20.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: audit 2024-04-02T04:50:19.431279+0000 mon.c (mon.2) 204 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: audit 2024-04-02T04:50:19.444042+0000 mon.a (mon.0) 1401 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[21528]: audit 2024-04-02T04:50:19.664254+0000 mon.c (mon.2) 205 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.052670+0000 osd.0 (osd.0) 147 : cluster 0 3.1 scrub starts 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.055246+0000 osd.0 (osd.0) 148 : cluster 0 3.1 scrub ok 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.173573+0000 osd.2 (osd.2) 97 : cluster 0 15.a scrub starts 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.176457+0000 osd.2 (osd.2) 98 : cluster 0 15.a scrub ok 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.339502+0000 osd.1 (osd.1) 159 : cluster 0 2.15 scrub starts 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.340949+0000 osd.1 (osd.1) 160 : cluster 0 2.15 scrub ok 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: audit 2024-04-02T04:50:19.419045+0000 mon.a (mon.0) 1397 : audit 1 from='client.? 172.21.15.69:0/2758212486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-72145-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: audit 2024-04-02T04:50:19.419216+0000 mon.a (mon.0) 1398 : audit 1 from='client.? 172.21.15.69:0/2631156749' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-72153-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:20.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: audit 2024-04-02T04:50:19.419386+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:20.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.428551+0000 mon.a (mon.0) 1400 : cluster 0 osdmap e180: 8 total, 7 up, 8 in 2024-04-02T04:50:20.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: audit 2024-04-02T04:50:19.431279+0000 mon.c (mon.2) 204 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:20.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: audit 2024-04-02T04:50:19.444042+0000 mon.a (mon.0) 1401 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:20.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:20 smithi069 bash[28904]: audit 2024-04-02T04:50:19.664254+0000 mon.c (mon.2) 205 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:20.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:20.835 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:20.938 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:20.959 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:20.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:21.062 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:21.085 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:21.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:21.206 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:21.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:21.336 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:21.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:21.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:21.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:21.751 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:21.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:21.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:21.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:19.899739+0000 mgr.y (mgr.14538) 261 : cluster 0 pgmap v247: 556 pgs: 1 active+clean+snaptrim, 113 active+undersized, 261 active+clean, 128 unknown, 16 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s; 110/876 objects degraded (12.557%) 2024-04-02T04:50:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.170483+0000 osd.2 (osd.2) 99 : cluster 0 2.10 scrub starts 2024-04-02T04:50:22.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.172621+0000 osd.2 (osd.2) 100 : cluster 0 2.10 scrub ok 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.217079+0000 osd.3 (osd.3) 175 : cluster 0 4.19 scrub starts 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.217925+0000 osd.3 (osd.3) 176 : cluster 0 4.19 scrub ok 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.301565+0000 osd.1 (osd.1) 161 : cluster 0 3.4 scrub starts 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.307273+0000 osd.1 (osd.1) 162 : cluster 0 3.4 scrub ok 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.418977+0000 mon.a (mon.0) 1402 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: audit 2024-04-02T04:50:20.429677+0000 mon.a (mon.0) 1403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]': finished 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: audit 2024-04-02T04:50:20.485814+0000 mon.b (mon.1) 145 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: cluster 2024-04-02T04:50:20.486205+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e181: 8 total, 7 up, 8 in 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: audit 2024-04-02T04:50:20.490650+0000 mon.c (mon.2) 206 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]: dispatch 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: audit 2024-04-02T04:50:20.493348+0000 mon.a (mon.0) 1405 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: audit 2024-04-02T04:50:20.495358+0000 mon.a (mon.0) 1406 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]: dispatch 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[21528]: audit 2024-04-02T04:50:20.665707+0000 mon.c (mon.2) 207 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:22.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:19.899739+0000 mgr.y (mgr.14538) 261 : cluster 0 pgmap v247: 556 pgs: 1 active+clean+snaptrim, 113 active+undersized, 261 active+clean, 128 unknown, 16 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s; 110/876 objects degraded (12.557%) 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.170483+0000 osd.2 (osd.2) 99 : cluster 0 2.10 scrub starts 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.172621+0000 osd.2 (osd.2) 100 : cluster 0 2.10 scrub ok 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.217079+0000 osd.3 (osd.3) 175 : cluster 0 4.19 scrub starts 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.217925+0000 osd.3 (osd.3) 176 : cluster 0 4.19 scrub ok 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.301565+0000 osd.1 (osd.1) 161 : cluster 0 3.4 scrub starts 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.307273+0000 osd.1 (osd.1) 162 : cluster 0 3.4 scrub ok 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.418977+0000 mon.a (mon.0) 1402 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: audit 2024-04-02T04:50:20.429677+0000 mon.a (mon.0) 1403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]': finished 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: audit 2024-04-02T04:50:20.485814+0000 mon.b (mon.1) 145 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: cluster 2024-04-02T04:50:20.486205+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e181: 8 total, 7 up, 8 in 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: audit 2024-04-02T04:50:20.490650+0000 mon.c (mon.2) 206 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]: dispatch 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: audit 2024-04-02T04:50:20.493348+0000 mon.a (mon.0) 1405 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: audit 2024-04-02T04:50:20.495358+0000 mon.a (mon.0) 1406 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]: dispatch 2024-04-02T04:50:22.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:21 smithi069 bash[28904]: audit 2024-04-02T04:50:20.665707+0000 mon.c (mon.2) 207 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:22.114 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:22.138 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:19.899739+0000 mgr.y (mgr.14538) 261 : cluster 0 pgmap v247: 556 pgs: 1 active+clean+snaptrim, 113 active+undersized, 261 active+clean, 128 unknown, 16 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s; 110/876 objects degraded (12.557%) 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.170483+0000 osd.2 (osd.2) 99 : cluster 0 2.10 scrub starts 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.172621+0000 osd.2 (osd.2) 100 : cluster 0 2.10 scrub ok 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.217079+0000 osd.3 (osd.3) 175 : cluster 0 4.19 scrub starts 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.217925+0000 osd.3 (osd.3) 176 : cluster 0 4.19 scrub ok 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.301565+0000 osd.1 (osd.1) 161 : cluster 0 3.4 scrub starts 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.307273+0000 osd.1 (osd.1) 162 : cluster 0 3.4 scrub ok 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.418977+0000 mon.a (mon.0) 1402 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: audit 2024-04-02T04:50:20.429677+0000 mon.a (mon.0) 1403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-11"}]': finished 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: audit 2024-04-02T04:50:20.485814+0000 mon.b (mon.1) 145 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: cluster 2024-04-02T04:50:20.486205+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e181: 8 total, 7 up, 8 in 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: audit 2024-04-02T04:50:20.490650+0000 mon.c (mon.2) 206 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]: dispatch 2024-04-02T04:50:22.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: audit 2024-04-02T04:50:20.493348+0000 mon.a (mon.0) 1405 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:22.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: audit 2024-04-02T04:50:20.495358+0000 mon.a (mon.0) 1406 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]: dispatch 2024-04-02T04:50:22.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:21 smithi098 bash[23790]: audit 2024-04-02T04:50:20.665707+0000 mon.c (mon.2) 207 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:22.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:22.263 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:22.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:22.391 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:22.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:22.523 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:22.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:22.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:22.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:22.774 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:22.877 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:22.900 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:22.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:23.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:23.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:23.154 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:23.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:50:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:50:23.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.019431+0000 osd.0 (osd.0) 149 : cluster 0 2.13 scrub starts 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.021186+0000 osd.0 (osd.0) 150 : cluster 0 2.13 scrub ok 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.432081+0000 mon.a (mon.0) 1407 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.435500+0000 mon.a (mon.0) 1408 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.435778+0000 mon.a (mon.0) 1409 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]': finished 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.450623+0000 mon.a (mon.0) 1410 : cluster 0 osdmap e182: 8 total, 7 up, 8 in 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.452757+0000 mon.c (mon.2) 208 : audit 1 from='client.? 172.21.15.69:0/2629970712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.464211+0000 mon.c (mon.2) 209 : audit 1 from='client.? 172.21.15.69:0/3059702661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.667046+0000 mon.c (mon.2) 210 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.845265+0000 osd.6 (osd.6) 113 : cluster 0 2.14 scrub starts 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.847531+0000 osd.6 (osd.6) 114 : cluster 0 2.14 scrub ok 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.910707+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:21.920371+0000 mon.a (mon.0) 1412 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:22.440543+0000 mon.a (mon.0) 1413 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:22.440765+0000 mon.a (mon.0) 1414 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: cluster 2024-04-02T04:50:22.453035+0000 mon.a (mon.0) 1415 : cluster 0 osdmap e183: 8 total, 7 up, 8 in 2024-04-02T04:50:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:22.455687+0000 mon.b (mon.1) 146 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]: dispatch 2024-04-02T04:50:23.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:22 smithi098 bash[23790]: audit 2024-04-02T04:50:22.460937+0000 mon.a (mon.0) 1416 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]: dispatch 2024-04-02T04:50:23.256 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.019431+0000 osd.0 (osd.0) 149 : cluster 0 2.13 scrub starts 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.021186+0000 osd.0 (osd.0) 150 : cluster 0 2.13 scrub ok 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.432081+0000 mon.a (mon.0) 1407 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.435500+0000 mon.a (mon.0) 1408 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.435778+0000 mon.a (mon.0) 1409 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]': finished 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.450623+0000 mon.a (mon.0) 1410 : cluster 0 osdmap e182: 8 total, 7 up, 8 in 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.452757+0000 mon.c (mon.2) 208 : audit 1 from='client.? 172.21.15.69:0/2629970712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.464211+0000 mon.c (mon.2) 209 : audit 1 from='client.? 172.21.15.69:0/3059702661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.667046+0000 mon.c (mon.2) 210 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.845265+0000 osd.6 (osd.6) 113 : cluster 0 2.14 scrub starts 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.847531+0000 osd.6 (osd.6) 114 : cluster 0 2.14 scrub ok 2024-04-02T04:50:23.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.910707+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:21.920371+0000 mon.a (mon.0) 1412 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:22.440543+0000 mon.a (mon.0) 1413 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:22.440765+0000 mon.a (mon.0) 1414 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: cluster 2024-04-02T04:50:22.453035+0000 mon.a (mon.0) 1415 : cluster 0 osdmap e183: 8 total, 7 up, 8 in 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:22.455687+0000 mon.b (mon.1) 146 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]: dispatch 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[28904]: audit 2024-04-02T04:50:22.460937+0000 mon.a (mon.0) 1416 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]: dispatch 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.019431+0000 osd.0 (osd.0) 149 : cluster 0 2.13 scrub starts 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.021186+0000 osd.0 (osd.0) 150 : cluster 0 2.13 scrub ok 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.432081+0000 mon.a (mon.0) 1407 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.435500+0000 mon.a (mon.0) 1408 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72366-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.435778+0000 mon.a (mon.0) 1409 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-11", "mode": "writeback"}]': finished 2024-04-02T04:50:23.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.450623+0000 mon.a (mon.0) 1410 : cluster 0 osdmap e182: 8 total, 7 up, 8 in 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.452757+0000 mon.c (mon.2) 208 : audit 1 from='client.? 172.21.15.69:0/2629970712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.464211+0000 mon.c (mon.2) 209 : audit 1 from='client.? 172.21.15.69:0/3059702661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.667046+0000 mon.c (mon.2) 210 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.845265+0000 osd.6 (osd.6) 113 : cluster 0 2.14 scrub starts 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.847531+0000 osd.6 (osd.6) 114 : cluster 0 2.14 scrub ok 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.910707+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:21.920371+0000 mon.a (mon.0) 1412 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:22.440543+0000 mon.a (mon.0) 1413 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-72145-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:22.440765+0000 mon.a (mon.0) 1414 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: cluster 2024-04-02T04:50:22.453035+0000 mon.a (mon.0) 1415 : cluster 0 osdmap e183: 8 total, 7 up, 8 in 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:22.455687+0000 mon.b (mon.1) 146 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]: dispatch 2024-04-02T04:50:23.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:22 smithi069 bash[21528]: audit 2024-04-02T04:50:22.460937+0000 mon.a (mon.0) 1416 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]: dispatch 2024-04-02T04:50:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:23.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:23.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:23.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:23.661 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:50:23.661 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:50:23.661 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:50:23.661 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:50:23.661 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:50:23.663 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:23.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:23.785 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:23.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:23.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:23.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.949+0000 7fb42fb14700 1 -- 172.21.15.69:0/577343912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806ab00 msgr2=0x7fb42806aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:23.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.949+0000 7fb42fb14700 1 --2- 172.21.15.69:0/577343912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806ab00 0x7fb42806aee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb424009990 tx=0x7fb424009ca0 comp rx=0 tx=0).stop 2024-04-02T04:50:23.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- 172.21.15.69:0/577343912 shutdown_connections 2024-04-02T04:50:23.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 --2- 172.21.15.69:0/577343912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb428107680 0x7fb42810f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:23.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 --2- 172.21.15.69:0/577343912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806ab00 0x7fb42806aee0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:23.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 --2- 172.21.15.69:0/577343912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb42806b4b0 0x7fb428106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:23.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- 172.21.15.69:0/577343912 >> 172.21.15.69:0/577343912 conn(0x7fb428078240 msgr2=0x7fb428078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:23.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- 172.21.15.69:0/577343912 shutdown_connections 2024-04-02T04:50:23.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- 172.21.15.69:0/577343912 wait complete. 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 librados: starting msgr at 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 librados: starting objecter 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 Processor -- start 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- start start 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 librados: setting wanted keys 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 librados: calling monclient init 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb42806ab00 0x7fb428198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:23.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 0x7fb428198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:23.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb428107680 0x7fb42819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:23.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb428111a90 con 0x7fb42806ab00 2024-04-02T04:50:23.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb428111910 con 0x7fb428107680 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42fb14700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb428111c10 con 0x7fb42806b4b0 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 0x7fb428198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 0x7fb428198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34732/0 (socket says 172.21.15.69:34732) 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 -- 172.21.15.69:0/4225045362 learned_addr learned my addr 172.21.15.69:0/4225045362 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d8b0700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb42806ab00 0x7fb428198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42e0b1700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb428107680 0x7fb42819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 -- 172.21.15.69:0/4225045362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb428107680 msgr2=0x7fb42819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb428107680 0x7fb42819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 -- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb42806ab00 msgr2=0x7fb428198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb42806ab00 0x7fb428198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 -- 172.21.15.69:0/4225045362 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42819d040 con 0x7fb42806b4b0 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d8b0700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb42806ab00 0x7fb428198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:50:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42e0b1700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb428107680 0x7fb42819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:50:23.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb42d0af700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 0x7fb428198570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb41800d980 tx=0x7fb41800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:23.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb41effd700 10 client.?.objecter ms_handle_connect 0x7fb42806b4b0 2024-04-02T04:50:23.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.953+0000 7fb41effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418017070 con 0x7fb42806b4b0 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb418009db0 con 0x7fb42806b4b0 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42819d330 con 0x7fb42806b4b0 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb42fb14700 10 client.25727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41800f3c0 con 0x7fb42806b4b0 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4281a4bb0 con 0x7fb42806b4b0 2024-04-02T04:50:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb42fb14700 1 librados: init done 2024-04-02T04:50:23.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb418009930 con 0x7fb42806b4b0 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb41406f6b0 0x7fb414071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(185..185 src has 74..185) ==== 14149+0+0 (secure 0 0 0) 0x7fb41808f520 con 0x7fb42806b4b0 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 10 client.25727.objecter ms_dispatch 0x7fb428001550 osd_map(185..185 src has 74..185) 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 3 client.25727.objecter handle_osd_map got epochs [185,185] > 0 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 3 client.25727.objecter handle_osd_map decoding full epoch 185 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb42d8b0700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb41406f6b0 0x7fb414071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb41effd700 20 client.25727.objecter dump_active .. 0 homeless 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.973+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb42804ff00 con 0x7fb42806b4b0 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.977+0000 7fb42d8b0700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb41406f6b0 0x7fb414071b70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb42400b2e0 tx=0x7fb42401a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:23.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.977+0000 7fb41effd700 10 client.25727.objecter ms_handle_connect 0x7fb41406f6b0 2024-04-02T04:50:23.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:23.985+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb41805b180 con 0x7fb42806b4b0 2024-04-02T04:50:24.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:24.040 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:24.143 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:24.166 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:24.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 -- 172.21.15.69:0/4245332175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 msgr2=0x7f30f41018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 --2- 172.21.15.69:0/4245332175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 0x7f30f41018e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f30e80099c0 tx=0x7f30e8009cd0 comp rx=0 tx=0).stop 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 -- 172.21.15.69:0/4245332175 shutdown_connections 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 --2- 172.21.15.69:0/4245332175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f4101e20 0x7f30f410dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 --2- 172.21.15.69:0/4245332175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f4107350 0x7f30f4107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 --2- 172.21.15.69:0/4245332175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 0x7f30f41018e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 -- 172.21.15.69:0/4245332175 >> 172.21.15.69:0/4245332175 conn(0x7f30f4077ea0 msgr2=0x7f30f40782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 -- 172.21.15.69:0/4245332175 shutdown_connections 2024-04-02T04:50:24.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 -- 172.21.15.69:0/4245332175 wait complete. 2024-04-02T04:50:24.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 librados: starting msgr at 2024-04-02T04:50:24.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 librados: starting objecter 2024-04-02T04:50:24.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 Processor -- start 2024-04-02T04:50:24.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 -- start start 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 librados: setting wanted keys 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.165+0000 7f30f90d1700 1 librados: calling monclient init 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 0x7f30f4102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f4101e20 0x7f30f41033d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f4107350 0x7f30f4103910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4110a10 con 0x7f30f4101e20 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4110890 con 0x7f30f4107350 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f30f4110b90 con 0x7f30f4101460 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f4101e20 0x7f30f41033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f2d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 0x7f30f4102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:24.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f4101e20 0x7f30f41033d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55034/0 (socket says 172.21.15.69:55034) 2024-04-02T04:50:24.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f259c700 1 -- 172.21.15.69:0/1918550930 learned_addr learned my addr 172.21.15.69:0/1918550930 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:24.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f4107350 0x7f30f4103910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 -- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 msgr2=0x7f30f4102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f4101460 0x7f30f4102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 -- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f4101e20 msgr2=0x7f30f41033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f4101e20 0x7f30f41033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f41aaac0 con 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f4107350 0x7f30f4103910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00cc50 tx=0x7f30ec00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30dbfff700 10 client.?.objecter ms_handle_connect 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30dbfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec016020 con 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f41aad50 con 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f30ec012420 con 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec01b660 con 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 10 client.25375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f41ab1b0 con 0x7f30f4107350 2024-04-02T04:50:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 librados: init done 2024-04-02T04:50:24.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.169+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f30f41028d0 con 0x7f30f4107350 2024-04-02T04:50:24.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f30ec019020 con 0x7f30f4107350 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f30dc06f5e0 0x7f30dc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(185..185 src has 74..185) ==== 14149+0+0 (secure 0 0 0) 0x7f30ec08f630 con 0x7f30f4107350 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 osd_map(185..185 src has 74..185) 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 3 client.25375.objecter handle_osd_map got epochs [185,185] > 0 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 3 client.25375.objecter handle_osd_map decoding full epoch 185 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f2d9d700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f30dc06f5e0 0x7f30dc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 20 client.25375.objecter dump_active .. 0 homeless 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v185) ==== 54+0+16 (secure 0 0 0) 0x7f30ec020030 con 0x7f30f4107350 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30dc06f5e0 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f2d9d700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f30dc06f5e0 0x7f30dc071aa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f30e4005950 tx=0x7f30e400b410 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 10 client.25375.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 1 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f30f4062e50 0x7f30f4063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 1 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 0 1 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41043c0 osd=0 4 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 10 client.25375.objecter _send_command 1 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30f404ff00 con 0x7f30f4062e50 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f259c700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f30f4062e50 0x7f30f4063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f4062e50 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30f259c700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f30f4062e50 0x7f30f4063230 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f404ff00 con 0x7f30f4062e50 2024-04-02T04:50:24.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 1: 0 ) 2024-04-02T04:50:24.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 10 client.25375.objecter _finish_command 1 = system:0 2024-04-02T04:50:24.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41043c0 osd=0 4 2024-04-02T04:50:24.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.173+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 0 1 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 10 client.25375.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 2 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 2 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41043c0 osd=0 3 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 0 2 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41043c0 osd=0 4 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 10 client.25375.objecter _send_command 2 2024-04-02T04:50:24.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f40628d0 con 0x7f30f4062e50 2024-04-02T04:50:24.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f40628d0 con 0x7f30f4062e50 2024-04-02T04:50:24.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30dbfff700 10 client.25375.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41043c0 osd=0 4 2024-04-02T04:50:24.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 0 2 2024-04-02T04:50:24.194 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:50:24.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 10 client.25375.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 3 2024-04-02T04:50:24.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f30f410ad20 0x7f30f41af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 1 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402db30 osd=1 4 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 10 client.25375.objecter _send_command 3 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f30f402dca0 con 0x7f30f410ad20 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f2d9d700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f30f410ad20 0x7f30f41af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f410ad20 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.193+0000 7f30f2d9d700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f30f410ad20 0x7f30f41af460 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.197+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f402dca0 con 0x7f30f410ad20 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.197+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 3: 0 ) 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.197+0000 7f30dbfff700 10 client.25375.objecter _finish_command 3 = system:0 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.197+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f402db30 osd=1 4 2024-04-02T04:50:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.197+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 1 3 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 10 client.25375.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 4 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 4 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f402db30 osd=1 3 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 1 4 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402db30 osd=1 4 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 10 client.25375.objecter _send_command 4 2024-04-02T04:50:24.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f41af9a0 con 0x7f30f410ad20 2024-04-02T04:50:24.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f41af9a0 con 0x7f30f410ad20 2024-04-02T04:50:24.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30dbfff700 10 client.25375.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f402db30 osd=1 4 2024-04-02T04:50:24.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 1 4 2024-04-02T04:50:24.216 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:50:24.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 10 client.25375.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 5 2024-04-02T04:50:24.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f30f410a400 0x7f30f41b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 5 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 2 5 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f410a290 osd=2 4 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 10 client.25375.objecter _send_command 5 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.213+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f30f402d690 con 0x7f30f410a400 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30f259c700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f30f410a400 0x7f30f41b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30f259c700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f30f410a400 0x7f30f41b2ce0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f410a400 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f402d690 con 0x7f30f410a400 2024-04-02T04:50:24.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 5: 0 ) 2024-04-02T04:50:24.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30dbfff700 10 client.25375.objecter _finish_command 5 = system:0 2024-04-02T04:50:24.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f410a290 osd=2 4 2024-04-02T04:50:24.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.217+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 2 5 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 10 client.25375.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 6 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 6 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f410a290 osd=2 3 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 2 6 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f410a290 osd=2 4 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 10 client.25375.objecter _send_command 6 2024-04-02T04:50:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f402cc00 con 0x7f30f410a400 2024-04-02T04:50:24.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f402cc00 con 0x7f30f410a400 2024-04-02T04:50:24.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30dbfff700 10 client.25375.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f410a290 osd=2 4 2024-04-02T04:50:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 2 6 2024-04-02T04:50:24.235 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 10 client.25375.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 7 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f30f41b0830 0x7f30f41b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 7 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 3 7 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402d000 osd=3 4 2024-04-02T04:50:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 10 client.25375.objecter _send_command 7 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f30f41b9210 con 0x7f30f41b0830 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f30f41b0830 0x7f30f41b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f41b0830 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.233+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f30f41b0830 0x7f30f41b63b0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.237+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f41b9210 con 0x7f30f41b0830 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.237+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 7: 0 ) 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.237+0000 7f30dbfff700 10 client.25375.objecter _finish_command 7 = system:0 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.237+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f402d000 osd=3 4 2024-04-02T04:50:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.237+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 3 7 2024-04-02T04:50:24.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.245+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fb428101120 con 0x7fb42806b4b0 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 10 client.25375.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 8 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 8 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f402d000 osd=3 3 2024-04-02T04:50:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 3 8 2024-04-02T04:50:24.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f402d000 osd=3 4 2024-04-02T04:50:24.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 10 client.25375.objecter _send_command 8 2024-04-02T04:50:24.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f41afec0 con 0x7f30f41b0830 2024-04-02T04:50:24.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f41afec0 con 0x7f30f41b0830 2024-04-02T04:50:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30dbfff700 10 client.25375.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f402d000 osd=3 4 2024-04-02T04:50:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 3 8 2024-04-02T04:50:24.255 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 10 client.25375.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 9 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f30f41afc70 0x7f30f41b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 9 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 4 9 2024-04-02T04:50:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b9260 osd=4 4 2024-04-02T04:50:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 10 client.25375.objecter _send_command 9 2024-04-02T04:50:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.253+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f30f41b4080 con 0x7f30f41afc70 2024-04-02T04:50:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30f2d9d700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f30f41afc70 0x7f30f41b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f41afc70 2024-04-02T04:50:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30f2d9d700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f30f41afc70 0x7f30f41b7a70 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f41b4080 con 0x7f30f41afc70 2024-04-02T04:50:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 9: 0 ) 2024-04-02T04:50:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30dbfff700 10 client.25375.objecter _finish_command 9 = system:0 2024-04-02T04:50:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41b9260 osd=4 4 2024-04-02T04:50:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.257+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 4 9 2024-04-02T04:50:24.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:24.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 10 client.25375.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 10 2024-04-02T04:50:24.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 10 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41b9260 osd=4 3 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 4 10 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b9260 osd=4 4 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 10 client.25375.objecter _send_command 10 2024-04-02T04:50:24.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.273+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f41b4080 con 0x7f30f41afc70 2024-04-02T04:50:24.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f41b4080 con 0x7f30f41afc70 2024-04-02T04:50:24.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30dbfff700 10 client.25375.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41b9260 osd=4 4 2024-04-02T04:50:24.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 4 10 2024-04-02T04:50:24.279 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:50:24.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 10 client.25375.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 11 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f30f41b84c0 0x7f30f41b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 11 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 5 11 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b4080 osd=5 4 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 10 client.25375.objecter _send_command 11 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f30f41bfd90 con 0x7f30f41b84c0 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.277+0000 7f30f259c700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f30f41b84c0 0x7f30f41b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f41b84c0 2024-04-02T04:50:24.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30f259c700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f30f41b84c0 0x7f30f41b8940 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f41bfd90 con 0x7f30f41b84c0 2024-04-02T04:50:24.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 11: 0 ) 2024-04-02T04:50:24.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30dbfff700 10 client.25375.objecter _finish_command 11 = system:0 2024-04-02T04:50:24.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41b4080 osd=5 4 2024-04-02T04:50:24.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.281+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 5 11 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 10 client.25375.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 12 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 12 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41b4080 osd=5 3 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 5 12 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41b4080 osd=5 4 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 10 client.25375.objecter _send_command 12 2024-04-02T04:50:24.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.297+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f41bd470 con 0x7f30f41b84c0 2024-04-02T04:50:24.299 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:24.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f41bd470 con 0x7f30f41b84c0 2024-04-02T04:50:24.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 10 client.25375.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41b4080 osd=5 4 2024-04-02T04:50:24.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 5 12 2024-04-02T04:50:24.301 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 10 client.25375.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 13 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f30f41bdaf0 0x7f30f41bfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 13 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 6 13 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41bd470 osd=6 4 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 10 client.25375.objecter _send_command 13 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f30f41c2ec0 con 0x7f30f41bdaf0 2024-04-02T04:50:24.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f30f41bdaf0 0x7f30f41bfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 10 client.25375.objecter ms_handle_connect 0x7f30f41bdaf0 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30f359e700 1 --2- 172.21.15.69:0/1918550930 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f30f41bdaf0 0x7f30f41bfee0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30f41c2ec0 con 0x7f30f41bdaf0 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 13: 0 ) 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 10 client.25375.objecter _finish_command 13 = system:0 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41bd470 osd=6 4 2024-04-02T04:50:24.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.301+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 6 13 2024-04-02T04:50:24.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 10 client.25375.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:50:24.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 14 2024-04-02T04:50:24.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 20 client.25375.objecter _get_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_remove -1 14 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 20 client.25375.objecter get_session s=0x7f30f41bd470 osd=6 3 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign 6 14 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 20 client.25375.objecter put_session s=0x7f30f41bd470 osd=6 4 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 10 client.25375.objecter _send_command 14 2024-04-02T04:50:24.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.317+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f30f41b00d0 con 0x7f30f41bdaf0 2024-04-02T04:50:24.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30f41b00d0 con 0x7f30f41bdaf0 2024-04-02T04:50:24.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30dbfff700 10 client.25375.objecter ms_dispatch 0x7f30f4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:24.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30dbfff700 10 client.25375.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:24.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30dbfff700 20 client.25375.objecter put_session s=0x7f30f41bd470 osd=6 4 2024-04-02T04:50:24.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30dbfff700 15 client.25375.objecter _session_command_op_remove 6 14 2024-04-02T04:50:24.321 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f90d1700 10 client.25375.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f90d1700 15 client.25375.objecter _session_command_op_assign -1 15 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f90d1700 20 client.25375.objecter _get_session osd=-1 returning homeless 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f90d1700 10 client.25375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=186}) -- 0x7f30f41c1cd0 con 0x7f30f4107350 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f90d1700 1 -- 172.21.15.69:0/1918550930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f30f41c2050 con 0x7f30f4107350 2024-04-02T04:50:24.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30dbfff700 1 -- 172.21.15.69:0/1918550930 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_get_version_reply(handle=1 version=185) ==== 24+0+0 (secure 0 0 0) 0x7f30ec08bb20 con 0x7f30f4107350 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f159a700 10 client.25375.objecter _check_command_map_dne tid 15 current 185 map_dne_bound 185 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f159a700 10 client.25375.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.321+0000 7f30f159a700 15 client.25375.objecter _session_command_op_remove -1 15 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:50:24.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:50:24.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:24.426 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:24.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:24.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: cluster 2024-04-02T04:50:21.901231+0000 mgr.y (mgr.14538) 262 : cluster 0 pgmap v250: 588 pgs: 1 active+clean+snaptrim, 113 active+undersized, 261 active+clean, 160 unknown, 16 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 110/876 objects degraded (12.557%) 2024-04-02T04:50:24.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:22.568418+0000 mon.c (mon.2) 211 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:24.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:22.668433+0000 mon.c (mon.2) 212 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:24.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: cluster 2024-04-02T04:50:22.822936+0000 osd.6 (osd.6) 115 : cluster 0 10.a scrub starts 2024-04-02T04:50:24.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: cluster 2024-04-02T04:50:22.824986+0000 osd.6 (osd.6) 116 : cluster 0 10.a scrub ok 2024-04-02T04:50:24.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:22.951993+0000 mon.a (mon.0) 1417 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:23.442870+0000 mon.a (mon.0) 1418 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]': finished 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:23.442979+0000 mon.a (mon.0) 1419 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:23.444594+0000 mon.b (mon.1) 147 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: cluster 2024-04-02T04:50:23.449379+0000 mon.a (mon.0) 1420 : cluster 0 osdmap e184: 8 total, 7 up, 8 in 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:23.451308+0000 mon.c (mon.2) 213 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:23.453312+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:24.541 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:24 smithi098 bash[23790]: audit 2024-04-02T04:50:23.453807+0000 mon.a (mon.0) 1422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:24.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: cluster 2024-04-02T04:50:21.901231+0000 mgr.y (mgr.14538) 262 : cluster 0 pgmap v250: 588 pgs: 1 active+clean+snaptrim, 113 active+undersized, 261 active+clean, 160 unknown, 16 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 110/876 objects degraded (12.557%) 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:22.568418+0000 mon.c (mon.2) 211 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:22.668433+0000 mon.c (mon.2) 212 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: cluster 2024-04-02T04:50:22.822936+0000 osd.6 (osd.6) 115 : cluster 0 10.a scrub starts 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: cluster 2024-04-02T04:50:22.824986+0000 osd.6 (osd.6) 116 : cluster 0 10.a scrub ok 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:22.951993+0000 mon.a (mon.0) 1417 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:24.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:23.442870+0000 mon.a (mon.0) 1418 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]': finished 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:23.442979+0000 mon.a (mon.0) 1419 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:23.444594+0000 mon.b (mon.1) 147 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: cluster 2024-04-02T04:50:23.449379+0000 mon.a (mon.0) 1420 : cluster 0 osdmap e184: 8 total, 7 up, 8 in 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:23.451308+0000 mon.c (mon.2) 213 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:23.453312+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[21528]: audit 2024-04-02T04:50:23.453807+0000 mon.a (mon.0) 1422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: cluster 2024-04-02T04:50:21.901231+0000 mgr.y (mgr.14538) 262 : cluster 0 pgmap v250: 588 pgs: 1 active+clean+snaptrim, 113 active+undersized, 261 active+clean, 160 unknown, 16 creating+peering, 37 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 110/876 objects degraded (12.557%) 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:22.568418+0000 mon.c (mon.2) 211 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:22.668433+0000 mon.c (mon.2) 212 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:24.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: cluster 2024-04-02T04:50:22.822936+0000 osd.6 (osd.6) 115 : cluster 0 10.a scrub starts 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: cluster 2024-04-02T04:50:22.824986+0000 osd.6 (osd.6) 116 : cluster 0 10.a scrub ok 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:22.951993+0000 mon.a (mon.0) 1417 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:23.442870+0000 mon.a (mon.0) 1418 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache", "force_nonempty":""}]': finished 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:23.442979+0000 mon.a (mon.0) 1419 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:23.444594+0000 mon.b (mon.1) 147 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: cluster 2024-04-02T04:50:23.449379+0000 mon.a (mon.0) 1420 : cluster 0 osdmap e184: 8 total, 7 up, 8 in 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:23.451308+0000 mon.c (mon.2) 213 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:23.453312+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:24.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:24 smithi069 bash[28904]: audit 2024-04-02T04:50:23.453807+0000 mon.a (mon.0) 1422 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]: dispatch 2024-04-02T04:50:24.682 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:24.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:24.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:24.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:24.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:24.937 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.973+0000 7fb41effd700 1 -- 172.21.15.69:0/4225045362 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v186) ==== 101+0+0 (secure 0 0 0) 0x7fb41805ebd0 con 0x7fb42806b4b0 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.973+0000 7fb42fb14700 10 librados: watch_flush enter 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.973+0000 7fb42fb14700 10 librados: watch_flush exit 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.973+0000 7fb42fb14700 20 client.25727.objecter shutdown clearing up homeless session... 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.973+0000 7fb42fb14700 10 client.25727.objecter successfully canceled tick 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb41406f6b0 msgr2=0x7fb414071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb41406f6b0 0x7fb414071b70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb42400b2e0 tx=0x7fb42401a040 comp rx=0 tx=0).stop 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 msgr2=0x7fb428198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 0x7fb428198570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb41800d980 tx=0x7fb41800dc90 comp rx=0 tx=0).stop 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 shutdown_connections 2024-04-02T04:50:25.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb41406f6b0 0x7fb414071b70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb428107680 0x7fb42819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb42806ab00 0x7fb428198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 --2- 172.21.15.69:0/4225045362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb42806b4b0 0x7fb428198570 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 >> 172.21.15.69:0/4225045362 conn(0x7fb428078240 msgr2=0x7fb4280fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 shutdown_connections 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 -- 172.21.15.69:0/4225045362 wait complete. 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:24.977+0000 7fb42fb14700 1 librados: shutdown 2024-04-02T04:50:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:50:25.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:25.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:25.314 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:25.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:25.444 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:25.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:25.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:25.692 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:23.036439+0000 mgr.y (mgr.14538) 263 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:25.692 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:23.232051+0000 osd.3 (osd.3) 177 : cluster 0 2.1e scrub starts 2024-04-02T04:50:25.692 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:23.234210+0000 osd.3 (osd.3) 178 : cluster 0 2.1e scrub ok 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:23.669401+0000 mon.c (mon.2) 214 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:23.902696+0000 mgr.y (mgr.14538) 264 : cluster 0 pgmap v253: 588 pgs: 2 active+clean+snaptrim, 135 active+undersized, 312 active+clean, 64 creating+peering, 32 unknown, 43 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 259 KiB/s wr, 9 op/s; 119/903 objects degraded (13.178%) 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:23.953432+0000 mon.a (mon.0) 1423 : cluster 3 Health check update: Degraded data redundancy: 110/876 objects degraded (12.557%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:23.954929+0000 mon.a (mon.0) 1424 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:23.957753+0000 mon.a (mon.0) 1425 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:23.957984+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]': finished 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:23.970158+0000 mon.b (mon.1) 148 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.171830+0000 mon.b (mon.1) 149 : audit 0 from='client.? 172.21.15.69:0/1918550930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.250199+0000 mon.c (mon.2) 215 : audit 1 from='client.? 172.21.15.69:0/4225045362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:24.442419+0000 mon.a (mon.0) 1427 : cluster 0 osdmap e185: 8 total, 7 up, 8 in 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.454797+0000 mon.a (mon.0) 1428 : audit 1 from='client.? 172.21.15.69:0/4013348432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-72153-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.457174+0000 mon.a (mon.0) 1429 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.670702+0000 mon.c (mon.2) 216 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.877669+0000 mon.a (mon.0) 1430 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:50:25.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:24.959776+0000 mon.a (mon.0) 1431 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.962629+0000 mon.a (mon.0) 1432 : audit 1 from='client.? 172.21.15.69:0/4013348432' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-72153-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.962852+0000 mon.a (mon.0) 1433 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]': finished 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.963072+0000 mon.a (mon.0) 1434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: audit 2024-04-02T04:50:24.981855+0000 mon.b (mon.1) 150 : audit 1 from='client.? 172.21.15.69:0/1093822697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[28904]: cluster 2024-04-02T04:50:24.986414+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e186: 8 total, 7 up, 8 in 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:23.036439+0000 mgr.y (mgr.14538) 263 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:23.232051+0000 osd.3 (osd.3) 177 : cluster 0 2.1e scrub starts 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:23.234210+0000 osd.3 (osd.3) 178 : cluster 0 2.1e scrub ok 2024-04-02T04:50:25.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:23.669401+0000 mon.c (mon.2) 214 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:23.902696+0000 mgr.y (mgr.14538) 264 : cluster 0 pgmap v253: 588 pgs: 2 active+clean+snaptrim, 135 active+undersized, 312 active+clean, 64 creating+peering, 32 unknown, 43 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 259 KiB/s wr, 9 op/s; 119/903 objects degraded (13.178%) 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:23.953432+0000 mon.a (mon.0) 1423 : cluster 3 Health check update: Degraded data redundancy: 110/876 objects degraded (12.557%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:23.954929+0000 mon.a (mon.0) 1424 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:23.957753+0000 mon.a (mon.0) 1425 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:23.957984+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]': finished 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:23.970158+0000 mon.b (mon.1) 148 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.171830+0000 mon.b (mon.1) 149 : audit 0 from='client.? 172.21.15.69:0/1918550930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.250199+0000 mon.c (mon.2) 215 : audit 1 from='client.? 172.21.15.69:0/4225045362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:24.442419+0000 mon.a (mon.0) 1427 : cluster 0 osdmap e185: 8 total, 7 up, 8 in 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.454797+0000 mon.a (mon.0) 1428 : audit 1 from='client.? 172.21.15.69:0/4013348432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-72153-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.457174+0000 mon.a (mon.0) 1429 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.670702+0000 mon.c (mon.2) 216 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.877669+0000 mon.a (mon.0) 1430 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:24.959776+0000 mon.a (mon.0) 1431 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:50:25.695 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.962629+0000 mon.a (mon.0) 1432 : audit 1 from='client.? 172.21.15.69:0/4013348432' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-72153-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:25.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.962852+0000 mon.a (mon.0) 1433 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]': finished 2024-04-02T04:50:25.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.963072+0000 mon.a (mon.0) 1434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:50:25.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: audit 2024-04-02T04:50:24.981855+0000 mon.b (mon.1) 150 : audit 1 from='client.? 172.21.15.69:0/1093822697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:25.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:25 smithi069 bash[21528]: cluster 2024-04-02T04:50:24.986414+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e186: 8 total, 7 up, 8 in 2024-04-02T04:50:25.698 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:23.036439+0000 mgr.y (mgr.14538) 263 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:23.232051+0000 osd.3 (osd.3) 177 : cluster 0 2.1e scrub starts 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:23.234210+0000 osd.3 (osd.3) 178 : cluster 0 2.1e scrub ok 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:23.669401+0000 mon.c (mon.2) 214 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:23.902696+0000 mgr.y (mgr.14538) 264 : cluster 0 pgmap v253: 588 pgs: 2 active+clean+snaptrim, 135 active+undersized, 312 active+clean, 64 creating+peering, 32 unknown, 43 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 259 KiB/s wr, 9 op/s; 119/903 objects degraded (13.178%) 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:23.953432+0000 mon.a (mon.0) 1423 : cluster 3 Health check update: Degraded data redundancy: 110/876 objects degraded (12.557%), 37 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:23.954929+0000 mon.a (mon.0) 1424 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:23.957753+0000 mon.a (mon.0) 1425 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72366-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:23.957984+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-11"}]': finished 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:23.970158+0000 mon.b (mon.1) 148 : audit 1 from='client.? 172.21.15.69:0/1861889037' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.171830+0000 mon.b (mon.1) 149 : audit 0 from='client.? 172.21.15.69:0/1918550930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.250199+0000 mon.c (mon.2) 215 : audit 1 from='client.? 172.21.15.69:0/4225045362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:24.442419+0000 mon.a (mon.0) 1427 : cluster 0 osdmap e185: 8 total, 7 up, 8 in 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.454797+0000 mon.a (mon.0) 1428 : audit 1 from='client.? 172.21.15.69:0/4013348432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-72153-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.457174+0000 mon.a (mon.0) 1429 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.670702+0000 mon.c (mon.2) 216 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.877669+0000 mon.a (mon.0) 1430 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:24.959776+0000 mon.a (mon.0) 1431 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.962629+0000 mon.a (mon.0) 1432 : audit 1 from='client.? 172.21.15.69:0/4013348432' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-72153-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.962852+0000 mon.a (mon.0) 1433 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72366-10", "tierpool":"test-rados-api-smithi069-72366-10-cache"}]': finished 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.963072+0000 mon.a (mon.0) 1434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: audit 2024-04-02T04:50:24.981855+0000 mon.b (mon.1) 150 : audit 1 from='client.? 172.21.15.69:0/1093822697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:25.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:25 smithi098 bash[23790]: cluster 2024-04-02T04:50:24.986414+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e186: 8 total, 7 up, 8 in 2024-04-02T04:50:25.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-02T04:50:25.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.829 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:25.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:25.954 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:25.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:26.056 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:26.077 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:26.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:26.208 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:26.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:26.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:26.440 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:26.461 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:26.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:26.583 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:26.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:26.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:26.842 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:50:26.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:50:26.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:26.965 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:26.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: cluster 2024-04-02T04:50:24.858181+0000 osd.6 (osd.6) 117 : cluster 0 10.b scrub starts 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: cluster 2024-04-02T04:50:24.860235+0000 osd.6 (osd.6) 118 : cluster 0 10.b scrub ok 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: cluster 2024-04-02T04:50:25.449933+0000 mon.a (mon.0) 1436 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: audit 2024-04-02T04:50:25.480494+0000 mon.a (mon.0) 1437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: audit 2024-04-02T04:50:25.672350+0000 mon.c (mon.2) 217 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: cluster 2024-04-02T04:50:25.821651+0000 osd.6 (osd.6) 119 : cluster 0 10.1c scrub starts 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: cluster 2024-04-02T04:50:25.822826+0000 osd.6 (osd.6) 120 : cluster 0 10.1c scrub ok 2024-04-02T04:50:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: audit 2024-04-02T04:50:26.342604+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: audit 2024-04-02T04:50:26.368901+0000 mon.c (mon.2) 218 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.372236+0000 mon.a (mon.0) 1439 : cluster 0 osdmap e187: 8 total, 7 up, 8 in 2024-04-02T04:50:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: audit 2024-04-02T04:50:26.396632+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:26 smithi098 bash[23790]: audit 2024-04-02T04:50:26.397130+0000 mon.a (mon.0) 1441 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:27.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:27.094 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:27.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:27.224 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: cluster 2024-04-02T04:50:24.858181+0000 osd.6 (osd.6) 117 : cluster 0 10.b scrub starts 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: cluster 2024-04-02T04:50:24.860235+0000 osd.6 (osd.6) 118 : cluster 0 10.b scrub ok 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: cluster 2024-04-02T04:50:25.449933+0000 mon.a (mon.0) 1436 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: audit 2024-04-02T04:50:25.480494+0000 mon.a (mon.0) 1437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: audit 2024-04-02T04:50:25.672350+0000 mon.c (mon.2) 217 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: cluster 2024-04-02T04:50:25.821651+0000 osd.6 (osd.6) 119 : cluster 0 10.1c scrub starts 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: cluster 2024-04-02T04:50:25.822826+0000 osd.6 (osd.6) 120 : cluster 0 10.1c scrub ok 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: audit 2024-04-02T04:50:26.342604+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: audit 2024-04-02T04:50:26.368901+0000 mon.c (mon.2) 218 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.372236+0000 mon.a (mon.0) 1439 : cluster 0 osdmap e187: 8 total, 7 up, 8 in 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: audit 2024-04-02T04:50:26.396632+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[28904]: audit 2024-04-02T04:50:26.397130+0000 mon.a (mon.0) 1441 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: cluster 2024-04-02T04:50:24.858181+0000 osd.6 (osd.6) 117 : cluster 0 10.b scrub starts 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: cluster 2024-04-02T04:50:24.860235+0000 osd.6 (osd.6) 118 : cluster 0 10.b scrub ok 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: cluster 2024-04-02T04:50:25.449933+0000 mon.a (mon.0) 1436 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: audit 2024-04-02T04:50:25.480494+0000 mon.a (mon.0) 1437 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: audit 2024-04-02T04:50:25.672350+0000 mon.c (mon.2) 217 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: cluster 2024-04-02T04:50:25.821651+0000 osd.6 (osd.6) 119 : cluster 0 10.1c scrub starts 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: cluster 2024-04-02T04:50:25.822826+0000 osd.6 (osd.6) 120 : cluster 0 10.1c scrub ok 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: audit 2024-04-02T04:50:26.342604+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-72145-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:27.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: audit 2024-04-02T04:50:26.368901+0000 mon.c (mon.2) 218 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.372236+0000 mon.a (mon.0) 1439 : cluster 0 osdmap e187: 8 total, 7 up, 8 in 2024-04-02T04:50:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: audit 2024-04-02T04:50:26.396632+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:26 smithi069 bash[21528]: audit 2024-04-02T04:50:26.397130+0000 mon.a (mon.0) 1441 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:27.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:27.359 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:27.365 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: Running main() from gmock_main.cc 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment set-up. 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolList (19062 ms) 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (12696 ms) 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (6041 ms) 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (6008 ms) 2024-04-02T04:50:27.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (6017 ms) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6076 ms) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6783 ms) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6698 ms) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7933 ms) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools (77315 ms total) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment tear-down 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (77315 ms total) 2024-04-02T04:50:27.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-02T04:50:27.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:27.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:27.611 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:27.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:27.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:27.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:27.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:27.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:25.903410+0000 mgr.y (mgr.14538) 265 : cluster 0 pgmap v256: 556 pgs: 1 active+clean+snaptrim, 112 active+undersized, 275 active+clean, 32 creating+peering, 96 unknown, 40 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 257 KiB/s wr, 4 op/s; 115/888 objects degraded (12.950%) 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:25.997659+0000 osd.0 (osd.0) 151 : cluster 0 3.17 scrub starts 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:25.999890+0000 osd.0 (osd.0) 152 : cluster 0 3.17 scrub ok 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.340670+0000 osd.1 (osd.1) 163 : cluster 0 6.4 scrub starts 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.342417+0000 osd.1 (osd.1) 164 : cluster 0 6.4 scrub ok 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.507482+0000 osd.5 (osd.5) 141 : cluster 0 3.c scrub starts 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.509286+0000 osd.5 (osd.5) 142 : cluster 0 3.c scrub ok 2024-04-02T04:50:27.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: audit 2024-04-02T04:50:26.673974+0000 mon.c (mon.2) 219 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.843177+0000 osd.6 (osd.6) 121 : cluster 0 13.1 scrub starts 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:26.845890+0000 osd.6 (osd.6) 122 : cluster 0 13.1 scrub ok 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: audit 2024-04-02T04:50:27.347675+0000 mon.a (mon.0) 1442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: audit 2024-04-02T04:50:27.348003+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: cluster 2024-04-02T04:50:27.361985+0000 mon.a (mon.0) 1444 : cluster 0 osdmap e188: 8 total, 7 up, 8 in 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: audit 2024-04-02T04:50:27.365175+0000 mon.c (mon.2) 220 : audit 1 from='client.? 172.21.15.69:0/3412615522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: audit 2024-04-02T04:50:27.370994+0000 mon.a (mon.0) 1445 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:27.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:27 smithi098 bash[23790]: audit 2024-04-02T04:50:27.372098+0000 mon.a (mon.0) 1446 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:28.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:28.130 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:28.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:28.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:25.903410+0000 mgr.y (mgr.14538) 265 : cluster 0 pgmap v256: 556 pgs: 1 active+clean+snaptrim, 112 active+undersized, 275 active+clean, 32 creating+peering, 96 unknown, 40 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 257 KiB/s wr, 4 op/s; 115/888 objects degraded (12.950%) 2024-04-02T04:50:28.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:25.997659+0000 osd.0 (osd.0) 151 : cluster 0 3.17 scrub starts 2024-04-02T04:50:28.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:25.999890+0000 osd.0 (osd.0) 152 : cluster 0 3.17 scrub ok 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.340670+0000 osd.1 (osd.1) 163 : cluster 0 6.4 scrub starts 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.342417+0000 osd.1 (osd.1) 164 : cluster 0 6.4 scrub ok 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.507482+0000 osd.5 (osd.5) 141 : cluster 0 3.c scrub starts 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.509286+0000 osd.5 (osd.5) 142 : cluster 0 3.c scrub ok 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: audit 2024-04-02T04:50:26.673974+0000 mon.c (mon.2) 219 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.843177+0000 osd.6 (osd.6) 121 : cluster 0 13.1 scrub starts 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:26.845890+0000 osd.6 (osd.6) 122 : cluster 0 13.1 scrub ok 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: audit 2024-04-02T04:50:27.347675+0000 mon.a (mon.0) 1442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: audit 2024-04-02T04:50:27.348003+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: cluster 2024-04-02T04:50:27.361985+0000 mon.a (mon.0) 1444 : cluster 0 osdmap e188: 8 total, 7 up, 8 in 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: audit 2024-04-02T04:50:27.365175+0000 mon.c (mon.2) 220 : audit 1 from='client.? 172.21.15.69:0/3412615522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:28.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: audit 2024-04-02T04:50:27.370994+0000 mon.a (mon.0) 1445 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[28904]: audit 2024-04-02T04:50:27.372098+0000 mon.a (mon.0) 1446 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:25.903410+0000 mgr.y (mgr.14538) 265 : cluster 0 pgmap v256: 556 pgs: 1 active+clean+snaptrim, 112 active+undersized, 275 active+clean, 32 creating+peering, 96 unknown, 40 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 257 KiB/s wr, 4 op/s; 115/888 objects degraded (12.950%) 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:25.997659+0000 osd.0 (osd.0) 151 : cluster 0 3.17 scrub starts 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:25.999890+0000 osd.0 (osd.0) 152 : cluster 0 3.17 scrub ok 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.340670+0000 osd.1 (osd.1) 163 : cluster 0 6.4 scrub starts 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.342417+0000 osd.1 (osd.1) 164 : cluster 0 6.4 scrub ok 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.507482+0000 osd.5 (osd.5) 141 : cluster 0 3.c scrub starts 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.509286+0000 osd.5 (osd.5) 142 : cluster 0 3.c scrub ok 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: audit 2024-04-02T04:50:26.673974+0000 mon.c (mon.2) 219 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.843177+0000 osd.6 (osd.6) 121 : cluster 0 13.1 scrub starts 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:26.845890+0000 osd.6 (osd.6) 122 : cluster 0 13.1 scrub ok 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: audit 2024-04-02T04:50:27.347675+0000 mon.a (mon.0) 1442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:28.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: audit 2024-04-02T04:50:27.348003+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:28.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: cluster 2024-04-02T04:50:27.361985+0000 mon.a (mon.0) 1444 : cluster 0 osdmap e188: 8 total, 7 up, 8 in 2024-04-02T04:50:28.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: audit 2024-04-02T04:50:27.365175+0000 mon.c (mon.2) 220 : audit 1 from='client.? 172.21.15.69:0/3412615522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:28.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: audit 2024-04-02T04:50:27.370994+0000 mon.a (mon.0) 1445 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]: dispatch 2024-04-02T04:50:28.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:27 smithi069 bash[21528]: audit 2024-04-02T04:50:27.372098+0000 mon.a (mon.0) 1446 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:28.263 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:28.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:28.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:28.534 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:28.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.663 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:28.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:50:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:50:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:50:28.793 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:28.856 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:50:28.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:50:28.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:28.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:28.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:29.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:29.050 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:29.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:29.179 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:29.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:29.308 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:29.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:29.411 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:29.430 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: cluster 2024-04-02T04:50:27.008369+0000 osd.2 (osd.2) 101 : cluster 0 15.e scrub starts 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: cluster 2024-04-02T04:50:27.009292+0000 osd.2 (osd.2) 102 : cluster 0 15.e scrub ok 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:27.675585+0000 mon.c (mon.2) 221 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.243669+0000 osd.3 (osd.3) 179 : cluster 0 2.19 scrub starts 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.247643+0000 osd.3 (osd.3) 180 : cluster 0 2.19 scrub ok 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.353880+0000 mon.a (mon.0) 1447 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.354032+0000 mon.a (mon.0) 1448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.363535+0000 mon.a (mon.0) 1449 : cluster 0 osdmap e189: 8 total, 7 up, 8 in 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.372523+0000 mon.c (mon.2) 222 : audit 1 from='client.? 172.21.15.69:0/2821291575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.373419+0000 mon.a (mon.0) 1450 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.415391+0000 mon.b (mon.1) 151 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.677160+0000 mon.c (mon.2) 223 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.859583+0000 mon.b (mon.1) 152 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:29 smithi098 bash[23790]: audit 2024-04-02T04:50:28.862539+0000 mon.a (mon.0) 1451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:29.559 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:29.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: cluster 2024-04-02T04:50:27.008369+0000 osd.2 (osd.2) 101 : cluster 0 15.e scrub starts 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: cluster 2024-04-02T04:50:27.009292+0000 osd.2 (osd.2) 102 : cluster 0 15.e scrub ok 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:27.675585+0000 mon.c (mon.2) 221 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.243669+0000 osd.3 (osd.3) 179 : cluster 0 2.19 scrub starts 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.247643+0000 osd.3 (osd.3) 180 : cluster 0 2.19 scrub ok 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.353880+0000 mon.a (mon.0) 1447 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.354032+0000 mon.a (mon.0) 1448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.363535+0000 mon.a (mon.0) 1449 : cluster 0 osdmap e189: 8 total, 7 up, 8 in 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.372523+0000 mon.c (mon.2) 222 : audit 1 from='client.? 172.21.15.69:0/2821291575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.373419+0000 mon.a (mon.0) 1450 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.415391+0000 mon.b (mon.1) 151 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.677160+0000 mon.c (mon.2) 223 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.859583+0000 mon.b (mon.1) 152 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[28904]: audit 2024-04-02T04:50:28.862539+0000 mon.a (mon.0) 1451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: cluster 2024-04-02T04:50:27.008369+0000 osd.2 (osd.2) 101 : cluster 0 15.e scrub starts 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: cluster 2024-04-02T04:50:27.009292+0000 osd.2 (osd.2) 102 : cluster 0 15.e scrub ok 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:27.675585+0000 mon.c (mon.2) 221 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.243669+0000 osd.3 (osd.3) 179 : cluster 0 2.19 scrub starts 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.247643+0000 osd.3 (osd.3) 180 : cluster 0 2.19 scrub ok 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.353880+0000 mon.a (mon.0) 1447 : audit 1 from='client.? 172.21.15.69:0/1438082132' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-72378-10"}]': finished 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.354032+0000 mon.a (mon.0) 1448 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-72153-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.363535+0000 mon.a (mon.0) 1449 : cluster 0 osdmap e189: 8 total, 7 up, 8 in 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.372523+0000 mon.c (mon.2) 222 : audit 1 from='client.? 172.21.15.69:0/2821291575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.373419+0000 mon.a (mon.0) 1450 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:29.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.415391+0000 mon.b (mon.1) 151 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.677160+0000 mon.c (mon.2) 223 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:29.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.859583+0000 mon.b (mon.1) 152 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:29 smithi069 bash[21528]: audit 2024-04-02T04:50:28.862539+0000 mon.a (mon.0) 1451 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:29.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:29.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:29.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:29.928 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:29.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:30.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:30.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:30.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:30.189 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:30.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:30.318 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:30.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:27.562651+0000 osd.4 (osd.4) 157 : cluster 0 6.1e scrub starts 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:27.563819+0000 osd.4 (osd.4) 158 : cluster 0 6.1e scrub ok 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:27.905409+0000 mgr.y (mgr.14538) 266 : cluster 0 pgmap v259: 548 pgs: 132 active+undersized, 337 active+clean, 32 creating+peering, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 27 KiB/s wr, 30 op/s; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.024649+0000 osd.0 (osd.0) 153 : cluster 0 3.6 deep-scrub starts 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.030651+0000 osd.0 (osd.0) 154 : cluster 0 3.6 deep-scrub ok 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.259866+0000 osd.1 (osd.1) 165 : cluster 0 13.10 scrub starts 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:28.261602+0000 osd.1 (osd.1) 166 : cluster 0 13.10 scrub ok 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.051154+0000 osd.0 (osd.0) 155 : cluster 0 13.12 deep-scrub starts 2024-04-02T04:50:30.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.052217+0000 osd.0 (osd.0) 156 : cluster 0 13.12 deep-scrub ok 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.281484+0000 osd.3 (osd.3) 181 : cluster 0 4.1d deep-scrub starts 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.283393+0000 osd.3 (osd.3) 182 : cluster 0 4.1d deep-scrub ok 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.317869+0000 mon.a (mon.0) 1452 : cluster 3 Health check update: Degraded data redundancy: 151/1191 objects degraded (12.678%), 47 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.323315+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.327265+0000 mon.b (mon.1) 153 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.331560+0000 mon.a (mon.0) 1454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.356099+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.356159+0000 mon.a (mon.0) 1456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.361096+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e190: 8 total, 7 up, 8 in 2024-04-02T04:50:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.361370+0000 mon.b (mon.1) 154 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:27.562651+0000 osd.4 (osd.4) 157 : cluster 0 6.1e scrub starts 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:27.563819+0000 osd.4 (osd.4) 158 : cluster 0 6.1e scrub ok 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:27.905409+0000 mgr.y (mgr.14538) 266 : cluster 0 pgmap v259: 548 pgs: 132 active+undersized, 337 active+clean, 32 creating+peering, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 27 KiB/s wr, 30 op/s; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.024649+0000 osd.0 (osd.0) 153 : cluster 0 3.6 deep-scrub starts 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.030651+0000 osd.0 (osd.0) 154 : cluster 0 3.6 deep-scrub ok 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.259866+0000 osd.1 (osd.1) 165 : cluster 0 13.10 scrub starts 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:28.261602+0000 osd.1 (osd.1) 166 : cluster 0 13.10 scrub ok 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.051154+0000 osd.0 (osd.0) 155 : cluster 0 13.12 deep-scrub starts 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.052217+0000 osd.0 (osd.0) 156 : cluster 0 13.12 deep-scrub ok 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.281484+0000 osd.3 (osd.3) 181 : cluster 0 4.1d deep-scrub starts 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.283393+0000 osd.3 (osd.3) 182 : cluster 0 4.1d deep-scrub ok 2024-04-02T04:50:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.317869+0000 mon.a (mon.0) 1452 : cluster 3 Health check update: Degraded data redundancy: 151/1191 objects degraded (12.678%), 47 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.323315+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.327265+0000 mon.b (mon.1) 153 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.331560+0000 mon.a (mon.0) 1454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.356099+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.356159+0000 mon.a (mon.0) 1456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.361096+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e190: 8 total, 7 up, 8 in 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.361370+0000 mon.b (mon.1) 154 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.369391+0000 mon.a (mon.0) 1458 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.532485+0000 osd.4 (osd.4) 159 : cluster 0 6.2 scrub starts 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.535296+0000 osd.4 (osd.4) 160 : cluster 0 6.2 scrub ok 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:29.678399+0000 mon.c (mon.2) 224 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:30.135182+0000 mon.c (mon.2) 225 : audit 0 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:30.137999+0000 mon.c (mon.2) 226 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[21528]: audit 2024-04-02T04:50:30.138332+0000 mon.a (mon.0) 1459 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.369391+0000 mon.a (mon.0) 1458 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.532485+0000 osd.4 (osd.4) 159 : cluster 0 6.2 scrub starts 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.535296+0000 osd.4 (osd.4) 160 : cluster 0 6.2 scrub ok 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:29.678399+0000 mon.c (mon.2) 224 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:30.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:30.135182+0000 mon.c (mon.2) 225 : audit 0 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T04:50:30.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:30.137999+0000 mon.c (mon.2) 226 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:30.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:30 smithi069 bash[28904]: audit 2024-04-02T04:50:30.138332+0000 mon.a (mon.0) 1459 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:30.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:30.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:30.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:30.708 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:27.562651+0000 osd.4 (osd.4) 157 : cluster 0 6.1e scrub starts 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:27.563819+0000 osd.4 (osd.4) 158 : cluster 0 6.1e scrub ok 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:27.905409+0000 mgr.y (mgr.14538) 266 : cluster 0 pgmap v259: 548 pgs: 132 active+undersized, 337 active+clean, 32 creating+peering, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 27 KiB/s wr, 30 op/s; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.024649+0000 osd.0 (osd.0) 153 : cluster 0 3.6 deep-scrub starts 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.030651+0000 osd.0 (osd.0) 154 : cluster 0 3.6 deep-scrub ok 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.259866+0000 osd.1 (osd.1) 165 : cluster 0 13.10 scrub starts 2024-04-02T04:50:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:28.261602+0000 osd.1 (osd.1) 166 : cluster 0 13.10 scrub ok 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.051154+0000 osd.0 (osd.0) 155 : cluster 0 13.12 deep-scrub starts 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.052217+0000 osd.0 (osd.0) 156 : cluster 0 13.12 deep-scrub ok 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.281484+0000 osd.3 (osd.3) 181 : cluster 0 4.1d deep-scrub starts 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.283393+0000 osd.3 (osd.3) 182 : cluster 0 4.1d deep-scrub ok 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.317869+0000 mon.a (mon.0) 1452 : cluster 3 Health check update: Degraded data redundancy: 151/1191 objects degraded (12.678%), 47 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.323315+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.327265+0000 mon.b (mon.1) 153 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.331560+0000 mon.a (mon.0) 1454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.356099+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-72145-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.356159+0000 mon.a (mon.0) 1456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.361096+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e190: 8 total, 7 up, 8 in 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.361370+0000 mon.b (mon.1) 154 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.369391+0000 mon.a (mon.0) 1458 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:30.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.532485+0000 osd.4 (osd.4) 159 : cluster 0 6.2 scrub starts 2024-04-02T04:50:30.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.535296+0000 osd.4 (osd.4) 160 : cluster 0 6.2 scrub ok 2024-04-02T04:50:30.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:29.678399+0000 mon.c (mon.2) 224 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:30.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:30.135182+0000 mon.c (mon.2) 225 : audit 0 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T04:50:30.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:30.137999+0000 mon.c (mon.2) 226 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:30.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:30 smithi098 bash[23790]: audit 2024-04-02T04:50:30.138332+0000 mon.a (mon.0) 1459 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:30.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:30.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:30.964 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:30.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:31.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:31.090 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:31.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:31.216 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:31.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:31.340 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: Running main() from gmock_main.cc 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] Global test environment set-up. 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.Limits (16814 ms) 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (11611 ms) 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.ReadOp (7599 ms) 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-02T04:50:31.396 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.SparseRead (5496 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.RoundTrip (4705 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (4281 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (5164 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (4428 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (4742 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.Trunc (4305 ms) 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-02T04:50:31.397 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.Remove (3017 ms) 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2538 ms) 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.RmXattr (3368 ms) 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3025 ms) 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] 14 tests from NeoRadosIo (81093 ms total) 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] Global test environment tear-down 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [==========] 14 tests from 1 test suite ran. (81093 ms total) 2024-04-02T04:50:31.398 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ PASSED ] 14 tests. 2024-04-02T04:50:31.414 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:31.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.804512+0000 osd.6 (osd.6) 123 : cluster 0 10.d scrub starts 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.806741+0000 osd.6 (osd.6) 124 : cluster 0 10.d scrub ok 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.907012+0000 mgr.y (mgr.14538) 267 : cluster 0 pgmap v262: 548 pgs: 111 active+undersized, 294 active+clean, 96 unknown, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 27 KiB/s wr, 28 op/s; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.990138+0000 osd.2 (osd.2) 103 : cluster 0 41.f deep-scrub starts 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:29.991163+0000 osd.2 (osd.2) 104 : cluster 0 41.f deep-scrub ok 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:30.066352+0000 osd.0 (osd.0) 157 : cluster 0 10.1e deep-scrub starts 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:30.067841+0000 osd.0 (osd.0) 158 : cluster 0 10.1e deep-scrub ok 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:30.292097+0000 osd.3 (osd.3) 183 : cluster 0 3.1e scrub starts 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:30.296184+0000 osd.3 (osd.3) 184 : cluster 0 3.1e scrub ok 2024-04-02T04:50:31.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.367286+0000 mon.a (mon.0) 1460 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.378422+0000 mon.c (mon.2) 227 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2"}]: dispatch 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.381846+0000 mon.c (mon.2) 228 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.679375+0000 mon.c (mon.2) 229 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: cluster 2024-04-02T04:50:30.822069+0000 mon.a (mon.0) 1461 : cluster 0 osdmap e191: 8 total, 7 up, 8 in 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.823547+0000 mon.a (mon.0) 1462 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.824874+0000 mon.c (mon.2) 230 : audit 1 from='client.? 172.21.15.69:0/1628092070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[21528]: audit 2024-04-02T04:50:30.830062+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.804512+0000 osd.6 (osd.6) 123 : cluster 0 10.d scrub starts 2024-04-02T04:50:31.461 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.806741+0000 osd.6 (osd.6) 124 : cluster 0 10.d scrub ok 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.907012+0000 mgr.y (mgr.14538) 267 : cluster 0 pgmap v262: 548 pgs: 111 active+undersized, 294 active+clean, 96 unknown, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 27 KiB/s wr, 28 op/s; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.990138+0000 osd.2 (osd.2) 103 : cluster 0 41.f deep-scrub starts 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:29.991163+0000 osd.2 (osd.2) 104 : cluster 0 41.f deep-scrub ok 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:30.066352+0000 osd.0 (osd.0) 157 : cluster 0 10.1e deep-scrub starts 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:30.067841+0000 osd.0 (osd.0) 158 : cluster 0 10.1e deep-scrub ok 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:30.292097+0000 osd.3 (osd.3) 183 : cluster 0 3.1e scrub starts 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:30.296184+0000 osd.3 (osd.3) 184 : cluster 0 3.1e scrub ok 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.367286+0000 mon.a (mon.0) 1460 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.378422+0000 mon.c (mon.2) 227 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2"}]: dispatch 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.381846+0000 mon.c (mon.2) 228 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.679375+0000 mon.c (mon.2) 229 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: cluster 2024-04-02T04:50:30.822069+0000 mon.a (mon.0) 1461 : cluster 0 osdmap e191: 8 total, 7 up, 8 in 2024-04-02T04:50:31.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.823547+0000 mon.a (mon.0) 1462 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.824874+0000 mon.c (mon.2) 230 : audit 1 from='client.? 172.21.15.69:0/1628092070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:31 smithi069 bash[28904]: audit 2024-04-02T04:50:30.830062+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.472 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:31.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:31.596 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:31.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:31.718 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:31.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.804512+0000 osd.6 (osd.6) 123 : cluster 0 10.d scrub starts 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.806741+0000 osd.6 (osd.6) 124 : cluster 0 10.d scrub ok 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.907012+0000 mgr.y (mgr.14538) 267 : cluster 0 pgmap v262: 548 pgs: 111 active+undersized, 294 active+clean, 96 unknown, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 27 KiB/s wr, 28 op/s; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.990138+0000 osd.2 (osd.2) 103 : cluster 0 41.f deep-scrub starts 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:29.991163+0000 osd.2 (osd.2) 104 : cluster 0 41.f deep-scrub ok 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:30.066352+0000 osd.0 (osd.0) 157 : cluster 0 10.1e deep-scrub starts 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:30.067841+0000 osd.0 (osd.0) 158 : cluster 0 10.1e deep-scrub ok 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:30.292097+0000 osd.3 (osd.3) 183 : cluster 0 3.1e scrub starts 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:30.296184+0000 osd.3 (osd.3) 184 : cluster 0 3.1e scrub ok 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.367286+0000 mon.a (mon.0) 1460 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.378422+0000 mon.c (mon.2) 227 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2"}]: dispatch 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.381846+0000 mon.c (mon.2) 228 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.679375+0000 mon.c (mon.2) 229 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: cluster 2024-04-02T04:50:30.822069+0000 mon.a (mon.0) 1461 : cluster 0 osdmap e191: 8 total, 7 up, 8 in 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.823547+0000 mon.a (mon.0) 1462 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.824874+0000 mon.c (mon.2) 230 : audit 1 from='client.? 172.21.15.69:0/1628092070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:31 smithi098 bash[23790]: audit 2024-04-02T04:50:30.830062+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:31.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:31.846 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:31.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:31.977 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:31.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:32.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:32.107 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:32.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:32.236 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:32.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:32.359 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:32.462 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- 172.21.15.69:0/3173111040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 msgr2=0x7f4114100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- 172.21.15.69:0/3173111040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 0x7f4114100d60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4104009990 tx=0x7f4104009ca0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- 172.21.15.69:0/3173111040 shutdown_connections 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- 172.21.15.69:0/3173111040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 0x7f411410d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- 172.21.15.69:0/3173111040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f4114106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- 172.21.15.69:0/3173111040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 0x7f4114100d60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- 172.21.15.69:0/3173111040 >> 172.21.15.69:0/3173111040 conn(0x7f4114077740 msgr2=0x7f41140feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- 172.21.15.69:0/3173111040 shutdown_connections 2024-04-02T04:50:32.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- 172.21.15.69:0/3173111040 wait complete. 2024-04-02T04:50:32.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 librados: starting msgr at 2024-04-02T04:50:32.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 librados: starting objecter 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 Processor -- start 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- start start 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 librados: setting wanted keys 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 librados: calling monclient init 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 0x7f4114073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 0x7f4114073e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f411406dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f411406c400 con 0x7f41141068e0 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f411406c280 con 0x7f41141008e0 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f411a556700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f411406c580 con 0x7f41141012a0 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f4118af3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f411406dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f4118af3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f411406dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33926/0 (socket says 172.21.15.69:33926) 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f4118af3700 1 -- 172.21.15.69:0/4128356016 learned_addr learned my addr 172.21.15.69:0/4128356016 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:32.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.476+0000 7f41137fe700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 0x7f4114073e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41137fe700 1 -- 172.21.15.69:0/4128356016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 msgr2=0x7f4114073940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41137fe700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 0x7f4114073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41137fe700 1 -- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 msgr2=0x7f411406dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41137fe700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f411406dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41137fe700 1 -- 172.21.15.69:0/4128356016 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411406e4a0 con 0x7f41141012a0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f4118af3700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f411406dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41137fe700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 0x7f4114073e80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c00 tx=0x7f41040057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41117fa700 10 client.?.objecter ms_handle_connect 0x7f41141012a0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41117fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410400fc50 con 0x7f41141012a0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4104023070 con 0x7f41141012a0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104027410 con 0x7f41141012a0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411406e730 con 0x7f41141012a0 2024-04-02T04:50:32.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f411a556700 10 client.25796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:32.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f411410efd0 con 0x7f41141012a0 2024-04-02T04:50:32.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f411a556700 1 librados: init done 2024-04-02T04:50:32.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4114102020 con 0x7f41141012a0 2024-04-02T04:50:32.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.480+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4104025020 con 0x7f41141012a0 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f410006f620 0x7f4100071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(193..193 src has 74..193) ==== 13095+0+0 (secure 0 0 0) 0x7f4104094ad0 con 0x7f41141012a0 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 10 client.25796.objecter ms_dispatch 0x7f4114001550 osd_map(193..193 src has 74..193) 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 3 client.25796.objecter handle_osd_map got epochs [193,193] > 0 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 3 client.25796.objecter handle_osd_map decoding full epoch 193 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 20 client.25796.objecter dump_active .. 0 homeless 2024-04-02T04:50:32.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f4113fff700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f410006f620 0x7f4100071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:32.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f41117fa700 10 client.25796.objecter ms_handle_connect 0x7f410006f620 2024-04-02T04:50:32.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.484+0000 7f4113fff700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f410006f620 0x7f4100071ae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009b60 tx=0x7f40fc009620 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:32.487 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:32.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.488+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4104013070 con 0x7f41141012a0 2024-04-02T04:50:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:32.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:32.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.377981+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.378062+0000 mon.a (mon.0) 1465 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.378119+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: cluster 2024-04-02T04:50:31.382094+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e192: 8 total, 7 up, 8 in 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.392784+0000 mon.c (mon.2) 231 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.393626+0000 mon.c (mon.2) 232 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.395192+0000 mon.a (mon.0) 1468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.395493+0000 mon.a (mon.0) 1469 : audit 1 from='client.? 172.21.15.69:0/3860825036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-72145-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: cluster 2024-04-02T04:50:31.492153+0000 osd.4 (osd.4) 161 : cluster 0 4.e scrub starts 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: cluster 2024-04-02T04:50:31.493705+0000 osd.4 (osd.4) 162 : cluster 0 4.e scrub ok 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.630305+0000 mon.c (mon.2) 233 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.680507+0000 mon.c (mon.2) 234 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.903061+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:32.598 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.946975+0000 mon.a (mon.0) 1471 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[21528]: audit 2024-04-02T04:50:31.947436+0000 mon.b (mon.1) 155 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.377981+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.378062+0000 mon.a (mon.0) 1465 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.378119+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: cluster 2024-04-02T04:50:31.382094+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e192: 8 total, 7 up, 8 in 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.392784+0000 mon.c (mon.2) 231 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.393626+0000 mon.c (mon.2) 232 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.395192+0000 mon.a (mon.0) 1468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.395493+0000 mon.a (mon.0) 1469 : audit 1 from='client.? 172.21.15.69:0/3860825036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-72145-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:32.599 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: cluster 2024-04-02T04:50:31.492153+0000 osd.4 (osd.4) 161 : cluster 0 4.e scrub starts 2024-04-02T04:50:32.600 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: cluster 2024-04-02T04:50:31.493705+0000 osd.4 (osd.4) 162 : cluster 0 4.e scrub ok 2024-04-02T04:50:32.600 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.630305+0000 mon.c (mon.2) 233 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:32.600 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.680507+0000 mon.c (mon.2) 234 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:32.600 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.903061+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:32.600 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.946975+0000 mon.a (mon.0) 1471 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:32.600 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:32 smithi069 bash[28904]: audit 2024-04-02T04:50:31.947436+0000 mon.b (mon.1) 155 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:32.609 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:32.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.684+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f411404ff00 con 0x7f410006f620 2024-04-02T04:50:32.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.377981+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-72378-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.378062+0000 mon.a (mon.0) 1465 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-72304-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.378119+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-72153-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: cluster 2024-04-02T04:50:31.382094+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e192: 8 total, 7 up, 8 in 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.392784+0000 mon.c (mon.2) 231 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.393626+0000 mon.c (mon.2) 232 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.395192+0000 mon.a (mon.0) 1468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.395493+0000 mon.a (mon.0) 1469 : audit 1 from='client.? 172.21.15.69:0/3860825036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-72145-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: cluster 2024-04-02T04:50:31.492153+0000 osd.4 (osd.4) 161 : cluster 0 4.e scrub starts 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: cluster 2024-04-02T04:50:31.493705+0000 osd.4 (osd.4) 162 : cluster 0 4.e scrub ok 2024-04-02T04:50:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.630305+0000 mon.c (mon.2) 233 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.680507+0000 mon.c (mon.2) 234 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.903061+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:50:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.946975+0000 mon.a (mon.0) 1471 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:32.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:32 smithi098 bash[23790]: audit 2024-04-02T04:50:31.947436+0000 mon.b (mon.1) 155 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:32.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.728+0000 7f41117fa700 1 -- 172.21.15.69:0/4128356016 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+1245593 (secure 0 0 0) 0x7f40fc0097b0 con 0x7f410006f620 2024-04-02T04:50:32.741 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 10 librados: watch_flush enter 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 10 librados: watch_flush exit 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 20 client.25796.objecter shutdown clearing up homeless session... 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 10 client.25796.objecter successfully canceled tick 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f410006f620 msgr2=0x7f4100071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f410006f620 0x7f4100071ae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009b60 tx=0x7f40fc009620 comp rx=0 tx=0).stop 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 msgr2=0x7f4114073e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 0x7f4114073e80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c00 tx=0x7f41040057c0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 shutdown_connections 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f410006f620 0x7f4100071ae0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41141068e0 0x7f411406dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41141008e0 0x7f4114073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 --2- 172.21.15.69:0/4128356016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41141012a0 0x7f4114073e80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 >> 172.21.15.69:0/4128356016 conn(0x7f4114077740 msgr2=0x7f4114103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 shutdown_connections 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 -- 172.21.15.69:0/4128356016 wait complete. 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:32.736+0000 7f411a556700 1 librados: shutdown 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:32.761 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:50:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:32.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:32.888 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:32.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:32.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:33.010 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:33.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:33.134 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:33.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:50:33.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:33.264 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:33.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:33.390 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:33.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:33.510 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:33.613 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:33.635 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:31.778904+0000 osd.6 (osd.6) 125 : cluster 0 10.f scrub starts 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:31.781263+0000 osd.6 (osd.6) 126 : cluster 0 10.f scrub ok 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:31.908029+0000 mgr.y (mgr.14538) 268 : cluster 0 pgmap v265: 524 pgs: 111 active+undersized, 294 active+clean, 72 unknown, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.020153+0000 osd.2 (osd.2) 105 : cluster 0 15.11 scrub starts 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.021519+0000 osd.2 (osd.2) 106 : cluster 0 15.11 scrub ok 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.092077+0000 osd.0 (osd.0) 159 : cluster 0 3.1f scrub starts 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.096245+0000 osd.0 (osd.0) 160 : cluster 0 3.1f scrub ok 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.381676+0000 osd.3 (osd.3) 185 : cluster 0 6.11 deep-scrub starts 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.383412+0000 osd.3 (osd.3) 186 : cluster 0 6.11 deep-scrub ok 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.387447+0000 mon.a (mon.0) 1472 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-02T04:50:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.387547+0000 mon.a (mon.0) 1473 : audit 1 from='client.? 172.21.15.69:0/3860825036' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-72145-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.387633+0000 mon.a (mon.0) 1474 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: cluster 2024-04-02T04:50:32.393995+0000 mon.a (mon.0) 1475 : cluster 0 osdmap e193: 8 total, 7 up, 8 in 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.394363+0000 mon.c (mon.2) 235 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.395233+0000 mon.a (mon.0) 1476 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.395392+0000 mon.c (mon.2) 236 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.414164+0000 mon.a (mon.0) 1477 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:50:33.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:33 smithi098 bash[23790]: audit 2024-04-02T04:50:32.681606+0000 mon.c (mon.2) 237 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:33.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:33.760 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:31.778904+0000 osd.6 (osd.6) 125 : cluster 0 10.f scrub starts 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:31.781263+0000 osd.6 (osd.6) 126 : cluster 0 10.f scrub ok 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:31.908029+0000 mgr.y (mgr.14538) 268 : cluster 0 pgmap v265: 524 pgs: 111 active+undersized, 294 active+clean, 72 unknown, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.020153+0000 osd.2 (osd.2) 105 : cluster 0 15.11 scrub starts 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.021519+0000 osd.2 (osd.2) 106 : cluster 0 15.11 scrub ok 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.092077+0000 osd.0 (osd.0) 159 : cluster 0 3.1f scrub starts 2024-04-02T04:50:33.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.096245+0000 osd.0 (osd.0) 160 : cluster 0 3.1f scrub ok 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.381676+0000 osd.3 (osd.3) 185 : cluster 0 6.11 deep-scrub starts 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.383412+0000 osd.3 (osd.3) 186 : cluster 0 6.11 deep-scrub ok 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.387447+0000 mon.a (mon.0) 1472 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.387547+0000 mon.a (mon.0) 1473 : audit 1 from='client.? 172.21.15.69:0/3860825036' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-72145-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.387633+0000 mon.a (mon.0) 1474 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: cluster 2024-04-02T04:50:32.393995+0000 mon.a (mon.0) 1475 : cluster 0 osdmap e193: 8 total, 7 up, 8 in 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.394363+0000 mon.c (mon.2) 235 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.395233+0000 mon.a (mon.0) 1476 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.395392+0000 mon.c (mon.2) 236 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.414164+0000 mon.a (mon.0) 1477 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:50:33.853 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[21528]: audit 2024-04-02T04:50:32.681606+0000 mon.c (mon.2) 237 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:31.778904+0000 osd.6 (osd.6) 125 : cluster 0 10.f scrub starts 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:31.781263+0000 osd.6 (osd.6) 126 : cluster 0 10.f scrub ok 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:31.908029+0000 mgr.y (mgr.14538) 268 : cluster 0 pgmap v265: 524 pgs: 111 active+undersized, 294 active+clean, 72 unknown, 47 active+undersized+degraded; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 151/1191 objects degraded (12.678%) 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.020153+0000 osd.2 (osd.2) 105 : cluster 0 15.11 scrub starts 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.021519+0000 osd.2 (osd.2) 106 : cluster 0 15.11 scrub ok 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.092077+0000 osd.0 (osd.0) 159 : cluster 0 3.1f scrub starts 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.096245+0000 osd.0 (osd.0) 160 : cluster 0 3.1f scrub ok 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.381676+0000 osd.3 (osd.3) 185 : cluster 0 6.11 deep-scrub starts 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.383412+0000 osd.3 (osd.3) 186 : cluster 0 6.11 deep-scrub ok 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.387447+0000 mon.a (mon.0) 1472 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.387547+0000 mon.a (mon.0) 1473 : audit 1 from='client.? 172.21.15.69:0/3860825036' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-72145-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.387633+0000 mon.a (mon.0) 1474 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: cluster 2024-04-02T04:50:32.393995+0000 mon.a (mon.0) 1475 : cluster 0 osdmap e193: 8 total, 7 up, 8 in 2024-04-02T04:50:33.854 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.394363+0000 mon.c (mon.2) 235 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:50:33.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.395233+0000 mon.a (mon.0) 1476 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:50:33.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.395392+0000 mon.c (mon.2) 236 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:50:33.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.414164+0000 mon.a (mon.0) 1477 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:50:33.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:33 smithi069 bash[28904]: audit 2024-04-02T04:50:32.681606+0000 mon.c (mon.2) 237 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:33.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:33.886 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:33.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:33.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:34.010 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:34.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:34.135 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:34.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:34.263 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: Running main() from gmock_main.cc 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] Global test environment set-up. 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (18762 ms) 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-02T04:50:34.336 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (9606 ms) 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (10503 ms) 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (4612 ms) 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (6979 ms) 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (5137 ms) 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (4441 ms) 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-02T04:50:34.337 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (4748 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3265 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3052 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3532 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3383 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3023 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2940 ms) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (83983 ms total) 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: 2024-04-02T04:50:34.338 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] Global test environment tear-down 2024-04-02T04:50:34.339 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (83983 ms total) 2024-04-02T04:50:34.339 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-02T04:50:34.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:34.396 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-04-02T04:50:32.389343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227100652,"num_objects":395,"num_object_clones":4,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":2,"num_read":2608,"num_read_kb":25986,"num_write":2099,"num_write_kb":230396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11200000,"num_omap_keys":4000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2018,"ondisk_log_size":2018,"up":1112,"acting":1112,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1672,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1067348,"kb_used_data":685984,"kb_used_omap":12,"kb_used_meta":381171,"kb_avail":748893868,"statfs":{"total":767960285184,"available":766867320832,"internally_reserved":0,"allocated":702447616,"data_stored":693154964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":390319664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1792,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":-2,"num_read_kb":0,"num_write":-10,"num_write_kb":-10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032159"},"pg_stats":[{"pgid":"175.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1a","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.399663+0000","last_change":"2024-04-02T04:50:27.391289+0000","last_active":"2024-04-02T04:50:27.399663+0000","last_peered":"2024-04-02T04:50:27.399663+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.391289+0000","last_became_peered":"2024-04-02T04:50:27.391289+0000","last_unstale":"2024-04-02T04:50:27.399663+0000","last_undegraded":"2024-04-02T04:50:27.399663+0000","last_fullsized":"2024-04-02T04:50:27.357482+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.121156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"175.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1b","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.463278+0000","last_change":"2024-04-02T04:50:27.461514+0000","last_active":"2024-04-02T04:50:27.463278+0000","last_peered":"2024-04-02T04:50:27.463278+0000","last_clean":"2024-04-02T04:50:27.463278+0000","last_became_active":"2024-04-02T04:50:27.461142+0000","last_became_peered":"2024-04-02T04:50:27.461142+0000","last_unstale":"2024-04-02T04:50:27.463278+0000","last_undegraded":"2024-04-02T04:50:27.463278+0000","last_fullsized":"2024-04-02T04:50:27.463278+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.800087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"175.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.18","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.436229+0000","last_change":"2024-04-02T04:50:27.435434+0000","last_active":"2024-04-02T04:50:27.436229+0000","last_peered":"2024-04-02T04:50:27.436229+0000","last_clean":"2024-04-02T04:50:27.436229+0000","last_became_active":"2024-04-02T04:50:27.434978+0000","last_became_peered":"2024-04-02T04:50:27.434978+0000","last_unstale":"2024-04-02T04:50:27.436229+0000","last_undegraded":"2024-04-02T04:50:27.436229+0000","last_fullsized":"2024-04-02T04:50:27.436229+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.843445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"175.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.19","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.456198+0000","last_change":"2024-04-02T04:50:27.454668+0000","last_active":"2024-04-02T04:50:27.456198+0000","last_peered":"2024-04-02T04:50:27.456198+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.454668+0000","last_became_peered":"2024-04-02T04:50:27.454668+0000","last_unstale":"2024-04-02T04:50:27.456198+0000","last_undegraded":"2024-04-02T04:50:27.456198+0000","last_fullsized":"2024-04-02T04:50:27.353984+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.529875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"175.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1e","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.464977+0000","last_change":"2024-04-02T04:50:27.463638+0000","last_active":"2024-04-02T04:50:27.464977+0000","last_peered":"2024-04-02T04:50:27.464977+0000","last_clean":"2024-04-02T04:50:27.464977+0000","last_became_active":"2024-04-02T04:50:27.463279+0000","last_became_peered":"2024-04-02T04:50:27.463279+0000","last_unstale":"2024-04-02T04:50:27.464977+0000","last_undegraded":"2024-04-02T04:50:27.464977+0000","last_fullsized":"2024-04-02T04:50:27.464977+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.664143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"175.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1f","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.465265+0000","last_change":"2024-04-02T04:50:27.464769+0000","last_active":"2024-04-02T04:50:27.465265+0000","last_peered":"2024-04-02T04:50:27.465265+0000","last_clean":"2024-04-02T04:50:27.465265+0000","last_became_active":"2024-04-02T04:50:27.462550+0000","last_became_peered":"2024-04-02T04:50:27.462550+0000","last_unstale":"2024-04-02T04:50:27.465265+0000","last_undegraded":"2024-04-02T04:50:27.465265+0000","last_fullsized":"2024-04-02T04:50:27.465265+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.681736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"175.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1c","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.464361+0000","last_change":"2024-04-02T04:50:27.463383+0000","last_active":"2024-04-02T04:50:27.464361+0000","last_peered":"2024-04-02T04:50:27.464361+0000","last_clean":"2024-04-02T04:50:27.464361+0000","last_became_active":"2024-04-02T04:50:27.462580+0000","last_became_peered":"2024-04-02T04:50:27.462580+0000","last_unstale":"2024-04-02T04:50:27.464361+0000","last_undegraded":"2024-04-02T04:50:27.464361+0000","last_fullsized":"2024-04-02T04:50:27.464361+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:49.841703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"175.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1d","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.455381+0000","last_change":"2024-04-02T04:50:27.445968+0000","last_active":"2024-04-02T04:50:27.455381+0000","last_peered":"2024-04-02T04:50:27.455381+0000","last_clean":"2024-04-02T04:50:27.455381+0000","last_became_active":"2024-04-02T04:50:27.441751+0000","last_became_peered":"2024-04-02T04:50:27.441751+0000","last_unstale":"2024-04-02T04:50:27.455381+0000","last_undegraded":"2024-04-02T04:50:27.455381+0000","last_fullsized":"2024-04-02T04:50:27.455381+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.565619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"175.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.12","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.456327+0000","last_change":"2024-04-02T04:50:27.455809+0000","last_active":"2024-04-02T04:50:27.456327+0000","last_peered":"2024-04-02T04:50:27.456327+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.455809+0000","last_became_peered":"2024-04-02T04:50:27.455809+0000","last_unstale":"2024-04-02T04:50:27.456327+0000","last_undegraded":"2024-04-02T04:50:27.456327+0000","last_fullsized":"2024-04-02T04:50:27.371468+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:46.659439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"175.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.13","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.426516+0000","last_change":"2024-04-02T04:50:27.424958+0000","last_active":"2024-04-02T04:50:27.426516+0000","last_peered":"2024-04-02T04:50:27.426516+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.424958+0000","last_became_peered":"2024-04-02T04:50:27.424958+0000","last_unstale":"2024-04-02T04:50:27.426516+0000","last_undegraded":"2024-04-02T04:50:27.426516+0000","last_fullsized":"2024-04-02T04:50:27.353406+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:46.951225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"175.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.10","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.413981+0000","last_change":"2024-04-02T04:50:27.413520+0000","last_active":"2024-04-02T04:50:27.413981+0000","last_peered":"2024-04-02T04:50:27.413981+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.413520+0000","last_became_peered":"2024-04-02T04:50:27.413520+0000","last_unstale":"2024-04-02T04:50:27.413981+0000","last_undegraded":"2024-04-02T04:50:27.413981+0000","last_fullsized":"2024-04-02T04:50:27.354145+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.809978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"175.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.11","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.449805+0000","last_change":"2024-04-02T04:50:27.449537+0000","last_active":"2024-04-02T04:50:27.449805+0000","last_peered":"2024-04-02T04:50:27.449805+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.449537+0000","last_became_peered":"2024-04-02T04:50:27.449537+0000","last_unstale":"2024-04-02T04:50:27.449805+0000","last_undegraded":"2024-04-02T04:50:27.449805+0000","last_fullsized":"2024-04-02T04:50:27.387079+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.070270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"175.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.16","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.424528+0000","last_change":"2024-04-02T04:50:27.415780+0000","last_active":"2024-04-02T04:50:27.424528+0000","last_peered":"2024-04-02T04:50:27.424528+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.415780+0000","last_became_peered":"2024-04-02T04:50:27.415780+0000","last_unstale":"2024-04-02T04:50:27.424528+0000","last_undegraded":"2024-04-02T04:50:27.424528+0000","last_fullsized":"2024-04-02T04:50:27.377984+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:48.999703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"175.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.17","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.443311+0000","last_change":"2024-04-02T04:50:27.421841+0000","last_active":"2024-04-02T04:50:27.443311+0000","last_peered":"2024-04-02T04:50:27.443311+0000","last_clean":"2024-04-02T04:50:27.443311+0000","last_became_active":"2024-04-02T04:50:27.421547+0000","last_became_peered":"2024-04-02T04:50:27.421547+0000","last_unstale":"2024-04-02T04:50:27.443311+0000","last_undegraded":"2024-04-02T04:50:27.443311+0000","last_fullsized":"2024-04-02T04:50:27.443311+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:42.282712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"175.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.14","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.433201+0000","last_change":"2024-04-02T04:50:27.425118+0000","last_active":"2024-04-02T04:50:27.433201+0000","last_peered":"2024-04-02T04:50:27.433201+0000","last_clean":"2024-04-02T04:50:27.433201+0000","last_became_active":"2024-04-02T04:50:27.424880+0000","last_became_peered":"2024-04-02T04:50:27.424880+0000","last_unstale":"2024-04-02T04:50:27.433201+0000","last_undegraded":"2024-04-02T04:50:27.433201+0000","last_fullsized":"2024-04-02T04:50:27.433201+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.088897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"175.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.15","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.436672+0000","last_change":"2024-04-02T04:50:27.420080+0000","last_active":"2024-04-02T04:50:27.436672+0000","last_peered":"2024-04-02T04:50:27.436672+0000","last_clean":"2024-04-02T04:50:27.436672+0000","last_became_active":"2024-04-02T04:50:27.419484+0000","last_became_peered":"2024-04-02T04:50:27.419484+0000","last_unstale":"2024-04-02T04:50:27.436672+0000","last_undegraded":"2024-04-02T04:50:27.436672+0000","last_fullsized":"2024-04-02T04:50:27.436672+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.522191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"175.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.a","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.406582+0000","last_change":"2024-04-02T04:50:27.403868+0000","last_active":"2024-04-02T04:50:27.406582+0000","last_peered":"2024-04-02T04:50:27.406582+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.403868+0000","last_became_peered":"2024-04-02T04:50:27.403868+0000","last_unstale":"2024-04-02T04:50:27.406582+0000","last_undegraded":"2024-04-02T04:50:27.406582+0000","last_fullsized":"2024-04-02T04:50:27.367349+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:37.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"175.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.b","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.463373+0000","last_change":"2024-04-02T04:50:27.460488+0000","last_active":"2024-04-02T04:50:27.463373+0000","last_peered":"2024-04-02T04:50:27.463373+0000","last_clean":"2024-04-02T04:50:27.463373+0000","last_became_active":"2024-04-02T04:50:27.460175+0000","last_became_peered":"2024-04-02T04:50:27.460175+0000","last_unstale":"2024-04-02T04:50:27.463373+0000","last_undegraded":"2024-04-02T04:50:27.463373+0000","last_fullsized":"2024-04-02T04:50:27.463373+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.735077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"175.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.8","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.441135+0000","last_change":"2024-04-02T04:50:27.440712+0000","last_active":"2024-04-02T04:50:27.441135+0000","last_peered":"2024-04-02T04:50:27.441135+0000","last_clean":"2024-04-02T04:50:27.441135+0000","last_became_active":"2024-04-02T04:50:27.440344+0000","last_became_peered":"2024-04-02T04:50:27.440344+0000","last_unstale":"2024-04-02T04:50:27.441135+0000","last_undegraded":"2024-04-02T04:50:27.441135+0000","last_fullsized":"2024-04-02T04:50:27.441135+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:41.539256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"175.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.9","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.433738+0000","last_change":"2024-04-02T04:50:27.429874+0000","last_active":"2024-04-02T04:50:27.433738+0000","last_peered":"2024-04-02T04:50:27.433738+0000","last_clean":"2024-04-02T04:50:27.433738+0000","last_became_active":"2024-04-02T04:50:27.429570+0000","last_became_peered":"2024-04-02T04:50:27.429570+0000","last_unstale":"2024-04-02T04:50:27.433738+0000","last_undegraded":"2024-04-02T04:50:27.433738+0000","last_fullsized":"2024-04-02T04:50:27.433738+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:38.908722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"175.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.e","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.435895+0000","last_change":"2024-04-02T04:50:27.432208+0000","last_active":"2024-04-02T04:50:27.435895+0000","last_peered":"2024-04-02T04:50:27.435895+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.432208+0000","last_became_peered":"2024-04-02T04:50:27.432208+0000","last_unstale":"2024-04-02T04:50:27.435895+0000","last_undegraded":"2024-04-02T04:50:27.435895+0000","last_fullsized":"2024-04-02T04:50:27.373021+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.017906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"175.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.f","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.447600+0000","last_change":"2024-04-02T04:50:27.445683+0000","last_active":"2024-04-02T04:50:27.447600+0000","last_peered":"2024-04-02T04:50:27.447600+0000","last_clean":"2024-04-02T04:50:27.447600+0000","last_became_active":"2024-04-02T04:50:27.433139+0000","last_became_peered":"2024-04-02T04:50:27.433139+0000","last_unstale":"2024-04-02T04:50:27.447600+0000","last_undegraded":"2024-04-02T04:50:27.447600+0000","last_fullsized":"2024-04-02T04:50:27.447600+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:35.826238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"175.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.c","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.422516+0000","last_change":"2024-04-02T04:50:27.422132+0000","last_active":"2024-04-02T04:50:27.422516+0000","last_peered":"2024-04-02T04:50:27.422516+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.422132+0000","last_became_peered":"2024-04-02T04:50:27.422132+0000","last_unstale":"2024-04-02T04:50:27.422516+0000","last_undegraded":"2024-04-02T04:50:27.422516+0000","last_fullsized":"2024-04-02T04:50:27.371044+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:50.861368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"175.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"173.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.d","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.424412+0000","last_change":"2024-04-02T04:50:27.419686+0000","last_active":"2024-04-02T04:50:27.424412+0000","last_peered":"2024-04-02T04:50:27.424412+0000","last_clean":"2024-04-02T04:50:27.424412+0000","last_became_active":"2024-04-02T04:50:27.414906+0000","last_became_peered":"2024-04-02T04:50:27.414906+0000","last_unstale":"2024-04-02T04:50:27.424412+0000","last_undegraded":"2024-04-02T04:50:27.424412+0000","last_fullsized":"2024-04-02T04:50:27.424412+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:54.513468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"175.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.2","version":"0'0","reported_seq":7,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.447322+0000","last_change":"2024-04-02T04:50:27.440268+0000","last_active":"2024-04-02T04:50:27.447322+0000","last_peered":"2024-04-02T04:50:27.447322+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.440268+0000","last_became_peered":"2024-04-02T04:50:27.440268+0000","last_unstale":"2024-04-02T04:50:27.447322+0000","last_undegraded":"2024-04-02T04:50:27.447322+0000","last_fullsized":"2024-04-02T04:50:27.357466+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:54.663787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"175.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.3","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.439814+0000","last_change":"2024-04-02T04:50:27.435113+0000","last_active":"2024-04-02T04:50:27.439814+0000","last_peered":"2024-04-02T04:50:27.439814+0000","last_clean":"2024-04-02T04:50:27.439814+0000","last_became_active":"2024-04-02T04:50:27.434796+0000","last_became_peered":"2024-04-02T04:50:27.434796+0000","last_unstale":"2024-04-02T04:50:27.439814+0000","last_undegraded":"2024-04-02T04:50:27.439814+0000","last_fullsized":"2024-04-02T04:50:27.439814+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.793540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"175.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.0","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.423604+0000","last_change":"2024-04-02T04:50:27.421416+0000","last_active":"2024-04-02T04:50:27.423604+0000","last_peered":"2024-04-02T04:50:27.423604+0000","last_clean":"2024-04-02T04:50:27.423604+0000","last_became_active":"2024-04-02T04:50:27.421222+0000","last_became_peered":"2024-04-02T04:50:27.421222+0000","last_unstale":"2024-04-02T04:50:27.423604+0000","last_undegraded":"2024-04-02T04:50:27.423604+0000","last_fullsized":"2024-04-02T04:50:27.423604+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:55.677418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"175.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.409930+0000","last_change":"2024-04-02T04:50:27.396011+0000","last_active":"2024-04-02T04:50:27.409930+0000","last_peered":"2024-04-02T04:50:27.409930+0000","last_clean":"2024-04-02T04:50:27.409930+0000","last_became_active":"2024-04-02T04:50:27.395102+0000","last_became_peered":"2024-04-02T04:50:27.395102+0000","last_unstale":"2024-04-02T04:50:27.409930+0000","last_undegraded":"2024-04-02T04:50:27.409930+0000","last_fullsized":"2024-04-02T04:50:27.409930+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.523650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"175.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.6","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.433760+0000","last_change":"2024-04-02T04:50:27.428585+0000","last_active":"2024-04-02T04:50:27.433760+0000","last_peered":"2024-04-02T04:50:27.433760+0000","last_clean":"2024-04-02T04:50:27.433760+0000","last_became_active":"2024-04-02T04:50:27.428019+0000","last_became_peered":"2024-04-02T04:50:27.428019+0000","last_unstale":"2024-04-02T04:50:27.433760+0000","last_undegraded":"2024-04-02T04:50:27.433760+0000","last_fullsized":"2024-04-02T04:50:27.433760+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:29.240853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"175.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.7","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.447636+0000","last_change":"2024-04-02T04:50:27.446873+0000","last_active":"2024-04-02T04:50:27.447636+0000","last_peered":"2024-04-02T04:50:27.447636+0000","last_clean":"2024-04-02T04:50:27.447636+0000","last_became_active":"2024-04-02T04:50:27.443081+0000","last_became_peered":"2024-04-02T04:50:27.443081+0000","last_unstale":"2024-04-02T04:50:27.447636+0000","last_undegraded":"2024-04-02T04:50:27.447636+0000","last_fullsized":"2024-04-02T04:50:27.447636+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.035347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"175.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.4","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.438319+0000","last_change":"2024-04-02T04:50:27.436782+0000","last_active":"2024-04-02T04:50:27.438319+0000","last_peered":"2024-04-02T04:50:27.438319+0000","last_clean":"2024-04-02T04:50:27.438319+0000","last_became_active":"2024-04-02T04:50:27.436487+0000","last_became_peered":"2024-04-02T04:50:27.436487+0000","last_unstale":"2024-04-02T04:50:27.438319+0000","last_undegraded":"2024-04-02T04:50:27.438319+0000","last_fullsized":"2024-04-02T04:50:27.438319+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:36.887735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"175.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:32.378694+0000","last_change":"2024-04-02T04:50:32.378694+0000","last_active":"2024-04-02T04:50:32.378694+0000","last_peered":"2024-04-02T04:50:32.378694+0000","last_clean":"2024-04-02T04:50:32.378694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:32.378694+0000","last_undegraded":"2024-04-02T04:50:32.378694+0000","last_fullsized":"2024-04-02T04:50:32.378694+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.378694+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"174.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:50:31.367708+0000","last_change":"2024-04-02T04:50:31.367708+0000","last_active":"2024-04-02T04:50:31.367708+0000","last_peered":"2024-04-02T04:50:31.367708+0000","last_clean":"2024-04-02T04:50:31.367708+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:50:31.367708+0000","last_undegraded":"2024-04-02T04:50:31.367708+0000","last_fullsized":"2024-04-02T04:50:31.367708+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:31.367708+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.367708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.5","version":"0'0","reported_seq":10,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.467771+0000","last_change":"2024-04-02T04:50:27.456107+0000","last_active":"2024-04-02T04:50:27.467771+0000","last_peered":"2024-04-02T04:50:27.467771+0000","last_clean":"2024-04-02T04:50:27.467771+0000","last_became_active":"2024-04-02T04:50:27.455179+0000","last_became_peered":"2024-04-02T04:50:27.455179+0000","last_unstale":"2024-04-02T04:50:27.467771+0000","last_undegraded":"2024-04-02T04:50:27.467771+0000","last_fullsized":"2024-04-02T04:50:27.467771+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:29.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"112.f","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.391513+0000","last_change":"2024-04-02T04:49:59.477750+0000","last_active":"2024-04-02T04:50:27.391513+0000","last_peered":"2024-04-02T04:50:27.391513+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.477750+0000","last_became_peered":"2024-04-02T04:49:59.477750+0000","last_unstale":"2024-04-02T04:50:27.391513+0000","last_undegraded":"2024-04-02T04:50:27.391513+0000","last_fullsized":"2024-04-02T04:49:59.449974+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:59.924640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"102.19","version":"188'5","reported_seq":85,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.547263+0000","last_change":"2024-04-02T04:50:27.532331+0000","last_active":"2024-04-02T04:50:27.547263+0000","last_peered":"2024-04-02T04:50:27.547263+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.255616+0000","last_became_peered":"2024-04-02T04:49:55.255616+0000","last_unstale":"2024-04-02T04:50:27.547263+0000","last_undegraded":"2024-04-02T04:50:27.523866+0000","last_fullsized":"2024-04-02T04:49:55.215309+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:59.929697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"112.e","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.381920+0000","last_change":"2024-04-02T04:50:22.478913+0000","last_active":"2024-04-02T04:50:27.381920+0000","last_peered":"2024-04-02T04:50:27.381920+0000","last_clean":"2024-04-02T04:50:27.381920+0000","last_became_active":"2024-04-02T04:49:59.480145+0000","last_became_peered":"2024-04-02T04:49:59.480145+0000","last_unstale":"2024-04-02T04:50:27.381920+0000","last_undegraded":"2024-04-02T04:50:27.381920+0000","last_fullsized":"2024-04-02T04:50:27.381920+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:09.002483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023371849999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.18","version":"188'2","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.494280+0000","last_change":"2024-04-02T04:50:22.505340+0000","last_active":"2024-04-02T04:50:27.494280+0000","last_peered":"2024-04-02T04:50:27.494280+0000","last_clean":"2024-04-02T04:50:27.494280+0000","last_became_active":"2024-04-02T04:49:56.572185+0000","last_became_peered":"2024-04-02T04:49:56.572185+0000","last_unstale":"2024-04-02T04:50:27.494280+0000","last_undegraded":"2024-04-02T04:50:27.494280+0000","last_fullsized":"2024-04-02T04:50:27.494280+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.232444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058825730000000003,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.d","version":"187'1","reported_seq":80,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.411909+0000","last_change":"2024-04-02T04:50:27.411777+0000","last_active":"2024-04-02T04:50:27.411909+0000","last_peered":"2024-04-02T04:50:27.411909+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.505134+0000","last_became_peered":"2024-04-02T04:49:59.505134+0000","last_unstale":"2024-04-02T04:50:27.411909+0000","last_undegraded":"2024-04-02T04:50:26.457661+0000","last_fullsized":"2024-04-02T04:49:59.425511+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.248072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.1b","version":"188'3","reported_seq":90,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.497701+0000","last_change":"2024-04-02T04:50:22.518367+0000","last_active":"2024-04-02T04:50:27.497701+0000","last_peered":"2024-04-02T04:50:27.497701+0000","last_clean":"2024-04-02T04:50:27.497701+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:50:27.497701+0000","last_undegraded":"2024-04-02T04:50:27.497701+0000","last_fullsized":"2024-04-02T04:50:27.497701+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:58.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083027329999999996,"stat_sum":{"num_bytes":20,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.c","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411935+0000","last_change":"2024-04-02T04:50:22.490674+0000","last_active":"2024-04-02T04:50:27.411935+0000","last_peered":"2024-04-02T04:50:27.411935+0000","last_clean":"2024-04-02T04:50:27.411935+0000","last_became_active":"2024-04-02T04:49:59.507790+0000","last_became_peered":"2024-04-02T04:49:59.507790+0000","last_unstale":"2024-04-02T04:50:27.411935+0000","last_undegraded":"2024-04-02T04:50:27.411935+0000","last_fullsized":"2024-04-02T04:50:27.411935+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:17.093391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030454452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.1a","version":"188'4","reported_seq":90,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.582633+0000","last_change":"2024-04-02T04:50:22.531972+0000","last_active":"2024-04-02T04:50:27.582633+0000","last_peered":"2024-04-02T04:50:27.582633+0000","last_clean":"2024-04-02T04:50:27.582633+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:50:27.582633+0000","last_undegraded":"2024-04-02T04:50:27.582633+0000","last_fullsized":"2024-04-02T04:50:27.582633+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:57.856098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042805730000000002,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.b","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404517+0000","last_change":"2024-04-02T04:50:22.482827+0000","last_active":"2024-04-02T04:50:27.404517+0000","last_peered":"2024-04-02T04:50:27.404517+0000","last_clean":"2024-04-02T04:50:27.404517+0000","last_became_active":"2024-04-02T04:49:59.485124+0000","last_became_peered":"2024-04-02T04:49:59.485124+0000","last_unstale":"2024-04-02T04:50:27.404517+0000","last_undegraded":"2024-04-02T04:50:27.404517+0000","last_fullsized":"2024-04-02T04:50:27.404517+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:17.185582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036518870000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.1d","version":"188'5","reported_seq":91,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.516862+0000","last_change":"2024-04-02T04:50:22.473542+0000","last_active":"2024-04-02T04:50:27.516862+0000","last_peered":"2024-04-02T04:50:27.516862+0000","last_clean":"2024-04-02T04:50:27.516862+0000","last_became_active":"2024-04-02T04:49:56.582887+0000","last_became_peered":"2024-04-02T04:49:56.582887+0000","last_unstale":"2024-04-02T04:50:27.516862+0000","last_undegraded":"2024-04-02T04:50:27.516862+0000","last_fullsized":"2024-04-02T04:50:27.516862+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.118154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049525999999999997,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.a","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.386518+0000","last_change":"2024-04-02T04:50:22.490261+0000","last_active":"2024-04-02T04:50:27.386518+0000","last_peered":"2024-04-02T04:50:27.386518+0000","last_clean":"2024-04-02T04:50:27.386518+0000","last_became_active":"2024-04-02T04:50:00.902590+0000","last_became_peered":"2024-04-02T04:50:00.902590+0000","last_unstale":"2024-04-02T04:50:27.386518+0000","last_undegraded":"2024-04-02T04:50:27.386518+0000","last_fullsized":"2024-04-02T04:50:27.386518+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.002680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055320359999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.1c","version":"188'2","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.478404+0000","last_change":"2024-04-02T04:50:22.559788+0000","last_active":"2024-04-02T04:50:27.478404+0000","last_peered":"2024-04-02T04:50:27.478404+0000","last_clean":"2024-04-02T04:50:27.478404+0000","last_became_active":"2024-04-02T04:49:56.570832+0000","last_became_peered":"2024-04-02T04:49:56.570832+0000","last_unstale":"2024-04-02T04:50:27.478404+0000","last_undegraded":"2024-04-02T04:50:27.478404+0000","last_fullsized":"2024-04-02T04:50:27.478404+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:00.391216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014858387000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.9","version":"167'4","reported_seq":83,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.368389+0000","last_change":"2024-04-02T04:50:02.188165+0000","last_active":"2024-04-02T04:50:27.368389+0000","last_peered":"2024-04-02T04:50:27.368389+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.491429+0000","last_became_peered":"2024-04-02T04:49:59.491429+0000","last_unstale":"2024-04-02T04:50:27.368389+0000","last_undegraded":"2024-04-02T04:50:00.849855+0000","last_fullsized":"2024-04-02T04:49:59.442007+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:25.982922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"102.1f","version":"188'2","reported_seq":89,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.481469+0000","last_change":"2024-04-02T04:50:22.492987+0000","last_active":"2024-04-02T04:50:27.481469+0000","last_peered":"2024-04-02T04:50:27.481469+0000","last_clean":"2024-04-02T04:50:27.481469+0000","last_became_active":"2024-04-02T04:49:55.249289+0000","last_became_peered":"2024-04-02T04:49:55.249289+0000","last_unstale":"2024-04-02T04:50:27.481469+0000","last_undegraded":"2024-04-02T04:50:27.481469+0000","last_fullsized":"2024-04-02T04:50:27.481469+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:11.681581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025262501999999999,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.8","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.362913+0000","last_change":"2024-04-02T04:50:22.467386+0000","last_active":"2024-04-02T04:50:27.362913+0000","last_peered":"2024-04-02T04:50:27.362913+0000","last_clean":"2024-04-02T04:50:27.362913+0000","last_became_active":"2024-04-02T04:49:59.499152+0000","last_became_peered":"2024-04-02T04:49:59.499152+0000","last_unstale":"2024-04-02T04:50:27.362913+0000","last_undegraded":"2024-04-02T04:50:27.362913+0000","last_fullsized":"2024-04-02T04:50:27.362913+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011206226,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.1e","version":"188'1","reported_seq":87,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432187+0000","last_change":"2024-04-02T04:50:22.555071+0000","last_active":"2024-04-02T04:50:27.432187+0000","last_peered":"2024-04-02T04:50:27.432187+0000","last_clean":"2024-04-02T04:50:27.432187+0000","last_became_active":"2024-04-02T04:49:56.563968+0000","last_became_peered":"2024-04-02T04:49:56.563968+0000","last_unstale":"2024-04-02T04:50:27.432187+0000","last_undegraded":"2024-04-02T04:50:27.432187+0000","last_fullsized":"2024-04-02T04:50:27.432187+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:11.875422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026877908999999998,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.7","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.393499+0000","last_change":"2024-04-02T04:49:59.487017+0000","last_active":"2024-04-02T04:50:27.393499+0000","last_peered":"2024-04-02T04:50:27.393499+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.487017+0000","last_became_peered":"2024-04-02T04:49:59.487017+0000","last_unstale":"2024-04-02T04:50:27.393499+0000","last_undegraded":"2024-04-02T04:50:27.393499+0000","last_fullsized":"2024-04-02T04:49:59.435079+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:02.909532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"102.11","version":"188'8","reported_seq":89,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.524006+0000","last_change":"2024-04-02T04:50:18.405897+0000","last_active":"2024-04-02T04:50:27.524006+0000","last_peered":"2024-04-02T04:50:27.524006+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.569001+0000","last_became_peered":"2024-04-02T04:49:56.569001+0000","last_unstale":"2024-04-02T04:50:27.524006+0000","last_undegraded":"2024-04-02T04:50:17.425678+0000","last_fullsized":"2024-04-02T04:49:56.526356+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:03.186292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"112.6","version":"0'0","reported_seq":77,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.406741+0000","last_change":"2024-04-02T04:50:00.928296+0000","last_active":"2024-04-02T04:50:27.406741+0000","last_peered":"2024-04-02T04:50:27.406741+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:50:00.928296+0000","last_became_peered":"2024-04-02T04:50:00.928296+0000","last_unstale":"2024-04-02T04:50:27.406741+0000","last_undegraded":"2024-04-02T04:50:27.406741+0000","last_fullsized":"2024-04-02T04:50:00.811670+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.808524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.10","version":"188'2","reported_seq":82,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.476730+0000","last_change":"2024-04-02T04:50:27.548680+0000","last_active":"2024-04-02T04:50:27.476730+0000","last_peered":"2024-04-02T04:50:27.476730+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.243496+0000","last_became_peered":"2024-04-02T04:49:55.243496+0000","last_unstale":"2024-04-02T04:50:27.476730+0000","last_undegraded":"2024-04-02T04:50:27.476730+0000","last_fullsized":"2024-04-02T04:49:55.211603+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:55.394446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"112.5","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.422835+0000","last_change":"2024-04-02T04:49:59.509070+0000","last_active":"2024-04-02T04:50:27.422835+0000","last_peered":"2024-04-02T04:50:27.422835+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.509070+0000","last_became_peered":"2024-04-02T04:49:59.509070+0000","last_unstale":"2024-04-02T04:50:27.422835+0000","last_undegraded":"2024-04-02T04:50:27.422835+0000","last_fullsized":"2024-04-02T04:49:59.470066+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:09.706550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"102.13","version":"188'8","reported_seq":88,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.562037+0000","last_change":"2024-04-02T04:50:18.418843+0000","last_active":"2024-04-02T04:50:27.562037+0000","last_peered":"2024-04-02T04:50:27.562037+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.554860+0000","last_became_peered":"2024-04-02T04:49:56.554860+0000","last_unstale":"2024-04-02T04:50:27.562037+0000","last_undegraded":"2024-04-02T04:50:17.426824+0000","last_fullsized":"2024-04-02T04:49:56.518531+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.867143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"112.4","version":"182'5","reported_seq":122,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.395765+0000","last_change":"2024-04-02T04:50:22.551893+0000","last_active":"2024-04-02T04:50:27.395765+0000","last_peered":"2024-04-02T04:50:27.395765+0000","last_clean":"2024-04-02T04:50:27.395765+0000","last_became_active":"2024-04-02T04:49:59.508354+0000","last_became_peered":"2024-04-02T04:49:59.508354+0000","last_unstale":"2024-04-02T04:50:27.395765+0000","last_undegraded":"2024-04-02T04:50:27.395765+0000","last_fullsized":"2024-04-02T04:50:27.395765+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.857759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026654245,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.12","version":"188'1","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.386669+0000","last_change":"2024-04-02T04:50:22.490795+0000","last_active":"2024-04-02T04:50:27.386669+0000","last_peered":"2024-04-02T04:50:27.386669+0000","last_clean":"2024-04-02T04:50:27.386669+0000","last_became_active":"2024-04-02T04:49:55.255700+0000","last_became_peered":"2024-04-02T04:49:55.255700+0000","last_unstale":"2024-04-02T04:50:27.386669+0000","last_undegraded":"2024-04-02T04:50:27.386669+0000","last_fullsized":"2024-04-02T04:50:27.386669+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.569012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054572539999999999,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.3","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.401767+0000","last_change":"2024-04-02T04:50:22.476562+0000","last_active":"2024-04-02T04:50:27.401767+0000","last_peered":"2024-04-02T04:50:27.401767+0000","last_clean":"2024-04-02T04:50:27.401767+0000","last_became_active":"2024-04-02T04:49:59.504159+0000","last_became_peered":"2024-04-02T04:49:59.504159+0000","last_unstale":"2024-04-02T04:50:27.401767+0000","last_undegraded":"2024-04-02T04:50:27.401767+0000","last_fullsized":"2024-04-02T04:50:27.401767+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:05.638224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01602688,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.15","version":"188'5","reported_seq":86,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.504239+0000","last_change":"2024-04-02T04:50:09.415659+0000","last_active":"2024-04-02T04:50:27.504239+0000","last_peered":"2024-04-02T04:50:27.504239+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.561896+0000","last_became_peered":"2024-04-02T04:49:56.561896+0000","last_unstale":"2024-04-02T04:50:27.504239+0000","last_undegraded":"2024-04-02T04:50:08.454406+0000","last_fullsized":"2024-04-02T04:49:56.542252+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:21.920813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"112.2","version":"0'0","reported_seq":77,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.396904+0000","last_change":"2024-04-02T04:50:00.913752+0000","last_active":"2024-04-02T04:50:27.396904+0000","last_peered":"2024-04-02T04:50:27.396904+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:50:00.913752+0000","last_became_peered":"2024-04-02T04:50:00.913752+0000","last_unstale":"2024-04-02T04:50:27.396904+0000","last_undegraded":"2024-04-02T04:50:27.396904+0000","last_fullsized":"2024-04-02T04:50:00.792602+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:13.844697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.14","version":"188'9","reported_seq":96,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.569655+0000","last_change":"2024-04-02T04:50:22.458516+0000","last_active":"2024-04-02T04:50:27.569655+0000","last_peered":"2024-04-02T04:50:27.569655+0000","last_clean":"2024-04-02T04:50:27.569655+0000","last_became_active":"2024-04-02T04:49:55.269267+0000","last_became_peered":"2024-04-02T04:49:55.269267+0000","last_unstale":"2024-04-02T04:50:27.569655+0000","last_undegraded":"2024-04-02T04:50:27.569655+0000","last_fullsized":"2024-04-02T04:50:27.569655+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:08.571913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037804760000000001,"stat_sum":{"num_bytes":72,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.370314+0000","last_change":"2024-04-02T04:50:22.514475+0000","last_active":"2024-04-02T04:50:27.370314+0000","last_peered":"2024-04-02T04:50:27.370314+0000","last_clean":"2024-04-02T04:50:27.370314+0000","last_became_active":"2024-04-02T04:49:59.477101+0000","last_became_peered":"2024-04-02T04:49:59.477101+0000","last_unstale":"2024-04-02T04:50:27.370314+0000","last_undegraded":"2024-04-02T04:50:27.370314+0000","last_fullsized":"2024-04-02T04:50:27.370314+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:00.549845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016813317000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.17","version":"188'2","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.539593+0000","last_change":"2024-04-02T04:50:22.556038+0000","last_active":"2024-04-02T04:50:27.539593+0000","last_peered":"2024-04-02T04:50:27.539593+0000","last_clean":"2024-04-02T04:50:27.539593+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:50:27.539593+0000","last_undegraded":"2024-04-02T04:50:27.539593+0000","last_fullsized":"2024-04-02T04:50:27.539593+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:57.716650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015217125,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.0","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.385845+0000","last_change":"2024-04-02T04:50:22.510756+0000","last_active":"2024-04-02T04:50:27.385845+0000","last_peered":"2024-04-02T04:50:27.385845+0000","last_clean":"2024-04-02T04:50:27.385845+0000","last_became_active":"2024-04-02T04:49:59.485524+0000","last_became_peered":"2024-04-02T04:49:59.485524+0000","last_unstale":"2024-04-02T04:50:27.385845+0000","last_undegraded":"2024-04-02T04:50:27.385845+0000","last_fullsized":"2024-04-02T04:50:27.385845+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.154039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015095956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.16","version":"188'1","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.387516+0000","last_change":"2024-04-02T04:50:22.465703+0000","last_active":"2024-04-02T04:50:27.387516+0000","last_peered":"2024-04-02T04:50:27.387516+0000","last_clean":"2024-04-02T04:50:27.387516+0000","last_became_active":"2024-04-02T04:49:55.269224+0000","last_became_peered":"2024-04-02T04:49:55.269224+0000","last_unstale":"2024-04-02T04:50:27.387516+0000","last_undegraded":"2024-04-02T04:50:27.387516+0000","last_fullsized":"2024-04-02T04:50:27.387516+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:15.241463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078454270000000003,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1f","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.418193+0000","last_change":"2024-04-02T04:50:22.481531+0000","last_active":"2024-04-02T04:50:27.418193+0000","last_peered":"2024-04-02T04:50:27.418193+0000","last_clean":"2024-04-02T04:50:27.418193+0000","last_became_active":"2024-04-02T04:50:00.922141+0000","last_became_peered":"2024-04-02T04:50:00.922141+0000","last_unstale":"2024-04-02T04:50:27.418193+0000","last_undegraded":"2024-04-02T04:50:27.418193+0000","last_fullsized":"2024-04-02T04:50:27.418193+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:20.803586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053604409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.9","version":"188'2","reported_seq":81,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.479688+0000","last_change":"2024-04-02T04:50:27.575689+0000","last_active":"2024-04-02T04:50:27.479688+0000","last_peered":"2024-04-02T04:50:27.479688+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.562137+0000","last_became_peered":"2024-04-02T04:49:56.562137+0000","last_unstale":"2024-04-02T04:50:27.479688+0000","last_undegraded":"2024-04-02T04:50:27.479688+0000","last_fullsized":"2024-04-02T04:49:56.524836+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.224801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"112.1e","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.449886+0000","last_change":"2024-04-02T04:49:59.477351+0000","last_active":"2024-04-02T04:50:27.449886+0000","last_peered":"2024-04-02T04:50:27.449886+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.477351+0000","last_became_peered":"2024-04-02T04:49:59.477351+0000","last_unstale":"2024-04-02T04:50:27.449886+0000","last_undegraded":"2024-04-02T04:50:27.449886+0000","last_fullsized":"2024-04-02T04:49:59.450534+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.801855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.8","version":"188'1","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.395715+0000","last_change":"2024-04-02T04:50:22.511423+0000","last_active":"2024-04-02T04:50:27.395715+0000","last_peered":"2024-04-02T04:50:27.395715+0000","last_clean":"2024-04-02T04:50:27.395715+0000","last_became_active":"2024-04-02T04:49:55.259553+0000","last_became_peered":"2024-04-02T04:49:55.259553+0000","last_unstale":"2024-04-02T04:50:27.395715+0000","last_undegraded":"2024-04-02T04:50:27.395715+0000","last_fullsized":"2024-04-02T04:50:27.395715+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:01.164871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012597687999999999,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1d","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.433606+0000","last_change":"2024-04-02T04:50:22.505311+0000","last_active":"2024-04-02T04:50:27.433606+0000","last_peered":"2024-04-02T04:50:27.433606+0000","last_clean":"2024-04-02T04:50:27.433606+0000","last_became_active":"2024-04-02T04:49:59.503670+0000","last_became_peered":"2024-04-02T04:49:59.503670+0000","last_unstale":"2024-04-02T04:50:27.433606+0000","last_undegraded":"2024-04-02T04:50:27.433606+0000","last_fullsized":"2024-04-02T04:50:27.433606+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.277158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021245564000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.b","version":"188'4","reported_seq":91,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.546144+0000","last_change":"2024-04-02T04:50:22.478126+0000","last_active":"2024-04-02T04:50:27.546144+0000","last_peered":"2024-04-02T04:50:27.546144+0000","last_clean":"2024-04-02T04:50:27.546144+0000","last_became_active":"2024-04-02T04:49:55.267102+0000","last_became_peered":"2024-04-02T04:49:55.267102+0000","last_unstale":"2024-04-02T04:50:27.546144+0000","last_undegraded":"2024-04-02T04:50:27.546144+0000","last_fullsized":"2024-04-02T04:50:27.546144+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.180615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01285824,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1c","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.395037+0000","last_change":"2024-04-02T04:49:59.504008+0000","last_active":"2024-04-02T04:50:27.395037+0000","last_peered":"2024-04-02T04:50:27.395037+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:49:59.504008+0000","last_became_peered":"2024-04-02T04:49:59.504008+0000","last_unstale":"2024-04-02T04:50:27.395037+0000","last_undegraded":"2024-04-02T04:50:27.395037+0000","last_fullsized":"2024-04-02T04:49:59.416156+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:12.364842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.a","version":"188'4","reported_seq":91,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.515503+0000","last_change":"2024-04-02T04:50:22.479090+0000","last_active":"2024-04-02T04:50:27.515503+0000","last_peered":"2024-04-02T04:50:27.515503+0000","last_clean":"2024-04-02T04:50:27.515503+0000","last_became_active":"2024-04-02T04:49:55.252341+0000","last_became_peered":"2024-04-02T04:49:55.252341+0000","last_unstale":"2024-04-02T04:50:27.515503+0000","last_undegraded":"2024-04-02T04:50:27.515503+0000","last_fullsized":"2024-04-02T04:50:27.515503+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.894636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058763110000000004,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.1b","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.403514+0000","last_change":"2024-04-02T04:50:22.551425+0000","last_active":"2024-04-02T04:50:27.403514+0000","last_peered":"2024-04-02T04:50:27.403514+0000","last_clean":"2024-04-02T04:50:27.403514+0000","last_became_active":"2024-04-02T04:49:59.499859+0000","last_became_peered":"2024-04-02T04:49:59.499859+0000","last_unstale":"2024-04-02T04:50:27.403514+0000","last_undegraded":"2024-04-02T04:50:27.403514+0000","last_fullsized":"2024-04-02T04:50:27.403514+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.782786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034261475,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.d","version":"188'7","reported_seq":86,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.579951+0000","last_change":"2024-04-02T04:50:27.508913+0000","last_active":"2024-04-02T04:50:27.579951+0000","last_peered":"2024-04-02T04:50:27.579951+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.566448+0000","last_became_peered":"2024-04-02T04:49:56.566448+0000","last_unstale":"2024-04-02T04:50:27.579951+0000","last_undegraded":"2024-04-02T04:50:27.492108+0000","last_fullsized":"2024-04-02T04:49:56.525265+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:03.656097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"112.1a","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.398560+0000","last_change":"2024-04-02T04:50:22.534031+0000","last_active":"2024-04-02T04:50:27.398560+0000","last_peered":"2024-04-02T04:50:27.398560+0000","last_clean":"2024-04-02T04:50:27.398560+0000","last_became_active":"2024-04-02T04:50:00.914382+0000","last_became_peered":"2024-04-02T04:50:00.914382+0000","last_unstale":"2024-04-02T04:50:27.398560+0000","last_undegraded":"2024-04-02T04:50:27.398560+0000","last_fullsized":"2024-04-02T04:50:27.398560+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:01.458212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005912507,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.c","version":"188'4","reported_seq":91,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.502634+0000","last_change":"2024-04-02T04:50:22.514035+0000","last_active":"2024-04-02T04:50:27.502634+0000","last_peered":"2024-04-02T04:50:27.502634+0000","last_clean":"2024-04-02T04:50:27.502634+0000","last_became_active":"2024-04-02T04:49:55.252273+0000","last_became_peered":"2024-04-02T04:49:55.252273+0000","last_unstale":"2024-04-02T04:50:27.502634+0000","last_undegraded":"2024-04-02T04:50:27.502634+0000","last_fullsized":"2024-04-02T04:50:27.502634+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.487760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011904270999999999,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.19","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.383923+0000","last_change":"2024-04-02T04:50:22.491273+0000","last_active":"2024-04-02T04:50:27.383923+0000","last_peered":"2024-04-02T04:50:27.383923+0000","last_clean":"2024-04-02T04:50:27.383923+0000","last_became_active":"2024-04-02T04:50:00.922789+0000","last_became_peered":"2024-04-02T04:50:00.922789+0000","last_unstale":"2024-04-02T04:50:27.383923+0000","last_undegraded":"2024-04-02T04:50:27.383923+0000","last_fullsized":"2024-04-02T04:50:27.383923+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:05.617750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052873390000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.f","version":"188'7","reported_seq":95,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.534583+0000","last_change":"2024-04-02T04:50:24.531979+0000","last_active":"2024-04-02T04:50:27.534583+0000","last_peered":"2024-04-02T04:50:27.534583+0000","last_clean":"2024-04-02T04:50:27.534583+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:50:27.534583+0000","last_undegraded":"2024-04-02T04:50:27.534583+0000","last_fullsized":"2024-04-02T04:50:27.534583+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:16.474141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.54930396999999997,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.18","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.387879+0000","last_change":"2024-04-02T04:50:22.495006+0000","last_active":"2024-04-02T04:50:27.387879+0000","last_peered":"2024-04-02T04:50:27.387879+0000","last_clean":"2024-04-02T04:50:27.387879+0000","last_became_active":"2024-04-02T04:49:59.507112+0000","last_became_peered":"2024-04-02T04:49:59.507112+0000","last_unstale":"2024-04-02T04:50:27.387879+0000","last_undegraded":"2024-04-02T04:50:27.387879+0000","last_fullsized":"2024-04-02T04:50:27.387879+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:24.964723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033680900999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.e","version":"188'8","reported_seq":96,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.495344+0000","last_change":"2024-04-02T04:50:22.556197+0000","last_active":"2024-04-02T04:50:27.495344+0000","last_peered":"2024-04-02T04:50:27.495344+0000","last_clean":"2024-04-02T04:50:27.495344+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:50:27.495344+0000","last_undegraded":"2024-04-02T04:50:27.495344+0000","last_fullsized":"2024-04-02T04:50:27.495344+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:20.219023+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.048399875000000002,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.17","version":"171'9","reported_seq":132,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.437914+0000","last_change":"2024-04-02T04:50:22.503107+0000","last_active":"2024-04-02T04:50:27.437914+0000","last_peered":"2024-04-02T04:50:27.437914+0000","last_clean":"2024-04-02T04:50:27.437914+0000","last_became_active":"2024-04-02T04:49:59.505203+0000","last_became_peered":"2024-04-02T04:49:59.505203+0000","last_unstale":"2024-04-02T04:50:27.437914+0000","last_undegraded":"2024-04-02T04:50:27.437914+0000","last_fullsized":"2024-04-02T04:50:27.437914+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:24.104805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023060961000000001,"stat_sum":{"num_bytes":384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.1","version":"188'3","reported_seq":83,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.514211+0000","last_change":"2024-04-02T04:50:27.514211+0000","last_active":"2024-04-02T04:50:27.514211+0000","last_peered":"2024-04-02T04:50:27.514211+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.245537+0000","last_became_peered":"2024-04-02T04:49:55.245537+0000","last_unstale":"2024-04-02T04:50:27.514211+0000","last_undegraded":"2024-04-02T04:50:27.498272+0000","last_fullsized":"2024-04-02T04:49:55.215487+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.399956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":3}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"112.16","version":"186'4","reported_seq":80,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.431874+0000","last_change":"2024-04-02T04:50:24.023253+0000","last_active":"2024-04-02T04:50:27.431874+0000","last_peered":"2024-04-02T04:50:27.431874+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:50:00.916125+0000","last_became_peered":"2024-04-02T04:50:00.916125+0000","last_unstale":"2024-04-02T04:50:27.431874+0000","last_undegraded":"2024-04-02T04:50:23.537660+0000","last_fullsized":"2024-04-02T04:50:00.809923+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.301543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.0","version":"188'5","reported_seq":92,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.587755+0000","last_change":"2024-04-02T04:50:22.510383+0000","last_active":"2024-04-02T04:50:27.587755+0000","last_peered":"2024-04-02T04:50:27.587755+0000","last_clean":"2024-04-02T04:50:27.587755+0000","last_became_active":"2024-04-02T04:49:55.268700+0000","last_became_peered":"2024-04-02T04:49:55.268700+0000","last_unstale":"2024-04-02T04:50:27.587755+0000","last_undegraded":"2024-04-02T04:50:27.587755+0000","last_fullsized":"2024-04-02T04:50:27.587755+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:15.862857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019278297,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.15","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377377+0000","last_change":"2024-04-02T04:50:22.475257+0000","last_active":"2024-04-02T04:50:27.377377+0000","last_peered":"2024-04-02T04:50:27.377377+0000","last_clean":"2024-04-02T04:50:27.377377+0000","last_became_active":"2024-04-02T04:49:59.507091+0000","last_became_peered":"2024-04-02T04:49:59.507091+0000","last_unstale":"2024-04-02T04:50:27.377377+0000","last_undegraded":"2024-04-02T04:50:27.377377+0000","last_fullsized":"2024-04-02T04:50:27.377377+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:24.338942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099931299999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.3","version":"188'2","reported_seq":89,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.507792+0000","last_change":"2024-04-02T04:50:22.482285+0000","last_active":"2024-04-02T04:50:27.507792+0000","last_peered":"2024-04-02T04:50:27.507792+0000","last_clean":"2024-04-02T04:50:27.507792+0000","last_became_active":"2024-04-02T04:49:55.257987+0000","last_became_peered":"2024-04-02T04:49:55.257987+0000","last_unstale":"2024-04-02T04:50:27.507792+0000","last_undegraded":"2024-04-02T04:50:27.507792+0000","last_fullsized":"2024-04-02T04:50:27.507792+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.072656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054451289999999999,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.14","version":"0'0","reported_seq":77,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.410011+0000","last_change":"2024-04-02T04:50:00.903763+0000","last_active":"2024-04-02T04:50:27.410011+0000","last_peered":"2024-04-02T04:50:27.410011+0000","last_clean":"2024-04-02T04:49:58.380085+0000","last_became_active":"2024-04-02T04:50:00.903763+0000","last_became_peered":"2024-04-02T04:50:00.903763+0000","last_unstale":"2024-04-02T04:50:27.410011+0000","last_undegraded":"2024-04-02T04:50:27.410011+0000","last_fullsized":"2024-04-02T04:50:00.796098+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.717159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.2","version":"188'2","reported_seq":89,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.552411+0000","last_change":"2024-04-02T04:50:22.491445+0000","last_active":"2024-04-02T04:50:27.552411+0000","last_peered":"2024-04-02T04:50:27.552411+0000","last_clean":"2024-04-02T04:50:27.552411+0000","last_became_active":"2024-04-02T04:49:55.235417+0000","last_became_peered":"2024-04-02T04:49:55.235417+0000","last_unstale":"2024-04-02T04:50:27.552411+0000","last_undegraded":"2024-04-02T04:50:27.552411+0000","last_fullsized":"2024-04-02T04:50:27.552411+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:59.585712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058798909999999999,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.13","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.401272+0000","last_change":"2024-04-02T04:50:22.474796+0000","last_active":"2024-04-02T04:50:27.401272+0000","last_peered":"2024-04-02T04:50:27.401272+0000","last_clean":"2024-04-02T04:50:27.401272+0000","last_became_active":"2024-04-02T04:49:59.492007+0000","last_became_peered":"2024-04-02T04:49:59.492007+0000","last_unstale":"2024-04-02T04:50:27.401272+0000","last_undegraded":"2024-04-02T04:50:27.401272+0000","last_fullsized":"2024-04-02T04:50:27.401272+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.926339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016278107999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.5","version":"188'5","reported_seq":91,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.558201+0000","last_change":"2024-04-02T04:50:22.537559+0000","last_active":"2024-04-02T04:50:27.558201+0000","last_peered":"2024-04-02T04:50:27.558201+0000","last_clean":"2024-04-02T04:50:27.558201+0000","last_became_active":"2024-04-02T04:49:56.553311+0000","last_became_peered":"2024-04-02T04:49:56.553311+0000","last_unstale":"2024-04-02T04:50:27.558201+0000","last_undegraded":"2024-04-02T04:50:27.558201+0000","last_fullsized":"2024-04-02T04:50:27.558201+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:17.705383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022060244999999999,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.12","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.417845+0000","last_change":"2024-04-02T04:50:22.486279+0000","last_active":"2024-04-02T04:50:27.417845+0000","last_peered":"2024-04-02T04:50:27.417845+0000","last_clean":"2024-04-02T04:50:27.417845+0000","last_became_active":"2024-04-02T04:49:59.487169+0000","last_became_peered":"2024-04-02T04:49:59.487169+0000","last_unstale":"2024-04-02T04:50:27.417845+0000","last_undegraded":"2024-04-02T04:50:27.417845+0000","last_fullsized":"2024-04-02T04:50:27.417845+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:12.206885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034628126000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.4","version":"188'1","reported_seq":80,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.405580+0000","last_change":"2024-04-02T04:49:56.578685+0000","last_active":"2024-04-02T04:50:27.405580+0000","last_peered":"2024-04-02T04:50:27.405580+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.578685+0000","last_became_peered":"2024-04-02T04:49:56.578685+0000","last_unstale":"2024-04-02T04:50:27.405580+0000","last_undegraded":"2024-04-02T04:50:27.405580+0000","last_fullsized":"2024-04-02T04:49:56.549435+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.766543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"112.11","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.423347+0000","last_change":"2024-04-02T04:50:22.484878+0000","last_active":"2024-04-02T04:50:27.423347+0000","last_peered":"2024-04-02T04:50:27.423347+0000","last_clean":"2024-04-02T04:50:27.423347+0000","last_became_active":"2024-04-02T04:49:59.486764+0000","last_became_peered":"2024-04-02T04:49:59.486764+0000","last_unstale":"2024-04-02T04:50:27.423347+0000","last_undegraded":"2024-04-02T04:50:27.423347+0000","last_fullsized":"2024-04-02T04:50:27.423347+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:09.629970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033619405999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.7","version":"188'1","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.419753+0000","last_change":"2024-04-02T04:50:22.482989+0000","last_active":"2024-04-02T04:50:27.419753+0000","last_peered":"2024-04-02T04:50:27.419753+0000","last_clean":"2024-04-02T04:50:27.419753+0000","last_became_active":"2024-04-02T04:49:55.261913+0000","last_became_peered":"2024-04-02T04:49:55.261913+0000","last_unstale":"2024-04-02T04:50:27.419753+0000","last_undegraded":"2024-04-02T04:50:27.419753+0000","last_fullsized":"2024-04-02T04:50:27.419753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:56.924275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044479259999999996,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"112.10","version":"178'20","reported_seq":139,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.410614+0000","last_change":"2024-04-02T04:50:22.494952+0000","last_active":"2024-04-02T04:50:27.410614+0000","last_peered":"2024-04-02T04:50:27.410614+0000","last_clean":"2024-04-02T04:50:27.410614+0000","last_became_active":"2024-04-02T04:49:59.499751+0000","last_became_peered":"2024-04-02T04:49:59.499751+0000","last_unstale":"2024-04-02T04:50:27.410614+0000","last_undegraded":"2024-04-02T04:50:27.410614+0000","last_fullsized":"2024-04-02T04:50:27.410614+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:58.380085+0000","last_clean_scrub_stamp":"2024-04-02T04:49:58.380085+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:12.362853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098097919999999995,"stat_sum":{"num_bytes":1280,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"9"}]},{"pgid":"102.6","version":"188'2","reported_seq":88,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.515887+0000","last_change":"2024-04-02T04:50:22.479243+0000","last_active":"2024-04-02T04:50:27.515887+0000","last_peered":"2024-04-02T04:50:27.515887+0000","last_clean":"2024-04-02T04:50:27.515887+0000","last_became_active":"2024-04-02T04:49:56.581977+0000","last_became_peered":"2024-04-02T04:49:56.581977+0000","last_unstale":"2024-04-02T04:50:27.515887+0000","last_undegraded":"2024-04-02T04:50:27.515887+0000","last_fullsized":"2024-04-02T04:50:27.515887+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:15.059141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025991510000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"43.14","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377419+0000","last_change":"2024-04-02T04:49:29.828822+0000","last_active":"2024-04-02T04:50:27.377419+0000","last_peered":"2024-04-02T04:50:27.377419+0000","last_clean":"2024-04-02T04:50:27.377419+0000","last_became_active":"2024-04-02T04:49:29.828535+0000","last_became_peered":"2024-04-02T04:49:29.828535+0000","last_unstale":"2024-04-02T04:50:27.377419+0000","last_undegraded":"2024-04-02T04:50:27.377419+0000","last_fullsized":"2024-04-02T04:50:27.377419+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.418991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.427772+0000","last_change":"2024-04-02T04:49:47.428191+0000","last_active":"2024-04-02T04:50:27.427772+0000","last_peered":"2024-04-02T04:50:27.427772+0000","last_clean":"2024-04-02T04:49:45.347176+0000","last_became_active":"2024-04-02T04:49:47.428191+0000","last_became_peered":"2024-04-02T04:49:47.428191+0000","last_unstale":"2024-04-02T04:50:27.427772+0000","last_undegraded":"2024-04-02T04:50:27.427772+0000","last_fullsized":"2024-04-02T04:49:47.367526+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:49.779207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.441875+0000","last_change":"2024-04-02T04:49:28.771791+0000","last_active":"2024-04-02T04:50:27.441875+0000","last_peered":"2024-04-02T04:50:27.441875+0000","last_clean":"2024-04-02T04:50:27.441875+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:50:27.441875+0000","last_undegraded":"2024-04-02T04:50:27.441875+0000","last_fullsized":"2024-04-02T04:50:27.441875+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.806079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428273+0000","last_change":"2024-04-02T04:49:28.787801+0000","last_active":"2024-04-02T04:50:27.428273+0000","last_peered":"2024-04-02T04:50:27.428273+0000","last_clean":"2024-04-02T04:50:27.428273+0000","last_became_active":"2024-04-02T04:49:28.787404+0000","last_became_peered":"2024-04-02T04:49:28.787404+0000","last_unstale":"2024-04-02T04:50:27.428273+0000","last_undegraded":"2024-04-02T04:50:27.428273+0000","last_fullsized":"2024-04-02T04:50:27.428273+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.441768+0000","last_change":"2024-04-02T04:49:28.787925+0000","last_active":"2024-04-02T04:50:27.441768+0000","last_peered":"2024-04-02T04:50:27.441768+0000","last_clean":"2024-04-02T04:50:27.441768+0000","last_became_active":"2024-04-02T04:49:28.783594+0000","last_became_peered":"2024-04-02T04:49:28.783594+0000","last_unstale":"2024-04-02T04:50:27.441768+0000","last_undegraded":"2024-04-02T04:50:27.441768+0000","last_fullsized":"2024-04-02T04:50:27.441768+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.600394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.408504+0000","last_change":"2024-04-02T04:49:28.726311+0000","last_active":"2024-04-02T04:50:27.408504+0000","last_peered":"2024-04-02T04:50:27.408504+0000","last_clean":"2024-04-02T04:50:27.408504+0000","last_became_active":"2024-04-02T04:49:28.726013+0000","last_became_peered":"2024-04-02T04:49:28.726013+0000","last_unstale":"2024-04-02T04:50:27.408504+0000","last_undegraded":"2024-04-02T04:50:27.408504+0000","last_fullsized":"2024-04-02T04:50:27.408504+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.527996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":110,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.382340+0000","last_change":"2024-04-02T04:49:47.479797+0000","last_active":"2024-04-02T04:50:27.382340+0000","last_peered":"2024-04-02T04:50:27.382340+0000","last_clean":"2024-04-02T04:49:45.330040+0000","last_became_active":"2024-04-02T04:49:47.479797+0000","last_became_peered":"2024-04-02T04:49:47.479797+0000","last_unstale":"2024-04-02T04:50:27.382340+0000","last_undegraded":"2024-04-02T04:50:27.382340+0000","last_fullsized":"2024-04-02T04:49:47.405691+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:54.174610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389776+0000","last_change":"2024-04-02T04:49:28.759806+0000","last_active":"2024-04-02T04:50:27.389776+0000","last_peered":"2024-04-02T04:50:27.389776+0000","last_clean":"2024-04-02T04:50:27.389776+0000","last_became_active":"2024-04-02T04:49:28.759334+0000","last_became_peered":"2024-04-02T04:49:28.759334+0000","last_unstale":"2024-04-02T04:50:27.389776+0000","last_undegraded":"2024-04-02T04:50:27.389776+0000","last_fullsized":"2024-04-02T04:50:27.389776+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:45.096946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.417348+0000","last_change":"2024-04-02T04:49:47.517914+0000","last_active":"2024-04-02T04:50:27.417348+0000","last_peered":"2024-04-02T04:50:27.417348+0000","last_clean":"2024-04-02T04:49:45.379252+0000","last_became_active":"2024-04-02T04:49:47.517914+0000","last_became_peered":"2024-04-02T04:49:47.517914+0000","last_unstale":"2024-04-02T04:50:27.417348+0000","last_undegraded":"2024-04-02T04:50:27.417348+0000","last_fullsized":"2024-04-02T04:49:47.408049+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.639026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.393384+0000","last_change":"2024-04-02T04:49:47.477571+0000","last_active":"2024-04-02T04:50:27.393384+0000","last_peered":"2024-04-02T04:50:27.393384+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.477571+0000","last_became_peered":"2024-04-02T04:49:47.477571+0000","last_unstale":"2024-04-02T04:50:27.393384+0000","last_undegraded":"2024-04-02T04:50:27.393384+0000","last_fullsized":"2024-04-02T04:49:47.404962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:50.428685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.411489+0000","last_change":"2024-04-02T04:49:47.517552+0000","last_active":"2024-04-02T04:50:27.411489+0000","last_peered":"2024-04-02T04:50:27.411489+0000","last_clean":"2024-04-02T04:49:45.378670+0000","last_became_active":"2024-04-02T04:49:47.517552+0000","last_became_peered":"2024-04-02T04:49:47.517552+0000","last_unstale":"2024-04-02T04:50:27.411489+0000","last_undegraded":"2024-04-02T04:50:27.411489+0000","last_fullsized":"2024-04-02T04:49:47.406289+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.423452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.438579+0000","last_change":"2024-04-02T04:49:28.761881+0000","last_active":"2024-04-02T04:50:27.438579+0000","last_peered":"2024-04-02T04:50:27.438579+0000","last_clean":"2024-04-02T04:50:27.438579+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:50:27.438579+0000","last_undegraded":"2024-04-02T04:50:27.438579+0000","last_fullsized":"2024-04-02T04:50:27.438579+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:50.082959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.367123+0000","last_change":"2024-04-02T04:49:29.819002+0000","last_active":"2024-04-02T04:50:27.367123+0000","last_peered":"2024-04-02T04:50:27.367123+0000","last_clean":"2024-04-02T04:50:27.367123+0000","last_became_active":"2024-04-02T04:49:29.818524+0000","last_became_peered":"2024-04-02T04:49:29.818524+0000","last_unstale":"2024-04-02T04:50:27.367123+0000","last_undegraded":"2024-04-02T04:50:27.367123+0000","last_fullsized":"2024-04-02T04:50:27.367123+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.271364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.407504+0000","last_change":"2024-04-02T04:49:28.769390+0000","last_active":"2024-04-02T04:50:27.407504+0000","last_peered":"2024-04-02T04:50:27.407504+0000","last_clean":"2024-04-02T04:50:27.407504+0000","last_became_active":"2024-04-02T04:49:28.769072+0000","last_became_peered":"2024-04-02T04:49:28.769072+0000","last_unstale":"2024-04-02T04:50:27.407504+0000","last_undegraded":"2024-04-02T04:50:27.407504+0000","last_fullsized":"2024-04-02T04:50:27.407504+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.314034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.370140+0000","last_change":"2024-04-02T04:49:29.802333+0000","last_active":"2024-04-02T04:50:27.370140+0000","last_peered":"2024-04-02T04:50:27.370140+0000","last_clean":"2024-04-02T04:50:27.370140+0000","last_became_active":"2024-04-02T04:49:29.801654+0000","last_became_peered":"2024-04-02T04:49:29.801654+0000","last_unstale":"2024-04-02T04:50:27.370140+0000","last_undegraded":"2024-04-02T04:50:27.370140+0000","last_fullsized":"2024-04-02T04:50:27.370140+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.493192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.366363+0000","last_change":"2024-04-02T04:49:28.760456+0000","last_active":"2024-04-02T04:50:27.366363+0000","last_peered":"2024-04-02T04:50:27.366363+0000","last_clean":"2024-04-02T04:50:27.366363+0000","last_became_active":"2024-04-02T04:49:28.756548+0000","last_became_peered":"2024-04-02T04:49:28.756548+0000","last_unstale":"2024-04-02T04:50:27.366363+0000","last_undegraded":"2024-04-02T04:50:27.366363+0000","last_fullsized":"2024-04-02T04:50:27.366363+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:55.623518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.420804+0000","last_change":"2024-04-02T04:49:28.741572+0000","last_active":"2024-04-02T04:50:27.420804+0000","last_peered":"2024-04-02T04:50:27.420804+0000","last_clean":"2024-04-02T04:50:27.420804+0000","last_became_active":"2024-04-02T04:49:28.741277+0000","last_became_peered":"2024-04-02T04:49:28.741277+0000","last_unstale":"2024-04-02T04:50:27.420804+0000","last_undegraded":"2024-04-02T04:50:27.420804+0000","last_fullsized":"2024-04-02T04:50:27.420804+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.201349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.376336+0000","last_change":"2024-04-02T04:49:28.740669+0000","last_active":"2024-04-02T04:50:27.376336+0000","last_peered":"2024-04-02T04:50:27.376336+0000","last_clean":"2024-04-02T04:50:27.376336+0000","last_became_active":"2024-04-02T04:49:28.719086+0000","last_became_peered":"2024-04-02T04:49:28.719086+0000","last_unstale":"2024-04-02T04:50:27.376336+0000","last_undegraded":"2024-04-02T04:50:27.376336+0000","last_fullsized":"2024-04-02T04:50:27.376336+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389701+0000","last_change":"2024-04-02T04:49:28.743542+0000","last_active":"2024-04-02T04:50:27.389701+0000","last_peered":"2024-04-02T04:50:27.389701+0000","last_clean":"2024-04-02T04:50:27.389701+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:50:27.389701+0000","last_undegraded":"2024-04-02T04:50:27.389701+0000","last_fullsized":"2024-04-02T04:50:27.389701+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.730554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.371633+0000","last_change":"2024-04-02T04:49:28.732308+0000","last_active":"2024-04-02T04:50:27.371633+0000","last_peered":"2024-04-02T04:50:27.371633+0000","last_clean":"2024-04-02T04:50:27.371633+0000","last_became_active":"2024-04-02T04:49:28.731836+0000","last_became_peered":"2024-04-02T04:49:28.731836+0000","last_unstale":"2024-04-02T04:50:27.371633+0000","last_undegraded":"2024-04-02T04:50:27.371633+0000","last_fullsized":"2024-04-02T04:50:27.371633+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.445708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.376242+0000","last_change":"2024-04-02T04:49:28.737637+0000","last_active":"2024-04-02T04:50:27.376242+0000","last_peered":"2024-04-02T04:50:27.376242+0000","last_clean":"2024-04-02T04:50:27.376242+0000","last_became_active":"2024-04-02T04:49:28.727703+0000","last_became_peered":"2024-04-02T04:49:28.727703+0000","last_unstale":"2024-04-02T04:50:27.376242+0000","last_undegraded":"2024-04-02T04:50:27.376242+0000","last_fullsized":"2024-04-02T04:50:27.376242+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.317806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.398059+0000","last_change":"2024-04-02T04:49:47.517107+0000","last_active":"2024-04-02T04:50:27.398059+0000","last_peered":"2024-04-02T04:50:27.398059+0000","last_clean":"2024-04-02T04:49:45.381150+0000","last_became_active":"2024-04-02T04:49:47.517107+0000","last_became_peered":"2024-04-02T04:49:47.517107+0000","last_unstale":"2024-04-02T04:50:27.398059+0000","last_undegraded":"2024-04-02T04:50:27.398059+0000","last_fullsized":"2024-04-02T04:49:47.429362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:52.136494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.365086+0000","last_change":"2024-04-02T04:49:47.444375+0000","last_active":"2024-04-02T04:50:27.365086+0000","last_peered":"2024-04-02T04:50:27.365086+0000","last_clean":"2024-04-02T04:49:45.331671+0000","last_became_active":"2024-04-02T04:49:47.444375+0000","last_became_peered":"2024-04-02T04:49:47.444375+0000","last_unstale":"2024-04-02T04:50:27.365086+0000","last_undegraded":"2024-04-02T04:50:27.365086+0000","last_fullsized":"2024-04-02T04:49:47.336937+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.040294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.365161+0000","last_change":"2024-04-02T04:49:47.381163+0000","last_active":"2024-04-02T04:50:27.365161+0000","last_peered":"2024-04-02T04:50:27.365161+0000","last_clean":"2024-04-02T04:49:45.330129+0000","last_became_active":"2024-04-02T04:49:47.381163+0000","last_became_peered":"2024-04-02T04:49:47.381163+0000","last_unstale":"2024-04-02T04:50:27.365161+0000","last_undegraded":"2024-04-02T04:50:27.365161+0000","last_fullsized":"2024-04-02T04:49:47.338896+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:49.582556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.444575+0000","last_change":"2024-04-02T04:49:28.767169+0000","last_active":"2024-04-02T04:50:27.444575+0000","last_peered":"2024-04-02T04:50:27.444575+0000","last_clean":"2024-04-02T04:50:27.444575+0000","last_became_active":"2024-04-02T04:49:28.762640+0000","last_became_peered":"2024-04-02T04:49:28.762640+0000","last_unstale":"2024-04-02T04:50:27.444575+0000","last_undegraded":"2024-04-02T04:50:27.444575+0000","last_fullsized":"2024-04-02T04:50:27.444575+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.772239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.452970+0000","last_change":"2024-04-02T04:49:28.768223+0000","last_active":"2024-04-02T04:50:27.452970+0000","last_peered":"2024-04-02T04:50:27.452970+0000","last_clean":"2024-04-02T04:50:27.452970+0000","last_became_active":"2024-04-02T04:49:28.767868+0000","last_became_peered":"2024-04-02T04:49:28.767868+0000","last_unstale":"2024-04-02T04:50:27.452970+0000","last_undegraded":"2024-04-02T04:50:27.452970+0000","last_fullsized":"2024-04-02T04:50:27.452970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:41.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.419496+0000","last_change":"2024-04-02T04:49:47.508050+0000","last_active":"2024-04-02T04:50:27.419496+0000","last_peered":"2024-04-02T04:50:27.419496+0000","last_clean":"2024-04-02T04:49:45.381990+0000","last_became_active":"2024-04-02T04:49:47.508050+0000","last_became_peered":"2024-04-02T04:49:47.508050+0000","last_unstale":"2024-04-02T04:50:27.419496+0000","last_undegraded":"2024-04-02T04:50:27.419496+0000","last_fullsized":"2024-04-02T04:49:47.429001+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.743446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.435013+0000","last_change":"2024-04-02T04:49:47.490495+0000","last_active":"2024-04-02T04:50:27.435013+0000","last_peered":"2024-04-02T04:50:27.435013+0000","last_clean":"2024-04-02T04:49:45.351290+0000","last_became_active":"2024-04-02T04:49:47.490495+0000","last_became_peered":"2024-04-02T04:49:47.490495+0000","last_unstale":"2024-04-02T04:50:27.435013+0000","last_undegraded":"2024-04-02T04:50:27.435013+0000","last_fullsized":"2024-04-02T04:49:47.384086+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:55.485488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.426719+0000","last_change":"2024-04-02T04:49:28.720521+0000","last_active":"2024-04-02T04:50:27.426719+0000","last_peered":"2024-04-02T04:50:27.426719+0000","last_clean":"2024-04-02T04:50:27.426719+0000","last_became_active":"2024-04-02T04:49:28.720246+0000","last_became_peered":"2024-04-02T04:49:28.720246+0000","last_unstale":"2024-04-02T04:50:27.426719+0000","last_undegraded":"2024-04-02T04:50:27.426719+0000","last_fullsized":"2024-04-02T04:50:27.426719+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.410086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.453119+0000","last_change":"2024-04-02T04:49:28.782201+0000","last_active":"2024-04-02T04:50:27.453119+0000","last_peered":"2024-04-02T04:50:27.453119+0000","last_clean":"2024-04-02T04:50:27.453119+0000","last_became_active":"2024-04-02T04:49:28.781235+0000","last_became_peered":"2024-04-02T04:49:28.781235+0000","last_unstale":"2024-04-02T04:50:27.453119+0000","last_undegraded":"2024-04-02T04:50:27.453119+0000","last_fullsized":"2024-04-02T04:50:27.453119+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.912877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.424219+0000","last_change":"2024-04-02T04:49:28.787959+0000","last_active":"2024-04-02T04:50:27.424219+0000","last_peered":"2024-04-02T04:50:27.424219+0000","last_clean":"2024-04-02T04:50:27.424219+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:50:27.424219+0000","last_undegraded":"2024-04-02T04:50:27.424219+0000","last_fullsized":"2024-04-02T04:50:27.424219+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.288476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.391952+0000","last_change":"2024-04-02T04:49:47.389948+0000","last_active":"2024-04-02T04:50:27.391952+0000","last_peered":"2024-04-02T04:50:27.391952+0000","last_clean":"2024-04-02T04:49:45.343249+0000","last_became_active":"2024-04-02T04:49:47.389948+0000","last_became_peered":"2024-04-02T04:49:47.389948+0000","last_unstale":"2024-04-02T04:50:27.391952+0000","last_undegraded":"2024-04-02T04:50:27.391952+0000","last_fullsized":"2024-04-02T04:49:47.366723+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.034689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428472+0000","last_change":"2024-04-02T04:49:28.750685+0000","last_active":"2024-04-02T04:50:27.428472+0000","last_peered":"2024-04-02T04:50:27.428472+0000","last_clean":"2024-04-02T04:50:27.428472+0000","last_became_active":"2024-04-02T04:49:28.750373+0000","last_became_peered":"2024-04-02T04:49:28.750373+0000","last_unstale":"2024-04-02T04:50:27.428472+0000","last_undegraded":"2024-04-02T04:50:27.428472+0000","last_fullsized":"2024-04-02T04:50:27.428472+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:41.002139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.394225+0000","last_change":"2024-04-02T04:49:47.389543+0000","last_active":"2024-04-02T04:50:27.394225+0000","last_peered":"2024-04-02T04:50:27.394225+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.389543+0000","last_became_peered":"2024-04-02T04:49:47.389543+0000","last_unstale":"2024-04-02T04:50:27.394225+0000","last_undegraded":"2024-04-02T04:50:27.394225+0000","last_fullsized":"2024-04-02T04:49:47.363220+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.639925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.407532+0000","last_change":"2024-04-02T04:49:47.391015+0000","last_active":"2024-04-02T04:50:27.407532+0000","last_peered":"2024-04-02T04:50:27.407532+0000","last_clean":"2024-04-02T04:49:45.363593+0000","last_became_active":"2024-04-02T04:49:47.391015+0000","last_became_peered":"2024-04-02T04:49:47.391015+0000","last_unstale":"2024-04-02T04:50:27.407532+0000","last_undegraded":"2024-04-02T04:50:27.407532+0000","last_fullsized":"2024-04-02T04:49:47.362030+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.283206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.385834+0000","last_change":"2024-04-02T04:49:47.434747+0000","last_active":"2024-04-02T04:50:27.385834+0000","last_peered":"2024-04-02T04:50:27.385834+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.434747+0000","last_became_peered":"2024-04-02T04:49:47.434747+0000","last_unstale":"2024-04-02T04:50:27.385834+0000","last_undegraded":"2024-04-02T04:50:27.385834+0000","last_fullsized":"2024-04-02T04:49:47.346344+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:46.288298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.357579+0000","last_change":"2024-04-02T04:49:28.741181+0000","last_active":"2024-04-02T04:50:27.357579+0000","last_peered":"2024-04-02T04:50:27.357579+0000","last_clean":"2024-04-02T04:50:27.357579+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:50:27.357579+0000","last_undegraded":"2024-04-02T04:50:27.357579+0000","last_fullsized":"2024-04-02T04:50:27.357579+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:49.387519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405448+0000","last_change":"2024-04-02T04:49:28.787621+0000","last_active":"2024-04-02T04:50:27.405448+0000","last_peered":"2024-04-02T04:50:27.405448+0000","last_clean":"2024-04-02T04:50:27.405448+0000","last_became_active":"2024-04-02T04:49:28.785195+0000","last_became_peered":"2024-04-02T04:49:28.785195+0000","last_unstale":"2024-04-02T04:50:27.405448+0000","last_undegraded":"2024-04-02T04:50:27.405448+0000","last_fullsized":"2024-04-02T04:50:27.405448+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:46.794953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411188+0000","last_change":"2024-04-02T04:49:28.743937+0000","last_active":"2024-04-02T04:50:27.411188+0000","last_peered":"2024-04-02T04:50:27.411188+0000","last_clean":"2024-04-02T04:50:27.411188+0000","last_became_active":"2024-04-02T04:49:28.741450+0000","last_became_peered":"2024-04-02T04:49:28.741450+0000","last_unstale":"2024-04-02T04:50:27.411188+0000","last_undegraded":"2024-04-02T04:50:27.411188+0000","last_fullsized":"2024-04-02T04:50:27.411188+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.773893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.399744+0000","last_change":"2024-04-02T04:49:28.731835+0000","last_active":"2024-04-02T04:50:27.399744+0000","last_peered":"2024-04-02T04:50:27.399744+0000","last_clean":"2024-04-02T04:50:27.399744+0000","last_became_active":"2024-04-02T04:49:28.731343+0000","last_became_peered":"2024-04-02T04:49:28.731343+0000","last_unstale":"2024-04-02T04:50:27.399744+0000","last_undegraded":"2024-04-02T04:50:27.399744+0000","last_fullsized":"2024-04-02T04:50:27.399744+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.085604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411487+0000","last_change":"2024-04-02T04:49:28.698491+0000","last_active":"2024-04-02T04:50:27.411487+0000","last_peered":"2024-04-02T04:50:27.411487+0000","last_clean":"2024-04-02T04:50:27.411487+0000","last_became_active":"2024-04-02T04:49:28.698110+0000","last_became_peered":"2024-04-02T04:49:28.698110+0000","last_unstale":"2024-04-02T04:50:27.411487+0000","last_undegraded":"2024-04-02T04:50:27.411487+0000","last_fullsized":"2024-04-02T04:50:27.411487+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.403086+0000","last_change":"2024-04-02T04:49:47.445509+0000","last_active":"2024-04-02T04:50:27.403086+0000","last_peered":"2024-04-02T04:50:27.403086+0000","last_clean":"2024-04-02T04:49:45.370130+0000","last_became_active":"2024-04-02T04:49:47.445509+0000","last_became_peered":"2024-04-02T04:49:47.445509+0000","last_unstale":"2024-04-02T04:50:27.403086+0000","last_undegraded":"2024-04-02T04:50:27.403086+0000","last_fullsized":"2024-04-02T04:49:47.391250+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.355962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405239+0000","last_change":"2024-04-02T04:49:28.748883+0000","last_active":"2024-04-02T04:50:27.405239+0000","last_peered":"2024-04-02T04:50:27.405239+0000","last_clean":"2024-04-02T04:50:27.405239+0000","last_became_active":"2024-04-02T04:49:28.746056+0000","last_became_peered":"2024-04-02T04:49:28.746056+0000","last_unstale":"2024-04-02T04:50:27.405239+0000","last_undegraded":"2024-04-02T04:50:27.405239+0000","last_fullsized":"2024-04-02T04:50:27.405239+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.610229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.366651+0000","last_change":"2024-04-02T04:49:47.388801+0000","last_active":"2024-04-02T04:50:27.366651+0000","last_peered":"2024-04-02T04:50:27.366651+0000","last_clean":"2024-04-02T04:49:45.330442+0000","last_became_active":"2024-04-02T04:49:47.388801+0000","last_became_peered":"2024-04-02T04:49:47.388801+0000","last_unstale":"2024-04-02T04:50:27.366651+0000","last_undegraded":"2024-04-02T04:50:27.366651+0000","last_fullsized":"2024-04-02T04:49:47.339290+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.000319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.365724+0000","last_change":"2024-04-02T04:49:47.444341+0000","last_active":"2024-04-02T04:50:27.365724+0000","last_peered":"2024-04-02T04:50:27.365724+0000","last_clean":"2024-04-02T04:49:45.331782+0000","last_became_active":"2024-04-02T04:49:47.444341+0000","last_became_peered":"2024-04-02T04:49:47.444341+0000","last_unstale":"2024-04-02T04:50:27.365724+0000","last_undegraded":"2024-04-02T04:50:27.365724+0000","last_fullsized":"2024-04-02T04:49:47.336855+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.922032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.369826+0000","last_change":"2024-04-02T04:49:29.857781+0000","last_active":"2024-04-02T04:50:27.369826+0000","last_peered":"2024-04-02T04:50:27.369826+0000","last_clean":"2024-04-02T04:50:27.369826+0000","last_became_active":"2024-04-02T04:49:29.857200+0000","last_became_peered":"2024-04-02T04:49:29.857200+0000","last_unstale":"2024-04-02T04:50:27.369826+0000","last_undegraded":"2024-04-02T04:50:27.369826+0000","last_fullsized":"2024-04-02T04:50:27.369826+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.627758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.384409+0000","last_change":"2024-04-02T04:49:28.747635+0000","last_active":"2024-04-02T04:50:27.384409+0000","last_peered":"2024-04-02T04:50:27.384409+0000","last_clean":"2024-04-02T04:50:27.384409+0000","last_became_active":"2024-04-02T04:49:28.746855+0000","last_became_peered":"2024-04-02T04:49:28.746855+0000","last_unstale":"2024-04-02T04:50:27.384409+0000","last_undegraded":"2024-04-02T04:50:27.384409+0000","last_fullsized":"2024-04-02T04:50:27.384409+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.103588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.380898+0000","last_change":"2024-04-02T04:49:47.472640+0000","last_active":"2024-04-02T04:50:27.380898+0000","last_peered":"2024-04-02T04:50:27.380898+0000","last_clean":"2024-04-02T04:49:45.352140+0000","last_became_active":"2024-04-02T04:49:47.472640+0000","last_became_peered":"2024-04-02T04:49:47.472640+0000","last_unstale":"2024-04-02T04:50:27.380898+0000","last_undegraded":"2024-04-02T04:50:27.380898+0000","last_fullsized":"2024-04-02T04:49:47.377460+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:45.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377326+0000","last_change":"2024-04-02T04:49:29.875000+0000","last_active":"2024-04-02T04:50:27.377326+0000","last_peered":"2024-04-02T04:50:27.377326+0000","last_clean":"2024-04-02T04:50:27.377326+0000","last_became_active":"2024-04-02T04:49:29.874539+0000","last_became_peered":"2024-04-02T04:49:29.874539+0000","last_unstale":"2024-04-02T04:50:27.377326+0000","last_undegraded":"2024-04-02T04:50:27.377326+0000","last_fullsized":"2024-04-02T04:50:27.377326+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.749738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.427926+0000","last_change":"2024-04-02T04:49:47.452282+0000","last_active":"2024-04-02T04:50:27.427926+0000","last_peered":"2024-04-02T04:50:27.427926+0000","last_clean":"2024-04-02T04:49:45.347315+0000","last_became_active":"2024-04-02T04:49:47.452282+0000","last_became_peered":"2024-04-02T04:49:47.452282+0000","last_unstale":"2024-04-02T04:50:27.427926+0000","last_undegraded":"2024-04-02T04:50:27.427926+0000","last_fullsized":"2024-04-02T04:49:47.364280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.055234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.393585+0000","last_change":"2024-04-02T04:49:29.833094+0000","last_active":"2024-04-02T04:50:27.393585+0000","last_peered":"2024-04-02T04:50:27.393585+0000","last_clean":"2024-04-02T04:50:27.393585+0000","last_became_active":"2024-04-02T04:49:29.830080+0000","last_became_peered":"2024-04-02T04:49:29.830080+0000","last_unstale":"2024-04-02T04:50:27.393585+0000","last_undegraded":"2024-04-02T04:50:27.393585+0000","last_fullsized":"2024-04-02T04:50:27.393585+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:34.558193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.412130+0000","last_change":"2024-04-02T04:49:47.477237+0000","last_active":"2024-04-02T04:50:27.412130+0000","last_peered":"2024-04-02T04:50:27.412130+0000","last_clean":"2024-04-02T04:49:45.366065+0000","last_became_active":"2024-04-02T04:49:47.477237+0000","last_became_peered":"2024-04-02T04:49:47.477237+0000","last_unstale":"2024-04-02T04:50:27.412130+0000","last_undegraded":"2024-04-02T04:50:27.412130+0000","last_fullsized":"2024-04-02T04:49:47.430531+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.439258+0000","last_change":"2024-04-02T04:49:28.781126+0000","last_active":"2024-04-02T04:50:27.439258+0000","last_peered":"2024-04-02T04:50:27.439258+0000","last_clean":"2024-04-02T04:50:27.439258+0000","last_became_active":"2024-04-02T04:49:28.780779+0000","last_became_peered":"2024-04-02T04:49:28.780779+0000","last_unstale":"2024-04-02T04:50:27.439258+0000","last_undegraded":"2024-04-02T04:50:27.439258+0000","last_fullsized":"2024-04-02T04:50:27.439258+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.131020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428150+0000","last_change":"2024-04-02T04:49:28.788472+0000","last_active":"2024-04-02T04:50:27.428150+0000","last_peered":"2024-04-02T04:50:27.428150+0000","last_clean":"2024-04-02T04:50:27.428150+0000","last_became_active":"2024-04-02T04:49:28.784897+0000","last_became_peered":"2024-04-02T04:49:28.784897+0000","last_unstale":"2024-04-02T04:50:27.428150+0000","last_undegraded":"2024-04-02T04:50:27.428150+0000","last_fullsized":"2024-04-02T04:50:27.428150+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.376986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.388237+0000","last_change":"2024-04-02T04:49:28.744580+0000","last_active":"2024-04-02T04:50:27.388237+0000","last_peered":"2024-04-02T04:50:27.388237+0000","last_clean":"2024-04-02T04:50:27.388237+0000","last_became_active":"2024-04-02T04:49:28.744153+0000","last_became_peered":"2024-04-02T04:49:28.744153+0000","last_unstale":"2024-04-02T04:50:27.388237+0000","last_undegraded":"2024-04-02T04:50:27.388237+0000","last_fullsized":"2024-04-02T04:50:27.388237+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.499989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.438461+0000","last_change":"2024-04-02T04:49:28.761364+0000","last_active":"2024-04-02T04:50:27.438461+0000","last_peered":"2024-04-02T04:50:27.438461+0000","last_clean":"2024-04-02T04:50:27.438461+0000","last_became_active":"2024-04-02T04:49:28.760661+0000","last_became_peered":"2024-04-02T04:49:28.760661+0000","last_unstale":"2024-04-02T04:50:27.438461+0000","last_undegraded":"2024-04-02T04:50:27.438461+0000","last_fullsized":"2024-04-02T04:50:27.438461+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:52.138288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.433951+0000","last_change":"2024-04-02T04:49:28.771528+0000","last_active":"2024-04-02T04:50:27.433951+0000","last_peered":"2024-04-02T04:50:27.433951+0000","last_clean":"2024-04-02T04:50:27.433951+0000","last_became_active":"2024-04-02T04:49:28.769787+0000","last_became_peered":"2024-04-02T04:49:28.769787+0000","last_unstale":"2024-04-02T04:50:27.433951+0000","last_undegraded":"2024-04-02T04:50:27.433951+0000","last_fullsized":"2024-04-02T04:50:27.433951+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.897668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.436644+0000","last_change":"2024-04-02T04:49:28.728679+0000","last_active":"2024-04-02T04:50:27.436644+0000","last_peered":"2024-04-02T04:50:27.436644+0000","last_clean":"2024-04-02T04:50:27.436644+0000","last_became_active":"2024-04-02T04:49:28.728416+0000","last_became_peered":"2024-04-02T04:49:28.728416+0000","last_unstale":"2024-04-02T04:50:27.436644+0000","last_undegraded":"2024-04-02T04:50:27.436644+0000","last_fullsized":"2024-04-02T04:50:27.436644+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:55.454335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.409952+0000","last_change":"2024-04-02T04:49:28.728977+0000","last_active":"2024-04-02T04:50:27.409952+0000","last_peered":"2024-04-02T04:50:27.409952+0000","last_clean":"2024-04-02T04:50:27.409952+0000","last_became_active":"2024-04-02T04:49:28.727661+0000","last_became_peered":"2024-04-02T04:49:28.727661+0000","last_unstale":"2024-04-02T04:50:27.409952+0000","last_undegraded":"2024-04-02T04:50:27.409952+0000","last_fullsized":"2024-04-02T04:50:27.409952+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:46.089165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.444503+0000","last_change":"2024-04-02T04:49:28.762112+0000","last_active":"2024-04-02T04:50:27.444503+0000","last_peered":"2024-04-02T04:50:27.444503+0000","last_clean":"2024-04-02T04:50:27.444503+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:50:27.444503+0000","last_undegraded":"2024-04-02T04:50:27.444503+0000","last_fullsized":"2024-04-02T04:50:27.444503+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.501134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.386310+0000","last_change":"2024-04-02T04:49:28.745274+0000","last_active":"2024-04-02T04:50:27.386310+0000","last_peered":"2024-04-02T04:50:27.386310+0000","last_clean":"2024-04-02T04:50:27.386310+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:50:27.386310+0000","last_undegraded":"2024-04-02T04:50:27.386310+0000","last_fullsized":"2024-04-02T04:50:27.386310+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.869332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.397201+0000","last_change":"2024-04-02T04:49:47.512403+0000","last_active":"2024-04-02T04:50:27.397201+0000","last_peered":"2024-04-02T04:50:27.397201+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.512403+0000","last_became_peered":"2024-04-02T04:49:47.512403+0000","last_unstale":"2024-04-02T04:50:27.397201+0000","last_undegraded":"2024-04-02T04:50:27.397201+0000","last_fullsized":"2024-04-02T04:49:47.379434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:47.191361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411587+0000","last_change":"2024-04-02T04:49:28.726174+0000","last_active":"2024-04-02T04:50:27.411587+0000","last_peered":"2024-04-02T04:50:27.411587+0000","last_clean":"2024-04-02T04:50:27.411587+0000","last_became_active":"2024-04-02T04:49:28.725896+0000","last_became_peered":"2024-04-02T04:49:28.725896+0000","last_unstale":"2024-04-02T04:50:27.411587+0000","last_undegraded":"2024-04-02T04:50:27.411587+0000","last_fullsized":"2024-04-02T04:50:27.411587+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:46.748968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.363153+0000","last_change":"2024-04-02T04:49:28.724761+0000","last_active":"2024-04-02T04:50:27.363153+0000","last_peered":"2024-04-02T04:50:27.363153+0000","last_clean":"2024-04-02T04:50:27.363153+0000","last_became_active":"2024-04-02T04:49:28.724387+0000","last_became_peered":"2024-04-02T04:49:28.724387+0000","last_unstale":"2024-04-02T04:50:27.363153+0000","last_undegraded":"2024-04-02T04:50:27.363153+0000","last_fullsized":"2024-04-02T04:50:27.363153+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.293721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.371398+0000","last_change":"2024-04-02T04:49:29.857549+0000","last_active":"2024-04-02T04:50:27.371398+0000","last_peered":"2024-04-02T04:50:27.371398+0000","last_clean":"2024-04-02T04:50:27.371398+0000","last_became_active":"2024-04-02T04:49:29.856708+0000","last_became_peered":"2024-04-02T04:49:29.856708+0000","last_unstale":"2024-04-02T04:50:27.371398+0000","last_undegraded":"2024-04-02T04:50:27.371398+0000","last_fullsized":"2024-04-02T04:50:27.371398+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.358415+0000","last_change":"2024-04-02T04:49:29.816487+0000","last_active":"2024-04-02T04:50:27.358415+0000","last_peered":"2024-04-02T04:50:27.358415+0000","last_clean":"2024-04-02T04:50:27.358415+0000","last_became_active":"2024-04-02T04:49:29.816194+0000","last_became_peered":"2024-04-02T04:49:29.816194+0000","last_unstale":"2024-04-02T04:50:27.358415+0000","last_undegraded":"2024-04-02T04:50:27.358415+0000","last_fullsized":"2024-04-02T04:50:27.358415+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.398658+0000","last_change":"2024-04-02T04:49:47.427831+0000","last_active":"2024-04-02T04:50:27.398658+0000","last_peered":"2024-04-02T04:50:27.398658+0000","last_clean":"2024-04-02T04:49:45.321587+0000","last_became_active":"2024-04-02T04:49:47.427831+0000","last_became_peered":"2024-04-02T04:49:47.427831+0000","last_unstale":"2024-04-02T04:50:27.398658+0000","last_undegraded":"2024-04-02T04:50:27.398658+0000","last_fullsized":"2024-04-02T04:49:47.369909+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.868774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.388508+0000","last_change":"2024-04-02T04:49:47.387543+0000","last_active":"2024-04-02T04:50:27.388508+0000","last_peered":"2024-04-02T04:50:27.388508+0000","last_clean":"2024-04-02T04:49:45.306095+0000","last_became_active":"2024-04-02T04:49:47.387543+0000","last_became_peered":"2024-04-02T04:49:47.387543+0000","last_unstale":"2024-04-02T04:50:27.388508+0000","last_undegraded":"2024-04-02T04:50:27.388508+0000","last_fullsized":"2024-04-02T04:49:47.345085+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:51.541798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375855+0000","last_change":"2024-04-02T04:49:28.739636+0000","last_active":"2024-04-02T04:50:27.375855+0000","last_peered":"2024-04-02T04:50:27.375855+0000","last_clean":"2024-04-02T04:50:27.375855+0000","last_became_active":"2024-04-02T04:49:28.734672+0000","last_became_peered":"2024-04-02T04:49:28.734672+0000","last_unstale":"2024-04-02T04:50:27.375855+0000","last_undegraded":"2024-04-02T04:50:27.375855+0000","last_fullsized":"2024-04-02T04:50:27.375855+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.378147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.396152+0000","last_change":"2024-04-02T04:49:28.749558+0000","last_active":"2024-04-02T04:50:27.396152+0000","last_peered":"2024-04-02T04:50:27.396152+0000","last_clean":"2024-04-02T04:50:27.396152+0000","last_became_active":"2024-04-02T04:49:28.749072+0000","last_became_peered":"2024-04-02T04:49:28.749072+0000","last_unstale":"2024-04-02T04:50:27.396152+0000","last_undegraded":"2024-04-02T04:50:27.396152+0000","last_fullsized":"2024-04-02T04:50:27.396152+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.539036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.358969+0000","last_change":"2024-04-02T04:49:29.864378+0000","last_active":"2024-04-02T04:50:27.358969+0000","last_peered":"2024-04-02T04:50:27.358969+0000","last_clean":"2024-04-02T04:50:27.358969+0000","last_became_active":"2024-04-02T04:49:29.858586+0000","last_became_peered":"2024-04-02T04:49:29.858586+0000","last_unstale":"2024-04-02T04:50:27.358969+0000","last_undegraded":"2024-04-02T04:50:27.358969+0000","last_fullsized":"2024-04-02T04:50:27.358969+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:34.464402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.370754+0000","last_change":"2024-04-02T04:49:47.469970+0000","last_active":"2024-04-02T04:50:27.370754+0000","last_peered":"2024-04-02T04:50:27.370754+0000","last_clean":"2024-04-02T04:49:45.328796+0000","last_became_active":"2024-04-02T04:49:47.469970+0000","last_became_peered":"2024-04-02T04:49:47.469970+0000","last_unstale":"2024-04-02T04:50:27.370754+0000","last_undegraded":"2024-04-02T04:50:27.370754+0000","last_fullsized":"2024-04-02T04:49:47.382953+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:54.725625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.428012+0000","last_change":"2024-04-02T04:49:47.457335+0000","last_active":"2024-04-02T04:50:27.428012+0000","last_peered":"2024-04-02T04:50:27.428012+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.457335+0000","last_became_peered":"2024-04-02T04:49:47.457335+0000","last_unstale":"2024-04-02T04:50:27.428012+0000","last_undegraded":"2024-04-02T04:50:27.428012+0000","last_fullsized":"2024-04-02T04:49:47.405947+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:47.391533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.388545+0000","last_change":"2024-04-02T04:49:28.751480+0000","last_active":"2024-04-02T04:50:27.388545+0000","last_peered":"2024-04-02T04:50:27.388545+0000","last_clean":"2024-04-02T04:50:27.388545+0000","last_became_active":"2024-04-02T04:49:28.750713+0000","last_became_peered":"2024-04-02T04:49:28.750713+0000","last_unstale":"2024-04-02T04:50:27.388545+0000","last_undegraded":"2024-04-02T04:50:27.388545+0000","last_fullsized":"2024-04-02T04:50:27.388545+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:52.675524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.439426+0000","last_change":"2024-04-02T04:49:28.768154+0000","last_active":"2024-04-02T04:50:27.439426+0000","last_peered":"2024-04-02T04:50:27.439426+0000","last_clean":"2024-04-02T04:50:27.439426+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:50:27.439426+0000","last_undegraded":"2024-04-02T04:50:27.439426+0000","last_fullsized":"2024-04-02T04:50:27.439426+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.843475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.409181+0000","last_change":"2024-04-02T04:49:47.500400+0000","last_active":"2024-04-02T04:50:27.409181+0000","last_peered":"2024-04-02T04:50:27.409181+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.500400+0000","last_became_peered":"2024-04-02T04:49:47.500400+0000","last_unstale":"2024-04-02T04:50:27.409181+0000","last_undegraded":"2024-04-02T04:50:27.409181+0000","last_fullsized":"2024-04-02T04:49:47.430108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.117902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.436729+0000","last_change":"2024-04-02T04:49:28.718550+0000","last_active":"2024-04-02T04:50:27.436729+0000","last_peered":"2024-04-02T04:50:27.436729+0000","last_clean":"2024-04-02T04:50:27.436729+0000","last_became_active":"2024-04-02T04:49:28.716933+0000","last_became_peered":"2024-04-02T04:49:28.716933+0000","last_unstale":"2024-04-02T04:50:27.436729+0000","last_undegraded":"2024-04-02T04:50:27.436729+0000","last_fullsized":"2024-04-02T04:50:27.436729+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428013+0000","last_change":"2024-04-02T04:49:28.780348+0000","last_active":"2024-04-02T04:50:27.428013+0000","last_peered":"2024-04-02T04:50:27.428013+0000","last_clean":"2024-04-02T04:50:27.428013+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:50:27.428013+0000","last_undegraded":"2024-04-02T04:50:27.428013+0000","last_fullsized":"2024-04-02T04:50:27.428013+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.369482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.452882+0000","last_change":"2024-04-02T04:49:28.757581+0000","last_active":"2024-04-02T04:50:27.452882+0000","last_peered":"2024-04-02T04:50:27.452882+0000","last_clean":"2024-04-02T04:50:27.452882+0000","last_became_active":"2024-04-02T04:49:28.757217+0000","last_became_peered":"2024-04-02T04:49:28.757217+0000","last_unstale":"2024-04-02T04:50:27.452882+0000","last_undegraded":"2024-04-02T04:50:27.452882+0000","last_fullsized":"2024-04-02T04:50:27.452882+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:53.435598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.404013+0000","last_change":"2024-04-02T04:49:47.508156+0000","last_active":"2024-04-02T04:50:27.404013+0000","last_peered":"2024-04-02T04:50:27.404013+0000","last_clean":"2024-04-02T04:49:45.359498+0000","last_became_active":"2024-04-02T04:49:47.508156+0000","last_became_peered":"2024-04-02T04:49:47.508156+0000","last_unstale":"2024-04-02T04:50:27.404013+0000","last_undegraded":"2024-04-02T04:50:27.404013+0000","last_fullsized":"2024-04-02T04:49:47.430692+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.183577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.417190+0000","last_change":"2024-04-02T04:49:28.759053+0000","last_active":"2024-04-02T04:50:27.417190+0000","last_peered":"2024-04-02T04:50:27.417190+0000","last_clean":"2024-04-02T04:50:27.417190+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:50:27.417190+0000","last_undegraded":"2024-04-02T04:50:27.417190+0000","last_fullsized":"2024-04-02T04:50:27.417190+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.424914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.428137+0000","last_change":"2024-04-02T04:49:47.501136+0000","last_active":"2024-04-02T04:50:27.428137+0000","last_peered":"2024-04-02T04:50:27.428137+0000","last_clean":"2024-04-02T04:49:45.381195+0000","last_became_active":"2024-04-02T04:49:47.501136+0000","last_became_peered":"2024-04-02T04:49:47.501136+0000","last_unstale":"2024-04-02T04:50:27.428137+0000","last_undegraded":"2024-04-02T04:50:27.428137+0000","last_fullsized":"2024-04-02T04:49:47.416717+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:47.610284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.417535+0000","last_change":"2024-04-02T04:49:47.510048+0000","last_active":"2024-04-02T04:50:27.417535+0000","last_peered":"2024-04-02T04:50:27.417535+0000","last_clean":"2024-04-02T04:49:45.381640+0000","last_became_active":"2024-04-02T04:49:47.510048+0000","last_became_peered":"2024-04-02T04:49:47.510048+0000","last_unstale":"2024-04-02T04:50:27.417535+0000","last_undegraded":"2024-04-02T04:50:27.417535+0000","last_fullsized":"2024-04-02T04:49:47.408507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.285216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.394741+0000","last_change":"2024-04-02T04:49:28.765539+0000","last_active":"2024-04-02T04:50:27.394741+0000","last_peered":"2024-04-02T04:50:27.394741+0000","last_clean":"2024-04-02T04:50:27.394741+0000","last_became_active":"2024-04-02T04:49:28.765229+0000","last_became_peered":"2024-04-02T04:49:28.765229+0000","last_unstale":"2024-04-02T04:50:27.394741+0000","last_undegraded":"2024-04-02T04:50:27.394741+0000","last_fullsized":"2024-04-02T04:50:27.394741+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.539872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.384401+0000","last_change":"2024-04-02T04:49:47.441206+0000","last_active":"2024-04-02T04:50:27.384401+0000","last_peered":"2024-04-02T04:50:27.384401+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.441206+0000","last_became_peered":"2024-04-02T04:49:47.441206+0000","last_unstale":"2024-04-02T04:50:27.384401+0000","last_undegraded":"2024-04-02T04:50:27.384401+0000","last_fullsized":"2024-04-02T04:49:47.375248+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.834302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405649+0000","last_change":"2024-04-02T04:49:28.759884+0000","last_active":"2024-04-02T04:50:27.405649+0000","last_peered":"2024-04-02T04:50:27.405649+0000","last_clean":"2024-04-02T04:50:27.405649+0000","last_became_active":"2024-04-02T04:49:28.749451+0000","last_became_peered":"2024-04-02T04:49:28.749451+0000","last_unstale":"2024-04-02T04:50:27.405649+0000","last_undegraded":"2024-04-02T04:50:27.405649+0000","last_fullsized":"2024-04-02T04:50:27.405649+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.018925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":111,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.397405+0000","last_change":"2024-04-02T04:49:47.479424+0000","last_active":"2024-04-02T04:50:27.397405+0000","last_peered":"2024-04-02T04:50:27.397405+0000","last_clean":"2024-04-02T04:49:45.342017+0000","last_became_active":"2024-04-02T04:49:47.479424+0000","last_became_peered":"2024-04-02T04:49:47.479424+0000","last_unstale":"2024-04-02T04:50:27.397405+0000","last_undegraded":"2024-04-02T04:50:27.397405+0000","last_fullsized":"2024-04-02T04:49:47.413098+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.688068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.361675+0000","last_change":"2024-04-02T04:49:28.748601+0000","last_active":"2024-04-02T04:50:27.361675+0000","last_peered":"2024-04-02T04:50:27.361675+0000","last_clean":"2024-04-02T04:50:27.361675+0000","last_became_active":"2024-04-02T04:49:28.743589+0000","last_became_peered":"2024-04-02T04:49:28.743589+0000","last_unstale":"2024-04-02T04:50:27.361675+0000","last_undegraded":"2024-04-02T04:50:27.361675+0000","last_fullsized":"2024-04-02T04:50:27.361675+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:41.508379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":110,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.369941+0000","last_change":"2024-04-02T04:49:47.502106+0000","last_active":"2024-04-02T04:50:27.369941+0000","last_peered":"2024-04-02T04:50:27.369941+0000","last_clean":"2024-04-02T04:49:45.334274+0000","last_became_active":"2024-04-02T04:49:47.502106+0000","last_became_peered":"2024-04-02T04:49:47.502106+0000","last_unstale":"2024-04-02T04:50:27.369941+0000","last_undegraded":"2024-04-02T04:50:27.369941+0000","last_fullsized":"2024-04-02T04:49:47.399330+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:53.045960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":107,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.358207+0000","last_change":"2024-04-02T04:49:29.805620+0000","last_active":"2024-04-02T04:50:27.358207+0000","last_peered":"2024-04-02T04:50:27.358207+0000","last_clean":"2024-04-02T04:50:27.358207+0000","last_became_active":"2024-04-02T04:49:29.805341+0000","last_became_peered":"2024-04-02T04:49:29.805341+0000","last_unstale":"2024-04-02T04:50:27.358207+0000","last_undegraded":"2024-04-02T04:50:27.358207+0000","last_fullsized":"2024-04-02T04:50:27.358207+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:51.511423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":81,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.394297+0000","last_change":"2024-04-02T04:49:47.482774+0000","last_active":"2024-04-02T04:50:27.394297+0000","last_peered":"2024-04-02T04:50:27.394297+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.482774+0000","last_became_peered":"2024-04-02T04:49:47.482774+0000","last_unstale":"2024-04-02T04:50:27.394297+0000","last_undegraded":"2024-04-02T04:50:27.394297+0000","last_fullsized":"2024-04-02T04:49:47.403371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.380369+0000","last_change":"2024-04-02T04:49:28.773379+0000","last_active":"2024-04-02T04:50:27.380369+0000","last_peered":"2024-04-02T04:50:27.380369+0000","last_clean":"2024-04-02T04:50:27.380369+0000","last_became_active":"2024-04-02T04:49:28.766979+0000","last_became_peered":"2024-04-02T04:49:28.766979+0000","last_unstale":"2024-04-02T04:50:27.380369+0000","last_undegraded":"2024-04-02T04:50:27.380369+0000","last_fullsized":"2024-04-02T04:50:27.380369+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.330686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.415670+0000","last_change":"2024-04-02T04:49:28.756847+0000","last_active":"2024-04-02T04:50:27.415670+0000","last_peered":"2024-04-02T04:50:27.415670+0000","last_clean":"2024-04-02T04:50:27.415670+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:50:27.415670+0000","last_undegraded":"2024-04-02T04:50:27.415670+0000","last_fullsized":"2024-04-02T04:50:27.415670+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.195571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432120+0000","last_change":"2024-04-02T04:49:28.753851+0000","last_active":"2024-04-02T04:50:27.432120+0000","last_peered":"2024-04-02T04:50:27.432120+0000","last_clean":"2024-04-02T04:50:27.432120+0000","last_became_active":"2024-04-02T04:49:28.753376+0000","last_became_peered":"2024-04-02T04:49:28.753376+0000","last_unstale":"2024-04-02T04:50:27.432120+0000","last_undegraded":"2024-04-02T04:50:27.432120+0000","last_fullsized":"2024-04-02T04:50:27.432120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.430472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.385107+0000","last_change":"2024-04-02T04:49:28.757796+0000","last_active":"2024-04-02T04:50:27.385107+0000","last_peered":"2024-04-02T04:50:27.385107+0000","last_clean":"2024-04-02T04:50:27.385107+0000","last_became_active":"2024-04-02T04:49:28.757583+0000","last_became_peered":"2024-04-02T04:49:28.757583+0000","last_unstale":"2024-04-02T04:50:27.385107+0000","last_undegraded":"2024-04-02T04:50:27.385107+0000","last_fullsized":"2024-04-02T04:50:27.385107+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.135695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":108,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.381011+0000","last_change":"2024-04-02T04:49:28.726706+0000","last_active":"2024-04-02T04:50:27.381011+0000","last_peered":"2024-04-02T04:50:27.381011+0000","last_clean":"2024-04-02T04:50:27.381011+0000","last_became_active":"2024-04-02T04:49:28.726425+0000","last_became_peered":"2024-04-02T04:49:28.726425+0000","last_unstale":"2024-04-02T04:50:27.381011+0000","last_undegraded":"2024-04-02T04:50:27.381011+0000","last_fullsized":"2024-04-02T04:50:27.381011+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:48.682138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"139'5","reported_seq":126,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.421267+0000","last_change":"2024-04-02T04:50:26.846127+0000","last_active":"2024-04-02T04:50:27.421267+0000","last_peered":"2024-04-02T04:50:27.421267+0000","last_clean":"2024-04-02T04:50:27.421267+0000","last_became_active":"2024-04-02T04:49:26.170913+0000","last_became_peered":"2024-04-02T04:49:26.170913+0000","last_unstale":"2024-04-02T04:50:27.421267+0000","last_undegraded":"2024-04-02T04:50:27.421267+0000","last_fullsized":"2024-04-02T04:50:27.421267+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:50:26.845900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.845900+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:49.621130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389453+0000","last_change":"2024-04-02T04:49:26.217363+0000","last_active":"2024-04-02T04:50:27.389453+0000","last_peered":"2024-04-02T04:50:27.389453+0000","last_clean":"2024-04-02T04:50:27.389453+0000","last_became_active":"2024-04-02T04:49:26.216834+0000","last_became_peered":"2024-04-02T04:49:26.216834+0000","last_unstale":"2024-04-02T04:50:27.389453+0000","last_undegraded":"2024-04-02T04:50:27.389453+0000","last_fullsized":"2024-04-02T04:50:27.389453+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.629993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.366384+0000","last_change":"2024-04-02T04:49:47.366762+0000","last_active":"2024-04-02T04:50:27.366384+0000","last_peered":"2024-04-02T04:50:27.366384+0000","last_clean":"2024-04-02T04:49:45.330332+0000","last_became_active":"2024-04-02T04:49:47.366762+0000","last_became_peered":"2024-04-02T04:49:47.366762+0000","last_unstale":"2024-04-02T04:50:27.366384+0000","last_undegraded":"2024-04-02T04:50:27.366384+0000","last_fullsized":"2024-04-02T04:49:47.338850+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.797658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.797658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.617779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":123,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.421553+0000","last_change":"2024-04-02T04:49:47.409183+0000","last_active":"2024-04-02T04:50:27.421553+0000","last_peered":"2024-04-02T04:50:27.421553+0000","last_clean":"2024-04-02T04:44:01.886062+0000","last_became_active":"2024-04-02T04:49:47.409183+0000","last_became_peered":"2024-04-02T04:49:47.409183+0000","last_unstale":"2024-04-02T04:50:27.421553+0000","last_undegraded":"2024-04-02T04:49:47.369263+0000","last_fullsized":"2024-04-02T04:49:47.369128+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":258,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.433052+0000","last_change":"2024-04-02T04:49:47.494397+0000","last_active":"2024-04-02T04:50:27.433052+0000","last_peered":"2024-04-02T04:50:27.433052+0000","last_clean":"2024-04-02T04:49:45.361282+0000","last_became_active":"2024-04-02T04:49:47.494397+0000","last_became_peered":"2024-04-02T04:49:47.494397+0000","last_unstale":"2024-04-02T04:50:27.433052+0000","last_undegraded":"2024-04-02T04:50:27.433052+0000","last_fullsized":"2024-04-02T04:49:47.425453+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.658688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.658688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.007284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":251,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.368232+0000","last_change":"2024-04-02T04:50:11.467388+0000","last_active":"2024-04-02T04:50:27.368232+0000","last_peered":"2024-04-02T04:50:27.368232+0000","last_clean":"2024-04-02T04:50:27.368232+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:50:27.368232+0000","last_undegraded":"2024-04-02T04:50:27.368232+0000","last_fullsized":"2024-04-02T04:50:27.368232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.467182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.467182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.047738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":131,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404896+0000","last_change":"2024-04-02T04:50:00.510347+0000","last_active":"2024-04-02T04:50:27.404896+0000","last_peered":"2024-04-02T04:50:27.404896+0000","last_clean":"2024-04-02T04:50:27.404896+0000","last_became_active":"2024-04-02T04:49:26.524826+0000","last_became_peered":"2024-04-02T04:49:26.524826+0000","last_unstale":"2024-04-02T04:50:27.404896+0000","last_undegraded":"2024-04-02T04:50:27.404896+0000","last_fullsized":"2024-04-02T04:50:27.404896+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:00.510236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:00.510236+0000","last_clean_scrub_stamp":"2024-04-02T04:50:00.510236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:14.432249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.366475+0000","last_change":"2024-04-02T04:49:26.248455+0000","last_active":"2024-04-02T04:50:27.366475+0000","last_peered":"2024-04-02T04:50:27.366475+0000","last_clean":"2024-04-02T04:50:27.366475+0000","last_became_active":"2024-04-02T04:49:26.241757+0000","last_became_peered":"2024-04-02T04:49:26.241757+0000","last_unstale":"2024-04-02T04:50:27.366475+0000","last_undegraded":"2024-04-02T04:50:27.366475+0000","last_fullsized":"2024-04-02T04:50:27.366475+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.591349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.409799+0000","last_change":"2024-04-02T04:49:26.298137+0000","last_active":"2024-04-02T04:50:27.409799+0000","last_peered":"2024-04-02T04:50:27.409799+0000","last_clean":"2024-04-02T04:50:27.409799+0000","last_became_active":"2024-04-02T04:49:26.296797+0000","last_became_peered":"2024-04-02T04:49:26.296797+0000","last_unstale":"2024-04-02T04:50:27.409799+0000","last_undegraded":"2024-04-02T04:50:27.409799+0000","last_fullsized":"2024-04-02T04:50:27.409799+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.151578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.416623+0000","last_change":"2024-04-02T04:49:12.993598+0000","last_active":"2024-04-02T04:50:27.416623+0000","last_peered":"2024-04-02T04:50:27.416623+0000","last_clean":"2024-04-02T04:50:27.416623+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:50:27.416623+0000","last_undegraded":"2024-04-02T04:50:27.416623+0000","last_fullsized":"2024-04-02T04:50:27.416623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.993519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":287,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.391497+0000","last_change":"2024-04-02T04:50:26.509397+0000","last_active":"2024-04-02T04:50:27.391497+0000","last_peered":"2024-04-02T04:50:27.391497+0000","last_clean":"2024-04-02T04:50:27.391497+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:50:27.391497+0000","last_undegraded":"2024-04-02T04:50:27.391497+0000","last_fullsized":"2024-04-02T04:50:27.391497+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:50:26.509291+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.509291+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.366019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":264,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.412681+0000","last_change":"2024-04-02T04:49:05.021205+0000","last_active":"2024-04-02T04:50:27.412681+0000","last_peered":"2024-04-02T04:50:27.412681+0000","last_clean":"2024-04-02T04:50:27.412681+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:50:27.412681+0000","last_undegraded":"2024-04-02T04:50:27.412681+0000","last_fullsized":"2024-04-02T04:50:27.412681+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.020988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.020988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":240,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.384902+0000","last_change":"2024-04-02T04:49:47.516977+0000","last_active":"2024-04-02T04:50:27.384902+0000","last_peered":"2024-04-02T04:50:27.384902+0000","last_clean":"2024-04-02T04:49:45.355630+0000","last_became_active":"2024-04-02T04:49:47.516977+0000","last_became_peered":"2024-04-02T04:49:47.516977+0000","last_unstale":"2024-04-02T04:50:27.384902+0000","last_undegraded":"2024-04-02T04:50:27.384902+0000","last_fullsized":"2024-04-02T04:49:47.384135+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.996159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":131,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.376472+0000","last_change":"2024-04-02T04:50:06.927866+0000","last_active":"2024-04-02T04:50:27.376472+0000","last_peered":"2024-04-02T04:50:27.376472+0000","last_clean":"2024-04-02T04:50:27.376472+0000","last_became_active":"2024-04-02T04:49:26.294550+0000","last_became_peered":"2024-04-02T04:49:26.294550+0000","last_unstale":"2024-04-02T04:50:27.376472+0000","last_undegraded":"2024-04-02T04:50:27.376472+0000","last_fullsized":"2024-04-02T04:50:27.376472+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:06.927770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:06.927770+0000","last_clean_scrub_stamp":"2024-04-02T04:50:06.927770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.535044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"139'5","reported_seq":120,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411682+0000","last_change":"2024-04-02T04:49:25.143416+0000","last_active":"2024-04-02T04:50:27.411682+0000","last_peered":"2024-04-02T04:50:27.411682+0000","last_clean":"2024-04-02T04:50:27.411682+0000","last_became_active":"2024-04-02T04:49:25.139936+0000","last_became_peered":"2024-04-02T04:49:25.139936+0000","last_unstale":"2024-04-02T04:50:27.411682+0000","last_undegraded":"2024-04-02T04:50:27.411682+0000","last_fullsized":"2024-04-02T04:50:27.411682+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.777813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":114,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.394572+0000","last_change":"2024-04-02T04:49:25.143965+0000","last_active":"2024-04-02T04:50:27.394572+0000","last_peered":"2024-04-02T04:50:27.394572+0000","last_clean":"2024-04-02T04:50:27.394572+0000","last_became_active":"2024-04-02T04:49:25.143060+0000","last_became_peered":"2024-04-02T04:49:25.143060+0000","last_unstale":"2024-04-02T04:50:27.394572+0000","last_undegraded":"2024-04-02T04:50:27.394572+0000","last_fullsized":"2024-04-02T04:50:27.394572+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":260,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.408220+0000","last_change":"2024-04-02T04:49:47.472350+0000","last_active":"2024-04-02T04:50:27.408220+0000","last_peered":"2024-04-02T04:50:27.408220+0000","last_clean":"2024-04-02T04:49:45.385689+0000","last_became_active":"2024-04-02T04:49:47.472350+0000","last_became_peered":"2024-04-02T04:49:47.472350+0000","last_unstale":"2024-04-02T04:50:27.408220+0000","last_undegraded":"2024-04-02T04:50:27.408220+0000","last_fullsized":"2024-04-02T04:49:47.393316+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.719146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.719146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.782205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":126,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.402643+0000","last_change":"2024-04-02T04:49:47.381178+0000","last_active":"2024-04-02T04:50:27.402643+0000","last_peered":"2024-04-02T04:50:27.402643+0000","last_clean":"2024-04-02T04:44:01.856616+0000","last_became_active":"2024-04-02T04:49:47.381178+0000","last_became_peered":"2024-04-02T04:49:47.381178+0000","last_unstale":"2024-04-02T04:50:27.402643+0000","last_undegraded":"2024-04-02T04:49:47.359978+0000","last_fullsized":"2024-04-02T04:49:47.359801+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:00.356122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.356122+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":263,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.410656+0000","last_change":"2024-04-02T04:50:12.312248+0000","last_active":"2024-04-02T04:50:27.410656+0000","last_peered":"2024-04-02T04:50:27.410656+0000","last_clean":"2024-04-02T04:50:27.410656+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:50:27.410656+0000","last_undegraded":"2024-04-02T04:50:27.410656+0000","last_fullsized":"2024-04-02T04:50:27.410656+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:12.312147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.312147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.986984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.376280+0000","last_change":"2024-04-02T04:49:47.438287+0000","last_active":"2024-04-02T04:50:27.376280+0000","last_peered":"2024-04-02T04:50:27.376280+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.438287+0000","last_became_peered":"2024-04-02T04:49:47.438287+0000","last_unstale":"2024-04-02T04:50:27.376280+0000","last_undegraded":"2024-04-02T04:50:27.376280+0000","last_fullsized":"2024-04-02T04:49:47.386231+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.437949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.437949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.007314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"138'1","reported_seq":127,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.416017+0000","last_change":"2024-04-02T04:49:47.453712+0000","last_active":"2024-04-02T04:50:27.416017+0000","last_peered":"2024-04-02T04:50:27.416017+0000","last_clean":"2024-04-02T04:49:45.363008+0000","last_became_active":"2024-04-02T04:49:47.453712+0000","last_became_peered":"2024-04-02T04:49:47.453712+0000","last_unstale":"2024-04-02T04:50:27.416017+0000","last_undegraded":"2024-04-02T04:49:47.388056+0000","last_fullsized":"2024-04-02T04:49:47.387907+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'1","last_scrub_stamp":"2024-04-02T04:49:30.760395+0000","last_deep_scrub":"138'1","last_deep_scrub_stamp":"2024-04-02T04:49:30.760395+0000","last_clean_scrub_stamp":"2024-04-02T04:49:30.760395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested deep scrub scheduled @ 2024-04-02T04:50:28.842831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.434222+0000","last_change":"2024-04-02T04:49:26.277902+0000","last_active":"2024-04-02T04:50:27.434222+0000","last_peered":"2024-04-02T04:50:27.434222+0000","last_clean":"2024-04-02T04:50:27.434222+0000","last_became_active":"2024-04-02T04:49:26.271155+0000","last_became_peered":"2024-04-02T04:49:26.271155+0000","last_unstale":"2024-04-02T04:50:27.434222+0000","last_undegraded":"2024-04-02T04:50:27.434222+0000","last_fullsized":"2024-04-02T04:50:27.434222+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.714127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.396420+0000","last_change":"2024-04-02T04:49:26.266092+0000","last_active":"2024-04-02T04:50:27.396420+0000","last_peered":"2024-04-02T04:50:27.396420+0000","last_clean":"2024-04-02T04:50:27.396420+0000","last_became_active":"2024-04-02T04:49:26.265834+0000","last_became_peered":"2024-04-02T04:49:26.265834+0000","last_unstale":"2024-04-02T04:50:27.396420+0000","last_undegraded":"2024-04-02T04:50:27.396420+0000","last_fullsized":"2024-04-02T04:50:27.396420+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.729034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":306,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.409134+0000","last_change":"2024-04-02T04:49:10.763649+0000","last_active":"2024-04-02T04:50:27.409134+0000","last_peered":"2024-04-02T04:50:27.409134+0000","last_clean":"2024-04-02T04:50:27.409134+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:50:27.409134+0000","last_undegraded":"2024-04-02T04:50:27.409134+0000","last_fullsized":"2024-04-02T04:50:27.409134+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:49:10.763404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.763404+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.679681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.387010+0000","last_change":"2024-04-02T04:49:47.385876+0000","last_active":"2024-04-02T04:50:27.387010+0000","last_peered":"2024-04-02T04:50:27.387010+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.385876+0000","last_became_peered":"2024-04-02T04:49:47.385876+0000","last_unstale":"2024-04-02T04:50:27.387010+0000","last_undegraded":"2024-04-02T04:50:27.387010+0000","last_fullsized":"2024-04-02T04:49:47.345306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.439457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.439457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":346,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432507+0000","last_change":"2024-04-02T04:50:15.516760+0000","last_active":"2024-04-02T04:50:27.432507+0000","last_peered":"2024-04-02T04:50:27.432507+0000","last_clean":"2024-04-02T04:50:27.432507+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:50:27.432507+0000","last_undegraded":"2024-04-02T04:50:27.432507+0000","last_fullsized":"2024-04-02T04:50:27.432507+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:50:15.516538+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.516538+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.144029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.445261+0000","last_change":"2024-04-02T04:50:13.288238+0000","last_active":"2024-04-02T04:50:27.445261+0000","last_peered":"2024-04-02T04:50:27.445261+0000","last_clean":"2024-04-02T04:50:27.445261+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:50:27.445261+0000","last_undegraded":"2024-04-02T04:50:27.445261+0000","last_fullsized":"2024-04-02T04:50:27.445261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:13.288136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.288136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:13.866854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":133,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.416005+0000","last_change":"2024-04-02T04:50:01.028518+0000","last_active":"2024-04-02T04:50:27.416005+0000","last_peered":"2024-04-02T04:50:27.416005+0000","last_clean":"2024-04-02T04:50:27.416005+0000","last_became_active":"2024-04-02T04:49:25.130306+0000","last_became_peered":"2024-04-02T04:49:25.130306+0000","last_unstale":"2024-04-02T04:50:27.416005+0000","last_undegraded":"2024-04-02T04:50:27.416005+0000","last_fullsized":"2024-04-02T04:50:27.416005+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:01.028336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:01.028336+0000","last_clean_scrub_stamp":"2024-04-02T04:50:01.028336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:15.612397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.431700+0000","last_change":"2024-04-02T04:49:26.242898+0000","last_active":"2024-04-02T04:50:27.431700+0000","last_peered":"2024-04-02T04:50:27.431700+0000","last_clean":"2024-04-02T04:50:27.431700+0000","last_became_active":"2024-04-02T04:49:26.241469+0000","last_became_peered":"2024-04-02T04:49:26.241469+0000","last_unstale":"2024-04-02T04:50:27.431700+0000","last_undegraded":"2024-04-02T04:50:27.431700+0000","last_fullsized":"2024-04-02T04:50:27.431700+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389142+0000","last_change":"2024-04-02T04:49:26.312549+0000","last_active":"2024-04-02T04:50:27.389142+0000","last_peered":"2024-04-02T04:50:27.389142+0000","last_clean":"2024-04-02T04:50:27.389142+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:50:27.389142+0000","last_undegraded":"2024-04-02T04:50:27.389142+0000","last_fullsized":"2024-04-02T04:50:27.389142+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.194362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.417676+0000","last_change":"2024-04-02T04:48:52.546612+0000","last_active":"2024-04-02T04:50:27.417676+0000","last_peered":"2024-04-02T04:50:27.417676+0000","last_clean":"2024-04-02T04:50:27.417676+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:50:27.417676+0000","last_undegraded":"2024-04-02T04:50:27.417676+0000","last_fullsized":"2024-04-02T04:50:27.417676+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.546510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.546510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":294,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.387974+0000","last_change":"2024-04-02T04:50:17.228750+0000","last_active":"2024-04-02T04:50:27.387974+0000","last_peered":"2024-04-02T04:50:27.387974+0000","last_clean":"2024-04-02T04:50:27.387974+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:50:27.387974+0000","last_undegraded":"2024-04-02T04:50:27.387974+0000","last_fullsized":"2024-04-02T04:50:27.387974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:50:17.228650+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.228650+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:44.745235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.389334+0000","last_change":"2024-04-02T04:49:47.504164+0000","last_active":"2024-04-02T04:50:27.389334+0000","last_peered":"2024-04-02T04:50:27.389334+0000","last_clean":"2024-04-02T04:49:45.357271+0000","last_became_active":"2024-04-02T04:49:47.504164+0000","last_became_peered":"2024-04-02T04:49:47.504164+0000","last_unstale":"2024-04-02T04:50:27.389334+0000","last_undegraded":"2024-04-02T04:50:27.389334+0000","last_fullsized":"2024-04-02T04:49:47.406612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.071225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":252,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432790+0000","last_change":"2024-04-02T04:50:18.501638+0000","last_active":"2024-04-02T04:50:27.432790+0000","last_peered":"2024-04-02T04:50:27.432790+0000","last_clean":"2024-04-02T04:50:27.432790+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:50:27.432790+0000","last_undegraded":"2024-04-02T04:50:27.432790+0000","last_fullsized":"2024-04-02T04:50:27.432790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.501528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.501528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:42.986033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.7","version":"163'7","reported_seq":150,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.422695+0000","last_change":"2024-04-02T04:50:00.295724+0000","last_active":"2024-04-02T04:50:27.422695+0000","last_peered":"2024-04-02T04:50:27.422695+0000","last_clean":"2024-04-02T04:50:27.422695+0000","last_became_active":"2024-04-02T04:49:26.311168+0000","last_became_peered":"2024-04-02T04:49:26.311168+0000","last_unstale":"2024-04-02T04:50:27.422695+0000","last_undegraded":"2024-04-02T04:50:27.422695+0000","last_fullsized":"2024-04-02T04:50:27.422695+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'7","last_scrub_stamp":"2024-04-02T04:50:00.295560+0000","last_deep_scrub":"163'7","last_deep_scrub_stamp":"2024-04-02T04:50:00.295560+0000","last_clean_scrub_stamp":"2024-04-02T04:50:00.295560+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:29.054311+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":51902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.6","version":"139'9","reported_seq":122,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428329+0000","last_change":"2024-04-02T04:49:26.487620+0000","last_active":"2024-04-02T04:50:27.428329+0000","last_peered":"2024-04-02T04:50:27.428329+0000","last_clean":"2024-04-02T04:50:27.428329+0000","last_became_active":"2024-04-02T04:49:26.487105+0000","last_became_peered":"2024-04-02T04:49:26.487105+0000","last_unstale":"2024-04-02T04:50:27.428329+0000","last_undegraded":"2024-04-02T04:50:27.428329+0000","last_fullsized":"2024-04-02T04:50:27.428329+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":129,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.416005+0000","last_change":"2024-04-02T04:49:47.408871+0000","last_active":"2024-04-02T04:50:27.416005+0000","last_peered":"2024-04-02T04:50:27.416005+0000","last_clean":"2024-04-02T04:49:45.308505+0000","last_became_active":"2024-04-02T04:49:47.408871+0000","last_became_peered":"2024-04-02T04:49:47.408871+0000","last_unstale":"2024-04-02T04:50:27.416005+0000","last_undegraded":"2024-04-02T04:49:47.350015+0000","last_fullsized":"2024-04-02T04:49:47.349865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.718573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":323,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.388309+0000","last_change":"2024-04-02T04:49:47.511581+0000","last_active":"2024-04-02T04:50:27.388309+0000","last_peered":"2024-04-02T04:50:27.388309+0000","last_clean":"2024-04-02T04:49:45.312272+0000","last_became_active":"2024-04-02T04:49:47.511581+0000","last_became_peered":"2024-04-02T04:49:47.511581+0000","last_unstale":"2024-04-02T04:50:27.388309+0000","last_undegraded":"2024-04-02T04:49:47.361633+0000","last_fullsized":"2024-04-02T04:49:47.361362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:49:24.269994+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.269994+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":271,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375407+0000","last_change":"2024-04-02T04:49:15.013882+0000","last_active":"2024-04-02T04:50:27.375407+0000","last_peered":"2024-04-02T04:50:27.375407+0000","last_clean":"2024-04-02T04:50:27.375407+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:50:27.375407+0000","last_undegraded":"2024-04-02T04:50:27.375407+0000","last_fullsized":"2024-04-02T04:50:27.375407+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.013758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.013758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.629763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":197,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.445329+0000","last_change":"2024-04-02T04:49:19.588206+0000","last_active":"2024-04-02T04:50:27.445329+0000","last_peered":"2024-04-02T04:50:27.445329+0000","last_clean":"2024-04-02T04:50:27.445329+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:50:27.445329+0000","last_undegraded":"2024-04-02T04:50:27.445329+0000","last_fullsized":"2024-04-02T04:50:27.445329+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:19.587996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:19.587996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:45.975007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":251,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.403212+0000","last_change":"2024-04-02T04:49:00.820022+0000","last_active":"2024-04-02T04:50:27.403212+0000","last_peered":"2024-04-02T04:50:27.403212+0000","last_clean":"2024-04-02T04:50:27.403212+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:50:27.403212+0000","last_undegraded":"2024-04-02T04:50:27.403212+0000","last_fullsized":"2024-04-02T04:50:27.403212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.819876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:34.755051+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.819876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"164'4","reported_seq":142,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404201+0000","last_change":"2024-04-02T04:50:01.296134+0000","last_active":"2024-04-02T04:50:27.404201+0000","last_peered":"2024-04-02T04:50:27.404201+0000","last_clean":"2024-04-02T04:50:27.404201+0000","last_became_active":"2024-04-02T04:49:26.361801+0000","last_became_peered":"2024-04-02T04:49:26.361801+0000","last_unstale":"2024-04-02T04:50:27.404201+0000","last_undegraded":"2024-04-02T04:50:27.404201+0000","last_fullsized":"2024-04-02T04:50:27.404201+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4","last_scrub_stamp":"2024-04-02T04:50:01.296025+0000","last_deep_scrub":"164'4","last_deep_scrub_stamp":"2024-04-02T04:50:01.296025+0000","last_clean_scrub_stamp":"2024-04-02T04:50:01.296025+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:12.400525+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":82,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.397384+0000","last_change":"2024-04-02T04:49:47.396455+0000","last_active":"2024-04-02T04:50:27.397384+0000","last_peered":"2024-04-02T04:50:27.397384+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.396455+0000","last_became_peered":"2024-04-02T04:49:47.396455+0000","last_unstale":"2024-04-02T04:50:27.397384+0000","last_undegraded":"2024-04-02T04:50:27.397384+0000","last_fullsized":"2024-04-02T04:49:47.375223+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:41.712329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.387758+0000","last_change":"2024-04-02T04:49:26.252236+0000","last_active":"2024-04-02T04:50:27.387758+0000","last_peered":"2024-04-02T04:50:27.387758+0000","last_clean":"2024-04-02T04:50:27.387758+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:50:27.387758+0000","last_undegraded":"2024-04-02T04:50:27.387758+0000","last_fullsized":"2024-04-02T04:50:27.387758+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:34.080744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":303,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.417264+0000","last_change":"2024-04-02T04:49:47.453493+0000","last_active":"2024-04-02T04:50:27.417264+0000","last_peered":"2024-04-02T04:50:27.417264+0000","last_clean":"2024-04-02T04:49:45.308750+0000","last_became_active":"2024-04-02T04:49:47.453493+0000","last_became_peered":"2024-04-02T04:49:47.453493+0000","last_unstale":"2024-04-02T04:50:27.417264+0000","last_undegraded":"2024-04-02T04:49:47.351067+0000","last_fullsized":"2024-04-02T04:49:47.350938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:48:49.534769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.534769+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.561606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.403908+0000","last_change":"2024-04-02T04:49:47.521979+0000","last_active":"2024-04-02T04:50:27.403908+0000","last_peered":"2024-04-02T04:50:27.403908+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.521979+0000","last_became_peered":"2024-04-02T04:49:47.521979+0000","last_unstale":"2024-04-02T04:50:27.403908+0000","last_undegraded":"2024-04-02T04:50:27.403908+0000","last_fullsized":"2024-04-02T04:49:47.391120+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.344546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":337,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432111+0000","last_change":"2024-04-02T04:49:04.648126+0000","last_active":"2024-04-02T04:50:27.432111+0000","last_peered":"2024-04-02T04:50:27.432111+0000","last_clean":"2024-04-02T04:50:27.432111+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:50:27.432111+0000","last_undegraded":"2024-04-02T04:50:27.432111+0000","last_fullsized":"2024-04-02T04:50:27.432111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:49:04.647949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.647949+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":245,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377570+0000","last_change":"2024-04-02T04:48:45.485574+0000","last_active":"2024-04-02T04:50:27.377570+0000","last_peered":"2024-04-02T04:50:27.377570+0000","last_clean":"2024-04-02T04:50:27.377570+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:50:27.377570+0000","last_undegraded":"2024-04-02T04:50:27.377570+0000","last_fullsized":"2024-04-02T04:50:27.377570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.485441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.485441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"168'6","reported_seq":141,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432773+0000","last_change":"2024-04-02T04:50:06.842874+0000","last_active":"2024-04-02T04:50:27.432773+0000","last_peered":"2024-04-02T04:50:27.432773+0000","last_clean":"2024-04-02T04:50:27.432773+0000","last_became_active":"2024-04-02T04:49:26.261364+0000","last_became_peered":"2024-04-02T04:49:26.261364+0000","last_unstale":"2024-04-02T04:50:27.432773+0000","last_undegraded":"2024-04-02T04:50:27.432773+0000","last_fullsized":"2024-04-02T04:50:27.432773+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'6","last_scrub_stamp":"2024-04-02T04:50:06.842743+0000","last_deep_scrub":"168'6","last_deep_scrub_stamp":"2024-04-02T04:50:06.842743+0000","last_clean_scrub_stamp":"2024-04-02T04:50:06.842743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:11.430117+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":115,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.357830+0000","last_change":"2024-04-02T04:49:26.319975+0000","last_active":"2024-04-02T04:50:27.357830+0000","last_peered":"2024-04-02T04:50:27.357830+0000","last_clean":"2024-04-02T04:50:27.357830+0000","last_became_active":"2024-04-02T04:49:26.319266+0000","last_became_peered":"2024-04-02T04:49:26.319266+0000","last_unstale":"2024-04-02T04:50:27.357830+0000","last_undegraded":"2024-04-02T04:50:27.357830+0000","last_fullsized":"2024-04-02T04:50:27.357830+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.449699+0000","last_change":"2024-04-02T04:49:26.274127+0000","last_active":"2024-04-02T04:50:27.449699+0000","last_peered":"2024-04-02T04:50:27.449699+0000","last_clean":"2024-04-02T04:50:27.449699+0000","last_became_active":"2024-04-02T04:49:26.255429+0000","last_became_peered":"2024-04-02T04:49:26.255429+0000","last_unstale":"2024-04-02T04:50:27.449699+0000","last_undegraded":"2024-04-02T04:50:27.449699+0000","last_fullsized":"2024-04-02T04:50:27.449699+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":249,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.417537+0000","last_change":"2024-04-02T04:49:13.637761+0000","last_active":"2024-04-02T04:50:27.417537+0000","last_peered":"2024-04-02T04:50:27.417537+0000","last_clean":"2024-04-02T04:50:27.417537+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:50:27.417537+0000","last_undegraded":"2024-04-02T04:50:27.417537+0000","last_fullsized":"2024-04-02T04:50:27.417537+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:13.637518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:13.637518+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.637518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":272,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.415060+0000","last_change":"2024-04-02T04:48:43.043307+0000","last_active":"2024-04-02T04:50:27.415060+0000","last_peered":"2024-04-02T04:50:27.415060+0000","last_clean":"2024-04-02T04:50:27.415060+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:50:27.415060+0000","last_undegraded":"2024-04-02T04:50:27.415060+0000","last_fullsized":"2024-04-02T04:50:27.415060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:48:43.043155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:43.043155+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":269,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.449919+0000","last_change":"2024-04-02T04:49:47.470291+0000","last_active":"2024-04-02T04:50:27.449919+0000","last_peered":"2024-04-02T04:50:27.449919+0000","last_clean":"2024-04-02T04:49:45.387254+0000","last_became_active":"2024-04-02T04:49:47.470291+0000","last_became_peered":"2024-04-02T04:49:47.470291+0000","last_unstale":"2024-04-02T04:50:27.449919+0000","last_undegraded":"2024-04-02T04:50:27.449919+0000","last_fullsized":"2024-04-02T04:49:47.418421+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.690362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.690362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.702707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":630,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.388650+0000","last_change":"2024-04-02T04:48:53.454617+0000","last_active":"2024-04-02T04:50:27.388650+0000","last_peered":"2024-04-02T04:50:27.388650+0000","last_clean":"2024-04-02T04:50:27.388650+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:50:27.388650+0000","last_undegraded":"2024-04-02T04:50:27.388650+0000","last_fullsized":"2024-04-02T04:50:27.388650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:48:53.454462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:53.454462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.428321+0000","last_change":"2024-04-02T04:49:47.470355+0000","last_active":"2024-04-02T04:50:27.428321+0000","last_peered":"2024-04-02T04:50:27.428321+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.470355+0000","last_became_peered":"2024-04-02T04:49:47.470355+0000","last_unstale":"2024-04-02T04:50:27.428321+0000","last_undegraded":"2024-04-02T04:50:27.428321+0000","last_fullsized":"2024-04-02T04:49:47.399308+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.a","version":"138'1","reported_seq":122,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.399113+0000","last_change":"2024-04-02T04:50:19.176673+0000","last_active":"2024-04-02T04:50:27.399113+0000","last_peered":"2024-04-02T04:50:27.399113+0000","last_clean":"2024-04-02T04:50:27.399113+0000","last_became_active":"2024-04-02T04:49:26.311848+0000","last_became_peered":"2024-04-02T04:49:26.311848+0000","last_unstale":"2024-04-02T04:50:27.399113+0000","last_undegraded":"2024-04-02T04:50:27.399113+0000","last_fullsized":"2024-04-02T04:50:27.399113+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'1","last_scrub_stamp":"2024-04-02T04:50:19.176468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.176468+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:36.402780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.367857+0000","last_change":"2024-04-02T04:49:26.343335+0000","last_active":"2024-04-02T04:50:27.367857+0000","last_peered":"2024-04-02T04:50:27.367857+0000","last_clean":"2024-04-02T04:50:27.367857+0000","last_became_active":"2024-04-02T04:49:26.343060+0000","last_became_peered":"2024-04-02T04:49:26.343060+0000","last_unstale":"2024-04-02T04:50:27.367857+0000","last_undegraded":"2024-04-02T04:50:27.367857+0000","last_fullsized":"2024-04-02T04:50:27.367857+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.833519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428624+0000","last_change":"2024-04-02T04:49:26.257833+0000","last_active":"2024-04-02T04:50:27.428624+0000","last_peered":"2024-04-02T04:50:27.428624+0000","last_clean":"2024-04-02T04:50:27.428624+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:50:27.428624+0000","last_undegraded":"2024-04-02T04:50:27.428624+0000","last_fullsized":"2024-04-02T04:50:27.428624+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.955371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"153'13","reported_seq":352,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.388429+0000","last_change":"2024-04-02T04:49:47.488932+0000","last_active":"2024-04-02T04:50:27.388429+0000","last_peered":"2024-04-02T04:50:27.388429+0000","last_clean":"2024-04-02T04:49:45.312370+0000","last_became_active":"2024-04-02T04:49:47.488932+0000","last_became_peered":"2024-04-02T04:49:47.488932+0000","last_unstale":"2024-04-02T04:50:27.388429+0000","last_undegraded":"2024-04-02T04:49:47.362506+0000","last_fullsized":"2024-04-02T04:49:47.362177+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:48:38.477825+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.477825+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":296,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.394200+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:50:27.394200+0000","last_peered":"2024-04-02T04:50:27.394200+0000","last_clean":"2024-04-02T04:49:45.321982+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:50:27.394200+0000","last_undegraded":"2024-04-02T04:49:47.406158+0000","last_fullsized":"2024-04-02T04:49:47.406050+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:49:07.374876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.374876+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.407872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":267,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.420400+0000","last_change":"2024-04-02T04:49:21.056756+0000","last_active":"2024-04-02T04:50:27.420400+0000","last_peered":"2024-04-02T04:50:27.420400+0000","last_clean":"2024-04-02T04:50:27.420400+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:50:27.420400+0000","last_undegraded":"2024-04-02T04:50:27.420400+0000","last_fullsized":"2024-04-02T04:50:27.420400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:49:21.056557+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:49:21.056557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.416507+0000","last_change":"2024-04-02T04:49:47.479633+0000","last_active":"2024-04-02T04:50:27.416507+0000","last_peered":"2024-04-02T04:50:27.416507+0000","last_clean":"2024-04-02T04:49:45.308628+0000","last_became_active":"2024-04-02T04:49:47.479633+0000","last_became_peered":"2024-04-02T04:49:47.479633+0000","last_unstale":"2024-04-02T04:50:27.416507+0000","last_undegraded":"2024-04-02T04:50:27.416507+0000","last_fullsized":"2024-04-02T04:49:47.350430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.555854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.555854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":245,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411575+0000","last_change":"2024-04-02T04:49:06.652181+0000","last_active":"2024-04-02T04:50:27.411575+0000","last_peered":"2024-04-02T04:50:27.411575+0000","last_clean":"2024-04-02T04:50:27.411575+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:50:27.411575+0000","last_undegraded":"2024-04-02T04:50:27.411575+0000","last_fullsized":"2024-04-02T04:50:27.411575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.651924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.651924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":121,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.396256+0000","last_change":"2024-04-02T04:50:18.190140+0000","last_active":"2024-04-02T04:50:27.396256+0000","last_peered":"2024-04-02T04:50:27.396256+0000","last_clean":"2024-04-02T04:50:27.396256+0000","last_became_active":"2024-04-02T04:49:26.293911+0000","last_became_peered":"2024-04-02T04:49:26.293911+0000","last_unstale":"2024-04-02T04:50:27.396256+0000","last_undegraded":"2024-04-02T04:50:27.396256+0000","last_fullsized":"2024-04-02T04:50:27.396256+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.189971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.189971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.631891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.a","version":"139'10","reported_seq":123,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.399009+0000","last_change":"2024-04-02T04:49:26.363291+0000","last_active":"2024-04-02T04:50:27.399009+0000","last_peered":"2024-04-02T04:50:27.399009+0000","last_clean":"2024-04-02T04:50:27.399009+0000","last_became_active":"2024-04-02T04:49:26.341161+0000","last_became_peered":"2024-04-02T04:49:26.341161+0000","last_unstale":"2024-04-02T04:50:27.399009+0000","last_undegraded":"2024-04-02T04:50:27.399009+0000","last_fullsized":"2024-04-02T04:50:27.399009+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.612361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432640+0000","last_change":"2024-04-02T04:49:26.186537+0000","last_active":"2024-04-02T04:50:27.432640+0000","last_peered":"2024-04-02T04:50:27.432640+0000","last_clean":"2024-04-02T04:50:27.432640+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:50:27.432640+0000","last_undegraded":"2024-04-02T04:50:27.432640+0000","last_fullsized":"2024-04-02T04:50:27.432640+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"153'13","reported_seq":352,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.416314+0000","last_change":"2024-04-02T04:49:47.472920+0000","last_active":"2024-04-02T04:50:27.416314+0000","last_peered":"2024-04-02T04:50:27.416314+0000","last_clean":"2024-04-02T04:49:45.374829+0000","last_became_active":"2024-04-02T04:49:47.472920+0000","last_became_peered":"2024-04-02T04:49:47.472920+0000","last_unstale":"2024-04-02T04:50:27.416314+0000","last_undegraded":"2024-04-02T04:49:47.384585+0000","last_fullsized":"2024-04-02T04:49:47.384414+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.546903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":421,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405269+0000","last_change":"2024-04-02T04:50:20.307539+0000","last_active":"2024-04-02T04:50:27.405269+0000","last_peered":"2024-04-02T04:50:27.405269+0000","last_clean":"2024-04-02T04:50:27.405269+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:50:27.405269+0000","last_undegraded":"2024-04-02T04:50:27.405269+0000","last_fullsized":"2024-04-02T04:50:27.405269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:50:20.307283+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.307283+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:24.958471+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.410874+0000","last_change":"2024-04-02T04:49:47.431753+0000","last_active":"2024-04-02T04:50:27.410874+0000","last_peered":"2024-04-02T04:50:27.410874+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.431753+0000","last_became_peered":"2024-04-02T04:49:47.431753+0000","last_unstale":"2024-04-02T04:50:27.410874+0000","last_undegraded":"2024-04-02T04:50:27.410874+0000","last_fullsized":"2024-04-02T04:49:47.405769+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":267,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.408839+0000","last_change":"2024-04-02T04:49:15.756968+0000","last_active":"2024-04-02T04:50:27.408839+0000","last_peered":"2024-04-02T04:50:27.408839+0000","last_clean":"2024-04-02T04:50:27.408839+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:50:27.408839+0000","last_undegraded":"2024-04-02T04:50:27.408839+0000","last_fullsized":"2024-04-02T04:50:27.408839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:49:15.756715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.756715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.794192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":253,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375668+0000","last_change":"2024-04-02T04:50:09.280418+0000","last_active":"2024-04-02T04:50:27.375668+0000","last_peered":"2024-04-02T04:50:27.375668+0000","last_clean":"2024-04-02T04:50:27.375668+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:50:27.375668+0000","last_undegraded":"2024-04-02T04:50:27.375668+0000","last_fullsized":"2024-04-02T04:50:27.375668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.280337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.280337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:37.314038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.8","version":"167'1006","reported_seq":2141,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.357715+0000","last_change":"2024-04-02T04:50:05.915610+0000","last_active":"2024-04-02T04:50:27.357715+0000","last_peered":"2024-04-02T04:50:27.357715+0000","last_clean":"2024-04-02T04:50:27.357715+0000","last_became_active":"2024-04-02T04:49:26.275687+0000","last_became_peered":"2024-04-02T04:49:26.275687+0000","last_unstale":"2024-04-02T04:50:27.357715+0000","last_undegraded":"2024-04-02T04:50:27.357715+0000","last_fullsized":"2024-04-02T04:50:27.357715+0000","mapping_epoch":135,"log_start":"138'200","ondisk_log_start":"138'200","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'1006","last_scrub_stamp":"2024-04-02T04:50:05.915183+0000","last_deep_scrub":"167'1006","last_deep_scrub_stamp":"2024-04-02T04:50:05.915183+0000","last_clean_scrub_stamp":"2024-04-02T04:50:05.915183+0000","objects_scrubbed":3,"log_size":806,"log_dups_size":200,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.137810+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1003,"num_read_kb":3,"num_write":1002,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":117,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.404072+0000","last_change":"2024-04-02T04:49:47.516900+0000","last_active":"2024-04-02T04:50:27.404072+0000","last_peered":"2024-04-02T04:50:27.404072+0000","last_clean":"2024-04-02T04:49:45.354881+0000","last_became_active":"2024-04-02T04:49:47.516900+0000","last_became_peered":"2024-04-02T04:49:47.516900+0000","last_unstale":"2024-04-02T04:50:27.404072+0000","last_undegraded":"2024-04-02T04:50:27.404072+0000","last_fullsized":"2024-04-02T04:49:47.391507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.318297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":264,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.416461+0000","last_change":"2024-04-02T04:50:13.057737+0000","last_active":"2024-04-02T04:50:27.416461+0000","last_peered":"2024-04-02T04:50:27.416461+0000","last_clean":"2024-04-02T04:50:27.416461+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:50:27.416461+0000","last_undegraded":"2024-04-02T04:50:27.416461+0000","last_fullsized":"2024-04-02T04:50:27.416461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:13.057654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.057654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:33.157077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.371901+0000","last_change":"2024-04-02T04:49:47.509591+0000","last_active":"2024-04-02T04:50:27.371901+0000","last_peered":"2024-04-02T04:50:27.371901+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.509591+0000","last_became_peered":"2024-04-02T04:49:47.509591+0000","last_unstale":"2024-04-02T04:50:27.371901+0000","last_undegraded":"2024-04-02T04:50:27.371901+0000","last_fullsized":"2024-04-02T04:49:47.387091+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.441320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.441320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.338902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":295,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.384679+0000","last_change":"2024-04-02T04:50:19.055361+0000","last_active":"2024-04-02T04:50:27.384679+0000","last_peered":"2024-04-02T04:50:27.384679+0000","last_clean":"2024-04-02T04:50:27.384679+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:50:27.384679+0000","last_undegraded":"2024-04-02T04:50:27.384679+0000","last_fullsized":"2024-04-02T04:50:27.384679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:50:19.055250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.055250+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.015434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":261,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404802+0000","last_change":"2024-04-02T04:50:26.342752+0000","last_active":"2024-04-02T04:50:27.404802+0000","last_peered":"2024-04-02T04:50:27.404802+0000","last_clean":"2024-04-02T04:50:27.404802+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:50:27.404802+0000","last_undegraded":"2024-04-02T04:50:27.404802+0000","last_fullsized":"2024-04-02T04:50:27.404802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.342429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.342429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.858608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.445870+0000","last_change":"2024-04-02T04:49:47.470287+0000","last_active":"2024-04-02T04:50:27.445870+0000","last_peered":"2024-04-02T04:50:27.445870+0000","last_clean":"2024-04-02T04:49:45.386144+0000","last_became_active":"2024-04-02T04:49:47.470287+0000","last_became_peered":"2024-04-02T04:49:47.470287+0000","last_unstale":"2024-04-02T04:50:27.445870+0000","last_undegraded":"2024-04-02T04:50:27.445870+0000","last_fullsized":"2024-04-02T04:49:47.418077+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.841448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"138'1","reported_seq":90,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.432637+0000","last_change":"2024-04-02T04:49:47.490664+0000","last_active":"2024-04-02T04:50:27.432637+0000","last_peered":"2024-04-02T04:50:27.432637+0000","last_clean":"2024-04-02T04:49:26.267044+0000","last_became_active":"2024-04-02T04:49:47.490664+0000","last_became_peered":"2024-04-02T04:49:47.490664+0000","last_unstale":"2024-04-02T04:50:27.432637+0000","last_undegraded":"2024-04-02T04:49:47.386184+0000","last_fullsized":"2024-04-02T04:49:47.383669+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.287027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":121,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.422943+0000","last_change":"2024-04-02T04:50:24.860538+0000","last_active":"2024-04-02T04:50:27.422943+0000","last_peered":"2024-04-02T04:50:27.422943+0000","last_clean":"2024-04-02T04:50:27.422943+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:50:27.422943+0000","last_undegraded":"2024-04-02T04:50:27.422943+0000","last_fullsized":"2024-04-02T04:50:27.422943+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:24.860246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:24.860246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.486260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":261,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.414539+0000","last_change":"2024-04-02T04:49:47.413615+0000","last_active":"2024-04-02T04:50:27.414539+0000","last_peered":"2024-04-02T04:50:27.414539+0000","last_clean":"2024-04-02T04:49:45.390373+0000","last_became_active":"2024-04-02T04:49:47.413615+0000","last_became_peered":"2024-04-02T04:49:47.413615+0000","last_unstale":"2024-04-02T04:50:27.414539+0000","last_undegraded":"2024-04-02T04:50:27.414539+0000","last_fullsized":"2024-04-02T04:49:47.396592+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.569058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.569058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.820777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":390,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.403604+0000","last_change":"2024-04-02T04:49:47.451735+0000","last_active":"2024-04-02T04:50:27.403604+0000","last_peered":"2024-04-02T04:50:27.403604+0000","last_clean":"2024-04-02T04:44:53.853321+0000","last_became_active":"2024-04-02T04:49:47.451735+0000","last_became_peered":"2024-04-02T04:49:47.451735+0000","last_unstale":"2024-04-02T04:50:27.403604+0000","last_undegraded":"2024-04-02T04:49:47.390635+0000","last_fullsized":"2024-04-02T04:49:47.390485+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.459185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.403118+0000","last_change":"2024-04-02T04:49:47.435042+0000","last_active":"2024-04-02T04:50:27.403118+0000","last_peered":"2024-04-02T04:50:27.403118+0000","last_clean":"2024-04-02T04:49:45.341176+0000","last_became_active":"2024-04-02T04:49:47.435042+0000","last_became_peered":"2024-04-02T04:49:47.435042+0000","last_unstale":"2024-04-02T04:50:27.403118+0000","last_undegraded":"2024-04-02T04:50:27.403118+0000","last_fullsized":"2024-04-02T04:49:47.351952+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:39.903683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.903683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":281,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.358900+0000","last_change":"2024-04-02T04:49:04.488216+0000","last_active":"2024-04-02T04:50:27.358900+0000","last_peered":"2024-04-02T04:50:27.358900+0000","last_clean":"2024-04-02T04:50:27.358900+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:50:27.358900+0000","last_undegraded":"2024-04-02T04:50:27.358900+0000","last_fullsized":"2024-04-02T04:50:27.358900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:49:04.488030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.488030+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.470177+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":251,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.400280+0000","last_change":"2024-04-02T04:49:25.573578+0000","last_active":"2024-04-02T04:50:27.400280+0000","last_peered":"2024-04-02T04:50:27.400280+0000","last_clean":"2024-04-02T04:50:27.400280+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:50:27.400280+0000","last_undegraded":"2024-04-02T04:50:27.400280+0000","last_fullsized":"2024-04-02T04:50:27.400280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.573285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:53.245097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404631+0000","last_change":"2024-04-02T04:49:26.310768+0000","last_active":"2024-04-02T04:50:27.404631+0000","last_peered":"2024-04-02T04:50:27.404631+0000","last_clean":"2024-04-02T04:50:27.404631+0000","last_became_active":"2024-04-02T04:49:26.290196+0000","last_became_peered":"2024-04-02T04:49:26.290196+0000","last_unstale":"2024-04-02T04:50:27.404631+0000","last_undegraded":"2024-04-02T04:50:27.404631+0000","last_fullsized":"2024-04-02T04:50:27.404631+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:37.932029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":121,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432778+0000","last_change":"2024-04-02T04:50:27.009407+0000","last_active":"2024-04-02T04:50:27.432778+0000","last_peered":"2024-04-02T04:50:27.432778+0000","last_clean":"2024-04-02T04:50:27.432778+0000","last_became_active":"2024-04-02T04:49:26.286715+0000","last_became_peered":"2024-04-02T04:49:26.286715+0000","last_unstale":"2024-04-02T04:50:27.432778+0000","last_undegraded":"2024-04-02T04:50:27.432778+0000","last_fullsized":"2024-04-02T04:50:27.432778+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.009297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.009297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.023957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.376917+0000","last_change":"2024-04-02T04:49:47.382055+0000","last_active":"2024-04-02T04:50:27.376917+0000","last_peered":"2024-04-02T04:50:27.376917+0000","last_clean":"2024-04-02T04:49:45.332186+0000","last_became_active":"2024-04-02T04:49:47.382055+0000","last_became_peered":"2024-04-02T04:49:47.382055+0000","last_unstale":"2024-04-02T04:50:27.376917+0000","last_undegraded":"2024-04-02T04:50:27.376917+0000","last_fullsized":"2024-04-02T04:49:47.337410+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.422629+0000","last_change":"2024-04-02T04:50:11.329816+0000","last_active":"2024-04-02T04:50:27.422629+0000","last_peered":"2024-04-02T04:50:27.422629+0000","last_clean":"2024-04-02T04:50:27.422629+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:50:27.422629+0000","last_undegraded":"2024-04-02T04:50:27.422629+0000","last_fullsized":"2024-04-02T04:50:27.422629+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.329645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.329645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.297573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":272,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375771+0000","last_change":"2024-04-02T04:50:14.307591+0000","last_active":"2024-04-02T04:50:27.375771+0000","last_peered":"2024-04-02T04:50:27.375771+0000","last_clean":"2024-04-02T04:50:27.375771+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:50:27.375771+0000","last_undegraded":"2024-04-02T04:50:27.375771+0000","last_fullsized":"2024-04-02T04:50:27.375771+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:14.307478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.307478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.776147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":317,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.371732+0000","last_change":"2024-04-02T04:50:16.319646+0000","last_active":"2024-04-02T04:50:27.371732+0000","last_peered":"2024-04-02T04:50:27.371732+0000","last_clean":"2024-04-02T04:50:27.371732+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:50:27.371732+0000","last_undegraded":"2024-04-02T04:50:27.371732+0000","last_fullsized":"2024-04-02T04:50:27.371732+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:50:16.319556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:16.319556+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:16.400689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.433064+0000","last_change":"2024-04-02T04:49:47.492471+0000","last_active":"2024-04-02T04:50:27.433064+0000","last_peered":"2024-04-02T04:50:27.433064+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.492471+0000","last_became_peered":"2024-04-02T04:49:47.492471+0000","last_unstale":"2024-04-02T04:50:27.433064+0000","last_undegraded":"2024-04-02T04:50:27.433064+0000","last_fullsized":"2024-04-02T04:49:47.411395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":82,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.391176+0000","last_change":"2024-04-02T04:49:47.388497+0000","last_active":"2024-04-02T04:50:27.391176+0000","last_peered":"2024-04-02T04:50:27.391176+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.388497+0000","last_became_peered":"2024-04-02T04:49:47.388497+0000","last_unstale":"2024-04-02T04:50:27.391176+0000","last_undegraded":"2024-04-02T04:50:27.391176+0000","last_fullsized":"2024-04-02T04:49:47.356622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.306464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.c","version":"140'5","reported_seq":123,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.386335+0000","last_change":"2024-04-02T04:49:26.259801+0000","last_active":"2024-04-02T04:50:27.386335+0000","last_peered":"2024-04-02T04:50:27.386335+0000","last_clean":"2024-04-02T04:50:27.386335+0000","last_became_active":"2024-04-02T04:49:26.259515+0000","last_became_peered":"2024-04-02T04:49:26.259515+0000","last_unstale":"2024-04-02T04:50:27.386335+0000","last_undegraded":"2024-04-02T04:50:27.386335+0000","last_fullsized":"2024-04-02T04:50:27.386335+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.678243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2808,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":121,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.407837+0000","last_change":"2024-04-02T04:50:22.825218+0000","last_active":"2024-04-02T04:50:27.407837+0000","last_peered":"2024-04-02T04:50:27.407837+0000","last_clean":"2024-04-02T04:50:27.407837+0000","last_became_active":"2024-04-02T04:49:26.229137+0000","last_became_peered":"2024-04-02T04:49:26.229137+0000","last_unstale":"2024-04-02T04:50:27.407837+0000","last_undegraded":"2024-04-02T04:50:27.407837+0000","last_fullsized":"2024-04-02T04:50:27.407837+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:22.825000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:22.825000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.909236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404898+0000","last_change":"2024-04-02T04:49:17.093584+0000","last_active":"2024-04-02T04:50:27.404898+0000","last_peered":"2024-04-02T04:50:27.404898+0000","last_clean":"2024-04-02T04:50:27.404898+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:50:27.404898+0000","last_undegraded":"2024-04-02T04:50:27.404898+0000","last_fullsized":"2024-04-02T04:50:27.404898+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:17.093485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:49:17.093485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:34.662176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":309,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.410829+0000","last_change":"2024-04-02T04:50:12.546833+0000","last_active":"2024-04-02T04:50:27.410829+0000","last_peered":"2024-04-02T04:50:27.410829+0000","last_clean":"2024-04-02T04:50:27.410829+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:50:27.410829+0000","last_undegraded":"2024-04-02T04:50:27.410829+0000","last_fullsized":"2024-04-02T04:50:27.410829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:50:12.546683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.546683+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:34.121347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":264,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.367124+0000","last_change":"2024-04-02T04:50:15.550230+0000","last_active":"2024-04-02T04:50:27.367124+0000","last_peered":"2024-04-02T04:50:27.367124+0000","last_clean":"2024-04-02T04:50:27.367124+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:50:27.367124+0000","last_undegraded":"2024-04-02T04:50:27.367124+0000","last_fullsized":"2024-04-02T04:50:27.367124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.550133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.550133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:20.615818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":251,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.385523+0000","last_change":"2024-04-02T04:49:47.486462+0000","last_active":"2024-04-02T04:50:27.385523+0000","last_peered":"2024-04-02T04:50:27.385523+0000","last_clean":"2024-04-02T04:49:45.345774+0000","last_became_active":"2024-04-02T04:49:47.486462+0000","last_became_peered":"2024-04-02T04:49:47.486462+0000","last_unstale":"2024-04-02T04:50:27.385523+0000","last_undegraded":"2024-04-02T04:49:47.409192+0000","last_fullsized":"2024-04-02T04:49:47.409080+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:48:49.483159+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.483159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"139'5","reported_seq":121,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.428632+0000","last_change":"2024-04-02T04:49:47.490156+0000","last_active":"2024-04-02T04:50:27.428632+0000","last_peered":"2024-04-02T04:50:27.428632+0000","last_clean":"2024-04-02T04:49:45.349908+0000","last_became_active":"2024-04-02T04:49:47.490156+0000","last_became_peered":"2024-04-02T04:49:47.490156+0000","last_unstale":"2024-04-02T04:50:27.428632+0000","last_undegraded":"2024-04-02T04:49:47.382039+0000","last_fullsized":"2024-04-02T04:49:47.381919+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.408191+0000","last_change":"2024-04-02T04:49:47.400444+0000","last_active":"2024-04-02T04:50:27.408191+0000","last_peered":"2024-04-02T04:50:27.408191+0000","last_clean":"2024-04-02T04:49:45.390160+0000","last_became_active":"2024-04-02T04:49:47.400444+0000","last_became_peered":"2024-04-02T04:49:47.400444+0000","last_unstale":"2024-04-02T04:50:27.408191+0000","last_undegraded":"2024-04-02T04:50:27.408191+0000","last_fullsized":"2024-04-02T04:49:47.375552+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.820265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"138'2","reported_seq":115,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.402358+0000","last_change":"2024-04-02T04:49:26.270988+0000","last_active":"2024-04-02T04:50:27.402358+0000","last_peered":"2024-04-02T04:50:27.402358+0000","last_clean":"2024-04-02T04:50:27.402358+0000","last_became_active":"2024-04-02T04:49:26.270708+0000","last_became_peered":"2024-04-02T04:49:26.270708+0000","last_unstale":"2024-04-02T04:50:27.402358+0000","last_undegraded":"2024-04-02T04:50:27.402358+0000","last_fullsized":"2024-04-02T04:50:27.402358+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.582787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":82,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.418245+0000","last_change":"2024-04-02T04:49:47.428471+0000","last_active":"2024-04-02T04:50:27.418245+0000","last_peered":"2024-04-02T04:50:27.418245+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.428471+0000","last_became_peered":"2024-04-02T04:49:47.428471+0000","last_unstale":"2024-04-02T04:50:27.418245+0000","last_undegraded":"2024-04-02T04:50:27.418245+0000","last_fullsized":"2024-04-02T04:49:47.368822+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389673+0000","last_change":"2024-04-02T04:49:16.090440+0000","last_active":"2024-04-02T04:50:27.389673+0000","last_peered":"2024-04-02T04:50:27.389673+0000","last_clean":"2024-04-02T04:50:27.389673+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:50:27.389673+0000","last_undegraded":"2024-04-02T04:50:27.389673+0000","last_fullsized":"2024-04-02T04:50:27.389673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:16.090227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:16.090227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.146520+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":361,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.401966+0000","last_change":"2024-04-02T04:49:15.791519+0000","last_active":"2024-04-02T04:50:27.401966+0000","last_peered":"2024-04-02T04:50:27.401966+0000","last_clean":"2024-04-02T04:50:27.401966+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:50:27.401966+0000","last_undegraded":"2024-04-02T04:50:27.401966+0000","last_fullsized":"2024-04-02T04:50:27.401966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:49:15.791281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.791281+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:39.859753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":276,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.404993+0000","last_change":"2024-04-02T04:49:47.508665+0000","last_active":"2024-04-02T04:50:27.404993+0000","last_peered":"2024-04-02T04:50:27.404993+0000","last_clean":"2024-04-02T04:49:45.365160+0000","last_became_active":"2024-04-02T04:49:47.508665+0000","last_became_peered":"2024-04-02T04:49:47.508665+0000","last_unstale":"2024-04-02T04:50:27.404993+0000","last_undegraded":"2024-04-02T04:49:47.437341+0000","last_fullsized":"2024-04-02T04:49:47.437226+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:49:13.057821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.057821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.601208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":254,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.403503+0000","last_change":"2024-04-02T04:49:11.991346+0000","last_active":"2024-04-02T04:50:27.403503+0000","last_peered":"2024-04-02T04:50:27.403503+0000","last_clean":"2024-04-02T04:50:27.403503+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:50:27.403503+0000","last_undegraded":"2024-04-02T04:50:27.403503+0000","last_fullsized":"2024-04-02T04:50:27.403503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.991232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.991232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.193266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.9","version":"140'4","reported_seq":134,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.416498+0000","last_change":"2024-04-02T04:49:47.455899+0000","last_active":"2024-04-02T04:50:27.416498+0000","last_peered":"2024-04-02T04:50:27.416498+0000","last_clean":"2024-04-02T04:49:45.363108+0000","last_became_active":"2024-04-02T04:49:47.455899+0000","last_became_peered":"2024-04-02T04:49:47.455899+0000","last_unstale":"2024-04-02T04:50:27.416498+0000","last_undegraded":"2024-04-02T04:49:47.390457+0000","last_fullsized":"2024-04-02T04:49:47.390327+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'4","last_scrub_stamp":"2024-04-02T04:49:31.824957+0000","last_deep_scrub":"140'4","last_deep_scrub_stamp":"2024-04-02T04:49:31.824957+0000","last_clean_scrub_stamp":"2024-04-02T04:49:31.824957+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested deep scrub scheduled @ 2024-04-02T04:50:28.842859+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":27325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"139'5","reported_seq":96,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.371033+0000","last_change":"2024-04-02T04:49:47.503964+0000","last_active":"2024-04-02T04:50:27.371033+0000","last_peered":"2024-04-02T04:50:27.371033+0000","last_clean":"2024-04-02T04:49:27.480374+0000","last_became_active":"2024-04-02T04:49:47.503964+0000","last_became_peered":"2024-04-02T04:49:47.503964+0000","last_unstale":"2024-04-02T04:50:27.371033+0000","last_undegraded":"2024-04-02T04:49:47.388434+0000","last_fullsized":"2024-04-02T04:49:47.383380+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.449351+0000","last_change":"2024-04-02T04:49:26.199475+0000","last_active":"2024-04-02T04:50:27.449351+0000","last_peered":"2024-04-02T04:50:27.449351+0000","last_clean":"2024-04-02T04:50:27.449351+0000","last_became_active":"2024-04-02T04:49:26.198942+0000","last_became_peered":"2024-04-02T04:49:26.198942+0000","last_unstale":"2024-04-02T04:50:27.449351+0000","last_undegraded":"2024-04-02T04:50:27.449351+0000","last_fullsized":"2024-04-02T04:50:27.449351+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.592960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":346,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.416900+0000","last_change":"2024-04-02T04:49:16.603545+0000","last_active":"2024-04-02T04:50:27.416900+0000","last_peered":"2024-04-02T04:50:27.416900+0000","last_clean":"2024-04-02T04:50:27.416900+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:50:27.416900+0000","last_undegraded":"2024-04-02T04:50:27.416900+0000","last_fullsized":"2024-04-02T04:50:27.416900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:16.603444+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:49:16.603444+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.235195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":129,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.428928+0000","last_change":"2024-04-02T04:49:47.489637+0000","last_active":"2024-04-02T04:50:27.428928+0000","last_peered":"2024-04-02T04:50:27.428928+0000","last_clean":"2024-04-02T04:44:01.859319+0000","last_became_active":"2024-04-02T04:49:47.489637+0000","last_became_peered":"2024-04-02T04:49:47.489637+0000","last_unstale":"2024-04-02T04:50:27.428928+0000","last_undegraded":"2024-04-02T04:49:47.370487+0000","last_fullsized":"2024-04-02T04:49:47.370305+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":283,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.418173+0000","last_change":"2024-04-02T04:49:47.409343+0000","last_active":"2024-04-02T04:50:27.418173+0000","last_peered":"2024-04-02T04:50:27.418173+0000","last_clean":"2024-04-02T04:49:45.339425+0000","last_became_active":"2024-04-02T04:49:47.409343+0000","last_became_peered":"2024-04-02T04:49:47.409343+0000","last_unstale":"2024-04-02T04:50:27.418173+0000","last_undegraded":"2024-04-02T04:49:47.369170+0000","last_fullsized":"2024-04-02T04:49:47.369009+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:48:39.608255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.608255+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.387975+0000","last_change":"2024-04-02T04:49:47.502658+0000","last_active":"2024-04-02T04:50:27.387975+0000","last_peered":"2024-04-02T04:50:27.387975+0000","last_clean":"2024-04-02T04:49:45.343348+0000","last_became_active":"2024-04-02T04:49:47.502658+0000","last_became_peered":"2024-04-02T04:49:47.502658+0000","last_unstale":"2024-04-02T04:50:27.387975+0000","last_undegraded":"2024-04-02T04:50:27.387975+0000","last_fullsized":"2024-04-02T04:49:47.417371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.488397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.488397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.280603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"169'2","reported_seq":133,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.398842+0000","last_change":"2024-04-02T04:50:08.268505+0000","last_active":"2024-04-02T04:50:27.398842+0000","last_peered":"2024-04-02T04:50:27.398842+0000","last_clean":"2024-04-02T04:50:27.398842+0000","last_became_active":"2024-04-02T04:49:26.312680+0000","last_became_peered":"2024-04-02T04:49:26.312680+0000","last_unstale":"2024-04-02T04:50:27.398842+0000","last_undegraded":"2024-04-02T04:50:27.398842+0000","last_fullsized":"2024-04-02T04:50:27.398842+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'2","last_scrub_stamp":"2024-04-02T04:50:08.268401+0000","last_deep_scrub":"169'2","last_deep_scrub_stamp":"2024-04-02T04:50:08.268401+0000","last_clean_scrub_stamp":"2024-04-02T04:50:08.268401+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:09.766543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.376129+0000","last_change":"2024-04-02T04:49:26.295160+0000","last_active":"2024-04-02T04:50:27.376129+0000","last_peered":"2024-04-02T04:50:27.376129+0000","last_clean":"2024-04-02T04:50:27.376129+0000","last_became_active":"2024-04-02T04:49:26.292029+0000","last_became_peered":"2024-04-02T04:49:26.292029+0000","last_unstale":"2024-04-02T04:50:27.376129+0000","last_undegraded":"2024-04-02T04:50:27.376129+0000","last_fullsized":"2024-04-02T04:50:27.376129+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:41.201469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":129,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.423538+0000","last_change":"2024-04-02T04:49:47.409578+0000","last_active":"2024-04-02T04:50:27.423538+0000","last_peered":"2024-04-02T04:50:27.423538+0000","last_clean":"2024-04-02T04:44:01.867622+0000","last_became_active":"2024-04-02T04:49:47.409578+0000","last_became_peered":"2024-04-02T04:49:47.409578+0000","last_unstale":"2024-04-02T04:50:27.423538+0000","last_undegraded":"2024-04-02T04:49:47.352200+0000","last_fullsized":"2024-04-02T04:49:47.352060+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":271,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375903+0000","last_change":"2024-04-02T04:50:20.172828+0000","last_active":"2024-04-02T04:50:27.375903+0000","last_peered":"2024-04-02T04:50:27.375903+0000","last_clean":"2024-04-02T04:50:27.375903+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:50:27.375903+0000","last_undegraded":"2024-04-02T04:50:27.375903+0000","last_fullsized":"2024-04-02T04:50:27.375903+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:20.172630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.172630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.769221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":252,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.416751+0000","last_change":"2024-04-02T04:49:47.492319+0000","last_active":"2024-04-02T04:50:27.416751+0000","last_peered":"2024-04-02T04:50:27.416751+0000","last_clean":"2024-04-02T04:49:45.387118+0000","last_became_active":"2024-04-02T04:49:47.492319+0000","last_became_peered":"2024-04-02T04:49:47.492319+0000","last_unstale":"2024-04-02T04:50:27.416751+0000","last_undegraded":"2024-04-02T04:50:27.416751+0000","last_fullsized":"2024-04-02T04:49:47.417925+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.103147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.996248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":246,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.410880+0000","last_change":"2024-04-02T04:49:47.511784+0000","last_active":"2024-04-02T04:50:27.410880+0000","last_peered":"2024-04-02T04:50:27.410880+0000","last_clean":"2024-04-02T04:49:45.315951+0000","last_became_active":"2024-04-02T04:49:47.511784+0000","last_became_peered":"2024-04-02T04:49:47.511784+0000","last_unstale":"2024-04-02T04:50:27.410880+0000","last_undegraded":"2024-04-02T04:50:27.410880+0000","last_fullsized":"2024-04-02T04:49:47.381831+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.646709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.646709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.007265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"139'5","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.445681+0000","last_change":"2024-04-02T04:49:26.178484+0000","last_active":"2024-04-02T04:50:27.445681+0000","last_peered":"2024-04-02T04:50:27.445681+0000","last_clean":"2024-04-02T04:50:27.445681+0000","last_became_active":"2024-04-02T04:49:26.178040+0000","last_became_peered":"2024-04-02T04:49:26.178040+0000","last_unstale":"2024-04-02T04:50:27.445681+0000","last_undegraded":"2024-04-02T04:50:27.445681+0000","last_fullsized":"2024-04-02T04:50:27.445681+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.358939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1d","version":"138'1","reported_seq":114,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.429399+0000","last_change":"2024-04-02T04:49:26.508534+0000","last_active":"2024-04-02T04:50:27.429399+0000","last_peered":"2024-04-02T04:50:27.429399+0000","last_clean":"2024-04-02T04:50:27.429399+0000","last_became_active":"2024-04-02T04:49:26.508071+0000","last_became_peered":"2024-04-02T04:49:26.508071+0000","last_unstale":"2024-04-02T04:50:27.429399+0000","last_undegraded":"2024-04-02T04:50:27.429399+0000","last_fullsized":"2024-04-02T04:50:27.429399+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.781786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.370790+0000","last_change":"2024-04-02T04:49:26.259224+0000","last_active":"2024-04-02T04:50:27.370790+0000","last_peered":"2024-04-02T04:50:27.370790+0000","last_clean":"2024-04-02T04:50:27.370790+0000","last_became_active":"2024-04-02T04:49:26.258674+0000","last_became_peered":"2024-04-02T04:49:26.258674+0000","last_unstale":"2024-04-02T04:50:27.370790+0000","last_undegraded":"2024-04-02T04:50:27.370790+0000","last_fullsized":"2024-04-02T04:50:27.370790+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.883933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":267,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.388044+0000","last_change":"2024-04-02T04:50:17.746744+0000","last_active":"2024-04-02T04:50:27.388044+0000","last_peered":"2024-04-02T04:50:27.388044+0000","last_clean":"2024-04-02T04:50:27.388044+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:50:27.388044+0000","last_undegraded":"2024-04-02T04:50:27.388044+0000","last_fullsized":"2024-04-02T04:50:27.388044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:50:17.746578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.746578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:21.436393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":265,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.421693+0000","last_change":"2024-04-02T04:49:07.675897+0000","last_active":"2024-04-02T04:50:27.421693+0000","last_peered":"2024-04-02T04:50:27.421693+0000","last_clean":"2024-04-02T04:50:27.421693+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:50:27.421693+0000","last_undegraded":"2024-04-02T04:50:27.421693+0000","last_fullsized":"2024-04-02T04:50:27.421693+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.675606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.311827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":258,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.394474+0000","last_change":"2024-04-02T04:49:47.477916+0000","last_active":"2024-04-02T04:50:27.394474+0000","last_peered":"2024-04-02T04:50:27.394474+0000","last_clean":"2024-04-02T04:49:45.322247+0000","last_became_active":"2024-04-02T04:49:47.477916+0000","last_became_peered":"2024-04-02T04:49:47.477916+0000","last_unstale":"2024-04-02T04:50:27.394474+0000","last_undegraded":"2024-04-02T04:50:27.394474+0000","last_fullsized":"2024-04-02T04:49:47.406503+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.485165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.485165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.413961+0000","last_change":"2024-04-02T04:49:47.412587+0000","last_active":"2024-04-02T04:50:27.413961+0000","last_peered":"2024-04-02T04:50:27.413961+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.412587+0000","last_became_peered":"2024-04-02T04:49:47.412587+0000","last_unstale":"2024-04-02T04:50:27.413961+0000","last_undegraded":"2024-04-02T04:50:27.413961+0000","last_fullsized":"2024-04-02T04:49:47.396252+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.304153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.304153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.429684+0000","last_change":"2024-04-02T04:49:26.254782+0000","last_active":"2024-04-02T04:50:27.429684+0000","last_peered":"2024-04-02T04:50:27.429684+0000","last_clean":"2024-04-02T04:50:27.429684+0000","last_became_active":"2024-04-02T04:49:26.250745+0000","last_became_peered":"2024-04-02T04:49:26.250745+0000","last_unstale":"2024-04-02T04:50:27.429684+0000","last_undegraded":"2024-04-02T04:50:27.429684+0000","last_fullsized":"2024-04-02T04:50:27.429684+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405679+0000","last_change":"2024-04-02T04:49:26.264906+0000","last_active":"2024-04-02T04:50:27.405679+0000","last_peered":"2024-04-02T04:50:27.405679+0000","last_clean":"2024-04-02T04:50:27.405679+0000","last_became_active":"2024-04-02T04:49:26.264596+0000","last_became_peered":"2024-04-02T04:49:26.264596+0000","last_unstale":"2024-04-02T04:50:27.405679+0000","last_undegraded":"2024-04-02T04:50:27.405679+0000","last_fullsized":"2024-04-02T04:50:27.405679+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:34.380907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.407191+0000","last_change":"2024-04-02T04:49:26.229870+0000","last_active":"2024-04-02T04:50:27.407191+0000","last_peered":"2024-04-02T04:50:27.407191+0000","last_clean":"2024-04-02T04:50:27.407191+0000","last_became_active":"2024-04-02T04:49:26.229293+0000","last_became_peered":"2024-04-02T04:49:26.229293+0000","last_unstale":"2024-04-02T04:50:27.407191+0000","last_undegraded":"2024-04-02T04:50:27.407191+0000","last_fullsized":"2024-04-02T04:50:27.407191+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.073846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":117,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.410235+0000","last_change":"2024-04-02T04:49:47.452491+0000","last_active":"2024-04-02T04:50:27.410235+0000","last_peered":"2024-04-02T04:50:27.410235+0000","last_clean":"2024-04-02T04:44:01.831181+0000","last_became_active":"2024-04-02T04:49:47.452491+0000","last_became_peered":"2024-04-02T04:49:47.452491+0000","last_unstale":"2024-04-02T04:50:27.410235+0000","last_undegraded":"2024-04-02T04:49:47.355108+0000","last_fullsized":"2024-04-02T04:49:47.354938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":258,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.366955+0000","last_change":"2024-04-02T04:49:47.419557+0000","last_active":"2024-04-02T04:50:27.366955+0000","last_peered":"2024-04-02T04:50:27.366955+0000","last_clean":"2024-04-02T04:49:45.336531+0000","last_became_active":"2024-04-02T04:49:47.419557+0000","last_became_peered":"2024-04-02T04:49:47.419557+0000","last_unstale":"2024-04-02T04:50:27.366955+0000","last_undegraded":"2024-04-02T04:50:27.366955+0000","last_fullsized":"2024-04-02T04:49:47.341828+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.899356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.899356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"153'13","reported_seq":343,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.377836+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:50:27.377836+0000","last_peered":"2024-04-02T04:50:27.377836+0000","last_clean":"2024-04-02T04:49:45.328987+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:50:27.377836+0000","last_undegraded":"2024-04-02T04:49:47.427380+0000","last_fullsized":"2024-04-02T04:49:47.427156+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.416516+0000","last_change":"2024-04-02T04:49:47.491986+0000","last_active":"2024-04-02T04:50:27.416516+0000","last_peered":"2024-04-02T04:50:27.416516+0000","last_clean":"2024-04-02T04:49:45.387071+0000","last_became_active":"2024-04-02T04:49:47.491986+0000","last_became_peered":"2024-04-02T04:49:47.491986+0000","last_unstale":"2024-04-02T04:50:27.416516+0000","last_undegraded":"2024-04-02T04:50:27.416516+0000","last_fullsized":"2024-04-02T04:49:47.416002+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.986307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.986307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"139'7","reported_seq":98,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.410336+0000","last_change":"2024-04-02T04:49:47.507515+0000","last_active":"2024-04-02T04:50:27.410336+0000","last_peered":"2024-04-02T04:50:27.410336+0000","last_clean":"2024-04-02T04:49:27.511662+0000","last_became_active":"2024-04-02T04:49:47.507515+0000","last_became_peered":"2024-04-02T04:49:47.507515+0000","last_unstale":"2024-04-02T04:50:27.410336+0000","last_undegraded":"2024-04-02T04:49:47.419015+0000","last_fullsized":"2024-04-02T04:49:47.418904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.819045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":115,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.394654+0000","last_change":"2024-04-02T04:49:25.133267+0000","last_active":"2024-04-02T04:50:27.394654+0000","last_peered":"2024-04-02T04:50:27.394654+0000","last_clean":"2024-04-02T04:50:27.394654+0000","last_became_active":"2024-04-02T04:49:25.133058+0000","last_became_peered":"2024-04-02T04:49:25.133058+0000","last_unstale":"2024-04-02T04:50:27.394654+0000","last_undegraded":"2024-04-02T04:50:27.394654+0000","last_fullsized":"2024-04-02T04:50:27.394654+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.759386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":117,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.382131+0000","last_change":"2024-04-02T04:49:47.480176+0000","last_active":"2024-04-02T04:50:27.382131+0000","last_peered":"2024-04-02T04:50:27.382131+0000","last_clean":"2024-04-02T04:49:45.335187+0000","last_became_active":"2024-04-02T04:49:47.480176+0000","last_became_peered":"2024-04-02T04:49:47.480176+0000","last_unstale":"2024-04-02T04:50:27.382131+0000","last_undegraded":"2024-04-02T04:50:27.382131+0000","last_fullsized":"2024-04-02T04:49:47.401165+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":280,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.402907+0000","last_change":"2024-04-02T04:49:47.492714+0000","last_active":"2024-04-02T04:50:27.402907+0000","last_peered":"2024-04-02T04:50:27.402907+0000","last_clean":"2024-04-02T04:49:45.370028+0000","last_became_active":"2024-04-02T04:49:47.492714+0000","last_became_peered":"2024-04-02T04:49:47.492714+0000","last_unstale":"2024-04-02T04:50:27.402907+0000","last_undegraded":"2024-04-02T04:49:47.390470+0000","last_fullsized":"2024-04-02T04:49:47.390328+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:49:14.989379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.989379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":265,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.413350+0000","last_change":"2024-04-02T04:50:21.021320+0000","last_active":"2024-04-02T04:50:27.413350+0000","last_peered":"2024-04-02T04:50:27.413350+0000","last_clean":"2024-04-02T04:50:27.413350+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:50:27.413350+0000","last_undegraded":"2024-04-02T04:50:27.413350+0000","last_fullsized":"2024-04-02T04:50:27.413350+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.021194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.021194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:46.253328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"153'12","reported_seq":343,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.402726+0000","last_change":"2024-04-02T04:49:47.418645+0000","last_active":"2024-04-02T04:50:27.402726+0000","last_peered":"2024-04-02T04:50:27.402726+0000","last_clean":"2024-04-02T04:49:45.363313+0000","last_became_active":"2024-04-02T04:49:47.418645+0000","last_became_peered":"2024-04-02T04:49:47.418645+0000","last_unstale":"2024-04-02T04:50:27.402726+0000","last_undegraded":"2024-04-02T04:49:47.360244+0000","last_fullsized":"2024-04-02T04:49:47.360013+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:41.932321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.932321+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":244,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.442510+0000","last_change":"2024-04-02T04:49:11.708233+0000","last_active":"2024-04-02T04:50:27.442510+0000","last_peered":"2024-04-02T04:50:27.442510+0000","last_clean":"2024-04-02T04:50:27.442510+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:50:27.442510+0000","last_undegraded":"2024-04-02T04:50:27.442510+0000","last_fullsized":"2024-04-02T04:50:27.442510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.707910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.707910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.434142+0000","last_change":"2024-04-02T04:49:26.281485+0000","last_active":"2024-04-02T04:50:27.434142+0000","last_peered":"2024-04-02T04:50:27.434142+0000","last_clean":"2024-04-02T04:50:27.434142+0000","last_became_active":"2024-04-02T04:49:26.280984+0000","last_became_peered":"2024-04-02T04:49:26.280984+0000","last_unstale":"2024-04-02T04:50:27.434142+0000","last_undegraded":"2024-04-02T04:50:27.434142+0000","last_fullsized":"2024-04-02T04:50:27.434142+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:36.143155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1e","version":"138'1","reported_seq":117,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.432571+0000","last_change":"2024-04-02T04:49:47.508696+0000","last_active":"2024-04-02T04:50:27.432571+0000","last_peered":"2024-04-02T04:50:27.432571+0000","last_clean":"2024-04-02T04:49:45.356369+0000","last_became_active":"2024-04-02T04:49:47.508696+0000","last_became_peered":"2024-04-02T04:49:47.508696+0000","last_unstale":"2024-04-02T04:50:27.432571+0000","last_undegraded":"2024-04-02T04:49:47.380594+0000","last_fullsized":"2024-04-02T04:49:47.380468+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.007333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":121,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.438047+0000","last_change":"2024-04-02T04:50:25.822947+0000","last_active":"2024-04-02T04:50:27.438047+0000","last_peered":"2024-04-02T04:50:27.438047+0000","last_clean":"2024-04-02T04:50:27.438047+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:50:27.438047+0000","last_undegraded":"2024-04-02T04:50:27.438047+0000","last_fullsized":"2024-04-02T04:50:27.438047+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:25.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:40.783877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":126,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.377452+0000","last_change":"2024-04-02T04:49:47.479230+0000","last_active":"2024-04-02T04:50:27.377452+0000","last_peered":"2024-04-02T04:50:27.377452+0000","last_clean":"2024-04-02T04:44:01.845646+0000","last_became_active":"2024-04-02T04:49:47.479230+0000","last_became_peered":"2024-04-02T04:49:47.479230+0000","last_unstale":"2024-04-02T04:50:27.377452+0000","last_undegraded":"2024-04-02T04:49:47.401783+0000","last_fullsized":"2024-04-02T04:49:47.401678+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:49:23.491068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:23.491068+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:49.357553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":265,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.438138+0000","last_change":"2024-04-02T04:50:21.847768+0000","last_active":"2024-04-02T04:50:27.438138+0000","last_peered":"2024-04-02T04:50:27.438138+0000","last_clean":"2024-04-02T04:50:27.438138+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:50:27.438138+0000","last_undegraded":"2024-04-02T04:50:27.438138+0000","last_fullsized":"2024-04-02T04:50:27.438138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.847551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.847551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:50.902451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":256,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.388973+0000","last_change":"2024-04-02T04:50:10.316715+0000","last_active":"2024-04-02T04:50:27.388973+0000","last_peered":"2024-04-02T04:50:27.388973+0000","last_clean":"2024-04-02T04:50:27.388973+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:50:27.388973+0000","last_undegraded":"2024-04-02T04:50:27.388973+0000","last_fullsized":"2024-04-02T04:50:27.388973+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:10.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:10.316429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:26.018226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":253,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.403404+0000","last_change":"2024-04-02T04:50:11.029007+0000","last_active":"2024-04-02T04:50:27.403404+0000","last_peered":"2024-04-02T04:50:27.403404+0000","last_clean":"2024-04-02T04:50:27.403404+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:50:27.403404+0000","last_undegraded":"2024-04-02T04:50:27.403404+0000","last_fullsized":"2024-04-02T04:50:27.403404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.028926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.028926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.528508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"139'10","reported_seq":123,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405115+0000","last_change":"2024-04-02T04:49:26.267537+0000","last_active":"2024-04-02T04:50:27.405115+0000","last_peered":"2024-04-02T04:50:27.405115+0000","last_clean":"2024-04-02T04:50:27.405115+0000","last_became_active":"2024-04-02T04:49:26.260148+0000","last_became_peered":"2024-04-02T04:49:26.260148+0000","last_unstale":"2024-04-02T04:50:27.405115+0000","last_undegraded":"2024-04-02T04:50:27.405115+0000","last_fullsized":"2024-04-02T04:50:27.405115+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.19","version":"163'6","reported_seq":122,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.417248+0000","last_change":"2024-04-02T04:49:26.488893+0000","last_active":"2024-04-02T04:50:27.417248+0000","last_peered":"2024-04-02T04:50:27.417248+0000","last_clean":"2024-04-02T04:50:27.417248+0000","last_became_active":"2024-04-02T04:49:26.488111+0000","last_became_peered":"2024-04-02T04:49:26.488111+0000","last_unstale":"2024-04-02T04:50:27.417248+0000","last_undegraded":"2024-04-02T04:50:27.417248+0000","last_fullsized":"2024-04-02T04:50:27.417248+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.428977+0000","last_change":"2024-04-02T04:49:26.264150+0000","last_active":"2024-04-02T04:50:27.428977+0000","last_peered":"2024-04-02T04:50:27.428977+0000","last_clean":"2024-04-02T04:50:27.428977+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:50:27.428977+0000","last_undegraded":"2024-04-02T04:50:27.428977+0000","last_fullsized":"2024-04-02T04:50:27.428977+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":294,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.397176+0000","last_change":"2024-04-02T04:49:47.392866+0000","last_active":"2024-04-02T04:50:27.397176+0000","last_peered":"2024-04-02T04:50:27.397176+0000","last_clean":"2024-04-02T04:49:45.343613+0000","last_became_active":"2024-04-02T04:49:47.392866+0000","last_became_peered":"2024-04-02T04:49:47.392866+0000","last_unstale":"2024-04-02T04:50:27.397176+0000","last_undegraded":"2024-04-02T04:49:47.367503+0000","last_fullsized":"2024-04-02T04:49:47.367381+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":264,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.412462+0000","last_change":"2024-04-02T04:50:19.341033+0000","last_active":"2024-04-02T04:50:27.412462+0000","last_peered":"2024-04-02T04:50:27.412462+0000","last_clean":"2024-04-02T04:50:27.412462+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:50:27.412462+0000","last_undegraded":"2024-04-02T04:50:27.412462+0000","last_fullsized":"2024-04-02T04:50:27.412462+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:19.340953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.340953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:28.851404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":256,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405673+0000","last_change":"2024-04-02T04:49:12.685154+0000","last_active":"2024-04-02T04:50:27.405673+0000","last_peered":"2024-04-02T04:50:27.405673+0000","last_clean":"2024-04-02T04:50:27.405673+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:50:27.405673+0000","last_undegraded":"2024-04-02T04:50:27.405673+0000","last_fullsized":"2024-04-02T04:50:27.405673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.684916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.684916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.981975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":245,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.357830+0000","last_change":"2024-04-02T04:48:44.527799+0000","last_active":"2024-04-02T04:50:27.357830+0000","last_peered":"2024-04-02T04:50:27.357830+0000","last_clean":"2024-04-02T04:50:27.357830+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:50:27.357830+0000","last_undegraded":"2024-04-02T04:50:27.357830+0000","last_fullsized":"2024-04-02T04:50:27.357830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.527676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.527676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"139'5","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389589+0000","last_change":"2024-04-02T04:49:26.283455+0000","last_active":"2024-04-02T04:50:27.389589+0000","last_peered":"2024-04-02T04:50:27.389589+0000","last_clean":"2024-04-02T04:50:27.389589+0000","last_became_active":"2024-04-02T04:49:26.275418+0000","last_became_peered":"2024-04-02T04:49:26.275418+0000","last_unstale":"2024-04-02T04:50:27.389589+0000","last_undegraded":"2024-04-02T04:50:27.389589+0000","last_fullsized":"2024-04-02T04:50:27.389589+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:37.878462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.372534+0000","last_change":"2024-04-02T04:49:47.473514+0000","last_active":"2024-04-02T04:50:27.372534+0000","last_peered":"2024-04-02T04:50:27.372534+0000","last_clean":"2024-04-02T04:49:45.307113+0000","last_became_active":"2024-04-02T04:49:47.473514+0000","last_became_peered":"2024-04-02T04:49:47.473514+0000","last_unstale":"2024-04-02T04:50:27.372534+0000","last_undegraded":"2024-04-02T04:50:27.372534+0000","last_fullsized":"2024-04-02T04:49:47.359648+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:45.667225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":114,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.411933+0000","last_change":"2024-04-02T04:49:25.082702+0000","last_active":"2024-04-02T04:50:27.411933+0000","last_peered":"2024-04-02T04:50:27.411933+0000","last_clean":"2024-04-02T04:50:27.411933+0000","last_became_active":"2024-04-02T04:49:25.080486+0000","last_became_peered":"2024-04-02T04:49:25.080486+0000","last_unstale":"2024-04-02T04:50:27.411933+0000","last_undegraded":"2024-04-02T04:50:27.411933+0000","last_fullsized":"2024-04-02T04:50:27.411933+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":282,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.418923+0000","last_change":"2024-04-02T04:50:25.999986+0000","last_active":"2024-04-02T04:50:27.418923+0000","last_peered":"2024-04-02T04:50:27.418923+0000","last_clean":"2024-04-02T04:50:27.418923+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:50:27.418923+0000","last_undegraded":"2024-04-02T04:50:27.418923+0000","last_fullsized":"2024-04-02T04:50:27.418923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:50:25.999894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.999894+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.329856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":263,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.391711+0000","last_change":"2024-04-02T04:49:01.797690+0000","last_active":"2024-04-02T04:50:27.391711+0000","last_peered":"2024-04-02T04:50:27.391711+0000","last_clean":"2024-04-02T04:50:27.391711+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:50:27.391711+0000","last_undegraded":"2024-04-02T04:50:27.391711+0000","last_fullsized":"2024-04-02T04:50:27.391711+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.797580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:36.285639+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.797580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.381691+0000","last_change":"2024-04-02T04:49:18.346180+0000","last_active":"2024-04-02T04:50:27.381691+0000","last_peered":"2024-04-02T04:50:27.381691+0000","last_clean":"2024-04-02T04:50:27.381691+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:50:27.381691+0000","last_undegraded":"2024-04-02T04:50:27.381691+0000","last_fullsized":"2024-04-02T04:50:27.381691+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:18.346039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:18.346039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:34.789931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.399412+0000","last_change":"2024-04-02T04:49:47.512747+0000","last_active":"2024-04-02T04:50:27.399412+0000","last_peered":"2024-04-02T04:50:27.399412+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.512747+0000","last_became_peered":"2024-04-02T04:49:47.512747+0000","last_unstale":"2024-04-02T04:50:27.399412+0000","last_undegraded":"2024-04-02T04:50:27.399412+0000","last_fullsized":"2024-04-02T04:49:47.402486+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.372299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.372299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.007300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":118,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.360433+0000","last_change":"2024-04-02T04:49:47.444258+0000","last_active":"2024-04-02T04:50:27.360433+0000","last_peered":"2024-04-02T04:50:27.360433+0000","last_clean":"2024-04-02T04:49:45.328612+0000","last_became_active":"2024-04-02T04:49:47.444258+0000","last_became_peered":"2024-04-02T04:49:47.444258+0000","last_unstale":"2024-04-02T04:50:27.360433+0000","last_undegraded":"2024-04-02T04:50:27.360433+0000","last_fullsized":"2024-04-02T04:49:47.397778+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.376511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"163'2","reported_seq":91,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.408088+0000","last_change":"2024-04-02T04:49:47.399926+0000","last_active":"2024-04-02T04:50:27.408088+0000","last_peered":"2024-04-02T04:50:27.408088+0000","last_clean":"2024-04-02T04:49:26.261528+0000","last_became_active":"2024-04-02T04:49:47.399926+0000","last_became_peered":"2024-04-02T04:49:47.399926+0000","last_unstale":"2024-04-02T04:50:27.408088+0000","last_undegraded":"2024-04-02T04:49:47.374959+0000","last_fullsized":"2024-04-02T04:49:47.374821+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2807,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":82,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.426181+0000","last_change":"2024-04-02T04:49:47.517978+0000","last_active":"2024-04-02T04:50:27.426181+0000","last_peered":"2024-04-02T04:50:27.426181+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.517978+0000","last_became_peered":"2024-04-02T04:49:47.517978+0000","last_unstale":"2024-04-02T04:50:27.426181+0000","last_undegraded":"2024-04-02T04:50:27.426181+0000","last_fullsized":"2024-04-02T04:49:47.456750+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:43.736186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":302,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.390534+0000","last_change":"2024-04-02T04:49:47.430162+0000","last_active":"2024-04-02T04:50:27.390534+0000","last_peered":"2024-04-02T04:50:27.390534+0000","last_clean":"2024-04-02T04:49:45.364593+0000","last_became_active":"2024-04-02T04:49:47.430162+0000","last_became_peered":"2024-04-02T04:49:47.430162+0000","last_unstale":"2024-04-02T04:50:27.390534+0000","last_undegraded":"2024-04-02T04:49:47.355994+0000","last_fullsized":"2024-04-02T04:49:47.355865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:49:03.788837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.788837+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":273,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.449477+0000","last_change":"2024-04-02T04:50:08.818906+0000","last_active":"2024-04-02T04:50:27.449477+0000","last_peered":"2024-04-02T04:50:27.449477+0000","last_clean":"2024-04-02T04:50:27.449477+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:50:27.449477+0000","last_undegraded":"2024-04-02T04:50:27.449477+0000","last_fullsized":"2024-04-02T04:50:27.449477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:08.818819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:50:08.818819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:37.696499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":252,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.358563+0000","last_change":"2024-04-02T04:49:47.443764+0000","last_active":"2024-04-02T04:50:27.358563+0000","last_peered":"2024-04-02T04:50:27.358563+0000","last_clean":"2024-04-02T04:49:45.325466+0000","last_became_active":"2024-04-02T04:49:47.443764+0000","last_became_peered":"2024-04-02T04:49:47.443764+0000","last_unstale":"2024-04-02T04:50:27.358563+0000","last_undegraded":"2024-04-02T04:50:27.358563+0000","last_fullsized":"2024-04-02T04:49:47.396652+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.351209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.836879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":245,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.388097+0000","last_change":"2024-04-02T04:49:25.285134+0000","last_active":"2024-04-02T04:50:27.388097+0000","last_peered":"2024-04-02T04:50:27.388097+0000","last_clean":"2024-04-02T04:50:27.388097+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:50:27.388097+0000","last_undegraded":"2024-04-02T04:50:27.388097+0000","last_fullsized":"2024-04-02T04:50:27.388097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.284922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.284922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:46.491722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"138'2","reported_seq":91,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.380791+0000","last_change":"2024-04-02T04:49:47.436385+0000","last_active":"2024-04-02T04:50:27.380791+0000","last_peered":"2024-04-02T04:50:27.380791+0000","last_clean":"2024-04-02T04:49:26.658740+0000","last_became_active":"2024-04-02T04:49:47.436385+0000","last_became_peered":"2024-04-02T04:49:47.436385+0000","last_unstale":"2024-04-02T04:50:27.380791+0000","last_undegraded":"2024-04-02T04:50:27.380791+0000","last_fullsized":"2024-04-02T04:49:47.371603+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"138'2","reported_seq":116,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.368510+0000","last_change":"2024-04-02T04:49:26.246497+0000","last_active":"2024-04-02T04:50:27.368510+0000","last_peered":"2024-04-02T04:50:27.368510+0000","last_clean":"2024-04-02T04:50:27.368510+0000","last_became_active":"2024-04-02T04:49:26.241436+0000","last_became_peered":"2024-04-02T04:49:26.241436+0000","last_unstale":"2024-04-02T04:50:27.368510+0000","last_undegraded":"2024-04-02T04:50:27.368510+0000","last_fullsized":"2024-04-02T04:50:27.368510+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.392017+0000","last_change":"2024-04-02T04:49:47.393078+0000","last_active":"2024-04-02T04:50:27.392017+0000","last_peered":"2024-04-02T04:50:27.392017+0000","last_clean":"2024-04-02T04:49:45.306613+0000","last_became_active":"2024-04-02T04:49:47.393078+0000","last_became_peered":"2024-04-02T04:49:47.393078+0000","last_unstale":"2024-04-02T04:50:27.392017+0000","last_undegraded":"2024-04-02T04:50:27.392017+0000","last_fullsized":"2024-04-02T04:49:47.373401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:29.534274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"133'50","reported_seq":336,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375132+0000","last_change":"2024-04-02T04:49:02.986990+0000","last_active":"2024-04-02T04:50:27.375132+0000","last_peered":"2024-04-02T04:50:27.375132+0000","last_clean":"2024-04-02T04:50:27.375132+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:50:27.375132+0000","last_undegraded":"2024-04-02T04:50:27.375132+0000","last_fullsized":"2024-04-02T04:50:27.375132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.986701+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":266,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.365242+0000","last_change":"2024-04-02T04:49:47.388339+0000","last_active":"2024-04-02T04:50:27.365242+0000","last_peered":"2024-04-02T04:50:27.365242+0000","last_clean":"2024-04-02T04:49:45.330232+0000","last_became_active":"2024-04-02T04:49:47.388339+0000","last_became_peered":"2024-04-02T04:49:47.388339+0000","last_unstale":"2024-04-02T04:50:27.365242+0000","last_undegraded":"2024-04-02T04:50:27.365242+0000","last_fullsized":"2024-04-02T04:49:47.337757+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:51.870038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:51.870038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":261,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.416104+0000","last_change":"2024-04-02T04:49:47.491600+0000","last_active":"2024-04-02T04:50:27.416104+0000","last_peered":"2024-04-02T04:50:27.416104+0000","last_clean":"2024-04-02T04:49:45.386942+0000","last_became_active":"2024-04-02T04:49:47.491600+0000","last_became_peered":"2024-04-02T04:49:47.491600+0000","last_unstale":"2024-04-02T04:50:27.416104+0000","last_undegraded":"2024-04-02T04:50:27.416104+0000","last_fullsized":"2024-04-02T04:49:47.415538+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.087064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":90,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.386635+0000","last_change":"2024-04-02T04:49:47.434796+0000","last_active":"2024-04-02T04:50:27.386635+0000","last_peered":"2024-04-02T04:50:27.386635+0000","last_clean":"2024-04-02T04:44:01.344239+0000","last_became_active":"2024-04-02T04:49:47.434796+0000","last_became_peered":"2024-04-02T04:49:47.434796+0000","last_unstale":"2024-04-02T04:50:27.386635+0000","last_undegraded":"2024-04-02T04:49:47.346909+0000","last_fullsized":"2024-04-02T04:49:47.346781+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:48:42.394228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.394228+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.17","version":"139'5","reported_seq":118,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.397776+0000","last_change":"2024-04-02T04:49:26.275021+0000","last_active":"2024-04-02T04:50:27.397776+0000","last_peered":"2024-04-02T04:50:27.397776+0000","last_clean":"2024-04-02T04:50:27.397776+0000","last_became_active":"2024-04-02T04:49:26.272870+0000","last_became_peered":"2024-04-02T04:49:26.272870+0000","last_unstale":"2024-04-02T04:50:27.397776+0000","last_undegraded":"2024-04-02T04:50:27.397776+0000","last_fullsized":"2024-04-02T04:50:27.397776+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.165434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.15","version":"140'6","reported_seq":101,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.411767+0000","last_change":"2024-04-02T04:49:47.507735+0000","last_active":"2024-04-02T04:50:27.411767+0000","last_peered":"2024-04-02T04:50:27.411767+0000","last_clean":"2024-04-02T04:49:29.072617+0000","last_became_active":"2024-04-02T04:49:47.507735+0000","last_became_peered":"2024-04-02T04:49:47.507735+0000","last_unstale":"2024-04-02T04:50:27.411767+0000","last_undegraded":"2024-04-02T04:49:47.407132+0000","last_fullsized":"2024-04-02T04:49:47.406966+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.397376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165900,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.376338+0000","last_change":"2024-04-02T04:49:47.478289+0000","last_active":"2024-04-02T04:50:27.376338+0000","last_peered":"2024-04-02T04:50:27.376338+0000","last_clean":"2024-04-02T04:49:45.332087+0000","last_became_active":"2024-04-02T04:49:47.478289+0000","last_became_peered":"2024-04-02T04:49:47.478289+0000","last_unstale":"2024-04-02T04:50:27.376338+0000","last_undegraded":"2024-04-02T04:50:27.376338+0000","last_fullsized":"2024-04-02T04:49:47.337387+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.691788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":289,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.381580+0000","last_change":"2024-04-02T04:50:14.221770+0000","last_active":"2024-04-02T04:50:27.381580+0000","last_peered":"2024-04-02T04:50:27.381580+0000","last_clean":"2024-04-02T04:50:27.381580+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:50:27.381580+0000","last_undegraded":"2024-04-02T04:50:27.381580+0000","last_fullsized":"2024-04-02T04:50:27.381580+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:50:14.221684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.221684+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.077859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":259,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.358775+0000","last_change":"2024-04-02T04:47:58.472809+0000","last_active":"2024-04-02T04:50:27.358775+0000","last_peered":"2024-04-02T04:50:27.358775+0000","last_clean":"2024-04-02T04:50:27.358775+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:50:27.358775+0000","last_undegraded":"2024-04-02T04:50:27.358775+0000","last_fullsized":"2024-04-02T04:50:27.358775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:47:58.472703+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:47:58.472703+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":350,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.450601+0000","last_change":"2024-04-02T04:50:12.840880+0000","last_active":"2024-04-02T04:50:27.450601+0000","last_peered":"2024-04-02T04:50:27.450601+0000","last_clean":"2024-04-02T04:50:27.450601+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:50:27.450601+0000","last_undegraded":"2024-04-02T04:50:27.450601+0000","last_fullsized":"2024-04-02T04:50:27.450601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:12.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.840785+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:28.314441+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":252,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.412883+0000","last_change":"2024-04-02T04:50:15.299404+0000","last_active":"2024-04-02T04:50:27.412883+0000","last_peered":"2024-04-02T04:50:27.412883+0000","last_clean":"2024-04-02T04:50:27.412883+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:50:27.412883+0000","last_undegraded":"2024-04-02T04:50:27.412883+0000","last_fullsized":"2024-04-02T04:50:27.412883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.299316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.299316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:43.462635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"139'5","reported_seq":120,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.391766+0000","last_change":"2024-04-02T04:49:25.091835+0000","last_active":"2024-04-02T04:50:27.391766+0000","last_peered":"2024-04-02T04:50:27.391766+0000","last_clean":"2024-04-02T04:50:27.391766+0000","last_became_active":"2024-04-02T04:49:25.091539+0000","last_became_peered":"2024-04-02T04:49:25.091539+0000","last_unstale":"2024-04-02T04:50:27.391766+0000","last_undegraded":"2024-04-02T04:50:27.391766+0000","last_fullsized":"2024-04-02T04:50:27.391766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.404542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"138'6","reported_seq":119,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.443794+0000","last_change":"2024-04-02T04:49:26.193162+0000","last_active":"2024-04-02T04:50:27.443794+0000","last_peered":"2024-04-02T04:50:27.443794+0000","last_clean":"2024-04-02T04:50:27.443794+0000","last_became_active":"2024-04-02T04:49:26.188086+0000","last_became_peered":"2024-04-02T04:49:26.188086+0000","last_unstale":"2024-04-02T04:50:27.443794+0000","last_undegraded":"2024-04-02T04:50:27.443794+0000","last_fullsized":"2024-04-02T04:50:27.443794+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.408617+0000","last_change":"2024-04-02T04:49:26.261255+0000","last_active":"2024-04-02T04:50:27.408617+0000","last_peered":"2024-04-02T04:50:27.408617+0000","last_clean":"2024-04-02T04:50:27.408617+0000","last_became_active":"2024-04-02T04:49:26.256165+0000","last_became_peered":"2024-04-02T04:49:26.256165+0000","last_unstale":"2024-04-02T04:50:27.408617+0000","last_undegraded":"2024-04-02T04:50:27.408617+0000","last_fullsized":"2024-04-02T04:50:27.408617+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":281,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.413523+0000","last_change":"2024-04-02T04:49:47.432220+0000","last_active":"2024-04-02T04:50:27.413523+0000","last_peered":"2024-04-02T04:50:27.413523+0000","last_clean":"2024-04-02T04:49:45.374676+0000","last_became_active":"2024-04-02T04:49:47.432220+0000","last_became_peered":"2024-04-02T04:49:47.432220+0000","last_unstale":"2024-04-02T04:50:27.413523+0000","last_undegraded":"2024-04-02T04:49:47.384090+0000","last_fullsized":"2024-04-02T04:49:47.383905+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:49:05.019939+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.019939+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.408755+0000","last_change":"2024-04-02T04:49:47.404868+0000","last_active":"2024-04-02T04:50:27.408755+0000","last_peered":"2024-04-02T04:50:27.408755+0000","last_clean":"2024-04-02T04:49:45.386009+0000","last_became_active":"2024-04-02T04:49:47.404868+0000","last_became_peered":"2024-04-02T04:49:47.404868+0000","last_unstale":"2024-04-02T04:50:27.408755+0000","last_undegraded":"2024-04-02T04:50:27.408755+0000","last_fullsized":"2024-04-02T04:49:47.375401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:14.778569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.778569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.595705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":255,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.410793+0000","last_change":"2024-04-02T04:49:13.722710+0000","last_active":"2024-04-02T04:50:27.410793+0000","last_peered":"2024-04-02T04:50:27.410793+0000","last_clean":"2024-04-02T04:50:27.410793+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:50:27.410793+0000","last_undegraded":"2024-04-02T04:50:27.410793+0000","last_fullsized":"2024-04-02T04:50:27.410793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:13.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:42.434894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":253,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.563896+0000","last_change":"2024-04-02T04:50:27.563922+0000","last_active":"2024-04-02T04:50:27.563896+0000","last_peered":"2024-04-02T04:50:27.563896+0000","last_clean":"2024-04-02T04:50:27.563896+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:50:27.563896+0000","last_undegraded":"2024-04-02T04:50:27.563896+0000","last_fullsized":"2024-04-02T04:50:27.563896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.563824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.563824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:43.307638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.432144+0000","last_change":"2024-04-02T04:49:26.508086+0000","last_active":"2024-04-02T04:50:27.432144+0000","last_peered":"2024-04-02T04:50:27.432144+0000","last_clean":"2024-04-02T04:50:27.432144+0000","last_became_active":"2024-04-02T04:49:26.507403+0000","last_became_peered":"2024-04-02T04:49:26.507403+0000","last_unstale":"2024-04-02T04:50:27.432144+0000","last_undegraded":"2024-04-02T04:50:27.432144+0000","last_fullsized":"2024-04-02T04:50:27.432144+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:40.204165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.441984+0000","last_change":"2024-04-02T04:49:26.547790+0000","last_active":"2024-04-02T04:50:27.441984+0000","last_peered":"2024-04-02T04:50:27.441984+0000","last_clean":"2024-04-02T04:50:27.441984+0000","last_became_active":"2024-04-02T04:49:26.547495+0000","last_became_peered":"2024-04-02T04:49:26.547495+0000","last_unstale":"2024-04-02T04:50:27.441984+0000","last_undegraded":"2024-04-02T04:50:27.441984+0000","last_fullsized":"2024-04-02T04:50:27.441984+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:28.877721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":116,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.398246+0000","last_change":"2024-04-02T04:49:47.517198+0000","last_active":"2024-04-02T04:50:27.398246+0000","last_peered":"2024-04-02T04:50:27.398246+0000","last_clean":"2024-04-02T04:49:45.381250+0000","last_became_active":"2024-04-02T04:49:47.517198+0000","last_became_peered":"2024-04-02T04:49:47.517198+0000","last_unstale":"2024-04-02T04:50:27.398246+0000","last_undegraded":"2024-04-02T04:50:27.398246+0000","last_fullsized":"2024-04-02T04:49:47.430351+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:44.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":308,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.410520+0000","last_change":"2024-04-02T04:50:14.537273+0000","last_active":"2024-04-02T04:50:27.410520+0000","last_peered":"2024-04-02T04:50:27.410520+0000","last_clean":"2024-04-02T04:50:27.410520+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:50:27.410520+0000","last_undegraded":"2024-04-02T04:50:27.410520+0000","last_fullsized":"2024-04-02T04:50:27.410520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:50:14.537029+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.537029+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:35.811114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.382433+0000","last_change":"2024-04-02T04:49:47.468861+0000","last_active":"2024-04-02T04:50:27.382433+0000","last_peered":"2024-04-02T04:50:27.382433+0000","last_clean":"2024-04-02T04:49:45.338835+0000","last_became_active":"2024-04-02T04:49:47.468861+0000","last_became_peered":"2024-04-02T04:49:47.468861+0000","last_unstale":"2024-04-02T04:50:27.382433+0000","last_undegraded":"2024-04-02T04:50:27.382433+0000","last_fullsized":"2024-04-02T04:49:47.401513+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.329420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.329420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.125964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":249,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377746+0000","last_change":"2024-04-02T04:47:57.459178+0000","last_active":"2024-04-02T04:50:27.377746+0000","last_peered":"2024-04-02T04:50:27.377746+0000","last_clean":"2024-04-02T04:50:27.377746+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:50:27.377746+0000","last_undegraded":"2024-04-02T04:50:27.377746+0000","last_fullsized":"2024-04-02T04:50:27.377746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:57.459040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:47:57.459040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":246,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.394375+0000","last_change":"2024-04-02T04:49:05.447259+0000","last_active":"2024-04-02T04:50:27.394375+0000","last_peered":"2024-04-02T04:50:27.394375+0000","last_clean":"2024-04-02T04:50:27.394375+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:50:27.394375+0000","last_undegraded":"2024-04-02T04:50:27.394375+0000","last_fullsized":"2024-04-02T04:50:27.394375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.447020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.447020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"138'2","reported_seq":120,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.393465+0000","last_change":"2024-04-02T04:49:47.455012+0000","last_active":"2024-04-02T04:50:27.393465+0000","last_peered":"2024-04-02T04:50:27.393465+0000","last_clean":"2024-04-02T04:49:45.373759+0000","last_became_active":"2024-04-02T04:49:47.455012+0000","last_became_peered":"2024-04-02T04:49:47.455012+0000","last_unstale":"2024-04-02T04:50:27.393465+0000","last_undegraded":"2024-04-02T04:50:27.393465+0000","last_fullsized":"2024-04-02T04:49:47.384636+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.16","version":"138'1","reported_seq":116,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.424143+0000","last_change":"2024-04-02T04:49:25.081930+0000","last_active":"2024-04-02T04:50:27.424143+0000","last_peered":"2024-04-02T04:50:27.424143+0000","last_clean":"2024-04-02T04:50:27.424143+0000","last_became_active":"2024-04-02T04:49:25.079693+0000","last_became_peered":"2024-04-02T04:49:25.079693+0000","last_unstale":"2024-04-02T04:50:27.424143+0000","last_undegraded":"2024-04-02T04:50:27.424143+0000","last_fullsized":"2024-04-02T04:50:27.424143+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.404711+0000","last_change":"2024-04-02T04:49:26.310960+0000","last_active":"2024-04-02T04:50:27.404711+0000","last_peered":"2024-04-02T04:50:27.404711+0000","last_clean":"2024-04-02T04:50:27.404711+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:50:27.404711+0000","last_undegraded":"2024-04-02T04:50:27.404711+0000","last_fullsized":"2024-04-02T04:50:27.404711+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":294,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.402504+0000","last_change":"2024-04-02T04:50:13.554140+0000","last_active":"2024-04-02T04:50:27.402504+0000","last_peered":"2024-04-02T04:50:27.402504+0000","last_clean":"2024-04-02T04:50:27.402504+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:50:27.402504+0000","last_undegraded":"2024-04-02T04:50:27.402504+0000","last_fullsized":"2024-04-02T04:50:27.402504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:50:13.553801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.553801+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.234795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":87,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.390755+0000","last_change":"2024-04-02T04:49:47.435134+0000","last_active":"2024-04-02T04:50:27.390755+0000","last_peered":"2024-04-02T04:50:27.390755+0000","last_clean":"2024-04-02T04:43:54.216783+0000","last_became_active":"2024-04-02T04:49:47.435134+0000","last_became_peered":"2024-04-02T04:49:47.435134+0000","last_unstale":"2024-04-02T04:50:27.390755+0000","last_undegraded":"2024-04-02T04:49:47.360301+0000","last_fullsized":"2024-04-02T04:49:47.360179+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:48:48.436655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:48.436655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.505599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.412105+0000","last_change":"2024-04-02T04:49:01.626968+0000","last_active":"2024-04-02T04:50:27.412105+0000","last_peered":"2024-04-02T04:50:27.412105+0000","last_clean":"2024-04-02T04:50:27.412105+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:50:27.412105+0000","last_undegraded":"2024-04-02T04:50:27.412105+0000","last_fullsized":"2024-04-02T04:50:27.412105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.626750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.626750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.603064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.403197+0000","last_change":"2024-04-02T04:49:47.495495+0000","last_active":"2024-04-02T04:50:27.403197+0000","last_peered":"2024-04-02T04:50:27.403197+0000","last_clean":"2024-04-02T04:49:45.352563+0000","last_became_active":"2024-04-02T04:49:47.495495+0000","last_became_peered":"2024-04-02T04:49:47.495495+0000","last_unstale":"2024-04-02T04:50:27.403197+0000","last_undegraded":"2024-04-02T04:50:27.403197+0000","last_fullsized":"2024-04-02T04:49:47.378202+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.996221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"139'9","reported_seq":100,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.416834+0000","last_change":"2024-04-02T04:49:47.518710+0000","last_active":"2024-04-02T04:50:27.416834+0000","last_peered":"2024-04-02T04:50:27.416834+0000","last_clean":"2024-04-02T04:49:27.516310+0000","last_became_active":"2024-04-02T04:49:47.518710+0000","last_became_peered":"2024-04-02T04:49:47.518710+0000","last_unstale":"2024-04-02T04:50:27.416834+0000","last_undegraded":"2024-04-02T04:49:47.393557+0000","last_fullsized":"2024-04-02T04:49:47.393345+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:41.734977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375648+0000","last_change":"2024-04-02T04:49:26.278994+0000","last_active":"2024-04-02T04:50:27.375648+0000","last_peered":"2024-04-02T04:50:27.375648+0000","last_clean":"2024-04-02T04:50:27.375648+0000","last_became_active":"2024-04-02T04:49:26.278736+0000","last_became_peered":"2024-04-02T04:49:26.278736+0000","last_unstale":"2024-04-02T04:50:27.375648+0000","last_undegraded":"2024-04-02T04:50:27.375648+0000","last_fullsized":"2024-04-02T04:50:27.375648+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:32.909634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":114,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377195+0000","last_change":"2024-04-02T04:49:25.094911+0000","last_active":"2024-04-02T04:50:27.377195+0000","last_peered":"2024-04-02T04:50:27.377195+0000","last_clean":"2024-04-02T04:50:27.377195+0000","last_became_active":"2024-04-02T04:49:25.094609+0000","last_became_peered":"2024-04-02T04:49:25.094609+0000","last_unstale":"2024-04-02T04:50:27.377195+0000","last_undegraded":"2024-04-02T04:50:27.377195+0000","last_fullsized":"2024-04-02T04:50:27.377195+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":276,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.391913+0000","last_change":"2024-04-02T04:48:46.888055+0000","last_active":"2024-04-02T04:50:27.391913+0000","last_peered":"2024-04-02T04:50:27.391913+0000","last_clean":"2024-04-02T04:50:27.391913+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:50:27.391913+0000","last_undegraded":"2024-04-02T04:50:27.391913+0000","last_fullsized":"2024-04-02T04:50:27.391913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:46.887903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:46.887903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":79,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.433279+0000","last_change":"2024-04-02T04:49:47.470693+0000","last_active":"2024-04-02T04:50:27.433279+0000","last_peered":"2024-04-02T04:50:27.433279+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.470693+0000","last_became_peered":"2024-04-02T04:49:47.470693+0000","last_unstale":"2024-04-02T04:50:27.433279+0000","last_undegraded":"2024-04-02T04:50:27.433279+0000","last_fullsized":"2024-04-02T04:49:47.411420+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:22.452700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:22.452700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.545355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":256,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.405464+0000","last_change":"2024-04-02T04:50:09.464337+0000","last_active":"2024-04-02T04:50:27.405464+0000","last_peered":"2024-04-02T04:50:27.405464+0000","last_clean":"2024-04-02T04:50:27.405464+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:50:27.405464+0000","last_undegraded":"2024-04-02T04:50:27.405464+0000","last_fullsized":"2024-04-02T04:50:27.405464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.464222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.464222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.598443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":259,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377165+0000","last_change":"2024-04-02T04:50:10.507552+0000","last_active":"2024-04-02T04:50:27.377165+0000","last_peered":"2024-04-02T04:50:27.377165+0000","last_clean":"2024-04-02T04:50:27.377165+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:50:27.377165+0000","last_undegraded":"2024-04-02T04:50:27.377165+0000","last_fullsized":"2024-04-02T04:50:27.377165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:10.507422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:10.507422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.898282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":115,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.394536+0000","last_change":"2024-04-02T04:49:25.078375+0000","last_active":"2024-04-02T04:50:27.394536+0000","last_peered":"2024-04-02T04:50:27.394536+0000","last_clean":"2024-04-02T04:50:27.394536+0000","last_became_active":"2024-04-02T04:49:25.076275+0000","last_became_peered":"2024-04-02T04:49:25.076275+0000","last_unstale":"2024-04-02T04:50:27.394536+0000","last_undegraded":"2024-04-02T04:50:27.394536+0000","last_fullsized":"2024-04-02T04:50:27.394536+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"140'4","reported_seq":120,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.389269+0000","last_change":"2024-04-02T04:49:26.173459+0000","last_active":"2024-04-02T04:50:27.389269+0000","last_peered":"2024-04-02T04:50:27.389269+0000","last_clean":"2024-04-02T04:50:27.389269+0000","last_became_active":"2024-04-02T04:49:26.172824+0000","last_became_peered":"2024-04-02T04:49:26.172824+0000","last_unstale":"2024-04-02T04:50:27.389269+0000","last_undegraded":"2024-04-02T04:50:27.389269+0000","last_fullsized":"2024-04-02T04:50:27.389269+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.450636+0000","last_change":"2024-04-02T04:49:26.280288+0000","last_active":"2024-04-02T04:50:27.450636+0000","last_peered":"2024-04-02T04:50:27.450636+0000","last_clean":"2024-04-02T04:50:27.450636+0000","last_became_active":"2024-04-02T04:49:26.275375+0000","last_became_peered":"2024-04-02T04:49:26.275375+0000","last_unstale":"2024-04-02T04:50:27.450636+0000","last_undegraded":"2024-04-02T04:50:27.450636+0000","last_fullsized":"2024-04-02T04:50:27.450636+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:38.509604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":285,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.419056+0000","last_change":"2024-04-02T04:49:01.074976+0000","last_active":"2024-04-02T04:50:27.419056+0000","last_peered":"2024-04-02T04:50:27.419056+0000","last_clean":"2024-04-02T04:50:27.419056+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:50:27.419056+0000","last_undegraded":"2024-04-02T04:50:27.419056+0000","last_fullsized":"2024-04-02T04:50:27.419056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:01.074728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.074728+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":266,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.385263+0000","last_change":"2024-04-02T04:50:23.234439+0000","last_active":"2024-04-02T04:50:27.385263+0000","last_peered":"2024-04-02T04:50:27.385263+0000","last_clean":"2024-04-02T04:50:27.385263+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:50:27.385263+0000","last_undegraded":"2024-04-02T04:50:27.385263+0000","last_fullsized":"2024-04-02T04:50:27.385263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:23.234223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:23.234223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.525539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.390487+0000","last_change":"2024-04-02T04:49:05.650145+0000","last_active":"2024-04-02T04:50:27.390487+0000","last_peered":"2024-04-02T04:50:27.390487+0000","last_clean":"2024-04-02T04:50:27.390487+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:50:27.390487+0000","last_undegraded":"2024-04-02T04:50:27.390487+0000","last_fullsized":"2024-04-02T04:50:27.390487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.649928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:27.752553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":244,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.410243+0000","last_change":"2024-04-02T04:49:07.677962+0000","last_active":"2024-04-02T04:50:27.410243+0000","last_peered":"2024-04-02T04:50:27.410243+0000","last_clean":"2024-04-02T04:50:27.410243+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:50:27.410243+0000","last_undegraded":"2024-04-02T04:50:27.410243+0000","last_fullsized":"2024-04-02T04:50:27.410243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.677740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.970457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"139'7","reported_seq":120,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.375246+0000","last_change":"2024-04-02T04:49:26.246176+0000","last_active":"2024-04-02T04:50:27.375246+0000","last_peered":"2024-04-02T04:50:27.375246+0000","last_clean":"2024-04-02T04:50:27.375246+0000","last_became_active":"2024-04-02T04:49:26.241390+0000","last_became_peered":"2024-04-02T04:49:26.241390+0000","last_unstale":"2024-04-02T04:50:27.375246+0000","last_undegraded":"2024-04-02T04:50:27.375246+0000","last_fullsized":"2024-04-02T04:50:27.375246+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:35.581851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.403626+0000","last_change":"2024-04-02T04:49:26.279858+0000","last_active":"2024-04-02T04:50:27.403626+0000","last_peered":"2024-04-02T04:50:27.403626+0000","last_clean":"2024-04-02T04:50:27.403626+0000","last_became_active":"2024-04-02T04:49:26.274276+0000","last_became_peered":"2024-04-02T04:49:26.274276+0000","last_unstale":"2024-04-02T04:50:27.403626+0000","last_undegraded":"2024-04-02T04:50:27.403626+0000","last_fullsized":"2024-04-02T04:50:27.403626+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.273839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.442274+0000","last_change":"2024-04-02T04:49:26.537210+0000","last_active":"2024-04-02T04:50:27.442274+0000","last_peered":"2024-04-02T04:50:27.442274+0000","last_clean":"2024-04-02T04:50:27.442274+0000","last_became_active":"2024-04-02T04:49:26.536918+0000","last_became_peered":"2024-04-02T04:49:26.536918+0000","last_unstale":"2024-04-02T04:50:27.442274+0000","last_undegraded":"2024-04-02T04:50:27.442274+0000","last_fullsized":"2024-04-02T04:50:27.442274+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":281,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.385638+0000","last_change":"2024-04-02T04:48:41.484207+0000","last_active":"2024-04-02T04:50:27.385638+0000","last_peered":"2024-04-02T04:50:27.385638+0000","last_clean":"2024-04-02T04:50:27.385638+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:50:27.385638+0000","last_undegraded":"2024-04-02T04:50:27.385638+0000","last_fullsized":"2024-04-02T04:50:27.385638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:48:41.484047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.484047+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":261,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.419234+0000","last_change":"2024-04-02T04:49:47.432079+0000","last_active":"2024-04-02T04:50:27.419234+0000","last_peered":"2024-04-02T04:50:27.419234+0000","last_clean":"2024-04-02T04:49:45.381663+0000","last_became_active":"2024-04-02T04:49:47.432079+0000","last_became_peered":"2024-04-02T04:49:47.432079+0000","last_unstale":"2024-04-02T04:50:27.419234+0000","last_undegraded":"2024-04-02T04:50:27.419234+0000","last_fullsized":"2024-04-02T04:49:47.406616+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:30.996193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.358005+0000","last_change":"2024-04-02T04:50:20.218009+0000","last_active":"2024-04-02T04:50:27.358005+0000","last_peered":"2024-04-02T04:50:27.358005+0000","last_clean":"2024-04-02T04:50:27.358005+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:50:27.358005+0000","last_undegraded":"2024-04-02T04:50:27.358005+0000","last_fullsized":"2024-04-02T04:50:27.358005+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:20.217928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.217928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.687003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":248,"reported_epoch":188,"state":"active+undersized","last_fresh":"2024-04-02T04:50:27.381747+0000","last_change":"2024-04-02T04:49:47.454112+0000","last_active":"2024-04-02T04:50:27.381747+0000","last_peered":"2024-04-02T04:50:27.381747+0000","last_clean":"2024-04-02T04:49:45.335087+0000","last_became_active":"2024-04-02T04:49:47.454112+0000","last_became_peered":"2024-04-02T04:49:47.454112+0000","last_unstale":"2024-04-02T04:50:27.381747+0000","last_undegraded":"2024-04-02T04:50:27.381747+0000","last_fullsized":"2024-04-02T04:49:47.400802+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.389577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.389577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:33.115272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":113,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-02T04:50:27.377045+0000","last_change":"2024-04-02T04:49:26.276875+0000","last_active":"2024-04-02T04:50:27.377045+0000","last_peered":"2024-04-02T04:50:27.377045+0000","last_clean":"2024-04-02T04:50:27.377045+0000","last_became_active":"2024-04-02T04:49:26.269993+0000","last_became_peered":"2024-04-02T04:49:26.269993+0000","last_unstale":"2024-04-02T04:50:27.377045+0000","last_undegraded":"2024-04-02T04:50:27.377045+0000","last_fullsized":"2024-04-02T04:50:27.377045+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"140'1","reported_seq":123,"reported_epoch":188,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:50:27.361860+0000","last_change":"2024-04-02T04:49:47.503022+0000","last_active":"2024-04-02T04:50:27.361860+0000","last_peered":"2024-04-02T04:50:27.361860+0000","last_clean":"2024-04-02T04:49:45.313557+0000","last_became_active":"2024-04-02T04:49:47.503022+0000","last_became_peered":"2024-04-02T04:49:47.503022+0000","last_unstale":"2024-04-02T04:50:27.361860+0000","last_undegraded":"2024-04-02T04:49:47.390158+0000","last_fullsized":"2024-04-02T04:49:47.390046+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:50:31.452214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":175,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":174,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":165,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":85,"acting":85,"num_store_stats":7},{"poolid":112,"num_pg":32,"stat_sum":{"num_bytes":2176,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":11,"num_read_kb":4,"num_write":28,"num_write_kb":1049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":6016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":85,"acting":85,"num_store_stats":7},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":843,"num_objects":108,"num_object_clones":2,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":1,"num_read":6,"num_read_kb":1,"num_write":110,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1159168,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":87,"acting":87,"num_store_stats":7},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":87,"acting":87,"num_store_stats":8},{"poolid":173,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":226499262,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":1010,"num_read_kb":24584,"num_write":1047,"num_write_kb":226712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11200000,"num_omap_keys":4000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679804928,"data_stored":679494986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":87,"acting":87,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":86,"acting":86,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743504,"num_pgs":239,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111160,"kb_used_data":75820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93633992,"statfs":{"total":95995035648,"available":95881207808,"internally_reserved":0,"allocated":77639680,"data_stored":76942360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103268,"num_pgs":217,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155000,"kb_used_data":101592,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93590152,"statfs":{"total":95995035648,"available":95836315648,"internally_reserved":0,"allocated":104030208,"data_stored":102830933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574999,"num_pgs":193,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":252768,"kb_used_data":199376,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93492384,"statfs":{"total":95995035648,"available":95736201216,"internally_reserved":0,"allocated":204161024,"data_stored":202899944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":198,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130412,"kb_used_data":77004,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93614740,"statfs":{"total":95995035648,"available":95861493760,"internally_reserved":0,"allocated":78852096,"data_stored":77661485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378765,"num_pgs":214,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109424,"kb_used_data":51856,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93635728,"statfs":{"total":95995035648,"available":95882985472,"internally_reserved":0,"allocated":53100544,"data_stored":51904358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051715,"num_pgs":198,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183132,"kb_used_data":125580,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93562020,"statfs":{"total":95995035648,"available":95807508480,"internally_reserved":0,"allocated":128593920,"data_stored":127401152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822777,"num_pgs":223,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62796,"kb_used_data":27444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93682356,"statfs":{"total":95995035648,"available":95930732544,"internally_reserved":0,"allocated":28102656,"data_stored":26758041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463021,"num_pgs":190,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62656,"kb_used_data":27312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93682496,"statfs":{"total":95995035648,"available":95930875904,"internally_reserved":0,"allocated":27967488,"data_stored":26756691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75538432,"data_stored":75500401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201371648,"data_stored":201326726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50356224,"data_stored":50331693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":125874176,"data_stored":125829188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25214976,"data_stored":25169713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25202688,"data_stored":25169693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100716544,"data_stored":100669988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":75530240,"data_stored":75497584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":112,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:50:34.416 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-04-02T04:50:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.417 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:34.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:34.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:32.689687+0000 mgr.y (mgr.14538) 269 : audit 0 from='client.25796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.047586+0000 mgr.y (mgr.14538) 270 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: cluster 2024-04-02T04:50:33.295547+0000 osd.1 (osd.1) 167 : cluster 0 3.19 scrub starts 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: cluster 2024-04-02T04:50:33.298470+0000 osd.1 (osd.1) 168 : cluster 0 3.19 scrub ok 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.391945+0000 mon.a (mon.0) 1478 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]': finished 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.392034+0000 mon.a (mon.0) 1479 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: cluster 2024-04-02T04:50:33.398668+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e194: 8 total, 7 up, 8 in 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.405419+0000 mon.c (mon.2) 238 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]: dispatch 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.406050+0000 mon.c (mon.2) 239 : audit 1 from='client.? 172.21.15.69:0/1435793473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.420144+0000 mon.c (mon.2) 240 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:50:34.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.420979+0000 mon.a (mon.0) 1481 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]: dispatch 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.421456+0000 mon.a (mon.0) 1482 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.422048+0000 mon.a (mon.0) 1483 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:33.682830+0000 mon.c (mon.2) 241 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.318663+0000 mon.a (mon.0) 1484 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.319696+0000 mon.a (mon.0) 1485 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:34.321386+0000 mon.a (mon.0) 1486 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]': finished 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:34.321466+0000 mon.a (mon.0) 1487 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:34.321521+0000 mon.a (mon.0) 1488 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]': finished 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.328002+0000 mon.a (mon.0) 1489 : cluster 0 osdmap e195: 8 total, 7 up, 8 in 2024-04-02T04:50:34.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:34 smithi098 bash[23790]: audit 2024-04-02T04:50:34.342975+0000 mon.a (mon.0) 1490 : audit 1 from='client.? 172.21.15.69:0/1676760754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-72145-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:32.689687+0000 mgr.y (mgr.14538) 269 : audit 0 from='client.25796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.047586+0000 mgr.y (mgr.14538) 270 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: cluster 2024-04-02T04:50:33.295547+0000 osd.1 (osd.1) 167 : cluster 0 3.19 scrub starts 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: cluster 2024-04-02T04:50:33.298470+0000 osd.1 (osd.1) 168 : cluster 0 3.19 scrub ok 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.391945+0000 mon.a (mon.0) 1478 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]': finished 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.392034+0000 mon.a (mon.0) 1479 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: cluster 2024-04-02T04:50:33.398668+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e194: 8 total, 7 up, 8 in 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.405419+0000 mon.c (mon.2) 238 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.406050+0000 mon.c (mon.2) 239 : audit 1 from='client.? 172.21.15.69:0/1435793473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.420144+0000 mon.c (mon.2) 240 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.420979+0000 mon.a (mon.0) 1481 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.421456+0000 mon.a (mon.0) 1482 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.422048+0000 mon.a (mon.0) 1483 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:50:34.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:33.682830+0000 mon.c (mon.2) 241 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.318663+0000 mon.a (mon.0) 1484 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.319696+0000 mon.a (mon.0) 1485 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:34.321386+0000 mon.a (mon.0) 1486 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]': finished 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:32.689687+0000 mgr.y (mgr.14538) 269 : audit 0 from='client.25796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.047586+0000 mgr.y (mgr.14538) 270 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: cluster 2024-04-02T04:50:33.295547+0000 osd.1 (osd.1) 167 : cluster 0 3.19 scrub starts 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: cluster 2024-04-02T04:50:33.298470+0000 osd.1 (osd.1) 168 : cluster 0 3.19 scrub ok 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.391945+0000 mon.a (mon.0) 1478 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-13"}]': finished 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.392034+0000 mon.a (mon.0) 1479 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: cluster 2024-04-02T04:50:33.398668+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e194: 8 total, 7 up, 8 in 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.405419+0000 mon.c (mon.2) 238 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]: dispatch 2024-04-02T04:50:34.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.406050+0000 mon.c (mon.2) 239 : audit 1 from='client.? 172.21.15.69:0/1435793473' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.420144+0000 mon.c (mon.2) 240 : audit 1 from='client.? 172.21.15.69:0/50059295' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.420979+0000 mon.a (mon.0) 1481 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]: dispatch 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.421456+0000 mon.a (mon.0) 1482 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.422048+0000 mon.a (mon.0) 1483 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:33.682830+0000 mon.c (mon.2) 241 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.318663+0000 mon.a (mon.0) 1484 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.319696+0000 mon.a (mon.0) 1485 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:34.321386+0000 mon.a (mon.0) 1486 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-13", "mode": "writeback"}]': finished 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:34.321466+0000 mon.a (mon.0) 1487 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:34.321521+0000 mon.a (mon.0) 1488 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]': finished 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.328002+0000 mon.a (mon.0) 1489 : cluster 0 osdmap e195: 8 total, 7 up, 8 in 2024-04-02T04:50:34.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[21528]: audit 2024-04-02T04:50:34.342975+0000 mon.a (mon.0) 1490 : audit 1 from='client.? 172.21.15.69:0/1676760754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-72145-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:34.321466+0000 mon.a (mon.0) 1487 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-72153-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:34.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:34.321521+0000 mon.a (mon.0) 1488 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-72304-1","app":"app1","key":"key1"}]': finished 2024-04-02T04:50:34.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.328002+0000 mon.a (mon.0) 1489 : cluster 0 osdmap e195: 8 total, 7 up, 8 in 2024-04-02T04:50:34.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:34 smithi069 bash[28904]: audit 2024-04-02T04:50:34.342975+0000 mon.a (mon.0) 1490 : audit 1 from='client.? 172.21.15.69:0/1676760754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-72145-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:34.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:34.663 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:34.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:34.791 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:34.799 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:34.799 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:34.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:34.912 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:34.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:35.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:35.037 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:35.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:35.164 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:35.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:35.339 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: seed 72304 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (488 ms) 2024-04-02T04:50:35.340 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (19 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (26 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (14 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (7 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (8 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-02T04:50:35.341 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (23 ms) 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (1 ms) 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (11 ms) 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (13 ms) 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2199 ms) 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (465 ms) 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-02T04:50:35.342 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting, doing copies 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60808 ms) 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (8 ms) 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (5 ms) 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4195 ms) 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-02T04:50:35.343 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-02T04:50:35.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:35.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:35.543 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:35.645 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:35.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:35.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:35.797 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:35.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:35.924 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:35.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:36.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:33.908966+0000 mgr.y (mgr.14538) 271 : cluster 0 pgmap v268: 524 pgs: 106 active+undersized, 274 active+clean, 32 creating+peering, 64 unknown, 48 active+undersized+degraded; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 100 KiB/s wr, 99 op/s; 286/2394 objects degraded (11.947%) 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.276179+0000 osd.1 (osd.1) 169 : cluster 0 2.d scrub starts 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.277543+0000 osd.1 (osd.1) 170 : cluster 0 2.d scrub ok 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.409084+0000 mon.a (mon.0) 1491 : cluster 3 Health check update: Degraded data redundancy: 286/2394 objects degraded (11.947%), 48 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.461019+0000 osd.3 (osd.3) 187 : cluster 0 6.c scrub starts 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.462060+0000 osd.3 (osd.3) 188 : cluster 0 6.c scrub ok 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.477092+0000 osd.5 (osd.5) 143 : cluster 0 3.1c scrub starts 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:34.479084+0000 osd.5 (osd.5) 144 : cluster 0 3.1c scrub ok 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:34.683941+0000 mon.c (mon.2) 242 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.250589+0000 mon.c (mon.2) 243 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.251520+0000 mon.c (mon.2) 244 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-04-02T04:50:36.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.252343+0000 mon.c (mon.2) 245 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.253058+0000 mon.c (mon.2) 246 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.253688+0000 mon.c (mon.2) 247 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.254247+0000 mon.c (mon.2) 248 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.254865+0000 mon.c (mon.2) 249 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.255465+0000 mon.c (mon.2) 250 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.256107+0000 mon.c (mon.2) 251 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.256675+0000 mon.c (mon.2) 252 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.323586+0000 mon.a (mon.0) 1492 : audit 1 from='client.? 172.21.15.69:0/1676760754' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi069-72145-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.336070+0000 mon.a (mon.0) 1493 : cluster 0 osdmap e196: 8 total, 7 up, 8 in 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.387308+0000 mon.a (mon.0) 1494 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:36.041 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:35 smithi098 bash[23790]: audit 2024-04-02T04:50:35.684871+0000 mon.c (mon.2) 253 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:36.053 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:36.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:36.183 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:36.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:33.908966+0000 mgr.y (mgr.14538) 271 : cluster 0 pgmap v268: 524 pgs: 106 active+undersized, 274 active+clean, 32 creating+peering, 64 unknown, 48 active+undersized+degraded; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 100 KiB/s wr, 99 op/s; 286/2394 objects degraded (11.947%) 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.276179+0000 osd.1 (osd.1) 169 : cluster 0 2.d scrub starts 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.277543+0000 osd.1 (osd.1) 170 : cluster 0 2.d scrub ok 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.409084+0000 mon.a (mon.0) 1491 : cluster 3 Health check update: Degraded data redundancy: 286/2394 objects degraded (11.947%), 48 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.461019+0000 osd.3 (osd.3) 187 : cluster 0 6.c scrub starts 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.462060+0000 osd.3 (osd.3) 188 : cluster 0 6.c scrub ok 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.477092+0000 osd.5 (osd.5) 143 : cluster 0 3.1c scrub starts 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:34.479084+0000 osd.5 (osd.5) 144 : cluster 0 3.1c scrub ok 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:34.683941+0000 mon.c (mon.2) 242 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.250589+0000 mon.c (mon.2) 243 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.251520+0000 mon.c (mon.2) 244 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.252343+0000 mon.c (mon.2) 245 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-04-02T04:50:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.253058+0000 mon.c (mon.2) 246 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.253688+0000 mon.c (mon.2) 247 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.254247+0000 mon.c (mon.2) 248 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.254865+0000 mon.c (mon.2) 249 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.255465+0000 mon.c (mon.2) 250 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.256107+0000 mon.c (mon.2) 251 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.256675+0000 mon.c (mon.2) 252 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:33.908966+0000 mgr.y (mgr.14538) 271 : cluster 0 pgmap v268: 524 pgs: 106 active+undersized, 274 active+clean, 32 creating+peering, 64 unknown, 48 active+undersized+degraded; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 100 KiB/s wr, 99 op/s; 286/2394 objects degraded (11.947%) 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.276179+0000 osd.1 (osd.1) 169 : cluster 0 2.d scrub starts 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.277543+0000 osd.1 (osd.1) 170 : cluster 0 2.d scrub ok 2024-04-02T04:50:36.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.409084+0000 mon.a (mon.0) 1491 : cluster 3 Health check update: Degraded data redundancy: 286/2394 objects degraded (11.947%), 48 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.461019+0000 osd.3 (osd.3) 187 : cluster 0 6.c scrub starts 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.462060+0000 osd.3 (osd.3) 188 : cluster 0 6.c scrub ok 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.477092+0000 osd.5 (osd.5) 143 : cluster 0 3.1c scrub starts 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:34.479084+0000 osd.5 (osd.5) 144 : cluster 0 3.1c scrub ok 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:34.683941+0000 mon.c (mon.2) 242 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.250589+0000 mon.c (mon.2) 243 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.251520+0000 mon.c (mon.2) 244 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.252343+0000 mon.c (mon.2) 245 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.253058+0000 mon.c (mon.2) 246 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.253688+0000 mon.c (mon.2) 247 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.254247+0000 mon.c (mon.2) 248 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.254865+0000 mon.c (mon.2) 249 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.255465+0000 mon.c (mon.2) 250 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.256107+0000 mon.c (mon.2) 251 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-04-02T04:50:36.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.256675+0000 mon.c (mon.2) 252 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.323586+0000 mon.a (mon.0) 1492 : audit 1 from='client.? 172.21.15.69:0/1676760754' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi069-72145-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.336070+0000 mon.a (mon.0) 1493 : cluster 0 osdmap e196: 8 total, 7 up, 8 in 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.387308+0000 mon.a (mon.0) 1494 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[21528]: audit 2024-04-02T04:50:35.684871+0000 mon.c (mon.2) 253 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.323586+0000 mon.a (mon.0) 1492 : audit 1 from='client.? 172.21.15.69:0/1676760754' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi069-72145-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.336070+0000 mon.a (mon.0) 1493 : cluster 0 osdmap e196: 8 total, 7 up, 8 in 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.387308+0000 mon.a (mon.0) 1494 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:36.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:35 smithi069 bash[28904]: audit 2024-04-02T04:50:35.684871+0000 mon.c (mon.2) 253 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:36.286 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:36.311 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:36.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:36.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:36.561 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:36.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:36.685 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:36.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:36.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:36.942 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:36.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:37.045 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:37.072 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:37.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.048344+0000 osd.0 (osd.0) 161 : cluster 0 4.b scrub starts 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.049650+0000 osd.0 (osd.0) 162 : cluster 0 4.b scrub ok 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.250815+0000 mgr.y (mgr.14538) 272 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.251700+0000 mgr.y (mgr.14538) 273 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.252512+0000 mgr.y (mgr.14538) 274 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.253259+0000 mgr.y (mgr.14538) 275 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-04-02T04:50:37.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.253857+0000 mgr.y (mgr.14538) 276 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.254413+0000 mgr.y (mgr.14538) 277 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.255031+0000 mgr.y (mgr.14538) 278 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.255649+0000 mgr.y (mgr.14538) 279 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.256273+0000 mgr.y (mgr.14538) 280 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.256801+0000 mgr.y (mgr.14538) 281 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.450307+0000 osd.5 (osd.5) 145 : cluster 0 165.1 scrub starts 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.452397+0000 osd.5 (osd.5) 146 : cluster 0 165.1 scrub ok 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.582919+0000 osd.4 (osd.4) 163 : cluster 0 165.7 scrub starts 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.583409+0000 osd.3 (osd.3) 189 : cluster 0 165.9 scrub starts 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.587356+0000 osd.3 (osd.3) 190 : cluster 0 165.9 scrub ok 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.589281+0000 osd.4 (osd.4) 164 : cluster 0 165.7 scrub ok 2024-04-02T04:50:37.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.822059+0000 mon.a (mon.0) 1495 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:35.823418+0000 mon.a (mon.0) 1496 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72304-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:35.909989+0000 mgr.y (mgr.14538) 282 : cluster 0 pgmap v271: 460 pgs: 104 active+undersized, 251 active+clean, 64 unknown, 41 active+undersized+degraded; 600 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 277/2289 objects degraded (12.101%) 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.075564+0000 osd.2 (osd.2) 107 : cluster 0 165.5 scrub starts 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.081678+0000 osd.2 (osd.2) 108 : cluster 0 165.5 scrub ok 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.224056+0000 osd.1 (osd.1) 171 : cluster 0 165.3 deep-scrub starts 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.233117+0000 osd.1 (osd.1) 172 : cluster 0 165.3 deep-scrub ok 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:36.326441+0000 mon.a (mon.0) 1497 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72304-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.331376+0000 mon.a (mon.0) 1498 : cluster 0 osdmap e197: 8 total, 7 up, 8 in 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:36.332342+0000 mon.a (mon.0) 1499 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72304-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:36.334529+0000 mon.c (mon.2) 254 : audit 1 from='client.? 172.21.15.69:0/1315613526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:36.335057+0000 mon.a (mon.0) 1500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:37.188 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[28904]: audit 2024-04-02T04:50:36.685961+0000 mon.c (mon.2) 255 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.048344+0000 osd.0 (osd.0) 161 : cluster 0 4.b scrub starts 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.049650+0000 osd.0 (osd.0) 162 : cluster 0 4.b scrub ok 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.250815+0000 mgr.y (mgr.14538) 272 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.251700+0000 mgr.y (mgr.14538) 273 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.252512+0000 mgr.y (mgr.14538) 274 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.253259+0000 mgr.y (mgr.14538) 275 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.253857+0000 mgr.y (mgr.14538) 276 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.254413+0000 mgr.y (mgr.14538) 277 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.255031+0000 mgr.y (mgr.14538) 278 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.255649+0000 mgr.y (mgr.14538) 279 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.256273+0000 mgr.y (mgr.14538) 280 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.256801+0000 mgr.y (mgr.14538) 281 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-04-02T04:50:37.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.450307+0000 osd.5 (osd.5) 145 : cluster 0 165.1 scrub starts 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.452397+0000 osd.5 (osd.5) 146 : cluster 0 165.1 scrub ok 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.582919+0000 osd.4 (osd.4) 163 : cluster 0 165.7 scrub starts 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.583409+0000 osd.3 (osd.3) 189 : cluster 0 165.9 scrub starts 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.587356+0000 osd.3 (osd.3) 190 : cluster 0 165.9 scrub ok 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.589281+0000 osd.4 (osd.4) 164 : cluster 0 165.7 scrub ok 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.822059+0000 mon.a (mon.0) 1495 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:35.823418+0000 mon.a (mon.0) 1496 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72304-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:35.909989+0000 mgr.y (mgr.14538) 282 : cluster 0 pgmap v271: 460 pgs: 104 active+undersized, 251 active+clean, 64 unknown, 41 active+undersized+degraded; 600 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 277/2289 objects degraded (12.101%) 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.075564+0000 osd.2 (osd.2) 107 : cluster 0 165.5 scrub starts 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.081678+0000 osd.2 (osd.2) 108 : cluster 0 165.5 scrub ok 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.224056+0000 osd.1 (osd.1) 171 : cluster 0 165.3 deep-scrub starts 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.233117+0000 osd.1 (osd.1) 172 : cluster 0 165.3 deep-scrub ok 2024-04-02T04:50:37.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:36.326441+0000 mon.a (mon.0) 1497 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72304-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:37.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.331376+0000 mon.a (mon.0) 1498 : cluster 0 osdmap e197: 8 total, 7 up, 8 in 2024-04-02T04:50:37.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:36.332342+0000 mon.a (mon.0) 1499 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72304-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:37.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:36.334529+0000 mon.c (mon.2) 254 : audit 1 from='client.? 172.21.15.69:0/1315613526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:37.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:36.335057+0000 mon.a (mon.0) 1500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:37.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:36 smithi069 bash[21528]: audit 2024-04-02T04:50:36.685961+0000 mon.c (mon.2) 255 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.048344+0000 osd.0 (osd.0) 161 : cluster 0 4.b scrub starts 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.049650+0000 osd.0 (osd.0) 162 : cluster 0 4.b scrub ok 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.250815+0000 mgr.y (mgr.14538) 272 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.251700+0000 mgr.y (mgr.14538) 273 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.252512+0000 mgr.y (mgr.14538) 274 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.253259+0000 mgr.y (mgr.14538) 275 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-04-02T04:50:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.253857+0000 mgr.y (mgr.14538) 276 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.254413+0000 mgr.y (mgr.14538) 277 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.255031+0000 mgr.y (mgr.14538) 278 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.255649+0000 mgr.y (mgr.14538) 279 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.256273+0000 mgr.y (mgr.14538) 280 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.256801+0000 mgr.y (mgr.14538) 281 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.450307+0000 osd.5 (osd.5) 145 : cluster 0 165.1 scrub starts 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.452397+0000 osd.5 (osd.5) 146 : cluster 0 165.1 scrub ok 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.582919+0000 osd.4 (osd.4) 163 : cluster 0 165.7 scrub starts 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.583409+0000 osd.3 (osd.3) 189 : cluster 0 165.9 scrub starts 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.587356+0000 osd.3 (osd.3) 190 : cluster 0 165.9 scrub ok 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.589281+0000 osd.4 (osd.4) 164 : cluster 0 165.7 scrub ok 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.822059+0000 mon.a (mon.0) 1495 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:35.823418+0000 mon.a (mon.0) 1496 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72304-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:37.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:35.909989+0000 mgr.y (mgr.14538) 282 : cluster 0 pgmap v271: 460 pgs: 104 active+undersized, 251 active+clean, 64 unknown, 41 active+undersized+degraded; 600 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 277/2289 objects degraded (12.101%) 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.075564+0000 osd.2 (osd.2) 107 : cluster 0 165.5 scrub starts 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.081678+0000 osd.2 (osd.2) 108 : cluster 0 165.5 scrub ok 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.224056+0000 osd.1 (osd.1) 171 : cluster 0 165.3 deep-scrub starts 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.233117+0000 osd.1 (osd.1) 172 : cluster 0 165.3 deep-scrub ok 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:36.326441+0000 mon.a (mon.0) 1497 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72304-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.331376+0000 mon.a (mon.0) 1498 : cluster 0 osdmap e197: 8 total, 7 up, 8 in 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:36.332342+0000 mon.a (mon.0) 1499 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72304-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:36.334529+0000 mon.c (mon.2) 254 : audit 1 from='client.? 172.21.15.69:0/1315613526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:36.335057+0000 mon.a (mon.0) 1500 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:37.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:36 smithi098 bash[23790]: audit 2024-04-02T04:50:36.685961+0000 mon.c (mon.2) 255 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:37.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:37.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:37.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:37.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:37.725 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:37.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:37.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:37.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.415272+0000 osd.5 (osd.5) 147 : cluster 0 165.8 scrub starts 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.417263+0000 osd.5 (osd.5) 148 : cluster 0 165.8 scrub ok 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.604205+0000 osd.3 (osd.3) 191 : cluster 0 165.0 deep-scrub starts 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:36.613137+0000 osd.3 (osd.3) 192 : cluster 0 165.0 deep-scrub ok 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.073973+0000 osd.2 (osd.2) 109 : cluster 0 13.8 scrub starts 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.075733+0000 osd.2 (osd.2) 110 : cluster 0 13.8 scrub ok 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.101554+0000 osd.0 (osd.0) 163 : cluster 0 41.1b scrub starts 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.102723+0000 osd.0 (osd.0) 164 : cluster 0 41.1b scrub ok 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.274102+0000 osd.1 (osd.1) 173 : cluster 0 165.4 scrub starts 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.276166+0000 osd.1 (osd.1) 174 : cluster 0 165.4 scrub ok 2024-04-02T04:50:38.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: audit 2024-04-02T04:50:37.330079+0000 mon.a (mon.0) 1501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.589518+0000 osd.3 (osd.3) 193 : cluster 0 165.6 deep-scrub starts 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: audit 2024-04-02T04:50:37.687148+0000 mon.c (mon.2) 256 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.771605+0000 mon.a (mon.0) 1502 : cluster 0 osdmap e198: 8 total, 7 up, 8 in 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[21528]: audit 2024-04-02T04:50:37.773231+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.69:0/1782413495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-72145-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.415272+0000 osd.5 (osd.5) 147 : cluster 0 165.8 scrub starts 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.417263+0000 osd.5 (osd.5) 148 : cluster 0 165.8 scrub ok 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.604205+0000 osd.3 (osd.3) 191 : cluster 0 165.0 deep-scrub starts 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:36.613137+0000 osd.3 (osd.3) 192 : cluster 0 165.0 deep-scrub ok 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.073973+0000 osd.2 (osd.2) 109 : cluster 0 13.8 scrub starts 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.075733+0000 osd.2 (osd.2) 110 : cluster 0 13.8 scrub ok 2024-04-02T04:50:38.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.101554+0000 osd.0 (osd.0) 163 : cluster 0 41.1b scrub starts 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.102723+0000 osd.0 (osd.0) 164 : cluster 0 41.1b scrub ok 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.274102+0000 osd.1 (osd.1) 173 : cluster 0 165.4 scrub starts 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.276166+0000 osd.1 (osd.1) 174 : cluster 0 165.4 scrub ok 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: audit 2024-04-02T04:50:37.330079+0000 mon.a (mon.0) 1501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.589518+0000 osd.3 (osd.3) 193 : cluster 0 165.6 deep-scrub starts 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: audit 2024-04-02T04:50:37.687148+0000 mon.c (mon.2) 256 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.771605+0000 mon.a (mon.0) 1502 : cluster 0 osdmap e198: 8 total, 7 up, 8 in 2024-04-02T04:50:38.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:37 smithi069 bash[28904]: audit 2024-04-02T04:50:37.773231+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.69:0/1782413495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-72145-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:38.091 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:38.118 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:38.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.415272+0000 osd.5 (osd.5) 147 : cluster 0 165.8 scrub starts 2024-04-02T04:50:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.417263+0000 osd.5 (osd.5) 148 : cluster 0 165.8 scrub ok 2024-04-02T04:50:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.604205+0000 osd.3 (osd.3) 191 : cluster 0 165.0 deep-scrub starts 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:36.613137+0000 osd.3 (osd.3) 192 : cluster 0 165.0 deep-scrub ok 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.073973+0000 osd.2 (osd.2) 109 : cluster 0 13.8 scrub starts 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.075733+0000 osd.2 (osd.2) 110 : cluster 0 13.8 scrub ok 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.101554+0000 osd.0 (osd.0) 163 : cluster 0 41.1b scrub starts 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.102723+0000 osd.0 (osd.0) 164 : cluster 0 41.1b scrub ok 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.274102+0000 osd.1 (osd.1) 173 : cluster 0 165.4 scrub starts 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.276166+0000 osd.1 (osd.1) 174 : cluster 0 165.4 scrub ok 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: audit 2024-04-02T04:50:37.330079+0000 mon.a (mon.0) 1501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-72153-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.589518+0000 osd.3 (osd.3) 193 : cluster 0 165.6 deep-scrub starts 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: audit 2024-04-02T04:50:37.687148+0000 mon.c (mon.2) 256 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.771605+0000 mon.a (mon.0) 1502 : cluster 0 osdmap e198: 8 total, 7 up, 8 in 2024-04-02T04:50:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:37 smithi098 bash[23790]: audit 2024-04-02T04:50:37.773231+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.69:0/1782413495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-72145-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:38.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:38.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:38.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:38.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:38.640 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:38.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:38.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:38.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:38.908 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:50:38.908 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:50:38.919 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:50:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:50:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:50:39.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:39.025 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:39.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:39.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:39.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:39.415 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:39.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-04-02T04:50:39.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:50:39.417 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-02T04:50:39.417 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-02T04:50:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:39.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:39.545 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:39.648 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:39.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.791 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.403521+0000 osd.5 (osd.5) 149 : cluster 0 6.d deep-scrub starts 2024-04-02T04:50:39.791 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.405663+0000 osd.5 (osd.5) 150 : cluster 0 6.d deep-scrub ok 2024-04-02T04:50:39.791 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.598516+0000 osd.3 (osd.3) 194 : cluster 0 165.6 deep-scrub ok 2024-04-02T04:50:39.791 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.839200+0000 osd.6 (osd.6) 127 : cluster 0 10.12 scrub starts 2024-04-02T04:50:39.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.840807+0000 osd.6 (osd.6) 128 : cluster 0 10.12 scrub ok 2024-04-02T04:50:39.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: audit 2024-04-02T04:50:38.332986+0000 mon.a (mon.0) 1504 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72304-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:39.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: audit 2024-04-02T04:50:38.333085+0000 mon.a (mon.0) 1505 : audit 1 from='client.? 172.21.15.69:0/1782413495' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-72145-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:39.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: cluster 2024-04-02T04:50:38.343097+0000 mon.a (mon.0) 1506 : cluster 0 osdmap e199: 8 total, 7 up, 8 in 2024-04-02T04:50:39.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:39 smithi098 bash[23790]: audit 2024-04-02T04:50:38.688561+0000 mon.c (mon.2) 257 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:39.806 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:39.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:39.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.403521+0000 osd.5 (osd.5) 149 : cluster 0 6.d deep-scrub starts 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.405663+0000 osd.5 (osd.5) 150 : cluster 0 6.d deep-scrub ok 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.598516+0000 osd.3 (osd.3) 194 : cluster 0 165.6 deep-scrub ok 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.839200+0000 osd.6 (osd.6) 127 : cluster 0 10.12 scrub starts 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.840807+0000 osd.6 (osd.6) 128 : cluster 0 10.12 scrub ok 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: audit 2024-04-02T04:50:38.332986+0000 mon.a (mon.0) 1504 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72304-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: audit 2024-04-02T04:50:38.333085+0000 mon.a (mon.0) 1505 : audit 1 from='client.? 172.21.15.69:0/1782413495' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-72145-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: cluster 2024-04-02T04:50:38.343097+0000 mon.a (mon.0) 1506 : cluster 0 osdmap e199: 8 total, 7 up, 8 in 2024-04-02T04:50:40.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[21528]: audit 2024-04-02T04:50:38.688561+0000 mon.c (mon.2) 257 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.403521+0000 osd.5 (osd.5) 149 : cluster 0 6.d deep-scrub starts 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.405663+0000 osd.5 (osd.5) 150 : cluster 0 6.d deep-scrub ok 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.598516+0000 osd.3 (osd.3) 194 : cluster 0 165.6 deep-scrub ok 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.839200+0000 osd.6 (osd.6) 127 : cluster 0 10.12 scrub starts 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.840807+0000 osd.6 (osd.6) 128 : cluster 0 10.12 scrub ok 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: audit 2024-04-02T04:50:38.332986+0000 mon.a (mon.0) 1504 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72304-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: audit 2024-04-02T04:50:38.333085+0000 mon.a (mon.0) 1505 : audit 1 from='client.? 172.21.15.69:0/1782413495' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-72145-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: cluster 2024-04-02T04:50:38.343097+0000 mon.a (mon.0) 1506 : cluster 0 osdmap e199: 8 total, 7 up, 8 in 2024-04-02T04:50:40.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:39 smithi069 bash[28904]: audit 2024-04-02T04:50:38.688561+0000 mon.c (mon.2) 257 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:40.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:40.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:40.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:40.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:40.321 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:40.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:40.448 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:40.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:40.573 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:40.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:40.709 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:40.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:37.912901+0000 mgr.y (mgr.14538) 283 : cluster 0 pgmap v274: 492 pgs: 1 active+clean+snaptrim, 121 active+undersized, 285 active+clean, 32 creating+peering, 53 active+undersized+degraded; 610 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 358/2964 objects degraded (12.078%) 2024-04-02T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.106564+0000 osd.2 (osd.2) 111 : cluster 0 40.11 scrub starts 2024-04-02T04:50:40.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:40.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:40.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:40.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:37.912901+0000 mgr.y (mgr.14538) 283 : cluster 0 pgmap v274: 492 pgs: 1 active+clean+snaptrim, 121 active+undersized, 285 active+clean, 32 creating+peering, 53 active+undersized+degraded; 610 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 358/2964 objects degraded (12.078%) 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.106564+0000 osd.2 (osd.2) 111 : cluster 0 40.11 scrub starts 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.108129+0000 osd.2 (osd.2) 112 : cluster 0 40.11 scrub ok 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.184852+0000 osd.0 (osd.0) 165 : cluster 0 10.4 scrub starts 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.186715+0000 osd.0 (osd.0) 166 : cluster 0 10.4 scrub ok 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.319652+0000 osd.1 (osd.1) 175 : cluster 0 10.14 deep-scrub starts 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.321595+0000 osd.1 (osd.1) 176 : cluster 0 10.14 deep-scrub ok 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.422334+0000 osd.5 (osd.5) 151 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.424561+0000 osd.5 (osd.5) 152 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.630084+0000 osd.3 (osd.3) 195 : cluster 0 5.0 scrub starts 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.632166+0000 osd.3 (osd.3) 196 : cluster 0 5.0 scrub ok 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.657765+0000 mon.a (mon.0) 1507 : cluster 3 Health check update: Degraded data redundancy: 358/2964 objects degraded (12.078%), 53 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.657805+0000 mon.a (mon.0) 1508 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.675401+0000 mon.a (mon.0) 1509 : cluster 0 osdmap e200: 8 total, 7 up, 8 in 2024-04-02T04:50:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: audit 2024-04-02T04:50:39.680722+0000 mon.a (mon.0) 1510 : audit 1 from='client.? 172.21.15.69:0/2860479876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-72153-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: audit 2024-04-02T04:50:39.683424+0000 mon.a (mon.0) 1511 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:40 smithi098 bash[23790]: audit 2024-04-02T04:50:39.691892+0000 mon.c (mon.2) 258 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:41.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:41.194 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:41.206 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:37.912901+0000 mgr.y (mgr.14538) 283 : cluster 0 pgmap v274: 492 pgs: 1 active+clean+snaptrim, 121 active+undersized, 285 active+clean, 32 creating+peering, 53 active+undersized+degraded; 610 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 358/2964 objects degraded (12.078%) 2024-04-02T04:50:41.206 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.106564+0000 osd.2 (osd.2) 111 : cluster 0 40.11 scrub starts 2024-04-02T04:50:41.206 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.108129+0000 osd.2 (osd.2) 112 : cluster 0 40.11 scrub ok 2024-04-02T04:50:41.206 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.184852+0000 osd.0 (osd.0) 165 : cluster 0 10.4 scrub starts 2024-04-02T04:50:41.206 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.186715+0000 osd.0 (osd.0) 166 : cluster 0 10.4 scrub ok 2024-04-02T04:50:41.206 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.319652+0000 osd.1 (osd.1) 175 : cluster 0 10.14 deep-scrub starts 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.321595+0000 osd.1 (osd.1) 176 : cluster 0 10.14 deep-scrub ok 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.422334+0000 osd.5 (osd.5) 151 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.424561+0000 osd.5 (osd.5) 152 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.630084+0000 osd.3 (osd.3) 195 : cluster 0 5.0 scrub starts 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.632166+0000 osd.3 (osd.3) 196 : cluster 0 5.0 scrub ok 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.657765+0000 mon.a (mon.0) 1507 : cluster 3 Health check update: Degraded data redundancy: 358/2964 objects degraded (12.078%), 53 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.657805+0000 mon.a (mon.0) 1508 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.675401+0000 mon.a (mon.0) 1509 : cluster 0 osdmap e200: 8 total, 7 up, 8 in 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: audit 2024-04-02T04:50:39.680722+0000 mon.a (mon.0) 1510 : audit 1 from='client.? 172.21.15.69:0/2860479876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-72153-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: audit 2024-04-02T04:50:39.683424+0000 mon.a (mon.0) 1511 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[28904]: audit 2024-04-02T04:50:39.691892+0000 mon.c (mon.2) 258 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:41.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.108129+0000 osd.2 (osd.2) 112 : cluster 0 40.11 scrub ok 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.184852+0000 osd.0 (osd.0) 165 : cluster 0 10.4 scrub starts 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.186715+0000 osd.0 (osd.0) 166 : cluster 0 10.4 scrub ok 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.319652+0000 osd.1 (osd.1) 175 : cluster 0 10.14 deep-scrub starts 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.321595+0000 osd.1 (osd.1) 176 : cluster 0 10.14 deep-scrub ok 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.422334+0000 osd.5 (osd.5) 151 : cluster 0 2.16 deep-scrub starts 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.424561+0000 osd.5 (osd.5) 152 : cluster 0 2.16 deep-scrub ok 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.630084+0000 osd.3 (osd.3) 195 : cluster 0 5.0 scrub starts 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.632166+0000 osd.3 (osd.3) 196 : cluster 0 5.0 scrub ok 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.657765+0000 mon.a (mon.0) 1507 : cluster 3 Health check update: Degraded data redundancy: 358/2964 objects degraded (12.078%), 53 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.657805+0000 mon.a (mon.0) 1508 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.675401+0000 mon.a (mon.0) 1509 : cluster 0 osdmap e200: 8 total, 7 up, 8 in 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: audit 2024-04-02T04:50:39.680722+0000 mon.a (mon.0) 1510 : audit 1 from='client.? 172.21.15.69:0/2860479876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-72153-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: audit 2024-04-02T04:50:39.683424+0000 mon.a (mon.0) 1511 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:40 smithi069 bash[21528]: audit 2024-04-02T04:50:39.691892+0000 mon.c (mon.2) 258 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:41.221 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:41.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:41.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:41.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:41.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:41.712 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:39.914449+0000 mgr.y (mgr.14538) 284 : cluster 0 pgmap v277: 500 pgs: 93 active+undersized, 251 active+clean, 104 unknown, 52 active+undersized+degraded; 610 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 6.0 KiB/s wr, 78 op/s; 357/2961 objects degraded (12.057%) 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.067840+0000 osd.0 (osd.0) 167 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.067959+0000 osd.0 (osd.0) 168 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.205777+0000 osd.0 (osd.0) 169 : cluster 0 6.0 scrub starts 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.207996+0000 osd.0 (osd.0) 170 : cluster 0 6.0 scrub ok 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.670283+0000 osd.3 (osd.3) 197 : cluster 0 6.1f scrub starts 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.671919+0000 osd.3 (osd.3) 198 : cluster 0 6.1f scrub ok 2024-04-02T04:50:41.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.678562+0000 mon.a (mon.0) 1512 : audit 1 from='client.? 172.21.15.69:0/2860479876' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-72153-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.678657+0000 mon.a (mon.0) 1513 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: cluster 2024-04-02T04:50:40.685953+0000 mon.a (mon.0) 1514 : cluster 0 osdmap e201: 8 total, 7 up, 8 in 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.696426+0000 mon.b (mon.1) 156 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.699101+0000 mon.c (mon.2) 259 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.700115+0000 mon.a (mon.0) 1515 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.701731+0000 mon.a (mon.0) 1516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.704007+0000 mon.c (mon.2) 260 : audit 1 from='client.? 172.21.15.69:0/70208617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[21528]: audit 2024-04-02T04:50:40.704980+0000 mon.a (mon.0) 1517 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:39.914449+0000 mgr.y (mgr.14538) 284 : cluster 0 pgmap v277: 500 pgs: 93 active+undersized, 251 active+clean, 104 unknown, 52 active+undersized+degraded; 610 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 6.0 KiB/s wr, 78 op/s; 357/2961 objects degraded (12.057%) 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.067840+0000 osd.0 (osd.0) 167 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.067959+0000 osd.0 (osd.0) 168 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:41.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.205777+0000 osd.0 (osd.0) 169 : cluster 0 6.0 scrub starts 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.207996+0000 osd.0 (osd.0) 170 : cluster 0 6.0 scrub ok 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.670283+0000 osd.3 (osd.3) 197 : cluster 0 6.1f scrub starts 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.671919+0000 osd.3 (osd.3) 198 : cluster 0 6.1f scrub ok 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.678562+0000 mon.a (mon.0) 1512 : audit 1 from='client.? 172.21.15.69:0/2860479876' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-72153-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.678657+0000 mon.a (mon.0) 1513 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: cluster 2024-04-02T04:50:40.685953+0000 mon.a (mon.0) 1514 : cluster 0 osdmap e201: 8 total, 7 up, 8 in 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.696426+0000 mon.b (mon.1) 156 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.699101+0000 mon.c (mon.2) 259 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.700115+0000 mon.a (mon.0) 1515 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.701731+0000 mon.a (mon.0) 1516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.704007+0000 mon.c (mon.2) 260 : audit 1 from='client.? 172.21.15.69:0/70208617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:41 smithi069 bash[28904]: audit 2024-04-02T04:50:40.704980+0000 mon.a (mon.0) 1517 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.761 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:41.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:39.914449+0000 mgr.y (mgr.14538) 284 : cluster 0 pgmap v277: 500 pgs: 93 active+undersized, 251 active+clean, 104 unknown, 52 active+undersized+degraded; 610 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 6.0 KiB/s wr, 78 op/s; 357/2961 objects degraded (12.057%) 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.067840+0000 osd.0 (osd.0) 167 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.067959+0000 osd.0 (osd.0) 168 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.205777+0000 osd.0 (osd.0) 169 : cluster 0 6.0 scrub starts 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.207996+0000 osd.0 (osd.0) 170 : cluster 0 6.0 scrub ok 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.670283+0000 osd.3 (osd.3) 197 : cluster 0 6.1f scrub starts 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.671919+0000 osd.3 (osd.3) 198 : cluster 0 6.1f scrub ok 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.678562+0000 mon.a (mon.0) 1512 : audit 1 from='client.? 172.21.15.69:0/2860479876' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-72153-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:41.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.678657+0000 mon.a (mon.0) 1513 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: cluster 2024-04-02T04:50:40.685953+0000 mon.a (mon.0) 1514 : cluster 0 osdmap e201: 8 total, 7 up, 8 in 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.696426+0000 mon.b (mon.1) 156 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.699101+0000 mon.c (mon.2) 259 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.700115+0000 mon.a (mon.0) 1515 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.701731+0000 mon.a (mon.0) 1516 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.704007+0000 mon.c (mon.2) 260 : audit 1 from='client.? 172.21.15.69:0/70208617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:41 smithi098 bash[23790]: audit 2024-04-02T04:50:40.704980+0000 mon.a (mon.0) 1517 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:41.892 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:41.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:41.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:42.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:42.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:42.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:42.275 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:42.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:42.403 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:42.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-02T04:50:42.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:42.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:42.653 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc api_snapshots: Running main() from gmock_main.cc 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (6358 ms) 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (5471 ms) 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (8453 ms) 2024-04-02T04:50:42.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5495 ms) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (25777 ms total) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4368 ms) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (5152 ms) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6149 ms) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (15669 ms total) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2970 ms) 2024-04-02T04:50:42.705 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2467 ms) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1544 ms) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1549 ms) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8530 ms total) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4000 ms) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4334 ms) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8334 ms total) 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-04-02T04:50:42.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-02T04:50:42.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (92641 ms total) 2024-04-02T04:50:42.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-02T04:50:42.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:41.424950+0000 osd.5 (osd.5) 153 : cluster 0 13.9 scrub starts 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:41.426736+0000 osd.5 (osd.5) 154 : cluster 0 13.9 scrub ok 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:41.682327+0000 mon.a (mon.0) 1518 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:41.682489+0000 mon.a (mon.0) 1519 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:41.682691+0000 mon.a (mon.0) 1520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:41.688340+0000 mon.b (mon.1) 157 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:42.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:41.691564+0000 mon.a (mon.0) 1521 : cluster 0 osdmap e202: 8 total, 7 up, 8 in 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:41.694916+0000 mon.a (mon.0) 1522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:41.712968+0000 mon.c (mon.2) 261 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:41.915671+0000 mgr.y (mgr.14538) 285 : cluster 0 pgmap v280: 492 pgs: 88 active+undersized, 249 active+clean, 104 unknown, 51 active+undersized+degraded; 594 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 355/2949 objects degraded (12.038%) 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:42.523999+0000 osd.4 (osd.4) 165 : cluster 0 6.17 scrub starts 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:42.525986+0000 osd.4 (osd.4) 166 : cluster 0 6.17 scrub ok 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: audit 2024-04-02T04:50:42.684923+0000 mon.a (mon.0) 1523 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[28904]: cluster 2024-04-02T04:50:42.691080+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e203: 8 total, 7 up, 8 in 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:41.424950+0000 osd.5 (osd.5) 153 : cluster 0 13.9 scrub starts 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:41.426736+0000 osd.5 (osd.5) 154 : cluster 0 13.9 scrub ok 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:41.682327+0000 mon.a (mon.0) 1518 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:41.682489+0000 mon.a (mon.0) 1519 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:41.682691+0000 mon.a (mon.0) 1520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:42.765 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:41.688340+0000 mon.b (mon.1) 157 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:41.691564+0000 mon.a (mon.0) 1521 : cluster 0 osdmap e202: 8 total, 7 up, 8 in 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:41.694916+0000 mon.a (mon.0) 1522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:41.712968+0000 mon.c (mon.2) 261 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:41.915671+0000 mgr.y (mgr.14538) 285 : cluster 0 pgmap v280: 492 pgs: 88 active+undersized, 249 active+clean, 104 unknown, 51 active+undersized+degraded; 594 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 355/2949 objects degraded (12.038%) 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:42.523999+0000 osd.4 (osd.4) 165 : cluster 0 6.17 scrub starts 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:42.525986+0000 osd.4 (osd.4) 166 : cluster 0 6.17 scrub ok 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: audit 2024-04-02T04:50:42.684923+0000 mon.a (mon.0) 1523 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:42.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:42 smithi069 bash[21528]: cluster 2024-04-02T04:50:42.691080+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e203: 8 total, 7 up, 8 in 2024-04-02T04:50:42.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:42.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:42.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:41.424950+0000 osd.5 (osd.5) 153 : cluster 0 13.9 scrub starts 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:41.426736+0000 osd.5 (osd.5) 154 : cluster 0 13.9 scrub ok 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:41.682327+0000 mon.a (mon.0) 1518 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72304-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:41.682489+0000 mon.a (mon.0) 1519 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:41.682691+0000 mon.a (mon.0) 1520 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-72145-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:41.688340+0000 mon.b (mon.1) 157 : audit 1 from='client.? 172.21.15.69:0/2181497743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:41.691564+0000 mon.a (mon.0) 1521 : cluster 0 osdmap e202: 8 total, 7 up, 8 in 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:41.694916+0000 mon.a (mon.0) 1522 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]: dispatch 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:41.712968+0000 mon.c (mon.2) 261 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:41.915671+0000 mgr.y (mgr.14538) 285 : cluster 0 pgmap v280: 492 pgs: 88 active+undersized, 249 active+clean, 104 unknown, 51 active+undersized+degraded; 594 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 355/2949 objects degraded (12.038%) 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:42.523999+0000 osd.4 (osd.4) 165 : cluster 0 6.17 scrub starts 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:42.525986+0000 osd.4 (osd.4) 166 : cluster 0 6.17 scrub ok 2024-04-02T04:50:42.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: audit 2024-04-02T04:50:42.684923+0000 mon.a (mon.0) 1523 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-72378-15"}]': finished 2024-04-02T04:50:42.891 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:42 smithi098 bash[23790]: cluster 2024-04-02T04:50:42.691080+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e203: 8 total, 7 up, 8 in 2024-04-02T04:50:42.904 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:42.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:43.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:43.033 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:43.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:43.160 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:43.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:50:43.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:43.289 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:43.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:43.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:43.548 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:43.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:43.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:43.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:43.800 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:43.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: audit 2024-04-02T04:50:42.704911+0000 mon.a (mon.0) 1525 : audit 1 from='client.? 172.21.15.69:0/3566438242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-72153-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: audit 2024-04-02T04:50:42.715678+0000 mon.c (mon.2) 262 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: cluster 2024-04-02T04:50:43.357778+0000 osd.1 (osd.1) 177 : cluster 0 2.6 scrub starts 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: cluster 2024-04-02T04:50:43.360446+0000 osd.1 (osd.1) 178 : cluster 0 2.6 scrub ok 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: cluster 2024-04-02T04:50:43.511796+0000 osd.4 (osd.4) 167 : cluster 0 4.c scrub starts 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: cluster 2024-04-02T04:50:43.512882+0000 osd.4 (osd.4) 168 : cluster 0 4.c scrub ok 2024-04-02T04:50:43.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:43 smithi098 bash[23790]: audit 2024-04-02T04:50:43.689699+0000 mon.a (mon.0) 1526 : audit 1 from='client.? 172.21.15.69:0/3566438242' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-72153-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:43.928 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:43.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: audit 2024-04-02T04:50:42.704911+0000 mon.a (mon.0) 1525 : audit 1 from='client.? 172.21.15.69:0/3566438242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-72153-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: audit 2024-04-02T04:50:42.715678+0000 mon.c (mon.2) 262 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: cluster 2024-04-02T04:50:43.357778+0000 osd.1 (osd.1) 177 : cluster 0 2.6 scrub starts 2024-04-02T04:50:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: cluster 2024-04-02T04:50:43.360446+0000 osd.1 (osd.1) 178 : cluster 0 2.6 scrub ok 2024-04-02T04:50:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: cluster 2024-04-02T04:50:43.511796+0000 osd.4 (osd.4) 167 : cluster 0 4.c scrub starts 2024-04-02T04:50:44.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: cluster 2024-04-02T04:50:43.512882+0000 osd.4 (osd.4) 168 : cluster 0 4.c scrub ok 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[28904]: audit 2024-04-02T04:50:43.689699+0000 mon.a (mon.0) 1526 : audit 1 from='client.? 172.21.15.69:0/3566438242' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-72153-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: audit 2024-04-02T04:50:42.704911+0000 mon.a (mon.0) 1525 : audit 1 from='client.? 172.21.15.69:0/3566438242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-72153-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: audit 2024-04-02T04:50:42.715678+0000 mon.c (mon.2) 262 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: cluster 2024-04-02T04:50:43.357778+0000 osd.1 (osd.1) 177 : cluster 0 2.6 scrub starts 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: cluster 2024-04-02T04:50:43.360446+0000 osd.1 (osd.1) 178 : cluster 0 2.6 scrub ok 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: cluster 2024-04-02T04:50:43.511796+0000 osd.4 (osd.4) 167 : cluster 0 4.c scrub starts 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: cluster 2024-04-02T04:50:43.512882+0000 osd.4 (osd.4) 168 : cluster 0 4.c scrub ok 2024-04-02T04:50:44.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:43 smithi069 bash[21528]: audit 2024-04-02T04:50:43.689699+0000 mon.a (mon.0) 1526 : audit 1 from='client.? 172.21.15.69:0/3566438242' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-72153-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:44.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:44.050 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:44.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:44.174 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/3292011947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c74107620 msgr2=0x7f7c7410f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- 172.21.15.69:0/3292011947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c74107620 0x7f7c7410f1a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00b0d0 tx=0x7f7c6c00b3e0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/3292011947 shutdown_connections 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- 172.21.15.69:0/3292011947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c74107620 0x7f7c7410f1a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- 172.21.15.69:0/3292011947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406a1d0 0x7f7c7406a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- 172.21.15.69:0/3292011947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7406ab80 0x7f7c74106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/3292011947 >> 172.21.15.69:0/3292011947 conn(0x7f7c740771e0 msgr2=0x7f7c740775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/3292011947 shutdown_connections 2024-04-02T04:50:44.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/3292011947 wait complete. 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 librados: starting msgr at 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 librados: starting objecter 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 Processor -- start 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- start start 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 librados: setting wanted keys 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 librados: calling monclient init 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7406a1d0 0x7f7c74101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406ab80 0x7f7c741017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c74107620 0x7f7c74101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74111b20 con 0x7f7c7406ab80 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741119a0 con 0x7f7c74107620 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7c74111ca0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c727fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406ab80 0x7f7c741017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c727fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406ab80 0x7f7c741017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43646/0 (socket says 172.21.15.69:43646) 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c727fc700 1 -- 172.21.15.69:0/155274050 learned_addr learned my addr 172.21.15.69:0/155274050 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7406a1d0 0x7f7c74101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c737fe700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c74107620 0x7f7c74101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 -- 172.21.15.69:0/155274050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c74107620 msgr2=0x7f7c74101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c74107620 0x7f7c74101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 -- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406ab80 msgr2=0x7f7c741017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406ab80 0x7f7c741017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 -- 172.21.15.69:0/155274050 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c741a65e0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c72ffd700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7406a1d0 0x7f7c74101270 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400e9f0 tx=0x7f7c6400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c5bfff700 10 client.?.objecter ms_handle_connect 0x7f7c7406a1d0 2024-04-02T04:50:44.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c5bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400cb70 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c741a68d0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7c640044a0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64003cb0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 10 client.25856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c741a6e60 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 librados: init done 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.176+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c7404ff00 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7c64018430 con 0x7f7c7406a1d0 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5c06f5e0 0x7f7c5c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(204..204 src has 74..204) ==== 12238+0+0 (secure 0 0 0) 0x7f7c6408f550 con 0x7f7c7406a1d0 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 10 client.25856.objecter ms_dispatch 0x7f7c74001550 osd_map(204..204 src has 74..204) 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 3 client.25856.objecter handle_osd_map got epochs [204,204] > 0 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 3 client.25856.objecter handle_osd_map decoding full epoch 204 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c727fc700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5c06f5e0 0x7f7c5c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 20 client.25856.objecter dump_active .. 0 homeless 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c5bfff700 10 client.25856.objecter ms_handle_connect 0x7f7c5c06f5e0 2024-04-02T04:50:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.180+0000 7f7c727fc700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5c06f5e0 0x7f7c5c071aa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c74108a20 tx=0x7f7c6800b460 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.184+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c64063bf0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:44.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- 172.21.15.69:0/2594831445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90108340 msgr2=0x7f8c90108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- 172.21.15.69:0/2594831445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90108340 0x7f8c90108720 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009990 tx=0x7f8c80009ca0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- 172.21.15.69:0/2594831445 shutdown_connections 2024-04-02T04:50:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- 172.21.15.69:0/2594831445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c90102c80 0x7f8c9010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- 172.21.15.69:0/2594831445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90108340 0x7f8c90108720 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- 172.21.15.69:0/2594831445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c901022c0 0x7f8c90102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- 172.21.15.69:0/2594831445 >> 172.21.15.69:0/2594831445 conn(0x7f8c900793f0 msgr2=0x7f8c90079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- 172.21.15.69:0/2594831445 shutdown_connections 2024-04-02T04:50:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- 172.21.15.69:0/2594831445 wait complete. 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 librados: starting msgr at 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 librados: starting objecter 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 Processor -- start 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- start start 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 librados: setting wanted keys 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 librados: calling monclient init 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c901022c0 0x7f8c90074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c90102c80 0x7f8c90074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c90108340 0x7f8c90074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c90111a20 con 0x7f8c901022c0 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c901118a0 con 0x7f8c90102c80 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c94dde700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8c90111ba0 con 0x7f8c90108340 2024-04-02T04:50:44.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c901022c0 0x7f8c90074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c901022c0 0x7f8c90074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43662/0 (socket says 172.21.15.69:43662) 2024-04-02T04:50:44.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 -- 172.21.15.69:0/3237109789 learned_addr learned my addr 172.21.15.69:0/3237109789 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:44.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c90108340 0x7f8c90074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c90102c80 0x7f8c90074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f7c741a70f0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 -- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c90108340 msgr2=0x7f8c90074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c90108340 0x7f8c90074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 -- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c90102c80 msgr2=0x7f8c90074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c90102c80 0x7f8c90074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8e59c700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c901a6530 con 0x7f8c901022c0 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.412+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c90108340 0x7f8c90074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c901022c0 0x7f8c90074540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8c80000c00 tx=0x7f8c800047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 10 client.?.objecter ms_handle_connect 0x7f8c901022c0 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80027070 con 0x7f8c901022c0 2024-04-02T04:50:44.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c901a67c0 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8c80022470 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80017650 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 10 client.15795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c901a6c70 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 1 librados: init done 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8c9004ff00 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8c8002a070 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7806f5e0 0x7f8c78071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(204..204 src has 74..204) ==== 12238+0+0 (secure 0 0 0) 0x7f8c80090a20 con 0x7f8c901022c0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 osd_map(204..204 src has 74..204) 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 3 client.15795.objecter handle_osd_map got epochs [204,204] > 0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 3 client.15795.objecter handle_osd_map decoding full epoch 204 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7806f5e0 0x7f8c78071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 20 client.15795.objecter dump_active .. 0 homeless 2024-04-02T04:50:44.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v204) ==== 54+0+16 (secure 0 0 0) 0x7f8c80031070 con 0x7f8c901022c0 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 10 client.15795.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 1 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8c90062e50 0x7f8c90063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 1 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 0 1 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c90075a70 osd=0 4 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 10 client.15795.objecter _send_command 1 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c901aa060 con 0x7f8c90062e50 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c7806f5e0 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8c7806f5e0 0x7f8c78071aa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8c84005fd0 tx=0x7f8c8400a560 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8c90062e50 0x7f8c90063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c90062e50 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.416+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8c90062e50 0x7f8c90063230 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.420+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c901aa060 con 0x7f8c90062e50 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.420+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 1: 0 ) 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.420+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 1 = system:0 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.420+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c90075a70 osd=0 4 2024-04-02T04:50:44.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.420+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 0 1 2024-04-02T04:50:44.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 10 client.15795.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 2 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 2 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c90075a70 osd=0 3 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 0 2 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c90075a70 osd=0 4 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 10 client.15795.objecter _send_command 2 2024-04-02T04:50:44.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.436+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c900628d0 con 0x7f8c90062e50 2024-04-02T04:50:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c900628d0 con 0x7f8c90062e50 2024-04-02T04:50:44.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c90075a70 osd=0 4 2024-04-02T04:50:44.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 0 2 2024-04-02T04:50:44.440 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:50:44.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 10 client.15795.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8c9002c9b0 0x7f8c9002cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 1 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a8a0 osd=1 4 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 10 client.15795.objecter _send_command 3 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8c9010af20 con 0x7f8c9002c9b0 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8c9002c9b0 0x7f8c9002cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c9002c9b0 2024-04-02T04:50:44.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8c9002c9b0 0x7f8c9002cdd0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c9010af20 con 0x7f8c9002c9b0 2024-04-02T04:50:44.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 3: 0 ) 2024-04-02T04:50:44.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 3 = system:0 2024-04-02T04:50:44.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c9010a8a0 osd=1 4 2024-04-02T04:50:44.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.440+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 1 3 2024-04-02T04:50:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:44.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 10 client.15795.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 4 2024-04-02T04:50:44.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 4 2024-04-02T04:50:44.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c9010a8a0 osd=1 3 2024-04-02T04:50:44.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 1 4 2024-04-02T04:50:44.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a8a0 osd=1 4 2024-04-02T04:50:44.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 10 client.15795.objecter _send_command 4 2024-04-02T04:50:44.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.456+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c9010a320 con 0x7f8c9002c9b0 2024-04-02T04:50:44.460 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c9010a320 con 0x7f8c9002c9b0 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c9010a8a0 osd=1 4 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 1 4 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 10 client.15795.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 5 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8c901ac220 0x7f8c901ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 5 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 2 5 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a720 osd=2 4 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 10 client.15795.objecter _send_command 5 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8c901b1470 con 0x7f8c901ac220 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8c901ac220 0x7f8c901ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8c901ac220 0x7f8c901ae610 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c901ac220 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c901b1470 con 0x7f8c901ac220 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 5: 0 ) 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 5 = system:0 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c9010a720 osd=2 4 2024-04-02T04:50:44.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.460+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 2 5 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 10 client.15795.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 6 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 6 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c9010a720 osd=2 3 2024-04-02T04:50:44.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 2 6 2024-04-02T04:50:44.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c9010a720 osd=2 4 2024-04-02T04:50:44.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 10 client.15795.objecter _send_command 6 2024-04-02T04:50:44.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.476+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c9002d560 con 0x7f8c901ac220 2024-04-02T04:50:44.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c9002d560 con 0x7f8c901ac220 2024-04-02T04:50:44.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c9010a720 osd=2 4 2024-04-02T04:50:44.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 2 6 2024-04-02T04:50:44.481 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 10 client.15795.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 7 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8c901afcd0 0x7f8c901b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 7 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 3 7 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a96b0 osd=3 4 2024-04-02T04:50:44.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 10 client.15795.objecter _send_command 7 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8c901a9d80 con 0x7f8c901afcd0 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8c901afcd0 0x7f8c901b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c901afcd0 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8c901afcd0 0x7f8c901b0130 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c901a9d80 con 0x7f8c901afcd0 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 7: 0 ) 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 7 = system:0 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901a96b0 osd=3 4 2024-04-02T04:50:44.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.480+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 3 7 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 10 client.15795.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 8 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 8 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901a96b0 osd=3 3 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 3 8 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a96b0 osd=3 4 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 10 client.15795.objecter _send_command 8 2024-04-02T04:50:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c901a9130 con 0x7f8c901afcd0 2024-04-02T04:50:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c901a9130 con 0x7f8c901afcd0 2024-04-02T04:50:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901a96b0 osd=3 4 2024-04-02T04:50:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 3 8 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 10 client.15795.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 9 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8c901b3630 0x7f8c901b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 9 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 4 9 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a8ee0 osd=4 4 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 10 client.15795.objecter _send_command 9 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8c901a90a0 con 0x7f8c901b3630 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8c901b3630 0x7f8c901b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c901b3630 2024-04-02T04:50:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.500+0000 7f8c8e59c700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8c901b3630 0x7f8c901b5a20 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.504+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c901a90a0 con 0x7f8c901b3630 2024-04-02T04:50:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.504+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 9: 0 ) 2024-04-02T04:50:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.504+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 9 = system:0 2024-04-02T04:50:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.504+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901a8ee0 osd=4 4 2024-04-02T04:50:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.504+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 4 9 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 10 client.15795.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 10 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 10 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901a8ee0 osd=4 3 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 4 10 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901a8ee0 osd=4 4 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 10 client.15795.objecter _send_command 10 2024-04-02T04:50:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.520+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c901b5f60 con 0x7f8c901b3630 2024-04-02T04:50:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c901b5f60 con 0x7f8c901b3630 2024-04-02T04:50:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901a8ee0 osd=4 4 2024-04-02T04:50:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 4 10 2024-04-02T04:50:44.524 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 10 client.15795.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 11 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8c901b6680 0x7f8c901b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 11 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 5 11 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901b5f60 osd=5 4 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 10 client.15795.objecter _send_command 11 2024-04-02T04:50:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8c901bba20 con 0x7f8c901b6680 2024-04-02T04:50:44.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8c901b6680 0x7f8c901b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c901b6680 2024-04-02T04:50:44.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c8dd9b700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8c901b6680 0x7f8c901b8a70 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c901bba20 con 0x7f8c901b6680 2024-04-02T04:50:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 11: 0 ) 2024-04-02T04:50:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 11 = system:0 2024-04-02T04:50:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901b5f60 osd=5 4 2024-04-02T04:50:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.524+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 5 11 2024-04-02T04:50:44.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:44.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 10 client.15795.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 12 2024-04-02T04:50:44.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 12 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901b5f60 osd=5 3 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 5 12 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901b5f60 osd=5 4 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 10 client.15795.objecter _send_command 12 2024-04-02T04:50:44.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.544+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c901bba20 con 0x7f8c901b6680 2024-04-02T04:50:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c901bba20 con 0x7f8c901b6680 2024-04-02T04:50:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901b5f60 osd=5 4 2024-04-02T04:50:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 5 12 2024-04-02T04:50:44.549 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:50:44.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 10 client.15795.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 13 2024-04-02T04:50:44.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8c901bad80 0x7f8c901bb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 13 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 6 13 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901bba20 osd=6 4 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 10 client.15795.objecter _send_command 13 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8c901bedc0 con 0x7f8c901bad80 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8c901bad80 0x7f8c901bb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c7f7fe700 10 client.15795.objecter ms_handle_connect 0x7f8c901bad80 2024-04-02T04:50:44.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.548+0000 7f8c8ed9d700 1 --2- 172.21.15.69:0/3237109789 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8c901bad80 0x7f8c901bb1e0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.552+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c901bedc0 con 0x7f8c901bad80 2024-04-02T04:50:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.552+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 13: 0 ) 2024-04-02T04:50:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.552+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 13 = system:0 2024-04-02T04:50:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.552+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901bba20 osd=6 4 2024-04-02T04:50:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.552+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 6 13 2024-04-02T04:50:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 10 client.15795.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 14 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 20 client.15795.objecter _get_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_remove -1 14 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 20 client.15795.objecter get_session s=0x7f8c901bba20 osd=6 3 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign 6 14 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 20 client.15795.objecter put_session s=0x7f8c901bba20 osd=6 4 2024-04-02T04:50:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 10 client.15795.objecter _send_command 14 2024-04-02T04:50:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.572+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c901b9020 con 0x7f8c901bad80 2024-04-02T04:50:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c901b9020 con 0x7f8c901bad80 2024-04-02T04:50:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c7f7fe700 10 client.15795.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:50:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c7f7fe700 10 client.15795.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:50:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c7f7fe700 20 client.15795.objecter put_session s=0x7f8c901bba20 osd=6 4 2024-04-02T04:50:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c7f7fe700 15 client.15795.objecter _session_command_op_remove 6 14 2024-04-02T04:50:44.577 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:50:44.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c94dde700 10 client.15795.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:50:44.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c94dde700 15 client.15795.objecter _session_command_op_assign -1 15 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c94dde700 20 client.15795.objecter _get_session osd=-1 returning homeless 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c94dde700 10 client.15795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=205}) -- 0x7f8c901ba400 con 0x7f8c901022c0 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c94dde700 1 -- 172.21.15.69:0/3237109789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8c901b9020 con 0x7f8c901022c0 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c7f7fe700 1 -- 172.21.15.69:0/3237109789 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_get_version_reply(handle=1 version=204) ==== 24+0+0 (secure 0 0 0) 0x7f8c8008d690 con 0x7f8c901022c0 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c8cd99700 10 client.15795.objecter _check_command_map_dne tid 15 current 204 map_dne_bound 204 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c8cd99700 10 client.15795.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.576+0000 7f8c8cd99700 15 client.15795.objecter _session_command_op_remove -1 15 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:50:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:50:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:50:44.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.664+0000 7f7c5bfff700 1 -- 172.21.15.69:0/155274050 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v205) ==== 87+0+0 (secure 0 0 0) 0x7f7c64059fb0 con 0x7f7c7406a1d0 2024-04-02T04:50:44.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.668+0000 7f7c79717700 10 librados: watch_flush enter 2024-04-02T04:50:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.668+0000 7f7c79717700 10 librados: watch_flush exit 2024-04-02T04:50:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.668+0000 7f7c79717700 20 client.25856.objecter shutdown clearing up homeless session... 2024-04-02T04:50:44.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.668+0000 7f7c79717700 10 client.25856.objecter successfully canceled tick 2024-04-02T04:50:44.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.672+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5c06f5e0 msgr2=0x7f7c5c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.672+0000 7f7c79717700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5c06f5e0 0x7f7c5c071aa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c74108a20 tx=0x7f7c6800b460 comp rx=0 tx=0).stop 2024-04-02T04:50:44.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.672+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7406a1d0 msgr2=0x7f7c74101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:44.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.672+0000 7f7c79717700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7406a1d0 0x7f7c74101270 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400e9f0 tx=0x7f7c6400ed00 comp rx=0 tx=0).stop 2024-04-02T04:50:44.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.684+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 shutdown_connections 2024-04-02T04:50:44.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.684+0000 7f7c79717700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7c5c06f5e0 0x7f7c5c071aa0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.684+0000 7f7c79717700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c74107620 0x7f7c74101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.684+0000 7f7c79717700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7406a1d0 0x7f7c74101270 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.684+0000 7f7c79717700 1 --2- 172.21.15.69:0/155274050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7406ab80 0x7f7c741017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:44.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.684+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 >> 172.21.15.69:0/155274050 conn(0x7f7c740771e0 msgr2=0x7f7c740ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:44.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.688+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 shutdown_connections 2024-04-02T04:50:44.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.688+0000 7f7c79717700 1 -- 172.21.15.69:0/155274050 wait complete. 2024-04-02T04:50:44.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:44.688+0000 7f7c79717700 1 librados: shutdown 2024-04-02T04:50:44.699 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:50:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:44.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:44.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:44.976 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:44.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:45.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:45.100 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:45.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:45.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:43.052824+0000 mgr.y (mgr.14538) 286 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:43.715039+0000 mon.b (mon.1) 158 : audit 1 from='client.? 172.21.15.69:0/422627931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:43.715546+0000 mon.a (mon.0) 1527 : cluster 0 osdmap e204: 8 total, 7 up, 8 in 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:43.717228+0000 mon.c (mon.2) 263 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:43.735297+0000 mon.a (mon.0) 1528 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:43.735568+0000 mon.a (mon.0) 1529 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:43.917242+0000 mgr.y (mgr.14538) 287 : cluster 0 pgmap v283: 452 pgs: 80 active+undersized, 228 active+clean, 32 creating+peering, 64 unknown, 48 active+undersized+degraded; 592 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.139446+0000 osd.2 (osd.2) 113 : cluster 0 13.a scrub starts 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.141581+0000 osd.2 (osd.2) 114 : cluster 0 13.a scrub ok 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.351512+0000 osd.1 (osd.1) 179 : cluster 0 2.9 scrub starts 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.352501+0000 osd.1 (osd.1) 180 : cluster 0 2.9 scrub ok 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.419647+0000 mon.c (mon.2) 264 : audit 1 from='client.? 172.21.15.69:0/155274050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.419980+0000 mon.a (mon.0) 1530 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.421681+0000 mon.a (mon.0) 1531 : audit 0 from='client.? 172.21.15.69:0/3237109789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.659302+0000 mon.a (mon.0) 1532 : cluster 3 Health check update: Degraded data redundancy: 355/2949 objects degraded (12.038%), 51 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.659336+0000 mon.a (mon.0) 1533 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.660325+0000 mon.a (mon.0) 1534 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.662543+0000 mon.a (mon.0) 1535 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.662596+0000 mon.a (mon.0) 1536 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.662676+0000 mon.a (mon.0) 1537 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.666314+0000 mon.a (mon.0) 1538 : cluster 0 osdmap e205: 8 total, 7 up, 8 in 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.667461+0000 mon.a (mon.0) 1539 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.670231+0000 mon.b (mon.1) 159 : audit 1 from='client.? 172.21.15.69:0/2527228822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:45 smithi098 bash[23790]: audit 2024-04-02T04:50:44.676064+0000 mon.a (mon.0) 1540 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.226 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:45.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:45.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:43.052824+0000 mgr.y (mgr.14538) 286 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:45.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:43.715039+0000 mon.b (mon.1) 158 : audit 1 from='client.? 172.21.15.69:0/422627931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:43.715546+0000 mon.a (mon.0) 1527 : cluster 0 osdmap e204: 8 total, 7 up, 8 in 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:43.717228+0000 mon.c (mon.2) 263 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:43.735297+0000 mon.a (mon.0) 1528 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:43.735568+0000 mon.a (mon.0) 1529 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:43.917242+0000 mgr.y (mgr.14538) 287 : cluster 0 pgmap v283: 452 pgs: 80 active+undersized, 228 active+clean, 32 creating+peering, 64 unknown, 48 active+undersized+degraded; 592 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.139446+0000 osd.2 (osd.2) 113 : cluster 0 13.a scrub starts 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.141581+0000 osd.2 (osd.2) 114 : cluster 0 13.a scrub ok 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.351512+0000 osd.1 (osd.1) 179 : cluster 0 2.9 scrub starts 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.352501+0000 osd.1 (osd.1) 180 : cluster 0 2.9 scrub ok 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.419647+0000 mon.c (mon.2) 264 : audit 1 from='client.? 172.21.15.69:0/155274050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.419980+0000 mon.a (mon.0) 1530 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.421681+0000 mon.a (mon.0) 1531 : audit 0 from='client.? 172.21.15.69:0/3237109789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.659302+0000 mon.a (mon.0) 1532 : cluster 3 Health check update: Degraded data redundancy: 355/2949 objects degraded (12.038%), 51 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.659336+0000 mon.a (mon.0) 1533 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:45.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.660325+0000 mon.a (mon.0) 1534 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.662543+0000 mon.a (mon.0) 1535 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:43.052824+0000 mgr.y (mgr.14538) 286 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:43.715039+0000 mon.b (mon.1) 158 : audit 1 from='client.? 172.21.15.69:0/422627931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:43.715546+0000 mon.a (mon.0) 1527 : cluster 0 osdmap e204: 8 total, 7 up, 8 in 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:43.717228+0000 mon.c (mon.2) 263 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:43.735297+0000 mon.a (mon.0) 1528 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:45.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:43.735568+0000 mon.a (mon.0) 1529 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:43.917242+0000 mgr.y (mgr.14538) 287 : cluster 0 pgmap v283: 452 pgs: 80 active+undersized, 228 active+clean, 32 creating+peering, 64 unknown, 48 active+undersized+degraded; 592 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.139446+0000 osd.2 (osd.2) 113 : cluster 0 13.a scrub starts 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.141581+0000 osd.2 (osd.2) 114 : cluster 0 13.a scrub ok 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.351512+0000 osd.1 (osd.1) 179 : cluster 0 2.9 scrub starts 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.352501+0000 osd.1 (osd.1) 180 : cluster 0 2.9 scrub ok 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.419647+0000 mon.c (mon.2) 264 : audit 1 from='client.? 172.21.15.69:0/155274050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.419980+0000 mon.a (mon.0) 1530 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.421681+0000 mon.a (mon.0) 1531 : audit 0 from='client.? 172.21.15.69:0/3237109789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.659302+0000 mon.a (mon.0) 1532 : cluster 3 Health check update: Degraded data redundancy: 355/2949 objects degraded (12.038%), 51 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.659336+0000 mon.a (mon.0) 1533 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.660325+0000 mon.a (mon.0) 1534 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:50:45.343 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.662543+0000 mon.a (mon.0) 1535 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.662596+0000 mon.a (mon.0) 1536 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.662676+0000 mon.a (mon.0) 1537 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.666314+0000 mon.a (mon.0) 1538 : cluster 0 osdmap e205: 8 total, 7 up, 8 in 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.667461+0000 mon.a (mon.0) 1539 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.670231+0000 mon.b (mon.1) 159 : audit 1 from='client.? 172.21.15.69:0/2527228822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[21528]: audit 2024-04-02T04:50:44.676064+0000 mon.a (mon.0) 1540 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.662596+0000 mon.a (mon.0) 1536 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-72145-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:45.344 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.662676+0000 mon.a (mon.0) 1537 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:50:45.345 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.666314+0000 mon.a (mon.0) 1538 : cluster 0 osdmap e205: 8 total, 7 up, 8 in 2024-04-02T04:50:45.345 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.667461+0000 mon.a (mon.0) 1539 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]: dispatch 2024-04-02T04:50:45.345 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.670231+0000 mon.b (mon.1) 159 : audit 1 from='client.? 172.21.15.69:0/2527228822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.345 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:45 smithi069 bash[28904]: audit 2024-04-02T04:50:44.676064+0000 mon.a (mon.0) 1540 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:45.355 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:45.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:45.484 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:45.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:45.614 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:45.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:45.747 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:45.850 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:45.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:45.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:45.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:46.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:46.129 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:46.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:46.252 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:50:46.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:50:46.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:46.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:46.510 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:46.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:46.624 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.255951+0000 osd.0 (osd.0) 171 : cluster 0 41.9 scrub starts 2024-04-02T04:50:46.624 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.257735+0000 osd.0 (osd.0) 172 : cluster 0 41.9 scrub ok 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.470342+0000 osd.5 (osd.5) 155 : cluster 0 43.6 scrub starts 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: cluster 2024-04-02T04:50:44.471501+0000 osd.5 (osd.5) 156 : cluster 0 43.6 scrub ok 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: audit 2024-04-02T04:50:44.720081+0000 mon.c (mon.2) 265 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: audit 2024-04-02T04:50:45.665493+0000 mon.a (mon.0) 1541 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: audit 2024-04-02T04:50:45.665594+0000 mon.a (mon.0) 1542 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: cluster 2024-04-02T04:50:45.672138+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e206: 8 total, 7 up, 8 in 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: audit 2024-04-02T04:50:45.676487+0000 mon.c (mon.2) 266 : audit 1 from='client.? 172.21.15.69:0/1120642269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: audit 2024-04-02T04:50:45.679564+0000 mon.a (mon.0) 1544 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:46.625 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:46 smithi098 bash[23790]: audit 2024-04-02T04:50:45.723028+0000 mon.c (mon.2) 267 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:46.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:46.762 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:46.812 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:46.831 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.255951+0000 osd.0 (osd.0) 171 : cluster 0 41.9 scrub starts 2024-04-02T04:50:46.831 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.257735+0000 osd.0 (osd.0) 172 : cluster 0 41.9 scrub ok 2024-04-02T04:50:46.831 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.470342+0000 osd.5 (osd.5) 155 : cluster 0 43.6 scrub starts 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: cluster 2024-04-02T04:50:44.471501+0000 osd.5 (osd.5) 156 : cluster 0 43.6 scrub ok 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: audit 2024-04-02T04:50:44.720081+0000 mon.c (mon.2) 265 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: audit 2024-04-02T04:50:45.665493+0000 mon.a (mon.0) 1541 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: audit 2024-04-02T04:50:45.665594+0000 mon.a (mon.0) 1542 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: cluster 2024-04-02T04:50:45.672138+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e206: 8 total, 7 up, 8 in 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: audit 2024-04-02T04:50:45.676487+0000 mon.c (mon.2) 266 : audit 1 from='client.? 172.21.15.69:0/1120642269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: audit 2024-04-02T04:50:45.679564+0000 mon.a (mon.0) 1544 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[28904]: audit 2024-04-02T04:50:45.723028+0000 mon.c (mon.2) 267 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.255951+0000 osd.0 (osd.0) 171 : cluster 0 41.9 scrub starts 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.257735+0000 osd.0 (osd.0) 172 : cluster 0 41.9 scrub ok 2024-04-02T04:50:46.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.470342+0000 osd.5 (osd.5) 155 : cluster 0 43.6 scrub starts 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: cluster 2024-04-02T04:50:44.471501+0000 osd.5 (osd.5) 156 : cluster 0 43.6 scrub ok 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: audit 2024-04-02T04:50:44.720081+0000 mon.c (mon.2) 265 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: audit 2024-04-02T04:50:45.665493+0000 mon.a (mon.0) 1541 : audit 1 from='client.? 172.21.15.69:0/36550613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72304-24"}]': finished 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: audit 2024-04-02T04:50:45.665594+0000 mon.a (mon.0) 1542 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72389-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: cluster 2024-04-02T04:50:45.672138+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e206: 8 total, 7 up, 8 in 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: audit 2024-04-02T04:50:45.676487+0000 mon.c (mon.2) 266 : audit 1 from='client.? 172.21.15.69:0/1120642269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: audit 2024-04-02T04:50:45.679564+0000 mon.a (mon.0) 1544 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:46.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:46 smithi069 bash[21528]: audit 2024-04-02T04:50:45.723028+0000 mon.c (mon.2) 267 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:46.865 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:46.883 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:46.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:46.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:47.114 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:47.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:47.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:47.250 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:50:47.266 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:47.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:47.394 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:47.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:47.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:47.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:47.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: cluster 2024-04-02T04:50:45.918047+0000 mgr.y (mgr.14538) 288 : cluster 0 pgmap v286: 452 pgs: 80 active+undersized, 228 active+clean, 96 unknown, 48 active+undersized+degraded; 592 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:50:47.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.671838+0000 mon.a (mon.0) 1545 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: cluster 2024-04-02T04:50:46.680243+0000 mon.a (mon.0) 1546 : cluster 0 osdmap e207: 8 total, 7 up, 8 in 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.691934+0000 mon.c (mon.2) 268 : audit 1 from='client.? 172.21.15.69:0/4145311892' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.693084+0000 mon.a (mon.0) 1547 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.696202+0000 mon.a (mon.0) 1548 : audit 1 from='client.? 172.21.15.69:0/1470643160' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-72304-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.724691+0000 mon.c (mon.2) 269 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.978653+0000 mon.b (mon.1) 160 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:47.770 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:47 smithi098 bash[23790]: audit 2024-04-02T04:50:46.978909+0000 mon.a (mon.0) 1549 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:47.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:47.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:47.908 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:47.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: cluster 2024-04-02T04:50:45.918047+0000 mgr.y (mgr.14538) 288 : cluster 0 pgmap v286: 452 pgs: 80 active+undersized, 228 active+clean, 96 unknown, 48 active+undersized+degraded; 592 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.671838+0000 mon.a (mon.0) 1545 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: cluster 2024-04-02T04:50:46.680243+0000 mon.a (mon.0) 1546 : cluster 0 osdmap e207: 8 total, 7 up, 8 in 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.691934+0000 mon.c (mon.2) 268 : audit 1 from='client.? 172.21.15.69:0/4145311892' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.693084+0000 mon.a (mon.0) 1547 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.696202+0000 mon.a (mon.0) 1548 : audit 1 from='client.? 172.21.15.69:0/1470643160' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-72304-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.724691+0000 mon.c (mon.2) 269 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.978653+0000 mon.b (mon.1) 160 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[28904]: audit 2024-04-02T04:50:46.978909+0000 mon.a (mon.0) 1549 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:47.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: cluster 2024-04-02T04:50:45.918047+0000 mgr.y (mgr.14538) 288 : cluster 0 pgmap v286: 452 pgs: 80 active+undersized, 228 active+clean, 96 unknown, 48 active+undersized+degraded; 592 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.671838+0000 mon.a (mon.0) 1545 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: cluster 2024-04-02T04:50:46.680243+0000 mon.a (mon.0) 1546 : cluster 0 osdmap e207: 8 total, 7 up, 8 in 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.691934+0000 mon.c (mon.2) 268 : audit 1 from='client.? 172.21.15.69:0/4145311892' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.693084+0000 mon.a (mon.0) 1547 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.696202+0000 mon.a (mon.0) 1548 : audit 1 from='client.? 172.21.15.69:0/1470643160' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-72304-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.724691+0000 mon.c (mon.2) 269 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.978653+0000 mon.b (mon.1) 160 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:47.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:47 smithi069 bash[21528]: audit 2024-04-02T04:50:46.978909+0000 mon.a (mon.0) 1549 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:50:48.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:48.037 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:48.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:48.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:48.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:50:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:48.328 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:50:48.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:48.410 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:48.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:48.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:48.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:48.662 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:48.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:48.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:48.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:48.944 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:48 smithi098 bash[23790]: audit 2024-04-02T04:50:47.680347+0000 mon.a (mon.0) 1550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:48.944 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:48 smithi098 bash[23790]: audit 2024-04-02T04:50:47.680431+0000 mon.a (mon.0) 1551 : audit 1 from='client.? 172.21.15.69:0/1470643160' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-72304-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:48.944 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:48 smithi098 bash[23790]: cluster 2024-04-02T04:50:47.684655+0000 mon.a (mon.0) 1552 : cluster 0 osdmap e208: 8 total, 7 up, 8 in 2024-04-02T04:50:48.944 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:48 smithi098 bash[23790]: audit 2024-04-02T04:50:47.727933+0000 mon.c (mon.2) 270 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:49.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:49.022 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:50:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:50:49.022 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[21528]: audit 2024-04-02T04:50:47.680347+0000 mon.a (mon.0) 1550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:49.022 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[21528]: audit 2024-04-02T04:50:47.680431+0000 mon.a (mon.0) 1551 : audit 1 from='client.? 172.21.15.69:0/1470643160' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-72304-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:49.023 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[21528]: cluster 2024-04-02T04:50:47.684655+0000 mon.a (mon.0) 1552 : cluster 0 osdmap e208: 8 total, 7 up, 8 in 2024-04-02T04:50:49.023 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[21528]: audit 2024-04-02T04:50:47.727933+0000 mon.c (mon.2) 270 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:49.023 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[28904]: audit 2024-04-02T04:50:47.680347+0000 mon.a (mon.0) 1550 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-72145-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:49.023 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[28904]: audit 2024-04-02T04:50:47.680431+0000 mon.a (mon.0) 1551 : audit 1 from='client.? 172.21.15.69:0/1470643160' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-72304-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:49.023 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[28904]: cluster 2024-04-02T04:50:47.684655+0000 mon.a (mon.0) 1552 : cluster 0 osdmap e208: 8 total, 7 up, 8 in 2024-04-02T04:50:49.023 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:48 smithi069 bash[28904]: audit 2024-04-02T04:50:47.727933+0000 mon.c (mon.2) 270 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:49.033 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:49.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:49.159 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:49.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:49.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:49.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:49.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:49.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:49.671 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:49.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:49.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: cluster 2024-04-02T04:50:47.919910+0000 mgr.y (mgr.14538) 289 : cluster 0 pgmap v289: 484 pgs: 128 active+undersized, 306 active+clean, 50 active+undersized+degraded; 600 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s; 352/2940 objects degraded (11.973%) 2024-04-02T04:50:49.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: cluster 2024-04-02T04:50:48.701024+0000 mon.a (mon.0) 1553 : cluster 0 osdmap e209: 8 total, 7 up, 8 in 2024-04-02T04:50:49.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: audit 2024-04-02T04:50:48.710540+0000 mon.c (mon.2) 271 : audit 1 from='client.? 172.21.15.69:0/3930140329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:49.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: audit 2024-04-02T04:50:48.714261+0000 mon.a (mon.0) 1554 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:49.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: audit 2024-04-02T04:50:48.731260+0000 mon.c (mon.2) 272 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:49.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: audit 2024-04-02T04:50:49.623396+0000 mon.a (mon.0) 1555 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:49.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: audit 2024-04-02T04:50:49.624728+0000 mon.a (mon.0) 1556 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:49.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: audit 2024-04-02T04:50:49.625084+0000 mon.a (mon.0) 1557 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:49.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:49 smithi098 bash[23790]: cluster 2024-04-02T04:50:49.661290+0000 mon.a (mon.0) 1558 : cluster 3 Health check update: Degraded data redundancy: 352/2940 objects degraded (11.973%), 50 pgs degraded, 108 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:49.795 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:49.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:49.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: cluster 2024-04-02T04:50:47.919910+0000 mgr.y (mgr.14538) 289 : cluster 0 pgmap v289: 484 pgs: 128 active+undersized, 306 active+clean, 50 active+undersized+degraded; 600 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s; 352/2940 objects degraded (11.973%) 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: cluster 2024-04-02T04:50:48.701024+0000 mon.a (mon.0) 1553 : cluster 0 osdmap e209: 8 total, 7 up, 8 in 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: audit 2024-04-02T04:50:48.710540+0000 mon.c (mon.2) 271 : audit 1 from='client.? 172.21.15.69:0/3930140329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: audit 2024-04-02T04:50:48.714261+0000 mon.a (mon.0) 1554 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: audit 2024-04-02T04:50:48.731260+0000 mon.c (mon.2) 272 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: audit 2024-04-02T04:50:49.623396+0000 mon.a (mon.0) 1555 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: audit 2024-04-02T04:50:49.624728+0000 mon.a (mon.0) 1556 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: audit 2024-04-02T04:50:49.625084+0000 mon.a (mon.0) 1557 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:49.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[21528]: cluster 2024-04-02T04:50:49.661290+0000 mon.a (mon.0) 1558 : cluster 3 Health check update: Degraded data redundancy: 352/2940 objects degraded (11.973%), 50 pgs degraded, 108 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: cluster 2024-04-02T04:50:47.919910+0000 mgr.y (mgr.14538) 289 : cluster 0 pgmap v289: 484 pgs: 128 active+undersized, 306 active+clean, 50 active+undersized+degraded; 600 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s; 352/2940 objects degraded (11.973%) 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: cluster 2024-04-02T04:50:48.701024+0000 mon.a (mon.0) 1553 : cluster 0 osdmap e209: 8 total, 7 up, 8 in 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: audit 2024-04-02T04:50:48.710540+0000 mon.c (mon.2) 271 : audit 1 from='client.? 172.21.15.69:0/3930140329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: audit 2024-04-02T04:50:48.714261+0000 mon.a (mon.0) 1554 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: audit 2024-04-02T04:50:48.731260+0000 mon.c (mon.2) 272 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: audit 2024-04-02T04:50:49.623396+0000 mon.a (mon.0) 1555 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: audit 2024-04-02T04:50:49.624728+0000 mon.a (mon.0) 1556 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: audit 2024-04-02T04:50:49.625084+0000 mon.a (mon.0) 1557 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:49.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:49 smithi069 bash[28904]: cluster 2024-04-02T04:50:49.661290+0000 mon.a (mon.0) 1558 : cluster 3 Health check update: Degraded data redundancy: 352/2940 objects degraded (11.973%), 50 pgs degraded, 108 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:50.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:50.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- 172.21.15.69:0/2060555191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06ab80 msgr2=0x7f79ac106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- 172.21.15.69:0/2060555191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06ab80 0x7f79ac106ea0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f79a00099c0 tx=0x7f79a0009cd0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- 172.21.15.69:0/2060555191 shutdown_connections 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- 172.21.15.69:0/2060555191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- 172.21.15.69:0/2060555191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06a1d0 0x7f79ac06a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- 172.21.15.69:0/2060555191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06ab80 0x7f79ac106ea0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- 172.21.15.69:0/2060555191 >> 172.21.15.69:0/2060555191 conn(0x7f79ac0771e0 msgr2=0x7f79ac0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- 172.21.15.69:0/2060555191 shutdown_connections 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- 172.21.15.69:0/2060555191 wait complete. 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 librados: starting msgr at 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 librados: starting objecter 2024-04-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 Processor -- start 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- start start 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 librados: setting wanted keys 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 librados: calling monclient init 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06a1d0 0x7f79ac101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06ab80 0x7f79ac1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac111a20 con 0x7f79ac06a1d0 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac1118a0 con 0x7f79ac06ab80 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b3003700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79ac111ba0 con 0x7f79ac107580 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b0d9f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06a1d0 0x7f79ac101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34648/0 (socket says 172.21.15.69:34648) 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 -- 172.21.15.69:0/680113056 learned_addr learned my addr 172.21.15.69:0/680113056 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79abfff700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06ab80 0x7f79ac1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 -- 172.21.15.69:0/680113056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06ab80 msgr2=0x7f79ac1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06ab80 0x7f79ac1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 -- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06a1d0 msgr2=0x7f79ac101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06a1d0 0x7f79ac101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b15a0700 1 -- 172.21.15.69:0/680113056 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac1a64e0 con 0x7f79ac107580 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.156+0000 7f79b0d9f700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06a1d0 0x7f79ac101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b15a0700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac101cf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f79a400ef30 tx=0x7f79a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 10 client.?.objecter ms_handle_connect 0x7f79ac107580 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a40092e0 con 0x7f79ac107580 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac1a6770 con 0x7f79ac107580 2024-04-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f79a4004bb0 con 0x7f79ac107580 2024-04-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4004020 con 0x7f79ac107580 2024-04-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b3003700 10 client.25922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ac1a6c30 con 0x7f79ac107580 2024-04-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b3003700 1 librados: init done 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79ac04ff00 con 0x7f79ac107580 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f79a401b030 con 0x7f79ac107580 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f799406f5e0 0x7f7994071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b0d9f700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f799406f5e0 0x7f7994071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(210..210 src has 74..210) ==== 12562+0+0 (secure 0 0 0) 0x7f79a408efc0 con 0x7f79ac107580 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 10 client.25922.objecter ms_dispatch 0x7f79ac001550 osd_map(210..210 src has 74..210) 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 3 client.25922.objecter handle_osd_map got epochs [210,210] > 0 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 3 client.25922.objecter handle_osd_map decoding full epoch 210 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 20 client.25922.objecter dump_active .. 0 homeless 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79a9ffb700 10 client.25922.objecter ms_handle_connect 0x7f799406f5e0 2024-04-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.160+0000 7f79b0d9f700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f799406f5e0 0x7f7994071aa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f799c005950 tx=0x7f799c00a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:50:50.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.164+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f79a4022020 con 0x7f79ac107580 2024-04-02T04:50:50.175 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:50.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:50.304 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.380+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f79ac1a6e60 con 0x7f79ac107580 2024-04-02T04:50:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79a9ffb700 1 -- 172.21.15.69:0/680113056 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v210) ==== 81+0+83037 (secure 0 0 0) 0x7f79a4010040 con 0x7f79ac107580 2024-04-02T04:50:50.385 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 210, 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "created": "2024-04-02T04:33:43.348385+0000", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "modified": "2024-04-02T04:50:49.691226+0000", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "last_up_change": "2024-04-02T04:49:46.297901+0000", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "last_in_change": "2024-04-02T04:42:47.064987+0000", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_num": 5801984, 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_set": [ 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "noscrub", 2024-04-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi069.stdout: "pglog_hardlimit", 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "purged_snapdirs", 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_deletes", 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "sortbitwise" 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_version": 39, 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "full_ratio": 0.94999998807907104, 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_snapshot": "", 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_max": 195, 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd": 8, 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "require_min_compat_client": "luminous", 2024-04-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi069.stdout: "min_compat_client": "luminous", 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "require_osd_release": "squid", 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_crimson": false, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "pools": [ 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 1, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".mgr", 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:38:57.671714+0000", 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 1, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 1, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 1, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 1, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 1, 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "23", 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_max": 32, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_min": 1 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": {} 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 8, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 8, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.25, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 2, 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".rgw.root", 2024-04-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:52.801763+0000", 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "79", 2024-04-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 3, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.log", 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:54.772439+0000", 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "81", 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 4, 2024-04-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.control", 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:56.696111+0000", 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "83", 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.2400000095367432, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.2400000095367432, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 5, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "datapool", 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:57.254108+0000", 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 3, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 3, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 3, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 3, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 3, 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "89", 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 2, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 89, 2024-04-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd": {} 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.6700000762939453, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.6700000762939453, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 6, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.meta", 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:58.846023+0000", 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "85", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_bias": 4 2024-04-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 10, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:11.754451+0000", 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.4900000095367432, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.4900000095367432, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.190000057220459, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.190000057220459, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 13, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosList_smithi069-72185-1", 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:13.034188+0000", 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 40, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace.1804289383", 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.538674+0000", 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 41, 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test_handler_errorsmithi069-72850-1", 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.947398+0000", 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.445 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.446 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.447 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:50:50.448 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 43, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace2.846930886", 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:25.876339+0000", 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.449 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.450 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.451 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.452 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.453 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.454 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-6", 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:53.761868+0000", 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "194", 2024-04-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "194", 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "194", 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "194", 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 6, 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 193, 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [ 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: 165 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": 165, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": 165, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.458 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.459 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 165, 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-13", 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:50:25.448764+0000", 2024-04-02T04:50:50.460 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8201, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "195", 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "194", 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "194", 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "194", 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.462 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 6, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 193, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": 102, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "writeback", 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.463 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.464 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 188, 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ReusePurgedSnapsmithi069-73023-11", 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:50:43.600663+0000", 2024-04-02T04:50:50.466 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.467 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "207", 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.468 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 3, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 207, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.469 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 193, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "RoundTripWriteSamePP_smithi069-72153-18", 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:50:48.167563+0000", 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "210", 2024-04-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 194, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosChecksum/1_smithi069-72304-30", 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:50:49.201772+0000", 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "210", 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 195, 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "OperateMtime_smithi069-72145-20", 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:50:49.617962+0000", 2024-04-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "210", 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b", 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 8, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6802", 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6803", 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6804", 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6805", 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6808", 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6809", 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6806", 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6807", 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6803/1087500595", 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6805/1087500595", 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6809/1087500595", 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6807/1087500595", 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a", 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 13, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6810", 2024-04-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6811", 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6812", 2024-04-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6813", 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6816", 2024-04-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6817", 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6814", 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6815", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6811/2651666015", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6813/2651666015", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6817/2651666015", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6815/2651666015", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7", 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 19, 2024-04-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6818", 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6819", 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6820", 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6821", 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6824", 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6825", 2024-04-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6822", 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6823", 2024-04-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6819/619998764", 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6821/619998764", 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6825/619998764", 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6823/619998764", 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed", 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 29, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6826", 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6827", 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6828", 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6829", 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6832", 2024-04-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6833", 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6830", 2024-04-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6831", 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6827/207593208", 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6829/207593208", 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6833/207593208", 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6831/207593208", 2024-04-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852", 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 36, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6800", 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.515 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6801", 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6802", 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6803", 2024-04-02T04:50:50.516 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6806", 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6807", 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.517 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6804", 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6805", 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6801/2065947829", 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6803/2065947829", 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6807/2065947829", 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6805/2065947829", 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe", 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 47, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6808", 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6809", 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6810", 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6811", 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6814", 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6815", 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6812", 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6813", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6809/219578185", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6811/219578185", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6815/219578185", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6813/219578185", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727", 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 58, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 209, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6816", 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6817", 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6818", 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6819", 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6822", 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6823", 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6820", 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6821", 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6817/1226172499", 2024-04-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6819/1226172499", 2024-04-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6823/1226172499", 2024-04-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6821/1226172499", 2024-04-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3", 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 0, 2024-04-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 69, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 151, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 152, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6824", 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6825", 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6826", 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6827", 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6830", 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6831", 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6828", 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6829", 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6825/148009079", 2024-04-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6827/148009079", 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6831/148009079", 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6829/148009079", 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "exists" 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_xinfo": [ 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:40.067963+0000", 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:02.984661+0000", 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:49:41.743901+0000", 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:49:43.718549+0000", 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:06.431911+0000", 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:05.075525+0000", 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:49:37.025718+0000", 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:49:15.075904+0000", 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 150 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap": [], 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_items": [ 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "2.9", 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 7, 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.6", 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 5 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.f", 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.17", 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_primaries": [], 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_temp": [], 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_temp": [], 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklist": { 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2738867930": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1798147282": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/52352690": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2928651991": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2763215770": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/3657409442": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/879855771": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1025808168": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1168952113": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1892956550": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/54578327": "2024-04-03T04:44:51.699278+0000" 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: "range_blocklist": {}, 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profiles": { 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "default": { 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van" 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16": { 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-device-class": "", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-num-failure-domains": "0", 2024-04-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-osds-per-failure-domain": "0", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-root": "default", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van", 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "w": "8" 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps_queue": [ 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 17, 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 36, 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 38, 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 66, 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 101, 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 103, 2024-04-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 112, 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 9 2024-04-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 120, 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 137, 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 145, 2024-04-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 146, 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 154, 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 164, 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 3 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 173, 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 176, 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 188, 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 189, 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "new_removed_snaps": [], 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "new_purged_snaps": [], 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_node_flags": {}, 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "device_class_flags": {}, 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode": { 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_enabled": false, 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_bucket_count": 0, 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "degraded_stretch_mode": 0, 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "recovering_stretch_mode": 0, 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_bucket": 0 2024-04-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:50:50.563 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:50:50.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:50.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 10 librados: watch_flush enter 2024-04-02T04:50:50.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 10 librados: watch_flush exit 2024-04-02T04:50:50.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 20 client.25922.objecter shutdown clearing up homeless session... 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 10 client.25922.objecter successfully canceled tick 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f799406f5e0 msgr2=0x7f7994071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f799406f5e0 0x7f7994071aa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f799c005950 tx=0x7f799c00a300 comp rx=0 tx=0).stop 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 msgr2=0x7f79ac101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac101cf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f79a400ef30 tx=0x7f79a400c3a0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 shutdown_connections 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f799406f5e0 0x7f7994071aa0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79ac107580 0x7f79ac101cf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac06a1d0 0x7f79ac101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 --2- 172.21.15.69:0/680113056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ac06ab80 0x7f79ac1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 >> 172.21.15.69:0/680113056 conn(0x7f79ac0771e0 msgr2=0x7f79ac0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 shutdown_connections 2024-04-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 -- 172.21.15.69:0/680113056 wait complete. 2024-04-02T04:50:50.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:50:50.384+0000 7f79b3003700 1 librados: shutdown 2024-04-02T04:50:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:50.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:50.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:50.855 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:50.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:50.984 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:50.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:51.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:51.107 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:51.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:50:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:51.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.694515+0000 mon.a (mon.0) 1559 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.694796+0000 mon.a (mon.0) 1560 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:51.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: cluster 2024-04-02T04:50:49.700785+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e210: 8 total, 7 up, 8 in 2024-04-02T04:50:51.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.701839+0000 mon.a (mon.0) 1562 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-72389-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:51.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.707354+0000 mon.c (mon.2) 273 : audit 1 from='client.? 172.21.15.69:0/2960985830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.710085+0000 mon.b (mon.1) 161 : audit 1 from='client.? 172.21.15.69:0/422631764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.732638+0000 mon.c (mon.2) 274 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.749848+0000 mon.a (mon.0) 1563 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:49.750658+0000 mon.a (mon.0) 1564 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: cluster 2024-04-02T04:50:49.920706+0000 mgr.y (mgr.14538) 290 : cluster 0 pgmap v292: 484 pgs: 96 unknown, 90 active+undersized, 250 active+clean, 48 active+undersized+degraded; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2931 objects degraded (11.941%) 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:50.384559+0000 mon.c (mon.2) 275 : audit 0 from='client.? 172.21.15.69:0/680113056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:50.699447+0000 mon.a (mon.0) 1565 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:50.699781+0000 mon.a (mon.0) 1566 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[28904]: audit 2024-04-02T04:50:50.733617+0000 mon.c (mon.2) 276 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.694515+0000 mon.a (mon.0) 1559 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.694796+0000 mon.a (mon.0) 1560 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: cluster 2024-04-02T04:50:49.700785+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e210: 8 total, 7 up, 8 in 2024-04-02T04:50:51.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.701839+0000 mon.a (mon.0) 1562 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-72389-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.707354+0000 mon.c (mon.2) 273 : audit 1 from='client.? 172.21.15.69:0/2960985830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.710085+0000 mon.b (mon.1) 161 : audit 1 from='client.? 172.21.15.69:0/422631764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.732638+0000 mon.c (mon.2) 274 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.749848+0000 mon.a (mon.0) 1563 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:49.750658+0000 mon.a (mon.0) 1564 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: cluster 2024-04-02T04:50:49.920706+0000 mgr.y (mgr.14538) 290 : cluster 0 pgmap v292: 484 pgs: 96 unknown, 90 active+undersized, 250 active+clean, 48 active+undersized+degraded; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2931 objects degraded (11.941%) 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:50.384559+0000 mon.c (mon.2) 275 : audit 0 from='client.? 172.21.15.69:0/680113056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:50.699447+0000 mon.a (mon.0) 1565 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:50.699781+0000 mon.a (mon.0) 1566 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:51 smithi069 bash[21528]: audit 2024-04-02T04:50:50.733617+0000 mon.c (mon.2) 276 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:51.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:51.354 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:51.394 INFO:tasks.thrashosds.thrasher:j is {'epoch': 210, 'fsid': 'f978400e-f0a9-11ee-b647-cb9ed24678a4', 'created': '2024-04-02T04:33:43.348385+0000', 'modified': '2024-04-02T04:50:49.691226+0000', 'last_up_change': '2024-04-02T04:49:46.297901+0000', 'last_in_change': '2024-04-02T04:42:47.064987+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 39, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 195, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-02T04:43:52.801763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-02T04:43:54.772439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-02T04:43:56.696111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-04-02T04:43:57.254108+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-02T04:43:58.846023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-02T04:49:11.754451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi069-72185-1', 'create_time': '2024-04-02T04:49:13.034188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 40, 'pool_name': 'poolrace.1804289383', 'create_time': '2024-04-02T04:49:24.538674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 41, 'pool_name': 'test_handler_errorsmithi069-72850-1', 'create_time': '2024-04-02T04:49:24.947398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 43, 'pool_name': 'poolrace2.846930886', 'create_time': '2024-04-02T04:49:25.876339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 102, 'pool_name': 'test-rados-api-smithi069-72349-6', 'create_time': '2024-04-02T04:49:53.761868+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '194', 'last_force_op_resend_prenautilus': '194', 'last_force_op_resend_preluminous': '194', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 193, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [165], 'tier_of': -1, 'read_tier': 165, 'write_tier': 165, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 165, 'pool_name': 'test-rados-api-smithi069-72349-13', 'create_time': '2024-04-02T04:50:25.448764+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '194', 'last_force_op_resend_prenautilus': '194', 'last_force_op_resend_preluminous': '194', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 193, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 102, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 188, 'pool_name': 'ReusePurgedSnapsmithi069-73023-11', 'create_time': '2024-04-02T04:50:43.600663+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 207, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 193, 'pool_name': 'RoundTripWriteSamePP_smithi069-72153-18', 'create_time': '2024-04-02T04:50:48.167563+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 194, 'pool_name': 'LibRadosChecksum/1_smithi069-72304-30', 'create_time': '2024-04-02T04:50:49.201772+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 195, 'pool_name': 'OperateMtime_smithi069-72145-20', 'create_time': '2024-04-02T04:50:49.617962+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1087500595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1087500595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 1087500595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1087500595}]}, 'public_addr': '172.21.15.69:6803/1087500595', 'cluster_addr': '172.21.15.69:6805/1087500595', 'heartbeat_back_addr': '172.21.15.69:6809/1087500595', 'heartbeat_front_addr': '172.21.15.69:6807/1087500595', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'afb71182-d4f7-416d-ae44-90f28eab4b7a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6811', 'nonce': 2651666015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6812', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2651666015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6816', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6817', 'nonce': 2651666015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6815', 'nonce': 2651666015}]}, 'public_addr': '172.21.15.69:6811/2651666015', 'cluster_addr': '172.21.15.69:6813/2651666015', 'heartbeat_back_addr': '172.21.15.69:6817/2651666015', 'heartbeat_front_addr': '172.21.15.69:6815/2651666015', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6818', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6819', 'nonce': 619998764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6820', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6821', 'nonce': 619998764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6825', 'nonce': 619998764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6822', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 619998764}]}, 'public_addr': '172.21.15.69:6819/619998764', 'cluster_addr': '172.21.15.69:6821/619998764', 'heartbeat_back_addr': '172.21.15.69:6825/619998764', 'heartbeat_front_addr': '172.21.15.69:6823/619998764', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '20e4526d-2707-44d0-b13a-af36019c42ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6826', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 207593208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 207593208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 207593208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 207593208}]}, 'public_addr': '172.21.15.69:6827/207593208', 'cluster_addr': '172.21.15.69:6829/207593208', 'heartbeat_back_addr': '172.21.15.69:6833/207593208', 'heartbeat_front_addr': '172.21.15.69:6831/207593208', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 2065947829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 2065947829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 2065947829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 2065947829}]}, 'public_addr': '172.21.15.98:6801/2065947829', 'cluster_addr': '172.21.15.98:6803/2065947829', 'heartbeat_back_addr': '172.21.15.98:6807/2065947829', 'heartbeat_front_addr': '172.21.15.98:6805/2065947829', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '902326af-4a40-47c8-a3b8-d5d5536f90fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 219578185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 219578185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6814', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 219578185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6813', 'nonce': 219578185}]}, 'public_addr': '172.21.15.98:6809/219578185', 'cluster_addr': '172.21.15.98:6811/219578185', 'heartbeat_back_addr': '172.21.15.98:6815/219578185', 'heartbeat_front_addr': '172.21.15.98:6813/219578185', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 1226172499}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 1226172499}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 1226172499}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 1226172499}]}, 'public_addr': '172.21.15.98:6817/1226172499', 'cluster_addr': '172.21.15.98:6819/1226172499', 'heartbeat_back_addr': '172.21.15.98:6823/1226172499', 'heartbeat_front_addr': '172.21.15.98:6821/1226172499', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 151, 'down_at': 152, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 148009079}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 148009079}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 148009079}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 148009079}]}, 'public_addr': '172.21.15.98:6825/148009079', 'cluster_addr': '172.21.15.98:6827/148009079', 'heartbeat_back_addr': '172.21.15.98:6831/148009079', 'heartbeat_front_addr': '172.21.15.98:6829/148009079', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:40.067963+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:02.984661+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:49:41.743901+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:49:43.718549+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:06.431911+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:05.075525+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:49:37.025718+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:49:15.075904+0000', 'dead_epoch': 150}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.69:0/2738867930': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/1798147282': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/52352690': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/2928651991': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:6800/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/2763215770': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/3657409442': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/879855771': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1025808168': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1168952113': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/1892956550': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6801/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/54578327': '2024-04-03T04:44:51.699278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosSnapshotsECPP_smithi069-72389-16': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 36, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 38, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 66, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 101, 'snaps': [{'begin': 2, 'length': 4}, {'begin': 7, 'length': 2}]}, {'pool': 102, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 103, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 2, 'length': 9}, {'begin': 14, 'length': 1}]}, {'pool': 120, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 137, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 145, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 146, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 164, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 173, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-02T04:50:51.394 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.694515+0000 mon.a (mon.0) 1559 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-72153-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.694796+0000 mon.a (mon.0) 1560 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: cluster 2024-04-02T04:50:49.700785+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e210: 8 total, 7 up, 8 in 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.701839+0000 mon.a (mon.0) 1562 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-72389-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.707354+0000 mon.c (mon.2) 273 : audit 1 from='client.? 172.21.15.69:0/2960985830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.710085+0000 mon.b (mon.1) 161 : audit 1 from='client.? 172.21.15.69:0/422631764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.732638+0000 mon.c (mon.2) 274 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.749848+0000 mon.a (mon.0) 1563 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:49.750658+0000 mon.a (mon.0) 1564 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:51.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: cluster 2024-04-02T04:50:49.920706+0000 mgr.y (mgr.14538) 290 : cluster 0 pgmap v292: 484 pgs: 96 unknown, 90 active+undersized, 250 active+clean, 48 active+undersized+degraded; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2931 objects degraded (11.941%) 2024-04-02T04:50:51.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:50.384559+0000 mon.c (mon.2) 275 : audit 0 from='client.? 172.21.15.69:0/680113056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:50:51.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:50.699447+0000 mon.a (mon.0) 1565 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72145-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:50.699781+0000 mon.a (mon.0) 1566 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-72304-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:51.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:51 smithi098 bash[23790]: audit 2024-04-02T04:50:50.733617+0000 mon.c (mon.2) 276 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:51.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:51.494 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:51.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:51.597 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:51.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:51.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:51.889 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:51.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:51.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:52.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-02T04:50:52.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:52.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:52.271 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:52.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:52.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:52.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:52.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: cluster 2024-04-02T04:50:50.874238+0000 osd.3 (osd.3) 199 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: cluster 2024-04-02T04:50:50.874314+0000 osd.3 (osd.3) 200 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: cluster 2024-04-02T04:50:51.162384+0000 mon.a (mon.0) 1567 : cluster 0 osdmap e211: 8 total, 7 up, 8 in 2024-04-02T04:50:52.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: cluster 2024-04-02T04:50:51.170217+0000 mon.a (mon.0) 1568 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:52.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: audit 2024-04-02T04:50:51.703065+0000 mon.a (mon.0) 1569 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-72389-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:50:52.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: cluster 2024-04-02T04:50:51.719683+0000 mon.a (mon.0) 1570 : cluster 0 osdmap e212: 8 total, 7 up, 8 in 2024-04-02T04:50:52.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: audit 2024-04-02T04:50:51.734399+0000 mon.c (mon.2) 277 : audit 1 from='client.? 172.21.15.69:0/2010422105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:52.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: audit 2024-04-02T04:50:51.735494+0000 mon.c (mon.2) 278 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:52.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:52 smithi098 bash[23790]: audit 2024-04-02T04:50:51.736192+0000 mon.a (mon.0) 1571 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:52.763 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:52.794 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:52.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:52.922 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:52.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:53.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:53.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: cluster 2024-04-02T04:50:50.874238+0000 osd.3 (osd.3) 199 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:53.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: cluster 2024-04-02T04:50:50.874314+0000 osd.3 (osd.3) 200 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:53.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: cluster 2024-04-02T04:50:51.162384+0000 mon.a (mon.0) 1567 : cluster 0 osdmap e211: 8 total, 7 up, 8 in 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: cluster 2024-04-02T04:50:51.170217+0000 mon.a (mon.0) 1568 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: audit 2024-04-02T04:50:51.703065+0000 mon.a (mon.0) 1569 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-72389-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: cluster 2024-04-02T04:50:51.719683+0000 mon.a (mon.0) 1570 : cluster 0 osdmap e212: 8 total, 7 up, 8 in 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: audit 2024-04-02T04:50:51.734399+0000 mon.c (mon.2) 277 : audit 1 from='client.? 172.21.15.69:0/2010422105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: audit 2024-04-02T04:50:51.735494+0000 mon.c (mon.2) 278 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[21528]: audit 2024-04-02T04:50:51.736192+0000 mon.a (mon.0) 1571 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: cluster 2024-04-02T04:50:50.874238+0000 osd.3 (osd.3) 199 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: cluster 2024-04-02T04:50:50.874314+0000 osd.3 (osd.3) 200 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:53.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: cluster 2024-04-02T04:50:51.162384+0000 mon.a (mon.0) 1567 : cluster 0 osdmap e211: 8 total, 7 up, 8 in 2024-04-02T04:50:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: cluster 2024-04-02T04:50:51.170217+0000 mon.a (mon.0) 1568 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: audit 2024-04-02T04:50:51.703065+0000 mon.a (mon.0) 1569 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-72389-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:50:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: cluster 2024-04-02T04:50:51.719683+0000 mon.a (mon.0) 1570 : cluster 0 osdmap e212: 8 total, 7 up, 8 in 2024-04-02T04:50:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: audit 2024-04-02T04:50:51.734399+0000 mon.c (mon.2) 277 : audit 1 from='client.? 172.21.15.69:0/2010422105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: audit 2024-04-02T04:50:51.735494+0000 mon.c (mon.2) 278 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:53.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:52 smithi069 bash[28904]: audit 2024-04-02T04:50:51.736192+0000 mon.a (mon.0) 1571 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:53.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:53.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:50:53.181 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:53.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:53.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:53.434 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:53.537 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:53.565 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:53.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: cluster 2024-04-02T04:50:51.921461+0000 mgr.y (mgr.14538) 291 : cluster 0 pgmap v295: 428 pgs: 40 unknown, 90 active+undersized, 250 active+clean, 48 active+undersized+degraded; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 350/2931 objects degraded (11.941%) 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: audit 2024-04-02T04:50:52.709626+0000 mon.a (mon.0) 1572 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: cluster 2024-04-02T04:50:52.722731+0000 mon.a (mon.0) 1573 : cluster 0 osdmap e213: 8 total, 7 up, 8 in 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: audit 2024-04-02T04:50:52.731193+0000 mon.c (mon.2) 279 : audit 1 from='client.? 172.21.15.69:0/3578872779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: audit 2024-04-02T04:50:52.737763+0000 mon.a (mon.0) 1574 : audit 1 from='client.? 172.21.15.69:0/3488474326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-72145-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: audit 2024-04-02T04:50:52.739134+0000 mon.a (mon.0) 1575 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[21528]: audit 2024-04-02T04:50:52.741429+0000 mon.c (mon.2) 280 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:53.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: cluster 2024-04-02T04:50:51.921461+0000 mgr.y (mgr.14538) 291 : cluster 0 pgmap v295: 428 pgs: 40 unknown, 90 active+undersized, 250 active+clean, 48 active+undersized+degraded; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 350/2931 objects degraded (11.941%) 2024-04-02T04:50:53.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: audit 2024-04-02T04:50:52.709626+0000 mon.a (mon.0) 1572 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:53.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: cluster 2024-04-02T04:50:52.722731+0000 mon.a (mon.0) 1573 : cluster 0 osdmap e213: 8 total, 7 up, 8 in 2024-04-02T04:50:53.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: audit 2024-04-02T04:50:52.731193+0000 mon.c (mon.2) 279 : audit 1 from='client.? 172.21.15.69:0/3578872779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: audit 2024-04-02T04:50:52.737763+0000 mon.a (mon.0) 1574 : audit 1 from='client.? 172.21.15.69:0/3488474326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-72145-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: audit 2024-04-02T04:50:52.739134+0000 mon.a (mon.0) 1575 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:53 smithi069 bash[28904]: audit 2024-04-02T04:50:52.741429+0000 mon.c (mon.2) 280 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:53.694 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:53.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:53.818 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:53.891 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:53.894 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:53.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:53.940 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:53.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:53.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: cluster 2024-04-02T04:50:51.921461+0000 mgr.y (mgr.14538) 291 : cluster 0 pgmap v295: 428 pgs: 40 unknown, 90 active+undersized, 250 active+clean, 48 active+undersized+degraded; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 350/2931 objects degraded (11.941%) 2024-04-02T04:50:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: audit 2024-04-02T04:50:52.709626+0000 mon.a (mon.0) 1572 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-72153-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: cluster 2024-04-02T04:50:52.722731+0000 mon.a (mon.0) 1573 : cluster 0 osdmap e213: 8 total, 7 up, 8 in 2024-04-02T04:50:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: audit 2024-04-02T04:50:52.731193+0000 mon.c (mon.2) 279 : audit 1 from='client.? 172.21.15.69:0/3578872779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: audit 2024-04-02T04:50:52.737763+0000 mon.a (mon.0) 1574 : audit 1 from='client.? 172.21.15.69:0/3488474326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-72145-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: audit 2024-04-02T04:50:52.739134+0000 mon.a (mon.0) 1575 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:53.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:53 smithi098 bash[23790]: audit 2024-04-02T04:50:52.741429+0000 mon.c (mon.2) 280 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:54.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:54.067 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:54.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:54.191 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:54.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:54.321 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:54.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:54.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:54.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:54.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:54.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:54.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:54.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[21528]: audit 2024-04-02T04:50:53.060549+0000 mgr.y (mgr.14538) 292 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:54.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[21528]: audit 2024-04-02T04:50:53.712246+0000 mon.a (mon.0) 1576 : audit 1 from='client.? 172.21.15.69:0/3488474326' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-72145-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[21528]: audit 2024-04-02T04:50:53.712369+0000 mon.a (mon.0) 1577 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[21528]: cluster 2024-04-02T04:50:53.717837+0000 mon.a (mon.0) 1578 : cluster 0 osdmap e214: 8 total, 7 up, 8 in 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[21528]: audit 2024-04-02T04:50:53.744987+0000 mon.c (mon.2) 281 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[28904]: audit 2024-04-02T04:50:53.060549+0000 mgr.y (mgr.14538) 292 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[28904]: audit 2024-04-02T04:50:53.712246+0000 mon.a (mon.0) 1576 : audit 1 from='client.? 172.21.15.69:0/3488474326' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-72145-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[28904]: audit 2024-04-02T04:50:53.712369+0000 mon.a (mon.0) 1577 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[28904]: cluster 2024-04-02T04:50:53.717837+0000 mon.a (mon.0) 1578 : cluster 0 osdmap e214: 8 total, 7 up, 8 in 2024-04-02T04:50:54.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:54 smithi069 bash[28904]: audit 2024-04-02T04:50:53.744987+0000 mon.c (mon.2) 281 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:54.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:54.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:54 smithi098 bash[23790]: audit 2024-04-02T04:50:53.060549+0000 mgr.y (mgr.14538) 292 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:50:54.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:54 smithi098 bash[23790]: audit 2024-04-02T04:50:53.712246+0000 mon.a (mon.0) 1576 : audit 1 from='client.? 172.21.15.69:0/3488474326' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-72145-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:54.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:54 smithi098 bash[23790]: audit 2024-04-02T04:50:53.712369+0000 mon.a (mon.0) 1577 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-72304-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:54.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:54 smithi098 bash[23790]: cluster 2024-04-02T04:50:53.717837+0000 mon.a (mon.0) 1578 : cluster 0 osdmap e214: 8 total, 7 up, 8 in 2024-04-02T04:50:54.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:54 smithi098 bash[23790]: audit 2024-04-02T04:50:53.744987+0000 mon.c (mon.2) 281 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:54.981 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:54.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:54.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:55.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:55.108 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:55.211 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:55.230 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:55.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:55.357 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:50:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:50:55.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:55.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:55.618 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:55.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: cluster 2024-04-02T04:50:53.922251+0000 mgr.y (mgr.14538) 293 : cluster 0 pgmap v298: 460 pgs: 64 creating+peering, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2934 objects degraded (11.929%) 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: cluster 2024-04-02T04:50:54.662730+0000 mon.a (mon.0) 1579 : cluster 3 Health check update: Degraded data redundancy: 350/2931 objects degraded (11.941%), 48 pgs degraded, 108 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: cluster 2024-04-02T04:50:54.678591+0000 mon.a (mon.0) 1580 : cluster 0 osdmap e215: 8 total, 7 up, 8 in 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.695777+0000 mon.a (mon.0) 1581 : audit 1 from='client.? 172.21.15.69:0/2288117367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-72153-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.729118+0000 mon.b (mon.1) 162 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.733486+0000 mon.b (mon.1) 163 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.733495+0000 mon.a (mon.0) 1582 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.736649+0000 mon.b (mon.1) 164 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.738426+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.741167+0000 mon.a (mon.0) 1584 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:55 smithi098 bash[23790]: audit 2024-04-02T04:50:54.746318+0000 mon.c (mon.2) 282 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:55.748 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:55.851 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:55.876 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: cluster 2024-04-02T04:50:53.922251+0000 mgr.y (mgr.14538) 293 : cluster 0 pgmap v298: 460 pgs: 64 creating+peering, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2934 objects degraded (11.929%) 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: cluster 2024-04-02T04:50:54.662730+0000 mon.a (mon.0) 1579 : cluster 3 Health check update: Degraded data redundancy: 350/2931 objects degraded (11.941%), 48 pgs degraded, 108 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: cluster 2024-04-02T04:50:54.678591+0000 mon.a (mon.0) 1580 : cluster 0 osdmap e215: 8 total, 7 up, 8 in 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.695777+0000 mon.a (mon.0) 1581 : audit 1 from='client.? 172.21.15.69:0/2288117367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-72153-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.729118+0000 mon.b (mon.1) 162 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.733486+0000 mon.b (mon.1) 163 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.733495+0000 mon.a (mon.0) 1582 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.736649+0000 mon.b (mon.1) 164 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.738426+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.741167+0000 mon.a (mon.0) 1584 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:55.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[21528]: audit 2024-04-02T04:50:54.746318+0000 mon.c (mon.2) 282 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: cluster 2024-04-02T04:50:53.922251+0000 mgr.y (mgr.14538) 293 : cluster 0 pgmap v298: 460 pgs: 64 creating+peering, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2934 objects degraded (11.929%) 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: cluster 2024-04-02T04:50:54.662730+0000 mon.a (mon.0) 1579 : cluster 3 Health check update: Degraded data redundancy: 350/2931 objects degraded (11.941%), 48 pgs degraded, 108 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: cluster 2024-04-02T04:50:54.678591+0000 mon.a (mon.0) 1580 : cluster 0 osdmap e215: 8 total, 7 up, 8 in 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.695777+0000 mon.a (mon.0) 1581 : audit 1 from='client.? 172.21.15.69:0/2288117367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-72153-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.729118+0000 mon.b (mon.1) 162 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.733486+0000 mon.b (mon.1) 163 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.733495+0000 mon.a (mon.0) 1582 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.736649+0000 mon.b (mon.1) 164 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.738426+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:55.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.741167+0000 mon.a (mon.0) 1584 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:50:55.879 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:55 smithi069 bash[28904]: audit 2024-04-02T04:50:54.746318+0000 mon.c (mon.2) 282 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:55.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:55.929 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:50:55.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:56.002 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:56.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:56.129 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:56.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:56.259 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:56.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:56.384 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:56.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:56.514 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:56.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:56.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:56.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: audit 2024-04-02T04:50:55.670802+0000 mon.a (mon.0) 1585 : audit 1 from='client.? 172.21.15.69:0/2288117367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-72153-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: audit 2024-04-02T04:50:55.671030+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: audit 2024-04-02T04:50:55.678778+0000 mon.b (mon.1) 165 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: cluster 2024-04-02T04:50:55.682155+0000 mon.a (mon.0) 1587 : cluster 0 osdmap e216: 8 total, 7 up, 8 in 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: audit 2024-04-02T04:50:55.692716+0000 mon.a (mon.0) 1588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: audit 2024-04-02T04:50:55.698691+0000 mon.a (mon.0) 1589 : audit 1 from='client.? 172.21.15.69:0/1157155124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-72145-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:56.758 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:56 smithi098 bash[23790]: audit 2024-04-02T04:50:55.747431+0000 mon.c (mon.2) 283 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:50:56.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:56.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: audit 2024-04-02T04:50:55.670802+0000 mon.a (mon.0) 1585 : audit 1 from='client.? 172.21.15.69:0/2288117367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-72153-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:56.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: audit 2024-04-02T04:50:55.671030+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: audit 2024-04-02T04:50:55.678778+0000 mon.b (mon.1) 165 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: cluster 2024-04-02T04:50:55.682155+0000 mon.a (mon.0) 1587 : cluster 0 osdmap e216: 8 total, 7 up, 8 in 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: audit 2024-04-02T04:50:55.692716+0000 mon.a (mon.0) 1588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: audit 2024-04-02T04:50:55.698691+0000 mon.a (mon.0) 1589 : audit 1 from='client.? 172.21.15.69:0/1157155124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-72145-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[28904]: audit 2024-04-02T04:50:55.747431+0000 mon.c (mon.2) 283 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: audit 2024-04-02T04:50:55.670802+0000 mon.a (mon.0) 1585 : audit 1 from='client.? 172.21.15.69:0/2288117367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-72153-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: audit 2024-04-02T04:50:55.671030+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: audit 2024-04-02T04:50:55.678778+0000 mon.b (mon.1) 165 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: cluster 2024-04-02T04:50:55.682155+0000 mon.a (mon.0) 1587 : cluster 0 osdmap e216: 8 total, 7 up, 8 in 2024-04-02T04:50:56.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: audit 2024-04-02T04:50:55.692716+0000 mon.a (mon.0) 1588 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:56.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: audit 2024-04-02T04:50:55.698691+0000 mon.a (mon.0) 1589 : audit 1 from='client.? 172.21.15.69:0/1157155124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-72145-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:56.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:56 smithi069 bash[21528]: audit 2024-04-02T04:50:55.747431+0000 mon.c (mon.2) 283 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:56.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:56.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:57.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:57.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:57.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:57.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:57.295 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:57.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:57.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:57.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:57.681 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:57.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:57.804 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:57.828 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[21528]: cluster 2024-04-02T04:50:55.923179+0000 mgr.y (mgr.14538) 294 : cluster 0 pgmap v301: 460 pgs: 64 unknown, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2934 objects degraded (11.929%) 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[21528]: cluster 2024-04-02T04:50:56.670643+0000 mon.a (mon.0) 1590 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[21528]: audit 2024-04-02T04:50:56.680105+0000 mon.a (mon.0) 1591 : audit 1 from='client.? 172.21.15.69:0/1157155124' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-72145-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[21528]: audit 2024-04-02T04:50:56.748368+0000 mon.c (mon.2) 284 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[21528]: cluster 2024-04-02T04:50:57.132977+0000 mon.a (mon.0) 1592 : cluster 0 osdmap e217: 8 total, 7 up, 8 in 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[28904]: cluster 2024-04-02T04:50:55.923179+0000 mgr.y (mgr.14538) 294 : cluster 0 pgmap v301: 460 pgs: 64 unknown, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2934 objects degraded (11.929%) 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[28904]: cluster 2024-04-02T04:50:56.670643+0000 mon.a (mon.0) 1590 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[28904]: audit 2024-04-02T04:50:56.680105+0000 mon.a (mon.0) 1591 : audit 1 from='client.? 172.21.15.69:0/1157155124' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-72145-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[28904]: audit 2024-04-02T04:50:56.748368+0000 mon.c (mon.2) 284 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:57.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:57 smithi069 bash[28904]: cluster 2024-04-02T04:50:57.132977+0000 mon.a (mon.0) 1592 : cluster 0 osdmap e217: 8 total, 7 up, 8 in 2024-04-02T04:50:57.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:57.930 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:57.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:57.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:57 smithi098 bash[23790]: cluster 2024-04-02T04:50:55.923179+0000 mgr.y (mgr.14538) 294 : cluster 0 pgmap v301: 460 pgs: 64 unknown, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 350/2934 objects degraded (11.929%) 2024-04-02T04:50:57.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:57 smithi098 bash[23790]: cluster 2024-04-02T04:50:56.670643+0000 mon.a (mon.0) 1590 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:50:57.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:57 smithi098 bash[23790]: audit 2024-04-02T04:50:56.680105+0000 mon.a (mon.0) 1591 : audit 1 from='client.? 172.21.15.69:0/1157155124' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-72145-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:57.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:57 smithi098 bash[23790]: audit 2024-04-02T04:50:56.748368+0000 mon.c (mon.2) 284 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:57.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:57 smithi098 bash[23790]: cluster 2024-04-02T04:50:57.132977+0000 mon.a (mon.0) 1592 : cluster 0 osdmap e217: 8 total, 7 up, 8 in 2024-04-02T04:50:58.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:58.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:50:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:50:58.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:58.182 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:58.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:58.312 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:58.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:58.439 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:58.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:58.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:58.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: seed 72389 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-02T04:50:58.706 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (6355 ms) 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (5471 ms) 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (8453 ms) 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2909 ms) 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (5986 ms) 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (29175 ms total) 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-02T04:50:58.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (5237 ms) 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (5732 ms) 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (7985 ms) 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4080 ms) 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2595 ms) 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:463: Skipped 2024-04-02T04:50:58.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi069-72389-7 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18320 ms) 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (43949 ms total) 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-02T04:50:58.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5053 ms) 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5061 ms total) 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2956 ms) 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2445 ms) 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-02T04:50:58.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1575 ms) 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: audit 2024-04-02T04:50:57.691319+0000 mon.a (mon.0) 1593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: cluster 2024-04-02T04:50:57.695322+0000 mon.a (mon.0) 1594 : cluster 0 osdmap e218: 8 total, 7 up, 8 in 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: audit 2024-04-02T04:50:57.700754+0000 mon.c (mon.2) 285 : audit 1 from='client.? 172.21.15.69:0/1528794956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: audit 2024-04-02T04:50:57.701517+0000 mon.a (mon.0) 1595 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: audit 2024-04-02T04:50:57.749266+0000 mon.c (mon.2) 286 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: cluster 2024-04-02T04:50:58.331414+0000 osd.6 (osd.6) 129 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:58 smithi098 bash[23790]: cluster 2024-04-02T04:50:58.331549+0000 osd.6 (osd.6) 130 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:58.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:50:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:50:58.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: audit 2024-04-02T04:50:57.691319+0000 mon.a (mon.0) 1593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:50:58.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: cluster 2024-04-02T04:50:57.695322+0000 mon.a (mon.0) 1594 : cluster 0 osdmap e218: 8 total, 7 up, 8 in 2024-04-02T04:50:58.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: audit 2024-04-02T04:50:57.700754+0000 mon.c (mon.2) 285 : audit 1 from='client.? 172.21.15.69:0/1528794956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:58.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: audit 2024-04-02T04:50:57.701517+0000 mon.a (mon.0) 1595 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:58.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: audit 2024-04-02T04:50:57.749266+0000 mon.c (mon.2) 286 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: cluster 2024-04-02T04:50:58.331414+0000 osd.6 (osd.6) 129 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[21528]: cluster 2024-04-02T04:50:58.331549+0000 osd.6 (osd.6) 130 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: audit 2024-04-02T04:50:57.691319+0000 mon.a (mon.0) 1593 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-72304-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: cluster 2024-04-02T04:50:57.695322+0000 mon.a (mon.0) 1594 : cluster 0 osdmap e218: 8 total, 7 up, 8 in 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: audit 2024-04-02T04:50:57.700754+0000 mon.c (mon.2) 285 : audit 1 from='client.? 172.21.15.69:0/1528794956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: audit 2024-04-02T04:50:57.701517+0000 mon.a (mon.0) 1595 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: audit 2024-04-02T04:50:57.749266+0000 mon.c (mon.2) 286 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: cluster 2024-04-02T04:50:58.331414+0000 osd.6 (osd.6) 129 : cluster 0 purged_snaps scrub starts 2024-04-02T04:50:58.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:58 smithi069 bash[28904]: cluster 2024-04-02T04:50:58.331549+0000 osd.6 (osd.6) 130 : cluster 0 purged_snaps scrub ok 2024-04-02T04:50:58.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:58.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:58.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:58.933 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:58.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:59.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:59.064 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:50:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:50:59.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:50:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:50:59.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:59.324 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:59.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:59.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:59.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:59.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:50:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:50:59.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:50:59.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: cluster 2024-04-02T04:50:57.926782+0000 mgr.y (mgr.14538) 295 : cluster 0 pgmap v304: 436 pgs: 40 creating+peering, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 350/2943 objects degraded (11.893%) 2024-04-02T04:50:59.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: audit 2024-04-02T04:50:58.696046+0000 mon.a (mon.0) 1596 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:59.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: cluster 2024-04-02T04:50:58.706467+0000 mon.a (mon.0) 1597 : cluster 0 osdmap e219: 8 total, 7 up, 8 in 2024-04-02T04:50:59.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: audit 2024-04-02T04:50:58.712291+0000 mon.a (mon.0) 1598 : audit 1 from='client.? 172.21.15.69:0/3757114845' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-72145-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:59.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: audit 2024-04-02T04:50:58.750249+0000 mon.c (mon.2) 287 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:59.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: cluster 2024-04-02T04:50:59.665228+0000 mon.a (mon.0) 1599 : cluster 3 Health check update: Degraded data redundancy: 350/2943 objects degraded (11.893%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:59.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: audit 2024-04-02T04:50:59.699201+0000 mon.a (mon.0) 1600 : audit 1 from='client.? 172.21.15.69:0/3757114845' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-72145-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:59.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: cluster 2024-04-02T04:50:59.705535+0000 mon.a (mon.0) 1601 : cluster 0 osdmap e220: 8 total, 7 up, 8 in 2024-04-02T04:50:59.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: audit 2024-04-02T04:50:59.708462+0000 mon.b (mon.1) 166 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:59.851 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:59.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:50:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:50:59.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:59.983 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[28904]: audit 2024-04-02T04:50:59.713132+0000 mon.a (mon.0) 1602 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:59.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: cluster 2024-04-02T04:50:57.926782+0000 mgr.y (mgr.14538) 295 : cluster 0 pgmap v304: 436 pgs: 40 creating+peering, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 350/2943 objects degraded (11.893%) 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: audit 2024-04-02T04:50:58.696046+0000 mon.a (mon.0) 1596 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: cluster 2024-04-02T04:50:58.706467+0000 mon.a (mon.0) 1597 : cluster 0 osdmap e219: 8 total, 7 up, 8 in 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: audit 2024-04-02T04:50:58.712291+0000 mon.a (mon.0) 1598 : audit 1 from='client.? 172.21.15.69:0/3757114845' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-72145-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: audit 2024-04-02T04:50:58.750249+0000 mon.c (mon.2) 287 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: cluster 2024-04-02T04:50:59.665228+0000 mon.a (mon.0) 1599 : cluster 3 Health check update: Degraded data redundancy: 350/2943 objects degraded (11.893%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: audit 2024-04-02T04:50:59.699201+0000 mon.a (mon.0) 1600 : audit 1 from='client.? 172.21.15.69:0/3757114845' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-72145-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: cluster 2024-04-02T04:50:59.705535+0000 mon.a (mon.0) 1601 : cluster 0 osdmap e220: 8 total, 7 up, 8 in 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: audit 2024-04-02T04:50:59.708462+0000 mon.b (mon.1) 166 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:59.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:50:59 smithi069 bash[21528]: audit 2024-04-02T04:50:59.713132+0000 mon.a (mon.0) 1602 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:50:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:50:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:50:59.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:00.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:00.124 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:00.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: cluster 2024-04-02T04:50:57.926782+0000 mgr.y (mgr.14538) 295 : cluster 0 pgmap v304: 436 pgs: 40 creating+peering, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 350/2943 objects degraded (11.893%) 2024-04-02T04:51:00.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: audit 2024-04-02T04:50:58.696046+0000 mon.a (mon.0) 1596 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-72153-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: cluster 2024-04-02T04:50:58.706467+0000 mon.a (mon.0) 1597 : cluster 0 osdmap e219: 8 total, 7 up, 8 in 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: audit 2024-04-02T04:50:58.712291+0000 mon.a (mon.0) 1598 : audit 1 from='client.? 172.21.15.69:0/3757114845' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-72145-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: audit 2024-04-02T04:50:58.750249+0000 mon.c (mon.2) 287 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: cluster 2024-04-02T04:50:59.665228+0000 mon.a (mon.0) 1599 : cluster 3 Health check update: Degraded data redundancy: 350/2943 objects degraded (11.893%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: audit 2024-04-02T04:50:59.699201+0000 mon.a (mon.0) 1600 : audit 1 from='client.? 172.21.15.69:0/3757114845' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-72145-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: cluster 2024-04-02T04:50:59.705535+0000 mon.a (mon.0) 1601 : cluster 0 osdmap e220: 8 total, 7 up, 8 in 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: audit 2024-04-02T04:50:59.708462+0000 mon.b (mon.1) 166 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:00.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:50:59 smithi098 bash[23790]: audit 2024-04-02T04:50:59.713132+0000 mon.a (mon.0) 1602 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:00.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:00.256 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:00.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:00.383 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:00.486 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:00.510 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:00.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:00.633 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:00.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:00.748 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:51:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:51:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:51:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:51:00.748 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:51:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:00.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:00.896 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:00.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: audit 2024-04-02T04:50:59.751707+0000 mon.c (mon.2) 288 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: cluster 2024-04-02T04:50:59.928578+0000 mgr.y (mgr.14538) 296 : cluster 0 pgmap v307: 428 pgs: 32 unknown, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 350/2943 objects degraded (11.893%) 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: audit 2024-04-02T04:51:00.703500+0000 mon.a (mon.0) 1603 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: cluster 2024-04-02T04:51:00.714828+0000 mon.a (mon.0) 1604 : cluster 0 osdmap e221: 8 total, 7 up, 8 in 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: audit 2024-04-02T04:51:00.716045+0000 mon.b (mon.1) 167 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: audit 2024-04-02T04:51:00.735041+0000 mon.a (mon.0) 1605 : audit 1 from='client.? 172.21.15.69:0/942008359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72153-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:00.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:00 smithi098 bash[23790]: audit 2024-04-02T04:51:00.736467+0000 mon.a (mon.0) 1606 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:01.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:01.022 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:01.060 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: audit 2024-04-02T04:50:59.751707+0000 mon.c (mon.2) 288 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: cluster 2024-04-02T04:50:59.928578+0000 mgr.y (mgr.14538) 296 : cluster 0 pgmap v307: 428 pgs: 32 unknown, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 350/2943 objects degraded (11.893%) 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: audit 2024-04-02T04:51:00.703500+0000 mon.a (mon.0) 1603 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: cluster 2024-04-02T04:51:00.714828+0000 mon.a (mon.0) 1604 : cluster 0 osdmap e221: 8 total, 7 up, 8 in 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: audit 2024-04-02T04:51:00.716045+0000 mon.b (mon.1) 167 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: audit 2024-04-02T04:51:00.735041+0000 mon.a (mon.0) 1605 : audit 1 from='client.? 172.21.15.69:0/942008359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72153-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[21528]: audit 2024-04-02T04:51:00.736467+0000 mon.a (mon.0) 1606 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:01.061 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: audit 2024-04-02T04:50:59.751707+0000 mon.c (mon.2) 288 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.062 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: cluster 2024-04-02T04:50:59.928578+0000 mgr.y (mgr.14538) 296 : cluster 0 pgmap v307: 428 pgs: 32 unknown, 93 active+undersized, 255 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 350/2943 objects degraded (11.893%) 2024-04-02T04:51:01.062 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: audit 2024-04-02T04:51:00.703500+0000 mon.a (mon.0) 1603 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:51:01.062 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: cluster 2024-04-02T04:51:00.714828+0000 mon.a (mon.0) 1604 : cluster 0 osdmap e221: 8 total, 7 up, 8 in 2024-04-02T04:51:01.062 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: audit 2024-04-02T04:51:00.716045+0000 mon.b (mon.1) 167 : audit 1 from='client.? 172.21.15.69:0/2157606784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:01.062 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: audit 2024-04-02T04:51:00.735041+0000 mon.a (mon.0) 1605 : audit 1 from='client.? 172.21.15.69:0/942008359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72153-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.062 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:00 smithi069 bash[28904]: audit 2024-04-02T04:51:00.736467+0000 mon.a (mon.0) 1606 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]: dispatch 2024-04-02T04:51:01.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:01.150 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:01.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:01.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:01.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:01.536 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:01.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-02T04:51:01.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:01.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pPP (68298 ms total) 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (535 ms) 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (535 ms total) 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (68 ms) 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (11 ms) 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (79 ms total) 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:51:01.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (496 ms) 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (501 ms total) 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (52 ms) 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (57 ms total) 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:51:01.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1061 ms) 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1061 ms total) 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (111670 ms total) 2024-04-02T04:51:01.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: Running main() from gmock_main.cc 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] Global test environment set-up. 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (24704 ms) 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (12292 ms) 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-02T04:51:01.727 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (6461 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (7013 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (9564 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (60034 ms total) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (7997 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5585 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8424 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5952 ms) 2024-04-02T04:51:01.728 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4357 ms) 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi069-73023-11. 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: Waiting for snaps to purge. 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19010 ms) 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (51325 ms total) 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] Global test environment tear-down 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (111359 ms total) 2024-04-02T04:51:01.729 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-02T04:51:01.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:01.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:01.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:01.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:00.753561+0000 mon.c (mon.2) 289 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:01.706059+0000 mon.a (mon.0) 1607 : audit 1 from='client.? 172.21.15.69:0/942008359' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72153-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:01.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:01.706189+0000 mon.a (mon.0) 1608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:51:01.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: cluster 2024-04-02T04:51:01.715316+0000 mon.a (mon.0) 1609 : cluster 0 osdmap e222: 8 total, 7 up, 8 in 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:01.719993+0000 mon.c (mon.2) 290 : audit 1 from='client.? 172.21.15.69:0/1867612821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:01.722548+0000 mon.a (mon.0) 1610 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:01.724219+0000 mon.a (mon.0) 1611 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[28904]: audit 2024-04-02T04:51:01.754786+0000 mon.c (mon.2) 291 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:00.753561+0000 mon.c (mon.2) 289 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:01.706059+0000 mon.a (mon.0) 1607 : audit 1 from='client.? 172.21.15.69:0/942008359' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72153-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:01.706189+0000 mon.a (mon.0) 1608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: cluster 2024-04-02T04:51:01.715316+0000 mon.a (mon.0) 1609 : cluster 0 osdmap e222: 8 total, 7 up, 8 in 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:01.719993+0000 mon.c (mon.2) 290 : audit 1 from='client.? 172.21.15.69:0/1867612821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:01.722548+0000 mon.a (mon.0) 1610 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:51:01.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:01.724219+0000 mon.a (mon.0) 1611 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.914 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:01 smithi069 bash[21528]: audit 2024-04-02T04:51:01.754786+0000 mon.c (mon.2) 291 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.926 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:01.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:01.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:00.753561+0000 mon.c (mon.2) 289 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:01.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:01.706059+0000 mon.a (mon.0) 1607 : audit 1 from='client.? 172.21.15.69:0/942008359' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-72153-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:01.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:01.706189+0000 mon.a (mon.0) 1608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-72304-36"}]': finished 2024-04-02T04:51:01.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: cluster 2024-04-02T04:51:01.715316+0000 mon.a (mon.0) 1609 : cluster 0 osdmap e222: 8 total, 7 up, 8 in 2024-04-02T04:51:01.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:01.719993+0000 mon.c (mon.2) 290 : audit 1 from='client.? 172.21.15.69:0/1867612821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:01.722548+0000 mon.a (mon.0) 1610 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:51:01.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:01.724219+0000 mon.a (mon.0) 1611 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:01.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:01 smithi098 bash[23790]: audit 2024-04-02T04:51:01.754786+0000 mon.c (mon.2) 291 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:02.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:02.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:02.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.316 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:02.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:02.446 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:02.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.576 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:02.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.705 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:02.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.834 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:02.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:02.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:02.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:03.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:03.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:03.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:03.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:03.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:03.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: cluster 2024-04-02T04:51:01.929614+0000 mgr.y (mgr.14538) 297 : cluster 0 pgmap v310: 420 pgs: 64 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: audit 2024-04-02T04:51:02.017445+0000 mon.b (mon.1) 168 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: audit 2024-04-02T04:51:02.708636+0000 mon.a (mon.0) 1612 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: audit 2024-04-02T04:51:02.708757+0000 mon.a (mon.0) 1613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: cluster 2024-04-02T04:51:02.713319+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e223: 8 total, 7 up, 8 in 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: audit 2024-04-02T04:51:02.713819+0000 mon.a (mon.0) 1615 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[28904]: audit 2024-04-02T04:51:02.755774+0000 mon.c (mon.2) 292 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: cluster 2024-04-02T04:51:01.929614+0000 mgr.y (mgr.14538) 297 : cluster 0 pgmap v310: 420 pgs: 64 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: audit 2024-04-02T04:51:02.017445+0000 mon.b (mon.1) 168 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:03.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: audit 2024-04-02T04:51:02.708636+0000 mon.a (mon.0) 1612 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:51:03.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: audit 2024-04-02T04:51:02.708757+0000 mon.a (mon.0) 1613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:03.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: cluster 2024-04-02T04:51:02.713319+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e223: 8 total, 7 up, 8 in 2024-04-02T04:51:03.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: audit 2024-04-02T04:51:02.713819+0000 mon.a (mon.0) 1615 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:51:03.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:03 smithi069 bash[21528]: audit 2024-04-02T04:51:02.755774+0000 mon.c (mon.2) 292 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:03.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:03.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:51:03.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: cluster 2024-04-02T04:51:01.929614+0000 mgr.y (mgr.14538) 297 : cluster 0 pgmap v310: 420 pgs: 64 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:03.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: audit 2024-04-02T04:51:02.017445+0000 mon.b (mon.1) 168 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:03.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: audit 2024-04-02T04:51:02.708636+0000 mon.a (mon.0) 1612 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:51:03.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: audit 2024-04-02T04:51:02.708757+0000 mon.a (mon.0) 1613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-72145-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:03.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: cluster 2024-04-02T04:51:02.713319+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e223: 8 total, 7 up, 8 in 2024-04-02T04:51:03.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: audit 2024-04-02T04:51:02.713819+0000 mon.a (mon.0) 1615 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]: dispatch 2024-04-02T04:51:03.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:03 smithi098 bash[23790]: audit 2024-04-02T04:51:02.755774+0000 mon.c (mon.2) 292 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:03.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:03.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:03.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:03.862 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/1194127968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec107220 msgr2=0x7ff5ec107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/1194127968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec107220 0x7ff5ec107600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80098f0 tx=0x7ff5e8009c00 comp rx=0 tx=0).stop 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/1194127968 shutdown_connections 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/1194127968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 0x7ff5ec10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/1194127968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec107220 0x7ff5ec107600 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/1194127968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec1011a0 0x7ff5ec101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/1194127968 >> 172.21.15.69:0/1194127968 conn(0x7ff5ec0771e0 msgr2=0x7ff5ec0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/1194127968 shutdown_connections 2024-04-02T04:51:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/1194127968 wait complete. 2024-04-02T04:51:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 librados: starting msgr at 2024-04-02T04:51:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 librados: starting objecter 2024-04-02T04:51:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 Processor -- start 2024-04-02T04:51:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- start start 2024-04-02T04:51:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 librados: setting wanted keys 2024-04-02T04:51:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 librados: calling monclient init 2024-04-02T04:51:03.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec1011a0 0x7ff5ec196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 0x7ff5ec197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec107220 0x7ff5ec19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec110910 con 0x7ff5ec101b60 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec110790 con 0x7ff5ec107220 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.964+0000 7ff5f3ba0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff5ec110a90 con 0x7ff5ec1011a0 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f193c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec1011a0 0x7ff5ec196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f193c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec1011a0 0x7ff5ec196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44842/0 (socket says 172.21.15.69:44842) 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f193c700 1 -- 172.21.15.69:0/482569044 learned_addr learned my addr 172.21.15.69:0/482569044 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 0x7ff5ec197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f213d700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec107220 0x7ff5ec19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 -- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec1011a0 msgr2=0x7ff5ec196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec1011a0 0x7ff5ec196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 -- 172.21.15.69:0/482569044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec107220 msgr2=0x7ff5ec19b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec107220 0x7ff5ec19b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 -- 172.21.15.69:0/482569044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec19bf30 con 0x7ff5ec101b60 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f213d700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec107220 0x7ff5ec19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5f113b700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 0x7ff5ec197460 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00c970 tx=0x7ff5dc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5e2ffd700 10 client.?.objecter ms_handle_connect 0x7ff5ec101b60 2024-04-02T04:51:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:03.968+0000 7ff5e2ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:03.990 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:03.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:04.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:04.121 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.188+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0078f0 con 0x7ff5ec101b60 2024-04-02T04:51:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.188+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ec19c220 con 0x7ff5ec101b60 2024-04-02T04:51:04.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.188+0000 7ff5f3ba0700 10 client.15918.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.188+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ec1a3aa0 con 0x7ff5ec101b60 2024-04-02T04:51:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.188+0000 7ff5f3ba0700 1 librados: init done 2024-04-02T04:51:04.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.188+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ec04ff00 con 0x7ff5ec101b60 2024-04-02T04:51:04.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.196+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff5dc018400 con 0x7ff5ec101b60 2024-04-02T04:51:04.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.196+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc021650 con 0x7ff5ec101b60 2024-04-02T04:51:04.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff5dc018900 con 0x7ff5ec101b60 2024-04-02T04:51:04.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff5d806f5e0 0x7ff5d8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(224..224 src has 74..224) ==== 11116+0+0 (secure 0 0 0) 0x7ff5dc08f210 con 0x7ff5ec101b60 2024-04-02T04:51:04.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 10 client.15918.objecter ms_dispatch 0x7ff5ec001550 osd_map(224..224 src has 74..224) 2024-04-02T04:51:04.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 3 client.15918.objecter handle_osd_map got epochs [224,224] > 0 2024-04-02T04:51:04.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 3 client.15918.objecter handle_osd_map decoding full epoch 224 2024-04-02T04:51:04.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5f193c700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff5d806f5e0 0x7ff5d8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 20 client.15918.objecter dump_active .. 0 homeless 2024-04-02T04:51:04.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5e2ffd700 10 client.15918.objecter ms_handle_connect 0x7ff5d806f5e0 2024-04-02T04:51:04.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.200+0000 7ff5f193c700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff5d806f5e0 0x7ff5d8071aa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009680 tx=0x7ff5e8019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.204+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff5dc027080 con 0x7ff5ec101b60 2024-04-02T04:51:04.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:04.245 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:04.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:04.370 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:04.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.444+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff5ec1a3d30 con 0x7ff5ec101b60 2024-04-02T04:51:04.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:04.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: audit 2024-04-02T04:51:03.068389+0000 mgr.y (mgr.14538) 298 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: cluster 2024-04-02T04:51:03.182432+0000 mon.a (mon.0) 1616 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: audit 2024-04-02T04:51:03.713790+0000 mon.a (mon.0) 1617 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: cluster 2024-04-02T04:51:03.724981+0000 mon.a (mon.0) 1618 : cluster 0 osdmap e224: 8 total, 7 up, 8 in 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: audit 2024-04-02T04:51:03.728513+0000 mon.c (mon.2) 293 : audit 1 from='client.? 172.21.15.69:0/1818782193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: audit 2024-04-02T04:51:03.730227+0000 mon.a (mon.0) 1619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: audit 2024-04-02T04:51:03.756768+0000 mon.c (mon.2) 294 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:04 smithi098 bash[23790]: audit 2024-04-02T04:51:03.758386+0000 mon.c (mon.2) 295 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:04.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 -- 172.21.15.69:0/1041634043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654104460 msgr2=0x7f06541048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 --2- 172.21.15.69:0/1041634043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654104460 0x7f06541048e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f063c009990 tx=0x7f063c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 -- 172.21.15.69:0/1041634043 shutdown_connections 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 --2- 172.21.15.69:0/1041634043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 0x7f0654111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 --2- 172.21.15.69:0/1041634043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f065410a4f0 0x7f065410a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 --2- 172.21.15.69:0/1041634043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654104460 0x7f06541048e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 -- 172.21.15.69:0/1041634043 >> 172.21.15.69:0/1041634043 conn(0x7f06540fff50 msgr2=0x7f0654102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 -- 172.21.15.69:0/1041634043 shutdown_connections 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 -- 172.21.15.69:0/1041634043 wait complete. 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 librados: starting msgr at 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 librados: starting objecter 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 Processor -- start 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 -- start start 2024-04-02T04:51:04.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 librados: setting wanted keys 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.492+0000 7f065993f700 1 librados: calling monclient init 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654104460 0x7f065419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 0x7f065419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f065410a4f0 0x7f065419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0654113c30 con 0x7f065410a4f0 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0654113ab0 con 0x7f0654104460 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0654113db0 con 0x7f0654104e20 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 0x7f065419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f065410a4f0 0x7f065419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 0x7f065419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44858/0 (socket says 172.21.15.69:44858) 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064bfff700 1 -- 172.21.15.69:0/1916207143 learned_addr learned my addr 172.21.15.69:0/1916207143 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:04.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654104460 0x7f065419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.498 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 -- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 msgr2=0x7f065419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 0x7f065419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 -- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654104460 msgr2=0x7f065419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0654104460 0x7f065419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065419f240 con 0x7f065410a4f0 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0654104e20 0x7f065419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f065410a4f0 0x7f065419eb00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f064c00efb0 tx=0x7f064c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 10 client.?.objecter ms_handle_connect 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c003a00 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f064c00ce20 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065419f530 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c004670 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 10 client.15924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06541a6db0 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065993f700 1 librados: init done 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0638005100 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f064c012050 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f064006f5e0 0x7f0640071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(224..224 src has 74..224) ==== 11116+0+0 (secure 0 0 0) 0x7f064c08edc0 con 0x7f065410a4f0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f064006f5e0 0x7f0640071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 osd_map(224..224 src has 74..224) 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 3 client.15924.objecter handle_osd_map got epochs [224,224] > 0 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 3 client.15924.objecter handle_osd_map decoding full epoch 224 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 20 client.15924.objecter dump_active .. 0 homeless 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v224) ==== 54+0+16 (secure 0 0 0) 0x7f064c090020 con 0x7f065410a4f0 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 10 client.15924.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f064006f5e0 0x7f0640071aa0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0644009be0 tx=0x7f0644009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 1 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f064006f5e0 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0638004730 0x7f0638004b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 1 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 0 1 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380053e0 osd=0 4 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 10 client.15924.objecter _send_command 1 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0638008b70 con 0x7f0638004730 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.496+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0638004730 0x7f0638004b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f0638004730 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f0638004730 0x7f0638004b50 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0638008b70 con 0x7f0638004730 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 1: 0 ) 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f065893d700 10 client.15924.objecter _finish_command 1 = system:0 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f06380053e0 osd=0 4 2024-04-02T04:51:04.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.500+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 0 1 2024-04-02T04:51:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 10 client.15924.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 2 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 2 2024-04-02T04:51:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f06380053e0 osd=0 3 2024-04-02T04:51:04.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 0 2 2024-04-02T04:51:04.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380053e0 osd=0 4 2024-04-02T04:51:04.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 10 client.15924.objecter _send_command 2 2024-04-02T04:51:04.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06380041e0 con 0x7f0638004730 2024-04-02T04:51:04.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06380041e0 con 0x7f0638004730 2024-04-02T04:51:04.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f065893d700 10 client.15924.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f06380053e0 osd=0 4 2024-04-02T04:51:04.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 0 2 2024-04-02T04:51:04.519 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 10 client.15924.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f0638002750 0x7f0638002bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 1 3 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380081c0 osd=1 4 2024-04-02T04:51:04.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 10 client.15924.objecter _send_command 3 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f06380089a0 con 0x7f0638002750 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.516+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f0638002750 0x7f0638002bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f0638002750 0x7f0638002bc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f0638002750 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06380089a0 con 0x7f0638002750 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 3: 0 ) 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f065893d700 10 client.15924.objecter _finish_command 3 = system:0 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f06380081c0 osd=1 4 2024-04-02T04:51:04.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.520+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 1 3 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 10 client.15924.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 4 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 4 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f06380081c0 osd=1 3 2024-04-02T04:51:04.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 1 4 2024-04-02T04:51:04.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380081c0 osd=1 4 2024-04-02T04:51:04.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 10 client.15924.objecter _send_command 4 2024-04-02T04:51:04.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0638007d00 con 0x7f0638002750 2024-04-02T04:51:04.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0638007d00 con 0x7f0638002750 2024-04-02T04:51:04.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f065893d700 10 client.15924.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f06380081c0 osd=1 4 2024-04-02T04:51:04.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 1 4 2024-04-02T04:51:04.539 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 10 client.15924.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 5 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f063800b240 0x7f063800d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 5 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 2 5 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638007f90 osd=2 4 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 10 client.15924.objecter _send_command 5 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0638010490 con 0x7f063800b240 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.536+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f063800b240 0x7f063800d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f063800b240 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f063800b240 0x7f063800d630 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0638010490 con 0x7f063800b240 2024-04-02T04:51:04.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 5: 0 ) 2024-04-02T04:51:04.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f065893d700 10 client.15924.objecter _finish_command 5 = system:0 2024-04-02T04:51:04.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638007f90 osd=2 4 2024-04-02T04:51:04.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.540+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 2 5 2024-04-02T04:51:04.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 10 client.15924.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 6 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 6 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638007f90 osd=2 3 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 2 6 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638007f90 osd=2 4 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 10 client.15924.objecter _send_command 6 2024-04-02T04:51:04.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0638003720 con 0x7f063800b240 2024-04-02T04:51:04.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0638003720 con 0x7f063800b240 2024-04-02T04:51:04.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f065893d700 10 client.15924.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638007f90 osd=2 4 2024-04-02T04:51:04.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.556+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 2 6 2024-04-02T04:51:04.560 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:51:04.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 10 client.15924.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 7 2024-04-02T04:51:04.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f063800ecf0 0x7f063800f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 7 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 3 7 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638001b50 osd=3 4 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 10 client.15924.objecter _send_command 7 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f063800f6c0 con 0x7f063800ecf0 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f063800ecf0 0x7f063800f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f063800ecf0 2024-04-02T04:51:04.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f063800ecf0 0x7f063800f180 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f063800f6c0 con 0x7f063800ecf0 2024-04-02T04:51:04.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 7: 0 ) 2024-04-02T04:51:04.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f065893d700 10 client.15924.objecter _finish_command 7 = system:0 2024-04-02T04:51:04.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638001b50 osd=3 4 2024-04-02T04:51:04.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.560+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 3 7 2024-04-02T04:51:04.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 10 client.15924.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 8 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 8 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638001b50 osd=3 3 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 3 8 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638001b50 osd=3 4 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 10 client.15924.objecter _send_command 8 2024-04-02T04:51:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.576+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06380100d0 con 0x7f063800ecf0 2024-04-02T04:51:04.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06380100d0 con 0x7f063800ecf0 2024-04-02T04:51:04.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 10 client.15924.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638001b50 osd=3 4 2024-04-02T04:51:04.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 3 8 2024-04-02T04:51:04.580 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 10 client.15924.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 9 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f06380100a0 0x7f0638014740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 9 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 4 9 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380023a0 osd=4 4 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 10 client.15924.objecter _send_command 9 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0638017800 con 0x7f06380100a0 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f06380100a0 0x7f0638014740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f0652ffd700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f06380100a0 0x7f0638014740 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f06380100a0 2024-04-02T04:51:04.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0638017800 con 0x7f06380100a0 2024-04-02T04:51:04.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 9: 0 ) 2024-04-02T04:51:04.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 10 client.15924.objecter _finish_command 9 = system:0 2024-04-02T04:51:04.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f06380023a0 osd=4 4 2024-04-02T04:51:04.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.580+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 4 9 2024-04-02T04:51:04.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 10 client.15924.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 10 2024-04-02T04:51:04.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 10 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f06380023a0 osd=4 3 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 4 10 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f06380023a0 osd=4 4 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 10 client.15924.objecter _send_command 10 2024-04-02T04:51:04.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.596+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0638016590 con 0x7f06380100a0 2024-04-02T04:51:04.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0638016590 con 0x7f06380100a0 2024-04-02T04:51:04.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f065893d700 10 client.15924.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f06380023a0 osd=4 4 2024-04-02T04:51:04.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 4 10 2024-04-02T04:51:04.603 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 10 client.15924.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 11 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f0638016cb0 0x7f0638017110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 11 2024-04-02T04:51:04.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 5 11 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638016590 osd=5 4 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 10 client.15924.objecter _send_command 11 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.600+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f063801ac10 con 0x7f0638016cb0 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f0638016cb0 0x7f0638017110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f0638016cb0 2024-04-02T04:51:04.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f064bfff700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f0638016cb0 0x7f0638017110 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:04.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f063c033a50 con 0x7f0638016cb0 2024-04-02T04:51:04.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 11: 0 ) 2024-04-02T04:51:04.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f065893d700 10 client.15924.objecter _finish_command 11 = system:0 2024-04-02T04:51:04.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638016590 osd=5 4 2024-04-02T04:51:04.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.604+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 5 11 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 10 client.15924.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 12 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 12 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638016590 osd=5 3 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 5 12 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638016590 osd=5 4 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 10 client.15924.objecter _send_command 12 2024-04-02T04:51:04.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.620+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0638019a50 con 0x7f0638016cb0 2024-04-02T04:51:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:04.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0638019a50 con 0x7f0638016cb0 2024-04-02T04:51:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f065893d700 10 client.15924.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638016590 osd=5 4 2024-04-02T04:51:04.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 5 12 2024-04-02T04:51:04.626 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 10 client.15924.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 13 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f063801a1c0 0x7f063801a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 13 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 6 13 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638019a50 osd=6 4 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 10 client.15924.objecter _send_command 13 2024-04-02T04:51:04.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0638016070 con 0x7f063801a1c0 2024-04-02T04:51:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f063801a1c0 0x7f063801a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f065893d700 10 client.15924.objecter ms_handle_connect 0x7f063801a1c0 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.624+0000 7f06537fe700 1 --2- 172.21.15.69:0/1916207143 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f063801a1c0 0x7f063801a600 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.628+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0638016070 con 0x7f063801a1c0 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.628+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 13: 0 ) 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.628+0000 7f065893d700 10 client.15924.objecter _finish_command 13 = system:0 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.628+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638019a50 osd=6 4 2024-04-02T04:51:04.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.628+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 6 13 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 10 client.15924.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 14 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 20 client.15924.objecter _get_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_remove -1 14 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 20 client.15924.objecter get_session s=0x7f0638019a50 osd=6 3 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign 6 14 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 20 client.15924.objecter put_session s=0x7f0638019a50 osd=6 4 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 10 client.15924.objecter _send_command 14 2024-04-02T04:51:04.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.644+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0638016070 con 0x7f063801a1c0 2024-04-02T04:51:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0638016070 con 0x7f063801a1c0 2024-04-02T04:51:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f065893d700 10 client.15924.objecter ms_dispatch 0x7f0654001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f065893d700 10 client.15924.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f065893d700 20 client.15924.objecter put_session s=0x7f0638019a50 osd=6 4 2024-04-02T04:51:04.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f065893d700 15 client.15924.objecter _session_command_op_remove 6 14 2024-04-02T04:51:04.650 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f064a7fc700 10 client.15924.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f064a7fc700 15 client.15924.objecter _session_command_op_assign -1 15 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f064a7fc700 20 client.15924.objecter _get_session osd=-1 returning homeless 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f064a7fc700 10 client.15924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=225}) -- 0x7f06380162e0 con 0x7f065410a4f0 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f064a7fc700 1 -- 172.21.15.69:0/1916207143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f063801e5d0 con 0x7f065410a4f0 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f065893d700 1 -- 172.21.15.69:0/1916207143 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_get_version_reply(handle=1 version=224) ==== 24+0+0 (secure 0 0 0) 0x7f064c05ac30 con 0x7f065410a4f0 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f0651ffb700 10 client.15924.objecter _check_command_map_dne tid 15 current 224 map_dne_bound 224 2024-04-02T04:51:04.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f0651ffb700 10 client.15924.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.648+0000 7f0651ffb700 15 client.15924.objecter _session_command_op_remove -1 15 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:51:04.652 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: audit 2024-04-02T04:51:03.068389+0000 mgr.y (mgr.14538) 298 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: cluster 2024-04-02T04:51:03.182432+0000 mon.a (mon.0) 1616 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: audit 2024-04-02T04:51:03.713790+0000 mon.a (mon.0) 1617 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: cluster 2024-04-02T04:51:03.724981+0000 mon.a (mon.0) 1618 : cluster 0 osdmap e224: 8 total, 7 up, 8 in 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: audit 2024-04-02T04:51:03.728513+0000 mon.c (mon.2) 293 : audit 1 from='client.? 172.21.15.69:0/1818782193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: audit 2024-04-02T04:51:03.730227+0000 mon.a (mon.0) 1619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: audit 2024-04-02T04:51:03.756768+0000 mon.c (mon.2) 294 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:04.653 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[21528]: audit 2024-04-02T04:51:03.758386+0000 mon.c (mon.2) 295 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: audit 2024-04-02T04:51:03.068389+0000 mgr.y (mgr.14538) 298 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: cluster 2024-04-02T04:51:03.182432+0000 mon.a (mon.0) 1616 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: audit 2024-04-02T04:51:03.713790+0000 mon.a (mon.0) 1617 : audit 1 from='client.? 172.21.15.69:0/1653232639' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-72389-16"}]': finished 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: cluster 2024-04-02T04:51:03.724981+0000 mon.a (mon.0) 1618 : cluster 0 osdmap e224: 8 total, 7 up, 8 in 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: audit 2024-04-02T04:51:03.728513+0000 mon.c (mon.2) 293 : audit 1 from='client.? 172.21.15.69:0/1818782193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: audit 2024-04-02T04:51:03.730227+0000 mon.a (mon.0) 1619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: audit 2024-04-02T04:51:03.756768+0000 mon.c (mon.2) 294 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:04.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:04 smithi069 bash[28904]: audit 2024-04-02T04:51:03.758386+0000 mon.c (mon.2) 295 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:04.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.664+0000 7ff5e2ffd700 1 -- 172.21.15.69:0/482569044 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v225) ==== 97+0+0 (secure 0 0 0) 0x7ff5dc014030 con 0x7ff5ec101b60 2024-04-02T04:51:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 10 librados: watch_flush enter 2024-04-02T04:51:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 10 librados: watch_flush exit 2024-04-02T04:51:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 20 client.15918.objecter shutdown clearing up homeless session... 2024-04-02T04:51:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 10 client.15918.objecter successfully canceled tick 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff5d806f5e0 msgr2=0x7ff5d8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff5d806f5e0 0x7ff5d8071aa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009680 tx=0x7ff5e8019040 comp rx=0 tx=0).stop 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 msgr2=0x7ff5ec197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 0x7ff5ec197460 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00c970 tx=0x7ff5dc00cc80 comp rx=0 tx=0).stop 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 shutdown_connections 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff5d806f5e0 0x7ff5d8071aa0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ec107220 0x7ff5ec19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5ec1011a0 0x7ff5ec196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 --2- 172.21.15.69:0/482569044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5ec101b60 0x7ff5ec197460 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00c970 tx=0x7ff5dc00cc80 comp rx=0 tx=0).stop 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 >> 172.21.15.69:0/482569044 conn(0x7ff5ec0771e0 msgr2=0x7ff5ec104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 shutdown_connections 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.668+0000 7ff5f3ba0700 1 -- 172.21.15.69:0/482569044 wait complete. 2024-04-02T04:51:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:04.672+0000 7ff5f3ba0700 1 librados: shutdown 2024-04-02T04:51:04.673 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:51:04.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:04.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:04.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:04.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:05.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:05.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: cluster 2024-04-02T04:51:03.585872+0000 osd.2 (osd.2) 115 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: cluster 2024-04-02T04:51:03.586011+0000 osd.2 (osd.2) 116 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: cluster 2024-04-02T04:51:03.931238+0000 mgr.y (mgr.14538) 299 : cluster 0 pgmap v313: 388 pgs: 32 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.190907+0000 mon.a (mon.0) 1620 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.192009+0000 mon.c (mon.2) 296 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.201660+0000 mon.a (mon.0) 1621 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.203593+0000 mon.c (mon.2) 297 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.208009+0000 mon.a (mon.0) 1622 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:05.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.450781+0000 mon.a (mon.0) 1623 : audit 1 from='client.? 172.21.15.69:0/482569044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:05.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.501976+0000 mon.a (mon.0) 1624 : audit 0 from='client.? 172.21.15.69:0/1916207143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:05.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: cluster 2024-04-02T04:51:04.666733+0000 mon.a (mon.0) 1625 : cluster 3 Health check update: Degraded data redundancy: 350/2928 objects degraded (11.954%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:05.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: cluster 2024-04-02T04:51:04.667723+0000 mon.a (mon.0) 1626 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:05.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:05.269 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:05.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: cluster 2024-04-02T04:51:03.585872+0000 osd.2 (osd.2) 115 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: cluster 2024-04-02T04:51:03.586011+0000 osd.2 (osd.2) 116 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: cluster 2024-04-02T04:51:03.931238+0000 mgr.y (mgr.14538) 299 : cluster 0 pgmap v313: 388 pgs: 32 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.190907+0000 mon.a (mon.0) 1620 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.192009+0000 mon.c (mon.2) 296 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.201660+0000 mon.a (mon.0) 1621 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.203593+0000 mon.c (mon.2) 297 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.208009+0000 mon.a (mon.0) 1622 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.450781+0000 mon.a (mon.0) 1623 : audit 1 from='client.? 172.21.15.69:0/482569044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.501976+0000 mon.a (mon.0) 1624 : audit 0 from='client.? 172.21.15.69:0/1916207143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: cluster 2024-04-02T04:51:04.666733+0000 mon.a (mon.0) 1625 : cluster 3 Health check update: Degraded data redundancy: 350/2928 objects degraded (11.954%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: cluster 2024-04-02T04:51:04.667723+0000 mon.a (mon.0) 1626 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.669438+0000 mon.a (mon.0) 1627 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.669522+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.669617+0000 mon.a (mon.0) 1629 : audit 1 from='client.? 172.21.15.69:0/482569044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: cluster 2024-04-02T04:51:04.673466+0000 mon.a (mon.0) 1630 : cluster 0 osdmap e225: 8 total, 7 up, 8 in 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.676529+0000 mon.c (mon.2) 298 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.676846+0000 mon.c (mon.2) 299 : audit 1 from='client.? 172.21.15.69:0/152488484' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.712501+0000 mon.a (mon.0) 1631 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.712855+0000 mon.a (mon.0) 1632 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:05.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:05 smithi098 bash[23790]: audit 2024-04-02T04:51:04.757860+0000 mon.c (mon.2) 300 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:05.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.669438+0000 mon.a (mon.0) 1627 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.669522+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.669617+0000 mon.a (mon.0) 1629 : audit 1 from='client.? 172.21.15.69:0/482569044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: cluster 2024-04-02T04:51:04.673466+0000 mon.a (mon.0) 1630 : cluster 0 osdmap e225: 8 total, 7 up, 8 in 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.676529+0000 mon.c (mon.2) 298 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.676846+0000 mon.c (mon.2) 299 : audit 1 from='client.? 172.21.15.69:0/152488484' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.712501+0000 mon.a (mon.0) 1631 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.712855+0000 mon.a (mon.0) 1632 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:05.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[21528]: audit 2024-04-02T04:51:04.757860+0000 mon.c (mon.2) 300 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: cluster 2024-04-02T04:51:03.585872+0000 osd.2 (osd.2) 115 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: cluster 2024-04-02T04:51:03.586011+0000 osd.2 (osd.2) 116 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: cluster 2024-04-02T04:51:03.931238+0000 mgr.y (mgr.14538) 299 : cluster 0 pgmap v313: 388 pgs: 32 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.190907+0000 mon.a (mon.0) 1620 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.192009+0000 mon.c (mon.2) 296 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.201660+0000 mon.a (mon.0) 1621 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.203593+0000 mon.c (mon.2) 297 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.208009+0000 mon.a (mon.0) 1622 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.450781+0000 mon.a (mon.0) 1623 : audit 1 from='client.? 172.21.15.69:0/482569044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.501976+0000 mon.a (mon.0) 1624 : audit 0 from='client.? 172.21.15.69:0/1916207143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: cluster 2024-04-02T04:51:04.666733+0000 mon.a (mon.0) 1625 : cluster 3 Health check update: Degraded data redundancy: 350/2928 objects degraded (11.954%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: cluster 2024-04-02T04:51:04.667723+0000 mon.a (mon.0) 1626 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:05.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.669438+0000 mon.a (mon.0) 1627 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-72153-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.669522+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.669617+0000 mon.a (mon.0) 1629 : audit 1 from='client.? 172.21.15.69:0/482569044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: cluster 2024-04-02T04:51:04.673466+0000 mon.a (mon.0) 1630 : cluster 0 osdmap e225: 8 total, 7 up, 8 in 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.676529+0000 mon.c (mon.2) 298 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.676846+0000 mon.c (mon.2) 299 : audit 1 from='client.? 172.21.15.69:0/152488484' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.712501+0000 mon.a (mon.0) 1631 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.712855+0000 mon.a (mon.0) 1632 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:05.517 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:05 smithi069 bash[28904]: audit 2024-04-02T04:51:04.757860+0000 mon.c (mon.2) 300 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:05.529 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:05.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:05.660 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:05.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:05.782 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:05.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:05.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:05.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:06.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:06.038 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:06.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:06.164 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:06.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:06.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:06.398 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:51:06.416 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:06.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:06.546 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:06.649 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:06.670 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:06.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:06.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:06.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:06.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:06.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:06.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:06 smithi098 bash[23790]: audit 2024-04-02T04:51:05.673377+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:06.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:06 smithi098 bash[23790]: audit 2024-04-02T04:51:05.758727+0000 mon.c (mon.2) 301 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:06.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:06 smithi098 bash[23790]: cluster 2024-04-02T04:51:06.149659+0000 mon.a (mon.0) 1634 : cluster 0 osdmap e226: 8 total, 7 up, 8 in 2024-04-02T04:51:07.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:06 smithi069 bash[21528]: audit 2024-04-02T04:51:05.673377+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:07.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:06 smithi069 bash[21528]: audit 2024-04-02T04:51:05.758727+0000 mon.c (mon.2) 301 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:07.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:06 smithi069 bash[21528]: cluster 2024-04-02T04:51:06.149659+0000 mon.a (mon.0) 1634 : cluster 0 osdmap e226: 8 total, 7 up, 8 in 2024-04-02T04:51:07.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:06 smithi069 bash[28904]: audit 2024-04-02T04:51:05.673377+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-72145-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:07.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:06 smithi069 bash[28904]: audit 2024-04-02T04:51:05.758727+0000 mon.c (mon.2) 301 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:07.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:06 smithi069 bash[28904]: cluster 2024-04-02T04:51:06.149659+0000 mon.a (mon.0) 1634 : cluster 0 osdmap e226: 8 total, 7 up, 8 in 2024-04-02T04:51:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:51:07.085 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:51:07.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:07.166 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 -- 172.21.15.69:0/4172712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40102c90 msgr2=0x7fef4010f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 --2- 172.21.15.69:0/4172712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40102c90 0x7fef4010f170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b0a0 tx=0x7fef3800b3b0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 -- 172.21.15.69:0/4172712785 shutdown_connections 2024-04-02T04:51:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 --2- 172.21.15.69:0/4172712785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef40102c90 0x7fef4010f170 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 --2- 172.21.15.69:0/4172712785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40108330 0x7fef40108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 --2- 172.21.15.69:0/4172712785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef401022f0 0x7fef40102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 -- 172.21.15.69:0/4172712785 >> 172.21.15.69:0/4172712785 conn(0x7fef400793f0 msgr2=0x7fef40079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 -- 172.21.15.69:0/4172712785 shutdown_connections 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 -- 172.21.15.69:0/4172712785 wait complete. 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 librados: starting msgr at 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 librados: starting objecter 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.196+0000 7fef47552700 1 Processor -- start 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- start start 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 librados: setting wanted keys 2024-04-02T04:51:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 librados: calling monclient init 2024-04-02T04:51:07.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 0x7fef401980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40102c90 0x7fef40198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 0x7fef4019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef40111b20 con 0x7fef40108330 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef401119a0 con 0x7fef40102c90 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fef40111ca0 con 0x7fef401022f0 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef45aef700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 0x7fef4019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 0x7fef401980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef45aef700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 0x7fef4019c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35534/0 (socket says 172.21.15.69:35534) 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 0x7fef401980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44930/0 (socket says 172.21.15.69:44930) 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef45aef700 1 -- 172.21.15.69:0/6828100 learned_addr learned my addr 172.21.15.69:0/6828100 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef44aed700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40102c90 0x7fef40198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 -- 172.21.15.69:0/6828100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40102c90 msgr2=0x7fef40198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40102c90 0x7fef40198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 -- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 msgr2=0x7fef4019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 0x7fef4019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 -- 172.21.15.69:0/6828100 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4019d0e0 con 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef44aed700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40102c90 0x7fef40198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef45aef700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 0x7fef4019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef452ee700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 0x7fef401980d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00b450 tx=0x7fef3c00b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 10 client.?.objecter ms_handle_connect 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c015020 con 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fef3c011cb0 con 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4019d3d0 con 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c010c40 con 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 10 client.26012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef40074470 con 0x7fef401022f0 2024-04-02T04:51:07.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 librados: init done 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4004ff00 con 0x7fef401022f0 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fef3c018020 con 0x7fef401022f0 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fef2c06f5e0 0x7fef2c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(227..227 src has 74..227) ==== 12142+0+0 (secure 0 0 0) 0x7fef3c08e940 con 0x7fef401022f0 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 10 client.26012.objecter ms_dispatch 0x7fef40001550 osd_map(227..227 src has 74..227) 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 3 client.26012.objecter handle_osd_map got epochs [227,227] > 0 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 3 client.26012.objecter handle_osd_map decoding full epoch 227 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef44aed700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fef2c06f5e0 0x7fef2c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 20 client.26012.objecter dump_active .. 0 homeless 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef367fc700 10 client.26012.objecter ms_handle_connect 0x7fef2c06f5e0 2024-04-02T04:51:07.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.200+0000 7fef44aed700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fef2c06f5e0 0x7fef2c071aa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fef40079b10 tx=0x7fef30009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:07.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.208+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fef3c01f020 con 0x7fef401022f0 2024-04-02T04:51:07.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:07.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:07.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.404+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef401097c0 con 0x7fef2c06f5e0 2024-04-02T04:51:07.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.440+0000 7fef367fc700 1 -- 172.21.15.69:0/6828100 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+1006013 (secure 0 0 0) 0x7fef401097c0 con 0x7fef2c06f5e0 2024-04-02T04:51:07.451 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 10 librados: watch_flush enter 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 10 librados: watch_flush exit 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 20 client.26012.objecter shutdown clearing up homeless session... 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 10 client.26012.objecter successfully canceled tick 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fef2c06f5e0 msgr2=0x7fef2c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fef2c06f5e0 0x7fef2c071aa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fef40079b10 tx=0x7fef30009260 comp rx=0 tx=0).stop 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 msgr2=0x7fef401980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 0x7fef401980d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00b450 tx=0x7fef3c00b760 comp rx=0 tx=0).stop 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 shutdown_connections 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fef2c06f5e0 0x7fef2c071aa0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef40108330 0x7fef4019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fef401022f0 0x7fef401980d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 --2- 172.21.15.69:0/6828100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40102c90 0x7fef40198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 >> 172.21.15.69:0/6828100 conn(0x7fef400793f0 msgr2=0x7fef401090b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 shutdown_connections 2024-04-02T04:51:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 -- 172.21.15.69:0/6828100 wait complete. 2024-04-02T04:51:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:07.444+0000 7fef47552700 1 librados: shutdown 2024-04-02T04:51:07.465 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:51:07.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.539 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:07.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:07.667 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:07.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:07.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[28904]: cluster 2024-04-02T04:51:05.932860+0000 mgr.y (mgr.14538) 300 : cluster 0 pgmap v316: 388 pgs: 32 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:07.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[28904]: audit 2024-04-02T04:51:06.682458+0000 mon.a (mon.0) 1635 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:07.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[28904]: cluster 2024-04-02T04:51:06.688279+0000 mon.a (mon.0) 1636 : cluster 0 osdmap e227: 8 total, 7 up, 8 in 2024-04-02T04:51:07.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[28904]: audit 2024-04-02T04:51:06.709296+0000 mon.a (mon.0) 1637 : audit 1 from='client.? 172.21.15.69:0/3666695339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-72153-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:07.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[28904]: audit 2024-04-02T04:51:06.759690+0000 mon.c (mon.2) 302 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[21528]: cluster 2024-04-02T04:51:05.932860+0000 mgr.y (mgr.14538) 300 : cluster 0 pgmap v316: 388 pgs: 32 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[21528]: audit 2024-04-02T04:51:06.682458+0000 mon.a (mon.0) 1635 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[21528]: cluster 2024-04-02T04:51:06.688279+0000 mon.a (mon.0) 1636 : cluster 0 osdmap e227: 8 total, 7 up, 8 in 2024-04-02T04:51:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[21528]: audit 2024-04-02T04:51:06.709296+0000 mon.a (mon.0) 1637 : audit 1 from='client.? 172.21.15.69:0/3666695339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-72153-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:07 smithi069 bash[21528]: audit 2024-04-02T04:51:06.759690+0000 mon.c (mon.2) 302 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:07.787 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:07.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.913 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:07.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:07.950 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:07 smithi098 bash[23790]: cluster 2024-04-02T04:51:05.932860+0000 mgr.y (mgr.14538) 300 : cluster 0 pgmap v316: 388 pgs: 32 unknown, 80 active+undersized, 228 active+clean, 48 active+undersized+degraded; 592 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 350/2928 objects degraded (11.954%) 2024-04-02T04:51:07.950 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:07 smithi098 bash[23790]: audit 2024-04-02T04:51:06.682458+0000 mon.a (mon.0) 1635 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:07.950 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:07 smithi098 bash[23790]: cluster 2024-04-02T04:51:06.688279+0000 mon.a (mon.0) 1636 : cluster 0 osdmap e227: 8 total, 7 up, 8 in 2024-04-02T04:51:07.950 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:07 smithi098 bash[23790]: audit 2024-04-02T04:51:06.709296+0000 mon.a (mon.0) 1637 : audit 1 from='client.? 172.21.15.69:0/3666695339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-72153-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:07.951 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:07 smithi098 bash[23790]: audit 2024-04-02T04:51:06.759690+0000 mon.c (mon.2) 302 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:08.016 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:08.040 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:08.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:08.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:08.293 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:08.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:08.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:08.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:08.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:08.571 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-04-02T04:51:06.684322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605895,"num_objects":976,"num_object_clones":543,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":1,"num_read":1924,"num_read_kb":1655,"num_write":1424,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2152,"ondisk_log_size":2152,"up":940,"acting":940,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1348,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":571112,"kb_used_data":109152,"kb_used_omap":12,"kb_used_meta":461747,"kb_avail":749390104,"statfs":{"total":767960285184,"available":767375466496,"internally_reserved":0,"allocated":111771648,"data_stored":92865710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":472829488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.973590"},"pg_stats":[{"pgid":"210.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"209.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"210.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:51:06.673847+0000","last_change":"2024-04-02T04:51:06.673847+0000","last_active":"2024-04-02T04:51:06.673847+0000","last_peered":"2024-04-02T04:51:06.673847+0000","last_clean":"2024-04-02T04:51:06.673847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:51:06.673847+0000","last_undegraded":"2024-04-02T04:51:06.673847+0000","last_fullsized":"2024-04-02T04:51:06.673847+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:06.673847+0000","last_clean_scrub_stamp":"2024-04-02T04:51:06.673847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"165.1a","version":"195'4","reported_seq":89,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.732914+0000","last_change":"2024-04-02T04:50:34.521208+0000","last_active":"2024-04-02T04:51:02.732914+0000","last_peered":"2024-04-02T04:51:02.732914+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.391289+0000","last_became_peered":"2024-04-02T04:50:27.391289+0000","last_unstale":"2024-04-02T04:51:02.732914+0000","last_undegraded":"2024-04-02T04:50:34.520997+0000","last_fullsized":"2024-04-02T04:50:27.357482+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:26.391102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"165.1b","version":"195'5","reported_seq":93,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.750099+0000","last_change":"2024-04-02T04:50:27.461514+0000","last_active":"2024-04-02T04:51:02.750099+0000","last_peered":"2024-04-02T04:51:02.750099+0000","last_clean":"2024-04-02T04:51:02.750099+0000","last_became_active":"2024-04-02T04:50:27.461142+0000","last_became_peered":"2024-04-02T04:50:27.461142+0000","last_unstale":"2024-04-02T04:51:02.750099+0000","last_undegraded":"2024-04-02T04:51:02.750099+0000","last_fullsized":"2024-04-02T04:51:02.750099+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:40.873838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"165.18","version":"195'4","reported_seq":91,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.753028+0000","last_change":"2024-04-02T04:50:27.435434+0000","last_active":"2024-04-02T04:51:02.753028+0000","last_peered":"2024-04-02T04:51:02.753028+0000","last_clean":"2024-04-02T04:51:02.753028+0000","last_became_active":"2024-04-02T04:50:27.434978+0000","last_became_peered":"2024-04-02T04:50:27.434978+0000","last_unstale":"2024-04-02T04:51:02.753028+0000","last_undegraded":"2024-04-02T04:51:02.753028+0000","last_fullsized":"2024-04-02T04:51:02.753028+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.570223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"165.19","version":"195'9","reported_seq":97,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.729813+0000","last_change":"2024-04-02T04:50:34.529833+0000","last_active":"2024-04-02T04:51:02.729813+0000","last_peered":"2024-04-02T04:51:02.729813+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.454668+0000","last_became_peered":"2024-04-02T04:50:27.454668+0000","last_unstale":"2024-04-02T04:51:02.729813+0000","last_undegraded":"2024-04-02T04:50:34.528694+0000","last_fullsized":"2024-04-02T04:50:27.353984+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:53.962926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"165.1e","version":"0'0","reported_seq":84,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.765840+0000","last_change":"2024-04-02T04:50:27.463638+0000","last_active":"2024-04-02T04:51:02.765840+0000","last_peered":"2024-04-02T04:51:02.765840+0000","last_clean":"2024-04-02T04:51:02.765840+0000","last_became_active":"2024-04-02T04:50:27.463279+0000","last_became_peered":"2024-04-02T04:50:27.463279+0000","last_unstale":"2024-04-02T04:51:02.765840+0000","last_undegraded":"2024-04-02T04:51:02.765840+0000","last_fullsized":"2024-04-02T04:51:02.765840+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.349271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"165.1f","version":"195'6","reported_seq":96,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.764933+0000","last_change":"2024-04-02T04:50:27.464769+0000","last_active":"2024-04-02T04:51:02.764933+0000","last_peered":"2024-04-02T04:51:02.764933+0000","last_clean":"2024-04-02T04:51:02.764933+0000","last_became_active":"2024-04-02T04:50:27.462550+0000","last_became_peered":"2024-04-02T04:50:27.462550+0000","last_unstale":"2024-04-02T04:51:02.764933+0000","last_undegraded":"2024-04-02T04:51:02.764933+0000","last_fullsized":"2024-04-02T04:51:02.764933+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.172375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"165.1c","version":"195'6","reported_seq":95,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.753553+0000","last_change":"2024-04-02T04:50:27.463383+0000","last_active":"2024-04-02T04:51:02.753553+0000","last_peered":"2024-04-02T04:51:02.753553+0000","last_clean":"2024-04-02T04:51:02.753553+0000","last_became_active":"2024-04-02T04:50:27.462580+0000","last_became_peered":"2024-04-02T04:50:27.462580+0000","last_unstale":"2024-04-02T04:51:02.753553+0000","last_undegraded":"2024-04-02T04:51:02.753553+0000","last_fullsized":"2024-04-02T04:51:02.753553+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:34.996182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"165.1d","version":"195'15","reported_seq":112,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.724723+0000","last_change":"2024-04-02T04:50:27.445968+0000","last_active":"2024-04-02T04:51:02.724723+0000","last_peered":"2024-04-02T04:51:02.724723+0000","last_clean":"2024-04-02T04:51:02.724723+0000","last_became_active":"2024-04-02T04:50:27.441751+0000","last_became_peered":"2024-04-02T04:50:27.441751+0000","last_unstale":"2024-04-02T04:51:02.724723+0000","last_undegraded":"2024-04-02T04:51:02.724723+0000","last_fullsized":"2024-04-02T04:51:02.724723+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.448401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"165.12","version":"195'3","reported_seq":86,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.718251+0000","last_change":"2024-04-02T04:50:34.940161+0000","last_active":"2024-04-02T04:51:02.718251+0000","last_peered":"2024-04-02T04:51:02.718251+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.455809+0000","last_became_peered":"2024-04-02T04:50:27.455809+0000","last_unstale":"2024-04-02T04:51:02.718251+0000","last_undegraded":"2024-04-02T04:50:34.937927+0000","last_fullsized":"2024-04-02T04:50:27.371468+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:48.395895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"165.13","version":"195'8","reported_seq":94,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.743167+0000","last_change":"2024-04-02T04:50:34.468045+0000","last_active":"2024-04-02T04:51:02.743167+0000","last_peered":"2024-04-02T04:51:02.743167+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.424958+0000","last_became_peered":"2024-04-02T04:50:27.424958+0000","last_unstale":"2024-04-02T04:51:02.743167+0000","last_undegraded":"2024-04-02T04:50:34.466799+0000","last_fullsized":"2024-04-02T04:50:27.353406+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:46.202061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"165.10","version":"195'6","reported_seq":92,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.729524+0000","last_change":"2024-04-02T04:50:34.453905+0000","last_active":"2024-04-02T04:51:02.729524+0000","last_peered":"2024-04-02T04:51:02.729524+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.413520+0000","last_became_peered":"2024-04-02T04:50:27.413520+0000","last_unstale":"2024-04-02T04:51:02.729524+0000","last_undegraded":"2024-04-02T04:50:34.452874+0000","last_fullsized":"2024-04-02T04:50:27.354145+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.747262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"165.11","version":"195'19","reported_seq":166,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.732903+0000","last_change":"2024-04-02T04:50:34.404746+0000","last_active":"2024-04-02T04:51:02.732903+0000","last_peered":"2024-04-02T04:51:02.732903+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.449537+0000","last_became_peered":"2024-04-02T04:50:27.449537+0000","last_unstale":"2024-04-02T04:51:02.732903+0000","last_undegraded":"2024-04-02T04:50:34.402326+0000","last_fullsized":"2024-04-02T04:50:27.387079+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":19}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"165.16","version":"195'1","reported_seq":83,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.740990+0000","last_change":"2024-04-02T04:50:34.479648+0000","last_active":"2024-04-02T04:51:02.740990+0000","last_peered":"2024-04-02T04:51:02.740990+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.415780+0000","last_became_peered":"2024-04-02T04:50:27.415780+0000","last_unstale":"2024-04-02T04:51:02.740990+0000","last_undegraded":"2024-04-02T04:50:34.478617+0000","last_fullsized":"2024-04-02T04:50:27.377984+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:32.705156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"165.17","version":"195'8","reported_seq":99,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748789+0000","last_change":"2024-04-02T04:50:27.421841+0000","last_active":"2024-04-02T04:51:02.748789+0000","last_peered":"2024-04-02T04:51:02.748789+0000","last_clean":"2024-04-02T04:51:02.748789+0000","last_became_active":"2024-04-02T04:50:27.421547+0000","last_became_peered":"2024-04-02T04:50:27.421547+0000","last_unstale":"2024-04-02T04:51:02.748789+0000","last_undegraded":"2024-04-02T04:51:02.748789+0000","last_fullsized":"2024-04-02T04:51:02.748789+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.204335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"165.14","version":"195'25","reported_seq":128,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.715976+0000","last_change":"2024-04-02T04:50:27.425118+0000","last_active":"2024-04-02T04:51:02.715976+0000","last_peered":"2024-04-02T04:51:02.715976+0000","last_clean":"2024-04-02T04:51:02.715976+0000","last_became_active":"2024-04-02T04:50:27.424880+0000","last_became_peered":"2024-04-02T04:50:27.424880+0000","last_unstale":"2024-04-02T04:51:02.715976+0000","last_undegraded":"2024-04-02T04:51:02.715976+0000","last_fullsized":"2024-04-02T04:51:02.715976+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:40.480643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":25,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"165.15","version":"195'8","reported_seq":97,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728847+0000","last_change":"2024-04-02T04:50:27.420080+0000","last_active":"2024-04-02T04:51:02.728847+0000","last_peered":"2024-04-02T04:51:02.728847+0000","last_clean":"2024-04-02T04:51:02.728847+0000","last_became_active":"2024-04-02T04:50:27.419484+0000","last_became_peered":"2024-04-02T04:50:27.419484+0000","last_unstale":"2024-04-02T04:51:02.728847+0000","last_undegraded":"2024-04-02T04:51:02.728847+0000","last_fullsized":"2024-04-02T04:51:02.728847+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:29.551760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"165.a","version":"195'8","reported_seq":95,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.721432+0000","last_change":"2024-04-02T04:50:34.441432+0000","last_active":"2024-04-02T04:51:02.721432+0000","last_peered":"2024-04-02T04:51:02.721432+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.403868+0000","last_became_peered":"2024-04-02T04:50:27.403868+0000","last_unstale":"2024-04-02T04:51:02.721432+0000","last_undegraded":"2024-04-02T04:50:34.440252+0000","last_fullsized":"2024-04-02T04:50:27.367349+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:46.632861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"165.b","version":"195'5","reported_seq":93,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.722521+0000","last_change":"2024-04-02T04:50:27.460488+0000","last_active":"2024-04-02T04:51:02.722521+0000","last_peered":"2024-04-02T04:51:02.722521+0000","last_clean":"2024-04-02T04:51:02.722521+0000","last_became_active":"2024-04-02T04:50:27.460175+0000","last_became_peered":"2024-04-02T04:50:27.460175+0000","last_unstale":"2024-04-02T04:51:02.722521+0000","last_undegraded":"2024-04-02T04:51:02.722521+0000","last_fullsized":"2024-04-02T04:51:02.722521+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:50.721039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"165.8","version":"195'3","reported_seq":99,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728835+0000","last_change":"2024-04-02T04:50:36.417379+0000","last_active":"2024-04-02T04:51:02.728835+0000","last_peered":"2024-04-02T04:51:02.728835+0000","last_clean":"2024-04-02T04:51:02.728835+0000","last_became_active":"2024-04-02T04:50:27.440344+0000","last_became_peered":"2024-04-02T04:50:27.440344+0000","last_unstale":"2024-04-02T04:51:02.728835+0000","last_undegraded":"2024-04-02T04:51:02.728835+0000","last_fullsized":"2024-04-02T04:51:02.728835+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'3","last_scrub_stamp":"2024-04-02T04:50:36.417269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:36.417269+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:51.496212+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"165.9","version":"195'3","reported_seq":98,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727302+0000","last_change":"2024-04-02T04:50:35.587551+0000","last_active":"2024-04-02T04:51:02.727302+0000","last_peered":"2024-04-02T04:51:02.727302+0000","last_clean":"2024-04-02T04:51:02.727302+0000","last_became_active":"2024-04-02T04:50:27.429570+0000","last_became_peered":"2024-04-02T04:50:27.429570+0000","last_unstale":"2024-04-02T04:51:02.727302+0000","last_undegraded":"2024-04-02T04:51:02.727302+0000","last_fullsized":"2024-04-02T04:51:02.727302+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'3","last_scrub_stamp":"2024-04-02T04:50:35.587366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:35.587366+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.405618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"165.e","version":"195'9","reported_seq":97,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.749755+0000","last_change":"2024-04-02T04:50:34.447025+0000","last_active":"2024-04-02T04:51:02.749755+0000","last_peered":"2024-04-02T04:51:02.749755+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.432208+0000","last_became_peered":"2024-04-02T04:50:27.432208+0000","last_unstale":"2024-04-02T04:51:02.749755+0000","last_undegraded":"2024-04-02T04:50:34.445852+0000","last_fullsized":"2024-04-02T04:50:27.373021+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:44.780921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"165.f","version":"195'3","reported_seq":89,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.758425+0000","last_change":"2024-04-02T04:50:27.445683+0000","last_active":"2024-04-02T04:51:02.758425+0000","last_peered":"2024-04-02T04:51:02.758425+0000","last_clean":"2024-04-02T04:51:02.758425+0000","last_became_active":"2024-04-02T04:50:27.433139+0000","last_became_peered":"2024-04-02T04:50:27.433139+0000","last_unstale":"2024-04-02T04:51:02.758425+0000","last_undegraded":"2024-04-02T04:51:02.758425+0000","last_fullsized":"2024-04-02T04:51:02.758425+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:40.223498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"165.c","version":"195'9","reported_seq":96,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.716665+0000","last_change":"2024-04-02T04:50:34.530511+0000","last_active":"2024-04-02T04:51:02.716665+0000","last_peered":"2024-04-02T04:51:02.716665+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.422132+0000","last_became_peered":"2024-04-02T04:50:27.422132+0000","last_unstale":"2024-04-02T04:51:02.716665+0000","last_undegraded":"2024-04-02T04:50:34.530397+0000","last_fullsized":"2024-04-02T04:50:27.371044+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:36.257126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"165.d","version":"195'16","reported_seq":112,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748542+0000","last_change":"2024-04-02T04:50:27.419686+0000","last_active":"2024-04-02T04:51:02.748542+0000","last_peered":"2024-04-02T04:51:02.748542+0000","last_clean":"2024-04-02T04:51:02.748542+0000","last_became_active":"2024-04-02T04:50:27.414906+0000","last_became_peered":"2024-04-02T04:50:27.414906+0000","last_unstale":"2024-04-02T04:51:02.748542+0000","last_undegraded":"2024-04-02T04:51:02.748542+0000","last_fullsized":"2024-04-02T04:51:02.748542+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:29.974664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":16,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"165.2","version":"195'3","reported_seq":87,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.736733+0000","last_change":"2024-04-02T04:50:34.691854+0000","last_active":"2024-04-02T04:51:02.736733+0000","last_peered":"2024-04-02T04:51:02.736733+0000","last_clean":"2024-04-02T04:50:26.335590+0000","last_became_active":"2024-04-02T04:50:27.440268+0000","last_became_peered":"2024-04-02T04:50:27.440268+0000","last_unstale":"2024-04-02T04:51:02.736733+0000","last_undegraded":"2024-04-02T04:50:34.690842+0000","last_fullsized":"2024-04-02T04:50:27.357466+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.335590+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-04-02T04:51:03.226866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"165.3","version":"196'8","reported_seq":105,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731397+0000","last_change":"2024-04-02T04:50:36.233432+0000","last_active":"2024-04-02T04:51:02.731397+0000","last_peered":"2024-04-02T04:51:02.731397+0000","last_clean":"2024-04-02T04:51:02.731397+0000","last_became_active":"2024-04-02T04:50:27.434796+0000","last_became_peered":"2024-04-02T04:50:27.434796+0000","last_unstale":"2024-04-02T04:51:02.731397+0000","last_undegraded":"2024-04-02T04:51:02.731397+0000","last_fullsized":"2024-04-02T04:51:02.731397+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'8","last_scrub_stamp":"2024-04-02T04:50:36.233128+0000","last_deep_scrub":"196'8","last_deep_scrub_stamp":"2024-04-02T04:50:36.233128+0000","last_clean_scrub_stamp":"2024-04-02T04:50:36.233128+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:05.857429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"165.0","version":"197'16","reported_seq":114,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712608+0000","last_change":"2024-04-02T04:50:36.613240+0000","last_active":"2024-04-02T04:51:02.712608+0000","last_peered":"2024-04-02T04:51:02.712608+0000","last_clean":"2024-04-02T04:51:02.712608+0000","last_became_active":"2024-04-02T04:50:27.421222+0000","last_became_peered":"2024-04-02T04:50:27.421222+0000","last_unstale":"2024-04-02T04:51:02.712608+0000","last_undegraded":"2024-04-02T04:51:02.712608+0000","last_fullsized":"2024-04-02T04:51:02.712608+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'16","last_scrub_stamp":"2024-04-02T04:50:36.613141+0000","last_deep_scrub":"197'16","last_deep_scrub_stamp":"2024-04-02T04:50:36.613141+0000","last_clean_scrub_stamp":"2024-04-02T04:50:36.613141+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:05.085189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"165.1","version":"195'5","reported_seq":101,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.734544+0000","last_change":"2024-04-02T04:50:35.452507+0000","last_active":"2024-04-02T04:51:02.734544+0000","last_peered":"2024-04-02T04:51:02.734544+0000","last_clean":"2024-04-02T04:51:02.734544+0000","last_became_active":"2024-04-02T04:50:27.395102+0000","last_became_peered":"2024-04-02T04:50:27.395102+0000","last_unstale":"2024-04-02T04:51:02.734544+0000","last_undegraded":"2024-04-02T04:51:02.734544+0000","last_fullsized":"2024-04-02T04:51:02.734544+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'5","last_scrub_stamp":"2024-04-02T04:50:35.452403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:35.452403+0000","objects_scrubbed":5,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:38.988323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"165.6","version":"198'10","reported_seq":106,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.716711+0000","last_change":"2024-04-02T04:50:37.598633+0000","last_active":"2024-04-02T04:51:02.716711+0000","last_peered":"2024-04-02T04:51:02.716711+0000","last_clean":"2024-04-02T04:51:02.716711+0000","last_became_active":"2024-04-02T04:50:27.428019+0000","last_became_peered":"2024-04-02T04:50:27.428019+0000","last_unstale":"2024-04-02T04:51:02.716711+0000","last_undegraded":"2024-04-02T04:51:02.716711+0000","last_fullsized":"2024-04-02T04:51:02.716711+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10","last_scrub_stamp":"2024-04-02T04:50:37.598521+0000","last_deep_scrub":"198'10","last_deep_scrub_stamp":"2024-04-02T04:50:37.598521+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.598521+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:43.764757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"165.7","version":"195'2","reported_seq":97,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.757120+0000","last_change":"2024-04-02T04:50:35.589507+0000","last_active":"2024-04-02T04:51:02.757120+0000","last_peered":"2024-04-02T04:51:02.757120+0000","last_clean":"2024-04-02T04:51:02.757120+0000","last_became_active":"2024-04-02T04:50:27.443081+0000","last_became_peered":"2024-04-02T04:50:27.443081+0000","last_unstale":"2024-04-02T04:51:02.757120+0000","last_undegraded":"2024-04-02T04:51:02.757120+0000","last_fullsized":"2024-04-02T04:51:02.757120+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'2","last_scrub_stamp":"2024-04-02T04:50:35.589292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:35.589292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:00.797354+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"165.4","version":"0'0","reported_seq":93,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.733089+0000","last_change":"2024-04-02T04:50:37.276278+0000","last_active":"2024-04-02T04:51:02.733089+0000","last_peered":"2024-04-02T04:51:02.733089+0000","last_clean":"2024-04-02T04:51:02.733089+0000","last_became_active":"2024-04-02T04:50:27.436487+0000","last_became_peered":"2024-04-02T04:50:27.436487+0000","last_unstale":"2024-04-02T04:51:02.733089+0000","last_undegraded":"2024-04-02T04:51:02.733089+0000","last_fullsized":"2024-04-02T04:51:02.733089+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.276172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.276172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:57.895131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"165.5","version":"195'11","reported_seq":113,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727978+0000","last_change":"2024-04-02T04:50:36.081778+0000","last_active":"2024-04-02T04:51:02.727978+0000","last_peered":"2024-04-02T04:51:02.727978+0000","last_clean":"2024-04-02T04:51:02.727978+0000","last_became_active":"2024-04-02T04:50:27.455179+0000","last_became_peered":"2024-04-02T04:50:27.455179+0000","last_unstale":"2024-04-02T04:51:02.727978+0000","last_undegraded":"2024-04-02T04:51:02.727978+0000","last_fullsized":"2024-04-02T04:51:02.727978+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-04-02T04:50:36.081682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:26.335590+0000","last_clean_scrub_stamp":"2024-04-02T04:50:36.081682+0000","objects_scrubbed":11,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:43.344360+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.19","version":"192'45","reported_seq":193,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.733536+0000","last_change":"2024-04-02T04:50:27.532331+0000","last_active":"2024-04-02T04:51:02.733536+0000","last_peered":"2024-04-02T04:51:02.733536+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.255616+0000","last_became_peered":"2024-04-02T04:49:55.255616+0000","last_unstale":"2024-04-02T04:51:02.733536+0000","last_undegraded":"2024-04-02T04:50:27.523866+0000","last_fullsized":"2024-04-02T04:49:55.215309+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:21.822455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":25}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.18","version":"192'18","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730745+0000","last_change":"2024-04-02T04:50:22.505340+0000","last_active":"2024-04-02T04:51:02.730745+0000","last_peered":"2024-04-02T04:51:02.730745+0000","last_clean":"2024-04-02T04:51:02.730745+0000","last_became_active":"2024-04-02T04:49:56.572185+0000","last_became_peered":"2024-04-02T04:49:56.572185+0000","last_unstale":"2024-04-02T04:51:02.730745+0000","last_undegraded":"2024-04-02T04:51:02.730745+0000","last_fullsized":"2024-04-02T04:51:02.730745+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:17.799041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058825730000000003,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1b","version":"192'19","reported_seq":181,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731534+0000","last_change":"2024-04-02T04:50:22.518367+0000","last_active":"2024-04-02T04:51:02.731534+0000","last_peered":"2024-04-02T04:51:02.731534+0000","last_clean":"2024-04-02T04:51:02.731534+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:51:02.731534+0000","last_undegraded":"2024-04-02T04:51:02.731534+0000","last_fullsized":"2024-04-02T04:51:02.731534+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:08.916831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083027329999999996,"stat_sum":{"num_bytes":108,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1a","version":"192'36","reported_seq":187,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.737019+0000","last_change":"2024-04-02T04:50:22.531972+0000","last_active":"2024-04-02T04:51:02.737019+0000","last_peered":"2024-04-02T04:51:02.737019+0000","last_clean":"2024-04-02T04:51:02.737019+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:51:02.737019+0000","last_undegraded":"2024-04-02T04:51:02.737019+0000","last_fullsized":"2024-04-02T04:51:02.737019+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042805730000000002,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1d","version":"192'45","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.719902+0000","last_change":"2024-04-02T04:50:22.473542+0000","last_active":"2024-04-02T04:51:02.719902+0000","last_peered":"2024-04-02T04:51:02.719902+0000","last_clean":"2024-04-02T04:51:02.719902+0000","last_became_active":"2024-04-02T04:49:56.582887+0000","last_became_peered":"2024-04-02T04:49:56.582887+0000","last_unstale":"2024-04-02T04:51:02.719902+0000","last_undegraded":"2024-04-02T04:51:02.719902+0000","last_fullsized":"2024-04-02T04:51:02.719902+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:20.300185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049525999999999997,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1c","version":"192'18","reported_seq":180,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728945+0000","last_change":"2024-04-02T04:50:22.559788+0000","last_active":"2024-04-02T04:51:02.728945+0000","last_peered":"2024-04-02T04:51:02.728945+0000","last_clean":"2024-04-02T04:51:02.728945+0000","last_became_active":"2024-04-02T04:49:56.570832+0000","last_became_peered":"2024-04-02T04:49:56.570832+0000","last_unstale":"2024-04-02T04:51:02.728945+0000","last_undegraded":"2024-04-02T04:51:02.728945+0000","last_fullsized":"2024-04-02T04:51:02.728945+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.635056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014858387000000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1f","version":"192'18","reported_seq":181,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.757326+0000","last_change":"2024-04-02T04:50:22.492987+0000","last_active":"2024-04-02T04:51:02.757326+0000","last_peered":"2024-04-02T04:51:02.757326+0000","last_clean":"2024-04-02T04:51:02.757326+0000","last_became_active":"2024-04-02T04:49:55.249289+0000","last_became_peered":"2024-04-02T04:49:55.249289+0000","last_unstale":"2024-04-02T04:51:02.757326+0000","last_undegraded":"2024-04-02T04:51:02.757326+0000","last_fullsized":"2024-04-02T04:51:02.757326+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.154349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025262501999999999,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1e","version":"192'9","reported_seq":167,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.720929+0000","last_change":"2024-04-02T04:50:22.555071+0000","last_active":"2024-04-02T04:51:02.720929+0000","last_peered":"2024-04-02T04:51:02.720929+0000","last_clean":"2024-04-02T04:51:02.720929+0000","last_became_active":"2024-04-02T04:49:56.563968+0000","last_became_peered":"2024-04-02T04:49:56.563968+0000","last_unstale":"2024-04-02T04:51:02.720929+0000","last_undegraded":"2024-04-02T04:51:02.720929+0000","last_fullsized":"2024-04-02T04:51:02.720929+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:14.841662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026877908999999998,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.11","version":"192'48","reported_seq":209,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.724252+0000","last_change":"2024-04-02T04:50:18.405897+0000","last_active":"2024-04-02T04:51:02.724252+0000","last_peered":"2024-04-02T04:51:02.724252+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.569001+0000","last_became_peered":"2024-04-02T04:49:56.569001+0000","last_unstale":"2024-04-02T04:51:02.724252+0000","last_undegraded":"2024-04-02T04:50:17.425678+0000","last_fullsized":"2024-04-02T04:49:56.526356+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273,"num_objects":27,"num_object_clones":21,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":27}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.10","version":"192'18","reported_seq":174,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.716972+0000","last_change":"2024-04-02T04:50:27.548680+0000","last_active":"2024-04-02T04:51:02.716972+0000","last_peered":"2024-04-02T04:51:02.716972+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.243496+0000","last_became_peered":"2024-04-02T04:49:55.243496+0000","last_unstale":"2024-04-02T04:51:02.716972+0000","last_undegraded":"2024-04-02T04:50:27.476730+0000","last_fullsized":"2024-04-02T04:49:55.211603+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:17.367368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.13","version":"192'48","reported_seq":196,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.731647+0000","last_change":"2024-04-02T04:50:18.418843+0000","last_active":"2024-04-02T04:51:02.731647+0000","last_peered":"2024-04-02T04:51:02.731647+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.554860+0000","last_became_peered":"2024-04-02T04:49:56.554860+0000","last_unstale":"2024-04-02T04:51:02.731647+0000","last_undegraded":"2024-04-02T04:50:17.426824+0000","last_fullsized":"2024-04-02T04:49:56.518531+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.895952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":27,"num_object_clones":21,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":27}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.12","version":"192'9","reported_seq":172,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.720629+0000","last_change":"2024-04-02T04:50:22.490795+0000","last_active":"2024-04-02T04:51:02.720629+0000","last_peered":"2024-04-02T04:51:02.720629+0000","last_clean":"2024-04-02T04:51:02.720629+0000","last_became_active":"2024-04-02T04:49:55.255700+0000","last_became_peered":"2024-04-02T04:49:55.255700+0000","last_unstale":"2024-04-02T04:51:02.720629+0000","last_undegraded":"2024-04-02T04:51:02.720629+0000","last_fullsized":"2024-04-02T04:51:02.720629+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054572539999999999,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.15","version":"192'37","reported_seq":190,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.739156+0000","last_change":"2024-04-02T04:50:09.415659+0000","last_active":"2024-04-02T04:51:02.739156+0000","last_peered":"2024-04-02T04:51:02.739156+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.561896+0000","last_became_peered":"2024-04-02T04:49:56.561896+0000","last_unstale":"2024-04-02T04:51:02.739156+0000","last_undegraded":"2024-04-02T04:50:08.454406+0000","last_fullsized":"2024-04-02T04:49:56.542252+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:08.340536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.14","version":"192'81","reported_seq":242,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.737720+0000","last_change":"2024-04-02T04:50:22.458516+0000","last_active":"2024-04-02T04:51:02.737720+0000","last_peered":"2024-04-02T04:51:02.737720+0000","last_clean":"2024-04-02T04:51:02.737720+0000","last_became_active":"2024-04-02T04:49:55.269267+0000","last_became_peered":"2024-04-02T04:49:55.269267+0000","last_unstale":"2024-04-02T04:51:02.737720+0000","last_undegraded":"2024-04-02T04:51:02.737720+0000","last_fullsized":"2024-04-02T04:51:02.737720+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.178388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037804760000000001,"stat_sum":{"num_bytes":468,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":45,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.17","version":"192'18","reported_seq":182,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.723820+0000","last_change":"2024-04-02T04:50:22.556038+0000","last_active":"2024-04-02T04:51:02.723820+0000","last_peered":"2024-04-02T04:51:02.723820+0000","last_clean":"2024-04-02T04:51:02.723820+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:51:02.723820+0000","last_undegraded":"2024-04-02T04:51:02.723820+0000","last_fullsized":"2024-04-02T04:51:02.723820+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:05.048804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015217125,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.16","version":"192'9","reported_seq":170,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.743238+0000","last_change":"2024-04-02T04:50:22.465703+0000","last_active":"2024-04-02T04:51:02.743238+0000","last_peered":"2024-04-02T04:51:02.743238+0000","last_clean":"2024-04-02T04:51:02.743238+0000","last_became_active":"2024-04-02T04:49:55.269224+0000","last_became_peered":"2024-04-02T04:49:55.269224+0000","last_unstale":"2024-04-02T04:51:02.743238+0000","last_undegraded":"2024-04-02T04:51:02.743238+0000","last_fullsized":"2024-04-02T04:51:02.743238+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078454270000000003,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.9","version":"192'18","reported_seq":169,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.723722+0000","last_change":"2024-04-02T04:50:27.575689+0000","last_active":"2024-04-02T04:51:02.723722+0000","last_peered":"2024-04-02T04:51:02.723722+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.562137+0000","last_became_peered":"2024-04-02T04:49:56.562137+0000","last_unstale":"2024-04-02T04:51:02.723722+0000","last_undegraded":"2024-04-02T04:50:27.479688+0000","last_fullsized":"2024-04-02T04:49:56.524836+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.592133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.8","version":"192'9","reported_seq":172,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.759696+0000","last_change":"2024-04-02T04:50:22.511423+0000","last_active":"2024-04-02T04:51:02.759696+0000","last_peered":"2024-04-02T04:51:02.759696+0000","last_clean":"2024-04-02T04:51:02.759696+0000","last_became_active":"2024-04-02T04:49:55.259553+0000","last_became_peered":"2024-04-02T04:49:55.259553+0000","last_unstale":"2024-04-02T04:51:02.759696+0000","last_undegraded":"2024-04-02T04:51:02.759696+0000","last_fullsized":"2024-04-02T04:51:02.759696+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.892233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012597687999999999,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.b","version":"192'36","reported_seq":190,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.743917+0000","last_change":"2024-04-02T04:50:22.478126+0000","last_active":"2024-04-02T04:51:02.743917+0000","last_peered":"2024-04-02T04:51:02.743917+0000","last_clean":"2024-04-02T04:51:02.743917+0000","last_became_active":"2024-04-02T04:49:55.267102+0000","last_became_peered":"2024-04-02T04:49:55.267102+0000","last_unstale":"2024-04-02T04:51:02.743917+0000","last_undegraded":"2024-04-02T04:51:02.743917+0000","last_fullsized":"2024-04-02T04:51:02.743917+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01285824,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.a","version":"192'36","reported_seq":194,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.736328+0000","last_change":"2024-04-02T04:50:22.479090+0000","last_active":"2024-04-02T04:51:02.736328+0000","last_peered":"2024-04-02T04:51:02.736328+0000","last_clean":"2024-04-02T04:51:02.736328+0000","last_became_active":"2024-04-02T04:49:55.252341+0000","last_became_peered":"2024-04-02T04:49:55.252341+0000","last_unstale":"2024-04-02T04:51:02.736328+0000","last_undegraded":"2024-04-02T04:51:02.736328+0000","last_fullsized":"2024-04-02T04:51:02.736328+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:14.334043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058763110000000004,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.d","version":"192'63","reported_seq":212,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.728108+0000","last_change":"2024-04-02T04:50:27.508913+0000","last_active":"2024-04-02T04:51:02.728108+0000","last_peered":"2024-04-02T04:51:02.728108+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.566448+0000","last_became_peered":"2024-04-02T04:49:56.566448+0000","last_unstale":"2024-04-02T04:51:02.728108+0000","last_undegraded":"2024-04-02T04:50:27.492108+0000","last_fullsized":"2024-04-02T04:49:56.525265+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":35}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.c","version":"192'36","reported_seq":195,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.759078+0000","last_change":"2024-04-02T04:50:22.514035+0000","last_active":"2024-04-02T04:51:02.759078+0000","last_peered":"2024-04-02T04:51:02.759078+0000","last_clean":"2024-04-02T04:51:02.759078+0000","last_became_active":"2024-04-02T04:49:55.252273+0000","last_became_peered":"2024-04-02T04:49:55.252273+0000","last_unstale":"2024-04-02T04:51:02.759078+0000","last_undegraded":"2024-04-02T04:51:02.759078+0000","last_fullsized":"2024-04-02T04:51:02.759078+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:10.784572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011904270999999999,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.f","version":"192'23","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.765364+0000","last_change":"2024-04-02T04:50:24.531979+0000","last_active":"2024-04-02T04:51:02.765364+0000","last_peered":"2024-04-02T04:51:02.765364+0000","last_clean":"2024-04-02T04:51:02.765364+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:51:02.765364+0000","last_undegraded":"2024-04-02T04:51:02.765364+0000","last_fullsized":"2024-04-02T04:51:02.765364+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:14.147702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.54930396999999997,"stat_sum":{"num_bytes":109,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.e","version":"192'32","reported_seq":196,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.740095+0000","last_change":"2024-04-02T04:50:22.556197+0000","last_active":"2024-04-02T04:51:02.740095+0000","last_peered":"2024-04-02T04:51:02.740095+0000","last_clean":"2024-04-02T04:51:02.740095+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:51:02.740095+0000","last_undegraded":"2024-04-02T04:51:02.740095+0000","last_fullsized":"2024-04-02T04:51:02.740095+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.261436+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.048399875000000002,"stat_sum":{"num_bytes":161,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.1","version":"192'27","reported_seq":179,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.734230+0000","last_change":"2024-04-02T04:50:27.514211+0000","last_active":"2024-04-02T04:51:02.734230+0000","last_peered":"2024-04-02T04:51:02.734230+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.245537+0000","last_became_peered":"2024-04-02T04:49:55.245537+0000","last_unstale":"2024-04-02T04:51:02.734230+0000","last_undegraded":"2024-04-02T04:50:27.498272+0000","last_fullsized":"2024-04-02T04:49:55.215487+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:03.855460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":15}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.0","version":"192'45","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.735571+0000","last_change":"2024-04-02T04:50:22.510383+0000","last_active":"2024-04-02T04:51:02.735571+0000","last_peered":"2024-04-02T04:51:02.735571+0000","last_clean":"2024-04-02T04:51:02.735571+0000","last_became_active":"2024-04-02T04:49:55.268700+0000","last_became_peered":"2024-04-02T04:49:55.268700+0000","last_unstale":"2024-04-02T04:51:02.735571+0000","last_undegraded":"2024-04-02T04:51:02.735571+0000","last_fullsized":"2024-04-02T04:51:02.735571+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:11.091967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019278297,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.3","version":"192'18","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730426+0000","last_change":"2024-04-02T04:50:22.482285+0000","last_active":"2024-04-02T04:51:02.730426+0000","last_peered":"2024-04-02T04:51:02.730426+0000","last_clean":"2024-04-02T04:51:02.730426+0000","last_became_active":"2024-04-02T04:49:55.257987+0000","last_became_peered":"2024-04-02T04:49:55.257987+0000","last_unstale":"2024-04-02T04:51:02.730426+0000","last_undegraded":"2024-04-02T04:51:02.730426+0000","last_fullsized":"2024-04-02T04:51:02.730426+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:05.899667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054451289999999999,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.2","version":"192'18","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.720748+0000","last_change":"2024-04-02T04:50:22.491445+0000","last_active":"2024-04-02T04:51:02.720748+0000","last_peered":"2024-04-02T04:51:02.720748+0000","last_clean":"2024-04-02T04:51:02.720748+0000","last_became_active":"2024-04-02T04:49:55.235417+0000","last_became_peered":"2024-04-02T04:49:55.235417+0000","last_unstale":"2024-04-02T04:51:02.720748+0000","last_undegraded":"2024-04-02T04:51:02.720748+0000","last_fullsized":"2024-04-02T04:51:02.720748+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058798909999999999,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.5","version":"192'45","reported_seq":202,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.735561+0000","last_change":"2024-04-02T04:50:22.537559+0000","last_active":"2024-04-02T04:51:02.735561+0000","last_peered":"2024-04-02T04:51:02.735561+0000","last_clean":"2024-04-02T04:51:02.735561+0000","last_became_active":"2024-04-02T04:49:56.553311+0000","last_became_peered":"2024-04-02T04:49:56.553311+0000","last_unstale":"2024-04-02T04:51:02.735561+0000","last_undegraded":"2024-04-02T04:51:02.735561+0000","last_fullsized":"2024-04-02T04:51:02.735561+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.013140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022060244999999999,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.4","version":"192'9","reported_seq":160,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.745324+0000","last_change":"2024-04-02T04:50:28.406787+0000","last_active":"2024-04-02T04:51:02.745324+0000","last_peered":"2024-04-02T04:51:02.745324+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.578685+0000","last_became_peered":"2024-04-02T04:49:56.578685+0000","last_unstale":"2024-04-02T04:51:02.745324+0000","last_undegraded":"2024-04-02T04:50:27.585397+0000","last_fullsized":"2024-04-02T04:49:56.549435+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:08.741208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.7","version":"192'9","reported_seq":171,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751346+0000","last_change":"2024-04-02T04:50:22.482989+0000","last_active":"2024-04-02T04:51:02.751346+0000","last_peered":"2024-04-02T04:51:02.751346+0000","last_clean":"2024-04-02T04:51:02.751346+0000","last_became_active":"2024-04-02T04:49:55.261913+0000","last_became_peered":"2024-04-02T04:49:55.261913+0000","last_unstale":"2024-04-02T04:51:02.751346+0000","last_undegraded":"2024-04-02T04:51:02.751346+0000","last_fullsized":"2024-04-02T04:51:02.751346+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:09.387310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044479259999999996,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.6","version":"192'18","reported_seq":179,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717114+0000","last_change":"2024-04-02T04:50:22.479243+0000","last_active":"2024-04-02T04:51:02.717114+0000","last_peered":"2024-04-02T04:51:02.717114+0000","last_clean":"2024-04-02T04:51:02.717114+0000","last_became_active":"2024-04-02T04:49:56.581977+0000","last_became_peered":"2024-04-02T04:49:56.581977+0000","last_unstale":"2024-04-02T04:51:02.717114+0000","last_undegraded":"2024-04-02T04:51:02.717114+0000","last_fullsized":"2024-04-02T04:51:02.717114+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025991510000000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"43.14","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.719697+0000","last_change":"2024-04-02T04:49:29.828822+0000","last_active":"2024-04-02T04:51:02.719697+0000","last_peered":"2024-04-02T04:51:02.719697+0000","last_clean":"2024-04-02T04:51:02.719697+0000","last_became_active":"2024-04-02T04:49:29.828535+0000","last_became_peered":"2024-04-02T04:49:29.828535+0000","last_unstale":"2024-04-02T04:51:02.719697+0000","last_undegraded":"2024-04-02T04:51:02.719697+0000","last_fullsized":"2024-04-02T04:51:02.719697+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.746550+0000","last_change":"2024-04-02T04:49:47.428191+0000","last_active":"2024-04-02T04:51:02.746550+0000","last_peered":"2024-04-02T04:51:02.746550+0000","last_clean":"2024-04-02T04:49:45.347176+0000","last_became_active":"2024-04-02T04:49:47.428191+0000","last_became_peered":"2024-04-02T04:49:47.428191+0000","last_unstale":"2024-04-02T04:51:02.746550+0000","last_undegraded":"2024-04-02T04:51:02.746550+0000","last_fullsized":"2024-04-02T04:49:47.367526+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.758164+0000","last_change":"2024-04-02T04:49:28.771791+0000","last_active":"2024-04-02T04:51:02.758164+0000","last_peered":"2024-04-02T04:51:02.758164+0000","last_clean":"2024-04-02T04:51:02.758164+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:51:02.758164+0000","last_undegraded":"2024-04-02T04:51:02.758164+0000","last_fullsized":"2024-04-02T04:51:02.758164+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717380+0000","last_change":"2024-04-02T04:49:28.787801+0000","last_active":"2024-04-02T04:51:02.717380+0000","last_peered":"2024-04-02T04:51:02.717380+0000","last_clean":"2024-04-02T04:51:02.717380+0000","last_became_active":"2024-04-02T04:49:28.787404+0000","last_became_peered":"2024-04-02T04:49:28.787404+0000","last_unstale":"2024-04-02T04:51:02.717380+0000","last_undegraded":"2024-04-02T04:51:02.717380+0000","last_fullsized":"2024-04-02T04:51:02.717380+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.758062+0000","last_change":"2024-04-02T04:49:28.787925+0000","last_active":"2024-04-02T04:51:02.758062+0000","last_peered":"2024-04-02T04:51:02.758062+0000","last_clean":"2024-04-02T04:51:02.758062+0000","last_became_active":"2024-04-02T04:49:28.783594+0000","last_became_peered":"2024-04-02T04:49:28.783594+0000","last_unstale":"2024-04-02T04:51:02.758062+0000","last_undegraded":"2024-04-02T04:51:02.758062+0000","last_fullsized":"2024-04-02T04:51:02.758062+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741599+0000","last_change":"2024-04-02T04:49:28.726311+0000","last_active":"2024-04-02T04:51:02.741599+0000","last_peered":"2024-04-02T04:51:02.741599+0000","last_clean":"2024-04-02T04:51:02.741599+0000","last_became_active":"2024-04-02T04:49:28.726013+0000","last_became_peered":"2024-04-02T04:49:28.726013+0000","last_unstale":"2024-04-02T04:51:02.741599+0000","last_undegraded":"2024-04-02T04:51:02.741599+0000","last_fullsized":"2024-04-02T04:51:02.741599+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":180,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.721628+0000","last_change":"2024-04-02T04:49:47.479797+0000","last_active":"2024-04-02T04:51:02.721628+0000","last_peered":"2024-04-02T04:51:02.721628+0000","last_clean":"2024-04-02T04:49:45.330040+0000","last_became_active":"2024-04-02T04:49:47.479797+0000","last_became_peered":"2024-04-02T04:49:47.479797+0000","last_unstale":"2024-04-02T04:51:02.721628+0000","last_undegraded":"2024-04-02T04:51:02.721628+0000","last_fullsized":"2024-04-02T04:49:47.405691+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.740084+0000","last_change":"2024-04-02T04:49:28.759806+0000","last_active":"2024-04-02T04:51:02.740084+0000","last_peered":"2024-04-02T04:51:02.740084+0000","last_clean":"2024-04-02T04:51:02.740084+0000","last_became_active":"2024-04-02T04:49:28.759334+0000","last_became_peered":"2024-04-02T04:49:28.759334+0000","last_unstale":"2024-04-02T04:51:02.740084+0000","last_undegraded":"2024-04-02T04:51:02.740084+0000","last_fullsized":"2024-04-02T04:51:02.740084+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.748307+0000","last_change":"2024-04-02T04:49:47.517914+0000","last_active":"2024-04-02T04:51:02.748307+0000","last_peered":"2024-04-02T04:51:02.748307+0000","last_clean":"2024-04-02T04:49:45.379252+0000","last_became_active":"2024-04-02T04:49:47.517914+0000","last_became_peered":"2024-04-02T04:49:47.517914+0000","last_unstale":"2024-04-02T04:51:02.748307+0000","last_undegraded":"2024-04-02T04:51:02.748307+0000","last_fullsized":"2024-04-02T04:49:47.408049+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.720801+0000","last_change":"2024-04-02T04:49:47.477571+0000","last_active":"2024-04-02T04:51:02.720801+0000","last_peered":"2024-04-02T04:51:02.720801+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.477571+0000","last_became_peered":"2024-04-02T04:49:47.477571+0000","last_unstale":"2024-04-02T04:51:02.720801+0000","last_undegraded":"2024-04-02T04:51:02.720801+0000","last_fullsized":"2024-04-02T04:49:47.404962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.741711+0000","last_change":"2024-04-02T04:49:47.517552+0000","last_active":"2024-04-02T04:51:02.741711+0000","last_peered":"2024-04-02T04:51:02.741711+0000","last_clean":"2024-04-02T04:49:45.378670+0000","last_became_active":"2024-04-02T04:49:47.517552+0000","last_became_peered":"2024-04-02T04:49:47.517552+0000","last_unstale":"2024-04-02T04:51:02.741711+0000","last_undegraded":"2024-04-02T04:51:02.741711+0000","last_fullsized":"2024-04-02T04:49:47.406289+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.753398+0000","last_change":"2024-04-02T04:49:28.761881+0000","last_active":"2024-04-02T04:51:02.753398+0000","last_peered":"2024-04-02T04:51:02.753398+0000","last_clean":"2024-04-02T04:51:02.753398+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:51:02.753398+0000","last_undegraded":"2024-04-02T04:51:02.753398+0000","last_fullsized":"2024-04-02T04:51:02.753398+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.892089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712367+0000","last_change":"2024-04-02T04:49:29.819002+0000","last_active":"2024-04-02T04:51:02.712367+0000","last_peered":"2024-04-02T04:51:02.712367+0000","last_clean":"2024-04-02T04:51:02.712367+0000","last_became_active":"2024-04-02T04:49:29.818524+0000","last_became_peered":"2024-04-02T04:49:29.818524+0000","last_unstale":"2024-04-02T04:51:02.712367+0000","last_undegraded":"2024-04-02T04:51:02.712367+0000","last_fullsized":"2024-04-02T04:51:02.712367+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.732762+0000","last_change":"2024-04-02T04:49:28.769390+0000","last_active":"2024-04-02T04:51:02.732762+0000","last_peered":"2024-04-02T04:51:02.732762+0000","last_clean":"2024-04-02T04:51:02.732762+0000","last_became_active":"2024-04-02T04:49:28.769072+0000","last_became_peered":"2024-04-02T04:49:28.769072+0000","last_unstale":"2024-04-02T04:51:02.732762+0000","last_undegraded":"2024-04-02T04:51:02.732762+0000","last_fullsized":"2024-04-02T04:51:02.732762+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.715242+0000","last_change":"2024-04-02T04:49:29.802333+0000","last_active":"2024-04-02T04:51:02.715242+0000","last_peered":"2024-04-02T04:51:02.715242+0000","last_clean":"2024-04-02T04:51:02.715242+0000","last_became_active":"2024-04-02T04:49:29.801654+0000","last_became_peered":"2024-04-02T04:49:29.801654+0000","last_unstale":"2024-04-02T04:51:02.715242+0000","last_undegraded":"2024-04-02T04:51:02.715242+0000","last_fullsized":"2024-04-02T04:51:02.715242+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748050+0000","last_change":"2024-04-02T04:49:28.760456+0000","last_active":"2024-04-02T04:51:02.748050+0000","last_peered":"2024-04-02T04:51:02.748050+0000","last_clean":"2024-04-02T04:51:02.748050+0000","last_became_active":"2024-04-02T04:49:28.756548+0000","last_became_peered":"2024-04-02T04:49:28.756548+0000","last_unstale":"2024-04-02T04:51:02.748050+0000","last_undegraded":"2024-04-02T04:51:02.748050+0000","last_fullsized":"2024-04-02T04:51:02.748050+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.746036+0000","last_change":"2024-04-02T04:49:28.741572+0000","last_active":"2024-04-02T04:51:02.746036+0000","last_peered":"2024-04-02T04:51:02.746036+0000","last_clean":"2024-04-02T04:51:02.746036+0000","last_became_active":"2024-04-02T04:49:28.741277+0000","last_became_peered":"2024-04-02T04:49:28.741277+0000","last_unstale":"2024-04-02T04:51:02.746036+0000","last_undegraded":"2024-04-02T04:51:02.746036+0000","last_fullsized":"2024-04-02T04:51:02.746036+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727560+0000","last_change":"2024-04-02T04:49:28.740669+0000","last_active":"2024-04-02T04:51:02.727560+0000","last_peered":"2024-04-02T04:51:02.727560+0000","last_clean":"2024-04-02T04:51:02.727560+0000","last_became_active":"2024-04-02T04:49:28.719086+0000","last_became_peered":"2024-04-02T04:49:28.719086+0000","last_unstale":"2024-04-02T04:51:02.727560+0000","last_undegraded":"2024-04-02T04:51:02.727560+0000","last_fullsized":"2024-04-02T04:51:02.727560+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728567+0000","last_change":"2024-04-02T04:49:28.743542+0000","last_active":"2024-04-02T04:51:02.728567+0000","last_peered":"2024-04-02T04:51:02.728567+0000","last_clean":"2024-04-02T04:51:02.728567+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:51:02.728567+0000","last_undegraded":"2024-04-02T04:51:02.728567+0000","last_fullsized":"2024-04-02T04:51:02.728567+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.723778+0000","last_change":"2024-04-02T04:49:28.732308+0000","last_active":"2024-04-02T04:51:02.723778+0000","last_peered":"2024-04-02T04:51:02.723778+0000","last_clean":"2024-04-02T04:51:02.723778+0000","last_became_active":"2024-04-02T04:49:28.731836+0000","last_became_peered":"2024-04-02T04:49:28.731836+0000","last_unstale":"2024-04-02T04:51:02.723778+0000","last_undegraded":"2024-04-02T04:51:02.723778+0000","last_fullsized":"2024-04-02T04:51:02.723778+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731434+0000","last_change":"2024-04-02T04:50:39.108210+0000","last_active":"2024-04-02T04:51:02.731434+0000","last_peered":"2024-04-02T04:51:02.731434+0000","last_clean":"2024-04-02T04:51:02.731434+0000","last_became_active":"2024-04-02T04:49:28.727703+0000","last_became_peered":"2024-04-02T04:49:28.727703+0000","last_unstale":"2024-04-02T04:51:02.731434+0000","last_undegraded":"2024-04-02T04:51:02.731434+0000","last_fullsized":"2024-04-02T04:51:02.731434+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.108133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.108133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:51.097133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.732179+0000","last_change":"2024-04-02T04:49:47.517107+0000","last_active":"2024-04-02T04:51:02.732179+0000","last_peered":"2024-04-02T04:51:02.732179+0000","last_clean":"2024-04-02T04:49:45.381150+0000","last_became_active":"2024-04-02T04:49:47.517107+0000","last_became_peered":"2024-04-02T04:49:47.517107+0000","last_unstale":"2024-04-02T04:51:02.732179+0000","last_undegraded":"2024-04-02T04:51:02.732179+0000","last_fullsized":"2024-04-02T04:49:47.429362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.735632+0000","last_change":"2024-04-02T04:49:47.444375+0000","last_active":"2024-04-02T04:51:02.735632+0000","last_peered":"2024-04-02T04:51:02.735632+0000","last_clean":"2024-04-02T04:49:45.331671+0000","last_became_active":"2024-04-02T04:49:47.444375+0000","last_became_peered":"2024-04-02T04:49:47.444375+0000","last_unstale":"2024-04-02T04:51:02.735632+0000","last_undegraded":"2024-04-02T04:51:02.735632+0000","last_fullsized":"2024-04-02T04:49:47.336937+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.745159+0000","last_change":"2024-04-02T04:49:47.381163+0000","last_active":"2024-04-02T04:51:02.745159+0000","last_peered":"2024-04-02T04:51:02.745159+0000","last_clean":"2024-04-02T04:49:45.330129+0000","last_became_active":"2024-04-02T04:49:47.381163+0000","last_became_peered":"2024-04-02T04:49:47.381163+0000","last_unstale":"2024-04-02T04:51:02.745159+0000","last_undegraded":"2024-04-02T04:51:02.745159+0000","last_fullsized":"2024-04-02T04:49:47.338896+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.759367+0000","last_change":"2024-04-02T04:49:28.767169+0000","last_active":"2024-04-02T04:51:02.759367+0000","last_peered":"2024-04-02T04:51:02.759367+0000","last_clean":"2024-04-02T04:51:02.759367+0000","last_became_active":"2024-04-02T04:49:28.762640+0000","last_became_peered":"2024-04-02T04:49:28.762640+0000","last_unstale":"2024-04-02T04:51:02.759367+0000","last_undegraded":"2024-04-02T04:51:02.759367+0000","last_fullsized":"2024-04-02T04:51:02.759367+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.766568+0000","last_change":"2024-04-02T04:49:28.768223+0000","last_active":"2024-04-02T04:51:02.766568+0000","last_peered":"2024-04-02T04:51:02.766568+0000","last_clean":"2024-04-02T04:51:02.766568+0000","last_became_active":"2024-04-02T04:49:28.767868+0000","last_became_peered":"2024-04-02T04:49:28.767868+0000","last_unstale":"2024-04-02T04:51:02.766568+0000","last_undegraded":"2024-04-02T04:51:02.766568+0000","last_fullsized":"2024-04-02T04:51:02.766568+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.751162+0000","last_change":"2024-04-02T04:49:47.508050+0000","last_active":"2024-04-02T04:51:02.751162+0000","last_peered":"2024-04-02T04:51:02.751162+0000","last_clean":"2024-04-02T04:49:45.381990+0000","last_became_active":"2024-04-02T04:49:47.508050+0000","last_became_peered":"2024-04-02T04:49:47.508050+0000","last_unstale":"2024-04-02T04:51:02.751162+0000","last_undegraded":"2024-04-02T04:51:02.751162+0000","last_fullsized":"2024-04-02T04:49:47.429001+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.738383+0000","last_change":"2024-04-02T04:49:47.490495+0000","last_active":"2024-04-02T04:51:02.738383+0000","last_peered":"2024-04-02T04:51:02.738383+0000","last_clean":"2024-04-02T04:49:45.351290+0000","last_became_active":"2024-04-02T04:49:47.490495+0000","last_became_peered":"2024-04-02T04:49:47.490495+0000","last_unstale":"2024-04-02T04:51:02.738383+0000","last_undegraded":"2024-04-02T04:51:02.738383+0000","last_fullsized":"2024-04-02T04:49:47.384086+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.742288+0000","last_change":"2024-04-02T04:49:28.720521+0000","last_active":"2024-04-02T04:51:02.742288+0000","last_peered":"2024-04-02T04:51:02.742288+0000","last_clean":"2024-04-02T04:51:02.742288+0000","last_became_active":"2024-04-02T04:49:28.720246+0000","last_became_peered":"2024-04-02T04:49:28.720246+0000","last_unstale":"2024-04-02T04:51:02.742288+0000","last_undegraded":"2024-04-02T04:51:02.742288+0000","last_fullsized":"2024-04-02T04:51:02.742288+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.766735+0000","last_change":"2024-04-02T04:49:28.782201+0000","last_active":"2024-04-02T04:51:02.766735+0000","last_peered":"2024-04-02T04:51:02.766735+0000","last_clean":"2024-04-02T04:51:02.766735+0000","last_became_active":"2024-04-02T04:49:28.781235+0000","last_became_peered":"2024-04-02T04:49:28.781235+0000","last_unstale":"2024-04-02T04:51:02.766735+0000","last_undegraded":"2024-04-02T04:51:02.766735+0000","last_fullsized":"2024-04-02T04:51:02.766735+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.746243+0000","last_change":"2024-04-02T04:49:28.787959+0000","last_active":"2024-04-02T04:51:02.746243+0000","last_peered":"2024-04-02T04:51:02.746243+0000","last_clean":"2024-04-02T04:51:02.746243+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:51:02.746243+0000","last_undegraded":"2024-04-02T04:51:02.746243+0000","last_fullsized":"2024-04-02T04:51:02.746243+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.718180+0000","last_change":"2024-04-02T04:49:47.389948+0000","last_active":"2024-04-02T04:51:02.718180+0000","last_peered":"2024-04-02T04:51:02.718180+0000","last_clean":"2024-04-02T04:49:45.343249+0000","last_became_active":"2024-04-02T04:49:47.389948+0000","last_became_peered":"2024-04-02T04:49:47.389948+0000","last_unstale":"2024-04-02T04:51:02.718180+0000","last_undegraded":"2024-04-02T04:51:02.718180+0000","last_fullsized":"2024-04-02T04:49:47.366723+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717688+0000","last_change":"2024-04-02T04:49:28.750685+0000","last_active":"2024-04-02T04:51:02.717688+0000","last_peered":"2024-04-02T04:51:02.717688+0000","last_clean":"2024-04-02T04:51:02.717688+0000","last_became_active":"2024-04-02T04:49:28.750373+0000","last_became_peered":"2024-04-02T04:49:28.750373+0000","last_unstale":"2024-04-02T04:51:02.717688+0000","last_undegraded":"2024-04-02T04:51:02.717688+0000","last_fullsized":"2024-04-02T04:51:02.717688+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.753624+0000","last_change":"2024-04-02T04:49:47.389543+0000","last_active":"2024-04-02T04:51:02.753624+0000","last_peered":"2024-04-02T04:51:02.753624+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.389543+0000","last_became_peered":"2024-04-02T04:49:47.389543+0000","last_unstale":"2024-04-02T04:51:02.753624+0000","last_undegraded":"2024-04-02T04:51:02.753624+0000","last_fullsized":"2024-04-02T04:49:47.363220+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.755778+0000","last_change":"2024-04-02T04:49:47.391015+0000","last_active":"2024-04-02T04:51:02.755778+0000","last_peered":"2024-04-02T04:51:02.755778+0000","last_clean":"2024-04-02T04:49:45.363593+0000","last_became_active":"2024-04-02T04:49:47.391015+0000","last_became_peered":"2024-04-02T04:49:47.391015+0000","last_unstale":"2024-04-02T04:51:02.755778+0000","last_undegraded":"2024-04-02T04:51:02.755778+0000","last_fullsized":"2024-04-02T04:49:47.362030+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.738306+0000","last_change":"2024-04-02T04:49:47.434747+0000","last_active":"2024-04-02T04:51:02.738306+0000","last_peered":"2024-04-02T04:51:02.738306+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.434747+0000","last_became_peered":"2024-04-02T04:49:47.434747+0000","last_unstale":"2024-04-02T04:51:02.738306+0000","last_undegraded":"2024-04-02T04:51:02.738306+0000","last_fullsized":"2024-04-02T04:49:47.346344+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731037+0000","last_change":"2024-04-02T04:49:28.741181+0000","last_active":"2024-04-02T04:51:02.731037+0000","last_peered":"2024-04-02T04:51:02.731037+0000","last_clean":"2024-04-02T04:51:02.731037+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:51:02.731037+0000","last_undegraded":"2024-04-02T04:51:02.731037+0000","last_fullsized":"2024-04-02T04:51:02.731037+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.716536+0000","last_change":"2024-04-02T04:49:28.787621+0000","last_active":"2024-04-02T04:51:02.716536+0000","last_peered":"2024-04-02T04:51:02.716536+0000","last_clean":"2024-04-02T04:51:02.716536+0000","last_became_active":"2024-04-02T04:49:28.785195+0000","last_became_peered":"2024-04-02T04:49:28.785195+0000","last_unstale":"2024-04-02T04:51:02.716536+0000","last_undegraded":"2024-04-02T04:51:02.716536+0000","last_fullsized":"2024-04-02T04:51:02.716536+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731004+0000","last_change":"2024-04-02T04:49:28.743937+0000","last_active":"2024-04-02T04:51:02.731004+0000","last_peered":"2024-04-02T04:51:02.731004+0000","last_clean":"2024-04-02T04:51:02.731004+0000","last_became_active":"2024-04-02T04:49:28.741450+0000","last_became_peered":"2024-04-02T04:49:28.741450+0000","last_unstale":"2024-04-02T04:51:02.731004+0000","last_undegraded":"2024-04-02T04:51:02.731004+0000","last_fullsized":"2024-04-02T04:51:02.731004+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.714615+0000","last_change":"2024-04-02T04:49:28.731835+0000","last_active":"2024-04-02T04:51:02.714615+0000","last_peered":"2024-04-02T04:51:02.714615+0000","last_clean":"2024-04-02T04:51:02.714615+0000","last_became_active":"2024-04-02T04:49:28.731343+0000","last_became_peered":"2024-04-02T04:49:28.731343+0000","last_unstale":"2024-04-02T04:51:02.714615+0000","last_undegraded":"2024-04-02T04:51:02.714615+0000","last_fullsized":"2024-04-02T04:51:02.714615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730686+0000","last_change":"2024-04-02T04:50:37.102849+0000","last_active":"2024-04-02T04:51:02.730686+0000","last_peered":"2024-04-02T04:51:02.730686+0000","last_clean":"2024-04-02T04:51:02.730686+0000","last_became_active":"2024-04-02T04:49:28.698110+0000","last_became_peered":"2024-04-02T04:49:28.698110+0000","last_unstale":"2024-04-02T04:51:02.730686+0000","last_undegraded":"2024-04-02T04:51:02.730686+0000","last_fullsized":"2024-04-02T04:51:02.730686+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.102732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.102732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:53.585452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.741269+0000","last_change":"2024-04-02T04:49:47.445509+0000","last_active":"2024-04-02T04:51:02.741269+0000","last_peered":"2024-04-02T04:51:02.741269+0000","last_clean":"2024-04-02T04:49:45.370130+0000","last_became_active":"2024-04-02T04:49:47.445509+0000","last_became_peered":"2024-04-02T04:49:47.445509+0000","last_unstale":"2024-04-02T04:51:02.741269+0000","last_undegraded":"2024-04-02T04:51:02.741269+0000","last_fullsized":"2024-04-02T04:49:47.391250+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.716266+0000","last_change":"2024-04-02T04:49:28.748883+0000","last_active":"2024-04-02T04:51:02.716266+0000","last_peered":"2024-04-02T04:51:02.716266+0000","last_clean":"2024-04-02T04:51:02.716266+0000","last_became_active":"2024-04-02T04:49:28.746056+0000","last_became_peered":"2024-04-02T04:49:28.746056+0000","last_unstale":"2024-04-02T04:51:02.716266+0000","last_undegraded":"2024-04-02T04:51:02.716266+0000","last_fullsized":"2024-04-02T04:51:02.716266+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.751985+0000","last_change":"2024-04-02T04:49:47.388801+0000","last_active":"2024-04-02T04:51:02.751985+0000","last_peered":"2024-04-02T04:51:02.751985+0000","last_clean":"2024-04-02T04:49:45.330442+0000","last_became_active":"2024-04-02T04:49:47.388801+0000","last_became_peered":"2024-04-02T04:49:47.388801+0000","last_unstale":"2024-04-02T04:51:02.751985+0000","last_undegraded":"2024-04-02T04:51:02.751985+0000","last_fullsized":"2024-04-02T04:49:47.339290+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.747508+0000","last_change":"2024-04-02T04:49:47.444341+0000","last_active":"2024-04-02T04:51:02.747508+0000","last_peered":"2024-04-02T04:51:02.747508+0000","last_clean":"2024-04-02T04:49:45.331782+0000","last_became_active":"2024-04-02T04:49:47.444341+0000","last_became_peered":"2024-04-02T04:49:47.444341+0000","last_unstale":"2024-04-02T04:51:02.747508+0000","last_undegraded":"2024-04-02T04:51:02.747508+0000","last_fullsized":"2024-04-02T04:49:47.336855+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.714916+0000","last_change":"2024-04-02T04:49:29.857781+0000","last_active":"2024-04-02T04:51:02.714916+0000","last_peered":"2024-04-02T04:51:02.714916+0000","last_clean":"2024-04-02T04:51:02.714916+0000","last_became_active":"2024-04-02T04:49:29.857200+0000","last_became_peered":"2024-04-02T04:49:29.857200+0000","last_unstale":"2024-04-02T04:51:02.714916+0000","last_undegraded":"2024-04-02T04:51:02.714916+0000","last_fullsized":"2024-04-02T04:51:02.714916+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.718762+0000","last_change":"2024-04-02T04:49:28.747635+0000","last_active":"2024-04-02T04:51:02.718762+0000","last_peered":"2024-04-02T04:51:02.718762+0000","last_clean":"2024-04-02T04:51:02.718762+0000","last_became_active":"2024-04-02T04:49:28.746855+0000","last_became_peered":"2024-04-02T04:49:28.746855+0000","last_unstale":"2024-04-02T04:51:02.718762+0000","last_undegraded":"2024-04-02T04:51:02.718762+0000","last_fullsized":"2024-04-02T04:51:02.718762+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.721922+0000","last_change":"2024-04-02T04:49:47.472640+0000","last_active":"2024-04-02T04:51:02.721922+0000","last_peered":"2024-04-02T04:51:02.721922+0000","last_clean":"2024-04-02T04:49:45.352140+0000","last_became_active":"2024-04-02T04:49:47.472640+0000","last_became_peered":"2024-04-02T04:49:47.472640+0000","last_unstale":"2024-04-02T04:51:02.721922+0000","last_undegraded":"2024-04-02T04:51:02.721922+0000","last_fullsized":"2024-04-02T04:49:47.377460+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.719677+0000","last_change":"2024-04-02T04:49:29.875000+0000","last_active":"2024-04-02T04:51:02.719677+0000","last_peered":"2024-04-02T04:51:02.719677+0000","last_clean":"2024-04-02T04:51:02.719677+0000","last_became_active":"2024-04-02T04:49:29.874539+0000","last_became_peered":"2024-04-02T04:49:29.874539+0000","last_unstale":"2024-04-02T04:51:02.719677+0000","last_undegraded":"2024-04-02T04:51:02.719677+0000","last_fullsized":"2024-04-02T04:51:02.719677+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.747090+0000","last_change":"2024-04-02T04:49:47.452282+0000","last_active":"2024-04-02T04:51:02.747090+0000","last_peered":"2024-04-02T04:51:02.747090+0000","last_clean":"2024-04-02T04:49:45.347315+0000","last_became_active":"2024-04-02T04:49:47.452282+0000","last_became_peered":"2024-04-02T04:49:47.452282+0000","last_unstale":"2024-04-02T04:51:02.747090+0000","last_undegraded":"2024-04-02T04:51:02.747090+0000","last_fullsized":"2024-04-02T04:49:47.364280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.721305+0000","last_change":"2024-04-02T04:49:29.833094+0000","last_active":"2024-04-02T04:51:02.721305+0000","last_peered":"2024-04-02T04:51:02.721305+0000","last_clean":"2024-04-02T04:51:02.721305+0000","last_became_active":"2024-04-02T04:49:29.830080+0000","last_became_peered":"2024-04-02T04:49:29.830080+0000","last_unstale":"2024-04-02T04:51:02.721305+0000","last_undegraded":"2024-04-02T04:51:02.721305+0000","last_fullsized":"2024-04-02T04:51:02.721305+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.737309+0000","last_change":"2024-04-02T04:49:47.477237+0000","last_active":"2024-04-02T04:51:02.737309+0000","last_peered":"2024-04-02T04:51:02.737309+0000","last_clean":"2024-04-02T04:49:45.366065+0000","last_became_active":"2024-04-02T04:49:47.477237+0000","last_became_peered":"2024-04-02T04:49:47.477237+0000","last_unstale":"2024-04-02T04:51:02.737309+0000","last_undegraded":"2024-04-02T04:51:02.737309+0000","last_fullsized":"2024-04-02T04:49:47.430531+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.766931+0000","last_change":"2024-04-02T04:49:28.781126+0000","last_active":"2024-04-02T04:51:02.766931+0000","last_peered":"2024-04-02T04:51:02.766931+0000","last_clean":"2024-04-02T04:51:02.766931+0000","last_became_active":"2024-04-02T04:49:28.780779+0000","last_became_peered":"2024-04-02T04:49:28.780779+0000","last_unstale":"2024-04-02T04:51:02.766931+0000","last_undegraded":"2024-04-02T04:51:02.766931+0000","last_fullsized":"2024-04-02T04:51:02.766931+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.747874+0000","last_change":"2024-04-02T04:49:28.788472+0000","last_active":"2024-04-02T04:51:02.747874+0000","last_peered":"2024-04-02T04:51:02.747874+0000","last_clean":"2024-04-02T04:51:02.747874+0000","last_became_active":"2024-04-02T04:49:28.784897+0000","last_became_peered":"2024-04-02T04:49:28.784897+0000","last_unstale":"2024-04-02T04:51:02.747874+0000","last_undegraded":"2024-04-02T04:51:02.747874+0000","last_fullsized":"2024-04-02T04:51:02.747874+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.756022+0000","last_change":"2024-04-02T04:50:44.471617+0000","last_active":"2024-04-02T04:51:02.756022+0000","last_peered":"2024-04-02T04:51:02.756022+0000","last_clean":"2024-04-02T04:51:02.756022+0000","last_became_active":"2024-04-02T04:49:28.744153+0000","last_became_peered":"2024-04-02T04:49:28.744153+0000","last_unstale":"2024-04-02T04:51:02.756022+0000","last_undegraded":"2024-04-02T04:51:02.756022+0000","last_fullsized":"2024-04-02T04:51:02.756022+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.471508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.471508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:59.409627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751812+0000","last_change":"2024-04-02T04:49:28.761364+0000","last_active":"2024-04-02T04:51:02.751812+0000","last_peered":"2024-04-02T04:51:02.751812+0000","last_clean":"2024-04-02T04:51:02.751812+0000","last_became_active":"2024-04-02T04:49:28.760661+0000","last_became_peered":"2024-04-02T04:49:28.760661+0000","last_unstale":"2024-04-02T04:51:02.751812+0000","last_undegraded":"2024-04-02T04:51:02.751812+0000","last_fullsized":"2024-04-02T04:51:02.751812+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.892132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.773846+0000","last_change":"2024-04-02T04:49:28.771528+0000","last_active":"2024-04-02T04:51:02.773846+0000","last_peered":"2024-04-02T04:51:02.773846+0000","last_clean":"2024-04-02T04:51:02.773846+0000","last_became_active":"2024-04-02T04:49:28.769787+0000","last_became_peered":"2024-04-02T04:49:28.769787+0000","last_unstale":"2024-04-02T04:51:02.773846+0000","last_undegraded":"2024-04-02T04:51:02.773846+0000","last_fullsized":"2024-04-02T04:51:02.773846+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.724039+0000","last_change":"2024-04-02T04:49:28.728679+0000","last_active":"2024-04-02T04:51:02.724039+0000","last_peered":"2024-04-02T04:51:02.724039+0000","last_clean":"2024-04-02T04:51:02.724039+0000","last_became_active":"2024-04-02T04:49:28.728416+0000","last_became_peered":"2024-04-02T04:49:28.728416+0000","last_unstale":"2024-04-02T04:51:02.724039+0000","last_undegraded":"2024-04-02T04:51:02.724039+0000","last_fullsized":"2024-04-02T04:51:02.724039+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.729064+0000","last_change":"2024-04-02T04:49:28.728977+0000","last_active":"2024-04-02T04:51:02.729064+0000","last_peered":"2024-04-02T04:51:02.729064+0000","last_clean":"2024-04-02T04:51:02.729064+0000","last_became_active":"2024-04-02T04:49:28.727661+0000","last_became_peered":"2024-04-02T04:49:28.727661+0000","last_unstale":"2024-04-02T04:51:02.729064+0000","last_undegraded":"2024-04-02T04:51:02.729064+0000","last_fullsized":"2024-04-02T04:51:02.729064+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.759393+0000","last_change":"2024-04-02T04:49:28.762112+0000","last_active":"2024-04-02T04:51:02.759393+0000","last_peered":"2024-04-02T04:51:02.759393+0000","last_clean":"2024-04-02T04:51:02.759393+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:51:02.759393+0000","last_undegraded":"2024-04-02T04:51:02.759393+0000","last_fullsized":"2024-04-02T04:51:02.759393+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751424+0000","last_change":"2024-04-02T04:49:28.745274+0000","last_active":"2024-04-02T04:51:02.751424+0000","last_peered":"2024-04-02T04:51:02.751424+0000","last_clean":"2024-04-02T04:51:02.751424+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:51:02.751424+0000","last_undegraded":"2024-04-02T04:51:02.751424+0000","last_fullsized":"2024-04-02T04:51:02.751424+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.737152+0000","last_change":"2024-04-02T04:49:47.512403+0000","last_active":"2024-04-02T04:51:02.737152+0000","last_peered":"2024-04-02T04:51:02.737152+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.512403+0000","last_became_peered":"2024-04-02T04:49:47.512403+0000","last_unstale":"2024-04-02T04:51:02.737152+0000","last_undegraded":"2024-04-02T04:51:02.737152+0000","last_fullsized":"2024-04-02T04:49:47.379434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730810+0000","last_change":"2024-04-02T04:49:28.726174+0000","last_active":"2024-04-02T04:51:02.730810+0000","last_peered":"2024-04-02T04:51:02.730810+0000","last_clean":"2024-04-02T04:51:02.730810+0000","last_became_active":"2024-04-02T04:49:28.725896+0000","last_became_peered":"2024-04-02T04:49:28.725896+0000","last_unstale":"2024-04-02T04:51:02.730810+0000","last_undegraded":"2024-04-02T04:51:02.730810+0000","last_fullsized":"2024-04-02T04:51:02.730810+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.735384+0000","last_change":"2024-04-02T04:49:28.724761+0000","last_active":"2024-04-02T04:51:02.735384+0000","last_peered":"2024-04-02T04:51:02.735384+0000","last_clean":"2024-04-02T04:51:02.735384+0000","last_became_active":"2024-04-02T04:49:28.724387+0000","last_became_peered":"2024-04-02T04:49:28.724387+0000","last_unstale":"2024-04-02T04:51:02.735384+0000","last_undegraded":"2024-04-02T04:51:02.735384+0000","last_fullsized":"2024-04-02T04:51:02.735384+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.715372+0000","last_change":"2024-04-02T04:49:29.857549+0000","last_active":"2024-04-02T04:51:02.715372+0000","last_peered":"2024-04-02T04:51:02.715372+0000","last_clean":"2024-04-02T04:51:02.715372+0000","last_became_active":"2024-04-02T04:49:29.856708+0000","last_became_peered":"2024-04-02T04:49:29.856708+0000","last_unstale":"2024-04-02T04:51:02.715372+0000","last_undegraded":"2024-04-02T04:51:02.715372+0000","last_fullsized":"2024-04-02T04:51:02.715372+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712631+0000","last_change":"2024-04-02T04:49:29.816487+0000","last_active":"2024-04-02T04:51:02.712631+0000","last_peered":"2024-04-02T04:51:02.712631+0000","last_clean":"2024-04-02T04:51:02.712631+0000","last_became_active":"2024-04-02T04:49:29.816194+0000","last_became_peered":"2024-04-02T04:49:29.816194+0000","last_unstale":"2024-04-02T04:51:02.712631+0000","last_undegraded":"2024-04-02T04:51:02.712631+0000","last_fullsized":"2024-04-02T04:51:02.712631+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.737424+0000","last_change":"2024-04-02T04:49:47.427831+0000","last_active":"2024-04-02T04:51:02.737424+0000","last_peered":"2024-04-02T04:51:02.737424+0000","last_clean":"2024-04-02T04:49:45.321587+0000","last_became_active":"2024-04-02T04:49:47.427831+0000","last_became_peered":"2024-04-02T04:49:47.427831+0000","last_unstale":"2024-04-02T04:51:02.737424+0000","last_undegraded":"2024-04-02T04:51:02.737424+0000","last_fullsized":"2024-04-02T04:49:47.369909+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.714556+0000","last_change":"2024-04-02T04:49:47.387543+0000","last_active":"2024-04-02T04:51:02.714556+0000","last_peered":"2024-04-02T04:51:02.714556+0000","last_clean":"2024-04-02T04:49:45.306095+0000","last_became_active":"2024-04-02T04:49:47.387543+0000","last_became_peered":"2024-04-02T04:49:47.387543+0000","last_unstale":"2024-04-02T04:51:02.714556+0000","last_undegraded":"2024-04-02T04:51:02.714556+0000","last_fullsized":"2024-04-02T04:49:47.345085+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728065+0000","last_change":"2024-04-02T04:49:28.739636+0000","last_active":"2024-04-02T04:51:02.728065+0000","last_peered":"2024-04-02T04:51:02.728065+0000","last_clean":"2024-04-02T04:51:02.728065+0000","last_became_active":"2024-04-02T04:49:28.734672+0000","last_became_peered":"2024-04-02T04:49:28.734672+0000","last_unstale":"2024-04-02T04:51:02.728065+0000","last_undegraded":"2024-04-02T04:51:02.728065+0000","last_fullsized":"2024-04-02T04:51:02.728065+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.718148+0000","last_change":"2024-04-02T04:49:28.749558+0000","last_active":"2024-04-02T04:51:02.718148+0000","last_peered":"2024-04-02T04:51:02.718148+0000","last_clean":"2024-04-02T04:51:02.718148+0000","last_became_active":"2024-04-02T04:49:28.749072+0000","last_became_peered":"2024-04-02T04:49:28.749072+0000","last_unstale":"2024-04-02T04:51:02.718148+0000","last_undegraded":"2024-04-02T04:51:02.718148+0000","last_fullsized":"2024-04-02T04:51:02.718148+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.726525+0000","last_change":"2024-04-02T04:49:29.864378+0000","last_active":"2024-04-02T04:51:02.726525+0000","last_peered":"2024-04-02T04:51:02.726525+0000","last_clean":"2024-04-02T04:51:02.726525+0000","last_became_active":"2024-04-02T04:49:29.858586+0000","last_became_peered":"2024-04-02T04:49:29.858586+0000","last_unstale":"2024-04-02T04:51:02.726525+0000","last_undegraded":"2024-04-02T04:51:02.726525+0000","last_fullsized":"2024-04-02T04:51:02.726525+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.723036+0000","last_change":"2024-04-02T04:49:47.469970+0000","last_active":"2024-04-02T04:51:02.723036+0000","last_peered":"2024-04-02T04:51:02.723036+0000","last_clean":"2024-04-02T04:49:45.328796+0000","last_became_active":"2024-04-02T04:49:47.469970+0000","last_became_peered":"2024-04-02T04:49:47.469970+0000","last_unstale":"2024-04-02T04:51:02.723036+0000","last_undegraded":"2024-04-02T04:51:02.723036+0000","last_fullsized":"2024-04-02T04:49:47.382953+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.261447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.749250+0000","last_change":"2024-04-02T04:49:47.457335+0000","last_active":"2024-04-02T04:51:02.749250+0000","last_peered":"2024-04-02T04:51:02.749250+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.457335+0000","last_became_peered":"2024-04-02T04:49:47.457335+0000","last_unstale":"2024-04-02T04:51:02.749250+0000","last_undegraded":"2024-04-02T04:51:02.749250+0000","last_fullsized":"2024-04-02T04:49:47.405947+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.892010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.756568+0000","last_change":"2024-04-02T04:49:28.751480+0000","last_active":"2024-04-02T04:51:02.756568+0000","last_peered":"2024-04-02T04:51:02.756568+0000","last_clean":"2024-04-02T04:51:02.756568+0000","last_became_active":"2024-04-02T04:49:28.750713+0000","last_became_peered":"2024-04-02T04:49:28.750713+0000","last_unstale":"2024-04-02T04:51:02.756568+0000","last_undegraded":"2024-04-02T04:51:02.756568+0000","last_fullsized":"2024-04-02T04:51:02.756568+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.767189+0000","last_change":"2024-04-02T04:49:28.768154+0000","last_active":"2024-04-02T04:51:02.767189+0000","last_peered":"2024-04-02T04:51:02.767189+0000","last_clean":"2024-04-02T04:51:02.767189+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:51:02.767189+0000","last_undegraded":"2024-04-02T04:51:02.767189+0000","last_fullsized":"2024-04-02T04:51:02.767189+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.735688+0000","last_change":"2024-04-02T04:49:47.500400+0000","last_active":"2024-04-02T04:51:02.735688+0000","last_peered":"2024-04-02T04:51:02.735688+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.500400+0000","last_became_peered":"2024-04-02T04:49:47.500400+0000","last_unstale":"2024-04-02T04:51:02.735688+0000","last_undegraded":"2024-04-02T04:51:02.735688+0000","last_fullsized":"2024-04-02T04:49:47.430108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.724149+0000","last_change":"2024-04-02T04:50:29.991272+0000","last_active":"2024-04-02T04:51:02.724149+0000","last_peered":"2024-04-02T04:51:02.724149+0000","last_clean":"2024-04-02T04:51:02.724149+0000","last_became_active":"2024-04-02T04:49:28.716933+0000","last_became_peered":"2024-04-02T04:49:28.716933+0000","last_unstale":"2024-04-02T04:51:02.724149+0000","last_undegraded":"2024-04-02T04:51:02.724149+0000","last_fullsized":"2024-04-02T04:51:02.724149+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.991167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.149519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.747622+0000","last_change":"2024-04-02T04:49:28.780348+0000","last_active":"2024-04-02T04:51:02.747622+0000","last_peered":"2024-04-02T04:51:02.747622+0000","last_clean":"2024-04-02T04:51:02.747622+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:51:02.747622+0000","last_undegraded":"2024-04-02T04:51:02.747622+0000","last_fullsized":"2024-04-02T04:51:02.747622+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.766363+0000","last_change":"2024-04-02T04:49:28.757581+0000","last_active":"2024-04-02T04:51:02.766363+0000","last_peered":"2024-04-02T04:51:02.766363+0000","last_clean":"2024-04-02T04:51:02.766363+0000","last_became_active":"2024-04-02T04:49:28.757217+0000","last_became_peered":"2024-04-02T04:49:28.757217+0000","last_unstale":"2024-04-02T04:51:02.766363+0000","last_undegraded":"2024-04-02T04:51:02.766363+0000","last_fullsized":"2024-04-02T04:51:02.766363+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.892179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.742924+0000","last_change":"2024-04-02T04:49:47.508156+0000","last_active":"2024-04-02T04:51:02.742924+0000","last_peered":"2024-04-02T04:51:02.742924+0000","last_clean":"2024-04-02T04:49:45.359498+0000","last_became_active":"2024-04-02T04:49:47.508156+0000","last_became_peered":"2024-04-02T04:49:47.508156+0000","last_unstale":"2024-04-02T04:51:02.742924+0000","last_undegraded":"2024-04-02T04:51:02.742924+0000","last_fullsized":"2024-04-02T04:49:47.430692+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.750574+0000","last_change":"2024-04-02T04:49:28.759053+0000","last_active":"2024-04-02T04:51:02.750574+0000","last_peered":"2024-04-02T04:51:02.750574+0000","last_clean":"2024-04-02T04:51:02.750574+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:51:02.750574+0000","last_undegraded":"2024-04-02T04:51:02.750574+0000","last_fullsized":"2024-04-02T04:51:02.750574+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.742395+0000","last_change":"2024-04-02T04:49:47.501136+0000","last_active":"2024-04-02T04:51:02.742395+0000","last_peered":"2024-04-02T04:51:02.742395+0000","last_clean":"2024-04-02T04:49:45.381195+0000","last_became_active":"2024-04-02T04:49:47.501136+0000","last_became_peered":"2024-04-02T04:49:47.501136+0000","last_unstale":"2024-04-02T04:51:02.742395+0000","last_undegraded":"2024-04-02T04:51:02.742395+0000","last_fullsized":"2024-04-02T04:49:47.416717+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.748544+0000","last_change":"2024-04-02T04:49:47.510048+0000","last_active":"2024-04-02T04:51:02.748544+0000","last_peered":"2024-04-02T04:51:02.748544+0000","last_clean":"2024-04-02T04:49:45.381640+0000","last_became_active":"2024-04-02T04:49:47.510048+0000","last_became_peered":"2024-04-02T04:49:47.510048+0000","last_unstale":"2024-04-02T04:51:02.748544+0000","last_undegraded":"2024-04-02T04:51:02.748544+0000","last_fullsized":"2024-04-02T04:49:47.408507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.742329+0000","last_change":"2024-04-02T04:49:28.765539+0000","last_active":"2024-04-02T04:51:02.742329+0000","last_peered":"2024-04-02T04:51:02.742329+0000","last_clean":"2024-04-02T04:51:02.742329+0000","last_became_active":"2024-04-02T04:49:28.765229+0000","last_became_peered":"2024-04-02T04:49:28.765229+0000","last_unstale":"2024-04-02T04:51:02.742329+0000","last_undegraded":"2024-04-02T04:51:02.742329+0000","last_fullsized":"2024-04-02T04:51:02.742329+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.735879+0000","last_change":"2024-04-02T04:49:47.441206+0000","last_active":"2024-04-02T04:51:02.735879+0000","last_peered":"2024-04-02T04:51:02.735879+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.441206+0000","last_became_peered":"2024-04-02T04:49:47.441206+0000","last_unstale":"2024-04-02T04:51:02.735879+0000","last_undegraded":"2024-04-02T04:51:02.735879+0000","last_fullsized":"2024-04-02T04:49:47.375248+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.716793+0000","last_change":"2024-04-02T04:49:28.759884+0000","last_active":"2024-04-02T04:51:02.716793+0000","last_peered":"2024-04-02T04:51:02.716793+0000","last_clean":"2024-04-02T04:51:02.716793+0000","last_became_active":"2024-04-02T04:49:28.749451+0000","last_became_peered":"2024-04-02T04:49:28.749451+0000","last_unstale":"2024-04-02T04:51:02.716793+0000","last_undegraded":"2024-04-02T04:51:02.716793+0000","last_fullsized":"2024-04-02T04:51:02.716793+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.718765+0000","last_change":"2024-04-02T04:49:47.479424+0000","last_active":"2024-04-02T04:51:02.718765+0000","last_peered":"2024-04-02T04:51:02.718765+0000","last_clean":"2024-04-02T04:49:45.342017+0000","last_became_active":"2024-04-02T04:49:47.479424+0000","last_became_peered":"2024-04-02T04:49:47.479424+0000","last_unstale":"2024-04-02T04:51:02.718765+0000","last_undegraded":"2024-04-02T04:51:02.718765+0000","last_fullsized":"2024-04-02T04:49:47.413098+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.735134+0000","last_change":"2024-04-02T04:49:28.748601+0000","last_active":"2024-04-02T04:51:02.735134+0000","last_peered":"2024-04-02T04:51:02.735134+0000","last_clean":"2024-04-02T04:51:02.735134+0000","last_became_active":"2024-04-02T04:49:28.743589+0000","last_became_peered":"2024-04-02T04:49:28.743589+0000","last_unstale":"2024-04-02T04:51:02.735134+0000","last_undegraded":"2024-04-02T04:51:02.735134+0000","last_fullsized":"2024-04-02T04:51:02.735134+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":180,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.715049+0000","last_change":"2024-04-02T04:49:47.502106+0000","last_active":"2024-04-02T04:51:02.715049+0000","last_peered":"2024-04-02T04:51:02.715049+0000","last_clean":"2024-04-02T04:49:45.334274+0000","last_became_active":"2024-04-02T04:49:47.502106+0000","last_became_peered":"2024-04-02T04:49:47.502106+0000","last_unstale":"2024-04-02T04:51:02.715049+0000","last_undegraded":"2024-04-02T04:51:02.715049+0000","last_fullsized":"2024-04-02T04:49:47.399330+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":177,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712493+0000","last_change":"2024-04-02T04:49:29.805620+0000","last_active":"2024-04-02T04:51:02.712493+0000","last_peered":"2024-04-02T04:51:02.712493+0000","last_clean":"2024-04-02T04:51:02.712493+0000","last_became_active":"2024-04-02T04:49:29.805341+0000","last_became_peered":"2024-04-02T04:49:29.805341+0000","last_unstale":"2024-04-02T04:51:02.712493+0000","last_undegraded":"2024-04-02T04:51:02.712493+0000","last_fullsized":"2024-04-02T04:51:02.712493+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":151,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.721015+0000","last_change":"2024-04-02T04:49:47.482774+0000","last_active":"2024-04-02T04:51:02.721015+0000","last_peered":"2024-04-02T04:51:02.721015+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.482774+0000","last_became_peered":"2024-04-02T04:49:47.482774+0000","last_unstale":"2024-04-02T04:51:02.721015+0000","last_undegraded":"2024-04-02T04:51:02.721015+0000","last_fullsized":"2024-04-02T04:49:47.403371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.753305+0000","last_change":"2024-04-02T04:49:28.773379+0000","last_active":"2024-04-02T04:51:02.753305+0000","last_peered":"2024-04-02T04:51:02.753305+0000","last_clean":"2024-04-02T04:51:02.753305+0000","last_became_active":"2024-04-02T04:49:28.766979+0000","last_became_peered":"2024-04-02T04:49:28.766979+0000","last_unstale":"2024-04-02T04:51:02.753305+0000","last_undegraded":"2024-04-02T04:51:02.753305+0000","last_fullsized":"2024-04-02T04:51:02.753305+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.714901+0000","last_change":"2024-04-02T04:49:28.756847+0000","last_active":"2024-04-02T04:51:02.714901+0000","last_peered":"2024-04-02T04:51:02.714901+0000","last_clean":"2024-04-02T04:51:02.714901+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:51:02.714901+0000","last_undegraded":"2024-04-02T04:51:02.714901+0000","last_fullsized":"2024-04-02T04:51:02.714901+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.750396+0000","last_change":"2024-04-02T04:49:28.753851+0000","last_active":"2024-04-02T04:51:02.750396+0000","last_peered":"2024-04-02T04:51:02.750396+0000","last_clean":"2024-04-02T04:51:02.750396+0000","last_became_active":"2024-04-02T04:49:28.753376+0000","last_became_peered":"2024-04-02T04:49:28.753376+0000","last_unstale":"2024-04-02T04:51:02.750396+0000","last_undegraded":"2024-04-02T04:51:02.750396+0000","last_fullsized":"2024-04-02T04:51:02.750396+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.719731+0000","last_change":"2024-04-02T04:50:44.257998+0000","last_active":"2024-04-02T04:51:02.719731+0000","last_peered":"2024-04-02T04:51:02.719731+0000","last_clean":"2024-04-02T04:51:02.719731+0000","last_became_active":"2024-04-02T04:49:28.757583+0000","last_became_peered":"2024-04-02T04:49:28.757583+0000","last_unstale":"2024-04-02T04:51:02.719731+0000","last_undegraded":"2024-04-02T04:51:02.719731+0000","last_fullsized":"2024-04-02T04:51:02.719731+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.257747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.257747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:04.856689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":178,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.722279+0000","last_change":"2024-04-02T04:49:28.726706+0000","last_active":"2024-04-02T04:51:02.722279+0000","last_peered":"2024-04-02T04:51:02.722279+0000","last_clean":"2024-04-02T04:51:02.722279+0000","last_became_active":"2024-04-02T04:49:28.726425+0000","last_became_peered":"2024-04-02T04:49:28.726425+0000","last_unstale":"2024-04-02T04:51:02.722279+0000","last_undegraded":"2024-04-02T04:51:02.722279+0000","last_fullsized":"2024-04-02T04:51:02.722279+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"139'5","reported_seq":196,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.740932+0000","last_change":"2024-04-02T04:50:26.846127+0000","last_active":"2024-04-02T04:51:02.740932+0000","last_peered":"2024-04-02T04:51:02.740932+0000","last_clean":"2024-04-02T04:51:02.740932+0000","last_became_active":"2024-04-02T04:49:26.170913+0000","last_became_peered":"2024-04-02T04:49:26.170913+0000","last_unstale":"2024-04-02T04:51:02.740932+0000","last_undegraded":"2024-04-02T04:51:02.740932+0000","last_fullsized":"2024-04-02T04:51:02.740932+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:50:26.845900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.845900+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:49.621130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.759223+0000","last_change":"2024-04-02T04:49:26.217363+0000","last_active":"2024-04-02T04:51:02.759223+0000","last_peered":"2024-04-02T04:51:02.759223+0000","last_clean":"2024-04-02T04:51:02.759223+0000","last_became_active":"2024-04-02T04:49:26.216834+0000","last_became_peered":"2024-04-02T04:49:26.216834+0000","last_unstale":"2024-04-02T04:51:02.759223+0000","last_undegraded":"2024-04-02T04:51:02.759223+0000","last_fullsized":"2024-04-02T04:51:02.759223+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":329,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.746170+0000","last_change":"2024-04-02T04:49:47.366762+0000","last_active":"2024-04-02T04:51:02.746170+0000","last_peered":"2024-04-02T04:51:02.746170+0000","last_clean":"2024-04-02T04:49:45.330332+0000","last_became_active":"2024-04-02T04:49:47.366762+0000","last_became_peered":"2024-04-02T04:49:47.366762+0000","last_unstale":"2024-04-02T04:51:02.746170+0000","last_undegraded":"2024-04-02T04:51:02.746170+0000","last_fullsized":"2024-04-02T04:49:47.338850+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.797658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.797658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":193,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.758573+0000","last_change":"2024-04-02T04:49:47.409183+0000","last_active":"2024-04-02T04:51:02.758573+0000","last_peered":"2024-04-02T04:51:02.758573+0000","last_clean":"2024-04-02T04:44:01.886062+0000","last_became_active":"2024-04-02T04:49:47.409183+0000","last_became_peered":"2024-04-02T04:49:47.409183+0000","last_unstale":"2024-04-02T04:51:02.758573+0000","last_undegraded":"2024-04-02T04:49:47.369263+0000","last_fullsized":"2024-04-02T04:49:47.369128+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":329,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.721961+0000","last_change":"2024-04-02T04:49:47.494397+0000","last_active":"2024-04-02T04:51:02.721961+0000","last_peered":"2024-04-02T04:51:02.721961+0000","last_clean":"2024-04-02T04:49:45.361282+0000","last_became_active":"2024-04-02T04:49:47.494397+0000","last_became_peered":"2024-04-02T04:49:47.494397+0000","last_unstale":"2024-04-02T04:51:02.721961+0000","last_undegraded":"2024-04-02T04:51:02.721961+0000","last_fullsized":"2024-04-02T04:49:47.425453+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.658688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.658688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":322,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.755431+0000","last_change":"2024-04-02T04:50:11.467388+0000","last_active":"2024-04-02T04:51:02.755431+0000","last_peered":"2024-04-02T04:51:02.755431+0000","last_clean":"2024-04-02T04:51:02.755431+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:51:02.755431+0000","last_undegraded":"2024-04-02T04:51:02.755431+0000","last_fullsized":"2024-04-02T04:51:02.755431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.467182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.467182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.047738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.754412+0000","last_change":"2024-04-02T04:49:26.248455+0000","last_active":"2024-04-02T04:51:02.754412+0000","last_peered":"2024-04-02T04:51:02.754412+0000","last_clean":"2024-04-02T04:51:02.754412+0000","last_became_active":"2024-04-02T04:49:26.241757+0000","last_became_peered":"2024-04-02T04:49:26.241757+0000","last_unstale":"2024-04-02T04:51:02.754412+0000","last_undegraded":"2024-04-02T04:51:02.754412+0000","last_fullsized":"2024-04-02T04:51:02.754412+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728778+0000","last_change":"2024-04-02T04:49:26.298137+0000","last_active":"2024-04-02T04:51:02.728778+0000","last_peered":"2024-04-02T04:51:02.728778+0000","last_clean":"2024-04-02T04:51:02.728778+0000","last_became_active":"2024-04-02T04:49:26.296797+0000","last_became_peered":"2024-04-02T04:49:26.296797+0000","last_unstale":"2024-04-02T04:51:02.728778+0000","last_undegraded":"2024-04-02T04:51:02.728778+0000","last_fullsized":"2024-04-02T04:51:02.728778+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731872+0000","last_change":"2024-04-02T04:50:35.049746+0000","last_active":"2024-04-02T04:51:02.731872+0000","last_peered":"2024-04-02T04:51:02.731872+0000","last_clean":"2024-04-02T04:51:02.731872+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:51:02.731872+0000","last_undegraded":"2024-04-02T04:51:02.731872+0000","last_fullsized":"2024-04-02T04:51:02.731872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:35.049654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:50:35.049654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.104749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":358,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741429+0000","last_change":"2024-04-02T04:50:26.509397+0000","last_active":"2024-04-02T04:51:02.741429+0000","last_peered":"2024-04-02T04:51:02.741429+0000","last_clean":"2024-04-02T04:51:02.741429+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:51:02.741429+0000","last_undegraded":"2024-04-02T04:51:02.741429+0000","last_fullsized":"2024-04-02T04:51:02.741429+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:50:26.509291+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.509291+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.366019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":343,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.738739+0000","last_change":"2024-04-02T04:50:34.277643+0000","last_active":"2024-04-02T04:51:02.738739+0000","last_peered":"2024-04-02T04:51:02.738739+0000","last_clean":"2024-04-02T04:51:02.738739+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:51:02.738739+0000","last_undegraded":"2024-04-02T04:51:02.738739+0000","last_fullsized":"2024-04-02T04:51:02.738739+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:34.277549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.277549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:35.859980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":311,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.719517+0000","last_change":"2024-04-02T04:49:47.516977+0000","last_active":"2024-04-02T04:51:02.719517+0000","last_peered":"2024-04-02T04:51:02.719517+0000","last_clean":"2024-04-02T04:49:45.355630+0000","last_became_active":"2024-04-02T04:49:47.516977+0000","last_became_peered":"2024-04-02T04:49:47.516977+0000","last_unstale":"2024-04-02T04:51:02.719517+0000","last_undegraded":"2024-04-02T04:51:02.719517+0000","last_fullsized":"2024-04-02T04:49:47.384135+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"139'5","reported_seq":190,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730919+0000","last_change":"2024-04-02T04:49:25.143416+0000","last_active":"2024-04-02T04:51:02.730919+0000","last_peered":"2024-04-02T04:51:02.730919+0000","last_clean":"2024-04-02T04:51:02.730919+0000","last_became_active":"2024-04-02T04:49:25.139936+0000","last_became_peered":"2024-04-02T04:49:25.139936+0000","last_unstale":"2024-04-02T04:51:02.730919+0000","last_undegraded":"2024-04-02T04:51:02.730919+0000","last_fullsized":"2024-04-02T04:51:02.730919+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":192,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.737124+0000","last_change":"2024-04-02T04:50:39.186924+0000","last_active":"2024-04-02T04:51:02.737124+0000","last_peered":"2024-04-02T04:51:02.737124+0000","last_clean":"2024-04-02T04:51:02.737124+0000","last_became_active":"2024-04-02T04:49:25.143060+0000","last_became_peered":"2024-04-02T04:49:25.143060+0000","last_unstale":"2024-04-02T04:51:02.737124+0000","last_undegraded":"2024-04-02T04:51:02.737124+0000","last_fullsized":"2024-04-02T04:51:02.737124+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.186729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.186729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:48.141413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":331,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.751449+0000","last_change":"2024-04-02T04:49:47.472350+0000","last_active":"2024-04-02T04:51:02.751449+0000","last_peered":"2024-04-02T04:51:02.751449+0000","last_clean":"2024-04-02T04:49:45.385689+0000","last_became_active":"2024-04-02T04:49:47.472350+0000","last_became_peered":"2024-04-02T04:49:47.472350+0000","last_unstale":"2024-04-02T04:51:02.751449+0000","last_undegraded":"2024-04-02T04:51:02.751449+0000","last_fullsized":"2024-04-02T04:49:47.393316+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.719146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.719146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":196,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.749942+0000","last_change":"2024-04-02T04:49:47.381178+0000","last_active":"2024-04-02T04:51:02.749942+0000","last_peered":"2024-04-02T04:51:02.749942+0000","last_clean":"2024-04-02T04:44:01.856616+0000","last_became_active":"2024-04-02T04:49:47.381178+0000","last_became_peered":"2024-04-02T04:49:47.381178+0000","last_unstale":"2024-04-02T04:51:02.749942+0000","last_undegraded":"2024-04-02T04:49:47.359978+0000","last_fullsized":"2024-04-02T04:49:47.359801+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:00.356122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.356122+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":334,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730007+0000","last_change":"2024-04-02T04:50:12.312248+0000","last_active":"2024-04-02T04:51:02.730007+0000","last_peered":"2024-04-02T04:51:02.730007+0000","last_clean":"2024-04-02T04:51:02.730007+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:51:02.730007+0000","last_undegraded":"2024-04-02T04:51:02.730007+0000","last_fullsized":"2024-04-02T04:51:02.730007+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:12.312147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.312147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.986984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.731814+0000","last_change":"2024-04-02T04:49:47.438287+0000","last_active":"2024-04-02T04:51:02.731814+0000","last_peered":"2024-04-02T04:51:02.731814+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.438287+0000","last_became_peered":"2024-04-02T04:49:47.438287+0000","last_unstale":"2024-04-02T04:51:02.731814+0000","last_undegraded":"2024-04-02T04:51:02.731814+0000","last_fullsized":"2024-04-02T04:49:47.386231+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.437949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.437949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751012+0000","last_change":"2024-04-02T04:49:26.277902+0000","last_active":"2024-04-02T04:51:02.751012+0000","last_peered":"2024-04-02T04:51:02.751012+0000","last_clean":"2024-04-02T04:51:02.751012+0000","last_became_active":"2024-04-02T04:49:26.271155+0000","last_became_peered":"2024-04-02T04:49:26.271155+0000","last_unstale":"2024-04-02T04:51:02.751012+0000","last_undegraded":"2024-04-02T04:51:02.751012+0000","last_fullsized":"2024-04-02T04:51:02.751012+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.718278+0000","last_change":"2024-04-02T04:49:26.266092+0000","last_active":"2024-04-02T04:51:02.718278+0000","last_peered":"2024-04-02T04:51:02.718278+0000","last_clean":"2024-04-02T04:51:02.718278+0000","last_became_active":"2024-04-02T04:49:26.265834+0000","last_became_peered":"2024-04-02T04:49:26.265834+0000","last_unstale":"2024-04-02T04:51:02.718278+0000","last_undegraded":"2024-04-02T04:51:02.718278+0000","last_fullsized":"2024-04-02T04:51:02.718278+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":376,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.734784+0000","last_change":"2024-04-02T04:49:10.763649+0000","last_active":"2024-04-02T04:51:02.734784+0000","last_peered":"2024-04-02T04:51:02.734784+0000","last_clean":"2024-04-02T04:51:02.734784+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:51:02.734784+0000","last_undegraded":"2024-04-02T04:51:02.734784+0000","last_fullsized":"2024-04-02T04:51:02.734784+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:49:10.763404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.763404+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.717838+0000","last_change":"2024-04-02T04:49:47.385876+0000","last_active":"2024-04-02T04:51:02.717838+0000","last_peered":"2024-04-02T04:51:02.717838+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.385876+0000","last_became_peered":"2024-04-02T04:49:47.385876+0000","last_unstale":"2024-04-02T04:51:02.717838+0000","last_undegraded":"2024-04-02T04:51:02.717838+0000","last_fullsized":"2024-04-02T04:49:47.345306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.439457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.439457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":424,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.737861+0000","last_change":"2024-04-02T04:50:15.516760+0000","last_active":"2024-04-02T04:51:02.737861+0000","last_peered":"2024-04-02T04:51:02.737861+0000","last_clean":"2024-04-02T04:51:02.737861+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:51:02.737861+0000","last_undegraded":"2024-04-02T04:51:02.737861+0000","last_fullsized":"2024-04-02T04:51:02.737861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:50:15.516538+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.516538+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.144029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":330,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.724658+0000","last_change":"2024-04-02T04:50:13.288238+0000","last_active":"2024-04-02T04:51:02.724658+0000","last_peered":"2024-04-02T04:51:02.724658+0000","last_clean":"2024-04-02T04:51:02.724658+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:51:02.724658+0000","last_undegraded":"2024-04-02T04:51:02.724658+0000","last_fullsized":"2024-04-02T04:51:02.724658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:13.288136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.288136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:13.866854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.736290+0000","last_change":"2024-04-02T04:49:26.242898+0000","last_active":"2024-04-02T04:51:02.736290+0000","last_peered":"2024-04-02T04:51:02.736290+0000","last_clean":"2024-04-02T04:51:02.736290+0000","last_became_active":"2024-04-02T04:49:26.241469+0000","last_became_peered":"2024-04-02T04:49:26.241469+0000","last_unstale":"2024-04-02T04:51:02.736290+0000","last_undegraded":"2024-04-02T04:51:02.736290+0000","last_fullsized":"2024-04-02T04:51:02.736290+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727901+0000","last_change":"2024-04-02T04:49:26.312549+0000","last_active":"2024-04-02T04:51:02.727901+0000","last_peered":"2024-04-02T04:51:02.727901+0000","last_clean":"2024-04-02T04:51:02.727901+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:51:02.727901+0000","last_undegraded":"2024-04-02T04:51:02.727901+0000","last_fullsized":"2024-04-02T04:51:02.727901+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731528+0000","last_change":"2024-04-02T04:50:43.513008+0000","last_active":"2024-04-02T04:51:02.731528+0000","last_peered":"2024-04-02T04:51:02.731528+0000","last_clean":"2024-04-02T04:51:02.731528+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:51:02.731528+0000","last_undegraded":"2024-04-02T04:51:02.731528+0000","last_fullsized":"2024-04-02T04:51:02.731528+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:43.512888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.512888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:03.332189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":365,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717717+0000","last_change":"2024-04-02T04:50:17.228750+0000","last_active":"2024-04-02T04:51:02.717717+0000","last_peered":"2024-04-02T04:51:02.717717+0000","last_clean":"2024-04-02T04:51:02.717717+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:51:02.717717+0000","last_undegraded":"2024-04-02T04:51:02.717717+0000","last_fullsized":"2024-04-02T04:51:02.717717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:50:17.228650+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.228650+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:44.745235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":338,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.728005+0000","last_change":"2024-04-02T04:49:47.504164+0000","last_active":"2024-04-02T04:51:02.728005+0000","last_peered":"2024-04-02T04:51:02.728005+0000","last_clean":"2024-04-02T04:49:45.357271+0000","last_became_active":"2024-04-02T04:49:47.504164+0000","last_became_peered":"2024-04-02T04:49:47.504164+0000","last_unstale":"2024-04-02T04:51:02.728005+0000","last_undegraded":"2024-04-02T04:51:02.728005+0000","last_fullsized":"2024-04-02T04:49:47.406612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.071225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":323,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751239+0000","last_change":"2024-04-02T04:50:18.501638+0000","last_active":"2024-04-02T04:51:02.751239+0000","last_peered":"2024-04-02T04:51:02.751239+0000","last_clean":"2024-04-02T04:51:02.751239+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:51:02.751239+0000","last_undegraded":"2024-04-02T04:51:02.751239+0000","last_fullsized":"2024-04-02T04:51:02.751239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.501528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.501528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:42.986033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"139'9","reported_seq":192,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748592+0000","last_change":"2024-04-02T04:49:26.487620+0000","last_active":"2024-04-02T04:51:02.748592+0000","last_peered":"2024-04-02T04:51:02.748592+0000","last_clean":"2024-04-02T04:51:02.748592+0000","last_became_active":"2024-04-02T04:49:26.487105+0000","last_became_peered":"2024-04-02T04:49:26.487105+0000","last_unstale":"2024-04-02T04:51:02.748592+0000","last_undegraded":"2024-04-02T04:51:02.748592+0000","last_fullsized":"2024-04-02T04:51:02.748592+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":199,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.715122+0000","last_change":"2024-04-02T04:49:47.408871+0000","last_active":"2024-04-02T04:51:02.715122+0000","last_peered":"2024-04-02T04:51:02.715122+0000","last_clean":"2024-04-02T04:49:45.308505+0000","last_became_active":"2024-04-02T04:49:47.408871+0000","last_became_peered":"2024-04-02T04:49:47.408871+0000","last_unstale":"2024-04-02T04:51:02.715122+0000","last_undegraded":"2024-04-02T04:49:47.350015+0000","last_fullsized":"2024-04-02T04:49:47.349865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":393,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.713018+0000","last_change":"2024-04-02T04:49:47.511581+0000","last_active":"2024-04-02T04:51:02.713018+0000","last_peered":"2024-04-02T04:51:02.713018+0000","last_clean":"2024-04-02T04:49:45.312272+0000","last_became_active":"2024-04-02T04:49:47.511581+0000","last_became_peered":"2024-04-02T04:49:47.511581+0000","last_unstale":"2024-04-02T04:51:02.713018+0000","last_undegraded":"2024-04-02T04:49:47.361633+0000","last_fullsized":"2024-04-02T04:49:47.361362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:49:24.269994+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.269994+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":349,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727712+0000","last_change":"2024-04-02T04:50:44.352622+0000","last_active":"2024-04-02T04:51:02.727712+0000","last_peered":"2024-04-02T04:51:02.727712+0000","last_clean":"2024-04-02T04:51:02.727712+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:51:02.727712+0000","last_undegraded":"2024-04-02T04:51:02.727712+0000","last_fullsized":"2024-04-02T04:51:02.727712+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.352505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.352505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:56.008048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":267,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728925+0000","last_change":"2024-04-02T04:49:19.588206+0000","last_active":"2024-04-02T04:51:02.728925+0000","last_peered":"2024-04-02T04:51:02.728925+0000","last_clean":"2024-04-02T04:51:02.728925+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:51:02.728925+0000","last_undegraded":"2024-04-02T04:51:02.728925+0000","last_fullsized":"2024-04-02T04:51:02.728925+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:19.587996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:19.587996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":330,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.754880+0000","last_change":"2024-04-02T04:50:37.405854+0000","last_active":"2024-04-02T04:51:02.754880+0000","last_peered":"2024-04-02T04:51:02.754880+0000","last_clean":"2024-04-02T04:51:02.754880+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:51:02.754880+0000","last_undegraded":"2024-04-02T04:51:02.754880+0000","last_fullsized":"2024-04-02T04:51:02.754880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.405673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.241866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":152,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.765949+0000","last_change":"2024-04-02T04:49:47.396455+0000","last_active":"2024-04-02T04:51:02.765949+0000","last_peered":"2024-04-02T04:51:02.765949+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.396455+0000","last_became_peered":"2024-04-02T04:49:47.396455+0000","last_unstale":"2024-04-02T04:51:02.765949+0000","last_undegraded":"2024-04-02T04:51:02.765949+0000","last_fullsized":"2024-04-02T04:49:47.375223+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741281+0000","last_change":"2024-04-02T04:49:26.252236+0000","last_active":"2024-04-02T04:51:02.741281+0000","last_peered":"2024-04-02T04:51:02.741281+0000","last_clean":"2024-04-02T04:51:02.741281+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:51:02.741281+0000","last_undegraded":"2024-04-02T04:51:02.741281+0000","last_fullsized":"2024-04-02T04:51:02.741281+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":373,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.731545+0000","last_change":"2024-04-02T04:49:47.453493+0000","last_active":"2024-04-02T04:51:02.731545+0000","last_peered":"2024-04-02T04:51:02.731545+0000","last_clean":"2024-04-02T04:49:45.308750+0000","last_became_active":"2024-04-02T04:49:47.453493+0000","last_became_peered":"2024-04-02T04:49:47.453493+0000","last_unstale":"2024-04-02T04:51:02.731545+0000","last_undegraded":"2024-04-02T04:49:47.351067+0000","last_fullsized":"2024-04-02T04:49:47.350938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:48:49.534769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.534769+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.731406+0000","last_change":"2024-04-02T04:49:47.521979+0000","last_active":"2024-04-02T04:51:02.731406+0000","last_peered":"2024-04-02T04:51:02.731406+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.521979+0000","last_became_peered":"2024-04-02T04:49:47.521979+0000","last_unstale":"2024-04-02T04:51:02.731406+0000","last_undegraded":"2024-04-02T04:51:02.731406+0000","last_fullsized":"2024-04-02T04:49:47.391120+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.344546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":423,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.750771+0000","last_change":"2024-04-02T04:50:31.493819+0000","last_active":"2024-04-02T04:51:02.750771+0000","last_peered":"2024-04-02T04:51:02.750771+0000","last_clean":"2024-04-02T04:51:02.750771+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:51:02.750771+0000","last_undegraded":"2024-04-02T04:51:02.750771+0000","last_fullsized":"2024-04-02T04:51:02.750771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:31.493715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.493715+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:41.738948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":324,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.719872+0000","last_change":"2024-04-02T04:50:34.462138+0000","last_active":"2024-04-02T04:51:02.719872+0000","last_peered":"2024-04-02T04:51:02.719872+0000","last_clean":"2024-04-02T04:51:02.719872+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:51:02.719872+0000","last_undegraded":"2024-04-02T04:51:02.719872+0000","last_fullsized":"2024-04-02T04:51:02.719872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:34.462064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.462064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:51.535481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":193,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731123+0000","last_change":"2024-04-02T04:50:37.076041+0000","last_active":"2024-04-02T04:51:02.731123+0000","last_peered":"2024-04-02T04:51:02.731123+0000","last_clean":"2024-04-02T04:51:02.731123+0000","last_became_active":"2024-04-02T04:49:26.319266+0000","last_became_peered":"2024-04-02T04:49:26.319266+0000","last_unstale":"2024-04-02T04:51:02.731123+0000","last_undegraded":"2024-04-02T04:51:02.731123+0000","last_fullsized":"2024-04-02T04:51:02.731123+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:50:37.075742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.075742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:01.825952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.767904+0000","last_change":"2024-04-02T04:50:31.781484+0000","last_active":"2024-04-02T04:51:02.767904+0000","last_peered":"2024-04-02T04:51:02.767904+0000","last_clean":"2024-04-02T04:51:02.767904+0000","last_became_active":"2024-04-02T04:49:26.255429+0000","last_became_peered":"2024-04-02T04:49:26.255429+0000","last_unstale":"2024-04-02T04:51:02.767904+0000","last_undegraded":"2024-04-02T04:51:02.767904+0000","last_fullsized":"2024-04-02T04:51:02.767904+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.781273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.781273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:36.148689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":319,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.739166+0000","last_change":"2024-04-02T04:49:13.637761+0000","last_active":"2024-04-02T04:51:02.739166+0000","last_peered":"2024-04-02T04:51:02.739166+0000","last_clean":"2024-04-02T04:51:02.739166+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:51:02.739166+0000","last_undegraded":"2024-04-02T04:51:02.739166+0000","last_fullsized":"2024-04-02T04:51:02.739166+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:13.637518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:13.637518+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.637518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"188'18","reported_seq":356,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741806+0000","last_change":"2024-04-02T04:50:28.030755+0000","last_active":"2024-04-02T04:51:02.741806+0000","last_peered":"2024-04-02T04:51:02.741806+0000","last_clean":"2024-04-02T04:51:02.741806+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:51:02.741806+0000","last_undegraded":"2024-04-02T04:51:02.741806+0000","last_fullsized":"2024-04-02T04:51:02.741806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'18","last_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.030658+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.699370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":339,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.768044+0000","last_change":"2024-04-02T04:49:47.470291+0000","last_active":"2024-04-02T04:51:02.768044+0000","last_peered":"2024-04-02T04:51:02.768044+0000","last_clean":"2024-04-02T04:49:45.387254+0000","last_became_active":"2024-04-02T04:49:47.470291+0000","last_became_peered":"2024-04-02T04:49:47.470291+0000","last_unstale":"2024-04-02T04:51:02.768044+0000","last_undegraded":"2024-04-02T04:51:02.768044+0000","last_fullsized":"2024-04-02T04:49:47.418421+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.690362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.690362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":743,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.713463+0000","last_change":"2024-04-02T04:50:39.632326+0000","last_active":"2024-04-02T04:51:02.713463+0000","last_peered":"2024-04-02T04:51:02.713463+0000","last_clean":"2024-04-02T04:51:02.713463+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:51:02.713463+0000","last_undegraded":"2024-04-02T04:51:02.713463+0000","last_fullsized":"2024-04-02T04:51:02.713463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:50:39.632171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.632171+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:06.420877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.742954+0000","last_change":"2024-04-02T04:49:47.470355+0000","last_active":"2024-04-02T04:51:02.742954+0000","last_peered":"2024-04-02T04:51:02.742954+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.470355+0000","last_became_peered":"2024-04-02T04:49:47.470355+0000","last_unstale":"2024-04-02T04:51:02.742954+0000","last_undegraded":"2024-04-02T04:51:02.742954+0000","last_fullsized":"2024-04-02T04:49:47.399308+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.890904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748303+0000","last_change":"2024-04-02T04:50:41.426843+0000","last_active":"2024-04-02T04:51:02.748303+0000","last_peered":"2024-04-02T04:51:02.748303+0000","last_clean":"2024-04-02T04:51:02.748303+0000","last_became_active":"2024-04-02T04:49:26.343060+0000","last_became_peered":"2024-04-02T04:49:26.343060+0000","last_unstale":"2024-04-02T04:51:02.748303+0000","last_undegraded":"2024-04-02T04:51:02.748303+0000","last_fullsized":"2024-04-02T04:51:02.748303+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:41.426741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:41.426741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:49.997920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.749100+0000","last_change":"2024-04-02T04:49:26.257833+0000","last_active":"2024-04-02T04:51:02.749100+0000","last_peered":"2024-04-02T04:51:02.749100+0000","last_clean":"2024-04-02T04:51:02.749100+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:51:02.749100+0000","last_undegraded":"2024-04-02T04:51:02.749100+0000","last_fullsized":"2024-04-02T04:51:02.749100+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"153'13","reported_seq":429,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.713093+0000","last_change":"2024-04-02T04:49:47.488932+0000","last_active":"2024-04-02T04:51:02.713093+0000","last_peered":"2024-04-02T04:51:02.713093+0000","last_clean":"2024-04-02T04:49:45.312370+0000","last_became_active":"2024-04-02T04:49:47.488932+0000","last_became_peered":"2024-04-02T04:49:47.488932+0000","last_unstale":"2024-04-02T04:51:02.713093+0000","last_undegraded":"2024-04-02T04:49:47.362506+0000","last_fullsized":"2024-04-02T04:49:47.362177+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:48:38.477825+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.477825+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":366,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.722051+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:51:02.722051+0000","last_peered":"2024-04-02T04:51:02.722051+0000","last_clean":"2024-04-02T04:49:45.321982+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:51:02.722051+0000","last_undegraded":"2024-04-02T04:49:47.406158+0000","last_fullsized":"2024-04-02T04:49:47.406050+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:49:07.374876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.374876+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":345,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.742320+0000","last_change":"2024-04-02T04:50:43.360677+0000","last_active":"2024-04-02T04:51:02.742320+0000","last_peered":"2024-04-02T04:51:02.742320+0000","last_clean":"2024-04-02T04:51:02.742320+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:51:02.742320+0000","last_undegraded":"2024-04-02T04:51:02.742320+0000","last_fullsized":"2024-04-02T04:51:02.742320+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:50:43.360457+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.360457+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:54.430085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":329,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.715185+0000","last_change":"2024-04-02T04:49:47.479633+0000","last_active":"2024-04-02T04:51:02.715185+0000","last_peered":"2024-04-02T04:51:02.715185+0000","last_clean":"2024-04-02T04:49:45.308628+0000","last_became_active":"2024-04-02T04:49:47.479633+0000","last_became_peered":"2024-04-02T04:49:47.479633+0000","last_unstale":"2024-04-02T04:51:02.715185+0000","last_undegraded":"2024-04-02T04:51:02.715185+0000","last_fullsized":"2024-04-02T04:49:47.350430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.555854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.555854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":323,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730972+0000","last_change":"2024-04-02T04:50:29.535635+0000","last_active":"2024-04-02T04:51:02.730972+0000","last_peered":"2024-04-02T04:51:02.730972+0000","last_clean":"2024-04-02T04:51:02.730972+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:51:02.730972+0000","last_undegraded":"2024-04-02T04:51:02.730972+0000","last_fullsized":"2024-04-02T04:51:02.730972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.535314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.535314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.006171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"139'10","reported_seq":201,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.737695+0000","last_change":"2024-04-02T04:50:44.141711+0000","last_active":"2024-04-02T04:51:02.737695+0000","last_peered":"2024-04-02T04:51:02.737695+0000","last_clean":"2024-04-02T04:51:02.737695+0000","last_became_active":"2024-04-02T04:49:26.341161+0000","last_became_peered":"2024-04-02T04:49:26.341161+0000","last_unstale":"2024-04-02T04:51:02.737695+0000","last_undegraded":"2024-04-02T04:51:02.737695+0000","last_fullsized":"2024-04-02T04:51:02.737695+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:50:44.141586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.141586+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:13.742820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.772449+0000","last_change":"2024-04-02T04:50:29.806953+0000","last_active":"2024-04-02T04:51:02.772449+0000","last_peered":"2024-04-02T04:51:02.772449+0000","last_clean":"2024-04-02T04:51:02.772449+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:51:02.772449+0000","last_undegraded":"2024-04-02T04:51:02.772449+0000","last_fullsized":"2024-04-02T04:51:02.772449+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.806751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.806751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.900759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"153'13","reported_seq":429,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.731538+0000","last_change":"2024-04-02T04:49:47.472920+0000","last_active":"2024-04-02T04:51:02.731538+0000","last_peered":"2024-04-02T04:51:02.731538+0000","last_clean":"2024-04-02T04:49:45.374829+0000","last_became_active":"2024-04-02T04:49:47.472920+0000","last_became_peered":"2024-04-02T04:49:47.472920+0000","last_unstale":"2024-04-02T04:51:02.731538+0000","last_undegraded":"2024-04-02T04:49:47.384585+0000","last_fullsized":"2024-04-02T04:49:47.384414+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":498,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.745188+0000","last_change":"2024-04-02T04:50:20.307539+0000","last_active":"2024-04-02T04:51:02.745188+0000","last_peered":"2024-04-02T04:51:02.745188+0000","last_clean":"2024-04-02T04:51:02.745188+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:51:02.745188+0000","last_undegraded":"2024-04-02T04:51:02.745188+0000","last_fullsized":"2024-04-02T04:51:02.745188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:50:20.307283+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.307283+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:24.958471+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.741407+0000","last_change":"2024-04-02T04:49:47.431753+0000","last_active":"2024-04-02T04:51:02.741407+0000","last_peered":"2024-04-02T04:51:02.741407+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.431753+0000","last_became_peered":"2024-04-02T04:49:47.431753+0000","last_unstale":"2024-04-02T04:51:02.741407+0000","last_undegraded":"2024-04-02T04:51:02.741407+0000","last_fullsized":"2024-04-02T04:49:47.405769+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":337,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.734429+0000","last_change":"2024-04-02T04:49:15.756968+0000","last_active":"2024-04-02T04:51:02.734429+0000","last_peered":"2024-04-02T04:51:02.734429+0000","last_clean":"2024-04-02T04:51:02.734429+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:51:02.734429+0000","last_undegraded":"2024-04-02T04:51:02.734429+0000","last_fullsized":"2024-04-02T04:51:02.734429+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:49:15.756715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.756715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891064+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":323,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727960+0000","last_change":"2024-04-02T04:50:09.280418+0000","last_active":"2024-04-02T04:51:02.727960+0000","last_peered":"2024-04-02T04:51:02.727960+0000","last_clean":"2024-04-02T04:51:02.727960+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:51:02.727960+0000","last_undegraded":"2024-04-02T04:51:02.727960+0000","last_fullsized":"2024-04-02T04:51:02.727960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.280337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.280337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:37.314038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":187,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.731601+0000","last_change":"2024-04-02T04:49:47.516900+0000","last_active":"2024-04-02T04:51:02.731601+0000","last_peered":"2024-04-02T04:51:02.731601+0000","last_clean":"2024-04-02T04:49:45.354881+0000","last_became_active":"2024-04-02T04:49:47.516900+0000","last_became_peered":"2024-04-02T04:49:47.516900+0000","last_unstale":"2024-04-02T04:51:02.731601+0000","last_undegraded":"2024-04-02T04:51:02.731601+0000","last_fullsized":"2024-04-02T04:49:47.391507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":334,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.747849+0000","last_change":"2024-04-02T04:50:13.057737+0000","last_active":"2024-04-02T04:51:02.747849+0000","last_peered":"2024-04-02T04:51:02.747849+0000","last_clean":"2024-04-02T04:51:02.747849+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:51:02.747849+0000","last_undegraded":"2024-04-02T04:51:02.747849+0000","last_fullsized":"2024-04-02T04:51:02.747849+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:13.057654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.057654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:33.157077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.723996+0000","last_change":"2024-04-02T04:49:47.509591+0000","last_active":"2024-04-02T04:51:02.723996+0000","last_peered":"2024-04-02T04:51:02.723996+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.509591+0000","last_became_peered":"2024-04-02T04:49:47.509591+0000","last_unstale":"2024-04-02T04:51:02.723996+0000","last_undegraded":"2024-04-02T04:51:02.723996+0000","last_fullsized":"2024-04-02T04:49:47.387091+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.441320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.441320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":365,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.719159+0000","last_change":"2024-04-02T04:50:19.055361+0000","last_active":"2024-04-02T04:51:02.719159+0000","last_peered":"2024-04-02T04:51:02.719159+0000","last_clean":"2024-04-02T04:51:02.719159+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:51:02.719159+0000","last_undegraded":"2024-04-02T04:51:02.719159+0000","last_fullsized":"2024-04-02T04:51:02.719159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:50:19.055250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.055250+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.015434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":331,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.744987+0000","last_change":"2024-04-02T04:50:26.342752+0000","last_active":"2024-04-02T04:51:02.744987+0000","last_peered":"2024-04-02T04:51:02.744987+0000","last_clean":"2024-04-02T04:51:02.744987+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:51:02.744987+0000","last_undegraded":"2024-04-02T04:51:02.744987+0000","last_fullsized":"2024-04-02T04:51:02.744987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.342429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.342429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.858608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.767479+0000","last_change":"2024-04-02T04:49:47.470287+0000","last_active":"2024-04-02T04:51:02.767479+0000","last_peered":"2024-04-02T04:51:02.767479+0000","last_clean":"2024-04-02T04:49:45.386144+0000","last_became_active":"2024-04-02T04:49:47.470287+0000","last_became_peered":"2024-04-02T04:49:47.470287+0000","last_unstale":"2024-04-02T04:51:02.767479+0000","last_undegraded":"2024-04-02T04:51:02.767479+0000","last_fullsized":"2024-04-02T04:49:47.418077+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.890968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.742181+0000","last_change":"2024-04-02T04:50:24.860538+0000","last_active":"2024-04-02T04:51:02.742181+0000","last_peered":"2024-04-02T04:51:02.742181+0000","last_clean":"2024-04-02T04:51:02.742181+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:51:02.742181+0000","last_undegraded":"2024-04-02T04:51:02.742181+0000","last_fullsized":"2024-04-02T04:51:02.742181+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:24.860246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:24.860246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.486260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":331,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.773513+0000","last_change":"2024-04-02T04:49:47.413615+0000","last_active":"2024-04-02T04:51:02.773513+0000","last_peered":"2024-04-02T04:51:02.773513+0000","last_clean":"2024-04-02T04:49:45.390373+0000","last_became_active":"2024-04-02T04:49:47.413615+0000","last_became_peered":"2024-04-02T04:49:47.413615+0000","last_unstale":"2024-04-02T04:51:02.773513+0000","last_undegraded":"2024-04-02T04:51:02.773513+0000","last_fullsized":"2024-04-02T04:49:47.396592+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.569058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.569058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.890949+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":460,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.730571+0000","last_change":"2024-04-02T04:49:47.451735+0000","last_active":"2024-04-02T04:51:02.730571+0000","last_peered":"2024-04-02T04:51:02.730571+0000","last_clean":"2024-04-02T04:44:53.853321+0000","last_became_active":"2024-04-02T04:49:47.451735+0000","last_became_peered":"2024-04-02T04:49:47.451735+0000","last_unstale":"2024-04-02T04:51:02.730571+0000","last_undegraded":"2024-04-02T04:49:47.390635+0000","last_fullsized":"2024-04-02T04:49:47.390485+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":329,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.745371+0000","last_change":"2024-04-02T04:49:47.435042+0000","last_active":"2024-04-02T04:51:02.745371+0000","last_peered":"2024-04-02T04:51:02.745371+0000","last_clean":"2024-04-02T04:49:45.341176+0000","last_became_active":"2024-04-02T04:49:47.435042+0000","last_became_peered":"2024-04-02T04:49:47.435042+0000","last_unstale":"2024-04-02T04:51:02.745371+0000","last_undegraded":"2024-04-02T04:51:02.745371+0000","last_fullsized":"2024-04-02T04:49:47.351952+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:39.903683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.903683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":351,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712841+0000","last_change":"2024-04-02T04:49:04.488216+0000","last_active":"2024-04-02T04:51:02.712841+0000","last_peered":"2024-04-02T04:51:02.712841+0000","last_clean":"2024-04-02T04:51:02.712841+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:51:02.712841+0000","last_undegraded":"2024-04-02T04:51:02.712841+0000","last_fullsized":"2024-04-02T04:51:02.712841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:49:04.488030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.488030+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":322,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.761169+0000","last_change":"2024-04-02T04:49:25.573578+0000","last_active":"2024-04-02T04:51:02.761169+0000","last_peered":"2024-04-02T04:51:02.761169+0000","last_clean":"2024-04-02T04:51:02.761169+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:51:02.761169+0000","last_undegraded":"2024-04-02T04:51:02.761169+0000","last_fullsized":"2024-04-02T04:51:02.761169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.573285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.743118+0000","last_change":"2024-04-02T04:49:26.310768+0000","last_active":"2024-04-02T04:51:02.743118+0000","last_peered":"2024-04-02T04:51:02.743118+0000","last_clean":"2024-04-02T04:51:02.743118+0000","last_became_active":"2024-04-02T04:49:26.290196+0000","last_became_peered":"2024-04-02T04:49:26.290196+0000","last_unstale":"2024-04-02T04:51:02.743118+0000","last_undegraded":"2024-04-02T04:51:02.743118+0000","last_fullsized":"2024-04-02T04:51:02.743118+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.751753+0000","last_change":"2024-04-02T04:49:47.382055+0000","last_active":"2024-04-02T04:51:02.751753+0000","last_peered":"2024-04-02T04:51:02.751753+0000","last_clean":"2024-04-02T04:49:45.332186+0000","last_became_active":"2024-04-02T04:49:47.382055+0000","last_became_peered":"2024-04-02T04:49:47.382055+0000","last_unstale":"2024-04-02T04:51:02.751753+0000","last_undegraded":"2024-04-02T04:51:02.751753+0000","last_fullsized":"2024-04-02T04:49:47.337410+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741925+0000","last_change":"2024-04-02T04:50:11.329816+0000","last_active":"2024-04-02T04:51:02.741925+0000","last_peered":"2024-04-02T04:51:02.741925+0000","last_clean":"2024-04-02T04:51:02.741925+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:51:02.741925+0000","last_undegraded":"2024-04-02T04:51:02.741925+0000","last_fullsized":"2024-04-02T04:51:02.741925+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.329645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.329645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.297573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":342,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.729797+0000","last_change":"2024-04-02T04:50:14.307591+0000","last_active":"2024-04-02T04:51:02.729797+0000","last_peered":"2024-04-02T04:51:02.729797+0000","last_clean":"2024-04-02T04:51:02.729797+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:51:02.729797+0000","last_undegraded":"2024-04-02T04:51:02.729797+0000","last_fullsized":"2024-04-02T04:51:02.729797+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:14.307478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.307478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.776147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":387,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.723889+0000","last_change":"2024-04-02T04:50:16.319646+0000","last_active":"2024-04-02T04:51:02.723889+0000","last_peered":"2024-04-02T04:51:02.723889+0000","last_clean":"2024-04-02T04:51:02.723889+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:51:02.723889+0000","last_undegraded":"2024-04-02T04:51:02.723889+0000","last_fullsized":"2024-04-02T04:51:02.723889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:50:16.319556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:16.319556+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:16.400689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.772706+0000","last_change":"2024-04-02T04:49:47.492471+0000","last_active":"2024-04-02T04:51:02.772706+0000","last_peered":"2024-04-02T04:51:02.772706+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.492471+0000","last_became_peered":"2024-04-02T04:49:47.492471+0000","last_unstale":"2024-04-02T04:51:02.772706+0000","last_undegraded":"2024-04-02T04:51:02.772706+0000","last_fullsized":"2024-04-02T04:49:47.411395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.890928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":152,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.760161+0000","last_change":"2024-04-02T04:49:47.388497+0000","last_active":"2024-04-02T04:51:02.760161+0000","last_peered":"2024-04-02T04:51:02.760161+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.388497+0000","last_became_peered":"2024-04-02T04:49:47.388497+0000","last_unstale":"2024-04-02T04:51:02.760161+0000","last_undegraded":"2024-04-02T04:51:02.760161+0000","last_fullsized":"2024-04-02T04:49:47.356622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.733159+0000","last_change":"2024-04-02T04:50:22.825218+0000","last_active":"2024-04-02T04:51:02.733159+0000","last_peered":"2024-04-02T04:51:02.733159+0000","last_clean":"2024-04-02T04:51:02.733159+0000","last_became_active":"2024-04-02T04:49:26.229137+0000","last_became_peered":"2024-04-02T04:49:26.229137+0000","last_unstale":"2024-04-02T04:51:02.733159+0000","last_undegraded":"2024-04-02T04:51:02.733159+0000","last_fullsized":"2024-04-02T04:51:02.733159+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:22.825000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:22.825000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.909236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.743438+0000","last_change":"2024-04-02T04:49:17.093584+0000","last_active":"2024-04-02T04:51:02.743438+0000","last_peered":"2024-04-02T04:51:02.743438+0000","last_clean":"2024-04-02T04:51:02.743438+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:51:02.743438+0000","last_undegraded":"2024-04-02T04:51:02.743438+0000","last_fullsized":"2024-04-02T04:51:02.743438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:17.093485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:49:17.093485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":379,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730502+0000","last_change":"2024-04-02T04:50:12.546833+0000","last_active":"2024-04-02T04:51:02.730502+0000","last_peered":"2024-04-02T04:51:02.730502+0000","last_clean":"2024-04-02T04:51:02.730502+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:51:02.730502+0000","last_undegraded":"2024-04-02T04:51:02.730502+0000","last_fullsized":"2024-04-02T04:51:02.730502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:50:12.546683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.546683+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:34.121347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":334,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.755055+0000","last_change":"2024-04-02T04:50:15.550230+0000","last_active":"2024-04-02T04:51:02.755055+0000","last_peered":"2024-04-02T04:51:02.755055+0000","last_clean":"2024-04-02T04:51:02.755055+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:51:02.755055+0000","last_undegraded":"2024-04-02T04:51:02.755055+0000","last_fullsized":"2024-04-02T04:51:02.755055+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.550133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.550133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:20.615818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":321,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.722067+0000","last_change":"2024-04-02T04:49:47.486462+0000","last_active":"2024-04-02T04:51:02.722067+0000","last_peered":"2024-04-02T04:51:02.722067+0000","last_clean":"2024-04-02T04:49:45.345774+0000","last_became_active":"2024-04-02T04:49:47.486462+0000","last_became_peered":"2024-04-02T04:49:47.486462+0000","last_unstale":"2024-04-02T04:51:02.722067+0000","last_undegraded":"2024-04-02T04:49:47.409192+0000","last_fullsized":"2024-04-02T04:49:47.409080+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:48:49.483159+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.483159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"139'5","reported_seq":191,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.729928+0000","last_change":"2024-04-02T04:49:47.490156+0000","last_active":"2024-04-02T04:51:02.729928+0000","last_peered":"2024-04-02T04:51:02.729928+0000","last_clean":"2024-04-02T04:49:45.349908+0000","last_became_active":"2024-04-02T04:49:47.490156+0000","last_became_peered":"2024-04-02T04:49:47.490156+0000","last_unstale":"2024-04-02T04:51:02.729928+0000","last_undegraded":"2024-04-02T04:49:47.382039+0000","last_fullsized":"2024-04-02T04:49:47.381919+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"138'2","reported_seq":185,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.744664+0000","last_change":"2024-04-02T04:49:26.270988+0000","last_active":"2024-04-02T04:51:02.744664+0000","last_peered":"2024-04-02T04:51:02.744664+0000","last_clean":"2024-04-02T04:51:02.744664+0000","last_became_active":"2024-04-02T04:49:26.270708+0000","last_became_peered":"2024-04-02T04:49:26.270708+0000","last_unstale":"2024-04-02T04:51:02.744664+0000","last_undegraded":"2024-04-02T04:51:02.744664+0000","last_fullsized":"2024-04-02T04:51:02.744664+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":152,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.732341+0000","last_change":"2024-04-02T04:49:47.428471+0000","last_active":"2024-04-02T04:51:02.732341+0000","last_peered":"2024-04-02T04:51:02.732341+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.428471+0000","last_became_peered":"2024-04-02T04:49:47.428471+0000","last_unstale":"2024-04-02T04:51:02.732341+0000","last_undegraded":"2024-04-02T04:51:02.732341+0000","last_fullsized":"2024-04-02T04:49:47.368822+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728488+0000","last_change":"2024-04-02T04:49:16.090440+0000","last_active":"2024-04-02T04:51:02.728488+0000","last_peered":"2024-04-02T04:51:02.728488+0000","last_clean":"2024-04-02T04:51:02.728488+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:51:02.728488+0000","last_undegraded":"2024-04-02T04:51:02.728488+0000","last_fullsized":"2024-04-02T04:51:02.728488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:16.090227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:16.090227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":438,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.749154+0000","last_change":"2024-04-02T04:49:15.791519+0000","last_active":"2024-04-02T04:51:02.749154+0000","last_peered":"2024-04-02T04:51:02.749154+0000","last_clean":"2024-04-02T04:51:02.749154+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:51:02.749154+0000","last_undegraded":"2024-04-02T04:51:02.749154+0000","last_fullsized":"2024-04-02T04:51:02.749154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:49:15.791281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.791281+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":346,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.743542+0000","last_change":"2024-04-02T04:49:47.508665+0000","last_active":"2024-04-02T04:51:02.743542+0000","last_peered":"2024-04-02T04:51:02.743542+0000","last_clean":"2024-04-02T04:49:45.365160+0000","last_became_active":"2024-04-02T04:49:47.508665+0000","last_became_peered":"2024-04-02T04:49:47.508665+0000","last_unstale":"2024-04-02T04:51:02.743542+0000","last_undegraded":"2024-04-02T04:49:47.437341+0000","last_fullsized":"2024-04-02T04:49:47.437226+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:49:13.057821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.057821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":332,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730400+0000","last_change":"2024-04-02T04:50:40.208265+0000","last_active":"2024-04-02T04:51:02.730400+0000","last_peered":"2024-04-02T04:51:02.730400+0000","last_clean":"2024-04-02T04:51:02.730400+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:51:02.730400+0000","last_undegraded":"2024-04-02T04:51:02.730400+0000","last_fullsized":"2024-04-02T04:51:02.730400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.208005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.208005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:50.016358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"139'5","reported_seq":166,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.723244+0000","last_change":"2024-04-02T04:49:47.503964+0000","last_active":"2024-04-02T04:51:02.723244+0000","last_peered":"2024-04-02T04:51:02.723244+0000","last_clean":"2024-04-02T04:49:27.480374+0000","last_became_active":"2024-04-02T04:49:47.503964+0000","last_became_peered":"2024-04-02T04:49:47.503964+0000","last_unstale":"2024-04-02T04:51:02.723244+0000","last_undegraded":"2024-04-02T04:49:47.388434+0000","last_fullsized":"2024-04-02T04:49:47.383380+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.767650+0000","last_change":"2024-04-02T04:49:26.199475+0000","last_active":"2024-04-02T04:51:02.767650+0000","last_peered":"2024-04-02T04:51:02.767650+0000","last_clean":"2024-04-02T04:51:02.767650+0000","last_became_active":"2024-04-02T04:49:26.198942+0000","last_became_peered":"2024-04-02T04:49:26.198942+0000","last_unstale":"2024-04-02T04:51:02.767650+0000","last_undegraded":"2024-04-02T04:51:02.767650+0000","last_fullsized":"2024-04-02T04:51:02.767650+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":424,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731231+0000","last_change":"2024-04-02T04:49:16.603545+0000","last_active":"2024-04-02T04:51:02.731231+0000","last_peered":"2024-04-02T04:51:02.731231+0000","last_clean":"2024-04-02T04:51:02.731231+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:51:02.731231+0000","last_undegraded":"2024-04-02T04:51:02.731231+0000","last_fullsized":"2024-04-02T04:51:02.731231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:16.603444+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:49:16.603444+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":199,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.750258+0000","last_change":"2024-04-02T04:49:47.489637+0000","last_active":"2024-04-02T04:51:02.750258+0000","last_peered":"2024-04-02T04:51:02.750258+0000","last_clean":"2024-04-02T04:44:01.859319+0000","last_became_active":"2024-04-02T04:49:47.489637+0000","last_became_peered":"2024-04-02T04:49:47.489637+0000","last_unstale":"2024-04-02T04:51:02.750258+0000","last_undegraded":"2024-04-02T04:49:47.370487+0000","last_fullsized":"2024-04-02T04:49:47.370305+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":354,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.758326+0000","last_change":"2024-04-02T04:49:47.409343+0000","last_active":"2024-04-02T04:51:02.758326+0000","last_peered":"2024-04-02T04:51:02.758326+0000","last_clean":"2024-04-02T04:49:45.339425+0000","last_became_active":"2024-04-02T04:49:47.409343+0000","last_became_peered":"2024-04-02T04:49:47.409343+0000","last_unstale":"2024-04-02T04:51:02.758326+0000","last_undegraded":"2024-04-02T04:49:47.369170+0000","last_fullsized":"2024-04-02T04:49:47.369009+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:48:39.608255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.608255+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":319,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.717823+0000","last_change":"2024-04-02T04:49:47.502658+0000","last_active":"2024-04-02T04:51:02.717823+0000","last_peered":"2024-04-02T04:51:02.717823+0000","last_clean":"2024-04-02T04:49:45.343348+0000","last_became_active":"2024-04-02T04:49:47.502658+0000","last_became_peered":"2024-04-02T04:49:47.502658+0000","last_unstale":"2024-04-02T04:51:02.717823+0000","last_undegraded":"2024-04-02T04:51:02.717823+0000","last_fullsized":"2024-04-02T04:49:47.417371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.488397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.488397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731272+0000","last_change":"2024-04-02T04:49:26.295160+0000","last_active":"2024-04-02T04:51:02.731272+0000","last_peered":"2024-04-02T04:51:02.731272+0000","last_clean":"2024-04-02T04:51:02.731272+0000","last_became_active":"2024-04-02T04:49:26.292029+0000","last_became_peered":"2024-04-02T04:49:26.292029+0000","last_unstale":"2024-04-02T04:51:02.731272+0000","last_undegraded":"2024-04-02T04:51:02.731272+0000","last_fullsized":"2024-04-02T04:51:02.731272+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":199,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.745602+0000","last_change":"2024-04-02T04:49:47.409578+0000","last_active":"2024-04-02T04:51:02.745602+0000","last_peered":"2024-04-02T04:51:02.745602+0000","last_clean":"2024-04-02T04:44:01.867622+0000","last_became_active":"2024-04-02T04:49:47.409578+0000","last_became_peered":"2024-04-02T04:49:47.409578+0000","last_unstale":"2024-04-02T04:51:02.745602+0000","last_undegraded":"2024-04-02T04:49:47.352200+0000","last_fullsized":"2024-04-02T04:49:47.352060+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":342,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731627+0000","last_change":"2024-04-02T04:50:20.172828+0000","last_active":"2024-04-02T04:51:02.731627+0000","last_peered":"2024-04-02T04:51:02.731627+0000","last_clean":"2024-04-02T04:51:02.731627+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:51:02.731627+0000","last_undegraded":"2024-04-02T04:51:02.731627+0000","last_fullsized":"2024-04-02T04:51:02.731627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:20.172630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.172630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:45.769221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":322,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.747985+0000","last_change":"2024-04-02T04:49:47.492319+0000","last_active":"2024-04-02T04:51:02.747985+0000","last_peered":"2024-04-02T04:51:02.747985+0000","last_clean":"2024-04-02T04:49:45.387118+0000","last_became_active":"2024-04-02T04:49:47.492319+0000","last_became_peered":"2024-04-02T04:49:47.492319+0000","last_unstale":"2024-04-02T04:51:02.747985+0000","last_undegraded":"2024-04-02T04:51:02.747985+0000","last_fullsized":"2024-04-02T04:49:47.417925+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.103147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":317,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.736851+0000","last_change":"2024-04-02T04:49:47.511784+0000","last_active":"2024-04-02T04:51:02.736851+0000","last_peered":"2024-04-02T04:51:02.736851+0000","last_clean":"2024-04-02T04:49:45.315951+0000","last_became_active":"2024-04-02T04:49:47.511784+0000","last_became_peered":"2024-04-02T04:49:47.511784+0000","last_unstale":"2024-04-02T04:51:02.736851+0000","last_undegraded":"2024-04-02T04:51:02.736851+0000","last_fullsized":"2024-04-02T04:49:47.381831+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.646709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.646709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"139'5","reported_seq":188,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.767417+0000","last_change":"2024-04-02T04:49:26.178484+0000","last_active":"2024-04-02T04:51:02.767417+0000","last_peered":"2024-04-02T04:51:02.767417+0000","last_clean":"2024-04-02T04:51:02.767417+0000","last_became_active":"2024-04-02T04:49:26.178040+0000","last_became_peered":"2024-04-02T04:49:26.178040+0000","last_unstale":"2024-04-02T04:51:02.767417+0000","last_undegraded":"2024-04-02T04:51:02.767417+0000","last_fullsized":"2024-04-02T04:51:02.767417+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727299+0000","last_change":"2024-04-02T04:49:26.259224+0000","last_active":"2024-04-02T04:51:02.727299+0000","last_peered":"2024-04-02T04:51:02.727299+0000","last_clean":"2024-04-02T04:51:02.727299+0000","last_became_active":"2024-04-02T04:49:26.258674+0000","last_became_peered":"2024-04-02T04:49:26.258674+0000","last_unstale":"2024-04-02T04:51:02.727299+0000","last_undegraded":"2024-04-02T04:51:02.727299+0000","last_fullsized":"2024-04-02T04:51:02.727299+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":338,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741546+0000","last_change":"2024-04-02T04:50:17.746744+0000","last_active":"2024-04-02T04:51:02.741546+0000","last_peered":"2024-04-02T04:51:02.741546+0000","last_clean":"2024-04-02T04:51:02.741546+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:51:02.741546+0000","last_undegraded":"2024-04-02T04:51:02.741546+0000","last_fullsized":"2024-04-02T04:51:02.741546+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:50:17.746578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.746578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:21.436393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.742527+0000","last_change":"2024-04-02T04:49:07.675897+0000","last_active":"2024-04-02T04:51:02.742527+0000","last_peered":"2024-04-02T04:51:02.742527+0000","last_clean":"2024-04-02T04:51:02.742527+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:51:02.742527+0000","last_undegraded":"2024-04-02T04:51:02.742527+0000","last_fullsized":"2024-04-02T04:51:02.742527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.675606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":328,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.725490+0000","last_change":"2024-04-02T04:49:47.477916+0000","last_active":"2024-04-02T04:51:02.725490+0000","last_peered":"2024-04-02T04:51:02.725490+0000","last_clean":"2024-04-02T04:49:45.322247+0000","last_became_active":"2024-04-02T04:49:47.477916+0000","last_became_peered":"2024-04-02T04:49:47.477916+0000","last_unstale":"2024-04-02T04:51:02.725490+0000","last_undegraded":"2024-04-02T04:51:02.725490+0000","last_fullsized":"2024-04-02T04:49:47.406503+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.485165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.485165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.773611+0000","last_change":"2024-04-02T04:49:47.412587+0000","last_active":"2024-04-02T04:51:02.773611+0000","last_peered":"2024-04-02T04:51:02.773611+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.412587+0000","last_became_peered":"2024-04-02T04:49:47.412587+0000","last_unstale":"2024-04-02T04:51:02.773611+0000","last_undegraded":"2024-04-02T04:51:02.773611+0000","last_fullsized":"2024-04-02T04:49:47.396252+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.304153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.304153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.890989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748316+0000","last_change":"2024-04-02T04:49:26.254782+0000","last_active":"2024-04-02T04:51:02.748316+0000","last_peered":"2024-04-02T04:51:02.748316+0000","last_clean":"2024-04-02T04:51:02.748316+0000","last_became_active":"2024-04-02T04:49:26.250745+0000","last_became_peered":"2024-04-02T04:49:26.250745+0000","last_unstale":"2024-04-02T04:51:02.748316+0000","last_undegraded":"2024-04-02T04:51:02.748316+0000","last_fullsized":"2024-04-02T04:51:02.748316+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727316+0000","last_change":"2024-04-02T04:49:26.229870+0000","last_active":"2024-04-02T04:51:02.727316+0000","last_peered":"2024-04-02T04:51:02.727316+0000","last_clean":"2024-04-02T04:51:02.727316+0000","last_became_active":"2024-04-02T04:49:26.229293+0000","last_became_peered":"2024-04-02T04:49:26.229293+0000","last_unstale":"2024-04-02T04:51:02.727316+0000","last_undegraded":"2024-04-02T04:51:02.727316+0000","last_fullsized":"2024-04-02T04:51:02.727316+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":187,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.718405+0000","last_change":"2024-04-02T04:49:47.452491+0000","last_active":"2024-04-02T04:51:02.718405+0000","last_peered":"2024-04-02T04:51:02.718405+0000","last_clean":"2024-04-02T04:44:01.831181+0000","last_became_active":"2024-04-02T04:49:47.452491+0000","last_became_peered":"2024-04-02T04:49:47.452491+0000","last_unstale":"2024-04-02T04:51:02.718405+0000","last_undegraded":"2024-04-02T04:49:47.355108+0000","last_fullsized":"2024-04-02T04:49:47.354938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":329,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.754745+0000","last_change":"2024-04-02T04:49:47.419557+0000","last_active":"2024-04-02T04:51:02.754745+0000","last_peered":"2024-04-02T04:51:02.754745+0000","last_clean":"2024-04-02T04:49:45.336531+0000","last_became_active":"2024-04-02T04:49:47.419557+0000","last_became_peered":"2024-04-02T04:49:47.419557+0000","last_unstale":"2024-04-02T04:51:02.754745+0000","last_undegraded":"2024-04-02T04:51:02.754745+0000","last_fullsized":"2024-04-02T04:49:47.341828+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.899356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.899356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"153'13","reported_seq":421,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.720465+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:51:02.720465+0000","last_peered":"2024-04-02T04:51:02.720465+0000","last_clean":"2024-04-02T04:49:45.328987+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:51:02.720465+0000","last_undegraded":"2024-04-02T04:49:47.427380+0000","last_fullsized":"2024-04-02T04:49:47.427156+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.810361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":318,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.747940+0000","last_change":"2024-04-02T04:49:47.491986+0000","last_active":"2024-04-02T04:51:02.747940+0000","last_peered":"2024-04-02T04:51:02.747940+0000","last_clean":"2024-04-02T04:49:45.387071+0000","last_became_active":"2024-04-02T04:49:47.491986+0000","last_became_peered":"2024-04-02T04:49:47.491986+0000","last_unstale":"2024-04-02T04:51:02.747940+0000","last_undegraded":"2024-04-02T04:51:02.747940+0000","last_fullsized":"2024-04-02T04:49:47.416002+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.986307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.986307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"139'7","reported_seq":168,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.729161+0000","last_change":"2024-04-02T04:49:47.507515+0000","last_active":"2024-04-02T04:51:02.729161+0000","last_peered":"2024-04-02T04:51:02.729161+0000","last_clean":"2024-04-02T04:49:27.511662+0000","last_became_active":"2024-04-02T04:49:47.507515+0000","last_became_peered":"2024-04-02T04:49:47.507515+0000","last_unstale":"2024-04-02T04:51:02.729161+0000","last_undegraded":"2024-04-02T04:49:47.419015+0000","last_fullsized":"2024-04-02T04:49:47.418904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":187,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.716362+0000","last_change":"2024-04-02T04:49:47.480176+0000","last_active":"2024-04-02T04:51:02.716362+0000","last_peered":"2024-04-02T04:51:02.716362+0000","last_clean":"2024-04-02T04:49:45.335187+0000","last_became_active":"2024-04-02T04:49:47.480176+0000","last_became_peered":"2024-04-02T04:49:47.480176+0000","last_unstale":"2024-04-02T04:51:02.716362+0000","last_undegraded":"2024-04-02T04:51:02.716362+0000","last_fullsized":"2024-04-02T04:49:47.401165+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":351,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.721661+0000","last_change":"2024-04-02T04:49:47.492714+0000","last_active":"2024-04-02T04:51:02.721661+0000","last_peered":"2024-04-02T04:51:02.721661+0000","last_clean":"2024-04-02T04:49:45.370028+0000","last_became_active":"2024-04-02T04:49:47.492714+0000","last_became_peered":"2024-04-02T04:49:47.492714+0000","last_unstale":"2024-04-02T04:51:02.721661+0000","last_undegraded":"2024-04-02T04:49:47.390470+0000","last_fullsized":"2024-04-02T04:49:47.390328+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:49:14.989379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.989379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731107+0000","last_change":"2024-04-02T04:50:21.021320+0000","last_active":"2024-04-02T04:51:02.731107+0000","last_peered":"2024-04-02T04:51:02.731107+0000","last_clean":"2024-04-02T04:51:02.731107+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:51:02.731107+0000","last_undegraded":"2024-04-02T04:51:02.731107+0000","last_fullsized":"2024-04-02T04:51:02.731107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.021194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.021194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:46.253328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"153'12","reported_seq":420,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.750541+0000","last_change":"2024-04-02T04:49:47.418645+0000","last_active":"2024-04-02T04:51:02.750541+0000","last_peered":"2024-04-02T04:51:02.750541+0000","last_clean":"2024-04-02T04:49:45.363313+0000","last_became_active":"2024-04-02T04:49:47.418645+0000","last_became_peered":"2024-04-02T04:49:47.418645+0000","last_unstale":"2024-04-02T04:51:02.750541+0000","last_undegraded":"2024-04-02T04:49:47.360244+0000","last_fullsized":"2024-04-02T04:49:47.360013+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:41.932321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.932321+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":322,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.761495+0000","last_change":"2024-04-02T04:50:42.526150+0000","last_active":"2024-04-02T04:51:02.761495+0000","last_peered":"2024-04-02T04:51:02.761495+0000","last_clean":"2024-04-02T04:51:02.761495+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:51:02.761495+0000","last_undegraded":"2024-04-02T04:51:02.761495+0000","last_fullsized":"2024-04-02T04:51:02.761495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:42.525995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:42.525995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:58.603189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.750999+0000","last_change":"2024-04-02T04:49:26.281485+0000","last_active":"2024-04-02T04:51:02.750999+0000","last_peered":"2024-04-02T04:51:02.750999+0000","last_clean":"2024-04-02T04:51:02.750999+0000","last_became_active":"2024-04-02T04:49:26.280984+0000","last_became_peered":"2024-04-02T04:49:26.280984+0000","last_unstale":"2024-04-02T04:51:02.750999+0000","last_undegraded":"2024-04-02T04:51:02.750999+0000","last_fullsized":"2024-04-02T04:51:02.750999+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751264+0000","last_change":"2024-04-02T04:50:25.822947+0000","last_active":"2024-04-02T04:51:02.751264+0000","last_peered":"2024-04-02T04:51:02.751264+0000","last_clean":"2024-04-02T04:51:02.751264+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:51:02.751264+0000","last_undegraded":"2024-04-02T04:51:02.751264+0000","last_fullsized":"2024-04-02T04:51:02.751264+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:25.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:40.783877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":196,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.720654+0000","last_change":"2024-04-02T04:49:47.479230+0000","last_active":"2024-04-02T04:51:02.720654+0000","last_peered":"2024-04-02T04:51:02.720654+0000","last_clean":"2024-04-02T04:44:01.845646+0000","last_became_active":"2024-04-02T04:49:47.479230+0000","last_became_peered":"2024-04-02T04:49:47.479230+0000","last_unstale":"2024-04-02T04:51:02.720654+0000","last_undegraded":"2024-04-02T04:49:47.401783+0000","last_fullsized":"2024-04-02T04:49:47.401678+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:49:23.491068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:23.491068+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751874+0000","last_change":"2024-04-02T04:50:21.847768+0000","last_active":"2024-04-02T04:51:02.751874+0000","last_peered":"2024-04-02T04:51:02.751874+0000","last_clean":"2024-04-02T04:51:02.751874+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:51:02.751874+0000","last_undegraded":"2024-04-02T04:51:02.751874+0000","last_fullsized":"2024-04-02T04:51:02.751874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.847551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.847551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:50.902451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727777+0000","last_change":"2024-04-02T04:50:10.316715+0000","last_active":"2024-04-02T04:51:02.727777+0000","last_peered":"2024-04-02T04:51:02.727777+0000","last_clean":"2024-04-02T04:51:02.727777+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:51:02.727777+0000","last_undegraded":"2024-04-02T04:51:02.727777+0000","last_fullsized":"2024-04-02T04:51:02.727777+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:10.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:10.316429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:26.018226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":324,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730289+0000","last_change":"2024-04-02T04:50:11.029007+0000","last_active":"2024-04-02T04:51:02.730289+0000","last_peered":"2024-04-02T04:51:02.730289+0000","last_clean":"2024-04-02T04:51:02.730289+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:51:02.730289+0000","last_undegraded":"2024-04-02T04:51:02.730289+0000","last_fullsized":"2024-04-02T04:51:02.730289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.028926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.028926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:18.528508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"139'10","reported_seq":193,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717040+0000","last_change":"2024-04-02T04:49:26.267537+0000","last_active":"2024-04-02T04:51:02.717040+0000","last_peered":"2024-04-02T04:51:02.717040+0000","last_clean":"2024-04-02T04:51:02.717040+0000","last_became_active":"2024-04-02T04:49:26.260148+0000","last_became_peered":"2024-04-02T04:49:26.260148+0000","last_unstale":"2024-04-02T04:51:02.717040+0000","last_undegraded":"2024-04-02T04:51:02.717040+0000","last_fullsized":"2024-04-02T04:51:02.717040+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717965+0000","last_change":"2024-04-02T04:49:26.264150+0000","last_active":"2024-04-02T04:51:02.717965+0000","last_peered":"2024-04-02T04:51:02.717965+0000","last_clean":"2024-04-02T04:51:02.717965+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:51:02.717965+0000","last_undegraded":"2024-04-02T04:51:02.717965+0000","last_fullsized":"2024-04-02T04:51:02.717965+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":365,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.729741+0000","last_change":"2024-04-02T04:49:47.392866+0000","last_active":"2024-04-02T04:51:02.729741+0000","last_peered":"2024-04-02T04:51:02.729741+0000","last_clean":"2024-04-02T04:49:45.343613+0000","last_became_active":"2024-04-02T04:49:47.392866+0000","last_became_peered":"2024-04-02T04:49:47.392866+0000","last_unstale":"2024-04-02T04:51:02.729741+0000","last_undegraded":"2024-04-02T04:49:47.367503+0000","last_fullsized":"2024-04-02T04:49:47.367381+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":334,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.738650+0000","last_change":"2024-04-02T04:50:19.341033+0000","last_active":"2024-04-02T04:51:02.738650+0000","last_peered":"2024-04-02T04:51:02.738650+0000","last_clean":"2024-04-02T04:51:02.738650+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:51:02.738650+0000","last_undegraded":"2024-04-02T04:51:02.738650+0000","last_fullsized":"2024-04-02T04:51:02.738650+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:19.340953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.340953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:28.851404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717680+0000","last_change":"2024-04-02T04:49:12.685154+0000","last_active":"2024-04-02T04:51:02.717680+0000","last_peered":"2024-04-02T04:51:02.717680+0000","last_clean":"2024-04-02T04:51:02.717680+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:51:02.717680+0000","last_undegraded":"2024-04-02T04:51:02.717680+0000","last_fullsized":"2024-04-02T04:51:02.717680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.684916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.684916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":324,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.729673+0000","last_change":"2024-04-02T04:50:32.383573+0000","last_active":"2024-04-02T04:51:02.729673+0000","last_peered":"2024-04-02T04:51:02.729673+0000","last_clean":"2024-04-02T04:51:02.729673+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:51:02.729673+0000","last_undegraded":"2024-04-02T04:51:02.729673+0000","last_fullsized":"2024-04-02T04:51:02.729673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.383421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:50.603625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"139'5","reported_seq":188,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.728378+0000","last_change":"2024-04-02T04:49:26.283455+0000","last_active":"2024-04-02T04:51:02.728378+0000","last_peered":"2024-04-02T04:51:02.728378+0000","last_clean":"2024-04-02T04:51:02.728378+0000","last_became_active":"2024-04-02T04:49:26.275418+0000","last_became_peered":"2024-04-02T04:49:26.275418+0000","last_unstale":"2024-04-02T04:51:02.728378+0000","last_undegraded":"2024-04-02T04:51:02.728378+0000","last_fullsized":"2024-04-02T04:51:02.728378+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":192,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741709+0000","last_change":"2024-04-02T04:50:30.067941+0000","last_active":"2024-04-02T04:51:02.741709+0000","last_peered":"2024-04-02T04:51:02.741709+0000","last_clean":"2024-04-02T04:51:02.741709+0000","last_became_active":"2024-04-02T04:49:25.080486+0000","last_became_peered":"2024-04-02T04:49:25.080486+0000","last_unstale":"2024-04-02T04:51:02.741709+0000","last_undegraded":"2024-04-02T04:51:02.741709+0000","last_fullsized":"2024-04-02T04:51:02.741709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:50:30.067845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:41.056343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":352,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.748858+0000","last_change":"2024-04-02T04:50:25.999986+0000","last_active":"2024-04-02T04:51:02.748858+0000","last_peered":"2024-04-02T04:51:02.748858+0000","last_clean":"2024-04-02T04:51:02.748858+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:51:02.748858+0000","last_undegraded":"2024-04-02T04:51:02.748858+0000","last_fullsized":"2024-04-02T04:51:02.748858+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:50:25.999894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.999894+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.329856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":341,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.762111+0000","last_change":"2024-04-02T04:50:39.424795+0000","last_active":"2024-04-02T04:51:02.762111+0000","last_peered":"2024-04-02T04:51:02.762111+0000","last_clean":"2024-04-02T04:51:02.762111+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:51:02.762111+0000","last_undegraded":"2024-04-02T04:51:02.762111+0000","last_fullsized":"2024-04-02T04:51:02.762111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.424575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:05.259870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":328,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712963+0000","last_change":"2024-04-02T04:49:18.346180+0000","last_active":"2024-04-02T04:51:02.712963+0000","last_peered":"2024-04-02T04:51:02.712963+0000","last_clean":"2024-04-02T04:51:02.712963+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:51:02.712963+0000","last_undegraded":"2024-04-02T04:51:02.712963+0000","last_fullsized":"2024-04-02T04:51:02.712963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:18.346039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:18.346039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.738059+0000","last_change":"2024-04-02T04:49:47.512747+0000","last_active":"2024-04-02T04:51:02.738059+0000","last_peered":"2024-04-02T04:51:02.738059+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.512747+0000","last_became_peered":"2024-04-02T04:49:47.512747+0000","last_unstale":"2024-04-02T04:51:02.738059+0000","last_undegraded":"2024-04-02T04:51:02.738059+0000","last_fullsized":"2024-04-02T04:49:47.402486+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.372299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.372299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":188,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.726982+0000","last_change":"2024-04-02T04:49:47.444258+0000","last_active":"2024-04-02T04:51:02.726982+0000","last_peered":"2024-04-02T04:51:02.726982+0000","last_clean":"2024-04-02T04:49:45.328612+0000","last_became_active":"2024-04-02T04:49:47.444258+0000","last_became_peered":"2024-04-02T04:49:47.444258+0000","last_unstale":"2024-04-02T04:51:02.726982+0000","last_undegraded":"2024-04-02T04:51:02.726982+0000","last_fullsized":"2024-04-02T04:49:47.397778+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":152,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.742095+0000","last_change":"2024-04-02T04:49:47.517978+0000","last_active":"2024-04-02T04:51:02.742095+0000","last_peered":"2024-04-02T04:51:02.742095+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.517978+0000","last_became_peered":"2024-04-02T04:49:47.517978+0000","last_unstale":"2024-04-02T04:51:02.742095+0000","last_undegraded":"2024-04-02T04:51:02.742095+0000","last_fullsized":"2024-04-02T04:49:47.456750+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":372,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.759842+0000","last_change":"2024-04-02T04:49:47.430162+0000","last_active":"2024-04-02T04:51:02.759842+0000","last_peered":"2024-04-02T04:51:02.759842+0000","last_clean":"2024-04-02T04:49:45.364593+0000","last_became_active":"2024-04-02T04:49:47.430162+0000","last_became_peered":"2024-04-02T04:49:47.430162+0000","last_unstale":"2024-04-02T04:51:02.759842+0000","last_undegraded":"2024-04-02T04:49:47.355994+0000","last_fullsized":"2024-04-02T04:49:47.355865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:49:03.788837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.788837+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":343,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.767851+0000","last_change":"2024-04-02T04:50:08.818906+0000","last_active":"2024-04-02T04:51:02.767851+0000","last_peered":"2024-04-02T04:51:02.767851+0000","last_clean":"2024-04-02T04:51:02.767851+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:51:02.767851+0000","last_undegraded":"2024-04-02T04:51:02.767851+0000","last_fullsized":"2024-04-02T04:51:02.767851+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:08.818819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:50:08.818819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:37.696499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":323,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.726253+0000","last_change":"2024-04-02T04:49:47.443764+0000","last_active":"2024-04-02T04:51:02.726253+0000","last_peered":"2024-04-02T04:51:02.726253+0000","last_clean":"2024-04-02T04:49:45.325466+0000","last_became_active":"2024-04-02T04:49:47.443764+0000","last_became_peered":"2024-04-02T04:49:47.443764+0000","last_unstale":"2024-04-02T04:51:02.726253+0000","last_undegraded":"2024-04-02T04:51:02.726253+0000","last_fullsized":"2024-04-02T04:49:47.396652+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.351209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.810497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":316,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.718059+0000","last_change":"2024-04-02T04:49:25.285134+0000","last_active":"2024-04-02T04:51:02.718059+0000","last_peered":"2024-04-02T04:51:02.718059+0000","last_clean":"2024-04-02T04:51:02.718059+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:51:02.718059+0000","last_undegraded":"2024-04-02T04:51:02.718059+0000","last_fullsized":"2024-04-02T04:51:02.718059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.284922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.284922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.810750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"138'2","reported_seq":161,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.719193+0000","last_change":"2024-04-02T04:49:47.436385+0000","last_active":"2024-04-02T04:51:02.719193+0000","last_peered":"2024-04-02T04:51:02.719193+0000","last_clean":"2024-04-02T04:49:26.658740+0000","last_became_active":"2024-04-02T04:49:47.436385+0000","last_became_peered":"2024-04-02T04:49:47.436385+0000","last_unstale":"2024-04-02T04:51:02.719193+0000","last_undegraded":"2024-04-02T04:51:02.719193+0000","last_fullsized":"2024-04-02T04:49:47.371603+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.715532+0000","last_change":"2024-04-02T04:49:47.393078+0000","last_active":"2024-04-02T04:51:02.715532+0000","last_peered":"2024-04-02T04:51:02.715532+0000","last_clean":"2024-04-02T04:49:45.306613+0000","last_became_active":"2024-04-02T04:49:47.393078+0000","last_became_peered":"2024-04-02T04:49:47.393078+0000","last_unstale":"2024-04-02T04:51:02.715532+0000","last_undegraded":"2024-04-02T04:51:02.715532+0000","last_fullsized":"2024-04-02T04:49:47.373401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"133'50","reported_seq":414,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727448+0000","last_change":"2024-04-02T04:50:33.298622+0000","last_active":"2024-04-02T04:51:02.727448+0000","last_peered":"2024-04-02T04:51:02.727448+0000","last_clean":"2024-04-02T04:51:02.727448+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:51:02.727448+0000","last_undegraded":"2024-04-02T04:51:02.727448+0000","last_fullsized":"2024-04-02T04:51:02.727448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:50:33.298479+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:50:33.298479+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:40.620907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.745673+0000","last_change":"2024-04-02T04:49:47.388339+0000","last_active":"2024-04-02T04:51:02.745673+0000","last_peered":"2024-04-02T04:51:02.745673+0000","last_clean":"2024-04-02T04:49:45.330232+0000","last_became_active":"2024-04-02T04:49:47.388339+0000","last_became_peered":"2024-04-02T04:49:47.388339+0000","last_unstale":"2024-04-02T04:51:02.745673+0000","last_undegraded":"2024-04-02T04:51:02.745673+0000","last_fullsized":"2024-04-02T04:49:47.337757+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:51.870038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:51.870038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":331,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.741908+0000","last_change":"2024-04-02T04:49:47.491600+0000","last_active":"2024-04-02T04:51:02.741908+0000","last_peered":"2024-04-02T04:51:02.741908+0000","last_clean":"2024-04-02T04:49:45.386942+0000","last_became_active":"2024-04-02T04:49:47.491600+0000","last_became_peered":"2024-04-02T04:49:47.491600+0000","last_unstale":"2024-04-02T04:51:02.741908+0000","last_undegraded":"2024-04-02T04:51:02.741908+0000","last_fullsized":"2024-04-02T04:49:47.415538+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":160,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.739090+0000","last_change":"2024-04-02T04:49:47.434796+0000","last_active":"2024-04-02T04:51:02.739090+0000","last_peered":"2024-04-02T04:51:02.739090+0000","last_clean":"2024-04-02T04:44:01.344239+0000","last_became_active":"2024-04-02T04:49:47.434796+0000","last_became_peered":"2024-04-02T04:49:47.434796+0000","last_unstale":"2024-04-02T04:51:02.739090+0000","last_undegraded":"2024-04-02T04:49:47.346909+0000","last_fullsized":"2024-04-02T04:49:47.346781+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:48:42.394228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.394228+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.17","version":"139'5","reported_seq":188,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.755736+0000","last_change":"2024-04-02T04:49:26.275021+0000","last_active":"2024-04-02T04:51:02.755736+0000","last_peered":"2024-04-02T04:51:02.755736+0000","last_clean":"2024-04-02T04:51:02.755736+0000","last_became_active":"2024-04-02T04:49:26.272870+0000","last_became_peered":"2024-04-02T04:49:26.272870+0000","last_unstale":"2024-04-02T04:51:02.755736+0000","last_undegraded":"2024-04-02T04:51:02.755736+0000","last_fullsized":"2024-04-02T04:51:02.755736+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.751206+0000","last_change":"2024-04-02T04:49:47.478289+0000","last_active":"2024-04-02T04:51:02.751206+0000","last_peered":"2024-04-02T04:51:02.751206+0000","last_clean":"2024-04-02T04:49:45.332087+0000","last_became_active":"2024-04-02T04:49:47.478289+0000","last_became_peered":"2024-04-02T04:49:47.478289+0000","last_unstale":"2024-04-02T04:51:02.751206+0000","last_undegraded":"2024-04-02T04:51:02.751206+0000","last_fullsized":"2024-04-02T04:49:47.337387+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":359,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.712842+0000","last_change":"2024-04-02T04:50:14.221770+0000","last_active":"2024-04-02T04:51:02.712842+0000","last_peered":"2024-04-02T04:51:02.712842+0000","last_clean":"2024-04-02T04:51:02.712842+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:51:02.712842+0000","last_undegraded":"2024-04-02T04:51:02.712842+0000","last_fullsized":"2024-04-02T04:51:02.712842+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:50:14.221684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.221684+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:30.077859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":337,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.726353+0000","last_change":"2024-04-02T04:50:28.247971+0000","last_active":"2024-04-02T04:51:02.726353+0000","last_peered":"2024-04-02T04:51:02.726353+0000","last_clean":"2024-04-02T04:51:02.726353+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:51:02.726353+0000","last_undegraded":"2024-04-02T04:51:02.726353+0000","last_fullsized":"2024-04-02T04:51:02.726353+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:50:28.247657+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.247657+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:52.942034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":427,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.770832+0000","last_change":"2024-04-02T04:50:12.840880+0000","last_active":"2024-04-02T04:51:02.770832+0000","last_peered":"2024-04-02T04:51:02.770832+0000","last_clean":"2024-04-02T04:51:02.770832+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:51:02.770832+0000","last_undegraded":"2024-04-02T04:51:02.770832+0000","last_fullsized":"2024-04-02T04:51:02.770832+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:12.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.840785+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:28.314441+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":322,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.738958+0000","last_change":"2024-04-02T04:50:15.299404+0000","last_active":"2024-04-02T04:51:02.738958+0000","last_peered":"2024-04-02T04:51:02.738958+0000","last_clean":"2024-04-02T04:51:02.738958+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:51:02.738958+0000","last_undegraded":"2024-04-02T04:51:02.738958+0000","last_fullsized":"2024-04-02T04:51:02.738958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.299316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.299316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:43.462635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"139'5","reported_seq":190,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.724456+0000","last_change":"2024-04-02T04:49:25.091835+0000","last_active":"2024-04-02T04:51:02.724456+0000","last_peered":"2024-04-02T04:51:02.724456+0000","last_clean":"2024-04-02T04:51:02.724456+0000","last_became_active":"2024-04-02T04:49:25.091539+0000","last_became_peered":"2024-04-02T04:49:25.091539+0000","last_unstale":"2024-04-02T04:51:02.724456+0000","last_undegraded":"2024-04-02T04:51:02.724456+0000","last_fullsized":"2024-04-02T04:51:02.724456+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.733508+0000","last_change":"2024-04-02T04:50:37.840934+0000","last_active":"2024-04-02T04:51:02.733508+0000","last_peered":"2024-04-02T04:51:02.733508+0000","last_clean":"2024-04-02T04:51:02.733508+0000","last_became_active":"2024-04-02T04:49:26.256165+0000","last_became_peered":"2024-04-02T04:49:26.256165+0000","last_unstale":"2024-04-02T04:51:02.733508+0000","last_undegraded":"2024-04-02T04:51:02.733508+0000","last_fullsized":"2024-04-02T04:51:02.733508+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.840813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.840813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:59.655281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":351,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.731290+0000","last_change":"2024-04-02T04:49:47.432220+0000","last_active":"2024-04-02T04:51:02.731290+0000","last_peered":"2024-04-02T04:51:02.731290+0000","last_clean":"2024-04-02T04:49:45.374676+0000","last_became_active":"2024-04-02T04:49:47.432220+0000","last_became_peered":"2024-04-02T04:49:47.432220+0000","last_unstale":"2024-04-02T04:51:02.731290+0000","last_undegraded":"2024-04-02T04:49:47.384090+0000","last_fullsized":"2024-04-02T04:49:47.383905+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:49:05.019939+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.019939+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":338,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.733869+0000","last_change":"2024-04-02T04:49:47.404868+0000","last_active":"2024-04-02T04:51:02.733869+0000","last_peered":"2024-04-02T04:51:02.733869+0000","last_clean":"2024-04-02T04:49:45.386009+0000","last_became_active":"2024-04-02T04:49:47.404868+0000","last_became_peered":"2024-04-02T04:49:47.404868+0000","last_unstale":"2024-04-02T04:51:02.733869+0000","last_undegraded":"2024-04-02T04:51:02.733869+0000","last_fullsized":"2024-04-02T04:49:47.375401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:14.778569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.778569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":325,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.732845+0000","last_change":"2024-04-02T04:49:13.722710+0000","last_active":"2024-04-02T04:51:02.732845+0000","last_peered":"2024-04-02T04:51:02.732845+0000","last_clean":"2024-04-02T04:51:02.732845+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:51:02.732845+0000","last_undegraded":"2024-04-02T04:51:02.732845+0000","last_fullsized":"2024-04-02T04:51:02.732845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:13.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:07.341310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":323,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.751014+0000","last_change":"2024-04-02T04:50:27.563922+0000","last_active":"2024-04-02T04:51:02.751014+0000","last_peered":"2024-04-02T04:51:02.751014+0000","last_clean":"2024-04-02T04:51:02.751014+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:51:02.751014+0000","last_undegraded":"2024-04-02T04:51:02.751014+0000","last_fullsized":"2024-04-02T04:51:02.751014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.563824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.563824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:43.307638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.737050+0000","last_change":"2024-04-02T04:49:26.508086+0000","last_active":"2024-04-02T04:51:02.737050+0000","last_peered":"2024-04-02T04:51:02.737050+0000","last_clean":"2024-04-02T04:51:02.737050+0000","last_became_active":"2024-04-02T04:49:26.507403+0000","last_became_peered":"2024-04-02T04:49:26.507403+0000","last_unstale":"2024-04-02T04:51:02.737050+0000","last_undegraded":"2024-04-02T04:51:02.737050+0000","last_fullsized":"2024-04-02T04:51:02.737050+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":186,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.732371+0000","last_change":"2024-04-02T04:49:47.517198+0000","last_active":"2024-04-02T04:51:02.732371+0000","last_peered":"2024-04-02T04:51:02.732371+0000","last_clean":"2024-04-02T04:49:45.381250+0000","last_became_active":"2024-04-02T04:49:47.517198+0000","last_became_peered":"2024-04-02T04:49:47.517198+0000","last_unstale":"2024-04-02T04:51:02.732371+0000","last_undegraded":"2024-04-02T04:51:02.732371+0000","last_fullsized":"2024-04-02T04:49:47.430351+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":378,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.730663+0000","last_change":"2024-04-02T04:50:14.537273+0000","last_active":"2024-04-02T04:51:02.730663+0000","last_peered":"2024-04-02T04:51:02.730663+0000","last_clean":"2024-04-02T04:51:02.730663+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:51:02.730663+0000","last_undegraded":"2024-04-02T04:51:02.730663+0000","last_fullsized":"2024-04-02T04:51:02.730663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:50:14.537029+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.537029+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:35.811114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":338,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.717491+0000","last_change":"2024-04-02T04:49:47.468861+0000","last_active":"2024-04-02T04:51:02.717491+0000","last_peered":"2024-04-02T04:51:02.717491+0000","last_clean":"2024-04-02T04:49:45.338835+0000","last_became_active":"2024-04-02T04:49:47.468861+0000","last_became_peered":"2024-04-02T04:49:47.468861+0000","last_unstale":"2024-04-02T04:51:02.717491+0000","last_undegraded":"2024-04-02T04:51:02.717491+0000","last_fullsized":"2024-04-02T04:49:47.401513+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.329420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.329420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.720739+0000","last_change":"2024-04-02T04:50:29.283629+0000","last_active":"2024-04-02T04:51:02.720739+0000","last_peered":"2024-04-02T04:51:02.720739+0000","last_clean":"2024-04-02T04:51:02.720739+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:51:02.720739+0000","last_undegraded":"2024-04-02T04:51:02.720739+0000","last_fullsized":"2024-04-02T04:51:02.720739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.283405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:44.837277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":324,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.725386+0000","last_change":"2024-04-02T04:50:40.672009+0000","last_active":"2024-04-02T04:51:02.725386+0000","last_peered":"2024-04-02T04:51:02.725386+0000","last_clean":"2024-04-02T04:51:02.725386+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:51:02.725386+0000","last_undegraded":"2024-04-02T04:51:02.725386+0000","last_fullsized":"2024-04-02T04:51:02.725386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.671923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.671923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:04.472833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"138'2","reported_seq":190,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.736713+0000","last_change":"2024-04-02T04:49:47.455012+0000","last_active":"2024-04-02T04:51:02.736713+0000","last_peered":"2024-04-02T04:51:02.736713+0000","last_clean":"2024-04-02T04:49:45.373759+0000","last_became_active":"2024-04-02T04:49:47.455012+0000","last_became_peered":"2024-04-02T04:49:47.455012+0000","last_unstale":"2024-04-02T04:51:02.736713+0000","last_undegraded":"2024-04-02T04:51:02.736713+0000","last_fullsized":"2024-04-02T04:49:47.384636+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.743220+0000","last_change":"2024-04-02T04:50:39.321796+0000","last_active":"2024-04-02T04:51:02.743220+0000","last_peered":"2024-04-02T04:51:02.743220+0000","last_clean":"2024-04-02T04:51:02.743220+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:51:02.743220+0000","last_undegraded":"2024-04-02T04:51:02.743220+0000","last_fullsized":"2024-04-02T04:51:02.743220+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.321605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:42.605407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":364,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.749644+0000","last_change":"2024-04-02T04:50:13.554140+0000","last_active":"2024-04-02T04:51:02.749644+0000","last_peered":"2024-04-02T04:51:02.749644+0000","last_clean":"2024-04-02T04:51:02.749644+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:51:02.749644+0000","last_undegraded":"2024-04-02T04:51:02.749644+0000","last_fullsized":"2024-04-02T04:51:02.749644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:50:13.553801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.553801+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.234795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":157,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.740552+0000","last_change":"2024-04-02T04:49:47.435134+0000","last_active":"2024-04-02T04:51:02.740552+0000","last_peered":"2024-04-02T04:51:02.740552+0000","last_clean":"2024-04-02T04:43:54.216783+0000","last_became_active":"2024-04-02T04:49:47.435134+0000","last_became_peered":"2024-04-02T04:49:47.435134+0000","last_unstale":"2024-04-02T04:51:02.740552+0000","last_undegraded":"2024-04-02T04:49:47.360301+0000","last_fullsized":"2024-04-02T04:49:47.360179+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:48:48.436655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:48.436655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":318,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.731779+0000","last_change":"2024-04-02T04:49:01.626968+0000","last_active":"2024-04-02T04:51:02.731779+0000","last_peered":"2024-04-02T04:51:02.731779+0000","last_clean":"2024-04-02T04:51:02.731779+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:51:02.731779+0000","last_undegraded":"2024-04-02T04:51:02.731779+0000","last_fullsized":"2024-04-02T04:51:02.731779+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.626750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.626750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":318,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.722463+0000","last_change":"2024-04-02T04:49:47.495495+0000","last_active":"2024-04-02T04:51:02.722463+0000","last_peered":"2024-04-02T04:51:02.722463+0000","last_clean":"2024-04-02T04:49:45.352563+0000","last_became_active":"2024-04-02T04:49:47.495495+0000","last_became_peered":"2024-04-02T04:49:47.495495+0000","last_unstale":"2024-04-02T04:51:02.722463+0000","last_undegraded":"2024-04-02T04:51:02.722463+0000","last_fullsized":"2024-04-02T04:49:47.378202+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"139'9","reported_seq":170,"reported_epoch":223,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:51:02.732060+0000","last_change":"2024-04-02T04:49:47.518710+0000","last_active":"2024-04-02T04:51:02.732060+0000","last_peered":"2024-04-02T04:51:02.732060+0000","last_clean":"2024-04-02T04:49:27.516310+0000","last_became_active":"2024-04-02T04:49:47.518710+0000","last_became_peered":"2024-04-02T04:49:47.518710+0000","last_unstale":"2024-04-02T04:51:02.732060+0000","last_undegraded":"2024-04-02T04:49:47.393557+0000","last_fullsized":"2024-04-02T04:49:47.393345+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.284177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":184,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.720071+0000","last_change":"2024-04-02T04:49:25.094911+0000","last_active":"2024-04-02T04:51:02.720071+0000","last_peered":"2024-04-02T04:51:02.720071+0000","last_clean":"2024-04-02T04:51:02.720071+0000","last_became_active":"2024-04-02T04:49:25.094609+0000","last_became_peered":"2024-04-02T04:49:25.094609+0000","last_unstale":"2024-04-02T04:51:02.720071+0000","last_undegraded":"2024-04-02T04:51:02.720071+0000","last_fullsized":"2024-04-02T04:51:02.720071+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":354,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.741864+0000","last_change":"2024-04-02T04:50:34.479206+0000","last_active":"2024-04-02T04:51:02.741864+0000","last_peered":"2024-04-02T04:51:02.741864+0000","last_clean":"2024-04-02T04:51:02.741864+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:51:02.741864+0000","last_undegraded":"2024-04-02T04:51:02.741864+0000","last_fullsized":"2024-04-02T04:51:02.741864+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:50:34.479089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.479089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:44.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":149,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.773010+0000","last_change":"2024-04-02T04:49:47.470693+0000","last_active":"2024-04-02T04:51:02.773010+0000","last_peered":"2024-04-02T04:51:02.773010+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.470693+0000","last_became_peered":"2024-04-02T04:49:47.470693+0000","last_unstale":"2024-04-02T04:51:02.773010+0000","last_undegraded":"2024-04-02T04:51:02.773010+0000","last_fullsized":"2024-04-02T04:49:47.411420+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:22.452700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:22.452700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":326,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.729555+0000","last_change":"2024-04-02T04:50:09.464337+0000","last_active":"2024-04-02T04:51:02.729555+0000","last_peered":"2024-04-02T04:51:02.729555+0000","last_clean":"2024-04-02T04:51:02.729555+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:51:02.729555+0000","last_undegraded":"2024-04-02T04:51:02.729555+0000","last_fullsized":"2024-04-02T04:51:02.729555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.464222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.464222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:23.598443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":329,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.752298+0000","last_change":"2024-04-02T04:50:10.507552+0000","last_active":"2024-04-02T04:51:02.752298+0000","last_peered":"2024-04-02T04:51:02.752298+0000","last_clean":"2024-04-02T04:51:02.752298+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:51:02.752298+0000","last_undegraded":"2024-04-02T04:51:02.752298+0000","last_fullsized":"2024-04-02T04:51:02.752298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:10.507422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:10.507422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:31.898282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":193,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.721271+0000","last_change":"2024-04-02T04:50:29.052309+0000","last_active":"2024-04-02T04:51:02.721271+0000","last_peered":"2024-04-02T04:51:02.721271+0000","last_clean":"2024-04-02T04:51:02.721271+0000","last_became_active":"2024-04-02T04:49:25.076275+0000","last_became_peered":"2024-04-02T04:49:25.076275+0000","last_unstale":"2024-04-02T04:51:02.721271+0000","last_undegraded":"2024-04-02T04:51:02.721271+0000","last_fullsized":"2024-04-02T04:51:02.721271+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.052221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:47.466508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.765278+0000","last_change":"2024-04-02T04:49:26.280288+0000","last_active":"2024-04-02T04:51:02.765278+0000","last_peered":"2024-04-02T04:51:02.765278+0000","last_clean":"2024-04-02T04:51:02.765278+0000","last_became_active":"2024-04-02T04:49:26.275375+0000","last_became_peered":"2024-04-02T04:49:26.275375+0000","last_unstale":"2024-04-02T04:51:02.765278+0000","last_undegraded":"2024-04-02T04:51:02.765278+0000","last_fullsized":"2024-04-02T04:51:02.765278+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:06.891626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":363,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.750655+0000","last_change":"2024-04-02T04:50:32.096624+0000","last_active":"2024-04-02T04:51:02.750655+0000","last_peered":"2024-04-02T04:51:02.750655+0000","last_clean":"2024-04-02T04:51:02.750655+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:51:02.750655+0000","last_undegraded":"2024-04-02T04:51:02.750655+0000","last_fullsized":"2024-04-02T04:51:02.750655+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:50:32.096257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.096257+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:37.477437+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.721939+0000","last_change":"2024-04-02T04:50:23.234439+0000","last_active":"2024-04-02T04:51:02.721939+0000","last_peered":"2024-04-02T04:51:02.721939+0000","last_clean":"2024-04-02T04:51:02.721939+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:51:02.721939+0000","last_undegraded":"2024-04-02T04:51:02.721939+0000","last_fullsized":"2024-04-02T04:51:02.721939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:23.234223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:23.234223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:39.525539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":318,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.715796+0000","last_change":"2024-04-02T04:49:05.650145+0000","last_active":"2024-04-02T04:51:02.715796+0000","last_peered":"2024-04-02T04:51:02.715796+0000","last_clean":"2024-04-02T04:51:02.715796+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:51:02.715796+0000","last_undegraded":"2024-04-02T04:51:02.715796+0000","last_fullsized":"2024-04-02T04:51:02.715796+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.649928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":314,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.717291+0000","last_change":"2024-04-02T04:49:07.677962+0000","last_active":"2024-04-02T04:51:02.717291+0000","last_peered":"2024-04-02T04:51:02.717291+0000","last_clean":"2024-04-02T04:51:02.717291+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:51:02.717291+0000","last_undegraded":"2024-04-02T04:51:02.717291+0000","last_fullsized":"2024-04-02T04:51:02.717291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.677740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"139'7","reported_seq":190,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.727561+0000","last_change":"2024-04-02T04:49:26.246176+0000","last_active":"2024-04-02T04:51:02.727561+0000","last_peered":"2024-04-02T04:51:02.727561+0000","last_clean":"2024-04-02T04:51:02.727561+0000","last_became_active":"2024-04-02T04:49:26.241390+0000","last_became_peered":"2024-04-02T04:49:26.241390+0000","last_unstale":"2024-04-02T04:51:02.727561+0000","last_undegraded":"2024-04-02T04:51:02.727561+0000","last_fullsized":"2024-04-02T04:51:02.727561+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.758709+0000","last_change":"2024-04-02T04:49:26.537210+0000","last_active":"2024-04-02T04:51:02.758709+0000","last_peered":"2024-04-02T04:51:02.758709+0000","last_clean":"2024-04-02T04:51:02.758709+0000","last_became_active":"2024-04-02T04:49:26.536918+0000","last_became_peered":"2024-04-02T04:49:26.536918+0000","last_unstale":"2024-04-02T04:51:02.758709+0000","last_undegraded":"2024-04-02T04:51:02.758709+0000","last_fullsized":"2024-04-02T04:51:02.758709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":359,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.722055+0000","last_change":"2024-04-02T04:50:30.296523+0000","last_active":"2024-04-02T04:51:02.722055+0000","last_peered":"2024-04-02T04:51:02.722055+0000","last_clean":"2024-04-02T04:51:02.722055+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:51:02.722055+0000","last_undegraded":"2024-04-02T04:51:02.722055+0000","last_fullsized":"2024-04-02T04:51:02.722055+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:50:30.296198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:30.296198+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:41.402471+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":331,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.750914+0000","last_change":"2024-04-02T04:49:47.432079+0000","last_active":"2024-04-02T04:51:02.750914+0000","last_peered":"2024-04-02T04:51:02.750914+0000","last_clean":"2024-04-02T04:49:45.381663+0000","last_became_active":"2024-04-02T04:49:47.432079+0000","last_became_peered":"2024-04-02T04:49:47.432079+0000","last_unstale":"2024-04-02T04:51:02.750914+0000","last_undegraded":"2024-04-02T04:51:02.750914+0000","last_fullsized":"2024-04-02T04:49:47.406616+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:04.283509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.725979+0000","last_change":"2024-04-02T04:50:20.218009+0000","last_active":"2024-04-02T04:51:02.725979+0000","last_peered":"2024-04-02T04:51:02.725979+0000","last_clean":"2024-04-02T04:51:02.725979+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:51:02.725979+0000","last_undegraded":"2024-04-02T04:51:02.725979+0000","last_fullsized":"2024-04-02T04:51:02.725979+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:20.217928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.217928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:22.687003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":318,"reported_epoch":223,"state":"active+undersized","last_fresh":"2024-04-02T04:51:02.715683+0000","last_change":"2024-04-02T04:49:47.454112+0000","last_active":"2024-04-02T04:51:02.715683+0000","last_peered":"2024-04-02T04:51:02.715683+0000","last_clean":"2024-04-02T04:49:45.335087+0000","last_became_active":"2024-04-02T04:49:47.454112+0000","last_became_peered":"2024-04-02T04:49:47.454112+0000","last_unstale":"2024-04-02T04:51:02.715683+0000","last_undegraded":"2024-04-02T04:51:02.715683+0000","last_fullsized":"2024-04-02T04:49:47.400802+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.389577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.389577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T04:51:02.724584+0000","last_change":"2024-04-02T04:50:28.261755+0000","last_active":"2024-04-02T04:51:02.724584+0000","last_peered":"2024-04-02T04:51:02.724584+0000","last_clean":"2024-04-02T04:51:02.724584+0000","last_became_active":"2024-04-02T04:49:26.269993+0000","last_became_peered":"2024-04-02T04:49:26.269993+0000","last_unstale":"2024-04-02T04:51:02.724584+0000","last_undegraded":"2024-04-02T04:51:02.724584+0000","last_fullsized":"2024-04-02T04:51:02.724584+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:28.261610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.261610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:29.668362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":210,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":209,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":165,"num_pg":32,"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2392064,"data_stored":6031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":85,"acting":85,"num_store_stats":7},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":5243,"num_objects":508,"num_object_clones":402,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":1,"num_read":86,"num_read_kb":1,"num_write":510,"num_write_kb":509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5517312,"data_stored":13932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":918,"ondisk_log_size":918,"up":87,"acting":87,"num_store_stats":7},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":87,"acting":87,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":86,"acting":86,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":646,"ondisk_log_size":646,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743504,"num_pgs":239,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111160,"kb_used_data":75820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93633992,"statfs":{"total":95995035648,"available":95881207808,"internally_reserved":0,"allocated":77639680,"data_stored":76942360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103275,"num_pgs":162,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68984,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93676168,"statfs":{"total":95995035648,"available":95924396032,"internally_reserved":0,"allocated":5996544,"data_stored":2695496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67708,"kb_used_data":4568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93677444,"statfs":{"total":95995035648,"available":95925702656,"internally_reserved":0,"allocated":4677632,"data_stored":2104663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738536,"num_pgs":164,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68052,"kb_used_data":4920,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93677100,"statfs":{"total":95995035648,"available":95925350400,"internally_reserved":0,"allocated":5038080,"data_stored":2693581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378772,"num_pgs":165,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67924,"kb_used_data":4792,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93677228,"statfs":{"total":95995035648,"available":95925481472,"internally_reserved":0,"allocated":4907008,"data_stored":2105752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051722,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71764,"kb_used_data":4484,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673388,"statfs":{"total":95995035648,"available":95921549312,"internally_reserved":0,"allocated":4591616,"data_stored":2115593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822784,"num_pgs":168,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49352,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93695800,"statfs":{"total":95995035648,"available":95944499200,"internally_reserved":0,"allocated":4366336,"data_stored":2104047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463028,"num_pgs":147,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66168,"kb_used_data":4448,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93678984,"statfs":{"total":95995035648,"available":95927279616,"internally_reserved":0,"allocated":4554752,"data_stored":2104218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":1508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":929792,"data_stored":2331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":897024,"data_stored":2258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":634880,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":1161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":696320,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1302528,"data_stored":3294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":1077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":165,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":1373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:51:08.586 INFO:tasks.thrashosds.thrasher:Setting 102.7 to [5, 0, 4] 2024-04-02T04:51:08.586 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '102.7', '5', '0', '4'] 2024-04-02T04:51:08.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 102.7 5 0 4 2024-04-02T04:51:08.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:08.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:08.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:08.919 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:51:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:51:08.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: audit 2024-04-02T04:51:07.409852+0000 mgr.y (mgr.14538) 301 : audit 0 from='client.26012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:51:08.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: audit 2024-04-02T04:51:07.685181+0000 mon.a (mon.0) 1638 : audit 1 from='client.? 172.21.15.69:0/3666695339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-72153-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:08.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: cluster 2024-04-02T04:51:07.689711+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e228: 8 total, 7 up, 8 in 2024-04-02T04:51:08.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: audit 2024-04-02T04:51:07.697346+0000 mon.c (mon.2) 303 : audit 1 from='client.? 172.21.15.69:0/4172830563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:08.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: audit 2024-04-02T04:51:07.705339+0000 mon.a (mon.0) 1640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: audit 2024-04-02T04:51:07.760604+0000 mon.c (mon.2) 304 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: audit 2024-04-02T04:51:08.689291+0000 mon.a (mon.0) 1641 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[28904]: cluster 2024-04-02T04:51:08.693224+0000 mon.a (mon.0) 1642 : cluster 0 osdmap e229: 8 total, 7 up, 8 in 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: audit 2024-04-02T04:51:07.409852+0000 mgr.y (mgr.14538) 301 : audit 0 from='client.26012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: audit 2024-04-02T04:51:07.685181+0000 mon.a (mon.0) 1638 : audit 1 from='client.? 172.21.15.69:0/3666695339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-72153-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: cluster 2024-04-02T04:51:07.689711+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e228: 8 total, 7 up, 8 in 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: audit 2024-04-02T04:51:07.697346+0000 mon.c (mon.2) 303 : audit 1 from='client.? 172.21.15.69:0/4172830563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: audit 2024-04-02T04:51:07.705339+0000 mon.a (mon.0) 1640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: audit 2024-04-02T04:51:07.760604+0000 mon.c (mon.2) 304 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: audit 2024-04-02T04:51:08.689291+0000 mon.a (mon.0) 1641 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:08.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:08 smithi069 bash[21528]: cluster 2024-04-02T04:51:08.693224+0000 mon.a (mon.0) 1642 : cluster 0 osdmap e229: 8 total, 7 up, 8 in 2024-04-02T04:51:08.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:51:08.932 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:08.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: audit 2024-04-02T04:51:07.409852+0000 mgr.y (mgr.14538) 301 : audit 0 from='client.26012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: audit 2024-04-02T04:51:07.685181+0000 mon.a (mon.0) 1638 : audit 1 from='client.? 172.21.15.69:0/3666695339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-72153-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: cluster 2024-04-02T04:51:07.689711+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e228: 8 total, 7 up, 8 in 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: audit 2024-04-02T04:51:07.697346+0000 mon.c (mon.2) 303 : audit 1 from='client.? 172.21.15.69:0/4172830563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: audit 2024-04-02T04:51:07.705339+0000 mon.a (mon.0) 1640 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: audit 2024-04-02T04:51:07.760604+0000 mon.c (mon.2) 304 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: audit 2024-04-02T04:51:08.689291+0000 mon.a (mon.0) 1641 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-72145-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:08 smithi098 bash[23790]: cluster 2024-04-02T04:51:08.693224+0000 mon.a (mon.0) 1642 : cluster 0 osdmap e229: 8 total, 7 up, 8 in 2024-04-02T04:51:09.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:09.061 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:09.086 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:51:09.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:09.184 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:09.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:09.306 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:09.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:09.430 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:09.536 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:09.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:09.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: Running main() from gmock_main.cc 2024-04-02T04:51:09.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-02T04:51:09.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment set-up. 2024-04-02T04:51:09.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.TooBig (17188 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (11580 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (10503 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4613 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (6963 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3815 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3771 ms) 2024-04-02T04:51:09.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (4346 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3301 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4462 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2962 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3916 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3455 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2575 ms) 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-02T04:51:09.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Flush (2934 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3353 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3008 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2990 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3025 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3014 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2965 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3011 ms) 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-02T04:51:09.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3019 ms) 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3007 ms) 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2969 ms) 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3062 ms) 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio (119811 ms total) 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2024-04-02T04:51:09.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-02T04:51:09.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:09.812 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:09.915 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:09.938 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:09.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:09.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:09 smithi098 bash[23790]: cluster 2024-04-02T04:51:07.934821+0000 mgr.y (mgr.14538) 302 : cluster 0 pgmap v319: 428 pgs: 27 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 32 creating+peering, 95 active+undersized, 214 active+clean, 48 active+undersized+degraded; 599 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 350/2718 objects degraded (12.877%) 2024-04-02T04:51:09.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:09 smithi098 bash[23790]: cluster 2024-04-02T04:51:08.698832+0000 mon.a (mon.0) 1643 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:09.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:09 smithi098 bash[23790]: audit 2024-04-02T04:51:08.762451+0000 mon.c (mon.2) 305 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:09.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:09 smithi098 bash[23790]: audit 2024-04-02T04:51:08.850236+0000 mon.c (mon.2) 306 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:09.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:09 smithi098 bash[23790]: audit 2024-04-02T04:51:09.195757+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:09.949 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:09 smithi098 bash[23790]: cluster 2024-04-02T04:51:09.668844+0000 mon.a (mon.0) 1645 : cluster 3 Health check update: Degraded data redundancy: 350/2718 objects degraded (12.877%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:10.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[28904]: cluster 2024-04-02T04:51:07.934821+0000 mgr.y (mgr.14538) 302 : cluster 0 pgmap v319: 428 pgs: 27 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 32 creating+peering, 95 active+undersized, 214 active+clean, 48 active+undersized+degraded; 599 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 350/2718 objects degraded (12.877%) 2024-04-02T04:51:10.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[28904]: cluster 2024-04-02T04:51:08.698832+0000 mon.a (mon.0) 1643 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:10.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[28904]: audit 2024-04-02T04:51:08.762451+0000 mon.c (mon.2) 305 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:10.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[28904]: audit 2024-04-02T04:51:08.850236+0000 mon.c (mon.2) 306 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:10.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[28904]: audit 2024-04-02T04:51:09.195757+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:10.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[28904]: cluster 2024-04-02T04:51:09.668844+0000 mon.a (mon.0) 1645 : cluster 3 Health check update: Degraded data redundancy: 350/2718 objects degraded (12.877%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[21528]: cluster 2024-04-02T04:51:07.934821+0000 mgr.y (mgr.14538) 302 : cluster 0 pgmap v319: 428 pgs: 27 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 32 creating+peering, 95 active+undersized, 214 active+clean, 48 active+undersized+degraded; 599 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 350/2718 objects degraded (12.877%) 2024-04-02T04:51:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[21528]: cluster 2024-04-02T04:51:08.698832+0000 mon.a (mon.0) 1643 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[21528]: audit 2024-04-02T04:51:08.762451+0000 mon.c (mon.2) 305 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[21528]: audit 2024-04-02T04:51:08.850236+0000 mon.c (mon.2) 306 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[21528]: audit 2024-04-02T04:51:09.195757+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:09 smithi069 bash[21528]: cluster 2024-04-02T04:51:09.668844+0000 mon.a (mon.0) 1645 : cluster 3 Health check update: Degraded data redundancy: 350/2718 objects degraded (12.877%), 48 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:10.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.064 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:10.147 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:51:10.147 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:51:10.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:10.187 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:10.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:10.312 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:10.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:10.436 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:10.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.563 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:10.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.692 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:10.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:10.813 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.728352+0000 mon.a (mon.0) 1646 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: cluster 2024-04-02T04:51:09.735823+0000 mon.a (mon.0) 1647 : cluster 0 osdmap e230: 8 total, 7 up, 8 in 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.740832+0000 mon.c (mon.2) 307 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.741948+0000 mon.a (mon.0) 1648 : audit 1 from='client.? 172.21.15.69:0/4128003077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-72153-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.757728+0000 mon.a (mon.0) 1649 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.768795+0000 mon.c (mon.2) 308 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.790398+0000 mon.a (mon.0) 1650 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.790877+0000 mon.a (mon.0) 1651 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.791246+0000 mon.a (mon.0) 1652 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-72145-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:09.882262+0000 mon.b (mon.1) 169 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: cluster 2024-04-02T04:51:09.935747+0000 mgr.y (mgr.14538) 303 : cluster 0 pgmap v322: 396 pgs: 27 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 32 unknown, 82 active+undersized, 195 active+clean, 48 active+undersized+degraded; 599 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s; 350/2718 objects degraded (12.877%) 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:10.562249+0000 mon.b (mon.1) 170 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:10.563744+0000 mon.b (mon.1) 171 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:51:10.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[28904]: audit 2024-04-02T04:51:10.574451+0000 mon.a (mon.0) 1653 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.728352+0000 mon.a (mon.0) 1646 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: cluster 2024-04-02T04:51:09.735823+0000 mon.a (mon.0) 1647 : cluster 0 osdmap e230: 8 total, 7 up, 8 in 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.740832+0000 mon.c (mon.2) 307 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.741948+0000 mon.a (mon.0) 1648 : audit 1 from='client.? 172.21.15.69:0/4128003077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-72153-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.757728+0000 mon.a (mon.0) 1649 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.768795+0000 mon.c (mon.2) 308 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.790398+0000 mon.a (mon.0) 1650 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.790877+0000 mon.a (mon.0) 1651 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.791246+0000 mon.a (mon.0) 1652 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-72145-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:09.882262+0000 mon.b (mon.1) 169 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: cluster 2024-04-02T04:51:09.935747+0000 mgr.y (mgr.14538) 303 : cluster 0 pgmap v322: 396 pgs: 27 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 32 unknown, 82 active+undersized, 195 active+clean, 48 active+undersized+degraded; 599 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s; 350/2718 objects degraded (12.877%) 2024-04-02T04:51:10.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:10.562249+0000 mon.b (mon.1) 170 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:10.563744+0000 mon.b (mon.1) 171 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:51:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:10 smithi069 bash[21528]: audit 2024-04-02T04:51:10.574451+0000 mon.a (mon.0) 1653 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:10.918 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.937 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:10.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:11.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:11.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.728352+0000 mon.a (mon.0) 1646 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: cluster 2024-04-02T04:51:09.735823+0000 mon.a (mon.0) 1647 : cluster 0 osdmap e230: 8 total, 7 up, 8 in 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.740832+0000 mon.c (mon.2) 307 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.741948+0000 mon.a (mon.0) 1648 : audit 1 from='client.? 172.21.15.69:0/4128003077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-72153-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.757728+0000 mon.a (mon.0) 1649 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]: dispatch 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.768795+0000 mon.c (mon.2) 308 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.790398+0000 mon.a (mon.0) 1650 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:11.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.790877+0000 mon.a (mon.0) 1651 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:11.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.791246+0000 mon.a (mon.0) 1652 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-72145-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:11.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:09.882262+0000 mon.b (mon.1) 169 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:11.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: cluster 2024-04-02T04:51:09.935747+0000 mgr.y (mgr.14538) 303 : cluster 0 pgmap v322: 396 pgs: 27 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 32 unknown, 82 active+undersized, 195 active+clean, 48 active+undersized+degraded; 599 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s; 350/2718 objects degraded (12.877%) 2024-04-02T04:51:11.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:10.562249+0000 mon.b (mon.1) 170 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:11.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:10.563744+0000 mon.b (mon.1) 171 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:51:11.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:10 smithi098 bash[23790]: audit 2024-04-02T04:51:10.574451+0000 mon.a (mon.0) 1653 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:11.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:11.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:11.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:11.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:11.437 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:11.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:11.564 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:11.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:11.693 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:11.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:11.821 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:11.926 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:11.948 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:11.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:12.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:12.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:12.205 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: cluster 2024-04-02T04:51:10.729087+0000 mon.a (mon.0) 1654 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: audit 2024-04-02T04:51:10.736843+0000 mon.a (mon.0) 1655 : audit 1 from='client.? 172.21.15.69:0/4128003077' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-72153-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: audit 2024-04-02T04:51:10.736928+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]': finished 2024-04-02T04:51:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: audit 2024-04-02T04:51:10.737035+0000 mon.a (mon.0) 1657 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-72145-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: cluster 2024-04-02T04:51:10.742349+0000 mon.a (mon.0) 1658 : cluster 0 osdmap e231: 8 total, 7 up, 8 in 2024-04-02T04:51:12.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: audit 2024-04-02T04:51:10.743098+0000 mon.a (mon.0) 1659 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-72145-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[28904]: audit 2024-04-02T04:51:10.770106+0000 mon.c (mon.2) 309 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: cluster 2024-04-02T04:51:10.729087+0000 mon.a (mon.0) 1654 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: audit 2024-04-02T04:51:10.736843+0000 mon.a (mon.0) 1655 : audit 1 from='client.? 172.21.15.69:0/4128003077' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-72153-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: audit 2024-04-02T04:51:10.736928+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]': finished 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: audit 2024-04-02T04:51:10.737035+0000 mon.a (mon.0) 1657 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-72145-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: cluster 2024-04-02T04:51:10.742349+0000 mon.a (mon.0) 1658 : cluster 0 osdmap e231: 8 total, 7 up, 8 in 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: audit 2024-04-02T04:51:10.743098+0000 mon.a (mon.0) 1659 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-72145-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:12 smithi069 bash[21528]: audit 2024-04-02T04:51:10.770106+0000 mon.c (mon.2) 309 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:12.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:12.330 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:12.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:12.458 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: cluster 2024-04-02T04:51:10.729087+0000 mon.a (mon.0) 1654 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: audit 2024-04-02T04:51:10.736843+0000 mon.a (mon.0) 1655 : audit 1 from='client.? 172.21.15.69:0/4128003077' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-72153-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: audit 2024-04-02T04:51:10.736928+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-13"}]': finished 2024-04-02T04:51:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: audit 2024-04-02T04:51:10.737035+0000 mon.a (mon.0) 1657 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-72145-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:12.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: cluster 2024-04-02T04:51:10.742349+0000 mon.a (mon.0) 1658 : cluster 0 osdmap e231: 8 total, 7 up, 8 in 2024-04-02T04:51:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: audit 2024-04-02T04:51:10.743098+0000 mon.a (mon.0) 1659 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-72145-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:12.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:12 smithi098 bash[23790]: audit 2024-04-02T04:51:10.770106+0000 mon.c (mon.2) 309 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:12.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:12.587 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:12.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:12.715 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:12.818 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:12.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:12.965 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:12.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:13.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:13.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:13.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:51:13.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:11.771546+0000 mon.c (mon.2) 310 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:13.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: cluster 2024-04-02T04:51:11.936999+0000 mgr.y (mgr.14538) 304 : cluster 0 pgmap v325: 332 pgs: 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 82 active+undersized, 193 active+clean, 37 active+undersized+degraded; 597 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 271/2085 objects degraded (12.998%) 2024-04-02T04:51:13.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: cluster 2024-04-02T04:51:12.174398+0000 mon.a (mon.0) 1660 : cluster 0 osdmap e232: 8 total, 7 up, 8 in 2024-04-02T04:51:13.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:12.773320+0000 mon.c (mon.2) 311 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:13.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:13.085991+0000 mon.a (mon.0) 1661 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-72145-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:13.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: cluster 2024-04-02T04:51:13.091451+0000 mon.a (mon.0) 1662 : cluster 0 osdmap e233: 8 total, 7 up, 8 in 2024-04-02T04:51:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:13.097235+0000 mon.c (mon.2) 312 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:13.097927+0000 mon.a (mon.0) 1663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:13.105550+0000 mon.c (mon.2) 313 : audit 1 from='client.? 172.21.15.69:0/823979957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:13 smithi098 bash[23790]: audit 2024-04-02T04:51:13.107407+0000 mon.a (mon.0) 1664 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.237 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:13.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:13.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:13.385 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:13.385 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:13.385 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:13.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:11.771546+0000 mon.c (mon.2) 310 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: cluster 2024-04-02T04:51:11.936999+0000 mgr.y (mgr.14538) 304 : cluster 0 pgmap v325: 332 pgs: 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 82 active+undersized, 193 active+clean, 37 active+undersized+degraded; 597 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 271/2085 objects degraded (12.998%) 2024-04-02T04:51:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: cluster 2024-04-02T04:51:12.174398+0000 mon.a (mon.0) 1660 : cluster 0 osdmap e232: 8 total, 7 up, 8 in 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:12.773320+0000 mon.c (mon.2) 311 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:13.085991+0000 mon.a (mon.0) 1661 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-72145-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: cluster 2024-04-02T04:51:13.091451+0000 mon.a (mon.0) 1662 : cluster 0 osdmap e233: 8 total, 7 up, 8 in 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:13.097235+0000 mon.c (mon.2) 312 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:13.097927+0000 mon.a (mon.0) 1663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:13.105550+0000 mon.c (mon.2) 313 : audit 1 from='client.? 172.21.15.69:0/823979957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[21528]: audit 2024-04-02T04:51:13.107407+0000 mon.a (mon.0) 1664 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:11.771546+0000 mon.c (mon.2) 310 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: cluster 2024-04-02T04:51:11.936999+0000 mgr.y (mgr.14538) 304 : cluster 0 pgmap v325: 332 pgs: 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 82 active+undersized, 193 active+clean, 37 active+undersized+degraded; 597 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 271/2085 objects degraded (12.998%) 2024-04-02T04:51:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: cluster 2024-04-02T04:51:12.174398+0000 mon.a (mon.0) 1660 : cluster 0 osdmap e232: 8 total, 7 up, 8 in 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:12.773320+0000 mon.c (mon.2) 311 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:13.085991+0000 mon.a (mon.0) 1661 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-72145-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: cluster 2024-04-02T04:51:13.091451+0000 mon.a (mon.0) 1662 : cluster 0 osdmap e233: 8 total, 7 up, 8 in 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:13.097235+0000 mon.c (mon.2) 312 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:13.097927+0000 mon.a (mon.0) 1663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:13.105550+0000 mon.c (mon.2) 313 : audit 1 from='client.? 172.21.15.69:0/823979957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:13 smithi069 bash[28904]: audit 2024-04-02T04:51:13.107407+0000 mon.a (mon.0) 1664 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:13.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:13.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:13.756 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:13.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:13.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:13.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.014 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:14.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.151 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:14.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:14.277 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:14.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:14 smithi098 bash[23790]: audit 2024-04-02T04:51:13.076495+0000 mgr.y (mgr.14538) 305 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:14.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:14 smithi098 bash[23790]: audit 2024-04-02T04:51:13.774390+0000 mon.c (mon.2) 314 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:14.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:14 smithi098 bash[23790]: audit 2024-04-02T04:51:14.089611+0000 mon.a (mon.0) 1665 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:14.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:14 smithi098 bash[23790]: audit 2024-04-02T04:51:14.089837+0000 mon.a (mon.0) 1666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:14.391 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:14 smithi098 bash[23790]: cluster 2024-04-02T04:51:14.098946+0000 mon.a (mon.0) 1667 : cluster 0 osdmap e234: 8 total, 7 up, 8 in 2024-04-02T04:51:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:14.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[21528]: audit 2024-04-02T04:51:13.076495+0000 mgr.y (mgr.14538) 305 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[21528]: audit 2024-04-02T04:51:13.774390+0000 mon.c (mon.2) 314 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[21528]: audit 2024-04-02T04:51:14.089611+0000 mon.a (mon.0) 1665 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[21528]: audit 2024-04-02T04:51:14.089837+0000 mon.a (mon.0) 1666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[21528]: cluster 2024-04-02T04:51:14.098946+0000 mon.a (mon.0) 1667 : cluster 0 osdmap e234: 8 total, 7 up, 8 in 2024-04-02T04:51:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[28904]: audit 2024-04-02T04:51:13.076495+0000 mgr.y (mgr.14538) 305 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:14.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[28904]: audit 2024-04-02T04:51:13.774390+0000 mon.c (mon.2) 314 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:14.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[28904]: audit 2024-04-02T04:51:14.089611+0000 mon.a (mon.0) 1665 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:14.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[28904]: audit 2024-04-02T04:51:14.089837+0000 mon.a (mon.0) 1666 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-72153-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:14.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:14 smithi069 bash[28904]: cluster 2024-04-02T04:51:14.098946+0000 mon.a (mon.0) 1667 : cluster 0 osdmap e234: 8 total, 7 up, 8 in 2024-04-02T04:51:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:14.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:14.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:14.787 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:14.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:14.913 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:14.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:15.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:15.045 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:15.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:15.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: cluster 2024-04-02T04:51:13.938053+0000 mgr.y (mgr.14538) 306 : cluster 0 pgmap v327: 404 pgs: 72 unknown, 13 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 81 active+undersized, 189 active+clean, 38 active+undersized+degraded; 620 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s wr, 1 op/s; 272/1968 objects degraded (13.821%) 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: cluster 2024-04-02T04:51:14.178982+0000 mon.a (mon.0) 1668 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: cluster 2024-04-02T04:51:14.670889+0000 mon.a (mon.0) 1669 : cluster 3 Health check update: Degraded data redundancy: 272/1968 objects degraded (13.821%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: cluster 2024-04-02T04:51:14.683927+0000 mon.a (mon.0) 1670 : cluster 0 osdmap e235: 8 total, 7 up, 8 in 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: audit 2024-04-02T04:51:14.696450+0000 mon.a (mon.0) 1671 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: audit 2024-04-02T04:51:14.719759+0000 mon.c (mon.2) 315 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: audit 2024-04-02T04:51:14.720704+0000 mon.a (mon.0) 1672 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:15.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:15 smithi098 bash[23790]: audit 2024-04-02T04:51:14.775547+0000 mon.c (mon.2) 316 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:15.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:15.435 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:15.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: cluster 2024-04-02T04:51:13.938053+0000 mgr.y (mgr.14538) 306 : cluster 0 pgmap v327: 404 pgs: 72 unknown, 13 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 81 active+undersized, 189 active+clean, 38 active+undersized+degraded; 620 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s wr, 1 op/s; 272/1968 objects degraded (13.821%) 2024-04-02T04:51:15.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: cluster 2024-04-02T04:51:14.178982+0000 mon.a (mon.0) 1668 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:15.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: cluster 2024-04-02T04:51:14.670889+0000 mon.a (mon.0) 1669 : cluster 3 Health check update: Degraded data redundancy: 272/1968 objects degraded (13.821%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:15.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: cluster 2024-04-02T04:51:14.683927+0000 mon.a (mon.0) 1670 : cluster 0 osdmap e235: 8 total, 7 up, 8 in 2024-04-02T04:51:15.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: audit 2024-04-02T04:51:14.696450+0000 mon.a (mon.0) 1671 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: audit 2024-04-02T04:51:14.719759+0000 mon.c (mon.2) 315 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: audit 2024-04-02T04:51:14.720704+0000 mon.a (mon.0) 1672 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[28904]: audit 2024-04-02T04:51:14.775547+0000 mon.c (mon.2) 316 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: cluster 2024-04-02T04:51:13.938053+0000 mgr.y (mgr.14538) 306 : cluster 0 pgmap v327: 404 pgs: 72 unknown, 13 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 81 active+undersized, 189 active+clean, 38 active+undersized+degraded; 620 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s wr, 1 op/s; 272/1968 objects degraded (13.821%) 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: cluster 2024-04-02T04:51:14.178982+0000 mon.a (mon.0) 1668 : cluster 3 Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: cluster 2024-04-02T04:51:14.670889+0000 mon.a (mon.0) 1669 : cluster 3 Health check update: Degraded data redundancy: 272/1968 objects degraded (13.821%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: cluster 2024-04-02T04:51:14.683927+0000 mon.a (mon.0) 1670 : cluster 0 osdmap e235: 8 total, 7 up, 8 in 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: audit 2024-04-02T04:51:14.696450+0000 mon.a (mon.0) 1671 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:15.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: audit 2024-04-02T04:51:14.719759+0000 mon.c (mon.2) 315 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:15.438 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: audit 2024-04-02T04:51:14.720704+0000 mon.a (mon.0) 1672 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:15.438 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:15 smithi069 bash[21528]: audit 2024-04-02T04:51:14.775547+0000 mon.c (mon.2) 316 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:15.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:15.562 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:15.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:15.693 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:15.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:51:15.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:15.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:15.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:16.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:16.189 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:16.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:16.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:16.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:16.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:16.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:16.678 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:16.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.693906+0000 mon.a (mon.0) 1673 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.694012+0000 mon.a (mon.0) 1674 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: cluster 2024-04-02T04:51:15.697643+0000 mon.a (mon.0) 1675 : cluster 0 osdmap e236: 8 total, 7 up, 8 in 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.699471+0000 mon.a (mon.0) 1676 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.703744+0000 mon.c (mon.2) 317 : audit 1 from='client.? 172.21.15.69:0/1068279702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.706243+0000 mon.c (mon.2) 318 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.712066+0000 mon.a (mon.0) 1677 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.712339+0000 mon.a (mon.0) 1678 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:16.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:16 smithi098 bash[23790]: audit 2024-04-02T04:51:15.776501+0000 mon.c (mon.2) 319 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:16.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:16.930 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:16.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:17.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:17.056 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:17.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:17.188 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.693906+0000 mon.a (mon.0) 1673 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.694012+0000 mon.a (mon.0) 1674 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: cluster 2024-04-02T04:51:15.697643+0000 mon.a (mon.0) 1675 : cluster 0 osdmap e236: 8 total, 7 up, 8 in 2024-04-02T04:51:17.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.699471+0000 mon.a (mon.0) 1676 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.703744+0000 mon.c (mon.2) 317 : audit 1 from='client.? 172.21.15.69:0/1068279702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.706243+0000 mon.c (mon.2) 318 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.712066+0000 mon.a (mon.0) 1677 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.712339+0000 mon.a (mon.0) 1678 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[21528]: audit 2024-04-02T04:51:15.776501+0000 mon.c (mon.2) 319 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.693906+0000 mon.a (mon.0) 1673 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.694012+0000 mon.a (mon.0) 1674 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: cluster 2024-04-02T04:51:15.697643+0000 mon.a (mon.0) 1675 : cluster 0 osdmap e236: 8 total, 7 up, 8 in 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.699471+0000 mon.a (mon.0) 1676 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.703744+0000 mon.c (mon.2) 317 : audit 1 from='client.? 172.21.15.69:0/1068279702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.706243+0000 mon.c (mon.2) 318 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.712066+0000 mon.a (mon.0) 1677 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:17.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.712339+0000 mon.a (mon.0) 1678 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:17.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:16 smithi069 bash[28904]: audit 2024-04-02T04:51:15.776501+0000 mon.c (mon.2) 319 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:17.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:17.316 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:17.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:17.439 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:17.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:17.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:17.697 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:17.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:17.836 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:17.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:17.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: cluster 2024-04-02T04:51:15.939903+0000 mgr.y (mgr.14538) 307 : cluster 0 pgmap v331: 396 pgs: 64 unknown, 13 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 81 active+undersized, 189 active+clean, 38 active+undersized+degraded; 620 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 2 op/s; 272/1968 objects degraded (13.821%) 2024-04-02T04:51:17.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: cluster 2024-04-02T04:51:16.385628+0000 osd.5 (osd.5) 157 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: cluster 2024-04-02T04:51:16.385778+0000 osd.5 (osd.5) 158 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.697343+0000 mon.a (mon.0) 1679 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.697527+0000 mon.a (mon.0) 1680 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.697719+0000 mon.a (mon.0) 1681 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]': finished 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.709810+0000 mon.c (mon.2) 320 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: cluster 2024-04-02T04:51:16.710247+0000 mon.a (mon.0) 1682 : cluster 0 osdmap e237: 8 total, 7 up, 8 in 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.712600+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.822194+0000 mon.c (mon.2) 321 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.843640+0000 mon.b (mon.1) 172 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.846050+0000 mon.b (mon.1) 173 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.848380+0000 mon.b (mon.1) 174 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.849051+0000 mon.a (mon.0) 1684 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.850624+0000 mon.a (mon.0) 1685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: cluster 2024-04-02T04:51:15.939903+0000 mgr.y (mgr.14538) 307 : cluster 0 pgmap v331: 396 pgs: 64 unknown, 13 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 81 active+undersized, 189 active+clean, 38 active+undersized+degraded; 620 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 2 op/s; 272/1968 objects degraded (13.821%) 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: cluster 2024-04-02T04:51:16.385628+0000 osd.5 (osd.5) 157 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: cluster 2024-04-02T04:51:16.385778+0000 osd.5 (osd.5) 158 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.697343+0000 mon.a (mon.0) 1679 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.697527+0000 mon.a (mon.0) 1680 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.697719+0000 mon.a (mon.0) 1681 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]': finished 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.709810+0000 mon.c (mon.2) 320 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]: dispatch 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: cluster 2024-04-02T04:51:16.710247+0000 mon.a (mon.0) 1682 : cluster 0 osdmap e237: 8 total, 7 up, 8 in 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.712600+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]: dispatch 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.822194+0000 mon.c (mon.2) 321 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:17.954 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.843640+0000 mon.b (mon.1) 172 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.846050+0000 mon.b (mon.1) 173 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.848380+0000 mon.b (mon.1) 174 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.849051+0000 mon.a (mon.0) 1684 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.850624+0000 mon.a (mon.0) 1685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:16.852989+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:17.060602+0000 mon.b (mon.1) 175 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:17.062076+0000 mon.a (mon.0) 1687 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: cluster 2024-04-02T04:51:17.697630+0000 mon.a (mon.0) 1688 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:17.699640+0000 mon.a (mon.0) 1689 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]': finished 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:17.699756+0000 mon.a (mon.0) 1690 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:17.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: cluster 2024-04-02T04:51:17.705508+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e238: 8 total, 7 up, 8 in 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:17.708549+0000 mon.b (mon.1) 176 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[21528]: audit 2024-04-02T04:51:17.713284+0000 mon.a (mon.0) 1692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:16.852989+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:17.060602+0000 mon.b (mon.1) 175 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:17.062076+0000 mon.a (mon.0) 1687 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: cluster 2024-04-02T04:51:17.697630+0000 mon.a (mon.0) 1688 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:17.699640+0000 mon.a (mon.0) 1689 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]': finished 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:17.699756+0000 mon.a (mon.0) 1690 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: cluster 2024-04-02T04:51:17.705508+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e238: 8 total, 7 up, 8 in 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:17.708549+0000 mon.b (mon.1) 176 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:17 smithi069 bash[28904]: audit 2024-04-02T04:51:17.713284+0000 mon.a (mon.0) 1692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.963 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:17.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: cluster 2024-04-02T04:51:15.939903+0000 mgr.y (mgr.14538) 307 : cluster 0 pgmap v331: 396 pgs: 64 unknown, 13 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 81 active+undersized, 189 active+clean, 38 active+undersized+degraded; 620 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 2 op/s; 272/1968 objects degraded (13.821%) 2024-04-02T04:51:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: cluster 2024-04-02T04:51:16.385628+0000 osd.5 (osd.5) 157 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: cluster 2024-04-02T04:51:16.385778+0000 osd.5 (osd.5) 158 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.697343+0000 mon.a (mon.0) 1679 : audit 1 from='client.? 172.21.15.69:0/2367793125' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-72145-27"}]': finished 2024-04-02T04:51:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.697527+0000 mon.a (mon.0) 1680 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-72153-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.697719+0000 mon.a (mon.0) 1681 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-15"}]': finished 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.709810+0000 mon.c (mon.2) 320 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: cluster 2024-04-02T04:51:16.710247+0000 mon.a (mon.0) 1682 : cluster 0 osdmap e237: 8 total, 7 up, 8 in 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.712600+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.822194+0000 mon.c (mon.2) 321 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.843640+0000 mon.b (mon.1) 172 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.846050+0000 mon.b (mon.1) 173 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.848380+0000 mon.b (mon.1) 174 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.849051+0000 mon.a (mon.0) 1684 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.850624+0000 mon.a (mon.0) 1685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:16.852989+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:17.060602+0000 mon.b (mon.1) 175 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:17.062076+0000 mon.a (mon.0) 1687 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: cluster 2024-04-02T04:51:17.697630+0000 mon.a (mon.0) 1688 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:17.699640+0000 mon.a (mon.0) 1689 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-15", "mode": "writeback"}]': finished 2024-04-02T04:51:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:17.699756+0000 mon.a (mon.0) 1690 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-72145-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: cluster 2024-04-02T04:51:17.705508+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e238: 8 total, 7 up, 8 in 2024-04-02T04:51:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:17.708549+0000 mon.b (mon.1) 176 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:17 smithi098 bash[23790]: audit 2024-04-02T04:51:17.713284+0000 mon.a (mon.0) 1692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:18.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:18.091 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:18.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:18.216 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:18.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:18.349 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:18.451 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:18.477 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:18.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:18.603 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:18.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:18.740 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:18.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:18.869 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:18.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:18.981 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:51:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:51:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:51:18.997 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:18.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:19.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:19.121 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:19.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:19.233 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[28904]: audit 2024-04-02T04:51:17.826406+0000 mon.c (mon.2) 322 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:19.233 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[28904]: cluster 2024-04-02T04:51:17.943278+0000 mgr.y (mgr.14538) 308 : cluster 0 pgmap v334: 364 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 93 active+undersized, 212 active+clean, 38 active+undersized+degraded; 628 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 KiB/s wr, 4 op/s; 273/1860 objects degraded (14.677%) 2024-04-02T04:51:19.233 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[28904]: cluster 2024-04-02T04:51:18.051258+0000 osd.1 (osd.1) 181 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:19.234 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[28904]: cluster 2024-04-02T04:51:18.051396+0000 osd.1 (osd.1) 182 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:19.234 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[21528]: audit 2024-04-02T04:51:17.826406+0000 mon.c (mon.2) 322 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:19.234 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[21528]: cluster 2024-04-02T04:51:17.943278+0000 mgr.y (mgr.14538) 308 : cluster 0 pgmap v334: 364 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 93 active+undersized, 212 active+clean, 38 active+undersized+degraded; 628 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 KiB/s wr, 4 op/s; 273/1860 objects degraded (14.677%) 2024-04-02T04:51:19.234 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[21528]: cluster 2024-04-02T04:51:18.051258+0000 osd.1 (osd.1) 181 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:19.234 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:19 smithi069 bash[21528]: cluster 2024-04-02T04:51:18.051396+0000 osd.1 (osd.1) 182 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:19.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:19.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:19.355 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:51:19.355 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:51:19.377 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:19 smithi098 bash[23790]: audit 2024-04-02T04:51:17.826406+0000 mon.c (mon.2) 322 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:19 smithi098 bash[23790]: cluster 2024-04-02T04:51:17.943278+0000 mgr.y (mgr.14538) 308 : cluster 0 pgmap v334: 364 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 93 active+undersized, 212 active+clean, 38 active+undersized+degraded; 628 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 KiB/s wr, 4 op/s; 273/1860 objects degraded (14.677%) 2024-04-02T04:51:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:19 smithi098 bash[23790]: cluster 2024-04-02T04:51:18.051258+0000 osd.1 (osd.1) 181 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:19 smithi098 bash[23790]: cluster 2024-04-02T04:51:18.051396+0000 osd.1 (osd.1) 182 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:19.480 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:19.507 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:19.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:19.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:19.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:19.904 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:19.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:20.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:20.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:20.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-02T04:51:20.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:20.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: cluster 2024-04-02T04:51:18.611525+0000 osd.4 (osd.4) 169 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: cluster 2024-04-02T04:51:18.611722+0000 osd.4 (osd.4) 170 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:18.781847+0000 mon.c (mon.2) 323 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:18.828704+0000 mon.c (mon.2) 324 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: cluster 2024-04-02T04:51:19.179117+0000 mon.a (mon.0) 1693 : cluster 0 osdmap e239: 8 total, 7 up, 8 in 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: cluster 2024-04-02T04:51:19.179382+0000 mon.a (mon.0) 1694 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.180407+0000 mon.c (mon.2) 325 : audit 1 from='client.? 172.21.15.69:0/4043317560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.191745+0000 mon.a (mon.0) 1695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.192053+0000 mon.a (mon.0) 1696 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: cluster 2024-04-02T04:51:19.674830+0000 mon.a (mon.0) 1697 : cluster 3 Health check update: Degraded data redundancy: 273/1860 objects degraded (14.677%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.729818+0000 mon.a (mon.0) 1698 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.730118+0000 mon.a (mon.0) 1699 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.730364+0000 mon.a (mon.0) 1700 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.735222+0000 mon.c (mon.2) 326 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:20.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: cluster 2024-04-02T04:51:19.740189+0000 mon.a (mon.0) 1701 : cluster 0 osdmap e240: 8 total, 7 up, 8 in 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.743134+0000 mon.c (mon.2) 327 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: cluster 2024-04-02T04:51:18.611525+0000 osd.4 (osd.4) 169 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: cluster 2024-04-02T04:51:18.611722+0000 osd.4 (osd.4) 170 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:18.781847+0000 mon.c (mon.2) 323 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:18.828704+0000 mon.c (mon.2) 324 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: cluster 2024-04-02T04:51:19.179117+0000 mon.a (mon.0) 1693 : cluster 0 osdmap e239: 8 total, 7 up, 8 in 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: cluster 2024-04-02T04:51:19.179382+0000 mon.a (mon.0) 1694 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.180407+0000 mon.c (mon.2) 325 : audit 1 from='client.? 172.21.15.69:0/4043317560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.191745+0000 mon.a (mon.0) 1695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.192053+0000 mon.a (mon.0) 1696 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:20.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: cluster 2024-04-02T04:51:19.674830+0000 mon.a (mon.0) 1697 : cluster 3 Health check update: Degraded data redundancy: 273/1860 objects degraded (14.677%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.729818+0000 mon.a (mon.0) 1698 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.730118+0000 mon.a (mon.0) 1699 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.730364+0000 mon.a (mon.0) 1700 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.735222+0000 mon.c (mon.2) 326 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: cluster 2024-04-02T04:51:19.740189+0000 mon.a (mon.0) 1701 : cluster 0 osdmap e240: 8 total, 7 up, 8 in 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.743134+0000 mon.c (mon.2) 327 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.746744+0000 mon.a (mon.0) 1702 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.751058+0000 mon.a (mon.0) 1703 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[21528]: audit 2024-04-02T04:51:19.829734+0000 mon.c (mon.2) 328 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.746744+0000 mon.a (mon.0) 1702 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.751058+0000 mon.a (mon.0) 1703 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:20.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:20 smithi069 bash[28904]: audit 2024-04-02T04:51:19.829734+0000 mon.c (mon.2) 328 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:20.420 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: cluster 2024-04-02T04:51:18.611525+0000 osd.4 (osd.4) 169 : cluster 0 purged_snaps scrub starts 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: cluster 2024-04-02T04:51:18.611722+0000 osd.4 (osd.4) 170 : cluster 0 purged_snaps scrub ok 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:18.781847+0000 mon.c (mon.2) 323 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:18.828704+0000 mon.c (mon.2) 324 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: cluster 2024-04-02T04:51:19.179117+0000 mon.a (mon.0) 1693 : cluster 0 osdmap e239: 8 total, 7 up, 8 in 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: cluster 2024-04-02T04:51:19.179382+0000 mon.a (mon.0) 1694 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.180407+0000 mon.c (mon.2) 325 : audit 1 from='client.? 172.21.15.69:0/4043317560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.191745+0000 mon.a (mon.0) 1695 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.192053+0000 mon.a (mon.0) 1696 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: cluster 2024-04-02T04:51:19.674830+0000 mon.a (mon.0) 1697 : cluster 3 Health check update: Degraded data redundancy: 273/1860 objects degraded (14.677%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.729818+0000 mon.a (mon.0) 1698 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-72145-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.730118+0000 mon.a (mon.0) 1699 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.730364+0000 mon.a (mon.0) 1700 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-72153-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.735222+0000 mon.c (mon.2) 326 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: cluster 2024-04-02T04:51:19.740189+0000 mon.a (mon.0) 1701 : cluster 0 osdmap e240: 8 total, 7 up, 8 in 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.743134+0000 mon.c (mon.2) 327 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.746744+0000 mon.a (mon.0) 1702 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]: dispatch 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.751058+0000 mon.a (mon.0) 1703 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:20.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:20 smithi098 bash[23790]: audit 2024-04-02T04:51:19.829734+0000 mon.c (mon.2) 328 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:20.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:20.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.677 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:20.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:20.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.937 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:20.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:21.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:21.066 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:21.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:21.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: cluster 2024-04-02T04:51:19.945108+0000 mgr.y (mgr.14538) 309 : cluster 0 pgmap v337: 396 pgs: 40 unknown, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 92 active+undersized, 209 active+clean, 37 active+undersized+degraded; 588 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 271/1842 objects degraded (14.712%) 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: cluster 2024-04-02T04:51:20.730465+0000 mon.a (mon.0) 1704 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: audit 2024-04-02T04:51:20.732448+0000 mon.a (mon.0) 1705 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]': finished 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: audit 2024-04-02T04:51:20.732557+0000 mon.a (mon.0) 1706 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: cluster 2024-04-02T04:51:20.736096+0000 mon.a (mon.0) 1707 : cluster 0 osdmap e241: 8 total, 7 up, 8 in 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: audit 2024-04-02T04:51:20.745331+0000 mon.c (mon.2) 329 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:21.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:21 smithi098 bash[23790]: audit 2024-04-02T04:51:20.764219+0000 mon.a (mon.0) 1708 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:21.326 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:21.429 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:21.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: cluster 2024-04-02T04:51:19.945108+0000 mgr.y (mgr.14538) 309 : cluster 0 pgmap v337: 396 pgs: 40 unknown, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 92 active+undersized, 209 active+clean, 37 active+undersized+degraded; 588 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 271/1842 objects degraded (14.712%) 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: cluster 2024-04-02T04:51:20.730465+0000 mon.a (mon.0) 1704 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: audit 2024-04-02T04:51:20.732448+0000 mon.a (mon.0) 1705 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]': finished 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: audit 2024-04-02T04:51:20.732557+0000 mon.a (mon.0) 1706 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: cluster 2024-04-02T04:51:20.736096+0000 mon.a (mon.0) 1707 : cluster 0 osdmap e241: 8 total, 7 up, 8 in 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: audit 2024-04-02T04:51:20.745331+0000 mon.c (mon.2) 329 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[28904]: audit 2024-04-02T04:51:20.764219+0000 mon.a (mon.0) 1708 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: cluster 2024-04-02T04:51:19.945108+0000 mgr.y (mgr.14538) 309 : cluster 0 pgmap v337: 396 pgs: 40 unknown, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 92 active+undersized, 209 active+clean, 37 active+undersized+degraded; 588 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 271/1842 objects degraded (14.712%) 2024-04-02T04:51:21.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: cluster 2024-04-02T04:51:20.730465+0000 mon.a (mon.0) 1704 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:21.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: audit 2024-04-02T04:51:20.732448+0000 mon.a (mon.0) 1705 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-15"}]': finished 2024-04-02T04:51:21.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: audit 2024-04-02T04:51:20.732557+0000 mon.a (mon.0) 1706 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:21.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: cluster 2024-04-02T04:51:20.736096+0000 mon.a (mon.0) 1707 : cluster 0 osdmap e241: 8 total, 7 up, 8 in 2024-04-02T04:51:21.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: audit 2024-04-02T04:51:20.745331+0000 mon.c (mon.2) 329 : audit 1 from='client.? 172.21.15.69:0/3443600823' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:21.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:21 smithi069 bash[21528]: audit 2024-04-02T04:51:20.764219+0000 mon.a (mon.0) 1708 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]: dispatch 2024-04-02T04:51:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:21.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:21.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:21.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:21.717 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:21.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrip: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-02T04:51:21.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2009 ms) 2024-04-02T04:51:21.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8985 ms total) 2024-04-02T04:51:21.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4103 ms) 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3893 ms) 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4034 ms) 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12030 ms total) 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (131702 ms total) 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-02T04:51:21.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-02T04:51:21.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:21.841 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:21.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:21.964 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:21.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:22.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:22.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:22.226 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:22.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:22.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:22.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:20.830640+0000 mon.c (mon.2) 330 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:22.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:21.749575+0000 mon.a (mon.0) 1709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:22.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: cluster 2024-04-02T04:51:21.753686+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e242: 8 total, 7 up, 8 in 2024-04-02T04:51:22.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:21.759493+0000 mon.b (mon.1) 177 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:22.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:21.764678+0000 mon.a (mon.0) 1711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:22.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:21.766256+0000 mon.c (mon.2) 331 : audit 1 from='client.? 172.21.15.69:0/3999996153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:22.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:21.767144+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:22.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:22 smithi098 bash[23790]: audit 2024-04-02T04:51:21.831845+0000 mon.c (mon.2) 332 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:22.478 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:20.830640+0000 mon.c (mon.2) 330 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:21.749575+0000 mon.a (mon.0) 1709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: cluster 2024-04-02T04:51:21.753686+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e242: 8 total, 7 up, 8 in 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:21.759493+0000 mon.b (mon.1) 177 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:21.764678+0000 mon.a (mon.0) 1711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:21.766256+0000 mon.c (mon.2) 331 : audit 1 from='client.? 172.21.15.69:0/3999996153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:21.767144+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[21528]: audit 2024-04-02T04:51:21.831845+0000 mon.c (mon.2) 332 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:22.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:20.830640+0000 mon.c (mon.2) 330 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:21.749575+0000 mon.a (mon.0) 1709 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-72389-21"}]': finished 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: cluster 2024-04-02T04:51:21.753686+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e242: 8 total, 7 up, 8 in 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:21.759493+0000 mon.b (mon.1) 177 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:21.764678+0000 mon.a (mon.0) 1711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:21.766256+0000 mon.c (mon.2) 331 : audit 1 from='client.? 172.21.15.69:0/3999996153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:21.767144+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:22.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:22 smithi069 bash[28904]: audit 2024-04-02T04:51:21.831845+0000 mon.c (mon.2) 332 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.559+0000 7f05919f3700 1 -- 172.21.15.69:0/1589155165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c1011a0 msgr2=0x7f058c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.559+0000 7f05919f3700 1 --2- 172.21.15.69:0/1589155165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c1011a0 0x7f058c101620 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0580009990 tx=0x7f0580009ca0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/1589155165 shutdown_connections 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 --2- 172.21.15.69:0/1589155165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058c101b60 0x7f058c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 --2- 172.21.15.69:0/1589155165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c107220 0x7f058c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 --2- 172.21.15.69:0/1589155165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c1011a0 0x7f058c101620 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/1589155165 >> 172.21.15.69:0/1589155165 conn(0x7f058c0771e0 msgr2=0x7f058c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/1589155165 shutdown_connections 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/1589155165 wait complete. 2024-04-02T04:51:22.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 librados: starting msgr at 2024-04-02T04:51:22.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 librados: starting objecter 2024-04-02T04:51:22.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 Processor -- start 2024-04-02T04:51:22.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- start start 2024-04-02T04:51:22.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 librados: setting wanted keys 2024-04-02T04:51:22.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 librados: calling monclient init 2024-04-02T04:51:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 0x7f058c102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c101b60 0x7f058c103360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058c107220 0x7f058c1038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f058c110930 con 0x7f058c101b60 2024-04-02T04:51:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f058c1107b0 con 0x7f058c107220 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f058c110ab0 con 0x7f058c1011a0 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 0x7f058c102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 0x7f058c102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34906/0 (socket says 172.21.15.69:34906) 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058affd700 1 -- 172.21.15.69:0/3838671543 learned_addr learned my addr 172.21.15.69:0/3838671543 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c101b60 0x7f058c103360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058b7fe700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058c107220 0x7f058c1038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 -- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 msgr2=0x7f058c102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 0x7f058c102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 -- 172.21.15.69:0/3838671543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058c107220 msgr2=0x7f058c1038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058c107220 0x7f058c1038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 -- 172.21.15.69:0/3838671543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c1a6510 con 0x7f058c101b60 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058affd700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 0x7f058c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:51:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f058a7fc700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c101b60 0x7f058c103360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f05800059d0 tx=0x7f05800047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05909f1700 10 client.?.objecter ms_handle_connect 0x7f058c101b60 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05909f1700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580027070 con 0x7f058c101b60 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0580022470 con 0x7f058c101b60 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580017650 con 0x7f058c101b60 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c1a67a0 con 0x7f058c101b60 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 10 client.15984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c1a6c50 con 0x7f058c101b60 2024-04-02T04:51:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 librados: init done 2024-04-02T04:51:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.563+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058c04ff00 con 0x7f058c101b60 2024-04-02T04:51:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f058002a070 con 0x7f058c101b60 2024-04-02T04:51:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f057406f5e0 0x7f0574071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(242..242 src has 74..242) ==== 11251+0+0 (secure 0 0 0) 0x7f058008ff90 con 0x7f058c101b60 2024-04-02T04:51:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 10 client.15984.objecter ms_dispatch 0x7f058c001550 osd_map(242..242 src has 74..242) 2024-04-02T04:51:22.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 3 client.15984.objecter handle_osd_map got epochs [242,242] > 0 2024-04-02T04:51:22.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 3 client.15984.objecter handle_osd_map decoding full epoch 242 2024-04-02T04:51:22.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f058affd700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f057406f5e0 0x7f0574071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 20 client.15984.objecter dump_active .. 0 homeless 2024-04-02T04:51:22.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f05909f1700 10 client.15984.objecter ms_handle_connect 0x7f057406f5e0 2024-04-02T04:51:22.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.567+0000 7f058affd700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f057406f5e0 0x7f0574071aa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f057c009680 tx=0x7f057c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.571+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0580013070 con 0x7f058c101b60 2024-04-02T04:51:22.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:22.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 -- 172.21.15.69:0/85901805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf41011a0 msgr2=0x7fdaf4101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 --2- 172.21.15.69:0/85901805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf41011a0 0x7fdaf4101620 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009990 tx=0x7fdae8009ca0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 -- 172.21.15.69:0/85901805 shutdown_connections 2024-04-02T04:51:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 --2- 172.21.15.69:0/85901805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf4101b60 0x7fdaf410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 --2- 172.21.15.69:0/85901805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf4107220 0x7fdaf4107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 --2- 172.21.15.69:0/85901805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf41011a0 0x7fdaf4101620 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 -- 172.21.15.69:0/85901805 >> 172.21.15.69:0/85901805 conn(0x7fdaf40771e0 msgr2=0x7fdaf40775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 -- 172.21.15.69:0/85901805 shutdown_connections 2024-04-02T04:51:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.611+0000 7fdafa014700 1 -- 172.21.15.69:0/85901805 wait complete. 2024-04-02T04:51:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 librados: starting msgr at 2024-04-02T04:51:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 librados: starting objecter 2024-04-02T04:51:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 Processor -- start 2024-04-02T04:51:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- start start 2024-04-02T04:51:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 librados: setting wanted keys 2024-04-02T04:51:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 librados: calling monclient init 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf41011a0 0x7fdaf4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf4101b60 0x7fdaf4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf4107220 0x7fdaf419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf4111b50 con 0x7fdaf4107220 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41119d0 con 0x7fdaf4101b60 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdaf4111cd0 con 0x7fdaf41011a0 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf41011a0 0x7fdaf4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf41011a0 0x7fdaf4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34930/0 (socket says 172.21.15.69:34930) 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 -- 172.21.15.69:0/1144499848 learned_addr learned my addr 172.21.15.69:0/1144499848 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf4107220 0x7fdaf419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 -- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf4101b60 msgr2=0x7fdaf4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf4101b60 0x7fdaf4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf4101b60 0x7fdaf4198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 -- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf4107220 msgr2=0x7fdaf419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf4107220 0x7fdaf419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf419d010 con 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf4107220 0x7fdaf419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf41011a0 0x7fdaf4198000 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdae400c920 tx=0x7fdae400cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf0ff9700 10 client.?.objecter ms_handle_connect 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf0ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4007b00 con 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fdae4004cb0 con 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4005550 con 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf419d300 con 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 10 client.26063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf410e810 con 0x7fdaf41011a0 2024-04-02T04:51:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 librados: init done 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdaf404ff00 con 0x7fdaf41011a0 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.615+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fdae4007ca0 con 0x7fdaf41011a0 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdadc06f5e0 0x7fdadc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(242..242 src has 74..242) ==== 11251+0+0 (secure 0 0 0) 0x7fdae408ee10 con 0x7fdaf41011a0 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 osd_map(242..242 src has 74..242) 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 3 client.26063.objecter handle_osd_map got epochs [242,242] > 0 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 3 client.26063.objecter handle_osd_map decoding full epoch 242 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdadc06f5e0 0x7fdadc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 20 client.26063.objecter dump_active .. 0 homeless 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v242) ==== 54+0+16 (secure 0 0 0) 0x7fdae4014030 con 0x7fdaf41011a0 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 10 client.26063.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 1 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fdaf40630a0 0x7fdaf4107f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 1 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 0 1 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4199360 osd=0 4 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 10 client.26063.objecter _send_command 1 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdaf41095d0 con 0x7fdaf40630a0 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fdaf40630a0 0x7fdaf4107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdadc06f5e0 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdadc06f5e0 0x7fdadc071aa0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdae800b7f0 tx=0x7fdae8005ea0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf40630a0 2024-04-02T04:51:22.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fdaf40630a0 0x7fdaf4107f10 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf41095d0 con 0x7fdaf40630a0 2024-04-02T04:51:22.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 1: 0 ) 2024-04-02T04:51:22.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 1 = system:0 2024-04-02T04:51:22.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf4199360 osd=0 4 2024-04-02T04:51:22.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.619+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 0 1 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 10 client.26063.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 2 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 2 2024-04-02T04:51:22.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf4199360 osd=0 3 2024-04-02T04:51:22.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 0 2 2024-04-02T04:51:22.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4199360 osd=0 4 2024-04-02T04:51:22.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 10 client.26063.objecter _send_command 2 2024-04-02T04:51:22.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf4108450 con 0x7fdaf40630a0 2024-04-02T04:51:22.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf4108450 con 0x7fdaf40630a0 2024-04-02T04:51:22.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf4199360 osd=0 4 2024-04-02T04:51:22.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.639+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 0 2 2024-04-02T04:51:22.643 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 10 client.26063.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 3 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fdaf4062c90 0x7fdaf410ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 3 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 1 3 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4062b20 osd=1 4 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 10 client.26063.objecter _send_command 3 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdaf41107b0 con 0x7fdaf4062c90 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fdaf4062c90 0x7fdaf410ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf4062c90 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fdaf4062c90 0x7fdaf410ead0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf41107b0 con 0x7fdaf4062c90 2024-04-02T04:51:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 3: 0 ) 2024-04-02T04:51:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 3 = system:0 2024-04-02T04:51:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf4062b20 osd=1 4 2024-04-02T04:51:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.643+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 1 3 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 10 client.26063.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 4 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 4 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf4062b20 osd=1 3 2024-04-02T04:51:22.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 1 4 2024-04-02T04:51:22.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf4062b20 osd=1 4 2024-04-02T04:51:22.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 10 client.26063.objecter _send_command 4 2024-04-02T04:51:22.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf410f010 con 0x7fdaf4062c90 2024-04-02T04:51:22.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf410f010 con 0x7fdaf4062c90 2024-04-02T04:51:22.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf4062b20 osd=1 4 2024-04-02T04:51:22.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 1 4 2024-04-02T04:51:22.666 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 10 client.26063.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 5 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fdaf402d180 0x7fdaf402d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 5 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 2 5 2024-04-02T04:51:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41103a0 osd=2 4 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 10 client.26063.objecter _send_command 5 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdaf4110510 con 0x7fdaf402d180 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fdaf402d180 0x7fdaf402d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf402d180 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.663+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fdaf402d180 0x7fdaf402d5e0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.667+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf4110510 con 0x7fdaf402d180 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.667+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 5: 0 ) 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.667+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 5 = system:0 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.667+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf41103a0 osd=2 4 2024-04-02T04:51:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.667+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 2 5 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 10 client.26063.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 6 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 6 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf41103a0 osd=2 3 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 2 6 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41103a0 osd=2 4 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 10 client.26063.objecter _send_command 6 2024-04-02T04:51:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf402cc00 con 0x7fdaf402d180 2024-04-02T04:51:22.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf402cc00 con 0x7fdaf402d180 2024-04-02T04:51:22.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf41103a0 osd=2 4 2024-04-02T04:51:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 2 6 2024-04-02T04:51:22.687 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:51:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 10 client.26063.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 7 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fdaf410fea0 0x7fdaf41b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 7 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 3 7 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf402d000 osd=3 4 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 10 client.26063.objecter _send_command 7 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdaf41b4fb0 con 0x7fdaf410fea0 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.683+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fdaf410fea0 0x7fdaf41b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fdaf410fea0 0x7fdaf41b2150 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf410fea0 2024-04-02T04:51:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf41b4fb0 con 0x7fdaf410fea0 2024-04-02T04:51:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 7: 0 ) 2024-04-02T04:51:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 7 = system:0 2024-04-02T04:51:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf402d000 osd=3 4 2024-04-02T04:51:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.687+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 3 7 2024-04-02T04:51:22.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 10 client.26063.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 8 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 8 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf402d000 osd=3 3 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 3 8 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf402d000 osd=3 4 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 10 client.26063.objecter _send_command 8 2024-04-02T04:51:22.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf410f530 con 0x7fdaf410fea0 2024-04-02T04:51:22.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf410f530 con 0x7fdaf410fea0 2024-04-02T04:51:22.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf402d000 osd=3 4 2024-04-02T04:51:22.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.723+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 3 8 2024-04-02T04:51:22.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:51:22.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 10 client.26063.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 9 2024-04-02T04:51:22.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fdaf41afca0 0x7fdaf41b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 9 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 4 9 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f530 osd=4 4 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 10 client.26063.objecter _send_command 9 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdaf41b8940 con 0x7fdaf41afca0 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fdaf41afca0 0x7fdaf41b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf37fe700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fdaf41afca0 0x7fdaf41b3810 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf41afca0 2024-04-02T04:51:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:22.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf41b8940 con 0x7fdaf41afca0 2024-04-02T04:51:22.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 9: 0 ) 2024-04-02T04:51:22.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 9 = system:0 2024-04-02T04:51:22.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf410f530 osd=4 4 2024-04-02T04:51:22.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.727+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 4 9 2024-04-02T04:51:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 10 client.26063.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 10 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 10 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf410f530 osd=4 3 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 4 10 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f530 osd=4 4 2024-04-02T04:51:22.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 10 client.26063.objecter _send_command 10 2024-04-02T04:51:22.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.763+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf41b3d50 con 0x7fdaf41afca0 2024-04-02T04:51:22.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf41b3d50 con 0x7fdaf41afca0 2024-04-02T04:51:22.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf410f530 osd=4 4 2024-04-02T04:51:22.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 4 10 2024-04-02T04:51:22.769 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 10 client.26063.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 11 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fdaf41b2e60 0x7fdaf41b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 11 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 5 11 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f2e0 osd=5 4 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 10 client.26063.objecter _send_command 11 2024-04-02T04:51:22.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.767+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdaf41bbad0 con 0x7fdaf41b2e60 2024-04-02T04:51:22.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.771+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fdaf41b2e60 0x7fdaf41b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3251261378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 msgr2=0x7f0e741015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3251261378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e741015f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009990 tx=0x7f0e64009ca0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3251261378 shutdown_connections 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3251261378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e74101b30 0x7f0e7410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3251261378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3251261378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e741015f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3251261378 >> 172.21.15.69:0/3251261378 conn(0x7f0e74078110 msgr2=0x7f0e74078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:22.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf2ffd700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fdaf41b2e60 0x7fdaf41b32c0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf41b2e60 2024-04-02T04:51:22.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3251261378 shutdown_connections 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3251261378 wait complete. 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 librados: starting msgr at 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 librados: starting objecter 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 Processor -- start 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 -- start start 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 librados: setting wanted keys 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7f0e7b4fa700 1 librados: calling monclient init 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf41bbad0 con 0x7fdaf41b2e60 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 11: 0 ) 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 11 = system:0 2024-04-02T04:51:22.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf410f2e0 osd=5 4 2024-04-02T04:51:22.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.779+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 5 11 2024-04-02T04:51:22.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e7419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e74101b30 0x7f0e7419f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741a3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741a3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741a3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34934/0 (socket says 172.21.15.69:34934) 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79296700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e7419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79296700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e7419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.69:37326/0 (socket says 172.21.15.69:37326) 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 -- 172.21.15.69:0/3238200247 learned_addr learned my addr 172.21.15.69:0/3238200247 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e74110a50 con 0x7f0e74101b30 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e741108d0 con 0x7f0e74101170 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0e74110bd0 con 0x7f0e741071e0 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e78a95700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e74101b30 0x7f0e7419f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 -- 172.21.15.69:0/3238200247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 msgr2=0x7f0e7419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e7419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 -- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e74101b30 msgr2=0x7f0e7419f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e74101b30 0x7f0e7419f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:22.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e741a3800 con 0x7f0e741071e0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e6a7fc700 10 client.?.objecter ms_handle_connect 0x7f0e741071e0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e6a7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e79a97700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741a3130 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b550 tx=0x7f0e7000b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70015020 con 0x7f0e741071e0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0e70011cb0 con 0x7f0e741071e0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e741a3af0 con 0x7f0e741071e0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 10 client.26066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e741ab3e0 con 0x7f0e741071e0 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.783+0000 7f0e7b4fa700 1 librados: init done 2024-04-02T04:51:22.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.787+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70010d20 con 0x7f0e741071e0 2024-04-02T04:51:22.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.787+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e74102890 con 0x7f0e741071e0 2024-04-02T04:51:22.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0e70018020 con 0x7f0e741071e0 2024-04-02T04:51:22.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0e6006f580 0x7f0e60071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(243..243 src has 74..243) ==== 11306+0+0 (secure 0 0 0) 0x7f0e7008e5b0 con 0x7f0e741071e0 2024-04-02T04:51:22.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 10 client.26066.objecter ms_dispatch 0x7f0e74001550 osd_map(243..243 src has 74..243) 2024-04-02T04:51:22.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 3 client.26066.objecter handle_osd_map got epochs [243,243] > 0 2024-04-02T04:51:22.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 3 client.26066.objecter handle_osd_map decoding full epoch 243 2024-04-02T04:51:22.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.791+0000 7f0e6a7fc700 20 client.26066.objecter dump_active .. 0 homeless 2024-04-02T04:51:22.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.795+0000 7f0e79296700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0e6006f580 0x7f0e60071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.795+0000 7f0e6a7fc700 10 client.26066.objecter ms_handle_connect 0x7f0e6006f580 2024-04-02T04:51:22.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.795+0000 7f0e79296700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0e6006f580 0x7f0e60071a40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00a400 tx=0x7f0e6c00d040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.799+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0e7001f020 con 0x7f0e741071e0 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 10 client.26063.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 12 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 12 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf410f2e0 osd=5 3 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 5 12 2024-04-02T04:51:22.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf410f2e0 osd=5 4 2024-04-02T04:51:22.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 10 client.26063.objecter _send_command 12 2024-04-02T04:51:22.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.811+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf41b91b0 con 0x7fdaf41b2e60 2024-04-02T04:51:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf41b91b0 con 0x7fdaf41b2e60 2024-04-02T04:51:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf410f2e0 osd=5 4 2024-04-02T04:51:22.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 5 12 2024-04-02T04:51:22.817 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 10 client.26063.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 13 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fdaf41b98d0 0x7fdaf41b9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 13 2024-04-02T04:51:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 6 13 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41b91b0 osd=6 4 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 10 client.26063.objecter _send_command 13 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdaf41beec0 con 0x7fdaf41b98d0 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fdaf41b98d0 0x7fdaf41b9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 10 client.26063.objecter ms_handle_connect 0x7fdaf41b98d0 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf3fff700 1 --2- 172.21.15.69:0/1144499848 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fdaf41b98d0 0x7fdaf41b9cf0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf41beec0 con 0x7fdaf41b98d0 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 13: 0 ) 2024-04-02T04:51:22.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 13 = system:0 2024-04-02T04:51:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf41b91b0 osd=6 4 2024-04-02T04:51:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.815+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 6 13 2024-04-02T04:51:22.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:22.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 10 client.26063.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:22.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 14 2024-04-02T04:51:22.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 20 client.26063.objecter _get_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 15 client.26063.objecter _session_command_op_remove -1 14 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 20 client.26063.objecter get_session s=0x7fdaf41b91b0 osd=6 3 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign 6 14 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 20 client.26063.objecter put_session s=0x7fdaf41b91b0 osd=6 4 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 10 client.26063.objecter _send_command 14 2024-04-02T04:51:22.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.835+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdaf41ba340 con 0x7fdaf41b98d0 2024-04-02T04:51:22.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdaf41ba340 con 0x7fdaf41b98d0 2024-04-02T04:51:22.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:22.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 10 client.26063.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:22.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 20 client.26063.objecter put_session s=0x7fdaf41b91b0 osd=6 4 2024-04-02T04:51:22.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 15 client.26063.objecter _session_command_op_remove 6 14 2024-04-02T04:51:22.841 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:51:22.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdafa014700 10 client.26063.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:22.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdafa014700 15 client.26063.objecter _session_command_op_assign -1 15 2024-04-02T04:51:22.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdafa014700 20 client.26063.objecter _get_session osd=-1 returning homeless 2024-04-02T04:51:22.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdafa014700 10 client.26063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=243}) -- 0x7fdaf41bb6d0 con 0x7fdaf41011a0 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdafa014700 1 -- 172.21.15.69:0/1144499848 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdaf41ba340 con 0x7fdaf41011a0 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(243..243 src has 74..243) ==== 1098+0+0 (secure 0 0 0) 0x7fdae405d040 con 0x7fdaf41011a0 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 10 client.26063.objecter ms_dispatch 0x7fdaf4001550 osd_map(243..243 src has 74..243) 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 3 client.26063.objecter handle_osd_map got epochs [243,243] > 242 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 3 client.26063.objecter handle_osd_map decoding incremental epoch 243 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 10 client.26063.objecter checking command 15 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 10 client.26063.objecter _check_command_map_dne tid 15 current 243 map_dne_bound 0 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 20 client.26063.objecter dump_active .. 1 homeless 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf0ff9700 1 -- 172.21.15.69:0/1144499848 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_get_version_reply(handle=1 version=243) ==== 24+0+0 (secure 0 0 0) 0x7fdae4063820 con 0x7fdaf41011a0 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf1ffb700 10 client.26063.objecter _check_command_map_dne tid 15 current 243 map_dne_bound 243 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf1ffb700 10 client.26063.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.839+0000 7fdaf1ffb700 15 client.26063.objecter _session_command_op_remove -1 15 2024-04-02T04:51:22.842 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:51:22.843 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:51:22.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:22.843+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]} v 0) -- 0x7f058c1a6e80 con 0x7f058c101b60 2024-04-02T04:51:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:22.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:22.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:22.978 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:22.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.023+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f0e7404ff00 con 0x7f0e741071e0 2024-04-02T04:51:23.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:23.104 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:23.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:23.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:51:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:23.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:23.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: cluster 2024-04-02T04:51:21.946880+0000 mgr.y (mgr.14538) 310 : cluster 0 pgmap v340: 356 pgs: 32 unknown, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 271/1842 objects degraded (14.712%) 2024-04-02T04:51:23.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.623448+0000 mon.c (mon.2) 333 : audit 0 from='client.? 172.21.15.69:0/1144499848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.754698+0000 mon.a (mon.0) 1713 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.754802+0000 mon.a (mon.0) 1714 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.754838+0000 mon.b (mon.1) 178 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: cluster 2024-04-02T04:51:22.759311+0000 mon.a (mon.0) 1715 : cluster 0 osdmap e243: 8 total, 7 up, 8 in 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.760300+0000 mon.a (mon.0) 1716 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.766204+0000 mon.c (mon.2) 334 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.772530+0000 mon.a (mon.0) 1717 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.832867+0000 mon.c (mon.2) 335 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:22.850095+0000 mon.a (mon.0) 1718 : audit 1 from='client.? 172.21.15.69:0/3838671543' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:23.028748+0000 mon.c (mon.2) 336 : audit 1 from='client.? 172.21.15.69:0/3238200247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:51:23.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:23 smithi098 bash[23790]: audit 2024-04-02T04:51:23.029166+0000 mon.a (mon.0) 1719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:51:23.365 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:23.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: cluster 2024-04-02T04:51:21.946880+0000 mgr.y (mgr.14538) 310 : cluster 0 pgmap v340: 356 pgs: 32 unknown, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 271/1842 objects degraded (14.712%) 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.623448+0000 mon.c (mon.2) 333 : audit 0 from='client.? 172.21.15.69:0/1144499848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.754698+0000 mon.a (mon.0) 1713 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.754802+0000 mon.a (mon.0) 1714 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.754838+0000 mon.b (mon.1) 178 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: cluster 2024-04-02T04:51:22.759311+0000 mon.a (mon.0) 1715 : cluster 0 osdmap e243: 8 total, 7 up, 8 in 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.760300+0000 mon.a (mon.0) 1716 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.766204+0000 mon.c (mon.2) 334 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:23.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.772530+0000 mon.a (mon.0) 1717 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.832867+0000 mon.c (mon.2) 335 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:22.850095+0000 mon.a (mon.0) 1718 : audit 1 from='client.? 172.21.15.69:0/3838671543' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:23.028748+0000 mon.c (mon.2) 336 : audit 1 from='client.? 172.21.15.69:0/3238200247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[28904]: audit 2024-04-02T04:51:23.029166+0000 mon.a (mon.0) 1719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: cluster 2024-04-02T04:51:21.946880+0000 mgr.y (mgr.14538) 310 : cluster 0 pgmap v340: 356 pgs: 32 unknown, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 271/1842 objects degraded (14.712%) 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.623448+0000 mon.c (mon.2) 333 : audit 0 from='client.? 172.21.15.69:0/1144499848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.754698+0000 mon.a (mon.0) 1713 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.754802+0000 mon.a (mon.0) 1714 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-72153-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.754838+0000 mon.b (mon.1) 178 : audit 1 from='client.? 172.21.15.69:0/1456953027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:23.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: cluster 2024-04-02T04:51:22.759311+0000 mon.a (mon.0) 1715 : cluster 0 osdmap e243: 8 total, 7 up, 8 in 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.760300+0000 mon.a (mon.0) 1716 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]: dispatch 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.766204+0000 mon.c (mon.2) 334 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.772530+0000 mon.a (mon.0) 1717 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.832867+0000 mon.c (mon.2) 335 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:22.850095+0000 mon.a (mon.0) 1718 : audit 1 from='client.? 172.21.15.69:0/3838671543' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]: dispatch 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:23.028748+0000 mon.c (mon.2) 336 : audit 1 from='client.? 172.21.15.69:0/3238200247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:51:23.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:23 smithi069 bash[21528]: audit 2024-04-02T04:51:23.029166+0000 mon.a (mon.0) 1719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:51:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:23.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:23.624 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:23.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:23.752 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:23.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.751+0000 7f05909f1700 1 -- 172.21.15.69:0/3838671543 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]=0 set 102.7 pg_upmap mapping to [5,0,4] v244) ==== 131+0+0 (secure 0 0 0) 0x7f058005c840 con 0x7f058c101b60 2024-04-02T04:51:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:23.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 10 librados: watch_flush enter 2024-04-02T04:51:23.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 10 librados: watch_flush exit 2024-04-02T04:51:23.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 20 client.15984.objecter shutdown clearing up homeless session... 2024-04-02T04:51:23.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 10 client.15984.objecter successfully canceled tick 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f057406f5e0 msgr2=0x7f0574071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f057406f5e0 0x7f0574071aa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f057c009680 tx=0x7f057c009260 comp rx=0 tx=0).stop 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c101b60 msgr2=0x7f058c103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c101b60 0x7f058c103360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f05800059d0 tx=0x7f05800047b0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 shutdown_connections 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f057406f5e0 0x7f0574071aa0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058c107220 0x7f058c1038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f058c1011a0 0x7f058c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 --2- 172.21.15.69:0/3838671543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f058c101b60 0x7f058c103360 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 >> 172.21.15.69:0/3838671543 conn(0x7f058c0771e0 msgr2=0x7f058c069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:23.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 shutdown_connections 2024-04-02T04:51:23.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 -- 172.21.15.69:0/3838671543 wait complete. 2024-04-02T04:51:23.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.755+0000 7f05919f3700 1 librados: shutdown 2024-04-02T04:51:23.758 INFO:teuthology.orchestra.run.smithi069.stderr:set 102.7 pg_upmap mapping to [5,0,4] 2024-04-02T04:51:23.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-04-02T04:51:23.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (17163 ms) 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (32753 ms) 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8515 ms) 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (4345 ms) 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-02T04:51:23.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3746 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (4019 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2960 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2942 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2967 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3042 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2933 ms) 2024-04-02T04:51:23.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3021 ms) 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3363 ms) 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2948 ms) 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3017 ms) 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3481 ms) 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2559 ms) 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-02T04:51:23.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3407 ms) 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2612 ms) 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2978 ms) 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3427 ms) 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2559 ms) 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3467 ms) 2024-04-02T04:51:23.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-02T04:51:23.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2514 ms) 2024-04-02T04:51:23.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-02T04:51:23.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3016 ms) 2024-04-02T04:51:23.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-02T04:51:23.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3039 ms) 2024-04-02T04:51:23.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.759+0000 7f0e6a7fc700 1 -- 172.21.15.69:0/3238200247 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v244) ==== 91+0+0 (secure 0 0 0) 0x7f0e7005ad30 con 0x7f0e741071e0 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 10 librados: watch_flush enter 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 10 librados: watch_flush exit 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 20 client.26066.objecter shutdown clearing up homeless session... 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 10 client.26066.objecter successfully canceled tick 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0e6006f580 msgr2=0x7f0e60071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0e6006f580 0x7f0e60071a40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00a400 tx=0x7f0e6c00d040 comp rx=0 tx=0).stop 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 msgr2=0x7f0e741a3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741a3130 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b550 tx=0x7f0e7000b860 comp rx=0 tx=0).stop 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 shutdown_connections 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0e6006f580 0x7f0e60071a40 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00a400 tx=0x7f0e6c00d040 comp rx=0 tx=0).stop 2024-04-02T04:51:23.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e741071e0 0x7f0e741a3130 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e74101170 0x7f0e7419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 --2- 172.21.15.69:0/3238200247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e74101b30 0x7f0e7419f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:23.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.763+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 >> 172.21.15.69:0/3238200247 conn(0x7f0e74078110 msgr2=0x7f0e7406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:23.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.771+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 shutdown_connections 2024-04-02T04:51:23.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.771+0000 7f0e7b4fa700 1 -- 172.21.15.69:0/3238200247 wait complete. 2024-04-02T04:51:23.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:23.771+0000 7f0e7b4fa700 1 librados: shutdown 2024-04-02T04:51:23.802 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:51:23.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:23.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:23.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:24.003 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:51:24.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:51:24.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:24.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:24.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.078841+0000 mgr.y (mgr.14538) 311 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: cluster 2024-04-02T04:51:23.755446+0000 mon.a (mon.0) 1720 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.757299+0000 mon.a (mon.0) 1721 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.757406+0000 mon.a (mon.0) 1722 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.757470+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.69:0/3838671543' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]': finished 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.757522+0000 mon.a (mon.0) 1724 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: cluster 2024-04-02T04:51:23.763523+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e244: 8 total, 7 up, 8 in 2024-04-02T04:51:24.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.813263+0000 mon.a (mon.0) 1726 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:24.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.815674+0000 mon.a (mon.0) 1727 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:24.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.818289+0000 mon.c (mon.2) 337 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:24.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.819402+0000 mon.a (mon.0) 1728 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-72145-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:24.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[21528]: audit 2024-04-02T04:51:23.819949+0000 mon.a (mon.0) 1729 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:24.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.078841+0000 mgr.y (mgr.14538) 311 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: cluster 2024-04-02T04:51:23.755446+0000 mon.a (mon.0) 1720 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.757299+0000 mon.a (mon.0) 1721 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.757406+0000 mon.a (mon.0) 1722 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.757470+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.69:0/3838671543' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]': finished 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.757522+0000 mon.a (mon.0) 1724 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:51:24.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: cluster 2024-04-02T04:51:23.763523+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e244: 8 total, 7 up, 8 in 2024-04-02T04:51:24.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.813263+0000 mon.a (mon.0) 1726 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:24.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.815674+0000 mon.a (mon.0) 1727 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:24.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.818289+0000 mon.c (mon.2) 337 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:24.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.819402+0000 mon.a (mon.0) 1728 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-72145-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:24.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:24 smithi098 bash[23790]: audit 2024-04-02T04:51:23.819949+0000 mon.a (mon.0) 1729 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:24.396 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:24.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.078841+0000 mgr.y (mgr.14538) 311 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: cluster 2024-04-02T04:51:23.755446+0000 mon.a (mon.0) 1720 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.757299+0000 mon.a (mon.0) 1721 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-72145-28"}]': finished 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.757406+0000 mon.a (mon.0) 1722 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.757470+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.69:0/3838671543' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "102.7", "id": [5, 0, 4]}]': finished 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.757522+0000 mon.a (mon.0) 1724 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:51:24.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: cluster 2024-04-02T04:51:23.763523+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e244: 8 total, 7 up, 8 in 2024-04-02T04:51:24.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.813263+0000 mon.a (mon.0) 1726 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:24.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.815674+0000 mon.a (mon.0) 1727 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:24.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.818289+0000 mon.c (mon.2) 337 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:24.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.819402+0000 mon.a (mon.0) 1728 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-72145-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:24.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:24 smithi069 bash[28904]: audit 2024-04-02T04:51:23.819949+0000 mon.a (mon.0) 1729 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:24.630 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:24.653 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:24.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:24.781 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:24.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:24.908 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:24.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:51:25.011 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:25.038 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:25.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:25.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:25.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:23.836128+0000 mon.c (mon.2) 338 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:25.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: cluster 2024-04-02T04:51:23.947726+0000 mgr.y (mgr.14538) 312 : cluster 0 pgmap v343: 356 pgs: 32 creating+peering, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 271/1725 objects degraded (15.710%) 2024-04-02T04:51:25.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.676727+0000 mon.a (mon.0) 1730 : cluster 3 Health check update: Degraded data redundancy: 271/1842 objects degraded (14.712%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.679736+0000 mon.a (mon.0) 1731 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-72145-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.679866+0000 mon.a (mon.0) 1732 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.685031+0000 mon.a (mon.0) 1733 : cluster 0 osdmap e245: 8 total, 7 up, 8 in 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.685519+0000 mon.a (mon.0) 1734 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-72145-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.691597+0000 mon.c (mon.2) 339 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.692441+0000 mon.a (mon.0) 1735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.693365+0000 mon.c (mon.2) 340 : audit 1 from='client.? 172.21.15.69:0/952746770' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.694445+0000 mon.a (mon.0) 1736 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.757358+0000 mon.a (mon.0) 1737 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[21528]: audit 2024-04-02T04:51:24.837666+0000 mon.c (mon.2) 341 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:23.836128+0000 mon.c (mon.2) 338 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:25.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: cluster 2024-04-02T04:51:23.947726+0000 mgr.y (mgr.14538) 312 : cluster 0 pgmap v343: 356 pgs: 32 creating+peering, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 271/1725 objects degraded (15.710%) 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.676727+0000 mon.a (mon.0) 1730 : cluster 3 Health check update: Degraded data redundancy: 271/1842 objects degraded (14.712%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.679736+0000 mon.a (mon.0) 1731 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-72145-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.679866+0000 mon.a (mon.0) 1732 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.685031+0000 mon.a (mon.0) 1733 : cluster 0 osdmap e245: 8 total, 7 up, 8 in 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.685519+0000 mon.a (mon.0) 1734 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-72145-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.691597+0000 mon.c (mon.2) 339 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.692441+0000 mon.a (mon.0) 1735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.693365+0000 mon.c (mon.2) 340 : audit 1 from='client.? 172.21.15.69:0/952746770' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.694445+0000 mon.a (mon.0) 1736 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.757358+0000 mon.a (mon.0) 1737 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:25 smithi069 bash[28904]: audit 2024-04-02T04:51:24.837666+0000 mon.c (mon.2) 341 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:25.298 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:25.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:25.429 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:25.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:23.836128+0000 mon.c (mon.2) 338 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: cluster 2024-04-02T04:51:23.947726+0000 mgr.y (mgr.14538) 312 : cluster 0 pgmap v343: 356 pgs: 32 creating+peering, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 271/1725 objects degraded (15.710%) 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.676727+0000 mon.a (mon.0) 1730 : cluster 3 Health check update: Degraded data redundancy: 271/1842 objects degraded (14.712%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.679736+0000 mon.a (mon.0) 1731 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-72145-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.679866+0000 mon.a (mon.0) 1732 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.685031+0000 mon.a (mon.0) 1733 : cluster 0 osdmap e245: 8 total, 7 up, 8 in 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.685519+0000 mon.a (mon.0) 1734 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-72145-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.691597+0000 mon.c (mon.2) 339 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.692441+0000 mon.a (mon.0) 1735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.693365+0000 mon.c (mon.2) 340 : audit 1 from='client.? 172.21.15.69:0/952746770' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.694445+0000 mon.a (mon.0) 1736 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.757358+0000 mon.a (mon.0) 1737 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:25 smithi098 bash[23790]: audit 2024-04-02T04:51:24.837666+0000 mon.c (mon.2) 341 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:25.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:25.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:25.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:25.821 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:25.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:25.946 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:25.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:26.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:26.073 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:26.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:51:26.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:26.197 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:26.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:26.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:26.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.603191+0000 osd.5 (osd.5) 159 : cluster 0 2.2 scrub starts 2024-04-02T04:51:26.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.605674+0000 osd.5 (osd.5) 160 : cluster 0 2.2 scrub ok 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.635588+0000 osd.3 (osd.3) 201 : cluster 0 10.15 scrub starts 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:24.637845+0000 osd.3 (osd.3) 202 : cluster 0 10.15 scrub ok 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:25.123370+0000 osd.0 (osd.0) 173 : cluster 0 40.d scrub starts 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:25.124919+0000 osd.0 (osd.0) 174 : cluster 0 40.d scrub ok 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: audit 2024-04-02T04:51:25.684297+0000 mon.a (mon.0) 1738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]': finished 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: audit 2024-04-02T04:51:25.684498+0000 mon.a (mon.0) 1739 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: audit 2024-04-02T04:51:25.695055+0000 mon.c (mon.2) 342 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]: dispatch 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: audit 2024-04-02T04:51:25.839505+0000 mon.c (mon.2) 343 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.151379+0000 mon.a (mon.0) 1740 : cluster 0 osdmap e246: 8 total, 7 up, 8 in 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[21528]: audit 2024-04-02T04:51:26.152528+0000 mon.a (mon.0) 1741 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]: dispatch 2024-04-02T04:51:26.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.603191+0000 osd.5 (osd.5) 159 : cluster 0 2.2 scrub starts 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.605674+0000 osd.5 (osd.5) 160 : cluster 0 2.2 scrub ok 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.635588+0000 osd.3 (osd.3) 201 : cluster 0 10.15 scrub starts 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:24.637845+0000 osd.3 (osd.3) 202 : cluster 0 10.15 scrub ok 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:25.123370+0000 osd.0 (osd.0) 173 : cluster 0 40.d scrub starts 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:25.124919+0000 osd.0 (osd.0) 174 : cluster 0 40.d scrub ok 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: audit 2024-04-02T04:51:25.684297+0000 mon.a (mon.0) 1738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]': finished 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: audit 2024-04-02T04:51:25.684498+0000 mon.a (mon.0) 1739 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: audit 2024-04-02T04:51:25.695055+0000 mon.c (mon.2) 342 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]: dispatch 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: audit 2024-04-02T04:51:25.839505+0000 mon.c (mon.2) 343 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.151379+0000 mon.a (mon.0) 1740 : cluster 0 osdmap e246: 8 total, 7 up, 8 in 2024-04-02T04:51:26.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:26 smithi069 bash[28904]: audit 2024-04-02T04:51:26.152528+0000 mon.a (mon.0) 1741 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]: dispatch 2024-04-02T04:51:26.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.603191+0000 osd.5 (osd.5) 159 : cluster 0 2.2 scrub starts 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.605674+0000 osd.5 (osd.5) 160 : cluster 0 2.2 scrub ok 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.635588+0000 osd.3 (osd.3) 201 : cluster 0 10.15 scrub starts 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:24.637845+0000 osd.3 (osd.3) 202 : cluster 0 10.15 scrub ok 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:25.123370+0000 osd.0 (osd.0) 173 : cluster 0 40.d scrub starts 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:25.124919+0000 osd.0 (osd.0) 174 : cluster 0 40.d scrub ok 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: audit 2024-04-02T04:51:25.684297+0000 mon.a (mon.0) 1738 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-17"}]': finished 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: audit 2024-04-02T04:51:25.684498+0000 mon.a (mon.0) 1739 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-72153-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: audit 2024-04-02T04:51:25.695055+0000 mon.c (mon.2) 342 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]: dispatch 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: audit 2024-04-02T04:51:25.839505+0000 mon.c (mon.2) 343 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.151379+0000 mon.a (mon.0) 1740 : cluster 0 osdmap e246: 8 total, 7 up, 8 in 2024-04-02T04:51:26.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:26 smithi098 bash[23790]: audit 2024-04-02T04:51:26.152528+0000 mon.a (mon.0) 1741 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]: dispatch 2024-04-02T04:51:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:26.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:26.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:26.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:26.846 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:26.949 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:26.971 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:26.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:27.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:27.098 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:27.201 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:27.203 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:51:27.221 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:27.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:27.351 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:27.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:25.639623+0000 osd.5 (osd.5) 161 : cluster 0 13.17 scrub starts 2024-04-02T04:51:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:25.643179+0000 osd.5 (osd.5) 162 : cluster 0 13.17 scrub ok 2024-04-02T04:51:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:25.949359+0000 mgr.y (mgr.14538) 313 : cluster 0 pgmap v346: 388 pgs: 32 unknown, 32 creating+peering, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 271/1725 objects degraded (15.710%) 2024-04-02T04:51:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.161207+0000 osd.2 (osd.2) 117 : cluster 0 43.3 scrub starts 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.162788+0000 osd.2 (osd.2) 118 : cluster 0 43.3 scrub ok 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.233620+0000 osd.1 (osd.1) 183 : cluster 0 102.e scrub starts 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.235655+0000 osd.1 (osd.1) 184 : cluster 0 102.e scrub ok 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.344150+0000 osd.4 (osd.4) 171 : cluster 0 10.1d scrub starts 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.346341+0000 osd.4 (osd.4) 172 : cluster 0 10.1d scrub ok 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.686308+0000 mon.a (mon.0) 1742 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: audit 2024-04-02T04:51:26.694785+0000 mon.a (mon.0) 1743 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-72145-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: audit 2024-04-02T04:51:26.695042+0000 mon.a (mon.0) 1744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]': finished 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.701883+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e247: 8 total, 7 up, 8 in 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: audit 2024-04-02T04:51:26.804212+0000 mon.c (mon.2) 344 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: audit 2024-04-02T04:51:26.841641+0000 mon.c (mon.2) 345 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:27.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[28904]: audit 2024-04-02T04:51:27.183340+0000 mon.a (mon.0) 1746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:25.639623+0000 osd.5 (osd.5) 161 : cluster 0 13.17 scrub starts 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:25.643179+0000 osd.5 (osd.5) 162 : cluster 0 13.17 scrub ok 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:25.949359+0000 mgr.y (mgr.14538) 313 : cluster 0 pgmap v346: 388 pgs: 32 unknown, 32 creating+peering, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 271/1725 objects degraded (15.710%) 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.161207+0000 osd.2 (osd.2) 117 : cluster 0 43.3 scrub starts 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.162788+0000 osd.2 (osd.2) 118 : cluster 0 43.3 scrub ok 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.233620+0000 osd.1 (osd.1) 183 : cluster 0 102.e scrub starts 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.235655+0000 osd.1 (osd.1) 184 : cluster 0 102.e scrub ok 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.344150+0000 osd.4 (osd.4) 171 : cluster 0 10.1d scrub starts 2024-04-02T04:51:27.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.346341+0000 osd.4 (osd.4) 172 : cluster 0 10.1d scrub ok 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.686308+0000 mon.a (mon.0) 1742 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: audit 2024-04-02T04:51:26.694785+0000 mon.a (mon.0) 1743 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-72145-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: audit 2024-04-02T04:51:26.695042+0000 mon.a (mon.0) 1744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]': finished 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.701883+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e247: 8 total, 7 up, 8 in 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: audit 2024-04-02T04:51:26.804212+0000 mon.c (mon.2) 344 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: audit 2024-04-02T04:51:26.841641+0000 mon.c (mon.2) 345 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:27.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:27 smithi069 bash[21528]: audit 2024-04-02T04:51:27.183340+0000 mon.a (mon.0) 1746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:27.479 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:27.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:27.608 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:27.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:25.639623+0000 osd.5 (osd.5) 161 : cluster 0 13.17 scrub starts 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:25.643179+0000 osd.5 (osd.5) 162 : cluster 0 13.17 scrub ok 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:25.949359+0000 mgr.y (mgr.14538) 313 : cluster 0 pgmap v346: 388 pgs: 32 unknown, 32 creating+peering, 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 80 active+undersized, 189 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 271/1725 objects degraded (15.710%) 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.161207+0000 osd.2 (osd.2) 117 : cluster 0 43.3 scrub starts 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.162788+0000 osd.2 (osd.2) 118 : cluster 0 43.3 scrub ok 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.233620+0000 osd.1 (osd.1) 183 : cluster 0 102.e scrub starts 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.235655+0000 osd.1 (osd.1) 184 : cluster 0 102.e scrub ok 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.344150+0000 osd.4 (osd.4) 171 : cluster 0 10.1d scrub starts 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.346341+0000 osd.4 (osd.4) 172 : cluster 0 10.1d scrub ok 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.686308+0000 mon.a (mon.0) 1742 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: audit 2024-04-02T04:51:26.694785+0000 mon.a (mon.0) 1743 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-72145-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: audit 2024-04-02T04:51:26.695042+0000 mon.a (mon.0) 1744 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-17", "mode": "writeback"}]': finished 2024-04-02T04:51:27.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.701883+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e247: 8 total, 7 up, 8 in 2024-04-02T04:51:27.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: audit 2024-04-02T04:51:26.804212+0000 mon.c (mon.2) 344 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:27.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: audit 2024-04-02T04:51:26.841641+0000 mon.c (mon.2) 345 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:27.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:27 smithi098 bash[23790]: audit 2024-04-02T04:51:27.183340+0000 mon.a (mon.0) 1746 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:27.743 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:27.847 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:27.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:27.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:27.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:28.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:28.134 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:28.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:28.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:28.393 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:28.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.659464+0000 osd.5 (osd.5) 163 : cluster 0 43.1 scrub starts 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:26.661370+0000 osd.5 (osd.5) 164 : cluster 0 43.1 scrub ok 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.140844+0000 osd.0 (osd.0) 175 : cluster 0 13.3 scrub starts 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.141438+0000 osd.2 (osd.2) 119 : cluster 0 10.3 scrub starts 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.142899+0000 osd.2 (osd.2) 120 : cluster 0 10.3 scrub ok 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.142901+0000 osd.0 (osd.0) 176 : cluster 0 13.3 scrub ok 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.254918+0000 osd.1 (osd.1) 185 : cluster 0 102.5 scrub starts 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.257874+0000 osd.1 (osd.1) 186 : cluster 0 102.5 scrub ok 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: audit 2024-04-02T04:51:27.698725+0000 mon.a (mon.0) 1747 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:28.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.706748+0000 mon.a (mon.0) 1748 : cluster 0 osdmap e248: 8 total, 7 up, 8 in 2024-04-02T04:51:28.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: audit 2024-04-02T04:51:27.712707+0000 mon.c (mon.2) 346 : audit 1 from='client.? 172.21.15.69:0/4013500518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:28.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: audit 2024-04-02T04:51:27.713695+0000 mon.a (mon.0) 1749 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:28.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: audit 2024-04-02T04:51:27.714166+0000 mon.c (mon.2) 347 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:28.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: audit 2024-04-02T04:51:27.715182+0000 mon.a (mon.0) 1750 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:28.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:28 smithi098 bash[23790]: audit 2024-04-02T04:51:27.842775+0000 mon.c (mon.2) 348 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:28.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.659464+0000 osd.5 (osd.5) 163 : cluster 0 43.1 scrub starts 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:26.661370+0000 osd.5 (osd.5) 164 : cluster 0 43.1 scrub ok 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.140844+0000 osd.0 (osd.0) 175 : cluster 0 13.3 scrub starts 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.141438+0000 osd.2 (osd.2) 119 : cluster 0 10.3 scrub starts 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.142899+0000 osd.2 (osd.2) 120 : cluster 0 10.3 scrub ok 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.142901+0000 osd.0 (osd.0) 176 : cluster 0 13.3 scrub ok 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.254918+0000 osd.1 (osd.1) 185 : cluster 0 102.5 scrub starts 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.257874+0000 osd.1 (osd.1) 186 : cluster 0 102.5 scrub ok 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: audit 2024-04-02T04:51:27.698725+0000 mon.a (mon.0) 1747 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.706748+0000 mon.a (mon.0) 1748 : cluster 0 osdmap e248: 8 total, 7 up, 8 in 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: audit 2024-04-02T04:51:27.712707+0000 mon.c (mon.2) 346 : audit 1 from='client.? 172.21.15.69:0/4013500518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: audit 2024-04-02T04:51:27.713695+0000 mon.a (mon.0) 1749 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:28.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: audit 2024-04-02T04:51:27.714166+0000 mon.c (mon.2) 347 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: audit 2024-04-02T04:51:27.715182+0000 mon.a (mon.0) 1750 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21528]: audit 2024-04-02T04:51:27.842775+0000 mon.c (mon.2) 348 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.659464+0000 osd.5 (osd.5) 163 : cluster 0 43.1 scrub starts 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:26.661370+0000 osd.5 (osd.5) 164 : cluster 0 43.1 scrub ok 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.140844+0000 osd.0 (osd.0) 175 : cluster 0 13.3 scrub starts 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.141438+0000 osd.2 (osd.2) 119 : cluster 0 10.3 scrub starts 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.142899+0000 osd.2 (osd.2) 120 : cluster 0 10.3 scrub ok 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.142901+0000 osd.0 (osd.0) 176 : cluster 0 13.3 scrub ok 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.254918+0000 osd.1 (osd.1) 185 : cluster 0 102.5 scrub starts 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.257874+0000 osd.1 (osd.1) 186 : cluster 0 102.5 scrub ok 2024-04-02T04:51:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: audit 2024-04-02T04:51:27.698725+0000 mon.a (mon.0) 1747 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.706748+0000 mon.a (mon.0) 1748 : cluster 0 osdmap e248: 8 total, 7 up, 8 in 2024-04-02T04:51:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: audit 2024-04-02T04:51:27.712707+0000 mon.c (mon.2) 346 : audit 1 from='client.? 172.21.15.69:0/4013500518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: audit 2024-04-02T04:51:27.713695+0000 mon.a (mon.0) 1749 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: audit 2024-04-02T04:51:27.714166+0000 mon.c (mon.2) 347 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: audit 2024-04-02T04:51:27.715182+0000 mon.a (mon.0) 1750 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]: dispatch 2024-04-02T04:51:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[28904]: audit 2024-04-02T04:51:27.842775+0000 mon.c (mon.2) 348 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:28.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:28.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:28.694 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:51:28.695 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:51:28.695 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:51:28.695 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:51:28.695 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:51:28.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:28.779 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:28.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:28.891 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:51:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:51:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:51:28.902 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:28.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:29.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:29.032 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:29.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:29.161 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:29.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:29.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.693591+0000 osd.3 (osd.3) 203 : cluster 0 41.3 scrub starts 2024-04-02T04:51:29.275 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.695431+0000 osd.3 (osd.3) 204 : cluster 0 41.3 scrub ok 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:27.951514+0000 mgr.y (mgr.14538) 314 : cluster 0 pgmap v349: 396 pgs: 32 creating+peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 100 active+undersized, 212 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s; 272/1653 objects degraded (16.455%); 2 B/s, 1 objects/s recovering 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:28.129183+0000 osd.2 (osd.2) 121 : cluster 0 10.18 scrub starts 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:28.130751+0000 osd.2 (osd.2) 122 : cluster 0 10.18 scrub ok 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:28.313138+0000 osd.4 (osd.4) 173 : cluster 0 13.1e scrub starts 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:28.314673+0000 osd.4 (osd.4) 174 : cluster 0 13.1e scrub ok 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:28.700240+0000 mon.a (mon.0) 1751 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: audit 2024-04-02T04:51:28.712023+0000 mon.a (mon.0) 1752 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: audit 2024-04-02T04:51:28.712280+0000 mon.a (mon.0) 1753 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]': finished 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: cluster 2024-04-02T04:51:28.721671+0000 mon.a (mon.0) 1754 : cluster 0 osdmap e249: 8 total, 7 up, 8 in 2024-04-02T04:51:29.276 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: audit 2024-04-02T04:51:28.725135+0000 mon.a (mon.0) 1755 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:29.277 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:29 smithi098 bash[23790]: audit 2024-04-02T04:51:28.844247+0000 mon.c (mon.2) 349 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:29.394 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:29.405 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.693591+0000 osd.3 (osd.3) 203 : cluster 0 41.3 scrub starts 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.695431+0000 osd.3 (osd.3) 204 : cluster 0 41.3 scrub ok 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:27.951514+0000 mgr.y (mgr.14538) 314 : cluster 0 pgmap v349: 396 pgs: 32 creating+peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 100 active+undersized, 212 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s; 272/1653 objects degraded (16.455%); 2 B/s, 1 objects/s recovering 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:28.129183+0000 osd.2 (osd.2) 121 : cluster 0 10.18 scrub starts 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:28.130751+0000 osd.2 (osd.2) 122 : cluster 0 10.18 scrub ok 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:28.313138+0000 osd.4 (osd.4) 173 : cluster 0 13.1e scrub starts 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:28.314673+0000 osd.4 (osd.4) 174 : cluster 0 13.1e scrub ok 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:28.700240+0000 mon.a (mon.0) 1751 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: audit 2024-04-02T04:51:28.712023+0000 mon.a (mon.0) 1752 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: audit 2024-04-02T04:51:28.712280+0000 mon.a (mon.0) 1753 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]': finished 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: cluster 2024-04-02T04:51:28.721671+0000 mon.a (mon.0) 1754 : cluster 0 osdmap e249: 8 total, 7 up, 8 in 2024-04-02T04:51:29.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: audit 2024-04-02T04:51:28.725135+0000 mon.a (mon.0) 1755 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[28904]: audit 2024-04-02T04:51:28.844247+0000 mon.c (mon.2) 349 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.693591+0000 osd.3 (osd.3) 203 : cluster 0 41.3 scrub starts 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.695431+0000 osd.3 (osd.3) 204 : cluster 0 41.3 scrub ok 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:27.951514+0000 mgr.y (mgr.14538) 314 : cluster 0 pgmap v349: 396 pgs: 32 creating+peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 100 active+undersized, 212 active+clean, 37 active+undersized+degraded; 588 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s; 272/1653 objects degraded (16.455%); 2 B/s, 1 objects/s recovering 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:28.129183+0000 osd.2 (osd.2) 121 : cluster 0 10.18 scrub starts 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:28.130751+0000 osd.2 (osd.2) 122 : cluster 0 10.18 scrub ok 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:28.313138+0000 osd.4 (osd.4) 173 : cluster 0 13.1e scrub starts 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:28.314673+0000 osd.4 (osd.4) 174 : cluster 0 13.1e scrub ok 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:28.700240+0000 mon.a (mon.0) 1751 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: audit 2024-04-02T04:51:28.712023+0000 mon.a (mon.0) 1752 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-72153-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:29.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: audit 2024-04-02T04:51:28.712280+0000 mon.a (mon.0) 1753 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-17"}]': finished 2024-04-02T04:51:29.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: cluster 2024-04-02T04:51:28.721671+0000 mon.a (mon.0) 1754 : cluster 0 osdmap e249: 8 total, 7 up, 8 in 2024-04-02T04:51:29.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: audit 2024-04-02T04:51:28.725135+0000 mon.a (mon.0) 1755 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:29.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:29 smithi069 bash[21528]: audit 2024-04-02T04:51:28.844247+0000 mon.c (mon.2) 349 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:29.420 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:29.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:29.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-02T04:51:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:29.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:29.778 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:29.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:29.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:29.922 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:29.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:30.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:30.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:30.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:30.180 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:30.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-04-02T04:51:30.198 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:51:30.198 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-02T04:51:30.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-02T04:51:30.283 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:30.306 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:30.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:30.428 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:30.531 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:30.551 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:30.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:30.680 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.316633+0000 osd.1 (osd.1) 187 : cluster 0 10.19 scrub starts 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.318988+0000 osd.1 (osd.1) 188 : cluster 0 10.19 scrub ok 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.321849+0000 osd.4 (osd.4) 175 : cluster 0 13.6 scrub starts 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.324010+0000 osd.4 (osd.4) 176 : cluster 0 13.6 scrub ok 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.678803+0000 mon.a (mon.0) 1756 : cluster 3 Health check update: Degraded data redundancy: 272/1653 objects degraded (16.455%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: audit 2024-04-02T04:51:29.720384+0000 mon.a (mon.0) 1757 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.727955+0000 mon.a (mon.0) 1758 : cluster 0 osdmap e250: 8 total, 7 up, 8 in 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: audit 2024-04-02T04:51:29.742725+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[28904]: audit 2024-04-02T04:51:29.845240+0000 mon.c (mon.2) 350 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:30.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.316633+0000 osd.1 (osd.1) 187 : cluster 0 10.19 scrub starts 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.318988+0000 osd.1 (osd.1) 188 : cluster 0 10.19 scrub ok 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.321849+0000 osd.4 (osd.4) 175 : cluster 0 13.6 scrub starts 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.324010+0000 osd.4 (osd.4) 176 : cluster 0 13.6 scrub ok 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.678803+0000 mon.a (mon.0) 1756 : cluster 3 Health check update: Degraded data redundancy: 272/1653 objects degraded (16.455%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: audit 2024-04-02T04:51:29.720384+0000 mon.a (mon.0) 1757 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.727955+0000 mon.a (mon.0) 1758 : cluster 0 osdmap e250: 8 total, 7 up, 8 in 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: audit 2024-04-02T04:51:29.742725+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:30.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:30 smithi069 bash[21528]: audit 2024-04-02T04:51:29.845240+0000 mon.c (mon.2) 350 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:30.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:30.802 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:30.886 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:30.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:30.924 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:30.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.316633+0000 osd.1 (osd.1) 187 : cluster 0 10.19 scrub starts 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.318988+0000 osd.1 (osd.1) 188 : cluster 0 10.19 scrub ok 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.321849+0000 osd.4 (osd.4) 175 : cluster 0 13.6 scrub starts 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.324010+0000 osd.4 (osd.4) 176 : cluster 0 13.6 scrub ok 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.678803+0000 mon.a (mon.0) 1756 : cluster 3 Health check update: Degraded data redundancy: 272/1653 objects degraded (16.455%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: audit 2024-04-02T04:51:29.720384+0000 mon.a (mon.0) 1757 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.727955+0000 mon.a (mon.0) 1758 : cluster 0 osdmap e250: 8 total, 7 up, 8 in 2024-04-02T04:51:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: audit 2024-04-02T04:51:29.742725+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]: dispatch 2024-04-02T04:51:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:30 smithi098 bash[23790]: audit 2024-04-02T04:51:29.845240+0000 mon.c (mon.2) 350 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:31.027 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:31.053 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:31.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:31.180 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:31.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:31.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:31.437 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:31.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:31.565 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:31.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:31.696 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:31.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:31.818 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:31.906 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:31.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:31.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.647239+0000 osd.3 (osd.3) 205 : cluster 0 40.2 scrub starts 2024-04-02T04:51:31.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.649409+0000 osd.3 (osd.3) 206 : cluster 0 40.2 scrub ok 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.672001+0000 osd.5 (osd.5) 165 : cluster 0 40.c scrub starts 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.673790+0000 osd.5 (osd.5) 166 : cluster 0 40.c scrub ok 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:29.952396+0000 mgr.y (mgr.14538) 315 : cluster 0 pgmap v352: 324 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 80 active+undersized, 192 active+clean, 37 active+undersized+degraded; 587 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 272/1650 objects degraded (16.485%); 2 B/s, 1 objects/s recovering 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.296707+0000 osd.4 (osd.4) 177 : cluster 0 13.1b scrub starts 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.299906+0000 osd.4 (osd.4) 178 : cluster 0 13.1b scrub ok 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.304878+0000 osd.1 (osd.1) 189 : cluster 0 41.1f scrub starts 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.306337+0000 osd.1 (osd.1) 190 : cluster 0 41.1f scrub ok 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.640805+0000 osd.5 (osd.5) 167 : cluster 0 10.6 scrub starts 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.643297+0000 osd.5 (osd.5) 168 : cluster 0 10.6 scrub ok 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.723379+0000 mon.a (mon.0) 1760 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.725186+0000 mon.a (mon.0) 1761 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: cluster 2024-04-02T04:51:30.735266+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e251: 8 total, 7 up, 8 in 2024-04-02T04:51:31.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.739644+0000 mon.c (mon.2) 351 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.740268+0000 mon.c (mon.2) 352 : audit 1 from='client.? 172.21.15.69:0/996993638' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.745885+0000 mon.a (mon.0) 1763 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.746081+0000 mon.a (mon.0) 1764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.776662+0000 mon.c (mon.2) 353 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.776976+0000 mon.a (mon.0) 1765 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.777824+0000 mon.c (mon.2) 354 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.778097+0000 mon.a (mon.0) 1766 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.778533+0000 mon.c (mon.2) 355 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.778818+0000 mon.a (mon.0) 1767 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:31.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[21528]: audit 2024-04-02T04:51:30.846399+0000 mon.c (mon.2) 356 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.647239+0000 osd.3 (osd.3) 205 : cluster 0 40.2 scrub starts 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.649409+0000 osd.3 (osd.3) 206 : cluster 0 40.2 scrub ok 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.672001+0000 osd.5 (osd.5) 165 : cluster 0 40.c scrub starts 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.673790+0000 osd.5 (osd.5) 166 : cluster 0 40.c scrub ok 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:29.952396+0000 mgr.y (mgr.14538) 315 : cluster 0 pgmap v352: 324 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 80 active+undersized, 192 active+clean, 37 active+undersized+degraded; 587 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 272/1650 objects degraded (16.485%); 2 B/s, 1 objects/s recovering 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.296707+0000 osd.4 (osd.4) 177 : cluster 0 13.1b scrub starts 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.299906+0000 osd.4 (osd.4) 178 : cluster 0 13.1b scrub ok 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.304878+0000 osd.1 (osd.1) 189 : cluster 0 41.1f scrub starts 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.306337+0000 osd.1 (osd.1) 190 : cluster 0 41.1f scrub ok 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.640805+0000 osd.5 (osd.5) 167 : cluster 0 10.6 scrub starts 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.643297+0000 osd.5 (osd.5) 168 : cluster 0 10.6 scrub ok 2024-04-02T04:51:31.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.723379+0000 mon.a (mon.0) 1760 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.725186+0000 mon.a (mon.0) 1761 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: cluster 2024-04-02T04:51:30.735266+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e251: 8 total, 7 up, 8 in 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.739644+0000 mon.c (mon.2) 351 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.740268+0000 mon.c (mon.2) 352 : audit 1 from='client.? 172.21.15.69:0/996993638' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.745885+0000 mon.a (mon.0) 1763 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.746081+0000 mon.a (mon.0) 1764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.776662+0000 mon.c (mon.2) 353 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.776976+0000 mon.a (mon.0) 1765 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.777824+0000 mon.c (mon.2) 354 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.778097+0000 mon.a (mon.0) 1766 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.778533+0000 mon.c (mon.2) 355 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.778818+0000 mon.a (mon.0) 1767 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:31.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:31 smithi069 bash[28904]: audit 2024-04-02T04:51:30.846399+0000 mon.c (mon.2) 356 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:31.943 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:31.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.647239+0000 osd.3 (osd.3) 205 : cluster 0 40.2 scrub starts 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.649409+0000 osd.3 (osd.3) 206 : cluster 0 40.2 scrub ok 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.672001+0000 osd.5 (osd.5) 165 : cluster 0 40.c scrub starts 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.673790+0000 osd.5 (osd.5) 166 : cluster 0 40.c scrub ok 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:29.952396+0000 mgr.y (mgr.14538) 315 : cluster 0 pgmap v352: 324 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 80 active+undersized, 192 active+clean, 37 active+undersized+degraded; 587 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 272/1650 objects degraded (16.485%); 2 B/s, 1 objects/s recovering 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.296707+0000 osd.4 (osd.4) 177 : cluster 0 13.1b scrub starts 2024-04-02T04:51:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.299906+0000 osd.4 (osd.4) 178 : cluster 0 13.1b scrub ok 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.304878+0000 osd.1 (osd.1) 189 : cluster 0 41.1f scrub starts 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.306337+0000 osd.1 (osd.1) 190 : cluster 0 41.1f scrub ok 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.640805+0000 osd.5 (osd.5) 167 : cluster 0 10.6 scrub starts 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.643297+0000 osd.5 (osd.5) 168 : cluster 0 10.6 scrub ok 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.723379+0000 mon.a (mon.0) 1760 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.725186+0000 mon.a (mon.0) 1761 : audit 1 from='client.? 172.21.15.69:0/2876630749' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-72145-29"}]': finished 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: cluster 2024-04-02T04:51:30.735266+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e251: 8 total, 7 up, 8 in 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.739644+0000 mon.c (mon.2) 351 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.740268+0000 mon.c (mon.2) 352 : audit 1 from='client.? 172.21.15.69:0/996993638' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.745885+0000 mon.a (mon.0) 1763 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.746081+0000 mon.a (mon.0) 1764 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.776662+0000 mon.c (mon.2) 353 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.776976+0000 mon.a (mon.0) 1765 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.777824+0000 mon.c (mon.2) 354 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.778097+0000 mon.a (mon.0) 1766 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:31.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.778533+0000 mon.c (mon.2) 355 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:31.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.778818+0000 mon.a (mon.0) 1767 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:31.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:31 smithi098 bash[23790]: audit 2024-04-02T04:51:30.846399+0000 mon.c (mon.2) 356 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:32.045 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:32.070 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:32.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:32.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:32.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:32.325 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:32.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:32.453 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:32.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:32.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:32.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:32.713 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:32.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:32.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:32.957 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: cluster 2024-04-02T04:51:31.152955+0000 osd.2 (osd.2) 123 : cluster 0 10.5 scrub starts 2024-04-02T04:51:32.957 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: cluster 2024-04-02T04:51:31.154900+0000 osd.2 (osd.2) 124 : cluster 0 10.5 scrub ok 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.728444+0000 mon.a (mon.0) 1768 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.728623+0000 mon.a (mon.0) 1769 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.728870+0000 mon.a (mon.0) 1770 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.738436+0000 mon.c (mon.2) 357 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: cluster 2024-04-02T04:51:31.741364+0000 mon.a (mon.0) 1771 : cluster 0 osdmap e252: 8 total, 7 up, 8 in 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.743201+0000 mon.c (mon.2) 358 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.745119+0000 mon.a (mon.0) 1772 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.745817+0000 mon.a (mon.0) 1773 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:31.847523+0000 mon.c (mon.2) 359 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: cluster 2024-04-02T04:51:31.953401+0000 mgr.y (mgr.14538) 316 : cluster 0 pgmap v355: 388 pgs: 64 unknown, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 80 active+undersized, 192 active+clean, 37 active+undersized+degraded; 587 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 272/1650 objects degraded (16.485%) 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: audit 2024-04-02T04:51:32.076555+0000 mon.b (mon.1) 179 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:32.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: cluster 2024-04-02T04:51:32.579063+0000 osd.3 (osd.3) 207 : cluster 0 13.16 scrub starts 2024-04-02T04:51:32.959 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:32 smithi098 bash[23790]: cluster 2024-04-02T04:51:32.581222+0000 osd.3 (osd.3) 208 : cluster 0 13.16 scrub ok 2024-04-02T04:51:32.971 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:32.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: cluster 2024-04-02T04:51:31.152955+0000 osd.2 (osd.2) 123 : cluster 0 10.5 scrub starts 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: cluster 2024-04-02T04:51:31.154900+0000 osd.2 (osd.2) 124 : cluster 0 10.5 scrub ok 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.728444+0000 mon.a (mon.0) 1768 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.728623+0000 mon.a (mon.0) 1769 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.728870+0000 mon.a (mon.0) 1770 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.738436+0000 mon.c (mon.2) 357 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: cluster 2024-04-02T04:51:31.741364+0000 mon.a (mon.0) 1771 : cluster 0 osdmap e252: 8 total, 7 up, 8 in 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.743201+0000 mon.c (mon.2) 358 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.745119+0000 mon.a (mon.0) 1772 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.745817+0000 mon.a (mon.0) 1773 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:33.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:31.847523+0000 mon.c (mon.2) 359 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: cluster 2024-04-02T04:51:31.953401+0000 mgr.y (mgr.14538) 316 : cluster 0 pgmap v355: 388 pgs: 64 unknown, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 80 active+undersized, 192 active+clean, 37 active+undersized+degraded; 587 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 272/1650 objects degraded (16.485%) 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: audit 2024-04-02T04:51:32.076555+0000 mon.b (mon.1) 179 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: cluster 2024-04-02T04:51:32.579063+0000 osd.3 (osd.3) 207 : cluster 0 13.16 scrub starts 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[28904]: cluster 2024-04-02T04:51:32.581222+0000 osd.3 (osd.3) 208 : cluster 0 13.16 scrub ok 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: cluster 2024-04-02T04:51:31.152955+0000 osd.2 (osd.2) 123 : cluster 0 10.5 scrub starts 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: cluster 2024-04-02T04:51:31.154900+0000 osd.2 (osd.2) 124 : cluster 0 10.5 scrub ok 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.728444+0000 mon.a (mon.0) 1768 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.728623+0000 mon.a (mon.0) 1769 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-72153-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.728870+0000 mon.a (mon.0) 1770 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-72145-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.738436+0000 mon.c (mon.2) 357 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: cluster 2024-04-02T04:51:31.741364+0000 mon.a (mon.0) 1771 : cluster 0 osdmap e252: 8 total, 7 up, 8 in 2024-04-02T04:51:33.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.743201+0000 mon.c (mon.2) 358 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.745119+0000 mon.a (mon.0) 1772 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.745817+0000 mon.a (mon.0) 1773 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:31.847523+0000 mon.c (mon.2) 359 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: cluster 2024-04-02T04:51:31.953401+0000 mgr.y (mgr.14538) 316 : cluster 0 pgmap v355: 388 pgs: 64 unknown, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 80 active+undersized, 192 active+clean, 37 active+undersized+degraded; 587 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 272/1650 objects degraded (16.485%) 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: audit 2024-04-02T04:51:32.076555+0000 mon.b (mon.1) 179 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: cluster 2024-04-02T04:51:32.579063+0000 osd.3 (osd.3) 207 : cluster 0 13.16 scrub starts 2024-04-02T04:51:33.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:32 smithi069 bash[21528]: cluster 2024-04-02T04:51:32.581222+0000 osd.3 (osd.3) 208 : cluster 0 13.16 scrub ok 2024-04-02T04:51:33.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:33.096 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:33.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:33.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:51:33.223 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:33.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:33.352 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:33.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:33.478 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:33.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:33.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:33.729 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[28904]: audit 2024-04-02T04:51:32.740325+0000 mon.a (mon.0) 1774 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[28904]: audit 2024-04-02T04:51:32.755380+0000 mon.c (mon.2) 360 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[28904]: audit 2024-04-02T04:51:32.848771+0000 mon.c (mon.2) 361 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[28904]: audit 2024-04-02T04:51:33.090177+0000 mgr.y (mgr.14538) 317 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.211083+0000 mon.a (mon.0) 1775 : cluster 0 osdmap e253: 8 total, 7 up, 8 in 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[28904]: audit 2024-04-02T04:51:33.212298+0000 mon.a (mon.0) 1776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[21528]: audit 2024-04-02T04:51:32.740325+0000 mon.a (mon.0) 1774 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[21528]: audit 2024-04-02T04:51:32.755380+0000 mon.c (mon.2) 360 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:33.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[21528]: audit 2024-04-02T04:51:32.848771+0000 mon.c (mon.2) 361 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[21528]: audit 2024-04-02T04:51:33.090177+0000 mgr.y (mgr.14538) 317 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.211083+0000 mon.a (mon.0) 1775 : cluster 0 osdmap e253: 8 total, 7 up, 8 in 2024-04-02T04:51:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:33 smithi069 bash[21528]: audit 2024-04-02T04:51:33.212298+0000 mon.a (mon.0) 1776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:33.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:33.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:33.952 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[23790]: audit 2024-04-02T04:51:32.740325+0000 mon.a (mon.0) 1774 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[23790]: audit 2024-04-02T04:51:32.755380+0000 mon.c (mon.2) 360 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[23790]: audit 2024-04-02T04:51:32.848771+0000 mon.c (mon.2) 361 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[23790]: audit 2024-04-02T04:51:33.090177+0000 mgr.y (mgr.14538) 317 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.211083+0000 mon.a (mon.0) 1775 : cluster 0 osdmap e253: 8 total, 7 up, 8 in 2024-04-02T04:51:33.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:33 smithi098 bash[23790]: audit 2024-04-02T04:51:33.212298+0000 mon.a (mon.0) 1776 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:33.974 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:33.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:34.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:34.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:34.233 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:51:34.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:34.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:34.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:34.483 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:34.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:34.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:34.753 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:34.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.175104+0000 osd.2 (osd.2) 125 : cluster 0 4.1c scrub starts 2024-04-02T04:51:34.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.176939+0000 osd.2 (osd.2) 126 : cluster 0 4.1c scrub ok 2024-04-02T04:51:34.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.687380+0000 osd.5 (osd.5) 169 : cluster 0 13.b scrub starts 2024-04-02T04:51:34.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.689741+0000 osd.5 (osd.5) 170 : cluster 0 13.b scrub ok 2024-04-02T04:51:34.775 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.770211+0000 mon.a (mon.0) 1777 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.770307+0000 mon.a (mon.0) 1778 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]': finished 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.775916+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e254: 8 total, 7 up, 8 in 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.778273+0000 mon.c (mon.2) 362 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]: dispatch 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.780472+0000 mon.a (mon.0) 1780 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]: dispatch 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.786636+0000 mon.c (mon.2) 363 : audit 1 from='client.? 172.21.15.69:0/2642753262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.787382+0000 mon.a (mon.0) 1781 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:33.850206+0000 mon.c (mon.2) 364 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.175104+0000 osd.2 (osd.2) 125 : cluster 0 4.1c scrub starts 2024-04-02T04:51:34.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.176939+0000 osd.2 (osd.2) 126 : cluster 0 4.1c scrub ok 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.687380+0000 osd.5 (osd.5) 169 : cluster 0 13.b scrub starts 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.689741+0000 osd.5 (osd.5) 170 : cluster 0 13.b scrub ok 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.770211+0000 mon.a (mon.0) 1777 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.770307+0000 mon.a (mon.0) 1778 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]': finished 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.775916+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e254: 8 total, 7 up, 8 in 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.778273+0000 mon.c (mon.2) 362 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]: dispatch 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.780472+0000 mon.a (mon.0) 1780 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]: dispatch 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.786636+0000 mon.c (mon.2) 363 : audit 1 from='client.? 172.21.15.69:0/2642753262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.787382+0000 mon.a (mon.0) 1781 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:33.850206+0000 mon.c (mon.2) 364 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:33.955165+0000 mgr.y (mgr.14538) 318 : cluster 0 pgmap v358: 396 pgs: 40 unknown, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 91 active+undersized, 218 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 272/1557 objects degraded (17.469%) 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.003315+0000 osd.6 (osd.6) 131 : cluster 0 5.2 scrub starts 2024-04-02T04:51:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.006007+0000 osd.6 (osd.6) 132 : cluster 0 5.2 scrub ok 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.341948+0000 osd.1 (osd.1) 191 : cluster 0 4.4 scrub starts 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.344535+0000 osd.1 (osd.1) 192 : cluster 0 4.4 scrub ok 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.680941+0000 mon.a (mon.0) 1782 : cluster 3 Health check update: Degraded data redundancy: 272/1650 objects degraded (16.485%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.682769+0000 mon.a (mon.0) 1783 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:34.687051+0000 mon.a (mon.0) 1784 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]': finished 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: audit 2024-04-02T04:51:34.687257+0000 mon.a (mon.0) 1785 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:34.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.700440+0000 mon.a (mon.0) 1786 : cluster 0 osdmap e255: 8 total, 7 up, 8 in 2024-04-02T04:51:34.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:34.879 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:34.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.175104+0000 osd.2 (osd.2) 125 : cluster 0 4.1c scrub starts 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.176939+0000 osd.2 (osd.2) 126 : cluster 0 4.1c scrub ok 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.687380+0000 osd.5 (osd.5) 169 : cluster 0 13.b scrub starts 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.689741+0000 osd.5 (osd.5) 170 : cluster 0 13.b scrub ok 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.770211+0000 mon.a (mon.0) 1777 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-72145-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.770307+0000 mon.a (mon.0) 1778 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-19"}]': finished 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.775916+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e254: 8 total, 7 up, 8 in 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.778273+0000 mon.c (mon.2) 362 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]: dispatch 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.780472+0000 mon.a (mon.0) 1780 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]: dispatch 2024-04-02T04:51:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.786636+0000 mon.c (mon.2) 363 : audit 1 from='client.? 172.21.15.69:0/2642753262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.787382+0000 mon.a (mon.0) 1781 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:33.850206+0000 mon.c (mon.2) 364 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:33.955165+0000 mgr.y (mgr.14538) 318 : cluster 0 pgmap v358: 396 pgs: 40 unknown, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 91 active+undersized, 218 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 272/1557 objects degraded (17.469%) 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.003315+0000 osd.6 (osd.6) 131 : cluster 0 5.2 scrub starts 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.006007+0000 osd.6 (osd.6) 132 : cluster 0 5.2 scrub ok 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.341948+0000 osd.1 (osd.1) 191 : cluster 0 4.4 scrub starts 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.344535+0000 osd.1 (osd.1) 192 : cluster 0 4.4 scrub ok 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.680941+0000 mon.a (mon.0) 1782 : cluster 3 Health check update: Degraded data redundancy: 272/1650 objects degraded (16.485%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.682769+0000 mon.a (mon.0) 1783 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:34.687051+0000 mon.a (mon.0) 1784 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]': finished 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: audit 2024-04-02T04:51:34.687257+0000 mon.a (mon.0) 1785 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:34 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.700440+0000 mon.a (mon.0) 1786 : cluster 0 osdmap e255: 8 total, 7 up, 8 in 2024-04-02T04:51:34.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:33.955165+0000 mgr.y (mgr.14538) 318 : cluster 0 pgmap v358: 396 pgs: 40 unknown, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 91 active+undersized, 218 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 272/1557 objects degraded (17.469%) 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.003315+0000 osd.6 (osd.6) 131 : cluster 0 5.2 scrub starts 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.006007+0000 osd.6 (osd.6) 132 : cluster 0 5.2 scrub ok 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.341948+0000 osd.1 (osd.1) 191 : cluster 0 4.4 scrub starts 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.344535+0000 osd.1 (osd.1) 192 : cluster 0 4.4 scrub ok 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.680941+0000 mon.a (mon.0) 1782 : cluster 3 Health check update: Degraded data redundancy: 272/1650 objects degraded (16.485%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.682769+0000 mon.a (mon.0) 1783 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:34.687051+0000 mon.a (mon.0) 1784 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-19", "mode": "writeback"}]': finished 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: audit 2024-04-02T04:51:34.687257+0000 mon.a (mon.0) 1785 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-72153-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:34.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:34 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.700440+0000 mon.a (mon.0) 1786 : cluster 0 osdmap e255: 8 total, 7 up, 8 in 2024-04-02T04:51:35.004 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:35.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:35.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:35.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:35.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:35.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:35.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:35.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:35.634 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:35.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:35.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:35.776 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:35.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:35.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:34.776522+0000 mon.c (mon.2) 365 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:34.777323+0000 mon.a (mon.0) 1787 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:34.851236+0000 mon.c (mon.2) 366 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.201941+0000 mon.c (mon.2) 367 : audit 1 from='client.? 172.21.15.69:0/2642753262' entity='client.admin' cmd=[{ 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.201963+0000 mon.c (mon.2) 368 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.201972+0000 mon.c (mon.2) 369 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.201983+0000 mon.c (mon.2) 370 : audit 1 "var": "eio", 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.201994+0000 mon.c (mon.2) 371 : audit 1 "val": "true" 2024-04-02T04:51:35.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202003+0000 mon.c (mon.2) 372 : audit 1 }]: dispatch 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202474+0000 mon.a (mon.0) 1788 : audit 1 from='client.? ' entity='client.admin' cmd=[{ 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202505+0000 mon.a (mon.0) 1789 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202522+0000 mon.a (mon.0) 1790 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202537+0000 mon.a (mon.0) 1791 : audit 1 "var": "eio", 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202553+0000 mon.a (mon.0) 1792 : audit 1 "val": "true" 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.202571+0000 mon.a (mon.0) 1793 : audit 1 }]: dispatch 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:34.776522+0000 mon.c (mon.2) 365 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:34.777323+0000 mon.a (mon.0) 1787 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:34.851236+0000 mon.c (mon.2) 366 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.201941+0000 mon.c (mon.2) 367 : audit 1 from='client.? 172.21.15.69:0/2642753262' entity='client.admin' cmd=[{ 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.201963+0000 mon.c (mon.2) 368 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:35.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.201972+0000 mon.c (mon.2) 369 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.201983+0000 mon.c (mon.2) 370 : audit 1 "var": "eio", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.201994+0000 mon.c (mon.2) 371 : audit 1 "val": "true" 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202003+0000 mon.c (mon.2) 372 : audit 1 }]: dispatch 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202474+0000 mon.a (mon.0) 1788 : audit 1 from='client.? ' entity='client.admin' cmd=[{ 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202505+0000 mon.a (mon.0) 1789 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202522+0000 mon.a (mon.0) 1790 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202537+0000 mon.a (mon.0) 1791 : audit 1 "var": "eio", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202553+0000 mon.a (mon.0) 1792 : audit 1 "val": "true" 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.202571+0000 mon.a (mon.0) 1793 : audit 1 }]: dispatch 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689045+0000 mon.a (mon.0) 1794 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689138+0000 mon.a (mon.0) 1795 : audit 1 from='client.? ' entity='client.admin' cmd='[{ 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689168+0000 mon.a (mon.0) 1796 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689188+0000 mon.a (mon.0) 1797 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689206+0000 mon.a (mon.0) 1798 : audit 1 "var": "eio", 2024-04-02T04:51:35.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689222+0000 mon.a (mon.0) 1799 : audit 1 "val": "true" 2024-04-02T04:51:36.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:34.776522+0000 mon.c (mon.2) 365 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:34.777323+0000 mon.a (mon.0) 1787 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:34.851236+0000 mon.c (mon.2) 366 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.201941+0000 mon.c (mon.2) 367 : audit 1 from='client.? 172.21.15.69:0/2642753262' entity='client.admin' cmd=[{ 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.201963+0000 mon.c (mon.2) 368 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.201972+0000 mon.c (mon.2) 369 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.201983+0000 mon.c (mon.2) 370 : audit 1 "var": "eio", 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.201994+0000 mon.c (mon.2) 371 : audit 1 "val": "true" 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202003+0000 mon.c (mon.2) 372 : audit 1 }]: dispatch 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202474+0000 mon.a (mon.0) 1788 : audit 1 from='client.? ' entity='client.admin' cmd=[{ 2024-04-02T04:51:36.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202505+0000 mon.a (mon.0) 1789 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202522+0000 mon.a (mon.0) 1790 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202537+0000 mon.a (mon.0) 1791 : audit 1 "var": "eio", 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202553+0000 mon.a (mon.0) 1792 : audit 1 "val": "true" 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.202571+0000 mon.a (mon.0) 1793 : audit 1 }]: dispatch 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689045+0000 mon.a (mon.0) 1794 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689138+0000 mon.a (mon.0) 1795 : audit 1 from='client.? ' entity='client.admin' cmd='[{ 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689168+0000 mon.a (mon.0) 1796 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689188+0000 mon.a (mon.0) 1797 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689206+0000 mon.a (mon.0) 1798 : audit 1 "var": "eio", 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689222+0000 mon.a (mon.0) 1799 : audit 1 "val": "true" 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.689240+0000 mon.a (mon.0) 1800 : audit 1 }]': finished 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: cluster 2024-04-02T04:51:35.695519+0000 mon.a (mon.0) 1801 : cluster 0 osdmap e256: 8 total, 7 up, 8 in 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.696410+0000 mon.c (mon.2) 373 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:36.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.696986+0000 mon.a (mon.0) 1802 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:36.031 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.697295+0000 mon.c (mon.2) 374 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:36.031 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:35 smithi098 bash[23790]: audit 2024-04-02T04:51:35.697860+0000 mon.a (mon.0) 1803 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:36.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.689240+0000 mon.a (mon.0) 1800 : audit 1 }]': finished 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: cluster 2024-04-02T04:51:35.695519+0000 mon.a (mon.0) 1801 : cluster 0 osdmap e256: 8 total, 7 up, 8 in 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.696410+0000 mon.c (mon.2) 373 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.696986+0000 mon.a (mon.0) 1802 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.697295+0000 mon.c (mon.2) 374 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[28904]: audit 2024-04-02T04:51:35.697860+0000 mon.a (mon.0) 1803 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689045+0000 mon.a (mon.0) 1794 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:36.144 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689138+0000 mon.a (mon.0) 1795 : audit 1 from='client.? ' entity='client.admin' cmd='[{ 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689168+0000 mon.a (mon.0) 1796 : audit 1 "prefix": "osd pool set", 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689188+0000 mon.a (mon.0) 1797 : audit 1 "pool": "PoolEIOFlag_smithi069-72153-33", 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689206+0000 mon.a (mon.0) 1798 : audit 1 "var": "eio", 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689222+0000 mon.a (mon.0) 1799 : audit 1 "val": "true" 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.689240+0000 mon.a (mon.0) 1800 : audit 1 }]': finished 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: cluster 2024-04-02T04:51:35.695519+0000 mon.a (mon.0) 1801 : cluster 0 osdmap e256: 8 total, 7 up, 8 in 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.696410+0000 mon.c (mon.2) 373 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.696986+0000 mon.a (mon.0) 1802 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]: dispatch 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.697295+0000 mon.c (mon.2) 374 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:36.145 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:35 smithi069 bash[21528]: audit 2024-04-02T04:51:35.697860+0000 mon.a (mon.0) 1803 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:36.160 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:36.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:36.289 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:36.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:36.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:36.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:36.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:36.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:36.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:36.803 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:36.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:36.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:37.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:37.066 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:37.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:37.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:34.999155+0000 osd.0 (osd.0) 177 : cluster 0 41.19 scrub starts 2024-04-02T04:51:37.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:35.000528+0000 osd.0 (osd.0) 178 : cluster 0 41.19 scrub ok 2024-04-02T04:51:37.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:35.719747+0000 osd.5 (osd.5) 171 : cluster 0 3.5 scrub starts 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:35.725749+0000 osd.5 (osd.5) 172 : cluster 0 3.5 scrub ok 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: audit 2024-04-02T04:51:35.852440+0000 mon.c (mon.2) 375 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:35.957046+0000 mgr.y (mgr.14538) 319 : cluster 0 pgmap v361: 388 pgs: 32 unknown, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 91 active+undersized, 218 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 272/1557 objects degraded (17.469%) 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.094990+0000 osd.6 (osd.6) 133 : cluster 0 41.7 scrub starts 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.096824+0000 osd.6 (osd.6) 134 : cluster 0 41.7 scrub ok 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.395208+0000 osd.4 (osd.4) 179 : cluster 0 10.17 scrub starts 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.397206+0000 osd.4 (osd.4) 180 : cluster 0 10.17 scrub ok 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.551711+0000 osd.3 (osd.3) 209 : cluster 0 43.4 scrub starts 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.554148+0000 osd.3 (osd.3) 210 : cluster 0 43.4 scrub ok 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.689702+0000 mon.a (mon.0) 1804 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: audit 2024-04-02T04:51:36.691376+0000 mon.a (mon.0) 1805 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]': finished 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: audit 2024-04-02T04:51:36.691461+0000 mon.a (mon.0) 1806 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.695703+0000 mon.a (mon.0) 1807 : cluster 0 osdmap e257: 8 total, 7 up, 8 in 2024-04-02T04:51:37.306 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: audit 2024-04-02T04:51:36.703573+0000 mon.c (mon.2) 376 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:37.307 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:37 smithi098 bash[23790]: audit 2024-04-02T04:51:36.704280+0000 mon.a (mon.0) 1808 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:37.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:34.999155+0000 osd.0 (osd.0) 177 : cluster 0 41.19 scrub starts 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:35.000528+0000 osd.0 (osd.0) 178 : cluster 0 41.19 scrub ok 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:35.719747+0000 osd.5 (osd.5) 171 : cluster 0 3.5 scrub starts 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:35.725749+0000 osd.5 (osd.5) 172 : cluster 0 3.5 scrub ok 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: audit 2024-04-02T04:51:35.852440+0000 mon.c (mon.2) 375 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:35.957046+0000 mgr.y (mgr.14538) 319 : cluster 0 pgmap v361: 388 pgs: 32 unknown, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 91 active+undersized, 218 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 272/1557 objects degraded (17.469%) 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.094990+0000 osd.6 (osd.6) 133 : cluster 0 41.7 scrub starts 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.096824+0000 osd.6 (osd.6) 134 : cluster 0 41.7 scrub ok 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.395208+0000 osd.4 (osd.4) 179 : cluster 0 10.17 scrub starts 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.397206+0000 osd.4 (osd.4) 180 : cluster 0 10.17 scrub ok 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.551711+0000 osd.3 (osd.3) 209 : cluster 0 43.4 scrub starts 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.554148+0000 osd.3 (osd.3) 210 : cluster 0 43.4 scrub ok 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.689702+0000 mon.a (mon.0) 1804 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: audit 2024-04-02T04:51:36.691376+0000 mon.a (mon.0) 1805 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]': finished 2024-04-02T04:51:37.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: audit 2024-04-02T04:51:36.691461+0000 mon.a (mon.0) 1806 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.695703+0000 mon.a (mon.0) 1807 : cluster 0 osdmap e257: 8 total, 7 up, 8 in 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: audit 2024-04-02T04:51:36.703573+0000 mon.c (mon.2) 376 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[28904]: audit 2024-04-02T04:51:36.704280+0000 mon.a (mon.0) 1808 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:34.999155+0000 osd.0 (osd.0) 177 : cluster 0 41.19 scrub starts 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:35.000528+0000 osd.0 (osd.0) 178 : cluster 0 41.19 scrub ok 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:35.719747+0000 osd.5 (osd.5) 171 : cluster 0 3.5 scrub starts 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:35.725749+0000 osd.5 (osd.5) 172 : cluster 0 3.5 scrub ok 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: audit 2024-04-02T04:51:35.852440+0000 mon.c (mon.2) 375 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:35.957046+0000 mgr.y (mgr.14538) 319 : cluster 0 pgmap v361: 388 pgs: 32 unknown, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 91 active+undersized, 218 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 272/1557 objects degraded (17.469%) 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.094990+0000 osd.6 (osd.6) 133 : cluster 0 41.7 scrub starts 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.096824+0000 osd.6 (osd.6) 134 : cluster 0 41.7 scrub ok 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.395208+0000 osd.4 (osd.4) 179 : cluster 0 10.17 scrub starts 2024-04-02T04:51:37.436 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.397206+0000 osd.4 (osd.4) 180 : cluster 0 10.17 scrub ok 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.551711+0000 osd.3 (osd.3) 209 : cluster 0 43.4 scrub starts 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.554148+0000 osd.3 (osd.3) 210 : cluster 0 43.4 scrub ok 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.689702+0000 mon.a (mon.0) 1804 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: audit 2024-04-02T04:51:36.691376+0000 mon.a (mon.0) 1805 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-19"}]': finished 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: audit 2024-04-02T04:51:36.691461+0000 mon.a (mon.0) 1806 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.695703+0000 mon.a (mon.0) 1807 : cluster 0 osdmap e257: 8 total, 7 up, 8 in 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: audit 2024-04-02T04:51:36.703573+0000 mon.c (mon.2) 376 : audit 1 from='client.? 172.21.15.69:0/1657029577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:37.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:37 smithi069 bash[21528]: audit 2024-04-02T04:51:36.704280+0000 mon.a (mon.0) 1808 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]: dispatch 2024-04-02T04:51:37.450 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:37.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:37.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:37.719 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:37.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:37.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- 172.21.15.69:0/829167189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81024b0 msgr2=0x7f68d8105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- 172.21.15.69:0/829167189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81024b0 0x7f68d8105fe0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0099c0 tx=0x7f68cc009cd0 comp rx=0 tx=0).stop 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- 172.21.15.69:0/829167189 shutdown_connections 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- 172.21.15.69:0/829167189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68d81066c0 0x7f68d810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- 172.21.15.69:0/829167189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68d8101b00 0x7f68d8101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- 172.21.15.69:0/829167189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81024b0 0x7f68d8105fe0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:37.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- 172.21.15.69:0/829167189 >> 172.21.15.69:0/829167189 conn(0x7f68d8077740 msgr2=0x7f68d80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:37.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- 172.21.15.69:0/829167189 shutdown_connections 2024-04-02T04:51:37.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- 172.21.15.69:0/829167189 wait complete. 2024-04-02T04:51:37.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 librados: starting msgr at 2024-04-02T04:51:37.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 librados: starting objecter 2024-04-02T04:51:37.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 Processor -- start 2024-04-02T04:51:37.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- start start 2024-04-02T04:51:37.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 librados: setting wanted keys 2024-04-02T04:51:37.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 librados: calling monclient init 2024-04-02T04:51:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68d8101b00 0x7f68d8197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68d81024b0 0x7f68d8197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81066c0 0x7f68d819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68d77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81066c0 0x7f68d819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68d77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81066c0 0x7f68d819bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34008/0 (socket says 172.21.15.69:34008) 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68d77fe700 1 -- 172.21.15.69:0/2619647443 learned_addr learned my addr 172.21.15.69:0/2619647443 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68d81024b0 0x7f68d8197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.927+0000 7f68dd4fc700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8110bc0 con 0x7f68d81024b0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68dd4fc700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8110a40 con 0x7f68d8101b00 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68dd4fc700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f68d8110d40 con 0x7f68d81066c0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68d8101b00 0x7f68d8197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d77fe700 1 -- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68d8101b00 msgr2=0x7f68d8197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68d8101b00 0x7f68d8197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d77fe700 1 -- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68d81024b0 msgr2=0x7f68d8197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68d81024b0 0x7f68d8197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d77fe700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d819c210 con 0x7f68d81066c0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68d81066c0 0x7f68d819bad0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f68d0007ee0 tx=0x7f68d000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68bffff700 10 client.?.objecter ms_handle_connect 0x7f68d81066c0 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68bffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d000ddd0 con 0x7f68d81066c0 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f68d0004020 con 0x7f68d81066c0 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0015660 con 0x7f68d81066c0 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68dd4fc700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d819c500 con 0x7f68d81066c0 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68dd4fc700 10 client.26117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68dd4fc700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d819c900 con 0x7f68d81066c0 2024-04-02T04:51:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68dd4fc700 1 librados: init done 2024-04-02T04:51:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.931+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f68d0003a00 con 0x7f68d81066c0 2024-04-02T04:51:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f68c006f5e0 0x7f68c0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(258..258 src has 74..258) ==== 10837+0+0 (secure 0 0 0) 0x7f68d008e8a0 con 0x7f68d81066c0 2024-04-02T04:51:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 osd_map(258..258 src has 74..258) 2024-04-02T04:51:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 3 client.26117.objecter handle_osd_map got epochs [258,258] > 0 2024-04-02T04:51:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 3 client.26117.objecter handle_osd_map decoding full epoch 258 2024-04-02T04:51:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 20 client.26117.objecter dump_active .. 0 homeless 2024-04-02T04:51:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f68d8102b50 con 0x7f68d81066c0 2024-04-02T04:51:37.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v258) ==== 54+0+16 (secure 0 0 0) 0x7f68d819c900 con 0x7f68d81066c0 2024-04-02T04:51:37.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 10 client.26117.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:37.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 1 2024-04-02T04:51:37.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f68d8108240 0x7f68d8108620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 1 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 0 1 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8198560 osd=0 4 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 10 client.26117.objecter _send_command 1 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68d804ff00 con 0x7f68d8108240 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f68c006f5e0 0x7f68c0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f68d8108240 0x7f68d8108620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d8108240 2024-04-02T04:51:37.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f68d8108240 0x7f68d8108620 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:37.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d804ff00 con 0x7f68d8108240 2024-04-02T04:51:37.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 1: 0 ) 2024-04-02T04:51:37.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 10 client.26117.objecter _finish_command 1 = system:0 2024-04-02T04:51:37.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d8198560 osd=0 4 2024-04-02T04:51:37.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.935+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 0 1 2024-04-02T04:51:37.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.939+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f68c006f5e0 0x7f68c0071aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f68c8006fd0 tx=0x7f68c8008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:37.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.939+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68c006f5e0 2024-04-02T04:51:37.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 10 client.26117.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 2 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 2 2024-04-02T04:51:37.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d8198560 osd=0 3 2024-04-02T04:51:37.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 0 2 2024-04-02T04:51:37.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8198560 osd=0 4 2024-04-02T04:51:37.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 10 client.26117.objecter _send_command 2 2024-04-02T04:51:37.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d8069d50 con 0x7f68d8108240 2024-04-02T04:51:37.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d8069d50 con 0x7f68d8108240 2024-04-02T04:51:37.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:37.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bffff700 10 client.26117.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:37.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d8198560 osd=0 4 2024-04-02T04:51:37.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 0 2 2024-04-02T04:51:37.955 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 10 client.26117.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f68d81a62a0 0x7f68d81a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 1 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8100f40 osd=1 4 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 10 client.26117.objecter _send_command 3 2024-04-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f68d81010b0 con 0x7f68d81a62a0 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.955+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f68d81a62a0 0x7f68d81a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f68d81a62a0 0x7f68d81a8690 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d81a62a0 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68c8013420 con 0x7f68d81a62a0 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 3: 0 ) 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68bffff700 10 client.26117.objecter _finish_command 3 = system:0 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d8100f40 osd=1 4 2024-04-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.959+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 1 3 2024-04-02T04:51:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:37.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 10 client.26117.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 4 2024-04-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 4 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d8100f40 osd=1 3 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 1 4 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8100f40 osd=1 4 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 10 client.26117.objecter _send_command 4 2024-04-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.975+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d8108db0 con 0x7f68d81a62a0 2024-04-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d8108db0 con 0x7f68d81a62a0 2024-04-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bffff700 10 client.26117.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d8100f40 osd=1 4 2024-04-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 1 4 2024-04-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 10 client.26117.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 5 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f68d81a9d50 0x7f68d81aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 5 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 2 5 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8062e50 osd=2 4 2024-04-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 10 client.26117.objecter _send_command 5 2024-04-02T04:51:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f68d81aa6d0 con 0x7f68d81a9d50 2024-04-02T04:51:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f68d81a9d50 0x7f68d81aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f68d81a9d50 0x7f68d81aa190 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:37.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.979+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d81a9d50 2024-04-02T04:51:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.983+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d81aa6d0 con 0x7f68d81a9d50 2024-04-02T04:51:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.983+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 5: 0 ) 2024-04-02T04:51:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.983+0000 7f68bffff700 10 client.26117.objecter _finish_command 5 = system:0 2024-04-02T04:51:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.983+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d8062e50 osd=2 4 2024-04-02T04:51:37.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.983+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 2 5 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 10 client.26117.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 6 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 6 2024-04-02T04:51:37.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d8062e50 osd=2 3 2024-04-02T04:51:37.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 2 6 2024-04-02T04:51:37.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d8062e50 osd=2 4 2024-04-02T04:51:37.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 10 client.26117.objecter _send_command 6 2024-04-02T04:51:37.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:37.999+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d81ab0e0 con 0x7f68d81a9d50 2024-04-02T04:51:38.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d81ab0e0 con 0x7f68d81a9d50 2024-04-02T04:51:38.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:38.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 10 client.26117.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:38.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d8062e50 osd=2 4 2024-04-02T04:51:38.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 2 6 2024-04-02T04:51:38.000 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:51:38.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 10 client.26117.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:38.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 7 2024-04-02T04:51:38.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f68d8062a40 0x7f68d81b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 7 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 3 7 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d80628d0 osd=3 4 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 10 client.26117.objecter _send_command 7 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f68d81b6d60 con 0x7f68d8062a40 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f68d8062a40 0x7f68d81b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d8062a40 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f68d8062a40 0x7f68d81b3e30 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d81b6d60 con 0x7f68d8062a40 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 7: 0 ) 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 10 client.26117.objecter _finish_command 7 = system:0 2024-04-02T04:51:38.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d80628d0 osd=3 4 2024-04-02T04:51:38.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.003+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 3 7 2024-04-02T04:51:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 10 client.26117.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 8 2024-04-02T04:51:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 8 2024-04-02T04:51:38.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d80628d0 osd=3 3 2024-04-02T04:51:38.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 3 8 2024-04-02T04:51:38.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d80628d0 osd=3 4 2024-04-02T04:51:38.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 10 client.26117.objecter _send_command 8 2024-04-02T04:51:38.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d81aabf0 con 0x7f68d8062a40 2024-04-02T04:51:38.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d81aabf0 con 0x7f68d8062a40 2024-04-02T04:51:38.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:38.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bffff700 10 client.26117.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:38.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d80628d0 osd=3 4 2024-04-02T04:51:38.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 3 8 2024-04-02T04:51:38.022 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:51:38.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 10 client.26117.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 9 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f68d81a9900 0x7f68d81b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 9 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 4 9 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81aabf0 osd=4 4 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 10 client.26117.objecter _send_command 9 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f68d81b4a80 con 0x7f68d81a9900 2024-04-02T04:51:38.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f68d81a9900 0x7f68d81b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d81a9900 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.023+0000 7f68d6ffd700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f68d81a9900 0x7f68d81b43a0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.027+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d81b4a80 con 0x7f68d81a9900 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.027+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 9: 0 ) 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.027+0000 7f68bffff700 10 client.26117.objecter _finish_command 9 = system:0 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.027+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d81aabf0 osd=4 4 2024-04-02T04:51:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.027+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 4 9 2024-04-02T04:51:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 10 client.26117.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 10 2024-04-02T04:51:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 10 2024-04-02T04:51:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d81aabf0 osd=4 3 2024-04-02T04:51:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 4 10 2024-04-02T04:51:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81aabf0 osd=4 4 2024-04-02T04:51:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 10 client.26117.objecter _send_command 10 2024-04-02T04:51:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d81b4a80 con 0x7f68d81a9900 2024-04-02T04:51:38.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d81b4a80 con 0x7f68d81a9900 2024-04-02T04:51:38.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:38.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bffff700 10 client.26117.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:38.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d81aabf0 osd=4 4 2024-04-02T04:51:38.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 4 10 2024-04-02T04:51:38.047 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 10 client.26117.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 11 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f68d81b65e0 0x7f68d81bafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 11 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 5 11 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81b4a80 osd=5 4 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 10 client.26117.objecter _send_command 11 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.047+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f68d81bde20 con 0x7f68d81b65e0 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f68d81b65e0 0x7f68d81bafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d81b65e0 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68d67fc700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f68d81b65e0 0x7f68d81bafc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d81bde20 con 0x7f68d81b65e0 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 11: 0 ) 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68bffff700 10 client.26117.objecter _finish_command 11 = system:0 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d81b4a80 osd=5 4 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.051+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 5 11 2024-04-02T04:51:38.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 10 client.26117.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:38.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 12 2024-04-02T04:51:38.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 12 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d81b4a80 osd=5 3 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 5 12 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81b4a80 osd=5 4 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 10 client.26117.objecter _send_command 12 2024-04-02T04:51:38.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.067+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d81bb500 con 0x7f68d81b65e0 2024-04-02T04:51:38.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d81bb500 con 0x7f68d81b65e0 2024-04-02T04:51:38.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:38.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 10 client.26117.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:38.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d81b4a80 osd=5 4 2024-04-02T04:51:38.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 5 12 2024-04-02T04:51:38.072 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 10 client.26117.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 13 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f68d81bbb80 0x7f68d81bdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 13 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 6 13 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81bb500 osd=6 4 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 10 client.26117.objecter _send_command 13 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f68d81c1050 con 0x7f68d81bbb80 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f68d81bbb80 0x7f68d81bdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68d77fe700 1 --2- 172.21.15.69:0/2619647443 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f68d81bbb80 0x7f68d81bdf90 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 10 client.26117.objecter ms_handle_connect 0x7f68d81bbb80 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d81c1050 con 0x7f68d81bbb80 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 13: 0 ) 2024-04-02T04:51:38.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 10 client.26117.objecter _finish_command 13 = system:0 2024-04-02T04:51:38.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d81bb500 osd=6 4 2024-04-02T04:51:38.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.075+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 6 13 2024-04-02T04:51:38.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:38.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 10 client.26117.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:51:38.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 14 2024-04-02T04:51:38.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 20 client.26117.objecter _get_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_remove -1 14 2024-04-02T04:51:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 20 client.26117.objecter get_session s=0x7f68d81bb500 osd=6 3 2024-04-02T04:51:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign 6 14 2024-04-02T04:51:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 20 client.26117.objecter put_session s=0x7f68d81bb500 osd=6 4 2024-04-02T04:51:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 10 client.26117.objecter _send_command 14 2024-04-02T04:51:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f68d81bf790 con 0x7f68d81bbb80 2024-04-02T04:51:38.094 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:38.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f68d81bf790 con 0x7f68d81bbb80 2024-04-02T04:51:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bffff700 10 client.26117.objecter ms_dispatch 0x7f68d8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bffff700 10 client.26117.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bffff700 20 client.26117.objecter put_session s=0x7f68d81bb500 osd=6 4 2024-04-02T04:51:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bffff700 15 client.26117.objecter _session_command_op_remove 6 14 2024-04-02T04:51:38.095 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:51:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 10 client.26117.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 15 client.26117.objecter _session_command_op_assign -1 15 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 20 client.26117.objecter _get_session osd=-1 returning homeless 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 10 client.26117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=259}) -- 0x7f68d81bf9a0 con 0x7f68d81066c0 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bdffb700 1 -- 172.21.15.69:0/2619647443 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f68d81bf790 con 0x7f68d81066c0 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68bffff700 1 -- 172.21.15.69:0/2619647443 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_get_version_reply(handle=1 version=258) ==== 24+0+0 (secure 0 0 0) 0x7f68d005be60 con 0x7f68d81066c0 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68d57fa700 10 client.26117.objecter _check_command_map_dne tid 15 current 258 map_dne_bound 258 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68d57fa700 10 client.26117.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.095+0000 7f68d57fa700 15 client.26117.objecter _session_command_op_remove -1 15 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:51:38.096 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:51:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:51:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:51:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:51:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:51:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:51:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/1078724705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60107220 msgr2=0x7fcf60107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- 172.21.15.69:0/1078724705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60107220 0x7fcf60107600 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcf500098f0 tx=0x7fcf50009c00 comp rx=0 tx=0).stop 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/1078724705 shutdown_connections 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- 172.21.15.69:0/1078724705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf60101b60 0x7fcf6010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- 172.21.15.69:0/1078724705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60107220 0x7fcf60107600 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- 172.21.15.69:0/1078724705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf601011a0 0x7fcf60101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/1078724705 >> 172.21.15.69:0/1078724705 conn(0x7fcf600771e0 msgr2=0x7fcf600775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:38.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/1078724705 shutdown_connections 2024-04-02T04:51:38.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/1078724705 wait complete. 2024-04-02T04:51:38.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 librados: starting msgr at 2024-04-02T04:51:38.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 librados: starting objecter 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 Processor -- start 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- start start 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 librados: setting wanted keys 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 librados: calling monclient init 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf601011a0 0x7fcf60198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60101b60 0x7fcf601993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 0x7fcf601a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcf60110960 con 0x7fcf60101b60 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf601107e0 con 0x7fcf601011a0 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcf60110ae0 con 0x7fcf60107220 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60101b60 0x7fcf601993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 0x7fcf601a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60101b60 0x7fcf601993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34128/0 (socket says 172.21.15.69:34128) 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 0x7fcf601a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34034/0 (socket says 172.21.15.69:34034) 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5f7fe700 1 -- 172.21.15.69:0/4229702324 learned_addr learned my addr 172.21.15.69:0/4229702324 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5ffff700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf601011a0 0x7fcf60198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 -- 172.21.15.69:0/4229702324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf601011a0 msgr2=0x7fcf60198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf601011a0 0x7fcf60198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 -- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60101b60 msgr2=0x7fcf601993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60101b60 0x7fcf601993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 -- 172.21.15.69:0/4229702324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf601a4420 con 0x7fcf60107220 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf64edc700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 0x7fcf601a3ee0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800cc50 tx=0x7fcf5800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5d7fa700 10 client.?.objecter ms_handle_connect 0x7fcf60107220 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5d7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58016020 con 0x7fcf60107220 2024-04-02T04:51:38.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fcf58012420 con 0x7fcf60107220 2024-04-02T04:51:38.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf601a4650 con 0x7fcf60107220 2024-04-02T04:51:38.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf580115e0 con 0x7fcf60107220 2024-04-02T04:51:38.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 10 client.26123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:38.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf601a4b40 con 0x7fcf60107220 2024-04-02T04:51:38.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 librados: init done 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.167+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6004ff00 con 0x7fcf60107220 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fcf58019020 con 0x7fcf60107220 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf4806f5e0 0x7fcf48071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(258..258 src has 74..258) ==== 10837+0+0 (secure 0 0 0) 0x7fcf5808eb40 con 0x7fcf60107220 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 10 client.26123.objecter ms_dispatch 0x7fcf60001550 osd_map(258..258 src has 74..258) 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 3 client.26123.objecter handle_osd_map got epochs [258,258] > 0 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 3 client.26123.objecter handle_osd_map decoding full epoch 258 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5ffff700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf4806f5e0 0x7fcf48071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 20 client.26123.objecter dump_active .. 0 homeless 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5d7fa700 10 client.26123.objecter ms_handle_connect 0x7fcf4806f5e0 2024-04-02T04:51:38.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.171+0000 7fcf5ffff700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf4806f5e0 0x7fcf48071aa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcf50005e90 tx=0x7fcf50019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:38.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.175+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf58020020 con 0x7fcf60107220 2024-04-02T04:51:38.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:38.217 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:38.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.704414+0000 osd.5 (osd.5) 173 : cluster 0 43.9 scrub starts 2024-04-02T04:51:38.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:36.711170+0000 osd.5 (osd.5) 174 : cluster 0 43.9 scrub ok 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:36.853730+0000 mon.c (mon.2) 377 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.174795+0000 mon.a (mon.0) 1809 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.219869+0000 osd.2 (osd.2) 127 : cluster 0 4.f scrub starts 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.220981+0000 osd.2 (osd.2) 128 : cluster 0 4.f scrub ok 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.348678+0000 osd.4 (osd.4) 181 : cluster 0 13.5 scrub starts 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.350145+0000 osd.4 (osd.4) 182 : cluster 0 13.5 scrub ok 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:37.695309+0000 mon.a (mon.0) 1810 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.699751+0000 mon.a (mon.0) 1811 : cluster 0 osdmap e258: 8 total, 7 up, 8 in 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:37.705820+0000 mon.a (mon.0) 1812 : audit 1 from='client.? 172.21.15.69:0/3435268291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-72153-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:37.753748+0000 mon.b (mon.1) 180 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:37.855003+0000 mon.c (mon.2) 378 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:37.936729+0000 mon.c (mon.2) 379 : audit 0 from='client.? 172.21.15.69:0/2619647443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:38.162603+0000 mon.b (mon.1) 181 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:38.163928+0000 mon.b (mon.1) 182 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:38.165834+0000 mon.a (mon.0) 1813 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:38.167350+0000 mon.a (mon.0) 1814 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:38 smithi098 bash[23790]: audit 2024-04-02T04:51:38.168096+0000 mon.a (mon.0) 1815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:51:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:51:38.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:38.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.391+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fcf601a4e00 con 0x7fcf60107220 2024-04-02T04:51:38.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.704414+0000 osd.5 (osd.5) 173 : cluster 0 43.9 scrub starts 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:36.711170+0000 osd.5 (osd.5) 174 : cluster 0 43.9 scrub ok 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:36.853730+0000 mon.c (mon.2) 377 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.174795+0000 mon.a (mon.0) 1809 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.219869+0000 osd.2 (osd.2) 127 : cluster 0 4.f scrub starts 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.220981+0000 osd.2 (osd.2) 128 : cluster 0 4.f scrub ok 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.348678+0000 osd.4 (osd.4) 181 : cluster 0 13.5 scrub starts 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.350145+0000 osd.4 (osd.4) 182 : cluster 0 13.5 scrub ok 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:37.695309+0000 mon.a (mon.0) 1810 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.699751+0000 mon.a (mon.0) 1811 : cluster 0 osdmap e258: 8 total, 7 up, 8 in 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:37.705820+0000 mon.a (mon.0) 1812 : audit 1 from='client.? 172.21.15.69:0/3435268291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-72153-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:38.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:37.753748+0000 mon.b (mon.1) 180 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:37.855003+0000 mon.c (mon.2) 378 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:37.936729+0000 mon.c (mon.2) 379 : audit 0 from='client.? 172.21.15.69:0/2619647443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:38.162603+0000 mon.b (mon.1) 181 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:38.163928+0000 mon.b (mon.1) 182 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:38.165834+0000 mon.a (mon.0) 1813 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:38.167350+0000 mon.a (mon.0) 1814 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21528]: audit 2024-04-02T04:51:38.168096+0000 mon.a (mon.0) 1815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.704414+0000 osd.5 (osd.5) 173 : cluster 0 43.9 scrub starts 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:36.711170+0000 osd.5 (osd.5) 174 : cluster 0 43.9 scrub ok 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:36.853730+0000 mon.c (mon.2) 377 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.174795+0000 mon.a (mon.0) 1809 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:38.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.219869+0000 osd.2 (osd.2) 127 : cluster 0 4.f scrub starts 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.220981+0000 osd.2 (osd.2) 128 : cluster 0 4.f scrub ok 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.348678+0000 osd.4 (osd.4) 181 : cluster 0 13.5 scrub starts 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.350145+0000 osd.4 (osd.4) 182 : cluster 0 13.5 scrub ok 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:37.695309+0000 mon.a (mon.0) 1810 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-72145-30"}]': finished 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.699751+0000 mon.a (mon.0) 1811 : cluster 0 osdmap e258: 8 total, 7 up, 8 in 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:37.705820+0000 mon.a (mon.0) 1812 : audit 1 from='client.? 172.21.15.69:0/3435268291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-72153-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:37.753748+0000 mon.b (mon.1) 180 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:37.855003+0000 mon.c (mon.2) 378 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:37.936729+0000 mon.c (mon.2) 379 : audit 0 from='client.? 172.21.15.69:0/2619647443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:38.162603+0000 mon.b (mon.1) 181 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:38.163928+0000 mon.b (mon.1) 182 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:38.165834+0000 mon.a (mon.0) 1813 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:38.167350+0000 mon.a (mon.0) 1814 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:38.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[28904]: audit 2024-04-02T04:51:38.168096+0000 mon.a (mon.0) 1815 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:38.461 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:38.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:38.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:38.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.723+0000 7fcf5d7fa700 1 -- 172.21.15.69:0/4229702324 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v259) ==== 101+0+0 (secure 0 0 0) 0x7fcf5805b3f0 con 0x7fcf60107220 2024-04-02T04:51:38.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 10 librados: watch_flush enter 2024-04-02T04:51:38.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 10 librados: watch_flush exit 2024-04-02T04:51:38.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 20 client.26123.objecter shutdown clearing up homeless session... 2024-04-02T04:51:38.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 10 client.26123.objecter successfully canceled tick 2024-04-02T04:51:38.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf4806f5e0 msgr2=0x7fcf48071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:38.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf4806f5e0 0x7fcf48071aa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcf50005e90 tx=0x7fcf50019040 comp rx=0 tx=0).stop 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 msgr2=0x7fcf601a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 0x7fcf601a3ee0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800cc50 tx=0x7fcf5800cf60 comp rx=0 tx=0).stop 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 shutdown_connections 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf4806f5e0 0x7fcf48071aa0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf60107220 0x7fcf601a3ee0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf601011a0 0x7fcf60198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 --2- 172.21.15.69:0/4229702324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf60101b60 0x7fcf601993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:38.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.735+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 >> 172.21.15.69:0/4229702324 conn(0x7fcf600771e0 msgr2=0x7fcf60107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:38.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.739+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 shutdown_connections 2024-04-02T04:51:38.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.739+0000 7fcf6693f700 1 -- 172.21.15.69:0/4229702324 wait complete. 2024-04-02T04:51:38.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:38.739+0000 7fcf6693f700 1 librados: shutdown 2024-04-02T04:51:38.738 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:38.742 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:51:38.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:38.868 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:38.971 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:38.994 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:51:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:51:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:51:38.997 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:38.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:39.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-02T04:51:39.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:39.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:39.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:39.253 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: cluster 2024-04-02T04:51:37.958736+0000 mgr.y (mgr.14538) 320 : cluster 0 pgmap v364: 356 pgs: 32 creating+peering, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: cluster 2024-04-02T04:51:38.236002+0000 osd.1 (osd.1) 193 : cluster 0 13.11 scrub starts 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: cluster 2024-04-02T04:51:38.237545+0000 osd.1 (osd.1) 194 : cluster 0 13.11 scrub ok 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.392196+0000 mon.c (mon.2) 380 : audit 1 from='client.? 172.21.15.69:0/4229702324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.392598+0000 mon.a (mon.0) 1816 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: cluster 2024-04-02T04:51:38.697473+0000 mon.a (mon.0) 1817 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.705860+0000 mon.a (mon.0) 1818 : audit 1 from='client.? 172.21.15.69:0/3435268291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-72153-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.706069+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.706283+0000 mon.a (mon.0) 1820 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:51:39.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.712929+0000 mon.b (mon.1) 183 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: cluster 2024-04-02T04:51:38.726697+0000 mon.a (mon.0) 1821 : cluster 0 osdmap e259: 8 total, 7 up, 8 in 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.732392+0000 mon.c (mon.2) 381 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.736793+0000 mon.a (mon.0) 1822 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.737284+0000 mon.a (mon.0) 1823 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[28904]: audit 2024-04-02T04:51:38.856076+0000 mon.c (mon.2) 382 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: cluster 2024-04-02T04:51:37.958736+0000 mgr.y (mgr.14538) 320 : cluster 0 pgmap v364: 356 pgs: 32 creating+peering, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: cluster 2024-04-02T04:51:38.236002+0000 osd.1 (osd.1) 193 : cluster 0 13.11 scrub starts 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: cluster 2024-04-02T04:51:38.237545+0000 osd.1 (osd.1) 194 : cluster 0 13.11 scrub ok 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.392196+0000 mon.c (mon.2) 380 : audit 1 from='client.? 172.21.15.69:0/4229702324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:39.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.392598+0000 mon.a (mon.0) 1816 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: cluster 2024-04-02T04:51:38.697473+0000 mon.a (mon.0) 1817 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.705860+0000 mon.a (mon.0) 1818 : audit 1 from='client.? 172.21.15.69:0/3435268291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-72153-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.706069+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.706283+0000 mon.a (mon.0) 1820 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.712929+0000 mon.b (mon.1) 183 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: cluster 2024-04-02T04:51:38.726697+0000 mon.a (mon.0) 1821 : cluster 0 osdmap e259: 8 total, 7 up, 8 in 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.732392+0000 mon.c (mon.2) 381 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.736793+0000 mon.a (mon.0) 1822 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.737284+0000 mon.a (mon.0) 1823 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:39.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:39 smithi069 bash[21528]: audit 2024-04-02T04:51:38.856076+0000 mon.c (mon.2) 382 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:39.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:39.385 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: cluster 2024-04-02T04:51:37.958736+0000 mgr.y (mgr.14538) 320 : cluster 0 pgmap v364: 356 pgs: 32 creating+peering, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: cluster 2024-04-02T04:51:38.236002+0000 osd.1 (osd.1) 193 : cluster 0 13.11 scrub starts 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: cluster 2024-04-02T04:51:38.237545+0000 osd.1 (osd.1) 194 : cluster 0 13.11 scrub ok 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.392196+0000 mon.c (mon.2) 380 : audit 1 from='client.? 172.21.15.69:0/4229702324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.392598+0000 mon.a (mon.0) 1816 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: cluster 2024-04-02T04:51:38.697473+0000 mon.a (mon.0) 1817 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.705860+0000 mon.a (mon.0) 1818 : audit 1 from='client.? 172.21.15.69:0/3435268291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-72153-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.706069+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-72145-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.706283+0000 mon.a (mon.0) 1820 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:51:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.712929+0000 mon.b (mon.1) 183 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: cluster 2024-04-02T04:51:38.726697+0000 mon.a (mon.0) 1821 : cluster 0 osdmap e259: 8 total, 7 up, 8 in 2024-04-02T04:51:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.732392+0000 mon.c (mon.2) 381 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.736793+0000 mon.a (mon.0) 1822 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.737284+0000 mon.a (mon.0) 1823 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:39 smithi098 bash[23790]: audit 2024-04-02T04:51:38.856076+0000 mon.c (mon.2) 382 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:39.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:39.510 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:39.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:39.643 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:39.746 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:39.788 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:39.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:39.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:40.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:40.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:40.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.166 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:40.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:40.300 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:40.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.418 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: cluster 2024-04-02T04:51:38.660504+0000 osd.5 (osd.5) 175 : cluster 0 13.2 scrub starts 2024-04-02T04:51:40.418 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: cluster 2024-04-02T04:51:38.662070+0000 osd.5 (osd.5) 176 : cluster 0 13.2 scrub ok 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: cluster 2024-04-02T04:51:39.046546+0000 osd.6 (osd.6) 135 : cluster 0 10.1a deep-scrub starts 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: cluster 2024-04-02T04:51:39.048977+0000 osd.6 (osd.6) 136 : cluster 0 10.1a deep-scrub ok 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: cluster 2024-04-02T04:51:39.684403+0000 mon.a (mon.0) 1824 : cluster 3 Health check update: Degraded data redundancy: 272/1533 objects degraded (17.743%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: audit 2024-04-02T04:51:39.710951+0000 mon.a (mon.0) 1825 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: audit 2024-04-02T04:51:39.795278+0000 mon.c (mon.2) 383 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: audit 2024-04-02T04:51:39.856947+0000 mon.c (mon.2) 384 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.184190+0000 mon.a (mon.0) 1826 : cluster 0 osdmap e260: 8 total, 7 up, 8 in 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[28904]: audit 2024-04-02T04:51:40.185563+0000 mon.a (mon.0) 1827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: cluster 2024-04-02T04:51:38.660504+0000 osd.5 (osd.5) 175 : cluster 0 13.2 scrub starts 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: cluster 2024-04-02T04:51:38.662070+0000 osd.5 (osd.5) 176 : cluster 0 13.2 scrub ok 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: cluster 2024-04-02T04:51:39.046546+0000 osd.6 (osd.6) 135 : cluster 0 10.1a deep-scrub starts 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: cluster 2024-04-02T04:51:39.048977+0000 osd.6 (osd.6) 136 : cluster 0 10.1a deep-scrub ok 2024-04-02T04:51:40.419 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: cluster 2024-04-02T04:51:39.684403+0000 mon.a (mon.0) 1824 : cluster 3 Health check update: Degraded data redundancy: 272/1533 objects degraded (17.743%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:40.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: audit 2024-04-02T04:51:39.710951+0000 mon.a (mon.0) 1825 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:40.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: audit 2024-04-02T04:51:39.795278+0000 mon.c (mon.2) 383 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:40.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: audit 2024-04-02T04:51:39.856947+0000 mon.c (mon.2) 384 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:40.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.184190+0000 mon.a (mon.0) 1826 : cluster 0 osdmap e260: 8 total, 7 up, 8 in 2024-04-02T04:51:40.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:40 smithi069 bash[21528]: audit 2024-04-02T04:51:40.185563+0000 mon.a (mon.0) 1827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:40.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: cluster 2024-04-02T04:51:38.660504+0000 osd.5 (osd.5) 175 : cluster 0 13.2 scrub starts 2024-04-02T04:51:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: cluster 2024-04-02T04:51:38.662070+0000 osd.5 (osd.5) 176 : cluster 0 13.2 scrub ok 2024-04-02T04:51:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: cluster 2024-04-02T04:51:39.046546+0000 osd.6 (osd.6) 135 : cluster 0 10.1a deep-scrub starts 2024-04-02T04:51:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: cluster 2024-04-02T04:51:39.048977+0000 osd.6 (osd.6) 136 : cluster 0 10.1a deep-scrub ok 2024-04-02T04:51:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: cluster 2024-04-02T04:51:39.684403+0000 mon.a (mon.0) 1824 : cluster 3 Health check update: Degraded data redundancy: 272/1533 objects degraded (17.743%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: audit 2024-04-02T04:51:39.710951+0000 mon.a (mon.0) 1825 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: audit 2024-04-02T04:51:39.795278+0000 mon.c (mon.2) 383 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: audit 2024-04-02T04:51:39.856947+0000 mon.c (mon.2) 384 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.184190+0000 mon.a (mon.0) 1826 : cluster 0 osdmap e260: 8 total, 7 up, 8 in 2024-04-02T04:51:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:40 smithi098 bash[23790]: audit 2024-04-02T04:51:40.185563+0000 mon.a (mon.0) 1827 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:40.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:40.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:40.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:51:40.713 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:51:40.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.799 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:40.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.923 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:40.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:41.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:41.052 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAid 0x55c99fcab770 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab880 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab990 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac090 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac1e0 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac310 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac480 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb4e10 2024-04-02T04:51:41.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcabbe0 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcabd30 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcabe40 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcabf70 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb53d0 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb4f20 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5070 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5180 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5290 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb59b0 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5500 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5650 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5780 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5ae0 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5fe0 2024-04-02T04:51:41.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb60f0 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb66d0 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5bf0 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5d20 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb5e50 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6820 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6220 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab170 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab2c0 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcab3f0 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6e00 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6950 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6330 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6460 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6590 2024-04-02T04:51:41.148 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7400 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6f50 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb70a0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb71b0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb72c0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7a00 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7550 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb76a0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb77b0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7b50 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7ca0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb78e0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8190 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7df0 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb7f00 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8050 2024-04-02T04:51:41.149 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8750 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb82a0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6a60 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6b90 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb6cc0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8d50 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb88a0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb89d0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8ae0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8c30 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9330 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8e80 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8fd0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb90e0 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9460 2024-04-02T04:51:41.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9950 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9ab0 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba090 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb95b0 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9700 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9830 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba1c0 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9be0 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9d30 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb9e60 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba2f0 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba7c0 2024-04-02T04:51:41.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba8f0 2024-04-02T04:51:41.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:41.179 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:41.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:41.309 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:41.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:39.642212+0000 osd.5 (osd.5) 177 : cluster 0 41.14 scrub starts 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:39.643938+0000 osd.5 (osd.5) 178 : cluster 0 41.14 scrub ok 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:39.959906+0000 mgr.y (mgr.14538) 321 : cluster 0 pgmap v367: 356 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.066084+0000 osd.6 (osd.6) 137 : cluster 0 2.11 scrub starts 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.072252+0000 osd.6 (osd.6) 138 : cluster 0 2.11 scrub ok 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.384389+0000 osd.4 (osd.4) 183 : cluster 0 4.1 deep-scrub starts 2024-04-02T04:51:41.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.385752+0000 osd.4 (osd.4) 184 : cluster 0 4.1 deep-scrub ok 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.719373+0000 mon.a (mon.0) 1828 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.719450+0000 mon.a (mon.0) 1829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.722900+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e261: 8 total, 7 up, 8 in 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.724312+0000 mon.c (mon.2) 385 : audit 1 from='client.? 172.21.15.69:0/3226152626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.725053+0000 mon.a (mon.0) 1831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.732068+0000 mon.c (mon.2) 386 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.733138+0000 mon.a (mon.0) 1832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[21528]: audit 2024-04-02T04:51:40.858209+0000 mon.c (mon.2) 387 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:39.642212+0000 osd.5 (osd.5) 177 : cluster 0 41.14 scrub starts 2024-04-02T04:51:41.425 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:39.643938+0000 osd.5 (osd.5) 178 : cluster 0 41.14 scrub ok 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:39.959906+0000 mgr.y (mgr.14538) 321 : cluster 0 pgmap v367: 356 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.066084+0000 osd.6 (osd.6) 137 : cluster 0 2.11 scrub starts 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.072252+0000 osd.6 (osd.6) 138 : cluster 0 2.11 scrub ok 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.384389+0000 osd.4 (osd.4) 183 : cluster 0 4.1 deep-scrub starts 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.385752+0000 osd.4 (osd.4) 184 : cluster 0 4.1 deep-scrub ok 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.719373+0000 mon.a (mon.0) 1828 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.719450+0000 mon.a (mon.0) 1829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.722900+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e261: 8 total, 7 up, 8 in 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.724312+0000 mon.c (mon.2) 385 : audit 1 from='client.? 172.21.15.69:0/3226152626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.725053+0000 mon.a (mon.0) 1831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.732068+0000 mon.c (mon.2) 386 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.733138+0000 mon.a (mon.0) 1832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:41.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:41 smithi069 bash[28904]: audit 2024-04-02T04:51:40.858209+0000 mon.c (mon.2) 387 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:41.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:39.642212+0000 osd.5 (osd.5) 177 : cluster 0 41.14 scrub starts 2024-04-02T04:51:41.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:39.643938+0000 osd.5 (osd.5) 178 : cluster 0 41.14 scrub ok 2024-04-02T04:51:41.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:39.959906+0000 mgr.y (mgr.14538) 321 : cluster 0 pgmap v367: 356 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:41.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.066084+0000 osd.6 (osd.6) 137 : cluster 0 2.11 scrub starts 2024-04-02T04:51:41.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.072252+0000 osd.6 (osd.6) 138 : cluster 0 2.11 scrub ok 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.384389+0000 osd.4 (osd.4) 183 : cluster 0 4.1 deep-scrub starts 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.385752+0000 osd.4 (osd.4) 184 : cluster 0 4.1 deep-scrub ok 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.719373+0000 mon.a (mon.0) 1828 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-72145-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.719450+0000 mon.a (mon.0) 1829 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.722900+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e261: 8 total, 7 up, 8 in 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.724312+0000 mon.c (mon.2) 385 : audit 1 from='client.? 172.21.15.69:0/3226152626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.725053+0000 mon.a (mon.0) 1831 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.732068+0000 mon.c (mon.2) 386 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.733138+0000 mon.a (mon.0) 1832 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:41 smithi098 bash[23790]: audit 2024-04-02T04:51:40.858209+0000 mon.c (mon.2) 387 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:41.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:41.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:41.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:41.697 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:41.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:41.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:41.950 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:41.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/3418919466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 msgr2=0x7faa0c10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- 172.21.15.69:0/3418919466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c10ec50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009990 tx=0x7fa9fc009ca0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/3418919466 shutdown_connections 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- 172.21.15.69:0/3418919466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- 172.21.15.69:0/3418919466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c10ec50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- 172.21.15.69:0/3418919466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa0c108870 0x7faa0c108cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/3418919466 >> 172.21.15.69:0/3418919466 conn(0x7faa0c075a10 msgr2=0x7faa0c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/3418919466 shutdown_connections 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/3418919466 wait complete. 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 librados: starting msgr at 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 librados: starting objecter 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 Processor -- start 2024-04-02T04:51:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- start start 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 librados: setting wanted keys 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 librados: calling monclient init 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa0c108870 0x7faa0c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c1a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1180e0 con 0x7faa0c109230 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c117f60 con 0x7faa0c108870 2024-04-02T04:51:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faa0c118260 con 0x7faa0c10e870 2024-04-02T04:51:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1132c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c1a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1132c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34154/0 (socket says 172.21.15.69:34154) 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c1a2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34070/0 (socket says 172.21.15.69:34070) 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1132c700 1 -- 172.21.15.69:0/331980160 learned_addr learned my addr 172.21.15.69:0/331980160 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa11b2d700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa0c108870 0x7faa0c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 -- 172.21.15.69:0/331980160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa0c108870 msgr2=0x7faa0c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa0c108870 0x7faa0c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 -- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 msgr2=0x7faa0c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 -- 172.21.15.69:0/331980160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0c1a3660 con 0x7faa0c10e870 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1132c700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa1232e700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c1a2f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7faa0400e900 tx=0x7faa0400ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa0affd700 10 client.?.objecter ms_handle_connect 0x7faa0c10e870 2024-04-02T04:51:42.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa0affd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04004220 con 0x7faa0c10e870 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7faa040043c0 con 0x7faa0c10e870 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0c1a3950 con 0x7faa0c10e870 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04003c30 con 0x7faa0c10e870 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 10 client.26135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0c1ab1d0 con 0x7faa0c10e870 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.003+0000 7faa13d91700 1 librados: init done 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0c110e00 con 0x7faa0c10e870 2024-04-02T04:51:42.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7faa0400c780 con 0x7faa0c10e870 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa9f406f5e0 0x7fa9f4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(262..262 src has 74..262) ==== 12110+0+0 (secure 0 0 0) 0x7faa0408f380 con 0x7faa0c10e870 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 10 client.26135.objecter ms_dispatch 0x7faa0c001550 osd_map(262..262 src has 74..262) 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 3 client.26135.objecter handle_osd_map got epochs [262,262] > 0 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 3 client.26135.objecter handle_osd_map decoding full epoch 262 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa11b2d700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa9f406f5e0 0x7fa9f4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 20 client.26135.objecter dump_active .. 0 homeless 2024-04-02T04:51:42.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa11b2d700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa9f406f5e0 0x7fa9f4071aa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00b2e0 tx=0x7fa9fc005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:42.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.007+0000 7faa0affd700 10 client.26135.objecter ms_handle_connect 0x7fa9f406f5e0 2024-04-02T04:51:42.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.011+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faa0405b740 con 0x7faa0c10e870 2024-04-02T04:51:42.053 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:42.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:42.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:42.203 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:42.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.235+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7faa0c19f9b0 con 0x7faa0c10e870 2024-04-02T04:51:42.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.235+0000 7faa0affd700 1 -- 172.21.15.69:0/331980160 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v262) ==== 81+0+77230 (secure 0 0 0) 0x7faa0405f190 con 0x7faa0c10e870 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 262, 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "created": "2024-04-02T04:33:43.348385+0000", 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "modified": "2024-04-02T04:51:41.719381+0000", 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "last_up_change": "2024-04-02T04:49:46.297901+0000", 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "last_in_change": "2024-04-02T04:42:47.064987+0000", 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_num": 5799936, 2024-04-02T04:51:42.236 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_set": [ 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "pglog_hardlimit", 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "purged_snapdirs", 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_deletes", 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "sortbitwise" 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_version": 54, 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "full_ratio": 0.94999998807907104, 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_snapshot": "", 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_max": 231, 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd": 8, 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "require_min_compat_client": "luminous", 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "min_compat_client": "luminous", 2024-04-02T04:51:42.237 INFO:teuthology.orchestra.run.smithi069.stdout: "require_osd_release": "squid", 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_crimson": false, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pools": [ 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 1, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".mgr", 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:38:57.671714+0000", 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.238 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 1, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 1, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 1, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 1, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 1, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.239 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "23", 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.240 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.241 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.242 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_max": 32, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_min": 1 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": {} 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 8, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 8, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.25, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.243 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 2, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".rgw.root", 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:52.801763+0000", 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.244 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.245 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "79", 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.246 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.247 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:51:42.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 3, 2024-04-02T04:51:42.249 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.log", 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:54.772439+0000", 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.250 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "81", 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.251 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.252 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.253 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.254 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 4, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.control", 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:56.696111+0000", 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.256 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "83", 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.257 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.258 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.259 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.2400000095367432, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.2400000095367432, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.260 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 5, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "datapool", 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:57.254108+0000", 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.261 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 3, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 3, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 3, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 3, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 3, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.262 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "89", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 2, 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 89, 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.263 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.264 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.265 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd": {} 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.6700000762939453, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.6700000762939453, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 6, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.meta", 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:58.846023+0000", 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.267 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "85", 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.269 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_bias": 4 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 10, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:11.754451+0000", 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.273 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.274 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.275 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.277 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.4900000095367432, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.4900000095367432, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.190000057220459, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.190000057220459, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 13, 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosList_smithi069-72185-1", 2024-04-02T04:51:42.278 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:13.034188+0000", 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.279 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.280 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.281 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.282 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.283 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 40, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace.1804289383", 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.538674+0000", 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.284 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.285 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.286 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.287 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.288 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.289 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 41, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test_handler_errorsmithi069-72850-1", 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.947398+0000", 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.290 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.291 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.292 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.293 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.294 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.295 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 43, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace2.846930886", 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:25.876339+0000", 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.296 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:51:42.297 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.298 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.299 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.300 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-6", 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:53.761868+0000", 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.301 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.302 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "262", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "262", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "262", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "262", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 7, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 261, 2024-04-02T04:51:42.303 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [ 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: 229 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": 229, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": 229, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.305 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:51:42.306 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 229, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-21", 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:51:38.165375+0000", 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.307 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.308 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "262", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "262", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "262", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "262", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.309 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 7, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 261, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": 102, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.310 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.311 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.312 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 230, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "RoundTripAppend_smithi069-72145-31", 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:51:40.184017+0000", 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:51:42.313 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 3, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 1, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 8, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 8, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 8, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 8, 2024-04-02T04:51:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 8, 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "261", 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.315 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.316 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "testprofile-RoundTripAppend_smithi069-72145-31", 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.317 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 8192, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {} 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 231, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ReadIntoBufferlist_smithi069-72153-35", 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:51:40.622800+0000", 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:51:42.318 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:51:42.319 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "262", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:51:42.320 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:51:42.321 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:51:42.322 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.323 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b", 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 8, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6802", 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6803", 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.325 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6804", 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6805", 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.326 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6808", 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6809", 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6806", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6807", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6803/1087500595", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6805/1087500595", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6809/1087500595", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6807/1087500595", 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.328 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a", 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 13, 2024-04-02T04:51:42.329 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6810", 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6811", 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.330 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6812", 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6813", 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6816", 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6817", 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6814", 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6815", 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.333 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6811/2651666015", 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6813/2651666015", 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6817/2651666015", 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6815/2651666015", 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7", 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.334 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 19, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6818", 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6819", 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6820", 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.336 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6821", 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6824", 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.337 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6825", 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6822", 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6823", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6819/619998764", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6821/619998764", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6825/619998764", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6823/619998764", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed", 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 29, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.340 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6826", 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6827", 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.341 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6828", 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6829", 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.342 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6832", 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6833", 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6830", 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6831", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6827/207593208", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6829/207593208", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6833/207593208", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6831/207593208", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852", 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 36, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.345 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6800", 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6801", 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.346 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6802", 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6803", 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.347 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6806", 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6807", 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6804", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6805", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6801/2065947829", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6803/2065947829", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6807/2065947829", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6805/2065947829", 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.349 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe", 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 47, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.350 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6808", 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6809", 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6810", 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6811", 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.352 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6814", 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6815", 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.353 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6812", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6813", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6809/219578185", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6811/219578185", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6815/219578185", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6813/219578185", 2024-04-02T04:51:42.354 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727", 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 58, 2024-04-02T04:51:42.355 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 261, 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6816", 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6817", 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.356 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6818", 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6819", 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.357 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6822", 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6823", 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.358 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6820", 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6821", 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:51:42.359 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6817/1226172499", 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6819/1226172499", 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6823/1226172499", 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6821/1226172499", 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3", 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 0, 2024-04-02T04:51:42.360 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 69, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 151, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 152, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6824", 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6825", 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6826", 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.362 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6827", 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6830", 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.363 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6831", 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6828", 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6829", 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:51:42.364 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6825/148009079", 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6827/148009079", 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6831/148009079", 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6829/148009079", 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "exists" 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_xinfo": [ 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.365 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:40.067963+0000", 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.366 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:18.051401+0000", 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:03.586018+0000", 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.367 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:50.874317+0000", 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:18.611731+0000", 2024-04-02T04:51:42.368 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:16.385784+0000", 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.369 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:58.331556+0000", 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:49:15.075904+0000", 2024-04-02T04:51:42.370 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 150 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap": [ 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "102.7", 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: 5, 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: 0, 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: 4 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_items": [ 2024-04-02T04:51:42.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "2.9", 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 7, 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.6", 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 5 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.f", 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.17", 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:51:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_primaries": [], 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_temp": [], 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_temp": [], 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklist": { 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2738867930": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1798147282": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/52352690": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2928651991": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:51:42.374 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2763215770": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/3657409442": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/879855771": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1025808168": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1168952113": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1892956550": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/54578327": "2024-04-03T04:44:51.699278+0000" 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "range_blocklist": {}, 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profiles": { 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "default": { 2024-04-02T04:51:42.375 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van" 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "testprofile-RoundTripAppend_smithi069-72145-31": { 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-device-class": "", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-num-failure-domains": "0", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-osds-per-failure-domain": "0", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-root": "default", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:51:42.376 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van", 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "w": "8" 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps_queue": [ 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 17, 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 36, 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 38, 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 66, 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 101, 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.379 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 6 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.380 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 103, 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 112, 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.381 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 9 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 120, 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:51:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 137, 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 145, 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.383 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 146, 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.384 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 154, 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 164, 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 3 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 165, 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 173, 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.386 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 176, 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 188, 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 189, 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 196, 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 209, 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:51:42.389 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 6 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 214, 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: "new_removed_snaps": [], 2024-04-02T04:51:42.390 INFO:teuthology.orchestra.run.smithi069.stdout: "new_purged_snaps": [], 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_node_flags": {}, 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "device_class_flags": {}, 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode": { 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_enabled": false, 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_bucket_count": 0, 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "degraded_stretch_mode": 0, 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "recovering_stretch_mode": 0, 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_bucket": 0 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:51:42.391 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:51:42.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:42.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 10 librados: watch_flush enter 2024-04-02T04:51:42.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 10 librados: watch_flush exit 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 20 client.26135.objecter shutdown clearing up homeless session... 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 10 client.26135.objecter successfully canceled tick 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa9f406f5e0 msgr2=0x7fa9f4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa9f406f5e0 0x7fa9f4071aa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00b2e0 tx=0x7fa9fc005960 comp rx=0 tx=0).stop 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 msgr2=0x7faa0c1a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c1a2f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7faa0400e900 tx=0x7faa0400ec10 comp rx=0 tx=0).stop 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 shutdown_connections 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa9f406f5e0 0x7fa9f4071aa0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c10e870 0x7faa0c1a2f20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa0c108870 0x7faa0c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 --2- 172.21.15.69:0/331980160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c109230 0x7faa0c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 >> 172.21.15.69:0/331980160 conn(0x7faa0c075a10 msgr2=0x7faa0c076490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 shutdown_connections 2024-04-02T04:51:42.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 -- 172.21.15.69:0/331980160 wait complete. 2024-04-02T04:51:42.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:42.239+0000 7faa13d91700 1 librados: shutdown 2024-04-02T04:51:42.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.622750+0000 osd.5 (osd.5) 179 : cluster 0 41.13 scrub starts 2024-04-02T04:51:42.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: cluster 2024-04-02T04:51:40.625000+0000 osd.5 (osd.5) 180 : cluster 0 41.13 scrub ok 2024-04-02T04:51:42.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: cluster 2024-04-02T04:51:41.088636+0000 osd.6 (osd.6) 139 : cluster 0 43.1c scrub starts 2024-04-02T04:51:42.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: cluster 2024-04-02T04:51:41.090701+0000 osd.6 (osd.6) 140 : cluster 0 43.1c scrub ok 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: audit 2024-04-02T04:51:41.722713+0000 mon.a (mon.0) 1833 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: audit 2024-04-02T04:51:41.722995+0000 mon.a (mon.0) 1834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]': finished 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: cluster 2024-04-02T04:51:41.732572+0000 mon.a (mon.0) 1835 : cluster 0 osdmap e262: 8 total, 7 up, 8 in 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: audit 2024-04-02T04:51:41.736820+0000 mon.c (mon.2) 388 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]: dispatch 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: audit 2024-04-02T04:51:41.739630+0000 mon.a (mon.0) 1836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]: dispatch 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[21528]: audit 2024-04-02T04:51:41.859171+0000 mon.c (mon.2) 389 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.622750+0000 osd.5 (osd.5) 179 : cluster 0 41.13 scrub starts 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: cluster 2024-04-02T04:51:40.625000+0000 osd.5 (osd.5) 180 : cluster 0 41.13 scrub ok 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: cluster 2024-04-02T04:51:41.088636+0000 osd.6 (osd.6) 139 : cluster 0 43.1c scrub starts 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: cluster 2024-04-02T04:51:41.090701+0000 osd.6 (osd.6) 140 : cluster 0 43.1c scrub ok 2024-04-02T04:51:42.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: audit 2024-04-02T04:51:41.722713+0000 mon.a (mon.0) 1833 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:42.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: audit 2024-04-02T04:51:41.722995+0000 mon.a (mon.0) 1834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]': finished 2024-04-02T04:51:42.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: cluster 2024-04-02T04:51:41.732572+0000 mon.a (mon.0) 1835 : cluster 0 osdmap e262: 8 total, 7 up, 8 in 2024-04-02T04:51:42.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: audit 2024-04-02T04:51:41.736820+0000 mon.c (mon.2) 388 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]: dispatch 2024-04-02T04:51:42.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: audit 2024-04-02T04:51:41.739630+0000 mon.a (mon.0) 1836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]: dispatch 2024-04-02T04:51:42.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:42 smithi069 bash[28904]: audit 2024-04-02T04:51:41.859171+0000 mon.c (mon.2) 389 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:42.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:42.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.622750+0000 osd.5 (osd.5) 179 : cluster 0 41.13 scrub starts 2024-04-02T04:51:42.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: cluster 2024-04-02T04:51:40.625000+0000 osd.5 (osd.5) 180 : cluster 0 41.13 scrub ok 2024-04-02T04:51:42.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: cluster 2024-04-02T04:51:41.088636+0000 osd.6 (osd.6) 139 : cluster 0 43.1c scrub starts 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: cluster 2024-04-02T04:51:41.090701+0000 osd.6 (osd.6) 140 : cluster 0 43.1c scrub ok 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: audit 2024-04-02T04:51:41.722713+0000 mon.a (mon.0) 1833 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-72153-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: audit 2024-04-02T04:51:41.722995+0000 mon.a (mon.0) 1834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-21"}]': finished 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: cluster 2024-04-02T04:51:41.732572+0000 mon.a (mon.0) 1835 : cluster 0 osdmap e262: 8 total, 7 up, 8 in 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: audit 2024-04-02T04:51:41.736820+0000 mon.c (mon.2) 388 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]: dispatch 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: audit 2024-04-02T04:51:41.739630+0000 mon.a (mon.0) 1836 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]: dispatch 2024-04-02T04:51:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:42 smithi098 bash[23790]: audit 2024-04-02T04:51:41.859171+0000 mon.c (mon.2) 389 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:42.514 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:42.538 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:42.641 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:42.669 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:42.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:51:42.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:51:42.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:42.799 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:42.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:42.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:42.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:43.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:43.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:43.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:43.191 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:43.196 INFO:tasks.thrashosds.thrasher:j is {'epoch': 262, 'fsid': 'f978400e-f0a9-11ee-b647-cb9ed24678a4', 'created': '2024-04-02T04:33:43.348385+0000', 'modified': '2024-04-02T04:51:41.719381+0000', 'last_up_change': '2024-04-02T04:49:46.297901+0000', 'last_in_change': '2024-04-02T04:42:47.064987+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 54, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 231, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-02T04:43:52.801763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-02T04:43:54.772439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-02T04:43:56.696111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-04-02T04:43:57.254108+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-02T04:43:58.846023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-02T04:49:11.754451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi069-72185-1', 'create_time': '2024-04-02T04:49:13.034188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 40, 'pool_name': 'poolrace.1804289383', 'create_time': '2024-04-02T04:49:24.538674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 41, 'pool_name': 'test_handler_errorsmithi069-72850-1', 'create_time': '2024-04-02T04:49:24.947398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 43, 'pool_name': 'poolrace2.846930886', 'create_time': '2024-04-02T04:49:25.876339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 102, 'pool_name': 'test-rados-api-smithi069-72349-6', 'create_time': '2024-04-02T04:49:53.761868+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '262', 'last_force_op_resend_prenautilus': '262', 'last_force_op_resend_preluminous': '262', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 261, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [229], 'tier_of': -1, 'read_tier': 229, 'write_tier': 229, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 229, 'pool_name': 'test-rados-api-smithi069-72349-21', 'create_time': '2024-04-02T04:51:38.165375+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '262', 'last_force_op_resend_prenautilus': '262', 'last_force_op_resend_preluminous': '262', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 261, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 102, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 230, 'pool_name': 'RoundTripAppend_smithi069-72145-31', 'create_time': '2024-04-02T04:51:40.184017+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '261', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-RoundTripAppend_smithi069-72145-31', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 231, 'pool_name': 'ReadIntoBufferlist_smithi069-72153-35', 'create_time': '2024-04-02T04:51:40.622800+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1087500595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1087500595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 1087500595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1087500595}]}, 'public_addr': '172.21.15.69:6803/1087500595', 'cluster_addr': '172.21.15.69:6805/1087500595', 'heartbeat_back_addr': '172.21.15.69:6809/1087500595', 'heartbeat_front_addr': '172.21.15.69:6807/1087500595', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'afb71182-d4f7-416d-ae44-90f28eab4b7a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6811', 'nonce': 2651666015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6812', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2651666015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6816', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6817', 'nonce': 2651666015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6815', 'nonce': 2651666015}]}, 'public_addr': '172.21.15.69:6811/2651666015', 'cluster_addr': '172.21.15.69:6813/2651666015', 'heartbeat_back_addr': '172.21.15.69:6817/2651666015', 'heartbeat_front_addr': '172.21.15.69:6815/2651666015', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6818', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6819', 'nonce': 619998764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6820', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6821', 'nonce': 619998764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6825', 'nonce': 619998764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6822', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 619998764}]}, 'public_addr': '172.21.15.69:6819/619998764', 'cluster_addr': '172.21.15.69:6821/619998764', 'heartbeat_back_addr': '172.21.15.69:6825/619998764', 'heartbeat_front_addr': '172.21.15.69:6823/619998764', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '20e4526d-2707-44d0-b13a-af36019c42ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6826', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 207593208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 207593208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 207593208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 207593208}]}, 'public_addr': '172.21.15.69:6827/207593208', 'cluster_addr': '172.21.15.69:6829/207593208', 'heartbeat_back_addr': '172.21.15.69:6833/207593208', 'heartbeat_front_addr': '172.21.15.69:6831/207593208', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 2065947829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 2065947829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 2065947829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 2065947829}]}, 'public_addr': '172.21.15.98:6801/2065947829', 'cluster_addr': '172.21.15.98:6803/2065947829', 'heartbeat_back_addr': '172.21.15.98:6807/2065947829', 'heartbeat_front_addr': '172.21.15.98:6805/2065947829', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '902326af-4a40-47c8-a3b8-d5d5536f90fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 219578185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 219578185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6814', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 219578185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6813', 'nonce': 219578185}]}, 'public_addr': '172.21.15.98:6809/219578185', 'cluster_addr': '172.21.15.98:6811/219578185', 'heartbeat_back_addr': '172.21.15.98:6815/219578185', 'heartbeat_front_addr': '172.21.15.98:6813/219578185', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 1226172499}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 1226172499}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 1226172499}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 1226172499}]}, 'public_addr': '172.21.15.98:6817/1226172499', 'cluster_addr': '172.21.15.98:6819/1226172499', 'heartbeat_back_addr': '172.21.15.98:6823/1226172499', 'heartbeat_front_addr': '172.21.15.98:6821/1226172499', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 151, 'down_at': 152, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 148009079}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 148009079}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 148009079}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 148009079}]}, 'public_addr': '172.21.15.98:6825/148009079', 'cluster_addr': '172.21.15.98:6827/148009079', 'heartbeat_back_addr': '172.21.15.98:6831/148009079', 'heartbeat_front_addr': '172.21.15.98:6829/148009079', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:40.067963+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:18.051401+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:03.586018+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:50.874317+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:18.611731+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:16.385784+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:58.331556+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:49:15.075904+0000', 'dead_epoch': 150}], 'pg_upmap': [{'pgid': '102.7', 'osds': [5, 0, 4]}], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.69:0/2738867930': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/1798147282': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/52352690': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/2928651991': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:6800/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/2763215770': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/3657409442': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/879855771': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1025808168': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1168952113': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/1892956550': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6801/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/54578327': '2024-04-03T04:44:51.699278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-RoundTripAppend_smithi069-72145-31': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 36, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 38, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 66, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 101, 'snaps': [{'begin': 2, 'length': 4}, {'begin': 7, 'length': 2}]}, {'pool': 102, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 103, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 2, 'length': 9}, {'begin': 14, 'length': 1}]}, {'pool': 120, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 137, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 145, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 146, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 164, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 173, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 196, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 209, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-02T04:51:43.197 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 102.7 2024-04-02T04:51:43.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap 102.7 2024-04-02T04:51:43.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:43.321 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:43.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:51:43.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: cluster 2024-04-02T04:51:41.961610+0000 mgr.y (mgr.14538) 322 : cluster 0 pgmap v370: 396 pgs: 72 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:43.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: audit 2024-04-02T04:51:42.236073+0000 mon.c (mon.2) 390 : audit 0 from='client.? 172.21.15.69:0/331980160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.309803+0000 osd.1 (osd.1) 195 : cluster 0 13.1a deep-scrub starts 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.313228+0000 osd.1 (osd.1) 196 : cluster 0 13.1a deep-scrub ok 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.722618+0000 mon.a (mon.0) 1837 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.723789+0000 mon.a (mon.0) 1838 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: audit 2024-04-02T04:51:42.731784+0000 mon.a (mon.0) 1839 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]': finished 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.738192+0000 mon.a (mon.0) 1840 : cluster 0 osdmap e263: 8 total, 7 up, 8 in 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: audit 2024-04-02T04:51:42.744910+0000 mon.b (mon.1) 184 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:43.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:43 smithi098 bash[23790]: audit 2024-04-02T04:51:42.749994+0000 mon.a (mon.0) 1841 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:43.448 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: cluster 2024-04-02T04:51:41.961610+0000 mgr.y (mgr.14538) 322 : cluster 0 pgmap v370: 396 pgs: 72 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: audit 2024-04-02T04:51:42.236073+0000 mon.c (mon.2) 390 : audit 0 from='client.? 172.21.15.69:0/331980160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.309803+0000 osd.1 (osd.1) 195 : cluster 0 13.1a deep-scrub starts 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.313228+0000 osd.1 (osd.1) 196 : cluster 0 13.1a deep-scrub ok 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.722618+0000 mon.a (mon.0) 1837 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.723789+0000 mon.a (mon.0) 1838 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: audit 2024-04-02T04:51:42.731784+0000 mon.a (mon.0) 1839 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]': finished 2024-04-02T04:51:43.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.738192+0000 mon.a (mon.0) 1840 : cluster 0 osdmap e263: 8 total, 7 up, 8 in 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: audit 2024-04-02T04:51:42.744910+0000 mon.b (mon.1) 184 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[21528]: audit 2024-04-02T04:51:42.749994+0000 mon.a (mon.0) 1841 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: cluster 2024-04-02T04:51:41.961610+0000 mgr.y (mgr.14538) 322 : cluster 0 pgmap v370: 396 pgs: 72 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 203 active+clean, 37 active+undersized+degraded; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 272/1533 objects degraded (17.743%) 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: audit 2024-04-02T04:51:42.236073+0000 mon.c (mon.2) 390 : audit 0 from='client.? 172.21.15.69:0/331980160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.309803+0000 osd.1 (osd.1) 195 : cluster 0 13.1a deep-scrub starts 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.313228+0000 osd.1 (osd.1) 196 : cluster 0 13.1a deep-scrub ok 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.722618+0000 mon.a (mon.0) 1837 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.723789+0000 mon.a (mon.0) 1838 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: audit 2024-04-02T04:51:42.731784+0000 mon.a (mon.0) 1839 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-21", "mode": "writeback"}]': finished 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.738192+0000 mon.a (mon.0) 1840 : cluster 0 osdmap e263: 8 total, 7 up, 8 in 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: audit 2024-04-02T04:51:42.744910+0000 mon.b (mon.1) 184 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:43.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:43 smithi069 bash[28904]: audit 2024-04-02T04:51:42.749994+0000 mon.a (mon.0) 1841 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:43.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:43.574 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:43.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:43.702 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:43.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:43.829 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:51:43.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:43.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:43.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:44.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:44.090 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:44.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:44.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:44.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.222083+0000 osd.2 (osd.2) 129 : cluster 0 41.5 scrub starts 2024-04-02T04:51:44.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:42.223089+0000 osd.2 (osd.2) 130 : cluster 0 41.5 scrub ok 2024-04-02T04:51:44.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:42.860146+0000 mon.c (mon.2) 391 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:44.334 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:42.878284+0000 mon.c (mon.2) 392 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:43.064883+0000 osd.6 (osd.6) 141 : cluster 0 3.a scrub starts 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:43.069684+0000 osd.6 (osd.6) 142 : cluster 0 3.a scrub ok 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.100713+0000 mgr.y (mgr.14538) 323 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:43.196843+0000 osd.2 (osd.2) 131 : cluster 0 43.18 scrub starts 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:43.198039+0000 osd.2 (osd.2) 132 : cluster 0 43.18 scrub ok 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.230020+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.736324+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.736402+0000 mon.a (mon.0) 1844 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: cluster 2024-04-02T04:51:43.739836+0000 mon.a (mon.0) 1845 : cluster 0 osdmap e264: 8 total, 7 up, 8 in 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.746766+0000 mon.b (mon.1) 185 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.751045+0000 mon.a (mon.0) 1846 : audit 1 from='client.? 172.21.15.69:0/3952449955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.752230+0000 mon.a (mon.0) 1847 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.758830+0000 mon.c (mon.2) 393 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:44.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.759439+0000 mon.a (mon.0) 1848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[21528]: audit 2024-04-02T04:51:43.861909+0000 mon.c (mon.2) 394 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.222083+0000 osd.2 (osd.2) 129 : cluster 0 41.5 scrub starts 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:42.223089+0000 osd.2 (osd.2) 130 : cluster 0 41.5 scrub ok 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:42.860146+0000 mon.c (mon.2) 391 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:42.878284+0000 mon.c (mon.2) 392 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:43.064883+0000 osd.6 (osd.6) 141 : cluster 0 3.a scrub starts 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:43.069684+0000 osd.6 (osd.6) 142 : cluster 0 3.a scrub ok 2024-04-02T04:51:44.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.100713+0000 mgr.y (mgr.14538) 323 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:43.196843+0000 osd.2 (osd.2) 131 : cluster 0 43.18 scrub starts 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:43.198039+0000 osd.2 (osd.2) 132 : cluster 0 43.18 scrub ok 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.230020+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.736324+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.736402+0000 mon.a (mon.0) 1844 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: cluster 2024-04-02T04:51:43.739836+0000 mon.a (mon.0) 1845 : cluster 0 osdmap e264: 8 total, 7 up, 8 in 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.746766+0000 mon.b (mon.1) 185 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.751045+0000 mon.a (mon.0) 1846 : audit 1 from='client.? 172.21.15.69:0/3952449955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.752230+0000 mon.a (mon.0) 1847 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.758830+0000 mon.c (mon.2) 393 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.759439+0000 mon.a (mon.0) 1848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:44.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:44 smithi069 bash[28904]: audit 2024-04-02T04:51:43.861909+0000 mon.c (mon.2) 394 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:44.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:44.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:44.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.222083+0000 osd.2 (osd.2) 129 : cluster 0 41.5 scrub starts 2024-04-02T04:51:44.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:42.223089+0000 osd.2 (osd.2) 130 : cluster 0 41.5 scrub ok 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:42.860146+0000 mon.c (mon.2) 391 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:42.878284+0000 mon.c (mon.2) 392 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:43.064883+0000 osd.6 (osd.6) 141 : cluster 0 3.a scrub starts 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:43.069684+0000 osd.6 (osd.6) 142 : cluster 0 3.a scrub ok 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.100713+0000 mgr.y (mgr.14538) 323 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:43.196843+0000 osd.2 (osd.2) 131 : cluster 0 43.18 scrub starts 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:43.198039+0000 osd.2 (osd.2) 132 : cluster 0 43.18 scrub ok 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.230020+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.736324+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.736402+0000 mon.a (mon.0) 1844 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: cluster 2024-04-02T04:51:43.739836+0000 mon.a (mon.0) 1845 : cluster 0 osdmap e264: 8 total, 7 up, 8 in 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.746766+0000 mon.b (mon.1) 185 : audit 1 from='client.? 172.21.15.69:0/4195475675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.751045+0000 mon.a (mon.0) 1846 : audit 1 from='client.? 172.21.15.69:0/3952449955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:44.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.752230+0000 mon.a (mon.0) 1847 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]: dispatch 2024-04-02T04:51:44.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.758830+0000 mon.c (mon.2) 393 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:44.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.759439+0000 mon.a (mon.0) 1848 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]: dispatch 2024-04-02T04:51:44.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:44 smithi098 bash[23790]: audit 2024-04-02T04:51:43.861909+0000 mon.c (mon.2) 394 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:44.475 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:44.578 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:44.605 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:44.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:44.764 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:44.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:44.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:44.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:44.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:44.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:45.126 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:45.150 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:45.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:45.272 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:45.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:45.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: cluster 2024-04-02T04:51:43.962717+0000 mgr.y (mgr.14538) 324 : cluster 0 pgmap v373: 388 pgs: 32 unknown, 1 active+clean+snaptrim, 89 active+undersized, 228 active+clean, 38 active+undersized+degraded; 587 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s; 273/1521 objects degraded (17.949%) 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.201361+0000 osd.2 (osd.2) 133 : cluster 0 43.7 scrub starts 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.205729+0000 osd.2 (osd.2) 134 : cluster 0 43.7 scrub ok 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.687966+0000 mon.a (mon.0) 1849 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.690510+0000 mon.a (mon.0) 1850 : audit 1 from='client.? 172.21.15.69:0/3952449955' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.690727+0000 mon.a (mon.0) 1851 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:45.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.690962+0000 mon.a (mon.0) 1852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]': finished 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.696490+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e265: 8 total, 7 up, 8 in 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.738917+0000 mon.a (mon.0) 1854 : cluster 3 Health check update: Degraded data redundancy: 273/1521 objects degraded (17.949%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.772396+0000 mon.c (mon.2) 395 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.773081+0000 mon.a (mon.0) 1855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.773994+0000 mon.c (mon.2) 396 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.774565+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.775358+0000 mon.c (mon.2) 397 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.776206+0000 mon.a (mon.0) 1857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[21528]: audit 2024-04-02T04:51:44.862989+0000 mon.c (mon.2) 398 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: cluster 2024-04-02T04:51:43.962717+0000 mgr.y (mgr.14538) 324 : cluster 0 pgmap v373: 388 pgs: 32 unknown, 1 active+clean+snaptrim, 89 active+undersized, 228 active+clean, 38 active+undersized+degraded; 587 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s; 273/1521 objects degraded (17.949%) 2024-04-02T04:51:45.402 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.201361+0000 osd.2 (osd.2) 133 : cluster 0 43.7 scrub starts 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.205729+0000 osd.2 (osd.2) 134 : cluster 0 43.7 scrub ok 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.687966+0000 mon.a (mon.0) 1849 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.690510+0000 mon.a (mon.0) 1850 : audit 1 from='client.? 172.21.15.69:0/3952449955' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.690727+0000 mon.a (mon.0) 1851 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.690962+0000 mon.a (mon.0) 1852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]': finished 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.696490+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e265: 8 total, 7 up, 8 in 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.738917+0000 mon.a (mon.0) 1854 : cluster 3 Health check update: Degraded data redundancy: 273/1521 objects degraded (17.949%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.772396+0000 mon.c (mon.2) 395 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.773081+0000 mon.a (mon.0) 1855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.773994+0000 mon.c (mon.2) 396 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.774565+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.775358+0000 mon.c (mon.2) 397 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.776206+0000 mon.a (mon.0) 1857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:45.403 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:45 smithi069 bash[28904]: audit 2024-04-02T04:51:44.862989+0000 mon.c (mon.2) 398 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:45.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: cluster 2024-04-02T04:51:43.962717+0000 mgr.y (mgr.14538) 324 : cluster 0 pgmap v373: 388 pgs: 32 unknown, 1 active+clean+snaptrim, 89 active+undersized, 228 active+clean, 38 active+undersized+degraded; 587 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s; 273/1521 objects degraded (17.949%) 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.201361+0000 osd.2 (osd.2) 133 : cluster 0 43.7 scrub starts 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.205729+0000 osd.2 (osd.2) 134 : cluster 0 43.7 scrub ok 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.687966+0000 mon.a (mon.0) 1849 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.690510+0000 mon.a (mon.0) 1850 : audit 1 from='client.? 172.21.15.69:0/3952449955' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.690727+0000 mon.a (mon.0) 1851 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-72145-31"}]': finished 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.690962+0000 mon.a (mon.0) 1852 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-21"}]': finished 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.696490+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e265: 8 total, 7 up, 8 in 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.738917+0000 mon.a (mon.0) 1854 : cluster 3 Health check update: Degraded data redundancy: 273/1521 objects degraded (17.949%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.772396+0000 mon.c (mon.2) 395 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.773081+0000 mon.a (mon.0) 1855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.773994+0000 mon.c (mon.2) 396 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.774565+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:45.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.775358+0000 mon.c (mon.2) 397 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:45.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.776206+0000 mon.a (mon.0) 1857 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:45.510 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:45 smithi098 bash[23790]: audit 2024-04-02T04:51:44.862989+0000 mon.c (mon.2) 398 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:45.526 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:45.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:45.656 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:45.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:45.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:45.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:46.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:46.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:46.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:46.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:46.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:46.283 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:46.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:46.410 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:46.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:46.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:46.671 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:46.774 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:46.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:46.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:46.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:46.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:46.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.927279+0000 osd.0 (osd.0) 179 : cluster 0 40.16 scrub starts 2024-04-02T04:51:46.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:44.929536+0000 osd.0 (osd.0) 180 : cluster 0 40.16 scrub ok 2024-04-02T04:51:46.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:45.520500+0000 osd.5 (osd.5) 181 : cluster 0 41.8 scrub starts 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:45.521584+0000 osd.5 (osd.5) 182 : cluster 0 41.8 scrub ok 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: audit 2024-04-02T04:51:45.708090+0000 mon.a (mon.0) 1858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:45.712166+0000 mon.a (mon.0) 1859 : cluster 0 osdmap e266: 8 total, 7 up, 8 in 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: audit 2024-04-02T04:51:45.723445+0000 mon.c (mon.2) 399 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: audit 2024-04-02T04:51:45.724519+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: audit 2024-04-02T04:51:45.724729+0000 mon.c (mon.2) 400 : audit 1 from='client.? 172.21.15.69:0/3534651428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: audit 2024-04-02T04:51:45.864363+0000 mon.c (mon.2) 401 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: audit 2024-04-02T04:51:46.146498+0000 mon.a (mon.0) 1861 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:46.282208+0000 osd.4 (osd.4) 185 : cluster 0 43.15 scrub starts 2024-04-02T04:51:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:46 smithi098 bash[23790]: cluster 2024-04-02T04:51:46.284014+0000 osd.4 (osd.4) 186 : cluster 0 43.15 scrub ok 2024-04-02T04:51:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.927279+0000 osd.0 (osd.0) 179 : cluster 0 40.16 scrub starts 2024-04-02T04:51:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:44.929536+0000 osd.0 (osd.0) 180 : cluster 0 40.16 scrub ok 2024-04-02T04:51:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:45.520500+0000 osd.5 (osd.5) 181 : cluster 0 41.8 scrub starts 2024-04-02T04:51:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:45.521584+0000 osd.5 (osd.5) 182 : cluster 0 41.8 scrub ok 2024-04-02T04:51:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: audit 2024-04-02T04:51:45.708090+0000 mon.a (mon.0) 1858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:45.712166+0000 mon.a (mon.0) 1859 : cluster 0 osdmap e266: 8 total, 7 up, 8 in 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: audit 2024-04-02T04:51:45.723445+0000 mon.c (mon.2) 399 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: audit 2024-04-02T04:51:45.724519+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: audit 2024-04-02T04:51:45.724729+0000 mon.c (mon.2) 400 : audit 1 from='client.? 172.21.15.69:0/3534651428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: audit 2024-04-02T04:51:45.864363+0000 mon.c (mon.2) 401 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: audit 2024-04-02T04:51:46.146498+0000 mon.a (mon.0) 1861 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:46.282208+0000 osd.4 (osd.4) 185 : cluster 0 43.15 scrub starts 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[21528]: cluster 2024-04-02T04:51:46.284014+0000 osd.4 (osd.4) 186 : cluster 0 43.15 scrub ok 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.927279+0000 osd.0 (osd.0) 179 : cluster 0 40.16 scrub starts 2024-04-02T04:51:47.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:44.929536+0000 osd.0 (osd.0) 180 : cluster 0 40.16 scrub ok 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:45.520500+0000 osd.5 (osd.5) 181 : cluster 0 41.8 scrub starts 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:45.521584+0000 osd.5 (osd.5) 182 : cluster 0 41.8 scrub ok 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: audit 2024-04-02T04:51:45.708090+0000 mon.a (mon.0) 1858 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-72145-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:45.712166+0000 mon.a (mon.0) 1859 : cluster 0 osdmap e266: 8 total, 7 up, 8 in 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: audit 2024-04-02T04:51:45.723445+0000 mon.c (mon.2) 399 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: audit 2024-04-02T04:51:45.724519+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: audit 2024-04-02T04:51:45.724729+0000 mon.c (mon.2) 400 : audit 1 from='client.? 172.21.15.69:0/3534651428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: audit 2024-04-02T04:51:45.864363+0000 mon.c (mon.2) 401 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: audit 2024-04-02T04:51:46.146498+0000 mon.a (mon.0) 1861 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:46.282208+0000 osd.4 (osd.4) 185 : cluster 0 43.15 scrub starts 2024-04-02T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:46 smithi069 bash[28904]: cluster 2024-04-02T04:51:46.284014+0000 osd.4 (osd.4) 186 : cluster 0 43.15 scrub ok 2024-04-02T04:51:47.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:47.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:47.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:47.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:47.444 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:51:47.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:47.555 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:51:47.555 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:51:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:47.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:47.687 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:47.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: cluster 2024-04-02T04:51:45.964140+0000 mgr.y (mgr.14538) 325 : cluster 0 pgmap v376: 388 pgs: 64 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 587 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 272/1515 objects degraded (17.954%) 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: audit 2024-04-02T04:51:46.721555+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: audit 2024-04-02T04:51:46.865813+0000 mon.c (mon.2) 402 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: cluster 2024-04-02T04:51:47.188479+0000 mon.a (mon.0) 1863 : cluster 0 osdmap e267: 8 total, 7 up, 8 in 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: audit 2024-04-02T04:51:47.189312+0000 mon.c (mon.2) 403 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: audit 2024-04-02T04:51:47.642837+0000 mon.a (mon.0) 1864 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: audit 2024-04-02T04:51:47.646021+0000 mon.b (mon.1) 186 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:47.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[21528]: audit 2024-04-02T04:51:47.648440+0000 mon.a (mon.0) 1865 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: cluster 2024-04-02T04:51:45.964140+0000 mgr.y (mgr.14538) 325 : cluster 0 pgmap v376: 388 pgs: 64 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 587 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 272/1515 objects degraded (17.954%) 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: audit 2024-04-02T04:51:46.721555+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: audit 2024-04-02T04:51:46.865813+0000 mon.c (mon.2) 402 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: cluster 2024-04-02T04:51:47.188479+0000 mon.a (mon.0) 1863 : cluster 0 osdmap e267: 8 total, 7 up, 8 in 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: audit 2024-04-02T04:51:47.189312+0000 mon.c (mon.2) 403 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: audit 2024-04-02T04:51:47.642837+0000 mon.a (mon.0) 1864 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: audit 2024-04-02T04:51:47.646021+0000 mon.b (mon.1) 186 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:47.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:47 smithi069 bash[28904]: audit 2024-04-02T04:51:47.648440+0000 mon.a (mon.0) 1865 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:47.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:47.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: cluster 2024-04-02T04:51:45.964140+0000 mgr.y (mgr.14538) 325 : cluster 0 pgmap v376: 388 pgs: 64 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 587 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 272/1515 objects degraded (17.954%) 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: audit 2024-04-02T04:51:46.721555+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: audit 2024-04-02T04:51:46.865813+0000 mon.c (mon.2) 402 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: cluster 2024-04-02T04:51:47.188479+0000 mon.a (mon.0) 1863 : cluster 0 osdmap e267: 8 total, 7 up, 8 in 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: audit 2024-04-02T04:51:47.189312+0000 mon.c (mon.2) 403 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: audit 2024-04-02T04:51:47.642837+0000 mon.a (mon.0) 1864 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:47.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: audit 2024-04-02T04:51:47.646021+0000 mon.b (mon.1) 186 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:47.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:47 smithi098 bash[23790]: audit 2024-04-02T04:51:47.648440+0000 mon.a (mon.0) 1865 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:51:47.935 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:47.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:48.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:48.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:48.195 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:48.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:48.323 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:48.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:48.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-02T04:51:48.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:48.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:48.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:48.825 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:48.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:48.957 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: cluster 2024-04-02T04:51:47.138830+0000 osd.2 (osd.2) 135 : cluster 0 102.1c scrub starts 2024-04-02T04:51:48.957 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: cluster 2024-04-02T04:51:47.142159+0000 osd.2 (osd.2) 136 : cluster 0 102.1c scrub ok 2024-04-02T04:51:48.957 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: audit 2024-04-02T04:51:47.732674+0000 mon.a (mon.0) 1866 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: audit 2024-04-02T04:51:47.732756+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: cluster 2024-04-02T04:51:47.739056+0000 mon.a (mon.0) 1868 : cluster 0 osdmap e268: 8 total, 7 up, 8 in 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: audit 2024-04-02T04:51:47.740918+0000 mon.c (mon.2) 404 : audit 1 from='client.? 172.21.15.69:0/1332857880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: audit 2024-04-02T04:51:47.746011+0000 mon.a (mon.0) 1869 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: audit 2024-04-02T04:51:47.866948+0000 mon.c (mon.2) 405 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: cluster 2024-04-02T04:51:47.968210+0000 mgr.y (mgr.14538) 326 : cluster 0 pgmap v379: 460 pgs: 1 active+clean+snaptrim, 40 creating+peering, 114 active+undersized, 268 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s; 273/1524 objects degraded (17.913%) 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.309272+0000 osd.4 (osd.4) 187 : cluster 0 4.1a scrub starts 2024-04-02T04:51:48.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:48 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.311596+0000 osd.4 (osd.4) 188 : cluster 0 4.1a scrub ok 2024-04-02T04:51:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:48.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:49.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:51:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:51:49.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: cluster 2024-04-02T04:51:47.138830+0000 osd.2 (osd.2) 135 : cluster 0 102.1c scrub starts 2024-04-02T04:51:49.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: cluster 2024-04-02T04:51:47.142159+0000 osd.2 (osd.2) 136 : cluster 0 102.1c scrub ok 2024-04-02T04:51:49.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: audit 2024-04-02T04:51:47.732674+0000 mon.a (mon.0) 1866 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:49.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: audit 2024-04-02T04:51:47.732756+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:49.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: cluster 2024-04-02T04:51:47.739056+0000 mon.a (mon.0) 1868 : cluster 0 osdmap e268: 8 total, 7 up, 8 in 2024-04-02T04:51:49.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: audit 2024-04-02T04:51:47.740918+0000 mon.c (mon.2) 404 : audit 1 from='client.? 172.21.15.69:0/1332857880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:49.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: audit 2024-04-02T04:51:47.746011+0000 mon.a (mon.0) 1869 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:49.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: audit 2024-04-02T04:51:47.866948+0000 mon.c (mon.2) 405 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:49.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: cluster 2024-04-02T04:51:47.968210+0000 mgr.y (mgr.14538) 326 : cluster 0 pgmap v379: 460 pgs: 1 active+clean+snaptrim, 40 creating+peering, 114 active+undersized, 268 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s; 273/1524 objects degraded (17.913%) 2024-04-02T04:51:49.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.309272+0000 osd.4 (osd.4) 187 : cluster 0 4.1a scrub starts 2024-04-02T04:51:49.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.311596+0000 osd.4 (osd.4) 188 : cluster 0 4.1a scrub ok 2024-04-02T04:51:49.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: cluster 2024-04-02T04:51:47.138830+0000 osd.2 (osd.2) 135 : cluster 0 102.1c scrub starts 2024-04-02T04:51:49.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: cluster 2024-04-02T04:51:47.142159+0000 osd.2 (osd.2) 136 : cluster 0 102.1c scrub ok 2024-04-02T04:51:49.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: audit 2024-04-02T04:51:47.732674+0000 mon.a (mon.0) 1866 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-72145-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: audit 2024-04-02T04:51:47.732756+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: cluster 2024-04-02T04:51:47.739056+0000 mon.a (mon.0) 1868 : cluster 0 osdmap e268: 8 total, 7 up, 8 in 2024-04-02T04:51:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: audit 2024-04-02T04:51:47.740918+0000 mon.c (mon.2) 404 : audit 1 from='client.? 172.21.15.69:0/1332857880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: audit 2024-04-02T04:51:47.746011+0000 mon.a (mon.0) 1869 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: audit 2024-04-02T04:51:47.866948+0000 mon.c (mon.2) 405 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:49.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: cluster 2024-04-02T04:51:47.968210+0000 mgr.y (mgr.14538) 326 : cluster 0 pgmap v379: 460 pgs: 1 active+clean+snaptrim, 40 creating+peering, 114 active+undersized, 268 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s; 273/1524 objects degraded (17.913%) 2024-04-02T04:51:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.309272+0000 osd.4 (osd.4) 187 : cluster 0 4.1a scrub starts 2024-04-02T04:51:49.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:48 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.311596+0000 osd.4 (osd.4) 188 : cluster 0 4.1a scrub ok 2024-04-02T04:51:49.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:49.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:49.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:49.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:49.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:49.477 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:49.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:49.606 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:49.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:49.735 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:49.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:49.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.137494+0000 osd.2 (osd.2) 137 : cluster 0 102.17 scrub starts 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.141386+0000 osd.2 (osd.2) 138 : cluster 0 102.17 scrub ok 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.734814+0000 mon.a (mon.0) 1870 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: audit 2024-04-02T04:51:48.736358+0000 mon.a (mon.0) 1871 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.744669+0000 mon.a (mon.0) 1872 : cluster 0 osdmap e269: 8 total, 7 up, 8 in 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: audit 2024-04-02T04:51:48.824648+0000 mon.c (mon.2) 406 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: audit 2024-04-02T04:51:48.825097+0000 mon.a (mon.0) 1873 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: audit 2024-04-02T04:51:48.867974+0000 mon.c (mon.2) 407 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.300071+0000 osd.4 (osd.4) 189 : cluster 0 4.18 scrub starts 2024-04-02T04:51:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:49 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.302536+0000 osd.4 (osd.4) 190 : cluster 0 4.18 scrub ok 2024-04-02T04:51:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:49.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.137494+0000 osd.2 (osd.2) 137 : cluster 0 102.17 scrub starts 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.141386+0000 osd.2 (osd.2) 138 : cluster 0 102.17 scrub ok 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.734814+0000 mon.a (mon.0) 1870 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: audit 2024-04-02T04:51:48.736358+0000 mon.a (mon.0) 1871 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.744669+0000 mon.a (mon.0) 1872 : cluster 0 osdmap e269: 8 total, 7 up, 8 in 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: audit 2024-04-02T04:51:48.824648+0000 mon.c (mon.2) 406 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: audit 2024-04-02T04:51:48.825097+0000 mon.a (mon.0) 1873 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: audit 2024-04-02T04:51:48.867974+0000 mon.c (mon.2) 407 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.300071+0000 osd.4 (osd.4) 189 : cluster 0 4.18 scrub starts 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.302536+0000 osd.4 (osd.4) 190 : cluster 0 4.18 scrub ok 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.137494+0000 osd.2 (osd.2) 137 : cluster 0 102.17 scrub starts 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.141386+0000 osd.2 (osd.2) 138 : cluster 0 102.17 scrub ok 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.734814+0000 mon.a (mon.0) 1870 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: audit 2024-04-02T04:51:48.736358+0000 mon.a (mon.0) 1871 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-72153-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.744669+0000 mon.a (mon.0) 1872 : cluster 0 osdmap e269: 8 total, 7 up, 8 in 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: audit 2024-04-02T04:51:48.824648+0000 mon.c (mon.2) 406 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: audit 2024-04-02T04:51:48.825097+0000 mon.a (mon.0) 1873 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: audit 2024-04-02T04:51:48.867974+0000 mon.c (mon.2) 407 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.300071+0000 osd.4 (osd.4) 189 : cluster 0 4.18 scrub starts 2024-04-02T04:51:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:49 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.302536+0000 osd.4 (osd.4) 190 : cluster 0 4.18 scrub ok 2024-04-02T04:51:50.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:50.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:50.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:50.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:50.482 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:50.509 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:50.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:50.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:50.763 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:50.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:50.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.956402+0000 osd.0 (osd.0) 181 : cluster 0 40.3 deep-scrub starts 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: cluster 2024-04-02T04:51:48.958361+0000 osd.0 (osd.0) 182 : cluster 0 40.3 deep-scrub ok 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.752888+0000 osd.3 (osd.3) 211 : cluster 0 3.2 scrub starts 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.756574+0000 osd.3 (osd.3) 212 : cluster 0 3.2 scrub ok 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: audit 2024-04-02T04:51:49.760786+0000 mon.a (mon.0) 1874 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.767296+0000 mon.a (mon.0) 1875 : cluster 0 osdmap e270: 8 total, 7 up, 8 in 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: audit 2024-04-02T04:51:49.773015+0000 mon.c (mon.2) 408 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: audit 2024-04-02T04:51:49.784951+0000 mon.c (mon.2) 409 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: audit 2024-04-02T04:51:49.787142+0000 mon.a (mon.0) 1876 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: audit 2024-04-02T04:51:49.788199+0000 mon.a (mon.0) 1877 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:50.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: audit 2024-04-02T04:51:49.869384+0000 mon.c (mon.2) 410 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:50.877 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:50 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.970050+0000 mgr.y (mgr.14538) 327 : cluster 0 pgmap v382: 420 pgs: 1 active+clean+snaptrim, 114 active+undersized, 268 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 273/1524 objects degraded (17.913%) 2024-04-02T04:51:50.891 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:50.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:50.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:51.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:51.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.956402+0000 osd.0 (osd.0) 181 : cluster 0 40.3 deep-scrub starts 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: cluster 2024-04-02T04:51:48.958361+0000 osd.0 (osd.0) 182 : cluster 0 40.3 deep-scrub ok 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.752888+0000 osd.3 (osd.3) 211 : cluster 0 3.2 scrub starts 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.756574+0000 osd.3 (osd.3) 212 : cluster 0 3.2 scrub ok 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: audit 2024-04-02T04:51:49.760786+0000 mon.a (mon.0) 1874 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.767296+0000 mon.a (mon.0) 1875 : cluster 0 osdmap e270: 8 total, 7 up, 8 in 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: audit 2024-04-02T04:51:49.773015+0000 mon.c (mon.2) 408 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: audit 2024-04-02T04:51:49.784951+0000 mon.c (mon.2) 409 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: audit 2024-04-02T04:51:49.787142+0000 mon.a (mon.0) 1876 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: audit 2024-04-02T04:51:49.788199+0000 mon.a (mon.0) 1877 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: audit 2024-04-02T04:51:49.869384+0000 mon.c (mon.2) 410 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.970050+0000 mgr.y (mgr.14538) 327 : cluster 0 pgmap v382: 420 pgs: 1 active+clean+snaptrim, 114 active+undersized, 268 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 273/1524 objects degraded (17.913%) 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.956402+0000 osd.0 (osd.0) 181 : cluster 0 40.3 deep-scrub starts 2024-04-02T04:51:51.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: cluster 2024-04-02T04:51:48.958361+0000 osd.0 (osd.0) 182 : cluster 0 40.3 deep-scrub ok 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.752888+0000 osd.3 (osd.3) 211 : cluster 0 3.2 scrub starts 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.756574+0000 osd.3 (osd.3) 212 : cluster 0 3.2 scrub ok 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: audit 2024-04-02T04:51:49.760786+0000 mon.a (mon.0) 1874 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.767296+0000 mon.a (mon.0) 1875 : cluster 0 osdmap e270: 8 total, 7 up, 8 in 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: audit 2024-04-02T04:51:49.773015+0000 mon.c (mon.2) 408 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: audit 2024-04-02T04:51:49.784951+0000 mon.c (mon.2) 409 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: audit 2024-04-02T04:51:49.787142+0000 mon.a (mon.0) 1876 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: audit 2024-04-02T04:51:49.788199+0000 mon.a (mon.0) 1877 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: audit 2024-04-02T04:51:49.869384+0000 mon.c (mon.2) 410 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:51.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:50 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.970050+0000 mgr.y (mgr.14538) 327 : cluster 0 pgmap v382: 420 pgs: 1 active+clean+snaptrim, 114 active+undersized, 268 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 273/1524 objects degraded (17.913%) 2024-04-02T04:51:51.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:51.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:51.274 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:51.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:51.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:51.533 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:51.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:51.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:51.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:51.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:51.927 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:51.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:52.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:52.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.982328+0000 osd.0 (osd.0) 183 : cluster 0 40.8 scrub starts 2024-04-02T04:51:52.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:49.983961+0000 osd.0 (osd.0) 184 : cluster 0 40.8 scrub ok 2024-04-02T04:51:52.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:50.149431+0000 osd.2 (osd.2) 139 : cluster 0 6.f scrub starts 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:50.152198+0000 osd.2 (osd.2) 140 : cluster 0 6.f scrub ok 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.786621+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]': finished 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.786803+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:50.794639+0000 mon.a (mon.0) 1880 : cluster 0 osdmap e271: 8 total, 7 up, 8 in 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.795746+0000 mon.c (mon.2) 411 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]: dispatch 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.798003+0000 mon.c (mon.2) 412 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.806373+0000 mon.a (mon.0) 1881 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]: dispatch 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.806814+0000 mon.a (mon.0) 1882 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: audit 2024-04-02T04:51:50.870464+0000 mon.c (mon.2) 413 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.038128+0000 osd.6 (osd.6) 143 : cluster 0 41.12 deep-scrub starts 2024-04-02T04:51:52.060 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:51 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.039442+0000 osd.6 (osd.6) 144 : cluster 0 41.12 deep-scrub ok 2024-04-02T04:51:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:52.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.982328+0000 osd.0 (osd.0) 183 : cluster 0 40.8 scrub starts 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:49.983961+0000 osd.0 (osd.0) 184 : cluster 0 40.8 scrub ok 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:50.149431+0000 osd.2 (osd.2) 139 : cluster 0 6.f scrub starts 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:50.152198+0000 osd.2 (osd.2) 140 : cluster 0 6.f scrub ok 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.786621+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]': finished 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.786803+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:50.794639+0000 mon.a (mon.0) 1880 : cluster 0 osdmap e271: 8 total, 7 up, 8 in 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.795746+0000 mon.c (mon.2) 411 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]: dispatch 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.798003+0000 mon.c (mon.2) 412 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:52.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.806373+0000 mon.a (mon.0) 1881 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]: dispatch 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.806814+0000 mon.a (mon.0) 1882 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: audit 2024-04-02T04:51:50.870464+0000 mon.c (mon.2) 413 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.038128+0000 osd.6 (osd.6) 143 : cluster 0 41.12 deep-scrub starts 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.039442+0000 osd.6 (osd.6) 144 : cluster 0 41.12 deep-scrub ok 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.982328+0000 osd.0 (osd.0) 183 : cluster 0 40.8 scrub starts 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:49.983961+0000 osd.0 (osd.0) 184 : cluster 0 40.8 scrub ok 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:50.149431+0000 osd.2 (osd.2) 139 : cluster 0 6.f scrub starts 2024-04-02T04:51:52.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:50.152198+0000 osd.2 (osd.2) 140 : cluster 0 6.f scrub ok 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.786621+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-23"}]': finished 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.786803+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:50.794639+0000 mon.a (mon.0) 1880 : cluster 0 osdmap e271: 8 total, 7 up, 8 in 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.795746+0000 mon.c (mon.2) 411 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]: dispatch 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.798003+0000 mon.c (mon.2) 412 : audit 1 from='client.? 172.21.15.69:0/3540652019' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.806373+0000 mon.a (mon.0) 1881 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]: dispatch 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.806814+0000 mon.a (mon.0) 1882 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]: dispatch 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: audit 2024-04-02T04:51:50.870464+0000 mon.c (mon.2) 413 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.038128+0000 osd.6 (osd.6) 143 : cluster 0 41.12 deep-scrub starts 2024-04-02T04:51:52.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:51 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.039442+0000 osd.6 (osd.6) 144 : cluster 0 41.12 deep-scrub ok 2024-04-02T04:51:52.186 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:52.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:52.314 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:52.416 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:52.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:52.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:52.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:52.851 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:52.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:52.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:53.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:50.986291+0000 osd.0 (osd.0) 185 : cluster 0 102.3 scrub starts 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:50.994391+0000 osd.0 (osd.0) 186 : cluster 0 102.3 scrub ok 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.739141+0000 osd.3 (osd.3) 213 : cluster 0 41.1 scrub starts 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.741372+0000 osd.3 (osd.3) 214 : cluster 0 41.1 scrub ok 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.787125+0000 mon.a (mon.0) 1883 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:51.789042+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]': finished 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:51.789149+0000 mon.a (mon.0) 1885 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:53.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.794341+0000 mon.a (mon.0) 1886 : cluster 0 osdmap e272: 8 total, 7 up, 8 in 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:51.829828+0000 mon.c (mon.2) 414 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:51.871553+0000 mon.c (mon.2) 415 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:51.971708+0000 mgr.y (mgr.14538) 328 : cluster 0 pgmap v385: 356 pgs: 1 active+clean+snaptrim, 93 active+undersized, 225 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 273/1521 objects degraded (17.949%) 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:52.074982+0000 osd.6 (osd.6) 145 : cluster 0 40.1e scrub starts 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:52.077902+0000 osd.6 (osd.6) 146 : cluster 0 40.1e scrub ok 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:52.267322+0000 mon.a (mon.0) 1887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:52.268001+0000 mon.c (mon.2) 416 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:52.268681+0000 mon.a (mon.0) 1888 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:52.269393+0000 mon.c (mon.2) 417 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: audit 2024-04-02T04:51:52.269739+0000 mon.a (mon.0) 1889 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:52.388877+0000 osd.4 (osd.4) 191 : cluster 0 40.e scrub starts 2024-04-02T04:51:53.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:52 smithi098 bash[23790]: cluster 2024-04-02T04:51:52.390684+0000 osd.4 (osd.4) 192 : cluster 0 40.e scrub ok 2024-04-02T04:51:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:53.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:53.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:51:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:50.986291+0000 osd.0 (osd.0) 185 : cluster 0 102.3 scrub starts 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:50.994391+0000 osd.0 (osd.0) 186 : cluster 0 102.3 scrub ok 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.739141+0000 osd.3 (osd.3) 213 : cluster 0 41.1 scrub starts 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.741372+0000 osd.3 (osd.3) 214 : cluster 0 41.1 scrub ok 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.787125+0000 mon.a (mon.0) 1883 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:51.789042+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]': finished 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:51.789149+0000 mon.a (mon.0) 1885 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:53.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.794341+0000 mon.a (mon.0) 1886 : cluster 0 osdmap e272: 8 total, 7 up, 8 in 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:51.829828+0000 mon.c (mon.2) 414 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:51.871553+0000 mon.c (mon.2) 415 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:51.971708+0000 mgr.y (mgr.14538) 328 : cluster 0 pgmap v385: 356 pgs: 1 active+clean+snaptrim, 93 active+undersized, 225 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 273/1521 objects degraded (17.949%) 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:52.074982+0000 osd.6 (osd.6) 145 : cluster 0 40.1e scrub starts 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:52.077902+0000 osd.6 (osd.6) 146 : cluster 0 40.1e scrub ok 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:52.267322+0000 mon.a (mon.0) 1887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:52.268001+0000 mon.c (mon.2) 416 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:52.268681+0000 mon.a (mon.0) 1888 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:52.269393+0000 mon.c (mon.2) 417 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: audit 2024-04-02T04:51:52.269739+0000 mon.a (mon.0) 1889 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:52.388877+0000 osd.4 (osd.4) 191 : cluster 0 40.e scrub starts 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[21528]: cluster 2024-04-02T04:51:52.390684+0000 osd.4 (osd.4) 192 : cluster 0 40.e scrub ok 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:50.986291+0000 osd.0 (osd.0) 185 : cluster 0 102.3 scrub starts 2024-04-02T04:51:53.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:50.994391+0000 osd.0 (osd.0) 186 : cluster 0 102.3 scrub ok 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.739141+0000 osd.3 (osd.3) 213 : cluster 0 41.1 scrub starts 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.741372+0000 osd.3 (osd.3) 214 : cluster 0 41.1 scrub ok 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.787125+0000 mon.a (mon.0) 1883 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:51.789042+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-23", "mode": "writeback"}]': finished 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:51.789149+0000 mon.a (mon.0) 1885 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-72145-32"}]': finished 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.794341+0000 mon.a (mon.0) 1886 : cluster 0 osdmap e272: 8 total, 7 up, 8 in 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:51.829828+0000 mon.c (mon.2) 414 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:51.871553+0000 mon.c (mon.2) 415 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:51.971708+0000 mgr.y (mgr.14538) 328 : cluster 0 pgmap v385: 356 pgs: 1 active+clean+snaptrim, 93 active+undersized, 225 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 273/1521 objects degraded (17.949%) 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:52.074982+0000 osd.6 (osd.6) 145 : cluster 0 40.1e scrub starts 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:52.077902+0000 osd.6 (osd.6) 146 : cluster 0 40.1e scrub ok 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:52.267322+0000 mon.a (mon.0) 1887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:52.268001+0000 mon.c (mon.2) 416 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:52.268681+0000 mon.a (mon.0) 1888 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:52.269393+0000 mon.c (mon.2) 417 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:53.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: audit 2024-04-02T04:51:52.269739+0000 mon.a (mon.0) 1889 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:53.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:52.388877+0000 osd.4 (osd.4) 191 : cluster 0 40.e scrub starts 2024-04-02T04:51:53.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:52 smithi069 bash[28904]: cluster 2024-04-02T04:51:52.390684+0000 osd.4 (osd.4) 192 : cluster 0 40.e scrub ok 2024-04-02T04:51:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:51:53.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:53.369 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:53.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:53.498 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:53.600 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:53.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:53.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:53.750 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:53.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:53.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: cluster 2024-04-02T04:51:52.802897+0000 mon.a (mon.0) 1890 : cluster 3 Health check update: Degraded data redundancy: 273/1521 objects degraded (17.949%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:53.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.818697+0000 mon.a (mon.0) 1891 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:53.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: cluster 2024-04-02T04:51:52.827694+0000 mon.a (mon.0) 1892 : cluster 0 osdmap e273: 8 total, 7 up, 8 in 2024-04-02T04:51:53.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.835826+0000 mon.c (mon.2) 418 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.840837+0000 mon.a (mon.0) 1893 : audit 1 from='client.? 172.21.15.69:0/3925174490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:53.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.841208+0000 mon.a (mon.0) 1894 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.872623+0000 mon.c (mon.2) 419 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.873423+0000 mon.c (mon.2) 420 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:52.873775+0000 mon.a (mon.0) 1895 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[21528]: audit 2024-04-02T04:51:53.108572+0000 mgr.y (mgr.14538) 329 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: cluster 2024-04-02T04:51:52.802897+0000 mon.a (mon.0) 1890 : cluster 3 Health check update: Degraded data redundancy: 273/1521 objects degraded (17.949%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.818697+0000 mon.a (mon.0) 1891 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: cluster 2024-04-02T04:51:52.827694+0000 mon.a (mon.0) 1892 : cluster 0 osdmap e273: 8 total, 7 up, 8 in 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.835826+0000 mon.c (mon.2) 418 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.840837+0000 mon.a (mon.0) 1893 : audit 1 from='client.? 172.21.15.69:0/3925174490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:53.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.841208+0000 mon.a (mon.0) 1894 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:53.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.872623+0000 mon.c (mon.2) 419 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:53.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.873423+0000 mon.c (mon.2) 420 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:53.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:52.873775+0000 mon.a (mon.0) 1895 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:53.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:53 smithi069 bash[28904]: audit 2024-04-02T04:51:53.108572+0000 mgr.y (mgr.14538) 329 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:53.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:53.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:54.000 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:54.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:54.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: cluster 2024-04-02T04:51:52.802897+0000 mon.a (mon.0) 1890 : cluster 3 Health check update: Degraded data redundancy: 273/1521 objects degraded (17.949%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:54.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.818697+0000 mon.a (mon.0) 1891 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-72145-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:51:54.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: cluster 2024-04-02T04:51:52.827694+0000 mon.a (mon.0) 1892 : cluster 0 osdmap e273: 8 total, 7 up, 8 in 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.835826+0000 mon.c (mon.2) 418 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.840837+0000 mon.a (mon.0) 1893 : audit 1 from='client.? 172.21.15.69:0/3925174490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.841208+0000 mon.a (mon.0) 1894 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.872623+0000 mon.c (mon.2) 419 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.873423+0000 mon.c (mon.2) 420 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:52.873775+0000 mon.a (mon.0) 1895 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:51:54.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:53 smithi098 bash[23790]: audit 2024-04-02T04:51:53.108572+0000 mgr.y (mgr.14538) 329 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:51:54.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:54.256 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:54.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:54.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:54.511 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:54.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:54.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:54.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.013609+0000 osd.0 (osd.0) 187 : cluster 0 102.12 scrub starts 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.016573+0000 osd.0 (osd.0) 188 : cluster 0 102.12 scrub ok 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.142731+0000 osd.2 (osd.2) 141 : cluster 0 102.1a scrub starts 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.146918+0000 osd.2 (osd.2) 142 : cluster 0 102.1a scrub ok 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.756738+0000 osd.3 (osd.3) 215 : cluster 0 40.7 scrub starts 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.758415+0000 osd.3 (osd.3) 216 : cluster 0 40.7 scrub ok 2024-04-02T04:51:54.882 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:53.873680+0000 mon.c (mon.2) 421 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:53.900494+0000 mon.a (mon.0) 1896 : audit 1 from='client.? 172.21.15.69:0/3925174490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:53.900604+0000 mon.a (mon.0) 1897 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:53.912049+0000 mon.c (mon.2) 422 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.972579+0000 mgr.y (mgr.14538) 330 : cluster 0 pgmap v388: 388 pgs: 32 creating+peering, 6 active+clean+snaptrim, 92 active+undersized, 220 active+clean, 38 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s; 275/1518 objects degraded (18.116%) 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:54.350335+0000 mon.a (mon.0) 1898 : cluster 0 osdmap e274: 8 total, 7 up, 8 in 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:54.353280+0000 mon.a (mon.0) 1899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:54.801921+0000 mon.a (mon.0) 1900 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:54.803976+0000 mon.a (mon.0) 1901 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:54.807157+0000 mon.a (mon.0) 1902 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:54.807418+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]': finished 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: cluster 2024-04-02T04:51:54.816906+0000 mon.a (mon.0) 1904 : cluster 0 osdmap e275: 8 total, 7 up, 8 in 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:54.818241+0000 mon.c (mon.2) 423 : audit 1 from='client.? 172.21.15.69:0/97625888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:54.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:54 smithi098 bash[23790]: audit 2024-04-02T04:51:54.819858+0000 mon.a (mon.0) 1905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:54.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:54.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:55.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:55.146 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:55.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.013609+0000 osd.0 (osd.0) 187 : cluster 0 102.12 scrub starts 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.016573+0000 osd.0 (osd.0) 188 : cluster 0 102.12 scrub ok 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.142731+0000 osd.2 (osd.2) 141 : cluster 0 102.1a scrub starts 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.146918+0000 osd.2 (osd.2) 142 : cluster 0 102.1a scrub ok 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.756738+0000 osd.3 (osd.3) 215 : cluster 0 40.7 scrub starts 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.758415+0000 osd.3 (osd.3) 216 : cluster 0 40.7 scrub ok 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:53.873680+0000 mon.c (mon.2) 421 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:53.900494+0000 mon.a (mon.0) 1896 : audit 1 from='client.? 172.21.15.69:0/3925174490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:53.900604+0000 mon.a (mon.0) 1897 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:53.912049+0000 mon.c (mon.2) 422 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.972579+0000 mgr.y (mgr.14538) 330 : cluster 0 pgmap v388: 388 pgs: 32 creating+peering, 6 active+clean+snaptrim, 92 active+undersized, 220 active+clean, 38 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s; 275/1518 objects degraded (18.116%) 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:54.350335+0000 mon.a (mon.0) 1898 : cluster 0 osdmap e274: 8 total, 7 up, 8 in 2024-04-02T04:51:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:54.353280+0000 mon.a (mon.0) 1899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:54.801921+0000 mon.a (mon.0) 1900 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:54.803976+0000 mon.a (mon.0) 1901 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:54.807157+0000 mon.a (mon.0) 1902 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:54.807418+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]': finished 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: cluster 2024-04-02T04:51:54.816906+0000 mon.a (mon.0) 1904 : cluster 0 osdmap e275: 8 total, 7 up, 8 in 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:54.818241+0000 mon.c (mon.2) 423 : audit 1 from='client.? 172.21.15.69:0/97625888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[21528]: audit 2024-04-02T04:51:54.819858+0000 mon.a (mon.0) 1905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.013609+0000 osd.0 (osd.0) 187 : cluster 0 102.12 scrub starts 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.016573+0000 osd.0 (osd.0) 188 : cluster 0 102.12 scrub ok 2024-04-02T04:51:55.262 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.142731+0000 osd.2 (osd.2) 141 : cluster 0 102.1a scrub starts 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.146918+0000 osd.2 (osd.2) 142 : cluster 0 102.1a scrub ok 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.756738+0000 osd.3 (osd.3) 215 : cluster 0 40.7 scrub starts 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.758415+0000 osd.3 (osd.3) 216 : cluster 0 40.7 scrub ok 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:53.873680+0000 mon.c (mon.2) 421 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:53.900494+0000 mon.a (mon.0) 1896 : audit 1 from='client.? 172.21.15.69:0/3925174490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:53.900604+0000 mon.a (mon.0) 1897 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:53.912049+0000 mon.c (mon.2) 422 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.972579+0000 mgr.y (mgr.14538) 330 : cluster 0 pgmap v388: 388 pgs: 32 creating+peering, 6 active+clean+snaptrim, 92 active+undersized, 220 active+clean, 38 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s; 275/1518 objects degraded (18.116%) 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:54.350335+0000 mon.a (mon.0) 1898 : cluster 0 osdmap e274: 8 total, 7 up, 8 in 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:54.353280+0000 mon.a (mon.0) 1899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]: dispatch 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:54.801921+0000 mon.a (mon.0) 1900 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:54.803976+0000 mon.a (mon.0) 1901 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:54.807157+0000 mon.a (mon.0) 1902 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-72145-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:55.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:54.807418+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-23"}]': finished 2024-04-02T04:51:55.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: cluster 2024-04-02T04:51:54.816906+0000 mon.a (mon.0) 1904 : cluster 0 osdmap e275: 8 total, 7 up, 8 in 2024-04-02T04:51:55.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:54.818241+0000 mon.c (mon.2) 423 : audit 1 from='client.? 172.21.15.69:0/97625888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:55.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:54 smithi069 bash[28904]: audit 2024-04-02T04:51:54.819858+0000 mon.a (mon.0) 1905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:55.272 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:55.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:55.398 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:55.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:55.525 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:55.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:55.653 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:55.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:55.779 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:55.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:55.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:56.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:56.032 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- 172.21.15.69:0/1562640483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4806a1d0 msgr2=0x7f0c4806a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/1562640483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4806a1d0 0x7f0c4806a5b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0c380099c0 tx=0x7f0c38009cd0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- 172.21.15.69:0/1562640483 shutdown_connections 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/1562640483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c48107580 0x7f0c4810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/1562640483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4806a1d0 0x7f0c4806a5b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/1562640483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4806ab80 0x7f0c48106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- 172.21.15.69:0/1562640483 >> 172.21.15.69:0/1562640483 conn(0x7f0c480771e0 msgr2=0x7f0c480775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- 172.21.15.69:0/1562640483 shutdown_connections 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- 172.21.15.69:0/1562640483 wait complete. 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 librados: starting msgr at 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 librados: starting objecter 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 Processor -- start 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- start start 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 librados: setting wanted keys 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 librados: calling monclient init 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 0x7f0c48198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4806ab80 0x7f0c48198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c48107580 0x7f0c4819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c48111ac0 con 0x7f0c4806a1d0 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c48111940 con 0x7f0c4806ab80 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4ed92700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0c48111c40 con 0x7f0c48107580 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4cb2e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 0x7f0c48198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4cb2e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 0x7f0c48198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49498/0 (socket says 172.21.15.69:49498) 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4d32f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c48107580 0x7f0c4819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c4cb2e700 1 -- 172.21.15.69:0/2262895501 learned_addr learned my addr 172.21.15.69:0/2262895501 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.059+0000 7f0c47fff700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4806ab80 0x7f0c48198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4d32f700 1 -- 172.21.15.69:0/2262895501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4806ab80 msgr2=0x7f0c48198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4d32f700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4806ab80 0x7f0c48198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4d32f700 1 -- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 msgr2=0x7f0c48198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4d32f700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 0x7f0c48198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4d32f700 1 -- 172.21.15.69:0/2262895501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4819d060 con 0x7f0c48107580 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4cb2e700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 0x7f0c48198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4d32f700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c48107580 0x7f0c4819c920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0c40007f10 tx=0x7f0c4000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 10 client.?.objecter ms_handle_connect 0x7f0c48107580 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c40004db0 con 0x7f0c48107580 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0c40003a00 con 0x7f0c48107580 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c4819d350 con 0x7f0c48107580 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4ed92700 10 client.26183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c481a4bd0 con 0x7f0c48107580 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4ed92700 1 librados: init done 2024-04-02T04:51:56.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c400044c0 con 0x7f0c48107580 2024-04-02T04:51:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c28005120 con 0x7f0c48107580 2024-04-02T04:51:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0c4001c020 con 0x7f0c48107580 2024-04-02T04:51:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0c3406f5e0 0x7f0c34071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(276..276 src has 74..276) ==== 12080+0+0 (secure 0 0 0) 0x7f0c4008e8c0 con 0x7f0c48107580 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 10 client.26183.objecter ms_dispatch 0x7f0c48001550 osd_map(276..276 src has 74..276) 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 3 client.26183.objecter handle_osd_map got epochs [276,276] > 0 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 3 client.26183.objecter handle_osd_map decoding full epoch 276 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4cb2e700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0c3406f5e0 0x7f0c34071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 20 client.26183.objecter dump_active .. 0 homeless 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c4cb2e700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0c3406f5e0 0x7f0c34071aa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0c38003860 tx=0x7f0c38005fb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.063+0000 7f0c45ffb700 10 client.26183.objecter ms_handle_connect 0x7f0c3406f5e0 2024-04-02T04:51:56.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.067+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c40010070 con 0x7f0c48107580 2024-04-02T04:51:56.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:56.155 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:56.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:56.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.303+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd rm-pg-upmap", "pgid": "102.7"} v 0) -- 0x7f0c28005410 con 0x7f0c48107580 2024-04-02T04:51:56.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 -- 172.21.15.69:0/1623904617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 msgr2=0x7faea8106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 --2- 172.21.15.69:0/1623904617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea8106ef0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fae980099c0 tx=0x7fae98009cd0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 -- 172.21.15.69:0/1623904617 shutdown_connections 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 --2- 172.21.15.69:0/1623904617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea8107620 0x7faea810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 --2- 172.21.15.69:0/1623904617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faea806a1d0 0x7faea806a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 --2- 172.21.15.69:0/1623904617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea8106ef0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 -- 172.21.15.69:0/1623904617 >> 172.21.15.69:0/1623904617 conn(0x7faea80771e0 msgr2=0x7faea80775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 -- 172.21.15.69:0/1623904617 shutdown_connections 2024-04-02T04:51:56.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.391+0000 7faeafac1700 1 -- 172.21.15.69:0/1623904617 wait complete. 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 librados: starting msgr at 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 librados: starting objecter 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 Processor -- start 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 -- start start 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 librados: setting wanted keys 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 librados: calling monclient init 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea806a1d0 0x7faea8198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faea8107620 0x7faea819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faea8111b80 con 0x7faea8107620 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faea8111a00 con 0x7faea806a1d0 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faea8111d00 con 0x7faea806ab80 2024-04-02T04:51:56.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:43972/0 (socket says 172.21.15.69:43972) 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 -- 172.21.15.69:0/3407373028 learned_addr learned my addr 172.21.15.69:0/3407373028 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead85d700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea806a1d0 0x7faea8198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeae05e700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faea8107620 0x7faea819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 -- 172.21.15.69:0/3407373028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea806a1d0 msgr2=0x7faea8198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea806a1d0 0x7faea8198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 -- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faea8107620 msgr2=0x7faea819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faea8107620 0x7faea819ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 -- 172.21.15.69:0/3407373028 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea819d170 con 0x7faea806ab80 2024-04-02T04:51:56.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faead05c700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea81986a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fae98009f50 tx=0x7fae980057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7fae9effd700 10 client.?.objecter ms_handle_connect 0x7faea806ab80 2024-04-02T04:51:56.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7fae9effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:56.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98027070 con 0x7faea806ab80 2024-04-02T04:51:56.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 -- 172.21.15.69:0/3407373028 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea819d400 con 0x7faea806ab80 2024-04-02T04:51:56.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fae98003fd0 con 0x7faea806ab80 2024-04-02T04:51:56.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 10 client.26189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:56.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98022d20 con 0x7faea806ab80 2024-04-02T04:51:56.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 -- 172.21.15.69:0/3407373028 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faea81011a0 con 0x7faea806ab80 2024-04-02T04:51:56.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.395+0000 7faeafac1700 1 librados: init done 2024-04-02T04:51:56.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fae9802a070 con 0x7faea806ab80 2024-04-02T04:51:56.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae90005120 con 0x7faea806ab80 2024-04-02T04:51:56.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fae9406f4b0 0x7fae94071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7faead85d700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fae9406f4b0 0x7fae94071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(276..276 src has 74..276) ==== 12080+0+0 (secure 0 0 0) 0x7fae98090490 con 0x7faea806ab80 2024-04-02T04:51:56.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 10 client.26189.objecter ms_dispatch 0x7faea8001550 osd_map(276..276 src has 74..276) 2024-04-02T04:51:56.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 3 client.26189.objecter handle_osd_map got epochs [276,276] > 0 2024-04-02T04:51:56.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 3 client.26189.objecter handle_osd_map decoding full epoch 276 2024-04-02T04:51:56.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 20 client.26189.objecter dump_active .. 0 homeless 2024-04-02T04:51:56.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7faead85d700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fae9406f4b0 0x7fae94071970 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faea4005950 tx=0x7faea400a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.399+0000 7fae9effd700 10 client.26189.objecter ms_handle_connect 0x7fae9406f4b0 2024-04-02T04:51:56.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.964454+0000 osd.0 (osd.0) 189 : cluster 0 6.10 scrub starts 2024-04-02T04:51:56.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:53.965807+0000 osd.0 (osd.0) 190 : cluster 0 6.10 scrub ok 2024-04-02T04:51:56.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:54.720038+0000 osd.3 (osd.3) 217 : cluster 0 4.10 scrub starts 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:54.722040+0000 osd.3 (osd.3) 218 : cluster 0 4.10 scrub ok 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: audit 2024-04-02T04:51:54.874739+0000 mon.c (mon.2) 424 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.331707+0000 osd.1 (osd.1) 197 : cluster 0 13.c scrub starts 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.332902+0000 osd.1 (osd.1) 198 : cluster 0 13.c scrub ok 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: audit 2024-04-02T04:51:55.808991+0000 mon.a (mon.0) 1906 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.812575+0000 mon.a (mon.0) 1907 : cluster 0 osdmap e276: 8 total, 7 up, 8 in 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.964454+0000 osd.0 (osd.0) 189 : cluster 0 6.10 scrub starts 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:53.965807+0000 osd.0 (osd.0) 190 : cluster 0 6.10 scrub ok 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:54.720038+0000 osd.3 (osd.3) 217 : cluster 0 4.10 scrub starts 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:54.722040+0000 osd.3 (osd.3) 218 : cluster 0 4.10 scrub ok 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: audit 2024-04-02T04:51:54.874739+0000 mon.c (mon.2) 424 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:56.399 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.331707+0000 osd.1 (osd.1) 197 : cluster 0 13.c scrub starts 2024-04-02T04:51:56.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.332902+0000 osd.1 (osd.1) 198 : cluster 0 13.c scrub ok 2024-04-02T04:51:56.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: audit 2024-04-02T04:51:55.808991+0000 mon.a (mon.0) 1906 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:56.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:56 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.812575+0000 mon.a (mon.0) 1907 : cluster 0 osdmap e276: 8 total, 7 up, 8 in 2024-04-02T04:51:56.409 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:56.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.411+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fae98013070 con 0x7faea806ab80 2024-04-02T04:51:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- 172.21.15.69:0/3001232977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 msgr2=0x7f96641028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- 172.21.15.69:0/3001232977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 0x7f96641028a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9654009920 tx=0x7f9654009c30 comp rx=0 tx=0).stop 2024-04-02T04:51:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- 172.21.15.69:0/3001232977 shutdown_connections 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- 172.21.15.69:0/3001232977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96641075a0 0x7f966410f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- 172.21.15.69:0/3001232977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 0x7f96641028a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- 172.21.15.69:0/3001232977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664102e70 0x7f9664106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- 172.21.15.69:0/3001232977 >> 172.21.15.69:0/3001232977 conn(0x7f96640fdd10 msgr2=0x7f9664100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- 172.21.15.69:0/3001232977 shutdown_connections 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- 172.21.15.69:0/3001232977 wait complete. 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 librados: starting msgr at 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 librados: starting objecter 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 Processor -- start 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- start start 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 librados: setting wanted keys 2024-04-02T04:51:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 librados: calling monclient init 2024-04-02T04:51:56.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 0x7f9664074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9664102e70 0x7f9664074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96641075a0 0x7f9664074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9664111aa0 con 0x7f96641075a0 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9664111920 con 0x7f9664102e70 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f966a299700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9664111c20 con 0x7f96641024c0 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96641075a0 0x7f9664074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96641075a0 0x7f9664074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49530/0 (socket says 172.21.15.69:49530) 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 -- 172.21.15.69:0/2125942760 learned_addr learned my addr 172.21.15.69:0/2125942760 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 0x7f9664074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9664102e70 0x7f9664074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 -- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 msgr2=0x7f9664074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96641024c0 0x7f9664074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 -- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9664102e70 msgr2=0x7f9664074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9664102e70 0x7f9664074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96641a66a0 con 0x7f96641075a0 2024-04-02T04:51:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9664102e70 0x7f9664074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:51:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96641075a0 0x7f9664074fc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f965c00ada0 tx=0x7f965c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f96617fa700 10 client.?.objecter ms_handle_connect 0x7f96641075a0 2024-04-02T04:51:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.463+0000 7f96617fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:51:56.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:56.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.964454+0000 osd.0 (osd.0) 189 : cluster 0 6.10 scrub starts 2024-04-02T04:51:56.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:53.965807+0000 osd.0 (osd.0) 190 : cluster 0 6.10 scrub ok 2024-04-02T04:51:56.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:54.720038+0000 osd.3 (osd.3) 217 : cluster 0 4.10 scrub starts 2024-04-02T04:51:56.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:54.722040+0000 osd.3 (osd.3) 218 : cluster 0 4.10 scrub ok 2024-04-02T04:51:56.524 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: audit 2024-04-02T04:51:54.874739+0000 mon.c (mon.2) 424 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:56.524 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.331707+0000 osd.1 (osd.1) 197 : cluster 0 13.c scrub starts 2024-04-02T04:51:56.524 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.332902+0000 osd.1 (osd.1) 198 : cluster 0 13.c scrub ok 2024-04-02T04:51:56.524 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: audit 2024-04-02T04:51:55.808991+0000 mon.a (mon.0) 1906 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:56.524 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:56 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.812575+0000 mon.a (mon.0) 1907 : cluster 0 osdmap e276: 8 total, 7 up, 8 in 2024-04-02T04:51:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:56.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:56.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.643+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fae90005410 con 0x7faea806ab80 2024-04-02T04:51:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:56.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.687+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c017020 con 0x7f96641075a0 2024-04-02T04:51:56.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.687+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96641a6990 con 0x7f96641075a0 2024-04-02T04:51:56.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.687+0000 7f966a299700 10 client.16110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:56.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.687+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96641a6f20 con 0x7f96641075a0 2024-04-02T04:51:56.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.687+0000 7f966a299700 1 librados: init done 2024-04-02T04:51:56.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f966404ff00 con 0x7f96641075a0 2024-04-02T04:51:56.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f965c00ce00 con 0x7f96641075a0 2024-04-02T04:51:56.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c01c870 con 0x7f96641075a0 2024-04-02T04:51:56.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f965c00c980 con 0x7f96641075a0 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f964c06f5e0 0x7f964c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(276..276 src has 74..276) ==== 12080+0+0 (secure 0 0 0) 0x7f965c08f6e0 con 0x7f96641075a0 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 osd_map(276..276 src has 74..276) 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 3 client.16110.objecter handle_osd_map got epochs [276,276] > 0 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 3 client.16110.objecter handle_osd_map decoding full epoch 276 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f964c06f5e0 0x7f964c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 20 client.16110.objecter dump_active .. 0 homeless 2024-04-02T04:51:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v276) ==== 54+0+16 (secure 0 0 0) 0x7f965c021020 con 0x7f96641075a0 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f964c06f5e0 0x7f964c071aa0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f965400c010 tx=0x7f9654019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.691+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f964c06f5e0 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 10 client.16110.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 1 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9664062e50 0x7f9664063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 1 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 0 1 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664075a70 osd=0 4 2024-04-02T04:51:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 10 client.16110.objecter _send_command 1 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96641aa340 con 0x7f9664062e50 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9664062e50 0x7f9664063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f9664062e50 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9664062e50 0x7f9664063230 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96641aa340 con 0x7f9664062e50 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 1: 0 ) 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96617fa700 10 client.16110.objecter _finish_command 1 = system:0 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f9664075a70 osd=0 4 2024-04-02T04:51:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.695+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 0 1 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 10 client.16110.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 2 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 2 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f9664075a70 osd=0 3 2024-04-02T04:51:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 0 2 2024-04-02T04:51:56.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664075a70 osd=0 4 2024-04-02T04:51:56.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 10 client.16110.objecter _send_command 2 2024-04-02T04:51:56.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96640628d0 con 0x7f9664062e50 2024-04-02T04:51:56.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96640628d0 con 0x7f9664062e50 2024-04-02T04:51:56.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f96617fa700 10 client.16110.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f9664075a70 osd=0 4 2024-04-02T04:51:56.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 0 2 2024-04-02T04:51:56.711 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 10 client.16110.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9664109da0 0x7f966402c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 1 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664062cd0 osd=1 4 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 10 client.16110.objecter _send_command 3 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.711+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f96641adca0 con 0x7f9664109da0 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9664109da0 0x7f966402c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f9664109da0 2024-04-02T04:51:56.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9664109da0 0x7f966402c9b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96641adca0 con 0x7f9664109da0 2024-04-02T04:51:56.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 3: 0 ) 2024-04-02T04:51:56.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f96617fa700 10 client.16110.objecter _finish_command 3 = system:0 2024-04-02T04:51:56.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f9664062cd0 osd=1 4 2024-04-02T04:51:56.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.715+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 1 3 2024-04-02T04:51:56.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 10 client.16110.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 4 2024-04-02T04:51:56.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 4 2024-04-02T04:51:56.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f9664062cd0 osd=1 3 2024-04-02T04:51:56.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 1 4 2024-04-02T04:51:56.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f9664062cd0 osd=1 4 2024-04-02T04:51:56.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 10 client.16110.objecter _send_command 4 2024-04-02T04:51:56.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9664109310 con 0x7f9664109da0 2024-04-02T04:51:56.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9664109310 con 0x7f9664109da0 2024-04-02T04:51:56.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96617fa700 10 client.16110.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f9664062cd0 osd=1 4 2024-04-02T04:51:56.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 1 4 2024-04-02T04:51:56.731 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 10 client.16110.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 5 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f96641ac500 0x7f96641ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 5 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 2 5 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9990 osd=2 4 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 10 client.16110.objecter _send_command 5 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f96641aa010 con 0x7f96641ac500 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f96641ac500 0x7f96641ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f96641ac500 2024-04-02T04:51:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.731+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f96641ac500 0x7f96641ac920 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.735+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96641aa010 con 0x7f96641ac500 2024-04-02T04:51:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.735+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 5: 0 ) 2024-04-02T04:51:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.735+0000 7f96617fa700 10 client.16110.objecter _finish_command 5 = system:0 2024-04-02T04:51:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.735+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641a9990 osd=2 4 2024-04-02T04:51:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.735+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 2 5 2024-04-02T04:51:56.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 10 client.16110.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 6 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 6 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641a9990 osd=2 3 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 2 6 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9990 osd=2 4 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 10 client.16110.objecter _send_command 6 2024-04-02T04:51:56.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96641a9410 con 0x7f96641ac500 2024-04-02T04:51:56.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96641a9410 con 0x7f96641ac500 2024-04-02T04:51:56.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 10 client.16110.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641a9990 osd=2 4 2024-04-02T04:51:56.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 2 6 2024-04-02T04:51:56.750 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:51:56.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 10 client.16110.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 7 2024-04-02T04:51:56.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f96641afe60 0x7f96641b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 7 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 3 7 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9810 osd=3 4 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 10 client.16110.objecter _send_command 7 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f96641b50b0 con 0x7f96641afe60 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f96641afe60 0x7f96641b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f96641afe60 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f96641afe60 0x7f96641b2250 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96641b50b0 con 0x7f96641afe60 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 7: 0 ) 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 10 client.16110.objecter _finish_command 7 = system:0 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641a9810 osd=3 4 2024-04-02T04:51:56.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.751+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 3 7 2024-04-02T04:51:56.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 10 client.16110.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 8 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 8 2024-04-02T04:51:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641a9810 osd=3 3 2024-04-02T04:51:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 3 8 2024-04-02T04:51:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641a9810 osd=3 4 2024-04-02T04:51:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 10 client.16110.objecter _send_command 8 2024-04-02T04:51:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96641ad170 con 0x7f96641afe60 2024-04-02T04:51:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96641ad170 con 0x7f96641afe60 2024-04-02T04:51:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f96617fa700 10 client.16110.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641a9810 osd=3 4 2024-04-02T04:51:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 3 8 2024-04-02T04:51:56.770 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 10 client.16110.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 9 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f966402dbd0 0x7f96641b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 9 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 4 9 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641ad170 osd=4 4 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 10 client.16110.objecter _send_command 9 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f96641b8a10 con 0x7f966402dbd0 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.771+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f966402dbd0 0x7f96641b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f966402dbd0 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f9663fff700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f966402dbd0 0x7f96641b3910 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96641b8a10 con 0x7f966402dbd0 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 9: 0 ) 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f96617fa700 10 client.16110.objecter _finish_command 9 = system:0 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641ad170 osd=4 4 2024-04-02T04:51:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.775+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 4 9 2024-04-02T04:51:56.782 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 10 client.16110.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 10 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 10 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641ad170 osd=4 3 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 4 10 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641ad170 osd=4 4 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 10 client.16110.objecter _send_command 10 2024-04-02T04:51:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.791+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96641b3e50 con 0x7f966402dbd0 2024-04-02T04:51:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96641b3e50 con 0x7f966402dbd0 2024-04-02T04:51:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 10 client.16110.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641ad170 osd=4 4 2024-04-02T04:51:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 4 10 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 10 client.16110.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 11 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f96641b44d0 0x7f96641b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 11 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 5 11 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641b3e50 osd=5 4 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 10 client.16110.objecter _send_command 11 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f96641bbba0 con 0x7f96641b44d0 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f96641b44d0 0x7f96641b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f96641b44d0 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96637fe700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f96641b44d0 0x7f96641b4930 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96641bbba0 con 0x7f96641b44d0 2024-04-02T04:51:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 11: 0 ) 2024-04-02T04:51:56.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 10 client.16110.objecter _finish_command 11 = system:0 2024-04-02T04:51:56.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641b3e50 osd=5 4 2024-04-02T04:51:56.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.795+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 5 11 2024-04-02T04:51:56.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.811+0000 7f0c45ffb700 1 -- 172.21.15.69:0/2262895501 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]=0 clear 102.7 pg_upmap mapping v277) ==== 108+0+0 (secure 0 0 0) 0x7f0c4001c310 con 0x7f0c48107580 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 10 client.16110.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign -1 12 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 20 client.16110.objecter _get_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 15 client.16110.objecter _session_command_op_remove -1 12 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 20 client.16110.objecter get_session s=0x7f96641b3e50 osd=5 3 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 15 client.16110.objecter _session_command_op_assign 5 12 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 20 client.16110.objecter put_session s=0x7f96641b3e50 osd=5 4 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 10 client.16110.objecter _send_command 12 2024-04-02T04:51:56.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f966a299700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96641b9280 con 0x7f96641b44d0 2024-04-02T04:51:56.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.815+0000 7f0c4ed92700 10 librados: watch_flush enter 2024-04-02T04:51:56.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 10 librados: watch_flush exit 2024-04-02T04:51:56.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 20 client.26183.objecter shutdown clearing up homeless session... 2024-04-02T04:51:56.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 10 client.26183.objecter successfully canceled tick 2024-04-02T04:51:56.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0c3406f5e0 msgr2=0x7f0c34071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0c3406f5e0 0x7f0c34071aa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0c38003860 tx=0x7f0c38005fb0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c48107580 msgr2=0x7f0c4819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c48107580 0x7f0c4819c920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0c40007f10 tx=0x7f0c4000d3b0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 shutdown_connections 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0c3406f5e0 0x7f0c34071aa0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c48107580 0x7f0c4819c920 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c4806a1d0 0x7f0c48198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 --2- 172.21.15.69:0/2262895501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c4806ab80 0x7f0c48198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 >> 172.21.15.69:0/2262895501 conn(0x7f0c480771e0 msgr2=0x7f0c480ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 shutdown_connections 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 -- 172.21.15.69:0/2262895501 wait complete. 2024-04-02T04:51:56.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7f0c4ed92700 1 librados: shutdown 2024-04-02T04:51:56.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.819+0000 7fae9effd700 1 -- 172.21.15.69:0/3407373028 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v277) ==== 87+0+0 (secure 0 0 0) 0x7fae9805c8b0 con 0x7faea806ab80 2024-04-02T04:51:56.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 10 librados: watch_flush enter 2024-04-02T04:51:56.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 10 librados: watch_flush exit 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 20 client.26189.objecter shutdown clearing up homeless session... 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 10 client.26189.objecter successfully canceled tick 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fae9406f4b0 msgr2=0x7fae94071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fae9406f4b0 0x7fae94071970 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faea4005950 tx=0x7faea400a300 comp rx=0 tx=0).stop 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 msgr2=0x7faea81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea81986a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fae98009f50 tx=0x7fae980057c0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 shutdown_connections 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fae9406f4b0 0x7fae94071970 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faea8107620 0x7faea819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea806a1d0 0x7faea8198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 --2- 172.21.15.69:0/3407373028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faea806ab80 0x7faea81986a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:51:56.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.823+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 >> 172.21.15.69:0/3407373028 conn(0x7faea80771e0 msgr2=0x7faea80ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:51:56.824 INFO:teuthology.orchestra.run.smithi069.stderr:clear 102.7 pg_upmap mapping 2024-04-02T04:51:56.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.827+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 shutdown_connections 2024-04-02T04:51:56.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.827+0000 7fae9cff9700 1 -- 172.21.15.69:0/3407373028 wait complete. 2024-04-02T04:51:56.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.827+0000 7fae9cff9700 1 librados: shutdown 2024-04-02T04:51:56.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.831+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96641b9280 con 0x7f96641b44d0 2024-04-02T04:51:56.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f96617fa700 10 client.16110.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f96641b3e50 osd=5 4 2024-04-02T04:51:56.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 5 12 2024-04-02T04:51:56.832 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:51:56.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 10 client.16110.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 15 client.16110.objecter _session_command_op_assign -1 13 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f963c002ff0 0x7f963c0054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 20 client.16110.objecter _get_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 20 client.16110.objecter put_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 20 client.16110.objecter _get_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 15 client.16110.objecter _session_command_op_remove -1 13 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 20 client.16110.objecter get_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 15 client.16110.objecter _session_command_op_assign 6 13 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 20 client.16110.objecter put_session s=0x7f963c0028e0 osd=6 4 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 10 client.16110.objecter _send_command 13 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f964affd700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f963c008530 con 0x7f963c002ff0 2024-04-02T04:51:56.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f963c002ff0 0x7f963c0054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:51:56.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f96617fa700 10 client.16110.objecter ms_handle_connect 0x7f963c002ff0 2024-04-02T04:51:56.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.835+0000 7f9668836700 1 --2- 172.21.15.69:0/2125942760 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f963c002ff0 0x7f963c0054b0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:51:56.835 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:51:56.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.839+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f963c008530 con 0x7f963c002ff0 2024-04-02T04:51:56.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.839+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 13: 0 ) 2024-04-02T04:51:56.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.839+0000 7f96617fa700 10 client.16110.objecter _finish_command 13 = system:0 2024-04-02T04:51:56.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.839+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f963c0028e0 osd=6 4 2024-04-02T04:51:56.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.839+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 6 13 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 10 client.16110.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 15 client.16110.objecter _session_command_op_assign -1 14 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 20 client.16110.objecter _get_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 20 client.16110.objecter put_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 20 client.16110.objecter _get_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 15 client.16110.objecter _session_command_op_remove -1 14 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 20 client.16110.objecter get_session s=0x7f963c0028e0 osd=6 3 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 15 client.16110.objecter _session_command_op_assign 6 14 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 20 client.16110.objecter put_session s=0x7f963c0028e0 osd=6 4 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 10 client.16110.objecter _send_command 14 2024-04-02T04:51:56.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.863+0000 7f964affd700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f963c001700 con 0x7f963c002ff0 2024-04-02T04:51:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f963c001700 con 0x7f963c002ff0 2024-04-02T04:51:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:51:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 10 client.16110.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:51:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 20 client.16110.objecter put_session s=0x7f963c0028e0 osd=6 4 2024-04-02T04:51:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 15 client.16110.objecter _session_command_op_remove 6 14 2024-04-02T04:51:56.864 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f964affd700 10 client.16110.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f964affd700 15 client.16110.objecter _session_command_op_assign -1 15 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f964affd700 20 client.16110.objecter _get_session osd=-1 returning homeless 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f964affd700 10 client.16110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f964affd700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=277}) -- 0x7f963c0086a0 con 0x7f96641075a0 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f964affd700 1 -- 172.21.15.69:0/2125942760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f963c008a80 con 0x7f96641075a0 2024-04-02T04:51:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 7 ==== osd_map(277..277 src has 74..277) ==== 1392+0+0 (secure 0 0 0) 0x7f965c08c3f0 con 0x7f96641075a0 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 10 client.16110.objecter ms_dispatch 0x7f9664001550 osd_map(277..277 src has 74..277) 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 3 client.16110.objecter handle_osd_map got epochs [277,277] > 276 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 3 client.16110.objecter handle_osd_map decoding incremental epoch 277 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 10 client.16110.objecter checking command 15 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 10 client.16110.objecter _check_command_map_dne tid 15 current 277 map_dne_bound 0 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 20 client.16110.objecter dump_active .. 1 homeless 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96617fa700 1 -- 172.21.15.69:0/2125942760 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7f965c05ad20 con 0x7f96641075a0 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96627fc700 10 client.16110.objecter _check_command_map_dne tid 15 current 277 map_dne_bound 277 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96627fc700 10 client.16110.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:51:56.867+0000 7f96627fc700 15 client.16110.objecter _session_command_op_remove -1 15 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:51:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:51:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:51:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:51:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:51:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:51:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:51:56.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:56.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:56.960 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:51:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:51:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:51:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:51:56.961 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:51:57.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:57.030 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:57.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.164151+0000 osd.2 (osd.2) 143 : cluster 0 102.1e scrub starts 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.166512+0000 osd.2 (osd.2) 144 : cluster 0 102.1e scrub ok 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.539300+0000 osd.5 (osd.5) 183 : cluster 0 41.e scrub starts 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.540647+0000 osd.5 (osd.5) 184 : cluster 0 41.e scrub ok 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:55.875869+0000 mon.c (mon.2) 425 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:55.973368+0000 mgr.y (mgr.14538) 331 : cluster 0 pgmap v391: 396 pgs: 40 unknown, 32 creating+peering, 5 active+clean+snaptrim, 80 active+undersized, 202 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 274/1515 objects degraded (18.086%) 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:56.181752+0000 osd.2 (osd.2) 145 : cluster 0 102.18 scrub starts 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:56.184362+0000 osd.2 (osd.2) 146 : cluster 0 102.18 scrub ok 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.307791+0000 mon.c (mon.2) 426 : audit 1 from='client.? 172.21.15.69:0/2262895501' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]: dispatch 2024-04-02T04:51:57.260 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.647467+0000 mon.c (mon.2) 427 : audit 1 from='client.? 172.21.15.69:0/3407373028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.693832+0000 mon.a (mon.0) 1908 : audit 0 from='client.? 172.21.15.69:0/2125942760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.694477+0000 mon.a (mon.0) 1909 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]: dispatch 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.694620+0000 mon.a (mon.0) 1910 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:56.809760+0000 mon.a (mon.0) 1911 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.811047+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]': finished 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.811124+0000 mon.a (mon.0) 1913 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:51:57.261 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.813432+0000 mon.b (mon.1) 187 : audit 1 from='client.? 172.21.15.69:0/3890049415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:51:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:51:57.395 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:57.422 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: cluster 2024-04-02T04:51:56.815007+0000 mon.a (mon.0) 1914 : cluster 0 osdmap e277: 8 total, 7 up, 8 in 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.816446+0000 mon.c (mon.2) 428 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.816927+0000 mon.a (mon.0) 1915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.819697+0000 mon.a (mon.0) 1916 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.826393+0000 mon.c (mon.2) 429 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.835400+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:57.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:57 smithi098 bash[23790]: audit 2024-04-02T04:51:56.877022+0000 mon.c (mon.2) 430 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:57.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.164151+0000 osd.2 (osd.2) 143 : cluster 0 102.1e scrub starts 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.166512+0000 osd.2 (osd.2) 144 : cluster 0 102.1e scrub ok 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.539300+0000 osd.5 (osd.5) 183 : cluster 0 41.e scrub starts 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.540647+0000 osd.5 (osd.5) 184 : cluster 0 41.e scrub ok 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:55.875869+0000 mon.c (mon.2) 425 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:55.973368+0000 mgr.y (mgr.14538) 331 : cluster 0 pgmap v391: 396 pgs: 40 unknown, 32 creating+peering, 5 active+clean+snaptrim, 80 active+undersized, 202 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 274/1515 objects degraded (18.086%) 2024-04-02T04:51:57.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:56.181752+0000 osd.2 (osd.2) 145 : cluster 0 102.18 scrub starts 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:56.184362+0000 osd.2 (osd.2) 146 : cluster 0 102.18 scrub ok 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.307791+0000 mon.c (mon.2) 426 : audit 1 from='client.? 172.21.15.69:0/2262895501' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.647467+0000 mon.c (mon.2) 427 : audit 1 from='client.? 172.21.15.69:0/3407373028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.693832+0000 mon.a (mon.0) 1908 : audit 0 from='client.? 172.21.15.69:0/2125942760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.694477+0000 mon.a (mon.0) 1909 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.694620+0000 mon.a (mon.0) 1910 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:56.809760+0000 mon.a (mon.0) 1911 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.811047+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]': finished 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.811124+0000 mon.a (mon.0) 1913 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.813432+0000 mon.b (mon.1) 187 : audit 1 from='client.? 172.21.15.69:0/3890049415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: cluster 2024-04-02T04:51:56.815007+0000 mon.a (mon.0) 1914 : cluster 0 osdmap e277: 8 total, 7 up, 8 in 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.816446+0000 mon.c (mon.2) 428 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.816927+0000 mon.a (mon.0) 1915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.164151+0000 osd.2 (osd.2) 143 : cluster 0 102.1e scrub starts 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.166512+0000 osd.2 (osd.2) 144 : cluster 0 102.1e scrub ok 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.539300+0000 osd.5 (osd.5) 183 : cluster 0 41.e scrub starts 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.540647+0000 osd.5 (osd.5) 184 : cluster 0 41.e scrub ok 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:55.875869+0000 mon.c (mon.2) 425 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:55.973368+0000 mgr.y (mgr.14538) 331 : cluster 0 pgmap v391: 396 pgs: 40 unknown, 32 creating+peering, 5 active+clean+snaptrim, 80 active+undersized, 202 active+clean, 37 active+undersized+degraded; 587 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 274/1515 objects degraded (18.086%) 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:56.181752+0000 osd.2 (osd.2) 145 : cluster 0 102.18 scrub starts 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:56.184362+0000 osd.2 (osd.2) 146 : cluster 0 102.18 scrub ok 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.307791+0000 mon.c (mon.2) 426 : audit 1 from='client.? 172.21.15.69:0/2262895501' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]: dispatch 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.647467+0000 mon.c (mon.2) 427 : audit 1 from='client.? 172.21.15.69:0/3407373028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:51:57.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.693832+0000 mon.a (mon.0) 1908 : audit 0 from='client.? 172.21.15.69:0/2125942760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.694477+0000 mon.a (mon.0) 1909 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.694620+0000 mon.a (mon.0) 1910 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:56.809760+0000 mon.a (mon.0) 1911 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.811047+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "102.7"}]': finished 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.811124+0000 mon.a (mon.0) 1913 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.813432+0000 mon.b (mon.1) 187 : audit 1 from='client.? 172.21.15.69:0/3890049415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: cluster 2024-04-02T04:51:56.815007+0000 mon.a (mon.0) 1914 : cluster 0 osdmap e277: 8 total, 7 up, 8 in 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.816446+0000 mon.c (mon.2) 428 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.816927+0000 mon.a (mon.0) 1915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.819697+0000 mon.a (mon.0) 1916 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.826393+0000 mon.c (mon.2) 429 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.835400+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[21528]: audit 2024-04-02T04:51:56.877022+0000 mon.c (mon.2) 430 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.819697+0000 mon.a (mon.0) 1916 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.826393+0000 mon.c (mon.2) 429 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.835400+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:57 smithi069 bash[28904]: audit 2024-04-02T04:51:56.877022+0000 mon.c (mon.2) 430 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:57.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:57.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:57.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-02T04:51:57.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:57.811 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:57.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:57.931 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:57.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:58.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:58.057 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:58.163 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:58.185 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:58.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:58.312 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:51:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:51:58.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: cluster 2024-04-02T04:51:56.576036+0000 osd.5 (osd.5) 185 : cluster 0 6.7 scrub starts 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: cluster 2024-04-02T04:51:56.577221+0000 osd.5 (osd.5) 186 : cluster 0 6.7 scrub ok 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: audit 2024-04-02T04:51:57.813061+0000 mon.a (mon.0) 1918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: audit 2024-04-02T04:51:57.813168+0000 mon.a (mon.0) 1919 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: audit 2024-04-02T04:51:57.813228+0000 mon.a (mon.0) 1920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: cluster 2024-04-02T04:51:57.821444+0000 mon.a (mon.0) 1921 : cluster 0 osdmap e278: 8 total, 7 up, 8 in 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: audit 2024-04-02T04:51:57.823781+0000 mon.c (mon.2) 431 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: audit 2024-04-02T04:51:57.826659+0000 mon.a (mon.0) 1922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:58.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21528]: audit 2024-04-02T04:51:57.877898+0000 mon.c (mon.2) 432 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: cluster 2024-04-02T04:51:56.576036+0000 osd.5 (osd.5) 185 : cluster 0 6.7 scrub starts 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: cluster 2024-04-02T04:51:56.577221+0000 osd.5 (osd.5) 186 : cluster 0 6.7 scrub ok 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: audit 2024-04-02T04:51:57.813061+0000 mon.a (mon.0) 1918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: audit 2024-04-02T04:51:57.813168+0000 mon.a (mon.0) 1919 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: audit 2024-04-02T04:51:57.813228+0000 mon.a (mon.0) 1920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: cluster 2024-04-02T04:51:57.821444+0000 mon.a (mon.0) 1921 : cluster 0 osdmap e278: 8 total, 7 up, 8 in 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: audit 2024-04-02T04:51:57.823781+0000 mon.c (mon.2) 431 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: audit 2024-04-02T04:51:57.826659+0000 mon.a (mon.0) 1922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:58.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[28904]: audit 2024-04-02T04:51:57.877898+0000 mon.c (mon.2) 432 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:58.439 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:58.542 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:58.567 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:58.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:58.683 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: cluster 2024-04-02T04:51:56.576036+0000 osd.5 (osd.5) 185 : cluster 0 6.7 scrub starts 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: cluster 2024-04-02T04:51:56.577221+0000 osd.5 (osd.5) 186 : cluster 0 6.7 scrub ok 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: audit 2024-04-02T04:51:57.813061+0000 mon.a (mon.0) 1918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: audit 2024-04-02T04:51:57.813168+0000 mon.a (mon.0) 1919 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: audit 2024-04-02T04:51:57.813228+0000 mon.a (mon.0) 1920 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: cluster 2024-04-02T04:51:57.821444+0000 mon.a (mon.0) 1921 : cluster 0 osdmap e278: 8 total, 7 up, 8 in 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: audit 2024-04-02T04:51:57.823781+0000 mon.c (mon.2) 431 : audit 1 from='client.? 172.21.15.69:0/2944377695' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: audit 2024-04-02T04:51:57.826659+0000 mon.a (mon.0) 1922 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]: dispatch 2024-04-02T04:51:58.684 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:58 smithi098 bash[23790]: audit 2024-04-02T04:51:57.877898+0000 mon.c (mon.2) 432 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:51:58.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:58.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:58.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:58.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:59.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:51:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:51:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:51:59.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:59.114 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:59.217 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:59.244 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:51:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:51:59.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:59.358 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: cluster 2024-04-02T04:51:57.975137+0000 mgr.y (mgr.14538) 332 : cluster 0 pgmap v394: 452 pgs: 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 280/1506 objects degraded (18.592%) 2024-04-02T04:51:59.358 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: cluster 2024-04-02T04:51:58.813171+0000 mon.a (mon.0) 1923 : cluster 3 Health check update: Degraded data redundancy: 280/1506 objects degraded (18.592%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:59.358 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.828506+0000 mon.a (mon.0) 1924 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: cluster 2024-04-02T04:51:58.833113+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e279: 8 total, 7 up, 8 in 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.847163+0000 mon.c (mon.2) 433 : audit 1 from='client.? 172.21.15.69:0/3531658135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.849101+0000 mon.a (mon.0) 1926 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.872883+0000 mon.a (mon.0) 1927 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.873992+0000 mon.a (mon.0) 1928 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.874419+0000 mon.a (mon.0) 1929 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-72145-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:59.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.878772+0000 mon.c (mon.2) 434 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:59.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.885373+0000 mon.c (mon.2) 435 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:59.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[28904]: audit 2024-04-02T04:51:58.885653+0000 mon.a (mon.0) 1930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:59.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: cluster 2024-04-02T04:51:57.975137+0000 mgr.y (mgr.14538) 332 : cluster 0 pgmap v394: 452 pgs: 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 280/1506 objects degraded (18.592%) 2024-04-02T04:51:59.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: cluster 2024-04-02T04:51:58.813171+0000 mon.a (mon.0) 1923 : cluster 3 Health check update: Degraded data redundancy: 280/1506 objects degraded (18.592%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.828506+0000 mon.a (mon.0) 1924 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: cluster 2024-04-02T04:51:58.833113+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e279: 8 total, 7 up, 8 in 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.847163+0000 mon.c (mon.2) 433 : audit 1 from='client.? 172.21.15.69:0/3531658135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.849101+0000 mon.a (mon.0) 1926 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.872883+0000 mon.a (mon.0) 1927 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.873992+0000 mon.a (mon.0) 1928 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.874419+0000 mon.a (mon.0) 1929 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-72145-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.878772+0000 mon.c (mon.2) 434 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.885373+0000 mon.c (mon.2) 435 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:59.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:51:59 smithi069 bash[21528]: audit 2024-04-02T04:51:58.885653+0000 mon.a (mon.0) 1930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:59.371 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:51:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:51:59.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:59.501 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:51:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:51:59.502 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:51:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:51:59.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:59.629 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:59.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: cluster 2024-04-02T04:51:57.975137+0000 mgr.y (mgr.14538) 332 : cluster 0 pgmap v394: 452 pgs: 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 280/1506 objects degraded (18.592%) 2024-04-02T04:51:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: cluster 2024-04-02T04:51:58.813171+0000 mon.a (mon.0) 1923 : cluster 3 Health check update: Degraded data redundancy: 280/1506 objects degraded (18.592%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:51:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.828506+0000 mon.a (mon.0) 1924 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-72145-33"}]': finished 2024-04-02T04:51:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: cluster 2024-04-02T04:51:58.833113+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e279: 8 total, 7 up, 8 in 2024-04-02T04:51:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.847163+0000 mon.c (mon.2) 433 : audit 1 from='client.? 172.21.15.69:0/3531658135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.849101+0000 mon.a (mon.0) 1926 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:51:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.872883+0000 mon.a (mon.0) 1927 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:51:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.873992+0000 mon.a (mon.0) 1928 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:51:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.874419+0000 mon.a (mon.0) 1929 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-72145-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:51:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.878772+0000 mon.c (mon.2) 434 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:51:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.885373+0000 mon.c (mon.2) 435 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:51:59 smithi098 bash[23790]: audit 2024-04-02T04:51:58.885653+0000 mon.a (mon.0) 1930 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:51:59.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:51:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:51:59.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:59.899 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:51:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:51:59.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:00.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:00.023 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:00.126 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:00.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:00.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:00.277 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:00.380 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:00.401 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:00.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:00.534 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:00.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:00.662 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:00.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:00.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:00.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:00.903 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.832929+0000 mon.a (mon.0) 1931 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:00.903 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.833160+0000 mon.a (mon.0) 1932 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-72145-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.833329+0000 mon.a (mon.0) 1933 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: cluster 2024-04-02T04:51:59.839882+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e280: 8 total, 7 up, 8 in 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.847440+0000 mon.c (mon.2) 436 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.853018+0000 mon.a (mon.0) 1935 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-72145-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.853638+0000 mon.a (mon.0) 1936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: audit 2024-04-02T04:51:59.879950+0000 mon.c (mon.2) 437 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: cluster 2024-04-02T04:51:59.976406+0000 mgr.y (mgr.14538) 333 : cluster 0 pgmap v397: 484 pgs: 32 unknown, 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 280/1506 objects degraded (18.592%) 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: cluster 2024-04-02T04:52:00.431376+0000 osd.0 (osd.0) 191 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[28904]: cluster 2024-04-02T04:52:00.431453+0000 osd.0 (osd.0) 192 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:00.904 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.832929+0000 mon.a (mon.0) 1931 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.833160+0000 mon.a (mon.0) 1932 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-72145-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.833329+0000 mon.a (mon.0) 1933 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: cluster 2024-04-02T04:51:59.839882+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e280: 8 total, 7 up, 8 in 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.847440+0000 mon.c (mon.2) 436 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.853018+0000 mon.a (mon.0) 1935 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-72145-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.853638+0000 mon.a (mon.0) 1936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: audit 2024-04-02T04:51:59.879950+0000 mon.c (mon.2) 437 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: cluster 2024-04-02T04:51:59.976406+0000 mgr.y (mgr.14538) 333 : cluster 0 pgmap v397: 484 pgs: 32 unknown, 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 280/1506 objects degraded (18.592%) 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: cluster 2024-04-02T04:52:00.431376+0000 osd.0 (osd.0) 191 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:00.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:00 smithi069 bash[21528]: cluster 2024-04-02T04:52:00.431453+0000 osd.0 (osd.0) 192 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:00.918 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:00.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:01.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:01.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:01.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:01.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:01.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.832929+0000 mon.a (mon.0) 1931 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:01.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.833160+0000 mon.a (mon.0) 1932 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-72145-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:01.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.833329+0000 mon.a (mon.0) 1933 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:01.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: cluster 2024-04-02T04:51:59.839882+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e280: 8 total, 7 up, 8 in 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.847440+0000 mon.c (mon.2) 436 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.853018+0000 mon.a (mon.0) 1935 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-72145-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.853638+0000 mon.a (mon.0) 1936 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: audit 2024-04-02T04:51:59.879950+0000 mon.c (mon.2) 437 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: cluster 2024-04-02T04:51:59.976406+0000 mgr.y (mgr.14538) 333 : cluster 0 pgmap v397: 484 pgs: 32 unknown, 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 280/1506 objects degraded (18.592%) 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: cluster 2024-04-02T04:52:00.431376+0000 osd.0 (osd.0) 191 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:00 smithi098 bash[23790]: cluster 2024-04-02T04:52:00.431453+0000 osd.0 (osd.0) 192 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:01.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:01.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:01.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:01.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:52:01.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:52:01.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:01.587 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:01.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:01.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:01.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:01.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: audit 2024-04-02T04:52:00.849240+0000 mon.a (mon.0) 1937 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]': finished 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: audit 2024-04-02T04:52:00.866399+0000 mon.c (mon.2) 438 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]: dispatch 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: audit 2024-04-02T04:52:00.882483+0000 mon.c (mon.2) 439 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: cluster 2024-04-02T04:52:01.315206+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e281: 8 total, 7 up, 8 in 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: audit 2024-04-02T04:52:01.316719+0000 mon.a (mon.0) 1939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]: dispatch 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: audit 2024-04-02T04:52:01.317020+0000 mon.c (mon.2) 440 : audit 1 from='client.? 172.21.15.69:0/65787901' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:01.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:01 smithi098 bash[23790]: audit 2024-04-02T04:52:01.317801+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:01.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:02.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:02.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: audit 2024-04-02T04:52:00.849240+0000 mon.a (mon.0) 1937 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]': finished 2024-04-02T04:52:02.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: audit 2024-04-02T04:52:00.866399+0000 mon.c (mon.2) 438 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]: dispatch 2024-04-02T04:52:02.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: audit 2024-04-02T04:52:00.882483+0000 mon.c (mon.2) 439 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:02.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: cluster 2024-04-02T04:52:01.315206+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e281: 8 total, 7 up, 8 in 2024-04-02T04:52:02.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: audit 2024-04-02T04:52:01.316719+0000 mon.a (mon.0) 1939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]: dispatch 2024-04-02T04:52:02.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: audit 2024-04-02T04:52:01.317020+0000 mon.c (mon.2) 440 : audit 1 from='client.? 172.21.15.69:0/65787901' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[21528]: audit 2024-04-02T04:52:01.317801+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: audit 2024-04-02T04:52:00.849240+0000 mon.a (mon.0) 1937 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-25"}]': finished 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: audit 2024-04-02T04:52:00.866399+0000 mon.c (mon.2) 438 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]: dispatch 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: audit 2024-04-02T04:52:00.882483+0000 mon.c (mon.2) 439 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: cluster 2024-04-02T04:52:01.315206+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e281: 8 total, 7 up, 8 in 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: audit 2024-04-02T04:52:01.316719+0000 mon.a (mon.0) 1939 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]: dispatch 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: audit 2024-04-02T04:52:01.317020+0000 mon.c (mon.2) 440 : audit 1 from='client.? 172.21.15.69:0/65787901' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:02.081 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:01 smithi069 bash[28904]: audit 2024-04-02T04:52:01.317801+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:02.094 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:02.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:02.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:02.357 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:02.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:02.488 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:52:02.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:02.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:02.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:02.746 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:02.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:02.880 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:02.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:02.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: cluster 2024-04-02T04:52:01.850061+0000 mon.a (mon.0) 1941 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:01.857939+0000 mon.a (mon.0) 1942 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-72145-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:01.858043+0000 mon.a (mon.0) 1943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]': finished 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:01.858140+0000 mon.a (mon.0) 1944 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: cluster 2024-04-02T04:52:01.861748+0000 mon.a (mon.0) 1945 : cluster 0 osdmap e282: 8 total, 7 up, 8 in 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:01.884047+0000 mon.c (mon.2) 441 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:01.943718+0000 mon.c (mon.2) 442 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:01.944088+0000 mon.a (mon.0) 1946 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: cluster 2024-04-02T04:52:01.977296+0000 mgr.y (mgr.14538) 334 : cluster 0 pgmap v400: 524 pgs: 72 unknown, 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 280/1506 objects degraded (18.592%) 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:02.670398+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:02.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[21528]: audit 2024-04-02T04:52:02.670436+0000 mon.b (mon.1) 188 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: cluster 2024-04-02T04:52:01.850061+0000 mon.a (mon.0) 1941 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:01.857939+0000 mon.a (mon.0) 1942 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-72145-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:01.858043+0000 mon.a (mon.0) 1943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]': finished 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:01.858140+0000 mon.a (mon.0) 1944 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: cluster 2024-04-02T04:52:01.861748+0000 mon.a (mon.0) 1945 : cluster 0 osdmap e282: 8 total, 7 up, 8 in 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:01.884047+0000 mon.c (mon.2) 441 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:01.943718+0000 mon.c (mon.2) 442 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:01.944088+0000 mon.a (mon.0) 1946 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: cluster 2024-04-02T04:52:01.977296+0000 mgr.y (mgr.14538) 334 : cluster 0 pgmap v400: 524 pgs: 72 unknown, 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 280/1506 objects degraded (18.592%) 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:02.670398+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:02.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:02 smithi069 bash[28904]: audit 2024-04-02T04:52:02.670436+0000 mon.b (mon.1) 188 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:03.016 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: cluster 2024-04-02T04:52:01.850061+0000 mon.a (mon.0) 1941 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:01.857939+0000 mon.a (mon.0) 1942 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-72145-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:01.858043+0000 mon.a (mon.0) 1943 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-25", "mode": "writeback"}]': finished 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:01.858140+0000 mon.a (mon.0) 1944 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-72153-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: cluster 2024-04-02T04:52:01.861748+0000 mon.a (mon.0) 1945 : cluster 0 osdmap e282: 8 total, 7 up, 8 in 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:01.884047+0000 mon.c (mon.2) 441 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:01.943718+0000 mon.c (mon.2) 442 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:01.944088+0000 mon.a (mon.0) 1946 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: cluster 2024-04-02T04:52:01.977296+0000 mgr.y (mgr.14538) 334 : cluster 0 pgmap v400: 524 pgs: 72 unknown, 1 active, 9 creating+activating, 1 activating+degraded, 6 creating+activating+undersized, 132 active+undersized, 266 active+clean, 37 active+undersized+degraded; 587 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 280/1506 objects degraded (18.592%) 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:02.670398+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:03.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:02 smithi098 bash[23790]: audit 2024-04-02T04:52:02.670436+0000 mon.b (mon.1) 188 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:03.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:03.144 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:03.247 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:03.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:03.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:03.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:52:03.400 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:03.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:03.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:03.660 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:03.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:03.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:03.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: cluster 2024-04-02T04:52:02.861829+0000 mon.a (mon.0) 1948 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: audit 2024-04-02T04:52:02.872392+0000 mon.a (mon.0) 1949 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: cluster 2024-04-02T04:52:02.878117+0000 mon.a (mon.0) 1950 : cluster 0 osdmap e283: 8 total, 7 up, 8 in 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: audit 2024-04-02T04:52:02.884608+0000 mon.c (mon.2) 443 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: audit 2024-04-02T04:52:02.885143+0000 mon.a (mon.0) 1951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: audit 2024-04-02T04:52:02.886152+0000 mon.c (mon.2) 444 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:03.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:03 smithi098 bash[23790]: audit 2024-04-02T04:52:03.115768+0000 mgr.y (mgr.14538) 335 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:03.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:04.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:04.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: cluster 2024-04-02T04:52:02.861829+0000 mon.a (mon.0) 1948 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: audit 2024-04-02T04:52:02.872392+0000 mon.a (mon.0) 1949 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: cluster 2024-04-02T04:52:02.878117+0000 mon.a (mon.0) 1950 : cluster 0 osdmap e283: 8 total, 7 up, 8 in 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: audit 2024-04-02T04:52:02.884608+0000 mon.c (mon.2) 443 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: audit 2024-04-02T04:52:02.885143+0000 mon.a (mon.0) 1951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: audit 2024-04-02T04:52:02.886152+0000 mon.c (mon.2) 444 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[21528]: audit 2024-04-02T04:52:03.115768+0000 mgr.y (mgr.14538) 335 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: cluster 2024-04-02T04:52:02.861829+0000 mon.a (mon.0) 1948 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: audit 2024-04-02T04:52:02.872392+0000 mon.a (mon.0) 1949 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: cluster 2024-04-02T04:52:02.878117+0000 mon.a (mon.0) 1950 : cluster 0 osdmap e283: 8 total, 7 up, 8 in 2024-04-02T04:52:04.159 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: audit 2024-04-02T04:52:02.884608+0000 mon.c (mon.2) 443 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:04.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: audit 2024-04-02T04:52:02.885143+0000 mon.a (mon.0) 1951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]: dispatch 2024-04-02T04:52:04.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: audit 2024-04-02T04:52:02.886152+0000 mon.c (mon.2) 444 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:04.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:03 smithi069 bash[28904]: audit 2024-04-02T04:52:03.115768+0000 mgr.y (mgr.14538) 335 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:04.172 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:04.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:04.296 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:04.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:04.424 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:04.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-04-02T04:52:04.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:52:04.509 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-02T04:52:04.509 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-02T04:52:04.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:04.581 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:04.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:04.707 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:04.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:04.839 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:04.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:04.966 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:04.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:05.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:05.095 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:05.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:05.223 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:05.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:05.349 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:05.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: cluster 2024-04-02T04:52:03.872890+0000 mon.a (mon.0) 1952 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: audit 2024-04-02T04:52:03.875118+0000 mon.a (mon.0) 1953 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]': finished 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: cluster 2024-04-02T04:52:03.882728+0000 mon.a (mon.0) 1954 : cluster 0 osdmap e284: 8 total, 7 up, 8 in 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: audit 2024-04-02T04:52:03.888972+0000 mon.a (mon.0) 1955 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: audit 2024-04-02T04:52:03.894755+0000 mon.c (mon.2) 445 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: cluster 2024-04-02T04:52:03.978735+0000 mgr.y (mgr.14538) 336 : cluster 0 pgmap v403: 452 pgs: 138 active+undersized, 277 active+clean, 37 active+undersized+degraded; 587 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 274/1512 objects degraded (18.122%); 3 B/s, 1 objects/s recovering 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: audit 2024-04-02T04:52:04.811726+0000 mon.a (mon.0) 1956 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: cluster 2024-04-02T04:52:04.817927+0000 mon.a (mon.0) 1957 : cluster 0 osdmap e285: 8 total, 7 up, 8 in 2024-04-02T04:52:05.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[28904]: audit 2024-04-02T04:52:04.819547+0000 mon.a (mon.0) 1958 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: cluster 2024-04-02T04:52:03.872890+0000 mon.a (mon.0) 1952 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: audit 2024-04-02T04:52:03.875118+0000 mon.a (mon.0) 1953 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]': finished 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: cluster 2024-04-02T04:52:03.882728+0000 mon.a (mon.0) 1954 : cluster 0 osdmap e284: 8 total, 7 up, 8 in 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: audit 2024-04-02T04:52:03.888972+0000 mon.a (mon.0) 1955 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: audit 2024-04-02T04:52:03.894755+0000 mon.c (mon.2) 445 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: cluster 2024-04-02T04:52:03.978735+0000 mgr.y (mgr.14538) 336 : cluster 0 pgmap v403: 452 pgs: 138 active+undersized, 277 active+clean, 37 active+undersized+degraded; 587 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 274/1512 objects degraded (18.122%); 3 B/s, 1 objects/s recovering 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: audit 2024-04-02T04:52:04.811726+0000 mon.a (mon.0) 1956 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: cluster 2024-04-02T04:52:04.817927+0000 mon.a (mon.0) 1957 : cluster 0 osdmap e285: 8 total, 7 up, 8 in 2024-04-02T04:52:05.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:05 smithi069 bash[21528]: audit 2024-04-02T04:52:04.819547+0000 mon.a (mon.0) 1958 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: cluster 2024-04-02T04:52:03.872890+0000 mon.a (mon.0) 1952 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: audit 2024-04-02T04:52:03.875118+0000 mon.a (mon.0) 1953 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-25"}]': finished 2024-04-02T04:52:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: cluster 2024-04-02T04:52:03.882728+0000 mon.a (mon.0) 1954 : cluster 0 osdmap e284: 8 total, 7 up, 8 in 2024-04-02T04:52:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: audit 2024-04-02T04:52:03.888972+0000 mon.a (mon.0) 1955 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: audit 2024-04-02T04:52:03.894755+0000 mon.c (mon.2) 445 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: cluster 2024-04-02T04:52:03.978735+0000 mgr.y (mgr.14538) 336 : cluster 0 pgmap v403: 452 pgs: 138 active+undersized, 277 active+clean, 37 active+undersized+degraded; 587 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 274/1512 objects degraded (18.122%); 3 B/s, 1 objects/s recovering 2024-04-02T04:52:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: audit 2024-04-02T04:52:04.811726+0000 mon.a (mon.0) 1956 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: cluster 2024-04-02T04:52:04.817927+0000 mon.a (mon.0) 1957 : cluster 0 osdmap e285: 8 total, 7 up, 8 in 2024-04-02T04:52:05.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:05 smithi098 bash[23790]: audit 2024-04-02T04:52:04.819547+0000 mon.a (mon.0) 1958 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]: dispatch 2024-04-02T04:52:05.479 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:05.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:05.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:05.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:05.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:05.872 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:05.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:05.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:06.004 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:06.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:06.224 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:06.225 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:06.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:06.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:06.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:06.369 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:04.896577+0000 mon.c (mon.2) 446 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: cluster 2024-04-02T04:52:05.266614+0000 mon.a (mon.0) 1959 : cluster 3 Health check update: Degraded data redundancy: 274/1512 objects degraded (18.122%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.816437+0000 mon.a (mon.0) 1960 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: cluster 2024-04-02T04:52:05.828504+0000 mon.a (mon.0) 1961 : cluster 0 osdmap e286: 8 total, 7 up, 8 in 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.833074+0000 mon.c (mon.2) 447 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.836586+0000 mon.a (mon.0) 1962 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.870885+0000 mon.a (mon.0) 1963 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.873245+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.874716+0000 mon.a (mon.0) 1965 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-72145-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:06.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:06 smithi098 bash[23790]: audit 2024-04-02T04:52:05.898056+0000 mon.c (mon.2) 448 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:06.385 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14457.0:44 4.1f 4:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:43:57.497204+0000", 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 488.88413542699999, 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.81366403300000001, 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.386 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14457", 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1965820815", 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 44 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497204+0000", 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497204+0000", 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497206+0000", 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9069999999999999e-06 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497216+0000", 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0210000000000001e-05 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497244+0000", 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8022999999999999e-05 2024-04-02T04:52:06.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497267+0000", 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2379999999999999e-05 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497406+0000", 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013954899999999999 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:57.497431+0000", 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4702000000000002e-05 2024-04-02T04:52:06.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.303043+0000", 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.80561248500000004 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.303113+0000", 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9672999999999995e-05 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.303189+0000", 2024-04-02T04:52:06.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6007999999999995e-05 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.303511+0000", 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000321906 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.303709+0000", 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019813599999999999 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.310758+0000", 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070487980000000002 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.310818+0000", 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0402999999999997e-05 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:43:58.310868+0000", 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9852999999999999e-05 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.24851.0:2 29.1e 29:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:49:26.029991+0000", 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 160.351348094, 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.32016878500000001, 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.24851", 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/3334667875", 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.393 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.029991+0000", 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.029991+0000", 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.029993+0000", 2024-04-02T04:52:06.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.973e-06 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.030004+0000", 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0838e-05 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.030016+0000", 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1656e-05 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.030027+0000", 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1107999999999999e-05 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.071679+0000", 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041652346 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.071694+0000", 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.438e-05 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.178726+0000", 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.107031894 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.178850+0000", 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012445199999999999 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.178988+0000", 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013761200000000001 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.182394+0000", 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034060789999999998 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.292530+0000", 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11013635300000001 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.350045+0000", 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.057514634000000002 2024-04-02T04:52:06.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.350106+0000", 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0977999999999998e-05 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:26.350160+0000", 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4481999999999998e-05 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:416 15.4 e140/135 -17:242a8acf:::temp_15.4_0_24164_1:head v 0'0, mlcod=0'0)", 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:49:29.186580+0000", 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 157.194759336, 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043873493999999999, 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.186580+0000", 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.186580+0000", 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.186583+0000", 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3e-06 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.216577+0000", 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029994733999999999 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.216604+0000", 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6281000000000001e-05 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.216616+0000", 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2004999999999999e-05 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.216705+0000", 2024-04-02T04:52:06.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9779999999999998e-05 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.216728+0000", 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2266999999999999e-05 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.230347+0000", 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013619678 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.230454+0000", 2024-04-02T04:52:06.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106449 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.24758.0:2070 15.9 15:951b79d9:test-rados-api-smithi069-72304-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:49:29.410091+0000", 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 156.97124830300001, 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.123936754, 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.24758", 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/50059295", 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2070 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.410091+0000", 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.410091+0000", 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.410094+0000", 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8899999999999999e-06 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.448014+0000", 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037919815000000003 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.448082+0000", 2024-04-02T04:52:06.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8235e-05 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.448102+0000", 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9667999999999999e-05 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.448228+0000", 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012661500000000001 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.448445+0000", 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021663100000000001 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,7", 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.448675+0000", 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022980099999999999 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.507894+0000", 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.059218926999999998 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.532726+0000", 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024832178999999999 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.533846+0000", 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001119781 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.533941+0000", 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5309000000000005e-05 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:29.534028+0000", 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6903000000000005e-05 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.15162.0:4 48.7 48:e7132e20:nspace::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:49:31.582958+0000", 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 154.79838180799999, 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.572024367, 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.15162", 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2993715898", 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.582958+0000", 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.582958+0000", 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.582960+0000", 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6299999999999998e-06 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.582969+0000", 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2409999999999995e-06 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.582983+0000", 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3927999999999999e-05 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.582995+0000", 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2424e-05 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.586162+0000", 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031674289999999998 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:31.586173+0000", 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1017e-05 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.144340+0000", 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5581664659999999 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.144481+0000", 2024-04-02T04:52:06.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014095399999999999 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,2", 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.144642+0000", 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000161092 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.145329+0000", 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00068718000000000002 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.147020+0000", 2024-04-02T04:52:06.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001691236 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.154910+0000", 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078895900000000001 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.154945+0000", 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5358999999999998e-05 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:35.154982+0000", 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6820999999999999e-05 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25241.0:2 57.c 57:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:49:33.716839+0000", 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 152.664500995, 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7140647959999997, 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25241", 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/3915961410", 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.716839+0000", 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.716839+0000", 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.716842+0000", 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0469999999999998e-06 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.716858+0000", 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6540000000000001e-05 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.716890+0000", 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2119000000000002e-05 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.716915+0000", 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5017000000000001e-05 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.731091+0000", 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014176022 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.731114+0000", 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2606e-05 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.413699+0000", 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6825847710000001 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.415717+0000", 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002018733 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,4", 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.416318+0000", 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00060039200000000005 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.422255+0000", 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059372030000000003 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.422659+0000", 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040407300000000002 2024-04-02T04:52:06.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.430787+0000", 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0081283099999999997 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.430853+0000", 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5480000000000003e-05 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.430903+0000", 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0482999999999998e-05 2024-04-02T04:52:06.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25184.0:3 53.f 53:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:49:33.796548+0000", 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 152.58479133099999, 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6343113530000002, 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25184", 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2929247525", 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3 2024-04-02T04:52:06.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796548+0000", 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796548+0000", 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796550+0000", 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.818e-06 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796557+0000", 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8040000000000004e-06 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796569+0000", 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2573e-05 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796582+0000", 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2658e-05 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796660+0000", 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7979000000000007e-05 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:33.796669+0000", 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9830000000000002e-06 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.402226+0000", 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6055566749999999 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.402348+0000", 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012233599999999999 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.402502+0000", 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015397499999999999 2024-04-02T04:52:06.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.406374+0000", 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003871949 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.407110+0000", 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00073620599999999999 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.430733+0000", 2024-04-02T04:52:06.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.023622575 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.430802+0000", 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9354e-05 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:49:36.430860+0000", 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7467999999999999e-05 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.15510.0:2 124.0 124:0384ab8e:::oid:head [write 0~0,setxattr key1 (5) in=4b,omap-clear] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:05.086860+0000", 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.294479309, 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96874455900000001, 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.15510", 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/669224138", 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.431 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.086860+0000", 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.086860+0000", 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.086863+0000", 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6299999999999998e-06 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.086874+0000", 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1157e-05 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.086898+0000", 2024-04-02T04:52:06.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4426e-05 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.086916+0000", 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7404000000000001e-05 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.087039+0000", 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012336199999999999 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.087055+0000", 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5916000000000001e-05 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.040948+0000", 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95389294300000005 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.041046+0000", 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7564000000000003e-05 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.041165+0000", 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119085 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.041529+0000", 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036473000000000001 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.055521+0000", 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013991526000000001 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.055554+0000", 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2573999999999997e-05 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.055605+0000", 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1242000000000002e-05 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25210.0:11 112.17 112:ee4881bd:test-rados-api-smithi069-72389-9::foo:head [write 0~128] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:05.949472+0000", 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.431867103, 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.068767235999999995, 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25210", 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2140839855", 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.949472+0000", 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.949472+0000", 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.949474+0000", 2024-04-02T04:52:06.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7680000000000001e-06 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.949483+0000", 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6680000000000007e-06 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.949495+0000", 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2516e-05 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.949509+0000", 2024-04-02T04:52:06.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3597e-05 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.957309+0000", 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077999310000000004 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.957317+0000", 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0870000000000007e-06 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.993913+0000", 2024-04-02T04:52:06.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036596139 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.994216+0000", 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030283799999999999 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.994814+0000", 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00059788099999999998 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:05.999076+0000", 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042619650000000004 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.000923+0000", 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018476300000000001 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.018149+0000", 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017226009 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.018205+0000", 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5587000000000002e-05 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:06.018240+0000", 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4619999999999997e-05 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25210.0:16 112.17 112:ee4881bd:test-rados-api-smithi069-72389-9::foo:head [write 128~128] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:08.383662+0000", 2024-04-02T04:52:06.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 117.997677831, 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.095853753, 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25210", 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2140839855", 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 16 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383662+0000", 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383662+0000", 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383664+0000", 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.852e-06 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383673+0000", 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3789999999999995e-06 2024-04-02T04:52:06.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383694+0000", 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1307000000000001e-05 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383706+0000", 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1834999999999999e-05 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383788+0000", 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1954999999999997e-05 2024-04-02T04:52:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.383796+0000", 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7190000000000001e-06 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.413435+0000", 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029639293000000001 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.413545+0000", 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110289 2024-04-02T04:52:06.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.413796+0000", 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025072700000000001 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.420256+0000", 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064601499999999996 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.436009+0000", 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015753133999999998 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.479424+0000", 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043414411999999999 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.479463+0000", 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9678000000000002e-05 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:08.479515+0000", 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2023e-05 2024-04-02T04:52:06.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25625.0:3 137.0 137:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:12.752286+0000", 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.62905384600001, 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.082598255999999995, 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25625", 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/320423736", 2024-04-02T04:52:06.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752286+0000", 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752286+0000", 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752288+0000", 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.227e-06 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752297+0000", 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1570000000000006e-06 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752311+0000", 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.429e-05 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752325+0000", 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3606e-05 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.759756+0000", 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074310289999999996 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.759766+0000", 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0193e-05 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.803976+0000", 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.044209994000000002 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.804215+0000", 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000239077 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.804362+0000", 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014705799999999999 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.805080+0000", 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00071789 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.805475+0000", 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000394483 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.834766+0000", 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.02929112 2024-04-02T04:52:06.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.834831+0000", 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4760999999999999e-05 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.834884+0000", 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3371000000000001e-05 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25622.0:2 140.1b 140:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:12.752795+0000", 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.62854408299999, 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97515813299999998, 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25622", 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/813734156", 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752795+0000", 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752795+0000", 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752796+0000", 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0020000000000001e-06 2024-04-02T04:52:06.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752802+0000", 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2789999999999998e-06 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752813+0000", 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1197e-05 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.752823+0000", 2024-04-02T04:52:06.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0346000000000001e-05 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.778140+0000", 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.025316822999999999 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:12.778149+0000", 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3719999999999994e-06 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.704575+0000", 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.92642503200000004 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.704719+0000", 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014485400000000001 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,4", 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.704870+0000", 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150276 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.715400+0000", 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010530329999999999 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.717177+0000", 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001777076 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.727776+0000", 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010598675 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.727869+0000", 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3070000000000002e-05 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:13.727954+0000", 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4801000000000003e-05 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25312.0:2 142.6 142:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-04-02T04:52:06.464 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:14.688713+0000", 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.692627019, 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.103279367, 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25312", 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/769728975", 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688713+0000", 2024-04-02T04:52:06.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688713+0000", 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688715+0000", 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7769999999999998e-06 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688726+0000", 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0647e-05 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688749+0000", 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2855999999999999e-05 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688768+0000", 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9596e-05 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688909+0000", 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014071299999999999 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.688927+0000", 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.806e-05 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.765822+0000", 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.076894910999999996 2024-04-02T04:52:06.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.765834+0000", 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2143999999999999e-05 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.785334+0000", 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019500027999999999 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.785462+0000", 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012808399999999999 2024-04-02T04:52:06.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.785629+0000", 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000166985 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.788767+0000", 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031378959999999998 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.791843+0000", 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030755380000000001 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.791902+0000", 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9314000000000002e-05 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:14.791992+0000", 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9818e-05 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25625.0:8 137.0 137:0384ab8e:::oid:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:16.037017+0000", 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 110.344322694, 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.065085365000000006, 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25625", 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/320423736", 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 8 2024-04-02T04:52:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.037017+0000", 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.037017+0000", 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.037021+0000", 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.817e-06 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.037037+0000", 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5993000000000001e-05 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.039243+0000", 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022063640000000002 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.039272+0000", 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9456999999999999e-05 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.053417+0000", 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014144274 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.053427+0000", 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8700000000000004e-06 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.087541+0000", 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034114229000000003 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.087655+0000", 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114119 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.087836+0000", 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018061100000000001 2024-04-02T04:52:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.090047+0000", 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022118350000000001 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.091667+0000", 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001619297 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.102013+0000", 2024-04-02T04:52:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010346579 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.102070+0000", 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6805000000000002e-05 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:16.102102+0000", 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2115e-05 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25682.0:2 153.1b 153:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:19.428047+0000", 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 106.95329282199999, 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.102656726, 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25682", 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2025367561", 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428047+0000", 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428047+0000", 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428049+0000", 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.874e-06 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428056+0000", 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7079999999999997e-06 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428069+0000", 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3125999999999999e-05 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428082+0000", 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2391999999999999e-05 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428147+0000", 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4678000000000007e-05 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:19.428156+0000", 2024-04-02T04:52:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9590000000000007e-06 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.526710+0000", 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.098553463 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.526837+0000", 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012700499999999999 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-04-02T04:52:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.527010+0000", 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000173196 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.530086+0000", 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030755689999999998 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.530591+0000", 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00050536899999999998 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.530616+0000", 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4457e-05 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.530659+0000", 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3489000000000003e-05 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:20.530703+0000", 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4440999999999997e-05 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:256 146.e 146:73b4a3b3:test-rados-api-smithi069-72349-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e182)", 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:21.447779+0000", 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 104.933560782, 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.071852535999999995, 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T04:52:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 256 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.447779+0000", 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.447779+0000", 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.447782+0000", 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7499999999999999e-06 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.447798+0000", 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6504e-05 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.447832+0000", 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4189000000000001e-05 2024-04-02T04:52:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.447860+0000", 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8096999999999999e-05 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.448082+0000", 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022195799999999999 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.448131+0000", 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9106e-05 2024-04-02T04:52:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.492996+0000", 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.044864322999999998 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.493289+0000", 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029374300000000001 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.501462+0000", 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0081725510000000001 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.501562+0000", 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9912000000000002e-05 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.518478+0000", 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016915725 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.518552+0000", 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4783000000000006e-05 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:21.519631+0000", 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010788950000000001 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25907.0:2 194.1c 194:3ee403da:test-rados-api-smithi069-72304-31::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:50.727140+0000", 2024-04-02T04:52:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 75.654199711999993, 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.47901767200000001, 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25907", 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2960985830", 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727140+0000", 2024-04-02T04:52:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727140+0000", 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727144+0000", 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.693e-06 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727169+0000", 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5109e-05 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727203+0000", 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4164e-05 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727230+0000", 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7233000000000002e-05 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727399+0000", 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016850200000000001 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.727415+0000", 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6829000000000001e-05 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.758524+0000", 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.031108888000000001 2024-04-02T04:52:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:50.758536+0000", 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1969e-05 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.187329+0000", 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.42879279399999998 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.187459+0000", 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012967600000000001 2024-04-02T04:52:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.187603+0000", 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014432900000000001 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.188221+0000", 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061765500000000005 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.205945+0000", 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017723892000000002 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.206015+0000", 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0108e-05 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:51.206158+0000", 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000142831 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:2081 102.f 102:f2af07b9:test-rados-api-smithi069-72349-26::foo:head [writefull 0~5] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:51:58.834944+0000", 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.5463951930000004, 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046229852000000002, 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2081 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834944+0000", 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834944+0000", 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834947+0000", 2024-04-02T04:52:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834967+0000", 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9167000000000001e-05 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835000+0000", 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3853000000000003e-05 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835027+0000", 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6682999999999998e-05 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835206+0000", 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000178772 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835225+0000", 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.895e-05 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.878478+0000", 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043253536000000002 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.878682+0000", 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203518 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.879050+0000", 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036813500000000001 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.880257+0000", 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001207025 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.880408+0000", 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150982 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.880998+0000", 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058958899999999996 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.881093+0000", 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5283000000000002e-05 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.881174+0000", 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1268999999999997e-05 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.2.19:90 102.f 102:f2af07b9:test-rados-api-smithi069-72349-26::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e282)", 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:01.879028+0000", 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.5023119630000004, 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043008441000000001, 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879028+0000", 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879028+0000", 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879031+0000", 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1039999999999998e-06 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879045+0000", 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4363e-05 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879076+0000", 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0929999999999997e-05 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879105+0000", 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8846000000000001e-05 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879412+0000", 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030738400000000001 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879431+0000", 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8580000000000002e-05 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.921838+0000", 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.042407591000000001 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.921941+0000", 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102705 2024-04-02T04:52:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.922036+0000", 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4938000000000005e-05 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.2.19:91 102.f 102:f2af07b9:test-rados-api-smithi069-72349-26::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e282)", 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:01.879162+0000", 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.5021779569999998, 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.045282401, 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:52:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879162+0000", 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879162+0000", 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879163+0000", 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.663e-06 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879175+0000", 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1667e-05 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879210+0000", 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5562999999999998e-05 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879230+0000", 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9242999999999999e-05 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879586+0000", 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035638100000000002 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.879602+0000", 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6209e-05 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.923792+0000", 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.044189715999999997 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.924065+0000", 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027301499999999999 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.924444+0000", 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037894399999999998 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:52:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:52:06.516 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:52:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:06.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:04.896577+0000 mon.c (mon.2) 446 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: cluster 2024-04-02T04:52:05.266614+0000 mon.a (mon.0) 1959 : cluster 3 Health check update: Degraded data redundancy: 274/1512 objects degraded (18.122%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.816437+0000 mon.a (mon.0) 1960 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: cluster 2024-04-02T04:52:05.828504+0000 mon.a (mon.0) 1961 : cluster 0 osdmap e286: 8 total, 7 up, 8 in 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.833074+0000 mon.c (mon.2) 447 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.836586+0000 mon.a (mon.0) 1962 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:06.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.870885+0000 mon.a (mon.0) 1963 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.873245+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.874716+0000 mon.a (mon.0) 1965 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-72145-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[28904]: audit 2024-04-02T04:52:05.898056+0000 mon.c (mon.2) 448 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:04.896577+0000 mon.c (mon.2) 446 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: cluster 2024-04-02T04:52:05.266614+0000 mon.a (mon.0) 1959 : cluster 3 Health check update: Degraded data redundancy: 274/1512 objects degraded (18.122%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.816437+0000 mon.a (mon.0) 1960 : audit 1 from='client.? 172.21.15.69:0/3023083818' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-72145-34"}]': finished 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: cluster 2024-04-02T04:52:05.828504+0000 mon.a (mon.0) 1961 : cluster 0 osdmap e286: 8 total, 7 up, 8 in 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.833074+0000 mon.c (mon.2) 447 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.836586+0000 mon.a (mon.0) 1962 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.870885+0000 mon.a (mon.0) 1963 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:06.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.873245+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:06.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.874716+0000 mon.a (mon.0) 1965 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-72145-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:06.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:06 smithi069 bash[21528]: audit 2024-04-02T04:52:05.898056+0000 mon.c (mon.2) 448 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:06.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:06.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:06.770 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:06.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:06.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:06.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:07.025 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:07.128 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:07.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:07.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-02T04:52:07.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.398 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[21528]: cluster 2024-04-02T04:52:05.980597+0000 mgr.y (mgr.14538) 337 : cluster 0 pgmap v406: 388 pgs: 32 unknown, 99 active+undersized, 220 active+clean, 37 active+undersized+degraded; 587 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 274/1512 objects degraded (18.122%); 3 B/s, 1 objects/s recovering 2024-04-02T04:52:07.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[21528]: audit 2024-04-02T04:52:06.819463+0000 mon.a (mon.0) 1966 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:07.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[21528]: audit 2024-04-02T04:52:06.819695+0000 mon.a (mon.0) 1967 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-72145-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:07.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[21528]: cluster 2024-04-02T04:52:06.828545+0000 mon.a (mon.0) 1968 : cluster 0 osdmap e287: 8 total, 7 up, 8 in 2024-04-02T04:52:07.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[21528]: audit 2024-04-02T04:52:06.845973+0000 mon.a (mon.0) 1969 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi069-72145-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:07.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[21528]: audit 2024-04-02T04:52:06.898882+0000 mon.c (mon.2) 449 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:07.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[28904]: cluster 2024-04-02T04:52:05.980597+0000 mgr.y (mgr.14538) 337 : cluster 0 pgmap v406: 388 pgs: 32 unknown, 99 active+undersized, 220 active+clean, 37 active+undersized+degraded; 587 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 274/1512 objects degraded (18.122%); 3 B/s, 1 objects/s recovering 2024-04-02T04:52:07.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[28904]: audit 2024-04-02T04:52:06.819463+0000 mon.a (mon.0) 1966 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:07.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[28904]: audit 2024-04-02T04:52:06.819695+0000 mon.a (mon.0) 1967 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-72145-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:07.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[28904]: cluster 2024-04-02T04:52:06.828545+0000 mon.a (mon.0) 1968 : cluster 0 osdmap e287: 8 total, 7 up, 8 in 2024-04-02T04:52:07.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[28904]: audit 2024-04-02T04:52:06.845973+0000 mon.a (mon.0) 1969 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi069-72145-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:07.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:07 smithi069 bash[28904]: audit 2024-04-02T04:52:06.898882+0000 mon.c (mon.2) 449 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:07.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:07.529 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:07.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:07 smithi098 bash[23790]: cluster 2024-04-02T04:52:05.980597+0000 mgr.y (mgr.14538) 337 : cluster 0 pgmap v406: 388 pgs: 32 unknown, 99 active+undersized, 220 active+clean, 37 active+undersized+degraded; 587 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 274/1512 objects degraded (18.122%); 3 B/s, 1 objects/s recovering 2024-04-02T04:52:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:07 smithi098 bash[23790]: audit 2024-04-02T04:52:06.819463+0000 mon.a (mon.0) 1966 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:07 smithi098 bash[23790]: audit 2024-04-02T04:52:06.819695+0000 mon.a (mon.0) 1967 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-72145-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:07 smithi098 bash[23790]: cluster 2024-04-02T04:52:06.828545+0000 mon.a (mon.0) 1968 : cluster 0 osdmap e287: 8 total, 7 up, 8 in 2024-04-02T04:52:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:07 smithi098 bash[23790]: audit 2024-04-02T04:52:06.845973+0000 mon.a (mon.0) 1969 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi069-72145-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:07 smithi098 bash[23790]: audit 2024-04-02T04:52:06.898882+0000 mon.c (mon.2) 449 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:07.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.787 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:07.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:07.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:08.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:08.045 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:08.148 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:08.170 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:08.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:08.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:08.533 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:08.550 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:08.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:08.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:08.792 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:52:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:52:08.804 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:08.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:08 smithi098 bash[23790]: audit 2024-04-02T04:52:07.899954+0000 mon.c (mon.2) 450 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:08.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:08 smithi098 bash[23790]: cluster 2024-04-02T04:52:07.982488+0000 mgr.y (mgr.14538) 338 : cluster 0 pgmap v409: 388 pgs: 28 creating+peering, 4 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:08.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:08 smithi098 bash[23790]: cluster 2024-04-02T04:52:08.266671+0000 mon.a (mon.0) 1970 : cluster 0 osdmap e288: 8 total, 7 up, 8 in 2024-04-02T04:52:08.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:08 smithi098 bash[23790]: audit 2024-04-02T04:52:08.269484+0000 mon.a (mon.0) 1971 : audit 1 from='client.? 172.21.15.69:0/1791372199' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-72153-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:08.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:08 smithi098 bash[23790]: audit 2024-04-02T04:52:08.279559+0000 mon.c (mon.2) 451 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:08.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:08 smithi098 bash[23790]: audit 2024-04-02T04:52:08.280065+0000 mon.a (mon.0) 1972 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:08.930 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:08.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21528]: audit 2024-04-02T04:52:07.899954+0000 mon.c (mon.2) 450 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21528]: cluster 2024-04-02T04:52:07.982488+0000 mgr.y (mgr.14538) 338 : cluster 0 pgmap v409: 388 pgs: 28 creating+peering, 4 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21528]: cluster 2024-04-02T04:52:08.266671+0000 mon.a (mon.0) 1970 : cluster 0 osdmap e288: 8 total, 7 up, 8 in 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21528]: audit 2024-04-02T04:52:08.269484+0000 mon.a (mon.0) 1971 : audit 1 from='client.? 172.21.15.69:0/1791372199' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-72153-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21528]: audit 2024-04-02T04:52:08.279559+0000 mon.c (mon.2) 451 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[21528]: audit 2024-04-02T04:52:08.280065+0000 mon.a (mon.0) 1972 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[28904]: audit 2024-04-02T04:52:07.899954+0000 mon.c (mon.2) 450 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[28904]: cluster 2024-04-02T04:52:07.982488+0000 mgr.y (mgr.14538) 338 : cluster 0 pgmap v409: 388 pgs: 28 creating+peering, 4 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[28904]: cluster 2024-04-02T04:52:08.266671+0000 mon.a (mon.0) 1970 : cluster 0 osdmap e288: 8 total, 7 up, 8 in 2024-04-02T04:52:09.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[28904]: audit 2024-04-02T04:52:08.269484+0000 mon.a (mon.0) 1971 : audit 1 from='client.? 172.21.15.69:0/1791372199' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-72153-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:09.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[28904]: audit 2024-04-02T04:52:08.279559+0000 mon.c (mon.2) 451 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:09.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:08 smithi069 bash[28904]: audit 2024-04-02T04:52:08.280065+0000 mon.a (mon.0) 1972 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:09.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:09.055 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:09.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:09.185 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:09.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:09.316 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:09.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:09.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:09.573 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:09.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:09.705 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:09.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:09.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:09.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: audit 2024-04-02T04:52:08.830805+0000 mon.a (mon.0) 1973 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi069-72145-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:09.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: audit 2024-04-02T04:52:08.830881+0000 mon.a (mon.0) 1974 : audit 1 from='client.? 172.21.15.69:0/1791372199' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-72153-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: audit 2024-04-02T04:52:08.830932+0000 mon.a (mon.0) 1975 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: audit 2024-04-02T04:52:08.836803+0000 mon.c (mon.2) 452 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: cluster 2024-04-02T04:52:08.846719+0000 mon.a (mon.0) 1976 : cluster 0 osdmap e289: 8 total, 7 up, 8 in 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: audit 2024-04-02T04:52:08.848714+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: audit 2024-04-02T04:52:08.900990+0000 mon.c (mon.2) 453 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: cluster 2024-04-02T04:52:08.941562+0000 osd.3 (osd.3) 219 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[21528]: cluster 2024-04-02T04:52:08.941741+0000 osd.3 (osd.3) 220 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: audit 2024-04-02T04:52:08.830805+0000 mon.a (mon.0) 1973 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi069-72145-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: audit 2024-04-02T04:52:08.830881+0000 mon.a (mon.0) 1974 : audit 1 from='client.? 172.21.15.69:0/1791372199' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-72153-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: audit 2024-04-02T04:52:08.830932+0000 mon.a (mon.0) 1975 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:09.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: audit 2024-04-02T04:52:08.836803+0000 mon.c (mon.2) 452 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:09.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: cluster 2024-04-02T04:52:08.846719+0000 mon.a (mon.0) 1976 : cluster 0 osdmap e289: 8 total, 7 up, 8 in 2024-04-02T04:52:09.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: audit 2024-04-02T04:52:08.848714+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:09.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: audit 2024-04-02T04:52:08.900990+0000 mon.c (mon.2) 453 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:09.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: cluster 2024-04-02T04:52:08.941562+0000 osd.3 (osd.3) 219 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:09.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:09 smithi069 bash[28904]: cluster 2024-04-02T04:52:08.941741+0000 osd.3 (osd.3) 220 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:09.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:09.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:09.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:10.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:10.079 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:10.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:10.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: audit 2024-04-02T04:52:08.830805+0000 mon.a (mon.0) 1973 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi069-72145-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:10.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: audit 2024-04-02T04:52:08.830881+0000 mon.a (mon.0) 1974 : audit 1 from='client.? 172.21.15.69:0/1791372199' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-72153-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:10.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: audit 2024-04-02T04:52:08.830932+0000 mon.a (mon.0) 1975 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:10.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: audit 2024-04-02T04:52:08.836803+0000 mon.c (mon.2) 452 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:10.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: cluster 2024-04-02T04:52:08.846719+0000 mon.a (mon.0) 1976 : cluster 0 osdmap e289: 8 total, 7 up, 8 in 2024-04-02T04:52:10.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: audit 2024-04-02T04:52:08.848714+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:10.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: audit 2024-04-02T04:52:08.900990+0000 mon.c (mon.2) 453 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:10.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: cluster 2024-04-02T04:52:08.941562+0000 osd.3 (osd.3) 219 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:10.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:09 smithi098 bash[23790]: cluster 2024-04-02T04:52:08.941741+0000 osd.3 (osd.3) 220 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:10.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:10.327 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:10.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:10.452 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:10.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:10.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:10.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:10.638 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:10.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:10.708 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:10.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:10.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:10.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:10.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: audit 2024-04-02T04:52:09.842767+0000 mon.a (mon.0) 1978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]': finished 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: cluster 2024-04-02T04:52:09.850434+0000 mon.a (mon.0) 1979 : cluster 0 osdmap e290: 8 total, 7 up, 8 in 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: audit 2024-04-02T04:52:09.854486+0000 mon.c (mon.2) 454 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]: dispatch 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: audit 2024-04-02T04:52:09.856158+0000 mon.a (mon.0) 1980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]: dispatch 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: audit 2024-04-02T04:52:09.901934+0000 mon.c (mon.2) 455 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: cluster 2024-04-02T04:52:09.983338+0000 mgr.y (mgr.14538) 339 : cluster 0 pgmap v412: 364 pgs: 8 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:10 smithi098 bash[23790]: audit 2024-04-02T04:52:10.616689+0000 mon.b (mon.1) 189 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:11.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:11.078 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: audit 2024-04-02T04:52:09.842767+0000 mon.a (mon.0) 1978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]': finished 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: cluster 2024-04-02T04:52:09.850434+0000 mon.a (mon.0) 1979 : cluster 0 osdmap e290: 8 total, 7 up, 8 in 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: audit 2024-04-02T04:52:09.854486+0000 mon.c (mon.2) 454 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]: dispatch 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: audit 2024-04-02T04:52:09.856158+0000 mon.a (mon.0) 1980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]: dispatch 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: audit 2024-04-02T04:52:09.901934+0000 mon.c (mon.2) 455 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: cluster 2024-04-02T04:52:09.983338+0000 mgr.y (mgr.14538) 339 : cluster 0 pgmap v412: 364 pgs: 8 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[21528]: audit 2024-04-02T04:52:10.616689+0000 mon.b (mon.1) 189 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: audit 2024-04-02T04:52:09.842767+0000 mon.a (mon.0) 1978 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-27"}]': finished 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: cluster 2024-04-02T04:52:09.850434+0000 mon.a (mon.0) 1979 : cluster 0 osdmap e290: 8 total, 7 up, 8 in 2024-04-02T04:52:11.087 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: audit 2024-04-02T04:52:09.854486+0000 mon.c (mon.2) 454 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]: dispatch 2024-04-02T04:52:11.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: audit 2024-04-02T04:52:09.856158+0000 mon.a (mon.0) 1980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]: dispatch 2024-04-02T04:52:11.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: audit 2024-04-02T04:52:09.901934+0000 mon.c (mon.2) 455 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:11.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: cluster 2024-04-02T04:52:09.983338+0000 mgr.y (mgr.14538) 339 : cluster 0 pgmap v412: 364 pgs: 8 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:11.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:10 smithi069 bash[28904]: audit 2024-04-02T04:52:10.616689+0000 mon.b (mon.1) 189 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:11.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.203 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:11.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:11.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:11.458 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:11.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:11.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:11.717 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:11.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:11.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:11.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: cluster 2024-04-02T04:52:10.843759+0000 mon.a (mon.0) 1981 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:10.850739+0000 mon.a (mon.0) 1982 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]': finished 2024-04-02T04:52:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: cluster 2024-04-02T04:52:10.856192+0000 mon.a (mon.0) 1983 : cluster 0 osdmap e291: 8 total, 7 up, 8 in 2024-04-02T04:52:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:10.858656+0000 mon.c (mon.2) 456 : audit 1 from='client.? 172.21.15.69:0/1759678489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:10.859064+0000 mon.a (mon.0) 1984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:10.861372+0000 mon.a (mon.0) 1985 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:10.903389+0000 mon.c (mon.2) 457 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:11.293755+0000 mon.b (mon.1) 190 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:11.295916+0000 mon.b (mon.1) 191 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[21528]: audit 2024-04-02T04:52:11.310590+0000 mon.a (mon.0) 1986 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: cluster 2024-04-02T04:52:10.843759+0000 mon.a (mon.0) 1981 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:10.850739+0000 mon.a (mon.0) 1982 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]': finished 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: cluster 2024-04-02T04:52:10.856192+0000 mon.a (mon.0) 1983 : cluster 0 osdmap e291: 8 total, 7 up, 8 in 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:10.858656+0000 mon.c (mon.2) 456 : audit 1 from='client.? 172.21.15.69:0/1759678489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:10.859064+0000 mon.a (mon.0) 1984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:10.861372+0000 mon.a (mon.0) 1985 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:10.903389+0000 mon.c (mon.2) 457 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:11.293755+0000 mon.b (mon.1) 190 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:11.295916+0000 mon.b (mon.1) 191 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:12.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:11 smithi069 bash[28904]: audit 2024-04-02T04:52:11.310590+0000 mon.a (mon.0) 1986 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:12.072 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:12.100 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:12.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:12.213 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: cluster 2024-04-02T04:52:10.843759+0000 mon.a (mon.0) 1981 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:10.850739+0000 mon.a (mon.0) 1982 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-27", "mode": "writeback"}]': finished 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: cluster 2024-04-02T04:52:10.856192+0000 mon.a (mon.0) 1983 : cluster 0 osdmap e291: 8 total, 7 up, 8 in 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:10.858656+0000 mon.c (mon.2) 456 : audit 1 from='client.? 172.21.15.69:0/1759678489' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:10.859064+0000 mon.a (mon.0) 1984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:10.861372+0000 mon.a (mon.0) 1985 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:10.903389+0000 mon.c (mon.2) 457 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:11.293755+0000 mon.b (mon.1) 190 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:11.295916+0000 mon.b (mon.1) 191 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:12.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:11 smithi098 bash[23790]: audit 2024-04-02T04:52:11.310590+0000 mon.a (mon.0) 1986 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:12.228 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:12.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:12.352 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:12.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:12.475 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- 172.21.15.69:0/2320483240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf281022f0 msgr2=0x7fdf28102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/2320483240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf281022f0 0x7fdf28102750 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0098f0 tx=0x7fdf1c009c00 comp rx=0 tx=0).stop 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- 172.21.15.69:0/2320483240 shutdown_connections 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/2320483240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf2810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/2320483240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf28108300 0x7fdf281086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/2320483240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf281022f0 0x7fdf28102750 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- 172.21.15.69:0/2320483240 >> 172.21.15.69:0/2320483240 conn(0x7fdf28079310 msgr2=0x7fdf28079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- 172.21.15.69:0/2320483240 shutdown_connections 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- 172.21.15.69:0/2320483240 wait complete. 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 librados: starting msgr at 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 librados: starting objecter 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 Processor -- start 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- start start 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 librados: setting wanted keys 2024-04-02T04:52:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 librados: calling monclient init 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf281022f0 0x7fdf281981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf281986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 0x7fdf2819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28111a90 con 0x7fdf28108300 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28111910 con 0x7fdf281022f0 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf2d5da700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdf28111c10 con 0x7fdf28102c90 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf267fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf281986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 0x7fdf2819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf267fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf281986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56608/0 (socket says 172.21.15.69:56608) 2024-04-02T04:52:12.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 0x7fdf2819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42406/0 (socket says 172.21.15.69:42406) 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf267fc700 1 -- 172.21.15.69:0/1973832456 learned_addr learned my addr 172.21.15.69:0/1973832456 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf26ffd700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf281022f0 0x7fdf281981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 -- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 msgr2=0x7fdf281986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf281986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 -- 172.21.15.69:0/1973832456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf281022f0 msgr2=0x7fdf281981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf281022f0 0x7fdf281981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf277fe700 1 -- 172.21.15.69:0/1973832456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2819d170 con 0x7fdf28108300 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.507+0000 7fdf267fc700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf281986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf277fe700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 0x7fdf2819ca30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009cd0 tx=0x7fdf2000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 10 client.?.objecter ms_handle_connect 0x7fdf28108300 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:12.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2000cdc0 con 0x7fdf28108300 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fdf20003a00 con 0x7fdf28108300 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf200044f0 con 0x7fdf28108300 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf2819d400 con 0x7fdf28108300 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf2d5da700 10 client.16149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf281a4ce0 con 0x7fdf28108300 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf2d5da700 1 librados: init done 2024-04-02T04:52:12.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf2810a6e0 con 0x7fdf28108300 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fdf2001a020 con 0x7fdf28108300 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf1006f5e0 0x7fdf10071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(292..292 src has 74..292) ==== 11439+0+0 (secure 0 0 0) 0x7fdf2008e8e0 con 0x7fdf28108300 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 10 client.16149.objecter ms_dispatch 0x7fdf28001550 osd_map(292..292 src has 74..292) 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 3 client.16149.objecter handle_osd_map got epochs [292,292] > 0 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 3 client.16149.objecter handle_osd_map decoding full epoch 292 2024-04-02T04:52:12.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf26ffd700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf1006f5e0 0x7fdf10071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 20 client.16149.objecter dump_active .. 0 homeless 2024-04-02T04:52:12.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf0ffff700 10 client.16149.objecter ms_handle_connect 0x7fdf1006f5e0 2024-04-02T04:52:12.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.511+0000 7fdf26ffd700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf1006f5e0 0x7fdf10071aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf18005fd0 tx=0x7fdf18005db0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.519+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf20010040 con 0x7fdf28108300 2024-04-02T04:52:12.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/1932058682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24108300 msgr2=0x7f4f241086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/1932058682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24108300 0x7f4f241086e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009990 tx=0x7f4f18009ca0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/1932058682 shutdown_connections 2024-04-02T04:52:12.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/1932058682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24102cb0 0x7f4f2410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/1932058682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24108300 0x7f4f241086e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/1932058682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f241022f0 0x7f4f24102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/1932058682 >> 172.21.15.69:0/1932058682 conn(0x7f4f240793a0 msgr2=0x7f4f240797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/1932058682 shutdown_connections 2024-04-02T04:52:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/1932058682 wait complete. 2024-04-02T04:52:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 librados: starting msgr at 2024-04-02T04:52:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 librados: starting objecter 2024-04-02T04:52:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 Processor -- start 2024-04-02T04:52:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- start start 2024-04-02T04:52:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 librados: setting wanted keys 2024-04-02T04:52:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 librados: calling monclient init 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f241022f0 0x7f4f241980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24102cb0 0x7f4f24198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f24108300 0x7f4f2419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f21d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24102cb0 0x7f4f24198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f22d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f24108300 0x7f4f2419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f21d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24102cb0 0x7f4f24198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56626/0 (socket says 172.21.15.69:56626) 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f21d9b700 1 -- 172.21.15.69:0/3298165942 learned_addr learned my addr 172.21.15.69:0/3298165942 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4f24111ac0 con 0x7f4f24108300 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f24111940 con 0x7f4f241022f0 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4f24111c40 con 0x7f4f24102cb0 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.551+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f241022f0 0x7f4f241980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f21d9b700 1 -- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f241022f0 msgr2=0x7f4f241980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f241022f0 0x7f4f241980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f21d9b700 1 -- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f24108300 msgr2=0x7f4f2419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f24108300 0x7f4f2419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f21d9b700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2419d100 con 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f24102cb0 0x7f4f24198630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009f60 tx=0x7f4f14009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 10 client.?.objecter ms_handle_connect 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14013be0 con 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4f140044a0 con 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2419d3f0 con 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14003680 con 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f28cb6700 10 client.26243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f24074470 con 0x7f4f24102cb0 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f28cb6700 1 librados: init done 2024-04-02T04:52:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f117fa700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4f08005100 con 0x7f4f24102cb0 2024-04-02T04:52:12.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4f14022020 con 0x7f4f24102cb0 2024-04-02T04:52:12.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.555+0000 7f4f137fe700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4f0c06f5e0 0x7f4f0c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(292..292 src has 74..292) ==== 11439+0+0 (secure 0 0 0) 0x7f4f140931d0 con 0x7f4f24102cb0 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 osd_map(292..292 src has 74..292) 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 3 client.26243.objecter handle_osd_map got epochs [292,292] > 0 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4f0c06f5e0 0x7f4f0c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 3 client.26243.objecter handle_osd_map decoding full epoch 292 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 20 client.26243.objecter dump_active .. 0 homeless 2024-04-02T04:52:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v292) ==== 54+0+16 (secure 0 0 0) 0x7f4f14095770 con 0x7f4f24102cb0 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f0c06f5e0 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4f0c06f5e0 0x7f4f0c071aa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006010 tx=0x7f4f1801a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 1 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4f24062e50 0x7f4f24063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 1 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 0 1 2024-04-02T04:52:12.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241039b0 osd=0 4 2024-04-02T04:52:12.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 10 client.26243.objecter _send_command 1 2024-04-02T04:52:12.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f2404ff00 con 0x7f4f24062e50 2024-04-02T04:52:12.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f22d9d700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4f24062e50 0x7f4f24063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f22d9d700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4f24062e50 0x7f4f24063230 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.559+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f24062e50 2024-04-02T04:52:12.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.563+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f2404ff00 con 0x7f4f24062e50 2024-04-02T04:52:12.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.563+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 1: 0 ) 2024-04-02T04:52:12.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.563+0000 7f4f137fe700 10 client.26243.objecter _finish_command 1 = system:0 2024-04-02T04:52:12.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.563+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241039b0 osd=0 4 2024-04-02T04:52:12.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.563+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 0 1 2024-04-02T04:52:12.578 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 2 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 2 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241039b0 osd=0 3 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 0 2 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241039b0 osd=0 4 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 10 client.26243.objecter _send_command 2 2024-04-02T04:52:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.583+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f24199450 con 0x7f4f24062e50 2024-04-02T04:52:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f24199450 con 0x7f4f24062e50 2024-04-02T04:52:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 10 client.26243.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241039b0 osd=0 4 2024-04-02T04:52:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 0 2 2024-04-02T04:52:12.584 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4f24109830 0x7f4f24109c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 1 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f24062b20 osd=1 4 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 10 client.26243.objecter _send_command 3 2024-04-02T04:52:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4f24062c90 con 0x7f4f24109830 2024-04-02T04:52:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4f24109830 0x7f4f24109c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f24109830 2024-04-02T04:52:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4f24109830 0x7f4f24109c10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f1800bda0 con 0x7f4f24109830 2024-04-02T04:52:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 3: 0 ) 2024-04-02T04:52:12.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 10 client.26243.objecter _finish_command 3 = system:0 2024-04-02T04:52:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f24062b20 osd=1 4 2024-04-02T04:52:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.587+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 1 3 2024-04-02T04:52:12.601 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 4 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 4 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f24062b20 osd=1 3 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 1 4 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f24062b20 osd=1 4 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 10 client.26243.objecter _send_command 4 2024-04-02T04:52:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.607+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f241092b0 con 0x7f4f24109830 2024-04-02T04:52:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f241092b0 con 0x7f4f24109830 2024-04-02T04:52:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 10 client.26243.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f24062b20 osd=1 4 2024-04-02T04:52:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 1 4 2024-04-02T04:52:12.609 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 5 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4f2402d690 0x7f4f241a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 5 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 2 5 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241096b0 osd=2 4 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 10 client.26243.objecter _send_command 5 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4f241acc70 con 0x7f4f2402d690 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4f2402d690 0x7f4f241a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f2402d690 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4f2402d690 0x7f4f241a9e10 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f241acc70 con 0x7f4f2402d690 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 5: 0 ) 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 10 client.26243.objecter _finish_command 5 = system:0 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241096b0 osd=2 4 2024-04-02T04:52:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.611+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 2 5 2024-04-02T04:52:12.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 6 2024-04-02T04:52:12.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 6 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241096b0 osd=2 3 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 2 6 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241096b0 osd=2 4 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 10 client.26243.objecter _send_command 6 2024-04-02T04:52:12.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.643+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f2402cc60 con 0x7f4f2402d690 2024-04-02T04:52:12.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f2402cc60 con 0x7f4f2402d690 2024-04-02T04:52:12.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 10 client.26243.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241096b0 osd=2 4 2024-04-02T04:52:12.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 2 6 2024-04-02T04:52:12.645 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:52:12.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 7 2024-04-02T04:52:12.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4f241a7960 0x7f4f241ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 7 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 3 7 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f2402cc60 osd=3 4 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 10 client.26243.objecter _send_command 7 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4f241b03c0 con 0x7f4f241a7960 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f22d9d700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4f241a7960 0x7f4f241ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f241a7960 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f22d9d700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4f241a7960 0x7f4f241ad4e0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f241b03c0 con 0x7f4f241a7960 2024-04-02T04:52:12.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 7: 0 ) 2024-04-02T04:52:12.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 10 client.26243.objecter _finish_command 7 = system:0 2024-04-02T04:52:12.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f2402cc60 osd=3 4 2024-04-02T04:52:12.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.647+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 3 7 2024-04-02T04:52:12.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 8 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 8 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f2402cc60 osd=3 3 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 3 8 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f2402cc60 osd=3 4 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 10 client.26243.objecter _send_command 8 2024-04-02T04:52:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f241a6ed0 con 0x7f4f241a7960 2024-04-02T04:52:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f241a6ed0 con 0x7f4f241a7960 2024-04-02T04:52:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f137fe700 10 client.26243.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f2402cc60 osd=3 4 2024-04-02T04:52:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 3 8 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 9 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4f241ab030 0x7f4f241ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 9 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 4 9 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6c80 osd=4 4 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 10 client.26243.objecter _send_command 9 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4f241a6df0 con 0x7f4f241ab030 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.667+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4f241ab030 0x7f4f241ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f241ab030 2024-04-02T04:52:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f2259c700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4f241ab030 0x7f4f241ada20 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f241a6df0 con 0x7f4f241ab030 2024-04-02T04:52:12.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 9: 0 ) 2024-04-02T04:52:12.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f137fe700 10 client.26243.objecter _finish_command 9 = system:0 2024-04-02T04:52:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241a6c80 osd=4 4 2024-04-02T04:52:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.671+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 4 9 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 10 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 10 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241a6c80 osd=4 3 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 4 10 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6c80 osd=4 4 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 10 client.26243.objecter _send_command 10 2024-04-02T04:52:12.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.691+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f241a6df0 con 0x7f4f241ab030 2024-04-02T04:52:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f241a6df0 con 0x7f4f241ab030 2024-04-02T04:52:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f137fe700 10 client.26243.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241a6c80 osd=4 4 2024-04-02T04:52:12.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 4 10 2024-04-02T04:52:12.693 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 11 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4f241ae4d0 0x7f4f241ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 11 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 5 11 2024-04-02T04:52:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6df0 osd=5 4 2024-04-02T04:52:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 10 client.26243.objecter _send_command 11 2024-04-02T04:52:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4f241aef90 con 0x7f4f241ae4d0 2024-04-02T04:52:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4f241ae4d0 0x7f4f241ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f241ae4d0 2024-04-02T04:52:12.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.695+0000 7f4f21d9b700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4f241ae4d0 0x7f4f241ae930 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.699+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f241aef90 con 0x7f4f241ae4d0 2024-04-02T04:52:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.699+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 11: 0 ) 2024-04-02T04:52:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.699+0000 7f4f137fe700 10 client.26243.objecter _finish_command 11 = system:0 2024-04-02T04:52:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.699+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241a6df0 osd=5 4 2024-04-02T04:52:12.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.699+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 5 11 2024-04-02T04:52:12.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 12 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 12 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241a6df0 osd=5 3 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 5 12 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241a6df0 osd=5 4 2024-04-02T04:52:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 10 client.26243.objecter _send_command 12 2024-04-02T04:52:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f241b45f0 con 0x7f4f241ae4d0 2024-04-02T04:52:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f241b45f0 con 0x7f4f241ae4d0 2024-04-02T04:52:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f137fe700 10 client.26243.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241a6df0 osd=5 4 2024-04-02T04:52:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 5 12 2024-04-02T04:52:12.735 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:52:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 13 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4f241b4c70 0x7f4f241b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 13 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 6 13 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241b45f0 osd=6 4 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 10 client.26243.objecter _send_command 13 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4f241ba100 con 0x7f4f241b4c70 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.735+0000 7f4f22d9d700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4f241b4c70 0x7f4f241b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f137fe700 10 client.26243.objecter ms_handle_connect 0x7f4f241b4c70 2024-04-02T04:52:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f22d9d700 1 --2- 172.21.15.69:0/3298165942 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4f241b4c70 0x7f4f241b50b0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f241ba100 con 0x7f4f241b4c70 2024-04-02T04:52:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 13: 0 ) 2024-04-02T04:52:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f137fe700 10 client.26243.objecter _finish_command 13 = system:0 2024-04-02T04:52:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241b45f0 osd=6 4 2024-04-02T04:52:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.739+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 6 13 2024-04-02T04:52:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 14 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 20 client.26243.objecter _get_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_remove -1 14 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 20 client.26243.objecter get_session s=0x7f4f241b45f0 osd=6 3 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign 6 14 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 20 client.26243.objecter put_session s=0x7f4f241b45f0 osd=6 4 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 10 client.26243.objecter _send_command 14 2024-04-02T04:52:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.771+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4f241b55f0 con 0x7f4f241b4c70 2024-04-02T04:52:12.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4f241b55f0 con 0x7f4f241b4c70 2024-04-02T04:52:12.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f137fe700 10 client.26243.objecter ms_dispatch 0x7f4f24001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:12.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f137fe700 10 client.26243.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:12.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f137fe700 20 client.26243.objecter put_session s=0x7f4f241b45f0 osd=6 4 2024-04-02T04:52:12.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f137fe700 15 client.26243.objecter _session_command_op_remove 6 14 2024-04-02T04:52:12.773 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f28cb6700 10 client.26243.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f28cb6700 15 client.26243.objecter _session_command_op_assign -1 15 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f28cb6700 20 client.26243.objecter _get_session osd=-1 returning homeless 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f28cb6700 10 client.26243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=293}) -- 0x7f4f241b6980 con 0x7f4f24102cb0 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f28cb6700 1 -- 172.21.15.69:0/3298165942 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f241b55f0 con 0x7f4f24102cb0 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f137fe700 1 -- 172.21.15.69:0/3298165942 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_get_version_reply(handle=1 version=292) ==== 24+0+0 (secure 0 0 0) 0x7f4f1409a070 con 0x7f4f24102cb0 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f20d99700 10 client.26243.objecter _check_command_map_dne tid 15 current 292 map_dne_bound 292 2024-04-02T04:52:12.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f20d99700 10 client.26243.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.775+0000 7f4f20d99700 15 client.26243.objecter _session_command_op_remove -1 15 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:52:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:52:12.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.815+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fdf281994c0 con 0x7fdf28108300 2024-04-02T04:52:12.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.871+0000 7fdf0ffff700 1 -- 172.21.15.69:0/1973832456 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v293) ==== 97+0+0 (secure 0 0 0) 0x7fdf2005afc0 con 0x7fdf28108300 2024-04-02T04:52:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:12.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.871+0000 7fdf2d5da700 10 librados: watch_flush enter 2024-04-02T04:52:12.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 10 librados: watch_flush exit 2024-04-02T04:52:12.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 20 client.16149.objecter shutdown clearing up homeless session... 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 10 client.16149.objecter successfully canceled tick 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf1006f5e0 msgr2=0x7fdf10071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf1006f5e0 0x7fdf10071aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf18005fd0 tx=0x7fdf18005db0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 msgr2=0x7fdf2819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 0x7fdf2819ca30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009cd0 tx=0x7fdf2000c620 comp rx=0 tx=0).stop 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 shutdown_connections 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf1006f5e0 0x7fdf10071aa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf28108300 0x7fdf2819ca30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf281022f0 0x7fdf281981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 --2- 172.21.15.69:0/1973832456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf28102c90 0x7fdf281986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 >> 172.21.15.69:0/1973832456 conn(0x7fdf28079310 msgr2=0x7fdf28100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 shutdown_connections 2024-04-02T04:52:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 -- 172.21.15.69:0/1973832456 wait complete. 2024-04-02T04:52:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:12.875+0000 7fdf2d5da700 1 librados: shutdown 2024-04-02T04:52:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:52:12.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:11.864880+0000 mon.a (mon.0) 1987 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:11.865112+0000 mon.a (mon.0) 1988 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: cluster 2024-04-02T04:52:11.874259+0000 mon.a (mon.0) 1989 : cluster 0 osdmap e292: 8 total, 7 up, 8 in 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:11.881254+0000 mon.a (mon.0) 1990 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:11.906952+0000 mon.c (mon.2) 458 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:11.947055+0000 mon.c (mon.2) 459 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:11.947393+0000 mon.a (mon.0) 1991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: cluster 2024-04-02T04:52:11.984474+0000 mgr.y (mgr.14538) 340 : cluster 0 pgmap v415: 388 pgs: 32 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:12.559251+0000 mon.c (mon.2) 460 : audit 0 from='client.? 172.21.15.69:0/3298165942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: audit 2024-04-02T04:52:12.818465+0000 mon.a (mon.0) 1992 : audit 1 from='client.? 172.21.15.69:0/1973832456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:52:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:12 smithi098 bash[23790]: cluster 2024-04-02T04:52:12.864833+0000 mon.a (mon.0) 1993 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:12.995 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:12.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:13.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:13.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:13.126 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:52:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:13.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:11.864880+0000 mon.a (mon.0) 1987 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:11.865112+0000 mon.a (mon.0) 1988 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: cluster 2024-04-02T04:52:11.874259+0000 mon.a (mon.0) 1989 : cluster 0 osdmap e292: 8 total, 7 up, 8 in 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:11.881254+0000 mon.a (mon.0) 1990 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:11.906952+0000 mon.c (mon.2) 458 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:11.947055+0000 mon.c (mon.2) 459 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:11.947393+0000 mon.a (mon.0) 1991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: cluster 2024-04-02T04:52:11.984474+0000 mgr.y (mgr.14538) 340 : cluster 0 pgmap v415: 388 pgs: 32 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:12.559251+0000 mon.c (mon.2) 460 : audit 0 from='client.? 172.21.15.69:0/3298165942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: audit 2024-04-02T04:52:12.818465+0000 mon.a (mon.0) 1992 : audit 1 from='client.? 172.21.15.69:0/1973832456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:52:13.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[28904]: cluster 2024-04-02T04:52:12.864833+0000 mon.a (mon.0) 1993 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:11.864880+0000 mon.a (mon.0) 1987 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-72153-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:11.865112+0000 mon.a (mon.0) 1988 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: cluster 2024-04-02T04:52:11.874259+0000 mon.a (mon.0) 1989 : cluster 0 osdmap e292: 8 total, 7 up, 8 in 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:11.881254+0000 mon.a (mon.0) 1990 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]: dispatch 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:11.906952+0000 mon.c (mon.2) 458 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:11.947055+0000 mon.c (mon.2) 459 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:11.947393+0000 mon.a (mon.0) 1991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: cluster 2024-04-02T04:52:11.984474+0000 mgr.y (mgr.14538) 340 : cluster 0 pgmap v415: 388 pgs: 32 unknown, 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 587 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 276/1521 objects degraded (18.146%) 2024-04-02T04:52:13.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:12.559251+0000 mon.c (mon.2) 460 : audit 0 from='client.? 172.21.15.69:0/3298165942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:13.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: audit 2024-04-02T04:52:12.818465+0000 mon.a (mon.0) 1992 : audit 1 from='client.? 172.21.15.69:0/1973832456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:52:13.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:12 smithi069 bash[21528]: cluster 2024-04-02T04:52:12.864833+0000 mon.a (mon.0) 1993 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:13.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:13.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:13.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:13.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:13.520 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:13.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:13.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:13.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:13.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:13.881 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:13.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:13.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:14.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: cluster 2024-04-02T04:52:12.865941+0000 mon.a (mon.0) 1994 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.871588+0000 mon.a (mon.0) 1995 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.871667+0000 mon.a (mon.0) 1996 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.871744+0000 mon.a (mon.0) 1997 : audit 1 from='client.? 172.21.15.69:0/1973832456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: cluster 2024-04-02T04:52:12.876832+0000 mon.a (mon.0) 1998 : cluster 0 osdmap e293: 8 total, 7 up, 8 in 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.880657+0000 mon.c (mon.2) 461 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:14.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.881015+0000 mon.a (mon.0) 1999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.897658+0000 mon.c (mon.2) 462 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.897973+0000 mon.a (mon.0) 2000 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.898308+0000 mon.b (mon.1) 192 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.898640+0000 mon.c (mon.2) 463 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.898870+0000 mon.a (mon.0) 2001 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.899493+0000 mon.c (mon.2) 464 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.899790+0000 mon.a (mon.0) 2002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.900294+0000 mon.b (mon.1) 193 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.901291+0000 mon.b (mon.1) 194 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.902743+0000 mon.a (mon.0) 2003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.904576+0000 mon.a (mon.0) 2004 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.905575+0000 mon.a (mon.0) 2005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:12.908205+0000 mon.c (mon.2) 465 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[21528]: audit 2024-04-02T04:52:13.124556+0000 mgr.y (mgr.14538) 341 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: cluster 2024-04-02T04:52:12.865941+0000 mon.a (mon.0) 1994 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.871588+0000 mon.a (mon.0) 1995 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.871667+0000 mon.a (mon.0) 1996 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.871744+0000 mon.a (mon.0) 1997 : audit 1 from='client.? 172.21.15.69:0/1973832456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: cluster 2024-04-02T04:52:12.876832+0000 mon.a (mon.0) 1998 : cluster 0 osdmap e293: 8 total, 7 up, 8 in 2024-04-02T04:52:14.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.880657+0000 mon.c (mon.2) 461 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.881015+0000 mon.a (mon.0) 1999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.897658+0000 mon.c (mon.2) 462 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.897973+0000 mon.a (mon.0) 2000 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.898308+0000 mon.b (mon.1) 192 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.898640+0000 mon.c (mon.2) 463 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.898870+0000 mon.a (mon.0) 2001 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.899493+0000 mon.c (mon.2) 464 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.899790+0000 mon.a (mon.0) 2002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.900294+0000 mon.b (mon.1) 193 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.901291+0000 mon.b (mon.1) 194 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.902743+0000 mon.a (mon.0) 2003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.904576+0000 mon.a (mon.0) 2004 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.905575+0000 mon.a (mon.0) 2005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:12.908205+0000 mon.c (mon.2) 465 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:13 smithi069 bash[28904]: audit 2024-04-02T04:52:13.124556+0000 mgr.y (mgr.14538) 341 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: cluster 2024-04-02T04:52:12.865941+0000 mon.a (mon.0) 1994 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.871588+0000 mon.a (mon.0) 1995 : audit 1 from='client.? 172.21.15.69:0/1187512078' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-72145-35"}]': finished 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.871667+0000 mon.a (mon.0) 1996 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.871744+0000 mon.a (mon.0) 1997 : audit 1 from='client.? 172.21.15.69:0/1973832456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: cluster 2024-04-02T04:52:12.876832+0000 mon.a (mon.0) 1998 : cluster 0 osdmap e293: 8 total, 7 up, 8 in 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.880657+0000 mon.c (mon.2) 461 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:14.032 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.881015+0000 mon.a (mon.0) 1999 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.897658+0000 mon.c (mon.2) 462 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.897973+0000 mon.a (mon.0) 2000 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.898308+0000 mon.b (mon.1) 192 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.898640+0000 mon.c (mon.2) 463 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.898870+0000 mon.a (mon.0) 2001 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.899493+0000 mon.c (mon.2) 464 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.899790+0000 mon.a (mon.0) 2002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.900294+0000 mon.b (mon.1) 193 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.901291+0000 mon.b (mon.1) 194 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.902743+0000 mon.a (mon.0) 2003 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.904576+0000 mon.a (mon.0) 2004 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.905575+0000 mon.a (mon.0) 2005 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:12.908205+0000 mon.c (mon.2) 465 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:14.033 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:13 smithi098 bash[23790]: audit 2024-04-02T04:52:13.124556+0000 mgr.y (mgr.14538) 341 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:14.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:14.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:14.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:14.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:14.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:14.432 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:14.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.561 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:14.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.690 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:14.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:14.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.952 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:14.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:15.056 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:15.086 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:52:15.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:52:15.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:15.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:15.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:15.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:15.591 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:15.660 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:15.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:15.817 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:15.839 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:15.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:15.964 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:15.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:16.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:16.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:16.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:16.211 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: cluster 2024-04-02T04:52:13.872786+0000 mon.a (mon.0) 2006 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.885538+0000 mon.a (mon.0) 2007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]': finished 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.885756+0000 mon.a (mon.0) 2008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.885971+0000 mon.a (mon.0) 2009 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.892323+0000 mon.b (mon.1) 195 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: cluster 2024-04-02T04:52:13.893501+0000 mon.a (mon.0) 2010 : cluster 0 osdmap e294: 8 total, 7 up, 8 in 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.901627+0000 mon.a (mon.0) 2011 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.901686+0000 mon.c (mon.2) 466 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.903417+0000 mon.a (mon.0) 2012 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: audit 2024-04-02T04:52:13.909661+0000 mon.c (mon.2) 467 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: cluster 2024-04-02T04:52:13.985374+0000 mgr.y (mgr.14538) 342 : cluster 0 pgmap v418: 356 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 91 active+undersized, 225 active+clean, 37 active+undersized+degraded; 587 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s; 278/1524 objects degraded (18.241%) 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[28904]: cluster 2024-04-02T04:52:14.811988+0000 mon.a (mon.0) 2013 : cluster 3 Health check update: Degraded data redundancy: 276/1521 objects degraded (18.146%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: cluster 2024-04-02T04:52:13.872786+0000 mon.a (mon.0) 2006 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.885538+0000 mon.a (mon.0) 2007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]': finished 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.885756+0000 mon.a (mon.0) 2008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.885971+0000 mon.a (mon.0) 2009 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.892323+0000 mon.b (mon.1) 195 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: cluster 2024-04-02T04:52:13.893501+0000 mon.a (mon.0) 2010 : cluster 0 osdmap e294: 8 total, 7 up, 8 in 2024-04-02T04:52:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.901627+0000 mon.a (mon.0) 2011 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.901686+0000 mon.c (mon.2) 466 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.903417+0000 mon.a (mon.0) 2012 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: audit 2024-04-02T04:52:13.909661+0000 mon.c (mon.2) 467 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: cluster 2024-04-02T04:52:13.985374+0000 mgr.y (mgr.14538) 342 : cluster 0 pgmap v418: 356 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 91 active+undersized, 225 active+clean, 37 active+undersized+degraded; 587 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s; 278/1524 objects degraded (18.241%) 2024-04-02T04:52:16.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:16 smithi069 bash[21528]: cluster 2024-04-02T04:52:14.811988+0000 mon.a (mon.0) 2013 : cluster 3 Health check update: Degraded data redundancy: 276/1521 objects degraded (18.146%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:16.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:16.338 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:16.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:16.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: cluster 2024-04-02T04:52:13.872786+0000 mon.a (mon.0) 2006 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:16.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.885538+0000 mon.a (mon.0) 2007 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-27"}]': finished 2024-04-02T04:52:16.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.885756+0000 mon.a (mon.0) 2008 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-72145-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:16.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.885971+0000 mon.a (mon.0) 2009 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:16.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.892323+0000 mon.b (mon.1) 195 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:16.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: cluster 2024-04-02T04:52:13.893501+0000 mon.a (mon.0) 2010 : cluster 0 osdmap e294: 8 total, 7 up, 8 in 2024-04-02T04:52:16.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.901627+0000 mon.a (mon.0) 2011 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:16.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.901686+0000 mon.c (mon.2) 466 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:16.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.903417+0000 mon.a (mon.0) 2012 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:16.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: audit 2024-04-02T04:52:13.909661+0000 mon.c (mon.2) 467 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:16.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: cluster 2024-04-02T04:52:13.985374+0000 mgr.y (mgr.14538) 342 : cluster 0 pgmap v418: 356 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 91 active+undersized, 225 active+clean, 37 active+undersized+degraded; 587 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s; 278/1524 objects degraded (18.241%) 2024-04-02T04:52:16.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:16 smithi098 bash[23790]: cluster 2024-04-02T04:52:14.811988+0000 mon.a (mon.0) 2013 : cluster 3 Health check update: Degraded data redundancy: 276/1521 objects degraded (18.146%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:16.468 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:16.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:16.599 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:16.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:16.729 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:16.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:16.854 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:16.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:16.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:17.086 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:17.113 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:17.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:17.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:52:17.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:14.911486+0000 mon.c (mon.2) 468 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: cluster 2024-04-02T04:52:15.723665+0000 mon.a (mon.0) 2014 : cluster 0 osdmap e295: 8 total, 7 up, 8 in 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:15.912537+0000 mon.c (mon.2) 469 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: cluster 2024-04-02T04:52:15.986109+0000 mgr.y (mgr.14538) 343 : cluster 0 pgmap v420: 324 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 204 active+clean, 37 active+undersized+degraded; 587 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 745 B/s wr, 2 op/s; 278/1524 objects degraded (18.241%) 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:16.166905+0000 mon.a (mon.0) 2015 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:16.166982+0000 mon.a (mon.0) 2016 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: cluster 2024-04-02T04:52:16.169992+0000 mon.a (mon.0) 2017 : cluster 0 osdmap e296: 8 total, 7 up, 8 in 2024-04-02T04:52:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:16.192076+0000 mon.c (mon.2) 470 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:16.192518+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:17 smithi098 bash[23790]: audit 2024-04-02T04:52:16.913691+0000 mon.c (mon.2) 471 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:17.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:17.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:14.911486+0000 mon.c (mon.2) 468 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: cluster 2024-04-02T04:52:15.723665+0000 mon.a (mon.0) 2014 : cluster 0 osdmap e295: 8 total, 7 up, 8 in 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:15.912537+0000 mon.c (mon.2) 469 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: cluster 2024-04-02T04:52:15.986109+0000 mgr.y (mgr.14538) 343 : cluster 0 pgmap v420: 324 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 204 active+clean, 37 active+undersized+degraded; 587 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 745 B/s wr, 2 op/s; 278/1524 objects degraded (18.241%) 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:16.166905+0000 mon.a (mon.0) 2015 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:16.166982+0000 mon.a (mon.0) 2016 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: cluster 2024-04-02T04:52:16.169992+0000 mon.a (mon.0) 2017 : cluster 0 osdmap e296: 8 total, 7 up, 8 in 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:16.192076+0000 mon.c (mon.2) 470 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:16.192518+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:17.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[28904]: audit 2024-04-02T04:52:16.913691+0000 mon.c (mon.2) 471 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:14.911486+0000 mon.c (mon.2) 468 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: cluster 2024-04-02T04:52:15.723665+0000 mon.a (mon.0) 2014 : cluster 0 osdmap e295: 8 total, 7 up, 8 in 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:15.912537+0000 mon.c (mon.2) 469 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: cluster 2024-04-02T04:52:15.986109+0000 mgr.y (mgr.14538) 343 : cluster 0 pgmap v420: 324 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 80 active+undersized, 204 active+clean, 37 active+undersized+degraded; 587 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 745 B/s wr, 2 op/s; 278/1524 objects degraded (18.241%) 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:16.166905+0000 mon.a (mon.0) 2015 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:16.166982+0000 mon.a (mon.0) 2016 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-72145-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: cluster 2024-04-02T04:52:16.169992+0000 mon.a (mon.0) 2017 : cluster 0 osdmap e296: 8 total, 7 up, 8 in 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:16.192076+0000 mon.c (mon.2) 470 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:16.192518+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:17.363 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:17 smithi069 bash[21528]: audit 2024-04-02T04:52:16.913691+0000 mon.c (mon.2) 471 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:17.368 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:17.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:17.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:52:17.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:17.497 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:17.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:17.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/1379195359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8314102cb0 msgr2=0x7f831410f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- 172.21.15.69:0/1379195359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8314102cb0 0x7f831410f120 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f830c009fc0 tx=0x7f830c00b330 comp rx=0 tx=0).stop 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/1379195359 shutdown_connections 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- 172.21.15.69:0/1379195359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8314102cb0 0x7f831410f120 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- 172.21.15.69:0/1379195359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 0x7f83141086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- 172.21.15.69:0/1379195359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83141022f0 0x7f8314102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/1379195359 >> 172.21.15.69:0/1379195359 conn(0x7f83140793a0 msgr2=0x7f83140797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/1379195359 shutdown_connections 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/1379195359 wait complete. 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 librados: starting msgr at 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 librados: starting objecter 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 Processor -- start 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- start start 2024-04-02T04:52:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 librados: setting wanted keys 2024-04-02T04:52:17.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 librados: calling monclient init 2024-04-02T04:52:17.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 0x7f8314074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:17.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 0x7f8314074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 0x7f8314074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8312ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 0x7f8314074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 0x7f8314074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8312ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 0x7f8314074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51204/0 (socket says 172.21.15.69:51204) 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 0x7f8314074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46362/0 (socket says 172.21.15.69:46362) 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8312ffd700 1 -- 172.21.15.69:0/205898330 learned_addr learned my addr 172.21.15.69:0/205898330 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8314111aa0 con 0x7f83141022f0 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8314111920 con 0x7f8314108300 2024-04-02T04:52:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8314111c20 con 0x7f8314102cb0 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 -- 172.21.15.69:0/205898330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 msgr2=0x7f8314074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83137fe700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 0x7f8314074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 0x7f8314074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 -- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 msgr2=0x7f8314074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 0x7f8314074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83141a64d0 con 0x7f8314102cb0 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8312ffd700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 0x7f8314074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83137fe700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 0x7f8314074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f83127fc700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 0x7f8314074a80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f830800eb90 tx=0x7f830800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f82fbfff700 10 client.?.objecter ms_handle_connect 0x7f8314102cb0 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f82fbfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830800ccf0 con 0x7f8314102cb0 2024-04-02T04:52:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f83080044a0 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83141a67c0 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.578+0000 7f8319625700 10 client.26261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f8319625700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83141a6ca0 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f8319625700 1 librados: init done 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308010430 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f83080173d0 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f4005120 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f82fc06f4b0 0x7f82fc071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(297..297 src has 74..297) ==== 12583+0+0 (secure 0 0 0) 0x7f8308092060 con 0x7f8314102cb0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 10 client.26261.objecter ms_dispatch 0x7f8314001550 osd_map(297..297 src has 74..297) 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 3 client.26261.objecter handle_osd_map got epochs [297,297] > 0 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 3 client.26261.objecter handle_osd_map decoding full epoch 297 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.582+0000 7f82fbfff700 20 client.26261.objecter dump_active .. 0 homeless 2024-04-02T04:52:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.586+0000 7f8312ffd700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f82fc06f4b0 0x7f82fc071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.586+0000 7f82fbfff700 10 client.26261.objecter ms_handle_connect 0x7f82fc06f4b0 2024-04-02T04:52:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.586+0000 7f8312ffd700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f82fc06f4b0 0x7f82fc071970 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8304005950 tx=0x7f830400b500 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.586+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f830805b980 con 0x7f8314102cb0 2024-04-02T04:52:17.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:17.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:17.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.780 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:17.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:17.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.810+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f82f40056e0 con 0x7f8314102cb0 2024-04-02T04:52:17.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.814+0000 7f82fbfff700 1 -- 172.21.15.69:0/205898330 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v297) ==== 81+0+77537 (secure 0 0 0) 0x7f830805f3d0 con 0x7f8314102cb0 2024-04-02T04:52:17.814 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 297, 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "created": "2024-04-02T04:33:43.348385+0000", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "modified": "2024-04-02T04:52:17.169365+0000", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "last_up_change": "2024-04-02T04:49:46.297901+0000", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "last_in_change": "2024-04-02T04:42:47.064987+0000", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_num": 5806080, 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_set": [ 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "nodeep-scrub", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "noscrub", 2024-04-02T04:52:17.815 INFO:teuthology.orchestra.run.smithi069.stdout: "pglog_hardlimit", 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "purged_snapdirs", 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_deletes", 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "sortbitwise" 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_version": 64, 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "full_ratio": 0.94999998807907104, 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_snapshot": "", 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_max": 251, 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd": 8, 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "require_min_compat_client": "luminous", 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "min_compat_client": "luminous", 2024-04-02T04:52:17.816 INFO:teuthology.orchestra.run.smithi069.stdout: "require_osd_release": "squid", 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_crimson": false, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "pools": [ 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 1, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".mgr", 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:38:57.671714+0000", 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.817 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 1, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 1, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 1, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 1, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 1, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.818 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "23", 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.819 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.820 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:52:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_max": 32, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_min": 1 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": {} 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 8, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 8, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.25, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 2, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".rgw.root", 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:52.801763+0000", 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.823 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.824 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "79", 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.825 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.826 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.827 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.828 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 3, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.log", 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:54.772439+0000", 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.829 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.830 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "81", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.831 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.832 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:52:17.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 4, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.control", 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:56.696111+0000", 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.835 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.836 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "83", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.837 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.838 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.839 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.2400000095367432, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.2400000095367432, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 5, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "datapool", 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:57.254108+0000", 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.841 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 3, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 3, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 3, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 3, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 3, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "89", 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.842 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 2, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 89, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.843 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.844 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd": {} 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.6700000762939453, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.6700000762939453, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:52:17.845 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 6, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.meta", 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:58.846023+0000", 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.846 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "85", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_bias": 4 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.851 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 10, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:11.754451+0000", 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.852 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:52:17.853 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.854 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.855 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.4900000095367432, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.4900000095367432, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.190000057220459, 2024-04-02T04:52:17.856 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.190000057220459, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 13, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosList_smithi069-72185-1", 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:13.034188+0000", 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.858 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.859 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.860 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.861 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 40, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace.1804289383", 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.538674+0000", 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.863 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.864 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.865 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.866 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:52:17.867 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 41, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test_handler_errorsmithi069-72850-1", 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.947398+0000", 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.868 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.869 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.870 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.871 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.872 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.873 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 43, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace2.846930886", 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:25.876339+0000", 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.874 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.875 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.876 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.877 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.4900000095367432, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.4900000095367432, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:52:17.878 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-6", 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:53.761868+0000", 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.879 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.880 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "294", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "293", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "293", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "293", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 10, 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 292, 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.881 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.882 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.883 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 249, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "SimpleWritePP_smithi069-72153-46", 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:52:15.274801+0000", 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:52:17.884 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 3, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 1, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 8, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 8, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 8, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 8, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 8, 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.885 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "296", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.886 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.887 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "testprofile-SimpleWritePP_smithi069-72153-46", 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 8192, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.888 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {} 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 250, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "FlushAsync_smithi069-72145-36", 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:52:15.723589+0000", 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 3, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 2, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.889 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 8, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 8, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 8, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 8, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 8, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.890 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "296", 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.891 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "testprofile-FlushAsync_smithi069-72145-36", 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.892 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 8192, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {} 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 251, 2024-04-02T04:52:17.893 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-29", 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:52:16.153360+0000", 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:52:17.894 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "297", 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:52:17.895 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:52:17.896 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:52:17.897 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:52:17.898 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b", 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 8, 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6802", 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6803", 2024-04-02T04:52:17.900 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6804", 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6805", 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.901 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6808", 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6809", 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.902 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6806", 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6807", 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.903 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6803/1087500595", 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6805/1087500595", 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6809/1087500595", 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6807/1087500595", 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a", 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.904 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 13, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6810", 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.905 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6811", 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6812", 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.906 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6813", 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6816", 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6817", 2024-04-02T04:52:17.907 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6814", 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6815", 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:52:17.908 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6811/2651666015", 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6813/2651666015", 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6817/2651666015", 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6815/2651666015", 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7", 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.909 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 19, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6818", 2024-04-02T04:52:17.910 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6819", 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6820", 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6821", 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.912 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6824", 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6825", 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.913 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6822", 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6823", 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6819/619998764", 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6821/619998764", 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6825/619998764", 2024-04-02T04:52:17.914 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6823/619998764", 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed", 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 29, 2024-04-02T04:52:17.915 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6826", 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6827", 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6828", 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6829", 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.917 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6832", 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6833", 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.918 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6830", 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6831", 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6827/207593208", 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6829/207593208", 2024-04-02T04:52:17.919 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6833/207593208", 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6831/207593208", 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852", 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.920 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 36, 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6800", 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6801", 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.921 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6802", 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6803", 2024-04-02T04:52:17.922 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6806", 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6807", 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.923 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6804", 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6805", 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:52:17.924 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6801/2065947829", 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6803/2065947829", 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6807/2065947829", 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6805/2065947829", 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe", 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 47, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.926 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6808", 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6809", 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.927 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6810", 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6811", 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.928 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6814", 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6815", 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.929 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6812", 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6813", 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.930 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6809/219578185", 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6811/219578185", 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6815/219578185", 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6813/219578185", 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727", 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:52:17.931 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 58, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 296, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.932 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6816", 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6817", 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6818", 2024-04-02T04:52:17.933 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6819", 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6822", 2024-04-02T04:52:17.934 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6823", 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6820", 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.935 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6821", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6817/1226172499", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6819/1226172499", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6823/1226172499", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6821/1226172499", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.936 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3", 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 0, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 69, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 151, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 152, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:52:17.937 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6824", 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6825", 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:52:17.938 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6826", 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6827", 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.939 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6830", 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6831", 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.940 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6828", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6829", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 148009079 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6825/148009079", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6827/148009079", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6831/148009079", 2024-04-02T04:52:17.941 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6829/148009079", 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "exists" 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_xinfo": [ 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.942 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:52:00.431456+0000", 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:18.051401+0000", 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.943 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:03.586018+0000", 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.944 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:52:08.941743+0000", 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:18.611731+0000", 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.945 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:51:16.385784+0000", 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.946 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:50:58.331556+0000", 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:49:15.075904+0000", 2024-04-02T04:52:17.947 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 150 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap": [], 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_items": [ 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "2.9", 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 7, 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.6", 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 5 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.f", 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:52:17.949 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.17", 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:52:17.950 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_primaries": [], 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_temp": [], 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_temp": [], 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklist": { 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2738867930": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1798147282": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/52352690": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2928651991": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2763215770": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/3657409442": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/879855771": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:52:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1025808168": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1168952113": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1892956550": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/54578327": "2024-04-03T04:44:51.699278+0000" 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "range_blocklist": {}, 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profiles": { 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "default": { 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:52:17.952 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van" 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "testprofile-FlushAsync_smithi069-72145-36": { 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-device-class": "", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-num-failure-domains": "0", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-osds-per-failure-domain": "0", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-root": "default", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van", 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: "w": "8" 2024-04-02T04:52:17.953 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "testprofile-SimpleWritePP_smithi069-72153-46": { 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-device-class": "", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-num-failure-domains": "0", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-osds-per-failure-domain": "0", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-root": "default", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van", 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: "w": "8" 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.954 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps_queue": [ 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 17, 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 36, 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.955 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 38, 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 66, 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 101, 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:52:17.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 7 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 10, 2024-04-02T04:52:17.958 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 103, 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.959 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 112, 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 9 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 120, 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 137, 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.961 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 145, 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 146, 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.962 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 154, 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.963 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 164, 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 3 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 165, 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:52:17.964 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 173, 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 176, 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 188, 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.966 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 189, 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 196, 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 209, 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 6 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 214, 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:52:17.968 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 234, 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 8, 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.969 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 245, 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 10, 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "new_removed_snaps": [], 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "new_purged_snaps": [], 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_node_flags": {}, 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "device_class_flags": {}, 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode": { 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_enabled": false, 2024-04-02T04:52:17.970 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_bucket_count": 0, 2024-04-02T04:52:17.971 INFO:teuthology.orchestra.run.smithi069.stdout: "degraded_stretch_mode": 0, 2024-04-02T04:52:17.971 INFO:teuthology.orchestra.run.smithi069.stdout: "recovering_stretch_mode": 0, 2024-04-02T04:52:17.971 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_bucket": 0 2024-04-02T04:52:17.971 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:17.971 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:52:17.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 10 librados: watch_flush enter 2024-04-02T04:52:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 10 librados: watch_flush exit 2024-04-02T04:52:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 20 client.26261.objecter shutdown clearing up homeless session... 2024-04-02T04:52:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 10 client.26261.objecter successfully canceled tick 2024-04-02T04:52:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f82fc06f4b0 msgr2=0x7f82fc071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f82fc06f4b0 0x7f82fc071970 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8304005950 tx=0x7f830400b500 comp rx=0 tx=0).stop 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 msgr2=0x7f8314074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 0x7f8314074a80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f830800eb90 tx=0x7f830800eea0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 shutdown_connections 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f82fc06f4b0 0x7f82fc071970 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8314108300 0x7f8314074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83141022f0 0x7f8314074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 --2- 172.21.15.69:0/205898330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8314102cb0 0x7f8314074a80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 >> 172.21.15.69:0/205898330 conn(0x7f83140793a0 msgr2=0x7f8314105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 shutdown_connections 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 -- 172.21.15.69:0/205898330 wait complete. 2024-04-02T04:52:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:17.818+0000 7f82f9ffb700 1 librados: shutdown 2024-04-02T04:52:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:18.104 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:18.129 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:18.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:18.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:18.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:18.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21528]: audit 2024-04-02T04:52:17.172595+0000 mon.a (mon.0) 2019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:18.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21528]: cluster 2024-04-02T04:52:17.176514+0000 mon.a (mon.0) 2020 : cluster 0 osdmap e297: 8 total, 7 up, 8 in 2024-04-02T04:52:18.369 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21528]: audit 2024-04-02T04:52:17.696030+0000 mon.a (mon.0) 2021 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21528]: audit 2024-04-02T04:52:17.696507+0000 mon.b (mon.1) 196 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21528]: audit 2024-04-02T04:52:17.814344+0000 mon.c (mon.2) 472 : audit 0 from='client.? 172.21.15.69:0/205898330' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21528]: audit 2024-04-02T04:52:17.914911+0000 mon.c (mon.2) 473 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[28904]: audit 2024-04-02T04:52:17.172595+0000 mon.a (mon.0) 2019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[28904]: cluster 2024-04-02T04:52:17.176514+0000 mon.a (mon.0) 2020 : cluster 0 osdmap e297: 8 total, 7 up, 8 in 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[28904]: audit 2024-04-02T04:52:17.696030+0000 mon.a (mon.0) 2021 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[28904]: audit 2024-04-02T04:52:17.696507+0000 mon.b (mon.1) 196 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[28904]: audit 2024-04-02T04:52:17.814344+0000 mon.c (mon.2) 472 : audit 0 from='client.? 172.21.15.69:0/205898330' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:52:18.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[28904]: audit 2024-04-02T04:52:17.914911+0000 mon.c (mon.2) 473 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:18.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:18 smithi098 bash[23790]: audit 2024-04-02T04:52:17.172595+0000 mon.a (mon.0) 2019 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:18 smithi098 bash[23790]: cluster 2024-04-02T04:52:17.176514+0000 mon.a (mon.0) 2020 : cluster 0 osdmap e297: 8 total, 7 up, 8 in 2024-04-02T04:52:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:18 smithi098 bash[23790]: audit 2024-04-02T04:52:17.696030+0000 mon.a (mon.0) 2021 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:52:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:18 smithi098 bash[23790]: audit 2024-04-02T04:52:17.696507+0000 mon.b (mon.1) 196 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:18 smithi098 bash[23790]: audit 2024-04-02T04:52:17.814344+0000 mon.c (mon.2) 472 : audit 0 from='client.? 172.21.15.69:0/205898330' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:52:18.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:18 smithi098 bash[23790]: audit 2024-04-02T04:52:17.914911+0000 mon.c (mon.2) 473 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:18.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:18.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:18.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:18.647 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:18.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:18.778 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:18.881 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:18.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:18.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:19.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:19.019 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:52:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:52:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:52:19.036 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:19.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:19.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: cluster 2024-04-02T04:52:17.989917+0000 mgr.y (mgr.14538) 344 : cluster 0 pgmap v423: 372 pgs: 97 active+undersized, 238 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 1000 KiB/s wr, 0 op/s; 278/1533 objects degraded (18.134%) 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: cluster 2024-04-02T04:52:18.191131+0000 mon.a (mon.0) 2022 : cluster 0 osdmap e298: 8 total, 7 up, 8 in 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: audit 2024-04-02T04:52:18.191326+0000 mon.b (mon.1) 197 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: audit 2024-04-02T04:52:18.193093+0000 mon.c (mon.2) 474 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: audit 2024-04-02T04:52:18.194668+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: audit 2024-04-02T04:52:18.196446+0000 mon.a (mon.0) 2024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: cluster 2024-04-02T04:52:18.693990+0000 mon.a (mon.0) 2025 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[28904]: audit 2024-04-02T04:52:18.915937+0000 mon.c (mon.2) 475 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: cluster 2024-04-02T04:52:17.989917+0000 mgr.y (mgr.14538) 344 : cluster 0 pgmap v423: 372 pgs: 97 active+undersized, 238 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 1000 KiB/s wr, 0 op/s; 278/1533 objects degraded (18.134%) 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: cluster 2024-04-02T04:52:18.191131+0000 mon.a (mon.0) 2022 : cluster 0 osdmap e298: 8 total, 7 up, 8 in 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: audit 2024-04-02T04:52:18.191326+0000 mon.b (mon.1) 197 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: audit 2024-04-02T04:52:18.193093+0000 mon.c (mon.2) 474 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: audit 2024-04-02T04:52:18.194668+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: audit 2024-04-02T04:52:18.196446+0000 mon.a (mon.0) 2024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: cluster 2024-04-02T04:52:18.693990+0000 mon.a (mon.0) 2025 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:19 smithi069 bash[21528]: audit 2024-04-02T04:52:18.915937+0000 mon.c (mon.2) 475 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:19.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:19.335 INFO:tasks.thrashosds.thrasher:j is {'epoch': 297, 'fsid': 'f978400e-f0a9-11ee-b647-cb9ed24678a4', 'created': '2024-04-02T04:33:43.348385+0000', 'modified': '2024-04-02T04:52:17.169365+0000', 'last_up_change': '2024-04-02T04:49:46.297901+0000', 'last_in_change': '2024-04-02T04:42:47.064987+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 64, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 251, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-02T04:43:52.801763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-02T04:43:54.772439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-02T04:43:56.696111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-04-02T04:43:57.254108+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-02T04:43:58.846023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-02T04:49:11.754451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi069-72185-1', 'create_time': '2024-04-02T04:49:13.034188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 40, 'pool_name': 'poolrace.1804289383', 'create_time': '2024-04-02T04:49:24.538674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 41, 'pool_name': 'test_handler_errorsmithi069-72850-1', 'create_time': '2024-04-02T04:49:24.947398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 43, 'pool_name': 'poolrace2.846930886', 'create_time': '2024-04-02T04:49:25.876339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 102, 'pool_name': 'test-rados-api-smithi069-72349-6', 'create_time': '2024-04-02T04:49:53.761868+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '294', 'last_force_op_resend': '293', 'last_force_op_resend_prenautilus': '293', 'last_force_op_resend_preluminous': '293', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 292, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 249, 'pool_name': 'SimpleWritePP_smithi069-72153-46', 'create_time': '2024-04-02T04:52:15.274801+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '296', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-SimpleWritePP_smithi069-72153-46', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 250, 'pool_name': 'FlushAsync_smithi069-72145-36', 'create_time': '2024-04-02T04:52:15.723589+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 2, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '296', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-FlushAsync_smithi069-72145-36', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 251, 'pool_name': 'test-rados-api-smithi069-72349-29', 'create_time': '2024-04-02T04:52:16.153360+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '297', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1087500595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1087500595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 1087500595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1087500595}]}, 'public_addr': '172.21.15.69:6803/1087500595', 'cluster_addr': '172.21.15.69:6805/1087500595', 'heartbeat_back_addr': '172.21.15.69:6809/1087500595', 'heartbeat_front_addr': '172.21.15.69:6807/1087500595', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'afb71182-d4f7-416d-ae44-90f28eab4b7a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6811', 'nonce': 2651666015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6812', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2651666015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6816', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6817', 'nonce': 2651666015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6815', 'nonce': 2651666015}]}, 'public_addr': '172.21.15.69:6811/2651666015', 'cluster_addr': '172.21.15.69:6813/2651666015', 'heartbeat_back_addr': '172.21.15.69:6817/2651666015', 'heartbeat_front_addr': '172.21.15.69:6815/2651666015', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6818', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6819', 'nonce': 619998764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6820', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6821', 'nonce': 619998764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6825', 'nonce': 619998764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6822', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 619998764}]}, 'public_addr': '172.21.15.69:6819/619998764', 'cluster_addr': '172.21.15.69:6821/619998764', 'heartbeat_back_addr': '172.21.15.69:6825/619998764', 'heartbeat_front_addr': '172.21.15.69:6823/619998764', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '20e4526d-2707-44d0-b13a-af36019c42ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6826', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 207593208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 207593208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 207593208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 207593208}]}, 'public_addr': '172.21.15.69:6827/207593208', 'cluster_addr': '172.21.15.69:6829/207593208', 'heartbeat_back_addr': '172.21.15.69:6833/207593208', 'heartbeat_front_addr': '172.21.15.69:6831/207593208', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 2065947829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 2065947829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 2065947829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 2065947829}]}, 'public_addr': '172.21.15.98:6801/2065947829', 'cluster_addr': '172.21.15.98:6803/2065947829', 'heartbeat_back_addr': '172.21.15.98:6807/2065947829', 'heartbeat_front_addr': '172.21.15.98:6805/2065947829', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '902326af-4a40-47c8-a3b8-d5d5536f90fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 219578185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 219578185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6814', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 219578185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6813', 'nonce': 219578185}]}, 'public_addr': '172.21.15.98:6809/219578185', 'cluster_addr': '172.21.15.98:6811/219578185', 'heartbeat_back_addr': '172.21.15.98:6815/219578185', 'heartbeat_front_addr': '172.21.15.98:6813/219578185', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 1226172499}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 1226172499}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 1226172499}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 1226172499}]}, 'public_addr': '172.21.15.98:6817/1226172499', 'cluster_addr': '172.21.15.98:6819/1226172499', 'heartbeat_back_addr': '172.21.15.98:6823/1226172499', 'heartbeat_front_addr': '172.21.15.98:6821/1226172499', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 151, 'down_at': 152, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 148009079}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 148009079}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 148009079}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 148009079}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 148009079}]}, 'public_addr': '172.21.15.98:6825/148009079', 'cluster_addr': '172.21.15.98:6827/148009079', 'heartbeat_back_addr': '172.21.15.98:6831/148009079', 'heartbeat_front_addr': '172.21.15.98:6829/148009079', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:52:00.431456+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:18.051401+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:03.586018+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:52:08.941743+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:18.611731+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:51:16.385784+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:50:58.331556+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:49:15.075904+0000', 'dead_epoch': 150}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.69:0/2738867930': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/1798147282': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/52352690': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/2928651991': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:6800/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/2763215770': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/3657409442': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/879855771': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1025808168': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1168952113': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/1892956550': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6801/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/54578327': '2024-04-03T04:44:51.699278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-FlushAsync_smithi069-72145-36': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-SimpleWritePP_smithi069-72153-46': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 36, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 38, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 66, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 101, 'snaps': [{'begin': 2, 'length': 4}, {'begin': 7, 'length': 2}]}, {'pool': 102, 'snaps': [{'begin': 2, 'length': 7}, {'begin': 10, 'length': 1}]}, {'pool': 103, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 2, 'length': 9}, {'begin': 14, 'length': 1}]}, {'pool': 120, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 137, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 145, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 146, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 164, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 173, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 196, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 209, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 10, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-02T04:52:19.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:52:19.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:19.446 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: cluster 2024-04-02T04:52:17.989917+0000 mgr.y (mgr.14538) 344 : cluster 0 pgmap v423: 372 pgs: 97 active+undersized, 238 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 1000 KiB/s wr, 0 op/s; 278/1533 objects degraded (18.134%) 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: cluster 2024-04-02T04:52:18.191131+0000 mon.a (mon.0) 2022 : cluster 0 osdmap e298: 8 total, 7 up, 8 in 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: audit 2024-04-02T04:52:18.191326+0000 mon.b (mon.1) 197 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: audit 2024-04-02T04:52:18.193093+0000 mon.c (mon.2) 474 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: audit 2024-04-02T04:52:18.194668+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: audit 2024-04-02T04:52:18.196446+0000 mon.a (mon.0) 2024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: cluster 2024-04-02T04:52:18.693990+0000 mon.a (mon.0) 2025 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:19.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:19 smithi098 bash[23790]: audit 2024-04-02T04:52:18.915937+0000 mon.c (mon.2) 475 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:19.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:19.568 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:19.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:19.696 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:19.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:19.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:19.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:19.959 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:19.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:20.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:20.088 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:20.191 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:20.225 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:20.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:20.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.191659+0000 mon.a (mon.0) 2026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.191856+0000 mon.a (mon.0) 2027 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.194195+0000 mon.b (mon.1) 198 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: cluster 2024-04-02T04:52:19.201090+0000 mon.a (mon.0) 2028 : cluster 0 osdmap e299: 8 total, 7 up, 8 in 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.204678+0000 mon.a (mon.0) 2029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.213533+0000 mon.c (mon.2) 476 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.227974+0000 mon.a (mon.0) 2030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.229893+0000 mon.c (mon.2) 477 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.230914+0000 mon.a (mon.0) 2031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: cluster 2024-04-02T04:52:19.815299+0000 mon.a (mon.0) 2032 : cluster 3 Health check update: Degraded data redundancy: 278/1533 objects degraded (18.134%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:19.917213+0000 mon.c (mon.2) 478 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:20.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:20.197005+0000 mon.a (mon.0) 2033 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:20.197200+0000 mon.a (mon.0) 2034 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: audit 2024-04-02T04:52:20.197366+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[21528]: cluster 2024-04-02T04:52:20.205534+0000 mon.a (mon.0) 2036 : cluster 0 osdmap e300: 8 total, 7 up, 8 in 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.191659+0000 mon.a (mon.0) 2026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.191856+0000 mon.a (mon.0) 2027 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.194195+0000 mon.b (mon.1) 198 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: cluster 2024-04-02T04:52:19.201090+0000 mon.a (mon.0) 2028 : cluster 0 osdmap e299: 8 total, 7 up, 8 in 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.204678+0000 mon.a (mon.0) 2029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.213533+0000 mon.c (mon.2) 476 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.227974+0000 mon.a (mon.0) 2030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:20.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.229893+0000 mon.c (mon.2) 477 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.230914+0000 mon.a (mon.0) 2031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: cluster 2024-04-02T04:52:19.815299+0000 mon.a (mon.0) 2032 : cluster 3 Health check update: Degraded data redundancy: 278/1533 objects degraded (18.134%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:19.917213+0000 mon.c (mon.2) 478 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:20.197005+0000 mon.a (mon.0) 2033 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:20.197200+0000 mon.a (mon.0) 2034 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: audit 2024-04-02T04:52:20.197366+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:20.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:20 smithi069 bash[28904]: cluster 2024-04-02T04:52:20.205534+0000 mon.a (mon.0) 2036 : cluster 0 osdmap e300: 8 total, 7 up, 8 in 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.191659+0000 mon.a (mon.0) 2026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.191856+0000 mon.a (mon.0) 2027 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.194195+0000 mon.b (mon.1) 198 : audit 1 from='client.? 172.21.15.69:0/1796860482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: cluster 2024-04-02T04:52:19.201090+0000 mon.a (mon.0) 2028 : cluster 0 osdmap e299: 8 total, 7 up, 8 in 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.204678+0000 mon.a (mon.0) 2029 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]: dispatch 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.213533+0000 mon.c (mon.2) 476 : audit 1 from='client.? 172.21.15.69:0/2499816773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.227974+0000 mon.a (mon.0) 2030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]: dispatch 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.229893+0000 mon.c (mon.2) 477 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.230914+0000 mon.a (mon.0) 2031 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: cluster 2024-04-02T04:52:19.815299+0000 mon.a (mon.0) 2032 : cluster 3 Health check update: Degraded data redundancy: 278/1533 objects degraded (18.134%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:19.917213+0000 mon.c (mon.2) 478 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:20.197005+0000 mon.a (mon.0) 2033 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-46"}]': finished 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:20.197200+0000 mon.a (mon.0) 2034 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-72145-36"}]': finished 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: audit 2024-04-02T04:52:20.197366+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:20 smithi098 bash[23790]: cluster 2024-04-02T04:52:20.205534+0000 mon.a (mon.0) 2036 : cluster 0 osdmap e300: 8 total, 7 up, 8 in 2024-04-02T04:52:20.484 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:20.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:20.687 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:52:20.687 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:52:20.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:20.840 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:20.866 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:20.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:20.995 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:20.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:21.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:21.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:21.253 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:21.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:21.383 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:21.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: cluster 2024-04-02T04:52:19.991728+0000 mgr.y (mgr.14538) 345 : cluster 0 pgmap v426: 356 pgs: 93 active+undersized, 226 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:21.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.209767+0000 mon.c (mon.2) 479 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:21.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.220256+0000 mon.a (mon.0) 2037 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.240053+0000 mon.a (mon.0) 2038 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.241012+0000 mon.c (mon.2) 480 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.241304+0000 mon.a (mon.0) 2039 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.241744+0000 mon.a (mon.0) 2040 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.242044+0000 mon.a (mon.0) 2041 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.242373+0000 mon.c (mon.2) 481 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.242614+0000 mon.a (mon.0) 2042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.243096+0000 mon.c (mon.2) 482 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.243415+0000 mon.a (mon.0) 2043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: cluster 2024-04-02T04:52:20.344383+0000 osd.6 (osd.6) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: cluster 2024-04-02T04:52:20.344510+0000 osd.6 (osd.6) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:20.918636+0000 mon.c (mon.2) 483 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.199706+0000 mon.a (mon.0) 2044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]': finished 2024-04-02T04:52:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.199981+0000 mon.a (mon.0) 2045 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.200167+0000 mon.a (mon.0) 2046 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: cluster 2024-04-02T04:52:21.207434+0000 mon.a (mon.0) 2047 : cluster 0 osdmap e301: 8 total, 7 up, 8 in 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.212155+0000 mon.c (mon.2) 484 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]: dispatch 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.212679+0000 mon.a (mon.0) 2048 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.213738+0000 mon.a (mon.0) 2049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]: dispatch 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.214468+0000 mon.c (mon.2) 485 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:21 smithi098 bash[23790]: audit 2024-04-02T04:52:21.215206+0000 mon.a (mon.0) 2050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:21.513 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:21.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: cluster 2024-04-02T04:52:19.991728+0000 mgr.y (mgr.14538) 345 : cluster 0 pgmap v426: 356 pgs: 93 active+undersized, 226 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:21.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.209767+0000 mon.c (mon.2) 479 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:21.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.220256+0000 mon.a (mon.0) 2037 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:21.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.240053+0000 mon.a (mon.0) 2038 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.241012+0000 mon.c (mon.2) 480 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.241304+0000 mon.a (mon.0) 2039 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.241744+0000 mon.a (mon.0) 2040 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.242044+0000 mon.a (mon.0) 2041 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.242373+0000 mon.c (mon.2) 481 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.242614+0000 mon.a (mon.0) 2042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.243096+0000 mon.c (mon.2) 482 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.243415+0000 mon.a (mon.0) 2043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: cluster 2024-04-02T04:52:20.344383+0000 osd.6 (osd.6) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: cluster 2024-04-02T04:52:20.344510+0000 osd.6 (osd.6) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:21.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:20.918636+0000 mon.c (mon.2) 483 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: cluster 2024-04-02T04:52:19.991728+0000 mgr.y (mgr.14538) 345 : cluster 0 pgmap v426: 356 pgs: 93 active+undersized, 226 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.209767+0000 mon.c (mon.2) 479 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.220256+0000 mon.a (mon.0) 2037 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.240053+0000 mon.a (mon.0) 2038 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.241012+0000 mon.c (mon.2) 480 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.241304+0000 mon.a (mon.0) 2039 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.241744+0000 mon.a (mon.0) 2040 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.242044+0000 mon.a (mon.0) 2041 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.242373+0000 mon.c (mon.2) 481 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.242614+0000 mon.a (mon.0) 2042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.243096+0000 mon.c (mon.2) 482 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.243415+0000 mon.a (mon.0) 2043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: cluster 2024-04-02T04:52:20.344383+0000 osd.6 (osd.6) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: cluster 2024-04-02T04:52:20.344510+0000 osd.6 (osd.6) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:20.918636+0000 mon.c (mon.2) 483 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.199706+0000 mon.a (mon.0) 2044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]': finished 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.199981+0000 mon.a (mon.0) 2045 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.200167+0000 mon.a (mon.0) 2046 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: cluster 2024-04-02T04:52:21.207434+0000 mon.a (mon.0) 2047 : cluster 0 osdmap e301: 8 total, 7 up, 8 in 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.212155+0000 mon.c (mon.2) 484 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.212679+0000 mon.a (mon.0) 2048 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.213738+0000 mon.a (mon.0) 2049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]: dispatch 2024-04-02T04:52:21.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.214468+0000 mon.c (mon.2) 485 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[28904]: audit 2024-04-02T04:52:21.215206+0000 mon.a (mon.0) 2050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.199706+0000 mon.a (mon.0) 2044 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-29"}]': finished 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.199981+0000 mon.a (mon.0) 2045 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-72153-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.200167+0000 mon.a (mon.0) 2046 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: cluster 2024-04-02T04:52:21.207434+0000 mon.a (mon.0) 2047 : cluster 0 osdmap e301: 8 total, 7 up, 8 in 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.212155+0000 mon.c (mon.2) 484 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]: dispatch 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.212679+0000 mon.a (mon.0) 2048 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.213738+0000 mon.a (mon.0) 2049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]: dispatch 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.214468+0000 mon.c (mon.2) 485 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:21 smithi069 bash[21528]: audit 2024-04-02T04:52:21.215206+0000 mon.a (mon.0) 2050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:21.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:21.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-02T04:52:21.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:21.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:21.905 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:21.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:22.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:22.036 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:22.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:22.166 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:22.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:22.288 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:22.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:22.414 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:22.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:22.542 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:22.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:22.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:22.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:22.899 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:22.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:23.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:23.047 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:23.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:23.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:23.212 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:23.212 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:23.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:23.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:23.415 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:52:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:52:23.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:23 smithi098 bash[23790]: audit 2024-04-02T04:52:21.919982+0000 mon.c (mon.2) 486 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:23.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:23 smithi098 bash[23790]: cluster 2024-04-02T04:52:22.200298+0000 mon.a (mon.0) 2051 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:23.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:23 smithi098 bash[23790]: audit 2024-04-02T04:52:22.201919+0000 mon.a (mon.0) 2052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]': finished 2024-04-02T04:52:23.431 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:23 smithi069 bash[21528]: audit 2024-04-02T04:52:21.919982+0000 mon.c (mon.2) 486 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:23 smithi069 bash[21528]: cluster 2024-04-02T04:52:22.200298+0000 mon.a (mon.0) 2051 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:23 smithi069 bash[21528]: audit 2024-04-02T04:52:22.201919+0000 mon.a (mon.0) 2052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]': finished 2024-04-02T04:52:23.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:23 smithi069 bash[28904]: audit 2024-04-02T04:52:21.919982+0000 mon.c (mon.2) 486 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:23.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:23 smithi069 bash[28904]: cluster 2024-04-02T04:52:22.200298+0000 mon.a (mon.0) 2051 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:23.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:23 smithi069 bash[28904]: audit 2024-04-02T04:52:22.201919+0000 mon.a (mon.0) 2052 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-29", "mode": "writeback"}]': finished 2024-04-02T04:52:23.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:23.559 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:23.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:23.792 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:23.819 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:23.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:23.947 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:23.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:24.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:24.076 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:24.180 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:24.206 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:24.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: cluster 2024-04-02T04:52:21.993540+0000 mgr.y (mgr.14538) 346 : cluster 0 pgmap v429: 356 pgs: 93 active+undersized, 226 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: audit 2024-04-02T04:52:22.921164+0000 mon.c (mon.2) 487 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: cluster 2024-04-02T04:52:23.082160+0000 mon.a (mon.0) 2053 : cluster 0 osdmap e302: 8 total, 7 up, 8 in 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: audit 2024-04-02T04:52:23.132639+0000 mgr.y (mgr.14538) 347 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: audit 2024-04-02T04:52:23.205135+0000 mon.a (mon.0) 2054 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: audit 2024-04-02T04:52:23.205336+0000 mon.a (mon.0) 2055 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: cluster 2024-04-02T04:52:23.211266+0000 mon.a (mon.0) 2056 : cluster 0 osdmap e303: 8 total, 7 up, 8 in 2024-04-02T04:52:24.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:24 smithi098 bash[23790]: audit 2024-04-02T04:52:23.922368+0000 mon.c (mon.2) 488 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:24.229 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:24.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:24.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: cluster 2024-04-02T04:52:21.993540+0000 mgr.y (mgr.14538) 346 : cluster 0 pgmap v429: 356 pgs: 93 active+undersized, 226 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: audit 2024-04-02T04:52:22.921164+0000 mon.c (mon.2) 487 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: cluster 2024-04-02T04:52:23.082160+0000 mon.a (mon.0) 2053 : cluster 0 osdmap e302: 8 total, 7 up, 8 in 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: audit 2024-04-02T04:52:23.132639+0000 mgr.y (mgr.14538) 347 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: audit 2024-04-02T04:52:23.205135+0000 mon.a (mon.0) 2054 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: audit 2024-04-02T04:52:23.205336+0000 mon.a (mon.0) 2055 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:24.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: cluster 2024-04-02T04:52:23.211266+0000 mon.a (mon.0) 2056 : cluster 0 osdmap e303: 8 total, 7 up, 8 in 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[28904]: audit 2024-04-02T04:52:23.922368+0000 mon.c (mon.2) 488 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: cluster 2024-04-02T04:52:21.993540+0000 mgr.y (mgr.14538) 346 : cluster 0 pgmap v429: 356 pgs: 93 active+undersized, 226 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: audit 2024-04-02T04:52:22.921164+0000 mon.c (mon.2) 487 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: cluster 2024-04-02T04:52:23.082160+0000 mon.a (mon.0) 2053 : cluster 0 osdmap e302: 8 total, 7 up, 8 in 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: audit 2024-04-02T04:52:23.132639+0000 mgr.y (mgr.14538) 347 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: audit 2024-04-02T04:52:23.205135+0000 mon.a (mon.0) 2054 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-72153-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: audit 2024-04-02T04:52:23.205336+0000 mon.a (mon.0) 2055 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-72145-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: cluster 2024-04-02T04:52:23.211266+0000 mon.a (mon.0) 2056 : cluster 0 osdmap e303: 8 total, 7 up, 8 in 2024-04-02T04:52:24.335 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:24 smithi069 bash[21528]: audit 2024-04-02T04:52:23.922368+0000 mon.c (mon.2) 488 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:24.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:24.465 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:24.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:24.596 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:24.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:24.725 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:24.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:24.864 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:24.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:24.992 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:24.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:25.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:25.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: cluster 2024-04-02T04:52:23.995502+0000 mgr.y (mgr.14538) 348 : cluster 0 pgmap v432: 372 pgs: 16 unknown, 92 active+undersized, 226 active+clean, 38 active+undersized+degraded; 8.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 255 B/s wr, 15 op/s; 279/1533 objects degraded (18.200%) 2024-04-02T04:52:25.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: cluster 2024-04-02T04:52:24.205087+0000 mon.a (mon.0) 2057 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:25.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: cluster 2024-04-02T04:52:24.221876+0000 mon.a (mon.0) 2058 : cluster 0 osdmap e304: 8 total, 7 up, 8 in 2024-04-02T04:52:25.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: cluster 2024-04-02T04:52:24.818239+0000 mon.a (mon.0) 2059 : cluster 3 Health check update: Degraded data redundancy: 279/1533 objects degraded (18.200%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:25.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: cluster 2024-04-02T04:52:24.830504+0000 mon.a (mon.0) 2060 : cluster 0 osdmap e305: 8 total, 7 up, 8 in 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: audit 2024-04-02T04:52:24.834767+0000 mon.a (mon.0) 2061 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: audit 2024-04-02T04:52:24.844171+0000 mon.c (mon.2) 489 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: audit 2024-04-02T04:52:24.844930+0000 mon.a (mon.0) 2062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: audit 2024-04-02T04:52:24.923733+0000 mon.c (mon.2) 490 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: audit 2024-04-02T04:52:24.949971+0000 mon.c (mon.2) 491 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[21528]: audit 2024-04-02T04:52:24.950379+0000 mon.a (mon.0) 2063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: cluster 2024-04-02T04:52:23.995502+0000 mgr.y (mgr.14538) 348 : cluster 0 pgmap v432: 372 pgs: 16 unknown, 92 active+undersized, 226 active+clean, 38 active+undersized+degraded; 8.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 255 B/s wr, 15 op/s; 279/1533 objects degraded (18.200%) 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: cluster 2024-04-02T04:52:24.205087+0000 mon.a (mon.0) 2057 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: cluster 2024-04-02T04:52:24.221876+0000 mon.a (mon.0) 2058 : cluster 0 osdmap e304: 8 total, 7 up, 8 in 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: cluster 2024-04-02T04:52:24.818239+0000 mon.a (mon.0) 2059 : cluster 3 Health check update: Degraded data redundancy: 279/1533 objects degraded (18.200%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:25.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: cluster 2024-04-02T04:52:24.830504+0000 mon.a (mon.0) 2060 : cluster 0 osdmap e305: 8 total, 7 up, 8 in 2024-04-02T04:52:25.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: audit 2024-04-02T04:52:24.834767+0000 mon.a (mon.0) 2061 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:25.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: audit 2024-04-02T04:52:24.844171+0000 mon.c (mon.2) 489 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:25.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: audit 2024-04-02T04:52:24.844930+0000 mon.a (mon.0) 2062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:25.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: audit 2024-04-02T04:52:24.923733+0000 mon.c (mon.2) 490 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:25.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: audit 2024-04-02T04:52:24.949971+0000 mon.c (mon.2) 491 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:25.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:25 smithi069 bash[28904]: audit 2024-04-02T04:52:24.950379+0000 mon.a (mon.0) 2063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:25.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: cluster 2024-04-02T04:52:23.995502+0000 mgr.y (mgr.14538) 348 : cluster 0 pgmap v432: 372 pgs: 16 unknown, 92 active+undersized, 226 active+clean, 38 active+undersized+degraded; 8.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 255 B/s wr, 15 op/s; 279/1533 objects degraded (18.200%) 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: cluster 2024-04-02T04:52:24.205087+0000 mon.a (mon.0) 2057 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: cluster 2024-04-02T04:52:24.221876+0000 mon.a (mon.0) 2058 : cluster 0 osdmap e304: 8 total, 7 up, 8 in 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: cluster 2024-04-02T04:52:24.818239+0000 mon.a (mon.0) 2059 : cluster 3 Health check update: Degraded data redundancy: 279/1533 objects degraded (18.200%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: cluster 2024-04-02T04:52:24.830504+0000 mon.a (mon.0) 2060 : cluster 0 osdmap e305: 8 total, 7 up, 8 in 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: audit 2024-04-02T04:52:24.834767+0000 mon.a (mon.0) 2061 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: audit 2024-04-02T04:52:24.844171+0000 mon.c (mon.2) 489 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: audit 2024-04-02T04:52:24.844930+0000 mon.a (mon.0) 2062 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: audit 2024-04-02T04:52:24.923733+0000 mon.c (mon.2) 490 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: audit 2024-04-02T04:52:24.949971+0000 mon.c (mon.2) 491 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:25.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:25 smithi098 bash[23790]: audit 2024-04-02T04:52:24.950379+0000 mon.a (mon.0) 2063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:25.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:25.379 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:25.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:25.510 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:25.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:25.745 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:25.768 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:25.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:25.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:25.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:25.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:26.026 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:26.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:26.153 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:26.256 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:26.269 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:26.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:26.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:26.548 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:26.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:26.676 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:26.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:26.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:26.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.824349+0000 mon.a (mon.0) 2064 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.824431+0000 mon.a (mon.0) 2065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.824528+0000 mon.a (mon.0) 2066 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: cluster 2024-04-02T04:52:25.828245+0000 mon.a (mon.0) 2067 : cluster 0 osdmap e306: 8 total, 7 up, 8 in 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.829567+0000 mon.a (mon.0) 2068 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.831721+0000 mon.c (mon.2) 492 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.832305+0000 mon.a (mon.0) 2069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.832617+0000 mon.c (mon.2) 493 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.834143+0000 mon.a (mon.0) 2070 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: audit 2024-04-02T04:52:25.925035+0000 mon.c (mon.2) 494 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:26.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:26 smithi098 bash[23790]: cluster 2024-04-02T04:52:25.997395+0000 mgr.y (mgr.14538) 349 : cluster 0 pgmap v436: 356 pgs: 92 active+undersized, 226 active+clean, 38 active+undersized+degraded; 8.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 270 B/s wr, 16 op/s; 279/1533 objects degraded (18.200%) 2024-04-02T04:52:26.936 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:26.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.824349+0000 mon.a (mon.0) 2064 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.824431+0000 mon.a (mon.0) 2065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.824528+0000 mon.a (mon.0) 2066 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: cluster 2024-04-02T04:52:25.828245+0000 mon.a (mon.0) 2067 : cluster 0 osdmap e306: 8 total, 7 up, 8 in 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.829567+0000 mon.a (mon.0) 2068 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.831721+0000 mon.c (mon.2) 492 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:27.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.832305+0000 mon.a (mon.0) 2069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.832617+0000 mon.c (mon.2) 493 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.834143+0000 mon.a (mon.0) 2070 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: audit 2024-04-02T04:52:25.925035+0000 mon.c (mon.2) 494 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[21528]: cluster 2024-04-02T04:52:25.997395+0000 mgr.y (mgr.14538) 349 : cluster 0 pgmap v436: 356 pgs: 92 active+undersized, 226 active+clean, 38 active+undersized+degraded; 8.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 270 B/s wr, 16 op/s; 279/1533 objects degraded (18.200%) 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.824349+0000 mon.a (mon.0) 2064 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.824431+0000 mon.a (mon.0) 2065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.824528+0000 mon.a (mon.0) 2066 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: cluster 2024-04-02T04:52:25.828245+0000 mon.a (mon.0) 2067 : cluster 0 osdmap e306: 8 total, 7 up, 8 in 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.829567+0000 mon.a (mon.0) 2068 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.831721+0000 mon.c (mon.2) 492 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.832305+0000 mon.a (mon.0) 2069 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]: dispatch 2024-04-02T04:52:27.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.832617+0000 mon.c (mon.2) 493 : audit 1 from='client.? 172.21.15.69:0/2206429443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:27.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.834143+0000 mon.a (mon.0) 2070 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]: dispatch 2024-04-02T04:52:27.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: audit 2024-04-02T04:52:25.925035+0000 mon.c (mon.2) 494 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:27.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:26 smithi069 bash[28904]: cluster 2024-04-02T04:52:25.997395+0000 mgr.y (mgr.14538) 349 : cluster 0 pgmap v436: 356 pgs: 92 active+undersized, 226 active+clean, 38 active+undersized+degraded; 8.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 270 B/s wr, 16 op/s; 279/1533 objects degraded (18.200%) 2024-04-02T04:52:27.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:27.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:27.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:27.324 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:27.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:27.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:27.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.711 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:27.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:27.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.959 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: cluster 2024-04-02T04:52:26.825534+0000 mon.a (mon.0) 2071 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:27.959 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.836166+0000 mon.a (mon.0) 2072 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:27.959 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.836244+0000 mon.a (mon.0) 2073 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]': finished 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.836346+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: cluster 2024-04-02T04:52:26.842557+0000 mon.a (mon.0) 2075 : cluster 0 osdmap e307: 8 total, 7 up, 8 in 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.875936+0000 mon.c (mon.2) 495 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.876212+0000 mon.c (mon.2) 496 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.876405+0000 mon.a (mon.0) 2076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.876678+0000 mon.a (mon.0) 2077 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.877002+0000 mon.c (mon.2) 497 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.877629+0000 mon.a (mon.0) 2078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.877668+0000 mon.c (mon.2) 498 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.877976+0000 mon.a (mon.0) 2079 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.878330+0000 mon.c (mon.2) 499 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.878587+0000 mon.c (mon.2) 500 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.878725+0000 mon.a (mon.0) 2080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:27.960 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.879117+0000 mon.a (mon.0) 2081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:27.961 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: audit 2024-04-02T04:52:26.926596+0000 mon.c (mon.2) 501 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:27.961 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: cluster 2024-04-02T04:52:27.008330+0000 osd.1 (osd.1) 199 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:27.961 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:27 smithi098 bash[23790]: cluster 2024-04-02T04:52:27.008461+0000 osd.1 (osd.1) 200 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:27.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: cluster 2024-04-02T04:52:26.825534+0000 mon.a (mon.0) 2071 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:27.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:27.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:28.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:28.099 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:28.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:28.229 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.836166+0000 mon.a (mon.0) 2072 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.836244+0000 mon.a (mon.0) 2073 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]': finished 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.836346+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: cluster 2024-04-02T04:52:26.842557+0000 mon.a (mon.0) 2075 : cluster 0 osdmap e307: 8 total, 7 up, 8 in 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.875936+0000 mon.c (mon.2) 495 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.876212+0000 mon.c (mon.2) 496 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.876405+0000 mon.a (mon.0) 2076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.876678+0000 mon.a (mon.0) 2077 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.877002+0000 mon.c (mon.2) 497 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.877629+0000 mon.a (mon.0) 2078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.877668+0000 mon.c (mon.2) 498 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.877976+0000 mon.a (mon.0) 2079 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.878330+0000 mon.c (mon.2) 499 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.878587+0000 mon.c (mon.2) 500 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.878725+0000 mon.a (mon.0) 2080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: cluster 2024-04-02T04:52:26.825534+0000 mon.a (mon.0) 2071 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.836166+0000 mon.a (mon.0) 2072 : audit 1 from='client.? 172.21.15.69:0/3534547343' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-72153-47"}]': finished 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.836244+0000 mon.a (mon.0) 2073 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-29"}]': finished 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.836346+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-72145-37"}]': finished 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: cluster 2024-04-02T04:52:26.842557+0000 mon.a (mon.0) 2075 : cluster 0 osdmap e307: 8 total, 7 up, 8 in 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.875936+0000 mon.c (mon.2) 495 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.876212+0000 mon.c (mon.2) 496 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.876405+0000 mon.a (mon.0) 2076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.876678+0000 mon.a (mon.0) 2077 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.877002+0000 mon.c (mon.2) 497 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.877629+0000 mon.a (mon.0) 2078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.877668+0000 mon.c (mon.2) 498 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.877976+0000 mon.a (mon.0) 2079 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.878330+0000 mon.c (mon.2) 499 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.878587+0000 mon.c (mon.2) 500 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.879117+0000 mon.a (mon.0) 2081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: audit 2024-04-02T04:52:26.926596+0000 mon.c (mon.2) 501 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: cluster 2024-04-02T04:52:27.008330+0000 osd.1 (osd.1) 199 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[21528]: cluster 2024-04-02T04:52:27.008461+0000 osd.1 (osd.1) 200 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:28.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.878725+0000 mon.a (mon.0) 2080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.879117+0000 mon.a (mon.0) 2081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:28.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: audit 2024-04-02T04:52:26.926596+0000 mon.c (mon.2) 501 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:28.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: cluster 2024-04-02T04:52:27.008330+0000 osd.1 (osd.1) 199 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:28.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:27 smithi069 bash[28904]: cluster 2024-04-02T04:52:27.008461+0000 osd.1 (osd.1) 200 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:28.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:28.357 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:28.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:28.489 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:28.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:28.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:28.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:28.750 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:28.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:52:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:52:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:52:28.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:28.887 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:28.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:28.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:29.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:29.144 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:29.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:29.377 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:29.401 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:29.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:29.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:29.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:29.655 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:29.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:29.776 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:29.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.856857+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.857090+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: cluster 2024-04-02T04:52:27.866304+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e308: 8 total, 7 up, 8 in 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.870442+0000 mon.c (mon.2) 502 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.870832+0000 mon.c (mon.2) 503 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.871203+0000 mon.a (mon.0) 2085 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.871858+0000 mon.a (mon.0) 2086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:29.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: audit 2024-04-02T04:52:27.927847+0000 mon.c (mon.2) 504 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[21528]: cluster 2024-04-02T04:52:28.000953+0000 mgr.y (mgr.14538) 350 : cluster 0 pgmap v439: 324 pgs: 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.856857+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.857090+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: cluster 2024-04-02T04:52:27.866304+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e308: 8 total, 7 up, 8 in 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.870442+0000 mon.c (mon.2) 502 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.870832+0000 mon.c (mon.2) 503 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.871203+0000 mon.a (mon.0) 2085 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.871858+0000 mon.a (mon.0) 2086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: audit 2024-04-02T04:52:27.927847+0000 mon.c (mon.2) 504 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:29.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:29 smithi069 bash[28904]: cluster 2024-04-02T04:52:28.000953+0000 mgr.y (mgr.14538) 350 : cluster 0 pgmap v439: 324 pgs: 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:29.910 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:29.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.856857+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-72145-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.857090+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-72153-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: cluster 2024-04-02T04:52:27.866304+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e308: 8 total, 7 up, 8 in 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.870442+0000 mon.c (mon.2) 502 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.870832+0000 mon.c (mon.2) 503 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.871203+0000 mon.a (mon.0) 2085 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.871858+0000 mon.a (mon.0) 2086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: audit 2024-04-02T04:52:27.927847+0000 mon.c (mon.2) 504 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:29 smithi098 bash[23790]: cluster 2024-04-02T04:52:28.000953+0000 mgr.y (mgr.14538) 350 : cluster 0 pgmap v439: 324 pgs: 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:30.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:30.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:30.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:30.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:30.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:30.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:30.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:30.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:30.562 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:30.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:30.691 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:30.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:30.816 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:28.929239+0000 mon.c (mon.2) 505 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: cluster 2024-04-02T04:52:29.720168+0000 mon.a (mon.0) 2087 : cluster 0 osdmap e309: 8 total, 7 up, 8 in 2024-04-02T04:52:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.721508+0000 mon.c (mon.2) 506 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.727090+0000 mon.a (mon.0) 2088 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: cluster 2024-04-02T04:52:29.821125+0000 mon.a (mon.0) 2089 : cluster 3 Health check update: Degraded data redundancy: 278/1527 objects degraded (18.206%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: cluster 2024-04-02T04:52:29.821168+0000 mon.a (mon.0) 2090 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.865415+0000 mon.a (mon.0) 2091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.865518+0000 mon.a (mon.0) 2092 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.865603+0000 mon.a (mon.0) 2093 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: cluster 2024-04-02T04:52:29.870762+0000 mon.a (mon.0) 2094 : cluster 0 osdmap e310: 8 total, 7 up, 8 in 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.872558+0000 mon.c (mon.2) 507 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.874352+0000 mon.a (mon.0) 2095 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: audit 2024-04-02T04:52:29.930080+0000 mon.c (mon.2) 508 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[28904]: cluster 2024-04-02T04:52:30.002737+0000 mgr.y (mgr.14538) 351 : cluster 0 pgmap v442: 372 pgs: 48 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:28.929239+0000 mon.c (mon.2) 505 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: cluster 2024-04-02T04:52:29.720168+0000 mon.a (mon.0) 2087 : cluster 0 osdmap e309: 8 total, 7 up, 8 in 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.721508+0000 mon.c (mon.2) 506 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.727090+0000 mon.a (mon.0) 2088 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: cluster 2024-04-02T04:52:29.821125+0000 mon.a (mon.0) 2089 : cluster 3 Health check update: Degraded data redundancy: 278/1527 objects degraded (18.206%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: cluster 2024-04-02T04:52:29.821168+0000 mon.a (mon.0) 2090 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.865415+0000 mon.a (mon.0) 2091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.865518+0000 mon.a (mon.0) 2092 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.865603+0000 mon.a (mon.0) 2093 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: cluster 2024-04-02T04:52:29.870762+0000 mon.a (mon.0) 2094 : cluster 0 osdmap e310: 8 total, 7 up, 8 in 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.872558+0000 mon.c (mon.2) 507 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.874352+0000 mon.a (mon.0) 2095 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: audit 2024-04-02T04:52:29.930080+0000 mon.c (mon.2) 508 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:30 smithi069 bash[21528]: cluster 2024-04-02T04:52:30.002737+0000 mgr.y (mgr.14538) 351 : cluster 0 pgmap v442: 372 pgs: 48 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:30.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:30.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:28.929239+0000 mon.c (mon.2) 505 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: cluster 2024-04-02T04:52:29.720168+0000 mon.a (mon.0) 2087 : cluster 0 osdmap e309: 8 total, 7 up, 8 in 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.721508+0000 mon.c (mon.2) 506 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.727090+0000 mon.a (mon.0) 2088 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: cluster 2024-04-02T04:52:29.821125+0000 mon.a (mon.0) 2089 : cluster 3 Health check update: Degraded data redundancy: 278/1527 objects degraded (18.206%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: cluster 2024-04-02T04:52:29.821168+0000 mon.a (mon.0) 2090 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.865415+0000 mon.a (mon.0) 2091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-72145-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.865518+0000 mon.a (mon.0) 2092 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-72153-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.865603+0000 mon.a (mon.0) 2093 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: cluster 2024-04-02T04:52:29.870762+0000 mon.a (mon.0) 2094 : cluster 0 osdmap e310: 8 total, 7 up, 8 in 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.872558+0000 mon.c (mon.2) 507 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.874352+0000 mon.a (mon.0) 2095 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: audit 2024-04-02T04:52:29.930080+0000 mon.c (mon.2) 508 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:30 smithi098 bash[23790]: cluster 2024-04-02T04:52:30.002737+0000 mgr.y (mgr.14538) 351 : cluster 0 pgmap v442: 372 pgs: 48 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:31.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:31.075 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:31.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:31.309 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:31.335 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:31.438 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:31.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:31.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:31.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:31.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:31.722 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:31.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:31.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:31.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:32.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:32.094 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: cluster 2024-04-02T04:52:30.484922+0000 osd.5 (osd.5) 187 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: cluster 2024-04-02T04:52:30.485121+0000 osd.5 (osd.5) 188 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: audit 2024-04-02T04:52:30.867853+0000 mon.a (mon.0) 2096 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: cluster 2024-04-02T04:52:30.872025+0000 mon.a (mon.0) 2097 : cluster 0 osdmap e311: 8 total, 7 up, 8 in 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: audit 2024-04-02T04:52:30.875965+0000 mon.c (mon.2) 509 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: audit 2024-04-02T04:52:30.883471+0000 mon.a (mon.0) 2098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[21528]: audit 2024-04-02T04:52:30.930962+0000 mon.c (mon.2) 510 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: cluster 2024-04-02T04:52:30.484922+0000 osd.5 (osd.5) 187 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: cluster 2024-04-02T04:52:30.485121+0000 osd.5 (osd.5) 188 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: audit 2024-04-02T04:52:30.867853+0000 mon.a (mon.0) 2096 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: cluster 2024-04-02T04:52:30.872025+0000 mon.a (mon.0) 2097 : cluster 0 osdmap e311: 8 total, 7 up, 8 in 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: audit 2024-04-02T04:52:30.875965+0000 mon.c (mon.2) 509 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: audit 2024-04-02T04:52:30.883471+0000 mon.a (mon.0) 2098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:32.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:31 smithi069 bash[28904]: audit 2024-04-02T04:52:30.930962+0000 mon.c (mon.2) 510 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:32.109 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:32.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: cluster 2024-04-02T04:52:30.484922+0000 osd.5 (osd.5) 187 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: cluster 2024-04-02T04:52:30.485121+0000 osd.5 (osd.5) 188 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: audit 2024-04-02T04:52:30.867853+0000 mon.a (mon.0) 2096 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: cluster 2024-04-02T04:52:30.872025+0000 mon.a (mon.0) 2097 : cluster 0 osdmap e311: 8 total, 7 up, 8 in 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: audit 2024-04-02T04:52:30.875965+0000 mon.c (mon.2) 509 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: audit 2024-04-02T04:52:30.883471+0000 mon.a (mon.0) 2098 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:31 smithi098 bash[23790]: audit 2024-04-02T04:52:30.930962+0000 mon.c (mon.2) 510 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:32.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/3817180817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78074d90 msgr2=0x7fab78112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:32.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- 172.21.15.69:0/3817180817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78074d90 0x7fab78112f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009990 tx=0x7fab6c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/3817180817 shutdown_connections 2024-04-02T04:52:32.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- 172.21.15.69:0/3817180817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78113440 0x7fab78115830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- 172.21.15.69:0/3817180817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074470 0x7fab78074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- 172.21.15.69:0/3817180817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78074d90 0x7fab78112f00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/3817180817 >> 172.21.15.69:0/3817180817 conn(0x7fab78079360 msgr2=0x7fab78079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:32.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/3817180817 shutdown_connections 2024-04-02T04:52:32.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/3817180817 wait complete. 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 librados: starting msgr at 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 librados: starting objecter 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 Processor -- start 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- start start 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 librados: setting wanted keys 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 librados: calling monclient init 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78074470 0x7fab7819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074d90 0x7fab7819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 0x7fab781a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab781180e0 con 0x7fab78074470 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab78117f60 con 0x7fab78074d90 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fab78118260 con 0x7fab78113440 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab76d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78074470 0x7fab7819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 0x7fab781a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 0x7fab781a2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36956/0 (socket says 172.21.15.69:36956) 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab76d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78074470 0x7fab7819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53792/0 (socket says 172.21.15.69:53792) 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 -- 172.21.15.69:0/2278786369 learned_addr learned my addr 172.21.15.69:0/2278786369 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7659c700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074d90 0x7fab7819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 -- 172.21.15.69:0/2278786369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074d90 msgr2=0x7fab7819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074d90 0x7fab7819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 -- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78074470 msgr2=0x7fab7819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78074470 0x7fab7819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 -- 172.21.15.69:0/2278786369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab781a3680 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7659c700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074d90 0x7fab7819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7759e700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 0x7fab781a2f40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fab7000ef90 tx=0x7fab7000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab5ffff700 10 client.?.objecter ms_handle_connect 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab5ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab700092e0 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fab70004bb0 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab781a3910 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70004020 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 10 client.26291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab781ab1f0 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.302+0000 7fab7d138700 1 librados: init done 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7804ff00 con 0x7fab78113440 2024-04-02T04:52:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fab7001b030 con 0x7fab78113440 2024-04-02T04:52:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fab6006f5e0 0x7fab60071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(312..312 src has 74..312) ==== 11795+0+0 (secure 0 0 0) 0x7fab7008ecf0 con 0x7fab78113440 2024-04-02T04:52:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab76d9d700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fab6006f5e0 0x7fab60071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 10 client.26291.objecter ms_dispatch 0x7fab78001550 osd_map(312..312 src has 74..312) 2024-04-02T04:52:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 3 client.26291.objecter handle_osd_map got epochs [312,312] > 0 2024-04-02T04:52:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 3 client.26291.objecter handle_osd_map decoding full epoch 312 2024-04-02T04:52:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 20 client.26291.objecter dump_active .. 0 homeless 2024-04-02T04:52:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab5ffff700 10 client.26291.objecter ms_handle_connect 0x7fab6006f5e0 2024-04-02T04:52:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.306+0000 7fab76d9d700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fab6006f5e0 0x7fab60071aa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fab68005950 tx=0x7fab6800a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.310+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fab70093070 con 0x7fab78113440 2024-04-02T04:52:32.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:32.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:32.485 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:32.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.530+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fab781ab3c0 con 0x7fab78113440 2024-04-02T04:52:32.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:32.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:32.744 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:32.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:32.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:32.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.910+0000 7fab5ffff700 1 -- 172.21.15.69:0/2278786369 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v313) ==== 91+0+0 (secure 0 0 0) 0x7fab70022020 con 0x7fab78113440 2024-04-02T04:52:32.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.914+0000 7fab7d138700 10 librados: watch_flush enter 2024-04-02T04:52:32.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.914+0000 7fab7d138700 10 librados: watch_flush exit 2024-04-02T04:52:32.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.914+0000 7fab7d138700 20 client.26291.objecter shutdown clearing up homeless session... 2024-04-02T04:52:32.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.914+0000 7fab7d138700 10 client.26291.objecter successfully canceled tick 2024-04-02T04:52:32.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fab6006f5e0 msgr2=0x7fab60071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fab6006f5e0 0x7fab60071aa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fab68005950 tx=0x7fab6800a300 comp rx=0 tx=0).stop 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 msgr2=0x7fab781a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 0x7fab781a2f40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fab7000ef90 tx=0x7fab7000c3a0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 shutdown_connections 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fab6006f5e0 0x7fab60071aa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab78113440 0x7fab781a2f40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab78074470 0x7fab7819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 --2- 172.21.15.69:0/2278786369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab78074d90 0x7fab7819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 >> 172.21.15.69:0/2278786369 conn(0x7fab78079360 msgr2=0x7fab78111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 shutdown_connections 2024-04-02T04:52:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 -- 172.21.15.69:0/2278786369 wait complete. 2024-04-02T04:52:32.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:32.918+0000 7fab7d138700 1 librados: shutdown 2024-04-02T04:52:32.918 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:52:32.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.000 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:33.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.879582+0000 mon.a (mon.0) 2099 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]': finished 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.884707+0000 mon.c (mon.2) 511 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]: dispatch 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: cluster 2024-04-02T04:52:31.894316+0000 mon.a (mon.0) 2100 : cluster 0 osdmap e312: 8 total, 7 up, 8 in 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.895223+0000 mon.a (mon.0) 2101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]: dispatch 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.895602+0000 mon.c (mon.2) 512 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.895944+0000 mon.c (mon.2) 513 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.896286+0000 mon.a (mon.0) 2102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:33.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.896610+0000 mon.a (mon.0) 2103 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:31.931945+0000 mon.c (mon.2) 514 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: cluster 2024-04-02T04:52:32.003875+0000 mgr.y (mgr.14538) 352 : cluster 0 pgmap v445: 356 pgs: 32 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:32.535302+0000 mon.c (mon.2) 515 : audit 1 from='client.? 172.21.15.69:0/2278786369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:52:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:32.535586+0000 mon.a (mon.0) 2104 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:52:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:32 smithi098 bash[23790]: audit 2024-04-02T04:52:32.712342+0000 mon.b (mon.1) 199 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:33.129 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:33.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:52:33.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.879582+0000 mon.a (mon.0) 2099 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]': finished 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.884707+0000 mon.c (mon.2) 511 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: cluster 2024-04-02T04:52:31.894316+0000 mon.a (mon.0) 2100 : cluster 0 osdmap e312: 8 total, 7 up, 8 in 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.895223+0000 mon.a (mon.0) 2101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.895602+0000 mon.c (mon.2) 512 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.895944+0000 mon.c (mon.2) 513 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.896286+0000 mon.a (mon.0) 2102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.896610+0000 mon.a (mon.0) 2103 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:31.931945+0000 mon.c (mon.2) 514 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:33.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: cluster 2024-04-02T04:52:32.003875+0000 mgr.y (mgr.14538) 352 : cluster 0 pgmap v445: 356 pgs: 32 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:32.535302+0000 mon.c (mon.2) 515 : audit 1 from='client.? 172.21.15.69:0/2278786369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:32.535586+0000 mon.a (mon.0) 2104 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[21528]: audit 2024-04-02T04:52:32.712342+0000 mon.b (mon.1) 199 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.879582+0000 mon.a (mon.0) 2099 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-31"}]': finished 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.884707+0000 mon.c (mon.2) 511 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: cluster 2024-04-02T04:52:31.894316+0000 mon.a (mon.0) 2100 : cluster 0 osdmap e312: 8 total, 7 up, 8 in 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.895223+0000 mon.a (mon.0) 2101 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.895602+0000 mon.c (mon.2) 512 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.895944+0000 mon.c (mon.2) 513 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:33.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.896286+0000 mon.a (mon.0) 2102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:33.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.896610+0000 mon.a (mon.0) 2103 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:33.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:31.931945+0000 mon.c (mon.2) 514 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:33.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: cluster 2024-04-02T04:52:32.003875+0000 mgr.y (mgr.14538) 352 : cluster 0 pgmap v445: 356 pgs: 32 unknown, 1 active+clean+snaptrim, 80 active+undersized, 206 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:33.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:32.535302+0000 mon.c (mon.2) 515 : audit 1 from='client.? 172.21.15.69:0/2278786369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:52:33.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:32.535586+0000 mon.a (mon.0) 2104 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:52:33.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:32 smithi069 bash[28904]: audit 2024-04-02T04:52:32.712342+0000 mon.b (mon.1) 199 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:33.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:33.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:33.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:33.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:33.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:33.641 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/626642174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 msgr2=0x7f3598107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- 172.21.15.69:0/626642174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 0x7f3598107600 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f35880098f0 tx=0x7f3588009c00 comp rx=0 tx=0).stop 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/626642174 shutdown_connections 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- 172.21.15.69:0/626642174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598101b60 0x7f359810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- 172.21.15.69:0/626642174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 0x7f3598107600 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- 172.21.15.69:0/626642174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35981011a0 0x7f3598101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/626642174 >> 172.21.15.69:0/626642174 conn(0x7f35980771e0 msgr2=0x7f35980775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/626642174 shutdown_connections 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/626642174 wait complete. 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 librados: starting msgr at 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 librados: starting objecter 2024-04-02T04:52:33.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 Processor -- start 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- start start 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 librados: setting wanted keys 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 librados: calling monclient init 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35981011a0 0x7f3598196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598101b60 0x7f3598197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 0x7f3598197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3598110970 con 0x7f3598107220 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35981107f0 con 0x7f3598101b60 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3598110af0 con 0x7f35981011a0 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35981011a0 0x7f3598196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f3596d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 0x7f3598197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35981011a0 0x7f3598196cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36976/0 (socket says 172.21.15.69:36976) 2024-04-02T04:52:33.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 -- 172.21.15.69:0/2690011343 learned_addr learned my addr 172.21.15.69:0/2690011343 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598101b60 0x7f3598197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 -- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598101b60 msgr2=0x7f3598197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598101b60 0x7f3598197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 -- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 msgr2=0x7f3598197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 0x7f3598197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35981a6560 con 0x7f35981011a0 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3598101b60 0x7f3598197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3598107220 0x7f3598197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35981011a0 0x7f3598196cf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f35880059d0 tx=0x7f358800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358ffff700 10 client.?.objecter ms_handle_connect 0x7f35981011a0 2024-04-02T04:52:33.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:33.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588027070 con 0x7f35981011a0 2024-04-02T04:52:33.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35981a67f0 con 0x7f35981011a0 2024-04-02T04:52:33.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f358800bb60 con 0x7f35981011a0 2024-04-02T04:52:33.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588004930 con 0x7f35981011a0 2024-04-02T04:52:33.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 10 client.26297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:33.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35981a6cc0 con 0x7f35981011a0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.654+0000 7f359cd34700 1 librados: init done 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f359804ff00 con 0x7f35981011a0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f358802a070 con 0x7f35981011a0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f358406f5e0 0x7f3584071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(313..313 src has 74..313) ==== 11142+0+0 (secure 0 0 0) 0x7f3588090380 con 0x7f35981011a0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 osd_map(313..313 src has 74..313) 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 3 client.26297.objecter handle_osd_map got epochs [313,313] > 0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 3 client.26297.objecter handle_osd_map decoding full epoch 313 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f358406f5e0 0x7f3584071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 20 client.26297.objecter dump_active .. 0 homeless 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v313) ==== 54+0+16 (secure 0 0 0) 0x7f3588031070 con 0x7f35981011a0 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 10 client.26297.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 1 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3598062e50 0x7f3598063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 1 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 0 1 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359806a560 osd=0 4 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 10 client.26297.objecter _send_command 1 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35981aa0b0 con 0x7f3598062e50 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f358406f5e0 0x7f3584071aa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3598198e50 tx=0x7f3580009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f358406f5e0 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3598062e50 0x7f3598063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f3598062e50 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3598062e50 0x7f3598063230 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35981aa0b0 con 0x7f3598062e50 2024-04-02T04:52:33.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 1: 0 ) 2024-04-02T04:52:33.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 10 client.26297.objecter _finish_command 1 = system:0 2024-04-02T04:52:33.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f359806a560 osd=0 4 2024-04-02T04:52:33.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.658+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 0 1 2024-04-02T04:52:33.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 10 client.26297.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 2 2024-04-02T04:52:33.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 2 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f359806a560 osd=0 3 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 0 2 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359806a560 osd=0 4 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 10 client.26297.objecter _send_command 2 2024-04-02T04:52:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35980628d0 con 0x7f3598062e50 2024-04-02T04:52:33.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35980628d0 con 0x7f3598062e50 2024-04-02T04:52:33.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f358ffff700 10 client.26297.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f359806a560 osd=0 4 2024-04-02T04:52:33.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 0 2 2024-04-02T04:52:33.679 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:52:33.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 10 client.26297.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 3 2024-04-02T04:52:33.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f359802c9b0 0x7f359802cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 1 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109780 osd=1 4 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 10 client.26297.objecter _send_command 3 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3598109e00 con 0x7f359802c9b0 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f359802c9b0 0x7f359802cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f359802c9b0 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.678+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f359802c9b0 0x7f359802cd90 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.682+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3598109e00 con 0x7f359802c9b0 2024-04-02T04:52:33.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.682+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 3: 0 ) 2024-04-02T04:52:33.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.682+0000 7f358ffff700 10 client.26297.objecter _finish_command 3 = system:0 2024-04-02T04:52:33.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.682+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f3598109780 osd=1 4 2024-04-02T04:52:33.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.682+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 1 3 2024-04-02T04:52:33.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 10 client.26297.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 4 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 4 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f3598109780 osd=1 3 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 1 4 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109780 osd=1 4 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 10 client.26297.objecter _send_command 4 2024-04-02T04:52:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3598109200 con 0x7f359802c9b0 2024-04-02T04:52:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3598109200 con 0x7f359802c9b0 2024-04-02T04:52:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f358ffff700 10 client.26297.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f3598109780 osd=1 4 2024-04-02T04:52:33.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 1 4 2024-04-02T04:52:33.699 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:52:33.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 10 client.26297.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 5 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35981ac270 0x7f35981ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 5 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 2 5 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109600 osd=2 4 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 10 client.26297.objecter _send_command 5 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f359802dfb0 con 0x7f35981ac270 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.698+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35981ac270 0x7f35981ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f35981ac270 2024-04-02T04:52:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35981ac270 0x7f35981ae660 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f359802dfb0 con 0x7f35981ac270 2024-04-02T04:52:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 5: 0 ) 2024-04-02T04:52:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358ffff700 10 client.26297.objecter _finish_command 5 = system:0 2024-04-02T04:52:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f3598109600 osd=2 4 2024-04-02T04:52:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.702+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 2 5 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 10 client.26297.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 6 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 6 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f3598109600 osd=2 3 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 2 6 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f3598109600 osd=2 4 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 10 client.26297.objecter _send_command 6 2024-04-02T04:52:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f359802d580 con 0x7f35981ac270 2024-04-02T04:52:33.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f359802d580 con 0x7f35981ac270 2024-04-02T04:52:33.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f358ffff700 10 client.26297.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f3598109600 osd=2 4 2024-04-02T04:52:33.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 2 6 2024-04-02T04:52:33.719 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 10 client.26297.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 7 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35981a9c10 0x7f35981b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 7 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 3 7 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359802d580 osd=3 4 2024-04-02T04:52:33.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 10 client.26297.objecter _send_command 7 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f35981b4ca0 con 0x7f35981a9c10 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.718+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35981a9c10 0x7f35981b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f35981a9c10 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35981a9c10 0x7f35981b1d30 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35981b4ca0 con 0x7f35981a9c10 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 7: 0 ) 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f358ffff700 10 client.26297.objecter _finish_command 7 = system:0 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f359802d580 osd=3 4 2024-04-02T04:52:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.722+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 3 7 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 10 client.26297.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 8 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 8 2024-04-02T04:52:33.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f359802d580 osd=3 3 2024-04-02T04:52:33.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 3 8 2024-04-02T04:52:33.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f359802d580 osd=3 4 2024-04-02T04:52:33.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 10 client.26297.objecter _send_command 8 2024-04-02T04:52:33.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35981a9180 con 0x7f35981a9c10 2024-04-02T04:52:33.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35981a9180 con 0x7f35981a9c10 2024-04-02T04:52:33.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f358ffff700 10 client.26297.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f359802d580 osd=3 4 2024-04-02T04:52:33.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.738+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 3 8 2024-04-02T04:52:33.739 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 10 client.26297.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 9 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35981af880 0x7f35981b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 9 2024-04-02T04:52:33.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 4 9 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981a9180 osd=4 4 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 10 client.26297.objecter _send_command 9 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f35981b8690 con 0x7f35981af880 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35981af880 0x7f35981b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f35981af880 2024-04-02T04:52:33.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f359659c700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35981af880 0x7f35981b3490 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35981b8690 con 0x7f35981af880 2024-04-02T04:52:33.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 9: 0 ) 2024-04-02T04:52:33.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f358ffff700 10 client.26297.objecter _finish_command 9 = system:0 2024-04-02T04:52:33.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f35981a9180 osd=4 4 2024-04-02T04:52:33.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.742+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 4 9 2024-04-02T04:52:33.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 10 client.26297.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 10 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 10 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f35981a9180 osd=4 3 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 4 10 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981a9180 osd=4 4 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 10 client.26297.objecter _send_command 10 2024-04-02T04:52:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35981b39d0 con 0x7f35981af880 2024-04-02T04:52:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35981b39d0 con 0x7f35981af880 2024-04-02T04:52:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f358ffff700 10 client.26297.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f35981a9180 osd=4 4 2024-04-02T04:52:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.762+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 4 10 2024-04-02T04:52:33.763 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 10 client.26297.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 11 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35981b4120 0x7f35981b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 11 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 5 11 2024-04-02T04:52:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b39d0 osd=5 4 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 10 client.26297.objecter _send_command 11 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f35981bb870 con 0x7f35981b4120 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35981b4120 0x7f35981b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358dd9b700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35981b4120 0x7f35981b4580 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f35981b4120 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35981bb870 con 0x7f35981b4120 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 11: 0 ) 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358ffff700 10 client.26297.objecter _finish_command 11 = system:0 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f35981b39d0 osd=5 4 2024-04-02T04:52:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.766+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 5 11 2024-04-02T04:52:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 10 client.26297.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 12 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 12 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f35981b39d0 osd=5 3 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 5 12 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b39d0 osd=5 4 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 10 client.26297.objecter _send_command 12 2024-04-02T04:52:33.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.782+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35981b8f00 con 0x7f35981b4120 2024-04-02T04:52:33.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35981b8f00 con 0x7f35981b4120 2024-04-02T04:52:33.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f358ffff700 10 client.26297.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f35981b39d0 osd=5 4 2024-04-02T04:52:33.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 5 12 2024-04-02T04:52:33.786 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:52:33.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 10 client.26297.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 13 2024-04-02T04:52:33.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35981b9580 0x7f35981b99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:33.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 13 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 6 13 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b8f00 osd=6 4 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 10 client.26297.objecter _send_command 13 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f35981bec10 con 0x7f35981b9580 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35981b9580 0x7f35981b99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f358ffff700 10 client.26297.objecter ms_handle_connect 0x7f35981b9580 2024-04-02T04:52:33.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.786+0000 7f3596d9d700 1 --2- 172.21.15.69:0/2690011343 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35981b9580 0x7f35981b99c0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:33.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.790+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35981bec10 con 0x7f35981b9580 2024-04-02T04:52:33.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.790+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 13: 0 ) 2024-04-02T04:52:33.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.790+0000 7f358ffff700 10 client.26297.objecter _finish_command 13 = system:0 2024-04-02T04:52:33.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.790+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f35981b8f00 osd=6 4 2024-04-02T04:52:33.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.790+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 6 13 2024-04-02T04:52:33.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 10 client.26297.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:52:33.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 14 2024-04-02T04:52:33.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 20 client.26297.objecter _get_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 15 client.26297.objecter _session_command_op_remove -1 14 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 20 client.26297.objecter get_session s=0x7f35981b8f00 osd=6 3 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign 6 14 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 20 client.26297.objecter put_session s=0x7f35981b8f00 osd=6 4 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 10 client.26297.objecter _send_command 14 2024-04-02T04:52:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35981bda90 con 0x7f35981b9580 2024-04-02T04:52:33.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35981bda90 con 0x7f35981b9580 2024-04-02T04:52:33.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f358ffff700 10 client.26297.objecter ms_dispatch 0x7f3598001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:33.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f358ffff700 10 client.26297.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:33.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f358ffff700 20 client.26297.objecter put_session s=0x7f35981b8f00 osd=6 4 2024-04-02T04:52:33.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f358ffff700 15 client.26297.objecter _session_command_op_remove 6 14 2024-04-02T04:52:33.807 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 10 client.26297.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 15 client.26297.objecter _session_command_op_assign -1 15 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 20 client.26297.objecter _get_session osd=-1 returning homeless 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 10 client.26297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.806+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=314}) -- 0x7f35981bee80 con 0x7f35981011a0 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.810+0000 7f359cd34700 1 -- 172.21.15.69:0/2690011343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35981bf050 con 0x7f35981011a0 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.810+0000 7f358ffff700 1 -- 172.21.15.69:0/2690011343 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7f358805bf40 con 0x7f35981011a0 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.810+0000 7f359559a700 10 client.26297.objecter _check_command_map_dne tid 15 current 313 map_dne_bound 313 2024-04-02T04:52:33.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.810+0000 7f359559a700 10 client.26297.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:33.810+0000 7f359559a700 15 client.26297.objecter _session_command_op_remove -1 15 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:52:33.809 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:52:33.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:33.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:33.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:34.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: cluster 2024-04-02T04:52:32.881145+0000 mon.a (mon.0) 2105 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: cluster 2024-04-02T04:52:32.881238+0000 mon.a (mon.0) 2106 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.891035+0000 mon.a (mon.0) 2107 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]': finished 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.891242+0000 mon.a (mon.0) 2108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.891429+0000 mon.a (mon.0) 2109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.891611+0000 mon.a (mon.0) 2110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:52:34.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: cluster 2024-04-02T04:52:32.904705+0000 mon.a (mon.0) 2111 : cluster 0 osdmap e313: 8 total, 7 up, 8 in 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.904792+0000 mon.c (mon.2) 516 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.905391+0000 mon.c (mon.2) 517 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.912793+0000 mon.a (mon.0) 2112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.913492+0000 mon.a (mon.0) 2113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:32.933063+0000 mon.c (mon.2) 518 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:33.042143+0000 mon.c (mon.2) 519 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:33.042449+0000 mon.a (mon.0) 2114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:33.140549+0000 mgr.y (mgr.14538) 353 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: cluster 2024-04-02T04:52:33.269238+0000 osd.0 (osd.0) 193 : cluster 0 3.1 scrub starts 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: cluster 2024-04-02T04:52:33.272442+0000 osd.0 (osd.0) 194 : cluster 0 3.1 scrub ok 2024-04-02T04:52:34.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:33 smithi098 bash[23790]: audit 2024-04-02T04:52:33.660726+0000 mon.c (mon.2) 520 : audit 0 from='client.? 172.21.15.69:0/2690011343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:34.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:34.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: cluster 2024-04-02T04:52:32.881145+0000 mon.a (mon.0) 2105 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:34.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: cluster 2024-04-02T04:52:32.881238+0000 mon.a (mon.0) 2106 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:52:34.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.891035+0000 mon.a (mon.0) 2107 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]': finished 2024-04-02T04:52:34.080 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.891242+0000 mon.a (mon.0) 2108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.891429+0000 mon.a (mon.0) 2109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.891611+0000 mon.a (mon.0) 2110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: cluster 2024-04-02T04:52:32.904705+0000 mon.a (mon.0) 2111 : cluster 0 osdmap e313: 8 total, 7 up, 8 in 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.904792+0000 mon.c (mon.2) 516 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.905391+0000 mon.c (mon.2) 517 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.912793+0000 mon.a (mon.0) 2112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.913492+0000 mon.a (mon.0) 2113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:32.933063+0000 mon.c (mon.2) 518 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:33.042143+0000 mon.c (mon.2) 519 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:33.042449+0000 mon.a (mon.0) 2114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:33.140549+0000 mgr.y (mgr.14538) 353 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: cluster 2024-04-02T04:52:33.269238+0000 osd.0 (osd.0) 193 : cluster 0 3.1 scrub starts 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: cluster 2024-04-02T04:52:33.272442+0000 osd.0 (osd.0) 194 : cluster 0 3.1 scrub ok 2024-04-02T04:52:34.081 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[21528]: audit 2024-04-02T04:52:33.660726+0000 mon.c (mon.2) 520 : audit 0 from='client.? 172.21.15.69:0/2690011343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: cluster 2024-04-02T04:52:32.881145+0000 mon.a (mon.0) 2105 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: cluster 2024-04-02T04:52:32.881238+0000 mon.a (mon.0) 2106 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.891035+0000 mon.a (mon.0) 2107 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-31", "mode": "writeback"}]': finished 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.891242+0000 mon.a (mon.0) 2108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.891429+0000 mon.a (mon.0) 2109 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.891611+0000 mon.a (mon.0) 2110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: cluster 2024-04-02T04:52:32.904705+0000 mon.a (mon.0) 2111 : cluster 0 osdmap e313: 8 total, 7 up, 8 in 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.904792+0000 mon.c (mon.2) 516 : audit 1 from='client.? 172.21.15.69:0/3508737953' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.905391+0000 mon.c (mon.2) 517 : audit 1 from='client.? 172.21.15.69:0/2920716639' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.912793+0000 mon.a (mon.0) 2112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]: dispatch 2024-04-02T04:52:34.082 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.913492+0000 mon.a (mon.0) 2113 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]: dispatch 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:32.933063+0000 mon.c (mon.2) 518 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:33.042143+0000 mon.c (mon.2) 519 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:33.042449+0000 mon.a (mon.0) 2114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:33.140549+0000 mgr.y (mgr.14538) 353 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: cluster 2024-04-02T04:52:33.269238+0000 osd.0 (osd.0) 193 : cluster 0 3.1 scrub starts 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: cluster 2024-04-02T04:52:33.272442+0000 osd.0 (osd.0) 194 : cluster 0 3.1 scrub ok 2024-04-02T04:52:34.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:33 smithi069 bash[28904]: audit 2024-04-02T04:52:33.660726+0000 mon.c (mon.2) 520 : audit 0 from='client.? 172.21.15.69:0/2690011343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:34.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:34.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:34.287 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:34.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:34.415 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:34.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:34.544 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:34.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:34.672 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:34.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:34.802 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:34.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:34.929 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:34.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:35.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:35.163 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:35.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:34 smithi069 bash[28904]: cluster 2024-04-02T04:52:33.664637+0000 osd.5 (osd.5) 189 : cluster 0 102.b scrub starts 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:34 smithi069 bash[28904]: cluster 2024-04-02T04:52:33.669164+0000 osd.5 (osd.5) 190 : cluster 0 102.b scrub ok 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.902337+0000 mon.a (mon.0) 2115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.902435+0000 mon.a (mon.0) 2116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.902523+0000 mon.a (mon.0) 2117 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: cluster 2024-04-02T04:52:33.909016+0000 mon.a (mon.0) 2118 : cluster 0 osdmap e314: 8 total, 7 up, 8 in 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.910846+0000 mon.c (mon.2) 521 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.912350+0000 mon.a (mon.0) 2119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.930433+0000 mon.c (mon.2) 522 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.931261+0000 mon.a (mon.0) 2120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.931594+0000 mon.c (mon.2) 523 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.931860+0000 mon.c (mon.2) 524 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.931946+0000 mon.a (mon.0) 2121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.932336+0000 mon.a (mon.0) 2122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.932479+0000 mon.c (mon.2) 525 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.932742+0000 mon.a (mon.0) 2123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:33.664637+0000 osd.5 (osd.5) 189 : cluster 0 102.b scrub starts 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:33.669164+0000 osd.5 (osd.5) 190 : cluster 0 102.b scrub ok 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.902337+0000 mon.a (mon.0) 2115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.902435+0000 mon.a (mon.0) 2116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.902523+0000 mon.a (mon.0) 2117 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:33.909016+0000 mon.a (mon.0) 2118 : cluster 0 osdmap e314: 8 total, 7 up, 8 in 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.910846+0000 mon.c (mon.2) 521 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.912350+0000 mon.a (mon.0) 2119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.930433+0000 mon.c (mon.2) 522 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.931261+0000 mon.a (mon.0) 2120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.931594+0000 mon.c (mon.2) 523 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.931860+0000 mon.c (mon.2) 524 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.931946+0000 mon.a (mon.0) 2121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.932336+0000 mon.a (mon.0) 2122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.932479+0000 mon.c (mon.2) 525 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.932742+0000 mon.a (mon.0) 2123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.932939+0000 mon.c (mon.2) 526 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.933218+0000 mon.a (mon.0) 2124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.933346+0000 mon.c (mon.2) 527 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.933637+0000 mon.a (mon.0) 2125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:33.934059+0000 mon.c (mon.2) 528 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.004798+0000 mgr.y (mgr.14538) 354 : cluster 0 pgmap v448: 356 pgs: 89 active+undersized, 230 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:35.178 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.324874+0000 osd.1 (osd.1) 201 : cluster 0 40.12 scrub starts 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.326878+0000 osd.1 (osd.1) 202 : cluster 0 40.12 scrub ok 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.823204+0000 mon.a (mon.0) 2126 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.825008+0000 mon.a (mon.0) 2127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]': finished 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.825179+0000 mon.a (mon.0) 2128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.825305+0000 mon.a (mon.0) 2129 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.831149+0000 mon.a (mon.0) 2130 : cluster 0 osdmap e315: 8 total, 7 up, 8 in 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.835345+0000 mon.c (mon.2) 529 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.835731+0000 mon.c (mon.2) 530 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.835924+0000 mon.a (mon.0) 2131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.179 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[28904]: audit 2024-04-02T04:52:34.836796+0000 mon.a (mon.0) 2132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.932939+0000 mon.c (mon.2) 526 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.933218+0000 mon.a (mon.0) 2124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.933346+0000 mon.c (mon.2) 527 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.933637+0000 mon.a (mon.0) 2125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:33.934059+0000 mon.c (mon.2) 528 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.004798+0000 mgr.y (mgr.14538) 354 : cluster 0 pgmap v448: 356 pgs: 89 active+undersized, 230 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.324874+0000 osd.1 (osd.1) 201 : cluster 0 40.12 scrub starts 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.326878+0000 osd.1 (osd.1) 202 : cluster 0 40.12 scrub ok 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.823204+0000 mon.a (mon.0) 2126 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.825008+0000 mon.a (mon.0) 2127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]': finished 2024-04-02T04:52:35.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.825179+0000 mon.a (mon.0) 2128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:35.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.825305+0000 mon.a (mon.0) 2129 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:35.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.831149+0000 mon.a (mon.0) 2130 : cluster 0 osdmap e315: 8 total, 7 up, 8 in 2024-04-02T04:52:35.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.835345+0000 mon.c (mon.2) 529 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.835731+0000 mon.c (mon.2) 530 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.835924+0000 mon.a (mon.0) 2131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:35 smithi069 bash[21528]: audit 2024-04-02T04:52:34.836796+0000 mon.a (mon.0) 2132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:33.664637+0000 osd.5 (osd.5) 189 : cluster 0 102.b scrub starts 2024-04-02T04:52:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:33.669164+0000 osd.5 (osd.5) 190 : cluster 0 102.b scrub ok 2024-04-02T04:52:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.902337+0000 mon.a (mon.0) 2115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-72153-48"}]': finished 2024-04-02T04:52:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.902435+0000 mon.a (mon.0) 2116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-72145-38"}]': finished 2024-04-02T04:52:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.902523+0000 mon.a (mon.0) 2117 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:33.909016+0000 mon.a (mon.0) 2118 : cluster 0 osdmap e314: 8 total, 7 up, 8 in 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.910846+0000 mon.c (mon.2) 521 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.912350+0000 mon.a (mon.0) 2119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.930433+0000 mon.c (mon.2) 522 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.931261+0000 mon.a (mon.0) 2120 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.931594+0000 mon.c (mon.2) 523 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.931860+0000 mon.c (mon.2) 524 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.931946+0000 mon.a (mon.0) 2121 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.932336+0000 mon.a (mon.0) 2122 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.932479+0000 mon.c (mon.2) 525 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.932742+0000 mon.a (mon.0) 2123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.932939+0000 mon.c (mon.2) 526 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.933218+0000 mon.a (mon.0) 2124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.933346+0000 mon.c (mon.2) 527 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.933637+0000 mon.a (mon.0) 2125 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:33.934059+0000 mon.c (mon.2) 528 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.004798+0000 mgr.y (mgr.14538) 354 : cluster 0 pgmap v448: 356 pgs: 89 active+undersized, 230 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.324874+0000 osd.1 (osd.1) 201 : cluster 0 40.12 scrub starts 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.326878+0000 osd.1 (osd.1) 202 : cluster 0 40.12 scrub ok 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.823204+0000 mon.a (mon.0) 2126 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.825008+0000 mon.a (mon.0) 2127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-31"}]': finished 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.825179+0000 mon.a (mon.0) 2128 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-72145-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.825305+0000 mon.a (mon.0) 2129 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-72153-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.831149+0000 mon.a (mon.0) 2130 : cluster 0 osdmap e315: 8 total, 7 up, 8 in 2024-04-02T04:52:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.835345+0000 mon.c (mon.2) 529 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.835731+0000 mon.c (mon.2) 530 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.835924+0000 mon.a (mon.0) 2131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:35.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:34 smithi098 bash[23790]: audit 2024-04-02T04:52:34.836796+0000 mon.a (mon.0) 2132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:35.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:35.315 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:35.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:35.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:35.573 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:35.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:35.701 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:35.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:35.830 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:35.946 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:35.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:36.072 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:36.095 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:36.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:36.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:36.345 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:36.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:36.473 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:36.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:36.597 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:36.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:36.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:36.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:36.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.408300+0000 osd.6 (osd.6) 149 : cluster 0 13.4 scrub starts 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.409930+0000 osd.6 (osd.6) 150 : cluster 0 13.4 scrub ok 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.431706+0000 osd.2 (osd.2) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.431842+0000 osd.2 (osd.2) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.709590+0000 osd.5 (osd.5) 191 : cluster 0 102.14 scrub starts 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: cluster 2024-04-02T04:52:34.712593+0000 osd.5 (osd.5) 192 : cluster 0 102.14 scrub ok 2024-04-02T04:52:36.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:36 smithi098 bash[23790]: audit 2024-04-02T04:52:35.037002+0000 mon.c (mon.2) 531 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:36.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:37.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:37.111 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:37.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.408300+0000 osd.6 (osd.6) 149 : cluster 0 13.4 scrub starts 2024-04-02T04:52:37.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.409930+0000 osd.6 (osd.6) 150 : cluster 0 13.4 scrub ok 2024-04-02T04:52:37.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.431706+0000 osd.2 (osd.2) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:37.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.431842+0000 osd.2 (osd.2) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.709590+0000 osd.5 (osd.5) 191 : cluster 0 102.14 scrub starts 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: cluster 2024-04-02T04:52:34.712593+0000 osd.5 (osd.5) 192 : cluster 0 102.14 scrub ok 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[28904]: audit 2024-04-02T04:52:35.037002+0000 mon.c (mon.2) 531 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.408300+0000 osd.6 (osd.6) 149 : cluster 0 13.4 scrub starts 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.409930+0000 osd.6 (osd.6) 150 : cluster 0 13.4 scrub ok 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.431706+0000 osd.2 (osd.2) 147 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.431842+0000 osd.2 (osd.2) 148 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.709590+0000 osd.5 (osd.5) 191 : cluster 0 102.14 scrub starts 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: cluster 2024-04-02T04:52:34.712593+0000 osd.5 (osd.5) 192 : cluster 0 102.14 scrub ok 2024-04-02T04:52:37.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:36 smithi069 bash[21528]: audit 2024-04-02T04:52:35.037002+0000 mon.c (mon.2) 531 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.215 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:37.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4183407556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c101150 msgr2=0x7f653c10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- 172.21.15.69:0/4183407556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c101150 0x7f653c10f120 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f653400b0d0 tx=0x7f653400b3e0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4183407556 shutdown_connections 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- 172.21.15.69:0/4183407556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c101150 0x7f653c10f120 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- 172.21.15.69:0/4183407556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 0x7f653c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- 172.21.15.69:0/4183407556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c100790 0x7f653c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4183407556 >> 172.21.15.69:0/4183407556 conn(0x7f653c077740 msgr2=0x7f653c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4183407556 shutdown_connections 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4183407556 wait complete. 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 librados: starting msgr at 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 librados: starting objecter 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 Processor -- start 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- start start 2024-04-02T04:52:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 librados: setting wanted keys 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 librados: calling monclient init 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 0x7f653c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c101150 0x7f653c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 0x7f653c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f653c111b80 con 0x7f653c107260 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f653c111a00 con 0x7f653c101150 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f653c111d00 con 0x7f653c100790 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 0x7f653c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 0x7f653c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37028/0 (socket says 172.21.15.69:37028) 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f6540cea700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 0x7f653c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 -- 172.21.15.69:0/4155980551 learned_addr learned my addr 172.21.15.69:0/4155980551 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653b7fe700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c101150 0x7f653c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 -- 172.21.15.69:0/4155980551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c101150 msgr2=0x7f653c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c101150 0x7f653c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 -- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 msgr2=0x7f653c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 0x7f653c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 -- 172.21.15.69:0/4155980551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c19d000 con 0x7f653c100790 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f6540cea700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 0x7f653c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f653bfff700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 0x7f653c198040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f652c00eb30 tx=0x7f652c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f65397fa700 10 client.?.objecter ms_handle_connect 0x7f653c100790 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f65397fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c00cc90 con 0x7f653c100790 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c19d2f0 con 0x7f653c100790 2024-04-02T04:52:37.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f652c00ce30 con 0x7f653c100790 2024-04-02T04:52:37.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c003df0 con 0x7f653c100790 2024-04-02T04:52:37.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 10 client.26315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:37.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c1a4b70 con 0x7f653c100790 2024-04-02T04:52:37.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 librados: init done 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653c04ff00 con 0x7f653c100790 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.290+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f652c004000 con 0x7f653c100790 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f652406f5e0 0x7f6524071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(317..317 src has 74..317) ==== 12598+0+0 (secure 0 0 0) 0x7f652c08f720 con 0x7f653c100790 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 10 client.26315.objecter ms_dispatch 0x7f653c001550 osd_map(317..317 src has 74..317) 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 3 client.26315.objecter handle_osd_map got epochs [317,317] > 0 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 3 client.26315.objecter handle_osd_map decoding full epoch 317 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f653b7fe700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f652406f5e0 0x7f6524071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 20 client.26315.objecter dump_active .. 0 homeless 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f65397fa700 10 client.26315.objecter ms_handle_connect 0x7f652406f5e0 2024-04-02T04:52:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.294+0000 7f653b7fe700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f652406f5e0 0x7f6524071aa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f653c0fff40 tx=0x7f6530005eb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:37.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.298+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f652c05b8f0 con 0x7f653c100790 2024-04-02T04:52:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:52:37.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:52:37.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:37.352 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:37.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:37.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.502+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f653c105450 con 0x7f652406f5e0 2024-04-02T04:52:37.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.530+0000 7f65397fa700 1 -- 172.21.15.69:0/4155980551 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+948762 (secure 0 0 0) 0x7f653c105450 con 0x7f652406f5e0 2024-04-02T04:52:37.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:52:37.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 10 librados: watch_flush enter 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 10 librados: watch_flush exit 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 20 client.26315.objecter shutdown clearing up homeless session... 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 10 client.26315.objecter successfully canceled tick 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f652406f5e0 msgr2=0x7f6524071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f652406f5e0 0x7f6524071aa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f653c0fff40 tx=0x7f6530005eb0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 msgr2=0x7f653c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 0x7f653c198040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f652c00eb30 tx=0x7f652c00ee40 comp rx=0 tx=0).stop 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 shutdown_connections 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f652406f5e0 0x7f6524071aa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f653c107260 0x7f653c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f653c100790 0x7f653c198040 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 --2- 172.21.15.69:0/4155980551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f653c101150 0x7f653c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 >> 172.21.15.69:0/4155980551 conn(0x7f653c077740 msgr2=0x7f653c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:37.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 shutdown_connections 2024-04-02T04:52:37.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 -- 172.21.15.69:0/4155980551 wait complete. 2024-04-02T04:52:37.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:37.538+0000 7f654274d700 1 librados: shutdown 2024-04-02T04:52:37.552 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:52:37.583 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:37.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:37.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:37.727 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:37.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:35.785278+0000 osd.3 (osd.3) 221 : cluster 0 40.13 scrub starts 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:35.787720+0000 osd.3 (osd.3) 222 : cluster 0 40.13 scrub ok 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:36.005912+0000 mgr.y (mgr.14538) 355 : cluster 0 pgmap v451: 324 pgs: 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:36.038719+0000 mon.c (mon.2) 532 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:36.308505+0000 osd.1 (osd.1) 203 : cluster 0 43.12 scrub starts 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:36.311295+0000 osd.1 (osd.1) 204 : cluster 0 43.12 scrub ok 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:36.665916+0000 osd.5 (osd.5) 193 : cluster 0 43.11 scrub starts 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:36.684667+0000 osd.5 (osd.5) 194 : cluster 0 43.11 scrub ok 2024-04-02T04:52:37.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:36.717706+0000 mon.a (mon.0) 2133 : cluster 0 osdmap e316: 8 total, 7 up, 8 in 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:37.039907+0000 mon.c (mon.2) 533 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:37.154386+0000 mon.a (mon.0) 2134 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:37.154486+0000 mon.a (mon.0) 2135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: cluster 2024-04-02T04:52:37.157127+0000 mon.a (mon.0) 2136 : cluster 0 osdmap e317: 8 total, 7 up, 8 in 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:37.167763+0000 mon.c (mon.2) 534 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:37.169161+0000 mon.a (mon.0) 2137 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[21528]: audit 2024-04-02T04:52:37.504796+0000 mgr.y (mgr.14538) 356 : audit 0 from='client.26315 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:35.785278+0000 osd.3 (osd.3) 221 : cluster 0 40.13 scrub starts 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:35.787720+0000 osd.3 (osd.3) 222 : cluster 0 40.13 scrub ok 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:36.005912+0000 mgr.y (mgr.14538) 355 : cluster 0 pgmap v451: 324 pgs: 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:37.835 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:36.038719+0000 mon.c (mon.2) 532 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:36.308505+0000 osd.1 (osd.1) 203 : cluster 0 43.12 scrub starts 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:36.311295+0000 osd.1 (osd.1) 204 : cluster 0 43.12 scrub ok 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:36.665916+0000 osd.5 (osd.5) 193 : cluster 0 43.11 scrub starts 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:36.684667+0000 osd.5 (osd.5) 194 : cluster 0 43.11 scrub ok 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:36.717706+0000 mon.a (mon.0) 2133 : cluster 0 osdmap e316: 8 total, 7 up, 8 in 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:37.039907+0000 mon.c (mon.2) 533 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:37.154386+0000 mon.a (mon.0) 2134 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:37.154486+0000 mon.a (mon.0) 2135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: cluster 2024-04-02T04:52:37.157127+0000 mon.a (mon.0) 2136 : cluster 0 osdmap e317: 8 total, 7 up, 8 in 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:37.167763+0000 mon.c (mon.2) 534 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:37.169161+0000 mon.a (mon.0) 2137 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:37.836 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:37 smithi069 bash[28904]: audit 2024-04-02T04:52:37.504796+0000 mgr.y (mgr.14538) 356 : audit 0 from='client.26315 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:52:37.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:35.785278+0000 osd.3 (osd.3) 221 : cluster 0 40.13 scrub starts 2024-04-02T04:52:37.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:35.787720+0000 osd.3 (osd.3) 222 : cluster 0 40.13 scrub ok 2024-04-02T04:52:37.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:36.005912+0000 mgr.y (mgr.14538) 355 : cluster 0 pgmap v451: 324 pgs: 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1527 objects degraded (18.206%) 2024-04-02T04:52:37.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:36.038719+0000 mon.c (mon.2) 532 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:36.308505+0000 osd.1 (osd.1) 203 : cluster 0 43.12 scrub starts 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:36.311295+0000 osd.1 (osd.1) 204 : cluster 0 43.12 scrub ok 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:36.665916+0000 osd.5 (osd.5) 193 : cluster 0 43.11 scrub starts 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:36.684667+0000 osd.5 (osd.5) 194 : cluster 0 43.11 scrub ok 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:36.717706+0000 mon.a (mon.0) 2133 : cluster 0 osdmap e316: 8 total, 7 up, 8 in 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:37.039907+0000 mon.c (mon.2) 533 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:37.154386+0000 mon.a (mon.0) 2134 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-72145-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:37.154486+0000 mon.a (mon.0) 2135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-72153-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: cluster 2024-04-02T04:52:37.157127+0000 mon.a (mon.0) 2136 : cluster 0 osdmap e317: 8 total, 7 up, 8 in 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:37.167763+0000 mon.c (mon.2) 534 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:37.169161+0000 mon.a (mon.0) 2137 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:37.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:37 smithi098 bash[23790]: audit 2024-04-02T04:52:37.504796+0000 mgr.y (mgr.14538) 356 : audit 0 from='client.26315 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:52:37.858 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:37.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:37.988 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:37.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:38.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:52:38.091 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:38.110 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:38.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:38.238 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:38.262 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:52:38.262 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:52:38.262 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:52:38.262 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:52:38.262 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:52:38.342 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:38.362 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:38.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:38.485 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:38.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:38.611 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:38.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:38.739 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:38.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:38.870 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:38.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:38.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.007977+0000 mgr.y (mgr.14538) 357 : cluster 0 pgmap v453: 372 pgs: 48 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 249 B/s wr, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: audit 2024-04-02T04:52:38.041034+0000 mon.c (mon.2) 535 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: audit 2024-04-02T04:52:38.156532+0000 mon.a (mon.0) 2138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.161610+0000 mon.a (mon.0) 2139 : cluster 0 osdmap e318: 8 total, 7 up, 8 in 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: audit 2024-04-02T04:52:38.163950+0000 mon.c (mon.2) 536 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: audit 2024-04-02T04:52:38.169676+0000 mon.a (mon.0) 2140 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.433724+0000 osd.4 (osd.4) 193 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:38.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:38 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.433904+0000 osd.4 (osd.4) 194 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:38.998 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:52:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:52:38.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.007977+0000 mgr.y (mgr.14538) 357 : cluster 0 pgmap v453: 372 pgs: 48 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 249 B/s wr, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: audit 2024-04-02T04:52:38.041034+0000 mon.c (mon.2) 535 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: audit 2024-04-02T04:52:38.156532+0000 mon.a (mon.0) 2138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.161610+0000 mon.a (mon.0) 2139 : cluster 0 osdmap e318: 8 total, 7 up, 8 in 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: audit 2024-04-02T04:52:38.163950+0000 mon.c (mon.2) 536 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: audit 2024-04-02T04:52:38.169676+0000 mon.a (mon.0) 2140 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.433724+0000 osd.4 (osd.4) 193 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:38.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.433904+0000 osd.4 (osd.4) 194 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:39.000 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.007977+0000 mgr.y (mgr.14538) 357 : cluster 0 pgmap v453: 372 pgs: 48 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 249 B/s wr, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: audit 2024-04-02T04:52:38.041034+0000 mon.c (mon.2) 535 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: audit 2024-04-02T04:52:38.156532+0000 mon.a (mon.0) 2138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.161610+0000 mon.a (mon.0) 2139 : cluster 0 osdmap e318: 8 total, 7 up, 8 in 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: audit 2024-04-02T04:52:38.163950+0000 mon.c (mon.2) 536 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: audit 2024-04-02T04:52:38.169676+0000 mon.a (mon.0) 2140 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.433724+0000 osd.4 (osd.4) 193 : cluster 0 purged_snaps scrub starts 2024-04-02T04:52:39.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:38 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.433904+0000 osd.4 (osd.4) 194 : cluster 0 purged_snaps scrub ok 2024-04-02T04:52:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:39.013 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-04-02T04:52:37.158055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4795303,"num_objects":509,"num_object_clones":142,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":2,"num_read":1801,"num_read_kb":1523,"num_write":1453,"num_write_kb":11251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771129,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2499,"ondisk_log_size":2499,"up":855,"acting":855,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1192,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":644140,"kb_used_data":128144,"kb_used_omap":12,"kb_used_meta":515827,"kb_avail":749317076,"statfs":{"total":767960285184,"available":767300685824,"internally_reserved":0,"allocated":131219456,"data_stored":113946462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":528207408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.265575"},"pg_stats":[{"pgid":"259.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"259.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"257.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:52:37.147949+0000","last_change":"2024-04-02T04:52:37.147949+0000","last_active":"2024-04-02T04:52:37.147949+0000","last_peered":"2024-04-02T04:52:37.147949+0000","last_clean":"2024-04-02T04:52:37.147949+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:52:37.147949+0000","last_undegraded":"2024-04-02T04:52:37.147949+0000","last_fullsized":"2024-04-02T04:52:37.147949+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:52:37.147949+0000","last_clean_scrub_stamp":"2024-04-02T04:52:37.147949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"102.19","version":"192'45","reported_seq":419,"reported_epoch":311,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:30.880805+0000","last_change":"2024-04-02T04:50:27.532331+0000","last_active":"2024-04-02T04:52:30.880805+0000","last_peered":"2024-04-02T04:52:30.880805+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.255616+0000","last_became_peered":"2024-04-02T04:49:55.255616+0000","last_unstale":"2024-04-02T04:52:30.880805+0000","last_undegraded":"2024-04-02T04:50:27.523866+0000","last_fullsized":"2024-04-02T04:49:55.215309+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":25}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.18","version":"263'35","reported_seq":460,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.888382+0000","last_change":"2024-04-02T04:52:24.858717+0000","last_active":"2024-04-02T04:52:30.888382+0000","last_peered":"2024-04-02T04:52:30.888382+0000","last_clean":"2024-04-02T04:52:30.888382+0000","last_became_active":"2024-04-02T04:49:56.572185+0000","last_became_peered":"2024-04-02T04:49:56.572185+0000","last_unstale":"2024-04-02T04:52:30.888382+0000","last_undegraded":"2024-04-02T04:52:30.888382+0000","last_fullsized":"2024-04-02T04:52:30.888382+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'35","last_scrub_stamp":"2024-04-02T04:51:56.184367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.184367+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:07.322467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031551970000000002,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1b","version":"263'37","reported_seq":463,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.441743+0000","last_change":"2024-04-02T04:52:24.887845+0000","last_active":"2024-04-02T04:52:32.441743+0000","last_peered":"2024-04-02T04:52:32.441743+0000","last_clean":"2024-04-02T04:52:32.441743+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:52:32.441743+0000","last_undegraded":"2024-04-02T04:52:32.441743+0000","last_fullsized":"2024-04-02T04:52:32.441743+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041402610000000001,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1a","version":"252'68","reported_seq":481,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.884902+0000","last_change":"2024-04-02T04:52:24.861534+0000","last_active":"2024-04-02T04:52:30.884902+0000","last_peered":"2024-04-02T04:52:30.884902+0000","last_clean":"2024-04-02T04:52:30.884902+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:52:30.884902+0000","last_undegraded":"2024-04-02T04:52:30.884902+0000","last_fullsized":"2024-04-02T04:52:30.884902+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'68","last_scrub_stamp":"2024-04-02T04:51:53.146929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.146929+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:01.347211+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044478449999999998,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1d","version":"248'85","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891583+0000","last_change":"2024-04-02T04:52:24.859133+0000","last_active":"2024-04-02T04:52:31.891583+0000","last_peered":"2024-04-02T04:52:31.891583+0000","last_clean":"2024-04-02T04:52:31.891583+0000","last_became_active":"2024-04-02T04:49:56.582887+0000","last_became_peered":"2024-04-02T04:49:56.582887+0000","last_unstale":"2024-04-02T04:52:31.891583+0000","last_undegraded":"2024-04-02T04:52:31.891583+0000","last_fullsized":"2024-04-02T04:52:31.891583+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001500936,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1c","version":"244'34","reported_seq":462,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.890480+0000","last_change":"2024-04-02T04:52:24.860732+0000","last_active":"2024-04-02T04:52:30.890480+0000","last_peered":"2024-04-02T04:52:30.890480+0000","last_clean":"2024-04-02T04:52:30.890480+0000","last_became_active":"2024-04-02T04:49:56.570832+0000","last_became_peered":"2024-04-02T04:49:56.570832+0000","last_unstale":"2024-04-02T04:52:30.890480+0000","last_undegraded":"2024-04-02T04:52:30.890480+0000","last_fullsized":"2024-04-02T04:52:30.890480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'34","last_scrub_stamp":"2024-04-02T04:51:47.142172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:47.142172+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:47.554788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039078569999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1f","version":"273'39","reported_seq":465,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.896009+0000","last_change":"2024-04-02T04:52:24.843938+0000","last_active":"2024-04-02T04:52:31.896009+0000","last_peered":"2024-04-02T04:52:31.896009+0000","last_clean":"2024-04-02T04:52:31.896009+0000","last_became_active":"2024-04-02T04:49:55.249289+0000","last_became_peered":"2024-04-02T04:49:55.249289+0000","last_unstale":"2024-04-02T04:52:31.896009+0000","last_undegraded":"2024-04-02T04:52:31.896009+0000","last_fullsized":"2024-04-02T04:52:31.896009+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039518549999999998,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1e","version":"248'17","reported_seq":445,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894887+0000","last_change":"2024-04-02T04:52:24.861811+0000","last_active":"2024-04-02T04:52:30.894887+0000","last_peered":"2024-04-02T04:52:30.894887+0000","last_clean":"2024-04-02T04:52:30.894887+0000","last_became_active":"2024-04-02T04:49:56.563968+0000","last_became_peered":"2024-04-02T04:49:56.563968+0000","last_unstale":"2024-04-02T04:52:30.894887+0000","last_undegraded":"2024-04-02T04:52:30.894887+0000","last_fullsized":"2024-04-02T04:52:30.894887+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:51:55.166519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.166519+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:24.630363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002240077,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.11","version":"192'48","reported_seq":435,"reported_epoch":311,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:30.883982+0000","last_change":"2024-04-02T04:50:18.405897+0000","last_active":"2024-04-02T04:52:30.883982+0000","last_peered":"2024-04-02T04:52:30.883982+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.569001+0000","last_became_peered":"2024-04-02T04:49:56.569001+0000","last_unstale":"2024-04-02T04:52:30.883982+0000","last_undegraded":"2024-04-02T04:50:17.425678+0000","last_fullsized":"2024-04-02T04:49:56.526356+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273,"num_objects":27,"num_object_clones":21,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":27}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.10","version":"192'18","reported_seq":401,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.882860+0000","last_change":"2024-04-02T04:50:27.548680+0000","last_active":"2024-04-02T04:52:31.882860+0000","last_peered":"2024-04-02T04:52:31.882860+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.243496+0000","last_became_peered":"2024-04-02T04:49:55.243496+0000","last_unstale":"2024-04-02T04:52:31.882860+0000","last_undegraded":"2024-04-02T04:50:27.476730+0000","last_fullsized":"2024-04-02T04:49:55.211603+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.13","version":"247'50","reported_seq":430,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.441827+0000","last_change":"2024-04-02T04:50:18.418843+0000","last_active":"2024-04-02T04:52:32.441827+0000","last_peered":"2024-04-02T04:52:32.441827+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.554860+0000","last_became_peered":"2024-04-02T04:49:56.554860+0000","last_unstale":"2024-04-02T04:52:32.441827+0000","last_undegraded":"2024-04-02T04:50:17.426824+0000","last_fullsized":"2024-04-02T04:49:56.518531+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":6,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.12","version":"240'17","reported_seq":451,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886325+0000","last_change":"2024-04-02T04:52:24.852711+0000","last_active":"2024-04-02T04:52:31.886325+0000","last_peered":"2024-04-02T04:52:31.886325+0000","last_clean":"2024-04-02T04:52:31.886325+0000","last_became_active":"2024-04-02T04:49:55.255700+0000","last_became_peered":"2024-04-02T04:49:55.255700+0000","last_unstale":"2024-04-02T04:52:31.886325+0000","last_undegraded":"2024-04-02T04:52:31.886325+0000","last_fullsized":"2024-04-02T04:52:31.886325+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-04-02T04:51:53.016585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.016585+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:04.745749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.016320628,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.15","version":"192'37","reported_seq":420,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.457490+0000","last_change":"2024-04-02T04:50:09.415659+0000","last_active":"2024-04-02T04:52:32.457490+0000","last_peered":"2024-04-02T04:52:32.457490+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.561896+0000","last_became_peered":"2024-04-02T04:49:56.561896+0000","last_unstale":"2024-04-02T04:52:32.457490+0000","last_undegraded":"2024-04-02T04:50:08.454406+0000","last_fullsized":"2024-04-02T04:49:56.542252+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.14","version":"273'153","reported_seq":563,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.619034+0000","last_change":"2024-04-02T04:52:24.880382+0000","last_active":"2024-04-02T04:52:32.619034+0000","last_peered":"2024-04-02T04:52:32.619034+0000","last_clean":"2024-04-02T04:52:32.619034+0000","last_became_active":"2024-04-02T04:49:55.269267+0000","last_became_peered":"2024-04-02T04:49:55.269267+0000","last_unstale":"2024-04-02T04:52:32.619034+0000","last_undegraded":"2024-04-02T04:52:32.619034+0000","last_fullsized":"2024-04-02T04:52:32.619034+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011648324,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":45,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.17","version":"249'34","reported_seq":464,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895131+0000","last_change":"2024-04-02T04:52:24.864288+0000","last_active":"2024-04-02T04:52:30.895131+0000","last_peered":"2024-04-02T04:52:30.895131+0000","last_clean":"2024-04-02T04:52:30.895131+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:52:30.895131+0000","last_undegraded":"2024-04-02T04:52:30.895131+0000","last_fullsized":"2024-04-02T04:52:30.895131+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'34","last_scrub_stamp":"2024-04-02T04:51:48.141397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.141397+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.858294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0024840909999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.16","version":"252'17","reported_seq":443,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.621329+0000","last_change":"2024-04-02T04:52:24.881096+0000","last_active":"2024-04-02T04:52:32.621329+0000","last_peered":"2024-04-02T04:52:32.621329+0000","last_clean":"2024-04-02T04:52:32.621329+0000","last_became_active":"2024-04-02T04:49:55.269224+0000","last_became_peered":"2024-04-02T04:49:55.269224+0000","last_unstale":"2024-04-02T04:52:32.621329+0000","last_undegraded":"2024-04-02T04:52:32.621329+0000","last_fullsized":"2024-04-02T04:52:32.621329+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095323079999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.9","version":"192'18","reported_seq":395,"reported_epoch":311,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:30.882527+0000","last_change":"2024-04-02T04:50:27.575689+0000","last_active":"2024-04-02T04:52:30.882527+0000","last_peered":"2024-04-02T04:52:30.882527+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.562137+0000","last_became_peered":"2024-04-02T04:49:56.562137+0000","last_unstale":"2024-04-02T04:52:30.882527+0000","last_undegraded":"2024-04-02T04:50:27.479688+0000","last_fullsized":"2024-04-02T04:49:56.524836+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.8","version":"248'17","reported_seq":442,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.893263+0000","last_change":"2024-04-02T04:52:24.914086+0000","last_active":"2024-04-02T04:52:30.893263+0000","last_peered":"2024-04-02T04:52:30.893263+0000","last_clean":"2024-04-02T04:52:30.893263+0000","last_became_active":"2024-04-02T04:49:55.259553+0000","last_became_peered":"2024-04-02T04:49:55.259553+0000","last_unstale":"2024-04-02T04:52:30.893263+0000","last_undegraded":"2024-04-02T04:52:30.893263+0000","last_fullsized":"2024-04-02T04:52:30.893263+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087158440000000004,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.b","version":"258'73","reported_seq":486,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.619420+0000","last_change":"2024-04-02T04:52:24.879895+0000","last_active":"2024-04-02T04:52:32.619420+0000","last_peered":"2024-04-02T04:52:32.619420+0000","last_clean":"2024-04-02T04:52:32.619420+0000","last_became_active":"2024-04-02T04:49:55.267102+0000","last_became_peered":"2024-04-02T04:49:55.267102+0000","last_unstale":"2024-04-02T04:52:32.619420+0000","last_undegraded":"2024-04-02T04:52:32.619420+0000","last_fullsized":"2024-04-02T04:52:32.619420+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011470035,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.a","version":"254'68","reported_seq":483,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.618186+0000","last_change":"2024-04-02T04:52:24.872340+0000","last_active":"2024-04-02T04:52:32.618186+0000","last_peered":"2024-04-02T04:52:32.618186+0000","last_clean":"2024-04-02T04:52:32.618186+0000","last_became_active":"2024-04-02T04:49:55.252341+0000","last_became_peered":"2024-04-02T04:49:55.252341+0000","last_unstale":"2024-04-02T04:52:32.618186+0000","last_undegraded":"2024-04-02T04:52:32.618186+0000","last_fullsized":"2024-04-02T04:52:32.618186+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055600299999999997,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.d","version":"288'69","reported_seq":442,"reported_epoch":311,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:30.894368+0000","last_change":"2024-04-02T04:50:27.508913+0000","last_active":"2024-04-02T04:52:30.894368+0000","last_peered":"2024-04-02T04:52:30.894368+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.566448+0000","last_became_peered":"2024-04-02T04:49:56.566448+0000","last_unstale":"2024-04-02T04:52:30.894368+0000","last_undegraded":"2024-04-02T04:50:27.492108+0000","last_fullsized":"2024-04-02T04:49:56.525265+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":39,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":39}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.c","version":"249'68","reported_seq":481,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894960+0000","last_change":"2024-04-02T04:52:24.913101+0000","last_active":"2024-04-02T04:52:30.894960+0000","last_peered":"2024-04-02T04:52:30.894960+0000","last_clean":"2024-04-02T04:52:30.894960+0000","last_became_active":"2024-04-02T04:49:55.252273+0000","last_became_peered":"2024-04-02T04:49:55.252273+0000","last_unstale":"2024-04-02T04:52:30.894960+0000","last_undegraded":"2024-04-02T04:52:30.894960+0000","last_fullsized":"2024-04-02T04:52:30.894960+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087463609999999994,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.f","version":"306'60","reported_seq":534,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894643+0000","last_change":"2024-04-02T04:52:28.268443+0000","last_active":"2024-04-02T04:52:30.894643+0000","last_peered":"2024-04-02T04:52:30.894643+0000","last_clean":"2024-04-02T04:52:30.894643+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:52:30.894643+0000","last_undegraded":"2024-04-02T04:52:30.894643+0000","last_fullsized":"2024-04-02T04:52:30.894643+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26022431000000001,"stat_sum":{"num_bytes":4194354,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.e","version":"229'56","reported_seq":490,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.451039+0000","last_change":"2024-04-02T04:52:24.889528+0000","last_active":"2024-04-02T04:52:32.451039+0000","last_peered":"2024-04-02T04:52:32.451039+0000","last_clean":"2024-04-02T04:52:32.451039+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:52:32.451039+0000","last_undegraded":"2024-04-02T04:52:32.451039+0000","last_fullsized":"2024-04-02T04:52:32.451039+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'56","last_scrub_stamp":"2024-04-02T04:51:26.235660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.235660+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:41.425356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044202829999999997,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.1","version":"269'30","reported_seq":407,"reported_epoch":311,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:30.881886+0000","last_change":"2024-04-02T04:50:27.514211+0000","last_active":"2024-04-02T04:52:30.881886+0000","last_peered":"2024-04-02T04:52:30.881886+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.245537+0000","last_became_peered":"2024-04-02T04:49:55.245537+0000","last_unstale":"2024-04-02T04:52:30.881886+0000","last_undegraded":"2024-04-02T04:50:27.498272+0000","last_fullsized":"2024-04-02T04:49:55.215487+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":17}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.0","version":"292'90","reported_seq":498,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.881575+0000","last_change":"2024-04-02T04:52:24.911918+0000","last_active":"2024-04-02T04:52:30.881575+0000","last_peered":"2024-04-02T04:52:30.881575+0000","last_clean":"2024-04-02T04:52:30.881575+0000","last_became_active":"2024-04-02T04:49:55.268700+0000","last_became_peered":"2024-04-02T04:49:55.268700+0000","last_unstale":"2024-04-02T04:52:30.881575+0000","last_undegraded":"2024-04-02T04:52:30.881575+0000","last_fullsized":"2024-04-02T04:52:30.881575+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015952999999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.3","version":"242'34","reported_seq":461,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886217+0000","last_change":"2024-04-02T04:52:24.851717+0000","last_active":"2024-04-02T04:52:31.886217+0000","last_peered":"2024-04-02T04:52:31.886217+0000","last_clean":"2024-04-02T04:52:31.886217+0000","last_became_active":"2024-04-02T04:49:55.257987+0000","last_became_peered":"2024-04-02T04:49:55.257987+0000","last_unstale":"2024-04-02T04:52:31.886217+0000","last_undegraded":"2024-04-02T04:52:31.886217+0000","last_fullsized":"2024-04-02T04:52:31.886217+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-04-02T04:51:50.994396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.994396+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:07.729629+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.015914708,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.2","version":"273'39","reported_seq":459,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886418+0000","last_change":"2024-04-02T04:52:24.853004+0000","last_active":"2024-04-02T04:52:31.886418+0000","last_peered":"2024-04-02T04:52:31.886418+0000","last_clean":"2024-04-02T04:52:31.886418+0000","last_became_active":"2024-04-02T04:49:55.235417+0000","last_became_peered":"2024-04-02T04:49:55.235417+0000","last_unstale":"2024-04-02T04:52:31.886418+0000","last_undegraded":"2024-04-02T04:52:31.886418+0000","last_fullsized":"2024-04-02T04:52:31.886418+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015312144999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.5","version":"229'85","reported_seq":508,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.448730+0000","last_change":"2024-04-02T04:52:24.890065+0000","last_active":"2024-04-02T04:52:32.448730+0000","last_peered":"2024-04-02T04:52:32.448730+0000","last_clean":"2024-04-02T04:52:32.448730+0000","last_became_active":"2024-04-02T04:49:56.553311+0000","last_became_peered":"2024-04-02T04:49:56.553311+0000","last_unstale":"2024-04-02T04:52:32.448730+0000","last_undegraded":"2024-04-02T04:52:32.448730+0000","last_fullsized":"2024-04-02T04:52:32.448730+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'85","last_scrub_stamp":"2024-04-02T04:51:27.257882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.257882+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041738679999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.4","version":"192'9","reported_seq":390,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.452413+0000","last_change":"2024-04-02T04:50:28.406787+0000","last_active":"2024-04-02T04:52:32.452413+0000","last_peered":"2024-04-02T04:52:32.452413+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.578685+0000","last_became_peered":"2024-04-02T04:49:56.578685+0000","last_unstale":"2024-04-02T04:52:32.452413+0000","last_undegraded":"2024-04-02T04:50:27.585397+0000","last_fullsized":"2024-04-02T04:49:56.549435+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.7","version":"255'18","reported_seq":420,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.890787+0000","last_change":"2024-04-02T04:52:24.853360+0000","last_active":"2024-04-02T04:52:31.890787+0000","last_peered":"2024-04-02T04:52:31.890787+0000","last_clean":"2024-04-02T04:52:31.890787+0000","last_became_active":"2024-04-02T04:51:57.862004+0000","last_became_peered":"2024-04-02T04:51:57.862004+0000","last_unstale":"2024-04-02T04:52:31.890787+0000","last_undegraded":"2024-04-02T04:52:31.890787+0000","last_fullsized":"2024-04-02T04:52:31.890787+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019926739999999998,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"102.6","version":"236'34","reported_seq":457,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.895558+0000","last_change":"2024-04-02T04:52:24.865251+0000","last_active":"2024-04-02T04:52:31.895558+0000","last_peered":"2024-04-02T04:52:31.895558+0000","last_clean":"2024-04-02T04:52:31.895558+0000","last_became_active":"2024-04-02T04:49:56.581977+0000","last_became_peered":"2024-04-02T04:49:56.581977+0000","last_unstale":"2024-04-02T04:52:31.895558+0000","last_undegraded":"2024-04-02T04:52:31.895558+0000","last_fullsized":"2024-04-02T04:52:31.895558+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028397800000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"}]},{"pgid":"43.14","version":"0'0","reported_seq":356,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891710+0000","last_change":"2024-04-02T04:49:29.828822+0000","last_active":"2024-04-02T04:52:31.891710+0000","last_peered":"2024-04-02T04:52:31.891710+0000","last_clean":"2024-04-02T04:52:31.891710+0000","last_became_active":"2024-04-02T04:49:29.828535+0000","last_became_peered":"2024-04-02T04:49:29.828535+0000","last_unstale":"2024-04-02T04:52:31.891710+0000","last_undegraded":"2024-04-02T04:52:31.891710+0000","last_fullsized":"2024-04-02T04:52:31.891710+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.898254+0000","last_change":"2024-04-02T04:49:47.428191+0000","last_active":"2024-04-02T04:52:31.898254+0000","last_peered":"2024-04-02T04:52:31.898254+0000","last_clean":"2024-04-02T04:49:45.347176+0000","last_became_active":"2024-04-02T04:49:47.428191+0000","last_became_peered":"2024-04-02T04:49:47.428191+0000","last_unstale":"2024-04-02T04:52:31.898254+0000","last_undegraded":"2024-04-02T04:52:31.898254+0000","last_fullsized":"2024-04-02T04:49:47.367526+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.898629+0000","last_change":"2024-04-02T04:49:28.771791+0000","last_active":"2024-04-02T04:52:31.898629+0000","last_peered":"2024-04-02T04:52:31.898629+0000","last_clean":"2024-04-02T04:52:31.898629+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:52:31.898629+0000","last_undegraded":"2024-04-02T04:52:31.898629+0000","last_fullsized":"2024-04-02T04:52:31.898629+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":365,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887792+0000","last_change":"2024-04-02T04:51:46.284187+0000","last_active":"2024-04-02T04:52:31.887792+0000","last_peered":"2024-04-02T04:52:31.887792+0000","last_clean":"2024-04-02T04:52:31.887792+0000","last_became_active":"2024-04-02T04:49:28.787404+0000","last_became_peered":"2024-04-02T04:49:28.787404+0000","last_unstale":"2024-04-02T04:52:31.887792+0000","last_undegraded":"2024-04-02T04:52:31.887792+0000","last_fullsized":"2024-04-02T04:52:31.887792+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:46.284019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:46.284019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.098571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.898605+0000","last_change":"2024-04-02T04:49:28.787925+0000","last_active":"2024-04-02T04:52:31.898605+0000","last_peered":"2024-04-02T04:52:31.898605+0000","last_clean":"2024-04-02T04:52:31.898605+0000","last_became_active":"2024-04-02T04:49:28.783594+0000","last_became_peered":"2024-04-02T04:49:28.783594+0000","last_unstale":"2024-04-02T04:52:31.898605+0000","last_undegraded":"2024-04-02T04:52:31.898605+0000","last_fullsized":"2024-04-02T04:52:31.898605+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888981+0000","last_change":"2024-04-02T04:51:44.929741+0000","last_active":"2024-04-02T04:52:31.888981+0000","last_peered":"2024-04-02T04:52:31.888981+0000","last_clean":"2024-04-02T04:52:31.888981+0000","last_became_active":"2024-04-02T04:49:28.726013+0000","last_became_peered":"2024-04-02T04:49:28.726013+0000","last_unstale":"2024-04-02T04:52:31.888981+0000","last_undegraded":"2024-04-02T04:52:31.888981+0000","last_fullsized":"2024-04-02T04:52:31.888981+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.929545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.929545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:56.903108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.895591+0000","last_change":"2024-04-02T04:49:47.479797+0000","last_active":"2024-04-02T04:52:31.895591+0000","last_peered":"2024-04-02T04:52:31.895591+0000","last_clean":"2024-04-02T04:49:45.330040+0000","last_became_active":"2024-04-02T04:49:47.479797+0000","last_became_peered":"2024-04-02T04:49:47.479797+0000","last_unstale":"2024-04-02T04:52:31.895591+0000","last_undegraded":"2024-04-02T04:52:31.895591+0000","last_fullsized":"2024-04-02T04:49:47.405691+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.619936+0000","last_change":"2024-04-02T04:51:39.644093+0000","last_active":"2024-04-02T04:52:32.619936+0000","last_peered":"2024-04-02T04:52:32.619936+0000","last_clean":"2024-04-02T04:52:32.619936+0000","last_became_active":"2024-04-02T04:49:28.759334+0000","last_became_peered":"2024-04-02T04:49:28.759334+0000","last_unstale":"2024-04-02T04:52:32.619936+0000","last_undegraded":"2024-04-02T04:52:32.619936+0000","last_fullsized":"2024-04-02T04:52:32.619936+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:43.476605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":358,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.888639+0000","last_change":"2024-04-02T04:49:47.517914+0000","last_active":"2024-04-02T04:52:31.888639+0000","last_peered":"2024-04-02T04:52:31.888639+0000","last_clean":"2024-04-02T04:49:45.379252+0000","last_became_active":"2024-04-02T04:49:47.517914+0000","last_became_peered":"2024-04-02T04:49:47.517914+0000","last_unstale":"2024-04-02T04:52:31.888639+0000","last_undegraded":"2024-04-02T04:52:31.888639+0000","last_fullsized":"2024-04-02T04:49:47.408049+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":330,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.895998+0000","last_change":"2024-04-02T04:49:47.477571+0000","last_active":"2024-04-02T04:52:31.895998+0000","last_peered":"2024-04-02T04:52:31.895998+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.477571+0000","last_became_peered":"2024-04-02T04:49:47.477571+0000","last_unstale":"2024-04-02T04:52:31.895998+0000","last_undegraded":"2024-04-02T04:52:31.895998+0000","last_fullsized":"2024-04-02T04:49:47.404962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":358,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.888424+0000","last_change":"2024-04-02T04:49:47.517552+0000","last_active":"2024-04-02T04:52:31.888424+0000","last_peered":"2024-04-02T04:52:31.888424+0000","last_clean":"2024-04-02T04:49:45.378670+0000","last_became_active":"2024-04-02T04:49:47.517552+0000","last_became_peered":"2024-04-02T04:49:47.517552+0000","last_unstale":"2024-04-02T04:52:31.888424+0000","last_undegraded":"2024-04-02T04:52:31.888424+0000","last_fullsized":"2024-04-02T04:49:47.406289+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894528+0000","last_change":"2024-04-02T04:49:28.761881+0000","last_active":"2024-04-02T04:52:30.894528+0000","last_peered":"2024-04-02T04:52:30.894528+0000","last_clean":"2024-04-02T04:52:30.894528+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:52:30.894528+0000","last_undegraded":"2024-04-02T04:52:30.894528+0000","last_fullsized":"2024-04-02T04:52:30.894528+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":356,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.883231+0000","last_change":"2024-04-02T04:49:29.819002+0000","last_active":"2024-04-02T04:52:31.883231+0000","last_peered":"2024-04-02T04:52:31.883231+0000","last_clean":"2024-04-02T04:52:31.883231+0000","last_became_active":"2024-04-02T04:49:29.818524+0000","last_became_peered":"2024-04-02T04:49:29.818524+0000","last_unstale":"2024-04-02T04:52:31.883231+0000","last_undegraded":"2024-04-02T04:52:31.883231+0000","last_fullsized":"2024-04-02T04:52:31.883231+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.883131+0000","last_change":"2024-04-02T04:51:51.039566+0000","last_active":"2024-04-02T04:52:30.883131+0000","last_peered":"2024-04-02T04:52:30.883131+0000","last_clean":"2024-04-02T04:52:30.883131+0000","last_became_active":"2024-04-02T04:49:28.769072+0000","last_became_peered":"2024-04-02T04:49:28.769072+0000","last_unstale":"2024-04-02T04:52:30.883131+0000","last_undegraded":"2024-04-02T04:52:30.883131+0000","last_fullsized":"2024-04-02T04:52:30.883131+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.039450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:00.166954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":356,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886203+0000","last_change":"2024-04-02T04:49:29.802333+0000","last_active":"2024-04-02T04:52:31.886203+0000","last_peered":"2024-04-02T04:52:31.886203+0000","last_clean":"2024-04-02T04:52:31.886203+0000","last_became_active":"2024-04-02T04:49:29.801654+0000","last_became_peered":"2024-04-02T04:49:29.801654+0000","last_unstale":"2024-04-02T04:52:31.886203+0000","last_undegraded":"2024-04-02T04:52:31.886203+0000","last_fullsized":"2024-04-02T04:52:31.886203+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":356,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.618596+0000","last_change":"2024-04-02T04:49:28.760456+0000","last_active":"2024-04-02T04:52:32.618596+0000","last_peered":"2024-04-02T04:52:32.618596+0000","last_clean":"2024-04-02T04:52:32.618596+0000","last_became_active":"2024-04-02T04:49:28.756548+0000","last_became_peered":"2024-04-02T04:49:28.756548+0000","last_unstale":"2024-04-02T04:52:32.618596+0000","last_undegraded":"2024-04-02T04:52:32.618596+0000","last_fullsized":"2024-04-02T04:52:32.618596+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.620629+0000","last_change":"2024-04-02T04:51:40.625223+0000","last_active":"2024-04-02T04:52:32.620629+0000","last_peered":"2024-04-02T04:52:32.620629+0000","last_clean":"2024-04-02T04:52:32.620629+0000","last_became_active":"2024-04-02T04:49:28.741277+0000","last_became_peered":"2024-04-02T04:49:28.741277+0000","last_unstale":"2024-04-02T04:52:32.620629+0000","last_undegraded":"2024-04-02T04:52:32.620629+0000","last_fullsized":"2024-04-02T04:52:32.620629+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.625010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.625010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:53.527872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.444105+0000","last_change":"2024-04-02T04:49:28.740669+0000","last_active":"2024-04-02T04:52:32.444105+0000","last_peered":"2024-04-02T04:52:32.444105+0000","last_clean":"2024-04-02T04:52:32.444105+0000","last_became_active":"2024-04-02T04:49:28.719086+0000","last_became_peered":"2024-04-02T04:49:28.719086+0000","last_unstale":"2024-04-02T04:52:32.444105+0000","last_undegraded":"2024-04-02T04:52:32.444105+0000","last_fullsized":"2024-04-02T04:52:32.444105+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.449070+0000","last_change":"2024-04-02T04:49:28.743542+0000","last_active":"2024-04-02T04:52:32.449070+0000","last_peered":"2024-04-02T04:52:32.449070+0000","last_clean":"2024-04-02T04:52:32.449070+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:52:32.449070+0000","last_undegraded":"2024-04-02T04:52:32.449070+0000","last_fullsized":"2024-04-02T04:52:32.449070+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.442872+0000","last_change":"2024-04-02T04:49:28.732308+0000","last_active":"2024-04-02T04:52:32.442872+0000","last_peered":"2024-04-02T04:52:32.442872+0000","last_clean":"2024-04-02T04:52:32.442872+0000","last_became_active":"2024-04-02T04:49:28.731836+0000","last_became_peered":"2024-04-02T04:49:28.731836+0000","last_unstale":"2024-04-02T04:52:32.442872+0000","last_undegraded":"2024-04-02T04:52:32.442872+0000","last_fullsized":"2024-04-02T04:52:32.442872+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.885046+0000","last_change":"2024-04-02T04:50:39.108210+0000","last_active":"2024-04-02T04:52:30.885046+0000","last_peered":"2024-04-02T04:52:30.885046+0000","last_clean":"2024-04-02T04:52:30.885046+0000","last_became_active":"2024-04-02T04:49:28.727703+0000","last_became_peered":"2024-04-02T04:49:28.727703+0000","last_unstale":"2024-04-02T04:52:30.885046+0000","last_undegraded":"2024-04-02T04:52:30.885046+0000","last_fullsized":"2024-04-02T04:52:30.885046+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.108133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.108133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.443782+0000","last_change":"2024-04-02T04:49:47.517107+0000","last_active":"2024-04-02T04:52:32.443782+0000","last_peered":"2024-04-02T04:52:32.443782+0000","last_clean":"2024-04-02T04:49:45.381150+0000","last_became_active":"2024-04-02T04:49:47.517107+0000","last_became_peered":"2024-04-02T04:49:47.517107+0000","last_unstale":"2024-04-02T04:52:32.443782+0000","last_undegraded":"2024-04-02T04:52:32.443782+0000","last_fullsized":"2024-04-02T04:49:47.429362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618288+0000","last_change":"2024-04-02T04:49:47.444375+0000","last_active":"2024-04-02T04:52:32.618288+0000","last_peered":"2024-04-02T04:52:32.618288+0000","last_clean":"2024-04-02T04:49:45.331671+0000","last_became_active":"2024-04-02T04:49:47.444375+0000","last_became_peered":"2024-04-02T04:49:47.444375+0000","last_unstale":"2024-04-02T04:52:32.618288+0000","last_undegraded":"2024-04-02T04:52:32.618288+0000","last_fullsized":"2024-04-02T04:49:47.336937+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618352+0000","last_change":"2024-04-02T04:49:47.381163+0000","last_active":"2024-04-02T04:52:32.618352+0000","last_peered":"2024-04-02T04:52:32.618352+0000","last_clean":"2024-04-02T04:49:45.330129+0000","last_became_active":"2024-04-02T04:49:47.381163+0000","last_became_peered":"2024-04-02T04:49:47.381163+0000","last_unstale":"2024-04-02T04:52:32.618352+0000","last_undegraded":"2024-04-02T04:52:32.618352+0000","last_fullsized":"2024-04-02T04:49:47.338896+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.896011+0000","last_change":"2024-04-02T04:51:41.090918+0000","last_active":"2024-04-02T04:52:30.896011+0000","last_peered":"2024-04-02T04:52:30.896011+0000","last_clean":"2024-04-02T04:52:30.896011+0000","last_became_active":"2024-04-02T04:49:28.762640+0000","last_became_peered":"2024-04-02T04:49:28.762640+0000","last_unstale":"2024-04-02T04:52:30.896011+0000","last_undegraded":"2024-04-02T04:52:30.896011+0000","last_fullsized":"2024-04-02T04:52:30.896011+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:41.090714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:41.090714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:45.248800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.896556+0000","last_change":"2024-04-02T04:49:28.768223+0000","last_active":"2024-04-02T04:52:30.896556+0000","last_peered":"2024-04-02T04:52:30.896556+0000","last_clean":"2024-04-02T04:52:30.896556+0000","last_became_active":"2024-04-02T04:49:28.767868+0000","last_became_peered":"2024-04-02T04:49:28.767868+0000","last_unstale":"2024-04-02T04:52:30.896556+0000","last_undegraded":"2024-04-02T04:52:30.896556+0000","last_fullsized":"2024-04-02T04:52:30.896556+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":358,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.890688+0000","last_change":"2024-04-02T04:49:47.508050+0000","last_active":"2024-04-02T04:52:31.890688+0000","last_peered":"2024-04-02T04:52:31.890688+0000","last_clean":"2024-04-02T04:49:45.381990+0000","last_became_active":"2024-04-02T04:49:47.508050+0000","last_became_peered":"2024-04-02T04:49:47.508050+0000","last_unstale":"2024-04-02T04:52:31.890688+0000","last_undegraded":"2024-04-02T04:52:31.890688+0000","last_fullsized":"2024-04-02T04:49:47.429001+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.895672+0000","last_change":"2024-04-02T04:49:47.490495+0000","last_active":"2024-04-02T04:52:31.895672+0000","last_peered":"2024-04-02T04:52:31.895672+0000","last_clean":"2024-04-02T04:49:45.351290+0000","last_became_active":"2024-04-02T04:49:47.490495+0000","last_became_peered":"2024-04-02T04:49:47.490495+0000","last_unstale":"2024-04-02T04:52:31.895672+0000","last_undegraded":"2024-04-02T04:52:31.895672+0000","last_fullsized":"2024-04-02T04:49:47.384086+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":365,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.455457+0000","last_change":"2024-04-02T04:51:30.306477+0000","last_active":"2024-04-02T04:52:32.455457+0000","last_peered":"2024-04-02T04:52:32.455457+0000","last_clean":"2024-04-02T04:52:32.455457+0000","last_became_active":"2024-04-02T04:49:28.720246+0000","last_became_peered":"2024-04-02T04:49:28.720246+0000","last_unstale":"2024-04-02T04:52:32.455457+0000","last_undegraded":"2024-04-02T04:52:32.455457+0000","last_fullsized":"2024-04-02T04:52:32.455457+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.306343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.306343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:48.542387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.902667+0000","last_change":"2024-04-02T04:51:52.078040+0000","last_active":"2024-04-02T04:52:30.902667+0000","last_peered":"2024-04-02T04:52:30.902667+0000","last_clean":"2024-04-02T04:52:30.902667+0000","last_became_active":"2024-04-02T04:49:28.781235+0000","last_became_peered":"2024-04-02T04:49:28.781235+0000","last_unstale":"2024-04-02T04:52:30.902667+0000","last_undegraded":"2024-04-02T04:52:30.902667+0000","last_fullsized":"2024-04-02T04:52:30.902667+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.077908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.077908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:10.867557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.896137+0000","last_change":"2024-04-02T04:49:28.787959+0000","last_active":"2024-04-02T04:52:31.896137+0000","last_peered":"2024-04-02T04:52:31.896137+0000","last_clean":"2024-04-02T04:52:31.896137+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:52:31.896137+0000","last_undegraded":"2024-04-02T04:52:31.896137+0000","last_fullsized":"2024-04-02T04:52:31.896137+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.889660+0000","last_change":"2024-04-02T04:49:47.389948+0000","last_active":"2024-04-02T04:52:31.889660+0000","last_peered":"2024-04-02T04:52:31.889660+0000","last_clean":"2024-04-02T04:49:45.343249+0000","last_became_active":"2024-04-02T04:49:47.389948+0000","last_became_peered":"2024-04-02T04:49:47.389948+0000","last_unstale":"2024-04-02T04:52:31.889660+0000","last_undegraded":"2024-04-02T04:52:31.889660+0000","last_fullsized":"2024-04-02T04:49:47.366723+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888479+0000","last_change":"2024-04-02T04:49:28.750685+0000","last_active":"2024-04-02T04:52:31.888479+0000","last_peered":"2024-04-02T04:52:31.888479+0000","last_clean":"2024-04-02T04:52:31.888479+0000","last_became_active":"2024-04-02T04:49:28.750373+0000","last_became_peered":"2024-04-02T04:49:28.750373+0000","last_unstale":"2024-04-02T04:52:31.888479+0000","last_undegraded":"2024-04-02T04:52:31.888479+0000","last_fullsized":"2024-04-02T04:52:31.888479+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":329,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.622743+0000","last_change":"2024-04-02T04:49:47.389543+0000","last_active":"2024-04-02T04:52:32.622743+0000","last_peered":"2024-04-02T04:52:32.622743+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.389543+0000","last_became_peered":"2024-04-02T04:49:47.389543+0000","last_unstale":"2024-04-02T04:52:32.622743+0000","last_undegraded":"2024-04-02T04:52:32.622743+0000","last_fullsized":"2024-04-02T04:49:47.363220+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.622602+0000","last_change":"2024-04-02T04:49:47.391015+0000","last_active":"2024-04-02T04:52:32.622602+0000","last_peered":"2024-04-02T04:52:32.622602+0000","last_clean":"2024-04-02T04:49:45.363593+0000","last_became_active":"2024-04-02T04:49:47.391015+0000","last_became_peered":"2024-04-02T04:49:47.391015+0000","last_unstale":"2024-04-02T04:52:32.622602+0000","last_undegraded":"2024-04-02T04:52:32.622602+0000","last_fullsized":"2024-04-02T04:49:47.362030+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":329,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.619228+0000","last_change":"2024-04-02T04:49:47.434747+0000","last_active":"2024-04-02T04:52:32.619228+0000","last_peered":"2024-04-02T04:52:32.619228+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.434747+0000","last_became_peered":"2024-04-02T04:49:47.434747+0000","last_unstale":"2024-04-02T04:52:32.619228+0000","last_undegraded":"2024-04-02T04:52:32.619228+0000","last_fullsized":"2024-04-02T04:49:47.346344+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.888718+0000","last_change":"2024-04-02T04:51:43.198139+0000","last_active":"2024-04-02T04:52:30.888718+0000","last_peered":"2024-04-02T04:52:30.888718+0000","last_clean":"2024-04-02T04:52:30.888718+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:52:30.888718+0000","last_undegraded":"2024-04-02T04:52:30.888718+0000","last_fullsized":"2024-04-02T04:52:30.888718+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:43.198044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.198044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:50.401921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885836+0000","last_change":"2024-04-02T04:49:28.787621+0000","last_active":"2024-04-02T04:52:31.885836+0000","last_peered":"2024-04-02T04:52:31.885836+0000","last_clean":"2024-04-02T04:52:31.885836+0000","last_became_active":"2024-04-02T04:49:28.785195+0000","last_became_peered":"2024-04-02T04:49:28.785195+0000","last_unstale":"2024-04-02T04:52:31.885836+0000","last_undegraded":"2024-04-02T04:52:31.885836+0000","last_fullsized":"2024-04-02T04:52:31.885836+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.895064+0000","last_change":"2024-04-02T04:49:28.743937+0000","last_active":"2024-04-02T04:52:31.895064+0000","last_peered":"2024-04-02T04:52:31.895064+0000","last_clean":"2024-04-02T04:52:31.895064+0000","last_became_active":"2024-04-02T04:49:28.741450+0000","last_became_peered":"2024-04-02T04:49:28.741450+0000","last_unstale":"2024-04-02T04:52:31.895064+0000","last_undegraded":"2024-04-02T04:52:31.895064+0000","last_fullsized":"2024-04-02T04:52:31.895064+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886011+0000","last_change":"2024-04-02T04:49:28.731835+0000","last_active":"2024-04-02T04:52:31.886011+0000","last_peered":"2024-04-02T04:52:31.886011+0000","last_clean":"2024-04-02T04:52:31.886011+0000","last_became_active":"2024-04-02T04:49:28.731343+0000","last_became_peered":"2024-04-02T04:49:28.731343+0000","last_unstale":"2024-04-02T04:52:31.886011+0000","last_undegraded":"2024-04-02T04:52:31.886011+0000","last_fullsized":"2024-04-02T04:52:31.886011+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886503+0000","last_change":"2024-04-02T04:50:37.102849+0000","last_active":"2024-04-02T04:52:31.886503+0000","last_peered":"2024-04-02T04:52:31.886503+0000","last_clean":"2024-04-02T04:52:31.886503+0000","last_became_active":"2024-04-02T04:49:28.698110+0000","last_became_peered":"2024-04-02T04:49:28.698110+0000","last_unstale":"2024-04-02T04:52:31.886503+0000","last_undegraded":"2024-04-02T04:52:31.886503+0000","last_fullsized":"2024-04-02T04:52:31.886503+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.102732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.102732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":358,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.887409+0000","last_change":"2024-04-02T04:49:47.445509+0000","last_active":"2024-04-02T04:52:31.887409+0000","last_peered":"2024-04-02T04:52:31.887409+0000","last_clean":"2024-04-02T04:49:45.370130+0000","last_became_active":"2024-04-02T04:49:47.445509+0000","last_became_peered":"2024-04-02T04:49:47.445509+0000","last_unstale":"2024-04-02T04:52:31.887409+0000","last_undegraded":"2024-04-02T04:52:31.887409+0000","last_fullsized":"2024-04-02T04:49:47.391250+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885835+0000","last_change":"2024-04-02T04:49:28.748883+0000","last_active":"2024-04-02T04:52:31.885835+0000","last_peered":"2024-04-02T04:52:31.885835+0000","last_clean":"2024-04-02T04:52:31.885835+0000","last_became_active":"2024-04-02T04:49:28.746056+0000","last_became_peered":"2024-04-02T04:49:28.746056+0000","last_unstale":"2024-04-02T04:52:31.885835+0000","last_undegraded":"2024-04-02T04:52:31.885835+0000","last_fullsized":"2024-04-02T04:52:31.885835+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618642+0000","last_change":"2024-04-02T04:49:47.388801+0000","last_active":"2024-04-02T04:52:32.618642+0000","last_peered":"2024-04-02T04:52:32.618642+0000","last_clean":"2024-04-02T04:49:45.330442+0000","last_became_active":"2024-04-02T04:49:47.388801+0000","last_became_peered":"2024-04-02T04:49:47.388801+0000","last_unstale":"2024-04-02T04:52:32.618642+0000","last_undegraded":"2024-04-02T04:52:32.618642+0000","last_fullsized":"2024-04-02T04:49:47.339290+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618661+0000","last_change":"2024-04-02T04:49:47.444341+0000","last_active":"2024-04-02T04:52:32.618661+0000","last_peered":"2024-04-02T04:52:32.618661+0000","last_clean":"2024-04-02T04:49:45.331782+0000","last_became_active":"2024-04-02T04:49:47.444341+0000","last_became_peered":"2024-04-02T04:49:47.444341+0000","last_unstale":"2024-04-02T04:52:32.618661+0000","last_undegraded":"2024-04-02T04:52:32.618661+0000","last_fullsized":"2024-04-02T04:49:47.336855+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":356,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885897+0000","last_change":"2024-04-02T04:49:29.857781+0000","last_active":"2024-04-02T04:52:31.885897+0000","last_peered":"2024-04-02T04:52:31.885897+0000","last_clean":"2024-04-02T04:52:31.885897+0000","last_became_active":"2024-04-02T04:49:29.857200+0000","last_became_peered":"2024-04-02T04:49:29.857200+0000","last_unstale":"2024-04-02T04:52:31.885897+0000","last_undegraded":"2024-04-02T04:52:31.885897+0000","last_fullsized":"2024-04-02T04:52:31.885897+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.884211+0000","last_change":"2024-04-02T04:51:35.000630+0000","last_active":"2024-04-02T04:52:31.884211+0000","last_peered":"2024-04-02T04:52:31.884211+0000","last_clean":"2024-04-02T04:52:31.884211+0000","last_became_active":"2024-04-02T04:49:28.746855+0000","last_became_peered":"2024-04-02T04:49:28.746855+0000","last_unstale":"2024-04-02T04:52:31.884211+0000","last_undegraded":"2024-04-02T04:52:31.884211+0000","last_fullsized":"2024-04-02T04:52:31.884211+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:35.000533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.000533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:56.518037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":358,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.883505+0000","last_change":"2024-04-02T04:49:47.472640+0000","last_active":"2024-04-02T04:52:31.883505+0000","last_peered":"2024-04-02T04:52:31.883505+0000","last_clean":"2024-04-02T04:49:45.352140+0000","last_became_active":"2024-04-02T04:49:47.472640+0000","last_became_peered":"2024-04-02T04:49:47.472640+0000","last_unstale":"2024-04-02T04:52:31.883505+0000","last_undegraded":"2024-04-02T04:52:31.883505+0000","last_fullsized":"2024-04-02T04:49:47.377460+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891611+0000","last_change":"2024-04-02T04:51:36.554382+0000","last_active":"2024-04-02T04:52:31.891611+0000","last_peered":"2024-04-02T04:52:31.891611+0000","last_clean":"2024-04-02T04:52:31.891611+0000","last_became_active":"2024-04-02T04:49:29.874539+0000","last_became_peered":"2024-04-02T04:49:29.874539+0000","last_unstale":"2024-04-02T04:52:31.891611+0000","last_undegraded":"2024-04-02T04:52:31.891611+0000","last_fullsized":"2024-04-02T04:52:31.891611+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.554164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.554164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:46.989972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.898407+0000","last_change":"2024-04-02T04:49:47.452282+0000","last_active":"2024-04-02T04:52:31.898407+0000","last_peered":"2024-04-02T04:52:31.898407+0000","last_clean":"2024-04-02T04:49:45.347315+0000","last_became_active":"2024-04-02T04:49:47.452282+0000","last_became_peered":"2024-04-02T04:49:47.452282+0000","last_unstale":"2024-04-02T04:52:31.898407+0000","last_undegraded":"2024-04-02T04:52:31.898407+0000","last_fullsized":"2024-04-02T04:49:47.364280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.896000+0000","last_change":"2024-04-02T04:51:53.758572+0000","last_active":"2024-04-02T04:52:31.896000+0000","last_peered":"2024-04-02T04:52:31.896000+0000","last_clean":"2024-04-02T04:52:31.896000+0000","last_became_active":"2024-04-02T04:49:29.830080+0000","last_became_peered":"2024-04-02T04:49:29.830080+0000","last_unstale":"2024-04-02T04:52:31.896000+0000","last_undegraded":"2024-04-02T04:52:31.896000+0000","last_fullsized":"2024-04-02T04:52:31.896000+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.758420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.758420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:20.112324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.451670+0000","last_change":"2024-04-02T04:49:47.477237+0000","last_active":"2024-04-02T04:52:32.451670+0000","last_peered":"2024-04-02T04:52:32.451670+0000","last_clean":"2024-04-02T04:49:45.366065+0000","last_became_active":"2024-04-02T04:49:47.477237+0000","last_became_peered":"2024-04-02T04:49:47.477237+0000","last_unstale":"2024-04-02T04:52:32.451670+0000","last_undegraded":"2024-04-02T04:52:32.451670+0000","last_fullsized":"2024-04-02T04:49:47.430531+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894809+0000","last_change":"2024-04-02T04:51:36.096984+0000","last_active":"2024-04-02T04:52:30.894809+0000","last_peered":"2024-04-02T04:52:30.894809+0000","last_clean":"2024-04-02T04:52:30.894809+0000","last_became_active":"2024-04-02T04:49:28.780779+0000","last_became_peered":"2024-04-02T04:49:28.780779+0000","last_unstale":"2024-04-02T04:52:30.894809+0000","last_undegraded":"2024-04-02T04:52:30.894809+0000","last_fullsized":"2024-04-02T04:52:30.894809+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.096832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.096832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:39.960736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.898957+0000","last_change":"2024-04-02T04:49:28.788472+0000","last_active":"2024-04-02T04:52:31.898957+0000","last_peered":"2024-04-02T04:52:31.898957+0000","last_clean":"2024-04-02T04:52:31.898957+0000","last_became_active":"2024-04-02T04:49:28.784897+0000","last_became_peered":"2024-04-02T04:49:28.784897+0000","last_unstale":"2024-04-02T04:52:31.898957+0000","last_undegraded":"2024-04-02T04:52:31.898957+0000","last_fullsized":"2024-04-02T04:52:31.898957+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.622288+0000","last_change":"2024-04-02T04:50:44.471617+0000","last_active":"2024-04-02T04:52:32.622288+0000","last_peered":"2024-04-02T04:52:32.622288+0000","last_clean":"2024-04-02T04:52:32.622288+0000","last_became_active":"2024-04-02T04:49:28.744153+0000","last_became_peered":"2024-04-02T04:49:28.744153+0000","last_unstale":"2024-04-02T04:52:32.622288+0000","last_undegraded":"2024-04-02T04:52:32.622288+0000","last_fullsized":"2024-04-02T04:52:32.622288+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.471508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.471508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894080+0000","last_change":"2024-04-02T04:49:28.761364+0000","last_active":"2024-04-02T04:52:30.894080+0000","last_peered":"2024-04-02T04:52:30.894080+0000","last_clean":"2024-04-02T04:52:30.894080+0000","last_became_active":"2024-04-02T04:49:28.760661+0000","last_became_peered":"2024-04-02T04:49:28.760661+0000","last_unstale":"2024-04-02T04:52:30.894080+0000","last_undegraded":"2024-04-02T04:52:30.894080+0000","last_fullsized":"2024-04-02T04:52:30.894080+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.904889+0000","last_change":"2024-04-02T04:49:28.771528+0000","last_active":"2024-04-02T04:52:30.904889+0000","last_peered":"2024-04-02T04:52:30.904889+0000","last_clean":"2024-04-02T04:52:30.904889+0000","last_became_active":"2024-04-02T04:49:28.769787+0000","last_became_peered":"2024-04-02T04:49:28.769787+0000","last_unstale":"2024-04-02T04:52:30.904889+0000","last_undegraded":"2024-04-02T04:52:30.904889+0000","last_fullsized":"2024-04-02T04:52:30.904889+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.898547+0000","last_change":"2024-04-02T04:51:44.205952+0000","last_active":"2024-04-02T04:52:30.898547+0000","last_peered":"2024-04-02T04:52:30.898547+0000","last_clean":"2024-04-02T04:52:30.898547+0000","last_became_active":"2024-04-02T04:49:28.728416+0000","last_became_peered":"2024-04-02T04:49:28.728416+0000","last_unstale":"2024-04-02T04:52:30.898547+0000","last_undegraded":"2024-04-02T04:52:30.898547+0000","last_fullsized":"2024-04-02T04:52:30.898547+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.205739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.205739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:08.804825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.898492+0000","last_change":"2024-04-02T04:51:42.223186+0000","last_active":"2024-04-02T04:52:30.898492+0000","last_peered":"2024-04-02T04:52:30.898492+0000","last_clean":"2024-04-02T04:52:30.898492+0000","last_became_active":"2024-04-02T04:49:28.727661+0000","last_became_peered":"2024-04-02T04:49:28.727661+0000","last_unstale":"2024-04-02T04:52:30.898492+0000","last_undegraded":"2024-04-02T04:52:30.898492+0000","last_fullsized":"2024-04-02T04:52:30.898492+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:42.223094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:42.223094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:44.031995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895749+0000","last_change":"2024-04-02T04:49:28.762112+0000","last_active":"2024-04-02T04:52:30.895749+0000","last_peered":"2024-04-02T04:52:30.895749+0000","last_clean":"2024-04-02T04:52:30.895749+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:52:30.895749+0000","last_undegraded":"2024-04-02T04:52:30.895749+0000","last_fullsized":"2024-04-02T04:52:30.895749+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.882920+0000","last_change":"2024-04-02T04:49:28.745274+0000","last_active":"2024-04-02T04:52:30.882920+0000","last_peered":"2024-04-02T04:52:30.882920+0000","last_clean":"2024-04-02T04:52:30.882920+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:52:30.882920+0000","last_undegraded":"2024-04-02T04:52:30.882920+0000","last_fullsized":"2024-04-02T04:52:30.882920+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":327,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.883557+0000","last_change":"2024-04-02T04:49:47.512403+0000","last_active":"2024-04-02T04:52:30.883557+0000","last_peered":"2024-04-02T04:52:30.883557+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.512403+0000","last_became_peered":"2024-04-02T04:49:47.512403+0000","last_unstale":"2024-04-02T04:52:30.883557+0000","last_undegraded":"2024-04-02T04:52:30.883557+0000","last_fullsized":"2024-04-02T04:49:47.379434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886623+0000","last_change":"2024-04-02T04:51:48.958576+0000","last_active":"2024-04-02T04:52:31.886623+0000","last_peered":"2024-04-02T04:52:31.886623+0000","last_clean":"2024-04-02T04:52:31.886623+0000","last_became_active":"2024-04-02T04:49:28.725896+0000","last_became_peered":"2024-04-02T04:49:28.725896+0000","last_unstale":"2024-04-02T04:52:31.886623+0000","last_undegraded":"2024-04-02T04:52:31.886623+0000","last_fullsized":"2024-04-02T04:52:31.886623+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.958371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:14.943164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.618216+0000","last_change":"2024-04-02T04:51:26.661470+0000","last_active":"2024-04-02T04:52:32.618216+0000","last_peered":"2024-04-02T04:52:32.618216+0000","last_clean":"2024-04-02T04:52:32.618216+0000","last_became_active":"2024-04-02T04:49:28.724387+0000","last_became_peered":"2024-04-02T04:49:28.724387+0000","last_unstale":"2024-04-02T04:52:32.618216+0000","last_undegraded":"2024-04-02T04:52:32.618216+0000","last_fullsized":"2024-04-02T04:52:32.618216+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.661376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.661376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:54.466284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886299+0000","last_change":"2024-04-02T04:51:27.695556+0000","last_active":"2024-04-02T04:52:31.886299+0000","last_peered":"2024-04-02T04:52:31.886299+0000","last_clean":"2024-04-02T04:52:31.886299+0000","last_became_active":"2024-04-02T04:49:29.856708+0000","last_became_peered":"2024-04-02T04:49:29.856708+0000","last_unstale":"2024-04-02T04:52:31.886299+0000","last_undegraded":"2024-04-02T04:52:31.886299+0000","last_fullsized":"2024-04-02T04:52:31.886299+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:27.695437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.695437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885373+0000","last_change":"2024-04-02T04:51:29.649636+0000","last_active":"2024-04-02T04:52:31.885373+0000","last_peered":"2024-04-02T04:52:31.885373+0000","last_clean":"2024-04-02T04:52:31.885373+0000","last_became_active":"2024-04-02T04:49:29.816194+0000","last_became_peered":"2024-04-02T04:49:29.816194+0000","last_unstale":"2024-04-02T04:52:31.885373+0000","last_undegraded":"2024-04-02T04:52:31.885373+0000","last_fullsized":"2024-04-02T04:52:31.885373+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.649422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.649422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:52.792304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":357,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.883960+0000","last_change":"2024-04-02T04:49:47.427831+0000","last_active":"2024-04-02T04:52:30.883960+0000","last_peered":"2024-04-02T04:52:30.883960+0000","last_clean":"2024-04-02T04:49:45.321587+0000","last_became_active":"2024-04-02T04:49:47.427831+0000","last_became_peered":"2024-04-02T04:49:47.427831+0000","last_unstale":"2024-04-02T04:52:30.883960+0000","last_undegraded":"2024-04-02T04:52:30.883960+0000","last_fullsized":"2024-04-02T04:49:47.369909+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.887008+0000","last_change":"2024-04-02T04:49:47.387543+0000","last_active":"2024-04-02T04:52:31.887008+0000","last_peered":"2024-04-02T04:52:31.887008+0000","last_clean":"2024-04-02T04:49:45.306095+0000","last_became_active":"2024-04-02T04:49:47.387543+0000","last_became_peered":"2024-04-02T04:49:47.387543+0000","last_unstale":"2024-04-02T04:52:31.887008+0000","last_undegraded":"2024-04-02T04:52:31.887008+0000","last_fullsized":"2024-04-02T04:49:47.345085+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.444136+0000","last_change":"2024-04-02T04:49:28.739636+0000","last_active":"2024-04-02T04:52:32.444136+0000","last_peered":"2024-04-02T04:52:32.444136+0000","last_clean":"2024-04-02T04:52:32.444136+0000","last_became_active":"2024-04-02T04:49:28.734672+0000","last_became_peered":"2024-04-02T04:49:28.734672+0000","last_unstale":"2024-04-02T04:52:32.444136+0000","last_undegraded":"2024-04-02T04:52:32.444136+0000","last_fullsized":"2024-04-02T04:52:32.444136+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.889335+0000","last_change":"2024-04-02T04:51:26.162872+0000","last_active":"2024-04-02T04:52:30.889335+0000","last_peered":"2024-04-02T04:52:30.889335+0000","last_clean":"2024-04-02T04:52:30.889335+0000","last_became_active":"2024-04-02T04:49:28.749072+0000","last_became_peered":"2024-04-02T04:49:28.749072+0000","last_unstale":"2024-04-02T04:52:30.889335+0000","last_undegraded":"2024-04-02T04:52:30.889335+0000","last_fullsized":"2024-04-02T04:52:30.889335+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.162792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.162792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.458531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888535+0000","last_change":"2024-04-02T04:51:51.741608+0000","last_active":"2024-04-02T04:52:31.888535+0000","last_peered":"2024-04-02T04:52:31.888535+0000","last_clean":"2024-04-02T04:52:31.888535+0000","last_became_active":"2024-04-02T04:49:29.858586+0000","last_became_peered":"2024-04-02T04:49:29.858586+0000","last_unstale":"2024-04-02T04:52:31.888535+0000","last_undegraded":"2024-04-02T04:52:31.888535+0000","last_fullsized":"2024-04-02T04:52:31.888535+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.741382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.741382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:52.319699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.440944+0000","last_change":"2024-04-02T04:49:47.469970+0000","last_active":"2024-04-02T04:52:32.440944+0000","last_peered":"2024-04-02T04:52:32.440944+0000","last_clean":"2024-04-02T04:49:45.328796+0000","last_became_active":"2024-04-02T04:49:47.469970+0000","last_became_peered":"2024-04-02T04:49:47.469970+0000","last_unstale":"2024-04-02T04:52:32.440944+0000","last_undegraded":"2024-04-02T04:52:32.440944+0000","last_fullsized":"2024-04-02T04:49:47.382953+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":327,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.887856+0000","last_change":"2024-04-02T04:49:47.457335+0000","last_active":"2024-04-02T04:52:30.887856+0000","last_peered":"2024-04-02T04:52:30.887856+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.457335+0000","last_became_peered":"2024-04-02T04:49:47.457335+0000","last_unstale":"2024-04-02T04:52:30.887856+0000","last_undegraded":"2024-04-02T04:52:30.887856+0000","last_fullsized":"2024-04-02T04:49:47.405947+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.622387+0000","last_change":"2024-04-02T04:51:55.540774+0000","last_active":"2024-04-02T04:52:32.622387+0000","last_peered":"2024-04-02T04:52:32.622387+0000","last_clean":"2024-04-02T04:52:32.622387+0000","last_became_active":"2024-04-02T04:49:28.750713+0000","last_became_peered":"2024-04-02T04:49:28.750713+0000","last_unstale":"2024-04-02T04:52:32.622387+0000","last_undegraded":"2024-04-02T04:52:32.622387+0000","last_fullsized":"2024-04-02T04:52:32.622387+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.540654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.540654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:16.003120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895005+0000","last_change":"2024-04-02T04:49:28.768154+0000","last_active":"2024-04-02T04:52:30.895005+0000","last_peered":"2024-04-02T04:52:30.895005+0000","last_clean":"2024-04-02T04:52:30.895005+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:52:30.895005+0000","last_undegraded":"2024-04-02T04:52:30.895005+0000","last_fullsized":"2024-04-02T04:52:30.895005+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":330,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.449019+0000","last_change":"2024-04-02T04:49:47.500400+0000","last_active":"2024-04-02T04:52:32.449019+0000","last_peered":"2024-04-02T04:52:32.449019+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.500400+0000","last_became_peered":"2024-04-02T04:49:47.500400+0000","last_unstale":"2024-04-02T04:52:32.449019+0000","last_undegraded":"2024-04-02T04:52:32.449019+0000","last_fullsized":"2024-04-02T04:49:47.430108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.898669+0000","last_change":"2024-04-02T04:50:29.991272+0000","last_active":"2024-04-02T04:52:30.898669+0000","last_peered":"2024-04-02T04:52:30.898669+0000","last_clean":"2024-04-02T04:52:30.898669+0000","last_became_active":"2024-04-02T04:49:28.716933+0000","last_became_peered":"2024-04-02T04:49:28.716933+0000","last_unstale":"2024-04-02T04:52:30.898669+0000","last_undegraded":"2024-04-02T04:52:30.898669+0000","last_fullsized":"2024-04-02T04:52:30.898669+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.991167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":365,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.898864+0000","last_change":"2024-04-02T04:51:52.390967+0000","last_active":"2024-04-02T04:52:31.898864+0000","last_peered":"2024-04-02T04:52:31.898864+0000","last_clean":"2024-04-02T04:52:31.898864+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:52:31.898864+0000","last_undegraded":"2024-04-02T04:52:31.898864+0000","last_fullsized":"2024-04-02T04:52:31.898864+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.390694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.390694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:58.042023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.896296+0000","last_change":"2024-04-02T04:49:28.757581+0000","last_active":"2024-04-02T04:52:30.896296+0000","last_peered":"2024-04-02T04:52:30.896296+0000","last_clean":"2024-04-02T04:52:30.896296+0000","last_became_active":"2024-04-02T04:49:28.757217+0000","last_became_peered":"2024-04-02T04:49:28.757217+0000","last_unstale":"2024-04-02T04:52:30.896296+0000","last_undegraded":"2024-04-02T04:52:30.896296+0000","last_fullsized":"2024-04-02T04:52:30.896296+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.444216+0000","last_change":"2024-04-02T04:49:47.508156+0000","last_active":"2024-04-02T04:52:32.444216+0000","last_peered":"2024-04-02T04:52:32.444216+0000","last_clean":"2024-04-02T04:49:45.359498+0000","last_became_active":"2024-04-02T04:49:47.508156+0000","last_became_peered":"2024-04-02T04:49:47.508156+0000","last_unstale":"2024-04-02T04:52:32.444216+0000","last_undegraded":"2024-04-02T04:52:32.444216+0000","last_fullsized":"2024-04-02T04:49:47.430692+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888512+0000","last_change":"2024-04-02T04:51:25.125020+0000","last_active":"2024-04-02T04:52:31.888512+0000","last_peered":"2024-04-02T04:52:31.888512+0000","last_clean":"2024-04-02T04:52:31.888512+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:52:31.888512+0000","last_undegraded":"2024-04-02T04:52:31.888512+0000","last_fullsized":"2024-04-02T04:52:31.888512+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:25.124929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.124929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":360,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.455890+0000","last_change":"2024-04-02T04:49:47.501136+0000","last_active":"2024-04-02T04:52:32.455890+0000","last_peered":"2024-04-02T04:52:32.455890+0000","last_clean":"2024-04-02T04:49:45.381195+0000","last_became_active":"2024-04-02T04:49:47.501136+0000","last_became_peered":"2024-04-02T04:49:47.501136+0000","last_unstale":"2024-04-02T04:52:32.455890+0000","last_undegraded":"2024-04-02T04:52:32.455890+0000","last_fullsized":"2024-04-02T04:49:47.416717+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":358,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.888649+0000","last_change":"2024-04-02T04:49:47.510048+0000","last_active":"2024-04-02T04:52:31.888649+0000","last_peered":"2024-04-02T04:52:31.888649+0000","last_clean":"2024-04-02T04:49:45.381640+0000","last_became_active":"2024-04-02T04:49:47.510048+0000","last_became_peered":"2024-04-02T04:49:47.510048+0000","last_unstale":"2024-04-02T04:52:31.888649+0000","last_undegraded":"2024-04-02T04:52:31.888649+0000","last_fullsized":"2024-04-02T04:49:47.408507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.621148+0000","last_change":"2024-04-02T04:51:29.673932+0000","last_active":"2024-04-02T04:52:32.621148+0000","last_peered":"2024-04-02T04:52:32.621148+0000","last_clean":"2024-04-02T04:52:32.621148+0000","last_became_active":"2024-04-02T04:49:28.765229+0000","last_became_peered":"2024-04-02T04:49:28.765229+0000","last_unstale":"2024-04-02T04:52:32.621148+0000","last_undegraded":"2024-04-02T04:52:32.621148+0000","last_fullsized":"2024-04-02T04:52:32.621148+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.673796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.673796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:43.374162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":327,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.881905+0000","last_change":"2024-04-02T04:49:47.441206+0000","last_active":"2024-04-02T04:52:30.881905+0000","last_peered":"2024-04-02T04:52:30.881905+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.441206+0000","last_became_peered":"2024-04-02T04:49:47.441206+0000","last_unstale":"2024-04-02T04:52:30.881905+0000","last_undegraded":"2024-04-02T04:52:30.881905+0000","last_fullsized":"2024-04-02T04:49:47.375248+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886120+0000","last_change":"2024-04-02T04:49:28.759884+0000","last_active":"2024-04-02T04:52:31.886120+0000","last_peered":"2024-04-02T04:52:31.886120+0000","last_clean":"2024-04-02T04:52:31.886120+0000","last_became_active":"2024-04-02T04:49:28.749451+0000","last_became_peered":"2024-04-02T04:49:28.749451+0000","last_unstale":"2024-04-02T04:52:31.886120+0000","last_undegraded":"2024-04-02T04:52:31.886120+0000","last_fullsized":"2024-04-02T04:52:31.886120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":357,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.894573+0000","last_change":"2024-04-02T04:49:47.479424+0000","last_active":"2024-04-02T04:52:30.894573+0000","last_peered":"2024-04-02T04:52:30.894573+0000","last_clean":"2024-04-02T04:49:45.342017+0000","last_became_active":"2024-04-02T04:49:47.479424+0000","last_became_peered":"2024-04-02T04:49:47.479424+0000","last_unstale":"2024-04-02T04:52:30.894573+0000","last_undegraded":"2024-04-02T04:52:30.894573+0000","last_fullsized":"2024-04-02T04:49:47.413098+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.618150+0000","last_change":"2024-04-02T04:51:36.711304+0000","last_active":"2024-04-02T04:52:32.618150+0000","last_peered":"2024-04-02T04:52:32.618150+0000","last_clean":"2024-04-02T04:52:32.618150+0000","last_became_active":"2024-04-02T04:49:28.743589+0000","last_became_peered":"2024-04-02T04:49:28.743589+0000","last_unstale":"2024-04-02T04:52:32.618150+0000","last_undegraded":"2024-04-02T04:52:32.618150+0000","last_fullsized":"2024-04-02T04:52:32.618150+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.711178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.711178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:03.649209+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":359,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.885998+0000","last_change":"2024-04-02T04:49:47.502106+0000","last_active":"2024-04-02T04:52:31.885998+0000","last_peered":"2024-04-02T04:52:31.885998+0000","last_clean":"2024-04-02T04:49:45.334274+0000","last_became_active":"2024-04-02T04:49:47.502106+0000","last_became_peered":"2024-04-02T04:49:47.502106+0000","last_unstale":"2024-04-02T04:52:31.885998+0000","last_undegraded":"2024-04-02T04:52:31.885998+0000","last_fullsized":"2024-04-02T04:49:47.399330+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":356,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885276+0000","last_change":"2024-04-02T04:49:29.805620+0000","last_active":"2024-04-02T04:52:31.885276+0000","last_peered":"2024-04-02T04:52:31.885276+0000","last_clean":"2024-04-02T04:52:31.885276+0000","last_became_active":"2024-04-02T04:49:29.805341+0000","last_became_peered":"2024-04-02T04:49:29.805341+0000","last_unstale":"2024-04-02T04:52:31.885276+0000","last_undegraded":"2024-04-02T04:52:31.885276+0000","last_fullsized":"2024-04-02T04:52:31.885276+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":328,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.886638+0000","last_change":"2024-04-02T04:49:47.482774+0000","last_active":"2024-04-02T04:52:31.886638+0000","last_peered":"2024-04-02T04:52:31.886638+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.482774+0000","last_became_peered":"2024-04-02T04:49:47.482774+0000","last_unstale":"2024-04-02T04:52:31.886638+0000","last_undegraded":"2024-04-02T04:52:31.886638+0000","last_fullsized":"2024-04-02T04:49:47.403371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.619632+0000","last_change":"2024-04-02T04:51:45.521690+0000","last_active":"2024-04-02T04:52:32.619632+0000","last_peered":"2024-04-02T04:52:32.619632+0000","last_clean":"2024-04-02T04:52:32.619632+0000","last_became_active":"2024-04-02T04:49:28.766979+0000","last_became_peered":"2024-04-02T04:49:28.766979+0000","last_unstale":"2024-04-02T04:52:32.619632+0000","last_undegraded":"2024-04-02T04:52:32.619632+0000","last_fullsized":"2024-04-02T04:52:32.619632+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:45.521589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:45.521589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:53.635294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":357,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886294+0000","last_change":"2024-04-02T04:49:28.756847+0000","last_active":"2024-04-02T04:52:31.886294+0000","last_peered":"2024-04-02T04:52:31.886294+0000","last_clean":"2024-04-02T04:52:31.886294+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:52:31.886294+0000","last_undegraded":"2024-04-02T04:52:31.886294+0000","last_fullsized":"2024-04-02T04:52:31.886294+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.885851+0000","last_change":"2024-04-02T04:49:28.753851+0000","last_active":"2024-04-02T04:52:30.885851+0000","last_peered":"2024-04-02T04:52:30.885851+0000","last_clean":"2024-04-02T04:52:30.885851+0000","last_became_active":"2024-04-02T04:49:28.753376+0000","last_became_peered":"2024-04-02T04:49:28.753376+0000","last_unstale":"2024-04-02T04:52:30.885851+0000","last_undegraded":"2024-04-02T04:52:30.885851+0000","last_fullsized":"2024-04-02T04:52:30.885851+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885850+0000","last_change":"2024-04-02T04:50:44.257998+0000","last_active":"2024-04-02T04:52:31.885850+0000","last_peered":"2024-04-02T04:52:31.885850+0000","last_clean":"2024-04-02T04:52:31.885850+0000","last_became_active":"2024-04-02T04:49:28.757583+0000","last_became_peered":"2024-04-02T04:49:28.757583+0000","last_unstale":"2024-04-02T04:52:31.885850+0000","last_undegraded":"2024-04-02T04:52:31.885850+0000","last_fullsized":"2024-04-02T04:52:31.885850+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.257747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.257747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":363,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.884187+0000","last_change":"2024-04-02T04:51:49.984059+0000","last_active":"2024-04-02T04:52:31.884187+0000","last_peered":"2024-04-02T04:52:31.884187+0000","last_clean":"2024-04-02T04:52:31.884187+0000","last_became_active":"2024-04-02T04:49:28.726425+0000","last_became_peered":"2024-04-02T04:49:28.726425+0000","last_unstale":"2024-04-02T04:52:31.884187+0000","last_undegraded":"2024-04-02T04:52:31.884187+0000","last_fullsized":"2024-04-02T04:52:31.884187+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.983966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.983966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:00.906764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"139'5","reported_seq":372,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.883737+0000","last_change":"2024-04-02T04:50:26.846127+0000","last_active":"2024-04-02T04:52:30.883737+0000","last_peered":"2024-04-02T04:52:30.883737+0000","last_clean":"2024-04-02T04:52:30.883737+0000","last_became_active":"2024-04-02T04:49:26.170913+0000","last_became_peered":"2024-04-02T04:49:26.170913+0000","last_unstale":"2024-04-02T04:52:30.883737+0000","last_undegraded":"2024-04-02T04:52:30.883737+0000","last_fullsized":"2024-04-02T04:52:30.883737+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:50:26.845900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.845900+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.230189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":369,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.622519+0000","last_change":"2024-04-02T04:51:30.643523+0000","last_active":"2024-04-02T04:52:32.622519+0000","last_peered":"2024-04-02T04:52:32.622519+0000","last_clean":"2024-04-02T04:52:32.622519+0000","last_became_active":"2024-04-02T04:49:26.216834+0000","last_became_peered":"2024-04-02T04:49:26.216834+0000","last_unstale":"2024-04-02T04:52:32.622519+0000","last_undegraded":"2024-04-02T04:52:32.622519+0000","last_fullsized":"2024-04-02T04:52:32.622519+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.643306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.643306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:42.736061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":506,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618671+0000","last_change":"2024-04-02T04:49:47.366762+0000","last_active":"2024-04-02T04:52:32.618671+0000","last_peered":"2024-04-02T04:52:32.618671+0000","last_clean":"2024-04-02T04:49:45.330332+0000","last_became_active":"2024-04-02T04:49:47.366762+0000","last_became_peered":"2024-04-02T04:49:47.366762+0000","last_unstale":"2024-04-02T04:52:32.618671+0000","last_undegraded":"2024-04-02T04:52:32.618671+0000","last_fullsized":"2024-04-02T04:49:47.338850+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.797658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.797658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":372,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.899127+0000","last_change":"2024-04-02T04:49:47.409183+0000","last_active":"2024-04-02T04:52:31.899127+0000","last_peered":"2024-04-02T04:52:31.899127+0000","last_clean":"2024-04-02T04:44:01.886062+0000","last_became_active":"2024-04-02T04:49:47.409183+0000","last_became_peered":"2024-04-02T04:49:47.409183+0000","last_unstale":"2024-04-02T04:52:31.899127+0000","last_undegraded":"2024-04-02T04:49:47.369263+0000","last_fullsized":"2024-04-02T04:49:47.369128+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":504,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.895180+0000","last_change":"2024-04-02T04:49:47.494397+0000","last_active":"2024-04-02T04:52:30.895180+0000","last_peered":"2024-04-02T04:52:30.895180+0000","last_clean":"2024-04-02T04:49:45.361282+0000","last_became_active":"2024-04-02T04:49:47.494397+0000","last_became_peered":"2024-04-02T04:49:47.494397+0000","last_unstale":"2024-04-02T04:52:30.895180+0000","last_undegraded":"2024-04-02T04:52:30.895180+0000","last_fullsized":"2024-04-02T04:49:47.425453+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.658688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.658688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":499,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.617137+0000","last_change":"2024-04-02T04:50:11.467388+0000","last_active":"2024-04-02T04:52:32.617137+0000","last_peered":"2024-04-02T04:52:32.617137+0000","last_clean":"2024-04-02T04:52:32.617137+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:52:32.617137+0000","last_undegraded":"2024-04-02T04:52:32.617137+0000","last_fullsized":"2024-04-02T04:52:32.617137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.467182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.467182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":369,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.618845+0000","last_change":"2024-04-02T04:51:38.662185+0000","last_active":"2024-04-02T04:52:32.618845+0000","last_peered":"2024-04-02T04:52:32.618845+0000","last_clean":"2024-04-02T04:52:32.618845+0000","last_became_active":"2024-04-02T04:49:26.241757+0000","last_became_peered":"2024-04-02T04:49:26.241757+0000","last_unstale":"2024-04-02T04:52:32.618845+0000","last_undegraded":"2024-04-02T04:52:32.618845+0000","last_fullsized":"2024-04-02T04:52:32.618845+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:38.662076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.662076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:58.177149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895204+0000","last_change":"2024-04-02T04:51:31.155149+0000","last_active":"2024-04-02T04:52:30.895204+0000","last_peered":"2024-04-02T04:52:30.895204+0000","last_clean":"2024-04-02T04:52:30.895204+0000","last_became_active":"2024-04-02T04:49:26.296797+0000","last_became_peered":"2024-04-02T04:49:26.296797+0000","last_unstale":"2024-04-02T04:52:30.895204+0000","last_undegraded":"2024-04-02T04:52:30.895204+0000","last_fullsized":"2024-04-02T04:52:30.895204+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:31.154912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:31.154912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:37.185092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":512,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887886+0000","last_change":"2024-04-02T04:50:35.049746+0000","last_active":"2024-04-02T04:52:31.887886+0000","last_peered":"2024-04-02T04:52:31.887886+0000","last_clean":"2024-04-02T04:52:31.887886+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:52:31.887886+0000","last_undegraded":"2024-04-02T04:52:31.887886+0000","last_fullsized":"2024-04-02T04:52:31.887886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:35.049654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:50:35.049654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":535,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.620784+0000","last_change":"2024-04-02T04:50:26.509397+0000","last_active":"2024-04-02T04:52:32.620784+0000","last_peered":"2024-04-02T04:52:32.620784+0000","last_clean":"2024-04-02T04:52:32.620784+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:52:32.620784+0000","last_undegraded":"2024-04-02T04:52:32.620784+0000","last_fullsized":"2024-04-02T04:52:32.620784+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:50:26.509291+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.509291+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":521,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.455340+0000","last_change":"2024-04-02T04:50:34.277643+0000","last_active":"2024-04-02T04:52:32.455340+0000","last_peered":"2024-04-02T04:52:32.455340+0000","last_clean":"2024-04-02T04:52:32.455340+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:52:32.455340+0000","last_undegraded":"2024-04-02T04:52:32.455340+0000","last_fullsized":"2024-04-02T04:52:32.455340+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:34.277549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:16.442234+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.277549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":487,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.885642+0000","last_change":"2024-04-02T04:49:47.516977+0000","last_active":"2024-04-02T04:52:31.885642+0000","last_peered":"2024-04-02T04:52:31.885642+0000","last_clean":"2024-04-02T04:49:45.355630+0000","last_became_active":"2024-04-02T04:49:47.516977+0000","last_became_peered":"2024-04-02T04:49:47.516977+0000","last_unstale":"2024-04-02T04:52:31.885642+0000","last_undegraded":"2024-04-02T04:52:31.885642+0000","last_fullsized":"2024-04-02T04:49:47.384135+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"139'5","reported_seq":375,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886739+0000","last_change":"2024-04-02T04:51:27.143046+0000","last_active":"2024-04-02T04:52:31.886739+0000","last_peered":"2024-04-02T04:52:31.886739+0000","last_clean":"2024-04-02T04:52:31.886739+0000","last_became_active":"2024-04-02T04:49:25.139936+0000","last_became_peered":"2024-04-02T04:49:25.139936+0000","last_unstale":"2024-04-02T04:52:31.886739+0000","last_undegraded":"2024-04-02T04:52:31.886739+0000","last_fullsized":"2024-04-02T04:52:31.886739+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:27.142908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.142908+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.491255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":369,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887075+0000","last_change":"2024-04-02T04:50:39.186924+0000","last_active":"2024-04-02T04:52:31.887075+0000","last_peered":"2024-04-02T04:52:31.887075+0000","last_clean":"2024-04-02T04:52:31.887075+0000","last_became_active":"2024-04-02T04:49:25.143060+0000","last_became_peered":"2024-04-02T04:49:25.143060+0000","last_unstale":"2024-04-02T04:52:31.887075+0000","last_undegraded":"2024-04-02T04:52:31.887075+0000","last_fullsized":"2024-04-02T04:52:31.887075+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.186729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.186729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":506,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.884136+0000","last_change":"2024-04-02T04:49:47.472350+0000","last_active":"2024-04-02T04:52:30.884136+0000","last_peered":"2024-04-02T04:52:30.884136+0000","last_clean":"2024-04-02T04:49:45.385689+0000","last_became_active":"2024-04-02T04:49:47.472350+0000","last_became_peered":"2024-04-02T04:49:47.472350+0000","last_unstale":"2024-04-02T04:52:30.884136+0000","last_undegraded":"2024-04-02T04:52:30.884136+0000","last_fullsized":"2024-04-02T04:49:47.393316+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.719146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.719146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":374,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.621536+0000","last_change":"2024-04-02T04:49:47.381178+0000","last_active":"2024-04-02T04:52:32.621536+0000","last_peered":"2024-04-02T04:52:32.621536+0000","last_clean":"2024-04-02T04:44:01.856616+0000","last_became_active":"2024-04-02T04:49:47.381178+0000","last_became_peered":"2024-04-02T04:49:47.381178+0000","last_unstale":"2024-04-02T04:52:32.621536+0000","last_undegraded":"2024-04-02T04:49:47.359978+0000","last_fullsized":"2024-04-02T04:49:47.359801+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:00.356122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.356122+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":509,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.892014+0000","last_change":"2024-04-02T04:50:12.312248+0000","last_active":"2024-04-02T04:52:30.892014+0000","last_peered":"2024-04-02T04:52:30.892014+0000","last_clean":"2024-04-02T04:52:30.892014+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:52:30.892014+0000","last_undegraded":"2024-04-02T04:52:30.892014+0000","last_fullsized":"2024-04-02T04:52:30.892014+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:12.312147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.312147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.889534+0000","last_change":"2024-04-02T04:49:47.438287+0000","last_active":"2024-04-02T04:52:30.889534+0000","last_peered":"2024-04-02T04:52:30.889534+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.438287+0000","last_became_peered":"2024-04-02T04:49:47.438287+0000","last_unstale":"2024-04-02T04:52:30.889534+0000","last_undegraded":"2024-04-02T04:52:30.889534+0000","last_fullsized":"2024-04-02T04:49:47.386231+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.437949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.437949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":359,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.893494+0000","last_change":"2024-04-02T04:49:26.277902+0000","last_active":"2024-04-02T04:52:30.893494+0000","last_peered":"2024-04-02T04:52:30.893494+0000","last_clean":"2024-04-02T04:52:30.893494+0000","last_became_active":"2024-04-02T04:49:26.271155+0000","last_became_peered":"2024-04-02T04:49:26.271155+0000","last_unstale":"2024-04-02T04:52:30.893494+0000","last_undegraded":"2024-04-02T04:52:30.893494+0000","last_fullsized":"2024-04-02T04:52:30.893494+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894397+0000","last_change":"2024-04-02T04:51:27.143038+0000","last_active":"2024-04-02T04:52:30.894397+0000","last_peered":"2024-04-02T04:52:30.894397+0000","last_clean":"2024-04-02T04:52:30.894397+0000","last_became_active":"2024-04-02T04:49:26.265834+0000","last_became_peered":"2024-04-02T04:49:26.265834+0000","last_unstale":"2024-04-02T04:52:30.894397+0000","last_undegraded":"2024-04-02T04:52:30.894397+0000","last_fullsized":"2024-04-02T04:52:30.894397+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:27.142905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.142905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:49.357864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":559,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.904003+0000","last_change":"2024-04-02T04:51:43.069937+0000","last_active":"2024-04-02T04:52:30.904003+0000","last_peered":"2024-04-02T04:52:30.904003+0000","last_clean":"2024-04-02T04:52:30.904003+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:52:30.904003+0000","last_undegraded":"2024-04-02T04:52:30.904003+0000","last_fullsized":"2024-04-02T04:52:30.904003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:51:43.069694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.069694+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:01.917989+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":328,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.890951+0000","last_change":"2024-04-02T04:49:47.385876+0000","last_active":"2024-04-02T04:52:31.890951+0000","last_peered":"2024-04-02T04:52:31.890951+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.385876+0000","last_became_peered":"2024-04-02T04:49:47.385876+0000","last_unstale":"2024-04-02T04:52:31.890951+0000","last_undegraded":"2024-04-02T04:52:31.890951+0000","last_fullsized":"2024-04-02T04:49:47.345306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.439457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.439457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":620,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.895508+0000","last_change":"2024-04-02T04:50:15.516760+0000","last_active":"2024-04-02T04:52:31.895508+0000","last_peered":"2024-04-02T04:52:31.895508+0000","last_clean":"2024-04-02T04:52:31.895508+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:52:31.895508+0000","last_undegraded":"2024-04-02T04:52:31.895508+0000","last_fullsized":"2024-04-02T04:52:31.895508+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:50:15.516538+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.516538+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":513,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.899196+0000","last_change":"2024-04-02T04:51:50.152380+0000","last_active":"2024-04-02T04:52:30.899196+0000","last_peered":"2024-04-02T04:52:30.899196+0000","last_clean":"2024-04-02T04:52:30.899196+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:52:30.899196+0000","last_undegraded":"2024-04-02T04:52:30.899196+0000","last_fullsized":"2024-04-02T04:52:30.899196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:50.152204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.152204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:56.369549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.893928+0000","last_change":"2024-04-02T04:51:37.350257+0000","last_active":"2024-04-02T04:52:31.893928+0000","last_peered":"2024-04-02T04:52:31.893928+0000","last_clean":"2024-04-02T04:52:31.893928+0000","last_became_active":"2024-04-02T04:49:26.241469+0000","last_became_peered":"2024-04-02T04:49:26.241469+0000","last_unstale":"2024-04-02T04:52:31.893928+0000","last_undegraded":"2024-04-02T04:52:31.893928+0000","last_fullsized":"2024-04-02T04:52:31.893928+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:37.350152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:37.350152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:43.325459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":362,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.447315+0000","last_change":"2024-04-02T04:49:26.312549+0000","last_active":"2024-04-02T04:52:32.447315+0000","last_peered":"2024-04-02T04:52:32.447315+0000","last_clean":"2024-04-02T04:52:32.447315+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:52:32.447315+0000","last_undegraded":"2024-04-02T04:52:32.447315+0000","last_fullsized":"2024-04-02T04:52:32.447315+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.893832+0000","last_change":"2024-04-02T04:50:43.513008+0000","last_active":"2024-04-02T04:52:31.893832+0000","last_peered":"2024-04-02T04:52:31.893832+0000","last_clean":"2024-04-02T04:52:31.893832+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:52:31.893832+0000","last_undegraded":"2024-04-02T04:52:31.893832+0000","last_fullsized":"2024-04-02T04:52:31.893832+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:43.512888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.512888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887310+0000","last_change":"2024-04-02T04:50:17.228750+0000","last_active":"2024-04-02T04:52:31.887310+0000","last_peered":"2024-04-02T04:52:31.887310+0000","last_clean":"2024-04-02T04:52:31.887310+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:52:31.887310+0000","last_undegraded":"2024-04-02T04:52:31.887310+0000","last_fullsized":"2024-04-02T04:52:31.887310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:50:17.228650+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.228650+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":516,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.447563+0000","last_change":"2024-04-02T04:49:47.504164+0000","last_active":"2024-04-02T04:52:32.447563+0000","last_peered":"2024-04-02T04:52:32.447563+0000","last_clean":"2024-04-02T04:49:45.357271+0000","last_became_active":"2024-04-02T04:49:47.504164+0000","last_became_peered":"2024-04-02T04:49:47.504164+0000","last_unstale":"2024-04-02T04:52:32.447563+0000","last_undegraded":"2024-04-02T04:52:32.447563+0000","last_fullsized":"2024-04-02T04:49:47.406612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.071225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.391977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.900672+0000","last_change":"2024-04-02T04:50:18.501638+0000","last_active":"2024-04-02T04:52:31.900672+0000","last_peered":"2024-04-02T04:52:31.900672+0000","last_clean":"2024-04-02T04:52:31.900672+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:52:31.900672+0000","last_undegraded":"2024-04-02T04:52:31.900672+0000","last_fullsized":"2024-04-02T04:52:31.900672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.501528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.501528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"139'9","reported_seq":379,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.899469+0000","last_change":"2024-04-02T04:51:29.324138+0000","last_active":"2024-04-02T04:52:31.899469+0000","last_peered":"2024-04-02T04:52:31.899469+0000","last_clean":"2024-04-02T04:52:31.899469+0000","last_became_active":"2024-04-02T04:49:26.487105+0000","last_became_peered":"2024-04-02T04:49:26.487105+0000","last_unstale":"2024-04-02T04:52:31.899469+0000","last_undegraded":"2024-04-02T04:52:31.899469+0000","last_fullsized":"2024-04-02T04:52:31.899469+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'9","last_scrub_stamp":"2024-04-02T04:51:29.324015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.324015+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:53.494806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":378,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.886564+0000","last_change":"2024-04-02T04:49:47.408871+0000","last_active":"2024-04-02T04:52:31.886564+0000","last_peered":"2024-04-02T04:52:31.886564+0000","last_clean":"2024-04-02T04:49:45.308505+0000","last_became_active":"2024-04-02T04:49:47.408871+0000","last_became_peered":"2024-04-02T04:49:47.408871+0000","last_unstale":"2024-04-02T04:52:31.886564+0000","last_undegraded":"2024-04-02T04:49:47.350015+0000","last_fullsized":"2024-04-02T04:49:47.349865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":571,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.883771+0000","last_change":"2024-04-02T04:49:47.511581+0000","last_active":"2024-04-02T04:52:31.883771+0000","last_peered":"2024-04-02T04:52:31.883771+0000","last_clean":"2024-04-02T04:49:45.312272+0000","last_became_active":"2024-04-02T04:49:47.511581+0000","last_became_peered":"2024-04-02T04:49:47.511581+0000","last_unstale":"2024-04-02T04:52:31.883771+0000","last_undegraded":"2024-04-02T04:49:47.361633+0000","last_fullsized":"2024-04-02T04:49:47.361362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:49:24.269994+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.269994+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":527,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.447118+0000","last_change":"2024-04-02T04:50:44.352622+0000","last_active":"2024-04-02T04:52:32.447118+0000","last_peered":"2024-04-02T04:52:32.447118+0000","last_clean":"2024-04-02T04:52:32.447118+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:52:32.447118+0000","last_undegraded":"2024-04-02T04:52:32.447118+0000","last_fullsized":"2024-04-02T04:52:32.447118+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.352505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.352505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.395018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":450,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.899485+0000","last_change":"2024-04-02T04:51:37.221061+0000","last_active":"2024-04-02T04:52:30.899485+0000","last_peered":"2024-04-02T04:52:30.899485+0000","last_clean":"2024-04-02T04:52:30.899485+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:52:30.899485+0000","last_undegraded":"2024-04-02T04:52:30.899485+0000","last_fullsized":"2024-04-02T04:52:30.899485+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:37.220985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:37.220985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:41.124902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":507,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.617176+0000","last_change":"2024-04-02T04:50:37.405854+0000","last_active":"2024-04-02T04:52:32.617176+0000","last_peered":"2024-04-02T04:52:32.617176+0000","last_clean":"2024-04-02T04:52:32.617176+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:52:32.617176+0000","last_undegraded":"2024-04-02T04:52:32.617176+0000","last_fullsized":"2024-04-02T04:52:32.617176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.405673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639653+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":328,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.887547+0000","last_change":"2024-04-02T04:49:47.396455+0000","last_active":"2024-04-02T04:52:30.887547+0000","last_peered":"2024-04-02T04:52:30.887547+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.396455+0000","last_became_peered":"2024-04-02T04:49:47.396455+0000","last_unstale":"2024-04-02T04:52:30.887547+0000","last_undegraded":"2024-04-02T04:52:30.887547+0000","last_fullsized":"2024-04-02T04:49:47.375223+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":359,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.883204+0000","last_change":"2024-04-02T04:49:26.252236+0000","last_active":"2024-04-02T04:52:30.883204+0000","last_peered":"2024-04-02T04:52:30.883204+0000","last_clean":"2024-04-02T04:52:30.883204+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:52:30.883204+0000","last_undegraded":"2024-04-02T04:52:30.883204+0000","last_fullsized":"2024-04-02T04:52:30.883204+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":551,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.887882+0000","last_change":"2024-04-02T04:49:47.453493+0000","last_active":"2024-04-02T04:52:31.887882+0000","last_peered":"2024-04-02T04:52:31.887882+0000","last_clean":"2024-04-02T04:49:45.308750+0000","last_became_active":"2024-04-02T04:49:47.453493+0000","last_became_peered":"2024-04-02T04:49:47.453493+0000","last_unstale":"2024-04-02T04:52:31.887882+0000","last_undegraded":"2024-04-02T04:49:47.351067+0000","last_fullsized":"2024-04-02T04:49:47.350938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:48:49.534769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.534769+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":326,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.884874+0000","last_change":"2024-04-02T04:49:47.521979+0000","last_active":"2024-04-02T04:52:31.884874+0000","last_peered":"2024-04-02T04:52:31.884874+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.521979+0000","last_became_peered":"2024-04-02T04:49:47.521979+0000","last_unstale":"2024-04-02T04:52:31.884874+0000","last_undegraded":"2024-04-02T04:52:31.884874+0000","last_fullsized":"2024-04-02T04:49:47.391120+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.344546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":619,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.900251+0000","last_change":"2024-04-02T04:50:31.493819+0000","last_active":"2024-04-02T04:52:31.900251+0000","last_peered":"2024-04-02T04:52:31.900251+0000","last_clean":"2024-04-02T04:52:31.900251+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:52:31.900251+0000","last_undegraded":"2024-04-02T04:52:31.900251+0000","last_fullsized":"2024-04-02T04:52:31.900251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:31.493715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.493715+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891848+0000","last_change":"2024-04-02T04:50:34.462138+0000","last_active":"2024-04-02T04:52:31.891848+0000","last_peered":"2024-04-02T04:52:31.891848+0000","last_clean":"2024-04-02T04:52:31.891848+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:52:31.891848+0000","last_undegraded":"2024-04-02T04:52:31.891848+0000","last_fullsized":"2024-04-02T04:52:31.891848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:34.462064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.462064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":369,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.888775+0000","last_change":"2024-04-02T04:50:37.076041+0000","last_active":"2024-04-02T04:52:30.888775+0000","last_peered":"2024-04-02T04:52:30.888775+0000","last_clean":"2024-04-02T04:52:30.888775+0000","last_became_active":"2024-04-02T04:49:26.319266+0000","last_became_peered":"2024-04-02T04:49:26.319266+0000","last_unstale":"2024-04-02T04:52:30.888775+0000","last_undegraded":"2024-04-02T04:52:30.888775+0000","last_fullsized":"2024-04-02T04:52:30.888775+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:50:37.075742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.075742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895825+0000","last_change":"2024-04-02T04:50:31.781484+0000","last_active":"2024-04-02T04:52:30.895825+0000","last_peered":"2024-04-02T04:52:30.895825+0000","last_clean":"2024-04-02T04:52:30.895825+0000","last_became_active":"2024-04-02T04:49:26.255429+0000","last_became_peered":"2024-04-02T04:49:26.255429+0000","last_unstale":"2024-04-02T04:52:30.895825+0000","last_undegraded":"2024-04-02T04:52:30.895825+0000","last_fullsized":"2024-04-02T04:52:30.895825+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.781273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.781273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.230045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":506,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891900+0000","last_change":"2024-04-02T04:51:40.385896+0000","last_active":"2024-04-02T04:52:31.891900+0000","last_peered":"2024-04-02T04:52:31.891900+0000","last_clean":"2024-04-02T04:52:31.891900+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:52:31.891900+0000","last_undegraded":"2024-04-02T04:52:31.891900+0000","last_fullsized":"2024-04-02T04:52:31.891900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.385759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:04.004405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"188'18","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.889107+0000","last_change":"2024-04-02T04:50:28.030755+0000","last_active":"2024-04-02T04:52:31.889107+0000","last_peered":"2024-04-02T04:52:31.889107+0000","last_clean":"2024-04-02T04:52:31.889107+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:52:31.889107+0000","last_undegraded":"2024-04-02T04:52:31.889107+0000","last_fullsized":"2024-04-02T04:52:31.889107+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'18","last_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.030658+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.895925+0000","last_change":"2024-04-02T04:49:47.470291+0000","last_active":"2024-04-02T04:52:30.895925+0000","last_peered":"2024-04-02T04:52:30.895925+0000","last_clean":"2024-04-02T04:49:45.387254+0000","last_became_active":"2024-04-02T04:49:47.470291+0000","last_became_peered":"2024-04-02T04:49:47.470291+0000","last_unstale":"2024-04-02T04:52:30.895925+0000","last_undegraded":"2024-04-02T04:52:30.895925+0000","last_fullsized":"2024-04-02T04:49:47.418421+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.690362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.690362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":1012,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.884287+0000","last_change":"2024-04-02T04:50:39.632326+0000","last_active":"2024-04-02T04:52:31.884287+0000","last_peered":"2024-04-02T04:52:31.884287+0000","last_clean":"2024-04-02T04:52:31.884287+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:52:31.884287+0000","last_undegraded":"2024-04-02T04:52:31.884287+0000","last_fullsized":"2024-04-02T04:52:31.884287+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:50:39.632171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.632171+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":495,"num_read_kb":490,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.884771+0000","last_change":"2024-04-02T04:49:47.470355+0000","last_active":"2024-04-02T04:52:30.884771+0000","last_peered":"2024-04-02T04:52:30.884771+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.470355+0000","last_became_peered":"2024-04-02T04:49:47.470355+0000","last_unstale":"2024-04-02T04:52:30.884771+0000","last_undegraded":"2024-04-02T04:52:30.884771+0000","last_fullsized":"2024-04-02T04:49:47.399308+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":369,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.618843+0000","last_change":"2024-04-02T04:50:41.426843+0000","last_active":"2024-04-02T04:52:32.618843+0000","last_peered":"2024-04-02T04:52:32.618843+0000","last_clean":"2024-04-02T04:52:32.618843+0000","last_became_active":"2024-04-02T04:49:26.343060+0000","last_became_peered":"2024-04-02T04:49:26.343060+0000","last_unstale":"2024-04-02T04:52:32.618843+0000","last_undegraded":"2024-04-02T04:52:32.618843+0000","last_fullsized":"2024-04-02T04:52:32.618843+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:41.426741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:41.426741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":362,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.899779+0000","last_change":"2024-04-02T04:49:26.257833+0000","last_active":"2024-04-02T04:52:31.899779+0000","last_peered":"2024-04-02T04:52:31.899779+0000","last_clean":"2024-04-02T04:52:31.899779+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:52:31.899779+0000","last_undegraded":"2024-04-02T04:52:31.899779+0000","last_fullsized":"2024-04-02T04:52:31.899779+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"153'13","reported_seq":626,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.884014+0000","last_change":"2024-04-02T04:49:47.488932+0000","last_active":"2024-04-02T04:52:31.884014+0000","last_peered":"2024-04-02T04:52:31.884014+0000","last_clean":"2024-04-02T04:49:45.312370+0000","last_became_active":"2024-04-02T04:49:47.488932+0000","last_became_peered":"2024-04-02T04:49:47.488932+0000","last_unstale":"2024-04-02T04:52:31.884014+0000","last_undegraded":"2024-04-02T04:49:47.362506+0000","last_fullsized":"2024-04-02T04:49:47.362177+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:48:38.477825+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.477825+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":544,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.896719+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:52:31.896719+0000","last_peered":"2024-04-02T04:52:31.896719+0000","last_clean":"2024-04-02T04:49:45.321982+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:52:31.896719+0000","last_undegraded":"2024-04-02T04:49:47.406158+0000","last_fullsized":"2024-04-02T04:49:47.406050+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:49:07.374876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.374876+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":524,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.451305+0000","last_change":"2024-04-02T04:50:43.360677+0000","last_active":"2024-04-02T04:52:32.451305+0000","last_peered":"2024-04-02T04:52:32.451305+0000","last_clean":"2024-04-02T04:52:32.451305+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:52:32.451305+0000","last_undegraded":"2024-04-02T04:52:32.451305+0000","last_fullsized":"2024-04-02T04:52:32.451305+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:50:43.360457+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.360457+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":508,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.886600+0000","last_change":"2024-04-02T04:49:47.479633+0000","last_active":"2024-04-02T04:52:31.886600+0000","last_peered":"2024-04-02T04:52:31.886600+0000","last_clean":"2024-04-02T04:49:45.308628+0000","last_became_active":"2024-04-02T04:49:47.479633+0000","last_became_peered":"2024-04-02T04:49:47.479633+0000","last_unstale":"2024-04-02T04:52:31.886600+0000","last_undegraded":"2024-04-02T04:52:31.886600+0000","last_fullsized":"2024-04-02T04:49:47.350430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.555854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.555854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887442+0000","last_change":"2024-04-02T04:50:29.535635+0000","last_active":"2024-04-02T04:52:31.887442+0000","last_peered":"2024-04-02T04:52:31.887442+0000","last_clean":"2024-04-02T04:52:31.887442+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:52:31.887442+0000","last_undegraded":"2024-04-02T04:52:31.887442+0000","last_fullsized":"2024-04-02T04:52:31.887442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.535314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.535314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"139'10","reported_seq":377,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.884233+0000","last_change":"2024-04-02T04:50:44.141711+0000","last_active":"2024-04-02T04:52:30.884233+0000","last_peered":"2024-04-02T04:52:30.884233+0000","last_clean":"2024-04-02T04:52:30.884233+0000","last_became_active":"2024-04-02T04:49:26.341161+0000","last_became_peered":"2024-04-02T04:49:26.341161+0000","last_unstale":"2024-04-02T04:52:30.884233+0000","last_undegraded":"2024-04-02T04:52:30.884233+0000","last_fullsized":"2024-04-02T04:52:30.884233+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:50:44.141586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.141586+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.887637+0000","last_change":"2024-04-02T04:50:29.806953+0000","last_active":"2024-04-02T04:52:30.887637+0000","last_peered":"2024-04-02T04:52:30.887637+0000","last_clean":"2024-04-02T04:52:30.887637+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:52:30.887637+0000","last_undegraded":"2024-04-02T04:52:30.887637+0000","last_fullsized":"2024-04-02T04:52:30.887637+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.806751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.806751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"153'13","reported_seq":624,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.887532+0000","last_change":"2024-04-02T04:49:47.472920+0000","last_active":"2024-04-02T04:52:31.887532+0000","last_peered":"2024-04-02T04:52:31.887532+0000","last_clean":"2024-04-02T04:49:45.374829+0000","last_became_active":"2024-04-02T04:49:47.472920+0000","last_became_peered":"2024-04-02T04:49:47.472920+0000","last_unstale":"2024-04-02T04:52:31.887532+0000","last_undegraded":"2024-04-02T04:49:47.384585+0000","last_fullsized":"2024-04-02T04:49:47.384414+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":695,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.448233+0000","last_change":"2024-04-02T04:50:20.307539+0000","last_active":"2024-04-02T04:52:32.448233+0000","last_peered":"2024-04-02T04:52:32.448233+0000","last_clean":"2024-04-02T04:52:32.448233+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:52:32.448233+0000","last_undegraded":"2024-04-02T04:52:32.448233+0000","last_fullsized":"2024-04-02T04:52:32.448233+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:50:20.307283+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.307283+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":326,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.888187+0000","last_change":"2024-04-02T04:49:47.431753+0000","last_active":"2024-04-02T04:52:31.888187+0000","last_peered":"2024-04-02T04:52:31.888187+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.431753+0000","last_became_peered":"2024-04-02T04:49:47.431753+0000","last_unstale":"2024-04-02T04:52:31.888187+0000","last_undegraded":"2024-04-02T04:52:31.888187+0000","last_fullsized":"2024-04-02T04:49:47.405769+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":521,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.903877+0000","last_change":"2024-04-02T04:51:34.006250+0000","last_active":"2024-04-02T04:52:30.903877+0000","last_peered":"2024-04-02T04:52:30.903877+0000","last_clean":"2024-04-02T04:52:30.903877+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:52:30.903877+0000","last_undegraded":"2024-04-02T04:52:30.903877+0000","last_fullsized":"2024-04-02T04:52:30.903877+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:51:34.006018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.006018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:49.364840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.443885+0000","last_change":"2024-04-02T04:50:09.280418+0000","last_active":"2024-04-02T04:52:32.443885+0000","last_peered":"2024-04-02T04:52:32.443885+0000","last_clean":"2024-04-02T04:52:32.443885+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:52:32.443885+0000","last_undegraded":"2024-04-02T04:52:32.443885+0000","last_fullsized":"2024-04-02T04:52:32.443885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.280337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.280337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.885073+0000","last_change":"2024-04-02T04:49:47.516900+0000","last_active":"2024-04-02T04:52:31.885073+0000","last_peered":"2024-04-02T04:52:31.885073+0000","last_clean":"2024-04-02T04:49:45.354881+0000","last_became_active":"2024-04-02T04:49:47.516900+0000","last_became_peered":"2024-04-02T04:49:47.516900+0000","last_unstale":"2024-04-02T04:52:31.885073+0000","last_undegraded":"2024-04-02T04:52:31.885073+0000","last_fullsized":"2024-04-02T04:49:47.391507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":510,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.889271+0000","last_change":"2024-04-02T04:50:13.057737+0000","last_active":"2024-04-02T04:52:31.889271+0000","last_peered":"2024-04-02T04:52:31.889271+0000","last_clean":"2024-04-02T04:52:31.889271+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:52:31.889271+0000","last_undegraded":"2024-04-02T04:52:31.889271+0000","last_fullsized":"2024-04-02T04:52:31.889271+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:13.057654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.057654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":328,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.442975+0000","last_change":"2024-04-02T04:49:47.509591+0000","last_active":"2024-04-02T04:52:32.442975+0000","last_peered":"2024-04-02T04:52:32.442975+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.509591+0000","last_became_peered":"2024-04-02T04:49:47.509591+0000","last_unstale":"2024-04-02T04:52:32.442975+0000","last_undegraded":"2024-04-02T04:52:32.442975+0000","last_fullsized":"2024-04-02T04:49:47.387091+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.441320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.441320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.391864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885404+0000","last_change":"2024-04-02T04:50:19.055361+0000","last_active":"2024-04-02T04:52:31.885404+0000","last_peered":"2024-04-02T04:52:31.885404+0000","last_clean":"2024-04-02T04:52:31.885404+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:52:31.885404+0000","last_undegraded":"2024-04-02T04:52:31.885404+0000","last_fullsized":"2024-04-02T04:52:31.885404+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:50:19.055250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.055250+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":510,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.440118+0000","last_change":"2024-04-02T04:50:26.342752+0000","last_active":"2024-04-02T04:52:32.440118+0000","last_peered":"2024-04-02T04:52:32.440118+0000","last_clean":"2024-04-02T04:52:32.440118+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:52:32.440118+0000","last_undegraded":"2024-04-02T04:52:32.440118+0000","last_fullsized":"2024-04-02T04:52:32.440118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.342429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.342429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":362,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.895303+0000","last_change":"2024-04-02T04:49:47.470287+0000","last_active":"2024-04-02T04:52:30.895303+0000","last_peered":"2024-04-02T04:52:30.895303+0000","last_clean":"2024-04-02T04:49:45.386144+0000","last_became_active":"2024-04-02T04:49:47.470287+0000","last_became_peered":"2024-04-02T04:49:47.470287+0000","last_unstale":"2024-04-02T04:52:30.895303+0000","last_undegraded":"2024-04-02T04:52:30.895303+0000","last_fullsized":"2024-04-02T04:49:47.418077+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.881740+0000","last_change":"2024-04-02T04:50:24.860538+0000","last_active":"2024-04-02T04:52:30.881740+0000","last_peered":"2024-04-02T04:52:30.881740+0000","last_clean":"2024-04-02T04:52:30.881740+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:52:30.881740+0000","last_undegraded":"2024-04-02T04:52:30.881740+0000","last_fullsized":"2024-04-02T04:52:30.881740+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:24.860246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:24.860246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":507,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.904461+0000","last_change":"2024-04-02T04:49:47.413615+0000","last_active":"2024-04-02T04:52:30.904461+0000","last_peered":"2024-04-02T04:52:30.904461+0000","last_clean":"2024-04-02T04:49:45.390373+0000","last_became_active":"2024-04-02T04:49:47.413615+0000","last_became_peered":"2024-04-02T04:49:47.413615+0000","last_unstale":"2024-04-02T04:52:30.904461+0000","last_undegraded":"2024-04-02T04:52:30.904461+0000","last_fullsized":"2024-04-02T04:49:47.396592+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.569058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.569058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":637,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.884604+0000","last_change":"2024-04-02T04:49:47.451735+0000","last_active":"2024-04-02T04:52:31.884604+0000","last_peered":"2024-04-02T04:52:31.884604+0000","last_clean":"2024-04-02T04:44:53.853321+0000","last_became_active":"2024-04-02T04:49:47.451735+0000","last_became_peered":"2024-04-02T04:49:47.451735+0000","last_unstale":"2024-04-02T04:52:31.884604+0000","last_undegraded":"2024-04-02T04:49:47.390635+0000","last_fullsized":"2024-04-02T04:49:47.390485+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":507,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.619983+0000","last_change":"2024-04-02T04:49:47.435042+0000","last_active":"2024-04-02T04:52:32.619983+0000","last_peered":"2024-04-02T04:52:32.619983+0000","last_clean":"2024-04-02T04:49:45.341176+0000","last_became_active":"2024-04-02T04:49:47.435042+0000","last_became_peered":"2024-04-02T04:49:47.435042+0000","last_unstale":"2024-04-02T04:52:32.619983+0000","last_undegraded":"2024-04-02T04:52:32.619983+0000","last_fullsized":"2024-04-02T04:49:47.351952+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:39.903683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.903683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.885530+0000","last_change":"2024-04-02T04:51:49.756835+0000","last_active":"2024-04-02T04:52:31.885530+0000","last_peered":"2024-04-02T04:52:31.885530+0000","last_clean":"2024-04-02T04:52:31.885530+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:52:31.885530+0000","last_undegraded":"2024-04-02T04:52:31.885530+0000","last_fullsized":"2024-04-02T04:52:31.885530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:49.756586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.756586+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.569836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":507,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.617271+0000","last_change":"2024-04-02T04:51:56.577347+0000","last_active":"2024-04-02T04:52:32.617271+0000","last_peered":"2024-04-02T04:52:32.617271+0000","last_clean":"2024-04-02T04:52:32.617271+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:52:32.617271+0000","last_undegraded":"2024-04-02T04:52:32.617271+0000","last_fullsized":"2024-04-02T04:52:32.617271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:56.577226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.577226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:14.507486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.447137+0000","last_change":"2024-04-02T04:51:55.332978+0000","last_active":"2024-04-02T04:52:32.447137+0000","last_peered":"2024-04-02T04:52:32.447137+0000","last_clean":"2024-04-02T04:52:32.447137+0000","last_became_active":"2024-04-02T04:49:26.290196+0000","last_became_peered":"2024-04-02T04:49:26.290196+0000","last_unstale":"2024-04-02T04:52:32.447137+0000","last_undegraded":"2024-04-02T04:52:32.447137+0000","last_fullsized":"2024-04-02T04:52:32.447137+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.332906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.332906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:11.573999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.619192+0000","last_change":"2024-04-02T04:49:47.382055+0000","last_active":"2024-04-02T04:52:32.619192+0000","last_peered":"2024-04-02T04:52:32.619192+0000","last_clean":"2024-04-02T04:49:45.332186+0000","last_became_active":"2024-04-02T04:49:47.382055+0000","last_became_peered":"2024-04-02T04:49:47.382055+0000","last_unstale":"2024-04-02T04:52:32.619192+0000","last_undegraded":"2024-04-02T04:52:32.619192+0000","last_fullsized":"2024-04-02T04:49:47.337410+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.452953+0000","last_change":"2024-04-02T04:50:11.329816+0000","last_active":"2024-04-02T04:52:32.452953+0000","last_peered":"2024-04-02T04:52:32.452953+0000","last_clean":"2024-04-02T04:52:32.452953+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:52:32.452953+0000","last_undegraded":"2024-04-02T04:52:32.452953+0000","last_fullsized":"2024-04-02T04:52:32.452953+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:11.329645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:11.329645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":518,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.884468+0000","last_change":"2024-04-02T04:50:14.307591+0000","last_active":"2024-04-02T04:52:30.884468+0000","last_peered":"2024-04-02T04:52:30.884468+0000","last_clean":"2024-04-02T04:52:30.884468+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:52:30.884468+0000","last_undegraded":"2024-04-02T04:52:30.884468+0000","last_fullsized":"2024-04-02T04:52:30.884468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:14.307478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.307478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":566,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.442900+0000","last_change":"2024-04-02T04:50:16.319646+0000","last_active":"2024-04-02T04:52:32.442900+0000","last_peered":"2024-04-02T04:52:32.442900+0000","last_clean":"2024-04-02T04:52:32.442900+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:52:32.442900+0000","last_undegraded":"2024-04-02T04:52:32.442900+0000","last_fullsized":"2024-04-02T04:52:32.442900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:50:16.319556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:16.319556+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.903305+0000","last_change":"2024-04-02T04:49:47.492471+0000","last_active":"2024-04-02T04:52:30.903305+0000","last_peered":"2024-04-02T04:52:30.903305+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.492471+0000","last_became_peered":"2024-04-02T04:49:47.492471+0000","last_unstale":"2024-04-02T04:52:30.903305+0000","last_undegraded":"2024-04-02T04:52:30.903305+0000","last_fullsized":"2024-04-02T04:49:47.411395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":330,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.622921+0000","last_change":"2024-04-02T04:49:47.388497+0000","last_active":"2024-04-02T04:52:32.622921+0000","last_peered":"2024-04-02T04:52:32.622921+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.388497+0000","last_became_peered":"2024-04-02T04:49:47.388497+0000","last_unstale":"2024-04-02T04:52:32.622921+0000","last_undegraded":"2024-04-02T04:52:32.622921+0000","last_fullsized":"2024-04-02T04:49:47.356622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.883574+0000","last_change":"2024-04-02T04:50:22.825218+0000","last_active":"2024-04-02T04:52:30.883574+0000","last_peered":"2024-04-02T04:52:30.883574+0000","last_clean":"2024-04-02T04:52:30.883574+0000","last_became_active":"2024-04-02T04:49:26.229137+0000","last_became_peered":"2024-04-02T04:49:26.229137+0000","last_unstale":"2024-04-02T04:52:30.883574+0000","last_undegraded":"2024-04-02T04:52:30.883574+0000","last_fullsized":"2024-04-02T04:52:30.883574+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:22.825000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:22.825000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":514,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.447676+0000","last_change":"2024-04-02T04:51:34.344914+0000","last_active":"2024-04-02T04:52:32.447676+0000","last_peered":"2024-04-02T04:52:32.447676+0000","last_clean":"2024-04-02T04:52:32.447676+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:52:32.447676+0000","last_undegraded":"2024-04-02T04:52:32.447676+0000","last_fullsized":"2024-04-02T04:52:32.447676+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:34.344554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.344554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:44.929327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":558,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.895361+0000","last_change":"2024-04-02T04:50:12.546833+0000","last_active":"2024-04-02T04:52:31.895361+0000","last_peered":"2024-04-02T04:52:31.895361+0000","last_clean":"2024-04-02T04:52:31.895361+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:52:31.895361+0000","last_undegraded":"2024-04-02T04:52:31.895361+0000","last_fullsized":"2024-04-02T04:52:31.895361+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:50:12.546683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.546683+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":520,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.619304+0000","last_change":"2024-04-02T04:51:24.605907+0000","last_active":"2024-04-02T04:52:32.619304+0000","last_peered":"2024-04-02T04:52:32.619304+0000","last_clean":"2024-04-02T04:52:32.619304+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:52:32.619304+0000","last_undegraded":"2024-04-02T04:52:32.619304+0000","last_fullsized":"2024-04-02T04:52:32.619304+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:24.605687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:51:24.605687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":499,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.895894+0000","last_change":"2024-04-02T04:49:47.486462+0000","last_active":"2024-04-02T04:52:31.895894+0000","last_peered":"2024-04-02T04:52:31.895894+0000","last_clean":"2024-04-02T04:49:45.345774+0000","last_became_active":"2024-04-02T04:49:47.486462+0000","last_became_peered":"2024-04-02T04:49:47.486462+0000","last_unstale":"2024-04-02T04:52:31.895894+0000","last_undegraded":"2024-04-02T04:49:47.409192+0000","last_fullsized":"2024-04-02T04:49:47.409080+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:48:49.483159+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.483159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"139'5","reported_seq":370,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.888177+0000","last_change":"2024-04-02T04:49:47.490156+0000","last_active":"2024-04-02T04:52:31.888177+0000","last_peered":"2024-04-02T04:52:31.888177+0000","last_clean":"2024-04-02T04:49:45.349908+0000","last_became_active":"2024-04-02T04:49:47.490156+0000","last_became_peered":"2024-04-02T04:49:47.490156+0000","last_unstale":"2024-04-02T04:52:31.888177+0000","last_undegraded":"2024-04-02T04:49:47.382039+0000","last_fullsized":"2024-04-02T04:49:47.381919+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"138'2","reported_seq":371,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.619775+0000","last_change":"2024-04-02T04:51:33.690122+0000","last_active":"2024-04-02T04:52:32.619775+0000","last_peered":"2024-04-02T04:52:32.619775+0000","last_clean":"2024-04-02T04:52:32.619775+0000","last_became_active":"2024-04-02T04:49:26.270708+0000","last_became_peered":"2024-04-02T04:49:26.270708+0000","last_unstale":"2024-04-02T04:52:32.619775+0000","last_undegraded":"2024-04-02T04:52:32.619775+0000","last_fullsized":"2024-04-02T04:52:32.619775+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:51:33.689760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:33.689760+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:57.568983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":331,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.893850+0000","last_change":"2024-04-02T04:49:47.428471+0000","last_active":"2024-04-02T04:52:31.893850+0000","last_peered":"2024-04-02T04:52:31.893850+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.428471+0000","last_became_peered":"2024-04-02T04:49:47.428471+0000","last_unstale":"2024-04-02T04:52:31.893850+0000","last_undegraded":"2024-04-02T04:52:31.893850+0000","last_fullsized":"2024-04-02T04:49:47.368822+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":506,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.448814+0000","last_change":"2024-04-02T04:49:16.090440+0000","last_active":"2024-04-02T04:52:32.448814+0000","last_peered":"2024-04-02T04:52:32.448814+0000","last_clean":"2024-04-02T04:52:32.448814+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:52:32.448814+0000","last_undegraded":"2024-04-02T04:52:32.448814+0000","last_fullsized":"2024-04-02T04:52:32.448814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:16.090227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:16.090227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392450+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":642,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.621234+0000","last_change":"2024-04-02T04:51:35.726385+0000","last_active":"2024-04-02T04:52:32.621234+0000","last_peered":"2024-04-02T04:52:32.621234+0000","last_clean":"2024-04-02T04:52:32.621234+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:52:32.621234+0000","last_undegraded":"2024-04-02T04:52:32.621234+0000","last_fullsized":"2024-04-02T04:52:32.621234+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:35.725768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.725768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:46.350350+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":525,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.451332+0000","last_change":"2024-04-02T04:49:47.508665+0000","last_active":"2024-04-02T04:52:32.451332+0000","last_peered":"2024-04-02T04:52:32.451332+0000","last_clean":"2024-04-02T04:49:45.365160+0000","last_became_active":"2024-04-02T04:49:47.508665+0000","last_became_peered":"2024-04-02T04:49:47.508665+0000","last_unstale":"2024-04-02T04:52:32.451332+0000","last_undegraded":"2024-04-02T04:49:47.437341+0000","last_fullsized":"2024-04-02T04:49:47.437226+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:49:13.057821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.057821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":509,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.884469+0000","last_change":"2024-04-02T04:50:40.208265+0000","last_active":"2024-04-02T04:52:31.884469+0000","last_peered":"2024-04-02T04:52:31.884469+0000","last_clean":"2024-04-02T04:52:31.884469+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:52:31.884469+0000","last_undegraded":"2024-04-02T04:52:31.884469+0000","last_fullsized":"2024-04-02T04:52:31.884469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.208005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.208005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"139'5","reported_seq":345,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.441495+0000","last_change":"2024-04-02T04:49:47.503964+0000","last_active":"2024-04-02T04:52:32.441495+0000","last_peered":"2024-04-02T04:52:32.441495+0000","last_clean":"2024-04-02T04:49:27.480374+0000","last_became_active":"2024-04-02T04:49:47.503964+0000","last_became_peered":"2024-04-02T04:49:47.503964+0000","last_unstale":"2024-04-02T04:52:32.441495+0000","last_undegraded":"2024-04-02T04:49:47.388434+0000","last_fullsized":"2024-04-02T04:49:47.383380+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.391929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":359,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895364+0000","last_change":"2024-04-02T04:49:26.199475+0000","last_active":"2024-04-02T04:52:30.895364+0000","last_peered":"2024-04-02T04:52:30.895364+0000","last_clean":"2024-04-02T04:52:30.895364+0000","last_became_active":"2024-04-02T04:49:26.198942+0000","last_became_peered":"2024-04-02T04:49:26.198942+0000","last_unstale":"2024-04-02T04:52:30.895364+0000","last_undegraded":"2024-04-02T04:52:30.895364+0000","last_fullsized":"2024-04-02T04:52:30.895364+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":620,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887488+0000","last_change":"2024-04-02T04:49:16.603545+0000","last_active":"2024-04-02T04:52:31.887488+0000","last_peered":"2024-04-02T04:52:31.887488+0000","last_clean":"2024-04-02T04:52:31.887488+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:52:31.887488+0000","last_undegraded":"2024-04-02T04:52:31.887488+0000","last_fullsized":"2024-04-02T04:52:31.887488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:16.603444+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:49:16.603444+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":378,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.900074+0000","last_change":"2024-04-02T04:49:47.489637+0000","last_active":"2024-04-02T04:52:31.900074+0000","last_peered":"2024-04-02T04:52:31.900074+0000","last_clean":"2024-04-02T04:44:01.859319+0000","last_became_active":"2024-04-02T04:49:47.489637+0000","last_became_peered":"2024-04-02T04:49:47.489637+0000","last_unstale":"2024-04-02T04:52:31.900074+0000","last_undegraded":"2024-04-02T04:49:47.370487+0000","last_fullsized":"2024-04-02T04:49:47.370305+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":532,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.899054+0000","last_change":"2024-04-02T04:49:47.409343+0000","last_active":"2024-04-02T04:52:31.899054+0000","last_peered":"2024-04-02T04:52:31.899054+0000","last_clean":"2024-04-02T04:49:45.339425+0000","last_became_active":"2024-04-02T04:49:47.409343+0000","last_became_peered":"2024-04-02T04:49:47.409343+0000","last_unstale":"2024-04-02T04:52:31.899054+0000","last_undegraded":"2024-04-02T04:49:47.369170+0000","last_fullsized":"2024-04-02T04:49:47.369009+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:48:39.608255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.608255+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387297+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":497,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.887403+0000","last_change":"2024-04-02T04:49:47.502658+0000","last_active":"2024-04-02T04:52:31.887403+0000","last_peered":"2024-04-02T04:52:31.887403+0000","last_clean":"2024-04-02T04:49:45.343348+0000","last_became_active":"2024-04-02T04:49:47.502658+0000","last_became_peered":"2024-04-02T04:49:47.502658+0000","last_unstale":"2024-04-02T04:52:31.887403+0000","last_undegraded":"2024-04-02T04:52:31.887403+0000","last_fullsized":"2024-04-02T04:49:47.417371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.488397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.488397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.888959+0000","last_change":"2024-04-02T04:51:28.130891+0000","last_active":"2024-04-02T04:52:30.888959+0000","last_peered":"2024-04-02T04:52:30.888959+0000","last_clean":"2024-04-02T04:52:30.888959+0000","last_became_active":"2024-04-02T04:49:26.292029+0000","last_became_peered":"2024-04-02T04:49:26.292029+0000","last_unstale":"2024-04-02T04:52:30.888959+0000","last_undegraded":"2024-04-02T04:52:30.888959+0000","last_fullsized":"2024-04-02T04:52:30.888959+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:28.130758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:28.130758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:42.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":378,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.894778+0000","last_change":"2024-04-02T04:49:47.409578+0000","last_active":"2024-04-02T04:52:31.894778+0000","last_peered":"2024-04-02T04:52:31.894778+0000","last_clean":"2024-04-02T04:44:01.867622+0000","last_became_active":"2024-04-02T04:49:47.409578+0000","last_became_peered":"2024-04-02T04:49:47.409578+0000","last_unstale":"2024-04-02T04:52:31.894778+0000","last_undegraded":"2024-04-02T04:49:47.352200+0000","last_fullsized":"2024-04-02T04:49:47.352060+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":517,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.889067+0000","last_change":"2024-04-02T04:50:20.172828+0000","last_active":"2024-04-02T04:52:30.889067+0000","last_peered":"2024-04-02T04:52:30.889067+0000","last_clean":"2024-04-02T04:52:30.889067+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:52:30.889067+0000","last_undegraded":"2024-04-02T04:52:30.889067+0000","last_fullsized":"2024-04-02T04:52:30.889067+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:20.172630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.172630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":499,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.889487+0000","last_change":"2024-04-02T04:49:47.492319+0000","last_active":"2024-04-02T04:52:31.889487+0000","last_peered":"2024-04-02T04:52:31.889487+0000","last_clean":"2024-04-02T04:49:45.387118+0000","last_became_active":"2024-04-02T04:49:47.492319+0000","last_became_peered":"2024-04-02T04:49:47.492319+0000","last_unstale":"2024-04-02T04:52:31.889487+0000","last_undegraded":"2024-04-02T04:52:31.889487+0000","last_fullsized":"2024-04-02T04:49:47.417925+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.103147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":492,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.894500+0000","last_change":"2024-04-02T04:49:47.511784+0000","last_active":"2024-04-02T04:52:30.894500+0000","last_peered":"2024-04-02T04:52:30.894500+0000","last_clean":"2024-04-02T04:49:45.315951+0000","last_became_active":"2024-04-02T04:49:47.511784+0000","last_became_peered":"2024-04-02T04:49:47.511784+0000","last_unstale":"2024-04-02T04:52:30.894500+0000","last_undegraded":"2024-04-02T04:52:30.894500+0000","last_fullsized":"2024-04-02T04:49:47.381831+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.646709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.646709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"139'5","reported_seq":364,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895001+0000","last_change":"2024-04-02T04:49:26.178484+0000","last_active":"2024-04-02T04:52:30.895001+0000","last_peered":"2024-04-02T04:52:30.895001+0000","last_clean":"2024-04-02T04:52:30.895001+0000","last_became_active":"2024-04-02T04:49:26.178040+0000","last_became_peered":"2024-04-02T04:49:26.178040+0000","last_unstale":"2024-04-02T04:52:30.895001+0000","last_undegraded":"2024-04-02T04:52:30.895001+0000","last_fullsized":"2024-04-02T04:52:30.895001+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.441492+0000","last_change":"2024-04-02T04:51:29.319201+0000","last_active":"2024-04-02T04:52:32.441492+0000","last_peered":"2024-04-02T04:52:32.441492+0000","last_clean":"2024-04-02T04:52:32.441492+0000","last_became_active":"2024-04-02T04:49:26.258674+0000","last_became_peered":"2024-04-02T04:49:26.258674+0000","last_unstale":"2024-04-02T04:52:32.441492+0000","last_undegraded":"2024-04-02T04:52:32.441492+0000","last_fullsized":"2024-04-02T04:52:32.441492+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.318999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.318999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:37.437604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":513,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.883475+0000","last_change":"2024-04-02T04:50:17.746744+0000","last_active":"2024-04-02T04:52:30.883475+0000","last_peered":"2024-04-02T04:52:30.883475+0000","last_clean":"2024-04-02T04:52:30.883475+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:52:30.883475+0000","last_undegraded":"2024-04-02T04:52:30.883475+0000","last_fullsized":"2024-04-02T04:52:30.883475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:50:17.746578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.746578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":519,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.885004+0000","last_change":"2024-04-02T04:51:40.072460+0000","last_active":"2024-04-02T04:52:30.885004+0000","last_peered":"2024-04-02T04:52:30.885004+0000","last_clean":"2024-04-02T04:52:30.885004+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:52:30.885004+0000","last_undegraded":"2024-04-02T04:52:30.885004+0000","last_fullsized":"2024-04-02T04:52:30.885004+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.072262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.072262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:46.565472+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":507,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.896942+0000","last_change":"2024-04-02T04:49:47.477916+0000","last_active":"2024-04-02T04:52:31.896942+0000","last_peered":"2024-04-02T04:52:31.896942+0000","last_clean":"2024-04-02T04:49:45.322247+0000","last_became_active":"2024-04-02T04:49:47.477916+0000","last_became_peered":"2024-04-02T04:49:47.477916+0000","last_unstale":"2024-04-02T04:52:31.896942+0000","last_undegraded":"2024-04-02T04:52:31.896942+0000","last_fullsized":"2024-04-02T04:49:47.406503+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.485165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.485165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.904653+0000","last_change":"2024-04-02T04:49:47.412587+0000","last_active":"2024-04-02T04:52:30.904653+0000","last_peered":"2024-04-02T04:52:30.904653+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.412587+0000","last_became_peered":"2024-04-02T04:49:47.412587+0000","last_unstale":"2024-04-02T04:52:30.904653+0000","last_undegraded":"2024-04-02T04:52:30.904653+0000","last_fullsized":"2024-04-02T04:49:47.396252+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.304153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.304153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.899209+0000","last_change":"2024-04-02T04:51:28.314780+0000","last_active":"2024-04-02T04:52:31.899209+0000","last_peered":"2024-04-02T04:52:31.899209+0000","last_clean":"2024-04-02T04:52:31.899209+0000","last_became_active":"2024-04-02T04:49:26.250745+0000","last_became_peered":"2024-04-02T04:49:26.250745+0000","last_unstale":"2024-04-02T04:52:31.899209+0000","last_undegraded":"2024-04-02T04:52:31.899209+0000","last_fullsized":"2024-04-02T04:52:31.899209+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:28.314678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:28.314678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:38.299397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.882684+0000","last_change":"2024-04-02T04:51:39.049216+0000","last_active":"2024-04-02T04:52:30.882684+0000","last_peered":"2024-04-02T04:52:30.882684+0000","last_clean":"2024-04-02T04:52:30.882684+0000","last_became_active":"2024-04-02T04:49:26.229293+0000","last_became_peered":"2024-04-02T04:49:26.229293+0000","last_unstale":"2024-04-02T04:52:30.882684+0000","last_undegraded":"2024-04-02T04:52:30.882684+0000","last_fullsized":"2024-04-02T04:52:30.882684+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.048988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:04.966675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":366,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.890807+0000","last_change":"2024-04-02T04:49:47.452491+0000","last_active":"2024-04-02T04:52:31.890807+0000","last_peered":"2024-04-02T04:52:31.890807+0000","last_clean":"2024-04-02T04:44:01.831181+0000","last_became_active":"2024-04-02T04:49:47.452491+0000","last_became_peered":"2024-04-02T04:49:47.452491+0000","last_unstale":"2024-04-02T04:52:31.890807+0000","last_undegraded":"2024-04-02T04:49:47.355108+0000","last_fullsized":"2024-04-02T04:49:47.354938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":506,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.619017+0000","last_change":"2024-04-02T04:49:47.419557+0000","last_active":"2024-04-02T04:52:32.619017+0000","last_peered":"2024-04-02T04:52:32.619017+0000","last_clean":"2024-04-02T04:49:45.336531+0000","last_became_active":"2024-04-02T04:49:47.419557+0000","last_became_peered":"2024-04-02T04:49:47.419557+0000","last_unstale":"2024-04-02T04:52:32.619017+0000","last_undegraded":"2024-04-02T04:52:32.619017+0000","last_fullsized":"2024-04-02T04:49:47.341828+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.899356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.899356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"153'13","reported_seq":617,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.892015+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:52:31.892015+0000","last_peered":"2024-04-02T04:52:31.892015+0000","last_clean":"2024-04-02T04:49:45.328987+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:52:31.892015+0000","last_undegraded":"2024-04-02T04:49:47.427380+0000","last_fullsized":"2024-04-02T04:49:47.427156+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677278+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":495,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.889330+0000","last_change":"2024-04-02T04:49:47.491986+0000","last_active":"2024-04-02T04:52:31.889330+0000","last_peered":"2024-04-02T04:52:31.889330+0000","last_clean":"2024-04-02T04:49:45.387071+0000","last_became_active":"2024-04-02T04:49:47.491986+0000","last_became_peered":"2024-04-02T04:49:47.491986+0000","last_unstale":"2024-04-02T04:52:31.889330+0000","last_undegraded":"2024-04-02T04:52:31.889330+0000","last_fullsized":"2024-04-02T04:49:47.416002+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.986307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.986307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"139'7","reported_seq":344,"reported_epoch":311,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:30.898627+0000","last_change":"2024-04-02T04:49:47.507515+0000","last_active":"2024-04-02T04:52:30.898627+0000","last_peered":"2024-04-02T04:52:30.898627+0000","last_clean":"2024-04-02T04:49:27.511662+0000","last_became_active":"2024-04-02T04:49:47.507515+0000","last_became_peered":"2024-04-02T04:49:47.507515+0000","last_unstale":"2024-04-02T04:52:30.898627+0000","last_undegraded":"2024-04-02T04:49:47.419015+0000","last_fullsized":"2024-04-02T04:49:47.418904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":366,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.886804+0000","last_change":"2024-04-02T04:49:47.480176+0000","last_active":"2024-04-02T04:52:31.886804+0000","last_peered":"2024-04-02T04:52:31.886804+0000","last_clean":"2024-04-02T04:49:45.335187+0000","last_became_active":"2024-04-02T04:49:47.480176+0000","last_became_peered":"2024-04-02T04:49:47.480176+0000","last_unstale":"2024-04-02T04:52:31.886804+0000","last_undegraded":"2024-04-02T04:52:31.886804+0000","last_fullsized":"2024-04-02T04:49:47.401165+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":527,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.887184+0000","last_change":"2024-04-02T04:49:47.492714+0000","last_active":"2024-04-02T04:52:31.887184+0000","last_peered":"2024-04-02T04:52:31.887184+0000","last_clean":"2024-04-02T04:49:45.370028+0000","last_became_active":"2024-04-02T04:49:47.492714+0000","last_became_peered":"2024-04-02T04:49:47.492714+0000","last_unstale":"2024-04-02T04:52:31.887184+0000","last_undegraded":"2024-04-02T04:49:47.390470+0000","last_fullsized":"2024-04-02T04:49:47.390328+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:49:14.989379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.989379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":512,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887002+0000","last_change":"2024-04-02T04:50:21.021320+0000","last_active":"2024-04-02T04:52:31.887002+0000","last_peered":"2024-04-02T04:52:31.887002+0000","last_clean":"2024-04-02T04:52:31.887002+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:52:31.887002+0000","last_undegraded":"2024-04-02T04:52:31.887002+0000","last_fullsized":"2024-04-02T04:52:31.887002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.021194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.021194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"153'12","reported_seq":616,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.621569+0000","last_change":"2024-04-02T04:49:47.418645+0000","last_active":"2024-04-02T04:52:32.621569+0000","last_peered":"2024-04-02T04:52:32.621569+0000","last_clean":"2024-04-02T04:49:45.363313+0000","last_became_active":"2024-04-02T04:49:47.418645+0000","last_became_peered":"2024-04-02T04:49:47.418645+0000","last_unstale":"2024-04-02T04:52:32.621569+0000","last_undegraded":"2024-04-02T04:49:47.360244+0000","last_fullsized":"2024-04-02T04:49:47.360013+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:41.932321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.932321+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.899930+0000","last_change":"2024-04-02T04:50:42.526150+0000","last_active":"2024-04-02T04:52:31.899930+0000","last_peered":"2024-04-02T04:52:31.899930+0000","last_clean":"2024-04-02T04:52:31.899930+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:52:31.899930+0000","last_undegraded":"2024-04-02T04:52:31.899930+0000","last_fullsized":"2024-04-02T04:52:31.899930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:42.525995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:42.525995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":359,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.893341+0000","last_change":"2024-04-02T04:49:26.281485+0000","last_active":"2024-04-02T04:52:30.893341+0000","last_peered":"2024-04-02T04:52:30.893341+0000","last_clean":"2024-04-02T04:52:30.893341+0000","last_became_active":"2024-04-02T04:49:26.280984+0000","last_became_peered":"2024-04-02T04:49:26.280984+0000","last_unstale":"2024-04-02T04:52:30.893341+0000","last_undegraded":"2024-04-02T04:52:30.893341+0000","last_fullsized":"2024-04-02T04:52:30.893341+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.893604+0000","last_change":"2024-04-02T04:50:25.822947+0000","last_active":"2024-04-02T04:52:30.893604+0000","last_peered":"2024-04-02T04:52:30.893604+0000","last_clean":"2024-04-02T04:52:30.893604+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:52:30.893604+0000","last_undegraded":"2024-04-02T04:52:30.893604+0000","last_fullsized":"2024-04-02T04:52:30.893604+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:25.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.230139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":375,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.891785+0000","last_change":"2024-04-02T04:49:47.479230+0000","last_active":"2024-04-02T04:52:31.891785+0000","last_peered":"2024-04-02T04:52:31.891785+0000","last_clean":"2024-04-02T04:44:01.845646+0000","last_became_active":"2024-04-02T04:49:47.479230+0000","last_became_peered":"2024-04-02T04:49:47.479230+0000","last_unstale":"2024-04-02T04:52:31.891785+0000","last_undegraded":"2024-04-02T04:49:47.401783+0000","last_fullsized":"2024-04-02T04:49:47.401678+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:49:23.491068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:23.491068+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":511,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.893757+0000","last_change":"2024-04-02T04:50:21.847768+0000","last_active":"2024-04-02T04:52:30.893757+0000","last_peered":"2024-04-02T04:52:30.893757+0000","last_clean":"2024-04-02T04:52:30.893757+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:52:30.893757+0000","last_undegraded":"2024-04-02T04:52:30.893757+0000","last_fullsized":"2024-04-02T04:52:30.893757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.847551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.847551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.447060+0000","last_change":"2024-04-02T04:50:10.316715+0000","last_active":"2024-04-02T04:52:32.447060+0000","last_peered":"2024-04-02T04:52:32.447060+0000","last_clean":"2024-04-02T04:52:32.447060+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:52:32.447060+0000","last_undegraded":"2024-04-02T04:52:32.447060+0000","last_fullsized":"2024-04-02T04:52:32.447060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:10.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:10.316429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":508,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.884342+0000","last_change":"2024-04-02T04:51:53.965925+0000","last_active":"2024-04-02T04:52:31.884342+0000","last_peered":"2024-04-02T04:52:31.884342+0000","last_clean":"2024-04-02T04:52:31.884342+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:52:31.884342+0000","last_undegraded":"2024-04-02T04:52:31.884342+0000","last_fullsized":"2024-04-02T04:52:31.884342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.965811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.965811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:56.102854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"139'10","reported_seq":380,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887910+0000","last_change":"2024-04-02T04:51:30.300205+0000","last_active":"2024-04-02T04:52:31.887910+0000","last_peered":"2024-04-02T04:52:31.887910+0000","last_clean":"2024-04-02T04:52:31.887910+0000","last_became_active":"2024-04-02T04:49:26.260148+0000","last_became_peered":"2024-04-02T04:49:26.260148+0000","last_unstale":"2024-04-02T04:52:31.887910+0000","last_undegraded":"2024-04-02T04:52:31.887910+0000","last_fullsized":"2024-04-02T04:52:31.887910+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:51:30.299917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.299917+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:52.994896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891017+0000","last_change":"2024-04-02T04:51:26.346563+0000","last_active":"2024-04-02T04:52:31.891017+0000","last_peered":"2024-04-02T04:52:31.891017+0000","last_clean":"2024-04-02T04:52:31.891017+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:52:31.891017+0000","last_undegraded":"2024-04-02T04:52:31.891017+0000","last_fullsized":"2024-04-02T04:52:31.891017+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.346350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.346350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:38.179083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":543,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.890453+0000","last_change":"2024-04-02T04:49:47.392866+0000","last_active":"2024-04-02T04:52:31.890453+0000","last_peered":"2024-04-02T04:52:31.890453+0000","last_clean":"2024-04-02T04:49:45.343613+0000","last_became_active":"2024-04-02T04:49:47.392866+0000","last_became_peered":"2024-04-02T04:49:47.392866+0000","last_unstale":"2024-04-02T04:52:31.890453+0000","last_undegraded":"2024-04-02T04:49:47.367503+0000","last_fullsized":"2024-04-02T04:49:47.367381+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387208+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":513,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.452947+0000","last_change":"2024-04-02T04:50:19.341033+0000","last_active":"2024-04-02T04:52:32.452947+0000","last_peered":"2024-04-02T04:52:32.452947+0000","last_clean":"2024-04-02T04:52:32.452947+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:52:32.452947+0000","last_undegraded":"2024-04-02T04:52:32.452947+0000","last_fullsized":"2024-04-02T04:52:32.452947+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:19.340953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:19.340953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.393996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888408+0000","last_change":"2024-04-02T04:49:12.685154+0000","last_active":"2024-04-02T04:52:31.888408+0000","last_peered":"2024-04-02T04:52:31.888408+0000","last_clean":"2024-04-02T04:52:31.888408+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:52:31.888408+0000","last_undegraded":"2024-04-02T04:52:31.888408+0000","last_fullsized":"2024-04-02T04:52:31.888408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.684916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.684916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887980+0000","last_change":"2024-04-02T04:50:32.383573+0000","last_active":"2024-04-02T04:52:31.887980+0000","last_peered":"2024-04-02T04:52:31.887980+0000","last_clean":"2024-04-02T04:52:31.887980+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:52:31.887980+0000","last_undegraded":"2024-04-02T04:52:31.887980+0000","last_fullsized":"2024-04-02T04:52:31.887980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.383421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"262'6","reported_seq":376,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.448512+0000","last_change":"2024-04-02T04:51:42.313312+0000","last_active":"2024-04-02T04:52:32.448512+0000","last_peered":"2024-04-02T04:52:32.448512+0000","last_clean":"2024-04-02T04:52:32.448512+0000","last_became_active":"2024-04-02T04:49:26.275418+0000","last_became_peered":"2024-04-02T04:49:26.275418+0000","last_unstale":"2024-04-02T04:52:32.448512+0000","last_undegraded":"2024-04-02T04:52:32.448512+0000","last_fullsized":"2024-04-02T04:52:32.448512+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'6","last_scrub_stamp":"2024-04-02T04:51:42.313232+0000","last_deep_scrub":"262'6","last_deep_scrub_stamp":"2024-04-02T04:51:42.313232+0000","last_clean_scrub_stamp":"2024-04-02T04:51:42.313232+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:58.430993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":369,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.889034+0000","last_change":"2024-04-02T04:50:30.067941+0000","last_active":"2024-04-02T04:52:31.889034+0000","last_peered":"2024-04-02T04:52:31.889034+0000","last_clean":"2024-04-02T04:52:31.889034+0000","last_became_active":"2024-04-02T04:49:25.080486+0000","last_became_peered":"2024-04-02T04:49:25.080486+0000","last_unstale":"2024-04-02T04:52:31.889034+0000","last_undegraded":"2024-04-02T04:52:31.889034+0000","last_fullsized":"2024-04-02T04:52:31.889034+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:50:30.067845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":529,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.890228+0000","last_change":"2024-04-02T04:50:25.999986+0000","last_active":"2024-04-02T04:52:31.890228+0000","last_peered":"2024-04-02T04:52:31.890228+0000","last_clean":"2024-04-02T04:52:31.890228+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:52:31.890228+0000","last_undegraded":"2024-04-02T04:52:31.890228+0000","last_fullsized":"2024-04-02T04:52:31.890228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:50:25.999894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.999894+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":519,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.622978+0000","last_change":"2024-04-02T04:50:39.424795+0000","last_active":"2024-04-02T04:52:32.622978+0000","last_peered":"2024-04-02T04:52:32.622978+0000","last_clean":"2024-04-02T04:52:32.622978+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:52:32.622978+0000","last_undegraded":"2024-04-02T04:52:32.622978+0000","last_fullsized":"2024-04-02T04:52:32.622978+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.424575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":514,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.883675+0000","last_change":"2024-04-02T04:51:54.722249+0000","last_active":"2024-04-02T04:52:31.883675+0000","last_peered":"2024-04-02T04:52:31.883675+0000","last_clean":"2024-04-02T04:52:31.883675+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:52:31.883675+0000","last_undegraded":"2024-04-02T04:52:31.883675+0000","last_fullsized":"2024-04-02T04:52:31.883675+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:54.722054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:54.722054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:55.046704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.884548+0000","last_change":"2024-04-02T04:49:47.512747+0000","last_active":"2024-04-02T04:52:30.884548+0000","last_peered":"2024-04-02T04:52:30.884548+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.512747+0000","last_became_peered":"2024-04-02T04:49:47.512747+0000","last_unstale":"2024-04-02T04:52:30.884548+0000","last_undegraded":"2024-04-02T04:52:30.884548+0000","last_fullsized":"2024-04-02T04:49:47.402486+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.372299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.372299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":367,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.888887+0000","last_change":"2024-04-02T04:49:47.444258+0000","last_active":"2024-04-02T04:52:31.888887+0000","last_peered":"2024-04-02T04:52:31.888887+0000","last_clean":"2024-04-02T04:49:45.328612+0000","last_became_active":"2024-04-02T04:49:47.444258+0000","last_became_peered":"2024-04-02T04:49:47.444258+0000","last_unstale":"2024-04-02T04:52:31.888887+0000","last_undegraded":"2024-04-02T04:52:31.888887+0000","last_fullsized":"2024-04-02T04:49:47.397778+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":331,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.455219+0000","last_change":"2024-04-02T04:49:47.517978+0000","last_active":"2024-04-02T04:52:32.455219+0000","last_peered":"2024-04-02T04:52:32.455219+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.517978+0000","last_became_peered":"2024-04-02T04:49:47.517978+0000","last_unstale":"2024-04-02T04:52:32.455219+0000","last_undegraded":"2024-04-02T04:52:32.455219+0000","last_fullsized":"2024-04-02T04:49:47.456750+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":550,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.622715+0000","last_change":"2024-04-02T04:49:47.430162+0000","last_active":"2024-04-02T04:52:32.622715+0000","last_peered":"2024-04-02T04:52:32.622715+0000","last_clean":"2024-04-02T04:49:45.364593+0000","last_became_active":"2024-04-02T04:49:47.430162+0000","last_became_peered":"2024-04-02T04:49:47.430162+0000","last_unstale":"2024-04-02T04:52:32.622715+0000","last_undegraded":"2024-04-02T04:49:47.355994+0000","last_fullsized":"2024-04-02T04:49:47.355865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:49:03.788837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.788837+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638883+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":519,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.895625+0000","last_change":"2024-04-02T04:50:08.818906+0000","last_active":"2024-04-02T04:52:30.895625+0000","last_peered":"2024-04-02T04:52:30.895625+0000","last_clean":"2024-04-02T04:52:30.895625+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:52:30.895625+0000","last_undegraded":"2024-04-02T04:52:30.895625+0000","last_fullsized":"2024-04-02T04:52:30.895625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:08.818819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:50:08.818819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.230091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.888283+0000","last_change":"2024-04-02T04:49:47.443764+0000","last_active":"2024-04-02T04:52:31.888283+0000","last_peered":"2024-04-02T04:52:31.888283+0000","last_clean":"2024-04-02T04:49:45.325466+0000","last_became_active":"2024-04-02T04:49:47.443764+0000","last_became_peered":"2024-04-02T04:49:47.443764+0000","last_unstale":"2024-04-02T04:52:31.888283+0000","last_undegraded":"2024-04-02T04:52:31.888283+0000","last_fullsized":"2024-04-02T04:49:47.396652+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.351209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":494,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887541+0000","last_change":"2024-04-02T04:49:25.285134+0000","last_active":"2024-04-02T04:52:31.887541+0000","last_peered":"2024-04-02T04:52:31.887541+0000","last_clean":"2024-04-02T04:52:31.887541+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:52:31.887541+0000","last_undegraded":"2024-04-02T04:52:31.887541+0000","last_fullsized":"2024-04-02T04:52:31.887541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.284922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.284922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"138'2","reported_seq":338,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.884061+0000","last_change":"2024-04-02T04:49:47.436385+0000","last_active":"2024-04-02T04:52:31.884061+0000","last_peered":"2024-04-02T04:52:31.884061+0000","last_clean":"2024-04-02T04:49:26.658740+0000","last_became_active":"2024-04-02T04:49:47.436385+0000","last_became_peered":"2024-04-02T04:49:47.436385+0000","last_unstale":"2024-04-02T04:52:31.884061+0000","last_undegraded":"2024-04-02T04:52:31.884061+0000","last_fullsized":"2024-04-02T04:49:47.371603+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":365,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.890394+0000","last_change":"2024-04-02T04:49:47.393078+0000","last_active":"2024-04-02T04:52:31.890394+0000","last_peered":"2024-04-02T04:52:31.890394+0000","last_clean":"2024-04-02T04:49:45.306613+0000","last_became_active":"2024-04-02T04:49:47.393078+0000","last_became_peered":"2024-04-02T04:49:47.393078+0000","last_unstale":"2024-04-02T04:52:31.890394+0000","last_undegraded":"2024-04-02T04:52:31.890394+0000","last_fullsized":"2024-04-02T04:49:47.373401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"133'50","reported_seq":593,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.441907+0000","last_change":"2024-04-02T04:50:33.298622+0000","last_active":"2024-04-02T04:52:32.441907+0000","last_peered":"2024-04-02T04:52:32.441907+0000","last_clean":"2024-04-02T04:52:32.441907+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:52:32.441907+0000","last_undegraded":"2024-04-02T04:52:32.441907+0000","last_fullsized":"2024-04-02T04:52:32.441907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:50:33.298479+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:50:33.298479+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":514,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618357+0000","last_change":"2024-04-02T04:49:47.388339+0000","last_active":"2024-04-02T04:52:32.618357+0000","last_peered":"2024-04-02T04:52:32.618357+0000","last_clean":"2024-04-02T04:49:45.330232+0000","last_became_active":"2024-04-02T04:49:47.388339+0000","last_became_peered":"2024-04-02T04:49:47.388339+0000","last_unstale":"2024-04-02T04:52:32.618357+0000","last_undegraded":"2024-04-02T04:52:32.618357+0000","last_fullsized":"2024-04-02T04:49:47.337757+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:51.870038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:51.870038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":508,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.889137+0000","last_change":"2024-04-02T04:49:47.491600+0000","last_active":"2024-04-02T04:52:31.889137+0000","last_peered":"2024-04-02T04:52:31.889137+0000","last_clean":"2024-04-02T04:49:45.386942+0000","last_became_active":"2024-04-02T04:49:47.491600+0000","last_became_peered":"2024-04-02T04:49:47.491600+0000","last_unstale":"2024-04-02T04:52:31.889137+0000","last_undegraded":"2024-04-02T04:52:31.889137+0000","last_fullsized":"2024-04-02T04:49:47.415538+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":338,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.619792+0000","last_change":"2024-04-02T04:49:47.434796+0000","last_active":"2024-04-02T04:52:32.619792+0000","last_peered":"2024-04-02T04:52:32.619792+0000","last_clean":"2024-04-02T04:44:01.344239+0000","last_became_active":"2024-04-02T04:49:47.434796+0000","last_became_peered":"2024-04-02T04:49:47.434796+0000","last_unstale":"2024-04-02T04:52:32.619792+0000","last_undegraded":"2024-04-02T04:49:47.346909+0000","last_fullsized":"2024-04-02T04:49:47.346781+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:48:42.394228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.394228+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.17","version":"139'5","reported_seq":374,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.623067+0000","last_change":"2024-04-02T04:51:25.643429+0000","last_active":"2024-04-02T04:52:32.623067+0000","last_peered":"2024-04-02T04:52:32.623067+0000","last_clean":"2024-04-02T04:52:32.623067+0000","last_became_active":"2024-04-02T04:49:26.272870+0000","last_became_peered":"2024-04-02T04:49:26.272870+0000","last_unstale":"2024-04-02T04:52:32.623067+0000","last_undegraded":"2024-04-02T04:52:32.623067+0000","last_fullsized":"2024-04-02T04:52:32.623067+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:25.643193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.643193+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":364,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.618950+0000","last_change":"2024-04-02T04:49:47.478289+0000","last_active":"2024-04-02T04:52:32.618950+0000","last_peered":"2024-04-02T04:52:32.618950+0000","last_clean":"2024-04-02T04:49:45.332087+0000","last_became_active":"2024-04-02T04:49:47.478289+0000","last_became_peered":"2024-04-02T04:49:47.478289+0000","last_unstale":"2024-04-02T04:52:32.618950+0000","last_undegraded":"2024-04-02T04:52:32.618950+0000","last_fullsized":"2024-04-02T04:49:47.337387+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.883571+0000","last_change":"2024-04-02T04:50:14.221770+0000","last_active":"2024-04-02T04:52:31.883571+0000","last_peered":"2024-04-02T04:52:31.883571+0000","last_clean":"2024-04-02T04:52:31.883571+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:52:31.883571+0000","last_undegraded":"2024-04-02T04:52:31.883571+0000","last_fullsized":"2024-04-02T04:52:31.883571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:50:14.221684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.221684+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":516,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888427+0000","last_change":"2024-04-02T04:50:28.247971+0000","last_active":"2024-04-02T04:52:31.888427+0000","last_peered":"2024-04-02T04:52:31.888427+0000","last_clean":"2024-04-02T04:52:31.888427+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:52:31.888427+0000","last_undegraded":"2024-04-02T04:52:31.888427+0000","last_fullsized":"2024-04-02T04:52:31.888427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:50:28.247657+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.247657+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":621,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.896123+0000","last_change":"2024-04-02T04:50:12.840880+0000","last_active":"2024-04-02T04:52:30.896123+0000","last_peered":"2024-04-02T04:52:30.896123+0000","last_clean":"2024-04-02T04:52:30.896123+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:52:30.896123+0000","last_undegraded":"2024-04-02T04:52:30.896123+0000","last_fullsized":"2024-04-02T04:52:30.896123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:12.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.840785+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229869+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.440175+0000","last_change":"2024-04-02T04:50:15.299404+0000","last_active":"2024-04-02T04:52:32.440175+0000","last_peered":"2024-04-02T04:52:32.440175+0000","last_clean":"2024-04-02T04:52:32.440175+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:52:32.440175+0000","last_undegraded":"2024-04-02T04:52:32.440175+0000","last_fullsized":"2024-04-02T04:52:32.440175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.299316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.299316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"139'5","reported_seq":377,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.896157+0000","last_change":"2024-04-02T04:51:32.581419+0000","last_active":"2024-04-02T04:52:31.896157+0000","last_peered":"2024-04-02T04:52:31.896157+0000","last_clean":"2024-04-02T04:52:31.896157+0000","last_became_active":"2024-04-02T04:49:25.091539+0000","last_became_peered":"2024-04-02T04:49:25.091539+0000","last_unstale":"2024-04-02T04:52:31.896157+0000","last_undegraded":"2024-04-02T04:52:31.896157+0000","last_fullsized":"2024-04-02T04:52:31.896157+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:32.581233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:32.581233+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:00.548171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.884666+0000","last_change":"2024-04-02T04:50:37.840934+0000","last_active":"2024-04-02T04:52:30.884666+0000","last_peered":"2024-04-02T04:52:30.884666+0000","last_clean":"2024-04-02T04:52:30.884666+0000","last_became_active":"2024-04-02T04:49:26.256165+0000","last_became_peered":"2024-04-02T04:49:26.256165+0000","last_unstale":"2024-04-02T04:52:30.884666+0000","last_undegraded":"2024-04-02T04:52:30.884666+0000","last_fullsized":"2024-04-02T04:52:30.884666+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.840813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.840813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.230272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":528,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.887248+0000","last_change":"2024-04-02T04:49:47.432220+0000","last_active":"2024-04-02T04:52:31.887248+0000","last_peered":"2024-04-02T04:52:31.887248+0000","last_clean":"2024-04-02T04:49:45.374676+0000","last_became_active":"2024-04-02T04:49:47.432220+0000","last_became_peered":"2024-04-02T04:49:47.432220+0000","last_unstale":"2024-04-02T04:52:31.887248+0000","last_undegraded":"2024-04-02T04:49:47.384090+0000","last_fullsized":"2024-04-02T04:49:47.383905+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:49:05.019939+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.019939+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":514,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.903016+0000","last_change":"2024-04-02T04:49:47.404868+0000","last_active":"2024-04-02T04:52:30.903016+0000","last_peered":"2024-04-02T04:52:30.903016+0000","last_clean":"2024-04-02T04:49:45.386009+0000","last_became_active":"2024-04-02T04:49:47.404868+0000","last_became_peered":"2024-04-02T04:49:47.404868+0000","last_unstale":"2024-04-02T04:52:30.903016+0000","last_undegraded":"2024-04-02T04:52:30.903016+0000","last_fullsized":"2024-04-02T04:49:47.375401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:14.778569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.778569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.229142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":509,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.894420+0000","last_change":"2024-04-02T04:51:33.177056+0000","last_active":"2024-04-02T04:52:30.894420+0000","last_peered":"2024-04-02T04:52:30.894420+0000","last_clean":"2024-04-02T04:52:30.894420+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:52:30.894420+0000","last_undegraded":"2024-04-02T04:52:30.894420+0000","last_fullsized":"2024-04-02T04:52:30.894420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:33.176943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:33.176943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:37.240411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.900556+0000","last_change":"2024-04-02T04:50:27.563922+0000","last_active":"2024-04-02T04:52:31.900556+0000","last_peered":"2024-04-02T04:52:31.900556+0000","last_clean":"2024-04-02T04:52:31.900556+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:52:31.900556+0000","last_undegraded":"2024-04-02T04:52:31.900556+0000","last_fullsized":"2024-04-02T04:52:31.900556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.563824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.563824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":362,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.894612+0000","last_change":"2024-04-02T04:49:26.508086+0000","last_active":"2024-04-02T04:52:31.894612+0000","last_peered":"2024-04-02T04:52:31.894612+0000","last_clean":"2024-04-02T04:52:31.894612+0000","last_became_active":"2024-04-02T04:49:26.507403+0000","last_became_peered":"2024-04-02T04:49:26.507403+0000","last_unstale":"2024-04-02T04:52:31.894612+0000","last_undegraded":"2024-04-02T04:52:31.894612+0000","last_fullsized":"2024-04-02T04:52:31.894612+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":365,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:32.442748+0000","last_change":"2024-04-02T04:49:47.517198+0000","last_active":"2024-04-02T04:52:32.442748+0000","last_peered":"2024-04-02T04:52:32.442748+0000","last_clean":"2024-04-02T04:49:45.381250+0000","last_became_active":"2024-04-02T04:49:47.517198+0000","last_became_peered":"2024-04-02T04:49:47.517198+0000","last_unstale":"2024-04-02T04:52:32.442748+0000","last_undegraded":"2024-04-02T04:52:32.442748+0000","last_fullsized":"2024-04-02T04:49:47.430351+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.392836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":557,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887118+0000","last_change":"2024-04-02T04:50:14.537273+0000","last_active":"2024-04-02T04:52:31.887118+0000","last_peered":"2024-04-02T04:52:31.887118+0000","last_clean":"2024-04-02T04:52:31.887118+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:52:31.887118+0000","last_undegraded":"2024-04-02T04:52:31.887118+0000","last_fullsized":"2024-04-02T04:52:31.887118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:50:14.537029+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.537029+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":517,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.887046+0000","last_change":"2024-04-02T04:49:47.468861+0000","last_active":"2024-04-02T04:52:31.887046+0000","last_peered":"2024-04-02T04:52:31.887046+0000","last_clean":"2024-04-02T04:49:45.338835+0000","last_became_active":"2024-04-02T04:49:47.468861+0000","last_became_peered":"2024-04-02T04:49:47.468861+0000","last_unstale":"2024-04-02T04:52:31.887046+0000","last_undegraded":"2024-04-02T04:52:31.887046+0000","last_fullsized":"2024-04-02T04:49:47.401513+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.329420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.329420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":506,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.892123+0000","last_change":"2024-04-02T04:50:29.283629+0000","last_active":"2024-04-02T04:52:31.892123+0000","last_peered":"2024-04-02T04:52:31.892123+0000","last_clean":"2024-04-02T04:52:31.892123+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:52:31.892123+0000","last_undegraded":"2024-04-02T04:52:31.892123+0000","last_fullsized":"2024-04-02T04:52:31.892123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.283405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":503,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.896880+0000","last_change":"2024-04-02T04:50:40.672009+0000","last_active":"2024-04-02T04:52:31.896880+0000","last_peered":"2024-04-02T04:52:31.896880+0000","last_clean":"2024-04-02T04:52:31.896880+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:52:31.896880+0000","last_undegraded":"2024-04-02T04:52:31.896880+0000","last_fullsized":"2024-04-02T04:52:31.896880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.671923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.671923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"138'2","reported_seq":367,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.886784+0000","last_change":"2024-04-02T04:49:47.455012+0000","last_active":"2024-04-02T04:52:31.886784+0000","last_peered":"2024-04-02T04:52:31.886784+0000","last_clean":"2024-04-02T04:49:45.373759+0000","last_became_active":"2024-04-02T04:49:47.455012+0000","last_became_peered":"2024-04-02T04:49:47.455012+0000","last_unstale":"2024-04-02T04:52:31.886784+0000","last_undegraded":"2024-04-02T04:52:31.886784+0000","last_fullsized":"2024-04-02T04:49:47.384636+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.447409+0000","last_change":"2024-04-02T04:50:39.321796+0000","last_active":"2024-04-02T04:52:32.447409+0000","last_peered":"2024-04-02T04:52:32.447409+0000","last_clean":"2024-04-02T04:52:32.447409+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:52:32.447409+0000","last_undegraded":"2024-04-02T04:52:32.447409+0000","last_fullsized":"2024-04-02T04:52:32.447409+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.321605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.621389+0000","last_change":"2024-04-02T04:50:13.554140+0000","last_active":"2024-04-02T04:52:32.621389+0000","last_peered":"2024-04-02T04:52:32.621389+0000","last_clean":"2024-04-02T04:52:32.621389+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:52:32.621389+0000","last_undegraded":"2024-04-02T04:52:32.621389+0000","last_fullsized":"2024-04-02T04:52:32.621389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:50:13.553801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:13.553801+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":335,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:32.620116+0000","last_change":"2024-04-02T04:49:47.435134+0000","last_active":"2024-04-02T04:52:32.620116+0000","last_peered":"2024-04-02T04:52:32.620116+0000","last_clean":"2024-04-02T04:43:54.216783+0000","last_became_active":"2024-04-02T04:49:47.435134+0000","last_became_peered":"2024-04-02T04:49:47.435134+0000","last_unstale":"2024-04-02T04:52:32.620116+0000","last_undegraded":"2024-04-02T04:49:47.360301+0000","last_fullsized":"2024-04-02T04:49:47.360179+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:48:48.436655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:48.436655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.638766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887777+0000","last_change":"2024-04-02T04:51:48.311838+0000","last_active":"2024-04-02T04:52:31.887777+0000","last_peered":"2024-04-02T04:52:31.887777+0000","last_clean":"2024-04-02T04:52:31.887777+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:52:31.887777+0000","last_undegraded":"2024-04-02T04:52:31.887777+0000","last_fullsized":"2024-04-02T04:52:31.887777+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.311616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.311616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:10.232371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":495,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.884301+0000","last_change":"2024-04-02T04:49:47.495495+0000","last_active":"2024-04-02T04:52:31.884301+0000","last_peered":"2024-04-02T04:52:31.884301+0000","last_clean":"2024-04-02T04:49:45.352563+0000","last_became_active":"2024-04-02T04:49:47.495495+0000","last_became_peered":"2024-04-02T04:49:47.495495+0000","last_unstale":"2024-04-02T04:52:31.884301+0000","last_undegraded":"2024-04-02T04:52:31.884301+0000","last_fullsized":"2024-04-02T04:49:47.378202+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"139'9","reported_seq":347,"reported_epoch":312,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:52:31.888002+0000","last_change":"2024-04-02T04:49:47.518710+0000","last_active":"2024-04-02T04:52:31.888002+0000","last_peered":"2024-04-02T04:52:31.888002+0000","last_clean":"2024-04-02T04:49:27.516310+0000","last_became_active":"2024-04-02T04:49:47.518710+0000","last_became_peered":"2024-04-02T04:49:47.518710+0000","last_unstale":"2024-04-02T04:52:31.888002+0000","last_undegraded":"2024-04-02T04:49:47.393557+0000","last_fullsized":"2024-04-02T04:49:47.393345+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":371,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.891658+0000","last_change":"2024-04-02T04:51:24.638019+0000","last_active":"2024-04-02T04:52:31.891658+0000","last_peered":"2024-04-02T04:52:31.891658+0000","last_clean":"2024-04-02T04:52:31.891658+0000","last_became_active":"2024-04-02T04:49:25.094609+0000","last_became_peered":"2024-04-02T04:49:25.094609+0000","last_unstale":"2024-04-02T04:52:31.891658+0000","last_undegraded":"2024-04-02T04:52:31.891658+0000","last_fullsized":"2024-04-02T04:52:31.891658+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:24.637853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:24.637853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:38.018793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":532,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.620880+0000","last_change":"2024-04-02T04:50:34.479206+0000","last_active":"2024-04-02T04:52:32.620880+0000","last_peered":"2024-04-02T04:52:32.620880+0000","last_clean":"2024-04-02T04:52:32.620880+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:52:32.620880+0000","last_undegraded":"2024-04-02T04:52:32.620880+0000","last_fullsized":"2024-04-02T04:52:32.620880+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:50:34.479089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.479089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+undersized","last_fresh":"2024-04-02T04:52:30.903610+0000","last_change":"2024-04-02T04:49:47.470693+0000","last_active":"2024-04-02T04:52:30.903610+0000","last_peered":"2024-04-02T04:52:30.903610+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.470693+0000","last_became_peered":"2024-04-02T04:49:47.470693+0000","last_unstale":"2024-04-02T04:52:30.903610+0000","last_undegraded":"2024-04-02T04:52:30.903610+0000","last_fullsized":"2024-04-02T04:49:47.411420+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:22.452700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:22.452700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888366+0000","last_change":"2024-04-02T04:50:09.464337+0000","last_active":"2024-04-02T04:52:31.888366+0000","last_peered":"2024-04-02T04:52:31.888366+0000","last_clean":"2024-04-02T04:52:31.888366+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:52:31.888366+0000","last_undegraded":"2024-04-02T04:52:31.888366+0000","last_fullsized":"2024-04-02T04:52:31.888366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.464222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.464222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.388407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":507,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.617456+0000","last_change":"2024-04-02T04:50:10.507552+0000","last_active":"2024-04-02T04:52:32.617456+0000","last_peered":"2024-04-02T04:52:32.617456+0000","last_clean":"2024-04-02T04:52:32.617456+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:52:32.617456+0000","last_undegraded":"2024-04-02T04:52:32.617456+0000","last_fullsized":"2024-04-02T04:52:32.617456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:10.507422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:10.507422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.639506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886898+0000","last_change":"2024-04-02T04:50:29.052309+0000","last_active":"2024-04-02T04:52:31.886898+0000","last_peered":"2024-04-02T04:52:31.886898+0000","last_clean":"2024-04-02T04:52:31.886898+0000","last_became_active":"2024-04-02T04:49:25.076275+0000","last_became_peered":"2024-04-02T04:49:25.076275+0000","last_unstale":"2024-04-02T04:52:31.886898+0000","last_undegraded":"2024-04-02T04:52:31.886898+0000","last_fullsized":"2024-04-02T04:52:31.886898+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.052221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":359,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-02T04:52:30.887198+0000","last_change":"2024-04-02T04:49:26.280288+0000","last_active":"2024-04-02T04:52:30.887198+0000","last_peered":"2024-04-02T04:52:30.887198+0000","last_clean":"2024-04-02T04:52:30.887198+0000","last_became_active":"2024-04-02T04:49:26.275375+0000","last_became_peered":"2024-04-02T04:49:26.275375+0000","last_unstale":"2024-04-02T04:52:30.887198+0000","last_undegraded":"2024-04-02T04:52:30.887198+0000","last_fullsized":"2024-04-02T04:52:30.887198+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.228817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.890264+0000","last_change":"2024-04-02T04:50:32.096624+0000","last_active":"2024-04-02T04:52:31.890264+0000","last_peered":"2024-04-02T04:52:31.890264+0000","last_clean":"2024-04-02T04:52:31.890264+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:52:31.890264+0000","last_undegraded":"2024-04-02T04:52:31.890264+0000","last_fullsized":"2024-04-02T04:52:31.890264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:50:32.096257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.096257+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247460+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":515,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.895885+0000","last_change":"2024-04-02T04:50:23.234439+0000","last_active":"2024-04-02T04:52:31.895885+0000","last_peered":"2024-04-02T04:52:31.895885+0000","last_clean":"2024-04-02T04:52:31.895885+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:52:31.895885+0000","last_undegraded":"2024-04-02T04:52:31.895885+0000","last_fullsized":"2024-04-02T04:52:31.895885+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:23.234223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:23.234223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":505,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.887661+0000","last_change":"2024-04-02T04:51:49.302781+0000","last_active":"2024-04-02T04:52:31.887661+0000","last_peered":"2024-04-02T04:52:31.887661+0000","last_clean":"2024-04-02T04:52:31.887661+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:52:31.887661+0000","last_undegraded":"2024-04-02T04:52:31.887661+0000","last_fullsized":"2024-04-02T04:52:31.887661+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.302549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.302549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:17.024873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":493,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.886567+0000","last_change":"2024-04-02T04:49:07.677962+0000","last_active":"2024-04-02T04:52:31.886567+0000","last_peered":"2024-04-02T04:52:31.886567+0000","last_clean":"2024-04-02T04:52:31.886567+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:52:31.886567+0000","last_undegraded":"2024-04-02T04:52:31.886567+0000","last_fullsized":"2024-04-02T04:52:31.886567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.677740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:36.387557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"139'7","reported_seq":377,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.442198+0000","last_change":"2024-04-02T04:51:38.237676+0000","last_active":"2024-04-02T04:52:32.442198+0000","last_peered":"2024-04-02T04:52:32.442198+0000","last_clean":"2024-04-02T04:52:32.442198+0000","last_became_active":"2024-04-02T04:49:26.241390+0000","last_became_peered":"2024-04-02T04:49:26.241390+0000","last_unstale":"2024-04-02T04:52:32.442198+0000","last_undegraded":"2024-04-02T04:52:32.442198+0000","last_fullsized":"2024-04-02T04:52:32.442198+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-04-02T04:51:38.237550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.237550+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:56.945794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.899659+0000","last_change":"2024-04-02T04:51:36.397355+0000","last_active":"2024-04-02T04:52:31.899659+0000","last_peered":"2024-04-02T04:52:31.899659+0000","last_clean":"2024-04-02T04:52:31.899659+0000","last_became_active":"2024-04-02T04:49:26.536918+0000","last_became_peered":"2024-04-02T04:49:26.536918+0000","last_unstale":"2024-04-02T04:52:31.899659+0000","last_undegraded":"2024-04-02T04:52:31.899659+0000","last_fullsized":"2024-04-02T04:52:31.899659+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.397213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.397213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:57.885856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.895981+0000","last_change":"2024-04-02T04:50:30.296523+0000","last_active":"2024-04-02T04:52:31.895981+0000","last_peered":"2024-04-02T04:52:31.895981+0000","last_clean":"2024-04-02T04:52:31.895981+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:52:31.895981+0000","last_undegraded":"2024-04-02T04:52:31.895981+0000","last_fullsized":"2024-04-02T04:52:31.895981+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:50:30.296198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:30.296198+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":508,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.890400+0000","last_change":"2024-04-02T04:49:47.432079+0000","last_active":"2024-04-02T04:52:31.890400+0000","last_peered":"2024-04-02T04:52:31.890400+0000","last_clean":"2024-04-02T04:49:45.381663+0000","last_became_active":"2024-04-02T04:49:47.432079+0000","last_became_peered":"2024-04-02T04:49:47.432079+0000","last_unstale":"2024-04-02T04:52:31.890400+0000","last_undegraded":"2024-04-02T04:52:31.890400+0000","last_fullsized":"2024-04-02T04:49:47.406616+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:32.247088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":506,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:31.888139+0000","last_change":"2024-04-02T04:50:20.218009+0000","last_active":"2024-04-02T04:52:31.888139+0000","last_peered":"2024-04-02T04:52:31.888139+0000","last_clean":"2024-04-02T04:52:31.888139+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:52:31.888139+0000","last_undegraded":"2024-04-02T04:52:31.888139+0000","last_fullsized":"2024-04-02T04:52:31.888139+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:20.217928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:20.217928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":497,"reported_epoch":312,"state":"active+undersized","last_fresh":"2024-04-02T04:52:31.886575+0000","last_change":"2024-04-02T04:49:47.454112+0000","last_active":"2024-04-02T04:52:31.886575+0000","last_peered":"2024-04-02T04:52:31.886575+0000","last_clean":"2024-04-02T04:49:45.335087+0000","last_became_active":"2024-04-02T04:49:47.454112+0000","last_became_peered":"2024-04-02T04:49:47.454112+0000","last_unstale":"2024-04-02T04:52:31.886575+0000","last_undegraded":"2024-04-02T04:52:31.886575+0000","last_fullsized":"2024-04-02T04:49:47.400802+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.389577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.389577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:34.677410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":370,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-04-02T04:52:32.443558+0000","last_change":"2024-04-02T04:50:28.261755+0000","last_active":"2024-04-02T04:52:32.443558+0000","last_peered":"2024-04-02T04:52:32.443558+0000","last_clean":"2024-04-02T04:52:32.443558+0000","last_became_active":"2024-04-02T04:49:26.269993+0000","last_became_peered":"2024-04-02T04:49:26.269993+0000","last_unstale":"2024-04-02T04:52:32.443558+0000","last_undegraded":"2024-04-02T04:52:32.443558+0000","last_fullsized":"2024-04-02T04:52:32.443558+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:28.261610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.261610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:52:33.394108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":259,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":258,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":257,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":4196932,"num_objects":262,"num_object_clones":142,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":2,"num_read":95,"num_read_kb":1,"num_write":539,"num_write_kb":8725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15015936,"data_stored":12588957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1502,"ondisk_log_size":1502,"up":87,"acting":87,"num_store_stats":7},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":87,"acting":87,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":86,"acting":86,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":646,"ondisk_log_size":646,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":495,"num_read_kb":490,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743504,"num_pgs":239,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111160,"kb_used_data":75820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93633992,"statfs":{"total":95995035648,"available":95881207808,"internally_reserved":0,"allocated":77639680,"data_stored":76942360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103293,"num_pgs":140,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86108,"kb_used_data":10508,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659044,"statfs":{"total":95995035648,"available":95906861056,"internally_reserved":0,"allocated":10760192,"data_stored":8100216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":144,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81220,"kb_used_data":9780,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93663932,"statfs":{"total":95995035648,"available":95911866368,"internally_reserved":0,"allocated":10014720,"data_stored":7510854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738554,"num_pgs":125,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77412,"kb_used_data":5964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93667740,"statfs":{"total":95995035648,"available":95915765760,"internally_reserved":0,"allocated":6107136,"data_stored":3915207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378790,"num_pgs":131,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81332,"kb_used_data":5732,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663820,"statfs":{"total":95995035648,"available":95911751680,"internally_reserved":0,"allocated":5869568,"data_stored":3313089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051740,"num_pgs":132,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80944,"kb_used_data":5336,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93664208,"statfs":{"total":95995035648,"available":95912148992,"internally_reserved":0,"allocated":5464064,"data_stored":3324488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822802,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67216,"kb_used_data":9648,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93677936,"statfs":{"total":95995035648,"available":95926206464,"internally_reserved":0,"allocated":9879552,"data_stored":7520599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463046,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58748,"kb_used_data":5356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93686404,"statfs":{"total":95995035648,"available":95934877696,"internally_reserved":0,"allocated":5484544,"data_stored":3319649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4673536,"data_stored":4195485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":1324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":4195017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4853760,"data_stored":4195943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:52:39.026 INFO:tasks.thrashosds.thrasher:Setting 41.a to [1, 4, 3] 2024-04-02T04:52:39.026 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '41.a', '1', '4', '3'] 2024-04-02T04:52:39.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 41.a 1 4 3 2024-04-02T04:52:39.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:39.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:39.251 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:39.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:52:39.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:39.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:39.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:39.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:39.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:39.763 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-02T04:52:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:39.881 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:39.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.473130+0000 osd.2 (osd.2) 149 : cluster 0 2.c scrub starts 2024-04-02T04:52:39.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.475453+0000 osd.2 (osd.2) 150 : cluster 0 2.c scrub ok 2024-04-02T04:52:39.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.659995+0000 osd.5 (osd.5) 195 : cluster 0 102.16 scrub starts 2024-04-02T04:52:39.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.664078+0000 osd.5 (osd.5) 196 : cluster 0 102.16 scrub ok 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.723207+0000 mon.a (mon.0) 2141 : cluster 3 Health check update: Degraded data redundancy: 278/1530 objects degraded (18.170%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.723281+0000 mon.a (mon.0) 2142 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.807792+0000 osd.3 (osd.3) 223 : cluster 0 43.1b scrub starts 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:38.809803+0000 osd.3 (osd.3) 224 : cluster 0 43.1b scrub ok 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.042370+0000 mon.c (mon.2) 537 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.160473+0000 mon.a (mon.0) 2143 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:39.164487+0000 mon.a (mon.0) 2144 : cluster 0 osdmap e319: 8 total, 7 up, 8 in 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.171127+0000 mon.c (mon.2) 538 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.171682+0000 mon.c (mon.2) 539 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.172821+0000 mon.a (mon.0) 2145 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.173538+0000 mon.a (mon.0) 2146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.176733+0000 mon.c (mon.2) 540 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: audit 2024-04-02T04:52:39.178307+0000 mon.a (mon.0) 2147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:39.364205+0000 osd.6 (osd.6) 151 : cluster 0 40.5 scrub starts 2024-04-02T04:52:39.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[21528]: cluster 2024-04-02T04:52:39.367112+0000 osd.6 (osd.6) 152 : cluster 0 40.5 scrub ok 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.473130+0000 osd.2 (osd.2) 149 : cluster 0 2.c scrub starts 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.475453+0000 osd.2 (osd.2) 150 : cluster 0 2.c scrub ok 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.659995+0000 osd.5 (osd.5) 195 : cluster 0 102.16 scrub starts 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.664078+0000 osd.5 (osd.5) 196 : cluster 0 102.16 scrub ok 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.723207+0000 mon.a (mon.0) 2141 : cluster 3 Health check update: Degraded data redundancy: 278/1530 objects degraded (18.170%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.723281+0000 mon.a (mon.0) 2142 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.807792+0000 osd.3 (osd.3) 223 : cluster 0 43.1b scrub starts 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:38.809803+0000 osd.3 (osd.3) 224 : cluster 0 43.1b scrub ok 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.042370+0000 mon.c (mon.2) 537 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.160473+0000 mon.a (mon.0) 2143 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:39.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:39.164487+0000 mon.a (mon.0) 2144 : cluster 0 osdmap e319: 8 total, 7 up, 8 in 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.171127+0000 mon.c (mon.2) 538 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.171682+0000 mon.c (mon.2) 539 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.172821+0000 mon.a (mon.0) 2145 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.173538+0000 mon.a (mon.0) 2146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.176733+0000 mon.c (mon.2) 540 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: audit 2024-04-02T04:52:39.178307+0000 mon.a (mon.0) 2147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:39.364205+0000 osd.6 (osd.6) 151 : cluster 0 40.5 scrub starts 2024-04-02T04:52:39.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:39 smithi069 bash[28904]: cluster 2024-04-02T04:52:39.367112+0000 osd.6 (osd.6) 152 : cluster 0 40.5 scrub ok 2024-04-02T04:52:39.904 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:39.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:40.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:40.018 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.473130+0000 osd.2 (osd.2) 149 : cluster 0 2.c scrub starts 2024-04-02T04:52:40.018 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.475453+0000 osd.2 (osd.2) 150 : cluster 0 2.c scrub ok 2024-04-02T04:52:40.018 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.659995+0000 osd.5 (osd.5) 195 : cluster 0 102.16 scrub starts 2024-04-02T04:52:40.018 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.664078+0000 osd.5 (osd.5) 196 : cluster 0 102.16 scrub ok 2024-04-02T04:52:40.018 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.723207+0000 mon.a (mon.0) 2141 : cluster 3 Health check update: Degraded data redundancy: 278/1530 objects degraded (18.170%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:40.018 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.723281+0000 mon.a (mon.0) 2142 : cluster 3 Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.807792+0000 osd.3 (osd.3) 223 : cluster 0 43.1b scrub starts 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:38.809803+0000 osd.3 (osd.3) 224 : cluster 0 43.1b scrub ok 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.042370+0000 mon.c (mon.2) 537 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.160473+0000 mon.a (mon.0) 2143 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:39.164487+0000 mon.a (mon.0) 2144 : cluster 0 osdmap e319: 8 total, 7 up, 8 in 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.171127+0000 mon.c (mon.2) 538 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.171682+0000 mon.c (mon.2) 539 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.172821+0000 mon.a (mon.0) 2145 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.173538+0000 mon.a (mon.0) 2146 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.176733+0000 mon.c (mon.2) 540 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: audit 2024-04-02T04:52:39.178307+0000 mon.a (mon.0) 2147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:39.364205+0000 osd.6 (osd.6) 151 : cluster 0 40.5 scrub starts 2024-04-02T04:52:40.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:39 smithi098 bash[23790]: cluster 2024-04-02T04:52:39.367112+0000 osd.6 (osd.6) 152 : cluster 0 40.5 scrub ok 2024-04-02T04:52:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:40.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:40.165 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:40.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:40.293 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:40.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:40.423 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:40.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:40.553 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:40.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:40.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:40.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:40.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:40.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:40.943 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:40.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:41.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:41.058 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.008943+0000 mgr.y (mgr.14538) 358 : cluster 0 pgmap v456: 356 pgs: 32 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 245 B/s wr, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:41.058 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.044059+0000 mon.c (mon.2) 541 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:41.058 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.166106+0000 mon.a (mon.0) 2148 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.166279+0000 mon.a (mon.0) 2149 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.166396+0000 mon.a (mon.0) 2150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]': finished 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.180626+0000 mon.c (mon.2) 542 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]: dispatch 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.180924+0000 mon.c (mon.2) 543 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.181175+0000 mon.c (mon.2) 544 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.181430+0000 mon.a (mon.0) 2151 : cluster 0 osdmap e320: 8 total, 7 up, 8 in 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.185186+0000 mon.a (mon.0) 2152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]: dispatch 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.185951+0000 mon.a (mon.0) 2153 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: audit 2024-04-02T04:52:40.186815+0000 mon.a (mon.0) 2154 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.349106+0000 osd.4 (osd.4) 195 : cluster 0 4.13 scrub starts 2024-04-02T04:52:41.059 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:40 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.351366+0000 osd.4 (osd.4) 196 : cluster 0 4.13 scrub ok 2024-04-02T04:52:41.073 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:41.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:41.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.008943+0000 mgr.y (mgr.14538) 358 : cluster 0 pgmap v456: 356 pgs: 32 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 245 B/s wr, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:41.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.044059+0000 mon.c (mon.2) 541 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:41.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.166106+0000 mon.a (mon.0) 2148 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.166279+0000 mon.a (mon.0) 2149 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.166396+0000 mon.a (mon.0) 2150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]': finished 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.180626+0000 mon.c (mon.2) 542 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]: dispatch 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.180924+0000 mon.c (mon.2) 543 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.181175+0000 mon.c (mon.2) 544 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.181430+0000 mon.a (mon.0) 2151 : cluster 0 osdmap e320: 8 total, 7 up, 8 in 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.185186+0000 mon.a (mon.0) 2152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]: dispatch 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.185951+0000 mon.a (mon.0) 2153 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: audit 2024-04-02T04:52:40.186815+0000 mon.a (mon.0) 2154 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.349106+0000 osd.4 (osd.4) 195 : cluster 0 4.13 scrub starts 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.351366+0000 osd.4 (osd.4) 196 : cluster 0 4.13 scrub ok 2024-04-02T04:52:41.192 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.008943+0000 mgr.y (mgr.14538) 358 : cluster 0 pgmap v456: 356 pgs: 32 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 245 B/s wr, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.044059+0000 mon.c (mon.2) 541 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.166106+0000 mon.a (mon.0) 2148 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.166279+0000 mon.a (mon.0) 2149 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.166396+0000 mon.a (mon.0) 2150 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-33"}]': finished 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.180626+0000 mon.c (mon.2) 542 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.180924+0000 mon.c (mon.2) 543 : audit 1 from='client.? 172.21.15.69:0/2861385427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.181175+0000 mon.c (mon.2) 544 : audit 1 from='client.? 172.21.15.69:0/1225121733' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.181430+0000 mon.a (mon.0) 2151 : cluster 0 osdmap e320: 8 total, 7 up, 8 in 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.185186+0000 mon.a (mon.0) 2152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.185951+0000 mon.a (mon.0) 2153 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: audit 2024-04-02T04:52:40.186815+0000 mon.a (mon.0) 2154 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]: dispatch 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.349106+0000 osd.4 (osd.4) 195 : cluster 0 4.13 scrub starts 2024-04-02T04:52:41.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:40 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.351366+0000 osd.4 (osd.4) 196 : cluster 0 4.13 scrub ok 2024-04-02T04:52:41.217 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:41.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:41.348 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:41.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:41.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:41.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:41.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:41.851 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.285122+0000 osd.0 (osd.0) 195 : cluster 0 102.7 scrub starts 2024-04-02T04:52:41.851 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.288501+0000 osd.0 (osd.0) 196 : cluster 0 102.7 scrub ok 2024-04-02T04:52:41.851 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.785207+0000 osd.3 (osd.3) 225 : cluster 0 43.10 scrub starts 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:40.787469+0000 osd.3 (osd.3) 226 : cluster 0 43.10 scrub ok 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.045592+0000 mon.c (mon.2) 545 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.167722+0000 mon.a (mon.0) 2155 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.170264+0000 mon.a (mon.0) 2156 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]': finished 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.170413+0000 mon.a (mon.0) 2157 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.170581+0000 mon.a (mon.0) 2158 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.179653+0000 mon.a (mon.0) 2159 : cluster 0 osdmap e321: 8 total, 7 up, 8 in 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.221407+0000 mon.a (mon.0) 2160 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.223177+0000 mon.a (mon.0) 2161 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.223638+0000 mon.a (mon.0) 2162 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-72153-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.226666+0000 mon.a (mon.0) 2163 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.227677+0000 mon.a (mon.0) 2164 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.228053+0000 mon.a (mon.0) 2165 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-72145-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:41.852 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.306736+0000 osd.1 (osd.1) 205 : cluster 0 4.2 scrub starts 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.308293+0000 osd.1 (osd.1) 206 : cluster 0 4.2 scrub ok 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.327015+0000 osd.0 (osd.0) 197 : cluster 0 102.2 scrub starts 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.329206+0000 osd.0 (osd.0) 198 : cluster 0 102.2 scrub ok 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.342422+0000 osd.4 (osd.4) 197 : cluster 0 6.1a scrub starts 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: cluster 2024-04-02T04:52:41.343659+0000 osd.4 (osd.4) 198 : cluster 0 6.1a scrub ok 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.385196+0000 mon.c (mon.2) 546 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:41.853 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:41 smithi098 bash[23790]: audit 2024-04-02T04:52:41.385685+0000 mon.a (mon.0) 2166 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:41.865 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:41.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:41.994 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:41.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.285122+0000 osd.0 (osd.0) 195 : cluster 0 102.7 scrub starts 2024-04-02T04:52:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.288501+0000 osd.0 (osd.0) 196 : cluster 0 102.7 scrub ok 2024-04-02T04:52:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.785207+0000 osd.3 (osd.3) 225 : cluster 0 43.10 scrub starts 2024-04-02T04:52:42.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:40.787469+0000 osd.3 (osd.3) 226 : cluster 0 43.10 scrub ok 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.045592+0000 mon.c (mon.2) 545 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.167722+0000 mon.a (mon.0) 2155 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.170264+0000 mon.a (mon.0) 2156 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]': finished 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.170413+0000 mon.a (mon.0) 2157 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.170581+0000 mon.a (mon.0) 2158 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.179653+0000 mon.a (mon.0) 2159 : cluster 0 osdmap e321: 8 total, 7 up, 8 in 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.221407+0000 mon.a (mon.0) 2160 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.223177+0000 mon.a (mon.0) 2161 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.223638+0000 mon.a (mon.0) 2162 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-72153-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.226666+0000 mon.a (mon.0) 2163 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.227677+0000 mon.a (mon.0) 2164 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.228053+0000 mon.a (mon.0) 2165 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-72145-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.306736+0000 osd.1 (osd.1) 205 : cluster 0 4.2 scrub starts 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.308293+0000 osd.1 (osd.1) 206 : cluster 0 4.2 scrub ok 2024-04-02T04:52:42.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.327015+0000 osd.0 (osd.0) 197 : cluster 0 102.2 scrub starts 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.329206+0000 osd.0 (osd.0) 198 : cluster 0 102.2 scrub ok 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.285122+0000 osd.0 (osd.0) 195 : cluster 0 102.7 scrub starts 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.288501+0000 osd.0 (osd.0) 196 : cluster 0 102.7 scrub ok 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.785207+0000 osd.3 (osd.3) 225 : cluster 0 43.10 scrub starts 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:40.787469+0000 osd.3 (osd.3) 226 : cluster 0 43.10 scrub ok 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.045592+0000 mon.c (mon.2) 545 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.167722+0000 mon.a (mon.0) 2155 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.170264+0000 mon.a (mon.0) 2156 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-33", "mode": "writeback"}]': finished 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.170413+0000 mon.a (mon.0) 2157 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-72145-39"}]': finished 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.170581+0000 mon.a (mon.0) 2158 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-72153-49"}]': finished 2024-04-02T04:52:42.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.179653+0000 mon.a (mon.0) 2159 : cluster 0 osdmap e321: 8 total, 7 up, 8 in 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.221407+0000 mon.a (mon.0) 2160 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.223177+0000 mon.a (mon.0) 2161 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.223638+0000 mon.a (mon.0) 2162 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-72153-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.226666+0000 mon.a (mon.0) 2163 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.227677+0000 mon.a (mon.0) 2164 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.228053+0000 mon.a (mon.0) 2165 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-72145-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.306736+0000 osd.1 (osd.1) 205 : cluster 0 4.2 scrub starts 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.308293+0000 osd.1 (osd.1) 206 : cluster 0 4.2 scrub ok 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.327015+0000 osd.0 (osd.0) 197 : cluster 0 102.2 scrub starts 2024-04-02T04:52:42.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.329206+0000 osd.0 (osd.0) 198 : cluster 0 102.2 scrub ok 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.342422+0000 osd.4 (osd.4) 197 : cluster 0 6.1a scrub starts 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: cluster 2024-04-02T04:52:41.343659+0000 osd.4 (osd.4) 198 : cluster 0 6.1a scrub ok 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.385196+0000 mon.c (mon.2) 546 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[21528]: audit 2024-04-02T04:52:41.385685+0000 mon.a (mon.0) 2166 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.342422+0000 osd.4 (osd.4) 197 : cluster 0 6.1a scrub starts 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: cluster 2024-04-02T04:52:41.343659+0000 osd.4 (osd.4) 198 : cluster 0 6.1a scrub ok 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.385196+0000 mon.c (mon.2) 546 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:42.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:41 smithi069 bash[28904]: audit 2024-04-02T04:52:41.385685+0000 mon.a (mon.0) 2166 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:42.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:42.124 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:42.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:42.253 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:42.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:42.379 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:42.482 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:42.507 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:42.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:42.638 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:42.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:42.803 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:42.803 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:42.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:42.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:43.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:43.028 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:43.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:43.142 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.010631+0000 mgr.y (mgr.14538) 359 : cluster 0 pgmap v459: 356 pgs: 32 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:43.142 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.047356+0000 mon.c (mon.2) 547 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.172773+0000 mon.a (mon.0) 2167 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-72153-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.172973+0000 mon.a (mon.0) 2168 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-72145-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.173159+0000 mon.a (mon.0) 2169 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.181621+0000 mon.a (mon.0) 2170 : cluster 0 osdmap e322: 8 total, 7 up, 8 in 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.183608+0000 mon.c (mon.2) 548 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.186043+0000 mon.a (mon.0) 2171 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-72153-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.186838+0000 mon.a (mon.0) 2172 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-72145-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: audit 2024-04-02T04:52:42.187356+0000 mon.a (mon.0) 2173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.265070+0000 osd.1 (osd.1) 207 : cluster 0 10.2 scrub starts 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.268940+0000 osd.1 (osd.1) 208 : cluster 0 10.2 scrub ok 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.313117+0000 osd.4 (osd.4) 199 : cluster 0 40.9 scrub starts 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.318789+0000 osd.4 (osd.4) 200 : cluster 0 40.9 scrub ok 2024-04-02T04:52:43.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.010631+0000 mgr.y (mgr.14538) 359 : cluster 0 pgmap v459: 356 pgs: 32 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.047356+0000 mon.c (mon.2) 547 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.172773+0000 mon.a (mon.0) 2167 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-72153-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.172973+0000 mon.a (mon.0) 2168 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-72145-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.173159+0000 mon.a (mon.0) 2169 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.181621+0000 mon.a (mon.0) 2170 : cluster 0 osdmap e322: 8 total, 7 up, 8 in 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.183608+0000 mon.c (mon.2) 548 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.186043+0000 mon.a (mon.0) 2171 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-72153-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.186838+0000 mon.a (mon.0) 2172 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-72145-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: audit 2024-04-02T04:52:42.187356+0000 mon.a (mon.0) 2173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.265070+0000 osd.1 (osd.1) 207 : cluster 0 10.2 scrub starts 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.268940+0000 osd.1 (osd.1) 208 : cluster 0 10.2 scrub ok 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.313117+0000 osd.4 (osd.4) 199 : cluster 0 40.9 scrub starts 2024-04-02T04:52:43.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:42 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.318789+0000 osd.4 (osd.4) 200 : cluster 0 40.9 scrub ok 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.010631+0000 mgr.y (mgr.14538) 359 : cluster 0 pgmap v459: 356 pgs: 32 creating+peering, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.047356+0000 mon.c (mon.2) 547 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.172773+0000 mon.a (mon.0) 2167 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-72153-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.172973+0000 mon.a (mon.0) 2168 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-72145-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.173159+0000 mon.a (mon.0) 2169 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.181621+0000 mon.a (mon.0) 2170 : cluster 0 osdmap e322: 8 total, 7 up, 8 in 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.183608+0000 mon.c (mon.2) 548 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.186043+0000 mon.a (mon.0) 2171 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-72153-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.186838+0000 mon.a (mon.0) 2172 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-72145-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: audit 2024-04-02T04:52:42.187356+0000 mon.a (mon.0) 2173 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]: dispatch 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.265070+0000 osd.1 (osd.1) 207 : cluster 0 10.2 scrub starts 2024-04-02T04:52:43.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.268940+0000 osd.1 (osd.1) 208 : cluster 0 10.2 scrub ok 2024-04-02T04:52:43.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.313117+0000 osd.4 (osd.4) 199 : cluster 0 40.9 scrub starts 2024-04-02T04:52:43.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:42 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.318789+0000 osd.4 (osd.4) 200 : cluster 0 40.9 scrub ok 2024-04-02T04:52:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:43.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:43.285 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:43.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:43.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:52:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:52:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:43.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:43.543 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:43.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:43.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:52:43.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:43.924 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:43.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:44.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:44.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:44.168 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:44 smithi098 bash[23790]: audit 2024-04-02T04:52:43.048976+0000 mon.c (mon.2) 549 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:44.168 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:44 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.173642+0000 mon.a (mon.0) 2174 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:44.168 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:44 smithi098 bash[23790]: audit 2024-04-02T04:52:43.175456+0000 mon.a (mon.0) 2175 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]': finished 2024-04-02T04:52:44.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:44.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:44 smithi069 bash[28904]: audit 2024-04-02T04:52:43.048976+0000 mon.c (mon.2) 549 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:44.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:44 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.173642+0000 mon.a (mon.0) 2174 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:44.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:44 smithi069 bash[28904]: audit 2024-04-02T04:52:43.175456+0000 mon.a (mon.0) 2175 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]': finished 2024-04-02T04:52:44.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:44 smithi069 bash[21528]: audit 2024-04-02T04:52:43.048976+0000 mon.c (mon.2) 549 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:44.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:44 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.173642+0000 mon.a (mon.0) 2174 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:44.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:44 smithi069 bash[21528]: audit 2024-04-02T04:52:43.175456+0000 mon.a (mon.0) 2175 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-33"}]': finished 2024-04-02T04:52:44.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:44.318 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:44.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:44.447 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:44.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:44.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:44.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:44.707 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:44.809 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:44.837 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:44.948 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:44.970 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:44.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:45.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.099 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:45.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.228 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:45.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.438626+0000 osd.6 (osd.6) 153 : cluster 0 10.0 scrub starts 2024-04-02T04:52:45.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:42.440843+0000 osd.6 (osd.6) 154 : cluster 0 10.0 scrub ok 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:43.151842+0000 mgr.y (mgr.14538) 360 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.273477+0000 osd.4 (osd.4) 201 : cluster 0 43.1a scrub starts 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.275412+0000 osd.4 (osd.4) 202 : cluster 0 43.1a scrub ok 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.423040+0000 osd.6 (osd.6) 155 : cluster 0 13.1c scrub starts 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.424595+0000 osd.6 (osd.6) 156 : cluster 0 13.1c scrub ok 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.649234+0000 osd.5 (osd.5) 197 : cluster 0 102.a scrub starts 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:43.653319+0000 osd.5 (osd.5) 198 : cluster 0 102.a scrub ok 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.012503+0000 mgr.y (mgr.14538) 361 : cluster 0 pgmap v462: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:44.050232+0000 mon.c (mon.2) 550 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.080161+0000 mon.a (mon.0) 2176 : cluster 0 osdmap e323: 8 total, 7 up, 8 in 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:44.177448+0000 mon.a (mon.0) 2177 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-72153-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:44.177525+0000 mon.a (mon.0) 2178 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-72145-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.183308+0000 mon.a (mon.0) 2179 : cluster 0 osdmap e324: 8 total, 7 up, 8 in 2024-04-02T04:52:45.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.250562+0000 osd.1 (osd.1) 209 : cluster 0 40.1 scrub starts 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.252096+0000 osd.1 (osd.1) 210 : cluster 0 40.1 scrub ok 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.826148+0000 mon.a (mon.0) 2180 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.834683+0000 mon.a (mon.0) 2181 : cluster 0 osdmap e325: 8 total, 7 up, 8 in 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:44.836699+0000 mon.c (mon.2) 551 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:44.863551+0000 mon.a (mon.0) 2182 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[21528]: audit 2024-04-02T04:52:45.051848+0000 mon.c (mon.2) 552 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.438626+0000 osd.6 (osd.6) 153 : cluster 0 10.0 scrub starts 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:42.440843+0000 osd.6 (osd.6) 154 : cluster 0 10.0 scrub ok 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:43.151842+0000 mgr.y (mgr.14538) 360 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.273477+0000 osd.4 (osd.4) 201 : cluster 0 43.1a scrub starts 2024-04-02T04:52:45.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.275412+0000 osd.4 (osd.4) 202 : cluster 0 43.1a scrub ok 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.423040+0000 osd.6 (osd.6) 155 : cluster 0 13.1c scrub starts 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.424595+0000 osd.6 (osd.6) 156 : cluster 0 13.1c scrub ok 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.649234+0000 osd.5 (osd.5) 197 : cluster 0 102.a scrub starts 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:43.653319+0000 osd.5 (osd.5) 198 : cluster 0 102.a scrub ok 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.012503+0000 mgr.y (mgr.14538) 361 : cluster 0 pgmap v462: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:44.050232+0000 mon.c (mon.2) 550 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.080161+0000 mon.a (mon.0) 2176 : cluster 0 osdmap e323: 8 total, 7 up, 8 in 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:44.177448+0000 mon.a (mon.0) 2177 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-72153-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:44.177525+0000 mon.a (mon.0) 2178 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-72145-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.183308+0000 mon.a (mon.0) 2179 : cluster 0 osdmap e324: 8 total, 7 up, 8 in 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.250562+0000 osd.1 (osd.1) 209 : cluster 0 40.1 scrub starts 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.252096+0000 osd.1 (osd.1) 210 : cluster 0 40.1 scrub ok 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.826148+0000 mon.a (mon.0) 2180 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.834683+0000 mon.a (mon.0) 2181 : cluster 0 osdmap e325: 8 total, 7 up, 8 in 2024-04-02T04:52:45.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:44.836699+0000 mon.c (mon.2) 551 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:45.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:44.863551+0000 mon.a (mon.0) 2182 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:45.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:45 smithi069 bash[28904]: audit 2024-04-02T04:52:45.051848+0000 mon.c (mon.2) 552 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:45.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.438626+0000 osd.6 (osd.6) 153 : cluster 0 10.0 scrub starts 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:42.440843+0000 osd.6 (osd.6) 154 : cluster 0 10.0 scrub ok 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:43.151842+0000 mgr.y (mgr.14538) 360 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.273477+0000 osd.4 (osd.4) 201 : cluster 0 43.1a scrub starts 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.275412+0000 osd.4 (osd.4) 202 : cluster 0 43.1a scrub ok 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.423040+0000 osd.6 (osd.6) 155 : cluster 0 13.1c scrub starts 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.424595+0000 osd.6 (osd.6) 156 : cluster 0 13.1c scrub ok 2024-04-02T04:52:45.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.649234+0000 osd.5 (osd.5) 197 : cluster 0 102.a scrub starts 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:43.653319+0000 osd.5 (osd.5) 198 : cluster 0 102.a scrub ok 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.012503+0000 mgr.y (mgr.14538) 361 : cluster 0 pgmap v462: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:44.050232+0000 mon.c (mon.2) 550 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.080161+0000 mon.a (mon.0) 2176 : cluster 0 osdmap e323: 8 total, 7 up, 8 in 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:44.177448+0000 mon.a (mon.0) 2177 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-72153-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:44.177525+0000 mon.a (mon.0) 2178 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-72145-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.183308+0000 mon.a (mon.0) 2179 : cluster 0 osdmap e324: 8 total, 7 up, 8 in 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.250562+0000 osd.1 (osd.1) 209 : cluster 0 40.1 scrub starts 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.252096+0000 osd.1 (osd.1) 210 : cluster 0 40.1 scrub ok 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.826148+0000 mon.a (mon.0) 2180 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.834683+0000 mon.a (mon.0) 2181 : cluster 0 osdmap e325: 8 total, 7 up, 8 in 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:44.836699+0000 mon.c (mon.2) 551 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:44.863551+0000 mon.a (mon.0) 2182 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:45.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:45 smithi098 bash[23790]: audit 2024-04-02T04:52:45.051848+0000 mon.c (mon.2) 552 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:45.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.488 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:45.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:45.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:45.748 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:45.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:45.886 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:45.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:45.992 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:46.020 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:46.123 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:46.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:46.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.774053+0000 osd.3 (osd.3) 227 : cluster 0 43.14 scrub starts 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: cluster 2024-04-02T04:52:44.775668+0000 osd.3 (osd.3) 228 : cluster 0 43.14 scrub ok 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: cluster 2024-04-02T04:52:45.219231+0000 osd.1 (osd.1) 211 : cluster 0 41.10 scrub starts 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: cluster 2024-04-02T04:52:45.221438+0000 osd.1 (osd.1) 212 : cluster 0 41.10 scrub ok 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: audit 2024-04-02T04:52:45.832551+0000 mon.a (mon.0) 2183 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: cluster 2024-04-02T04:52:45.840920+0000 mon.a (mon.0) 2184 : cluster 0 osdmap e326: 8 total, 7 up, 8 in 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: audit 2024-04-02T04:52:45.846589+0000 mon.a (mon.0) 2185 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: audit 2024-04-02T04:52:45.847232+0000 mon.a (mon.0) 2186 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: audit 2024-04-02T04:52:45.857433+0000 mon.c (mon.2) 553 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: audit 2024-04-02T04:52:45.858795+0000 mon.a (mon.0) 2187 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:46.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[21528]: audit 2024-04-02T04:52:46.053330+0000 mon.c (mon.2) 554 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.774053+0000 osd.3 (osd.3) 227 : cluster 0 43.14 scrub starts 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: cluster 2024-04-02T04:52:44.775668+0000 osd.3 (osd.3) 228 : cluster 0 43.14 scrub ok 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: cluster 2024-04-02T04:52:45.219231+0000 osd.1 (osd.1) 211 : cluster 0 41.10 scrub starts 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: cluster 2024-04-02T04:52:45.221438+0000 osd.1 (osd.1) 212 : cluster 0 41.10 scrub ok 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: audit 2024-04-02T04:52:45.832551+0000 mon.a (mon.0) 2183 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: cluster 2024-04-02T04:52:45.840920+0000 mon.a (mon.0) 2184 : cluster 0 osdmap e326: 8 total, 7 up, 8 in 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: audit 2024-04-02T04:52:45.846589+0000 mon.a (mon.0) 2185 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: audit 2024-04-02T04:52:45.847232+0000 mon.a (mon.0) 2186 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: audit 2024-04-02T04:52:45.857433+0000 mon.c (mon.2) 553 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:46.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: audit 2024-04-02T04:52:45.858795+0000 mon.a (mon.0) 2187 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:46.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:46 smithi069 bash[28904]: audit 2024-04-02T04:52:46.053330+0000 mon.c (mon.2) 554 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:46.281 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:46.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.774053+0000 osd.3 (osd.3) 227 : cluster 0 43.14 scrub starts 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: cluster 2024-04-02T04:52:44.775668+0000 osd.3 (osd.3) 228 : cluster 0 43.14 scrub ok 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: cluster 2024-04-02T04:52:45.219231+0000 osd.1 (osd.1) 211 : cluster 0 41.10 scrub starts 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: cluster 2024-04-02T04:52:45.221438+0000 osd.1 (osd.1) 212 : cluster 0 41.10 scrub ok 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: audit 2024-04-02T04:52:45.832551+0000 mon.a (mon.0) 2183 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: cluster 2024-04-02T04:52:45.840920+0000 mon.a (mon.0) 2184 : cluster 0 osdmap e326: 8 total, 7 up, 8 in 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: audit 2024-04-02T04:52:45.846589+0000 mon.a (mon.0) 2185 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: audit 2024-04-02T04:52:45.847232+0000 mon.a (mon.0) 2186 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:46.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: audit 2024-04-02T04:52:45.857433+0000 mon.c (mon.2) 553 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: audit 2024-04-02T04:52:45.858795+0000 mon.a (mon.0) 2187 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:46 smithi098 bash[23790]: audit 2024-04-02T04:52:46.053330+0000 mon.c (mon.2) 554 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:46.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:46.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:46.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:46.668 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:46.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:46.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.918 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:46.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:47.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:47.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:47.165 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:47.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:47.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.014572+0000 mgr.y (mgr.14538) 362 : cluster 0 pgmap v466: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 B/s wr, 2 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:47.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.332328+0000 osd.4 (osd.4) 203 : cluster 0 40.6 scrub starts 2024-04-02T04:52:47.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.334935+0000 osd.4 (osd.4) 204 : cluster 0 40.6 scrub ok 2024-04-02T04:52:47.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.568465+0000 osd.2 (osd.2) 151 : cluster 0 2.1 scrub starts 2024-04-02T04:52:47.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.569556+0000 osd.2 (osd.2) 152 : cluster 0 2.1 scrub ok 2024-04-02T04:52:47.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.842467+0000 mon.a (mon.0) 2188 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:47.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.842543+0000 mon.a (mon.0) 2189 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:47.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.842616+0000 mon.a (mon.0) 2190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.846924+0000 mon.a (mon.0) 2191 : cluster 0 osdmap e327: 8 total, 7 up, 8 in 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.848110+0000 mon.c (mon.2) 555 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.849098+0000 mon.a (mon.0) 2192 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.850032+0000 mon.a (mon.0) 2193 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:46.852358+0000 mon.a (mon.0) 2194 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[21528]: audit 2024-04-02T04:52:47.054483+0000 mon.c (mon.2) 556 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:47.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.014572+0000 mgr.y (mgr.14538) 362 : cluster 0 pgmap v466: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 B/s wr, 2 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.332328+0000 osd.4 (osd.4) 203 : cluster 0 40.6 scrub starts 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.334935+0000 osd.4 (osd.4) 204 : cluster 0 40.6 scrub ok 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.568465+0000 osd.2 (osd.2) 151 : cluster 0 2.1 scrub starts 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.569556+0000 osd.2 (osd.2) 152 : cluster 0 2.1 scrub ok 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.842467+0000 mon.a (mon.0) 2188 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.842543+0000 mon.a (mon.0) 2189 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.842616+0000 mon.a (mon.0) 2190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.846924+0000 mon.a (mon.0) 2191 : cluster 0 osdmap e327: 8 total, 7 up, 8 in 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.848110+0000 mon.c (mon.2) 555 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.849098+0000 mon.a (mon.0) 2192 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.850032+0000 mon.a (mon.0) 2193 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:46.852358+0000 mon.a (mon.0) 2194 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:47.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:47 smithi069 bash[28904]: audit 2024-04-02T04:52:47.054483+0000 mon.c (mon.2) 556 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:47.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:47.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:47.414 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:47.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.014572+0000 mgr.y (mgr.14538) 362 : cluster 0 pgmap v466: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 B/s wr, 2 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:47.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.332328+0000 osd.4 (osd.4) 203 : cluster 0 40.6 scrub starts 2024-04-02T04:52:47.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.334935+0000 osd.4 (osd.4) 204 : cluster 0 40.6 scrub ok 2024-04-02T04:52:47.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.568465+0000 osd.2 (osd.2) 151 : cluster 0 2.1 scrub starts 2024-04-02T04:52:47.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.569556+0000 osd.2 (osd.2) 152 : cluster 0 2.1 scrub ok 2024-04-02T04:52:47.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.842467+0000 mon.a (mon.0) 2188 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.842543+0000 mon.a (mon.0) 2189 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.842616+0000 mon.a (mon.0) 2190 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.846924+0000 mon.a (mon.0) 2191 : cluster 0 osdmap e327: 8 total, 7 up, 8 in 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.848110+0000 mon.c (mon.2) 555 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.849098+0000 mon.a (mon.0) 2192 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]: dispatch 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.850032+0000 mon.a (mon.0) 2193 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]: dispatch 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:46.852358+0000 mon.a (mon.0) 2194 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:47 smithi098 bash[23790]: audit 2024-04-02T04:52:47.054483+0000 mon.c (mon.2) 556 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:47.516 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:47.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:47.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:47.799 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:47.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:47.925 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:47.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:48.027 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:48.051 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:48.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:48.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:48.312 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:48.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.684780+0000 osd.5 (osd.5) 199 : cluster 0 6.a scrub starts 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: cluster 2024-04-02T04:52:46.685929+0000 osd.5 (osd.5) 200 : cluster 0 6.a scrub ok 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.750168+0000 mon.b (mon.1) 200 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.845025+0000 mon.a (mon.0) 2195 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.845283+0000 mon.a (mon.0) 2196 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.845456+0000 mon.a (mon.0) 2197 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.850876+0000 mon.c (mon.2) 557 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]: dispatch 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: cluster 2024-04-02T04:52:47.852201+0000 mon.a (mon.0) 2198 : cluster 0 osdmap e328: 8 total, 7 up, 8 in 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.853020+0000 mon.a (mon.0) 2199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]: dispatch 2024-04-02T04:52:48.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.873339+0000 mon.a (mon.0) 2200 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.874437+0000 mon.a (mon.0) 2201 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:48 smithi098 bash[23790]: audit 2024-04-02T04:52:47.874863+0000 mon.a (mon.0) 2202 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-72145-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:48.476 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:48.578 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:48.589 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.684780+0000 osd.5 (osd.5) 199 : cluster 0 6.a scrub starts 2024-04-02T04:52:48.589 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: cluster 2024-04-02T04:52:46.685929+0000 osd.5 (osd.5) 200 : cluster 0 6.a scrub ok 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.750168+0000 mon.b (mon.1) 200 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.845025+0000 mon.a (mon.0) 2195 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.845283+0000 mon.a (mon.0) 2196 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.845456+0000 mon.a (mon.0) 2197 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.850876+0000 mon.c (mon.2) 557 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]: dispatch 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: cluster 2024-04-02T04:52:47.852201+0000 mon.a (mon.0) 2198 : cluster 0 osdmap e328: 8 total, 7 up, 8 in 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.853020+0000 mon.a (mon.0) 2199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]: dispatch 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.873339+0000 mon.a (mon.0) 2200 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.874437+0000 mon.a (mon.0) 2201 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21528]: audit 2024-04-02T04:52:47.874863+0000 mon.a (mon.0) 2202 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-72145-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:48.590 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.684780+0000 osd.5 (osd.5) 199 : cluster 0 6.a scrub starts 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: cluster 2024-04-02T04:52:46.685929+0000 osd.5 (osd.5) 200 : cluster 0 6.a scrub ok 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.750168+0000 mon.b (mon.1) 200 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.845025+0000 mon.a (mon.0) 2195 : audit 1 from='client.? 172.21.15.69:0/845592075' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-72145-40"}]': finished 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.845283+0000 mon.a (mon.0) 2196 : audit 1 from='client.? 172.21.15.69:0/1129903165' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-72153-50"}]': finished 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.845456+0000 mon.a (mon.0) 2197 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.850876+0000 mon.c (mon.2) 557 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]: dispatch 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: cluster 2024-04-02T04:52:47.852201+0000 mon.a (mon.0) 2198 : cluster 0 osdmap e328: 8 total, 7 up, 8 in 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.853020+0000 mon.a (mon.0) 2199 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]: dispatch 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.873339+0000 mon.a (mon.0) 2200 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.874437+0000 mon.a (mon.0) 2201 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:48.591 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[28904]: audit 2024-04-02T04:52:47.874863+0000 mon.a (mon.0) 2202 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-72145-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:48.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:48.777 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:52:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:52:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:52:48.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:48.876 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:48.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:48.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:49.003 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:49.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:49.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:49.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:49.263 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:49.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:49.393 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:49.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:49.524 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:47.727851+0000 osd.5 (osd.5) 201 : cluster 0 6.19 scrub starts 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:47.730628+0000 osd.5 (osd.5) 202 : cluster 0 6.19 scrub ok 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:47.738988+0000 osd.3 (osd.3) 229 : cluster 0 6.13 scrub starts 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:47.741738+0000 osd.3 (osd.3) 230 : cluster 0 6.13 scrub ok 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.016226+0000 mgr.y (mgr.14538) 363 : cluster 0 pgmap v469: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:48.055514+0000 mon.c (mon.2) 558 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.217687+0000 osd.1 (osd.1) 213 : cluster 0 102.1b scrub starts 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.221387+0000 osd.1 (osd.1) 214 : cluster 0 102.1b scrub ok 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.598066+0000 osd.2 (osd.2) 153 : cluster 0 41.f scrub starts 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.599588+0000 osd.2 (osd.2) 154 : cluster 0 41.f scrub ok 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.724295+0000 osd.5 (osd.5) 203 : cluster 0 3.1d scrub starts 2024-04-02T04:52:49.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.727577+0000 osd.5 (osd.5) 204 : cluster 0 3.1d scrub ok 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.846321+0000 mon.a (mon.0) 2203 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:48.849209+0000 mon.a (mon.0) 2204 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]': finished 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:48.849488+0000 mon.a (mon.0) 2205 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-72145-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: cluster 2024-04-02T04:52:48.855518+0000 mon.a (mon.0) 2206 : cluster 0 osdmap e329: 8 total, 7 up, 8 in 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:48.858043+0000 mon.c (mon.2) 559 : audit 1 from='client.? 172.21.15.69:0/225957613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:48.860442+0000 mon.a (mon.0) 2207 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-72145-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:48.862417+0000 mon.a (mon.0) 2208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[21528]: audit 2024-04-02T04:52:49.056527+0000 mon.c (mon.2) 560 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:47.727851+0000 osd.5 (osd.5) 201 : cluster 0 6.19 scrub starts 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:47.730628+0000 osd.5 (osd.5) 202 : cluster 0 6.19 scrub ok 2024-04-02T04:52:49.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:47.738988+0000 osd.3 (osd.3) 229 : cluster 0 6.13 scrub starts 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:47.741738+0000 osd.3 (osd.3) 230 : cluster 0 6.13 scrub ok 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.016226+0000 mgr.y (mgr.14538) 363 : cluster 0 pgmap v469: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:48.055514+0000 mon.c (mon.2) 558 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.217687+0000 osd.1 (osd.1) 213 : cluster 0 102.1b scrub starts 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.221387+0000 osd.1 (osd.1) 214 : cluster 0 102.1b scrub ok 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.598066+0000 osd.2 (osd.2) 153 : cluster 0 41.f scrub starts 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.599588+0000 osd.2 (osd.2) 154 : cluster 0 41.f scrub ok 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.724295+0000 osd.5 (osd.5) 203 : cluster 0 3.1d scrub starts 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.727577+0000 osd.5 (osd.5) 204 : cluster 0 3.1d scrub ok 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.846321+0000 mon.a (mon.0) 2203 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:48.849209+0000 mon.a (mon.0) 2204 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]': finished 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:48.849488+0000 mon.a (mon.0) 2205 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-72145-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:49.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: cluster 2024-04-02T04:52:48.855518+0000 mon.a (mon.0) 2206 : cluster 0 osdmap e329: 8 total, 7 up, 8 in 2024-04-02T04:52:49.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:48.858043+0000 mon.c (mon.2) 559 : audit 1 from='client.? 172.21.15.69:0/225957613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:49.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:48.860442+0000 mon.a (mon.0) 2207 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-72145-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:49.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:48.862417+0000 mon.a (mon.0) 2208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:49.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:49 smithi069 bash[28904]: audit 2024-04-02T04:52:49.056527+0000 mon.c (mon.2) 560 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:49.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:49.653 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:49.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:47.727851+0000 osd.5 (osd.5) 201 : cluster 0 6.19 scrub starts 2024-04-02T04:52:49.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:47.730628+0000 osd.5 (osd.5) 202 : cluster 0 6.19 scrub ok 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:47.738988+0000 osd.3 (osd.3) 229 : cluster 0 6.13 scrub starts 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:47.741738+0000 osd.3 (osd.3) 230 : cluster 0 6.13 scrub ok 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.016226+0000 mgr.y (mgr.14538) 363 : cluster 0 pgmap v469: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:48.055514+0000 mon.c (mon.2) 558 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.217687+0000 osd.1 (osd.1) 213 : cluster 0 102.1b scrub starts 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.221387+0000 osd.1 (osd.1) 214 : cluster 0 102.1b scrub ok 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.598066+0000 osd.2 (osd.2) 153 : cluster 0 41.f scrub starts 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.599588+0000 osd.2 (osd.2) 154 : cluster 0 41.f scrub ok 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.724295+0000 osd.5 (osd.5) 203 : cluster 0 3.1d scrub starts 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.727577+0000 osd.5 (osd.5) 204 : cluster 0 3.1d scrub ok 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.846321+0000 mon.a (mon.0) 2203 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:48.849209+0000 mon.a (mon.0) 2204 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-35", "mode": "writeback"}]': finished 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:48.849488+0000 mon.a (mon.0) 2205 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-72145-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: cluster 2024-04-02T04:52:48.855518+0000 mon.a (mon.0) 2206 : cluster 0 osdmap e329: 8 total, 7 up, 8 in 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:48.858043+0000 mon.c (mon.2) 559 : audit 1 from='client.? 172.21.15.69:0/225957613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:48.860442+0000 mon.a (mon.0) 2207 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-72145-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:49.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:48.862417+0000 mon.a (mon.0) 2208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:49.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:49 smithi098 bash[23790]: audit 2024-04-02T04:52:49.056527+0000 mon.c (mon.2) 560 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:49.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:49.784 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:49.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:49.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:50.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:50.039 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:50.142 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:50.161 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:50.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:50.293 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:50.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.175396+0000 osd.1 (osd.1) 215 : cluster 0 3.0 scrub starts 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.180510+0000 osd.1 (osd.1) 216 : cluster 0 3.0 scrub ok 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.241969+0000 osd.4 (osd.4) 205 : cluster 0 13.15 scrub starts 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.244249+0000 osd.4 (osd.4) 206 : cluster 0 13.15 scrub ok 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.308928+0000 osd.0 (osd.0) 199 : cluster 0 4.6 scrub starts 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.310934+0000 osd.0 (osd.0) 200 : cluster 0 4.6 scrub ok 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: audit 2024-04-02T04:52:49.853159+0000 mon.a (mon.0) 2209 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[21528]: cluster 2024-04-02T04:52:50.330535+0000 mon.a (mon.0) 2210 : cluster 0 osdmap e330: 8 total, 7 up, 8 in 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.175396+0000 osd.1 (osd.1) 215 : cluster 0 3.0 scrub starts 2024-04-02T04:52:50.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.180510+0000 osd.1 (osd.1) 216 : cluster 0 3.0 scrub ok 2024-04-02T04:52:50.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.241969+0000 osd.4 (osd.4) 205 : cluster 0 13.15 scrub starts 2024-04-02T04:52:50.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.244249+0000 osd.4 (osd.4) 206 : cluster 0 13.15 scrub ok 2024-04-02T04:52:50.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.308928+0000 osd.0 (osd.0) 199 : cluster 0 4.6 scrub starts 2024-04-02T04:52:50.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.310934+0000 osd.0 (osd.0) 200 : cluster 0 4.6 scrub ok 2024-04-02T04:52:50.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: audit 2024-04-02T04:52:49.853159+0000 mon.a (mon.0) 2209 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:50.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:50 smithi069 bash[28904]: cluster 2024-04-02T04:52:50.330535+0000 mon.a (mon.0) 2210 : cluster 0 osdmap e330: 8 total, 7 up, 8 in 2024-04-02T04:52:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:50.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:50.551 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:50.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:50.665 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.175396+0000 osd.1 (osd.1) 215 : cluster 0 3.0 scrub starts 2024-04-02T04:52:50.665 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.180510+0000 osd.1 (osd.1) 216 : cluster 0 3.0 scrub ok 2024-04-02T04:52:50.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.241969+0000 osd.4 (osd.4) 205 : cluster 0 13.15 scrub starts 2024-04-02T04:52:50.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.244249+0000 osd.4 (osd.4) 206 : cluster 0 13.15 scrub ok 2024-04-02T04:52:50.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.308928+0000 osd.0 (osd.0) 199 : cluster 0 4.6 scrub starts 2024-04-02T04:52:50.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.310934+0000 osd.0 (osd.0) 200 : cluster 0 4.6 scrub ok 2024-04-02T04:52:50.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: audit 2024-04-02T04:52:49.853159+0000 mon.a (mon.0) 2209 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-72153-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:50.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:50 smithi098 bash[23790]: cluster 2024-04-02T04:52:50.330535+0000 mon.a (mon.0) 2210 : cluster 0 osdmap e330: 8 total, 7 up, 8 in 2024-04-02T04:52:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:50.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:50.914 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:50.935 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:50.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:51.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.063 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:51.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:51.193 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:51.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:51.424 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.450 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:51.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:51.580 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:51.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.486208+0000 osd.6 (osd.6) 157 : cluster 0 10.16 scrub starts 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.488708+0000 osd.6 (osd.6) 158 : cluster 0 10.16 scrub ok 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.760146+0000 osd.3 (osd.3) 231 : cluster 0 40.a scrub starts 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:49.762406+0000 osd.3 (osd.3) 232 : cluster 0 40.a scrub ok 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:50.017876+0000 mgr.y (mgr.14538) 364 : cluster 0 pgmap v472: 388 pgs: 32 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.057563+0000 mon.c (mon.2) 561 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:50.649554+0000 osd.2 (osd.2) 155 : cluster 0 2.10 scrub starts 2024-04-02T04:52:51.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:50.651850+0000 osd.2 (osd.2) 156 : cluster 0 2.10 scrub ok 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.861355+0000 mon.a (mon.0) 2211 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-72145-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: cluster 2024-04-02T04:52:50.867670+0000 mon.a (mon.0) 2212 : cluster 0 osdmap e331: 8 total, 7 up, 8 in 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.902347+0000 mon.a (mon.0) 2213 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.903966+0000 mon.a (mon.0) 2214 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.904453+0000 mon.a (mon.0) 2215 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.976025+0000 mon.c (mon.2) 562 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:50.976475+0000 mon.a (mon.0) 2216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:51.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:51 smithi098 bash[23790]: audit 2024-04-02T04:52:51.058932+0000 mon.c (mon.2) 563 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:51.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.486208+0000 osd.6 (osd.6) 157 : cluster 0 10.16 scrub starts 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.488708+0000 osd.6 (osd.6) 158 : cluster 0 10.16 scrub ok 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.760146+0000 osd.3 (osd.3) 231 : cluster 0 40.a scrub starts 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:49.762406+0000 osd.3 (osd.3) 232 : cluster 0 40.a scrub ok 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:50.017876+0000 mgr.y (mgr.14538) 364 : cluster 0 pgmap v472: 388 pgs: 32 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.057563+0000 mon.c (mon.2) 561 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:50.649554+0000 osd.2 (osd.2) 155 : cluster 0 2.10 scrub starts 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:50.651850+0000 osd.2 (osd.2) 156 : cluster 0 2.10 scrub ok 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.861355+0000 mon.a (mon.0) 2211 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-72145-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: cluster 2024-04-02T04:52:50.867670+0000 mon.a (mon.0) 2212 : cluster 0 osdmap e331: 8 total, 7 up, 8 in 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.902347+0000 mon.a (mon.0) 2213 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.903966+0000 mon.a (mon.0) 2214 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.904453+0000 mon.a (mon.0) 2215 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:51.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.976025+0000 mon.c (mon.2) 562 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:50.976475+0000 mon.a (mon.0) 2216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[21528]: audit 2024-04-02T04:52:51.058932+0000 mon.c (mon.2) 563 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.486208+0000 osd.6 (osd.6) 157 : cluster 0 10.16 scrub starts 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.488708+0000 osd.6 (osd.6) 158 : cluster 0 10.16 scrub ok 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.760146+0000 osd.3 (osd.3) 231 : cluster 0 40.a scrub starts 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:49.762406+0000 osd.3 (osd.3) 232 : cluster 0 40.a scrub ok 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:50.017876+0000 mgr.y (mgr.14538) 364 : cluster 0 pgmap v472: 388 pgs: 32 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.057563+0000 mon.c (mon.2) 561 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:51.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:50.649554+0000 osd.2 (osd.2) 155 : cluster 0 2.10 scrub starts 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:50.651850+0000 osd.2 (osd.2) 156 : cluster 0 2.10 scrub ok 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.861355+0000 mon.a (mon.0) 2211 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-72145-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: cluster 2024-04-02T04:52:50.867670+0000 mon.a (mon.0) 2212 : cluster 0 osdmap e331: 8 total, 7 up, 8 in 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.902347+0000 mon.a (mon.0) 2213 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.903966+0000 mon.a (mon.0) 2214 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.904453+0000 mon.a (mon.0) 2215 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.976025+0000 mon.c (mon.2) 562 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:50.976475+0000 mon.a (mon.0) 2216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:51.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:51 smithi069 bash[28904]: audit 2024-04-02T04:52:51.058932+0000 mon.c (mon.2) 563 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:51.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.839 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:51.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.965 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:51.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:52.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:52.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:52.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:52.356 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:52.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:52.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.246294+0000 osd.1 (osd.1) 217 : cluster 0 3.4 scrub starts 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.249484+0000 osd.1 (osd.1) 218 : cluster 0 3.4 scrub ok 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.284750+0000 osd.4 (osd.4) 207 : cluster 0 40.1d scrub starts 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.287027+0000 osd.4 (osd.4) 208 : cluster 0 40.1d scrub ok 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: audit 2024-04-02T04:52:51.863706+0000 mon.a (mon.0) 2217 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: audit 2024-04-02T04:52:51.863786+0000 mon.a (mon.0) 2218 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.869785+0000 mon.a (mon.0) 2219 : cluster 0 osdmap e332: 8 total, 7 up, 8 in 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: audit 2024-04-02T04:52:51.873433+0000 mon.a (mon.0) 2220 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-72153-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: audit 2024-04-02T04:52:51.893095+0000 mon.c (mon.2) 564 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: audit 2024-04-02T04:52:51.893469+0000 mon.a (mon.0) 2221 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:52.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:52 smithi098 bash[23790]: audit 2024-04-02T04:52:52.060716+0000 mon.c (mon.2) 565 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:52.615 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:52.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:52.745 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.246294+0000 osd.1 (osd.1) 217 : cluster 0 3.4 scrub starts 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.249484+0000 osd.1 (osd.1) 218 : cluster 0 3.4 scrub ok 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.284750+0000 osd.4 (osd.4) 207 : cluster 0 40.1d scrub starts 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.287027+0000 osd.4 (osd.4) 208 : cluster 0 40.1d scrub ok 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: audit 2024-04-02T04:52:51.863706+0000 mon.a (mon.0) 2217 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: audit 2024-04-02T04:52:51.863786+0000 mon.a (mon.0) 2218 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.869785+0000 mon.a (mon.0) 2219 : cluster 0 osdmap e332: 8 total, 7 up, 8 in 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: audit 2024-04-02T04:52:51.873433+0000 mon.a (mon.0) 2220 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-72153-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: audit 2024-04-02T04:52:51.893095+0000 mon.c (mon.2) 564 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: audit 2024-04-02T04:52:51.893469+0000 mon.a (mon.0) 2221 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[28904]: audit 2024-04-02T04:52:52.060716+0000 mon.c (mon.2) 565 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.246294+0000 osd.1 (osd.1) 217 : cluster 0 3.4 scrub starts 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.249484+0000 osd.1 (osd.1) 218 : cluster 0 3.4 scrub ok 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.284750+0000 osd.4 (osd.4) 207 : cluster 0 40.1d scrub starts 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.287027+0000 osd.4 (osd.4) 208 : cluster 0 40.1d scrub ok 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: audit 2024-04-02T04:52:51.863706+0000 mon.a (mon.0) 2217 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: audit 2024-04-02T04:52:51.863786+0000 mon.a (mon.0) 2218 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.869785+0000 mon.a (mon.0) 2219 : cluster 0 osdmap e332: 8 total, 7 up, 8 in 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: audit 2024-04-02T04:52:51.873433+0000 mon.a (mon.0) 2220 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-72153-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: audit 2024-04-02T04:52:51.893095+0000 mon.c (mon.2) 564 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: audit 2024-04-02T04:52:51.893469+0000 mon.a (mon.0) 2221 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:52.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:52 smithi069 bash[21528]: audit 2024-04-02T04:52:52.060716+0000 mon.c (mon.2) 565 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:52.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:52.878 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:52.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:52.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:53.012 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:53.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:53.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:53.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:53.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:53.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.765047+0000 osd.3 (osd.3) 233 : cluster 0 102.1d scrub starts 2024-04-02T04:52:53.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:51.767212+0000 osd.3 (osd.3) 234 : cluster 0 102.1d scrub ok 2024-04-02T04:52:53.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.018875+0000 mgr.y (mgr.14538) 365 : cluster 0 pgmap v475: 364 pgs: 8 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.298978+0000 osd.0 (osd.0) 201 : cluster 0 3.1f scrub starts 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.302865+0000 osd.0 (osd.0) 202 : cluster 0 3.1f scrub ok 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.663823+0000 osd.2 (osd.2) 157 : cluster 0 40.11 scrub starts 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.665312+0000 osd.2 (osd.2) 158 : cluster 0 40.11 scrub ok 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.863801+0000 mon.a (mon.0) 2222 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: audit 2024-04-02T04:52:52.877104+0000 mon.a (mon.0) 2223 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: audit 2024-04-02T04:52:53.062439+0000 mon.c (mon.2) 566 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.334562+0000 mon.a (mon.0) 2224 : cluster 0 osdmap e333: 8 total, 7 up, 8 in 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[21528]: audit 2024-04-02T04:52:53.336525+0000 mon.a (mon.0) 2225 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.765047+0000 osd.3 (osd.3) 233 : cluster 0 102.1d scrub starts 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:51.767212+0000 osd.3 (osd.3) 234 : cluster 0 102.1d scrub ok 2024-04-02T04:52:53.383 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.018875+0000 mgr.y (mgr.14538) 365 : cluster 0 pgmap v475: 364 pgs: 8 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.298978+0000 osd.0 (osd.0) 201 : cluster 0 3.1f scrub starts 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.302865+0000 osd.0 (osd.0) 202 : cluster 0 3.1f scrub ok 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.663823+0000 osd.2 (osd.2) 157 : cluster 0 40.11 scrub starts 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.665312+0000 osd.2 (osd.2) 158 : cluster 0 40.11 scrub ok 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.863801+0000 mon.a (mon.0) 2222 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: audit 2024-04-02T04:52:52.877104+0000 mon.a (mon.0) 2223 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: audit 2024-04-02T04:52:53.062439+0000 mon.c (mon.2) 566 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.334562+0000 mon.a (mon.0) 2224 : cluster 0 osdmap e333: 8 total, 7 up, 8 in 2024-04-02T04:52:53.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:53 smithi069 bash[28904]: audit 2024-04-02T04:52:53.336525+0000 mon.a (mon.0) 2225 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:53.392 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:53.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.765047+0000 osd.3 (osd.3) 233 : cluster 0 102.1d scrub starts 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:51.767212+0000 osd.3 (osd.3) 234 : cluster 0 102.1d scrub ok 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.018875+0000 mgr.y (mgr.14538) 365 : cluster 0 pgmap v475: 364 pgs: 8 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 278/1530 objects degraded (18.170%) 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.298978+0000 osd.0 (osd.0) 201 : cluster 0 3.1f scrub starts 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.302865+0000 osd.0 (osd.0) 202 : cluster 0 3.1f scrub ok 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.663823+0000 osd.2 (osd.2) 157 : cluster 0 40.11 scrub starts 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.665312+0000 osd.2 (osd.2) 158 : cluster 0 40.11 scrub ok 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.863801+0000 mon.a (mon.0) 2222 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: audit 2024-04-02T04:52:52.877104+0000 mon.a (mon.0) 2223 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: audit 2024-04-02T04:52:53.062439+0000 mon.c (mon.2) 566 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.334562+0000 mon.a (mon.0) 2224 : cluster 0 osdmap e333: 8 total, 7 up, 8 in 2024-04-02T04:52:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:53 smithi098 bash[23790]: audit 2024-04-02T04:52:53.336525+0000 mon.a (mon.0) 2225 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:53.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:53.518 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:53.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:53.648 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:53.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:53.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:53.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:53.918 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:53.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:53.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/2595128790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 msgr2=0x7f23901086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- 172.21.15.69:0/2595128790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f23901086d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2380009990 tx=0x7f2380009ca0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/2595128790 shutdown_connections 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- 172.21.15.69:0/2595128790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f239006a910 0x7f239010f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- 172.21.15.69:0/2595128790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f23901086d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- 172.21.15.69:0/2595128790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f239006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/2595128790 >> 172.21.15.69:0/2595128790 conn(0x7f23900771e0 msgr2=0x7f23900775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/2595128790 shutdown_connections 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/2595128790 wait complete. 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 librados: starting msgr at 2024-04-02T04:52:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 librados: starting objecter 2024-04-02T04:52:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 Processor -- start 2024-04-02T04:52:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- start start 2024-04-02T04:52:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 librados: setting wanted keys 2024-04-02T04:52:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 librados: calling monclient init 2024-04-02T04:52:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f2390101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f239006a910 0x7f23901017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f2390101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2390111ab0 con 0x7f23901082f0 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2390111930 con 0x7f239006a910 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2390111c30 con 0x7f2390069f70 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f2390101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f2390101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:43712/0 (socket says 172.21.15.69:43712) 2024-04-02T04:52:54.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 -- 172.21.15.69:0/4234676247 learned_addr learned my addr 172.21.15.69:0/4234676247 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2394bf3700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f2390101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 -- 172.21.15.69:0/4234676247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f239006a910 msgr2=0x7f23901017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f239006a910 0x7f23901017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 -- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 msgr2=0x7f2390101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f2390101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 -- 172.21.15.69:0/4234676247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23901a65d0 con 0x7f2390069f70 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2394bf3700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f2390101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238ffff700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f2390101270 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2380000c00 tx=0x7f23800047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238d7fa700 10 client.?.objecter ms_handle_connect 0x7f2390069f70 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238d7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2380027070 con 0x7f2390069f70 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23901a6860 con 0x7f2390069f70 2024-04-02T04:52:54.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f2380022470 con 0x7f2390069f70 2024-04-02T04:52:54.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2380017650 con 0x7f2390069f70 2024-04-02T04:52:54.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 10 client.26351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:54.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23901a6d10 con 0x7f2390069f70 2024-04-02T04:52:54.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 librados: init done 2024-04-02T04:52:54.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.022+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239004ff00 con 0x7f2390069f70 2024-04-02T04:52:54.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f238002a070 con 0x7f2390069f70 2024-04-02T04:52:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f237806f5e0 0x7f2378071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(334..334 src has 74..334) ==== 12019+0+0 (secure 0 0 0) 0x7f2380090170 con 0x7f2390069f70 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 10 client.26351.objecter ms_dispatch 0x7f2390001550 osd_map(334..334 src has 74..334) 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 3 client.26351.objecter handle_osd_map got epochs [334,334] > 0 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 3 client.26351.objecter handle_osd_map decoding full epoch 334 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238f7fe700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f237806f5e0 0x7f2378071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 20 client.26351.objecter dump_active .. 0 homeless 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238d7fa700 10 client.26351.objecter ms_handle_connect 0x7f237806f5e0 2024-04-02T04:52:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.026+0000 7f238f7fe700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f237806f5e0 0x7f2378071aa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2384005950 tx=0x7f23840058e0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2380013070 con 0x7f2390069f70 2024-04-02T04:52:54.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 -- 172.21.15.69:0/21602257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 msgr2=0x7fb19810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/21602257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb19810f120 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb190009f90 tx=0x7fb19000b330 comp rx=0 tx=0).stop 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 -- 172.21.15.69:0/21602257 shutdown_connections 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/21602257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb19810f120 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/21602257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198108300 0x7fb1981086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/21602257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 -- 172.21.15.69:0/21602257 >> 172.21.15.69:0/21602257 conn(0x7fb1980793a0 msgr2=0x7fb1980797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 -- 172.21.15.69:0/21602257 shutdown_connections 2024-04-02T04:52:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.030+0000 7fb19f8f5700 1 -- 172.21.15.69:0/21602257 wait complete. 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 librados: starting msgr at 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 librados: starting objecter 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 Processor -- start 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- start start 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 librados: setting wanted keys 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 librados: calling monclient init 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb198198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19ce90700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb198198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198108300 0x7fb19819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39394/0 (socket says 172.21.15.69:39394) 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb198111aa0 con 0x7fb1981022f0 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 -- 172.21.15.69:0/3827138854 learned_addr learned my addr 172.21.15.69:0/3827138854 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb198111920 con 0x7fb198108300 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb198111c20 con 0x7fb198102cb0 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 -- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 msgr2=0x7fb198198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb198198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 -- 172.21.15.69:0/3827138854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198108300 msgr2=0x7fb19819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198108300 0x7fb19819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19819cef0 con 0x7fb1981022f0 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19ce90700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb198198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19d691700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198197ee0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb19400c970 tx=0x7fb19400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb18e7fc700 10 client.?.objecter ms_handle_connect 0x7fb1981022f0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb18e7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194007aa0 con 0x7fb1981022f0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb194004cb0 con 0x7fb1981022f0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19819d1e0 con 0x7fb1981022f0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194005470 con 0x7fb1981022f0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 10 client.16266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1981a4a60 con 0x7fb1981022f0 2024-04-02T04:52:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 librados: init done 2024-04-02T04:52:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19804ff00 con 0x7fb1981022f0 2024-04-02T04:52:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.034+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb194004750 con 0x7fb1981022f0 2024-04-02T04:52:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb18406f5e0 0x7fb184071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb19ce90700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb18406f5e0 0x7fb184071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(334..334 src has 74..334) ==== 12019+0+0 (secure 0 0 0) 0x7fb19408f0f0 con 0x7fb1981022f0 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb19ce90700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb18406f5e0 0x7fb184071aa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb188009750 tx=0x7fb188009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 10 client.16266.objecter ms_dispatch 0x7fb198001550 osd_map(334..334 src has 74..334) 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 3 client.16266.objecter handle_osd_map got epochs [334,334] > 0 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 3 client.16266.objecter handle_osd_map decoding full epoch 334 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 20 client.16266.objecter dump_active .. 0 homeless 2024-04-02T04:52:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.038+0000 7fb18e7fc700 10 client.16266.objecter ms_handle_connect 0x7fb18406f5e0 2024-04-02T04:52:54.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.042+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb19405b4a0 con 0x7fb1981022f0 2024-04-02T04:52:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:54.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:54.180 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 -- 172.21.15.69:0/3148355675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69dc102e60 msgr2=0x7f69dc106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 --2- 172.21.15.69:0/3148355675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69dc102e60 0x7f69dc106eb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0099c0 tx=0x7f69cc009cd0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 -- 172.21.15.69:0/3148355675 shutdown_connections 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 --2- 172.21.15.69:0/3148355675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc107590 0x7f69dc10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 --2- 172.21.15.69:0/3148355675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc1024b0 0x7f69dc102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 --2- 172.21.15.69:0/3148355675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69dc102e60 0x7f69dc106eb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 -- 172.21.15.69:0/3148355675 >> 172.21.15.69:0/3148355675 conn(0x7f69dc079610 msgr2=0x7f69dc079a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 -- 172.21.15.69:0/3148355675 shutdown_connections 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7f69e4582700 1 -- 172.21.15.69:0/3148355675 wait complete. 2024-04-02T04:52:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.274+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fb198199240 con 0x7fb1981022f0 2024-04-02T04:52:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 librados: starting msgr at 2024-04-02T04:52:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 librados: starting objecter 2024-04-02T04:52:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 Processor -- start 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- start start 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 librados: setting wanted keys 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 librados: calling monclient init 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69dc1024b0 0x7f69dc1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc102e60 0x7f69dc198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc107590 0x7f69dc19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc111a90 con 0x7f69dc102e60 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc111910 con 0x7f69dc107590 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f69dc111c10 con 0x7f69dc1024b0 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e1b1d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc102e60 0x7f69dc198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e1b1d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc102e60 0x7f69dc198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39408/0 (socket says 172.21.15.69:39408) 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e1b1d700 1 -- 172.21.15.69:0/3611478386 learned_addr learned my addr 172.21.15.69:0/3611478386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:52:54.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69dc1024b0 0x7f69dc1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc107590 0x7f69dc19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 -- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc107590 msgr2=0x7f69dc19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc107590 0x7f69dc19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 -- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc102e60 msgr2=0x7f69dc198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc102e60 0x7f69dc198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69dc19d0e0 con 0x7f69dc1024b0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69dc102e60 0x7f69dc198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69dc1024b0 0x7f69dc1980d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f69d800e9f0 tx=0x7f69d800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69d37fe700 10 client.?.objecter ms_handle_connect 0x7f69dc1024b0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69d37fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d800cbd0 con 0x7f69dc1024b0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f69d800cd70 con 0x7f69dc1024b0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69dc19d3d0 con 0x7f69dc1024b0 2024-04-02T04:52:54.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 10 client.26363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:54.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69dc1a4c50 con 0x7f69dc1024b0 2024-04-02T04:52:54.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 librados: init done 2024-04-02T04:52:54.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8018680 con 0x7f69dc1024b0 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.278+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f69c0005130 con 0x7f69dc1024b0 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f69d8018820 con 0x7f69dc1024b0 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f69c806f5e0 0x7f69c8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(334..334 src has 74..334) ==== 12019+0+0 (secure 0 0 0) 0x7f69d808f830 con 0x7f69dc1024b0 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 osd_map(334..334 src has 74..334) 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 3 client.26363.objecter handle_osd_map got epochs [334,334] > 0 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 3 client.26363.objecter handle_osd_map decoding full epoch 334 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f69c806f5e0 0x7f69c8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 20 client.26363.objecter dump_active .. 0 homeless 2024-04-02T04:52:54.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v334) ==== 54+0+16 (secure 0 0 0) 0x7f69d8090080 con 0x7f69dc1024b0 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f69c806f5e0 0x7f69c8071aa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0037a0 tx=0x7f69cc019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c806f5e0 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 10 client.26363.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 1 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69c0004760 0x7f69c0004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 1 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 0 1 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0005410 osd=0 4 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 10 client.26363.objecter _send_command 1 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69c0008ba0 con 0x7f69c0004760 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69c0004760 0x7f69c0004b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69c0004760 0x7f69c0004b80 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c0004760 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c0008ba0 con 0x7f69c0004760 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 1: 0 ) 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 10 client.26363.objecter _finish_command 1 = system:0 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c0005410 osd=0 4 2024-04-02T04:52:54.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.282+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 0 1 2024-04-02T04:52:54.284 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:54.285 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24628.0:2 14.3 14:c39a01fb:test-rados-api-smithi069-72410-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:26.014734+0000", 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 208.27434975099999, 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.39154399400000001, 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.286 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24628", 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2510353129", 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014734+0000", 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014734+0000", 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014736+0000", 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0509999999999999e-06 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014743+0000", 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4429999999999997e-06 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014756+0000", 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2435e-05 2024-04-02T04:52:54.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014769+0000", 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2996000000000001e-05 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014849+0000", 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0257000000000001e-05 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.014857+0000", 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8310000000000001e-06 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.135978+0000", 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.121120754 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.136100+0000", 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012249799999999999 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,3", 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.136229+0000", 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000129126 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.162670+0000", 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.026440254999999999 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.281533+0000", 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.118863593 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.406174+0000", 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.124641113 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.406233+0000", 2024-04-02T04:52:54.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8171000000000003e-05 2024-04-02T04:52:54.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.406278+0000", 2024-04-02T04:52:54.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5470999999999999e-05 2024-04-02T04:52:54.293 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.293 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.293 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24809.0:2 27.15 27:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:26.031450+0000", 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 208.25763398800001, 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.35376824299999998, 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24809", 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/271762309", 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031450+0000", 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031450+0000", 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031454+0000", 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8720000000000004e-06 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031486+0000", 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2619000000000001e-05 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031510+0000", 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3470000000000001e-05 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031530+0000", 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0431000000000001e-05 2024-04-02T04:52:54.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031881+0000", 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00035117200000000002 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.031892+0000", 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0433e-05 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.074387+0000", 2024-04-02T04:52:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.042494795000000002 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.074397+0000", 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0241000000000001e-05 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.385140+0000", 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.31074319299999997 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.385218+0000", 2024-04-02T04:52:54.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8016999999999995e-05 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24920.0:2 36.18 36:1d577f2e:test-rados-api-smithi069-72378-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:26.165432+0000", 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 208.123651857, 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.25210887799999998, 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.299 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24920", 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1522194686", 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.165432+0000", 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.165432+0000", 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.165433+0000", 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6020000000000001e-06 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.165441+0000", 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4719999999999998e-06 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.165454+0000", 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2578000000000001e-05 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.165464+0000", 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0925e-05 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.166087+0000", 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000622286 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.166096+0000", 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3219999999999999e-06 2024-04-02T04:52:54.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.306661+0000", 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.14056513000000001 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.306793+0000", 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013224499999999999 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,6", 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.306940+0000", 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000146429 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322000+0000", 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015060328 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.366496+0000", 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.044495701999999998 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.417461+0000", 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.050964655999999997 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.417506+0000", 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5189999999999999e-05 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.417541+0000", 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5012999999999998e-05 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.24758.0:1987 15.8 e139/135 15:1569c6cd:test-rados-api-smithi069-72304-15::foo:head v 139'972, mlcod=139'970)", 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:28.611965+0000", 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 205.677118971, 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.069591239999999999, 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611965+0000", 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611965+0000", 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611966+0000", 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3599999999999999e-06 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611973+0000", 2024-04-02T04:52:54.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1509999999999998e-06 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611985+0000", 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1375999999999999e-05 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611993+0000", 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6899999999999998e-06 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.653935+0000", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.041941612000000003 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.653957+0000", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2422e-05 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.681519+0000", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.027561535000000002 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.681556+0000", 2024-04-02T04:52:54.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7094000000000001e-05 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.24758.0:2059 15.7 e140/135 15:e4929e40:test-rados-api-smithi069-72304-16::big:head v 140'1, mlcod=0'0)", 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:28.888388+0000", 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 205.40069614199999, 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.080569994000000006, 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.888388+0000", 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.888388+0000", 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.888389+0000", 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8299999999999995e-07 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.911761+0000", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023371959000000001 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.911818+0000", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7348e-05 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.911840+0000", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2124999999999999e-05 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.911940+0000", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9607999999999999e-05 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.911963+0000", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3345000000000001e-05 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.968909+0000", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.056946518000000002 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.968958+0000", 2024-04-02T04:52:54.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8108000000000001e-05 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.7.0:261 15.15 e140/135 -17:accf5d73:::temp_15.15_0_24283_1:head v 0'0, mlcod=138'4)", 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:28.997380+0000", 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 205.29170352099999, 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.034481820000000003, 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.997380+0000", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.997380+0000", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.997382+0000", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3459999999999999e-06 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.012640+0000", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015258314 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.012661+0000", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0772999999999999e-05 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.012670+0000", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7860000000000008e-06 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.012750+0000", 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9808999999999994e-05 2024-04-02T04:52:54.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.012782+0000", 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1897999999999999e-05 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.031778+0000", 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.018995356000000001 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.031862+0000", 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4537999999999996e-05 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.317 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.24758.0:2074 15.7 e140/135 15:e36060b3:test-rados-api-smithi069-72304-17::big3:head v 140'3, mlcod=140'1)", 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:29.589306+0000", 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 204.69977816700001, 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.093102381999999997, 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.589306+0000", 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.589306+0000", 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.589308+0000", 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6860000000000002e-06 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.610710+0000", 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.021402109999999998 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.610750+0000", 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9181000000000003e-05 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.610768+0000", 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8158000000000001e-05 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.610883+0000", 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000114802 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.610931+0000", 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8412000000000003e-05 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.681431+0000", 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.070499876000000003 2024-04-02T04:52:54.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.682408+0000", 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00097715699999999998 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25184.0:2 53.1a 53:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:31.562742+0000", 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 202.72634162599999, 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.232107514, 2024-04-02T04:52:54.322 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25184", 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2929247525", 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562742+0000", 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562742+0000", 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562743+0000", 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.17e-06 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562748+0000", 2024-04-02T04:52:54.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.011e-06 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562758+0000", 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0059e-05 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562769+0000", 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1034e-05 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562832+0000", 2024-04-02T04:52:54.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2230999999999994e-05 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.562843+0000", 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1133e-05 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.784377+0000", 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2215347369999998 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.784502+0000", 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012415500000000001 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 5,6", 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.784638+0000", 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000136243 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.788072+0000", 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034337310000000002 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.789415+0000", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001343038 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.794695+0000", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052804210000000004 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.794783+0000", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7752999999999998e-05 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:33.794850+0000", 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6798000000000007e-05 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24994.0:2 64.1e 64:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:36.392241+0000", 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 197.89684281199999, 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3757462469999999, 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.329 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24994", 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/231433776", 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392241+0000", 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392241+0000", 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392242+0000", 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.497e-06 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392254+0000", 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1605e-05 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392264+0000", 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7520000000000008e-06 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392275+0000", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0767999999999999e-05 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392332+0000", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6960999999999999e-05 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:36.392341+0000", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2229999999999996e-06 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.764287+0000", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3719464210000001 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.764409+0000", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012210899999999999 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,4", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.764542+0000", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013265799999999999 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.765452+0000", 2024-04-02T04:52:54.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000909993 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.766036+0000", 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058437699999999997 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.767900+0000", 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018638839999999999 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.767947+0000", 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6727999999999997e-05 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.767987+0000", 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0271000000000001e-05 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWriteReply(115.4s0 166/164 ECSubWriteReply(tid=506, last_complete=166'47, committed=1, applied=1))", 2024-04-02T04:52:54.335 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:04.277278+0000", 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 170.011805791, 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.41365864200000002, 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.277278+0000", 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.277278+0000", 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.277279+0000", 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8100000000000002e-07 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.277283+0000", 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8609999999999997e-06 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.690725+0000", 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.41344107200000002 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.690738+0000", 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.377e-05 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.690873+0000", 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013504000000000001 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.690937+0000", 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3218000000000004e-05 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25442.0:46 110.4 110:20000000::::head [pgnls start_epoch 167] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e167)", 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:04.691848+0000", 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 169.59723528500001, 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.040077705999999998, 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25442", 2024-04-02T04:52:54.339 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 46 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691848+0000", 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691848+0000", 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691850+0000", 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.906e-06 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691858+0000", 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8490000000000006e-06 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691875+0000", 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7e-05 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691889+0000", 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3703000000000001e-05 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691947+0000", 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8538000000000001e-05 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.691953+0000", 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9220000000000002e-06 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.731739+0000", 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.039786015000000001 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.731831+0000", 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1415000000000005e-05 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.731926+0000", 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5358000000000004e-05 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15462.0:495 115.4s0 115:34008337:::494:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:04.803759+0000", 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 169.485325168, 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.047053853, 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15462", 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1621702341", 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 495 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803759+0000", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803759+0000", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803759+0000", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.7199999999999998e-07 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803763+0000", 2024-04-02T04:52:54.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5700000000000001e-06 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803778+0000", 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.525e-05 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803783+0000", 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3800000000000002e-06 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803834+0000", 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0569000000000003e-05 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.803898+0000", 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4263000000000005e-05 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.804071+0000", 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017291600000000001 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.804329+0000", 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025762799999999998 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.850771+0000", 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.046442512999999998 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.850812+0000", 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1091999999999999e-05 2024-04-02T04:52:54.348 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25568.0:2 128.15 128:ad551702:::test:head [create,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:06.066095+0000", 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 168.222988536, 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4192534509999999, 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25568", 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3776732179", 2024-04-02T04:52:54.349 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066095+0000", 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066095+0000", 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066097+0000", 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5039999999999999e-06 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066103+0000", 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.7120000000000003e-06 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066116+0000", 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2737e-05 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066128+0000", 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1409999999999999e-05 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066205+0000", 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7267000000000004e-05 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.066219+0000", 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3678e-05 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.460079+0000", 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.39386049 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.460202+0000", 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012341199999999999 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 2,3", 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.460332+0000", 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012956799999999999 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.463740+0000", 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034075490000000002 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.465888+0000", 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0021483890000000001 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.485271+0000", 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019382648999999998 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.485311+0000", 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0003000000000003e-05 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.485349+0000", 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8083000000000001e-05 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25553.0:104 125.1e 125:79d83b4f:::test_97:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e168)", 2024-04-02T04:52:54.355 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:06.198745+0000", 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 168.09033881600001, 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.246926637, 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25553", 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2131827544", 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 104 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198745+0000", 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198745+0000", 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198746+0000", 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0800000000000004e-07 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198750+0000", 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1239999999999998e-06 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198758+0000", 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3089999999999996e-06 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198763+0000", 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0520000000000001e-06 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198811+0000", 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7754999999999999e-05 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.198878+0000", 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6806999999999999e-05 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.445672+0000", 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.24679388199999999 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25568.0:399 128.10 128:0868e552:::test_392:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e169)", 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:08.411822+0000", 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 165.87726203700001, 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.077518554000000003, 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25568", 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3776732179", 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 399 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411822+0000", 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411822+0000", 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411823+0000", 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5060000000000001e-06 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411830+0000", 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4509999999999997e-06 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411840+0000", 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9720000000000004e-06 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411849+0000", 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3279999999999995e-06 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.411893+0000", 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4280999999999998e-05 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.412017+0000", 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000123266 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.489340+0000", 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.077323749999999997 2024-04-02T04:52:54.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25568.0:714 128.15 128:ad5c968e:::test_707:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e170)", 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:08.932348+0000", 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 165.35673620899999, 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.055733626000000001, 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.364 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25568", 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3776732179", 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 714 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932348+0000", 2024-04-02T04:52:54.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932348+0000", 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932349+0000", 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.42e-06 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932355+0000", 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2600000000000002e-06 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932366+0000", 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0924000000000001e-05 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932373+0000", 2024-04-02T04:52:54.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4880000000000004e-06 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.932417+0000", 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4156999999999998e-05 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.986270+0000", 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.053853107999999997 2024-04-02T04:52:54.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.988081+0000", 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018112689999999999 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25568.0:999 128.15 128:af7a18d4:::test_992:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e170)", 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:09.380284+0000", 2024-04-02T04:52:54.369 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 164.90879941099999, 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.057356896999999997, 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25568", 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3776732179", 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 999 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380284+0000", 2024-04-02T04:52:54.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380284+0000", 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380286+0000", 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.877e-06 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380296+0000", 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8379999999999997e-06 2024-04-02T04:52:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380315+0000", 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9361e-05 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380326+0000", 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0767999999999999e-05 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380398+0000", 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2292000000000005e-05 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.380537+0000", 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000138078 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:09.437641+0000", 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.057104683000000003 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.373 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25291.0:2 139.0 139:0384ab8e:::oid:head [append 0~64] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:11.732378+0000", 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 162.55670524799999, 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.109035633, 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25291", 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3596831311", 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.732378+0000", 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.732378+0000", 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.732381+0000", 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4320000000000002e-06 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.732393+0000", 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.236e-05 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.732417+0000", 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4003999999999999e-05 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.732438+0000", 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0602e-05 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.737615+0000", 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0051769800000000003 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:11.737643+0000", 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7835e-05 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.821346+0000", 2024-04-02T04:52:54.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.083703632 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.821446+0000", 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9171000000000003e-05 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3", 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.821572+0000", 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000126019 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.838046+0000", 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016474105999999999 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.840278+0000", 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022322359999999999 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.840355+0000", 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7213999999999996e-05 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.841414+0000", 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010590420000000001 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15570.0:2 138.0 138:0384ab8e:::oid:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:12.168483+0000", 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 162.12060066199999, 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.67317640099999998, 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.380 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15570", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1263654461", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168483+0000", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168483+0000", 2024-04-02T04:52:54.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168485+0000", 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9e-06 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168493+0000", 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1019999999999997e-06 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168509+0000", 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6368999999999999e-05 2024-04-02T04:52:54.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168522+0000", 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2951e-05 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168589+0000", 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6495999999999995e-05 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.168601+0000", 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1909000000000001e-05 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.821105+0000", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.65250418200000004 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.821240+0000", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013456199999999999 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.821403+0000", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000163034 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:52:54.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.836275+0000", 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014872386 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.841484+0000", 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052086859999999997 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.841550+0000", 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6247000000000005e-05 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.841659+0000", 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000109577 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.16134.0:1 247.6s0 247:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:52:08.856814+0000", 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 45.432269410000004, 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.051519758, 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:52:54.386 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.16134", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1187512078", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.856814+0000", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.856814+0000", 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:52:54.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.856816+0000", 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.604e-06 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.856822+0000", 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6860000000000003e-06 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.856833+0000", 2024-04-02T04:52:54.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0902e-05 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.856843+0000", 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0154e-05 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.857239+0000", 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00039605499999999997 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:08.857242+0000", 2024-04-02T04:52:54.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.72e-06 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:09.905030+0000", 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.047787281 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:09.905174+0000", 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014466699999999999 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:52:54.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:09.905739+0000", 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00056433099999999995 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:09.906953+0000", 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0012146500000000001 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:09.908269+0000", 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0013160660000000001 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:52:54.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:09.908334+0000", 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4641999999999995e-05 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:52:54.392 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:52:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.393 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:54.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 10 client.26363.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 2 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 2 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c0005410 osd=0 3 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 0 2 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0005410 osd=0 4 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 10 client.26363.objecter _send_command 2 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c0008500 con 0x7f69c0004760 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c0008500 con 0x7f69c0004760 2024-04-02T04:52:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69d37fe700 10 client.26363.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c0005410 osd=0 4 2024-04-02T04:52:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 0 2 2024-04-02T04:52:54.395 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:52:54.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 10 client.26363.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 3 2024-04-02T04:52:54.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f69c0002cd0 0x7f69c0003110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 3 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 1 3 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00089b0 osd=1 4 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 10 client.26363.objecter _send_command 3 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f69c0003870 con 0x7f69c0002cd0 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f69c0002cd0 0x7f69c0003110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c0002cd0 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.322+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f69c0002cd0 0x7f69c0003110 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.326+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c0003870 con 0x7f69c0002cd0 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.326+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 3: 0 ) 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.326+0000 7f69d37fe700 10 client.26363.objecter _finish_command 3 = system:0 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.326+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c00089b0 osd=1 4 2024-04-02T04:52:54.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.326+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 1 3 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 10 client.26363.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 4 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 4 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c00089b0 osd=1 3 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 1 4 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00089b0 osd=1 4 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 10 client.26363.objecter _send_command 4 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c0004280 con 0x7f69c0002cd0 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c0004280 con 0x7f69c0002cd0 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69d37fe700 10 client.26363.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c00089b0 osd=1 4 2024-04-02T04:52:54.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 1 4 2024-04-02T04:52:54.399 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:52:54.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 10 client.26363.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 5 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f69c000b270 0x7f69c000d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 5 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 2 5 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0007fe0 osd=2 4 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 10 client.26363.objecter _send_command 5 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f69c00104c0 con 0x7f69c000b270 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f69c000b270 0x7f69c000d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c000b270 2024-04-02T04:52:54.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.346+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f69c000b270 0x7f69c000d660 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.350+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c00104c0 con 0x7f69c000b270 2024-04-02T04:52:54.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.350+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 5: 0 ) 2024-04-02T04:52:54.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.350+0000 7f69d37fe700 10 client.26363.objecter _finish_command 5 = system:0 2024-04-02T04:52:54.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.350+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c0007fe0 osd=2 4 2024-04-02T04:52:54.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.350+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 2 5 2024-04-02T04:52:54.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.362+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]} v 0) -- 0x7f23901a6f70 con 0x7f2390069f70 2024-04-02T04:52:54.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 10 client.26363.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 6 2024-04-02T04:52:54.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 6 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c0007fe0 osd=2 3 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 2 6 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0007fe0 osd=2 4 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 10 client.26363.objecter _send_command 6 2024-04-02T04:52:54.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.366+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c0003d70 con 0x7f69c000b270 2024-04-02T04:52:54.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c0003d70 con 0x7f69c000b270 2024-04-02T04:52:54.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 10 client.26363.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c0007fe0 osd=2 4 2024-04-02T04:52:54.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 2 6 2024-04-02T04:52:54.404 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:52:54.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 10 client.26363.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 7 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69c000ed20 0x7f69c000f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 7 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 3 7 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0004000 osd=3 4 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 10 client.26363.objecter _send_command 7 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f69c000f710 con 0x7f69c000ed20 2024-04-02T04:52:54.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69c000ed20 0x7f69c000f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69c000ed20 0x7f69c000f1d0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c000ed20 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c000f710 con 0x7f69c000ed20 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 7: 0 ) 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 10 client.26363.objecter _finish_command 7 = system:0 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c0004000 osd=3 4 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.370+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 3 7 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 10 client.26363.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 8 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 8 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c0004000 osd=3 3 2024-04-02T04:52:54.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 3 8 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c0004000 osd=3 4 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 10 client.26363.objecter _send_command 8 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.386+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c000f710 con 0x7f69c000ed20 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c000f710 con 0x7f69c000ed20 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69d37fe700 10 client.26363.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c0004000 osd=3 4 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 3 8 2024-04-02T04:52:54.408 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 10 client.26363.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 9 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69c0012680 0x7f69c0014a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 9 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 4 9 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c000f710 osd=4 4 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 10 client.26363.objecter _send_command 9 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f69c00178d0 con 0x7f69c0012680 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69c0012680 0x7f69c0014a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c0012680 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.390+0000 7f69e231e700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69c0012680 0x7f69c0014a70 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.394+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c00178d0 con 0x7f69c0012680 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.394+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 9: 0 ) 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.394+0000 7f69d37fe700 10 client.26363.objecter _finish_command 9 = system:0 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.394+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c000f710 osd=4 4 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.394+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 4 9 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 10 client.26363.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 10 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 10 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c000f710 osd=4 3 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 4 10 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c000f710 osd=4 4 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 10 client.26363.objecter _send_command 10 2024-04-02T04:52:54.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.410+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c00193c0 con 0x7f69c0012680 2024-04-02T04:52:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c00193c0 con 0x7f69c0012680 2024-04-02T04:52:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69d37fe700 10 client.26363.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c000f710 osd=4 4 2024-04-02T04:52:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 4 10 2024-04-02T04:52:54.413 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:52:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 10 client.26363.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 11 2024-04-02T04:52:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69c0019ae0 0x7f69c001bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 11 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 5 11 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00193c0 osd=5 4 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 10 client.26363.objecter _send_command 11 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f69c001ef00 con 0x7f69c0019ae0 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69c0019ae0 0x7f69c001bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c0019ae0 2024-04-02T04:52:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.414+0000 7f69e1b1d700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69c0019ae0 0x7f69c001bef0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.418+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c001ef00 con 0x7f69c0019ae0 2024-04-02T04:52:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.418+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 11: 0 ) 2024-04-02T04:52:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.418+0000 7f69d37fe700 10 client.26363.objecter _finish_command 11 = system:0 2024-04-02T04:52:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.418+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c00193c0 osd=5 4 2024-04-02T04:52:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.418+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 5 11 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 10 client.26363.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 12 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 12 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c00193c0 osd=5 3 2024-04-02T04:52:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 5 12 2024-04-02T04:52:54.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c00193c0 osd=5 4 2024-04-02T04:52:54.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 10 client.26363.objecter _send_command 12 2024-04-02T04:52:54.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.434+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c001ef00 con 0x7f69c0019ae0 2024-04-02T04:52:54.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c001ef00 con 0x7f69c0019ae0 2024-04-02T04:52:54.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69d37fe700 10 client.26363.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c00193c0 osd=5 4 2024-04-02T04:52:54.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 5 12 2024-04-02T04:52:54.436 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:52:54.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 10 client.26363.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 13 2024-04-02T04:52:54.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69c001e220 0x7f69c001e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:52:54.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 13 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 6 13 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c001ef00 osd=6 4 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 10 client.26363.objecter _send_command 13 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f69c0022300 con 0x7f69c001e220 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69c001e220 0x7f69c001e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69d37fe700 10 client.26363.objecter ms_handle_connect 0x7f69c001e220 2024-04-02T04:52:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.438+0000 7f69e2b1f700 1 --2- 172.21.15.69:0/3611478386 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69c001e220 0x7f69c001e6a0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:52:54.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.442+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69c0022300 con 0x7f69c001e220 2024-04-02T04:52:54.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.442+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 13: 0 ) 2024-04-02T04:52:54.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.442+0000 7f69d37fe700 10 client.26363.objecter _finish_command 13 = system:0 2024-04-02T04:52:54.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.442+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c001ef00 osd=6 4 2024-04-02T04:52:54.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.442+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 6 13 2024-04-02T04:52:54.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 10 client.26363.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:52:54.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 14 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 20 client.26363.objecter _get_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 15 client.26363.objecter _session_command_op_remove -1 14 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 20 client.26363.objecter get_session s=0x7f69c001ef00 osd=6 3 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign 6 14 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 20 client.26363.objecter put_session s=0x7f69c001ef00 osd=6 4 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 10 client.26363.objecter _send_command 14 2024-04-02T04:52:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.458+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69c001c540 con 0x7f69c001e220 2024-04-02T04:52:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69c001c540 con 0x7f69c001e220 2024-04-02T04:52:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d37fe700 10 client.26363.objecter ms_dispatch 0x7f69dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:52:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d37fe700 10 client.26363.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:52:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d37fe700 20 client.26363.objecter put_session s=0x7f69c001ef00 osd=6 4 2024-04-02T04:52:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d37fe700 15 client.26363.objecter _session_command_op_remove 6 14 2024-04-02T04:52:54.461 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69e4582700 10 client.26363.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69e4582700 15 client.26363.objecter _session_command_op_assign -1 15 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69e4582700 20 client.26363.objecter _get_session osd=-1 returning homeless 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69e4582700 10 client.26363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=335}) -- 0x7f69c001d8d0 con 0x7f69dc1024b0 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69e4582700 1 -- 172.21.15.69:0/3611478386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69c001c540 con 0x7f69dc1024b0 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d37fe700 1 -- 172.21.15.69:0/3611478386 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_get_version_reply(handle=1 version=334) ==== 24+0+0 (secure 0 0 0) 0x7f69d808c570 con 0x7f69dc1024b0 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d3fff700 10 client.26363.objecter _check_command_map_dne tid 15 current 334 map_dne_bound 334 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d3fff700 10 client.26363.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.462+0000 7f69d3fff700 15 client.26363.objecter _session_command_op_remove -1 15 2024-04-02T04:52:54.462 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:52:54.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:52:54.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.415865+0000 osd.6 (osd.6) 159 : cluster 0 43.b scrub starts 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:52.417767+0000 osd.6 (osd.6) 160 : cluster 0 43.b scrub ok 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:53.163103+0000 mgr.y (mgr.14538) 366 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.666666+0000 osd.2 (osd.2) 159 : cluster 0 13.8 scrub starts 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.668855+0000 osd.2 (osd.2) 160 : cluster 0 13.8 scrub ok 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.800223+0000 osd.5 (osd.5) 205 : cluster 0 3.1c scrub starts 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.802707+0000 osd.5 (osd.5) 206 : cluster 0 3.1c scrub ok 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:53.880493+0000 mon.a (mon.0) 2226 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-72153-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:53.880607+0000 mon.a (mon.0) 2227 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.885256+0000 mon.a (mon.0) 2228 : cluster 0 osdmap e334: 8 total, 7 up, 8 in 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:53.887620+0000 mon.a (mon.0) 2229 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:53.945979+0000 mon.c (mon.2) 567 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:53.946458+0000 mon.a (mon.0) 2230 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:54.063792+0000 mon.c (mon.2) 568 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:54.280092+0000 mon.a (mon.0) 2231 : audit 1 from='client.? 172.21.15.69:0/3827138854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[28904]: audit 2024-04-02T04:52:54.283103+0000 mon.c (mon.2) 569 : audit 0 from='client.? 172.21.15.69:0/3611478386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.415865+0000 osd.6 (osd.6) 159 : cluster 0 43.b scrub starts 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:52.417767+0000 osd.6 (osd.6) 160 : cluster 0 43.b scrub ok 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:53.163103+0000 mgr.y (mgr.14538) 366 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.666666+0000 osd.2 (osd.2) 159 : cluster 0 13.8 scrub starts 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.668855+0000 osd.2 (osd.2) 160 : cluster 0 13.8 scrub ok 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.800223+0000 osd.5 (osd.5) 205 : cluster 0 3.1c scrub starts 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.802707+0000 osd.5 (osd.5) 206 : cluster 0 3.1c scrub ok 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:53.880493+0000 mon.a (mon.0) 2226 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-72153-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:53.880607+0000 mon.a (mon.0) 2227 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.885256+0000 mon.a (mon.0) 2228 : cluster 0 osdmap e334: 8 total, 7 up, 8 in 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:53.887620+0000 mon.a (mon.0) 2229 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:53.945979+0000 mon.c (mon.2) 567 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:54.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:53.946458+0000 mon.a (mon.0) 2230 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:54.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:54.063792+0000 mon.c (mon.2) 568 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:54.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:54.280092+0000 mon.a (mon.0) 2231 : audit 1 from='client.? 172.21.15.69:0/3827138854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:52:54.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:54 smithi069 bash[21528]: audit 2024-04-02T04:52:54.283103+0000 mon.c (mon.2) 569 : audit 0 from='client.? 172.21.15.69:0/3611478386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:54.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:54.512 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:54.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.415865+0000 osd.6 (osd.6) 159 : cluster 0 43.b scrub starts 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:52.417767+0000 osd.6 (osd.6) 160 : cluster 0 43.b scrub ok 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:53.163103+0000 mgr.y (mgr.14538) 366 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.666666+0000 osd.2 (osd.2) 159 : cluster 0 13.8 scrub starts 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.668855+0000 osd.2 (osd.2) 160 : cluster 0 13.8 scrub ok 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.800223+0000 osd.5 (osd.5) 205 : cluster 0 3.1c scrub starts 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.802707+0000 osd.5 (osd.5) 206 : cluster 0 3.1c scrub ok 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:53.880493+0000 mon.a (mon.0) 2226 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-72153-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:54.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:53.880607+0000 mon.a (mon.0) 2227 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.885256+0000 mon.a (mon.0) 2228 : cluster 0 osdmap e334: 8 total, 7 up, 8 in 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:53.887620+0000 mon.a (mon.0) 2229 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]: dispatch 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:53.945979+0000 mon.c (mon.2) 567 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:53.946458+0000 mon.a (mon.0) 2230 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:54.063792+0000 mon.c (mon.2) 568 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:54.280092+0000 mon.a (mon.0) 2231 : audit 1 from='client.? 172.21.15.69:0/3827138854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:52:54.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:54 smithi098 bash[23790]: audit 2024-04-02T04:52:54.283103+0000 mon.c (mon.2) 569 : audit 0 from='client.? 172.21.15.69:0/3611478386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:52:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:54.743 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:54.769 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:54.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.830+0000 7fb18e7fc700 1 -- 172.21.15.69:0/3827138854 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v335) ==== 101+0+0 (secure 0 0 0) 0x7fb19405eef0 con 0x7fb1981022f0 2024-04-02T04:52:54.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 10 librados: watch_flush enter 2024-04-02T04:52:54.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 10 librados: watch_flush exit 2024-04-02T04:52:54.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 20 client.16266.objecter shutdown clearing up homeless session... 2024-04-02T04:52:54.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 10 client.16266.objecter successfully canceled tick 2024-04-02T04:52:54.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb18406f5e0 msgr2=0x7fb184071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.842+0000 7f238d7fa700 1 -- 172.21.15.69:0/4234676247 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]=0 set 41.a pg_upmap mapping to [1,4,3] v335) ==== 129+0+0 (secure 0 0 0) 0x7f238005c580 con 0x7f2390069f70 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb18406f5e0 0x7fb184071aa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb188009750 tx=0x7fb188009310 comp rx=0 tx=0).stop 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 msgr2=0x7fb198197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198197ee0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb19400c970 tx=0x7fb19400cc80 comp rx=0 tx=0).stop 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 shutdown_connections 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb18406f5e0 0x7fb184071aa0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb198108300 0x7fb19819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1981022f0 0x7fb198197ee0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 --2- 172.21.15.69:0/3827138854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb198102cb0 0x7fb198198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.834+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 >> 172.21.15.69:0/3827138854 conn(0x7fb1980793a0 msgr2=0x7fb198105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:54.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.842+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 shutdown_connections 2024-04-02T04:52:54.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.846+0000 7fb19f8f5700 1 -- 172.21.15.69:0/3827138854 wait complete. 2024-04-02T04:52:54.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7fb19f8f5700 1 librados: shutdown 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 10 librados: watch_flush enter 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 10 librados: watch_flush exit 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 20 client.26351.objecter shutdown clearing up homeless session... 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 10 client.26351.objecter successfully canceled tick 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f237806f5e0 msgr2=0x7f2378071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f237806f5e0 0x7f2378071aa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2384005950 tx=0x7f23840058e0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 msgr2=0x7f2390101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f2390101270 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2380000c00 tx=0x7f23800047b0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 shutdown_connections 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f237806f5e0 0x7f2378071aa0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23901082f0 0x7f2390101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2390069f70 0x7f2390101270 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 --2- 172.21.15.69:0/4234676247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f239006a910 0x7f23901017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 >> 172.21.15.69:0/4234676247 conn(0x7f23900771e0 msgr2=0x7f2390109070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 shutdown_connections 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 -- 172.21.15.69:0/4234676247 wait complete. 2024-04-02T04:52:54.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:52:54.850+0000 7f2396656700 1 librados: shutdown 2024-04-02T04:52:54.861 INFO:teuthology.orchestra.run.smithi069.stderr:set 41.a pg_upmap mapping to [1,4,3] 2024-04-02T04:52:54.861 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:52:54.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:54.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:55.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:55.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:55.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:55.284 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:55.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.408955+0000 osd.6 (osd.6) 161 : cluster 0 40.f scrub starts 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.410903+0000 osd.6 (osd.6) 162 : cluster 0 40.f scrub ok 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.765611+0000 osd.3 (osd.3) 235 : cluster 0 102.6 scrub starts 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:53.769662+0000 osd.3 (osd.3) 236 : cluster 0 102.6 scrub ok 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:54.019696+0000 mgr.y (mgr.14538) 367 : cluster 0 pgmap v478: 364 pgs: 8 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:54.292100+0000 osd.1 (osd.1) 219 : cluster 0 4.12 scrub starts 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:54.293635+0000 osd.1 (osd.1) 220 : cluster 0 4.12 scrub ok 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.367201+0000 mon.c (mon.2) 570 : audit 1 from='client.? 172.21.15.69:0/4234676247' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]: dispatch 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.367549+0000 mon.a (mon.0) 2232 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]: dispatch 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:54.831637+0000 mon.a (mon.0) 2233 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.833722+0000 mon.a (mon.0) 2234 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.833805+0000 mon.a (mon.0) 2235 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:55.400 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.833899+0000 mon.a (mon.0) 2236 : audit 1 from='client.? 172.21.15.69:0/3827138854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.834010+0000 mon.a (mon.0) 2237 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]': finished 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:54.839455+0000 mon.a (mon.0) 2238 : cluster 0 osdmap e335: 8 total, 7 up, 8 in 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.846567+0000 mon.c (mon.2) 571 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.856330+0000 mon.a (mon.0) 2239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: cluster 2024-04-02T04:52:54.880451+0000 mon.a (mon.0) 2240 : cluster 3 Health check update: Degraded data redundancy: 278/1548 objects degraded (17.959%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.896398+0000 mon.c (mon.2) 572 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.897134+0000 mon.a (mon.0) 2241 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.898654+0000 mon.c (mon.2) 573 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.899119+0000 mon.a (mon.0) 2242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.900090+0000 mon.c (mon.2) 574 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:54.900525+0000 mon.a (mon.0) 2243 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[28904]: audit 2024-04-02T04:52:55.064777+0000 mon.c (mon.2) 575 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.408955+0000 osd.6 (osd.6) 161 : cluster 0 40.f scrub starts 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.410903+0000 osd.6 (osd.6) 162 : cluster 0 40.f scrub ok 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.765611+0000 osd.3 (osd.3) 235 : cluster 0 102.6 scrub starts 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:53.769662+0000 osd.3 (osd.3) 236 : cluster 0 102.6 scrub ok 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:54.019696+0000 mgr.y (mgr.14538) 367 : cluster 0 pgmap v478: 364 pgs: 8 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:54.292100+0000 osd.1 (osd.1) 219 : cluster 0 4.12 scrub starts 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:54.293635+0000 osd.1 (osd.1) 220 : cluster 0 4.12 scrub ok 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.367201+0000 mon.c (mon.2) 570 : audit 1 from='client.? 172.21.15.69:0/4234676247' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]: dispatch 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.367549+0000 mon.a (mon.0) 2232 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]: dispatch 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:54.831637+0000 mon.a (mon.0) 2233 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.833722+0000 mon.a (mon.0) 2234 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:55.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.833805+0000 mon.a (mon.0) 2235 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.833899+0000 mon.a (mon.0) 2236 : audit 1 from='client.? 172.21.15.69:0/3827138854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.834010+0000 mon.a (mon.0) 2237 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]': finished 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:54.839455+0000 mon.a (mon.0) 2238 : cluster 0 osdmap e335: 8 total, 7 up, 8 in 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.846567+0000 mon.c (mon.2) 571 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.856330+0000 mon.a (mon.0) 2239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: cluster 2024-04-02T04:52:54.880451+0000 mon.a (mon.0) 2240 : cluster 3 Health check update: Degraded data redundancy: 278/1548 objects degraded (17.959%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.896398+0000 mon.c (mon.2) 572 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.897134+0000 mon.a (mon.0) 2241 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.898654+0000 mon.c (mon.2) 573 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.899119+0000 mon.a (mon.0) 2242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.900090+0000 mon.c (mon.2) 574 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:54.900525+0000 mon.a (mon.0) 2243 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:55.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:55 smithi069 bash[21528]: audit 2024-04-02T04:52:55.064777+0000 mon.c (mon.2) 575 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:55.414 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:55.517 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:55.544 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:55.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:55.659 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.408955+0000 osd.6 (osd.6) 161 : cluster 0 40.f scrub starts 2024-04-02T04:52:55.659 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.410903+0000 osd.6 (osd.6) 162 : cluster 0 40.f scrub ok 2024-04-02T04:52:55.659 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.765611+0000 osd.3 (osd.3) 235 : cluster 0 102.6 scrub starts 2024-04-02T04:52:55.659 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:53.769662+0000 osd.3 (osd.3) 236 : cluster 0 102.6 scrub ok 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:54.019696+0000 mgr.y (mgr.14538) 367 : cluster 0 pgmap v478: 364 pgs: 8 unknown, 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:54.292100+0000 osd.1 (osd.1) 219 : cluster 0 4.12 scrub starts 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:54.293635+0000 osd.1 (osd.1) 220 : cluster 0 4.12 scrub ok 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.367201+0000 mon.c (mon.2) 570 : audit 1 from='client.? 172.21.15.69:0/4234676247' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]: dispatch 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.367549+0000 mon.a (mon.0) 2232 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]: dispatch 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:54.831637+0000 mon.a (mon.0) 2233 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.833722+0000 mon.a (mon.0) 2234 : audit 1 from='client.? 172.21.15.69:0/2522793461' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-72145-41"}]': finished 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.833805+0000 mon.a (mon.0) 2235 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.833899+0000 mon.a (mon.0) 2236 : audit 1 from='client.? 172.21.15.69:0/3827138854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.834010+0000 mon.a (mon.0) 2237 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "41.a", "id": [1, 4, 3]}]': finished 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:54.839455+0000 mon.a (mon.0) 2238 : cluster 0 osdmap e335: 8 total, 7 up, 8 in 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.846567+0000 mon.c (mon.2) 571 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.856330+0000 mon.a (mon.0) 2239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]: dispatch 2024-04-02T04:52:55.660 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: cluster 2024-04-02T04:52:54.880451+0000 mon.a (mon.0) 2240 : cluster 3 Health check update: Degraded data redundancy: 278/1548 objects degraded (17.959%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.896398+0000 mon.c (mon.2) 572 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.897134+0000 mon.a (mon.0) 2241 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.898654+0000 mon.c (mon.2) 573 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.899119+0000 mon.a (mon.0) 2242 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.900090+0000 mon.c (mon.2) 574 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:54.900525+0000 mon.a (mon.0) 2243 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:55.661 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:55 smithi098 bash[23790]: audit 2024-04-02T04:52:55.064777+0000 mon.c (mon.2) 575 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:55.674 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:55.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:55.804 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:55.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:55.930 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:55.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:56.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:56.059 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:56.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:56.186 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:56.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:56.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:56.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: cluster 2024-04-02T04:52:55.332685+0000 osd.0 (osd.0) 203 : cluster 0 4.b scrub starts 2024-04-02T04:52:56.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: cluster 2024-04-02T04:52:55.334970+0000 osd.0 (osd.0) 204 : cluster 0 4.b scrub ok 2024-04-02T04:52:56.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: cluster 2024-04-02T04:52:55.611356+0000 osd.2 (osd.2) 161 : cluster 0 13.a scrub starts 2024-04-02T04:52:56.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: cluster 2024-04-02T04:52:55.613390+0000 osd.2 (osd.2) 162 : cluster 0 13.a scrub ok 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: cluster 2024-04-02T04:52:55.848491+0000 mon.a (mon.0) 2244 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: audit 2024-04-02T04:52:55.850783+0000 mon.a (mon.0) 2245 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: audit 2024-04-02T04:52:55.851003+0000 mon.a (mon.0) 2246 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: cluster 2024-04-02T04:52:55.863130+0000 mon.a (mon.0) 2247 : cluster 0 osdmap e336: 8 total, 7 up, 8 in 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: audit 2024-04-02T04:52:55.867395+0000 mon.c (mon.2) 576 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: audit 2024-04-02T04:52:55.870282+0000 mon.a (mon.0) 2248 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: audit 2024-04-02T04:52:55.871244+0000 mon.a (mon.0) 2249 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[28904]: audit 2024-04-02T04:52:56.066041+0000 mon.c (mon.2) 577 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: cluster 2024-04-02T04:52:55.332685+0000 osd.0 (osd.0) 203 : cluster 0 4.b scrub starts 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: cluster 2024-04-02T04:52:55.334970+0000 osd.0 (osd.0) 204 : cluster 0 4.b scrub ok 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: cluster 2024-04-02T04:52:55.611356+0000 osd.2 (osd.2) 161 : cluster 0 13.a scrub starts 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: cluster 2024-04-02T04:52:55.613390+0000 osd.2 (osd.2) 162 : cluster 0 13.a scrub ok 2024-04-02T04:52:56.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: cluster 2024-04-02T04:52:55.848491+0000 mon.a (mon.0) 2244 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: audit 2024-04-02T04:52:55.850783+0000 mon.a (mon.0) 2245 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: audit 2024-04-02T04:52:55.851003+0000 mon.a (mon.0) 2246 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: cluster 2024-04-02T04:52:55.863130+0000 mon.a (mon.0) 2247 : cluster 0 osdmap e336: 8 total, 7 up, 8 in 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: audit 2024-04-02T04:52:55.867395+0000 mon.c (mon.2) 576 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: audit 2024-04-02T04:52:55.870282+0000 mon.a (mon.0) 2248 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: audit 2024-04-02T04:52:55.871244+0000 mon.a (mon.0) 2249 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:56.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:56 smithi069 bash[21528]: audit 2024-04-02T04:52:56.066041+0000 mon.c (mon.2) 577 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:56.448 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:56.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:56.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:56.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:56.709 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: cluster 2024-04-02T04:52:55.332685+0000 osd.0 (osd.0) 203 : cluster 0 4.b scrub starts 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: cluster 2024-04-02T04:52:55.334970+0000 osd.0 (osd.0) 204 : cluster 0 4.b scrub ok 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: cluster 2024-04-02T04:52:55.611356+0000 osd.2 (osd.2) 161 : cluster 0 13.a scrub starts 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: cluster 2024-04-02T04:52:55.613390+0000 osd.2 (osd.2) 162 : cluster 0 13.a scrub ok 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: cluster 2024-04-02T04:52:55.848491+0000 mon.a (mon.0) 2244 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: audit 2024-04-02T04:52:55.850783+0000 mon.a (mon.0) 2245 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-35"}]': finished 2024-04-02T04:52:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: audit 2024-04-02T04:52:55.851003+0000 mon.a (mon.0) 2246 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-72145-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: cluster 2024-04-02T04:52:55.863130+0000 mon.a (mon.0) 2247 : cluster 0 osdmap e336: 8 total, 7 up, 8 in 2024-04-02T04:52:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: audit 2024-04-02T04:52:55.867395+0000 mon.c (mon.2) 576 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: audit 2024-04-02T04:52:55.870282+0000 mon.a (mon.0) 2248 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: audit 2024-04-02T04:52:55.871244+0000 mon.a (mon.0) 2249 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:52:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:56 smithi098 bash[23790]: audit 2024-04-02T04:52:56.066041+0000 mon.c (mon.2) 577 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:56.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:56.839 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:56.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:56.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:57.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:57.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:57.222 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:57.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:57.351 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:57.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-02T04:52:57.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:57.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:52:57.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:52:57.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:57.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:57.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:57.747 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:57.850 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:57.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: cluster 2024-04-02T04:52:56.021460+0000 mgr.y (mgr.14538) 368 : cluster 0 pgmap v481: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:52:57.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: cluster 2024-04-02T04:52:56.595025+0000 osd.2 (osd.2) 163 : cluster 0 43.3 scrub starts 2024-04-02T04:52:57.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: cluster 2024-04-02T04:52:56.596964+0000 osd.2 (osd.2) 164 : cluster 0 43.3 scrub ok 2024-04-02T04:52:57.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: audit 2024-04-02T04:52:56.859215+0000 mon.a (mon.0) 2250 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:57.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: audit 2024-04-02T04:52:57.067281+0000 mon.c (mon.2) 578 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:57.864 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: cluster 2024-04-02T04:52:57.306661+0000 mon.a (mon.0) 2251 : cluster 0 osdmap e337: 8 total, 7 up, 8 in 2024-04-02T04:52:57.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[21528]: audit 2024-04-02T04:52:57.308363+0000 mon.a (mon.0) 2252 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: cluster 2024-04-02T04:52:56.021460+0000 mgr.y (mgr.14538) 368 : cluster 0 pgmap v481: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: cluster 2024-04-02T04:52:56.595025+0000 osd.2 (osd.2) 163 : cluster 0 43.3 scrub starts 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: cluster 2024-04-02T04:52:56.596964+0000 osd.2 (osd.2) 164 : cluster 0 43.3 scrub ok 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: audit 2024-04-02T04:52:56.859215+0000 mon.a (mon.0) 2250 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: audit 2024-04-02T04:52:57.067281+0000 mon.c (mon.2) 578 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: cluster 2024-04-02T04:52:57.306661+0000 mon.a (mon.0) 2251 : cluster 0 osdmap e337: 8 total, 7 up, 8 in 2024-04-02T04:52:57.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:57 smithi069 bash[28904]: audit 2024-04-02T04:52:57.308363+0000 mon.a (mon.0) 2252 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:57.888 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:57.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:57.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: cluster 2024-04-02T04:52:56.021460+0000 mgr.y (mgr.14538) 368 : cluster 0 pgmap v481: 356 pgs: 92 active+undersized, 227 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: cluster 2024-04-02T04:52:56.595025+0000 osd.2 (osd.2) 163 : cluster 0 43.3 scrub starts 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: cluster 2024-04-02T04:52:56.596964+0000 osd.2 (osd.2) 164 : cluster 0 43.3 scrub ok 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: audit 2024-04-02T04:52:56.859215+0000 mon.a (mon.0) 2250 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: audit 2024-04-02T04:52:57.067281+0000 mon.c (mon.2) 578 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: cluster 2024-04-02T04:52:57.306661+0000 mon.a (mon.0) 2251 : cluster 0 osdmap e337: 8 total, 7 up, 8 in 2024-04-02T04:52:58.003 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:57 smithi098 bash[23790]: audit 2024-04-02T04:52:57.308363+0000 mon.a (mon.0) 2252 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]: dispatch 2024-04-02T04:52:58.019 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:58.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:58.147 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:52:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:52:58.250 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:58.275 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:58.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:58.403 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:52:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:52:58.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:58.533 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:58.636 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:58.663 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:58.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:58.789 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:52:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:52:58.892 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:58.904 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:52:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:56.821539+0000 osd.3 (osd.3) 237 : cluster 0 4.19 scrub starts 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:56.823608+0000 osd.3 (osd.3) 238 : cluster 0 4.19 scrub ok 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:57.361894+0000 osd.6 (osd.6) 163 : cluster 0 10.7 scrub starts 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:57.364224+0000 osd.6 (osd.6) 164 : cluster 0 10.7 scrub ok 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:57.639563+0000 osd.2 (osd.2) 165 : cluster 0 10.5 scrub starts 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:57.640860+0000 osd.2 (osd.2) 166 : cluster 0 10.5 scrub ok 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.864119+0000 mon.a (mon.0) 2253 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.864391+0000 mon.a (mon.0) 2254 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:58.905 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:57.872391+0000 mon.a (mon.0) 2255 : cluster 0 osdmap e338: 8 total, 7 up, 8 in 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.892143+0000 mon.c (mon.2) 579 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.893699+0000 mon.a (mon.0) 2256 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.917005+0000 mon.a (mon.0) 2257 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.918340+0000 mon.a (mon.0) 2258 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:57.918691+0000 mon.a (mon.0) 2259 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.025344+0000 mgr.y (mgr.14538) 369 : cluster 0 pgmap v484: 364 pgs: 32 creating+peering, 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 278/1536 objects degraded (18.099%) 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: audit 2024-04-02T04:52:58.069009+0000 mon.c (mon.2) 580 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.342857+0000 osd.4 (osd.4) 209 : cluster 0 40.17 scrub starts 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.345222+0000 osd.4 (osd.4) 210 : cluster 0 40.17 scrub ok 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.359435+0000 osd.1 (osd.1) 221 : cluster 0 41.a scrub starts 2024-04-02T04:52:58.906 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.361045+0000 osd.1 (osd.1) 222 : cluster 0 41.a scrub ok 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:56.821539+0000 osd.3 (osd.3) 237 : cluster 0 4.19 scrub starts 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:56.823608+0000 osd.3 (osd.3) 238 : cluster 0 4.19 scrub ok 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:57.361894+0000 osd.6 (osd.6) 163 : cluster 0 10.7 scrub starts 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:57.364224+0000 osd.6 (osd.6) 164 : cluster 0 10.7 scrub ok 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:57.639563+0000 osd.2 (osd.2) 165 : cluster 0 10.5 scrub starts 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:57.640860+0000 osd.2 (osd.2) 166 : cluster 0 10.5 scrub ok 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.864119+0000 mon.a (mon.0) 2253 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.864391+0000 mon.a (mon.0) 2254 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:57.872391+0000 mon.a (mon.0) 2255 : cluster 0 osdmap e338: 8 total, 7 up, 8 in 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.892143+0000 mon.c (mon.2) 579 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:58.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.893699+0000 mon.a (mon.0) 2256 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.917005+0000 mon.a (mon.0) 2257 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.918340+0000 mon.a (mon.0) 2258 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:57.918691+0000 mon.a (mon.0) 2259 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.025344+0000 mgr.y (mgr.14538) 369 : cluster 0 pgmap v484: 364 pgs: 32 creating+peering, 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 278/1536 objects degraded (18.099%) 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: audit 2024-04-02T04:52:58.069009+0000 mon.c (mon.2) 580 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.342857+0000 osd.4 (osd.4) 209 : cluster 0 40.17 scrub starts 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.345222+0000 osd.4 (osd.4) 210 : cluster 0 40.17 scrub ok 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.359435+0000 osd.1 (osd.1) 221 : cluster 0 41.a scrub starts 2024-04-02T04:52:58.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:58 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.361045+0000 osd.1 (osd.1) 222 : cluster 0 41.a scrub ok 2024-04-02T04:52:58.922 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:58.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:59.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:56.821539+0000 osd.3 (osd.3) 237 : cluster 0 4.19 scrub starts 2024-04-02T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:56.823608+0000 osd.3 (osd.3) 238 : cluster 0 4.19 scrub ok 2024-04-02T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:57.361894+0000 osd.6 (osd.6) 163 : cluster 0 10.7 scrub starts 2024-04-02T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:57.364224+0000 osd.6 (osd.6) 164 : cluster 0 10.7 scrub ok 2024-04-02T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:57.639563+0000 osd.2 (osd.2) 165 : cluster 0 10.5 scrub starts 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:57.640860+0000 osd.2 (osd.2) 166 : cluster 0 10.5 scrub ok 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.864119+0000 mon.a (mon.0) 2253 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-72145-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.864391+0000 mon.a (mon.0) 2254 : audit 1 from='client.? 172.21.15.69:0/250149322' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-72153-52"}]': finished 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:57.872391+0000 mon.a (mon.0) 2255 : cluster 0 osdmap e338: 8 total, 7 up, 8 in 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.892143+0000 mon.c (mon.2) 579 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.893699+0000 mon.a (mon.0) 2256 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.917005+0000 mon.a (mon.0) 2257 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.918340+0000 mon.a (mon.0) 2258 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:57.918691+0000 mon.a (mon.0) 2259 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.025344+0000 mgr.y (mgr.14538) 369 : cluster 0 pgmap v484: 364 pgs: 32 creating+peering, 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 278/1536 objects degraded (18.099%) 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: audit 2024-04-02T04:52:58.069009+0000 mon.c (mon.2) 580 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.342857+0000 osd.4 (osd.4) 209 : cluster 0 40.17 scrub starts 2024-04-02T04:52:59.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.345222+0000 osd.4 (osd.4) 210 : cluster 0 40.17 scrub ok 2024-04-02T04:52:59.039 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.359435+0000 osd.1 (osd.1) 221 : cluster 0 41.a scrub starts 2024-04-02T04:52:59.039 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:58 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.361045+0000 osd.1 (osd.1) 222 : cluster 0 41.a scrub ok 2024-04-02T04:52:59.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:59.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:59.181 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:52:59.284 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:59.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:59.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:59.542 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:52:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:52:59.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:52:59.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:59.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:52:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:52:59.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:52:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:52:59.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.689421+0000 osd.2 (osd.2) 167 : cluster 0 4.1c scrub starts 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.690488+0000 osd.2 (osd.2) 168 : cluster 0 4.1c scrub ok 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.804540+0000 osd.3 (osd.3) 239 : cluster 0 3.18 scrub starts 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.806319+0000 osd.3 (osd.3) 240 : cluster 0 3.18 scrub ok 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.865723+0000 mon.a (mon.0) 2260 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: audit 2024-04-02T04:52:58.867031+0000 mon.a (mon.0) 2261 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: audit 2024-04-02T04:52:58.867132+0000 mon.a (mon.0) 2262 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:58.873041+0000 mon.a (mon.0) 2263 : cluster 0 osdmap e339: 8 total, 7 up, 8 in 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: audit 2024-04-02T04:52:58.875834+0000 mon.a (mon.0) 2264 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-72153-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:59.912 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: audit 2024-04-02T04:52:58.879372+0000 mon.c (mon.2) 581 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: audit 2024-04-02T04:52:58.889820+0000 mon.a (mon.0) 2265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: audit 2024-04-02T04:52:59.070679+0000 mon.c (mon.2) 582 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:59.360788+0000 osd.6 (osd.6) 165 : cluster 0 41.1e scrub starts 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:59.362738+0000 osd.6 (osd.6) 166 : cluster 0 41.1e scrub ok 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:59.383217+0000 osd.1 (osd.1) 223 : cluster 0 4.7 scrub starts 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[21528]: cluster 2024-04-02T04:52:59.385093+0000 osd.1 (osd.1) 224 : cluster 0 4.7 scrub ok 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.689421+0000 osd.2 (osd.2) 167 : cluster 0 4.1c scrub starts 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.690488+0000 osd.2 (osd.2) 168 : cluster 0 4.1c scrub ok 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.804540+0000 osd.3 (osd.3) 239 : cluster 0 3.18 scrub starts 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.806319+0000 osd.3 (osd.3) 240 : cluster 0 3.18 scrub ok 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.865723+0000 mon.a (mon.0) 2260 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: audit 2024-04-02T04:52:58.867031+0000 mon.a (mon.0) 2261 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: audit 2024-04-02T04:52:58.867132+0000 mon.a (mon.0) 2262 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:59.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:58.873041+0000 mon.a (mon.0) 2263 : cluster 0 osdmap e339: 8 total, 7 up, 8 in 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: audit 2024-04-02T04:52:58.875834+0000 mon.a (mon.0) 2264 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-72153-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: audit 2024-04-02T04:52:58.879372+0000 mon.c (mon.2) 581 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: audit 2024-04-02T04:52:58.889820+0000 mon.a (mon.0) 2265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: audit 2024-04-02T04:52:59.070679+0000 mon.c (mon.2) 582 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:59.360788+0000 osd.6 (osd.6) 165 : cluster 0 41.1e scrub starts 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:59.362738+0000 osd.6 (osd.6) 166 : cluster 0 41.1e scrub ok 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:59.383217+0000 osd.1 (osd.1) 223 : cluster 0 4.7 scrub starts 2024-04-02T04:52:59.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:52:59 smithi069 bash[28904]: cluster 2024-04-02T04:52:59.385093+0000 osd.1 (osd.1) 224 : cluster 0 4.7 scrub ok 2024-04-02T04:52:59.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:59.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.689421+0000 osd.2 (osd.2) 167 : cluster 0 4.1c scrub starts 2024-04-02T04:52:59.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.690488+0000 osd.2 (osd.2) 168 : cluster 0 4.1c scrub ok 2024-04-02T04:52:59.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.804540+0000 osd.3 (osd.3) 239 : cluster 0 3.18 scrub starts 2024-04-02T04:52:59.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.806319+0000 osd.3 (osd.3) 240 : cluster 0 3.18 scrub ok 2024-04-02T04:52:59.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.865723+0000 mon.a (mon.0) 2260 : cluster 3 Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:52:59.947 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: audit 2024-04-02T04:52:58.867031+0000 mon.a (mon.0) 2261 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: audit 2024-04-02T04:52:58.867132+0000 mon.a (mon.0) 2262 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:58.873041+0000 mon.a (mon.0) 2263 : cluster 0 osdmap e339: 8 total, 7 up, 8 in 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: audit 2024-04-02T04:52:58.875834+0000 mon.a (mon.0) 2264 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-72153-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: audit 2024-04-02T04:52:58.879372+0000 mon.c (mon.2) 581 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: audit 2024-04-02T04:52:58.889820+0000 mon.a (mon.0) 2265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: audit 2024-04-02T04:52:59.070679+0000 mon.c (mon.2) 582 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:59.360788+0000 osd.6 (osd.6) 165 : cluster 0 41.1e scrub starts 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:59.362738+0000 osd.6 (osd.6) 166 : cluster 0 41.1e scrub ok 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:59.383217+0000 osd.1 (osd.1) 223 : cluster 0 4.7 scrub starts 2024-04-02T04:52:59.948 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:52:59 smithi098 bash[23790]: cluster 2024-04-02T04:52:59.385093+0000 osd.1 (osd.1) 224 : cluster 0 4.7 scrub ok 2024-04-02T04:52:59.959 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:52:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:52:59.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:00.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:00.087 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:00.189 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:00.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:00.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:00.343 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:00.446 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:00.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:00.572 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:00.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:00.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:00.722 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:00.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:00.852 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:00.955 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: audit 2024-04-02T04:52:59.896344+0000 mon.a (mon.0) 2266 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: audit 2024-04-02T04:52:59.908012+0000 mon.c (mon.2) 583 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.027034+0000 mgr.y (mgr.14538) 370 : cluster 0 pgmap v487: 356 pgs: 27 creating+peering, 5 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 278/1536 objects degraded (18.099%) 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: audit 2024-04-02T04:53:00.072187+0000 mon.c (mon.2) 584 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.343024+0000 mon.a (mon.0) 2267 : cluster 0 osdmap e340: 8 total, 7 up, 8 in 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: audit 2024-04-02T04:53:00.343741+0000 mon.a (mon.0) 2268 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: audit 2024-04-02T04:53:00.345010+0000 mon.c (mon.2) 585 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: audit 2024-04-02T04:53:00.345460+0000 mon.a (mon.0) 2269 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.380777+0000 osd.0 (osd.0) 205 : cluster 0 10.1e scrub starts 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.388667+0000 osd.0 (osd.0) 206 : cluster 0 10.1e scrub ok 2024-04-02T04:53:00.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: audit 2024-04-02T04:52:59.896344+0000 mon.a (mon.0) 2266 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: audit 2024-04-02T04:52:59.908012+0000 mon.c (mon.2) 583 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.027034+0000 mgr.y (mgr.14538) 370 : cluster 0 pgmap v487: 356 pgs: 27 creating+peering, 5 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 278/1536 objects degraded (18.099%) 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: audit 2024-04-02T04:53:00.072187+0000 mon.c (mon.2) 584 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.343024+0000 mon.a (mon.0) 2267 : cluster 0 osdmap e340: 8 total, 7 up, 8 in 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: audit 2024-04-02T04:53:00.343741+0000 mon.a (mon.0) 2268 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: audit 2024-04-02T04:53:00.345010+0000 mon.c (mon.2) 585 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: audit 2024-04-02T04:53:00.345460+0000 mon.a (mon.0) 2269 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.380777+0000 osd.0 (osd.0) 205 : cluster 0 10.1e scrub starts 2024-04-02T04:53:00.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.388667+0000 osd.0 (osd.0) 206 : cluster 0 10.1e scrub ok 2024-04-02T04:53:00.978 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:00.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:01.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:01.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: audit 2024-04-02T04:52:59.896344+0000 mon.a (mon.0) 2266 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:01.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: audit 2024-04-02T04:52:59.908012+0000 mon.c (mon.2) 583 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:01.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.027034+0000 mgr.y (mgr.14538) 370 : cluster 0 pgmap v487: 356 pgs: 27 creating+peering, 5 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 278/1536 objects degraded (18.099%) 2024-04-02T04:53:01.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: audit 2024-04-02T04:53:00.072187+0000 mon.c (mon.2) 584 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:01.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.343024+0000 mon.a (mon.0) 2267 : cluster 0 osdmap e340: 8 total, 7 up, 8 in 2024-04-02T04:53:01.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: audit 2024-04-02T04:53:00.343741+0000 mon.a (mon.0) 2268 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:01.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: audit 2024-04-02T04:53:00.345010+0000 mon.c (mon.2) 585 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: audit 2024-04-02T04:53:00.345460+0000 mon.a (mon.0) 2269 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.380777+0000 osd.0 (osd.0) 205 : cluster 0 10.1e scrub starts 2024-04-02T04:53:01.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.388667+0000 osd.0 (osd.0) 206 : cluster 0 10.1e scrub ok 2024-04-02T04:53:01.108 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:01.211 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:01.239 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:01.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:01.471 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:01.499 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:01.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:01.628 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:01.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:01.755 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:01.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:01.884 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:01.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:01.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.402797+0000 osd.6 (osd.6) 167 : cluster 0 40.4 scrub starts 2024-04-02T04:53:01.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.409602+0000 osd.6 (osd.6) 168 : cluster 0 40.4 scrub ok 2024-04-02T04:53:01.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.908541+0000 mon.a (mon.0) 2270 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-72153-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:01.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.908763+0000 mon.a (mon.0) 2271 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]': finished 2024-04-02T04:53:01.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.908931+0000 mon.a (mon.0) 2272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:53:01.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.920557+0000 mon.c (mon.2) 586 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: cluster 2024-04-02T04:53:00.925054+0000 mon.a (mon.0) 2273 : cluster 0 osdmap e341: 8 total, 7 up, 8 in 2024-04-02T04:53:01.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.929741+0000 mon.c (mon.2) 587 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]: dispatch 2024-04-02T04:53:01.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.932739+0000 mon.a (mon.0) 2274 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:00.934056+0000 mon.a (mon.0) 2275 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]: dispatch 2024-04-02T04:53:01.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: audit 2024-04-02T04:53:01.073448+0000 mon.c (mon.2) 588 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:01.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: cluster 2024-04-02T04:53:01.366653+0000 osd.6 (osd.6) 169 : cluster 0 13.1f scrub starts 2024-04-02T04:53:01.925 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:01 smithi098 bash[23790]: cluster 2024-04-02T04:53:01.369731+0000 osd.6 (osd.6) 170 : cluster 0 13.1f scrub ok 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: stte 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6955 ms) 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7061 ms) 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6971 ms) 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6976 ms) 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7007 ms) 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-02T04:53:01.932 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7079 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7042 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6983 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7045 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7340 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6628 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7066 ms) 2024-04-02T04:53:01.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7274 ms) 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6667 ms) 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6995 ms) 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7082 ms) 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112171 ms total) 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment tear-down 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (231982 ms total) 2024-04-02T04:53:01.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-02T04:53:01.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.402797+0000 osd.6 (osd.6) 167 : cluster 0 40.4 scrub starts 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.409602+0000 osd.6 (osd.6) 168 : cluster 0 40.4 scrub ok 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.908541+0000 mon.a (mon.0) 2270 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-72153-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.908763+0000 mon.a (mon.0) 2271 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]': finished 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.908931+0000 mon.a (mon.0) 2272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.920557+0000 mon.c (mon.2) 586 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: cluster 2024-04-02T04:53:00.925054+0000 mon.a (mon.0) 2273 : cluster 0 osdmap e341: 8 total, 7 up, 8 in 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.929741+0000 mon.c (mon.2) 587 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]: dispatch 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.932739+0000 mon.a (mon.0) 2274 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:00.934056+0000 mon.a (mon.0) 2275 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]: dispatch 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: audit 2024-04-02T04:53:01.073448+0000 mon.c (mon.2) 588 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:01.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: cluster 2024-04-02T04:53:01.366653+0000 osd.6 (osd.6) 169 : cluster 0 13.1f scrub starts 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[21528]: cluster 2024-04-02T04:53:01.369731+0000 osd.6 (osd.6) 170 : cluster 0 13.1f scrub ok 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.402797+0000 osd.6 (osd.6) 167 : cluster 0 40.4 scrub starts 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.409602+0000 osd.6 (osd.6) 168 : cluster 0 40.4 scrub ok 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.908541+0000 mon.a (mon.0) 2270 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-72153-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.908763+0000 mon.a (mon.0) 2271 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-37"}]': finished 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.908931+0000 mon.a (mon.0) 2272 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.920557+0000 mon.c (mon.2) 586 : audit 1 from='client.? 172.21.15.69:0/2598453363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: cluster 2024-04-02T04:53:00.925054+0000 mon.a (mon.0) 2273 : cluster 0 osdmap e341: 8 total, 7 up, 8 in 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.929741+0000 mon.c (mon.2) 587 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]: dispatch 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.932739+0000 mon.a (mon.0) 2274 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]: dispatch 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:00.934056+0000 mon.a (mon.0) 2275 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]: dispatch 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: audit 2024-04-02T04:53:01.073448+0000 mon.c (mon.2) 588 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:01.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: cluster 2024-04-02T04:53:01.366653+0000 osd.6 (osd.6) 169 : cluster 0 13.1f scrub starts 2024-04-02T04:53:01.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:01 smithi069 bash[28904]: cluster 2024-04-02T04:53:01.369731+0000 osd.6 (osd.6) 170 : cluster 0 13.1f scrub ok 2024-04-02T04:53:02.008 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:02.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:02.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:02.203 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:02.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:02.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:02.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:02.388 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:02.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:02.516 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:02.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:02.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:02.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:02.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:02.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:02.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-04-02T04:53:02.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:53:02.923 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-02T04:53:02.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:53:03.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:03.027 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:03.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:03.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:01.792128+0000 osd.3 (osd.3) 241 : cluster 0 2.1e scrub starts 2024-04-02T04:53:03.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:01.793650+0000 osd.3 (osd.3) 242 : cluster 0 2.1e scrub ok 2024-04-02T04:53:03.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:01.909195+0000 mon.a (mon.0) 2276 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:03.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:01.911233+0000 mon.a (mon.0) 2277 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:53:03.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:01.911329+0000 mon.a (mon.0) 2278 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]': finished 2024-04-02T04:53:03.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:01.915325+0000 mon.a (mon.0) 2279 : cluster 0 osdmap e342: 8 total, 7 up, 8 in 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:01.988873+0000 mon.c (mon.2) 589 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:01.989249+0000 mon.a (mon.0) 2280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:02.027977+0000 mgr.y (mgr.14538) 371 : cluster 0 pgmap v490: 364 pgs: 27 creating+peering, 13 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 278/1536 objects degraded (18.099%) 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:02.075241+0000 mon.c (mon.2) 590 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:02.291780+0000 osd.4 (osd.4) 211 : cluster 0 43.19 deep-scrub starts 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: cluster 2024-04-02T04:53:02.294298+0000 osd.4 (osd.4) 212 : cluster 0 43.19 deep-scrub ok 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:02.797482+0000 mon.b (mon.1) 201 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[28904]: audit 2024-04-02T04:53:02.798443+0000 mon.a (mon.0) 2281 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:01.792128+0000 osd.3 (osd.3) 241 : cluster 0 2.1e scrub starts 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:01.793650+0000 osd.3 (osd.3) 242 : cluster 0 2.1e scrub ok 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:01.909195+0000 mon.a (mon.0) 2276 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:01.911233+0000 mon.a (mon.0) 2277 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:53:03.139 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:01.911329+0000 mon.a (mon.0) 2278 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]': finished 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:01.915325+0000 mon.a (mon.0) 2279 : cluster 0 osdmap e342: 8 total, 7 up, 8 in 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:01.988873+0000 mon.c (mon.2) 589 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:01.989249+0000 mon.a (mon.0) 2280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:02.027977+0000 mgr.y (mgr.14538) 371 : cluster 0 pgmap v490: 364 pgs: 27 creating+peering, 13 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 278/1536 objects degraded (18.099%) 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:02.075241+0000 mon.c (mon.2) 590 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:02.291780+0000 osd.4 (osd.4) 211 : cluster 0 43.19 deep-scrub starts 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: cluster 2024-04-02T04:53:02.294298+0000 osd.4 (osd.4) 212 : cluster 0 43.19 deep-scrub ok 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:02.797482+0000 mon.b (mon.1) 201 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:03.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:02 smithi069 bash[21528]: audit 2024-04-02T04:53:02.798443+0000 mon.a (mon.0) 2281 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:03.149 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout:arted 0x55c99fcbaed0 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba420 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba530 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcba660 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb020 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbaa20 2024-04-02T04:53:03.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb83b0 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb84e0 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcb8630 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb620 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb170 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb2a0 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb3d0 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb500 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbbc00 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb750 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb880 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbb990 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbbac0 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc1e0 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbbd30 2024-04-02T04:53:03.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbbe60 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbbf90 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc0a0 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc7e0 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc330 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc460 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc590 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc930 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbce20 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbcf50 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbca60 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbcbb0 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbc6c0 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbcce0 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd550 2024-04-02T04:53:03.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd0a0 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd1f0 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd300 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd430 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbdb50 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd6a0 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd7d0 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbd8e0 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbda10 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe130 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbdc80 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbddd0 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbdf40 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe260 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe750 2024-04-02T04:53:03.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe880 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe390 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe4e0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe610 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbe9b0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbee80 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbefb0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbeac0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbec10 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbed60 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf0c0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf5b0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf6e0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbfca0 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf210 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf360 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf470 2024-04-02T04:53:03.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbfdd0 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf7f0 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbf960 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbfad0 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbff00 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc03d0 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0500 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0ae0 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0030 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0140 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0290 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0c30 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0630 2024-04-02T04:53:03.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0780 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:01.792128+0000 osd.3 (osd.3) 241 : cluster 0 2.1e scrub starts 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:01.793650+0000 osd.3 (osd.3) 242 : cluster 0 2.1e scrub ok 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:01.909195+0000 mon.a (mon.0) 2276 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:01.911233+0000 mon.a (mon.0) 2277 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-72145-42"}]': finished 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:01.911329+0000 mon.a (mon.0) 2278 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-37", "mode": "writeback"}]': finished 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:01.915325+0000 mon.a (mon.0) 2279 : cluster 0 osdmap e342: 8 total, 7 up, 8 in 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:01.988873+0000 mon.c (mon.2) 589 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:01.989249+0000 mon.a (mon.0) 2280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:02.027977+0000 mgr.y (mgr.14538) 371 : cluster 0 pgmap v490: 364 pgs: 27 creating+peering, 13 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 278/1536 objects degraded (18.099%) 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:02.075241+0000 mon.c (mon.2) 590 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:02.291780+0000 osd.4 (osd.4) 211 : cluster 0 43.19 deep-scrub starts 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: cluster 2024-04-02T04:53:02.294298+0000 osd.4 (osd.4) 212 : cluster 0 43.19 deep-scrub ok 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:02.797482+0000 mon.b (mon.1) 201 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:03.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:02 smithi098 bash[23790]: audit 2024-04-02T04:53:02.798443+0000 mon.a (mon.0) 2281 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:03.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:53:03.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:03.277 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:03.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:03.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:03.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:03.662 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:03.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:03.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:03.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:03.924 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:03.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:04.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:04.055 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:04.158 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:04.169 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: cluster 2024-04-02T04:53:02.766542+0000 osd.3 (osd.3) 243 : cluster 0 3.1e deep-scrub starts 2024-04-02T04:53:04.169 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: cluster 2024-04-02T04:53:02.777761+0000 osd.3 (osd.3) 244 : cluster 0 3.1e deep-scrub ok 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: audit 2024-04-02T04:53:02.930614+0000 mon.a (mon.0) 2282 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: audit 2024-04-02T04:53:02.937294+0000 mon.c (mon.2) 591 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: cluster 2024-04-02T04:53:02.937627+0000 mon.a (mon.0) 2283 : cluster 0 osdmap e343: 8 total, 7 up, 8 in 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: audit 2024-04-02T04:53:02.938751+0000 mon.a (mon.0) 2284 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: audit 2024-04-02T04:53:02.939143+0000 mon.a (mon.0) 2285 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: audit 2024-04-02T04:53:03.076517+0000 mon.c (mon.2) 592 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[28904]: audit 2024-04-02T04:53:03.173093+0000 mgr.y (mgr.14538) 372 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: cluster 2024-04-02T04:53:02.766542+0000 osd.3 (osd.3) 243 : cluster 0 3.1e deep-scrub starts 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: cluster 2024-04-02T04:53:02.777761+0000 osd.3 (osd.3) 244 : cluster 0 3.1e deep-scrub ok 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: audit 2024-04-02T04:53:02.930614+0000 mon.a (mon.0) 2282 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: audit 2024-04-02T04:53:02.937294+0000 mon.c (mon.2) 591 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:04.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: cluster 2024-04-02T04:53:02.937627+0000 mon.a (mon.0) 2283 : cluster 0 osdmap e343: 8 total, 7 up, 8 in 2024-04-02T04:53:04.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: audit 2024-04-02T04:53:02.938751+0000 mon.a (mon.0) 2284 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:04.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: audit 2024-04-02T04:53:02.939143+0000 mon.a (mon.0) 2285 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:53:04.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: audit 2024-04-02T04:53:03.076517+0000 mon.c (mon.2) 592 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:04.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:03 smithi069 bash[21528]: audit 2024-04-02T04:53:03.173093+0000 mgr.y (mgr.14538) 372 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:04.185 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: cluster 2024-04-02T04:53:02.766542+0000 osd.3 (osd.3) 243 : cluster 0 3.1e deep-scrub starts 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: cluster 2024-04-02T04:53:02.777761+0000 osd.3 (osd.3) 244 : cluster 0 3.1e deep-scrub ok 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: audit 2024-04-02T04:53:02.930614+0000 mon.a (mon.0) 2282 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: audit 2024-04-02T04:53:02.937294+0000 mon.c (mon.2) 591 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: cluster 2024-04-02T04:53:02.937627+0000 mon.a (mon.0) 2283 : cluster 0 osdmap e343: 8 total, 7 up, 8 in 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: audit 2024-04-02T04:53:02.938751+0000 mon.a (mon.0) 2284 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]: dispatch 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: audit 2024-04-02T04:53:02.939143+0000 mon.a (mon.0) 2285 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: audit 2024-04-02T04:53:03.076517+0000 mon.c (mon.2) 592 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:04.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:03 smithi098 bash[23790]: audit 2024-04-02T04:53:03.173093+0000 mgr.y (mgr.14538) 372 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:04.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:04.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:04.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:04.442 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:04.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:04.572 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:04.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:04.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:04.901 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:04.902 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:04.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:04.957 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:04.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:05.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:05.081 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:05.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:05.205 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:05.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:05.326 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:05.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:03.657604+0000 osd.5 (osd.5) 207 : cluster 0 6.d scrub starts 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:03.659583+0000 osd.5 (osd.5) 208 : cluster 0 6.d scrub ok 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:03.931710+0000 mon.a (mon.0) 2286 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: audit 2024-04-02T04:53:03.947330+0000 mon.a (mon.0) 2287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]': finished 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: audit 2024-04-02T04:53:03.947582+0000 mon.a (mon.0) 2288 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:03.954284+0000 mon.a (mon.0) 2289 : cluster 0 osdmap e344: 8 total, 7 up, 8 in 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: audit 2024-04-02T04:53:03.956866+0000 mon.a (mon.0) 2290 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.029805+0000 mgr.y (mgr.14538) 373 : cluster 0 pgmap v493: 356 pgs: 95 active+undersized, 224 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: audit 2024-04-02T04:53:04.077709+0000 mon.c (mon.2) 593 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.378109+0000 osd.4 (osd.4) 213 : cluster 0 4.9 scrub starts 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.380699+0000 osd.4 (osd.4) 214 : cluster 0 4.9 scrub ok 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.834002+0000 mon.a (mon.0) 2291 : cluster 3 Health check update: Degraded data redundancy: 278/1536 objects degraded (18.099%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:05.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.834062+0000 mon.a (mon.0) 2292 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: audit 2024-04-02T04:53:04.836608+0000 mon.a (mon.0) 2293 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.841764+0000 mon.a (mon.0) 2294 : cluster 0 osdmap e345: 8 total, 7 up, 8 in 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:03.657604+0000 osd.5 (osd.5) 207 : cluster 0 6.d scrub starts 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:03.659583+0000 osd.5 (osd.5) 208 : cluster 0 6.d scrub ok 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:03.931710+0000 mon.a (mon.0) 2286 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: audit 2024-04-02T04:53:03.947330+0000 mon.a (mon.0) 2287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]': finished 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: audit 2024-04-02T04:53:03.947582+0000 mon.a (mon.0) 2288 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:03.954284+0000 mon.a (mon.0) 2289 : cluster 0 osdmap e344: 8 total, 7 up, 8 in 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: audit 2024-04-02T04:53:03.956866+0000 mon.a (mon.0) 2290 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.029805+0000 mgr.y (mgr.14538) 373 : cluster 0 pgmap v493: 356 pgs: 95 active+undersized, 224 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: audit 2024-04-02T04:53:04.077709+0000 mon.c (mon.2) 593 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:05.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.378109+0000 osd.4 (osd.4) 213 : cluster 0 4.9 scrub starts 2024-04-02T04:53:05.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.380699+0000 osd.4 (osd.4) 214 : cluster 0 4.9 scrub ok 2024-04-02T04:53:05.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.834002+0000 mon.a (mon.0) 2291 : cluster 3 Health check update: Degraded data redundancy: 278/1536 objects degraded (18.099%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:05.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.834062+0000 mon.a (mon.0) 2292 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:05.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: audit 2024-04-02T04:53:04.836608+0000 mon.a (mon.0) 2293 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:05.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:05 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.841764+0000 mon.a (mon.0) 2294 : cluster 0 osdmap e345: 8 total, 7 up, 8 in 2024-04-02T04:53:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:05.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:05.566 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:03.657604+0000 osd.5 (osd.5) 207 : cluster 0 6.d scrub starts 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:03.659583+0000 osd.5 (osd.5) 208 : cluster 0 6.d scrub ok 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:03.931710+0000 mon.a (mon.0) 2286 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: audit 2024-04-02T04:53:03.947330+0000 mon.a (mon.0) 2287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-37"}]': finished 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: audit 2024-04-02T04:53:03.947582+0000 mon.a (mon.0) 2288 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:03.954284+0000 mon.a (mon.0) 2289 : cluster 0 osdmap e344: 8 total, 7 up, 8 in 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: audit 2024-04-02T04:53:03.956866+0000 mon.a (mon.0) 2290 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]: dispatch 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.029805+0000 mgr.y (mgr.14538) 373 : cluster 0 pgmap v493: 356 pgs: 95 active+undersized, 224 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: audit 2024-04-02T04:53:04.077709+0000 mon.c (mon.2) 593 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.378109+0000 osd.4 (osd.4) 213 : cluster 0 4.9 scrub starts 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.380699+0000 osd.4 (osd.4) 214 : cluster 0 4.9 scrub ok 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.834002+0000 mon.a (mon.0) 2291 : cluster 3 Health check update: Degraded data redundancy: 278/1536 objects degraded (18.099%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.834062+0000 mon.a (mon.0) 2292 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: audit 2024-04-02T04:53:04.836608+0000 mon.a (mon.0) 2293 : audit 1 from='client.? 172.21.15.69:0/4103513852' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-72153-53"}]': finished 2024-04-02T04:53:05.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:05 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.841764+0000 mon.a (mon.0) 2294 : cluster 0 osdmap e345: 8 total, 7 up, 8 in 2024-04-02T04:53:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:05.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:05.709 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:05.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:05.838 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:05.940 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:05.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:06.065 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:06.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.220 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:06.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:06.348 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:06.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.636560+0000 osd.5 (osd.5) 209 : cluster 0 13.9 scrub starts 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.638632+0000 osd.5 (osd.5) 210 : cluster 0 13.9 scrub ok 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.788608+0000 osd.3 (osd.3) 245 : cluster 0 3.b scrub starts 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:04.790912+0000 osd.3 (osd.3) 246 : cluster 0 3.b scrub ok 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:04.880401+0000 mon.c (mon.2) 594 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.079417+0000 mon.c (mon.2) 595 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:06.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.296748+0000 mon.a (mon.0) 2295 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.297278+0000 mon.c (mon.2) 596 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.304480+0000 mon.a (mon.0) 2296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.304984+0000 mon.c (mon.2) 597 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.309585+0000 mon.a (mon.0) 2297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:05.329567+0000 osd.0 (osd.0) 207 : cluster 0 2.13 scrub starts 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:05.331017+0000 osd.0 (osd.0) 208 : cluster 0 2.13 scrub ok 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:05.436093+0000 osd.1 (osd.1) 225 : cluster 0 2.15 scrub starts 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:05.438184+0000 osd.1 (osd.1) 226 : cluster 0 2.15 scrub ok 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.850526+0000 mon.a (mon.0) 2298 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.858583+0000 mon.c (mon.2) 598 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: cluster 2024-04-02T04:53:05.859517+0000 mon.a (mon.0) 2299 : cluster 0 osdmap e346: 8 total, 7 up, 8 in 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.862204+0000 mon.a (mon.0) 2300 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.867422+0000 mon.c (mon.2) 599 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:05.867917+0000 mon.a (mon.0) 2301 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:06.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:06 smithi098 bash[23790]: audit 2024-04-02T04:53:06.080877+0000 mon.c (mon.2) 600 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:06.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.609 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:06.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.636560+0000 osd.5 (osd.5) 209 : cluster 0 13.9 scrub starts 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.638632+0000 osd.5 (osd.5) 210 : cluster 0 13.9 scrub ok 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.788608+0000 osd.3 (osd.3) 245 : cluster 0 3.b scrub starts 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:04.790912+0000 osd.3 (osd.3) 246 : cluster 0 3.b scrub ok 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:04.880401+0000 mon.c (mon.2) 594 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.079417+0000 mon.c (mon.2) 595 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.296748+0000 mon.a (mon.0) 2295 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.297278+0000 mon.c (mon.2) 596 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.304480+0000 mon.a (mon.0) 2296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.304984+0000 mon.c (mon.2) 597 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:06.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.309585+0000 mon.a (mon.0) 2297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:05.329567+0000 osd.0 (osd.0) 207 : cluster 0 2.13 scrub starts 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:05.331017+0000 osd.0 (osd.0) 208 : cluster 0 2.13 scrub ok 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:05.436093+0000 osd.1 (osd.1) 225 : cluster 0 2.15 scrub starts 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:05.438184+0000 osd.1 (osd.1) 226 : cluster 0 2.15 scrub ok 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.850526+0000 mon.a (mon.0) 2298 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.858583+0000 mon.c (mon.2) 598 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: cluster 2024-04-02T04:53:05.859517+0000 mon.a (mon.0) 2299 : cluster 0 osdmap e346: 8 total, 7 up, 8 in 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.862204+0000 mon.a (mon.0) 2300 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.636560+0000 osd.5 (osd.5) 209 : cluster 0 13.9 scrub starts 2024-04-02T04:53:06.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.638632+0000 osd.5 (osd.5) 210 : cluster 0 13.9 scrub ok 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.788608+0000 osd.3 (osd.3) 245 : cluster 0 3.b scrub starts 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:04.790912+0000 osd.3 (osd.3) 246 : cluster 0 3.b scrub ok 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:04.880401+0000 mon.c (mon.2) 594 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.079417+0000 mon.c (mon.2) 595 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.296748+0000 mon.a (mon.0) 2295 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.297278+0000 mon.c (mon.2) 596 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.304480+0000 mon.a (mon.0) 2296 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.304984+0000 mon.c (mon.2) 597 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.309585+0000 mon.a (mon.0) 2297 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:05.329567+0000 osd.0 (osd.0) 207 : cluster 0 2.13 scrub starts 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:05.331017+0000 osd.0 (osd.0) 208 : cluster 0 2.13 scrub ok 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:05.436093+0000 osd.1 (osd.1) 225 : cluster 0 2.15 scrub starts 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:05.438184+0000 osd.1 (osd.1) 226 : cluster 0 2.15 scrub ok 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.850526+0000 mon.a (mon.0) 2298 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-72153-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.858583+0000 mon.c (mon.2) 598 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: cluster 2024-04-02T04:53:05.859517+0000 mon.a (mon.0) 2299 : cluster 0 osdmap e346: 8 total, 7 up, 8 in 2024-04-02T04:53:06.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.862204+0000 mon.a (mon.0) 2300 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:06.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.867422+0000 mon.c (mon.2) 599 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:06.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:05.867917+0000 mon.a (mon.0) 2301 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:06.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[21528]: audit 2024-04-02T04:53:06.080877+0000 mon.c (mon.2) 600 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:06.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.867422+0000 mon.c (mon.2) 599 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:06.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:05.867917+0000 mon.a (mon.0) 2301 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:06.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:06 smithi069 bash[28904]: audit 2024-04-02T04:53:06.080877+0000 mon.c (mon.2) 600 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:06.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:06.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:06.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:06.982 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:53:06.982 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:53:06.999 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:07.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:07.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.257 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:07.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:07.380 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:07.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:07 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.030825+0000 mgr.y (mgr.14538) 374 : cluster 0 pgmap v496: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:07.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:07 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.380158+0000 osd.0 (osd.0) 209 : cluster 0 13.12 scrub starts 2024-04-02T04:53:07.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:07 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.381928+0000 osd.0 (osd.0) 210 : cluster 0 13.12 scrub ok 2024-04-02T04:53:07.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:07 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.391963+0000 osd.1 (osd.1) 227 : cluster 0 13.10 scrub starts 2024-04-02T04:53:07.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:07 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.393739+0000 osd.1 (osd.1) 228 : cluster 0 13.10 scrub ok 2024-04-02T04:53:07.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:07 smithi098 bash[23790]: audit 2024-04-02T04:53:06.860676+0000 mon.a (mon.0) 2302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:07.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.510 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:07.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.030825+0000 mgr.y (mgr.14538) 374 : cluster 0 pgmap v496: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:07.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.380158+0000 osd.0 (osd.0) 209 : cluster 0 13.12 scrub starts 2024-04-02T04:53:07.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.381928+0000 osd.0 (osd.0) 210 : cluster 0 13.12 scrub ok 2024-04-02T04:53:07.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.391963+0000 osd.1 (osd.1) 227 : cluster 0 13.10 scrub starts 2024-04-02T04:53:07.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.393739+0000 osd.1 (osd.1) 228 : cluster 0 13.10 scrub ok 2024-04-02T04:53:07.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[21528]: audit 2024-04-02T04:53:06.860676+0000 mon.a (mon.0) 2302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:07.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.030825+0000 mgr.y (mgr.14538) 374 : cluster 0 pgmap v496: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:07.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.380158+0000 osd.0 (osd.0) 209 : cluster 0 13.12 scrub starts 2024-04-02T04:53:07.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.381928+0000 osd.0 (osd.0) 210 : cluster 0 13.12 scrub ok 2024-04-02T04:53:07.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.391963+0000 osd.1 (osd.1) 227 : cluster 0 13.10 scrub starts 2024-04-02T04:53:07.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.393739+0000 osd.1 (osd.1) 228 : cluster 0 13.10 scrub ok 2024-04-02T04:53:07.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:07 smithi069 bash[28904]: audit 2024-04-02T04:53:06.860676+0000 mon.a (mon.0) 2302 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:07.612 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:07.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:07.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.898 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:07.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:08.000 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:08.027 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:08.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:08.157 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:08.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:08.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:08.416 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:08.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:08.545 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:08.648 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.474157+0000 osd.6 (osd.6) 171 : cluster 0 43.0 scrub starts 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:06.475979+0000 osd.6 (osd.6) 172 : cluster 0 43.0 scrub ok 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:06.876435+0000 mon.c (mon.2) 601 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:07.082586+0000 mon.c (mon.2) 602 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.306252+0000 mon.a (mon.0) 2303 : cluster 0 osdmap e347: 8 total, 7 up, 8 in 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:07.307912+0000 mon.a (mon.0) 2304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.386661+0000 osd.4 (osd.4) 215 : cluster 0 41.17 scrub starts 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.388939+0000 osd.4 (osd.4) 216 : cluster 0 41.17 scrub ok 2024-04-02T04:53:08.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.409072+0000 osd.0 (osd.0) 211 : cluster 0 3.6 scrub starts 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.411386+0000 osd.0 (osd.0) 212 : cluster 0 3.6 scrub ok 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.784220+0000 osd.2 (osd.2) 169 : cluster 0 4.f scrub starts 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.785749+0000 osd.2 (osd.2) 170 : cluster 0 4.f scrub ok 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:07.864218+0000 mon.a (mon.0) 2305 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:07.864452+0000 mon.a (mon.0) 2306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:07.870983+0000 mon.c (mon.2) 603 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.874326+0000 mon.a (mon.0) 2307 : cluster 0 osdmap e348: 8 total, 7 up, 8 in 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:07.886607+0000 mon.a (mon.0) 2308 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[28904]: audit 2024-04-02T04:53:08.083568+0000 mon.c (mon.2) 604 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.474157+0000 osd.6 (osd.6) 171 : cluster 0 43.0 scrub starts 2024-04-02T04:53:08.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:06.475979+0000 osd.6 (osd.6) 172 : cluster 0 43.0 scrub ok 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:06.876435+0000 mon.c (mon.2) 601 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:07.082586+0000 mon.c (mon.2) 602 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.306252+0000 mon.a (mon.0) 2303 : cluster 0 osdmap e347: 8 total, 7 up, 8 in 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:07.307912+0000 mon.a (mon.0) 2304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.386661+0000 osd.4 (osd.4) 215 : cluster 0 41.17 scrub starts 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.388939+0000 osd.4 (osd.4) 216 : cluster 0 41.17 scrub ok 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.409072+0000 osd.0 (osd.0) 211 : cluster 0 3.6 scrub starts 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.411386+0000 osd.0 (osd.0) 212 : cluster 0 3.6 scrub ok 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.784220+0000 osd.2 (osd.2) 169 : cluster 0 4.f scrub starts 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.785749+0000 osd.2 (osd.2) 170 : cluster 0 4.f scrub ok 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:07.864218+0000 mon.a (mon.0) 2305 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:08.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:07.864452+0000 mon.a (mon.0) 2306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:08.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:07.870983+0000 mon.c (mon.2) 603 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:08.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.874326+0000 mon.a (mon.0) 2307 : cluster 0 osdmap e348: 8 total, 7 up, 8 in 2024-04-02T04:53:08.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:07.886607+0000 mon.a (mon.0) 2308 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:08.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21528]: audit 2024-04-02T04:53:08.083568+0000 mon.c (mon.2) 604 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.474157+0000 osd.6 (osd.6) 171 : cluster 0 43.0 scrub starts 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:06.475979+0000 osd.6 (osd.6) 172 : cluster 0 43.0 scrub ok 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:06.876435+0000 mon.c (mon.2) 601 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:07.082586+0000 mon.c (mon.2) 602 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.306252+0000 mon.a (mon.0) 2303 : cluster 0 osdmap e347: 8 total, 7 up, 8 in 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:07.307912+0000 mon.a (mon.0) 2304 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.386661+0000 osd.4 (osd.4) 215 : cluster 0 41.17 scrub starts 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.388939+0000 osd.4 (osd.4) 216 : cluster 0 41.17 scrub ok 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.409072+0000 osd.0 (osd.0) 211 : cluster 0 3.6 scrub starts 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.411386+0000 osd.0 (osd.0) 212 : cluster 0 3.6 scrub ok 2024-04-02T04:53:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.784220+0000 osd.2 (osd.2) 169 : cluster 0 4.f scrub starts 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.785749+0000 osd.2 (osd.2) 170 : cluster 0 4.f scrub ok 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:07.864218+0000 mon.a (mon.0) 2305 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-72153-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:07.864452+0000 mon.a (mon.0) 2306 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:07.870983+0000 mon.c (mon.2) 603 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.874326+0000 mon.a (mon.0) 2307 : cluster 0 osdmap e348: 8 total, 7 up, 8 in 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:07.886607+0000 mon.a (mon.0) 2308 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:08 smithi098 bash[23790]: audit 2024-04-02T04:53:08.083568+0000 mon.c (mon.2) 604 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:08.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:08.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:08.917 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:53:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:53:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:53:08.930 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:08.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:09.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:09.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:09.192 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:09.295 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:09.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.465404+0000 osd.6 (osd.6) 173 : cluster 0 40.14 scrub starts 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.467838+0000 osd.6 (osd.6) 174 : cluster 0 40.14 scrub ok 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.597575+0000 osd.5 (osd.5) 211 : cluster 0 3.c scrub starts 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: cluster 2024-04-02T04:53:07.600369+0000 osd.5 (osd.5) 212 : cluster 0 3.c scrub ok 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: cluster 2024-04-02T04:53:08.032889+0000 mgr.y (mgr.14538) 375 : cluster 0 pgmap v499: 364 pgs: 8 creating+peering, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: audit 2024-04-02T04:53:08.873968+0000 mon.a (mon.0) 2309 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]': finished 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: audit 2024-04-02T04:53:08.883303+0000 mon.c (mon.2) 605 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]: dispatch 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: cluster 2024-04-02T04:53:08.883306+0000 mon.a (mon.0) 2310 : cluster 0 osdmap e349: 8 total, 7 up, 8 in 2024-04-02T04:53:09.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: audit 2024-04-02T04:53:08.889974+0000 mon.a (mon.0) 2311 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]: dispatch 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[21528]: audit 2024-04-02T04:53:09.084613+0000 mon.c (mon.2) 606 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.465404+0000 osd.6 (osd.6) 173 : cluster 0 40.14 scrub starts 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.467838+0000 osd.6 (osd.6) 174 : cluster 0 40.14 scrub ok 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.597575+0000 osd.5 (osd.5) 211 : cluster 0 3.c scrub starts 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: cluster 2024-04-02T04:53:07.600369+0000 osd.5 (osd.5) 212 : cluster 0 3.c scrub ok 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: cluster 2024-04-02T04:53:08.032889+0000 mgr.y (mgr.14538) 375 : cluster 0 pgmap v499: 364 pgs: 8 creating+peering, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: audit 2024-04-02T04:53:08.873968+0000 mon.a (mon.0) 2309 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]': finished 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: audit 2024-04-02T04:53:08.883303+0000 mon.c (mon.2) 605 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]: dispatch 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: cluster 2024-04-02T04:53:08.883306+0000 mon.a (mon.0) 2310 : cluster 0 osdmap e349: 8 total, 7 up, 8 in 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: audit 2024-04-02T04:53:08.889974+0000 mon.a (mon.0) 2311 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]: dispatch 2024-04-02T04:53:09.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:09 smithi069 bash[28904]: audit 2024-04-02T04:53:09.084613+0000 mon.c (mon.2) 606 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:09.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:09.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.465404+0000 osd.6 (osd.6) 173 : cluster 0 40.14 scrub starts 2024-04-02T04:53:09.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.467838+0000 osd.6 (osd.6) 174 : cluster 0 40.14 scrub ok 2024-04-02T04:53:09.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.597575+0000 osd.5 (osd.5) 211 : cluster 0 3.c scrub starts 2024-04-02T04:53:09.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: cluster 2024-04-02T04:53:07.600369+0000 osd.5 (osd.5) 212 : cluster 0 3.c scrub ok 2024-04-02T04:53:09.567 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: cluster 2024-04-02T04:53:08.032889+0000 mgr.y (mgr.14538) 375 : cluster 0 pgmap v499: 364 pgs: 8 creating+peering, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:09.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: audit 2024-04-02T04:53:08.873968+0000 mon.a (mon.0) 2309 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-39"}]': finished 2024-04-02T04:53:09.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: audit 2024-04-02T04:53:08.883303+0000 mon.c (mon.2) 605 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]: dispatch 2024-04-02T04:53:09.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: cluster 2024-04-02T04:53:08.883306+0000 mon.a (mon.0) 2310 : cluster 0 osdmap e349: 8 total, 7 up, 8 in 2024-04-02T04:53:09.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: audit 2024-04-02T04:53:08.889974+0000 mon.a (mon.0) 2311 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]: dispatch 2024-04-02T04:53:09.568 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:09 smithi098 bash[23790]: audit 2024-04-02T04:53:09.084613+0000 mon.c (mon.2) 606 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:09.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:09.712 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:09.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:09.842 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:09.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-02T04:53:09.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:09.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:10.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:10.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:10.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:10.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:10.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:10.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:08.574840+0000 osd.5 (osd.5) 213 : cluster 0 43.6 scrub starts 2024-04-02T04:53:10.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:08.577214+0000 osd.5 (osd.5) 214 : cluster 0 43.6 scrub ok 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:09.819128+0000 osd.2 (osd.2) 171 : cluster 0 10.18 scrub starts 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:09.820691+0000 osd.2 (osd.2) 172 : cluster 0 10.18 scrub ok 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:09.836044+0000 mon.a (mon.0) 2312 : cluster 3 Health check update: Degraded data redundancy: 278/1539 objects degraded (18.064%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:09.874276+0000 mon.a (mon.0) 2313 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: audit 2024-04-02T04:53:09.909511+0000 mon.a (mon.0) 2314 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]': finished 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: cluster 2024-04-02T04:53:09.914504+0000 mon.a (mon.0) 2315 : cluster 0 osdmap e350: 8 total, 7 up, 8 in 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: audit 2024-04-02T04:53:09.915701+0000 mon.c (mon.2) 607 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: audit 2024-04-02T04:53:09.916068+0000 mon.a (mon.0) 2316 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: audit 2024-04-02T04:53:10.039604+0000 mon.c (mon.2) 608 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: audit 2024-04-02T04:53:10.039995+0000 mon.a (mon.0) 2317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[28904]: audit 2024-04-02T04:53:10.085674+0000 mon.c (mon.2) 609 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:10.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:08.574840+0000 osd.5 (osd.5) 213 : cluster 0 43.6 scrub starts 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:08.577214+0000 osd.5 (osd.5) 214 : cluster 0 43.6 scrub ok 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:09.819128+0000 osd.2 (osd.2) 171 : cluster 0 10.18 scrub starts 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:09.820691+0000 osd.2 (osd.2) 172 : cluster 0 10.18 scrub ok 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:09.836044+0000 mon.a (mon.0) 2312 : cluster 3 Health check update: Degraded data redundancy: 278/1539 objects degraded (18.064%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:09.874276+0000 mon.a (mon.0) 2313 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: audit 2024-04-02T04:53:09.909511+0000 mon.a (mon.0) 2314 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]': finished 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: cluster 2024-04-02T04:53:09.914504+0000 mon.a (mon.0) 2315 : cluster 0 osdmap e350: 8 total, 7 up, 8 in 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: audit 2024-04-02T04:53:09.915701+0000 mon.c (mon.2) 607 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: audit 2024-04-02T04:53:09.916068+0000 mon.a (mon.0) 2316 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: audit 2024-04-02T04:53:10.039604+0000 mon.c (mon.2) 608 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: audit 2024-04-02T04:53:10.039995+0000 mon.a (mon.0) 2317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:10.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:10 smithi069 bash[21528]: audit 2024-04-02T04:53:10.085674+0000 mon.c (mon.2) 609 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:10.475 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:10.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:08.574840+0000 osd.5 (osd.5) 213 : cluster 0 43.6 scrub starts 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:08.577214+0000 osd.5 (osd.5) 214 : cluster 0 43.6 scrub ok 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:09.819128+0000 osd.2 (osd.2) 171 : cluster 0 10.18 scrub starts 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:09.820691+0000 osd.2 (osd.2) 172 : cluster 0 10.18 scrub ok 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:09.836044+0000 mon.a (mon.0) 2312 : cluster 3 Health check update: Degraded data redundancy: 278/1539 objects degraded (18.064%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:09.874276+0000 mon.a (mon.0) 2313 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: audit 2024-04-02T04:53:09.909511+0000 mon.a (mon.0) 2314 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-39", "mode": "writeback"}]': finished 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: cluster 2024-04-02T04:53:09.914504+0000 mon.a (mon.0) 2315 : cluster 0 osdmap e350: 8 total, 7 up, 8 in 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: audit 2024-04-02T04:53:09.915701+0000 mon.c (mon.2) 607 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: audit 2024-04-02T04:53:09.916068+0000 mon.a (mon.0) 2316 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: audit 2024-04-02T04:53:10.039604+0000 mon.c (mon.2) 608 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:10.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: audit 2024-04-02T04:53:10.039995+0000 mon.a (mon.0) 2317 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:10.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:10 smithi098 bash[23790]: audit 2024-04-02T04:53:10.085674+0000 mon.c (mon.2) 609 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:10.605 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:10.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:10.736 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:10.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:10.971 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:10.994 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:10.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:11.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:11.121 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:11.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:11.252 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:11.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:11.384 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:11.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:11.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:11.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.033767+0000 mgr.y (mgr.14538) 376 : cluster 0 pgmap v502: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.334182+0000 osd.0 (osd.0) 213 : cluster 0 10.4 scrub starts 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.335597+0000 osd.0 (osd.0) 214 : cluster 0 10.4 scrub ok 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.414029+0000 osd.1 (osd.1) 229 : cluster 0 2.d deep-scrub starts 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.415761+0000 osd.1 (osd.1) 230 : cluster 0 2.d deep-scrub ok 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.910769+0000 mon.a (mon.0) 2318 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:10.912259+0000 mon.a (mon.0) 2319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:10.912495+0000 mon.a (mon.0) 2320 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:10.922458+0000 mon.c (mon.2) 610 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:10.922913+0000 mon.c (mon.2) 611 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:11.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.925976+0000 mon.a (mon.0) 2321 : cluster 0 osdmap e351: 8 total, 7 up, 8 in 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:10.930253+0000 mon.a (mon.0) 2322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:10.931161+0000 mon.a (mon.0) 2323 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[28904]: audit 2024-04-02T04:53:11.086768+0000 mon.c (mon.2) 612 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.033767+0000 mgr.y (mgr.14538) 376 : cluster 0 pgmap v502: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.334182+0000 osd.0 (osd.0) 213 : cluster 0 10.4 scrub starts 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.335597+0000 osd.0 (osd.0) 214 : cluster 0 10.4 scrub ok 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.414029+0000 osd.1 (osd.1) 229 : cluster 0 2.d deep-scrub starts 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.415761+0000 osd.1 (osd.1) 230 : cluster 0 2.d deep-scrub ok 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.910769+0000 mon.a (mon.0) 2318 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:10.912259+0000 mon.a (mon.0) 2319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:11.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:10.912495+0000 mon.a (mon.0) 2320 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:53:11.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:10.922458+0000 mon.c (mon.2) 610 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:11.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:10.922913+0000 mon.c (mon.2) 611 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:11.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.925976+0000 mon.a (mon.0) 2321 : cluster 0 osdmap e351: 8 total, 7 up, 8 in 2024-04-02T04:53:11.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:10.930253+0000 mon.a (mon.0) 2322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:11.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:10.931161+0000 mon.a (mon.0) 2323 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:11.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:11 smithi069 bash[21528]: audit 2024-04-02T04:53:11.086768+0000 mon.c (mon.2) 612 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:11.638 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.033767+0000 mgr.y (mgr.14538) 376 : cluster 0 pgmap v502: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.334182+0000 osd.0 (osd.0) 213 : cluster 0 10.4 scrub starts 2024-04-02T04:53:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.335597+0000 osd.0 (osd.0) 214 : cluster 0 10.4 scrub ok 2024-04-02T04:53:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.414029+0000 osd.1 (osd.1) 229 : cluster 0 2.d deep-scrub starts 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.415761+0000 osd.1 (osd.1) 230 : cluster 0 2.d deep-scrub ok 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.910769+0000 mon.a (mon.0) 2318 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:10.912259+0000 mon.a (mon.0) 2319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:10.912495+0000 mon.a (mon.0) 2320 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:10.922458+0000 mon.c (mon.2) 610 : audit 1 from='client.? 172.21.15.69:0/3675239409' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:10.922913+0000 mon.c (mon.2) 611 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.925976+0000 mon.a (mon.0) 2321 : cluster 0 osdmap e351: 8 total, 7 up, 8 in 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:10.930253+0000 mon.a (mon.0) 2322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]: dispatch 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:10.931161+0000 mon.a (mon.0) 2323 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]: dispatch 2024-04-02T04:53:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:11 smithi098 bash[23790]: audit 2024-04-02T04:53:11.086768+0000 mon.c (mon.2) 612 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:11.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:11.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:11.890 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:11.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:11.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:12.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:12.135 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:12.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:12.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:12.388 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:12.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.490085+0000 osd.6 (osd.6) 175 : cluster 0 41.4 scrub starts 2024-04-02T04:53:12.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: cluster 2024-04-02T04:53:10.491889+0000 osd.6 (osd.6) 176 : cluster 0 41.4 scrub ok 2024-04-02T04:53:12.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: cluster 2024-04-02T04:53:11.305956+0000 osd.0 (osd.0) 215 : cluster 0 6.0 scrub starts 2024-04-02T04:53:12.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: cluster 2024-04-02T04:53:11.307242+0000 osd.0 (osd.0) 216 : cluster 0 6.0 scrub ok 2024-04-02T04:53:12.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.375125+0000 mon.b (mon.1) 202 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:12.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: cluster 2024-04-02T04:53:11.912729+0000 mon.a (mon.0) 2324 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.914182+0000 mon.a (mon.0) 2325 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.914279+0000 mon.a (mon.0) 2326 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]': finished 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: cluster 2024-04-02T04:53:11.919442+0000 mon.a (mon.0) 2327 : cluster 0 osdmap e352: 8 total, 7 up, 8 in 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.950427+0000 mon.c (mon.2) 613 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.950875+0000 mon.a (mon.0) 2328 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.951505+0000 mon.c (mon.2) 614 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.951763+0000 mon.a (mon.0) 2329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.952274+0000 mon.c (mon.2) 615 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:11.952523+0000 mon.a (mon.0) 2330 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:12.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:12.087889+0000 mon.c (mon.2) 616 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:12.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:12.109171+0000 mon.b (mon.1) 203 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:12.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:12.110450+0000 mon.b (mon.1) 204 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:12.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[21528]: audit 2024-04-02T04:53:12.120021+0000 mon.a (mon.0) 2331 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:12.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.490085+0000 osd.6 (osd.6) 175 : cluster 0 41.4 scrub starts 2024-04-02T04:53:12.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: cluster 2024-04-02T04:53:10.491889+0000 osd.6 (osd.6) 176 : cluster 0 41.4 scrub ok 2024-04-02T04:53:12.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: cluster 2024-04-02T04:53:11.305956+0000 osd.0 (osd.0) 215 : cluster 0 6.0 scrub starts 2024-04-02T04:53:12.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: cluster 2024-04-02T04:53:11.307242+0000 osd.0 (osd.0) 216 : cluster 0 6.0 scrub ok 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.375125+0000 mon.b (mon.1) 202 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: cluster 2024-04-02T04:53:11.912729+0000 mon.a (mon.0) 2324 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.914182+0000 mon.a (mon.0) 2325 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.914279+0000 mon.a (mon.0) 2326 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]': finished 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: cluster 2024-04-02T04:53:11.919442+0000 mon.a (mon.0) 2327 : cluster 0 osdmap e352: 8 total, 7 up, 8 in 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.950427+0000 mon.c (mon.2) 613 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.950875+0000 mon.a (mon.0) 2328 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.951505+0000 mon.c (mon.2) 614 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.951763+0000 mon.a (mon.0) 2329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.952274+0000 mon.c (mon.2) 615 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:12.393 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:11.952523+0000 mon.a (mon.0) 2330 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:12.087889+0000 mon.c (mon.2) 616 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:12.109171+0000 mon.b (mon.1) 203 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:12.110450+0000 mon.b (mon.1) 204 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:12 smithi069 bash[28904]: audit 2024-04-02T04:53:12.120021+0000 mon.a (mon.0) 2331 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:12.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:12.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:12.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:12.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.490085+0000 osd.6 (osd.6) 175 : cluster 0 41.4 scrub starts 2024-04-02T04:53:12.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: cluster 2024-04-02T04:53:10.491889+0000 osd.6 (osd.6) 176 : cluster 0 41.4 scrub ok 2024-04-02T04:53:12.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: cluster 2024-04-02T04:53:11.305956+0000 osd.0 (osd.0) 215 : cluster 0 6.0 scrub starts 2024-04-02T04:53:12.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: cluster 2024-04-02T04:53:11.307242+0000 osd.0 (osd.0) 216 : cluster 0 6.0 scrub ok 2024-04-02T04:53:12.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.375125+0000 mon.b (mon.1) 202 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: cluster 2024-04-02T04:53:11.912729+0000 mon.a (mon.0) 2324 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.914182+0000 mon.a (mon.0) 2325 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-72153-54"}]': finished 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.914279+0000 mon.a (mon.0) 2326 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-39"}]': finished 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: cluster 2024-04-02T04:53:11.919442+0000 mon.a (mon.0) 2327 : cluster 0 osdmap e352: 8 total, 7 up, 8 in 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.950427+0000 mon.c (mon.2) 613 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.950875+0000 mon.a (mon.0) 2328 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.951505+0000 mon.c (mon.2) 614 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.951763+0000 mon.a (mon.0) 2329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.952274+0000 mon.c (mon.2) 615 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:11.952523+0000 mon.a (mon.0) 2330 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:12.087889+0000 mon.c (mon.2) 616 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:12.109171+0000 mon.b (mon.1) 203 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:12.110450+0000 mon.b (mon.1) 204 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:12 smithi098 bash[23790]: audit 2024-04-02T04:53:12.120021+0000 mon.a (mon.0) 2331 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:12.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:12.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:12.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: seed 72349 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (1307 ms) 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (25506 ms) 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (14 ms) 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (26827 ms total) 2024-04-02T04:53:12.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (10755 ms) 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (9168 ms) 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9851 ms) 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [3] 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [4,3] 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-02T04:53:12.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting some heads 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47188 ms) 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9591 ms) 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7965 ms) 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-02T04:53:12.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7989 ms) 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8004 ms) 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10090 ms) 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9016 ms) 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10894 ms) 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12146 ms) 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-02T04:53:12.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8848 ms) 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7474 ms) 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13115 ms) 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7539 ms) 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-02T04:53:12.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8093 ms) 2024-04-02T04:53:13.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:13.039 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:13.142 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:13.169 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:13.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:13.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:13.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:53:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: cluster 2024-04-02T04:53:11.471775+0000 osd.6 (osd.6) 177 : cluster 0 43.e scrub starts 2024-04-02T04:53:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: cluster 2024-04-02T04:53:11.473091+0000 osd.6 (osd.6) 178 : cluster 0 43.e scrub ok 2024-04-02T04:53:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: cluster 2024-04-02T04:53:12.034520+0000 mgr.y (mgr.14538) 377 : cluster 0 pgmap v505: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: audit 2024-04-02T04:53:12.925776+0000 mon.a (mon.0) 2332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: cluster 2024-04-02T04:53:12.934567+0000 mon.a (mon.0) 2333 : cluster 0 osdmap e353: 8 total, 7 up, 8 in 2024-04-02T04:53:13.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: audit 2024-04-02T04:53:12.939466+0000 mon.c (mon.2) 617 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:13.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:13 smithi098 bash[23790]: audit 2024-04-02T04:53:12.940763+0000 mon.a (mon.0) 2334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: cluster 2024-04-02T04:53:11.471775+0000 osd.6 (osd.6) 177 : cluster 0 43.e scrub starts 2024-04-02T04:53:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: cluster 2024-04-02T04:53:11.473091+0000 osd.6 (osd.6) 178 : cluster 0 43.e scrub ok 2024-04-02T04:53:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: cluster 2024-04-02T04:53:12.034520+0000 mgr.y (mgr.14538) 377 : cluster 0 pgmap v505: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: audit 2024-04-02T04:53:12.925776+0000 mon.a (mon.0) 2332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:13.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: cluster 2024-04-02T04:53:12.934567+0000 mon.a (mon.0) 2333 : cluster 0 osdmap e353: 8 total, 7 up, 8 in 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: audit 2024-04-02T04:53:12.939466+0000 mon.c (mon.2) 617 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[21528]: audit 2024-04-02T04:53:12.940763+0000 mon.a (mon.0) 2334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: cluster 2024-04-02T04:53:11.471775+0000 osd.6 (osd.6) 177 : cluster 0 43.e scrub starts 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: cluster 2024-04-02T04:53:11.473091+0000 osd.6 (osd.6) 178 : cluster 0 43.e scrub ok 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: cluster 2024-04-02T04:53:12.034520+0000 mgr.y (mgr.14538) 377 : cluster 0 pgmap v505: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 4.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 278/1539 objects degraded (18.064%) 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: audit 2024-04-02T04:53:12.925776+0000 mon.a (mon.0) 2332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-72153-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:13.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: cluster 2024-04-02T04:53:12.934567+0000 mon.a (mon.0) 2333 : cluster 0 osdmap e353: 8 total, 7 up, 8 in 2024-04-02T04:53:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: audit 2024-04-02T04:53:12.939466+0000 mon.c (mon.2) 617 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:13.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:13 smithi069 bash[28904]: audit 2024-04-02T04:53:12.940763+0000 mon.a (mon.0) 2334 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:13.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:13.561 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:13.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:13.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.828 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:13.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:13.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:13.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:13.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:14.064 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:14.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 -- 172.21.15.69:0/3046866039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 msgr2=0x7f00a01027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 --2- 172.21.15.69:0/3046866039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 0x7f00a01027a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0094009990 tx=0x7f0094009ca0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 -- 172.21.15.69:0/3046866039 shutdown_connections 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 --2- 172.21.15.69:0/3046866039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a0102ce0 0x7f00a010f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 --2- 172.21.15.69:0/3046866039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a0108390 0x7f00a0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 --2- 172.21.15.69:0/3046866039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 0x7f00a01027a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 -- 172.21.15.69:0/3046866039 >> 172.21.15.69:0/3046866039 conn(0x7f00a0079310 msgr2=0x7f00a0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 -- 172.21.15.69:0/3046866039 shutdown_connections 2024-04-02T04:53:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.077+0000 7f00a5026700 1 -- 172.21.15.69:0/3046866039 wait complete. 2024-04-02T04:53:14.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 librados: starting msgr at 2024-04-02T04:53:14.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 librados: starting objecter 2024-04-02T04:53:14.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 Processor -- start 2024-04-02T04:53:14.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- start start 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 librados: setting wanted keys 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 librados: calling monclient init 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 0x7f00a0074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a0102ce0 0x7f00a0074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a0108390 0x7f00a0074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0111b10 con 0x7f00a0102ce0 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0111990 con 0x7f00a0108390 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f00a0111c90 con 0x7f00a0102320 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 0x7f00a0074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 0x7f00a0074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49208/0 (socket says 172.21.15.69:49208) 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 -- 172.21.15.69:0/2279077980 learned_addr learned my addr 172.21.15.69:0/2279077980 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a0102ce0 0x7f00a0074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a0108390 0x7f00a0074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 -- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a0108390 msgr2=0x7f00a0074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00a0108390 0x7f00a0074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 -- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a0102ce0 msgr2=0x7f00a0074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a0102ce0 0x7f00a0074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a01a65e0 con 0x7f00a0102320 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a0102ce0 0x7f00a0074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a0102320 0x7f00a0074540 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f009000c970 tx=0x7f009000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f0087fff700 10 client.?.objecter ms_handle_connect 0x7f00a0102320 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f0087fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090007aa0 con 0x7f00a0102320 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a01a68d0 con 0x7f00a0102320 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 10 client.26393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a01a6e10 con 0x7f00a0102320 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 librados: init done 2024-04-02T04:53:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0090004cb0 con 0x7f00a0102320 2024-04-02T04:53:14.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090005470 con 0x7f00a0102320 2024-04-02T04:53:14.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f00a004ff00 con 0x7f00a0102320 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.081+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f009000f450 con 0x7f00a0102320 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f008806f4b0 0x7f0088071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f008806f4b0 0x7f0088071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(354..354 src has 74..354) ==== 11411+0+0 (secure 0 0 0) 0x7f009008ed20 con 0x7f00a0102320 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 osd_map(354..354 src has 74..354) 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 3 client.26393.objecter handle_osd_map got epochs [354,354] > 0 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 3 client.26393.objecter handle_osd_map decoding full epoch 354 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 20 client.26393.objecter dump_active .. 0 homeless 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v354) ==== 54+0+16 (secure 0 0 0) 0x7f0090091270 con 0x7f00a0102320 2024-04-02T04:53:14.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f008806f4b0 0x7f0088071970 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f00a0079aa0 tx=0x7f009401a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f008806f4b0 2024-04-02T04:53:14.087 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 10 client.26393.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 1 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f00a0062e50 0x7f00a0063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 1 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 0 1 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0075a70 osd=0 4 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 10 client.26393.objecter _send_command 1 2024-04-02T04:53:14.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00a01aa230 con 0x7f00a0062e50 2024-04-02T04:53:14.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f00a0062e50 0x7f00a0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f00a0062e50 0x7f00a0063230 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:14.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.085+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a0062e50 2024-04-02T04:53:14.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.089+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01aa230 con 0x7f00a0062e50 2024-04-02T04:53:14.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.089+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 1: 0 ) 2024-04-02T04:53:14.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.089+0000 7f0087fff700 10 client.26393.objecter _finish_command 1 = system:0 2024-04-02T04:53:14.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.089+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a0075a70 osd=0 4 2024-04-02T04:53:14.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.089+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 0 1 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 10 client.26393.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 2 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 2 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a0075a70 osd=0 3 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 0 2 2024-04-02T04:53:14.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0075a70 osd=0 4 2024-04-02T04:53:14.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 10 client.26393.objecter _send_command 2 2024-04-02T04:53:14.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.105+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a00628d0 con 0x7f00a0062e50 2024-04-02T04:53:14.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a00628d0 con 0x7f00a0062e50 2024-04-02T04:53:14.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 10 client.26393.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a0075a70 osd=0 4 2024-04-02T04:53:14.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 0 2 2024-04-02T04:53:14.108 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:53:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 10 client.26393.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 3 2024-04-02T04:53:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f00a010ad90 0x7f00a002c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 3 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 1 3 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0062cd0 osd=1 4 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 10 client.26393.objecter _send_command 3 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f00a01adb90 con 0x7f00a010ad90 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f00a010ad90 0x7f00a002c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a010ad90 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f00a010ad90 0x7f00a002c9b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01adb90 con 0x7f00a010ad90 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 3: 0 ) 2024-04-02T04:53:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 10 client.26393.objecter _finish_command 3 = system:0 2024-04-02T04:53:14.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a0062cd0 osd=1 4 2024-04-02T04:53:14.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.109+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 1 3 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 10 client.26393.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 4 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 4 2024-04-02T04:53:14.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a0062cd0 osd=1 3 2024-04-02T04:53:14.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 1 4 2024-04-02T04:53:14.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a0062cd0 osd=1 4 2024-04-02T04:53:14.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 10 client.26393.objecter _send_command 4 2024-04-02T04:53:14.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.125+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a010a300 con 0x7f00a010ad90 2024-04-02T04:53:14.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a010a300 con 0x7f00a010ad90 2024-04-02T04:53:14.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 10 client.26393.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a0062cd0 osd=1 4 2024-04-02T04:53:14.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 1 4 2024-04-02T04:53:14.128 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 10 client.26393.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 5 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f00a01ac3f0 0x7f00a01ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 5 2024-04-02T04:53:14.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 2 5 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9880 osd=2 4 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 10 client.26393.objecter _send_command 5 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f00a01a9f00 con 0x7f00a01ac3f0 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f00a01ac3f0 0x7f00a01ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f00a01ac3f0 0x7f00a01ac810 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a01ac3f0 2024-04-02T04:53:14.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01a9f00 con 0x7f00a01ac3f0 2024-04-02T04:53:14.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 5: 0 ) 2024-04-02T04:53:14.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 10 client.26393.objecter _finish_command 5 = system:0 2024-04-02T04:53:14.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01a9880 osd=2 4 2024-04-02T04:53:14.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.129+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 2 5 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 10 client.26393.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 6 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 6 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01a9880 osd=2 3 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 2 6 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9880 osd=2 4 2024-04-02T04:53:14.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 10 client.26393.objecter _send_command 6 2024-04-02T04:53:14.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.149+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a01a9300 con 0x7f00a01ac3f0 2024-04-02T04:53:14.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a01a9300 con 0x7f00a01ac3f0 2024-04-02T04:53:14.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 10 client.26393.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01a9880 osd=2 4 2024-04-02T04:53:14.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 2 6 2024-04-02T04:53:14.153 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:53:14.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 10 client.26393.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 7 2024-04-02T04:53:14.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f00a01afd50 0x7f00a01b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 7 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 3 7 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01aa0b0 osd=3 4 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 10 client.26393.objecter _send_command 7 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f00a01a9550 con 0x7f00a01afd50 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f00a01afd50 0x7f00a01b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a01afd50 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f00a01afd50 0x7f00a01b2140 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01a9550 con 0x7f00a01afd50 2024-04-02T04:53:14.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 7: 0 ) 2024-04-02T04:53:14.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 10 client.26393.objecter _finish_command 7 = system:0 2024-04-02T04:53:14.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01aa0b0 osd=3 4 2024-04-02T04:53:14.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.153+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 3 7 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 10 client.26393.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 8 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 8 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01aa0b0 osd=3 3 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 3 8 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.169+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01aa0b0 osd=3 4 2024-04-02T04:53:14.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 10 client.26393.objecter _send_command 8 2024-04-02T04:53:14.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a01a9550 con 0x7f00a01afd50 2024-04-02T04:53:14.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a01a9550 con 0x7f00a01afd50 2024-04-02T04:53:14.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f0087fff700 10 client.26393.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01aa0b0 osd=3 4 2024-04-02T04:53:14.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 3 8 2024-04-02T04:53:14.175 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:53:14.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 10 client.26393.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 9 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f00a002d6c0 0x7f00a002db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 9 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 4 9 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9550 osd=4 4 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 10 client.26393.objecter _send_command 9 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.173+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f00a01b8130 con 0x7f00a002d6c0 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f00a002d6c0 0x7f00a002db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f009ed9d700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f00a002d6c0 0x7f00a002db20 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a002d6c0 2024-04-02T04:53:14.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01b8130 con 0x7f00a002d6c0 2024-04-02T04:53:14.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 9: 0 ) 2024-04-02T04:53:14.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f0087fff700 10 client.26393.objecter _finish_command 9 = system:0 2024-04-02T04:53:14.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01a9550 osd=4 4 2024-04-02T04:53:14.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.177+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 4 9 2024-04-02T04:53:14.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 10 client.26393.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 10 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 10 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01a9550 osd=4 3 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 4 10 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01a9550 osd=4 4 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 10 client.26393.objecter _send_command 10 2024-04-02T04:53:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.197+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a002d140 con 0x7f00a002d6c0 2024-04-02T04:53:14.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.201+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a002d140 con 0x7f00a002d6c0 2024-04-02T04:53:14.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.201+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.201+0000 7f0087fff700 10 client.26393.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.201+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01a9550 osd=4 4 2024-04-02T04:53:14.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.201+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 4 10 2024-04-02T04:53:14.203 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:53:14.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 10 client.26393.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 11 2024-04-02T04:53:14.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f00a002cef0 0x7f00a01b6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 11 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 5 11 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01ace10 osd=5 4 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 10 client.26393.objecter _send_command 11 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f00a01b8100 con 0x7f00a002cef0 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f00a002cef0 0x7f00a01b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a002cef0 2024-04-02T04:53:14.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f009e59c700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f00a002cef0 0x7f00a01b6990 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01b8100 con 0x7f00a002cef0 2024-04-02T04:53:14.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 11: 0 ) 2024-04-02T04:53:14.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f0087fff700 10 client.26393.objecter _finish_command 11 = system:0 2024-04-02T04:53:14.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01ace10 osd=5 4 2024-04-02T04:53:14.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.205+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 5 11 2024-04-02T04:53:14.217 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 10 client.26393.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 12 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 12 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01ace10 osd=5 3 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 5 12 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01ace10 osd=5 4 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 10 client.26393.objecter _send_command 12 2024-04-02T04:53:14.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a01b8100 con 0x7f00a002cef0 2024-04-02T04:53:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a01b8100 con 0x7f00a002cef0 2024-04-02T04:53:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f0087fff700 10 client.26393.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01ace10 osd=5 4 2024-04-02T04:53:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 5 12 2024-04-02T04:53:14.227 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 10 client.26393.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 13 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f00a01b73e0 0x7f00a01b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 13 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 6 13 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01b8100 osd=6 4 2024-04-02T04:53:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 10 client.26393.objecter _send_command 13 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f00a01b3460 con 0x7f00a01b73e0 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.225+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f00a01b73e0 0x7f00a01b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f0087fff700 10 client.26393.objecter ms_handle_connect 0x7f00a01b73e0 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f009f59e700 1 --2- 172.21.15.69:0/2279077980 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f00a01b73e0 0x7f00a01b7860 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00a01b3460 con 0x7f00a01b73e0 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 13: 0 ) 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f0087fff700 10 client.26393.objecter _finish_command 13 = system:0 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01b8100 osd=6 4 2024-04-02T04:53:14.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.229+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 6 13 2024-04-02T04:53:14.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 10 client.26393.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 14 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 20 client.26393.objecter _get_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 15 client.26393.objecter _session_command_op_remove -1 14 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 20 client.26393.objecter get_session s=0x7f00a01b8100 osd=6 3 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign 6 14 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 20 client.26393.objecter put_session s=0x7f00a01b8100 osd=6 4 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 10 client.26393.objecter _send_command 14 2024-04-02T04:53:14.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f00a01b3460 con 0x7f00a01b73e0 2024-04-02T04:53:14.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f00a01b3460 con 0x7f00a01b73e0 2024-04-02T04:53:14.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f0087fff700 10 client.26393.objecter ms_dispatch 0x7f00a0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:14.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f0087fff700 10 client.26393.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:14.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f0087fff700 20 client.26393.objecter put_session s=0x7f00a01b8100 osd=6 4 2024-04-02T04:53:14.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f0087fff700 15 client.26393.objecter _session_command_op_remove 6 14 2024-04-02T04:53:14.251 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:53:14.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 10 client.26393.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:14.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 15 client.26393.objecter _session_command_op_assign -1 15 2024-04-02T04:53:14.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 20 client.26393.objecter _get_session osd=-1 returning homeless 2024-04-02T04:53:14.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 10 client.26393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:14.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=355}) -- 0x7f00a01b3400 con 0x7f00a0102320 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f00a5026700 1 -- 172.21.15.69:0/2279077980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00a01bec80 con 0x7f00a0102320 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f0087fff700 1 -- 172.21.15.69:0/2279077980 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_get_version_reply(handle=1 version=354) ==== 24+0+0 (secure 0 0 0) 0x7f0090068050 con 0x7f00a0102320 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f009d59a700 10 client.26393.objecter _check_command_map_dne tid 15 current 354 map_dne_bound 354 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f009d59a700 10 client.26393.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.249+0000 7f009d59a700 15 client.26393.objecter _session_command_op_remove -1 15 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:53:14.252 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:53:14.253 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:53:14.321 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2531833952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 msgr2=0x7fe0b010f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2531833952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b010f130 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009f90 tx=0x7fe0a800b330 comp rx=0 tx=0).stop 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2531833952 shutdown_connections 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2531833952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b010f130 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2531833952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0b01082f0 0x7fe0b01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2531833952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0b0069f70 0x7fe0b006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2531833952 >> 172.21.15.69:0/2531833952 conn(0x7fe0b00771e0 msgr2=0x7fe0b00775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2531833952 shutdown_connections 2024-04-02T04:53:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.321+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2531833952 wait complete. 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 librados: starting msgr at 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 librados: starting objecter 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 Processor -- start 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- start start 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 librados: setting wanted keys 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 librados: calling monclient init 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0b0069f70 0x7fe0b019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0b01082f0 0x7fe0b019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b0111ab0 con 0x7fe0b0069f70 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b0111930 con 0x7fe0b01082f0 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b0111c30 con 0x7fe0b006a910 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49230/0 (socket says 172.21.15.69:49230) 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 -- 172.21.15.69:0/2580987783 learned_addr learned my addr 172.21.15.69:0/2580987783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b5761700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0b0069f70 0x7fe0b019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 -- 172.21.15.69:0/2580987783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0b01082f0 msgr2=0x7fe0b019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0b01082f0 0x7fe0b019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 -- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0b0069f70 msgr2=0x7fe0b019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0b0069f70 0x7fe0b019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 -- 172.21.15.69:0/2580987783 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b019f240 con 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b4f60700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b019a770 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000ec10 tx=0x7fe0a000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0a67fc700 10 client.?.objecter ms_handle_connect 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0a67fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a000cde0 con 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b019f530 con 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 10 client.26399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b01a6db0 con 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe0a00044a0 con 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 librados: init done 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0010430 con 0x7fe0b006a910 2024-04-02T04:53:14.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b004ff00 con 0x7fe0b006a910 2024-04-02T04:53:14.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.325+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe0a0004000 con 0x7fe0b006a910 2024-04-02T04:53:14.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.329+0000 7fe0a67fc700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe09c06f5e0 0x7fe09c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:14.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.329+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(354..354 src has 74..354) ==== 11411+0+0 (secure 0 0 0) 0x7fe0a008f4e0 con 0x7fe0b006a910 2024-04-02T04:53:14.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.329+0000 7fe0a67fc700 10 client.26399.objecter ms_dispatch 0x7fe0b0001550 osd_map(354..354 src has 74..354) 2024-04-02T04:53:14.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.329+0000 7fe0a67fc700 3 client.26399.objecter handle_osd_map got epochs [354,354] > 0 2024-04-02T04:53:14.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.329+0000 7fe0a67fc700 3 client.26399.objecter handle_osd_map decoding full epoch 354 2024-04-02T04:53:14.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.329+0000 7fe0a67fc700 20 client.26399.objecter dump_active .. 0 homeless 2024-04-02T04:53:14.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.333+0000 7fe0b5761700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe09c06f5e0 0x7fe09c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:14.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.333+0000 7fe0b5761700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe09c06f5e0 0x7fe09c071aa0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac006fd0 tx=0x7fe0ac008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:14.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.333+0000 7fe0a67fc700 10 client.26399.objecter ms_handle_connect 0x7fe09c06f5e0 2024-04-02T04:53:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.333+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe0a0016330 con 0x7fe0b006a910 2024-04-02T04:53:14.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:14.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:14.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:14 smithi098 bash[23790]: cluster 2024-04-02T04:53:12.622666+0000 osd.5 (osd.5) 215 : cluster 0 2.16 scrub starts 2024-04-02T04:53:14.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:14 smithi098 bash[23790]: cluster 2024-04-02T04:53:12.624924+0000 osd.5 (osd.5) 216 : cluster 0 2.16 scrub ok 2024-04-02T04:53:14.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:14 smithi098 bash[23790]: audit 2024-04-02T04:53:13.088902+0000 mon.c (mon.2) 618 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:14.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:14 smithi098 bash[23790]: audit 2024-04-02T04:53:13.184330+0000 mgr.y (mgr.14538) 378 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:14.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:14 smithi098 bash[23790]: audit 2024-04-02T04:53:14.087986+0000 mon.c (mon.2) 619 : audit 0 from='client.? 172.21.15.69:0/2279077980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:14.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[28904]: cluster 2024-04-02T04:53:12.622666+0000 osd.5 (osd.5) 215 : cluster 0 2.16 scrub starts 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[28904]: cluster 2024-04-02T04:53:12.624924+0000 osd.5 (osd.5) 216 : cluster 0 2.16 scrub ok 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[28904]: audit 2024-04-02T04:53:13.088902+0000 mon.c (mon.2) 618 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[28904]: audit 2024-04-02T04:53:13.184330+0000 mgr.y (mgr.14538) 378 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[28904]: audit 2024-04-02T04:53:14.087986+0000 mon.c (mon.2) 619 : audit 0 from='client.? 172.21.15.69:0/2279077980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[21528]: cluster 2024-04-02T04:53:12.622666+0000 osd.5 (osd.5) 215 : cluster 0 2.16 scrub starts 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[21528]: cluster 2024-04-02T04:53:12.624924+0000 osd.5 (osd.5) 216 : cluster 0 2.16 scrub ok 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[21528]: audit 2024-04-02T04:53:13.088902+0000 mon.c (mon.2) 618 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[21528]: audit 2024-04-02T04:53:13.184330+0000 mgr.y (mgr.14538) 378 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:14 smithi069 bash[21528]: audit 2024-04-02T04:53:14.087986+0000 mon.c (mon.2) 619 : audit 0 from='client.? 172.21.15.69:0/2279077980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:14.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.549+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fe0b01a7040 con 0x7fe0b006a910 2024-04-02T04:53:14.574 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:14.596 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:14.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:14.723 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:14.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:14.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.853+0000 7fe0a67fc700 1 -- 172.21.15.69:0/2580987783 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v355) ==== 87+0+0 (secure 0 0 0) 0x7fe0a005bb50 con 0x7fe0b006a910 2024-04-02T04:53:14.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:14.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.873+0000 7fe0b79c5700 10 librados: watch_flush enter 2024-04-02T04:53:14.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.873+0000 7fe0b79c5700 10 librados: watch_flush exit 2024-04-02T04:53:14.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.873+0000 7fe0b79c5700 20 client.26399.objecter shutdown clearing up homeless session... 2024-04-02T04:53:14.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.873+0000 7fe0b79c5700 10 client.26399.objecter successfully canceled tick 2024-04-02T04:53:14.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe09c06f5e0 msgr2=0x7fe09c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe09c06f5e0 0x7fe09c071aa0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac006fd0 tx=0x7fe0ac008040 comp rx=0 tx=0).stop 2024-04-02T04:53:14.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 msgr2=0x7fe0b019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:14.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b019a770 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000ec10 tx=0x7fe0a000c2d0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 shutdown_connections 2024-04-02T04:53:14.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe09c06f5e0 0x7fe09c071aa0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0b01082f0 0x7fe0b019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0b0069f70 0x7fe0b019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 --2- 172.21.15.69:0/2580987783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0b006a910 0x7fe0b019a770 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:14.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 >> 172.21.15.69:0/2580987783 conn(0x7fe0b00771e0 msgr2=0x7fe0b010d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:14.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.877+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 shutdown_connections 2024-04-02T04:53:14.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.881+0000 7fe0b79c5700 1 -- 172.21.15.69:0/2580987783 wait complete. 2024-04-02T04:53:14.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:14.881+0000 7fe0b79c5700 1 librados: shutdown 2024-04-02T04:53:14.884 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:53:14.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:14.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:15.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:15.105 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:15.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:15.229 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:15.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:15.351 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:15.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:15.480 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:15.583 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:15.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:15.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:13.576914+0000 osd.5 (osd.5) 217 : cluster 0 2.2 scrub starts 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:13.578406+0000 osd.5 (osd.5) 218 : cluster 0 2.2 scrub ok 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:13.850513+0000 osd.2 (osd.2) 173 : cluster 0 41.5 scrub starts 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:13.855118+0000 osd.2 (osd.2) 174 : cluster 0 41.5 scrub ok 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.035269+0000 mgr.y (mgr.14538) 379 : cluster 0 pgmap v508: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.090394+0000 mon.c (mon.2) 620 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.406258+0000 mon.a (mon.0) 2335 : cluster 0 osdmap e354: 8 total, 7 up, 8 in 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.409089+0000 mon.c (mon.2) 621 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.416073+0000 mon.a (mon.0) 2336 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.441139+0000 osd.6 (osd.6) 179 : cluster 0 102.f scrub starts 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.447892+0000 osd.6 (osd.6) 180 : cluster 0 102.f scrub ok 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.553262+0000 mon.c (mon.2) 622 : audit 1 from='client.? 172.21.15.69:0/2580987783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.553818+0000 mon.a (mon.0) 2337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.839880+0000 mon.a (mon.0) 2338 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.842333+0000 mon.a (mon.0) 2339 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.842582+0000 mon.a (mon.0) 2340 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.842769+0000 mon.a (mon.0) 2341 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.855305+0000 mon.a (mon.0) 2342 : cluster 0 osdmap e355: 8 total, 7 up, 8 in 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.882566+0000 mon.c (mon.2) 623 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:14.893914+0000 mon.a (mon.0) 2343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.905110+0000 osd.2 (osd.2) 175 : cluster 0 10.3 scrub starts 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.931387+0000 mon.a (mon.0) 2344 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:15 smithi098 bash[23790]: audit 2024-04-02T04:53:15.091435+0000 mon.c (mon.2) 624 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:13.576914+0000 osd.5 (osd.5) 217 : cluster 0 2.2 scrub starts 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:13.578406+0000 osd.5 (osd.5) 218 : cluster 0 2.2 scrub ok 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:13.850513+0000 osd.2 (osd.2) 173 : cluster 0 41.5 scrub starts 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:13.855118+0000 osd.2 (osd.2) 174 : cluster 0 41.5 scrub ok 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.035269+0000 mgr.y (mgr.14538) 379 : cluster 0 pgmap v508: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.090394+0000 mon.c (mon.2) 620 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.406258+0000 mon.a (mon.0) 2335 : cluster 0 osdmap e354: 8 total, 7 up, 8 in 2024-04-02T04:53:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.409089+0000 mon.c (mon.2) 621 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.416073+0000 mon.a (mon.0) 2336 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.441139+0000 osd.6 (osd.6) 179 : cluster 0 102.f scrub starts 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.447892+0000 osd.6 (osd.6) 180 : cluster 0 102.f scrub ok 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.553262+0000 mon.c (mon.2) 622 : audit 1 from='client.? 172.21.15.69:0/2580987783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.553818+0000 mon.a (mon.0) 2337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.839880+0000 mon.a (mon.0) 2338 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.842333+0000 mon.a (mon.0) 2339 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.842582+0000 mon.a (mon.0) 2340 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.842769+0000 mon.a (mon.0) 2341 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.855305+0000 mon.a (mon.0) 2342 : cluster 0 osdmap e355: 8 total, 7 up, 8 in 2024-04-02T04:53:15.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.882566+0000 mon.c (mon.2) 623 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:13.576914+0000 osd.5 (osd.5) 217 : cluster 0 2.2 scrub starts 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:13.578406+0000 osd.5 (osd.5) 218 : cluster 0 2.2 scrub ok 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:13.850513+0000 osd.2 (osd.2) 173 : cluster 0 41.5 scrub starts 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:13.855118+0000 osd.2 (osd.2) 174 : cluster 0 41.5 scrub ok 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.035269+0000 mgr.y (mgr.14538) 379 : cluster 0 pgmap v508: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.090394+0000 mon.c (mon.2) 620 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.406258+0000 mon.a (mon.0) 2335 : cluster 0 osdmap e354: 8 total, 7 up, 8 in 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.409089+0000 mon.c (mon.2) 621 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.416073+0000 mon.a (mon.0) 2336 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.441139+0000 osd.6 (osd.6) 179 : cluster 0 102.f scrub starts 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.447892+0000 osd.6 (osd.6) 180 : cluster 0 102.f scrub ok 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.553262+0000 mon.c (mon.2) 622 : audit 1 from='client.? 172.21.15.69:0/2580987783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:53:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.553818+0000 mon.a (mon.0) 2337 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.839880+0000 mon.a (mon.0) 2338 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.842333+0000 mon.a (mon.0) 2339 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-72153-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.842582+0000 mon.a (mon.0) 2340 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.842769+0000 mon.a (mon.0) 2341 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.855305+0000 mon.a (mon.0) 2342 : cluster 0 osdmap e355: 8 total, 7 up, 8 in 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.882566+0000 mon.c (mon.2) 623 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:14.893914+0000 mon.a (mon.0) 2343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.905110+0000 osd.2 (osd.2) 175 : cluster 0 10.3 scrub starts 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.931387+0000 mon.a (mon.0) 2344 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:15.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[21528]: audit 2024-04-02T04:53:15.091435+0000 mon.c (mon.2) 624 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:15.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:14.893914+0000 mon.a (mon.0) 2343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:15.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.905110+0000 osd.2 (osd.2) 175 : cluster 0 10.3 scrub starts 2024-04-02T04:53:15.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.931387+0000 mon.a (mon.0) 2344 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:15.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:15 smithi069 bash[28904]: audit 2024-04-02T04:53:15.091435+0000 mon.c (mon.2) 624 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:15.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:15.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:15.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:15.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:15.999 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:16.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:16.128 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:16.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:16.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:16.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.834844+0000 osd.3 (osd.3) 247 : cluster 0 4.1d scrub starts 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: cluster 2024-04-02T04:53:14.837453+0000 osd.3 (osd.3) 248 : cluster 0 4.1d scrub ok 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: audit 2024-04-02T04:53:15.847051+0000 mon.a (mon.0) 2345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: cluster 2024-04-02T04:53:15.856810+0000 mon.a (mon.0) 2346 : cluster 0 osdmap e356: 8 total, 7 up, 8 in 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: audit 2024-04-02T04:53:15.859475+0000 mon.c (mon.2) 625 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: audit 2024-04-02T04:53:15.860304+0000 mon.a (mon.0) 2347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:16.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[21528]: audit 2024-04-02T04:53:16.092883+0000 mon.c (mon.2) 626 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.834844+0000 osd.3 (osd.3) 247 : cluster 0 4.1d scrub starts 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: cluster 2024-04-02T04:53:14.837453+0000 osd.3 (osd.3) 248 : cluster 0 4.1d scrub ok 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: audit 2024-04-02T04:53:15.847051+0000 mon.a (mon.0) 2345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: cluster 2024-04-02T04:53:15.856810+0000 mon.a (mon.0) 2346 : cluster 0 osdmap e356: 8 total, 7 up, 8 in 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: audit 2024-04-02T04:53:15.859475+0000 mon.c (mon.2) 625 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: audit 2024-04-02T04:53:15.860304+0000 mon.a (mon.0) 2347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:16.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:16 smithi069 bash[28904]: audit 2024-04-02T04:53:16.092883+0000 mon.c (mon.2) 626 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:16.519 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:16.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:16.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:53:16.672 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:53:16.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.834844+0000 osd.3 (osd.3) 247 : cluster 0 4.1d scrub starts 2024-04-02T04:53:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: cluster 2024-04-02T04:53:14.837453+0000 osd.3 (osd.3) 248 : cluster 0 4.1d scrub ok 2024-04-02T04:53:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: audit 2024-04-02T04:53:15.847051+0000 mon.a (mon.0) 2345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: cluster 2024-04-02T04:53:15.856810+0000 mon.a (mon.0) 2346 : cluster 0 osdmap e356: 8 total, 7 up, 8 in 2024-04-02T04:53:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: audit 2024-04-02T04:53:15.859475+0000 mon.c (mon.2) 625 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: audit 2024-04-02T04:53:15.860304+0000 mon.a (mon.0) 2347 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:16 smithi098 bash[23790]: audit 2024-04-02T04:53:16.092883+0000 mon.c (mon.2) 626 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:16.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.777 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:16.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:16.905 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:16.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:17.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:17.028 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:17.028 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- 172.21.15.69:0/2215970772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4107220 msgr2=0x7f1da4107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/2215970772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4107220 0x7f1da4107600 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1da000ba80 tx=0x7f1da000bd90 comp rx=0 tx=0).stop 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- 172.21.15.69:0/2215970772 shutdown_connections 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/2215970772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1da4101b60 0x7f1da410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/2215970772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4107220 0x7f1da4107600 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/2215970772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da41011a0 0x7f1da4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- 172.21.15.69:0/2215970772 >> 172.21.15.69:0/2215970772 conn(0x7f1da40771e0 msgr2=0x7f1da40775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- 172.21.15.69:0/2215970772 shutdown_connections 2024-04-02T04:53:17.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- 172.21.15.69:0/2215970772 wait complete. 2024-04-02T04:53:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 librados: starting msgr at 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 librados: starting objecter 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 Processor -- start 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- start start 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 librados: setting wanted keys 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 librados: calling monclient init 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 0x7f1da4195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:17.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da4101b60 0x7f1da41962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1da4107220 0x7f1da419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410f870 con 0x7f1da41011a0 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.029+0000 7f1dac2c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410f6f0 con 0x7f1da4101b60 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1dac2c6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1da410f9f0 con 0x7f1da4107220 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 0x7f1da4195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 0x7f1da4195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35434/0 (socket says 172.21.15.69:35434) 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 -- 172.21.15.69:0/1327584244 learned_addr learned my addr 172.21.15.69:0/1327584244 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa863700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1da4107220 0x7f1da419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1da9861700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da4101b60 0x7f1da41962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 -- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1da4107220 msgr2=0x7f1da419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1da4107220 0x7f1da419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 -- 172.21.15.69:0/1327584244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da4101b60 msgr2=0x7f1da41962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da4101b60 0x7f1da41962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 -- 172.21.15.69:0/1327584244 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da419ad70 con 0x7f1da41011a0 2024-04-02T04:53:17.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1da9861700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da4101b60 0x7f1da41962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1daa062700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 0x7f1da4195d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1da000da00 tx=0x7f1da000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 10 client.?.objecter ms_handle_connect 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0005240 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da419b000 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f1da00046d0 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1dac2c6700 10 client.16308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0028670 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41a28e0 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1dac2c6700 1 librados: init done 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f1da0026020 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da4102860 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1d9006f4b0 0x7f1d90071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(357..357 src has 74..357) ==== 11432+0+0 (secure 0 0 0) 0x7f1da0011030 con 0x7f1da41011a0 2024-04-02T04:53:17.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 10 client.16308.objecter ms_dispatch 0x7f1da4001550 osd_map(357..357 src has 74..357) 2024-04-02T04:53:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 3 client.16308.objecter handle_osd_map got epochs [357,357] > 0 2024-04-02T04:53:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 3 client.16308.objecter handle_osd_map decoding full epoch 357 2024-04-02T04:53:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1da9861700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1d9006f4b0 0x7f1d90071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 20 client.16308.objecter dump_active .. 0 homeless 2024-04-02T04:53:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1d9b7fe700 10 client.16308.objecter ms_handle_connect 0x7f1d9006f4b0 2024-04-02T04:53:17.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.033+0000 7f1da9861700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1d9006f4b0 0x7f1d90071970 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1d94005fd0 tx=0x7f1d94005e20 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:17.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.041+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1da002d060 con 0x7f1da41011a0 2024-04-02T04:53:17.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:17.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.245+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1da404ff00 con 0x7f1d9006f4b0 2024-04-02T04:53:17.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:17.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.281+0000 7f1d9b7fe700 1 -- 172.21.15.69:0/1327584244 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+903041 (secure 0 0 0) 0x7f1da404ff00 con 0x7f1d9006f4b0 2024-04-02T04:53:17.291 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:53:17.299 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:17.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 10 librados: watch_flush enter 2024-04-02T04:53:17.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 10 librados: watch_flush exit 2024-04-02T04:53:17.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 20 client.16308.objecter shutdown clearing up homeless session... 2024-04-02T04:53:17.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 10 client.16308.objecter successfully canceled tick 2024-04-02T04:53:17.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1d9006f4b0 msgr2=0x7f1d90071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:17.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1d9006f4b0 0x7f1d90071970 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1d94005fd0 tx=0x7f1d94005e20 comp rx=0 tx=0).stop 2024-04-02T04:53:17.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 msgr2=0x7f1da4195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:17.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 0x7f1da4195d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1da000da00 tx=0x7f1da000da30 comp rx=0 tx=0).stop 2024-04-02T04:53:17.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 shutdown_connections 2024-04-02T04:53:17.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1d9006f4b0 0x7f1d90071970 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1da4107220 0x7f1da419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da41011a0 0x7f1da4195d60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 --2- 172.21.15.69:0/1327584244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da4101b60 0x7f1da41962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:17.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 >> 172.21.15.69:0/1327584244 conn(0x7f1da40771e0 msgr2=0x7f1da4107ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:17.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 shutdown_connections 2024-04-02T04:53:17.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 -- 172.21.15.69:0/1327584244 wait complete. 2024-04-02T04:53:17.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:17.289+0000 7f1dac2c6700 1 librados: shutdown 2024-04-02T04:53:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:17.306 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:53:17.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:17.433 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: cluster 2024-04-02T04:53:16.036925+0000 mgr.y (mgr.14538) 380 : cluster 0 pgmap v511: 364 pgs: 40 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: audit 2024-04-02T04:53:16.855253+0000 mon.a (mon.0) 2348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: cluster 2024-04-02T04:53:16.859284+0000 mon.a (mon.0) 2349 : cluster 0 osdmap e357: 8 total, 7 up, 8 in 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: audit 2024-04-02T04:53:16.867910+0000 mon.c (mon.2) 627 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: audit 2024-04-02T04:53:16.868358+0000 mon.a (mon.0) 2350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: audit 2024-04-02T04:53:16.868643+0000 mon.c (mon.2) 628 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: audit 2024-04-02T04:53:16.868937+0000 mon.a (mon.0) 2351 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:17.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[21528]: audit 2024-04-02T04:53:17.094469+0000 mon.c (mon.2) 629 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: cluster 2024-04-02T04:53:16.036925+0000 mgr.y (mgr.14538) 380 : cluster 0 pgmap v511: 364 pgs: 40 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: audit 2024-04-02T04:53:16.855253+0000 mon.a (mon.0) 2348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: cluster 2024-04-02T04:53:16.859284+0000 mon.a (mon.0) 2349 : cluster 0 osdmap e357: 8 total, 7 up, 8 in 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: audit 2024-04-02T04:53:16.867910+0000 mon.c (mon.2) 627 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: audit 2024-04-02T04:53:16.868358+0000 mon.a (mon.0) 2350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: audit 2024-04-02T04:53:16.868643+0000 mon.c (mon.2) 628 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: audit 2024-04-02T04:53:16.868937+0000 mon.a (mon.0) 2351 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:17.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:17 smithi069 bash[28904]: audit 2024-04-02T04:53:17.094469+0000 mon.c (mon.2) 629 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:17.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:17.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:17.690 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: cluster 2024-04-02T04:53:16.036925+0000 mgr.y (mgr.14538) 380 : cluster 0 pgmap v511: 364 pgs: 40 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: audit 2024-04-02T04:53:16.855253+0000 mon.a (mon.0) 2348 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: cluster 2024-04-02T04:53:16.859284+0000 mon.a (mon.0) 2349 : cluster 0 osdmap e357: 8 total, 7 up, 8 in 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: audit 2024-04-02T04:53:16.867910+0000 mon.c (mon.2) 627 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: audit 2024-04-02T04:53:16.868358+0000 mon.a (mon.0) 2350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: audit 2024-04-02T04:53:16.868643+0000 mon.c (mon.2) 628 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: audit 2024-04-02T04:53:16.868937+0000 mon.a (mon.0) 2351 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:17 smithi098 bash[23790]: audit 2024-04-02T04:53:17.094469+0000 mon.c (mon.2) 629 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:17.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:17.820 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:17.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:17.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:18.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:18.074 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:18.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:18.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:18.332 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:18.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: cluster 2024-04-02T04:53:17.163789+0000 osd.0 (osd.0) 217 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: cluster 2024-04-02T04:53:17.163980+0000 osd.0 (osd.0) 218 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.252007+0000 mgr.y (mgr.14538) 381 : audit 0 from='client.16308 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.841116+0000 mon.a (mon.0) 2352 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.841198+0000 mon.b (mon.1) 205 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.857006+0000 mon.a (mon.0) 2353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.857098+0000 mon.a (mon.0) 2354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: cluster 2024-04-02T04:53:17.860899+0000 mon.a (mon.0) 2355 : cluster 0 osdmap e358: 8 total, 7 up, 8 in 2024-04-02T04:53:18.446 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.861371+0000 mon.c (mon.2) 630 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:53:18.447 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.861667+0000 mon.c (mon.2) 631 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:18.447 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.862498+0000 mon.a (mon.0) 2356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:53:18.447 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:17.862842+0000 mon.a (mon.0) 2357 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:18.447 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:18 smithi098 bash[23790]: audit 2024-04-02T04:53:18.095775+0000 mon.c (mon.2) 632 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:18.461 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:18.564 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:18.589 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:18.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: cluster 2024-04-02T04:53:17.163789+0000 osd.0 (osd.0) 217 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: cluster 2024-04-02T04:53:17.163980+0000 osd.0 (osd.0) 218 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.252007+0000 mgr.y (mgr.14538) 381 : audit 0 from='client.16308 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.841116+0000 mon.a (mon.0) 2352 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.841198+0000 mon.b (mon.1) 205 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.857006+0000 mon.a (mon.0) 2353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.857098+0000 mon.a (mon.0) 2354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: cluster 2024-04-02T04:53:17.860899+0000 mon.a (mon.0) 2355 : cluster 0 osdmap e358: 8 total, 7 up, 8 in 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.861371+0000 mon.c (mon.2) 630 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:53:18.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.861667+0000 mon.c (mon.2) 631 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.862498+0000 mon.a (mon.0) 2356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:17.862842+0000 mon.a (mon.0) 2357 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[28904]: audit 2024-04-02T04:53:18.095775+0000 mon.c (mon.2) 632 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: cluster 2024-04-02T04:53:17.163789+0000 osd.0 (osd.0) 217 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: cluster 2024-04-02T04:53:17.163980+0000 osd.0 (osd.0) 218 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.252007+0000 mgr.y (mgr.14538) 381 : audit 0 from='client.16308 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.841116+0000 mon.a (mon.0) 2352 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:53:18.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.841198+0000 mon.b (mon.1) 205 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.857006+0000 mon.a (mon.0) 2353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.857098+0000 mon.a (mon.0) 2354 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: cluster 2024-04-02T04:53:17.860899+0000 mon.a (mon.0) 2355 : cluster 0 osdmap e358: 8 total, 7 up, 8 in 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.861371+0000 mon.c (mon.2) 630 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.861667+0000 mon.c (mon.2) 631 : audit 1 from='client.? 172.21.15.69:0/49085792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.862498+0000 mon.a (mon.0) 2356 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:17.862842+0000 mon.a (mon.0) 2357 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]: dispatch 2024-04-02T04:53:18.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21528]: audit 2024-04-02T04:53:18.095775+0000 mon.c (mon.2) 632 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:18.687 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-04-02T04:53:16.856729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8795329,"num_objects":514,"num_object_clones":143,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":2,"num_read":1845,"num_read_kb":1566,"num_write":1465,"num_write_kb":15168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771129,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2522,"ondisk_log_size":2522,"up":855,"acting":855,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":656968,"kb_used_data":144980,"kb_used_omap":12,"kb_used_meta":511667,"kb_avail":749304248,"statfs":{"total":767960285184,"available":767287549952,"internally_reserved":0,"allocated":148459520,"data_stored":129673909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":523947568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.929733"},"pg_stats":[{"pgid":"271.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"271.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:53:13.925670+0000","last_change":"2024-04-02T04:53:13.925670+0000","last_active":"2024-04-02T04:53:13.925670+0000","last_peered":"2024-04-02T04:53:13.925670+0000","last_clean":"2024-04-02T04:53:13.925670+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:53:13.925670+0000","last_undegraded":"2024-04-02T04:53:13.925670+0000","last_fullsized":"2024-04-02T04:53:13.925670+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:13.925670+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.925670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"102.19","version":"192'45","reported_seq":524,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.936293+0000","last_change":"2024-04-02T04:50:27.532331+0000","last_active":"2024-04-02T04:53:11.936293+0000","last_peered":"2024-04-02T04:53:11.936293+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.255616+0000","last_became_peered":"2024-04-02T04:49:55.255616+0000","last_unstale":"2024-04-02T04:53:11.936293+0000","last_undegraded":"2024-04-02T04:50:27.523866+0000","last_fullsized":"2024-04-02T04:49:55.215309+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":25}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.18","version":"263'35","reported_seq":567,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.925878+0000","last_change":"2024-04-02T04:52:53.911292+0000","last_active":"2024-04-02T04:53:11.925878+0000","last_peered":"2024-04-02T04:53:11.925878+0000","last_clean":"2024-04-02T04:53:11.925878+0000","last_became_active":"2024-04-02T04:49:56.572185+0000","last_became_peered":"2024-04-02T04:49:56.572185+0000","last_unstale":"2024-04-02T04:53:11.925878+0000","last_undegraded":"2024-04-02T04:53:11.925878+0000","last_fullsized":"2024-04-02T04:53:11.925878+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'35","last_scrub_stamp":"2024-04-02T04:51:56.184367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.184367+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:22.983929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0094289309999999998,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1b","version":"263'37","reported_seq":578,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933362+0000","last_change":"2024-04-02T04:52:53.928225+0000","last_active":"2024-04-02T04:53:11.933362+0000","last_peered":"2024-04-02T04:53:11.933362+0000","last_clean":"2024-04-02T04:53:11.933362+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:53:11.933362+0000","last_undegraded":"2024-04-02T04:53:11.933362+0000","last_fullsized":"2024-04-02T04:53:11.933362+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'37","last_scrub_stamp":"2024-04-02T04:52:48.221399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.221399+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:58.690278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016351759999999999,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1a","version":"342'73","reported_seq":593,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.928275+0000","last_change":"2024-04-02T04:52:53.912906+0000","last_active":"2024-04-02T04:53:11.928275+0000","last_peered":"2024-04-02T04:53:11.928275+0000","last_clean":"2024-04-02T04:53:11.928275+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:53:11.928275+0000","last_undegraded":"2024-04-02T04:53:11.928275+0000","last_fullsized":"2024-04-02T04:53:11.928275+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'68","last_scrub_stamp":"2024-04-02T04:51:53.146929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.146929+0000","objects_scrubbed":4,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:21.730192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00057379499999999999,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1d","version":"248'85","reported_seq":620,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921238+0000","last_change":"2024-04-02T04:52:53.907859+0000","last_active":"2024-04-02T04:53:11.921238+0000","last_peered":"2024-04-02T04:53:11.921238+0000","last_clean":"2024-04-02T04:53:11.921238+0000","last_became_active":"2024-04-02T04:49:56.582887+0000","last_became_peered":"2024-04-02T04:49:56.582887+0000","last_unstale":"2024-04-02T04:53:11.921238+0000","last_undegraded":"2024-04-02T04:53:11.921238+0000","last_fullsized":"2024-04-02T04:53:11.921238+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'85","last_scrub_stamp":"2024-04-02T04:52:51.767218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.767218+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:01.181697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043202969999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1c","version":"244'34","reported_seq":569,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.929265+0000","last_change":"2024-04-02T04:52:53.919138+0000","last_active":"2024-04-02T04:53:11.929265+0000","last_peered":"2024-04-02T04:53:11.929265+0000","last_clean":"2024-04-02T04:53:11.929265+0000","last_became_active":"2024-04-02T04:49:56.570832+0000","last_became_peered":"2024-04-02T04:49:56.570832+0000","last_unstale":"2024-04-02T04:53:11.929265+0000","last_undegraded":"2024-04-02T04:53:11.929265+0000","last_fullsized":"2024-04-02T04:53:11.929265+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'34","last_scrub_stamp":"2024-04-02T04:51:47.142172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:47.142172+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:15.987374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00060106500000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1f","version":"334'47","reported_seq":579,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917889+0000","last_change":"2024-04-02T04:52:55.923999+0000","last_active":"2024-04-02T04:53:11.917889+0000","last_peered":"2024-04-02T04:53:11.917889+0000","last_clean":"2024-04-02T04:53:11.917889+0000","last_became_active":"2024-04-02T04:49:55.249289+0000","last_became_peered":"2024-04-02T04:49:55.249289+0000","last_unstale":"2024-04-02T04:53:11.917889+0000","last_undegraded":"2024-04-02T04:53:11.917889+0000","last_fullsized":"2024-04-02T04:53:11.917889+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063291931999999995,"stat_sum":{"num_bytes":29,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1e","version":"248'17","reported_seq":552,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.929765+0000","last_change":"2024-04-02T04:52:53.925077+0000","last_active":"2024-04-02T04:53:11.929765+0000","last_peered":"2024-04-02T04:53:11.929765+0000","last_clean":"2024-04-02T04:53:11.929765+0000","last_became_active":"2024-04-02T04:49:56.563968+0000","last_became_peered":"2024-04-02T04:49:56.563968+0000","last_unstale":"2024-04-02T04:53:11.929765+0000","last_undegraded":"2024-04-02T04:53:11.929765+0000","last_fullsized":"2024-04-02T04:53:11.929765+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:51:55.166519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.166519+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:13.943894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00090914499999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.11","version":"192'48","reported_seq":538,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.926539+0000","last_change":"2024-04-02T04:50:18.405897+0000","last_active":"2024-04-02T04:53:11.926539+0000","last_peered":"2024-04-02T04:53:11.926539+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.569001+0000","last_became_peered":"2024-04-02T04:49:56.569001+0000","last_unstale":"2024-04-02T04:53:11.926539+0000","last_undegraded":"2024-04-02T04:50:17.425678+0000","last_fullsized":"2024-04-02T04:49:56.526356+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273,"num_objects":27,"num_object_clones":21,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":27}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.10","version":"192'18","reported_seq":504,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.917205+0000","last_change":"2024-04-02T04:50:27.548680+0000","last_active":"2024-04-02T04:53:11.917205+0000","last_peered":"2024-04-02T04:53:11.917205+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.243496+0000","last_became_peered":"2024-04-02T04:49:55.243496+0000","last_unstale":"2024-04-02T04:53:11.917205+0000","last_undegraded":"2024-04-02T04:50:27.476730+0000","last_fullsized":"2024-04-02T04:49:55.211603+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.13","version":"247'50","reported_seq":533,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.935599+0000","last_change":"2024-04-02T04:50:18.418843+0000","last_active":"2024-04-02T04:53:11.935599+0000","last_peered":"2024-04-02T04:53:11.935599+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.554860+0000","last_became_peered":"2024-04-02T04:49:56.554860+0000","last_unstale":"2024-04-02T04:53:11.935599+0000","last_undegraded":"2024-04-02T04:50:17.426824+0000","last_fullsized":"2024-04-02T04:49:56.518531+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":6,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.12","version":"240'17","reported_seq":558,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.920050+0000","last_change":"2024-04-02T04:52:53.896169+0000","last_active":"2024-04-02T04:53:11.920050+0000","last_peered":"2024-04-02T04:53:11.920050+0000","last_clean":"2024-04-02T04:53:11.920050+0000","last_became_active":"2024-04-02T04:49:55.255700+0000","last_became_peered":"2024-04-02T04:49:55.255700+0000","last_unstale":"2024-04-02T04:53:11.920050+0000","last_undegraded":"2024-04-02T04:53:11.920050+0000","last_fullsized":"2024-04-02T04:53:11.920050+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-04-02T04:51:53.016585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.016585+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002296219,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.15","version":"192'37","reported_seq":523,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.937440+0000","last_change":"2024-04-02T04:50:09.415659+0000","last_active":"2024-04-02T04:53:11.937440+0000","last_peered":"2024-04-02T04:53:11.937440+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.561896+0000","last_became_peered":"2024-04-02T04:49:56.561896+0000","last_unstale":"2024-04-02T04:53:11.937440+0000","last_undegraded":"2024-04-02T04:50:08.454406+0000","last_fullsized":"2024-04-02T04:49:56.542252+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.14","version":"350'154","reported_seq":679,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950371+0000","last_change":"2024-04-02T04:52:53.927028+0000","last_active":"2024-04-02T04:53:11.950371+0000","last_peered":"2024-04-02T04:53:11.950371+0000","last_clean":"2024-04-02T04:53:11.950371+0000","last_became_active":"2024-04-02T04:49:55.269267+0000","last_became_peered":"2024-04-02T04:49:55.269267+0000","last_unstale":"2024-04-02T04:53:11.950371+0000","last_undegraded":"2024-04-02T04:53:11.950371+0000","last_fullsized":"2024-04-02T04:53:11.950371+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'153","last_scrub_stamp":"2024-04-02T04:52:34.712599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.712599+0000","objects_scrubbed":9,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:01.712013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0025973229999999999,"stat_sum":{"num_bytes":4000107,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":46,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.17","version":"313'35","reported_seq":572,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930536+0000","last_change":"2024-04-02T04:52:53.928093+0000","last_active":"2024-04-02T04:53:11.930536+0000","last_peered":"2024-04-02T04:53:11.930536+0000","last_clean":"2024-04-02T04:53:11.930536+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:53:11.930536+0000","last_undegraded":"2024-04-02T04:53:11.930536+0000","last_fullsized":"2024-04-02T04:53:11.930536+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'34","last_scrub_stamp":"2024-04-02T04:51:48.141397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.141397+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:17.044814+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0021229199999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.16","version":"252'17","reported_seq":558,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.951051+0000","last_change":"2024-04-02T04:52:53.931293+0000","last_active":"2024-04-02T04:53:11.951051+0000","last_peered":"2024-04-02T04:53:11.951051+0000","last_clean":"2024-04-02T04:53:11.951051+0000","last_became_active":"2024-04-02T04:49:55.269224+0000","last_became_peered":"2024-04-02T04:49:55.269224+0000","last_unstale":"2024-04-02T04:53:11.951051+0000","last_undegraded":"2024-04-02T04:53:11.951051+0000","last_fullsized":"2024-04-02T04:53:11.951051+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'17","last_scrub_stamp":"2024-04-02T04:52:38.664090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.664090+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:57.079994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031222559999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.9","version":"321'20","reported_seq":502,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.926353+0000","last_change":"2024-04-02T04:50:27.575689+0000","last_active":"2024-04-02T04:53:11.926353+0000","last_peered":"2024-04-02T04:53:11.926353+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.562137+0000","last_became_peered":"2024-04-02T04:49:56.562137+0000","last_unstale":"2024-04-02T04:53:11.926353+0000","last_undegraded":"2024-04-02T04:50:27.479688+0000","last_fullsized":"2024-04-02T04:49:56.524836+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.8","version":"248'17","reported_seq":551,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941662+0000","last_change":"2024-04-02T04:52:53.906611+0000","last_active":"2024-04-02T04:53:11.941662+0000","last_peered":"2024-04-02T04:53:11.941662+0000","last_clean":"2024-04-02T04:53:11.941662+0000","last_became_active":"2024-04-02T04:49:55.259553+0000","last_became_peered":"2024-04-02T04:49:55.259553+0000","last_unstale":"2024-04-02T04:53:11.941662+0000","last_undegraded":"2024-04-02T04:53:11.941662+0000","last_fullsized":"2024-04-02T04:53:11.941662+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069807660000000002,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.b","version":"258'73","reported_seq":601,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950184+0000","last_change":"2024-04-02T04:52:53.926194+0000","last_active":"2024-04-02T04:53:11.950184+0000","last_peered":"2024-04-02T04:53:11.950184+0000","last_clean":"2024-04-02T04:53:11.950184+0000","last_became_active":"2024-04-02T04:49:55.267102+0000","last_became_peered":"2024-04-02T04:49:55.267102+0000","last_unstale":"2024-04-02T04:53:11.950184+0000","last_undegraded":"2024-04-02T04:53:11.950184+0000","last_fullsized":"2024-04-02T04:53:11.950184+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'73","last_scrub_stamp":"2024-04-02T04:52:33.669178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.669178+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.684633+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003027484,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.a","version":"254'68","reported_seq":598,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950014+0000","last_change":"2024-04-02T04:52:53.919322+0000","last_active":"2024-04-02T04:53:11.950014+0000","last_peered":"2024-04-02T04:53:11.950014+0000","last_clean":"2024-04-02T04:53:11.950014+0000","last_became_active":"2024-04-02T04:49:55.252341+0000","last_became_peered":"2024-04-02T04:49:55.252341+0000","last_unstale":"2024-04-02T04:53:11.950014+0000","last_undegraded":"2024-04-02T04:53:11.950014+0000","last_fullsized":"2024-04-02T04:53:11.950014+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'68","last_scrub_stamp":"2024-04-02T04:52:43.653347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.653347+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:08.320707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00117381,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.d","version":"288'69","reported_seq":545,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.929793+0000","last_change":"2024-04-02T04:50:27.508913+0000","last_active":"2024-04-02T04:53:11.929793+0000","last_peered":"2024-04-02T04:53:11.929793+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.566448+0000","last_became_peered":"2024-04-02T04:49:56.566448+0000","last_unstale":"2024-04-02T04:53:11.929793+0000","last_undegraded":"2024-04-02T04:50:27.492108+0000","last_fullsized":"2024-04-02T04:49:56.525265+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":39,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":39}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.c","version":"249'68","reported_seq":590,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942529+0000","last_change":"2024-04-02T04:52:53.920193+0000","last_active":"2024-04-02T04:53:11.942529+0000","last_peered":"2024-04-02T04:53:11.942529+0000","last_clean":"2024-04-02T04:53:11.942529+0000","last_became_active":"2024-04-02T04:49:55.252273+0000","last_became_peered":"2024-04-02T04:49:55.252273+0000","last_unstale":"2024-04-02T04:53:11.942529+0000","last_undegraded":"2024-04-02T04:53:11.942529+0000","last_fullsized":"2024-04-02T04:53:11.942529+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093620490000000008,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.f","version":"306'60","reported_seq":643,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.938454+0000","last_change":"2024-04-02T04:52:53.926339+0000","last_active":"2024-04-02T04:53:11.938454+0000","last_peered":"2024-04-02T04:53:11.938454+0000","last_clean":"2024-04-02T04:53:11.938454+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:53:11.938454+0000","last_undegraded":"2024-04-02T04:53:11.938454+0000","last_fullsized":"2024-04-02T04:53:11.938454+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048377230000000004,"stat_sum":{"num_bytes":4194354,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.e","version":"229'56","reported_seq":597,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.939993+0000","last_change":"2024-04-02T04:52:53.933725+0000","last_active":"2024-04-02T04:53:11.939993+0000","last_peered":"2024-04-02T04:53:11.939993+0000","last_clean":"2024-04-02T04:53:11.939993+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:53:11.939993+0000","last_undegraded":"2024-04-02T04:53:11.939993+0000","last_fullsized":"2024-04-02T04:53:11.939993+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'56","last_scrub_stamp":"2024-04-02T04:51:26.235660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.235660+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000480698,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1","version":"269'30","reported_seq":512,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.936125+0000","last_change":"2024-04-02T04:50:27.514211+0000","last_active":"2024-04-02T04:53:11.936125+0000","last_peered":"2024-04-02T04:53:11.936125+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:55.245537+0000","last_became_peered":"2024-04-02T04:49:55.245537+0000","last_unstale":"2024-04-02T04:53:11.936125+0000","last_undegraded":"2024-04-02T04:50:27.498272+0000","last_fullsized":"2024-04-02T04:49:55.215487+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":17}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.0","version":"292'90","reported_seq":607,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936078+0000","last_change":"2024-04-02T04:52:53.902619+0000","last_active":"2024-04-02T04:53:11.936078+0000","last_peered":"2024-04-02T04:53:11.936078+0000","last_clean":"2024-04-02T04:53:11.936078+0000","last_became_active":"2024-04-02T04:49:55.268700+0000","last_became_peered":"2024-04-02T04:49:55.268700+0000","last_unstale":"2024-04-02T04:53:11.936078+0000","last_undegraded":"2024-04-02T04:53:11.936078+0000","last_fullsized":"2024-04-02T04:53:11.936078+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073717599999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.3","version":"242'34","reported_seq":568,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.923279+0000","last_change":"2024-04-02T04:52:53.892322+0000","last_active":"2024-04-02T04:53:11.923279+0000","last_peered":"2024-04-02T04:53:11.923279+0000","last_clean":"2024-04-02T04:53:11.923279+0000","last_became_active":"2024-04-02T04:49:55.257987+0000","last_became_peered":"2024-04-02T04:49:55.257987+0000","last_unstale":"2024-04-02T04:53:11.923279+0000","last_undegraded":"2024-04-02T04:53:11.923279+0000","last_fullsized":"2024-04-02T04:53:11.923279+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-04-02T04:51:50.994396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.994396+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:20.498792+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.003033806,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.2","version":"273'39","reported_seq":574,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.920305+0000","last_change":"2024-04-02T04:52:53.896472+0000","last_active":"2024-04-02T04:53:11.920305+0000","last_peered":"2024-04-02T04:53:11.920305+0000","last_clean":"2024-04-02T04:53:11.920305+0000","last_became_active":"2024-04-02T04:49:55.235417+0000","last_became_peered":"2024-04-02T04:49:55.235417+0000","last_unstale":"2024-04-02T04:53:11.920305+0000","last_undegraded":"2024-04-02T04:53:11.920305+0000","last_fullsized":"2024-04-02T04:53:11.920305+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'39","last_scrub_stamp":"2024-04-02T04:52:41.329212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.329212+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:51.236088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023115700000000002,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.5","version":"229'85","reported_seq":615,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936353+0000","last_change":"2024-04-02T04:52:53.930302+0000","last_active":"2024-04-02T04:53:11.936353+0000","last_peered":"2024-04-02T04:53:11.936353+0000","last_clean":"2024-04-02T04:53:11.936353+0000","last_became_active":"2024-04-02T04:49:56.553311+0000","last_became_peered":"2024-04-02T04:49:56.553311+0000","last_unstale":"2024-04-02T04:53:11.936353+0000","last_undegraded":"2024-04-02T04:53:11.936353+0000","last_fullsized":"2024-04-02T04:53:11.936353+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'85","last_scrub_stamp":"2024-04-02T04:51:27.257882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.257882+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00035259499999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.4","version":"192'9","reported_seq":493,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.936756+0000","last_change":"2024-04-02T04:50:28.406787+0000","last_active":"2024-04-02T04:53:11.936756+0000","last_peered":"2024-04-02T04:53:11.936756+0000","last_clean":"2024-04-02T04:49:53.797011+0000","last_became_active":"2024-04-02T04:49:56.578685+0000","last_became_peered":"2024-04-02T04:49:56.578685+0000","last_unstale":"2024-04-02T04:53:11.936756+0000","last_undegraded":"2024-04-02T04:50:27.585397+0000","last_fullsized":"2024-04-02T04:49:56.549435+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.7","version":"255'18","reported_seq":535,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.927647+0000","last_change":"2024-04-02T04:52:53.897232+0000","last_active":"2024-04-02T04:53:11.927647+0000","last_peered":"2024-04-02T04:53:11.927647+0000","last_clean":"2024-04-02T04:53:11.927647+0000","last_became_active":"2024-04-02T04:51:57.862004+0000","last_became_peered":"2024-04-02T04:51:57.862004+0000","last_unstale":"2024-04-02T04:53:11.927647+0000","last_undegraded":"2024-04-02T04:53:11.927647+0000","last_fullsized":"2024-04-02T04:53:11.927647+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'18","last_scrub_stamp":"2024-04-02T04:52:40.288509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.288509+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:02.046535+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0020366989999999999,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.6","version":"236'34","reported_seq":572,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921928+0000","last_change":"2024-04-02T04:52:53.905214+0000","last_active":"2024-04-02T04:53:11.921928+0000","last_peered":"2024-04-02T04:53:11.921928+0000","last_clean":"2024-04-02T04:53:11.921928+0000","last_became_active":"2024-04-02T04:49:56.581977+0000","last_became_peered":"2024-04-02T04:49:56.581977+0000","last_unstale":"2024-04-02T04:53:11.921928+0000","last_undegraded":"2024-04-02T04:53:11.921928+0000","last_fullsized":"2024-04-02T04:53:11.921928+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-02T04:52:53.769683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.769683+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:05.305449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003762409,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"43.14","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.920605+0000","last_change":"2024-04-02T04:52:44.775768+0000","last_active":"2024-04-02T04:53:11.920605+0000","last_peered":"2024-04-02T04:53:11.920605+0000","last_clean":"2024-04-02T04:53:11.920605+0000","last_became_active":"2024-04-02T04:49:29.828535+0000","last_became_peered":"2024-04-02T04:49:29.828535+0000","last_unstale":"2024-04-02T04:53:11.920605+0000","last_undegraded":"2024-04-02T04:53:11.920605+0000","last_fullsized":"2024-04-02T04:53:11.920605+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.775672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.775672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:03.357544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919976+0000","last_change":"2024-04-02T04:49:47.428191+0000","last_active":"2024-04-02T04:53:11.919976+0000","last_peered":"2024-04-02T04:53:11.919976+0000","last_clean":"2024-04-02T04:49:45.347176+0000","last_became_active":"2024-04-02T04:49:47.428191+0000","last_became_peered":"2024-04-02T04:49:47.428191+0000","last_unstale":"2024-04-02T04:53:11.919976+0000","last_undegraded":"2024-04-02T04:53:11.919976+0000","last_fullsized":"2024-04-02T04:49:47.367526+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919406+0000","last_change":"2024-04-02T04:52:58.345528+0000","last_active":"2024-04-02T04:53:11.919406+0000","last_peered":"2024-04-02T04:53:11.919406+0000","last_clean":"2024-04-02T04:53:11.919406+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:53:11.919406+0000","last_undegraded":"2024-04-02T04:53:11.919406+0000","last_fullsized":"2024-04-02T04:53:11.919406+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.345234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.345234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:06.059336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916697+0000","last_change":"2024-04-02T04:51:46.284187+0000","last_active":"2024-04-02T04:53:11.916697+0000","last_peered":"2024-04-02T04:53:11.916697+0000","last_clean":"2024-04-02T04:53:11.916697+0000","last_became_active":"2024-04-02T04:49:28.787404+0000","last_became_peered":"2024-04-02T04:49:28.787404+0000","last_unstale":"2024-04-02T04:53:11.916697+0000","last_undegraded":"2024-04-02T04:53:11.916697+0000","last_fullsized":"2024-04-02T04:53:11.916697+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:46.284019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:46.284019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.098571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919298+0000","last_change":"2024-04-02T04:53:07.389165+0000","last_active":"2024-04-02T04:53:11.919298+0000","last_peered":"2024-04-02T04:53:11.919298+0000","last_clean":"2024-04-02T04:53:11.919298+0000","last_became_active":"2024-04-02T04:49:28.783594+0000","last_became_peered":"2024-04-02T04:49:28.783594+0000","last_unstale":"2024-04-02T04:53:11.919298+0000","last_undegraded":"2024-04-02T04:53:11.919298+0000","last_fullsized":"2024-04-02T04:53:11.919298+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.388952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.388952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:26.015336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926660+0000","last_change":"2024-04-02T04:51:44.929741+0000","last_active":"2024-04-02T04:53:11.926660+0000","last_peered":"2024-04-02T04:53:11.926660+0000","last_clean":"2024-04-02T04:53:11.926660+0000","last_became_active":"2024-04-02T04:49:28.726013+0000","last_became_peered":"2024-04-02T04:49:28.726013+0000","last_unstale":"2024-04-02T04:53:11.926660+0000","last_undegraded":"2024-04-02T04:53:11.926660+0000","last_fullsized":"2024-04-02T04:53:11.926660+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.929545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.929545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.922139+0000","last_change":"2024-04-02T04:49:47.479797+0000","last_active":"2024-04-02T04:53:11.922139+0000","last_peered":"2024-04-02T04:53:11.922139+0000","last_clean":"2024-04-02T04:49:45.330040+0000","last_became_active":"2024-04-02T04:49:47.479797+0000","last_became_peered":"2024-04-02T04:49:47.479797+0000","last_unstale":"2024-04-02T04:53:11.922139+0000","last_undegraded":"2024-04-02T04:53:11.922139+0000","last_fullsized":"2024-04-02T04:49:47.405691+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.951433+0000","last_change":"2024-04-02T04:51:39.644093+0000","last_active":"2024-04-02T04:53:11.951433+0000","last_peered":"2024-04-02T04:53:11.951433+0000","last_clean":"2024-04-02T04:53:11.951433+0000","last_became_active":"2024-04-02T04:49:28.759334+0000","last_became_peered":"2024-04-02T04:49:28.759334+0000","last_unstale":"2024-04-02T04:53:11.951433+0000","last_undegraded":"2024-04-02T04:53:11.951433+0000","last_fullsized":"2024-04-02T04:53:11.951433+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":438,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.925278+0000","last_change":"2024-04-02T04:49:47.517914+0000","last_active":"2024-04-02T04:53:11.925278+0000","last_peered":"2024-04-02T04:53:11.925278+0000","last_clean":"2024-04-02T04:49:45.379252+0000","last_became_active":"2024-04-02T04:49:47.517914+0000","last_became_peered":"2024-04-02T04:49:47.517914+0000","last_unstale":"2024-04-02T04:53:11.925278+0000","last_undegraded":"2024-04-02T04:53:11.925278+0000","last_fullsized":"2024-04-02T04:49:47.408049+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":410,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.921757+0000","last_change":"2024-04-02T04:49:47.477571+0000","last_active":"2024-04-02T04:53:11.921757+0000","last_peered":"2024-04-02T04:53:11.921757+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.477571+0000","last_became_peered":"2024-04-02T04:49:47.477571+0000","last_unstale":"2024-04-02T04:53:11.921757+0000","last_undegraded":"2024-04-02T04:53:11.921757+0000","last_fullsized":"2024-04-02T04:49:47.404962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":438,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.924878+0000","last_change":"2024-04-02T04:49:47.517552+0000","last_active":"2024-04-02T04:53:11.924878+0000","last_peered":"2024-04-02T04:53:11.924878+0000","last_clean":"2024-04-02T04:49:45.378670+0000","last_became_active":"2024-04-02T04:49:47.517552+0000","last_became_peered":"2024-04-02T04:49:47.517552+0000","last_unstale":"2024-04-02T04:53:11.924878+0000","last_undegraded":"2024-04-02T04:53:11.924878+0000","last_fullsized":"2024-04-02T04:49:47.406289+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942440+0000","last_change":"2024-04-02T04:53:07.468104+0000","last_active":"2024-04-02T04:53:11.942440+0000","last_peered":"2024-04-02T04:53:11.942440+0000","last_clean":"2024-04-02T04:53:11.942440+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:53:11.942440+0000","last_undegraded":"2024-04-02T04:53:11.942440+0000","last_fullsized":"2024-04-02T04:53:11.942440+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.467848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.467848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:09.323511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917121+0000","last_change":"2024-04-02T04:52:40.787636+0000","last_active":"2024-04-02T04:53:11.917121+0000","last_peered":"2024-04-02T04:53:11.917121+0000","last_clean":"2024-04-02T04:53:11.917121+0000","last_became_active":"2024-04-02T04:49:29.818524+0000","last_became_peered":"2024-04-02T04:49:29.818524+0000","last_unstale":"2024-04-02T04:53:11.917121+0000","last_undegraded":"2024-04-02T04:53:11.917121+0000","last_fullsized":"2024-04-02T04:53:11.917121+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.787478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.787478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:58.491931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936853+0000","last_change":"2024-04-02T04:51:51.039566+0000","last_active":"2024-04-02T04:53:11.936853+0000","last_peered":"2024-04-02T04:53:11.936853+0000","last_clean":"2024-04-02T04:53:11.936853+0000","last_became_active":"2024-04-02T04:49:28.769072+0000","last_became_peered":"2024-04-02T04:49:28.769072+0000","last_unstale":"2024-04-02T04:53:11.936853+0000","last_undegraded":"2024-04-02T04:53:11.936853+0000","last_fullsized":"2024-04-02T04:53:11.936853+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.039450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919282+0000","last_change":"2024-04-02T04:52:35.787957+0000","last_active":"2024-04-02T04:53:11.919282+0000","last_peered":"2024-04-02T04:53:11.919282+0000","last_clean":"2024-04-02T04:53:11.919282+0000","last_became_active":"2024-04-02T04:49:29.801654+0000","last_became_peered":"2024-04-02T04:49:29.801654+0000","last_unstale":"2024-04-02T04:53:11.919282+0000","last_undegraded":"2024-04-02T04:53:11.919282+0000","last_fullsized":"2024-04-02T04:53:11.919282+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:35.787733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:35.787733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:55.272197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.949113+0000","last_change":"2024-04-02T04:52:36.685201+0000","last_active":"2024-04-02T04:53:11.949113+0000","last_peered":"2024-04-02T04:53:11.949113+0000","last_clean":"2024-04-02T04:53:11.949113+0000","last_became_active":"2024-04-02T04:49:28.756548+0000","last_became_peered":"2024-04-02T04:49:28.756548+0000","last_unstale":"2024-04-02T04:53:11.949113+0000","last_undegraded":"2024-04-02T04:53:11.949113+0000","last_fullsized":"2024-04-02T04:53:11.949113+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.684685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.684685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:38.649119+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950570+0000","last_change":"2024-04-02T04:51:40.625223+0000","last_active":"2024-04-02T04:53:11.950570+0000","last_peered":"2024-04-02T04:53:11.950570+0000","last_clean":"2024-04-02T04:53:11.950570+0000","last_became_active":"2024-04-02T04:49:28.741277+0000","last_became_peered":"2024-04-02T04:49:28.741277+0000","last_unstale":"2024-04-02T04:53:11.950570+0000","last_undegraded":"2024-04-02T04:53:11.950570+0000","last_fullsized":"2024-04-02T04:53:11.950570+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.625010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.625010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.934135+0000","last_change":"2024-04-02T04:52:34.327003+0000","last_active":"2024-04-02T04:53:11.934135+0000","last_peered":"2024-04-02T04:53:11.934135+0000","last_clean":"2024-04-02T04:53:11.934135+0000","last_became_active":"2024-04-02T04:49:28.719086+0000","last_became_peered":"2024-04-02T04:49:28.719086+0000","last_unstale":"2024-04-02T04:53:11.934135+0000","last_undegraded":"2024-04-02T04:53:11.934135+0000","last_fullsized":"2024-04-02T04:53:11.934135+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.326885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.326885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:39.077081+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.935110+0000","last_change":"2024-04-02T04:52:36.311512+0000","last_active":"2024-04-02T04:53:11.935110+0000","last_peered":"2024-04-02T04:53:11.935110+0000","last_clean":"2024-04-02T04:53:11.935110+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:53:11.935110+0000","last_undegraded":"2024-04-02T04:53:11.935110+0000","last_fullsized":"2024-04-02T04:53:11.935110+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.311307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.238730+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933045+0000","last_change":"2024-04-02T04:52:45.221582+0000","last_active":"2024-04-02T04:53:11.933045+0000","last_peered":"2024-04-02T04:53:11.933045+0000","last_clean":"2024-04-02T04:53:11.933045+0000","last_became_active":"2024-04-02T04:49:28.731836+0000","last_became_peered":"2024-04-02T04:49:28.731836+0000","last_unstale":"2024-04-02T04:53:11.933045+0000","last_undegraded":"2024-04-02T04:53:11.933045+0000","last_fullsized":"2024-04-02T04:53:11.933045+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:45.221445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:45.221445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:57.458264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.927982+0000","last_change":"2024-04-02T04:52:52.665425+0000","last_active":"2024-04-02T04:53:11.927982+0000","last_peered":"2024-04-02T04:53:11.927982+0000","last_clean":"2024-04-02T04:53:11.927982+0000","last_became_active":"2024-04-02T04:49:28.727703+0000","last_became_peered":"2024-04-02T04:49:28.727703+0000","last_unstale":"2024-04-02T04:53:11.927982+0000","last_undegraded":"2024-04-02T04:53:11.927982+0000","last_fullsized":"2024-04-02T04:53:11.927982+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.665317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.665317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:56.472500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":440,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.935630+0000","last_change":"2024-04-02T04:49:47.517107+0000","last_active":"2024-04-02T04:53:11.935630+0000","last_peered":"2024-04-02T04:53:11.935630+0000","last_clean":"2024-04-02T04:49:45.381150+0000","last_became_active":"2024-04-02T04:49:47.517107+0000","last_became_peered":"2024-04-02T04:49:47.517107+0000","last_unstale":"2024-04-02T04:53:11.935630+0000","last_undegraded":"2024-04-02T04:53:11.935630+0000","last_fullsized":"2024-04-02T04:49:47.429362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.948912+0000","last_change":"2024-04-02T04:49:47.444375+0000","last_active":"2024-04-02T04:53:11.948912+0000","last_peered":"2024-04-02T04:53:11.948912+0000","last_clean":"2024-04-02T04:49:45.331671+0000","last_became_active":"2024-04-02T04:49:47.444375+0000","last_became_peered":"2024-04-02T04:49:47.444375+0000","last_unstale":"2024-04-02T04:53:11.948912+0000","last_undegraded":"2024-04-02T04:53:11.948912+0000","last_fullsized":"2024-04-02T04:49:47.336937+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.948940+0000","last_change":"2024-04-02T04:49:47.381163+0000","last_active":"2024-04-02T04:53:11.948940+0000","last_peered":"2024-04-02T04:53:11.948940+0000","last_clean":"2024-04-02T04:49:45.330129+0000","last_became_active":"2024-04-02T04:49:47.381163+0000","last_became_peered":"2024-04-02T04:49:47.381163+0000","last_unstale":"2024-04-02T04:53:11.948940+0000","last_undegraded":"2024-04-02T04:53:11.948940+0000","last_fullsized":"2024-04-02T04:49:47.338896+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.945617+0000","last_change":"2024-04-02T04:51:41.090918+0000","last_active":"2024-04-02T04:53:11.945617+0000","last_peered":"2024-04-02T04:53:11.945617+0000","last_clean":"2024-04-02T04:53:11.945617+0000","last_became_active":"2024-04-02T04:49:28.762640+0000","last_became_peered":"2024-04-02T04:49:28.762640+0000","last_unstale":"2024-04-02T04:53:11.945617+0000","last_undegraded":"2024-04-02T04:53:11.945617+0000","last_fullsized":"2024-04-02T04:53:11.945617+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:41.090714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:41.090714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941691+0000","last_change":"2024-04-02T04:52:59.362968+0000","last_active":"2024-04-02T04:53:11.941691+0000","last_peered":"2024-04-02T04:53:11.941691+0000","last_clean":"2024-04-02T04:53:11.941691+0000","last_became_active":"2024-04-02T04:49:28.767868+0000","last_became_peered":"2024-04-02T04:49:28.767868+0000","last_unstale":"2024-04-02T04:53:11.941691+0000","last_undegraded":"2024-04-02T04:53:11.941691+0000","last_fullsized":"2024-04-02T04:53:11.941691+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.362748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.362748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:16.883581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":438,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.927562+0000","last_change":"2024-04-02T04:49:47.508050+0000","last_active":"2024-04-02T04:53:11.927562+0000","last_peered":"2024-04-02T04:53:11.927562+0000","last_clean":"2024-04-02T04:49:45.381990+0000","last_became_active":"2024-04-02T04:49:47.508050+0000","last_became_peered":"2024-04-02T04:49:47.508050+0000","last_unstale":"2024-04-02T04:53:11.927562+0000","last_undegraded":"2024-04-02T04:53:11.927562+0000","last_fullsized":"2024-04-02T04:49:47.429001+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.921794+0000","last_change":"2024-04-02T04:49:47.490495+0000","last_active":"2024-04-02T04:53:11.921794+0000","last_peered":"2024-04-02T04:53:11.921794+0000","last_clean":"2024-04-02T04:49:45.351290+0000","last_became_active":"2024-04-02T04:49:47.490495+0000","last_became_peered":"2024-04-02T04:49:47.490495+0000","last_unstale":"2024-04-02T04:53:11.921794+0000","last_undegraded":"2024-04-02T04:53:11.921794+0000","last_fullsized":"2024-04-02T04:49:47.384086+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.940434+0000","last_change":"2024-04-02T04:51:30.306477+0000","last_active":"2024-04-02T04:53:11.940434+0000","last_peered":"2024-04-02T04:53:11.940434+0000","last_clean":"2024-04-02T04:53:11.940434+0000","last_became_active":"2024-04-02T04:49:28.720246+0000","last_became_peered":"2024-04-02T04:49:28.720246+0000","last_unstale":"2024-04-02T04:53:11.940434+0000","last_undegraded":"2024-04-02T04:53:11.940434+0000","last_fullsized":"2024-04-02T04:53:11.940434+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.306343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.306343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942075+0000","last_change":"2024-04-02T04:51:52.078040+0000","last_active":"2024-04-02T04:53:11.942075+0000","last_peered":"2024-04-02T04:53:11.942075+0000","last_clean":"2024-04-02T04:53:11.942075+0000","last_became_active":"2024-04-02T04:49:28.781235+0000","last_became_peered":"2024-04-02T04:49:28.781235+0000","last_unstale":"2024-04-02T04:53:11.942075+0000","last_undegraded":"2024-04-02T04:53:11.942075+0000","last_fullsized":"2024-04-02T04:53:11.942075+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.077908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.077908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":436,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919858+0000","last_change":"2024-04-02T04:49:28.787959+0000","last_active":"2024-04-02T04:53:11.919858+0000","last_peered":"2024-04-02T04:53:11.919858+0000","last_clean":"2024-04-02T04:53:11.919858+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:53:11.919858+0000","last_undegraded":"2024-04-02T04:53:11.919858+0000","last_fullsized":"2024-04-02T04:53:11.919858+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.916757+0000","last_change":"2024-04-02T04:49:47.389948+0000","last_active":"2024-04-02T04:53:11.916757+0000","last_peered":"2024-04-02T04:53:11.916757+0000","last_clean":"2024-04-02T04:49:45.343249+0000","last_became_active":"2024-04-02T04:49:47.389948+0000","last_became_peered":"2024-04-02T04:49:47.389948+0000","last_unstale":"2024-04-02T04:53:11.916757+0000","last_undegraded":"2024-04-02T04:53:11.916757+0000","last_fullsized":"2024-04-02T04:49:47.366723+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917183+0000","last_change":"2024-04-02T04:52:51.287269+0000","last_active":"2024-04-02T04:53:11.917183+0000","last_peered":"2024-04-02T04:53:11.917183+0000","last_clean":"2024-04-02T04:53:11.917183+0000","last_became_active":"2024-04-02T04:49:28.750373+0000","last_became_peered":"2024-04-02T04:49:28.750373+0000","last_unstale":"2024-04-02T04:53:11.917183+0000","last_undegraded":"2024-04-02T04:53:11.917183+0000","last_fullsized":"2024-04-02T04:53:11.917183+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:51.287037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.287037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:18.722013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":409,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.952366+0000","last_change":"2024-04-02T04:49:47.389543+0000","last_active":"2024-04-02T04:53:11.952366+0000","last_peered":"2024-04-02T04:53:11.952366+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.389543+0000","last_became_peered":"2024-04-02T04:49:47.389543+0000","last_unstale":"2024-04-02T04:53:11.952366+0000","last_undegraded":"2024-04-02T04:53:11.952366+0000","last_fullsized":"2024-04-02T04:49:47.363220+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.952625+0000","last_change":"2024-04-02T04:49:47.391015+0000","last_active":"2024-04-02T04:53:11.952625+0000","last_peered":"2024-04-02T04:53:11.952625+0000","last_clean":"2024-04-02T04:49:45.363593+0000","last_became_active":"2024-04-02T04:49:47.391015+0000","last_became_peered":"2024-04-02T04:49:47.391015+0000","last_unstale":"2024-04-02T04:53:11.952625+0000","last_undegraded":"2024-04-02T04:53:11.952625+0000","last_fullsized":"2024-04-02T04:49:47.362030+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":409,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.950767+0000","last_change":"2024-04-02T04:49:47.434747+0000","last_active":"2024-04-02T04:53:11.950767+0000","last_peered":"2024-04-02T04:53:11.950767+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.434747+0000","last_became_peered":"2024-04-02T04:49:47.434747+0000","last_unstale":"2024-04-02T04:53:11.950767+0000","last_undegraded":"2024-04-02T04:53:11.950767+0000","last_fullsized":"2024-04-02T04:49:47.346344+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":442,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926160+0000","last_change":"2024-04-02T04:51:43.198139+0000","last_active":"2024-04-02T04:53:11.926160+0000","last_peered":"2024-04-02T04:53:11.926160+0000","last_clean":"2024-04-02T04:53:11.926160+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:53:11.926160+0000","last_undegraded":"2024-04-02T04:53:11.926160+0000","last_fullsized":"2024-04-02T04:53:11.926160+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:43.198044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.198044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":436,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916156+0000","last_change":"2024-04-02T04:49:28.787621+0000","last_active":"2024-04-02T04:53:11.916156+0000","last_peered":"2024-04-02T04:53:11.916156+0000","last_clean":"2024-04-02T04:53:11.916156+0000","last_became_active":"2024-04-02T04:49:28.785195+0000","last_became_peered":"2024-04-02T04:49:28.785195+0000","last_unstale":"2024-04-02T04:53:11.916156+0000","last_undegraded":"2024-04-02T04:53:11.916156+0000","last_fullsized":"2024-04-02T04:53:11.916156+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":436,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922118+0000","last_change":"2024-04-02T04:49:28.743937+0000","last_active":"2024-04-02T04:53:11.922118+0000","last_peered":"2024-04-02T04:53:11.922118+0000","last_clean":"2024-04-02T04:53:11.922118+0000","last_became_active":"2024-04-02T04:49:28.741450+0000","last_became_peered":"2024-04-02T04:49:28.741450+0000","last_unstale":"2024-04-02T04:53:11.922118+0000","last_undegraded":"2024-04-02T04:53:11.922118+0000","last_fullsized":"2024-04-02T04:53:11.922118+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916172+0000","last_change":"2024-04-02T04:53:02.294549+0000","last_active":"2024-04-02T04:53:11.916172+0000","last_peered":"2024-04-02T04:53:11.916172+0000","last_clean":"2024-04-02T04:53:11.916172+0000","last_became_active":"2024-04-02T04:49:28.731343+0000","last_became_peered":"2024-04-02T04:49:28.731343+0000","last_unstale":"2024-04-02T04:53:11.916172+0000","last_undegraded":"2024-04-02T04:53:11.916172+0000","last_fullsized":"2024-04-02T04:53:11.916172+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.294309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.735531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.923451+0000","last_change":"2024-04-02T04:50:37.102849+0000","last_active":"2024-04-02T04:53:11.923451+0000","last_peered":"2024-04-02T04:53:11.923451+0000","last_clean":"2024-04-02T04:53:11.923451+0000","last_became_active":"2024-04-02T04:49:28.698110+0000","last_became_peered":"2024-04-02T04:49:28.698110+0000","last_unstale":"2024-04-02T04:53:11.923451+0000","last_undegraded":"2024-04-02T04:53:11.923451+0000","last_fullsized":"2024-04-02T04:53:11.923451+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.102732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.102732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":438,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.921705+0000","last_change":"2024-04-02T04:49:47.445509+0000","last_active":"2024-04-02T04:53:11.921705+0000","last_peered":"2024-04-02T04:53:11.921705+0000","last_clean":"2024-04-02T04:49:45.370130+0000","last_became_active":"2024-04-02T04:49:47.445509+0000","last_became_peered":"2024-04-02T04:49:47.445509+0000","last_unstale":"2024-04-02T04:53:11.921705+0000","last_undegraded":"2024-04-02T04:53:11.921705+0000","last_fullsized":"2024-04-02T04:49:47.391250+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916150+0000","last_change":"2024-04-02T04:52:43.275598+0000","last_active":"2024-04-02T04:53:11.916150+0000","last_peered":"2024-04-02T04:53:11.916150+0000","last_clean":"2024-04-02T04:53:11.916150+0000","last_became_active":"2024-04-02T04:49:28.746056+0000","last_became_peered":"2024-04-02T04:49:28.746056+0000","last_unstale":"2024-04-02T04:53:11.916150+0000","last_undegraded":"2024-04-02T04:53:11.916150+0000","last_fullsized":"2024-04-02T04:53:11.916150+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:43.275420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.275420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:05.341488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.949156+0000","last_change":"2024-04-02T04:49:47.388801+0000","last_active":"2024-04-02T04:53:11.949156+0000","last_peered":"2024-04-02T04:53:11.949156+0000","last_clean":"2024-04-02T04:49:45.330442+0000","last_became_active":"2024-04-02T04:49:47.388801+0000","last_became_peered":"2024-04-02T04:49:47.388801+0000","last_unstale":"2024-04-02T04:53:11.949156+0000","last_undegraded":"2024-04-02T04:53:11.949156+0000","last_fullsized":"2024-04-02T04:49:47.339290+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.948982+0000","last_change":"2024-04-02T04:49:47.444341+0000","last_active":"2024-04-02T04:53:11.948982+0000","last_peered":"2024-04-02T04:53:11.948982+0000","last_clean":"2024-04-02T04:49:45.331782+0000","last_became_active":"2024-04-02T04:49:47.444341+0000","last_became_peered":"2024-04-02T04:49:47.444341+0000","last_unstale":"2024-04-02T04:53:11.948982+0000","last_undegraded":"2024-04-02T04:53:11.948982+0000","last_fullsized":"2024-04-02T04:49:47.336855+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919191+0000","last_change":"2024-04-02T04:52:38.809910+0000","last_active":"2024-04-02T04:53:11.919191+0000","last_peered":"2024-04-02T04:53:11.919191+0000","last_clean":"2024-04-02T04:53:11.919191+0000","last_became_active":"2024-04-02T04:49:29.857200+0000","last_became_peered":"2024-04-02T04:49:29.857200+0000","last_unstale":"2024-04-02T04:53:11.919191+0000","last_undegraded":"2024-04-02T04:53:11.919191+0000","last_fullsized":"2024-04-02T04:53:11.919191+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.809807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.809807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:55.783074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918453+0000","last_change":"2024-04-02T04:51:35.000630+0000","last_active":"2024-04-02T04:53:11.918453+0000","last_peered":"2024-04-02T04:53:11.918453+0000","last_clean":"2024-04-02T04:53:11.918453+0000","last_became_active":"2024-04-02T04:49:28.746855+0000","last_became_peered":"2024-04-02T04:49:28.746855+0000","last_unstale":"2024-04-02T04:53:11.918453+0000","last_undegraded":"2024-04-02T04:53:11.918453+0000","last_fullsized":"2024-04-02T04:53:11.918453+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:35.000533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.000533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":438,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.917875+0000","last_change":"2024-04-02T04:49:47.472640+0000","last_active":"2024-04-02T04:53:11.917875+0000","last_peered":"2024-04-02T04:53:11.917875+0000","last_clean":"2024-04-02T04:49:45.352140+0000","last_became_active":"2024-04-02T04:49:47.472640+0000","last_became_peered":"2024-04-02T04:49:47.472640+0000","last_unstale":"2024-04-02T04:53:11.917875+0000","last_undegraded":"2024-04-02T04:53:11.917875+0000","last_fullsized":"2024-04-02T04:49:47.377460+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.920474+0000","last_change":"2024-04-02T04:51:36.554382+0000","last_active":"2024-04-02T04:53:11.920474+0000","last_peered":"2024-04-02T04:53:11.920474+0000","last_clean":"2024-04-02T04:53:11.920474+0000","last_became_active":"2024-04-02T04:49:29.874539+0000","last_became_peered":"2024-04-02T04:49:29.874539+0000","last_unstale":"2024-04-02T04:53:11.920474+0000","last_undegraded":"2024-04-02T04:53:11.920474+0000","last_fullsized":"2024-04-02T04:53:11.920474+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.554164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.554164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.920014+0000","last_change":"2024-04-02T04:49:47.452282+0000","last_active":"2024-04-02T04:53:11.920014+0000","last_peered":"2024-04-02T04:53:11.920014+0000","last_clean":"2024-04-02T04:49:45.347315+0000","last_became_active":"2024-04-02T04:49:47.452282+0000","last_became_peered":"2024-04-02T04:49:47.452282+0000","last_unstale":"2024-04-02T04:53:11.920014+0000","last_undegraded":"2024-04-02T04:53:11.920014+0000","last_fullsized":"2024-04-02T04:49:47.364280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921881+0000","last_change":"2024-04-02T04:51:53.758572+0000","last_active":"2024-04-02T04:53:11.921881+0000","last_peered":"2024-04-02T04:53:11.921881+0000","last_clean":"2024-04-02T04:53:11.921881+0000","last_became_active":"2024-04-02T04:49:29.830080+0000","last_became_peered":"2024-04-02T04:49:29.830080+0000","last_unstale":"2024-04-02T04:53:11.921881+0000","last_undegraded":"2024-04-02T04:53:11.921881+0000","last_fullsized":"2024-04-02T04:53:11.921881+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.758420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.758420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:20.112324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":440,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.936680+0000","last_change":"2024-04-02T04:49:47.477237+0000","last_active":"2024-04-02T04:53:11.936680+0000","last_peered":"2024-04-02T04:53:11.936680+0000","last_clean":"2024-04-02T04:49:45.366065+0000","last_became_active":"2024-04-02T04:49:47.477237+0000","last_became_peered":"2024-04-02T04:49:47.477237+0000","last_unstale":"2024-04-02T04:53:11.936680+0000","last_undegraded":"2024-04-02T04:53:11.936680+0000","last_fullsized":"2024-04-02T04:49:47.430531+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942078+0000","last_change":"2024-04-02T04:51:36.096984+0000","last_active":"2024-04-02T04:53:11.942078+0000","last_peered":"2024-04-02T04:53:11.942078+0000","last_clean":"2024-04-02T04:53:11.942078+0000","last_became_active":"2024-04-02T04:49:28.780779+0000","last_became_peered":"2024-04-02T04:49:28.780779+0000","last_unstale":"2024-04-02T04:53:11.942078+0000","last_undegraded":"2024-04-02T04:53:11.942078+0000","last_fullsized":"2024-04-02T04:53:11.942078+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.096832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.096832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921869+0000","last_change":"2024-04-02T04:52:46.335166+0000","last_active":"2024-04-02T04:53:11.921869+0000","last_peered":"2024-04-02T04:53:11.921869+0000","last_clean":"2024-04-02T04:53:11.921869+0000","last_became_active":"2024-04-02T04:49:28.784897+0000","last_became_peered":"2024-04-02T04:49:28.784897+0000","last_unstale":"2024-04-02T04:53:11.921869+0000","last_undegraded":"2024-04-02T04:53:11.921869+0000","last_fullsized":"2024-04-02T04:53:11.921869+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.334949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.334949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.534127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":452,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.951576+0000","last_change":"2024-04-02T04:53:08.577436+0000","last_active":"2024-04-02T04:53:11.951576+0000","last_peered":"2024-04-02T04:53:11.951576+0000","last_clean":"2024-04-02T04:53:11.951576+0000","last_became_active":"2024-04-02T04:49:28.744153+0000","last_became_peered":"2024-04-02T04:49:28.744153+0000","last_unstale":"2024-04-02T04:53:11.951576+0000","last_undegraded":"2024-04-02T04:53:11.951576+0000","last_fullsized":"2024-04-02T04:53:11.951576+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:08.577227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:08.577227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:12.696737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942213+0000","last_change":"2024-04-02T04:53:10.492126+0000","last_active":"2024-04-02T04:53:11.942213+0000","last_peered":"2024-04-02T04:53:11.942213+0000","last_clean":"2024-04-02T04:53:11.942213+0000","last_became_active":"2024-04-02T04:49:28.760661+0000","last_became_peered":"2024-04-02T04:49:28.760661+0000","last_unstale":"2024-04-02T04:53:11.942213+0000","last_undegraded":"2024-04-02T04:53:11.942213+0000","last_fullsized":"2024-04-02T04:53:11.942213+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.491899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.491899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:32.562654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.945740+0000","last_change":"2024-04-02T04:52:39.367465+0000","last_active":"2024-04-02T04:53:11.945740+0000","last_peered":"2024-04-02T04:53:11.945740+0000","last_clean":"2024-04-02T04:53:11.945740+0000","last_became_active":"2024-04-02T04:49:28.769787+0000","last_became_peered":"2024-04-02T04:49:28.769787+0000","last_unstale":"2024-04-02T04:53:11.945740+0000","last_undegraded":"2024-04-02T04:53:11.945740+0000","last_fullsized":"2024-04-02T04:53:11.945740+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:39.367130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:39.367130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:56.573097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":442,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930577+0000","last_change":"2024-04-02T04:51:44.205952+0000","last_active":"2024-04-02T04:53:11.930577+0000","last_peered":"2024-04-02T04:53:11.930577+0000","last_clean":"2024-04-02T04:53:11.930577+0000","last_became_active":"2024-04-02T04:49:28.728416+0000","last_became_peered":"2024-04-02T04:49:28.728416+0000","last_unstale":"2024-04-02T04:53:11.930577+0000","last_undegraded":"2024-04-02T04:53:11.930577+0000","last_fullsized":"2024-04-02T04:53:11.930577+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.205739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.205739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":442,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930592+0000","last_change":"2024-04-02T04:51:42.223186+0000","last_active":"2024-04-02T04:53:11.930592+0000","last_peered":"2024-04-02T04:53:11.930592+0000","last_clean":"2024-04-02T04:53:11.930592+0000","last_became_active":"2024-04-02T04:49:28.727661+0000","last_became_peered":"2024-04-02T04:49:28.727661+0000","last_unstale":"2024-04-02T04:53:11.930592+0000","last_undegraded":"2024-04-02T04:53:11.930592+0000","last_fullsized":"2024-04-02T04:53:11.930592+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:42.223094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:42.223094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.945498+0000","last_change":"2024-04-02T04:53:00.409859+0000","last_active":"2024-04-02T04:53:11.945498+0000","last_peered":"2024-04-02T04:53:11.945498+0000","last_clean":"2024-04-02T04:53:11.945498+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:53:11.945498+0000","last_undegraded":"2024-04-02T04:53:11.945498+0000","last_fullsized":"2024-04-02T04:53:11.945498+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.409610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.409610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:05.010589+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.937775+0000","last_change":"2024-04-02T04:53:06.476121+0000","last_active":"2024-04-02T04:53:11.937775+0000","last_peered":"2024-04-02T04:53:11.937775+0000","last_clean":"2024-04-02T04:53:11.937775+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:53:11.937775+0000","last_undegraded":"2024-04-02T04:53:11.937775+0000","last_fullsized":"2024-04-02T04:53:11.937775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.475984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.475984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:09.567682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":407,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.928369+0000","last_change":"2024-04-02T04:49:47.512403+0000","last_active":"2024-04-02T04:53:11.928369+0000","last_peered":"2024-04-02T04:53:11.928369+0000","last_clean":"2024-04-02T04:49:24.948823+0000","last_became_active":"2024-04-02T04:49:47.512403+0000","last_became_peered":"2024-04-02T04:49:47.512403+0000","last_unstale":"2024-04-02T04:53:11.928369+0000","last_undegraded":"2024-04-02T04:53:11.928369+0000","last_fullsized":"2024-04-02T04:49:47.379434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.923482+0000","last_change":"2024-04-02T04:51:48.958576+0000","last_active":"2024-04-02T04:53:11.923482+0000","last_peered":"2024-04-02T04:53:11.923482+0000","last_clean":"2024-04-02T04:53:11.923482+0000","last_became_active":"2024-04-02T04:49:28.725896+0000","last_became_peered":"2024-04-02T04:49:28.725896+0000","last_unstale":"2024-04-02T04:53:11.923482+0000","last_undegraded":"2024-04-02T04:53:11.923482+0000","last_fullsized":"2024-04-02T04:53:11.923482+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.958371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:14.943164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.948866+0000","last_change":"2024-04-02T04:51:26.661470+0000","last_active":"2024-04-02T04:53:11.948866+0000","last_peered":"2024-04-02T04:53:11.948866+0000","last_clean":"2024-04-02T04:53:11.948866+0000","last_became_active":"2024-04-02T04:49:28.724387+0000","last_became_peered":"2024-04-02T04:49:28.724387+0000","last_unstale":"2024-04-02T04:53:11.948866+0000","last_undegraded":"2024-04-02T04:53:11.948866+0000","last_fullsized":"2024-04-02T04:53:11.948866+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.661376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.661376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919366+0000","last_change":"2024-04-02T04:51:27.695556+0000","last_active":"2024-04-02T04:53:11.919366+0000","last_peered":"2024-04-02T04:53:11.919366+0000","last_clean":"2024-04-02T04:53:11.919366+0000","last_became_active":"2024-04-02T04:49:29.856708+0000","last_became_peered":"2024-04-02T04:49:29.856708+0000","last_unstale":"2024-04-02T04:53:11.919366+0000","last_undegraded":"2024-04-02T04:53:11.919366+0000","last_fullsized":"2024-04-02T04:53:11.919366+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:27.695437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.695437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918726+0000","last_change":"2024-04-02T04:51:29.649636+0000","last_active":"2024-04-02T04:53:11.918726+0000","last_peered":"2024-04-02T04:53:11.918726+0000","last_clean":"2024-04-02T04:53:11.918726+0000","last_became_active":"2024-04-02T04:49:29.816194+0000","last_became_peered":"2024-04-02T04:49:29.816194+0000","last_unstale":"2024-04-02T04:53:11.918726+0000","last_undegraded":"2024-04-02T04:53:11.918726+0000","last_fullsized":"2024-04-02T04:53:11.918726+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.649422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.649422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":437,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.928581+0000","last_change":"2024-04-02T04:49:47.427831+0000","last_active":"2024-04-02T04:53:11.928581+0000","last_peered":"2024-04-02T04:53:11.928581+0000","last_clean":"2024-04-02T04:49:45.321587+0000","last_became_active":"2024-04-02T04:49:47.427831+0000","last_became_peered":"2024-04-02T04:49:47.427831+0000","last_unstale":"2024-04-02T04:53:11.928581+0000","last_undegraded":"2024-04-02T04:53:11.928581+0000","last_fullsized":"2024-04-02T04:49:47.369909+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.916572+0000","last_change":"2024-04-02T04:49:47.387543+0000","last_active":"2024-04-02T04:53:11.916572+0000","last_peered":"2024-04-02T04:53:11.916572+0000","last_clean":"2024-04-02T04:49:45.306095+0000","last_became_active":"2024-04-02T04:49:47.387543+0000","last_became_peered":"2024-04-02T04:49:47.387543+0000","last_unstale":"2024-04-02T04:53:11.916572+0000","last_undegraded":"2024-04-02T04:53:11.916572+0000","last_fullsized":"2024-04-02T04:49:47.345085+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933520+0000","last_change":"2024-04-02T04:52:44.252234+0000","last_active":"2024-04-02T04:53:11.933520+0000","last_peered":"2024-04-02T04:53:11.933520+0000","last_clean":"2024-04-02T04:53:11.933520+0000","last_became_active":"2024-04-02T04:49:28.734672+0000","last_became_peered":"2024-04-02T04:49:28.734672+0000","last_unstale":"2024-04-02T04:53:11.933520+0000","last_undegraded":"2024-04-02T04:53:11.933520+0000","last_fullsized":"2024-04-02T04:53:11.933520+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.252102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.252102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:07.491596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.928999+0000","last_change":"2024-04-02T04:52:56.597071+0000","last_active":"2024-04-02T04:53:11.928999+0000","last_peered":"2024-04-02T04:53:11.928999+0000","last_clean":"2024-04-02T04:53:11.928999+0000","last_became_active":"2024-04-02T04:49:28.749072+0000","last_became_peered":"2024-04-02T04:49:28.749072+0000","last_unstale":"2024-04-02T04:53:11.928999+0000","last_undegraded":"2024-04-02T04:53:11.928999+0000","last_fullsized":"2024-04-02T04:53:11.928999+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.596968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.596968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:26.432888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917821+0000","last_change":"2024-04-02T04:51:51.741608+0000","last_active":"2024-04-02T04:53:11.917821+0000","last_peered":"2024-04-02T04:53:11.917821+0000","last_clean":"2024-04-02T04:53:11.917821+0000","last_became_active":"2024-04-02T04:49:29.858586+0000","last_became_peered":"2024-04-02T04:49:29.858586+0000","last_unstale":"2024-04-02T04:53:11.917821+0000","last_undegraded":"2024-04-02T04:53:11.917821+0000","last_fullsized":"2024-04-02T04:53:11.917821+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.741382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.741382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":440,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.932404+0000","last_change":"2024-04-02T04:49:47.469970+0000","last_active":"2024-04-02T04:53:11.932404+0000","last_peered":"2024-04-02T04:53:11.932404+0000","last_clean":"2024-04-02T04:49:45.328796+0000","last_became_active":"2024-04-02T04:49:47.469970+0000","last_became_peered":"2024-04-02T04:49:47.469970+0000","last_unstale":"2024-04-02T04:53:11.932404+0000","last_undegraded":"2024-04-02T04:53:11.932404+0000","last_fullsized":"2024-04-02T04:49:47.382953+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":408,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.941590+0000","last_change":"2024-04-02T04:49:47.457335+0000","last_active":"2024-04-02T04:53:11.941590+0000","last_peered":"2024-04-02T04:53:11.941590+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.457335+0000","last_became_peered":"2024-04-02T04:49:47.457335+0000","last_unstale":"2024-04-02T04:53:11.941590+0000","last_undegraded":"2024-04-02T04:53:11.941590+0000","last_fullsized":"2024-04-02T04:49:47.405947+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.463226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.952074+0000","last_change":"2024-04-02T04:51:55.540774+0000","last_active":"2024-04-02T04:53:11.952074+0000","last_peered":"2024-04-02T04:53:11.952074+0000","last_clean":"2024-04-02T04:53:11.952074+0000","last_became_active":"2024-04-02T04:49:28.750713+0000","last_became_peered":"2024-04-02T04:49:28.750713+0000","last_unstale":"2024-04-02T04:53:11.952074+0000","last_undegraded":"2024-04-02T04:53:11.952074+0000","last_fullsized":"2024-04-02T04:53:11.952074+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.540654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.540654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:16.003120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942164+0000","last_change":"2024-04-02T04:52:53.411217+0000","last_active":"2024-04-02T04:53:11.942164+0000","last_peered":"2024-04-02T04:53:11.942164+0000","last_clean":"2024-04-02T04:53:11.942164+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:53:11.942164+0000","last_undegraded":"2024-04-02T04:53:11.942164+0000","last_fullsized":"2024-04-02T04:53:11.942164+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:53.410913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.410913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:16.643056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":410,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.936449+0000","last_change":"2024-04-02T04:49:47.500400+0000","last_active":"2024-04-02T04:53:11.936449+0000","last_peered":"2024-04-02T04:53:11.936449+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.500400+0000","last_became_peered":"2024-04-02T04:49:47.500400+0000","last_unstale":"2024-04-02T04:53:11.936449+0000","last_undegraded":"2024-04-02T04:53:11.936449+0000","last_fullsized":"2024-04-02T04:49:47.430108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930645+0000","last_change":"2024-04-02T04:52:48.599707+0000","last_active":"2024-04-02T04:53:11.930645+0000","last_peered":"2024-04-02T04:53:11.930645+0000","last_clean":"2024-04-02T04:53:11.930645+0000","last_became_active":"2024-04-02T04:49:28.716933+0000","last_became_peered":"2024-04-02T04:49:28.716933+0000","last_unstale":"2024-04-02T04:53:11.930645+0000","last_undegraded":"2024-04-02T04:53:11.930645+0000","last_fullsized":"2024-04-02T04:53:11.930645+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:48.599595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.599595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:11.497654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921864+0000","last_change":"2024-04-02T04:51:52.390967+0000","last_active":"2024-04-02T04:53:11.921864+0000","last_peered":"2024-04-02T04:53:11.921864+0000","last_clean":"2024-04-02T04:53:11.921864+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:53:11.921864+0000","last_undegraded":"2024-04-02T04:53:11.921864+0000","last_fullsized":"2024-04-02T04:53:11.921864+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.390694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.390694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941680+0000","last_change":"2024-04-02T04:53:11.473195+0000","last_active":"2024-04-02T04:53:11.941680+0000","last_peered":"2024-04-02T04:53:11.941680+0000","last_clean":"2024-04-02T04:53:11.941680+0000","last_became_active":"2024-04-02T04:49:28.757217+0000","last_became_peered":"2024-04-02T04:49:28.757217+0000","last_unstale":"2024-04-02T04:53:11.941680+0000","last_undegraded":"2024-04-02T04:53:11.941680+0000","last_fullsized":"2024-04-02T04:53:11.941680+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.473096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.473096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:11.825694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":440,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.935773+0000","last_change":"2024-04-02T04:49:47.508156+0000","last_active":"2024-04-02T04:53:11.935773+0000","last_peered":"2024-04-02T04:53:11.935773+0000","last_clean":"2024-04-02T04:49:45.359498+0000","last_became_active":"2024-04-02T04:49:47.508156+0000","last_became_peered":"2024-04-02T04:49:47.508156+0000","last_unstale":"2024-04-02T04:53:11.935773+0000","last_undegraded":"2024-04-02T04:53:11.935773+0000","last_fullsized":"2024-04-02T04:49:47.430692+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.925181+0000","last_change":"2024-04-02T04:51:25.125020+0000","last_active":"2024-04-02T04:53:11.925181+0000","last_peered":"2024-04-02T04:53:11.925181+0000","last_clean":"2024-04-02T04:53:11.925181+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:53:11.925181+0000","last_undegraded":"2024-04-02T04:53:11.925181+0000","last_fullsized":"2024-04-02T04:53:11.925181+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:25.124929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.124929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":440,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.940597+0000","last_change":"2024-04-02T04:49:47.501136+0000","last_active":"2024-04-02T04:53:11.940597+0000","last_peered":"2024-04-02T04:53:11.940597+0000","last_clean":"2024-04-02T04:49:45.381195+0000","last_became_active":"2024-04-02T04:49:47.501136+0000","last_became_peered":"2024-04-02T04:49:47.501136+0000","last_unstale":"2024-04-02T04:53:11.940597+0000","last_undegraded":"2024-04-02T04:53:11.940597+0000","last_fullsized":"2024-04-02T04:49:47.416717+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":438,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.925440+0000","last_change":"2024-04-02T04:49:47.510048+0000","last_active":"2024-04-02T04:53:11.925440+0000","last_peered":"2024-04-02T04:53:11.925440+0000","last_clean":"2024-04-02T04:49:45.381640+0000","last_became_active":"2024-04-02T04:49:47.510048+0000","last_became_peered":"2024-04-02T04:49:47.510048+0000","last_unstale":"2024-04-02T04:53:11.925440+0000","last_undegraded":"2024-04-02T04:53:11.925440+0000","last_fullsized":"2024-04-02T04:49:47.408507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.954300+0000","last_change":"2024-04-02T04:51:29.673932+0000","last_active":"2024-04-02T04:53:11.954300+0000","last_peered":"2024-04-02T04:53:11.954300+0000","last_clean":"2024-04-02T04:53:11.954300+0000","last_became_active":"2024-04-02T04:49:28.765229+0000","last_became_peered":"2024-04-02T04:49:28.765229+0000","last_unstale":"2024-04-02T04:53:11.954300+0000","last_undegraded":"2024-04-02T04:53:11.954300+0000","last_fullsized":"2024-04-02T04:53:11.954300+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.673796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.673796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":408,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.936286+0000","last_change":"2024-04-02T04:49:47.441206+0000","last_active":"2024-04-02T04:53:11.936286+0000","last_peered":"2024-04-02T04:53:11.936286+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.441206+0000","last_became_peered":"2024-04-02T04:49:47.441206+0000","last_unstale":"2024-04-02T04:53:11.936286+0000","last_undegraded":"2024-04-02T04:53:11.936286+0000","last_fullsized":"2024-04-02T04:49:47.375248+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":51,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933594+0000","last_change":"2024-04-02T04:52:58.361166+0000","last_active":"2024-04-02T04:53:11.933594+0000","last_peered":"2024-04-02T04:53:11.933594+0000","last_clean":"2024-04-02T04:53:11.933594+0000","last_became_active":"2024-04-02T04:52:55.872480+0000","last_became_peered":"2024-04-02T04:52:55.872480+0000","last_unstale":"2024-04-02T04:53:11.933594+0000","last_undegraded":"2024-04-02T04:53:11.933594+0000","last_fullsized":"2024-04-02T04:53:11.933594+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.361050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.361050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:12.351179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":437,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.929375+0000","last_change":"2024-04-02T04:49:47.479424+0000","last_active":"2024-04-02T04:53:11.929375+0000","last_peered":"2024-04-02T04:53:11.929375+0000","last_clean":"2024-04-02T04:49:45.342017+0000","last_became_active":"2024-04-02T04:49:47.479424+0000","last_became_peered":"2024-04-02T04:49:47.479424+0000","last_unstale":"2024-04-02T04:53:11.929375+0000","last_undegraded":"2024-04-02T04:53:11.929375+0000","last_fullsized":"2024-04-02T04:49:47.413098+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.948802+0000","last_change":"2024-04-02T04:51:36.711304+0000","last_active":"2024-04-02T04:53:11.948802+0000","last_peered":"2024-04-02T04:53:11.948802+0000","last_clean":"2024-04-02T04:53:11.948802+0000","last_became_active":"2024-04-02T04:49:28.743589+0000","last_became_peered":"2024-04-02T04:49:28.743589+0000","last_unstale":"2024-04-02T04:53:11.948802+0000","last_undegraded":"2024-04-02T04:53:11.948802+0000","last_fullsized":"2024-04-02T04:53:11.948802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.711178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.711178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":439,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919223+0000","last_change":"2024-04-02T04:49:47.502106+0000","last_active":"2024-04-02T04:53:11.919223+0000","last_peered":"2024-04-02T04:53:11.919223+0000","last_clean":"2024-04-02T04:49:45.334274+0000","last_became_active":"2024-04-02T04:49:47.502106+0000","last_became_peered":"2024-04-02T04:49:47.502106+0000","last_unstale":"2024-04-02T04:53:11.919223+0000","last_undegraded":"2024-04-02T04:53:11.919223+0000","last_fullsized":"2024-04-02T04:49:47.399330+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918634+0000","last_change":"2024-04-02T04:52:49.762675+0000","last_active":"2024-04-02T04:53:11.918634+0000","last_peered":"2024-04-02T04:53:11.918634+0000","last_clean":"2024-04-02T04:53:11.918634+0000","last_became_active":"2024-04-02T04:49:29.805341+0000","last_became_peered":"2024-04-02T04:49:29.805341+0000","last_unstale":"2024-04-02T04:53:11.918634+0000","last_undegraded":"2024-04-02T04:53:11.918634+0000","last_fullsized":"2024-04-02T04:53:11.918634+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.762442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.762442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:05.791883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":408,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.920817+0000","last_change":"2024-04-02T04:49:47.482774+0000","last_active":"2024-04-02T04:53:11.920817+0000","last_peered":"2024-04-02T04:53:11.920817+0000","last_clean":"2024-04-02T04:49:25.981622+0000","last_became_active":"2024-04-02T04:49:47.482774+0000","last_became_peered":"2024-04-02T04:49:47.482774+0000","last_unstale":"2024-04-02T04:53:11.920817+0000","last_undegraded":"2024-04-02T04:53:11.920817+0000","last_fullsized":"2024-04-02T04:49:47.403371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.949424+0000","last_change":"2024-04-02T04:51:45.521690+0000","last_active":"2024-04-02T04:53:11.949424+0000","last_peered":"2024-04-02T04:53:11.949424+0000","last_clean":"2024-04-02T04:53:11.949424+0000","last_became_active":"2024-04-02T04:49:28.766979+0000","last_became_peered":"2024-04-02T04:49:28.766979+0000","last_unstale":"2024-04-02T04:53:11.949424+0000","last_undegraded":"2024-04-02T04:53:11.949424+0000","last_fullsized":"2024-04-02T04:53:11.949424+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:45.521589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:45.521589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916351+0000","last_change":"2024-04-02T04:52:42.318994+0000","last_active":"2024-04-02T04:53:11.916351+0000","last_peered":"2024-04-02T04:53:11.916351+0000","last_clean":"2024-04-02T04:53:11.916351+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:53:11.916351+0000","last_undegraded":"2024-04-02T04:53:11.916351+0000","last_fullsized":"2024-04-02T04:53:11.916351+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.318800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.318800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:44.825391+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941438+0000","last_change":"2024-04-02T04:52:52.417888+0000","last_active":"2024-04-02T04:53:11.941438+0000","last_peered":"2024-04-02T04:53:11.941438+0000","last_clean":"2024-04-02T04:53:11.941438+0000","last_became_active":"2024-04-02T04:49:28.753376+0000","last_became_peered":"2024-04-02T04:49:28.753376+0000","last_unstale":"2024-04-02T04:53:11.941438+0000","last_undegraded":"2024-04-02T04:53:11.941438+0000","last_fullsized":"2024-04-02T04:53:11.941438+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.417771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.417771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:11.133064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919681+0000","last_change":"2024-04-02T04:50:44.257998+0000","last_active":"2024-04-02T04:53:11.919681+0000","last_peered":"2024-04-02T04:53:11.919681+0000","last_clean":"2024-04-02T04:53:11.919681+0000","last_became_active":"2024-04-02T04:49:28.757583+0000","last_became_peered":"2024-04-02T04:49:28.757583+0000","last_unstale":"2024-04-02T04:53:11.919681+0000","last_undegraded":"2024-04-02T04:53:11.919681+0000","last_fullsized":"2024-04-02T04:53:11.919681+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.257747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.257747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918061+0000","last_change":"2024-04-02T04:51:49.984059+0000","last_active":"2024-04-02T04:53:11.918061+0000","last_peered":"2024-04-02T04:53:11.918061+0000","last_clean":"2024-04-02T04:53:11.918061+0000","last_became_active":"2024-04-02T04:49:28.726425+0000","last_became_peered":"2024-04-02T04:49:28.726425+0000","last_unstale":"2024-04-02T04:53:11.918061+0000","last_undegraded":"2024-04-02T04:53:11.918061+0000","last_fullsized":"2024-04-02T04:53:11.918061+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.983966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.983966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"139'5","reported_seq":453,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.937809+0000","last_change":"2024-04-02T04:50:26.846127+0000","last_active":"2024-04-02T04:53:11.937809+0000","last_peered":"2024-04-02T04:53:11.937809+0000","last_clean":"2024-04-02T04:53:11.937809+0000","last_became_active":"2024-04-02T04:49:26.170913+0000","last_became_peered":"2024-04-02T04:49:26.170913+0000","last_unstale":"2024-04-02T04:53:11.937809+0000","last_undegraded":"2024-04-02T04:53:11.937809+0000","last_fullsized":"2024-04-02T04:53:11.937809+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:50:26.845900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.845900+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.952268+0000","last_change":"2024-04-02T04:51:30.643523+0000","last_active":"2024-04-02T04:53:11.952268+0000","last_peered":"2024-04-02T04:53:11.952268+0000","last_clean":"2024-04-02T04:53:11.952268+0000","last_became_active":"2024-04-02T04:49:26.216834+0000","last_became_peered":"2024-04-02T04:49:26.216834+0000","last_unstale":"2024-04-02T04:53:11.952268+0000","last_undegraded":"2024-04-02T04:53:11.952268+0000","last_fullsized":"2024-04-02T04:53:11.952268+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.643306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.643306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":587,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.949125+0000","last_change":"2024-04-02T04:49:47.366762+0000","last_active":"2024-04-02T04:53:11.949125+0000","last_peered":"2024-04-02T04:53:11.949125+0000","last_clean":"2024-04-02T04:49:45.330332+0000","last_became_active":"2024-04-02T04:49:47.366762+0000","last_became_peered":"2024-04-02T04:49:47.366762+0000","last_unstale":"2024-04-02T04:53:11.949125+0000","last_undegraded":"2024-04-02T04:53:11.949125+0000","last_fullsized":"2024-04-02T04:49:47.338850+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.797658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.797658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":451,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.919918+0000","last_change":"2024-04-02T04:49:47.409183+0000","last_active":"2024-04-02T04:53:11.919918+0000","last_peered":"2024-04-02T04:53:11.919918+0000","last_clean":"2024-04-02T04:44:01.886062+0000","last_became_active":"2024-04-02T04:49:47.409183+0000","last_became_peered":"2024-04-02T04:49:47.409183+0000","last_unstale":"2024-04-02T04:53:11.919918+0000","last_undegraded":"2024-04-02T04:49:47.369263+0000","last_fullsized":"2024-04-02T04:49:47.369128+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":585,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.930320+0000","last_change":"2024-04-02T04:49:47.494397+0000","last_active":"2024-04-02T04:53:11.930320+0000","last_peered":"2024-04-02T04:53:11.930320+0000","last_clean":"2024-04-02T04:49:45.361282+0000","last_became_active":"2024-04-02T04:49:47.494397+0000","last_became_peered":"2024-04-02T04:49:47.494397+0000","last_unstale":"2024-04-02T04:53:11.930320+0000","last_undegraded":"2024-04-02T04:53:11.930320+0000","last_fullsized":"2024-04-02T04:49:47.425453+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:42.658688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.658688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.951081+0000","last_change":"2024-04-02T04:52:46.686069+0000","last_active":"2024-04-02T04:53:11.951081+0000","last_peered":"2024-04-02T04:53:11.951081+0000","last_clean":"2024-04-02T04:53:11.951081+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:53:11.951081+0000","last_undegraded":"2024-04-02T04:53:11.951081+0000","last_fullsized":"2024-04-02T04:53:11.951081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.685936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.685936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:02.921313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950491+0000","last_change":"2024-04-02T04:51:38.662185+0000","last_active":"2024-04-02T04:53:11.950491+0000","last_peered":"2024-04-02T04:53:11.950491+0000","last_clean":"2024-04-02T04:53:11.950491+0000","last_became_active":"2024-04-02T04:49:26.241757+0000","last_became_peered":"2024-04-02T04:49:26.241757+0000","last_unstale":"2024-04-02T04:53:11.950491+0000","last_undegraded":"2024-04-02T04:53:11.950491+0000","last_fullsized":"2024-04-02T04:53:11.950491+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:38.662076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.662076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":455,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930413+0000","last_change":"2024-04-02T04:52:57.641001+0000","last_active":"2024-04-02T04:53:11.930413+0000","last_peered":"2024-04-02T04:53:11.930413+0000","last_clean":"2024-04-02T04:53:11.930413+0000","last_became_active":"2024-04-02T04:49:26.296797+0000","last_became_peered":"2024-04-02T04:49:26.296797+0000","last_unstale":"2024-04-02T04:53:11.930413+0000","last_undegraded":"2024-04-02T04:53:11.930413+0000","last_fullsized":"2024-04-02T04:53:11.930413+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.640866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.640866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:12.765945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":601,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.924577+0000","last_change":"2024-04-02T04:52:55.335191+0000","last_active":"2024-04-02T04:53:11.924577+0000","last_peered":"2024-04-02T04:53:11.924577+0000","last_clean":"2024-04-02T04:53:11.924577+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:53:11.924577+0000","last_undegraded":"2024-04-02T04:53:11.924577+0000","last_fullsized":"2024-04-02T04:53:11.924577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:55.334980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.334980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:59.648881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"105'15","reported_seq":624,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.952245+0000","last_change":"2024-04-02T04:53:07.600488+0000","last_active":"2024-04-02T04:53:11.952245+0000","last_peered":"2024-04-02T04:53:11.952245+0000","last_clean":"2024-04-02T04:53:11.952245+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:53:11.952245+0000","last_undegraded":"2024-04-02T04:53:11.952245+0000","last_fullsized":"2024-04-02T04:53:11.952245+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:53:07.600375+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.600375+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:32.336251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":610,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.937075+0000","last_change":"2024-04-02T04:53:10.415912+0000","last_active":"2024-04-02T04:53:11.937075+0000","last_peered":"2024-04-02T04:53:11.937075+0000","last_clean":"2024-04-02T04:53:11.937075+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:53:11.937075+0000","last_undegraded":"2024-04-02T04:53:11.937075+0000","last_fullsized":"2024-04-02T04:53:11.937075+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.415766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:17.931537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":568,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919252+0000","last_change":"2024-04-02T04:49:47.516977+0000","last_active":"2024-04-02T04:53:11.919252+0000","last_peered":"2024-04-02T04:53:11.919252+0000","last_clean":"2024-04-02T04:49:45.355630+0000","last_became_active":"2024-04-02T04:49:47.516977+0000","last_became_peered":"2024-04-02T04:49:47.516977+0000","last_unstale":"2024-04-02T04:53:11.919252+0000","last_undegraded":"2024-04-02T04:53:11.919252+0000","last_fullsized":"2024-04-02T04:49:47.384135+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"139'5","reported_seq":455,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.923550+0000","last_change":"2024-04-02T04:51:27.143046+0000","last_active":"2024-04-02T04:53:11.923550+0000","last_peered":"2024-04-02T04:53:11.923550+0000","last_clean":"2024-04-02T04:53:11.923550+0000","last_became_active":"2024-04-02T04:49:25.139936+0000","last_became_peered":"2024-04-02T04:49:25.139936+0000","last_unstale":"2024-04-02T04:53:11.923550+0000","last_undegraded":"2024-04-02T04:53:11.923550+0000","last_fullsized":"2024-04-02T04:53:11.923550+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:27.142908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.142908+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":457,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926846+0000","last_change":"2024-04-02T04:53:10.335701+0000","last_active":"2024-04-02T04:53:11.926846+0000","last_peered":"2024-04-02T04:53:11.926846+0000","last_clean":"2024-04-02T04:53:11.926846+0000","last_became_active":"2024-04-02T04:49:25.143060+0000","last_became_peered":"2024-04-02T04:49:25.143060+0000","last_unstale":"2024-04-02T04:53:11.926846+0000","last_undegraded":"2024-04-02T04:53:11.926846+0000","last_fullsized":"2024-04-02T04:53:11.926846+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.335602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.335602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:28.770529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.936785+0000","last_change":"2024-04-02T04:49:47.472350+0000","last_active":"2024-04-02T04:53:11.936785+0000","last_peered":"2024-04-02T04:53:11.936785+0000","last_clean":"2024-04-02T04:49:45.385689+0000","last_became_active":"2024-04-02T04:49:47.472350+0000","last_became_peered":"2024-04-02T04:49:47.472350+0000","last_unstale":"2024-04-02T04:53:11.936785+0000","last_undegraded":"2024-04-02T04:53:11.936785+0000","last_fullsized":"2024-04-02T04:49:47.393316+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.719146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.719146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":454,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.951126+0000","last_change":"2024-04-02T04:49:47.381178+0000","last_active":"2024-04-02T04:53:11.951126+0000","last_peered":"2024-04-02T04:53:11.951126+0000","last_clean":"2024-04-02T04:44:01.856616+0000","last_became_active":"2024-04-02T04:49:47.381178+0000","last_became_peered":"2024-04-02T04:49:47.381178+0000","last_unstale":"2024-04-02T04:53:11.951126+0000","last_undegraded":"2024-04-02T04:49:47.359978+0000","last_fullsized":"2024-04-02T04:49:47.359801+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:00.356122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.356122+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":598,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.929760+0000","last_change":"2024-04-02T04:52:38.475693+0000","last_active":"2024-04-02T04:53:11.929760+0000","last_peered":"2024-04-02T04:53:11.929760+0000","last_clean":"2024-04-02T04:53:11.929760+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:53:11.929760+0000","last_undegraded":"2024-04-02T04:53:11.929760+0000","last_fullsized":"2024-04-02T04:53:11.929760+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.475462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.475462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.957384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":405,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.926802+0000","last_change":"2024-04-02T04:49:47.438287+0000","last_active":"2024-04-02T04:53:11.926802+0000","last_peered":"2024-04-02T04:53:11.926802+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.438287+0000","last_became_peered":"2024-04-02T04:49:47.438287+0000","last_unstale":"2024-04-02T04:53:11.926802+0000","last_undegraded":"2024-04-02T04:53:11.926802+0000","last_fullsized":"2024-04-02T04:49:47.386231+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.437949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.437949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941791+0000","last_change":"2024-04-02T04:52:34.410043+0000","last_active":"2024-04-02T04:53:11.941791+0000","last_peered":"2024-04-02T04:53:11.941791+0000","last_clean":"2024-04-02T04:53:11.941791+0000","last_became_active":"2024-04-02T04:49:26.271155+0000","last_became_peered":"2024-04-02T04:49:26.271155+0000","last_unstale":"2024-04-02T04:53:11.941791+0000","last_undegraded":"2024-04-02T04:53:11.941791+0000","last_fullsized":"2024-04-02T04:53:11.941791+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.409934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.409934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.101727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":447,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.929137+0000","last_change":"2024-04-02T04:51:27.143038+0000","last_active":"2024-04-02T04:53:11.929137+0000","last_peered":"2024-04-02T04:53:11.929137+0000","last_clean":"2024-04-02T04:53:11.929137+0000","last_became_active":"2024-04-02T04:49:26.265834+0000","last_became_peered":"2024-04-02T04:49:26.265834+0000","last_unstale":"2024-04-02T04:53:11.929137+0000","last_undegraded":"2024-04-02T04:53:11.929137+0000","last_fullsized":"2024-04-02T04:53:11.929137+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:27.142905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.142905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"85'26","reported_seq":641,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.938154+0000","last_change":"2024-04-02T04:51:43.069937+0000","last_active":"2024-04-02T04:53:11.938154+0000","last_peered":"2024-04-02T04:53:11.938154+0000","last_clean":"2024-04-02T04:53:11.938154+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:53:11.938154+0000","last_undegraded":"2024-04-02T04:53:11.938154+0000","last_fullsized":"2024-04-02T04:53:11.938154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:51:43.069694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.069694+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":407,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919961+0000","last_change":"2024-04-02T04:49:47.385876+0000","last_active":"2024-04-02T04:53:11.919961+0000","last_peered":"2024-04-02T04:53:11.919961+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.385876+0000","last_became_peered":"2024-04-02T04:49:47.385876+0000","last_unstale":"2024-04-02T04:53:11.919961+0000","last_undegraded":"2024-04-02T04:53:11.919961+0000","last_fullsized":"2024-04-02T04:49:47.345306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.439457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.439457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":708,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919980+0000","last_change":"2024-04-02T04:50:15.516760+0000","last_active":"2024-04-02T04:53:11.919980+0000","last_peered":"2024-04-02T04:53:11.919980+0000","last_clean":"2024-04-02T04:53:11.919980+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:53:11.919980+0000","last_undegraded":"2024-04-02T04:53:11.919980+0000","last_fullsized":"2024-04-02T04:53:11.919980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:50:15.516538+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.516538+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930878+0000","last_change":"2024-04-02T04:51:50.152380+0000","last_active":"2024-04-02T04:53:11.930878+0000","last_peered":"2024-04-02T04:53:11.930878+0000","last_clean":"2024-04-02T04:53:11.930878+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:53:11.930878+0000","last_undegraded":"2024-04-02T04:53:11.930878+0000","last_fullsized":"2024-04-02T04:53:11.930878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:50.152204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.152204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919289+0000","last_change":"2024-04-02T04:51:37.350257+0000","last_active":"2024-04-02T04:53:11.919289+0000","last_peered":"2024-04-02T04:53:11.919289+0000","last_clean":"2024-04-02T04:53:11.919289+0000","last_became_active":"2024-04-02T04:49:26.241469+0000","last_became_peered":"2024-04-02T04:49:26.241469+0000","last_unstale":"2024-04-02T04:53:11.919289+0000","last_undegraded":"2024-04-02T04:53:11.919289+0000","last_fullsized":"2024-04-02T04:53:11.919289+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:37.350152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:37.350152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.934499+0000","last_change":"2024-04-02T04:52:42.269689+0000","last_active":"2024-04-02T04:53:11.934499+0000","last_peered":"2024-04-02T04:53:11.934499+0000","last_clean":"2024-04-02T04:53:11.934499+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:53:11.934499+0000","last_undegraded":"2024-04-02T04:53:11.934499+0000","last_fullsized":"2024-04-02T04:53:11.934499+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.268948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.268948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:00.033647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":585,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917587+0000","last_change":"2024-04-02T04:50:43.513008+0000","last_active":"2024-04-02T04:53:11.917587+0000","last_peered":"2024-04-02T04:53:11.917587+0000","last_clean":"2024-04-02T04:53:11.917587+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:53:11.917587+0000","last_undegraded":"2024-04-02T04:53:11.917587+0000","last_fullsized":"2024-04-02T04:53:11.917587+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:43.512888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.512888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"112'20","reported_seq":632,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919878+0000","last_change":"2024-04-02T04:53:04.791017+0000","last_active":"2024-04-02T04:53:11.919878+0000","last_peered":"2024-04-02T04:53:11.919878+0000","last_clean":"2024-04-02T04:53:11.919878+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:53:11.919878+0000","last_undegraded":"2024-04-02T04:53:11.919878+0000","last_fullsized":"2024-04-02T04:53:11.919878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:53:04.790916+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.790916+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:10.920544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":597,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.934592+0000","last_change":"2024-04-02T04:49:47.504164+0000","last_active":"2024-04-02T04:53:11.934592+0000","last_peered":"2024-04-02T04:53:11.934592+0000","last_clean":"2024-04-02T04:49:45.357271+0000","last_became_active":"2024-04-02T04:49:47.504164+0000","last_became_peered":"2024-04-02T04:49:47.504164+0000","last_unstale":"2024-04-02T04:53:11.934592+0000","last_undegraded":"2024-04-02T04:53:11.934592+0000","last_fullsized":"2024-04-02T04:49:47.406612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.071225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":581,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922417+0000","last_change":"2024-04-02T04:50:18.501638+0000","last_active":"2024-04-02T04:53:11.922417+0000","last_peered":"2024-04-02T04:53:11.922417+0000","last_clean":"2024-04-02T04:53:11.922417+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:53:11.922417+0000","last_undegraded":"2024-04-02T04:53:11.922417+0000","last_fullsized":"2024-04-02T04:53:11.922417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.501528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.501528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"139'9","reported_seq":458,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922065+0000","last_change":"2024-04-02T04:51:29.324138+0000","last_active":"2024-04-02T04:53:11.922065+0000","last_peered":"2024-04-02T04:53:11.922065+0000","last_clean":"2024-04-02T04:53:11.922065+0000","last_became_active":"2024-04-02T04:49:26.487105+0000","last_became_peered":"2024-04-02T04:49:26.487105+0000","last_unstale":"2024-04-02T04:53:11.922065+0000","last_undegraded":"2024-04-02T04:53:11.922065+0000","last_fullsized":"2024-04-02T04:53:11.922065+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'9","last_scrub_stamp":"2024-04-02T04:51:29.324015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.324015+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":457,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.916464+0000","last_change":"2024-04-02T04:49:47.408871+0000","last_active":"2024-04-02T04:53:11.916464+0000","last_peered":"2024-04-02T04:53:11.916464+0000","last_clean":"2024-04-02T04:49:45.308505+0000","last_became_active":"2024-04-02T04:49:47.408871+0000","last_became_peered":"2024-04-02T04:49:47.408871+0000","last_unstale":"2024-04-02T04:53:11.916464+0000","last_undegraded":"2024-04-02T04:49:47.350015+0000","last_fullsized":"2024-04-02T04:49:47.349865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"103'36","reported_seq":652,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.917694+0000","last_change":"2024-04-02T04:49:47.511581+0000","last_active":"2024-04-02T04:53:11.917694+0000","last_peered":"2024-04-02T04:53:11.917694+0000","last_clean":"2024-04-02T04:49:45.312272+0000","last_became_active":"2024-04-02T04:49:47.511581+0000","last_became_peered":"2024-04-02T04:49:47.511581+0000","last_unstale":"2024-04-02T04:53:11.917694+0000","last_undegraded":"2024-04-02T04:49:47.361633+0000","last_fullsized":"2024-04-02T04:49:47.361362+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:49:24.269994+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.269994+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":608,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933385+0000","last_change":"2024-04-02T04:50:44.352622+0000","last_active":"2024-04-02T04:53:11.933385+0000","last_peered":"2024-04-02T04:53:11.933385+0000","last_clean":"2024-04-02T04:53:11.933385+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:53:11.933385+0000","last_undegraded":"2024-04-02T04:53:11.933385+0000","last_fullsized":"2024-04-02T04:53:11.933385+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.352505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.352505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":539,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.930997+0000","last_change":"2024-04-02T04:53:07.785865+0000","last_active":"2024-04-02T04:53:11.930997+0000","last_peered":"2024-04-02T04:53:11.930997+0000","last_clean":"2024-04-02T04:53:11.930997+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:53:11.930997+0000","last_undegraded":"2024-04-02T04:53:11.930997+0000","last_fullsized":"2024-04-02T04:53:11.930997+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.785753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.785753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.493653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":596,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.951383+0000","last_change":"2024-04-02T04:53:03.659862+0000","last_active":"2024-04-02T04:53:11.951383+0000","last_peered":"2024-04-02T04:53:11.951383+0000","last_clean":"2024-04-02T04:53:11.951383+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:53:11.951383+0000","last_undegraded":"2024-04-02T04:53:11.951383+0000","last_fullsized":"2024-04-02T04:53:11.951383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:03.659593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:53:03.659593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:30.238058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":409,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.941786+0000","last_change":"2024-04-02T04:49:47.396455+0000","last_active":"2024-04-02T04:53:11.941786+0000","last_peered":"2024-04-02T04:53:11.941786+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.396455+0000","last_became_peered":"2024-04-02T04:49:47.396455+0000","last_unstale":"2024-04-02T04:53:11.941786+0000","last_undegraded":"2024-04-02T04:53:11.941786+0000","last_fullsized":"2024-04-02T04:49:47.375223+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.937902+0000","last_change":"2024-04-02T04:52:42.441055+0000","last_active":"2024-04-02T04:53:11.937902+0000","last_peered":"2024-04-02T04:53:11.937902+0000","last_clean":"2024-04-02T04:53:11.937902+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:53:11.937902+0000","last_undegraded":"2024-04-02T04:53:11.937902+0000","last_fullsized":"2024-04-02T04:53:11.937902+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.440852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.440852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.682661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":631,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.917451+0000","last_change":"2024-04-02T04:49:47.453493+0000","last_active":"2024-04-02T04:53:11.917451+0000","last_peered":"2024-04-02T04:53:11.917451+0000","last_clean":"2024-04-02T04:49:45.308750+0000","last_became_active":"2024-04-02T04:49:47.453493+0000","last_became_peered":"2024-04-02T04:49:47.453493+0000","last_unstale":"2024-04-02T04:53:11.917451+0000","last_undegraded":"2024-04-02T04:49:47.351067+0000","last_fullsized":"2024-04-02T04:49:47.350938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-02T04:48:49.534769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.534769+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":406,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.918851+0000","last_change":"2024-04-02T04:49:47.521979+0000","last_active":"2024-04-02T04:53:11.918851+0000","last_peered":"2024-04-02T04:53:11.918851+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.521979+0000","last_became_peered":"2024-04-02T04:49:47.521979+0000","last_unstale":"2024-04-02T04:53:11.918851+0000","last_undegraded":"2024-04-02T04:53:11.918851+0000","last_fullsized":"2024-04-02T04:49:47.391120+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.344546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":707,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922317+0000","last_change":"2024-04-02T04:50:31.493819+0000","last_active":"2024-04-02T04:53:11.922317+0000","last_peered":"2024-04-02T04:53:11.922317+0000","last_clean":"2024-04-02T04:53:11.922317+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:53:11.922317+0000","last_undegraded":"2024-04-02T04:53:11.922317+0000","last_fullsized":"2024-04-02T04:53:11.922317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:31.493715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.493715+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":583,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.920851+0000","last_change":"2024-04-02T04:50:34.462138+0000","last_active":"2024-04-02T04:53:11.920851+0000","last_peered":"2024-04-02T04:53:11.920851+0000","last_clean":"2024-04-02T04:53:11.920851+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:53:11.920851+0000","last_undegraded":"2024-04-02T04:53:11.920851+0000","last_fullsized":"2024-04-02T04:53:11.920851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:34.462064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.462064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":457,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926285+0000","last_change":"2024-04-02T04:52:53.669076+0000","last_active":"2024-04-02T04:53:11.926285+0000","last_peered":"2024-04-02T04:53:11.926285+0000","last_clean":"2024-04-02T04:53:11.926285+0000","last_became_active":"2024-04-02T04:49:26.319266+0000","last_became_peered":"2024-04-02T04:49:26.319266+0000","last_unstale":"2024-04-02T04:53:11.926285+0000","last_undegraded":"2024-04-02T04:53:11.926285+0000","last_fullsized":"2024-04-02T04:53:11.926285+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:52:53.668865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.668865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:05.234817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.945630+0000","last_change":"2024-04-02T04:50:31.781484+0000","last_active":"2024-04-02T04:53:11.945630+0000","last_peered":"2024-04-02T04:53:11.945630+0000","last_clean":"2024-04-02T04:53:11.945630+0000","last_became_active":"2024-04-02T04:49:26.255429+0000","last_became_peered":"2024-04-02T04:49:26.255429+0000","last_unstale":"2024-04-02T04:53:11.945630+0000","last_undegraded":"2024-04-02T04:53:11.945630+0000","last_fullsized":"2024-04-02T04:53:11.945630+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.781273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.781273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":585,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917878+0000","last_change":"2024-04-02T04:51:40.385896+0000","last_active":"2024-04-02T04:53:11.917878+0000","last_peered":"2024-04-02T04:53:11.917878+0000","last_clean":"2024-04-02T04:53:11.917878+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:53:11.917878+0000","last_undegraded":"2024-04-02T04:53:11.917878+0000","last_fullsized":"2024-04-02T04:53:11.917878+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.385759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"188'18","reported_seq":621,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926851+0000","last_change":"2024-04-02T04:53:07.411481+0000","last_active":"2024-04-02T04:53:11.926851+0000","last_peered":"2024-04-02T04:53:11.926851+0000","last_clean":"2024-04-02T04:53:11.926851+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:53:11.926851+0000","last_undegraded":"2024-04-02T04:53:11.926851+0000","last_fullsized":"2024-04-02T04:53:11.926851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'18","last_scrub_stamp":"2024-04-02T04:53:07.411390+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.411390+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:19.500892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":596,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.945719+0000","last_change":"2024-04-02T04:49:47.470291+0000","last_active":"2024-04-02T04:53:11.945719+0000","last_peered":"2024-04-02T04:53:11.945719+0000","last_clean":"2024-04-02T04:49:45.387254+0000","last_became_active":"2024-04-02T04:49:47.470291+0000","last_became_peered":"2024-04-02T04:49:47.470291+0000","last_unstale":"2024-04-02T04:53:11.945719+0000","last_undegraded":"2024-04-02T04:53:11.945719+0000","last_fullsized":"2024-04-02T04:49:47.418421+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.690362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.690362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":1132,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918135+0000","last_change":"2024-04-02T04:50:39.632326+0000","last_active":"2024-04-02T04:53:11.918135+0000","last_peered":"2024-04-02T04:53:11.918135+0000","last_clean":"2024-04-02T04:53:11.918135+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:53:11.918135+0000","last_undegraded":"2024-04-02T04:53:11.918135+0000","last_fullsized":"2024-04-02T04:53:11.918135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:50:39.632171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.632171+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":535,"num_read_kb":530,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":406,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.938420+0000","last_change":"2024-04-02T04:49:47.470355+0000","last_active":"2024-04-02T04:53:11.938420+0000","last_peered":"2024-04-02T04:53:11.938420+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.470355+0000","last_became_peered":"2024-04-02T04:49:47.470355+0000","last_unstale":"2024-04-02T04:53:11.938420+0000","last_undegraded":"2024-04-02T04:53:11.938420+0000","last_fullsized":"2024-04-02T04:49:47.399308+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:49.224505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:47:49.224505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":457,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.949206+0000","last_change":"2024-04-02T04:53:04.638778+0000","last_active":"2024-04-02T04:53:11.949206+0000","last_peered":"2024-04-02T04:53:11.949206+0000","last_clean":"2024-04-02T04:53:11.949206+0000","last_became_active":"2024-04-02T04:49:26.343060+0000","last_became_peered":"2024-04-02T04:49:26.343060+0000","last_unstale":"2024-04-02T04:53:11.949206+0000","last_undegraded":"2024-04-02T04:53:11.949206+0000","last_fullsized":"2024-04-02T04:53:11.949206+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:04.638640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.638640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.668090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":441,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922225+0000","last_change":"2024-04-02T04:49:26.257833+0000","last_active":"2024-04-02T04:53:11.922225+0000","last_peered":"2024-04-02T04:53:11.922225+0000","last_clean":"2024-04-02T04:53:11.922225+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:53:11.922225+0000","last_undegraded":"2024-04-02T04:53:11.922225+0000","last_fullsized":"2024-04-02T04:53:11.922225+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"153'13","reported_seq":714,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.917904+0000","last_change":"2024-04-02T04:49:47.488932+0000","last_active":"2024-04-02T04:53:11.917904+0000","last_peered":"2024-04-02T04:53:11.917904+0000","last_clean":"2024-04-02T04:49:45.312370+0000","last_became_active":"2024-04-02T04:49:47.488932+0000","last_became_peered":"2024-04-02T04:49:47.488932+0000","last_unstale":"2024-04-02T04:53:11.917904+0000","last_undegraded":"2024-04-02T04:49:47.362506+0000","last_fullsized":"2024-04-02T04:49:47.362177+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'12","last_scrub_stamp":"2024-04-02T04:48:38.477825+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:48:38.477825+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":625,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.922349+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:53:11.922349+0000","last_peered":"2024-04-02T04:53:11.922349+0000","last_clean":"2024-04-02T04:49:45.321982+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:53:11.922349+0000","last_undegraded":"2024-04-02T04:49:47.406158+0000","last_fullsized":"2024-04-02T04:49:47.406050+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:49:07.374876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.374876+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":604,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.940162+0000","last_change":"2024-04-02T04:50:43.360677+0000","last_active":"2024-04-02T04:53:11.940162+0000","last_peered":"2024-04-02T04:53:11.940162+0000","last_clean":"2024-04-02T04:53:11.940162+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:53:11.940162+0000","last_undegraded":"2024-04-02T04:53:11.940162+0000","last_fullsized":"2024-04-02T04:53:11.940162+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:50:43.360457+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.360457+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":587,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.916708+0000","last_change":"2024-04-02T04:49:47.479633+0000","last_active":"2024-04-02T04:53:11.916708+0000","last_peered":"2024-04-02T04:53:11.916708+0000","last_clean":"2024-04-02T04:49:45.308628+0000","last_became_active":"2024-04-02T04:49:47.479633+0000","last_became_peered":"2024-04-02T04:49:47.479633+0000","last_unstale":"2024-04-02T04:53:11.916708+0000","last_undegraded":"2024-04-02T04:53:11.916708+0000","last_fullsized":"2024-04-02T04:49:47.350430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.555854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.555854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":581,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917034+0000","last_change":"2024-04-02T04:50:29.535635+0000","last_active":"2024-04-02T04:53:11.917034+0000","last_peered":"2024-04-02T04:53:11.917034+0000","last_clean":"2024-04-02T04:53:11.917034+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:53:11.917034+0000","last_undegraded":"2024-04-02T04:53:11.917034+0000","last_fullsized":"2024-04-02T04:53:11.917034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.535314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.535314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"139'10","reported_seq":465,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.928690+0000","last_change":"2024-04-02T04:52:55.613507+0000","last_active":"2024-04-02T04:53:11.928690+0000","last_peered":"2024-04-02T04:53:11.928690+0000","last_clean":"2024-04-02T04:53:11.928690+0000","last_became_active":"2024-04-02T04:49:26.341161+0000","last_became_peered":"2024-04-02T04:49:26.341161+0000","last_unstale":"2024-04-02T04:53:11.928690+0000","last_undegraded":"2024-04-02T04:53:11.928690+0000","last_fullsized":"2024-04-02T04:53:11.928690+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:52:55.613396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.613396+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:08.083376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941443+0000","last_change":"2024-04-02T04:50:29.806953+0000","last_active":"2024-04-02T04:53:11.941443+0000","last_peered":"2024-04-02T04:53:11.941443+0000","last_clean":"2024-04-02T04:53:11.941443+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:53:11.941443+0000","last_undegraded":"2024-04-02T04:53:11.941443+0000","last_fullsized":"2024-04-02T04:53:11.941443+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.806751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.806751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"153'13","reported_seq":712,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.923775+0000","last_change":"2024-04-02T04:49:47.472920+0000","last_active":"2024-04-02T04:53:11.923775+0000","last_peered":"2024-04-02T04:53:11.923775+0000","last_clean":"2024-04-02T04:49:45.374829+0000","last_became_active":"2024-04-02T04:49:47.472920+0000","last_became_peered":"2024-04-02T04:49:47.472920+0000","last_unstale":"2024-04-02T04:53:11.923775+0000","last_undegraded":"2024-04-02T04:49:47.384585+0000","last_fullsized":"2024-04-02T04:49:47.384414+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"121'47","reported_seq":791,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936230+0000","last_change":"2024-04-02T04:52:51.249602+0000","last_active":"2024-04-02T04:53:11.936230+0000","last_peered":"2024-04-02T04:53:11.936230+0000","last_clean":"2024-04-02T04:53:11.936230+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:53:11.936230+0000","last_undegraded":"2024-04-02T04:53:11.936230+0000","last_fullsized":"2024-04-02T04:53:11.936230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:52:51.249489+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.249489+0000","objects_scrubbed":11,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:19.032401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":406,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.924680+0000","last_change":"2024-04-02T04:49:47.431753+0000","last_active":"2024-04-02T04:53:11.924680+0000","last_peered":"2024-04-02T04:53:11.924680+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.431753+0000","last_became_peered":"2024-04-02T04:49:47.431753+0000","last_unstale":"2024-04-02T04:53:11.924680+0000","last_undegraded":"2024-04-02T04:53:11.924680+0000","last_fullsized":"2024-04-02T04:49:47.405769+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":602,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942422+0000","last_change":"2024-04-02T04:51:34.006250+0000","last_active":"2024-04-02T04:53:11.942422+0000","last_peered":"2024-04-02T04:53:11.942422+0000","last_clean":"2024-04-02T04:53:11.942422+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:53:11.942422+0000","last_undegraded":"2024-04-02T04:53:11.942422+0000","last_fullsized":"2024-04-02T04:53:11.942422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:51:34.006018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.006018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":582,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933484+0000","last_change":"2024-04-02T04:50:09.280418+0000","last_active":"2024-04-02T04:53:11.933484+0000","last_peered":"2024-04-02T04:53:11.933484+0000","last_clean":"2024-04-02T04:53:11.933484+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:53:11.933484+0000","last_undegraded":"2024-04-02T04:53:11.933484+0000","last_fullsized":"2024-04-02T04:53:11.933484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.280337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.280337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919044+0000","last_change":"2024-04-02T04:49:47.516900+0000","last_active":"2024-04-02T04:53:11.919044+0000","last_peered":"2024-04-02T04:53:11.919044+0000","last_clean":"2024-04-02T04:49:45.354881+0000","last_became_active":"2024-04-02T04:49:47.516900+0000","last_became_peered":"2024-04-02T04:49:47.516900+0000","last_unstale":"2024-04-02T04:53:11.919044+0000","last_undegraded":"2024-04-02T04:53:11.919044+0000","last_fullsized":"2024-04-02T04:49:47.391507+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":599,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.927202+0000","last_change":"2024-04-02T04:52:49.311146+0000","last_active":"2024-04-02T04:53:11.927202+0000","last_peered":"2024-04-02T04:53:11.927202+0000","last_clean":"2024-04-02T04:53:11.927202+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:53:11.927202+0000","last_undegraded":"2024-04-02T04:53:11.927202+0000","last_fullsized":"2024-04-02T04:53:11.927202+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.310943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.310943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:06.272620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":408,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.933164+0000","last_change":"2024-04-02T04:49:47.509591+0000","last_active":"2024-04-02T04:53:11.933164+0000","last_peered":"2024-04-02T04:53:11.933164+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.509591+0000","last_became_peered":"2024-04-02T04:49:47.509591+0000","last_unstale":"2024-04-02T04:53:11.933164+0000","last_undegraded":"2024-04-02T04:53:11.933164+0000","last_fullsized":"2024-04-02T04:49:47.387091+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.441320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.441320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'20","reported_seq":630,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918867+0000","last_change":"2024-04-02T04:52:33.272616+0000","last_active":"2024-04-02T04:53:11.918867+0000","last_peered":"2024-04-02T04:53:11.918867+0000","last_clean":"2024-04-02T04:53:11.918867+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:53:11.918867+0000","last_undegraded":"2024-04-02T04:53:11.918867+0000","last_fullsized":"2024-04-02T04:53:11.918867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:52:33.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.272448+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:39.428809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":590,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.935997+0000","last_change":"2024-04-02T04:50:26.342752+0000","last_active":"2024-04-02T04:53:11.935997+0000","last_peered":"2024-04-02T04:53:11.935997+0000","last_clean":"2024-04-02T04:53:11.935997+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:53:11.935997+0000","last_undegraded":"2024-04-02T04:53:11.935997+0000","last_fullsized":"2024-04-02T04:53:11.935997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.342429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.342429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":443,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.942360+0000","last_change":"2024-04-02T04:49:47.470287+0000","last_active":"2024-04-02T04:53:11.942360+0000","last_peered":"2024-04-02T04:53:11.942360+0000","last_clean":"2024-04-02T04:49:45.386144+0000","last_became_active":"2024-04-02T04:49:47.470287+0000","last_became_peered":"2024-04-02T04:49:47.470287+0000","last_unstale":"2024-04-02T04:53:11.942360+0000","last_undegraded":"2024-04-02T04:53:11.942360+0000","last_fullsized":"2024-04-02T04:49:47.418077+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936798+0000","last_change":"2024-04-02T04:50:24.860538+0000","last_active":"2024-04-02T04:53:11.936798+0000","last_peered":"2024-04-02T04:53:11.936798+0000","last_clean":"2024-04-02T04:53:11.936798+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:53:11.936798+0000","last_undegraded":"2024-04-02T04:53:11.936798+0000","last_fullsized":"2024-04-02T04:53:11.936798+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:24.860246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:24.860246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.945280+0000","last_change":"2024-04-02T04:49:47.413615+0000","last_active":"2024-04-02T04:53:11.945280+0000","last_peered":"2024-04-02T04:53:11.945280+0000","last_clean":"2024-04-02T04:49:45.390373+0000","last_became_active":"2024-04-02T04:49:47.413615+0000","last_became_peered":"2024-04-02T04:49:47.413615+0000","last_unstale":"2024-04-02T04:53:11.945280+0000","last_undegraded":"2024-04-02T04:53:11.945280+0000","last_fullsized":"2024-04-02T04:49:47.396592+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:45.569058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:45.569058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":717,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.918589+0000","last_change":"2024-04-02T04:49:47.451735+0000","last_active":"2024-04-02T04:53:11.918589+0000","last_peered":"2024-04-02T04:53:11.918589+0000","last_clean":"2024-04-02T04:44:53.853321+0000","last_became_active":"2024-04-02T04:49:47.451735+0000","last_became_peered":"2024-04-02T04:49:47.451735+0000","last_unstale":"2024-04-02T04:53:11.918589+0000","last_undegraded":"2024-04-02T04:49:47.390635+0000","last_fullsized":"2024-04-02T04:49:47.390485+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":587,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.950425+0000","last_change":"2024-04-02T04:49:47.435042+0000","last_active":"2024-04-02T04:53:11.950425+0000","last_peered":"2024-04-02T04:53:11.950425+0000","last_clean":"2024-04-02T04:49:45.341176+0000","last_became_active":"2024-04-02T04:49:47.435042+0000","last_became_peered":"2024-04-02T04:49:47.435042+0000","last_unstale":"2024-04-02T04:53:11.950425+0000","last_undegraded":"2024-04-02T04:53:11.950425+0000","last_fullsized":"2024-04-02T04:49:47.351952+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:39.903683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.903683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":618,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918841+0000","last_change":"2024-04-02T04:51:49.756835+0000","last_active":"2024-04-02T04:53:11.918841+0000","last_peered":"2024-04-02T04:53:11.918841+0000","last_clean":"2024-04-02T04:53:11.918841+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:53:11.918841+0000","last_undegraded":"2024-04-02T04:53:11.918841+0000","last_fullsized":"2024-04-02T04:53:11.918841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:49.756586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.756586+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.952881+0000","last_change":"2024-04-02T04:51:56.577347+0000","last_active":"2024-04-02T04:53:11.952881+0000","last_peered":"2024-04-02T04:53:11.952881+0000","last_clean":"2024-04-02T04:53:11.952881+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:53:11.952881+0000","last_undegraded":"2024-04-02T04:53:11.952881+0000","last_fullsized":"2024-04-02T04:53:11.952881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:56.577226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.577226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:14.507486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.935879+0000","last_change":"2024-04-02T04:51:55.332978+0000","last_active":"2024-04-02T04:53:11.935879+0000","last_peered":"2024-04-02T04:53:11.935879+0000","last_clean":"2024-04-02T04:53:11.935879+0000","last_became_active":"2024-04-02T04:49:26.290196+0000","last_became_peered":"2024-04-02T04:49:26.290196+0000","last_unstale":"2024-04-02T04:53:11.935879+0000","last_undegraded":"2024-04-02T04:53:11.935879+0000","last_fullsized":"2024-04-02T04:53:11.935879+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.332906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.332906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.949314+0000","last_change":"2024-04-02T04:49:47.382055+0000","last_active":"2024-04-02T04:53:11.949314+0000","last_peered":"2024-04-02T04:53:11.949314+0000","last_clean":"2024-04-02T04:49:45.332186+0000","last_became_active":"2024-04-02T04:49:47.382055+0000","last_became_peered":"2024-04-02T04:49:47.382055+0000","last_unstale":"2024-04-02T04:53:11.949314+0000","last_undegraded":"2024-04-02T04:53:11.949314+0000","last_fullsized":"2024-04-02T04:49:47.337410+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.940271+0000","last_change":"2024-04-02T04:52:59.385229+0000","last_active":"2024-04-02T04:53:11.940271+0000","last_peered":"2024-04-02T04:53:11.940271+0000","last_clean":"2024-04-02T04:53:11.940271+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:53:11.940271+0000","last_undegraded":"2024-04-02T04:53:11.940271+0000","last_fullsized":"2024-04-02T04:53:11.940271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.385111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.385111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:14.836761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":606,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926783+0000","last_change":"2024-04-02T04:52:46.569670+0000","last_active":"2024-04-02T04:53:11.926783+0000","last_peered":"2024-04-02T04:53:11.926783+0000","last_clean":"2024-04-02T04:53:11.926783+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:53:11.926783+0000","last_undegraded":"2024-04-02T04:53:11.926783+0000","last_fullsized":"2024-04-02T04:53:11.926783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.569560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.569560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:52.328763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":654,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933049+0000","last_change":"2024-04-02T04:52:49.180778+0000","last_active":"2024-04-02T04:53:11.933049+0000","last_peered":"2024-04-02T04:53:11.933049+0000","last_clean":"2024-04-02T04:53:11.933049+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:53:11.933049+0000","last_undegraded":"2024-04-02T04:53:11.933049+0000","last_fullsized":"2024-04-02T04:53:11.933049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:52:49.180521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.180521+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.223519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":406,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.941573+0000","last_change":"2024-04-02T04:49:47.492471+0000","last_active":"2024-04-02T04:53:11.941573+0000","last_peered":"2024-04-02T04:53:11.941573+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.492471+0000","last_became_peered":"2024-04-02T04:49:47.492471+0000","last_unstale":"2024-04-02T04:53:11.941573+0000","last_undegraded":"2024-04-02T04:53:11.941573+0000","last_fullsized":"2024-04-02T04:49:47.411395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:53.270717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:53.270717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":410,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.953505+0000","last_change":"2024-04-02T04:49:47.388497+0000","last_active":"2024-04-02T04:53:11.953505+0000","last_peered":"2024-04-02T04:53:11.953505+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.388497+0000","last_became_peered":"2024-04-02T04:49:47.388497+0000","last_unstale":"2024-04-02T04:53:11.953505+0000","last_undegraded":"2024-04-02T04:53:11.953505+0000","last_fullsized":"2024-04-02T04:49:47.356622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936604+0000","last_change":"2024-04-02T04:50:22.825218+0000","last_active":"2024-04-02T04:53:11.936604+0000","last_peered":"2024-04-02T04:53:11.936604+0000","last_clean":"2024-04-02T04:53:11.936604+0000","last_became_active":"2024-04-02T04:49:26.229137+0000","last_became_peered":"2024-04-02T04:49:26.229137+0000","last_unstale":"2024-04-02T04:53:11.936604+0000","last_undegraded":"2024-04-02T04:53:11.936604+0000","last_fullsized":"2024-04-02T04:53:11.936604+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:22.825000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:22.825000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936093+0000","last_change":"2024-04-02T04:51:34.344914+0000","last_active":"2024-04-02T04:53:11.936093+0000","last_peered":"2024-04-02T04:53:11.936093+0000","last_clean":"2024-04-02T04:53:11.936093+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:53:11.936093+0000","last_undegraded":"2024-04-02T04:53:11.936093+0000","last_fullsized":"2024-04-02T04:53:11.936093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:34.344554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.344554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'25","reported_seq":637,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922015+0000","last_change":"2024-04-02T04:50:12.546833+0000","last_active":"2024-04-02T04:53:11.922015+0000","last_peered":"2024-04-02T04:53:11.922015+0000","last_clean":"2024-04-02T04:53:11.922015+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:53:11.922015+0000","last_undegraded":"2024-04-02T04:53:11.922015+0000","last_fullsized":"2024-04-02T04:53:11.922015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:50:12.546683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.546683+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":600,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950858+0000","last_change":"2024-04-02T04:51:24.605907+0000","last_active":"2024-04-02T04:53:11.950858+0000","last_peered":"2024-04-02T04:53:11.950858+0000","last_clean":"2024-04-02T04:53:11.950858+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:53:11.950858+0000","last_undegraded":"2024-04-02T04:53:11.950858+0000","last_fullsized":"2024-04-02T04:53:11.950858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:24.605687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:51:24.605687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":580,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.922413+0000","last_change":"2024-04-02T04:49:47.486462+0000","last_active":"2024-04-02T04:53:11.922413+0000","last_peered":"2024-04-02T04:53:11.922413+0000","last_clean":"2024-04-02T04:49:45.345774+0000","last_became_active":"2024-04-02T04:49:47.486462+0000","last_became_peered":"2024-04-02T04:49:47.486462+0000","last_unstale":"2024-04-02T04:53:11.922413+0000","last_undegraded":"2024-04-02T04:49:47.409192+0000","last_fullsized":"2024-04-02T04:49:47.409080+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:48:49.483159+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.483159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"139'5","reported_seq":449,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.917383+0000","last_change":"2024-04-02T04:49:47.490156+0000","last_active":"2024-04-02T04:53:11.917383+0000","last_peered":"2024-04-02T04:53:11.917383+0000","last_clean":"2024-04-02T04:49:45.349908+0000","last_became_active":"2024-04-02T04:49:47.490156+0000","last_became_peered":"2024-04-02T04:49:47.490156+0000","last_unstale":"2024-04-02T04:53:11.917383+0000","last_undegraded":"2024-04-02T04:49:47.382039+0000","last_fullsized":"2024-04-02T04:49:47.381919+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"138'2","reported_seq":451,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950236+0000","last_change":"2024-04-02T04:51:33.690122+0000","last_active":"2024-04-02T04:53:11.950236+0000","last_peered":"2024-04-02T04:53:11.950236+0000","last_clean":"2024-04-02T04:53:11.950236+0000","last_became_active":"2024-04-02T04:49:26.270708+0000","last_became_peered":"2024-04-02T04:49:26.270708+0000","last_unstale":"2024-04-02T04:53:11.950236+0000","last_undegraded":"2024-04-02T04:53:11.950236+0000","last_fullsized":"2024-04-02T04:53:11.950236+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:51:33.689760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:33.689760+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":410,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.917776+0000","last_change":"2024-04-02T04:49:47.428471+0000","last_active":"2024-04-02T04:53:11.917776+0000","last_peered":"2024-04-02T04:53:11.917776+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.428471+0000","last_became_peered":"2024-04-02T04:49:47.428471+0000","last_unstale":"2024-04-02T04:53:11.917776+0000","last_undegraded":"2024-04-02T04:53:11.917776+0000","last_fullsized":"2024-04-02T04:49:47.368822+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.935020+0000","last_change":"2024-04-02T04:52:41.308406+0000","last_active":"2024-04-02T04:53:11.935020+0000","last_peered":"2024-04-02T04:53:11.935020+0000","last_clean":"2024-04-02T04:53:11.935020+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:53:11.935020+0000","last_undegraded":"2024-04-02T04:53:11.935020+0000","last_fullsized":"2024-04-02T04:53:11.935020+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.308301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.308301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:55.444595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":730,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950819+0000","last_change":"2024-04-02T04:51:35.726385+0000","last_active":"2024-04-02T04:53:11.950819+0000","last_peered":"2024-04-02T04:53:11.950819+0000","last_clean":"2024-04-02T04:53:11.950819+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:53:11.950819+0000","last_undegraded":"2024-04-02T04:53:11.950819+0000","last_fullsized":"2024-04-02T04:53:11.950819+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:35.725768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.725768+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":605,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.936122+0000","last_change":"2024-04-02T04:49:47.508665+0000","last_active":"2024-04-02T04:53:11.936122+0000","last_peered":"2024-04-02T04:53:11.936122+0000","last_clean":"2024-04-02T04:49:45.365160+0000","last_became_active":"2024-04-02T04:49:47.508665+0000","last_became_peered":"2024-04-02T04:49:47.508665+0000","last_unstale":"2024-04-02T04:53:11.936122+0000","last_undegraded":"2024-04-02T04:49:47.437341+0000","last_fullsized":"2024-04-02T04:49:47.437226+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:49:13.057821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:13.057821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":597,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918492+0000","last_change":"2024-04-02T04:53:11.307355+0000","last_active":"2024-04-02T04:53:11.918492+0000","last_peered":"2024-04-02T04:53:11.918492+0000","last_clean":"2024-04-02T04:53:11.918492+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:53:11.918492+0000","last_undegraded":"2024-04-02T04:53:11.918492+0000","last_fullsized":"2024-04-02T04:53:11.918492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.307248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.307248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:40.654703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"139'5","reported_seq":425,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.932690+0000","last_change":"2024-04-02T04:49:47.503964+0000","last_active":"2024-04-02T04:53:11.932690+0000","last_peered":"2024-04-02T04:53:11.932690+0000","last_clean":"2024-04-02T04:49:27.480374+0000","last_became_active":"2024-04-02T04:49:47.503964+0000","last_became_peered":"2024-04-02T04:49:47.503964+0000","last_unstale":"2024-04-02T04:53:11.932690+0000","last_undegraded":"2024-04-02T04:49:47.388434+0000","last_fullsized":"2024-04-02T04:49:47.383380+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942555+0000","last_change":"2024-04-02T04:52:57.364583+0000","last_active":"2024-04-02T04:53:11.942555+0000","last_peered":"2024-04-02T04:53:11.942555+0000","last_clean":"2024-04-02T04:53:11.942555+0000","last_became_active":"2024-04-02T04:49:26.198942+0000","last_became_peered":"2024-04-02T04:49:26.198942+0000","last_unstale":"2024-04-02T04:53:11.942555+0000","last_undegraded":"2024-04-02T04:53:11.942555+0000","last_fullsized":"2024-04-02T04:53:11.942555+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.364241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.364241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:25.621398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":716,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917334+0000","last_change":"2024-04-02T04:53:04.380937+0000","last_active":"2024-04-02T04:53:11.917334+0000","last_peered":"2024-04-02T04:53:11.917334+0000","last_clean":"2024-04-02T04:53:11.917334+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:53:11.917334+0000","last_undegraded":"2024-04-02T04:53:11.917334+0000","last_fullsized":"2024-04-02T04:53:11.917334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:53:04.380713+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.380713+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:27.532844+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":457,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.922301+0000","last_change":"2024-04-02T04:49:47.489637+0000","last_active":"2024-04-02T04:53:11.922301+0000","last_peered":"2024-04-02T04:53:11.922301+0000","last_clean":"2024-04-02T04:44:01.859319+0000","last_became_active":"2024-04-02T04:49:47.489637+0000","last_became_peered":"2024-04-02T04:49:47.489637+0000","last_unstale":"2024-04-02T04:53:11.922301+0000","last_undegraded":"2024-04-02T04:49:47.370487+0000","last_fullsized":"2024-04-02T04:49:47.370305+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":612,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.919774+0000","last_change":"2024-04-02T04:49:47.409343+0000","last_active":"2024-04-02T04:53:11.919774+0000","last_peered":"2024-04-02T04:53:11.919774+0000","last_clean":"2024-04-02T04:49:45.339425+0000","last_became_active":"2024-04-02T04:49:47.409343+0000","last_became_peered":"2024-04-02T04:49:47.409343+0000","last_unstale":"2024-04-02T04:53:11.919774+0000","last_undegraded":"2024-04-02T04:49:47.369170+0000","last_fullsized":"2024-04-02T04:49:47.369009+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:48:39.608255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:39.608255+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":578,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919978+0000","last_change":"2024-04-02T04:49:47.502658+0000","last_active":"2024-04-02T04:53:11.919978+0000","last_peered":"2024-04-02T04:53:11.919978+0000","last_clean":"2024-04-02T04:49:45.343348+0000","last_became_active":"2024-04-02T04:49:47.502658+0000","last_became_peered":"2024-04-02T04:49:47.502658+0000","last_unstale":"2024-04-02T04:53:11.919978+0000","last_undegraded":"2024-04-02T04:53:11.919978+0000","last_fullsized":"2024-04-02T04:49:47.417371+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:01.488397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:49:01.488397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":455,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926494+0000","last_change":"2024-04-02T04:53:09.820846+0000","last_active":"2024-04-02T04:53:11.926494+0000","last_peered":"2024-04-02T04:53:11.926494+0000","last_clean":"2024-04-02T04:53:11.926494+0000","last_became_active":"2024-04-02T04:49:26.292029+0000","last_became_peered":"2024-04-02T04:49:26.292029+0000","last_unstale":"2024-04-02T04:53:11.926494+0000","last_undegraded":"2024-04-02T04:53:11.926494+0000","last_fullsized":"2024-04-02T04:53:11.926494+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:09.820697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:09.820697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:20.156473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"85'17","reported_seq":457,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.919332+0000","last_change":"2024-04-02T04:49:47.409578+0000","last_active":"2024-04-02T04:53:11.919332+0000","last_peered":"2024-04-02T04:53:11.919332+0000","last_clean":"2024-04-02T04:44:01.867622+0000","last_became_active":"2024-04-02T04:49:47.409578+0000","last_became_peered":"2024-04-02T04:49:47.409578+0000","last_unstale":"2024-04-02T04:53:11.919332+0000","last_undegraded":"2024-04-02T04:49:47.352200+0000","last_fullsized":"2024-04-02T04:49:47.352060+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":606,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926659+0000","last_change":"2024-04-02T04:52:50.652084+0000","last_active":"2024-04-02T04:53:11.926659+0000","last_peered":"2024-04-02T04:53:11.926659+0000","last_clean":"2024-04-02T04:53:11.926659+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:53:11.926659+0000","last_undegraded":"2024-04-02T04:53:11.926659+0000","last_fullsized":"2024-04-02T04:53:11.926659+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:50.651865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:52:50.651865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:05.465692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":579,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.927370+0000","last_change":"2024-04-02T04:49:47.492319+0000","last_active":"2024-04-02T04:53:11.927370+0000","last_peered":"2024-04-02T04:53:11.927370+0000","last_clean":"2024-04-02T04:49:45.387118+0000","last_became_active":"2024-04-02T04:49:47.492319+0000","last_became_peered":"2024-04-02T04:49:47.492319+0000","last_unstale":"2024-04-02T04:53:11.927370+0000","last_undegraded":"2024-04-02T04:53:11.927370+0000","last_fullsized":"2024-04-02T04:49:47.417925+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.103147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":572,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.929947+0000","last_change":"2024-04-02T04:49:47.511784+0000","last_active":"2024-04-02T04:53:11.929947+0000","last_peered":"2024-04-02T04:53:11.929947+0000","last_clean":"2024-04-02T04:49:45.315951+0000","last_became_active":"2024-04-02T04:49:47.511784+0000","last_became_peered":"2024-04-02T04:49:47.511784+0000","last_unstale":"2024-04-02T04:53:11.929947+0000","last_undegraded":"2024-04-02T04:53:11.929947+0000","last_fullsized":"2024-04-02T04:49:47.381831+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:44.646709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:44.646709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"139'5","reported_seq":453,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942250+0000","last_change":"2024-04-02T04:53:01.369857+0000","last_active":"2024-04-02T04:53:11.942250+0000","last_peered":"2024-04-02T04:53:11.942250+0000","last_clean":"2024-04-02T04:53:11.942250+0000","last_became_active":"2024-04-02T04:49:26.178040+0000","last_became_peered":"2024-04-02T04:49:26.178040+0000","last_unstale":"2024-04-02T04:53:11.942250+0000","last_undegraded":"2024-04-02T04:53:11.942250+0000","last_fullsized":"2024-04-02T04:53:11.942250+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:53:01.369737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:01.369737+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:30.638792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933158+0000","last_change":"2024-04-02T04:51:29.319201+0000","last_active":"2024-04-02T04:53:11.933158+0000","last_peered":"2024-04-02T04:53:11.933158+0000","last_clean":"2024-04-02T04:53:11.933158+0000","last_became_active":"2024-04-02T04:49:26.258674+0000","last_became_peered":"2024-04-02T04:49:26.258674+0000","last_unstale":"2024-04-02T04:53:11.933158+0000","last_undegraded":"2024-04-02T04:53:11.933158+0000","last_fullsized":"2024-04-02T04:53:11.933158+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.318999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.318999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"85'4","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.938011+0000","last_change":"2024-04-02T04:50:17.746744+0000","last_active":"2024-04-02T04:53:11.938011+0000","last_peered":"2024-04-02T04:53:11.938011+0000","last_clean":"2024-04-02T04:53:11.938011+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:53:11.938011+0000","last_undegraded":"2024-04-02T04:53:11.938011+0000","last_fullsized":"2024-04-02T04:53:11.938011+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:50:17.746578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.746578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":600,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.938271+0000","last_change":"2024-04-02T04:51:40.072460+0000","last_active":"2024-04-02T04:53:11.938271+0000","last_peered":"2024-04-02T04:53:11.938271+0000","last_clean":"2024-04-02T04:53:11.938271+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:53:11.938271+0000","last_undegraded":"2024-04-02T04:53:11.938271+0000","last_fullsized":"2024-04-02T04:53:11.938271+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.072262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.072262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":587,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.922517+0000","last_change":"2024-04-02T04:49:47.477916+0000","last_active":"2024-04-02T04:53:11.922517+0000","last_peered":"2024-04-02T04:53:11.922517+0000","last_clean":"2024-04-02T04:49:45.322247+0000","last_became_active":"2024-04-02T04:49:47.477916+0000","last_became_peered":"2024-04-02T04:49:47.477916+0000","last_unstale":"2024-04-02T04:53:11.922517+0000","last_undegraded":"2024-04-02T04:53:11.922517+0000","last_fullsized":"2024-04-02T04:49:47.406503+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.485165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.485165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":406,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.945491+0000","last_change":"2024-04-02T04:49:47.412587+0000","last_active":"2024-04-02T04:53:11.945491+0000","last_peered":"2024-04-02T04:53:11.945491+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.412587+0000","last_became_peered":"2024-04-02T04:49:47.412587+0000","last_unstale":"2024-04-02T04:53:11.945491+0000","last_undegraded":"2024-04-02T04:53:11.945491+0000","last_fullsized":"2024-04-02T04:49:47.396252+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:59.304153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:59.304153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921992+0000","last_change":"2024-04-02T04:51:28.314780+0000","last_active":"2024-04-02T04:53:11.921992+0000","last_peered":"2024-04-02T04:53:11.921992+0000","last_clean":"2024-04-02T04:53:11.921992+0000","last_became_active":"2024-04-02T04:49:26.250745+0000","last_became_peered":"2024-04-02T04:49:26.250745+0000","last_unstale":"2024-04-02T04:53:11.921992+0000","last_undegraded":"2024-04-02T04:53:11.921992+0000","last_fullsized":"2024-04-02T04:53:11.921992+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:28.314678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:28.314678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936321+0000","last_change":"2024-04-02T04:51:39.049216+0000","last_active":"2024-04-02T04:53:11.936321+0000","last_peered":"2024-04-02T04:53:11.936321+0000","last_clean":"2024-04-02T04:53:11.936321+0000","last_became_active":"2024-04-02T04:49:26.229293+0000","last_became_peered":"2024-04-02T04:49:26.229293+0000","last_unstale":"2024-04-02T04:53:11.936321+0000","last_undegraded":"2024-04-02T04:53:11.936321+0000","last_fullsized":"2024-04-02T04:53:11.936321+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.048988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":445,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.921775+0000","last_change":"2024-04-02T04:49:47.452491+0000","last_active":"2024-04-02T04:53:11.921775+0000","last_peered":"2024-04-02T04:53:11.921775+0000","last_clean":"2024-04-02T04:44:01.831181+0000","last_became_active":"2024-04-02T04:49:47.452491+0000","last_became_peered":"2024-04-02T04:49:47.452491+0000","last_unstale":"2024-04-02T04:53:11.921775+0000","last_undegraded":"2024-04-02T04:49:47.355108+0000","last_fullsized":"2024-04-02T04:49:47.354938+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:00.302308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.302308+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":587,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.950758+0000","last_change":"2024-04-02T04:49:47.419557+0000","last_active":"2024-04-02T04:53:11.950758+0000","last_peered":"2024-04-02T04:53:11.950758+0000","last_clean":"2024-04-02T04:49:45.336531+0000","last_became_active":"2024-04-02T04:49:47.419557+0000","last_became_peered":"2024-04-02T04:49:47.419557+0000","last_unstale":"2024-04-02T04:53:11.950758+0000","last_undegraded":"2024-04-02T04:53:11.950758+0000","last_fullsized":"2024-04-02T04:49:47.341828+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:40.899356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:40.899356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:17.620408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"153'13","reported_seq":705,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.921043+0000","last_change":"2024-04-02T04:49:47.502490+0000","last_active":"2024-04-02T04:53:11.921043+0000","last_peered":"2024-04-02T04:53:11.921043+0000","last_clean":"2024-04-02T04:49:45.328987+0000","last_became_active":"2024-04-02T04:49:47.502490+0000","last_became_peered":"2024-04-02T04:49:47.502490+0000","last_unstale":"2024-04-02T04:53:11.921043+0000","last_undegraded":"2024-04-02T04:49:47.427380+0000","last_fullsized":"2024-04-02T04:49:47.427156+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.448123+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":575,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.927264+0000","last_change":"2024-04-02T04:49:47.491986+0000","last_active":"2024-04-02T04:53:11.927264+0000","last_peered":"2024-04-02T04:53:11.927264+0000","last_clean":"2024-04-02T04:49:45.387071+0000","last_became_active":"2024-04-02T04:49:47.491986+0000","last_became_peered":"2024-04-02T04:49:47.491986+0000","last_unstale":"2024-04-02T04:53:11.927264+0000","last_undegraded":"2024-04-02T04:53:11.927264+0000","last_fullsized":"2024-04-02T04:49:47.416002+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.986307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.986307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"139'7","reported_seq":424,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.930642+0000","last_change":"2024-04-02T04:49:47.507515+0000","last_active":"2024-04-02T04:53:11.930642+0000","last_peered":"2024-04-02T04:53:11.930642+0000","last_clean":"2024-04-02T04:49:27.511662+0000","last_became_active":"2024-04-02T04:49:47.507515+0000","last_became_peered":"2024-04-02T04:49:47.507515+0000","last_unstale":"2024-04-02T04:53:11.930642+0000","last_undegraded":"2024-04-02T04:49:47.419015+0000","last_fullsized":"2024-04-02T04:49:47.418904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":446,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919628+0000","last_change":"2024-04-02T04:49:47.480176+0000","last_active":"2024-04-02T04:53:11.919628+0000","last_peered":"2024-04-02T04:53:11.919628+0000","last_clean":"2024-04-02T04:49:45.335187+0000","last_became_active":"2024-04-02T04:49:47.480176+0000","last_became_peered":"2024-04-02T04:49:47.480176+0000","last_unstale":"2024-04-02T04:53:11.919628+0000","last_undegraded":"2024-04-02T04:53:11.919628+0000","last_fullsized":"2024-04-02T04:49:47.401165+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":607,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.921302+0000","last_change":"2024-04-02T04:49:47.492714+0000","last_active":"2024-04-02T04:53:11.921302+0000","last_peered":"2024-04-02T04:53:11.921302+0000","last_clean":"2024-04-02T04:49:45.370028+0000","last_became_active":"2024-04-02T04:49:47.492714+0000","last_became_peered":"2024-04-02T04:49:47.492714+0000","last_unstale":"2024-04-02T04:53:11.921302+0000","last_undegraded":"2024-04-02T04:49:47.390470+0000","last_fullsized":"2024-04-02T04:49:47.390328+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:49:14.989379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.989379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":600,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.923665+0000","last_change":"2024-04-02T04:53:05.331108+0000","last_active":"2024-04-02T04:53:11.923665+0000","last_peered":"2024-04-02T04:53:11.923665+0000","last_clean":"2024-04-02T04:53:11.923665+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:53:11.923665+0000","last_undegraded":"2024-04-02T04:53:11.923665+0000","last_fullsized":"2024-04-02T04:53:11.923665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.331020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.331020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:21.500198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"153'12","reported_seq":704,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.951186+0000","last_change":"2024-04-02T04:49:47.418645+0000","last_active":"2024-04-02T04:53:11.951186+0000","last_peered":"2024-04-02T04:53:11.951186+0000","last_clean":"2024-04-02T04:49:45.363313+0000","last_became_active":"2024-04-02T04:49:47.418645+0000","last_became_peered":"2024-04-02T04:49:47.418645+0000","last_unstale":"2024-04-02T04:53:11.951186+0000","last_undegraded":"2024-04-02T04:49:47.360244+0000","last_fullsized":"2024-04-02T04:49:47.360013+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:48:41.932321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:48:41.932321+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":580,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922039+0000","last_change":"2024-04-02T04:50:42.526150+0000","last_active":"2024-04-02T04:53:11.922039+0000","last_peered":"2024-04-02T04:53:11.922039+0000","last_clean":"2024-04-02T04:53:11.922039+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:53:11.922039+0000","last_undegraded":"2024-04-02T04:53:11.922039+0000","last_fullsized":"2024-04-02T04:53:11.922039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:42.525995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:42.525995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941625+0000","last_change":"2024-04-02T04:52:43.424718+0000","last_active":"2024-04-02T04:53:11.941625+0000","last_peered":"2024-04-02T04:53:11.941625+0000","last_clean":"2024-04-02T04:53:11.941625+0000","last_became_active":"2024-04-02T04:49:26.280984+0000","last_became_peered":"2024-04-02T04:49:26.280984+0000","last_unstale":"2024-04-02T04:53:11.941625+0000","last_undegraded":"2024-04-02T04:53:11.941625+0000","last_fullsized":"2024-04-02T04:53:11.941625+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:43.424601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.424601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:54.827733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.941918+0000","last_change":"2024-04-02T04:50:25.822947+0000","last_active":"2024-04-02T04:53:11.941918+0000","last_peered":"2024-04-02T04:53:11.941918+0000","last_clean":"2024-04-02T04:53:11.941918+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:53:11.941918+0000","last_undegraded":"2024-04-02T04:53:11.941918+0000","last_fullsized":"2024-04-02T04:53:11.941918+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:25.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":455,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.921566+0000","last_change":"2024-04-02T04:49:47.479230+0000","last_active":"2024-04-02T04:53:11.921566+0000","last_peered":"2024-04-02T04:53:11.921566+0000","last_clean":"2024-04-02T04:44:01.845646+0000","last_became_active":"2024-04-02T04:49:47.479230+0000","last_became_peered":"2024-04-02T04:49:47.479230+0000","last_unstale":"2024-04-02T04:53:11.921566+0000","last_undegraded":"2024-04-02T04:49:47.401783+0000","last_fullsized":"2024-04-02T04:49:47.401678+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:49:23.491068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:23.491068+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":592,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.942009+0000","last_change":"2024-04-02T04:50:21.847768+0000","last_active":"2024-04-02T04:53:11.942009+0000","last_peered":"2024-04-02T04:53:11.942009+0000","last_clean":"2024-04-02T04:53:11.942009+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:53:11.942009+0000","last_undegraded":"2024-04-02T04:53:11.942009+0000","last_fullsized":"2024-04-02T04:53:11.942009+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.847551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.847551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":593,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.934407+0000","last_change":"2024-04-02T04:52:54.293779+0000","last_active":"2024-04-02T04:53:11.934407+0000","last_peered":"2024-04-02T04:53:11.934407+0000","last_clean":"2024-04-02T04:53:11.934407+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:53:11.934407+0000","last_undegraded":"2024-04-02T04:53:11.934407+0000","last_fullsized":"2024-04-02T04:53:11.934407+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:54.293646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:54.293646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:07.211616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.918397+0000","last_change":"2024-04-02T04:51:53.965925+0000","last_active":"2024-04-02T04:53:11.918397+0000","last_peered":"2024-04-02T04:53:11.918397+0000","last_clean":"2024-04-02T04:53:11.918397+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:53:11.918397+0000","last_undegraded":"2024-04-02T04:53:11.918397+0000","last_fullsized":"2024-04-02T04:53:11.918397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.965811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.965811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"139'10","reported_seq":459,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919600+0000","last_change":"2024-04-02T04:51:30.300205+0000","last_active":"2024-04-02T04:53:11.919600+0000","last_peered":"2024-04-02T04:53:11.919600+0000","last_clean":"2024-04-02T04:53:11.919600+0000","last_became_active":"2024-04-02T04:49:26.260148+0000","last_became_peered":"2024-04-02T04:49:26.260148+0000","last_unstale":"2024-04-02T04:53:11.919600+0000","last_undegraded":"2024-04-02T04:53:11.919600+0000","last_fullsized":"2024-04-02T04:53:11.919600+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:51:30.299917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.299917+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917788+0000","last_change":"2024-04-02T04:51:26.346563+0000","last_active":"2024-04-02T04:53:11.917788+0000","last_peered":"2024-04-02T04:53:11.917788+0000","last_clean":"2024-04-02T04:53:11.917788+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:53:11.917788+0000","last_undegraded":"2024-04-02T04:53:11.917788+0000","last_fullsized":"2024-04-02T04:53:11.917788+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.346350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.346350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"85'22","reported_seq":622,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.917434+0000","last_change":"2024-04-02T04:49:47.392866+0000","last_active":"2024-04-02T04:53:11.917434+0000","last_peered":"2024-04-02T04:53:11.917434+0000","last_clean":"2024-04-02T04:49:45.343613+0000","last_became_active":"2024-04-02T04:49:47.392866+0000","last_became_peered":"2024-04-02T04:49:47.392866+0000","last_unstale":"2024-04-02T04:53:11.917434+0000","last_undegraded":"2024-04-02T04:49:47.367503+0000","last_fullsized":"2024-04-02T04:49:47.367381+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:48:00.691378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:00.691378+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":601,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.936973+0000","last_change":"2024-04-02T04:53:05.438467+0000","last_active":"2024-04-02T04:53:11.936973+0000","last_peered":"2024-04-02T04:53:11.936973+0000","last_clean":"2024-04-02T04:53:11.936973+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:53:11.936973+0000","last_undegraded":"2024-04-02T04:53:11.936973+0000","last_fullsized":"2024-04-02T04:53:11.936973+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.438193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.438193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:25.272805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":592,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919860+0000","last_change":"2024-04-02T04:52:40.351590+0000","last_active":"2024-04-02T04:53:11.919860+0000","last_peered":"2024-04-02T04:53:11.919860+0000","last_clean":"2024-04-02T04:53:11.919860+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:53:11.919860+0000","last_undegraded":"2024-04-02T04:53:11.919860+0000","last_fullsized":"2024-04-02T04:53:11.919860+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.351381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.351381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:41.542456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":582,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917099+0000","last_change":"2024-04-02T04:50:32.383573+0000","last_active":"2024-04-02T04:53:11.917099+0000","last_peered":"2024-04-02T04:53:11.917099+0000","last_clean":"2024-04-02T04:53:11.917099+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:53:11.917099+0000","last_undegraded":"2024-04-02T04:53:11.917099+0000","last_fullsized":"2024-04-02T04:53:11.917099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.383421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"262'6","reported_seq":456,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.934919+0000","last_change":"2024-04-02T04:51:42.313312+0000","last_active":"2024-04-02T04:53:11.934919+0000","last_peered":"2024-04-02T04:53:11.934919+0000","last_clean":"2024-04-02T04:53:11.934919+0000","last_became_active":"2024-04-02T04:49:26.275418+0000","last_became_peered":"2024-04-02T04:49:26.275418+0000","last_unstale":"2024-04-02T04:53:11.934919+0000","last_undegraded":"2024-04-02T04:53:11.934919+0000","last_fullsized":"2024-04-02T04:53:11.934919+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'6","last_scrub_stamp":"2024-04-02T04:51:42.313232+0000","last_deep_scrub":"262'6","last_deep_scrub_stamp":"2024-04-02T04:51:42.313232+0000","last_clean_scrub_stamp":"2024-04-02T04:51:42.313232+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":457,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926753+0000","last_change":"2024-04-02T04:53:00.389103+0000","last_active":"2024-04-02T04:53:11.926753+0000","last_peered":"2024-04-02T04:53:11.926753+0000","last_clean":"2024-04-02T04:53:11.926753+0000","last_became_active":"2024-04-02T04:49:25.080486+0000","last_became_peered":"2024-04-02T04:49:25.080486+0000","last_unstale":"2024-04-02T04:53:11.926753+0000","last_undegraded":"2024-04-02T04:53:11.926753+0000","last_fullsized":"2024-04-02T04:53:11.926753+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.388684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.388684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:22.291684+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":609,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.926902+0000","last_change":"2024-04-02T04:50:25.999986+0000","last_active":"2024-04-02T04:53:11.926902+0000","last_peered":"2024-04-02T04:53:11.926902+0000","last_clean":"2024-04-02T04:53:11.926902+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:53:11.926902+0000","last_undegraded":"2024-04-02T04:53:11.926902+0000","last_fullsized":"2024-04-02T04:53:11.926902+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:50:25.999894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.999894+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":607,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:12.625101+0000","last_change":"2024-04-02T04:53:12.625161+0000","last_active":"2024-04-02T04:53:12.625101+0000","last_peered":"2024-04-02T04:53:12.625101+0000","last_clean":"2024-04-02T04:53:12.625101+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:53:12.625101+0000","last_undegraded":"2024-04-02T04:53:12.625101+0000","last_fullsized":"2024-04-02T04:53:12.625101+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:12.624937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:53:12.624937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:13.488562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917613+0000","last_change":"2024-04-02T04:51:54.722249+0000","last_active":"2024-04-02T04:53:11.917613+0000","last_peered":"2024-04-02T04:53:11.917613+0000","last_clean":"2024-04-02T04:53:11.917613+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:53:11.917613+0000","last_undegraded":"2024-04-02T04:53:11.917613+0000","last_fullsized":"2024-04-02T04:53:11.917613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:54.722054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:54.722054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":405,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.928902+0000","last_change":"2024-04-02T04:49:47.512747+0000","last_active":"2024-04-02T04:53:11.928902+0000","last_peered":"2024-04-02T04:53:11.928902+0000","last_clean":"2024-04-02T04:43:59.289274+0000","last_became_active":"2024-04-02T04:49:47.512747+0000","last_became_peered":"2024-04-02T04:49:47.512747+0000","last_unstale":"2024-04-02T04:53:11.928902+0000","last_undegraded":"2024-04-02T04:53:11.928902+0000","last_fullsized":"2024-04-02T04:49:47.402486+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.372299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.372299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:12.782454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":447,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.918198+0000","last_change":"2024-04-02T04:49:47.444258+0000","last_active":"2024-04-02T04:53:11.918198+0000","last_peered":"2024-04-02T04:53:11.918198+0000","last_clean":"2024-04-02T04:49:45.328612+0000","last_became_active":"2024-04-02T04:49:47.444258+0000","last_became_peered":"2024-04-02T04:49:47.444258+0000","last_unstale":"2024-04-02T04:53:11.918198+0000","last_undegraded":"2024-04-02T04:53:11.918198+0000","last_fullsized":"2024-04-02T04:49:47.397778+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":411,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.940389+0000","last_change":"2024-04-02T04:49:47.517978+0000","last_active":"2024-04-02T04:53:11.940389+0000","last_peered":"2024-04-02T04:53:11.940389+0000","last_clean":"2024-04-02T04:49:15.972156+0000","last_became_active":"2024-04-02T04:49:47.517978+0000","last_became_peered":"2024-04-02T04:49:47.517978+0000","last_unstale":"2024-04-02T04:53:11.940389+0000","last_undegraded":"2024-04-02T04:53:11.940389+0000","last_fullsized":"2024-04-02T04:49:47.456750+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":630,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.953203+0000","last_change":"2024-04-02T04:49:47.430162+0000","last_active":"2024-04-02T04:53:11.953203+0000","last_peered":"2024-04-02T04:53:11.953203+0000","last_clean":"2024-04-02T04:49:45.364593+0000","last_became_active":"2024-04-02T04:49:47.430162+0000","last_became_peered":"2024-04-02T04:49:47.430162+0000","last_unstale":"2024-04-02T04:53:11.953203+0000","last_undegraded":"2024-04-02T04:49:47.355994+0000","last_fullsized":"2024-04-02T04:49:47.355865+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:49:03.788837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.788837+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":600,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.945536+0000","last_change":"2024-04-02T04:50:08.818906+0000","last_active":"2024-04-02T04:53:11.945536+0000","last_peered":"2024-04-02T04:53:11.945536+0000","last_clean":"2024-04-02T04:53:11.945536+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:53:11.945536+0000","last_undegraded":"2024-04-02T04:53:11.945536+0000","last_fullsized":"2024-04-02T04:53:11.945536+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:08.818819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:50:08.818819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":581,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.917422+0000","last_change":"2024-04-02T04:49:47.443764+0000","last_active":"2024-04-02T04:53:11.917422+0000","last_peered":"2024-04-02T04:53:11.917422+0000","last_clean":"2024-04-02T04:49:45.325466+0000","last_became_active":"2024-04-02T04:49:47.443764+0000","last_became_peered":"2024-04-02T04:49:47.443764+0000","last_unstale":"2024-04-02T04:53:11.917422+0000","last_undegraded":"2024-04-02T04:53:11.917422+0000","last_fullsized":"2024-04-02T04:49:47.396652+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.351209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":582,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.920002+0000","last_change":"2024-04-02T04:52:47.741959+0000","last_active":"2024-04-02T04:53:11.920002+0000","last_peered":"2024-04-02T04:53:11.920002+0000","last_clean":"2024-04-02T04:53:11.920002+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:53:11.920002+0000","last_undegraded":"2024-04-02T04:53:11.920002+0000","last_fullsized":"2024-04-02T04:53:11.920002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.741751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.741751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:58.511816+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"138'2","reported_seq":418,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.917669+0000","last_change":"2024-04-02T04:49:47.436385+0000","last_active":"2024-04-02T04:53:11.917669+0000","last_peered":"2024-04-02T04:53:11.917669+0000","last_clean":"2024-04-02T04:49:26.658740+0000","last_became_active":"2024-04-02T04:49:47.436385+0000","last_became_peered":"2024-04-02T04:49:47.436385+0000","last_unstale":"2024-04-02T04:53:11.917669+0000","last_undegraded":"2024-04-02T04:53:11.917669+0000","last_fullsized":"2024-04-02T04:49:47.371603+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.917830+0000","last_change":"2024-04-02T04:49:47.393078+0000","last_active":"2024-04-02T04:53:11.917830+0000","last_peered":"2024-04-02T04:53:11.917830+0000","last_clean":"2024-04-02T04:49:45.306613+0000","last_became_active":"2024-04-02T04:49:47.393078+0000","last_became_peered":"2024-04-02T04:49:47.393078+0000","last_unstale":"2024-04-02T04:53:11.917830+0000","last_undegraded":"2024-04-02T04:53:11.917830+0000","last_fullsized":"2024-04-02T04:49:47.373401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"133'50","reported_seq":673,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933258+0000","last_change":"2024-04-02T04:50:33.298622+0000","last_active":"2024-04-02T04:53:11.933258+0000","last_peered":"2024-04-02T04:53:11.933258+0000","last_clean":"2024-04-02T04:53:11.933258+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:53:11.933258+0000","last_undegraded":"2024-04-02T04:53:11.933258+0000","last_fullsized":"2024-04-02T04:53:11.933258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:50:33.298479+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:50:33.298479+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.948975+0000","last_change":"2024-04-02T04:49:47.388339+0000","last_active":"2024-04-02T04:53:11.948975+0000","last_peered":"2024-04-02T04:53:11.948975+0000","last_clean":"2024-04-02T04:49:45.330232+0000","last_became_active":"2024-04-02T04:49:47.388339+0000","last_became_peered":"2024-04-02T04:49:47.388339+0000","last_unstale":"2024-04-02T04:53:11.948975+0000","last_undegraded":"2024-04-02T04:53:11.948975+0000","last_fullsized":"2024-04-02T04:49:47.337757+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:51.870038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:51.870038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.926951+0000","last_change":"2024-04-02T04:49:47.491600+0000","last_active":"2024-04-02T04:53:11.926951+0000","last_peered":"2024-04-02T04:53:11.926951+0000","last_clean":"2024-04-02T04:49:45.386942+0000","last_became_active":"2024-04-02T04:49:47.491600+0000","last_became_peered":"2024-04-02T04:49:47.491600+0000","last_unstale":"2024-04-02T04:53:11.926951+0000","last_undegraded":"2024-04-02T04:53:11.926951+0000","last_fullsized":"2024-04-02T04:49:47.415538+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":418,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.951039+0000","last_change":"2024-04-02T04:49:47.434796+0000","last_active":"2024-04-02T04:53:11.951039+0000","last_peered":"2024-04-02T04:53:11.951039+0000","last_clean":"2024-04-02T04:44:01.344239+0000","last_became_active":"2024-04-02T04:49:47.434796+0000","last_became_peered":"2024-04-02T04:49:47.434796+0000","last_unstale":"2024-04-02T04:53:11.951039+0000","last_undegraded":"2024-04-02T04:49:47.346909+0000","last_fullsized":"2024-04-02T04:49:47.346781+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:48:42.394228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:42.394228+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.17","version":"139'5","reported_seq":454,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.952597+0000","last_change":"2024-04-02T04:51:25.643429+0000","last_active":"2024-04-02T04:53:11.952597+0000","last_peered":"2024-04-02T04:53:11.952597+0000","last_clean":"2024-04-02T04:53:11.952597+0000","last_became_active":"2024-04-02T04:49:26.272870+0000","last_became_peered":"2024-04-02T04:49:26.272870+0000","last_unstale":"2024-04-02T04:53:11.952597+0000","last_undegraded":"2024-04-02T04:53:11.952597+0000","last_fullsized":"2024-04-02T04:53:11.952597+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:25.643193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.643193+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":444,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.949215+0000","last_change":"2024-04-02T04:49:47.478289+0000","last_active":"2024-04-02T04:53:11.949215+0000","last_peered":"2024-04-02T04:53:11.949215+0000","last_clean":"2024-04-02T04:49:45.332087+0000","last_became_active":"2024-04-02T04:49:47.478289+0000","last_became_peered":"2024-04-02T04:49:47.478289+0000","last_unstale":"2024-04-02T04:53:11.949215+0000","last_undegraded":"2024-04-02T04:53:11.949215+0000","last_fullsized":"2024-04-02T04:49:47.337387+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"85'15","reported_seq":626,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917509+0000","last_change":"2024-04-02T04:52:58.806428+0000","last_active":"2024-04-02T04:53:11.917509+0000","last_peered":"2024-04-02T04:53:11.917509+0000","last_clean":"2024-04-02T04:53:11.917509+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:53:11.917509+0000","last_undegraded":"2024-04-02T04:53:11.917509+0000","last_fullsized":"2024-04-02T04:53:11.917509+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:58.806324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.806324+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:10.998925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":596,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917621+0000","last_change":"2024-04-02T04:50:28.247971+0000","last_active":"2024-04-02T04:53:11.917621+0000","last_peered":"2024-04-02T04:53:11.917621+0000","last_clean":"2024-04-02T04:53:11.917621+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:53:11.917621+0000","last_undegraded":"2024-04-02T04:53:11.917621+0000","last_fullsized":"2024-04-02T04:53:11.917621+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:50:28.247657+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.247657+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":710,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.945722+0000","last_change":"2024-04-02T04:50:12.840880+0000","last_active":"2024-04-02T04:53:11.945722+0000","last_peered":"2024-04-02T04:53:11.945722+0000","last_clean":"2024-04-02T04:53:11.945722+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:53:11.945722+0000","last_undegraded":"2024-04-02T04:53:11.945722+0000","last_fullsized":"2024-04-02T04:53:11.945722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:12.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.840785+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":581,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.937190+0000","last_change":"2024-04-02T04:50:15.299404+0000","last_active":"2024-04-02T04:53:11.937190+0000","last_peered":"2024-04-02T04:53:11.937190+0000","last_clean":"2024-04-02T04:53:11.937190+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:53:11.937190+0000","last_undegraded":"2024-04-02T04:53:11.937190+0000","last_fullsized":"2024-04-02T04:53:11.937190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.299316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.299316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"139'5","reported_seq":457,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922672+0000","last_change":"2024-04-02T04:51:32.581419+0000","last_active":"2024-04-02T04:53:11.922672+0000","last_peered":"2024-04-02T04:53:11.922672+0000","last_clean":"2024-04-02T04:53:11.922672+0000","last_became_active":"2024-04-02T04:49:25.091539+0000","last_became_peered":"2024-04-02T04:49:25.091539+0000","last_unstale":"2024-04-02T04:53:11.922672+0000","last_undegraded":"2024-04-02T04:53:11.922672+0000","last_fullsized":"2024-04-02T04:53:11.922672+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:32.581233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:32.581233+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.937832+0000","last_change":"2024-04-02T04:50:37.840934+0000","last_active":"2024-04-02T04:53:11.937832+0000","last_peered":"2024-04-02T04:53:11.937832+0000","last_clean":"2024-04-02T04:53:11.937832+0000","last_became_active":"2024-04-02T04:49:26.256165+0000","last_became_peered":"2024-04-02T04:49:26.256165+0000","last_unstale":"2024-04-02T04:53:11.937832+0000","last_undegraded":"2024-04-02T04:53:11.937832+0000","last_fullsized":"2024-04-02T04:53:11.937832+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.840813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.840813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"116'14","reported_seq":608,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.923745+0000","last_change":"2024-04-02T04:49:47.432220+0000","last_active":"2024-04-02T04:53:11.923745+0000","last_peered":"2024-04-02T04:53:11.923745+0000","last_clean":"2024-04-02T04:49:45.374676+0000","last_became_active":"2024-04-02T04:49:47.432220+0000","last_became_peered":"2024-04-02T04:49:47.432220+0000","last_unstale":"2024-04-02T04:53:11.923745+0000","last_undegraded":"2024-04-02T04:49:47.384090+0000","last_fullsized":"2024-04-02T04:49:47.383905+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:49:05.019939+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.019939+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":595,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.937944+0000","last_change":"2024-04-02T04:49:47.404868+0000","last_active":"2024-04-02T04:53:11.937944+0000","last_peered":"2024-04-02T04:53:11.937944+0000","last_clean":"2024-04-02T04:49:45.386009+0000","last_became_active":"2024-04-02T04:49:47.404868+0000","last_became_peered":"2024-04-02T04:49:47.404868+0000","last_unstale":"2024-04-02T04:53:11.937944+0000","last_undegraded":"2024-04-02T04:53:11.937944+0000","last_fullsized":"2024-04-02T04:49:47.375401+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:14.778569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.778569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":597,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.929893+0000","last_change":"2024-04-02T04:52:58.690594+0000","last_active":"2024-04-02T04:53:11.929893+0000","last_peered":"2024-04-02T04:53:11.929893+0000","last_clean":"2024-04-02T04:53:11.929893+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:53:11.929893+0000","last_undegraded":"2024-04-02T04:53:11.929893+0000","last_fullsized":"2024-04-02T04:53:11.929893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.690492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.690492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:00.009825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":581,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922368+0000","last_change":"2024-04-02T04:50:27.563922+0000","last_active":"2024-04-02T04:53:11.922368+0000","last_peered":"2024-04-02T04:53:11.922368+0000","last_clean":"2024-04-02T04:53:11.922368+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:53:11.922368+0000","last_undegraded":"2024-04-02T04:53:11.922368+0000","last_fullsized":"2024-04-02T04:53:11.922368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.563824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.563824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919675+0000","last_change":"2024-04-02T04:52:49.244563+0000","last_active":"2024-04-02T04:53:11.919675+0000","last_peered":"2024-04-02T04:53:11.919675+0000","last_clean":"2024-04-02T04:53:11.919675+0000","last_became_active":"2024-04-02T04:49:26.507403+0000","last_became_peered":"2024-04-02T04:49:26.507403+0000","last_unstale":"2024-04-02T04:53:11.919675+0000","last_undegraded":"2024-04-02T04:53:11.919675+0000","last_fullsized":"2024-04-02T04:53:11.919675+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.244258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.244258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:00.773526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":445,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.935866+0000","last_change":"2024-04-02T04:49:47.517198+0000","last_active":"2024-04-02T04:53:11.935866+0000","last_peered":"2024-04-02T04:53:11.935866+0000","last_clean":"2024-04-02T04:49:45.381250+0000","last_became_active":"2024-04-02T04:49:47.517198+0000","last_became_peered":"2024-04-02T04:49:47.517198+0000","last_unstale":"2024-04-02T04:53:11.935866+0000","last_undegraded":"2024-04-02T04:53:11.935866+0000","last_fullsized":"2024-04-02T04:49:47.430351+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"121'29","reported_seq":636,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916459+0000","last_change":"2024-04-02T04:50:14.537273+0000","last_active":"2024-04-02T04:53:11.916459+0000","last_peered":"2024-04-02T04:53:11.916459+0000","last_clean":"2024-04-02T04:53:11.916459+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:53:11.916459+0000","last_undegraded":"2024-04-02T04:53:11.916459+0000","last_fullsized":"2024-04-02T04:53:11.916459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:50:14.537029+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.537029+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":597,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919827+0000","last_change":"2024-04-02T04:49:47.468861+0000","last_active":"2024-04-02T04:53:11.919827+0000","last_peered":"2024-04-02T04:53:11.919827+0000","last_clean":"2024-04-02T04:49:45.338835+0000","last_became_active":"2024-04-02T04:49:47.468861+0000","last_became_peered":"2024-04-02T04:49:47.468861+0000","last_unstale":"2024-04-02T04:53:11.919827+0000","last_undegraded":"2024-04-02T04:53:11.919827+0000","last_fullsized":"2024-04-02T04:49:47.401513+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.329420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.329420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":586,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921731+0000","last_change":"2024-04-02T04:50:29.283629+0000","last_active":"2024-04-02T04:53:11.921731+0000","last_peered":"2024-04-02T04:53:11.921731+0000","last_clean":"2024-04-02T04:53:11.921731+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:53:11.921731+0000","last_undegraded":"2024-04-02T04:53:11.921731+0000","last_fullsized":"2024-04-02T04:53:11.921731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.283405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":583,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922380+0000","last_change":"2024-04-02T04:50:40.672009+0000","last_active":"2024-04-02T04:53:11.922380+0000","last_peered":"2024-04-02T04:53:11.922380+0000","last_clean":"2024-04-02T04:53:11.922380+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:53:11.922380+0000","last_undegraded":"2024-04-02T04:53:11.922380+0000","last_fullsized":"2024-04-02T04:53:11.922380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.671923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.671923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"138'2","reported_seq":447,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.926412+0000","last_change":"2024-04-02T04:49:47.455012+0000","last_active":"2024-04-02T04:53:11.926412+0000","last_peered":"2024-04-02T04:53:11.926412+0000","last_clean":"2024-04-02T04:49:45.373759+0000","last_became_active":"2024-04-02T04:49:47.455012+0000","last_became_peered":"2024-04-02T04:49:47.455012+0000","last_unstale":"2024-04-02T04:53:11.926412+0000","last_undegraded":"2024-04-02T04:53:11.926412+0000","last_fullsized":"2024-04-02T04:49:47.384636+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":450,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.935989+0000","last_change":"2024-04-02T04:50:39.321796+0000","last_active":"2024-04-02T04:53:11.935989+0000","last_peered":"2024-04-02T04:53:11.935989+0000","last_clean":"2024-04-02T04:53:11.935989+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:53:11.935989+0000","last_undegraded":"2024-04-02T04:53:11.935989+0000","last_fullsized":"2024-04-02T04:53:11.935989+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.321605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"85'18","reported_seq":630,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.950986+0000","last_change":"2024-04-02T04:52:48.727813+0000","last_active":"2024-04-02T04:53:11.950986+0000","last_peered":"2024-04-02T04:53:11.950986+0000","last_clean":"2024-04-02T04:53:11.950986+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:53:11.950986+0000","last_undegraded":"2024-04-02T04:53:11.950986+0000","last_fullsized":"2024-04-02T04:53:11.950986+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:52:48.727589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.727589+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.632589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":415,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.952060+0000","last_change":"2024-04-02T04:49:47.435134+0000","last_active":"2024-04-02T04:53:11.952060+0000","last_peered":"2024-04-02T04:53:11.952060+0000","last_clean":"2024-04-02T04:43:54.216783+0000","last_became_active":"2024-04-02T04:49:47.435134+0000","last_became_peered":"2024-04-02T04:49:47.435134+0000","last_unstale":"2024-04-02T04:53:11.952060+0000","last_undegraded":"2024-04-02T04:49:47.360301+0000","last_fullsized":"2024-04-02T04:49:47.360179+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:48:48.436655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:48.436655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":584,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917297+0000","last_change":"2024-04-02T04:51:48.311838+0000","last_active":"2024-04-02T04:53:11.917297+0000","last_peered":"2024-04-02T04:53:11.917297+0000","last_clean":"2024-04-02T04:53:11.917297+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:53:11.917297+0000","last_undegraded":"2024-04-02T04:53:11.917297+0000","last_fullsized":"2024-04-02T04:53:11.917297+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.311616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.311616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":575,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.918201+0000","last_change":"2024-04-02T04:49:47.495495+0000","last_active":"2024-04-02T04:53:11.918201+0000","last_peered":"2024-04-02T04:53:11.918201+0000","last_clean":"2024-04-02T04:49:45.352563+0000","last_became_active":"2024-04-02T04:49:47.495495+0000","last_became_peered":"2024-04-02T04:49:47.495495+0000","last_unstale":"2024-04-02T04:53:11.918201+0000","last_undegraded":"2024-04-02T04:53:11.918201+0000","last_fullsized":"2024-04-02T04:49:47.378202+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"139'9","reported_seq":427,"reported_epoch":352,"state":"active+undersized+degraded","last_fresh":"2024-04-02T04:53:11.924229+0000","last_change":"2024-04-02T04:49:47.518710+0000","last_active":"2024-04-02T04:53:11.924229+0000","last_peered":"2024-04-02T04:53:11.924229+0000","last_clean":"2024-04-02T04:49:27.516310+0000","last_became_active":"2024-04-02T04:49:47.518710+0000","last_became_peered":"2024-04-02T04:49:47.518710+0000","last_unstale":"2024-04-02T04:53:11.924229+0000","last_undegraded":"2024-04-02T04:49:47.393557+0000","last_fullsized":"2024-04-02T04:49:47.393345+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":451,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921361+0000","last_change":"2024-04-02T04:51:24.638019+0000","last_active":"2024-04-02T04:53:11.921361+0000","last_peered":"2024-04-02T04:53:11.921361+0000","last_clean":"2024-04-02T04:53:11.921361+0000","last_became_active":"2024-04-02T04:49:25.094609+0000","last_became_peered":"2024-04-02T04:49:25.094609+0000","last_unstale":"2024-04-02T04:53:11.921361+0000","last_undegraded":"2024-04-02T04:53:11.921361+0000","last_fullsized":"2024-04-02T04:53:11.921361+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:24.637853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:24.637853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"85'15","reported_seq":620,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.952351+0000","last_change":"2024-04-02T04:52:53.802878+0000","last_active":"2024-04-02T04:53:11.952351+0000","last_peered":"2024-04-02T04:53:11.952351+0000","last_clean":"2024-04-02T04:53:11.952351+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:53:11.952351+0000","last_undegraded":"2024-04-02T04:53:11.952351+0000","last_fullsized":"2024-04-02T04:53:11.952351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:53.802714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.802714+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:21.332468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":406,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.941646+0000","last_change":"2024-04-02T04:49:47.470693+0000","last_active":"2024-04-02T04:53:11.941646+0000","last_peered":"2024-04-02T04:53:11.941646+0000","last_clean":"2024-04-02T04:43:53.179277+0000","last_became_active":"2024-04-02T04:49:47.470693+0000","last_became_peered":"2024-04-02T04:49:47.470693+0000","last_unstale":"2024-04-02T04:53:11.941646+0000","last_undegraded":"2024-04-02T04:53:11.941646+0000","last_fullsized":"2024-04-02T04:49:47.411420+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:22.452700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:22.452700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":584,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.919771+0000","last_change":"2024-04-02T04:50:09.464337+0000","last_active":"2024-04-02T04:53:11.919771+0000","last_peered":"2024-04-02T04:53:11.919771+0000","last_clean":"2024-04-02T04:53:11.919771+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:53:11.919771+0000","last_undegraded":"2024-04-02T04:53:11.919771+0000","last_fullsized":"2024-04-02T04:53:11.919771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.464222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.464222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":595,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.949410+0000","last_change":"2024-04-02T04:52:47.730759+0000","last_active":"2024-04-02T04:53:11.949410+0000","last_peered":"2024-04-02T04:53:11.949410+0000","last_clean":"2024-04-02T04:53:11.949410+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:53:11.949410+0000","last_undegraded":"2024-04-02T04:53:11.949410+0000","last_fullsized":"2024-04-02T04:53:11.949410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.730634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.730634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:48.839408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":458,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921117+0000","last_change":"2024-04-02T04:53:06.382048+0000","last_active":"2024-04-02T04:53:11.921117+0000","last_peered":"2024-04-02T04:53:11.921117+0000","last_clean":"2024-04-02T04:53:11.921117+0000","last_became_active":"2024-04-02T04:49:25.076275+0000","last_became_peered":"2024-04-02T04:49:25.076275+0000","last_unstale":"2024-04-02T04:53:11.921117+0000","last_undegraded":"2024-04-02T04:53:11.921117+0000","last_fullsized":"2024-04-02T04:53:11.921117+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.381932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.381932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.209600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":448,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.938237+0000","last_change":"2024-04-02T04:52:49.488923+0000","last_active":"2024-04-02T04:53:11.938237+0000","last_peered":"2024-04-02T04:53:11.938237+0000","last_clean":"2024-04-02T04:53:11.938237+0000","last_became_active":"2024-04-02T04:49:26.275375+0000","last_became_peered":"2024-04-02T04:49:26.275375+0000","last_unstale":"2024-04-02T04:53:11.938237+0000","last_undegraded":"2024-04-02T04:53:11.938237+0000","last_fullsized":"2024-04-02T04:53:11.938237+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.488717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.488717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:06.730484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":628,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.927020+0000","last_change":"2024-04-02T04:52:52.303113+0000","last_active":"2024-04-02T04:53:11.927020+0000","last_peered":"2024-04-02T04:53:11.927020+0000","last_clean":"2024-04-02T04:53:11.927020+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:53:11.927020+0000","last_undegraded":"2024-04-02T04:53:11.927020+0000","last_fullsized":"2024-04-02T04:53:11.927020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:52:52.302878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.302878+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.761680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":603,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922378+0000","last_change":"2024-04-02T04:53:01.793779+0000","last_active":"2024-04-02T04:53:11.922378+0000","last_peered":"2024-04-02T04:53:11.922378+0000","last_clean":"2024-04-02T04:53:11.922378+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:53:11.922378+0000","last_undegraded":"2024-04-02T04:53:11.922378+0000","last_fullsized":"2024-04-02T04:53:11.922378+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:01.793655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:01.793655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.044051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":584,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917069+0000","last_change":"2024-04-02T04:51:49.302781+0000","last_active":"2024-04-02T04:53:11.917069+0000","last_peered":"2024-04-02T04:53:11.917069+0000","last_clean":"2024-04-02T04:53:11.917069+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:53:11.917069+0000","last_undegraded":"2024-04-02T04:53:11.917069+0000","last_fullsized":"2024-04-02T04:53:11.917069+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.302549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.302549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:17.024873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":580,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.916423+0000","last_change":"2024-04-02T04:52:41.343784+0000","last_active":"2024-04-02T04:53:11.916423+0000","last_peered":"2024-04-02T04:53:11.916423+0000","last_clean":"2024-04-02T04:53:11.916423+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:53:11.916423+0000","last_undegraded":"2024-04-02T04:53:11.916423+0000","last_fullsized":"2024-04-02T04:53:11.916423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.343663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.343663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:59.505758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"139'7","reported_seq":457,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933271+0000","last_change":"2024-04-02T04:51:38.237676+0000","last_active":"2024-04-02T04:53:11.933271+0000","last_peered":"2024-04-02T04:53:11.933271+0000","last_clean":"2024-04-02T04:53:11.933271+0000","last_became_active":"2024-04-02T04:49:26.241390+0000","last_became_peered":"2024-04-02T04:49:26.241390+0000","last_unstale":"2024-04-02T04:53:11.933271+0000","last_undegraded":"2024-04-02T04:53:11.933271+0000","last_fullsized":"2024-04-02T04:53:11.933271+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-04-02T04:51:38.237550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.237550+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":449,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.921744+0000","last_change":"2024-04-02T04:51:36.397355+0000","last_active":"2024-04-02T04:53:11.921744+0000","last_peered":"2024-04-02T04:53:11.921744+0000","last_clean":"2024-04-02T04:53:11.921744+0000","last_became_active":"2024-04-02T04:49:26.536918+0000","last_became_peered":"2024-04-02T04:49:26.536918+0000","last_unstale":"2024-04-02T04:53:11.921744+0000","last_undegraded":"2024-04-02T04:53:11.921744+0000","last_fullsized":"2024-04-02T04:53:11.921744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.397213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.397213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"342'22","reported_seq":632,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.922532+0000","last_change":"2024-04-02T04:53:02.778087+0000","last_active":"2024-04-02T04:53:11.922532+0000","last_peered":"2024-04-02T04:53:11.922532+0000","last_clean":"2024-04-02T04:53:11.922532+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:53:11.922532+0000","last_undegraded":"2024-04-02T04:53:11.922532+0000","last_fullsized":"2024-04-02T04:53:11.922532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'22","last_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_deep_scrub":"342'22","last_deep_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.777776+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:23.211739+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":588,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.927361+0000","last_change":"2024-04-02T04:49:47.432079+0000","last_active":"2024-04-02T04:53:11.927361+0000","last_peered":"2024-04-02T04:53:11.927361+0000","last_clean":"2024-04-02T04:49:45.381663+0000","last_became_active":"2024-04-02T04:49:47.432079+0000","last_became_peered":"2024-04-02T04:49:47.432079+0000","last_unstale":"2024-04-02T04:53:11.927361+0000","last_undegraded":"2024-04-02T04:53:11.927361+0000","last_fullsized":"2024-04-02T04:49:47.406616+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":594,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.917278+0000","last_change":"2024-04-02T04:52:56.823906+0000","last_active":"2024-04-02T04:53:11.917278+0000","last_peered":"2024-04-02T04:53:11.917278+0000","last_clean":"2024-04-02T04:53:11.917278+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:53:11.917278+0000","last_undegraded":"2024-04-02T04:53:11.917278+0000","last_fullsized":"2024-04-02T04:53:11.917278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.823620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.823620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:13.652305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":577,"reported_epoch":352,"state":"active+undersized","last_fresh":"2024-04-02T04:53:11.919567+0000","last_change":"2024-04-02T04:49:47.454112+0000","last_active":"2024-04-02T04:53:11.919567+0000","last_peered":"2024-04-02T04:53:11.919567+0000","last_clean":"2024-04-02T04:49:45.335087+0000","last_became_active":"2024-04-02T04:49:47.454112+0000","last_became_peered":"2024-04-02T04:49:47.454112+0000","last_unstale":"2024-04-02T04:53:11.919567+0000","last_undegraded":"2024-04-02T04:53:11.919567+0000","last_fullsized":"2024-04-02T04:49:47.400802+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.389577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.389577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":458,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-04-02T04:53:11.933469+0000","last_change":"2024-04-02T04:53:06.393893+0000","last_active":"2024-04-02T04:53:11.933469+0000","last_peered":"2024-04-02T04:53:11.933469+0000","last_clean":"2024-04-02T04:53:11.933469+0000","last_became_active":"2024-04-02T04:49:26.269993+0000","last_became_peered":"2024-04-02T04:49:26.269993+0000","last_unstale":"2024-04-02T04:53:11.933469+0000","last_undegraded":"2024-04-02T04:53:11.933469+0000","last_fullsized":"2024-04-02T04:53:11.933469+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:23.095557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":271,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":8196958,"num_objects":267,"num_object_clones":143,"num_object_copies":801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":2,"num_read":99,"num_read_kb":4,"num_write":551,"num_write_kb":12642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27070464,"data_stored":24589035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1519,"ondisk_log_size":1519,"up":87,"acting":87,"num_store_stats":7},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":87,"acting":87,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":86,"acting":86,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":652,"ondisk_log_size":652,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":535,"num_read_kb":530,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743504,"num_pgs":239,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111160,"kb_used_data":75820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93633992,"statfs":{"total":95995035648,"available":95881207808,"internally_reserved":0,"allocated":77639680,"data_stored":76942360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103301,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90780,"kb_used_data":15152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93654372,"statfs":{"total":95995035648,"available":95902076928,"internally_reserved":0,"allocated":15515648,"data_stored":12636990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86144,"kb_used_data":10516,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659008,"statfs":{"total":95995035648,"available":95906824192,"internally_reserved":0,"allocated":10768384,"data_stored":8047620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":139,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82324,"kb_used_data":6684,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662828,"statfs":{"total":95995035648,"available":95910735872,"internally_reserved":0,"allocated":6844416,"data_stored":4442002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378798,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82112,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93663040,"statfs":{"total":95995035648,"available":95910952960,"internally_reserved":0,"allocated":6639616,"data_stored":3853961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051748,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73112,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93672040,"statfs":{"total":95995035648,"available":95920168960,"internally_reserved":0,"allocated":10203136,"data_stored":7851277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822810,"num_pgs":138,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67924,"kb_used_data":10368,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93677228,"statfs":{"total":95995035648,"available":95925481472,"internally_reserved":0,"allocated":10616832,"data_stored":8047380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463054,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63412,"kb_used_data":9992,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93681740,"statfs":{"total":95995035648,"available":95930101760,"internally_reserved":0,"allocated":10231808,"data_stored":7852319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":4195493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4145152,"data_stored":4000361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4497408,"data_stored":4195019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4000431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8859648,"data_stored":8195959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:53:18.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 1 2024-04-02T04:53:18.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:18.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:53:18.762 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:18.834 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:53:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:53:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:53:18.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:18.890 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:18.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:18.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:19.012 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:19.215 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:19.243 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:19.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:19.373 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:19.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:19.502 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:19.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: cluster 2024-04-02T04:53:18.040772+0000 mgr.y (mgr.14538) 382 : cluster 0 pgmap v514: 356 pgs: 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: cluster 2024-04-02T04:53:18.857050+0000 mon.a (mon.0) 2358 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.874881+0000 mon.a (mon.0) 2359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.875122+0000 mon.a (mon.0) 2360 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: cluster 2024-04-02T04:53:18.882189+0000 mon.a (mon.0) 2361 : cluster 0 osdmap e359: 8 total, 7 up, 8 in 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.912948+0000 mon.c (mon.2) 633 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.913847+0000 mon.a (mon.0) 2362 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.914579+0000 mon.c (mon.2) 634 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.914904+0000 mon.a (mon.0) 2363 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.915727+0000 mon.c (mon.2) 635 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:19.617 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:18.916102+0000 mon.a (mon.0) 2364 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[28904]: audit 2024-04-02T04:53:19.097446+0000 mon.c (mon.2) 636 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: cluster 2024-04-02T04:53:18.040772+0000 mgr.y (mgr.14538) 382 : cluster 0 pgmap v514: 356 pgs: 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: cluster 2024-04-02T04:53:18.857050+0000 mon.a (mon.0) 2358 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.874881+0000 mon.a (mon.0) 2359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.875122+0000 mon.a (mon.0) 2360 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: cluster 2024-04-02T04:53:18.882189+0000 mon.a (mon.0) 2361 : cluster 0 osdmap e359: 8 total, 7 up, 8 in 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.912948+0000 mon.c (mon.2) 633 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.913847+0000 mon.a (mon.0) 2362 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.914579+0000 mon.c (mon.2) 634 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.914904+0000 mon.a (mon.0) 2363 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.619 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.915727+0000 mon.c (mon.2) 635 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:19.619 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:18.916102+0000 mon.a (mon.0) 2364 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:19.619 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:19 smithi069 bash[21528]: audit 2024-04-02T04:53:19.097446+0000 mon.c (mon.2) 636 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:19.633 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: cluster 2024-04-02T04:53:18.040772+0000 mgr.y (mgr.14538) 382 : cluster 0 pgmap v514: 356 pgs: 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: cluster 2024-04-02T04:53:18.857050+0000 mon.a (mon.0) 2358 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.874881+0000 mon.a (mon.0) 2359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.875122+0000 mon.a (mon.0) 2360 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-72153-55"}]': finished 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: cluster 2024-04-02T04:53:18.882189+0000 mon.a (mon.0) 2361 : cluster 0 osdmap e359: 8 total, 7 up, 8 in 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.912948+0000 mon.c (mon.2) 633 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.913847+0000 mon.a (mon.0) 2362 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.914579+0000 mon.c (mon.2) 634 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.914904+0000 mon.a (mon.0) 2363 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.915727+0000 mon.c (mon.2) 635 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:19.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:18.916102+0000 mon.a (mon.0) 2364 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:19.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:19 smithi098 bash[23790]: audit 2024-04-02T04:53:19.097446+0000 mon.c (mon.2) 636 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:19.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:19.765 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:19.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:19.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:19.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:19.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:20.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:20.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:20.142 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:20.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:20.272 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:20.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:20.403 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:20.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:20.533 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:20.636 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:20.663 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:20.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:20.791 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:20.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.877325+0000 mon.a (mon.0) 2365 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: cluster 2024-04-02T04:53:19.882466+0000 mon.a (mon.0) 2366 : cluster 0 osdmap e360: 8 total, 7 up, 8 in 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.886695+0000 mon.c (mon.2) 637 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.888141+0000 mon.a (mon.0) 2367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.962464+0000 mon.c (mon.2) 638 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.962868+0000 mon.a (mon.0) 2368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:20.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.963526+0000 mon.c (mon.2) 639 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]: dispatch 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:19.963844+0000 mon.a (mon.0) 2369 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]: dispatch 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: cluster 2024-04-02T04:53:20.041704+0000 mgr.y (mgr.14538) 383 : cluster 0 pgmap v517: 356 pgs: 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[21528]: audit 2024-04-02T04:53:20.098751+0000 mon.c (mon.2) 640 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.877325+0000 mon.a (mon.0) 2365 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: cluster 2024-04-02T04:53:19.882466+0000 mon.a (mon.0) 2366 : cluster 0 osdmap e360: 8 total, 7 up, 8 in 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.886695+0000 mon.c (mon.2) 637 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.888141+0000 mon.a (mon.0) 2367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.962464+0000 mon.c (mon.2) 638 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:20.911 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.962868+0000 mon.a (mon.0) 2368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:20.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.963526+0000 mon.c (mon.2) 639 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]: dispatch 2024-04-02T04:53:20.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:19.963844+0000 mon.a (mon.0) 2369 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]: dispatch 2024-04-02T04:53:20.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: cluster 2024-04-02T04:53:20.041704+0000 mgr.y (mgr.14538) 383 : cluster 0 pgmap v517: 356 pgs: 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:20.912 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:20 smithi069 bash[28904]: audit 2024-04-02T04:53:20.098751+0000 mon.c (mon.2) 640 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:20.925 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:20.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:21.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:21.053 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:21.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:21.168 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.877325+0000 mon.a (mon.0) 2365 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-72153-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:21.168 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: cluster 2024-04-02T04:53:19.882466+0000 mon.a (mon.0) 2366 : cluster 0 osdmap e360: 8 total, 7 up, 8 in 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.886695+0000 mon.c (mon.2) 637 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.888141+0000 mon.a (mon.0) 2367 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.962464+0000 mon.c (mon.2) 638 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.962868+0000 mon.a (mon.0) 2368 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.963526+0000 mon.c (mon.2) 639 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]: dispatch 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:19.963844+0000 mon.a (mon.0) 2369 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]: dispatch 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: cluster 2024-04-02T04:53:20.041704+0000 mgr.y (mgr.14538) 383 : cluster 0 pgmap v517: 356 pgs: 91 active+undersized, 228 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:21.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:20 smithi098 bash[23790]: audit 2024-04-02T04:53:20.098751+0000 mon.c (mon.2) 640 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:21.184 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:21.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:21.291 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:53:21.291 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:53:21.291 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:53:21.291 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:53:21.291 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:53:21.311 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:21.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:21.440 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:21.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:21.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:21.697 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:21.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:21.828 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:21.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:21.955 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:21.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:21.972 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:21 smithi098 bash[23790]: audit 2024-04-02T04:53:20.885647+0000 mon.a (mon.0) 2370 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]': finished 2024-04-02T04:53:21.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:21 smithi098 bash[23790]: audit 2024-04-02T04:53:21.099957+0000 mon.c (mon.2) 641 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:21.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:21 smithi098 bash[23790]: cluster 2024-04-02T04:53:21.347068+0000 mon.a (mon.0) 2371 : cluster 0 osdmap e361: 8 total, 7 up, 8 in 2024-04-02T04:53:22.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:22.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-02T04:53:22.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:21 smithi069 bash[21528]: audit 2024-04-02T04:53:20.885647+0000 mon.a (mon.0) 2370 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]': finished 2024-04-02T04:53:22.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:21 smithi069 bash[21528]: audit 2024-04-02T04:53:21.099957+0000 mon.c (mon.2) 641 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:22.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:21 smithi069 bash[21528]: cluster 2024-04-02T04:53:21.347068+0000 mon.a (mon.0) 2371 : cluster 0 osdmap e361: 8 total, 7 up, 8 in 2024-04-02T04:53:22.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:21 smithi069 bash[28904]: audit 2024-04-02T04:53:20.885647+0000 mon.a (mon.0) 2370 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-41"}]': finished 2024-04-02T04:53:22.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:21 smithi069 bash[28904]: audit 2024-04-02T04:53:21.099957+0000 mon.c (mon.2) 641 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:22.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:21 smithi069 bash[28904]: cluster 2024-04-02T04:53:21.347068+0000 mon.a (mon.0) 2371 : cluster 0 osdmap e361: 8 total, 7 up, 8 in 2024-04-02T04:53:22.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:22.213 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:22.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:22.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:22.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:22.572 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:22.598 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:22.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:22.729 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:22.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:22.860 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:22.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[21528]: audit 2024-04-02T04:53:21.895490+0000 mon.a (mon.0) 2372 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[21528]: cluster 2024-04-02T04:53:21.898957+0000 mon.a (mon.0) 2373 : cluster 0 osdmap e362: 8 total, 7 up, 8 in 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[21528]: cluster 2024-04-02T04:53:22.043517+0000 mgr.y (mgr.14538) 384 : cluster 0 pgmap v520: 332 pgs: 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[21528]: audit 2024-04-02T04:53:22.101128+0000 mon.c (mon.2) 642 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[28904]: audit 2024-04-02T04:53:21.895490+0000 mon.a (mon.0) 2372 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[28904]: cluster 2024-04-02T04:53:21.898957+0000 mon.a (mon.0) 2373 : cluster 0 osdmap e362: 8 total, 7 up, 8 in 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[28904]: cluster 2024-04-02T04:53:22.043517+0000 mgr.y (mgr.14538) 384 : cluster 0 pgmap v520: 332 pgs: 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:22.971 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:22 smithi069 bash[28904]: audit 2024-04-02T04:53:22.101128+0000 mon.c (mon.2) 642 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:22.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:22.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:23.085 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:23.112 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:23.188 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:22 smithi098 bash[23790]: audit 2024-04-02T04:53:21.895490+0000 mon.a (mon.0) 2372 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-72153-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:23.188 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:22 smithi098 bash[23790]: cluster 2024-04-02T04:53:21.898957+0000 mon.a (mon.0) 2373 : cluster 0 osdmap e362: 8 total, 7 up, 8 in 2024-04-02T04:53:23.188 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:22 smithi098 bash[23790]: cluster 2024-04-02T04:53:22.043517+0000 mgr.y (mgr.14538) 384 : cluster 0 pgmap v520: 332 pgs: 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:23.188 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:22 smithi098 bash[23790]: audit 2024-04-02T04:53:22.101128+0000 mon.c (mon.2) 642 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:23.215 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:23.238 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:23.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:23.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:23.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:53:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:53:23.496 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:23.599 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:23.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:23.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:23.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:23.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:23.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:23.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:23 smithi098 bash[23790]: cluster 2024-04-02T04:53:22.910928+0000 mon.a (mon.0) 2374 : cluster 0 osdmap e363: 8 total, 7 up, 8 in 2024-04-02T04:53:23.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:23 smithi098 bash[23790]: audit 2024-04-02T04:53:22.920366+0000 mon.c (mon.2) 643 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:23.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:23 smithi098 bash[23790]: audit 2024-04-02T04:53:22.922760+0000 mon.a (mon.0) 2375 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:23.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:23 smithi098 bash[23790]: audit 2024-04-02T04:53:23.102216+0000 mon.c (mon.2) 644 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:23.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:23 smithi098 bash[23790]: audit 2024-04-02T04:53:23.192068+0000 mgr.y (mgr.14538) 385 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:24.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:24.131 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:24.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:24.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[28904]: cluster 2024-04-02T04:53:22.910928+0000 mon.a (mon.0) 2374 : cluster 0 osdmap e363: 8 total, 7 up, 8 in 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[28904]: audit 2024-04-02T04:53:22.920366+0000 mon.c (mon.2) 643 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[28904]: audit 2024-04-02T04:53:22.922760+0000 mon.a (mon.0) 2375 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[28904]: audit 2024-04-02T04:53:23.102216+0000 mon.c (mon.2) 644 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[28904]: audit 2024-04-02T04:53:23.192068+0000 mgr.y (mgr.14538) 385 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[21528]: cluster 2024-04-02T04:53:22.910928+0000 mon.a (mon.0) 2374 : cluster 0 osdmap e363: 8 total, 7 up, 8 in 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[21528]: audit 2024-04-02T04:53:22.920366+0000 mon.c (mon.2) 643 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[21528]: audit 2024-04-02T04:53:22.922760+0000 mon.a (mon.0) 2375 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[21528]: audit 2024-04-02T04:53:23.102216+0000 mon.c (mon.2) 644 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:24.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:23 smithi069 bash[21528]: audit 2024-04-02T04:53:23.192068+0000 mgr.y (mgr.14538) 385 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:24.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:24.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:24.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:24.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:24.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:24.649 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:24.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:24.779 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:24.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:24.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:24.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:25.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.037 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:25.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.163 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:25.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:23.908193+0000 mon.a (mon.0) 2376 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: cluster 2024-04-02T04:53:23.917351+0000 mon.a (mon.0) 2377 : cluster 0 osdmap e364: 8 total, 7 up, 8 in 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:23.926014+0000 mon.c (mon.2) 645 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:23.940952+0000 mon.c (mon.2) 646 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-72349-6","var": "pg_num","format": "json"}]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:23.952281+0000 mon.c (mon.2) 647 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:23.954003+0000 mon.a (mon.0) 2378 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:23.955009+0000 mon.a (mon.0) 2379 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: cluster 2024-04-02T04:53:24.045314+0000 mgr.y (mgr.14538) 386 : cluster 0 pgmap v523: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.103926+0000 mon.c (mon.2) 648 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: cluster 2024-04-02T04:53:24.844286+0000 mon.a (mon.0) 2380 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.848334+0000 mon.a (mon.0) 2381 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.848567+0000 mon.a (mon.0) 2382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: cluster 2024-04-02T04:53:24.856400+0000 mon.a (mon.0) 2383 : cluster 0 osdmap e365: 8 total, 7 up, 8 in 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.858532+0000 mon.c (mon.2) 649 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.859262+0000 mon.c (mon.2) 650 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-02T04:53:25.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.862258+0000 mon.a (mon.0) 2384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[21528]: audit 2024-04-02T04:53:24.863232+0000 mon.a (mon.0) 2385 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:23.908193+0000 mon.a (mon.0) 2376 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: cluster 2024-04-02T04:53:23.917351+0000 mon.a (mon.0) 2377 : cluster 0 osdmap e364: 8 total, 7 up, 8 in 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:23.926014+0000 mon.c (mon.2) 645 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:23.940952+0000 mon.c (mon.2) 646 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-72349-6","var": "pg_num","format": "json"}]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:23.952281+0000 mon.c (mon.2) 647 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:23.954003+0000 mon.a (mon.0) 2378 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:23.955009+0000 mon.a (mon.0) 2379 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: cluster 2024-04-02T04:53:24.045314+0000 mgr.y (mgr.14538) 386 : cluster 0 pgmap v523: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.103926+0000 mon.c (mon.2) 648 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:25.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: cluster 2024-04-02T04:53:24.844286+0000 mon.a (mon.0) 2380 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.848334+0000 mon.a (mon.0) 2381 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.848567+0000 mon.a (mon.0) 2382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: cluster 2024-04-02T04:53:24.856400+0000 mon.a (mon.0) 2383 : cluster 0 osdmap e365: 8 total, 7 up, 8 in 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.858532+0000 mon.c (mon.2) 649 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.859262+0000 mon.c (mon.2) 650 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.862258+0000 mon.a (mon.0) 2384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:24 smithi069 bash[28904]: audit 2024-04-02T04:53:24.863232+0000 mon.a (mon.0) 2385 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-02T04:53:25.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:23.908193+0000 mon.a (mon.0) 2376 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: cluster 2024-04-02T04:53:23.917351+0000 mon.a (mon.0) 2377 : cluster 0 osdmap e364: 8 total, 7 up, 8 in 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:23.926014+0000 mon.c (mon.2) 645 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:23.940952+0000 mon.c (mon.2) 646 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-72349-6","var": "pg_num","format": "json"}]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:23.952281+0000 mon.c (mon.2) 647 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:23.954003+0000 mon.a (mon.0) 2378 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:23.955009+0000 mon.a (mon.0) 2379 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: cluster 2024-04-02T04:53:24.045314+0000 mgr.y (mgr.14538) 386 : cluster 0 pgmap v523: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.103926+0000 mon.c (mon.2) 648 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: cluster 2024-04-02T04:53:24.844286+0000 mon.a (mon.0) 2380 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.848334+0000 mon.a (mon.0) 2381 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.848567+0000 mon.a (mon.0) 2382 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: cluster 2024-04-02T04:53:24.856400+0000 mon.a (mon.0) 2383 : cluster 0 osdmap e365: 8 total, 7 up, 8 in 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.858532+0000 mon.c (mon.2) 649 : audit 1 from='client.? 172.21.15.69:0/3582086137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.859262+0000 mon.c (mon.2) 650 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-02T04:53:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.862258+0000 mon.a (mon.0) 2384 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]: dispatch 2024-04-02T04:53:25.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:24 smithi098 bash[23790]: audit 2024-04-02T04:53:24.863232+0000 mon.a (mon.0) 2385 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-02T04:53:25.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.286 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:25.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:25.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:25.531 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:25.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:25.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:25.789 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:25.892 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:25.913 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:25.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:26.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:26.043 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:26.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:26.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:26.157 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.105610+0000 mon.c (mon.2) 651 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:26.157 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.851476+0000 mon.a (mon.0) 2386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:26.157 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.851573+0000 mon.a (mon.0) 2387 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]': finished 2024-04-02T04:53:26.157 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: cluster 2024-04-02T04:53:25.856659+0000 mon.a (mon.0) 2388 : cluster 0 osdmap e366: 8 total, 7 up, 8 in 2024-04-02T04:53:26.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.870801+0000 mon.c (mon.2) 652 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:26.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.871504+0000 mon.a (mon.0) 2389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:26.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.893039+0000 mon.a (mon.0) 2390 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:26.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.894117+0000 mon.a (mon.0) 2391 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:26.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:25 smithi098 bash[23790]: audit 2024-04-02T04:53:25.894507+0000 mon.a (mon.0) 2392 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-72153-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:26.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.105610+0000 mon.c (mon.2) 651 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.851476+0000 mon.a (mon.0) 2386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.851573+0000 mon.a (mon.0) 2387 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]': finished 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: cluster 2024-04-02T04:53:25.856659+0000 mon.a (mon.0) 2388 : cluster 0 osdmap e366: 8 total, 7 up, 8 in 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.870801+0000 mon.c (mon.2) 652 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.871504+0000 mon.a (mon.0) 2389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.893039+0000 mon.a (mon.0) 2390 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.894117+0000 mon.a (mon.0) 2391 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[21528]: audit 2024-04-02T04:53:25.894507+0000 mon.a (mon.0) 2392 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-72153-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:26.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.105610+0000 mon.c (mon.2) 651 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.851476+0000 mon.a (mon.0) 2386 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-72153-56"}]': finished 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.851573+0000 mon.a (mon.0) 2387 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_count","val": "8"}]': finished 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: cluster 2024-04-02T04:53:25.856659+0000 mon.a (mon.0) 2388 : cluster 0 osdmap e366: 8 total, 7 up, 8 in 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.870801+0000 mon.c (mon.2) 652 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.871504+0000 mon.a (mon.0) 2389 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.893039+0000 mon.a (mon.0) 2390 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.894117+0000 mon.a (mon.0) 2391 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:26.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:25 smithi069 bash[28904]: audit 2024-04-02T04:53:25.894507+0000 mon.a (mon.0) 2392 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-72153-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:26.302 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:26.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:26.436 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:26.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:26.563 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:26.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:26.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:26.930 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:26.960 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:26.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:27.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:27.089 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:27.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:27.203 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: cluster 2024-04-02T04:53:26.046330+0000 mgr.y (mgr.14538) 387 : cluster 0 pgmap v526: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:27.203 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: audit 2024-04-02T04:53:26.107081+0000 mon.c (mon.2) 653 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:27.203 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: audit 2024-04-02T04:53:26.853987+0000 mon.a (mon.0) 2393 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:27.203 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: audit 2024-04-02T04:53:26.854121+0000 mon.a (mon.0) 2394 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-72153-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: cluster 2024-04-02T04:53:26.858030+0000 mon.a (mon.0) 2395 : cluster 0 osdmap e367: 8 total, 7 up, 8 in 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: audit 2024-04-02T04:53:26.859573+0000 mon.a (mon.0) 2396 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-72153-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: audit 2024-04-02T04:53:26.859957+0000 mon.c (mon.2) 654 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[21528]: audit 2024-04-02T04:53:26.860744+0000 mon.a (mon.0) 2397 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: cluster 2024-04-02T04:53:26.046330+0000 mgr.y (mgr.14538) 387 : cluster 0 pgmap v526: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: audit 2024-04-02T04:53:26.107081+0000 mon.c (mon.2) 653 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: audit 2024-04-02T04:53:26.853987+0000 mon.a (mon.0) 2393 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: audit 2024-04-02T04:53:26.854121+0000 mon.a (mon.0) 2394 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-72153-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: cluster 2024-04-02T04:53:26.858030+0000 mon.a (mon.0) 2395 : cluster 0 osdmap e367: 8 total, 7 up, 8 in 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: audit 2024-04-02T04:53:26.859573+0000 mon.a (mon.0) 2396 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-72153-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: audit 2024-04-02T04:53:26.859957+0000 mon.c (mon.2) 654 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-02T04:53:27.204 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:27 smithi069 bash[28904]: audit 2024-04-02T04:53:26.860744+0000 mon.a (mon.0) 2397 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-02T04:53:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: cluster 2024-04-02T04:53:26.046330+0000 mgr.y (mgr.14538) 387 : cluster 0 pgmap v526: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: audit 2024-04-02T04:53:26.107081+0000 mon.c (mon.2) 653 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: audit 2024-04-02T04:53:26.853987+0000 mon.a (mon.0) 2393 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: audit 2024-04-02T04:53:26.854121+0000 mon.a (mon.0) 2394 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-72153-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: cluster 2024-04-02T04:53:26.858030+0000 mon.a (mon.0) 2395 : cluster 0 osdmap e367: 8 total, 7 up, 8 in 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: audit 2024-04-02T04:53:26.859573+0000 mon.a (mon.0) 2396 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-72153-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: audit 2024-04-02T04:53:26.859957+0000 mon.c (mon.2) 654 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-02T04:53:27.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:26 smithi098 bash[23790]: audit 2024-04-02T04:53:26.860744+0000 mon.a (mon.0) 2397 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-02T04:53:27.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:27.347 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:27.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:27.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:27.604 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:27.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:27.836 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:27.856 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:27.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:27.987 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:27.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:28.090 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:28.112 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:28.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:28.215 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 271:602f83fe:::foo:head 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8966 ms) 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 0 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 1 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 2 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 3 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 4 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 5 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 6 ls 1712033608,0 2024-04-02T04:53:28.216 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 7 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 8 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 9 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 10 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 11 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 12 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 13 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 14 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 15 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 16 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 17 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 18 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 19 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 20 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 21 ls 1712033608,0 2024-04-02T04:53:28.217 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 22 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 23 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 24 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 25 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 26 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 27 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 28 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 29 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 30 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 31 ls 1712033608,0 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6cac518f:::0:head 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:02547ec2:::1:head 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f905c69b:::2:head 2024-04-02T04:53:28.218 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:cfc208b3:::3:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d83876eb:::4:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b29083e3:::5:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c4fdafeb:::6:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:5c6b0b28:::7:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:bd63b0f1:::8:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e960b815:::9:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:52ea6a34:::10:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:89d3ae78:::11:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:de5d7c5f:::12:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:566253c9:::13:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:62a1935d:::14:head 2024-04-02T04:53:28.219 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:863748b0:::15:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3958e169:::16:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4d4dabf9:::17:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:8391935d:::18:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:28883081:::19:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:69259c59:::20:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4bdb80b7:::21:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a11c5d71:::22:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:271af37b:::23:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:95b121be:::24:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:58d1031b:::25:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:0a050783:::26:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c709704c:::27:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:cbe56eaf:::28:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:86b4b162:::29:head 2024-04-02T04:53:28.220 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:70d89383:::30:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:dd450c7c:::31:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6d5729b1:::32:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c388f3fb:::33:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:56cfea31:::34:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9dbc1bf7:::35:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:40b74ccd:::36:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4d5aaf42:::37:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:920f362c:::38:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6cc53222:::39:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9cad833f:::40:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:1ea84d41:::41:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c4480ef6:::42:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a694361e:::43:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d1bd33e9:::44:head 2024-04-02T04:53:28.221 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ddc2cd5d:::45:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:2b782207:::46:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7b187fca:::47:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:90ecdf6f:::48:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a5ed95fe:::49:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ea0eaa55:::50:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f33ef17b:::51:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a0d1b2f6:::52:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:60c5229e:::53:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:edcbc575:::54:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:102cf253:::55:head 2024-04-02T04:53:28.222 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:efb7fb0b:::56:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:50d0a326:::57:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d4dc5daf:::58:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3a130462:::59:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ec87ed71:::60:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d5bc9454:::61:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3ddfe313:::62:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7c2816b9:::63:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:47e00e4d:::64:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c6410c18:::65:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b48ed237:::66:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:cd63ad31:::67:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b179e92b:::68:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:0d9f741a:::69:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6d3352ae:::70:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c6d5c19e:::71:head 2024-04-02T04:53:28.223 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:bc4729c3:::72:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:77e930b9:::73:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:0abeecfd:::74:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b7c37e15:::75:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b6378398:::76:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:02bd68de:::77:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:cc795d2d:::78:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:630d4fea:::79:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e0d29ef5:::80:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:fd6f13d2:::81:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:606461d5:::82:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:eadbdc43:::83:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:8761d0bb:::84:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9ef0186f:::85:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e0d41294:::86:head 2024-04-02T04:53:28.224 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:961de695:::87:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:1423148f:::88:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:633a8fa2:::89:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a8653809:::90:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3dac8b33:::91:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:35aad435:::92:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f6dcc343:::93:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:dbbdad87:::94:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:1cb48ce0:::95:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:03cd461c:::96:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:17a4ea99:::97:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9993c9a7:::98:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6394211c:::99:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:94c7ae57:::100:head 2024-04-02T04:53:28.225 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6fdee5bb:::101:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9a477fd1:::102:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:eb850916:::103:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:affc56b9:::104:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b42dc814:::105:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f319f8f0:::106:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9a40b9de:::107:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:8b524f28:::108:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e3de589f:::109:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:90f90a5b:::110:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a7b4f1d7:::111:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:af51766e:::112:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b6f90bd1:::113:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e0261208:::114:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c9569ef7:::115:head 2024-04-02T04:53:28.226 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:61bebe50:::116:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:fe93412b:::117:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d3d38bee:::118:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3100ba0c:::119:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d0560ada:::120:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f0ea8b35:::121:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:766f231a:::122:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a07a2582:::123:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:bd7c6b3a:::124:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:fb2ddaff:::125:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4408e1fe:::126:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ee1df7a7:::127:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c3002909:::128:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4f48ffa9:::129:head 2024-04-02T04:53:28.227 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:edf38733:::130:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c08425c0:::131:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:5f902d98:::132:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:41ea2c93:::133:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:813cee13:::134:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:0131818d:::135:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:26ba5a85:::136:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:381b8a5a:::137:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:28797e47:::138:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:bfca7f22:::139:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:36807075:::140:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:80b03975:::141:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:5c15709b:::142:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f39ea15e:::143:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ea992956:::144:head 2024-04-02T04:53:28.228 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:48887b1c:::145:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:9f24a9dd:::146:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:987f100b:::147:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d2dd3581:::148:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7fed1808:::149:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c80b70e9:::150:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:85ed90f9:::151:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:36428b24:::152:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d044c34a:::153:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7c18bf58:::154:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d1c21232:::155:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a7a3c575:::156:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:87da0633:::157:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d5ac3822:::158:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3f20522d:::159:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6ca26563:::160:head 2024-04-02T04:53:28.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:532ce135:::161:head 2024-04-02T04:53:28.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:c78863e6:::162:head 2024-04-02T04:53:28.235 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:28.253 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:28.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:28.368 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:28 smithi069 bash[28904]: audit 2024-04-02T04:53:27.108411+0000 mon.c (mon.2) 655 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:28.368 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:28 smithi069 bash[28904]: audit 2024-04-02T04:53:27.856156+0000 mon.a (mon.0) 2398 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-02T04:53:28.375 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:28.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:28.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:28.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:28.630 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:28.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:28 smithi069 bash[21528]: audit 2024-04-02T04:53:27.108411+0000 mon.c (mon.2) 655 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:28.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:28 smithi069 bash[21528]: audit 2024-04-02T04:53:27.856156+0000 mon.a (mon.0) 2398 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-02T04:53:28.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:28 smithi098 bash[23790]: audit 2024-04-02T04:53:27.108411+0000 mon.c (mon.2) 655 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:28.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:28 smithi098 bash[23790]: audit 2024-04-02T04:53:27.856156+0000 mon.a (mon.0) 2398 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-02T04:53:28.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:28.761 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:28.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:28.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:28.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:29.024 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:29.031 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:53:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:53:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:53:29.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:29.153 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:29.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:29.283 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:29.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:29.408 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:29.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: cluster 2024-04-02T04:53:28.048190+0000 mgr.y (mgr.14538) 388 : cluster 0 pgmap v529: 356 pgs: 91 active+undersized, 227 active+clean, 38 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 280/1566 objects degraded (17.880%) 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.110115+0000 mon.c (mon.2) 656 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.219584+0000 mon.c (mon.2) 657 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-72349-43","var": "pg_num","format": "json"}]: dispatch 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: cluster 2024-04-02T04:53:28.321801+0000 mon.a (mon.0) 2399 : cluster 0 osdmap e368: 8 total, 7 up, 8 in 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.370529+0000 mon.c (mon.2) 658 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.370891+0000 mon.a (mon.0) 2400 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.371579+0000 mon.c (mon.2) 659 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]: dispatch 2024-04-02T04:53:29.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.372046+0000 mon.a (mon.0) 2401 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]: dispatch 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: cluster 2024-04-02T04:53:28.856320+0000 mon.a (mon.0) 2402 : cluster 3 Health check update: Degraded data redundancy: 280/1566 objects degraded (17.880%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.866190+0000 mon.a (mon.0) 2403 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-72153-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:28.866424+0000 mon.a (mon.0) 2404 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]': finished 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: cluster 2024-04-02T04:53:28.874572+0000 mon.a (mon.0) 2405 : cluster 0 osdmap e369: 8 total, 7 up, 8 in 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[21528]: audit 2024-04-02T04:53:29.111324+0000 mon.c (mon.2) 660 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: cluster 2024-04-02T04:53:28.048190+0000 mgr.y (mgr.14538) 388 : cluster 0 pgmap v529: 356 pgs: 91 active+undersized, 227 active+clean, 38 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 280/1566 objects degraded (17.880%) 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.110115+0000 mon.c (mon.2) 656 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.219584+0000 mon.c (mon.2) 657 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-72349-43","var": "pg_num","format": "json"}]: dispatch 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: cluster 2024-04-02T04:53:28.321801+0000 mon.a (mon.0) 2399 : cluster 0 osdmap e368: 8 total, 7 up, 8 in 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.370529+0000 mon.c (mon.2) 658 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.370891+0000 mon.a (mon.0) 2400 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:29.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.371579+0000 mon.c (mon.2) 659 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]: dispatch 2024-04-02T04:53:29.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.372046+0000 mon.a (mon.0) 2401 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]: dispatch 2024-04-02T04:53:29.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: cluster 2024-04-02T04:53:28.856320+0000 mon.a (mon.0) 2402 : cluster 3 Health check update: Degraded data redundancy: 280/1566 objects degraded (17.880%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:29.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.866190+0000 mon.a (mon.0) 2403 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-72153-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:29.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:28.866424+0000 mon.a (mon.0) 2404 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]': finished 2024-04-02T04:53:29.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: cluster 2024-04-02T04:53:28.874572+0000 mon.a (mon.0) 2405 : cluster 0 osdmap e369: 8 total, 7 up, 8 in 2024-04-02T04:53:29.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:29 smithi069 bash[28904]: audit 2024-04-02T04:53:29.111324+0000 mon.c (mon.2) 660 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:29.539 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:29.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:29.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: cluster 2024-04-02T04:53:28.048190+0000 mgr.y (mgr.14538) 388 : cluster 0 pgmap v529: 356 pgs: 91 active+undersized, 227 active+clean, 38 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 280/1566 objects degraded (17.880%) 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.110115+0000 mon.c (mon.2) 656 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.219584+0000 mon.c (mon.2) 657 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-72349-43","var": "pg_num","format": "json"}]: dispatch 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: cluster 2024-04-02T04:53:28.321801+0000 mon.a (mon.0) 2399 : cluster 0 osdmap e368: 8 total, 7 up, 8 in 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.370529+0000 mon.c (mon.2) 658 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.370891+0000 mon.a (mon.0) 2400 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.371579+0000 mon.c (mon.2) 659 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]: dispatch 2024-04-02T04:53:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.372046+0000 mon.a (mon.0) 2401 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]: dispatch 2024-04-02T04:53:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: cluster 2024-04-02T04:53:28.856320+0000 mon.a (mon.0) 2402 : cluster 3 Health check update: Degraded data redundancy: 280/1566 objects degraded (17.880%), 38 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.866190+0000 mon.a (mon.0) 2403 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-72153-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:28.866424+0000 mon.a (mon.0) 2404 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-43"}]': finished 2024-04-02T04:53:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: cluster 2024-04-02T04:53:28.874572+0000 mon.a (mon.0) 2405 : cluster 0 osdmap e369: 8 total, 7 up, 8 in 2024-04-02T04:53:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:29 smithi098 bash[23790]: audit 2024-04-02T04:53:29.111324+0000 mon.c (mon.2) 660 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:29.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:29.798 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:29.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:29.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:29.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:30.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:30.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:30.182 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:30.284 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:30.307 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:30.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:30.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:30.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:30.562 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:30.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:30.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:30 smithi069 bash[21528]: cluster 2024-04-02T04:53:29.847513+0000 mon.a (mon.0) 2406 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:30.676 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:30 smithi069 bash[21528]: cluster 2024-04-02T04:53:29.876372+0000 mon.a (mon.0) 2407 : cluster 0 osdmap e370: 8 total, 7 up, 8 in 2024-04-02T04:53:30.676 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:30 smithi069 bash[28904]: cluster 2024-04-02T04:53:29.847513+0000 mon.a (mon.0) 2406 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:30.676 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:30 smithi069 bash[28904]: cluster 2024-04-02T04:53:29.876372+0000 mon.a (mon.0) 2407 : cluster 0 osdmap e370: 8 total, 7 up, 8 in 2024-04-02T04:53:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:30.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:30 smithi098 bash[23790]: cluster 2024-04-02T04:53:29.847513+0000 mon.a (mon.0) 2406 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:30.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:30 smithi098 bash[23790]: cluster 2024-04-02T04:53:29.876372+0000 mon.a (mon.0) 2407 : cluster 0 osdmap e370: 8 total, 7 up, 8 in 2024-04-02T04:53:30.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:30.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:30.947 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:30.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:31.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:31.076 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:31.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:31.199 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:31.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:31.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.353+0000 7f588ae5d700 1 -- 172.21.15.69:0/3156668470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58841075b0 msgr2=0x7f588410f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.353+0000 7f588ae5d700 1 --2- 172.21.15.69:0/3156668470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58841075b0 0x7f588410f180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f587c00b0d0 tx=0x7f587c00b3e0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/3156668470 shutdown_connections 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 --2- 172.21.15.69:0/3156668470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58841075b0 0x7f588410f180 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 --2- 172.21.15.69:0/3156668470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5884102580 0x7f5884102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 --2- 172.21.15.69:0/3156668470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884102f30 0x7f5884106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/3156668470 >> 172.21.15.69:0/3156668470 conn(0x7f58840793a0 msgr2=0x7f58840797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/3156668470 shutdown_connections 2024-04-02T04:53:31.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/3156668470 wait complete. 2024-04-02T04:53:31.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 librados: starting msgr at 2024-04-02T04:53:31.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 librados: starting objecter 2024-04-02T04:53:31.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 Processor -- start 2024-04-02T04:53:31.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- start start 2024-04-02T04:53:31.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 librados: setting wanted keys 2024-04-02T04:53:31.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 librados: calling monclient init 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5884102580 0x7f5884074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 0x7f5884074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58841075b0 0x7f5884074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5884111ab0 con 0x7f58841075b0 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5884111930 con 0x7f5884102580 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5884111c30 con 0x7f5884102f30 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 0x7f5884074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f58893fa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58841075b0 0x7f5884074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 0x7f5884074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49572/0 (socket says 172.21.15.69:49572) 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 -- 172.21.15.69:0/703428386 learned_addr learned my addr 172.21.15.69:0/703428386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5888bf9700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5884102580 0x7f5884074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 -- 172.21.15.69:0/703428386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5884102580 msgr2=0x7f5884074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5884102580 0x7f5884074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 -- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58841075b0 msgr2=0x7f5884074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58841075b0 0x7f5884074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 -- 172.21.15.69:0/703428386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58841a6540 con 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5883fff700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 0x7f5884074a80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f587800ea30 tx=0x7f587800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5881ffb700 10 client.?.objecter ms_handle_connect 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5881ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587800cbd0 con 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f58780044a0 con 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58841a6830 con 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878003d30 con 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 10 client.26423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58841a6d70 con 0x7f5884102f30 2024-04-02T04:53:31.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 librados: init done 2024-04-02T04:53:31.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.357+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588404ff00 con 0x7f5884102f30 2024-04-02T04:53:31.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5878003f30 con 0x7f5884102f30 2024-04-02T04:53:31.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f587406f5e0 0x7f5874071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(371..371 src has 74..371) ==== 11409+0+0 (secure 0 0 0) 0x7f587808f1f0 con 0x7f5884102f30 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 10 client.26423.objecter ms_dispatch 0x7f5884001550 osd_map(371..371 src has 74..371) 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 3 client.26423.objecter handle_osd_map got epochs [371,371] > 0 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 3 client.26423.objecter handle_osd_map decoding full epoch 371 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5888bf9700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f587406f5e0 0x7f5874071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 20 client.26423.objecter dump_active .. 0 homeless 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5881ffb700 10 client.26423.objecter ms_handle_connect 0x7f587406f5e0 2024-04-02T04:53:31.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.361+0000 7f5888bf9700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f587406f5e0 0x7f5874071aa0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5870005950 tx=0x7f58700058e0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.365+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5878016330 con 0x7f5884102f30 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/1102731489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c107580 msgr2=0x7f8f9c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- 172.21.15.69:0/1102731489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c107580 0x7f8f9c10f160 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400b0d0 tx=0x7f8f9400b3e0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/1102731489 shutdown_connections 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- 172.21.15.69:0/1102731489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c107580 0x7f8f9c10f160 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- 172.21.15.69:0/1102731489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c06a1d0 0x7f8f9c06a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- 172.21.15.69:0/1102731489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06ab80 0x7f8f9c106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/1102731489 >> 172.21.15.69:0/1102731489 conn(0x7f8f9c0771e0 msgr2=0x7f8f9c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/1102731489 shutdown_connections 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/1102731489 wait complete. 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 librados: starting msgr at 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 librados: starting objecter 2024-04-02T04:53:31.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 Processor -- start 2024-04-02T04:53:31.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- start start 2024-04-02T04:53:31.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 librados: setting wanted keys 2024-04-02T04:53:31.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 librados: calling monclient init 2024-04-02T04:53:31.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06a1d0 0x7f8f9c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c06ab80 0x7f8f9c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 0x7f8f9c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c111a80 con 0x7f8f9c107580 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c111900 con 0x7f8f9c06a1d0 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f9c111c00 con 0x7f8f9c06ab80 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 0x7f8f9c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa080c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c06ab80 0x7f8f9c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 0x7f8f9c19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40634/0 (socket says 172.21.15.69:40634) 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 -- 172.21.15.69:0/3760708773 learned_addr learned my addr 172.21.15.69:0/3760708773 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa100d700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06a1d0 0x7f8f9c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 -- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c06ab80 msgr2=0x7f8f9c198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c06ab80 0x7f8f9c198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 -- 172.21.15.69:0/3760708773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06a1d0 msgr2=0x7f8f9c197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06a1d0 0x7f8f9c197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 -- 172.21.15.69:0/3760708773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9c19cef0 con 0x7f8f9c107580 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa100d700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06a1d0 0x7f8f9c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa180e700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 0x7f8f9c19c7b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8f94006070 tx=0x7f8f94007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8f927fc700 10 client.?.objecter ms_handle_connect 0x7f8f9c107580 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8f927fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94021020 con 0x7f8f9c107580 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8f94004c50 con 0x7f8f9c107580 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f9c19d180 con 0x7f8f9c107580 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94005050 con 0x7f8f9c107580 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 10 client.16332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f9c1a4a60 con 0x7f8f9c107580 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.401+0000 7f8fa3271700 1 librados: init done 2024-04-02T04:53:31.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f9c04ff00 con 0x7f8f9c107580 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8f94024020 con 0x7f8f9c107580 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f8806f5e0 0x7f8f88071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(371..371 src has 74..371) ==== 11409+0+0 (secure 0 0 0) 0x7f8f9400f020 con 0x7f8f9c107580 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 10 client.16332.objecter ms_dispatch 0x7f8f9c001550 osd_map(371..371 src has 74..371) 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 3 client.16332.objecter handle_osd_map got epochs [371,371] > 0 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 3 client.16332.objecter handle_osd_map decoding full epoch 371 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8fa100d700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f8806f5e0 0x7f8f88071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 20 client.16332.objecter dump_active .. 0 homeless 2024-04-02T04:53:31.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8f927fc700 10 client.16332.objecter ms_handle_connect 0x7f8f8806f5e0 2024-04-02T04:53:31.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.405+0000 7f8fa100d700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f8806f5e0 0x7f8f88071aa0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009cc0 tx=0x7f8f98009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.409+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8f9405c790 con 0x7f8f9c107580 2024-04-02T04:53:31.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:31.451 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:31.502 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: cluster 2024-04-02T04:53:30.016801+0000 osd.3 (osd.3) 249 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: cluster 2024-04-02T04:53:30.016960+0000 osd.3 (osd.3) 250 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: cluster 2024-04-02T04:53:30.049772+0000 mgr.y (mgr.14538) 389 : cluster 0 pgmap v532: 332 pgs: 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: audit 2024-04-02T04:53:30.112791+0000 mon.c (mon.2) 661 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: cluster 2024-04-02T04:53:30.883542+0000 mon.a (mon.0) 2408 : cluster 0 osdmap e371: 8 total, 7 up, 8 in 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: audit 2024-04-02T04:53:30.891819+0000 mon.a (mon.0) 2409 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: audit 2024-04-02T04:53:30.894967+0000 mon.c (mon.2) 662 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: audit 2024-04-02T04:53:30.896332+0000 mon.a (mon.0) 2410 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[21528]: audit 2024-04-02T04:53:31.113907+0000 mon.c (mon.2) 663 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: cluster 2024-04-02T04:53:30.016801+0000 osd.3 (osd.3) 249 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: cluster 2024-04-02T04:53:30.016960+0000 osd.3 (osd.3) 250 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:31.503 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: cluster 2024-04-02T04:53:30.049772+0000 mgr.y (mgr.14538) 389 : cluster 0 pgmap v532: 332 pgs: 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:31.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: audit 2024-04-02T04:53:30.112791+0000 mon.c (mon.2) 661 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:31.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: cluster 2024-04-02T04:53:30.883542+0000 mon.a (mon.0) 2408 : cluster 0 osdmap e371: 8 total, 7 up, 8 in 2024-04-02T04:53:31.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: audit 2024-04-02T04:53:30.891819+0000 mon.a (mon.0) 2409 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:31.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: audit 2024-04-02T04:53:30.894967+0000 mon.c (mon.2) 662 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:31.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: audit 2024-04-02T04:53:30.896332+0000 mon.a (mon.0) 2410 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:31.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:31 smithi069 bash[28904]: audit 2024-04-02T04:53:31.113907+0000 mon.c (mon.2) 663 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:31.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:31.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.605+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f58841a6fd0 con 0x7f5884102f30 2024-04-02T04:53:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.637+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7f8f9c1a4c30 con 0x7f8f9c107580 2024-04-02T04:53:31.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:31.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: cluster 2024-04-02T04:53:30.016801+0000 osd.3 (osd.3) 249 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:31.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: cluster 2024-04-02T04:53:30.016960+0000 osd.3 (osd.3) 250 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:31.695 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: cluster 2024-04-02T04:53:30.049772+0000 mgr.y (mgr.14538) 389 : cluster 0 pgmap v532: 332 pgs: 8 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:31.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: audit 2024-04-02T04:53:30.112791+0000 mon.c (mon.2) 661 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:31.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: cluster 2024-04-02T04:53:30.883542+0000 mon.a (mon.0) 2408 : cluster 0 osdmap e371: 8 total, 7 up, 8 in 2024-04-02T04:53:31.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: audit 2024-04-02T04:53:30.891819+0000 mon.a (mon.0) 2409 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:31.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: audit 2024-04-02T04:53:30.894967+0000 mon.c (mon.2) 662 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:31.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: audit 2024-04-02T04:53:30.896332+0000 mon.a (mon.0) 2410 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:31.696 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:31 smithi098 bash[23790]: audit 2024-04-02T04:53:31.113907+0000 mon.c (mon.2) 663 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:31.712 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/2189302886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 msgr2=0x7f3c54100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- 172.21.15.69:0/2189302886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 0x7f3c54100d60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800aaa0 tx=0x7f3c4800adb0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/2189302886 shutdown_connections 2024-04-02T04:53:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- 172.21.15.69:0/2189302886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c541012a0 0x7f3c5410d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- 172.21.15.69:0/2189302886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c541068e0 0x7f3c54106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- 172.21.15.69:0/2189302886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 0x7f3c54100d60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/2189302886 >> 172.21.15.69:0/2189302886 conn(0x7f3c54077740 msgr2=0x7f3c540feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/2189302886 shutdown_connections 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/2189302886 wait complete. 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 librados: starting msgr at 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 librados: starting objecter 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 Processor -- start 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- start start 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 librados: setting wanted keys 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 librados: calling monclient init 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 0x7f3c54198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c541012a0 0x7f3c54198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c541068e0 0x7f3c5419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5410f910 con 0x7f3c541068e0 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5410f790 con 0x7f3c541012a0 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3c5410fa90 con 0x7f3c541008e0 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 0x7f3c54198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 0x7f3c54198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49594/0 (socket says 172.21.15.69:49594) 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 -- 172.21.15.69:0/626485702 learned_addr learned my addr 172.21.15.69:0/626485702 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c541068e0 0x7f3c5419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c541012a0 0x7f3c54198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 -- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c541012a0 msgr2=0x7f3c54198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c541012a0 0x7f3c54198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 -- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c541068e0 msgr2=0x7f3c5419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c541068e0 0x7f3c5419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5419d030 con 0x7f3c541008e0 2024-04-02T04:53:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c541008e0 0x7f3c54198020 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400d980 tx=0x7f3c4400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c50ff9700 10 client.?.objecter ms_handle_connect 0x7f3c541008e0 2024-04-02T04:53:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c50ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44017070 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3c44005020 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5419d320 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44009990 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 10 client.26435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c541a4ba0 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 librados: init done 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3c5404ff00 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.749+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3c44004a30 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c3c06f5e0 0x7f3c3c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(371..371 src has 74..371) ==== 11409+0+0 (secure 0 0 0) 0x7f3c4408ed40 con 0x7f3c541008e0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c3c06f5e0 0x7f3c3c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 osd_map(371..371 src has 74..371) 2024-04-02T04:53:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 3 client.26435.objecter handle_osd_map got epochs [371,371] > 0 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 3 client.26435.objecter handle_osd_map decoding full epoch 371 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 20 client.26435.objecter dump_active .. 0 homeless 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v371) ==== 54+0+16 (secure 0 0 0) 0x7f3c44020020 con 0x7f3c541008e0 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 10 client.26435.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 1 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3c540630a0 0x7f3c54109750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 1 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 0 1 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54199380 osd=0 4 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 10 client.26435.objecter _send_command 1 2024-04-02T04:53:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c5410ae10 con 0x7f3c540630a0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c3c06f5e0 0x7f3c3c071aa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800c6d0 tx=0x7f3c48005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c3c06f5e0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3c540630a0 0x7f3c54109750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c540630a0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3c540630a0 0x7f3c54109750 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c5410ae10 con 0x7f3c540630a0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 1: 0 ) 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 1 = system:0 2024-04-02T04:53:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c54199380 osd=0 4 2024-04-02T04:53:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.753+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 0 1 2024-04-02T04:53:31.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 10 client.26435.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 2 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 2 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c54199380 osd=0 3 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 0 2 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54199380 osd=0 4 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 10 client.26435.objecter _send_command 2 2024-04-02T04:53:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.769+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c54109c90 con 0x7f3c540630a0 2024-04-02T04:53:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c54109c90 con 0x7f3c540630a0 2024-04-02T04:53:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c54199380 osd=0 4 2024-04-02T04:53:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 0 2 2024-04-02T04:53:31.773 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:53:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 10 client.26435.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3c54062c90 0x7f3c541a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 1 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54062b20 osd=1 4 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 10 client.26435.objecter _send_command 3 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3c5410ab80 con 0x7f3c54062c90 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3c54062c90 0x7f3c541a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c54062c90 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3c54062c90 0x7f3c541a8d90 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c5410ab80 con 0x7f3c54062c90 2024-04-02T04:53:31.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 3: 0 ) 2024-04-02T04:53:31.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 3 = system:0 2024-04-02T04:53:31.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c54062b20 osd=1 4 2024-04-02T04:53:31.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.773+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 1 3 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 10 client.26435.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 4 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 4 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c54062b20 osd=1 3 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 1 4 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c54062b20 osd=1 4 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 10 client.26435.objecter _send_command 4 2024-04-02T04:53:31.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.789+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c5410a0f0 con 0x7f3c54062c90 2024-04-02T04:53:31.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c5410a0f0 con 0x7f3c54062c90 2024-04-02T04:53:31.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c54062b20 osd=1 4 2024-04-02T04:53:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 1 4 2024-04-02T04:53:31.793 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 10 client.26435.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 5 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3c540fcab0 0x7f3c540fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 5 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 2 5 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402d180 osd=2 4 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 10 client.26435.objecter _send_command 5 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3c5402d800 con 0x7f3c540fcab0 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3c540fcab0 0x7f3c540fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c540fcab0 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3c540fcab0 0x7f3c540fcef0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c5402d800 con 0x7f3c540fcab0 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 5: 0 ) 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 5 = system:0 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c5402d180 osd=2 4 2024-04-02T04:53:31.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.793+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 2 5 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 10 client.26435.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 6 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 6 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c5402d180 osd=2 3 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 2 6 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402d180 osd=2 4 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 10 client.26435.objecter _send_command 6 2024-04-02T04:53:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c5402cc00 con 0x7f3c540fcab0 2024-04-02T04:53:31.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c5402cc00 con 0x7f3c540fcab0 2024-04-02T04:53:31.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c5402d180 osd=2 4 2024-04-02T04:53:31.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 2 6 2024-04-02T04:53:31.813 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 10 client.26435.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 7 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3c541a9aa0 0x7f3c541a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 7 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 3 7 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402cc00 osd=3 4 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 10 client.26435.objecter _send_command 7 2024-04-02T04:53:31.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3c541b51a0 con 0x7f3c541a9aa0 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3c541a9aa0 0x7f3c541a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c541a9aa0 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3c541a9aa0 0x7f3c541a9f00 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c541b51a0 con 0x7f3c541a9aa0 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 7: 0 ) 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 7 = system:0 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c5402cc00 osd=3 4 2024-04-02T04:53:31.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.813+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 3 7 2024-04-02T04:53:31.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 10 client.26435.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 8 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 8 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c5402cc00 osd=3 3 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 3 8 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c5402cc00 osd=3 4 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 10 client.26435.objecter _send_command 8 2024-04-02T04:53:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c541a9520 con 0x7f3c541a9aa0 2024-04-02T04:53:31.833 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:31.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c541a9520 con 0x7f3c541a9aa0 2024-04-02T04:53:31.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c5402cc00 osd=3 4 2024-04-02T04:53:31.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 3 8 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 10 client.26435.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 9 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3c541a9690 0x7f3c541b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 9 2024-04-02T04:53:31.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 4 9 2024-04-02T04:53:31.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541a9520 osd=4 4 2024-04-02T04:53:31.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 10 client.26435.objecter _send_command 9 2024-04-02T04:53:31.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.833+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3c540fe110 con 0x7f3c541a9690 2024-04-02T04:53:31.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3c541a9690 0x7f3c541b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:31.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c541a9690 2024-04-02T04:53:31.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c537fe700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3c541a9690 0x7f3c541b5a10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c540fe110 con 0x7f3c541a9690 2024-04-02T04:53:31.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 9: 0 ) 2024-04-02T04:53:31.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 9 = system:0 2024-04-02T04:53:31.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c541a9520 osd=4 4 2024-04-02T04:53:31.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.837+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 4 9 2024-04-02T04:53:31.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 10 client.26435.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 10 2024-04-02T04:53:31.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 10 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c541a9520 osd=4 3 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 4 10 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541a9520 osd=4 4 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 10 client.26435.objecter _send_command 10 2024-04-02T04:53:31.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.853+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c540fe110 con 0x7f3c541a9690 2024-04-02T04:53:31.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c540fe110 con 0x7f3c541a9690 2024-04-02T04:53:31.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c541a9520 osd=4 4 2024-04-02T04:53:31.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 4 10 2024-04-02T04:53:31.859 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 10 client.26435.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 11 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3c541b6460 0x7f3c541b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 11 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 5 11 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c540fe110 osd=5 4 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 10 client.26435.objecter _send_command 11 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3c541bb6b0 con 0x7f3c541b6460 2024-04-02T04:53:31.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3c541b6460 0x7f3c541b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c541b6460 2024-04-02T04:53:31.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c52ffd700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3c541b6460 0x7f3c541b8850 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c541bb6b0 con 0x7f3c541b6460 2024-04-02T04:53:31.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 11: 0 ) 2024-04-02T04:53:31.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 11 = system:0 2024-04-02T04:53:31.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c540fe110 osd=5 4 2024-04-02T04:53:31.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.861+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 5 11 2024-04-02T04:53:31.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.877+0000 7f8f927fc700 1 -- 172.21.15.69:0/3760708773 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v372) ==== 129+0+0 (secure 0 0 0) 0x7f8f940601e0 con 0x7f8f9c107580 2024-04-02T04:53:31.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.877+0000 7f8fa3271700 10 librados: watch_flush enter 2024-04-02T04:53:31.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 10 librados: watch_flush exit 2024-04-02T04:53:31.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 20 client.16332.objecter shutdown clearing up homeless session... 2024-04-02T04:53:31.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 10 client.16332.objecter successfully canceled tick 2024-04-02T04:53:31.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f8806f5e0 msgr2=0x7f8f88071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f8806f5e0 0x7f8f88071aa0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009cc0 tx=0x7f8f98009260 comp rx=0 tx=0).stop 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 msgr2=0x7f8f9c19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 0x7f8f9c19c7b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8f94006070 tx=0x7f8f94007a20 comp rx=0 tx=0).stop 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 shutdown_connections 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f8806f5e0 0x7f8f88071aa0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c107580 0x7f8f9c19c7b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c06a1d0 0x7f8f9c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 --2- 172.21.15.69:0/3760708773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f9c06ab80 0x7f8f9c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 >> 172.21.15.69:0/3760708773 conn(0x7f8f9c0771e0 msgr2=0x7f8f9c0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 shutdown_connections 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 -- 172.21.15.69:0/3760708773 wait complete. 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f8fa3271700 1 librados: shutdown 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 10 client.26435.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 12 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 12 2024-04-02T04:53:31.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c540fe110 osd=5 3 2024-04-02T04:53:31.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 5 12 2024-04-02T04:53:31.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c540fe110 osd=5 4 2024-04-02T04:53:31.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 10 client.26435.objecter _send_command 12 2024-04-02T04:53:31.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.881+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c541b8d90 con 0x7f3c541b6460 2024-04-02T04:53:31.886 INFO:teuthology.orchestra.run.smithi069.stderr:set pool 1 pgp_num to 1 2024-04-02T04:53:31.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.885+0000 7f5881ffb700 1 -- 172.21.15.69:0/703428386 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v372) ==== 97+0+0 (secure 0 0 0) 0x7f587805b860 con 0x7f5884102f30 2024-04-02T04:53:31.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.889+0000 7f588ae5d700 10 librados: watch_flush enter 2024-04-02T04:53:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c541b8d90 con 0x7f3c541b6460 2024-04-02T04:53:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c540fe110 osd=5 4 2024-04-02T04:53:31.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 5 12 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 10 librados: watch_flush exit 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 20 client.26423.objecter shutdown clearing up homeless session... 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 10 client.26423.objecter successfully canceled tick 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f587406f5e0 msgr2=0x7f5874071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f587406f5e0 0x7f5874071aa0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5870005950 tx=0x7f58700058e0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 msgr2=0x7f5884074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.893+0000 7f588ae5d700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 0x7f5884074a80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f587800ea30 tx=0x7f587800ed40 comp rx=0 tx=0).stop 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 shutdown_connections 2024-04-02T04:53:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f587406f5e0 0x7f5874071aa0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58841075b0 0x7f5884074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5884102580 0x7f5884074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 --2- 172.21.15.69:0/703428386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5884102f30 0x7f5884074a80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 >> 172.21.15.69:0/703428386 conn(0x7f58840793a0 msgr2=0x7f5884100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 shutdown_connections 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 -- 172.21.15.69:0/703428386 wait complete. 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.897+0000 7f588ae5d700 1 librados: shutdown 2024-04-02T04:53:31.899 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:53:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 10 client.26435.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 13 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3c541b9410 0x7f3c541bb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 13 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 6 13 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541b8d90 osd=6 4 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 10 client.26435.objecter _send_command 13 2024-04-02T04:53:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.901+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3c541be940 con 0x7f3c541b9410 2024-04-02T04:53:31.902 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:53:31.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.905+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3c541b9410 0x7f3c541bb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:31.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.917+0000 7f3c53fff700 1 --2- 172.21.15.69:0/626485702 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3c541b9410 0x7f3c541bb880 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:31.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.917+0000 7f3c50ff9700 10 client.26435.objecter ms_handle_connect 0x7f3c541b9410 2024-04-02T04:53:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.925+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c541be940 con 0x7f3c541b9410 2024-04-02T04:53:31.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.925+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 13: 0 ) 2024-04-02T04:53:31.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.925+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 13 = system:0 2024-04-02T04:53:31.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.925+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c541b8d90 osd=6 4 2024-04-02T04:53:31.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.925+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 6 13 2024-04-02T04:53:31.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:31.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 10 client.26435.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:53:31.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 14 2024-04-02T04:53:31.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 20 client.26435.objecter _get_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_remove -1 14 2024-04-02T04:53:31.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 20 client.26435.objecter get_session s=0x7f3c541b8d90 osd=6 3 2024-04-02T04:53:31.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign 6 14 2024-04-02T04:53:31.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 20 client.26435.objecter put_session s=0x7f3c541b8d90 osd=6 4 2024-04-02T04:53:31.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 10 client.26435.objecter _send_command 14 2024-04-02T04:53:31.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.945+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c541bd6d0 con 0x7f3c541b9410 2024-04-02T04:53:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3c541bd6d0 con 0x7f3c541b9410 2024-04-02T04:53:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 10 client.26435.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 20 client.26435.objecter put_session s=0x7f3c541b8d90 osd=6 4 2024-04-02T04:53:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 15 client.26435.objecter _session_command_op_remove 6 14 2024-04-02T04:53:31.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c59c99700 10 client.26435.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c59c99700 15 client.26435.objecter _session_command_op_assign -1 15 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c59c99700 20 client.26435.objecter _get_session osd=-1 returning homeless 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c59c99700 10 client.26435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=372}) -- 0x7f3c541bd910 con 0x7f3c541008e0 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c59c99700 1 -- 172.21.15.69:0/626485702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c541bd6d0 con 0x7f3c541008e0 2024-04-02T04:53:31.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(372..372 src has 74..372) ==== 1146+0+0 (secure 0 0 0) 0x7f3c4408bce0 con 0x7f3c541008e0 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 10 client.26435.objecter ms_dispatch 0x7f3c54001550 osd_map(372..372 src has 74..372) 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 3 client.26435.objecter handle_osd_map got epochs [372,372] > 371 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 3 client.26435.objecter handle_osd_map decoding incremental epoch 372 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 10 client.26435.objecter checking command 15 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 10 client.26435.objecter _check_command_map_dne tid 15 current 372 map_dne_bound 0 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 20 client.26435.objecter dump_active .. 1 homeless 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c50ff9700 1 -- 172.21.15.69:0/626485702 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_get_version_reply(handle=1 version=372) ==== 24+0+0 (secure 0 0 0) 0x7f3c4405aa80 con 0x7f3c541008e0 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c51ffb700 10 client.26435.objecter _check_command_map_dne tid 15 current 372 map_dne_bound 372 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c51ffb700 10 client.26435.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:31.949+0000 7f3c51ffb700 15 client.26435.objecter _session_command_op_remove -1 15 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:53:31.950 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:53:31.951 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:53:31.951 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:53:31.951 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:53:31.951 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:53:31.951 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:53:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:31.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:32.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:32.076 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:32.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:32.203 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:32.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:32.330 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:32.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:32.460 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:32.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: cluster 2024-04-02T04:53:31.064906+0000 osd.6 (osd.6) 181 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: cluster 2024-04-02T04:53:31.065039+0000 osd.6 (osd.6) 182 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.611420+0000 mon.c (mon.2) 664 : audit 1 from='client.? 172.21.15.69:0/703428386' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.611861+0000 mon.a (mon.0) 2411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.643410+0000 mon.a (mon.0) 2412 : audit 1 from='client.? 172.21.15.69:0/3760708773' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.755967+0000 mon.c (mon.2) 665 : audit 0 from='client.? 172.21.15.69:0/626485702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: cluster 2024-04-02T04:53:31.879385+0000 mon.a (mon.0) 2413 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.881051+0000 mon.a (mon.0) 2414 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.881117+0000 mon.a (mon.0) 2415 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:32.593 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.881182+0000 mon.a (mon.0) 2416 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:53:32.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.881252+0000 mon.a (mon.0) 2417 : audit 1 from='client.? 172.21.15.69:0/3760708773' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-02T04:53:32.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: cluster 2024-04-02T04:53:31.887506+0000 mon.a (mon.0) 2418 : cluster 0 osdmap e372: 8 total, 7 up, 8 in 2024-04-02T04:53:32.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.889812+0000 mon.c (mon.2) 666 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:32.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.890838+0000 mon.a (mon.0) 2419 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:32.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:31.891260+0000 mon.a (mon.0) 2420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:32.594 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:32 smithi098 bash[23790]: audit 2024-04-02T04:53:32.114930+0000 mon.c (mon.2) 667 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:32.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:32.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: cluster 2024-04-02T04:53:31.064906+0000 osd.6 (osd.6) 181 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:32.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: cluster 2024-04-02T04:53:31.065039+0000 osd.6 (osd.6) 182 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:32.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.611420+0000 mon.c (mon.2) 664 : audit 1 from='client.? 172.21.15.69:0/703428386' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:53:32.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.611861+0000 mon.a (mon.0) 2411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:53:32.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.643410+0000 mon.a (mon.0) 2412 : audit 1 from='client.? 172.21.15.69:0/3760708773' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:53:32.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.755967+0000 mon.c (mon.2) 665 : audit 0 from='client.? 172.21.15.69:0/626485702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: cluster 2024-04-02T04:53:31.879385+0000 mon.a (mon.0) 2413 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.881051+0000 mon.a (mon.0) 2414 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.881117+0000 mon.a (mon.0) 2415 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.881182+0000 mon.a (mon.0) 2416 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.881252+0000 mon.a (mon.0) 2417 : audit 1 from='client.? 172.21.15.69:0/3760708773' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: cluster 2024-04-02T04:53:31.887506+0000 mon.a (mon.0) 2418 : cluster 0 osdmap e372: 8 total, 7 up, 8 in 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.889812+0000 mon.c (mon.2) 666 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.890838+0000 mon.a (mon.0) 2419 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:31.891260+0000 mon.a (mon.0) 2420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[21528]: audit 2024-04-02T04:53:32.114930+0000 mon.c (mon.2) 667 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:32.708 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: cluster 2024-04-02T04:53:31.064906+0000 osd.6 (osd.6) 181 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: cluster 2024-04-02T04:53:31.065039+0000 osd.6 (osd.6) 182 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.611420+0000 mon.c (mon.2) 664 : audit 1 from='client.? 172.21.15.69:0/703428386' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.611861+0000 mon.a (mon.0) 2411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.643410+0000 mon.a (mon.0) 2412 : audit 1 from='client.? 172.21.15.69:0/3760708773' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.755967+0000 mon.c (mon.2) 665 : audit 0 from='client.? 172.21.15.69:0/626485702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: cluster 2024-04-02T04:53:31.879385+0000 mon.a (mon.0) 2413 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.881051+0000 mon.a (mon.0) 2414 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.881117+0000 mon.a (mon.0) 2415 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.881182+0000 mon.a (mon.0) 2416 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.881252+0000 mon.a (mon.0) 2417 : audit 1 from='client.? 172.21.15.69:0/3760708773' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: cluster 2024-04-02T04:53:31.887506+0000 mon.a (mon.0) 2418 : cluster 0 osdmap e372: 8 total, 7 up, 8 in 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.889812+0000 mon.c (mon.2) 666 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:32.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.890838+0000 mon.a (mon.0) 2419 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]: dispatch 2024-04-02T04:53:32.710 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:31.891260+0000 mon.a (mon.0) 2420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:32.710 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:32 smithi069 bash[28904]: audit 2024-04-02T04:53:32.114930+0000 mon.c (mon.2) 667 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:32.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:32.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:32.852 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:32.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:32.977 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:32.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:33.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:33.105 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:33.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:33.235 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:33.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:33.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:53:33.341 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: cluster 2024-04-02T04:53:32.050642+0000 mgr.y (mgr.14538) 390 : cluster 0 pgmap v535: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.860542+0000 mon.b (mon.1) 206 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.889714+0000 mon.a (mon.0) 2421 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.889788+0000 mon.a (mon.0) 2422 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: cluster 2024-04-02T04:53:32.893781+0000 mon.a (mon.0) 2423 : cluster 0 osdmap e373: 8 total, 7 up, 8 in 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.895175+0000 mon.c (mon.2) 668 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.898640+0000 mon.a (mon.0) 2424 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.918369+0000 mon.c (mon.2) 669 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.918704+0000 mon.a (mon.0) 2425 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.919384+0000 mon.c (mon.2) 670 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.919665+0000 mon.a (mon.0) 2426 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.920173+0000 mon.c (mon.2) 671 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:33.342 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:32.920449+0000 mon.a (mon.0) 2427 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:33.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:33 smithi098 bash[23790]: audit 2024-04-02T04:53:33.116037+0000 mon.c (mon.2) 672 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:33.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:33.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: cluster 2024-04-02T04:53:32.050642+0000 mgr.y (mgr.14538) 390 : cluster 0 pgmap v535: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:33.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.860542+0000 mon.b (mon.1) 206 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:33.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.889714+0000 mon.a (mon.0) 2421 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.889788+0000 mon.a (mon.0) 2422 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: cluster 2024-04-02T04:53:32.893781+0000 mon.a (mon.0) 2423 : cluster 0 osdmap e373: 8 total, 7 up, 8 in 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.895175+0000 mon.c (mon.2) 668 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.898640+0000 mon.a (mon.0) 2424 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.918369+0000 mon.c (mon.2) 669 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.918704+0000 mon.a (mon.0) 2425 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.919384+0000 mon.c (mon.2) 670 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.919665+0000 mon.a (mon.0) 2426 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.920173+0000 mon.c (mon.2) 671 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:32.920449+0000 mon.a (mon.0) 2427 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[21528]: audit 2024-04-02T04:53:33.116037+0000 mon.c (mon.2) 672 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: cluster 2024-04-02T04:53:32.050642+0000 mgr.y (mgr.14538) 390 : cluster 0 pgmap v535: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:33.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.860542+0000 mon.b (mon.1) 206 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.889714+0000 mon.a (mon.0) 2421 : audit 1 from='client.? 172.21.15.69:0/616633040' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-72153-57"}]': finished 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.889788+0000 mon.a (mon.0) 2422 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: cluster 2024-04-02T04:53:32.893781+0000 mon.a (mon.0) 2423 : cluster 0 osdmap e373: 8 total, 7 up, 8 in 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.895175+0000 mon.c (mon.2) 668 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.898640+0000 mon.a (mon.0) 2424 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.918369+0000 mon.c (mon.2) 669 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.918704+0000 mon.a (mon.0) 2425 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.919384+0000 mon.c (mon.2) 670 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.919665+0000 mon.a (mon.0) 2426 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.920173+0000 mon.c (mon.2) 671 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:32.920449+0000 mon.a (mon.0) 2427 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:33.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:33 smithi069 bash[28904]: audit 2024-04-02T04:53:33.116037+0000 mon.c (mon.2) 672 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:33.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:33.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:33.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:33.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:33.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:33.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:34.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:34.144 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:34.246 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:34.272 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:34.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:34.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:34.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:34.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:34.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.203437+0000 mgr.y (mgr.14538) 391 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.892433+0000 mon.a (mon.0) 2428 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]': finished 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.892739+0000 mon.a (mon.0) 2429 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: cluster 2024-04-02T04:53:33.904043+0000 mon.a (mon.0) 2430 : cluster 0 osdmap e374: 8 total, 7 up, 8 in 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.905847+0000 mon.c (mon.2) 673 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.906471+0000 mon.c (mon.2) 674 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.907713+0000 mon.a (mon.0) 2431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:33.908739+0000 mon.a (mon.0) 2432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T04:53:34.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:34 smithi098 bash[23790]: audit 2024-04-02T04:53:34.117818+0000 mon.c (mon.2) 675 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:34.659 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:34.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:34.788 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.203437+0000 mgr.y (mgr.14538) 391 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.892433+0000 mon.a (mon.0) 2428 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]': finished 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.892739+0000 mon.a (mon.0) 2429 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: cluster 2024-04-02T04:53:33.904043+0000 mon.a (mon.0) 2430 : cluster 0 osdmap e374: 8 total, 7 up, 8 in 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.905847+0000 mon.c (mon.2) 673 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.906471+0000 mon.c (mon.2) 674 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.907713+0000 mon.a (mon.0) 2431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:33.908739+0000 mon.a (mon.0) 2432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[21528]: audit 2024-04-02T04:53:34.117818+0000 mon.c (mon.2) 675 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:34.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.203437+0000 mgr.y (mgr.14538) 391 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.892433+0000 mon.a (mon.0) 2428 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_count","val": "3"}]': finished 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.892739+0000 mon.a (mon.0) 2429 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-72153-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: cluster 2024-04-02T04:53:33.904043+0000 mon.a (mon.0) 2430 : cluster 0 osdmap e374: 8 total, 7 up, 8 in 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.905847+0000 mon.c (mon.2) 673 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.906471+0000 mon.c (mon.2) 674 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.907713+0000 mon.a (mon.0) 2431 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:33.908739+0000 mon.a (mon.0) 2432 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T04:53:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:34 smithi069 bash[28904]: audit 2024-04-02T04:53:34.117818+0000 mon.c (mon.2) 675 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:34.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:34.917 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:34.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:35.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:35.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:35.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:35.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: cluster 2024-04-02T04:53:34.051944+0000 mgr.y (mgr.14538) 392 : cluster 0 pgmap v538: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: cluster 2024-04-02T04:53:34.848967+0000 mon.a (mon.0) 2433 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: audit 2024-04-02T04:53:34.852306+0000 mon.a (mon.0) 2434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]': finished 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: audit 2024-04-02T04:53:34.865321+0000 mon.c (mon.2) 676 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: audit 2024-04-02T04:53:35.119607+0000 mon.c (mon.2) 677 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: cluster 2024-04-02T04:53:35.301255+0000 mon.a (mon.0) 2435 : cluster 0 osdmap e375: 8 total, 7 up, 8 in 2024-04-02T04:53:35.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:35 smithi098 bash[23790]: audit 2024-04-02T04:53:35.302421+0000 mon.a (mon.0) 2436 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:35.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: cluster 2024-04-02T04:53:34.051944+0000 mgr.y (mgr.14538) 392 : cluster 0 pgmap v538: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:35.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: cluster 2024-04-02T04:53:34.848967+0000 mon.a (mon.0) 2433 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:35.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: audit 2024-04-02T04:53:34.852306+0000 mon.a (mon.0) 2434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]': finished 2024-04-02T04:53:35.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: audit 2024-04-02T04:53:34.865321+0000 mon.c (mon.2) 676 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:35.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: audit 2024-04-02T04:53:35.119607+0000 mon.c (mon.2) 677 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:35.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: cluster 2024-04-02T04:53:35.301255+0000 mon.a (mon.0) 2435 : cluster 0 osdmap e375: 8 total, 7 up, 8 in 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[21528]: audit 2024-04-02T04:53:35.302421+0000 mon.a (mon.0) 2436 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: cluster 2024-04-02T04:53:34.051944+0000 mgr.y (mgr.14538) 392 : cluster 0 pgmap v538: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: cluster 2024-04-02T04:53:34.848967+0000 mon.a (mon.0) 2433 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: audit 2024-04-02T04:53:34.852306+0000 mon.a (mon.0) 2434 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_period","val": "3"}]': finished 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: audit 2024-04-02T04:53:34.865321+0000 mon.c (mon.2) 676 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: audit 2024-04-02T04:53:35.119607+0000 mon.c (mon.2) 677 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: cluster 2024-04-02T04:53:35.301255+0000 mon.a (mon.0) 2435 : cluster 0 osdmap e375: 8 total, 7 up, 8 in 2024-04-02T04:53:35.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:35 smithi069 bash[28904]: audit 2024-04-02T04:53:35.302421+0000 mon.a (mon.0) 2436 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:35.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:35.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:35.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:35.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:35.851 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:35.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:35.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:36.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:36.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:36.236 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:36.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:36.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:53:36.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:53:36.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:36.490 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:36.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:36.614 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:36.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:36.740 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:36.744 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:53:36.744 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:53:36.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:36.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:36.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:37.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: audit 2024-04-02T04:53:35.855278+0000 mon.a (mon.0) 2437 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:37.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: audit 2024-04-02T04:53:35.855505+0000 mon.a (mon.0) 2438 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: cluster 2024-04-02T04:53:35.864378+0000 mon.a (mon.0) 2439 : cluster 0 osdmap e376: 8 total, 7 up, 8 in 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: audit 2024-04-02T04:53:35.866006+0000 mon.c (mon.2) 678 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: audit 2024-04-02T04:53:35.880386+0000 mon.a (mon.0) 2440 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: cluster 2024-04-02T04:53:36.053804+0000 mgr.y (mgr.14538) 393 : cluster 0 pgmap v541: 364 pgs: 8 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[21528]: audit 2024-04-02T04:53:36.121524+0000 mon.c (mon.2) 679 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: audit 2024-04-02T04:53:35.855278+0000 mon.a (mon.0) 2437 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: audit 2024-04-02T04:53:35.855505+0000 mon.a (mon.0) 2438 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: cluster 2024-04-02T04:53:35.864378+0000 mon.a (mon.0) 2439 : cluster 0 osdmap e376: 8 total, 7 up, 8 in 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: audit 2024-04-02T04:53:35.866006+0000 mon.c (mon.2) 678 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: audit 2024-04-02T04:53:35.880386+0000 mon.a (mon.0) 2440 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: cluster 2024-04-02T04:53:36.053804+0000 mgr.y (mgr.14538) 393 : cluster 0 pgmap v541: 364 pgs: 8 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:37.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:36 smithi069 bash[28904]: audit 2024-04-02T04:53:36.121524+0000 mon.c (mon.2) 679 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:37.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.142 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: audit 2024-04-02T04:53:35.855278+0000 mon.a (mon.0) 2437 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-72153-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: audit 2024-04-02T04:53:35.855505+0000 mon.a (mon.0) 2438 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: cluster 2024-04-02T04:53:35.864378+0000 mon.a (mon.0) 2439 : cluster 0 osdmap e376: 8 total, 7 up, 8 in 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: audit 2024-04-02T04:53:35.866006+0000 mon.c (mon.2) 678 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: audit 2024-04-02T04:53:35.880386+0000 mon.a (mon.0) 2440 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: cluster 2024-04-02T04:53:36.053804+0000 mgr.y (mgr.14538) 393 : cluster 0 pgmap v541: 364 pgs: 8 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:36 smithi098 bash[23790]: audit 2024-04-02T04:53:36.121524+0000 mon.c (mon.2) 679 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:37.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:37.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:37.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:37.404 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:37.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:37.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:37.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-02T04:53:37.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.788 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:37.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:37.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[21528]: cluster 2024-04-02T04:53:36.855216+0000 mon.a (mon.0) 2441 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:37.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[21528]: audit 2024-04-02T04:53:36.864571+0000 mon.a (mon.0) 2442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-02T04:53:37.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[21528]: cluster 2024-04-02T04:53:36.871785+0000 mon.a (mon.0) 2443 : cluster 0 osdmap e377: 8 total, 7 up, 8 in 2024-04-02T04:53:37.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[21528]: audit 2024-04-02T04:53:37.123307+0000 mon.c (mon.2) 680 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:37.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[28904]: cluster 2024-04-02T04:53:36.855216+0000 mon.a (mon.0) 2441 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:37.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[28904]: audit 2024-04-02T04:53:36.864571+0000 mon.a (mon.0) 2442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-02T04:53:37.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[28904]: cluster 2024-04-02T04:53:36.871785+0000 mon.a (mon.0) 2443 : cluster 0 osdmap e377: 8 total, 7 up, 8 in 2024-04-02T04:53:37.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:37 smithi069 bash[28904]: audit 2024-04-02T04:53:37.123307+0000 mon.c (mon.2) 680 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:37.914 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:37.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:37.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:38.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:38.043 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:38.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:38.172 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:38.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:37 smithi098 bash[23790]: cluster 2024-04-02T04:53:36.855216+0000 mon.a (mon.0) 2441 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:38.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:37 smithi098 bash[23790]: audit 2024-04-02T04:53:36.864571+0000 mon.a (mon.0) 2442 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-02T04:53:38.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:37 smithi098 bash[23790]: cluster 2024-04-02T04:53:36.871785+0000 mon.a (mon.0) 2443 : cluster 0 osdmap e377: 8 total, 7 up, 8 in 2024-04-02T04:53:38.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:37 smithi098 bash[23790]: audit 2024-04-02T04:53:37.123307+0000 mon.c (mon.2) 680 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:38.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:38.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:38.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:53:38.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:53:38.434 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:38.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:38.563 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:38.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:38.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:38.821 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:38.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:38.951 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:38.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:38.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:38 smithi098 bash[23790]: cluster 2024-04-02T04:53:37.870396+0000 mon.a (mon.0) 2444 : cluster 0 osdmap e378: 8 total, 7 up, 8 in 2024-04-02T04:53:38.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:38 smithi098 bash[23790]: audit 2024-04-02T04:53:37.884559+0000 mon.c (mon.2) 681 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:38.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:38 smithi098 bash[23790]: audit 2024-04-02T04:53:37.885261+0000 mon.a (mon.0) 2445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:38.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:38 smithi098 bash[23790]: cluster 2024-04-02T04:53:38.057816+0000 mgr.y (mgr.14538) 394 : cluster 0 pgmap v544: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:38.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:38 smithi098 bash[23790]: audit 2024-04-02T04:53:38.125164+0000 mon.c (mon.2) 682 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:39.027 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:53:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:53:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[21528]: cluster 2024-04-02T04:53:37.870396+0000 mon.a (mon.0) 2444 : cluster 0 osdmap e378: 8 total, 7 up, 8 in 2024-04-02T04:53:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[21528]: audit 2024-04-02T04:53:37.884559+0000 mon.c (mon.2) 681 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:39.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[21528]: audit 2024-04-02T04:53:37.885261+0000 mon.a (mon.0) 2445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[21528]: cluster 2024-04-02T04:53:38.057816+0000 mgr.y (mgr.14538) 394 : cluster 0 pgmap v544: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[21528]: audit 2024-04-02T04:53:38.125164+0000 mon.c (mon.2) 682 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[28904]: cluster 2024-04-02T04:53:37.870396+0000 mon.a (mon.0) 2444 : cluster 0 osdmap e378: 8 total, 7 up, 8 in 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[28904]: audit 2024-04-02T04:53:37.884559+0000 mon.c (mon.2) 681 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[28904]: audit 2024-04-02T04:53:37.885261+0000 mon.a (mon.0) 2445 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[28904]: cluster 2024-04-02T04:53:38.057816+0000 mgr.y (mgr.14538) 394 : cluster 0 pgmap v544: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:39.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:38 smithi069 bash[28904]: audit 2024-04-02T04:53:38.125164+0000 mon.c (mon.2) 682 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:39.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:39.079 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:39.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:39.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:39.338 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:39.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:39.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:39.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:39.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:39.732 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:39.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:39.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:39.988 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:39.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:40.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:40.102 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:39 smithi098 bash[23790]: audit 2024-04-02T04:53:38.891106+0000 mon.a (mon.0) 2446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:40.102 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:39 smithi098 bash[23790]: cluster 2024-04-02T04:53:38.899041+0000 mon.a (mon.0) 2447 : cluster 0 osdmap e379: 8 total, 7 up, 8 in 2024-04-02T04:53:40.102 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:39 smithi098 bash[23790]: audit 2024-04-02T04:53:38.906731+0000 mon.c (mon.2) 683 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:40.102 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:39 smithi098 bash[23790]: audit 2024-04-02T04:53:38.908544+0000 mon.a (mon.0) 2448 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:40.102 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:39 smithi098 bash[23790]: audit 2024-04-02T04:53:39.126898+0000 mon.c (mon.2) 684 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:40.102 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:39 smithi098 bash[23790]: cluster 2024-04-02T04:53:39.851446+0000 mon.a (mon.0) 2449 : cluster 3 Health check update: Degraded data redundancy: 278/1545 objects degraded (17.994%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:40.115 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:40.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:40.229 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[21528]: audit 2024-04-02T04:53:38.891106+0000 mon.a (mon.0) 2446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[21528]: cluster 2024-04-02T04:53:38.899041+0000 mon.a (mon.0) 2447 : cluster 0 osdmap e379: 8 total, 7 up, 8 in 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[21528]: audit 2024-04-02T04:53:38.906731+0000 mon.c (mon.2) 683 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[21528]: audit 2024-04-02T04:53:38.908544+0000 mon.a (mon.0) 2448 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[21528]: audit 2024-04-02T04:53:39.126898+0000 mon.c (mon.2) 684 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[21528]: cluster 2024-04-02T04:53:39.851446+0000 mon.a (mon.0) 2449 : cluster 3 Health check update: Degraded data redundancy: 278/1545 objects degraded (17.994%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[28904]: audit 2024-04-02T04:53:38.891106+0000 mon.a (mon.0) 2446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[28904]: cluster 2024-04-02T04:53:38.899041+0000 mon.a (mon.0) 2447 : cluster 0 osdmap e379: 8 total, 7 up, 8 in 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[28904]: audit 2024-04-02T04:53:38.906731+0000 mon.c (mon.2) 683 : audit 1 from='client.? 172.21.15.69:0/928542033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[28904]: audit 2024-04-02T04:53:38.908544+0000 mon.a (mon.0) 2448 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]: dispatch 2024-04-02T04:53:40.230 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[28904]: audit 2024-04-02T04:53:39.126898+0000 mon.c (mon.2) 684 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:40.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:39 smithi069 bash[28904]: cluster 2024-04-02T04:53:39.851446+0000 mon.a (mon.0) 2449 : cluster 3 Health check update: Degraded data redundancy: 278/1545 objects degraded (17.994%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:40.245 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:40.278 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-04-02T04:53:40.278 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:53:40.278 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-04-02T04:53:40.278 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-04-02T04:53:40.279 DEBUG:teuthology.orchestra.run.smithi098:osd.7> sudo journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7.service 2024-04-02T04:53:40.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl start ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7 2024-04-02T04:53:40.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-02T04:53:40.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:40.391 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:40 smithi098 systemd[1]: Started Ceph osd.7 for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T04:53:40.407 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:40.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:40.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:40.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:40.661 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:40.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:40.791 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:40.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:40.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:41.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.941007+0000 mon.a (mon.0) 2450 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: cluster 2024-04-02T04:53:39.951867+0000 mon.a (mon.0) 2451 : cluster 0 osdmap e380: 8 total, 7 up, 8 in 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.984464+0000 mon.c (mon.2) 685 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.985026+0000 mon.a (mon.0) 2452 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.985635+0000 mon.c (mon.2) 686 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.985953+0000 mon.a (mon.0) 2453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.986481+0000 mon.c (mon.2) 687 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:39.986811+0000 mon.a (mon.0) 2454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: cluster 2024-04-02T04:53:40.058818+0000 mgr.y (mgr.14538) 395 : cluster 0 pgmap v547: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:41.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[28904]: audit 2024-04-02T04:53:40.128849+0000 mon.c (mon.2) 688 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.941007+0000 mon.a (mon.0) 2450 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: cluster 2024-04-02T04:53:39.951867+0000 mon.a (mon.0) 2451 : cluster 0 osdmap e380: 8 total, 7 up, 8 in 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.984464+0000 mon.c (mon.2) 685 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.985026+0000 mon.a (mon.0) 2452 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.985635+0000 mon.c (mon.2) 686 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.985953+0000 mon.a (mon.0) 2453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.986481+0000 mon.c (mon.2) 687 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:39.986811+0000 mon.a (mon.0) 2454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: cluster 2024-04-02T04:53:40.058818+0000 mgr.y (mgr.14538) 395 : cluster 0 pgmap v547: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:41.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:40 smithi069 bash[21528]: audit 2024-04-02T04:53:40.128849+0000 mon.c (mon.2) 688 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:41.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:41.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:41.181 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.941007+0000 mon.a (mon.0) 2450 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-72153-58"}]': finished 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: cluster 2024-04-02T04:53:39.951867+0000 mon.a (mon.0) 2451 : cluster 0 osdmap e380: 8 total, 7 up, 8 in 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.984464+0000 mon.c (mon.2) 685 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.985026+0000 mon.a (mon.0) 2452 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.985635+0000 mon.c (mon.2) 686 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.985953+0000 mon.a (mon.0) 2453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:41.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.986481+0000 mon.c (mon.2) 687 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:39.986811+0000 mon.a (mon.0) 2454 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: cluster 2024-04-02T04:53:40.058818+0000 mgr.y (mgr.14538) 395 : cluster 0 pgmap v547: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:40 smithi098 bash[23790]: audit 2024-04-02T04:53:40.128849+0000 mon.c (mon.2) 688 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:41.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:41.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:41.441 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:41.544 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:41.570 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:41.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:41.702 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:41.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:41.832 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:41.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:41.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:42.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:42.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:42.201 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T04:53:42.202 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[69281]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-02T04:53:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:42.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:42.330 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:42.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:42.460 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:42.522 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:42.522 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:42.522 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:42.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[28904]: audit 2024-04-02T04:53:40.976217+0000 mon.a (mon.0) 2455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[28904]: cluster 2024-04-02T04:53:40.984415+0000 mon.a (mon.0) 2456 : cluster 0 osdmap e381: 8 total, 7 up, 8 in 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[28904]: audit 2024-04-02T04:53:40.989343+0000 mon.c (mon.2) 689 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[28904]: audit 2024-04-02T04:53:40.991287+0000 mon.a (mon.0) 2457 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[28904]: audit 2024-04-02T04:53:41.130729+0000 mon.c (mon.2) 690 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[21528]: audit 2024-04-02T04:53:40.976217+0000 mon.a (mon.0) 2455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[21528]: cluster 2024-04-02T04:53:40.984415+0000 mon.a (mon.0) 2456 : cluster 0 osdmap e381: 8 total, 7 up, 8 in 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[21528]: audit 2024-04-02T04:53:40.989343+0000 mon.c (mon.2) 689 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[21528]: audit 2024-04-02T04:53:40.991287+0000 mon.a (mon.0) 2457 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:42.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:42 smithi069 bash[21528]: audit 2024-04-02T04:53:41.130729+0000 mon.c (mon.2) 690 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:42.581 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:42.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:42.711 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:42.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:42.855 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[23790]: audit 2024-04-02T04:53:40.976217+0000 mon.a (mon.0) 2455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:42.855 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[23790]: cluster 2024-04-02T04:53:40.984415+0000 mon.a (mon.0) 2456 : cluster 0 osdmap e381: 8 total, 7 up, 8 in 2024-04-02T04:53:42.855 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[23790]: audit 2024-04-02T04:53:40.989343+0000 mon.c (mon.2) 689 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:42.855 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[23790]: audit 2024-04-02T04:53:40.991287+0000 mon.a (mon.0) 2457 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:42.855 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:42 smithi098 bash[23790]: audit 2024-04-02T04:53:41.130729+0000 mon.c (mon.2) 690 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:42.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:42.964 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:42.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:43.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:43.092 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:43.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:43.222 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:43.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:43.352 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:43.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:43.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:53:43.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[23790]: cluster 2024-04-02T04:53:42.060543+0000 mgr.y (mgr.14538) 396 : cluster 0 pgmap v550: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:43.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[23790]: audit 2024-04-02T04:53:42.132449+0000 mon.c (mon.2) 691 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:43.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[23790]: cluster 2024-04-02T04:53:42.442022+0000 mon.a (mon.0) 2458 : cluster 0 osdmap e382: 8 total, 7 up, 8 in 2024-04-02T04:53:43.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[23790]: audit 2024-04-02T04:53:42.988673+0000 mon.a (mon.0) 2459 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:43.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[23790]: cluster 2024-04-02T04:53:42.996344+0000 mon.a (mon.0) 2460 : cluster 0 osdmap e383: 8 total, 7 up, 8 in 2024-04-02T04:53:43.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:43 smithi098 bash[23790]: audit 2024-04-02T04:53:43.134190+0000 mon.c (mon.2) 692 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:43.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:43.611 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:43.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[21528]: cluster 2024-04-02T04:53:42.060543+0000 mgr.y (mgr.14538) 396 : cluster 0 pgmap v550: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[21528]: audit 2024-04-02T04:53:42.132449+0000 mon.c (mon.2) 691 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[21528]: cluster 2024-04-02T04:53:42.442022+0000 mon.a (mon.0) 2458 : cluster 0 osdmap e382: 8 total, 7 up, 8 in 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[21528]: audit 2024-04-02T04:53:42.988673+0000 mon.a (mon.0) 2459 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[21528]: cluster 2024-04-02T04:53:42.996344+0000 mon.a (mon.0) 2460 : cluster 0 osdmap e383: 8 total, 7 up, 8 in 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[21528]: audit 2024-04-02T04:53:43.134190+0000 mon.c (mon.2) 692 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[28904]: cluster 2024-04-02T04:53:42.060543+0000 mgr.y (mgr.14538) 396 : cluster 0 pgmap v550: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 278/1545 objects degraded (17.994%) 2024-04-02T04:53:43.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[28904]: audit 2024-04-02T04:53:42.132449+0000 mon.c (mon.2) 691 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:43.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[28904]: cluster 2024-04-02T04:53:42.442022+0000 mon.a (mon.0) 2458 : cluster 0 osdmap e382: 8 total, 7 up, 8 in 2024-04-02T04:53:43.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[28904]: audit 2024-04-02T04:53:42.988673+0000 mon.a (mon.0) 2459 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-72153-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:43.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[28904]: cluster 2024-04-02T04:53:42.996344+0000 mon.a (mon.0) 2460 : cluster 0 osdmap e383: 8 total, 7 up, 8 in 2024-04-02T04:53:43.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:43 smithi069 bash[28904]: audit 2024-04-02T04:53:43.134190+0000 mon.c (mon.2) 692 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:43.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:43.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:43.870 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:43.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:43.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.009 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:44.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:44.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:44.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:44.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:44.504 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:44 smithi098 bash[69630]: debug 2024-04-02T04:53:44.422+0000 7f5afb120700 1 -- 172.21.15.98:0/658662589 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3188e7680 con 0x55f318924800 2024-04-02T04:53:44.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:44 smithi098 bash[23790]: audit 2024-04-02T04:53:43.212824+0000 mgr.y (mgr.14538) 397 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:44.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:44 smithi098 bash[23790]: cluster 2024-04-02T04:53:43.999181+0000 mon.a (mon.0) 2461 : cluster 0 osdmap e384: 8 total, 7 up, 8 in 2024-04-02T04:53:44.505 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:44 smithi098 bash[23790]: audit 2024-04-02T04:53:44.135649+0000 mon.c (mon.2) 693 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:44.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.647 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:44.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:44.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:44 smithi069 bash[21528]: audit 2024-04-02T04:53:43.212824+0000 mgr.y (mgr.14538) 397 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:44.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:44 smithi069 bash[21528]: cluster 2024-04-02T04:53:43.999181+0000 mon.a (mon.0) 2461 : cluster 0 osdmap e384: 8 total, 7 up, 8 in 2024-04-02T04:53:44.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:44 smithi069 bash[21528]: audit 2024-04-02T04:53:44.135649+0000 mon.c (mon.2) 693 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:44.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:44 smithi069 bash[28904]: audit 2024-04-02T04:53:43.212824+0000 mgr.y (mgr.14538) 397 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:44.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:44 smithi069 bash[28904]: cluster 2024-04-02T04:53:43.999181+0000 mon.a (mon.0) 2461 : cluster 0 osdmap e384: 8 total, 7 up, 8 in 2024-04-02T04:53:44.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:44 smithi069 bash[28904]: audit 2024-04-02T04:53:44.135649+0000 mon.c (mon.2) 693 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:44.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:44.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:45.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.031 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:45.134 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.155 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:45.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:44 smithi098 bash[69630]: debug 2024-04-02T04:53:44.978+0000 7f5b042b87c0 -1 Falling back to public interface 2024-04-02T04:53:45.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:45.280 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:45.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:45.403 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:45.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: cluster 2024-04-02T04:53:44.061668+0000 mgr.y (mgr.14538) 398 : cluster 0 pgmap v553: 364 pgs: 8 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:53:45.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: cluster 2024-04-02T04:53:44.853749+0000 mon.a (mon.0) 2462 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:45.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: cluster 2024-04-02T04:53:44.866813+0000 mon.a (mon.0) 2463 : cluster 0 osdmap e385: 8 total, 7 up, 8 in 2024-04-02T04:53:45.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: audit 2024-04-02T04:53:44.870223+0000 mon.c (mon.2) 694 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:45.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: audit 2024-04-02T04:53:44.879379+0000 mon.a (mon.0) 2464 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: cluster 2024-04-02T04:53:44.991375+0000 mon.a (mon.0) 2465 : cluster 3 Health check update: Degraded data redundancy: 278/1548 objects degraded (17.959%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[21528]: audit 2024-04-02T04:53:45.137162+0000 mon.c (mon.2) 695 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: cluster 2024-04-02T04:53:44.061668+0000 mgr.y (mgr.14538) 398 : cluster 0 pgmap v553: 364 pgs: 8 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: cluster 2024-04-02T04:53:44.853749+0000 mon.a (mon.0) 2462 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: cluster 2024-04-02T04:53:44.866813+0000 mon.a (mon.0) 2463 : cluster 0 osdmap e385: 8 total, 7 up, 8 in 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: audit 2024-04-02T04:53:44.870223+0000 mon.c (mon.2) 694 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: audit 2024-04-02T04:53:44.879379+0000 mon.a (mon.0) 2464 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: cluster 2024-04-02T04:53:44.991375+0000 mon.a (mon.0) 2465 : cluster 3 Health check update: Degraded data redundancy: 278/1548 objects degraded (17.959%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:45.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:45 smithi069 bash[28904]: audit 2024-04-02T04:53:45.137162+0000 mon.c (mon.2) 695 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:45.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.653 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: cluster 2024-04-02T04:53:44.061668+0000 mgr.y (mgr.14538) 398 : cluster 0 pgmap v553: 364 pgs: 8 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: cluster 2024-04-02T04:53:44.853749+0000 mon.a (mon.0) 2462 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: cluster 2024-04-02T04:53:44.866813+0000 mon.a (mon.0) 2463 : cluster 0 osdmap e385: 8 total, 7 up, 8 in 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: audit 2024-04-02T04:53:44.870223+0000 mon.c (mon.2) 694 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: audit 2024-04-02T04:53:44.879379+0000 mon.a (mon.0) 2464 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: cluster 2024-04-02T04:53:44.991375+0000 mon.a (mon.0) 2465 : cluster 3 Health check update: Degraded data redundancy: 278/1548 objects degraded (17.959%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:45.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:45 smithi098 bash[23790]: audit 2024-04-02T04:53:45.137162+0000 mon.c (mon.2) 695 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:45.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:45.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.917 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:45.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:46.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:46.047 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:46.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:46.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:46.308 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:46.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:46.439 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:46.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:46.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:46.693 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:53:46.698 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:46.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:46.828 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7462ddf6::o.RoundTripAppendPP (3022 ms) 2024-04-02T04:53:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2942 ms) 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3023 ms) 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3478 ms) 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: setting pool EIO 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: max_success 99, min_failed 100 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3492 ms) 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3481 ms) 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (150232 ms total) 2024-04-02T04:53:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2564 ms) 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9054 ms) 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15047 ms) 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3006 ms) 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29671 ms total) 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-02T04:53:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3025 ms) 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3025 ms total) 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13965 ms) 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7067 ms) 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7274 ms) 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6667 ms) 2024-04-02T04:53:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3021 ms) 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7016 ms) 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6957 ms) 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7079 ms) 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6961 ms) 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6976 ms) 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-02T04:53:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7035 ms) 2024-04-02T04:53:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-02T04:53:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7067 ms) 2024-04-02T04:53:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-02T04:53:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6936 ms) 2024-04-02T04:53:46.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:46.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:46 smithi098 bash[23790]: audit 2024-04-02T04:53:45.874879+0000 mon.a (mon.0) 2466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:46.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:46 smithi098 bash[23790]: cluster 2024-04-02T04:53:45.881250+0000 mon.a (mon.0) 2467 : cluster 0 osdmap e386: 8 total, 7 up, 8 in 2024-04-02T04:53:46.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:46 smithi098 bash[23790]: audit 2024-04-02T04:53:45.883102+0000 mon.c (mon.2) 696 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:46.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:46 smithi098 bash[23790]: audit 2024-04-02T04:53:45.883732+0000 mon.a (mon.0) 2468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:46.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:46 smithi098 bash[23790]: cluster 2024-04-02T04:53:46.062833+0000 mgr.y (mgr.14538) 399 : cluster 0 pgmap v556: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:53:46.943 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:46 smithi098 bash[23790]: audit 2024-04-02T04:53:46.138442+0000 mon.c (mon.2) 697 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:46.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[21528]: audit 2024-04-02T04:53:45.874879+0000 mon.a (mon.0) 2466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[21528]: cluster 2024-04-02T04:53:45.881250+0000 mon.a (mon.0) 2467 : cluster 0 osdmap e386: 8 total, 7 up, 8 in 2024-04-02T04:53:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[21528]: audit 2024-04-02T04:53:45.883102+0000 mon.c (mon.2) 696 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[21528]: audit 2024-04-02T04:53:45.883732+0000 mon.a (mon.0) 2468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[21528]: cluster 2024-04-02T04:53:46.062833+0000 mgr.y (mgr.14538) 399 : cluster 0 pgmap v556: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[21528]: audit 2024-04-02T04:53:46.138442+0000 mon.c (mon.2) 697 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[28904]: audit 2024-04-02T04:53:45.874879+0000 mon.a (mon.0) 2466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[28904]: cluster 2024-04-02T04:53:45.881250+0000 mon.a (mon.0) 2467 : cluster 0 osdmap e386: 8 total, 7 up, 8 in 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[28904]: audit 2024-04-02T04:53:45.883102+0000 mon.c (mon.2) 696 : audit 1 from='client.? 172.21.15.69:0/3391289957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[28904]: audit 2024-04-02T04:53:45.883732+0000 mon.a (mon.0) 2468 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]: dispatch 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[28904]: cluster 2024-04-02T04:53:46.062833+0000 mgr.y (mgr.14538) 399 : cluster 0 pgmap v556: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 278/1548 objects degraded (17.959%) 2024-04-02T04:53:47.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:46 smithi069 bash[28904]: audit 2024-04-02T04:53:46.138442+0000 mon.c (mon.2) 697 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:47.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:47.089 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:47.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:47.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:47.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:47.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:47.467 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:47.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:47.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:47.725 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:47.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:47.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:47.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:47.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:47.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:48.085 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[21528]: audit 2024-04-02T04:53:46.885870+0000 mon.a (mon.0) 2469 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[21528]: cluster 2024-04-02T04:53:46.898503+0000 mon.a (mon.0) 2470 : cluster 0 osdmap e387: 8 total, 7 up, 8 in 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[21528]: audit 2024-04-02T04:53:47.139983+0000 mon.c (mon.2) 698 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[21528]: cluster 2024-04-02T04:53:47.388020+0000 osd.1 (osd.1) 231 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[21528]: cluster 2024-04-02T04:53:47.388122+0000 osd.1 (osd.1) 232 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[21528]: audit 2024-04-02T04:53:47.876414+0000 mon.b (mon.1) 207 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[28904]: audit 2024-04-02T04:53:46.885870+0000 mon.a (mon.0) 2469 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[28904]: cluster 2024-04-02T04:53:46.898503+0000 mon.a (mon.0) 2470 : cluster 0 osdmap e387: 8 total, 7 up, 8 in 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[28904]: audit 2024-04-02T04:53:47.139983+0000 mon.c (mon.2) 698 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[28904]: cluster 2024-04-02T04:53:47.388020+0000 osd.1 (osd.1) 231 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:48.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[28904]: cluster 2024-04-02T04:53:47.388122+0000 osd.1 (osd.1) 232 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:48.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:47 smithi069 bash[28904]: audit 2024-04-02T04:53:47.876414+0000 mon.b (mon.1) 207 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:48.111 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:48.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:48.222 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[69630]: debug 2024-04-02T04:53:48.138+0000 7f5b042b87c0 -1 osd.7 152 log_to_monitors true 2024-04-02T04:53:48.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:47 smithi098 bash[23790]: audit 2024-04-02T04:53:46.885870+0000 mon.a (mon.0) 2469 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-72153-59"}]': finished 2024-04-02T04:53:48.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:47 smithi098 bash[23790]: cluster 2024-04-02T04:53:46.898503+0000 mon.a (mon.0) 2470 : cluster 0 osdmap e387: 8 total, 7 up, 8 in 2024-04-02T04:53:48.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:47 smithi098 bash[23790]: audit 2024-04-02T04:53:47.139983+0000 mon.c (mon.2) 698 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:48.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:47 smithi098 bash[23790]: cluster 2024-04-02T04:53:47.388020+0000 osd.1 (osd.1) 231 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:48.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:47 smithi098 bash[23790]: cluster 2024-04-02T04:53:47.388122+0000 osd.1 (osd.1) 232 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:48.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:47 smithi098 bash[23790]: audit 2024-04-02T04:53:47.876414+0000 mon.b (mon.1) 207 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:48.240 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:48.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:48.369 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:48.472 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:48.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:48.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:48.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:48.757 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:48.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:48.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:48.919 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:53:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: cluster 2024-04-02T04:53:47.895571+0000 mon.a (mon.0) 2471 : cluster 0 osdmap e388: 8 total, 7 up, 8 in 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:47.918859+0000 mon.a (mon.0) 2472 : audit 1 from='client.? 172.21.15.69:0/3704678881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: cluster 2024-04-02T04:53:47.895571+0000 mon.a (mon.0) 2471 : cluster 0 osdmap e388: 8 total, 7 up, 8 in 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:47.918859+0000 mon.a (mon.0) 2472 : audit 1 from='client.? 172.21.15.69:0/3704678881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: cluster 2024-04-02T04:53:48.066831+0000 mgr.y (mgr.14538) 400 : cluster 0 pgmap v559: 388 pgs: 27 creating+peering, 5 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 278/1554 objects degraded (17.889%) 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.141434+0000 mon.c (mon.2) 699 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.148158+0000 mon.b (mon.1) 208 : audit 1 from='osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:53:48.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.152975+0000 mon.a (mon.0) 2473 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:53:48.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.893457+0000 mon.a (mon.0) 2474 : audit 1 from='client.? 172.21.15.69:0/3704678881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:48.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.893626+0000 mon.a (mon.0) 2475 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:53:48.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: cluster 2024-04-02T04:53:48.898603+0000 mon.a (mon.0) 2476 : cluster 0 osdmap e389: 8 total, 7 up, 8 in 2024-04-02T04:53:48.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.900179+0000 mon.b (mon.1) 209 : audit 1 from='osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:53:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[21528]: audit 2024-04-02T04:53:48.905242+0000 mon.a (mon.0) 2477 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:53:48.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: cluster 2024-04-02T04:53:48.066831+0000 mgr.y (mgr.14538) 400 : cluster 0 pgmap v559: 388 pgs: 27 creating+peering, 5 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 278/1554 objects degraded (17.889%) 2024-04-02T04:53:48.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.141434+0000 mon.c (mon.2) 699 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:48.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.148158+0000 mon.b (mon.1) 208 : audit 1 from='osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:53:48.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.152975+0000 mon.a (mon.0) 2473 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:53:48.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.893457+0000 mon.a (mon.0) 2474 : audit 1 from='client.? 172.21.15.69:0/3704678881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:48.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.893626+0000 mon.a (mon.0) 2475 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:53:48.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: cluster 2024-04-02T04:53:48.898603+0000 mon.a (mon.0) 2476 : cluster 0 osdmap e389: 8 total, 7 up, 8 in 2024-04-02T04:53:48.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.900179+0000 mon.b (mon.1) 209 : audit 1 from='osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:53:48.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:48 smithi069 bash[28904]: audit 2024-04-02T04:53:48.905242+0000 mon.a (mon.0) 2477 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:53:49.010 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:49.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:49.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: cluster 2024-04-02T04:53:47.895571+0000 mon.a (mon.0) 2471 : cluster 0 osdmap e388: 8 total, 7 up, 8 in 2024-04-02T04:53:49.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:47.918859+0000 mon.a (mon.0) 2472 : audit 1 from='client.? 172.21.15.69:0/3704678881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:49.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: cluster 2024-04-02T04:53:48.066831+0000 mgr.y (mgr.14538) 400 : cluster 0 pgmap v559: 388 pgs: 27 creating+peering, 5 unknown, 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 278/1554 objects degraded (17.889%) 2024-04-02T04:53:49.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.141434+0000 mon.c (mon.2) 699 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.148158+0000 mon.b (mon.1) 208 : audit 1 from='osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.152975+0000 mon.a (mon.0) 2473 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.893457+0000 mon.a (mon.0) 2474 : audit 1 from='client.? 172.21.15.69:0/3704678881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-72153-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.893626+0000 mon.a (mon.0) 2475 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: cluster 2024-04-02T04:53:48.898603+0000 mon.a (mon.0) 2476 : cluster 0 osdmap e389: 8 total, 7 up, 8 in 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.900179+0000 mon.b (mon.1) 209 : audit 1 from='osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:53:49.123 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:48 smithi098 bash[23790]: audit 2024-04-02T04:53:48.905242+0000 mon.a (mon.0) 2477 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-02T04:53:49.135 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:49.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:49.265 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:49.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:49.397 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:49.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:49.524 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:49.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:49.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:49.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:49.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:49.780 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:49.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:49.912 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:49.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:50.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:50.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:50.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: audit 2024-04-02T04:53:48.991684+0000 mon.c (mon.2) 700 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:50.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: audit 2024-04-02T04:53:48.992080+0000 mon.a (mon.0) 2478 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:50.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: audit 2024-04-02T04:53:48.992866+0000 mon.c (mon.2) 701 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]: dispatch 2024-04-02T04:53:50.149 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: audit 2024-04-02T04:53:48.993272+0000 mon.a (mon.0) 2479 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]: dispatch 2024-04-02T04:53:50.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: audit 2024-04-02T04:53:49.142725+0000 mon.c (mon.2) 702 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:50.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: cluster 2024-04-02T04:53:49.857857+0000 mon.a (mon.0) 2480 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:50.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: audit 2024-04-02T04:53:49.898604+0000 mon.a (mon.0) 2481 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]': finished 2024-04-02T04:53:50.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:49 smithi098 bash[23790]: cluster 2024-04-02T04:53:49.905610+0000 mon.a (mon.0) 2482 : cluster 0 osdmap e390: 8 total, 7 up, 8 in 2024-04-02T04:53:50.163 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:50.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:50.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: audit 2024-04-02T04:53:48.991684+0000 mon.c (mon.2) 700 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:50.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: audit 2024-04-02T04:53:48.992080+0000 mon.a (mon.0) 2478 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: audit 2024-04-02T04:53:48.992866+0000 mon.c (mon.2) 701 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: audit 2024-04-02T04:53:48.993272+0000 mon.a (mon.0) 2479 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: audit 2024-04-02T04:53:49.142725+0000 mon.c (mon.2) 702 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: cluster 2024-04-02T04:53:49.857857+0000 mon.a (mon.0) 2480 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: audit 2024-04-02T04:53:49.898604+0000 mon.a (mon.0) 2481 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]': finished 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[21528]: cluster 2024-04-02T04:53:49.905610+0000 mon.a (mon.0) 2482 : cluster 0 osdmap e390: 8 total, 7 up, 8 in 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: audit 2024-04-02T04:53:48.991684+0000 mon.c (mon.2) 700 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: audit 2024-04-02T04:53:48.992080+0000 mon.a (mon.0) 2478 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: audit 2024-04-02T04:53:48.992866+0000 mon.c (mon.2) 701 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]: dispatch 2024-04-02T04:53:50.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: audit 2024-04-02T04:53:48.993272+0000 mon.a (mon.0) 2479 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]: dispatch 2024-04-02T04:53:50.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: audit 2024-04-02T04:53:49.142725+0000 mon.c (mon.2) 702 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:50.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: cluster 2024-04-02T04:53:49.857857+0000 mon.a (mon.0) 2480 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:50.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: audit 2024-04-02T04:53:49.898604+0000 mon.a (mon.0) 2481 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-45"}]': finished 2024-04-02T04:53:50.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:49 smithi069 bash[28904]: cluster 2024-04-02T04:53:49.905610+0000 mon.a (mon.0) 2482 : cluster 0 osdmap e390: 8 total, 7 up, 8 in 2024-04-02T04:53:50.291 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:50.394 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:50.508 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:53:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:53:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T04:53:50.508 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:53:50.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.551 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:50.654 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:50.680 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:50.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:50.808 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:50.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.939 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:50.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:51.042 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:51.067 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:51.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:51.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:51.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:51.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: audit 2024-04-02T04:53:49.960461+0000 mon.a (mon.0) 2483 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: audit 2024-04-02T04:53:49.961888+0000 mon.a (mon.0) 2484 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: audit 2024-04-02T04:53:49.962460+0000 mon.a (mon.0) 2485 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-72153-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:51.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: cluster 2024-04-02T04:53:50.067741+0000 mgr.y (mgr.14538) 401 : cluster 0 pgmap v562: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 278/1554 objects degraded (17.889%) 2024-04-02T04:53:51.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: audit 2024-04-02T04:53:50.146155+0000 mon.c (mon.2) 703 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:51.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: audit 2024-04-02T04:53:50.901503+0000 mon.a (mon.0) 2486 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-72153-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:51.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: cluster 2024-04-02T04:53:50.909259+0000 mon.a (mon.0) 2487 : cluster 0 osdmap e391: 8 total, 7 up, 8 in 2024-04-02T04:53:51.436 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:51 smithi098 bash[23790]: audit 2024-04-02T04:53:50.914584+0000 mon.a (mon.0) 2488 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-72153-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:51.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:51.572 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:51.606 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:53:51.606 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:53:51.632 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: audit 2024-04-02T04:53:49.960461+0000 mon.a (mon.0) 2483 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.632 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: audit 2024-04-02T04:53:49.961888+0000 mon.a (mon.0) 2484 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: audit 2024-04-02T04:53:49.962460+0000 mon.a (mon.0) 2485 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-72153-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: cluster 2024-04-02T04:53:50.067741+0000 mgr.y (mgr.14538) 401 : cluster 0 pgmap v562: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 278/1554 objects degraded (17.889%) 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: audit 2024-04-02T04:53:50.146155+0000 mon.c (mon.2) 703 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: audit 2024-04-02T04:53:50.901503+0000 mon.a (mon.0) 2486 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-72153-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: cluster 2024-04-02T04:53:50.909259+0000 mon.a (mon.0) 2487 : cluster 0 osdmap e391: 8 total, 7 up, 8 in 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[21528]: audit 2024-04-02T04:53:50.914584+0000 mon.a (mon.0) 2488 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-72153-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: audit 2024-04-02T04:53:49.960461+0000 mon.a (mon.0) 2483 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: audit 2024-04-02T04:53:49.961888+0000 mon.a (mon.0) 2484 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: audit 2024-04-02T04:53:49.962460+0000 mon.a (mon.0) 2485 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-72153-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:51.633 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: cluster 2024-04-02T04:53:50.067741+0000 mgr.y (mgr.14538) 401 : cluster 0 pgmap v562: 356 pgs: 90 active+undersized, 229 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 278/1554 objects degraded (17.889%) 2024-04-02T04:53:51.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: audit 2024-04-02T04:53:50.146155+0000 mon.c (mon.2) 703 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:51.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: audit 2024-04-02T04:53:50.901503+0000 mon.a (mon.0) 2486 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-72153-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:53:51.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: cluster 2024-04-02T04:53:50.909259+0000 mon.a (mon.0) 2487 : cluster 0 osdmap e391: 8 total, 7 up, 8 in 2024-04-02T04:53:51.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:51 smithi069 bash[28904]: audit 2024-04-02T04:53:50.914584+0000 mon.a (mon.0) 2488 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-72153-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:51.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:51.693 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:51.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:51.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/2176695335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e40104450 msgr2=0x7f7e401048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- 172.21.15.69:0/2176695335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e40104450 0x7f7e401048d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009990 tx=0x7f7e34009ca0 comp rx=0 tx=0).stop 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/2176695335 shutdown_connections 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- 172.21.15.69:0/2176695335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e40104e10 0x7f7e40111320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- 172.21.15.69:0/2176695335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e4010a4e0 0x7f7e4010a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- 172.21.15.69:0/2176695335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e40104450 0x7f7e401048d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/2176695335 >> 172.21.15.69:0/2176695335 conn(0x7f7e400fff20 msgr2=0x7f7e40102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/2176695335 shutdown_connections 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/2176695335 wait complete. 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 librados: starting msgr at 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 librados: starting objecter 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 Processor -- start 2024-04-02T04:53:51.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- start start 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 librados: setting wanted keys 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 librados: calling monclient init 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e40104450 0x7f7e4019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e40104e10 0x7f7e4019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e4010a4e0 0x7f7e4019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40113c80 con 0x7f7e40104e10 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40113b00 con 0x7f7e40104450 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7e40113e00 con 0x7f7e4010a4e0 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e45693700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e4010a4e0 0x7f7e4019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e40104e10 0x7f7e4019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e45693700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e4010a4e0 0x7f7e4019eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:35518/0 (socket says 172.21.15.69:35518) 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e40104e10 0x7f7e4019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59754/0 (socket says 172.21.15.69:59754) 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e45693700 1 -- 172.21.15.69:0/3032011001 learned_addr learned my addr 172.21.15.69:0/3032011001 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:51.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e40104450 0x7f7e4019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 -- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e4010a4e0 msgr2=0x7f7e4019eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e4010a4e0 0x7f7e4019eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 -- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e40104450 msgr2=0x7f7e4019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e40104450 0x7f7e4019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4019f270 con 0x7f7e40104e10 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e4010a4e0 0x7f7e4019eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e40104450 0x7f7e4019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e40104e10 0x7f7e4019a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006010 tx=0x7f7e340057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3dffb700 10 client.?.objecter ms_handle_connect 0x7f7e40104e10 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34027070 con 0x7f7e40104e10 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4019f500 con 0x7f7e40104e10 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7e34022470 con 0x7f7e40104e10 2024-04-02T04:53:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 10 client.16374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:51.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34017650 con 0x7f7e40104e10 2024-04-02T04:53:51.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e401a6de0 con 0x7f7e40104e10 2024-04-02T04:53:51.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.797+0000 7f7e470f6700 1 librados: init done 2024-04-02T04:53:51.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7e4004ff00 con 0x7f7e40104e10 2024-04-02T04:53:51.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7e3402a070 con 0x7f7e40104e10 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7e2806f5e0 0x7f7e28071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(391..391 src has 74..391) ==== 10940+0+0 (secure 0 0 0) 0x7f7e3408fa20 con 0x7f7e40104e10 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 osd_map(391..391 src has 74..391) 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 3 client.16374.objecter handle_osd_map got epochs [391,391] > 0 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 3 client.16374.objecter handle_osd_map decoding full epoch 391 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7e2806f5e0 0x7f7e28071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 20 client.16374.objecter dump_active .. 0 homeless 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v391) ==== 54+0+16 (secure 0 0 0) 0x7f7e34013070 con 0x7f7e40104e10 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 10 client.16374.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 1 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7e4010baa0 0x7f7e4010be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e2806f5e0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7e2806f5e0 0x7f7e28071aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009fa0 tx=0x7f7e38009570 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 1 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 0 1 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e4019b5c0 osd=0 4 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 10 client.16374.objecter _send_command 1 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e401aa140 con 0x7f7e4010baa0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7e4010baa0 0x7f7e4010be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7e4010baa0 0x7f7e4010be80 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e4010baa0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e401aa140 con 0x7f7e4010baa0 2024-04-02T04:53:51.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 1: 0 ) 2024-04-02T04:53:51.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 1 = system:0 2024-04-02T04:53:51.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e4019b5c0 osd=0 4 2024-04-02T04:53:51.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.801+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 0 1 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 10 client.16374.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 2 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 2 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e4019b5c0 osd=0 3 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 0 2 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e4019b5c0 osd=0 4 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 10 client.16374.objecter _send_command 2 2024-04-02T04:53:51.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e4010b520 con 0x7f7e4010baa0 2024-04-02T04:53:51.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e4010b520 con 0x7f7e4010baa0 2024-04-02T04:53:51.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e4019b5c0 osd=0 4 2024-04-02T04:53:51.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 0 2 2024-04-02T04:53:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 10 client.16374.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 3 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7e4002c9b0 0x7f7e4002cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 3 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 1 3 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062e50 osd=1 4 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 10 client.16374.objecter _send_command 3 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7e400634d0 con 0x7f7e4002c9b0 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7e4002c9b0 0x7f7e4002cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e4002c9b0 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7e4002c9b0 0x7f7e4002cdd0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e400634d0 con 0x7f7e4002c9b0 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 3: 0 ) 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 3 = system:0 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e40062e50 osd=1 4 2024-04-02T04:53:51.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.821+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 1 3 2024-04-02T04:53:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 10 client.16374.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 4 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 4 2024-04-02T04:53:51.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e40062e50 osd=1 3 2024-04-02T04:53:51.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 1 4 2024-04-02T04:53:51.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062e50 osd=1 4 2024-04-02T04:53:51.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 10 client.16374.objecter _send_command 4 2024-04-02T04:53:51.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e400628d0 con 0x7f7e4002c9b0 2024-04-02T04:53:51.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e400628d0 con 0x7f7e4002c9b0 2024-04-02T04:53:51.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e40062e50 osd=1 4 2024-04-02T04:53:51.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 1 4 2024-04-02T04:53:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 10 client.16374.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 5 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7e401ac300 0x7f7e401ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 5 2024-04-02T04:53:51.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 2 5 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062cd0 osd=2 4 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 10 client.16374.objecter _send_command 5 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7e401b1550 con 0x7f7e401ac300 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7e401ac300 0x7f7e401ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e401ac300 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7e401ac300 0x7f7e401ae6f0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e401b1550 con 0x7f7e401ac300 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 5: 0 ) 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 5 = system:0 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e40062cd0 osd=2 4 2024-04-02T04:53:51.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.845+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 2 5 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 10 client.16374.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 6 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 6 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e40062cd0 osd=2 3 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 2 6 2024-04-02T04:53:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e40062cd0 osd=2 4 2024-04-02T04:53:51.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 10 client.16374.objecter _send_command 6 2024-04-02T04:53:51.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e4002d560 con 0x7f7e401ac300 2024-04-02T04:53:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e4002d560 con 0x7f7e401ac300 2024-04-02T04:53:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e40062cd0 osd=2 4 2024-04-02T04:53:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 2 6 2024-04-02T04:53:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 10 client.16374.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 7 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f7e401afdb0 0x7f7e401b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 7 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 3 7 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9790 osd=3 4 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 10 client.16374.objecter _send_command 7 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7e401a9e10 con 0x7f7e401afdb0 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f7e401afdb0 0x7f7e401b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e401afdb0 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f7e401afdb0 0x7f7e401b01f0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e401a9e10 con 0x7f7e401afdb0 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 7: 0 ) 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 7 = system:0 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401a9790 osd=3 4 2024-04-02T04:53:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.865+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 3 7 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 10 client.16374.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 8 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 8 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401a9790 osd=3 3 2024-04-02T04:53:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 3 8 2024-04-02T04:53:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9790 osd=3 4 2024-04-02T04:53:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 10 client.16374.objecter _send_command 8 2024-04-02T04:53:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.885+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e401a9210 con 0x7f7e401afdb0 2024-04-02T04:53:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e401a9210 con 0x7f7e401afdb0 2024-04-02T04:53:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401a9790 osd=3 4 2024-04-02T04:53:51.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 3 8 2024-04-02T04:53:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 10 client.16374.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 9 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f7e401a9380 0x7f7e401b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 9 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 4 9 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9210 osd=4 4 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 10 client.16374.objecter _send_command 9 2024-04-02T04:53:51.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7e401b86a0 con 0x7f7e401a9380 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f7e401a9380 0x7f7e401b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e401a9380 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e44e92700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f7e401a9380 0x7f7e401b5720 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e401b86a0 con 0x7f7e401a9380 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 9: 0 ) 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 9 = system:0 2024-04-02T04:53:51.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401a9210 osd=4 4 2024-04-02T04:53:51.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.889+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 4 9 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 10 client.16374.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 10 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 10 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401a9210 osd=4 3 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 4 10 2024-04-02T04:53:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401a9210 osd=4 4 2024-04-02T04:53:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 10 client.16374.objecter _send_command 10 2024-04-02T04:53:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e401b7430 con 0x7f7e401a9380 2024-04-02T04:53:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e401b7430 con 0x7f7e401a9380 2024-04-02T04:53:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401a9210 osd=4 4 2024-04-02T04:53:51.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 4 10 2024-04-02T04:53:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:53:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 10 client.16374.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 11 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f7e401b7b50 0x7f7e401b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 11 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 5 11 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401b7430 osd=5 4 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 10 client.16374.objecter _send_command 11 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7e401bba90 con 0x7f7e401b7b50 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f7e401b7b50 0x7f7e401b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e401b7b50 2024-04-02T04:53:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.929+0000 7f7e3ffff700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f7e401b7b50 0x7f7e401b7fb0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.933+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e401bba90 con 0x7f7e401b7b50 2024-04-02T04:53:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.933+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 11: 0 ) 2024-04-02T04:53:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.933+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 11 = system:0 2024-04-02T04:53:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.933+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401b7430 osd=5 4 2024-04-02T04:53:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.933+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 5 11 2024-04-02T04:53:51.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:51.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 10 client.16374.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 12 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 12 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401b7430 osd=5 3 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 5 12 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401b7430 osd=5 4 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 10 client.16374.objecter _send_command 12 2024-04-02T04:53:51.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e401bba90 con 0x7f7e401b7b50 2024-04-02T04:53:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.949+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e401bba90 con 0x7f7e401b7b50 2024-04-02T04:53:51.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401b7430 osd=5 4 2024-04-02T04:53:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 5 12 2024-04-02T04:53:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:53:51.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 10 client.16374.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 13 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f7e401bae20 0x7f7e401bb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 13 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 6 13 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401bba90 osd=6 4 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 10 client.16374.objecter _send_command 13 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7e401bee30 con 0x7f7e401bae20 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f7e401bae20 0x7f7e401bb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:51.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 10 client.16374.objecter ms_handle_connect 0x7f7e401bae20 2024-04-02T04:53:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e45693700 1 --2- 172.21.15.69:0/3032011001 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f7e401bae20 0x7f7e401bb260 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e401bee30 con 0x7f7e401bae20 2024-04-02T04:53:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 13: 0 ) 2024-04-02T04:53:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 13 = system:0 2024-04-02T04:53:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401bba90 osd=6 4 2024-04-02T04:53:51.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.953+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 6 13 2024-04-02T04:53:51.963 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:51.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 10 client.16374.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 14 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 20 client.16374.objecter _get_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_remove -1 14 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 20 client.16374.objecter get_session s=0x7f7e401bba90 osd=6 3 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign 6 14 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 20 client.16374.objecter put_session s=0x7f7e401bba90 osd=6 4 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 10 client.16374.objecter _send_command 14 2024-04-02T04:53:51.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.969+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e401b5c60 con 0x7f7e401bae20 2024-04-02T04:53:51.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e401b5c60 con 0x7f7e401bae20 2024-04-02T04:53:51.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:53:51.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e3dffb700 10 client.16374.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:53:51.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e3dffb700 20 client.16374.objecter put_session s=0x7f7e401bba90 osd=6 4 2024-04-02T04:53:51.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e3dffb700 15 client.16374.objecter _session_command_op_remove 6 14 2024-04-02T04:53:51.972 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:53:51.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e470f6700 10 client.16374.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:53:51.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e470f6700 15 client.16374.objecter _session_command_op_assign -1 15 2024-04-02T04:53:51.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e470f6700 20 client.16374.objecter _get_session osd=-1 returning homeless 2024-04-02T04:53:51.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e470f6700 10 client.16374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:51.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=392}) -- 0x7f7e401bee60 con 0x7f7e40104e10 2024-04-02T04:53:51.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:51.973+0000 7f7e470f6700 1 -- 172.21.15.69:0/3032011001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e401bf1a0 con 0x7f7e40104e10 2024-04-02T04:53:52.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.025+0000 7ff343520700 1 -- 172.21.15.69:0/2255214078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c1011a0 msgr2=0x7ff33c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:52.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.025+0000 7ff343520700 1 --2- 172.21.15.69:0/2255214078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c1011a0 0x7ff33c101620 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009990 tx=0x7ff32c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.025+0000 7ff343520700 1 -- 172.21.15.69:0/2255214078 shutdown_connections 2024-04-02T04:53:52.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.025+0000 7ff343520700 1 --2- 172.21.15.69:0/2255214078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c101b60 0x7ff33c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.025+0000 7ff343520700 1 --2- 172.21.15.69:0/2255214078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c107220 0x7ff33c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.025+0000 7ff343520700 1 --2- 172.21.15.69:0/2255214078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c1011a0 0x7ff33c101620 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- 172.21.15.69:0/2255214078 >> 172.21.15.69:0/2255214078 conn(0x7ff33c0771e0 msgr2=0x7ff33c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- 172.21.15.69:0/2255214078 shutdown_connections 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- 172.21.15.69:0/2255214078 wait complete. 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 librados: starting msgr at 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 librados: starting objecter 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 Processor -- start 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- start start 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 librados: setting wanted keys 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 librados: calling monclient init 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c1011a0 0x7ff33c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c101b60 0x7ff33c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:52.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c107220 0x7ff33c19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:52.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c110970 con 0x7ff33c1011a0 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c1107f0 con 0x7ff33c101b60 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c110af0 con 0x7ff33c107220 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3412bc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c1011a0 0x7ff33c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3412bc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c1011a0 0x7ff33c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59770/0 (socket says 172.21.15.69:59770) 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3412bc700 1 -- 172.21.15.69:0/3204760425 learned_addr learned my addr 172.21.15.69:0/3204760425 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c107220 0x7ff33c19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff340abb700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c101b60 0x7ff33c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 -- 172.21.15.69:0/3204760425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c101b60 msgr2=0x7ff33c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c101b60 0x7ff33c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 -- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c1011a0 msgr2=0x7ff33c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:52.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c1011a0 0x7ff33c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 -- 172.21.15.69:0/3204760425 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c19f160 con 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff341abd700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c107220 0x7ff33c19ea20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff334007f10 tx=0x7ff33400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3327fc700 10 client.?.objecter ms_handle_connect 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3327fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334004db0 con 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c19f450 con 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff334009e20 con 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334015400 con 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 10 client.26477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c1a6cd0 con 0x7ff33c107220 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 librados: init done 2024-04-02T04:53:52.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33c04ff00 con 0x7ff33c107220 2024-04-02T04:53:52.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.029+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff33400db20 con 0x7ff33c107220 2024-04-02T04:53:52.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff32806f5e0 0x7ff328071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(392..392 src has 74..392) ==== 11419+0+0 (secure 0 0 0) 0x7ff33408e870 con 0x7ff33c107220 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 10 client.26477.objecter ms_dispatch 0x7ff33c001550 osd_map(392..392 src has 74..392) 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 3 client.26477.objecter handle_osd_map got epochs [392,392] > 0 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 3 client.26477.objecter handle_osd_map decoding full epoch 392 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3412bc700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff32806f5e0 0x7ff328071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 20 client.26477.objecter dump_active .. 0 homeless 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3327fc700 10 client.26477.objecter ms_handle_connect 0x7ff32806f5e0 2024-04-02T04:53:52.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.033+0000 7ff3412bc700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff32806f5e0 0x7ff328071aa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff338005950 tx=0x7ff33800b500 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:53:52.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.037+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff334010070 con 0x7ff33c107220 2024-04-02T04:53:52.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:52.084 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:52.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:52.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:52.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.253+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7ff33c1a6fc0 con 0x7ff33c107220 2024-04-02T04:53:52.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 7 ==== osd_map(392..392 src has 74..392) ==== 1745+0+0 (secure 0 0 0) 0x7f7e3408fe00 con 0x7f7e40104e10 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 10 client.16374.objecter ms_dispatch 0x7f7e40001550 osd_map(392..392 src has 74..392) 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 3 client.16374.objecter handle_osd_map got epochs [392,392] > 391 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 3 client.16374.objecter handle_osd_map decoding incremental epoch 392 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 10 client.16374.objecter checking command 15 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 10 client.16374.objecter _check_command_map_dne tid 15 current 392 map_dne_bound 0 2024-04-02T04:53:52.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 20 client.16374.objecter dump_active .. 1 homeless 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3dffb700 1 -- 172.21.15.69:0/3032011001 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_get_version_reply(handle=1 version=392) ==== 24+0+0 (secure 0 0 0) 0x7f7e3405b9f0 con 0x7f7e40104e10 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3effd700 10 client.16374.objecter _check_command_map_dne tid 15 current 392 map_dne_bound 392 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3effd700 10 client.16374.objecter _finish_command 15 = osd:6 osd down 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.345+0000 7f7e3effd700 15 client.16374.objecter _session_command_op_remove -1 15 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:53:52.345 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:53:52.346 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:53:52.346 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:53:52.346 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:53:52.346 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:53:52.346 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:53:52.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-02T04:53:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:52.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:52.472 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:52.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: cluster 2024-04-02T04:53:49.102873+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: cluster 2024-04-02T04:53:49.103011+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:51.147573+0000 mon.c (mon.2) 704 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:51.804527+0000 mon.a (mon.0) 2489 : audit 0 from='client.? 172.21.15.69:0/3032011001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:52.148922+0000 mon.c (mon.2) 705 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:52.256851+0000 mon.c (mon.2) 706 : audit 1 from='client.? 172.21.15.69:0/3204760425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: cluster 2024-04-02T04:53:52.346441+0000 mon.a (mon.0) 2490 : cluster 0 osdmap e392: 8 total, 7 up, 8 in 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:52.347834+0000 mon.c (mon.2) 707 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:52.353256+0000 mon.a (mon.0) 2491 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:53:52.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:52 smithi098 bash[23790]: audit 2024-04-02T04:53:52.353773+0000 mon.a (mon.0) 2492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:52.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: cluster 2024-04-02T04:53:49.102873+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: cluster 2024-04-02T04:53:49.103011+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:51.147573+0000 mon.c (mon.2) 704 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:51.804527+0000 mon.a (mon.0) 2489 : audit 0 from='client.? 172.21.15.69:0/3032011001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:52.148922+0000 mon.c (mon.2) 705 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:52.256851+0000 mon.c (mon.2) 706 : audit 1 from='client.? 172.21.15.69:0/3204760425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: cluster 2024-04-02T04:53:52.346441+0000 mon.a (mon.0) 2490 : cluster 0 osdmap e392: 8 total, 7 up, 8 in 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:52.347834+0000 mon.c (mon.2) 707 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:52.353256+0000 mon.a (mon.0) 2491 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[28904]: audit 2024-04-02T04:53:52.353773+0000 mon.a (mon.0) 2492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: cluster 2024-04-02T04:53:49.102873+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: cluster 2024-04-02T04:53:49.103011+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:51.147573+0000 mon.c (mon.2) 704 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:52.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:51.804527+0000 mon.a (mon.0) 2489 : audit 0 from='client.? 172.21.15.69:0/3032011001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:53:52.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:52.148922+0000 mon.c (mon.2) 705 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:52.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:52.256851+0000 mon.c (mon.2) 706 : audit 1 from='client.? 172.21.15.69:0/3204760425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:53:52.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: cluster 2024-04-02T04:53:52.346441+0000 mon.a (mon.0) 2490 : cluster 0 osdmap e392: 8 total, 7 up, 8 in 2024-04-02T04:53:52.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:52.347834+0000 mon.c (mon.2) 707 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:52.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:52.353256+0000 mon.a (mon.0) 2491 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:53:52.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:52 smithi069 bash[21528]: audit 2024-04-02T04:53:52.353773+0000 mon.a (mon.0) 2492 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:53:52.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-02T04:53:52.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:52.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:52.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:52.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.913+0000 7ff3327fc700 1 -- 172.21.15.69:0/3204760425 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v393) ==== 91+0+0 (secure 0 0 0) 0x7ff33405af60 con 0x7ff33c107220 2024-04-02T04:53:52.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.917+0000 7ff343520700 10 librados: watch_flush enter 2024-04-02T04:53:52.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 10 librados: watch_flush exit 2024-04-02T04:53:52.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 20 client.26477.objecter shutdown clearing up homeless session... 2024-04-02T04:53:52.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 10 client.26477.objecter successfully canceled tick 2024-04-02T04:53:52.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff32806f5e0 msgr2=0x7ff328071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:52.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff32806f5e0 0x7ff328071aa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff338005950 tx=0x7ff33800b500 comp rx=0 tx=0).stop 2024-04-02T04:53:52.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c107220 msgr2=0x7ff33c19ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:53:52.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c107220 0x7ff33c19ea20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff334007f10 tx=0x7ff33400d3b0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 shutdown_connections 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff32806f5e0 0x7ff328071aa0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff33c107220 0x7ff33c19ea20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff33c1011a0 0x7ff33c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 --2- 172.21.15.69:0/3204760425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c101b60 0x7ff33c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 >> 172.21.15.69:0/3204760425 conn(0x7ff33c0771e0 msgr2=0x7ff33c0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 shutdown_connections 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 -- 172.21.15.69:0/3204760425 wait complete. 2024-04-02T04:53:52.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:53:52.921+0000 7ff343520700 1 librados: shutdown 2024-04-02T04:53:52.923 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:53:52.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:52.976 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:52.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:53.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:53.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:53.228 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:53.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:53.359 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:53.371 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[69630]: debug 2024-04-02T04:53:53.002+0000 7f5af5710700 -1 osd.7 391 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:53:53.371 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:53:53.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: cluster 2024-04-02T04:53:52.068755+0000 mgr.y (mgr.14538) 402 : cluster 0 pgmap v565: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: cluster 2024-04-02T04:53:52.903340+0000 mon.a (mon.0) 2493 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: cluster 2024-04-02T04:53:52.904443+0000 mon.a (mon.0) 2494 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: audit 2024-04-02T04:53:52.913496+0000 mon.a (mon.0) 2495 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-72153-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: audit 2024-04-02T04:53:52.913577+0000 mon.a (mon.0) 2496 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: audit 2024-04-02T04:53:52.913635+0000 mon.a (mon.0) 2497 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: cluster 2024-04-02T04:53:52.918159+0000 mon.a (mon.0) 2498 : cluster 0 osdmap e393: 8 total, 7 up, 8 in 2024-04-02T04:53:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: audit 2024-04-02T04:53:52.985079+0000 mon.c (mon.2) 708 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: audit 2024-04-02T04:53:52.985465+0000 mon.a (mon.0) 2499 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:53 smithi098 bash[23790]: audit 2024-04-02T04:53:53.150273+0000 mon.c (mon.2) 709 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-02T04:53:53.507 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-02T04:53:53.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: cluster 2024-04-02T04:53:52.068755+0000 mgr.y (mgr.14538) 402 : cluster 0 pgmap v565: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: cluster 2024-04-02T04:53:52.903340+0000 mon.a (mon.0) 2493 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: cluster 2024-04-02T04:53:52.904443+0000 mon.a (mon.0) 2494 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: audit 2024-04-02T04:53:52.913496+0000 mon.a (mon.0) 2495 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-72153-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: audit 2024-04-02T04:53:52.913577+0000 mon.a (mon.0) 2496 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: audit 2024-04-02T04:53:52.913635+0000 mon.a (mon.0) 2497 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: cluster 2024-04-02T04:53:52.918159+0000 mon.a (mon.0) 2498 : cluster 0 osdmap e393: 8 total, 7 up, 8 in 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: audit 2024-04-02T04:53:52.985079+0000 mon.c (mon.2) 708 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: audit 2024-04-02T04:53:52.985465+0000 mon.a (mon.0) 2499 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:53.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[21528]: audit 2024-04-02T04:53:53.150273+0000 mon.c (mon.2) 709 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: cluster 2024-04-02T04:53:52.068755+0000 mgr.y (mgr.14538) 402 : cluster 0 pgmap v565: 356 pgs: 32 unknown, 80 active+undersized, 207 active+clean, 37 active+undersized+degraded; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 278/1542 objects degraded (18.029%) 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: cluster 2024-04-02T04:53:52.903340+0000 mon.a (mon.0) 2493 : cluster 3 Health check update: Degraded data redundancy: 278/1542 objects degraded (18.029%), 37 pgs degraded, 117 pgs undersized (PG_DEGRADED) 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: cluster 2024-04-02T04:53:52.904443+0000 mon.a (mon.0) 2494 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: audit 2024-04-02T04:53:52.913496+0000 mon.a (mon.0) 2495 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-72153-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: audit 2024-04-02T04:53:52.913577+0000 mon.a (mon.0) 2496 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: audit 2024-04-02T04:53:52.913635+0000 mon.a (mon.0) 2497 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: cluster 2024-04-02T04:53:52.918159+0000 mon.a (mon.0) 2498 : cluster 0 osdmap e393: 8 total, 7 up, 8 in 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: audit 2024-04-02T04:53:52.985079+0000 mon.c (mon.2) 708 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: audit 2024-04-02T04:53:52.985465+0000 mon.a (mon.0) 2499 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:53:53.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:53 smithi069 bash[28904]: audit 2024-04-02T04:53:53.150273+0000 mon.c (mon.2) 709 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:53.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:53.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:53.747 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:53.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:53.878 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:53.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:53.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:54.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:54.132 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:54.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:54.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:54.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:54.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:54.519 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:54.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: audit 2024-04-02T04:53:53.221055+0000 mgr.y (mgr.14538) 403 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.354639+0000 osd.1 (osd.1) 233 : cluster 0 6.1 scrub starts 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.356297+0000 osd.1 (osd.1) 234 : cluster 0 6.1 scrub ok 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.766166+0000 osd.5 (osd.5) 219 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.766333+0000 osd.5 (osd.5) 220 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.915298+0000 mon.a (mon.0) 2500 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: audit 2024-04-02T04:53:53.917649+0000 mon.a (mon.0) 2501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: audit 2024-04-02T04:53:53.917992+0000 mon.b (mon.1) 210 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.925993+0000 mon.a (mon.0) 2502 : cluster 1 osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] boot 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.926103+0000 mon.a (mon.0) 2503 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: audit 2024-04-02T04:53:53.932844+0000 mon.c (mon.2) 710 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:53:54.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: audit 2024-04-02T04:53:53.938348+0000 mon.a (mon.0) 2504 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[21528]: audit 2024-04-02T04:53:54.151561+0000 mon.c (mon.2) 711 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: audit 2024-04-02T04:53:53.221055+0000 mgr.y (mgr.14538) 403 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.354639+0000 osd.1 (osd.1) 233 : cluster 0 6.1 scrub starts 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.356297+0000 osd.1 (osd.1) 234 : cluster 0 6.1 scrub ok 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.766166+0000 osd.5 (osd.5) 219 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.766333+0000 osd.5 (osd.5) 220 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.915298+0000 mon.a (mon.0) 2500 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: audit 2024-04-02T04:53:53.917649+0000 mon.a (mon.0) 2501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: audit 2024-04-02T04:53:53.917992+0000 mon.b (mon.1) 210 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:53:54.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.925993+0000 mon.a (mon.0) 2502 : cluster 1 osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] boot 2024-04-02T04:53:54.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.926103+0000 mon.a (mon.0) 2503 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-04-02T04:53:54.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: audit 2024-04-02T04:53:53.932844+0000 mon.c (mon.2) 710 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:53:54.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: audit 2024-04-02T04:53:53.938348+0000 mon.a (mon.0) 2504 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:53:54.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:54 smithi069 bash[28904]: audit 2024-04-02T04:53:54.151561+0000 mon.c (mon.2) 711 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: audit 2024-04-02T04:53:53.221055+0000 mgr.y (mgr.14538) 403 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.354639+0000 osd.1 (osd.1) 233 : cluster 0 6.1 scrub starts 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.356297+0000 osd.1 (osd.1) 234 : cluster 0 6.1 scrub ok 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.766166+0000 osd.5 (osd.5) 219 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.766333+0000 osd.5 (osd.5) 220 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.915298+0000 mon.a (mon.0) 2500 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: audit 2024-04-02T04:53:53.917649+0000 mon.a (mon.0) 2501 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:53:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: audit 2024-04-02T04:53:53.917992+0000 mon.b (mon.1) 210 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:53:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.925993+0000 mon.a (mon.0) 2502 : cluster 1 osd.7 [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] boot 2024-04-02T04:53:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.926103+0000 mon.a (mon.0) 2503 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-04-02T04:53:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: audit 2024-04-02T04:53:53.932844+0000 mon.c (mon.2) 710 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:53:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: audit 2024-04-02T04:53:53.938348+0000 mon.a (mon.0) 2504 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:53:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:54 smithi098 bash[23790]: audit 2024-04-02T04:53:54.151561+0000 mon.c (mon.2) 711 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:54.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:54.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:54.941 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:54.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:55.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:55.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:55.295 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:55.317 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:55.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.736695+0000 osd.4 (osd.4) 217 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.736853+0000 osd.4 (osd.4) 218 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.995611+0000 osd.2 (osd.2) 176 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:53.995751+0000 osd.2 (osd.2) 177 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.053854+0000 osd.2 (osd.2) 178 : cluster 0 10.3 scrub starts 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.055034+0000 osd.2 (osd.2) 179 : cluster 0 10.3 scrub ok 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.071329+0000 mgr.y (mgr.14538) 404 : cluster 0 pgmap v568: 364 pgs: 38 peering, 3 creating+activating, 27 creating+peering, 6 unknown, 3 creating+activating+undersized, 57 active+undersized, 207 active+clean, 23 active+undersized+degraded; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 192/1494 objects degraded (12.851%) 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: audit 2024-04-02T04:53:54.863184+0000 mon.a (mon.0) 2505 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]': finished 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.871322+0000 mon.a (mon.0) 2506 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: audit 2024-04-02T04:53:54.873038+0000 mon.c (mon.2) 712 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]: dispatch 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: audit 2024-04-02T04:53:54.877589+0000 mon.a (mon.0) 2507 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]: dispatch 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.917689+0000 mon.a (mon.0) 2508 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 33 pgs peering (PG_AVAILABILITY) 2024-04-02T04:53:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.051152+0000 osd.7 (osd.7) 3 : cluster 0 6.3 scrub starts 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.052152+0000 osd.7 (osd.7) 4 : cluster 0 6.3 scrub ok 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[21528]: audit 2024-04-02T04:53:55.153071+0000 mon.c (mon.2) 713 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.736695+0000 osd.4 (osd.4) 217 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.736853+0000 osd.4 (osd.4) 218 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.995611+0000 osd.2 (osd.2) 176 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:53.995751+0000 osd.2 (osd.2) 177 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.053854+0000 osd.2 (osd.2) 178 : cluster 0 10.3 scrub starts 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.055034+0000 osd.2 (osd.2) 179 : cluster 0 10.3 scrub ok 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.071329+0000 mgr.y (mgr.14538) 404 : cluster 0 pgmap v568: 364 pgs: 38 peering, 3 creating+activating, 27 creating+peering, 6 unknown, 3 creating+activating+undersized, 57 active+undersized, 207 active+clean, 23 active+undersized+degraded; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 192/1494 objects degraded (12.851%) 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: audit 2024-04-02T04:53:54.863184+0000 mon.a (mon.0) 2505 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]': finished 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.871322+0000 mon.a (mon.0) 2506 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: audit 2024-04-02T04:53:54.873038+0000 mon.c (mon.2) 712 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]: dispatch 2024-04-02T04:53:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: audit 2024-04-02T04:53:54.877589+0000 mon.a (mon.0) 2507 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]: dispatch 2024-04-02T04:53:55.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.917689+0000 mon.a (mon.0) 2508 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 33 pgs peering (PG_AVAILABILITY) 2024-04-02T04:53:55.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.051152+0000 osd.7 (osd.7) 3 : cluster 0 6.3 scrub starts 2024-04-02T04:53:55.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.052152+0000 osd.7 (osd.7) 4 : cluster 0 6.3 scrub ok 2024-04-02T04:53:55.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:55 smithi069 bash[28904]: audit 2024-04-02T04:53:55.153071+0000 mon.c (mon.2) 713 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:55.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:53:55.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:55.577 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:55.680 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.736695+0000 osd.4 (osd.4) 217 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.736853+0000 osd.4 (osd.4) 218 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.995611+0000 osd.2 (osd.2) 176 : cluster 0 purged_snaps scrub starts 2024-04-02T04:53:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:53.995751+0000 osd.2 (osd.2) 177 : cluster 0 purged_snaps scrub ok 2024-04-02T04:53:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.053854+0000 osd.2 (osd.2) 178 : cluster 0 10.3 scrub starts 2024-04-02T04:53:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.055034+0000 osd.2 (osd.2) 179 : cluster 0 10.3 scrub ok 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.071329+0000 mgr.y (mgr.14538) 404 : cluster 0 pgmap v568: 364 pgs: 38 peering, 3 creating+activating, 27 creating+peering, 6 unknown, 3 creating+activating+undersized, 57 active+undersized, 207 active+clean, 23 active+undersized+degraded; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 192/1494 objects degraded (12.851%) 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: audit 2024-04-02T04:53:54.863184+0000 mon.a (mon.0) 2505 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-47"}]': finished 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.871322+0000 mon.a (mon.0) 2506 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: audit 2024-04-02T04:53:54.873038+0000 mon.c (mon.2) 712 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]: dispatch 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: audit 2024-04-02T04:53:54.877589+0000 mon.a (mon.0) 2507 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]: dispatch 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.917689+0000 mon.a (mon.0) 2508 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 33 pgs peering (PG_AVAILABILITY) 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.051152+0000 osd.7 (osd.7) 3 : cluster 0 6.3 scrub starts 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.052152+0000 osd.7 (osd.7) 4 : cluster 0 6.3 scrub ok 2024-04-02T04:53:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:55 smithi098 bash[23790]: audit 2024-04-02T04:53:55.153071+0000 mon.c (mon.2) 713 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:55.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:55.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:55.965 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:55.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:56.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:56.095 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:56.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:53:56.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:56.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:56.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:56.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.643314+0000 osd.4 (osd.4) 219 : cluster 0 40.1b scrub starts 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.645619+0000 osd.4 (osd.4) 220 : cluster 0 40.1b scrub ok 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.817163+0000 osd.6 (osd.6) 183 : cluster 0 102.c scrub starts 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:54.821013+0000 osd.6 (osd.6) 184 : cluster 0 102.c scrub ok 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.152713+0000 osd.3 (osd.3) 251 : cluster 0 4.14 scrub starts 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.155570+0000 osd.3 (osd.3) 252 : cluster 0 4.14 scrub ok 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.756947+0000 osd.5 (osd.5) 221 : cluster 0 2.3 scrub starts 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.758983+0000 osd.5 (osd.5) 222 : cluster 0 2.3 scrub ok 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.863643+0000 mon.a (mon.0) 2509 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: audit 2024-04-02T04:53:55.867967+0000 mon.a (mon.0) 2510 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]': finished 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:55.875870+0000 mon.a (mon.0) 2511 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: audit 2024-04-02T04:53:55.882064+0000 mon.c (mon.2) 714 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:56.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: audit 2024-04-02T04:53:55.883094+0000 mon.a (mon.0) 2512 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: audit 2024-04-02T04:53:55.884191+0000 mon.a (mon.0) 2513 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.022659+0000 osd.7 (osd.7) 5 : cluster 0 6.5 scrub starts 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.024442+0000 osd.7 (osd.7) 6 : cluster 0 6.5 scrub ok 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[21528]: audit 2024-04-02T04:53:56.155088+0000 mon.c (mon.2) 715 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.643314+0000 osd.4 (osd.4) 219 : cluster 0 40.1b scrub starts 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.645619+0000 osd.4 (osd.4) 220 : cluster 0 40.1b scrub ok 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.817163+0000 osd.6 (osd.6) 183 : cluster 0 102.c scrub starts 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:54.821013+0000 osd.6 (osd.6) 184 : cluster 0 102.c scrub ok 2024-04-02T04:53:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.152713+0000 osd.3 (osd.3) 251 : cluster 0 4.14 scrub starts 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.155570+0000 osd.3 (osd.3) 252 : cluster 0 4.14 scrub ok 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.756947+0000 osd.5 (osd.5) 221 : cluster 0 2.3 scrub starts 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.758983+0000 osd.5 (osd.5) 222 : cluster 0 2.3 scrub ok 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.863643+0000 mon.a (mon.0) 2509 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: audit 2024-04-02T04:53:55.867967+0000 mon.a (mon.0) 2510 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]': finished 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:55.875870+0000 mon.a (mon.0) 2511 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: audit 2024-04-02T04:53:55.882064+0000 mon.c (mon.2) 714 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: audit 2024-04-02T04:53:55.883094+0000 mon.a (mon.0) 2512 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: audit 2024-04-02T04:53:55.884191+0000 mon.a (mon.0) 2513 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.022659+0000 osd.7 (osd.7) 5 : cluster 0 6.5 scrub starts 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.024442+0000 osd.7 (osd.7) 6 : cluster 0 6.5 scrub ok 2024-04-02T04:53:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:56 smithi069 bash[28904]: audit 2024-04-02T04:53:56.155088+0000 mon.c (mon.2) 715 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:56.479 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:56.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:56.609 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:56.655 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:53:56.656 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:53:56.656 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-02T04:53:56.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:53:56.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.643314+0000 osd.4 (osd.4) 219 : cluster 0 40.1b scrub starts 2024-04-02T04:53:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.645619+0000 osd.4 (osd.4) 220 : cluster 0 40.1b scrub ok 2024-04-02T04:53:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.817163+0000 osd.6 (osd.6) 183 : cluster 0 102.c scrub starts 2024-04-02T04:53:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:54.821013+0000 osd.6 (osd.6) 184 : cluster 0 102.c scrub ok 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.152713+0000 osd.3 (osd.3) 251 : cluster 0 4.14 scrub starts 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.155570+0000 osd.3 (osd.3) 252 : cluster 0 4.14 scrub ok 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.756947+0000 osd.5 (osd.5) 221 : cluster 0 2.3 scrub starts 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.758983+0000 osd.5 (osd.5) 222 : cluster 0 2.3 scrub ok 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.863643+0000 mon.a (mon.0) 2509 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: audit 2024-04-02T04:53:55.867967+0000 mon.a (mon.0) 2510 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-47", "mode": "writeback"}]': finished 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:55.875870+0000 mon.a (mon.0) 2511 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: audit 2024-04-02T04:53:55.882064+0000 mon.c (mon.2) 714 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: audit 2024-04-02T04:53:55.883094+0000 mon.a (mon.0) 2512 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: audit 2024-04-02T04:53:55.884191+0000 mon.a (mon.0) 2513 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.022659+0000 osd.7 (osd.7) 5 : cluster 0 6.5 scrub starts 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.024442+0000 osd.7 (osd.7) 6 : cluster 0 6.5 scrub ok 2024-04-02T04:53:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:56 smithi098 bash[23790]: audit 2024-04-02T04:53:56.155088+0000 mon.c (mon.2) 715 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:56.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:56.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:56.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:57.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:57.132 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:57.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:57.261 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:57.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:57.393 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:57.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.073750+0000 mgr.y (mgr.14538) 405 : cluster 0 pgmap v571: 356 pgs: 38 peering, 3 creating+activating, 25 creating+peering, 3 creating+activating+undersized, 57 active+undersized, 207 active+clean, 23 active+undersized+degraded; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 192/1494 objects degraded (12.851%) 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.108687+0000 osd.3 (osd.3) 253 : cluster 0 4.0 scrub starts 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.111640+0000 osd.3 (osd.3) 254 : cluster 0 4.0 scrub ok 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.867972+0000 mon.a (mon.0) 2514 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: audit 2024-04-02T04:53:56.874655+0000 mon.a (mon.0) 2515 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: audit 2024-04-02T04:53:56.874752+0000 mon.a (mon.0) 2516 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:56.878313+0000 mon.a (mon.0) 2517 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: audit 2024-04-02T04:53:56.880083+0000 mon.c (mon.2) 716 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: audit 2024-04-02T04:53:56.882786+0000 mon.a (mon.0) 2518 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: audit 2024-04-02T04:53:56.883014+0000 mon.a (mon.0) 2519 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:57.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:57.030018+0000 osd.7 (osd.7) 7 : cluster 0 3.13 scrub starts 2024-04-02T04:53:57.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: cluster 2024-04-02T04:53:57.032533+0000 osd.7 (osd.7) 8 : cluster 0 3.13 scrub ok 2024-04-02T04:53:57.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:57 smithi098 bash[23790]: audit 2024-04-02T04:53:57.157070+0000 mon.c (mon.2) 717 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:57.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:57.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.073750+0000 mgr.y (mgr.14538) 405 : cluster 0 pgmap v571: 356 pgs: 38 peering, 3 creating+activating, 25 creating+peering, 3 creating+activating+undersized, 57 active+undersized, 207 active+clean, 23 active+undersized+degraded; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 192/1494 objects degraded (12.851%) 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.108687+0000 osd.3 (osd.3) 253 : cluster 0 4.0 scrub starts 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.111640+0000 osd.3 (osd.3) 254 : cluster 0 4.0 scrub ok 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.867972+0000 mon.a (mon.0) 2514 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: audit 2024-04-02T04:53:56.874655+0000 mon.a (mon.0) 2515 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: audit 2024-04-02T04:53:56.874752+0000 mon.a (mon.0) 2516 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:56.878313+0000 mon.a (mon.0) 2517 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: audit 2024-04-02T04:53:56.880083+0000 mon.c (mon.2) 716 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: audit 2024-04-02T04:53:56.882786+0000 mon.a (mon.0) 2518 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: audit 2024-04-02T04:53:56.883014+0000 mon.a (mon.0) 2519 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:57.030018+0000 osd.7 (osd.7) 7 : cluster 0 3.13 scrub starts 2024-04-02T04:53:57.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: cluster 2024-04-02T04:53:57.032533+0000 osd.7 (osd.7) 8 : cluster 0 3.13 scrub ok 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[28904]: audit 2024-04-02T04:53:57.157070+0000 mon.c (mon.2) 717 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.073750+0000 mgr.y (mgr.14538) 405 : cluster 0 pgmap v571: 356 pgs: 38 peering, 3 creating+activating, 25 creating+peering, 3 creating+activating+undersized, 57 active+undersized, 207 active+clean, 23 active+undersized+degraded; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 192/1494 objects degraded (12.851%) 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.108687+0000 osd.3 (osd.3) 253 : cluster 0 4.0 scrub starts 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.111640+0000 osd.3 (osd.3) 254 : cluster 0 4.0 scrub ok 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.867972+0000 mon.a (mon.0) 2514 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: audit 2024-04-02T04:53:56.874655+0000 mon.a (mon.0) 2515 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: audit 2024-04-02T04:53:56.874752+0000 mon.a (mon.0) 2516 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:56.878313+0000 mon.a (mon.0) 2517 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: audit 2024-04-02T04:53:56.880083+0000 mon.c (mon.2) 716 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: audit 2024-04-02T04:53:56.882786+0000 mon.a (mon.0) 2518 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]: dispatch 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: audit 2024-04-02T04:53:56.883014+0000 mon.a (mon.0) 2519 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:53:57.636 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:57.030018+0000 osd.7 (osd.7) 7 : cluster 0 3.13 scrub starts 2024-04-02T04:53:57.637 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: cluster 2024-04-02T04:53:57.032533+0000 osd.7 (osd.7) 8 : cluster 0 3.13 scrub ok 2024-04-02T04:53:57.637 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:57 smithi069 bash[21528]: audit 2024-04-02T04:53:57.157070+0000 mon.c (mon.2) 717 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:57.651 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:53:57.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:57.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:53:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:53:57.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:57.908 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:57.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:58.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:58.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:58.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:58.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:58.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:58.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.558 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:58.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:58.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:58.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:58.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:58.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:58.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.878101+0000 mon.a (mon.0) 2520 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:58.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.878171+0000 mon.a (mon.0) 2521 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:58.922 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: cluster 2024-04-02T04:53:57.881829+0000 mon.a (mon.0) 2522 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.888395+0000 mon.c (mon.2) 718 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.888947+0000 mon.a (mon.0) 2523 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.919705+0000 mon.a (mon.0) 2524 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.920766+0000 mon.a (mon.0) 2525 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:57.921144+0000 mon.a (mon.0) 2526 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.077533+0000 mgr.y (mgr.14538) 406 : cluster 0 pgmap v574: 356 pgs: 3 active+clean+snaptrim, 38 peering, 314 active+clean, 1 active+undersized+degraded; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s; 39/1383 objects degraded (2.820%); 259 B/s, 25 objects/s recovering 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.145160+0000 osd.3 (osd.3) 255 : cluster 0 6.6 scrub starts 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.149600+0000 osd.3 (osd.3) 256 : cluster 0 6.6 scrub ok 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: audit 2024-04-02T04:53:58.158856+0000 mon.c (mon.2) 719 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.556943+0000 osd.4 (osd.4) 221 : cluster 0 3.14 scrub starts 2024-04-02T04:53:58.923 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:53:58 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.561423+0000 osd.4 (osd.4) 222 : cluster 0 3.14 scrub ok 2024-04-02T04:53:58.936 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:58.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:53:58.984 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:53:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.878101+0000 mon.a (mon.0) 2520 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.878171+0000 mon.a (mon.0) 2521 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: cluster 2024-04-02T04:53:57.881829+0000 mon.a (mon.0) 2522 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.888395+0000 mon.c (mon.2) 718 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.888947+0000 mon.a (mon.0) 2523 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.919705+0000 mon.a (mon.0) 2524 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.920766+0000 mon.a (mon.0) 2525 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:57.921144+0000 mon.a (mon.0) 2526 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.077533+0000 mgr.y (mgr.14538) 406 : cluster 0 pgmap v574: 356 pgs: 3 active+clean+snaptrim, 38 peering, 314 active+clean, 1 active+undersized+degraded; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s; 39/1383 objects degraded (2.820%); 259 B/s, 25 objects/s recovering 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.145160+0000 osd.3 (osd.3) 255 : cluster 0 6.6 scrub starts 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.149600+0000 osd.3 (osd.3) 256 : cluster 0 6.6 scrub ok 2024-04-02T04:53:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: audit 2024-04-02T04:53:58.158856+0000 mon.c (mon.2) 719 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.556943+0000 osd.4 (osd.4) 221 : cluster 0 3.14 scrub starts 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.561423+0000 osd.4 (osd.4) 222 : cluster 0 3.14 scrub ok 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.878101+0000 mon.a (mon.0) 2520 : audit 1 from='client.? 172.21.15.69:0/2108977557' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-72153-61"}]': finished 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.878171+0000 mon.a (mon.0) 2521 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: cluster 2024-04-02T04:53:57.881829+0000 mon.a (mon.0) 2522 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.888395+0000 mon.c (mon.2) 718 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.888947+0000 mon.a (mon.0) 2523 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.919705+0000 mon.a (mon.0) 2524 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.920766+0000 mon.a (mon.0) 2525 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:57.921144+0000 mon.a (mon.0) 2526 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.077533+0000 mgr.y (mgr.14538) 406 : cluster 0 pgmap v574: 356 pgs: 3 active+clean+snaptrim, 38 peering, 314 active+clean, 1 active+undersized+degraded; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s; 39/1383 objects degraded (2.820%); 259 B/s, 25 objects/s recovering 2024-04-02T04:53:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.145160+0000 osd.3 (osd.3) 255 : cluster 0 6.6 scrub starts 2024-04-02T04:53:58.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.149600+0000 osd.3 (osd.3) 256 : cluster 0 6.6 scrub ok 2024-04-02T04:53:58.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: audit 2024-04-02T04:53:58.158856+0000 mon.c (mon.2) 719 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:53:58.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.556943+0000 osd.4 (osd.4) 221 : cluster 0 3.14 scrub starts 2024-04-02T04:53:58.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:53:58 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.561423+0000 osd.4 (osd.4) 222 : cluster 0 3.14 scrub ok 2024-04-02T04:53:59.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:53:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:53:59.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:59.193 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:53:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:53:59.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:59.322 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:53:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:53:59.425 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:59.451 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:59.554 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:59.581 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:53:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:53:59.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:59.712 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:53:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:53:59.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:59.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:53:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:53:59.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:53:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:53:59.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:00.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:00.099 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:00.203 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:00.229 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:00.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:00.359 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:00.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:00.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.878612+0000 mon.a (mon.0) 2527 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.879008+0000 mon.a (mon.0) 2528 : cluster 3 Health check update: Degraded data redundancy: 39/1383 objects degraded (2.820%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:58.880337+0000 mon.a (mon.0) 2529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:58.880437+0000 mon.a (mon.0) 2530 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:58.884252+0000 mon.a (mon.0) 2531 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:58.885509+0000 mon.c (mon.2) 720 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:58.889103+0000 mon.a (mon.0) 2532 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-72153-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:00.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:58.889414+0000 mon.a (mon.0) 2533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:59.101312+0000 osd.7 (osd.7) 9 : cluster 0 2.b scrub starts 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:59.105127+0000 osd.7 (osd.7) 10 : cluster 0 2.b scrub ok 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:59.160508+0000 mon.c (mon.2) 721 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:59.317652+0000 osd.1 (osd.1) 235 : cluster 0 2.4 scrub starts 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:59.321214+0000 osd.1 (osd.1) 236 : cluster 0 2.4 scrub ok 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:59.516360+0000 osd.4 (osd.4) 223 : cluster 0 2.f scrub starts 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: cluster 2024-04-02T04:53:59.519519+0000 osd.4 (osd.4) 224 : cluster 0 2.f scrub ok 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[28904]: audit 2024-04-02T04:53:59.884459+0000 mon.a (mon.0) 2534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.878612+0000 mon.a (mon.0) 2527 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.879008+0000 mon.a (mon.0) 2528 : cluster 3 Health check update: Degraded data redundancy: 39/1383 objects degraded (2.820%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-02T04:54:00.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:58.880337+0000 mon.a (mon.0) 2529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:58.880437+0000 mon.a (mon.0) 2530 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:58.884252+0000 mon.a (mon.0) 2531 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:58.885509+0000 mon.c (mon.2) 720 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:58.889103+0000 mon.a (mon.0) 2532 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-72153-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:58.889414+0000 mon.a (mon.0) 2533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:59.101312+0000 osd.7 (osd.7) 9 : cluster 0 2.b scrub starts 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:59.105127+0000 osd.7 (osd.7) 10 : cluster 0 2.b scrub ok 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:59.160508+0000 mon.c (mon.2) 721 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:59.317652+0000 osd.1 (osd.1) 235 : cluster 0 2.4 scrub starts 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:59.321214+0000 osd.1 (osd.1) 236 : cluster 0 2.4 scrub ok 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:59.516360+0000 osd.4 (osd.4) 223 : cluster 0 2.f scrub starts 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: cluster 2024-04-02T04:53:59.519519+0000 osd.4 (osd.4) 224 : cluster 0 2.f scrub ok 2024-04-02T04:54:00.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:00 smithi069 bash[21528]: audit 2024-04-02T04:53:59.884459+0000 mon.a (mon.0) 2534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T04:54:00.619 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:00.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:00.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.878612+0000 mon.a (mon.0) 2527 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:00.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.879008+0000 mon.a (mon.0) 2528 : cluster 3 Health check update: Degraded data redundancy: 39/1383 objects degraded (2.820%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:58.880337+0000 mon.a (mon.0) 2529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:58.880437+0000 mon.a (mon.0) 2530 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:58.884252+0000 mon.a (mon.0) 2531 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:58.885509+0000 mon.c (mon.2) 720 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:58.889103+0000 mon.a (mon.0) 2532 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-72153-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:58.889414+0000 mon.a (mon.0) 2533 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:59.101312+0000 osd.7 (osd.7) 9 : cluster 0 2.b scrub starts 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:59.105127+0000 osd.7 (osd.7) 10 : cluster 0 2.b scrub ok 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:59.160508+0000 mon.c (mon.2) 721 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:59.317652+0000 osd.1 (osd.1) 235 : cluster 0 2.4 scrub starts 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:59.321214+0000 osd.1 (osd.1) 236 : cluster 0 2.4 scrub ok 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:59.516360+0000 osd.4 (osd.4) 223 : cluster 0 2.f scrub starts 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: cluster 2024-04-02T04:53:59.519519+0000 osd.4 (osd.4) 224 : cluster 0 2.f scrub ok 2024-04-02T04:54:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:00 smithi098 bash[23790]: audit 2024-04-02T04:53:59.884459+0000 mon.a (mon.0) 2534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T04:54:00.749 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:00.759 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:00.760 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:00.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:00.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:00.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:01.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:01.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:01.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:01.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:01.262 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:01.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:01.390 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:01.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:53:59.891490+0000 mon.c (mon.2) 722 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T04:54:01.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.078496+0000 mgr.y (mgr.14538) 407 : cluster 0 pgmap v577: 356 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s wr, 8 op/s; 471 B/s, 47 objects/s recovering 2024-04-02T04:54:01.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.145237+0000 osd.7 (osd.7) 11 : cluster 0 6.1c scrub starts 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.147815+0000 osd.7 (osd.7) 12 : cluster 0 6.1c scrub ok 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:00.161673+0000 mon.c (mon.2) 723 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.339781+0000 mon.a (mon.0) 2535 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:00.341496+0000 mon.a (mon.0) 2536 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.884536+0000 mon.a (mon.0) 2537 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 33 pgs peering) 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.884602+0000 mon.a (mon.0) 2538 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1383 objects degraded (2.820%), 1 pg degraded, 1 pg undersized) 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:00.894808+0000 mon.a (mon.0) 2539 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-72153-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:00.894927+0000 mon.a (mon.0) 2540 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.903254+0000 mon.a (mon.0) 2541 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:00.917697+0000 mon.c (mon.2) 724 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:00.918447+0000 mon.a (mon.0) 2542 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.928103+0000 osd.2 (osd.2) 180 : cluster 0 2.e scrub starts 2024-04-02T04:54:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.929843+0000 osd.2 (osd.2) 181 : cluster 0 2.e scrub ok 2024-04-02T04:54:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:01 smithi098 bash[23790]: audit 2024-04-02T04:54:01.163042+0000 mon.c (mon.2) 725 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:01.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:01.599 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:53:59.891490+0000 mon.c (mon.2) 722 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T04:54:01.599 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.078496+0000 mgr.y (mgr.14538) 407 : cluster 0 pgmap v577: 356 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s wr, 8 op/s; 471 B/s, 47 objects/s recovering 2024-04-02T04:54:01.599 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.145237+0000 osd.7 (osd.7) 11 : cluster 0 6.1c scrub starts 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.147815+0000 osd.7 (osd.7) 12 : cluster 0 6.1c scrub ok 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:00.161673+0000 mon.c (mon.2) 723 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.339781+0000 mon.a (mon.0) 2535 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:00.341496+0000 mon.a (mon.0) 2536 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.884536+0000 mon.a (mon.0) 2537 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 33 pgs peering) 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.884602+0000 mon.a (mon.0) 2538 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1383 objects degraded (2.820%), 1 pg degraded, 1 pg undersized) 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:00.894808+0000 mon.a (mon.0) 2539 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-72153-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:00.894927+0000 mon.a (mon.0) 2540 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.903254+0000 mon.a (mon.0) 2541 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:00.917697+0000 mon.c (mon.2) 724 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:00.918447+0000 mon.a (mon.0) 2542 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T04:54:01.600 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.928103+0000 osd.2 (osd.2) 180 : cluster 0 2.e scrub starts 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.929843+0000 osd.2 (osd.2) 181 : cluster 0 2.e scrub ok 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[21528]: audit 2024-04-02T04:54:01.163042+0000 mon.c (mon.2) 725 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:53:59.891490+0000 mon.c (mon.2) 722 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.078496+0000 mgr.y (mgr.14538) 407 : cluster 0 pgmap v577: 356 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s wr, 8 op/s; 471 B/s, 47 objects/s recovering 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.145237+0000 osd.7 (osd.7) 11 : cluster 0 6.1c scrub starts 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.147815+0000 osd.7 (osd.7) 12 : cluster 0 6.1c scrub ok 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:00.161673+0000 mon.c (mon.2) 723 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.339781+0000 mon.a (mon.0) 2535 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:00.341496+0000 mon.a (mon.0) 2536 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.884536+0000 mon.a (mon.0) 2537 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 33 pgs peering) 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.884602+0000 mon.a (mon.0) 2538 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1383 objects degraded (2.820%), 1 pg degraded, 1 pg undersized) 2024-04-02T04:54:01.601 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:00.894808+0000 mon.a (mon.0) 2539 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-72153-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:00.894927+0000 mon.a (mon.0) 2540 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.903254+0000 mon.a (mon.0) 2541 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:00.917697+0000 mon.c (mon.2) 724 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:00.918447+0000 mon.a (mon.0) 2542 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.928103+0000 osd.2 (osd.2) 180 : cluster 0 2.e scrub starts 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.929843+0000 osd.2 (osd.2) 181 : cluster 0 2.e scrub ok 2024-04-02T04:54:01.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:01 smithi069 bash[28904]: audit 2024-04-02T04:54:01.163042+0000 mon.c (mon.2) 725 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:01.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:01.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:01.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:01.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:01.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:02.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:02.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:02.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:02.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:02.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.905669+0000 osd.6 (osd.6) 185 : cluster 0 4.5 scrub starts 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:00.909017+0000 osd.6 (osd.6) 186 : cluster 0 4.5 scrub ok 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.085812+0000 osd.3 (osd.3) 257 : cluster 0 6.b scrub starts 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.088121+0000 osd.3 (osd.3) 258 : cluster 0 6.b scrub ok 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.299403+0000 osd.1 (osd.1) 237 : cluster 0 2.a scrub starts 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.301072+0000 osd.1 (osd.1) 238 : cluster 0 2.a scrub ok 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: audit 2024-04-02T04:54:01.897675+0000 mon.a (mon.0) 2543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.902003+0000 osd.2 (osd.2) 182 : cluster 0 6.14 scrub starts 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.905589+0000 mon.a (mon.0) 2544 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.922326+0000 osd.2 (osd.2) 183 : cluster 0 6.14 scrub ok 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: audit 2024-04-02T04:54:02.000215+0000 mon.c (mon.2) 726 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: audit 2024-04-02T04:54:02.000783+0000 mon.a (mon.0) 2545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:02 smithi098 bash[23790]: audit 2024-04-02T04:54:02.164640+0000 mon.c (mon.2) 727 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:02.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:02.554 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:02.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:02.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.905669+0000 osd.6 (osd.6) 185 : cluster 0 4.5 scrub starts 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:00.909017+0000 osd.6 (osd.6) 186 : cluster 0 4.5 scrub ok 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.085812+0000 osd.3 (osd.3) 257 : cluster 0 6.b scrub starts 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.088121+0000 osd.3 (osd.3) 258 : cluster 0 6.b scrub ok 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.299403+0000 osd.1 (osd.1) 237 : cluster 0 2.a scrub starts 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.301072+0000 osd.1 (osd.1) 238 : cluster 0 2.a scrub ok 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: audit 2024-04-02T04:54:01.897675+0000 mon.a (mon.0) 2543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.902003+0000 osd.2 (osd.2) 182 : cluster 0 6.14 scrub starts 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.905589+0000 mon.a (mon.0) 2544 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.922326+0000 osd.2 (osd.2) 183 : cluster 0 6.14 scrub ok 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: audit 2024-04-02T04:54:02.000215+0000 mon.c (mon.2) 726 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:02.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: audit 2024-04-02T04:54:02.000783+0000 mon.a (mon.0) 2545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[21528]: audit 2024-04-02T04:54:02.164640+0000 mon.c (mon.2) 727 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.905669+0000 osd.6 (osd.6) 185 : cluster 0 4.5 scrub starts 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:00.909017+0000 osd.6 (osd.6) 186 : cluster 0 4.5 scrub ok 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.085812+0000 osd.3 (osd.3) 257 : cluster 0 6.b scrub starts 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.088121+0000 osd.3 (osd.3) 258 : cluster 0 6.b scrub ok 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.299403+0000 osd.1 (osd.1) 237 : cluster 0 2.a scrub starts 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.301072+0000 osd.1 (osd.1) 238 : cluster 0 2.a scrub ok 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: audit 2024-04-02T04:54:01.897675+0000 mon.a (mon.0) 2543 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.902003+0000 osd.2 (osd.2) 182 : cluster 0 6.14 scrub starts 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.905589+0000 mon.a (mon.0) 2544 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.922326+0000 osd.2 (osd.2) 183 : cluster 0 6.14 scrub ok 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: audit 2024-04-02T04:54:02.000215+0000 mon.c (mon.2) 726 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:02.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: audit 2024-04-02T04:54:02.000783+0000 mon.a (mon.0) 2545 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:02.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:02 smithi069 bash[28904]: audit 2024-04-02T04:54:02.164640+0000 mon.c (mon.2) 727 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:02.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:02.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:02.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:02.945 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:02.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:03.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:03.075 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:03.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:03.204 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:03.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:03.329 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:03.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:54:03.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:03.478 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:54:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:54:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:54:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:54:03.478 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:54:03.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.946723+0000 osd.6 (osd.6) 187 : cluster 0 4.a scrub starts 2024-04-02T04:54:03.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:01.950464+0000 osd.6 (osd.6) 188 : cluster 0 4.a scrub ok 2024-04-02T04:54:03.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.079108+0000 mgr.y (mgr.14538) 408 : cluster 0 pgmap v580: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s wr, 4 op/s; 211 B/s, 21 objects/s recovering 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.336088+0000 osd.1 (osd.1) 239 : cluster 0 40.0 scrub starts 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.337680+0000 osd.1 (osd.1) 240 : cluster 0 40.0 scrub ok 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.767397+0000 osd.5 (osd.5) 223 : cluster 0 2.12 scrub starts 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.769702+0000 osd.5 (osd.5) 224 : cluster 0 2.12 scrub ok 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.897576+0000 mon.a (mon.0) 2546 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:02.910914+0000 mon.a (mon.0) 2547 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: cluster 2024-04-02T04:54:02.919055+0000 mon.a (mon.0) 2548 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:02.920945+0000 mon.c (mon.2) 728 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:02.922928+0000 mon.a (mon.0) 2549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:02.923671+0000 mon.a (mon.0) 2550 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:02.940180+0000 mon.a (mon.0) 2551 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:02.942514+0000 mon.b (mon.1) 211 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:03.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[21528]: audit 2024-04-02T04:54:03.166102+0000 mon.c (mon.2) 729 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.946723+0000 osd.6 (osd.6) 187 : cluster 0 4.a scrub starts 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:01.950464+0000 osd.6 (osd.6) 188 : cluster 0 4.a scrub ok 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.079108+0000 mgr.y (mgr.14538) 408 : cluster 0 pgmap v580: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s wr, 4 op/s; 211 B/s, 21 objects/s recovering 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.336088+0000 osd.1 (osd.1) 239 : cluster 0 40.0 scrub starts 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.337680+0000 osd.1 (osd.1) 240 : cluster 0 40.0 scrub ok 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.767397+0000 osd.5 (osd.5) 223 : cluster 0 2.12 scrub starts 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.769702+0000 osd.5 (osd.5) 224 : cluster 0 2.12 scrub ok 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.897576+0000 mon.a (mon.0) 2546 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:02.910914+0000 mon.a (mon.0) 2547 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: cluster 2024-04-02T04:54:02.919055+0000 mon.a (mon.0) 2548 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:02.920945+0000 mon.c (mon.2) 728 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:03.558 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:02.922928+0000 mon.a (mon.0) 2549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:03.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:02.923671+0000 mon.a (mon.0) 2550 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:03.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:02.940180+0000 mon.a (mon.0) 2551 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:03.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:02.942514+0000 mon.b (mon.1) 211 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:03.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:03 smithi069 bash[28904]: audit 2024-04-02T04:54:03.166102+0000 mon.c (mon.2) 729 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:03.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:03.574 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.946723+0000 osd.6 (osd.6) 187 : cluster 0 4.a scrub starts 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:01.950464+0000 osd.6 (osd.6) 188 : cluster 0 4.a scrub ok 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.079108+0000 mgr.y (mgr.14538) 408 : cluster 0 pgmap v580: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s wr, 4 op/s; 211 B/s, 21 objects/s recovering 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.336088+0000 osd.1 (osd.1) 239 : cluster 0 40.0 scrub starts 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.337680+0000 osd.1 (osd.1) 240 : cluster 0 40.0 scrub ok 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.767397+0000 osd.5 (osd.5) 223 : cluster 0 2.12 scrub starts 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.769702+0000 osd.5 (osd.5) 224 : cluster 0 2.12 scrub ok 2024-04-02T04:54:03.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.897576+0000 mon.a (mon.0) 2546 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:02.910914+0000 mon.a (mon.0) 2547 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: cluster 2024-04-02T04:54:02.919055+0000 mon.a (mon.0) 2548 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:02.920945+0000 mon.c (mon.2) 728 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:02.922928+0000 mon.a (mon.0) 2549 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:02.923671+0000 mon.a (mon.0) 2550 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:02.940180+0000 mon.a (mon.0) 2551 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:02.942514+0000 mon.b (mon.1) 211 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:03.576 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:03 smithi098 bash[23790]: audit 2024-04-02T04:54:03.166102+0000 mon.c (mon.2) 729 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:03.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:03.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:03.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:03.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:04.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:04.102 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:04.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:04.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:04.360 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:04.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:04.489 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:04.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:04.602 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.203266+0000 osd.7 (osd.7) 13 : cluster 0 6.15 scrub starts 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.204672+0000 osd.7 (osd.7) 14 : cluster 0 6.15 scrub ok 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.232534+0000 mgr.y (mgr.14538) 409 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.561805+0000 osd.4 (osd.4) 225 : cluster 0 3.9 scrub starts 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.566016+0000 osd.4 (osd.4) 226 : cluster 0 3.9 scrub ok 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.728919+0000 osd.5 (osd.5) 225 : cluster 0 2.18 scrub starts 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.730059+0000 osd.5 (osd.5) 226 : cluster 0 2.18 scrub ok 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.881948+0000 osd.2 (osd.2) 184 : cluster 0 40.b scrub starts 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.883764+0000 osd.2 (osd.2) 185 : cluster 0 40.b scrub ok 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.912943+0000 mon.a (mon.0) 2552 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]': finished 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.913107+0000 mon.a (mon.0) 2553 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: cluster 2024-04-02T04:54:03.917615+0000 mon.a (mon.0) 2554 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.920659+0000 mon.a (mon.0) 2555 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.976019+0000 mon.c (mon.2) 730 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.976618+0000 mon.a (mon.0) 2556 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:04.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.977568+0000 mon.c (mon.2) 731 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:03.978086+0000 mon.a (mon.0) 2557 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[28904]: audit 2024-04-02T04:54:04.167986+0000 mon.c (mon.2) 732 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.203266+0000 osd.7 (osd.7) 13 : cluster 0 6.15 scrub starts 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.204672+0000 osd.7 (osd.7) 14 : cluster 0 6.15 scrub ok 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.232534+0000 mgr.y (mgr.14538) 409 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.561805+0000 osd.4 (osd.4) 225 : cluster 0 3.9 scrub starts 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.566016+0000 osd.4 (osd.4) 226 : cluster 0 3.9 scrub ok 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.728919+0000 osd.5 (osd.5) 225 : cluster 0 2.18 scrub starts 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.730059+0000 osd.5 (osd.5) 226 : cluster 0 2.18 scrub ok 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.881948+0000 osd.2 (osd.2) 184 : cluster 0 40.b scrub starts 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.883764+0000 osd.2 (osd.2) 185 : cluster 0 40.b scrub ok 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.912943+0000 mon.a (mon.0) 2552 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]': finished 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.913107+0000 mon.a (mon.0) 2553 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:04.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: cluster 2024-04-02T04:54:03.917615+0000 mon.a (mon.0) 2554 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-04-02T04:54:04.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.920659+0000 mon.a (mon.0) 2555 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:04.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.976019+0000 mon.c (mon.2) 730 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:04.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.976618+0000 mon.a (mon.0) 2556 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:04.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.977568+0000 mon.c (mon.2) 731 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:04.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:03.978086+0000 mon.a (mon.0) 2557 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:04.605 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:04 smithi069 bash[21528]: audit 2024-04-02T04:54:04.167986+0000 mon.c (mon.2) 732 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:04.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:04.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:04.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.203266+0000 osd.7 (osd.7) 13 : cluster 0 6.15 scrub starts 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.204672+0000 osd.7 (osd.7) 14 : cluster 0 6.15 scrub ok 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.232534+0000 mgr.y (mgr.14538) 409 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.561805+0000 osd.4 (osd.4) 225 : cluster 0 3.9 scrub starts 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.566016+0000 osd.4 (osd.4) 226 : cluster 0 3.9 scrub ok 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.728919+0000 osd.5 (osd.5) 225 : cluster 0 2.18 scrub starts 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.730059+0000 osd.5 (osd.5) 226 : cluster 0 2.18 scrub ok 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.881948+0000 osd.2 (osd.2) 184 : cluster 0 40.b scrub starts 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.883764+0000 osd.2 (osd.2) 185 : cluster 0 40.b scrub ok 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.912943+0000 mon.a (mon.0) 2552 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]': finished 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.913107+0000 mon.a (mon.0) 2553 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: cluster 2024-04-02T04:54:03.917615+0000 mon.a (mon.0) 2554 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.920659+0000 mon.a (mon.0) 2555 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]: dispatch 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.976019+0000 mon.c (mon.2) 730 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.976618+0000 mon.a (mon.0) 2556 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.977568+0000 mon.c (mon.2) 731 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:03.978086+0000 mon.a (mon.0) 2557 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-47"}]: dispatch 2024-04-02T04:54:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:04 smithi098 bash[23790]: audit 2024-04-02T04:54:04.167986+0000 mon.c (mon.2) 732 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:04.748 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:04.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:04.881 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:04.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:04.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:05.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:05.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:05.130 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:05.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:05.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:05.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:05.378 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:05.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:05.509 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:05.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.067544+0000 osd.3 (osd.3) 259 : cluster 0 4.11 scrub starts 2024-04-02T04:54:05.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.069395+0000 osd.3 (osd.3) 260 : cluster 0 4.11 scrub ok 2024-04-02T04:54:05.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.080413+0000 mgr.y (mgr.14538) 410 : cluster 0 pgmap v583: 356 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.336049+0000 osd.1 (osd.1) 241 : cluster 0 41.c scrub starts 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.337864+0000 osd.1 (osd.1) 242 : cluster 0 41.c scrub ok 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: audit 2024-04-02T04:54:04.867706+0000 mon.a (mon.0) 2558 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.876042+0000 mon.a (mon.0) 2559 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: audit 2024-04-02T04:54:04.925665+0000 mon.b (mon.1) 212 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: audit 2024-04-02T04:54:05.169456+0000 mon.c (mon.2) 733 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[21528]: audit 2024-04-02T04:54:05.345661+0000 mon.a (mon.0) 2560 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.067544+0000 osd.3 (osd.3) 259 : cluster 0 4.11 scrub starts 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.069395+0000 osd.3 (osd.3) 260 : cluster 0 4.11 scrub ok 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.080413+0000 mgr.y (mgr.14538) 410 : cluster 0 pgmap v583: 356 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.336049+0000 osd.1 (osd.1) 241 : cluster 0 41.c scrub starts 2024-04-02T04:54:05.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.337864+0000 osd.1 (osd.1) 242 : cluster 0 41.c scrub ok 2024-04-02T04:54:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: audit 2024-04-02T04:54:04.867706+0000 mon.a (mon.0) 2558 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.876042+0000 mon.a (mon.0) 2559 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-04-02T04:54:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: audit 2024-04-02T04:54:04.925665+0000 mon.b (mon.1) 212 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: audit 2024-04-02T04:54:05.169456+0000 mon.c (mon.2) 733 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:05.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:05 smithi069 bash[28904]: audit 2024-04-02T04:54:05.345661+0000 mon.a (mon.0) 2560 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:05.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.067544+0000 osd.3 (osd.3) 259 : cluster 0 4.11 scrub starts 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.069395+0000 osd.3 (osd.3) 260 : cluster 0 4.11 scrub ok 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.080413+0000 mgr.y (mgr.14538) 410 : cluster 0 pgmap v583: 356 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 350 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.336049+0000 osd.1 (osd.1) 241 : cluster 0 41.c scrub starts 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.337864+0000 osd.1 (osd.1) 242 : cluster 0 41.c scrub ok 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: audit 2024-04-02T04:54:04.867706+0000 mon.a (mon.0) 2558 : audit 1 from='client.? 172.21.15.69:0/778228701' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-72153-62"}]': finished 2024-04-02T04:54:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.876042+0000 mon.a (mon.0) 2559 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-04-02T04:54:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: audit 2024-04-02T04:54:04.925665+0000 mon.b (mon.1) 212 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: audit 2024-04-02T04:54:05.169456+0000 mon.c (mon.2) 733 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:05 smithi098 bash[23790]: audit 2024-04-02T04:54:05.345661+0000 mon.a (mon.0) 2560 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:05.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:05.767 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:05.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:05.906 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:05.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:06.009 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:06.035 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:06.138 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:06.163 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:06.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:06.294 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:06.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.963447+0000 osd.6 (osd.6) 189 : cluster 0 13.f scrub starts 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: cluster 2024-04-02T04:54:04.965184+0000 osd.6 (osd.6) 190 : cluster 0 13.f scrub ok 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.348460+0000 mon.b (mon.1) 213 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.357104+0000 mon.b (mon.1) 214 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.360482+0000 mon.a (mon.0) 2561 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.362379+0000 mon.a (mon.0) 2562 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: cluster 2024-04-02T04:54:05.879296+0000 osd.2 (osd.2) 186 : cluster 0 43.2 scrub starts 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: cluster 2024-04-02T04:54:05.882184+0000 osd.2 (osd.2) 187 : cluster 0 43.2 scrub ok 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.889685+0000 mon.a (mon.0) 2563 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: cluster 2024-04-02T04:54:05.895938+0000 mon.a (mon.0) 2564 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.898789+0000 mon.c (mon.2) 734 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.900133+0000 mon.b (mon.1) 215 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.408 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.900430+0000 mon.a (mon.0) 2565 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:05.907169+0000 mon.a (mon.0) 2566 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[28904]: audit 2024-04-02T04:54:06.170621+0000 mon.c (mon.2) 735 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.963447+0000 osd.6 (osd.6) 189 : cluster 0 13.f scrub starts 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: cluster 2024-04-02T04:54:04.965184+0000 osd.6 (osd.6) 190 : cluster 0 13.f scrub ok 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.348460+0000 mon.b (mon.1) 213 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.357104+0000 mon.b (mon.1) 214 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.360482+0000 mon.a (mon.0) 2561 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.362379+0000 mon.a (mon.0) 2562 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: cluster 2024-04-02T04:54:05.879296+0000 osd.2 (osd.2) 186 : cluster 0 43.2 scrub starts 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: cluster 2024-04-02T04:54:05.882184+0000 osd.2 (osd.2) 187 : cluster 0 43.2 scrub ok 2024-04-02T04:54:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.889685+0000 mon.a (mon.0) 2563 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:06.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: cluster 2024-04-02T04:54:05.895938+0000 mon.a (mon.0) 2564 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-04-02T04:54:06.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.898789+0000 mon.c (mon.2) 734 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:06.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.900133+0000 mon.b (mon.1) 215 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.900430+0000 mon.a (mon.0) 2565 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:06.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:05.907169+0000 mon.a (mon.0) 2566 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.410 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:06 smithi069 bash[21528]: audit 2024-04-02T04:54:06.170621+0000 mon.c (mon.2) 735 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:06.422 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:06.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:06.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:06.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.963447+0000 osd.6 (osd.6) 189 : cluster 0 13.f scrub starts 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: cluster 2024-04-02T04:54:04.965184+0000 osd.6 (osd.6) 190 : cluster 0 13.f scrub ok 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.348460+0000 mon.b (mon.1) 213 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.357104+0000 mon.b (mon.1) 214 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.360482+0000 mon.a (mon.0) 2561 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.362379+0000 mon.a (mon.0) 2562 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: cluster 2024-04-02T04:54:05.879296+0000 osd.2 (osd.2) 186 : cluster 0 43.2 scrub starts 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: cluster 2024-04-02T04:54:05.882184+0000 osd.2 (osd.2) 187 : cluster 0 43.2 scrub ok 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.889685+0000 mon.a (mon.0) 2563 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-72153-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: cluster 2024-04-02T04:54:05.895938+0000 mon.a (mon.0) 2564 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.898789+0000 mon.c (mon.2) 734 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.900133+0000 mon.b (mon.1) 215 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.900430+0000 mon.a (mon.0) 2565 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:06.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:05.907169+0000 mon.a (mon.0) 2566 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:06.668 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:06 smithi098 bash[23790]: audit 2024-04-02T04:54:06.170621+0000 mon.c (mon.2) 735 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:06.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-02T04:54:06.683 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:06.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:06.914 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:06.946 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:06.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:07.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:07.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:07.207 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:07.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:07.334 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:07.437 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:07.461 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:07.564 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:07.591 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:07.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:07.705 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: cluster 2024-04-02T04:54:05.617238+0000 osd.4 (osd.4) 227 : cluster 0 5.1 scrub starts 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: cluster 2024-04-02T04:54:05.619634+0000 osd.4 (osd.4) 228 : cluster 0 5.1 scrub ok 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: cluster 2024-04-02T04:54:06.081218+0000 mgr.y (mgr.14538) 411 : cluster 0 pgmap v586: 356 pgs: 32 unknown, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 318 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: cluster 2024-04-02T04:54:06.132155+0000 osd.3 (osd.3) 261 : cluster 0 13.19 scrub starts 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: cluster 2024-04-02T04:54:06.134217+0000 osd.3 (osd.3) 262 : cluster 0 13.19 scrub ok 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: audit 2024-04-02T04:54:06.898947+0000 mon.a (mon.0) 2567 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: audit 2024-04-02T04:54:06.946317+0000 mon.c (mon.2) 736 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: audit 2024-04-02T04:54:07.171734+0000 mon.c (mon.2) 737 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.339257+0000 mon.a (mon.0) 2568 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-04-02T04:54:07.706 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:07 smithi098 bash[23790]: audit 2024-04-02T04:54:07.343344+0000 mon.a (mon.0) 2569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:07.721 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:07.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: cluster 2024-04-02T04:54:05.617238+0000 osd.4 (osd.4) 227 : cluster 0 5.1 scrub starts 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: cluster 2024-04-02T04:54:05.619634+0000 osd.4 (osd.4) 228 : cluster 0 5.1 scrub ok 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: cluster 2024-04-02T04:54:06.081218+0000 mgr.y (mgr.14538) 411 : cluster 0 pgmap v586: 356 pgs: 32 unknown, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 318 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: cluster 2024-04-02T04:54:06.132155+0000 osd.3 (osd.3) 261 : cluster 0 13.19 scrub starts 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: cluster 2024-04-02T04:54:06.134217+0000 osd.3 (osd.3) 262 : cluster 0 13.19 scrub ok 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: audit 2024-04-02T04:54:06.898947+0000 mon.a (mon.0) 2567 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: audit 2024-04-02T04:54:06.946317+0000 mon.c (mon.2) 736 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: audit 2024-04-02T04:54:07.171734+0000 mon.c (mon.2) 737 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.339257+0000 mon.a (mon.0) 2568 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[21528]: audit 2024-04-02T04:54:07.343344+0000 mon.a (mon.0) 2569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: cluster 2024-04-02T04:54:05.617238+0000 osd.4 (osd.4) 227 : cluster 0 5.1 scrub starts 2024-04-02T04:54:07.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: cluster 2024-04-02T04:54:05.619634+0000 osd.4 (osd.4) 228 : cluster 0 5.1 scrub ok 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: cluster 2024-04-02T04:54:06.081218+0000 mgr.y (mgr.14538) 411 : cluster 0 pgmap v586: 356 pgs: 32 unknown, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 318 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: cluster 2024-04-02T04:54:06.132155+0000 osd.3 (osd.3) 261 : cluster 0 13.19 scrub starts 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: cluster 2024-04-02T04:54:06.134217+0000 osd.3 (osd.3) 262 : cluster 0 13.19 scrub ok 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: audit 2024-04-02T04:54:06.898947+0000 mon.a (mon.0) 2567 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: audit 2024-04-02T04:54:06.946317+0000 mon.c (mon.2) 736 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: audit 2024-04-02T04:54:07.171734+0000 mon.c (mon.2) 737 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.339257+0000 mon.a (mon.0) 2568 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-04-02T04:54:07.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:07 smithi069 bash[28904]: audit 2024-04-02T04:54:07.343344+0000 mon.a (mon.0) 2569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:07.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:07.851 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:07.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:07.977 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:07.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:08.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:08.103 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:08.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:08.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:08.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:08.361 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:08.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:08.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:08.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.752 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:08.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:08.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:08.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:08.993 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:54:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.621304+0000 osd.4 (osd.4) 229 : cluster 0 10.10 scrub starts 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.623870+0000 osd.4 (osd.4) 230 : cluster 0 10.10 scrub ok 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.627073+0000 osd.5 (osd.5) 227 : cluster 0 4.15 scrub starts 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.630045+0000 osd.5 (osd.5) 228 : cluster 0 4.15 scrub ok 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.828998+0000 osd.2 (osd.2) 188 : cluster 0 102.9 scrub starts 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.832691+0000 osd.2 (osd.2) 189 : cluster 0 102.9 scrub ok 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: audit 2024-04-02T04:54:07.901193+0000 mon.a (mon.0) 2570 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: audit 2024-04-02T04:54:07.901271+0000 mon.a (mon.0) 2571 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:07.905704+0000 mon.a (mon.0) 2572 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: audit 2024-04-02T04:54:07.907261+0000 mon.c (mon.2) 738 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: audit 2024-04-02T04:54:07.910532+0000 mon.a (mon.0) 2573 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.084010+0000 mgr.y (mgr.14538) 412 : cluster 0 pgmap v589: 364 pgs: 12 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 348 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[28904]: audit 2024-04-02T04:54:08.173199+0000 mon.c (mon.2) 739 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.621304+0000 osd.4 (osd.4) 229 : cluster 0 10.10 scrub starts 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.623870+0000 osd.4 (osd.4) 230 : cluster 0 10.10 scrub ok 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.627073+0000 osd.5 (osd.5) 227 : cluster 0 4.15 scrub starts 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.630045+0000 osd.5 (osd.5) 228 : cluster 0 4.15 scrub ok 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.828998+0000 osd.2 (osd.2) 188 : cluster 0 102.9 scrub starts 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.832691+0000 osd.2 (osd.2) 189 : cluster 0 102.9 scrub ok 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: audit 2024-04-02T04:54:07.901193+0000 mon.a (mon.0) 2570 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: audit 2024-04-02T04:54:07.901271+0000 mon.a (mon.0) 2571 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:54:08.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:07.905704+0000 mon.a (mon.0) 2572 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-04-02T04:54:08.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: audit 2024-04-02T04:54:07.907261+0000 mon.c (mon.2) 738 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:08.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: audit 2024-04-02T04:54:07.910532+0000 mon.a (mon.0) 2573 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:08.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.084010+0000 mgr.y (mgr.14538) 412 : cluster 0 pgmap v589: 364 pgs: 12 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 348 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:54:08.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:08 smithi069 bash[21528]: audit 2024-04-02T04:54:08.173199+0000 mon.c (mon.2) 739 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:09.006 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:09.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:09.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.621304+0000 osd.4 (osd.4) 229 : cluster 0 10.10 scrub starts 2024-04-02T04:54:09.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.623870+0000 osd.4 (osd.4) 230 : cluster 0 10.10 scrub ok 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.627073+0000 osd.5 (osd.5) 227 : cluster 0 4.15 scrub starts 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.630045+0000 osd.5 (osd.5) 228 : cluster 0 4.15 scrub ok 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.828998+0000 osd.2 (osd.2) 188 : cluster 0 102.9 scrub starts 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.832691+0000 osd.2 (osd.2) 189 : cluster 0 102.9 scrub ok 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: audit 2024-04-02T04:54:07.901193+0000 mon.a (mon.0) 2570 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-72153-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: audit 2024-04-02T04:54:07.901271+0000 mon.a (mon.0) 2571 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:07.905704+0000 mon.a (mon.0) 2572 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: audit 2024-04-02T04:54:07.907261+0000 mon.c (mon.2) 738 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: audit 2024-04-02T04:54:07.910532+0000 mon.a (mon.0) 2573 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.084010+0000 mgr.y (mgr.14538) 412 : cluster 0 pgmap v589: 364 pgs: 12 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 348 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:54:09.122 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:08 smithi098 bash[23790]: audit 2024-04-02T04:54:08.173199+0000 mon.c (mon.2) 739 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:09.134 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:09.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:09.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:09.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:09.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:09.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:09.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:09.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:09.771 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:09.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:09.903 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:09.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:10.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.207301+0000 osd.7 (osd.7) 15 : cluster 0 6.8 scrub starts 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.209699+0000 osd.7 (osd.7) 16 : cluster 0 6.8 scrub ok 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.663256+0000 osd.4 (osd.4) 231 : cluster 0 41.16 scrub starts 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.664541+0000 osd.4 (osd.4) 232 : cluster 0 41.16 scrub ok 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:08.910357+0000 mon.a (mon.0) 2574 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]': finished 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.918796+0000 mon.a (mon.0) 2575 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-04-02T04:54:10.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:08.920381+0000 mon.c (mon.2) 740 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]: dispatch 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:08.922035+0000 mon.a (mon.0) 2576 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]: dispatch 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.976550+0000 osd.6 (osd.6) 191 : cluster 0 2.7 scrub starts 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:08.978765+0000 osd.6 (osd.6) 192 : cluster 0 2.7 scrub ok 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.151449+0000 osd.3 (osd.3) 263 : cluster 0 2.1b scrub starts 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.153474+0000 osd.3 (osd.3) 264 : cluster 0 2.1b scrub ok 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:09.174460+0000 mon.c (mon.2) 741 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.867962+0000 mon.a (mon.0) 2577 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:09.869785+0000 mon.a (mon.0) 2578 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]': finished 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.874603+0000 mon.a (mon.0) 2579 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:09.880111+0000 mon.b (mon.1) 216 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[21528]: audit 2024-04-02T04:54:09.885400+0000 mon.a (mon.0) 2580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.207301+0000 osd.7 (osd.7) 15 : cluster 0 6.8 scrub starts 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.209699+0000 osd.7 (osd.7) 16 : cluster 0 6.8 scrub ok 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.663256+0000 osd.4 (osd.4) 231 : cluster 0 41.16 scrub starts 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.664541+0000 osd.4 (osd.4) 232 : cluster 0 41.16 scrub ok 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:08.910357+0000 mon.a (mon.0) 2574 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]': finished 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.918796+0000 mon.a (mon.0) 2575 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:08.920381+0000 mon.c (mon.2) 740 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]: dispatch 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:08.922035+0000 mon.a (mon.0) 2576 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]: dispatch 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.976550+0000 osd.6 (osd.6) 191 : cluster 0 2.7 scrub starts 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:08.978765+0000 osd.6 (osd.6) 192 : cluster 0 2.7 scrub ok 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.151449+0000 osd.3 (osd.3) 263 : cluster 0 2.1b scrub starts 2024-04-02T04:54:10.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.153474+0000 osd.3 (osd.3) 264 : cluster 0 2.1b scrub ok 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:09.174460+0000 mon.c (mon.2) 741 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.867962+0000 mon.a (mon.0) 2577 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:09.869785+0000 mon.a (mon.0) 2578 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]': finished 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.874603+0000 mon.a (mon.0) 2579 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:09.880111+0000 mon.b (mon.1) 216 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:09 smithi069 bash[28904]: audit 2024-04-02T04:54:09.885400+0000 mon.a (mon.0) 2580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.032 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:10.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.207301+0000 osd.7 (osd.7) 15 : cluster 0 6.8 scrub starts 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.209699+0000 osd.7 (osd.7) 16 : cluster 0 6.8 scrub ok 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.663256+0000 osd.4 (osd.4) 231 : cluster 0 41.16 scrub starts 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.664541+0000 osd.4 (osd.4) 232 : cluster 0 41.16 scrub ok 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:08.910357+0000 mon.a (mon.0) 2574 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-49"}]': finished 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.918796+0000 mon.a (mon.0) 2575 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:08.920381+0000 mon.c (mon.2) 740 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]: dispatch 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:08.922035+0000 mon.a (mon.0) 2576 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]: dispatch 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.976550+0000 osd.6 (osd.6) 191 : cluster 0 2.7 scrub starts 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:08.978765+0000 osd.6 (osd.6) 192 : cluster 0 2.7 scrub ok 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.151449+0000 osd.3 (osd.3) 263 : cluster 0 2.1b scrub starts 2024-04-02T04:54:10.163 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.153474+0000 osd.3 (osd.3) 264 : cluster 0 2.1b scrub ok 2024-04-02T04:54:10.164 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:09.174460+0000 mon.c (mon.2) 741 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:10.164 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.867962+0000 mon.a (mon.0) 2577 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:54:10.164 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:09.869785+0000 mon.a (mon.0) 2578 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-49", "mode": "readproxy"}]': finished 2024-04-02T04:54:10.164 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.874603+0000 mon.a (mon.0) 2579 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-04-02T04:54:10.164 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:09.880111+0000 mon.b (mon.1) 216 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.164 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:09 smithi098 bash[23790]: audit 2024-04-02T04:54:09.885400+0000 mon.a (mon.0) 2580 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:10.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:10.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:10.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:10.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:10.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:10.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.811 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:10.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:10.929 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.646081+0000 osd.4 (osd.4) 233 : cluster 0 10.1 scrub starts 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.648281+0000 osd.4 (osd.4) 234 : cluster 0 10.1 scrub ok 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.838955+0000 osd.2 (osd.2) 190 : cluster 0 102.1c scrub starts 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: cluster 2024-04-02T04:54:09.841450+0000 osd.2 (osd.2) 191 : cluster 0 102.1c scrub ok 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: cluster 2024-04-02T04:54:10.084932+0000 mgr.y (mgr.14538) 413 : cluster 0 pgmap v592: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: audit 2024-04-02T04:54:10.175875+0000 mon.c (mon.2) 742 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: audit 2024-04-02T04:54:10.872765+0000 mon.a (mon.0) 2581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: audit 2024-04-02T04:54:10.875029+0000 mon.b (mon.1) 217 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: cluster 2024-04-02T04:54:10.882617+0000 mon.a (mon.0) 2582 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-04-02T04:54:10.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:10 smithi098 bash[23790]: audit 2024-04-02T04:54:10.885685+0000 mon.a (mon.0) 2583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:10.949 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:10.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:11.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:11.080 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:11.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.646081+0000 osd.4 (osd.4) 233 : cluster 0 10.1 scrub starts 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.648281+0000 osd.4 (osd.4) 234 : cluster 0 10.1 scrub ok 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.838955+0000 osd.2 (osd.2) 190 : cluster 0 102.1c scrub starts 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: cluster 2024-04-02T04:54:09.841450+0000 osd.2 (osd.2) 191 : cluster 0 102.1c scrub ok 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: cluster 2024-04-02T04:54:10.084932+0000 mgr.y (mgr.14538) 413 : cluster 0 pgmap v592: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: audit 2024-04-02T04:54:10.175875+0000 mon.c (mon.2) 742 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: audit 2024-04-02T04:54:10.872765+0000 mon.a (mon.0) 2581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: audit 2024-04-02T04:54:10.875029+0000 mon.b (mon.1) 217 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: cluster 2024-04-02T04:54:10.882617+0000 mon.a (mon.0) 2582 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-04-02T04:54:11.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[21528]: audit 2024-04-02T04:54:10.885685+0000 mon.a (mon.0) 2583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.646081+0000 osd.4 (osd.4) 233 : cluster 0 10.1 scrub starts 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.648281+0000 osd.4 (osd.4) 234 : cluster 0 10.1 scrub ok 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.838955+0000 osd.2 (osd.2) 190 : cluster 0 102.1c scrub starts 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: cluster 2024-04-02T04:54:09.841450+0000 osd.2 (osd.2) 191 : cluster 0 102.1c scrub ok 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: cluster 2024-04-02T04:54:10.084932+0000 mgr.y (mgr.14538) 413 : cluster 0 pgmap v592: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: audit 2024-04-02T04:54:10.175875+0000 mon.c (mon.2) 742 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: audit 2024-04-02T04:54:10.872765+0000 mon.a (mon.0) 2581 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: audit 2024-04-02T04:54:10.875029+0000 mon.b (mon.1) 217 : audit 1 from='client.? 172.21.15.69:0/2058437918' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: cluster 2024-04-02T04:54:10.882617+0000 mon.a (mon.0) 2582 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-04-02T04:54:11.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:10 smithi069 bash[28904]: audit 2024-04-02T04:54:10.885685+0000 mon.a (mon.0) 2583 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]: dispatch 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/1097920935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 msgr2=0x7f6394106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- 172.21.15.69:0/1097920935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394106570 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6390009990 tx=0x7f6390009ca0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/1097920935 shutdown_connections 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- 172.21.15.69:0/1097920935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394102280 0x7f639410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- 172.21.15.69:0/1097920935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394106570 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- 172.21.15.69:0/1097920935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 0x7f6394101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/1097920935 >> 172.21.15.69:0/1097920935 conn(0x7f6394077740 msgr2=0x7f63940feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:11.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/1097920935 shutdown_connections 2024-04-02T04:54:11.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/1097920935 wait complete. 2024-04-02T04:54:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 librados: starting msgr at 2024-04-02T04:54:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 librados: starting objecter 2024-04-02T04:54:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 Processor -- start 2024-04-02T04:54:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- start start 2024-04-02T04:54:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 librados: setting wanted keys 2024-04-02T04:54:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 librados: calling monclient init 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 0x7f63941080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394102280 0x7f6394108610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394108b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6394110a20 con 0x7f63941018c0 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63941108a0 con 0x7f6394102280 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6394110ba0 con 0x7f6394106190 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394108b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639a4d4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 0x7f63941080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394108b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55732/0 (socket says 172.21.15.69:55732) 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639a4d4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 0x7f63941080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42120/0 (socket says 172.21.15.69:42120) 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 learned_addr learned my addr 172.21.15.69:0/2747262768 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394102280 0x7f6394108610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394102280 msgr2=0x7f6394108610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394102280 0x7f6394108610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 msgr2=0x7f63941080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 0x7f63941080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63941a65e0 con 0x7f6394106190 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63941018c0 0x7f63941080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394108b50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f638c00b9e0 tx=0x7f638c00bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f638b7fe700 10 client.?.objecter ms_handle_connect 0x7f6394106190 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f638b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:11.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c019070 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f638c00cdb0 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c014e10 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63941a68d0 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 10 client.26492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63941a6e10 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.197+0000 7f639c738700 1 librados: init done 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f639404ff00 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f638c01c070 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f638006f5e0 0x7f6380071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(411..411 src has 74..411) ==== 11113+0+0 (secure 0 0 0) 0x7f638c08e180 con 0x7f6394106190 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 osd_map(411..411 src has 74..411) 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 3 client.26492.objecter handle_osd_map got epochs [411,411] > 0 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 3 client.26492.objecter handle_osd_map decoding full epoch 411 2024-04-02T04:54:11.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f638006f5e0 0x7f6380071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 20 client.26492.objecter dump_active .. 0 homeless 2024-04-02T04:54:11.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v411) ==== 54+0+16 (secure 0 0 0) 0x7f638c0907a0 con 0x7f6394106190 2024-04-02T04:54:11.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 10 client.26492.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 1 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63940630a0 0x7f6394104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f638006f5e0 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f638006f5e0 0x7f6380071aa0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f639000b7f0 tx=0x7f639000b260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 1 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 0 1 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 4 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 10 client.26492.objecter _send_command 1 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6394105a50 con 0x7f63940630a0 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63940630a0 0x7f6394104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f63940630a0 2024-04-02T04:54:11.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63940630a0 0x7f6394104390 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6394105a50 con 0x7f63940630a0 2024-04-02T04:54:11.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 1: 0 ) 2024-04-02T04:54:11.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 10 client.26492.objecter _finish_command 1 = system:0 2024-04-02T04:54:11.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 4 2024-04-02T04:54:11.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.201+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 0 1 2024-04-02T04:54:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 10 client.26492.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 2 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 2 2024-04-02T04:54:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f6394109600 osd=0 3 2024-04-02T04:54:11.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 0 2 2024-04-02T04:54:11.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 4 2024-04-02T04:54:11.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 10 client.26492.objecter _send_command 2 2024-04-02T04:54:11.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.221+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f63941048d0 con 0x7f63940630a0 2024-04-02T04:54:11.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63941048d0 con 0x7f63940630a0 2024-04-02T04:54:11.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 10 client.26492.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 4 2024-04-02T04:54:11.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 0 2 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 10 client.26492.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 3 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6394062c90 0x7f63941ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 3 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 1 3 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 4 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 10 client.26492.objecter _send_command 3 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f63941057c0 con 0x7f6394062c90 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6394062c90 0x7f63941ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f6394062c90 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6394062c90 0x7f63941ab0c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941057c0 con 0x7f6394062c90 2024-04-02T04:54:11.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 3: 0 ) 2024-04-02T04:54:11.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 10 client.26492.objecter _finish_command 3 = system:0 2024-04-02T04:54:11.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 4 2024-04-02T04:54:11.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.225+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 1 3 2024-04-02T04:54:11.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 10 client.26492.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 4 2024-04-02T04:54:11.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 4 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f6394062b20 osd=1 3 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 1 4 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 4 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 10 client.26492.objecter _send_command 4 2024-04-02T04:54:11.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.241+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6394104d30 con 0x7f6394062c90 2024-04-02T04:54:11.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6394104d30 con 0x7f6394062c90 2024-04-02T04:54:11.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 10 client.26492.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 4 2024-04-02T04:54:11.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 1 4 2024-04-02T04:54:11.244 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 10 client.26492.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 5 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f639402d690 0x7f63941ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 5 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 2 5 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 4 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 10 client.26492.objecter _send_command 5 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f63941b15f0 con 0x7f639402d690 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f639402d690 0x7f63941ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f639402d690 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f639402d690 0x7f63941ae790 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941b15f0 con 0x7f639402d690 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 5: 0 ) 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 10 client.26492.objecter _finish_command 5 = system:0 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 4 2024-04-02T04:54:11.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.245+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 2 5 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 10 client.26492.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 6 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 6 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f6394105130 osd=2 3 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 2 6 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 4 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 10 client.26492.objecter _send_command 6 2024-04-02T04:54:11.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f639402cc00 con 0x7f639402d690 2024-04-02T04:54:11.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f639402cc00 con 0x7f639402d690 2024-04-02T04:54:11.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 10 client.26492.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 4 2024-04-02T04:54:11.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 2 6 2024-04-02T04:54:11.265 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 10 client.26492.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 7 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63941afe50 0x7f63941b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 7 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 3 7 2024-04-02T04:54:11.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 4 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 10 client.26492.objecter _send_command 7 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f63941ac4a0 con 0x7f63941afe50 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63941afe50 0x7f63941b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f63941afe50 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63941afe50 0x7f63941b0270 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941ac4a0 con 0x7f63941afe50 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 7: 0 ) 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 10 client.26492.objecter _finish_command 7 = system:0 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 4 2024-04-02T04:54:11.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.265+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 3 7 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 10 client.26492.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 8 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 8 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941abdd0 osd=3 3 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 3 8 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 4 2024-04-02T04:54:11.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 10 client.26492.objecter _send_command 8 2024-04-02T04:54:11.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.285+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f63941ab850 con 0x7f63941afe50 2024-04-02T04:54:11.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63941ab850 con 0x7f63941afe50 2024-04-02T04:54:11.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f638b7fe700 10 client.26492.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 4 2024-04-02T04:54:11.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 3 8 2024-04-02T04:54:11.288 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 10 client.26492.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 9 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63941ab9c0 0x7f63941b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 9 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 4 9 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 4 2024-04-02T04:54:11.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 10 client.26492.objecter _send_command 9 2024-04-02T04:54:11.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f63941b86b0 con 0x7f63941ab9c0 2024-04-02T04:54:11.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63941ab9c0 0x7f63941b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f63941ab9c0 2024-04-02T04:54:11.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.289+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63941ab9c0 0x7f63941b57c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.293+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941b86b0 con 0x7f63941ab9c0 2024-04-02T04:54:11.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.293+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 9: 0 ) 2024-04-02T04:54:11.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.293+0000 7f638b7fe700 10 client.26492.objecter _finish_command 9 = system:0 2024-04-02T04:54:11.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.293+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 4 2024-04-02T04:54:11.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.293+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 4 9 2024-04-02T04:54:11.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 10 client.26492.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 10 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 10 2024-04-02T04:54:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941ab850 osd=4 3 2024-04-02T04:54:11.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 4 10 2024-04-02T04:54:11.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 4 2024-04-02T04:54:11.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 10 client.26492.objecter _send_command 10 2024-04-02T04:54:11.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.309+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f63941b5d00 con 0x7f63941ab9c0 2024-04-02T04:54:11.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63941b5d00 con 0x7f63941ab9c0 2024-04-02T04:54:11.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f638b7fe700 10 client.26492.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 4 2024-04-02T04:54:11.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 4 10 2024-04-02T04:54:11.314 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 10 client.26492.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 11 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63941b6420 0x7f63941b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 11 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 5 11 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 4 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 10 client.26492.objecter _send_command 11 2024-04-02T04:54:11.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.313+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f63941bbaa0 con 0x7f63941b6420 2024-04-02T04:54:11.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63941b6420 0x7f63941b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f63941b6420 2024-04-02T04:54:11.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f6399cd3700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63941b6420 0x7f63941b6860 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941bbaa0 con 0x7f63941b6420 2024-04-02T04:54:11.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 11: 0 ) 2024-04-02T04:54:11.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f638b7fe700 10 client.26492.objecter _finish_command 11 = system:0 2024-04-02T04:54:11.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 4 2024-04-02T04:54:11.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.317+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 5 11 2024-04-02T04:54:11.330 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 10 client.26492.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 12 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 12 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941b5d00 osd=5 3 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 5 12 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 4 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 10 client.26492.objecter _send_command 12 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.341+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f63941bbaa0 con 0x7f63941b6420 2024-04-02T04:54:11.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63941bbaa0 con 0x7f63941b6420 2024-04-02T04:54:11.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f638b7fe700 10 client.26492.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 4 2024-04-02T04:54:11.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 5 12 2024-04-02T04:54:11.344 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 10 client.26492.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 13 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63941bae40 0x7f63941bb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 13 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 6 13 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 4 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 10 client.26492.objecter _send_command 13 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f63941bee40 con 0x7f63941bae40 2024-04-02T04:54:11.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63941bae40 0x7f63941bb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f63941bae40 2024-04-02T04:54:11.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.345+0000 7f639acd5700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63941bae40 0x7f63941bb2a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.349+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941bee40 con 0x7f63941bae40 2024-04-02T04:54:11.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.349+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 13: 0 ) 2024-04-02T04:54:11.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.349+0000 7f638b7fe700 10 client.26492.objecter _finish_command 13 = system:0 2024-04-02T04:54:11.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.349+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 4 2024-04-02T04:54:11.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.349+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 6 13 2024-04-02T04:54:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 10 client.26492.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 14 2024-04-02T04:54:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 14 2024-04-02T04:54:11.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941bbaa0 osd=6 3 2024-04-02T04:54:11.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 6 14 2024-04-02T04:54:11.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 4 2024-04-02T04:54:11.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 10 client.26492.objecter _send_command 14 2024-04-02T04:54:11.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.365+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f63941b6df0 con 0x7f63941bae40 2024-04-02T04:54:11.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63941b6df0 con 0x7f63941bae40 2024-04-02T04:54:11.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f638b7fe700 10 client.26492.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 4 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 6 14 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 10 client.26492.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 15 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63941b7510 0x7f63941b7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 15 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 7 15 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 4 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 10 client.26492.objecter _send_command 15 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f63941b81d0 con 0x7f63941b7510 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.369+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63941b7510 0x7f63941b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f638b7fe700 10 client.26492.objecter ms_handle_connect 0x7f63941b7510 2024-04-02T04:54:11.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f639a4d4700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63941b7510 0x7f63941b7990 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63941b81d0 con 0x7f63941b7510 2024-04-02T04:54:11.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 15: 0 ) 2024-04-02T04:54:11.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f638b7fe700 10 client.26492.objecter _finish_command 15 = system:0 2024-04-02T04:54:11.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 4 2024-04-02T04:54:11.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.373+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 7 15 2024-04-02T04:54:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.389+0000 7f639c738700 10 client.26492.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.389+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign -1 16 2024-04-02T04:54:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.389+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.389+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 20 client.26492.objecter _get_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 15 client.26492.objecter _session_command_op_remove -1 16 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 20 client.26492.objecter get_session s=0x7f63941b6df0 osd=7 3 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 15 client.26492.objecter _session_command_op_assign 7 16 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 4 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 10 client.26492.objecter _send_command 16 2024-04-02T04:54:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.393+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f63941b81d0 con 0x7f63941b7510 2024-04-02T04:54:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f638b7fe700 1 -- 172.21.15.69:0/2747262768 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63941b81d0 con 0x7f63941b7510 2024-04-02T04:54:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f638b7fe700 10 client.26492.objecter ms_dispatch 0x7f6394001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f638b7fe700 10 client.26492.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f638b7fe700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 4 2024-04-02T04:54:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f638b7fe700 15 client.26492.objecter _session_command_op_remove 7 16 2024-04-02T04:54:11.395 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:54:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 librados: watch_flush enter 2024-04-02T04:54:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 librados: watch_flush exit 2024-04-02T04:54:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.0 2024-04-02T04:54:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63940630a0 msgr2=0x7f6394104390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63940630a0 0x7f6394104390 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394109600 osd=0 1 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.1 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6394062c90 msgr2=0x7f63941ab0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6394062c90 0x7f63941ab0c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394062b20 osd=1 1 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.2 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f639402d690 msgr2=0x7f63941ae790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f639402d690 0x7f63941ae790 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f6394105130 osd=2 1 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.3 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63941afe50 msgr2=0x7f63941b0270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63941afe50 0x7f63941b0270 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941abdd0 osd=3 1 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.4 2024-04-02T04:54:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63941ab9c0 msgr2=0x7f63941b57c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63941ab9c0 0x7f63941b57c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941ab850 osd=4 1 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.5 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63941b6420 msgr2=0x7f63941b6860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63941b6420 0x7f63941b6860 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b5d00 osd=5 1 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.6 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63941bae40 msgr2=0x7f63941bb2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63941bae40 0x7f63941bb2a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941bbaa0 osd=6 1 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter close_session for osd.7 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63941b7510 msgr2=0x7f63941b7990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63941b7510 0x7f63941b7990 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter put_session s=0x7f63941b6df0 osd=7 1 2024-04-02T04:54:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 20 client.26492.objecter shutdown clearing up homeless session... 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 10 client.26492.objecter successfully canceled tick 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f638006f5e0 msgr2=0x7f6380071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f638006f5e0 0x7f6380071aa0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f639000b7f0 tx=0x7f639000b260 comp rx=0 tx=0).stop 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 msgr2=0x7f6394108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 --2- 172.21.15.69:0/2747262768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6394106190 0x7f6394108b50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f638c00b9e0 tx=0x7f638c00bcf0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 reap_dead start 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 reap_dead start 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 reap_dead start 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 reap_dead start 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 reap_dead start 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639acd5700 1 -- 172.21.15.69:0/2747262768 reap_dead start 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 shutdown_connections 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 >> 172.21.15.69:0/2747262768 conn(0x7f6394077740 msgr2=0x7f6394103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 shutdown_connections 2024-04-02T04:54:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 -- 172.21.15.69:0/2747262768 wait complete. 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.397+0000 7f639c738700 1 librados: shutdown 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:54:11.400 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:54:11.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:11.442 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:11.453 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/1714299724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4100790 msgr2=0x7ff8e4100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/1714299724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4100790 0x7ff8e4100c10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40099c0 tx=0x7ff8d4009cd0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/1714299724 shutdown_connections 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/1714299724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4101150 0x7ff8e410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/1714299724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e4107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/1714299724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4100790 0x7ff8e4100c10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/1714299724 >> 172.21.15.69:0/1714299724 conn(0x7ff8e4077740 msgr2=0x7ff8e40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:11.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/1714299724 shutdown_connections 2024-04-02T04:54:11.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/1714299724 wait complete. 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 librados: starting msgr at 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 librados: starting objecter 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 Processor -- start 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- start start 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 librados: setting wanted keys 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 librados: calling monclient init 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4100790 0x7ff8e4101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4101150 0x7ff8e4101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e41023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4111b40 con 0x7ff8e4101150 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41119c0 con 0x7ff8e4100790 2024-04-02T04:54:11.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e4111cc0 con 0x7ff8e4107260 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e41023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8e939a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4101150 0x7ff8e4101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e41023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55746/0 (socket says 172.21.15.69:55746) 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 -- 172.21.15.69:0/3154669265 learned_addr learned my addr 172.21.15.69:0/3154669265 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8e9b9b700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4100790 0x7ff8e4101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 -- 172.21.15.69:0/3154669265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4100790 msgr2=0x7ff8e4101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4100790 0x7ff8e4101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 -- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4101150 msgr2=0x7ff8e4101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4101150 0x7ff8e4101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 -- 172.21.15.69:0/3154669265 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e41a65c0 con 0x7ff8e4107260 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8e939a700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4101150 0x7ff8e4101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ea39c700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e41023f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007ee0 tx=0x7ff8dc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8daffd700 10 client.?.objecter ms_handle_connect 0x7ff8e4107260 2024-04-02T04:54:11.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8daffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00dbd0 con 0x7ff8e4107260 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff8dc00dd70 con 0x7ff8e4107260 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e41a68b0 con 0x7ff8e4107260 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc005840 con 0x7ff8e4107260 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 10 client.26498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a6dc0 con 0x7ff8e4107260 2024-04-02T04:54:11.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.461+0000 7ff8ebdff700 1 librados: init done 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e4103ab0 con 0x7ff8e4107260 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff8dc01a020 con 0x7ff8e4107260 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff8d006f5e0 0x7ff8d0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(411..411 src has 74..411) ==== 11113+0+0 (secure 0 0 0) 0x7ff8dc08e600 con 0x7ff8e4107260 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 10 client.26498.objecter ms_dispatch 0x7ff8e4001550 osd_map(411..411 src has 74..411) 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 3 client.26498.objecter handle_osd_map got epochs [411,411] > 0 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 3 client.26498.objecter handle_osd_map decoding full epoch 411 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8e9b9b700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff8d006f5e0 0x7ff8d0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 20 client.26498.objecter dump_active .. 0 homeless 2024-04-02T04:54:11.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8daffd700 10 client.26498.objecter ms_handle_connect 0x7ff8d006f5e0 2024-04-02T04:54:11.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.465+0000 7ff8e9b9b700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff8d006f5e0 0x7ff8d0071aa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009bd0 tx=0x7ff8e0009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.469+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff8dc010070 con 0x7ff8e4107260 2024-04-02T04:54:11.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:11.578 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:11.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:11.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.693+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ff8e4102ea0 con 0x7ff8e4107260 2024-04-02T04:54:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:11.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:11.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:11.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8daffd700 1 -- 172.21.15.69:0/3154669265 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v412) ==== 101+0+0 (secure 0 0 0) 0x7ff8dc05ae20 con 0x7ff8e4107260 2024-04-02T04:54:11.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 10 librados: watch_flush enter 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 10 librados: watch_flush exit 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 20 client.26498.objecter shutdown clearing up homeless session... 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 10 client.26498.objecter successfully canceled tick 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff8d006f5e0 msgr2=0x7ff8d0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff8d006f5e0 0x7ff8d0071aa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009bd0 tx=0x7ff8e0009260 comp rx=0 tx=0).stop 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 msgr2=0x7ff8e41023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e41023f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007ee0 tx=0x7ff8dc00d3b0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 shutdown_connections 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff8d006f5e0 0x7ff8d0071aa0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e4107260 0x7ff8e41023f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4100790 0x7ff8e4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 --2- 172.21.15.69:0/3154669265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e4101150 0x7ff8e4101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 >> 172.21.15.69:0/3154669265 conn(0x7ff8e4077740 msgr2=0x7ff8e4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:11.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 shutdown_connections 2024-04-02T04:54:11.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.881+0000 7ff8ebdff700 1 -- 172.21.15.69:0/3154669265 wait complete. 2024-04-02T04:54:11.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.885+0000 7ff8ebdff700 1 librados: shutdown 2024-04-02T04:54:11.884 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.909+0000 7f5e7bc03700 1 -- 172.21.15.69:0/3001398502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 msgr2=0x7f5e741096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.909+0000 7f5e7bc03700 1 --2- 172.21.15.69:0/3001398502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741096c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009990 tx=0x7f5e70009ca0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- 172.21.15.69:0/3001398502 shutdown_connections 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 --2- 172.21.15.69:0/3001398502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741096c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 --2- 172.21.15.69:0/3001398502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 0x7f5e7410e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 --2- 172.21.15.69:0/3001398502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7410ec60 0x7f5e7410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- 172.21.15.69:0/3001398502 >> 172.21.15.69:0/3001398502 conn(0x7f5e7406d3c0 msgr2=0x7f5e7406d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:11.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- 172.21.15.69:0/3001398502 shutdown_connections 2024-04-02T04:54:11.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- 172.21.15.69:0/3001398502 wait complete. 2024-04-02T04:54:11.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 librados: starting msgr at 2024-04-02T04:54:11.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 librados: starting objecter 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 Processor -- start 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- start start 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 librados: setting wanted keys 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 librados: calling monclient init 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741a7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7410ec60 0x7f5e741a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 0x7f5e741ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5e74120a50 con 0x7f5e741109e0 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e741208d0 con 0x7f5e7410ec60 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7bc03700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5e74120bd0 con 0x7f5e74109240 2024-04-02T04:54:11.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741a7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55772/0 (socket says 172.21.15.69:55772) 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 -- 172.21.15.69:0/2302395000 learned_addr learned my addr 172.21.15.69:0/2302395000 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7919e700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7410ec60 0x7f5e741a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7a1a0700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 0x7f5e741ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 -- 172.21.15.69:0/2302395000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7410ec60 msgr2=0x7f5e741a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7410ec60 0x7f5e741a7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 -- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 msgr2=0x7f5e741ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 0x7f5e741ab990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 -- 172.21.15.69:0/2302395000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e741ac0d0 con 0x7f5e74109240 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7a1a0700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 0x7f5e741ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e7999f700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741a7110 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0060b0 tx=0x7f5e6c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e6affd700 10 client.?.objecter ms_handle_connect 0x7f5e74109240 2024-04-02T04:54:11.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.913+0000 7f5e6affd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c01a070 con 0x7f5e74109240 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f5e6c00dd20 con 0x7f5e74109240 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c014600 con 0x7f5e74109240 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e7bc03700 1 -- 172.21.15.69:0/2302395000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e741ac3c0 con 0x7f5e74109240 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e7bc03700 10 client.26510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e7bc03700 1 -- 172.21.15.69:0/2302395000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e741b3c40 con 0x7f5e74109240 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e7bc03700 1 librados: init done 2024-04-02T04:54:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e5c005120 con 0x7f5e74109240 2024-04-02T04:54:11.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5e6c01d070 con 0x7f5e74109240 2024-04-02T04:54:11.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e6006f620 0x7f5e60071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:11.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(412..412 src has 74..412) ==== 11006+0+0 (secure 0 0 0) 0x7f5e6c08e500 con 0x7f5e74109240 2024-04-02T04:54:11.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 10 client.26510.objecter ms_dispatch 0x7f5e74001550 osd_map(412..412 src has 74..412) 2024-04-02T04:54:11.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 3 client.26510.objecter handle_osd_map got epochs [412,412] > 0 2024-04-02T04:54:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 3 client.26510.objecter handle_osd_map decoding full epoch 412 2024-04-02T04:54:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e7919e700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e6006f620 0x7f5e60071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 20 client.26510.objecter dump_active .. 0 homeless 2024-04-02T04:54:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e6affd700 10 client.26510.objecter ms_handle_connect 0x7f5e6006f620 2024-04-02T04:54:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.917+0000 7f5e7919e700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e6006f620 0x7f5e60071ae0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5e741102a0 tx=0x7f5e64009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:11.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:11.921+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5e6c010070 con 0x7f5e74109240 2024-04-02T04:54:11.940 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:11.959 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:11.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:11.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:10.912995+0000 mon.a (mon.0) 2584 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:11.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:10.940128+0000 osd.6 (osd.6) 193 : cluster 0 2.1a scrub starts 2024-04-02T04:54:11.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:10.943759+0000 osd.6 (osd.6) 194 : cluster 0 2.1a scrub ok 2024-04-02T04:54:11.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.177508+0000 mon.c (mon.2) 743 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:11.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.204643+0000 mon.c (mon.2) 744 : audit 0 from='client.? 172.21.15.69:0/2747262768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.291884+0000 osd.1 (osd.1) 243 : cluster 0 43.13 scrub starts 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.293558+0000 osd.1 (osd.1) 244 : cluster 0 43.13 scrub ok 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.698125+0000 mon.c (mon.2) 745 : audit 1 from='client.? 172.21.15.69:0/3154669265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.698616+0000 mon.a (mon.0) 2585 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.873815+0000 mon.a (mon.0) 2586 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.875776+0000 mon.a (mon.0) 2587 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.875886+0000 mon.a (mon.0) 2588 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.880345+0000 mon.a (mon.0) 2589 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-04-02T04:54:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:11 smithi098 bash[23790]: audit 2024-04-02T04:54:11.913318+0000 mon.a (mon.0) 2590 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:12.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:10.912995+0000 mon.a (mon.0) 2584 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:12.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:10.940128+0000 osd.6 (osd.6) 193 : cluster 0 2.1a scrub starts 2024-04-02T04:54:12.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:10.943759+0000 osd.6 (osd.6) 194 : cluster 0 2.1a scrub ok 2024-04-02T04:54:12.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.177508+0000 mon.c (mon.2) 743 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:12.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.204643+0000 mon.c (mon.2) 744 : audit 0 from='client.? 172.21.15.69:0/2747262768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.291884+0000 osd.1 (osd.1) 243 : cluster 0 43.13 scrub starts 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.293558+0000 osd.1 (osd.1) 244 : cluster 0 43.13 scrub ok 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.698125+0000 mon.c (mon.2) 745 : audit 1 from='client.? 172.21.15.69:0/3154669265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.698616+0000 mon.a (mon.0) 2585 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.873815+0000 mon.a (mon.0) 2586 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.875776+0000 mon.a (mon.0) 2587 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.875886+0000 mon.a (mon.0) 2588 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.880345+0000 mon.a (mon.0) 2589 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[21528]: audit 2024-04-02T04:54:11.913318+0000 mon.a (mon.0) 2590 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:10.912995+0000 mon.a (mon.0) 2584 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:12.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:10.940128+0000 osd.6 (osd.6) 193 : cluster 0 2.1a scrub starts 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:10.943759+0000 osd.6 (osd.6) 194 : cluster 0 2.1a scrub ok 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.177508+0000 mon.c (mon.2) 743 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.204643+0000 mon.c (mon.2) 744 : audit 0 from='client.? 172.21.15.69:0/2747262768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.291884+0000 osd.1 (osd.1) 243 : cluster 0 43.13 scrub starts 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.293558+0000 osd.1 (osd.1) 244 : cluster 0 43.13 scrub ok 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.698125+0000 mon.c (mon.2) 745 : audit 1 from='client.? 172.21.15.69:0/3154669265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.698616+0000 mon.a (mon.0) 2585 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.873815+0000 mon.a (mon.0) 2586 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.875776+0000 mon.a (mon.0) 2587 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-72153-63"}]': finished 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.875886+0000 mon.a (mon.0) 2588 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.880345+0000 mon.a (mon.0) 2589 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-04-02T04:54:12.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:11 smithi069 bash[28904]: audit 2024-04-02T04:54:11.913318+0000 mon.a (mon.0) 2590 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:12.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:12.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.129+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e5c000bc0 con 0x7f5e6006f620 2024-04-02T04:54:12.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:12.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.205+0000 7f5e6affd700 1 -- 172.21.15.69:0/2302395000 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+906517 (secure 0 0 0) 0x7f5e5c000bc0 con 0x7f5e6006f620 2024-04-02T04:54:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 10 librados: watch_flush enter 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 10 librados: watch_flush exit 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 20 client.26510.objecter shutdown clearing up homeless session... 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 10 client.26510.objecter successfully canceled tick 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e6006f620 msgr2=0x7f5e60071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e6006f620 0x7f5e60071ae0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5e741102a0 tx=0x7f5e64009190 comp rx=0 tx=0).stop 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 msgr2=0x7f5e741a7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741a7110 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0060b0 tx=0x7f5e6c00d3b0 comp rx=0 tx=0).stop 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 shutdown_connections 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e6006f620 0x7f5e60071ae0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e741109e0 0x7f5e741ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e74109240 0x7f5e741a7110 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:12.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 --2- 172.21.15.69:0/2302395000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e7410ec60 0x7f5e741a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:12.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 >> 172.21.15.69:0/2302395000 conn(0x7f5e7406d3c0 msgr2=0x7f5e74072db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:12.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 shutdown_connections 2024-04-02T04:54:12.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 -- 172.21.15.69:0/2302395000 wait complete. 2024-04-02T04:54:12.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:12.209+0000 7f5e68ff9700 1 librados: shutdown 2024-04-02T04:54:12.225 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:54:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.226 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:12.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:12.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:12.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:12.476 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:12.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:12.599 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:12.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:12.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:12.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:12.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:12.978 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:12.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:13.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:13.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.908313+0000 osd.2 (osd.2) 192 : cluster 0 43.18 scrub starts 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.909996+0000 osd.2 (osd.2) 193 : cluster 0 43.18 scrub ok 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:11.912052+0000 mon.c (mon.2) 746 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:11.916239+0000 mon.c (mon.2) 747 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:11.927741+0000 mon.a (mon.0) 2591 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:11.930121+0000 mon.c (mon.2) 748 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:11.930407+0000 mon.a (mon.0) 2592 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:13.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.952158+0000 osd.6 (osd.6) 195 : cluster 0 102.8 scrub starts 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: cluster 2024-04-02T04:54:11.954366+0000 osd.6 (osd.6) 196 : cluster 0 102.8 scrub ok 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.085470+0000 mgr.y (mgr.14538) 414 : cluster 0 pgmap v595: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:12.134158+0000 mgr.y (mgr.14538) 415 : audit 0 from='client.26510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:12.177058+0000 mon.b (mon.1) 218 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:12.178708+0000 mon.c (mon.2) 749 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:12.828188+0000 mon.b (mon.1) 219 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:12.829909+0000 mon.b (mon.1) 220 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:13.221 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:12 smithi098 bash[23790]: audit 2024-04-02T04:54:12.843206+0000 mon.a (mon.0) 2593 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:13.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.908313+0000 osd.2 (osd.2) 192 : cluster 0 43.18 scrub starts 2024-04-02T04:54:13.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.909996+0000 osd.2 (osd.2) 193 : cluster 0 43.18 scrub ok 2024-04-02T04:54:13.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:11.912052+0000 mon.c (mon.2) 746 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:11.916239+0000 mon.c (mon.2) 747 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:11.927741+0000 mon.a (mon.0) 2591 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:11.930121+0000 mon.c (mon.2) 748 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:11.930407+0000 mon.a (mon.0) 2592 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.952158+0000 osd.6 (osd.6) 195 : cluster 0 102.8 scrub starts 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: cluster 2024-04-02T04:54:11.954366+0000 osd.6 (osd.6) 196 : cluster 0 102.8 scrub ok 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.085470+0000 mgr.y (mgr.14538) 414 : cluster 0 pgmap v595: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:12.134158+0000 mgr.y (mgr.14538) 415 : audit 0 from='client.26510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:12.177058+0000 mon.b (mon.1) 218 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:12.178708+0000 mon.c (mon.2) 749 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:12.828188+0000 mon.b (mon.1) 219 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:12.829909+0000 mon.b (mon.1) 220 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[21528]: audit 2024-04-02T04:54:12.843206+0000 mon.a (mon.0) 2593 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:13.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.908313+0000 osd.2 (osd.2) 192 : cluster 0 43.18 scrub starts 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.909996+0000 osd.2 (osd.2) 193 : cluster 0 43.18 scrub ok 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:11.912052+0000 mon.c (mon.2) 746 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:11.916239+0000 mon.c (mon.2) 747 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:11.927741+0000 mon.a (mon.0) 2591 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:11.930121+0000 mon.c (mon.2) 748 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:11.930407+0000 mon.a (mon.0) 2592 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.952158+0000 osd.6 (osd.6) 195 : cluster 0 102.8 scrub starts 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: cluster 2024-04-02T04:54:11.954366+0000 osd.6 (osd.6) 196 : cluster 0 102.8 scrub ok 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.085470+0000 mgr.y (mgr.14538) 414 : cluster 0 pgmap v595: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 541 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:12.134158+0000 mgr.y (mgr.14538) 415 : audit 0 from='client.26510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:12.177058+0000 mon.b (mon.1) 218 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:12.178708+0000 mon.c (mon.2) 749 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:12.828188+0000 mon.b (mon.1) 219 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:12.829909+0000 mon.b (mon.1) 220 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:12 smithi069 bash[28904]: audit 2024-04-02T04:54:12.843206+0000 mon.a (mon.0) 2593 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:13.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:13.364 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:13.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:13.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:54:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:54:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:13.596 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:13.624 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:13.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:13.753 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:13.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:13.883 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:13.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:13.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:14.012 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:14.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:14.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:14.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:14.261 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:14.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:14.395 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:14.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:14.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.208799+0000 osd.7 (osd.7) 17 : cluster 0 3.d scrub starts 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.211217+0000 osd.7 (osd.7) 18 : cluster 0 3.d scrub ok 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.575521+0000 osd.5 (osd.5) 229 : cluster 0 4.8 scrub starts 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.577613+0000 osd.5 (osd.5) 230 : cluster 0 4.8 scrub ok 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.619490+0000 osd.4 (osd.4) 235 : cluster 0 13.d scrub starts 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.621180+0000 osd.4 (osd.4) 236 : cluster 0 13.d scrub ok 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.932480+0000 osd.6 (osd.6) 197 : cluster 0 102.0 deep-scrub starts 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.940352+0000 osd.6 (osd.6) 198 : cluster 0 102.0 deep-scrub ok 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: audit 2024-04-02T04:54:12.950514+0000 mon.a (mon.0) 2594 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: cluster 2024-04-02T04:54:12.958156+0000 mon.a (mon.0) 2595 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: audit 2024-04-02T04:54:12.962588+0000 mon.c (mon.2) 750 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: audit 2024-04-02T04:54:12.964711+0000 mon.a (mon.0) 2596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:14.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: audit 2024-04-02T04:54:13.180206+0000 mon.c (mon.2) 751 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[28904]: audit 2024-04-02T04:54:13.241258+0000 mgr.y (mgr.14538) 416 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.208799+0000 osd.7 (osd.7) 17 : cluster 0 3.d scrub starts 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.211217+0000 osd.7 (osd.7) 18 : cluster 0 3.d scrub ok 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.575521+0000 osd.5 (osd.5) 229 : cluster 0 4.8 scrub starts 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.577613+0000 osd.5 (osd.5) 230 : cluster 0 4.8 scrub ok 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.619490+0000 osd.4 (osd.4) 235 : cluster 0 13.d scrub starts 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.621180+0000 osd.4 (osd.4) 236 : cluster 0 13.d scrub ok 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.932480+0000 osd.6 (osd.6) 197 : cluster 0 102.0 deep-scrub starts 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.940352+0000 osd.6 (osd.6) 198 : cluster 0 102.0 deep-scrub ok 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: audit 2024-04-02T04:54:12.950514+0000 mon.a (mon.0) 2594 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: cluster 2024-04-02T04:54:12.958156+0000 mon.a (mon.0) 2595 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-04-02T04:54:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: audit 2024-04-02T04:54:12.962588+0000 mon.c (mon.2) 750 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: audit 2024-04-02T04:54:12.964711+0000 mon.a (mon.0) 2596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: audit 2024-04-02T04:54:13.180206+0000 mon.c (mon.2) 751 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:14 smithi069 bash[21528]: audit 2024-04-02T04:54:13.241258+0000 mgr.y (mgr.14538) 416 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:14.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.208799+0000 osd.7 (osd.7) 17 : cluster 0 3.d scrub starts 2024-04-02T04:54:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.211217+0000 osd.7 (osd.7) 18 : cluster 0 3.d scrub ok 2024-04-02T04:54:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.575521+0000 osd.5 (osd.5) 229 : cluster 0 4.8 scrub starts 2024-04-02T04:54:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.577613+0000 osd.5 (osd.5) 230 : cluster 0 4.8 scrub ok 2024-04-02T04:54:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.619490+0000 osd.4 (osd.4) 235 : cluster 0 13.d scrub starts 2024-04-02T04:54:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.621180+0000 osd.4 (osd.4) 236 : cluster 0 13.d scrub ok 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.932480+0000 osd.6 (osd.6) 197 : cluster 0 102.0 deep-scrub starts 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.940352+0000 osd.6 (osd.6) 198 : cluster 0 102.0 deep-scrub ok 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: audit 2024-04-02T04:54:12.950514+0000 mon.a (mon.0) 2594 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-72153-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: cluster 2024-04-02T04:54:12.958156+0000 mon.a (mon.0) 2595 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: audit 2024-04-02T04:54:12.962588+0000 mon.c (mon.2) 750 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: audit 2024-04-02T04:54:12.964711+0000 mon.a (mon.0) 2596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: audit 2024-04-02T04:54:13.180206+0000 mon.c (mon.2) 751 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:14 smithi098 bash[23790]: audit 2024-04-02T04:54:13.241258+0000 mgr.y (mgr.14538) 416 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:14.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:14.783 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:14.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:14.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:14.998 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-04-02T04:54:12.085183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8794604,"num_objects":443,"num_object_clones":52,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":1,"num_read":2332,"num_read_kb":2052,"num_write":1772,"num_write_kb":15189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1773016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3015,"ondisk_log_size":3015,"up":1068,"acting":1068,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1031,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":554480,"kb_used_data":83052,"kb_used_omap":68,"kb_used_meta":471163,"kb_avail":749406736,"statfs":{"total":767960285184,"available":767392497664,"internally_reserved":0,"allocated":85045248,"data_stored":62701973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69644,"internal_metadata":482471924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-57,"num_objects":-6,"num_object_clones":-6,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.003678"},"pg_stats":[{"pgid":"283.1f","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232353+0000","last_change":"2024-04-02T04:54:07.230677+0000","last_active":"2024-04-02T04:54:07.232353+0000","last_peered":"2024-04-02T04:54:07.232353+0000","last_clean":"2024-04-02T04:54:07.232353+0000","last_became_active":"2024-04-02T04:54:07.228713+0000","last_became_peered":"2024-04-02T04:54:07.228713+0000","last_unstale":"2024-04-02T04:54:07.232353+0000","last_undegraded":"2024-04-02T04:54:07.232353+0000","last_fullsized":"2024-04-02T04:54:07.232353+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:33.563435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"283.1e","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.947483+0000","last_change":"2024-04-02T04:54:06.945402+0000","last_active":"2024-04-02T04:54:06.947483+0000","last_peered":"2024-04-02T04:54:06.947483+0000","last_clean":"2024-04-02T04:54:06.947483+0000","last_became_active":"2024-04-02T04:54:06.945000+0000","last_became_peered":"2024-04-02T04:54:06.945000+0000","last_unstale":"2024-04-02T04:54:06.947483+0000","last_undegraded":"2024-04-02T04:54:06.947483+0000","last_fullsized":"2024-04-02T04:54:06.947483+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.919375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.1d","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.234119+0000","last_change":"2024-04-02T04:54:07.231812+0000","last_active":"2024-04-02T04:54:07.234119+0000","last_peered":"2024-04-02T04:54:07.234119+0000","last_clean":"2024-04-02T04:54:07.234119+0000","last_became_active":"2024-04-02T04:54:07.231498+0000","last_became_peered":"2024-04-02T04:54:07.231498+0000","last_unstale":"2024-04-02T04:54:07.234119+0000","last_undegraded":"2024-04-02T04:54:07.234119+0000","last_fullsized":"2024-04-02T04:54:07.234119+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:19.306981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"283.1c","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905689+0000","last_change":"2024-04-02T04:54:06.941220+0000","last_active":"2024-04-02T04:54:07.905689+0000","last_peered":"2024-04-02T04:54:07.905689+0000","last_clean":"2024-04-02T04:54:07.905689+0000","last_became_active":"2024-04-02T04:54:06.940783+0000","last_became_peered":"2024-04-02T04:54:06.940783+0000","last_unstale":"2024-04-02T04:54:07.905689+0000","last_undegraded":"2024-04-02T04:54:07.905689+0000","last_fullsized":"2024-04-02T04:54:07.905689+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:05.893158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"283.1b","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.235650+0000","last_change":"2024-04-02T04:54:07.234811+0000","last_active":"2024-04-02T04:54:07.235650+0000","last_peered":"2024-04-02T04:54:07.235650+0000","last_clean":"2024-04-02T04:54:07.235650+0000","last_became_active":"2024-04-02T04:54:07.233909+0000","last_became_peered":"2024-04-02T04:54:07.233909+0000","last_unstale":"2024-04-02T04:54:07.235650+0000","last_undegraded":"2024-04-02T04:54:07.235650+0000","last_fullsized":"2024-04-02T04:54:07.235650+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.482584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"283.1a","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905009+0000","last_change":"2024-04-02T04:54:06.928318+0000","last_active":"2024-04-02T04:54:07.905009+0000","last_peered":"2024-04-02T04:54:07.905009+0000","last_clean":"2024-04-02T04:54:07.905009+0000","last_became_active":"2024-04-02T04:54:06.927973+0000","last_became_peered":"2024-04-02T04:54:06.927973+0000","last_unstale":"2024-04-02T04:54:07.905009+0000","last_undegraded":"2024-04-02T04:54:07.905009+0000","last_fullsized":"2024-04-02T04:54:07.905009+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:21.207093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"283.19","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.234111+0000","last_change":"2024-04-02T04:54:07.229968+0000","last_active":"2024-04-02T04:54:07.234111+0000","last_peered":"2024-04-02T04:54:07.234111+0000","last_clean":"2024-04-02T04:54:07.234111+0000","last_became_active":"2024-04-02T04:54:07.228577+0000","last_became_peered":"2024-04-02T04:54:07.228577+0000","last_unstale":"2024-04-02T04:54:07.234111+0000","last_undegraded":"2024-04-02T04:54:07.234111+0000","last_fullsized":"2024-04-02T04:54:07.234111+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:25.163836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"283.18","version":"0'0","reported_seq":16,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.917295+0000","last_change":"2024-04-02T04:54:07.230565+0000","last_active":"2024-04-02T04:54:08.917295+0000","last_peered":"2024-04-02T04:54:08.917295+0000","last_clean":"2024-04-02T04:54:08.917295+0000","last_became_active":"2024-04-02T04:54:07.230212+0000","last_became_peered":"2024-04-02T04:54:07.230212+0000","last_unstale":"2024-04-02T04:54:08.917295+0000","last_undegraded":"2024-04-02T04:54:08.917295+0000","last_fullsized":"2024-04-02T04:54:08.917295+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:08.089868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"283.17","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912517+0000","last_change":"2024-04-02T04:54:06.933177+0000","last_active":"2024-04-02T04:54:07.912517+0000","last_peered":"2024-04-02T04:54:07.912517+0000","last_clean":"2024-04-02T04:54:07.912517+0000","last_became_active":"2024-04-02T04:54:06.932925+0000","last_became_peered":"2024-04-02T04:54:06.932925+0000","last_unstale":"2024-04-02T04:54:07.912517+0000","last_undegraded":"2024-04-02T04:54:07.912517+0000","last_fullsized":"2024-04-02T04:54:07.912517+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:32.942987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"283.16","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232620+0000","last_change":"2024-04-02T04:54:07.229033+0000","last_active":"2024-04-02T04:54:07.232620+0000","last_peered":"2024-04-02T04:54:07.232620+0000","last_clean":"2024-04-02T04:54:07.232620+0000","last_became_active":"2024-04-02T04:54:07.228670+0000","last_became_peered":"2024-04-02T04:54:07.228670+0000","last_unstale":"2024-04-02T04:54:07.232620+0000","last_undegraded":"2024-04-02T04:54:07.232620+0000","last_fullsized":"2024-04-02T04:54:07.232620+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:35.610454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"283.15","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.941338+0000","last_change":"2024-04-02T04:54:06.939945+0000","last_active":"2024-04-02T04:54:06.941338+0000","last_peered":"2024-04-02T04:54:06.941338+0000","last_clean":"2024-04-02T04:54:06.941338+0000","last_became_active":"2024-04-02T04:54:06.933633+0000","last_became_peered":"2024-04-02T04:54:06.933633+0000","last_unstale":"2024-04-02T04:54:06.941338+0000","last_undegraded":"2024-04-02T04:54:06.941338+0000","last_fullsized":"2024-04-02T04:54:06.941338+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.723165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"283.14","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.235675+0000","last_change":"2024-04-02T04:54:07.235195+0000","last_active":"2024-04-02T04:54:07.235675+0000","last_peered":"2024-04-02T04:54:07.235675+0000","last_clean":"2024-04-02T04:54:07.235675+0000","last_became_active":"2024-04-02T04:54:07.234815+0000","last_became_peered":"2024-04-02T04:54:07.234815+0000","last_unstale":"2024-04-02T04:54:07.235675+0000","last_undegraded":"2024-04-02T04:54:07.235675+0000","last_fullsized":"2024-04-02T04:54:07.235675+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:16.255988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"283.13","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908334+0000","last_change":"2024-04-02T04:54:06.942294+0000","last_active":"2024-04-02T04:54:07.908334+0000","last_peered":"2024-04-02T04:54:07.908334+0000","last_clean":"2024-04-02T04:54:07.908334+0000","last_became_active":"2024-04-02T04:54:06.941916+0000","last_became_peered":"2024-04-02T04:54:06.941916+0000","last_unstale":"2024-04-02T04:54:07.908334+0000","last_undegraded":"2024-04-02T04:54:07.908334+0000","last_fullsized":"2024-04-02T04:54:07.908334+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:32.818163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"283.12","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905207+0000","last_change":"2024-04-02T04:54:06.946806+0000","last_active":"2024-04-02T04:54:07.905207+0000","last_peered":"2024-04-02T04:54:07.905207+0000","last_clean":"2024-04-02T04:54:07.905207+0000","last_became_active":"2024-04-02T04:54:06.946521+0000","last_became_peered":"2024-04-02T04:54:06.946521+0000","last_unstale":"2024-04-02T04:54:07.905207+0000","last_undegraded":"2024-04-02T04:54:07.905207+0000","last_fullsized":"2024-04-02T04:54:07.905207+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:18.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"283.11","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.934261+0000","last_change":"2024-04-02T04:54:06.933693+0000","last_active":"2024-04-02T04:54:06.934261+0000","last_peered":"2024-04-02T04:54:06.934261+0000","last_clean":"2024-04-02T04:54:06.934261+0000","last_became_active":"2024-04-02T04:54:06.933170+0000","last_became_peered":"2024-04-02T04:54:06.933170+0000","last_unstale":"2024-04-02T04:54:06.934261+0000","last_undegraded":"2024-04-02T04:54:06.934261+0000","last_fullsized":"2024-04-02T04:54:06.934261+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:16.214635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.10","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.235601+0000","last_change":"2024-04-02T04:54:07.234146+0000","last_active":"2024-04-02T04:54:07.235601+0000","last_peered":"2024-04-02T04:54:07.235601+0000","last_clean":"2024-04-02T04:54:07.235601+0000","last_became_active":"2024-04-02T04:54:07.233721+0000","last_became_peered":"2024-04-02T04:54:07.233721+0000","last_unstale":"2024-04-02T04:54:07.235601+0000","last_undegraded":"2024-04-02T04:54:07.235601+0000","last_fullsized":"2024-04-02T04:54:07.235601+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:25.239351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"283.f","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.936215+0000","last_change":"2024-04-02T04:54:06.935880+0000","last_active":"2024-04-02T04:54:06.936215+0000","last_peered":"2024-04-02T04:54:06.936215+0000","last_clean":"2024-04-02T04:54:06.936215+0000","last_became_active":"2024-04-02T04:54:06.935539+0000","last_became_peered":"2024-04-02T04:54:06.935539+0000","last_unstale":"2024-04-02T04:54:06.936215+0000","last_undegraded":"2024-04-02T04:54:06.936215+0000","last_fullsized":"2024-04-02T04:54:06.936215+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.294880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.e","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.936209+0000","last_change":"2024-04-02T04:54:06.935723+0000","last_active":"2024-04-02T04:54:06.936209+0000","last_peered":"2024-04-02T04:54:06.936209+0000","last_clean":"2024-04-02T04:54:06.936209+0000","last_became_active":"2024-04-02T04:54:06.935432+0000","last_became_peered":"2024-04-02T04:54:06.935432+0000","last_unstale":"2024-04-02T04:54:06.936209+0000","last_undegraded":"2024-04-02T04:54:06.936209+0000","last_fullsized":"2024-04-02T04:54:06.936209+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:27.358073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.d","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.937139+0000","last_change":"2024-04-02T04:54:06.936365+0000","last_active":"2024-04-02T04:54:06.937139+0000","last_peered":"2024-04-02T04:54:06.937139+0000","last_clean":"2024-04-02T04:54:06.937139+0000","last_became_active":"2024-04-02T04:54:06.935087+0000","last_became_peered":"2024-04-02T04:54:06.935087+0000","last_unstale":"2024-04-02T04:54:06.937139+0000","last_undegraded":"2024-04-02T04:54:06.937139+0000","last_fullsized":"2024-04-02T04:54:06.937139+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:30.259454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"283.c","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905349+0000","last_change":"2024-04-02T04:54:07.234560+0000","last_active":"2024-04-02T04:54:07.905349+0000","last_peered":"2024-04-02T04:54:07.905349+0000","last_clean":"2024-04-02T04:54:07.905349+0000","last_became_active":"2024-04-02T04:54:07.233630+0000","last_became_peered":"2024-04-02T04:54:07.233630+0000","last_unstale":"2024-04-02T04:54:07.905349+0000","last_undegraded":"2024-04-02T04:54:07.905349+0000","last_fullsized":"2024-04-02T04:54:07.905349+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.526643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"283.b","version":"0'0","reported_seq":16,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920126+0000","last_change":"2024-04-02T04:54:07.229894+0000","last_active":"2024-04-02T04:54:08.920126+0000","last_peered":"2024-04-02T04:54:08.920126+0000","last_clean":"2024-04-02T04:54:08.920126+0000","last_became_active":"2024-04-02T04:54:07.229054+0000","last_became_peered":"2024-04-02T04:54:07.229054+0000","last_unstale":"2024-04-02T04:54:08.920126+0000","last_undegraded":"2024-04-02T04:54:08.920126+0000","last_fullsized":"2024-04-02T04:54:08.920126+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:24.211111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"283.a","version":"0'0","reported_seq":16,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918116+0000","last_change":"2024-04-02T04:54:06.931813+0000","last_active":"2024-04-02T04:54:08.918116+0000","last_peered":"2024-04-02T04:54:08.918116+0000","last_clean":"2024-04-02T04:54:08.918116+0000","last_became_active":"2024-04-02T04:54:06.930701+0000","last_became_peered":"2024-04-02T04:54:06.930701+0000","last_unstale":"2024-04-02T04:54:08.918116+0000","last_undegraded":"2024-04-02T04:54:08.918116+0000","last_fullsized":"2024-04-02T04:54:08.918116+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.322842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"283.9","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.939317+0000","last_change":"2024-04-02T04:54:06.938758+0000","last_active":"2024-04-02T04:54:06.939317+0000","last_peered":"2024-04-02T04:54:06.939317+0000","last_clean":"2024-04-02T04:54:06.939317+0000","last_became_active":"2024-04-02T04:54:06.938303+0000","last_became_peered":"2024-04-02T04:54:06.938303+0000","last_unstale":"2024-04-02T04:54:06.939317+0000","last_undegraded":"2024-04-02T04:54:06.939317+0000","last_fullsized":"2024-04-02T04:54:06.939317+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.695590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"283.8","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.937441+0000","last_change":"2024-04-02T04:54:06.936936+0000","last_active":"2024-04-02T04:54:06.937441+0000","last_peered":"2024-04-02T04:54:06.937441+0000","last_clean":"2024-04-02T04:54:06.937441+0000","last_became_active":"2024-04-02T04:54:06.936645+0000","last_became_peered":"2024-04-02T04:54:06.936645+0000","last_unstale":"2024-04-02T04:54:06.937441+0000","last_undegraded":"2024-04-02T04:54:06.937441+0000","last_fullsized":"2024-04-02T04:54:06.937441+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:22.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.7","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.940284+0000","last_change":"2024-04-02T04:54:06.939544+0000","last_active":"2024-04-02T04:54:06.940284+0000","last_peered":"2024-04-02T04:54:06.940284+0000","last_clean":"2024-04-02T04:54:06.940284+0000","last_became_active":"2024-04-02T04:54:06.939024+0000","last_became_peered":"2024-04-02T04:54:06.939024+0000","last_unstale":"2024-04-02T04:54:06.940284+0000","last_undegraded":"2024-04-02T04:54:06.940284+0000","last_fullsized":"2024-04-02T04:54:06.940284+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.695018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.6","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.237233+0000","last_change":"2024-04-02T04:54:07.236289+0000","last_active":"2024-04-02T04:54:07.237233+0000","last_peered":"2024-04-02T04:54:07.237233+0000","last_clean":"2024-04-02T04:54:07.237233+0000","last_became_active":"2024-04-02T04:54:07.234297+0000","last_became_peered":"2024-04-02T04:54:07.234297+0000","last_unstale":"2024-04-02T04:54:07.237233+0000","last_undegraded":"2024-04-02T04:54:07.237233+0000","last_fullsized":"2024-04-02T04:54:07.237233+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:30.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"283.5","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.940630+0000","last_change":"2024-04-02T04:54:06.934660+0000","last_active":"2024-04-02T04:54:06.940630+0000","last_peered":"2024-04-02T04:54:06.940630+0000","last_clean":"2024-04-02T04:54:06.940630+0000","last_became_active":"2024-04-02T04:54:06.933792+0000","last_became_peered":"2024-04-02T04:54:06.933792+0000","last_unstale":"2024-04-02T04:54:06.940630+0000","last_undegraded":"2024-04-02T04:54:06.940630+0000","last_fullsized":"2024-04-02T04:54:06.940630+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:11.498092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"283.4","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910322+0000","last_change":"2024-04-02T04:54:06.945278+0000","last_active":"2024-04-02T04:54:07.910322+0000","last_peered":"2024-04-02T04:54:07.910322+0000","last_clean":"2024-04-02T04:54:07.910322+0000","last_became_active":"2024-04-02T04:54:06.944959+0000","last_became_peered":"2024-04-02T04:54:06.944959+0000","last_unstale":"2024-04-02T04:54:07.910322+0000","last_undegraded":"2024-04-02T04:54:07.910322+0000","last_fullsized":"2024-04-02T04:54:07.910322+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:23.091370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"283.3","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910059+0000","last_change":"2024-04-02T04:54:06.945975+0000","last_active":"2024-04-02T04:54:07.910059+0000","last_peered":"2024-04-02T04:54:07.910059+0000","last_clean":"2024-04-02T04:54:07.910059+0000","last_became_active":"2024-04-02T04:54:06.945702+0000","last_became_peered":"2024-04-02T04:54:06.945702+0000","last_unstale":"2024-04-02T04:54:07.910059+0000","last_undegraded":"2024-04-02T04:54:07.910059+0000","last_fullsized":"2024-04-02T04:54:07.910059+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.074468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"283.2","version":"0'0","reported_seq":13,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905484+0000","last_change":"2024-04-02T04:54:06.923007+0000","last_active":"2024-04-02T04:54:07.905484+0000","last_peered":"2024-04-02T04:54:07.905484+0000","last_clean":"2024-04-02T04:54:07.905484+0000","last_became_active":"2024-04-02T04:54:06.922603+0000","last_became_peered":"2024-04-02T04:54:06.922603+0000","last_unstale":"2024-04-02T04:54:07.905484+0000","last_undegraded":"2024-04-02T04:54:07.905484+0000","last_fullsized":"2024-04-02T04:54:07.905484+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:18.935860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"283.1","version":"0'0","reported_seq":16,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918613+0000","last_change":"2024-04-02T04:54:07.229626+0000","last_active":"2024-04-02T04:54:08.918613+0000","last_peered":"2024-04-02T04:54:08.918613+0000","last_clean":"2024-04-02T04:54:08.918613+0000","last_became_active":"2024-04-02T04:54:07.229109+0000","last_became_peered":"2024-04-02T04:54:07.229109+0000","last_unstale":"2024-04-02T04:54:08.918613+0000","last_undegraded":"2024-04-02T04:54:08.918613+0000","last_fullsized":"2024-04-02T04:54:08.918613+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.906694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"283.0","version":"0'0","reported_seq":10,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.939963+0000","last_change":"2024-04-02T04:54:06.934590+0000","last_active":"2024-04-02T04:54:06.939963+0000","last_peered":"2024-04-02T04:54:06.939963+0000","last_clean":"2024-04-02T04:54:06.939963+0000","last_became_active":"2024-04-02T04:54:06.933568+0000","last_became_peered":"2024-04-02T04:54:06.933568+0000","last_unstale":"2024-04-02T04:54:06.939963+0000","last_undegraded":"2024-04-02T04:54:06.939963+0000","last_fullsized":"2024-04-02T04:54:06.939963+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":406,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:05.886385+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.886385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:19.866333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"102.19","version":"407'66","reported_seq":123,"reported_epoch":409,"state":"active+clean+snaptrim","last_fresh":"2024-04-02T04:54:08.919168+0000","last_change":"2024-04-02T04:54:03.007361+0000","last_active":"2024-04-02T04:54:08.919168+0000","last_peered":"2024-04-02T04:54:08.919168+0000","last_clean":"2024-04-02T04:54:08.919168+0000","last_became_active":"2024-04-02T04:53:54.919540+0000","last_became_peered":"2024-04-02T04:53:54.919540+0000","last_unstale":"2024-04-02T04:54:08.919168+0000","last_undegraded":"2024-04-02T04:54:08.919168+0000","last_fullsized":"2024-04-02T04:54:08.919168+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.121381688,"stat_sum":{"num_bytes":173,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"102.18","version":"263'35","reported_seq":686,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913808+0000","last_change":"2024-04-02T04:52:53.911292+0000","last_active":"2024-04-02T04:54:06.913808+0000","last_peered":"2024-04-02T04:54:06.913808+0000","last_clean":"2024-04-02T04:54:06.913808+0000","last_became_active":"2024-04-02T04:49:56.572185+0000","last_became_peered":"2024-04-02T04:49:56.572185+0000","last_unstale":"2024-04-02T04:54:06.913808+0000","last_undegraded":"2024-04-02T04:54:06.913808+0000","last_fullsized":"2024-04-02T04:54:06.913808+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'35","last_scrub_stamp":"2024-04-02T04:51:56.184367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.184367+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0094289309999999998,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1b","version":"263'37","reported_seq":700,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908020+0000","last_change":"2024-04-02T04:52:53.928225+0000","last_active":"2024-04-02T04:54:07.908020+0000","last_peered":"2024-04-02T04:54:07.908020+0000","last_clean":"2024-04-02T04:54:07.908020+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:54:07.908020+0000","last_undegraded":"2024-04-02T04:54:07.908020+0000","last_fullsized":"2024-04-02T04:54:07.908020+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'37","last_scrub_stamp":"2024-04-02T04:52:48.221399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.221399+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:17.357792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016351759999999999,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1a","version":"342'73","reported_seq":712,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.916957+0000","last_change":"2024-04-02T04:52:53.912906+0000","last_active":"2024-04-02T04:54:06.916957+0000","last_peered":"2024-04-02T04:54:06.916957+0000","last_clean":"2024-04-02T04:54:06.916957+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:54:06.916957+0000","last_undegraded":"2024-04-02T04:54:06.916957+0000","last_fullsized":"2024-04-02T04:54:06.916957+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'68","last_scrub_stamp":"2024-04-02T04:51:53.146929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.146929+0000","objects_scrubbed":4,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00057379499999999999,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1d","version":"248'85","reported_seq":743,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907733+0000","last_change":"2024-04-02T04:52:53.907859+0000","last_active":"2024-04-02T04:54:07.907733+0000","last_peered":"2024-04-02T04:54:07.907733+0000","last_clean":"2024-04-02T04:54:07.907733+0000","last_became_active":"2024-04-02T04:49:56.582887+0000","last_became_peered":"2024-04-02T04:49:56.582887+0000","last_unstale":"2024-04-02T04:54:07.907733+0000","last_undegraded":"2024-04-02T04:54:07.907733+0000","last_fullsized":"2024-04-02T04:54:07.907733+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'85","last_scrub_stamp":"2024-04-02T04:52:51.767218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.767218+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:20.149272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043202969999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1c","version":"393'35","reported_seq":689,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.918785+0000","last_change":"2024-04-02T04:52:53.919138+0000","last_active":"2024-04-02T04:54:06.918785+0000","last_peered":"2024-04-02T04:54:06.918785+0000","last_clean":"2024-04-02T04:54:06.918785+0000","last_became_active":"2024-04-02T04:49:56.570832+0000","last_became_peered":"2024-04-02T04:49:56.570832+0000","last_unstale":"2024-04-02T04:54:06.918785+0000","last_undegraded":"2024-04-02T04:54:06.918785+0000","last_fullsized":"2024-04-02T04:54:06.918785+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'34","last_scrub_stamp":"2024-04-02T04:51:47.142172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:47.142172+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00060106500000000002,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1f","version":"407'51","reported_seq":703,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.940988+0000","last_change":"2024-04-02T04:52:55.923999+0000","last_active":"2024-04-02T04:54:06.940988+0000","last_peered":"2024-04-02T04:54:06.940988+0000","last_clean":"2024-04-02T04:54:06.940988+0000","last_became_active":"2024-04-02T04:49:55.249289+0000","last_became_peered":"2024-04-02T04:49:55.249289+0000","last_unstale":"2024-04-02T04:54:06.940988+0000","last_undegraded":"2024-04-02T04:54:06.940988+0000","last_fullsized":"2024-04-02T04:54:06.940988+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063291931999999995,"stat_sum":{"num_bytes":61,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1e","version":"248'17","reported_seq":672,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921155+0000","last_change":"2024-04-02T04:52:53.925077+0000","last_active":"2024-04-02T04:54:06.921155+0000","last_peered":"2024-04-02T04:54:06.921155+0000","last_clean":"2024-04-02T04:54:06.921155+0000","last_became_active":"2024-04-02T04:49:56.563968+0000","last_became_peered":"2024-04-02T04:49:56.563968+0000","last_unstale":"2024-04-02T04:54:06.921155+0000","last_undegraded":"2024-04-02T04:54:06.921155+0000","last_fullsized":"2024-04-02T04:54:06.921155+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:51:55.166519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.166519+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00090914499999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.11","version":"407'71","reported_seq":748,"reported_epoch":407,"state":"active+clean+snaptrim","last_fresh":"2024-04-02T04:54:07.036787+0000","last_change":"2024-04-02T04:54:03.017817+0000","last_active":"2024-04-02T04:54:07.036787+0000","last_peered":"2024-04-02T04:54:07.036787+0000","last_clean":"2024-04-02T04:54:07.036787+0000","last_became_active":"2024-04-02T04:53:54.924990+0000","last_became_peered":"2024-04-02T04:53:54.924990+0000","last_unstale":"2024-04-02T04:54:07.036787+0000","last_undegraded":"2024-04-02T04:54:07.036787+0000","last_fullsized":"2024-04-02T04:54:07.036787+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":1.1036099319999999,"stat_sum":{"num_bytes":178,"num_objects":17,"num_object_clones":10,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"102.10","version":"401'34","reported_seq":704,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913833+0000","last_change":"2024-04-02T04:54:03.009266+0000","last_active":"2024-04-02T04:54:06.913833+0000","last_peered":"2024-04-02T04:54:06.913833+0000","last_clean":"2024-04-02T04:54:06.913833+0000","last_became_active":"2024-04-02T04:53:54.956913+0000","last_became_peered":"2024-04-02T04:53:54.956913+0000","last_unstale":"2024-04-02T04:54:06.913833+0000","last_undegraded":"2024-04-02T04:54:06.913833+0000","last_fullsized":"2024-04-02T04:54:06.913833+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0930477839999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.13","version":"395'92","reported_seq":795,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908070+0000","last_change":"2024-04-02T04:53:55.035257+0000","last_active":"2024-04-02T04:54:07.908070+0000","last_peered":"2024-04-02T04:54:07.908070+0000","last_clean":"2024-04-02T04:54:07.908070+0000","last_became_active":"2024-04-02T04:53:54.943248+0000","last_became_peered":"2024-04-02T04:53:54.943248+0000","last_unstale":"2024-04-02T04:54:07.908070+0000","last_undegraded":"2024-04-02T04:54:07.908070+0000","last_fullsized":"2024-04-02T04:54:07.908070+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.024959176,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.12","version":"393'18","reported_seq":677,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913201+0000","last_change":"2024-04-02T04:52:53.896169+0000","last_active":"2024-04-02T04:54:06.913201+0000","last_peered":"2024-04-02T04:54:06.913201+0000","last_clean":"2024-04-02T04:54:06.913201+0000","last_became_active":"2024-04-02T04:49:55.255700+0000","last_became_peered":"2024-04-02T04:49:55.255700+0000","last_unstale":"2024-04-02T04:54:06.913201+0000","last_undegraded":"2024-04-02T04:54:06.913201+0000","last_fullsized":"2024-04-02T04:54:06.913201+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-04-02T04:51:53.016585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.016585+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002296219,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.15","version":"395'69","reported_seq":761,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.913103+0000","last_change":"2024-04-02T04:53:55.044824+0000","last_active":"2024-04-02T04:54:07.913103+0000","last_peered":"2024-04-02T04:54:07.913103+0000","last_clean":"2024-04-02T04:54:07.913103+0000","last_became_active":"2024-04-02T04:53:54.955612+0000","last_became_peered":"2024-04-02T04:53:54.955612+0000","last_unstale":"2024-04-02T04:54:07.913103+0000","last_undegraded":"2024-04-02T04:54:07.913103+0000","last_fullsized":"2024-04-02T04:54:07.913103+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.016697489999999999,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.14","version":"350'154","reported_seq":798,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232331+0000","last_change":"2024-04-02T04:52:53.927028+0000","last_active":"2024-04-02T04:54:07.232331+0000","last_peered":"2024-04-02T04:54:07.232331+0000","last_clean":"2024-04-02T04:54:07.232331+0000","last_became_active":"2024-04-02T04:49:55.269267+0000","last_became_peered":"2024-04-02T04:49:55.269267+0000","last_unstale":"2024-04-02T04:54:07.232331+0000","last_undegraded":"2024-04-02T04:54:07.232331+0000","last_fullsized":"2024-04-02T04:54:07.232331+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'153","last_scrub_stamp":"2024-04-02T04:52:34.712599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.712599+0000","objects_scrubbed":9,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0025973229999999999,"stat_sum":{"num_bytes":4000107,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":46,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.17","version":"393'36","reported_seq":692,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921756+0000","last_change":"2024-04-02T04:52:53.928093+0000","last_active":"2024-04-02T04:54:06.921756+0000","last_peered":"2024-04-02T04:54:06.921756+0000","last_clean":"2024-04-02T04:54:06.921756+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:54:06.921756+0000","last_undegraded":"2024-04-02T04:54:06.921756+0000","last_fullsized":"2024-04-02T04:54:06.921756+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'34","last_scrub_stamp":"2024-04-02T04:51:48.141397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.141397+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0021229199999999999,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.16","version":"393'18","reported_seq":678,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232403+0000","last_change":"2024-04-02T04:52:53.931293+0000","last_active":"2024-04-02T04:54:07.232403+0000","last_peered":"2024-04-02T04:54:07.232403+0000","last_clean":"2024-04-02T04:54:07.232403+0000","last_became_active":"2024-04-02T04:49:55.269224+0000","last_became_peered":"2024-04-02T04:49:55.269224+0000","last_unstale":"2024-04-02T04:54:07.232403+0000","last_undegraded":"2024-04-02T04:54:07.232403+0000","last_fullsized":"2024-04-02T04:54:07.232403+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'17","last_scrub_stamp":"2024-04-02T04:52:38.664090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.664090+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031222559999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.9","version":"401'37","reported_seq":720,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.832908+0000","last_change":"2024-04-02T04:54:07.832992+0000","last_active":"2024-04-02T04:54:07.832908+0000","last_peered":"2024-04-02T04:54:07.832908+0000","last_clean":"2024-04-02T04:54:07.832908+0000","last_became_active":"2024-04-02T04:53:54.924508+0000","last_became_peered":"2024-04-02T04:53:54.924508+0000","last_unstale":"2024-04-02T04:54:07.832908+0000","last_undegraded":"2024-04-02T04:54:07.832908+0000","last_fullsized":"2024-04-02T04:54:07.832908+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'37","last_scrub_stamp":"2024-04-02T04:54:07.832705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.832705+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:26.772069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.0784295509999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.8","version":"248'17","reported_seq":668,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.917133+0000","last_change":"2024-04-02T04:52:53.906611+0000","last_active":"2024-04-02T04:54:06.917133+0000","last_peered":"2024-04-02T04:54:06.917133+0000","last_clean":"2024-04-02T04:54:06.917133+0000","last_became_active":"2024-04-02T04:49:55.259553+0000","last_became_peered":"2024-04-02T04:49:55.259553+0000","last_unstale":"2024-04-02T04:54:06.917133+0000","last_undegraded":"2024-04-02T04:54:06.917133+0000","last_fullsized":"2024-04-02T04:54:06.917133+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069807660000000002,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.b","version":"393'75","reported_seq":722,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230273+0000","last_change":"2024-04-02T04:52:53.926194+0000","last_active":"2024-04-02T04:54:07.230273+0000","last_peered":"2024-04-02T04:54:07.230273+0000","last_clean":"2024-04-02T04:54:07.230273+0000","last_became_active":"2024-04-02T04:49:55.267102+0000","last_became_peered":"2024-04-02T04:49:55.267102+0000","last_unstale":"2024-04-02T04:54:07.230273+0000","last_undegraded":"2024-04-02T04:54:07.230273+0000","last_fullsized":"2024-04-02T04:54:07.230273+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'73","last_scrub_stamp":"2024-04-02T04:52:33.669178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.669178+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003027484,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.a","version":"254'68","reported_seq":717,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229527+0000","last_change":"2024-04-02T04:52:53.919322+0000","last_active":"2024-04-02T04:54:07.229527+0000","last_peered":"2024-04-02T04:54:07.229527+0000","last_clean":"2024-04-02T04:54:07.229527+0000","last_became_active":"2024-04-02T04:49:55.252341+0000","last_became_peered":"2024-04-02T04:49:55.252341+0000","last_unstale":"2024-04-02T04:54:07.229527+0000","last_undegraded":"2024-04-02T04:54:07.229527+0000","last_fullsized":"2024-04-02T04:54:07.229527+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'68","last_scrub_stamp":"2024-04-02T04:52:43.653347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.653347+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00117381,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.d","version":"407'84","reported_seq":142,"reported_epoch":409,"state":"active+clean+snaptrim","last_fresh":"2024-04-02T04:54:08.925034+0000","last_change":"2024-04-02T04:54:01.030671+0000","last_active":"2024-04-02T04:54:08.925034+0000","last_peered":"2024-04-02T04:54:08.925034+0000","last_clean":"2024-04-02T04:54:08.925034+0000","last_became_active":"2024-04-02T04:53:54.929671+0000","last_became_peered":"2024-04-02T04:53:54.929671+0000","last_unstale":"2024-04-02T04:54:08.925034+0000","last_undegraded":"2024-04-02T04:54:08.925034+0000","last_fullsized":"2024-04-02T04:54:08.925034+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.86223859599999997,"stat_sum":{"num_bytes":342,"num_objects":33,"num_object_clones":23,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"102.c","version":"249'68","reported_seq":715,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.927468+0000","last_change":"2024-04-02T04:53:54.821231+0000","last_active":"2024-04-02T04:54:06.927468+0000","last_peered":"2024-04-02T04:54:06.927468+0000","last_clean":"2024-04-02T04:54:06.927468+0000","last_became_active":"2024-04-02T04:49:55.252273+0000","last_became_peered":"2024-04-02T04:49:55.252273+0000","last_unstale":"2024-04-02T04:54:06.927468+0000","last_undegraded":"2024-04-02T04:54:06.927468+0000","last_fullsized":"2024-04-02T04:54:06.927468+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'68","last_scrub_stamp":"2024-04-02T04:53:54.821023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.821023+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:55.464313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0093620490000000008,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.f","version":"393'63","reported_seq":771,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.927883+0000","last_change":"2024-04-02T04:53:14.448202+0000","last_active":"2024-04-02T04:54:06.927883+0000","last_peered":"2024-04-02T04:54:06.927883+0000","last_clean":"2024-04-02T04:54:06.927883+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:54:06.927883+0000","last_undegraded":"2024-04-02T04:54:06.927883+0000","last_fullsized":"2024-04-02T04:54:06.927883+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'60","last_scrub_stamp":"2024-04-02T04:53:14.447906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:53:14.447906+0000","objects_scrubbed":8,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:39.196281+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048377230000000004,"stat_sum":{"num_bytes":4194378,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":8214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.e","version":"229'56","reported_seq":719,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912404+0000","last_change":"2024-04-02T04:52:53.933725+0000","last_active":"2024-04-02T04:54:07.912404+0000","last_peered":"2024-04-02T04:54:07.912404+0000","last_clean":"2024-04-02T04:54:07.912404+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:54:07.912404+0000","last_undegraded":"2024-04-02T04:54:07.912404+0000","last_fullsized":"2024-04-02T04:54:07.912404+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'56","last_scrub_stamp":"2024-04-02T04:51:26.235660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.235660+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000480698,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.1","version":"401'44","reported_seq":106,"reported_epoch":409,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-02T04:54:08.919402+0000","last_change":"2024-04-02T04:54:03.007339+0000","last_active":"2024-04-02T04:54:08.919402+0000","last_peered":"2024-04-02T04:54:08.919402+0000","last_clean":"2024-04-02T04:54:08.919402+0000","last_became_active":"2024-04-02T04:53:54.923616+0000","last_became_peered":"2024-04-02T04:53:54.923616+0000","last_unstale":"2024-04-02T04:54:08.919402+0000","last_undegraded":"2024-04-02T04:54:08.919402+0000","last_fullsized":"2024-04-02T04:54:08.919402+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.105706815,"stat_sum":{"num_bytes":123,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"102.0","version":"292'90","reported_seq":725,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915527+0000","last_change":"2024-04-02T04:52:53.902619+0000","last_active":"2024-04-02T04:54:06.915527+0000","last_peered":"2024-04-02T04:54:06.915527+0000","last_clean":"2024-04-02T04:54:06.915527+0000","last_became_active":"2024-04-02T04:49:55.268700+0000","last_became_peered":"2024-04-02T04:49:55.268700+0000","last_unstale":"2024-04-02T04:54:06.915527+0000","last_undegraded":"2024-04-02T04:54:06.915527+0000","last_fullsized":"2024-04-02T04:54:06.915527+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073717599999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.3","version":"393'35","reported_seq":688,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913116+0000","last_change":"2024-04-02T04:52:53.892322+0000","last_active":"2024-04-02T04:54:06.913116+0000","last_peered":"2024-04-02T04:54:06.913116+0000","last_clean":"2024-04-02T04:54:06.913116+0000","last_became_active":"2024-04-02T04:49:55.257987+0000","last_became_peered":"2024-04-02T04:49:55.257987+0000","last_unstale":"2024-04-02T04:54:06.913116+0000","last_undegraded":"2024-04-02T04:54:06.913116+0000","last_fullsized":"2024-04-02T04:54:06.913116+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-04-02T04:51:50.994396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.994396+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.003033806,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.2","version":"273'39","reported_seq":693,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913388+0000","last_change":"2024-04-02T04:52:53.896472+0000","last_active":"2024-04-02T04:54:06.913388+0000","last_peered":"2024-04-02T04:54:06.913388+0000","last_clean":"2024-04-02T04:54:06.913388+0000","last_became_active":"2024-04-02T04:49:55.235417+0000","last_became_peered":"2024-04-02T04:49:55.235417+0000","last_unstale":"2024-04-02T04:54:06.913388+0000","last_undegraded":"2024-04-02T04:54:06.913388+0000","last_fullsized":"2024-04-02T04:54:06.913388+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'39","last_scrub_stamp":"2024-04-02T04:52:41.329212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.329212+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023115700000000002,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.5","version":"229'85","reported_seq":738,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910112+0000","last_change":"2024-04-02T04:52:53.930302+0000","last_active":"2024-04-02T04:54:07.910112+0000","last_peered":"2024-04-02T04:54:07.910112+0000","last_clean":"2024-04-02T04:54:07.910112+0000","last_became_active":"2024-04-02T04:49:56.553311+0000","last_became_peered":"2024-04-02T04:49:56.553311+0000","last_unstale":"2024-04-02T04:54:07.910112+0000","last_undegraded":"2024-04-02T04:54:07.910112+0000","last_fullsized":"2024-04-02T04:54:07.910112+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'85","last_scrub_stamp":"2024-04-02T04:51:27.257882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.257882+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00035259499999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.4","version":"395'17","reported_seq":684,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.911173+0000","last_change":"2024-04-02T04:53:54.976255+0000","last_active":"2024-04-02T04:54:07.911173+0000","last_peered":"2024-04-02T04:54:07.911173+0000","last_clean":"2024-04-02T04:54:07.911173+0000","last_became_active":"2024-04-02T04:53:54.939674+0000","last_became_peered":"2024-04-02T04:53:54.939674+0000","last_unstale":"2024-04-02T04:54:07.911173+0000","last_undegraded":"2024-04-02T04:54:07.911173+0000","last_fullsized":"2024-04-02T04:54:07.911173+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0099821739999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.7","version":"393'19","reported_seq":654,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926469+0000","last_change":"2024-04-02T04:52:53.897232+0000","last_active":"2024-04-02T04:54:06.926469+0000","last_peered":"2024-04-02T04:54:06.926469+0000","last_clean":"2024-04-02T04:54:06.926469+0000","last_became_active":"2024-04-02T04:51:57.862004+0000","last_became_peered":"2024-04-02T04:51:57.862004+0000","last_unstale":"2024-04-02T04:54:06.926469+0000","last_undegraded":"2024-04-02T04:54:06.926469+0000","last_fullsized":"2024-04-02T04:54:06.926469+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'18","last_scrub_stamp":"2024-04-02T04:52:40.288509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.288509+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0020366989999999999,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"102.6","version":"236'34","reported_seq":695,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908479+0000","last_change":"2024-04-02T04:52:53.905214+0000","last_active":"2024-04-02T04:54:07.908479+0000","last_peered":"2024-04-02T04:54:07.908479+0000","last_clean":"2024-04-02T04:54:07.908479+0000","last_became_active":"2024-04-02T04:49:56.581977+0000","last_became_peered":"2024-04-02T04:49:56.581977+0000","last_unstale":"2024-04-02T04:54:07.908479+0000","last_undegraded":"2024-04-02T04:54:07.908479+0000","last_fullsized":"2024-04-02T04:54:07.908479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-02T04:52:53.769683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.769683+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:21.301767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003762409,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"7"},{"start":"a","length":"3"},{"start":"e","length":"1"}]},{"pgid":"43.14","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906003+0000","last_change":"2024-04-02T04:52:44.775768+0000","last_active":"2024-04-02T04:54:07.906003+0000","last_peered":"2024-04-02T04:54:07.906003+0000","last_clean":"2024-04-02T04:54:07.906003+0000","last_became_active":"2024-04-02T04:49:29.828535+0000","last_became_peered":"2024-04-02T04:49:29.828535+0000","last_unstale":"2024-04-02T04:54:07.906003+0000","last_undegraded":"2024-04-02T04:54:07.906003+0000","last_fullsized":"2024-04-02T04:54:07.906003+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.775672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.775672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911026+0000","last_change":"2024-04-02T04:53:54.912014+0000","last_active":"2024-04-02T04:54:06.911026+0000","last_peered":"2024-04-02T04:54:06.911026+0000","last_clean":"2024-04-02T04:54:06.911026+0000","last_became_active":"2024-04-02T04:53:54.911630+0000","last_became_peered":"2024-04-02T04:53:54.911630+0000","last_unstale":"2024-04-02T04:54:06.911026+0000","last_undegraded":"2024-04-02T04:54:06.911026+0000","last_fullsized":"2024-04-02T04:54:06.911026+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.909758+0000","last_change":"2024-04-02T04:52:58.345528+0000","last_active":"2024-04-02T04:54:06.909758+0000","last_peered":"2024-04-02T04:54:06.909758+0000","last_clean":"2024-04-02T04:54:06.909758+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:54:06.909758+0000","last_undegraded":"2024-04-02T04:54:06.909758+0000","last_fullsized":"2024-04-02T04:54:06.909758+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.345234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.345234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.905829+0000","last_change":"2024-04-02T04:51:46.284187+0000","last_active":"2024-04-02T04:54:06.905829+0000","last_peered":"2024-04-02T04:54:06.905829+0000","last_clean":"2024-04-02T04:54:06.905829+0000","last_became_active":"2024-04-02T04:49:28.787404+0000","last_became_peered":"2024-04-02T04:49:28.787404+0000","last_unstale":"2024-04-02T04:54:06.905829+0000","last_undegraded":"2024-04-02T04:54:06.905829+0000","last_fullsized":"2024-04-02T04:54:06.905829+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:46.284019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:46.284019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.909586+0000","last_change":"2024-04-02T04:53:07.389165+0000","last_active":"2024-04-02T04:54:06.909586+0000","last_peered":"2024-04-02T04:54:06.909586+0000","last_clean":"2024-04-02T04:54:06.909586+0000","last_became_active":"2024-04-02T04:49:28.783594+0000","last_became_peered":"2024-04-02T04:49:28.783594+0000","last_unstale":"2024-04-02T04:54:06.909586+0000","last_undegraded":"2024-04-02T04:54:06.909586+0000","last_fullsized":"2024-04-02T04:54:06.909586+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.388952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.388952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:26.015336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923039+0000","last_change":"2024-04-02T04:51:44.929741+0000","last_active":"2024-04-02T04:54:06.923039+0000","last_peered":"2024-04-02T04:54:06.923039+0000","last_clean":"2024-04-02T04:54:06.923039+0000","last_became_active":"2024-04-02T04:49:28.726013+0000","last_became_peered":"2024-04-02T04:49:28.726013+0000","last_unstale":"2024-04-02T04:54:06.923039+0000","last_undegraded":"2024-04-02T04:54:06.923039+0000","last_fullsized":"2024-04-02T04:54:06.923039+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.929545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.929545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908322+0000","last_change":"2024-04-02T04:53:54.921543+0000","last_active":"2024-04-02T04:54:07.908322+0000","last_peered":"2024-04-02T04:54:07.908322+0000","last_clean":"2024-04-02T04:54:07.908322+0000","last_became_active":"2024-04-02T04:53:54.920768+0000","last_became_peered":"2024-04-02T04:53:54.920768+0000","last_unstale":"2024-04-02T04:54:07.908322+0000","last_undegraded":"2024-04-02T04:54:07.908322+0000","last_fullsized":"2024-04-02T04:54:07.908322+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232801+0000","last_change":"2024-04-02T04:51:39.644093+0000","last_active":"2024-04-02T04:54:07.232801+0000","last_peered":"2024-04-02T04:54:07.232801+0000","last_clean":"2024-04-02T04:54:07.232801+0000","last_became_active":"2024-04-02T04:49:28.759334+0000","last_became_peered":"2024-04-02T04:49:28.759334+0000","last_unstale":"2024-04-02T04:54:07.232801+0000","last_undegraded":"2024-04-02T04:54:07.232801+0000","last_fullsized":"2024-04-02T04:54:07.232801+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919082+0000","last_change":"2024-04-02T04:53:54.944915+0000","last_active":"2024-04-02T04:54:06.919082+0000","last_peered":"2024-04-02T04:54:06.919082+0000","last_clean":"2024-04-02T04:54:06.919082+0000","last_became_active":"2024-04-02T04:53:54.940561+0000","last_became_peered":"2024-04-02T04:53:54.940561+0000","last_unstale":"2024-04-02T04:54:06.919082+0000","last_undegraded":"2024-04-02T04:54:06.919082+0000","last_fullsized":"2024-04-02T04:54:06.919082+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.921454+0000","last_change":"2024-04-02T04:53:54.912392+0000","last_active":"2024-04-02T04:54:08.921454+0000","last_peered":"2024-04-02T04:54:08.921454+0000","last_clean":"2024-04-02T04:54:08.921454+0000","last_became_active":"2024-04-02T04:53:54.910223+0000","last_became_peered":"2024-04-02T04:53:54.910223+0000","last_unstale":"2024-04-02T04:54:08.921454+0000","last_undegraded":"2024-04-02T04:54:08.921454+0000","last_fullsized":"2024-04-02T04:54:08.921454+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.918833+0000","last_change":"2024-04-02T04:53:54.941362+0000","last_active":"2024-04-02T04:54:06.918833+0000","last_peered":"2024-04-02T04:54:06.918833+0000","last_clean":"2024-04-02T04:54:06.918833+0000","last_became_active":"2024-04-02T04:53:54.940053+0000","last_became_peered":"2024-04-02T04:53:54.940053+0000","last_unstale":"2024-04-02T04:54:06.918833+0000","last_undegraded":"2024-04-02T04:54:06.918833+0000","last_fullsized":"2024-04-02T04:54:06.918833+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.927242+0000","last_change":"2024-04-02T04:53:07.468104+0000","last_active":"2024-04-02T04:54:06.927242+0000","last_peered":"2024-04-02T04:54:06.927242+0000","last_clean":"2024-04-02T04:54:06.927242+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:54:06.927242+0000","last_undegraded":"2024-04-02T04:54:06.927242+0000","last_fullsized":"2024-04-02T04:54:06.927242+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.467848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.467848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:09.323511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904199+0000","last_change":"2024-04-02T04:52:40.787636+0000","last_active":"2024-04-02T04:54:07.904199+0000","last_peered":"2024-04-02T04:54:07.904199+0000","last_clean":"2024-04-02T04:54:07.904199+0000","last_became_active":"2024-04-02T04:49:29.818524+0000","last_became_peered":"2024-04-02T04:49:29.818524+0000","last_unstale":"2024-04-02T04:54:07.904199+0000","last_undegraded":"2024-04-02T04:54:07.904199+0000","last_fullsized":"2024-04-02T04:54:07.904199+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.787478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.787478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920309+0000","last_change":"2024-04-02T04:51:51.039566+0000","last_active":"2024-04-02T04:54:06.920309+0000","last_peered":"2024-04-02T04:54:06.920309+0000","last_clean":"2024-04-02T04:54:06.920309+0000","last_became_active":"2024-04-02T04:49:28.769072+0000","last_became_peered":"2024-04-02T04:49:28.769072+0000","last_unstale":"2024-04-02T04:54:06.920309+0000","last_undegraded":"2024-04-02T04:54:06.920309+0000","last_fullsized":"2024-04-02T04:54:06.920309+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.039450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908564+0000","last_change":"2024-04-02T04:52:35.787957+0000","last_active":"2024-04-02T04:54:07.908564+0000","last_peered":"2024-04-02T04:54:07.908564+0000","last_clean":"2024-04-02T04:54:07.908564+0000","last_became_active":"2024-04-02T04:49:29.801654+0000","last_became_peered":"2024-04-02T04:49:29.801654+0000","last_unstale":"2024-04-02T04:54:07.908564+0000","last_undegraded":"2024-04-02T04:54:07.908564+0000","last_fullsized":"2024-04-02T04:54:07.908564+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:35.787733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:35.787733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229817+0000","last_change":"2024-04-02T04:52:36.685201+0000","last_active":"2024-04-02T04:54:07.229817+0000","last_peered":"2024-04-02T04:54:07.229817+0000","last_clean":"2024-04-02T04:54:07.229817+0000","last_became_active":"2024-04-02T04:49:28.756548+0000","last_became_peered":"2024-04-02T04:49:28.756548+0000","last_unstale":"2024-04-02T04:54:07.229817+0000","last_undegraded":"2024-04-02T04:54:07.229817+0000","last_fullsized":"2024-04-02T04:54:07.229817+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.684685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.684685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230895+0000","last_change":"2024-04-02T04:51:40.625223+0000","last_active":"2024-04-02T04:54:07.230895+0000","last_peered":"2024-04-02T04:54:07.230895+0000","last_clean":"2024-04-02T04:54:07.230895+0000","last_became_active":"2024-04-02T04:49:28.741277+0000","last_became_peered":"2024-04-02T04:49:28.741277+0000","last_unstale":"2024-04-02T04:54:07.230895+0000","last_undegraded":"2024-04-02T04:54:07.230895+0000","last_fullsized":"2024-04-02T04:54:07.230895+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.625010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.625010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905866+0000","last_change":"2024-04-02T04:52:34.327003+0000","last_active":"2024-04-02T04:54:07.905866+0000","last_peered":"2024-04-02T04:54:07.905866+0000","last_clean":"2024-04-02T04:54:07.905866+0000","last_became_active":"2024-04-02T04:49:28.719086+0000","last_became_peered":"2024-04-02T04:49:28.719086+0000","last_unstale":"2024-04-02T04:54:07.905866+0000","last_undegraded":"2024-04-02T04:54:07.905866+0000","last_fullsized":"2024-04-02T04:54:07.905866+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.326885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.326885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907683+0000","last_change":"2024-04-02T04:52:36.311512+0000","last_active":"2024-04-02T04:54:07.907683+0000","last_peered":"2024-04-02T04:54:07.907683+0000","last_clean":"2024-04-02T04:54:07.907683+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:54:07.907683+0000","last_undegraded":"2024-04-02T04:54:07.907683+0000","last_fullsized":"2024-04-02T04:54:07.907683+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.311307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907081+0000","last_change":"2024-04-02T04:52:45.221582+0000","last_active":"2024-04-02T04:54:07.907081+0000","last_peered":"2024-04-02T04:54:07.907081+0000","last_clean":"2024-04-02T04:54:07.907081+0000","last_became_active":"2024-04-02T04:49:28.731836+0000","last_became_peered":"2024-04-02T04:49:28.731836+0000","last_unstale":"2024-04-02T04:54:07.907081+0000","last_undegraded":"2024-04-02T04:54:07.907081+0000","last_fullsized":"2024-04-02T04:54:07.907081+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:45.221445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:45.221445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":560,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.918437+0000","last_change":"2024-04-02T04:52:52.665425+0000","last_active":"2024-04-02T04:54:06.918437+0000","last_peered":"2024-04-02T04:54:06.918437+0000","last_clean":"2024-04-02T04:54:06.918437+0000","last_became_active":"2024-04-02T04:49:28.727703+0000","last_became_peered":"2024-04-02T04:49:28.727703+0000","last_unstale":"2024-04-02T04:54:06.918437+0000","last_undegraded":"2024-04-02T04:54:06.918437+0000","last_fullsized":"2024-04-02T04:54:06.918437+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.665317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.665317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":558,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910124+0000","last_change":"2024-04-02T04:53:54.943666+0000","last_active":"2024-04-02T04:54:07.910124+0000","last_peered":"2024-04-02T04:54:07.910124+0000","last_clean":"2024-04-02T04:54:07.910124+0000","last_became_active":"2024-04-02T04:53:54.942156+0000","last_became_peered":"2024-04-02T04:53:54.942156+0000","last_unstale":"2024-04-02T04:54:07.910124+0000","last_undegraded":"2024-04-02T04:54:07.910124+0000","last_fullsized":"2024-04-02T04:54:07.910124+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229662+0000","last_change":"2024-04-02T04:53:54.917125+0000","last_active":"2024-04-02T04:54:07.229662+0000","last_peered":"2024-04-02T04:54:07.229662+0000","last_clean":"2024-04-02T04:54:07.229662+0000","last_became_active":"2024-04-02T04:53:54.916251+0000","last_became_peered":"2024-04-02T04:53:54.916251+0000","last_unstale":"2024-04-02T04:54:07.229662+0000","last_undegraded":"2024-04-02T04:54:07.229662+0000","last_fullsized":"2024-04-02T04:54:07.229662+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.228620+0000","last_change":"2024-04-02T04:53:54.920771+0000","last_active":"2024-04-02T04:54:07.228620+0000","last_peered":"2024-04-02T04:54:07.228620+0000","last_clean":"2024-04-02T04:54:07.228620+0000","last_became_active":"2024-04-02T04:53:54.920154+0000","last_became_peered":"2024-04-02T04:53:54.920154+0000","last_unstale":"2024-04-02T04:54:07.228620+0000","last_undegraded":"2024-04-02T04:54:07.228620+0000","last_fullsized":"2024-04-02T04:54:07.228620+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":550,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928056+0000","last_change":"2024-04-02T04:51:41.090918+0000","last_active":"2024-04-02T04:54:06.928056+0000","last_peered":"2024-04-02T04:54:06.928056+0000","last_clean":"2024-04-02T04:54:06.928056+0000","last_became_active":"2024-04-02T04:49:28.762640+0000","last_became_peered":"2024-04-02T04:49:28.762640+0000","last_unstale":"2024-04-02T04:54:06.928056+0000","last_undegraded":"2024-04-02T04:54:06.928056+0000","last_fullsized":"2024-04-02T04:54:06.928056+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:41.090714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:41.090714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926692+0000","last_change":"2024-04-02T04:52:59.362968+0000","last_active":"2024-04-02T04:54:06.926692+0000","last_peered":"2024-04-02T04:54:06.926692+0000","last_clean":"2024-04-02T04:54:06.926692+0000","last_became_active":"2024-04-02T04:49:28.767868+0000","last_became_peered":"2024-04-02T04:49:28.767868+0000","last_unstale":"2024-04-02T04:54:06.926692+0000","last_undegraded":"2024-04-02T04:54:06.926692+0000","last_fullsized":"2024-04-02T04:54:06.926692+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.362748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.362748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:16.883581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926355+0000","last_change":"2024-04-02T04:53:54.955671+0000","last_active":"2024-04-02T04:54:06.926355+0000","last_peered":"2024-04-02T04:54:06.926355+0000","last_clean":"2024-04-02T04:54:06.926355+0000","last_became_active":"2024-04-02T04:53:54.955334+0000","last_became_peered":"2024-04-02T04:53:54.955334+0000","last_unstale":"2024-04-02T04:54:06.926355+0000","last_undegraded":"2024-04-02T04:54:06.926355+0000","last_fullsized":"2024-04-02T04:54:06.926355+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911704+0000","last_change":"2024-04-02T04:53:54.939129+0000","last_active":"2024-04-02T04:54:06.911704+0000","last_peered":"2024-04-02T04:54:06.911704+0000","last_clean":"2024-04-02T04:54:06.911704+0000","last_became_active":"2024-04-02T04:53:54.938792+0000","last_became_peered":"2024-04-02T04:53:54.938792+0000","last_unstale":"2024-04-02T04:54:06.911704+0000","last_undegraded":"2024-04-02T04:54:06.911704+0000","last_fullsized":"2024-04-02T04:54:06.911704+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912677+0000","last_change":"2024-04-02T04:51:30.306477+0000","last_active":"2024-04-02T04:54:07.912677+0000","last_peered":"2024-04-02T04:54:07.912677+0000","last_clean":"2024-04-02T04:54:07.912677+0000","last_became_active":"2024-04-02T04:49:28.720246+0000","last_became_peered":"2024-04-02T04:49:28.720246+0000","last_unstale":"2024-04-02T04:54:07.912677+0000","last_undegraded":"2024-04-02T04:54:07.912677+0000","last_fullsized":"2024-04-02T04:54:07.912677+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.306343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.306343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926970+0000","last_change":"2024-04-02T04:51:52.078040+0000","last_active":"2024-04-02T04:54:06.926970+0000","last_peered":"2024-04-02T04:54:06.926970+0000","last_clean":"2024-04-02T04:54:06.926970+0000","last_became_active":"2024-04-02T04:49:28.781235+0000","last_became_peered":"2024-04-02T04:49:28.781235+0000","last_unstale":"2024-04-02T04:54:06.926970+0000","last_undegraded":"2024-04-02T04:54:06.926970+0000","last_fullsized":"2024-04-02T04:54:06.926970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.077908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.077908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":546,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.910674+0000","last_change":"2024-04-02T04:49:28.787959+0000","last_active":"2024-04-02T04:54:06.910674+0000","last_peered":"2024-04-02T04:54:06.910674+0000","last_clean":"2024-04-02T04:54:06.910674+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:54:06.910674+0000","last_undegraded":"2024-04-02T04:54:06.910674+0000","last_fullsized":"2024-04-02T04:54:06.910674+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.905486+0000","last_change":"2024-04-02T04:53:54.918584+0000","last_active":"2024-04-02T04:54:06.905486+0000","last_peered":"2024-04-02T04:54:06.905486+0000","last_clean":"2024-04-02T04:54:06.905486+0000","last_became_active":"2024-04-02T04:53:54.917716+0000","last_became_peered":"2024-04-02T04:53:54.917716+0000","last_unstale":"2024-04-02T04:54:06.905486+0000","last_undegraded":"2024-04-02T04:54:06.905486+0000","last_fullsized":"2024-04-02T04:54:06.905486+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.905846+0000","last_change":"2024-04-02T04:52:51.287269+0000","last_active":"2024-04-02T04:54:06.905846+0000","last_peered":"2024-04-02T04:54:06.905846+0000","last_clean":"2024-04-02T04:54:06.905846+0000","last_became_active":"2024-04-02T04:49:28.750373+0000","last_became_peered":"2024-04-02T04:49:28.750373+0000","last_unstale":"2024-04-02T04:54:06.905846+0000","last_undegraded":"2024-04-02T04:54:06.905846+0000","last_fullsized":"2024-04-02T04:54:06.905846+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:51.287037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.287037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:18.722013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.921163+0000","last_change":"2024-04-02T04:53:54.926795+0000","last_active":"2024-04-02T04:54:08.921163+0000","last_peered":"2024-04-02T04:54:08.921163+0000","last_clean":"2024-04-02T04:54:08.921163+0000","last_became_active":"2024-04-02T04:53:54.923703+0000","last_became_peered":"2024-04-02T04:53:54.923703+0000","last_unstale":"2024-04-02T04:54:08.921163+0000","last_undegraded":"2024-04-02T04:54:08.921163+0000","last_fullsized":"2024-04-02T04:54:08.921163+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233596+0000","last_change":"2024-04-02T04:53:54.938266+0000","last_active":"2024-04-02T04:54:07.233596+0000","last_peered":"2024-04-02T04:54:07.233596+0000","last_clean":"2024-04-02T04:54:07.233596+0000","last_became_active":"2024-04-02T04:53:54.937941+0000","last_became_peered":"2024-04-02T04:53:54.937941+0000","last_unstale":"2024-04-02T04:54:07.233596+0000","last_undegraded":"2024-04-02T04:54:07.233596+0000","last_fullsized":"2024-04-02T04:54:07.233596+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920564+0000","last_change":"2024-04-02T04:53:54.944266+0000","last_active":"2024-04-02T04:54:08.920564+0000","last_peered":"2024-04-02T04:54:08.920564+0000","last_clean":"2024-04-02T04:54:08.920564+0000","last_became_active":"2024-04-02T04:53:54.943925+0000","last_became_peered":"2024-04-02T04:53:54.943925+0000","last_unstale":"2024-04-02T04:54:08.920564+0000","last_undegraded":"2024-04-02T04:54:08.920564+0000","last_fullsized":"2024-04-02T04:54:08.920564+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914283+0000","last_change":"2024-04-02T04:51:43.198139+0000","last_active":"2024-04-02T04:54:06.914283+0000","last_peered":"2024-04-02T04:54:06.914283+0000","last_clean":"2024-04-02T04:54:06.914283+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:54:06.914283+0000","last_undegraded":"2024-04-02T04:54:06.914283+0000","last_fullsized":"2024-04-02T04:54:06.914283+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:43.198044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.198044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":545,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.900877+0000","last_change":"2024-04-02T04:49:28.787621+0000","last_active":"2024-04-02T04:54:06.900877+0000","last_peered":"2024-04-02T04:54:06.900877+0000","last_clean":"2024-04-02T04:54:06.900877+0000","last_became_active":"2024-04-02T04:49:28.785195+0000","last_became_peered":"2024-04-02T04:49:28.785195+0000","last_unstale":"2024-04-02T04:54:06.900877+0000","last_undegraded":"2024-04-02T04:54:06.900877+0000","last_fullsized":"2024-04-02T04:54:06.900877+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.905450+0000","last_change":"2024-04-02T04:53:54.645844+0000","last_active":"2024-04-02T04:54:06.905450+0000","last_peered":"2024-04-02T04:54:06.905450+0000","last_clean":"2024-04-02T04:54:06.905450+0000","last_became_active":"2024-04-02T04:49:28.741450+0000","last_became_peered":"2024-04-02T04:49:28.741450+0000","last_unstale":"2024-04-02T04:54:06.905450+0000","last_undegraded":"2024-04-02T04:54:06.905450+0000","last_fullsized":"2024-04-02T04:54:06.905450+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:54.645629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.645629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:09.714263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901413+0000","last_change":"2024-04-02T04:53:02.294549+0000","last_active":"2024-04-02T04:54:06.901413+0000","last_peered":"2024-04-02T04:54:06.901413+0000","last_clean":"2024-04-02T04:54:06.901413+0000","last_became_active":"2024-04-02T04:49:28.731343+0000","last_became_peered":"2024-04-02T04:49:28.731343+0000","last_unstale":"2024-04-02T04:54:06.901413+0000","last_undegraded":"2024-04-02T04:54:06.901413+0000","last_fullsized":"2024-04-02T04:54:06.901413+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.294309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.735531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915903+0000","last_change":"2024-04-02T04:50:37.102849+0000","last_active":"2024-04-02T04:54:06.915903+0000","last_peered":"2024-04-02T04:54:06.915903+0000","last_clean":"2024-04-02T04:54:06.915903+0000","last_became_active":"2024-04-02T04:49:28.698110+0000","last_became_peered":"2024-04-02T04:49:28.698110+0000","last_unstale":"2024-04-02T04:54:06.915903+0000","last_undegraded":"2024-04-02T04:54:06.915903+0000","last_fullsized":"2024-04-02T04:54:06.915903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.102732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.102732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915270+0000","last_change":"2024-04-02T04:53:54.926479+0000","last_active":"2024-04-02T04:54:06.915270+0000","last_peered":"2024-04-02T04:54:06.915270+0000","last_clean":"2024-04-02T04:54:06.915270+0000","last_became_active":"2024-04-02T04:53:54.925994+0000","last_became_peered":"2024-04-02T04:53:54.925994+0000","last_unstale":"2024-04-02T04:54:06.915270+0000","last_undegraded":"2024-04-02T04:54:06.915270+0000","last_fullsized":"2024-04-02T04:54:06.915270+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.900412+0000","last_change":"2024-04-02T04:52:43.275598+0000","last_active":"2024-04-02T04:54:06.900412+0000","last_peered":"2024-04-02T04:54:06.900412+0000","last_clean":"2024-04-02T04:54:06.900412+0000","last_became_active":"2024-04-02T04:49:28.746056+0000","last_became_peered":"2024-04-02T04:49:28.746056+0000","last_unstale":"2024-04-02T04:54:06.900412+0000","last_undegraded":"2024-04-02T04:54:06.900412+0000","last_fullsized":"2024-04-02T04:54:06.900412+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:43.275420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.275420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.228734+0000","last_change":"2024-04-02T04:53:54.918857+0000","last_active":"2024-04-02T04:54:07.228734+0000","last_peered":"2024-04-02T04:54:07.228734+0000","last_clean":"2024-04-02T04:54:07.228734+0000","last_became_active":"2024-04-02T04:53:54.918532+0000","last_became_peered":"2024-04-02T04:53:54.918532+0000","last_unstale":"2024-04-02T04:54:07.228734+0000","last_undegraded":"2024-04-02T04:54:07.228734+0000","last_fullsized":"2024-04-02T04:54:07.228734+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230718+0000","last_change":"2024-04-02T04:53:54.917379+0000","last_active":"2024-04-02T04:54:07.230718+0000","last_peered":"2024-04-02T04:54:07.230718+0000","last_clean":"2024-04-02T04:54:07.230718+0000","last_became_active":"2024-04-02T04:53:54.916285+0000","last_became_peered":"2024-04-02T04:53:54.916285+0000","last_unstale":"2024-04-02T04:54:07.230718+0000","last_undegraded":"2024-04-02T04:54:07.230718+0000","last_fullsized":"2024-04-02T04:54:07.230718+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":556,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908393+0000","last_change":"2024-04-02T04:52:38.809910+0000","last_active":"2024-04-02T04:54:07.908393+0000","last_peered":"2024-04-02T04:54:07.908393+0000","last_clean":"2024-04-02T04:54:07.908393+0000","last_became_active":"2024-04-02T04:49:29.857200+0000","last_became_peered":"2024-04-02T04:49:29.857200+0000","last_unstale":"2024-04-02T04:54:07.908393+0000","last_undegraded":"2024-04-02T04:54:07.908393+0000","last_fullsized":"2024-04-02T04:54:07.908393+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.809807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.809807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915022+0000","last_change":"2024-04-02T04:51:35.000630+0000","last_active":"2024-04-02T04:54:06.915022+0000","last_peered":"2024-04-02T04:54:06.915022+0000","last_clean":"2024-04-02T04:54:06.915022+0000","last_became_active":"2024-04-02T04:49:28.746855+0000","last_became_peered":"2024-04-02T04:49:28.746855+0000","last_unstale":"2024-04-02T04:54:06.915022+0000","last_undegraded":"2024-04-02T04:54:06.915022+0000","last_fullsized":"2024-04-02T04:54:06.915022+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:35.000533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.000533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915640+0000","last_change":"2024-04-02T04:53:54.937487+0000","last_active":"2024-04-02T04:54:06.915640+0000","last_peered":"2024-04-02T04:54:06.915640+0000","last_clean":"2024-04-02T04:54:06.915640+0000","last_became_active":"2024-04-02T04:53:54.937162+0000","last_became_peered":"2024-04-02T04:53:54.937162+0000","last_unstale":"2024-04-02T04:54:06.915640+0000","last_undegraded":"2024-04-02T04:54:06.915640+0000","last_fullsized":"2024-04-02T04:54:06.915640+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905866+0000","last_change":"2024-04-02T04:51:36.554382+0000","last_active":"2024-04-02T04:54:07.905866+0000","last_peered":"2024-04-02T04:54:07.905866+0000","last_clean":"2024-04-02T04:54:07.905866+0000","last_became_active":"2024-04-02T04:49:29.874539+0000","last_became_peered":"2024-04-02T04:49:29.874539+0000","last_unstale":"2024-04-02T04:54:07.905866+0000","last_undegraded":"2024-04-02T04:54:07.905866+0000","last_fullsized":"2024-04-02T04:54:07.905866+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.554164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.554164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911376+0000","last_change":"2024-04-02T04:53:54.920901+0000","last_active":"2024-04-02T04:54:06.911376+0000","last_peered":"2024-04-02T04:54:06.911376+0000","last_clean":"2024-04-02T04:54:06.911376+0000","last_became_active":"2024-04-02T04:53:54.920551+0000","last_became_peered":"2024-04-02T04:53:54.920551+0000","last_unstale":"2024-04-02T04:54:06.911376+0000","last_undegraded":"2024-04-02T04:54:06.911376+0000","last_fullsized":"2024-04-02T04:54:06.911376+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908601+0000","last_change":"2024-04-02T04:51:53.758572+0000","last_active":"2024-04-02T04:54:07.908601+0000","last_peered":"2024-04-02T04:54:07.908601+0000","last_clean":"2024-04-02T04:54:07.908601+0000","last_became_active":"2024-04-02T04:49:29.830080+0000","last_became_peered":"2024-04-02T04:49:29.830080+0000","last_unstale":"2024-04-02T04:54:07.908601+0000","last_undegraded":"2024-04-02T04:54:07.908601+0000","last_fullsized":"2024-04-02T04:54:07.908601+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.758420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.758420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":558,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910925+0000","last_change":"2024-04-02T04:53:54.956174+0000","last_active":"2024-04-02T04:54:07.910925+0000","last_peered":"2024-04-02T04:54:07.910925+0000","last_clean":"2024-04-02T04:54:07.910925+0000","last_became_active":"2024-04-02T04:53:54.955517+0000","last_became_peered":"2024-04-02T04:53:54.955517+0000","last_unstale":"2024-04-02T04:54:07.910925+0000","last_undegraded":"2024-04-02T04:54:07.910925+0000","last_fullsized":"2024-04-02T04:54:07.910925+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.927719+0000","last_change":"2024-04-02T04:51:36.096984+0000","last_active":"2024-04-02T04:54:06.927719+0000","last_peered":"2024-04-02T04:54:06.927719+0000","last_clean":"2024-04-02T04:54:06.927719+0000","last_became_active":"2024-04-02T04:49:28.780779+0000","last_became_peered":"2024-04-02T04:49:28.780779+0000","last_unstale":"2024-04-02T04:54:06.927719+0000","last_undegraded":"2024-04-02T04:54:06.927719+0000","last_fullsized":"2024-04-02T04:54:06.927719+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.096832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.096832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911702+0000","last_change":"2024-04-02T04:52:46.335166+0000","last_active":"2024-04-02T04:54:06.911702+0000","last_peered":"2024-04-02T04:54:06.911702+0000","last_clean":"2024-04-02T04:54:06.911702+0000","last_became_active":"2024-04-02T04:49:28.784897+0000","last_became_peered":"2024-04-02T04:49:28.784897+0000","last_unstale":"2024-04-02T04:54:06.911702+0000","last_undegraded":"2024-04-02T04:54:06.911702+0000","last_fullsized":"2024-04-02T04:54:06.911702+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.334949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.334949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.534127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":562,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232748+0000","last_change":"2024-04-02T04:53:08.577436+0000","last_active":"2024-04-02T04:54:07.232748+0000","last_peered":"2024-04-02T04:54:07.232748+0000","last_clean":"2024-04-02T04:54:07.232748+0000","last_became_active":"2024-04-02T04:49:28.744153+0000","last_became_peered":"2024-04-02T04:49:28.744153+0000","last_unstale":"2024-04-02T04:54:07.232748+0000","last_undegraded":"2024-04-02T04:54:07.232748+0000","last_fullsized":"2024-04-02T04:54:07.232748+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:08.577227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:08.577227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:12.696737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926657+0000","last_change":"2024-04-02T04:53:10.492126+0000","last_active":"2024-04-02T04:54:06.926657+0000","last_peered":"2024-04-02T04:54:06.926657+0000","last_clean":"2024-04-02T04:54:06.926657+0000","last_became_active":"2024-04-02T04:49:28.760661+0000","last_became_peered":"2024-04-02T04:49:28.760661+0000","last_unstale":"2024-04-02T04:54:06.926657+0000","last_undegraded":"2024-04-02T04:54:06.926657+0000","last_fullsized":"2024-04-02T04:54:06.926657+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.491899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.491899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:32.562654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.929411+0000","last_change":"2024-04-02T04:52:39.367465+0000","last_active":"2024-04-02T04:54:06.929411+0000","last_peered":"2024-04-02T04:54:06.929411+0000","last_clean":"2024-04-02T04:54:06.929411+0000","last_became_active":"2024-04-02T04:49:28.769787+0000","last_became_peered":"2024-04-02T04:49:28.769787+0000","last_unstale":"2024-04-02T04:54:06.929411+0000","last_undegraded":"2024-04-02T04:54:06.929411+0000","last_fullsized":"2024-04-02T04:54:06.929411+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:39.367130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:39.367130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921919+0000","last_change":"2024-04-02T04:51:44.205952+0000","last_active":"2024-04-02T04:54:06.921919+0000","last_peered":"2024-04-02T04:54:06.921919+0000","last_clean":"2024-04-02T04:54:06.921919+0000","last_became_active":"2024-04-02T04:49:28.728416+0000","last_became_peered":"2024-04-02T04:49:28.728416+0000","last_unstale":"2024-04-02T04:54:06.921919+0000","last_undegraded":"2024-04-02T04:54:06.921919+0000","last_fullsized":"2024-04-02T04:54:06.921919+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.205739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.205739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":560,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921454+0000","last_change":"2024-04-02T04:53:13.855229+0000","last_active":"2024-04-02T04:54:06.921454+0000","last_peered":"2024-04-02T04:54:06.921454+0000","last_clean":"2024-04-02T04:54:06.921454+0000","last_became_active":"2024-04-02T04:49:28.727661+0000","last_became_peered":"2024-04-02T04:49:28.727661+0000","last_unstale":"2024-04-02T04:54:06.921454+0000","last_undegraded":"2024-04-02T04:54:06.921454+0000","last_fullsized":"2024-04-02T04:54:06.921454+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.855124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.855124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:43.490398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928051+0000","last_change":"2024-04-02T04:53:00.409859+0000","last_active":"2024-04-02T04:54:06.928051+0000","last_peered":"2024-04-02T04:54:06.928051+0000","last_clean":"2024-04-02T04:54:06.928051+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:54:06.928051+0000","last_undegraded":"2024-04-02T04:54:06.928051+0000","last_fullsized":"2024-04-02T04:54:06.928051+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.409610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.409610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.916114+0000","last_change":"2024-04-02T04:53:06.476121+0000","last_active":"2024-04-02T04:54:06.916114+0000","last_peered":"2024-04-02T04:54:06.916114+0000","last_clean":"2024-04-02T04:54:06.916114+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:54:06.916114+0000","last_undegraded":"2024-04-02T04:54:06.916114+0000","last_fullsized":"2024-04-02T04:54:06.916114+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.475984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.475984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:09.567682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918416+0000","last_change":"2024-04-02T04:53:54.918758+0000","last_active":"2024-04-02T04:54:08.918416+0000","last_peered":"2024-04-02T04:54:08.918416+0000","last_clean":"2024-04-02T04:54:08.918416+0000","last_became_active":"2024-04-02T04:53:54.918416+0000","last_became_peered":"2024-04-02T04:53:54.918416+0000","last_unstale":"2024-04-02T04:54:08.918416+0000","last_undegraded":"2024-04-02T04:54:08.918416+0000","last_fullsized":"2024-04-02T04:54:08.918416+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.918011+0000","last_change":"2024-04-02T04:51:48.958576+0000","last_active":"2024-04-02T04:54:06.918011+0000","last_peered":"2024-04-02T04:54:06.918011+0000","last_clean":"2024-04-02T04:54:06.918011+0000","last_became_active":"2024-04-02T04:49:28.725896+0000","last_became_peered":"2024-04-02T04:49:28.725896+0000","last_unstale":"2024-04-02T04:54:06.918011+0000","last_undegraded":"2024-04-02T04:54:06.918011+0000","last_fullsized":"2024-04-02T04:54:06.918011+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.958371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229575+0000","last_change":"2024-04-02T04:51:26.661470+0000","last_active":"2024-04-02T04:54:07.229575+0000","last_peered":"2024-04-02T04:54:07.229575+0000","last_clean":"2024-04-02T04:54:07.229575+0000","last_became_active":"2024-04-02T04:49:28.724387+0000","last_became_peered":"2024-04-02T04:49:28.724387+0000","last_unstale":"2024-04-02T04:54:07.229575+0000","last_undegraded":"2024-04-02T04:54:07.229575+0000","last_fullsized":"2024-04-02T04:54:07.229575+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.661376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.661376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908619+0000","last_change":"2024-04-02T04:51:27.695556+0000","last_active":"2024-04-02T04:54:07.908619+0000","last_peered":"2024-04-02T04:54:07.908619+0000","last_clean":"2024-04-02T04:54:07.908619+0000","last_became_active":"2024-04-02T04:49:29.856708+0000","last_became_peered":"2024-04-02T04:49:29.856708+0000","last_unstale":"2024-04-02T04:54:07.908619+0000","last_undegraded":"2024-04-02T04:54:07.908619+0000","last_fullsized":"2024-04-02T04:54:07.908619+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:27.695437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.695437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905979+0000","last_change":"2024-04-02T04:51:29.649636+0000","last_active":"2024-04-02T04:54:07.905979+0000","last_peered":"2024-04-02T04:54:07.905979+0000","last_clean":"2024-04-02T04:54:07.905979+0000","last_became_active":"2024-04-02T04:49:29.816194+0000","last_became_peered":"2024-04-02T04:49:29.816194+0000","last_unstale":"2024-04-02T04:54:07.905979+0000","last_undegraded":"2024-04-02T04:54:07.905979+0000","last_fullsized":"2024-04-02T04:54:07.905979+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.649422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.649422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":561,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.917401+0000","last_change":"2024-04-02T04:54:05.882430+0000","last_active":"2024-04-02T04:54:06.917401+0000","last_peered":"2024-04-02T04:54:06.917401+0000","last_clean":"2024-04-02T04:54:06.917401+0000","last_became_active":"2024-04-02T04:53:54.905940+0000","last_became_peered":"2024-04-02T04:53:54.905940+0000","last_unstale":"2024-04-02T04:54:06.917401+0000","last_undegraded":"2024-04-02T04:54:06.917401+0000","last_fullsized":"2024-04-02T04:54:06.917401+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.882197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.882197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:20.853459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.900994+0000","last_change":"2024-04-02T04:53:54.917101+0000","last_active":"2024-04-02T04:54:06.900994+0000","last_peered":"2024-04-02T04:54:06.900994+0000","last_clean":"2024-04-02T04:54:06.900994+0000","last_became_active":"2024-04-02T04:53:54.916649+0000","last_became_peered":"2024-04-02T04:53:54.916649+0000","last_unstale":"2024-04-02T04:54:06.900994+0000","last_undegraded":"2024-04-02T04:54:06.900994+0000","last_fullsized":"2024-04-02T04:54:06.900994+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907337+0000","last_change":"2024-04-02T04:52:44.252234+0000","last_active":"2024-04-02T04:54:07.907337+0000","last_peered":"2024-04-02T04:54:07.907337+0000","last_clean":"2024-04-02T04:54:07.907337+0000","last_became_active":"2024-04-02T04:49:28.734672+0000","last_became_peered":"2024-04-02T04:49:28.734672+0000","last_unstale":"2024-04-02T04:54:07.907337+0000","last_undegraded":"2024-04-02T04:54:07.907337+0000","last_fullsized":"2024-04-02T04:54:07.907337+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.252102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.252102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":560,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919405+0000","last_change":"2024-04-02T04:52:56.597071+0000","last_active":"2024-04-02T04:54:06.919405+0000","last_peered":"2024-04-02T04:54:06.919405+0000","last_clean":"2024-04-02T04:54:06.919405+0000","last_became_active":"2024-04-02T04:49:28.749072+0000","last_became_peered":"2024-04-02T04:49:28.749072+0000","last_unstale":"2024-04-02T04:54:06.919405+0000","last_undegraded":"2024-04-02T04:54:06.919405+0000","last_fullsized":"2024-04-02T04:54:06.919405+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.596968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.596968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:26.432888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904842+0000","last_change":"2024-04-02T04:51:51.741608+0000","last_active":"2024-04-02T04:54:07.904842+0000","last_peered":"2024-04-02T04:54:07.904842+0000","last_clean":"2024-04-02T04:54:07.904842+0000","last_became_active":"2024-04-02T04:49:29.858586+0000","last_became_peered":"2024-04-02T04:49:29.858586+0000","last_unstale":"2024-04-02T04:54:07.904842+0000","last_undegraded":"2024-04-02T04:54:07.904842+0000","last_fullsized":"2024-04-02T04:54:07.904842+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.741382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.741382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":566,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906554+0000","last_change":"2024-04-02T04:54:02.337934+0000","last_active":"2024-04-02T04:54:07.906554+0000","last_peered":"2024-04-02T04:54:07.906554+0000","last_clean":"2024-04-02T04:54:07.906554+0000","last_became_active":"2024-04-02T04:53:54.929052+0000","last_became_peered":"2024-04-02T04:53:54.929052+0000","last_unstale":"2024-04-02T04:54:07.906554+0000","last_undegraded":"2024-04-02T04:54:07.906554+0000","last_fullsized":"2024-04-02T04:54:07.906554+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:02.337688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:02.337688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:09.027443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.919971+0000","last_change":"2024-04-02T04:53:54.944277+0000","last_active":"2024-04-02T04:54:08.919971+0000","last_peered":"2024-04-02T04:54:08.919971+0000","last_clean":"2024-04-02T04:54:08.919971+0000","last_became_active":"2024-04-02T04:53:54.943480+0000","last_became_peered":"2024-04-02T04:53:54.943480+0000","last_unstale":"2024-04-02T04:54:08.919971+0000","last_undegraded":"2024-04-02T04:54:08.919971+0000","last_fullsized":"2024-04-02T04:54:08.919971+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.232866+0000","last_change":"2024-04-02T04:51:55.540774+0000","last_active":"2024-04-02T04:54:07.232866+0000","last_peered":"2024-04-02T04:54:07.232866+0000","last_clean":"2024-04-02T04:54:07.232866+0000","last_became_active":"2024-04-02T04:49:28.750713+0000","last_became_peered":"2024-04-02T04:49:28.750713+0000","last_unstale":"2024-04-02T04:54:07.232866+0000","last_undegraded":"2024-04-02T04:54:07.232866+0000","last_fullsized":"2024-04-02T04:54:07.232866+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.540654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.540654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.927990+0000","last_change":"2024-04-02T04:52:53.411217+0000","last_active":"2024-04-02T04:54:06.927990+0000","last_peered":"2024-04-02T04:54:06.927990+0000","last_clean":"2024-04-02T04:54:06.927990+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:54:06.927990+0000","last_undegraded":"2024-04-02T04:54:06.927990+0000","last_fullsized":"2024-04-02T04:54:06.927990+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:53.410913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.410913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:16.643056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.922784+0000","last_change":"2024-04-02T04:53:54.936976+0000","last_active":"2024-04-02T04:54:08.922784+0000","last_peered":"2024-04-02T04:54:08.922784+0000","last_clean":"2024-04-02T04:54:08.922784+0000","last_became_active":"2024-04-02T04:53:54.933626+0000","last_became_peered":"2024-04-02T04:53:54.933626+0000","last_unstale":"2024-04-02T04:54:08.922784+0000","last_undegraded":"2024-04-02T04:54:08.922784+0000","last_fullsized":"2024-04-02T04:54:08.922784+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":560,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.922005+0000","last_change":"2024-04-02T04:52:48.599707+0000","last_active":"2024-04-02T04:54:06.922005+0000","last_peered":"2024-04-02T04:54:06.922005+0000","last_clean":"2024-04-02T04:54:06.922005+0000","last_became_active":"2024-04-02T04:49:28.716933+0000","last_became_peered":"2024-04-02T04:49:28.716933+0000","last_unstale":"2024-04-02T04:54:06.922005+0000","last_undegraded":"2024-04-02T04:54:06.922005+0000","last_fullsized":"2024-04-02T04:54:06.922005+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:48.599595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.599595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:11.497654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911622+0000","last_change":"2024-04-02T04:51:52.390967+0000","last_active":"2024-04-02T04:54:06.911622+0000","last_peered":"2024-04-02T04:54:06.911622+0000","last_clean":"2024-04-02T04:54:06.911622+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:54:06.911622+0000","last_undegraded":"2024-04-02T04:54:06.911622+0000","last_fullsized":"2024-04-02T04:54:06.911622+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.390694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.390694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926427+0000","last_change":"2024-04-02T04:53:11.473195+0000","last_active":"2024-04-02T04:54:06.926427+0000","last_peered":"2024-04-02T04:54:06.926427+0000","last_clean":"2024-04-02T04:54:06.926427+0000","last_became_active":"2024-04-02T04:49:28.757217+0000","last_became_peered":"2024-04-02T04:49:28.757217+0000","last_unstale":"2024-04-02T04:54:06.926427+0000","last_undegraded":"2024-04-02T04:54:06.926427+0000","last_fullsized":"2024-04-02T04:54:06.926427+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.473096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.473096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:11.825694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":566,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910530+0000","last_change":"2024-04-02T04:54:04.337974+0000","last_active":"2024-04-02T04:54:07.910530+0000","last_peered":"2024-04-02T04:54:07.910530+0000","last_clean":"2024-04-02T04:54:07.910530+0000","last_became_active":"2024-04-02T04:53:54.940640+0000","last_became_peered":"2024-04-02T04:53:54.940640+0000","last_unstale":"2024-04-02T04:54:07.910530+0000","last_undegraded":"2024-04-02T04:54:07.910530+0000","last_fullsized":"2024-04-02T04:54:07.910530+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.337869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.337869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:07.693911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.918991+0000","last_change":"2024-04-02T04:51:25.125020+0000","last_active":"2024-04-02T04:54:06.918991+0000","last_peered":"2024-04-02T04:54:06.918991+0000","last_clean":"2024-04-02T04:54:06.918991+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:54:06.918991+0000","last_undegraded":"2024-04-02T04:54:06.918991+0000","last_fullsized":"2024-04-02T04:54:06.918991+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:25.124929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.124929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":558,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912771+0000","last_change":"2024-04-02T04:53:54.921421+0000","last_active":"2024-04-02T04:54:07.912771+0000","last_peered":"2024-04-02T04:54:07.912771+0000","last_clean":"2024-04-02T04:54:07.912771+0000","last_became_active":"2024-04-02T04:53:54.921036+0000","last_became_peered":"2024-04-02T04:53:54.921036+0000","last_unstale":"2024-04-02T04:54:07.912771+0000","last_undegraded":"2024-04-02T04:54:07.912771+0000","last_fullsized":"2024-04-02T04:54:07.912771+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":553,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919170+0000","last_change":"2024-04-02T04:53:54.945162+0000","last_active":"2024-04-02T04:54:06.919170+0000","last_peered":"2024-04-02T04:54:06.919170+0000","last_clean":"2024-04-02T04:54:06.919170+0000","last_became_active":"2024-04-02T04:53:54.941043+0000","last_became_peered":"2024-04-02T04:53:54.941043+0000","last_unstale":"2024-04-02T04:54:06.919170+0000","last_undegraded":"2024-04-02T04:54:06.919170+0000","last_fullsized":"2024-04-02T04:54:06.919170+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233586+0000","last_change":"2024-04-02T04:51:29.673932+0000","last_active":"2024-04-02T04:54:07.233586+0000","last_peered":"2024-04-02T04:54:07.233586+0000","last_clean":"2024-04-02T04:54:07.233586+0000","last_became_active":"2024-04-02T04:49:28.765229+0000","last_became_peered":"2024-04-02T04:49:28.765229+0000","last_unstale":"2024-04-02T04:54:07.233586+0000","last_undegraded":"2024-04-02T04:54:07.233586+0000","last_fullsized":"2024-04-02T04:54:07.233586+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.673796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.673796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.917897+0000","last_change":"2024-04-02T04:53:54.947489+0000","last_active":"2024-04-02T04:54:08.917897+0000","last_peered":"2024-04-02T04:54:08.917897+0000","last_clean":"2024-04-02T04:54:08.917897+0000","last_became_active":"2024-04-02T04:53:54.945599+0000","last_became_peered":"2024-04-02T04:53:54.945599+0000","last_unstale":"2024-04-02T04:54:08.917897+0000","last_undegraded":"2024-04-02T04:54:08.917897+0000","last_fullsized":"2024-04-02T04:54:08.917897+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":162,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905118+0000","last_change":"2024-04-02T04:52:58.361166+0000","last_active":"2024-04-02T04:54:07.905118+0000","last_peered":"2024-04-02T04:54:07.905118+0000","last_clean":"2024-04-02T04:54:07.905118+0000","last_became_active":"2024-04-02T04:52:55.872480+0000","last_became_peered":"2024-04-02T04:52:55.872480+0000","last_unstale":"2024-04-02T04:54:07.905118+0000","last_undegraded":"2024-04-02T04:54:07.905118+0000","last_fullsized":"2024-04-02T04:54:07.905118+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.361050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.361050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:12.351179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":561,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919872+0000","last_change":"2024-04-02T04:54:03.883895+0000","last_active":"2024-04-02T04:54:06.919872+0000","last_peered":"2024-04-02T04:54:06.919872+0000","last_clean":"2024-04-02T04:54:06.919872+0000","last_became_active":"2024-04-02T04:53:54.940670+0000","last_became_peered":"2024-04-02T04:53:54.940670+0000","last_unstale":"2024-04-02T04:54:06.919872+0000","last_undegraded":"2024-04-02T04:54:06.919872+0000","last_fullsized":"2024-04-02T04:54:06.919872+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.883769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.883769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.172141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229465+0000","last_change":"2024-04-02T04:51:36.711304+0000","last_active":"2024-04-02T04:54:07.229465+0000","last_peered":"2024-04-02T04:54:07.229465+0000","last_clean":"2024-04-02T04:54:07.229465+0000","last_became_active":"2024-04-02T04:49:28.743589+0000","last_became_peered":"2024-04-02T04:49:28.743589+0000","last_unstale":"2024-04-02T04:54:07.229465+0000","last_undegraded":"2024-04-02T04:54:07.229465+0000","last_fullsized":"2024-04-02T04:54:07.229465+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.711178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.711178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":558,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908407+0000","last_change":"2024-04-02T04:53:54.944777+0000","last_active":"2024-04-02T04:54:07.908407+0000","last_peered":"2024-04-02T04:54:07.908407+0000","last_clean":"2024-04-02T04:54:07.908407+0000","last_became_active":"2024-04-02T04:53:54.943555+0000","last_became_peered":"2024-04-02T04:53:54.943555+0000","last_unstale":"2024-04-02T04:54:07.908407+0000","last_undegraded":"2024-04-02T04:54:07.908407+0000","last_fullsized":"2024-04-02T04:54:07.908407+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":557,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905875+0000","last_change":"2024-04-02T04:52:49.762675+0000","last_active":"2024-04-02T04:54:07.905875+0000","last_peered":"2024-04-02T04:54:07.905875+0000","last_clean":"2024-04-02T04:54:07.905875+0000","last_became_active":"2024-04-02T04:49:29.805341+0000","last_became_peered":"2024-04-02T04:49:29.805341+0000","last_unstale":"2024-04-02T04:54:07.905875+0000","last_undegraded":"2024-04-02T04:54:07.905875+0000","last_fullsized":"2024-04-02T04:54:07.905875+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.762442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.762442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":51,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918795+0000","last_change":"2024-04-02T04:53:54.943643+0000","last_active":"2024-04-02T04:54:08.918795+0000","last_peered":"2024-04-02T04:54:08.918795+0000","last_clean":"2024-04-02T04:54:08.918795+0000","last_became_active":"2024-04-02T04:53:54.943220+0000","last_became_peered":"2024-04-02T04:53:54.943220+0000","last_unstale":"2024-04-02T04:54:08.918795+0000","last_undegraded":"2024-04-02T04:54:08.918795+0000","last_fullsized":"2024-04-02T04:54:08.918795+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229042+0000","last_change":"2024-04-02T04:51:45.521690+0000","last_active":"2024-04-02T04:54:07.229042+0000","last_peered":"2024-04-02T04:54:07.229042+0000","last_clean":"2024-04-02T04:54:07.229042+0000","last_became_active":"2024-04-02T04:49:28.766979+0000","last_became_peered":"2024-04-02T04:49:28.766979+0000","last_unstale":"2024-04-02T04:54:07.229042+0000","last_undegraded":"2024-04-02T04:54:07.229042+0000","last_fullsized":"2024-04-02T04:54:07.229042+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:45.521589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:45.521589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":554,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901688+0000","last_change":"2024-04-02T04:52:42.318994+0000","last_active":"2024-04-02T04:54:06.901688+0000","last_peered":"2024-04-02T04:54:06.901688+0000","last_clean":"2024-04-02T04:54:06.901688+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:54:06.901688+0000","last_undegraded":"2024-04-02T04:54:06.901688+0000","last_fullsized":"2024-04-02T04:54:06.901688+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.318800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.318800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.925220+0000","last_change":"2024-04-02T04:52:52.417888+0000","last_active":"2024-04-02T04:54:06.925220+0000","last_peered":"2024-04-02T04:54:06.925220+0000","last_clean":"2024-04-02T04:54:06.925220+0000","last_became_active":"2024-04-02T04:49:28.753376+0000","last_became_peered":"2024-04-02T04:49:28.753376+0000","last_unstale":"2024-04-02T04:54:06.925220+0000","last_undegraded":"2024-04-02T04:54:06.925220+0000","last_fullsized":"2024-04-02T04:54:06.925220+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.417771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.417771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:11.133064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915831+0000","last_change":"2024-04-02T04:50:44.257998+0000","last_active":"2024-04-02T04:54:06.915831+0000","last_peered":"2024-04-02T04:54:06.915831+0000","last_clean":"2024-04-02T04:54:06.915831+0000","last_became_active":"2024-04-02T04:49:28.757583+0000","last_became_peered":"2024-04-02T04:49:28.757583+0000","last_unstale":"2024-04-02T04:54:06.915831+0000","last_undegraded":"2024-04-02T04:54:06.915831+0000","last_fullsized":"2024-04-02T04:54:06.915831+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.257747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.257747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914139+0000","last_change":"2024-04-02T04:51:49.984059+0000","last_active":"2024-04-02T04:54:06.914139+0000","last_peered":"2024-04-02T04:54:06.914139+0000","last_clean":"2024-04-02T04:54:06.914139+0000","last_became_active":"2024-04-02T04:49:28.726425+0000","last_became_peered":"2024-04-02T04:49:28.726425+0000","last_unstale":"2024-04-02T04:54:06.914139+0000","last_undegraded":"2024-04-02T04:54:06.914139+0000","last_fullsized":"2024-04-02T04:54:06.914139+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.983966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.983966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"139'5","reported_seq":561,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919870+0000","last_change":"2024-04-02T04:50:26.846127+0000","last_active":"2024-04-02T04:54:06.919870+0000","last_peered":"2024-04-02T04:54:06.919870+0000","last_clean":"2024-04-02T04:54:06.919870+0000","last_became_active":"2024-04-02T04:49:26.170913+0000","last_became_peered":"2024-04-02T04:49:26.170913+0000","last_unstale":"2024-04-02T04:54:06.919870+0000","last_undegraded":"2024-04-02T04:54:06.919870+0000","last_fullsized":"2024-04-02T04:54:06.919870+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:50:26.845900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.845900+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233055+0000","last_change":"2024-04-02T04:51:30.643523+0000","last_active":"2024-04-02T04:54:07.233055+0000","last_peered":"2024-04-02T04:54:07.233055+0000","last_clean":"2024-04-02T04:54:07.233055+0000","last_became_active":"2024-04-02T04:49:26.216834+0000","last_became_peered":"2024-04-02T04:49:26.216834+0000","last_unstale":"2024-04-02T04:54:07.233055+0000","last_undegraded":"2024-04-02T04:54:07.233055+0000","last_fullsized":"2024-04-02T04:54:07.233055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.643306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.643306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":702,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.228725+0000","last_change":"2024-04-02T04:53:54.920493+0000","last_active":"2024-04-02T04:54:07.228725+0000","last_peered":"2024-04-02T04:54:07.228725+0000","last_clean":"2024-04-02T04:54:07.228725+0000","last_became_active":"2024-04-02T04:53:54.919695+0000","last_became_peered":"2024-04-02T04:53:54.919695+0000","last_unstale":"2024-04-02T04:54:07.228725+0000","last_undegraded":"2024-04-02T04:54:07.228725+0000","last_fullsized":"2024-04-02T04:54:07.228725+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:06.797658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:06.797658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"402'27","reported_seq":218,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.921920+0000","last_change":"2024-04-02T04:53:54.940511+0000","last_active":"2024-04-02T04:54:08.921920+0000","last_peered":"2024-04-02T04:54:08.921920+0000","last_clean":"2024-04-02T04:54:08.921920+0000","last_became_active":"2024-04-02T04:53:54.938166+0000","last_became_peered":"2024-04-02T04:53:54.938166+0000","last_unstale":"2024-04-02T04:54:08.921920+0000","last_undegraded":"2024-04-02T04:54:08.921920+0000","last_fullsized":"2024-04-02T04:54:08.921920+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:48:47.429018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:47.429018+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":708,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921641+0000","last_change":"2024-04-02T04:54:00.930078+0000","last_active":"2024-04-02T04:54:06.921641+0000","last_peered":"2024-04-02T04:54:06.921641+0000","last_clean":"2024-04-02T04:54:06.921641+0000","last_became_active":"2024-04-02T04:53:54.938837+0000","last_became_peered":"2024-04-02T04:53:54.938837+0000","last_unstale":"2024-04-02T04:54:06.921641+0000","last_undegraded":"2024-04-02T04:54:06.921641+0000","last_fullsized":"2024-04-02T04:54:06.921641+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.929851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.929851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:02.359890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":697,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230950+0000","last_change":"2024-04-02T04:52:46.686069+0000","last_active":"2024-04-02T04:54:07.230950+0000","last_peered":"2024-04-02T04:54:07.230950+0000","last_clean":"2024-04-02T04:54:07.230950+0000","last_became_active":"2024-04-02T04:44:00.348259+0000","last_became_peered":"2024-04-02T04:44:00.348259+0000","last_unstale":"2024-04-02T04:54:07.230950+0000","last_undegraded":"2024-04-02T04:54:07.230950+0000","last_fullsized":"2024-04-02T04:54:07.230950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.685936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.685936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229849+0000","last_change":"2024-04-02T04:51:38.662185+0000","last_active":"2024-04-02T04:54:07.229849+0000","last_peered":"2024-04-02T04:54:07.229849+0000","last_clean":"2024-04-02T04:54:07.229849+0000","last_became_active":"2024-04-02T04:49:26.241757+0000","last_became_peered":"2024-04-02T04:49:26.241757+0000","last_unstale":"2024-04-02T04:54:07.229849+0000","last_undegraded":"2024-04-02T04:54:07.229849+0000","last_fullsized":"2024-04-02T04:54:07.229849+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:38.662076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.662076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":565,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921296+0000","last_change":"2024-04-02T04:52:57.641001+0000","last_active":"2024-04-02T04:54:06.921296+0000","last_peered":"2024-04-02T04:54:06.921296+0000","last_clean":"2024-04-02T04:54:06.921296+0000","last_became_active":"2024-04-02T04:49:26.296797+0000","last_became_peered":"2024-04-02T04:49:26.296797+0000","last_unstale":"2024-04-02T04:54:06.921296+0000","last_undegraded":"2024-04-02T04:54:06.921296+0000","last_fullsized":"2024-04-02T04:54:06.921296+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.640866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.640866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:12.765945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920990+0000","last_change":"2024-04-02T04:52:55.335191+0000","last_active":"2024-04-02T04:54:06.920990+0000","last_peered":"2024-04-02T04:54:06.920990+0000","last_clean":"2024-04-02T04:54:06.920990+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:54:06.920990+0000","last_undegraded":"2024-04-02T04:54:06.920990+0000","last_fullsized":"2024-04-02T04:54:06.920990+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:55.334980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.334980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"402'25","reported_seq":748,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233075+0000","last_change":"2024-04-02T04:53:07.600488+0000","last_active":"2024-04-02T04:54:07.233075+0000","last_peered":"2024-04-02T04:54:07.233075+0000","last_clean":"2024-04-02T04:54:07.233075+0000","last_became_active":"2024-04-02T04:43:56.247780+0000","last_became_peered":"2024-04-02T04:43:56.247780+0000","last_unstale":"2024-04-02T04:54:07.233075+0000","last_undegraded":"2024-04-02T04:54:07.233075+0000","last_fullsized":"2024-04-02T04:54:07.233075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:53:07.600375+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.600375+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:32.336251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":721,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.911366+0000","last_change":"2024-04-02T04:53:10.415912+0000","last_active":"2024-04-02T04:54:07.911366+0000","last_peered":"2024-04-02T04:54:07.911366+0000","last_clean":"2024-04-02T04:54:07.911366+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:54:07.911366+0000","last_undegraded":"2024-04-02T04:54:07.911366+0000","last_fullsized":"2024-04-02T04:54:07.911366+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.415766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:17.931537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":682,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915920+0000","last_change":"2024-04-02T04:53:54.936368+0000","last_active":"2024-04-02T04:54:06.915920+0000","last_peered":"2024-04-02T04:54:06.915920+0000","last_clean":"2024-04-02T04:54:06.915920+0000","last_became_active":"2024-04-02T04:53:54.935778+0000","last_became_peered":"2024-04-02T04:53:54.935778+0000","last_unstale":"2024-04-02T04:54:06.915920+0000","last_undegraded":"2024-04-02T04:54:06.915920+0000","last_fullsized":"2024-04-02T04:54:06.915920+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:36.242786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:47:36.242786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"139'5","reported_seq":564,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921920+0000","last_change":"2024-04-02T04:51:27.143046+0000","last_active":"2024-04-02T04:54:06.921920+0000","last_peered":"2024-04-02T04:54:06.921920+0000","last_clean":"2024-04-02T04:54:06.921920+0000","last_became_active":"2024-04-02T04:49:25.139936+0000","last_became_peered":"2024-04-02T04:49:25.139936+0000","last_unstale":"2024-04-02T04:54:06.921920+0000","last_undegraded":"2024-04-02T04:54:06.921920+0000","last_fullsized":"2024-04-02T04:54:06.921920+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:27.142908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.142908+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":566,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919602+0000","last_change":"2024-04-02T04:53:10.335701+0000","last_active":"2024-04-02T04:54:06.919602+0000","last_peered":"2024-04-02T04:54:06.919602+0000","last_clean":"2024-04-02T04:54:06.919602+0000","last_became_active":"2024-04-02T04:49:25.143060+0000","last_became_peered":"2024-04-02T04:49:25.143060+0000","last_unstale":"2024-04-02T04:54:06.919602+0000","last_undegraded":"2024-04-02T04:54:06.919602+0000","last_fullsized":"2024-04-02T04:54:06.919602+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.335602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.335602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:28.770529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921049+0000","last_change":"2024-04-02T04:54:01.950572+0000","last_active":"2024-04-02T04:54:06.921049+0000","last_peered":"2024-04-02T04:54:06.921049+0000","last_clean":"2024-04-02T04:54:06.921049+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T04:54:06.921049+0000","last_undegraded":"2024-04-02T04:54:06.921049+0000","last_fullsized":"2024-04-02T04:54:06.921049+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.950469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.950469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:24.390817+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"402'31","reported_seq":230,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.923982+0000","last_change":"2024-04-02T04:53:54.945712+0000","last_active":"2024-04-02T04:54:08.923982+0000","last_peered":"2024-04-02T04:54:08.923982+0000","last_clean":"2024-04-02T04:54:08.923982+0000","last_became_active":"2024-04-02T04:53:54.945328+0000","last_became_peered":"2024-04-02T04:53:54.945328+0000","last_unstale":"2024-04-02T04:54:08.923982+0000","last_undegraded":"2024-04-02T04:54:08.923982+0000","last_fullsized":"2024-04-02T04:54:08.923982+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:00.356122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.356122+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":707,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919484+0000","last_change":"2024-04-02T04:52:38.475693+0000","last_active":"2024-04-02T04:54:06.919484+0000","last_peered":"2024-04-02T04:54:06.919484+0000","last_clean":"2024-04-02T04:54:06.919484+0000","last_became_active":"2024-04-02T04:43:54.213465+0000","last_became_peered":"2024-04-02T04:43:54.213465+0000","last_unstale":"2024-04-02T04:54:06.919484+0000","last_undegraded":"2024-04-02T04:54:06.919484+0000","last_fullsized":"2024-04-02T04:54:06.919484+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.475462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.475462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":189,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918320+0000","last_change":"2024-04-02T04:54:08.209919+0000","last_active":"2024-04-02T04:54:08.918320+0000","last_peered":"2024-04-02T04:54:08.918320+0000","last_clean":"2024-04-02T04:54:08.918320+0000","last_became_active":"2024-04-02T04:53:54.934051+0000","last_became_peered":"2024-04-02T04:53:54.934051+0000","last_unstale":"2024-04-02T04:54:08.918320+0000","last_undegraded":"2024-04-02T04:54:08.918320+0000","last_fullsized":"2024-04-02T04:54:08.918320+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.209709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.209709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:27.901075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926080+0000","last_change":"2024-04-02T04:52:34.410043+0000","last_active":"2024-04-02T04:54:06.926080+0000","last_peered":"2024-04-02T04:54:06.926080+0000","last_clean":"2024-04-02T04:54:06.926080+0000","last_became_active":"2024-04-02T04:49:26.271155+0000","last_became_peered":"2024-04-02T04:49:26.271155+0000","last_unstale":"2024-04-02T04:54:06.926080+0000","last_undegraded":"2024-04-02T04:54:06.926080+0000","last_fullsized":"2024-04-02T04:54:06.926080+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.409934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.409934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":568,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919501+0000","last_change":"2024-04-02T04:53:54.055128+0000","last_active":"2024-04-02T04:54:06.919501+0000","last_peered":"2024-04-02T04:54:06.919501+0000","last_clean":"2024-04-02T04:54:06.919501+0000","last_became_active":"2024-04-02T04:49:26.265834+0000","last_became_peered":"2024-04-02T04:49:26.265834+0000","last_unstale":"2024-04-02T04:54:06.919501+0000","last_undegraded":"2024-04-02T04:54:06.919501+0000","last_fullsized":"2024-04-02T04:54:06.919501+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:54.055039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.055039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:06.712487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"402'42","reported_seq":772,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.922054+0000","last_change":"2024-04-02T04:51:43.069937+0000","last_active":"2024-04-02T04:54:06.922054+0000","last_peered":"2024-04-02T04:54:06.922054+0000","last_clean":"2024-04-02T04:54:06.922054+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:54:06.922054+0000","last_undegraded":"2024-04-02T04:54:06.922054+0000","last_fullsized":"2024-04-02T04:54:06.922054+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:51:43.069694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.069694+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":187,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.919358+0000","last_change":"2024-04-02T04:53:59.105264+0000","last_active":"2024-04-02T04:54:08.919358+0000","last_peered":"2024-04-02T04:54:08.919358+0000","last_clean":"2024-04-02T04:54:08.919358+0000","last_became_active":"2024-04-02T04:53:54.931953+0000","last_became_peered":"2024-04-02T04:53:54.931953+0000","last_unstale":"2024-04-02T04:54:08.919358+0000","last_undegraded":"2024-04-02T04:54:08.919358+0000","last_fullsized":"2024-04-02T04:54:08.919358+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:59.105134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.105134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:16.434076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":828,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911354+0000","last_change":"2024-04-02T04:50:15.516760+0000","last_active":"2024-04-02T04:54:06.911354+0000","last_peered":"2024-04-02T04:54:06.911354+0000","last_clean":"2024-04-02T04:54:06.911354+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:54:06.911354+0000","last_undegraded":"2024-04-02T04:54:06.911354+0000","last_fullsized":"2024-04-02T04:54:06.911354+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:50:15.516538+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.516538+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":703,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.922519+0000","last_change":"2024-04-02T04:51:50.152380+0000","last_active":"2024-04-02T04:54:06.922519+0000","last_peered":"2024-04-02T04:54:06.922519+0000","last_clean":"2024-04-02T04:54:06.922519+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:54:06.922519+0000","last_undegraded":"2024-04-02T04:54:06.922519+0000","last_fullsized":"2024-04-02T04:54:06.922519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:50.152204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.152204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.906744+0000","last_change":"2024-04-02T04:51:37.350257+0000","last_active":"2024-04-02T04:54:06.906744+0000","last_peered":"2024-04-02T04:54:06.906744+0000","last_clean":"2024-04-02T04:54:06.906744+0000","last_became_active":"2024-04-02T04:49:26.241469+0000","last_became_peered":"2024-04-02T04:49:26.241469+0000","last_unstale":"2024-04-02T04:54:06.906744+0000","last_undegraded":"2024-04-02T04:54:06.906744+0000","last_fullsized":"2024-04-02T04:54:06.906744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:37.350152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:37.350152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":562,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906335+0000","last_change":"2024-04-02T04:52:42.269689+0000","last_active":"2024-04-02T04:54:07.906335+0000","last_peered":"2024-04-02T04:54:07.906335+0000","last_clean":"2024-04-02T04:54:07.906335+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:54:07.906335+0000","last_undegraded":"2024-04-02T04:54:07.906335+0000","last_fullsized":"2024-04-02T04:54:07.906335+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.268948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.268948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":694,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.906249+0000","last_change":"2024-04-02T04:50:43.513008+0000","last_active":"2024-04-02T04:54:06.906249+0000","last_peered":"2024-04-02T04:54:06.906249+0000","last_clean":"2024-04-02T04:54:06.906249+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:54:06.906249+0000","last_undegraded":"2024-04-02T04:54:06.906249+0000","last_fullsized":"2024-04-02T04:54:06.906249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:43.512888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.512888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"402'26","reported_seq":753,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909765+0000","last_change":"2024-04-02T04:53:04.791017+0000","last_active":"2024-04-02T04:54:07.909765+0000","last_peered":"2024-04-02T04:54:07.909765+0000","last_clean":"2024-04-02T04:54:07.909765+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:54:07.909765+0000","last_undegraded":"2024-04-02T04:54:07.909765+0000","last_fullsized":"2024-04-02T04:54:07.909765+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:53:04.790916+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.790916+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:10.920544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":722,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906460+0000","last_change":"2024-04-02T04:54:01.301186+0000","last_active":"2024-04-02T04:54:07.906460+0000","last_peered":"2024-04-02T04:54:07.906460+0000","last_clean":"2024-04-02T04:54:07.906460+0000","last_became_active":"2024-04-02T04:53:54.935402+0000","last_became_peered":"2024-04-02T04:53:54.935402+0000","last_unstale":"2024-04-02T04:54:07.906460+0000","last_undegraded":"2024-04-02T04:54:07.906460+0000","last_fullsized":"2024-04-02T04:54:07.906460+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.301076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.301076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:01.696601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":690,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.930705+0000","last_change":"2024-04-02T04:50:18.501638+0000","last_active":"2024-04-02T04:54:06.930705+0000","last_peered":"2024-04-02T04:54:06.930705+0000","last_clean":"2024-04-02T04:54:06.930705+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:54:06.930705+0000","last_undegraded":"2024-04-02T04:54:06.930705+0000","last_fullsized":"2024-04-02T04:54:06.930705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.501528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.501528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"139'9","reported_seq":568,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.912450+0000","last_change":"2024-04-02T04:51:29.324138+0000","last_active":"2024-04-02T04:54:06.912450+0000","last_peered":"2024-04-02T04:54:06.912450+0000","last_clean":"2024-04-02T04:54:06.912450+0000","last_became_active":"2024-04-02T04:49:26.487105+0000","last_became_peered":"2024-04-02T04:49:26.487105+0000","last_unstale":"2024-04-02T04:54:06.912450+0000","last_undegraded":"2024-04-02T04:54:06.912450+0000","last_fullsized":"2024-04-02T04:54:06.912450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'9","last_scrub_stamp":"2024-04-02T04:51:29.324015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.324015+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":573,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901886+0000","last_change":"2024-04-02T04:53:54.918674+0000","last_active":"2024-04-02T04:54:06.901886+0000","last_peered":"2024-04-02T04:54:06.901886+0000","last_clean":"2024-04-02T04:54:06.901886+0000","last_became_active":"2024-04-02T04:53:54.916707+0000","last_became_peered":"2024-04-02T04:53:54.916707+0000","last_unstale":"2024-04-02T04:54:06.901886+0000","last_undegraded":"2024-04-02T04:54:06.901886+0000","last_fullsized":"2024-04-02T04:54:06.901886+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"402'48","reported_seq":788,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904809+0000","last_change":"2024-04-02T04:53:54.927487+0000","last_active":"2024-04-02T04:54:07.904809+0000","last_peered":"2024-04-02T04:54:07.904809+0000","last_clean":"2024-04-02T04:54:07.904809+0000","last_became_active":"2024-04-02T04:53:54.926993+0000","last_became_peered":"2024-04-02T04:53:54.926993+0000","last_unstale":"2024-04-02T04:54:07.904809+0000","last_undegraded":"2024-04-02T04:54:07.904809+0000","last_fullsized":"2024-04-02T04:54:07.904809+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'36","last_scrub_stamp":"2024-04-02T04:49:24.269994+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.269994+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":719,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907043+0000","last_change":"2024-04-02T04:50:44.352622+0000","last_active":"2024-04-02T04:54:07.907043+0000","last_peered":"2024-04-02T04:54:07.907043+0000","last_clean":"2024-04-02T04:54:07.907043+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:54:07.907043+0000","last_undegraded":"2024-04-02T04:54:07.907043+0000","last_fullsized":"2024-04-02T04:54:07.907043+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.352505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.352505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":648,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.922727+0000","last_change":"2024-04-02T04:53:07.785865+0000","last_active":"2024-04-02T04:54:06.922727+0000","last_peered":"2024-04-02T04:54:06.922727+0000","last_clean":"2024-04-02T04:54:06.922727+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:54:06.922727+0000","last_undegraded":"2024-04-02T04:54:06.922727+0000","last_fullsized":"2024-04-02T04:54:06.922727+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.785753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.785753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.493653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":705,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.231710+0000","last_change":"2024-04-02T04:53:03.659862+0000","last_active":"2024-04-02T04:54:07.231710+0000","last_peered":"2024-04-02T04:54:07.231710+0000","last_clean":"2024-04-02T04:54:07.231710+0000","last_became_active":"2024-04-02T04:44:00.333235+0000","last_became_peered":"2024-04-02T04:44:00.333235+0000","last_unstale":"2024-04-02T04:54:07.231710+0000","last_undegraded":"2024-04-02T04:54:07.231710+0000","last_fullsized":"2024-04-02T04:54:07.231710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:03.659593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:53:03.659593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:30.238058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":52,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.921660+0000","last_change":"2024-04-02T04:53:54.926889+0000","last_active":"2024-04-02T04:54:08.921660+0000","last_peered":"2024-04-02T04:54:08.921660+0000","last_clean":"2024-04-02T04:54:08.921660+0000","last_became_active":"2024-04-02T04:53:54.924164+0000","last_became_peered":"2024-04-02T04:53:54.924164+0000","last_unstale":"2024-04-02T04:54:08.921660+0000","last_undegraded":"2024-04-02T04:54:08.921660+0000","last_fullsized":"2024-04-02T04:54:08.921660+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.916130+0000","last_change":"2024-04-02T04:52:42.441055+0000","last_active":"2024-04-02T04:54:06.916130+0000","last_peered":"2024-04-02T04:54:06.916130+0000","last_clean":"2024-04-02T04:54:06.916130+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:54:06.916130+0000","last_undegraded":"2024-04-02T04:54:06.916130+0000","last_fullsized":"2024-04-02T04:54:06.916130+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.440852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.440852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"402'30","reported_seq":763,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.902270+0000","last_change":"2024-04-02T04:54:03.566423+0000","last_active":"2024-04-02T04:54:06.902270+0000","last_peered":"2024-04-02T04:54:06.902270+0000","last_clean":"2024-04-02T04:54:06.902270+0000","last_became_active":"2024-04-02T04:53:54.917946+0000","last_became_peered":"2024-04-02T04:53:54.917946+0000","last_unstale":"2024-04-02T04:54:06.902270+0000","last_undegraded":"2024-04-02T04:54:06.902270+0000","last_fullsized":"2024-04-02T04:54:06.902270+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:54:03.566030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.566030+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.298004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":195,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.917592+0000","last_change":"2024-04-02T04:53:54.946259+0000","last_active":"2024-04-02T04:54:08.917592+0000","last_peered":"2024-04-02T04:54:08.917592+0000","last_clean":"2024-04-02T04:54:08.917592+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T04:54:08.917592+0000","last_undegraded":"2024-04-02T04:54:08.917592+0000","last_fullsized":"2024-04-02T04:54:08.917592+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.344546+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.344546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":827,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913814+0000","last_change":"2024-04-02T04:50:31.493819+0000","last_active":"2024-04-02T04:54:06.913814+0000","last_peered":"2024-04-02T04:54:06.913814+0000","last_clean":"2024-04-02T04:54:06.913814+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:54:06.913814+0000","last_undegraded":"2024-04-02T04:54:06.913814+0000","last_fullsized":"2024-04-02T04:54:06.913814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:31.493715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.493715+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":695,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906122+0000","last_change":"2024-04-02T04:50:34.462138+0000","last_active":"2024-04-02T04:54:07.906122+0000","last_peered":"2024-04-02T04:54:07.906122+0000","last_clean":"2024-04-02T04:54:07.906122+0000","last_became_active":"2024-04-02T04:44:00.358037+0000","last_became_peered":"2024-04-02T04:44:00.358037+0000","last_unstale":"2024-04-02T04:54:07.906122+0000","last_undegraded":"2024-04-02T04:54:07.906122+0000","last_fullsized":"2024-04-02T04:54:07.906122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:34.462064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:50:34.462064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":567,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914437+0000","last_change":"2024-04-02T04:52:53.669076+0000","last_active":"2024-04-02T04:54:06.914437+0000","last_peered":"2024-04-02T04:54:06.914437+0000","last_clean":"2024-04-02T04:54:06.914437+0000","last_became_active":"2024-04-02T04:49:26.319266+0000","last_became_peered":"2024-04-02T04:49:26.319266+0000","last_unstale":"2024-04-02T04:54:06.914437+0000","last_undegraded":"2024-04-02T04:54:06.914437+0000","last_fullsized":"2024-04-02T04:54:06.914437+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:52:53.668865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.668865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.929377+0000","last_change":"2024-04-02T04:50:31.781484+0000","last_active":"2024-04-02T04:54:06.929377+0000","last_peered":"2024-04-02T04:54:06.929377+0000","last_clean":"2024-04-02T04:54:06.929377+0000","last_became_active":"2024-04-02T04:49:26.255429+0000","last_became_peered":"2024-04-02T04:49:26.255429+0000","last_unstale":"2024-04-02T04:54:06.929377+0000","last_undegraded":"2024-04-02T04:54:06.929377+0000","last_fullsized":"2024-04-02T04:54:06.929377+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:31.781273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.781273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":694,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.902537+0000","last_change":"2024-04-02T04:51:40.385896+0000","last_active":"2024-04-02T04:54:06.902537+0000","last_peered":"2024-04-02T04:54:06.902537+0000","last_clean":"2024-04-02T04:54:06.902537+0000","last_became_active":"2024-04-02T04:43:58.276578+0000","last_became_peered":"2024-04-02T04:43:58.276578+0000","last_unstale":"2024-04-02T04:54:06.902537+0000","last_undegraded":"2024-04-02T04:54:06.902537+0000","last_fullsized":"2024-04-02T04:54:06.902537+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.385759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"402'30","reported_seq":747,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923252+0000","last_change":"2024-04-02T04:53:07.411481+0000","last_active":"2024-04-02T04:54:06.923252+0000","last_peered":"2024-04-02T04:54:06.923252+0000","last_clean":"2024-04-02T04:54:06.923252+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:54:06.923252+0000","last_undegraded":"2024-04-02T04:54:06.923252+0000","last_fullsized":"2024-04-02T04:54:06.923252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'18","last_scrub_stamp":"2024-04-02T04:53:07.411390+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.411390+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:19.500892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.929648+0000","last_change":"2024-04-02T04:53:54.913991+0000","last_active":"2024-04-02T04:54:06.929648+0000","last_peered":"2024-04-02T04:54:06.929648+0000","last_clean":"2024-04-02T04:54:06.929648+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T04:54:06.929648+0000","last_undegraded":"2024-04-02T04:54:06.929648+0000","last_fullsized":"2024-04-02T04:54:06.929648+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:08.690362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.690362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":1300,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.905037+0000","last_change":"2024-04-02T04:50:39.632326+0000","last_active":"2024-04-02T04:54:07.905037+0000","last_peered":"2024-04-02T04:54:07.905037+0000","last_clean":"2024-04-02T04:54:07.905037+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:54:07.905037+0000","last_undegraded":"2024-04-02T04:54:07.905037+0000","last_fullsized":"2024-04-02T04:54:07.905037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:50:39.632171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.632171+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":156,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920309+0000","last_change":"2024-04-02T04:53:55.052233+0000","last_active":"2024-04-02T04:54:08.920309+0000","last_peered":"2024-04-02T04:54:08.920309+0000","last_clean":"2024-04-02T04:54:08.920309+0000","last_became_active":"2024-04-02T04:53:54.936665+0000","last_became_peered":"2024-04-02T04:53:54.936665+0000","last_unstale":"2024-04-02T04:54:08.920309+0000","last_undegraded":"2024-04-02T04:54:08.920309+0000","last_fullsized":"2024-04-02T04:54:08.920309+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:55.052156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.052156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:23.649070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":567,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229886+0000","last_change":"2024-04-02T04:53:04.638778+0000","last_active":"2024-04-02T04:54:07.229886+0000","last_peered":"2024-04-02T04:54:07.229886+0000","last_clean":"2024-04-02T04:54:07.229886+0000","last_became_active":"2024-04-02T04:49:26.343060+0000","last_became_peered":"2024-04-02T04:49:26.343060+0000","last_unstale":"2024-04-02T04:54:07.229886+0000","last_undegraded":"2024-04-02T04:54:07.229886+0000","last_fullsized":"2024-04-02T04:54:07.229886+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:04.638640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.638640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.668090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":551,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913041+0000","last_change":"2024-04-02T04:49:26.257833+0000","last_active":"2024-04-02T04:54:06.913041+0000","last_peered":"2024-04-02T04:54:06.913041+0000","last_clean":"2024-04-02T04:54:06.913041+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:54:06.913041+0000","last_undegraded":"2024-04-02T04:54:06.913041+0000","last_fullsized":"2024-04-02T04:54:06.913041+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"395'14","reported_seq":853,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904926+0000","last_change":"2024-04-02T04:53:56.111933+0000","last_active":"2024-04-02T04:54:07.904926+0000","last_peered":"2024-04-02T04:54:07.904926+0000","last_clean":"2024-04-02T04:54:07.904926+0000","last_became_active":"2024-04-02T04:53:54.935396+0000","last_became_peered":"2024-04-02T04:53:54.935396+0000","last_unstale":"2024-04-02T04:54:07.904926+0000","last_undegraded":"2024-04-02T04:54:07.904926+0000","last_fullsized":"2024-04-02T04:54:07.904926+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'14","last_scrub_stamp":"2024-04-02T04:53:56.111652+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:53:56.111652+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:03.107367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"402'23","reported_seq":749,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910937+0000","last_change":"2024-04-02T04:53:54.920631+0000","last_active":"2024-04-02T04:54:07.910937+0000","last_peered":"2024-04-02T04:54:07.910937+0000","last_clean":"2024-04-02T04:54:07.910937+0000","last_became_active":"2024-04-02T04:53:54.912078+0000","last_became_peered":"2024-04-02T04:53:54.912078+0000","last_unstale":"2024-04-02T04:54:07.910937+0000","last_undegraded":"2024-04-02T04:54:07.910937+0000","last_fullsized":"2024-04-02T04:54:07.910937+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-02T04:49:07.374876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.374876+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":715,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912420+0000","last_change":"2024-04-02T04:50:43.360677+0000","last_active":"2024-04-02T04:54:07.912420+0000","last_peered":"2024-04-02T04:54:07.912420+0000","last_clean":"2024-04-02T04:54:07.912420+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:54:07.912420+0000","last_undegraded":"2024-04-02T04:54:07.912420+0000","last_fullsized":"2024-04-02T04:54:07.912420+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:50:43.360457+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.360457+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":710,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901979+0000","last_change":"2024-04-02T04:54:05.619856+0000","last_active":"2024-04-02T04:54:06.901979+0000","last_peered":"2024-04-02T04:54:06.901979+0000","last_clean":"2024-04-02T04:54:06.901979+0000","last_became_active":"2024-04-02T04:53:54.917485+0000","last_became_peered":"2024-04-02T04:53:54.917485+0000","last_unstale":"2024-04-02T04:54:06.901979+0000","last_undegraded":"2024-04-02T04:54:06.901979+0000","last_fullsized":"2024-04-02T04:54:06.901979+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.619646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.619646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:09.566907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":690,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901614+0000","last_change":"2024-04-02T04:50:29.535635+0000","last_active":"2024-04-02T04:54:06.901614+0000","last_peered":"2024-04-02T04:54:06.901614+0000","last_clean":"2024-04-02T04:54:06.901614+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:54:06.901614+0000","last_undegraded":"2024-04-02T04:54:06.901614+0000","last_fullsized":"2024-04-02T04:54:06.901614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.535314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.535314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"139'10","reported_seq":575,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.917499+0000","last_change":"2024-04-02T04:52:55.613507+0000","last_active":"2024-04-02T04:54:06.917499+0000","last_peered":"2024-04-02T04:54:06.917499+0000","last_clean":"2024-04-02T04:54:06.917499+0000","last_became_active":"2024-04-02T04:49:26.341161+0000","last_became_peered":"2024-04-02T04:49:26.341161+0000","last_unstale":"2024-04-02T04:54:06.917499+0000","last_undegraded":"2024-04-02T04:54:06.917499+0000","last_fullsized":"2024-04-02T04:54:06.917499+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:52:55.613396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.613396+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:08.083376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928419+0000","last_change":"2024-04-02T04:50:29.806953+0000","last_active":"2024-04-02T04:54:06.928419+0000","last_peered":"2024-04-02T04:54:06.928419+0000","last_clean":"2024-04-02T04:54:06.928419+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:54:06.928419+0000","last_undegraded":"2024-04-02T04:54:06.928419+0000","last_fullsized":"2024-04-02T04:54:06.928419+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.806751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.806751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"395'14","reported_seq":838,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920820+0000","last_change":"2024-04-02T04:53:54.940918+0000","last_active":"2024-04-02T04:54:06.920820+0000","last_peered":"2024-04-02T04:54:06.920820+0000","last_clean":"2024-04-02T04:54:06.920820+0000","last_became_active":"2024-04-02T04:53:54.940043+0000","last_became_peered":"2024-04-02T04:53:54.940043+0000","last_unstale":"2024-04-02T04:54:06.920820+0000","last_undegraded":"2024-04-02T04:54:06.920820+0000","last_fullsized":"2024-04-02T04:54:06.920820+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"402'59","reported_seq":931,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910910+0000","last_change":"2024-04-02T04:52:51.249602+0000","last_active":"2024-04-02T04:54:07.910910+0000","last_peered":"2024-04-02T04:54:07.910910+0000","last_clean":"2024-04-02T04:54:07.910910+0000","last_became_active":"2024-04-02T04:43:56.241529+0000","last_became_peered":"2024-04-02T04:43:56.241529+0000","last_unstale":"2024-04-02T04:54:07.910910+0000","last_undegraded":"2024-04-02T04:54:07.910910+0000","last_fullsized":"2024-04-02T04:54:07.910910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:52:51.249489+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.249489+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:19.032401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.923357+0000","last_change":"2024-04-02T04:53:54.944193+0000","last_active":"2024-04-02T04:54:08.923357+0000","last_peered":"2024-04-02T04:54:08.923357+0000","last_clean":"2024-04-02T04:54:08.923357+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T04:54:08.923357+0000","last_undegraded":"2024-04-02T04:54:08.923357+0000","last_fullsized":"2024-04-02T04:54:08.923357+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:52.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:52.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921786+0000","last_change":"2024-04-02T04:51:34.006250+0000","last_active":"2024-04-02T04:54:06.921786+0000","last_peered":"2024-04-02T04:54:06.921786+0000","last_clean":"2024-04-02T04:54:06.921786+0000","last_became_active":"2024-04-02T04:43:58.284079+0000","last_became_peered":"2024-04-02T04:43:58.284079+0000","last_unstale":"2024-04-02T04:54:06.921786+0000","last_undegraded":"2024-04-02T04:54:06.921786+0000","last_fullsized":"2024-04-02T04:54:06.921786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:51:34.006018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.006018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":701,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907239+0000","last_change":"2024-04-02T04:53:53.356428+0000","last_active":"2024-04-02T04:54:07.907239+0000","last_peered":"2024-04-02T04:54:07.907239+0000","last_clean":"2024-04-02T04:54:07.907239+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:54:07.907239+0000","last_undegraded":"2024-04-02T04:54:07.907239+0000","last_fullsized":"2024-04-02T04:54:07.907239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:53.356300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:53:53.356300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:55.430795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914848+0000","last_change":"2024-04-02T04:53:54.958148+0000","last_active":"2024-04-02T04:54:06.914848+0000","last_peered":"2024-04-02T04:54:06.914848+0000","last_clean":"2024-04-02T04:54:06.914848+0000","last_became_active":"2024-04-02T04:53:54.957413+0000","last_became_peered":"2024-04-02T04:53:54.957413+0000","last_unstale":"2024-04-02T04:54:06.914848+0000","last_undegraded":"2024-04-02T04:54:06.914848+0000","last_fullsized":"2024-04-02T04:54:06.914848+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":707,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923513+0000","last_change":"2024-04-02T04:52:49.311146+0000","last_active":"2024-04-02T04:54:06.923513+0000","last_peered":"2024-04-02T04:54:06.923513+0000","last_clean":"2024-04-02T04:54:06.923513+0000","last_became_active":"2024-04-02T04:45:53.943072+0000","last_became_peered":"2024-04-02T04:45:53.943072+0000","last_unstale":"2024-04-02T04:54:06.923513+0000","last_undegraded":"2024-04-02T04:54:06.923513+0000","last_fullsized":"2024-04-02T04:54:06.923513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.310943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.310943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918537+0000","last_change":"2024-04-02T04:53:54.951695+0000","last_active":"2024-04-02T04:54:08.918537+0000","last_peered":"2024-04-02T04:54:08.918537+0000","last_clean":"2024-04-02T04:54:08.918537+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T04:54:08.918537+0000","last_undegraded":"2024-04-02T04:54:08.918537+0000","last_fullsized":"2024-04-02T04:54:08.918537+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:49.441320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:49.441320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"402'34","reported_seq":759,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915440+0000","last_change":"2024-04-02T04:52:33.272616+0000","last_active":"2024-04-02T04:54:06.915440+0000","last_peered":"2024-04-02T04:54:06.915440+0000","last_clean":"2024-04-02T04:54:06.915440+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:54:06.915440+0000","last_undegraded":"2024-04-02T04:54:06.915440+0000","last_fullsized":"2024-04-02T04:54:06.915440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:52:33.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.272448+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":701,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910699+0000","last_change":"2024-04-02T04:50:26.342752+0000","last_active":"2024-04-02T04:54:07.910699+0000","last_peered":"2024-04-02T04:54:07.910699+0000","last_clean":"2024-04-02T04:54:07.910699+0000","last_became_active":"2024-04-02T04:44:00.343974+0000","last_became_peered":"2024-04-02T04:44:00.343974+0000","last_unstale":"2024-04-02T04:54:07.910699+0000","last_undegraded":"2024-04-02T04:54:07.910699+0000","last_fullsized":"2024-04-02T04:54:07.910699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.342429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.342429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":565,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928592+0000","last_change":"2024-04-02T04:54:04.965297+0000","last_active":"2024-04-02T04:54:06.928592+0000","last_peered":"2024-04-02T04:54:06.928592+0000","last_clean":"2024-04-02T04:54:06.928592+0000","last_became_active":"2024-04-02T04:53:54.913281+0000","last_became_peered":"2024-04-02T04:53:54.913281+0000","last_unstale":"2024-04-02T04:54:06.928592+0000","last_undegraded":"2024-04-02T04:54:06.928592+0000","last_fullsized":"2024-04-02T04:54:06.928592+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.965190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.965190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:10.343551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923551+0000","last_change":"2024-04-02T04:50:24.860538+0000","last_active":"2024-04-02T04:54:06.923551+0000","last_peered":"2024-04-02T04:54:06.923551+0000","last_clean":"2024-04-02T04:54:06.923551+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:54:06.923551+0000","last_undegraded":"2024-04-02T04:54:06.923551+0000","last_fullsized":"2024-04-02T04:54:06.923551+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:24.860246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:24.860246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928904+0000","last_change":"2024-04-02T04:54:00.910076+0000","last_active":"2024-04-02T04:54:06.928904+0000","last_peered":"2024-04-02T04:54:06.928904+0000","last_clean":"2024-04-02T04:54:06.928904+0000","last_became_active":"2024-04-02T04:53:54.934952+0000","last_became_peered":"2024-04-02T04:53:54.934952+0000","last_unstale":"2024-04-02T04:54:06.928904+0000","last_undegraded":"2024-04-02T04:54:06.928904+0000","last_fullsized":"2024-04-02T04:54:06.928904+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.909025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.909025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:19.903732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":450,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918508+0000","last_change":"2024-04-02T04:53:54.947927+0000","last_active":"2024-04-02T04:54:08.918508+0000","last_peered":"2024-04-02T04:54:08.918508+0000","last_clean":"2024-04-02T04:54:08.918508+0000","last_became_active":"2024-04-02T04:53:54.946098+0000","last_became_peered":"2024-04-02T04:53:54.946098+0000","last_unstale":"2024-04-02T04:54:08.918508+0000","last_undegraded":"2024-04-02T04:54:08.918508+0000","last_fullsized":"2024-04-02T04:54:08.918508+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":710,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230705+0000","last_change":"2024-04-02T04:53:55.759211+0000","last_active":"2024-04-02T04:54:07.230705+0000","last_peered":"2024-04-02T04:54:07.230705+0000","last_clean":"2024-04-02T04:54:07.230705+0000","last_became_active":"2024-04-02T04:53:54.936894+0000","last_became_peered":"2024-04-02T04:53:54.936894+0000","last_unstale":"2024-04-02T04:54:07.230705+0000","last_undegraded":"2024-04-02T04:54:07.230705+0000","last_fullsized":"2024-04-02T04:54:07.230705+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:55.758993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.758993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:20.880849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"402'26","reported_seq":745,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906145+0000","last_change":"2024-04-02T04:51:49.756835+0000","last_active":"2024-04-02T04:54:07.906145+0000","last_peered":"2024-04-02T04:54:07.906145+0000","last_clean":"2024-04-02T04:54:07.906145+0000","last_became_active":"2024-04-02T04:43:56.243507+0000","last_became_peered":"2024-04-02T04:43:56.243507+0000","last_unstale":"2024-04-02T04:54:07.906145+0000","last_undegraded":"2024-04-02T04:54:07.906145+0000","last_fullsized":"2024-04-02T04:54:07.906145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:49.756586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.756586+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":697,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233586+0000","last_change":"2024-04-02T04:51:56.577347+0000","last_active":"2024-04-02T04:54:07.233586+0000","last_peered":"2024-04-02T04:54:07.233586+0000","last_clean":"2024-04-02T04:54:07.233586+0000","last_became_active":"2024-04-02T04:44:00.338606+0000","last_became_peered":"2024-04-02T04:44:00.338606+0000","last_unstale":"2024-04-02T04:54:07.233586+0000","last_undegraded":"2024-04-02T04:54:07.233586+0000","last_fullsized":"2024-04-02T04:54:07.233586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:56.577226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.577226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":562,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910595+0000","last_change":"2024-04-02T04:51:55.332978+0000","last_active":"2024-04-02T04:54:07.910595+0000","last_peered":"2024-04-02T04:54:07.910595+0000","last_clean":"2024-04-02T04:54:07.910595+0000","last_became_active":"2024-04-02T04:49:26.290196+0000","last_became_peered":"2024-04-02T04:49:26.290196+0000","last_unstale":"2024-04-02T04:54:07.910595+0000","last_undegraded":"2024-04-02T04:54:07.910595+0000","last_fullsized":"2024-04-02T04:54:07.910595+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.332906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.332906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":560,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230130+0000","last_change":"2024-04-02T04:53:54.917935+0000","last_active":"2024-04-02T04:54:07.230130+0000","last_peered":"2024-04-02T04:54:07.230130+0000","last_clean":"2024-04-02T04:54:07.230130+0000","last_became_active":"2024-04-02T04:53:54.917650+0000","last_became_peered":"2024-04-02T04:53:54.917650+0000","last_unstale":"2024-04-02T04:54:07.230130+0000","last_undegraded":"2024-04-02T04:54:07.230130+0000","last_fullsized":"2024-04-02T04:54:07.230130+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":705,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912582+0000","last_change":"2024-04-02T04:52:59.385229+0000","last_active":"2024-04-02T04:54:07.912582+0000","last_peered":"2024-04-02T04:54:07.912582+0000","last_clean":"2024-04-02T04:54:07.912582+0000","last_became_active":"2024-04-02T04:43:58.301318+0000","last_became_peered":"2024-04-02T04:43:58.301318+0000","last_unstale":"2024-04-02T04:54:07.912582+0000","last_undegraded":"2024-04-02T04:54:07.912582+0000","last_fullsized":"2024-04-02T04:54:07.912582+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.385111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.385111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:14.836761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":715,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.917860+0000","last_change":"2024-04-02T04:52:46.569670+0000","last_active":"2024-04-02T04:54:06.917860+0000","last_peered":"2024-04-02T04:54:06.917860+0000","last_clean":"2024-04-02T04:54:06.917860+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:54:06.917860+0000","last_undegraded":"2024-04-02T04:54:06.917860+0000","last_fullsized":"2024-04-02T04:54:06.917860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.569560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.569560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"402'42","reported_seq":784,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907192+0000","last_change":"2024-04-02T04:52:49.180778+0000","last_active":"2024-04-02T04:54:07.907192+0000","last_peered":"2024-04-02T04:54:07.907192+0000","last_clean":"2024-04-02T04:54:07.907192+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:54:07.907192+0000","last_undegraded":"2024-04-02T04:54:07.907192+0000","last_fullsized":"2024-04-02T04:54:07.907192+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:52:49.180521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.180521+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":156,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.923484+0000","last_change":"2024-04-02T04:53:56.024607+0000","last_active":"2024-04-02T04:54:08.923484+0000","last_peered":"2024-04-02T04:54:08.923484+0000","last_clean":"2024-04-02T04:54:08.923484+0000","last_became_active":"2024-04-02T04:53:54.936793+0000","last_became_peered":"2024-04-02T04:53:54.936793+0000","last_unstale":"2024-04-02T04:54:08.923484+0000","last_undegraded":"2024-04-02T04:54:08.923484+0000","last_fullsized":"2024-04-02T04:54:08.923484+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:56.024450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:56.024450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:00.558897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":52,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920916+0000","last_change":"2024-04-02T04:53:54.927567+0000","last_active":"2024-04-02T04:54:08.920916+0000","last_peered":"2024-04-02T04:54:08.920916+0000","last_clean":"2024-04-02T04:54:08.920916+0000","last_became_active":"2024-04-02T04:53:54.926763+0000","last_became_peered":"2024-04-02T04:53:54.926763+0000","last_unstale":"2024-04-02T04:54:08.920916+0000","last_undegraded":"2024-04-02T04:54:08.920916+0000","last_fullsized":"2024-04-02T04:54:08.920916+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920622+0000","last_change":"2024-04-02T04:50:22.825218+0000","last_active":"2024-04-02T04:54:06.920622+0000","last_peered":"2024-04-02T04:54:06.920622+0000","last_clean":"2024-04-02T04:54:06.920622+0000","last_became_active":"2024-04-02T04:49:26.229137+0000","last_became_peered":"2024-04-02T04:49:26.229137+0000","last_unstale":"2024-04-02T04:54:06.920622+0000","last_undegraded":"2024-04-02T04:54:06.920622+0000","last_fullsized":"2024-04-02T04:54:06.920622+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:22.825000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:22.825000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":705,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910764+0000","last_change":"2024-04-02T04:51:34.344914+0000","last_active":"2024-04-02T04:54:07.910764+0000","last_peered":"2024-04-02T04:54:07.910764+0000","last_clean":"2024-04-02T04:54:07.910764+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:54:07.910764+0000","last_undegraded":"2024-04-02T04:54:07.910764+0000","last_fullsized":"2024-04-02T04:54:07.910764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:34.344554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.344554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'35","reported_seq":761,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.905108+0000","last_change":"2024-04-02T04:50:12.546833+0000","last_active":"2024-04-02T04:54:06.905108+0000","last_peered":"2024-04-02T04:54:06.905108+0000","last_clean":"2024-04-02T04:54:06.905108+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:54:06.905108+0000","last_undegraded":"2024-04-02T04:54:06.905108+0000","last_fullsized":"2024-04-02T04:54:06.905108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:50:12.546683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.546683+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":717,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230153+0000","last_change":"2024-04-02T04:53:13.578552+0000","last_active":"2024-04-02T04:54:07.230153+0000","last_peered":"2024-04-02T04:54:07.230153+0000","last_clean":"2024-04-02T04:54:07.230153+0000","last_became_active":"2024-04-02T04:43:54.226513+0000","last_became_peered":"2024-04-02T04:43:54.226513+0000","last_unstale":"2024-04-02T04:54:07.230153+0000","last_undegraded":"2024-04-02T04:54:07.230153+0000","last_fullsized":"2024-04-02T04:54:07.230153+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.578411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.578411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:42.763292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":706,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908691+0000","last_change":"2024-04-02T04:53:58.149905+0000","last_active":"2024-04-02T04:54:07.908691+0000","last_peered":"2024-04-02T04:54:07.908691+0000","last_clean":"2024-04-02T04:54:07.908691+0000","last_became_active":"2024-04-02T04:53:54.934877+0000","last_became_peered":"2024-04-02T04:53:54.934877+0000","last_unstale":"2024-04-02T04:54:07.908691+0000","last_undegraded":"2024-04-02T04:54:07.908691+0000","last_fullsized":"2024-04-02T04:54:07.908691+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:53:58.149611+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:53:58.149611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:00.756861+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"139'5","reported_seq":565,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.906099+0000","last_change":"2024-04-02T04:53:54.917270+0000","last_active":"2024-04-02T04:54:06.906099+0000","last_peered":"2024-04-02T04:54:06.906099+0000","last_clean":"2024-04-02T04:54:06.906099+0000","last_became_active":"2024-04-02T04:53:54.916847+0000","last_became_peered":"2024-04-02T04:53:54.916847+0000","last_unstale":"2024-04-02T04:54:06.906099+0000","last_undegraded":"2024-04-02T04:54:06.906099+0000","last_fullsized":"2024-04-02T04:54:06.906099+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"138'2","reported_seq":561,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230575+0000","last_change":"2024-04-02T04:51:33.690122+0000","last_active":"2024-04-02T04:54:07.230575+0000","last_peered":"2024-04-02T04:54:07.230575+0000","last_clean":"2024-04-02T04:54:07.230575+0000","last_became_active":"2024-04-02T04:49:26.270708+0000","last_became_peered":"2024-04-02T04:49:26.270708+0000","last_unstale":"2024-04-02T04:54:07.230575+0000","last_undegraded":"2024-04-02T04:54:07.230575+0000","last_fullsized":"2024-04-02T04:54:07.230575+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:51:33.689760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:33.689760+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":52,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920322+0000","last_change":"2024-04-02T04:53:54.943618+0000","last_active":"2024-04-02T04:54:08.920322+0000","last_peered":"2024-04-02T04:54:08.920322+0000","last_clean":"2024-04-02T04:54:08.920322+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T04:54:08.920322+0000","last_undegraded":"2024-04-02T04:54:08.920322+0000","last_fullsized":"2024-04-02T04:54:08.920322+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":705,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907590+0000","last_change":"2024-04-02T04:52:41.308406+0000","last_active":"2024-04-02T04:54:07.907590+0000","last_peered":"2024-04-02T04:54:07.907590+0000","last_clean":"2024-04-02T04:54:07.907590+0000","last_became_active":"2024-04-02T04:43:58.294856+0000","last_became_peered":"2024-04-02T04:43:58.294856+0000","last_unstale":"2024-04-02T04:54:07.907590+0000","last_undegraded":"2024-04-02T04:54:07.907590+0000","last_fullsized":"2024-04-02T04:54:07.907590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.308301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.308301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"402'28","reported_seq":868,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.231227+0000","last_change":"2024-04-02T04:51:35.726385+0000","last_active":"2024-04-02T04:54:07.231227+0000","last_peered":"2024-04-02T04:54:07.231227+0000","last_clean":"2024-04-02T04:54:07.231227+0000","last_became_active":"2024-04-02T04:43:56.238393+0000","last_became_peered":"2024-04-02T04:43:56.238393+0000","last_unstale":"2024-04-02T04:54:07.231227+0000","last_undegraded":"2024-04-02T04:54:07.231227+0000","last_fullsized":"2024-04-02T04:54:07.231227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:35.725768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.725768+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":730,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910805+0000","last_change":"2024-04-02T04:53:59.321337+0000","last_active":"2024-04-02T04:54:07.910805+0000","last_peered":"2024-04-02T04:54:07.910805+0000","last_clean":"2024-04-02T04:54:07.910805+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T04:54:07.910805+0000","last_undegraded":"2024-04-02T04:54:07.910805+0000","last_fullsized":"2024-04-02T04:54:07.910805+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:53:59.321220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.321220+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.462072+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":706,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914437+0000","last_change":"2024-04-02T04:53:11.307355+0000","last_active":"2024-04-02T04:54:06.914437+0000","last_peered":"2024-04-02T04:54:06.914437+0000","last_clean":"2024-04-02T04:54:06.914437+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:54:06.914437+0000","last_undegraded":"2024-04-02T04:54:06.914437+0000","last_fullsized":"2024-04-02T04:54:06.914437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.307248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.307248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:40.654703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"139'5","reported_seq":59,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918208+0000","last_change":"2024-04-02T04:53:54.932181+0000","last_active":"2024-04-02T04:54:08.918208+0000","last_peered":"2024-04-02T04:54:08.918208+0000","last_clean":"2024-04-02T04:54:08.918208+0000","last_became_active":"2024-04-02T04:53:54.931833+0000","last_became_peered":"2024-04-02T04:53:54.931833+0000","last_unstale":"2024-04-02T04:54:08.918208+0000","last_undegraded":"2024-04-02T04:54:08.918208+0000","last_fullsized":"2024-04-02T04:54:08.918208+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928841+0000","last_change":"2024-04-02T04:52:57.364583+0000","last_active":"2024-04-02T04:54:06.928841+0000","last_peered":"2024-04-02T04:54:06.928841+0000","last_clean":"2024-04-02T04:54:06.928841+0000","last_became_active":"2024-04-02T04:49:26.198942+0000","last_became_peered":"2024-04-02T04:49:26.198942+0000","last_unstale":"2024-04-02T04:54:06.928841+0000","last_undegraded":"2024-04-02T04:54:06.928841+0000","last_fullsized":"2024-04-02T04:54:06.928841+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.364241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.364241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:25.621398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":836,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.902174+0000","last_change":"2024-04-02T04:53:04.380937+0000","last_active":"2024-04-02T04:54:06.902174+0000","last_peered":"2024-04-02T04:54:06.902174+0000","last_clean":"2024-04-02T04:54:06.902174+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:54:06.902174+0000","last_undegraded":"2024-04-02T04:54:06.902174+0000","last_fullsized":"2024-04-02T04:54:06.902174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:53:04.380713+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.380713+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:27.532844+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"402'25","reported_seq":232,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920391+0000","last_change":"2024-04-02T04:53:54.929360+0000","last_active":"2024-04-02T04:54:08.920391+0000","last_peered":"2024-04-02T04:54:08.920391+0000","last_clean":"2024-04-02T04:54:08.920391+0000","last_became_active":"2024-04-02T04:53:54.927132+0000","last_became_peered":"2024-04-02T04:53:54.927132+0000","last_unstale":"2024-04-02T04:54:08.920391+0000","last_undegraded":"2024-04-02T04:54:08.920391+0000","last_fullsized":"2024-04-02T04:54:08.920391+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":735,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.910484+0000","last_change":"2024-04-02T04:53:59.519788+0000","last_active":"2024-04-02T04:54:06.910484+0000","last_peered":"2024-04-02T04:54:06.910484+0000","last_clean":"2024-04-02T04:54:06.910484+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T04:54:06.910484+0000","last_undegraded":"2024-04-02T04:54:06.910484+0000","last_fullsized":"2024-04-02T04:54:06.910484+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:53:59.519530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.519530+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:07.349244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":704,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909840+0000","last_change":"2024-04-02T04:54:01.088340+0000","last_active":"2024-04-02T04:54:07.909840+0000","last_peered":"2024-04-02T04:54:07.909840+0000","last_clean":"2024-04-02T04:54:07.909840+0000","last_became_active":"2024-04-02T04:53:54.944662+0000","last_became_peered":"2024-04-02T04:53:54.944662+0000","last_unstale":"2024-04-02T04:54:07.909840+0000","last_undegraded":"2024-04-02T04:54:07.909840+0000","last_fullsized":"2024-04-02T04:54:07.909840+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.088131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.088131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:01.594214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":565,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915334+0000","last_change":"2024-04-02T04:53:09.820846+0000","last_active":"2024-04-02T04:54:06.915334+0000","last_peered":"2024-04-02T04:54:06.915334+0000","last_clean":"2024-04-02T04:54:06.915334+0000","last_became_active":"2024-04-02T04:49:26.292029+0000","last_became_peered":"2024-04-02T04:49:26.292029+0000","last_unstale":"2024-04-02T04:54:06.915334+0000","last_undegraded":"2024-04-02T04:54:06.915334+0000","last_fullsized":"2024-04-02T04:54:06.915334+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:09.820697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:09.820697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:20.156473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"402'25","reported_seq":222,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920876+0000","last_change":"2024-04-02T04:53:54.923643+0000","last_active":"2024-04-02T04:54:08.920876+0000","last_peered":"2024-04-02T04:54:08.920876+0000","last_clean":"2024-04-02T04:54:08.920876+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T04:54:08.920876+0000","last_undegraded":"2024-04-02T04:54:08.920876+0000","last_fullsized":"2024-04-02T04:54:08.920876+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":715,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.915426+0000","last_change":"2024-04-02T04:52:50.652084+0000","last_active":"2024-04-02T04:54:06.915426+0000","last_peered":"2024-04-02T04:54:06.915426+0000","last_clean":"2024-04-02T04:54:06.915426+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:54:06.915426+0000","last_undegraded":"2024-04-02T04:54:06.915426+0000","last_fullsized":"2024-04-02T04:54:06.915426+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:50.651865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:52:50.651865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":694,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.924442+0000","last_change":"2024-04-02T04:53:54.956960+0000","last_active":"2024-04-02T04:54:06.924442+0000","last_peered":"2024-04-02T04:54:06.924442+0000","last_clean":"2024-04-02T04:54:06.924442+0000","last_became_active":"2024-04-02T04:53:54.956221+0000","last_became_peered":"2024-04-02T04:53:54.956221+0000","last_unstale":"2024-04-02T04:54:06.924442+0000","last_undegraded":"2024-04-02T04:54:06.924442+0000","last_fullsized":"2024-04-02T04:54:06.924442+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:00.103147+0000","last_clean_scrub_stamp":"2024-04-02T04:49:00.103147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":696,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920226+0000","last_change":"2024-04-02T04:54:01.922789+0000","last_active":"2024-04-02T04:54:06.920226+0000","last_peered":"2024-04-02T04:54:06.920226+0000","last_clean":"2024-04-02T04:54:06.920226+0000","last_became_active":"2024-04-02T04:53:54.925132+0000","last_became_peered":"2024-04-02T04:53:54.925132+0000","last_unstale":"2024-04-02T04:54:06.920226+0000","last_undegraded":"2024-04-02T04:54:06.920226+0000","last_fullsized":"2024-04-02T04:54:06.920226+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.922344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.922344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:15.507416+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1f","version":"139'5","reported_seq":561,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.928290+0000","last_change":"2024-04-02T04:53:01.369857+0000","last_active":"2024-04-02T04:54:06.928290+0000","last_peered":"2024-04-02T04:54:06.928290+0000","last_clean":"2024-04-02T04:54:06.928290+0000","last_became_active":"2024-04-02T04:49:26.178040+0000","last_became_peered":"2024-04-02T04:49:26.178040+0000","last_unstale":"2024-04-02T04:54:06.928290+0000","last_undegraded":"2024-04-02T04:54:06.928290+0000","last_fullsized":"2024-04-02T04:54:06.928290+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:53:01.369737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:01.369737+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:30.638792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":562,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906832+0000","last_change":"2024-04-02T04:51:29.319201+0000","last_active":"2024-04-02T04:54:07.906832+0000","last_peered":"2024-04-02T04:54:07.906832+0000","last_clean":"2024-04-02T04:54:07.906832+0000","last_became_active":"2024-04-02T04:49:26.258674+0000","last_became_peered":"2024-04-02T04:49:26.258674+0000","last_unstale":"2024-04-02T04:54:07.906832+0000","last_undegraded":"2024-04-02T04:54:07.906832+0000","last_fullsized":"2024-04-02T04:54:07.906832+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.318999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.318999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"402'8","reported_seq":708,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.916480+0000","last_change":"2024-04-02T04:50:17.746744+0000","last_active":"2024-04-02T04:54:06.916480+0000","last_peered":"2024-04-02T04:54:06.916480+0000","last_clean":"2024-04-02T04:54:06.916480+0000","last_became_active":"2024-04-02T04:43:56.249900+0000","last_became_peered":"2024-04-02T04:43:56.249900+0000","last_unstale":"2024-04-02T04:54:06.916480+0000","last_undegraded":"2024-04-02T04:54:06.916480+0000","last_fullsized":"2024-04-02T04:54:06.916480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-04-02T04:50:17.746578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:17.746578+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":708,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920435+0000","last_change":"2024-04-02T04:51:40.072460+0000","last_active":"2024-04-02T04:54:06.920435+0000","last_peered":"2024-04-02T04:54:06.920435+0000","last_clean":"2024-04-02T04:54:06.920435+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:54:06.920435+0000","last_undegraded":"2024-04-02T04:54:06.920435+0000","last_fullsized":"2024-04-02T04:54:06.920435+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.072262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.072262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":706,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910253+0000","last_change":"2024-04-02T04:53:54.921243+0000","last_active":"2024-04-02T04:54:07.910253+0000","last_peered":"2024-04-02T04:54:07.910253+0000","last_clean":"2024-04-02T04:54:07.910253+0000","last_became_active":"2024-04-02T04:53:54.920683+0000","last_became_peered":"2024-04-02T04:53:54.920683+0000","last_unstale":"2024-04-02T04:54:07.910253+0000","last_undegraded":"2024-04-02T04:54:07.910253+0000","last_fullsized":"2024-04-02T04:54:07.910253+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:02.485165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:49:02.485165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":179,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.923217+0000","last_change":"2024-04-02T04:54:03.204796+0000","last_active":"2024-04-02T04:54:08.923217+0000","last_peered":"2024-04-02T04:54:08.923217+0000","last_clean":"2024-04-02T04:54:08.923217+0000","last_became_active":"2024-04-02T04:53:54.933037+0000","last_became_peered":"2024-04-02T04:53:54.933037+0000","last_unstale":"2024-04-02T04:54:08.923217+0000","last_undegraded":"2024-04-02T04:54:08.923217+0000","last_fullsized":"2024-04-02T04:54:08.923217+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.204677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.204677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.899180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.912156+0000","last_change":"2024-04-02T04:51:28.314780+0000","last_active":"2024-04-02T04:54:06.912156+0000","last_peered":"2024-04-02T04:54:06.912156+0000","last_clean":"2024-04-02T04:54:06.912156+0000","last_became_active":"2024-04-02T04:49:26.250745+0000","last_became_peered":"2024-04-02T04:49:26.250745+0000","last_unstale":"2024-04-02T04:54:06.912156+0000","last_undegraded":"2024-04-02T04:54:06.912156+0000","last_fullsized":"2024-04-02T04:54:06.912156+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:28.314678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:28.314678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919698+0000","last_change":"2024-04-02T04:51:39.049216+0000","last_active":"2024-04-02T04:54:06.919698+0000","last_peered":"2024-04-02T04:54:06.919698+0000","last_clean":"2024-04-02T04:54:06.919698+0000","last_became_active":"2024-04-02T04:49:26.229293+0000","last_became_peered":"2024-04-02T04:49:26.229293+0000","last_unstale":"2024-04-02T04:54:06.919698+0000","last_undegraded":"2024-04-02T04:54:06.919698+0000","last_fullsized":"2024-04-02T04:54:06.919698+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.048988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"402'19","reported_seq":195,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.919509+0000","last_change":"2024-04-02T04:53:57.032666+0000","last_active":"2024-04-02T04:54:08.919509+0000","last_peered":"2024-04-02T04:54:08.919509+0000","last_clean":"2024-04-02T04:54:08.919509+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T04:54:08.919509+0000","last_undegraded":"2024-04-02T04:54:08.919509+0000","last_fullsized":"2024-04-02T04:54:08.919509+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:53:57.032539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:53:57.032539+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:00.442091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":710,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.229973+0000","last_change":"2024-04-02T04:54:02.769937+0000","last_active":"2024-04-02T04:54:07.229973+0000","last_peered":"2024-04-02T04:54:07.229973+0000","last_clean":"2024-04-02T04:54:07.229973+0000","last_became_active":"2024-04-02T04:53:54.898211+0000","last_became_peered":"2024-04-02T04:53:54.898211+0000","last_unstale":"2024-04-02T04:54:07.229973+0000","last_undegraded":"2024-04-02T04:54:07.229973+0000","last_fullsized":"2024-04-02T04:54:07.229973+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:02.769714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:02.769714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:32.056100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"395'14","reported_seq":844,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909084+0000","last_change":"2024-04-02T04:53:55.155728+0000","last_active":"2024-04-02T04:54:07.909084+0000","last_peered":"2024-04-02T04:54:07.909084+0000","last_clean":"2024-04-02T04:54:07.909084+0000","last_became_active":"2024-04-02T04:53:54.924949+0000","last_became_peered":"2024-04-02T04:53:54.924949+0000","last_unstale":"2024-04-02T04:54:07.909084+0000","last_undegraded":"2024-04-02T04:54:07.909084+0000","last_fullsized":"2024-04-02T04:54:07.909084+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'14","last_scrub_stamp":"2024-04-02T04:53:55.155578+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.155578+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.148406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":690,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923608+0000","last_change":"2024-04-02T04:53:54.957199+0000","last_active":"2024-04-02T04:54:06.923608+0000","last_peered":"2024-04-02T04:54:06.923608+0000","last_clean":"2024-04-02T04:54:06.923608+0000","last_became_active":"2024-04-02T04:53:54.956634+0000","last_became_peered":"2024-04-02T04:53:54.956634+0000","last_unstale":"2024-04-02T04:54:06.923608+0000","last_undegraded":"2024-04-02T04:54:06.923608+0000","last_fullsized":"2024-04-02T04:54:06.923608+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:03.986307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:01.454803+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.986307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"139'7","reported_seq":61,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.923793+0000","last_change":"2024-04-02T04:53:54.952789+0000","last_active":"2024-04-02T04:54:08.923793+0000","last_peered":"2024-04-02T04:54:08.923793+0000","last_clean":"2024-04-02T04:54:08.923793+0000","last_became_active":"2024-04-02T04:53:54.952321+0000","last_became_peered":"2024-04-02T04:53:54.952321+0000","last_unstale":"2024-04-02T04:54:08.923793+0000","last_undegraded":"2024-04-02T04:54:08.923793+0000","last_fullsized":"2024-04-02T04:54:08.923793+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":564,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909058+0000","last_change":"2024-04-02T04:53:54.945265+0000","last_active":"2024-04-02T04:54:07.909058+0000","last_peered":"2024-04-02T04:54:07.909058+0000","last_clean":"2024-04-02T04:54:07.909058+0000","last_became_active":"2024-04-02T04:53:54.944150+0000","last_became_peered":"2024-04-02T04:53:54.944150+0000","last_unstale":"2024-04-02T04:54:07.909058+0000","last_undegraded":"2024-04-02T04:54:07.909058+0000","last_fullsized":"2024-04-02T04:54:07.909058+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"402'15","reported_seq":731,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913820+0000","last_change":"2024-04-02T04:53:54.938967+0000","last_active":"2024-04-02T04:54:06.913820+0000","last_peered":"2024-04-02T04:54:06.913820+0000","last_clean":"2024-04-02T04:54:06.913820+0000","last_became_active":"2024-04-02T04:53:54.938547+0000","last_became_peered":"2024-04-02T04:53:54.938547+0000","last_unstale":"2024-04-02T04:54:06.913820+0000","last_undegraded":"2024-04-02T04:54:06.913820+0000","last_fullsized":"2024-04-02T04:54:06.913820+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:49:14.989379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.989379+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920545+0000","last_change":"2024-04-02T04:53:05.331108+0000","last_active":"2024-04-02T04:54:06.920545+0000","last_peered":"2024-04-02T04:54:06.920545+0000","last_clean":"2024-04-02T04:54:06.920545+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:54:06.920545+0000","last_undegraded":"2024-04-02T04:54:06.920545+0000","last_fullsized":"2024-04-02T04:54:06.920545+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.331020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.331020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:21.500198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"395'13","reported_seq":840,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.630233+0000","last_change":"2024-04-02T04:54:07.630306+0000","last_active":"2024-04-02T04:54:07.630233+0000","last_peered":"2024-04-02T04:54:07.630233+0000","last_clean":"2024-04-02T04:54:07.630233+0000","last_became_active":"2024-04-02T04:53:54.936626+0000","last_became_peered":"2024-04-02T04:53:54.936626+0000","last_unstale":"2024-04-02T04:54:07.630233+0000","last_undegraded":"2024-04-02T04:54:07.630233+0000","last_fullsized":"2024-04-02T04:54:07.630233+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'13","last_scrub_stamp":"2024-04-02T04:54:07.630056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.630056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.746887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":690,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.912239+0000","last_change":"2024-04-02T04:50:42.526150+0000","last_active":"2024-04-02T04:54:06.912239+0000","last_peered":"2024-04-02T04:54:06.912239+0000","last_clean":"2024-04-02T04:54:06.912239+0000","last_became_active":"2024-04-02T04:44:00.337013+0000","last_became_peered":"2024-04-02T04:44:00.337013+0000","last_unstale":"2024-04-02T04:54:06.912239+0000","last_undegraded":"2024-04-02T04:54:06.912239+0000","last_fullsized":"2024-04-02T04:54:06.912239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:42.525995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:42.525995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.925792+0000","last_change":"2024-04-02T04:52:43.424718+0000","last_active":"2024-04-02T04:54:06.925792+0000","last_peered":"2024-04-02T04:54:06.925792+0000","last_clean":"2024-04-02T04:54:06.925792+0000","last_became_active":"2024-04-02T04:49:26.280984+0000","last_became_peered":"2024-04-02T04:49:26.280984+0000","last_unstale":"2024-04-02T04:54:06.925792+0000","last_undegraded":"2024-04-02T04:54:06.925792+0000","last_fullsized":"2024-04-02T04:54:06.925792+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:43.424601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.424601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":555,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926102+0000","last_change":"2024-04-02T04:50:25.822947+0000","last_active":"2024-04-02T04:54:06.926102+0000","last_peered":"2024-04-02T04:54:06.926102+0000","last_clean":"2024-04-02T04:54:06.926102+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:54:06.926102+0000","last_undegraded":"2024-04-02T04:54:06.926102+0000","last_fullsized":"2024-04-02T04:54:06.926102+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:25.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"402'21","reported_seq":232,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.924165+0000","last_change":"2024-04-02T04:53:54.940529+0000","last_active":"2024-04-02T04:54:08.924165+0000","last_peered":"2024-04-02T04:54:08.924165+0000","last_clean":"2024-04-02T04:54:08.924165+0000","last_became_active":"2024-04-02T04:53:54.936610+0000","last_became_peered":"2024-04-02T04:53:54.936610+0000","last_unstale":"2024-04-02T04:54:08.924165+0000","last_undegraded":"2024-04-02T04:54:08.924165+0000","last_fullsized":"2024-04-02T04:54:08.924165+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:49:23.491068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:23.491068+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":700,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926370+0000","last_change":"2024-04-02T04:50:21.847768+0000","last_active":"2024-04-02T04:54:06.926370+0000","last_peered":"2024-04-02T04:54:06.926370+0000","last_clean":"2024-04-02T04:54:06.926370+0000","last_became_active":"2024-04-02T04:43:54.212632+0000","last_became_peered":"2024-04-02T04:43:54.212632+0000","last_unstale":"2024-04-02T04:54:06.926370+0000","last_undegraded":"2024-04-02T04:54:06.926370+0000","last_fullsized":"2024-04-02T04:54:06.926370+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.847551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.847551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":705,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906138+0000","last_change":"2024-04-02T04:52:54.293779+0000","last_active":"2024-04-02T04:54:07.906138+0000","last_peered":"2024-04-02T04:54:07.906138+0000","last_clean":"2024-04-02T04:54:07.906138+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:54:07.906138+0000","last_undegraded":"2024-04-02T04:54:07.906138+0000","last_fullsized":"2024-04-02T04:54:07.906138+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:54.293646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:54.293646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":697,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914343+0000","last_change":"2024-04-02T04:51:53.965925+0000","last_active":"2024-04-02T04:54:06.914343+0000","last_peered":"2024-04-02T04:54:06.914343+0000","last_clean":"2024-04-02T04:54:06.914343+0000","last_became_active":"2024-04-02T04:44:00.330066+0000","last_became_peered":"2024-04-02T04:44:00.330066+0000","last_unstale":"2024-04-02T04:54:06.914343+0000","last_undegraded":"2024-04-02T04:54:06.914343+0000","last_fullsized":"2024-04-02T04:54:06.914343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.965811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.965811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"139'10","reported_seq":568,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.902892+0000","last_change":"2024-04-02T04:51:30.300205+0000","last_active":"2024-04-02T04:54:06.902892+0000","last_peered":"2024-04-02T04:54:06.902892+0000","last_clean":"2024-04-02T04:54:06.902892+0000","last_became_active":"2024-04-02T04:49:26.260148+0000","last_became_peered":"2024-04-02T04:49:26.260148+0000","last_unstale":"2024-04-02T04:54:06.902892+0000","last_undegraded":"2024-04-02T04:54:06.902892+0000","last_fullsized":"2024-04-02T04:54:06.902892+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:51:30.299917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.299917+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":558,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.906242+0000","last_change":"2024-04-02T04:51:26.346563+0000","last_active":"2024-04-02T04:54:06.906242+0000","last_peered":"2024-04-02T04:54:06.906242+0000","last_clean":"2024-04-02T04:54:06.906242+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:54:06.906242+0000","last_undegraded":"2024-04-02T04:54:06.906242+0000","last_fullsized":"2024-04-02T04:54:06.906242+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.346350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.346350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"402'32","reported_seq":761,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.905978+0000","last_change":"2024-04-02T04:53:58.561674+0000","last_active":"2024-04-02T04:54:06.905978+0000","last_peered":"2024-04-02T04:54:06.905978+0000","last_clean":"2024-04-02T04:54:06.905978+0000","last_became_active":"2024-04-02T04:53:54.918208+0000","last_became_peered":"2024-04-02T04:53:54.918208+0000","last_unstale":"2024-04-02T04:54:06.905978+0000","last_undegraded":"2024-04-02T04:54:06.905978+0000","last_fullsized":"2024-04-02T04:54:06.905978+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:53:58.561437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:53:58.561437+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:07.047209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":713,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.911214+0000","last_change":"2024-04-02T04:53:05.438467+0000","last_active":"2024-04-02T04:54:07.911214+0000","last_peered":"2024-04-02T04:54:07.911214+0000","last_clean":"2024-04-02T04:54:07.911214+0000","last_became_active":"2024-04-02T04:43:54.221902+0000","last_became_peered":"2024-04-02T04:43:54.221902+0000","last_unstale":"2024-04-02T04:54:07.911214+0000","last_undegraded":"2024-04-02T04:54:07.911214+0000","last_fullsized":"2024-04-02T04:54:07.911214+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.438193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.438193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:25.272805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":702,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.904329+0000","last_change":"2024-04-02T04:52:40.351590+0000","last_active":"2024-04-02T04:54:06.904329+0000","last_peered":"2024-04-02T04:54:06.904329+0000","last_clean":"2024-04-02T04:54:06.904329+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:54:06.904329+0000","last_undegraded":"2024-04-02T04:54:06.904329+0000","last_fullsized":"2024-04-02T04:54:06.904329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.351381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.351381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":695,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904208+0000","last_change":"2024-04-02T04:50:32.383573+0000","last_active":"2024-04-02T04:54:07.904208+0000","last_peered":"2024-04-02T04:54:07.904208+0000","last_clean":"2024-04-02T04:54:07.904208+0000","last_became_active":"2024-04-02T04:44:00.332845+0000","last_became_peered":"2024-04-02T04:44:00.332845+0000","last_unstale":"2024-04-02T04:54:07.904208+0000","last_undegraded":"2024-04-02T04:54:07.904208+0000","last_fullsized":"2024-04-02T04:54:07.904208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:50:32.383421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"262'6","reported_seq":567,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907525+0000","last_change":"2024-04-02T04:51:42.313312+0000","last_active":"2024-04-02T04:54:07.907525+0000","last_peered":"2024-04-02T04:54:07.907525+0000","last_clean":"2024-04-02T04:54:07.907525+0000","last_became_active":"2024-04-02T04:49:26.275418+0000","last_became_peered":"2024-04-02T04:49:26.275418+0000","last_unstale":"2024-04-02T04:54:07.907525+0000","last_undegraded":"2024-04-02T04:54:07.907525+0000","last_fullsized":"2024-04-02T04:54:07.907525+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'6","last_scrub_stamp":"2024-04-02T04:51:42.313232+0000","last_deep_scrub":"262'6","last_deep_scrub_stamp":"2024-04-02T04:51:42.313232+0000","last_clean_scrub_stamp":"2024-04-02T04:51:42.313232+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":566,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923146+0000","last_change":"2024-04-02T04:53:00.389103+0000","last_active":"2024-04-02T04:54:06.923146+0000","last_peered":"2024-04-02T04:54:06.923146+0000","last_clean":"2024-04-02T04:54:06.923146+0000","last_became_active":"2024-04-02T04:49:25.080486+0000","last_became_peered":"2024-04-02T04:49:25.080486+0000","last_unstale":"2024-04-02T04:54:06.923146+0000","last_undegraded":"2024-04-02T04:54:06.923146+0000","last_fullsized":"2024-04-02T04:54:06.923146+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.388684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.388684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:22.291684+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"402'18","reported_seq":727,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.925894+0000","last_change":"2024-04-02T04:50:25.999986+0000","last_active":"2024-04-02T04:54:06.925894+0000","last_peered":"2024-04-02T04:54:06.925894+0000","last_clean":"2024-04-02T04:54:06.925894+0000","last_became_active":"2024-04-02T04:43:56.244885+0000","last_became_peered":"2024-04-02T04:43:56.244885+0000","last_unstale":"2024-04-02T04:54:06.925894+0000","last_undegraded":"2024-04-02T04:54:06.925894+0000","last_fullsized":"2024-04-02T04:54:06.925894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:50:25.999894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.999894+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":717,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233496+0000","last_change":"2024-04-02T04:53:12.625161+0000","last_active":"2024-04-02T04:54:07.233496+0000","last_peered":"2024-04-02T04:54:07.233496+0000","last_clean":"2024-04-02T04:54:07.233496+0000","last_became_active":"2024-04-02T04:43:54.224908+0000","last_became_peered":"2024-04-02T04:43:54.224908+0000","last_unstale":"2024-04-02T04:54:07.233496+0000","last_undegraded":"2024-04-02T04:54:07.233496+0000","last_fullsized":"2024-04-02T04:54:07.233496+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:12.624937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:53:12.624937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:13.488562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":707,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904595+0000","last_change":"2024-04-02T04:51:54.722249+0000","last_active":"2024-04-02T04:54:07.904595+0000","last_peered":"2024-04-02T04:54:07.904595+0000","last_clean":"2024-04-02T04:54:07.904595+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:54:07.904595+0000","last_undegraded":"2024-04-02T04:54:07.904595+0000","last_fullsized":"2024-04-02T04:54:07.904595+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:54.722054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:54.722054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":173,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.918698+0000","last_change":"2024-04-02T04:53:54.908948+0000","last_active":"2024-04-02T04:54:08.918698+0000","last_peered":"2024-04-02T04:54:08.918698+0000","last_clean":"2024-04-02T04:54:08.918698+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T04:54:08.918698+0000","last_undegraded":"2024-04-02T04:54:08.918698+0000","last_fullsized":"2024-04-02T04:54:08.918698+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:10.372299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:49:10.372299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":573,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909662+0000","last_change":"2024-04-02T04:54:06.134370+0000","last_active":"2024-04-02T04:54:07.909662+0000","last_peered":"2024-04-02T04:54:07.909662+0000","last_clean":"2024-04-02T04:54:07.909662+0000","last_became_active":"2024-04-02T04:53:54.915320+0000","last_became_peered":"2024-04-02T04:53:54.915320+0000","last_unstale":"2024-04-02T04:54:07.909662+0000","last_undegraded":"2024-04-02T04:54:07.909662+0000","last_fullsized":"2024-04-02T04:54:07.909662+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:06.134223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:06.134223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:09.105222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":52,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.921786+0000","last_change":"2024-04-02T04:53:54.929549+0000","last_active":"2024-04-02T04:54:08.921786+0000","last_peered":"2024-04-02T04:54:08.921786+0000","last_clean":"2024-04-02T04:54:08.921786+0000","last_became_active":"2024-04-02T04:53:54.927422+0000","last_became_peered":"2024-04-02T04:53:54.927422+0000","last_unstale":"2024-04-02T04:54:08.921786+0000","last_undegraded":"2024-04-02T04:54:08.921786+0000","last_fullsized":"2024-04-02T04:54:08.921786+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"402'27","reported_seq":755,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233294+0000","last_change":"2024-04-02T04:53:54.923965+0000","last_active":"2024-04-02T04:54:07.233294+0000","last_peered":"2024-04-02T04:54:07.233294+0000","last_clean":"2024-04-02T04:54:07.233294+0000","last_became_active":"2024-04-02T04:53:54.923580+0000","last_became_peered":"2024-04-02T04:53:54.923580+0000","last_unstale":"2024-04-02T04:54:07.233294+0000","last_undegraded":"2024-04-02T04:54:07.233294+0000","last_fullsized":"2024-04-02T04:54:07.233294+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-02T04:49:03.788837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:03.788837+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":708,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.929091+0000","last_change":"2024-04-02T04:50:08.818906+0000","last_active":"2024-04-02T04:54:06.929091+0000","last_peered":"2024-04-02T04:54:06.929091+0000","last_clean":"2024-04-02T04:54:06.929091+0000","last_became_active":"2024-04-02T04:43:54.221219+0000","last_became_peered":"2024-04-02T04:43:54.221219+0000","last_unstale":"2024-04-02T04:54:06.929091+0000","last_undegraded":"2024-04-02T04:54:06.929091+0000","last_fullsized":"2024-04-02T04:54:06.929091+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:08.818819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:50:08.818819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":708,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904511+0000","last_change":"2024-04-02T04:54:04.069490+0000","last_active":"2024-04-02T04:54:07.904511+0000","last_peered":"2024-04-02T04:54:07.904511+0000","last_clean":"2024-04-02T04:54:07.904511+0000","last_became_active":"2024-04-02T04:53:54.915756+0000","last_became_peered":"2024-04-02T04:53:54.915756+0000","last_unstale":"2024-04-02T04:54:07.904511+0000","last_undegraded":"2024-04-02T04:54:07.904511+0000","last_fullsized":"2024-04-02T04:54:07.904511+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.069399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.069399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.503633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":695,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909897+0000","last_change":"2024-04-02T04:52:47.741959+0000","last_active":"2024-04-02T04:54:07.909897+0000","last_peered":"2024-04-02T04:54:07.909897+0000","last_clean":"2024-04-02T04:54:07.909897+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:54:07.909897+0000","last_undegraded":"2024-04-02T04:54:07.909897+0000","last_fullsized":"2024-04-02T04:54:07.909897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.741751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.741751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"138'2","reported_seq":54,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.917311+0000","last_change":"2024-04-02T04:53:54.947231+0000","last_active":"2024-04-02T04:54:08.917311+0000","last_peered":"2024-04-02T04:54:08.917311+0000","last_clean":"2024-04-02T04:54:08.917311+0000","last_became_active":"2024-04-02T04:53:54.945406+0000","last_became_peered":"2024-04-02T04:53:54.945406+0000","last_unstale":"2024-04-02T04:54:08.917311+0000","last_undegraded":"2024-04-02T04:54:08.917311+0000","last_fullsized":"2024-04-02T04:54:08.917311+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":568,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.624111+0000","last_change":"2024-04-02T04:54:07.624201+0000","last_active":"2024-04-02T04:54:07.624111+0000","last_peered":"2024-04-02T04:54:07.624111+0000","last_clean":"2024-04-02T04:54:07.624111+0000","last_became_active":"2024-04-02T04:53:54.917005+0000","last_became_peered":"2024-04-02T04:53:54.917005+0000","last_unstale":"2024-04-02T04:54:07.624111+0000","last_undegraded":"2024-04-02T04:54:07.624111+0000","last_fullsized":"2024-04-02T04:54:07.624111+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:07.623881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.623881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.876277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"402'62","reported_seq":803,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906908+0000","last_change":"2024-04-02T04:50:33.298622+0000","last_active":"2024-04-02T04:54:07.906908+0000","last_peered":"2024-04-02T04:54:07.906908+0000","last_clean":"2024-04-02T04:54:07.906908+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:54:07.906908+0000","last_undegraded":"2024-04-02T04:54:07.906908+0000","last_fullsized":"2024-04-02T04:54:07.906908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:50:33.298479+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:50:33.298479+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":718,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.228608+0000","last_change":"2024-04-02T04:54:03.730180+0000","last_active":"2024-04-02T04:54:07.228608+0000","last_peered":"2024-04-02T04:54:07.228608+0000","last_clean":"2024-04-02T04:54:07.228608+0000","last_became_active":"2024-04-02T04:53:54.917714+0000","last_became_peered":"2024-04-02T04:53:54.917714+0000","last_unstale":"2024-04-02T04:54:07.228608+0000","last_undegraded":"2024-04-02T04:54:07.228608+0000","last_fullsized":"2024-04-02T04:54:07.228608+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.730067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.730067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.494014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":703,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.923358+0000","last_change":"2024-04-02T04:53:54.955657+0000","last_active":"2024-04-02T04:54:06.923358+0000","last_peered":"2024-04-02T04:54:06.923358+0000","last_clean":"2024-04-02T04:54:06.923358+0000","last_became_active":"2024-04-02T04:53:54.955305+0000","last_became_peered":"2024-04-02T04:53:54.955305+0000","last_unstale":"2024-04-02T04:54:06.923358+0000","last_undegraded":"2024-04-02T04:54:06.923358+0000","last_fullsized":"2024-04-02T04:54:06.923358+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":174,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920446+0000","last_change":"2024-04-02T04:54:00.148555+0000","last_active":"2024-04-02T04:54:08.920446+0000","last_peered":"2024-04-02T04:54:08.920446+0000","last_clean":"2024-04-02T04:54:08.920446+0000","last_became_active":"2024-04-02T04:53:54.915365+0000","last_became_peered":"2024-04-02T04:53:54.915365+0000","last_unstale":"2024-04-02T04:54:08.920446+0000","last_undegraded":"2024-04-02T04:54:08.920446+0000","last_fullsized":"2024-04-02T04:54:08.920446+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:54:00.147830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.147830+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:15.021369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.17","version":"139'5","reported_seq":564,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233037+0000","last_change":"2024-04-02T04:51:25.643429+0000","last_active":"2024-04-02T04:54:07.233037+0000","last_peered":"2024-04-02T04:54:07.233037+0000","last_clean":"2024-04-02T04:54:07.233037+0000","last_became_active":"2024-04-02T04:49:26.272870+0000","last_became_peered":"2024-04-02T04:49:26.272870+0000","last_unstale":"2024-04-02T04:54:07.233037+0000","last_undegraded":"2024-04-02T04:54:07.233037+0000","last_fullsized":"2024-04-02T04:54:07.233037+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:25.643193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.643193+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":560,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230002+0000","last_change":"2024-04-02T04:53:54.918075+0000","last_active":"2024-04-02T04:54:07.230002+0000","last_peered":"2024-04-02T04:54:07.230002+0000","last_clean":"2024-04-02T04:54:07.230002+0000","last_became_active":"2024-04-02T04:53:54.916741+0000","last_became_peered":"2024-04-02T04:53:54.916741+0000","last_unstale":"2024-04-02T04:54:07.230002+0000","last_undegraded":"2024-04-02T04:54:07.230002+0000","last_fullsized":"2024-04-02T04:54:07.230002+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"402'21","reported_seq":748,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904482+0000","last_change":"2024-04-02T04:52:58.806428+0000","last_active":"2024-04-02T04:54:07.904482+0000","last_peered":"2024-04-02T04:54:07.904482+0000","last_clean":"2024-04-02T04:54:07.904482+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:54:07.904482+0000","last_undegraded":"2024-04-02T04:54:07.904482+0000","last_fullsized":"2024-04-02T04:54:07.904482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:58.806324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.806324+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:10.998925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":709,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904609+0000","last_change":"2024-04-02T04:50:28.247971+0000","last_active":"2024-04-02T04:54:07.904609+0000","last_peered":"2024-04-02T04:54:07.904609+0000","last_clean":"2024-04-02T04:54:07.904609+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:54:07.904609+0000","last_undegraded":"2024-04-02T04:54:07.904609+0000","last_fullsized":"2024-04-02T04:54:07.904609+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:50:28.247657+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:50:28.247657+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'11","reported_seq":828,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.929898+0000","last_change":"2024-04-02T04:50:12.840880+0000","last_active":"2024-04-02T04:54:06.929898+0000","last_peered":"2024-04-02T04:54:06.929898+0000","last_clean":"2024-04-02T04:54:06.929898+0000","last_became_active":"2024-04-02T04:43:58.302033+0000","last_became_peered":"2024-04-02T04:43:58.302033+0000","last_unstale":"2024-04-02T04:54:06.929898+0000","last_undegraded":"2024-04-02T04:54:06.929898+0000","last_fullsized":"2024-04-02T04:54:06.929898+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:12.840785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.840785+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":693,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.912888+0000","last_change":"2024-04-02T04:50:15.299404+0000","last_active":"2024-04-02T04:54:07.912888+0000","last_peered":"2024-04-02T04:54:07.912888+0000","last_clean":"2024-04-02T04:54:07.912888+0000","last_became_active":"2024-04-02T04:44:00.353930+0000","last_became_peered":"2024-04-02T04:44:00.353930+0000","last_unstale":"2024-04-02T04:54:07.912888+0000","last_undegraded":"2024-04-02T04:54:07.912888+0000","last_fullsized":"2024-04-02T04:54:07.912888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:15.299316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.299316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"139'5","reported_seq":569,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909049+0000","last_change":"2024-04-02T04:51:32.581419+0000","last_active":"2024-04-02T04:54:07.909049+0000","last_peered":"2024-04-02T04:54:07.909049+0000","last_clean":"2024-04-02T04:54:07.909049+0000","last_became_active":"2024-04-02T04:49:25.091539+0000","last_became_peered":"2024-04-02T04:49:25.091539+0000","last_unstale":"2024-04-02T04:54:07.909049+0000","last_undegraded":"2024-04-02T04:54:07.909049+0000","last_fullsized":"2024-04-02T04:54:07.909049+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:32.581233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:32.581233+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921292+0000","last_change":"2024-04-02T04:50:37.840934+0000","last_active":"2024-04-02T04:54:06.921292+0000","last_peered":"2024-04-02T04:54:06.921292+0000","last_clean":"2024-04-02T04:54:06.921292+0000","last_became_active":"2024-04-02T04:49:26.256165+0000","last_became_peered":"2024-04-02T04:49:26.256165+0000","last_unstale":"2024-04-02T04:54:06.921292+0000","last_undegraded":"2024-04-02T04:54:06.921292+0000","last_fullsized":"2024-04-02T04:54:06.921292+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.840813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.840813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"402'16","reported_seq":726,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.920717+0000","last_change":"2024-04-02T04:53:54.940945+0000","last_active":"2024-04-02T04:54:06.920717+0000","last_peered":"2024-04-02T04:54:06.920717+0000","last_clean":"2024-04-02T04:54:06.920717+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T04:54:06.920717+0000","last_undegraded":"2024-04-02T04:54:06.920717+0000","last_fullsized":"2024-04-02T04:54:06.920717+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'14","last_scrub_stamp":"2024-04-02T04:49:05.019939+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:49:05.019939+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":709,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.921538+0000","last_change":"2024-04-02T04:53:54.903674+0000","last_active":"2024-04-02T04:54:06.921538+0000","last_peered":"2024-04-02T04:54:06.921538+0000","last_clean":"2024-04-02T04:54:06.921538+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T04:54:06.921538+0000","last_undegraded":"2024-04-02T04:54:06.921538+0000","last_fullsized":"2024-04-02T04:54:06.921538+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:14.778569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.778569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":707,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919999+0000","last_change":"2024-04-02T04:52:58.690594+0000","last_active":"2024-04-02T04:54:06.919999+0000","last_peered":"2024-04-02T04:54:06.919999+0000","last_clean":"2024-04-02T04:54:06.919999+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:54:06.919999+0000","last_undegraded":"2024-04-02T04:54:06.919999+0000","last_fullsized":"2024-04-02T04:54:06.919999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.690492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.690492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":691,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913843+0000","last_change":"2024-04-02T04:50:27.563922+0000","last_active":"2024-04-02T04:54:06.913843+0000","last_peered":"2024-04-02T04:54:06.913843+0000","last_clean":"2024-04-02T04:54:06.913843+0000","last_became_active":"2024-04-02T04:44:00.334068+0000","last_became_peered":"2024-04-02T04:44:00.334068+0000","last_unstale":"2024-04-02T04:54:06.913843+0000","last_undegraded":"2024-04-02T04:54:06.913843+0000","last_fullsized":"2024-04-02T04:54:06.913843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.563824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.563824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911508+0000","last_change":"2024-04-02T04:52:49.244563+0000","last_active":"2024-04-02T04:54:06.911508+0000","last_peered":"2024-04-02T04:54:06.911508+0000","last_clean":"2024-04-02T04:54:06.911508+0000","last_became_active":"2024-04-02T04:49:26.507403+0000","last_became_peered":"2024-04-02T04:49:26.507403+0000","last_unstale":"2024-04-02T04:54:06.911508+0000","last_undegraded":"2024-04-02T04:54:06.911508+0000","last_fullsized":"2024-04-02T04:54:06.911508+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.244258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.244258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":563,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910128+0000","last_change":"2024-04-02T04:53:54.943938+0000","last_active":"2024-04-02T04:54:07.910128+0000","last_peered":"2024-04-02T04:54:07.910128+0000","last_clean":"2024-04-02T04:54:07.910128+0000","last_became_active":"2024-04-02T04:53:54.942584+0000","last_became_peered":"2024-04-02T04:53:54.942584+0000","last_unstale":"2024-04-02T04:54:07.910128+0000","last_undegraded":"2024-04-02T04:54:07.910128+0000","last_fullsized":"2024-04-02T04:54:07.910128+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"402'35","reported_seq":755,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901453+0000","last_change":"2024-04-02T04:50:14.537273+0000","last_active":"2024-04-02T04:54:06.901453+0000","last_peered":"2024-04-02T04:54:06.901453+0000","last_clean":"2024-04-02T04:54:06.901453+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:54:06.901453+0000","last_undegraded":"2024-04-02T04:54:06.901453+0000","last_fullsized":"2024-04-02T04:54:06.901453+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:50:14.537029+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.537029+0000","objects_scrubbed":6,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":716,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909639+0000","last_change":"2024-04-02T04:53:54.945507+0000","last_active":"2024-04-02T04:54:07.909639+0000","last_peered":"2024-04-02T04:54:07.909639+0000","last_clean":"2024-04-02T04:54:07.909639+0000","last_became_active":"2024-04-02T04:53:54.944446+0000","last_became_peered":"2024-04-02T04:53:54.944446+0000","last_unstale":"2024-04-02T04:54:07.909639+0000","last_undegraded":"2024-04-02T04:54:07.909639+0000","last_fullsized":"2024-04-02T04:54:07.909639+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:12.329420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:12.329420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":707,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.911862+0000","last_change":"2024-04-02T04:53:14.837679+0000","last_active":"2024-04-02T04:54:07.911862+0000","last_peered":"2024-04-02T04:54:07.911862+0000","last_clean":"2024-04-02T04:54:07.911862+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:54:07.911862+0000","last_undegraded":"2024-04-02T04:54:07.911862+0000","last_fullsized":"2024-04-02T04:54:07.911862+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:14.837465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:53:14.837465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:40.038515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":695,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.909982+0000","last_change":"2024-04-02T04:50:40.672009+0000","last_active":"2024-04-02T04:54:07.909982+0000","last_peered":"2024-04-02T04:54:07.909982+0000","last_clean":"2024-04-02T04:54:07.909982+0000","last_became_active":"2024-04-02T04:44:00.346529+0000","last_became_peered":"2024-04-02T04:44:00.346529+0000","last_unstale":"2024-04-02T04:54:07.909982+0000","last_undegraded":"2024-04-02T04:54:07.909982+0000","last_fullsized":"2024-04-02T04:54:07.909982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.671923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.671923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"138'2","reported_seq":562,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.919443+0000","last_change":"2024-04-02T04:53:54.940409+0000","last_active":"2024-04-02T04:54:06.919443+0000","last_peered":"2024-04-02T04:54:06.919443+0000","last_clean":"2024-04-02T04:54:06.919443+0000","last_became_active":"2024-04-02T04:53:54.939949+0000","last_became_peered":"2024-04-02T04:53:54.939949+0000","last_unstale":"2024-04-02T04:54:06.919443+0000","last_undegraded":"2024-04-02T04:54:06.919443+0000","last_fullsized":"2024-04-02T04:54:06.919443+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":562,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.910637+0000","last_change":"2024-04-02T04:50:39.321796+0000","last_active":"2024-04-02T04:54:07.910637+0000","last_peered":"2024-04-02T04:54:07.910637+0000","last_clean":"2024-04-02T04:54:07.910637+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:54:07.910637+0000","last_undegraded":"2024-04-02T04:54:07.910637+0000","last_fullsized":"2024-04-02T04:54:07.910637+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.321605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"402'24","reported_seq":749,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.231408+0000","last_change":"2024-04-02T04:52:48.727813+0000","last_active":"2024-04-02T04:54:07.231408+0000","last_peered":"2024-04-02T04:54:07.231408+0000","last_clean":"2024-04-02T04:54:07.231408+0000","last_became_active":"2024-04-02T04:43:56.245463+0000","last_became_peered":"2024-04-02T04:43:56.245463+0000","last_unstale":"2024-04-02T04:54:07.231408+0000","last_undegraded":"2024-04-02T04:54:07.231408+0000","last_fullsized":"2024-04-02T04:54:07.231408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:52:48.727589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.727589+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.632589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":186,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.920188+0000","last_change":"2024-04-02T04:53:54.926722+0000","last_active":"2024-04-02T04:54:08.920188+0000","last_peered":"2024-04-02T04:54:08.920188+0000","last_clean":"2024-04-02T04:54:08.920188+0000","last_became_active":"2024-04-02T04:53:54.918406+0000","last_became_peered":"2024-04-02T04:53:54.918406+0000","last_unstale":"2024-04-02T04:54:08.920188+0000","last_undegraded":"2024-04-02T04:54:08.920188+0000","last_fullsized":"2024-04-02T04:54:08.920188+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:48:48.436655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:48:48.436655+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":694,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.902035+0000","last_change":"2024-04-02T04:51:48.311838+0000","last_active":"2024-04-02T04:54:06.902035+0000","last_peered":"2024-04-02T04:54:06.902035+0000","last_clean":"2024-04-02T04:54:06.902035+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:54:06.902035+0000","last_undegraded":"2024-04-02T04:54:06.902035+0000","last_fullsized":"2024-04-02T04:54:06.902035+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.311616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.311616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":690,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.914237+0000","last_change":"2024-04-02T04:53:54.958067+0000","last_active":"2024-04-02T04:54:06.914237+0000","last_peered":"2024-04-02T04:54:06.914237+0000","last_clean":"2024-04-02T04:54:06.914237+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T04:54:06.914237+0000","last_undegraded":"2024-04-02T04:54:06.914237+0000","last_fullsized":"2024-04-02T04:54:06.914237+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:48:02.193629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:48:02.193629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"139'9","reported_seq":63,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.919704+0000","last_change":"2024-04-02T04:53:54.947224+0000","last_active":"2024-04-02T04:54:08.919704+0000","last_peered":"2024-04-02T04:54:08.919704+0000","last_clean":"2024-04-02T04:54:08.919704+0000","last_became_active":"2024-04-02T04:53:54.945565+0000","last_became_peered":"2024-04-02T04:53:54.945565+0000","last_unstale":"2024-04-02T04:54:08.919704+0000","last_undegraded":"2024-04-02T04:54:08.919704+0000","last_fullsized":"2024-04-02T04:54:08.919704+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":563,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908514+0000","last_change":"2024-04-02T04:51:24.638019+0000","last_active":"2024-04-02T04:54:07.908514+0000","last_peered":"2024-04-02T04:54:07.908514+0000","last_clean":"2024-04-02T04:54:07.908514+0000","last_became_active":"2024-04-02T04:49:25.094609+0000","last_became_peered":"2024-04-02T04:49:25.094609+0000","last_unstale":"2024-04-02T04:54:07.908514+0000","last_undegraded":"2024-04-02T04:54:07.908514+0000","last_fullsized":"2024-04-02T04:54:07.908514+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:24.637853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:24.637853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"402'27","reported_seq":748,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.233354+0000","last_change":"2024-04-02T04:52:53.802878+0000","last_active":"2024-04-02T04:54:07.233354+0000","last_peered":"2024-04-02T04:54:07.233354+0000","last_clean":"2024-04-02T04:54:07.233354+0000","last_became_active":"2024-04-02T04:43:56.238858+0000","last_became_peered":"2024-04-02T04:43:56.238858+0000","last_unstale":"2024-04-02T04:54:07.233354+0000","last_undegraded":"2024-04-02T04:54:07.233354+0000","last_fullsized":"2024-04-02T04:54:07.233354+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:53.802714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.802714+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:21.332468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":199,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-02T04:54:08.923004+0000","last_change":"2024-04-02T04:53:54.943928+0000","last_active":"2024-04-02T04:54:08.923004+0000","last_peered":"2024-04-02T04:54:08.923004+0000","last_clean":"2024-04-02T04:54:08.923004+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T04:54:08.923004+0000","last_undegraded":"2024-04-02T04:54:08.923004+0000","last_fullsized":"2024-04-02T04:54:08.923004+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:22.452700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:22.452700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":694,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.903938+0000","last_change":"2024-04-02T04:50:09.464337+0000","last_active":"2024-04-02T04:54:06.903938+0000","last_peered":"2024-04-02T04:54:06.903938+0000","last_clean":"2024-04-02T04:54:06.903938+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:54:06.903938+0000","last_undegraded":"2024-04-02T04:54:06.903938+0000","last_fullsized":"2024-04-02T04:54:06.903938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.464222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.464222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":705,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.230242+0000","last_change":"2024-04-02T04:52:47.730759+0000","last_active":"2024-04-02T04:54:07.230242+0000","last_peered":"2024-04-02T04:54:07.230242+0000","last_clean":"2024-04-02T04:54:07.230242+0000","last_became_active":"2024-04-02T04:44:00.340613+0000","last_became_peered":"2024-04-02T04:44:00.340613+0000","last_unstale":"2024-04-02T04:54:07.230242+0000","last_undegraded":"2024-04-02T04:54:07.230242+0000","last_fullsized":"2024-04-02T04:54:07.230242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.730634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.730634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":567,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.913628+0000","last_change":"2024-04-02T04:53:06.382048+0000","last_active":"2024-04-02T04:54:06.913628+0000","last_peered":"2024-04-02T04:54:06.913628+0000","last_clean":"2024-04-02T04:54:06.913628+0000","last_became_active":"2024-04-02T04:49:25.076275+0000","last_became_peered":"2024-04-02T04:49:25.076275+0000","last_unstale":"2024-04-02T04:54:06.913628+0000","last_undegraded":"2024-04-02T04:54:06.913628+0000","last_fullsized":"2024-04-02T04:54:06.913628+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.381932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.381932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:29.209600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":556,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.925829+0000","last_change":"2024-04-02T04:52:49.488923+0000","last_active":"2024-04-02T04:54:06.925829+0000","last_peered":"2024-04-02T04:54:06.925829+0000","last_clean":"2024-04-02T04:54:06.925829+0000","last_became_active":"2024-04-02T04:49:26.275375+0000","last_became_peered":"2024-04-02T04:49:26.275375+0000","last_unstale":"2024-04-02T04:54:06.925829+0000","last_undegraded":"2024-04-02T04:54:06.925829+0000","last_fullsized":"2024-04-02T04:54:06.925829+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.488717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.488717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"402'25","reported_seq":748,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926014+0000","last_change":"2024-04-02T04:52:52.303113+0000","last_active":"2024-04-02T04:54:06.926014+0000","last_peered":"2024-04-02T04:54:06.926014+0000","last_clean":"2024-04-02T04:54:06.926014+0000","last_became_active":"2024-04-02T04:43:56.243336+0000","last_became_peered":"2024-04-02T04:43:56.243336+0000","last_unstale":"2024-04-02T04:54:06.926014+0000","last_undegraded":"2024-04-02T04:54:06.926014+0000","last_fullsized":"2024-04-02T04:54:06.926014+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:52:52.302878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.302878+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.761680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":716,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908622+0000","last_change":"2024-04-02T04:53:01.793779+0000","last_active":"2024-04-02T04:54:07.908622+0000","last_peered":"2024-04-02T04:54:07.908622+0000","last_clean":"2024-04-02T04:54:07.908622+0000","last_became_active":"2024-04-02T04:43:54.208527+0000","last_became_peered":"2024-04-02T04:43:54.208527+0000","last_unstale":"2024-04-02T04:54:07.908622+0000","last_undegraded":"2024-04-02T04:54:07.908622+0000","last_fullsized":"2024-04-02T04:54:07.908622+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:01.793655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:01.793655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:15.044051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":694,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901554+0000","last_change":"2024-04-02T04:51:49.302781+0000","last_active":"2024-04-02T04:54:06.901554+0000","last_peered":"2024-04-02T04:54:06.901554+0000","last_clean":"2024-04-02T04:54:06.901554+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:54:06.901554+0000","last_undegraded":"2024-04-02T04:54:06.901554+0000","last_fullsized":"2024-04-02T04:54:06.901554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.302549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.302549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":690,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.901290+0000","last_change":"2024-04-02T04:52:41.343784+0000","last_active":"2024-04-02T04:54:06.901290+0000","last_peered":"2024-04-02T04:54:06.901290+0000","last_clean":"2024-04-02T04:54:06.901290+0000","last_became_active":"2024-04-02T04:44:00.345046+0000","last_became_peered":"2024-04-02T04:44:00.345046+0000","last_unstale":"2024-04-02T04:54:06.901290+0000","last_undegraded":"2024-04-02T04:54:06.901290+0000","last_fullsized":"2024-04-02T04:54:06.901290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.343663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.343663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"139'7","reported_seq":569,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.906966+0000","last_change":"2024-04-02T04:51:38.237676+0000","last_active":"2024-04-02T04:54:07.906966+0000","last_peered":"2024-04-02T04:54:07.906966+0000","last_clean":"2024-04-02T04:54:07.906966+0000","last_became_active":"2024-04-02T04:49:26.241390+0000","last_became_peered":"2024-04-02T04:49:26.241390+0000","last_unstale":"2024-04-02T04:54:07.906966+0000","last_undegraded":"2024-04-02T04:54:07.906966+0000","last_fullsized":"2024-04-02T04:54:07.906966+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-04-02T04:51:38.237550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.237550+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":559,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.911778+0000","last_change":"2024-04-02T04:51:36.397355+0000","last_active":"2024-04-02T04:54:06.911778+0000","last_peered":"2024-04-02T04:54:06.911778+0000","last_clean":"2024-04-02T04:54:06.911778+0000","last_became_active":"2024-04-02T04:49:26.536918+0000","last_became_peered":"2024-04-02T04:49:26.536918+0000","last_unstale":"2024-04-02T04:54:06.911778+0000","last_undegraded":"2024-04-02T04:54:06.911778+0000","last_fullsized":"2024-04-02T04:54:06.911778+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.397213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.397213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"402'32","reported_seq":760,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908734+0000","last_change":"2024-04-02T04:53:02.778087+0000","last_active":"2024-04-02T04:54:07.908734+0000","last_peered":"2024-04-02T04:54:07.908734+0000","last_clean":"2024-04-02T04:54:07.908734+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:54:07.908734+0000","last_undegraded":"2024-04-02T04:54:07.908734+0000","last_fullsized":"2024-04-02T04:54:07.908734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'22","last_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_deep_scrub":"342'22","last_deep_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.777776+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:23.211739+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":702,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-04-02T04:54:06.926193+0000","last_change":"2024-04-02T04:53:54.925037+0000","last_active":"2024-04-02T04:54:06.926193+0000","last_peered":"2024-04-02T04:54:06.926193+0000","last_clean":"2024-04-02T04:54:06.926193+0000","last_became_active":"2024-04-02T04:53:54.924695+0000","last_became_peered":"2024-04-02T04:53:54.924695+0000","last_unstale":"2024-04-02T04:54:06.926193+0000","last_undegraded":"2024-04-02T04:54:06.926193+0000","last_fullsized":"2024-04-02T04:54:06.926193+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:47:54.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:47:54.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":707,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.904342+0000","last_change":"2024-04-02T04:52:56.823906+0000","last_active":"2024-04-02T04:54:07.904342+0000","last_peered":"2024-04-02T04:54:07.904342+0000","last_clean":"2024-04-02T04:54:07.904342+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:54:07.904342+0000","last_undegraded":"2024-04-02T04:54:07.904342+0000","last_fullsized":"2024-04-02T04:54:07.904342+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.823620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.823620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:13.652305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":696,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.908909+0000","last_change":"2024-04-02T04:53:54.945031+0000","last_active":"2024-04-02T04:54:07.908909+0000","last_peered":"2024-04-02T04:54:07.908909+0000","last_clean":"2024-04-02T04:54:07.908909+0000","last_became_active":"2024-04-02T04:53:54.943990+0000","last_became_peered":"2024-04-02T04:53:54.943990+0000","last_unstale":"2024-04-02T04:54:07.908909+0000","last_undegraded":"2024-04-02T04:54:07.908909+0000","last_fullsized":"2024-04-02T04:54:07.908909+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:09.389577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:49:09.389577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":570,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-02T04:54:07.907627+0000","last_change":"2024-04-02T04:53:06.393893+0000","last_active":"2024-04-02T04:54:07.907627+0000","last_peered":"2024-04-02T04:54:07.907627+0000","last_clean":"2024-04-02T04:54:07.907627+0000","last_became_active":"2024-04-02T04:49:26.269993+0000","last_became_peered":"2024-04-02T04:49:26.269993+0000","last_unstale":"2024-04-02T04:54:07.907627+0000","last_undegraded":"2024-04-02T04:54:07.907627+0000","last_fullsized":"2024-04-02T04:54:07.907627+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:54:23.095557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":283,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":8196233,"num_objects":196,"num_object_clones":52,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":1,"num_read":102,"num_read_kb":6,"num_write":572,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26947584,"data_stored":24588699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1722,"ondisk_log_size":1722,"up":96,"acting":96,"num_store_stats":8},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1412,"num_read_kb":1137,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":938,"ondisk_log_size":938,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114629,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40212,"kb_used_data":7860,"kb_used_omap":57,"kb_used_meta":32262,"kb_avail":93704940,"statfs":{"total":95995035648,"available":95953858560,"internally_reserved":0,"allocated":8048640,"data_stored":5146572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":33037198},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103312,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79080,"kb_used_data":15956,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93666072,"statfs":{"total":95995035648,"available":95914057728,"internally_reserved":0,"allocated":16338944,"data_stored":13325968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74368,"kb_used_data":11256,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670784,"statfs":{"total":95995035648,"available":95918882816,"internally_reserved":0,"allocated":11526144,"data_stored":8736440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738573,"num_pgs":131,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87372,"kb_used_data":7608,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93657780,"statfs":{"total":95995035648,"available":95905566720,"internally_reserved":0,"allocated":7790592,"data_stored":5131277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378809,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70428,"kb_used_data":7304,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674724,"statfs":{"total":95995035648,"available":95922917376,"internally_reserved":0,"allocated":7479296,"data_stored":4542986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051759,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78228,"kb_used_data":10920,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666924,"statfs":{"total":95995035648,"available":95914930176,"internally_reserved":0,"allocated":11182080,"data_stored":8540634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822821,"num_pgs":138,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56308,"kb_used_data":11220,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93688844,"statfs":{"total":95995035648,"available":95937376256,"internally_reserved":0,"allocated":11489280,"data_stored":8736469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463065,"num_pgs":133,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68484,"kb_used_data":10928,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93676668,"statfs":{"total":95995035648,"available":95924908032,"internally_reserved":0,"allocated":11190272,"data_stored":8541627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4481024,"data_stored":4195004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":4000409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4415488,"data_stored":4194799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":4000430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8720384,"data_stored":8195609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":283,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:54:15.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 1 2024-04-02T04:54:15.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:15.043 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:15.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:15.165 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:15.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:15.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:15.426 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:15.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:15.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:15.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:13.636618+0000 osd.4 (osd.4) 237 : cluster 0 43.1e scrub starts 2024-04-02T04:54:15.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:13.638958+0000 osd.4 (osd.4) 238 : cluster 0 43.1e scrub ok 2024-04-02T04:54:15.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.086227+0000 mgr.y (mgr.14538) 417 : cluster 0 pgmap v598: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: audit 2024-04-02T04:54:14.181430+0000 mon.c (mon.2) 752 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.243650+0000 osd.7 (osd.7) 19 : cluster 0 2.1c scrub starts 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.246457+0000 osd.7 (osd.7) 20 : cluster 0 2.1c scrub ok 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.418128+0000 mon.a (mon.0) 2597 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.555052+0000 osd.5 (osd.5) 231 : cluster 0 10.9 scrub starts 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.557157+0000 osd.5 (osd.5) 232 : cluster 0 10.9 scrub ok 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: audit 2024-04-02T04:54:14.873016+0000 mon.a (mon.0) 2598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.883325+0000 mon.a (mon.0) 2599 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-04-02T04:54:15.670 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:15 smithi098 bash[23790]: audit 2024-04-02T04:54:15.182505+0000 mon.c (mon.2) 753 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:15.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:13.636618+0000 osd.4 (osd.4) 237 : cluster 0 43.1e scrub starts 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:13.638958+0000 osd.4 (osd.4) 238 : cluster 0 43.1e scrub ok 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.086227+0000 mgr.y (mgr.14538) 417 : cluster 0 pgmap v598: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: audit 2024-04-02T04:54:14.181430+0000 mon.c (mon.2) 752 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.243650+0000 osd.7 (osd.7) 19 : cluster 0 2.1c scrub starts 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.246457+0000 osd.7 (osd.7) 20 : cluster 0 2.1c scrub ok 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.418128+0000 mon.a (mon.0) 2597 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.555052+0000 osd.5 (osd.5) 231 : cluster 0 10.9 scrub starts 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.557157+0000 osd.5 (osd.5) 232 : cluster 0 10.9 scrub ok 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: audit 2024-04-02T04:54:14.873016+0000 mon.a (mon.0) 2598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.883325+0000 mon.a (mon.0) 2599 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-04-02T04:54:15.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[21528]: audit 2024-04-02T04:54:15.182505+0000 mon.c (mon.2) 753 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:13.636618+0000 osd.4 (osd.4) 237 : cluster 0 43.1e scrub starts 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:13.638958+0000 osd.4 (osd.4) 238 : cluster 0 43.1e scrub ok 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.086227+0000 mgr.y (mgr.14538) 417 : cluster 0 pgmap v598: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: audit 2024-04-02T04:54:14.181430+0000 mon.c (mon.2) 752 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.243650+0000 osd.7 (osd.7) 19 : cluster 0 2.1c scrub starts 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.246457+0000 osd.7 (osd.7) 20 : cluster 0 2.1c scrub ok 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.418128+0000 mon.a (mon.0) 2597 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.555052+0000 osd.5 (osd.5) 231 : cluster 0 10.9 scrub starts 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.557157+0000 osd.5 (osd.5) 232 : cluster 0 10.9 scrub ok 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: audit 2024-04-02T04:54:14.873016+0000 mon.a (mon.0) 2598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-72153-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:15.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.883325+0000 mon.a (mon.0) 2599 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-04-02T04:54:15.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:15 smithi069 bash[28904]: audit 2024-04-02T04:54:15.182505+0000 mon.c (mon.2) 753 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:15.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:15.815 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:15.918 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:15.944 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:15.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:16.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:16.073 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:16.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:16.204 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:16.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:16.334 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:16.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:16.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.664865+0000 osd.4 (osd.4) 239 : cluster 0 41.6 scrub starts 2024-04-02T04:54:16.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:14.666541+0000 osd.4 (osd.4) 240 : cluster 0 41.6 scrub ok 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:15.107546+0000 osd.3 (osd.3) 265 : cluster 0 6.1b scrub starts 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:15.108985+0000 osd.3 (osd.3) 266 : cluster 0 6.1b scrub ok 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:15.254243+0000 osd.7 (osd.7) 21 : cluster 0 3.f deep-scrub starts 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:15.261662+0000 osd.7 (osd.7) 22 : cluster 0 3.f deep-scrub ok 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: cluster 2024-04-02T04:54:15.880481+0000 mon.a (mon.0) 2600 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[21528]: audit 2024-04-02T04:54:16.183699+0000 mon.c (mon.2) 754 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.664865+0000 osd.4 (osd.4) 239 : cluster 0 41.6 scrub starts 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:14.666541+0000 osd.4 (osd.4) 240 : cluster 0 41.6 scrub ok 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:15.107546+0000 osd.3 (osd.3) 265 : cluster 0 6.1b scrub starts 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:15.108985+0000 osd.3 (osd.3) 266 : cluster 0 6.1b scrub ok 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:15.254243+0000 osd.7 (osd.7) 21 : cluster 0 3.f deep-scrub starts 2024-04-02T04:54:16.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:15.261662+0000 osd.7 (osd.7) 22 : cluster 0 3.f deep-scrub ok 2024-04-02T04:54:16.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: cluster 2024-04-02T04:54:15.880481+0000 mon.a (mon.0) 2600 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-04-02T04:54:16.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:16 smithi069 bash[28904]: audit 2024-04-02T04:54:16.183699+0000 mon.c (mon.2) 754 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:16.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:16.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:16.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.664865+0000 osd.4 (osd.4) 239 : cluster 0 41.6 scrub starts 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:14.666541+0000 osd.4 (osd.4) 240 : cluster 0 41.6 scrub ok 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:15.107546+0000 osd.3 (osd.3) 265 : cluster 0 6.1b scrub starts 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:15.108985+0000 osd.3 (osd.3) 266 : cluster 0 6.1b scrub ok 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:15.254243+0000 osd.7 (osd.7) 21 : cluster 0 3.f deep-scrub starts 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:15.261662+0000 osd.7 (osd.7) 22 : cluster 0 3.f deep-scrub ok 2024-04-02T04:54:16.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: cluster 2024-04-02T04:54:15.880481+0000 mon.a (mon.0) 2600 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-04-02T04:54:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:16 smithi098 bash[23790]: audit 2024-04-02T04:54:16.183699+0000 mon.c (mon.2) 754 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:16.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:16.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:16.851 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:16.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:16.976 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:16.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:17.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:54:17.079 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:17.101 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:17.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:17.232 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:17.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:17.363 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:17.465 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:17.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: cluster 2024-04-02T04:54:16.084886+0000 osd.3 (osd.3) 267 : cluster 0 4.17 scrub starts 2024-04-02T04:54:17.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: cluster 2024-04-02T04:54:16.086769+0000 osd.3 (osd.3) 268 : cluster 0 4.17 scrub ok 2024-04-02T04:54:17.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: cluster 2024-04-02T04:54:16.087167+0000 mgr.y (mgr.14538) 418 : cluster 0 pgmap v601: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T04:54:17.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: cluster 2024-04-02T04:54:16.875454+0000 mon.a (mon.0) 2601 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:17.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: cluster 2024-04-02T04:54:16.891944+0000 mon.a (mon.0) 2602 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-04-02T04:54:17.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: audit 2024-04-02T04:54:16.894138+0000 mon.c (mon.2) 755 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:17.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: audit 2024-04-02T04:54:16.895053+0000 mon.a (mon.0) 2603 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:17.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:17 smithi098 bash[23790]: audit 2024-04-02T04:54:17.185527+0000 mon.c (mon.2) 756 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:17.491 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:17.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:17.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: cluster 2024-04-02T04:54:16.084886+0000 osd.3 (osd.3) 267 : cluster 0 4.17 scrub starts 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: cluster 2024-04-02T04:54:16.086769+0000 osd.3 (osd.3) 268 : cluster 0 4.17 scrub ok 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: cluster 2024-04-02T04:54:16.087167+0000 mgr.y (mgr.14538) 418 : cluster 0 pgmap v601: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: cluster 2024-04-02T04:54:16.875454+0000 mon.a (mon.0) 2601 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: cluster 2024-04-02T04:54:16.891944+0000 mon.a (mon.0) 2602 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: audit 2024-04-02T04:54:16.894138+0000 mon.c (mon.2) 755 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: audit 2024-04-02T04:54:16.895053+0000 mon.a (mon.0) 2603 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[21528]: audit 2024-04-02T04:54:17.185527+0000 mon.c (mon.2) 756 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: cluster 2024-04-02T04:54:16.084886+0000 osd.3 (osd.3) 267 : cluster 0 4.17 scrub starts 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: cluster 2024-04-02T04:54:16.086769+0000 osd.3 (osd.3) 268 : cluster 0 4.17 scrub ok 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: cluster 2024-04-02T04:54:16.087167+0000 mgr.y (mgr.14538) 418 : cluster 0 pgmap v601: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: cluster 2024-04-02T04:54:16.875454+0000 mon.a (mon.0) 2601 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:17.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: cluster 2024-04-02T04:54:16.891944+0000 mon.a (mon.0) 2602 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-04-02T04:54:17.605 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: audit 2024-04-02T04:54:16.894138+0000 mon.c (mon.2) 755 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:17.605 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: audit 2024-04-02T04:54:16.895053+0000 mon.a (mon.0) 2603 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:17.605 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:17 smithi069 bash[28904]: audit 2024-04-02T04:54:17.185527+0000 mon.c (mon.2) 756 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:17.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:17.790 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:54:17.790 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:54:17.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:17.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:17.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:17.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:18.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:18.130 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:18.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:18.262 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:18.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:18.496 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:18.522 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:18.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.128412+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub starts 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.131278+0000 osd.3 (osd.3) 270 : cluster 0 3.7 scrub ok 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.217118+0000 osd.7 (osd.7) 23 : cluster 0 2.5 scrub starts 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.219346+0000 osd.7 (osd.7) 24 : cluster 0 2.5 scrub ok 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: audit 2024-04-02T04:54:17.887267+0000 mon.a (mon.0) 2604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.891027+0000 mon.a (mon.0) 2605 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-04-02T04:54:18.638 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: audit 2024-04-02T04:54:17.894986+0000 mon.c (mon.2) 757 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: audit 2024-04-02T04:54:17.895531+0000 mon.a (mon.0) 2606 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.924325+0000 osd.6 (osd.6) 199 : cluster 0 3.10 scrub starts 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.927323+0000 osd.6 (osd.6) 200 : cluster 0 3.10 scrub ok 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.933692+0000 osd.2 (osd.2) 194 : cluster 0 102.1a scrub starts 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: cluster 2024-04-02T04:54:17.944936+0000 osd.2 (osd.2) 195 : cluster 0 102.1a scrub ok 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: audit 2024-04-02T04:54:17.971997+0000 mon.a (mon.0) 2607 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: audit 2024-04-02T04:54:17.972463+0000 mon.b (mon.1) 221 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:18.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:18 smithi098 bash[23790]: audit 2024-04-02T04:54:18.187218+0000 mon.c (mon.2) 758 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:18.653 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:18.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.128412+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub starts 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.131278+0000 osd.3 (osd.3) 270 : cluster 0 3.7 scrub ok 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.217118+0000 osd.7 (osd.7) 23 : cluster 0 2.5 scrub starts 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.219346+0000 osd.7 (osd.7) 24 : cluster 0 2.5 scrub ok 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: audit 2024-04-02T04:54:17.887267+0000 mon.a (mon.0) 2604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.891027+0000 mon.a (mon.0) 2605 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: audit 2024-04-02T04:54:17.894986+0000 mon.c (mon.2) 757 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: audit 2024-04-02T04:54:17.895531+0000 mon.a (mon.0) 2606 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:18.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.924325+0000 osd.6 (osd.6) 199 : cluster 0 3.10 scrub starts 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.927323+0000 osd.6 (osd.6) 200 : cluster 0 3.10 scrub ok 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.933692+0000 osd.2 (osd.2) 194 : cluster 0 102.1a scrub starts 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: cluster 2024-04-02T04:54:17.944936+0000 osd.2 (osd.2) 195 : cluster 0 102.1a scrub ok 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: audit 2024-04-02T04:54:17.971997+0000 mon.a (mon.0) 2607 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: audit 2024-04-02T04:54:17.972463+0000 mon.b (mon.1) 221 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21528]: audit 2024-04-02T04:54:18.187218+0000 mon.c (mon.2) 758 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.128412+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub starts 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.131278+0000 osd.3 (osd.3) 270 : cluster 0 3.7 scrub ok 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.217118+0000 osd.7 (osd.7) 23 : cluster 0 2.5 scrub starts 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.219346+0000 osd.7 (osd.7) 24 : cluster 0 2.5 scrub ok 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: audit 2024-04-02T04:54:17.887267+0000 mon.a (mon.0) 2604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.891027+0000 mon.a (mon.0) 2605 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: audit 2024-04-02T04:54:17.894986+0000 mon.c (mon.2) 757 : audit 1 from='client.? 172.21.15.69:0/683787247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: audit 2024-04-02T04:54:17.895531+0000 mon.a (mon.0) 2606 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]: dispatch 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.924325+0000 osd.6 (osd.6) 199 : cluster 0 3.10 scrub starts 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.927323+0000 osd.6 (osd.6) 200 : cluster 0 3.10 scrub ok 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.933692+0000 osd.2 (osd.2) 194 : cluster 0 102.1a scrub starts 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: cluster 2024-04-02T04:54:17.944936+0000 osd.2 (osd.2) 195 : cluster 0 102.1a scrub ok 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: audit 2024-04-02T04:54:17.971997+0000 mon.a (mon.0) 2607 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: audit 2024-04-02T04:54:17.972463+0000 mon.b (mon.1) 221 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[28904]: audit 2024-04-02T04:54:18.187218+0000 mon.c (mon.2) 758 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:18.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:18.777 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:54:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:54:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:54:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:18.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:18.911 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:18.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:19.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:19.038 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:19.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.168 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:19.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:19.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.426 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:19.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:19.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.671 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: cluster 2024-04-02T04:54:18.090122+0000 mgr.y (mgr.14538) 419 : cluster 0 pgmap v604: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: cluster 2024-04-02T04:54:18.246320+0000 osd.7 (osd.7) 25 : cluster 0 2.8 deep-scrub starts 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: cluster 2024-04-02T04:54:18.248853+0000 osd.7 (osd.7) 26 : cluster 0 2.8 deep-scrub ok 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: cluster 2024-04-02T04:54:18.877422+0000 osd.6 (osd.6) 201 : cluster 0 4.1f scrub starts 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: cluster 2024-04-02T04:54:18.879216+0000 osd.6 (osd.6) 202 : cluster 0 4.1f scrub ok 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.889463+0000 mon.a (mon.0) 2608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: cluster 2024-04-02T04:54:18.893678+0000 mon.a (mon.0) 2609 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.918100+0000 mon.c (mon.2) 759 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.918563+0000 mon.a (mon.0) 2610 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.919148+0000 mon.c (mon.2) 760 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.919506+0000 mon.a (mon.0) 2611 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.920032+0000 mon.c (mon.2) 761 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:18.920393+0000 mon.a (mon.0) 2612 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:19.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:19 smithi098 bash[23790]: audit 2024-04-02T04:54:19.188819+0000 mon.c (mon.2) 762 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:19.688 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: cluster 2024-04-02T04:54:18.090122+0000 mgr.y (mgr.14538) 419 : cluster 0 pgmap v604: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: cluster 2024-04-02T04:54:18.246320+0000 osd.7 (osd.7) 25 : cluster 0 2.8 deep-scrub starts 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: cluster 2024-04-02T04:54:18.248853+0000 osd.7 (osd.7) 26 : cluster 0 2.8 deep-scrub ok 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: cluster 2024-04-02T04:54:18.877422+0000 osd.6 (osd.6) 201 : cluster 0 4.1f scrub starts 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: cluster 2024-04-02T04:54:18.879216+0000 osd.6 (osd.6) 202 : cluster 0 4.1f scrub ok 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.889463+0000 mon.a (mon.0) 2608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: cluster 2024-04-02T04:54:18.893678+0000 mon.a (mon.0) 2609 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.918100+0000 mon.c (mon.2) 759 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.918563+0000 mon.a (mon.0) 2610 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.919148+0000 mon.c (mon.2) 760 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.919506+0000 mon.a (mon.0) 2611 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.920032+0000 mon.c (mon.2) 761 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:18.920393+0000 mon.a (mon.0) 2612 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[21528]: audit 2024-04-02T04:54:19.188819+0000 mon.c (mon.2) 762 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: cluster 2024-04-02T04:54:18.090122+0000 mgr.y (mgr.14538) 419 : cluster 0 pgmap v604: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: cluster 2024-04-02T04:54:18.246320+0000 osd.7 (osd.7) 25 : cluster 0 2.8 deep-scrub starts 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: cluster 2024-04-02T04:54:18.248853+0000 osd.7 (osd.7) 26 : cluster 0 2.8 deep-scrub ok 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: cluster 2024-04-02T04:54:18.877422+0000 osd.6 (osd.6) 201 : cluster 0 4.1f scrub starts 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: cluster 2024-04-02T04:54:18.879216+0000 osd.6 (osd.6) 202 : cluster 0 4.1f scrub ok 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.889463+0000 mon.a (mon.0) 2608 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-72153-64"}]': finished 2024-04-02T04:54:19.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: cluster 2024-04-02T04:54:18.893678+0000 mon.a (mon.0) 2609 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.918100+0000 mon.c (mon.2) 759 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.918563+0000 mon.a (mon.0) 2610 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.919148+0000 mon.c (mon.2) 760 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.919506+0000 mon.a (mon.0) 2611 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.920032+0000 mon.c (mon.2) 761 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:18.920393+0000 mon.a (mon.0) 2612 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:19.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:19 smithi069 bash[28904]: audit 2024-04-02T04:54:19.188819+0000 mon.c (mon.2) 762 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:19.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:19.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:19.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:19.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:20.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:20.073 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:20.156 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:20.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:20.204 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:20.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:20.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:20.431 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.198179+0000 osd.7 (osd.7) 27 : cluster 0 2.0 scrub starts 2024-04-02T04:54:20.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:20.462 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.198179+0000 osd.7 (osd.7) 27 : cluster 0 2.0 scrub starts 2024-04-02T04:54:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.199208+0000 osd.7 (osd.7) 28 : cluster 0 2.0 scrub ok 2024-04-02T04:54:20.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.200810+0000 osd.3 (osd.3) 271 : cluster 0 10.1b scrub starts 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.202174+0000 osd.3 (osd.3) 272 : cluster 0 10.1b scrub ok 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.357649+0000 osd.0 (osd.0) 219 : cluster 0 6.9 scrub starts 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.359774+0000 osd.0 (osd.0) 220 : cluster 0 6.9 scrub ok 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.537896+0000 osd.5 (osd.5) 233 : cluster 0 10.11 scrub starts 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.540340+0000 osd.5 (osd.5) 234 : cluster 0 10.11 scrub ok 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: audit 2024-04-02T04:54:19.875897+0000 mon.a (mon.0) 2613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: cluster 2024-04-02T04:54:19.885008+0000 mon.a (mon.0) 2614 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: audit 2024-04-02T04:54:19.887103+0000 mon.c (mon.2) 763 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: audit 2024-04-02T04:54:19.889377+0000 mon.a (mon.0) 2615 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: audit 2024-04-02T04:54:20.189765+0000 mon.c (mon.2) 764 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: audit 2024-04-02T04:54:20.308679+0000 mon.c (mon.2) 765 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:20 smithi098 bash[23790]: audit 2024-04-02T04:54:20.309371+0000 mon.a (mon.0) 2616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:20.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:20.596 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:20.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:20.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.199208+0000 osd.7 (osd.7) 28 : cluster 0 2.0 scrub ok 2024-04-02T04:54:20.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.200810+0000 osd.3 (osd.3) 271 : cluster 0 10.1b scrub starts 2024-04-02T04:54:20.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.202174+0000 osd.3 (osd.3) 272 : cluster 0 10.1b scrub ok 2024-04-02T04:54:20.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.357649+0000 osd.0 (osd.0) 219 : cluster 0 6.9 scrub starts 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.359774+0000 osd.0 (osd.0) 220 : cluster 0 6.9 scrub ok 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.537896+0000 osd.5 (osd.5) 233 : cluster 0 10.11 scrub starts 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.540340+0000 osd.5 (osd.5) 234 : cluster 0 10.11 scrub ok 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: audit 2024-04-02T04:54:19.875897+0000 mon.a (mon.0) 2613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: cluster 2024-04-02T04:54:19.885008+0000 mon.a (mon.0) 2614 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: audit 2024-04-02T04:54:19.887103+0000 mon.c (mon.2) 763 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: audit 2024-04-02T04:54:19.889377+0000 mon.a (mon.0) 2615 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: audit 2024-04-02T04:54:20.189765+0000 mon.c (mon.2) 764 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: audit 2024-04-02T04:54:20.308679+0000 mon.c (mon.2) 765 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[21528]: audit 2024-04-02T04:54:20.309371+0000 mon.a (mon.0) 2616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:20.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.198179+0000 osd.7 (osd.7) 27 : cluster 0 2.0 scrub starts 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.199208+0000 osd.7 (osd.7) 28 : cluster 0 2.0 scrub ok 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.200810+0000 osd.3 (osd.3) 271 : cluster 0 10.1b scrub starts 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.202174+0000 osd.3 (osd.3) 272 : cluster 0 10.1b scrub ok 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.357649+0000 osd.0 (osd.0) 219 : cluster 0 6.9 scrub starts 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.359774+0000 osd.0 (osd.0) 220 : cluster 0 6.9 scrub ok 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.537896+0000 osd.5 (osd.5) 233 : cluster 0 10.11 scrub starts 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.540340+0000 osd.5 (osd.5) 234 : cluster 0 10.11 scrub ok 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: audit 2024-04-02T04:54:19.875897+0000 mon.a (mon.0) 2613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-72153-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: cluster 2024-04-02T04:54:19.885008+0000 mon.a (mon.0) 2614 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: audit 2024-04-02T04:54:19.887103+0000 mon.c (mon.2) 763 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: audit 2024-04-02T04:54:19.889377+0000 mon.a (mon.0) 2615 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: audit 2024-04-02T04:54:20.189765+0000 mon.c (mon.2) 764 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: audit 2024-04-02T04:54:20.308679+0000 mon.c (mon.2) 765 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:20.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:20 smithi069 bash[28904]: audit 2024-04-02T04:54:20.309371+0000 mon.a (mon.0) 2616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:20.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:20.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:20.855 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:20.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:20.984 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:20.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:21.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:21.111 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:21.174 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:21.175 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:21.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:21.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:21.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:21.359 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:21.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: cluster 2024-04-02T04:54:20.091028+0000 mgr.y (mgr.14538) 420 : cluster 0 pgmap v607: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:21.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: cluster 2024-04-02T04:54:20.556166+0000 osd.5 (osd.5) 235 : cluster 0 41.18 scrub starts 2024-04-02T04:54:21.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: cluster 2024-04-02T04:54:20.558406+0000 osd.5 (osd.5) 236 : cluster 0 41.18 scrub ok 2024-04-02T04:54:21.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: audit 2024-04-02T04:54:20.879399+0000 mon.a (mon.0) 2617 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:21.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: cluster 2024-04-02T04:54:20.889246+0000 osd.6 (osd.6) 203 : cluster 0 10.b scrub starts 2024-04-02T04:54:21.462 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:21.488 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:21.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:21.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: audit 2024-04-02T04:54:20.889868+0000 mon.c (mon.2) 766 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: cluster 2024-04-02T04:54:20.898824+0000 osd.6 (osd.6) 204 : cluster 0 10.b scrub ok 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: audit 2024-04-02T04:54:21.190764+0000 mon.c (mon.2) 767 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.342232+0000 mon.a (mon.0) 2618 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[21528]: audit 2024-04-02T04:54:21.343881+0000 mon.a (mon.0) 2619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: cluster 2024-04-02T04:54:20.091028+0000 mgr.y (mgr.14538) 420 : cluster 0 pgmap v607: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: cluster 2024-04-02T04:54:20.556166+0000 osd.5 (osd.5) 235 : cluster 0 41.18 scrub starts 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: cluster 2024-04-02T04:54:20.558406+0000 osd.5 (osd.5) 236 : cluster 0 41.18 scrub ok 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: audit 2024-04-02T04:54:20.879399+0000 mon.a (mon.0) 2617 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: cluster 2024-04-02T04:54:20.889246+0000 osd.6 (osd.6) 203 : cluster 0 10.b scrub starts 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: audit 2024-04-02T04:54:20.889868+0000 mon.c (mon.2) 766 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:21.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: cluster 2024-04-02T04:54:20.898824+0000 osd.6 (osd.6) 204 : cluster 0 10.b scrub ok 2024-04-02T04:54:21.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: audit 2024-04-02T04:54:21.190764+0000 mon.c (mon.2) 767 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:21.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.342232+0000 mon.a (mon.0) 2618 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-04-02T04:54:21.604 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:21 smithi069 bash[28904]: audit 2024-04-02T04:54:21.343881+0000 mon.a (mon.0) 2619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:21.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:21.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: cluster 2024-04-02T04:54:20.091028+0000 mgr.y (mgr.14538) 420 : cluster 0 pgmap v607: 356 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: cluster 2024-04-02T04:54:20.556166+0000 osd.5 (osd.5) 235 : cluster 0 41.18 scrub starts 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: cluster 2024-04-02T04:54:20.558406+0000 osd.5 (osd.5) 236 : cluster 0 41.18 scrub ok 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: audit 2024-04-02T04:54:20.879399+0000 mon.a (mon.0) 2617 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: cluster 2024-04-02T04:54:20.889246+0000 osd.6 (osd.6) 203 : cluster 0 10.b scrub starts 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: audit 2024-04-02T04:54:20.889868+0000 mon.c (mon.2) 766 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: cluster 2024-04-02T04:54:20.898824+0000 osd.6 (osd.6) 204 : cluster 0 10.b scrub ok 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: audit 2024-04-02T04:54:21.190764+0000 mon.c (mon.2) 767 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.342232+0000 mon.a (mon.0) 2618 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-04-02T04:54:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:21 smithi098 bash[23790]: audit 2024-04-02T04:54:21.343881+0000 mon.a (mon.0) 2619 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:21.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:21.752 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:21.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:21.881 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:21.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:21.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:22.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:22.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:22.148 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:22.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:22.277 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:22.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:22.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:22.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:22.539 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:22.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:22.667 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:22.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.137877+0000 osd.3 (osd.3) 273 : cluster 0 3.8 scrub starts 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.140545+0000 osd.3 (osd.3) 274 : cluster 0 3.8 scrub ok 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.207779+0000 osd.1 (osd.1) 245 : cluster 0 10.13 scrub starts 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.209407+0000 osd.1 (osd.1) 246 : cluster 0 10.13 scrub ok 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.880170+0000 mon.a (mon.0) 2620 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:21.885378+0000 mon.a (mon.0) 2621 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:21.885628+0000 mon.a (mon.0) 2622 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]': finished 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: cluster 2024-04-02T04:54:21.892212+0000 mon.a (mon.0) 2623 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:21.937727+0000 mon.c (mon.2) 768 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:21.938389+0000 mon.a (mon.0) 2624 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:21.939457+0000 mon.c (mon.2) 769 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:21.940111+0000 mon.a (mon.0) 2625 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:22.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:22 smithi098 bash[23790]: audit 2024-04-02T04:54:22.191947+0000 mon.c (mon.2) 770 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:22.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:22.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.137877+0000 osd.3 (osd.3) 273 : cluster 0 3.8 scrub starts 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.140545+0000 osd.3 (osd.3) 274 : cluster 0 3.8 scrub ok 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.207779+0000 osd.1 (osd.1) 245 : cluster 0 10.13 scrub starts 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.209407+0000 osd.1 (osd.1) 246 : cluster 0 10.13 scrub ok 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.880170+0000 mon.a (mon.0) 2620 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:21.885378+0000 mon.a (mon.0) 2621 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:21.885628+0000 mon.a (mon.0) 2622 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]': finished 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: cluster 2024-04-02T04:54:21.892212+0000 mon.a (mon.0) 2623 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:21.937727+0000 mon.c (mon.2) 768 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:22.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:21.938389+0000 mon.a (mon.0) 2624 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:21.939457+0000 mon.c (mon.2) 769 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:21.940111+0000 mon.a (mon.0) 2625 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[21528]: audit 2024-04-02T04:54:22.191947+0000 mon.c (mon.2) 770 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.137877+0000 osd.3 (osd.3) 273 : cluster 0 3.8 scrub starts 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.140545+0000 osd.3 (osd.3) 274 : cluster 0 3.8 scrub ok 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.207779+0000 osd.1 (osd.1) 245 : cluster 0 10.13 scrub starts 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.209407+0000 osd.1 (osd.1) 246 : cluster 0 10.13 scrub ok 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.880170+0000 mon.a (mon.0) 2620 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:21.885378+0000 mon.a (mon.0) 2621 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-72153-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:22.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:21.885628+0000 mon.a (mon.0) 2622 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]': finished 2024-04-02T04:54:22.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: cluster 2024-04-02T04:54:21.892212+0000 mon.a (mon.0) 2623 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-04-02T04:54:22.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:21.937727+0000 mon.c (mon.2) 768 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:22.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:21.938389+0000 mon.a (mon.0) 2624 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:22.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:21.939457+0000 mon.c (mon.2) 769 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:22.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:21.940111+0000 mon.a (mon.0) 2625 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-49"}]: dispatch 2024-04-02T04:54:22.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:22 smithi069 bash[28904]: audit 2024-04-02T04:54:22.191947+0000 mon.c (mon.2) 770 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:22.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:22.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2:163:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:5d165639:::164:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f43765fc:::165:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b4c720e9:::166:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e694b040:::167:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:afa38db2:::168:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:77ba9f53:::169:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:87495034:::170:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7c96bf0e:::171:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:dbe346cc:::172:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:e943ec24:::173:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f97a9c0c:::174:head 2024-04-02T04:54:22.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:6f26e74d:::175:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4f95e106:::176:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:0e6f2f8f:::177:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:05db05f1:::178:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:38a78d66:::179:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:d095610b:::180:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:a1a9d709:::181:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:1e5d39db:::182:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:f7df4fb9:::183:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:03a7f161:::184:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ba70721e:::185:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:28e5662d:::186:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:973d52de:::187:head 2024-04-02T04:54:22.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:4303eb1c:::188:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:b990b48e:::189:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:29b8165b:::190:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:3547f197:::191:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:7e260936:::192:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:1abec7b1:::193:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:10fdda93:::194:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:15817eea:::195:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:770bab57:::196:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:ed9e13e7:::197:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:71471a8f:::198:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 274:10fb1d02:::199:head 2024-04-02T04:54:22.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7975 ms) 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1712033617 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,1712033622,1712033623,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,1712033622,1712033623,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,1712033622,1712033623,0 2024-04-02T04:54:22.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,1712033622,1712033623,1712033625,1712033626,0 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,1712033622,1712033623,1712033625,1712033626,0 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033617,1712033619,1712033620,1712033622,1712033623,1712033625,1712033626,0 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712033620,1712033622,1712033623,1712033625,1712033626,1712033628,1712033629,0 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1712033620, trimmed 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21035 ms) 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13967 ms) 2024-04-02T04:54:22.911 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-02T04:54:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:22.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:23.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:23.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:23.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:23.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:23.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:23.325 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:54:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:54:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:23.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:23.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:23.590 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:23.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:23.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[28904]: cluster 2024-04-02T04:54:22.092051+0000 mgr.y (mgr.14538) 421 : cluster 0 pgmap v610: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:23.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[28904]: cluster 2024-04-02T04:54:22.885421+0000 mon.a (mon.0) 2626 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:23.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[28904]: cluster 2024-04-02T04:54:22.909845+0000 mon.a (mon.0) 2627 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-04-02T04:54:23.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[28904]: audit 2024-04-02T04:54:23.193071+0000 mon.c (mon.2) 771 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:23.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[21528]: cluster 2024-04-02T04:54:22.092051+0000 mgr.y (mgr.14538) 421 : cluster 0 pgmap v610: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:23.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[21528]: cluster 2024-04-02T04:54:22.885421+0000 mon.a (mon.0) 2626 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:23.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[21528]: cluster 2024-04-02T04:54:22.909845+0000 mon.a (mon.0) 2627 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-04-02T04:54:23.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:23 smithi069 bash[21528]: audit 2024-04-02T04:54:23.193071+0000 mon.c (mon.2) 771 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:23.718 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:23.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:23 smithi098 bash[23790]: cluster 2024-04-02T04:54:22.092051+0000 mgr.y (mgr.14538) 421 : cluster 0 pgmap v610: 364 pgs: 8 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 352 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:23.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:23 smithi098 bash[23790]: cluster 2024-04-02T04:54:22.885421+0000 mon.a (mon.0) 2626 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:23.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:23 smithi098 bash[23790]: cluster 2024-04-02T04:54:22.909845+0000 mon.a (mon.0) 2627 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-04-02T04:54:23.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:23 smithi098 bash[23790]: audit 2024-04-02T04:54:23.193071+0000 mon.c (mon.2) 771 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:23.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:23.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:23.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:24.077 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:24.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:24.234 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:24.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:24.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:24.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:22.970268+0000 osd.2 (osd.2) 196 : cluster 0 102.11 scrub starts 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:22.972420+0000 osd.2 (osd.2) 197 : cluster 0 102.11 scrub ok 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:23.113306+0000 osd.3 (osd.3) 275 : cluster 0 43.16 deep-scrub starts 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:23.115285+0000 osd.3 (osd.3) 276 : cluster 0 43.16 deep-scrub ok 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: audit 2024-04-02T04:54:23.252489+0000 mgr.y (mgr.14538) 422 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:23.367225+0000 osd.0 (osd.0) 221 : cluster 0 2.1f scrub starts 2024-04-02T04:54:24.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:23.368267+0000 osd.0 (osd.0) 222 : cluster 0 2.1f scrub ok 2024-04-02T04:54:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: cluster 2024-04-02T04:54:23.901693+0000 mon.a (mon.0) 2628 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-04-02T04:54:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: audit 2024-04-02T04:54:23.907067+0000 mon.c (mon.2) 772 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: audit 2024-04-02T04:54:23.907861+0000 mon.a (mon.0) 2629 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: audit 2024-04-02T04:54:23.907966+0000 mon.c (mon.2) 773 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: audit 2024-04-02T04:54:23.908891+0000 mon.a (mon.0) 2630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:24 smithi098 bash[23790]: audit 2024-04-02T04:54:24.194702+0000 mon.c (mon.2) 774 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:24.495 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:24.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:24.626 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:24.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:24.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:22.970268+0000 osd.2 (osd.2) 196 : cluster 0 102.11 scrub starts 2024-04-02T04:54:24.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:22.972420+0000 osd.2 (osd.2) 197 : cluster 0 102.11 scrub ok 2024-04-02T04:54:24.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:23.113306+0000 osd.3 (osd.3) 275 : cluster 0 43.16 deep-scrub starts 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:23.115285+0000 osd.3 (osd.3) 276 : cluster 0 43.16 deep-scrub ok 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: audit 2024-04-02T04:54:23.252489+0000 mgr.y (mgr.14538) 422 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:23.367225+0000 osd.0 (osd.0) 221 : cluster 0 2.1f scrub starts 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:23.368267+0000 osd.0 (osd.0) 222 : cluster 0 2.1f scrub ok 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: cluster 2024-04-02T04:54:23.901693+0000 mon.a (mon.0) 2628 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: audit 2024-04-02T04:54:23.907067+0000 mon.c (mon.2) 772 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: audit 2024-04-02T04:54:23.907861+0000 mon.a (mon.0) 2629 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: audit 2024-04-02T04:54:23.907966+0000 mon.c (mon.2) 773 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: audit 2024-04-02T04:54:23.908891+0000 mon.a (mon.0) 2630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[28904]: audit 2024-04-02T04:54:24.194702+0000 mon.c (mon.2) 774 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:22.970268+0000 osd.2 (osd.2) 196 : cluster 0 102.11 scrub starts 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:22.972420+0000 osd.2 (osd.2) 197 : cluster 0 102.11 scrub ok 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:23.113306+0000 osd.3 (osd.3) 275 : cluster 0 43.16 deep-scrub starts 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:23.115285+0000 osd.3 (osd.3) 276 : cluster 0 43.16 deep-scrub ok 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: audit 2024-04-02T04:54:23.252489+0000 mgr.y (mgr.14538) 422 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:23.367225+0000 osd.0 (osd.0) 221 : cluster 0 2.1f scrub starts 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:23.368267+0000 osd.0 (osd.0) 222 : cluster 0 2.1f scrub ok 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: cluster 2024-04-02T04:54:23.901693+0000 mon.a (mon.0) 2628 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: audit 2024-04-02T04:54:23.907067+0000 mon.c (mon.2) 772 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: audit 2024-04-02T04:54:23.907861+0000 mon.a (mon.0) 2629 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: audit 2024-04-02T04:54:23.907966+0000 mon.c (mon.2) 773 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: audit 2024-04-02T04:54:23.908891+0000 mon.a (mon.0) 2630 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:24.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:24 smithi069 bash[21528]: audit 2024-04-02T04:54:24.194702+0000 mon.c (mon.2) 774 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:24.755 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:24.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:24.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:24.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:25.018 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:25.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:25.149 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ : started 0x55c99fcc08d0 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0d60 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1210 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1320 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1920 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0e70 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc0fc0 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc10d0 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1a30 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1470 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbab30 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbac80 2024-04-02T04:54:25.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcbadb0 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2010 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1b60 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1c90 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1da0 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1ed0 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2610 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2160 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc22b0 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc23e0 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2760 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2c50 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2da0 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2890 2024-04-02T04:54:25.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc29e0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2b10 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2ed0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3380 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc34b0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3a70 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc2fe0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3130 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc35c0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1580 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3260 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc16d0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc1800 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3bc0 2024-04-02T04:54:25.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3d10 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4310 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3e60 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3f90 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc40e0 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4420 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc48f0 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4a20 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5000 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4550 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4680 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc47d0 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5150 2024-04-02T04:54:25.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4b50 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4c80 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4db0 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc4ee0 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5730 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5280 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc36d0 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3820 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc3930 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5d30 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5880 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc59d0 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5b00 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5e80 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6330 2024-04-02T04:54:25.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6460 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6a40 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5f90 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc60e0 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc61f0 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6b90 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6590 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc66c0 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc67f0 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6920 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7190 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6ce0 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6e10 2024-04-02T04:54:25.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc6f20 2024-04-02T04:54:25.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:25.278 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:25.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.019898+0000 osd.2 (osd.2) 198 : cluster 0 102.17 scrub starts 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.021806+0000 osd.2 (osd.2) 199 : cluster 0 102.17 scrub ok 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.093492+0000 mgr.y (mgr.14538) 423 : cluster 0 pgmap v613: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 24 unknown, 1 active+clean+snaptrim_wait, 321 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.095425+0000 osd.3 (osd.3) 277 : cluster 0 41.b scrub starts 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.097030+0000 osd.3 (osd.3) 278 : cluster 0 41.b scrub ok 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.233919+0000 osd.7 (osd.7) 29 : cluster 0 13.7 scrub starts 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.235458+0000 osd.7 (osd.7) 30 : cluster 0 13.7 scrub ok 2024-04-02T04:54:25.440 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.384777+0000 osd.0 (osd.0) 223 : cluster 0 6.18 scrub starts 2024-04-02T04:54:25.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:25.522 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.386315+0000 osd.0 (osd.0) 224 : cluster 0 6.18 scrub ok 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:24.878650+0000 mon.a (mon.0) 2631 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:24.878883+0000 mon.a (mon.0) 2632 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: cluster 2024-04-02T04:54:24.883379+0000 mon.a (mon.0) 2633 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:24.883736+0000 mon.c (mon.2) 775 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:24.885787+0000 mon.a (mon.0) 2634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:24.953834+0000 mon.c (mon.2) 776 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:24.954177+0000 mon.a (mon.0) 2635 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:25.523 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:25 smithi098 bash[23790]: audit 2024-04-02T04:54:25.196408+0000 mon.c (mon.2) 777 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:25.538 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:25.613 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.019898+0000 osd.2 (osd.2) 198 : cluster 0 102.17 scrub starts 2024-04-02T04:54:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.021806+0000 osd.2 (osd.2) 199 : cluster 0 102.17 scrub ok 2024-04-02T04:54:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.093492+0000 mgr.y (mgr.14538) 423 : cluster 0 pgmap v613: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 24 unknown, 1 active+clean+snaptrim_wait, 321 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.095425+0000 osd.3 (osd.3) 277 : cluster 0 41.b scrub starts 2024-04-02T04:54:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.097030+0000 osd.3 (osd.3) 278 : cluster 0 41.b scrub ok 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.233919+0000 osd.7 (osd.7) 29 : cluster 0 13.7 scrub starts 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.235458+0000 osd.7 (osd.7) 30 : cluster 0 13.7 scrub ok 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.384777+0000 osd.0 (osd.0) 223 : cluster 0 6.18 scrub starts 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.386315+0000 osd.0 (osd.0) 224 : cluster 0 6.18 scrub ok 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:24.878650+0000 mon.a (mon.0) 2631 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:24.878883+0000 mon.a (mon.0) 2632 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: cluster 2024-04-02T04:54:24.883379+0000 mon.a (mon.0) 2633 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:24.883736+0000 mon.c (mon.2) 775 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:24.885787+0000 mon.a (mon.0) 2634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:24.953834+0000 mon.c (mon.2) 776 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:24.954177+0000 mon.a (mon.0) 2635 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:25.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[21528]: audit 2024-04-02T04:54:25.196408+0000 mon.c (mon.2) 777 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.019898+0000 osd.2 (osd.2) 198 : cluster 0 102.17 scrub starts 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.021806+0000 osd.2 (osd.2) 199 : cluster 0 102.17 scrub ok 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.093492+0000 mgr.y (mgr.14538) 423 : cluster 0 pgmap v613: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 24 unknown, 1 active+clean+snaptrim_wait, 321 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.095425+0000 osd.3 (osd.3) 277 : cluster 0 41.b scrub starts 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.097030+0000 osd.3 (osd.3) 278 : cluster 0 41.b scrub ok 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.233919+0000 osd.7 (osd.7) 29 : cluster 0 13.7 scrub starts 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.235458+0000 osd.7 (osd.7) 30 : cluster 0 13.7 scrub ok 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.384777+0000 osd.0 (osd.0) 223 : cluster 0 6.18 scrub starts 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.386315+0000 osd.0 (osd.0) 224 : cluster 0 6.18 scrub ok 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:24.878650+0000 mon.a (mon.0) 2631 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:25.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:24.878883+0000 mon.a (mon.0) 2632 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:25.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: cluster 2024-04-02T04:54:24.883379+0000 mon.a (mon.0) 2633 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-04-02T04:54:25.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:24.883736+0000 mon.c (mon.2) 775 : audit 1 from='client.? 172.21.15.69:0/812125742' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:25.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:24.885787+0000 mon.a (mon.0) 2634 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]: dispatch 2024-04-02T04:54:25.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:24.953834+0000 mon.c (mon.2) 776 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:25.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:24.954177+0000 mon.a (mon.0) 2635 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:54:25.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:25 smithi069 bash[28904]: audit 2024-04-02T04:54:25.196408+0000 mon.c (mon.2) 777 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:25.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:25.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:25.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:25.837 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:25.940 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:25.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:25.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:26.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:26.091 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:26.194 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:26.220 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:26.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:26.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:26.453 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:26.478 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:26.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.198466+0000 osd.1 (osd.1) 247 : cluster 0 43.f scrub starts 2024-04-02T04:54:26.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.202333+0000 osd.1 (osd.1) 248 : cluster 0 43.f scrub ok 2024-04-02T04:54:26.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.265928+0000 osd.7 (osd.7) 31 : cluster 0 6.12 scrub starts 2024-04-02T04:54:26.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.267622+0000 osd.7 (osd.7) 32 : cluster 0 6.12 scrub ok 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.601413+0000 osd.5 (osd.5) 237 : cluster 0 3.16 scrub starts 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.604409+0000 osd.5 (osd.5) 238 : cluster 0 3.16 scrub ok 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.880793+0000 mon.a (mon.0) 2636 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.880890+0000 mon.a (mon.0) 2637 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: cluster 2024-04-02T04:54:25.885344+0000 mon.a (mon.0) 2638 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.890453+0000 mon.c (mon.2) 778 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.890978+0000 mon.a (mon.0) 2639 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.911840+0000 mon.a (mon.0) 2640 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.912364+0000 mon.a (mon.0) 2641 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:25.912729+0000 mon.a (mon.0) 2642 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-72153-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[28904]: audit 2024-04-02T04:54:26.198180+0000 mon.c (mon.2) 779 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:26.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.198466+0000 osd.1 (osd.1) 247 : cluster 0 43.f scrub starts 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.202333+0000 osd.1 (osd.1) 248 : cluster 0 43.f scrub ok 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.265928+0000 osd.7 (osd.7) 31 : cluster 0 6.12 scrub starts 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.267622+0000 osd.7 (osd.7) 32 : cluster 0 6.12 scrub ok 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.601413+0000 osd.5 (osd.5) 237 : cluster 0 3.16 scrub starts 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.604409+0000 osd.5 (osd.5) 238 : cluster 0 3.16 scrub ok 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.880793+0000 mon.a (mon.0) 2636 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.880890+0000 mon.a (mon.0) 2637 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: cluster 2024-04-02T04:54:25.885344+0000 mon.a (mon.0) 2638 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.890453+0000 mon.c (mon.2) 778 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.890978+0000 mon.a (mon.0) 2639 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.911840+0000 mon.a (mon.0) 2640 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.912364+0000 mon.a (mon.0) 2641 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:25.912729+0000 mon.a (mon.0) 2642 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-72153-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:26.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:26 smithi069 bash[21528]: audit 2024-04-02T04:54:26.198180+0000 mon.c (mon.2) 779 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:26.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:26.608 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:26.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:26.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.198466+0000 osd.1 (osd.1) 247 : cluster 0 43.f scrub starts 2024-04-02T04:54:26.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.202333+0000 osd.1 (osd.1) 248 : cluster 0 43.f scrub ok 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.265928+0000 osd.7 (osd.7) 31 : cluster 0 6.12 scrub starts 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.267622+0000 osd.7 (osd.7) 32 : cluster 0 6.12 scrub ok 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.601413+0000 osd.5 (osd.5) 237 : cluster 0 3.16 scrub starts 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.604409+0000 osd.5 (osd.5) 238 : cluster 0 3.16 scrub ok 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.880793+0000 mon.a (mon.0) 2636 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-72153-65"}]': finished 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.880890+0000 mon.a (mon.0) 2637 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: cluster 2024-04-02T04:54:25.885344+0000 mon.a (mon.0) 2638 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.890453+0000 mon.c (mon.2) 778 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.890978+0000 mon.a (mon.0) 2639 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.911840+0000 mon.a (mon.0) 2640 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.912364+0000 mon.a (mon.0) 2641 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:25.912729+0000 mon.a (mon.0) 2642 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-72153-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:54:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:26 smithi098 bash[23790]: audit 2024-04-02T04:54:26.198180+0000 mon.c (mon.2) 779 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:26.739 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:26.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:26.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:26.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:26.997 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:26.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:27.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:27.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:27.251 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:27.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:27.381 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:27.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:27.512 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:27.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:27.642 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.094527+0000 mgr.y (mgr.14538) 424 : cluster 0 pgmap v616: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 24 unknown, 1 active+clean+snaptrim_wait, 321 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.229386+0000 osd.1 (osd.1) 249 : cluster 0 43.5 scrub starts 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.233987+0000 osd.1 (osd.1) 250 : cluster 0 43.5 scrub ok 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.264050+0000 osd.7 (osd.7) 33 : cluster 0 13.18 scrub starts 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.267085+0000 osd.7 (osd.7) 34 : cluster 0 13.18 scrub ok 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.391756+0000 osd.0 (osd.0) 225 : cluster 0 6.16 deep-scrub starts 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.393392+0000 osd.0 (osd.0) 226 : cluster 0 6.16 deep-scrub ok 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.641256+0000 osd.5 (osd.5) 239 : cluster 0 41.11 scrub starts 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.642603+0000 osd.5 (osd.5) 240 : cluster 0 41.11 scrub ok 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.866184+0000 osd.6 (osd.6) 205 : cluster 0 10.d scrub starts 2024-04-02T04:54:27.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.868516+0000 osd.6 (osd.6) 206 : cluster 0 10.d scrub ok 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: audit 2024-04-02T04:54:26.884701+0000 mon.a (mon.0) 2643 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]': finished 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: audit 2024-04-02T04:54:26.884797+0000 mon.a (mon.0) 2644 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-72153-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: audit 2024-04-02T04:54:26.890491+0000 mon.c (mon.2) 780 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]: dispatch 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.893947+0000 mon.a (mon.0) 2645 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: audit 2024-04-02T04:54:26.896624+0000 mon.a (mon.0) 2646 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-72153-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: audit 2024-04-02T04:54:26.897233+0000 mon.a (mon.0) 2647 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]: dispatch 2024-04-02T04:54:27.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:27 smithi098 bash[23790]: audit 2024-04-02T04:54:27.199589+0000 mon.c (mon.2) 781 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:27.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.094527+0000 mgr.y (mgr.14538) 424 : cluster 0 pgmap v616: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 24 unknown, 1 active+clean+snaptrim_wait, 321 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.229386+0000 osd.1 (osd.1) 249 : cluster 0 43.5 scrub starts 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.233987+0000 osd.1 (osd.1) 250 : cluster 0 43.5 scrub ok 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.264050+0000 osd.7 (osd.7) 33 : cluster 0 13.18 scrub starts 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.267085+0000 osd.7 (osd.7) 34 : cluster 0 13.18 scrub ok 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.391756+0000 osd.0 (osd.0) 225 : cluster 0 6.16 deep-scrub starts 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.393392+0000 osd.0 (osd.0) 226 : cluster 0 6.16 deep-scrub ok 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.641256+0000 osd.5 (osd.5) 239 : cluster 0 41.11 scrub starts 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.642603+0000 osd.5 (osd.5) 240 : cluster 0 41.11 scrub ok 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.866184+0000 osd.6 (osd.6) 205 : cluster 0 10.d scrub starts 2024-04-02T04:54:27.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.868516+0000 osd.6 (osd.6) 206 : cluster 0 10.d scrub ok 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: audit 2024-04-02T04:54:26.884701+0000 mon.a (mon.0) 2643 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]': finished 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: audit 2024-04-02T04:54:26.884797+0000 mon.a (mon.0) 2644 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-72153-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: audit 2024-04-02T04:54:26.890491+0000 mon.c (mon.2) 780 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]: dispatch 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.893947+0000 mon.a (mon.0) 2645 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: audit 2024-04-02T04:54:26.896624+0000 mon.a (mon.0) 2646 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-72153-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: audit 2024-04-02T04:54:26.897233+0000 mon.a (mon.0) 2647 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]: dispatch 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[21528]: audit 2024-04-02T04:54:27.199589+0000 mon.c (mon.2) 781 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.094527+0000 mgr.y (mgr.14538) 424 : cluster 0 pgmap v616: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 24 unknown, 1 active+clean+snaptrim_wait, 321 active+clean; 8.4 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.229386+0000 osd.1 (osd.1) 249 : cluster 0 43.5 scrub starts 2024-04-02T04:54:27.757 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.233987+0000 osd.1 (osd.1) 250 : cluster 0 43.5 scrub ok 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.264050+0000 osd.7 (osd.7) 33 : cluster 0 13.18 scrub starts 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.267085+0000 osd.7 (osd.7) 34 : cluster 0 13.18 scrub ok 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.391756+0000 osd.0 (osd.0) 225 : cluster 0 6.16 deep-scrub starts 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.393392+0000 osd.0 (osd.0) 226 : cluster 0 6.16 deep-scrub ok 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.641256+0000 osd.5 (osd.5) 239 : cluster 0 41.11 scrub starts 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.642603+0000 osd.5 (osd.5) 240 : cluster 0 41.11 scrub ok 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.866184+0000 osd.6 (osd.6) 205 : cluster 0 10.d scrub starts 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.868516+0000 osd.6 (osd.6) 206 : cluster 0 10.d scrub ok 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: audit 2024-04-02T04:54:26.884701+0000 mon.a (mon.0) 2643 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-51"}]': finished 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: audit 2024-04-02T04:54:26.884797+0000 mon.a (mon.0) 2644 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-72153-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: audit 2024-04-02T04:54:26.890491+0000 mon.c (mon.2) 780 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]: dispatch 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.893947+0000 mon.a (mon.0) 2645 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-04-02T04:54:27.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: audit 2024-04-02T04:54:26.896624+0000 mon.a (mon.0) 2646 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-72153-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:27.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: audit 2024-04-02T04:54:26.897233+0000 mon.a (mon.0) 2647 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]: dispatch 2024-04-02T04:54:27.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:27 smithi069 bash[28904]: audit 2024-04-02T04:54:27.199589+0000 mon.c (mon.2) 781 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:27.770 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:27.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:27.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:27.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:28.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:28.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:28.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:28.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:28.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:28.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:28.416 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:28.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.978148+0000 osd.2 (osd.2) 200 : cluster 0 6.f scrub starts 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:26.980656+0000 osd.2 (osd.2) 201 : cluster 0 6.f scrub ok 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.012863+0000 osd.3 (osd.3) 279 : cluster 0 6.11 scrub starts 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.015299+0000 osd.3 (osd.3) 280 : cluster 0 6.11 scrub ok 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.404947+0000 osd.0 (osd.0) 227 : cluster 0 3.1b scrub starts 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.408015+0000 osd.0 (osd.0) 228 : cluster 0 3.1b scrub ok 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.660652+0000 osd.5 (osd.5) 241 : cluster 0 41.1d scrub starts 2024-04-02T04:54:28.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.663039+0000 osd.5 (osd.5) 242 : cluster 0 41.1d scrub ok 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:27.885710+0000 mon.a (mon.0) 2648 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: audit 2024-04-02T04:54:27.888796+0000 mon.a (mon.0) 2649 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]': finished 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: audit 2024-04-02T04:54:27.982039+0000 mon.c (mon.2) 782 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: audit 2024-04-02T04:54:28.200715+0000 mon.c (mon.2) 783 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.328059+0000 mon.a (mon.0) 2650 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[28904]: audit 2024-04-02T04:54:28.329506+0000 mon.a (mon.0) 2651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.978148+0000 osd.2 (osd.2) 200 : cluster 0 6.f scrub starts 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:26.980656+0000 osd.2 (osd.2) 201 : cluster 0 6.f scrub ok 2024-04-02T04:54:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.012863+0000 osd.3 (osd.3) 279 : cluster 0 6.11 scrub starts 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.015299+0000 osd.3 (osd.3) 280 : cluster 0 6.11 scrub ok 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.404947+0000 osd.0 (osd.0) 227 : cluster 0 3.1b scrub starts 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.408015+0000 osd.0 (osd.0) 228 : cluster 0 3.1b scrub ok 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.660652+0000 osd.5 (osd.5) 241 : cluster 0 41.1d scrub starts 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.663039+0000 osd.5 (osd.5) 242 : cluster 0 41.1d scrub ok 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:27.885710+0000 mon.a (mon.0) 2648 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: audit 2024-04-02T04:54:27.888796+0000 mon.a (mon.0) 2649 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]': finished 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: audit 2024-04-02T04:54:27.982039+0000 mon.c (mon.2) 782 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: audit 2024-04-02T04:54:28.200715+0000 mon.c (mon.2) 783 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.328059+0000 mon.a (mon.0) 2650 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-04-02T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21528]: audit 2024-04-02T04:54:28.329506+0000 mon.a (mon.0) 2651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:54:28.540 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:28.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:28.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:28.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.978148+0000 osd.2 (osd.2) 200 : cluster 0 6.f scrub starts 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:26.980656+0000 osd.2 (osd.2) 201 : cluster 0 6.f scrub ok 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.012863+0000 osd.3 (osd.3) 279 : cluster 0 6.11 scrub starts 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.015299+0000 osd.3 (osd.3) 280 : cluster 0 6.11 scrub ok 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.404947+0000 osd.0 (osd.0) 227 : cluster 0 3.1b scrub starts 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.408015+0000 osd.0 (osd.0) 228 : cluster 0 3.1b scrub ok 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.660652+0000 osd.5 (osd.5) 241 : cluster 0 41.1d scrub starts 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.663039+0000 osd.5 (osd.5) 242 : cluster 0 41.1d scrub ok 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:27.885710+0000 mon.a (mon.0) 2648 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: audit 2024-04-02T04:54:27.888796+0000 mon.a (mon.0) 2649 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-51", "mode": "writeback"}]': finished 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: audit 2024-04-02T04:54:27.982039+0000 mon.c (mon.2) 782 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: audit 2024-04-02T04:54:28.200715+0000 mon.c (mon.2) 783 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.328059+0000 mon.a (mon.0) 2650 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-04-02T04:54:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:28 smithi098 bash[23790]: audit 2024-04-02T04:54:28.329506+0000 mon.a (mon.0) 2651 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:54:28.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:28.797 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:28.876 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:54:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:54:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:54:28.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:28.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:29.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:29.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:29.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:29.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:29.324 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:29.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:29.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:29.585 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:29.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.097391+0000 mgr.y (mgr.14538) 425 : cluster 0 pgmap v619: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 1 active+clean+snaptrim_wait, 345 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.367445+0000 osd.0 (osd.0) 229 : cluster 0 13.14 scrub starts 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.369213+0000 osd.0 (osd.0) 230 : cluster 0 13.14 scrub ok 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.639410+0000 osd.5 (osd.5) 243 : cluster 0 40.19 scrub starts 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.641778+0000 osd.5 (osd.5) 244 : cluster 0 40.19 scrub ok 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.852006+0000 osd.6 (osd.6) 207 : cluster 0 10.a deep-scrub starts 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.854270+0000 osd.6 (osd.6) 208 : cluster 0 10.a deep-scrub ok 2024-04-02T04:54:29.700 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: audit 2024-04-02T04:54:28.898208+0000 mon.a (mon.0) 2652 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-72153-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: audit 2024-04-02T04:54:28.898488+0000 mon.a (mon.0) 2653 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.906063+0000 mon.a (mon.0) 2654 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: audit 2024-04-02T04:54:28.912204+0000 mon.c (mon.2) 784 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: audit 2024-04-02T04:54:28.924327+0000 mon.a (mon.0) 2655 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[28904]: audit 2024-04-02T04:54:29.202256+0000 mon.c (mon.2) 785 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.097391+0000 mgr.y (mgr.14538) 425 : cluster 0 pgmap v619: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 1 active+clean+snaptrim_wait, 345 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.367445+0000 osd.0 (osd.0) 229 : cluster 0 13.14 scrub starts 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.369213+0000 osd.0 (osd.0) 230 : cluster 0 13.14 scrub ok 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.639410+0000 osd.5 (osd.5) 243 : cluster 0 40.19 scrub starts 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.641778+0000 osd.5 (osd.5) 244 : cluster 0 40.19 scrub ok 2024-04-02T04:54:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.852006+0000 osd.6 (osd.6) 207 : cluster 0 10.a deep-scrub starts 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.854270+0000 osd.6 (osd.6) 208 : cluster 0 10.a deep-scrub ok 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: audit 2024-04-02T04:54:28.898208+0000 mon.a (mon.0) 2652 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-72153-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: audit 2024-04-02T04:54:28.898488+0000 mon.a (mon.0) 2653 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.906063+0000 mon.a (mon.0) 2654 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: audit 2024-04-02T04:54:28.912204+0000 mon.c (mon.2) 784 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: audit 2024-04-02T04:54:28.924327+0000 mon.a (mon.0) 2655 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:54:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:29 smithi069 bash[21528]: audit 2024-04-02T04:54:29.202256+0000 mon.c (mon.2) 785 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.097391+0000 mgr.y (mgr.14538) 425 : cluster 0 pgmap v619: 356 pgs: 2 active+clean+snaptrim, 8 creating+peering, 1 active+clean+snaptrim_wait, 345 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:54:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.367445+0000 osd.0 (osd.0) 229 : cluster 0 13.14 scrub starts 2024-04-02T04:54:29.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.369213+0000 osd.0 (osd.0) 230 : cluster 0 13.14 scrub ok 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.639410+0000 osd.5 (osd.5) 243 : cluster 0 40.19 scrub starts 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.641778+0000 osd.5 (osd.5) 244 : cluster 0 40.19 scrub ok 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.852006+0000 osd.6 (osd.6) 207 : cluster 0 10.a deep-scrub starts 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.854270+0000 osd.6 (osd.6) 208 : cluster 0 10.a deep-scrub ok 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: audit 2024-04-02T04:54:28.898208+0000 mon.a (mon.0) 2652 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-72153-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: audit 2024-04-02T04:54:28.898488+0000 mon.a (mon.0) 2653 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.906063+0000 mon.a (mon.0) 2654 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: audit 2024-04-02T04:54:28.912204+0000 mon.c (mon.2) 784 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: audit 2024-04-02T04:54:28.924327+0000 mon.a (mon.0) 2655 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:54:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:29 smithi098 bash[23790]: audit 2024-04-02T04:54:29.202256+0000 mon.c (mon.2) 785 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:29.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:29.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:29.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:29.972 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:29.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:30.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:30.105 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:30.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:30.236 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:30.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:30.363 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:30.467 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:30.491 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:30.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:30.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:30.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.732548+0000 osd.4 (osd.4) 241 : cluster 0 10.e scrub starts 2024-04-02T04:54:30.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:28.733826+0000 osd.4 (osd.4) 242 : cluster 0 10.e scrub ok 2024-04-02T04:54:30.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.877634+0000 mon.a (mon.0) 2656 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:30.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: audit 2024-04-02T04:54:29.880314+0000 mon.a (mon.0) 2657 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.884913+0000 mon.a (mon.0) 2658 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: audit 2024-04-02T04:54:29.886272+0000 mon.c (mon.2) 786 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: audit 2024-04-02T04:54:29.894644+0000 mon.a (mon.0) 2659 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.923317+0000 osd.6 (osd.6) 209 : cluster 0 10.f deep-scrub starts 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.941605+0000 osd.6 (osd.6) 210 : cluster 0 10.f deep-scrub ok 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.949843+0000 osd.2 (osd.2) 202 : cluster 0 102.18 scrub starts 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.951320+0000 osd.2 (osd.2) 203 : cluster 0 102.18 scrub ok 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[21528]: audit 2024-04-02T04:54:30.204001+0000 mon.c (mon.2) 787 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.732548+0000 osd.4 (osd.4) 241 : cluster 0 10.e scrub starts 2024-04-02T04:54:30.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:28.733826+0000 osd.4 (osd.4) 242 : cluster 0 10.e scrub ok 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.877634+0000 mon.a (mon.0) 2656 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: audit 2024-04-02T04:54:29.880314+0000 mon.a (mon.0) 2657 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.884913+0000 mon.a (mon.0) 2658 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: audit 2024-04-02T04:54:29.886272+0000 mon.c (mon.2) 786 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: audit 2024-04-02T04:54:29.894644+0000 mon.a (mon.0) 2659 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.923317+0000 osd.6 (osd.6) 209 : cluster 0 10.f deep-scrub starts 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.941605+0000 osd.6 (osd.6) 210 : cluster 0 10.f deep-scrub ok 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.949843+0000 osd.2 (osd.2) 202 : cluster 0 102.18 scrub starts 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.951320+0000 osd.2 (osd.2) 203 : cluster 0 102.18 scrub ok 2024-04-02T04:54:30.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:30 smithi069 bash[28904]: audit 2024-04-02T04:54:30.204001+0000 mon.c (mon.2) 787 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:30.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:30.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.732548+0000 osd.4 (osd.4) 241 : cluster 0 10.e scrub starts 2024-04-02T04:54:30.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:28.733826+0000 osd.4 (osd.4) 242 : cluster 0 10.e scrub ok 2024-04-02T04:54:30.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.877634+0000 mon.a (mon.0) 2656 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:30.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: audit 2024-04-02T04:54:29.880314+0000 mon.a (mon.0) 2657 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:54:30.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.884913+0000 mon.a (mon.0) 2658 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-04-02T04:54:30.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: audit 2024-04-02T04:54:29.886272+0000 mon.c (mon.2) 786 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:54:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: audit 2024-04-02T04:54:29.894644+0000 mon.a (mon.0) 2659 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:54:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.923317+0000 osd.6 (osd.6) 209 : cluster 0 10.f deep-scrub starts 2024-04-02T04:54:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.941605+0000 osd.6 (osd.6) 210 : cluster 0 10.f deep-scrub ok 2024-04-02T04:54:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.949843+0000 osd.2 (osd.2) 202 : cluster 0 102.18 scrub starts 2024-04-02T04:54:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.951320+0000 osd.2 (osd.2) 203 : cluster 0 102.18 scrub ok 2024-04-02T04:54:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:30 smithi098 bash[23790]: audit 2024-04-02T04:54:30.204001+0000 mon.c (mon.2) 787 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:30.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:30.874 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:30.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:30.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:31.005 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:31.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:31.132 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:31.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:31.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:31.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:31.391 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:31.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.996914+0000 osd.3 (osd.3) 281 : cluster 0 6.c scrub starts 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: cluster 2024-04-02T04:54:29.998568+0000 osd.3 (osd.3) 282 : cluster 0 6.c scrub ok 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: cluster 2024-04-02T04:54:30.098385+0000 mgr.y (mgr.14538) 426 : cluster 0 pgmap v622: 364 pgs: 1 active+clean+snaptrim, 8 unknown, 355 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: cluster 2024-04-02T04:54:30.890975+0000 mon.a (mon.0) 2660 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: audit 2024-04-02T04:54:30.893447+0000 mon.a (mon.0) 2661 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: cluster 2024-04-02T04:54:30.900688+0000 mon.a (mon.0) 2662 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: audit 2024-04-02T04:54:30.904564+0000 mon.c (mon.2) 788 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:31.506 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: audit 2024-04-02T04:54:30.907258+0000 mon.a (mon.0) 2663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:31.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: audit 2024-04-02T04:54:30.907948+0000 mon.a (mon.0) 2664 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:31.507 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:31 smithi098 bash[23790]: audit 2024-04-02T04:54:31.205636+0000 mon.c (mon.2) 789 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:31.519 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:31.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:31.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.996914+0000 osd.3 (osd.3) 281 : cluster 0 6.c scrub starts 2024-04-02T04:54:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: cluster 2024-04-02T04:54:29.998568+0000 osd.3 (osd.3) 282 : cluster 0 6.c scrub ok 2024-04-02T04:54:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: cluster 2024-04-02T04:54:30.098385+0000 mgr.y (mgr.14538) 426 : cluster 0 pgmap v622: 364 pgs: 1 active+clean+snaptrim, 8 unknown, 355 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-02T04:54:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: cluster 2024-04-02T04:54:30.890975+0000 mon.a (mon.0) 2660 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: audit 2024-04-02T04:54:30.893447+0000 mon.a (mon.0) 2661 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:54:31.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: cluster 2024-04-02T04:54:30.900688+0000 mon.a (mon.0) 2662 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: audit 2024-04-02T04:54:30.904564+0000 mon.c (mon.2) 788 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: audit 2024-04-02T04:54:30.907258+0000 mon.a (mon.0) 2663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: audit 2024-04-02T04:54:30.907948+0000 mon.a (mon.0) 2664 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[21528]: audit 2024-04-02T04:54:31.205636+0000 mon.c (mon.2) 789 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.996914+0000 osd.3 (osd.3) 281 : cluster 0 6.c scrub starts 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: cluster 2024-04-02T04:54:29.998568+0000 osd.3 (osd.3) 282 : cluster 0 6.c scrub ok 2024-04-02T04:54:31.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: cluster 2024-04-02T04:54:30.098385+0000 mgr.y (mgr.14538) 426 : cluster 0 pgmap v622: 364 pgs: 1 active+clean+snaptrim, 8 unknown, 355 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: cluster 2024-04-02T04:54:30.890975+0000 mon.a (mon.0) 2660 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: audit 2024-04-02T04:54:30.893447+0000 mon.a (mon.0) 2661 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: cluster 2024-04-02T04:54:30.900688+0000 mon.a (mon.0) 2662 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: audit 2024-04-02T04:54:30.904564+0000 mon.c (mon.2) 788 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: audit 2024-04-02T04:54:30.907258+0000 mon.a (mon.0) 2663 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: audit 2024-04-02T04:54:30.907948+0000 mon.a (mon.0) 2664 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:31.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:31 smithi069 bash[28904]: audit 2024-04-02T04:54:31.205636+0000 mon.c (mon.2) 789 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.780 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:31.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:31.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:31.907 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:31.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:32.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:32.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:32.168 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:32.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:32.300 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 -- 172.21.15.69:0/236998304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 msgr2=0x7f9b0c10a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 --2- 172.21.15.69:0/236998304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c10a920 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009990 tx=0x7f9afc009ca0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 -- 172.21.15.69:0/236998304 shutdown_connections 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 --2- 172.21.15.69:0/236998304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104e50 0x7f9b0c111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 --2- 172.21.15.69:0/236998304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c10a920 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 --2- 172.21.15.69:0/236998304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104490 0x7f9b0c104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 -- 172.21.15.69:0/236998304 >> 172.21.15.69:0/236998304 conn(0x7f9b0c0fff20 msgr2=0x7f9b0c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 -- 172.21.15.69:0/236998304 shutdown_connections 2024-04-02T04:54:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 -- 172.21.15.69:0/236998304 wait complete. 2024-04-02T04:54:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 librados: starting msgr at 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 librados: starting objecter 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 Processor -- start 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 -- start start 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 librados: setting wanted keys 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.336+0000 7f9b14059700 1 librados: calling monclient init 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104490 0x7f9b0c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 0x7f9b0c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c113c90 con 0x7f9b0c104e50 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c113b10 con 0x7f9b0c104490 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9b0c113e10 con 0x7f9b0c10a540 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b115f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 0x7f9b0c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b115f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 0x7f9b0c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48598/0 (socket says 172.21.15.69:48598) 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b115f4700 1 -- 172.21.15.69:0/2932983959 learned_addr learned my addr 172.21.15.69:0/2932983959 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b11df5700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104490 0x7f9b0c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 -- 172.21.15.69:0/2932983959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104490 msgr2=0x7f9b0c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104490 0x7f9b0c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 -- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 msgr2=0x7f9b0c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 0x7f9b0c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 -- 172.21.15.69:0/2932983959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c19f200 con 0x7f9b0c10a540 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b11df5700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104490 0x7f9b0c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b115f4700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 0x7f9b0c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b125f6700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c19eac0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9b04007f10 tx=0x7f9b0400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 10 client.?.objecter ms_handle_connect 0x7f9b0c10a540 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0400ddd0 con 0x7f9b0c10a540 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9b04009e20 con 0x7f9b0c10a540 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0c19f4f0 con 0x7f9b0c10a540 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04015400 con 0x7f9b0c10a540 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 10 client.26528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0c1a6d70 con 0x7f9b0c10a540 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 librados: init done 2024-04-02T04:54:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b0c04ff00 con 0x7f9b0c10a540 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f9b04004ad0 con 0x7f9b0c10a540 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9af806f5e0 0x7f9af8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(432..432 src has 74..432) ==== 11139+0+0 (secure 0 0 0) 0x7f9b0408e850 con 0x7f9b0c10a540 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 10 client.26528.objecter ms_dispatch 0x7f9b0c001550 osd_map(432..432 src has 74..432) 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 3 client.26528.objecter handle_osd_map got epochs [432,432] > 0 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 3 client.26528.objecter handle_osd_map decoding full epoch 432 2024-04-02T04:54:32.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b11df5700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9af806f5e0 0x7f9af8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:32.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 20 client.26528.objecter dump_active .. 0 homeless 2024-04-02T04:54:32.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b11df5700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9af806f5e0 0x7f9af8071aa0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9afc006010 tx=0x7f9afc01a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:32.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.340+0000 7f9b02ffd700 10 client.26528.objecter ms_handle_connect 0x7f9af806f5e0 2024-04-02T04:54:32.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.348+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9b04010070 con 0x7f9b0c10a540 2024-04-02T04:54:32.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:32.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:32.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: cluster 2024-04-02T04:54:30.732354+0000 osd.4 (osd.4) 243 : cluster 0 41.1a scrub starts 2024-04-02T04:54:32.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: cluster 2024-04-02T04:54:30.734960+0000 osd.4 (osd.4) 244 : cluster 0 41.1a scrub ok 2024-04-02T04:54:32.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: cluster 2024-04-02T04:54:31.674934+0000 osd.5 (osd.5) 245 : cluster 0 40.10 scrub starts 2024-04-02T04:54:32.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: cluster 2024-04-02T04:54:31.677838+0000 osd.5 (osd.5) 246 : cluster 0 40.10 scrub ok 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: audit 2024-04-02T04:54:31.896089+0000 mon.a (mon.0) 2665 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: audit 2024-04-02T04:54:31.896360+0000 mon.a (mon.0) 2666 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: cluster 2024-04-02T04:54:31.904785+0000 mon.a (mon.0) 2667 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: audit 2024-04-02T04:54:31.909898+0000 mon.a (mon.0) 2668 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: audit 2024-04-02T04:54:31.910878+0000 mon.c (mon.2) 790 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: audit 2024-04-02T04:54:31.912421+0000 mon.a (mon.0) 2669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T04:54:32.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:32 smithi098 bash[23790]: audit 2024-04-02T04:54:32.206875+0000 mon.c (mon.2) 791 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:32.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.564+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7f9b0c1a6f40 con 0x7f9b0c10a540 2024-04-02T04:54:32.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:32.683 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: cluster 2024-04-02T04:54:30.732354+0000 osd.4 (osd.4) 243 : cluster 0 41.1a scrub starts 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: cluster 2024-04-02T04:54:30.734960+0000 osd.4 (osd.4) 244 : cluster 0 41.1a scrub ok 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: cluster 2024-04-02T04:54:31.674934+0000 osd.5 (osd.5) 245 : cluster 0 40.10 scrub starts 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: cluster 2024-04-02T04:54:31.677838+0000 osd.5 (osd.5) 246 : cluster 0 40.10 scrub ok 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: audit 2024-04-02T04:54:31.896089+0000 mon.a (mon.0) 2665 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: audit 2024-04-02T04:54:31.896360+0000 mon.a (mon.0) 2666 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: cluster 2024-04-02T04:54:31.904785+0000 mon.a (mon.0) 2667 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: audit 2024-04-02T04:54:31.909898+0000 mon.a (mon.0) 2668 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:32.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: audit 2024-04-02T04:54:31.910878+0000 mon.c (mon.2) 790 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: audit 2024-04-02T04:54:31.912421+0000 mon.a (mon.0) 2669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[28904]: audit 2024-04-02T04:54:32.206875+0000 mon.c (mon.2) 791 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: cluster 2024-04-02T04:54:30.732354+0000 osd.4 (osd.4) 243 : cluster 0 41.1a scrub starts 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: cluster 2024-04-02T04:54:30.734960+0000 osd.4 (osd.4) 244 : cluster 0 41.1a scrub ok 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: cluster 2024-04-02T04:54:31.674934+0000 osd.5 (osd.5) 245 : cluster 0 40.10 scrub starts 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: cluster 2024-04-02T04:54:31.677838+0000 osd.5 (osd.5) 246 : cluster 0 40.10 scrub ok 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: audit 2024-04-02T04:54:31.896089+0000 mon.a (mon.0) 2665 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: audit 2024-04-02T04:54:31.896360+0000 mon.a (mon.0) 2666 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: cluster 2024-04-02T04:54:31.904785+0000 mon.a (mon.0) 2667 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: audit 2024-04-02T04:54:31.909898+0000 mon.a (mon.0) 2668 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]: dispatch 2024-04-02T04:54:32.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: audit 2024-04-02T04:54:31.910878+0000 mon.c (mon.2) 790 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T04:54:32.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: audit 2024-04-02T04:54:31.912421+0000 mon.a (mon.0) 2669 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T04:54:32.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:32 smithi069 bash[21528]: audit 2024-04-02T04:54:32.206875+0000 mon.c (mon.2) 791 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:32.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:32.813 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:32.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.912+0000 7f9b02ffd700 1 -- 172.21.15.69:0/2932983959 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v433) ==== 129+0+0 (secure 0 0 0) 0x7f9b0405b050 con 0x7f9b0c10a540 2024-04-02T04:54:32.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:32.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_ 2024-04-02T04:54:32.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3008 ms) 2024-04-02T04:54:32.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-02T04:54:32.933 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7983 ms) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6993 ms) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6998 ms) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7015 ms) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6989 ms) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7041 ms) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (140048 ms total) 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-02T04:54:32.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (322976 ms total) 2024-04-02T04:54:32.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.932+0000 7f9b14059700 10 librados: watch_flush enter 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.932+0000 7f9b14059700 10 librados: watch_flush exit 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.932+0000 7f9b14059700 20 client.26528.objecter shutdown clearing up homeless session... 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.932+0000 7f9b14059700 10 client.26528.objecter successfully canceled tick 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9af806f5e0 msgr2=0x7f9af8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9af806f5e0 0x7f9af8071aa0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9afc006010 tx=0x7f9afc01a040 comp rx=0 tx=0).stop 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 msgr2=0x7f9b0c19eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c19eac0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9b04007f10 tx=0x7f9b0400d3b0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 shutdown_connections 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9af806f5e0 0x7f9af8071aa0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c10a540 0x7f9b0c19eac0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b0c104490 0x7f9b0c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 --2- 172.21.15.69:0/2932983959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c104e50 0x7f9b0c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:32.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 >> 172.21.15.69:0/2932983959 conn(0x7f9b0c0fff20 msgr2=0x7f9b0c10b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:32.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 shutdown_connections 2024-04-02T04:54:32.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 -- 172.21.15.69:0/2932983959 wait complete. 2024-04-02T04:54:32.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:32.936+0000 7f9b14059700 1 librados: shutdown 2024-04-02T04:54:32.937 INFO:teuthology.orchestra.run.smithi069.stderr:set pool 1 pgp_num to 1 2024-04-02T04:54:32.945 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:32.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:33.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:33.076 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:33.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:33.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:33.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:54:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:33.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:33.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:33.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:33.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: cluster 2024-04-02T04:54:32.099350+0000 mgr.y (mgr.14538) 427 : cluster 0 pgmap v625: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-02T04:54:33.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: cluster 2024-04-02T04:54:32.186361+0000 osd.1 (osd.1) 251 : cluster 0 102.4 scrub starts 2024-04-02T04:54:33.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: cluster 2024-04-02T04:54:32.190171+0000 osd.1 (osd.1) 252 : cluster 0 102.4 scrub ok 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.571083+0000 mon.c (mon.2) 792 : audit 1 from='client.? 172.21.15.69:0/2932983959' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.571839+0000 mon.a (mon.0) 2670 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.908449+0000 mon.a (mon.0) 2671 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.908681+0000 mon.a (mon.0) 2672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]': finished 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.908853+0000 mon.a (mon.0) 2673 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: cluster 2024-04-02T04:54:32.924113+0000 mon.a (mon.0) 2674 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.998081+0000 mon.b (mon.1) 222 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:32.998349+0000 mon.a (mon.0) 2675 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[28904]: audit 2024-04-02T04:54:33.208002+0000 mon.c (mon.2) 793 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: cluster 2024-04-02T04:54:32.099350+0000 mgr.y (mgr.14538) 427 : cluster 0 pgmap v625: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: cluster 2024-04-02T04:54:32.186361+0000 osd.1 (osd.1) 251 : cluster 0 102.4 scrub starts 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: cluster 2024-04-02T04:54:32.190171+0000 osd.1 (osd.1) 252 : cluster 0 102.4 scrub ok 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.571083+0000 mon.c (mon.2) 792 : audit 1 from='client.? 172.21.15.69:0/2932983959' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.571839+0000 mon.a (mon.0) 2670 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.908449+0000 mon.a (mon.0) 2671 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.908681+0000 mon.a (mon.0) 2672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]': finished 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.908853+0000 mon.a (mon.0) 2673 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: cluster 2024-04-02T04:54:32.924113+0000 mon.a (mon.0) 2674 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.998081+0000 mon.b (mon.1) 222 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:33.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:32.998349+0000 mon.a (mon.0) 2675 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:33.717 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:33 smithi069 bash[21528]: audit 2024-04-02T04:54:33.208002+0000 mon.c (mon.2) 793 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:33.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: cluster 2024-04-02T04:54:32.099350+0000 mgr.y (mgr.14538) 427 : cluster 0 pgmap v625: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-02T04:54:33.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: cluster 2024-04-02T04:54:32.186361+0000 osd.1 (osd.1) 251 : cluster 0 102.4 scrub starts 2024-04-02T04:54:33.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: cluster 2024-04-02T04:54:32.190171+0000 osd.1 (osd.1) 252 : cluster 0 102.4 scrub ok 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.571083+0000 mon.c (mon.2) 792 : audit 1 from='client.? 172.21.15.69:0/2932983959' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.571839+0000 mon.a (mon.0) 2670 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.908449+0000 mon.a (mon.0) 2671 : audit 1 from='client.? 172.21.15.69:0/2900523523' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-72153-66"}]': finished 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.908681+0000 mon.a (mon.0) 2672 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-51","var": "target_max_objects","val": "1"}]': finished 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.908853+0000 mon.a (mon.0) 2673 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: cluster 2024-04-02T04:54:32.924113+0000 mon.a (mon.0) 2674 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.998081+0000 mon.b (mon.1) 222 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:32.998349+0000 mon.a (mon.0) 2675 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:33.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:33 smithi098 bash[23790]: audit 2024-04-02T04:54:33.208002+0000 mon.c (mon.2) 793 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:33.727 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:33.830 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:33.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:33.985 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:33.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:34.088 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:34.114 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:34.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:34.245 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:34.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:34.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:34.487 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:34 smithi098 bash[23790]: cluster 2024-04-02T04:54:32.957960+0000 osd.3 (osd.3) 283 : cluster 0 2.19 scrub starts 2024-04-02T04:54:34.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:34 smithi098 bash[23790]: cluster 2024-04-02T04:54:32.960548+0000 osd.3 (osd.3) 284 : cluster 0 2.19 scrub ok 2024-04-02T04:54:34.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:34 smithi098 bash[23790]: audit 2024-04-02T04:54:33.257589+0000 mgr.y (mgr.14538) 428 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:34.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:34 smithi098 bash[23790]: cluster 2024-04-02T04:54:33.685896+0000 osd.5 (osd.5) 247 : cluster 0 13.17 deep-scrub starts 2024-04-02T04:54:34.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:34 smithi098 bash[23790]: cluster 2024-04-02T04:54:33.691595+0000 osd.5 (osd.5) 248 : cluster 0 13.17 deep-scrub ok 2024-04-02T04:54:34.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:34 smithi098 bash[23790]: audit 2024-04-02T04:54:34.209162+0000 mon.c (mon.2) 794 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:34.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:34.633 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:34.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:34.759 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:34.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[21528]: cluster 2024-04-02T04:54:32.957960+0000 osd.3 (osd.3) 283 : cluster 0 2.19 scrub starts 2024-04-02T04:54:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[21528]: cluster 2024-04-02T04:54:32.960548+0000 osd.3 (osd.3) 284 : cluster 0 2.19 scrub ok 2024-04-02T04:54:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[21528]: audit 2024-04-02T04:54:33.257589+0000 mgr.y (mgr.14538) 428 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[21528]: cluster 2024-04-02T04:54:33.685896+0000 osd.5 (osd.5) 247 : cluster 0 13.17 deep-scrub starts 2024-04-02T04:54:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[21528]: cluster 2024-04-02T04:54:33.691595+0000 osd.5 (osd.5) 248 : cluster 0 13.17 deep-scrub ok 2024-04-02T04:54:34.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[21528]: audit 2024-04-02T04:54:34.209162+0000 mon.c (mon.2) 794 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[28904]: cluster 2024-04-02T04:54:32.957960+0000 osd.3 (osd.3) 283 : cluster 0 2.19 scrub starts 2024-04-02T04:54:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[28904]: cluster 2024-04-02T04:54:32.960548+0000 osd.3 (osd.3) 284 : cluster 0 2.19 scrub ok 2024-04-02T04:54:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[28904]: audit 2024-04-02T04:54:33.257589+0000 mgr.y (mgr.14538) 428 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[28904]: cluster 2024-04-02T04:54:33.685896+0000 osd.5 (osd.5) 247 : cluster 0 13.17 deep-scrub starts 2024-04-02T04:54:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[28904]: cluster 2024-04-02T04:54:33.691595+0000 osd.5 (osd.5) 248 : cluster 0 13.17 deep-scrub ok 2024-04-02T04:54:34.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:34 smithi069 bash[28904]: audit 2024-04-02T04:54:34.209162+0000 mon.c (mon.2) 794 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:34.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:34.897 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:34.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:35.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:35.024 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:35.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- 172.21.15.69:0/1030188349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f421406a1d0 msgr2=0x7f421406a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- 172.21.15.69:0/1030188349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f421406a1d0 0x7f421406a5b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f42040099c0 tx=0x7f4204009cd0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- 172.21.15.69:0/1030188349 shutdown_connections 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- 172.21.15.69:0/1030188349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4214107620 0x7f421410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- 172.21.15.69:0/1030188349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f421406a1d0 0x7f421406a5b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- 172.21.15.69:0/1030188349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406ab80 0x7f4214106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- 172.21.15.69:0/1030188349 >> 172.21.15.69:0/1030188349 conn(0x7f42140771e0 msgr2=0x7f42140775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- 172.21.15.69:0/1030188349 shutdown_connections 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- 172.21.15.69:0/1030188349 wait complete. 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 librados: starting msgr at 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 librados: starting objecter 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 Processor -- start 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- start start 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 librados: setting wanted keys 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 librados: calling monclient init 2024-04-02T04:54:35.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 0x7f42141980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f421406ab80 0x7f42141985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 0x7f421419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4214111a80 con 0x7f4214107620 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4214111900 con 0x7f421406ab80 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4219960700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4214111c00 con 0x7f421406a1d0 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f42137fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 0x7f421419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4212ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 0x7f42141980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4212ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 0x7f42141980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:59592/0 (socket says 172.21.15.69:59592) 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f42137fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 0x7f421419c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48626/0 (socket says 172.21.15.69:48626) 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.112+0000 7f4212ffd700 1 -- 172.21.15.69:0/3938661975 learned_addr learned my addr 172.21.15.69:0/3938661975 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42127fc700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f421406ab80 0x7f42141985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42137fe700 1 -- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 msgr2=0x7f42141980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42137fe700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 0x7f42141980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42137fe700 1 -- 172.21.15.69:0/3938661975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f421406ab80 msgr2=0x7f42141985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42137fe700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f421406ab80 0x7f42141985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42137fe700 1 -- 172.21.15.69:0/3938661975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421419d0b0 con 0x7f4214107620 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4212ffd700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 0x7f42141980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42127fc700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f421406ab80 0x7f42141985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f42137fe700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 0x7f421419c970 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f420c00ab40 tx=0x7f420c00ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 10 client.?.objecter ms_handle_connect 0x7f4214107620 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c017020 con 0x7f4214107620 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f420c004bb0 con 0x7f4214107620 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421419d3a0 con 0x7f4214107620 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4219960700 10 client.16440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:35.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42141011a0 con 0x7f4214107620 2024-04-02T04:54:35.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4219960700 1 librados: init done 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c005280 con 0x7f4214107620 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4200005150 con 0x7f4214107620 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f420c00c640 con 0x7f4214107620 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f41fc06f4b0 0x7f41fc071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(434..434 src has 74..434) ==== 11060+0+0 (secure 0 0 0) 0x7f420c08e970 con 0x7f4214107620 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 10 client.16440.objecter ms_dispatch 0x7f4214001550 osd_map(434..434 src has 74..434) 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 3 client.16440.objecter handle_osd_map got epochs [434,434] > 0 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 3 client.16440.objecter handle_osd_map decoding full epoch 434 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4212ffd700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f41fc06f4b0 0x7f41fc071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 20 client.16440.objecter dump_active .. 0 homeless 2024-04-02T04:54:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f421895e700 10 client.16440.objecter ms_handle_connect 0x7f41fc06f4b0 2024-04-02T04:54:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.116+0000 7f4212ffd700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f41fc06f4b0 0x7f41fc071970 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4204009990 tx=0x7f4204005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.128+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f420c021020 con 0x7f4214107620 2024-04-02T04:54:35.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:35.188 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:54:35.188 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:54:35.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- 172.21.15.69:0/2675072804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101170 msgr2=0x7f195c1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- 172.21.15.69:0/2675072804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101170 0x7f195c1015f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1958009990 tx=0x7f1958009ca0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- 172.21.15.69:0/2675072804 shutdown_connections 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- 172.21.15.69:0/2675072804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101b30 0x7f195c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- 172.21.15.69:0/2675072804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f195c1071e0 0x7f195c1075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- 172.21.15.69:0/2675072804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101170 0x7f195c1015f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- 172.21.15.69:0/2675072804 >> 172.21.15.69:0/2675072804 conn(0x7f195c078110 msgr2=0x7f195c078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- 172.21.15.69:0/2675072804 shutdown_connections 2024-04-02T04:54:35.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- 172.21.15.69:0/2675072804 wait complete. 2024-04-02T04:54:35.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 librados: starting msgr at 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 librados: starting objecter 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 Processor -- start 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- start start 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 librados: setting wanted keys 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 librados: calling monclient init 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 0x7f195c10ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101b30 0x7f195c10f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f195c1071e0 0x7f195c10f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f195c111a80 con 0x7f195c101170 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f195c111900 con 0x7f195c1071e0 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f196413f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f195c111c00 con 0x7f195c101b30 2024-04-02T04:54:35.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 0x7f195c10ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 0x7f195c10ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48636/0 (socket says 172.21.15.69:48636) 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 -- 172.21.15.69:0/140023482 learned_addr learned my addr 172.21.15.69:0/140023482 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101b30 0x7f195c10f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101b30 msgr2=0x7f195c10f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f195c101b30 0x7f195c10f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f195c1071e0 msgr2=0x7f195c10f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.316+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f195c1071e0 0x7f195c10f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1961edb700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c1a65d0 con 0x7f195c101170 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 0x7f195c10ece0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f194c00b530 tx=0x7f194c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 10 client.?.objecter ms_handle_connect 0x7f195c101170 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c015020 con 0x7f195c101170 2024-04-02T04:54:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c1a68c0 con 0x7f195c101170 2024-04-02T04:54:35.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f194c011cb0 con 0x7f195c101170 2024-04-02T04:54:35.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c010d20 con 0x7f195c101170 2024-04-02T04:54:35.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 10 client.16446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:35.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c1a6dd0 con 0x7f195c101170 2024-04-02T04:54:35.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 1 librados: init done 2024-04-02T04:54:35.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f195c102890 con 0x7f195c101170 2024-04-02T04:54:35.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f194c018020 con 0x7f195c101170 2024-04-02T04:54:35.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f194806f5e0 0x7f1948071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f194806f5e0 0x7f1948071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(434..434 src has 74..434) ==== 11060+0+0 (secure 0 0 0) 0x7f194c08e6a0 con 0x7f195c101170 2024-04-02T04:54:35.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 osd_map(434..434 src has 74..434) 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 3 client.16446.objecter handle_osd_map got epochs [434,434] > 0 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 3 client.16446.objecter handle_osd_map decoding full epoch 434 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 20 client.16446.objecter dump_active .. 0 homeless 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v434) ==== 54+0+16 (secure 0 0 0) 0x7f194c01f020 con 0x7f195c101170 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 10 client.16446.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 1 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f194806f5e0 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f194806f5e0 0x7f1948071aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f19580098e0 tx=0x7f1958005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f195c062e50 0x7f195c063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 1 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 0 1 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 4 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 10 client.16446.objecter _send_command 1 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f195c1aa190 con 0x7f195c062e50 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f19626dc700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f195c062e50 0x7f195c063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f19626dc700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f195c062e50 0x7f195c063230 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.320+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c062e50 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.324+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1aa190 con 0x7f195c062e50 2024-04-02T04:54:35.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.324+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 1: 0 ) 2024-04-02T04:54:35.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.324+0000 7f1952ffd700 10 client.16446.objecter _finish_command 1 = system:0 2024-04-02T04:54:35.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.324+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 4 2024-04-02T04:54:35.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.324+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 0 1 2024-04-02T04:54:35.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 10 client.16446.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 2 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 2 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c04ff00 osd=0 3 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 0 2 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 4 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 10 client.16446.objecter _send_command 2 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c0628d0 con 0x7f195c062e50 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c0628d0 con 0x7f195c062e50 2024-04-02T04:54:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f1952ffd700 10 client.16446.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 4 2024-04-02T04:54:35.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 0 2 2024-04-02T04:54:35.343 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 10 client.16446.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f195c02c9b0 0x7f195c02cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 1 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 4 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 10 client.16446.objecter _send_command 3 2024-04-02T04:54:35.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f195c108d90 con 0x7f195c02c9b0 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.340+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f195c02c9b0 0x7f195c02cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f195c02c9b0 0x7f195c02cdd0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c02c9b0 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c108d90 con 0x7f195c02c9b0 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 3: 0 ) 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1952ffd700 10 client.16446.objecter _finish_command 3 = system:0 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 4 2024-04-02T04:54:35.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.344+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 1 3 2024-04-02T04:54:35.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 10 client.16446.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 4 2024-04-02T04:54:35.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 4 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c108710 osd=1 3 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 1 4 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 4 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 10 client.16446.objecter _send_command 4 2024-04-02T04:54:35.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c108190 con 0x7f195c02c9b0 2024-04-02T04:54:35.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c108190 con 0x7f195c02c9b0 2024-04-02T04:54:35.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f1952ffd700 10 client.16446.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 4 2024-04-02T04:54:35.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.360+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 1 4 2024-04-02T04:54:35.363 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:54:35.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 10 client.16446.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 5 2024-04-02T04:54:35.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f195c1ac350 0x7f195c1ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 5 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 2 5 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 4 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 10 client.16446.objecter _send_command 5 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f195c1b15a0 con 0x7f195c1ac350 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f195c1ac350 0x7f195c1ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f4200005710 con 0x7f4214107620 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c1ac350 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f195c1ac350 0x7f195c1ae740 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1b15a0 con 0x7f195c1ac350 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 5: 0 ) 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f1952ffd700 10 client.16446.objecter _finish_command 5 = system:0 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 4 2024-04-02T04:54:35.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.364+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 2 5 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 10 client.16446.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 6 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 6 2024-04-02T04:54:35.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c108590 osd=2 3 2024-04-02T04:54:35.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 2 6 2024-04-02T04:54:35.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 4 2024-04-02T04:54:35.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 10 client.16446.objecter _send_command 6 2024-04-02T04:54:35.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c02d560 con 0x7f195c1ac350 2024-04-02T04:54:35.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c02d560 con 0x7f195c1ac350 2024-04-02T04:54:35.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f1952ffd700 10 client.16446.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 4 2024-04-02T04:54:35.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 2 6 2024-04-02T04:54:35.382 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:54:35.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 10 client.16446.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 7 2024-04-02T04:54:35.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f195c1afe00 0x7f195c1b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 7 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 3 7 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 4 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 10 client.16446.objecter _send_command 7 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f195c1a9eb0 con 0x7f195c1afe00 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.380+0000 7f19626dc700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f195c1afe00 0x7f195c1b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c1afe00 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f19626dc700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f195c1afe00 0x7f195c1b0220 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1a9eb0 con 0x7f195c1afe00 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 7: 0 ) 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f1952ffd700 10 client.16446.objecter _finish_command 7 = system:0 2024-04-02T04:54:35.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 4 2024-04-02T04:54:35.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.384+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 3 7 2024-04-02T04:54:35.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:35.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 10 client.16446.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 8 2024-04-02T04:54:35.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 8 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1a97e0 osd=3 3 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 3 8 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 4 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 10 client.16446.objecter _send_command 8 2024-04-02T04:54:35.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.400+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c1a9eb0 con 0x7f195c1afe00 2024-04-02T04:54:35.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c1a9eb0 con 0x7f195c1afe00 2024-04-02T04:54:35.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1952ffd700 10 client.16446.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 4 2024-04-02T04:54:35.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 3 8 2024-04-02T04:54:35.404 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 10 client.16446.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 9 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f195c1a9310 0x7f195c1b5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 9 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 4 9 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 4 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 10 client.16446.objecter _send_command 9 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f195c1b85d0 con 0x7f195c1a9310 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f195c1a9310 0x7f195c1b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c1a9310 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.404+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f195c1a9310 0x7f195c1b5770 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.408+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1b85d0 con 0x7f195c1a9310 2024-04-02T04:54:35.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.408+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 9: 0 ) 2024-04-02T04:54:35.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.408+0000 7f1952ffd700 10 client.16446.objecter _finish_command 9 = system:0 2024-04-02T04:54:35.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.408+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 4 2024-04-02T04:54:35.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.408+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 4 9 2024-04-02T04:54:35.413 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:35.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 10 client.16446.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 10 2024-04-02T04:54:35.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 10 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1a9eb0 osd=4 3 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 4 10 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 4 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 10 client.16446.objecter _send_command 10 2024-04-02T04:54:35.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c1b5cb0 con 0x7f195c1a9310 2024-04-02T04:54:35.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c1b5cb0 con 0x7f195c1a9310 2024-04-02T04:54:35.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f1952ffd700 10 client.16446.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 4 2024-04-02T04:54:35.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 4 10 2024-04-02T04:54:35.426 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 10 client.16446.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 11 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f195c1b7bf0 0x7f195c1b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 11 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 5 11 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 4 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 10 client.16446.objecter _send_command 11 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.424+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f195c1bb940 con 0x7f195c1b7bf0 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f195c1b7bf0 0x7f195c1b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c1b7bf0 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f19616da700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f195c1b7bf0 0x7f195c1b7ff0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1bb940 con 0x7f195c1b7bf0 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 11: 0 ) 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f1952ffd700 10 client.16446.objecter _finish_command 11 = system:0 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 4 2024-04-02T04:54:35.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.428+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 5 11 2024-04-02T04:54:35.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 10 client.16446.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 12 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 12 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1b5cb0 osd=5 3 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 5 12 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 4 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 10 client.16446.objecter _send_command 12 2024-04-02T04:54:35.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.444+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c1b5ec0 con 0x7f195c1b7bf0 2024-04-02T04:54:35.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c1b5ec0 con 0x7f195c1b7bf0 2024-04-02T04:54:35.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 10 client.16446.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 4 2024-04-02T04:54:35.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 5 12 2024-04-02T04:54:35.447 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 10 client.16446.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 13 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f195c1b63d0 0x7f195c1b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 13 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 6 13 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 4 2024-04-02T04:54:35.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 10 client.16446.objecter _send_command 13 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f195c1b6d50 con 0x7f195c1b63d0 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f19626dc700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f195c1b63d0 0x7f195c1b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c1b63d0 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f19626dc700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f195c1b63d0 0x7f195c1b6810 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1b6d50 con 0x7f195c1b63d0 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 13: 0 ) 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 10 client.16446.objecter _finish_command 13 = system:0 2024-04-02T04:54:35.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 4 2024-04-02T04:54:35.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.448+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 6 13 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 10 client.16446.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 14 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 14 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1b7270 osd=6 3 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 6 14 2024-04-02T04:54:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 4 2024-04-02T04:54:35.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 10 client.16446.objecter _send_command 14 2024-04-02T04:54:35.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c1b6d50 con 0x7f195c1b63d0 2024-04-02T04:54:35.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c1b6d50 con 0x7f195c1b63d0 2024-04-02T04:54:35.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f1952ffd700 10 client.16446.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 4 2024-04-02T04:54:35.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 6 14 2024-04-02T04:54:35.470 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 10 client.16446.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 15 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f195c1bcce0 0x7f195c1bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 15 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 7 15 2024-04-02T04:54:35.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 4 2024-04-02T04:54:35.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 10 client.16446.objecter _send_command 15 2024-04-02T04:54:35.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f195c1c1f30 con 0x7f195c1bcce0 2024-04-02T04:54:35.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.468+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f195c1bcce0 0x7f195c1bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:35.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1952ffd700 10 client.16446.objecter ms_handle_connect 0x7f195c1bcce0 2024-04-02T04:54:35.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1961edb700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f195c1bcce0 0x7f195c1bf0d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:35.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c1c1f30 con 0x7f195c1bcce0 2024-04-02T04:54:35.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 15: 0 ) 2024-04-02T04:54:35.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1952ffd700 10 client.16446.objecter _finish_command 15 = system:0 2024-04-02T04:54:35.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 4 2024-04-02T04:54:35.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.472+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 7 15 2024-04-02T04:54:35.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 10 client.16446.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:54:35.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign -1 16 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 20 client.16446.objecter _get_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 15 client.16446.objecter _session_command_op_remove -1 16 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 20 client.16446.objecter get_session s=0x7f195c1b6d50 osd=7 3 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 15 client.16446.objecter _session_command_op_assign 7 16 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 4 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 10 client.16446.objecter _send_command 16 2024-04-02T04:54:35.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.488+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f195c1bf610 con 0x7f195c1bcce0 2024-04-02T04:54:35.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f1952ffd700 1 -- 172.21.15.69:0/140023482 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f195c1bf610 con 0x7f195c1bcce0 2024-04-02T04:54:35.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f1952ffd700 10 client.16446.objecter ms_dispatch 0x7f195c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:35.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f1952ffd700 10 client.16446.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:35.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f1952ffd700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 4 2024-04-02T04:54:35.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f1952ffd700 15 client.16446.objecter _session_command_op_remove 7 16 2024-04-02T04:54:35.493 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:54:35.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 librados: watch_flush enter 2024-04-02T04:54:35.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 librados: watch_flush exit 2024-04-02T04:54:35.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.0 2024-04-02T04:54:35.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f195c062e50 msgr2=0x7f195c063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f195c062e50 0x7f195c063230 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c04ff00 osd=0 1 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.1 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f195c02c9b0 msgr2=0x7f195c02cdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f195c02c9b0 0x7f195c02cdd0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108710 osd=1 1 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.2 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f195c1ac350 msgr2=0x7f195c1ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f195c1ac350 0x7f195c1ae740 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f19626dc700 1 -- 172.21.15.69:0/140023482 reap_dead start 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c108590 osd=2 1 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.3 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f195c1afe00 msgr2=0x7f195c1b0220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f195c1afe00 0x7f195c1b0220 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a97e0 osd=3 1 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.4 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f195c1a9310 msgr2=0x7f195c1b5770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f195c1a9310 0x7f195c1b5770 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1a9eb0 osd=4 1 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.5 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f195c1b7bf0 msgr2=0x7f195c1b7ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f195c1b7bf0 0x7f195c1b7ff0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b5cb0 osd=5 1 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.6 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f195c1b63d0 msgr2=0x7f195c1b6810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f195c1b63d0 0x7f195c1b6810 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b7270 osd=6 1 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter close_session for osd.7 2024-04-02T04:54:35.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f195c1bcce0 msgr2=0x7f195c1bf0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f195c1bcce0 0x7f195c1bf0d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter put_session s=0x7f195c1b6d50 osd=7 1 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 20 client.16446.objecter shutdown clearing up homeless session... 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 10 client.16446.objecter successfully canceled tick 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f19626dc700 1 -- 172.21.15.69:0/140023482 reap_dead start 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f194806f5e0 msgr2=0x7f1948071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f194806f5e0 0x7f1948071aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f19580098e0 tx=0x7f1958005960 comp rx=0 tx=0).stop 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 msgr2=0x7f195c10ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 0x7f195c10ece0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f194c00b530 tx=0x7f194c00b840 comp rx=0 tx=0).stop 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 shutdown_connections 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f195c101170 0x7f195c10ece0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 --2- 172.21.15.69:0/140023482 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f194806f5e0 0x7f1948071aa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 >> 172.21.15.69:0/140023482 conn(0x7f195c078110 msgr2=0x7f195c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 shutdown_connections 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 -- 172.21.15.69:0/140023482 wait complete. 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.492+0000 7f196413f700 1 librados: shutdown 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:54:35.499 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:54:35.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.103381+0000 mgr.y (mgr.14538) 429 : cluster 0 pgmap v627: 356 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 354 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:35.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.161585+0000 osd.1 (osd.1) 253 : cluster 0 102.13 scrub starts 2024-04-02T04:54:35.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.165265+0000 osd.1 (osd.1) 254 : cluster 0 102.13 scrub ok 2024-04-02T04:54:35.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.461441+0000 mon.a (mon.0) 2676 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-02T04:54:35.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.890920+0000 mon.a (mon.0) 2677 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-04-02T04:54:35.527 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.915264+0000 osd.2 (osd.2) 204 : cluster 0 102.1e scrub starts 2024-04-02T04:54:35.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.920012+0000 osd.2 (osd.2) 205 : cluster 0 102.1e scrub ok 2024-04-02T04:54:35.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.973922+0000 osd.6 (osd.6) 211 : cluster 0 2.17 scrub starts 2024-04-02T04:54:35.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: cluster 2024-04-02T04:54:34.979120+0000 osd.6 (osd.6) 212 : cluster 0 2.17 scrub ok 2024-04-02T04:54:35.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: audit 2024-04-02T04:54:35.210304+0000 mon.c (mon.2) 795 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:35.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: audit 2024-04-02T04:54:35.326194+0000 mon.a (mon.0) 2678 : audit 0 from='client.? 172.21.15.69:0/140023482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:35.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:35 smithi098 bash[23790]: audit 2024-04-02T04:54:35.368619+0000 mon.a (mon.0) 2679 : audit 1 from='client.? 172.21.15.69:0/3938661975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:54:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:35.647 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:35.671 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:35.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.103381+0000 mgr.y (mgr.14538) 429 : cluster 0 pgmap v627: 356 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 354 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.161585+0000 osd.1 (osd.1) 253 : cluster 0 102.13 scrub starts 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.165265+0000 osd.1 (osd.1) 254 : cluster 0 102.13 scrub ok 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.461441+0000 mon.a (mon.0) 2676 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.890920+0000 mon.a (mon.0) 2677 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.915264+0000 osd.2 (osd.2) 204 : cluster 0 102.1e scrub starts 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.920012+0000 osd.2 (osd.2) 205 : cluster 0 102.1e scrub ok 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.973922+0000 osd.6 (osd.6) 211 : cluster 0 2.17 scrub starts 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: cluster 2024-04-02T04:54:34.979120+0000 osd.6 (osd.6) 212 : cluster 0 2.17 scrub ok 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: audit 2024-04-02T04:54:35.210304+0000 mon.c (mon.2) 795 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: audit 2024-04-02T04:54:35.326194+0000 mon.a (mon.0) 2678 : audit 0 from='client.? 172.21.15.69:0/140023482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:35.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[21528]: audit 2024-04-02T04:54:35.368619+0000 mon.a (mon.0) 2679 : audit 1 from='client.? 172.21.15.69:0/3938661975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.103381+0000 mgr.y (mgr.14538) 429 : cluster 0 pgmap v627: 356 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 354 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.161585+0000 osd.1 (osd.1) 253 : cluster 0 102.13 scrub starts 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.165265+0000 osd.1 (osd.1) 254 : cluster 0 102.13 scrub ok 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.461441+0000 mon.a (mon.0) 2676 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.890920+0000 mon.a (mon.0) 2677 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.915264+0000 osd.2 (osd.2) 204 : cluster 0 102.1e scrub starts 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.920012+0000 osd.2 (osd.2) 205 : cluster 0 102.1e scrub ok 2024-04-02T04:54:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.973922+0000 osd.6 (osd.6) 211 : cluster 0 2.17 scrub starts 2024-04-02T04:54:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: cluster 2024-04-02T04:54:34.979120+0000 osd.6 (osd.6) 212 : cluster 0 2.17 scrub ok 2024-04-02T04:54:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: audit 2024-04-02T04:54:35.210304+0000 mon.c (mon.2) 795 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: audit 2024-04-02T04:54:35.326194+0000 mon.a (mon.0) 2678 : audit 0 from='client.? 172.21.15.69:0/140023482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:35.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:35 smithi069 bash[28904]: audit 2024-04-02T04:54:35.368619+0000 mon.a (mon.0) 2679 : audit 1 from='client.? 172.21.15.69:0/3938661975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:54:35.799 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:35.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.880+0000 7f421895e700 1 -- 172.21.15.69:0/3938661975 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v435) ==== 87+0+0 (secure 0 0 0) 0x7f420c05fe20 con 0x7f4214107620 2024-04-02T04:54:35.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 10 librados: watch_flush enter 2024-04-02T04:54:35.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 10 librados: watch_flush exit 2024-04-02T04:54:35.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 20 client.16440.objecter shutdown clearing up homeless session... 2024-04-02T04:54:35.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 10 client.16440.objecter successfully canceled tick 2024-04-02T04:54:35.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f41fc06f4b0 msgr2=0x7f41fc071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f41fc06f4b0 0x7f41fc071970 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4204009990 tx=0x7f4204005960 comp rx=0 tx=0).stop 2024-04-02T04:54:35.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 msgr2=0x7f421419c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:35.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.884+0000 7f4219960700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 0x7f421419c970 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f420c00ab40 tx=0x7f420c00ae50 comp rx=0 tx=0).stop 2024-04-02T04:54:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.888+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 shutdown_connections 2024-04-02T04:54:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.888+0000 7f4219960700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f41fc06f4b0 0x7f41fc071970 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.888+0000 7f4219960700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4214107620 0x7f421419c970 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.888+0000 7f4219960700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f421406a1d0 0x7f42141980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.888+0000 7f4219960700 1 --2- 172.21.15.69:0/3938661975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f421406ab80 0x7f42141985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.888+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 >> 172.21.15.69:0/3938661975 conn(0x7f42140771e0 msgr2=0x7f42140ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.892+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 shutdown_connections 2024-04-02T04:54:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.892+0000 7f4219960700 1 -- 172.21.15.69:0/3938661975 wait complete. 2024-04-02T04:54:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:35.892+0000 7f4219960700 1 librados: shutdown 2024-04-02T04:54:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:54:35.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:35.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:36.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:36.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:36.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:36.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:36.319 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:36.422 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:36.446 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:36.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-02T04:54:36.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:36.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.040191+0000 osd.0 (osd.0) 231 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.040359+0000 osd.0 (osd.0) 232 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.397364+0000 osd.0 (osd.0) 233 : cluster 0 4.16 deep-scrub starts 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.398672+0000 osd.0 (osd.0) 234 : cluster 0 4.16 deep-scrub ok 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.771547+0000 osd.5 (osd.5) 249 : cluster 0 10.6 scrub starts 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.773819+0000 osd.5 (osd.5) 250 : cluster 0 10.6 scrub ok 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.884896+0000 mon.a (mon.0) 2680 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: audit 2024-04-02T04:54:35.887111+0000 mon.a (mon.0) 2681 : audit 1 from='client.? 172.21.15.69:0/3938661975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: cluster 2024-04-02T04:54:35.893769+0000 mon.a (mon.0) 2682 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-04-02T04:54:36.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:36 smithi098 bash[23790]: audit 2024-04-02T04:54:36.211391+0000 mon.c (mon.2) 796 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:36.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.040191+0000 osd.0 (osd.0) 231 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:36.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.040359+0000 osd.0 (osd.0) 232 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.397364+0000 osd.0 (osd.0) 233 : cluster 0 4.16 deep-scrub starts 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.398672+0000 osd.0 (osd.0) 234 : cluster 0 4.16 deep-scrub ok 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.771547+0000 osd.5 (osd.5) 249 : cluster 0 10.6 scrub starts 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.773819+0000 osd.5 (osd.5) 250 : cluster 0 10.6 scrub ok 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.884896+0000 mon.a (mon.0) 2680 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: audit 2024-04-02T04:54:35.887111+0000 mon.a (mon.0) 2681 : audit 1 from='client.? 172.21.15.69:0/3938661975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: cluster 2024-04-02T04:54:35.893769+0000 mon.a (mon.0) 2682 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[21528]: audit 2024-04-02T04:54:36.211391+0000 mon.c (mon.2) 796 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:36.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.040191+0000 osd.0 (osd.0) 231 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.040359+0000 osd.0 (osd.0) 232 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.397364+0000 osd.0 (osd.0) 233 : cluster 0 4.16 deep-scrub starts 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.398672+0000 osd.0 (osd.0) 234 : cluster 0 4.16 deep-scrub ok 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.771547+0000 osd.5 (osd.5) 249 : cluster 0 10.6 scrub starts 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.773819+0000 osd.5 (osd.5) 250 : cluster 0 10.6 scrub ok 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.884896+0000 mon.a (mon.0) 2680 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: audit 2024-04-02T04:54:35.887111+0000 mon.a (mon.0) 2681 : audit 1 from='client.? 172.21.15.69:0/3938661975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: cluster 2024-04-02T04:54:35.893769+0000 mon.a (mon.0) 2682 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-04-02T04:54:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:36 smithi069 bash[28904]: audit 2024-04-02T04:54:36.211391+0000 mon.c (mon.2) 796 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:36.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:36.836 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:36.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:36.967 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:36.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:37.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:37.097 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:37.200 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:37.227 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:37.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:37.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:37.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:37.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:37.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:37.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:37 smithi098 bash[23790]: cluster 2024-04-02T04:54:36.104410+0000 mgr.y (mgr.14538) 430 : cluster 0 pgmap v630: 356 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 354 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:37.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:37 smithi098 bash[23790]: audit 2024-04-02T04:54:37.212426+0000 mon.c (mon.2) 797 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:37.744 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:37.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:37 smithi069 bash[21528]: cluster 2024-04-02T04:54:36.104410+0000 mgr.y (mgr.14538) 430 : cluster 0 pgmap v630: 356 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 354 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:37.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:37 smithi069 bash[21528]: audit 2024-04-02T04:54:37.212426+0000 mon.c (mon.2) 797 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:37.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:37 smithi069 bash[28904]: cluster 2024-04-02T04:54:36.104410+0000 mgr.y (mgr.14538) 430 : cluster 0 pgmap v630: 356 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 354 active+clean; 8.4 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:37.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:37 smithi069 bash[28904]: audit 2024-04-02T04:54:37.212426+0000 mon.c (mon.2) 797 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:37.847 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:37.876 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:37.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:37.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:38.006 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:38.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:38.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:38.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:38.395 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:38.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:38.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:38.655 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:38.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:38 smithi098 bash[23790]: audit 2024-04-02T04:54:38.213935+0000 mon.c (mon.2) 798 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:38.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:38.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:38 smithi069 bash[21528]: audit 2024-04-02T04:54:38.213935+0000 mon.c (mon.2) 798 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:38.768 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:54:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:54:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:54:38.768 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:38 smithi069 bash[28904]: audit 2024-04-02T04:54:38.213935+0000 mon.c (mon.2) 798 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:38.782 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:38.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:38.912 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:38.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:39.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:39.041 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:39.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:39.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:39.297 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:39.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:54:39.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:54:39.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:39.428 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:39.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:39 smithi069 bash[28904]: cluster 2024-04-02T04:54:38.107560+0000 mgr.y (mgr.14538) 431 : cluster 0 pgmap v631: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:39.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:39 smithi069 bash[28904]: audit 2024-04-02T04:54:39.215559+0000 mon.c (mon.2) 799 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:39.531 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:39.551 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:39.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:39.665 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:39 smithi098 bash[23790]: cluster 2024-04-02T04:54:38.107560+0000 mgr.y (mgr.14538) 431 : cluster 0 pgmap v631: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:39.665 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:39 smithi098 bash[23790]: audit 2024-04-02T04:54:39.215559+0000 mon.c (mon.2) 799 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:39.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:39 smithi069 bash[21528]: cluster 2024-04-02T04:54:38.107560+0000 mgr.y (mgr.14538) 431 : cluster 0 pgmap v631: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:39.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:39 smithi069 bash[21528]: audit 2024-04-02T04:54:39.215559+0000 mon.c (mon.2) 799 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:39.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:39.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:39.911 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:39.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:40.015 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:54:40.016 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:54:40.016 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:54:40.016 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd out 5 2024-04-02T04:54:40.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.086 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:40.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:40.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:40.347 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:40.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:40.477 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:40.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:40.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.736 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:40.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.862 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:40.965 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:40.991 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:40.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:41.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:41.105 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:40 smithi069 bash[21528]: cluster 2024-04-02T04:54:40.108559+0000 mgr.y (mgr.14538) 432 : cluster 0 pgmap v632: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:41.105 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:40 smithi069 bash[21528]: audit 2024-04-02T04:54:40.216585+0000 mon.c (mon.2) 800 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:41.105 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:40 smithi069 bash[28904]: cluster 2024-04-02T04:54:40.108559+0000 mgr.y (mgr.14538) 432 : cluster 0 pgmap v632: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:41.105 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:40 smithi069 bash[28904]: audit 2024-04-02T04:54:40.216585+0000 mon.c (mon.2) 800 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:40 smithi098 bash[23790]: cluster 2024-04-02T04:54:40.108559+0000 mgr.y (mgr.14538) 432 : cluster 0 pgmap v632: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:54:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:40 smithi098 bash[23790]: audit 2024-04-02T04:54:40.216585+0000 mon.c (mon.2) 800 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:41.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:41.273 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:41.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:41.381 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:41.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:41.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:41.642 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:41.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:41.772 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:41.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:41.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:42.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:42.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:41 smithi069 bash[21528]: audit 2024-04-02T04:54:41.217837+0000 mon.c (mon.2) 801 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:42.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:41 smithi069 bash[28904]: audit 2024-04-02T04:54:41.217837+0000 mon.c (mon.2) 801 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:42.034 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:42.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:42.163 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:42.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:41 smithi098 bash[23790]: audit 2024-04-02T04:54:41.217837+0000 mon.c (mon.2) 801 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:42.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:42.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:42.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:42.553 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:42.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:42.684 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:42.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:42.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:42.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:42 smithi098 bash[23790]: cluster 2024-04-02T04:54:42.109476+0000 mgr.y (mgr.14538) 433 : cluster 0 pgmap v633: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-04-02T04:54:42.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:42 smithi098 bash[23790]: audit 2024-04-02T04:54:42.219613+0000 mon.c (mon.2) 802 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:42.947 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:42.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:43.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:43.076 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:43.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:43.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:43.204 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:43.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:42 smithi069 bash[28904]: cluster 2024-04-02T04:54:42.109476+0000 mgr.y (mgr.14538) 433 : cluster 0 pgmap v633: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-04-02T04:54:43.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:42 smithi069 bash[28904]: audit 2024-04-02T04:54:42.219613+0000 mon.c (mon.2) 802 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:42 smithi069 bash[21528]: cluster 2024-04-02T04:54:42.109476+0000 mgr.y (mgr.14538) 433 : cluster 0 pgmap v633: 356 pgs: 1 active+clean+snaptrim, 355 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-04-02T04:54:43.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:42 smithi069 bash[21528]: audit 2024-04-02T04:54:42.219613+0000 mon.c (mon.2) 802 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:43.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:43.334 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:43.437 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:43.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:54:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:54:43.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:43.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:43.721 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:43.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:43.852 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:43.955 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:43.979 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:43.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:43.984 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:43.985 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:44.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:44.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:44.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:43 smithi098 bash[23790]: audit 2024-04-02T04:54:42.952352+0000 mon.c (mon.2) 803 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:44.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:43 smithi098 bash[23790]: audit 2024-04-02T04:54:42.953079+0000 mon.a (mon.0) 2683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:44.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:43 smithi098 bash[23790]: audit 2024-04-02T04:54:43.220833+0000 mon.c (mon.2) 804 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:44.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:43 smithi098 bash[23790]: audit 2024-04-02T04:54:43.266003+0000 mgr.y (mgr.14538) 434 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:44.241 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:44.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[28904]: audit 2024-04-02T04:54:42.952352+0000 mon.c (mon.2) 803 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:44.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[28904]: audit 2024-04-02T04:54:42.953079+0000 mon.a (mon.0) 2683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:44.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[28904]: audit 2024-04-02T04:54:43.220833+0000 mon.c (mon.2) 804 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:44.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[28904]: audit 2024-04-02T04:54:43.266003+0000 mgr.y (mgr.14538) 434 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:44.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[21528]: audit 2024-04-02T04:54:42.952352+0000 mon.c (mon.2) 803 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:44.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[21528]: audit 2024-04-02T04:54:42.953079+0000 mon.a (mon.0) 2683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:44.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[21528]: audit 2024-04-02T04:54:43.220833+0000 mon.c (mon.2) 804 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:44.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:43 smithi069 bash[21528]: audit 2024-04-02T04:54:43.266003+0000 mgr.y (mgr.14538) 434 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:44.332 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:44.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:44.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:44.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:44.496 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:44.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:44.625 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:44.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:44.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:44.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:44.886 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:44.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:44.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:45.018 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:45.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:45.148 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: audit 2024-04-02T04:54:43.923435+0000 mon.a (mon.0) 2684 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: cluster 2024-04-02T04:54:43.927879+0000 mon.a (mon.0) 2685 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: audit 2024-04-02T04:54:43.931044+0000 mon.c (mon.2) 805 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: audit 2024-04-02T04:54:43.931683+0000 mon.a (mon.0) 2686 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: cluster 2024-04-02T04:54:44.110890+0000 mgr.y (mgr.14538) 435 : cluster 0 pgmap v635: 356 pgs: 356 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: audit 2024-04-02T04:54:44.222562+0000 mon.c (mon.2) 806 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: audit 2024-04-02T04:54:44.887255+0000 mon.a (mon.0) 2687 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]': finished 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[28904]: cluster 2024-04-02T04:54:44.891508+0000 mon.a (mon.0) 2688 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: audit 2024-04-02T04:54:43.923435+0000 mon.a (mon.0) 2684 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: cluster 2024-04-02T04:54:43.927879+0000 mon.a (mon.0) 2685 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-04-02T04:54:45.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: audit 2024-04-02T04:54:43.931044+0000 mon.c (mon.2) 805 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:45.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: audit 2024-04-02T04:54:43.931683+0000 mon.a (mon.0) 2686 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:45.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: cluster 2024-04-02T04:54:44.110890+0000 mgr.y (mgr.14538) 435 : cluster 0 pgmap v635: 356 pgs: 356 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:54:45.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: audit 2024-04-02T04:54:44.222562+0000 mon.c (mon.2) 806 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:45.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: audit 2024-04-02T04:54:44.887255+0000 mon.a (mon.0) 2687 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]': finished 2024-04-02T04:54:45.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:44 smithi069 bash[21528]: cluster 2024-04-02T04:54:44.891508+0000 mon.a (mon.0) 2688 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-04-02T04:54:45.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: audit 2024-04-02T04:54:43.923435+0000 mon.a (mon.0) 2684 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: cluster 2024-04-02T04:54:43.927879+0000 mon.a (mon.0) 2685 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: audit 2024-04-02T04:54:43.931044+0000 mon.c (mon.2) 805 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: audit 2024-04-02T04:54:43.931683+0000 mon.a (mon.0) 2686 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: cluster 2024-04-02T04:54:44.110890+0000 mgr.y (mgr.14538) 435 : cluster 0 pgmap v635: 356 pgs: 356 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: audit 2024-04-02T04:54:44.222562+0000 mon.c (mon.2) 806 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: audit 2024-04-02T04:54:44.887255+0000 mon.a (mon.0) 2687 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]': finished 2024-04-02T04:54:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:44 smithi098 bash[23790]: cluster 2024-04-02T04:54:44.891508+0000 mon.a (mon.0) 2688 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-04-02T04:54:45.251 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:45.280 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:45.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:45.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:45.538 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:45.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:45.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:45.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:45.925 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:45.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:46.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:46.053 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:46.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:46.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:46.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: cluster 2024-04-02T04:54:44.751051+0000 osd.3 (osd.3) 285 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:46.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: cluster 2024-04-02T04:54:44.751341+0000 osd.3 (osd.3) 286 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:46.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:44.970447+0000 mon.c (mon.2) 807 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:46.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:44.971154+0000 mon.a (mon.0) 2689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:46.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:44.971873+0000 mon.c (mon.2) 808 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:46.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:44.972482+0000 mon.a (mon.0) 2690 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:46.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:45.224299+0000 mon.c (mon.2) 809 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:45.225437+0000 mon.c (mon.2) 810 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:54:46.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:45 smithi098 bash[23790]: audit 2024-04-02T04:54:45.225962+0000 mon.a (mon.0) 2691 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: cluster 2024-04-02T04:54:44.751051+0000 osd.3 (osd.3) 285 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: cluster 2024-04-02T04:54:44.751341+0000 osd.3 (osd.3) 286 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:44.970447+0000 mon.c (mon.2) 807 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:44.971154+0000 mon.a (mon.0) 2689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:44.971873+0000 mon.c (mon.2) 808 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:44.972482+0000 mon.a (mon.0) 2690 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:45.224299+0000 mon.c (mon.2) 809 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:45.225437+0000 mon.c (mon.2) 810 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[21528]: audit 2024-04-02T04:54:45.225962+0000 mon.a (mon.0) 2691 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:54:46.213 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: cluster 2024-04-02T04:54:44.751051+0000 osd.3 (osd.3) 285 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: cluster 2024-04-02T04:54:44.751341+0000 osd.3 (osd.3) 286 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:44.970447+0000 mon.c (mon.2) 807 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:44.971154+0000 mon.a (mon.0) 2689 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:44.971873+0000 mon.c (mon.2) 808 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:44.972482+0000 mon.a (mon.0) 2690 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-51"}]: dispatch 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:45.224299+0000 mon.c (mon.2) 809 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:45.225437+0000 mon.c (mon.2) 810 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:54:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:45 smithi069 bash[28904]: audit 2024-04-02T04:54:45.225962+0000 mon.a (mon.0) 2691 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:54:46.220 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:46.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:46.419 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:54:46.420 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:54:46.420 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:54:46.420 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:54:46.420 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:54:46.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:46.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.603 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:46.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:46.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:46.732 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:46.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:46.863 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:46.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:45.934239+0000 mon.a (mon.0) 2692 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]': finished 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: cluster 2024-04-02T04:54:45.944149+0000 mon.a (mon.0) 2693 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:45.948905+0000 mon.c (mon.2) 811 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:45.949970+0000 mon.c (mon.2) 812 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:45.951014+0000 mon.c (mon.2) 813 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:46.107730+0000 mon.b (mon.1) 223 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: cluster 2024-04-02T04:54:46.111789+0000 mgr.y (mgr.14538) 436 : cluster 0 pgmap v638: 324 pgs: 324 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:54:46.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:46.424083+0000 mon.a (mon.0) 2694 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:54:46.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[21528]: audit 2024-04-02T04:54:46.425553+0000 mon.a (mon.0) 2695 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:54:46.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:46.996 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:46.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:45.934239+0000 mon.a (mon.0) 2692 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]': finished 2024-04-02T04:54:46.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: cluster 2024-04-02T04:54:45.944149+0000 mon.a (mon.0) 2693 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-04-02T04:54:46.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:45.948905+0000 mon.c (mon.2) 811 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:45.949970+0000 mon.c (mon.2) 812 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:46.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:45.951014+0000 mon.c (mon.2) 813 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:54:46.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:46.107730+0000 mon.b (mon.1) 223 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:54:46.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: cluster 2024-04-02T04:54:46.111789+0000 mgr.y (mgr.14538) 436 : cluster 0 pgmap v638: 324 pgs: 324 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:54:46.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:46.424083+0000 mon.a (mon.0) 2694 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:54:46.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:46 smithi069 bash[28904]: audit 2024-04-02T04:54:46.425553+0000 mon.a (mon.0) 2695 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:54:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:47.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:47.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:45.934239+0000 mon.a (mon.0) 2692 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]': finished 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: cluster 2024-04-02T04:54:45.944149+0000 mon.a (mon.0) 2693 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:45.948905+0000 mon.c (mon.2) 811 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:45.949970+0000 mon.c (mon.2) 812 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:45.951014+0000 mon.c (mon.2) 813 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:46.107730+0000 mon.b (mon.1) 223 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: cluster 2024-04-02T04:54:46.111789+0000 mgr.y (mgr.14538) 436 : cluster 0 pgmap v638: 324 pgs: 324 active+clean; 8.4 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:46.424083+0000 mon.a (mon.0) 2694 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:54:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:46 smithi098 bash[23790]: audit 2024-04-02T04:54:46.425553+0000 mon.a (mon.0) 2695 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:54:47.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:47.252 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:47.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:47.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:47.513 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:47.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:47.641 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:47.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:47.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:47.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:48.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:48.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: cluster 2024-04-02T04:54:46.957129+0000 mon.a (mon.0) 2696 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: audit 2024-04-02T04:54:46.967799+0000 mon.a (mon.0) 2697 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]': finished 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: audit 2024-04-02T04:54:46.967968+0000 mon.a (mon.0) 2698 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: cluster 2024-04-02T04:54:46.972819+0000 mon.a (mon.0) 2699 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: audit 2024-04-02T04:54:46.976804+0000 mon.c (mon.2) 814 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: audit 2024-04-02T04:54:46.984178+0000 mon.c (mon.2) 815 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:48.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[28904]: audit 2024-04-02T04:54:46.984946+0000 mon.a (mon.0) 2700 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: cluster 2024-04-02T04:54:46.957129+0000 mon.a (mon.0) 2696 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: audit 2024-04-02T04:54:46.967799+0000 mon.a (mon.0) 2697 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]': finished 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: audit 2024-04-02T04:54:46.967968+0000 mon.a (mon.0) 2698 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: cluster 2024-04-02T04:54:46.972819+0000 mon.a (mon.0) 2699 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: audit 2024-04-02T04:54:46.976804+0000 mon.c (mon.2) 814 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: audit 2024-04-02T04:54:46.984178+0000 mon.c (mon.2) 815 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:48.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:47 smithi069 bash[21528]: audit 2024-04-02T04:54:46.984946+0000 mon.a (mon.0) 2700 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:48.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: cluster 2024-04-02T04:54:46.957129+0000 mon.a (mon.0) 2696 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: audit 2024-04-02T04:54:46.967799+0000 mon.a (mon.0) 2697 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]': finished 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: audit 2024-04-02T04:54:46.967968+0000 mon.a (mon.0) 2698 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: cluster 2024-04-02T04:54:46.972819+0000 mon.a (mon.0) 2699 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: audit 2024-04-02T04:54:46.976804+0000 mon.c (mon.2) 814 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: audit 2024-04-02T04:54:46.984178+0000 mon.c (mon.2) 815 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:48.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:47 smithi098 bash[23790]: audit 2024-04-02T04:54:46.984946+0000 mon.a (mon.0) 2700 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:48.308 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:48.411 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:48.437 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:48.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:48.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:48.697 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:48.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:48.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:48.886 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:54:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:54:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:54:48.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:48.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:48.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:49.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:49.085 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:49.165 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:48 smithi069 bash[21528]: cluster 2024-04-02T04:54:47.487677+0000 osd.6 (osd.6) 213 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:49.165 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:48 smithi069 bash[21528]: cluster 2024-04-02T04:54:47.488252+0000 osd.6 (osd.6) 214 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:49.165 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:48 smithi069 bash[21528]: audit 2024-04-02T04:54:47.970318+0000 mon.a (mon.0) 2701 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:49.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: cluster 2024-04-02T04:54:47.487677+0000 osd.6 (osd.6) 213 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: cluster 2024-04-02T04:54:47.488252+0000 osd.6 (osd.6) 214 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:47.970318+0000 mon.a (mon.0) 2701 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: cluster 2024-04-02T04:54:47.974973+0000 mon.a (mon.0) 2702 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:48.026149+0000 mon.a (mon.0) 2703 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:48.027437+0000 mon.b (mon.1) 224 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:48.071745+0000 mon.c (mon.2) 816 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:48.072173+0000 mon.a (mon.0) 2704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:48.072740+0000 mon.c (mon.2) 817 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-53"}]: dispatch 2024-04-02T04:54:49.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: audit 2024-04-02T04:54:48.073029+0000 mon.a (mon.0) 2705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-53"}]: dispatch 2024-04-02T04:54:49.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:48 smithi098 bash[23790]: cluster 2024-04-02T04:54:48.112483+0000 mgr.y (mgr.14538) 437 : cluster 0 pgmap v641: 356 pgs: 17 peering, 30 creating+peering, 2 unknown, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:49.211 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:49.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:49.345 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:49.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: cluster 2024-04-02T04:54:47.974973+0000 mon.a (mon.0) 2702 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-04-02T04:54:49.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: audit 2024-04-02T04:54:48.026149+0000 mon.a (mon.0) 2703 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:49.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: audit 2024-04-02T04:54:48.027437+0000 mon.b (mon.1) 224 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:49.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: audit 2024-04-02T04:54:48.071745+0000 mon.c (mon.2) 816 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: audit 2024-04-02T04:54:48.072173+0000 mon.a (mon.0) 2704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: audit 2024-04-02T04:54:48.072740+0000 mon.c (mon.2) 817 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-53"}]: dispatch 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: audit 2024-04-02T04:54:48.073029+0000 mon.a (mon.0) 2705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-53"}]: dispatch 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[21528]: cluster 2024-04-02T04:54:48.112483+0000 mgr.y (mgr.14538) 437 : cluster 0 pgmap v641: 356 pgs: 17 peering, 30 creating+peering, 2 unknown, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: cluster 2024-04-02T04:54:47.487677+0000 osd.6 (osd.6) 213 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: cluster 2024-04-02T04:54:47.488252+0000 osd.6 (osd.6) 214 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:47.970318+0000 mon.a (mon.0) 2701 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: cluster 2024-04-02T04:54:47.974973+0000 mon.a (mon.0) 2702 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:48.026149+0000 mon.a (mon.0) 2703 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:54:49.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:48.027437+0000 mon.b (mon.1) 224 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:49.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:48.071745+0000 mon.c (mon.2) 816 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:49.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:48.072173+0000 mon.a (mon.0) 2704 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:49.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:48.072740+0000 mon.c (mon.2) 817 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-53"}]: dispatch 2024-04-02T04:54:49.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: audit 2024-04-02T04:54:48.073029+0000 mon.a (mon.0) 2705 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-53"}]: dispatch 2024-04-02T04:54:49.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:49 smithi069 bash[28904]: cluster 2024-04-02T04:54:48.112483+0000 mgr.y (mgr.14538) 437 : cluster 0 pgmap v641: 356 pgs: 17 peering, 30 creating+peering, 2 unknown, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:54:49.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:49.578 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:49.602 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:49.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:49.834 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:49.859 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:49.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:49.986 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:49.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:50.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[21528]: cluster 2024-04-02T04:54:49.010560+0000 mon.a (mon.0) 2706 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[21528]: cluster 2024-04-02T04:54:49.022671+0000 mon.a (mon.0) 2707 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 8 pgs peering (PG_AVAILABILITY) 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[21528]: cluster 2024-04-02T04:54:49.895389+0000 mon.a (mon.0) 2708 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[21528]: audit 2024-04-02T04:54:49.905174+0000 mon.c (mon.2) 818 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[21528]: audit 2024-04-02T04:54:49.905708+0000 mon.a (mon.0) 2709 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[28904]: cluster 2024-04-02T04:54:49.010560+0000 mon.a (mon.0) 2706 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[28904]: cluster 2024-04-02T04:54:49.022671+0000 mon.a (mon.0) 2707 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 8 pgs peering (PG_AVAILABILITY) 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[28904]: cluster 2024-04-02T04:54:49.895389+0000 mon.a (mon.0) 2708 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-04-02T04:54:50.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[28904]: audit 2024-04-02T04:54:49.905174+0000 mon.c (mon.2) 818 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:50.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:50 smithi069 bash[28904]: audit 2024-04-02T04:54:49.905708+0000 mon.a (mon.0) 2709 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:50.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:50.219 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:50.245 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:50.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:50.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:50 smithi098 bash[23790]: cluster 2024-04-02T04:54:49.010560+0000 mon.a (mon.0) 2706 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-04-02T04:54:50.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:50 smithi098 bash[23790]: cluster 2024-04-02T04:54:49.022671+0000 mon.a (mon.0) 2707 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 8 pgs peering (PG_AVAILABILITY) 2024-04-02T04:54:50.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:50 smithi098 bash[23790]: cluster 2024-04-02T04:54:49.895389+0000 mon.a (mon.0) 2708 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-04-02T04:54:50.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:50 smithi098 bash[23790]: audit 2024-04-02T04:54:49.905174+0000 mon.c (mon.2) 818 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:50.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:50 smithi098 bash[23790]: audit 2024-04-02T04:54:49.905708+0000 mon.a (mon.0) 2709 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:50.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:50.506 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:50.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-02T04:54:50.609 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:50.630 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:50.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:50.761 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:50.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:50.893 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:50.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:50.996 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:51.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:51.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:51.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:50.109100+0000 mon.b (mon.1) 225 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: cluster 2024-04-02T04:54:50.113264+0000 mgr.y (mgr.14538) 438 : cluster 0 pgmap v644: 356 pgs: 32 unknown, 17 peering, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:50.117046+0000 mon.a (mon.0) 2710 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:50.892413+0000 mon.a (mon.0) 2711 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:50.892644+0000 mon.a (mon.0) 2712 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: cluster 2024-04-02T04:54:50.903567+0000 mon.a (mon.0) 2713 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:50.909561+0000 mon.c (mon.2) 819 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:54:51.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:51.003611+0000 mon.c (mon.2) 820 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:51.004077+0000 mon.a (mon.0) 2714 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:51.004817+0000 mon.c (mon.2) 821 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-55"}]: dispatch 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[21528]: audit 2024-04-02T04:54:51.005177+0000 mon.a (mon.0) 2715 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-55"}]: dispatch 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:50.109100+0000 mon.b (mon.1) 225 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: cluster 2024-04-02T04:54:50.113264+0000 mgr.y (mgr.14538) 438 : cluster 0 pgmap v644: 356 pgs: 32 unknown, 17 peering, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:50.117046+0000 mon.a (mon.0) 2710 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:50.892413+0000 mon.a (mon.0) 2711 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:50.892644+0000 mon.a (mon.0) 2712 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: cluster 2024-04-02T04:54:50.903567+0000 mon.a (mon.0) 2713 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-04-02T04:54:51.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:50.909561+0000 mon.c (mon.2) 819 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:54:51.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:51.003611+0000 mon.c (mon.2) 820 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:51.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:51.004077+0000 mon.a (mon.0) 2714 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:51.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:51.004817+0000 mon.c (mon.2) 821 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-55"}]: dispatch 2024-04-02T04:54:51.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:51 smithi069 bash[28904]: audit 2024-04-02T04:54:51.005177+0000 mon.a (mon.0) 2715 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-55"}]: dispatch 2024-04-02T04:54:51.279 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:51.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:51.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:50.109100+0000 mon.b (mon.1) 225 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-02T04:54:51.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: cluster 2024-04-02T04:54:50.113264+0000 mgr.y (mgr.14538) 438 : cluster 0 pgmap v644: 356 pgs: 32 unknown, 17 peering, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:54:51.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:50.117046+0000 mon.a (mon.0) 2710 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-02T04:54:51.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:50.892413+0000 mon.a (mon.0) 2711 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:51.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:50.892644+0000 mon.a (mon.0) 2712 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-02T04:54:51.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: cluster 2024-04-02T04:54:50.903567+0000 mon.a (mon.0) 2713 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-04-02T04:54:51.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:50.909561+0000 mon.c (mon.2) 819 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:54:51.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:51.003611+0000 mon.c (mon.2) 820 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:51.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:51.004077+0000 mon.a (mon.0) 2714 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:51.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:51.004817+0000 mon.c (mon.2) 821 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-55"}]: dispatch 2024-04-02T04:54:51.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:51 smithi098 bash[23790]: audit 2024-04-02T04:54:51.005177+0000 mon.a (mon.0) 2715 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-55"}]: dispatch 2024-04-02T04:54:51.409 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:51.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:51.540 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:51.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:51.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:51.795 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:51.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:51.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:52.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:52.056 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:52.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:52.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:52.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:52.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:52.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:52.553 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:52 smithi098 bash[23790]: cluster 2024-04-02T04:54:51.008681+0000 mon.a (mon.0) 2716 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:52.553 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:52 smithi098 bash[23790]: cluster 2024-04-02T04:54:51.903039+0000 mon.a (mon.0) 2717 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-04-02T04:54:52.569 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:52.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:52.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:52 smithi069 bash[21528]: cluster 2024-04-02T04:54:51.008681+0000 mon.a (mon.0) 2716 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:52.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:52 smithi069 bash[21528]: cluster 2024-04-02T04:54:51.903039+0000 mon.a (mon.0) 2717 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-04-02T04:54:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:52 smithi069 bash[28904]: cluster 2024-04-02T04:54:51.008681+0000 mon.a (mon.0) 2716 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:52.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:52 smithi069 bash[28904]: cluster 2024-04-02T04:54:51.903039+0000 mon.a (mon.0) 2717 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-04-02T04:54:52.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:52.830 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:52.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:52.961 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:52.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:53.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:53.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:53.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:53.219 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:53.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:53.345 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:53.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:53.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:54:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:54:53.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:53 smithi098 bash[23790]: cluster 2024-04-02T04:54:52.114237+0000 mgr.y (mgr.14538) 439 : cluster 0 pgmap v647: 324 pgs: 17 peering, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 63 B/s, 0 objects/s recovering 2024-04-02T04:54:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:53 smithi098 bash[23790]: cluster 2024-04-02T04:54:52.914237+0000 mon.a (mon.0) 2718 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-04-02T04:54:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:53 smithi098 bash[23790]: audit 2024-04-02T04:54:52.931761+0000 mon.c (mon.2) 822 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:53 smithi098 bash[23790]: audit 2024-04-02T04:54:52.933361+0000 mon.a (mon.0) 2719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:53.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:53.606 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:53.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:53.733 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[21528]: cluster 2024-04-02T04:54:52.114237+0000 mgr.y (mgr.14538) 439 : cluster 0 pgmap v647: 324 pgs: 17 peering, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 63 B/s, 0 objects/s recovering 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[21528]: cluster 2024-04-02T04:54:52.914237+0000 mon.a (mon.0) 2718 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[21528]: audit 2024-04-02T04:54:52.931761+0000 mon.c (mon.2) 822 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[21528]: audit 2024-04-02T04:54:52.933361+0000 mon.a (mon.0) 2719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[28904]: cluster 2024-04-02T04:54:52.114237+0000 mgr.y (mgr.14538) 439 : cluster 0 pgmap v647: 324 pgs: 17 peering, 307 active+clean; 8.4 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 63 B/s, 0 objects/s recovering 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[28904]: cluster 2024-04-02T04:54:52.914237+0000 mon.a (mon.0) 2718 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-04-02T04:54:53.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[28904]: audit 2024-04-02T04:54:52.931761+0000 mon.c (mon.2) 822 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:53 smithi069 bash[28904]: audit 2024-04-02T04:54:52.933361+0000 mon.a (mon.0) 2719 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:53.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:53.865 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:53.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:53.986 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:53.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:53.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:54.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 -- 172.21.15.69:0/751963535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c06ab00 msgr2=0x7f046c06aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 --2- 172.21.15.69:0/751963535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c06ab00 0x7f046c06aee0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f045c0098f0 tx=0x7f045c009c00 comp rx=0 tx=0).stop 2024-04-02T04:54:54.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 -- 172.21.15.69:0/751963535 shutdown_connections 2024-04-02T04:54:54.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 --2- 172.21.15.69:0/751963535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c107680 0x7f046c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 --2- 172.21.15.69:0/751963535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c06ab00 0x7f046c06aee0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 --2- 172.21.15.69:0/751963535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f046c06b4b0 0x7f046c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 -- 172.21.15.69:0/751963535 >> 172.21.15.69:0/751963535 conn(0x7f046c078240 msgr2=0x7f046c078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 -- 172.21.15.69:0/751963535 shutdown_connections 2024-04-02T04:54:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.044+0000 7f04709a5700 1 -- 172.21.15.69:0/751963535 wait complete. 2024-04-02T04:54:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 librados: starting msgr at 2024-04-02T04:54:54.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 librados: starting objecter 2024-04-02T04:54:54.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 Processor -- start 2024-04-02T04:54:54.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- start start 2024-04-02T04:54:54.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 librados: setting wanted keys 2024-04-02T04:54:54.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 librados: calling monclient init 2024-04-02T04:54:54.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 0x7f046c1011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f046c06b4b0 0x7f046c101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c107680 0x7f046c101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f046c111b50 con 0x7f046c06ab00 2024-04-02T04:54:54.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f046c1119d0 con 0x7f046c06b4b0 2024-04-02T04:54:54.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f046c111cd0 con 0x7f046c107680 2024-04-02T04:54:54.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 0x7f046c1011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 0x7f046c1011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35780/0 (socket says 172.21.15.69:35780) 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c107680 0x7f046c101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 -- 172.21.15.69:0/3503362957 learned_addr learned my addr 172.21.15.69:0/3503362957 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 -- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c107680 msgr2=0x7f046c101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c107680 0x7f046c101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 -- 172.21.15.69:0/3503362957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f046c06b4b0 msgr2=0x7f046c101730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f046c06b4b0 0x7f046c101730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 -- 172.21.15.69:0/3503362957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c1a6620 con 0x7f046c06ab00 2024-04-02T04:54:54.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f046a59c700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 0x7f046c1011f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f045c009230 tx=0x7f045c00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f045b7fe700 10 client.?.objecter ms_handle_connect 0x7f046c06ab00 2024-04-02T04:54:54.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f045b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:54.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c00b780 con 0x7f046c06ab00 2024-04-02T04:54:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046c1a68b0 con 0x7f046c06ab00 2024-04-02T04:54:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 10 client.16452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046c1a6d00 con 0x7f046c06ab00 2024-04-02T04:54:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 librados: init done 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f045c0054c0 con 0x7f046c06ab00 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c003a50 con 0x7f046c06ab00 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.048+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c005150 con 0x7f046c06ab00 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f045c00b920 con 0x7f046c06ab00 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045406f620 0x7f0454071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f0469d9b700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045406f620 0x7f0454071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f0469d9b700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045406f620 0x7f0454071ae0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f046c108a00 tx=0x7f046000b3f0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 10 client.16452.objecter ms_handle_connect 0x7f045406f620 2024-04-02T04:54:54.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(446..446 src has 74..446) ==== 10938+0+0 (secure 0 0 0) 0x7f045c0902b0 con 0x7f046c06ab00 2024-04-02T04:54:54.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 10 client.16452.objecter ms_dispatch 0x7f046c001550 osd_map(446..446 src has 74..446) 2024-04-02T04:54:54.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 3 client.16452.objecter handle_osd_map got epochs [446,446] > 0 2024-04-02T04:54:54.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 3 client.16452.objecter handle_osd_map decoding full epoch 446 2024-04-02T04:54:54.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.052+0000 7f045b7fe700 20 client.16452.objecter dump_active .. 0 homeless 2024-04-02T04:54:54.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.056+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f045c013030 con 0x7f046c06ab00 2024-04-02T04:54:54.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:54.115 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 -- 172.21.15.69:0/222670554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe734101cb0 msgr2=0x7fe734102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 --2- 172.21.15.69:0/222670554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe734101cb0 0x7fe734102090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe7240099c0 tx=0x7fe724009cd0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 -- 172.21.15.69:0/222670554 shutdown_connections 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 --2- 172.21.15.69:0/222670554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73410ed50 0x7fe734111140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 --2- 172.21.15.69:0/222670554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe734101cb0 0x7fe734102090 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 --2- 172.21.15.69:0/222670554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7341025d0 0x7fe73410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 -- 172.21.15.69:0/222670554 >> 172.21.15.69:0/222670554 conn(0x7fe7340771e0 msgr2=0x7fe7340775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 -- 172.21.15.69:0/222670554 shutdown_connections 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.120+0000 7fe7392cf700 1 -- 172.21.15.69:0/222670554 wait complete. 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 librados: starting msgr at 2024-04-02T04:54:54.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 librados: starting objecter 2024-04-02T04:54:54.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 Processor -- start 2024-04-02T04:54:54.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 -- start start 2024-04-02T04:54:54.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 librados: setting wanted keys 2024-04-02T04:54:54.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 librados: calling monclient init 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe734101cb0 0x7fe73419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7341025d0 0x7fe73419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe73410ed50 0x7fe73419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe734113ca0 con 0x7fe734101cb0 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe734113b20 con 0x7fe73410ed50 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe734113e20 con 0x7fe7341025d0 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7341025d0 0x7fe73419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7341025d0 0x7fe73419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57942/0 (socket says 172.21.15.69:57942) 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 -- 172.21.15.69:0/2405271954 learned_addr learned my addr 172.21.15.69:0/2405271954 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe734101cb0 0x7fe73419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe73410ed50 msgr2=0x7fe73419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe73410ed50 0x7fe73419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe734101cb0 msgr2=0x7fe73419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe734101cb0 0x7fe73419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73419f280 con 0x7fe7341025d0 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7341025d0 0x7fe73419a7b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe72800d950 tx=0x7fe72800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe71bfff700 10 client.?.objecter ms_handle_connect 0x7fe7341025d0 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe71bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728016070 con 0x7fe7341025d0 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe728009d40 con 0x7fe7341025d0 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728010e60 con 0x7fe7341025d0 2024-04-02T04:54:54.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73419f570 con 0x7fe7341025d0 2024-04-02T04:54:54.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 10 client.26552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:54.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7341a6df0 con 0x7fe7341025d0 2024-04-02T04:54:54.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe7392cf700 1 librados: init done 2024-04-02T04:54:54.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe73404ff00 con 0x7fe7341025d0 2024-04-02T04:54:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.124+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe728019070 con 0x7fe7341025d0 2024-04-02T04:54:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe71c06f5e0 0x7fe71c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(446..446 src has 74..446) ==== 10938+0+0 (secure 0 0 0) 0x7fe72808e8a0 con 0x7fe7341025d0 2024-04-02T04:54:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 osd_map(446..446 src has 74..446) 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 3 client.26552.objecter handle_osd_map got epochs [446,446] > 0 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 3 client.26552.objecter handle_osd_map decoding full epoch 446 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe71c06f5e0 0x7fe71c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 20 client.26552.objecter dump_active .. 0 homeless 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v446) ==== 54+0+16 (secure 0 0 0) 0x7fe728093220 con 0x7fe7341025d0 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 10 client.26552.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 1 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe7341a6fc0 0x7fe7341a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 1 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 0 1 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 4 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 10 client.26552.objecter _send_command 1 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7340635b0 con 0x7fe7341a6fc0 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe71c06f5e0 0x7fe71c071aa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe72400b580 tx=0x7fe724019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe71c06f5e0 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe7337fe700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe7341a6fc0 0x7fe7341a93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341a6fc0 2024-04-02T04:54:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe7337fe700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe7341a6fc0 0x7fe7341a93b0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7340635b0 con 0x7fe7341a6fc0 2024-04-02T04:54:54.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 1: 0 ) 2024-04-02T04:54:54.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 10 client.26552.objecter _finish_command 1 = system:0 2024-04-02T04:54:54.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 4 2024-04-02T04:54:54.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.128+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 0 1 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 10 client.26552.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 2 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 2 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe73419b5d0 osd=0 3 2024-04-02T04:54:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 0 2 2024-04-02T04:54:54.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 4 2024-04-02T04:54:54.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 10 client.26552.objecter _send_command 2 2024-04-02T04:54:54.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe734062b20 con 0x7fe7341a6fc0 2024-04-02T04:54:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe734062b20 con 0x7fe7341a6fc0 2024-04-02T04:54:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe71bfff700 10 client.26552.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 4 2024-04-02T04:54:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 0 2 2024-04-02T04:54:54.150 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:54:54.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 10 client.26552.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe7341094b0 0x7fe7341aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 1 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 4 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 10 client.26552.objecter _send_command 3 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe7341af8e0 con 0x7fe7341094b0 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe7341094b0 0x7fe7341aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341094b0 2024-04-02T04:54:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.148+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe7341094b0 0x7fe7341aca80 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.152+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7341af8e0 con 0x7fe7341094b0 2024-04-02T04:54:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.152+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 3: 0 ) 2024-04-02T04:54:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.152+0000 7fe71bfff700 10 client.26552.objecter _finish_command 3 = system:0 2024-04-02T04:54:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.152+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 4 2024-04-02T04:54:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.152+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 1 3 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/1666934620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70074d90 msgr2=0x7f6a70112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- 172.21.15.69:0/1666934620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70074d90 0x7f6a70112f30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009990 tx=0x7f6a64009ca0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/1666934620 shutdown_connections 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- 172.21.15.69:0/1666934620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a70113470 0x7f6a70115860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- 172.21.15.69:0/1666934620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074470 0x7f6a70074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- 172.21.15.69:0/1666934620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70074d90 0x7f6a70112f30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/1666934620 >> 172.21.15.69:0/1666934620 conn(0x7f6a700793a0 msgr2=0x7f6a700797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:54.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/1666934620 shutdown_connections 2024-04-02T04:54:54.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/1666934620 wait complete. 2024-04-02T04:54:54.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 librados: starting msgr at 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 librados: starting objecter 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 Processor -- start 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- start start 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 librados: setting wanted keys 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 librados: calling monclient init 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a70074470 0x7f6a7019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 0x7f6a7019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70113470 0x7f6a701a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a701180f0 con 0x7f6a70074d90 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a70117f70 con 0x7f6a70074470 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6a70118270 con 0x7f6a70113470 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 0x7f6a7019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 0x7f6a7019ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35822/0 (socket says 172.21.15.69:35822) 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 -- 172.21.15.69:0/2109031040 learned_addr learned my addr 172.21.15.69:0/2109031040 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6f7fe700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70113470 0x7f6a701a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 -- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70113470 msgr2=0x7f6a701a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70113470 0x7f6a701a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 -- 172.21.15.69:0/2109031040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a70074470 msgr2=0x7f6a7019e680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a70074470 0x7f6a7019e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 -- 172.21.15.69:0/2109031040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a701a3690 con 0x7f6a70074d90 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6f7fe700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70113470 0x7f6a701a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a6e7fc700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 0x7f6a7019ebc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6a64000c00 tx=0x7f6a640047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a74933700 10 client.?.objecter ms_handle_connect 0x7f6a70074d90 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a74933700 10 client.?.objecter resend_mon_ops 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64027070 con 0x7f6a70074d90 2024-04-02T04:54:54.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6a64022470 con 0x7f6a70074d90 2024-04-02T04:54:54.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64017650 con 0x7f6a70074d90 2024-04-02T04:54:54.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a701a3920 con 0x7f6a70074d90 2024-04-02T04:54:54.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 10 client.16464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:54:54.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a701ab200 con 0x7f6a70074d90 2024-04-02T04:54:54.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.159+0000 7f6a75935700 1 librados: init done 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a70075b60 con 0x7f6a70074d90 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6a6402a070 con 0x7f6a70074d90 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a5806f5e0 0x7f6a58071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(446..446 src has 74..446) ==== 10938+0+0 (secure 0 0 0) 0x7f6a6408fd90 con 0x7f6a70074d90 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 10 client.16464.objecter ms_dispatch 0x7f6a70001550 osd_map(446..446 src has 74..446) 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 3 client.16464.objecter handle_osd_map got epochs [446,446] > 0 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 3 client.16464.objecter handle_osd_map decoding full epoch 446 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a6effd700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a5806f5e0 0x7f6a58071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 20 client.16464.objecter dump_active .. 0 homeless 2024-04-02T04:54:54.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a6effd700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a5806f5e0 0x7f6a58071aa0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009b20 tx=0x7f6a60009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.163+0000 7f6a74933700 10 client.16464.objecter ms_handle_connect 0x7f6a5806f5e0 2024-04-02T04:54:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.167+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6a64013070 con 0x7f6a70074d90 2024-04-02T04:54:54.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 10 client.26552.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 4 2024-04-02T04:54:54.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 4 2024-04-02T04:54:54.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe734062f20 osd=1 3 2024-04-02T04:54:54.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 1 4 2024-04-02T04:54:54.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 4 2024-04-02T04:54:54.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 10 client.26552.objecter _send_command 4 2024-04-02T04:54:54.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe734108a20 con 0x7fe7341094b0 2024-04-02T04:54:54.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe734108a20 con 0x7fe7341094b0 2024-04-02T04:54:54.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe71bfff700 10 client.26552.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 4 2024-04-02T04:54:54.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 1 4 2024-04-02T04:54:54.174 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 10 client.26552.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 5 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe7341ae140 0x7fe7341ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 5 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 2 5 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 4 2024-04-02T04:54:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 10 client.26552.objecter _send_command 5 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe7341aa7d0 con 0x7fe7341ae140 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe7341ae140 0x7fe7341ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341ae140 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.171+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe7341ae140 0x7fe7341ae580 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.175+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7341aa7d0 con 0x7fe7341ae140 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.175+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 5: 0 ) 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.175+0000 7fe71bfff700 10 client.26552.objecter _finish_command 5 = system:0 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.175+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 4 2024-04-02T04:54:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.175+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 2 5 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 10 client.26552.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 6 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 6 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341aa0c0 osd=2 3 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 2 6 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 4 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 10 client.26552.objecter _send_command 6 2024-04-02T04:54:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.191+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7341a9b40 con 0x7fe7341ae140 2024-04-02T04:54:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7341a9b40 con 0x7fe7341ae140 2024-04-02T04:54:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 10 client.26552.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 4 2024-04-02T04:54:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 2 6 2024-04-02T04:54:54.195 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:54:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 10 client.26552.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 7 2024-04-02T04:54:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe7341b1aa0 0x7fe7341b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 7 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 3 7 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 4 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 10 client.26552.objecter _send_command 7 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe7341a9f00 con 0x7fe7341b1aa0 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe7337fe700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe7341b1aa0 0x7fe7341b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341b1aa0 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe7337fe700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe7341b1aa0 0x7fe7341b3e90 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe72c01a410 con 0x7fe7341b1aa0 2024-04-02T04:54:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 7: 0 ) 2024-04-02T04:54:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 10 client.26552.objecter _finish_command 7 = system:0 2024-04-02T04:54:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 4 2024-04-02T04:54:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.195+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 3 7 2024-04-02T04:54:54.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 10 client.26552.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 8 2024-04-02T04:54:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 8 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341a9d90 osd=3 3 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 3 8 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 4 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 10 client.26552.objecter _send_command 8 2024-04-02T04:54:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.227+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7341aee30 con 0x7fe7341b1aa0 2024-04-02T04:54:54.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7341aee30 con 0x7fe7341b1aa0 2024-04-02T04:54:54.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe71bfff700 10 client.26552.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 4 2024-04-02T04:54:54.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 3 8 2024-04-02T04:54:54.234 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:54:54.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 10 client.26552.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 9 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe7341adca0 0x7fe7341b7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 9 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 4 9 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 4 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 10 client.26552.objecter _send_command 9 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe7341ba3c0 con 0x7fe7341adca0 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.231+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe7341adca0 0x7fe7341b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341adca0 2024-04-02T04:54:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe7341adca0 0x7fe7341b7560 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7341ba3c0 con 0x7fe7341adca0 2024-04-02T04:54:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 9: 0 ) 2024-04-02T04:54:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe71bfff700 10 client.26552.objecter _finish_command 9 = system:0 2024-04-02T04:54:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 4 2024-04-02T04:54:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.235+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 4 9 2024-04-02T04:54:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:54.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 10 client.26552.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 10 2024-04-02T04:54:54.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 10 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341af090 osd=4 3 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 4 10 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 4 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 10 client.26552.objecter _send_command 10 2024-04-02T04:54:54.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.267+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7341b7aa0 con 0x7fe7341adca0 2024-04-02T04:54:54.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7341b7aa0 con 0x7fe7341adca0 2024-04-02T04:54:54.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe71bfff700 10 client.26552.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 4 2024-04-02T04:54:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 4 10 2024-04-02T04:54:54.275 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:54:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 10 client.26552.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 11 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe7341b8120 0x7fe7341ba510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 11 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 5 11 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 4 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 10 client.26552.objecter _send_command 11 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.271+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe7341bd520 con 0x7fe7341b8120 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe7341b8120 0x7fe7341ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341b8120 2024-04-02T04:54:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe7327fc700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe7341b8120 0x7fe7341ba510 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7341bd520 con 0x7fe7341b8120 2024-04-02T04:54:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 11: 0 ) 2024-04-02T04:54:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe71bfff700 10 client.26552.objecter _finish_command 11 = system:0 2024-04-02T04:54:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 4 2024-04-02T04:54:54.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.275+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 5 11 2024-04-02T04:54:54.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 10 client.26552.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 12 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 12 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341b7aa0 osd=5 3 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 5 12 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 4 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 10 client.26552.objecter _send_command 12 2024-04-02T04:54:54.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.307+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7341bc2f0 con 0x7fe7341b8120 2024-04-02T04:54:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7341bc2f0 con 0x7fe7341b8120 2024-04-02T04:54:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 10 client.26552.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 4 2024-04-02T04:54:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 5 12 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 10 client.26552.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 13 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe7341bca10 0x7fe7341bce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 13 2024-04-02T04:54:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 6 13 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 4 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 10 client.26552.objecter _send_command 13 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe7341c0940 con 0x7fe7341bca10 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe7337fe700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe7341bca10 0x7fe7341bce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341bca10 2024-04-02T04:54:54.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe7337fe700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe7341bca10 0x7fe7341bce70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7341c0940 con 0x7fe7341bca10 2024-04-02T04:54:54.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 13: 0 ) 2024-04-02T04:54:54.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 10 client.26552.objecter _finish_command 13 = system:0 2024-04-02T04:54:54.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 4 2024-04-02T04:54:54.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.311+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 6 13 2024-04-02T04:54:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.319+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd out", "ids": ["5"]} v 0) -- 0x7f044c005710 con 0x7f046c06ab00 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 10 client.26552.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 14 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 14 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341bc2f0 osd=6 3 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 6 14 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 4 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 10 client.26552.objecter _send_command 14 2024-04-02T04:54:54.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.331+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7341c0940 con 0x7fe7341bca10 2024-04-02T04:54:54.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7341c0940 con 0x7fe7341bca10 2024-04-02T04:54:54.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe71bfff700 10 client.26552.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 4 2024-04-02T04:54:54.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 6 14 2024-04-02T04:54:54.336 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 10 client.26552.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 15 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fe7341bfcd0 0x7fe7341c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 15 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 7 15 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 4 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 10 client.26552.objecter _send_command 15 2024-04-02T04:54:54.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe7341c3cb0 con 0x7fe7341bfcd0 2024-04-02T04:54:54.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fe7341bfcd0 0x7fe7341c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:54:54.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe71bfff700 10 client.26552.objecter ms_handle_connect 0x7fe7341bfcd0 2024-04-02T04:54:54.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.335+0000 7fe732ffd700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fe7341bfcd0 0x7fe7341c0130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:54:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.339+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7341c3cb0 con 0x7fe7341bfcd0 2024-04-02T04:54:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.339+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 15: 0 ) 2024-04-02T04:54:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.339+0000 7fe71bfff700 10 client.26552.objecter _finish_command 15 = system:0 2024-04-02T04:54:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.339+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 4 2024-04-02T04:54:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.339+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 7 15 2024-04-02T04:54:54.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:54.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 client.26552.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign -1 16 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter _get_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_remove -1 16 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter get_session s=0x7fe7341c0940 osd=7 3 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 15 client.26552.objecter _session_command_op_assign 7 16 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 4 2024-04-02T04:54:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 client.26552.objecter _send_command 16 2024-04-02T04:54:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7341c2b30 con 0x7fe7341bfcd0 2024-04-02T04:54:54.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe71bfff700 1 -- 172.21.15.69:0/2405271954 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7341c2b30 con 0x7fe7341bfcd0 2024-04-02T04:54:54.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe71bfff700 10 client.26552.objecter ms_dispatch 0x7fe734001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:54:54.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe71bfff700 10 client.26552.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:54:54.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe71bfff700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 4 2024-04-02T04:54:54.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe71bfff700 15 client.26552.objecter _session_command_op_remove 7 16 2024-04-02T04:54:54.359 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 librados: watch_flush enter 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 librados: watch_flush exit 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.0 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe7341a6fc0 msgr2=0x7fe7341a93b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fe7341a6fc0 0x7fe7341a93b0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe73419b5d0 osd=0 1 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.1 2024-04-02T04:54:54.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe7341094b0 msgr2=0x7fe7341aca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fe7341094b0 0x7fe7341aca80 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe734062f20 osd=1 1 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.2 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe7341ae140 msgr2=0x7fe7341ae580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fe7341ae140 0x7fe7341ae580 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.355+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341aa0c0 osd=2 1 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.3 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe7341b1aa0 msgr2=0x7fe7341b3e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fe7341b1aa0 0x7fe7341b3e90 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341a9d90 osd=3 1 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.4 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe7341adca0 msgr2=0x7fe7341b7560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fe7341adca0 0x7fe7341b7560 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341af090 osd=4 1 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.5 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe7341b8120 msgr2=0x7fe7341ba510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fe7341b8120 0x7fe7341ba510 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341b7aa0 osd=5 1 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.6 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe7341bca10 msgr2=0x7fe7341bce70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fe7341bca10 0x7fe7341bce70 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341bc2f0 osd=6 1 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 10 client.26552.objecter close_session for osd.7 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fe7341bfcd0 msgr2=0x7fe7341c0130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fe7341bfcd0 0x7fe7341c0130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 20 client.26552.objecter put_session s=0x7fe7341c0940 osd=7 1 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 20 client.26552.objecter shutdown clearing up homeless session... 2024-04-02T04:54:54.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 10 client.26552.objecter successfully canceled tick 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe7337fe700 1 -- 172.21.15.69:0/2405271954 reap_dead start 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe7337fe700 1 -- 172.21.15.69:0/2405271954 reap_dead start 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe71c06f5e0 msgr2=0x7fe71c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe71c06f5e0 0x7fe71c071aa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe72400b580 tx=0x7fe724019040 comp rx=0 tx=0).stop 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7341025d0 msgr2=0x7fe73419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 --2- 172.21.15.69:0/2405271954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7341025d0 0x7fe73419a7b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe72800d950 tx=0x7fe72800dc60 comp rx=0 tx=0).stop 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe7337fe700 1 -- 172.21.15.69:0/2405271954 reap_dead start 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe7337fe700 1 -- 172.21.15.69:0/2405271954 reap_dead start 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe7337fe700 1 -- 172.21.15.69:0/2405271954 reap_dead start 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 shutdown_connections 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 >> 172.21.15.69:0/2405271954 conn(0x7fe7340771e0 msgr2=0x7fe734100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 shutdown_connections 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 -- 172.21.15.69:0/2405271954 wait complete. 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.359+0000 7fe719ffb700 1 librados: shutdown 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:54:54.364 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:54:54.368 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.415+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f6a7019f9e0 con 0x7f6a70074d90 2024-04-02T04:54:54.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:54.498 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:54.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: audit 2024-04-02T04:54:53.273483+0000 mgr.y (mgr.14538) 440 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: audit 2024-04-02T04:54:53.907013+0000 mon.a (mon.0) 2720 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: cluster 2024-04-02T04:54:53.912794+0000 mon.a (mon.0) 2721 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: audit 2024-04-02T04:54:54.110938+0000 mon.b (mon.1) 226 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: audit 2024-04-02T04:54:54.116925+0000 mon.a (mon.0) 2722 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: audit 2024-04-02T04:54:54.131699+0000 mon.c (mon.2) 823 : audit 0 from='client.? 172.21.15.69:0/2405271954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: audit 2024-04-02T04:54:54.324295+0000 mon.a (mon.0) 2723 : audit 1 from='client.? 172.21.15.69:0/3503362957' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[28904]: cluster 2024-04-02T04:54:54.324369+0000 mon.a (mon.0) 2724 : cluster 1 Client client.admin marked osd.5 out, while it was still marked up 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: debug 2024-04-02T04:54:54.319+0000 7f02343d7700 -1 mon.a@0(leader).osd e446 definitely_dead 0 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: audit 2024-04-02T04:54:53.273483+0000 mgr.y (mgr.14538) 440 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: audit 2024-04-02T04:54:53.907013+0000 mon.a (mon.0) 2720 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:54.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: cluster 2024-04-02T04:54:53.912794+0000 mon.a (mon.0) 2721 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-04-02T04:54:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: audit 2024-04-02T04:54:54.110938+0000 mon.b (mon.1) 226 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-02T04:54:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: audit 2024-04-02T04:54:54.116925+0000 mon.a (mon.0) 2722 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-02T04:54:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: audit 2024-04-02T04:54:54.131699+0000 mon.c (mon.2) 823 : audit 0 from='client.? 172.21.15.69:0/2405271954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: audit 2024-04-02T04:54:54.324295+0000 mon.a (mon.0) 2723 : audit 1 from='client.? 172.21.15.69:0/3503362957' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-02T04:54:54.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:54 smithi069 bash[21528]: cluster 2024-04-02T04:54:54.324369+0000 mon.a (mon.0) 2724 : cluster 1 Client client.admin marked osd.5 out, while it was still marked up 2024-04-02T04:54:54.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:54.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: audit 2024-04-02T04:54:53.273483+0000 mgr.y (mgr.14538) 440 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: audit 2024-04-02T04:54:53.907013+0000 mon.a (mon.0) 2720 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: cluster 2024-04-02T04:54:53.912794+0000 mon.a (mon.0) 2721 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: audit 2024-04-02T04:54:54.110938+0000 mon.b (mon.1) 226 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: audit 2024-04-02T04:54:54.116925+0000 mon.a (mon.0) 2722 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: audit 2024-04-02T04:54:54.131699+0000 mon.c (mon.2) 823 : audit 0 from='client.? 172.21.15.69:0/2405271954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: audit 2024-04-02T04:54:54.324295+0000 mon.a (mon.0) 2723 : audit 1 from='client.? 172.21.15.69:0/3503362957' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-02T04:54:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:54 smithi098 bash[23790]: cluster 2024-04-02T04:54:54.324369+0000 mon.a (mon.0) 2724 : cluster 1 Client client.admin marked osd.5 out, while it was still marked up 2024-04-02T04:54:54.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:54.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:54.884 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:54.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.887+0000 7f6a74933700 1 -- 172.21.15.69:0/2109031040 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v447) ==== 97+0+0 (secure 0 0 0) 0x7f6a6405c660 con 0x7f6a70074d90 2024-04-02T04:54:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.887+0000 7f045b7fe700 1 -- 172.21.15.69:0/3503362957 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["5"]}]=0 marked out osd.5. v447) ==== 87+0+0 (secure 0 0 0) 0x7f045c05cb80 con 0x7f046c06ab00 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.887+0000 7f04709a5700 10 librados: watch_flush enter 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.887+0000 7f04709a5700 10 librados: watch_flush exit 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.887+0000 7f04709a5700 20 client.16452.objecter shutdown clearing up homeless session... 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.887+0000 7f04709a5700 10 client.16452.objecter successfully canceled tick 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045406f620 msgr2=0x7f0454071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045406f620 0x7f0454071ae0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f046c108a00 tx=0x7f046000b3f0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 msgr2=0x7f046c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 0x7f046c1011f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f045c009230 tx=0x7f045c00b560 comp rx=0 tx=0).stop 2024-04-02T04:54:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 shutdown_connections 2024-04-02T04:54:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045406f620 0x7f0454071ae0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c107680 0x7f046c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c06ab00 0x7f046c1011f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 --2- 172.21.15.69:0/3503362957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f046c06b4b0 0x7f046c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 >> 172.21.15.69:0/3503362957 conn(0x7f046c078240 msgr2=0x7f046c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 shutdown_connections 2024-04-02T04:54:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 10 librados: watch_flush enter 2024-04-02T04:54:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 10 librados: watch_flush exit 2024-04-02T04:54:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 20 client.16464.objecter shutdown clearing up homeless session... 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 10 client.16464.objecter successfully canceled tick 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a5806f5e0 msgr2=0x7f6a58071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a5806f5e0 0x7f6a58071aa0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009b20 tx=0x7f6a60009260 comp rx=0 tx=0).stop 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 msgr2=0x7f6a7019ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 0x7f6a7019ebc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6a64000c00 tx=0x7f6a640047b0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 shutdown_connections 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a5806f5e0 0x7f6a58071aa0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a70113470 0x7f6a701a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a70074470 0x7f6a7019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 --2- 172.21.15.69:0/2109031040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70074d90 0x7f6a7019ebc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 >> 172.21.15.69:0/2109031040 conn(0x7f6a700793a0 msgr2=0x7f6a7010ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 shutdown_connections 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 -- 172.21.15.69:0/2109031040 wait complete. 2024-04-02T04:54:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f6a75935700 1 librados: shutdown 2024-04-02T04:54:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 -- 172.21.15.69:0/3503362957 wait complete. 2024-04-02T04:54:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:54:54.891+0000 7f04709a5700 1 librados: shutdown 2024-04-02T04:54:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:54:54.911 INFO:teuthology.orchestra.run.smithi069.stderr:marked out osd.5. 2024-04-02T04:54:54.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:55.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:55.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:55.260 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:55.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:55.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:55.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: cluster 2024-04-02T04:54:54.115143+0000 mgr.y (mgr.14538) 441 : cluster 0 pgmap v650: 355 pgs: 2 creating+activating, 23 creating+peering, 6 unknown, 324 active+clean; 8.4 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 255 B/s, 3 objects/s recovering 2024-04-02T04:54:55.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.421092+0000 mon.a (mon.0) 2725 : audit 1 from='client.? 172.21.15.69:0/2109031040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:55.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.741446+0000 mon.c (mon.2) 824 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:55.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.742006+0000 mon.a (mon.0) 2726 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.742841+0000 mon.c (mon.2) 825 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-57"}]: dispatch 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.743321+0000 mon.a (mon.0) 2727 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-57"}]: dispatch 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: cluster 2024-04-02T04:54:54.889908+0000 mon.a (mon.0) 2728 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.891820+0000 mon.a (mon.0) 2729 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.891915+0000 mon.a (mon.0) 2730 : audit 1 from='client.? 172.21.15.69:0/3503362957' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: audit 2024-04-02T04:54:54.891962+0000 mon.a (mon.0) 2731 : audit 1 from='client.? 172.21.15.69:0/2109031040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: cluster 2024-04-02T04:54:54.896482+0000 mon.a (mon.0) 2732 : cluster 0 osdmap e447: 8 total, 8 up, 7 in 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[21528]: cluster 2024-04-02T04:54:54.922555+0000 mon.a (mon.0) 2733 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 8 pgs peering) 2024-04-02T04:54:55.386 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: cluster 2024-04-02T04:54:54.115143+0000 mgr.y (mgr.14538) 441 : cluster 0 pgmap v650: 355 pgs: 2 creating+activating, 23 creating+peering, 6 unknown, 324 active+clean; 8.4 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 255 B/s, 3 objects/s recovering 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.421092+0000 mon.a (mon.0) 2725 : audit 1 from='client.? 172.21.15.69:0/2109031040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.741446+0000 mon.c (mon.2) 824 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.742006+0000 mon.a (mon.0) 2726 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.742841+0000 mon.c (mon.2) 825 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-57"}]: dispatch 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.743321+0000 mon.a (mon.0) 2727 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-57"}]: dispatch 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: cluster 2024-04-02T04:54:54.889908+0000 mon.a (mon.0) 2728 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.891820+0000 mon.a (mon.0) 2729 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.891915+0000 mon.a (mon.0) 2730 : audit 1 from='client.? 172.21.15.69:0/3503362957' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: audit 2024-04-02T04:54:54.891962+0000 mon.a (mon.0) 2731 : audit 1 from='client.? 172.21.15.69:0/2109031040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: cluster 2024-04-02T04:54:54.896482+0000 mon.a (mon.0) 2732 : cluster 0 osdmap e447: 8 total, 8 up, 7 in 2024-04-02T04:54:55.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:55 smithi069 bash[28904]: cluster 2024-04-02T04:54:54.922555+0000 mon.a (mon.0) 2733 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 8 pgs peering) 2024-04-02T04:54:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:55.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:55.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: cluster 2024-04-02T04:54:54.115143+0000 mgr.y (mgr.14538) 441 : cluster 0 pgmap v650: 355 pgs: 2 creating+activating, 23 creating+peering, 6 unknown, 324 active+clean; 8.4 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 255 B/s, 3 objects/s recovering 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.421092+0000 mon.a (mon.0) 2725 : audit 1 from='client.? 172.21.15.69:0/2109031040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.741446+0000 mon.c (mon.2) 824 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.742006+0000 mon.a (mon.0) 2726 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.742841+0000 mon.c (mon.2) 825 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-57"}]: dispatch 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.743321+0000 mon.a (mon.0) 2727 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-57"}]: dispatch 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: cluster 2024-04-02T04:54:54.889908+0000 mon.a (mon.0) 2728 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.891820+0000 mon.a (mon.0) 2729 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.891915+0000 mon.a (mon.0) 2730 : audit 1 from='client.? 172.21.15.69:0/3503362957' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: audit 2024-04-02T04:54:54.891962+0000 mon.a (mon.0) 2731 : audit 1 from='client.? 172.21.15.69:0/2109031040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: cluster 2024-04-02T04:54:54.896482+0000 mon.a (mon.0) 2732 : cluster 0 osdmap e447: 8 total, 8 up, 7 in 2024-04-02T04:54:55.502 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:55 smithi098 bash[23790]: cluster 2024-04-02T04:54:54.922555+0000 mon.a (mon.0) 2733 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 8 pgs peering) 2024-04-02T04:54:55.516 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:55.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:55.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:55.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:55.780 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:55.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:55.921 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:55.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:56.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:56.048 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:56.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:56.173 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:56.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:56.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:56.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:56.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:56.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:56.695 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:56.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:56.825 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:56.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:56.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:56 smithi098 bash[23790]: cluster 2024-04-02T04:54:55.914534+0000 mon.a (mon.0) 2734 : cluster 0 osdmap e448: 8 total, 8 up, 7 in 2024-04-02T04:54:56.941 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:56 smithi098 bash[23790]: audit 2024-04-02T04:54:55.929558+0000 mon.c (mon.2) 826 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:56.941 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:56 smithi098 bash[23790]: audit 2024-04-02T04:54:55.930923+0000 mon.a (mon.0) 2735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:56.941 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:56 smithi098 bash[23790]: cluster 2024-04-02T04:54:56.115791+0000 mgr.y (mgr.14538) 442 : cluster 0 pgmap v653: 355 pgs: 32 unknown, 323 active+clean; 8.4 MiB data, 478 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 255 B/s, 3 objects/s recovering 2024-04-02T04:54:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:56.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:57.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:57.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:57.208 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:57.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[21528]: cluster 2024-04-02T04:54:55.914534+0000 mon.a (mon.0) 2734 : cluster 0 osdmap e448: 8 total, 8 up, 7 in 2024-04-02T04:54:57.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[21528]: audit 2024-04-02T04:54:55.929558+0000 mon.c (mon.2) 826 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:57.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[21528]: audit 2024-04-02T04:54:55.930923+0000 mon.a (mon.0) 2735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:57.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[21528]: cluster 2024-04-02T04:54:56.115791+0000 mgr.y (mgr.14538) 442 : cluster 0 pgmap v653: 355 pgs: 32 unknown, 323 active+clean; 8.4 MiB data, 478 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 255 B/s, 3 objects/s recovering 2024-04-02T04:54:57.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[28904]: cluster 2024-04-02T04:54:55.914534+0000 mon.a (mon.0) 2734 : cluster 0 osdmap e448: 8 total, 8 up, 7 in 2024-04-02T04:54:57.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[28904]: audit 2024-04-02T04:54:55.929558+0000 mon.c (mon.2) 826 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:57.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[28904]: audit 2024-04-02T04:54:55.930923+0000 mon.a (mon.0) 2735 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:54:57.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:56 smithi069 bash[28904]: cluster 2024-04-02T04:54:56.115791+0000 mgr.y (mgr.14538) 442 : cluster 0 pgmap v653: 355 pgs: 32 unknown, 323 active+clean; 8.4 MiB data, 478 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 255 B/s, 3 objects/s recovering 2024-04-02T04:54:57.311 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:54:57.335 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:57.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:54:57.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:57.461 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:54:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:54:57.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:57.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:57.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:54:57.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:57.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:57.985 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:57.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:58.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: cluster 2024-04-02T04:54:56.896889+0000 mon.a (mon.0) 2736 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: audit 2024-04-02T04:54:56.898372+0000 mon.a (mon.0) 2737 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: cluster 2024-04-02T04:54:56.907482+0000 mon.a (mon.0) 2738 : cluster 0 osdmap e449: 8 total, 8 up, 7 in 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: audit 2024-04-02T04:54:56.910136+0000 mon.c (mon.2) 827 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: audit 2024-04-02T04:54:56.982324+0000 mon.c (mon.2) 828 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: audit 2024-04-02T04:54:56.982748+0000 mon.a (mon.0) 2739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: audit 2024-04-02T04:54:56.983447+0000 mon.c (mon.2) 829 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-59"}]: dispatch 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[21528]: audit 2024-04-02T04:54:56.983817+0000 mon.a (mon.0) 2740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-59"}]: dispatch 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: cluster 2024-04-02T04:54:56.896889+0000 mon.a (mon.0) 2736 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: audit 2024-04-02T04:54:56.898372+0000 mon.a (mon.0) 2737 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:58.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: cluster 2024-04-02T04:54:56.907482+0000 mon.a (mon.0) 2738 : cluster 0 osdmap e449: 8 total, 8 up, 7 in 2024-04-02T04:54:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: audit 2024-04-02T04:54:56.910136+0000 mon.c (mon.2) 827 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:54:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: audit 2024-04-02T04:54:56.982324+0000 mon.c (mon.2) 828 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: audit 2024-04-02T04:54:56.982748+0000 mon.a (mon.0) 2739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: audit 2024-04-02T04:54:56.983447+0000 mon.c (mon.2) 829 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-59"}]: dispatch 2024-04-02T04:54:58.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:57 smithi069 bash[28904]: audit 2024-04-02T04:54:56.983817+0000 mon.a (mon.0) 2740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-59"}]: dispatch 2024-04-02T04:54:58.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: cluster 2024-04-02T04:54:56.896889+0000 mon.a (mon.0) 2736 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: audit 2024-04-02T04:54:56.898372+0000 mon.a (mon.0) 2737 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: cluster 2024-04-02T04:54:56.907482+0000 mon.a (mon.0) 2738 : cluster 0 osdmap e449: 8 total, 8 up, 7 in 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: audit 2024-04-02T04:54:56.910136+0000 mon.c (mon.2) 827 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: audit 2024-04-02T04:54:56.982324+0000 mon.c (mon.2) 828 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: audit 2024-04-02T04:54:56.982748+0000 mon.a (mon.0) 2739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: audit 2024-04-02T04:54:56.983447+0000 mon.c (mon.2) 829 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-59"}]: dispatch 2024-04-02T04:54:58.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:57 smithi098 bash[23790]: audit 2024-04-02T04:54:56.983817+0000 mon.a (mon.0) 2740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-59"}]: dispatch 2024-04-02T04:54:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:58.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:58.243 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:58.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:54:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:54:58.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:58.501 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:58.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:58.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:58.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:58.753 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:58.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:58.882 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:58.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:58.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:54:58.896 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:54:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:54:58.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:59.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:59.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[28904]: cluster 2024-04-02T04:54:57.944184+0000 mon.a (mon.0) 2741 : cluster 0 osdmap e450: 8 total, 8 up, 7 in 2024-04-02T04:54:59.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[28904]: audit 2024-04-02T04:54:58.113221+0000 mon.b (mon.1) 227 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-02T04:54:59.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[28904]: cluster 2024-04-02T04:54:58.117301+0000 mgr.y (mgr.14538) 443 : cluster 0 pgmap v656: 322 pgs: 2 active+undersized+remapped, 1 peering, 319 active+clean; 8.4 MiB data, 483 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1.5 KiB/s wr, 2 op/s; 977 KiB/s, 62 objects/s recovering 2024-04-02T04:54:59.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[28904]: audit 2024-04-02T04:54:58.416850+0000 mon.a (mon.0) 2742 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-02T04:54:59.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[28904]: cluster 2024-04-02T04:54:58.467776+0000 osd.1 (osd.1) 255 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[28904]: cluster 2024-04-02T04:54:58.467961+0000 osd.1 (osd.1) 256 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21528]: cluster 2024-04-02T04:54:57.944184+0000 mon.a (mon.0) 2741 : cluster 0 osdmap e450: 8 total, 8 up, 7 in 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21528]: audit 2024-04-02T04:54:58.113221+0000 mon.b (mon.1) 227 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21528]: cluster 2024-04-02T04:54:58.117301+0000 mgr.y (mgr.14538) 443 : cluster 0 pgmap v656: 322 pgs: 2 active+undersized+remapped, 1 peering, 319 active+clean; 8.4 MiB data, 483 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1.5 KiB/s wr, 2 op/s; 977 KiB/s, 62 objects/s recovering 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21528]: audit 2024-04-02T04:54:58.416850+0000 mon.a (mon.0) 2742 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21528]: cluster 2024-04-02T04:54:58.467776+0000 osd.1 (osd.1) 255 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:59.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:58 smithi069 bash[21528]: cluster 2024-04-02T04:54:58.467961+0000 osd.1 (osd.1) 256 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:59.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:59.120 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:58 smithi098 bash[23790]: cluster 2024-04-02T04:54:57.944184+0000 mon.a (mon.0) 2741 : cluster 0 osdmap e450: 8 total, 8 up, 7 in 2024-04-02T04:54:59.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:58 smithi098 bash[23790]: audit 2024-04-02T04:54:58.113221+0000 mon.b (mon.1) 227 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-02T04:54:59.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:58 smithi098 bash[23790]: cluster 2024-04-02T04:54:58.117301+0000 mgr.y (mgr.14538) 443 : cluster 0 pgmap v656: 322 pgs: 2 active+undersized+remapped, 1 peering, 319 active+clean; 8.4 MiB data, 483 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1.5 KiB/s wr, 2 op/s; 977 KiB/s, 62 objects/s recovering 2024-04-02T04:54:59.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:58 smithi098 bash[23790]: audit 2024-04-02T04:54:58.416850+0000 mon.a (mon.0) 2742 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-02T04:54:59.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:58 smithi098 bash[23790]: cluster 2024-04-02T04:54:58.467776+0000 osd.1 (osd.1) 255 : cluster 0 purged_snaps scrub starts 2024-04-02T04:54:59.121 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:58 smithi098 bash[23790]: cluster 2024-04-02T04:54:58.467961+0000 osd.1 (osd.1) 256 : cluster 0 purged_snaps scrub ok 2024-04-02T04:54:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:59.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:54:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:54:59.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:59.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:54:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:54:59.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:54:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:54:59.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:59.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:54:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:54:59.763 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:59.785 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:54:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:54:59.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:59.917 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:54:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:54:59.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:00.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:00.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:00.159 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[28904]: audit 2024-04-02T04:54:58.959119+0000 mon.a (mon.0) 2743 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[28904]: cluster 2024-04-02T04:54:58.977300+0000 mon.a (mon.0) 2744 : cluster 0 osdmap e451: 8 total, 8 up, 7 in 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[28904]: audit 2024-04-02T04:54:58.980285+0000 mon.c (mon.2) 830 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[28904]: audit 2024-04-02T04:54:58.985533+0000 mon.a (mon.0) 2745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[21528]: audit 2024-04-02T04:54:58.959119+0000 mon.a (mon.0) 2743 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[21528]: cluster 2024-04-02T04:54:58.977300+0000 mon.a (mon.0) 2744 : cluster 0 osdmap e451: 8 total, 8 up, 7 in 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[21528]: audit 2024-04-02T04:54:58.980285+0000 mon.c (mon.2) 830 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:00.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:54:59 smithi069 bash[21528]: audit 2024-04-02T04:54:58.985533+0000 mon.a (mon.0) 2745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:00.172 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:00.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:00.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:59 smithi098 bash[23790]: audit 2024-04-02T04:54:58.959119+0000 mon.a (mon.0) 2743 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-02T04:55:00.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:54:59 smithi098 bash[23790]: cluster 2024-04-02T04:54:58.977300+0000 mon.a (mon.0) 2744 : cluster 0 osdmap e451: 8 total, 8 up, 7 in 2024-04-02T04:55:00.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:00 smithi098 bash[23790]: audit 2024-04-02T04:54:58.980285+0000 mon.c (mon.2) 830 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:00.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:00 smithi098 bash[23790]: audit 2024-04-02T04:54:58.985533+0000 mon.a (mon.0) 2745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:00.301 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:00.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:00.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:00.434 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:55:00.434 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:55:00.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:00.553 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:00.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:00.680 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:00.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:00.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:00.938 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:00.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:01.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:01.052 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:00 smithi098 bash[23790]: audit 2024-04-02T04:54:59.961137+0000 mon.a (mon.0) 2746 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:01.052 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:00 smithi098 bash[23790]: cluster 2024-04-02T04:54:59.969993+0000 mon.a (mon.0) 2747 : cluster 0 osdmap e452: 8 total, 8 up, 7 in 2024-04-02T04:55:01.052 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:01 smithi098 bash[23790]: audit 2024-04-02T04:54:59.974555+0000 mon.c (mon.2) 831 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:01.052 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:01 smithi098 bash[23790]: audit 2024-04-02T04:54:59.994662+0000 mon.c (mon.2) 832 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:01.053 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:01 smithi098 bash[23790]: audit 2024-04-02T04:54:59.995400+0000 mon.a (mon.0) 2748 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:01.053 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:01 smithi098 bash[23790]: cluster 2024-04-02T04:55:00.118369+0000 mgr.y (mgr.14538) 444 : cluster 0 pgmap v659: 354 pgs: 7 creating+peering, 25 unknown, 2 active+undersized+remapped, 1 peering, 319 active+clean; 8.4 MiB data, 487 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s wr, 1 op/s; 977 KiB/s, 67 objects/s recovering 2024-04-02T04:55:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:01.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:01.195 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:01.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:00 smithi069 bash[21528]: audit 2024-04-02T04:54:59.961137+0000 mon.a (mon.0) 2746 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:00 smithi069 bash[21528]: cluster 2024-04-02T04:54:59.969993+0000 mon.a (mon.0) 2747 : cluster 0 osdmap e452: 8 total, 8 up, 7 in 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:00 smithi069 bash[21528]: audit 2024-04-02T04:54:59.974555+0000 mon.c (mon.2) 831 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:00 smithi069 bash[21528]: audit 2024-04-02T04:54:59.994662+0000 mon.c (mon.2) 832 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[21528]: audit 2024-04-02T04:54:59.995400+0000 mon.a (mon.0) 2748 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[21528]: cluster 2024-04-02T04:55:00.118369+0000 mgr.y (mgr.14538) 444 : cluster 0 pgmap v659: 354 pgs: 7 creating+peering, 25 unknown, 2 active+undersized+remapped, 1 peering, 319 active+clean; 8.4 MiB data, 487 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s wr, 1 op/s; 977 KiB/s, 67 objects/s recovering 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[28904]: audit 2024-04-02T04:54:59.961137+0000 mon.a (mon.0) 2746 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[28904]: cluster 2024-04-02T04:54:59.969993+0000 mon.a (mon.0) 2747 : cluster 0 osdmap e452: 8 total, 8 up, 7 in 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[28904]: audit 2024-04-02T04:54:59.974555+0000 mon.c (mon.2) 831 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[28904]: audit 2024-04-02T04:54:59.994662+0000 mon.c (mon.2) 832 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:01.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[28904]: audit 2024-04-02T04:54:59.995400+0000 mon.a (mon.0) 2748 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:01.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:01 smithi069 bash[28904]: cluster 2024-04-02T04:55:00.118369+0000 mgr.y (mgr.14538) 444 : cluster 0 pgmap v659: 354 pgs: 7 creating+peering, 25 unknown, 2 active+undersized+remapped, 1 peering, 319 active+clean; 8.4 MiB data, 487 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s wr, 1 op/s; 977 KiB/s, 67 objects/s recovering 2024-04-02T04:55:01.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:01.323 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:01.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-02T04:55:01.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:01.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:01.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:01.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:01.774 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:01.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:01.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:01.941 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:55:01.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:55:01.941 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-02T04:55:01.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:55:02.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:02.138 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[28904]: audit 2024-04-02T04:55:00.985046+0000 mon.a (mon.0) 2749 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[28904]: cluster 2024-04-02T04:55:00.993520+0000 mon.a (mon.0) 2750 : cluster 0 osdmap e453: 8 total, 8 up, 7 in 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[28904]: audit 2024-04-02T04:55:01.063344+0000 mon.c (mon.2) 833 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[28904]: audit 2024-04-02T04:55:01.064119+0000 mon.a (mon.0) 2751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[28904]: audit 2024-04-02T04:55:01.065193+0000 mon.c (mon.2) 834 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-61"}]: dispatch 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[28904]: audit 2024-04-02T04:55:01.065870+0000 mon.a (mon.0) 2752 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-61"}]: dispatch 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[21528]: audit 2024-04-02T04:55:00.985046+0000 mon.a (mon.0) 2749 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:02.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[21528]: cluster 2024-04-02T04:55:00.993520+0000 mon.a (mon.0) 2750 : cluster 0 osdmap e453: 8 total, 8 up, 7 in 2024-04-02T04:55:02.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[21528]: audit 2024-04-02T04:55:01.063344+0000 mon.c (mon.2) 833 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:02.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[21528]: audit 2024-04-02T04:55:01.064119+0000 mon.a (mon.0) 2751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:02.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[21528]: audit 2024-04-02T04:55:01.065193+0000 mon.c (mon.2) 834 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-61"}]: dispatch 2024-04-02T04:55:02.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:02 smithi069 bash[21528]: audit 2024-04-02T04:55:01.065870+0000 mon.a (mon.0) 2752 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-61"}]: dispatch 2024-04-02T04:55:02.156 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:02.260 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:02.284 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:02.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:02.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:02.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:02 smithi098 bash[23790]: audit 2024-04-02T04:55:00.985046+0000 mon.a (mon.0) 2749 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:02.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:02 smithi098 bash[23790]: cluster 2024-04-02T04:55:00.993520+0000 mon.a (mon.0) 2750 : cluster 0 osdmap e453: 8 total, 8 up, 7 in 2024-04-02T04:55:02.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:02 smithi098 bash[23790]: audit 2024-04-02T04:55:01.063344+0000 mon.c (mon.2) 833 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:02.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:02 smithi098 bash[23790]: audit 2024-04-02T04:55:01.064119+0000 mon.a (mon.0) 2751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:02.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:02 smithi098 bash[23790]: audit 2024-04-02T04:55:01.065193+0000 mon.c (mon.2) 834 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-61"}]: dispatch 2024-04-02T04:55:02.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:02 smithi098 bash[23790]: audit 2024-04-02T04:55:01.065870+0000 mon.a (mon.0) 2752 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-61"}]: dispatch 2024-04-02T04:55:02.514 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:02.538 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:02.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:02.669 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:02.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:02.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:02.926 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:02.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:03.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:03.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:03 smithi098 bash[23790]: cluster 2024-04-02T04:55:02.004714+0000 mon.a (mon.0) 2753 : cluster 0 osdmap e454: 8 total, 8 up, 7 in 2024-04-02T04:55:03.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:03 smithi098 bash[23790]: audit 2024-04-02T04:55:02.115475+0000 mon.b (mon.1) 228 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-02T04:55:03.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:03 smithi098 bash[23790]: cluster 2024-04-02T04:55:02.119270+0000 mgr.y (mgr.14538) 445 : cluster 0 pgmap v662: 321 pgs: 2 active+undersized+remapped, 1 peering, 318 active+clean; 8.4 MiB data, 487 MiB used, 625 GiB / 626 GiB avail; 302 B/s, 3 objects/s recovering 2024-04-02T04:55:03.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:03 smithi098 bash[23790]: audit 2024-04-02T04:55:02.448923+0000 mon.a (mon.0) 2754 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-02T04:55:03.061 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:03.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[21528]: cluster 2024-04-02T04:55:02.004714+0000 mon.a (mon.0) 2753 : cluster 0 osdmap e454: 8 total, 8 up, 7 in 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[21528]: audit 2024-04-02T04:55:02.115475+0000 mon.b (mon.1) 228 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[21528]: cluster 2024-04-02T04:55:02.119270+0000 mgr.y (mgr.14538) 445 : cluster 0 pgmap v662: 321 pgs: 2 active+undersized+remapped, 1 peering, 318 active+clean; 8.4 MiB data, 487 MiB used, 625 GiB / 626 GiB avail; 302 B/s, 3 objects/s recovering 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[21528]: audit 2024-04-02T04:55:02.448923+0000 mon.a (mon.0) 2754 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[28904]: cluster 2024-04-02T04:55:02.004714+0000 mon.a (mon.0) 2753 : cluster 0 osdmap e454: 8 total, 8 up, 7 in 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[28904]: audit 2024-04-02T04:55:02.115475+0000 mon.b (mon.1) 228 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[28904]: cluster 2024-04-02T04:55:02.119270+0000 mgr.y (mgr.14538) 445 : cluster 0 pgmap v662: 321 pgs: 2 active+undersized+remapped, 1 peering, 318 active+clean; 8.4 MiB data, 487 MiB used, 625 GiB / 626 GiB avail; 302 B/s, 3 objects/s recovering 2024-04-02T04:55:03.172 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:03 smithi069 bash[28904]: audit 2024-04-02T04:55:02.448923+0000 mon.a (mon.0) 2754 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-02T04:55:03.182 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:03.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:03.303 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:03.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:03.453 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:03.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:55:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:55:03.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:03.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:03.689 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:03.792 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:03.818 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:03.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:03.949 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:03.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:04.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:04.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:04.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:04.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: cluster 2024-04-02T04:55:02.639531+0000 osd.5 (osd.5) 251 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:04.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: cluster 2024-04-02T04:55:02.639941+0000 osd.5 (osd.5) 252 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:04.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: audit 2024-04-02T04:55:03.019750+0000 mon.a (mon.0) 2755 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-02T04:55:04.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: cluster 2024-04-02T04:55:03.031429+0000 mon.a (mon.0) 2756 : cluster 0 osdmap e455: 8 total, 8 up, 7 in 2024-04-02T04:55:04.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: audit 2024-04-02T04:55:03.034337+0000 mon.c (mon.2) 835 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:04.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: audit 2024-04-02T04:55:03.035241+0000 mon.a (mon.0) 2757 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:04.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: audit 2024-04-02T04:55:03.051836+0000 mon.b (mon.1) 229 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:04.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[21528]: audit 2024-04-02T04:55:03.284729+0000 mgr.y (mgr.14538) 446 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:04.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: cluster 2024-04-02T04:55:02.639531+0000 osd.5 (osd.5) 251 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:04.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: cluster 2024-04-02T04:55:02.639941+0000 osd.5 (osd.5) 252 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:04.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: audit 2024-04-02T04:55:03.019750+0000 mon.a (mon.0) 2755 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-02T04:55:04.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: cluster 2024-04-02T04:55:03.031429+0000 mon.a (mon.0) 2756 : cluster 0 osdmap e455: 8 total, 8 up, 7 in 2024-04-02T04:55:04.199 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: audit 2024-04-02T04:55:03.034337+0000 mon.c (mon.2) 835 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:04.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: audit 2024-04-02T04:55:03.035241+0000 mon.a (mon.0) 2757 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:04.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: audit 2024-04-02T04:55:03.051836+0000 mon.b (mon.1) 229 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:04.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:04 smithi069 bash[28904]: audit 2024-04-02T04:55:03.284729+0000 mgr.y (mgr.14538) 446 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:04.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:04.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:04.326 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: cluster 2024-04-02T04:55:02.639531+0000 osd.5 (osd.5) 251 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:04.326 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: cluster 2024-04-02T04:55:02.639941+0000 osd.5 (osd.5) 252 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:04.326 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: audit 2024-04-02T04:55:03.019750+0000 mon.a (mon.0) 2755 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-02T04:55:04.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: cluster 2024-04-02T04:55:03.031429+0000 mon.a (mon.0) 2756 : cluster 0 osdmap e455: 8 total, 8 up, 7 in 2024-04-02T04:55:04.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: audit 2024-04-02T04:55:03.034337+0000 mon.c (mon.2) 835 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:04.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: audit 2024-04-02T04:55:03.035241+0000 mon.a (mon.0) 2757 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:04.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: audit 2024-04-02T04:55:03.051836+0000 mon.b (mon.1) 229 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:04.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:04 smithi098 bash[23790]: audit 2024-04-02T04:55:03.284729+0000 mgr.y (mgr.14538) 446 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:04.340 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:04.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:04.471 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:04.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:04.603 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:04.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:04.728 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:04.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:04.858 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:04.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:04.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:04.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:05.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:05.100 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: audit 2024-04-02T04:55:04.022087+0000 mon.a (mon.0) 2758 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: cluster 2024-04-02T04:55:04.027827+0000 mon.a (mon.0) 2759 : cluster 0 osdmap e456: 8 total, 8 up, 7 in 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: audit 2024-04-02T04:55:04.067929+0000 mon.c (mon.2) 836 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: audit 2024-04-02T04:55:04.076568+0000 mon.c (mon.2) 837 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: audit 2024-04-02T04:55:04.077005+0000 mon.a (mon.0) 2760 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: cluster 2024-04-02T04:55:04.120266+0000 mgr.y (mgr.14538) 447 : cluster 0 pgmap v665: 353 pgs: 4 creating+peering, 28 unknown, 321 active+clean; 8.4 MiB data, 488 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: cluster 2024-04-02T04:55:04.892712+0000 mon.a (mon.0) 2761 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: audit 2024-04-02T04:55:04.896634+0000 mon.a (mon.0) 2762 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:05.101 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:05 smithi098 bash[23790]: cluster 2024-04-02T04:55:04.904591+0000 mon.a (mon.0) 2763 : cluster 0 osdmap e457: 8 total, 8 up, 7 in 2024-04-02T04:55:05.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:05.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:05.244 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:05.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:05.358 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: audit 2024-04-02T04:55:04.022087+0000 mon.a (mon.0) 2758 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:05.358 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: cluster 2024-04-02T04:55:04.027827+0000 mon.a (mon.0) 2759 : cluster 0 osdmap e456: 8 total, 8 up, 7 in 2024-04-02T04:55:05.358 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: audit 2024-04-02T04:55:04.067929+0000 mon.c (mon.2) 836 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: audit 2024-04-02T04:55:04.076568+0000 mon.c (mon.2) 837 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: audit 2024-04-02T04:55:04.077005+0000 mon.a (mon.0) 2760 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: cluster 2024-04-02T04:55:04.120266+0000 mgr.y (mgr.14538) 447 : cluster 0 pgmap v665: 353 pgs: 4 creating+peering, 28 unknown, 321 active+clean; 8.4 MiB data, 488 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: cluster 2024-04-02T04:55:04.892712+0000 mon.a (mon.0) 2761 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: audit 2024-04-02T04:55:04.896634+0000 mon.a (mon.0) 2762 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[21528]: cluster 2024-04-02T04:55:04.904591+0000 mon.a (mon.0) 2763 : cluster 0 osdmap e457: 8 total, 8 up, 7 in 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: audit 2024-04-02T04:55:04.022087+0000 mon.a (mon.0) 2758 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: cluster 2024-04-02T04:55:04.027827+0000 mon.a (mon.0) 2759 : cluster 0 osdmap e456: 8 total, 8 up, 7 in 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: audit 2024-04-02T04:55:04.067929+0000 mon.c (mon.2) 836 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: audit 2024-04-02T04:55:04.076568+0000 mon.c (mon.2) 837 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: audit 2024-04-02T04:55:04.077005+0000 mon.a (mon.0) 2760 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: cluster 2024-04-02T04:55:04.120266+0000 mgr.y (mgr.14538) 447 : cluster 0 pgmap v665: 353 pgs: 4 creating+peering, 28 unknown, 321 active+clean; 8.4 MiB data, 488 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-02T04:55:05.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: cluster 2024-04-02T04:55:04.892712+0000 mon.a (mon.0) 2761 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:55:05.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: audit 2024-04-02T04:55:04.896634+0000 mon.a (mon.0) 2762 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:05.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:05 smithi069 bash[28904]: cluster 2024-04-02T04:55:04.904591+0000 mon.a (mon.0) 2763 : cluster 0 osdmap e457: 8 total, 8 up, 7 in 2024-04-02T04:55:05.375 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:05.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:05.507 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:05.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:05.634 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:05.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:05.764 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:05.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:05.896 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:05.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:06.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:06.029 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:06.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:55:06.162 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:06.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:06.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:06 smithi069 bash[21528]: cluster 2024-04-02T04:55:04.916463+0000 osd.7 (osd.7) 35 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:06.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:06 smithi069 bash[21528]: cluster 2024-04-02T04:55:04.916772+0000 osd.7 (osd.7) 36 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:06.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:06 smithi069 bash[21528]: cluster 2024-04-02T04:55:05.906448+0000 mon.a (mon.0) 2764 : cluster 0 osdmap e458: 8 total, 8 up, 7 in 2024-04-02T04:55:06.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:06 smithi069 bash[28904]: cluster 2024-04-02T04:55:04.916463+0000 osd.7 (osd.7) 35 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:06.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:06 smithi069 bash[28904]: cluster 2024-04-02T04:55:04.916772+0000 osd.7 (osd.7) 36 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:06.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:06 smithi069 bash[28904]: cluster 2024-04-02T04:55:05.906448+0000 mon.a (mon.0) 2764 : cluster 0 osdmap e458: 8 total, 8 up, 7 in 2024-04-02T04:55:06.291 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:06.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:06.420 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:06.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:06 smithi098 bash[23790]: cluster 2024-04-02T04:55:04.916463+0000 osd.7 (osd.7) 35 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:06 smithi098 bash[23790]: cluster 2024-04-02T04:55:04.916772+0000 osd.7 (osd.7) 36 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:06.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:06 smithi098 bash[23790]: cluster 2024-04-02T04:55:05.906448+0000 mon.a (mon.0) 2764 : cluster 0 osdmap e458: 8 total, 8 up, 7 in 2024-04-02T04:55:06.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:06.653 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:06.674 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:06.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:06.797 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/1623107553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0106500 msgr2=0x7efdf010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- 172.21.15.69:0/1623107553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0106500 0x7efdf010e0d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efde800b0d0 tx=0x7efde800b3e0 comp rx=0 tx=0).stop 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/1623107553 shutdown_connections 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- 172.21.15.69:0/1623107553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0106500 0x7efdf010e0d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- 172.21.15.69:0/1623107553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdf0101420 0x7efdf0101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- 172.21.15.69:0/1623107553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0101dd0 0x7efdf0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/1623107553 >> 172.21.15.69:0/1623107553 conn(0x7efdf0078240 msgr2=0x7efdf0078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/1623107553 shutdown_connections 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/1623107553 wait complete. 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 librados: starting msgr at 2024-04-02T04:55:06.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 librados: starting objecter 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 Processor -- start 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- start start 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 librados: setting wanted keys 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 librados: calling monclient init 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 0x7efdf0102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0101dd0 0x7efdf0103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdf0106500 0x7efdf0103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efdf0110a80 con 0x7efdf0106500 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdf0110900 con 0x7efdf0101dd0 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efdf0110c00 con 0x7efdf0101420 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 0x7efdf0102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 0x7efdf0102ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47474/0 (socket says 172.21.15.69:47474) 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 -- 172.21.15.69:0/2737914966 learned_addr learned my addr 172.21.15.69:0/2737914966 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:06.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdf0106500 0x7efdf0103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0101dd0 0x7efdf0103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0101dd0 msgr2=0x7efdf0103400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0101dd0 0x7efdf0103400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdf0106500 msgr2=0x7efdf0103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdf0106500 0x7efdf0103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf01a6690 con 0x7efdf0101420 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0101dd0 0x7efdf0103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 0x7efdf0102ec0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efde400e9a0 tx=0x7efde400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdecff9700 10 client.?.objecter ms_handle_connect 0x7efdf0101420 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdecff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde400cbd0 con 0x7efdf0101420 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7efde400cd70 con 0x7efdf0101420 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf01a6980 con 0x7efdf0101420 2024-04-02T04:55:06.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4010e60 con 0x7efdf0101420 2024-04-02T04:55:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 10 client.26564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf01a6ec0 con 0x7efdf0101420 2024-04-02T04:55:06.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 librados: init done 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.859+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7efdf004ff00 con 0x7efdf0101420 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7efde4010430 con 0x7efdf0101420 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efdd806f5e0 0x7efdd8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(458..458 src has 74..458) ==== 10925+0+0 (secure 0 0 0) 0x7efde408f800 con 0x7efdf0101420 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 osd_map(458..458 src has 74..458) 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 3 client.26564.objecter handle_osd_map got epochs [458,458] > 0 2024-04-02T04:55:06.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 3 client.26564.objecter handle_osd_map decoding full epoch 458 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efdd806f5e0 0x7efdd8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 20 client.26564.objecter dump_active .. 0 homeless 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v458) ==== 54+0+16 (secure 0 0 0) 0x7efde4090080 con 0x7efdf0101420 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 10 client.26564.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 1 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdd806f5e0 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efdd806f5e0 0x7efdd8071aa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efdf0106250 tx=0x7efde0008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7efdf0062e50 0x7efdf0063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 1 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 0 1 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 4 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 10 client.26564.objecter _send_command 1 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdf01aa220 con 0x7efdf0062e50 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7efdf0062e50 0x7efdf0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7efdf0062e50 0x7efdf0063230 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf0062e50 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf01aa220 con 0x7efdf0062e50 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 1: 0 ) 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 10 client.26564.objecter _finish_command 1 = system:0 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 4 2024-04-02T04:55:06.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.863+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 0 1 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 10 client.26564.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 2 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 2 2024-04-02T04:55:06.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01043f0 osd=0 3 2024-04-02T04:55:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 0 2 2024-04-02T04:55:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 4 2024-04-02T04:55:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 10 client.26564.objecter _send_command 2 2024-04-02T04:55:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf00628d0 con 0x7efdf0062e50 2024-04-02T04:55:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf00628d0 con 0x7efdf0062e50 2024-04-02T04:55:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 10 client.26564.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 4 2024-04-02T04:55:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 0 2 2024-04-02T04:55:06.881 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 10 client.26564.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 3 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7efdf002c9b0 0x7efdf002cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 3 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 1 3 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 4 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 10 client.26564.objecter _send_command 3 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7efdf0109e30 con 0x7efdf002c9b0 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7efdf002c9b0 0x7efdf002cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf002c9b0 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7efdf002c9b0 0x7efdf002cdf0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf0109e30 con 0x7efdf002c9b0 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 3: 0 ) 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 10 client.26564.objecter _finish_command 3 = system:0 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 4 2024-04-02T04:55:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.883+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 1 3 2024-04-02T04:55:06.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 10 client.26564.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 4 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 4 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01097b0 osd=1 3 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 1 4 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 4 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 10 client.26564.objecter _send_command 4 2024-04-02T04:55:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf0109230 con 0x7efdf002c9b0 2024-04-02T04:55:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf0109230 con 0x7efdf002c9b0 2024-04-02T04:55:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdecff9700 10 client.26564.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 4 2024-04-02T04:55:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 1 4 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 10 client.26564.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 5 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7efdf01ac3e0 0x7efdf01ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 5 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 2 5 2024-04-02T04:55:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 4 2024-04-02T04:55:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 10 client.26564.objecter _send_command 5 2024-04-02T04:55:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7efdf01b1630 con 0x7efdf01ac3e0 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.903+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7efdf01ac3e0 0x7efdf01ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7efdf01ac3e0 0x7efdf01ae7d0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf01ac3e0 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf01b1630 con 0x7efdf01ac3e0 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 5: 0 ) 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdecff9700 10 client.26564.objecter _finish_command 5 = system:0 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 4 2024-04-02T04:55:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.907+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 2 5 2024-04-02T04:55:06.920 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:06.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 10 client.26564.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 6 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 6 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf0109fe0 osd=2 3 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 2 6 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 4 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 10 client.26564.objecter _send_command 6 2024-04-02T04:55:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf002d580 con 0x7efdf01ac3e0 2024-04-02T04:55:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf002d580 con 0x7efdf01ac3e0 2024-04-02T04:55:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdecff9700 10 client.26564.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 4 2024-04-02T04:55:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 2 6 2024-04-02T04:55:06.926 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:55:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 10 client.26564.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 7 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7efdf01a9d80 0x7efdf01b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 7 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 3 7 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 4 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 10 client.26564.objecter _send_command 7 2024-04-02T04:55:06.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.927+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7efdf002d910 con 0x7efdf01a9d80 2024-04-02T04:55:06.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.935+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7efdf01a9d80 0x7efdf01b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.939+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7efdf01a9d80 0x7efdf01b1ea0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.939+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf01a9d80 2024-04-02T04:55:06.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.943+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf002d910 con 0x7efdf01a9d80 2024-04-02T04:55:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.943+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 7: 0 ) 2024-04-02T04:55:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.943+0000 7efdecff9700 10 client.26564.objecter _finish_command 7 = system:0 2024-04-02T04:55:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.943+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 4 2024-04-02T04:55:06.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.943+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 3 7 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 10 client.26564.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 8 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 8 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf0109630 osd=3 3 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 3 8 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 4 2024-04-02T04:55:06.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 10 client.26564.objecter _send_command 8 2024-04-02T04:55:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.967+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf01a92f0 con 0x7efdf01a9d80 2024-04-02T04:55:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf01a92f0 con 0x7efdf01a9d80 2024-04-02T04:55:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 10 client.26564.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 4 2024-04-02T04:55:06.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 3 8 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 10 client.26564.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 9 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7efdf01af9f0 0x7efdf01b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 9 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 4 9 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 4 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 10 client.26564.objecter _send_command 9 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7efdf01a9210 con 0x7efdf01af9f0 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7efdf01af9f0 0x7efdf01b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf01af9f0 2024-04-02T04:55:06.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7efdf01af9f0 0x7efdf01b3560 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf01a9210 con 0x7efdf01af9f0 2024-04-02T04:55:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 9: 0 ) 2024-04-02T04:55:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 10 client.26564.objecter _finish_command 9 = system:0 2024-04-02T04:55:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 4 2024-04-02T04:55:06.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.971+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 4 9 2024-04-02T04:55:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 10 client.26564.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 10 2024-04-02T04:55:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 10 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01a90a0 osd=4 3 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 4 10 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 4 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 10 client.26564.objecter _send_command 10 2024-04-02T04:55:06.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.987+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf01a9210 con 0x7efdf01af9f0 2024-04-02T04:55:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf01a9210 con 0x7efdf01af9f0 2024-04-02T04:55:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdecff9700 10 client.26564.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 4 2024-04-02T04:55:06.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 4 10 2024-04-02T04:55:06.990 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:55:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 10 client.26564.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 11 2024-04-02T04:55:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7efdf01b3fb0 0x7efdf01b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:06.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 11 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 5 11 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 4 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 10 client.26564.objecter _send_command 11 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7efdf01bb670 con 0x7efdf01b3fb0 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7efdf01b3fb0 0x7efdf01b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf01b3fb0 2024-04-02T04:55:06.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.991+0000 7efdeeffd700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7efdf01b3fb0 0x7efdf01b4430 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.995+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf01bb670 con 0x7efdf01b3fb0 2024-04-02T04:55:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.995+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 11: 0 ) 2024-04-02T04:55:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.995+0000 7efdecff9700 10 client.26564.objecter _finish_command 11 = system:0 2024-04-02T04:55:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.995+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 4 2024-04-02T04:55:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:06.995+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 5 11 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 10 client.26564.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 12 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 12 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01a9210 osd=5 3 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 5 12 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 4 2024-04-02T04:55:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 10 client.26564.objecter _send_command 12 2024-04-02T04:55:07.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.011+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf01b8d50 con 0x7efdf01b3fb0 2024-04-02T04:55:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf01b8d50 con 0x7efdf01b3fb0 2024-04-02T04:55:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdecff9700 10 client.26564.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 4 2024-04-02T04:55:07.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 5 12 2024-04-02T04:55:07.013 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 10 client.26564.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 13 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7efdf01b9260 0x7efdf01bb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 13 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 6 13 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 4 2024-04-02T04:55:07.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 10 client.26564.objecter _send_command 13 2024-04-02T04:55:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7efdf01be6b0 con 0x7efdf01b9260 2024-04-02T04:55:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7efdf01b9260 0x7efdf01bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf01b9260 2024-04-02T04:55:07.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.015+0000 7efdeffff700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7efdf01b9260 0x7efdf01bb650 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.019+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf01be6b0 con 0x7efdf01b9260 2024-04-02T04:55:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.019+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 13: 0 ) 2024-04-02T04:55:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.019+0000 7efdecff9700 10 client.26564.objecter _finish_command 13 = system:0 2024-04-02T04:55:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.019+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 4 2024-04-02T04:55:07.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.019+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 6 13 2024-04-02T04:55:07.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 10 client.26564.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 14 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 14 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01b4ab0 osd=6 3 2024-04-02T04:55:07.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 6 14 2024-04-02T04:55:07.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 4 2024-04-02T04:55:07.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 10 client.26564.objecter _send_command 14 2024-04-02T04:55:07.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.035+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf01bd480 con 0x7efdf01b9260 2024-04-02T04:55:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf01bd480 con 0x7efdf01b9260 2024-04-02T04:55:07.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdecff9700 10 client.26564.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 4 2024-04-02T04:55:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 6 14 2024-04-02T04:55:07.038 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:55:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 10 client.26564.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 15 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7efdf01bdba0 0x7efdf01be000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 15 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 7 15 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 4 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 10 client.26564.objecter _send_command 15 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7efdf01c1a80 con 0x7efdf01bdba0 2024-04-02T04:55:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.039+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7efdf01bdba0 0x7efdf01be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdecff9700 10 client.26564.objecter ms_handle_connect 0x7efdf01bdba0 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdef7fe700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7efdf01bdba0 0x7efdf01be000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdf01c1a80 con 0x7efdf01bdba0 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 15: 0 ) 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdecff9700 10 client.26564.objecter _finish_command 15 = system:0 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 4 2024-04-02T04:55:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.043+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 7 15 2024-04-02T04:55:07.049 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 10 client.26564.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign -1 16 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 20 client.26564.objecter _get_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_remove -1 16 2024-04-02T04:55:07.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 20 client.26564.objecter get_session s=0x7efdf01bd480 osd=7 3 2024-04-02T04:55:07.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 15 client.26564.objecter _session_command_op_assign 7 16 2024-04-02T04:55:07.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 4 2024-04-02T04:55:07.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 10 client.26564.objecter _send_command 16 2024-04-02T04:55:07.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.063+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efdf01bbcb0 con 0x7efdf01bdba0 2024-04-02T04:55:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdecff9700 1 -- 172.21.15.69:0/2737914966 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7efdf01bbcb0 con 0x7efdf01bdba0 2024-04-02T04:55:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdecff9700 10 client.26564.objecter ms_dispatch 0x7efdf0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdecff9700 10 client.26564.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdecff9700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 4 2024-04-02T04:55:07.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdecff9700 15 client.26564.objecter _session_command_op_remove 7 16 2024-04-02T04:55:07.064 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:55:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 librados: watch_flush enter 2024-04-02T04:55:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 librados: watch_flush exit 2024-04-02T04:55:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.0 2024-04-02T04:55:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7efdf0062e50 msgr2=0x7efdf0063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7efdf0062e50 0x7efdf0063230 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01043f0 osd=0 1 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.1 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7efdf002c9b0 msgr2=0x7efdf002cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7efdf002c9b0 0x7efdf002cdf0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01097b0 osd=1 1 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.2 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7efdf01ac3e0 msgr2=0x7efdf01ae7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7efdf01ac3e0 0x7efdf01ae7d0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109fe0 osd=2 1 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.3 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7efdf01a9d80 msgr2=0x7efdf01b1ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7efdf01a9d80 0x7efdf01b1ea0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf0109630 osd=3 1 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.4 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7efdf01af9f0 msgr2=0x7efdf01b3560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7efdf01af9f0 0x7efdf01b3560 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a90a0 osd=4 1 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.5 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7efdf01b3fb0 msgr2=0x7efdf01b4430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7efdf01b3fb0 0x7efdf01b4430 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01a9210 osd=5 1 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.6 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdeffff700 1 -- 172.21.15.69:0/2737914966 reap_dead start 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7efdf01b9260 msgr2=0x7efdf01bb650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7efdf01b9260 0x7efdf01bb650 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdeffff700 1 -- 172.21.15.69:0/2737914966 reap_dead start 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdeffff700 1 -- 172.21.15.69:0/2737914966 reap_dead start 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01b4ab0 osd=6 1 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter close_session for osd.7 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7efdf01bdba0 msgr2=0x7efdf01be000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7efdf01bdba0 0x7efdf01be000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter put_session s=0x7efdf01bd480 osd=7 1 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 20 client.26564.objecter shutdown clearing up homeless session... 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 10 client.26564.objecter successfully canceled tick 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efdd806f5e0 msgr2=0x7efdd8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efdd806f5e0 0x7efdd8071aa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efdf0106250 tx=0x7efde0008040 comp rx=0 tx=0).stop 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 msgr2=0x7efdf0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 0x7efdf0102ec0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efde400e9a0 tx=0x7efde400ecb0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 shutdown_connections 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7efdf01bdba0 0x7efdf01be000 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7efdf01b9260 0x7efdf01bb650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdf0101420 0x7efdf0102ec0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 --2- 172.21.15.69:0/2737914966 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efdd806f5e0 0x7efdd8071aa0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 >> 172.21.15.69:0/2737914966 conn(0x7efdf0078240 msgr2=0x7efdf006a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 shutdown_connections 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 -- 172.21.15.69:0/2737914966 wait complete. 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:07.067+0000 7efdf5c03700 1 librados: shutdown 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:55:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:55:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:55:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:55:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:55:07.070 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:55:07.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:07.171 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:07.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:07.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[28904]: audit 2024-04-02T04:55:06.117329+0000 mon.b (mon.1) 230 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-02T04:55:07.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[28904]: cluster 2024-04-02T04:55:06.121187+0000 mgr.y (mgr.14538) 448 : cluster 0 pgmap v668: 352 pgs: 4 creating+peering, 28 unknown, 320 active+clean; 8.4 MiB data, 488 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:55:07.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[28904]: audit 2024-04-02T04:55:06.125028+0000 mon.a (mon.0) 2765 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-02T04:55:07.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[28904]: audit 2024-04-02T04:55:06.863779+0000 mon.c (mon.2) 838 : audit 0 from='client.? 172.21.15.69:0/2737914966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:07.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[28904]: audit 2024-04-02T04:55:06.902937+0000 mon.a (mon.0) 2766 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-02T04:55:07.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[28904]: cluster 2024-04-02T04:55:06.908177+0000 mon.a (mon.0) 2767 : cluster 0 osdmap e459: 8 total, 8 up, 7 in 2024-04-02T04:55:07.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[21528]: audit 2024-04-02T04:55:06.117329+0000 mon.b (mon.1) 230 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-02T04:55:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[21528]: cluster 2024-04-02T04:55:06.121187+0000 mgr.y (mgr.14538) 448 : cluster 0 pgmap v668: 352 pgs: 4 creating+peering, 28 unknown, 320 active+clean; 8.4 MiB data, 488 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:55:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[21528]: audit 2024-04-02T04:55:06.125028+0000 mon.a (mon.0) 2765 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-02T04:55:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[21528]: audit 2024-04-02T04:55:06.863779+0000 mon.c (mon.2) 838 : audit 0 from='client.? 172.21.15.69:0/2737914966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[21528]: audit 2024-04-02T04:55:06.902937+0000 mon.a (mon.0) 2766 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-02T04:55:07.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:07 smithi069 bash[21528]: cluster 2024-04-02T04:55:06.908177+0000 mon.a (mon.0) 2767 : cluster 0 osdmap e459: 8 total, 8 up, 7 in 2024-04-02T04:55:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.298 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:07.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:07.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:07.410 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:07 smithi098 bash[23790]: audit 2024-04-02T04:55:06.117329+0000 mon.b (mon.1) 230 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-02T04:55:07.410 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:07 smithi098 bash[23790]: cluster 2024-04-02T04:55:06.121187+0000 mgr.y (mgr.14538) 448 : cluster 0 pgmap v668: 352 pgs: 4 creating+peering, 28 unknown, 320 active+clean; 8.4 MiB data, 488 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:55:07.411 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:07 smithi098 bash[23790]: audit 2024-04-02T04:55:06.125028+0000 mon.a (mon.0) 2765 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-02T04:55:07.411 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:07 smithi098 bash[23790]: audit 2024-04-02T04:55:06.863779+0000 mon.c (mon.2) 838 : audit 0 from='client.? 172.21.15.69:0/2737914966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:07.411 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:07 smithi098 bash[23790]: audit 2024-04-02T04:55:06.902937+0000 mon.a (mon.0) 2766 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-02T04:55:07.411 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:07 smithi098 bash[23790]: cluster 2024-04-02T04:55:06.908177+0000 mon.a (mon.0) 2767 : cluster 0 osdmap e459: 8 total, 8 up, 7 in 2024-04-02T04:55:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:07.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:07.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:07.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:07.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:07.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:07.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:07.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:07.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:07.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:07.944 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:07.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:08.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:08.072 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:08.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:08.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:08.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:08 smithi098 bash[23790]: cluster 2024-04-02T04:55:07.018023+0000 osd.4 (osd.4) 245 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:08.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:08 smithi098 bash[23790]: cluster 2024-04-02T04:55:07.018296+0000 osd.4 (osd.4) 246 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:08.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:08 smithi098 bash[23790]: cluster 2024-04-02T04:55:07.912745+0000 mon.a (mon.0) 2768 : cluster 0 osdmap e460: 8 total, 8 up, 7 in 2024-04-02T04:55:08.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:08.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:08.527 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[21528]: cluster 2024-04-02T04:55:07.018023+0000 osd.4 (osd.4) 245 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:08.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[21528]: cluster 2024-04-02T04:55:07.018296+0000 osd.4 (osd.4) 246 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:08.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[21528]: cluster 2024-04-02T04:55:07.912745+0000 mon.a (mon.0) 2768 : cluster 0 osdmap e460: 8 total, 8 up, 7 in 2024-04-02T04:55:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[28904]: cluster 2024-04-02T04:55:07.018023+0000 osd.4 (osd.4) 245 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[28904]: cluster 2024-04-02T04:55:07.018296+0000 osd.4 (osd.4) 246 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:08.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[28904]: cluster 2024-04-02T04:55:07.912745+0000 mon.a (mon.0) 2768 : cluster 0 osdmap e460: 8 total, 8 up, 7 in 2024-04-02T04:55:08.565 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:08.592 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:08.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:08.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:08.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:08.962 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:55:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:55:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:55:08.977 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:08.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:09.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:09.102 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:09.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:09.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:09 smithi069 bash[21528]: cluster 2024-04-02T04:55:08.122939+0000 mgr.y (mgr.14538) 449 : cluster 0 pgmap v671: 352 pgs: 2 clean+premerge+peered, 4 creating+peering, 346 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-02T04:55:09.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:09 smithi069 bash[21528]: cluster 2024-04-02T04:55:08.917231+0000 mon.a (mon.0) 2769 : cluster 0 osdmap e461: 8 total, 8 up, 7 in 2024-04-02T04:55:09.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:09 smithi069 bash[28904]: cluster 2024-04-02T04:55:08.122939+0000 mgr.y (mgr.14538) 449 : cluster 0 pgmap v671: 352 pgs: 2 clean+premerge+peered, 4 creating+peering, 346 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-02T04:55:09.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:09 smithi069 bash[28904]: cluster 2024-04-02T04:55:08.917231+0000 mon.a (mon.0) 2769 : cluster 0 osdmap e461: 8 total, 8 up, 7 in 2024-04-02T04:55:09.226 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:09.290 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1127167782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec106500 msgr2=0x7fc6ec10e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1127167782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec106500 0x7fc6ec10e0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0099c0 tx=0x7fc6dc009cd0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1127167782 shutdown_connections 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1127167782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec106500 0x7fc6ec10e0d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1127167782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101420 0x7fc6ec101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1127167782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec101dd0 0x7fc6ec105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1127167782 >> 172.21.15.69:0/1127167782 conn(0x7fc6ec078240 msgr2=0x7fc6ec078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:09.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1127167782 shutdown_connections 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.311+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1127167782 wait complete. 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 librados: starting msgr at 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 librados: starting objecter 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 Processor -- start 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- start start 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 librados: setting wanted keys 2024-04-02T04:55:09.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 librados: calling monclient init 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 0x7fc6ec102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101dd0 0x7fc6ec103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec106500 0x7fc6ec103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1109f0 con 0x7fc6ec101420 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec110870 con 0x7fc6ec101dd0 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec110b70 con 0x7fc6ec106500 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 0x7fc6ec102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 0x7fc6ec102ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60020/0 (socket says 172.21.15.69:60020) 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 -- 172.21.15.69:0/1614673936 learned_addr learned my addr 172.21.15.69:0/1614673936 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1949700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec106500 0x7fc6ec103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:09.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 -- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec106500 msgr2=0x7fc6ec103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f0947700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101dd0 0x7fc6ec103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec106500 0x7fc6ec103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 -- 172.21.15.69:0/1614673936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101dd0 msgr2=0x7fc6ec103400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101dd0 0x7fc6ec103400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 -- 172.21.15.69:0/1614673936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec1a6550 con 0x7fc6ec101420 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1949700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec106500 0x7fc6ec103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f0947700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101dd0 0x7fc6ec103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f1148700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 0x7fc6ec102ec0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4007e20 tx=0x7fc6e400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 10 client.?.objecter ms_handle_connect 0x7fc6ec101420 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4016020 con 0x7fc6ec101420 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc6e400ad40 con 0x7fc6ec101420 2024-04-02T04:55:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40115e0 con 0x7fc6ec101420 2024-04-02T04:55:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec1a6840 con 0x7fc6ec101420 2024-04-02T04:55:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 10 client.16476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1a6d80 con 0x7fc6ec101420 2024-04-02T04:55:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 librados: init done 2024-04-02T04:55:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec04ff00 con 0x7fc6ec101420 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc6e4019020 con 0x7fc6ec101420 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc6d406f5e0 0x7fc6d4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.315+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(461..461 src has 74..461) ==== 10981+0+0 (secure 0 0 0) 0x7fc6e408e870 con 0x7fc6ec101420 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6ea7fc700 10 client.16476.objecter ms_dispatch 0x7fc6ec001550 osd_map(461..461 src has 74..461) 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6ea7fc700 3 client.16476.objecter handle_osd_map got epochs [461,461] > 0 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6ea7fc700 3 client.16476.objecter handle_osd_map decoding full epoch 461 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6f0947700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc6d406f5e0 0x7fc6d4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6ea7fc700 20 client.16476.objecter dump_active .. 0 homeless 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6ea7fc700 10 client.16476.objecter ms_handle_connect 0x7fc6d406f5e0 2024-04-02T04:55:09.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.319+0000 7fc6f0947700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc6d406f5e0 0x7fc6d4071aa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec06b8e0 tx=0x7fc6e0009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:09.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.323+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc6e4020070 con 0x7fc6ec101420 2024-04-02T04:55:09.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:09.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:09 smithi098 bash[23790]: cluster 2024-04-02T04:55:08.122939+0000 mgr.y (mgr.14538) 449 : cluster 0 pgmap v671: 352 pgs: 2 clean+premerge+peered, 4 creating+peering, 346 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-02T04:55:09.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:09 smithi098 bash[23790]: cluster 2024-04-02T04:55:08.917231+0000 mon.a (mon.0) 2769 : cluster 0 osdmap e461: 8 total, 8 up, 7 in 2024-04-02T04:55:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:09.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:09.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.543+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fc6ec1043a0 con 0x7fc6ec101420 2024-04-02T04:55:09.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:09.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:09.732 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:09.774 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:55:09.774 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:55:09.774 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T04:55:09.774 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:55:09.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:09.854 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:09.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.907+0000 7fc6ea7fc700 1 -- 172.21.15.69:0/1614673936 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v462) ==== 91+0+0 (secure 0 0 0) 0x7fc6ec1043a0 con 0x7fc6ec101420 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 10 librados: watch_flush enter 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 10 librados: watch_flush exit 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 20 client.16476.objecter shutdown clearing up homeless session... 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 10 client.16476.objecter successfully canceled tick 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc6d406f5e0 msgr2=0x7fc6d4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc6d406f5e0 0x7fc6d4071aa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec06b8e0 tx=0x7fc6e0009260 comp rx=0 tx=0).stop 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 msgr2=0x7fc6ec102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.915+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 0x7fc6ec102ec0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4007e20 tx=0x7fc6e400a340 comp rx=0 tx=0).stop 2024-04-02T04:55:09.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 shutdown_connections 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc6d406f5e0 0x7fc6d4071aa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6ec106500 0x7fc6ec103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6ec101420 0x7fc6ec102ec0 secure :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4007e20 tx=0x7fc6e400a340 comp rx=0 tx=0).stop 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 --2- 172.21.15.69:0/1614673936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec101dd0 0x7fc6ec103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 >> 172.21.15.69:0/1614673936 conn(0x7fc6ec078240 msgr2=0x7fc6ec105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 shutdown_connections 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 -- 172.21.15.69:0/1614673936 wait complete. 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:09.919+0000 7fc6f33ac700 1 librados: shutdown 2024-04-02T04:55:09.917 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:55:09.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:09.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:10.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:10.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:10 smithi098 bash[23790]: audit 2024-04-02T04:55:09.547459+0000 mon.a (mon.0) 2770 : audit 1 from='client.? 172.21.15.69:0/1614673936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:55:10.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:10 smithi098 bash[23790]: cluster 2024-04-02T04:55:09.908616+0000 mon.a (mon.0) 2771 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:10.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:10 smithi098 bash[23790]: audit 2024-04-02T04:55:09.911093+0000 mon.a (mon.0) 2772 : audit 1 from='client.? 172.21.15.69:0/1614673936' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:55:10.092 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:10 smithi098 bash[23790]: cluster 2024-04-02T04:55:09.919024+0000 mon.a (mon.0) 2773 : cluster 0 osdmap e462: 8 total, 8 up, 7 in 2024-04-02T04:55:10.107 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:10.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[21528]: audit 2024-04-02T04:55:09.547459+0000 mon.a (mon.0) 2770 : audit 1 from='client.? 172.21.15.69:0/1614673936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:55:10.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[21528]: cluster 2024-04-02T04:55:09.908616+0000 mon.a (mon.0) 2771 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:10.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[21528]: audit 2024-04-02T04:55:09.911093+0000 mon.a (mon.0) 2772 : audit 1 from='client.? 172.21.15.69:0/1614673936' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:55:10.200 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[21528]: cluster 2024-04-02T04:55:09.919024+0000 mon.a (mon.0) 2773 : cluster 0 osdmap e462: 8 total, 8 up, 7 in 2024-04-02T04:55:10.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[28904]: audit 2024-04-02T04:55:09.547459+0000 mon.a (mon.0) 2770 : audit 1 from='client.? 172.21.15.69:0/1614673936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:55:10.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[28904]: cluster 2024-04-02T04:55:09.908616+0000 mon.a (mon.0) 2771 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:10.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[28904]: audit 2024-04-02T04:55:09.911093+0000 mon.a (mon.0) 2772 : audit 1 from='client.? 172.21.15.69:0/1614673936' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:55:10.201 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:10 smithi069 bash[28904]: cluster 2024-04-02T04:55:09.919024+0000 mon.a (mon.0) 2773 : cluster 0 osdmap e462: 8 total, 8 up, 7 in 2024-04-02T04:55:10.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:10.238 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:10.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:10.370 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:10.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:10.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:10.629 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:10.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-02T04:55:10.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:10.759 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:10.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:55:10.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:10.900 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:10.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:11.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:11.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:11.150 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:11.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:11.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: cluster 2024-04-02T04:55:09.936875+0000 osd.4 (osd.4) 247 : cluster 0 13.e scrub starts 2024-04-02T04:55:11.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: cluster 2024-04-02T04:55:09.938276+0000 osd.4 (osd.4) 248 : cluster 0 13.e scrub ok 2024-04-02T04:55:11.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: audit 2024-04-02T04:55:10.119871+0000 mon.b (mon.1) 231 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-02T04:55:11.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: cluster 2024-04-02T04:55:10.123753+0000 mgr.y (mgr.14538) 450 : cluster 0 pgmap v674: 351 pgs: 1 clean+premerge+peered, 350 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:55:11.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: audit 2024-04-02T04:55:10.125935+0000 mon.a (mon.0) 2774 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-02T04:55:11.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: cluster 2024-04-02T04:55:10.258750+0000 osd.1 (osd.1) 257 : cluster 0 102.15 scrub starts 2024-04-02T04:55:11.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:11 smithi098 bash[23790]: cluster 2024-04-02T04:55:10.262622+0000 osd.1 (osd.1) 258 : cluster 0 102.15 scrub ok 2024-04-02T04:55:11.279 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: cluster 2024-04-02T04:55:09.936875+0000 osd.4 (osd.4) 247 : cluster 0 13.e scrub starts 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: cluster 2024-04-02T04:55:09.938276+0000 osd.4 (osd.4) 248 : cluster 0 13.e scrub ok 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: audit 2024-04-02T04:55:10.119871+0000 mon.b (mon.1) 231 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: cluster 2024-04-02T04:55:10.123753+0000 mgr.y (mgr.14538) 450 : cluster 0 pgmap v674: 351 pgs: 1 clean+premerge+peered, 350 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: audit 2024-04-02T04:55:10.125935+0000 mon.a (mon.0) 2774 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: cluster 2024-04-02T04:55:10.258750+0000 osd.1 (osd.1) 257 : cluster 0 102.15 scrub starts 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[21528]: cluster 2024-04-02T04:55:10.262622+0000 osd.1 (osd.1) 258 : cluster 0 102.15 scrub ok 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: cluster 2024-04-02T04:55:09.936875+0000 osd.4 (osd.4) 247 : cluster 0 13.e scrub starts 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: cluster 2024-04-02T04:55:09.938276+0000 osd.4 (osd.4) 248 : cluster 0 13.e scrub ok 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: audit 2024-04-02T04:55:10.119871+0000 mon.b (mon.1) 231 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-02T04:55:11.353 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: cluster 2024-04-02T04:55:10.123753+0000 mgr.y (mgr.14538) 450 : cluster 0 pgmap v674: 351 pgs: 1 clean+premerge+peered, 350 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:55:11.354 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: audit 2024-04-02T04:55:10.125935+0000 mon.a (mon.0) 2774 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-02T04:55:11.354 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: cluster 2024-04-02T04:55:10.258750+0000 osd.1 (osd.1) 257 : cluster 0 102.15 scrub starts 2024-04-02T04:55:11.354 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:11 smithi069 bash[28904]: cluster 2024-04-02T04:55:10.262622+0000 osd.1 (osd.1) 258 : cluster 0 102.15 scrub ok 2024-04-02T04:55:11.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:11.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:11.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:11.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:11.670 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:11.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:11.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:11.927 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:11.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:12.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:12.054 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:12.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:12.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:12.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.308 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:12.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:12.422 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:12 smithi098 bash[23790]: audit 2024-04-02T04:55:11.075950+0000 mon.a (mon.0) 2775 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-02T04:55:12.422 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:12 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.081737+0000 mon.a (mon.0) 2776 : cluster 0 osdmap e463: 8 total, 8 up, 7 in 2024-04-02T04:55:12.422 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:12 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.120736+0000 osd.0 (osd.0) 235 : cluster 0 13.b scrub starts 2024-04-02T04:55:12.423 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:12 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.124543+0000 osd.0 (osd.0) 236 : cluster 0 13.b scrub ok 2024-04-02T04:55:12.423 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:12 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.231029+0000 osd.1 (osd.1) 259 : cluster 0 6.1d scrub starts 2024-04-02T04:55:12.423 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:12 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.232525+0000 osd.1 (osd.1) 260 : cluster 0 6.1d scrub ok 2024-04-02T04:55:12.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[28904]: audit 2024-04-02T04:55:11.075950+0000 mon.a (mon.0) 2775 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.081737+0000 mon.a (mon.0) 2776 : cluster 0 osdmap e463: 8 total, 8 up, 7 in 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.120736+0000 osd.0 (osd.0) 235 : cluster 0 13.b scrub starts 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.124543+0000 osd.0 (osd.0) 236 : cluster 0 13.b scrub ok 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.231029+0000 osd.1 (osd.1) 259 : cluster 0 6.1d scrub starts 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.232525+0000 osd.1 (osd.1) 260 : cluster 0 6.1d scrub ok 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[21528]: audit 2024-04-02T04:55:11.075950+0000 mon.a (mon.0) 2775 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-02T04:55:12.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.081737+0000 mon.a (mon.0) 2776 : cluster 0 osdmap e463: 8 total, 8 up, 7 in 2024-04-02T04:55:12.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.120736+0000 osd.0 (osd.0) 235 : cluster 0 13.b scrub starts 2024-04-02T04:55:12.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.124543+0000 osd.0 (osd.0) 236 : cluster 0 13.b scrub ok 2024-04-02T04:55:12.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.231029+0000 osd.1 (osd.1) 259 : cluster 0 6.1d scrub starts 2024-04-02T04:55:12.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:12 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.232525+0000 osd.1 (osd.1) 260 : cluster 0 6.1d scrub ok 2024-04-02T04:55:12.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:12.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:12.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:12.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.958 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:12.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:13.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:13.081 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:13.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:55:13.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:13.207 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:13.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:13.331 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:13.332 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.394786+0000 osd.7 (osd.7) 37 : cluster 0 13.0 scrub starts 2024-04-02T04:55:13.332 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.403951+0000 osd.7 (osd.7) 38 : cluster 0 13.0 scrub ok 2024-04-02T04:55:13.332 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.974206+0000 osd.4 (osd.4) 249 : cluster 0 102.1f scrub starts 2024-04-02T04:55:13.332 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: cluster 2024-04-02T04:55:11.979929+0000 osd.4 (osd.4) 250 : cluster 0 102.1f scrub ok 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: cluster 2024-04-02T04:55:12.087349+0000 mon.a (mon.0) 2777 : cluster 0 osdmap e464: 8 total, 8 up, 7 in 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: cluster 2024-04-02T04:55:12.124587+0000 mgr.y (mgr.14538) 451 : cluster 0 pgmap v677: 351 pgs: 1 clean+premerge+peered, 350 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[21528]: audit 2024-04-02T04:55:12.889847+0000 mon.b (mon.1) 232 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.394786+0000 osd.7 (osd.7) 37 : cluster 0 13.0 scrub starts 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.403951+0000 osd.7 (osd.7) 38 : cluster 0 13.0 scrub ok 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.974206+0000 osd.4 (osd.4) 249 : cluster 0 102.1f scrub starts 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: cluster 2024-04-02T04:55:11.979929+0000 osd.4 (osd.4) 250 : cluster 0 102.1f scrub ok 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: cluster 2024-04-02T04:55:12.087349+0000 mon.a (mon.0) 2777 : cluster 0 osdmap e464: 8 total, 8 up, 7 in 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: cluster 2024-04-02T04:55:12.124587+0000 mgr.y (mgr.14538) 451 : cluster 0 pgmap v677: 351 pgs: 1 clean+premerge+peered, 350 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:13.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:13 smithi069 bash[28904]: audit 2024-04-02T04:55:12.889847+0000 mon.b (mon.1) 232 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/1978747161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e8102520 msgr2=0x7fb6e8102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- 172.21.15.69:0/1978747161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e8102520 0x7fb6e8102900 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80099c0 tx=0x7fb6d8009cd0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/1978747161 shutdown_connections 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- 172.21.15.69:0/1978747161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6e81075a0 0x7fb6e810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- 172.21.15.69:0/1978747161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e8102520 0x7fb6e8102900 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- 172.21.15.69:0/1978747161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/1978747161 >> 172.21.15.69:0/1978747161 conn(0x7fb6e8079360 msgr2=0x7fb6e8079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:13.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/1978747161 shutdown_connections 2024-04-02T04:55:13.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/1978747161 wait complete. 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 librados: starting msgr at 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 librados: starting objecter 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 Processor -- start 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- start start 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 librados: setting wanted keys 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 librados: calling monclient init 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6e8102520 0x7fb6e8074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:13.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e81075a0 0x7fb6e8074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e8111a50 con 0x7fb6e81075a0 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e81118d0 con 0x7fb6e8102520 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e8111bd0 con 0x7fb6e8102ed0 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ecda1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e81075a0 0x7fb6e8074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38656/0 (socket says 172.21.15.69:38656) 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 -- 172.21.15.69:0/552457806 learned_addr learned my addr 172.21.15.69:0/552457806 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e7fff700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6e8102520 0x7fb6e8074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 -- 172.21.15.69:0/552457806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6e8102520 msgr2=0x7fb6e8074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6e8102520 0x7fb6e8074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 -- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e81075a0 msgr2=0x7fb6e8074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e81075a0 0x7fb6e8074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 -- 172.21.15.69:0/552457806 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e81a65d0 con 0x7fb6e8102ed0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ecda1700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e81075a0 0x7fb6e8074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e77fe700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8074a80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc00ea30 tx=0x7fb6dc00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e57fa700 10 client.?.objecter ms_handle_connect 0x7fb6e8102ed0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e57fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc00cbd0 con 0x7fb6e8102ed0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb6dc0044a0 con 0x7fb6e8102ed0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e81a68c0 con 0x7fb6e8102ed0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc003c70 con 0x7fb6e8102ed0 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 10 client.26576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:13.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.435+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e81a6e00 con 0x7fb6e8102ed0 2024-04-02T04:55:13.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6ee804700 1 librados: init done 2024-04-02T04:55:13.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e804ff00 con 0x7fb6e8102ed0 2024-04-02T04:55:13.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:13.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb6dc003e70 con 0x7fb6e8102ed0 2024-04-02T04:55:13.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb6d006f5e0 0x7fb6d0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:13.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(465..465 src has 74..465) ==== 10953+0+0 (secure 0 0 0) 0x7fb6dc08ef10 con 0x7fb6e8102ed0 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 10 client.26576.objecter ms_dispatch 0x7fb6e8001550 osd_map(465..465 src has 74..465) 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 3 client.26576.objecter handle_osd_map got epochs [465,465] > 0 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 3 client.26576.objecter handle_osd_map decoding full epoch 465 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e7fff700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb6d006f5e0 0x7fb6d0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 20 client.26576.objecter dump_active .. 0 homeless 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e57fa700 10 client.26576.objecter ms_handle_connect 0x7fb6d006f5e0 2024-04-02T04:55:13.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.439+0000 7fb6e7fff700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb6d006f5e0 0x7fb6d0071aa0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80037a0 tx=0x7fb6d8019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:13.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.443+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb6dc05b750 con 0x7fb6e8102ed0 2024-04-02T04:55:13.451 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:55:13.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.394786+0000 osd.7 (osd.7) 37 : cluster 0 13.0 scrub starts 2024-04-02T04:55:13.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.403951+0000 osd.7 (osd.7) 38 : cluster 0 13.0 scrub ok 2024-04-02T04:55:13.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.974206+0000 osd.4 (osd.4) 249 : cluster 0 102.1f scrub starts 2024-04-02T04:55:13.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: cluster 2024-04-02T04:55:11.979929+0000 osd.4 (osd.4) 250 : cluster 0 102.1f scrub ok 2024-04-02T04:55:13.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: cluster 2024-04-02T04:55:12.087349+0000 mon.a (mon.0) 2777 : cluster 0 osdmap e464: 8 total, 8 up, 7 in 2024-04-02T04:55:13.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: cluster 2024-04-02T04:55:12.124587+0000 mgr.y (mgr.14538) 451 : cluster 0 pgmap v677: 351 pgs: 1 clean+premerge+peered, 350 active+clean; 8.4 MiB data, 492 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:13.452 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:13 smithi098 bash[23790]: audit 2024-04-02T04:55:12.889847+0000 mon.b (mon.1) 232 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:13.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:13.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.671+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6e8100840 con 0x7fb6d006f5e0 2024-04-02T04:55:13.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:13.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:13.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.723+0000 7fb6e57fa700 1 -- 172.21.15.69:0/552457806 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+893989 (secure 0 0 0) 0x7fb6e8100840 con 0x7fb6d006f5e0 2024-04-02T04:55:13.733 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:55:13.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 10 librados: watch_flush enter 2024-04-02T04:55:13.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 10 librados: watch_flush exit 2024-04-02T04:55:13.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 20 client.26576.objecter shutdown clearing up homeless session... 2024-04-02T04:55:13.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 10 client.26576.objecter successfully canceled tick 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb6d006f5e0 msgr2=0x7fb6d0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb6d006f5e0 0x7fb6d0071aa0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80037a0 tx=0x7fb6d8019040 comp rx=0 tx=0).stop 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 msgr2=0x7fb6e8074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8074a80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc00ea30 tx=0x7fb6dc00ed40 comp rx=0 tx=0).stop 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 shutdown_connections 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb6d006f5e0 0x7fb6d0071aa0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6e81075a0 0x7fb6e8074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6e8102520 0x7fb6e8074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 --2- 172.21.15.69:0/552457806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6e8102ed0 0x7fb6e8074a80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:13.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 >> 172.21.15.69:0/552457806 conn(0x7fb6e8079360 msgr2=0x7fb6e8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:13.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 shutdown_connections 2024-04-02T04:55:13.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 -- 172.21.15.69:0/552457806 wait complete. 2024-04-02T04:55:13.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:13.731+0000 7fb6ee804700 1 librados: shutdown 2024-04-02T04:55:13.746 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:55:13.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:13.835 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:13.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:13.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:14.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:14.085 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:14.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:14.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.045869+0000 osd.3 (osd.3) 287 : cluster 0 5.0 scrub starts 2024-04-02T04:55:14.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.047913+0000 osd.3 (osd.3) 288 : cluster 0 5.0 scrub ok 2024-04-02T04:55:14.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.106719+0000 mon.a (mon.0) 2778 : cluster 0 osdmap e465: 8 total, 8 up, 7 in 2024-04-02T04:55:14.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.109024+0000 osd.0 (osd.0) 237 : cluster 0 4.1e scrub starts 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.114434+0000 osd.0 (osd.0) 238 : cluster 0 4.1e scrub ok 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.293147+0000 mgr.y (mgr.14538) 452 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.424177+0000 mon.a (mon.0) 2779 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.435649+0000 mon.a (mon.0) 2780 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.475465+0000 mon.b (mon.1) 233 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.477572+0000 mon.b (mon.1) 234 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.478942+0000 mon.b (mon.1) 235 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.480630+0000 mon.a (mon.0) 2781 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.480904+0000 mon.b (mon.1) 236 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.482686+0000 mon.a (mon.0) 2782 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.484179+0000 mon.a (mon.0) 2783 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.485923+0000 mon.a (mon.0) 2784 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.201 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cephadm 2024-04-02T04:55:13.487130+0000 mgr.y (mgr.14538) 453 : cephadm 1 Adjusting osd_memory_target on smithi098 to 3292M 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.491785+0000 mon.a (mon.0) 2785 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:13.673199+0000 mgr.y (mgr.14538) 454 : audit 0 from='client.26576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.825688+0000 osd.2 (osd.2) 206 : cluster 0 2.1 scrub starts 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: cluster 2024-04-02T04:55:13.826917+0000 osd.2 (osd.2) 207 : cluster 0 2.1 scrub ok 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.025447+0000 mon.b (mon.1) 237 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.027300+0000 mon.b (mon.1) 238 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.028562+0000 mon.b (mon.1) 239 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.029717+0000 mon.b (mon.1) 240 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.030637+0000 mon.a (mon.0) 2786 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.032261+0000 mon.a (mon.0) 2787 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.033532+0000 mon.a (mon.0) 2788 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.034682+0000 mon.a (mon.0) 2789 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.202 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.037927+0000 mon.b (mon.1) 241 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:14.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.038785+0000 mon.b (mon.1) 242 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:14.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.040570+0000 mon.a (mon.0) 2790 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.203 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:14 smithi098 bash[23790]: audit 2024-04-02T04:55:14.050599+0000 mon.a (mon.0) 2791 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.214 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:14.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:14.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:14.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:14.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:14.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.045869+0000 osd.3 (osd.3) 287 : cluster 0 5.0 scrub starts 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.047913+0000 osd.3 (osd.3) 288 : cluster 0 5.0 scrub ok 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.106719+0000 mon.a (mon.0) 2778 : cluster 0 osdmap e465: 8 total, 8 up, 7 in 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.109024+0000 osd.0 (osd.0) 237 : cluster 0 4.1e scrub starts 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.114434+0000 osd.0 (osd.0) 238 : cluster 0 4.1e scrub ok 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.293147+0000 mgr.y (mgr.14538) 452 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.424177+0000 mon.a (mon.0) 2779 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.435649+0000 mon.a (mon.0) 2780 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.475465+0000 mon.b (mon.1) 233 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.477572+0000 mon.b (mon.1) 234 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.478942+0000 mon.b (mon.1) 235 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.480630+0000 mon.a (mon.0) 2781 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.480904+0000 mon.b (mon.1) 236 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.482686+0000 mon.a (mon.0) 2782 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.484179+0000 mon.a (mon.0) 2783 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.485923+0000 mon.a (mon.0) 2784 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cephadm 2024-04-02T04:55:13.487130+0000 mgr.y (mgr.14538) 453 : cephadm 1 Adjusting osd_memory_target on smithi098 to 3292M 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.491785+0000 mon.a (mon.0) 2785 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:13.673199+0000 mgr.y (mgr.14538) 454 : audit 0 from='client.26576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.825688+0000 osd.2 (osd.2) 206 : cluster 0 2.1 scrub starts 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: cluster 2024-04-02T04:55:13.826917+0000 osd.2 (osd.2) 207 : cluster 0 2.1 scrub ok 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.025447+0000 mon.b (mon.1) 237 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.045869+0000 osd.3 (osd.3) 287 : cluster 0 5.0 scrub starts 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.047913+0000 osd.3 (osd.3) 288 : cluster 0 5.0 scrub ok 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.106719+0000 mon.a (mon.0) 2778 : cluster 0 osdmap e465: 8 total, 8 up, 7 in 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.109024+0000 osd.0 (osd.0) 237 : cluster 0 4.1e scrub starts 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.114434+0000 osd.0 (osd.0) 238 : cluster 0 4.1e scrub ok 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.293147+0000 mgr.y (mgr.14538) 452 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.424177+0000 mon.a (mon.0) 2779 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.435649+0000 mon.a (mon.0) 2780 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.475465+0000 mon.b (mon.1) 233 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.477572+0000 mon.b (mon.1) 234 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.478942+0000 mon.b (mon.1) 235 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.480630+0000 mon.a (mon.0) 2781 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.480904+0000 mon.b (mon.1) 236 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.482686+0000 mon.a (mon.0) 2782 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.484179+0000 mon.a (mon.0) 2783 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.485923+0000 mon.a (mon.0) 2784 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cephadm 2024-04-02T04:55:13.487130+0000 mgr.y (mgr.14538) 453 : cephadm 1 Adjusting osd_memory_target on smithi098 to 3292M 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.491785+0000 mon.a (mon.0) 2785 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:13.673199+0000 mgr.y (mgr.14538) 454 : audit 0 from='client.26576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.825688+0000 osd.2 (osd.2) 206 : cluster 0 2.1 scrub starts 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: cluster 2024-04-02T04:55:13.826917+0000 osd.2 (osd.2) 207 : cluster 0 2.1 scrub ok 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.025447+0000 mon.b (mon.1) 237 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.027300+0000 mon.b (mon.1) 238 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.028562+0000 mon.b (mon.1) 239 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.029717+0000 mon.b (mon.1) 240 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.030637+0000 mon.a (mon.0) 2786 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.032261+0000 mon.a (mon.0) 2787 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.033532+0000 mon.a (mon.0) 2788 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.034682+0000 mon.a (mon.0) 2789 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.037927+0000 mon.b (mon.1) 241 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.038785+0000 mon.b (mon.1) 242 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.040570+0000 mon.a (mon.0) 2790 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[21528]: audit 2024-04-02T04:55:14.050599+0000 mon.a (mon.0) 2791 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.027300+0000 mon.b (mon.1) 238 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.028562+0000 mon.b (mon.1) 239 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.029717+0000 mon.b (mon.1) 240 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.030637+0000 mon.a (mon.0) 2786 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.032261+0000 mon.a (mon.0) 2787 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.033532+0000 mon.a (mon.0) 2788 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.034682+0000 mon.a (mon.0) 2789 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.037927+0000 mon.b (mon.1) 241 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.038785+0000 mon.b (mon.1) 242 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.040570+0000 mon.a (mon.0) 2790 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:14 smithi069 bash[28904]: audit 2024-04-02T04:55:14.050599+0000 mon.a (mon.0) 2791 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:14.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:14.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:14.625 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2024-04-02T04:55:13.101907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8794050,"num_objects":411,"num_object_clones":4,"num_object_copies":1233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":2409,"num_read_kb":2122,"num_write":1804,"num_write_kb":15215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":5777569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3147,"ondisk_log_size":3147,"up":1050,"acting":1050,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1053,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":508884,"kb_used_data":82460,"kb_used_omap":66,"kb_used_meta":426109,"kb_avail":655707180,"statfs":{"total":671965249536,"available":671444152320,"internally_reserved":0,"allocated":84439040,"data_stored":62883336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68049,"internal_metadata":436336175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":123,"num_objects":0,"num_object_clones":-1,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.190489"},"pg_stats":[{"pgid":"294.1f","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.113429+0000","last_change":"2024-04-02T04:55:04.059964+0000","last_active":"2024-04-02T04:55:12.113429+0000","last_peered":"2024-04-02T04:55:12.113429+0000","last_clean":"2024-04-02T04:55:12.113429+0000","last_became_active":"2024-04-02T04:55:04.059618+0000","last_became_peered":"2024-04-02T04:55:04.059618+0000","last_unstale":"2024-04-02T04:55:12.113429+0000","last_undegraded":"2024-04-02T04:55:12.113429+0000","last_fullsized":"2024-04-02T04:55:12.113429+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:18.895747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"294.1e","version":"457'1","reported_seq":28,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448541+0000","last_change":"2024-04-02T04:55:04.076989+0000","last_active":"2024-04-02T04:55:12.448541+0000","last_peered":"2024-04-02T04:55:12.448541+0000","last_clean":"2024-04-02T04:55:12.448541+0000","last_became_active":"2024-04-02T04:55:04.076667+0000","last_became_peered":"2024-04-02T04:55:04.076667+0000","last_unstale":"2024-04-02T04:55:12.448541+0000","last_undegraded":"2024-04-02T04:55:12.448541+0000","last_fullsized":"2024-04-02T04:55:12.448541+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:07.559429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.1d","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.109046+0000","last_change":"2024-04-02T04:55:04.066928+0000","last_active":"2024-04-02T04:55:12.109046+0000","last_peered":"2024-04-02T04:55:12.109046+0000","last_clean":"2024-04-02T04:55:12.109046+0000","last_became_active":"2024-04-02T04:55:04.066016+0000","last_became_peered":"2024-04-02T04:55:04.066016+0000","last_unstale":"2024-04-02T04:55:12.109046+0000","last_undegraded":"2024-04-02T04:55:12.109046+0000","last_fullsized":"2024-04-02T04:55:12.109046+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:08.801244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"294.1c","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099777+0000","last_change":"2024-04-02T04:55:04.069351+0000","last_active":"2024-04-02T04:55:12.099777+0000","last_peered":"2024-04-02T04:55:12.099777+0000","last_clean":"2024-04-02T04:55:12.099777+0000","last_became_active":"2024-04-02T04:55:04.069015+0000","last_became_peered":"2024-04-02T04:55:04.069015+0000","last_unstale":"2024-04-02T04:55:12.099777+0000","last_undegraded":"2024-04-02T04:55:12.099777+0000","last_fullsized":"2024-04-02T04:55:12.099777+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:05.012164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"294.1b","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080097+0000","last_change":"2024-04-02T04:55:04.062313+0000","last_active":"2024-04-02T04:55:12.080097+0000","last_peered":"2024-04-02T04:55:12.080097+0000","last_clean":"2024-04-02T04:55:12.080097+0000","last_became_active":"2024-04-02T04:55:04.061661+0000","last_became_peered":"2024-04-02T04:55:04.061661+0000","last_unstale":"2024-04-02T04:55:12.080097+0000","last_undegraded":"2024-04-02T04:55:12.080097+0000","last_fullsized":"2024-04-02T04:55:12.080097+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:16.438498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"294.1a","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079397+0000","last_change":"2024-04-02T04:55:04.063674+0000","last_active":"2024-04-02T04:55:12.079397+0000","last_peered":"2024-04-02T04:55:12.079397+0000","last_clean":"2024-04-02T04:55:12.079397+0000","last_became_active":"2024-04-02T04:55:04.062993+0000","last_became_peered":"2024-04-02T04:55:04.062993+0000","last_unstale":"2024-04-02T04:55:12.079397+0000","last_undegraded":"2024-04-02T04:55:12.079397+0000","last_fullsized":"2024-04-02T04:55:12.079397+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:27.993258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"294.19","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.444624+0000","last_change":"2024-04-02T04:55:04.076938+0000","last_active":"2024-04-02T04:55:12.444624+0000","last_peered":"2024-04-02T04:55:12.444624+0000","last_clean":"2024-04-02T04:55:12.444624+0000","last_became_active":"2024-04-02T04:55:04.076543+0000","last_became_peered":"2024-04-02T04:55:04.076543+0000","last_unstale":"2024-04-02T04:55:12.444624+0000","last_undegraded":"2024-04-02T04:55:12.444624+0000","last_fullsized":"2024-04-02T04:55:12.444624+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:08.162342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.18","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.116346+0000","last_change":"2024-04-02T04:55:04.070786+0000","last_active":"2024-04-02T04:55:12.116346+0000","last_peered":"2024-04-02T04:55:12.116346+0000","last_clean":"2024-04-02T04:55:12.116346+0000","last_became_active":"2024-04-02T04:55:04.069943+0000","last_became_peered":"2024-04-02T04:55:04.069943+0000","last_unstale":"2024-04-02T04:55:12.116346+0000","last_undegraded":"2024-04-02T04:55:12.116346+0000","last_fullsized":"2024-04-02T04:55:12.116346+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:16.854291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"294.17","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449144+0000","last_change":"2024-04-02T04:55:04.074604+0000","last_active":"2024-04-02T04:55:12.449144+0000","last_peered":"2024-04-02T04:55:12.449144+0000","last_clean":"2024-04-02T04:55:12.449144+0000","last_became_active":"2024-04-02T04:55:04.067620+0000","last_became_peered":"2024-04-02T04:55:04.067620+0000","last_unstale":"2024-04-02T04:55:12.449144+0000","last_undegraded":"2024-04-02T04:55:12.449144+0000","last_fullsized":"2024-04-02T04:55:12.449144+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:20.760638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.16","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119310+0000","last_change":"2024-04-02T04:55:04.072324+0000","last_active":"2024-04-02T04:55:12.119310+0000","last_peered":"2024-04-02T04:55:12.119310+0000","last_clean":"2024-04-02T04:55:12.119310+0000","last_became_active":"2024-04-02T04:55:04.069534+0000","last_became_peered":"2024-04-02T04:55:04.069534+0000","last_unstale":"2024-04-02T04:55:12.119310+0000","last_undegraded":"2024-04-02T04:55:12.119310+0000","last_fullsized":"2024-04-02T04:55:12.119310+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:22.729769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"294.15","version":"0'0","reported_seq":21,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933328+0000","last_change":"2024-04-02T04:55:04.077809+0000","last_active":"2024-04-02T04:55:08.933328+0000","last_peered":"2024-04-02T04:55:08.933328+0000","last_clean":"2024-04-02T04:55:08.933328+0000","last_became_active":"2024-04-02T04:55:04.077457+0000","last_became_peered":"2024-04-02T04:55:04.077457+0000","last_unstale":"2024-04-02T04:55:08.933328+0000","last_undegraded":"2024-04-02T04:55:08.933328+0000","last_fullsized":"2024-04-02T04:55:08.933328+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:23.014991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"294.14","version":"461'4","reported_seq":34,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080595+0000","last_change":"2024-04-02T04:55:04.054202+0000","last_active":"2024-04-02T04:55:12.080595+0000","last_peered":"2024-04-02T04:55:12.080595+0000","last_clean":"2024-04-02T04:55:12.080595+0000","last_became_active":"2024-04-02T04:55:04.053771+0000","last_became_peered":"2024-04-02T04:55:04.053771+0000","last_unstale":"2024-04-02T04:55:12.080595+0000","last_undegraded":"2024-04-02T04:55:12.080595+0000","last_fullsized":"2024-04-02T04:55:12.080595+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:14.254694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"294.13","version":"0'0","reported_seq":21,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932312+0000","last_change":"2024-04-02T04:55:04.074034+0000","last_active":"2024-04-02T04:55:08.932312+0000","last_peered":"2024-04-02T04:55:08.932312+0000","last_clean":"2024-04-02T04:55:08.932312+0000","last_became_active":"2024-04-02T04:55:04.073665+0000","last_became_peered":"2024-04-02T04:55:04.073665+0000","last_unstale":"2024-04-02T04:55:08.932312+0000","last_undegraded":"2024-04-02T04:55:08.932312+0000","last_fullsized":"2024-04-02T04:55:08.932312+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:24.602606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"294.12","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.116389+0000","last_change":"2024-04-02T04:55:04.067359+0000","last_active":"2024-04-02T04:55:12.116389+0000","last_peered":"2024-04-02T04:55:12.116389+0000","last_clean":"2024-04-02T04:55:12.116389+0000","last_became_active":"2024-04-02T04:55:04.066992+0000","last_became_peered":"2024-04-02T04:55:04.066992+0000","last_unstale":"2024-04-02T04:55:12.116389+0000","last_undegraded":"2024-04-02T04:55:12.116389+0000","last_fullsized":"2024-04-02T04:55:12.116389+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:20.638833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"294.11","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097007+0000","last_change":"2024-04-02T04:55:04.058544+0000","last_active":"2024-04-02T04:55:12.097007+0000","last_peered":"2024-04-02T04:55:12.097007+0000","last_clean":"2024-04-02T04:55:12.097007+0000","last_became_active":"2024-04-02T04:55:04.058156+0000","last_became_peered":"2024-04-02T04:55:04.058156+0000","last_unstale":"2024-04-02T04:55:12.097007+0000","last_undegraded":"2024-04-02T04:55:12.097007+0000","last_fullsized":"2024-04-02T04:55:12.097007+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:03.792903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"294.10","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.444178+0000","last_change":"2024-04-02T04:55:04.076089+0000","last_active":"2024-04-02T04:55:12.444178+0000","last_peered":"2024-04-02T04:55:12.444178+0000","last_clean":"2024-04-02T04:55:12.444178+0000","last_became_active":"2024-04-02T04:55:04.067632+0000","last_became_peered":"2024-04-02T04:55:04.067632+0000","last_unstale":"2024-04-02T04:55:12.444178+0000","last_undegraded":"2024-04-02T04:55:12.444178+0000","last_fullsized":"2024-04-02T04:55:12.444178+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:07.790670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.f","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097947+0000","last_change":"2024-04-02T04:55:04.063444+0000","last_active":"2024-04-02T04:55:12.097947+0000","last_peered":"2024-04-02T04:55:12.097947+0000","last_clean":"2024-04-02T04:55:12.097947+0000","last_became_active":"2024-04-02T04:55:04.063121+0000","last_became_peered":"2024-04-02T04:55:04.063121+0000","last_unstale":"2024-04-02T04:55:12.097947+0000","last_undegraded":"2024-04-02T04:55:12.097947+0000","last_fullsized":"2024-04-02T04:55:12.097947+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:14.988027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"294.e","version":"460'3","reported_seq":30,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110361+0000","last_change":"2024-04-02T04:55:04.063034+0000","last_active":"2024-04-02T04:55:12.110361+0000","last_peered":"2024-04-02T04:55:12.110361+0000","last_clean":"2024-04-02T04:55:12.110361+0000","last_became_active":"2024-04-02T04:55:04.062772+0000","last_became_peered":"2024-04-02T04:55:04.062772+0000","last_unstale":"2024-04-02T04:55:12.110361+0000","last_undegraded":"2024-04-02T04:55:12.110361+0000","last_fullsized":"2024-04-02T04:55:12.110361+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:29.623822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"294.d","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448020+0000","last_change":"2024-04-02T04:55:04.057800+0000","last_active":"2024-04-02T04:55:12.448020+0000","last_peered":"2024-04-02T04:55:12.448020+0000","last_clean":"2024-04-02T04:55:12.448020+0000","last_became_active":"2024-04-02T04:55:04.057406+0000","last_became_peered":"2024-04-02T04:55:04.057406+0000","last_unstale":"2024-04-02T04:55:12.448020+0000","last_undegraded":"2024-04-02T04:55:12.448020+0000","last_fullsized":"2024-04-02T04:55:12.448020+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:32.170517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.c","version":"457'2","reported_seq":30,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.090314+0000","last_change":"2024-04-02T04:55:04.059987+0000","last_active":"2024-04-02T04:55:12.090314+0000","last_peered":"2024-04-02T04:55:12.090314+0000","last_clean":"2024-04-02T04:55:12.090314+0000","last_became_active":"2024-04-02T04:55:04.059580+0000","last_became_peered":"2024-04-02T04:55:04.059580+0000","last_unstale":"2024-04-02T04:55:12.090314+0000","last_undegraded":"2024-04-02T04:55:12.090314+0000","last_fullsized":"2024-04-02T04:55:12.090314+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:22.134218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"294.b","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101528+0000","last_change":"2024-04-02T04:55:04.057828+0000","last_active":"2024-04-02T04:55:12.101528+0000","last_peered":"2024-04-02T04:55:12.101528+0000","last_clean":"2024-04-02T04:55:12.101528+0000","last_became_active":"2024-04-02T04:55:04.056342+0000","last_became_peered":"2024-04-02T04:55:04.056342+0000","last_unstale":"2024-04-02T04:55:12.101528+0000","last_undegraded":"2024-04-02T04:55:12.101528+0000","last_fullsized":"2024-04-02T04:55:12.101528+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:17.492543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"294.a","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.444613+0000","last_change":"2024-04-02T04:55:04.076125+0000","last_active":"2024-04-02T04:55:12.444613+0000","last_peered":"2024-04-02T04:55:12.444613+0000","last_clean":"2024-04-02T04:55:12.444613+0000","last_became_active":"2024-04-02T04:55:04.075780+0000","last_became_peered":"2024-04-02T04:55:04.075780+0000","last_unstale":"2024-04-02T04:55:12.444613+0000","last_undegraded":"2024-04-02T04:55:12.444613+0000","last_fullsized":"2024-04-02T04:55:12.444613+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:20.407394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.9","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.098636+0000","last_change":"2024-04-02T04:55:04.069845+0000","last_active":"2024-04-02T04:55:12.098636+0000","last_peered":"2024-04-02T04:55:12.098636+0000","last_clean":"2024-04-02T04:55:12.098636+0000","last_became_active":"2024-04-02T04:55:04.068958+0000","last_became_peered":"2024-04-02T04:55:04.068958+0000","last_unstale":"2024-04-02T04:55:12.098636+0000","last_undegraded":"2024-04-02T04:55:12.098636+0000","last_fullsized":"2024-04-02T04:55:12.098636+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:29.558104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"294.8","version":"0'0","reported_seq":21,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.930666+0000","last_change":"2024-04-02T04:55:04.071525+0000","last_active":"2024-04-02T04:55:08.930666+0000","last_peered":"2024-04-02T04:55:08.930666+0000","last_clean":"2024-04-02T04:55:08.930666+0000","last_became_active":"2024-04-02T04:55:04.070288+0000","last_became_peered":"2024-04-02T04:55:04.070288+0000","last_unstale":"2024-04-02T04:55:08.930666+0000","last_undegraded":"2024-04-02T04:55:08.930666+0000","last_fullsized":"2024-04-02T04:55:08.930666+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:30.921514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"294.7","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119516+0000","last_change":"2024-04-02T04:55:04.070565+0000","last_active":"2024-04-02T04:55:12.119516+0000","last_peered":"2024-04-02T04:55:12.119516+0000","last_clean":"2024-04-02T04:55:12.119516+0000","last_became_active":"2024-04-02T04:55:04.067641+0000","last_became_peered":"2024-04-02T04:55:04.067641+0000","last_unstale":"2024-04-02T04:55:12.119516+0000","last_undegraded":"2024-04-02T04:55:12.119516+0000","last_fullsized":"2024-04-02T04:55:12.119516+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:14.859435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"294.6","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110369+0000","last_change":"2024-04-02T04:55:04.061026+0000","last_active":"2024-04-02T04:55:12.110369+0000","last_peered":"2024-04-02T04:55:12.110369+0000","last_clean":"2024-04-02T04:55:12.110369+0000","last_became_active":"2024-04-02T04:55:04.058198+0000","last_became_peered":"2024-04-02T04:55:04.058198+0000","last_unstale":"2024-04-02T04:55:12.110369+0000","last_undegraded":"2024-04-02T04:55:12.110369+0000","last_fullsized":"2024-04-02T04:55:12.110369+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:05.018919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"294.5","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.094649+0000","last_change":"2024-04-02T04:55:04.059133+0000","last_active":"2024-04-02T04:55:12.094649+0000","last_peered":"2024-04-02T04:55:12.094649+0000","last_clean":"2024-04-02T04:55:12.094649+0000","last_became_active":"2024-04-02T04:55:04.057756+0000","last_became_peered":"2024-04-02T04:55:04.057756+0000","last_unstale":"2024-04-02T04:55:12.094649+0000","last_undegraded":"2024-04-02T04:55:12.094649+0000","last_fullsized":"2024-04-02T04:55:12.094649+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:07.335119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"294.4","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080879+0000","last_change":"2024-04-02T04:55:04.068795+0000","last_active":"2024-04-02T04:55:12.080879+0000","last_peered":"2024-04-02T04:55:12.080879+0000","last_clean":"2024-04-02T04:55:12.080879+0000","last_became_active":"2024-04-02T04:55:04.068084+0000","last_became_peered":"2024-04-02T04:55:04.068084+0000","last_unstale":"2024-04-02T04:55:12.080879+0000","last_undegraded":"2024-04-02T04:55:12.080879+0000","last_fullsized":"2024-04-02T04:55:12.080879+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:19.812597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"294.3","version":"0'0","reported_seq":21,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.937388+0000","last_change":"2024-04-02T04:55:04.074596+0000","last_active":"2024-04-02T04:55:08.937388+0000","last_peered":"2024-04-02T04:55:08.937388+0000","last_clean":"2024-04-02T04:55:08.937388+0000","last_became_active":"2024-04-02T04:55:04.073777+0000","last_became_peered":"2024-04-02T04:55:04.073777+0000","last_unstale":"2024-04-02T04:55:08.937388+0000","last_undegraded":"2024-04-02T04:55:08.937388+0000","last_fullsized":"2024-04-02T04:55:08.937388+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:04.331553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"294.2","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.116622+0000","last_change":"2024-04-02T04:55:04.069494+0000","last_active":"2024-04-02T04:55:12.116622+0000","last_peered":"2024-04-02T04:55:12.116622+0000","last_clean":"2024-04-02T04:55:12.116622+0000","last_became_active":"2024-04-02T04:55:04.067679+0000","last_became_peered":"2024-04-02T04:55:04.067679+0000","last_unstale":"2024-04-02T04:55:12.116622+0000","last_undegraded":"2024-04-02T04:55:12.116622+0000","last_fullsized":"2024-04-02T04:55:12.116622+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:25.647879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"294.1","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445315+0000","last_change":"2024-04-02T04:55:04.054533+0000","last_active":"2024-04-02T04:55:12.445315+0000","last_peered":"2024-04-02T04:55:12.445315+0000","last_clean":"2024-04-02T04:55:12.445315+0000","last_became_active":"2024-04-02T04:55:04.053285+0000","last_became_peered":"2024-04-02T04:55:04.053285+0000","last_unstale":"2024-04-02T04:55:12.445315+0000","last_undegraded":"2024-04-02T04:55:12.445315+0000","last_fullsized":"2024-04-02T04:55:12.445315+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:08.115602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"294.0","version":"0'0","reported_seq":27,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.087265+0000","last_change":"2024-04-02T04:55:04.056172+0000","last_active":"2024-04-02T04:55:12.087265+0000","last_peered":"2024-04-02T04:55:12.087265+0000","last_clean":"2024-04-02T04:55:12.087265+0000","last_became_active":"2024-04-02T04:55:04.055056+0000","last_became_peered":"2024-04-02T04:55:04.055056+0000","last_unstale":"2024-04-02T04:55:12.087265+0000","last_undegraded":"2024-04-02T04:55:12.087265+0000","last_fullsized":"2024-04-02T04:55:12.087265+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:03.012080+0000","last_clean_scrub_stamp":"2024-04-02T04:55:03.012080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:12.582380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"102.19","version":"423'86","reported_seq":293,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932718+0000","last_change":"2024-04-02T04:55:08.936154+0000","last_active":"2024-04-02T04:55:08.932718+0000","last_peered":"2024-04-02T04:55:08.932718+0000","last_clean":"2024-04-02T04:55:08.932718+0000","last_became_active":"2024-04-02T04:53:54.919540+0000","last_became_peered":"2024-04-02T04:53:54.919540+0000","last_unstale":"2024-04-02T04:55:08.932718+0000","last_undegraded":"2024-04-02T04:55:08.932718+0000","last_fullsized":"2024-04-02T04:55:08.932718+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046958260000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.18","version":"263'35","reported_seq":857,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103562+0000","last_change":"2024-04-02T04:55:08.951836+0000","last_active":"2024-04-02T04:55:12.103562+0000","last_peered":"2024-04-02T04:55:12.103562+0000","last_clean":"2024-04-02T04:55:12.103562+0000","last_became_active":"2024-04-02T04:54:55.950385+0000","last_became_peered":"2024-04-02T04:54:55.950385+0000","last_unstale":"2024-04-02T04:55:12.103562+0000","last_undegraded":"2024-04-02T04:55:12.103562+0000","last_fullsized":"2024-04-02T04:55:12.103562+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'35","last_scrub_stamp":"2024-04-02T04:54:29.951327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.951327+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:47.948317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015598321,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1b","version":"263'37","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117536+0000","last_change":"2024-04-02T04:55:08.928941+0000","last_active":"2024-04-02T04:55:12.117536+0000","last_peered":"2024-04-02T04:55:12.117536+0000","last_clean":"2024-04-02T04:55:12.117536+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:55:12.117536+0000","last_undegraded":"2024-04-02T04:55:12.117536+0000","last_fullsized":"2024-04-02T04:55:12.117536+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'37","last_scrub_stamp":"2024-04-02T04:52:48.221399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.221399+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0056034170000000003,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1a","version":"449'76","reported_seq":856,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.093381+0000","last_change":"2024-04-02T04:55:08.951858+0000","last_active":"2024-04-02T04:55:12.093381+0000","last_peered":"2024-04-02T04:55:12.093381+0000","last_clean":"2024-04-02T04:55:12.093381+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:55:12.093381+0000","last_undegraded":"2024-04-02T04:55:12.093381+0000","last_fullsized":"2024-04-02T04:55:12.093381+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'73","last_scrub_stamp":"2024-04-02T04:54:17.944946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.944946+0000","objects_scrubbed":5,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:45.088807+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.015555253,"stat_sum":{"num_bytes":62,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1d","version":"248'85","reported_seq":873,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101005+0000","last_change":"2024-04-02T04:55:08.923627+0000","last_active":"2024-04-02T04:55:12.101005+0000","last_peered":"2024-04-02T04:55:12.101005+0000","last_clean":"2024-04-02T04:55:12.101005+0000","last_became_active":"2024-04-02T04:49:56.582887+0000","last_became_peered":"2024-04-02T04:49:56.582887+0000","last_unstale":"2024-04-02T04:55:12.101005+0000","last_undegraded":"2024-04-02T04:55:12.101005+0000","last_fullsized":"2024-04-02T04:55:12.101005+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'85","last_scrub_stamp":"2024-04-02T04:52:51.767218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.767218+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061429470000000002,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1c","version":"393'35","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.090090+0000","last_change":"2024-04-02T04:55:08.952309+0000","last_active":"2024-04-02T04:55:12.090090+0000","last_peered":"2024-04-02T04:55:12.090090+0000","last_clean":"2024-04-02T04:55:12.090090+0000","last_became_active":"2024-04-02T04:49:56.570832+0000","last_became_peered":"2024-04-02T04:49:56.570832+0000","last_unstale":"2024-04-02T04:55:12.090090+0000","last_undegraded":"2024-04-02T04:55:12.090090+0000","last_fullsized":"2024-04-02T04:55:12.090090+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-04-02T04:54:09.841478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:09.841478+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:30.976963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.01239724,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1f","version":"443'55","reported_seq":849,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081902+0000","last_change":"2024-04-02T04:55:11.980101+0000","last_active":"2024-04-02T04:55:12.081902+0000","last_peered":"2024-04-02T04:55:12.081902+0000","last_clean":"2024-04-02T04:55:12.081902+0000","last_became_active":"2024-04-02T04:49:55.249289+0000","last_became_peered":"2024-04-02T04:49:55.249289+0000","last_unstale":"2024-04-02T04:55:12.081902+0000","last_undegraded":"2024-04-02T04:55:12.081902+0000","last_fullsized":"2024-04-02T04:55:12.081902+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'55","last_scrub_stamp":"2024-04-02T04:55:11.979935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:55:11.979935+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:26.514401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001525181,"stat_sum":{"num_bytes":71,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1e","version":"248'17","reported_seq":812,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112220+0000","last_change":"2024-04-02T04:55:08.954288+0000","last_active":"2024-04-02T04:55:12.112220+0000","last_peered":"2024-04-02T04:55:12.112220+0000","last_clean":"2024-04-02T04:55:12.112220+0000","last_became_active":"2024-04-02T04:49:56.563968+0000","last_became_peered":"2024-04-02T04:49:56.563968+0000","last_unstale":"2024-04-02T04:55:12.112220+0000","last_undegraded":"2024-04-02T04:55:12.112220+0000","last_fullsized":"2024-04-02T04:55:12.112220+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:54:34.920018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.920018+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:48.306904+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00055178300000000003,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.11","version":"419'91","reported_seq":939,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.109683+0000","last_change":"2024-04-02T04:55:08.947216+0000","last_active":"2024-04-02T04:55:12.109683+0000","last_peered":"2024-04-02T04:55:12.109683+0000","last_clean":"2024-04-02T04:55:12.109683+0000","last_became_active":"2024-04-02T04:53:54.924990+0000","last_became_peered":"2024-04-02T04:53:54.924990+0000","last_unstale":"2024-04-02T04:55:12.109683+0000","last_undegraded":"2024-04-02T04:55:12.109683+0000","last_fullsized":"2024-04-02T04:55:12.109683+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'91","last_scrub_stamp":"2024-04-02T04:54:22.972425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:22.972425+0000","objects_scrubbed":7,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:31.351523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011142802,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.10","version":"461'52","reported_seq":855,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.089829+0000","last_change":"2024-04-02T04:55:10.938361+0000","last_active":"2024-04-02T04:55:12.089829+0000","last_peered":"2024-04-02T04:55:12.089829+0000","last_clean":"2024-04-02T04:55:12.089829+0000","last_became_active":"2024-04-02T04:53:54.956913+0000","last_became_peered":"2024-04-02T04:53:54.956913+0000","last_unstale":"2024-04-02T04:55:12.089829+0000","last_undegraded":"2024-04-02T04:55:12.089829+0000","last_fullsized":"2024-04-02T04:55:12.089829+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.010868375,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.13","version":"440'94","reported_seq":935,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117604+0000","last_change":"2024-04-02T04:55:08.929281+0000","last_active":"2024-04-02T04:55:12.117604+0000","last_peered":"2024-04-02T04:55:12.117604+0000","last_clean":"2024-04-02T04:55:12.117604+0000","last_became_active":"2024-04-02T04:53:54.943248+0000","last_became_peered":"2024-04-02T04:53:54.943248+0000","last_unstale":"2024-04-02T04:55:12.117604+0000","last_undegraded":"2024-04-02T04:55:12.117604+0000","last_fullsized":"2024-04-02T04:55:12.117604+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'92","last_scrub_stamp":"2024-04-02T04:54:34.165272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.165272+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:39.276785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0047766880000000003,"stat_sum":{"num_bytes":63,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.12","version":"449'21","reported_seq":813,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099602+0000","last_change":"2024-04-02T04:55:08.946218+0000","last_active":"2024-04-02T04:55:12.099602+0000","last_peered":"2024-04-02T04:55:12.099602+0000","last_clean":"2024-04-02T04:55:12.099602+0000","last_became_active":"2024-04-02T04:49:55.255700+0000","last_became_peered":"2024-04-02T04:49:55.255700+0000","last_unstale":"2024-04-02T04:55:12.099602+0000","last_undegraded":"2024-04-02T04:55:12.099602+0000","last_fullsized":"2024-04-02T04:55:12.099602+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-04-02T04:51:53.016585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.016585+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117482+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0084629609999999997,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.15","version":"395'69","reported_seq":899,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.121679+0000","last_change":"2024-04-02T04:55:10.262879+0000","last_active":"2024-04-02T04:55:12.121679+0000","last_peered":"2024-04-02T04:55:12.121679+0000","last_clean":"2024-04-02T04:55:12.121679+0000","last_became_active":"2024-04-02T04:53:54.955612+0000","last_became_peered":"2024-04-02T04:53:54.955612+0000","last_unstale":"2024-04-02T04:55:12.121679+0000","last_undegraded":"2024-04-02T04:55:12.121679+0000","last_fullsized":"2024-04-02T04:55:12.121679+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'69","last_scrub_stamp":"2024-04-02T04:55:10.262637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:55:10.262637+0000","objects_scrubbed":5,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:18.434407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000814582,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.14","version":"350'154","reported_seq":822,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447635+0000","last_change":"2024-04-02T04:55:09.081674+0000","last_active":"2024-04-02T04:55:12.447635+0000","last_peered":"2024-04-02T04:55:12.447635+0000","last_clean":"2024-04-02T04:55:12.447635+0000","last_became_active":"2024-04-02T04:54:58.982282+0000","last_became_peered":"2024-04-02T04:54:58.982282+0000","last_unstale":"2024-04-02T04:55:12.447635+0000","last_undegraded":"2024-04-02T04:55:12.447635+0000","last_fullsized":"2024-04-02T04:55:12.447635+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'153","last_scrub_stamp":"2024-04-02T04:52:34.712599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.712599+0000","objects_scrubbed":9,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.901849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0075892900000000003,"stat_sum":{"num_bytes":4000107,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":46,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4000107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.17","version":"393'36","reported_seq":833,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097545+0000","last_change":"2024-04-02T04:55:08.952647+0000","last_active":"2024-04-02T04:55:12.097545+0000","last_peered":"2024-04-02T04:55:12.097545+0000","last_clean":"2024-04-02T04:55:12.097545+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:55:12.097545+0000","last_undegraded":"2024-04-02T04:55:12.097545+0000","last_fullsized":"2024-04-02T04:55:12.097545+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'36","last_scrub_stamp":"2024-04-02T04:54:24.021811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.021811+0000","objects_scrubbed":4,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:53.005307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0056061039999999998,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.16","version":"393'18","reported_seq":712,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449066+0000","last_change":"2024-04-02T04:55:09.082681+0000","last_active":"2024-04-02T04:55:12.449066+0000","last_peered":"2024-04-02T04:55:12.449066+0000","last_clean":"2024-04-02T04:55:12.449066+0000","last_became_active":"2024-04-02T04:54:56.013487+0000","last_became_peered":"2024-04-02T04:54:56.013487+0000","last_unstale":"2024-04-02T04:55:12.449066+0000","last_undegraded":"2024-04-02T04:55:12.449066+0000","last_fullsized":"2024-04-02T04:55:12.449066+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'17","last_scrub_stamp":"2024-04-02T04:52:38.664090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.664090+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.901798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021057670000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.9","version":"401'37","reported_seq":853,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.108748+0000","last_change":"2024-04-02T04:55:08.946502+0000","last_active":"2024-04-02T04:55:12.108748+0000","last_peered":"2024-04-02T04:55:12.108748+0000","last_clean":"2024-04-02T04:55:12.108748+0000","last_became_active":"2024-04-02T04:53:54.924508+0000","last_became_peered":"2024-04-02T04:53:54.924508+0000","last_unstale":"2024-04-02T04:55:12.108748+0000","last_undegraded":"2024-04-02T04:55:12.108748+0000","last_fullsized":"2024-04-02T04:55:12.108748+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'37","last_scrub_stamp":"2024-04-02T04:54:07.832705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.832705+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.797201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.010537447,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.8","version":"248'17","reported_seq":808,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445557+0000","last_change":"2024-04-02T04:55:09.076547+0000","last_active":"2024-04-02T04:55:12.445557+0000","last_peered":"2024-04-02T04:55:12.445557+0000","last_clean":"2024-04-02T04:55:12.445557+0000","last_became_active":"2024-04-02T04:49:55.259553+0000","last_became_peered":"2024-04-02T04:49:55.259553+0000","last_unstale":"2024-04-02T04:55:12.445557+0000","last_undegraded":"2024-04-02T04:55:12.445557+0000","last_fullsized":"2024-04-02T04:55:12.445557+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:54:11.954373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:11.954373+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.115924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0037285619999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.b","version":"393'75","reported_seq":792,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103552+0000","last_change":"2024-04-02T04:55:08.949865+0000","last_active":"2024-04-02T04:55:12.103552+0000","last_peered":"2024-04-02T04:55:12.103552+0000","last_clean":"2024-04-02T04:55:12.103552+0000","last_became_active":"2024-04-02T04:54:55.989598+0000","last_became_peered":"2024-04-02T04:54:55.989598+0000","last_unstale":"2024-04-02T04:55:12.103552+0000","last_undegraded":"2024-04-02T04:55:12.103552+0000","last_fullsized":"2024-04-02T04:55:12.103552+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'73","last_scrub_stamp":"2024-04-02T04:52:33.669178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.669178+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0021764789999999998,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.a","version":"254'68","reported_seq":405,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099275+0000","last_change":"2024-04-02T04:55:08.945734+0000","last_active":"2024-04-02T04:55:12.099275+0000","last_peered":"2024-04-02T04:55:12.099275+0000","last_clean":"2024-04-02T04:55:12.099275+0000","last_became_active":"2024-04-02T04:54:55.961329+0000","last_became_peered":"2024-04-02T04:54:55.961329+0000","last_unstale":"2024-04-02T04:55:12.099275+0000","last_undegraded":"2024-04-02T04:55:12.099275+0000","last_fullsized":"2024-04-02T04:55:12.099275+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'68","last_scrub_stamp":"2024-04-02T04:52:43.653347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.653347+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010361412,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.d","version":"434'130","reported_seq":336,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936974+0000","last_change":"2024-04-02T04:55:08.940751+0000","last_active":"2024-04-02T04:55:08.936974+0000","last_peered":"2024-04-02T04:55:08.936974+0000","last_clean":"2024-04-02T04:55:08.936974+0000","last_became_active":"2024-04-02T04:53:54.929671+0000","last_became_peered":"2024-04-02T04:53:54.929671+0000","last_unstale":"2024-04-02T04:55:08.936974+0000","last_undegraded":"2024-04-02T04:55:08.936974+0000","last_fullsized":"2024-04-02T04:55:08.936974+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050601379999999996,"stat_sum":{"num_bytes":95,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.c","version":"249'68","reported_seq":847,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448284+0000","last_change":"2024-04-02T04:55:09.081849+0000","last_active":"2024-04-02T04:55:12.448284+0000","last_peered":"2024-04-02T04:55:12.448284+0000","last_clean":"2024-04-02T04:55:12.448284+0000","last_became_active":"2024-04-02T04:49:55.252273+0000","last_became_peered":"2024-04-02T04:49:55.252273+0000","last_unstale":"2024-04-02T04:55:12.448284+0000","last_undegraded":"2024-04-02T04:55:12.448284+0000","last_fullsized":"2024-04-02T04:55:12.448284+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'68","last_scrub_stamp":"2024-04-02T04:53:54.821023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.821023+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.043213+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0037236679999999999,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.f","version":"393'63","reported_seq":903,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449471+0000","last_change":"2024-04-02T04:55:09.083552+0000","last_active":"2024-04-02T04:55:12.449471+0000","last_peered":"2024-04-02T04:55:12.449471+0000","last_clean":"2024-04-02T04:55:12.449471+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:55:12.449471+0000","last_undegraded":"2024-04-02T04:55:12.449471+0000","last_fullsized":"2024-04-02T04:55:12.449471+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'60","last_scrub_stamp":"2024-04-02T04:53:14.447906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:53:14.447906+0000","objects_scrubbed":8,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.901898+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0024518999999999999,"stat_sum":{"num_bytes":4194378,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":8214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.e","version":"425'57","reported_seq":852,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119865+0000","last_change":"2024-04-02T04:55:08.931052+0000","last_active":"2024-04-02T04:55:12.119865+0000","last_peered":"2024-04-02T04:55:12.119865+0000","last_clean":"2024-04-02T04:55:12.119865+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:55:12.119865+0000","last_undegraded":"2024-04-02T04:55:12.119865+0000","last_fullsized":"2024-04-02T04:55:12.119865+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'56","last_scrub_stamp":"2024-04-02T04:51:26.235660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.235660+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061928419999999996,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.1","version":"453'60","reported_seq":271,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.931784+0000","last_change":"2024-04-02T04:55:08.935987+0000","last_active":"2024-04-02T04:55:08.931784+0000","last_peered":"2024-04-02T04:55:08.931784+0000","last_clean":"2024-04-02T04:55:08.931784+0000","last_became_active":"2024-04-02T04:53:54.923616+0000","last_became_peered":"2024-04-02T04:53:54.923616+0000","last_unstale":"2024-04-02T04:55:08.931784+0000","last_undegraded":"2024-04-02T04:55:08.931784+0000","last_fullsized":"2024-04-02T04:55:08.931784+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046179359999999996,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.0","version":"446'101","reported_seq":945,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.444456+0000","last_change":"2024-04-02T04:55:09.073903+0000","last_active":"2024-04-02T04:55:12.444456+0000","last_peered":"2024-04-02T04:55:12.444456+0000","last_clean":"2024-04-02T04:55:12.444456+0000","last_became_active":"2024-04-02T04:54:58.978036+0000","last_became_peered":"2024-04-02T04:54:58.978036+0000","last_unstale":"2024-04-02T04:55:12.444456+0000","last_undegraded":"2024-04-02T04:55:12.444456+0000","last_fullsized":"2024-04-02T04:55:12.444456+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'95","last_scrub_stamp":"2024-04-02T04:54:12.940358+0000","last_deep_scrub":"412'95","last_deep_scrub_stamp":"2024-04-02T04:54:12.940358+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.940358+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:41.353352+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0062680599999999998,"stat_sum":{"num_bytes":70,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.3","version":"393'35","reported_seq":850,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101883+0000","last_change":"2024-04-02T04:55:08.949468+0000","last_active":"2024-04-02T04:55:12.101883+0000","last_peered":"2024-04-02T04:55:12.101883+0000","last_clean":"2024-04-02T04:55:12.101883+0000","last_became_active":"2024-04-02T04:54:55.988749+0000","last_became_peered":"2024-04-02T04:54:55.988749+0000","last_unstale":"2024-04-02T04:55:12.101883+0000","last_undegraded":"2024-04-02T04:55:12.101883+0000","last_fullsized":"2024-04-02T04:55:12.101883+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-04-02T04:51:50.994396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:50.994396+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117436+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.011251228,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.2","version":"453'41","reported_seq":827,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099940+0000","last_change":"2024-04-02T04:55:08.948863+0000","last_active":"2024-04-02T04:55:12.099940+0000","last_peered":"2024-04-02T04:55:12.099940+0000","last_clean":"2024-04-02T04:55:12.099940+0000","last_became_active":"2024-04-02T04:49:55.235417+0000","last_became_peered":"2024-04-02T04:49:55.235417+0000","last_unstale":"2024-04-02T04:55:12.099940+0000","last_undegraded":"2024-04-02T04:55:12.099940+0000","last_fullsized":"2024-04-02T04:55:12.099940+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'39","last_scrub_stamp":"2024-04-02T04:52:41.329212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.329212+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010833436,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.5","version":"229'85","reported_seq":925,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119293+0000","last_change":"2024-04-02T04:55:08.933296+0000","last_active":"2024-04-02T04:55:12.119293+0000","last_peered":"2024-04-02T04:55:12.119293+0000","last_clean":"2024-04-02T04:55:12.119293+0000","last_became_active":"2024-04-02T04:54:55.991571+0000","last_became_peered":"2024-04-02T04:54:55.991571+0000","last_unstale":"2024-04-02T04:55:12.119293+0000","last_undegraded":"2024-04-02T04:55:12.119293+0000","last_fullsized":"2024-04-02T04:55:12.119293+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'85","last_scrub_stamp":"2024-04-02T04:51:27.257882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.257882+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0036629140000000002,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.4","version":"446'20","reported_seq":848,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120872+0000","last_change":"2024-04-02T04:55:08.936918+0000","last_active":"2024-04-02T04:55:12.120872+0000","last_peered":"2024-04-02T04:55:12.120872+0000","last_clean":"2024-04-02T04:55:12.120872+0000","last_became_active":"2024-04-02T04:54:55.979745+0000","last_became_peered":"2024-04-02T04:54:55.979745+0000","last_unstale":"2024-04-02T04:55:12.120872+0000","last_undegraded":"2024-04-02T04:55:12.120872+0000","last_fullsized":"2024-04-02T04:55:12.120872+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'18","last_scrub_stamp":"2024-04-02T04:54:32.190177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:32.190177+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:56.424625+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.002952873,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":71,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.7","version":"393'19","reported_seq":808,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.115353+0000","last_change":"2024-04-02T04:55:08.950328+0000","last_active":"2024-04-02T04:55:12.115353+0000","last_peered":"2024-04-02T04:55:12.115353+0000","last_clean":"2024-04-02T04:55:12.115353+0000","last_became_active":"2024-04-02T04:54:56.004707+0000","last_became_peered":"2024-04-02T04:54:56.004707+0000","last_unstale":"2024-04-02T04:55:12.115353+0000","last_undegraded":"2024-04-02T04:55:12.115353+0000","last_fullsized":"2024-04-02T04:55:12.115353+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'18","last_scrub_stamp":"2024-04-02T04:52:40.288509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.288509+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.117654+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0012839570000000001,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":47,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"102.6","version":"425'35","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099466+0000","last_change":"2024-04-02T04:55:08.923080+0000","last_active":"2024-04-02T04:55:12.099466+0000","last_peered":"2024-04-02T04:55:12.099466+0000","last_clean":"2024-04-02T04:55:12.099466+0000","last_became_active":"2024-04-02T04:49:56.581977+0000","last_became_peered":"2024-04-02T04:49:56.581977+0000","last_unstale":"2024-04-02T04:55:12.099466+0000","last_undegraded":"2024-04-02T04:55:12.099466+0000","last_fullsized":"2024-04-02T04:55:12.099466+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-02T04:52:53.769683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.769683+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060384339999999996,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"43.14","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.096663+0000","last_change":"2024-04-02T04:54:55.986021+0000","last_active":"2024-04-02T04:55:12.096663+0000","last_peered":"2024-04-02T04:55:12.096663+0000","last_clean":"2024-04-02T04:55:12.096663+0000","last_became_active":"2024-04-02T04:54:55.985551+0000","last_became_peered":"2024-04-02T04:54:55.985551+0000","last_unstale":"2024-04-02T04:55:12.096663+0000","last_undegraded":"2024-04-02T04:55:12.096663+0000","last_fullsized":"2024-04-02T04:55:12.096663+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.775672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.775672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":677,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081610+0000","last_change":"2024-04-02T04:54:08.664647+0000","last_active":"2024-04-02T04:55:12.081610+0000","last_peered":"2024-04-02T04:55:12.081610+0000","last_clean":"2024-04-02T04:55:12.081610+0000","last_became_active":"2024-04-02T04:53:54.911630+0000","last_became_peered":"2024-04-02T04:53:54.911630+0000","last_unstale":"2024-04-02T04:55:12.081610+0000","last_undegraded":"2024-04-02T04:55:12.081610+0000","last_fullsized":"2024-04-02T04:55:12.081610+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.664547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.664547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:20.890228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082067+0000","last_change":"2024-04-02T04:52:58.345528+0000","last_active":"2024-04-02T04:55:12.082067+0000","last_peered":"2024-04-02T04:55:12.082067+0000","last_clean":"2024-04-02T04:55:12.082067+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:55:12.082067+0000","last_undegraded":"2024-04-02T04:55:12.082067+0000","last_fullsized":"2024-04-02T04:55:12.082067+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.345234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.345234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081039+0000","last_change":"2024-04-02T04:51:46.284187+0000","last_active":"2024-04-02T04:55:12.081039+0000","last_peered":"2024-04-02T04:55:12.081039+0000","last_clean":"2024-04-02T04:55:12.081039+0000","last_became_active":"2024-04-02T04:49:28.787404+0000","last_became_peered":"2024-04-02T04:49:28.787404+0000","last_unstale":"2024-04-02T04:55:12.081039+0000","last_undegraded":"2024-04-02T04:55:12.081039+0000","last_fullsized":"2024-04-02T04:55:12.081039+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:46.284019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:46.284019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082060+0000","last_change":"2024-04-02T04:53:07.389165+0000","last_active":"2024-04-02T04:55:12.082060+0000","last_peered":"2024-04-02T04:55:12.082060+0000","last_clean":"2024-04-02T04:55:12.082060+0000","last_became_active":"2024-04-02T04:49:28.783594+0000","last_became_peered":"2024-04-02T04:49:28.783594+0000","last_unstale":"2024-04-02T04:55:12.082060+0000","last_undegraded":"2024-04-02T04:55:12.082060+0000","last_fullsized":"2024-04-02T04:55:12.082060+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.388952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.388952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":666,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112220+0000","last_change":"2024-04-02T04:51:44.929741+0000","last_active":"2024-04-02T04:55:12.112220+0000","last_peered":"2024-04-02T04:55:12.112220+0000","last_clean":"2024-04-02T04:55:12.112220+0000","last_became_active":"2024-04-02T04:49:28.726013+0000","last_became_peered":"2024-04-02T04:49:28.726013+0000","last_unstale":"2024-04-02T04:55:12.112220+0000","last_undegraded":"2024-04-02T04:55:12.112220+0000","last_fullsized":"2024-04-02T04:55:12.112220+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.929545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.929545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099845+0000","last_change":"2024-04-02T04:54:23.115504+0000","last_active":"2024-04-02T04:55:12.099845+0000","last_peered":"2024-04-02T04:55:12.099845+0000","last_clean":"2024-04-02T04:55:12.099845+0000","last_became_active":"2024-04-02T04:53:54.920768+0000","last_became_peered":"2024-04-02T04:53:54.920768+0000","last_unstale":"2024-04-02T04:55:12.099845+0000","last_undegraded":"2024-04-02T04:55:12.099845+0000","last_fullsized":"2024-04-02T04:55:12.099845+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:23.115295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:23.115295+0000","last_clean_scrub_stamp":"2024-04-02T04:54:23.115295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.535270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.090516+0000","last_change":"2024-04-02T04:54:55.978445+0000","last_active":"2024-04-02T04:55:12.090516+0000","last_peered":"2024-04-02T04:55:12.090516+0000","last_clean":"2024-04-02T04:55:12.090516+0000","last_became_active":"2024-04-02T04:54:55.978157+0000","last_became_peered":"2024-04-02T04:54:55.978157+0000","last_unstale":"2024-04-02T04:55:12.090516+0000","last_undegraded":"2024-04-02T04:55:12.090516+0000","last_fullsized":"2024-04-02T04:55:12.090516+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.780242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":667,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110084+0000","last_change":"2024-04-02T04:53:54.944915+0000","last_active":"2024-04-02T04:55:12.110084+0000","last_peered":"2024-04-02T04:55:12.110084+0000","last_clean":"2024-04-02T04:55:12.110084+0000","last_became_active":"2024-04-02T04:53:54.940561+0000","last_became_peered":"2024-04-02T04:53:54.940561+0000","last_unstale":"2024-04-02T04:55:12.110084+0000","last_undegraded":"2024-04-02T04:55:12.110084+0000","last_fullsized":"2024-04-02T04:55:12.110084+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":155,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.940586+0000","last_change":"2024-04-02T04:53:54.912392+0000","last_active":"2024-04-02T04:55:08.940586+0000","last_peered":"2024-04-02T04:55:08.940586+0000","last_clean":"2024-04-02T04:55:08.940586+0000","last_became_active":"2024-04-02T04:53:54.910223+0000","last_became_peered":"2024-04-02T04:53:54.910223+0000","last_unstale":"2024-04-02T04:55:08.940586+0000","last_undegraded":"2024-04-02T04:55:08.940586+0000","last_fullsized":"2024-04-02T04:55:08.940586+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":667,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.109825+0000","last_change":"2024-04-02T04:53:54.941362+0000","last_active":"2024-04-02T04:55:12.109825+0000","last_peered":"2024-04-02T04:55:12.109825+0000","last_clean":"2024-04-02T04:55:12.109825+0000","last_became_active":"2024-04-02T04:53:54.940053+0000","last_became_peered":"2024-04-02T04:53:54.940053+0000","last_unstale":"2024-04-02T04:55:12.109825+0000","last_undegraded":"2024-04-02T04:55:12.109825+0000","last_fullsized":"2024-04-02T04:55:12.109825+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448228+0000","last_change":"2024-04-02T04:53:07.468104+0000","last_active":"2024-04-02T04:55:12.448228+0000","last_peered":"2024-04-02T04:55:12.448228+0000","last_clean":"2024-04-02T04:55:12.448228+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:55:12.448228+0000","last_undegraded":"2024-04-02T04:55:12.448228+0000","last_fullsized":"2024-04-02T04:55:12.448228+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.467848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.467848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.095405+0000","last_change":"2024-04-02T04:52:40.787636+0000","last_active":"2024-04-02T04:55:12.095405+0000","last_peered":"2024-04-02T04:55:12.095405+0000","last_clean":"2024-04-02T04:55:12.095405+0000","last_became_active":"2024-04-02T04:49:29.818524+0000","last_became_peered":"2024-04-02T04:49:29.818524+0000","last_unstale":"2024-04-02T04:55:12.095405+0000","last_undegraded":"2024-04-02T04:55:12.095405+0000","last_fullsized":"2024-04-02T04:55:12.095405+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.787478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.787478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":670,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445317+0000","last_change":"2024-04-02T04:54:55.984155+0000","last_active":"2024-04-02T04:55:12.445317+0000","last_peered":"2024-04-02T04:55:12.445317+0000","last_clean":"2024-04-02T04:55:12.445317+0000","last_became_active":"2024-04-02T04:54:55.979293+0000","last_became_peered":"2024-04-02T04:54:55.979293+0000","last_unstale":"2024-04-02T04:55:12.445317+0000","last_undegraded":"2024-04-02T04:55:12.445317+0000","last_fullsized":"2024-04-02T04:55:12.445317+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:51.039450+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.039450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097839+0000","last_change":"2024-04-02T04:54:55.971353+0000","last_active":"2024-04-02T04:55:12.097839+0000","last_peered":"2024-04-02T04:55:12.097839+0000","last_clean":"2024-04-02T04:55:12.097839+0000","last_became_active":"2024-04-02T04:54:55.971015+0000","last_became_peered":"2024-04-02T04:54:55.971015+0000","last_unstale":"2024-04-02T04:55:12.097839+0000","last_undegraded":"2024-04-02T04:55:12.097839+0000","last_fullsized":"2024-04-02T04:55:12.097839+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:35.787733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:35.787733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445368+0000","last_change":"2024-04-02T04:54:56.002495+0000","last_active":"2024-04-02T04:55:12.445368+0000","last_peered":"2024-04-02T04:55:12.445368+0000","last_clean":"2024-04-02T04:55:12.445368+0000","last_became_active":"2024-04-02T04:54:56.002212+0000","last_became_peered":"2024-04-02T04:54:56.002212+0000","last_unstale":"2024-04-02T04:55:12.445368+0000","last_undegraded":"2024-04-02T04:55:12.445368+0000","last_fullsized":"2024-04-02T04:55:12.445368+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.684685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.684685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080608+0000","last_change":"2024-04-02T04:54:56.005466+0000","last_active":"2024-04-02T04:55:12.080608+0000","last_peered":"2024-04-02T04:55:12.080608+0000","last_clean":"2024-04-02T04:55:12.080608+0000","last_became_active":"2024-04-02T04:54:56.001215+0000","last_became_peered":"2024-04-02T04:54:56.001215+0000","last_unstale":"2024-04-02T04:55:12.080608+0000","last_undegraded":"2024-04-02T04:55:12.080608+0000","last_fullsized":"2024-04-02T04:55:12.080608+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.625010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.625010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117685+0000","last_change":"2024-04-02T04:54:55.969224+0000","last_active":"2024-04-02T04:55:12.117685+0000","last_peered":"2024-04-02T04:55:12.117685+0000","last_clean":"2024-04-02T04:55:12.117685+0000","last_became_active":"2024-04-02T04:54:55.968826+0000","last_became_peered":"2024-04-02T04:54:55.968826+0000","last_unstale":"2024-04-02T04:55:12.117685+0000","last_undegraded":"2024-04-02T04:55:12.117685+0000","last_fullsized":"2024-04-02T04:55:12.117685+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.326885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.326885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118335+0000","last_change":"2024-04-02T04:52:36.311512+0000","last_active":"2024-04-02T04:55:12.118335+0000","last_peered":"2024-04-02T04:55:12.118335+0000","last_clean":"2024-04-02T04:55:12.118335+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:55:12.118335+0000","last_undegraded":"2024-04-02T04:55:12.118335+0000","last_fullsized":"2024-04-02T04:55:12.118335+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.311307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.116230+0000","last_change":"2024-04-02T04:54:55.979984+0000","last_active":"2024-04-02T04:55:12.116230+0000","last_peered":"2024-04-02T04:55:12.116230+0000","last_clean":"2024-04-02T04:55:12.116230+0000","last_became_active":"2024-04-02T04:54:55.979631+0000","last_became_peered":"2024-04-02T04:54:55.979631+0000","last_unstale":"2024-04-02T04:55:12.116230+0000","last_undegraded":"2024-04-02T04:55:12.116230+0000","last_fullsized":"2024-04-02T04:55:12.116230+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:45.221445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:45.221445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":680,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110946+0000","last_change":"2024-04-02T04:54:55.978436+0000","last_active":"2024-04-02T04:55:12.110946+0000","last_peered":"2024-04-02T04:55:12.110946+0000","last_clean":"2024-04-02T04:55:12.110946+0000","last_became_active":"2024-04-02T04:54:55.978095+0000","last_became_peered":"2024-04-02T04:54:55.978095+0000","last_unstale":"2024-04-02T04:55:12.110946+0000","last_undegraded":"2024-04-02T04:55:12.110946+0000","last_fullsized":"2024-04-02T04:55:12.110946+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.665317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.665317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.780420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118602+0000","last_change":"2024-04-02T04:54:11.293681+0000","last_active":"2024-04-02T04:55:12.118602+0000","last_peered":"2024-04-02T04:55:12.118602+0000","last_clean":"2024-04-02T04:55:12.118602+0000","last_became_active":"2024-04-02T04:53:54.942156+0000","last_became_peered":"2024-04-02T04:53:54.942156+0000","last_unstale":"2024-04-02T04:55:12.118602+0000","last_undegraded":"2024-04-02T04:55:12.118602+0000","last_fullsized":"2024-04-02T04:55:12.118602+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:11.293564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:11.293564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":565,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118868+0000","last_change":"2024-04-02T04:54:55.938735+0000","last_active":"2024-04-02T04:55:12.118868+0000","last_peered":"2024-04-02T04:55:12.118868+0000","last_clean":"2024-04-02T04:55:12.118868+0000","last_became_active":"2024-04-02T04:54:55.938419+0000","last_became_peered":"2024-04-02T04:54:55.938419+0000","last_unstale":"2024-04-02T04:55:12.118868+0000","last_undegraded":"2024-04-02T04:55:12.118868+0000","last_fullsized":"2024-04-02T04:55:12.118868+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.642610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.642610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:47.747441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":565,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103890+0000","last_change":"2024-04-02T04:54:55.978517+0000","last_active":"2024-04-02T04:55:12.103890+0000","last_peered":"2024-04-02T04:55:12.103890+0000","last_clean":"2024-04-02T04:55:12.103890+0000","last_became_active":"2024-04-02T04:54:55.978164+0000","last_became_peered":"2024-04-02T04:54:55.978164+0000","last_unstale":"2024-04-02T04:55:12.103890+0000","last_undegraded":"2024-04-02T04:55:12.103890+0000","last_fullsized":"2024-04-02T04:55:12.103890+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:31.677846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:31.677846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:56.355278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448581+0000","last_change":"2024-04-02T04:51:41.090918+0000","last_active":"2024-04-02T04:55:12.448581+0000","last_peered":"2024-04-02T04:55:12.448581+0000","last_clean":"2024-04-02T04:55:12.448581+0000","last_became_active":"2024-04-02T04:49:28.762640+0000","last_became_peered":"2024-04-02T04:49:28.762640+0000","last_unstale":"2024-04-02T04:55:12.448581+0000","last_undegraded":"2024-04-02T04:55:12.448581+0000","last_fullsized":"2024-04-02T04:55:12.448581+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:41.090714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:41.090714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449398+0000","last_change":"2024-04-02T04:52:59.362968+0000","last_active":"2024-04-02T04:55:12.449398+0000","last_peered":"2024-04-02T04:55:12.449398+0000","last_clean":"2024-04-02T04:55:12.449398+0000","last_became_active":"2024-04-02T04:49:28.767868+0000","last_became_peered":"2024-04-02T04:49:28.767868+0000","last_unstale":"2024-04-02T04:55:12.449398+0000","last_undegraded":"2024-04-02T04:55:12.449398+0000","last_fullsized":"2024-04-02T04:55:12.449398+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.362748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.362748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":666,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.115191+0000","last_change":"2024-04-02T04:53:54.955671+0000","last_active":"2024-04-02T04:55:12.115191+0000","last_peered":"2024-04-02T04:55:12.115191+0000","last_clean":"2024-04-02T04:55:12.115191+0000","last_became_active":"2024-04-02T04:53:54.955334+0000","last_became_peered":"2024-04-02T04:53:54.955334+0000","last_unstale":"2024-04-02T04:55:12.115191+0000","last_undegraded":"2024-04-02T04:55:12.115191+0000","last_fullsized":"2024-04-02T04:55:12.115191+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082761+0000","last_change":"2024-04-02T04:53:54.939129+0000","last_active":"2024-04-02T04:55:12.082761+0000","last_peered":"2024-04-02T04:55:12.082761+0000","last_clean":"2024-04-02T04:55:12.082761+0000","last_became_active":"2024-04-02T04:53:54.938792+0000","last_became_peered":"2024-04-02T04:53:54.938792+0000","last_unstale":"2024-04-02T04:55:12.082761+0000","last_undegraded":"2024-04-02T04:55:12.082761+0000","last_fullsized":"2024-04-02T04:55:12.082761+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.918912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120329+0000","last_change":"2024-04-02T04:54:55.988334+0000","last_active":"2024-04-02T04:55:12.120329+0000","last_peered":"2024-04-02T04:55:12.120329+0000","last_clean":"2024-04-02T04:55:12.120329+0000","last_became_active":"2024-04-02T04:54:55.987934+0000","last_became_peered":"2024-04-02T04:54:55.987934+0000","last_unstale":"2024-04-02T04:55:12.120329+0000","last_undegraded":"2024-04-02T04:55:12.120329+0000","last_fullsized":"2024-04-02T04:55:12.120329+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:30.306343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:30.306343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":670,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449472+0000","last_change":"2024-04-02T04:54:55.972702+0000","last_active":"2024-04-02T04:55:12.449472+0000","last_peered":"2024-04-02T04:55:12.449472+0000","last_clean":"2024-04-02T04:55:12.449472+0000","last_became_active":"2024-04-02T04:54:55.972111+0000","last_became_peered":"2024-04-02T04:54:55.972111+0000","last_unstale":"2024-04-02T04:55:12.449472+0000","last_undegraded":"2024-04-02T04:55:12.449472+0000","last_fullsized":"2024-04-02T04:55:12.449472+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.077908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.077908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081499+0000","last_change":"2024-04-02T04:54:13.639181+0000","last_active":"2024-04-02T04:55:12.081499+0000","last_peered":"2024-04-02T04:55:12.081499+0000","last_clean":"2024-04-02T04:55:12.081499+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:55:12.081499+0000","last_undegraded":"2024-04-02T04:55:12.081499+0000","last_fullsized":"2024-04-02T04:55:12.081499+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:13.638970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:13.638970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:30.892037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081037+0000","last_change":"2024-04-02T04:54:56.014261+0000","last_active":"2024-04-02T04:55:12.081037+0000","last_peered":"2024-04-02T04:55:12.081037+0000","last_clean":"2024-04-02T04:55:12.081037+0000","last_became_active":"2024-04-02T04:54:56.011823+0000","last_became_peered":"2024-04-02T04:54:56.011823+0000","last_unstale":"2024-04-02T04:55:12.081037+0000","last_undegraded":"2024-04-02T04:55:12.081037+0000","last_fullsized":"2024-04-02T04:55:12.081037+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.916373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081050+0000","last_change":"2024-04-02T04:54:56.015298+0000","last_active":"2024-04-02T04:55:12.081050+0000","last_peered":"2024-04-02T04:55:12.081050+0000","last_clean":"2024-04-02T04:55:12.081050+0000","last_became_active":"2024-04-02T04:54:56.011565+0000","last_became_peered":"2024-04-02T04:54:56.011565+0000","last_unstale":"2024-04-02T04:55:12.081050+0000","last_undegraded":"2024-04-02T04:55:12.081050+0000","last_fullsized":"2024-04-02T04:55:12.081050+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:51.287037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.287037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":161,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.940176+0000","last_change":"2024-04-02T04:54:55.989618+0000","last_active":"2024-04-02T04:55:08.940176+0000","last_peered":"2024-04-02T04:55:08.940176+0000","last_clean":"2024-04-02T04:55:08.940176+0000","last_became_active":"2024-04-02T04:54:55.989261+0000","last_became_peered":"2024-04-02T04:54:55.989261+0000","last_unstale":"2024-04-02T04:55:08.940176+0000","last_undegraded":"2024-04-02T04:55:08.940176+0000","last_fullsized":"2024-04-02T04:55:08.940176+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":565,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.121020+0000","last_change":"2024-04-02T04:54:55.992290+0000","last_active":"2024-04-02T04:55:12.121020+0000","last_peered":"2024-04-02T04:55:12.121020+0000","last_clean":"2024-04-02T04:55:12.121020+0000","last_became_active":"2024-04-02T04:54:55.991956+0000","last_became_peered":"2024-04-02T04:54:55.991956+0000","last_unstale":"2024-04-02T04:55:12.121020+0000","last_undegraded":"2024-04-02T04:55:12.121020+0000","last_fullsized":"2024-04-02T04:55:12.121020+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:27.663049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:27.663049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:29.817490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":161,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.939334+0000","last_change":"2024-04-02T04:54:55.971706+0000","last_active":"2024-04-02T04:55:08.939334+0000","last_peered":"2024-04-02T04:55:08.939334+0000","last_clean":"2024-04-02T04:55:08.939334+0000","last_became_active":"2024-04-02T04:54:55.970782+0000","last_became_peered":"2024-04-02T04:54:55.970782+0000","last_unstale":"2024-04-02T04:55:08.939334+0000","last_undegraded":"2024-04-02T04:55:08.939334+0000","last_fullsized":"2024-04-02T04:55:08.939334+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.104470+0000","last_change":"2024-04-02T04:54:11.910124+0000","last_active":"2024-04-02T04:55:12.104470+0000","last_peered":"2024-04-02T04:55:12.104470+0000","last_clean":"2024-04-02T04:55:12.104470+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:55:12.104470+0000","last_undegraded":"2024-04-02T04:55:12.104470+0000","last_fullsized":"2024-04-02T04:55:12.104470+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:11.910003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:11.910003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.197421+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079919+0000","last_change":"2024-04-02T04:54:30.735284+0000","last_active":"2024-04-02T04:55:12.079919+0000","last_peered":"2024-04-02T04:55:12.079919+0000","last_clean":"2024-04-02T04:55:12.079919+0000","last_became_active":"2024-04-02T04:49:28.785195+0000","last_became_peered":"2024-04-02T04:49:28.785195+0000","last_unstale":"2024-04-02T04:55:12.079919+0000","last_undegraded":"2024-04-02T04:55:12.079919+0000","last_fullsized":"2024-04-02T04:55:12.079919+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:30.734970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:30.734970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:39.014963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081631+0000","last_change":"2024-04-02T04:54:56.009133+0000","last_active":"2024-04-02T04:55:12.081631+0000","last_peered":"2024-04-02T04:55:12.081631+0000","last_clean":"2024-04-02T04:55:12.081631+0000","last_became_active":"2024-04-02T04:54:56.008791+0000","last_became_peered":"2024-04-02T04:54:56.008791+0000","last_unstale":"2024-04-02T04:55:12.081631+0000","last_undegraded":"2024-04-02T04:55:12.081631+0000","last_fullsized":"2024-04-02T04:55:12.081631+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:54.645629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.645629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:20.717710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079458+0000","last_change":"2024-04-02T04:54:56.011237+0000","last_active":"2024-04-02T04:55:12.079458+0000","last_peered":"2024-04-02T04:55:12.079458+0000","last_clean":"2024-04-02T04:55:12.079458+0000","last_became_active":"2024-04-02T04:54:56.008433+0000","last_became_peered":"2024-04-02T04:54:56.008433+0000","last_unstale":"2024-04-02T04:55:12.079458+0000","last_undegraded":"2024-04-02T04:55:12.079458+0000","last_fullsized":"2024-04-02T04:55:12.079458+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.294309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":666,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102184+0000","last_change":"2024-04-02T04:50:37.102849+0000","last_active":"2024-04-02T04:55:12.102184+0000","last_peered":"2024-04-02T04:55:12.102184+0000","last_clean":"2024-04-02T04:55:12.102184+0000","last_became_active":"2024-04-02T04:49:28.698110+0000","last_became_peered":"2024-04-02T04:49:28.698110+0000","last_unstale":"2024-04-02T04:55:12.102184+0000","last_undegraded":"2024-04-02T04:55:12.102184+0000","last_fullsized":"2024-04-02T04:55:12.102184+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.102732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.102732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":667,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101509+0000","last_change":"2024-04-02T04:53:54.926479+0000","last_active":"2024-04-02T04:55:12.101509+0000","last_peered":"2024-04-02T04:55:12.101509+0000","last_clean":"2024-04-02T04:55:12.101509+0000","last_became_active":"2024-04-02T04:53:54.925994+0000","last_became_peered":"2024-04-02T04:53:54.925994+0000","last_unstale":"2024-04-02T04:55:12.101509+0000","last_undegraded":"2024-04-02T04:55:12.101509+0000","last_fullsized":"2024-04-02T04:55:12.101509+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079789+0000","last_change":"2024-04-02T04:54:56.008799+0000","last_active":"2024-04-02T04:55:12.079789+0000","last_peered":"2024-04-02T04:55:12.079789+0000","last_clean":"2024-04-02T04:55:12.079789+0000","last_became_active":"2024-04-02T04:54:56.008363+0000","last_became_peered":"2024-04-02T04:54:56.008363+0000","last_unstale":"2024-04-02T04:55:12.079789+0000","last_undegraded":"2024-04-02T04:55:12.079789+0000","last_fullsized":"2024-04-02T04:55:12.079789+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:43.275420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.275420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":565,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079149+0000","last_change":"2024-04-02T04:54:55.996329+0000","last_active":"2024-04-02T04:55:12.079149+0000","last_peered":"2024-04-02T04:55:12.079149+0000","last_clean":"2024-04-02T04:55:12.079149+0000","last_became_active":"2024-04-02T04:54:55.995831+0000","last_became_peered":"2024-04-02T04:54:55.995831+0000","last_unstale":"2024-04-02T04:55:12.079149+0000","last_undegraded":"2024-04-02T04:55:12.079149+0000","last_fullsized":"2024-04-02T04:55:12.079149+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:20.558417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:20.558417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.097122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":559,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932905+0000","last_change":"2024-04-02T04:54:55.971295+0000","last_active":"2024-04-02T04:55:08.932905+0000","last_peered":"2024-04-02T04:55:08.932905+0000","last_clean":"2024-04-02T04:55:08.932905+0000","last_became_active":"2024-04-02T04:54:55.970851+0000","last_became_peered":"2024-04-02T04:54:55.970851+0000","last_unstale":"2024-04-02T04:55:08.932905+0000","last_undegraded":"2024-04-02T04:55:08.932905+0000","last_fullsized":"2024-04-02T04:55:08.932905+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.641792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.641792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:45.080877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.096614+0000","last_change":"2024-04-02T04:52:38.809910+0000","last_active":"2024-04-02T04:55:12.096614+0000","last_peered":"2024-04-02T04:55:12.096614+0000","last_clean":"2024-04-02T04:55:12.096614+0000","last_became_active":"2024-04-02T04:49:29.857200+0000","last_became_peered":"2024-04-02T04:49:29.857200+0000","last_unstale":"2024-04-02T04:55:12.096614+0000","last_undegraded":"2024-04-02T04:55:12.096614+0000","last_fullsized":"2024-04-02T04:55:12.096614+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.809807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.809807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":666,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100762+0000","last_change":"2024-04-02T04:51:35.000630+0000","last_active":"2024-04-02T04:55:12.100762+0000","last_peered":"2024-04-02T04:55:12.100762+0000","last_clean":"2024-04-02T04:55:12.100762+0000","last_became_active":"2024-04-02T04:49:28.746855+0000","last_became_peered":"2024-04-02T04:49:28.746855+0000","last_unstale":"2024-04-02T04:55:12.100762+0000","last_undegraded":"2024-04-02T04:55:12.100762+0000","last_fullsized":"2024-04-02T04:55:12.100762+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:35.000533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.000533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":673,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.090742+0000","last_change":"2024-04-02T04:54:55.962034+0000","last_active":"2024-04-02T04:55:12.090742+0000","last_peered":"2024-04-02T04:55:12.090742+0000","last_clean":"2024-04-02T04:55:12.090742+0000","last_became_active":"2024-04-02T04:54:55.961664+0000","last_became_peered":"2024-04-02T04:54:55.961664+0000","last_unstale":"2024-04-02T04:55:12.090742+0000","last_undegraded":"2024-04-02T04:55:12.090742+0000","last_fullsized":"2024-04-02T04:55:12.090742+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.096358+0000","last_change":"2024-04-02T04:51:36.554382+0000","last_active":"2024-04-02T04:55:12.096358+0000","last_peered":"2024-04-02T04:55:12.096358+0000","last_clean":"2024-04-02T04:55:12.096358+0000","last_became_active":"2024-04-02T04:49:29.874539+0000","last_became_peered":"2024-04-02T04:49:29.874539+0000","last_unstale":"2024-04-02T04:55:12.096358+0000","last_undegraded":"2024-04-02T04:55:12.096358+0000","last_fullsized":"2024-04-02T04:55:12.096358+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.554164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.554164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":677,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081635+0000","last_change":"2024-04-02T04:54:14.666642+0000","last_active":"2024-04-02T04:55:12.081635+0000","last_peered":"2024-04-02T04:55:12.081635+0000","last_clean":"2024-04-02T04:55:12.081635+0000","last_became_active":"2024-04-02T04:53:54.920551+0000","last_became_peered":"2024-04-02T04:53:54.920551+0000","last_unstale":"2024-04-02T04:55:12.081635+0000","last_undegraded":"2024-04-02T04:55:12.081635+0000","last_fullsized":"2024-04-02T04:55:12.081635+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:14.666546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:14.666546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:33.693694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.111398+0000","last_change":"2024-04-02T04:51:53.758572+0000","last_active":"2024-04-02T04:55:12.111398+0000","last_peered":"2024-04-02T04:55:12.111398+0000","last_clean":"2024-04-02T04:55:12.111398+0000","last_became_active":"2024-04-02T04:49:29.830080+0000","last_became_peered":"2024-04-02T04:49:29.830080+0000","last_unstale":"2024-04-02T04:55:12.111398+0000","last_undegraded":"2024-04-02T04:55:12.111398+0000","last_fullsized":"2024-04-02T04:55:12.111398+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.758420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.758420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119841+0000","last_change":"2024-04-02T04:54:26.234089+0000","last_active":"2024-04-02T04:55:12.119841+0000","last_peered":"2024-04-02T04:55:12.119841+0000","last_clean":"2024-04-02T04:55:12.119841+0000","last_became_active":"2024-04-02T04:53:54.955517+0000","last_became_peered":"2024-04-02T04:53:54.955517+0000","last_unstale":"2024-04-02T04:55:12.119841+0000","last_undegraded":"2024-04-02T04:55:12.119841+0000","last_fullsized":"2024-04-02T04:55:12.119841+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.233991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.233991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:41.669914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":670,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449598+0000","last_change":"2024-04-02T04:54:56.007860+0000","last_active":"2024-04-02T04:55:12.449598+0000","last_peered":"2024-04-02T04:55:12.449598+0000","last_clean":"2024-04-02T04:55:12.449598+0000","last_became_active":"2024-04-02T04:54:56.006494+0000","last_became_peered":"2024-04-02T04:54:56.006494+0000","last_unstale":"2024-04-02T04:55:12.449598+0000","last_undegraded":"2024-04-02T04:55:12.449598+0000","last_fullsized":"2024-04-02T04:55:12.449598+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.096832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.096832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081770+0000","last_change":"2024-04-02T04:54:56.005290+0000","last_active":"2024-04-02T04:55:12.081770+0000","last_peered":"2024-04-02T04:55:12.081770+0000","last_clean":"2024-04-02T04:55:12.081770+0000","last_became_active":"2024-04-02T04:54:56.004451+0000","last_became_peered":"2024-04-02T04:54:56.004451+0000","last_unstale":"2024-04-02T04:55:12.081770+0000","last_undegraded":"2024-04-02T04:55:12.081770+0000","last_fullsized":"2024-04-02T04:55:12.081770+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.334949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.334949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081456+0000","last_change":"2024-04-02T04:54:56.009048+0000","last_active":"2024-04-02T04:55:12.081456+0000","last_peered":"2024-04-02T04:55:12.081456+0000","last_clean":"2024-04-02T04:55:12.081456+0000","last_became_active":"2024-04-02T04:54:56.004907+0000","last_became_peered":"2024-04-02T04:54:56.004907+0000","last_unstale":"2024-04-02T04:55:12.081456+0000","last_undegraded":"2024-04-02T04:55:12.081456+0000","last_fullsized":"2024-04-02T04:55:12.081456+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:08.577227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:08.577227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448119+0000","last_change":"2024-04-02T04:53:10.492126+0000","last_active":"2024-04-02T04:55:12.448119+0000","last_peered":"2024-04-02T04:55:12.448119+0000","last_clean":"2024-04-02T04:55:12.448119+0000","last_became_active":"2024-04-02T04:49:28.760661+0000","last_became_peered":"2024-04-02T04:49:28.760661+0000","last_unstale":"2024-04-02T04:55:12.448119+0000","last_undegraded":"2024-04-02T04:55:12.448119+0000","last_fullsized":"2024-04-02T04:55:12.448119+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.491899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.491899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":670,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448948+0000","last_change":"2024-04-02T04:54:55.979416+0000","last_active":"2024-04-02T04:55:12.448948+0000","last_peered":"2024-04-02T04:55:12.448948+0000","last_clean":"2024-04-02T04:55:12.448948+0000","last_became_active":"2024-04-02T04:54:55.978988+0000","last_became_peered":"2024-04-02T04:54:55.978988+0000","last_unstale":"2024-04-02T04:55:12.448948+0000","last_undegraded":"2024-04-02T04:55:12.448948+0000","last_fullsized":"2024-04-02T04:55:12.448948+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:39.367130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:39.367130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":672,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.098534+0000","last_change":"2024-04-02T04:54:55.941918+0000","last_active":"2024-04-02T04:55:12.098534+0000","last_peered":"2024-04-02T04:55:12.098534+0000","last_clean":"2024-04-02T04:55:12.098534+0000","last_became_active":"2024-04-02T04:54:55.941554+0000","last_became_peered":"2024-04-02T04:54:55.941554+0000","last_unstale":"2024-04-02T04:55:12.098534+0000","last_undegraded":"2024-04-02T04:55:12.098534+0000","last_fullsized":"2024-04-02T04:55:12.098534+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:44.205739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:44.205739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.780284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.108750+0000","last_change":"2024-04-02T04:53:13.855229+0000","last_active":"2024-04-02T04:55:12.108750+0000","last_peered":"2024-04-02T04:55:12.108750+0000","last_clean":"2024-04-02T04:55:12.108750+0000","last_became_active":"2024-04-02T04:49:28.727661+0000","last_became_peered":"2024-04-02T04:49:28.727661+0000","last_unstale":"2024-04-02T04:55:12.108750+0000","last_undegraded":"2024-04-02T04:55:12.108750+0000","last_fullsized":"2024-04-02T04:55:12.108750+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.855124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.855124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448558+0000","last_change":"2024-04-02T04:53:00.409859+0000","last_active":"2024-04-02T04:55:12.448558+0000","last_peered":"2024-04-02T04:55:12.448558+0000","last_clean":"2024-04-02T04:55:12.448558+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:55:12.448558+0000","last_undegraded":"2024-04-02T04:55:12.448558+0000","last_fullsized":"2024-04-02T04:55:12.448558+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.409610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.409610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.444851+0000","last_change":"2024-04-02T04:53:06.476121+0000","last_active":"2024-04-02T04:55:12.444851+0000","last_peered":"2024-04-02T04:55:12.444851+0000","last_clean":"2024-04-02T04:55:12.444851+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:55:12.444851+0000","last_undegraded":"2024-04-02T04:55:12.444851+0000","last_fullsized":"2024-04-02T04:55:12.444851+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.475984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.475984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":155,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932097+0000","last_change":"2024-04-02T04:53:54.918758+0000","last_active":"2024-04-02T04:55:08.932097+0000","last_peered":"2024-04-02T04:55:08.932097+0000","last_clean":"2024-04-02T04:55:08.932097+0000","last_became_active":"2024-04-02T04:53:54.918416+0000","last_became_peered":"2024-04-02T04:53:54.918416+0000","last_unstale":"2024-04-02T04:55:08.932097+0000","last_undegraded":"2024-04-02T04:55:08.932097+0000","last_fullsized":"2024-04-02T04:55:08.932097+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":672,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102651+0000","last_change":"2024-04-02T04:54:55.989937+0000","last_active":"2024-04-02T04:55:12.102651+0000","last_peered":"2024-04-02T04:55:12.102651+0000","last_clean":"2024-04-02T04:55:12.102651+0000","last_became_active":"2024-04-02T04:54:55.989586+0000","last_became_peered":"2024-04-02T04:54:55.989586+0000","last_unstale":"2024-04-02T04:55:12.102651+0000","last_undegraded":"2024-04-02T04:55:12.102651+0000","last_fullsized":"2024-04-02T04:55:12.102651+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.958371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099559+0000","last_change":"2024-04-02T04:54:55.988854+0000","last_active":"2024-04-02T04:55:12.099559+0000","last_peered":"2024-04-02T04:55:12.099559+0000","last_clean":"2024-04-02T04:55:12.099559+0000","last_became_active":"2024-04-02T04:54:55.988651+0000","last_became_peered":"2024-04-02T04:54:55.988651+0000","last_unstale":"2024-04-02T04:55:12.099559+0000","last_undegraded":"2024-04-02T04:55:12.099559+0000","last_fullsized":"2024-04-02T04:55:12.099559+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.661376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.661376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102317+0000","last_change":"2024-04-02T04:51:27.695556+0000","last_active":"2024-04-02T04:55:12.102317+0000","last_peered":"2024-04-02T04:55:12.102317+0000","last_clean":"2024-04-02T04:55:12.102317+0000","last_became_active":"2024-04-02T04:49:29.856708+0000","last_became_peered":"2024-04-02T04:49:29.856708+0000","last_unstale":"2024-04-02T04:55:12.102317+0000","last_undegraded":"2024-04-02T04:55:12.102317+0000","last_fullsized":"2024-04-02T04:55:12.102317+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:27.695437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.695437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.093590+0000","last_change":"2024-04-02T04:51:29.649636+0000","last_active":"2024-04-02T04:55:12.093590+0000","last_peered":"2024-04-02T04:55:12.093590+0000","last_clean":"2024-04-02T04:55:12.093590+0000","last_became_active":"2024-04-02T04:49:29.816194+0000","last_became_peered":"2024-04-02T04:49:29.816194+0000","last_unstale":"2024-04-02T04:55:12.093590+0000","last_undegraded":"2024-04-02T04:55:12.093590+0000","last_fullsized":"2024-04-02T04:55:12.093590+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.649422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.649422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":681,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.094252+0000","last_change":"2024-04-02T04:54:55.928923+0000","last_active":"2024-04-02T04:55:12.094252+0000","last_peered":"2024-04-02T04:55:12.094252+0000","last_clean":"2024-04-02T04:55:12.094252+0000","last_became_active":"2024-04-02T04:54:55.928548+0000","last_became_peered":"2024-04-02T04:54:55.928548+0000","last_unstale":"2024-04-02T04:55:12.094252+0000","last_undegraded":"2024-04-02T04:55:12.094252+0000","last_fullsized":"2024-04-02T04:55:12.094252+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.882197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.882197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:32.957200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079317+0000","last_change":"2024-04-02T04:54:56.009403+0000","last_active":"2024-04-02T04:55:12.079317+0000","last_peered":"2024-04-02T04:55:12.079317+0000","last_clean":"2024-04-02T04:55:12.079317+0000","last_became_active":"2024-04-02T04:54:56.005204+0000","last_became_peered":"2024-04-02T04:54:56.005204+0000","last_unstale":"2024-04-02T04:55:12.079317+0000","last_undegraded":"2024-04-02T04:55:12.079317+0000","last_fullsized":"2024-04-02T04:55:12.079317+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.916527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118697+0000","last_change":"2024-04-02T04:54:55.969087+0000","last_active":"2024-04-02T04:55:12.118697+0000","last_peered":"2024-04-02T04:55:12.118697+0000","last_clean":"2024-04-02T04:55:12.118697+0000","last_became_active":"2024-04-02T04:54:55.968463+0000","last_became_peered":"2024-04-02T04:54:55.968463+0000","last_unstale":"2024-04-02T04:55:12.118697+0000","last_undegraded":"2024-04-02T04:55:12.118697+0000","last_fullsized":"2024-04-02T04:55:12.118697+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.252102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.252102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101354+0000","last_change":"2024-04-02T04:52:56.597071+0000","last_active":"2024-04-02T04:55:12.101354+0000","last_peered":"2024-04-02T04:55:12.101354+0000","last_clean":"2024-04-02T04:55:12.101354+0000","last_became_active":"2024-04-02T04:49:28.749072+0000","last_became_peered":"2024-04-02T04:49:28.749072+0000","last_unstale":"2024-04-02T04:55:12.101354+0000","last_undegraded":"2024-04-02T04:55:12.101354+0000","last_fullsized":"2024-04-02T04:55:12.101354+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.596968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.596968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100276+0000","last_change":"2024-04-02T04:51:51.741608+0000","last_active":"2024-04-02T04:55:12.100276+0000","last_peered":"2024-04-02T04:55:12.100276+0000","last_clean":"2024-04-02T04:55:12.100276+0000","last_became_active":"2024-04-02T04:49:29.858586+0000","last_became_peered":"2024-04-02T04:49:29.858586+0000","last_unstale":"2024-04-02T04:55:12.100276+0000","last_undegraded":"2024-04-02T04:55:12.100276+0000","last_fullsized":"2024-04-02T04:55:12.100276+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:51.741382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:51.741382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.115861+0000","last_change":"2024-04-02T04:54:02.337934+0000","last_active":"2024-04-02T04:55:12.115861+0000","last_peered":"2024-04-02T04:55:12.115861+0000","last_clean":"2024-04-02T04:55:12.115861+0000","last_became_active":"2024-04-02T04:53:54.929052+0000","last_became_peered":"2024-04-02T04:53:54.929052+0000","last_unstale":"2024-04-02T04:55:12.115861+0000","last_undegraded":"2024-04-02T04:55:12.115861+0000","last_fullsized":"2024-04-02T04:55:12.115861+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:02.337688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:02.337688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":155,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933051+0000","last_change":"2024-04-02T04:53:54.944277+0000","last_active":"2024-04-02T04:55:08.933051+0000","last_peered":"2024-04-02T04:55:08.933051+0000","last_clean":"2024-04-02T04:55:08.933051+0000","last_became_active":"2024-04-02T04:53:54.943480+0000","last_became_peered":"2024-04-02T04:53:54.943480+0000","last_unstale":"2024-04-02T04:55:08.933051+0000","last_undegraded":"2024-04-02T04:55:08.933051+0000","last_fullsized":"2024-04-02T04:55:08.933051+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101617+0000","last_change":"2024-04-02T04:54:55.986523+0000","last_active":"2024-04-02T04:55:12.101617+0000","last_peered":"2024-04-02T04:55:12.101617+0000","last_clean":"2024-04-02T04:55:12.101617+0000","last_became_active":"2024-04-02T04:54:55.985717+0000","last_became_peered":"2024-04-02T04:54:55.985717+0000","last_unstale":"2024-04-02T04:55:12.101617+0000","last_undegraded":"2024-04-02T04:55:12.101617+0000","last_fullsized":"2024-04-02T04:55:12.101617+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.540654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.540654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449616+0000","last_change":"2024-04-02T04:52:53.411217+0000","last_active":"2024-04-02T04:55:12.449616+0000","last_peered":"2024-04-02T04:55:12.449616+0000","last_clean":"2024-04-02T04:55:12.449616+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:55:12.449616+0000","last_undegraded":"2024-04-02T04:55:12.449616+0000","last_fullsized":"2024-04-02T04:55:12.449616+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:53.410913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.410913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":155,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.935626+0000","last_change":"2024-04-02T04:53:54.936976+0000","last_active":"2024-04-02T04:55:08.935626+0000","last_peered":"2024-04-02T04:55:08.935626+0000","last_clean":"2024-04-02T04:55:08.935626+0000","last_became_active":"2024-04-02T04:53:54.933626+0000","last_became_peered":"2024-04-02T04:53:54.933626+0000","last_unstale":"2024-04-02T04:55:08.935626+0000","last_undegraded":"2024-04-02T04:55:08.935626+0000","last_fullsized":"2024-04-02T04:55:08.935626+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.098877+0000","last_change":"2024-04-02T04:52:48.599707+0000","last_active":"2024-04-02T04:55:12.098877+0000","last_peered":"2024-04-02T04:55:12.098877+0000","last_clean":"2024-04-02T04:55:12.098877+0000","last_became_active":"2024-04-02T04:49:28.716933+0000","last_became_peered":"2024-04-02T04:49:28.716933+0000","last_unstale":"2024-04-02T04:55:12.098877+0000","last_undegraded":"2024-04-02T04:55:12.098877+0000","last_fullsized":"2024-04-02T04:55:12.098877+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:48.599595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.991167+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.599595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081743+0000","last_change":"2024-04-02T04:51:52.390967+0000","last_active":"2024-04-02T04:55:12.081743+0000","last_peered":"2024-04-02T04:55:12.081743+0000","last_clean":"2024-04-02T04:55:12.081743+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:55:12.081743+0000","last_undegraded":"2024-04-02T04:55:12.081743+0000","last_fullsized":"2024-04-02T04:55:12.081743+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.390694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.390694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449382+0000","last_change":"2024-04-02T04:53:11.473195+0000","last_active":"2024-04-02T04:55:12.449382+0000","last_peered":"2024-04-02T04:55:12.449382+0000","last_clean":"2024-04-02T04:55:12.449382+0000","last_became_active":"2024-04-02T04:49:28.757217+0000","last_became_peered":"2024-04-02T04:49:28.757217+0000","last_unstale":"2024-04-02T04:55:12.449382+0000","last_undegraded":"2024-04-02T04:55:12.449382+0000","last_fullsized":"2024-04-02T04:55:12.449382+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.473096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.473096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119443+0000","last_change":"2024-04-02T04:54:04.337974+0000","last_active":"2024-04-02T04:55:12.119443+0000","last_peered":"2024-04-02T04:55:12.119443+0000","last_clean":"2024-04-02T04:55:12.119443+0000","last_became_active":"2024-04-02T04:53:54.940640+0000","last_became_peered":"2024-04-02T04:53:54.940640+0000","last_unstale":"2024-04-02T04:55:12.119443+0000","last_undegraded":"2024-04-02T04:55:12.119443+0000","last_fullsized":"2024-04-02T04:55:12.119443+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.337869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.337869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.227233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":666,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.109900+0000","last_change":"2024-04-02T04:51:25.125020+0000","last_active":"2024-04-02T04:55:12.109900+0000","last_peered":"2024-04-02T04:55:12.109900+0000","last_clean":"2024-04-02T04:55:12.109900+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:55:12.109900+0000","last_undegraded":"2024-04-02T04:55:12.109900+0000","last_fullsized":"2024-04-02T04:55:12.109900+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:25.124929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:25.124929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120523+0000","last_change":"2024-04-02T04:54:25.202447+0000","last_active":"2024-04-02T04:55:12.120523+0000","last_peered":"2024-04-02T04:55:12.120523+0000","last_clean":"2024-04-02T04:55:12.120523+0000","last_became_active":"2024-04-02T04:53:54.921036+0000","last_became_peered":"2024-04-02T04:53:54.921036+0000","last_unstale":"2024-04-02T04:55:12.120523+0000","last_undegraded":"2024-04-02T04:55:12.120523+0000","last_fullsized":"2024-04-02T04:55:12.120523+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:25.202339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:25.202339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:46.782432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":667,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110078+0000","last_change":"2024-04-02T04:53:54.945162+0000","last_active":"2024-04-02T04:55:12.110078+0000","last_peered":"2024-04-02T04:55:12.110078+0000","last_clean":"2024-04-02T04:55:12.110078+0000","last_became_active":"2024-04-02T04:53:54.941043+0000","last_became_peered":"2024-04-02T04:53:54.941043+0000","last_unstale":"2024-04-02T04:55:12.110078+0000","last_undegraded":"2024-04-02T04:55:12.110078+0000","last_fullsized":"2024-04-02T04:55:12.110078+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:49:24.948823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097396+0000","last_change":"2024-04-02T04:54:55.940325+0000","last_active":"2024-04-02T04:55:12.097396+0000","last_peered":"2024-04-02T04:55:12.097396+0000","last_clean":"2024-04-02T04:55:12.097396+0000","last_became_active":"2024-04-02T04:54:55.939534+0000","last_became_peered":"2024-04-02T04:54:55.939534+0000","last_unstale":"2024-04-02T04:55:12.097396+0000","last_undegraded":"2024-04-02T04:55:12.097396+0000","last_fullsized":"2024-04-02T04:55:12.097396+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.673796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.673796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":155,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932209+0000","last_change":"2024-04-02T04:53:54.947489+0000","last_active":"2024-04-02T04:55:08.932209+0000","last_peered":"2024-04-02T04:55:08.932209+0000","last_clean":"2024-04-02T04:55:08.932209+0000","last_became_active":"2024-04-02T04:53:54.945599+0000","last_became_peered":"2024-04-02T04:53:54.945599+0000","last_unstale":"2024-04-02T04:55:08.932209+0000","last_undegraded":"2024-04-02T04:55:08.932209+0000","last_fullsized":"2024-04-02T04:55:08.932209+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":275,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117296+0000","last_change":"2024-04-02T04:52:58.361166+0000","last_active":"2024-04-02T04:55:12.117296+0000","last_peered":"2024-04-02T04:55:12.117296+0000","last_clean":"2024-04-02T04:55:12.117296+0000","last_became_active":"2024-04-02T04:52:55.872480+0000","last_became_peered":"2024-04-02T04:52:55.872480+0000","last_unstale":"2024-04-02T04:55:12.117296+0000","last_undegraded":"2024-04-02T04:55:12.117296+0000","last_fullsized":"2024-04-02T04:55:12.117296+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.361050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.361050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102505+0000","last_change":"2024-04-02T04:54:03.883895+0000","last_active":"2024-04-02T04:55:12.102505+0000","last_peered":"2024-04-02T04:55:12.102505+0000","last_clean":"2024-04-02T04:55:12.102505+0000","last_became_active":"2024-04-02T04:53:54.940670+0000","last_became_peered":"2024-04-02T04:53:54.940670+0000","last_unstale":"2024-04-02T04:55:12.102505+0000","last_undegraded":"2024-04-02T04:55:12.102505+0000","last_fullsized":"2024-04-02T04:55:12.102505+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.883769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.883769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.172141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097427+0000","last_change":"2024-04-02T04:54:55.930221+0000","last_active":"2024-04-02T04:55:12.097427+0000","last_peered":"2024-04-02T04:55:12.097427+0000","last_clean":"2024-04-02T04:55:12.097427+0000","last_became_active":"2024-04-02T04:54:55.928941+0000","last_became_peered":"2024-04-02T04:54:55.928941+0000","last_unstale":"2024-04-02T04:55:12.097427+0000","last_undegraded":"2024-04-02T04:55:12.097427+0000","last_fullsized":"2024-04-02T04:55:12.097427+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.711178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.711178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":678,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097603+0000","last_change":"2024-04-02T04:54:24.097179+0000","last_active":"2024-04-02T04:55:12.097603+0000","last_peered":"2024-04-02T04:55:12.097603+0000","last_clean":"2024-04-02T04:55:12.097603+0000","last_became_active":"2024-04-02T04:53:54.943555+0000","last_became_peered":"2024-04-02T04:53:54.943555+0000","last_unstale":"2024-04-02T04:55:12.097603+0000","last_undegraded":"2024-04-02T04:55:12.097603+0000","last_fullsized":"2024-04-02T04:55:12.097603+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:24.097035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.097035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:36.183732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.093332+0000","last_change":"2024-04-02T04:52:49.762675+0000","last_active":"2024-04-02T04:55:12.093332+0000","last_peered":"2024-04-02T04:55:12.093332+0000","last_clean":"2024-04-02T04:55:12.093332+0000","last_became_active":"2024-04-02T04:49:29.805341+0000","last_became_peered":"2024-04-02T04:49:29.805341+0000","last_unstale":"2024-04-02T04:55:12.093332+0000","last_undegraded":"2024-04-02T04:55:12.093332+0000","last_fullsized":"2024-04-02T04:55:12.093332+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.762442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.762442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":161,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933410+0000","last_change":"2024-04-02T04:54:55.956664+0000","last_active":"2024-04-02T04:55:08.933410+0000","last_peered":"2024-04-02T04:55:08.933410+0000","last_clean":"2024-04-02T04:55:08.933410+0000","last_became_active":"2024-04-02T04:54:55.956199+0000","last_became_peered":"2024-04-02T04:54:55.956199+0000","last_unstale":"2024-04-02T04:55:08.933410+0000","last_undegraded":"2024-04-02T04:55:08.933410+0000","last_fullsized":"2024-04-02T04:55:08.933410+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:49:25.981622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":46,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079318+0000","last_change":"2024-04-02T04:54:56.009161+0000","last_active":"2024-04-02T04:55:12.079318+0000","last_peered":"2024-04-02T04:55:12.079318+0000","last_clean":"2024-04-02T04:55:12.079318+0000","last_became_active":"2024-04-02T04:54:56.008209+0000","last_became_peered":"2024-04-02T04:54:56.008209+0000","last_unstale":"2024-04-02T04:55:12.079318+0000","last_undegraded":"2024-04-02T04:55:12.079318+0000","last_fullsized":"2024-04-02T04:55:12.079318+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:45.521589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:45.521589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079467+0000","last_change":"2024-04-02T04:52:42.318994+0000","last_active":"2024-04-02T04:55:12.079467+0000","last_peered":"2024-04-02T04:55:12.079467+0000","last_clean":"2024-04-02T04:55:12.079467+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:55:12.079467+0000","last_undegraded":"2024-04-02T04:55:12.079467+0000","last_fullsized":"2024-04-02T04:55:12.079467+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.318800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.318800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921004+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":664,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447411+0000","last_change":"2024-04-02T04:52:52.417888+0000","last_active":"2024-04-02T04:55:12.447411+0000","last_peered":"2024-04-02T04:55:12.447411+0000","last_clean":"2024-04-02T04:55:12.447411+0000","last_became_active":"2024-04-02T04:49:28.753376+0000","last_became_peered":"2024-04-02T04:49:28.753376+0000","last_unstale":"2024-04-02T04:55:12.447411+0000","last_undegraded":"2024-04-02T04:55:12.447411+0000","last_fullsized":"2024-04-02T04:55:12.447411+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.417771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.417771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":672,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.107617+0000","last_change":"2024-04-02T04:54:55.990682+0000","last_active":"2024-04-02T04:55:12.107617+0000","last_peered":"2024-04-02T04:55:12.107617+0000","last_clean":"2024-04-02T04:55:12.107617+0000","last_became_active":"2024-04-02T04:54:55.990160+0000","last_became_peered":"2024-04-02T04:54:55.990160+0000","last_unstale":"2024-04-02T04:55:12.107617+0000","last_undegraded":"2024-04-02T04:55:12.107617+0000","last_fullsized":"2024-04-02T04:55:12.107617+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.257747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.257747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":672,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.090998+0000","last_change":"2024-04-02T04:54:56.002298+0000","last_active":"2024-04-02T04:55:12.090998+0000","last_peered":"2024-04-02T04:55:12.090998+0000","last_clean":"2024-04-02T04:55:12.090998+0000","last_became_active":"2024-04-02T04:54:56.001980+0000","last_became_peered":"2024-04-02T04:54:56.001980+0000","last_unstale":"2024-04-02T04:55:12.090998+0000","last_undegraded":"2024-04-02T04:55:12.090998+0000","last_fullsized":"2024-04-02T04:55:12.090998+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.983966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.983966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"139'5","reported_seq":687,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445628+0000","last_change":"2024-04-02T04:50:26.846127+0000","last_active":"2024-04-02T04:55:12.445628+0000","last_peered":"2024-04-02T04:55:12.445628+0000","last_clean":"2024-04-02T04:55:12.445628+0000","last_became_active":"2024-04-02T04:49:26.170913+0000","last_became_peered":"2024-04-02T04:49:26.170913+0000","last_unstale":"2024-04-02T04:55:12.445628+0000","last_undegraded":"2024-04-02T04:55:12.445628+0000","last_fullsized":"2024-04-02T04:55:12.445628+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:50:26.845900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.845900+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":47,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112053+0000","last_change":"2024-04-02T04:54:56.017671+0000","last_active":"2024-04-02T04:55:12.112053+0000","last_peered":"2024-04-02T04:55:12.112053+0000","last_clean":"2024-04-02T04:55:12.112053+0000","last_became_active":"2024-04-02T04:54:56.017309+0000","last_became_peered":"2024-04-02T04:54:56.017309+0000","last_unstale":"2024-04-02T04:55:12.112053+0000","last_undegraded":"2024-04-02T04:55:12.112053+0000","last_fullsized":"2024-04-02T04:55:12.112053+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:35.773829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:35.773829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:03.864945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":707,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.931254+0000","last_change":"2024-04-02T04:54:55.990438+0000","last_active":"2024-04-02T04:55:08.931254+0000","last_peered":"2024-04-02T04:55:08.931254+0000","last_clean":"2024-04-02T04:55:08.931254+0000","last_became_active":"2024-04-02T04:54:55.990081+0000","last_became_peered":"2024-04-02T04:54:55.990081+0000","last_unstale":"2024-04-02T04:55:08.931254+0000","last_undegraded":"2024-04-02T04:55:08.931254+0000","last_fullsized":"2024-04-02T04:55:08.931254+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:12.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.577625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:37.113179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"415'28","reported_seq":331,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.941094+0000","last_change":"2024-04-02T04:54:15.261937+0000","last_active":"2024-04-02T04:55:08.941094+0000","last_peered":"2024-04-02T04:55:08.941094+0000","last_clean":"2024-04-02T04:55:08.941094+0000","last_became_active":"2024-04-02T04:53:54.938166+0000","last_became_peered":"2024-04-02T04:53:54.938166+0000","last_unstale":"2024-04-02T04:55:08.941094+0000","last_undegraded":"2024-04-02T04:55:08.941094+0000","last_fullsized":"2024-04-02T04:55:08.941094+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'28","last_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_deep_scrub":"415'28","last_deep_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_clean_scrub_stamp":"2024-04-02T04:54:15.261679+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:45.099247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.113693+0000","last_change":"2024-04-02T04:54:00.930078+0000","last_active":"2024-04-02T04:55:12.113693+0000","last_peered":"2024-04-02T04:55:12.113693+0000","last_clean":"2024-04-02T04:55:12.113693+0000","last_became_active":"2024-04-02T04:53:54.938837+0000","last_became_peered":"2024-04-02T04:53:54.938837+0000","last_unstale":"2024-04-02T04:55:12.113693+0000","last_undegraded":"2024-04-02T04:55:12.113693+0000","last_fullsized":"2024-04-02T04:55:12.113693+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.929851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.929851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.780513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":44,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445182+0000","last_change":"2024-04-02T04:54:56.003791+0000","last_active":"2024-04-02T04:55:12.445182+0000","last_peered":"2024-04-02T04:55:12.445182+0000","last_clean":"2024-04-02T04:55:12.445182+0000","last_became_active":"2024-04-02T04:54:56.002149+0000","last_became_peered":"2024-04-02T04:54:56.002149+0000","last_unstale":"2024-04-02T04:55:12.445182+0000","last_undegraded":"2024-04-02T04:55:12.445182+0000","last_fullsized":"2024-04-02T04:55:12.445182+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.685936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.685936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":55,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.095146+0000","last_change":"2024-04-02T04:54:55.986396+0000","last_active":"2024-04-02T04:55:12.095146+0000","last_peered":"2024-04-02T04:55:12.095146+0000","last_clean":"2024-04-02T04:55:12.095146+0000","last_became_active":"2024-04-02T04:54:55.985950+0000","last_became_peered":"2024-04-02T04:54:55.985950+0000","last_unstale":"2024-04-02T04:55:12.095146+0000","last_undegraded":"2024-04-02T04:55:12.095146+0000","last_fullsized":"2024-04-02T04:55:12.095146+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:38.662076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.662076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":679,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.108246+0000","last_change":"2024-04-02T04:52:57.641001+0000","last_active":"2024-04-02T04:55:12.108246+0000","last_peered":"2024-04-02T04:55:12.108246+0000","last_clean":"2024-04-02T04:55:12.108246+0000","last_became_active":"2024-04-02T04:49:26.296797+0000","last_became_peered":"2024-04-02T04:49:26.296797+0000","last_unstale":"2024-04-02T04:55:12.108246+0000","last_undegraded":"2024-04-02T04:55:12.108246+0000","last_fullsized":"2024-04-02T04:55:12.108246+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.640866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.640866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103500+0000","last_change":"2024-04-02T04:52:55.335191+0000","last_active":"2024-04-02T04:55:12.103500+0000","last_peered":"2024-04-02T04:55:12.103500+0000","last_clean":"2024-04-02T04:55:12.103500+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:55:12.103500+0000","last_undegraded":"2024-04-02T04:55:12.103500+0000","last_fullsized":"2024-04-02T04:55:12.103500+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:55.334980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.334980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"402'25","reported_seq":792,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.096896+0000","last_change":"2024-04-02T04:54:56.013356+0000","last_active":"2024-04-02T04:55:12.096896+0000","last_peered":"2024-04-02T04:55:12.096896+0000","last_clean":"2024-04-02T04:55:12.096896+0000","last_became_active":"2024-04-02T04:54:55.986229+0000","last_became_peered":"2024-04-02T04:54:55.986229+0000","last_unstale":"2024-04-02T04:55:12.096896+0000","last_undegraded":"2024-04-02T04:55:12.096896+0000","last_fullsized":"2024-04-02T04:55:12.096896+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:53:07.600375+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.600375+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.121289+0000","last_change":"2024-04-02T04:53:10.415912+0000","last_active":"2024-04-02T04:55:12.121289+0000","last_peered":"2024-04-02T04:55:12.121289+0000","last_clean":"2024-04-02T04:55:12.121289+0000","last_became_active":"2024-04-02T04:43:54.226045+0000","last_became_peered":"2024-04-02T04:43:54.226045+0000","last_unstale":"2024-04-02T04:55:12.121289+0000","last_undegraded":"2024-04-02T04:55:12.121289+0000","last_fullsized":"2024-04-02T04:55:12.121289+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.415766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":805,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.107742+0000","last_change":"2024-04-02T04:54:19.360031+0000","last_active":"2024-04-02T04:55:12.107742+0000","last_peered":"2024-04-02T04:55:12.107742+0000","last_clean":"2024-04-02T04:55:12.107742+0000","last_became_active":"2024-04-02T04:53:54.935778+0000","last_became_peered":"2024-04-02T04:53:54.935778+0000","last_unstale":"2024-04-02T04:55:12.107742+0000","last_undegraded":"2024-04-02T04:55:12.107742+0000","last_fullsized":"2024-04-02T04:55:12.107742+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.359786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.359786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:46.605763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"139'5","reported_seq":691,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102810+0000","last_change":"2024-04-02T04:51:27.143046+0000","last_active":"2024-04-02T04:55:12.102810+0000","last_peered":"2024-04-02T04:55:12.102810+0000","last_clean":"2024-04-02T04:55:12.102810+0000","last_became_active":"2024-04-02T04:49:25.139936+0000","last_became_peered":"2024-04-02T04:49:25.139936+0000","last_unstale":"2024-04-02T04:55:12.102810+0000","last_undegraded":"2024-04-02T04:55:12.102810+0000","last_fullsized":"2024-04-02T04:55:12.102810+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:27.142908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:27.142908+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":680,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.105193+0000","last_change":"2024-04-02T04:53:10.335701+0000","last_active":"2024-04-02T04:55:12.105193+0000","last_peered":"2024-04-02T04:55:12.105193+0000","last_clean":"2024-04-02T04:55:12.105193+0000","last_became_active":"2024-04-02T04:49:25.143060+0000","last_became_peered":"2024-04-02T04:49:25.143060+0000","last_unstale":"2024-04-02T04:55:12.105193+0000","last_undegraded":"2024-04-02T04:55:12.105193+0000","last_fullsized":"2024-04-02T04:55:12.105193+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.335602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.335602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445814+0000","last_change":"2024-04-02T04:54:01.950572+0000","last_active":"2024-04-02T04:55:12.445814+0000","last_peered":"2024-04-02T04:55:12.445814+0000","last_clean":"2024-04-02T04:55:12.445814+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T04:55:12.445814+0000","last_undegraded":"2024-04-02T04:55:12.445814+0000","last_fullsized":"2024-04-02T04:55:12.445814+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.950469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.950469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:24.390817+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"402'31","reported_seq":366,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.937210+0000","last_change":"2024-04-02T04:54:56.015627+0000","last_active":"2024-04-02T04:55:08.937210+0000","last_peered":"2024-04-02T04:55:08.937210+0000","last_clean":"2024-04-02T04:55:08.937210+0000","last_became_active":"2024-04-02T04:54:55.980317+0000","last_became_peered":"2024-04-02T04:54:55.980317+0000","last_unstale":"2024-04-02T04:55:08.937210+0000","last_undegraded":"2024-04-02T04:55:08.937210+0000","last_fullsized":"2024-04-02T04:55:08.937210+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'31","last_scrub_stamp":"2024-04-02T04:54:12.211222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.211222+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:31.612007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.091256+0000","last_change":"2024-04-02T04:54:55.952119+0000","last_active":"2024-04-02T04:55:12.091256+0000","last_peered":"2024-04-02T04:55:12.091256+0000","last_clean":"2024-04-02T04:55:12.091256+0000","last_became_active":"2024-04-02T04:54:55.951772+0000","last_became_peered":"2024-04-02T04:54:55.951772+0000","last_unstale":"2024-04-02T04:55:12.091256+0000","last_undegraded":"2024-04-02T04:55:12.091256+0000","last_fullsized":"2024-04-02T04:55:12.091256+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.475462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.475462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.780326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":293,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933200+0000","last_change":"2024-04-02T04:54:08.209919+0000","last_active":"2024-04-02T04:55:08.933200+0000","last_peered":"2024-04-02T04:55:08.933200+0000","last_clean":"2024-04-02T04:55:08.933200+0000","last_became_active":"2024-04-02T04:53:54.934051+0000","last_became_peered":"2024-04-02T04:53:54.934051+0000","last_unstale":"2024-04-02T04:55:08.933200+0000","last_undegraded":"2024-04-02T04:55:08.933200+0000","last_fullsized":"2024-04-02T04:55:08.933200+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.209709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.209709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:27.901075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":682,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447868+0000","last_change":"2024-04-02T04:52:34.410043+0000","last_active":"2024-04-02T04:55:12.447868+0000","last_peered":"2024-04-02T04:55:12.447868+0000","last_clean":"2024-04-02T04:55:12.447868+0000","last_became_active":"2024-04-02T04:49:26.271155+0000","last_became_peered":"2024-04-02T04:49:26.271155+0000","last_unstale":"2024-04-02T04:55:12.447868+0000","last_undegraded":"2024-04-02T04:55:12.447868+0000","last_fullsized":"2024-04-02T04:55:12.447868+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.409934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.409934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":688,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101608+0000","last_change":"2024-04-02T04:54:55.966455+0000","last_active":"2024-04-02T04:55:12.101608+0000","last_peered":"2024-04-02T04:55:12.101608+0000","last_clean":"2024-04-02T04:55:12.101608+0000","last_became_active":"2024-04-02T04:54:55.966128+0000","last_became_peered":"2024-04-02T04:54:55.966128+0000","last_unstale":"2024-04-02T04:55:12.101608+0000","last_undegraded":"2024-04-02T04:55:12.101608+0000","last_fullsized":"2024-04-02T04:55:12.101608+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:54.055039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.055039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.780559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"402'42","reported_seq":886,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445941+0000","last_change":"2024-04-02T04:51:43.069937+0000","last_active":"2024-04-02T04:55:12.445941+0000","last_peered":"2024-04-02T04:55:12.445941+0000","last_clean":"2024-04-02T04:55:12.445941+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:55:12.445941+0000","last_undegraded":"2024-04-02T04:55:12.445941+0000","last_fullsized":"2024-04-02T04:55:12.445941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:51:43.069694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.069694+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":297,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933620+0000","last_change":"2024-04-02T04:54:55.986887+0000","last_active":"2024-04-02T04:55:08.933620+0000","last_peered":"2024-04-02T04:55:08.933620+0000","last_clean":"2024-04-02T04:55:08.933620+0000","last_became_active":"2024-04-02T04:54:55.986521+0000","last_became_peered":"2024-04-02T04:54:55.986521+0000","last_unstale":"2024-04-02T04:55:08.933620+0000","last_undegraded":"2024-04-02T04:55:08.933620+0000","last_fullsized":"2024-04-02T04:55:08.933620+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:59.105134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.105134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.059348+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":955,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081973+0000","last_change":"2024-04-02T04:50:15.516760+0000","last_active":"2024-04-02T04:55:12.081973+0000","last_peered":"2024-04-02T04:55:12.081973+0000","last_clean":"2024-04-02T04:55:12.081973+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:55:12.081973+0000","last_undegraded":"2024-04-02T04:55:12.081973+0000","last_fullsized":"2024-04-02T04:55:12.081973+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:50:15.516538+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:56.704379+0000","last_clean_scrub_stamp":"2024-04-02T04:50:15.516538+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919840+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":825,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.108855+0000","last_change":"2024-04-02T04:54:26.980826+0000","last_active":"2024-04-02T04:55:12.108855+0000","last_peered":"2024-04-02T04:55:12.108855+0000","last_clean":"2024-04-02T04:55:12.108855+0000","last_became_active":"2024-04-02T04:44:00.346778+0000","last_became_peered":"2024-04-02T04:44:00.346778+0000","last_unstale":"2024-04-02T04:55:12.108855+0000","last_undegraded":"2024-04-02T04:55:12.108855+0000","last_fullsized":"2024-04-02T04:55:12.108855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.980664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.980664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:32.762006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":691,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081544+0000","last_change":"2024-04-02T04:54:56.011513+0000","last_active":"2024-04-02T04:55:12.081544+0000","last_peered":"2024-04-02T04:55:12.081544+0000","last_clean":"2024-04-02T04:55:12.081544+0000","last_became_active":"2024-04-02T04:54:56.008639+0000","last_became_peered":"2024-04-02T04:54:56.008639+0000","last_unstale":"2024-04-02T04:55:12.081544+0000","last_undegraded":"2024-04-02T04:55:12.081544+0000","last_fullsized":"2024-04-02T04:55:12.081544+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:37.350152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:37.350152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117961+0000","last_change":"2024-04-02T04:52:42.269689+0000","last_active":"2024-04-02T04:55:12.117961+0000","last_peered":"2024-04-02T04:55:12.117961+0000","last_clean":"2024-04-02T04:55:12.117961+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:55:12.117961+0000","last_undegraded":"2024-04-02T04:55:12.117961+0000","last_fullsized":"2024-04-02T04:55:12.117961+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.268948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.268948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":808,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081207+0000","last_change":"2024-04-02T04:50:43.513008+0000","last_active":"2024-04-02T04:55:12.081207+0000","last_peered":"2024-04-02T04:55:12.081207+0000","last_clean":"2024-04-02T04:55:12.081207+0000","last_became_active":"2024-04-02T04:43:58.287270+0000","last_became_peered":"2024-04-02T04:43:58.287270+0000","last_unstale":"2024-04-02T04:55:12.081207+0000","last_undegraded":"2024-04-02T04:55:12.081207+0000","last_fullsized":"2024-04-02T04:55:12.081207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:43.512888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.512888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"402'26","reported_seq":866,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110982+0000","last_change":"2024-04-02T04:53:04.791017+0000","last_active":"2024-04-02T04:55:12.110982+0000","last_peered":"2024-04-02T04:55:12.110982+0000","last_clean":"2024-04-02T04:55:12.110982+0000","last_became_active":"2024-04-02T04:43:56.229106+0000","last_became_peered":"2024-04-02T04:43:56.229106+0000","last_unstale":"2024-04-02T04:55:12.110982+0000","last_undegraded":"2024-04-02T04:55:12.110982+0000","last_fullsized":"2024-04-02T04:55:12.110982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:53:04.790916+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.790916+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118040+0000","last_change":"2024-04-02T04:54:01.301186+0000","last_active":"2024-04-02T04:55:12.118040+0000","last_peered":"2024-04-02T04:55:12.118040+0000","last_clean":"2024-04-02T04:55:12.118040+0000","last_became_active":"2024-04-02T04:53:54.935402+0000","last_became_peered":"2024-04-02T04:53:54.935402+0000","last_unstale":"2024-04-02T04:55:12.118040+0000","last_undegraded":"2024-04-02T04:55:12.118040+0000","last_fullsized":"2024-04-02T04:55:12.118040+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.301076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.301076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.227215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":804,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082891+0000","last_change":"2024-04-02T04:50:18.501638+0000","last_active":"2024-04-02T04:55:12.082891+0000","last_peered":"2024-04-02T04:55:12.082891+0000","last_clean":"2024-04-02T04:55:12.082891+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:55:12.082891+0000","last_undegraded":"2024-04-02T04:55:12.082891+0000","last_fullsized":"2024-04-02T04:55:12.082891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:18.501528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:18.501528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"139'9","reported_seq":706,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081893+0000","last_change":"2024-04-02T04:54:56.019988+0000","last_active":"2024-04-02T04:55:12.081893+0000","last_peered":"2024-04-02T04:55:12.081893+0000","last_clean":"2024-04-02T04:55:12.081893+0000","last_became_active":"2024-04-02T04:54:55.995097+0000","last_became_peered":"2024-04-02T04:54:55.995097+0000","last_unstale":"2024-04-02T04:55:12.081893+0000","last_undegraded":"2024-04-02T04:55:12.081893+0000","last_fullsized":"2024-04-02T04:55:12.081893+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'9","last_scrub_stamp":"2024-04-02T04:51:29.324015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.324015+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":695,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079626+0000","last_change":"2024-04-02T04:54:09.648490+0000","last_active":"2024-04-02T04:55:12.079626+0000","last_peered":"2024-04-02T04:55:12.079626+0000","last_clean":"2024-04-02T04:55:12.079626+0000","last_became_active":"2024-04-02T04:53:54.916707+0000","last_became_peered":"2024-04-02T04:53:54.916707+0000","last_unstale":"2024-04-02T04:55:12.079626+0000","last_undegraded":"2024-04-02T04:55:12.079626+0000","last_fullsized":"2024-04-02T04:55:12.079626+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T04:54:09.648292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:09.648292+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:14.604754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"402'48","reported_seq":909,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.089454+0000","last_change":"2024-04-02T04:54:21.140672+0000","last_active":"2024-04-02T04:55:12.089454+0000","last_peered":"2024-04-02T04:55:12.089454+0000","last_clean":"2024-04-02T04:55:12.089454+0000","last_became_active":"2024-04-02T04:53:54.926993+0000","last_became_peered":"2024-04-02T04:53:54.926993+0000","last_unstale":"2024-04-02T04:55:12.089454+0000","last_undegraded":"2024-04-02T04:55:12.089454+0000","last_fullsized":"2024-04-02T04:55:12.089454+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'48","last_scrub_stamp":"2024-04-02T04:54:21.140559+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:54:21.140559+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:36.522536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":832,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118057+0000","last_change":"2024-04-02T04:50:44.352622+0000","last_active":"2024-04-02T04:55:12.118057+0000","last_peered":"2024-04-02T04:55:12.118057+0000","last_clean":"2024-04-02T04:55:12.118057+0000","last_became_active":"2024-04-02T04:45:53.585759+0000","last_became_peered":"2024-04-02T04:45:53.585759+0000","last_unstale":"2024-04-02T04:55:12.118057+0000","last_undegraded":"2024-04-02T04:55:12.118057+0000","last_fullsized":"2024-04-02T04:55:12.118057+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:44.352505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:44.352505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":763,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.109884+0000","last_change":"2024-04-02T04:53:07.785865+0000","last_active":"2024-04-02T04:55:12.109884+0000","last_peered":"2024-04-02T04:55:12.109884+0000","last_clean":"2024-04-02T04:55:12.109884+0000","last_became_active":"2024-04-02T04:45:53.850129+0000","last_became_peered":"2024-04-02T04:45:53.850129+0000","last_unstale":"2024-04-02T04:55:12.109884+0000","last_undegraded":"2024-04-02T04:55:12.109884+0000","last_fullsized":"2024-04-02T04:55:12.109884+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.785753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.785753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":44,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.121407+0000","last_change":"2024-04-02T04:54:55.957462+0000","last_active":"2024-04-02T04:55:12.121407+0000","last_peered":"2024-04-02T04:55:12.121407+0000","last_clean":"2024-04-02T04:55:12.121407+0000","last_became_active":"2024-04-02T04:54:55.956870+0000","last_became_peered":"2024-04-02T04:54:55.956870+0000","last_unstale":"2024-04-02T04:55:12.121407+0000","last_undegraded":"2024-04-02T04:55:12.121407+0000","last_fullsized":"2024-04-02T04:55:12.121407+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:03.659593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:53:03.659593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":176,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.940919+0000","last_change":"2024-04-02T04:54:24.235563+0000","last_active":"2024-04-02T04:55:08.940919+0000","last_peered":"2024-04-02T04:55:08.940919+0000","last_clean":"2024-04-02T04:55:08.940919+0000","last_became_active":"2024-04-02T04:53:54.924164+0000","last_became_peered":"2024-04-02T04:53:54.924164+0000","last_unstale":"2024-04-02T04:55:08.940919+0000","last_undegraded":"2024-04-02T04:55:08.940919+0000","last_fullsized":"2024-04-02T04:55:08.940919+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:24.235462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.235462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:43.571643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445508+0000","last_change":"2024-04-02T04:52:42.441055+0000","last_active":"2024-04-02T04:55:12.445508+0000","last_peered":"2024-04-02T04:55:12.445508+0000","last_clean":"2024-04-02T04:55:12.445508+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:55:12.445508+0000","last_undegraded":"2024-04-02T04:55:12.445508+0000","last_fullsized":"2024-04-02T04:55:12.445508+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.440852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.440852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"402'30","reported_seq":878,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079861+0000","last_change":"2024-04-02T04:54:03.566423+0000","last_active":"2024-04-02T04:55:12.079861+0000","last_peered":"2024-04-02T04:55:12.079861+0000","last_clean":"2024-04-02T04:55:12.079861+0000","last_became_active":"2024-04-02T04:53:54.917946+0000","last_became_peered":"2024-04-02T04:53:54.917946+0000","last_unstale":"2024-04-02T04:55:12.079861+0000","last_undegraded":"2024-04-02T04:55:12.079861+0000","last_fullsized":"2024-04-02T04:55:12.079861+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:54:03.566030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.566030+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.298004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":307,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.931579+0000","last_change":"2024-04-02T04:54:18.249066+0000","last_active":"2024-04-02T04:55:08.931579+0000","last_peered":"2024-04-02T04:55:08.931579+0000","last_clean":"2024-04-02T04:55:08.931579+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T04:55:08.931579+0000","last_undegraded":"2024-04-02T04:55:08.931579+0000","last_fullsized":"2024-04-02T04:55:08.931579+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T04:54:18.248880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:30.958122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":954,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082797+0000","last_change":"2024-04-02T04:50:31.493819+0000","last_active":"2024-04-02T04:55:12.082797+0000","last_peered":"2024-04-02T04:55:12.082797+0000","last_clean":"2024-04-02T04:55:12.082797+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:55:12.082797+0000","last_undegraded":"2024-04-02T04:55:12.082797+0000","last_fullsized":"2024-04-02T04:55:12.082797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:50:31.493715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:31.493715+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":821,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.096881+0000","last_change":"2024-04-02T04:54:55.986278+0000","last_active":"2024-04-02T04:55:12.096881+0000","last_peered":"2024-04-02T04:55:12.096881+0000","last_clean":"2024-04-02T04:55:12.096881+0000","last_became_active":"2024-04-02T04:54:55.985670+0000","last_became_peered":"2024-04-02T04:54:55.985670+0000","last_unstale":"2024-04-02T04:55:12.096881+0000","last_undegraded":"2024-04-02T04:55:12.096881+0000","last_fullsized":"2024-04-02T04:55:12.096881+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:29.998572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.998572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:31.622673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":703,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.104736+0000","last_change":"2024-04-02T04:54:55.982484+0000","last_active":"2024-04-02T04:55:12.104736+0000","last_peered":"2024-04-02T04:55:12.104736+0000","last_clean":"2024-04-02T04:55:12.104736+0000","last_became_active":"2024-04-02T04:54:55.936909+0000","last_became_peered":"2024-04-02T04:54:55.936909+0000","last_unstale":"2024-04-02T04:55:12.104736+0000","last_undegraded":"2024-04-02T04:55:12.104736+0000","last_fullsized":"2024-04-02T04:55:12.104736+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:52:53.668865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.668865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":677,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449797+0000","last_change":"2024-04-02T04:54:29.941796+0000","last_active":"2024-04-02T04:55:12.449797+0000","last_peered":"2024-04-02T04:55:12.449797+0000","last_clean":"2024-04-02T04:55:12.449797+0000","last_became_active":"2024-04-02T04:49:26.255429+0000","last_became_peered":"2024-04-02T04:49:26.255429+0000","last_unstale":"2024-04-02T04:55:12.449797+0000","last_undegraded":"2024-04-02T04:55:12.449797+0000","last_fullsized":"2024-04-02T04:55:12.449797+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.941615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:43.357114+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":815,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079988+0000","last_change":"2024-04-02T04:54:56.011408+0000","last_active":"2024-04-02T04:55:12.079988+0000","last_peered":"2024-04-02T04:55:12.079988+0000","last_clean":"2024-04-02T04:55:12.079988+0000","last_became_active":"2024-04-02T04:54:56.011012+0000","last_became_peered":"2024-04-02T04:54:56.011012+0000","last_unstale":"2024-04-02T04:55:12.079988+0000","last_undegraded":"2024-04-02T04:55:12.079988+0000","last_fullsized":"2024-04-02T04:55:12.079988+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.385759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"402'30","reported_seq":862,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112551+0000","last_change":"2024-04-02T04:53:07.411481+0000","last_active":"2024-04-02T04:55:12.112551+0000","last_peered":"2024-04-02T04:55:12.112551+0000","last_clean":"2024-04-02T04:55:12.112551+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:55:12.112551+0000","last_undegraded":"2024-04-02T04:55:12.112551+0000","last_fullsized":"2024-04-02T04:55:12.112551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'18","last_scrub_stamp":"2024-04-02T04:53:07.411390+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.411390+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449812+0000","last_change":"2024-04-02T04:54:08.978978+0000","last_active":"2024-04-02T04:55:12.449812+0000","last_peered":"2024-04-02T04:55:12.449812+0000","last_clean":"2024-04-02T04:55:12.449812+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T04:55:12.449812+0000","last_undegraded":"2024-04-02T04:55:12.449812+0000","last_fullsized":"2024-04-02T04:55:12.449812+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.978775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.978775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.387419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":1477,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.090120+0000","last_change":"2024-04-02T04:50:39.632326+0000","last_active":"2024-04-02T04:55:12.090120+0000","last_peered":"2024-04-02T04:55:12.090120+0000","last_clean":"2024-04-02T04:55:12.090120+0000","last_became_active":"2024-04-02T04:43:58.299645+0000","last_became_peered":"2024-04-02T04:43:58.299645+0000","last_unstale":"2024-04-02T04:55:12.090120+0000","last_undegraded":"2024-04-02T04:55:12.090120+0000","last_fullsized":"2024-04-02T04:55:12.090120+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:50:39.632171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.632171+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":655,"num_read_kb":650,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":260,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.937834+0000","last_change":"2024-04-02T04:53:55.052233+0000","last_active":"2024-04-02T04:55:08.937834+0000","last_peered":"2024-04-02T04:55:08.937834+0000","last_clean":"2024-04-02T04:55:08.937834+0000","last_became_active":"2024-04-02T04:53:54.936665+0000","last_became_peered":"2024-04-02T04:53:54.936665+0000","last_unstale":"2024-04-02T04:55:08.937834+0000","last_undegraded":"2024-04-02T04:55:08.937834+0000","last_fullsized":"2024-04-02T04:55:08.937834+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:55.052156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.052156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:23.649070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":55,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100312+0000","last_change":"2024-04-02T04:54:55.987745+0000","last_active":"2024-04-02T04:55:12.100312+0000","last_peered":"2024-04-02T04:55:12.100312+0000","last_clean":"2024-04-02T04:55:12.100312+0000","last_became_active":"2024-04-02T04:54:55.987407+0000","last_became_peered":"2024-04-02T04:54:55.987407+0000","last_unstale":"2024-04-02T04:55:12.100312+0000","last_undegraded":"2024-04-02T04:55:12.100312+0000","last_fullsized":"2024-04-02T04:55:12.100312+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:04.638640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.638640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":673,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082013+0000","last_change":"2024-04-02T04:54:28.733964+0000","last_active":"2024-04-02T04:55:12.082013+0000","last_peered":"2024-04-02T04:55:12.082013+0000","last_clean":"2024-04-02T04:55:12.082013+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:55:12.082013+0000","last_undegraded":"2024-04-02T04:55:12.082013+0000","last_fullsized":"2024-04-02T04:55:12.082013+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.733831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.733831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:56.853976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"395'14","reported_seq":978,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.089723+0000","last_change":"2024-04-02T04:53:56.111933+0000","last_active":"2024-04-02T04:55:12.089723+0000","last_peered":"2024-04-02T04:55:12.089723+0000","last_clean":"2024-04-02T04:55:12.089723+0000","last_became_active":"2024-04-02T04:53:54.935396+0000","last_became_peered":"2024-04-02T04:53:54.935396+0000","last_unstale":"2024-04-02T04:55:12.089723+0000","last_undegraded":"2024-04-02T04:55:12.089723+0000","last_fullsized":"2024-04-02T04:55:12.089723+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'14","last_scrub_stamp":"2024-04-02T04:53:56.111652+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:53:56.111652+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"402'23","reported_seq":870,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112015+0000","last_change":"2024-04-02T04:54:17.131387+0000","last_active":"2024-04-02T04:55:12.112015+0000","last_peered":"2024-04-02T04:55:12.112015+0000","last_clean":"2024-04-02T04:55:12.112015+0000","last_became_active":"2024-04-02T04:53:54.912078+0000","last_became_peered":"2024-04-02T04:53:54.912078+0000","last_unstale":"2024-04-02T04:55:12.112015+0000","last_undegraded":"2024-04-02T04:55:12.112015+0000","last_fullsized":"2024-04-02T04:55:12.112015+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'23","last_scrub_stamp":"2024-04-02T04:54:17.131283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.131283+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:46.719560+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120080+0000","last_change":"2024-04-02T04:50:43.360677+0000","last_active":"2024-04-02T04:55:12.120080+0000","last_peered":"2024-04-02T04:55:12.120080+0000","last_clean":"2024-04-02T04:55:12.120080+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:55:12.120080+0000","last_undegraded":"2024-04-02T04:55:12.120080+0000","last_fullsized":"2024-04-02T04:55:12.120080+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:50:43.360457+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:50:43.360457+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":825,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079721+0000","last_change":"2024-04-02T04:54:05.619856+0000","last_active":"2024-04-02T04:55:12.079721+0000","last_peered":"2024-04-02T04:55:12.079721+0000","last_clean":"2024-04-02T04:55:12.079721+0000","last_became_active":"2024-04-02T04:53:54.917485+0000","last_became_peered":"2024-04-02T04:53:54.917485+0000","last_unstale":"2024-04-02T04:55:12.079721+0000","last_undegraded":"2024-04-02T04:55:12.079721+0000","last_fullsized":"2024-04-02T04:55:12.079721+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.619646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.619646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":805,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080249+0000","last_change":"2024-04-02T04:50:29.535635+0000","last_active":"2024-04-02T04:55:12.080249+0000","last_peered":"2024-04-02T04:55:12.080249+0000","last_clean":"2024-04-02T04:55:12.080249+0000","last_became_active":"2024-04-02T04:44:00.334316+0000","last_became_peered":"2024-04-02T04:44:00.334316+0000","last_unstale":"2024-04-02T04:55:12.080249+0000","last_undegraded":"2024-04-02T04:55:12.080249+0000","last_fullsized":"2024-04-02T04:55:12.080249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:29.535314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:29.535314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"139'10","reported_seq":707,"reported_epoch":464,"state":"clean+premerge+peered","last_fresh":"2024-04-02T04:55:12.117005+0000","last_change":"2024-04-02T04:55:12.117005+0000","last_active":"2024-04-02T04:55:11.085061+0000","last_peered":"2024-04-02T04:55:12.117005+0000","last_clean":"2024-04-02T04:55:12.117005+0000","last_became_active":"2024-04-02T04:49:26.341161+0000","last_became_peered":"2024-04-02T04:55:12.116616+0000","last_unstale":"2024-04-02T04:55:12.117005+0000","last_undegraded":"2024-04-02T04:55:12.117005+0000","last_fullsized":"2024-04-02T04:55:12.117005+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:52:55.613396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.613396+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":677,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448672+0000","last_change":"2024-04-02T04:54:26.868758+0000","last_active":"2024-04-02T04:55:12.448672+0000","last_peered":"2024-04-02T04:55:12.448672+0000","last_clean":"2024-04-02T04:55:12.448672+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:55:12.448672+0000","last_undegraded":"2024-04-02T04:55:12.448672+0000","last_fullsized":"2024-04-02T04:55:12.448672+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.868528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.868528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:43.857038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"451'15","reported_seq":976,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103289+0000","last_change":"2024-04-02T04:54:55.969766+0000","last_active":"2024-04-02T04:55:12.103289+0000","last_peered":"2024-04-02T04:55:12.103289+0000","last_clean":"2024-04-02T04:55:12.103289+0000","last_became_active":"2024-04-02T04:54:55.961735+0000","last_became_peered":"2024-04-02T04:54:55.961735+0000","last_unstale":"2024-04-02T04:55:12.103289+0000","last_undegraded":"2024-04-02T04:55:12.103289+0000","last_fullsized":"2024-04-02T04:55:12.103289+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:49:08.963583+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:49:08.963583+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"451'60","reported_seq":1087,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120765+0000","last_change":"2024-04-02T04:54:55.984048+0000","last_active":"2024-04-02T04:55:12.120765+0000","last_peered":"2024-04-02T04:55:12.120765+0000","last_clean":"2024-04-02T04:55:12.120765+0000","last_became_active":"2024-04-02T04:54:55.955778+0000","last_became_peered":"2024-04-02T04:54:55.955778+0000","last_unstale":"2024-04-02T04:55:12.120765+0000","last_undegraded":"2024-04-02T04:55:12.120765+0000","last_fullsized":"2024-04-02T04:55:12.120765+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:52:51.249489+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.249489+0000","objects_scrubbed":11,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":295,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936818+0000","last_change":"2024-04-02T04:54:17.219530+0000","last_active":"2024-04-02T04:55:08.936818+0000","last_peered":"2024-04-02T04:55:08.936818+0000","last_clean":"2024-04-02T04:55:08.936818+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T04:55:08.936818+0000","last_undegraded":"2024-04-02T04:55:08.936818+0000","last_fullsized":"2024-04-02T04:55:08.936818+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:17.219353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.219353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:33.338734+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":833,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445929+0000","last_change":"2024-04-02T04:54:56.024756+0000","last_active":"2024-04-02T04:55:12.445929+0000","last_peered":"2024-04-02T04:55:12.445929+0000","last_clean":"2024-04-02T04:55:12.445929+0000","last_became_active":"2024-04-02T04:54:56.004769+0000","last_became_peered":"2024-04-02T04:54:56.004769+0000","last_unstale":"2024-04-02T04:55:12.445929+0000","last_undegraded":"2024-04-02T04:55:12.445929+0000","last_fullsized":"2024-04-02T04:55:12.445929+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:51:34.006018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.006018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":814,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118612+0000","last_change":"2024-04-02T04:53:53.356428+0000","last_active":"2024-04-02T04:55:12.118612+0000","last_peered":"2024-04-02T04:55:12.118612+0000","last_clean":"2024-04-02T04:55:12.118612+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:55:12.118612+0000","last_undegraded":"2024-04-02T04:55:12.118612+0000","last_fullsized":"2024-04-02T04:55:12.118612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:53.356300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:53:53.356300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":673,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.098650+0000","last_change":"2024-04-02T04:53:54.958148+0000","last_active":"2024-04-02T04:55:12.098650+0000","last_peered":"2024-04-02T04:55:12.098650+0000","last_clean":"2024-04-02T04:55:12.098650+0000","last_became_active":"2024-04-02T04:53:54.957413+0000","last_became_peered":"2024-04-02T04:53:54.957413+0000","last_unstale":"2024-04-02T04:55:12.098650+0000","last_undegraded":"2024-04-02T04:55:12.098650+0000","last_fullsized":"2024-04-02T04:55:12.098650+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112929+0000","last_change":"2024-04-02T04:54:55.954958+0000","last_active":"2024-04-02T04:55:12.112929+0000","last_peered":"2024-04-02T04:55:12.112929+0000","last_clean":"2024-04-02T04:55:12.112929+0000","last_became_active":"2024-04-02T04:54:55.950537+0000","last_became_peered":"2024-04-02T04:54:55.950537+0000","last_unstale":"2024-04-02T04:55:12.112929+0000","last_undegraded":"2024-04-02T04:55:12.112929+0000","last_fullsized":"2024-04-02T04:55:12.112929+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.310943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.310943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":293,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936528+0000","last_change":"2024-04-02T04:54:19.199301+0000","last_active":"2024-04-02T04:55:08.936528+0000","last_peered":"2024-04-02T04:55:08.936528+0000","last_clean":"2024-04-02T04:55:08.936528+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T04:55:08.936528+0000","last_undegraded":"2024-04-02T04:55:08.936528+0000","last_fullsized":"2024-04-02T04:55:08.936528+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.199212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.199212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:39.344910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"402'34","reported_seq":874,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101672+0000","last_change":"2024-04-02T04:52:33.272616+0000","last_active":"2024-04-02T04:55:12.101672+0000","last_peered":"2024-04-02T04:55:12.101672+0000","last_clean":"2024-04-02T04:55:12.101672+0000","last_became_active":"2024-04-02T04:43:56.247678+0000","last_became_peered":"2024-04-02T04:43:56.247678+0000","last_unstale":"2024-04-02T04:55:12.101672+0000","last_undegraded":"2024-04-02T04:55:12.101672+0000","last_fullsized":"2024-04-02T04:55:12.101672+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:52:33.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.272448+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":820,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120373+0000","last_change":"2024-04-02T04:54:55.968790+0000","last_active":"2024-04-02T04:55:12.120373+0000","last_peered":"2024-04-02T04:55:12.120373+0000","last_clean":"2024-04-02T04:55:12.120373+0000","last_became_active":"2024-04-02T04:54:55.968410+0000","last_became_peered":"2024-04-02T04:54:55.968410+0000","last_unstale":"2024-04-02T04:55:12.120373+0000","last_undegraded":"2024-04-02T04:55:12.120373+0000","last_fullsized":"2024-04-02T04:55:12.120373+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:26.342429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:26.342429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.f","version":"139'5","reported_seq":598,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.941696+0000","last_change":"2024-04-02T04:54:53.941538+0000","last_active":"2024-04-02T04:55:08.941696+0000","last_peered":"2024-04-02T04:55:08.941696+0000","last_clean":"2024-04-02T04:55:08.941696+0000","last_became_active":"2024-04-02T04:54:53.941138+0000","last_became_peered":"2024-04-02T04:54:53.941138+0000","last_unstale":"2024-04-02T04:55:08.941696+0000","last_undegraded":"2024-04-02T04:55:08.941696+0000","last_fullsized":"2024-04-02T04:55:08.941696+0000","mapping_epoch":439,"log_start":"139'5","ondisk_log_start":"139'5","created":135,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.965190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.965190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":5,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.012701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":677,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447148+0000","last_change":"2024-04-02T04:54:20.899430+0000","last_active":"2024-04-02T04:55:12.447148+0000","last_peered":"2024-04-02T04:55:12.447148+0000","last_clean":"2024-04-02T04:55:12.447148+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:55:12.447148+0000","last_undegraded":"2024-04-02T04:55:12.447148+0000","last_fullsized":"2024-04-02T04:55:12.447148+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:20.898842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:20.898842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:25.716894+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448826+0000","last_change":"2024-04-02T04:54:00.910076+0000","last_active":"2024-04-02T04:55:12.448826+0000","last_peered":"2024-04-02T04:55:12.448826+0000","last_clean":"2024-04-02T04:55:12.448826+0000","last_became_active":"2024-04-02T04:53:54.934952+0000","last_became_peered":"2024-04-02T04:53:54.934952+0000","last_unstale":"2024-04-02T04:55:12.448826+0000","last_undegraded":"2024-04-02T04:55:12.448826+0000","last_fullsized":"2024-04-02T04:55:12.448826+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.909025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.909025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:19.903732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":555,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932771+0000","last_change":"2024-04-02T04:53:54.947927+0000","last_active":"2024-04-02T04:55:08.932771+0000","last_peered":"2024-04-02T04:55:08.932771+0000","last_clean":"2024-04-02T04:55:08.932771+0000","last_became_active":"2024-04-02T04:53:54.946098+0000","last_became_peered":"2024-04-02T04:53:54.946098+0000","last_unstale":"2024-04-02T04:55:08.932771+0000","last_undegraded":"2024-04-02T04:55:08.932771+0000","last_fullsized":"2024-04-02T04:55:08.932771+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:49:07.344606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:49:07.344606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":713,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102042+0000","last_change":"2024-04-02T04:54:55.932244+0000","last_active":"2024-04-02T04:55:12.102042+0000","last_peered":"2024-04-02T04:55:12.102042+0000","last_clean":"2024-04-02T04:55:12.102042+0000","last_became_active":"2024-04-02T04:54:55.931902+0000","last_became_peered":"2024-04-02T04:54:55.931902+0000","last_unstale":"2024-04-02T04:55:12.102042+0000","last_undegraded":"2024-04-02T04:55:12.102042+0000","last_fullsized":"2024-04-02T04:55:12.102042+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:55.758993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.758993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.389359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"402'26","reported_seq":878,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.093999+0000","last_change":"2024-04-02T04:54:56.031589+0000","last_active":"2024-04-02T04:55:12.093999+0000","last_peered":"2024-04-02T04:55:12.093999+0000","last_clean":"2024-04-02T04:55:12.093999+0000","last_became_active":"2024-04-02T04:54:55.985985+0000","last_became_peered":"2024-04-02T04:54:55.985985+0000","last_unstale":"2024-04-02T04:55:12.093999+0000","last_undegraded":"2024-04-02T04:55:12.093999+0000","last_fullsized":"2024-04-02T04:55:12.093999+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:49.756586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.756586+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":44,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.111611+0000","last_change":"2024-04-02T04:54:55.971470+0000","last_active":"2024-04-02T04:55:12.111611+0000","last_peered":"2024-04-02T04:55:12.111611+0000","last_clean":"2024-04-02T04:55:12.111611+0000","last_became_active":"2024-04-02T04:54:55.971083+0000","last_became_peered":"2024-04-02T04:54:55.971083+0000","last_unstale":"2024-04-02T04:55:12.111611+0000","last_undegraded":"2024-04-02T04:55:12.111611+0000","last_fullsized":"2024-04-02T04:55:12.111611+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:56.577226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:51:56.577226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":80,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447527+0000","last_change":"2024-04-02T04:55:05.956631+0000","last_active":"2024-04-02T04:55:12.447527+0000","last_peered":"2024-04-02T04:55:12.447527+0000","last_clean":"2024-04-02T04:55:12.447527+0000","last_became_active":"2024-04-02T04:55:05.955924+0000","last_became_peered":"2024-04-02T04:55:05.955924+0000","last_unstale":"2024-04-02T04:55:12.447527+0000","last_undegraded":"2024-04-02T04:55:12.447527+0000","last_fullsized":"2024-04-02T04:55:12.447527+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:55.332906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:55.332906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":571,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110370+0000","last_change":"2024-04-02T04:54:55.947450+0000","last_active":"2024-04-02T04:55:12.110370+0000","last_peered":"2024-04-02T04:55:12.110370+0000","last_clean":"2024-04-02T04:55:12.110370+0000","last_became_active":"2024-04-02T04:54:55.946927+0000","last_became_peered":"2024-04-02T04:54:55.946927+0000","last_unstale":"2024-04-02T04:55:12.110370+0000","last_undegraded":"2024-04-02T04:55:12.110370+0000","last_fullsized":"2024-04-02T04:55:12.110370+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:14.557167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:14.557167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:38.095260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.119995+0000","last_change":"2024-04-02T04:54:55.961082+0000","last_active":"2024-04-02T04:55:12.119995+0000","last_peered":"2024-04-02T04:55:12.119995+0000","last_clean":"2024-04-02T04:55:12.119995+0000","last_became_active":"2024-04-02T04:54:55.960718+0000","last_became_peered":"2024-04-02T04:54:55.960718+0000","last_unstale":"2024-04-02T04:55:12.119995+0000","last_undegraded":"2024-04-02T04:55:12.119995+0000","last_fullsized":"2024-04-02T04:55:12.119995+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.385111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.385111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110416+0000","last_change":"2024-04-02T04:52:46.569670+0000","last_active":"2024-04-02T04:55:12.110416+0000","last_peered":"2024-04-02T04:55:12.110416+0000","last_clean":"2024-04-02T04:55:12.110416+0000","last_became_active":"2024-04-02T04:43:54.215318+0000","last_became_peered":"2024-04-02T04:43:54.215318+0000","last_unstale":"2024-04-02T04:55:12.110416+0000","last_undegraded":"2024-04-02T04:55:12.110416+0000","last_fullsized":"2024-04-02T04:55:12.110416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.569560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.569560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"402'42","reported_seq":896,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.116319+0000","last_change":"2024-04-02T04:52:49.180778+0000","last_active":"2024-04-02T04:55:12.116319+0000","last_peered":"2024-04-02T04:55:12.116319+0000","last_clean":"2024-04-02T04:55:12.116319+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:55:12.116319+0000","last_undegraded":"2024-04-02T04:55:12.116319+0000","last_fullsized":"2024-04-02T04:55:12.116319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-02T04:52:49.180521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.180521+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":260,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936933+0000","last_change":"2024-04-02T04:53:56.024607+0000","last_active":"2024-04-02T04:55:08.936933+0000","last_peered":"2024-04-02T04:55:08.936933+0000","last_clean":"2024-04-02T04:55:08.936933+0000","last_became_active":"2024-04-02T04:53:54.936793+0000","last_became_peered":"2024-04-02T04:53:54.936793+0000","last_unstale":"2024-04-02T04:55:08.936933+0000","last_undegraded":"2024-04-02T04:55:08.936933+0000","last_fullsized":"2024-04-02T04:55:08.936933+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:56.024450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:56.024450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.439247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":104,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081870+0000","last_change":"2024-04-02T04:55:09.938372+0000","last_active":"2024-04-02T04:55:12.081870+0000","last_peered":"2024-04-02T04:55:12.081870+0000","last_clean":"2024-04-02T04:55:12.081870+0000","last_became_active":"2024-04-02T04:54:58.218406+0000","last_became_peered":"2024-04-02T04:54:58.218406+0000","last_unstale":"2024-04-02T04:55:12.081870+0000","last_undegraded":"2024-04-02T04:55:12.081870+0000","last_fullsized":"2024-04-02T04:55:12.081870+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:09.938280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:09.938280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:31.669105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":683,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445634+0000","last_change":"2024-04-02T04:54:56.005135+0000","last_active":"2024-04-02T04:55:12.445634+0000","last_peered":"2024-04-02T04:55:12.445634+0000","last_clean":"2024-04-02T04:55:12.445634+0000","last_became_active":"2024-04-02T04:54:56.004256+0000","last_became_peered":"2024-04-02T04:54:56.004256+0000","last_unstale":"2024-04-02T04:55:12.445634+0000","last_undegraded":"2024-04-02T04:55:12.445634+0000","last_fullsized":"2024-04-02T04:55:12.445634+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.854283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:56.391369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":818,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120499+0000","last_change":"2024-04-02T04:51:34.344914+0000","last_active":"2024-04-02T04:55:12.120499+0000","last_peered":"2024-04-02T04:55:12.120499+0000","last_clean":"2024-04-02T04:55:12.120499+0000","last_became_active":"2024-04-02T04:43:58.311739+0000","last_became_peered":"2024-04-02T04:43:58.311739+0000","last_unstale":"2024-04-02T04:55:12.120499+0000","last_undegraded":"2024-04-02T04:55:12.120499+0000","last_fullsized":"2024-04-02T04:55:12.120499+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:34.344554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:51:34.344554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'35","reported_seq":876,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081397+0000","last_change":"2024-04-02T04:50:12.546833+0000","last_active":"2024-04-02T04:55:12.081397+0000","last_peered":"2024-04-02T04:55:12.081397+0000","last_clean":"2024-04-02T04:55:12.081397+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:55:12.081397+0000","last_undegraded":"2024-04-02T04:55:12.081397+0000","last_fullsized":"2024-04-02T04:55:12.081397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-02T04:50:12.546683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:12.546683+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117832+0000","last_change":"2024-04-02T04:54:55.969695+0000","last_active":"2024-04-02T04:55:12.117832+0000","last_peered":"2024-04-02T04:55:12.117832+0000","last_clean":"2024-04-02T04:55:12.117832+0000","last_became_active":"2024-04-02T04:54:55.969352+0000","last_became_peered":"2024-04-02T04:54:55.969352+0000","last_unstale":"2024-04-02T04:55:12.117832+0000","last_undegraded":"2024-04-02T04:55:12.117832+0000","last_fullsized":"2024-04-02T04:55:12.117832+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.578411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.578411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100781+0000","last_change":"2024-04-02T04:53:58.149905+0000","last_active":"2024-04-02T04:55:12.100781+0000","last_peered":"2024-04-02T04:55:12.100781+0000","last_clean":"2024-04-02T04:55:12.100781+0000","last_became_active":"2024-04-02T04:53:54.934877+0000","last_became_peered":"2024-04-02T04:53:54.934877+0000","last_unstale":"2024-04-02T04:55:12.100781+0000","last_undegraded":"2024-04-02T04:55:12.100781+0000","last_fullsized":"2024-04-02T04:55:12.100781+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:53:58.149611+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:53:58.149611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.d","version":"139'7","reported_seq":729,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081265+0000","last_change":"2024-04-02T04:55:02.026913+0000","last_active":"2024-04-02T04:55:12.081265+0000","last_peered":"2024-04-02T04:55:12.081265+0000","last_clean":"2024-04-02T04:55:12.081265+0000","last_became_active":"2024-04-02T04:55:02.020819+0000","last_became_peered":"2024-04-02T04:55:02.020819+0000","last_unstale":"2024-04-02T04:55:12.081265+0000","last_undegraded":"2024-04-02T04:55:12.081265+0000","last_fullsized":"2024-04-02T04:55:12.081265+0000","mapping_epoch":447,"log_start":"139'7","ondisk_log_start":"139'7","created":135,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:54:12.621186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.621186+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:26.687187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"139'10","reported_seq":93,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101715+0000","last_change":"2024-04-02T04:55:11.124645+0000","last_active":"2024-04-02T04:55:12.101715+0000","last_peered":"2024-04-02T04:55:12.101715+0000","last_clean":"2024-04-02T04:55:12.101715+0000","last_became_active":"2024-04-02T04:55:09.936089+0000","last_became_peered":"2024-04-02T04:55:09.936089+0000","last_unstale":"2024-04-02T04:55:12.101715+0000","last_undegraded":"2024-04-02T04:55:12.101715+0000","last_fullsized":"2024-04-02T04:55:12.101715+0000","mapping_epoch":439,"log_start":"139'10","ondisk_log_start":"139'10","created":135,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10","last_scrub_stamp":"2024-04-02T04:55:11.124551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:11.124551+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:30.690130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":156,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936875+0000","last_change":"2024-04-02T04:53:54.943618+0000","last_active":"2024-04-02T04:55:08.936875+0000","last_peered":"2024-04-02T04:55:08.936875+0000","last_clean":"2024-04-02T04:55:08.936875+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T04:55:08.936875+0000","last_undegraded":"2024-04-02T04:55:08.936875+0000","last_fullsized":"2024-04-02T04:55:08.936875+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118267+0000","last_change":"2024-04-02T04:54:55.979795+0000","last_active":"2024-04-02T04:55:12.118267+0000","last_peered":"2024-04-02T04:55:12.118267+0000","last_clean":"2024-04-02T04:55:12.118267+0000","last_became_active":"2024-04-02T04:54:55.979416+0000","last_became_peered":"2024-04-02T04:54:55.979416+0000","last_unstale":"2024-04-02T04:55:12.118267+0000","last_undegraded":"2024-04-02T04:55:12.118267+0000","last_fullsized":"2024-04-02T04:55:12.118267+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.308301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.308301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"451'29","reported_seq":920,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102426+0000","last_change":"2024-04-02T04:54:56.032906+0000","last_active":"2024-04-02T04:55:12.102426+0000","last_peered":"2024-04-02T04:55:12.102426+0000","last_clean":"2024-04-02T04:55:12.102426+0000","last_became_active":"2024-04-02T04:54:55.934119+0000","last_became_peered":"2024-04-02T04:54:55.934119+0000","last_unstale":"2024-04-02T04:55:12.102426+0000","last_undegraded":"2024-04-02T04:55:12.102426+0000","last_fullsized":"2024-04-02T04:55:12.102426+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-02T04:51:35.725768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:35.725768+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":843,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120580+0000","last_change":"2024-04-02T04:53:59.321337+0000","last_active":"2024-04-02T04:55:12.120580+0000","last_peered":"2024-04-02T04:55:12.120580+0000","last_clean":"2024-04-02T04:55:12.120580+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T04:55:12.120580+0000","last_undegraded":"2024-04-02T04:55:12.120580+0000","last_fullsized":"2024-04-02T04:55:12.120580+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:53:59.321220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.321220+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.462072+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":820,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.092158+0000","last_change":"2024-04-02T04:53:11.307355+0000","last_active":"2024-04-02T04:55:12.092158+0000","last_peered":"2024-04-02T04:55:12.092158+0000","last_clean":"2024-04-02T04:55:12.092158+0000","last_became_active":"2024-04-02T04:44:00.344860+0000","last_became_peered":"2024-04-02T04:44:00.344860+0000","last_unstale":"2024-04-02T04:55:12.092158+0000","last_undegraded":"2024-04-02T04:55:12.092158+0000","last_fullsized":"2024-04-02T04:55:12.092158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.307248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.307248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"139'5","reported_seq":175,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933251+0000","last_change":"2024-04-02T04:53:54.932181+0000","last_active":"2024-04-02T04:55:08.933251+0000","last_peered":"2024-04-02T04:55:08.933251+0000","last_clean":"2024-04-02T04:55:08.933251+0000","last_became_active":"2024-04-02T04:53:54.931833+0000","last_became_peered":"2024-04-02T04:53:54.931833+0000","last_unstale":"2024-04-02T04:55:08.933251+0000","last_undegraded":"2024-04-02T04:55:08.933251+0000","last_fullsized":"2024-04-02T04:55:08.933251+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449700+0000","last_change":"2024-04-02T04:54:55.991572+0000","last_active":"2024-04-02T04:55:12.449700+0000","last_peered":"2024-04-02T04:55:12.449700+0000","last_clean":"2024-04-02T04:55:12.449700+0000","last_became_active":"2024-04-02T04:54:55.991180+0000","last_became_peered":"2024-04-02T04:54:55.991180+0000","last_unstale":"2024-04-02T04:55:12.449700+0000","last_undegraded":"2024-04-02T04:55:12.449700+0000","last_fullsized":"2024-04-02T04:55:12.449700+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.364241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.364241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"121'12","reported_seq":964,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079769+0000","last_change":"2024-04-02T04:53:04.380937+0000","last_active":"2024-04-02T04:55:12.079769+0000","last_peered":"2024-04-02T04:55:12.079769+0000","last_clean":"2024-04-02T04:55:12.079769+0000","last_became_active":"2024-04-02T04:43:58.296764+0000","last_became_peered":"2024-04-02T04:43:58.296764+0000","last_unstale":"2024-04-02T04:55:12.079769+0000","last_undegraded":"2024-04-02T04:55:12.079769+0000","last_fullsized":"2024-04-02T04:55:12.079769+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:53:04.380713+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.380713+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"402'25","reported_seq":336,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.937613+0000","last_change":"2024-04-02T04:53:54.929360+0000","last_active":"2024-04-02T04:55:08.937613+0000","last_peered":"2024-04-02T04:55:08.937613+0000","last_clean":"2024-04-02T04:55:08.937613+0000","last_became_active":"2024-04-02T04:53:54.927132+0000","last_became_peered":"2024-04-02T04:53:54.927132+0000","last_unstale":"2024-04-02T04:55:08.937613+0000","last_undegraded":"2024-04-02T04:55:08.937613+0000","last_fullsized":"2024-04-02T04:55:08.937613+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:49:04.429318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:04.429318+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":850,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082869+0000","last_change":"2024-04-02T04:53:59.519788+0000","last_active":"2024-04-02T04:55:12.082869+0000","last_peered":"2024-04-02T04:55:12.082869+0000","last_clean":"2024-04-02T04:55:12.082869+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T04:55:12.082869+0000","last_undegraded":"2024-04-02T04:55:12.082869+0000","last_fullsized":"2024-04-02T04:55:12.082869+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:53:59.519530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.519530+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":817,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.111191+0000","last_change":"2024-04-02T04:54:01.088340+0000","last_active":"2024-04-02T04:55:12.111191+0000","last_peered":"2024-04-02T04:55:12.111191+0000","last_clean":"2024-04-02T04:55:12.111191+0000","last_became_active":"2024-04-02T04:53:54.944662+0000","last_became_peered":"2024-04-02T04:53:54.944662+0000","last_unstale":"2024-04-02T04:55:12.111191+0000","last_undegraded":"2024-04-02T04:55:12.111191+0000","last_fullsized":"2024-04-02T04:55:12.111191+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.088131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.088131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":679,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.105202+0000","last_change":"2024-04-02T04:53:09.820846+0000","last_active":"2024-04-02T04:55:12.105202+0000","last_peered":"2024-04-02T04:55:12.105202+0000","last_clean":"2024-04-02T04:55:12.105202+0000","last_became_active":"2024-04-02T04:49:26.292029+0000","last_became_peered":"2024-04-02T04:49:26.292029+0000","last_unstale":"2024-04-02T04:55:12.105202+0000","last_undegraded":"2024-04-02T04:55:12.105202+0000","last_fullsized":"2024-04-02T04:55:12.105202+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:09.820697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:09.820697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"402'25","reported_seq":326,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.933056+0000","last_change":"2024-04-02T04:53:54.923643+0000","last_active":"2024-04-02T04:55:08.933056+0000","last_peered":"2024-04-02T04:55:08.933056+0000","last_clean":"2024-04-02T04:55:08.933056+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T04:55:08.933056+0000","last_undegraded":"2024-04-02T04:55:08.933056+0000","last_fullsized":"2024-04-02T04:55:08.933056+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:48:56.339802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:48:56.339802+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.105449+0000","last_change":"2024-04-02T04:52:50.652084+0000","last_active":"2024-04-02T04:55:12.105449+0000","last_peered":"2024-04-02T04:55:12.105449+0000","last_clean":"2024-04-02T04:55:12.105449+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:55:12.105449+0000","last_undegraded":"2024-04-02T04:55:12.105449+0000","last_fullsized":"2024-04-02T04:55:12.105449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:50.651865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:52:50.651865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.113212+0000","last_change":"2024-04-02T04:54:35.398791+0000","last_active":"2024-04-02T04:55:12.113212+0000","last_peered":"2024-04-02T04:55:12.113212+0000","last_clean":"2024-04-02T04:55:12.113212+0000","last_became_active":"2024-04-02T04:53:54.956221+0000","last_became_peered":"2024-04-02T04:53:54.956221+0000","last_unstale":"2024-04-02T04:55:12.113212+0000","last_undegraded":"2024-04-02T04:55:12.113212+0000","last_fullsized":"2024-04-02T04:55:12.113212+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_clean_scrub_stamp":"2024-04-02T04:54:35.398676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:38.063870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":810,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.091632+0000","last_change":"2024-04-02T04:54:01.922789+0000","last_active":"2024-04-02T04:55:12.091632+0000","last_peered":"2024-04-02T04:55:12.091632+0000","last_clean":"2024-04-02T04:55:12.091632+0000","last_became_active":"2024-04-02T04:53:54.925132+0000","last_became_peered":"2024-04-02T04:53:54.925132+0000","last_unstale":"2024-04-02T04:55:12.091632+0000","last_undegraded":"2024-04-02T04:55:12.091632+0000","last_fullsized":"2024-04-02T04:55:12.091632+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.922344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.922344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:15.507416+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117411+0000","last_change":"2024-04-02T04:51:29.319201+0000","last_active":"2024-04-02T04:55:12.117411+0000","last_peered":"2024-04-02T04:55:12.117411+0000","last_clean":"2024-04-02T04:55:12.117411+0000","last_became_active":"2024-04-02T04:49:26.258674+0000","last_became_peered":"2024-04-02T04:49:26.258674+0000","last_unstale":"2024-04-02T04:55:12.117411+0000","last_undegraded":"2024-04-02T04:55:12.117411+0000","last_fullsized":"2024-04-02T04:55:12.117411+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:29.318999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.318999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"402'8","reported_seq":841,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445011+0000","last_change":"2024-04-02T04:54:56.022677+0000","last_active":"2024-04-02T04:55:12.445011+0000","last_peered":"2024-04-02T04:55:12.445011+0000","last_clean":"2024-04-02T04:55:12.445011+0000","last_became_active":"2024-04-02T04:54:55.987132+0000","last_became_peered":"2024-04-02T04:54:55.987132+0000","last_unstale":"2024-04-02T04:55:12.445011+0000","last_undegraded":"2024-04-02T04:55:12.445011+0000","last_fullsized":"2024-04-02T04:55:12.445011+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'8","last_scrub_stamp":"2024-04-02T04:54:17.927334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.927334+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:44.851896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":821,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445909+0000","last_change":"2024-04-02T04:51:40.072460+0000","last_active":"2024-04-02T04:55:12.445909+0000","last_peered":"2024-04-02T04:55:12.445909+0000","last_clean":"2024-04-02T04:55:12.445909+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:55:12.445909+0000","last_undegraded":"2024-04-02T04:55:12.445909+0000","last_fullsized":"2024-04-02T04:55:12.445909+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:40.072262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:51:40.072262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":826,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112369+0000","last_change":"2024-04-02T04:54:16.087021+0000","last_active":"2024-04-02T04:55:12.112369+0000","last_peered":"2024-04-02T04:55:12.112369+0000","last_clean":"2024-04-02T04:55:12.112369+0000","last_became_active":"2024-04-02T04:53:54.920683+0000","last_became_peered":"2024-04-02T04:53:54.920683+0000","last_unstale":"2024-04-02T04:55:12.112369+0000","last_undegraded":"2024-04-02T04:55:12.112369+0000","last_fullsized":"2024-04-02T04:55:12.112369+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:16.086780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:54:16.086780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:24.987481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":283,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936546+0000","last_change":"2024-04-02T04:54:03.204796+0000","last_active":"2024-04-02T04:55:08.936546+0000","last_peered":"2024-04-02T04:55:08.936546+0000","last_clean":"2024-04-02T04:55:08.936546+0000","last_became_active":"2024-04-02T04:53:54.933037+0000","last_became_peered":"2024-04-02T04:53:54.933037+0000","last_unstale":"2024-04-02T04:55:08.936546+0000","last_undegraded":"2024-04-02T04:55:08.936546+0000","last_fullsized":"2024-04-02T04:55:08.936546+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.204677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.204677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.899180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445071+0000","last_change":"2024-04-02T04:54:56.001623+0000","last_active":"2024-04-02T04:55:12.445071+0000","last_peered":"2024-04-02T04:55:12.445071+0000","last_clean":"2024-04-02T04:55:12.445071+0000","last_became_active":"2024-04-02T04:54:56.001224+0000","last_became_peered":"2024-04-02T04:54:56.001224+0000","last_unstale":"2024-04-02T04:55:12.445071+0000","last_undegraded":"2024-04-02T04:55:12.445071+0000","last_fullsized":"2024-04-02T04:55:12.445071+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T04:51:39.048988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"402'19","reported_seq":299,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936880+0000","last_change":"2024-04-02T04:53:57.032666+0000","last_active":"2024-04-02T04:55:08.936880+0000","last_peered":"2024-04-02T04:55:08.936880+0000","last_clean":"2024-04-02T04:55:08.936880+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T04:55:08.936880+0000","last_undegraded":"2024-04-02T04:55:08.936880+0000","last_fullsized":"2024-04-02T04:55:08.936880+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:53:57.032539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:53:57.032539+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.439204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":713,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.094352+0000","last_change":"2024-04-02T04:54:55.935347+0000","last_active":"2024-04-02T04:55:12.094352+0000","last_peered":"2024-04-02T04:55:12.094352+0000","last_clean":"2024-04-02T04:55:12.094352+0000","last_became_active":"2024-04-02T04:54:55.933130+0000","last_became_peered":"2024-04-02T04:54:55.933130+0000","last_unstale":"2024-04-02T04:55:12.094352+0000","last_undegraded":"2024-04-02T04:55:12.094352+0000","last_fullsized":"2024-04-02T04:55:12.094352+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:02.769714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:02.769714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:26.268990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"395'14","reported_seq":969,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097465+0000","last_change":"2024-04-02T04:53:55.155728+0000","last_active":"2024-04-02T04:55:12.097465+0000","last_peered":"2024-04-02T04:55:12.097465+0000","last_clean":"2024-04-02T04:55:12.097465+0000","last_became_active":"2024-04-02T04:53:54.924949+0000","last_became_peered":"2024-04-02T04:53:54.924949+0000","last_unstale":"2024-04-02T04:55:12.097465+0000","last_undegraded":"2024-04-02T04:55:12.097465+0000","last_fullsized":"2024-04-02T04:55:12.097465+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'14","last_scrub_stamp":"2024-04-02T04:53:55.155578+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.155578+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:17.148406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":812,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.113015+0000","last_change":"2024-04-02T04:54:26.393528+0000","last_active":"2024-04-02T04:55:12.113015+0000","last_peered":"2024-04-02T04:55:12.113015+0000","last_clean":"2024-04-02T04:55:12.113015+0000","last_became_active":"2024-04-02T04:53:54.956634+0000","last_became_peered":"2024-04-02T04:53:54.956634+0000","last_unstale":"2024-04-02T04:55:12.113015+0000","last_undegraded":"2024-04-02T04:55:12.113015+0000","last_fullsized":"2024-04-02T04:55:12.113015+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.393399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:26.393399+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.393399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:36.904421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":685,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110623+0000","last_change":"2024-04-02T04:54:19.202284+0000","last_active":"2024-04-02T04:55:12.110623+0000","last_peered":"2024-04-02T04:55:12.110623+0000","last_clean":"2024-04-02T04:55:12.110623+0000","last_became_active":"2024-04-02T04:53:54.944150+0000","last_became_peered":"2024-04-02T04:53:54.944150+0000","last_unstale":"2024-04-02T04:55:12.110623+0000","last_undegraded":"2024-04-02T04:55:12.110623+0000","last_fullsized":"2024-04-02T04:55:12.110623+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.202179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.202179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.435824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"402'15","reported_seq":845,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100851+0000","last_change":"2024-04-02T04:53:54.938967+0000","last_active":"2024-04-02T04:55:12.100851+0000","last_peered":"2024-04-02T04:55:12.100851+0000","last_clean":"2024-04-02T04:55:12.100851+0000","last_became_active":"2024-04-02T04:53:54.938547+0000","last_became_peered":"2024-04-02T04:53:54.938547+0000","last_unstale":"2024-04-02T04:55:12.100851+0000","last_undegraded":"2024-04-02T04:55:12.100851+0000","last_fullsized":"2024-04-02T04:55:12.100851+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-02T04:49:14.989379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:14.989379+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103031+0000","last_change":"2024-04-02T04:53:05.331108+0000","last_active":"2024-04-02T04:55:12.103031+0000","last_peered":"2024-04-02T04:55:12.103031+0000","last_clean":"2024-04-02T04:55:12.103031+0000","last_became_active":"2024-04-02T04:43:54.212385+0000","last_became_peered":"2024-04-02T04:43:54.212385+0000","last_unstale":"2024-04-02T04:55:12.103031+0000","last_undegraded":"2024-04-02T04:55:12.103031+0000","last_fullsized":"2024-04-02T04:55:12.103031+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.331020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.331020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"451'14","reported_seq":840,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.937529+0000","last_change":"2024-04-02T04:54:56.019183+0000","last_active":"2024-04-02T04:55:08.937529+0000","last_peered":"2024-04-02T04:55:08.937529+0000","last_clean":"2024-04-02T04:55:08.937529+0000","last_became_active":"2024-04-02T04:54:55.980378+0000","last_became_peered":"2024-04-02T04:54:55.980378+0000","last_unstale":"2024-04-02T04:55:08.937529+0000","last_undegraded":"2024-04-02T04:55:08.937529+0000","last_fullsized":"2024-04-02T04:55:08.937529+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'13","last_scrub_stamp":"2024-04-02T04:54:07.630056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.630056+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:10.484064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":810,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.083016+0000","last_change":"2024-04-02T04:54:56.000771+0000","last_active":"2024-04-02T04:55:12.083016+0000","last_peered":"2024-04-02T04:55:12.083016+0000","last_clean":"2024-04-02T04:55:12.083016+0000","last_became_active":"2024-04-02T04:54:56.000394+0000","last_became_peered":"2024-04-02T04:54:56.000394+0000","last_unstale":"2024-04-02T04:55:12.083016+0000","last_undegraded":"2024-04-02T04:55:12.083016+0000","last_fullsized":"2024-04-02T04:55:12.083016+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:42.525995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:42.525995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":669,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447909+0000","last_change":"2024-04-02T04:50:25.822947+0000","last_active":"2024-04-02T04:55:12.447909+0000","last_peered":"2024-04-02T04:55:12.447909+0000","last_clean":"2024-04-02T04:55:12.447909+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:55:12.447909+0000","last_undegraded":"2024-04-02T04:55:12.447909+0000","last_fullsized":"2024-04-02T04:55:12.447909+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:25.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"402'21","reported_seq":336,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.937388+0000","last_change":"2024-04-02T04:53:54.940529+0000","last_active":"2024-04-02T04:55:08.937388+0000","last_peered":"2024-04-02T04:55:08.937388+0000","last_clean":"2024-04-02T04:55:08.937388+0000","last_became_active":"2024-04-02T04:53:54.936610+0000","last_became_peered":"2024-04-02T04:53:54.936610+0000","last_unstale":"2024-04-02T04:55:08.937388+0000","last_undegraded":"2024-04-02T04:55:08.937388+0000","last_fullsized":"2024-04-02T04:55:08.937388+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:49:23.491068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:49:23.491068+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.447991+0000","last_change":"2024-04-02T04:54:55.993606+0000","last_active":"2024-04-02T04:55:12.447991+0000","last_peered":"2024-04-02T04:55:12.447991+0000","last_clean":"2024-04-02T04:55:12.447991+0000","last_became_active":"2024-04-02T04:54:55.993068+0000","last_became_peered":"2024-04-02T04:54:55.993068+0000","last_unstale":"2024-04-02T04:55:12.447991+0000","last_undegraded":"2024-04-02T04:55:12.447991+0000","last_fullsized":"2024-04-02T04:55:12.447991+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:21.847551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:50:21.847551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":817,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117896+0000","last_change":"2024-04-02T04:52:54.293779+0000","last_active":"2024-04-02T04:55:12.117896+0000","last_peered":"2024-04-02T04:55:12.117896+0000","last_clean":"2024-04-02T04:55:12.117896+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:55:12.117896+0000","last_undegraded":"2024-04-02T04:55:12.117896+0000","last_fullsized":"2024-04-02T04:55:12.117896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:54.293646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:54.293646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":817,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.091512+0000","last_change":"2024-04-02T04:54:56.005487+0000","last_active":"2024-04-02T04:55:12.091512+0000","last_peered":"2024-04-02T04:55:12.091512+0000","last_clean":"2024-04-02T04:55:12.091512+0000","last_became_active":"2024-04-02T04:54:56.005128+0000","last_became_peered":"2024-04-02T04:54:56.005128+0000","last_unstale":"2024-04-02T04:55:12.091512+0000","last_undegraded":"2024-04-02T04:55:12.091512+0000","last_fullsized":"2024-04-02T04:55:12.091512+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.965811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.965811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":673,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081216+0000","last_change":"2024-04-02T04:51:26.346563+0000","last_active":"2024-04-02T04:55:12.081216+0000","last_peered":"2024-04-02T04:55:12.081216+0000","last_clean":"2024-04-02T04:55:12.081216+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:55:12.081216+0000","last_undegraded":"2024-04-02T04:55:12.081216+0000","last_fullsized":"2024-04-02T04:55:12.081216+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:26.346350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:26.346350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"402'32","reported_seq":875,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081215+0000","last_change":"2024-04-02T04:53:58.561674+0000","last_active":"2024-04-02T04:55:12.081215+0000","last_peered":"2024-04-02T04:55:12.081215+0000","last_clean":"2024-04-02T04:55:12.081215+0000","last_became_active":"2024-04-02T04:53:54.918208+0000","last_became_peered":"2024-04-02T04:53:54.918208+0000","last_unstale":"2024-04-02T04:55:12.081215+0000","last_undegraded":"2024-04-02T04:55:12.081215+0000","last_fullsized":"2024-04-02T04:55:12.081215+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:53:58.561437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:53:58.561437+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921711+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.121208+0000","last_change":"2024-04-02T04:54:55.988018+0000","last_active":"2024-04-02T04:55:12.121208+0000","last_peered":"2024-04-02T04:55:12.121208+0000","last_clean":"2024-04-02T04:55:12.121208+0000","last_became_active":"2024-04-02T04:54:55.987373+0000","last_became_peered":"2024-04-02T04:54:55.987373+0000","last_unstale":"2024-04-02T04:55:12.121208+0000","last_undegraded":"2024-04-02T04:55:12.121208+0000","last_fullsized":"2024-04-02T04:55:12.121208+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.438193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.438193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080855+0000","last_change":"2024-04-02T04:52:40.351590+0000","last_active":"2024-04-02T04:55:12.080855+0000","last_peered":"2024-04-02T04:55:12.080855+0000","last_clean":"2024-04-02T04:55:12.080855+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:55:12.080855+0000","last_undegraded":"2024-04-02T04:55:12.080855+0000","last_fullsized":"2024-04-02T04:55:12.080855+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.351381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.351381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":821,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.097864+0000","last_change":"2024-04-02T04:54:55.941899+0000","last_active":"2024-04-02T04:55:12.097864+0000","last_peered":"2024-04-02T04:55:12.097864+0000","last_clean":"2024-04-02T04:55:12.097864+0000","last_became_active":"2024-04-02T04:54:55.940993+0000","last_became_peered":"2024-04-02T04:54:55.940993+0000","last_unstale":"2024-04-02T04:55:12.097864+0000","last_undegraded":"2024-04-02T04:55:12.097864+0000","last_fullsized":"2024-04-02T04:55:12.097864+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:27.015307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:54:27.015307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:29.107143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":686,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112378+0000","last_change":"2024-04-02T04:54:56.017323+0000","last_active":"2024-04-02T04:55:12.112378+0000","last_peered":"2024-04-02T04:55:12.112378+0000","last_clean":"2024-04-02T04:55:12.112378+0000","last_became_active":"2024-04-02T04:54:56.017129+0000","last_became_peered":"2024-04-02T04:54:56.017129+0000","last_unstale":"2024-04-02T04:55:12.112378+0000","last_undegraded":"2024-04-02T04:55:12.112378+0000","last_fullsized":"2024-04-02T04:55:12.112378+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.388684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.388684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"402'18","reported_seq":857,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.114394+0000","last_change":"2024-04-02T04:54:56.054638+0000","last_active":"2024-04-02T04:55:12.114394+0000","last_peered":"2024-04-02T04:55:12.114394+0000","last_clean":"2024-04-02T04:55:12.114394+0000","last_became_active":"2024-04-02T04:54:56.010058+0000","last_became_peered":"2024-04-02T04:54:56.010058+0000","last_unstale":"2024-04-02T04:55:12.114394+0000","last_undegraded":"2024-04-02T04:55:12.114394+0000","last_fullsized":"2024-04-02T04:55:12.114394+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-02T04:50:25.999894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:50:25.999894+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":45,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449150+0000","last_change":"2024-04-02T04:54:55.973248+0000","last_active":"2024-04-02T04:55:12.449150+0000","last_peered":"2024-04-02T04:55:12.449150+0000","last_clean":"2024-04-02T04:55:12.449150+0000","last_became_active":"2024-04-02T04:54:55.972841+0000","last_became_peered":"2024-04-02T04:54:55.972841+0000","last_unstale":"2024-04-02T04:55:12.449150+0000","last_undegraded":"2024-04-02T04:55:12.449150+0000","last_fullsized":"2024-04-02T04:55:12.449150+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:12.624937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:53:12.624937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.089210+0000","last_change":"2024-04-02T04:51:54.722249+0000","last_active":"2024-04-02T04:55:12.089210+0000","last_peered":"2024-04-02T04:55:12.089210+0000","last_clean":"2024-04-02T04:55:12.089210+0000","last_became_active":"2024-04-02T04:43:58.298572+0000","last_became_peered":"2024-04-02T04:43:58.298572+0000","last_unstale":"2024-04-02T04:55:12.089210+0000","last_undegraded":"2024-04-02T04:55:12.089210+0000","last_fullsized":"2024-04-02T04:55:12.089210+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:54.722054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:54.722054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":285,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.932851+0000","last_change":"2024-04-02T04:54:25.267842+0000","last_active":"2024-04-02T04:55:08.932851+0000","last_peered":"2024-04-02T04:55:08.932851+0000","last_clean":"2024-04-02T04:55:08.932851+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T04:55:08.932851+0000","last_undegraded":"2024-04-02T04:55:08.932851+0000","last_fullsized":"2024-04-02T04:55:08.932851+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:25.267631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:54:25.267631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:36.943944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":54,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099115+0000","last_change":"2024-04-02T04:54:55.985303+0000","last_active":"2024-04-02T04:55:12.099115+0000","last_peered":"2024-04-02T04:55:12.099115+0000","last_clean":"2024-04-02T04:55:12.099115+0000","last_became_active":"2024-04-02T04:54:55.984950+0000","last_became_peered":"2024-04-02T04:54:55.984950+0000","last_unstale":"2024-04-02T04:55:12.099115+0000","last_undegraded":"2024-04-02T04:55:12.099115+0000","last_fullsized":"2024-04-02T04:55:12.099115+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:06.134223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:06.134223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:21.790516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":156,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.940822+0000","last_change":"2024-04-02T04:53:54.929549+0000","last_active":"2024-04-02T04:55:08.940822+0000","last_peered":"2024-04-02T04:55:08.940822+0000","last_clean":"2024-04-02T04:55:08.940822+0000","last_became_active":"2024-04-02T04:53:54.927422+0000","last_became_peered":"2024-04-02T04:53:54.927422+0000","last_unstale":"2024-04-02T04:55:08.940822+0000","last_undegraded":"2024-04-02T04:55:08.940822+0000","last_fullsized":"2024-04-02T04:55:08.940822+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"402'27","reported_seq":795,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.939471+0000","last_change":"2024-04-02T04:54:56.002828+0000","last_active":"2024-04-02T04:55:08.939471+0000","last_peered":"2024-04-02T04:55:08.939471+0000","last_clean":"2024-04-02T04:55:08.939471+0000","last_became_active":"2024-04-02T04:54:55.956215+0000","last_became_peered":"2024-04-02T04:54:55.956215+0000","last_unstale":"2024-04-02T04:55:08.939471+0000","last_undegraded":"2024-04-02T04:55:08.939471+0000","last_fullsized":"2024-04-02T04:55:08.939471+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T04:54:25.604414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:25.604414+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:45.465679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449727+0000","last_change":"2024-04-02T04:54:55.979542+0000","last_active":"2024-04-02T04:55:12.449727+0000","last_peered":"2024-04-02T04:55:12.449727+0000","last_clean":"2024-04-02T04:55:12.449727+0000","last_became_active":"2024-04-02T04:54:55.978936+0000","last_became_peered":"2024-04-02T04:54:55.978936+0000","last_unstale":"2024-04-02T04:55:12.449727+0000","last_undegraded":"2024-04-02T04:55:12.449727+0000","last_fullsized":"2024-04-02T04:55:12.449727+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:34.979136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.979136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:56.731115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":820,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099844+0000","last_change":"2024-04-02T04:54:04.069490+0000","last_active":"2024-04-02T04:55:12.099844+0000","last_peered":"2024-04-02T04:55:12.099844+0000","last_clean":"2024-04-02T04:55:12.099844+0000","last_became_active":"2024-04-02T04:53:54.915756+0000","last_became_peered":"2024-04-02T04:53:54.915756+0000","last_unstale":"2024-04-02T04:55:12.099844+0000","last_undegraded":"2024-04-02T04:55:12.099844+0000","last_fullsized":"2024-04-02T04:55:12.099844+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.069399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.069399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.503633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":807,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.111289+0000","last_change":"2024-04-02T04:52:47.741959+0000","last_active":"2024-04-02T04:55:12.111289+0000","last_peered":"2024-04-02T04:55:12.111289+0000","last_clean":"2024-04-02T04:55:12.111289+0000","last_became_active":"2024-04-02T04:44:00.358112+0000","last_became_peered":"2024-04-02T04:44:00.358112+0000","last_unstale":"2024-04-02T04:55:12.111289+0000","last_undegraded":"2024-04-02T04:55:12.111289+0000","last_fullsized":"2024-04-02T04:55:12.111289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.741751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.741751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.18","version":"138'2","reported_seq":85,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103112+0000","last_change":"2024-04-02T04:54:55.982028+0000","last_active":"2024-04-02T04:55:12.103112+0000","last_peered":"2024-04-02T04:55:12.103112+0000","last_clean":"2024-04-02T04:55:12.103112+0000","last_became_active":"2024-04-02T04:54:55.935579+0000","last_became_peered":"2024-04-02T04:54:55.935579+0000","last_unstale":"2024-04-02T04:55:12.103112+0000","last_undegraded":"2024-04-02T04:55:12.103112+0000","last_fullsized":"2024-04-02T04:55:12.103112+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:54:26.267106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.267106+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:56.083367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":682,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079632+0000","last_change":"2024-04-02T04:54:07.624201+0000","last_active":"2024-04-02T04:55:12.079632+0000","last_peered":"2024-04-02T04:55:12.079632+0000","last_clean":"2024-04-02T04:55:12.079632+0000","last_became_active":"2024-04-02T04:53:54.917005+0000","last_became_peered":"2024-04-02T04:53:54.917005+0000","last_unstale":"2024-04-02T04:55:12.079632+0000","last_undegraded":"2024-04-02T04:55:12.079632+0000","last_fullsized":"2024-04-02T04:55:12.079632+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:07.623881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.623881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.876277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"402'62","reported_seq":915,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.117814+0000","last_change":"2024-04-02T04:50:33.298622+0000","last_active":"2024-04-02T04:55:12.117814+0000","last_peered":"2024-04-02T04:55:12.117814+0000","last_clean":"2024-04-02T04:55:12.117814+0000","last_became_active":"2024-04-02T04:43:56.233273+0000","last_became_peered":"2024-04-02T04:43:56.233273+0000","last_unstale":"2024-04-02T04:55:12.117814+0000","last_undegraded":"2024-04-02T04:55:12.117814+0000","last_fullsized":"2024-04-02T04:55:12.117814+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'50","last_scrub_stamp":"2024-04-02T04:50:33.298479+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:50:33.298479+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":721,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079599+0000","last_change":"2024-04-02T04:54:56.011075+0000","last_active":"2024-04-02T04:55:12.079599+0000","last_peered":"2024-04-02T04:55:12.079599+0000","last_clean":"2024-04-02T04:55:12.079599+0000","last_became_active":"2024-04-02T04:54:56.008703+0000","last_became_peered":"2024-04-02T04:54:56.008703+0000","last_unstale":"2024-04-02T04:55:12.079599+0000","last_undegraded":"2024-04-02T04:55:12.079599+0000","last_fullsized":"2024-04-02T04:55:12.079599+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.730067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.730067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":817,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112717+0000","last_change":"2024-04-02T04:53:54.955657+0000","last_active":"2024-04-02T04:55:12.112717+0000","last_peered":"2024-04-02T04:55:12.112717+0000","last_clean":"2024-04-02T04:55:12.112717+0000","last_became_active":"2024-04-02T04:53:54.955305+0000","last_became_peered":"2024-04-02T04:53:54.955305+0000","last_unstale":"2024-04-02T04:55:12.112717+0000","last_undegraded":"2024-04-02T04:55:12.112717+0000","last_fullsized":"2024-04-02T04:55:12.112717+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:11.011018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:49:11.011018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":288,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936650+0000","last_change":"2024-04-02T04:54:55.960137+0000","last_active":"2024-04-02T04:55:08.936650+0000","last_peered":"2024-04-02T04:55:08.936650+0000","last_clean":"2024-04-02T04:55:08.936650+0000","last_became_active":"2024-04-02T04:54:55.934158+0000","last_became_peered":"2024-04-02T04:54:55.934158+0000","last_unstale":"2024-04-02T04:55:08.936650+0000","last_undegraded":"2024-04-02T04:55:08.936650+0000","last_fullsized":"2024-04-02T04:55:08.936650+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:54:00.147830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.147830+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:23.205001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.17","version":"433'6","reported_seq":70,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.940091+0000","last_change":"2024-04-02T04:54:48.056666+0000","last_active":"2024-04-02T04:55:08.940091+0000","last_peered":"2024-04-02T04:55:08.940091+0000","last_clean":"2024-04-02T04:55:08.940091+0000","last_became_active":"2024-04-02T04:54:48.046361+0000","last_became_peered":"2024-04-02T04:54:48.046361+0000","last_unstale":"2024-04-02T04:55:08.940091+0000","last_undegraded":"2024-04-02T04:55:08.940091+0000","last_fullsized":"2024-04-02T04:55:08.940091+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'6","last_scrub_stamp":"2024-04-02T04:54:33.691607+0000","last_deep_scrub":"433'6","last_deep_scrub_stamp":"2024-04-02T04:54:33.691607+0000","last_clean_scrub_stamp":"2024-04-02T04:54:33.691607+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:45.547999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":570,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101213+0000","last_change":"2024-04-02T04:54:55.990492+0000","last_active":"2024-04-02T04:55:12.101213+0000","last_peered":"2024-04-02T04:55:12.101213+0000","last_clean":"2024-04-02T04:55:12.101213+0000","last_became_active":"2024-04-02T04:54:55.989605+0000","last_became_peered":"2024-04-02T04:54:55.989605+0000","last_unstale":"2024-04-02T04:55:12.101213+0000","last_undegraded":"2024-04-02T04:55:12.101213+0000","last_fullsized":"2024-04-02T04:55:12.101213+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.540354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.540354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:37.177029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"402'21","reported_seq":860,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.088933+0000","last_change":"2024-04-02T04:52:58.806428+0000","last_active":"2024-04-02T04:55:12.088933+0000","last_peered":"2024-04-02T04:55:12.088933+0000","last_clean":"2024-04-02T04:55:12.088933+0000","last_became_active":"2024-04-02T04:43:56.234366+0000","last_became_peered":"2024-04-02T04:43:56.234366+0000","last_unstale":"2024-04-02T04:55:12.088933+0000","last_undegraded":"2024-04-02T04:55:12.088933+0000","last_fullsized":"2024-04-02T04:55:12.088933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:58.806324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.806324+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100018+0000","last_change":"2024-04-02T04:54:32.960737+0000","last_active":"2024-04-02T04:55:12.100018+0000","last_peered":"2024-04-02T04:55:12.100018+0000","last_clean":"2024-04-02T04:55:12.100018+0000","last_became_active":"2024-04-02T04:43:54.203026+0000","last_became_peered":"2024-04-02T04:43:54.203026+0000","last_unstale":"2024-04-02T04:55:12.100018+0000","last_undegraded":"2024-04-02T04:55:12.100018+0000","last_fullsized":"2024-04-02T04:55:12.100018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:54:32.960556+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:54:32.960556+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:02.900525+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"451'12","reported_seq":973,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449855+0000","last_change":"2024-04-02T04:54:56.025228+0000","last_active":"2024-04-02T04:55:12.449855+0000","last_peered":"2024-04-02T04:55:12.449855+0000","last_clean":"2024-04-02T04:55:12.449855+0000","last_became_active":"2024-04-02T04:54:56.006792+0000","last_became_peered":"2024-04-02T04:54:56.006792+0000","last_unstale":"2024-04-02T04:55:12.449855+0000","last_undegraded":"2024-04-02T04:55:12.449855+0000","last_fullsized":"2024-04-02T04:55:12.449855+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:54:18.879221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:18.879221+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:19.996343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.121430+0000","last_change":"2024-04-02T04:55:11.232633+0000","last_active":"2024-04-02T04:55:12.121430+0000","last_peered":"2024-04-02T04:55:12.121430+0000","last_clean":"2024-04-02T04:55:12.121430+0000","last_became_active":"2024-04-02T04:54:55.979124+0000","last_became_peered":"2024-04-02T04:54:55.979124+0000","last_unstale":"2024-04-02T04:55:12.121430+0000","last_undegraded":"2024-04-02T04:55:12.121430+0000","last_fullsized":"2024-04-02T04:55:12.121430+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:11.232532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:11.232532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:27.541964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"139'5","reported_seq":83,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081302+0000","last_change":"2024-04-02T04:54:56.022644+0000","last_active":"2024-04-02T04:55:12.081302+0000","last_peered":"2024-04-02T04:55:12.081302+0000","last_clean":"2024-04-02T04:55:12.081302+0000","last_became_active":"2024-04-02T04:54:55.995682+0000","last_became_peered":"2024-04-02T04:54:55.995682+0000","last_unstale":"2024-04-02T04:55:12.081302+0000","last_undegraded":"2024-04-02T04:55:12.081302+0000","last_fullsized":"2024-04-02T04:55:12.081302+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:51:32.581233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:32.581233+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445809+0000","last_change":"2024-04-02T04:54:55.972812+0000","last_active":"2024-04-02T04:55:12.445809+0000","last_peered":"2024-04-02T04:55:12.445809+0000","last_clean":"2024-04-02T04:55:12.445809+0000","last_became_active":"2024-04-02T04:54:55.972361+0000","last_became_peered":"2024-04-02T04:54:55.972361+0000","last_unstale":"2024-04-02T04:55:12.445809+0000","last_undegraded":"2024-04-02T04:55:12.445809+0000","last_fullsized":"2024-04-02T04:55:12.445809+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:37.840813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:50:37.840813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"402'16","reported_seq":848,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103196+0000","last_change":"2024-04-02T04:54:27.408260+0000","last_active":"2024-04-02T04:55:12.103196+0000","last_peered":"2024-04-02T04:55:12.103196+0000","last_clean":"2024-04-02T04:55:12.103196+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T04:55:12.103196+0000","last_undegraded":"2024-04-02T04:55:12.103196+0000","last_fullsized":"2024-04-02T04:55:12.103196+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'16","last_scrub_stamp":"2024-04-02T04:54:27.408029+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:54:27.408029+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:30.835150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.445878+0000","last_change":"2024-04-02T04:54:10.943919+0000","last_active":"2024-04-02T04:55:12.445878+0000","last_peered":"2024-04-02T04:55:12.445878+0000","last_clean":"2024-04-02T04:55:12.445878+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T04:55:12.445878+0000","last_undegraded":"2024-04-02T04:55:12.445878+0000","last_fullsized":"2024-04-02T04:55:12.445878+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:10.943768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:54:10.943768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:26.636177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":821,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.092146+0000","last_change":"2024-04-02T04:52:58.690594+0000","last_active":"2024-04-02T04:55:12.092146+0000","last_peered":"2024-04-02T04:55:12.092146+0000","last_clean":"2024-04-02T04:55:12.092146+0000","last_became_active":"2024-04-02T04:43:58.302345+0000","last_became_peered":"2024-04-02T04:43:58.302345+0000","last_unstale":"2024-04-02T04:55:12.092146+0000","last_undegraded":"2024-04-02T04:55:12.092146+0000","last_fullsized":"2024-04-02T04:55:12.092146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.690492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.690492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":811,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082871+0000","last_change":"2024-04-02T04:54:56.012078+0000","last_active":"2024-04-02T04:55:12.082871+0000","last_peered":"2024-04-02T04:55:12.082871+0000","last_clean":"2024-04-02T04:55:12.082871+0000","last_became_active":"2024-04-02T04:54:56.011705+0000","last_became_peered":"2024-04-02T04:54:56.011705+0000","last_unstale":"2024-04-02T04:55:12.082871+0000","last_undegraded":"2024-04-02T04:55:12.082871+0000","last_fullsized":"2024-04-02T04:55:12.082871+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:27.563824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:27.563824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":696,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081798+0000","last_change":"2024-04-02T04:54:56.009174+0000","last_active":"2024-04-02T04:55:12.081798+0000","last_peered":"2024-04-02T04:55:12.081798+0000","last_clean":"2024-04-02T04:55:12.081798+0000","last_became_active":"2024-04-02T04:54:56.004190+0000","last_became_peered":"2024-04-02T04:54:56.004190+0000","last_unstale":"2024-04-02T04:55:12.081798+0000","last_undegraded":"2024-04-02T04:55:12.081798+0000","last_fullsized":"2024-04-02T04:55:12.081798+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.244258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.244258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":683,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118683+0000","last_change":"2024-04-02T04:54:21.209523+0000","last_active":"2024-04-02T04:55:12.118683+0000","last_peered":"2024-04-02T04:55:12.118683+0000","last_clean":"2024-04-02T04:55:12.118683+0000","last_became_active":"2024-04-02T04:53:54.942584+0000","last_became_peered":"2024-04-02T04:53:54.942584+0000","last_unstale":"2024-04-02T04:55:12.118683+0000","last_undegraded":"2024-04-02T04:55:12.118683+0000","last_fullsized":"2024-04-02T04:55:12.118683+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:21.209411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:21.209411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:28.711443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"402'35","reported_seq":869,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080160+0000","last_change":"2024-04-02T04:50:14.537273+0000","last_active":"2024-04-02T04:55:12.080160+0000","last_peered":"2024-04-02T04:55:12.080160+0000","last_clean":"2024-04-02T04:55:12.080160+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:55:12.080160+0000","last_undegraded":"2024-04-02T04:55:12.080160+0000","last_fullsized":"2024-04-02T04:55:12.080160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'29","last_scrub_stamp":"2024-04-02T04:50:14.537029+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:50:14.537029+0000","objects_scrubbed":6,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":836,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110845+0000","last_change":"2024-04-02T04:54:09.153699+0000","last_active":"2024-04-02T04:55:12.110845+0000","last_peered":"2024-04-02T04:55:12.110845+0000","last_clean":"2024-04-02T04:55:12.110845+0000","last_became_active":"2024-04-02T04:53:54.944446+0000","last_became_peered":"2024-04-02T04:53:54.944446+0000","last_unstale":"2024-04-02T04:55:12.110845+0000","last_undegraded":"2024-04-02T04:55:12.110845+0000","last_fullsized":"2024-04-02T04:55:12.110845+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:09.153487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:09.153487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:21.256198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.110525+0000","last_change":"2024-04-02T04:53:14.837679+0000","last_active":"2024-04-02T04:55:12.110525+0000","last_peered":"2024-04-02T04:55:12.110525+0000","last_clean":"2024-04-02T04:55:12.110525+0000","last_became_active":"2024-04-02T04:43:58.298890+0000","last_became_peered":"2024-04-02T04:43:58.298890+0000","last_unstale":"2024-04-02T04:55:12.110525+0000","last_undegraded":"2024-04-02T04:55:12.110525+0000","last_fullsized":"2024-04-02T04:55:12.110525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:14.837465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:53:14.837465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":814,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.112235+0000","last_change":"2024-04-02T04:54:55.986536+0000","last_active":"2024-04-02T04:55:12.112235+0000","last_peered":"2024-04-02T04:55:12.112235+0000","last_clean":"2024-04-02T04:55:12.112235+0000","last_became_active":"2024-04-02T04:54:55.985747+0000","last_became_peered":"2024-04-02T04:54:55.985747+0000","last_unstale":"2024-04-02T04:55:12.112235+0000","last_undegraded":"2024-04-02T04:55:12.112235+0000","last_fullsized":"2024-04-02T04:55:12.112235+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:40.671923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:50:40.671923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.14","version":"138'2","reported_seq":90,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.448418+0000","last_change":"2024-04-02T04:54:48.055292+0000","last_active":"2024-04-02T04:55:12.448418+0000","last_peered":"2024-04-02T04:55:12.448418+0000","last_clean":"2024-04-02T04:55:12.448418+0000","last_became_active":"2024-04-02T04:54:48.040842+0000","last_became_peered":"2024-04-02T04:54:48.040842+0000","last_unstale":"2024-04-02T04:55:12.448418+0000","last_undegraded":"2024-04-02T04:55:12.448418+0000","last_fullsized":"2024-04-02T04:55:12.448418+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:54:28.369217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.369217+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:51.877092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":674,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.120642+0000","last_change":"2024-04-02T04:50:39.321796+0000","last_active":"2024-04-02T04:55:12.120642+0000","last_peered":"2024-04-02T04:55:12.120642+0000","last_clean":"2024-04-02T04:55:12.120642+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:55:12.120642+0000","last_undegraded":"2024-04-02T04:55:12.120642+0000","last_fullsized":"2024-04-02T04:55:12.120642+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:50:39.321605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"402'24","reported_seq":794,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081941+0000","last_change":"2024-04-02T04:54:56.051234+0000","last_active":"2024-04-02T04:55:12.081941+0000","last_peered":"2024-04-02T04:55:12.081941+0000","last_clean":"2024-04-02T04:55:12.081941+0000","last_became_active":"2024-04-02T04:54:56.014703+0000","last_became_peered":"2024-04-02T04:54:56.014703+0000","last_unstale":"2024-04-02T04:55:12.081941+0000","last_undegraded":"2024-04-02T04:55:12.081941+0000","last_fullsized":"2024-04-02T04:55:12.081941+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:52:48.727589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.727589+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":308,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.935966+0000","last_change":"2024-04-02T04:54:55.991236+0000","last_active":"2024-04-02T04:55:08.935966+0000","last_peered":"2024-04-02T04:55:08.935966+0000","last_clean":"2024-04-02T04:55:08.935966+0000","last_became_active":"2024-04-02T04:54:55.971317+0000","last_became_peered":"2024-04-02T04:54:55.971317+0000","last_unstale":"2024-04-02T04:55:08.935966+0000","last_undegraded":"2024-04-02T04:55:08.935966+0000","last_fullsized":"2024-04-02T04:55:08.935966+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:54:14.246467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:14.246467+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:21.438858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":808,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080381+0000","last_change":"2024-04-02T04:51:48.311838+0000","last_active":"2024-04-02T04:55:12.080381+0000","last_peered":"2024-04-02T04:55:12.080381+0000","last_clean":"2024-04-02T04:55:12.080381+0000","last_became_active":"2024-04-02T04:43:58.287331+0000","last_became_peered":"2024-04-02T04:43:58.287331+0000","last_unstale":"2024-04-02T04:55:12.080381+0000","last_undegraded":"2024-04-02T04:55:12.080381+0000","last_fullsized":"2024-04-02T04:55:12.080381+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.311616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.311616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":812,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.091245+0000","last_change":"2024-04-02T04:54:24.386420+0000","last_active":"2024-04-02T04:55:12.091245+0000","last_peered":"2024-04-02T04:55:12.091245+0000","last_clean":"2024-04-02T04:55:12.091245+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T04:55:12.091245+0000","last_undegraded":"2024-04-02T04:55:12.091245+0000","last_fullsized":"2024-04-02T04:55:12.091245+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:24.386320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.386320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:48.013202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.13","version":"139'9","reported_seq":588,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.103667+0000","last_change":"2024-04-02T04:54:48.055289+0000","last_active":"2024-04-02T04:55:12.103667+0000","last_peered":"2024-04-02T04:55:12.103667+0000","last_clean":"2024-04-02T04:55:12.103667+0000","last_became_active":"2024-04-02T04:54:48.025378+0000","last_became_peered":"2024-04-02T04:54:48.025378+0000","last_unstale":"2024-04-02T04:55:12.103667+0000","last_undegraded":"2024-04-02T04:55:12.103667+0000","last_fullsized":"2024-04-02T04:55:12.103667+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:49:15.972156+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":682,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.101648+0000","last_change":"2024-04-02T04:54:55.943251+0000","last_active":"2024-04-02T04:55:12.101648+0000","last_peered":"2024-04-02T04:55:12.101648+0000","last_clean":"2024-04-02T04:55:12.101648+0000","last_became_active":"2024-04-02T04:54:55.942912+0000","last_became_peered":"2024-04-02T04:54:55.942912+0000","last_unstale":"2024-04-02T04:55:12.101648+0000","last_undegraded":"2024-04-02T04:55:12.101648+0000","last_fullsized":"2024-04-02T04:55:12.101648+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:24.637853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:24.637853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"402'27","reported_seq":796,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.081045+0000","last_change":"2024-04-02T04:54:56.036522+0000","last_active":"2024-04-02T04:55:12.081045+0000","last_peered":"2024-04-02T04:55:12.081045+0000","last_clean":"2024-04-02T04:55:12.081045+0000","last_became_active":"2024-04-02T04:54:56.011842+0000","last_became_peered":"2024-04-02T04:54:56.011842+0000","last_unstale":"2024-04-02T04:55:12.081045+0000","last_undegraded":"2024-04-02T04:55:12.081045+0000","last_fullsized":"2024-04-02T04:55:12.081045+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:53.802714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.802714+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":303,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.936090+0000","last_change":"2024-04-02T04:53:54.943928+0000","last_active":"2024-04-02T04:55:08.936090+0000","last_peered":"2024-04-02T04:55:08.936090+0000","last_clean":"2024-04-02T04:55:08.936090+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T04:55:08.936090+0000","last_undegraded":"2024-04-02T04:55:08.936090+0000","last_fullsized":"2024-04-02T04:55:08.936090+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:22.452700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:49:22.452700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:13.438830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":808,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080798+0000","last_change":"2024-04-02T04:50:09.464337+0000","last_active":"2024-04-02T04:55:12.080798+0000","last_peered":"2024-04-02T04:55:12.080798+0000","last_clean":"2024-04-02T04:55:12.080798+0000","last_became_active":"2024-04-02T04:43:58.296800+0000","last_became_peered":"2024-04-02T04:43:58.296800+0000","last_unstale":"2024-04-02T04:55:12.080798+0000","last_undegraded":"2024-04-02T04:55:12.080798+0000","last_fullsized":"2024-04-02T04:55:12.080798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:50:09.464222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:50:09.464222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.919356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":44,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.118429+0000","last_change":"2024-04-02T04:54:55.939287+0000","last_active":"2024-04-02T04:55:12.118429+0000","last_peered":"2024-04-02T04:55:12.118429+0000","last_clean":"2024-04-02T04:55:12.118429+0000","last_became_active":"2024-04-02T04:54:55.938942+0000","last_became_peered":"2024-04-02T04:54:55.938942+0000","last_unstale":"2024-04-02T04:55:12.118429+0000","last_undegraded":"2024-04-02T04:55:12.118429+0000","last_fullsized":"2024-04-02T04:55:12.118429+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.730634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.730634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":54,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.092679+0000","last_change":"2024-04-02T04:54:55.949504+0000","last_active":"2024-04-02T04:55:12.092679+0000","last_peered":"2024-04-02T04:55:12.092679+0000","last_clean":"2024-04-02T04:55:12.092679+0000","last_became_active":"2024-04-02T04:54:55.949201+0000","last_became_peered":"2024-04-02T04:54:55.949201+0000","last_unstale":"2024-04-02T04:55:12.092679+0000","last_undegraded":"2024-04-02T04:55:12.092679+0000","last_fullsized":"2024-04-02T04:55:12.092679+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.381932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.052221+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.381932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":675,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.449069+0000","last_change":"2024-04-02T04:54:56.014628+0000","last_active":"2024-04-02T04:55:12.449069+0000","last_peered":"2024-04-02T04:55:12.449069+0000","last_clean":"2024-04-02T04:55:12.449069+0000","last_became_active":"2024-04-02T04:54:56.014277+0000","last_became_peered":"2024-04-02T04:54:56.014277+0000","last_unstale":"2024-04-02T04:55:12.449069+0000","last_undegraded":"2024-04-02T04:55:12.449069+0000","last_fullsized":"2024-04-02T04:55:12.449069+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.488717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.488717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"402'25","reported_seq":883,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.114585+0000","last_change":"2024-04-02T04:54:56.049199+0000","last_active":"2024-04-02T04:55:12.114585+0000","last_peered":"2024-04-02T04:55:12.114585+0000","last_clean":"2024-04-02T04:55:12.114585+0000","last_became_active":"2024-04-02T04:54:55.991156+0000","last_became_peered":"2024-04-02T04:54:55.991156+0000","last_unstale":"2024-04-02T04:55:12.114585+0000","last_undegraded":"2024-04-02T04:55:12.114585+0000","last_fullsized":"2024-04-02T04:55:12.114585+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:52:52.302878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.302878+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100610+0000","last_change":"2024-04-02T04:54:55.960085+0000","last_active":"2024-04-02T04:55:12.100610+0000","last_peered":"2024-04-02T04:55:12.100610+0000","last_clean":"2024-04-02T04:55:12.100610+0000","last_became_active":"2024-04-02T04:54:55.959699+0000","last_became_peered":"2024-04-02T04:54:55.959699+0000","last_unstale":"2024-04-02T04:55:12.100610+0000","last_undegraded":"2024-04-02T04:55:12.100610+0000","last_fullsized":"2024-04-02T04:55:12.100610+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:01.793655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:01.793655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":808,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.079471+0000","last_change":"2024-04-02T04:51:49.302781+0000","last_active":"2024-04-02T04:55:12.079471+0000","last_peered":"2024-04-02T04:55:12.079471+0000","last_clean":"2024-04-02T04:55:12.079471+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:55:12.079471+0000","last_undegraded":"2024-04-02T04:55:12.079471+0000","last_fullsized":"2024-04-02T04:55:12.079471+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.302549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.302549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":810,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.080091+0000","last_change":"2024-04-02T04:54:55.988287+0000","last_active":"2024-04-02T04:55:12.080091+0000","last_peered":"2024-04-02T04:55:12.080091+0000","last_clean":"2024-04-02T04:55:12.080091+0000","last_became_active":"2024-04-02T04:54:55.984764+0000","last_became_peered":"2024-04-02T04:54:55.984764+0000","last_unstale":"2024-04-02T04:55:12.080091+0000","last_undegraded":"2024-04-02T04:55:12.080091+0000","last_fullsized":"2024-04-02T04:55:12.080091+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.343663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.343663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.921308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.11","version":"139'7","reported_seq":98,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.446045+0000","last_change":"2024-04-02T04:54:48.058696+0000","last_active":"2024-04-02T04:55:12.446045+0000","last_peered":"2024-04-02T04:55:12.446045+0000","last_clean":"2024-04-02T04:55:12.446045+0000","last_became_active":"2024-04-02T04:54:48.046345+0000","last_became_peered":"2024-04-02T04:54:48.046345+0000","last_unstale":"2024-04-02T04:55:12.446045+0000","last_undegraded":"2024-04-02T04:55:12.446045+0000","last_fullsized":"2024-04-02T04:55:12.446045+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7","last_scrub_stamp":"2024-04-02T04:51:38.237550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:38.237550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":673,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.082911+0000","last_change":"2024-04-02T04:51:36.397355+0000","last_active":"2024-04-02T04:55:12.082911+0000","last_peered":"2024-04-02T04:55:12.082911+0000","last_clean":"2024-04-02T04:55:12.082911+0000","last_became_active":"2024-04-02T04:49:26.536918+0000","last_became_peered":"2024-04-02T04:49:26.536918+0000","last_unstale":"2024-04-02T04:55:12.082911+0000","last_undegraded":"2024-04-02T04:55:12.082911+0000","last_fullsized":"2024-04-02T04:55:12.082911+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:36.397213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:36.397213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:12.920530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"402'32","reported_seq":872,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.100939+0000","last_change":"2024-04-02T04:53:02.778087+0000","last_active":"2024-04-02T04:55:12.100939+0000","last_peered":"2024-04-02T04:55:12.100939+0000","last_clean":"2024-04-02T04:55:12.100939+0000","last_became_active":"2024-04-02T04:43:56.243717+0000","last_became_peered":"2024-04-02T04:43:56.243717+0000","last_unstale":"2024-04-02T04:55:12.100939+0000","last_undegraded":"2024-04-02T04:55:12.100939+0000","last_fullsized":"2024-04-02T04:55:12.100939+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'22","last_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_deep_scrub":"342'22","last_deep_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.777776+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.114843+0000","last_change":"2024-04-02T04:54:23.368378+0000","last_active":"2024-04-02T04:55:12.114843+0000","last_peered":"2024-04-02T04:55:12.114843+0000","last_clean":"2024-04-02T04:55:12.114843+0000","last_became_active":"2024-04-02T04:53:54.924695+0000","last_became_peered":"2024-04-02T04:53:54.924695+0000","last_unstale":"2024-04-02T04:55:12.114843+0000","last_undegraded":"2024-04-02T04:55:12.114843+0000","last_fullsized":"2024-04-02T04:55:12.114843+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:23.368272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:23.368272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:24.298352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.099578+0000","last_change":"2024-04-02T04:52:56.823906+0000","last_active":"2024-04-02T04:55:12.099578+0000","last_peered":"2024-04-02T04:55:12.099578+0000","last_clean":"2024-04-02T04:55:12.099578+0000","last_became_active":"2024-04-02T04:43:58.291907+0000","last_became_peered":"2024-04-02T04:43:58.291907+0000","last_unstale":"2024-04-02T04:55:12.099578+0000","last_undegraded":"2024-04-02T04:55:12.099578+0000","last_fullsized":"2024-04-02T04:55:12.099578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.823620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.823620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-04-02T04:55:12.102683+0000","last_change":"2024-04-02T04:54:15.109115+0000","last_active":"2024-04-02T04:55:12.102683+0000","last_peered":"2024-04-02T04:55:12.102683+0000","last_clean":"2024-04-02T04:55:12.102683+0000","last_became_active":"2024-04-02T04:53:54.943990+0000","last_became_peered":"2024-04-02T04:53:54.943990+0000","last_unstale":"2024-04-02T04:55:12.102683+0000","last_undegraded":"2024-04-02T04:55:12.102683+0000","last_fullsized":"2024-04-02T04:55:12.102683+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:15.108990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:54:15.108990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:34.115709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":63,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-04-02T04:55:08.930946+0000","last_change":"2024-04-02T04:54:48.004392+0000","last_active":"2024-04-02T04:55:08.930946+0000","last_peered":"2024-04-02T04:55:08.930946+0000","last_clean":"2024-04-02T04:55:08.930946+0000","last_became_active":"2024-04-02T04:54:48.003497+0000","last_became_peered":"2024-04-02T04:54:48.003497+0000","last_unstale":"2024-04-02T04:55:08.930946+0000","last_undegraded":"2024-04-02T04:55:08.930946+0000","last_fullsized":"2024-04-02T04:55:08.930946+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":294,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":7},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":8195793,"num_objects":161,"num_object_clones":4,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":4002365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30572544,"data_stored":28587574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1872,"ondisk_log_size":1872,"up":96,"acting":96,"num_store_stats":8},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":13,"num_pg":26,"stat_sum":{"num_bytes":1920,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":78,"acting":78,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1407,"num_read_kb":1132,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":1172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":941,"ondisk_log_size":941,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":650,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114641,"num_pgs":141,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49020,"kb_used_data":12508,"kb_used_omap":57,"kb_used_meta":36422,"kb_avail":93696132,"statfs":{"total":95995035648,"available":95944839168,"internally_reserved":0,"allocated":12808192,"data_stored":9773169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":37297038},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103325,"num_pgs":165,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84104,"kb_used_data":16796,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93661048,"statfs":{"total":95995035648,"available":95908913152,"internally_reserved":0,"allocated":17199104,"data_stored":14012697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":148,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79492,"kb_used_data":12196,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93665660,"statfs":{"total":95995035648,"available":95913635840,"internally_reserved":0,"allocated":12488704,"data_stored":9422106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":150,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75948,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669204,"statfs":{"total":95995035648,"available":95917264896,"internally_reserved":0,"allocated":8847360,"data_stored":5817989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378822,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75480,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669672,"statfs":{"total":95995035648,"available":95917744128,"internally_reserved":0,"allocated":8368128,"data_stored":5228454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051772,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83364,"kb_used_data":11920,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93661788,"statfs":{"total":95995035648,"available":95909670912,"internally_reserved":0,"allocated":12206080,"data_stored":9216650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822834,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61476,"kb_used_data":12228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93683676,"statfs":{"total":95995035648,"available":95932084224,"internally_reserved":0,"allocated":12521472,"data_stored":9412271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4472832,"data_stored":4194965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4000479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4448256,"data_stored":4194865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4067328,"data_stored":4000177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8581120,"data_stored":8195225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":4000608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":14,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":294,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:55:14.639 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-02T04:55:14.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pg_num 11 2024-04-02T04:55:14.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:14.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:14.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:14.847 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:14.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:14.975 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:14.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:15.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:15.102 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:15.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:15.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: cephadm 2024-04-02T04:55:14.035333+0000 mgr.y (mgr.14538) 455 : cephadm 1 Adjusting osd_memory_target on smithi069 to 3292M 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.107429+0000 mon.a (mon.0) 2792 : cluster 0 osdmap e466: 8 total, 8 up, 7 in 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: audit 2024-04-02T04:55:14.122137+0000 mon.b (mon.1) 243 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.125058+0000 mgr.y (mgr.14538) 456 : cluster 0 pgmap v680: 350 pgs: 1 clean+premerge+peered, 349 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: audit 2024-04-02T04:55:14.127538+0000 mon.a (mon.0) 2793 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.363638+0000 osd.1 (osd.1) 261 : cluster 0 3.19 scrub starts 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.368673+0000 osd.1 (osd.1) 262 : cluster 0 3.19 scrub ok 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: audit 2024-04-02T04:55:14.900670+0000 mon.a (mon.0) 2794 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-02T04:55:15.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:15 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.903672+0000 mon.a (mon.0) 2795 : cluster 0 osdmap e467: 8 total, 8 up, 7 in 2024-04-02T04:55:15.231 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:15.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:15.361 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:15.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:15.474 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:55:15.494 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:15.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: cephadm 2024-04-02T04:55:14.035333+0000 mgr.y (mgr.14538) 455 : cephadm 1 Adjusting osd_memory_target on smithi069 to 3292M 2024-04-02T04:55:15.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.107429+0000 mon.a (mon.0) 2792 : cluster 0 osdmap e466: 8 total, 8 up, 7 in 2024-04-02T04:55:15.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: audit 2024-04-02T04:55:14.122137+0000 mon.b (mon.1) 243 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-02T04:55:15.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.125058+0000 mgr.y (mgr.14538) 456 : cluster 0 pgmap v680: 350 pgs: 1 clean+premerge+peered, 349 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:15.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: audit 2024-04-02T04:55:14.127538+0000 mon.a (mon.0) 2793 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-02T04:55:15.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.363638+0000 osd.1 (osd.1) 261 : cluster 0 3.19 scrub starts 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.368673+0000 osd.1 (osd.1) 262 : cluster 0 3.19 scrub ok 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: audit 2024-04-02T04:55:14.900670+0000 mon.a (mon.0) 2794 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.903672+0000 mon.a (mon.0) 2795 : cluster 0 osdmap e467: 8 total, 8 up, 7 in 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: cephadm 2024-04-02T04:55:14.035333+0000 mgr.y (mgr.14538) 455 : cephadm 1 Adjusting osd_memory_target on smithi069 to 3292M 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.107429+0000 mon.a (mon.0) 2792 : cluster 0 osdmap e466: 8 total, 8 up, 7 in 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: audit 2024-04-02T04:55:14.122137+0000 mon.b (mon.1) 243 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.125058+0000 mgr.y (mgr.14538) 456 : cluster 0 pgmap v680: 350 pgs: 1 clean+premerge+peered, 349 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: audit 2024-04-02T04:55:14.127538+0000 mon.a (mon.0) 2793 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.363638+0000 osd.1 (osd.1) 261 : cluster 0 3.19 scrub starts 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.368673+0000 osd.1 (osd.1) 262 : cluster 0 3.19 scrub ok 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: audit 2024-04-02T04:55:14.900670+0000 mon.a (mon.0) 2794 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-02T04:55:15.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:15 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.903672+0000 mon.a (mon.0) 2795 : cluster 0 osdmap e467: 8 total, 8 up, 7 in 2024-04-02T04:55:15.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:15.621 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:15.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:15.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:15.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:15.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:15.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:16.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:16.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:16.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:16.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.296510+0000 osd.7 (osd.7) 39 : cluster 0 3.11 scrub starts 2024-04-02T04:55:16.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:14.300833+0000 osd.7 (osd.7) 40 : cluster 0 3.11 scrub ok 2024-04-02T04:55:16.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.098368+0000 osd.0 (osd.0) 239 : cluster 0 3.12 scrub starts 2024-04-02T04:55:16.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.110863+0000 osd.0 (osd.0) 240 : cluster 0 3.12 scrub ok 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.852137+0000 osd.2 (osd.2) 208 : cluster 0 41.14 scrub starts 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.854531+0000 osd.2 (osd.2) 209 : cluster 0 41.14 scrub ok 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.920571+0000 mon.a (mon.0) 2796 : cluster 0 osdmap e468: 8 total, 8 up, 7 in 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.296510+0000 osd.7 (osd.7) 39 : cluster 0 3.11 scrub starts 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:14.300833+0000 osd.7 (osd.7) 40 : cluster 0 3.11 scrub ok 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.098368+0000 osd.0 (osd.0) 239 : cluster 0 3.12 scrub starts 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.110863+0000 osd.0 (osd.0) 240 : cluster 0 3.12 scrub ok 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.852137+0000 osd.2 (osd.2) 208 : cluster 0 41.14 scrub starts 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.854531+0000 osd.2 (osd.2) 209 : cluster 0 41.14 scrub ok 2024-04-02T04:55:16.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:16 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.920571+0000 mon.a (mon.0) 2796 : cluster 0 osdmap e468: 8 total, 8 up, 7 in 2024-04-02T04:55:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:16.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:16.397 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:16.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.296510+0000 osd.7 (osd.7) 39 : cluster 0 3.11 scrub starts 2024-04-02T04:55:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:14.300833+0000 osd.7 (osd.7) 40 : cluster 0 3.11 scrub ok 2024-04-02T04:55:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.098368+0000 osd.0 (osd.0) 239 : cluster 0 3.12 scrub starts 2024-04-02T04:55:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.110863+0000 osd.0 (osd.0) 240 : cluster 0 3.12 scrub ok 2024-04-02T04:55:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.852137+0000 osd.2 (osd.2) 208 : cluster 0 41.14 scrub starts 2024-04-02T04:55:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.854531+0000 osd.2 (osd.2) 209 : cluster 0 41.14 scrub ok 2024-04-02T04:55:16.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:16 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.920571+0000 mon.a (mon.0) 2796 : cluster 0 osdmap e468: 8 total, 8 up, 7 in 2024-04-02T04:55:16.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:16.526 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:16.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:16.653 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:16.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:16.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:16.908 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:16.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:17.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:17.038 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:17.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:17.163 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:17.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:17 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.971795+0000 osd.3 (osd.3) 289 : cluster 0 6.1f scrub starts 2024-04-02T04:55:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:17 smithi098 bash[23790]: cluster 2024-04-02T04:55:15.974109+0000 osd.3 (osd.3) 290 : cluster 0 6.1f scrub ok 2024-04-02T04:55:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:17 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.126380+0000 mgr.y (mgr.14538) 457 : cluster 0 pgmap v683: 350 pgs: 1 clean+premerge+peered, 349 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:17 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.826695+0000 osd.2 (osd.2) 210 : cluster 0 43.7 scrub starts 2024-04-02T04:55:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:17 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.829697+0000 osd.2 (osd.2) 211 : cluster 0 43.7 scrub ok 2024-04-02T04:55:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:17 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.923049+0000 mon.a (mon.0) 2797 : cluster 0 osdmap e469: 8 total, 8 up, 7 in 2024-04-02T04:55:17.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:17.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.971795+0000 osd.3 (osd.3) 289 : cluster 0 6.1f scrub starts 2024-04-02T04:55:17.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[21528]: cluster 2024-04-02T04:55:15.974109+0000 osd.3 (osd.3) 290 : cluster 0 6.1f scrub ok 2024-04-02T04:55:17.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.126380+0000 mgr.y (mgr.14538) 457 : cluster 0 pgmap v683: 350 pgs: 1 clean+premerge+peered, 349 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:17.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.826695+0000 osd.2 (osd.2) 210 : cluster 0 43.7 scrub starts 2024-04-02T04:55:17.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.829697+0000 osd.2 (osd.2) 211 : cluster 0 43.7 scrub ok 2024-04-02T04:55:17.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.923049+0000 mon.a (mon.0) 2797 : cluster 0 osdmap e469: 8 total, 8 up, 7 in 2024-04-02T04:55:17.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.971795+0000 osd.3 (osd.3) 289 : cluster 0 6.1f scrub starts 2024-04-02T04:55:17.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[28904]: cluster 2024-04-02T04:55:15.974109+0000 osd.3 (osd.3) 290 : cluster 0 6.1f scrub ok 2024-04-02T04:55:17.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.126380+0000 mgr.y (mgr.14538) 457 : cluster 0 pgmap v683: 350 pgs: 1 clean+premerge+peered, 349 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:17.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.826695+0000 osd.2 (osd.2) 210 : cluster 0 43.7 scrub starts 2024-04-02T04:55:17.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.829697+0000 osd.2 (osd.2) 211 : cluster 0 43.7 scrub ok 2024-04-02T04:55:17.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:17 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.923049+0000 mon.a (mon.0) 2797 : cluster 0 osdmap e469: 8 total, 8 up, 7 in 2024-04-02T04:55:17.292 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:17.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:17.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:17.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:17.547 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:17.650 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:17.673 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:17.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:17.804 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:17.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:17.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:17.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:18.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:18.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:18.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.311350+0000 osd.7 (osd.7) 41 : cluster 0 3.e scrub starts 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.314005+0000 osd.7 (osd.7) 42 : cluster 0 3.e scrub ok 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.948054+0000 osd.3 (osd.3) 291 : cluster 0 10.15 scrub starts 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:16.955145+0000 osd.3 (osd.3) 292 : cluster 0 10.15 scrub ok 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.029146+0000 osd.0 (osd.0) 241 : cluster 0 40.1f scrub starts 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.030541+0000 osd.0 (osd.0) 242 : cluster 0 40.1f scrub ok 2024-04-02T04:55:18.194 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.390735+0000 osd.1 (osd.1) 263 : cluster 0 10.14 scrub starts 2024-04-02T04:55:18.195 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.392228+0000 osd.1 (osd.1) 264 : cluster 0 10.14 scrub ok 2024-04-02T04:55:18.195 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.790287+0000 osd.2 (osd.2) 212 : cluster 0 4.1c scrub starts 2024-04-02T04:55:18.195 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.792827+0000 osd.2 (osd.2) 213 : cluster 0 4.1c scrub ok 2024-04-02T04:55:18.195 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.928538+0000 mon.a (mon.0) 2798 : cluster 0 osdmap e470: 8 total, 8 up, 7 in 2024-04-02T04:55:18.195 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: audit 2024-04-02T04:55:18.069574+0000 mon.b (mon.1) 244 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:18.195 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:18 smithi098 bash[23790]: audit 2024-04-02T04:55:18.124218+0000 mon.b (mon.1) 245 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-02T04:55:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.311350+0000 osd.7 (osd.7) 41 : cluster 0 3.e scrub starts 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.314005+0000 osd.7 (osd.7) 42 : cluster 0 3.e scrub ok 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.948054+0000 osd.3 (osd.3) 291 : cluster 0 10.15 scrub starts 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:16.955145+0000 osd.3 (osd.3) 292 : cluster 0 10.15 scrub ok 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.029146+0000 osd.0 (osd.0) 241 : cluster 0 40.1f scrub starts 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.030541+0000 osd.0 (osd.0) 242 : cluster 0 40.1f scrub ok 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.390735+0000 osd.1 (osd.1) 263 : cluster 0 10.14 scrub starts 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.392228+0000 osd.1 (osd.1) 264 : cluster 0 10.14 scrub ok 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.790287+0000 osd.2 (osd.2) 212 : cluster 0 4.1c scrub starts 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.792827+0000 osd.2 (osd.2) 213 : cluster 0 4.1c scrub ok 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.928538+0000 mon.a (mon.0) 2798 : cluster 0 osdmap e470: 8 total, 8 up, 7 in 2024-04-02T04:55:18.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: audit 2024-04-02T04:55:18.069574+0000 mon.b (mon.1) 244 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[28904]: audit 2024-04-02T04:55:18.124218+0000 mon.b (mon.1) 245 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.311350+0000 osd.7 (osd.7) 41 : cluster 0 3.e scrub starts 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.314005+0000 osd.7 (osd.7) 42 : cluster 0 3.e scrub ok 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.948054+0000 osd.3 (osd.3) 291 : cluster 0 10.15 scrub starts 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:16.955145+0000 osd.3 (osd.3) 292 : cluster 0 10.15 scrub ok 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.029146+0000 osd.0 (osd.0) 241 : cluster 0 40.1f scrub starts 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.030541+0000 osd.0 (osd.0) 242 : cluster 0 40.1f scrub ok 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.390735+0000 osd.1 (osd.1) 263 : cluster 0 10.14 scrub starts 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.392228+0000 osd.1 (osd.1) 264 : cluster 0 10.14 scrub ok 2024-04-02T04:55:18.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.790287+0000 osd.2 (osd.2) 212 : cluster 0 4.1c scrub starts 2024-04-02T04:55:18.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.792827+0000 osd.2 (osd.2) 213 : cluster 0 4.1c scrub ok 2024-04-02T04:55:18.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.928538+0000 mon.a (mon.0) 2798 : cluster 0 osdmap e470: 8 total, 8 up, 7 in 2024-04-02T04:55:18.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: audit 2024-04-02T04:55:18.069574+0000 mon.b (mon.1) 244 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:18.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21528]: audit 2024-04-02T04:55:18.124218+0000 mon.b (mon.1) 245 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-02T04:55:18.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:18.326 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:18.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:18.457 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:18.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:18.588 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:18.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:18.702 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:18.718 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:18.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:18.866 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:55:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:55:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:55:18.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:18.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:19.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:19.166 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:19.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:19.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.858033+0000 osd.6 (osd.6) 215 : cluster 0 2.14 scrub starts 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: cluster 2024-04-02T04:55:17.859619+0000 osd.6 (osd.6) 216 : cluster 0 2.14 scrub ok 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.050032+0000 osd.0 (osd.0) 243 : cluster 0 41.15 scrub starts 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.051699+0000 osd.0 (osd.0) 244 : cluster 0 41.15 scrub ok 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.127971+0000 mgr.y (mgr.14538) 458 : cluster 0 pgmap v686: 349 pgs: 1 activating, 348 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: audit 2024-04-02T04:55:18.132132+0000 mon.a (mon.0) 2799 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: audit 2024-04-02T04:55:18.919584+0000 mon.a (mon.0) 2800 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-02T04:55:19.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:19 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.929192+0000 mon.a (mon.0) 2801 : cluster 0 osdmap e471: 8 total, 8 up, 7 in 2024-04-02T04:55:19.231 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:19.296 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:55:19.296 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:55:19.296 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:55:19.296 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:55:19.296 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:55:19.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:19.360 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:19.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:19.490 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.858033+0000 osd.6 (osd.6) 215 : cluster 0 2.14 scrub starts 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: cluster 2024-04-02T04:55:17.859619+0000 osd.6 (osd.6) 216 : cluster 0 2.14 scrub ok 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.050032+0000 osd.0 (osd.0) 243 : cluster 0 41.15 scrub starts 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.051699+0000 osd.0 (osd.0) 244 : cluster 0 41.15 scrub ok 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.127971+0000 mgr.y (mgr.14538) 458 : cluster 0 pgmap v686: 349 pgs: 1 activating, 348 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: audit 2024-04-02T04:55:18.132132+0000 mon.a (mon.0) 2799 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-02T04:55:19.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: audit 2024-04-02T04:55:18.919584+0000 mon.a (mon.0) 2800 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.929192+0000 mon.a (mon.0) 2801 : cluster 0 osdmap e471: 8 total, 8 up, 7 in 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.858033+0000 osd.6 (osd.6) 215 : cluster 0 2.14 scrub starts 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: cluster 2024-04-02T04:55:17.859619+0000 osd.6 (osd.6) 216 : cluster 0 2.14 scrub ok 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.050032+0000 osd.0 (osd.0) 243 : cluster 0 41.15 scrub starts 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.051699+0000 osd.0 (osd.0) 244 : cluster 0 41.15 scrub ok 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.127971+0000 mgr.y (mgr.14538) 458 : cluster 0 pgmap v686: 349 pgs: 1 activating, 348 active+clean; 8.4 MiB data, 497 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: audit 2024-04-02T04:55:18.132132+0000 mon.a (mon.0) 2799 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: audit 2024-04-02T04:55:18.919584+0000 mon.a (mon.0) 2800 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-02T04:55:19.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:19 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.929192+0000 mon.a (mon.0) 2801 : cluster 0 osdmap e471: 8 total, 8 up, 7 in 2024-04-02T04:55:19.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:19.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:19.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:19.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:19.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:19.881 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:19.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:19.984 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:20.008 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:20.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:20.135 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:20.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-02T04:55:20.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:20.264 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:20.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.315912+0000 osd.7 (osd.7) 43 : cluster 0 43.a scrub starts 2024-04-02T04:55:20.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.317643+0000 osd.7 (osd.7) 44 : cluster 0 43.a scrub ok 2024-04-02T04:55:20.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.980627+0000 osd.4 (osd.4) 251 : cluster 0 41.1c scrub starts 2024-04-02T04:55:20.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:18.981900+0000 osd.4 (osd.4) 252 : cluster 0 41.1c scrub ok 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:19.762561+0000 osd.2 (osd.2) 214 : cluster 0 13.a scrub starts 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:19.765547+0000 osd.2 (osd.2) 215 : cluster 0 13.a scrub ok 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[21528]: cluster 2024-04-02T04:55:19.912939+0000 mon.a (mon.0) 2802 : cluster 0 osdmap e472: 8 total, 8 up, 7 in 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.315912+0000 osd.7 (osd.7) 43 : cluster 0 43.a scrub starts 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.317643+0000 osd.7 (osd.7) 44 : cluster 0 43.a scrub ok 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.980627+0000 osd.4 (osd.4) 251 : cluster 0 41.1c scrub starts 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:18.981900+0000 osd.4 (osd.4) 252 : cluster 0 41.1c scrub ok 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:19.762561+0000 osd.2 (osd.2) 214 : cluster 0 13.a scrub starts 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:19.765547+0000 osd.2 (osd.2) 215 : cluster 0 13.a scrub ok 2024-04-02T04:55:20.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:20 smithi069 bash[28904]: cluster 2024-04-02T04:55:19.912939+0000 mon.a (mon.0) 2802 : cluster 0 osdmap e472: 8 total, 8 up, 7 in 2024-04-02T04:55:20.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:20.388 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.315912+0000 osd.7 (osd.7) 43 : cluster 0 43.a scrub starts 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.317643+0000 osd.7 (osd.7) 44 : cluster 0 43.a scrub ok 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.980627+0000 osd.4 (osd.4) 251 : cluster 0 41.1c scrub starts 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:18.981900+0000 osd.4 (osd.4) 252 : cluster 0 41.1c scrub ok 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:19.762561+0000 osd.2 (osd.2) 214 : cluster 0 13.a scrub starts 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:19.765547+0000 osd.2 (osd.2) 215 : cluster 0 13.a scrub ok 2024-04-02T04:55:20.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:20 smithi098 bash[23790]: cluster 2024-04-02T04:55:19.912939+0000 mon.a (mon.0) 2802 : cluster 0 osdmap e472: 8 total, 8 up, 7 in 2024-04-02T04:55:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:20.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:20.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:20.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:20.775 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:20.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:20.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:21.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:21.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:21.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:21.162 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:21.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:21.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:19.315446+0000 osd.7 (osd.7) 45 : cluster 0 10.c scrub starts 2024-04-02T04:55:21.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:19.317097+0000 osd.7 (osd.7) 46 : cluster 0 10.c scrub ok 2024-04-02T04:55:21.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.030946+0000 osd.3 (osd.3) 293 : cluster 0 41.3 scrub starts 2024-04-02T04:55:21.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.038102+0000 osd.3 (osd.3) 294 : cluster 0 41.3 scrub ok 2024-04-02T04:55:21.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.050143+0000 osd.4 (osd.4) 253 : cluster 0 41.0 scrub starts 2024-04-02T04:55:21.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.052918+0000 osd.4 (osd.4) 254 : cluster 0 41.0 scrub ok 2024-04-02T04:55:21.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.128598+0000 mgr.y (mgr.14538) 459 : cluster 0 pgmap v689: 349 pgs: 1 activating, 348 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:21.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:21 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.933451+0000 mon.a (mon.0) 2803 : cluster 0 osdmap e473: 8 total, 8 up, 7 in 2024-04-02T04:55:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- 172.21.15.69:0/1497056149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 msgr2=0x7f503810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1497056149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 0x7f503810f120 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f503000b0d0 tx=0x7f503000b3e0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- 172.21.15.69:0/1497056149 shutdown_connections 2024-04-02T04:55:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1497056149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 0x7f503810f120 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1497056149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038107260 0x7f5038107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1497056149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5038100790 0x7f5038100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- 172.21.15.69:0/1497056149 >> 172.21.15.69:0/1497056149 conn(0x7f5038077740 msgr2=0x7f50380feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- 172.21.15.69:0/1497056149 shutdown_connections 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- 172.21.15.69:0/1497056149 wait complete. 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 librados: starting msgr at 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 librados: starting objecter 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 Processor -- start 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- start start 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 librados: setting wanted keys 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 librados: calling monclient init 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038100790 0x7f5038198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 0x7f5038198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5038107260 0x7f503819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5038111b40 con 0x7f5038100790 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50381119c0 con 0x7f5038107260 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503cb6f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5038111cc0 con 0x7f5038101150 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038100790 0x7f5038198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f5035d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 0x7f5038198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038100790 0x7f5038198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44124/0 (socket says 172.21.15.69:44124) 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f503659c700 1 -- 172.21.15.69:0/1116612936 learned_addr learned my addr 172.21.15.69:0/1116612936 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.390+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5038107260 0x7f503819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503659c700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 msgr2=0x7f5038198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5038101150 0x7f5038198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503659c700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5038107260 msgr2=0x7f503819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5038107260 0x7f503819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503659c700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503819cfc0 con 0x7f5038100790 2024-04-02T04:55:21.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038100790 0x7f5038198000 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f502800eb30 tx=0x7f502800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 10 client.?.objecter ms_handle_connect 0x7f5038100790 2024-04-02T04:55:21.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:21.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502800cc90 con 0x7f5038100790 2024-04-02T04:55:21.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503819d2b0 con 0x7f5038100790 2024-04-02T04:55:21.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 10 client.16488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f502800ce30 con 0x7f5038100790 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50381a4b30 con 0x7f5038100790 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 1 librados: init done 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028003df0 con 0x7f5038100790 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f503804ff00 con 0x7f5038100790 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5028004000 con 0x7f5038100790 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f502006f5e0 0x7f5020071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(473..473 src has 74..473) ==== 10953+0+0 (secure 0 0 0) 0x7f502808f000 con 0x7f5038100790 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 osd_map(473..473 src has 74..473) 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 3 client.16488.objecter handle_osd_map got epochs [473,473] > 0 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 3 client.16488.objecter handle_osd_map decoding full epoch 473 2024-04-02T04:55:21.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f5035d9b700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f502006f5e0 0x7f5020071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 20 client.16488.objecter dump_active .. 0 homeless 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v473) ==== 54+0+16 (secure 0 0 0) 0x7f5028091620 con 0x7f5038100790 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 10 client.16488.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 1 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f502006f5e0 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f5035d9b700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f502006f5e0 0x7f5020071aa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f50380ffed0 tx=0x7f502c00a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5038062e50 0x7f5038063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 1 2024-04-02T04:55:21.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 0 1 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 4 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 10 client.16488.objecter _send_command 1 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50381a7f80 con 0x7f5038062e50 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.394+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5038062e50 0x7f5038063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5038062e50 0x7f5038063230 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f5038062e50 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50381a7f80 con 0x7f5038062e50 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 1: 0 ) 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f50277fe700 10 client.16488.objecter _finish_command 1 = system:0 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 4 2024-04-02T04:55:21.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.398+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 0 1 2024-04-02T04:55:21.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 10 client.16488.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 2 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 2 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f5038199310 osd=0 3 2024-04-02T04:55:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 0 2 2024-04-02T04:55:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 4 2024-04-02T04:55:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 10 client.16488.objecter _send_command 2 2024-04-02T04:55:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.414+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50380628d0 con 0x7f5038062e50 2024-04-02T04:55:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f50380628d0 con 0x7f5038062e50 2024-04-02T04:55:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 10 client.16488.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 4 2024-04-02T04:55:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 0 2 2024-04-02T04:55:21.416 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 10 client.16488.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f503802c9b0 0x7f503802cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 1 3 2024-04-02T04:55:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 4 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 10 client.16488.objecter _send_command 3 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5038109eb0 con 0x7f503802c9b0 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f503802c9b0 0x7f503802cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f503802c9b0 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f503802c9b0 0x7f503802cdb0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5038109eb0 con 0x7f503802c9b0 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 3: 0 ) 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 10 client.16488.objecter _finish_command 3 = system:0 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 4 2024-04-02T04:55:21.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.418+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 1 3 2024-04-02T04:55:21.424 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 10 client.16488.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 4 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 4 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f5038109830 osd=1 3 2024-04-02T04:55:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 1 4 2024-04-02T04:55:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 4 2024-04-02T04:55:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 10 client.16488.objecter _send_command 4 2024-04-02T04:55:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.434+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50381092b0 con 0x7f503802c9b0 2024-04-02T04:55:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f50381092b0 con 0x7f503802c9b0 2024-04-02T04:55:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 10 client.16488.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 4 2024-04-02T04:55:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 1 4 2024-04-02T04:55:21.437 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:55:21.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 10 client.16488.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 5 2024-04-02T04:55:21.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f50381aa140 0x7f50381ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 5 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 2 5 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 4 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 10 client.16488.objecter _send_command 5 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f50381af390 con 0x7f50381aa140 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f5035d9b700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f50381aa140 0x7f50381ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f50381aa140 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f5035d9b700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f50381aa140 0x7f50381ac530 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50381af390 con 0x7f50381aa140 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 5: 0 ) 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 10 client.16488.objecter _finish_command 5 = system:0 2024-04-02T04:55:21.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 4 2024-04-02T04:55:21.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.438+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 2 5 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 10 client.16488.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 6 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 6 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f503802dac0 osd=2 3 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 2 6 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 4 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 10 client.16488.objecter _send_command 6 2024-04-02T04:55:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503802d540 con 0x7f50381aa140 2024-04-02T04:55:21.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503802d540 con 0x7f50381aa140 2024-04-02T04:55:21.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f50277fe700 10 client.16488.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 4 2024-04-02T04:55:21.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 2 6 2024-04-02T04:55:21.459 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:55:21.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 10 client.16488.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 7 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f5038102a20 0x7f50381aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 7 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 3 7 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 4 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 10 client.16488.objecter _send_command 7 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f50381a7c50 con 0x7f5038102a20 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.458+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f5038102a20 0x7f50381aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f5038102a20 2024-04-02T04:55:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f5038102a20 0x7f50381aea80 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50381a7c50 con 0x7f5038102a20 2024-04-02T04:55:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 7: 0 ) 2024-04-02T04:55:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f50277fe700 10 client.16488.objecter _finish_command 7 = system:0 2024-04-02T04:55:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 4 2024-04-02T04:55:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.462+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 3 7 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:19.315446+0000 osd.7 (osd.7) 45 : cluster 0 10.c scrub starts 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:19.317097+0000 osd.7 (osd.7) 46 : cluster 0 10.c scrub ok 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.030946+0000 osd.3 (osd.3) 293 : cluster 0 41.3 scrub starts 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.038102+0000 osd.3 (osd.3) 294 : cluster 0 41.3 scrub ok 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.050143+0000 osd.4 (osd.4) 253 : cluster 0 41.0 scrub starts 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.052918+0000 osd.4 (osd.4) 254 : cluster 0 41.0 scrub ok 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.128598+0000 mgr.y (mgr.14538) 459 : cluster 0 pgmap v689: 349 pgs: 1 activating, 348 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.933451+0000 mon.a (mon.0) 2803 : cluster 0 osdmap e473: 8 total, 8 up, 7 in 2024-04-02T04:55:21.477 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:19.315446+0000 osd.7 (osd.7) 45 : cluster 0 10.c scrub starts 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:19.317097+0000 osd.7 (osd.7) 46 : cluster 0 10.c scrub ok 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.030946+0000 osd.3 (osd.3) 293 : cluster 0 41.3 scrub starts 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.038102+0000 osd.3 (osd.3) 294 : cluster 0 41.3 scrub ok 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.050143+0000 osd.4 (osd.4) 253 : cluster 0 41.0 scrub starts 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.052918+0000 osd.4 (osd.4) 254 : cluster 0 41.0 scrub ok 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.128598+0000 mgr.y (mgr.14538) 459 : cluster 0 pgmap v689: 349 pgs: 1 activating, 348 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:21.478 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:21 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.933451+0000 mon.a (mon.0) 2803 : cluster 0 osdmap e473: 8 total, 8 up, 7 in 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 10 client.16488.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 8 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 8 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381a75d0 osd=3 3 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 3 8 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 4 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 10 client.16488.objecter _send_command 8 2024-04-02T04:55:21.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.478+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5038102e80 con 0x7f5038102a20 2024-04-02T04:55:21.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5038102e80 con 0x7f5038102a20 2024-04-02T04:55:21.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f50277fe700 10 client.16488.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 4 2024-04-02T04:55:21.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 3 8 2024-04-02T04:55:21.482 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:55:21.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 10 client.16488.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 9 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f5038103390 0x7f50381b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 9 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 4 9 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 4 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 10 client.16488.objecter _send_command 9 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5038103810 con 0x7f5038103390 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f5038103390 0x7f50381b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f5038103390 2024-04-02T04:55:21.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.482+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f5038103390 0x7f50381b5960 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.486+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5038103810 con 0x7f5038103390 2024-04-02T04:55:21.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.486+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 9: 0 ) 2024-04-02T04:55:21.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.486+0000 7f50277fe700 10 client.16488.objecter _finish_command 9 = system:0 2024-04-02T04:55:21.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.486+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 4 2024-04-02T04:55:21.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.486+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 4 9 2024-04-02T04:55:21.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 10 client.16488.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 10 2024-04-02T04:55:21.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 10 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381a7e00 osd=4 3 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 4 10 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 4 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 10 client.16488.objecter _send_command 10 2024-04-02T04:55:21.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.502+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5038103810 con 0x7f5038103390 2024-04-02T04:55:21.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5038103810 con 0x7f5038103390 2024-04-02T04:55:21.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 10 client.16488.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 4 2024-04-02T04:55:21.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 4 10 2024-04-02T04:55:21.504 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 10 client.16488.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 11 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f50381b63b0 0x7f50381b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 11 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 5 11 2024-04-02T04:55:21.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 4 2024-04-02T04:55:21.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 10 client.16488.objecter _send_command 11 2024-04-02T04:55:21.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f50381bb9a0 con 0x7f50381b63b0 2024-04-02T04:55:21.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f5035d9b700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f50381b63b0 0x7f50381b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f50381b63b0 2024-04-02T04:55:21.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f5035d9b700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f50381b63b0 0x7f50381b67f0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50381bb9a0 con 0x7f50381b63b0 2024-04-02T04:55:21.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 11: 0 ) 2024-04-02T04:55:21.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 10 client.16488.objecter _finish_command 11 = system:0 2024-04-02T04:55:21.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 4 2024-04-02T04:55:21.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.506+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 5 11 2024-04-02T04:55:21.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:21.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 10 client.16488.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 12 2024-04-02T04:55:21.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 12 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f5038103810 osd=5 3 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 5 12 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 4 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 10 client.16488.objecter _send_command 12 2024-04-02T04:55:21.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.542+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50381b6e40 con 0x7f50381b63b0 2024-04-02T04:55:21.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f50381b6e40 con 0x7f50381b63b0 2024-04-02T04:55:21.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f50277fe700 10 client.16488.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 4 2024-04-02T04:55:21.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 5 12 2024-04-02T04:55:21.545 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:55:21.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 10 client.16488.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 13 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f50381b7560 0x7f50381b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 13 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 6 13 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 4 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 10 client.16488.objecter _send_command 13 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f50381b80e0 con 0x7f50381b7560 2024-04-02T04:55:21.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f50381b7560 0x7f50381b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f50381b7560 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.546+0000 7f5036d9d700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f50381b7560 0x7f50381b79a0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.550+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50381b80e0 con 0x7f50381b7560 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.550+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 13: 0 ) 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.550+0000 7f50277fe700 10 client.16488.objecter _finish_command 13 = system:0 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.550+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 4 2024-04-02T04:55:21.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.550+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 6 13 2024-04-02T04:55:21.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:21.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 10 client.16488.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 14 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 14 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381b6e40 osd=6 3 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 6 14 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 4 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 10 client.16488.objecter _send_command 14 2024-04-02T04:55:21.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50381b80a0 con 0x7f50381b7560 2024-04-02T04:55:21.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f50381b80a0 con 0x7f50381b7560 2024-04-02T04:55:21.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f50277fe700 10 client.16488.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 4 2024-04-02T04:55:21.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.566+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 6 14 2024-04-02T04:55:21.567 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 10 client.16488.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 15 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f50381bcd40 0x7f50381bf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 15 2024-04-02T04:55:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 7 15 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 4 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 10 client.16488.objecter _send_command 15 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f50381c1f90 con 0x7f50381bcd40 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f50381bcd40 0x7f50381bf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f50277fe700 10 client.16488.objecter ms_handle_connect 0x7f50381bcd40 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f503659c700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f50381bcd40 0x7f50381bf130 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:21.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50381c1f90 con 0x7f50381bcd40 2024-04-02T04:55:21.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 15: 0 ) 2024-04-02T04:55:21.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f50277fe700 10 client.16488.objecter _finish_command 15 = system:0 2024-04-02T04:55:21.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 4 2024-04-02T04:55:21.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.570+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 7 15 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 10 client.16488.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign -1 16 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 20 client.16488.objecter _get_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_remove -1 16 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 20 client.16488.objecter get_session s=0x7f50381b80a0 osd=7 3 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 15 client.16488.objecter _session_command_op_assign 7 16 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 4 2024-04-02T04:55:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 10 client.16488.objecter _send_command 16 2024-04-02T04:55:21.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.586+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50381bf670 con 0x7f50381bcd40 2024-04-02T04:55:21.588 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:55:21.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f50277fe700 1 -- 172.21.15.69:0/1116612936 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f50381bf670 con 0x7f50381bcd40 2024-04-02T04:55:21.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f50277fe700 10 client.16488.objecter ms_dispatch 0x7f5038001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:21.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f50277fe700 10 client.16488.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:21.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f50277fe700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 4 2024-04-02T04:55:21.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f50277fe700 15 client.16488.objecter _session_command_op_remove 7 16 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 librados: watch_flush enter 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 librados: watch_flush exit 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.0 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5038062e50 msgr2=0x7f5038063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5038062e50 0x7f5038063230 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038199310 osd=0 1 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.1 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f503802c9b0 msgr2=0x7f503802cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f503802c9b0 0x7f503802cdb0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038109830 osd=1 1 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.2 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f50381aa140 msgr2=0x7f50381ac530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f50381aa140 0x7f50381ac530 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f503802dac0 osd=2 1 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.3 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f5038102a20 msgr2=0x7f50381aea80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f5038102a20 0x7f50381aea80 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a75d0 osd=3 1 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.4 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f5038103390 msgr2=0x7f50381b5960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f5038103390 0x7f50381b5960 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f5036d9d700 1 -- 172.21.15.69:0/1116612936 reap_dead start 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f5036d9d700 1 -- 172.21.15.69:0/1116612936 reap_dead start 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381a7e00 osd=4 1 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.5 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f50381b63b0 msgr2=0x7f50381b67f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f50381b63b0 0x7f50381b67f0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f5038103810 osd=5 1 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.6 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f50381b7560 msgr2=0x7f50381b79a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f50381b7560 0x7f50381b79a0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b6e40 osd=6 1 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter close_session for osd.7 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f50381bcd40 msgr2=0x7f50381bf130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f50381bcd40 0x7f50381bf130 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter put_session s=0x7f50381b80a0 osd=7 1 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 20 client.16488.objecter shutdown clearing up homeless session... 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 10 client.16488.objecter successfully canceled tick 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f502006f5e0 msgr2=0x7f5020071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f502006f5e0 0x7f5020071aa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f50380ffed0 tx=0x7f502c00a300 comp rx=0 tx=0).stop 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038100790 msgr2=0x7f5038198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 --2- 172.21.15.69:0/1116612936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5038100790 0x7f5038198000 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f502800eb30 tx=0x7f502800ee40 comp rx=0 tx=0).stop 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f5036d9d700 1 -- 172.21.15.69:0/1116612936 reap_dead start 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 shutdown_connections 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 >> 172.21.15.69:0/1116612936 conn(0x7f5038077740 msgr2=0x7f5038104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 shutdown_connections 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 -- 172.21.15.69:0/1116612936 wait complete. 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:21.590+0000 7f503cb6f700 1 librados: shutdown 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:55:21.593 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:55:21.594 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:55:21.594 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:55:21.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:21.685 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:21.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:21.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:21.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:21.951 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:21.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:22.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:22.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.276683+0000 osd.7 (osd.7) 47 : cluster 0 3.15 scrub starts 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.279087+0000 osd.7 (osd.7) 48 : cluster 0 3.15 scrub ok 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.979727+0000 osd.4 (osd.4) 255 : cluster 0 43.1d scrub starts 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:20.982526+0000 osd.4 (osd.4) 256 : cluster 0 43.1d scrub ok 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.054730+0000 osd.3 (osd.3) 295 : cluster 0 40.c scrub starts 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.056748+0000 osd.3 (osd.3) 296 : cluster 0 40.c scrub ok 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: audit 2024-04-02T04:55:21.398311+0000 mon.a (mon.0) 2804 : audit 0 from='client.? 172.21.15.69:0/1116612936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.445583+0000 osd.1 (osd.1) 265 : cluster 0 6.4 scrub starts 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.446618+0000 osd.1 (osd.1) 266 : cluster 0 6.4 scrub ok 2024-04-02T04:55:22.196 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.933824+0000 mon.a (mon.0) 2805 : cluster 0 osdmap e474: 8 total, 8 up, 7 in 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: audit 2024-04-02T04:55:22.125754+0000 mon.b (mon.1) 246 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[21528]: audit 2024-04-02T04:55:22.133366+0000 mon.a (mon.0) 2806 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.276683+0000 osd.7 (osd.7) 47 : cluster 0 3.15 scrub starts 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.279087+0000 osd.7 (osd.7) 48 : cluster 0 3.15 scrub ok 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.979727+0000 osd.4 (osd.4) 255 : cluster 0 43.1d scrub starts 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:20.982526+0000 osd.4 (osd.4) 256 : cluster 0 43.1d scrub ok 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.054730+0000 osd.3 (osd.3) 295 : cluster 0 40.c scrub starts 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.056748+0000 osd.3 (osd.3) 296 : cluster 0 40.c scrub ok 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: audit 2024-04-02T04:55:21.398311+0000 mon.a (mon.0) 2804 : audit 0 from='client.? 172.21.15.69:0/1116612936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.445583+0000 osd.1 (osd.1) 265 : cluster 0 6.4 scrub starts 2024-04-02T04:55:22.197 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.446618+0000 osd.1 (osd.1) 266 : cluster 0 6.4 scrub ok 2024-04-02T04:55:22.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.933824+0000 mon.a (mon.0) 2805 : cluster 0 osdmap e474: 8 total, 8 up, 7 in 2024-04-02T04:55:22.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: audit 2024-04-02T04:55:22.125754+0000 mon.b (mon.1) 246 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-02T04:55:22.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:22 smithi069 bash[28904]: audit 2024-04-02T04:55:22.133366+0000 mon.a (mon.0) 2806 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-02T04:55:22.210 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:22.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:22.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.276683+0000 osd.7 (osd.7) 47 : cluster 0 3.15 scrub starts 2024-04-02T04:55:22.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.279087+0000 osd.7 (osd.7) 48 : cluster 0 3.15 scrub ok 2024-04-02T04:55:22.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.979727+0000 osd.4 (osd.4) 255 : cluster 0 43.1d scrub starts 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:20.982526+0000 osd.4 (osd.4) 256 : cluster 0 43.1d scrub ok 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.054730+0000 osd.3 (osd.3) 295 : cluster 0 40.c scrub starts 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.056748+0000 osd.3 (osd.3) 296 : cluster 0 40.c scrub ok 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: audit 2024-04-02T04:55:21.398311+0000 mon.a (mon.0) 2804 : audit 0 from='client.? 172.21.15.69:0/1116612936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.445583+0000 osd.1 (osd.1) 265 : cluster 0 6.4 scrub starts 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.446618+0000 osd.1 (osd.1) 266 : cluster 0 6.4 scrub ok 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.933824+0000 mon.a (mon.0) 2805 : cluster 0 osdmap e474: 8 total, 8 up, 7 in 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: audit 2024-04-02T04:55:22.125754+0000 mon.b (mon.1) 246 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-02T04:55:22.325 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:22 smithi098 bash[23790]: audit 2024-04-02T04:55:22.133366+0000 mon.a (mon.0) 2806 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-02T04:55:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:22.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:22.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:22.598 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:22.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:22.727 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:22.830 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:22.856 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:22.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:22.985 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:22.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:22.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:23.088 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:23.114 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:23.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:23.245 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:23.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:23.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.910294+0000 osd.6 (osd.6) 217 : cluster 0 13.1 scrub starts 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.913089+0000 osd.6 (osd.6) 218 : cluster 0 13.1 scrub ok 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.979057+0000 osd.0 (osd.0) 245 : cluster 0 13.13 scrub starts 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:21.983937+0000 osd.0 (osd.0) 246 : cluster 0 13.13 scrub ok 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.129575+0000 mgr.y (mgr.14538) 460 : cluster 0 pgmap v692: 348 pgs: 1 activating, 347 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.864736+0000 osd.2 (osd.2) 216 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.864917+0000 osd.2 (osd.2) 217 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: audit 2024-04-02T04:55:22.940763+0000 mon.a (mon.0) 2807 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-02T04:55:23.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:23 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.950606+0000 mon.a (mon.0) 2808 : cluster 0 osdmap e475: 8 total, 8 up, 7 in 2024-04-02T04:55:23.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.910294+0000 osd.6 (osd.6) 217 : cluster 0 13.1 scrub starts 2024-04-02T04:55:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.913089+0000 osd.6 (osd.6) 218 : cluster 0 13.1 scrub ok 2024-04-02T04:55:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.979057+0000 osd.0 (osd.0) 245 : cluster 0 13.13 scrub starts 2024-04-02T04:55:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:21.983937+0000 osd.0 (osd.0) 246 : cluster 0 13.13 scrub ok 2024-04-02T04:55:23.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.129575+0000 mgr.y (mgr.14538) 460 : cluster 0 pgmap v692: 348 pgs: 1 activating, 347 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.864736+0000 osd.2 (osd.2) 216 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.864917+0000 osd.2 (osd.2) 217 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: audit 2024-04-02T04:55:22.940763+0000 mon.a (mon.0) 2807 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.950606+0000 mon.a (mon.0) 2808 : cluster 0 osdmap e475: 8 total, 8 up, 7 in 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.910294+0000 osd.6 (osd.6) 217 : cluster 0 13.1 scrub starts 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.913089+0000 osd.6 (osd.6) 218 : cluster 0 13.1 scrub ok 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.979057+0000 osd.0 (osd.0) 245 : cluster 0 13.13 scrub starts 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:21.983937+0000 osd.0 (osd.0) 246 : cluster 0 13.13 scrub ok 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.129575+0000 mgr.y (mgr.14538) 460 : cluster 0 pgmap v692: 348 pgs: 1 activating, 347 active+clean; 8.4 MiB data, 498 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:23.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.864736+0000 osd.2 (osd.2) 216 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:23.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.864917+0000 osd.2 (osd.2) 217 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:23.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: audit 2024-04-02T04:55:22.940763+0000 mon.a (mon.0) 2807 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-02T04:55:23.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:23 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.950606+0000 mon.a (mon.0) 2808 : cluster 0 osdmap e475: 8 total, 8 up, 7 in 2024-04-02T04:55:23.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:23.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:23.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:23.893 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:23.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:23.996 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:24.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:24.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:24.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.262057+0000 osd.7 (osd.7) 49 : cluster 0 10.1f scrub starts 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.264125+0000 osd.7 (osd.7) 50 : cluster 0 10.1f scrub ok 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.879996+0000 osd.4 (osd.4) 257 : cluster 0 4.1b scrub starts 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:22.882033+0000 osd.4 (osd.4) 258 : cluster 0 4.1b scrub ok 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: audit 2024-04-02T04:55:23.301247+0000 mgr.y (mgr.14538) 461 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:23.766151+0000 osd.2 (osd.2) 218 : cluster 0 2.c scrub starts 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:23.768434+0000 osd.2 (osd.2) 219 : cluster 0 2.c scrub ok 2024-04-02T04:55:24.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[21528]: cluster 2024-04-02T04:55:23.952575+0000 mon.a (mon.0) 2809 : cluster 0 osdmap e476: 8 total, 8 up, 7 in 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.262057+0000 osd.7 (osd.7) 49 : cluster 0 10.1f scrub starts 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.264125+0000 osd.7 (osd.7) 50 : cluster 0 10.1f scrub ok 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.879996+0000 osd.4 (osd.4) 257 : cluster 0 4.1b scrub starts 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:22.882033+0000 osd.4 (osd.4) 258 : cluster 0 4.1b scrub ok 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: audit 2024-04-02T04:55:23.301247+0000 mgr.y (mgr.14538) 461 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:23.766151+0000 osd.2 (osd.2) 218 : cluster 0 2.c scrub starts 2024-04-02T04:55:24.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:23.768434+0000 osd.2 (osd.2) 219 : cluster 0 2.c scrub ok 2024-04-02T04:55:24.258 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:24 smithi069 bash[28904]: cluster 2024-04-02T04:55:23.952575+0000 mon.a (mon.0) 2809 : cluster 0 osdmap e476: 8 total, 8 up, 7 in 2024-04-02T04:55:24.271 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:24.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.403 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.262057+0000 osd.7 (osd.7) 49 : cluster 0 10.1f scrub starts 2024-04-02T04:55:24.403 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.264125+0000 osd.7 (osd.7) 50 : cluster 0 10.1f scrub ok 2024-04-02T04:55:24.403 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.879996+0000 osd.4 (osd.4) 257 : cluster 0 4.1b scrub starts 2024-04-02T04:55:24.403 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:22.882033+0000 osd.4 (osd.4) 258 : cluster 0 4.1b scrub ok 2024-04-02T04:55:24.403 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: audit 2024-04-02T04:55:23.301247+0000 mgr.y (mgr.14538) 461 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:24.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:23.766151+0000 osd.2 (osd.2) 218 : cluster 0 2.c scrub starts 2024-04-02T04:55:24.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:23.768434+0000 osd.2 (osd.2) 219 : cluster 0 2.c scrub ok 2024-04-02T04:55:24.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:24 smithi098 bash[23790]: cluster 2024-04-02T04:55:23.952575+0000 mon.a (mon.0) 2809 : cluster 0 osdmap e476: 8 total, 8 up, 7 in 2024-04-02T04:55:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:24.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.529 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:24.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:24.660 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:24.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.793 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:24.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.927 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:24.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:24.967 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:55:25.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:25.158 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:25.183 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:25 smithi098 bash[23790]: cluster 2024-04-02T04:55:23.870857+0000 osd.6 (osd.6) 219 : cluster 0 10.1c scrub starts 2024-04-02T04:55:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:25 smithi098 bash[23790]: cluster 2024-04-02T04:55:23.873182+0000 osd.6 (osd.6) 220 : cluster 0 10.1c scrub ok 2024-04-02T04:55:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:25 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.130681+0000 mgr.y (mgr.14538) 462 : cluster 0 pgmap v695: 348 pgs: 2 clean+premerge+peered, 346 active+clean; 8.4 MiB data, 502 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:55:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:25 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.499665+0000 osd.1 (osd.1) 267 : cluster 0 2.6 scrub starts 2024-04-02T04:55:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:25 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.503210+0000 osd.1 (osd.1) 268 : cluster 0 2.6 scrub ok 2024-04-02T04:55:25.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:25 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.917893+0000 mon.a (mon.0) 2810 : cluster 0 osdmap e477: 8 total, 8 up, 7 in 2024-04-02T04:55:25.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:25.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[21528]: cluster 2024-04-02T04:55:23.870857+0000 osd.6 (osd.6) 219 : cluster 0 10.1c scrub starts 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[21528]: cluster 2024-04-02T04:55:23.873182+0000 osd.6 (osd.6) 220 : cluster 0 10.1c scrub ok 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.130681+0000 mgr.y (mgr.14538) 462 : cluster 0 pgmap v695: 348 pgs: 2 clean+premerge+peered, 346 active+clean; 8.4 MiB data, 502 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.499665+0000 osd.1 (osd.1) 267 : cluster 0 2.6 scrub starts 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.503210+0000 osd.1 (osd.1) 268 : cluster 0 2.6 scrub ok 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.917893+0000 mon.a (mon.0) 2810 : cluster 0 osdmap e477: 8 total, 8 up, 7 in 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[28904]: cluster 2024-04-02T04:55:23.870857+0000 osd.6 (osd.6) 219 : cluster 0 10.1c scrub starts 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[28904]: cluster 2024-04-02T04:55:23.873182+0000 osd.6 (osd.6) 220 : cluster 0 10.1c scrub ok 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.130681+0000 mgr.y (mgr.14538) 462 : cluster 0 pgmap v695: 348 pgs: 2 clean+premerge+peered, 346 active+clean; 8.4 MiB data, 502 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.499665+0000 osd.1 (osd.1) 267 : cluster 0 2.6 scrub starts 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.503210+0000 osd.1 (osd.1) 268 : cluster 0 2.6 scrub ok 2024-04-02T04:55:25.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:25 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.917893+0000 mon.a (mon.0) 2810 : cluster 0 osdmap e477: 8 total, 8 up, 7 in 2024-04-02T04:55:25.312 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:25.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:25.442 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:25.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- 172.21.15.69:0/978235736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b01011a0 msgr2=0x7ff6b0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/978235736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b01011a0 0x7ff6b0101620 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009990 tx=0x7ff6a4009ca0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- 172.21.15.69:0/978235736 shutdown_connections 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/978235736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b0101b60 0x7ff6b010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/978235736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0107220 0x7ff6b0107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/978235736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b01011a0 0x7ff6b0101620 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- 172.21.15.69:0/978235736 >> 172.21.15.69:0/978235736 conn(0x7ff6b00771e0 msgr2=0x7ff6b00775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- 172.21.15.69:0/978235736 shutdown_connections 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- 172.21.15.69:0/978235736 wait complete. 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 librados: starting msgr at 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 librados: starting objecter 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 Processor -- start 2024-04-02T04:55:25.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- start start 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 librados: setting wanted keys 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 librados: calling monclient init 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b01011a0 0x7ff6b0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 0x7ff6b0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b0107220 0x7ff6b019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0110980 con 0x7ff6b0107220 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0110800 con 0x7ff6b01011a0 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6b4a32700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b0110b00 con 0x7ff6b0101b60 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6add9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 0x7ff6b0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6aed9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b0107220 0x7ff6b019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6add9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 0x7ff6b0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40582/0 (socket says 172.21.15.69:40582) 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6add9b700 1 -- 172.21.15.69:0/319750940 learned_addr learned my addr 172.21.15.69:0/319750940 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:25.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6ae59c700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b01011a0 0x7ff6b0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6aed9d700 1 -- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 msgr2=0x7ff6b0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6aed9d700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 0x7ff6b0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6aed9d700 1 -- 172.21.15.69:0/319750940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b01011a0 msgr2=0x7ff6b0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6aed9d700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b01011a0 0x7ff6b0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6aed9d700 1 -- 172.21.15.69:0/319750940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b019d020 con 0x7ff6b0107220 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.570+0000 7ff6add9b700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 0x7ff6b0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6aed9d700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b0107220 0x7ff6b019c8e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8007f10 tx=0x7ff6a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 10 client.?.objecter ms_handle_connect 0x7ff6b0107220 2024-04-02T04:55:25.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8004db0 con 0x7ff6b0107220 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b019d310 con 0x7ff6b0107220 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff6a8009e20 con 0x7ff6b0107220 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8015400 con 0x7ff6b0107220 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6b4a32700 10 client.16494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b0102d50 con 0x7ff6b0107220 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6b4a32700 1 librados: init done 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b004ff00 con 0x7ff6b0107220 2024-04-02T04:55:25.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff6a800db20 con 0x7ff6b0107220 2024-04-02T04:55:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:25.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff69806f5e0 0x7ff698071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(477..477 src has 74..477) ==== 10953+0+0 (secure 0 0 0) 0x7ff6a808f020 con 0x7ff6b0107220 2024-04-02T04:55:25.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 10 client.16494.objecter ms_dispatch 0x7ff6b0001550 osd_map(477..477 src has 74..477) 2024-04-02T04:55:25.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 3 client.16494.objecter handle_osd_map got epochs [477,477] > 0 2024-04-02T04:55:25.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 3 client.16494.objecter handle_osd_map decoding full epoch 477 2024-04-02T04:55:25.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6ae59c700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff69806f5e0 0x7ff698071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 20 client.16494.objecter dump_active .. 0 homeless 2024-04-02T04:55:25.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff6ae59c700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff69806f5e0 0x7ff698071aa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0005950 tx=0x7ff6a000b410 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:25.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.574+0000 7ff69f7fe700 10 client.16494.objecter ms_handle_connect 0x7ff69806f5e0 2024-04-02T04:55:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:25.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.582+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff6a8010070 con 0x7ff6b0107220 2024-04-02T04:55:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 -- 172.21.15.69:0/1411451515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 msgr2=0x7f815c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 --2- 172.21.15.69:0/1411451515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c101620 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8150009990 tx=0x7f8150009ca0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 -- 172.21.15.69:0/1411451515 shutdown_connections 2024-04-02T04:55:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 --2- 172.21.15.69:0/1411451515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f815c101b60 0x7f815c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 --2- 172.21.15.69:0/1411451515 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f815c107220 0x7f815c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 --2- 172.21.15.69:0/1411451515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c101620 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 -- 172.21.15.69:0/1411451515 >> 172.21.15.69:0/1411451515 conn(0x7f815c0771e0 msgr2=0x7f815c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 -- 172.21.15.69:0/1411451515 shutdown_connections 2024-04-02T04:55:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 -- 172.21.15.69:0/1411451515 wait complete. 2024-04-02T04:55:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 librados: starting msgr at 2024-04-02T04:55:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 librados: starting objecter 2024-04-02T04:55:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 Processor -- start 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 -- start start 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 librados: setting wanted keys 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.598+0000 7f8161573700 1 librados: calling monclient init 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f815c101b60 0x7f815c1973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f815c107220 0x7f815c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f815c110930 con 0x7f815c1011a0 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f815c1107b0 con 0x7f815c107220 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f815c110ab0 con 0x7f815c101b60 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44146/0 (socket says 172.21.15.69:44146) 2024-04-02T04:55:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 -- 172.21.15.69:0/473758178 learned_addr learned my addr 172.21.15.69:0/473758178 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815a7fc700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f815c101b60 0x7f815c1973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 -- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f815c101b60 msgr2=0x7f815c1973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f815c101b60 0x7f815c1973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 -- 172.21.15.69:0/473758178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f815c107220 msgr2=0x7f815c19b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f815c107220 0x7f815c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 -- 172.21.15.69:0/473758178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815c19beb0 con 0x7f815c1011a0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f815affd700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c196ea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f815400eb30 tx=0x7f815400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 10 client.?.objecter ms_handle_connect 0x7f815c1011a0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815400cc90 con 0x7f815c1011a0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f815400ce30 con 0x7f815c1011a0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815c19c1a0 con 0x7f815c1011a0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154003df0 con 0x7f815c1011a0 2024-04-02T04:55:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 10 client.16500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:25.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f815c1a3a20 con 0x7f815c1011a0 2024-04-02T04:55:25.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 librados: init done 2024-04-02T04:55:25.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815c102860 con 0x7f815c1011a0 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8154004000 con 0x7f815c1011a0 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.602+0000 7f8143fff700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f814406f5e0 0x7f8144071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(477..477 src has 74..477) ==== 10953+0+0 (secure 0 0 0) 0x7f815408f160 con 0x7f815c1011a0 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f8143fff700 10 client.16500.objecter ms_dispatch 0x7f815c001550 osd_map(477..477 src has 74..477) 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f8143fff700 3 client.16500.objecter handle_osd_map got epochs [477,477] > 0 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f8143fff700 3 client.16500.objecter handle_osd_map decoding full epoch 477 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f8143fff700 20 client.16500.objecter dump_active .. 0 homeless 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f815a7fc700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f814406f5e0 0x7f8144071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f815a7fc700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f814406f5e0 0x7f8144071aa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f815000b2e0 tx=0x7f8150005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.606+0000 7f8143fff700 10 client.16500.objecter ms_handle_connect 0x7f814406f5e0 2024-04-02T04:55:25.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.610+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f815405f470 con 0x7f815c1011a0 2024-04-02T04:55:25.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:25.706 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:25.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.806+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ff6b0102fb0 con 0x7ff6b0107220 2024-04-02T04:55:25.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:25.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.814+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) -- 0x7f815c1a3cb0 con 0x7f815c1011a0 2024-04-02T04:55:25.835 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.906+0000 7ff69f7fe700 1 -- 172.21.15.69:0/319750940 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v478) ==== 101+0+0 (secure 0 0 0) 0x7ff6a805b8e0 con 0x7ff6b0107220 2024-04-02T04:55:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.906+0000 7f8143fff700 1 -- 172.21.15.69:0/473758178 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v478) ==== 129+0+0 (secure 0 0 0) 0x7f8154063ca0 con 0x7f815c1011a0 2024-04-02T04:55:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7f8161573700 10 librados: watch_flush enter 2024-04-02T04:55:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7f8161573700 10 librados: watch_flush exit 2024-04-02T04:55:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7f8161573700 20 client.16500.objecter shutdown clearing up homeless session... 2024-04-02T04:55:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7f8161573700 10 client.16500.objecter successfully canceled tick 2024-04-02T04:55:25.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 10 librados: watch_flush enter 2024-04-02T04:55:25.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 10 librados: watch_flush exit 2024-04-02T04:55:25.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 20 client.16494.objecter shutdown clearing up homeless session... 2024-04-02T04:55:25.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 10 client.16494.objecter successfully canceled tick 2024-04-02T04:55:25.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff69806f5e0 msgr2=0x7ff698071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff69806f5e0 0x7ff698071aa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0005950 tx=0x7ff6a000b410 comp rx=0 tx=0).stop 2024-04-02T04:55:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b0107220 msgr2=0x7ff6b019c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b0107220 0x7ff6b019c8e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8007f10 tx=0x7ff6a800d3b0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 shutdown_connections 2024-04-02T04:55:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff69806f5e0 0x7ff698071aa0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6b0107220 0x7ff6b019c8e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b01011a0 0x7ff6b0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 --2- 172.21.15.69:0/319750940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6b0101b60 0x7ff6b0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.910+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 >> 172.21.15.69:0/319750940 conn(0x7ff6b00771e0 msgr2=0x7ff6b0069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:25.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 shutdown_connections 2024-04-02T04:55:25.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7ff6b4a32700 1 -- 172.21.15.69:0/319750940 wait complete. 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f814406f5e0 msgr2=0x7f8144071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f814406f5e0 0x7f8144071aa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f815000b2e0 tx=0x7f8150005960 comp rx=0 tx=0).stop 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 msgr2=0x7f815c196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c196ea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f815400eb30 tx=0x7f815400ee40 comp rx=0 tx=0).stop 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7ff6b4a32700 1 librados: shutdown 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 shutdown_connections 2024-04-02T04:55:25.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f814406f5e0 0x7f8144071aa0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f815c107220 0x7f815c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f815c1011a0 0x7f815c196ea0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 --2- 172.21.15.69:0/473758178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f815c101b60 0x7f815c1973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 >> 172.21.15.69:0/473758178 conn(0x7f815c0771e0 msgr2=0x7f815c107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 shutdown_connections 2024-04-02T04:55:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 -- 172.21.15.69:0/473758178 wait complete. 2024-04-02T04:55:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:55:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:25.918+0000 7f8161573700 1 librados: shutdown 2024-04-02T04:55:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:set pool 1 pg_num to 11 2024-04-02T04:55:25.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:25.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:26.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:55:26.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:26.129 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.803436+0000 osd.4 (osd.4) 259 : cluster 0 6.1e scrub starts 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.804930+0000 osd.4 (osd.4) 260 : cluster 0 6.1e scrub ok 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.888397+0000 osd.6 (osd.6) 221 : cluster 0 10.12 scrub starts 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: cluster 2024-04-02T04:55:24.890311+0000 osd.6 (osd.6) 222 : cluster 0 10.12 scrub ok 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:25.811631+0000 mon.a (mon.0) 2811 : audit 1 from='client.? 172.21.15.69:0/319750940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:25.818948+0000 mon.a (mon.0) 2812 : audit 1 from='client.? 172.21.15.69:0/473758178' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: cluster 2024-04-02T04:55:25.907045+0000 mon.a (mon.0) 2813 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:55:26.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:25.909076+0000 mon.a (mon.0) 2814 : audit 1 from='client.? 172.21.15.69:0/319750940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:25.909227+0000 mon.a (mon.0) 2815 : audit 1 from='client.? 172.21.15.69:0/473758178' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: cluster 2024-04-02T04:55:25.915755+0000 mon.a (mon.0) 2816 : cluster 0 osdmap e478: 8 total, 8 up, 7 in 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:25.987997+0000 mon.c (mon.2) 839 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:26.127441+0000 mon.b (mon.1) 247 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:26.128007+0000 mon.b (mon.1) 248 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:26.133430+0000 mon.a (mon.0) 2817 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:26.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:26 smithi098 bash[23790]: audit 2024-04-02T04:55:26.133731+0000 mon.a (mon.0) 2818 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-02T04:55:26.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:26.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.803436+0000 osd.4 (osd.4) 259 : cluster 0 6.1e scrub starts 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.804930+0000 osd.4 (osd.4) 260 : cluster 0 6.1e scrub ok 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.888397+0000 osd.6 (osd.6) 221 : cluster 0 10.12 scrub starts 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: cluster 2024-04-02T04:55:24.890311+0000 osd.6 (osd.6) 222 : cluster 0 10.12 scrub ok 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:25.811631+0000 mon.a (mon.0) 2811 : audit 1 from='client.? 172.21.15.69:0/319750940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:25.818948+0000 mon.a (mon.0) 2812 : audit 1 from='client.? 172.21.15.69:0/473758178' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: cluster 2024-04-02T04:55:25.907045+0000 mon.a (mon.0) 2813 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:25.909076+0000 mon.a (mon.0) 2814 : audit 1 from='client.? 172.21.15.69:0/319750940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:25.909227+0000 mon.a (mon.0) 2815 : audit 1 from='client.? 172.21.15.69:0/473758178' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: cluster 2024-04-02T04:55:25.915755+0000 mon.a (mon.0) 2816 : cluster 0 osdmap e478: 8 total, 8 up, 7 in 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:25.987997+0000 mon.c (mon.2) 839 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:26.127441+0000 mon.b (mon.1) 247 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:26.128007+0000 mon.b (mon.1) 248 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-02T04:55:26.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:26.133430+0000 mon.a (mon.0) 2817 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:26.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[21528]: audit 2024-04-02T04:55:26.133731+0000 mon.a (mon.0) 2818 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-02T04:55:26.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.803436+0000 osd.4 (osd.4) 259 : cluster 0 6.1e scrub starts 2024-04-02T04:55:26.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.804930+0000 osd.4 (osd.4) 260 : cluster 0 6.1e scrub ok 2024-04-02T04:55:26.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.888397+0000 osd.6 (osd.6) 221 : cluster 0 10.12 scrub starts 2024-04-02T04:55:26.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: cluster 2024-04-02T04:55:24.890311+0000 osd.6 (osd.6) 222 : cluster 0 10.12 scrub ok 2024-04-02T04:55:26.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:25.811631+0000 mon.a (mon.0) 2811 : audit 1 from='client.? 172.21.15.69:0/319750940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:26.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:25.818948+0000 mon.a (mon.0) 2812 : audit 1 from='client.? 172.21.15.69:0/473758178' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-02T04:55:26.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: cluster 2024-04-02T04:55:25.907045+0000 mon.a (mon.0) 2813 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:55:26.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:25.909076+0000 mon.a (mon.0) 2814 : audit 1 from='client.? 172.21.15.69:0/319750940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:55:26.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:25.909227+0000 mon.a (mon.0) 2815 : audit 1 from='client.? 172.21.15.69:0/473758178' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-02T04:55:26.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: cluster 2024-04-02T04:55:25.915755+0000 mon.a (mon.0) 2816 : cluster 0 osdmap e478: 8 total, 8 up, 7 in 2024-04-02T04:55:26.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:25.987997+0000 mon.c (mon.2) 839 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:26.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:26.127441+0000 mon.b (mon.1) 247 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:26.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:26.128007+0000 mon.b (mon.1) 248 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-02T04:55:26.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:26.133430+0000 mon.a (mon.0) 2817 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:26.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:26 smithi069 bash[28904]: audit 2024-04-02T04:55:26.133731+0000 mon.a (mon.0) 2818 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-02T04:55:26.249 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:26.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:26.373 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:26.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:26.503 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:26.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:26.626 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:26.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:26.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:26.887 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:26.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:26.990 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:27.017 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:27.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:27.211 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:27.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.097326+0000 osd.3 (osd.3) 297 : cluster 0 43.4 scrub starts 2024-04-02T04:55:27.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.098482+0000 osd.3 (osd.3) 298 : cluster 0 43.4 scrub ok 2024-04-02T04:55:27.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.131247+0000 mgr.y (mgr.14538) 463 : cluster 0 pgmap v698: 347 pgs: 1 clean+premerge+peered, 346 active+clean; 8.4 MiB data, 502 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T04:55:27.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.483703+0000 osd.1 (osd.1) 269 : cluster 0 2.9 deep-scrub starts 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.486656+0000 osd.1 (osd.1) 270 : cluster 0 2.9 deep-scrub ok 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: audit 2024-04-02T04:55:26.911783+0000 mon.a (mon.0) 2819 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: audit 2024-04-02T04:55:26.911974+0000 mon.a (mon.0) 2820 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.921253+0000 mon.a (mon.0) 2821 : cluster 0 osdmap e479: 8 total, 8 up, 7 in 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[21528]: audit 2024-04-02T04:55:26.989036+0000 mon.c (mon.2) 840 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.097326+0000 osd.3 (osd.3) 297 : cluster 0 43.4 scrub starts 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.098482+0000 osd.3 (osd.3) 298 : cluster 0 43.4 scrub ok 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.131247+0000 mgr.y (mgr.14538) 463 : cluster 0 pgmap v698: 347 pgs: 1 clean+premerge+peered, 346 active+clean; 8.4 MiB data, 502 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.483703+0000 osd.1 (osd.1) 269 : cluster 0 2.9 deep-scrub starts 2024-04-02T04:55:27.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.486656+0000 osd.1 (osd.1) 270 : cluster 0 2.9 deep-scrub ok 2024-04-02T04:55:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: audit 2024-04-02T04:55:26.911783+0000 mon.a (mon.0) 2819 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-02T04:55:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: audit 2024-04-02T04:55:26.911974+0000 mon.a (mon.0) 2820 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-02T04:55:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.921253+0000 mon.a (mon.0) 2821 : cluster 0 osdmap e479: 8 total, 8 up, 7 in 2024-04-02T04:55:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:27 smithi069 bash[28904]: audit 2024-04-02T04:55:26.989036+0000 mon.c (mon.2) 840 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:27.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:27.340 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:27.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:27.470 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.097326+0000 osd.3 (osd.3) 297 : cluster 0 43.4 scrub starts 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.098482+0000 osd.3 (osd.3) 298 : cluster 0 43.4 scrub ok 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.131247+0000 mgr.y (mgr.14538) 463 : cluster 0 pgmap v698: 347 pgs: 1 clean+premerge+peered, 346 active+clean; 8.4 MiB data, 502 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.483703+0000 osd.1 (osd.1) 269 : cluster 0 2.9 deep-scrub starts 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.486656+0000 osd.1 (osd.1) 270 : cluster 0 2.9 deep-scrub ok 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: audit 2024-04-02T04:55:26.911783+0000 mon.a (mon.0) 2819 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: audit 2024-04-02T04:55:26.911974+0000 mon.a (mon.0) 2820 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.921253+0000 mon.a (mon.0) 2821 : cluster 0 osdmap e479: 8 total, 8 up, 7 in 2024-04-02T04:55:27.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:27 smithi098 bash[23790]: audit 2024-04-02T04:55:26.989036+0000 mon.c (mon.2) 840 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:27.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:27.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:27.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:27.856 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:27.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:27.983 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:27.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:28.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:28.107 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:28.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:28.241 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.898554+0000 osd.6 (osd.6) 223 : cluster 0 10.1a scrub starts 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:26.900127+0000 osd.6 (osd.6) 224 : cluster 0 10.1a scrub ok 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.018612+0000 osd.0 (osd.0) 247 : cluster 0 4.3 scrub starts 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.022868+0000 osd.0 (osd.0) 248 : cluster 0 4.3 scrub ok 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.079625+0000 osd.3 (osd.3) 299 : cluster 0 43.9 scrub starts 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.081134+0000 osd.3 (osd.3) 300 : cluster 0 43.9 scrub ok 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.940412+0000 mon.a (mon.0) 2822 : cluster 0 osdmap e480: 8 total, 8 up, 7 in 2024-04-02T04:55:28.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: audit 2024-04-02T04:55:27.990598+0000 mon.c (mon.2) 841 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.898554+0000 osd.6 (osd.6) 223 : cluster 0 10.1a scrub starts 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:26.900127+0000 osd.6 (osd.6) 224 : cluster 0 10.1a scrub ok 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.018612+0000 osd.0 (osd.0) 247 : cluster 0 4.3 scrub starts 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.022868+0000 osd.0 (osd.0) 248 : cluster 0 4.3 scrub ok 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.079625+0000 osd.3 (osd.3) 299 : cluster 0 43.9 scrub starts 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.081134+0000 osd.3 (osd.3) 300 : cluster 0 43.9 scrub ok 2024-04-02T04:55:28.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.940412+0000 mon.a (mon.0) 2822 : cluster 0 osdmap e480: 8 total, 8 up, 7 in 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: audit 2024-04-02T04:55:27.990598+0000 mon.c (mon.2) 841 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: audit 2024-04-02T04:55:28.128927+0000 mon.b (mon.1) 249 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21528]: audit 2024-04-02T04:55:28.134473+0000 mon.a (mon.0) 2823 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.898554+0000 osd.6 (osd.6) 223 : cluster 0 10.1a scrub starts 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:26.900127+0000 osd.6 (osd.6) 224 : cluster 0 10.1a scrub ok 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.018612+0000 osd.0 (osd.0) 247 : cluster 0 4.3 scrub starts 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.022868+0000 osd.0 (osd.0) 248 : cluster 0 4.3 scrub ok 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.079625+0000 osd.3 (osd.3) 299 : cluster 0 43.9 scrub starts 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.081134+0000 osd.3 (osd.3) 300 : cluster 0 43.9 scrub ok 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.940412+0000 mon.a (mon.0) 2822 : cluster 0 osdmap e480: 8 total, 8 up, 7 in 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: audit 2024-04-02T04:55:27.990598+0000 mon.c (mon.2) 841 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:28.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: audit 2024-04-02T04:55:28.128927+0000 mon.b (mon.1) 249 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:28.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[28904]: audit 2024-04-02T04:55:28.134473+0000 mon.a (mon.0) 2823 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:28.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:28.372 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:28.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:28.501 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:28.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:28.632 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:28.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: audit 2024-04-02T04:55:28.128927+0000 mon.b (mon.1) 249 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:28.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:28 smithi098 bash[23790]: audit 2024-04-02T04:55:28.134473+0000 mon.a (mon.0) 2823 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-02T04:55:28.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:28.764 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:28.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:28.876 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:55:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:55:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:55:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:28.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:28.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:29.003 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:55:29.003 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:55:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:29.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:29.149 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:29.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:29.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:29.395 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.703001+0000 osd.4 (osd.4) 261 : cluster 0 3.3 scrub starts 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.706541+0000 osd.4 (osd.4) 262 : cluster 0 3.3 scrub ok 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.994032+0000 osd.0 (osd.0) 249 : cluster 0 40.18 scrub starts 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:27.995285+0000 osd.0 (osd.0) 250 : cluster 0 40.18 scrub ok 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.035719+0000 osd.3 (osd.3) 301 : cluster 0 41.1 deep-scrub starts 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.036835+0000 osd.3 (osd.3) 302 : cluster 0 41.1 deep-scrub ok 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.132705+0000 mgr.y (mgr.14538) 464 : cluster 0 pgmap v701: 357 pgs: 2 peering, 10 unknown, 1 clean+premerge+peered, 344 active+clean; 8.4 MiB data, 503 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: audit 2024-04-02T04:55:28.940044+0000 mon.a (mon.0) 2824 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.943599+0000 mon.a (mon.0) 2825 : cluster 0 osdmap e481: 8 total, 8 up, 7 in 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[28904]: audit 2024-04-02T04:55:28.991938+0000 mon.c (mon.2) 842 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:29.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.703001+0000 osd.4 (osd.4) 261 : cluster 0 3.3 scrub starts 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.706541+0000 osd.4 (osd.4) 262 : cluster 0 3.3 scrub ok 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.994032+0000 osd.0 (osd.0) 249 : cluster 0 40.18 scrub starts 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:27.995285+0000 osd.0 (osd.0) 250 : cluster 0 40.18 scrub ok 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.035719+0000 osd.3 (osd.3) 301 : cluster 0 41.1 deep-scrub starts 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.036835+0000 osd.3 (osd.3) 302 : cluster 0 41.1 deep-scrub ok 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.132705+0000 mgr.y (mgr.14538) 464 : cluster 0 pgmap v701: 357 pgs: 2 peering, 10 unknown, 1 clean+premerge+peered, 344 active+clean; 8.4 MiB data, 503 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: audit 2024-04-02T04:55:28.940044+0000 mon.a (mon.0) 2824 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.943599+0000 mon.a (mon.0) 2825 : cluster 0 osdmap e481: 8 total, 8 up, 7 in 2024-04-02T04:55:29.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:29 smithi069 bash[21528]: audit 2024-04-02T04:55:28.991938+0000 mon.c (mon.2) 842 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.703001+0000 osd.4 (osd.4) 261 : cluster 0 3.3 scrub starts 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.706541+0000 osd.4 (osd.4) 262 : cluster 0 3.3 scrub ok 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.994032+0000 osd.0 (osd.0) 249 : cluster 0 40.18 scrub starts 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:27.995285+0000 osd.0 (osd.0) 250 : cluster 0 40.18 scrub ok 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.035719+0000 osd.3 (osd.3) 301 : cluster 0 41.1 deep-scrub starts 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.036835+0000 osd.3 (osd.3) 302 : cluster 0 41.1 deep-scrub ok 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.132705+0000 mgr.y (mgr.14538) 464 : cluster 0 pgmap v701: 357 pgs: 2 peering, 10 unknown, 1 clean+premerge+peered, 344 active+clean; 8.4 MiB data, 503 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: audit 2024-04-02T04:55:28.940044+0000 mon.a (mon.0) 2824 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-02T04:55:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.943599+0000 mon.a (mon.0) 2825 : cluster 0 osdmap e481: 8 total, 8 up, 7 in 2024-04-02T04:55:29.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:29 smithi098 bash[23790]: audit 2024-04-02T04:55:28.991938+0000 mon.c (mon.2) 842 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:29.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:29.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:29.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:29.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:55:29.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:29.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-02T04:55:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:29.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:29.935 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:29.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:30.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:30.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:30.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:30.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:30.426 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:30.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.183103+0000 osd.7 (osd.7) 51 : cluster 0 1.1 deep-scrub starts 2024-04-02T04:55:30.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.185540+0000 osd.7 (osd.7) 52 : cluster 0 1.1 deep-scrub ok 2024-04-02T04:55:30.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.923628+0000 osd.6 (osd.6) 225 : cluster 0 13.11 scrub starts 2024-04-02T04:55:30.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.925477+0000 osd.6 (osd.6) 226 : cluster 0 13.11 scrub ok 2024-04-02T04:55:30.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.978032+0000 osd.0 (osd.0) 251 : cluster 0 10.8 scrub starts 2024-04-02T04:55:30.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:28.979320+0000 osd.0 (osd.0) 252 : cluster 0 10.8 scrub ok 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:29.086990+0000 osd.3 (osd.3) 303 : cluster 0 13.2 scrub starts 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:29.088340+0000 osd.3 (osd.3) 304 : cluster 0 13.2 scrub ok 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:29.147003+0000 osd.7 (osd.7) 53 : cluster 0 1.0 scrub starts 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:29.149124+0000 osd.7 (osd.7) 54 : cluster 0 1.0 scrub ok 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: cluster 2024-04-02T04:55:29.911324+0000 mon.a (mon.0) 2826 : cluster 0 osdmap e482: 8 total, 8 up, 7 in 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: audit 2024-04-02T04:55:29.993263+0000 mon.c (mon.2) 843 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: audit 2024-04-02T04:55:30.129990+0000 mon.b (mon.1) 250 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[28904]: audit 2024-04-02T04:55:30.137300+0000 mon.a (mon.0) 2827 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T04:55:30.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.183103+0000 osd.7 (osd.7) 51 : cluster 0 1.1 deep-scrub starts 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.185540+0000 osd.7 (osd.7) 52 : cluster 0 1.1 deep-scrub ok 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.923628+0000 osd.6 (osd.6) 225 : cluster 0 13.11 scrub starts 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.925477+0000 osd.6 (osd.6) 226 : cluster 0 13.11 scrub ok 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.978032+0000 osd.0 (osd.0) 251 : cluster 0 10.8 scrub starts 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:28.979320+0000 osd.0 (osd.0) 252 : cluster 0 10.8 scrub ok 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:29.086990+0000 osd.3 (osd.3) 303 : cluster 0 13.2 scrub starts 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:29.088340+0000 osd.3 (osd.3) 304 : cluster 0 13.2 scrub ok 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:29.147003+0000 osd.7 (osd.7) 53 : cluster 0 1.0 scrub starts 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:29.149124+0000 osd.7 (osd.7) 54 : cluster 0 1.0 scrub ok 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: cluster 2024-04-02T04:55:29.911324+0000 mon.a (mon.0) 2826 : cluster 0 osdmap e482: 8 total, 8 up, 7 in 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: audit 2024-04-02T04:55:29.993263+0000 mon.c (mon.2) 843 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: audit 2024-04-02T04:55:30.129990+0000 mon.b (mon.1) 250 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T04:55:30.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:30 smithi069 bash[21528]: audit 2024-04-02T04:55:30.137300+0000 mon.a (mon.0) 2827 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T04:55:30.453 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.183103+0000 osd.7 (osd.7) 51 : cluster 0 1.1 deep-scrub starts 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.185540+0000 osd.7 (osd.7) 52 : cluster 0 1.1 deep-scrub ok 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.923628+0000 osd.6 (osd.6) 225 : cluster 0 13.11 scrub starts 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.925477+0000 osd.6 (osd.6) 226 : cluster 0 13.11 scrub ok 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.978032+0000 osd.0 (osd.0) 251 : cluster 0 10.8 scrub starts 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:28.979320+0000 osd.0 (osd.0) 252 : cluster 0 10.8 scrub ok 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:29.086990+0000 osd.3 (osd.3) 303 : cluster 0 13.2 scrub starts 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:29.088340+0000 osd.3 (osd.3) 304 : cluster 0 13.2 scrub ok 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:29.147003+0000 osd.7 (osd.7) 53 : cluster 0 1.0 scrub starts 2024-04-02T04:55:30.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:29.149124+0000 osd.7 (osd.7) 54 : cluster 0 1.0 scrub ok 2024-04-02T04:55:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: cluster 2024-04-02T04:55:29.911324+0000 mon.a (mon.0) 2826 : cluster 0 osdmap e482: 8 total, 8 up, 7 in 2024-04-02T04:55:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: audit 2024-04-02T04:55:29.993263+0000 mon.c (mon.2) 843 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: audit 2024-04-02T04:55:30.129990+0000 mon.b (mon.1) 250 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T04:55:30.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:30 smithi098 bash[23790]: audit 2024-04-02T04:55:30.137300+0000 mon.a (mon.0) 2827 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T04:55:30.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:30.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:30.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:30.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:30.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:30.821 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:30.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:30.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:31.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:31.100 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:31.203 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:31.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:31.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:31.365 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:31.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:31.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.133666+0000 mgr.y (mgr.14538) 465 : cluster 0 pgmap v704: 356 pgs: 1 peering, 10 unknown, 345 active+clean; 8.0 MiB data, 503 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:31.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.146705+0000 osd.7 (osd.7) 55 : cluster 0 43.1f scrub starts 2024-04-02T04:55:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.148942+0000 osd.7 (osd.7) 56 : cluster 0 43.1f scrub ok 2024-04-02T04:55:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.514304+0000 osd.1 (osd.1) 271 : cluster 0 102.e scrub starts 2024-04-02T04:55:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.517094+0000 osd.1 (osd.1) 272 : cluster 0 102.e scrub ok 2024-04-02T04:55:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: audit 2024-04-02T04:55:30.923147+0000 mon.a (mon.0) 2828 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-02T04:55:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.930378+0000 mon.a (mon.0) 2829 : cluster 0 osdmap e483: 8 total, 8 up, 7 in 2024-04-02T04:55:31.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:31 smithi098 bash[23790]: audit 2024-04-02T04:55:30.994641+0000 mon.c (mon.2) 844 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:31.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.133666+0000 mgr.y (mgr.14538) 465 : cluster 0 pgmap v704: 356 pgs: 1 peering, 10 unknown, 345 active+clean; 8.0 MiB data, 503 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.146705+0000 osd.7 (osd.7) 55 : cluster 0 43.1f scrub starts 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.148942+0000 osd.7 (osd.7) 56 : cluster 0 43.1f scrub ok 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.514304+0000 osd.1 (osd.1) 271 : cluster 0 102.e scrub starts 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.517094+0000 osd.1 (osd.1) 272 : cluster 0 102.e scrub ok 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: audit 2024-04-02T04:55:30.923147+0000 mon.a (mon.0) 2828 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.930378+0000 mon.a (mon.0) 2829 : cluster 0 osdmap e483: 8 total, 8 up, 7 in 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[21528]: audit 2024-04-02T04:55:30.994641+0000 mon.c (mon.2) 844 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:31.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.133666+0000 mgr.y (mgr.14538) 465 : cluster 0 pgmap v704: 356 pgs: 1 peering, 10 unknown, 345 active+clean; 8.0 MiB data, 503 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.146705+0000 osd.7 (osd.7) 55 : cluster 0 43.1f scrub starts 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.148942+0000 osd.7 (osd.7) 56 : cluster 0 43.1f scrub ok 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.514304+0000 osd.1 (osd.1) 271 : cluster 0 102.e scrub starts 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.517094+0000 osd.1 (osd.1) 272 : cluster 0 102.e scrub ok 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: audit 2024-04-02T04:55:30.923147+0000 mon.a (mon.0) 2828 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.930378+0000 mon.a (mon.0) 2829 : cluster 0 osdmap e483: 8 total, 8 up, 7 in 2024-04-02T04:55:31.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:31 smithi069 bash[28904]: audit 2024-04-02T04:55:30.994641+0000 mon.c (mon.2) 844 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:31.600 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:31.629 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:31.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:31.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:31.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:31.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:32.018 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:32.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:32.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:32.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.995455+0000 osd.0 (osd.0) 253 : cluster 0 40.1a scrub starts 2024-04-02T04:55:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:30.997603+0000 osd.0 (osd.0) 254 : cluster 0 40.1a scrub ok 2024-04-02T04:55:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:31.069992+0000 osd.3 (osd.3) 305 : cluster 0 40.2 scrub starts 2024-04-02T04:55:32.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:31.072388+0000 osd.3 (osd.3) 306 : cluster 0 40.2 scrub ok 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:31.493163+0000 osd.1 (osd.1) 273 : cluster 0 10.19 scrub starts 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:31.495507+0000 osd.1 (osd.1) 274 : cluster 0 10.19 scrub ok 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: cluster 2024-04-02T04:55:31.962460+0000 mon.a (mon.0) 2830 : cluster 0 osdmap e484: 8 total, 8 up, 7 in 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[21528]: audit 2024-04-02T04:55:31.996388+0000 mon.c (mon.2) 845 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.995455+0000 osd.0 (osd.0) 253 : cluster 0 40.1a scrub starts 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:30.997603+0000 osd.0 (osd.0) 254 : cluster 0 40.1a scrub ok 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:31.069992+0000 osd.3 (osd.3) 305 : cluster 0 40.2 scrub starts 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:31.072388+0000 osd.3 (osd.3) 306 : cluster 0 40.2 scrub ok 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:31.493163+0000 osd.1 (osd.1) 273 : cluster 0 10.19 scrub starts 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:31.495507+0000 osd.1 (osd.1) 274 : cluster 0 10.19 scrub ok 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: cluster 2024-04-02T04:55:31.962460+0000 mon.a (mon.0) 2830 : cluster 0 osdmap e484: 8 total, 8 up, 7 in 2024-04-02T04:55:32.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:32 smithi069 bash[28904]: audit 2024-04-02T04:55:31.996388+0000 mon.c (mon.2) 845 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:32.280 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:32.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:32.404 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.995455+0000 osd.0 (osd.0) 253 : cluster 0 40.1a scrub starts 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:30.997603+0000 osd.0 (osd.0) 254 : cluster 0 40.1a scrub ok 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:31.069992+0000 osd.3 (osd.3) 305 : cluster 0 40.2 scrub starts 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:31.072388+0000 osd.3 (osd.3) 306 : cluster 0 40.2 scrub ok 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:31.493163+0000 osd.1 (osd.1) 273 : cluster 0 10.19 scrub starts 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:31.495507+0000 osd.1 (osd.1) 274 : cluster 0 10.19 scrub ok 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: cluster 2024-04-02T04:55:31.962460+0000 mon.a (mon.0) 2830 : cluster 0 osdmap e484: 8 total, 8 up, 7 in 2024-04-02T04:55:32.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:32 smithi098 bash[23790]: audit 2024-04-02T04:55:31.996388+0000 mon.c (mon.2) 845 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:32.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:32.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:32.664 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:32.696 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:55:32.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:55:32.696 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-02T04:55:32.696 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-02T04:55:32.696 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-02T04:55:32.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-02T04:55:32.767 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:32.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:32.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:32.914 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:32.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:33.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:33.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:33.175 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:33.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:33.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: cluster 2024-04-02T04:55:32.134641+0000 mgr.y (mgr.14538) 466 : cluster 0 pgmap v707: 356 pgs: 1 peering, 10 unknown, 345 active+clean; 8.0 MiB data, 503 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: cluster 2024-04-02T04:55:32.578372+0000 osd.4 (osd.4) 263 : cluster 0 3.1a scrub starts 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: cluster 2024-04-02T04:55:32.581154+0000 osd.4 (osd.4) 264 : cluster 0 3.1a scrub ok 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: cluster 2024-04-02T04:55:32.849328+0000 osd.2 (osd.2) 220 : cluster 0 13.8 scrub starts 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: cluster 2024-04-02T04:55:32.851309+0000 osd.2 (osd.2) 221 : cluster 0 13.8 scrub ok 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: cluster 2024-04-02T04:55:32.968450+0000 mon.a (mon.0) 2831 : cluster 0 osdmap e485: 8 total, 8 up, 7 in 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: audit 2024-04-02T04:55:32.998240+0000 mon.c (mon.2) 846 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: audit 2024-04-02T04:55:33.096954+0000 mon.b (mon.1) 251 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[21528]: audit 2024-04-02T04:55:33.098262+0000 mon.a (mon.0) 2832 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:33.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: cluster 2024-04-02T04:55:32.134641+0000 mgr.y (mgr.14538) 466 : cluster 0 pgmap v707: 356 pgs: 1 peering, 10 unknown, 345 active+clean; 8.0 MiB data, 503 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: cluster 2024-04-02T04:55:32.578372+0000 osd.4 (osd.4) 263 : cluster 0 3.1a scrub starts 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: cluster 2024-04-02T04:55:32.581154+0000 osd.4 (osd.4) 264 : cluster 0 3.1a scrub ok 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: cluster 2024-04-02T04:55:32.849328+0000 osd.2 (osd.2) 220 : cluster 0 13.8 scrub starts 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: cluster 2024-04-02T04:55:32.851309+0000 osd.2 (osd.2) 221 : cluster 0 13.8 scrub ok 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: cluster 2024-04-02T04:55:32.968450+0000 mon.a (mon.0) 2831 : cluster 0 osdmap e485: 8 total, 8 up, 7 in 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: audit 2024-04-02T04:55:32.998240+0000 mon.c (mon.2) 846 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: audit 2024-04-02T04:55:33.096954+0000 mon.b (mon.1) 251 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:33.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:33 smithi069 bash[28904]: audit 2024-04-02T04:55:33.098262+0000 mon.a (mon.0) 2832 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:33.305 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:33.307 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: cluster 2024-04-02T04:55:32.134641+0000 mgr.y (mgr.14538) 466 : cluster 0 pgmap v707: 356 pgs: 1 peering, 10 unknown, 345 active+clean; 8.0 MiB data, 503 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: cluster 2024-04-02T04:55:32.578372+0000 osd.4 (osd.4) 263 : cluster 0 3.1a scrub starts 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: cluster 2024-04-02T04:55:32.581154+0000 osd.4 (osd.4) 264 : cluster 0 3.1a scrub ok 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: cluster 2024-04-02T04:55:32.849328+0000 osd.2 (osd.2) 220 : cluster 0 13.8 scrub starts 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: cluster 2024-04-02T04:55:32.851309+0000 osd.2 (osd.2) 221 : cluster 0 13.8 scrub ok 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: cluster 2024-04-02T04:55:32.968450+0000 mon.a (mon.0) 2831 : cluster 0 osdmap e485: 8 total, 8 up, 7 in 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: audit 2024-04-02T04:55:32.998240+0000 mon.c (mon.2) 846 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: audit 2024-04-02T04:55:33.096954+0000 mon.b (mon.1) 251 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:33.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[23790]: audit 2024-04-02T04:55:33.098262+0000 mon.a (mon.0) 2832 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:33.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:33.419 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:55:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:55:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:33.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:33.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:33.696 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:33.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:33.825 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:33.864 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:33.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:33.955 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:33.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:34.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:34.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:34.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:34.213 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.024786+0000 osd.6 (osd.6) 227 : cluster 0 1.6 scrub starts 2024-04-02T04:55:34.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.026861+0000 osd.6 (osd.6) 228 : cluster 0 1.6 scrub ok 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.107736+0000 osd.3 (osd.3) 307 : cluster 0 1.4 scrub starts 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.110637+0000 osd.3 (osd.3) 308 : cluster 0 1.4 scrub ok 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: audit 2024-04-02T04:55:33.310016+0000 mgr.y (mgr.14538) 467 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.447502+0000 osd.1 (osd.1) 275 : cluster 0 1.3 scrub starts 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.448762+0000 osd.1 (osd.1) 276 : cluster 0 1.3 scrub ok 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.823347+0000 osd.2 (osd.2) 222 : cluster 0 1.8 scrub starts 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.825327+0000 osd.2 (osd.2) 223 : cluster 0 1.8 scrub ok 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.967118+0000 mon.a (mon.0) 2833 : cluster 0 osdmap e486: 8 total, 8 up, 7 in 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: audit 2024-04-02T04:55:33.999443+0000 mon.c (mon.2) 847 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: audit 2024-04-02T04:55:34.132437+0000 mon.b (mon.1) 252 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-02T04:55:34.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[21528]: audit 2024-04-02T04:55:34.139863+0000 mon.a (mon.0) 2834 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.024786+0000 osd.6 (osd.6) 227 : cluster 0 1.6 scrub starts 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.026861+0000 osd.6 (osd.6) 228 : cluster 0 1.6 scrub ok 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.107736+0000 osd.3 (osd.3) 307 : cluster 0 1.4 scrub starts 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.110637+0000 osd.3 (osd.3) 308 : cluster 0 1.4 scrub ok 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: audit 2024-04-02T04:55:33.310016+0000 mgr.y (mgr.14538) 467 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.447502+0000 osd.1 (osd.1) 275 : cluster 0 1.3 scrub starts 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.448762+0000 osd.1 (osd.1) 276 : cluster 0 1.3 scrub ok 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.823347+0000 osd.2 (osd.2) 222 : cluster 0 1.8 scrub starts 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.825327+0000 osd.2 (osd.2) 223 : cluster 0 1.8 scrub ok 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.967118+0000 mon.a (mon.0) 2833 : cluster 0 osdmap e486: 8 total, 8 up, 7 in 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: audit 2024-04-02T04:55:33.999443+0000 mon.c (mon.2) 847 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: audit 2024-04-02T04:55:34.132437+0000 mon.b (mon.1) 252 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-02T04:55:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:34 smithi069 bash[28904]: audit 2024-04-02T04:55:34.139863+0000 mon.a (mon.0) 2834 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-02T04:55:34.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:34.344 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:34.447 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:34.471 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.024786+0000 osd.6 (osd.6) 227 : cluster 0 1.6 scrub starts 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.026861+0000 osd.6 (osd.6) 228 : cluster 0 1.6 scrub ok 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.107736+0000 osd.3 (osd.3) 307 : cluster 0 1.4 scrub starts 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.110637+0000 osd.3 (osd.3) 308 : cluster 0 1.4 scrub ok 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: audit 2024-04-02T04:55:33.310016+0000 mgr.y (mgr.14538) 467 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.447502+0000 osd.1 (osd.1) 275 : cluster 0 1.3 scrub starts 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.448762+0000 osd.1 (osd.1) 276 : cluster 0 1.3 scrub ok 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.823347+0000 osd.2 (osd.2) 222 : cluster 0 1.8 scrub starts 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.825327+0000 osd.2 (osd.2) 223 : cluster 0 1.8 scrub ok 2024-04-02T04:55:34.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.967118+0000 mon.a (mon.0) 2833 : cluster 0 osdmap e486: 8 total, 8 up, 7 in 2024-04-02T04:55:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: audit 2024-04-02T04:55:33.999443+0000 mon.c (mon.2) 847 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: audit 2024-04-02T04:55:34.132437+0000 mon.b (mon.1) 252 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-02T04:55:34.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:34 smithi098 bash[23790]: audit 2024-04-02T04:55:34.139863+0000 mon.a (mon.0) 2834 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-02T04:55:34.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:55:34.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:34.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:34.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:34.859 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:34.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:34.989 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:34.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:35.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:35.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:35.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.261170+0000 osd.7 (osd.7) 57 : cluster 0 1.9 scrub starts 2024-04-02T04:55:35.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.263502+0000 osd.7 (osd.7) 58 : cluster 0 1.9 scrub ok 2024-04-02T04:55:35.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.601649+0000 osd.4 (osd.4) 265 : cluster 0 1.5 scrub starts 2024-04-02T04:55:35.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.603249+0000 osd.4 (osd.4) 266 : cluster 0 1.5 scrub ok 2024-04-02T04:55:35.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.952958+0000 osd.0 (osd.0) 255 : cluster 0 1.1 scrub starts 2024-04-02T04:55:35.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.954628+0000 osd.0 (osd.0) 256 : cluster 0 1.1 scrub ok 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.980328+0000 osd.6 (osd.6) 229 : cluster 0 1.7 scrub starts 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:33.982922+0000 osd.6 (osd.6) 230 : cluster 0 1.7 scrub ok 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.136184+0000 mgr.y (mgr.14538) 468 : cluster 0 pgmap v710: 355 pgs: 1 activating, 1 active+clean+snaptrim, 1 clean+premerge+peered, 6 peering, 346 active+clean; 8.3 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 331 KiB/s wr, 54 op/s; 380 KiB/s, 2 objects/s recovering 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.143664+0000 osd.3 (osd.3) 309 : cluster 0 4.10 scrub starts 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.146094+0000 osd.3 (osd.3) 310 : cluster 0 4.10 scrub ok 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.873506+0000 osd.2 (osd.2) 224 : cluster 0 1.a deep-scrub starts 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.875099+0000 osd.2 (osd.2) 225 : cluster 0 1.a deep-scrub ok 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: audit 2024-04-02T04:55:34.909767+0000 mon.a (mon.0) 2835 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.913679+0000 mon.a (mon.0) 2836 : cluster 0 osdmap e487: 8 total, 8 up, 7 in 2024-04-02T04:55:35.234 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:35 smithi098 bash[23790]: audit 2024-04-02T04:55:35.000616+0000 mon.c (mon.2) 848 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:35.249 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:35.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:35.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.261170+0000 osd.7 (osd.7) 57 : cluster 0 1.9 scrub starts 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.263502+0000 osd.7 (osd.7) 58 : cluster 0 1.9 scrub ok 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.601649+0000 osd.4 (osd.4) 265 : cluster 0 1.5 scrub starts 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.603249+0000 osd.4 (osd.4) 266 : cluster 0 1.5 scrub ok 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.952958+0000 osd.0 (osd.0) 255 : cluster 0 1.1 scrub starts 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.954628+0000 osd.0 (osd.0) 256 : cluster 0 1.1 scrub ok 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.980328+0000 osd.6 (osd.6) 229 : cluster 0 1.7 scrub starts 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:33.982922+0000 osd.6 (osd.6) 230 : cluster 0 1.7 scrub ok 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.136184+0000 mgr.y (mgr.14538) 468 : cluster 0 pgmap v710: 355 pgs: 1 activating, 1 active+clean+snaptrim, 1 clean+premerge+peered, 6 peering, 346 active+clean; 8.3 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 331 KiB/s wr, 54 op/s; 380 KiB/s, 2 objects/s recovering 2024-04-02T04:55:35.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.143664+0000 osd.3 (osd.3) 309 : cluster 0 4.10 scrub starts 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.146094+0000 osd.3 (osd.3) 310 : cluster 0 4.10 scrub ok 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.873506+0000 osd.2 (osd.2) 224 : cluster 0 1.a deep-scrub starts 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.875099+0000 osd.2 (osd.2) 225 : cluster 0 1.a deep-scrub ok 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: audit 2024-04-02T04:55:34.909767+0000 mon.a (mon.0) 2835 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.913679+0000 mon.a (mon.0) 2836 : cluster 0 osdmap e487: 8 total, 8 up, 7 in 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[21528]: audit 2024-04-02T04:55:35.000616+0000 mon.c (mon.2) 848 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.261170+0000 osd.7 (osd.7) 57 : cluster 0 1.9 scrub starts 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.263502+0000 osd.7 (osd.7) 58 : cluster 0 1.9 scrub ok 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.601649+0000 osd.4 (osd.4) 265 : cluster 0 1.5 scrub starts 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.603249+0000 osd.4 (osd.4) 266 : cluster 0 1.5 scrub ok 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.952958+0000 osd.0 (osd.0) 255 : cluster 0 1.1 scrub starts 2024-04-02T04:55:35.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.954628+0000 osd.0 (osd.0) 256 : cluster 0 1.1 scrub ok 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.980328+0000 osd.6 (osd.6) 229 : cluster 0 1.7 scrub starts 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:33.982922+0000 osd.6 (osd.6) 230 : cluster 0 1.7 scrub ok 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.136184+0000 mgr.y (mgr.14538) 468 : cluster 0 pgmap v710: 355 pgs: 1 activating, 1 active+clean+snaptrim, 1 clean+premerge+peered, 6 peering, 346 active+clean; 8.3 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 331 KiB/s wr, 54 op/s; 380 KiB/s, 2 objects/s recovering 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.143664+0000 osd.3 (osd.3) 309 : cluster 0 4.10 scrub starts 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.146094+0000 osd.3 (osd.3) 310 : cluster 0 4.10 scrub ok 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.873506+0000 osd.2 (osd.2) 224 : cluster 0 1.a deep-scrub starts 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.875099+0000 osd.2 (osd.2) 225 : cluster 0 1.a deep-scrub ok 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: audit 2024-04-02T04:55:34.909767+0000 mon.a (mon.0) 2835 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.913679+0000 mon.a (mon.0) 2836 : cluster 0 osdmap e487: 8 total, 8 up, 7 in 2024-04-02T04:55:35.499 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:35 smithi069 bash[28904]: audit 2024-04-02T04:55:35.000616+0000 mon.c (mon.2) 848 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:35.510 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:35.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:35.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- 172.21.15.69:0/3343349480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101e20 msgr2=0x7f446c10dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- 172.21.15.69:0/3343349480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101e20 0x7f446c10dfa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f446400b0d0 tx=0x7f446400b3e0 comp rx=0 tx=0).stop 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- 172.21.15.69:0/3343349480 shutdown_connections 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- 172.21.15.69:0/3343349480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101e20 0x7f446c10dfa0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- 172.21.15.69:0/3343349480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f446c107350 0x7f446c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- 172.21.15.69:0/3343349480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c101460 0x7f446c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:35.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- 172.21.15.69:0/3343349480 >> 172.21.15.69:0/3343349480 conn(0x7f446c077ea0 msgr2=0x7f446c0782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- 172.21.15.69:0/3343349480 shutdown_connections 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- 172.21.15.69:0/3343349480 wait complete. 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 librados: starting msgr at 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 librados: starting objecter 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 Processor -- start 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- start start 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 librados: setting wanted keys 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 librados: calling monclient init 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 0x7f446c10eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f446c101e20 0x7f446c10f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c107350 0x7f446c10f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f446c111b60 con 0x7f446c101460 2024-04-02T04:55:35.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f446c1119e0 con 0x7f446c107350 2024-04-02T04:55:35.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44732bd700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f446c111ce0 con 0x7f446c101e20 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 0x7f446c10eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4470858700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f446c101e20 0x7f446c10f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 0x7f446c10eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49152/0 (socket says 172.21.15.69:49152) 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 -- 172.21.15.69:0/1528830057 learned_addr learned my addr 172.21.15.69:0/1528830057 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c107350 0x7f446c10f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f446c101e20 msgr2=0x7f446c10f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f446c101e20 0x7f446c10f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c107350 msgr2=0x7f446c10f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c107350 0x7f446c10f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c1a6430 con 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f446c101e20 0x7f446c10f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 0x7f446c10eaf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f446800eb30 tx=0x7f446800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44627fc700 10 client.?.objecter ms_handle_connect 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44627fc700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446800cbd0 con 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.802+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f44680044a0 con 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446c1a6720 con 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468003d40 con 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 10 client.16506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f446c1a6cb0 con 0x7f446c101460 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 1 librados: init done 2024-04-02T04:55:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f446c04ff00 con 0x7f446c101460 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4468003ee0 con 0x7f446c101460 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f445806f5e0 0x7f4458071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(487..487 src has 74..487) ==== 10953+0+0 (secure 0 0 0) 0x7f446808f0f0 con 0x7f446c101460 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 osd_map(487..487 src has 74..487) 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 3 client.16506.objecter handle_osd_map got epochs [487,487] > 0 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 3 client.16506.objecter handle_osd_map decoding full epoch 487 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f445806f5e0 0x7f4458071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 20 client.16506.objecter dump_active .. 0 homeless 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v487) ==== 54+0+16 (secure 0 0 0) 0x7f4468091710 con 0x7f446c101460 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 10 client.16506.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 1 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f446c0630a0 0x7f446c107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 1 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 0 1 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 4 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 10 client.16506.objecter _send_command 1 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f446c1095a0 con 0x7f446c0630a0 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f445806f5e0 0x7f4458071aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f446c0784a0 tx=0x7f445c00a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f445806f5e0 2024-04-02T04:55:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f446c0630a0 0x7f446c107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c0630a0 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.806+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f446c0630a0 0x7f446c107ee0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.810+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1095a0 con 0x7f446c0630a0 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.810+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 1: 0 ) 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.810+0000 7f44627fc700 10 client.16506.objecter _finish_command 1 = system:0 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.810+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 4 2024-04-02T04:55:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.810+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 0 1 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 10 client.16506.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 2 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 2 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c10ffd0 osd=0 3 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 0 2 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 4 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 10 client.16506.objecter _send_command 2 2024-04-02T04:55:35.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c108420 con 0x7f446c0630a0 2024-04-02T04:55:35.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c108420 con 0x7f446c0630a0 2024-04-02T04:55:35.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44627fc700 10 client.16506.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 4 2024-04-02T04:55:35.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 0 2 2024-04-02T04:55:35.827 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:55:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 10 client.16506.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f446c062c90 0x7f446c1aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 1 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 4 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 10 client.16506.objecter _send_command 3 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f446c1092b0 con 0x7f446c062c90 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.826+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f446c062c90 0x7f446c1aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c062c90 2024-04-02T04:55:35.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f446c062c90 0x7f446c1aaf90 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1092b0 con 0x7f446c062c90 2024-04-02T04:55:35.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 3: 0 ) 2024-04-02T04:55:35.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f44627fc700 10 client.16506.objecter _finish_command 3 = system:0 2024-04-02T04:55:35.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 4 2024-04-02T04:55:35.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.830+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 1 3 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 10 client.16506.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 4 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 4 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c062b20 osd=1 3 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 1 4 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 4 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 10 client.16506.objecter _send_command 4 2024-04-02T04:55:35.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.846+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c108820 con 0x7f446c062c90 2024-04-02T04:55:35.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c108820 con 0x7f446c062c90 2024-04-02T04:55:35.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 10 client.16506.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 4 2024-04-02T04:55:35.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 1 4 2024-04-02T04:55:35.848 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 10 client.16506.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 5 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f446c02d690 0x7f446c1ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 5 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 2 5 2024-04-02T04:55:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 4 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 10 client.16506.objecter _send_command 5 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f446c1b14c0 con 0x7f446c02d690 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f446c02d690 0x7f446c1ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c02d690 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f446c02d690 0x7f446c1ae660 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1b14c0 con 0x7f446c02d690 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 5: 0 ) 2024-04-02T04:55:35.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 10 client.16506.objecter _finish_command 5 = system:0 2024-04-02T04:55:35.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 4 2024-04-02T04:55:35.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.850+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 2 5 2024-04-02T04:55:35.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 10 client.16506.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 6 2024-04-02T04:55:35.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 6 2024-04-02T04:55:35.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c02e040 osd=2 3 2024-04-02T04:55:35.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 2 6 2024-04-02T04:55:35.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 4 2024-04-02T04:55:35.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 10 client.16506.objecter _send_command 6 2024-04-02T04:55:35.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c02cc00 con 0x7f446c02d690 2024-04-02T04:55:35.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c02cc00 con 0x7f446c02d690 2024-04-02T04:55:35.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 10 client.16506.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 4 2024-04-02T04:55:35.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 2 6 2024-04-02T04:55:35.869 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:55:35.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 10 client.16506.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 7 2024-04-02T04:55:35.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f446c1ac1b0 0x7f446c1afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 7 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 3 7 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 4 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 10 client.16506.objecter _send_command 7 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f446c1b4e20 con 0x7f446c1ac1b0 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f446c1ac1b0 0x7f446c1afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c1ac1b0 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f446c1ac1b0 0x7f446c1afd20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1b4e20 con 0x7f446c1ac1b0 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 7: 0 ) 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 10 client.16506.objecter _finish_command 7 = system:0 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 4 2024-04-02T04:55:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.870+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 3 7 2024-04-02T04:55:35.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 10 client.16506.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 8 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 8 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c02cc00 osd=3 3 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 3 8 2024-04-02T04:55:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 4 2024-04-02T04:55:35.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 10 client.16506.objecter _send_command 8 2024-04-02T04:55:35.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c1ab780 con 0x7f446c1ac1b0 2024-04-02T04:55:35.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c1ab780 con 0x7f446c1ac1b0 2024-04-02T04:55:35.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44627fc700 10 client.16506.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 4 2024-04-02T04:55:35.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 3 8 2024-04-02T04:55:35.891 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.890+0000 7f44732bd700 10 client.16506.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 9 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f446c1af880 0x7f446c1b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 9 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 4 9 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 4 2024-04-02T04:55:35.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 10 client.16506.objecter _send_command 9 2024-04-02T04:55:35.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f446c1b84f0 con 0x7f446c1af880 2024-04-02T04:55:35.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f446c1af880 0x7f446c1b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.893 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:35.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c1af880 2024-04-02T04:55:35.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f446c1af880 0x7f446c1b5690 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:35.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:35.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1b84f0 con 0x7f446c1af880 2024-04-02T04:55:35.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 9: 0 ) 2024-04-02T04:55:35.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44627fc700 10 client.16506.objecter _finish_command 9 = system:0 2024-04-02T04:55:35.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 4 2024-04-02T04:55:35.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.894+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 4 9 2024-04-02T04:55:35.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 10 client.16506.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 10 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 10 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1ab530 osd=4 3 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 4 10 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 4 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 10 client.16506.objecter _send_command 10 2024-04-02T04:55:35.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.914+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c1b5bd0 con 0x7f446c1af880 2024-04-02T04:55:35.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c1b5bd0 con 0x7f446c1af880 2024-04-02T04:55:35.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 10 client.16506.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 4 2024-04-02T04:55:35.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 4 10 2024-04-02T04:55:35.916 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 10 client.16506.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 11 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f446c1b6250 0x7f446c1b8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 11 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 5 11 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 4 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 10 client.16506.objecter _send_command 11 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f446c1bb590 con 0x7f446c1b6250 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f446c1b6250 0x7f446c1b8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c1b6250 2024-04-02T04:55:35.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f4470858700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f446c1b6250 0x7f446c1b8640 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1bb590 con 0x7f446c1b6250 2024-04-02T04:55:35.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 11: 0 ) 2024-04-02T04:55:35.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 10 client.16506.objecter _finish_command 11 = system:0 2024-04-02T04:55:35.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 4 2024-04-02T04:55:35.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.918+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 5 11 2024-04-02T04:55:35.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 10 client.16506.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 12 2024-04-02T04:55:35.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 12 2024-04-02T04:55:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1b5bd0 osd=5 3 2024-04-02T04:55:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 5 12 2024-04-02T04:55:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 4 2024-04-02T04:55:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 10 client.16506.objecter _send_command 12 2024-04-02T04:55:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.938+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c1b8b80 con 0x7f446c1b6250 2024-04-02T04:55:35.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c1b8b80 con 0x7f446c1b6250 2024-04-02T04:55:35.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44627fc700 10 client.16506.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 4 2024-04-02T04:55:35.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 5 12 2024-04-02T04:55:35.941 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:55:35.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 10 client.16506.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 13 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f446c1b92a0 0x7f446c1b9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 13 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 6 13 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 4 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 10 client.16506.objecter _send_command 13 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f446c1be930 con 0x7f446c1b92a0 2024-04-02T04:55:35.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f446c1b92a0 0x7f446c1b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c1b92a0 2024-04-02T04:55:35.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.942+0000 7f447185a700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f446c1b92a0 0x7f446c1b9700 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.946+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1be930 con 0x7f446c1b92a0 2024-04-02T04:55:35.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.946+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 13: 0 ) 2024-04-02T04:55:35.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.946+0000 7f44627fc700 10 client.16506.objecter _finish_command 13 = system:0 2024-04-02T04:55:35.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.946+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 4 2024-04-02T04:55:35.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.946+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 6 13 2024-04-02T04:55:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 10 client.16506.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 14 2024-04-02T04:55:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 14 2024-04-02T04:55:35.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1b8b80 osd=6 3 2024-04-02T04:55:35.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 6 14 2024-04-02T04:55:35.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 4 2024-04-02T04:55:35.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 10 client.16506.objecter _send_command 14 2024-04-02T04:55:35.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.962+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c1bd7b0 con 0x7f446c1b92a0 2024-04-02T04:55:35.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c1bd7b0 con 0x7f446c1b92a0 2024-04-02T04:55:35.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:35.970 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44627fc700 10 client.16506.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 4 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 6 14 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 10 client.16506.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 15 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f446c1bded0 0x7f446c1be330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 15 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 7 15 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 4 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 10 client.16506.objecter _send_command 15 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f446c1c1ca0 con 0x7f446c1bded0 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.970+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f446c1bded0 0x7f446c1be330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:35.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.974+0000 7f44627fc700 10 client.16506.objecter ms_handle_connect 0x7f446c1bded0 2024-04-02T04:55:35.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.974+0000 7f4471059700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f446c1bded0 0x7f446c1be330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.978+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c1c1ca0 con 0x7f446c1bded0 2024-04-02T04:55:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.978+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 15: 0 ) 2024-04-02T04:55:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.978+0000 7f44627fc700 10 client.16506.objecter _finish_command 15 = system:0 2024-04-02T04:55:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.978+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 4 2024-04-02T04:55:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.978+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 7 15 2024-04-02T04:55:35.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 10 client.16506.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign -1 16 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 20 client.16506.objecter _get_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 15 client.16506.objecter _session_command_op_remove -1 16 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 20 client.16506.objecter get_session s=0x7f446c1bd7b0 osd=7 3 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 15 client.16506.objecter _session_command_op_assign 7 16 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 4 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 10 client.16506.objecter _send_command 16 2024-04-02T04:55:35.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:35.998+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f446c1b9c40 con 0x7f446c1bded0 2024-04-02T04:55:36.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44627fc700 1 -- 172.21.15.69:0/1528830057 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f446c1b9c40 con 0x7f446c1bded0 2024-04-02T04:55:36.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44627fc700 10 client.16506.objecter ms_dispatch 0x7f446c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:36.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44627fc700 10 client.16506.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:36.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44627fc700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 4 2024-04-02T04:55:36.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44627fc700 15 client.16506.objecter _session_command_op_remove 7 16 2024-04-02T04:55:36.002 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:55:36.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 librados: watch_flush enter 2024-04-02T04:55:36.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 librados: watch_flush exit 2024-04-02T04:55:36.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.0 2024-04-02T04:55:36.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f446c0630a0 msgr2=0x7f446c107ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f446c0630a0 0x7f446c107ee0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c10ffd0 osd=0 1 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.1 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f446c062c90 msgr2=0x7f446c1aaf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f446c062c90 0x7f446c1aaf90 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c062b20 osd=1 1 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.2 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f446c02d690 msgr2=0x7f446c1ae660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f446c02d690 0x7f446c1ae660 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02e040 osd=2 1 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.3 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f446c1ac1b0 msgr2=0x7f446c1afd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f446c1ac1b0 0x7f446c1afd20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c02cc00 osd=3 1 2024-04-02T04:55:36.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.4 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f446c1af880 msgr2=0x7f446c1b5690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f447185a700 1 -- 172.21.15.69:0/1528830057 reap_dead start 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f446c1af880 0x7f446c1b5690 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f447185a700 1 -- 172.21.15.69:0/1528830057 reap_dead start 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1ab530 osd=4 1 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.5 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f446c1b6250 msgr2=0x7f446c1b8640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f446c1b6250 0x7f446c1b8640 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b5bd0 osd=5 1 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.6 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f446c1b92a0 msgr2=0x7f446c1b9700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f446c1b92a0 0x7f446c1b9700 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1b8b80 osd=6 1 2024-04-02T04:55:36.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter close_session for osd.7 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f446c1bded0 msgr2=0x7f446c1be330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f446c1bded0 0x7f446c1be330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter put_session s=0x7f446c1bd7b0 osd=7 1 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 20 client.16506.objecter shutdown clearing up homeless session... 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 10 client.16506.objecter successfully canceled tick 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f445806f5e0 msgr2=0x7f4458071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f445806f5e0 0x7f4458071aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f446c0784a0 tx=0x7f445c00a380 comp rx=0 tx=0).stop 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f447185a700 1 -- 172.21.15.69:0/1528830057 reap_dead start 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 msgr2=0x7f446c10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 0x7f446c10eaf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f446800eb30 tx=0x7f446800ee40 comp rx=0 tx=0).stop 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 shutdown_connections 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 --2- 172.21.15.69:0/1528830057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f446c101460 0x7f446c10eaf0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 >> 172.21.15.69:0/1528830057 conn(0x7f446c077ea0 msgr2=0x7f446c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:36.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 shutdown_connections 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 -- 172.21.15.69:0/1528830057 wait complete. 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:36.002+0000 7f44732bd700 1 librados: shutdown 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:55:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:55:36.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:36.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:36.149 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:36.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:36.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.215527+0000 osd.7 (osd.7) 59 : cluster 0 1.2 scrub starts 2024-04-02T04:55:36.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.217237+0000 osd.7 (osd.7) 60 : cluster 0 1.2 scrub ok 2024-04-02T04:55:36.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.652101+0000 osd.4 (osd.4) 267 : cluster 0 4.d deep-scrub starts 2024-04-02T04:55:36.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:34.654299+0000 osd.4 (osd.4) 268 : cluster 0 4.d deep-scrub ok 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.115926+0000 osd.3 (osd.3) 311 : cluster 0 6.7 scrub starts 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.117767+0000 osd.3 (osd.3) 312 : cluster 0 6.7 scrub ok 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.368796+0000 osd.1 (osd.1) 277 : cluster 0 102.5 scrub starts 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.372899+0000 osd.1 (osd.1) 278 : cluster 0 102.5 scrub ok 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: audit 2024-04-02T04:55:35.809655+0000 mon.a (mon.0) 2837 : audit 0 from='client.? 172.21.15.69:0/1528830057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.869910+0000 osd.2 (osd.2) 226 : cluster 0 2.10 scrub starts 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.870861+0000 osd.2 (osd.2) 227 : cluster 0 2.10 scrub ok 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.970251+0000 mon.a (mon.0) 2838 : cluster 0 osdmap e488: 8 total, 8 up, 7 in 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[28904]: audit 2024-04-02T04:55:36.001866+0000 mon.c (mon.2) 849 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.215527+0000 osd.7 (osd.7) 59 : cluster 0 1.2 scrub starts 2024-04-02T04:55:36.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.217237+0000 osd.7 (osd.7) 60 : cluster 0 1.2 scrub ok 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.652101+0000 osd.4 (osd.4) 267 : cluster 0 4.d deep-scrub starts 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:34.654299+0000 osd.4 (osd.4) 268 : cluster 0 4.d deep-scrub ok 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.115926+0000 osd.3 (osd.3) 311 : cluster 0 6.7 scrub starts 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.117767+0000 osd.3 (osd.3) 312 : cluster 0 6.7 scrub ok 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.368796+0000 osd.1 (osd.1) 277 : cluster 0 102.5 scrub starts 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.372899+0000 osd.1 (osd.1) 278 : cluster 0 102.5 scrub ok 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: audit 2024-04-02T04:55:35.809655+0000 mon.a (mon.0) 2837 : audit 0 from='client.? 172.21.15.69:0/1528830057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.869910+0000 osd.2 (osd.2) 226 : cluster 0 2.10 scrub starts 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.870861+0000 osd.2 (osd.2) 227 : cluster 0 2.10 scrub ok 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.970251+0000 mon.a (mon.0) 2838 : cluster 0 osdmap e488: 8 total, 8 up, 7 in 2024-04-02T04:55:36.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:36 smithi069 bash[21528]: audit 2024-04-02T04:55:36.001866+0000 mon.c (mon.2) 849 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:36.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.215527+0000 osd.7 (osd.7) 59 : cluster 0 1.2 scrub starts 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.217237+0000 osd.7 (osd.7) 60 : cluster 0 1.2 scrub ok 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.652101+0000 osd.4 (osd.4) 267 : cluster 0 4.d deep-scrub starts 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:34.654299+0000 osd.4 (osd.4) 268 : cluster 0 4.d deep-scrub ok 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.115926+0000 osd.3 (osd.3) 311 : cluster 0 6.7 scrub starts 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.117767+0000 osd.3 (osd.3) 312 : cluster 0 6.7 scrub ok 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.368796+0000 osd.1 (osd.1) 277 : cluster 0 102.5 scrub starts 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.372899+0000 osd.1 (osd.1) 278 : cluster 0 102.5 scrub ok 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: audit 2024-04-02T04:55:35.809655+0000 mon.a (mon.0) 2837 : audit 0 from='client.? 172.21.15.69:0/1528830057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.869910+0000 osd.2 (osd.2) 226 : cluster 0 2.10 scrub starts 2024-04-02T04:55:36.384 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.870861+0000 osd.2 (osd.2) 227 : cluster 0 2.10 scrub ok 2024-04-02T04:55:36.385 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.970251+0000 mon.a (mon.0) 2838 : cluster 0 osdmap e488: 8 total, 8 up, 7 in 2024-04-02T04:55:36.385 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:36 smithi098 bash[23790]: audit 2024-04-02T04:55:36.001866+0000 mon.c (mon.2) 849 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:36.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:36.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:36.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:36.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:36.786 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:36.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:36.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:37.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:37.148 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:37.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:37.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.187447+0000 osd.7 (osd.7) 61 : cluster 0 43.8 scrub starts 2024-04-02T04:55:37.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[28904]: cluster 2024-04-02T04:55:35.189766+0000 osd.7 (osd.7) 62 : cluster 0 43.8 scrub ok 2024-04-02T04:55:37.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[28904]: cluster 2024-04-02T04:55:36.137158+0000 mgr.y (mgr.14538) 469 : cluster 0 pgmap v713: 355 pgs: 1 activating, 1 active+clean+snaptrim, 1 clean+premerge+peered, 6 peering, 346 active+clean; 8.3 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 331 KiB/s wr, 54 op/s; 380 KiB/s, 2 objects/s recovering 2024-04-02T04:55:37.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[28904]: cluster 2024-04-02T04:55:36.977540+0000 mon.a (mon.0) 2839 : cluster 0 osdmap e489: 8 total, 8 up, 7 in 2024-04-02T04:55:37.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[28904]: audit 2024-04-02T04:55:37.003042+0000 mon.c (mon.2) 850 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.187447+0000 osd.7 (osd.7) 61 : cluster 0 43.8 scrub starts 2024-04-02T04:55:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[21528]: cluster 2024-04-02T04:55:35.189766+0000 osd.7 (osd.7) 62 : cluster 0 43.8 scrub ok 2024-04-02T04:55:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[21528]: cluster 2024-04-02T04:55:36.137158+0000 mgr.y (mgr.14538) 469 : cluster 0 pgmap v713: 355 pgs: 1 activating, 1 active+clean+snaptrim, 1 clean+premerge+peered, 6 peering, 346 active+clean; 8.3 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 331 KiB/s wr, 54 op/s; 380 KiB/s, 2 objects/s recovering 2024-04-02T04:55:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[21528]: cluster 2024-04-02T04:55:36.977540+0000 mon.a (mon.0) 2839 : cluster 0 osdmap e489: 8 total, 8 up, 7 in 2024-04-02T04:55:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:37 smithi069 bash[21528]: audit 2024-04-02T04:55:37.003042+0000 mon.c (mon.2) 850 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:37.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:37.305 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:37.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:37.435 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:37 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.187447+0000 osd.7 (osd.7) 61 : cluster 0 43.8 scrub starts 2024-04-02T04:55:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:37 smithi098 bash[23790]: cluster 2024-04-02T04:55:35.189766+0000 osd.7 (osd.7) 62 : cluster 0 43.8 scrub ok 2024-04-02T04:55:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:37 smithi098 bash[23790]: cluster 2024-04-02T04:55:36.137158+0000 mgr.y (mgr.14538) 469 : cluster 0 pgmap v713: 355 pgs: 1 activating, 1 active+clean+snaptrim, 1 clean+premerge+peered, 6 peering, 346 active+clean; 8.3 MiB data, 504 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 331 KiB/s wr, 54 op/s; 380 KiB/s, 2 objects/s recovering 2024-04-02T04:55:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:37 smithi098 bash[23790]: cluster 2024-04-02T04:55:36.977540+0000 mon.a (mon.0) 2839 : cluster 0 osdmap e489: 8 total, 8 up, 7 in 2024-04-02T04:55:37.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:37 smithi098 bash[23790]: audit 2024-04-02T04:55:37.003042+0000 mon.c (mon.2) 850 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:37.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:37.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:37.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:37.692 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:37.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:37.822 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:37.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:37.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:38.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:38.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:38.191 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:36.609692+0000 osd.4 (osd.4) 269 : cluster 0 4.e scrub starts 2024-04-02T04:55:38.191 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:36.612117+0000 osd.4 (osd.4) 270 : cluster 0 4.e scrub ok 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.065596+0000 osd.0 (osd.0) 257 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.065850+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.842614+0000 osd.0 (osd.0) 259 : cluster 0 41.d scrub starts 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.844186+0000 osd.0 (osd.0) 260 : cluster 0 41.d scrub ok 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.990839+0000 mon.a (mon.0) 2840 : cluster 0 osdmap e490: 8 total, 8 up, 7 in 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: audit 2024-04-02T04:55:38.004669+0000 mon.c (mon.2) 851 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: audit 2024-04-02T04:55:38.134896+0000 mon.b (mon.1) 253 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-02T04:55:38.192 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:38 smithi098 bash[23790]: audit 2024-04-02T04:55:38.142363+0000 mon.a (mon.0) 2841 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-02T04:55:38.203 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:38.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:38.332 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:38.366 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:55:38.366 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:55:38.366 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T04:55:38.366 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:55:38.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:36.609692+0000 osd.4 (osd.4) 269 : cluster 0 4.e scrub starts 2024-04-02T04:55:38.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:36.612117+0000 osd.4 (osd.4) 270 : cluster 0 4.e scrub ok 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.065596+0000 osd.0 (osd.0) 257 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.065850+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.842614+0000 osd.0 (osd.0) 259 : cluster 0 41.d scrub starts 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.844186+0000 osd.0 (osd.0) 260 : cluster 0 41.d scrub ok 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.990839+0000 mon.a (mon.0) 2840 : cluster 0 osdmap e490: 8 total, 8 up, 7 in 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: audit 2024-04-02T04:55:38.004669+0000 mon.c (mon.2) 851 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: audit 2024-04-02T04:55:38.134896+0000 mon.b (mon.1) 253 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21528]: audit 2024-04-02T04:55:38.142363+0000 mon.a (mon.0) 2841 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:36.609692+0000 osd.4 (osd.4) 269 : cluster 0 4.e scrub starts 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:36.612117+0000 osd.4 (osd.4) 270 : cluster 0 4.e scrub ok 2024-04-02T04:55:38.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.065596+0000 osd.0 (osd.0) 257 : cluster 0 purged_snaps scrub starts 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.065850+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub ok 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.842614+0000 osd.0 (osd.0) 259 : cluster 0 41.d scrub starts 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.844186+0000 osd.0 (osd.0) 260 : cluster 0 41.d scrub ok 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.990839+0000 mon.a (mon.0) 2840 : cluster 0 osdmap e490: 8 total, 8 up, 7 in 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: audit 2024-04-02T04:55:38.004669+0000 mon.c (mon.2) 851 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: audit 2024-04-02T04:55:38.134896+0000 mon.b (mon.1) 253 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-02T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[28904]: audit 2024-04-02T04:55:38.142363+0000 mon.a (mon.0) 2841 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-02T04:55:38.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:38.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:38.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:38.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:38.782 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:38.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:38.842 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:38.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:38.957 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:55:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:55:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:55:38.972 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:38.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:39.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:39.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:39.230 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:39.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-02T04:55:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.965782+0000 osd.6 (osd.6) 231 : cluster 0 43.1c scrub starts 2024-04-02T04:55:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: cluster 2024-04-02T04:55:37.967195+0000 osd.6 (osd.6) 232 : cluster 0 43.1c scrub ok 2024-04-02T04:55:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: cluster 2024-04-02T04:55:38.138502+0000 mgr.y (mgr.14538) 470 : cluster 0 pgmap v716: 354 pgs: 1 peering, 353 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: cluster 2024-04-02T04:55:38.238545+0000 osd.7 (osd.7) 63 : cluster 0 40.1c scrub starts 2024-04-02T04:55:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: cluster 2024-04-02T04:55:38.240735+0000 osd.7 (osd.7) 64 : cluster 0 40.1c scrub ok 2024-04-02T04:55:39.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: audit 2024-04-02T04:55:38.991646+0000 mon.a (mon.0) 2842 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: cluster 2024-04-02T04:55:38.995216+0000 mon.a (mon.0) 2843 : cluster 0 osdmap e491: 8 total, 8 up, 7 in 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[21528]: audit 2024-04-02T04:55:39.006529+0000 mon.c (mon.2) 852 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.965782+0000 osd.6 (osd.6) 231 : cluster 0 43.1c scrub starts 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: cluster 2024-04-02T04:55:37.967195+0000 osd.6 (osd.6) 232 : cluster 0 43.1c scrub ok 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: cluster 2024-04-02T04:55:38.138502+0000 mgr.y (mgr.14538) 470 : cluster 0 pgmap v716: 354 pgs: 1 peering, 353 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: cluster 2024-04-02T04:55:38.238545+0000 osd.7 (osd.7) 63 : cluster 0 40.1c scrub starts 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: cluster 2024-04-02T04:55:38.240735+0000 osd.7 (osd.7) 64 : cluster 0 40.1c scrub ok 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: audit 2024-04-02T04:55:38.991646+0000 mon.a (mon.0) 2842 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: cluster 2024-04-02T04:55:38.995216+0000 mon.a (mon.0) 2843 : cluster 0 osdmap e491: 8 total, 8 up, 7 in 2024-04-02T04:55:39.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:39 smithi069 bash[28904]: audit 2024-04-02T04:55:39.006529+0000 mon.c (mon.2) 852 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:39.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.965782+0000 osd.6 (osd.6) 231 : cluster 0 43.1c scrub starts 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: cluster 2024-04-02T04:55:37.967195+0000 osd.6 (osd.6) 232 : cluster 0 43.1c scrub ok 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: cluster 2024-04-02T04:55:38.138502+0000 mgr.y (mgr.14538) 470 : cluster 0 pgmap v716: 354 pgs: 1 peering, 353 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: cluster 2024-04-02T04:55:38.238545+0000 osd.7 (osd.7) 63 : cluster 0 40.1c scrub starts 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: cluster 2024-04-02T04:55:38.240735+0000 osd.7 (osd.7) 64 : cluster 0 40.1c scrub ok 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: audit 2024-04-02T04:55:38.991646+0000 mon.a (mon.0) 2842 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: cluster 2024-04-02T04:55:38.995216+0000 mon.a (mon.0) 2843 : cluster 0 osdmap e491: 8 total, 8 up, 7 in 2024-04-02T04:55:39.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:39 smithi098 bash[23790]: audit 2024-04-02T04:55:39.006529+0000 mon.c (mon.2) 852 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:39.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:39.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:39.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:39.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:39.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:39.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:39.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:39.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:40.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:40.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:40.169 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:40.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:40.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:40.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:55:40.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:40.412 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[28904]: cluster 2024-04-02T04:55:39.140645+0000 osd.3 (osd.3) 313 : cluster 0 3.5 scrub starts 2024-04-02T04:55:40.412 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[28904]: cluster 2024-04-02T04:55:39.143345+0000 osd.3 (osd.3) 314 : cluster 0 3.5 scrub ok 2024-04-02T04:55:40.413 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[28904]: cluster 2024-04-02T04:55:39.916468+0000 mon.a (mon.0) 2844 : cluster 0 osdmap e492: 8 total, 8 up, 7 in 2024-04-02T04:55:40.413 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[28904]: audit 2024-04-02T04:55:40.007354+0000 mon.c (mon.2) 853 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:40.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[21528]: cluster 2024-04-02T04:55:39.140645+0000 osd.3 (osd.3) 313 : cluster 0 3.5 scrub starts 2024-04-02T04:55:40.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[21528]: cluster 2024-04-02T04:55:39.143345+0000 osd.3 (osd.3) 314 : cluster 0 3.5 scrub ok 2024-04-02T04:55:40.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[21528]: cluster 2024-04-02T04:55:39.916468+0000 mon.a (mon.0) 2844 : cluster 0 osdmap e492: 8 total, 8 up, 7 in 2024-04-02T04:55:40.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:40 smithi069 bash[21528]: audit 2024-04-02T04:55:40.007354+0000 mon.c (mon.2) 853 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:40.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:40.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 -- 172.21.15.69:0/816323203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 msgr2=0x7fd6e0074850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:40.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 --2- 172.21.15.69:0/816323203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 0x7fd6e0074850 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009990 tx=0x7fd6d4009ca0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 -- 172.21.15.69:0/816323203 shutdown_connections 2024-04-02T04:55:40.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 --2- 172.21.15.69:0/816323203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e010eca0 0x7fd6e0111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 --2- 172.21.15.69:0/816323203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 0x7fd6e0074850 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 --2- 172.21.15.69:0/816323203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e010e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 -- 172.21.15.69:0/816323203 >> 172.21.15.69:0/816323203 conn(0x7fd6e00793a0 msgr2=0x7fd6e00797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 -- 172.21.15.69:0/816323203 shutdown_connections 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 -- 172.21.15.69:0/816323203 wait complete. 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 librados: starting msgr at 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.466+0000 7fd6e5260700 1 librados: starting objecter 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 Processor -- start 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- start start 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 librados: setting wanted keys 2024-04-02T04:55:40.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 librados: calling monclient init 2024-04-02T04:55:40.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 0x7fd6e019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:40.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:40.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e010eca0 0x7fd6e019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:40.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e0113c50 con 0x7fd6e0074470 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e0113ad0 con 0x7fd6e010eca0 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e0113dd0 con 0x7fd6e0074d90 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6ded9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 0x7fd6e019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60976/0 (socket says 172.21.15.69:60976) 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 -- 172.21.15.69:0/2185383769 learned_addr learned my addr 172.21.15.69:0/2185383769 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6df59e700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e010eca0 0x7fd6e019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 -- 172.21.15.69:0/2185383769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e010eca0 msgr2=0x7fd6e019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e010eca0 0x7fd6e019eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 -- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 msgr2=0x7fd6e019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 0x7fd6e019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 -- 172.21.15.69:0/2185383769 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e019f2c0 con 0x7fd6e0074d90 2024-04-02T04:55:40.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6de59c700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e019a7f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000e9c0 tx=0x7fd6d000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:40.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 10 client.?.objecter ms_handle_connect 0x7fd6e0074d90 2024-04-02T04:55:40.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:40.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d000cbd0 con 0x7fd6e0074d90 2024-04-02T04:55:40.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd6d000cd70 con 0x7fd6e0074d90 2024-04-02T04:55:40.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e019f5b0 con 0x7fd6e0074d90 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0005610 con 0x7fd6e0074d90 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 10 client.26606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e01a6e30 con 0x7fd6e0074d90 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 librados: init done 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e004ff00 con 0x7fd6e0074d90 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd6d0004750 con 0x7fd6e0074d90 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd6c806f5e0 0x7fd6c8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(492..492 src has 74..492) ==== 10953+0+0 (secure 0 0 0) 0x7fd6d008f090 con 0x7fd6e0074d90 2024-04-02T04:55:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 10 client.26606.objecter ms_dispatch 0x7fd6e0001550 osd_map(492..492 src has 74..492) 2024-04-02T04:55:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 3 client.26606.objecter handle_osd_map got epochs [492,492] > 0 2024-04-02T04:55:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 3 client.26606.objecter handle_osd_map decoding full epoch 492 2024-04-02T04:55:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6ded9d700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd6c806f5e0 0x7fd6c8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.470+0000 7fd6c7fff700 20 client.26606.objecter dump_active .. 0 homeless 2024-04-02T04:55:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.474+0000 7fd6ded9d700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd6c806f5e0 0x7fd6c8071aa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400b2e0 tx=0x7fd6d4005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.474+0000 7fd6c7fff700 10 client.26606.objecter ms_handle_connect 0x7fd6c806f5e0 2024-04-02T04:55:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:40 smithi098 bash[23790]: cluster 2024-04-02T04:55:39.140645+0000 osd.3 (osd.3) 313 : cluster 0 3.5 scrub starts 2024-04-02T04:55:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:40 smithi098 bash[23790]: cluster 2024-04-02T04:55:39.143345+0000 osd.3 (osd.3) 314 : cluster 0 3.5 scrub ok 2024-04-02T04:55:40.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:40 smithi098 bash[23790]: cluster 2024-04-02T04:55:39.916468+0000 mon.a (mon.0) 2844 : cluster 0 osdmap e492: 8 total, 8 up, 7 in 2024-04-02T04:55:40.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:40 smithi098 bash[23790]: audit 2024-04-02T04:55:40.007354+0000 mon.c (mon.2) 853 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:40.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.478+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd6d005b8d0 con 0x7fd6e0074d90 2024-04-02T04:55:40.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:40.546 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:40.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:40.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:40.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.706+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd6e019b610 con 0x7fd6e0074d90 2024-04-02T04:55:40.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:40.806 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:40.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:40.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.934+0000 7fd6c7fff700 1 -- 172.21.15.69:0/2185383769 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v493) ==== 87+0+0 (secure 0 0 0) 0x7fd6d005f320 con 0x7fd6e0074d90 2024-04-02T04:55:40.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 10 librados: watch_flush enter 2024-04-02T04:55:40.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 10 librados: watch_flush exit 2024-04-02T04:55:40.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 20 client.26606.objecter shutdown clearing up homeless session... 2024-04-02T04:55:40.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 10 client.26606.objecter successfully canceled tick 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd6c806f5e0 msgr2=0x7fd6c8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd6c806f5e0 0x7fd6c8071aa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400b2e0 tx=0x7fd6d4005960 comp rx=0 tx=0).stop 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 msgr2=0x7fd6e019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e019a7f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000e9c0 tx=0x7fd6d000ecd0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 shutdown_connections 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd6c806f5e0 0x7fd6c8071aa0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e010eca0 0x7fd6e019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6e0074470 0x7fd6e019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 --2- 172.21.15.69:0/2185383769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6e0074d90 0x7fd6e019a7f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:40.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 >> 172.21.15.69:0/2185383769 conn(0x7fd6e00793a0 msgr2=0x7fd6e0106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:40.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.938+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 shutdown_connections 2024-04-02T04:55:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:40.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.942+0000 7fd6e5260700 1 -- 172.21.15.69:0/2185383769 wait complete. 2024-04-02T04:55:40.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:40.942+0000 7fd6e5260700 1 librados: shutdown 2024-04-02T04:55:40.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:40.944 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:55:41.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:41.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:41.199 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:41.302 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:41.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: cluster 2024-04-02T04:55:40.139491+0000 mgr.y (mgr.14538) 471 : cluster 0 pgmap v719: 354 pgs: 1 peering, 353 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:40.711297+0000 mon.c (mon.2) 854 : audit 1 from='client.? 172.21.15.69:0/2185383769' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:40.711635+0000 mon.a (mon.0) 2845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: cluster 2024-04-02T04:55:40.924664+0000 mon.a (mon.0) 2846 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:40.927042+0000 mon.a (mon.0) 2847 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: cluster 2024-04-02T04:55:40.934563+0000 mon.a (mon.0) 2848 : cluster 0 osdmap e493: 8 total, 8 up, 7 in 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:41.008194+0000 mon.c (mon.2) 855 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:41.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:41.009520+0000 mon.c (mon.2) 856 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:41.009769+0000 mon.a (mon.0) 2849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:41.010261+0000 mon.c (mon.2) 857 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-63"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[28904]: audit 2024-04-02T04:55:41.010476+0000 mon.a (mon.0) 2850 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-63"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: cluster 2024-04-02T04:55:40.139491+0000 mgr.y (mgr.14538) 471 : cluster 0 pgmap v719: 354 pgs: 1 peering, 353 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:40.711297+0000 mon.c (mon.2) 854 : audit 1 from='client.? 172.21.15.69:0/2185383769' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:40.711635+0000 mon.a (mon.0) 2845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: cluster 2024-04-02T04:55:40.924664+0000 mon.a (mon.0) 2846 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:40.927042+0000 mon.a (mon.0) 2847 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: cluster 2024-04-02T04:55:40.934563+0000 mon.a (mon.0) 2848 : cluster 0 osdmap e493: 8 total, 8 up, 7 in 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:41.008194+0000 mon.c (mon.2) 855 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:41.009520+0000 mon.c (mon.2) 856 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:41.445 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:41.009769+0000 mon.a (mon.0) 2849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:41.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:41.010261+0000 mon.c (mon.2) 857 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-63"}]: dispatch 2024-04-02T04:55:41.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:41 smithi069 bash[21528]: audit 2024-04-02T04:55:41.010476+0000 mon.a (mon.0) 2850 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-63"}]: dispatch 2024-04-02T04:55:41.458 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: cluster 2024-04-02T04:55:40.139491+0000 mgr.y (mgr.14538) 471 : cluster 0 pgmap v719: 354 pgs: 1 peering, 353 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:40.711297+0000 mon.c (mon.2) 854 : audit 1 from='client.? 172.21.15.69:0/2185383769' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:40.711635+0000 mon.a (mon.0) 2845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: cluster 2024-04-02T04:55:40.924664+0000 mon.a (mon.0) 2846 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:40.927042+0000 mon.a (mon.0) 2847 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: cluster 2024-04-02T04:55:40.934563+0000 mon.a (mon.0) 2848 : cluster 0 osdmap e493: 8 total, 8 up, 7 in 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:41.008194+0000 mon.c (mon.2) 855 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:41.009520+0000 mon.c (mon.2) 856 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:41.009769+0000 mon.a (mon.0) 2849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:41.010261+0000 mon.c (mon.2) 857 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-63"}]: dispatch 2024-04-02T04:55:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:41 smithi098 bash[23790]: audit 2024-04-02T04:55:41.010476+0000 mon.a (mon.0) 2850 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-63"}]: dispatch 2024-04-02T04:55:41.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:41.589 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:41.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:41.720 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:41.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:41.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:41.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:41.981 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:41.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:42.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:42.104 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:42.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:42.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:42.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:42.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:42.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:42.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:42.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:42.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:42.881 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:42.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:42.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:43.000 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:42 smithi098 bash[23790]: cluster 2024-04-02T04:55:41.960503+0000 mon.a (mon.0) 2851 : cluster 0 osdmap e494: 8 total, 8 up, 7 in 2024-04-02T04:55:43.001 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:42 smithi098 bash[23790]: audit 2024-04-02T04:55:42.009254+0000 mon.c (mon.2) 858 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:43.001 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:42 smithi098 bash[23790]: audit 2024-04-02T04:55:42.136595+0000 mon.b (mon.1) 254 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-02T04:55:43.001 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:42 smithi098 bash[23790]: cluster 2024-04-02T04:55:42.140427+0000 mgr.y (mgr.14538) 472 : cluster 0 pgmap v722: 321 pgs: 1 peering, 320 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:43.001 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:42 smithi098 bash[23790]: audit 2024-04-02T04:55:42.415290+0000 mon.a (mon.0) 2852 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-02T04:55:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:43.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:43.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[28904]: cluster 2024-04-02T04:55:41.960503+0000 mon.a (mon.0) 2851 : cluster 0 osdmap e494: 8 total, 8 up, 7 in 2024-04-02T04:55:43.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[28904]: audit 2024-04-02T04:55:42.009254+0000 mon.c (mon.2) 858 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:43.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[28904]: audit 2024-04-02T04:55:42.136595+0000 mon.b (mon.1) 254 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-02T04:55:43.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[28904]: cluster 2024-04-02T04:55:42.140427+0000 mgr.y (mgr.14538) 472 : cluster 0 pgmap v722: 321 pgs: 1 peering, 320 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:43.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[28904]: audit 2024-04-02T04:55:42.415290+0000 mon.a (mon.0) 2852 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-02T04:55:43.134 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[21528]: cluster 2024-04-02T04:55:41.960503+0000 mon.a (mon.0) 2851 : cluster 0 osdmap e494: 8 total, 8 up, 7 in 2024-04-02T04:55:43.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[21528]: audit 2024-04-02T04:55:42.009254+0000 mon.c (mon.2) 858 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:43.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[21528]: audit 2024-04-02T04:55:42.136595+0000 mon.b (mon.1) 254 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-02T04:55:43.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[21528]: cluster 2024-04-02T04:55:42.140427+0000 mgr.y (mgr.14538) 472 : cluster 0 pgmap v722: 321 pgs: 1 peering, 320 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:43.135 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:42 smithi069 bash[21528]: audit 2024-04-02T04:55:42.415290+0000 mon.a (mon.0) 2852 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-02T04:55:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:43.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:43.274 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:43.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:43.405 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:43.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:43.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:55:43.533 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:43.636 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:43.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:43.704 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:55:43.704 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-02T04:55:43.704 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:55:43.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:43.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:43.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:43.919 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:43.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:44.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:44.045 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:44.088 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:55:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[23790]: audit 2024-04-02T04:55:42.957116+0000 mon.a (mon.0) 2853 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-02T04:55:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[23790]: cluster 2024-04-02T04:55:42.962384+0000 mon.a (mon.0) 2854 : cluster 0 osdmap e495: 8 total, 8 up, 7 in 2024-04-02T04:55:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[23790]: audit 2024-04-02T04:55:42.965776+0000 mon.c (mon.2) 859 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:44.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[23790]: audit 2024-04-02T04:55:42.966562+0000 mon.a (mon.0) 2855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:44.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[23790]: audit 2024-04-02T04:55:43.010825+0000 mon.c (mon.2) 860 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:44.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:43 smithi098 bash[23790]: audit 2024-04-02T04:55:43.321013+0000 mgr.y (mgr.14538) 473 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:44.148 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[28904]: audit 2024-04-02T04:55:42.957116+0000 mon.a (mon.0) 2853 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[28904]: cluster 2024-04-02T04:55:42.962384+0000 mon.a (mon.0) 2854 : cluster 0 osdmap e495: 8 total, 8 up, 7 in 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[28904]: audit 2024-04-02T04:55:42.965776+0000 mon.c (mon.2) 859 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[28904]: audit 2024-04-02T04:55:42.966562+0000 mon.a (mon.0) 2855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[28904]: audit 2024-04-02T04:55:43.010825+0000 mon.c (mon.2) 860 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[28904]: audit 2024-04-02T04:55:43.321013+0000 mgr.y (mgr.14538) 473 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:44.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[21528]: audit 2024-04-02T04:55:42.957116+0000 mon.a (mon.0) 2853 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-02T04:55:44.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[21528]: cluster 2024-04-02T04:55:42.962384+0000 mon.a (mon.0) 2854 : cluster 0 osdmap e495: 8 total, 8 up, 7 in 2024-04-02T04:55:44.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[21528]: audit 2024-04-02T04:55:42.965776+0000 mon.c (mon.2) 859 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:44.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[21528]: audit 2024-04-02T04:55:42.966562+0000 mon.a (mon.0) 2855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:55:44.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[21528]: audit 2024-04-02T04:55:43.010825+0000 mon.c (mon.2) 860 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:44.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:43 smithi069 bash[21528]: audit 2024-04-02T04:55:43.321013+0000 mgr.y (mgr.14538) 473 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:44.172 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:44.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:44.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:44.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:44.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:44.533 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:44.560 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:44.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:44.689 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:44.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:44.819 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:44.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:44.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:44.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:45.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:45.084 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:45.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:55:45.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:45.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: audit 2024-04-02T04:55:43.981203+0000 mon.a (mon.0) 2856 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:45.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: cluster 2024-04-02T04:55:43.986456+0000 mon.a (mon.0) 2857 : cluster 0 osdmap e496: 8 total, 8 up, 7 in 2024-04-02T04:55:45.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: audit 2024-04-02T04:55:43.990282+0000 mon.c (mon.2) 861 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: audit 2024-04-02T04:55:44.003419+0000 mon.c (mon.2) 862 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: audit 2024-04-02T04:55:44.006462+0000 mon.a (mon.0) 2858 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: audit 2024-04-02T04:55:44.013861+0000 mon.c (mon.2) 863 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: cluster 2024-04-02T04:55:44.141398+0000 mgr.y (mgr.14538) 474 : cluster 0 pgmap v725: 353 pgs: 5 creating+activating, 2 peering, 10 creating+peering, 17 unknown, 319 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: audit 2024-04-02T04:55:44.916192+0000 mon.a (mon.0) 2859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[21528]: cluster 2024-04-02T04:55:44.924090+0000 mon.a (mon.0) 2860 : cluster 0 osdmap e497: 8 total, 8 up, 7 in 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: audit 2024-04-02T04:55:43.981203+0000 mon.a (mon.0) 2856 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: cluster 2024-04-02T04:55:43.986456+0000 mon.a (mon.0) 2857 : cluster 0 osdmap e496: 8 total, 8 up, 7 in 2024-04-02T04:55:45.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: audit 2024-04-02T04:55:43.990282+0000 mon.c (mon.2) 861 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: audit 2024-04-02T04:55:44.003419+0000 mon.c (mon.2) 862 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: audit 2024-04-02T04:55:44.006462+0000 mon.a (mon.0) 2858 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: audit 2024-04-02T04:55:44.013861+0000 mon.c (mon.2) 863 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: cluster 2024-04-02T04:55:44.141398+0000 mgr.y (mgr.14538) 474 : cluster 0 pgmap v725: 353 pgs: 5 creating+activating, 2 peering, 10 creating+peering, 17 unknown, 319 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: audit 2024-04-02T04:55:44.916192+0000 mon.a (mon.0) 2859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:44 smithi069 bash[28904]: cluster 2024-04-02T04:55:44.924090+0000 mon.a (mon.0) 2860 : cluster 0 osdmap e497: 8 total, 8 up, 7 in 2024-04-02T04:55:45.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:45.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: audit 2024-04-02T04:55:43.981203+0000 mon.a (mon.0) 2856 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:55:45.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: cluster 2024-04-02T04:55:43.986456+0000 mon.a (mon.0) 2857 : cluster 0 osdmap e496: 8 total, 8 up, 7 in 2024-04-02T04:55:45.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: audit 2024-04-02T04:55:43.990282+0000 mon.c (mon.2) 861 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:55:45.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: audit 2024-04-02T04:55:44.003419+0000 mon.c (mon.2) 862 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:45.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: audit 2024-04-02T04:55:44.006462+0000 mon.a (mon.0) 2858 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:55:45.329 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: audit 2024-04-02T04:55:44.013861+0000 mon.c (mon.2) 863 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:45.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: cluster 2024-04-02T04:55:44.141398+0000 mgr.y (mgr.14538) 474 : cluster 0 pgmap v725: 353 pgs: 5 creating+activating, 2 peering, 10 creating+peering, 17 unknown, 319 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:45.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: audit 2024-04-02T04:55:44.916192+0000 mon.a (mon.0) 2859 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:55:45.330 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:44 smithi098 bash[23790]: cluster 2024-04-02T04:55:44.924090+0000 mon.a (mon.0) 2860 : cluster 0 osdmap e497: 8 total, 8 up, 7 in 2024-04-02T04:55:45.345 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:45.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:45.477 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:45.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:45.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:45.737 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:45.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:45.869 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:45.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:45.996 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:45.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:46.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:46.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:45 smithi098 bash[23790]: audit 2024-04-02T04:55:45.015392+0000 mon.c (mon.2) 864 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:46.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:45 smithi098 bash[23790]: cluster 2024-04-02T04:55:45.937318+0000 mon.a (mon.0) 2861 : cluster 0 osdmap e498: 8 total, 8 up, 7 in 2024-04-02T04:55:46.124 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:46.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:46.254 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:46.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:45 smithi069 bash[21528]: audit 2024-04-02T04:55:45.015392+0000 mon.c (mon.2) 864 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:46.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:45 smithi069 bash[21528]: cluster 2024-04-02T04:55:45.937318+0000 mon.a (mon.0) 2861 : cluster 0 osdmap e498: 8 total, 8 up, 7 in 2024-04-02T04:55:46.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:45 smithi069 bash[28904]: audit 2024-04-02T04:55:45.015392+0000 mon.c (mon.2) 864 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:46.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:45 smithi069 bash[28904]: cluster 2024-04-02T04:55:45.937318+0000 mon.a (mon.0) 2861 : cluster 0 osdmap e498: 8 total, 8 up, 7 in 2024-04-02T04:55:46.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:46.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:46.520 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:46.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:46.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:46.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:46.782 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:46.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:46.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:47.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:47.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[21528]: audit 2024-04-02T04:55:46.016301+0000 mon.c (mon.2) 865 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[21528]: audit 2024-04-02T04:55:46.138728+0000 mon.b (mon.1) 255 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[21528]: cluster 2024-04-02T04:55:46.142371+0000 mgr.y (mgr.14538) 475 : cluster 0 pgmap v728: 352 pgs: 5 creating+activating, 1 peering, 10 creating+peering, 17 unknown, 319 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[21528]: audit 2024-04-02T04:55:46.146226+0000 mon.a (mon.0) 2862 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[21528]: audit 2024-04-02T04:55:46.933919+0000 mon.a (mon.0) 2863 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[21528]: cluster 2024-04-02T04:55:46.941832+0000 mon.a (mon.0) 2864 : cluster 0 osdmap e499: 8 total, 8 up, 7 in 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[28904]: audit 2024-04-02T04:55:46.016301+0000 mon.c (mon.2) 865 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[28904]: audit 2024-04-02T04:55:46.138728+0000 mon.b (mon.1) 255 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[28904]: cluster 2024-04-02T04:55:46.142371+0000 mgr.y (mgr.14538) 475 : cluster 0 pgmap v728: 352 pgs: 5 creating+activating, 1 peering, 10 creating+peering, 17 unknown, 319 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[28904]: audit 2024-04-02T04:55:46.146226+0000 mon.a (mon.0) 2862 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[28904]: audit 2024-04-02T04:55:46.933919+0000 mon.a (mon.0) 2863 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-02T04:55:47.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:46 smithi069 bash[28904]: cluster 2024-04-02T04:55:46.941832+0000 mon.a (mon.0) 2864 : cluster 0 osdmap e499: 8 total, 8 up, 7 in 2024-04-02T04:55:47.039 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:47.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:47.154 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:46 smithi098 bash[23790]: audit 2024-04-02T04:55:46.016301+0000 mon.c (mon.2) 865 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:47.154 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:46 smithi098 bash[23790]: audit 2024-04-02T04:55:46.138728+0000 mon.b (mon.1) 255 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-02T04:55:47.154 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:46 smithi098 bash[23790]: cluster 2024-04-02T04:55:46.142371+0000 mgr.y (mgr.14538) 475 : cluster 0 pgmap v728: 352 pgs: 5 creating+activating, 1 peering, 10 creating+peering, 17 unknown, 319 active+clean; 8.4 MiB data, 512 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:47.154 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:46 smithi098 bash[23790]: audit 2024-04-02T04:55:46.146226+0000 mon.a (mon.0) 2862 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-02T04:55:47.155 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:46 smithi098 bash[23790]: audit 2024-04-02T04:55:46.933919+0000 mon.a (mon.0) 2863 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-02T04:55:47.155 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:46 smithi098 bash[23790]: cluster 2024-04-02T04:55:46.941832+0000 mon.a (mon.0) 2864 : cluster 0 osdmap e499: 8 total, 8 up, 7 in 2024-04-02T04:55:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:47.207 INFO:tasks.workunit.client.0.smithi069.stdout:misc: started 0x55c99fcc7070 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7770 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc72c0 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc73f0 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7520 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc78c0 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7d70 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7ea0 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7650 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7ff0 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8140 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8700 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc5390 2024-04-02T04:55:47.208 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc54c0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc55f0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8850 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8250 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc83a0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc84d0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8980 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8e50 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8f80 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc85e0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc90b0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9200 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8a90 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8bc0 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc8d10 2024-04-02T04:55:47.209 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9330 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9820 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9950 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9f30 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9480 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc95d0 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9a80 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9bb0 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9cc0 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9df0 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca530 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca080 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca1b0 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc9700 2024-04-02T04:55:47.210 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca2c0 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca410 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca660 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccab10 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccac40 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb240 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca770 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca8c0 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccad90 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccaee0 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb010 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb390 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb860 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb990 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccbf70 2024-04-02T04:55:47.211 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb4c0 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcca9f0 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb610 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccb740 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccbac0 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc79d0 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7b20 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcc7c50 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc550 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc0a0 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc210 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc340 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc6a0 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcccb50 2024-04-02T04:55:47.212 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcccc80 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd260 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc7b0 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccc8e0 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccca10 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd3b0 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcccdb0 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcccee0 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd030 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd4e0 2024-04-02T04:55:47.213 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd9b0 2024-04-02T04:55:47.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:47.299 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:47.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:47.429 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:47.522 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:47.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:47.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:47.847 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:47.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:47.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:48.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:48.121 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:48 smithi069 bash[21528]: audit 2024-04-02T04:55:47.017371+0000 mon.c (mon.2) 866 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:48.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:48 smithi069 bash[21528]: cluster 2024-04-02T04:55:47.944379+0000 mon.a (mon.0) 2865 : cluster 0 osdmap e500: 8 total, 8 up, 7 in 2024-04-02T04:55:48.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:48 smithi069 bash[28904]: audit 2024-04-02T04:55:47.017371+0000 mon.c (mon.2) 866 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:48.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:48 smithi069 bash[28904]: cluster 2024-04-02T04:55:47.944379+0000 mon.a (mon.0) 2865 : cluster 0 osdmap e500: 8 total, 8 up, 7 in 2024-04-02T04:55:48.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:48.176 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:55:48.176 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:55:48.176 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:55:48.176 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:55:48.176 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:55:48.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:48.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:48 smithi098 bash[23790]: audit 2024-04-02T04:55:47.017371+0000 mon.c (mon.2) 866 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:48.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:48 smithi098 bash[23790]: cluster 2024-04-02T04:55:47.944379+0000 mon.a (mon.0) 2865 : cluster 0 osdmap e500: 8 total, 8 up, 7 in 2024-04-02T04:55:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:48.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:48.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:48.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:48.521 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:48.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:48.650 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:48.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:48.781 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:48.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:48.896 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:55:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:55:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:55:48.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:48.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:49.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-02T04:55:49.045 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:49.146 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[21528]: audit 2024-04-02T04:55:48.018717+0000 mon.c (mon.2) 867 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[21528]: audit 2024-04-02T04:55:48.141907+0000 mon.b (mon.1) 256 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[21528]: cluster 2024-04-02T04:55:48.143652+0000 mgr.y (mgr.14538) 476 : cluster 0 pgmap v731: 352 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 activating, 5 creating+activating, 341 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[21528]: audit 2024-04-02T04:55:48.144563+0000 mon.a (mon.0) 2866 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[21528]: cluster 2024-04-02T04:55:48.941565+0000 mon.a (mon.0) 2867 : cluster 0 osdmap e501: 8 total, 8 up, 7 in 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[28904]: audit 2024-04-02T04:55:48.018717+0000 mon.c (mon.2) 867 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[28904]: audit 2024-04-02T04:55:48.141907+0000 mon.b (mon.1) 256 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[28904]: cluster 2024-04-02T04:55:48.143652+0000 mgr.y (mgr.14538) 476 : cluster 0 pgmap v731: 352 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 activating, 5 creating+activating, 341 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-02T04:55:49.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[28904]: audit 2024-04-02T04:55:48.144563+0000 mon.a (mon.0) 2866 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:49.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:49 smithi069 bash[28904]: cluster 2024-04-02T04:55:48.941565+0000 mon.a (mon.0) 2867 : cluster 0 osdmap e501: 8 total, 8 up, 7 in 2024-04-02T04:55:49.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.172 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:49.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:49 smithi098 bash[23790]: audit 2024-04-02T04:55:48.018717+0000 mon.c (mon.2) 867 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:49.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:49 smithi098 bash[23790]: audit 2024-04-02T04:55:48.141907+0000 mon.b (mon.1) 256 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:49.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:49 smithi098 bash[23790]: cluster 2024-04-02T04:55:48.143652+0000 mgr.y (mgr.14538) 476 : cluster 0 pgmap v731: 352 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 activating, 5 creating+activating, 341 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-02T04:55:49.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:49 smithi098 bash[23790]: audit 2024-04-02T04:55:48.144563+0000 mon.a (mon.0) 2866 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:55:49.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:49 smithi098 bash[23790]: cluster 2024-04-02T04:55:48.941565+0000 mon.a (mon.0) 2867 : cluster 0 osdmap e501: 8 total, 8 up, 7 in 2024-04-02T04:55:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:49.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.424 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:49.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:49.556 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:49.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:49.685 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:49.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:49.812 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:49.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:49.948 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:49.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:50.012 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:55:50.013 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:55:50.013 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 6, 7] 2024-04-02T04:55:50.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd out 3 2024-04-02T04:55:50.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:50.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:50.197 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:50 smithi069 bash[21528]: audit 2024-04-02T04:55:49.019918+0000 mon.c (mon.2) 868 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:50.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:50 smithi069 bash[21528]: cluster 2024-04-02T04:55:49.947853+0000 mon.a (mon.0) 2868 : cluster 0 osdmap e502: 8 total, 8 up, 7 in 2024-04-02T04:55:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:50 smithi069 bash[28904]: audit 2024-04-02T04:55:49.019918+0000 mon.c (mon.2) 868 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:50.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:50 smithi069 bash[28904]: cluster 2024-04-02T04:55:49.947853+0000 mon.a (mon.0) 2868 : cluster 0 osdmap e502: 8 total, 8 up, 7 in 2024-04-02T04:55:50.211 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:50.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:50 smithi098 bash[23790]: audit 2024-04-02T04:55:49.019918+0000 mon.c (mon.2) 868 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:50.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:50 smithi098 bash[23790]: cluster 2024-04-02T04:55:49.947853+0000 mon.a (mon.0) 2868 : cluster 0 osdmap e502: 8 total, 8 up, 7 in 2024-04-02T04:55:50.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:50.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:50.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:50.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:50.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:50.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:50.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:50.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:51.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:51.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:51 smithi098 bash[23790]: audit 2024-04-02T04:55:50.020910+0000 mon.c (mon.2) 869 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:51.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:51 smithi098 bash[23790]: audit 2024-04-02T04:55:50.140531+0000 mon.b (mon.1) 257 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-02T04:55:51.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:51 smithi098 bash[23790]: cluster 2024-04-02T04:55:50.144139+0000 mgr.y (mgr.14538) 477 : cluster 0 pgmap v734: 351 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 activating, 346 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-02T04:55:51.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:51 smithi098 bash[23790]: audit 2024-04-02T04:55:50.146973+0000 mon.a (mon.0) 2869 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-02T04:55:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:51.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[21528]: audit 2024-04-02T04:55:50.020910+0000 mon.c (mon.2) 869 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[21528]: audit 2024-04-02T04:55:50.140531+0000 mon.b (mon.1) 257 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-02T04:55:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[21528]: cluster 2024-04-02T04:55:50.144139+0000 mgr.y (mgr.14538) 477 : cluster 0 pgmap v734: 351 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 activating, 346 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-02T04:55:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[21528]: audit 2024-04-02T04:55:50.146973+0000 mon.a (mon.0) 2869 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-02T04:55:51.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[28904]: audit 2024-04-02T04:55:50.020910+0000 mon.c (mon.2) 869 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:51.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[28904]: audit 2024-04-02T04:55:50.140531+0000 mon.b (mon.1) 257 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-02T04:55:51.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[28904]: cluster 2024-04-02T04:55:50.144139+0000 mgr.y (mgr.14538) 477 : cluster 0 pgmap v734: 351 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 activating, 346 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-02T04:55:51.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:51 smithi069 bash[28904]: audit 2024-04-02T04:55:50.146973+0000 mon.a (mon.0) 2869 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-02T04:55:51.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:51.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:51.387 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:51.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:51.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:51.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:51.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:51.780 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:51.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:51.909 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:51.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:52.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:52.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:52.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:52.172 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:52.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:52.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[21528]: audit 2024-04-02T04:55:51.012803+0000 mon.a (mon.0) 2870 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-02T04:55:52.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[21528]: cluster 2024-04-02T04:55:51.019059+0000 mon.a (mon.0) 2871 : cluster 0 osdmap e503: 8 total, 8 up, 7 in 2024-04-02T04:55:52.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[21528]: audit 2024-04-02T04:55:51.023900+0000 mon.c (mon.2) 870 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:52.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[21528]: audit 2024-04-02T04:55:52.025530+0000 mon.c (mon.2) 871 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:52.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[21528]: cluster 2024-04-02T04:55:52.026582+0000 mon.a (mon.0) 2872 : cluster 0 osdmap e504: 8 total, 8 up, 7 in 2024-04-02T04:55:52.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[28904]: audit 2024-04-02T04:55:51.012803+0000 mon.a (mon.0) 2870 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-02T04:55:52.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[28904]: cluster 2024-04-02T04:55:51.019059+0000 mon.a (mon.0) 2871 : cluster 0 osdmap e503: 8 total, 8 up, 7 in 2024-04-02T04:55:52.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[28904]: audit 2024-04-02T04:55:51.023900+0000 mon.c (mon.2) 870 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:52.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[28904]: audit 2024-04-02T04:55:52.025530+0000 mon.c (mon.2) 871 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:52.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:52 smithi069 bash[28904]: cluster 2024-04-02T04:55:52.026582+0000 mon.a (mon.0) 2872 : cluster 0 osdmap e504: 8 total, 8 up, 7 in 2024-04-02T04:55:52.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:52 smithi098 bash[23790]: audit 2024-04-02T04:55:51.012803+0000 mon.a (mon.0) 2870 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-02T04:55:52.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:52 smithi098 bash[23790]: cluster 2024-04-02T04:55:51.019059+0000 mon.a (mon.0) 2871 : cluster 0 osdmap e503: 8 total, 8 up, 7 in 2024-04-02T04:55:52.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:52 smithi098 bash[23790]: audit 2024-04-02T04:55:51.023900+0000 mon.c (mon.2) 870 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:52.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:52 smithi098 bash[23790]: audit 2024-04-02T04:55:52.025530+0000 mon.c (mon.2) 871 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:52.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:52 smithi098 bash[23790]: cluster 2024-04-02T04:55:52.026582+0000 mon.a (mon.0) 2872 : cluster 0 osdmap e504: 8 total, 8 up, 7 in 2024-04-02T04:55:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:52.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:52.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:52.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:52.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/3212937290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 msgr2=0x7fcf84101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- 172.21.15.69:0/3212937290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84101620 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009990 tx=0x7fcf70009ca0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/3212937290 shutdown_connections 2024-04-02T04:55:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- 172.21.15.69:0/3212937290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 0x7fcf8410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- 172.21.15.69:0/3212937290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf84107220 0x7fcf84107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- 172.21.15.69:0/3212937290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84101620 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/3212937290 >> 172.21.15.69:0/3212937290 conn(0x7fcf840771e0 msgr2=0x7fcf840775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/3212937290 shutdown_connections 2024-04-02T04:55:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/3212937290 wait complete. 2024-04-02T04:55:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 librados: starting msgr at 2024-04-02T04:55:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 librados: starting objecter 2024-04-02T04:55:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 Processor -- start 2024-04-02T04:55:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- start start 2024-04-02T04:55:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 librados: setting wanted keys 2024-04-02T04:55:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 librados: calling monclient init 2024-04-02T04:55:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 0x7fcf84198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf84107220 0x7fcf8419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcf84110970 con 0x7fcf841011a0 2024-04-02T04:55:52.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf841107f0 con 0x7fcf84107220 2024-04-02T04:55:52.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcf84110af0 con 0x7fcf84101b60 2024-04-02T04:55:52.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 0x7fcf84198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf88e1f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 0x7fcf84198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42548/0 (socket says 172.21.15.69:42548) 2024-04-02T04:55:52.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf88e1f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51720/0 (socket says 172.21.15.69:51720) 2024-04-02T04:55:52.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 -- 172.21.15.69:0/1865748997 learned_addr learned my addr 172.21.15.69:0/1865748997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:52.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf84107220 0x7fcf8419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf84107220 msgr2=0x7fcf8419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf84107220 0x7fcf8419c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 msgr2=0x7fcf84197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8419cf10 con 0x7fcf84101b60 2024-04-02T04:55:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf841011a0 0x7fcf84197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:55:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 0x7fcf84198440 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcf700057f0 tx=0x7fcf7000bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf81ffb700 10 client.?.objecter ms_handle_connect 0x7fcf84101b60 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf81ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70026070 con 0x7fcf84101b60 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8419d1a0 con 0x7fcf84101b60 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fcf70021410 con 0x7fcf84101b60 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70017650 con 0x7fcf84101b60 2024-04-02T04:55:52.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 10 client.26612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf841a4a80 con 0x7fcf84101b60 2024-04-02T04:55:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.749+0000 7fcf8b083700 1 librados: init done 2024-04-02T04:55:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fcf8404ff00 con 0x7fcf84101b60 2024-04-02T04:55:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fcf70029070 con 0x7fcf84101b60 2024-04-02T04:55:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf7406f5e0 0x7fcf74071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(504..504 src has 74..504) ==== 10953+0+0 (secure 0 0 0) 0x7fcf700901d0 con 0x7fcf84101b60 2024-04-02T04:55:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 osd_map(504..504 src has 74..504) 2024-04-02T04:55:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 3 client.26612.objecter handle_osd_map got epochs [504,504] > 0 2024-04-02T04:55:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 3 client.26612.objecter handle_osd_map decoding full epoch 504 2024-04-02T04:55:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf7406f5e0 0x7fcf74071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 20 client.26612.objecter dump_active .. 0 homeless 2024-04-02T04:55:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v504) ==== 54+0+16 (secure 0 0 0) 0x7fcf70030020 con 0x7fcf84101b60 2024-04-02T04:55:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 10 client.26612.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 1 2024-04-02T04:55:52.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fcf84062e50 0x7fcf84063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 1 2024-04-02T04:55:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 0 1 2024-04-02T04:55:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 4 2024-04-02T04:55:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 10 client.26612.objecter _send_command 1 2024-04-02T04:55:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf84104d60 con 0x7fcf84062e50 2024-04-02T04:55:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fcf84062e50 0x7fcf84063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf7406f5e0 0x7fcf74071aa0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800afd0 tx=0x7fcf7800a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf7406f5e0 2024-04-02T04:55:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf84062e50 2024-04-02T04:55:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fcf84062e50 0x7fcf84063230 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf84104d60 con 0x7fcf84062e50 2024-04-02T04:55:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 1: 0 ) 2024-04-02T04:55:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 1 = system:0 2024-04-02T04:55:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 4 2024-04-02T04:55:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.753+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 0 1 2024-04-02T04:55:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 10 client.26612.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 2 2024-04-02T04:55:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 2 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf84199260 osd=0 3 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 0 2 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 4 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 10 client.26612.objecter _send_command 2 2024-04-02T04:55:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf840628d0 con 0x7fcf84062e50 2024-04-02T04:55:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf840628d0 con 0x7fcf84062e50 2024-04-02T04:55:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 4 2024-04-02T04:55:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 0 2 2024-04-02T04:55:52.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:55:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 10 client.26612.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 3 2024-04-02T04:55:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fcf8410ac00 0x7fcf841ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 3 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 1 3 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 4 2024-04-02T04:55:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 10 client.26612.objecter _send_command 3 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fcf8402dca0 con 0x7fcf8410ac00 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fcf8410ac00 0x7fcf841ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf8410ac00 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fcf8410ac00 0x7fcf841ab070 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.777+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf8402dca0 con 0x7fcf8410ac00 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.781+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 3: 0 ) 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.781+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 3 = system:0 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.781+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 4 2024-04-02T04:55:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.781+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 1 3 2024-04-02T04:55:52.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 10 client.26612.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 4 2024-04-02T04:55:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 4 2024-04-02T04:55:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf8402db30 osd=1 3 2024-04-02T04:55:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 1 4 2024-04-02T04:55:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 4 2024-04-02T04:55:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 10 client.26612.objecter _send_command 4 2024-04-02T04:55:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.797+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841ab5b0 con 0x7fcf8410ac00 2024-04-02T04:55:52.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841ab5b0 con 0x7fcf8410ac00 2024-04-02T04:55:52.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 4 2024-04-02T04:55:52.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 1 4 2024-04-02T04:55:52.800 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 10 client.26612.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 5 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fcf8410a2e0 0x7fcf841ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 5 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 2 5 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 4 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 10 client.26612.objecter _send_command 5 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fcf8402d690 con 0x7fcf8410a2e0 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fcf8410a2e0 0x7fcf841ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf8410a2e0 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fcf8410a2e0 0x7fcf841ae8f0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf8402d690 con 0x7fcf8410a2e0 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 5: 0 ) 2024-04-02T04:55:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 5 = system:0 2024-04-02T04:55:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 4 2024-04-02T04:55:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.801+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 2 5 2024-04-02T04:55:52.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 10 client.26612.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 6 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 6 2024-04-02T04:55:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf8410a170 osd=2 3 2024-04-02T04:55:52.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 2 6 2024-04-02T04:55:52.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 4 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 10 client.26612.objecter _send_command 6 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.817+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841aee30 con 0x7fcf8410a2e0 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841aee30 con 0x7fcf8410a2e0 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 4 2024-04-02T04:55:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 2 6 2024-04-02T04:55:52.822 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 10 client.26612.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 7 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fcf841abf30 0x7fcf841ac350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 7 2024-04-02T04:55:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 3 7 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 4 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 10 client.26612.objecter _send_command 7 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fcf841b4bb0 con 0x7fcf841abf30 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fcf841abf30 0x7fcf841ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf841abf30 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fcf841abf30 0x7fcf841ac350 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf841b4bb0 con 0x7fcf841abf30 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 7: 0 ) 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 7 = system:0 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 4 2024-04-02T04:55:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.821+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 3 7 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 10 client.26612.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 8 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 8 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841aee30 osd=3 3 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 3 8 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 4 2024-04-02T04:55:52.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 10 client.26612.objecter _send_command 8 2024-04-02T04:55:52.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841ab9b0 con 0x7fcf841abf30 2024-04-02T04:55:52.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841ab9b0 con 0x7fcf841abf30 2024-04-02T04:55:52.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 4 2024-04-02T04:55:52.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 3 8 2024-04-02T04:55:52.842 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 10 client.26612.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 9 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fcf841afd20 0x7fcf841b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 9 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 4 9 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 4 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 10 client.26612.objecter _send_command 9 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fcf841b8570 con 0x7fcf841afd20 2024-04-02T04:55:52.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fcf841afd20 0x7fcf841b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf841afd20 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.841+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fcf841afd20 0x7fcf841b21d0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.845+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf841b8570 con 0x7fcf841afd20 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.845+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 9: 0 ) 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.845+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 9 = system:0 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.845+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 4 2024-04-02T04:55:52.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.845+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 4 9 2024-04-02T04:55:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 10 client.26612.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 10 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 10 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841ab9b0 osd=4 3 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 4 10 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 4 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 10 client.26612.objecter _send_command 10 2024-04-02T04:55:52.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841b2710 con 0x7fcf841afd20 2024-04-02T04:55:52.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841b2710 con 0x7fcf841afd20 2024-04-02T04:55:52.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 4 2024-04-02T04:55:52.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 4 10 2024-04-02T04:55:52.863 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 10 client.26612.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 11 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fcf841b2d90 0x7fcf841b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 11 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 5 11 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 4 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 10 client.26612.objecter _send_command 11 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.861+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fcf841bb700 con 0x7fcf841b2d90 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fcf841b2d90 0x7fcf841b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf83fff700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fcf841b2d90 0x7fcf841b31f0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf841b2d90 2024-04-02T04:55:52.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf841bb700 con 0x7fcf841b2d90 2024-04-02T04:55:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 11: 0 ) 2024-04-02T04:55:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 11 = system:0 2024-04-02T04:55:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 4 2024-04-02T04:55:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.865+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 5 11 2024-04-02T04:55:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 10 client.26612.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 12 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 12 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841b2710 osd=5 3 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 5 12 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 4 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 10 client.26612.objecter _send_command 12 2024-04-02T04:55:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.881+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841b8de0 con 0x7fcf841b2d90 2024-04-02T04:55:52.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841b8de0 con 0x7fcf841b2d90 2024-04-02T04:55:52.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 4 2024-04-02T04:55:52.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 5 12 2024-04-02T04:55:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 10 client.26612.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 13 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fcf841bacf0 0x7fcf841bb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 13 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 6 13 2024-04-02T04:55:52.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 4 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 10 client.26612.objecter _send_command 13 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fcf841beaa0 con 0x7fcf841bacf0 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.885+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fcf841bacf0 0x7fcf841bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf841bacf0 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf89620700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fcf841bacf0 0x7fcf841bb130 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf841beaa0 con 0x7fcf841bacf0 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 13: 0 ) 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 13 = system:0 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 4 2024-04-02T04:55:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.889+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 6 13 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 10 client.26612.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 14 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 14 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841b8de0 osd=6 3 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 6 14 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 4 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 10 client.26612.objecter _send_command 14 2024-04-02T04:55:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.905+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841abc20 con 0x7fcf841bacf0 2024-04-02T04:55:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841abc20 con 0x7fcf841bacf0 2024-04-02T04:55:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 4 2024-04-02T04:55:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 6 14 2024-04-02T04:55:52.909 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 10 client.26612.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 15 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fcf841ba170 0x7fcf841b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 15 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 7 15 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 4 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 10 client.26612.objecter _send_command 15 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fcf841b46b0 con 0x7fcf841ba170 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fcf841ba170 0x7fcf841b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf88e1f700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fcf841ba170 0x7fcf841b4170 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.909+0000 7fcf81ffb700 10 client.26612.objecter ms_handle_connect 0x7fcf841ba170 2024-04-02T04:55:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.913+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf841b46b0 con 0x7fcf841ba170 2024-04-02T04:55:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.913+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 15: 0 ) 2024-04-02T04:55:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.913+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 15 = system:0 2024-04-02T04:55:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.913+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 4 2024-04-02T04:55:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.913+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 7 15 2024-04-02T04:55:52.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:52.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 10 client.26612.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign -1 16 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 20 client.26612.objecter _get_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_remove -1 16 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 20 client.26612.objecter get_session s=0x7fcf841abc20 osd=7 3 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 15 client.26612.objecter _session_command_op_assign 7 16 2024-04-02T04:55:52.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 4 2024-04-02T04:55:52.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 10 client.26612.objecter _send_command 16 2024-04-02T04:55:52.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.929+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcf841b46b0 con 0x7fcf841ba170 2024-04-02T04:55:52.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf81ffb700 1 -- 172.21.15.69:0/1865748997 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcf841b46b0 con 0x7fcf841ba170 2024-04-02T04:55:52.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf81ffb700 10 client.26612.objecter ms_dispatch 0x7fcf84001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:55:52.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf81ffb700 10 client.26612.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:55:52.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf81ffb700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 4 2024-04-02T04:55:52.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf81ffb700 15 client.26612.objecter _session_command_op_remove 7 16 2024-04-02T04:55:52.935 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:55:52.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 librados: watch_flush enter 2024-04-02T04:55:52.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 librados: watch_flush exit 2024-04-02T04:55:52.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.0 2024-04-02T04:55:52.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fcf84062e50 msgr2=0x7fcf84063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fcf84062e50 0x7fcf84063230 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf84199260 osd=0 1 2024-04-02T04:55:52.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.1 2024-04-02T04:55:52.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fcf8410ac00 msgr2=0x7fcf841ab070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fcf8410ac00 0x7fcf841ab070 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8402db30 osd=1 1 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.2 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fcf8410a2e0 msgr2=0x7fcf841ae8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fcf8410a2e0 0x7fcf841ae8f0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf8410a170 osd=2 1 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.3 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fcf841abf30 msgr2=0x7fcf841ac350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fcf841abf30 0x7fcf841ac350 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841aee30 osd=3 1 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.4 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fcf841afd20 msgr2=0x7fcf841b21d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fcf841afd20 0x7fcf841b21d0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841ab9b0 osd=4 1 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.5 2024-04-02T04:55:52.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fcf841b2d90 msgr2=0x7fcf841b31f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fcf841b2d90 0x7fcf841b31f0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b2710 osd=5 1 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.6 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fcf841bacf0 msgr2=0x7fcf841bb130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fcf841bacf0 0x7fcf841bb130 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841b8de0 osd=6 1 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter close_session for osd.7 2024-04-02T04:55:52.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fcf841ba170 msgr2=0x7fcf841b4170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fcf841ba170 0x7fcf841b4170 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf89620700 1 -- 172.21.15.69:0/1865748997 reap_dead start 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter put_session s=0x7fcf841abc20 osd=7 1 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 20 client.26612.objecter shutdown clearing up homeless session... 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 10 client.26612.objecter successfully canceled tick 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf7406f5e0 msgr2=0x7fcf74071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcf7406f5e0 0x7fcf74071aa0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800afd0 tx=0x7fcf7800a380 comp rx=0 tx=0).stop 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf89620700 1 -- 172.21.15.69:0/1865748997 reap_dead start 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 msgr2=0x7fcf84198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 --2- 172.21.15.69:0/1865748997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf84101b60 0x7fcf84198440 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcf700057f0 tx=0x7fcf7000bfa0 comp rx=0 tx=0).stop 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf89620700 1 -- 172.21.15.69:0/1865748997 reap_dead start 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf89620700 1 -- 172.21.15.69:0/1865748997 reap_dead start 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf89620700 1 -- 172.21.15.69:0/1865748997 reap_dead start 2024-04-02T04:55:52.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 shutdown_connections 2024-04-02T04:55:52.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 >> 172.21.15.69:0/1865748997 conn(0x7fcf840771e0 msgr2=0x7fcf840ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:52.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 shutdown_connections 2024-04-02T04:55:52.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 -- 172.21.15.69:0/1865748997 wait complete. 2024-04-02T04:55:52.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:52.933+0000 7fcf8b083700 1 librados: shutdown 2024-04-02T04:55:52.944 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:55:52.944 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:55:52.944 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:55:52.944 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:55:52.944 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:55:52.944 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:55:52.945 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:55:52.945 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:55:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:52.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:53.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:53.072 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:53.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:53.186 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:53 smithi098 bash[23790]: cluster 2024-04-02T04:55:52.144671+0000 mgr.y (mgr.14538) 478 : cluster 0 pgmap v737: 351 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 activating, 346 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:53.186 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:53 smithi098 bash[23790]: audit 2024-04-02T04:55:52.756435+0000 mon.c (mon.2) 872 : audit 0 from='client.? 172.21.15.69:0/1865748997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:53.186 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:53 smithi098 bash[23790]: audit 2024-04-02T04:55:53.027088+0000 mon.c (mon.2) 873 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:53.201 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:53.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:53.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:53 smithi069 bash[21528]: cluster 2024-04-02T04:55:52.144671+0000 mgr.y (mgr.14538) 478 : cluster 0 pgmap v737: 351 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 activating, 346 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:53.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:53 smithi069 bash[21528]: audit 2024-04-02T04:55:52.756435+0000 mon.c (mon.2) 872 : audit 0 from='client.? 172.21.15.69:0/1865748997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:53.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:53 smithi069 bash[21528]: audit 2024-04-02T04:55:53.027088+0000 mon.c (mon.2) 873 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:53.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:53 smithi069 bash[28904]: cluster 2024-04-02T04:55:52.144671+0000 mgr.y (mgr.14538) 478 : cluster 0 pgmap v737: 351 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 activating, 346 active+clean; 8.4 MiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:55:53.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:53 smithi069 bash[28904]: audit 2024-04-02T04:55:52.756435+0000 mon.c (mon.2) 872 : audit 0 from='client.? 172.21.15.69:0/1865748997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:55:53.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:53 smithi069 bash[28904]: audit 2024-04-02T04:55:53.027088+0000 mon.c (mon.2) 873 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:53.332 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:53.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:53.447 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:55:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:55:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:53.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:53.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:55:53.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:53.846 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:53.949 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:53.977 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:53.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:54.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:54.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:54.217 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:54 smithi069 bash[21528]: cluster 2024-04-02T04:55:53.040244+0000 mon.a (mon.0) 2873 : cluster 0 osdmap e505: 8 total, 8 up, 7 in 2024-04-02T04:55:54.217 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:54 smithi069 bash[21528]: audit 2024-04-02T04:55:53.328915+0000 mgr.y (mgr.14538) 479 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:54.217 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:54 smithi069 bash[21528]: audit 2024-04-02T04:55:54.028102+0000 mon.c (mon.2) 874 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:54.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:54 smithi069 bash[28904]: cluster 2024-04-02T04:55:53.040244+0000 mon.a (mon.0) 2873 : cluster 0 osdmap e505: 8 total, 8 up, 7 in 2024-04-02T04:55:54.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:54 smithi069 bash[28904]: audit 2024-04-02T04:55:53.328915+0000 mgr.y (mgr.14538) 479 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:54.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:54 smithi069 bash[28904]: audit 2024-04-02T04:55:54.028102+0000 mon.c (mon.2) 874 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:54.219 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:54 smithi098 bash[23790]: cluster 2024-04-02T04:55:53.040244+0000 mon.a (mon.0) 2873 : cluster 0 osdmap e505: 8 total, 8 up, 7 in 2024-04-02T04:55:54.219 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:54 smithi098 bash[23790]: audit 2024-04-02T04:55:53.328915+0000 mgr.y (mgr.14538) 479 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:55:54.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:54.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:54.363 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:54.466 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:54.490 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:54.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:54.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:54.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:54.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:54.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:54.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:55.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:55.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:55.125 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:54 smithi098 bash[23790]: audit 2024-04-02T04:55:54.028102+0000 mon.c (mon.2) 874 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: cluster 2024-04-02T04:55:54.056886+0000 mon.a (mon.0) 2874 : cluster 0 osdmap e506: 8 total, 8 up, 7 in 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: audit 2024-04-02T04:55:54.141839+0000 mon.b (mon.1) 258 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: cluster 2024-04-02T04:55:54.145355+0000 mgr.y (mgr.14538) 480 : cluster 0 pgmap v740: 350 pgs: 1 peering, 349 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: audit 2024-04-02T04:55:54.147753+0000 mon.a (mon.0) 2875 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: audit 2024-04-02T04:55:54.920228+0000 mon.a (mon.0) 2876 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: cluster 2024-04-02T04:55:54.928170+0000 mon.a (mon.0) 2877 : cluster 0 osdmap e507: 8 total, 8 up, 7 in 2024-04-02T04:55:55.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:55 smithi098 bash[23790]: audit 2024-04-02T04:55:55.029299+0000 mon.c (mon.2) 875 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:55.135 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:55.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:55.260 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: cluster 2024-04-02T04:55:54.056886+0000 mon.a (mon.0) 2874 : cluster 0 osdmap e506: 8 total, 8 up, 7 in 2024-04-02T04:55:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: audit 2024-04-02T04:55:54.141839+0000 mon.b (mon.1) 258 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-02T04:55:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: cluster 2024-04-02T04:55:54.145355+0000 mgr.y (mgr.14538) 480 : cluster 0 pgmap v740: 350 pgs: 1 peering, 349 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: audit 2024-04-02T04:55:54.147753+0000 mon.a (mon.0) 2875 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-02T04:55:55.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: audit 2024-04-02T04:55:54.920228+0000 mon.a (mon.0) 2876 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: cluster 2024-04-02T04:55:54.928170+0000 mon.a (mon.0) 2877 : cluster 0 osdmap e507: 8 total, 8 up, 7 in 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[21528]: audit 2024-04-02T04:55:55.029299+0000 mon.c (mon.2) 875 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: cluster 2024-04-02T04:55:54.056886+0000 mon.a (mon.0) 2874 : cluster 0 osdmap e506: 8 total, 8 up, 7 in 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: audit 2024-04-02T04:55:54.141839+0000 mon.b (mon.1) 258 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: cluster 2024-04-02T04:55:54.145355+0000 mgr.y (mgr.14538) 480 : cluster 0 pgmap v740: 350 pgs: 1 peering, 349 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: audit 2024-04-02T04:55:54.147753+0000 mon.a (mon.0) 2875 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: audit 2024-04-02T04:55:54.920228+0000 mon.a (mon.0) 2876 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: cluster 2024-04-02T04:55:54.928170+0000 mon.a (mon.0) 2877 : cluster 0 osdmap e507: 8 total, 8 up, 7 in 2024-04-02T04:55:55.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:55 smithi069 bash[28904]: audit 2024-04-02T04:55:55.029299+0000 mon.c (mon.2) 875 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:55.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:55.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:55.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 -- 172.21.15.69:0/1872554506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 msgr2=0x7fc39806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:55.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1872554506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc39806b030 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc388009990 tx=0x7fc388009ca0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 -- 172.21.15.69:0/1872554506 shutdown_connections 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1872554506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1872554506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc39806b030 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1872554506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 0x7fc3981043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.489+0000 7fc39ece6700 1 -- 172.21.15.69:0/1872554506 >> 172.21.15.69:0/1872554506 conn(0x7fc3980771e0 msgr2=0x7fc3980775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- 172.21.15.69:0/1872554506 shutdown_connections 2024-04-02T04:55:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- 172.21.15.69:0/1872554506 wait complete. 2024-04-02T04:55:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 librados: starting msgr at 2024-04-02T04:55:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 librados: starting objecter 2024-04-02T04:55:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 Processor -- start 2024-04-02T04:55:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- start start 2024-04-02T04:55:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 librados: setting wanted keys 2024-04-02T04:55:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 librados: calling monclient init 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc3981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 0x7fc398198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc398111bd0 con 0x7fc39806ac50 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc398111a50 con 0x7fc398103f60 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc398111d50 con 0x7fc398104920 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ca82700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc3981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ca82700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc3981981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51744/0 (socket says 172.21.15.69:51744) 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42560/0 (socket says 172.21.15.69:42560) 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ca82700 1 -- 172.21.15.69:0/1217784965 learned_addr learned my addr 172.21.15.69:0/1217784965 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:55:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc397fff700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 0x7fc398198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 -- 172.21.15.69:0/1217784965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 msgr2=0x7fc398198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 0x7fc398198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 -- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 msgr2=0x7fc3981981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc3981981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 -- 172.21.15.69:0/1217784965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39819d1e0 con 0x7fc398104920 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc397fff700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 0x7fc398198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ca82700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc3981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39d283700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39819caa0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc39000efc0 tx=0x7fc39000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc395ffb700 10 client.?.objecter ms_handle_connect 0x7fc398104920 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc395ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:55:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3900092e0 con 0x7fc398104920 2024-04-02T04:55:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc390004bb0 con 0x7fc398104920 2024-04-02T04:55:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39819d470 con 0x7fc398104920 2024-04-02T04:55:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390004020 con 0x7fc398104920 2024-04-02T04:55:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 10 client.26618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:55:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3981a4d50 con 0x7fc398104920 2024-04-02T04:55:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 librados: init done 2024-04-02T04:55:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39804ff00 con 0x7fc398104920 2024-04-02T04:55:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.493+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc39001b030 con 0x7fc398104920 2024-04-02T04:55:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc38406f5e0 0x7fc384071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:55:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(507..507 src has 74..507) ==== 10953+0+0 (secure 0 0 0) 0x7fc39008e8f0 con 0x7fc398104920 2024-04-02T04:55:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 10 client.26618.objecter ms_dispatch 0x7fc398001550 osd_map(507..507 src has 74..507) 2024-04-02T04:55:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 3 client.26618.objecter handle_osd_map got epochs [507,507] > 0 2024-04-02T04:55:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 3 client.26618.objecter handle_osd_map decoding full epoch 507 2024-04-02T04:55:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc39ca82700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc38406f5e0 0x7fc384071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:55:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 20 client.26618.objecter dump_active .. 0 homeless 2024-04-02T04:55:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc39ca82700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc38406f5e0 0x7fc384071aa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc38800b2e0 tx=0x7fc38801a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:55:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.497+0000 7fc395ffb700 10 client.26618.objecter ms_handle_connect 0x7fc38406f5e0 2024-04-02T04:55:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.501+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc390022020 con 0x7fc398104920 2024-04-02T04:55:55.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:55.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:55.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.725+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fc3981a4f80 con 0x7fc398104920 2024-04-02T04:55:55.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:55.773 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:55.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:55.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:55.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:55.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.941+0000 7fc395ffb700 1 -- 172.21.15.69:0/1217784965 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v508) ==== 97+0+0 (secure 0 0 0) 0x7fc390010040 con 0x7fc398104920 2024-04-02T04:55:55.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 10 librados: watch_flush enter 2024-04-02T04:55:55.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 10 librados: watch_flush exit 2024-04-02T04:55:55.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 20 client.26618.objecter shutdown clearing up homeless session... 2024-04-02T04:55:55.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 10 client.26618.objecter successfully canceled tick 2024-04-02T04:55:55.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc38406f5e0 msgr2=0x7fc384071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:55.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc38406f5e0 0x7fc384071aa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc38800b2e0 tx=0x7fc38801a040 comp rx=0 tx=0).stop 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 msgr2=0x7fc39819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.953+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39819caa0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc39000efc0 tx=0x7fc39000c3a0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 shutdown_connections 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc38406f5e0 0x7fc384071aa0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc398104920 0x7fc39819caa0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39806ac50 0x7fc3981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 --2- 172.21.15.69:0/1217784965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc398103f60 0x7fc398198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 >> 172.21.15.69:0/1217784965 conn(0x7fc3980771e0 msgr2=0x7fc398109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 shutdown_connections 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 -- 172.21.15.69:0/1217784965 wait complete. 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:55:55.957+0000 7fc39ece6700 1 librados: shutdown 2024-04-02T04:55:55.958 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:55:56.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:56.028 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:56.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:56.158 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:56.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:56.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:56 smithi098 bash[23790]: audit 2024-04-02T04:55:55.731217+0000 mon.c (mon.2) 876 : audit 1 from='client.? 172.21.15.69:0/1217784965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:56.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:56 smithi098 bash[23790]: audit 2024-04-02T04:55:55.731643+0000 mon.a (mon.0) 2878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:56.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:56 smithi098 bash[23790]: cluster 2024-04-02T04:55:55.936211+0000 mon.a (mon.0) 2879 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:56.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:56 smithi098 bash[23790]: audit 2024-04-02T04:55:55.938446+0000 mon.a (mon.0) 2880 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:55:56.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:56 smithi098 bash[23790]: cluster 2024-04-02T04:55:55.946775+0000 mon.a (mon.0) 2881 : cluster 0 osdmap e508: 8 total, 8 up, 7 in 2024-04-02T04:55:56.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:56 smithi098 bash[23790]: audit 2024-04-02T04:55:56.030385+0000 mon.c (mon.2) 877 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[21528]: audit 2024-04-02T04:55:55.731217+0000 mon.c (mon.2) 876 : audit 1 from='client.? 172.21.15.69:0/1217784965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[21528]: audit 2024-04-02T04:55:55.731643+0000 mon.a (mon.0) 2878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[21528]: cluster 2024-04-02T04:55:55.936211+0000 mon.a (mon.0) 2879 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:56.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[21528]: audit 2024-04-02T04:55:55.938446+0000 mon.a (mon.0) 2880 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[21528]: cluster 2024-04-02T04:55:55.946775+0000 mon.a (mon.0) 2881 : cluster 0 osdmap e508: 8 total, 8 up, 7 in 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[21528]: audit 2024-04-02T04:55:56.030385+0000 mon.c (mon.2) 877 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[28904]: audit 2024-04-02T04:55:55.731217+0000 mon.c (mon.2) 876 : audit 1 from='client.? 172.21.15.69:0/1217784965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[28904]: audit 2024-04-02T04:55:55.731643+0000 mon.a (mon.0) 2878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[28904]: cluster 2024-04-02T04:55:55.936211+0000 mon.a (mon.0) 2879 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[28904]: audit 2024-04-02T04:55:55.938446+0000 mon.a (mon.0) 2880 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[28904]: cluster 2024-04-02T04:55:55.946775+0000 mon.a (mon.0) 2881 : cluster 0 osdmap e508: 8 total, 8 up, 7 in 2024-04-02T04:55:56.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:56 smithi069 bash[28904]: audit 2024-04-02T04:55:56.030385+0000 mon.c (mon.2) 877 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:56.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:56.415 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:56.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:56.545 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:56.649 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:56.674 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:56.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:55:56.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:56.803 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:56.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:56.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:57.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:57.064 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:57.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:57.190 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:57.226 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:55:57.226 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:55:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:55:57.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:57.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:57 smithi069 bash[21528]: cluster 2024-04-02T04:55:56.146284+0000 mgr.y (mgr.14538) 481 : cluster 0 pgmap v743: 350 pgs: 1 peering, 349 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:57.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:57 smithi069 bash[21528]: cluster 2024-04-02T04:55:56.945748+0000 mon.a (mon.0) 2882 : cluster 0 osdmap e509: 8 total, 8 up, 7 in 2024-04-02T04:55:57.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:57 smithi069 bash[21528]: audit 2024-04-02T04:55:57.031843+0000 mon.c (mon.2) 878 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:57.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:57 smithi069 bash[28904]: cluster 2024-04-02T04:55:56.146284+0000 mgr.y (mgr.14538) 481 : cluster 0 pgmap v743: 350 pgs: 1 peering, 349 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:57.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:57 smithi069 bash[28904]: cluster 2024-04-02T04:55:56.945748+0000 mon.a (mon.0) 2882 : cluster 0 osdmap e509: 8 total, 8 up, 7 in 2024-04-02T04:55:57.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:57 smithi069 bash[28904]: audit 2024-04-02T04:55:57.031843+0000 mon.c (mon.2) 878 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:55:57.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:57.433 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:57 smithi098 bash[23790]: cluster 2024-04-02T04:55:56.146284+0000 mgr.y (mgr.14538) 481 : cluster 0 pgmap v743: 350 pgs: 1 peering, 349 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-04-02T04:55:57.433 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:57 smithi098 bash[23790]: cluster 2024-04-02T04:55:56.945748+0000 mon.a (mon.0) 2882 : cluster 0 osdmap e509: 8 total, 8 up, 7 in 2024-04-02T04:55:57.433 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:57 smithi098 bash[23790]: audit 2024-04-02T04:55:57.031843+0000 mon.c (mon.2) 878 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:57.454 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:55:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:55:57.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:57.585 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:57.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:57.716 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:57.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:57.845 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:57.949 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:57.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:57.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:58.085 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:58.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-02T04:55:58.113 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:58.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:58.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:58.466 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:58.488 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:58.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:58.615 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:58.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:58.745 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:55:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:55:58.847 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:55:58.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:58.858 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:55:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:55:58.881 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:58.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:58.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:58.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: cluster 2024-04-02T04:55:57.950538+0000 mon.a (mon.0) 2883 : cluster 0 osdmap e510: 8 total, 8 up, 7 in 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:57.999895+0000 mon.c (mon.2) 879 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:58.000300+0000 mon.a (mon.0) 2884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:58.000849+0000 mon.c (mon.2) 880 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-65"}]: dispatch 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:58.001117+0000 mon.a (mon.0) 2885 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-65"}]: dispatch 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:58.033467+0000 mon.c (mon.2) 881 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:58.143982+0000 mon.b (mon.1) 259 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: cluster 2024-04-02T04:55:58.147509+0000 mgr.y (mgr.14538) 482 : cluster 0 pgmap v746: 349 pgs: 1 peering, 348 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:55:58.996 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:58 smithi098 bash[23790]: audit 2024-04-02T04:55:58.149447+0000 mon.a (mon.0) 2886 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-02T04:55:59.010 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:59.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: cluster 2024-04-02T04:55:57.950538+0000 mon.a (mon.0) 2883 : cluster 0 osdmap e510: 8 total, 8 up, 7 in 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:57.999895+0000 mon.c (mon.2) 879 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:58.000300+0000 mon.a (mon.0) 2884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:58.000849+0000 mon.c (mon.2) 880 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-65"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:58.001117+0000 mon.a (mon.0) 2885 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-65"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:58.033467+0000 mon.c (mon.2) 881 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:58.143982+0000 mon.b (mon.1) 259 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: cluster 2024-04-02T04:55:58.147509+0000 mgr.y (mgr.14538) 482 : cluster 0 pgmap v746: 349 pgs: 1 peering, 348 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[21528]: audit 2024-04-02T04:55:58.149447+0000 mon.a (mon.0) 2886 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-02T04:55:59.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: cluster 2024-04-02T04:55:57.950538+0000 mon.a (mon.0) 2883 : cluster 0 osdmap e510: 8 total, 8 up, 7 in 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:57.999895+0000 mon.c (mon.2) 879 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:58.000300+0000 mon.a (mon.0) 2884 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:58.000849+0000 mon.c (mon.2) 880 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-65"}]: dispatch 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:58.001117+0000 mon.a (mon.0) 2885 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-65"}]: dispatch 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:58.033467+0000 mon.c (mon.2) 881 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:58.143982+0000 mon.b (mon.1) 259 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: cluster 2024-04-02T04:55:58.147509+0000 mgr.y (mgr.14538) 482 : cluster 0 pgmap v746: 349 pgs: 1 peering, 348 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:55:59.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:58 smithi069 bash[28904]: audit 2024-04-02T04:55:58.149447+0000 mon.a (mon.0) 2886 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-02T04:55:59.134 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:59.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:59.263 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:55:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:55:59.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:59.388 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:55:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:55:59.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:55:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:55:59.620 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:59.648 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:55:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:55:59.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:59.776 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:55:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:55:59.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:59.905 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:55:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:55:59.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:00.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:00.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:59 smithi098 bash[23790]: audit 2024-04-02T04:55:58.953517+0000 mon.a (mon.0) 2887 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-02T04:56:00.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:59 smithi098 bash[23790]: cluster 2024-04-02T04:55:58.964737+0000 mon.a (mon.0) 2888 : cluster 0 osdmap e511: 8 total, 8 up, 7 in 2024-04-02T04:56:00.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:59 smithi098 bash[23790]: audit 2024-04-02T04:55:59.034491+0000 mon.c (mon.2) 882 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:00.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:55:59 smithi098 bash[23790]: cluster 2024-04-02T04:55:59.927802+0000 mon.a (mon.0) 2889 : cluster 0 osdmap e512: 8 total, 8 up, 7 in 2024-04-02T04:56:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:00.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:00.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:00.269 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[28904]: audit 2024-04-02T04:55:58.953517+0000 mon.a (mon.0) 2887 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[28904]: cluster 2024-04-02T04:55:58.964737+0000 mon.a (mon.0) 2888 : cluster 0 osdmap e511: 8 total, 8 up, 7 in 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[28904]: audit 2024-04-02T04:55:59.034491+0000 mon.c (mon.2) 882 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[28904]: cluster 2024-04-02T04:55:59.927802+0000 mon.a (mon.0) 2889 : cluster 0 osdmap e512: 8 total, 8 up, 7 in 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[21528]: audit 2024-04-02T04:55:58.953517+0000 mon.a (mon.0) 2887 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[21528]: cluster 2024-04-02T04:55:58.964737+0000 mon.a (mon.0) 2888 : cluster 0 osdmap e511: 8 total, 8 up, 7 in 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[21528]: audit 2024-04-02T04:55:59.034491+0000 mon.c (mon.2) 882 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:00.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:55:59 smithi069 bash[21528]: cluster 2024-04-02T04:55:59.927802+0000 mon.a (mon.0) 2889 : cluster 0 osdmap e512: 8 total, 8 up, 7 in 2024-04-02T04:56:00.284 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:00.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:00.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 -- 172.21.15.69:0/662952541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54107260 msgr2=0x7f4a54107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:00.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 --2- 172.21.15.69:0/662952541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54107260 0x7f4a54107640 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4a440099c0 tx=0x7f4a44009cd0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 -- 172.21.15.69:0/662952541 shutdown_connections 2024-04-02T04:56:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 --2- 172.21.15.69:0/662952541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a54101150 0x7f4a5410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 --2- 172.21.15.69:0/662952541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54107260 0x7f4a54107640 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 --2- 172.21.15.69:0/662952541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a54100790 0x7f4a54100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 -- 172.21.15.69:0/662952541 >> 172.21.15.69:0/662952541 conn(0x7f4a54077740 msgr2=0x7f4a540feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 -- 172.21.15.69:0/662952541 shutdown_connections 2024-04-02T04:56:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.469+0000 7f4a597fa700 1 -- 172.21.15.69:0/662952541 wait complete. 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 librados: starting msgr at 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 librados: starting objecter 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 Processor -- start 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- start start 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 librados: setting wanted keys 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 librados: calling monclient init 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 0x7f4a54198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a54101150 0x7f4a54198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a54107260 0x7f4a5419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111b80 con 0x7f4a54101150 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111a00 con 0x7f4a54107260 2024-04-02T04:56:00.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4a54111d00 con 0x7f4a54100790 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 0x7f4a54198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a527fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a54101150 0x7f4a54198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 0x7f4a54198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57788/0 (socket says 172.21.15.69:57788) 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 -- 172.21.15.69:0/3835980797 learned_addr learned my addr 172.21.15.69:0/3835980797 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a537fe700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a54107260 0x7f4a5419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 -- 172.21.15.69:0/3835980797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a54107260 msgr2=0x7f4a5419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a54107260 0x7f4a5419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 -- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a54101150 msgr2=0x7f4a54198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a54101150 0x7f4a54198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 -- 172.21.15.69:0/3835980797 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5419d0e0 con 0x7f4a54100790 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a52ffd700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 0x7f4a54198120 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005910 tx=0x7f4a4400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 10 client.?.objecter ms_handle_connect 0x7f4a54100790 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44027070 con 0x7f4a54100790 2024-04-02T04:56:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4a4400fc90 con 0x7f4a54100790 2024-04-02T04:56:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5419d370 con 0x7f4a54100790 2024-04-02T04:56:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44022c90 con 0x7f4a54100790 2024-04-02T04:56:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 10 client.26624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a541a4c50 con 0x7f4a54100790 2024-04-02T04:56:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 librados: init done 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a54103ab0 con 0x7f4a54100790 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4a4402a070 con 0x7f4a54100790 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a3c06f5e0 0x7f4a3c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(512..512 src has 74..512) ==== 10940+0+0 (secure 0 0 0) 0x7f4a4408fb60 con 0x7f4a54100790 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 10 client.26624.objecter ms_dispatch 0x7f4a54001550 osd_map(512..512 src has 74..512) 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 3 client.26624.objecter handle_osd_map got epochs [512,512] > 0 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a3bfff700 3 client.26624.objecter handle_osd_map decoding full epoch 512 2024-04-02T04:56:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.473+0000 7f4a527fc700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a3c06f5e0 0x7f4a3c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:00.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.477+0000 7f4a3bfff700 20 client.26624.objecter dump_active .. 0 homeless 2024-04-02T04:56:00.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.477+0000 7f4a3bfff700 10 client.26624.objecter ms_handle_connect 0x7f4a3c06f5e0 2024-04-02T04:56:00.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.477+0000 7f4a527fc700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a3c06f5e0 0x7f4a3c071aa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4a540fff90 tx=0x7f4a48009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:00.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.481+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a44013070 con 0x7f4a54100790 2024-04-02T04:56:00.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:00.545 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:00.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.701+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd out", "ids": ["3"]} v 0) -- 0x7f4a54199430 con 0x7f4a54100790 2024-04-02T04:56:00.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:00.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:00.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:00.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.933+0000 7f4a3bfff700 1 -- 172.21.15.69:0/3835980797 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["3"]}]=0 marked out osd.3. v513) ==== 87+0+0 (secure 0 0 0) 0x7f4a4405c3b0 con 0x7f4a54100790 2024-04-02T04:56:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:00.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 10 librados: watch_flush enter 2024-04-02T04:56:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 10 librados: watch_flush exit 2024-04-02T04:56:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 20 client.26624.objecter shutdown clearing up homeless session... 2024-04-02T04:56:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 10 client.26624.objecter successfully canceled tick 2024-04-02T04:56:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a3c06f5e0 msgr2=0x7f4a3c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a3c06f5e0 0x7f4a3c071aa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4a540fff90 tx=0x7f4a48009190 comp rx=0 tx=0).stop 2024-04-02T04:56:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 msgr2=0x7f4a54198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 0x7f4a54198120 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005910 tx=0x7f4a4400fa00 comp rx=0 tx=0).stop 2024-04-02T04:56:00.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 shutdown_connections 2024-04-02T04:56:00.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4a3c06f5e0 0x7f4a3c071aa0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a54107260 0x7f4a5419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54100790 0x7f4a54198120 secure :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005910 tx=0x7f4a4400fa00 comp rx=0 tx=0).stop 2024-04-02T04:56:00.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 --2- 172.21.15.69:0/3835980797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a54101150 0x7f4a54198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:00.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.957+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 >> 172.21.15.69:0/3835980797 conn(0x7f4a54077740 msgr2=0x7f4a54104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:00.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: debug 2024-04-02T04:56:00.705+0000 7f02343d7700 -1 mon.a@0(leader).osd e512 definitely_dead 0 2024-04-02T04:56:00.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.961+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 shutdown_connections 2024-04-02T04:56:00.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.961+0000 7f4a597fa700 1 -- 172.21.15.69:0/3835980797 wait complete. 2024-04-02T04:56:00.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:00.961+0000 7f4a597fa700 1 librados: shutdown 2024-04-02T04:56:00.965 INFO:teuthology.orchestra.run.smithi069.stderr:marked out osd.3. 2024-04-02T04:56:01.039 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:55:59.940741+0000 mon.c (mon.2) 883 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:55:59.969751+0000 mon.a (mon.0) 2890 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:56:00.035647+0000 mon.c (mon.2) 884 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: cluster 2024-04-02T04:56:00.148204+0000 mgr.y (mgr.14538) 483 : cluster 0 pgmap v749: 349 pgs: 32 unknown, 317 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:56:00.707785+0000 mon.c (mon.2) 885 : audit 1 from='client.? 172.21.15.69:0/3835980797' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:56:00.708470+0000 mon.a (mon.0) 2891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: cluster 2024-04-02T04:56:00.708564+0000 mon.a (mon.0) 2892 : cluster 1 Client client.admin marked osd.3 out, while it was still marked up 2024-04-02T04:56:01.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:56:00.926400+0000 mon.a (mon.0) 2893 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:56:00.926581+0000 mon.a (mon.0) 2894 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: cluster 2024-04-02T04:56:00.932924+0000 mon.a (mon.0) 2895 : cluster 0 osdmap e513: 8 total, 8 up, 6 in 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[21528]: audit 2024-04-02T04:56:00.944621+0000 mon.c (mon.2) 886 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:55:59.940741+0000 mon.c (mon.2) 883 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:55:59.969751+0000 mon.a (mon.0) 2890 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:56:00.035647+0000 mon.c (mon.2) 884 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: cluster 2024-04-02T04:56:00.148204+0000 mgr.y (mgr.14538) 483 : cluster 0 pgmap v749: 349 pgs: 32 unknown, 317 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:56:00.707785+0000 mon.c (mon.2) 885 : audit 1 from='client.? 172.21.15.69:0/3835980797' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:56:00.708470+0000 mon.a (mon.0) 2891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: cluster 2024-04-02T04:56:00.708564+0000 mon.a (mon.0) 2892 : cluster 1 Client client.admin marked osd.3 out, while it was still marked up 2024-04-02T04:56:01.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:56:00.926400+0000 mon.a (mon.0) 2893 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:01.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:56:00.926581+0000 mon.a (mon.0) 2894 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-02T04:56:01.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: cluster 2024-04-02T04:56:00.932924+0000 mon.a (mon.0) 2895 : cluster 0 osdmap e513: 8 total, 8 up, 6 in 2024-04-02T04:56:01.050 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:00 smithi069 bash[28904]: audit 2024-04-02T04:56:00.944621+0000 mon.c (mon.2) 886 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:01.059 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:01.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:01.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:55:59.940741+0000 mon.c (mon.2) 883 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:55:59.969751+0000 mon.a (mon.0) 2890 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:56:00.035647+0000 mon.c (mon.2) 884 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: cluster 2024-04-02T04:56:00.148204+0000 mgr.y (mgr.14538) 483 : cluster 0 pgmap v749: 349 pgs: 32 unknown, 317 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:56:00.707785+0000 mon.c (mon.2) 885 : audit 1 from='client.? 172.21.15.69:0/3835980797' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:56:00.708470+0000 mon.a (mon.0) 2891 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: cluster 2024-04-02T04:56:00.708564+0000 mon.a (mon.0) 2892 : cluster 1 Client client.admin marked osd.3 out, while it was still marked up 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:56:00.926400+0000 mon.a (mon.0) 2893 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:56:00.926581+0000 mon.a (mon.0) 2894 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: cluster 2024-04-02T04:56:00.932924+0000 mon.a (mon.0) 2895 : cluster 0 osdmap e513: 8 total, 8 up, 6 in 2024-04-02T04:56:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:00 smithi098 bash[23790]: audit 2024-04-02T04:56:00.944621+0000 mon.c (mon.2) 886 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:01.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:01.315 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:01.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:01.448 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:01.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:01.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:01.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:01.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:01.993 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:01.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:02.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:02.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:01 smithi098 bash[23790]: cluster 2024-04-02T04:56:00.961148+0000 mon.a (mon.0) 2896 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:02.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:01 smithi098 bash[23790]: audit 2024-04-02T04:56:01.036564+0000 mon.c (mon.2) 887 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:02.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:01 smithi098 bash[23790]: cluster 2024-04-02T04:56:01.940160+0000 mon.a (mon.0) 2897 : cluster 0 osdmap e514: 8 total, 8 up, 6 in 2024-04-02T04:56:02.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:02.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:02.245 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:02.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:01 smithi069 bash[21528]: cluster 2024-04-02T04:56:00.961148+0000 mon.a (mon.0) 2896 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:01 smithi069 bash[21528]: audit 2024-04-02T04:56:01.036564+0000 mon.c (mon.2) 887 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:01 smithi069 bash[21528]: cluster 2024-04-02T04:56:01.940160+0000 mon.a (mon.0) 2897 : cluster 0 osdmap e514: 8 total, 8 up, 6 in 2024-04-02T04:56:02.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:01 smithi069 bash[28904]: cluster 2024-04-02T04:56:00.961148+0000 mon.a (mon.0) 2896 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:02.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:01 smithi069 bash[28904]: audit 2024-04-02T04:56:01.036564+0000 mon.c (mon.2) 887 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:02.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:01 smithi069 bash[28904]: cluster 2024-04-02T04:56:01.940160+0000 mon.a (mon.0) 2897 : cluster 0 osdmap e514: 8 total, 8 up, 6 in 2024-04-02T04:56:02.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:02.377 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:02.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:02.539 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:02.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:02.637 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:02.743 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:02.769 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:02.857 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:56:02.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:02.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:02.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: Running main() from gmock_main.cc 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] Global test environment set-up. 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (412670 ms) 2024-04-02T04:56:02.955 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] 1 test from neocls_handler_error (412670 ms total) 2024-04-02T04:56:02.956 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: 2024-04-02T04:56:02.956 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] Global test environment tear-down 2024-04-02T04:56:02.956 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [==========] 1 test from 1 test suite ran. (412670 ms total) 2024-04-02T04:56:02.956 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [ PASSED ] 1 test. 2024-04-02T04:56:02.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[28904]: audit 2024-04-02T04:56:02.037444+0000 mon.c (mon.2) 888 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[28904]: audit 2024-04-02T04:56:02.145062+0000 mon.b (mon.1) 260 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[28904]: cluster 2024-04-02T04:56:02.148813+0000 mgr.y (mgr.14538) 484 : cluster 0 pgmap v752: 348 pgs: 32 unknown, 316 active+clean; 8.4 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 31 B/s, 0 objects/s recovering 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[28904]: audit 2024-04-02T04:56:02.150604+0000 mon.a (mon.0) 2898 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[28904]: audit 2024-04-02T04:56:02.933052+0000 mon.a (mon.0) 2899 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[28904]: cluster 2024-04-02T04:56:02.941146+0000 mon.a (mon.0) 2900 : cluster 0 osdmap e515: 8 total, 8 up, 6 in 2024-04-02T04:56:03.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[21528]: audit 2024-04-02T04:56:02.037444+0000 mon.c (mon.2) 888 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[21528]: audit 2024-04-02T04:56:02.145062+0000 mon.b (mon.1) 260 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-02T04:56:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[21528]: cluster 2024-04-02T04:56:02.148813+0000 mgr.y (mgr.14538) 484 : cluster 0 pgmap v752: 348 pgs: 32 unknown, 316 active+clean; 8.4 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 31 B/s, 0 objects/s recovering 2024-04-02T04:56:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[21528]: audit 2024-04-02T04:56:02.150604+0000 mon.a (mon.0) 2898 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-02T04:56:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[21528]: audit 2024-04-02T04:56:02.933052+0000 mon.a (mon.0) 2899 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-02T04:56:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:02 smithi069 bash[21528]: cluster 2024-04-02T04:56:02.941146+0000 mon.a (mon.0) 2900 : cluster 0 osdmap e515: 8 total, 8 up, 6 in 2024-04-02T04:56:03.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:03.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:02 smithi098 bash[23790]: audit 2024-04-02T04:56:02.037444+0000 mon.c (mon.2) 888 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:03.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:02 smithi098 bash[23790]: audit 2024-04-02T04:56:02.145062+0000 mon.b (mon.1) 260 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-02T04:56:03.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:02 smithi098 bash[23790]: cluster 2024-04-02T04:56:02.148813+0000 mgr.y (mgr.14538) 484 : cluster 0 pgmap v752: 348 pgs: 32 unknown, 316 active+clean; 8.4 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 31 B/s, 0 objects/s recovering 2024-04-02T04:56:03.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:02 smithi098 bash[23790]: audit 2024-04-02T04:56:02.150604+0000 mon.a (mon.0) 2898 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-02T04:56:03.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:02 smithi098 bash[23790]: audit 2024-04-02T04:56:02.933052+0000 mon.a (mon.0) 2899 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-02T04:56:03.131 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:02 smithi098 bash[23790]: cluster 2024-04-02T04:56:02.941146+0000 mon.a (mon.0) 2900 : cluster 0 osdmap e515: 8 total, 8 up, 6 in 2024-04-02T04:56:03.145 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:03.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:03.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:03.276 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:03.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:03.409 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:56:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:56:03.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:03.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:03.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:03.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:03.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:03.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:03.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:04.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:04.056 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:04.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:04.183 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:04.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:04.310 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:04.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:04.441 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: cluster 2024-04-02T04:56:02.945994+0000 osd.0 (osd.0) 261 : cluster 0 13.3 starting backfill to osd.7 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: cluster 2024-04-02T04:56:02.949004+0000 osd.0 (osd.0) 262 : cluster 0 13.b starting backfill to osd.7 from (0'0,0'0] MAX to 478'14 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: cluster 2024-04-02T04:56:02.955665+0000 osd.4 (osd.4) 271 : cluster 0 13.6 starting backfill to osd.0 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.038511+0000 mon.c (mon.2) 889 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.051054+0000 mon.c (mon.2) 890 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.051373+0000 mon.a (mon.0) 2901 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.051958+0000 mon.c (mon.2) 891 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-67"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.052237+0000 mon.a (mon.0) 2902 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-67"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.158813+0000 mon.b (mon.1) 261 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: audit 2024-04-02T04:56:03.337027+0000 mgr.y (mgr.14538) 485 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:04.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[21528]: cluster 2024-04-02T04:56:03.939388+0000 mon.a (mon.0) 2903 : cluster 0 osdmap e516: 8 total, 8 up, 6 in 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: cluster 2024-04-02T04:56:02.945994+0000 osd.0 (osd.0) 261 : cluster 0 13.3 starting backfill to osd.7 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: cluster 2024-04-02T04:56:02.949004+0000 osd.0 (osd.0) 262 : cluster 0 13.b starting backfill to osd.7 from (0'0,0'0] MAX to 478'14 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: cluster 2024-04-02T04:56:02.955665+0000 osd.4 (osd.4) 271 : cluster 0 13.6 starting backfill to osd.0 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.038511+0000 mon.c (mon.2) 889 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.051054+0000 mon.c (mon.2) 890 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.051373+0000 mon.a (mon.0) 2901 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.051958+0000 mon.c (mon.2) 891 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-67"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.052237+0000 mon.a (mon.0) 2902 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-67"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.158813+0000 mon.b (mon.1) 261 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: audit 2024-04-02T04:56:03.337027+0000 mgr.y (mgr.14538) 485 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:04 smithi069 bash[28904]: cluster 2024-04-02T04:56:03.939388+0000 mon.a (mon.0) 2903 : cluster 0 osdmap e516: 8 total, 8 up, 6 in 2024-04-02T04:56:04.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:04.573 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:04.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:04.685 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[51598]: logger=cleanup t=2024-04-02T04:56:04.498039616Z level=info msg="Completed cleanup jobs" duration=332.230494ms 2024-04-02T04:56:04.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: cluster 2024-04-02T04:56:02.945994+0000 osd.0 (osd.0) 261 : cluster 0 13.3 starting backfill to osd.7 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:04.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: cluster 2024-04-02T04:56:02.949004+0000 osd.0 (osd.0) 262 : cluster 0 13.b starting backfill to osd.7 from (0'0,0'0] MAX to 478'14 2024-04-02T04:56:04.685 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: cluster 2024-04-02T04:56:02.955665+0000 osd.4 (osd.4) 271 : cluster 0 13.6 starting backfill to osd.0 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.038511+0000 mon.c (mon.2) 889 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.051054+0000 mon.c (mon.2) 890 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.051373+0000 mon.a (mon.0) 2901 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.051958+0000 mon.c (mon.2) 891 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-67"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.052237+0000 mon.a (mon.0) 2902 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-67"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.158813+0000 mon.b (mon.1) 261 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: audit 2024-04-02T04:56:03.337027+0000 mgr.y (mgr.14538) 485 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:04.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:04 smithi098 bash[23790]: cluster 2024-04-02T04:56:03.939388+0000 mon.a (mon.0) 2903 : cluster 0 osdmap e516: 8 total, 8 up, 6 in 2024-04-02T04:56:04.701 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:04.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:04.824 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:04.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:04.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:05.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:05.097 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:05.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:05.274 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:05.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:05.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:05.472 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: cluster 2024-04-02T04:56:03.939754+0000 osd.4 (osd.4) 272 : cluster 0 13.d starting backfill to osd.6 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: audit 2024-04-02T04:56:04.039388+0000 mon.c (mon.2) 892 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: cluster 2024-04-02T04:56:04.150113+0000 mgr.y (mgr.14538) 486 : cluster 0 pgmap v755: 284 pgs: 2 peering, 2 activating+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 3 remapped+peering, 274 active+clean; 8.1 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s wr, 4 op/s; 46/1224 objects degraded (3.758%); 4/1224 objects misplaced (0.327%); 120 KiB/s, 59 objects/s recovering 2024-04-02T04:56:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: cluster 2024-04-02T04:56:04.937575+0000 mon.a (mon.0) 2904 : cluster 0 osdmap e517: 8 total, 8 up, 6 in 2024-04-02T04:56:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: audit 2024-04-02T04:56:04.940618+0000 mon.c (mon.2) 893 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: audit 2024-04-02T04:56:04.943059+0000 mon.a (mon.0) 2905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:05.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:05 smithi098 bash[23790]: audit 2024-04-02T04:56:05.040438+0000 mon.c (mon.2) 894 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:05.488 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:05.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: cluster 2024-04-02T04:56:03.939754+0000 osd.4 (osd.4) 272 : cluster 0 13.d starting backfill to osd.6 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: audit 2024-04-02T04:56:04.039388+0000 mon.c (mon.2) 892 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: cluster 2024-04-02T04:56:04.150113+0000 mgr.y (mgr.14538) 486 : cluster 0 pgmap v755: 284 pgs: 2 peering, 2 activating+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 3 remapped+peering, 274 active+clean; 8.1 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s wr, 4 op/s; 46/1224 objects degraded (3.758%); 4/1224 objects misplaced (0.327%); 120 KiB/s, 59 objects/s recovering 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: cluster 2024-04-02T04:56:04.937575+0000 mon.a (mon.0) 2904 : cluster 0 osdmap e517: 8 total, 8 up, 6 in 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: audit 2024-04-02T04:56:04.940618+0000 mon.c (mon.2) 893 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: audit 2024-04-02T04:56:04.943059+0000 mon.a (mon.0) 2905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[28904]: audit 2024-04-02T04:56:05.040438+0000 mon.c (mon.2) 894 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: cluster 2024-04-02T04:56:03.939754+0000 osd.4 (osd.4) 272 : cluster 0 13.d starting backfill to osd.6 from (0'0,0'0] MAX to 478'11 2024-04-02T04:56:05.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: audit 2024-04-02T04:56:04.039388+0000 mon.c (mon.2) 892 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:05.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: cluster 2024-04-02T04:56:04.150113+0000 mgr.y (mgr.14538) 486 : cluster 0 pgmap v755: 284 pgs: 2 peering, 2 activating+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 3 remapped+peering, 274 active+clean; 8.1 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s wr, 4 op/s; 46/1224 objects degraded (3.758%); 4/1224 objects misplaced (0.327%); 120 KiB/s, 59 objects/s recovering 2024-04-02T04:56:05.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: cluster 2024-04-02T04:56:04.937575+0000 mon.a (mon.0) 2904 : cluster 0 osdmap e517: 8 total, 8 up, 6 in 2024-04-02T04:56:05.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: audit 2024-04-02T04:56:04.940618+0000 mon.c (mon.2) 893 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:05.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: audit 2024-04-02T04:56:04.943059+0000 mon.a (mon.0) 2905 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:05.604 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:05 smithi069 bash[21528]: audit 2024-04-02T04:56:05.040438+0000 mon.c (mon.2) 894 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:05.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:05.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:05.746 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:05.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:05.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:05.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:05.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:06.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:06.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:06.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:06.265 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:06.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:06.395 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:06.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:06.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:06.657 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:06.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:06.781 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:06.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:06.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 -- 172.21.15.69:0/2333446293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 msgr2=0x7f91cc101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 --2- 172.21.15.69:0/2333446293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 0x7f91cc101220 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009920 tx=0x7f91bc009c30 comp rx=0 tx=0).stop 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 -- 172.21.15.69:0/2333446293 shutdown_connections 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 --2- 172.21.15.69:0/2333446293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc107610 0x7f91cc10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 --2- 172.21.15.69:0/2333446293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 0x7f91cc101220 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 --2- 172.21.15.69:0/2333446293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc1017f0 0x7f91cc106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 -- 172.21.15.69:0/2333446293 >> 172.21.15.69:0/2333446293 conn(0x7f91cc0764a0 msgr2=0x7f91cc0768b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 -- 172.21.15.69:0/2333446293 shutdown_connections 2024-04-02T04:56:06.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 -- 172.21.15.69:0/2333446293 wait complete. 2024-04-02T04:56:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 librados: starting msgr at 2024-04-02T04:56:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 librados: starting objecter 2024-04-02T04:56:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.893+0000 7f91d0dea700 1 Processor -- start 2024-04-02T04:56:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 -- start start 2024-04-02T04:56:06.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 librados: setting wanted keys 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 librados: calling monclient init 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 0x7f91cc101dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc1017f0 0x7f91cc102300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc107610 0x7f91cc102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc111ad0 con 0x7f91cc1017f0 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc111950 con 0x7f91cc107610 2024-04-02T04:56:06.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f91cc111c50 con 0x7f91cc100e40 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc1017f0 0x7f91cc102300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc1017f0 0x7f91cc102300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57682/0 (socket says 172.21.15.69:57682) 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 -- 172.21.15.69:0/948987024 learned_addr learned my addr 172.21.15.69:0/948987024 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc107610 0x7f91cc102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 0x7f91cc101dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 msgr2=0x7f91cc101dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 0x7f91cc101dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc107610 msgr2=0x7f91cc102840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc107610 0x7f91cc102840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91cc1a66a0 con 0x7f91cc1017f0 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91cc107610 0x7f91cc102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:06.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91cc100e40 0x7f91cc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:56:06.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc1017f0 0x7f91cc102300 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f91c000c8f0 tx=0x7f91c000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:06.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91bb7fe700 10 client.?.objecter ms_handle_connect 0x7f91cc1017f0 2024-04-02T04:56:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91bb7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0007940 con 0x7f91cc1017f0 2024-04-02T04:56:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f91c000f450 con 0x7f91cc1017f0 2024-04-02T04:56:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91cc1a6990 con 0x7f91cc1017f0 2024-04-02T04:56:06.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 10 client.16536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0018660 con 0x7f91cc1017f0 2024-04-02T04:56:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91cc1a6f20 con 0x7f91cc1017f0 2024-04-02T04:56:06.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.897+0000 7f91d0dea700 1 librados: init done 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f91c00188c0 con 0x7f91cc1017f0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91d0dea700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f91cc06a390 con 0x7f91cc1017f0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f91b406f4b0 0x7f91b4071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(518..518 src has 74..518) ==== 10551+0+0 (secure 0 0 0) 0x7f91c0091170 con 0x7f91cc1017f0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 osd_map(518..518 src has 74..518) 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 3 client.16536.objecter handle_osd_map got epochs [518,518] > 0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 3 client.16536.objecter handle_osd_map decoding full epoch 518 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f91b406f4b0 0x7f91b4071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 20 client.16536.objecter dump_active .. 0 homeless 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v518) ==== 54+0+16 (secure 0 0 0) 0x7f91c0014030 con 0x7f91cc1017f0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f91b406f4b0 0x7f91b4071970 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009680 tx=0x7f91bc005b40 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b406f4b0 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 10 client.16536.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:06.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 1 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f91b0002ff0 0x7f91b00054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 1 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 0 1 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 4 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 10 client.16536.objecter _send_command 1 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91b0008530 con 0x7f91b0002ff0 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f91b0002ff0 0x7f91b00054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b0002ff0 2024-04-02T04:56:06.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f91b0002ff0 0x7f91b00054b0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b0008530 con 0x7f91b0002ff0 2024-04-02T04:56:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 1: 0 ) 2024-04-02T04:56:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 1 = system:0 2024-04-02T04:56:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 4 2024-04-02T04:56:06.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.901+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 0 1 2024-04-02T04:56:06.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:06.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 10 client.16536.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 2 2024-04-02T04:56:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 2 2024-04-02T04:56:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b00028e0 osd=0 3 2024-04-02T04:56:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 0 2 2024-04-02T04:56:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 4 2024-04-02T04:56:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 10 client.16536.objecter _send_command 2 2024-04-02T04:56:06.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b0001240 con 0x7f91b0002ff0 2024-04-02T04:56:06.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b0001240 con 0x7f91b0002ff0 2024-04-02T04:56:06.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:06.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:06.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 4 2024-04-02T04:56:06.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 0 2 2024-04-02T04:56:06.923 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 10 client.16536.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 3 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f91b0006cb0 0x7f91b0007160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 3 2024-04-02T04:56:06.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 1 3 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 4 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 10 client.16536.objecter _send_command 3 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f91b00076a0 con 0x7f91b0006cb0 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.921+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f91b0006cb0 0x7f91b0007160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b0006cb0 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f91b0006cb0 0x7f91b0007160 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b00076a0 con 0x7f91b0006cb0 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 3: 0 ) 2024-04-02T04:56:06.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 3 = system:0 2024-04-02T04:56:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 4 2024-04-02T04:56:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.925+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 1 3 2024-04-02T04:56:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 10 client.16536.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 4 2024-04-02T04:56:06.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 4 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0001f80 osd=1 3 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 1 4 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 4 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 10 client.16536.objecter _send_command 4 2024-04-02T04:56:06.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b00080f0 con 0x7f91b0006cb0 2024-04-02T04:56:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b00080f0 con 0x7f91b0006cb0 2024-04-02T04:56:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 4 2024-04-02T04:56:06.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 1 4 2024-04-02T04:56:06.956 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:56:06.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 10 client.16536.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:06.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 5 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f91b000acf0 0x7f91b000d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 5 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 2 5 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 4 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 10 client.16536.objecter _send_command 5 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f91b0010020 con 0x7f91b000acf0 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.953+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f91b000acf0 0x7f91b000d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b000acf0 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f91b000acf0 0x7f91b000d0e0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:06.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b0010020 con 0x7f91b000acf0 2024-04-02T04:56:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 5: 0 ) 2024-04-02T04:56:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 5 = system:0 2024-04-02T04:56:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 4 2024-04-02T04:56:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.957+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 2 5 2024-04-02T04:56:06.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 10 client.16536.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 6 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 6 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0001cd0 osd=2 3 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 2 6 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 4 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 10 client.16536.objecter _send_command 6 2024-04-02T04:56:06.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.973+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b0007c00 con 0x7f91b000acf0 2024-04-02T04:56:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b0007c00 con 0x7f91b000acf0 2024-04-02T04:56:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 4 2024-04-02T04:56:06.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 2 6 2024-04-02T04:56:06.978 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:56:06.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 10 client.16536.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 7 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f91b000e7a0 0x7f91b000ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 7 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 3 7 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 4 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 10 client.16536.objecter _send_command 7 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f91b000f170 con 0x7f91b000e7a0 2024-04-02T04:56:06.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f91b000e7a0 0x7f91b000ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b000e7a0 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f91b000e7a0 0x7f91b000ec30 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b000f170 con 0x7f91b000e7a0 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 7: 0 ) 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 7 = system:0 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 4 2024-04-02T04:56:06.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.977+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 3 7 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 10 client.16536.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 8 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 8 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0007e90 osd=3 3 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 3 8 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 4 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 10 client.16536.objecter _send_command 8 2024-04-02T04:56:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:06.997+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b000fb80 con 0x7f91b000e7a0 2024-04-02T04:56:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b000fb80 con 0x7f91b000e7a0 2024-04-02T04:56:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 4 2024-04-02T04:56:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 3 8 2024-04-02T04:56:07.002 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 10 client.16536.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 9 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f91b00121e0 0x7f91b00145d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 9 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 4 9 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 4 2024-04-02T04:56:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 10 client.16536.objecter _send_command 9 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f91b0017430 con 0x7f91b00121e0 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f91b00121e0 0x7f91b00145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b00121e0 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.001+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f91b00121e0 0x7f91b00145d0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.005+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b0017430 con 0x7f91b00121e0 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.005+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 9: 0 ) 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.005+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 9 = system:0 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.005+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 4 2024-04-02T04:56:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.005+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 4 9 2024-04-02T04:56:07.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:05.928024+0000 mon.a (mon.0) 2906 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: cluster 2024-04-02T04:56:05.932458+0000 mon.a (mon.0) 2907 : cluster 0 osdmap e518: 8 total, 8 up, 6 in 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:05.934286+0000 mon.c (mon.2) 895 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.041926+0000 mon.c (mon.2) 896 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.042539+0000 mon.c (mon.2) 897 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.043098+0000 mon.a (mon.0) 2908 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.044186+0000 mon.c (mon.2) 898 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-69"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.044824+0000 mon.a (mon.0) 2909 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-69"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.147533+0000 mon.b (mon.1) 262 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: cluster 2024-04-02T04:56:06.151031+0000 mgr.y (mgr.14538) 487 : cluster 0 pgmap v758: 315 pgs: 32 unknown, 2 peering, 2 activating+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 2 remapped+peering, 274 active+clean; 8.1 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s wr, 0 op/s; 46/1218 objects degraded (3.777%); 4/1218 objects misplaced (0.328%); 120 KiB/s, 57 objects/s recovering 2024-04-02T04:56:07.018 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.154634+0000 mon.a (mon.0) 2910 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: cluster 2024-04-02T04:56:06.464394+0000 osd.1 (osd.1) 279 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: cluster 2024-04-02T04:56:06.464723+0000 osd.1 (osd.1) 280 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: cluster 2024-04-02T04:56:06.514776+0000 osd.3 (osd.3) 315 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: cluster 2024-04-02T04:56:06.515098+0000 osd.3 (osd.3) 316 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[28904]: audit 2024-04-02T04:56:06.904817+0000 mon.a (mon.0) 2911 : audit 0 from='client.? 172.21.15.69:0/948987024' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:05.928024+0000 mon.a (mon.0) 2906 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: cluster 2024-04-02T04:56:05.932458+0000 mon.a (mon.0) 2907 : cluster 0 osdmap e518: 8 total, 8 up, 6 in 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:05.934286+0000 mon.c (mon.2) 895 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.041926+0000 mon.c (mon.2) 896 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.042539+0000 mon.c (mon.2) 897 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.043098+0000 mon.a (mon.0) 2908 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.044186+0000 mon.c (mon.2) 898 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-69"}]: dispatch 2024-04-02T04:56:07.019 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.044824+0000 mon.a (mon.0) 2909 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-69"}]: dispatch 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.147533+0000 mon.b (mon.1) 262 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: cluster 2024-04-02T04:56:06.151031+0000 mgr.y (mgr.14538) 487 : cluster 0 pgmap v758: 315 pgs: 32 unknown, 2 peering, 2 activating+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 2 remapped+peering, 274 active+clean; 8.1 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s wr, 0 op/s; 46/1218 objects degraded (3.777%); 4/1218 objects misplaced (0.328%); 120 KiB/s, 57 objects/s recovering 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.154634+0000 mon.a (mon.0) 2910 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: cluster 2024-04-02T04:56:06.464394+0000 osd.1 (osd.1) 279 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: cluster 2024-04-02T04:56:06.464723+0000 osd.1 (osd.1) 280 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: cluster 2024-04-02T04:56:06.514776+0000 osd.3 (osd.3) 315 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: cluster 2024-04-02T04:56:06.515098+0000 osd.3 (osd.3) 316 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:07.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:06 smithi069 bash[21528]: audit 2024-04-02T04:56:06.904817+0000 mon.a (mon.0) 2911 : audit 0 from='client.? 172.21.15.69:0/948987024' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:07.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 10 client.16536.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:07.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 10 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 10 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b000fd90 osd=4 3 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 4 10 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 4 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 10 client.16536.objecter _send_command 10 2024-04-02T04:56:07.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.021+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b0014b10 con 0x7f91b00121e0 2024-04-02T04:56:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b0014b10 con 0x7f91b00121e0 2024-04-02T04:56:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 4 2024-04-02T04:56:07.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 4 10 2024-04-02T04:56:07.027 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 10 client.16536.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 11 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f91b0015190 0x7f91b0017580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 11 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 5 11 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 4 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 10 client.16536.objecter _send_command 11 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f91b001a5f0 con 0x7f91b0015190 2024-04-02T04:56:07.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f91b0015190 0x7f91b0017580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b0015190 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.025+0000 7f91c9d9b700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f91b0015190 0x7f91b0017580 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.029+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b001a5f0 con 0x7f91b0015190 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.029+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 11: 0 ) 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.029+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 11 = system:0 2024-04-02T04:56:07.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.029+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 4 2024-04-02T04:56:07.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.029+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 5 11 2024-04-02T04:56:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 10 client.16536.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 12 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 12 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0014b10 osd=5 3 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 5 12 2024-04-02T04:56:07.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 4 2024-04-02T04:56:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 10 client.16536.objecter _send_command 12 2024-04-02T04:56:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.045+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b0019380 con 0x7f91b0015190 2024-04-02T04:56:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b0019380 con 0x7f91b0015190 2024-04-02T04:56:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 4 2024-04-02T04:56:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 5 12 2024-04-02T04:56:07.051 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 10 client.16536.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 13 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f91b0019aa0 0x7f91b0019f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 13 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 6 13 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 4 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 10 client.16536.objecter _send_command 13 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.049+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f91b001da50 con 0x7f91b0019aa0 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f91b0019aa0 0x7f91b0019f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b0019aa0 2024-04-02T04:56:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91cad9d700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f91b0019aa0 0x7f91b0019f20 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b001da50 con 0x7f91b0019aa0 2024-04-02T04:56:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 13: 0 ) 2024-04-02T04:56:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 13 = system:0 2024-04-02T04:56:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 4 2024-04-02T04:56:07.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.053+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 6 13 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 10 client.16536.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 14 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 14 2024-04-02T04:56:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b0019380 osd=6 3 2024-04-02T04:56:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 6 14 2024-04-02T04:56:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 4 2024-04-02T04:56:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 10 client.16536.objecter _send_command 14 2024-04-02T04:56:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.069+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b001da10 con 0x7f91b0019aa0 2024-04-02T04:56:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b001da10 con 0x7f91b0019aa0 2024-04-02T04:56:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 4 2024-04-02T04:56:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 6 14 2024-04-02T04:56:07.076 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:56:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 10 client.16536.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 15 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f91b001cdb0 0x7f91b001d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 15 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 7 15 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 4 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 10 client.16536.objecter _send_command 15 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.073+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f91b0020dc0 con 0x7f91b001cdb0 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f91b001cdb0 0x7f91b001d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91ca59c700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f91b001cdb0 0x7f91b001d210 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91bb7fe700 10 client.16536.objecter ms_handle_connect 0x7f91b001cdb0 2024-04-02T04:56:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b0020dc0 con 0x7f91b001cdb0 2024-04-02T04:56:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 15: 0 ) 2024-04-02T04:56:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 15 = system:0 2024-04-02T04:56:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 4 2024-04-02T04:56:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.077+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 7 15 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 10 client.16536.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign -1 16 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 20 client.16536.objecter _get_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_remove -1 16 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 20 client.16536.objecter get_session s=0x7f91b001da10 osd=7 3 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 15 client.16536.objecter _session_command_op_assign 7 16 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 4 2024-04-02T04:56:07.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 10 client.16536.objecter _send_command 16 2024-04-02T04:56:07.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.097+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f91b0017bb0 con 0x7f91b001cdb0 2024-04-02T04:56:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91bb7fe700 1 -- 172.21.15.69:0/948987024 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f91b0017bb0 con 0x7f91b001cdb0 2024-04-02T04:56:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91bb7fe700 10 client.16536.objecter ms_dispatch 0x7f91cc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91bb7fe700 10 client.16536.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91bb7fe700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 4 2024-04-02T04:56:07.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91bb7fe700 15 client.16536.objecter _session_command_op_remove 7 16 2024-04-02T04:56:07.100 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:56:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 librados: watch_flush enter 2024-04-02T04:56:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 librados: watch_flush exit 2024-04-02T04:56:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.0 2024-04-02T04:56:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f91b0002ff0 msgr2=0x7f91b00054b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f91b0002ff0 0x7f91b00054b0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b00028e0 osd=0 1 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.1 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f91b0006cb0 msgr2=0x7f91b0007160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f91b0006cb0 0x7f91b0007160 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001f80 osd=1 1 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.2 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f91b000acf0 msgr2=0x7f91b000d0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f91b000acf0 0x7f91b000d0e0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0001cd0 osd=2 1 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.3 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f91b000e7a0 msgr2=0x7f91b000ec30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f91b000e7a0 0x7f91b000ec30 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0007e90 osd=3 1 2024-04-02T04:56:07.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.4 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f91b00121e0 msgr2=0x7f91b00145d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f91b00121e0 0x7f91b00145d0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b000fd90 osd=4 1 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.5 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f91b0015190 msgr2=0x7f91b0017580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f91b0015190 0x7f91b0017580 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0014b10 osd=5 1 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.6 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f91b0019aa0 msgr2=0x7f91b0019f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f91b0019aa0 0x7f91b0019f20 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b0019380 osd=6 1 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter close_session for osd.7 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f91b001cdb0 msgr2=0x7f91b001d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f91b001cdb0 0x7f91b001d210 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter put_session s=0x7f91b001da10 osd=7 1 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 20 client.16536.objecter shutdown clearing up homeless session... 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 10 client.16536.objecter successfully canceled tick 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91cad9d700 1 -- 172.21.15.69:0/948987024 reap_dead start 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91cad9d700 1 -- 172.21.15.69:0/948987024 reap_dead start 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f91b406f4b0 msgr2=0x7f91b4071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f91b406f4b0 0x7f91b4071970 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009680 tx=0x7f91bc005b40 comp rx=0 tx=0).stop 2024-04-02T04:56:07.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc1017f0 msgr2=0x7f91cc102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 --2- 172.21.15.69:0/948987024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91cc1017f0 0x7f91cc102300 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f91c000c8f0 tx=0x7f91c000cc00 comp rx=0 tx=0).stop 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91cad9d700 1 -- 172.21.15.69:0/948987024 reap_dead start 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91cad9d700 1 -- 172.21.15.69:0/948987024 reap_dead start 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91cad9d700 1 -- 172.21.15.69:0/948987024 reap_dead start 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 shutdown_connections 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 >> 172.21.15.69:0/948987024 conn(0x7f91cc0764a0 msgr2=0x7f91cc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 shutdown_connections 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 -- 172.21.15.69:0/948987024 wait complete. 2024-04-02T04:56:07.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:07.101+0000 7f91b97fa700 1 librados: shutdown 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:56:07.107 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:56:07.116 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7] out_osds: [5, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:56:07.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:56:07.116 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-02T04:56:07.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-02T04:56:07.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:07.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:05.928024+0000 mon.a (mon.0) 2906 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: cluster 2024-04-02T04:56:05.932458+0000 mon.a (mon.0) 2907 : cluster 0 osdmap e518: 8 total, 8 up, 6 in 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:05.934286+0000 mon.c (mon.2) 895 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.041926+0000 mon.c (mon.2) 896 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.042539+0000 mon.c (mon.2) 897 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.043098+0000 mon.a (mon.0) 2908 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.044186+0000 mon.c (mon.2) 898 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-69"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.044824+0000 mon.a (mon.0) 2909 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-69"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.147533+0000 mon.b (mon.1) 262 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: cluster 2024-04-02T04:56:06.151031+0000 mgr.y (mgr.14538) 487 : cluster 0 pgmap v758: 315 pgs: 32 unknown, 2 peering, 2 activating+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 2 remapped+peering, 274 active+clean; 8.1 MiB data, 436 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s wr, 0 op/s; 46/1218 objects degraded (3.777%); 4/1218 objects misplaced (0.328%); 120 KiB/s, 57 objects/s recovering 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.154634+0000 mon.a (mon.0) 2910 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: cluster 2024-04-02T04:56:06.464394+0000 osd.1 (osd.1) 279 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:07.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: cluster 2024-04-02T04:56:06.464723+0000 osd.1 (osd.1) 280 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:07.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: cluster 2024-04-02T04:56:06.514776+0000 osd.3 (osd.3) 315 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:07.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: cluster 2024-04-02T04:56:06.515098+0000 osd.3 (osd.3) 316 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:07.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:06 smithi098 bash[23790]: audit 2024-04-02T04:56:06.904817+0000 mon.a (mon.0) 2911 : audit 0 from='client.? 172.21.15.69:0/948987024' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:07.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:07.280 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:07.320 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:56:07.321 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:56:07.321 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T04:56:07.321 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:56:07.382 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:07.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:07.531 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:07.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:07.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:07.790 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:07.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:07.921 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:07.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:08.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:07 smithi069 bash[21528]: audit 2024-04-02T04:56:06.937536+0000 mon.a (mon.0) 2912 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-02T04:56:08.054 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:07 smithi069 bash[21528]: cluster 2024-04-02T04:56:06.944204+0000 mon.a (mon.0) 2913 : cluster 0 osdmap e519: 8 total, 8 up, 6 in 2024-04-02T04:56:08.054 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:07 smithi069 bash[21528]: audit 2024-04-02T04:56:07.043035+0000 mon.c (mon.2) 899 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:08.055 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:07 smithi069 bash[28904]: audit 2024-04-02T04:56:06.937536+0000 mon.a (mon.0) 2912 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-02T04:56:08.055 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:07 smithi069 bash[28904]: cluster 2024-04-02T04:56:06.944204+0000 mon.a (mon.0) 2913 : cluster 0 osdmap e519: 8 total, 8 up, 6 in 2024-04-02T04:56:08.055 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:07 smithi069 bash[28904]: audit 2024-04-02T04:56:07.043035+0000 mon.c (mon.2) 899 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:08.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:08.081 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:08.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-02T04:56:08.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:08.211 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:08.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:07 smithi098 bash[23790]: audit 2024-04-02T04:56:06.937536+0000 mon.a (mon.0) 2912 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-02T04:56:08.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:07 smithi098 bash[23790]: cluster 2024-04-02T04:56:06.944204+0000 mon.a (mon.0) 2913 : cluster 0 osdmap e519: 8 total, 8 up, 6 in 2024-04-02T04:56:08.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:07 smithi098 bash[23790]: audit 2024-04-02T04:56:07.043035+0000 mon.c (mon.2) 899 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:08.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:08.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:08.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:08.598 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:08.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:08.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:08.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:08.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:08.970 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:56:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:56:09.003 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:09.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:09.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: cluster 2024-04-02T04:56:07.956769+0000 mon.a (mon.0) 2914 : cluster 0 osdmap e520: 8 total, 8 up, 6 in 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: audit 2024-04-02T04:56:07.958720+0000 mon.c (mon.2) 900 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: audit 2024-04-02T04:56:07.959260+0000 mon.a (mon.0) 2915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: audit 2024-04-02T04:56:08.044003+0000 mon.c (mon.2) 901 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: cluster 2024-04-02T04:56:08.153263+0000 mgr.y (mgr.14538) 488 : cluster 0 pgmap v761: 315 pgs: 12 creating+peering, 20 unknown, 1 active+remapped+backfill_toofull, 4 peering, 278 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s; 2/1230 objects misplaced (0.163%); 1.0 MiB/s, 13 objects/s recovering 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: cluster 2024-04-02T04:56:08.249608+0000 osd.6 (osd.6) 233 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:09.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:08 smithi098 bash[23790]: cluster 2024-04-02T04:56:08.250030+0000 osd.6 (osd.6) 234 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:09.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:09.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: cluster 2024-04-02T04:56:07.956769+0000 mon.a (mon.0) 2914 : cluster 0 osdmap e520: 8 total, 8 up, 6 in 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: audit 2024-04-02T04:56:07.958720+0000 mon.c (mon.2) 900 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: audit 2024-04-02T04:56:07.959260+0000 mon.a (mon.0) 2915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: audit 2024-04-02T04:56:08.044003+0000 mon.c (mon.2) 901 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: cluster 2024-04-02T04:56:08.153263+0000 mgr.y (mgr.14538) 488 : cluster 0 pgmap v761: 315 pgs: 12 creating+peering, 20 unknown, 1 active+remapped+backfill_toofull, 4 peering, 278 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s; 2/1230 objects misplaced (0.163%); 1.0 MiB/s, 13 objects/s recovering 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: cluster 2024-04-02T04:56:08.249608+0000 osd.6 (osd.6) 233 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:08 smithi069 bash[28904]: cluster 2024-04-02T04:56:08.250030+0000 osd.6 (osd.6) 234 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: cluster 2024-04-02T04:56:07.956769+0000 mon.a (mon.0) 2914 : cluster 0 osdmap e520: 8 total, 8 up, 6 in 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: audit 2024-04-02T04:56:07.958720+0000 mon.c (mon.2) 900 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:09.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: audit 2024-04-02T04:56:07.959260+0000 mon.a (mon.0) 2915 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:09.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: audit 2024-04-02T04:56:08.044003+0000 mon.c (mon.2) 901 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:09.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: cluster 2024-04-02T04:56:08.153263+0000 mgr.y (mgr.14538) 488 : cluster 0 pgmap v761: 315 pgs: 12 creating+peering, 20 unknown, 1 active+remapped+backfill_toofull, 4 peering, 278 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s; 2/1230 objects misplaced (0.163%); 1.0 MiB/s, 13 objects/s recovering 2024-04-02T04:56:09.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: cluster 2024-04-02T04:56:08.249608+0000 osd.6 (osd.6) 233 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:09.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:09 smithi069 bash[21528]: cluster 2024-04-02T04:56:08.250030+0000 osd.6 (osd.6) 234 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:09.262 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:09.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:09.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:09.522 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:09.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:09.652 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:09.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:09.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:09.912 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:09.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:10.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:10.043 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:10.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:10.157 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: cluster 2024-04-02T04:56:08.951405+0000 mon.a (mon.0) 2916 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: cluster 2024-04-02T04:56:08.951505+0000 mon.a (mon.0) 2917 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: audit 2024-04-02T04:56:08.953037+0000 mon.a (mon.0) 2918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: cluster 2024-04-02T04:56:08.962076+0000 mon.a (mon.0) 2919 : cluster 0 osdmap e521: 8 total, 8 up, 6 in 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: audit 2024-04-02T04:56:09.003758+0000 mon.c (mon.2) 902 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: audit 2024-04-02T04:56:09.013746+0000 mon.c (mon.2) 903 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: audit 2024-04-02T04:56:09.014177+0000 mon.a (mon.0) 2920 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:56:10.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:10 smithi098 bash[23790]: audit 2024-04-02T04:56:09.045102+0000 mon.c (mon.2) 904 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:10.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:10.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: cluster 2024-04-02T04:56:08.951405+0000 mon.a (mon.0) 2916 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: cluster 2024-04-02T04:56:08.951505+0000 mon.a (mon.0) 2917 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: audit 2024-04-02T04:56:08.953037+0000 mon.a (mon.0) 2918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: cluster 2024-04-02T04:56:08.962076+0000 mon.a (mon.0) 2919 : cluster 0 osdmap e521: 8 total, 8 up, 6 in 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: audit 2024-04-02T04:56:09.003758+0000 mon.c (mon.2) 902 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: audit 2024-04-02T04:56:09.013746+0000 mon.c (mon.2) 903 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: audit 2024-04-02T04:56:09.014177+0000 mon.a (mon.0) 2920 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[21528]: audit 2024-04-02T04:56:09.045102+0000 mon.c (mon.2) 904 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: cluster 2024-04-02T04:56:08.951405+0000 mon.a (mon.0) 2916 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: cluster 2024-04-02T04:56:08.951505+0000 mon.a (mon.0) 2917 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:10.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: audit 2024-04-02T04:56:08.953037+0000 mon.a (mon.0) 2918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:10.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: cluster 2024-04-02T04:56:08.962076+0000 mon.a (mon.0) 2919 : cluster 0 osdmap e521: 8 total, 8 up, 6 in 2024-04-02T04:56:10.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: audit 2024-04-02T04:56:09.003758+0000 mon.c (mon.2) 902 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:10.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: audit 2024-04-02T04:56:09.013746+0000 mon.c (mon.2) 903 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:56:10.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: audit 2024-04-02T04:56:09.014177+0000 mon.a (mon.0) 2920 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:56:10.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:10 smithi069 bash[28904]: audit 2024-04-02T04:56:09.045102+0000 mon.c (mon.2) 904 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:10.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:10.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:10.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- 172.21.15.69:0/101736547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c108390 msgr2=0x7fb64c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- 172.21.15.69:0/101736547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c108390 0x7fb64c108770 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb648009990 tx=0x7fb648009ca0 comp rx=0 tx=0).stop 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- 172.21.15.69:0/101736547 shutdown_connections 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- 172.21.15.69:0/101736547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 0x7fb64c10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- 172.21.15.69:0/101736547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c108390 0x7fb64c108770 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- 172.21.15.69:0/101736547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c102320 0x7fb64c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- 172.21.15.69:0/101736547 >> 172.21.15.69:0/101736547 conn(0x7fb64c079310 msgr2=0x7fb64c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- 172.21.15.69:0/101736547 shutdown_connections 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- 172.21.15.69:0/101736547 wait complete. 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 librados: starting msgr at 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 librados: starting objecter 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 Processor -- start 2024-04-02T04:56:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- start start 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 librados: setting wanted keys 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 librados: calling monclient init 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 0x7fb64c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 0x7fb64c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c108390 0x7fb64c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c111af0 con 0x7fb64c102320 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c111970 con 0x7fb64c102ce0 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb64c111c70 con 0x7fb64c108390 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 0x7fb64c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 0x7fb64c074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54008/0 (socket says 172.21.15.69:54008) 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 -- 172.21.15.69:0/1635031641 learned_addr learned my addr 172.21.15.69:0/1635031641 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb652cc4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c108390 0x7fb64c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb651cc2700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 0x7fb64c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 -- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c108390 msgr2=0x7fb64c074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c108390 0x7fb64c074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 -- 172.21.15.69:0/1635031641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 msgr2=0x7fb64c074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 0x7fb64c074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 -- 172.21.15.69:0/1635031641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64c1a65d0 con 0x7fb64c102320 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb651cc2700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 0x7fb64c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb652cc4700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c108390 0x7fb64c074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6524c3700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 0x7fb64c074540 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb6480057f0 tx=0x7fb64800bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6437fe700 10 client.?.objecter ms_handle_connect 0x7fb64c102320 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6437fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648026070 con 0x7fb64c102320 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb648021410 con 0x7fb64c102320 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64c1a6860 con 0x7fb64c102320 2024-04-02T04:56:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.457+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648017650 con 0x7fb64c102320 2024-04-02T04:56:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb654727700 10 client.16542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb64c1a6d40 con 0x7fb64c102320 2024-04-02T04:56:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb654727700 1 librados: init done 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb64c04ff00 con 0x7fb64c102320 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb648029070 con 0x7fb64c102320 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb63806f5e0 0x7fb638071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(522..522 src has 74..522) ==== 10551+0+0 (secure 0 0 0) 0x7fb64808f8d0 con 0x7fb64c102320 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 10 client.16542.objecter ms_dispatch 0x7fb64c001550 osd_map(522..522 src has 74..522) 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 3 client.16542.objecter handle_osd_map got epochs [522,522] > 0 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 3 client.16542.objecter handle_osd_map decoding full epoch 522 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb651cc2700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb63806f5e0 0x7fb638071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 20 client.16542.objecter dump_active .. 0 homeless 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb6437fe700 10 client.16542.objecter ms_handle_connect 0x7fb63806f5e0 2024-04-02T04:56:10.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.461+0000 7fb651cc2700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb63806f5e0 0x7fb638071aa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb64c079ad0 tx=0x7fb63c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:10.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.465+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb648013070 con 0x7fb64c102320 2024-04-02T04:56:10.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:10.550 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:10.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:10.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:10.689+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fb64c1a7000 con 0x7fb64c102320 2024-04-02T04:56:10.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:10.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:10.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:10.939 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:10.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:11.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.017+0000 7fb6437fe700 1 -- 172.21.15.69:0/1635031641 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v523) ==== 91+0+0 (secure 0 0 0) 0x7fb64805c340 con 0x7fb64c102320 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 10 librados: watch_flush enter 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 10 librados: watch_flush exit 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 20 client.16542.objecter shutdown clearing up homeless session... 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 10 client.16542.objecter successfully canceled tick 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb63806f5e0 msgr2=0x7fb638071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb63806f5e0 0x7fb638071aa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb64c079ad0 tx=0x7fb63c009260 comp rx=0 tx=0).stop 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 msgr2=0x7fb64c074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 0x7fb64c074540 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb6480057f0 tx=0x7fb64800bfa0 comp rx=0 tx=0).stop 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 shutdown_connections 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb63806f5e0 0x7fb638071aa0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:11.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb64c108390 0x7fb64c074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:11.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb64c102320 0x7fb64c074540 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:11.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 --2- 172.21.15.69:0/1635031641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb64c102ce0 0x7fb64c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:11.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 >> 172.21.15.69:0/1635031641 conn(0x7fb64c079310 msgr2=0x7fb64c1090a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:11.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.025+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 shutdown_connections 2024-04-02T04:56:11.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.029+0000 7fb654727700 1 -- 172.21.15.69:0/1635031641 wait complete. 2024-04-02T04:56:11.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:11.029+0000 7fb654727700 1 librados: shutdown 2024-04-02T04:56:11.029 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:56:11.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:11.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:11.197 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: audit 2024-04-02T04:56:09.997228+0000 mon.a (mon.0) 2921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: cluster 2024-04-02T04:56:10.004111+0000 mon.a (mon.0) 2922 : cluster 0 osdmap e522: 8 total, 8 up, 6 in 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: audit 2024-04-02T04:56:10.046042+0000 mon.c (mon.2) 905 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: audit 2024-04-02T04:56:10.150994+0000 mon.b (mon.1) 263 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: cluster 2024-04-02T04:56:10.154361+0000 mgr.y (mgr.14538) 489 : cluster 0 pgmap v764: 314 pgs: 12 creating+peering, 13 unknown, 1 active+remapped+backfill_toofull, 1 peering, 287 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 2/1230 objects misplaced (0.163%); 1.0 MiB/s, 14 objects/s recovering 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: audit 2024-04-02T04:56:10.158139+0000 mon.a (mon.0) 2923 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:56:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:11 smithi098 bash[23790]: audit 2024-04-02T04:56:10.692916+0000 mon.a (mon.0) 2924 : audit 1 from='client.? 172.21.15.69:0/1635031641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:56:11.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: audit 2024-04-02T04:56:09.997228+0000 mon.a (mon.0) 2921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: cluster 2024-04-02T04:56:10.004111+0000 mon.a (mon.0) 2922 : cluster 0 osdmap e522: 8 total, 8 up, 6 in 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: audit 2024-04-02T04:56:10.046042+0000 mon.c (mon.2) 905 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: audit 2024-04-02T04:56:10.150994+0000 mon.b (mon.1) 263 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: cluster 2024-04-02T04:56:10.154361+0000 mgr.y (mgr.14538) 489 : cluster 0 pgmap v764: 314 pgs: 12 creating+peering, 13 unknown, 1 active+remapped+backfill_toofull, 1 peering, 287 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 2/1230 objects misplaced (0.163%); 1.0 MiB/s, 14 objects/s recovering 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: audit 2024-04-02T04:56:10.158139+0000 mon.a (mon.0) 2923 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[28904]: audit 2024-04-02T04:56:10.692916+0000 mon.a (mon.0) 2924 : audit 1 from='client.? 172.21.15.69:0/1635031641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: audit 2024-04-02T04:56:09.997228+0000 mon.a (mon.0) 2921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: cluster 2024-04-02T04:56:10.004111+0000 mon.a (mon.0) 2922 : cluster 0 osdmap e522: 8 total, 8 up, 6 in 2024-04-02T04:56:11.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: audit 2024-04-02T04:56:10.046042+0000 mon.c (mon.2) 905 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:11.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: audit 2024-04-02T04:56:10.150994+0000 mon.b (mon.1) 263 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:56:11.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: cluster 2024-04-02T04:56:10.154361+0000 mgr.y (mgr.14538) 489 : cluster 0 pgmap v764: 314 pgs: 12 creating+peering, 13 unknown, 1 active+remapped+backfill_toofull, 1 peering, 287 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 2/1230 objects misplaced (0.163%); 1.0 MiB/s, 14 objects/s recovering 2024-04-02T04:56:11.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: audit 2024-04-02T04:56:10.158139+0000 mon.a (mon.0) 2923 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-02T04:56:11.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:11 smithi069 bash[21528]: audit 2024-04-02T04:56:10.692916+0000 mon.a (mon.0) 2924 : audit 1 from='client.? 172.21.15.69:0/1635031641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:56:11.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:11.325 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:11.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:11.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:11.585 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:11.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:11.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:56:11.713 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:11.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:11.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:11.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:12.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:12.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[21528]: cluster 2024-04-02T04:56:11.011875+0000 mon.a (mon.0) 2925 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:56:12.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[21528]: audit 2024-04-02T04:56:11.022920+0000 mon.a (mon.0) 2926 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-02T04:56:12.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[21528]: audit 2024-04-02T04:56:11.023111+0000 mon.a (mon.0) 2927 : audit 1 from='client.? 172.21.15.69:0/1635031641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:56:12.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[21528]: cluster 2024-04-02T04:56:11.027756+0000 mon.a (mon.0) 2928 : cluster 0 osdmap e523: 8 total, 8 up, 6 in 2024-04-02T04:56:12.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[21528]: audit 2024-04-02T04:56:11.047094+0000 mon.c (mon.2) 906 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:12.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[28904]: cluster 2024-04-02T04:56:11.011875+0000 mon.a (mon.0) 2925 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:56:12.084 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[28904]: audit 2024-04-02T04:56:11.022920+0000 mon.a (mon.0) 2926 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-02T04:56:12.084 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[28904]: audit 2024-04-02T04:56:11.023111+0000 mon.a (mon.0) 2927 : audit 1 from='client.? 172.21.15.69:0/1635031641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:56:12.084 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[28904]: cluster 2024-04-02T04:56:11.027756+0000 mon.a (mon.0) 2928 : cluster 0 osdmap e523: 8 total, 8 up, 6 in 2024-04-02T04:56:12.084 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:12 smithi069 bash[28904]: audit 2024-04-02T04:56:11.047094+0000 mon.c (mon.2) 906 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:12.095 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:12.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:12 smithi098 bash[23790]: cluster 2024-04-02T04:56:11.011875+0000 mon.a (mon.0) 2925 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:56:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:12 smithi098 bash[23790]: audit 2024-04-02T04:56:11.022920+0000 mon.a (mon.0) 2926 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-72185-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-02T04:56:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:12 smithi098 bash[23790]: audit 2024-04-02T04:56:11.023111+0000 mon.a (mon.0) 2927 : audit 1 from='client.? 172.21.15.69:0/1635031641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:56:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:12 smithi098 bash[23790]: cluster 2024-04-02T04:56:11.027756+0000 mon.a (mon.0) 2928 : cluster 0 osdmap e523: 8 total, 8 up, 6 in 2024-04-02T04:56:12.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:12 smithi098 bash[23790]: audit 2024-04-02T04:56:11.047094+0000 mon.c (mon.2) 906 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:12.224 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:12.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:12.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:12.536 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:12.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:12.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:12.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:12.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:56:12.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:13.001 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:13.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:13.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.031479+0000 mon.a (mon.0) 2929 : cluster 0 osdmap e524: 8 total, 8 up, 6 in 2024-04-02T04:56:13.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.047622+0000 osd.0 (osd.0) 263 : cluster 0 102.10 scrub starts 2024-04-02T04:56:13.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: audit 2024-04-02T04:56:12.048267+0000 mon.c (mon.2) 907 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.050241+0000 osd.0 (osd.0) 264 : cluster 0 102.10 scrub ok 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.137758+0000 osd.6 (osd.6) 235 : cluster 0 13.4 scrub starts 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.139992+0000 osd.6 (osd.6) 236 : cluster 0 13.4 scrub ok 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.155176+0000 mgr.y (mgr.14538) 490 : cluster 0 pgmap v767: 314 pgs: 12 creating+peering, 13 unknown, 1 active+remapped+backfill_toofull, 1 peering, 287 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 2/1230 objects misplaced (0.163%) 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.189217+0000 osd.4 (osd.4) 273 : cluster 0 13.5 scrub starts 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.192208+0000 osd.4 (osd.4) 274 : cluster 0 13.5 scrub ok 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.444978+0000 osd.1 (osd.1) 281 : cluster 0 102.1b scrub starts 2024-04-02T04:56:13.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[23790]: cluster 2024-04-02T04:56:12.449159+0000 osd.1 (osd.1) 282 : cluster 0 102.1b scrub ok 2024-04-02T04:56:13.129 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:13.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:13.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.031479+0000 mon.a (mon.0) 2929 : cluster 0 osdmap e524: 8 total, 8 up, 6 in 2024-04-02T04:56:13.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.047622+0000 osd.0 (osd.0) 263 : cluster 0 102.10 scrub starts 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: audit 2024-04-02T04:56:12.048267+0000 mon.c (mon.2) 907 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.050241+0000 osd.0 (osd.0) 264 : cluster 0 102.10 scrub ok 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.137758+0000 osd.6 (osd.6) 235 : cluster 0 13.4 scrub starts 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.139992+0000 osd.6 (osd.6) 236 : cluster 0 13.4 scrub ok 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.155176+0000 mgr.y (mgr.14538) 490 : cluster 0 pgmap v767: 314 pgs: 12 creating+peering, 13 unknown, 1 active+remapped+backfill_toofull, 1 peering, 287 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 2/1230 objects misplaced (0.163%) 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.189217+0000 osd.4 (osd.4) 273 : cluster 0 13.5 scrub starts 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.192208+0000 osd.4 (osd.4) 274 : cluster 0 13.5 scrub ok 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.444978+0000 osd.1 (osd.1) 281 : cluster 0 102.1b scrub starts 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[21528]: cluster 2024-04-02T04:56:12.449159+0000 osd.1 (osd.1) 282 : cluster 0 102.1b scrub ok 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.031479+0000 mon.a (mon.0) 2929 : cluster 0 osdmap e524: 8 total, 8 up, 6 in 2024-04-02T04:56:13.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.047622+0000 osd.0 (osd.0) 263 : cluster 0 102.10 scrub starts 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: audit 2024-04-02T04:56:12.048267+0000 mon.c (mon.2) 907 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.050241+0000 osd.0 (osd.0) 264 : cluster 0 102.10 scrub ok 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.137758+0000 osd.6 (osd.6) 235 : cluster 0 13.4 scrub starts 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.139992+0000 osd.6 (osd.6) 236 : cluster 0 13.4 scrub ok 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.155176+0000 mgr.y (mgr.14538) 490 : cluster 0 pgmap v767: 314 pgs: 12 creating+peering, 13 unknown, 1 active+remapped+backfill_toofull, 1 peering, 287 active+clean; 8.1 MiB data, 445 MiB used, 536 GiB / 536 GiB avail; 2/1230 objects misplaced (0.163%) 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.189217+0000 osd.4 (osd.4) 273 : cluster 0 13.5 scrub starts 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.192208+0000 osd.4 (osd.4) 274 : cluster 0 13.5 scrub ok 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.444978+0000 osd.1 (osd.1) 281 : cluster 0 102.1b scrub starts 2024-04-02T04:56:13.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:13 smithi069 bash[28904]: cluster 2024-04-02T04:56:12.449159+0000 osd.1 (osd.1) 282 : cluster 0 102.1b scrub ok 2024-04-02T04:56:13.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:13.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:13.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:56:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:56:13.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:13.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:13.518 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:13.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:13.655 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:13.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:13.780 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:13.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:13.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:13.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:14.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:14.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:56:14.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.177 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:14.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:14.298 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:14.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[21528]: cluster 2024-04-02T04:56:13.038496+0000 mon.a (mon.0) 2930 : cluster 0 osdmap e525: 8 total, 8 up, 6 in 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[21528]: audit 2024-04-02T04:56:13.049478+0000 mon.c (mon.2) 908 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[21528]: cluster 2024-04-02T04:56:13.070898+0000 osd.0 (osd.0) 265 : cluster 0 102.3 scrub starts 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[21528]: cluster 2024-04-02T04:56:13.074242+0000 osd.0 (osd.0) 266 : cluster 0 102.3 scrub ok 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[21528]: audit 2024-04-02T04:56:13.346381+0000 mgr.y (mgr.14538) 491 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[28904]: cluster 2024-04-02T04:56:13.038496+0000 mon.a (mon.0) 2930 : cluster 0 osdmap e525: 8 total, 8 up, 6 in 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[28904]: audit 2024-04-02T04:56:13.049478+0000 mon.c (mon.2) 908 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[28904]: cluster 2024-04-02T04:56:13.070898+0000 osd.0 (osd.0) 265 : cluster 0 102.3 scrub starts 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[28904]: cluster 2024-04-02T04:56:13.074242+0000 osd.0 (osd.0) 266 : cluster 0 102.3 scrub ok 2024-04-02T04:56:14.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:14 smithi069 bash[28904]: audit 2024-04-02T04:56:13.346381+0000 mgr.y (mgr.14538) 491 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:14 smithi098 bash[23790]: cluster 2024-04-02T04:56:13.038496+0000 mon.a (mon.0) 2930 : cluster 0 osdmap e525: 8 total, 8 up, 6 in 2024-04-02T04:56:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:14 smithi098 bash[23790]: audit 2024-04-02T04:56:13.049478+0000 mon.c (mon.2) 908 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:14.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:14 smithi098 bash[23790]: cluster 2024-04-02T04:56:13.070898+0000 osd.0 (osd.0) 265 : cluster 0 102.3 scrub starts 2024-04-02T04:56:14.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:14 smithi098 bash[23790]: cluster 2024-04-02T04:56:13.074242+0000 osd.0 (osd.0) 266 : cluster 0 102.3 scrub ok 2024-04-02T04:56:14.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:14 smithi098 bash[23790]: audit 2024-04-02T04:56:13.346381+0000 mgr.y (mgr.14538) 491 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:14.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:14.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:14.543 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:14.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:14.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.794 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:14.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:14.922 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:14.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:15.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:15.044 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:15.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:15.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:13.966796+0000 osd.7 (osd.7) 65 : cluster 0 43.d scrub starts 2024-04-02T04:56:15.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:13.968436+0000 osd.7 (osd.7) 66 : cluster 0 43.d scrub ok 2024-04-02T04:56:15.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.038296+0000 mon.a (mon.0) 2931 : cluster 0 osdmap e526: 8 total, 8 up, 6 in 2024-04-02T04:56:15.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: audit 2024-04-02T04:56:14.050402+0000 mon.c (mon.2) 909 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: audit 2024-04-02T04:56:14.113316+0000 mon.b (mon.1) 264 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.150277+0000 osd.4 (osd.4) 275 : cluster 0 6.e scrub starts 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.151483+0000 osd.4 (osd.4) 276 : cluster 0 6.e scrub ok 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.155802+0000 mgr.y (mgr.14538) 492 : cluster 0 pgmap v770: 313 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 311 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: audit 2024-04-02T04:56:14.590944+0000 mon.a (mon.0) 2932 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: audit 2024-04-02T04:56:14.597860+0000 mon.a (mon.0) 2933 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.926945+0000 mon.a (mon.0) 2934 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.934482+0000 mon.a (mon.0) 2935 : cluster 0 osdmap e527: 8 total, 8 up, 6 in 2024-04-02T04:56:15.159 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:13.966796+0000 osd.7 (osd.7) 65 : cluster 0 43.d scrub starts 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:13.968436+0000 osd.7 (osd.7) 66 : cluster 0 43.d scrub ok 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.038296+0000 mon.a (mon.0) 2931 : cluster 0 osdmap e526: 8 total, 8 up, 6 in 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: audit 2024-04-02T04:56:14.050402+0000 mon.c (mon.2) 909 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: audit 2024-04-02T04:56:14.113316+0000 mon.b (mon.1) 264 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.150277+0000 osd.4 (osd.4) 275 : cluster 0 6.e scrub starts 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.151483+0000 osd.4 (osd.4) 276 : cluster 0 6.e scrub ok 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.155802+0000 mgr.y (mgr.14538) 492 : cluster 0 pgmap v770: 313 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 311 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: audit 2024-04-02T04:56:14.590944+0000 mon.a (mon.0) 2932 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: audit 2024-04-02T04:56:14.597860+0000 mon.a (mon.0) 2933 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.926945+0000 mon.a (mon.0) 2934 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:15.160 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:15 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.934482+0000 mon.a (mon.0) 2935 : cluster 0 osdmap e527: 8 total, 8 up, 6 in 2024-04-02T04:56:15.173 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:15.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:13.966796+0000 osd.7 (osd.7) 65 : cluster 0 43.d scrub starts 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:13.968436+0000 osd.7 (osd.7) 66 : cluster 0 43.d scrub ok 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.038296+0000 mon.a (mon.0) 2931 : cluster 0 osdmap e526: 8 total, 8 up, 6 in 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: audit 2024-04-02T04:56:14.050402+0000 mon.c (mon.2) 909 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: audit 2024-04-02T04:56:14.113316+0000 mon.b (mon.1) 264 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.150277+0000 osd.4 (osd.4) 275 : cluster 0 6.e scrub starts 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.151483+0000 osd.4 (osd.4) 276 : cluster 0 6.e scrub ok 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.155802+0000 mgr.y (mgr.14538) 492 : cluster 0 pgmap v770: 313 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 311 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:15.286 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: audit 2024-04-02T04:56:14.590944+0000 mon.a (mon.0) 2932 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:15.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: audit 2024-04-02T04:56:14.597860+0000 mon.a (mon.0) 2933 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:15.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.926945+0000 mon.a (mon.0) 2934 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:15.287 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:15 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.934482+0000 mon.a (mon.0) 2935 : cluster 0 osdmap e527: 8 total, 8 up, 6 in 2024-04-02T04:56:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:15.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:15.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:15.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:15.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:15.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:15.814 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:15.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:15.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:15.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:16.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:16.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:16.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:16.195 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:16.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:16.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- 172.21.15.69:0/4189544853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 msgr2=0x7f1800103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:16.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- 172.21.15.69:0/4189544853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f1800103490 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009920 tx=0x7f17f0009c30 comp rx=0 tx=0).stop 2024-04-02T04:56:16.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- 172.21.15.69:0/4189544853 shutdown_connections 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- 172.21.15.69:0/4189544853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 0x7f1800111390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- 172.21.15.69:0/4189544853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f1800103490 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- 172.21.15.69:0/4189544853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f1800101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- 172.21.15.69:0/4189544853 >> 172.21.15.69:0/4189544853 conn(0x7f1800076460 msgr2=0x7f1800076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- 172.21.15.69:0/4189544853 shutdown_connections 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- 172.21.15.69:0/4189544853 wait complete. 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 librados: starting msgr at 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 librados: starting objecter 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 Processor -- start 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- start start 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 librados: setting wanted keys 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 librados: calling monclient init 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f180019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:16.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 0x7f180019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f180019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1800113e10 con 0x7f1800100c00 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1800113c90 con 0x7f18001015a0 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1806b93700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1800113f90 con 0x7f18001030b0 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f180492f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f180019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f1805130700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f180019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f180492f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f180019a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54034/0 (socket says 172.21.15.69:54034) 2024-04-02T04:56:16.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.221+0000 7f180492f700 1 -- 172.21.15.69:0/2911287831 learned_addr learned my addr 172.21.15.69:0/2911287831 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.216753+0000 osd.2 (osd.2) 228 : cluster 0 40.11 scrub starts 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: cluster 2024-04-02T04:56:14.217792+0000 osd.2 (osd.2) 229 : cluster 0 40.11 scrub ok 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: audit 2024-04-02T04:56:15.053893+0000 mon.c (mon.2) 910 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: cluster 2024-04-02T04:56:15.167306+0000 osd.4 (osd.4) 277 : cluster 0 4.c scrub starts 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: cluster 2024-04-02T04:56:15.169178+0000 osd.4 (osd.4) 278 : cluster 0 4.c scrub ok 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: cluster 2024-04-02T04:56:15.450972+0000 osd.1 (osd.1) 283 : cluster 0 4.4 scrub starts 2024-04-02T04:56:16.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:16 smithi098 bash[23790]: cluster 2024-04-02T04:56:15.453128+0000 osd.1 (osd.1) 284 : cluster 0 4.4 scrub ok 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fffff700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 0x7f180019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f180492f700 1 -- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 msgr2=0x7f180019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f180492f700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f180019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f180492f700 1 -- 172.21.15.69:0/2911287831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 msgr2=0x7f180019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f180492f700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 0x7f180019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f180492f700 1 -- 172.21.15.69:0/2911287831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180019f3c0 con 0x7f1800100c00 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fffff700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 0x7f180019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f1805130700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f180019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f180492f700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f180019a3b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f17f0000c00 tx=0x7f17f000fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 10 client.?.objecter ms_handle_connect 0x7f1800100c00 2024-04-02T04:56:16.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f000b660 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f17f0005040 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180019f650 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0003960 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f1806b93700 10 client.16548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18001a6f30 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f1806b93700 1 librados: init done 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f180004ff00 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f17f000b800 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e806f5e0 0x7f17e8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(527..527 src has 74..527) ==== 10551+0+0 (secure 0 0 0) 0x7f17f008fd50 con 0x7f1800100c00 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fffff700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e806f5e0 0x7f17e8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 10 client.16548.objecter ms_dispatch 0x7f1800001550 osd_map(527..527 src has 74..527) 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 3 client.16548.objecter handle_osd_map got epochs [527,527] > 0 2024-04-02T04:56:16.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 3 client.16548.objecter handle_osd_map decoding full epoch 527 2024-04-02T04:56:16.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 20 client.16548.objecter dump_active .. 0 homeless 2024-04-02T04:56:16.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fdffb700 10 client.16548.objecter ms_handle_connect 0x7f17e806f5e0 2024-04-02T04:56:16.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.225+0000 7f17fffff700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e806f5e0 0x7f17e8071aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1800076a80 tx=0x7f17f4009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:16.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.229+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f17f0093030 con 0x7f1800100c00 2024-04-02T04:56:16.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:16.326 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:16.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:16.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:16.429 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:16.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.216753+0000 osd.2 (osd.2) 228 : cluster 0 40.11 scrub starts 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: cluster 2024-04-02T04:56:14.217792+0000 osd.2 (osd.2) 229 : cluster 0 40.11 scrub ok 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: audit 2024-04-02T04:56:15.053893+0000 mon.c (mon.2) 910 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: cluster 2024-04-02T04:56:15.167306+0000 osd.4 (osd.4) 277 : cluster 0 4.c scrub starts 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: cluster 2024-04-02T04:56:15.169178+0000 osd.4 (osd.4) 278 : cluster 0 4.c scrub ok 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: cluster 2024-04-02T04:56:15.450972+0000 osd.1 (osd.1) 283 : cluster 0 4.4 scrub starts 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[28904]: cluster 2024-04-02T04:56:15.453128+0000 osd.1 (osd.1) 284 : cluster 0 4.4 scrub ok 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.216753+0000 osd.2 (osd.2) 228 : cluster 0 40.11 scrub starts 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: cluster 2024-04-02T04:56:14.217792+0000 osd.2 (osd.2) 229 : cluster 0 40.11 scrub ok 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: audit 2024-04-02T04:56:15.053893+0000 mon.c (mon.2) 910 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: cluster 2024-04-02T04:56:15.167306+0000 osd.4 (osd.4) 277 : cluster 0 4.c scrub starts 2024-04-02T04:56:16.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: cluster 2024-04-02T04:56:15.169178+0000 osd.4 (osd.4) 278 : cluster 0 4.c scrub ok 2024-04-02T04:56:16.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: cluster 2024-04-02T04:56:15.450972+0000 osd.1 (osd.1) 283 : cluster 0 4.4 scrub starts 2024-04-02T04:56:16.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:16 smithi069 bash[21528]: cluster 2024-04-02T04:56:15.453128+0000 osd.1 (osd.1) 284 : cluster 0 4.4 scrub ok 2024-04-02T04:56:16.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.445+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f180002db30 con 0x7f1800100c00 2024-04-02T04:56:16.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.449+0000 7f17fdffb700 1 -- 172.21.15.69:0/2911287831 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v527) ==== 81+0+68023 (secure 0 0 0) 0x7f17f0013030 con 0x7f1800100c00 2024-04-02T04:56:16.452 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:56:16.452 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 527, 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "created": "2024-04-02T04:33:43.348385+0000", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "modified": "2024-04-02T04:56:14.927439+0000", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "last_up_change": "2024-04-02T04:53:53.913436+0000", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "last_in_change": "2024-04-02T04:56:00.922950+0000", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_num": 5804032, 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_set": [ 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "nodeep-scrub", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "pglog_hardlimit", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "purged_snapdirs", 2024-04-02T04:56:16.453 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_deletes", 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "sortbitwise" 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_version": 106, 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "full_ratio": 0.94999998807907104, 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_snapshot": "", 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_max": 298, 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd": 8, 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "require_min_compat_client": "luminous", 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "min_compat_client": "luminous", 2024-04-02T04:56:16.454 INFO:teuthology.orchestra.run.smithi069.stdout: "require_osd_release": "squid", 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_crimson": false, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pools": [ 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 1, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".mgr", 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:38:57.671714+0000", 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.455 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 11, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 11, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 11, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 11, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 11, 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.456 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "481", 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "479", 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.457 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.458 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.459 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_max": 32, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_min": 1 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": {} 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.9100000858306885, 2024-04-02T04:56:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.9100000858306885, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.1800000667572021, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.1800000667572021, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 2, 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".rgw.root", 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:52.801763+0000", 2024-04-02T04:56:16.461 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.462 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.463 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "79", 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.464 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.466 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:56:16.467 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 3, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.log", 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:54.772439+0000", 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.468 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "81", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.470 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.471 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.472 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.473 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 4, 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.control", 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:56.696111+0000", 2024-04-02T04:56:16.474 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.475 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "83", 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.476 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.477 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.478 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.479 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.25, 2024-04-02T04:56:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.25, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.690000057220459, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.690000057220459, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 5, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "datapool", 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:57.254108+0000", 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:56:16.481 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 3, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 3, 2024-04-02T04:56:16.482 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 3, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 3, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 3, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "89", 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.483 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 2, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 89, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.484 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.485 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd": {} 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.486 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.6700000762939453, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.6700000762939453, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 6, 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.meta", 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:58.846023+0000", 2024-04-02T04:56:16.487 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.488 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "85", 2024-04-02T04:56:16.489 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.490 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.491 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_bias": 4 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.492 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 10, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:11.754451+0000", 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.495 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.496 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.497 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.498 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.5099999904632568, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.5099999904632568, 2024-04-02T04:56:16.499 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.8799999952316284, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 13, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosList_smithi069-72185-1", 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:13.034188+0000", 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.500 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 11, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 11, 2024-04-02T04:56:16.501 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 11, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 11, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 11, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "13.b", 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 525, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 524, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 524, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "478'14", 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "478'11" 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "525", 2024-04-02T04:56:16.502 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "525", 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "523", 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.503 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.504 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.190000057220459, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.190000057220459, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.6399999856948853, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.6399999856948853, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.506 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 40, 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace.1804289383", 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:24.538674+0000", 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.507 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.508 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:56:16.509 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.510 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.511 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.512 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.513 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 43, 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "poolrace2.846930886", 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:25.876339+0000", 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.514 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.515 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "146", 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.516 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.517 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.518 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.75, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.75, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:56:16.519 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 102, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-6", 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:53.761868+0000", 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "523", 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "436", 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "436", 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "436", 2024-04-02T04:56:16.522 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 21, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 523, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.523 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.524 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: "fingerprint_algorithm": "sha1" 2024-04-02T04:56:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5, 2024-04-02T04:56:16.526 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 298, 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-71", 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:56:07.375432+0000", 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:56:16.527 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:56:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "522", 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:56:16.529 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:56:16.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:56:16.532 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.75, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.75, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.75, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.309999942779541, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.309999942779541, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.533 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b", 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 8, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 525, 2024-04-02T04:56:16.534 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6802", 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6803", 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.535 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6804", 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6805", 2024-04-02T04:56:16.536 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6808", 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6809", 2024-04-02T04:56:16.537 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6806", 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.538 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6807", 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6803/1087500595", 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6805/1087500595", 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6809/1087500595", 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6807/1087500595", 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a", 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 13, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 520, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.540 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6810", 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6811", 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.541 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6812", 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6813", 2024-04-02T04:56:16.542 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6816", 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.543 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6817", 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6814", 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6815", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6811/2651666015", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6813/2651666015", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6817/2651666015", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6815/2651666015", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7", 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 19, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 520, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.546 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6818", 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6819", 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.547 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6820", 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6821", 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.548 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6824", 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6825", 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6822", 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6823", 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.550 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6819/619998764", 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6821/619998764", 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6825/619998764", 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6823/619998764", 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed", 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.551 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 0, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 29, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 512, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6826", 2024-04-02T04:56:16.552 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6827", 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6828", 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6829", 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6832", 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6833", 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6830", 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6831", 2024-04-02T04:56:16.556 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6827/207593208", 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6829/207593208", 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6833/207593208", 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6831/207593208", 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.557 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852", 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 36, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 520, 2024-04-02T04:56:16.558 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6800", 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6801", 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.559 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6802", 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.560 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6803", 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6806", 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6807", 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6804", 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.562 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6805", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6801/2065947829", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6803/2065947829", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6807/2065947829", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6805/2065947829", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.563 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe", 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 0, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 47, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 445, 2024-04-02T04:56:16.564 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6808", 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6809", 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.565 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6810", 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6811", 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6814", 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6815", 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.567 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6812", 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6813", 2024-04-02T04:56:16.568 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6809/219578185", 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6811/219578185", 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6815/219578185", 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6813/219578185", 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.569 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727", 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 58, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 521, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.570 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6816", 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6817", 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.571 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6818", 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6819", 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.572 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6822", 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6823", 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.573 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6820", 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6821", 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.574 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6817/1226172499", 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6819/1226172499", 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6823/1226172499", 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6821/1226172499", 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3", 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:56:16.575 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 69, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 152, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 394, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 520, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 152, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6824", 2024-04-02T04:56:16.576 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6825", 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6826", 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6827", 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6830", 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6831", 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:56:16.579 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6828", 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6829", 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6825/575416324", 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6827/575416324", 2024-04-02T04:56:16.580 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6831/575416324", 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6829/575416324", 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_xinfo": [ 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.581 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:55:37.065858+0000", 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.582 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:56:06.464734+0000", 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:55:22.864920+0000", 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.583 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 65536, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:56:06.515102+0000", 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.584 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:55:07.018301+0000", 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.585 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 65536, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:55:02.640259+0000", 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:56:08.250034+0000", 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:56:16.586 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.587 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.587 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:56:16.587 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:56:16.587 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:56:16.587 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:56:16.589 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:55:04.922383+0000", 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 150 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap": [], 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_items": [ 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "2.9", 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 7, 2024-04-02T04:56:16.590 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.f", 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.17", 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_primaries": [], 2024-04-02T04:56:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_temp": [ 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "13.6", 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: 4, 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: 2, 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: 3 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_temp": [], 2024-04-02T04:56:16.593 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklist": { 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2738867930": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1798147282": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/52352690": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2928651991": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2763215770": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/3657409442": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/879855771": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:56:16.594 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1025808168": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1168952113": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1892956550": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/54578327": "2024-04-03T04:44:51.699278+0000" 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: "range_blocklist": {}, 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profiles": { 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "default": { 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van" 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps_queue": [ 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 17, 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 36, 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:56:16.597 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 38, 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 66, 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 101, 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 103, 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 112, 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 9 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:56:16.602 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 120, 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.603 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 137, 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 145, 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 146, 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.605 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 154, 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 164, 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.606 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 3 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 165, 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.607 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 173, 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 176, 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.608 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 188, 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.609 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 189, 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 196, 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:56:16.610 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 209, 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 6 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.611 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 214, 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 234, 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 8, 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.612 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 245, 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 10, 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 251, 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.613 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 11, 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 262, 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.614 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "new_removed_snaps": [], 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "new_purged_snaps": [], 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_node_flags": {}, 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "device_class_flags": {}, 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode": { 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_enabled": false, 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_bucket_count": 0, 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "degraded_stretch_mode": 0, 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "recovering_stretch_mode": 0, 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_bucket": 0 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:56:16.615 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:56:16.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 10 librados: watch_flush enter 2024-04-02T04:56:16.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 10 librados: watch_flush exit 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 20 client.16548.objecter shutdown clearing up homeless session... 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 10 client.16548.objecter successfully canceled tick 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e806f5e0 msgr2=0x7f17e8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e806f5e0 0x7f17e8071aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1800076a80 tx=0x7f17f4009310 comp rx=0 tx=0).stop 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 msgr2=0x7f180019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f180019a3b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f17f0000c00 tx=0x7f17f000fad0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 shutdown_connections 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e806f5e0 0x7f17e8071aa0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18001030b0 0x7f180019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1800100c00 0x7f180019a3b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 --2- 172.21.15.69:0/2911287831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18001015a0 0x7f180019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 >> 172.21.15.69:0/2911287831 conn(0x7f1800076460 msgr2=0x7f18000fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 shutdown_connections 2024-04-02T04:56:16.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 -- 172.21.15.69:0/2911287831 wait complete. 2024-04-02T04:56:16.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:16.453+0000 7f1806b93700 1 librados: shutdown 2024-04-02T04:56:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:16.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:16.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:16.663 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:56:16.664 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:56:16.664 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:56:16.664 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:56:16.664 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:56:16.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:16.745 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:16.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:16.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:16.873 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:16.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:16.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:17.003 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:17.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:17.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.025689+0000 osd.0 (osd.0) 267 : cluster 0 40.15 scrub starts 2024-04-02T04:56:17.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.027879+0000 osd.0 (osd.0) 268 : cluster 0 40.15 scrub ok 2024-04-02T04:56:17.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: audit 2024-04-02T04:56:16.055907+0000 mon.c (mon.2) 911 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:17.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.122230+0000 osd.4 (osd.4) 279 : cluster 0 6.2 scrub starts 2024-04-02T04:56:17.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.123500+0000 osd.4 (osd.4) 280 : cluster 0 6.2 scrub ok 2024-04-02T04:56:17.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.142855+0000 osd.6 (osd.6) 237 : cluster 0 2.11 scrub starts 2024-04-02T04:56:17.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.144269+0000 osd.6 (osd.6) 238 : cluster 0 2.11 scrub ok 2024-04-02T04:56:17.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.156307+0000 mgr.y (mgr.14538) 493 : cluster 0 pgmap v772: 313 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 311 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 991 B/s wr, 3 op/s; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:17.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:17 smithi098 bash[23790]: audit 2024-04-02T04:56:16.448512+0000 mon.a (mon.0) 2936 : audit 0 from='client.? 172.21.15.69:0/2911287831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:56:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:17.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.025689+0000 osd.0 (osd.0) 267 : cluster 0 40.15 scrub starts 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.027879+0000 osd.0 (osd.0) 268 : cluster 0 40.15 scrub ok 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: audit 2024-04-02T04:56:16.055907+0000 mon.c (mon.2) 911 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.122230+0000 osd.4 (osd.4) 279 : cluster 0 6.2 scrub starts 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.123500+0000 osd.4 (osd.4) 280 : cluster 0 6.2 scrub ok 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.142855+0000 osd.6 (osd.6) 237 : cluster 0 2.11 scrub starts 2024-04-02T04:56:17.337 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.144269+0000 osd.6 (osd.6) 238 : cluster 0 2.11 scrub ok 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.156307+0000 mgr.y (mgr.14538) 493 : cluster 0 pgmap v772: 313 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 311 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 991 B/s wr, 3 op/s; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[21528]: audit 2024-04-02T04:56:16.448512+0000 mon.a (mon.0) 2936 : audit 0 from='client.? 172.21.15.69:0/2911287831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.025689+0000 osd.0 (osd.0) 267 : cluster 0 40.15 scrub starts 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.027879+0000 osd.0 (osd.0) 268 : cluster 0 40.15 scrub ok 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: audit 2024-04-02T04:56:16.055907+0000 mon.c (mon.2) 911 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.122230+0000 osd.4 (osd.4) 279 : cluster 0 6.2 scrub starts 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.123500+0000 osd.4 (osd.4) 280 : cluster 0 6.2 scrub ok 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.142855+0000 osd.6 (osd.6) 237 : cluster 0 2.11 scrub starts 2024-04-02T04:56:17.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.144269+0000 osd.6 (osd.6) 238 : cluster 0 2.11 scrub ok 2024-04-02T04:56:17.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.156307+0000 mgr.y (mgr.14538) 493 : cluster 0 pgmap v772: 313 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 311 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 991 B/s wr, 3 op/s; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:17.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:17 smithi069 bash[28904]: audit 2024-04-02T04:56:16.448512+0000 mon.a (mon.0) 2936 : audit 0 from='client.? 172.21.15.69:0/2911287831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:56:17.341 INFO:tasks.thrashosds.thrasher:j is {'epoch': 527, 'fsid': 'f978400e-f0a9-11ee-b647-cb9ed24678a4', 'created': '2024-04-02T04:33:43.348385+0000', 'modified': '2024-04-02T04:56:14.927439+0000', 'last_up_change': '2024-04-02T04:53:53.913436+0000', 'last_in_change': '2024-04-02T04:56:00.922950+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 106, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 298, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '481', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '479', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9100000858306885, 'score_stable': 2.9100000858306885, 'optimal_score': 0.75, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-02T04:43:52.801763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-02T04:43:54.772439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-02T04:43:56.696111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-04-02T04:43:57.254108+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-02T04:43:58.846023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-02T04:49:11.754451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'LibRadosList_smithi069-72185-1', 'create_time': '2024-04-02T04:49:13.034188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '13.b', 'ready_epoch': 525, 'last_epoch_started': 524, 'last_epoch_clean': 524, 'source_version': "478'14", 'target_version': "478'11"}, 'last_change': '525', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '525', 'last_force_op_resend_preluminous': '523', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.75, 'raw_score_acting': 1.6399999856948853, 'raw_score_stable': 1.6399999856948853, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 40, 'pool_name': 'poolrace.1804289383', 'create_time': '2024-04-02T04:49:24.538674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 43, 'pool_name': 'poolrace2.846930886', 'create_time': '2024-04-02T04:49:25.876339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 102, 'pool_name': 'test-rados-api-smithi069-72349-6', 'create_time': '2024-04-02T04:49:53.761868+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '523', 'last_force_op_resend': '436', 'last_force_op_resend_prenautilus': '436', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 523, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 298, 'pool_name': 'test-rados-api-smithi069-72349-71', 'create_time': '2024-04-02T04:56:07.375432+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '522', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1087500595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1087500595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 1087500595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1087500595}]}, 'public_addr': '172.21.15.69:6803/1087500595', 'cluster_addr': '172.21.15.69:6805/1087500595', 'heartbeat_back_addr': '172.21.15.69:6809/1087500595', 'heartbeat_front_addr': '172.21.15.69:6807/1087500595', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'afb71182-d4f7-416d-ae44-90f28eab4b7a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6811', 'nonce': 2651666015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6812', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2651666015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6816', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6817', 'nonce': 2651666015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6815', 'nonce': 2651666015}]}, 'public_addr': '172.21.15.69:6811/2651666015', 'cluster_addr': '172.21.15.69:6813/2651666015', 'heartbeat_back_addr': '172.21.15.69:6817/2651666015', 'heartbeat_front_addr': '172.21.15.69:6815/2651666015', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6818', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6819', 'nonce': 619998764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6820', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6821', 'nonce': 619998764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6825', 'nonce': 619998764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6822', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 619998764}]}, 'public_addr': '172.21.15.69:6819/619998764', 'cluster_addr': '172.21.15.69:6821/619998764', 'heartbeat_back_addr': '172.21.15.69:6825/619998764', 'heartbeat_front_addr': '172.21.15.69:6823/619998764', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '20e4526d-2707-44d0-b13a-af36019c42ed', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 512, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6826', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 207593208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 207593208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 207593208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 207593208}]}, 'public_addr': '172.21.15.69:6827/207593208', 'cluster_addr': '172.21.15.69:6829/207593208', 'heartbeat_back_addr': '172.21.15.69:6833/207593208', 'heartbeat_front_addr': '172.21.15.69:6831/207593208', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 2065947829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 2065947829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 2065947829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 2065947829}]}, 'public_addr': '172.21.15.98:6801/2065947829', 'cluster_addr': '172.21.15.98:6803/2065947829', 'heartbeat_back_addr': '172.21.15.98:6807/2065947829', 'heartbeat_front_addr': '172.21.15.98:6805/2065947829', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '902326af-4a40-47c8-a3b8-d5d5536f90fe', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 445, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 219578185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 219578185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6814', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 219578185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6813', 'nonce': 219578185}]}, 'public_addr': '172.21.15.98:6809/219578185', 'cluster_addr': '172.21.15.98:6811/219578185', 'heartbeat_back_addr': '172.21.15.98:6815/219578185', 'heartbeat_front_addr': '172.21.15.98:6813/219578185', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 521, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 1226172499}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 1226172499}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 1226172499}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 1226172499}]}, 'public_addr': '172.21.15.98:6817/1226172499', 'cluster_addr': '172.21.15.98:6819/1226172499', 'heartbeat_back_addr': '172.21.15.98:6823/1226172499', 'heartbeat_front_addr': '172.21.15.98:6821/1226172499', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 69, 'last_clean_end': 152, 'up_from': 394, 'up_thru': 520, 'down_at': 152, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 575416324}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 575416324}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 575416324}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 575416324}]}, 'public_addr': '172.21.15.98:6825/575416324', 'cluster_addr': '172.21.15.98:6827/575416324', 'heartbeat_back_addr': '172.21.15.98:6831/575416324', 'heartbeat_front_addr': '172.21.15.98:6829/575416324', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:55:37.065858+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:56:06.464734+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:55:22.864920+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-02T04:56:06.515102+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:55:07.018301+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-02T04:55:02.640259+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:56:08.250034+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:55:04.922383+0000', 'dead_epoch': 150}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '13.6', 'osds': [4, 2, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.69:0/2738867930': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/1798147282': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/52352690': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/2928651991': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:6800/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/2763215770': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/3657409442': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/879855771': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1025808168': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1168952113': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/1892956550': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6801/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/54578327': '2024-04-03T04:44:51.699278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 36, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 38, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 66, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 101, 'snaps': [{'begin': 2, 'length': 4}, {'begin': 7, 'length': 2}]}, {'pool': 103, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 2, 'length': 9}, {'begin': 14, 'length': 1}]}, {'pool': 120, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 137, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 145, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 146, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 164, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 173, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 196, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 209, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 251, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 262, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-02T04:56:17.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:56:17.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:17.393 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:17.496 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:17.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-02T04:56:17.518 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:17.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:17.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:17.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:17.776 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:17.816 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:17.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:17.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:18.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:18.027 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:18.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.719794+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.720047+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.937325+0000 osd.7 (osd.7) 69 : cluster 0 1.4 scrub starts 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:16.938935+0000 osd.7 (osd.7) 70 : cluster 0 1.4 scrub ok 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: audit 2024-04-02T04:56:17.057965+0000 mon.c (mon.2) 912 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:17.077278+0000 mon.a (mon.0) 2937 : cluster 0 osdmap e528: 8 total, 8 up, 6 in 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:17.097641+0000 osd.4 (osd.4) 281 : cluster 0 6.17 scrub starts 2024-04-02T04:56:18.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: cluster 2024-04-02T04:56:17.098956+0000 osd.4 (osd.4) 282 : cluster 0 6.17 scrub ok 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[28904]: audit 2024-04-02T04:56:18.059772+0000 mon.c (mon.2) 913 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.719794+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.720047+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.937325+0000 osd.7 (osd.7) 69 : cluster 0 1.4 scrub starts 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:16.938935+0000 osd.7 (osd.7) 70 : cluster 0 1.4 scrub ok 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: audit 2024-04-02T04:56:17.057965+0000 mon.c (mon.2) 912 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:17.077278+0000 mon.a (mon.0) 2937 : cluster 0 osdmap e528: 8 total, 8 up, 6 in 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:17.097641+0000 osd.4 (osd.4) 281 : cluster 0 6.17 scrub starts 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: cluster 2024-04-02T04:56:17.098956+0000 osd.4 (osd.4) 282 : cluster 0 6.17 scrub ok 2024-04-02T04:56:18.143 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21528]: audit 2024-04-02T04:56:18.059772+0000 mon.c (mon.2) 913 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:18.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:18.286 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:18.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:18.389 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:18.412 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.719794+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.720047+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.937325+0000 osd.7 (osd.7) 69 : cluster 0 1.4 scrub starts 2024-04-02T04:56:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:16.938935+0000 osd.7 (osd.7) 70 : cluster 0 1.4 scrub ok 2024-04-02T04:56:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: audit 2024-04-02T04:56:17.057965+0000 mon.c (mon.2) 912 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:18.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:17.077278+0000 mon.a (mon.0) 2937 : cluster 0 osdmap e528: 8 total, 8 up, 6 in 2024-04-02T04:56:18.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:17.097641+0000 osd.4 (osd.4) 281 : cluster 0 6.17 scrub starts 2024-04-02T04:56:18.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: cluster 2024-04-02T04:56:17.098956+0000 osd.4 (osd.4) 282 : cluster 0 6.17 scrub ok 2024-04-02T04:56:18.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:18 smithi098 bash[23790]: audit 2024-04-02T04:56:18.059772+0000 mon.c (mon.2) 913 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:18.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:18.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:18.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:18.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:18.803 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:18.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:18.933 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:18.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:19.029 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:56:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:56:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:56:19.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:19.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:19.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:19.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:19.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:19.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:17.981825+0000 osd.0 (osd.0) 269 : cluster 0 3.17 scrub starts 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:17.986076+0000 osd.0 (osd.0) 270 : cluster 0 3.17 scrub ok 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.077554+0000 osd.4 (osd.4) 283 : cluster 0 10.1d scrub starts 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.079076+0000 osd.4 (osd.4) 284 : cluster 0 10.1d scrub ok 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.119450+0000 osd.4 (osd.4) 285 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.119645+0000 osd.4 (osd.4) 286 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.128456+0000 osd.6 (osd.6) 239 : cluster 0 3.2 scrub starts 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.130847+0000 osd.6 (osd.6) 240 : cluster 0 3.2 scrub ok 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.157560+0000 mgr.y (mgr.14538) 494 : cluster 0 pgmap v774: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 999 B/s rd, 2.0 KiB/s wr, 4 op/s; 2/1257 objects misplaced (0.159%); 99 B/s, 0 objects/s recovering 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: audit 2024-04-02T04:56:18.204259+0000 mon.a (mon.0) 2938 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: audit 2024-04-02T04:56:18.204311+0000 mon.b (mon.1) 265 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:19.431 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.939577+0000 osd.7 (osd.7) 71 : cluster 0 2.1d scrub starts 2024-04-02T04:56:19.432 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.940734+0000 osd.7 (osd.7) 72 : cluster 0 2.1d scrub ok 2024-04-02T04:56:19.432 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: audit 2024-04-02T04:56:19.061205+0000 mon.c (mon.2) 914 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:19.432 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:19 smithi098 bash[23790]: cluster 2024-04-02T04:56:19.079862+0000 mon.a (mon.0) 2939 : cluster 0 osdmap e529: 8 total, 8 up, 6 in 2024-04-02T04:56:19.446 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:17.981825+0000 osd.0 (osd.0) 269 : cluster 0 3.17 scrub starts 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:17.986076+0000 osd.0 (osd.0) 270 : cluster 0 3.17 scrub ok 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.077554+0000 osd.4 (osd.4) 283 : cluster 0 10.1d scrub starts 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.079076+0000 osd.4 (osd.4) 284 : cluster 0 10.1d scrub ok 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.119450+0000 osd.4 (osd.4) 285 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.119645+0000 osd.4 (osd.4) 286 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.128456+0000 osd.6 (osd.6) 239 : cluster 0 3.2 scrub starts 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.130847+0000 osd.6 (osd.6) 240 : cluster 0 3.2 scrub ok 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.157560+0000 mgr.y (mgr.14538) 494 : cluster 0 pgmap v774: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 999 B/s rd, 2.0 KiB/s wr, 4 op/s; 2/1257 objects misplaced (0.159%); 99 B/s, 0 objects/s recovering 2024-04-02T04:56:19.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: audit 2024-04-02T04:56:18.204259+0000 mon.a (mon.0) 2938 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: audit 2024-04-02T04:56:18.204311+0000 mon.b (mon.1) 265 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.939577+0000 osd.7 (osd.7) 71 : cluster 0 2.1d scrub starts 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.940734+0000 osd.7 (osd.7) 72 : cluster 0 2.1d scrub ok 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: audit 2024-04-02T04:56:19.061205+0000 mon.c (mon.2) 914 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[21528]: cluster 2024-04-02T04:56:19.079862+0000 mon.a (mon.0) 2939 : cluster 0 osdmap e529: 8 total, 8 up, 6 in 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:17.981825+0000 osd.0 (osd.0) 269 : cluster 0 3.17 scrub starts 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:17.986076+0000 osd.0 (osd.0) 270 : cluster 0 3.17 scrub ok 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.077554+0000 osd.4 (osd.4) 283 : cluster 0 10.1d scrub starts 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.079076+0000 osd.4 (osd.4) 284 : cluster 0 10.1d scrub ok 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.119450+0000 osd.4 (osd.4) 285 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:19.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.119645+0000 osd.4 (osd.4) 286 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.128456+0000 osd.6 (osd.6) 239 : cluster 0 3.2 scrub starts 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.130847+0000 osd.6 (osd.6) 240 : cluster 0 3.2 scrub ok 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.157560+0000 mgr.y (mgr.14538) 494 : cluster 0 pgmap v774: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.1 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 999 B/s rd, 2.0 KiB/s wr, 4 op/s; 2/1257 objects misplaced (0.159%); 99 B/s, 0 objects/s recovering 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: audit 2024-04-02T04:56:18.204259+0000 mon.a (mon.0) 2938 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: audit 2024-04-02T04:56:18.204311+0000 mon.b (mon.1) 265 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.939577+0000 osd.7 (osd.7) 71 : cluster 0 2.1d scrub starts 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.940734+0000 osd.7 (osd.7) 72 : cluster 0 2.1d scrub ok 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: audit 2024-04-02T04:56:19.061205+0000 mon.c (mon.2) 914 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:19.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:19 smithi069 bash[28904]: cluster 2024-04-02T04:56:19.079862+0000 mon.a (mon.0) 2939 : cluster 0 osdmap e529: 8 total, 8 up, 6 in 2024-04-02T04:56:19.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:19.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:19.705 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:19.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:19.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:19.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:19.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:20.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:20.081 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:20.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:20.207 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:20.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:20.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:20.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:20.460 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:20.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.960989+0000 osd.0 (osd.0) 271 : cluster 0 40.d scrub starts 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: cluster 2024-04-02T04:56:18.962004+0000 osd.0 (osd.0) 272 : cluster 0 40.d scrub ok 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: audit 2024-04-02T04:56:19.408413+0000 mon.a (mon.0) 2940 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: audit 2024-04-02T04:56:19.422350+0000 mon.a (mon.0) 2941 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: cluster 2024-04-02T04:56:19.910865+0000 osd.7 (osd.7) 73 : cluster 0 43.c scrub starts 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: cluster 2024-04-02T04:56:19.912955+0000 osd.7 (osd.7) 74 : cluster 0 43.c scrub ok 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[28904]: audit 2024-04-02T04:56:20.062216+0000 mon.c (mon.2) 915 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.960989+0000 osd.0 (osd.0) 271 : cluster 0 40.d scrub starts 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: cluster 2024-04-02T04:56:18.962004+0000 osd.0 (osd.0) 272 : cluster 0 40.d scrub ok 2024-04-02T04:56:20.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: audit 2024-04-02T04:56:19.408413+0000 mon.a (mon.0) 2940 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:20.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: audit 2024-04-02T04:56:19.422350+0000 mon.a (mon.0) 2941 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:20.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: cluster 2024-04-02T04:56:19.910865+0000 osd.7 (osd.7) 73 : cluster 0 43.c scrub starts 2024-04-02T04:56:20.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: cluster 2024-04-02T04:56:19.912955+0000 osd.7 (osd.7) 74 : cluster 0 43.c scrub ok 2024-04-02T04:56:20.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:20 smithi069 bash[21528]: audit 2024-04-02T04:56:20.062216+0000 mon.c (mon.2) 915 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:20.587 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:20.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:20.717 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:20.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.960989+0000 osd.0 (osd.0) 271 : cluster 0 40.d scrub starts 2024-04-02T04:56:20.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: cluster 2024-04-02T04:56:18.962004+0000 osd.0 (osd.0) 272 : cluster 0 40.d scrub ok 2024-04-02T04:56:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: audit 2024-04-02T04:56:19.408413+0000 mon.a (mon.0) 2940 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: audit 2024-04-02T04:56:19.422350+0000 mon.a (mon.0) 2941 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: cluster 2024-04-02T04:56:19.910865+0000 osd.7 (osd.7) 73 : cluster 0 43.c scrub starts 2024-04-02T04:56:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: cluster 2024-04-02T04:56:19.912955+0000 osd.7 (osd.7) 74 : cluster 0 43.c scrub ok 2024-04-02T04:56:20.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:20 smithi098 bash[23790]: audit 2024-04-02T04:56:20.062216+0000 mon.c (mon.2) 915 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:20.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:20.849 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:20.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:20.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:21.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:21.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:21.231 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:21.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:21.357 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:21.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:21.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.081415+0000 osd.4 (osd.4) 287 : cluster 0 10.17 scrub starts 2024-04-02T04:56:21.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.082785+0000 osd.4 (osd.4) 288 : cluster 0 10.17 scrub ok 2024-04-02T04:56:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.158439+0000 mgr.y (mgr.14538) 495 : cluster 0 pgmap v776: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1260 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.910823+0000 osd.7 (osd.7) 75 : cluster 0 43.17 scrub starts 2024-04-02T04:56:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.913434+0000 osd.7 (osd.7) 76 : cluster 0 43.17 scrub ok 2024-04-02T04:56:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: audit 2024-04-02T04:56:21.047425+0000 mon.a (mon.0) 2942 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: audit 2024-04-02T04:56:21.054944+0000 mon.a (mon.0) 2943 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:21.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:21 smithi098 bash[23790]: audit 2024-04-02T04:56:21.063488+0000 mon.c (mon.2) 916 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:21.487 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:21.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:21.611 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:21.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.081415+0000 osd.4 (osd.4) 287 : cluster 0 10.17 scrub starts 2024-04-02T04:56:21.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.082785+0000 osd.4 (osd.4) 288 : cluster 0 10.17 scrub ok 2024-04-02T04:56:21.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.158439+0000 mgr.y (mgr.14538) 495 : cluster 0 pgmap v776: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1260 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:21.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.910823+0000 osd.7 (osd.7) 75 : cluster 0 43.17 scrub starts 2024-04-02T04:56:21.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.913434+0000 osd.7 (osd.7) 76 : cluster 0 43.17 scrub ok 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: audit 2024-04-02T04:56:21.047425+0000 mon.a (mon.0) 2942 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: audit 2024-04-02T04:56:21.054944+0000 mon.a (mon.0) 2943 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[21528]: audit 2024-04-02T04:56:21.063488+0000 mon.c (mon.2) 916 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.081415+0000 osd.4 (osd.4) 287 : cluster 0 10.17 scrub starts 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.082785+0000 osd.4 (osd.4) 288 : cluster 0 10.17 scrub ok 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.158439+0000 mgr.y (mgr.14538) 495 : cluster 0 pgmap v776: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1260 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.910823+0000 osd.7 (osd.7) 75 : cluster 0 43.17 scrub starts 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.913434+0000 osd.7 (osd.7) 76 : cluster 0 43.17 scrub ok 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: audit 2024-04-02T04:56:21.047425+0000 mon.a (mon.0) 2942 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: audit 2024-04-02T04:56:21.054944+0000 mon.a (mon.0) 2943 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:21.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:21 smithi069 bash[28904]: audit 2024-04-02T04:56:21.063488+0000 mon.c (mon.2) 916 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:21.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:21.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:21.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:21.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:21.993 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:21.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:56:22.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:22.122 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:22.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:22.253 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:22.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:22.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:22.514 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:22.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.943230+0000 osd.0 (osd.0) 273 : cluster 0 43.1 scrub starts 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: cluster 2024-04-02T04:56:20.946540+0000 osd.0 (osd.0) 274 : cluster 0 43.1 scrub ok 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: audit 2024-04-02T04:56:21.623749+0000 mon.b (mon.1) 266 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: audit 2024-04-02T04:56:21.624794+0000 mon.b (mon.1) 267 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: audit 2024-04-02T04:56:21.635303+0000 mon.a (mon.0) 2944 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: cluster 2024-04-02T04:56:21.943668+0000 osd.7 (osd.7) 77 : cluster 0 102.19 scrub starts 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: cluster 2024-04-02T04:56:21.948309+0000 osd.7 (osd.7) 78 : cluster 0 102.19 scrub ok 2024-04-02T04:56:22.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[21528]: audit 2024-04-02T04:56:22.064872+0000 mon.c (mon.2) 917 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.943230+0000 osd.0 (osd.0) 273 : cluster 0 43.1 scrub starts 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: cluster 2024-04-02T04:56:20.946540+0000 osd.0 (osd.0) 274 : cluster 0 43.1 scrub ok 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: audit 2024-04-02T04:56:21.623749+0000 mon.b (mon.1) 266 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: audit 2024-04-02T04:56:21.624794+0000 mon.b (mon.1) 267 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: audit 2024-04-02T04:56:21.635303+0000 mon.a (mon.0) 2944 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: cluster 2024-04-02T04:56:21.943668+0000 osd.7 (osd.7) 77 : cluster 0 102.19 scrub starts 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: cluster 2024-04-02T04:56:21.948309+0000 osd.7 (osd.7) 78 : cluster 0 102.19 scrub ok 2024-04-02T04:56:22.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:22 smithi069 bash[28904]: audit 2024-04-02T04:56:22.064872+0000 mon.c (mon.2) 917 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.943230+0000 osd.0 (osd.0) 273 : cluster 0 43.1 scrub starts 2024-04-02T04:56:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: cluster 2024-04-02T04:56:20.946540+0000 osd.0 (osd.0) 274 : cluster 0 43.1 scrub ok 2024-04-02T04:56:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: audit 2024-04-02T04:56:21.623749+0000 mon.b (mon.1) 266 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:22.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: audit 2024-04-02T04:56:21.624794+0000 mon.b (mon.1) 267 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:22.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: audit 2024-04-02T04:56:21.635303+0000 mon.a (mon.0) 2944 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:22.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: cluster 2024-04-02T04:56:21.943668+0000 osd.7 (osd.7) 77 : cluster 0 102.19 scrub starts 2024-04-02T04:56:22.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: cluster 2024-04-02T04:56:21.948309+0000 osd.7 (osd.7) 78 : cluster 0 102.19 scrub ok 2024-04-02T04:56:22.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:22 smithi098 bash[23790]: audit 2024-04-02T04:56:22.064872+0000 mon.c (mon.2) 917 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:22.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:22.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:22.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:22.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:23.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:23.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:23.169 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:23.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:23.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:23.429 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:23.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:23.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[23790]: cluster 2024-04-02T04:56:21.980881+0000 osd.0 (osd.0) 275 : cluster 0 13.3 scrub starts 2024-04-02T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[23790]: cluster 2024-04-02T04:56:21.985151+0000 osd.0 (osd.0) 276 : cluster 0 13.3 scrub ok 2024-04-02T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[23790]: cluster 2024-04-02T04:56:22.159128+0000 mgr.y (mgr.14538) 496 : cluster 0 pgmap v777: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1260 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[23790]: cluster 2024-04-02T04:56:22.458402+0000 osd.1 (osd.1) 285 : cluster 0 43.10 scrub starts 2024-04-02T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[23790]: cluster 2024-04-02T04:56:22.460505+0000 osd.1 (osd.1) 286 : cluster 0 43.10 scrub ok 2024-04-02T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:23 smithi098 bash[23790]: audit 2024-04-02T04:56:23.066677+0000 mon.c (mon.2) 918 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:23.555 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:23.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:23.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[21528]: cluster 2024-04-02T04:56:21.980881+0000 osd.0 (osd.0) 275 : cluster 0 13.3 scrub starts 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[21528]: cluster 2024-04-02T04:56:21.985151+0000 osd.0 (osd.0) 276 : cluster 0 13.3 scrub ok 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[21528]: cluster 2024-04-02T04:56:22.159128+0000 mgr.y (mgr.14538) 496 : cluster 0 pgmap v777: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1260 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[21528]: cluster 2024-04-02T04:56:22.458402+0000 osd.1 (osd.1) 285 : cluster 0 43.10 scrub starts 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[21528]: cluster 2024-04-02T04:56:22.460505+0000 osd.1 (osd.1) 286 : cluster 0 43.10 scrub ok 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[21528]: audit 2024-04-02T04:56:23.066677+0000 mon.c (mon.2) 918 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[28904]: cluster 2024-04-02T04:56:21.980881+0000 osd.0 (osd.0) 275 : cluster 0 13.3 scrub starts 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[28904]: cluster 2024-04-02T04:56:21.985151+0000 osd.0 (osd.0) 276 : cluster 0 13.3 scrub ok 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[28904]: cluster 2024-04-02T04:56:22.159128+0000 mgr.y (mgr.14538) 496 : cluster 0 pgmap v777: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1260 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[28904]: cluster 2024-04-02T04:56:22.458402+0000 osd.1 (osd.1) 285 : cluster 0 43.10 scrub starts 2024-04-02T04:56:23.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[28904]: cluster 2024-04-02T04:56:22.460505+0000 osd.1 (osd.1) 286 : cluster 0 43.10 scrub ok 2024-04-02T04:56:23.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:23 smithi069 bash[28904]: audit 2024-04-02T04:56:23.066677+0000 mon.c (mon.2) 918 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:23.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:23.818 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:23.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:23.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:24.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:24.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:24.209 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:24.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:24.339 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:24.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:24.464 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:24.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[21528]: cluster 2024-04-02T04:56:23.022493+0000 osd.0 (osd.0) 277 : cluster 0 102.12 scrub starts 2024-04-02T04:56:24.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[21528]: cluster 2024-04-02T04:56:23.025179+0000 osd.0 (osd.0) 278 : cluster 0 102.12 scrub ok 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[21528]: audit 2024-04-02T04:56:23.356830+0000 mgr.y (mgr.14538) 497 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[21528]: cluster 2024-04-02T04:56:23.929280+0000 osd.7 (osd.7) 79 : cluster 0 102.1 scrub starts 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[21528]: cluster 2024-04-02T04:56:23.933379+0000 osd.7 (osd.7) 80 : cluster 0 102.1 scrub ok 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[21528]: audit 2024-04-02T04:56:24.068150+0000 mon.c (mon.2) 919 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[28904]: cluster 2024-04-02T04:56:23.022493+0000 osd.0 (osd.0) 277 : cluster 0 102.12 scrub starts 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[28904]: cluster 2024-04-02T04:56:23.025179+0000 osd.0 (osd.0) 278 : cluster 0 102.12 scrub ok 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[28904]: audit 2024-04-02T04:56:23.356830+0000 mgr.y (mgr.14538) 497 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[28904]: cluster 2024-04-02T04:56:23.929280+0000 osd.7 (osd.7) 79 : cluster 0 102.1 scrub starts 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[28904]: cluster 2024-04-02T04:56:23.933379+0000 osd.7 (osd.7) 80 : cluster 0 102.1 scrub ok 2024-04-02T04:56:24.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:24 smithi069 bash[28904]: audit 2024-04-02T04:56:24.068150+0000 mon.c (mon.2) 919 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:24.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:24.594 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/1310857359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 msgr2=0x7f8d2c102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- 172.21.15.69:0/1310857359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c102190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009990 tx=0x7f8d18009ca0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/1310857359 shutdown_connections 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- 172.21.15.69:0/1310857359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c1026d0 0x7f8d2c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- 172.21.15.69:0/1310857359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d2c107430 0x7f8d2c107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- 172.21.15.69:0/1310857359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c102190 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/1310857359 >> 172.21.15.69:0/1310857359 conn(0x7f8d2c076460 msgr2=0x7f8d2c076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/1310857359 shutdown_connections 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/1310857359 wait complete. 2024-04-02T04:56:24.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 librados: starting msgr at 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 librados: starting objecter 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 Processor -- start 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- start start 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 librados: setting wanted keys 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 librados: calling monclient init 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d2c1026d0 0x7f8d2c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c107430 0x7f8d2c19ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c110af0 con 0x7f8d2c107430 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c110970 con 0x7f8d2c1026d0 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2c110c70 con 0x7f8d2c101d10 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:58018/0 (socket says 172.21.15.69:58018) 2024-04-02T04:56:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 -- 172.21.15.69:0/3434339002 learned_addr learned my addr 172.21.15.69:0/3434339002 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c107430 0x7f8d2c19ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d2c1026d0 0x7f8d2c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d2c1026d0 msgr2=0x7f8d2c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d2c1026d0 0x7f8d2c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c107430 msgr2=0x7f8d2c19ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c107430 0x7f8d2c19ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c19d150 con 0x7f8d2c101d10 2024-04-02T04:56:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c107430 0x7f8d2c19ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c198140 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c970 tx=0x7f8d2000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d29ffb700 10 client.?.objecter ms_handle_connect 0x7f8d2c101d10 2024-04-02T04:56:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d29ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20007aa0 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8d20004cb0 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c19d440 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20005470 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 10 client.26645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c102dc0 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.688+0000 7f8d32e15700 1 librados: init done 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8d2c04ff00 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8d20004750 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8d1c0739f0 0x7f8d1c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(529..529 src has 74..529) ==== 10551+0+0 (secure 0 0 0) 0x7f8d2008eaa0 con 0x7f8d2c101d10 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 osd_map(529..529 src has 74..529) 2024-04-02T04:56:24.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 3 client.26645.objecter handle_osd_map got epochs [529,529] > 0 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 3 client.26645.objecter handle_osd_map decoding full epoch 529 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8d1c0739f0 0x7f8d1c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 20 client.26645.objecter dump_active .. 0 homeless 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v529) ==== 54+0+16 (secure 0 0 0) 0x7f8d20093220 con 0x7f8d2c101d10 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 10 client.26645.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 1 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8d2c062e50 0x7f8d2c063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 1 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 0 1 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 4 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 10 client.26645.objecter _send_command 1 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c104200 con 0x7f8d2c062e50 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8d2c062e50 0x7f8d2c063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8d1c0739f0 0x7f8d1c075eb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800b2e0 tx=0x7f8d1801a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d1c0739f0 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c062e50 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8d2c062e50 0x7f8d2c063230 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c104200 con 0x7f8d2c062e50 2024-04-02T04:56:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 1: 0 ) 2024-04-02T04:56:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 1 = system:0 2024-04-02T04:56:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 4 2024-04-02T04:56:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.692+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 0 1 2024-04-02T04:56:24.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 10 client.26645.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 2 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 2 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1994a0 osd=0 3 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 0 2 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 4 2024-04-02T04:56:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 10 client.26645.objecter _send_command 2 2024-04-02T04:56:24.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.712+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c104c40 con 0x7f8d2c062e50 2024-04-02T04:56:24.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c104c40 con 0x7f8d2c062e50 2024-04-02T04:56:24.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 4 2024-04-02T04:56:24.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 0 2 2024-04-02T04:56:24.717 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:56:24.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 10 client.26645.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8d2c103590 0x7f8d2c103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 1 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 4 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 10 client.26645.objecter _send_command 3 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c062b20 con 0x7f8d2c103590 2024-04-02T04:56:24.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8d2c103590 0x7f8d2c103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c103590 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8d2c103590 0x7f8d2c103970 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c062b20 con 0x7f8d2c103590 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 3: 0 ) 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 3 = system:0 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 4 2024-04-02T04:56:24.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.716+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 1 3 2024-04-02T04:56:24.720 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:24 smithi098 bash[23790]: cluster 2024-04-02T04:56:23.022493+0000 osd.0 (osd.0) 277 : cluster 0 102.12 scrub starts 2024-04-02T04:56:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:24 smithi098 bash[23790]: cluster 2024-04-02T04:56:23.025179+0000 osd.0 (osd.0) 278 : cluster 0 102.12 scrub ok 2024-04-02T04:56:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:24 smithi098 bash[23790]: audit 2024-04-02T04:56:23.356830+0000 mgr.y (mgr.14538) 497 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:24 smithi098 bash[23790]: cluster 2024-04-02T04:56:23.929280+0000 osd.7 (osd.7) 79 : cluster 0 102.1 scrub starts 2024-04-02T04:56:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:24 smithi098 bash[23790]: cluster 2024-04-02T04:56:23.933379+0000 osd.7 (osd.7) 80 : cluster 0 102.1 scrub ok 2024-04-02T04:56:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:24 smithi098 bash[23790]: audit 2024-04-02T04:56:24.068150+0000 mon.c (mon.2) 919 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:24.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 10 client.26645.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 4 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 4 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c104df0 osd=1 3 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 1 4 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 4 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 10 client.26645.objecter _send_command 4 2024-04-02T04:56:24.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.732+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c02c9b0 con 0x7f8d2c103590 2024-04-02T04:56:24.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c02c9b0 con 0x7f8d2c103590 2024-04-02T04:56:24.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 4 2024-04-02T04:56:24.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 1 4 2024-04-02T04:56:24.738 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 10 client.26645.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 5 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8d2c109870 0x7f8d2c109cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 5 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 2 5 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 4 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 10 client.26645.objecter _send_command 5 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c02deb0 con 0x7f8d2c109870 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8d2c109870 0x7f8d2c109cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c109870 2024-04-02T04:56:24.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8d2c109870 0x7f8d2c109cb0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c02deb0 con 0x7f8d2c109870 2024-04-02T04:56:24.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 5: 0 ) 2024-04-02T04:56:24.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 5 = system:0 2024-04-02T04:56:24.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 4 2024-04-02T04:56:24.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.736+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 2 5 2024-04-02T04:56:24.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 10 client.26645.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 6 2024-04-02T04:56:24.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 6 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c02dd40 osd=2 3 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 2 6 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 4 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 10 client.26645.objecter _send_command 6 2024-04-02T04:56:24.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c1092f0 con 0x7f8d2c109870 2024-04-02T04:56:24.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c1092f0 con 0x7f8d2c109870 2024-04-02T04:56:24.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 4 2024-04-02T04:56:24.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 2 6 2024-04-02T04:56:24.758 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 10 client.26645.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 7 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8d2c02d840 0x7f8d2c1ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 7 2024-04-02T04:56:24.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 3 7 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 4 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 10 client.26645.objecter _send_command 7 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c109540 con 0x7f8d2c02d840 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8d2c02d840 0x7f8d2c1ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c02d840 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8d2c02d840 0x7f8d2c1ade80 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c109540 con 0x7f8d2c02d840 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 7: 0 ) 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 7 = system:0 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 4 2024-04-02T04:56:24.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.756+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 3 7 2024-04-02T04:56:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 10 client.26645.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 8 2024-04-02T04:56:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 8 2024-04-02T04:56:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c062cd0 osd=3 3 2024-04-02T04:56:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 3 8 2024-04-02T04:56:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 4 2024-04-02T04:56:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 10 client.26645.objecter _send_command 8 2024-04-02T04:56:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c109540 con 0x7f8d2c02d840 2024-04-02T04:56:24.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c109540 con 0x7f8d2c02d840 2024-04-02T04:56:24.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 4 2024-04-02T04:56:24.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 3 8 2024-04-02T04:56:24.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 10 client.26645.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 9 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8d2c1ab4c0 0x7f8d2c1b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 9 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 4 9 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 4 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 10 client.26645.objecter _send_command 9 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c1ab9a0 con 0x7f8d2c1ab4c0 2024-04-02T04:56:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.776+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8d2c1ab4c0 0x7f8d2c1b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c1ab4c0 2024-04-02T04:56:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8d2c1ab4c0 0x7f8d2c1b1550 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c1ab9a0 con 0x7f8d2c1ab4c0 2024-04-02T04:56:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 9: 0 ) 2024-04-02T04:56:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 9 = system:0 2024-04-02T04:56:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 4 2024-04-02T04:56:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.780+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 4 9 2024-04-02T04:56:24.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 10 client.26645.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 10 2024-04-02T04:56:24.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 10 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c109540 osd=4 3 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 4 10 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 4 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 10 client.26645.objecter _send_command 10 2024-04-02T04:56:24.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.796+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c0008d0 con 0x7f8d2c1ab4c0 2024-04-02T04:56:24.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.800+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c0008d0 con 0x7f8d2c1ab4c0 2024-04-02T04:56:24.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.800+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.800+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.800+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 4 2024-04-02T04:56:24.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.800+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 4 10 2024-04-02T04:56:24.804 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 10 client.26645.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 11 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8d2c1b1fa0 0x7f8d2c1b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 11 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 5 11 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 4 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 10 client.26645.objecter _send_command 11 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c1b7280 con 0x7f8d2c1b1fa0 2024-04-02T04:56:24.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8d2c1b1fa0 0x7f8d2c1b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d2bfff700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8d2c1b1fa0 0x7f8d2c1b4390 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c1b1fa0 2024-04-02T04:56:24.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c1b7280 con 0x7f8d2c1b1fa0 2024-04-02T04:56:24.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 11: 0 ) 2024-04-02T04:56:24.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 11 = system:0 2024-04-02T04:56:24.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 4 2024-04-02T04:56:24.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.804+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 5 11 2024-04-02T04:56:24.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 10 client.26645.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 12 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 12 2024-04-02T04:56:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1abcf0 osd=5 3 2024-04-02T04:56:24.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 5 12 2024-04-02T04:56:24.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 4 2024-04-02T04:56:24.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 10 client.26645.objecter _send_command 12 2024-04-02T04:56:24.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.824+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c1b48d0 con 0x7f8d2c1b1fa0 2024-04-02T04:56:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c1b48d0 con 0x7f8d2c1b1fa0 2024-04-02T04:56:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 4 2024-04-02T04:56:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 5 12 2024-04-02T04:56:24.829 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 10 client.26645.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 13 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8d2c1b4ff0 0x7f8d2c1b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 13 2024-04-02T04:56:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 6 13 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 4 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 10 client.26645.objecter _send_command 13 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c1ba6a0 con 0x7f8d2c1b4ff0 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8d2c1b4ff0 0x7f8d2c1b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c1b4ff0 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d313b2700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8d2c1b4ff0 0x7f8d2c1b5450 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c1ba6a0 con 0x7f8d2c1b4ff0 2024-04-02T04:56:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 13: 0 ) 2024-04-02T04:56:24.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 13 = system:0 2024-04-02T04:56:24.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 4 2024-04-02T04:56:24.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.828+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 6 13 2024-04-02T04:56:24.845 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 10 client.26645.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 14 2024-04-02T04:56:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 14 2024-04-02T04:56:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1b48d0 osd=6 3 2024-04-02T04:56:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 6 14 2024-04-02T04:56:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 4 2024-04-02T04:56:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 10 client.26645.objecter _send_command 14 2024-04-02T04:56:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c1ba6a0 con 0x7f8d2c1b4ff0 2024-04-02T04:56:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c1ba6a0 con 0x7f8d2c1b4ff0 2024-04-02T04:56:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 4 2024-04-02T04:56:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 6 14 2024-04-02T04:56:24.856 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:56:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 10 client.26645.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 15 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f8d2c1b9a90 0x7f8d2c1b9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 15 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 7 15 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 4 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 10 client.26645.objecter _send_command 15 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.852+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8d2c1bda40 con 0x7f8d2c1b9a90 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f8d2c1b9a90 0x7f8d2c1b9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d29ffb700 10 client.26645.objecter ms_handle_connect 0x7f8d2c1b9a90 2024-04-02T04:56:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d30bb1700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f8d2c1b9a90 0x7f8d2c1b9ef0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d2c1bda40 con 0x7f8d2c1b9a90 2024-04-02T04:56:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 15: 0 ) 2024-04-02T04:56:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 15 = system:0 2024-04-02T04:56:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 4 2024-04-02T04:56:24.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.856+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 7 15 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 10 client.26645.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign -1 16 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 20 client.26645.objecter _get_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_remove -1 16 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 20 client.26645.objecter get_session s=0x7f8d2c1ba6a0 osd=7 3 2024-04-02T04:56:24.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 15 client.26645.objecter _session_command_op_assign 7 16 2024-04-02T04:56:24.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 4 2024-04-02T04:56:24.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 10 client.26645.objecter _send_command 16 2024-04-02T04:56:24.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.872+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8d2c1b59e0 con 0x7f8d2c1b9a90 2024-04-02T04:56:24.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d29ffb700 1 -- 172.21.15.69:0/3434339002 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8d2c1b59e0 con 0x7f8d2c1b9a90 2024-04-02T04:56:24.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d29ffb700 10 client.26645.objecter ms_dispatch 0x7f8d2c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:24.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d29ffb700 10 client.26645.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:24.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d29ffb700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 4 2024-04-02T04:56:24.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d29ffb700 15 client.26645.objecter _session_command_op_remove 7 16 2024-04-02T04:56:24.881 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:56:24.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 librados: watch_flush enter 2024-04-02T04:56:24.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 librados: watch_flush exit 2024-04-02T04:56:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.0 2024-04-02T04:56:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8d2c062e50 msgr2=0x7f8d2c063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8d2c062e50 0x7f8d2c063230 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1994a0 osd=0 1 2024-04-02T04:56:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.1 2024-04-02T04:56:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8d2c103590 msgr2=0x7f8d2c103970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f8d2c103590 0x7f8d2c103970 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c104df0 osd=1 1 2024-04-02T04:56:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.2 2024-04-02T04:56:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8d2c109870 msgr2=0x7f8d2c109cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f8d2c109870 0x7f8d2c109cb0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c02dd40 osd=2 1 2024-04-02T04:56:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.3 2024-04-02T04:56:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8d2c02d840 msgr2=0x7f8d2c1ade80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f8d2c02d840 0x7f8d2c1ade80 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d313b2700 1 -- 172.21.15.69:0/3434339002 reap_dead start 2024-04-02T04:56:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c062cd0 osd=3 1 2024-04-02T04:56:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.4 2024-04-02T04:56:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8d2c1ab4c0 msgr2=0x7f8d2c1b1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f8d2c1ab4c0 0x7f8d2c1b1550 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d313b2700 1 -- 172.21.15.69:0/3434339002 reap_dead start 2024-04-02T04:56:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c109540 osd=4 1 2024-04-02T04:56:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.5 2024-04-02T04:56:24.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8d2c1b1fa0 msgr2=0x7f8d2c1b4390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f8d2c1b1fa0 0x7f8d2c1b4390 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1abcf0 osd=5 1 2024-04-02T04:56:24.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.6 2024-04-02T04:56:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8d2c1b4ff0 msgr2=0x7f8d2c1b5450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f8d2c1b4ff0 0x7f8d2c1b5450 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1b48d0 osd=6 1 2024-04-02T04:56:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter close_session for osd.7 2024-04-02T04:56:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f8d2c1b9a90 msgr2=0x7f8d2c1b9ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f8d2c1b9a90 0x7f8d2c1b9ef0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter put_session s=0x7f8d2c1ba6a0 osd=7 1 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 20 client.26645.objecter shutdown clearing up homeless session... 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 10 client.26645.objecter successfully canceled tick 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8d1c0739f0 msgr2=0x7f8d1c075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8d1c0739f0 0x7f8d1c075eb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800b2e0 tx=0x7f8d1801a040 comp rx=0 tx=0).stop 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 msgr2=0x7f8d2c198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 --2- 172.21.15.69:0/3434339002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c101d10 0x7f8d2c198140 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c970 tx=0x7f8d2000cc80 comp rx=0 tx=0).stop 2024-04-02T04:56:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d313b2700 1 -- 172.21.15.69:0/3434339002 reap_dead start 2024-04-02T04:56:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 shutdown_connections 2024-04-02T04:56:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 >> 172.21.15.69:0/3434339002 conn(0x7f8d2c076460 msgr2=0x7f8d2c100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 shutdown_connections 2024-04-02T04:56:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 -- 172.21.15.69:0/3434339002 wait complete. 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:24.880+0000 7f8d32e15700 1 librados: shutdown 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:56:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:56:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:56:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:56:24.948 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:24.967 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:24.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:25.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:25.092 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:25.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:25.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:25.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:25 smithi098 bash[23790]: cluster 2024-04-02T04:56:24.162179+0000 mgr.y (mgr.14538) 498 : cluster 0 pgmap v778: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 2/1248 objects misplaced (0.160%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:25 smithi098 bash[23790]: cluster 2024-04-02T04:56:24.209560+0000 osd.2 (osd.2) 230 : cluster 0 43.3 scrub starts 2024-04-02T04:56:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:25 smithi098 bash[23790]: cluster 2024-04-02T04:56:24.211190+0000 osd.2 (osd.2) 231 : cluster 0 43.3 scrub ok 2024-04-02T04:56:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:25 smithi098 bash[23790]: audit 2024-04-02T04:56:24.697147+0000 mon.c (mon.2) 920 : audit 0 from='client.? 172.21.15.69:0/3434339002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:25 smithi098 bash[23790]: audit 2024-04-02T04:56:25.069194+0000 mon.c (mon.2) 921 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:25.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[21528]: cluster 2024-04-02T04:56:24.162179+0000 mgr.y (mgr.14538) 498 : cluster 0 pgmap v778: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 2/1248 objects misplaced (0.160%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:25.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[21528]: cluster 2024-04-02T04:56:24.209560+0000 osd.2 (osd.2) 230 : cluster 0 43.3 scrub starts 2024-04-02T04:56:25.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[21528]: cluster 2024-04-02T04:56:24.211190+0000 osd.2 (osd.2) 231 : cluster 0 43.3 scrub ok 2024-04-02T04:56:25.483 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[21528]: audit 2024-04-02T04:56:24.697147+0000 mon.c (mon.2) 920 : audit 0 from='client.? 172.21.15.69:0/3434339002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:25.483 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[21528]: audit 2024-04-02T04:56:25.069194+0000 mon.c (mon.2) 921 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:25.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[28904]: cluster 2024-04-02T04:56:24.162179+0000 mgr.y (mgr.14538) 498 : cluster 0 pgmap v778: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 2/1248 objects misplaced (0.160%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:25.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[28904]: cluster 2024-04-02T04:56:24.209560+0000 osd.2 (osd.2) 230 : cluster 0 43.3 scrub starts 2024-04-02T04:56:25.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[28904]: cluster 2024-04-02T04:56:24.211190+0000 osd.2 (osd.2) 231 : cluster 0 43.3 scrub ok 2024-04-02T04:56:25.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[28904]: audit 2024-04-02T04:56:24.697147+0000 mon.c (mon.2) 920 : audit 0 from='client.? 172.21.15.69:0/3434339002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:25.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:25 smithi069 bash[28904]: audit 2024-04-02T04:56:25.069194+0000 mon.c (mon.2) 921 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:25.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:25.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:25.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:25.739 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:56:25.740 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:56:25.743 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:25.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:25.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:25.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:25.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:26.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:26.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:26.266 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:26.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:26.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:26.397 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:26.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 -- 172.21.15.69:0/1853440252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4107260 msgr2=0x7f25f4107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/1853440252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4107260 0x7f25f4107640 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0099c0 tx=0x7f25ec009cd0 comp rx=0 tx=0).stop 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 -- 172.21.15.69:0/1853440252 shutdown_connections 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/1853440252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f4101150 0x7f25f410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/1853440252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4107260 0x7f25f4107640 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/1853440252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25f4100790 0x7f25f4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.472+0000 7f25fc5c8700 1 -- 172.21.15.69:0/1853440252 >> 172.21.15.69:0/1853440252 conn(0x7f25f4077740 msgr2=0x7f25f40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- 172.21.15.69:0/1853440252 shutdown_connections 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- 172.21.15.69:0/1853440252 wait complete. 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 librados: starting msgr at 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 librados: starting objecter 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 Processor -- start 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- start start 2024-04-02T04:56:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 librados: setting wanted keys 2024-04-02T04:56:26.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 librados: calling monclient init 2024-04-02T04:56:26.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25f4100790 0x7f25f41980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:26.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 0x7f25f41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f4107260 0x7f25f419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25f4111b00 con 0x7f25f4101150 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25f4111980 con 0x7f25f4100790 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f25f4111c80 con 0x7f25f4107260 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 0x7f25f41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 0x7f25f41985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36722/0 (socket says 172.21.15.69:36722) 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 -- 172.21.15.69:0/361320645 learned_addr learned my addr 172.21.15.69:0/361320645 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fab65700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f4107260 0x7f25f419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fa364700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25f4100790 0x7f25f41980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 -- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f4107260 msgr2=0x7f25f419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f4107260 0x7f25f419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 -- 172.21.15.69:0/361320645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25f4100790 msgr2=0x7f25f41980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25f4100790 0x7f25f41980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:26.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 -- 172.21.15.69:0/361320645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f419d0b0 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f9b63700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 0x7f25f41985e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f25e800d950 tx=0x7f25e800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f37fe700 10 client.?.objecter ms_handle_connect 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f37fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8016070 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f25e8009dc0 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e800f3c0 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f419d3a0 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 10 client.16560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f41a4c20 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 librados: init done 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f404ff00 con 0x7f25f4101150 2024-04-02T04:56:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.476+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f25e8019070 con 0x7f25f4101150 2024-04-02T04:56:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25dc06f5e0 0x7f25dc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(529..529 src has 74..529) ==== 10551+0+0 (secure 0 0 0) 0x7f25e808e7a0 con 0x7f25f4101150 2024-04-02T04:56:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 10 client.16560.objecter ms_dispatch 0x7f25f4001550 osd_map(529..529 src has 74..529) 2024-04-02T04:56:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 3 client.16560.objecter handle_osd_map got epochs [529,529] > 0 2024-04-02T04:56:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 3 client.16560.objecter handle_osd_map decoding full epoch 529 2024-04-02T04:56:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 20 client.16560.objecter dump_active .. 0 homeless 2024-04-02T04:56:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25fa364700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25dc06f5e0 0x7f25dc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25fa364700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25dc06f5e0 0x7f25dc071aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0037a0 tx=0x7f25ec000bc0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.480+0000 7f25f37fe700 10 client.16560.objecter ms_handle_connect 0x7f25dc06f5e0 2024-04-02T04:56:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.484+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f25e805b1f0 con 0x7f25f4101150 2024-04-02T04:56:26.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: cluster 2024-04-02T04:56:25.007646+0000 osd.0 (osd.0) 279 : cluster 0 40.3 scrub starts 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: cluster 2024-04-02T04:56:25.009019+0000 osd.0 (osd.0) 280 : cluster 0 40.3 scrub ok 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: cluster 2024-04-02T04:56:25.079392+0000 osd.4 (osd.4) 289 : cluster 0 4.1 scrub starts 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: cluster 2024-04-02T04:56:25.081816+0000 osd.4 (osd.4) 290 : cluster 0 4.1 scrub ok 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: cluster 2024-04-02T04:56:25.496411+0000 osd.1 (osd.1) 287 : cluster 0 40.1 scrub starts 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: cluster 2024-04-02T04:56:25.497851+0000 osd.1 (osd.1) 288 : cluster 0 40.1 scrub ok 2024-04-02T04:56:26.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[21528]: audit 2024-04-02T04:56:26.070468+0000 mon.c (mon.2) 922 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: cluster 2024-04-02T04:56:25.007646+0000 osd.0 (osd.0) 279 : cluster 0 40.3 scrub starts 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: cluster 2024-04-02T04:56:25.009019+0000 osd.0 (osd.0) 280 : cluster 0 40.3 scrub ok 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: cluster 2024-04-02T04:56:25.079392+0000 osd.4 (osd.4) 289 : cluster 0 4.1 scrub starts 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: cluster 2024-04-02T04:56:25.081816+0000 osd.4 (osd.4) 290 : cluster 0 4.1 scrub ok 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: cluster 2024-04-02T04:56:25.496411+0000 osd.1 (osd.1) 287 : cluster 0 40.1 scrub starts 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: cluster 2024-04-02T04:56:25.497851+0000 osd.1 (osd.1) 288 : cluster 0 40.1 scrub ok 2024-04-02T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:26 smithi069 bash[28904]: audit 2024-04-02T04:56:26.070468+0000 mon.c (mon.2) 922 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:26.519 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:26.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-02T04:56:26.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:26.639 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:26.704+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f25f41a4e50 con 0x7f25f4101150 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: cluster 2024-04-02T04:56:25.007646+0000 osd.0 (osd.0) 279 : cluster 0 40.3 scrub starts 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: cluster 2024-04-02T04:56:25.009019+0000 osd.0 (osd.0) 280 : cluster 0 40.3 scrub ok 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: cluster 2024-04-02T04:56:25.079392+0000 osd.4 (osd.4) 289 : cluster 0 4.1 scrub starts 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: cluster 2024-04-02T04:56:25.081816+0000 osd.4 (osd.4) 290 : cluster 0 4.1 scrub ok 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: cluster 2024-04-02T04:56:25.496411+0000 osd.1 (osd.1) 287 : cluster 0 40.1 scrub starts 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: cluster 2024-04-02T04:56:25.497851+0000 osd.1 (osd.1) 288 : cluster 0 40.1 scrub ok 2024-04-02T04:56:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:26 smithi098 bash[23790]: audit 2024-04-02T04:56:26.070468+0000 mon.c (mon.2) 922 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:26.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:26.865 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:26.891 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:26.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:26.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:27.017 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:27.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:27.144 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:27.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:27.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:27.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.436+0000 7f25f37fe700 1 -- 172.21.15.69:0/361320645 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v530) ==== 101+0+0 (secure 0 0 0) 0x7f25e805ec40 con 0x7f25f4101150 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 10 librados: watch_flush enter 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 10 librados: watch_flush exit 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 20 client.16560.objecter shutdown clearing up homeless session... 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 10 client.16560.objecter successfully canceled tick 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25dc06f5e0 msgr2=0x7f25dc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25dc06f5e0 0x7f25dc071aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0037a0 tx=0x7f25ec000bc0 comp rx=0 tx=0).stop 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 msgr2=0x7f25f41985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.440+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 0x7f25f41985e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f25e800d950 tx=0x7f25e800dc60 comp rx=0 tx=0).stop 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 shutdown_connections 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f25dc06f5e0 0x7f25dc071aa0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f4107260 0x7f25f419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25f4100790 0x7f25f41980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 --2- 172.21.15.69:0/361320645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f4101150 0x7f25f41985e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 >> 172.21.15.69:0/361320645 conn(0x7f25f4077740 msgr2=0x7f25f4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 shutdown_connections 2024-04-02T04:56:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 -- 172.21.15.69:0/361320645 wait complete. 2024-04-02T04:56:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:27.448+0000 7f25fc5c8700 1 librados: shutdown 2024-04-02T04:56:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:56:27.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:27.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:27 smithi098 bash[23790]: cluster 2024-04-02T04:56:26.163096+0000 mgr.y (mgr.14538) 499 : cluster 0 pgmap v779: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 2/1248 objects misplaced (0.160%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:27.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:27 smithi098 bash[23790]: cluster 2024-04-02T04:56:26.533847+0000 osd.1 (osd.1) 289 : cluster 0 3.0 scrub starts 2024-04-02T04:56:27.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:27 smithi098 bash[23790]: cluster 2024-04-02T04:56:26.536126+0000 osd.1 (osd.1) 290 : cluster 0 3.0 scrub ok 2024-04-02T04:56:27.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:27 smithi098 bash[23790]: audit 2024-04-02T04:56:26.711444+0000 mon.a (mon.0) 2945 : audit 1 from='client.? 172.21.15.69:0/361320645' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:56:27.522 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:27 smithi098 bash[23790]: audit 2024-04-02T04:56:27.072244+0000 mon.c (mon.2) 923 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:27.537 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:27.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:27.653 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[28904]: cluster 2024-04-02T04:56:26.163096+0000 mgr.y (mgr.14538) 499 : cluster 0 pgmap v779: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 2/1248 objects misplaced (0.160%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:27.653 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[28904]: cluster 2024-04-02T04:56:26.533847+0000 osd.1 (osd.1) 289 : cluster 0 3.0 scrub starts 2024-04-02T04:56:27.653 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[28904]: cluster 2024-04-02T04:56:26.536126+0000 osd.1 (osd.1) 290 : cluster 0 3.0 scrub ok 2024-04-02T04:56:27.653 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[28904]: audit 2024-04-02T04:56:26.711444+0000 mon.a (mon.0) 2945 : audit 1 from='client.? 172.21.15.69:0/361320645' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:56:27.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[28904]: audit 2024-04-02T04:56:27.072244+0000 mon.c (mon.2) 923 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:27.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[21528]: cluster 2024-04-02T04:56:26.163096+0000 mgr.y (mgr.14538) 499 : cluster 0 pgmap v779: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 2/1248 objects misplaced (0.160%); 0 B/s, 0 objects/s recovering 2024-04-02T04:56:27.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[21528]: cluster 2024-04-02T04:56:26.533847+0000 osd.1 (osd.1) 289 : cluster 0 3.0 scrub starts 2024-04-02T04:56:27.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[21528]: cluster 2024-04-02T04:56:26.536126+0000 osd.1 (osd.1) 290 : cluster 0 3.0 scrub ok 2024-04-02T04:56:27.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[21528]: audit 2024-04-02T04:56:26.711444+0000 mon.a (mon.0) 2945 : audit 1 from='client.? 172.21.15.69:0/361320645' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:56:27.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:27 smithi069 bash[21528]: audit 2024-04-02T04:56:27.072244+0000 mon.c (mon.2) 923 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:27.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:27.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:27.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:27.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:28.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:28.056 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:28.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:28.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:28.317 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:28.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:28.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:56:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:28.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:28.562 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:26.998390+0000 osd.0 (osd.0) 281 : cluster 0 40.16 scrub starts 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:26.999631+0000 osd.0 (osd.0) 282 : cluster 0 40.16 scrub ok 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:27.045514+0000 osd.6 (osd.6) 241 : cluster 0 5.2 scrub starts 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:27.047850+0000 osd.6 (osd.6) 242 : cluster 0 5.2 scrub ok 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:27.144157+0000 osd.2 (osd.2) 232 : cluster 0 10.5 scrub starts 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:27.145767+0000 osd.2 (osd.2) 233 : cluster 0 10.5 scrub ok 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:27.433271+0000 mon.a (mon.0) 2946 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: audit 2024-04-02T04:56:27.442360+0000 mon.a (mon.0) 2947 : audit 1 from='client.? 172.21.15.69:0/361320645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: cluster 2024-04-02T04:56:27.448500+0000 mon.a (mon.0) 2948 : cluster 0 osdmap e530: 8 total, 8 up, 6 in 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[28904]: audit 2024-04-02T04:56:28.073825+0000 mon.c (mon.2) 924 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:28.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:26.998390+0000 osd.0 (osd.0) 281 : cluster 0 40.16 scrub starts 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:26.999631+0000 osd.0 (osd.0) 282 : cluster 0 40.16 scrub ok 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:27.045514+0000 osd.6 (osd.6) 241 : cluster 0 5.2 scrub starts 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:27.047850+0000 osd.6 (osd.6) 242 : cluster 0 5.2 scrub ok 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:27.144157+0000 osd.2 (osd.2) 232 : cluster 0 10.5 scrub starts 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:27.145767+0000 osd.2 (osd.2) 233 : cluster 0 10.5 scrub ok 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:27.433271+0000 mon.a (mon.0) 2946 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: audit 2024-04-02T04:56:27.442360+0000 mon.a (mon.0) 2947 : audit 1 from='client.? 172.21.15.69:0/361320645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: cluster 2024-04-02T04:56:27.448500+0000 mon.a (mon.0) 2948 : cluster 0 osdmap e530: 8 total, 8 up, 6 in 2024-04-02T04:56:28.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21528]: audit 2024-04-02T04:56:28.073825+0000 mon.c (mon.2) 924 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:28.576 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:28.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:28.701 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:26.998390+0000 osd.0 (osd.0) 281 : cluster 0 40.16 scrub starts 2024-04-02T04:56:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:26.999631+0000 osd.0 (osd.0) 282 : cluster 0 40.16 scrub ok 2024-04-02T04:56:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:27.045514+0000 osd.6 (osd.6) 241 : cluster 0 5.2 scrub starts 2024-04-02T04:56:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:27.047850+0000 osd.6 (osd.6) 242 : cluster 0 5.2 scrub ok 2024-04-02T04:56:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:27.144157+0000 osd.2 (osd.2) 232 : cluster 0 10.5 scrub starts 2024-04-02T04:56:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:27.145767+0000 osd.2 (osd.2) 233 : cluster 0 10.5 scrub ok 2024-04-02T04:56:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:27.433271+0000 mon.a (mon.0) 2946 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:56:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: audit 2024-04-02T04:56:27.442360+0000 mon.a (mon.0) 2947 : audit 1 from='client.? 172.21.15.69:0/361320645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:56:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: cluster 2024-04-02T04:56:27.448500+0000 mon.a (mon.0) 2948 : cluster 0 osdmap e530: 8 total, 8 up, 6 in 2024-04-02T04:56:28.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:28 smithi098 bash[23790]: audit 2024-04-02T04:56:28.073825+0000 mon.c (mon.2) 924 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:28.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:28.918 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:56:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:56:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:56:28.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:28.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:29.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:29.086 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:29.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:29.217 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- 172.21.15.69:0/1953294968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460410e870 msgr2=0x7f460410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- 172.21.15.69:0/1953294968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460410e870 0x7f460410ec50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f45f8009990 tx=0x7f45f8009ca0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- 172.21.15.69:0/1953294968 shutdown_connections 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- 172.21.15.69:0/1953294968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f4604115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- 172.21.15.69:0/1953294968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460410e870 0x7f460410ec50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- 172.21.15.69:0/1953294968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4604108870 0x7f4604108cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- 172.21.15.69:0/1953294968 >> 172.21.15.69:0/1953294968 conn(0x7f4604075a10 msgr2=0x7f4604077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- 172.21.15.69:0/1953294968 shutdown_connections 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- 172.21.15.69:0/1953294968 wait complete. 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 librados: starting msgr at 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 librados: starting objecter 2024-04-02T04:56:29.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 Processor -- start 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- start start 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 librados: setting wanted keys 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 librados: calling monclient init 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4604108870 0x7f460419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f460419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460410e870 0x7f46041a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f46041180c0 con 0x7f4604108870 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4604117f40 con 0x7f460410e870 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4604118240 con 0x7f4604109230 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f460419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f460259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4604108870 0x7f460419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f460419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:45106/0 (socket says 172.21.15.69:45106) 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 -- 172.21.15.69:0/699904820 learned_addr learned my addr 172.21.15.69:0/699904820 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:29.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4602d9d700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460410e870 0x7f46041a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 -- 172.21.15.69:0/699904820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460410e870 msgr2=0x7f46041a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460410e870 0x7f46041a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 -- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4604108870 msgr2=0x7f460419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4604108870 0x7f460419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 -- 172.21.15.69:0/699904820 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46041a3620 con 0x7f4604109230 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f460259c700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4604108870 0x7f460419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4601d9b700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f460419eb50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e9a0 tx=0x7f45f400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f45f37fe700 10 client.?.objecter ms_handle_connect 0x7f4604109230 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f45f37fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f400ca10 con 0x7f4604109230 2024-04-02T04:56:29.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46041a3910 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f45f400cbb0 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4018420 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.288+0000 7f4608d98700 10 client.26657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46041ab190 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f4608d98700 1 librados: init done 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460404ff00 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f45f4010430 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ec06f5e0 0x7f45ec071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(530..530 src has 74..530) ==== 10551+0+0 (secure 0 0 0) 0x7f45f408ee20 con 0x7f4604109230 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 10 client.26657.objecter ms_dispatch 0x7f4604001550 osd_map(530..530 src has 74..530) 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 3 client.26657.objecter handle_osd_map got epochs [530,530] > 0 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 3 client.26657.objecter handle_osd_map decoding full epoch 530 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f460259c700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ec06f5e0 0x7f45ec071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 20 client.26657.objecter dump_active .. 0 homeless 2024-04-02T04:56:29.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f460259c700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ec06f5e0 0x7f45ec071aa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f45f80098e0 tx=0x7f45f8005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:29.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.292+0000 7f45f37fe700 10 client.26657.objecter ms_handle_connect 0x7f45ec06f5e0 2024-04-02T04:56:29.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.296+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f45f405b870 con 0x7f4604109230 2024-04-02T04:56:29.321 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:29.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:29.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:29.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.504+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f460419f970 con 0x7f45ec06f5e0 2024-04-02T04:56:29.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.528+0000 7f45f37fe700 1 -- 172.21.15.69:0/699904820 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+799906 (secure 0 0 0) 0x7f460419f970 con 0x7f45ec06f5e0 2024-04-02T04:56:29.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 10 librados: watch_flush enter 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 10 librados: watch_flush exit 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 20 client.26657.objecter shutdown clearing up homeless session... 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 10 client.26657.objecter successfully canceled tick 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ec06f5e0 msgr2=0x7f45ec071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ec06f5e0 0x7f45ec071aa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f45f80098e0 tx=0x7f45f8005960 comp rx=0 tx=0).stop 2024-04-02T04:56:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 msgr2=0x7f460419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.532+0000 7f4608d98700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f460419eb50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e9a0 tx=0x7f45f400ecb0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 shutdown_connections 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ec06f5e0 0x7f45ec071aa0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460410e870 0x7f46041a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4604108870 0x7f460419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 --2- 172.21.15.69:0/699904820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4604109230 0x7f460419eb50 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 >> 172.21.15.69:0/699904820 conn(0x7f4604075a10 msgr2=0x7f46040763d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 shutdown_connections 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 -- 172.21.15.69:0/699904820 wait complete. 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:29.536+0000 7f4608d98700 1 librados: shutdown 2024-04-02T04:56:29.550 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:56:29.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:29.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:29.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:28.107489+0000 osd.2 (osd.2) 234 : cluster 0 4.19 scrub starts 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:28.109696+0000 osd.2 (osd.2) 235 : cluster 0 4.19 scrub ok 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:28.164574+0000 mgr.y (mgr.14538) 500 : cluster 0 pgmap v781: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 338 B/s wr, 2 op/s; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:28.479315+0000 osd.1 (osd.1) 291 : cluster 0 43.12 scrub starts 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:28.480774+0000 osd.1 (osd.1) 292 : cluster 0 43.12 scrub ok 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:29.002149+0000 osd.7 (osd.7) 81 : cluster 0 102.d deep-scrub starts 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: cluster 2024-04-02T04:56:29.021345+0000 osd.7 (osd.7) 82 : cluster 0 102.d deep-scrub ok 2024-04-02T04:56:29.705 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[21528]: audit 2024-04-02T04:56:29.076398+0000 mon.c (mon.2) 925 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:28.107489+0000 osd.2 (osd.2) 234 : cluster 0 4.19 scrub starts 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:28.109696+0000 osd.2 (osd.2) 235 : cluster 0 4.19 scrub ok 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:28.164574+0000 mgr.y (mgr.14538) 500 : cluster 0 pgmap v781: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 338 B/s wr, 2 op/s; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:28.479315+0000 osd.1 (osd.1) 291 : cluster 0 43.12 scrub starts 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:28.480774+0000 osd.1 (osd.1) 292 : cluster 0 43.12 scrub ok 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:29.002149+0000 osd.7 (osd.7) 81 : cluster 0 102.d deep-scrub starts 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: cluster 2024-04-02T04:56:29.021345+0000 osd.7 (osd.7) 82 : cluster 0 102.d deep-scrub ok 2024-04-02T04:56:29.706 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:29 smithi069 bash[28904]: audit 2024-04-02T04:56:29.076398+0000 mon.c (mon.2) 925 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:28.107489+0000 osd.2 (osd.2) 234 : cluster 0 4.19 scrub starts 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:28.109696+0000 osd.2 (osd.2) 235 : cluster 0 4.19 scrub ok 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:28.164574+0000 mgr.y (mgr.14538) 500 : cluster 0 pgmap v781: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 338 B/s wr, 2 op/s; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:28.479315+0000 osd.1 (osd.1) 291 : cluster 0 43.12 scrub starts 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:28.480774+0000 osd.1 (osd.1) 292 : cluster 0 43.12 scrub ok 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:29.002149+0000 osd.7 (osd.7) 81 : cluster 0 102.d deep-scrub starts 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: cluster 2024-04-02T04:56:29.021345+0000 osd.7 (osd.7) 82 : cluster 0 102.d deep-scrub ok 2024-04-02T04:56:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:29 smithi098 bash[23790]: audit 2024-04-02T04:56:29.076398+0000 mon.c (mon.2) 925 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:29.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:29.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:29.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:29.988 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:29.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:30.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:30.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:30.248 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:30.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:30.378 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:30.471 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":781,"stamp":"2024-04-02T04:56:28.163357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8794219,"num_objects":416,"num_object_clones":5,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":1586,"num_read_kb":1198,"num_write":1691,"num_write_kb":12840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":774,"num_bytes_recovered":8272556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4430,"ondisk_log_size":4430,"up":939,"acting":939,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":939,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":456848,"kb_used_data":81236,"kb_used_omap":64,"kb_used_meta":375487,"kb_avail":562014064,"statfs":{"total":575970213888,"available":575502401536,"internally_reserved":0,"allocated":83185664,"data_stored":62204407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66456,"internal_metadata":384498792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1,"num_objects":-4,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.087323"},"pg_stats":[{"pgid":"298.1f","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925785+0000","last_change":"2024-04-02T04:56:09.011164+0000","last_active":"2024-04-02T04:56:19.925785+0000","last_peered":"2024-04-02T04:56:19.925785+0000","last_clean":"2024-04-02T04:56:19.925785+0000","last_became_active":"2024-04-02T04:56:09.010632+0000","last_became_peered":"2024-04-02T04:56:09.010632+0000","last_unstale":"2024-04-02T04:56:19.925785+0000","last_undegraded":"2024-04-02T04:56:19.925785+0000","last_fullsized":"2024-04-02T04:56:19.925785+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.001052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.1e","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921805+0000","last_change":"2024-04-02T04:56:09.015818+0000","last_active":"2024-04-02T04:56:19.921805+0000","last_peered":"2024-04-02T04:56:19.921805+0000","last_clean":"2024-04-02T04:56:19.921805+0000","last_became_active":"2024-04-02T04:56:09.014775+0000","last_became_peered":"2024-04-02T04:56:09.014775+0000","last_unstale":"2024-04-02T04:56:19.921805+0000","last_undegraded":"2024-04-02T04:56:19.921805+0000","last_fullsized":"2024-04-02T04:56:19.921805+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:16.519476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.1d","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450324+0000","last_change":"2024-04-02T04:56:09.002360+0000","last_active":"2024-04-02T04:56:27.450324+0000","last_peered":"2024-04-02T04:56:27.450324+0000","last_clean":"2024-04-02T04:56:27.450324+0000","last_became_active":"2024-04-02T04:56:09.000759+0000","last_became_peered":"2024-04-02T04:56:09.000759+0000","last_unstale":"2024-04-02T04:56:27.450324+0000","last_undegraded":"2024-04-02T04:56:27.450324+0000","last_fullsized":"2024-04-02T04:56:27.450324+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:09.410347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"298.1c","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919679+0000","last_change":"2024-04-02T04:56:09.013411+0000","last_active":"2024-04-02T04:56:19.919679+0000","last_peered":"2024-04-02T04:56:19.919679+0000","last_clean":"2024-04-02T04:56:19.919679+0000","last_became_active":"2024-04-02T04:56:09.012765+0000","last_became_peered":"2024-04-02T04:56:09.012765+0000","last_unstale":"2024-04-02T04:56:19.919679+0000","last_undegraded":"2024-04-02T04:56:19.919679+0000","last_fullsized":"2024-04-02T04:56:19.919679+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:17.987106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"298.1b","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914191+0000","last_change":"2024-04-02T04:56:09.009750+0000","last_active":"2024-04-02T04:56:19.914191+0000","last_peered":"2024-04-02T04:56:19.914191+0000","last_clean":"2024-04-02T04:56:19.914191+0000","last_became_active":"2024-04-02T04:56:09.004281+0000","last_became_peered":"2024-04-02T04:56:09.004281+0000","last_unstale":"2024-04-02T04:56:19.914191+0000","last_undegraded":"2024-04-02T04:56:19.914191+0000","last_fullsized":"2024-04-02T04:56:19.914191+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:08.879988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.1a","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.444500+0000","last_change":"2024-04-02T04:56:08.990843+0000","last_active":"2024-04-02T04:56:27.444500+0000","last_peered":"2024-04-02T04:56:27.444500+0000","last_clean":"2024-04-02T04:56:27.444500+0000","last_became_active":"2024-04-02T04:56:08.989967+0000","last_became_peered":"2024-04-02T04:56:08.989967+0000","last_unstale":"2024-04-02T04:56:27.444500+0000","last_undegraded":"2024-04-02T04:56:27.444500+0000","last_fullsized":"2024-04-02T04:56:27.444500+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:32.740332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"298.19","version":"523'3","reported_seq":30,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264587+0000","last_change":"2024-04-02T04:56:09.000818+0000","last_active":"2024-04-02T04:56:19.264587+0000","last_peered":"2024-04-02T04:56:19.264587+0000","last_clean":"2024-04-02T04:56:19.264587+0000","last_became_active":"2024-04-02T04:56:09.000146+0000","last_became_peered":"2024-04-02T04:56:09.000146+0000","last_unstale":"2024-04-02T04:56:19.264587+0000","last_undegraded":"2024-04-02T04:56:19.264587+0000","last_fullsized":"2024-04-02T04:56:19.264587+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.915684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"298.18","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.544610+0000","last_change":"2024-04-02T04:56:09.004716+0000","last_active":"2024-04-02T04:56:27.544610+0000","last_peered":"2024-04-02T04:56:27.544610+0000","last_clean":"2024-04-02T04:56:27.544610+0000","last_became_active":"2024-04-02T04:56:09.004327+0000","last_became_peered":"2024-04-02T04:56:09.004327+0000","last_unstale":"2024-04-02T04:56:27.544610+0000","last_undegraded":"2024-04-02T04:56:27.544610+0000","last_fullsized":"2024-04-02T04:56:27.544610+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:12.222822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"298.17","version":"522'2","reported_seq":29,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922408+0000","last_change":"2024-04-02T04:56:09.014880+0000","last_active":"2024-04-02T04:56:19.922408+0000","last_peered":"2024-04-02T04:56:19.922408+0000","last_clean":"2024-04-02T04:56:19.922408+0000","last_became_active":"2024-04-02T04:56:09.014355+0000","last_became_peered":"2024-04-02T04:56:09.014355+0000","last_unstale":"2024-04-02T04:56:19.922408+0000","last_undegraded":"2024-04-02T04:56:19.922408+0000","last_fullsized":"2024-04-02T04:56:19.922408+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:34.511410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.16","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920784+0000","last_change":"2024-04-02T04:56:09.003645+0000","last_active":"2024-04-02T04:56:19.920784+0000","last_peered":"2024-04-02T04:56:19.920784+0000","last_clean":"2024-04-02T04:56:19.920784+0000","last_became_active":"2024-04-02T04:56:09.002725+0000","last_became_peered":"2024-04-02T04:56:09.002725+0000","last_unstale":"2024-04-02T04:56:19.920784+0000","last_undegraded":"2024-04-02T04:56:19.920784+0000","last_fullsized":"2024-04-02T04:56:19.920784+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:08.670203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.15","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925825+0000","last_change":"2024-04-02T04:56:09.016257+0000","last_active":"2024-04-02T04:56:19.925825+0000","last_peered":"2024-04-02T04:56:19.925825+0000","last_clean":"2024-04-02T04:56:19.925825+0000","last_became_active":"2024-04-02T04:56:09.014875+0000","last_became_peered":"2024-04-02T04:56:09.014875+0000","last_unstale":"2024-04-02T04:56:19.925825+0000","last_undegraded":"2024-04-02T04:56:19.925825+0000","last_fullsized":"2024-04-02T04:56:19.925825+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:35.646460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.14","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449679+0000","last_change":"2024-04-02T04:56:08.992725+0000","last_active":"2024-04-02T04:56:27.449679+0000","last_peered":"2024-04-02T04:56:27.449679+0000","last_clean":"2024-04-02T04:56:27.449679+0000","last_became_active":"2024-04-02T04:56:08.990707+0000","last_became_peered":"2024-04-02T04:56:08.990707+0000","last_unstale":"2024-04-02T04:56:27.449679+0000","last_undegraded":"2024-04-02T04:56:27.449679+0000","last_fullsized":"2024-04-02T04:56:27.449679+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:35.179360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"298.13","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545612+0000","last_change":"2024-04-02T04:56:09.006443+0000","last_active":"2024-04-02T04:56:27.545612+0000","last_peered":"2024-04-02T04:56:27.545612+0000","last_clean":"2024-04-02T04:56:27.545612+0000","last_became_active":"2024-04-02T04:56:09.006052+0000","last_became_peered":"2024-04-02T04:56:09.006052+0000","last_unstale":"2024-04-02T04:56:27.545612+0000","last_undegraded":"2024-04-02T04:56:27.545612+0000","last_fullsized":"2024-04-02T04:56:27.545612+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:20.443808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"298.12","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264754+0000","last_change":"2024-04-02T04:56:09.016917+0000","last_active":"2024-04-02T04:56:19.264754+0000","last_peered":"2024-04-02T04:56:19.264754+0000","last_clean":"2024-04-02T04:56:19.264754+0000","last_became_active":"2024-04-02T04:56:09.016542+0000","last_became_peered":"2024-04-02T04:56:09.016542+0000","last_unstale":"2024-04-02T04:56:19.264754+0000","last_undegraded":"2024-04-02T04:56:19.264754+0000","last_fullsized":"2024-04-02T04:56:19.264754+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:16.068661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"298.11","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448048+0000","last_change":"2024-04-02T04:56:08.989670+0000","last_active":"2024-04-02T04:56:27.448048+0000","last_peered":"2024-04-02T04:56:27.448048+0000","last_clean":"2024-04-02T04:56:27.448048+0000","last_became_active":"2024-04-02T04:56:08.989340+0000","last_became_peered":"2024-04-02T04:56:08.989340+0000","last_unstale":"2024-04-02T04:56:27.448048+0000","last_undegraded":"2024-04-02T04:56:27.448048+0000","last_fullsized":"2024-04-02T04:56:27.448048+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:26.770190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"298.10","version":"522'2","reported_seq":29,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.917062+0000","last_change":"2024-04-02T04:56:09.007322+0000","last_active":"2024-04-02T04:56:19.917062+0000","last_peered":"2024-04-02T04:56:19.917062+0000","last_clean":"2024-04-02T04:56:19.917062+0000","last_became_active":"2024-04-02T04:56:09.006907+0000","last_became_peered":"2024-04-02T04:56:09.006907+0000","last_unstale":"2024-04-02T04:56:19.917062+0000","last_undegraded":"2024-04-02T04:56:19.917062+0000","last_fullsized":"2024-04-02T04:56:19.917062+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:15.168585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"298.f","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.447100+0000","last_change":"2024-04-02T04:56:08.993286+0000","last_active":"2024-04-02T04:56:27.447100+0000","last_peered":"2024-04-02T04:56:27.447100+0000","last_clean":"2024-04-02T04:56:27.447100+0000","last_became_active":"2024-04-02T04:56:08.987195+0000","last_became_peered":"2024-04-02T04:56:08.987195+0000","last_unstale":"2024-04-02T04:56:27.447100+0000","last_undegraded":"2024-04-02T04:56:27.447100+0000","last_fullsized":"2024-04-02T04:56:27.447100+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:31.693357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"298.e","version":"529'3","reported_seq":32,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546765+0000","last_change":"2024-04-02T04:56:08.994913+0000","last_active":"2024-04-02T04:56:27.546765+0000","last_peered":"2024-04-02T04:56:27.546765+0000","last_clean":"2024-04-02T04:56:27.546765+0000","last_became_active":"2024-04-02T04:56:08.991231+0000","last_became_peered":"2024-04-02T04:56:08.991231+0000","last_unstale":"2024-04-02T04:56:27.546765+0000","last_undegraded":"2024-04-02T04:56:27.546765+0000","last_fullsized":"2024-04-02T04:56:27.546765+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:36.607436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"298.d","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920427+0000","last_change":"2024-04-02T04:56:09.014631+0000","last_active":"2024-04-02T04:56:19.920427+0000","last_peered":"2024-04-02T04:56:19.920427+0000","last_clean":"2024-04-02T04:56:19.920427+0000","last_became_active":"2024-04-02T04:56:09.011580+0000","last_became_peered":"2024-04-02T04:56:09.011580+0000","last_unstale":"2024-04-02T04:56:19.920427+0000","last_undegraded":"2024-04-02T04:56:19.920427+0000","last_fullsized":"2024-04-02T04:56:19.920427+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:16.453976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"298.c","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919810+0000","last_change":"2024-04-02T04:56:08.998873+0000","last_active":"2024-04-02T04:56:19.919810+0000","last_peered":"2024-04-02T04:56:19.919810+0000","last_clean":"2024-04-02T04:56:19.919810+0000","last_became_active":"2024-04-02T04:56:08.997033+0000","last_became_peered":"2024-04-02T04:56:08.997033+0000","last_unstale":"2024-04-02T04:56:19.919810+0000","last_undegraded":"2024-04-02T04:56:19.919810+0000","last_fullsized":"2024-04-02T04:56:19.919810+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:37.693223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"298.b","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919122+0000","last_change":"2024-04-02T04:56:09.004638+0000","last_active":"2024-04-02T04:56:19.919122+0000","last_peered":"2024-04-02T04:56:19.919122+0000","last_clean":"2024-04-02T04:56:19.919122+0000","last_became_active":"2024-04-02T04:56:09.003907+0000","last_became_peered":"2024-04-02T04:56:09.003907+0000","last_unstale":"2024-04-02T04:56:19.919122+0000","last_undegraded":"2024-04-02T04:56:19.919122+0000","last_fullsized":"2024-04-02T04:56:19.919122+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.678435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"298.a","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264750+0000","last_change":"2024-04-02T04:56:08.997407+0000","last_active":"2024-04-02T04:56:19.264750+0000","last_peered":"2024-04-02T04:56:19.264750+0000","last_clean":"2024-04-02T04:56:19.264750+0000","last_became_active":"2024-04-02T04:56:08.997102+0000","last_became_peered":"2024-04-02T04:56:08.997102+0000","last_unstale":"2024-04-02T04:56:19.264750+0000","last_undegraded":"2024-04-02T04:56:19.264750+0000","last_fullsized":"2024-04-02T04:56:19.264750+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:26.460626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"298.9","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.917710+0000","last_change":"2024-04-02T04:56:09.000419+0000","last_active":"2024-04-02T04:56:19.917710+0000","last_peered":"2024-04-02T04:56:19.917710+0000","last_clean":"2024-04-02T04:56:19.917710+0000","last_became_active":"2024-04-02T04:56:08.999700+0000","last_became_peered":"2024-04-02T04:56:08.999700+0000","last_unstale":"2024-04-02T04:56:19.917710+0000","last_undegraded":"2024-04-02T04:56:19.917710+0000","last_fullsized":"2024-04-02T04:56:19.917710+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:25.851175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"298.8","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.544703+0000","last_change":"2024-04-02T04:56:09.012666+0000","last_active":"2024-04-02T04:56:27.544703+0000","last_peered":"2024-04-02T04:56:27.544703+0000","last_clean":"2024-04-02T04:56:27.544703+0000","last_became_active":"2024-04-02T04:56:09.012316+0000","last_became_peered":"2024-04-02T04:56:09.012316+0000","last_unstale":"2024-04-02T04:56:27.544703+0000","last_undegraded":"2024-04-02T04:56:27.544703+0000","last_fullsized":"2024-04-02T04:56:27.544703+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:30.665864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"298.7","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450815+0000","last_change":"2024-04-02T04:56:08.989844+0000","last_active":"2024-04-02T04:56:27.450815+0000","last_peered":"2024-04-02T04:56:27.450815+0000","last_clean":"2024-04-02T04:56:27.450815+0000","last_became_active":"2024-04-02T04:56:08.989269+0000","last_became_peered":"2024-04-02T04:56:08.989269+0000","last_unstale":"2024-04-02T04:56:27.450815+0000","last_undegraded":"2024-04-02T04:56:27.450815+0000","last_fullsized":"2024-04-02T04:56:27.450815+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:21.869119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"298.6","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266720+0000","last_change":"2024-04-02T04:56:09.009990+0000","last_active":"2024-04-02T04:56:19.266720+0000","last_peered":"2024-04-02T04:56:19.266720+0000","last_clean":"2024-04-02T04:56:19.266720+0000","last_became_active":"2024-04-02T04:56:09.009628+0000","last_became_peered":"2024-04-02T04:56:09.009628+0000","last_unstale":"2024-04-02T04:56:19.266720+0000","last_undegraded":"2024-04-02T04:56:19.266720+0000","last_fullsized":"2024-04-02T04:56:19.266720+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:36.503185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"298.5","version":"0'0","reported_seq":29,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450444+0000","last_change":"2024-04-02T04:56:09.004334+0000","last_active":"2024-04-02T04:56:27.450444+0000","last_peered":"2024-04-02T04:56:27.450444+0000","last_clean":"2024-04-02T04:56:27.450444+0000","last_became_active":"2024-04-02T04:56:09.003519+0000","last_became_peered":"2024-04-02T04:56:09.003519+0000","last_unstale":"2024-04-02T04:56:27.450444+0000","last_undegraded":"2024-04-02T04:56:27.450444+0000","last_fullsized":"2024-04-02T04:56:27.450444+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:26.349269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"298.4","version":"529'3","reported_seq":32,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.446097+0000","last_change":"2024-04-02T04:56:08.993326+0000","last_active":"2024-04-02T04:56:27.446097+0000","last_peered":"2024-04-02T04:56:27.446097+0000","last_clean":"2024-04-02T04:56:27.446097+0000","last_became_active":"2024-04-02T04:56:08.992549+0000","last_became_peered":"2024-04-02T04:56:08.992549+0000","last_unstale":"2024-04-02T04:56:27.446097+0000","last_undegraded":"2024-04-02T04:56:27.446097+0000","last_fullsized":"2024-04-02T04:56:27.446097+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:09.803071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"298.3","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266707+0000","last_change":"2024-04-02T04:56:08.998413+0000","last_active":"2024-04-02T04:56:19.266707+0000","last_peered":"2024-04-02T04:56:19.266707+0000","last_clean":"2024-04-02T04:56:19.266707+0000","last_became_active":"2024-04-02T04:56:08.993187+0000","last_became_peered":"2024-04-02T04:56:08.993187+0000","last_unstale":"2024-04-02T04:56:19.266707+0000","last_undegraded":"2024-04-02T04:56:19.266707+0000","last_fullsized":"2024-04-02T04:56:19.266707+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:09.974223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"298.2","version":"529'3","reported_seq":32,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545019+0000","last_change":"2024-04-02T04:56:09.006270+0000","last_active":"2024-04-02T04:56:27.545019+0000","last_peered":"2024-04-02T04:56:27.545019+0000","last_clean":"2024-04-02T04:56:27.545019+0000","last_became_active":"2024-04-02T04:56:09.005907+0000","last_became_peered":"2024-04-02T04:56:09.005907+0000","last_unstale":"2024-04-02T04:56:27.545019+0000","last_undegraded":"2024-04-02T04:56:27.545019+0000","last_fullsized":"2024-04-02T04:56:27.545019+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:10.082364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"298.1","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264639+0000","last_change":"2024-04-02T04:56:09.015062+0000","last_active":"2024-04-02T04:56:19.264639+0000","last_peered":"2024-04-02T04:56:19.264639+0000","last_clean":"2024-04-02T04:56:19.264639+0000","last_became_active":"2024-04-02T04:56:09.014648+0000","last_became_peered":"2024-04-02T04:56:09.014648+0000","last_unstale":"2024-04-02T04:56:19.264639+0000","last_undegraded":"2024-04-02T04:56:19.264639+0000","last_fullsized":"2024-04-02T04:56:19.264639+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:22.656130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"298.0","version":"529'5","reported_seq":34,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.443710+0000","last_change":"2024-04-02T04:56:08.991148+0000","last_active":"2024-04-02T04:56:27.443710+0000","last_peered":"2024-04-02T04:56:27.443710+0000","last_clean":"2024-04-02T04:56:27.443710+0000","last_became_active":"2024-04-02T04:56:08.990334+0000","last_became_peered":"2024-04-02T04:56:08.990334+0000","last_unstale":"2024-04-02T04:56:27.443710+0000","last_undegraded":"2024-04-02T04:56:27.443710+0000","last_fullsized":"2024-04-02T04:56:27.443710+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:07.938112+0000","last_clean_scrub_stamp":"2024-04-02T04:56:07.938112+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:30.538616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"102.19","version":"423'86","reported_seq":458,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:21.948483+0000","last_change":"2024-04-02T04:56:21.948566+0000","last_active":"2024-04-02T04:56:21.948483+0000","last_peered":"2024-04-02T04:56:21.948483+0000","last_clean":"2024-04-02T04:56:21.948483+0000","last_became_active":"2024-04-02T04:53:54.919540+0000","last_became_peered":"2024-04-02T04:53:54.919540+0000","last_unstale":"2024-04-02T04:56:21.948483+0000","last_undegraded":"2024-04-02T04:56:21.948483+0000","last_fullsized":"2024-04-02T04:56:21.948483+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'86","last_scrub_stamp":"2024-04-02T04:56:21.948321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:56:21.948321+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:32.845278+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0076407940000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"102.18","version":"263'35","reported_seq":1040,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.544996+0000","last_change":"2024-04-02T04:56:02.083934+0000","last_active":"2024-04-02T04:56:27.544996+0000","last_peered":"2024-04-02T04:56:27.544996+0000","last_clean":"2024-04-02T04:56:27.544996+0000","last_became_active":"2024-04-02T04:56:02.029427+0000","last_became_peered":"2024-04-02T04:56:02.029427+0000","last_unstale":"2024-04-02T04:56:27.544996+0000","last_undegraded":"2024-04-02T04:56:27.544996+0000","last_fullsized":"2024-04-02T04:56:27.544996+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'35","last_scrub_stamp":"2024-04-02T04:54:29.951327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.951327+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021713000000000001,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.1b","version":"263'37","reported_seq":989,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450514+0000","last_change":"2024-04-02T04:56:12.449430+0000","last_active":"2024-04-02T04:56:27.450514+0000","last_peered":"2024-04-02T04:56:27.450514+0000","last_clean":"2024-04-02T04:56:27.450514+0000","last_became_active":"2024-04-02T04:49:56.568053+0000","last_became_peered":"2024-04-02T04:49:56.568053+0000","last_unstale":"2024-04-02T04:56:27.450514+0000","last_undegraded":"2024-04-02T04:56:27.450514+0000","last_fullsized":"2024-04-02T04:56:27.450514+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'37","last_scrub_stamp":"2024-04-02T04:56:12.449170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:56:12.449170+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.655748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0035464099999999998,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.1a","version":"449'76","reported_seq":1009,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545526+0000","last_change":"2024-04-02T04:55:47.953849+0000","last_active":"2024-04-02T04:56:27.545526+0000","last_peered":"2024-04-02T04:56:27.545526+0000","last_clean":"2024-04-02T04:56:27.545526+0000","last_became_active":"2024-04-02T04:49:56.571944+0000","last_became_peered":"2024-04-02T04:49:56.571944+0000","last_unstale":"2024-04-02T04:56:27.545526+0000","last_undegraded":"2024-04-02T04:56:27.545526+0000","last_fullsized":"2024-04-02T04:56:27.545526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'73","last_scrub_stamp":"2024-04-02T04:54:17.944946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.944946+0000","objects_scrubbed":5,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00150468,"stat_sum":{"num_bytes":62,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.1d","version":"518'90","reported_seq":419,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.923762+0000","last_change":"2024-04-02T04:56:02.104164+0000","last_active":"2024-04-02T04:56:19.923762+0000","last_peered":"2024-04-02T04:56:19.923762+0000","last_clean":"2024-04-02T04:56:19.923762+0000","last_became_active":"2024-04-02T04:56:02.016805+0000","last_became_peered":"2024-04-02T04:56:02.016805+0000","last_unstale":"2024-04-02T04:56:19.923762+0000","last_undegraded":"2024-04-02T04:56:19.923762+0000","last_fullsized":"2024-04-02T04:56:19.923762+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'85","last_scrub_stamp":"2024-04-02T04:52:51.767218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.767218+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.060092379000000001,"stat_sum":{"num_bytes":65,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.1c","version":"393'35","reported_seq":1013,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546085+0000","last_change":"2024-04-02T04:56:02.111070+0000","last_active":"2024-04-02T04:56:27.546085+0000","last_peered":"2024-04-02T04:56:27.546085+0000","last_clean":"2024-04-02T04:56:27.546085+0000","last_became_active":"2024-04-02T04:56:02.042582+0000","last_became_peered":"2024-04-02T04:56:02.042582+0000","last_unstale":"2024-04-02T04:56:27.546085+0000","last_undegraded":"2024-04-02T04:56:27.546085+0000","last_fullsized":"2024-04-02T04:56:27.546085+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-04-02T04:54:09.841478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:09.841478+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0036239279999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.1f","version":"443'55","reported_seq":1051,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452103+0000","last_change":"2024-04-02T04:56:02.056736+0000","last_active":"2024-04-02T04:56:27.452103+0000","last_peered":"2024-04-02T04:56:27.452103+0000","last_clean":"2024-04-02T04:56:27.452103+0000","last_became_active":"2024-04-02T04:56:01.996315+0000","last_became_peered":"2024-04-02T04:56:01.996315+0000","last_unstale":"2024-04-02T04:56:27.452103+0000","last_undegraded":"2024-04-02T04:56:27.452103+0000","last_fullsized":"2024-04-02T04:56:27.452103+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'55","last_scrub_stamp":"2024-04-02T04:55:11.979935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:55:11.979935+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003935086,"stat_sum":{"num_bytes":71,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":71,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"102.1e","version":"248'17","reported_seq":984,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.547110+0000","last_change":"2024-04-02T04:56:02.115978+0000","last_active":"2024-04-02T04:56:27.547110+0000","last_peered":"2024-04-02T04:56:27.547110+0000","last_clean":"2024-04-02T04:56:27.547110+0000","last_became_active":"2024-04-02T04:56:02.048184+0000","last_became_peered":"2024-04-02T04:56:02.048184+0000","last_unstale":"2024-04-02T04:56:27.547110+0000","last_undegraded":"2024-04-02T04:56:27.547110+0000","last_fullsized":"2024-04-02T04:56:27.547110+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:54:34.920018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.920018+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0027788499999999998,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.11","version":"419'91","reported_seq":1092,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545077+0000","last_change":"2024-04-02T04:55:47.960672+0000","last_active":"2024-04-02T04:56:27.545077+0000","last_peered":"2024-04-02T04:56:27.545077+0000","last_clean":"2024-04-02T04:56:27.545077+0000","last_became_active":"2024-04-02T04:53:54.924990+0000","last_became_peered":"2024-04-02T04:53:54.924990+0000","last_unstale":"2024-04-02T04:56:27.545077+0000","last_undegraded":"2024-04-02T04:56:27.545077+0000","last_fullsized":"2024-04-02T04:56:27.545077+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'91","last_scrub_stamp":"2024-04-02T04:54:22.972425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:22.972425+0000","objects_scrubbed":7,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044624230000000001,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.10","version":"483'56","reported_seq":1022,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264485+0000","last_change":"2024-04-02T04:56:12.050422+0000","last_active":"2024-04-02T04:56:19.264485+0000","last_peered":"2024-04-02T04:56:19.264485+0000","last_clean":"2024-04-02T04:56:19.264485+0000","last_became_active":"2024-04-02T04:53:54.956913+0000","last_became_peered":"2024-04-02T04:53:54.956913+0000","last_unstale":"2024-04-02T04:56:19.264485+0000","last_undegraded":"2024-04-02T04:56:19.264485+0000","last_fullsized":"2024-04-02T04:56:19.264485+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'56","last_scrub_stamp":"2024-04-02T04:56:12.050253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:56:12.050253+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:21.629128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049466409999999999,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.13","version":"529'117","reported_seq":1108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450578+0000","last_change":"2024-04-02T04:55:47.987480+0000","last_active":"2024-04-02T04:56:27.450578+0000","last_peered":"2024-04-02T04:56:27.450578+0000","last_clean":"2024-04-02T04:56:27.450578+0000","last_became_active":"2024-04-02T04:53:54.943248+0000","last_became_peered":"2024-04-02T04:53:54.943248+0000","last_unstale":"2024-04-02T04:56:27.450578+0000","last_undegraded":"2024-04-02T04:56:27.450578+0000","last_fullsized":"2024-04-02T04:56:27.450578+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'92","last_scrub_stamp":"2024-04-02T04:54:34.165272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.165272+0000","objects_scrubbed":6,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0040958289999999996,"stat_sum":{"num_bytes":94,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":41,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.12","version":"449'21","reported_seq":994,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:23.025277+0000","last_change":"2024-04-02T04:56:23.025317+0000","last_active":"2024-04-02T04:56:23.025277+0000","last_peered":"2024-04-02T04:56:23.025277+0000","last_clean":"2024-04-02T04:56:23.025277+0000","last_became_active":"2024-04-02T04:56:02.034929+0000","last_became_peered":"2024-04-02T04:56:02.034929+0000","last_unstale":"2024-04-02T04:56:23.025277+0000","last_undegraded":"2024-04-02T04:56:23.025277+0000","last_fullsized":"2024-04-02T04:56:23.025277+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'21","last_scrub_stamp":"2024-04-02T04:56:23.025185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:56:23.025185+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:49.806740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013016219999999999,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.15","version":"500'85","reported_seq":1064,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452096+0000","last_change":"2024-04-02T04:55:47.993740+0000","last_active":"2024-04-02T04:56:27.452096+0000","last_peered":"2024-04-02T04:56:27.452096+0000","last_clean":"2024-04-02T04:56:27.452096+0000","last_became_active":"2024-04-02T04:53:54.955612+0000","last_became_peered":"2024-04-02T04:53:54.955612+0000","last_unstale":"2024-04-02T04:56:27.452096+0000","last_undegraded":"2024-04-02T04:56:27.452096+0000","last_fullsized":"2024-04-02T04:56:27.452096+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'69","last_scrub_stamp":"2024-04-02T04:55:10.262637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:55:10.262637+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0063899059999999999,"stat_sum":{"num_bytes":70,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.14","version":"350'154","reported_seq":1074,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920734+0000","last_change":"2024-04-02T04:56:04.964577+0000","last_active":"2024-04-02T04:56:19.920734+0000","last_peered":"2024-04-02T04:56:19.920734+0000","last_clean":"2024-04-02T04:56:19.920734+0000","last_became_active":"2024-04-02T04:56:04.963818+0000","last_became_peered":"2024-04-02T04:56:04.963818+0000","last_unstale":"2024-04-02T04:56:19.920734+0000","last_undegraded":"2024-04-02T04:56:19.920734+0000","last_fullsized":"2024-04-02T04:56:19.920734+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'153","last_scrub_stamp":"2024-04-02T04:52:34.712599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.712599+0000","objects_scrubbed":9,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0073890359999999999,"stat_sum":{"num_bytes":4000107,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":46,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":8000214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.17","version":"393'36","reported_seq":986,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.547501+0000","last_change":"2024-04-02T04:55:47.960824+0000","last_active":"2024-04-02T04:56:27.547501+0000","last_peered":"2024-04-02T04:56:27.547501+0000","last_clean":"2024-04-02T04:56:27.547501+0000","last_became_active":"2024-04-02T04:49:56.565251+0000","last_became_peered":"2024-04-02T04:49:56.565251+0000","last_unstale":"2024-04-02T04:56:27.547501+0000","last_undegraded":"2024-04-02T04:56:27.547501+0000","last_fullsized":"2024-04-02T04:56:27.547501+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'36","last_scrub_stamp":"2024-04-02T04:54:24.021811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.021811+0000","objects_scrubbed":4,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019823029999999999,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.16","version":"393'18","reported_seq":862,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924553+0000","last_change":"2024-04-02T04:55:47.951771+0000","last_active":"2024-04-02T04:56:19.924553+0000","last_peered":"2024-04-02T04:56:19.924553+0000","last_clean":"2024-04-02T04:56:19.924553+0000","last_became_active":"2024-04-02T04:54:56.013487+0000","last_became_peered":"2024-04-02T04:54:56.013487+0000","last_unstale":"2024-04-02T04:56:19.924553+0000","last_undegraded":"2024-04-02T04:56:19.924553+0000","last_fullsized":"2024-04-02T04:56:19.924553+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'17","last_scrub_stamp":"2024-04-02T04:52:38.664090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.664090+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039549629999999997,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.9","version":"401'37","reported_seq":1007,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.544978+0000","last_change":"2024-04-02T04:55:47.959954+0000","last_active":"2024-04-02T04:56:27.544978+0000","last_peered":"2024-04-02T04:56:27.544978+0000","last_clean":"2024-04-02T04:56:27.544978+0000","last_became_active":"2024-04-02T04:53:54.924508+0000","last_became_peered":"2024-04-02T04:53:54.924508+0000","last_unstale":"2024-04-02T04:56:27.544978+0000","last_undegraded":"2024-04-02T04:56:27.544978+0000","last_fullsized":"2024-04-02T04:56:27.544978+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'37","last_scrub_stamp":"2024-04-02T04:54:07.832705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.832705+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0041008490000000002,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"102.8","version":"248'17","reported_seq":977,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919417+0000","last_change":"2024-04-02T04:56:02.097847+0000","last_active":"2024-04-02T04:56:19.919417+0000","last_peered":"2024-04-02T04:56:19.919417+0000","last_clean":"2024-04-02T04:56:19.919417+0000","last_became_active":"2024-04-02T04:56:02.016649+0000","last_became_peered":"2024-04-02T04:56:02.016649+0000","last_unstale":"2024-04-02T04:56:19.919417+0000","last_undegraded":"2024-04-02T04:56:19.919417+0000","last_fullsized":"2024-04-02T04:56:19.919417+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'17","last_scrub_stamp":"2024-04-02T04:54:11.954373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:11.954373+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014192919999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.b","version":"393'75","reported_seq":943,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266622+0000","last_change":"2024-04-02T04:55:47.951680+0000","last_active":"2024-04-02T04:56:19.266622+0000","last_peered":"2024-04-02T04:56:19.266622+0000","last_clean":"2024-04-02T04:56:19.266622+0000","last_became_active":"2024-04-02T04:54:55.989598+0000","last_became_peered":"2024-04-02T04:54:55.989598+0000","last_unstale":"2024-04-02T04:56:19.266622+0000","last_undegraded":"2024-04-02T04:56:19.266622+0000","last_fullsized":"2024-04-02T04:56:19.266622+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'73","last_scrub_stamp":"2024-04-02T04:52:33.669178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.669178+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00080097999999999996,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.a","version":"254'68","reported_seq":604,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265198+0000","last_change":"2024-04-02T04:56:02.125995+0000","last_active":"2024-04-02T04:56:19.265198+0000","last_peered":"2024-04-02T04:56:19.265198+0000","last_clean":"2024-04-02T04:56:19.265198+0000","last_became_active":"2024-04-02T04:56:02.036389+0000","last_became_peered":"2024-04-02T04:56:02.036389+0000","last_unstale":"2024-04-02T04:56:19.265198+0000","last_undegraded":"2024-04-02T04:56:19.265198+0000","last_fullsized":"2024-04-02T04:56:19.265198+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'68","last_scrub_stamp":"2024-04-02T04:52:43.653347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.653347+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001735779,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.d","version":"434'130","reported_seq":493,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921371+0000","last_change":"2024-04-02T04:55:48.000331+0000","last_active":"2024-04-02T04:56:19.921371+0000","last_peered":"2024-04-02T04:56:19.921371+0000","last_clean":"2024-04-02T04:56:19.921371+0000","last_became_active":"2024-04-02T04:53:54.929671+0000","last_became_peered":"2024-04-02T04:53:54.929671+0000","last_unstale":"2024-04-02T04:56:19.921371+0000","last_undegraded":"2024-04-02T04:56:19.921371+0000","last_fullsized":"2024-04-02T04:56:19.921371+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:49:53.797011+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046557819999999998,"stat_sum":{"num_bytes":95,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"102.c","version":"249'68","reported_seq":1045,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922938+0000","last_change":"2024-04-02T04:56:02.086019+0000","last_active":"2024-04-02T04:56:19.922938+0000","last_peered":"2024-04-02T04:56:19.922938+0000","last_clean":"2024-04-02T04:56:19.922938+0000","last_became_active":"2024-04-02T04:56:02.012789+0000","last_became_peered":"2024-04-02T04:56:02.012789+0000","last_unstale":"2024-04-02T04:56:19.922938+0000","last_undegraded":"2024-04-02T04:56:19.922938+0000","last_fullsized":"2024-04-02T04:56:19.922938+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'68","last_scrub_stamp":"2024-04-02T04:53:54.821023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.821023+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048449239999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.f","version":"393'63","reported_seq":1053,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.916835+0000","last_change":"2024-04-02T04:55:47.950380+0000","last_active":"2024-04-02T04:56:19.916835+0000","last_peered":"2024-04-02T04:56:19.916835+0000","last_clean":"2024-04-02T04:56:19.916835+0000","last_became_active":"2024-04-02T04:49:55.259293+0000","last_became_peered":"2024-04-02T04:49:55.259293+0000","last_unstale":"2024-04-02T04:56:19.916835+0000","last_undegraded":"2024-04-02T04:56:19.916835+0000","last_fullsized":"2024-04-02T04:56:19.916835+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'60","last_scrub_stamp":"2024-04-02T04:53:14.447906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:53:14.447906+0000","objects_scrubbed":8,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003397069,"stat_sum":{"num_bytes":4194378,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":8214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.e","version":"425'57","reported_seq":1011,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451542+0000","last_change":"2024-04-02T04:55:47.989740+0000","last_active":"2024-04-02T04:56:27.451542+0000","last_peered":"2024-04-02T04:56:27.451542+0000","last_clean":"2024-04-02T04:56:27.451542+0000","last_became_active":"2024-04-02T04:49:56.570661+0000","last_became_peered":"2024-04-02T04:49:56.570661+0000","last_unstale":"2024-04-02T04:56:27.451542+0000","last_undegraded":"2024-04-02T04:56:27.451542+0000","last_fullsized":"2024-04-02T04:56:27.451542+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'57","last_scrub_stamp":"2024-04-02T04:55:30.517100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:55:30.517100+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:44.056285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00282044,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.1","version":"453'60","reported_seq":436,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:23.933550+0000","last_change":"2024-04-02T04:56:23.933628+0000","last_active":"2024-04-02T04:56:23.933550+0000","last_peered":"2024-04-02T04:56:23.933550+0000","last_clean":"2024-04-02T04:56:23.933550+0000","last_became_active":"2024-04-02T04:53:54.923616+0000","last_became_peered":"2024-04-02T04:53:54.923616+0000","last_unstale":"2024-04-02T04:56:23.933550+0000","last_undegraded":"2024-04-02T04:56:23.933550+0000","last_fullsized":"2024-04-02T04:56:23.933550+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'60","last_scrub_stamp":"2024-04-02T04:56:23.933390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:56:23.933390+0000","objects_scrubbed":6,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:36.658970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0023591010000000002,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"102.0","version":"446'101","reported_seq":1096,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.917796+0000","last_change":"2024-04-02T04:55:47.942698+0000","last_active":"2024-04-02T04:56:19.917796+0000","last_peered":"2024-04-02T04:56:19.917796+0000","last_clean":"2024-04-02T04:56:19.917796+0000","last_became_active":"2024-04-02T04:54:58.978036+0000","last_became_peered":"2024-04-02T04:54:58.978036+0000","last_unstale":"2024-04-02T04:56:19.917796+0000","last_undegraded":"2024-04-02T04:56:19.917796+0000","last_fullsized":"2024-04-02T04:56:19.917796+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'95","last_scrub_stamp":"2024-04-02T04:54:12.940358+0000","last_deep_scrub":"412'95","last_deep_scrub_stamp":"2024-04-02T04:54:12.940358+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.940358+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.005127169,"stat_sum":{"num_bytes":70,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"102.3","version":"393'35","reported_seq":1010,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265810+0000","last_change":"2024-04-02T04:56:13.074367+0000","last_active":"2024-04-02T04:56:19.265810+0000","last_peered":"2024-04-02T04:56:19.265810+0000","last_clean":"2024-04-02T04:56:19.265810+0000","last_became_active":"2024-04-02T04:54:55.988749+0000","last_became_peered":"2024-04-02T04:54:55.988749+0000","last_unstale":"2024-04-02T04:56:19.265810+0000","last_undegraded":"2024-04-02T04:56:19.265810+0000","last_fullsized":"2024-04-02T04:56:19.265810+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-04-02T04:56:13.074247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:56:13.074247+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:21.971042+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0019351329999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.2","version":"453'41","reported_seq":1013,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265320+0000","last_change":"2024-04-02T04:56:02.128543+0000","last_active":"2024-04-02T04:56:19.265320+0000","last_peered":"2024-04-02T04:56:19.265320+0000","last_clean":"2024-04-02T04:56:19.265320+0000","last_became_active":"2024-04-02T04:56:02.035534+0000","last_became_peered":"2024-04-02T04:56:02.035534+0000","last_unstale":"2024-04-02T04:56:19.265320+0000","last_undegraded":"2024-04-02T04:56:19.265320+0000","last_fullsized":"2024-04-02T04:56:19.265320+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'39","last_scrub_stamp":"2024-04-02T04:52:41.329212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.329212+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017237680000000001,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.5","version":"229'85","reported_seq":1085,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450620+0000","last_change":"2024-04-02T04:55:47.987698+0000","last_active":"2024-04-02T04:56:27.450620+0000","last_peered":"2024-04-02T04:56:27.450620+0000","last_clean":"2024-04-02T04:56:27.450620+0000","last_became_active":"2024-04-02T04:54:55.991571+0000","last_became_peered":"2024-04-02T04:54:55.991571+0000","last_unstale":"2024-04-02T04:56:27.450620+0000","last_undegraded":"2024-04-02T04:56:27.450620+0000","last_fullsized":"2024-04-02T04:56:27.450620+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'85","last_scrub_stamp":"2024-04-02T04:55:35.372910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:55:35.372910+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:47.776409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0040053700000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.4","version":"446'20","reported_seq":1000,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450193+0000","last_change":"2024-04-02T04:55:47.990955+0000","last_active":"2024-04-02T04:56:27.450193+0000","last_peered":"2024-04-02T04:56:27.450193+0000","last_clean":"2024-04-02T04:56:27.450193+0000","last_became_active":"2024-04-02T04:54:55.979745+0000","last_became_peered":"2024-04-02T04:54:55.979745+0000","last_unstale":"2024-04-02T04:56:27.450193+0000","last_undegraded":"2024-04-02T04:56:27.450193+0000","last_fullsized":"2024-04-02T04:56:27.450193+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'18","last_scrub_stamp":"2024-04-02T04:54:32.190177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:54:32.190177+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.001175094,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":71,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"102.7","version":"393'19","reported_seq":960,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.268038+0000","last_change":"2024-04-02T04:55:47.961348+0000","last_active":"2024-04-02T04:56:19.268038+0000","last_peered":"2024-04-02T04:56:19.268038+0000","last_clean":"2024-04-02T04:56:19.268038+0000","last_became_active":"2024-04-02T04:54:56.004707+0000","last_became_peered":"2024-04-02T04:54:56.004707+0000","last_unstale":"2024-04-02T04:56:19.268038+0000","last_undegraded":"2024-04-02T04:56:19.268038+0000","last_fullsized":"2024-04-02T04:56:19.268038+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'18","last_scrub_stamp":"2024-04-02T04:52:40.288509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.288509+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00071136599999999995,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":47,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"102.6","version":"425'35","reported_seq":801,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924449+0000","last_change":"2024-04-02T04:56:02.130489+0000","last_active":"2024-04-02T04:56:19.924449+0000","last_peered":"2024-04-02T04:56:19.924449+0000","last_clean":"2024-04-02T04:56:19.924449+0000","last_became_active":"2024-04-02T04:56:02.018845+0000","last_became_peered":"2024-04-02T04:56:02.018845+0000","last_unstale":"2024-04-02T04:56:19.924449+0000","last_undegraded":"2024-04-02T04:56:19.924449+0000","last_fullsized":"2024-04-02T04:56:19.924449+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-02T04:52:53.769683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:53.797011+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.769683+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003762409,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.14","version":"0'0","reported_seq":676,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921042+0000","last_change":"2024-04-02T04:56:02.016378+0000","last_active":"2024-04-02T04:56:19.921042+0000","last_peered":"2024-04-02T04:56:19.921042+0000","last_clean":"2024-04-02T04:56:19.921042+0000","last_became_active":"2024-04-02T04:56:02.015898+0000","last_became_peered":"2024-04-02T04:56:02.015898+0000","last_unstale":"2024-04-02T04:56:19.921042+0000","last_undegraded":"2024-04-02T04:56:19.921042+0000","last_fullsized":"2024-04-02T04:56:19.921042+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:44.775672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:44.775672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":800,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452521+0000","last_change":"2024-04-02T04:52:58.345528+0000","last_active":"2024-04-02T04:56:27.452521+0000","last_peered":"2024-04-02T04:56:27.452521+0000","last_clean":"2024-04-02T04:56:27.452521+0000","last_became_active":"2024-04-02T04:49:28.771136+0000","last_became_peered":"2024-04-02T04:49:28.771136+0000","last_unstale":"2024-04-02T04:56:27.452521+0000","last_undegraded":"2024-04-02T04:56:27.452521+0000","last_fullsized":"2024-04-02T04:56:27.452521+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:58.345234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.345234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.15","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.446687+0000","last_change":"2024-04-02T04:56:01.993032+0000","last_active":"2024-04-02T04:56:27.446687+0000","last_peered":"2024-04-02T04:56:27.446687+0000","last_clean":"2024-04-02T04:56:27.446687+0000","last_became_active":"2024-04-02T04:56:01.992516+0000","last_became_peered":"2024-04-02T04:56:01.992516+0000","last_unstale":"2024-04-02T04:56:27.446687+0000","last_undegraded":"2024-04-02T04:56:27.446687+0000","last_fullsized":"2024-04-02T04:56:27.446687+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:46.284019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:51:46.284019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":810,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:26.999743+0000","last_change":"2024-04-02T04:56:26.999785+0000","last_active":"2024-04-02T04:56:26.999743+0000","last_peered":"2024-04-02T04:56:26.999743+0000","last_clean":"2024-04-02T04:56:26.999743+0000","last_became_active":"2024-04-02T04:56:02.040684+0000","last_became_peered":"2024-04-02T04:56:02.040684+0000","last_unstale":"2024-04-02T04:56:26.999743+0000","last_undegraded":"2024-04-02T04:56:26.999743+0000","last_fullsized":"2024-04-02T04:56:26.999743+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:26.999638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:56:26.999638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:55.817304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.16","version":"0'0","reported_seq":566,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.547145+0000","last_change":"2024-04-02T04:56:02.040233+0000","last_active":"2024-04-02T04:56:27.547145+0000","last_peered":"2024-04-02T04:56:27.547145+0000","last_clean":"2024-04-02T04:56:27.547145+0000","last_became_active":"2024-04-02T04:56:02.035312+0000","last_became_peered":"2024-04-02T04:56:02.035312+0000","last_unstale":"2024-04-02T04:56:27.547145+0000","last_undegraded":"2024-04-02T04:56:27.547145+0000","last_fullsized":"2024-04-02T04:56:27.547145+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:23.115295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:23.115295+0000","last_clean_scrub_stamp":"2024-04-02T04:54:23.115295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":805,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267122+0000","last_change":"2024-04-02T04:56:16.028032+0000","last_active":"2024-04-02T04:56:19.267122+0000","last_peered":"2024-04-02T04:56:19.267122+0000","last_clean":"2024-04-02T04:56:19.267122+0000","last_became_active":"2024-04-02T04:53:54.940561+0000","last_became_peered":"2024-04-02T04:53:54.940561+0000","last_unstale":"2024-04-02T04:56:19.267122+0000","last_undegraded":"2024-04-02T04:56:19.267122+0000","last_fullsized":"2024-04-02T04:56:19.267122+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:16.027884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.027884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:41.615275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.17","version":"0'0","reported_seq":305,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:20.913556+0000","last_change":"2024-04-02T04:56:20.913606+0000","last_active":"2024-04-02T04:56:20.913556+0000","last_peered":"2024-04-02T04:56:20.913556+0000","last_clean":"2024-04-02T04:56:20.913556+0000","last_became_active":"2024-04-02T04:56:02.031007+0000","last_became_peered":"2024-04-02T04:56:02.031007+0000","last_unstale":"2024-04-02T04:56:20.913556+0000","last_undegraded":"2024-04-02T04:56:20.913556+0000","last_fullsized":"2024-04-02T04:56:20.913556+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:20.913442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:56:20.913442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:45.299935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":794,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.923412+0000","last_change":"2024-04-02T04:53:07.468104+0000","last_active":"2024-04-02T04:56:19.923412+0000","last_peered":"2024-04-02T04:56:19.923412+0000","last_clean":"2024-04-02T04:56:19.923412+0000","last_became_active":"2024-04-02T04:49:28.761058+0000","last_became_peered":"2024-04-02T04:49:28.761058+0000","last_unstale":"2024-04-02T04:56:19.923412+0000","last_undegraded":"2024-04-02T04:56:19.923412+0000","last_fullsized":"2024-04-02T04:56:19.923412+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.467848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.467848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.10","version":"0'0","reported_seq":55,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.447847+0000","last_change":"2024-04-02T04:56:22.460613+0000","last_active":"2024-04-02T04:56:27.447847+0000","last_peered":"2024-04-02T04:56:27.447847+0000","last_clean":"2024-04-02T04:56:27.447847+0000","last_became_active":"2024-04-02T04:56:02.051336+0000","last_became_peered":"2024-04-02T04:56:02.051336+0000","last_unstale":"2024-04-02T04:56:27.447847+0000","last_undegraded":"2024-04-02T04:56:27.447847+0000","last_fullsized":"2024-04-02T04:56:27.447847+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:22.460510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:56:22.460510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:25.176059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":678,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448850+0000","last_change":"2024-04-02T04:56:01.992839+0000","last_active":"2024-04-02T04:56:27.448850+0000","last_peered":"2024-04-02T04:56:27.448850+0000","last_clean":"2024-04-02T04:56:27.448850+0000","last_became_active":"2024-04-02T04:56:01.992381+0000","last_became_peered":"2024-04-02T04:56:01.992381+0000","last_unstale":"2024-04-02T04:56:27.448850+0000","last_undegraded":"2024-04-02T04:56:27.448850+0000","last_fullsized":"2024-04-02T04:56:27.448850+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:35.787733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:35.787733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.11","version":"0'0","reported_seq":175,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918300+0000","last_change":"2024-04-02T04:54:56.002495+0000","last_active":"2024-04-02T04:56:19.918300+0000","last_peered":"2024-04-02T04:56:19.918300+0000","last_clean":"2024-04-02T04:56:19.918300+0000","last_became_active":"2024-04-02T04:54:56.002212+0000","last_became_peered":"2024-04-02T04:54:56.002212+0000","last_unstale":"2024-04-02T04:56:19.918300+0000","last_undegraded":"2024-04-02T04:56:19.918300+0000","last_fullsized":"2024-04-02T04:56:19.918300+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.684685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.684685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448854+0000","last_change":"2024-04-02T04:54:55.969224+0000","last_active":"2024-04-02T04:56:27.448854+0000","last_peered":"2024-04-02T04:56:27.448854+0000","last_clean":"2024-04-02T04:56:27.448854+0000","last_became_active":"2024-04-02T04:54:55.968826+0000","last_became_peered":"2024-04-02T04:54:55.968826+0000","last_unstale":"2024-04-02T04:56:27.448854+0000","last_undegraded":"2024-04-02T04:56:27.448854+0000","last_fullsized":"2024-04-02T04:56:27.448854+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:34.326885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:34.326885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.12","version":"0'0","reported_seq":800,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449666+0000","last_change":"2024-04-02T04:52:36.311512+0000","last_active":"2024-04-02T04:56:27.449666+0000","last_peered":"2024-04-02T04:56:27.449666+0000","last_clean":"2024-04-02T04:56:27.449666+0000","last_became_active":"2024-04-02T04:49:28.741617+0000","last_became_peered":"2024-04-02T04:49:28.741617+0000","last_unstale":"2024-04-02T04:56:27.449666+0000","last_undegraded":"2024-04-02T04:56:27.449666+0000","last_fullsized":"2024-04-02T04:56:27.449666+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:36.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:36.311307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":821,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545762+0000","last_change":"2024-04-02T04:56:14.217879+0000","last_active":"2024-04-02T04:56:27.545762+0000","last_peered":"2024-04-02T04:56:27.545762+0000","last_clean":"2024-04-02T04:56:27.545762+0000","last_became_active":"2024-04-02T04:54:55.978095+0000","last_became_peered":"2024-04-02T04:54:55.978095+0000","last_unstale":"2024-04-02T04:56:27.545762+0000","last_undegraded":"2024-04-02T04:56:27.545762+0000","last_fullsized":"2024-04-02T04:56:27.545762+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:14.217796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:56:14.217796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:18.636319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.13","version":"0'0","reported_seq":809,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450854+0000","last_change":"2024-04-02T04:54:11.293681+0000","last_active":"2024-04-02T04:56:27.450854+0000","last_peered":"2024-04-02T04:56:27.450854+0000","last_clean":"2024-04-02T04:56:27.450854+0000","last_became_active":"2024-04-02T04:53:54.942156+0000","last_became_peered":"2024-04-02T04:53:54.942156+0000","last_unstale":"2024-04-02T04:56:27.450854+0000","last_undegraded":"2024-04-02T04:56:27.450854+0000","last_fullsized":"2024-04-02T04:56:27.450854+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:11.293564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:11.293564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":697,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545029+0000","last_change":"2024-04-02T04:54:55.978517+0000","last_active":"2024-04-02T04:56:27.545029+0000","last_peered":"2024-04-02T04:56:27.545029+0000","last_clean":"2024-04-02T04:56:27.545029+0000","last_became_active":"2024-04-02T04:54:55.978164+0000","last_became_peered":"2024-04-02T04:54:55.978164+0000","last_unstale":"2024-04-02T04:56:27.545029+0000","last_undegraded":"2024-04-02T04:56:27.545029+0000","last_fullsized":"2024-04-02T04:56:27.545029+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:31.677846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:31.677846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.1c","version":"0'0","reported_seq":807,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.923874+0000","last_change":"2024-04-02T04:56:02.020410+0000","last_active":"2024-04-02T04:56:19.923874+0000","last_peered":"2024-04-02T04:56:19.923874+0000","last_clean":"2024-04-02T04:56:19.923874+0000","last_became_active":"2024-04-02T04:56:02.019889+0000","last_became_peered":"2024-04-02T04:56:02.019889+0000","last_unstale":"2024-04-02T04:56:19.923874+0000","last_undegraded":"2024-04-02T04:56:19.923874+0000","last_fullsized":"2024-04-02T04:56:19.923874+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:37.967201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:37.967201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:04.439660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":805,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267977+0000","last_change":"2024-04-02T04:55:17.030696+0000","last_active":"2024-04-02T04:56:19.267977+0000","last_peered":"2024-04-02T04:56:19.267977+0000","last_clean":"2024-04-02T04:56:19.267977+0000","last_became_active":"2024-04-02T04:53:54.955334+0000","last_became_peered":"2024-04-02T04:53:54.955334+0000","last_unstale":"2024-04-02T04:56:19.267977+0000","last_undegraded":"2024-04-02T04:56:19.267977+0000","last_fullsized":"2024-04-02T04:56:19.267977+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.030546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.030546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:19.964451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1d","version":"0'0","reported_seq":809,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449651+0000","last_change":"2024-04-02T04:55:20.982762+0000","last_active":"2024-04-02T04:56:27.449651+0000","last_peered":"2024-04-02T04:56:27.449651+0000","last_clean":"2024-04-02T04:56:27.449651+0000","last_became_active":"2024-04-02T04:53:54.938792+0000","last_became_peered":"2024-04-02T04:53:54.938792+0000","last_unstale":"2024-04-02T04:56:27.449651+0000","last_undegraded":"2024-04-02T04:56:27.449651+0000","last_fullsized":"2024-04-02T04:56:27.449651+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:20.982539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:20.982539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:27.782777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":800,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.927800+0000","last_change":"2024-04-02T04:54:55.972702+0000","last_active":"2024-04-02T04:56:19.927800+0000","last_peered":"2024-04-02T04:56:19.927800+0000","last_clean":"2024-04-02T04:56:19.927800+0000","last_became_active":"2024-04-02T04:54:55.972111+0000","last_became_peered":"2024-04-02T04:54:55.972111+0000","last_unstale":"2024-04-02T04:56:19.927800+0000","last_undegraded":"2024-04-02T04:56:19.927800+0000","last_fullsized":"2024-04-02T04:56:19.927800+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.077908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.077908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.1e","version":"0'0","reported_seq":801,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448449+0000","last_change":"2024-04-02T04:54:13.639181+0000","last_active":"2024-04-02T04:56:27.448449+0000","last_peered":"2024-04-02T04:56:27.448449+0000","last_clean":"2024-04-02T04:56:27.448449+0000","last_became_active":"2024-04-02T04:49:28.785272+0000","last_became_peered":"2024-04-02T04:49:28.785272+0000","last_unstale":"2024-04-02T04:56:27.448449+0000","last_undegraded":"2024-04-02T04:56:27.448449+0000","last_fullsized":"2024-04-02T04:56:27.448449+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:13.638970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:13.638970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.446769+0000","last_change":"2024-04-02T04:54:56.015298+0000","last_active":"2024-04-02T04:56:27.446769+0000","last_peered":"2024-04-02T04:56:27.446769+0000","last_clean":"2024-04-02T04:56:27.446769+0000","last_became_active":"2024-04-02T04:54:56.011565+0000","last_became_peered":"2024-04-02T04:54:56.011565+0000","last_unstale":"2024-04-02T04:56:27.446769+0000","last_undegraded":"2024-04-02T04:56:27.446769+0000","last_fullsized":"2024-04-02T04:56:27.446769+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:51.287037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.287037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.1f","version":"0'0","reported_seq":305,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921879+0000","last_change":"2024-04-02T04:55:30.149166+0000","last_active":"2024-04-02T04:56:19.921879+0000","last_peered":"2024-04-02T04:56:19.921879+0000","last_clean":"2024-04-02T04:56:19.921879+0000","last_became_active":"2024-04-02T04:54:55.989261+0000","last_became_peered":"2024-04-02T04:54:55.989261+0000","last_unstale":"2024-04-02T04:56:19.921879+0000","last_undegraded":"2024-04-02T04:56:19.921879+0000","last_fullsized":"2024-04-02T04:56:19.921879+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:30.148952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:30.148952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:55.428579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":305,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920239+0000","last_change":"2024-04-02T04:55:38.240848+0000","last_active":"2024-04-02T04:56:19.920239+0000","last_peered":"2024-04-02T04:56:19.920239+0000","last_clean":"2024-04-02T04:56:19.920239+0000","last_became_active":"2024-04-02T04:54:55.970782+0000","last_became_peered":"2024-04-02T04:54:55.970782+0000","last_unstale":"2024-04-02T04:56:19.920239+0000","last_undegraded":"2024-04-02T04:56:19.920239+0000","last_fullsized":"2024-04-02T04:56:19.920239+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:38.240741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:55:38.240741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:39.718016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.18","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545187+0000","last_change":"2024-04-02T04:54:11.910124+0000","last_active":"2024-04-02T04:56:27.545187+0000","last_peered":"2024-04-02T04:56:27.545187+0000","last_clean":"2024-04-02T04:56:27.545187+0000","last_became_active":"2024-04-02T04:49:28.740911+0000","last_became_peered":"2024-04-02T04:49:28.740911+0000","last_unstale":"2024-04-02T04:56:27.545187+0000","last_undegraded":"2024-04-02T04:56:27.545187+0000","last_fullsized":"2024-04-02T04:56:27.545187+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:11.910003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:11.910003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":812,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452652+0000","last_change":"2024-04-02T04:56:02.011220+0000","last_active":"2024-04-02T04:56:27.452652+0000","last_peered":"2024-04-02T04:56:27.452652+0000","last_clean":"2024-04-02T04:56:27.452652+0000","last_became_active":"2024-04-02T04:56:02.010446+0000","last_became_peered":"2024-04-02T04:56:02.010446+0000","last_unstale":"2024-04-02T04:56:27.452652+0000","last_undegraded":"2024-04-02T04:56:27.452652+0000","last_fullsized":"2024-04-02T04:56:27.452652+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:54.645629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.645629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.19","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.444751+0000","last_change":"2024-04-02T04:54:56.011237+0000","last_active":"2024-04-02T04:56:27.444751+0000","last_peered":"2024-04-02T04:56:27.444751+0000","last_clean":"2024-04-02T04:56:27.444751+0000","last_became_active":"2024-04-02T04:54:56.008433+0000","last_became_peered":"2024-04-02T04:54:56.008433+0000","last_unstale":"2024-04-02T04:56:27.444751+0000","last_undegraded":"2024-04-02T04:56:27.444751+0000","last_fullsized":"2024-04-02T04:56:27.444751+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:02.294309+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.294309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":805,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265807+0000","last_change":"2024-04-02T04:55:30.997806+0000","last_active":"2024-04-02T04:56:19.265807+0000","last_peered":"2024-04-02T04:56:19.265807+0000","last_clean":"2024-04-02T04:56:19.265807+0000","last_became_active":"2024-04-02T04:53:54.925994+0000","last_became_peered":"2024-04-02T04:53:54.925994+0000","last_unstale":"2024-04-02T04:56:19.265807+0000","last_undegraded":"2024-04-02T04:56:19.265807+0000","last_fullsized":"2024-04-02T04:56:19.265807+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:30.997615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:55:30.997615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:43.113774+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1a","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445872+0000","last_change":"2024-04-02T04:54:56.008799+0000","last_active":"2024-04-02T04:56:27.445872+0000","last_peered":"2024-04-02T04:56:27.445872+0000","last_clean":"2024-04-02T04:56:27.445872+0000","last_became_active":"2024-04-02T04:54:56.008363+0000","last_became_peered":"2024-04-02T04:54:56.008363+0000","last_unstale":"2024-04-02T04:56:27.445872+0000","last_undegraded":"2024-04-02T04:56:27.445872+0000","last_fullsized":"2024-04-02T04:56:27.445872+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:43.275420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:43.275420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":694,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915410+0000","last_change":"2024-04-02T04:54:55.971295+0000","last_active":"2024-04-02T04:56:19.915410+0000","last_peered":"2024-04-02T04:56:19.915410+0000","last_clean":"2024-04-02T04:56:19.915410+0000","last_became_active":"2024-04-02T04:54:55.970851+0000","last_became_peered":"2024-04-02T04:54:55.970851+0000","last_unstale":"2024-04-02T04:56:19.915410+0000","last_undegraded":"2024-04-02T04:56:19.915410+0000","last_fullsized":"2024-04-02T04:56:19.915410+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.641792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.641792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.1b","version":"0'0","reported_seq":45,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919546+0000","last_change":"2024-04-02T04:56:02.020339+0000","last_active":"2024-04-02T04:56:19.919546+0000","last_peered":"2024-04-02T04:56:19.919546+0000","last_clean":"2024-04-02T04:56:19.919546+0000","last_became_active":"2024-04-02T04:56:02.019956+0000","last_became_peered":"2024-04-02T04:56:02.019956+0000","last_unstale":"2024-04-02T04:56:19.919546+0000","last_undegraded":"2024-04-02T04:56:19.919546+0000","last_fullsized":"2024-04-02T04:56:19.919546+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:38.809807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:38.809807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":817,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264703+0000","last_change":"2024-04-02T04:56:02.034225+0000","last_active":"2024-04-02T04:56:19.264703+0000","last_peered":"2024-04-02T04:56:19.264703+0000","last_clean":"2024-04-02T04:56:19.264703+0000","last_became_active":"2024-04-02T04:56:02.033876+0000","last_became_peered":"2024-04-02T04:56:02.033876+0000","last_unstale":"2024-04-02T04:56:19.264703+0000","last_undegraded":"2024-04-02T04:56:19.264703+0000","last_fullsized":"2024-04-02T04:56:19.264703+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:27.995289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:55:27.995289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:39.161729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":47,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449420+0000","last_change":"2024-04-02T04:56:02.036530+0000","last_active":"2024-04-02T04:56:27.449420+0000","last_peered":"2024-04-02T04:56:27.449420+0000","last_clean":"2024-04-02T04:56:27.449420+0000","last_became_active":"2024-04-02T04:56:02.033324+0000","last_became_peered":"2024-04-02T04:56:02.033324+0000","last_unstale":"2024-04-02T04:56:27.449420+0000","last_undegraded":"2024-04-02T04:56:27.449420+0000","last_fullsized":"2024-04-02T04:56:27.449420+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:26.098486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:26.098486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:50.930303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":45,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.927910+0000","last_change":"2024-04-02T04:56:02.012916+0000","last_active":"2024-04-02T04:56:19.927910+0000","last_peered":"2024-04-02T04:56:19.927910+0000","last_clean":"2024-04-02T04:56:19.927910+0000","last_became_active":"2024-04-02T04:56:02.012541+0000","last_became_peered":"2024-04-02T04:56:02.012541+0000","last_unstale":"2024-04-02T04:56:19.927910+0000","last_undegraded":"2024-04-02T04:56:19.927910+0000","last_fullsized":"2024-04-02T04:56:19.927910+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.758420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.758420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":809,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450834+0000","last_change":"2024-04-02T04:54:26.234089+0000","last_active":"2024-04-02T04:56:27.450834+0000","last_peered":"2024-04-02T04:56:27.450834+0000","last_clean":"2024-04-02T04:56:27.450834+0000","last_became_active":"2024-04-02T04:53:54.955517+0000","last_became_peered":"2024-04-02T04:53:54.955517+0000","last_unstale":"2024-04-02T04:56:27.450834+0000","last_undegraded":"2024-04-02T04:56:27.450834+0000","last_fullsized":"2024-04-02T04:56:27.450834+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.233991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.233991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":807,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448754+0000","last_change":"2024-04-02T04:54:56.005290+0000","last_active":"2024-04-02T04:56:27.448754+0000","last_peered":"2024-04-02T04:56:27.448754+0000","last_clean":"2024-04-02T04:56:27.448754+0000","last_became_active":"2024-04-02T04:54:56.004451+0000","last_became_peered":"2024-04-02T04:54:56.004451+0000","last_unstale":"2024-04-02T04:56:27.448754+0000","last_undegraded":"2024-04-02T04:56:27.448754+0000","last_fullsized":"2024-04-02T04:56:27.448754+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.334949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.334949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":178,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.447759+0000","last_change":"2024-04-02T04:54:56.009048+0000","last_active":"2024-04-02T04:56:27.447759+0000","last_peered":"2024-04-02T04:56:27.447759+0000","last_clean":"2024-04-02T04:56:27.447759+0000","last_became_active":"2024-04-02T04:54:56.004907+0000","last_became_peered":"2024-04-02T04:54:56.004907+0000","last_unstale":"2024-04-02T04:56:27.447759+0000","last_undegraded":"2024-04-02T04:56:27.447759+0000","last_fullsized":"2024-04-02T04:56:27.447759+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:08.577227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:08.577227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":800,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924753+0000","last_change":"2024-04-02T04:54:55.979416+0000","last_active":"2024-04-02T04:56:19.924753+0000","last_peered":"2024-04-02T04:56:19.924753+0000","last_clean":"2024-04-02T04:56:19.924753+0000","last_became_active":"2024-04-02T04:54:55.978988+0000","last_became_peered":"2024-04-02T04:54:55.978988+0000","last_unstale":"2024-04-02T04:56:19.924753+0000","last_undegraded":"2024-04-02T04:56:19.924753+0000","last_fullsized":"2024-04-02T04:56:19.924753+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:39.367130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:39.367130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":819,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.549424+0000","last_change":"2024-04-02T04:56:02.047640+0000","last_active":"2024-04-02T04:56:27.549424+0000","last_peered":"2024-04-02T04:56:27.549424+0000","last_clean":"2024-04-02T04:56:27.549424+0000","last_became_active":"2024-04-02T04:56:02.044097+0000","last_became_peered":"2024-04-02T04:56:02.044097+0000","last_unstale":"2024-04-02T04:56:27.549424+0000","last_undegraded":"2024-04-02T04:56:27.549424+0000","last_fullsized":"2024-04-02T04:56:27.549424+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:16.829708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:16.829708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:32.435246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":794,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.923717+0000","last_change":"2024-04-02T04:53:00.409859+0000","last_active":"2024-04-02T04:56:19.923717+0000","last_peered":"2024-04-02T04:56:19.923717+0000","last_clean":"2024-04-02T04:56:19.923717+0000","last_became_active":"2024-04-02T04:49:28.761080+0000","last_became_peered":"2024-04-02T04:49:28.761080+0000","last_unstale":"2024-04-02T04:56:19.923717+0000","last_undegraded":"2024-04-02T04:56:19.923717+0000","last_fullsized":"2024-04-02T04:56:19.923717+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.409610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.409610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":794,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918169+0000","last_change":"2024-04-02T04:53:06.476121+0000","last_active":"2024-04-02T04:56:19.918169+0000","last_peered":"2024-04-02T04:56:19.918169+0000","last_clean":"2024-04-02T04:56:19.918169+0000","last_became_active":"2024-04-02T04:49:28.744986+0000","last_became_peered":"2024-04-02T04:49:28.744986+0000","last_unstale":"2024-04-02T04:56:19.918169+0000","last_undegraded":"2024-04-02T04:56:19.918169+0000","last_fullsized":"2024-04-02T04:56:19.918169+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:06.475984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:06.475984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":811,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:25.009090+0000","last_change":"2024-04-02T04:56:25.009111+0000","last_active":"2024-04-02T04:56:25.009090+0000","last_peered":"2024-04-02T04:56:25.009090+0000","last_clean":"2024-04-02T04:56:25.009090+0000","last_became_active":"2024-04-02T04:54:55.989586+0000","last_became_peered":"2024-04-02T04:54:55.989586+0000","last_unstale":"2024-04-02T04:56:25.009090+0000","last_undegraded":"2024-04-02T04:56:25.009090+0000","last_fullsized":"2024-04-02T04:56:25.009090+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:25.009023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:48.958371+0000","last_clean_scrub_stamp":"2024-04-02T04:56:25.009023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:34.602790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":184,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:20.946646+0000","last_change":"2024-04-02T04:56:20.946691+0000","last_active":"2024-04-02T04:56:20.946646+0000","last_peered":"2024-04-02T04:56:20.946646+0000","last_clean":"2024-04-02T04:56:20.946646+0000","last_became_active":"2024-04-02T04:54:55.988651+0000","last_became_peered":"2024-04-02T04:54:55.988651+0000","last_unstale":"2024-04-02T04:56:20.946646+0000","last_undegraded":"2024-04-02T04:56:20.946646+0000","last_fullsized":"2024-04-02T04:56:20.946646+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:20.946546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:56:20.946546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:33.010082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":47,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448570+0000","last_change":"2024-04-02T04:56:02.053405+0000","last_active":"2024-04-02T04:56:27.448570+0000","last_peered":"2024-04-02T04:56:27.448570+0000","last_clean":"2024-04-02T04:56:27.448570+0000","last_became_active":"2024-04-02T04:56:02.053031+0000","last_became_peered":"2024-04-02T04:56:02.053031+0000","last_unstale":"2024-04-02T04:56:27.448570+0000","last_undegraded":"2024-04-02T04:56:27.448570+0000","last_fullsized":"2024-04-02T04:56:27.448570+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:31.072404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:55:31.072404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:43.712560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":814,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545673+0000","last_change":"2024-04-02T04:54:55.928923+0000","last_active":"2024-04-02T04:56:27.545673+0000","last_peered":"2024-04-02T04:56:27.545673+0000","last_clean":"2024-04-02T04:56:27.545673+0000","last_became_active":"2024-04-02T04:54:55.928548+0000","last_became_peered":"2024-04-02T04:54:55.928548+0000","last_unstale":"2024-04-02T04:56:27.545673+0000","last_undegraded":"2024-04-02T04:56:27.545673+0000","last_fullsized":"2024-04-02T04:56:27.545673+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.882197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.882197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":814,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449301+0000","last_change":"2024-04-02T04:56:25.497980+0000","last_active":"2024-04-02T04:56:27.449301+0000","last_peered":"2024-04-02T04:56:27.449301+0000","last_clean":"2024-04-02T04:56:27.449301+0000","last_became_active":"2024-04-02T04:54:55.968463+0000","last_became_peered":"2024-04-02T04:54:55.968463+0000","last_unstale":"2024-04-02T04:56:27.449301+0000","last_undegraded":"2024-04-02T04:56:27.449301+0000","last_fullsized":"2024-04-02T04:56:27.449301+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:25.497858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:56:25.497858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:27.591931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":821,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546132+0000","last_change":"2024-04-02T04:56:24.211292+0000","last_active":"2024-04-02T04:56:27.546132+0000","last_peered":"2024-04-02T04:56:27.546132+0000","last_clean":"2024-04-02T04:56:27.546132+0000","last_became_active":"2024-04-02T04:56:02.041840+0000","last_became_peered":"2024-04-02T04:56:02.041840+0000","last_unstale":"2024-04-02T04:56:27.546132+0000","last_undegraded":"2024-04-02T04:56:27.546132+0000","last_fullsized":"2024-04-02T04:56:27.546132+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:24.211195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:56:24.211195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:51.079380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":809,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.447917+0000","last_change":"2024-04-02T04:54:02.337934+0000","last_active":"2024-04-02T04:56:27.447917+0000","last_peered":"2024-04-02T04:56:27.447917+0000","last_clean":"2024-04-02T04:56:27.447917+0000","last_became_active":"2024-04-02T04:53:54.929052+0000","last_became_peered":"2024-04-02T04:53:54.929052+0000","last_unstale":"2024-04-02T04:56:27.447917+0000","last_undegraded":"2024-04-02T04:56:27.447917+0000","last_fullsized":"2024-04-02T04:56:27.447917+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:02.337688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:02.337688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.c","version":"0'0","reported_seq":299,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914802+0000","last_change":"2024-04-02T04:56:19.913193+0000","last_active":"2024-04-02T04:56:19.914802+0000","last_peered":"2024-04-02T04:56:19.914802+0000","last_clean":"2024-04-02T04:56:19.914802+0000","last_became_active":"2024-04-02T04:53:54.943480+0000","last_became_peered":"2024-04-02T04:53:54.943480+0000","last_unstale":"2024-04-02T04:56:19.914802+0000","last_undegraded":"2024-04-02T04:56:19.914802+0000","last_fullsized":"2024-04-02T04:56:19.914802+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:19.912968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:56:19.912968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:45.678419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":794,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928116+0000","last_change":"2024-04-02T04:52:53.411217+0000","last_active":"2024-04-02T04:56:19.928116+0000","last_peered":"2024-04-02T04:56:19.928116+0000","last_clean":"2024-04-02T04:56:19.928116+0000","last_became_active":"2024-04-02T04:49:28.767822+0000","last_became_peered":"2024-04-02T04:49:28.767822+0000","last_unstale":"2024-04-02T04:56:19.928116+0000","last_undegraded":"2024-04-02T04:56:19.928116+0000","last_fullsized":"2024-04-02T04:56:19.928116+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:53.410913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.410913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.d","version":"0'0","reported_seq":299,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921661+0000","last_change":"2024-04-02T04:56:13.968587+0000","last_active":"2024-04-02T04:56:19.921661+0000","last_peered":"2024-04-02T04:56:19.921661+0000","last_clean":"2024-04-02T04:56:19.921661+0000","last_became_active":"2024-04-02T04:53:54.933626+0000","last_became_peered":"2024-04-02T04:53:54.933626+0000","last_unstale":"2024-04-02T04:56:19.921661+0000","last_undegraded":"2024-04-02T04:56:19.921661+0000","last_fullsized":"2024-04-02T04:56:19.921661+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:13.968441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:56:13.968441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:43.189940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":801,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448508+0000","last_change":"2024-04-02T04:51:52.390967+0000","last_active":"2024-04-02T04:56:27.448508+0000","last_peered":"2024-04-02T04:56:27.448508+0000","last_clean":"2024-04-02T04:56:27.448508+0000","last_became_active":"2024-04-02T04:49:28.779928+0000","last_became_peered":"2024-04-02T04:49:28.779928+0000","last_unstale":"2024-04-02T04:56:27.448508+0000","last_undegraded":"2024-04-02T04:56:27.448508+0000","last_fullsized":"2024-04-02T04:56:27.448508+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:52.390694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:52.390694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.e","version":"0'0","reported_seq":800,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925338+0000","last_change":"2024-04-02T04:56:02.020235+0000","last_active":"2024-04-02T04:56:19.925338+0000","last_peered":"2024-04-02T04:56:19.925338+0000","last_clean":"2024-04-02T04:56:19.925338+0000","last_became_active":"2024-04-02T04:56:02.019161+0000","last_became_peered":"2024-04-02T04:56:02.019161+0000","last_unstale":"2024-04-02T04:56:19.925338+0000","last_undegraded":"2024-04-02T04:56:19.925338+0000","last_fullsized":"2024-04-02T04:56:19.925338+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.473096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.473096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":805,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267039+0000","last_change":"2024-04-02T04:56:18.962109+0000","last_active":"2024-04-02T04:56:19.267039+0000","last_peered":"2024-04-02T04:56:19.267039+0000","last_clean":"2024-04-02T04:56:19.267039+0000","last_became_active":"2024-04-02T04:49:28.758733+0000","last_became_peered":"2024-04-02T04:49:28.758733+0000","last_unstale":"2024-04-02T04:56:19.267039+0000","last_undegraded":"2024-04-02T04:56:19.267039+0000","last_fullsized":"2024-04-02T04:56:19.267039+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:18.962009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.962009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:42.925029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"43.f","version":"0'0","reported_seq":809,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452265+0000","last_change":"2024-04-02T04:54:25.202447+0000","last_active":"2024-04-02T04:56:27.452265+0000","last_peered":"2024-04-02T04:56:27.452265+0000","last_clean":"2024-04-02T04:56:27.452265+0000","last_became_active":"2024-04-02T04:53:54.921036+0000","last_became_peered":"2024-04-02T04:53:54.921036+0000","last_unstale":"2024-04-02T04:56:27.452265+0000","last_undegraded":"2024-04-02T04:56:27.452265+0000","last_fullsized":"2024-04-02T04:56:27.452265+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:25.202339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:54:25.202339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":48,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449518+0000","last_change":"2024-04-02T04:56:02.036165+0000","last_active":"2024-04-02T04:56:27.449518+0000","last_peered":"2024-04-02T04:56:27.449518+0000","last_clean":"2024-04-02T04:56:27.449518+0000","last_became_active":"2024-04-02T04:56:02.033787+0000","last_became_peered":"2024-04-02T04:56:02.033787+0000","last_unstale":"2024-04-02T04:56:27.449518+0000","last_undegraded":"2024-04-02T04:56:27.449518+0000","last_fullsized":"2024-04-02T04:56:27.449518+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:21.056758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:55:21.056758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:46.939978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.8","version":"0'0","reported_seq":299,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.913592+0000","last_change":"2024-04-02T04:55:35.190015+0000","last_active":"2024-04-02T04:56:19.913592+0000","last_peered":"2024-04-02T04:56:19.913592+0000","last_clean":"2024-04-02T04:56:19.913592+0000","last_became_active":"2024-04-02T04:53:54.945599+0000","last_became_peered":"2024-04-02T04:53:54.945599+0000","last_unstale":"2024-04-02T04:56:19.913592+0000","last_undegraded":"2024-04-02T04:56:19.913592+0000","last_fullsized":"2024-04-02T04:56:19.913592+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:35.189778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:35.189778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:48.701302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":808,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546559+0000","last_change":"2024-04-02T04:54:03.883895+0000","last_active":"2024-04-02T04:56:27.546559+0000","last_peered":"2024-04-02T04:56:27.546559+0000","last_clean":"2024-04-02T04:56:27.546559+0000","last_became_active":"2024-04-02T04:53:54.940670+0000","last_became_peered":"2024-04-02T04:53:54.940670+0000","last_unstale":"2024-04-02T04:56:27.546559+0000","last_undegraded":"2024-04-02T04:56:27.546559+0000","last_fullsized":"2024-04-02T04:56:27.546559+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.883769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.883769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.9","version":"0'0","reported_seq":48,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.544978+0000","last_change":"2024-04-02T04:56:02.058240+0000","last_active":"2024-04-02T04:56:27.544978+0000","last_peered":"2024-04-02T04:56:27.544978+0000","last_clean":"2024-04-02T04:56:27.544978+0000","last_became_active":"2024-04-02T04:56:02.057364+0000","last_became_peered":"2024-04-02T04:56:02.057364+0000","last_unstale":"2024-04-02T04:56:27.544978+0000","last_undegraded":"2024-04-02T04:56:27.544978+0000","last_fullsized":"2024-04-02T04:56:27.544978+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:27.081139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:27.081139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:39.667210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":47,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448698+0000","last_change":"2024-04-02T04:56:02.053642+0000","last_active":"2024-04-02T04:56:27.448698+0000","last_peered":"2024-04-02T04:56:27.448698+0000","last_clean":"2024-04-02T04:56:27.448698+0000","last_became_active":"2024-04-02T04:56:02.052684+0000","last_became_peered":"2024-04-02T04:56:02.052684+0000","last_unstale":"2024-04-02T04:56:27.448698+0000","last_undegraded":"2024-04-02T04:56:27.448698+0000","last_fullsized":"2024-04-02T04:56:27.448698+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.762442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.762442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.a","version":"0'0","reported_seq":305,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915071+0000","last_change":"2024-04-02T04:55:18.317781+0000","last_active":"2024-04-02T04:56:19.915071+0000","last_peered":"2024-04-02T04:56:19.915071+0000","last_clean":"2024-04-02T04:56:19.915071+0000","last_became_active":"2024-04-02T04:54:55.956199+0000","last_became_peered":"2024-04-02T04:54:55.956199+0000","last_unstale":"2024-04-02T04:56:19.915071+0000","last_undegraded":"2024-04-02T04:56:19.915071+0000","last_fullsized":"2024-04-02T04:56:19.915071+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:18.317648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:55:18.317648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:36.336467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":801,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.444898+0000","last_change":"2024-04-02T04:52:42.318994+0000","last_active":"2024-04-02T04:56:27.444898+0000","last_peered":"2024-04-02T04:56:27.444898+0000","last_clean":"2024-04-02T04:56:27.444898+0000","last_became_active":"2024-04-02T04:49:28.756547+0000","last_became_peered":"2024-04-02T04:49:28.756547+0000","last_unstale":"2024-04-02T04:56:27.444898+0000","last_undegraded":"2024-04-02T04:56:27.444898+0000","last_fullsized":"2024-04-02T04:56:27.444898+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.318800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.318800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.b","version":"0'0","reported_seq":800,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920369+0000","last_change":"2024-04-02T04:56:02.033228+0000","last_active":"2024-04-02T04:56:19.920369+0000","last_peered":"2024-04-02T04:56:19.920369+0000","last_clean":"2024-04-02T04:56:19.920369+0000","last_became_active":"2024-04-02T04:56:02.032872+0000","last_became_peered":"2024-04-02T04:56:02.032872+0000","last_unstale":"2024-04-02T04:56:19.920369+0000","last_undegraded":"2024-04-02T04:56:19.920369+0000","last_fullsized":"2024-04-02T04:56:19.920369+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:52.417771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.981622+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.417771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":809,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264718+0000","last_change":"2024-04-02T04:56:02.018549+0000","last_active":"2024-04-02T04:56:19.264718+0000","last_peered":"2024-04-02T04:56:19.264718+0000","last_clean":"2024-04-02T04:56:19.264718+0000","last_became_active":"2024-04-02T04:56:02.018161+0000","last_became_peered":"2024-04-02T04:56:02.018161+0000","last_unstale":"2024-04-02T04:56:19.264718+0000","last_undegraded":"2024-04-02T04:56:19.264718+0000","last_fullsized":"2024-04-02T04:56:19.264718+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.983966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:24.948823+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.983966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"478'9","reported_seq":868,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918485+0000","last_change":"2024-04-02T04:55:50.016632+0000","last_active":"2024-04-02T04:56:19.918485+0000","last_peered":"2024-04-02T04:56:19.918485+0000","last_clean":"2024-04-02T04:56:19.918485+0000","last_became_active":"2024-04-02T04:55:50.015524+0000","last_became_peered":"2024-04-02T04:55:50.015524+0000","last_unstale":"2024-04-02T04:56:19.918485+0000","last_undegraded":"2024-04-02T04:56:19.918485+0000","last_fullsized":"2024-04-02T04:56:19.918485+0000","mapping_epoch":135,"log_start":"478'9","ondisk_log_start":"478'9","created":135,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:55:21.913103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:21.913103+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:40.269329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":177,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267183+0000","last_change":"2024-04-02T04:54:56.017671+0000","last_active":"2024-04-02T04:56:19.267183+0000","last_peered":"2024-04-02T04:56:19.267183+0000","last_clean":"2024-04-02T04:56:19.267183+0000","last_became_active":"2024-04-02T04:54:56.017309+0000","last_became_peered":"2024-04-02T04:54:56.017309+0000","last_unstale":"2024-04-02T04:56:19.267183+0000","last_undegraded":"2024-04-02T04:56:19.267183+0000","last_fullsized":"2024-04-02T04:56:19.267183+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:35.773829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:35.773829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":842,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.913044+0000","last_change":"2024-04-02T04:54:55.990438+0000","last_active":"2024-04-02T04:56:19.913044+0000","last_peered":"2024-04-02T04:56:19.913044+0000","last_clean":"2024-04-02T04:56:19.913044+0000","last_became_active":"2024-04-02T04:54:55.990081+0000","last_became_peered":"2024-04-02T04:54:55.990081+0000","last_unstale":"2024-04-02T04:56:19.913044+0000","last_undegraded":"2024-04-02T04:56:19.913044+0000","last_fullsized":"2024-04-02T04:56:19.913044+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:12.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.577625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"415'28","reported_seq":467,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925574+0000","last_change":"2024-04-02T04:54:15.261937+0000","last_active":"2024-04-02T04:56:19.925574+0000","last_peered":"2024-04-02T04:56:19.925574+0000","last_clean":"2024-04-02T04:56:19.925574+0000","last_became_active":"2024-04-02T04:53:54.938166+0000","last_became_peered":"2024-04-02T04:53:54.938166+0000","last_unstale":"2024-04-02T04:56:19.925574+0000","last_undegraded":"2024-04-02T04:56:19.925574+0000","last_fullsized":"2024-04-02T04:56:19.925574+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'28","last_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_deep_scrub":"415'28","last_deep_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_clean_scrub_stamp":"2024-04-02T04:54:15.261679+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":961,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.547493+0000","last_change":"2024-04-02T04:56:02.040494+0000","last_active":"2024-04-02T04:56:27.547493+0000","last_peered":"2024-04-02T04:56:27.547493+0000","last_clean":"2024-04-02T04:56:27.547493+0000","last_became_active":"2024-04-02T04:56:02.039969+0000","last_became_peered":"2024-04-02T04:56:02.039969+0000","last_unstale":"2024-04-02T04:56:27.547493+0000","last_undegraded":"2024-04-02T04:56:27.547493+0000","last_fullsized":"2024-04-02T04:56:27.547493+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.929851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.929851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":173,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919989+0000","last_change":"2024-04-02T04:54:56.003791+0000","last_active":"2024-04-02T04:56:19.919989+0000","last_peered":"2024-04-02T04:56:19.919989+0000","last_clean":"2024-04-02T04:56:19.919989+0000","last_became_active":"2024-04-02T04:54:56.002149+0000","last_became_peered":"2024-04-02T04:54:56.002149+0000","last_unstale":"2024-04-02T04:56:19.919989+0000","last_undegraded":"2024-04-02T04:56:19.919989+0000","last_fullsized":"2024-04-02T04:56:19.919989+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:46.685936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:46.685936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":199,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449737+0000","last_change":"2024-04-02T04:56:02.018124+0000","last_active":"2024-04-02T04:56:27.449737+0000","last_peered":"2024-04-02T04:56:27.449737+0000","last_clean":"2024-04-02T04:56:27.449737+0000","last_became_active":"2024-04-02T04:56:02.014768+0000","last_became_peered":"2024-04-02T04:56:02.014768+0000","last_unstale":"2024-04-02T04:56:27.449737+0000","last_undegraded":"2024-04-02T04:56:27.449737+0000","last_fullsized":"2024-04-02T04:56:27.449737+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:29.088346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:29.088346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:57.554826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":826,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.547149+0000","last_change":"2024-04-02T04:56:27.145868+0000","last_active":"2024-04-02T04:56:27.547149+0000","last_peered":"2024-04-02T04:56:27.547149+0000","last_clean":"2024-04-02T04:56:27.547149+0000","last_became_active":"2024-04-02T04:56:02.030140+0000","last_became_peered":"2024-04-02T04:56:02.030140+0000","last_unstale":"2024-04-02T04:56:27.547149+0000","last_undegraded":"2024-04-02T04:56:27.547149+0000","last_fullsized":"2024-04-02T04:56:27.547149+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:27.145771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:27.145771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:46.741709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":954,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266562+0000","last_change":"2024-04-02T04:52:55.335191+0000","last_active":"2024-04-02T04:56:19.266562+0000","last_peered":"2024-04-02T04:56:19.266562+0000","last_clean":"2024-04-02T04:56:19.266562+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:56:19.266562+0000","last_undegraded":"2024-04-02T04:56:19.266562+0000","last_fullsized":"2024-04-02T04:56:19.266562+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:55.334980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:52:55.334980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"402'25","reported_seq":790,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922576+0000","last_change":"2024-04-02T04:56:02.061309+0000","last_active":"2024-04-02T04:56:19.922576+0000","last_peered":"2024-04-02T04:56:19.922576+0000","last_clean":"2024-04-02T04:56:19.922576+0000","last_became_active":"2024-04-02T04:56:02.013022+0000","last_became_peered":"2024-04-02T04:56:02.013022+0000","last_unstale":"2024-04-02T04:56:19.922576+0000","last_undegraded":"2024-04-02T04:56:19.922576+0000","last_fullsized":"2024-04-02T04:56:19.922576+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'15","last_scrub_stamp":"2024-04-02T04:53:07.600375+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.600375+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":970,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451758+0000","last_change":"2024-04-02T04:56:02.036683+0000","last_active":"2024-04-02T04:56:27.451758+0000","last_peered":"2024-04-02T04:56:27.451758+0000","last_clean":"2024-04-02T04:56:27.451758+0000","last_became_active":"2024-04-02T04:56:02.035898+0000","last_became_peered":"2024-04-02T04:56:02.035898+0000","last_unstale":"2024-04-02T04:56:27.451758+0000","last_undegraded":"2024-04-02T04:56:27.451758+0000","last_fullsized":"2024-04-02T04:56:27.451758+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.415766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":935,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265726+0000","last_change":"2024-04-02T04:54:19.360031+0000","last_active":"2024-04-02T04:56:19.265726+0000","last_peered":"2024-04-02T04:56:19.265726+0000","last_clean":"2024-04-02T04:56:19.265726+0000","last_became_active":"2024-04-02T04:53:54.935778+0000","last_became_peered":"2024-04-02T04:53:54.935778+0000","last_unstale":"2024-04-02T04:56:19.265726+0000","last_undegraded":"2024-04-02T04:56:19.265726+0000","last_fullsized":"2024-04-02T04:56:19.265726+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.359786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.359786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"478'14","reported_seq":899,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:21.985312+0000","last_change":"2024-04-02T04:56:21.985387+0000","last_active":"2024-04-02T04:56:21.985312+0000","last_peered":"2024-04-02T04:56:21.985312+0000","last_clean":"2024-04-02T04:56:21.985312+0000","last_became_active":"2024-04-02T04:56:14.051790+0000","last_became_peered":"2024-04-02T04:56:14.051790+0000","last_unstale":"2024-04-02T04:56:21.985312+0000","last_undegraded":"2024-04-02T04:56:21.985312+0000","last_fullsized":"2024-04-02T04:56:21.985312+0000","mapping_epoch":516,"log_start":"478'14","ondisk_log_start":"478'14","created":135,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'14","last_scrub_stamp":"2024-04-02T04:56:21.985162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:21.985162+0000","objects_scrubbed":4,"log_size":0,"log_dups_size":34,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:22.872298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":817,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266318+0000","last_change":"2024-04-02T04:56:02.033276+0000","last_active":"2024-04-02T04:56:19.266318+0000","last_peered":"2024-04-02T04:56:19.266318+0000","last_clean":"2024-04-02T04:56:19.266318+0000","last_became_active":"2024-04-02T04:56:02.032919+0000","last_became_peered":"2024-04-02T04:56:02.032919+0000","last_unstale":"2024-04-02T04:56:19.266318+0000","last_undegraded":"2024-04-02T04:56:19.266318+0000","last_fullsized":"2024-04-02T04:56:19.266318+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:10.335602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:10.335602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":952,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920910+0000","last_change":"2024-04-02T04:54:01.950572+0000","last_active":"2024-04-02T04:56:19.920910+0000","last_peered":"2024-04-02T04:56:19.920910+0000","last_clean":"2024-04-02T04:56:19.920910+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T04:56:19.920910+0000","last_undegraded":"2024-04-02T04:56:19.920910+0000","last_fullsized":"2024-04-02T04:56:19.920910+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.950469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.950469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"402'31","reported_seq":526,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.926086+0000","last_change":"2024-04-02T04:56:02.031363+0000","last_active":"2024-04-02T04:56:19.926086+0000","last_peered":"2024-04-02T04:56:19.926086+0000","last_clean":"2024-04-02T04:56:19.926086+0000","last_became_active":"2024-04-02T04:56:01.994332+0000","last_became_peered":"2024-04-02T04:56:01.994332+0000","last_unstale":"2024-04-02T04:56:19.926086+0000","last_undegraded":"2024-04-02T04:56:19.926086+0000","last_fullsized":"2024-04-02T04:56:19.926086+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'31","last_scrub_stamp":"2024-04-02T04:54:12.211222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:12.211222+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":968,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546493+0000","last_change":"2024-04-02T04:55:23.768650+0000","last_active":"2024-04-02T04:56:27.546493+0000","last_peered":"2024-04-02T04:56:27.546493+0000","last_clean":"2024-04-02T04:56:27.546493+0000","last_became_active":"2024-04-02T04:54:55.951772+0000","last_became_peered":"2024-04-02T04:54:55.951772+0000","last_unstale":"2024-04-02T04:56:27.546493+0000","last_undegraded":"2024-04-02T04:56:27.546493+0000","last_fullsized":"2024-04-02T04:56:27.546493+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:23.768444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:55:23.768444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:50.545802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":435,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914518+0000","last_change":"2024-04-02T04:56:02.031226+0000","last_active":"2024-04-02T04:56:19.914518+0000","last_peered":"2024-04-02T04:56:19.914518+0000","last_clean":"2024-04-02T04:56:19.914518+0000","last_became_active":"2024-04-02T04:56:02.030867+0000","last_became_peered":"2024-04-02T04:56:02.030867+0000","last_unstale":"2024-04-02T04:56:19.914518+0000","last_undegraded":"2024-04-02T04:56:19.914518+0000","last_fullsized":"2024-04-02T04:56:19.914518+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.209709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.209709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"101'128","reported_seq":924,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.544889+0000","last_change":"2024-04-02T04:56:04.966229+0000","last_active":"2024-04-02T04:56:27.544889+0000","last_peered":"2024-04-02T04:56:27.544889+0000","last_clean":"2024-04-02T04:56:27.544889+0000","last_became_active":"2024-04-02T04:56:04.964114+0000","last_became_peered":"2024-04-02T04:56:04.964114+0000","last_unstale":"2024-04-02T04:56:27.544889+0000","last_undegraded":"2024-04-02T04:56:27.544889+0000","last_fullsized":"2024-04-02T04:56:27.544889+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.825332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.825332+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:48.505963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.4","version":"138'2","reported_seq":871,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921611+0000","last_change":"2024-04-02T04:56:12.140376+0000","last_active":"2024-04-02T04:56:19.921611+0000","last_peered":"2024-04-02T04:56:19.921611+0000","last_clean":"2024-04-02T04:56:19.921611+0000","last_became_active":"2024-04-02T04:56:10.022107+0000","last_became_peered":"2024-04-02T04:56:10.022107+0000","last_unstale":"2024-04-02T04:56:19.921611+0000","last_undegraded":"2024-04-02T04:56:19.921611+0000","last_fullsized":"2024-04-02T04:56:19.921611+0000","mapping_epoch":135,"log_start":"138'2","ondisk_log_start":"138'2","created":135,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:56:12.140006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:12.140006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":2,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:41.518461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":827,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546140+0000","last_change":"2024-04-02T04:56:02.020266+0000","last_active":"2024-04-02T04:56:27.546140+0000","last_peered":"2024-04-02T04:56:27.546140+0000","last_clean":"2024-04-02T04:56:27.546140+0000","last_became_active":"2024-04-02T04:56:02.019929+0000","last_became_peered":"2024-04-02T04:56:02.019929+0000","last_unstale":"2024-04-02T04:56:27.546140+0000","last_undegraded":"2024-04-02T04:56:27.546140+0000","last_fullsized":"2024-04-02T04:56:27.546140+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:54.055039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:54.055039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"402'42","reported_seq":1015,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921926+0000","last_change":"2024-04-02T04:51:43.069937+0000","last_active":"2024-04-02T04:56:19.921926+0000","last_peered":"2024-04-02T04:56:19.921926+0000","last_clean":"2024-04-02T04:56:19.921926+0000","last_became_active":"2024-04-02T04:43:56.232305+0000","last_became_peered":"2024-04-02T04:43:56.232305+0000","last_unstale":"2024-04-02T04:56:19.921926+0000","last_undegraded":"2024-04-02T04:56:19.921926+0000","last_fullsized":"2024-04-02T04:56:19.921926+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'26","last_scrub_stamp":"2024-04-02T04:51:43.069694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:51:43.069694+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":439,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915732+0000","last_change":"2024-04-02T04:56:02.004400+0000","last_active":"2024-04-02T04:56:19.915732+0000","last_peered":"2024-04-02T04:56:19.915732+0000","last_clean":"2024-04-02T04:56:19.915732+0000","last_became_active":"2024-04-02T04:56:02.004005+0000","last_became_peered":"2024-04-02T04:56:02.004005+0000","last_unstale":"2024-04-02T04:56:19.915732+0000","last_undegraded":"2024-04-02T04:56:19.915732+0000","last_fullsized":"2024-04-02T04:56:19.915732+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:59.105134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.105134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"121'12","reported_seq":1111,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448722+0000","last_change":"2024-04-02T04:55:34.654439+0000","last_active":"2024-04-02T04:56:27.448722+0000","last_peered":"2024-04-02T04:56:27.448722+0000","last_clean":"2024-04-02T04:56:27.448722+0000","last_became_active":"2024-04-02T04:43:58.296884+0000","last_became_peered":"2024-04-02T04:43:58.296884+0000","last_unstale":"2024-04-02T04:56:27.448722+0000","last_undegraded":"2024-04-02T04:56:27.448722+0000","last_fullsized":"2024-04-02T04:56:27.448722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:55:34.654306+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:55:34.654306+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.654306+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:58.528958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":964,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.549637+0000","last_change":"2024-04-02T04:56:02.023567+0000","last_active":"2024-04-02T04:56:27.549637+0000","last_peered":"2024-04-02T04:56:27.549637+0000","last_clean":"2024-04-02T04:56:27.549637+0000","last_became_active":"2024-04-02T04:56:02.023213+0000","last_became_peered":"2024-04-02T04:56:02.023213+0000","last_unstale":"2024-04-02T04:56:27.549637+0000","last_undegraded":"2024-04-02T04:56:27.549637+0000","last_fullsized":"2024-04-02T04:56:27.549637+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.980664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.980664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'128","reported_seq":724,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.913170+0000","last_change":"2024-04-02T04:56:04.960407+0000","last_active":"2024-04-02T04:56:19.913170+0000","last_peered":"2024-04-02T04:56:19.913170+0000","last_clean":"2024-04-02T04:56:19.913170+0000","last_became_active":"2024-04-02T04:56:04.959832+0000","last_became_peered":"2024-04-02T04:56:04.959832+0000","last_unstale":"2024-04-02T04:56:19.913170+0000","last_undegraded":"2024-04-02T04:56:19.913170+0000","last_fullsized":"2024-04-02T04:56:19.913170+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.263513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.263513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:00.029986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.5","version":"478'11","reported_seq":888,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.447618+0000","last_change":"2024-04-02T04:56:12.192445+0000","last_active":"2024-04-02T04:56:27.447618+0000","last_peered":"2024-04-02T04:56:27.447618+0000","last_clean":"2024-04-02T04:56:27.447618+0000","last_became_active":"2024-04-02T04:56:05.977946+0000","last_became_peered":"2024-04-02T04:56:05.977946+0000","last_unstale":"2024-04-02T04:56:27.447618+0000","last_undegraded":"2024-04-02T04:56:27.447618+0000","last_fullsized":"2024-04-02T04:56:27.447618+0000","mapping_epoch":513,"log_start":"478'11","ondisk_log_start":"478'11","created":135,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-04-02T04:56:12.192218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:12.192218+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:16.852862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":805,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449253+0000","last_change":"2024-04-02T04:52:42.269689+0000","last_active":"2024-04-02T04:56:27.449253+0000","last_peered":"2024-04-02T04:56:27.449253+0000","last_clean":"2024-04-02T04:56:27.449253+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:56:27.449253+0000","last_undegraded":"2024-04-02T04:56:27.449253+0000","last_fullsized":"2024-04-02T04:56:27.449253+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.268948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.268948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":955,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449194+0000","last_change":"2024-04-02T04:56:15.169496+0000","last_active":"2024-04-02T04:56:27.449194+0000","last_peered":"2024-04-02T04:56:27.449194+0000","last_clean":"2024-04-02T04:56:27.449194+0000","last_became_active":"2024-04-02T04:56:02.028741+0000","last_became_peered":"2024-04-02T04:56:02.028741+0000","last_unstale":"2024-04-02T04:56:27.449194+0000","last_undegraded":"2024-04-02T04:56:27.449194+0000","last_fullsized":"2024-04-02T04:56:27.449194+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:15.169187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:56:15.169187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:22.096352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"402'26","reported_seq":793,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266510+0000","last_change":"2024-04-02T04:56:02.092153+0000","last_active":"2024-04-02T04:56:19.266510+0000","last_peered":"2024-04-02T04:56:19.266510+0000","last_clean":"2024-04-02T04:56:19.266510+0000","last_became_active":"2024-04-02T04:56:02.035603+0000","last_became_peered":"2024-04-02T04:56:02.035603+0000","last_unstale":"2024-04-02T04:56:19.266510+0000","last_undegraded":"2024-04-02T04:56:19.266510+0000","last_fullsized":"2024-04-02T04:56:19.266510+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'20","last_scrub_stamp":"2024-04-02T04:53:04.790916+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.790916+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":971,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449254+0000","last_change":"2024-04-02T04:56:02.030381+0000","last_active":"2024-04-02T04:56:27.449254+0000","last_peered":"2024-04-02T04:56:27.449254+0000","last_clean":"2024-04-02T04:56:27.449254+0000","last_became_active":"2024-04-02T04:56:02.030010+0000","last_became_peered":"2024-04-02T04:56:02.030010+0000","last_unstale":"2024-04-02T04:56:27.449254+0000","last_undegraded":"2024-04-02T04:56:27.449254+0000","last_fullsized":"2024-04-02T04:56:27.449254+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.301076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.301076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":945,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452327+0000","last_change":"2024-04-02T04:56:14.151593+0000","last_active":"2024-04-02T04:56:27.452327+0000","last_peered":"2024-04-02T04:56:27.452327+0000","last_clean":"2024-04-02T04:56:27.452327+0000","last_became_active":"2024-04-02T04:44:00.344957+0000","last_became_peered":"2024-04-02T04:44:00.344957+0000","last_unstale":"2024-04-02T04:56:27.452327+0000","last_undegraded":"2024-04-02T04:56:27.452327+0000","last_fullsized":"2024-04-02T04:56:27.452327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:14.151489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:14.151489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:33.073818+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"101'128","reported_seq":911,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545308+0000","last_change":"2024-04-02T04:55:34.875219+0000","last_active":"2024-04-02T04:56:27.545308+0000","last_peered":"2024-04-02T04:56:27.545308+0000","last_clean":"2024-04-02T04:56:27.545308+0000","last_became_active":"2024-04-02T04:55:32.987580+0000","last_became_peered":"2024-04-02T04:55:32.987580+0000","last_unstale":"2024-04-02T04:56:27.545308+0000","last_undegraded":"2024-04-02T04:56:27.545308+0000","last_fullsized":"2024-04-02T04:56:27.545308+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:34.875105+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:34.875105+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.875105+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:46.029802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.6","version":"478'11","reported_seq":892,"reported_epoch":530,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-02T04:56:27.448855+0000","last_change":"2024-04-02T04:56:02.991386+0000","last_active":"2024-04-02T04:56:27.448855+0000","last_peered":"2024-04-02T04:56:27.448855+0000","last_clean":"2024-04-02T04:55:59.966108+0000","last_became_active":"2024-04-02T04:56:02.985514+0000","last_became_peered":"2024-04-02T04:56:02.985514+0000","last_unstale":"2024-04-02T04:56:27.448855+0000","last_undegraded":"2024-04-02T04:56:27.448855+0000","last_fullsized":"2024-04-02T04:56:27.448855+0000","mapping_epoch":514,"log_start":"478'11","ondisk_log_start":"478'11","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'9","last_scrub_stamp":"2024-04-02T04:51:29.324015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:51:29.324015+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,3],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"2,3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":828,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445131+0000","last_change":"2024-04-02T04:54:09.648490+0000","last_active":"2024-04-02T04:56:27.445131+0000","last_peered":"2024-04-02T04:56:27.445131+0000","last_clean":"2024-04-02T04:56:27.445131+0000","last_became_active":"2024-04-02T04:53:54.916707+0000","last_became_peered":"2024-04-02T04:53:54.916707+0000","last_unstale":"2024-04-02T04:56:27.445131+0000","last_undegraded":"2024-04-02T04:56:27.445131+0000","last_fullsized":"2024-04-02T04:56:27.445131+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T04:54:09.648292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:09.648292+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"402'48","reported_seq":838,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448973+0000","last_change":"2024-04-02T04:56:02.078159+0000","last_active":"2024-04-02T04:56:27.448973+0000","last_peered":"2024-04-02T04:56:27.448973+0000","last_clean":"2024-04-02T04:56:27.448973+0000","last_became_active":"2024-04-02T04:56:02.037231+0000","last_became_peered":"2024-04-02T04:56:02.037231+0000","last_unstale":"2024-04-02T04:56:27.448973+0000","last_undegraded":"2024-04-02T04:56:27.448973+0000","last_fullsized":"2024-04-02T04:56:27.448973+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'48","last_scrub_stamp":"2024-04-02T04:54:21.140559+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:54:21.140559+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":976,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449047+0000","last_change":"2024-04-02T04:56:02.026219+0000","last_active":"2024-04-02T04:56:27.449047+0000","last_peered":"2024-04-02T04:56:27.449047+0000","last_clean":"2024-04-02T04:56:27.449047+0000","last_became_active":"2024-04-02T04:56:02.025505+0000","last_became_peered":"2024-04-02T04:56:02.025505+0000","last_unstale":"2024-04-02T04:56:27.449047+0000","last_undegraded":"2024-04-02T04:56:27.449047+0000","last_fullsized":"2024-04-02T04:56:27.449047+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:26.486667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:26.486667+0000","last_clean_scrub_stamp":"2024-04-02T04:55:26.486667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:42.987793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":901,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.549742+0000","last_change":"2024-04-02T04:56:02.035299+0000","last_active":"2024-04-02T04:56:27.549742+0000","last_peered":"2024-04-02T04:56:27.549742+0000","last_clean":"2024-04-02T04:56:27.549742+0000","last_became_active":"2024-04-02T04:56:02.034957+0000","last_became_peered":"2024-04-02T04:56:02.034957+0000","last_unstale":"2024-04-02T04:56:27.549742+0000","last_undegraded":"2024-04-02T04:56:27.549742+0000","last_fullsized":"2024-04-02T04:56:27.549742+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:07.785753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.785753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":174,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451917+0000","last_change":"2024-04-02T04:54:55.957462+0000","last_active":"2024-04-02T04:56:27.451917+0000","last_peered":"2024-04-02T04:56:27.451917+0000","last_clean":"2024-04-02T04:56:27.451917+0000","last_became_active":"2024-04-02T04:54:55.956870+0000","last_became_peered":"2024-04-02T04:54:55.956870+0000","last_unstale":"2024-04-02T04:56:27.451917+0000","last_undegraded":"2024-04-02T04:56:27.451917+0000","last_fullsized":"2024-04-02T04:56:27.451917+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:03.659593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:53:03.659593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"478'8","reported_seq":366,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922289+0000","last_change":"2024-04-02T04:55:57.971110+0000","last_active":"2024-04-02T04:56:19.922289+0000","last_peered":"2024-04-02T04:56:19.922289+0000","last_clean":"2024-04-02T04:56:19.922289+0000","last_became_active":"2024-04-02T04:55:57.970009+0000","last_became_peered":"2024-04-02T04:55:57.970009+0000","last_unstale":"2024-04-02T04:56:19.922289+0000","last_undegraded":"2024-04-02T04:56:19.922289+0000","last_fullsized":"2024-04-02T04:56:19.922289+0000","mapping_epoch":394,"log_start":"478'8","ondisk_log_start":"478'8","created":135,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:24.235462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.235462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":15,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":799,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918361+0000","last_change":"2024-04-02T04:52:42.441055+0000","last_active":"2024-04-02T04:56:19.918361+0000","last_peered":"2024-04-02T04:56:19.918361+0000","last_clean":"2024-04-02T04:56:19.918361+0000","last_became_active":"2024-04-02T04:49:26.245820+0000","last_became_peered":"2024-04-02T04:49:26.245820+0000","last_unstale":"2024-04-02T04:56:19.918361+0000","last_undegraded":"2024-04-02T04:56:19.918361+0000","last_fullsized":"2024-04-02T04:56:19.918361+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:42.440852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:42.440852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"402'30","reported_seq":1010,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445870+0000","last_change":"2024-04-02T04:54:03.566423+0000","last_active":"2024-04-02T04:56:27.445870+0000","last_peered":"2024-04-02T04:56:27.445870+0000","last_clean":"2024-04-02T04:56:27.445870+0000","last_became_active":"2024-04-02T04:53:54.917946+0000","last_became_peered":"2024-04-02T04:53:54.917946+0000","last_unstale":"2024-04-02T04:56:27.445870+0000","last_undegraded":"2024-04-02T04:56:27.445870+0000","last_fullsized":"2024-04-02T04:56:27.445870+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:54:03.566030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.566030+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":443,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.913316+0000","last_change":"2024-04-02T04:54:18.249066+0000","last_active":"2024-04-02T04:56:19.913316+0000","last_peered":"2024-04-02T04:56:19.913316+0000","last_clean":"2024-04-02T04:56:19.913316+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T04:56:19.913316+0000","last_undegraded":"2024-04-02T04:56:19.913316+0000","last_fullsized":"2024-04-02T04:56:19.913316+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T04:54:18.248880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"85'11","reported_seq":1110,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450263+0000","last_change":"2024-04-02T04:55:36.612229+0000","last_active":"2024-04-02T04:56:27.450263+0000","last_peered":"2024-04-02T04:56:27.450263+0000","last_clean":"2024-04-02T04:56:27.450263+0000","last_became_active":"2024-04-02T04:43:58.296903+0000","last_became_peered":"2024-04-02T04:43:58.296903+0000","last_unstale":"2024-04-02T04:56:27.450263+0000","last_undegraded":"2024-04-02T04:56:27.450263+0000","last_fullsized":"2024-04-02T04:56:27.450263+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:55:36.612122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:36.612122+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:47.526197+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":822,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.923590+0000","last_change":"2024-04-02T04:56:02.013868+0000","last_active":"2024-04-02T04:56:19.923590+0000","last_peered":"2024-04-02T04:56:19.923590+0000","last_clean":"2024-04-02T04:56:19.923590+0000","last_became_active":"2024-04-02T04:56:02.013446+0000","last_became_peered":"2024-04-02T04:56:02.013446+0000","last_unstale":"2024-04-02T04:56:19.923590+0000","last_undegraded":"2024-04-02T04:56:19.923590+0000","last_fullsized":"2024-04-02T04:56:19.923590+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:29.998572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:26.746930+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.998572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"101'128","reported_seq":656,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915868+0000","last_change":"2024-04-02T04:56:16.939050+0000","last_active":"2024-04-02T04:56:19.915868+0000","last_peered":"2024-04-02T04:56:19.915868+0000","last_clean":"2024-04-02T04:56:19.915868+0000","last_became_active":"2024-04-02T04:56:04.955998+0000","last_became_peered":"2024-04-02T04:56:04.955998+0000","last_unstale":"2024-04-02T04:56:19.915868+0000","last_undegraded":"2024-04-02T04:56:19.915868+0000","last_fullsized":"2024-04-02T04:56:19.915868+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:56:16.938940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.938940+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:41.684861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.8","version":"138'2","reported_seq":885,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545254+0000","last_change":"2024-04-02T04:55:32.851607+0000","last_active":"2024-04-02T04:56:27.545254+0000","last_peered":"2024-04-02T04:56:27.545254+0000","last_clean":"2024-04-02T04:56:27.545254+0000","last_became_active":"2024-04-02T04:55:21.951023+0000","last_became_peered":"2024-04-02T04:55:21.951023+0000","last_unstale":"2024-04-02T04:56:27.545254+0000","last_undegraded":"2024-04-02T04:56:27.545254+0000","last_fullsized":"2024-04-02T04:56:27.545254+0000","mapping_epoch":447,"log_start":"138'2","ondisk_log_start":"138'2","created":135,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-04-02T04:55:32.851320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:32.851320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":4,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:46.866419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":813,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928399+0000","last_change":"2024-04-02T04:56:02.021760+0000","last_active":"2024-04-02T04:56:19.928399+0000","last_peered":"2024-04-02T04:56:19.928399+0000","last_clean":"2024-04-02T04:56:19.928399+0000","last_became_active":"2024-04-02T04:56:02.021365+0000","last_became_peered":"2024-04-02T04:56:02.021365+0000","last_unstale":"2024-04-02T04:56:19.928399+0000","last_undegraded":"2024-04-02T04:56:19.928399+0000","last_fullsized":"2024-04-02T04:56:19.928399+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.941615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":956,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.446315+0000","last_change":"2024-04-02T04:56:25.082036+0000","last_active":"2024-04-02T04:56:27.446315+0000","last_peered":"2024-04-02T04:56:27.446315+0000","last_clean":"2024-04-02T04:56:27.446315+0000","last_became_active":"2024-04-02T04:54:56.011012+0000","last_became_peered":"2024-04-02T04:54:56.011012+0000","last_unstale":"2024-04-02T04:56:27.446315+0000","last_undegraded":"2024-04-02T04:56:27.446315+0000","last_fullsized":"2024-04-02T04:56:27.446315+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:25.081828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:56:25.081828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:32.650321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"402'30","reported_seq":992,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267777+0000","last_change":"2024-04-02T04:53:07.411481+0000","last_active":"2024-04-02T04:56:19.267777+0000","last_peered":"2024-04-02T04:56:19.267777+0000","last_clean":"2024-04-02T04:56:19.267777+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:56:19.267777+0000","last_undegraded":"2024-04-02T04:56:19.267777+0000","last_fullsized":"2024-04-02T04:56:19.267777+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'18","last_scrub_stamp":"2024-04-02T04:53:07.411390+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:53:07.411390+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":960,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928472+0000","last_change":"2024-04-02T04:54:08.978978+0000","last_active":"2024-04-02T04:56:19.928472+0000","last_peered":"2024-04-02T04:56:19.928472+0000","last_clean":"2024-04-02T04:56:19.928472+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T04:56:19.928472+0000","last_undegraded":"2024-04-02T04:56:19.928472+0000","last_fullsized":"2024-04-02T04:56:19.928472+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.978775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.978775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":119,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448251+0000","last_change":"2024-04-02T04:56:02.096558+0000","last_active":"2024-04-02T04:56:27.448251+0000","last_peered":"2024-04-02T04:56:27.448251+0000","last_clean":"2024-04-02T04:56:27.448251+0000","last_became_active":"2024-04-02T04:56:02.047540+0000","last_became_peered":"2024-04-02T04:56:02.047540+0000","last_unstale":"2024-04-02T04:56:27.448251+0000","last_undegraded":"2024-04-02T04:56:27.448251+0000","last_fullsized":"2024-04-02T04:56:27.448251+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:55:13.047925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:13.047925+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:39.136718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":396,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921162+0000","last_change":"2024-04-02T04:53:55.052233+0000","last_active":"2024-04-02T04:56:19.921162+0000","last_peered":"2024-04-02T04:56:19.921162+0000","last_clean":"2024-04-02T04:56:19.921162+0000","last_became_active":"2024-04-02T04:53:54.936665+0000","last_became_peered":"2024-04-02T04:53:54.936665+0000","last_unstale":"2024-04-02T04:56:19.921162+0000","last_undegraded":"2024-04-02T04:56:19.921162+0000","last_fullsized":"2024-04-02T04:56:19.921162+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:55.052156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.052156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'128","reported_seq":911,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.446273+0000","last_change":"2024-04-02T04:55:33.603363+0000","last_active":"2024-04-02T04:56:27.446273+0000","last_peered":"2024-04-02T04:56:27.446273+0000","last_clean":"2024-04-02T04:56:27.446273+0000","last_became_active":"2024-04-02T04:55:32.989836+0000","last_became_peered":"2024-04-02T04:55:32.989836+0000","last_unstale":"2024-04-02T04:56:27.446273+0000","last_undegraded":"2024-04-02T04:56:27.446273+0000","last_fullsized":"2024-04-02T04:56:27.446273+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.603255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.603255+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:58.924548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":226,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265171+0000","last_change":"2024-04-02T04:55:18.061491+0000","last_active":"2024-04-02T04:56:19.265171+0000","last_peered":"2024-04-02T04:56:19.265171+0000","last_clean":"2024-04-02T04:56:19.265171+0000","last_became_active":"2024-04-02T04:55:18.060898+0000","last_became_peered":"2024-04-02T04:55:18.060898+0000","last_unstale":"2024-04-02T04:56:19.265171+0000","last_undegraded":"2024-04-02T04:56:19.265171+0000","last_fullsized":"2024-04-02T04:56:19.265171+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:04.638640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.638640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":806,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449828+0000","last_change":"2024-04-02T04:54:28.733964+0000","last_active":"2024-04-02T04:56:27.449828+0000","last_peered":"2024-04-02T04:56:27.449828+0000","last_clean":"2024-04-02T04:56:27.449828+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:56:27.449828+0000","last_undegraded":"2024-04-02T04:56:27.449828+0000","last_fullsized":"2024-04-02T04:56:27.449828+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.733831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.733831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"516'15","reported_seq":856,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.913383+0000","last_change":"2024-04-02T04:56:02.027806+0000","last_active":"2024-04-02T04:56:19.913383+0000","last_peered":"2024-04-02T04:56:19.913383+0000","last_clean":"2024-04-02T04:56:19.913383+0000","last_became_active":"2024-04-02T04:56:02.000974+0000","last_became_peered":"2024-04-02T04:56:02.000974+0000","last_unstale":"2024-04-02T04:56:19.913383+0000","last_undegraded":"2024-04-02T04:56:19.913383+0000","last_fullsized":"2024-04-02T04:56:19.913383+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'14","last_scrub_stamp":"2024-04-02T04:53:56.111652+0000","last_deep_scrub":"105'12","last_deep_scrub_stamp":"2024-04-02T04:46:25.694340+0000","last_clean_scrub_stamp":"2024-04-02T04:53:56.111652+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"402'23","reported_seq":791,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925587+0000","last_change":"2024-04-02T04:56:02.065093+0000","last_active":"2024-04-02T04:56:19.925587+0000","last_peered":"2024-04-02T04:56:19.925587+0000","last_clean":"2024-04-02T04:56:19.925587+0000","last_became_active":"2024-04-02T04:56:02.005178+0000","last_became_peered":"2024-04-02T04:56:02.005178+0000","last_unstale":"2024-04-02T04:56:19.925587+0000","last_undegraded":"2024-04-02T04:56:19.925587+0000","last_fullsized":"2024-04-02T04:56:19.925587+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'23","last_scrub_stamp":"2024-04-02T04:54:17.131283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.131283+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":966,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451765+0000","last_change":"2024-04-02T04:55:24.503457+0000","last_active":"2024-04-02T04:56:27.451765+0000","last_peered":"2024-04-02T04:56:27.451765+0000","last_clean":"2024-04-02T04:56:27.451765+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:56:27.451765+0000","last_undegraded":"2024-04-02T04:56:27.451765+0000","last_fullsized":"2024-04-02T04:56:27.451765+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:55:24.503221+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.503221+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:31.924522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":964,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445710+0000","last_change":"2024-04-02T04:56:02.011216+0000","last_active":"2024-04-02T04:56:27.445710+0000","last_peered":"2024-04-02T04:56:27.445710+0000","last_clean":"2024-04-02T04:56:27.445710+0000","last_became_active":"2024-04-02T04:56:02.010848+0000","last_became_peered":"2024-04-02T04:56:02.010848+0000","last_unstale":"2024-04-02T04:56:27.445710+0000","last_undegraded":"2024-04-02T04:56:27.445710+0000","last_fullsized":"2024-04-02T04:56:27.445710+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:05.619646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:05.619646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":951,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448668+0000","last_change":"2024-04-02T04:56:16.123612+0000","last_active":"2024-04-02T04:56:27.448668+0000","last_peered":"2024-04-02T04:56:27.448668+0000","last_clean":"2024-04-02T04:56:27.448668+0000","last_became_active":"2024-04-02T04:56:02.014236+0000","last_became_peered":"2024-04-02T04:56:02.014236+0000","last_unstale":"2024-04-02T04:56:27.448668+0000","last_undegraded":"2024-04-02T04:56:27.448668+0000","last_fullsized":"2024-04-02T04:56:27.448668+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:16.123504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.123504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:44.365452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"101'128","reported_seq":436,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924240+0000","last_change":"2024-04-02T04:56:04.960957+0000","last_active":"2024-04-02T04:56:19.924240+0000","last_peered":"2024-04-02T04:56:19.924240+0000","last_clean":"2024-04-02T04:56:19.924240+0000","last_became_active":"2024-04-02T04:56:04.960532+0000","last_became_peered":"2024-04-02T04:56:04.960532+0000","last_unstale":"2024-04-02T04:56:19.924240+0000","last_undegraded":"2024-04-02T04:56:19.924240+0000","last_fullsized":"2024-04-02T04:56:19.924240+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.026882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.026882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:44.757246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.a","version":"478'12","reported_seq":890,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545836+0000","last_change":"2024-04-02T04:55:19.765903+0000","last_active":"2024-04-02T04:56:27.545836+0000","last_peered":"2024-04-02T04:56:27.545836+0000","last_clean":"2024-04-02T04:56:27.545836+0000","last_became_active":"2024-04-02T04:55:14.116832+0000","last_became_peered":"2024-04-02T04:55:14.116832+0000","last_unstale":"2024-04-02T04:56:27.545836+0000","last_undegraded":"2024-04-02T04:56:27.545836+0000","last_fullsized":"2024-04-02T04:56:27.545836+0000","mapping_epoch":135,"log_start":"262'6","ondisk_log_start":"262'6","created":135,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'6","last_scrub_stamp":"2024-04-02T04:55:19.765558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:19.765558+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":16,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:29.363290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":807,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924113+0000","last_change":"2024-04-02T04:54:26.868758+0000","last_active":"2024-04-02T04:56:19.924113+0000","last_peered":"2024-04-02T04:56:19.924113+0000","last_clean":"2024-04-02T04:56:19.924113+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:56:19.924113+0000","last_undegraded":"2024-04-02T04:56:19.924113+0000","last_fullsized":"2024-04-02T04:56:19.924113+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.868528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.868528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"451'15","reported_seq":1129,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.889131+0000","last_change":"2024-04-02T04:55:27.022983+0000","last_active":"2024-04-02T04:56:19.889131+0000","last_peered":"2024-04-02T04:56:19.889131+0000","last_clean":"2024-04-02T04:56:19.889131+0000","last_became_active":"2024-04-02T04:54:55.961735+0000","last_became_peered":"2024-04-02T04:54:55.961735+0000","last_unstale":"2024-04-02T04:56:19.889131+0000","last_undegraded":"2024-04-02T04:56:19.889131+0000","last_fullsized":"2024-04-02T04:56:19.889131+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'15","last_scrub_stamp":"2024-04-02T04:55:27.022873+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:55:27.022873+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:50.369764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"451'60","reported_seq":1232,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450071+0000","last_change":"2024-04-02T04:54:55.984048+0000","last_active":"2024-04-02T04:56:27.450071+0000","last_peered":"2024-04-02T04:56:27.450071+0000","last_clean":"2024-04-02T04:56:27.450071+0000","last_became_active":"2024-04-02T04:54:55.955778+0000","last_became_peered":"2024-04-02T04:54:55.955778+0000","last_unstale":"2024-04-02T04:56:27.450071+0000","last_undegraded":"2024-04-02T04:56:27.450071+0000","last_fullsized":"2024-04-02T04:56:27.450071+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'47","last_scrub_stamp":"2024-04-02T04:52:51.249489+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:52:51.249489+0000","objects_scrubbed":11,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":431,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925633+0000","last_change":"2024-04-02T04:54:17.219530+0000","last_active":"2024-04-02T04:56:19.925633+0000","last_peered":"2024-04-02T04:56:19.925633+0000","last_clean":"2024-04-02T04:56:19.925633+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T04:56:19.925633+0000","last_undegraded":"2024-04-02T04:56:19.925633+0000","last_fullsized":"2024-04-02T04:56:19.925633+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:17.219353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.219353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":970,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.048065+0000","last_change":"2024-04-02T04:56:27.048142+0000","last_active":"2024-04-02T04:56:27.048065+0000","last_peered":"2024-04-02T04:56:27.048065+0000","last_clean":"2024-04-02T04:56:27.048065+0000","last_became_active":"2024-04-02T04:54:56.004769+0000","last_became_peered":"2024-04-02T04:54:56.004769+0000","last_unstale":"2024-04-02T04:56:27.048065+0000","last_undegraded":"2024-04-02T04:56:27.048065+0000","last_fullsized":"2024-04-02T04:56:27.048065+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:56:27.047860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:27.047860+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:33.252126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":945,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449298+0000","last_change":"2024-04-02T04:53:53.356428+0000","last_active":"2024-04-02T04:56:27.449298+0000","last_peered":"2024-04-02T04:56:27.449298+0000","last_clean":"2024-04-02T04:56:27.449298+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:56:27.449298+0000","last_undegraded":"2024-04-02T04:56:27.449298+0000","last_fullsized":"2024-04-02T04:56:27.449298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:53.356300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:53:53.356300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'128","reported_seq":910,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450293+0000","last_change":"2024-04-02T04:55:33.448870+0000","last_active":"2024-04-02T04:56:27.450293+0000","last_peered":"2024-04-02T04:56:27.450293+0000","last_clean":"2024-04-02T04:56:27.450293+0000","last_became_active":"2024-04-02T04:55:32.982310+0000","last_became_peered":"2024-04-02T04:55:32.982310+0000","last_unstale":"2024-04-02T04:56:27.450293+0000","last_undegraded":"2024-04-02T04:56:27.450293+0000","last_fullsized":"2024-04-02T04:56:27.450293+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.448768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.448768+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:57.152452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":812,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265288+0000","last_change":"2024-04-02T04:55:28.979441+0000","last_active":"2024-04-02T04:56:19.265288+0000","last_peered":"2024-04-02T04:56:19.265288+0000","last_clean":"2024-04-02T04:56:19.265288+0000","last_became_active":"2024-04-02T04:53:54.957413+0000","last_became_peered":"2024-04-02T04:53:54.957413+0000","last_unstale":"2024-04-02T04:56:19.265288+0000","last_undegraded":"2024-04-02T04:56:19.265288+0000","last_fullsized":"2024-04-02T04:56:19.265288+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:28.979324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:28.979324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:40.657043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":958,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267962+0000","last_change":"2024-04-02T04:54:55.954958+0000","last_active":"2024-04-02T04:56:19.267962+0000","last_peered":"2024-04-02T04:56:19.267962+0000","last_clean":"2024-04-02T04:56:19.267962+0000","last_became_active":"2024-04-02T04:54:55.950537+0000","last_became_peered":"2024-04-02T04:54:55.950537+0000","last_unstale":"2024-04-02T04:56:19.267962+0000","last_undegraded":"2024-04-02T04:56:19.267962+0000","last_fullsized":"2024-04-02T04:56:19.267962+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.310943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.310943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":429,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914946+0000","last_change":"2024-04-02T04:54:19.199301+0000","last_active":"2024-04-02T04:56:19.914946+0000","last_peered":"2024-04-02T04:56:19.914946+0000","last_clean":"2024-04-02T04:56:19.914946+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T04:56:19.914946+0000","last_undegraded":"2024-04-02T04:56:19.914946+0000","last_fullsized":"2024-04-02T04:56:19.914946+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.199212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.199212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"402'34","reported_seq":1029,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265510+0000","last_change":"2024-04-02T04:56:02.061816+0000","last_active":"2024-04-02T04:56:19.265510+0000","last_peered":"2024-04-02T04:56:19.265510+0000","last_clean":"2024-04-02T04:56:19.265510+0000","last_became_active":"2024-04-02T04:56:01.998567+0000","last_became_peered":"2024-04-02T04:56:01.998567+0000","last_unstale":"2024-04-02T04:56:19.265510+0000","last_undegraded":"2024-04-02T04:56:19.265510+0000","last_fullsized":"2024-04-02T04:56:19.265510+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'20","last_scrub_stamp":"2024-04-02T04:52:33.272448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:33.272448+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":964,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449808+0000","last_change":"2024-04-02T04:56:02.038902+0000","last_active":"2024-04-02T04:56:27.449808+0000","last_peered":"2024-04-02T04:56:27.449808+0000","last_clean":"2024-04-02T04:56:27.449808+0000","last_became_active":"2024-04-02T04:56:02.038468+0000","last_became_peered":"2024-04-02T04:56:02.038468+0000","last_unstale":"2024-04-02T04:56:27.449808+0000","last_undegraded":"2024-04-02T04:56:27.449808+0000","last_fullsized":"2024-04-02T04:56:27.449808+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:21.446624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:21.446624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":807,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919786+0000","last_change":"2024-04-02T04:54:20.899430+0000","last_active":"2024-04-02T04:56:19.919786+0000","last_peered":"2024-04-02T04:56:19.919786+0000","last_clean":"2024-04-02T04:56:19.919786+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:56:19.919786+0000","last_undegraded":"2024-04-02T04:56:19.919786+0000","last_fullsized":"2024-04-02T04:56:19.919786+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:20.898842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:20.898842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":952,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924490+0000","last_change":"2024-04-02T04:54:00.910076+0000","last_active":"2024-04-02T04:56:19.924490+0000","last_peered":"2024-04-02T04:56:19.924490+0000","last_clean":"2024-04-02T04:56:19.924490+0000","last_became_active":"2024-04-02T04:53:54.934952+0000","last_became_peered":"2024-04-02T04:53:54.934952+0000","last_unstale":"2024-04-02T04:56:19.924490+0000","last_undegraded":"2024-04-02T04:56:19.924490+0000","last_fullsized":"2024-04-02T04:56:19.924490+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.909025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.909025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":708,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914048+0000","last_change":"2024-04-02T04:55:29.149388+0000","last_active":"2024-04-02T04:56:19.914048+0000","last_peered":"2024-04-02T04:56:19.914048+0000","last_clean":"2024-04-02T04:56:19.914048+0000","last_became_active":"2024-04-02T04:55:27.978786+0000","last_became_peered":"2024-04-02T04:55:27.978786+0000","last_unstale":"2024-04-02T04:56:19.914048+0000","last_undegraded":"2024-04-02T04:56:19.914048+0000","last_fullsized":"2024-04-02T04:56:19.914048+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:29.149138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:29.149138+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:58.305426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":851,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546281+0000","last_change":"2024-04-02T04:56:02.029767+0000","last_active":"2024-04-02T04:56:27.546281+0000","last_peered":"2024-04-02T04:56:27.546281+0000","last_clean":"2024-04-02T04:56:27.546281+0000","last_became_active":"2024-04-02T04:56:02.029404+0000","last_became_peered":"2024-04-02T04:56:02.029404+0000","last_unstale":"2024-04-02T04:56:27.546281+0000","last_undegraded":"2024-04-02T04:56:27.546281+0000","last_fullsized":"2024-04-02T04:56:27.546281+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:55.758993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.758993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"402'26","reported_seq":795,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922212+0000","last_change":"2024-04-02T04:56:18.130977+0000","last_active":"2024-04-02T04:56:19.922212+0000","last_peered":"2024-04-02T04:56:19.922212+0000","last_clean":"2024-04-02T04:56:19.922212+0000","last_became_active":"2024-04-02T04:56:02.007843+0000","last_became_peered":"2024-04-02T04:56:02.007843+0000","last_unstale":"2024-04-02T04:56:19.922212+0000","last_undegraded":"2024-04-02T04:56:19.922212+0000","last_fullsized":"2024-04-02T04:56:19.922212+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T04:56:18.130852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.130852+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:22.706510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452775+0000","last_change":"2024-04-02T04:56:02.029000+0000","last_active":"2024-04-02T04:56:27.452775+0000","last_peered":"2024-04-02T04:56:27.452775+0000","last_clean":"2024-04-02T04:56:27.452775+0000","last_became_active":"2024-04-02T04:56:02.028563+0000","last_became_peered":"2024-04-02T04:56:02.028563+0000","last_unstale":"2024-04-02T04:56:27.452775+0000","last_undegraded":"2024-04-02T04:56:27.452775+0000","last_fullsized":"2024-04-02T04:56:27.452775+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:35.117776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:55:35.117776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:42.212905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"101'128","reported_seq":715,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.913666+0000","last_change":"2024-04-02T04:55:34.217352+0000","last_active":"2024-04-02T04:56:19.913666+0000","last_peered":"2024-04-02T04:56:19.913666+0000","last_clean":"2024-04-02T04:56:19.913666+0000","last_became_active":"2024-04-02T04:55:32.995580+0000","last_became_peered":"2024-04-02T04:55:32.995580+0000","last_unstale":"2024-04-02T04:56:19.913666+0000","last_undegraded":"2024-04-02T04:56:19.913666+0000","last_fullsized":"2024-04-02T04:56:19.913666+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:34.217242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.217242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:37.277470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":703,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545195+0000","last_change":"2024-04-02T04:54:55.947450+0000","last_active":"2024-04-02T04:56:27.545195+0000","last_peered":"2024-04-02T04:56:27.545195+0000","last_clean":"2024-04-02T04:56:27.545195+0000","last_became_active":"2024-04-02T04:54:55.946927+0000","last_became_peered":"2024-04-02T04:54:55.946927+0000","last_unstale":"2024-04-02T04:56:27.545195+0000","last_undegraded":"2024-04-02T04:56:27.545195+0000","last_fullsized":"2024-04-02T04:56:27.545195+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:14.557167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:14.557167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":954,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452047+0000","last_change":"2024-04-02T04:54:55.961082+0000","last_active":"2024-04-02T04:56:27.452047+0000","last_peered":"2024-04-02T04:56:27.452047+0000","last_clean":"2024-04-02T04:56:27.452047+0000","last_became_active":"2024-04-02T04:54:55.960718+0000","last_became_peered":"2024-04-02T04:54:55.960718+0000","last_unstale":"2024-04-02T04:56:27.452047+0000","last_undegraded":"2024-04-02T04:56:27.452047+0000","last_fullsized":"2024-04-02T04:56:27.452047+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:59.385111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:59.385111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":977,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545263+0000","last_change":"2024-04-02T04:56:02.058776+0000","last_active":"2024-04-02T04:56:27.545263+0000","last_peered":"2024-04-02T04:56:27.545263+0000","last_clean":"2024-04-02T04:56:27.545263+0000","last_became_active":"2024-04-02T04:56:02.057830+0000","last_became_peered":"2024-04-02T04:56:02.057830+0000","last_unstale":"2024-04-02T04:56:27.545263+0000","last_undegraded":"2024-04-02T04:56:27.545263+0000","last_fullsized":"2024-04-02T04:56:27.545263+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:13.826925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:55:13.826925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:37.268559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"402'42","reported_seq":1035,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448546+0000","last_change":"2024-04-02T04:56:26.536234+0000","last_active":"2024-04-02T04:56:27.448546+0000","last_peered":"2024-04-02T04:56:27.448546+0000","last_clean":"2024-04-02T04:56:27.448546+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:56:27.448546+0000","last_undegraded":"2024-04-02T04:56:27.448546+0000","last_fullsized":"2024-04-02T04:56:27.448546+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'42","last_scrub_stamp":"2024-04-02T04:56:26.536132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:26.536132+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:44.181206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":402,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925911+0000","last_change":"2024-04-02T04:56:02.033529+0000","last_active":"2024-04-02T04:56:19.925911+0000","last_peered":"2024-04-02T04:56:19.925911+0000","last_clean":"2024-04-02T04:56:19.925911+0000","last_became_active":"2024-04-02T04:56:02.033034+0000","last_became_peered":"2024-04-02T04:56:02.033034+0000","last_unstale":"2024-04-02T04:56:19.925911+0000","last_undegraded":"2024-04-02T04:56:19.925911+0000","last_fullsized":"2024-04-02T04:56:19.925911+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:56.024450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:56.024450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'128","reported_seq":915,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264849+0000","last_change":"2024-04-02T04:55:33.954742+0000","last_active":"2024-04-02T04:56:19.264849+0000","last_peered":"2024-04-02T04:56:19.264849+0000","last_clean":"2024-04-02T04:56:19.264849+0000","last_became_active":"2024-04-02T04:55:32.984097+0000","last_became_peered":"2024-04-02T04:55:32.984097+0000","last_unstale":"2024-04-02T04:56:19.264849+0000","last_undegraded":"2024-04-02T04:56:19.264849+0000","last_fullsized":"2024-04-02T04:56:19.264849+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.954638+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:28.185555+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.954638+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:48.943503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":813,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920498+0000","last_change":"2024-04-02T04:54:56.005135+0000","last_active":"2024-04-02T04:56:19.920498+0000","last_peered":"2024-04-02T04:56:19.920498+0000","last_clean":"2024-04-02T04:56:19.920498+0000","last_became_active":"2024-04-02T04:54:56.004256+0000","last_became_peered":"2024-04-02T04:54:56.004256+0000","last_unstale":"2024-04-02T04:56:19.920498+0000","last_undegraded":"2024-04-02T04:56:19.920498+0000","last_fullsized":"2024-04-02T04:56:19.920498+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.854283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":962,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449878+0000","last_change":"2024-04-02T04:56:15.453426+0000","last_active":"2024-04-02T04:56:27.449878+0000","last_peered":"2024-04-02T04:56:27.449878+0000","last_clean":"2024-04-02T04:56:27.449878+0000","last_became_active":"2024-04-02T04:56:02.023667+0000","last_became_peered":"2024-04-02T04:56:02.023667+0000","last_unstale":"2024-04-02T04:56:27.449878+0000","last_undegraded":"2024-04-02T04:56:27.449878+0000","last_fullsized":"2024-04-02T04:56:27.449878+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:15.453139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:56:15.453139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:37.773076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'35","reported_seq":1017,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452440+0000","last_change":"2024-04-02T04:55:27.706686+0000","last_active":"2024-04-02T04:56:27.452440+0000","last_peered":"2024-04-02T04:56:27.452440+0000","last_clean":"2024-04-02T04:56:27.452440+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:56:27.452440+0000","last_undegraded":"2024-04-02T04:56:27.452440+0000","last_fullsized":"2024-04-02T04:56:27.452440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T04:55:27.706548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:27.706548+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:49.716511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":175,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449058+0000","last_change":"2024-04-02T04:54:55.969695+0000","last_active":"2024-04-02T04:56:27.449058+0000","last_peered":"2024-04-02T04:56:27.449058+0000","last_clean":"2024-04-02T04:56:27.449058+0000","last_became_active":"2024-04-02T04:54:55.969352+0000","last_became_peered":"2024-04-02T04:54:55.969352+0000","last_unstale":"2024-04-02T04:56:27.449058+0000","last_undegraded":"2024-04-02T04:56:27.449058+0000","last_fullsized":"2024-04-02T04:56:27.449058+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:13.578411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:13.578411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":107,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449455+0000","last_change":"2024-04-02T04:56:02.025517+0000","last_active":"2024-04-02T04:56:27.449455+0000","last_peered":"2024-04-02T04:56:27.449455+0000","last_clean":"2024-04-02T04:56:27.449455+0000","last_became_active":"2024-04-02T04:56:02.004830+0000","last_became_peered":"2024-04-02T04:56:02.004830+0000","last_unstale":"2024-04-02T04:56:27.449455+0000","last_undegraded":"2024-04-02T04:56:27.449455+0000","last_fullsized":"2024-04-02T04:56:27.449455+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:53:58.149611+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:45:41.707741+0000","last_clean_scrub_stamp":"2024-04-02T04:53:58.149611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"101'128","reported_seq":436,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924680+0000","last_change":"2024-04-02T04:56:04.951413+0000","last_active":"2024-04-02T04:56:19.924680+0000","last_peered":"2024-04-02T04:56:19.924680+0000","last_clean":"2024-04-02T04:56:19.924680+0000","last_became_active":"2024-04-02T04:56:04.950985+0000","last_became_peered":"2024-04-02T04:56:04.950985+0000","last_unstale":"2024-04-02T04:56:19.924680+0000","last_undegraded":"2024-04-02T04:56:19.924680+0000","last_fullsized":"2024-04-02T04:56:19.924680+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.982932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.982932+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:43.393010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":300,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915299+0000","last_change":"2024-04-02T04:55:19.317421+0000","last_active":"2024-04-02T04:56:19.915299+0000","last_peered":"2024-04-02T04:56:19.915299+0000","last_clean":"2024-04-02T04:56:19.915299+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T04:56:19.915299+0000","last_undegraded":"2024-04-02T04:56:19.915299+0000","last_fullsized":"2024-04-02T04:56:19.915299+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:19.317110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:19.317110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:32.005051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":955,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449522+0000","last_change":"2024-04-02T04:54:55.979795+0000","last_active":"2024-04-02T04:56:27.449522+0000","last_peered":"2024-04-02T04:56:27.449522+0000","last_clean":"2024-04-02T04:56:27.449522+0000","last_became_active":"2024-04-02T04:54:55.979416+0000","last_became_peered":"2024-04-02T04:54:55.979416+0000","last_unstale":"2024-04-02T04:56:27.449522+0000","last_undegraded":"2024-04-02T04:56:27.449522+0000","last_fullsized":"2024-04-02T04:56:27.449522+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.308301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.308301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"516'30","reported_seq":922,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.547031+0000","last_change":"2024-04-02T04:56:02.070819+0000","last_active":"2024-04-02T04:56:27.547031+0000","last_peered":"2024-04-02T04:56:27.547031+0000","last_clean":"2024-04-02T04:56:27.547031+0000","last_became_active":"2024-04-02T04:56:02.035037+0000","last_became_peered":"2024-04-02T04:56:02.035037+0000","last_unstale":"2024-04-02T04:56:27.547031+0000","last_undegraded":"2024-04-02T04:56:27.547031+0000","last_fullsized":"2024-04-02T04:56:27.547031+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'29","last_scrub_stamp":"2024-04-02T04:55:39.143350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:39.143350+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:07.974240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":974,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450019+0000","last_change":"2024-04-02T04:53:59.321337+0000","last_active":"2024-04-02T04:56:27.450019+0000","last_peered":"2024-04-02T04:56:27.450019+0000","last_clean":"2024-04-02T04:56:27.450019+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T04:56:27.450019+0000","last_undegraded":"2024-04-02T04:56:27.450019+0000","last_fullsized":"2024-04-02T04:56:27.450019+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:53:59.321220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.321220+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":957,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264991+0000","last_change":"2024-04-02T04:56:02.038697+0000","last_active":"2024-04-02T04:56:19.264991+0000","last_peered":"2024-04-02T04:56:19.264991+0000","last_clean":"2024-04-02T04:56:19.264991+0000","last_became_active":"2024-04-02T04:56:02.036172+0000","last_became_peered":"2024-04-02T04:56:02.036172+0000","last_unstale":"2024-04-02T04:56:19.264991+0000","last_undegraded":"2024-04-02T04:56:19.264991+0000","last_fullsized":"2024-04-02T04:56:19.264991+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:11.307248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:53:11.307248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"478'7","reported_seq":363,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914168+0000","last_change":"2024-04-02T04:55:54.065389+0000","last_active":"2024-04-02T04:56:19.914168+0000","last_peered":"2024-04-02T04:56:19.914168+0000","last_clean":"2024-04-02T04:56:19.914168+0000","last_became_active":"2024-04-02T04:55:54.064666+0000","last_became_peered":"2024-04-02T04:55:54.064666+0000","last_unstale":"2024-04-02T04:56:19.914168+0000","last_undegraded":"2024-04-02T04:56:19.914168+0000","last_fullsized":"2024-04-02T04:56:19.914168+0000","mapping_epoch":394,"log_start":"478'7","ondisk_log_start":"478'7","created":135,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'5","last_scrub_stamp":"2024-04-02T04:55:11.403972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:11.403972+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":7,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:38.640825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":811,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928222+0000","last_change":"2024-04-02T04:56:02.022016+0000","last_active":"2024-04-02T04:56:19.928222+0000","last_peered":"2024-04-02T04:56:19.928222+0000","last_clean":"2024-04-02T04:56:19.928222+0000","last_became_active":"2024-04-02T04:56:02.021356+0000","last_became_peered":"2024-04-02T04:56:02.021356+0000","last_unstale":"2024-04-02T04:56:19.928222+0000","last_undegraded":"2024-04-02T04:56:19.928222+0000","last_fullsized":"2024-04-02T04:56:19.928222+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:57.364241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:57.364241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"516'13","reported_seq":1121,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445801+0000","last_change":"2024-04-02T04:56:02.062017+0000","last_active":"2024-04-02T04:56:27.445801+0000","last_peered":"2024-04-02T04:56:27.445801+0000","last_clean":"2024-04-02T04:56:27.445801+0000","last_became_active":"2024-04-02T04:56:02.028732+0000","last_became_peered":"2024-04-02T04:56:02.028732+0000","last_unstale":"2024-04-02T04:56:27.445801+0000","last_undegraded":"2024-04-02T04:56:27.445801+0000","last_fullsized":"2024-04-02T04:56:27.445801+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:53:04.380713+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:53:04.380713+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"402'25","reported_seq":480,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921481+0000","last_change":"2024-04-02T04:55:16.314129+0000","last_active":"2024-04-02T04:56:19.921481+0000","last_peered":"2024-04-02T04:56:19.921481+0000","last_clean":"2024-04-02T04:56:19.921481+0000","last_became_active":"2024-04-02T04:53:54.927132+0000","last_became_peered":"2024-04-02T04:53:54.927132+0000","last_unstale":"2024-04-02T04:56:19.921481+0000","last_undegraded":"2024-04-02T04:56:19.921481+0000","last_fullsized":"2024-04-02T04:56:19.921481+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:55:16.314010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:16.314010+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":982,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452931+0000","last_change":"2024-04-02T04:53:59.519788+0000","last_active":"2024-04-02T04:56:27.452931+0000","last_peered":"2024-04-02T04:56:27.452931+0000","last_clean":"2024-04-02T04:56:27.452931+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T04:56:27.452931+0000","last_undegraded":"2024-04-02T04:56:27.452931+0000","last_fullsized":"2024-04-02T04:56:27.452931+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:53:59.519530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:59.519530+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":703,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915915+0000","last_change":"2024-04-02T04:56:02.008808+0000","last_active":"2024-04-02T04:56:19.915915+0000","last_peered":"2024-04-02T04:56:19.915915+0000","last_clean":"2024-04-02T04:56:19.915915+0000","last_became_active":"2024-04-02T04:56:02.008241+0000","last_became_peered":"2024-04-02T04:56:02.008241+0000","last_unstale":"2024-04-02T04:56:19.915915+0000","last_undegraded":"2024-04-02T04:56:19.915915+0000","last_fullsized":"2024-04-02T04:56:19.915915+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.088131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.088131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":817,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545509+0000","last_change":"2024-04-02T04:56:02.042468+0000","last_active":"2024-04-02T04:56:27.545509+0000","last_peered":"2024-04-02T04:56:27.545509+0000","last_clean":"2024-04-02T04:56:27.545509+0000","last_became_active":"2024-04-02T04:56:02.042131+0000","last_became_peered":"2024-04-02T04:56:02.042131+0000","last_unstale":"2024-04-02T04:56:27.545509+0000","last_undegraded":"2024-04-02T04:56:27.545509+0000","last_fullsized":"2024-04-02T04:56:27.545509+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:09.820697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:53:09.820697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"402'25","reported_seq":470,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915673+0000","last_change":"2024-04-02T04:55:14.301013+0000","last_active":"2024-04-02T04:56:19.915673+0000","last_peered":"2024-04-02T04:56:19.915673+0000","last_clean":"2024-04-02T04:56:19.915673+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T04:56:19.915673+0000","last_undegraded":"2024-04-02T04:56:19.915673+0000","last_fullsized":"2024-04-02T04:56:19.915673+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:55:14.300842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:14.300842+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:25.211354+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":970,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545426+0000","last_change":"2024-04-02T04:55:35.870955+0000","last_active":"2024-04-02T04:56:27.545426+0000","last_peered":"2024-04-02T04:56:27.545426+0000","last_clean":"2024-04-02T04:56:27.545426+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:56:27.545426+0000","last_undegraded":"2024-04-02T04:56:27.545426+0000","last_fullsized":"2024-04-02T04:56:27.545426+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:35.870866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:55:35.870866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:37.386776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":953,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.268070+0000","last_change":"2024-04-02T04:56:02.023034+0000","last_active":"2024-04-02T04:56:19.268070+0000","last_peered":"2024-04-02T04:56:19.268070+0000","last_clean":"2024-04-02T04:56:19.268070+0000","last_became_active":"2024-04-02T04:56:02.022662+0000","last_became_peered":"2024-04-02T04:56:02.022662+0000","last_unstale":"2024-04-02T04:56:19.268070+0000","last_undegraded":"2024-04-02T04:56:19.268070+0000","last_fullsized":"2024-04-02T04:56:19.268070+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_clean_scrub_stamp":"2024-04-02T04:54:35.398676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":943,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546707+0000","last_change":"2024-04-02T04:54:01.922789+0000","last_active":"2024-04-02T04:56:27.546707+0000","last_peered":"2024-04-02T04:56:27.546707+0000","last_clean":"2024-04-02T04:56:27.546707+0000","last_became_active":"2024-04-02T04:53:54.925132+0000","last_became_peered":"2024-04-02T04:53:54.925132+0000","last_unstale":"2024-04-02T04:56:27.546707+0000","last_undegraded":"2024-04-02T04:56:27.546707+0000","last_fullsized":"2024-04-02T04:56:27.546707+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.922344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.922344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":818,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448893+0000","last_change":"2024-04-02T04:56:02.047947+0000","last_active":"2024-04-02T04:56:27.448893+0000","last_peered":"2024-04-02T04:56:27.448893+0000","last_clean":"2024-04-02T04:56:27.448893+0000","last_became_active":"2024-04-02T04:56:02.047584+0000","last_became_peered":"2024-04-02T04:56:02.047584+0000","last_unstale":"2024-04-02T04:56:27.448893+0000","last_undegraded":"2024-04-02T04:56:27.448893+0000","last_fullsized":"2024-04-02T04:56:27.448893+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:31.495522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:31.495522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:33.682657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"402'8","reported_seq":983,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918466+0000","last_change":"2024-04-02T04:56:02.093705+0000","last_active":"2024-04-02T04:56:19.918466+0000","last_peered":"2024-04-02T04:56:19.918466+0000","last_clean":"2024-04-02T04:56:19.918466+0000","last_became_active":"2024-04-02T04:56:02.017218+0000","last_became_peered":"2024-04-02T04:56:02.017218+0000","last_unstale":"2024-04-02T04:56:19.918466+0000","last_undegraded":"2024-04-02T04:56:19.918466+0000","last_fullsized":"2024-04-02T04:56:19.918466+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'8","last_scrub_stamp":"2024-04-02T04:54:17.927334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.927334+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":959,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918676+0000","last_change":"2024-04-02T04:56:16.144378+0000","last_active":"2024-04-02T04:56:19.918676+0000","last_peered":"2024-04-02T04:56:19.918676+0000","last_clean":"2024-04-02T04:56:19.918676+0000","last_became_active":"2024-04-02T04:43:54.208939+0000","last_became_peered":"2024-04-02T04:43:54.208939+0000","last_unstale":"2024-04-02T04:56:19.918676+0000","last_undegraded":"2024-04-02T04:56:19.918676+0000","last_fullsized":"2024-04-02T04:56:19.918676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:16.144273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.144273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:42.315252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":713,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925940+0000","last_change":"2024-04-02T04:56:02.032704+0000","last_active":"2024-04-02T04:56:19.925940+0000","last_peered":"2024-04-02T04:56:19.925940+0000","last_clean":"2024-04-02T04:56:19.925940+0000","last_became_active":"2024-04-02T04:56:02.032262+0000","last_became_peered":"2024-04-02T04:56:02.032262+0000","last_unstale":"2024-04-02T04:56:19.925940+0000","last_undegraded":"2024-04-02T04:56:19.925940+0000","last_fullsized":"2024-04-02T04:56:19.925940+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:16.086780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:54:16.086780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":419,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922504+0000","last_change":"2024-04-02T04:54:03.204796+0000","last_active":"2024-04-02T04:56:19.922504+0000","last_peered":"2024-04-02T04:56:19.922504+0000","last_clean":"2024-04-02T04:56:19.922504+0000","last_became_active":"2024-04-02T04:53:54.933037+0000","last_became_peered":"2024-04-02T04:53:54.933037+0000","last_unstale":"2024-04-02T04:56:19.922504+0000","last_undegraded":"2024-04-02T04:56:19.922504+0000","last_fullsized":"2024-04-02T04:56:19.922504+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.204677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.204677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":812,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.919687+0000","last_change":"2024-04-02T04:55:26.900226+0000","last_active":"2024-04-02T04:56:19.919687+0000","last_peered":"2024-04-02T04:56:19.919687+0000","last_clean":"2024-04-02T04:56:19.919687+0000","last_became_active":"2024-04-02T04:54:56.001224+0000","last_became_peered":"2024-04-02T04:54:56.001224+0000","last_unstale":"2024-04-02T04:56:19.919687+0000","last_undegraded":"2024-04-02T04:56:19.919687+0000","last_fullsized":"2024-04-02T04:56:19.919687+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:26.900131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T04:55:26.900131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:49.548486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"402'19","reported_seq":435,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.916128+0000","last_change":"2024-04-02T04:53:57.032666+0000","last_active":"2024-04-02T04:56:19.916128+0000","last_peered":"2024-04-02T04:56:19.916128+0000","last_clean":"2024-04-02T04:56:19.916128+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T04:56:19.916128+0000","last_undegraded":"2024-04-02T04:56:19.916128+0000","last_fullsized":"2024-04-02T04:56:19.916128+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:53:57.032539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:53:57.032539+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":713,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914616+0000","last_change":"2024-04-02T04:56:01.998892+0000","last_active":"2024-04-02T04:56:19.914616+0000","last_peered":"2024-04-02T04:56:19.914616+0000","last_clean":"2024-04-02T04:56:19.914616+0000","last_became_active":"2024-04-02T04:56:01.998553+0000","last_became_peered":"2024-04-02T04:56:01.998553+0000","last_unstale":"2024-04-02T04:56:19.914616+0000","last_undegraded":"2024-04-02T04:56:19.914616+0000","last_fullsized":"2024-04-02T04:56:19.914616+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:02.769714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:02.769714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"516'15","reported_seq":222,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450180+0000","last_change":"2024-04-02T04:56:02.067529+0000","last_active":"2024-04-02T04:56:27.450180+0000","last_peered":"2024-04-02T04:56:27.450180+0000","last_clean":"2024-04-02T04:56:27.450180+0000","last_became_active":"2024-04-02T04:56:02.031453+0000","last_became_peered":"2024-04-02T04:56:02.031453+0000","last_unstale":"2024-04-02T04:56:27.450180+0000","last_undegraded":"2024-04-02T04:56:27.450180+0000","last_fullsized":"2024-04-02T04:56:27.450180+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'14","last_scrub_stamp":"2024-04-02T04:53:55.155578+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:53:55.155578+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":949,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.268060+0000","last_change":"2024-04-02T04:56:02.014756+0000","last_active":"2024-04-02T04:56:19.268060+0000","last_peered":"2024-04-02T04:56:19.268060+0000","last_clean":"2024-04-02T04:56:19.268060+0000","last_became_active":"2024-04-02T04:56:02.014127+0000","last_became_peered":"2024-04-02T04:56:02.014127+0000","last_unstale":"2024-04-02T04:56:19.268060+0000","last_undegraded":"2024-04-02T04:56:19.268060+0000","last_fullsized":"2024-04-02T04:56:19.268060+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.393399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:26.393399+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.393399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":573,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452363+0000","last_change":"2024-04-02T04:56:02.010903+0000","last_active":"2024-04-02T04:56:27.452363+0000","last_peered":"2024-04-02T04:56:27.452363+0000","last_clean":"2024-04-02T04:56:27.452363+0000","last_became_active":"2024-04-02T04:56:02.010397+0000","last_became_peered":"2024-04-02T04:56:02.010397+0000","last_unstale":"2024-04-02T04:56:27.452363+0000","last_undegraded":"2024-04-02T04:56:27.452363+0000","last_fullsized":"2024-04-02T04:56:27.452363+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.202179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.202179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"402'15","reported_seq":1000,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265563+0000","last_change":"2024-04-02T04:56:02.122165+0000","last_active":"2024-04-02T04:56:19.265563+0000","last_peered":"2024-04-02T04:56:19.265563+0000","last_clean":"2024-04-02T04:56:19.265563+0000","last_became_active":"2024-04-02T04:56:02.036547+0000","last_became_peered":"2024-04-02T04:56:02.036547+0000","last_unstale":"2024-04-02T04:56:19.265563+0000","last_undegraded":"2024-04-02T04:56:19.265563+0000","last_fullsized":"2024-04-02T04:56:19.265563+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'15","last_scrub_stamp":"2024-04-02T04:55:15.110870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:15.110870+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:42.362491+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":960,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266311+0000","last_change":"2024-04-02T04:56:02.010702+0000","last_active":"2024-04-02T04:56:19.266311+0000","last_peered":"2024-04-02T04:56:19.266311+0000","last_clean":"2024-04-02T04:56:19.266311+0000","last_became_active":"2024-04-02T04:56:02.009976+0000","last_became_peered":"2024-04-02T04:56:02.009976+0000","last_unstale":"2024-04-02T04:56:19.266311+0000","last_undegraded":"2024-04-02T04:56:19.266311+0000","last_fullsized":"2024-04-02T04:56:19.266311+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.331020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.331020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"516'15","reported_seq":1000,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.926353+0000","last_change":"2024-04-02T04:56:02.068211+0000","last_active":"2024-04-02T04:56:19.926353+0000","last_peered":"2024-04-02T04:56:19.926353+0000","last_clean":"2024-04-02T04:56:19.926353+0000","last_became_active":"2024-04-02T04:56:02.015343+0000","last_became_peered":"2024-04-02T04:56:02.015343+0000","last_unstale":"2024-04-02T04:56:19.926353+0000","last_undegraded":"2024-04-02T04:56:19.926353+0000","last_fullsized":"2024-04-02T04:56:19.926353+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'13","last_scrub_stamp":"2024-04-02T04:54:07.630056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.630056+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":951,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.453281+0000","last_change":"2024-04-02T04:56:17.099070+0000","last_active":"2024-04-02T04:56:27.453281+0000","last_peered":"2024-04-02T04:56:27.453281+0000","last_clean":"2024-04-02T04:56:27.453281+0000","last_became_active":"2024-04-02T04:54:56.000394+0000","last_became_peered":"2024-04-02T04:54:56.000394+0000","last_unstale":"2024-04-02T04:56:27.453281+0000","last_undegraded":"2024-04-02T04:56:27.453281+0000","last_fullsized":"2024-04-02T04:56:27.453281+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:17.098961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:17.098961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:45.485523+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":806,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922285+0000","last_change":"2024-04-02T04:55:23.873415+0000","last_active":"2024-04-02T04:56:19.922285+0000","last_peered":"2024-04-02T04:56:19.922285+0000","last_clean":"2024-04-02T04:56:19.922285+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:56:19.922285+0000","last_undegraded":"2024-04-02T04:56:19.922285+0000","last_fullsized":"2024-04-02T04:56:19.922285+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:23.873192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:23.873192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:32.624948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"402'21","reported_seq":498,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.926166+0000","last_change":"2024-04-02T04:56:02.056614+0000","last_active":"2024-04-02T04:56:19.926166+0000","last_peered":"2024-04-02T04:56:19.926166+0000","last_clean":"2024-04-02T04:56:19.926166+0000","last_became_active":"2024-04-02T04:56:01.998126+0000","last_became_peered":"2024-04-02T04:56:01.998126+0000","last_unstale":"2024-04-02T04:56:19.926166+0000","last_undegraded":"2024-04-02T04:56:19.926166+0000","last_fullsized":"2024-04-02T04:56:19.926166+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T04:55:20.279092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:20.279092+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:44.173843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":963,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922310+0000","last_change":"2024-04-02T04:56:02.020949+0000","last_active":"2024-04-02T04:56:19.922310+0000","last_peered":"2024-04-02T04:56:19.922310+0000","last_clean":"2024-04-02T04:56:19.922310+0000","last_became_active":"2024-04-02T04:56:02.020308+0000","last_became_peered":"2024-04-02T04:56:02.020308+0000","last_unstale":"2024-04-02T04:56:19.922310+0000","last_undegraded":"2024-04-02T04:56:19.922310+0000","last_fullsized":"2024-04-02T04:56:19.922310+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.859624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.859624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:19.937891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":948,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449149+0000","last_change":"2024-04-02T04:52:54.293779+0000","last_active":"2024-04-02T04:56:27.449149+0000","last_peered":"2024-04-02T04:56:27.449149+0000","last_clean":"2024-04-02T04:56:27.449149+0000","last_became_active":"2024-04-02T04:43:58.301671+0000","last_became_peered":"2024-04-02T04:43:58.301671+0000","last_unstale":"2024-04-02T04:56:27.449149+0000","last_undegraded":"2024-04-02T04:56:27.449149+0000","last_fullsized":"2024-04-02T04:56:27.449149+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:54.293646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:54.293646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":948,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264893+0000","last_change":"2024-04-02T04:54:56.005487+0000","last_active":"2024-04-02T04:56:19.264893+0000","last_peered":"2024-04-02T04:56:19.264893+0000","last_clean":"2024-04-02T04:56:19.264893+0000","last_became_active":"2024-04-02T04:54:56.005128+0000","last_became_peered":"2024-04-02T04:54:56.005128+0000","last_unstale":"2024-04-02T04:56:19.264893+0000","last_undegraded":"2024-04-02T04:56:19.264893+0000","last_fullsized":"2024-04-02T04:56:19.264893+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:53.965811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:51:53.965811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":813,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.446997+0000","last_change":"2024-04-02T04:56:18.079213+0000","last_active":"2024-04-02T04:56:27.446997+0000","last_peered":"2024-04-02T04:56:27.446997+0000","last_clean":"2024-04-02T04:56:27.446997+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:56:27.446997+0000","last_undegraded":"2024-04-02T04:56:27.446997+0000","last_fullsized":"2024-04-02T04:56:27.446997+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:18.079082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.079082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:43.594671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"402'32","reported_seq":1008,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448828+0000","last_change":"2024-04-02T04:53:58.561674+0000","last_active":"2024-04-02T04:56:27.448828+0000","last_peered":"2024-04-02T04:56:27.448828+0000","last_clean":"2024-04-02T04:56:27.448828+0000","last_became_active":"2024-04-02T04:53:54.918208+0000","last_became_peered":"2024-04-02T04:53:54.918208+0000","last_unstale":"2024-04-02T04:56:27.448828+0000","last_undegraded":"2024-04-02T04:56:27.448828+0000","last_fullsized":"2024-04-02T04:56:27.448828+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'22","last_scrub_stamp":"2024-04-02T04:53:58.561437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:53:58.561437+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":962,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451647+0000","last_change":"2024-04-02T04:54:55.988018+0000","last_active":"2024-04-02T04:56:27.451647+0000","last_peered":"2024-04-02T04:56:27.451647+0000","last_clean":"2024-04-02T04:56:27.451647+0000","last_became_active":"2024-04-02T04:54:55.987373+0000","last_became_peered":"2024-04-02T04:54:55.987373+0000","last_unstale":"2024-04-02T04:56:27.451647+0000","last_undegraded":"2024-04-02T04:56:27.451647+0000","last_fullsized":"2024-04-02T04:56:27.451647+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:05.438193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:05.438193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":949,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449976+0000","last_change":"2024-04-02T04:52:40.351590+0000","last_active":"2024-04-02T04:56:27.449976+0000","last_peered":"2024-04-02T04:56:27.449976+0000","last_clean":"2024-04-02T04:56:27.449976+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:56:27.449976+0000","last_undegraded":"2024-04-02T04:56:27.449976+0000","last_fullsized":"2024-04-02T04:56:27.449976+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:40.351381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:40.351381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":822,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265563+0000","last_change":"2024-04-02T04:56:02.028085+0000","last_active":"2024-04-02T04:56:19.265563+0000","last_peered":"2024-04-02T04:56:19.265563+0000","last_clean":"2024-04-02T04:56:19.265563+0000","last_became_active":"2024-04-02T04:56:02.027687+0000","last_became_peered":"2024-04-02T04:56:02.027687+0000","last_unstale":"2024-04-02T04:56:19.265563+0000","last_undegraded":"2024-04-02T04:56:19.265563+0000","last_fullsized":"2024-04-02T04:56:19.265563+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:27.015307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:54:27.015307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":817,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267294+0000","last_change":"2024-04-02T04:54:56.017323+0000","last_active":"2024-04-02T04:56:19.267294+0000","last_peered":"2024-04-02T04:56:19.267294+0000","last_clean":"2024-04-02T04:56:19.267294+0000","last_became_active":"2024-04-02T04:54:56.017129+0000","last_became_peered":"2024-04-02T04:54:56.017129+0000","last_unstale":"2024-04-02T04:56:19.267294+0000","last_undegraded":"2024-04-02T04:56:19.267294+0000","last_fullsized":"2024-04-02T04:56:19.267294+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:00.388684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:53:00.388684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"402'18","reported_seq":1012,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267299+0000","last_change":"2024-04-02T04:56:17.986393+0000","last_active":"2024-04-02T04:56:19.267299+0000","last_peered":"2024-04-02T04:56:19.267299+0000","last_clean":"2024-04-02T04:56:19.267299+0000","last_became_active":"2024-04-02T04:56:02.042783+0000","last_became_peered":"2024-04-02T04:56:02.042783+0000","last_unstale":"2024-04-02T04:56:19.267299+0000","last_undegraded":"2024-04-02T04:56:19.267299+0000","last_fullsized":"2024-04-02T04:56:19.267299+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'18","last_scrub_stamp":"2024-04-02T04:56:17.986089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:17.986089+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:27.034239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":174,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924748+0000","last_change":"2024-04-02T04:54:55.973248+0000","last_active":"2024-04-02T04:56:19.924748+0000","last_peered":"2024-04-02T04:56:19.924748+0000","last_clean":"2024-04-02T04:56:19.924748+0000","last_became_active":"2024-04-02T04:54:55.972841+0000","last_became_peered":"2024-04-02T04:54:55.972841+0000","last_unstale":"2024-04-02T04:56:19.924748+0000","last_undegraded":"2024-04-02T04:56:19.924748+0000","last_fullsized":"2024-04-02T04:56:19.924748+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:12.624937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:53:12.624937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":44,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448368+0000","last_change":"2024-04-02T04:56:02.037633+0000","last_active":"2024-04-02T04:56:27.448368+0000","last_peered":"2024-04-02T04:56:27.448368+0000","last_clean":"2024-04-02T04:56:27.448368+0000","last_became_active":"2024-04-02T04:56:02.036721+0000","last_became_peered":"2024-04-02T04:56:02.036721+0000","last_unstale":"2024-04-02T04:56:27.448368+0000","last_undegraded":"2024-04-02T04:56:27.448368+0000","last_fullsized":"2024-04-02T04:56:27.448368+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:34.146107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.146107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:36.777848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":421,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914840+0000","last_change":"2024-04-02T04:54:25.267842+0000","last_active":"2024-04-02T04:56:19.914840+0000","last_peered":"2024-04-02T04:56:19.914840+0000","last_clean":"2024-04-02T04:56:19.914840+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T04:56:19.914840+0000","last_undegraded":"2024-04-02T04:56:19.914840+0000","last_fullsized":"2024-04-02T04:56:19.914840+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:25.267631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:54:25.267631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":300,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922435+0000","last_change":"2024-04-02T04:55:22.264368+0000","last_active":"2024-04-02T04:56:19.922435+0000","last_peered":"2024-04-02T04:56:19.922435+0000","last_clean":"2024-04-02T04:56:19.922435+0000","last_became_active":"2024-04-02T04:53:54.927422+0000","last_became_peered":"2024-04-02T04:53:54.927422+0000","last_unstale":"2024-04-02T04:56:19.922435+0000","last_undegraded":"2024-04-02T04:56:19.922435+0000","last_fullsized":"2024-04-02T04:56:19.922435+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:22.264141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:22.264141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:23.446618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"402'27","reported_seq":950,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921630+0000","last_change":"2024-04-02T04:56:02.102341+0000","last_active":"2024-04-02T04:56:19.921630+0000","last_peered":"2024-04-02T04:56:19.921630+0000","last_clean":"2024-04-02T04:56:19.921630+0000","last_became_active":"2024-04-02T04:56:02.031805+0000","last_became_peered":"2024-04-02T04:56:02.031805+0000","last_unstale":"2024-04-02T04:56:19.921630+0000","last_undegraded":"2024-04-02T04:56:19.921630+0000","last_fullsized":"2024-04-02T04:56:19.921630+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T04:54:25.604414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:25.604414+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":965,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928264+0000","last_change":"2024-04-02T04:54:55.979542+0000","last_active":"2024-04-02T04:56:19.928264+0000","last_peered":"2024-04-02T04:56:19.928264+0000","last_clean":"2024-04-02T04:56:19.928264+0000","last_became_active":"2024-04-02T04:54:55.978936+0000","last_became_peered":"2024-04-02T04:54:55.978936+0000","last_unstale":"2024-04-02T04:56:19.928264+0000","last_undegraded":"2024-04-02T04:56:19.928264+0000","last_fullsized":"2024-04-02T04:56:19.928264+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:34.979136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.979136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":707,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.914369+0000","last_change":"2024-04-02T04:56:02.000413+0000","last_active":"2024-04-02T04:56:19.914369+0000","last_peered":"2024-04-02T04:56:19.914369+0000","last_clean":"2024-04-02T04:56:19.914369+0000","last_became_active":"2024-04-02T04:56:01.999978+0000","last_became_peered":"2024-04-02T04:56:01.999978+0000","last_unstale":"2024-04-02T04:56:19.914369+0000","last_undegraded":"2024-04-02T04:56:19.914369+0000","last_fullsized":"2024-04-02T04:56:19.914369+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:04.069399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:04.069399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":42,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266188+0000","last_change":"2024-04-02T04:56:02.028194+0000","last_active":"2024-04-02T04:56:19.266188+0000","last_peered":"2024-04-02T04:56:19.266188+0000","last_clean":"2024-04-02T04:56:19.266188+0000","last_became_active":"2024-04-02T04:56:02.027695+0000","last_became_peered":"2024-04-02T04:56:02.027695+0000","last_unstale":"2024-04-02T04:56:19.266188+0000","last_undegraded":"2024-04-02T04:56:19.266188+0000","last_fullsized":"2024-04-02T04:56:19.266188+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.741751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.741751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":815,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445190+0000","last_change":"2024-04-02T04:54:07.624201+0000","last_active":"2024-04-02T04:56:27.445190+0000","last_peered":"2024-04-02T04:56:27.445190+0000","last_clean":"2024-04-02T04:56:27.445190+0000","last_became_active":"2024-04-02T04:53:54.917005+0000","last_became_peered":"2024-04-02T04:53:54.917005+0000","last_unstale":"2024-04-02T04:56:27.445190+0000","last_undegraded":"2024-04-02T04:56:27.445190+0000","last_fullsized":"2024-04-02T04:56:27.445190+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:07.623881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:07.623881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"402'62","reported_seq":1084,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449042+0000","last_change":"2024-04-02T04:56:02.081823+0000","last_active":"2024-04-02T04:56:27.449042+0000","last_peered":"2024-04-02T04:56:27.449042+0000","last_clean":"2024-04-02T04:56:27.449042+0000","last_became_active":"2024-04-02T04:56:02.033693+0000","last_became_peered":"2024-04-02T04:56:02.033693+0000","last_unstale":"2024-04-02T04:56:27.449042+0000","last_undegraded":"2024-04-02T04:56:27.449042+0000","last_fullsized":"2024-04-02T04:56:27.449042+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'62","last_scrub_stamp":"2024-04-02T04:55:14.368686+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:55:14.368686+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:27.779859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":853,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.445162+0000","last_change":"2024-04-02T04:54:56.011075+0000","last_active":"2024-04-02T04:56:27.445162+0000","last_peered":"2024-04-02T04:56:27.445162+0000","last_clean":"2024-04-02T04:56:27.445162+0000","last_became_active":"2024-04-02T04:54:56.008703+0000","last_became_peered":"2024-04-02T04:54:56.008703+0000","last_unstale":"2024-04-02T04:56:27.445162+0000","last_undegraded":"2024-04-02T04:56:27.445162+0000","last_fullsized":"2024-04-02T04:56:27.445162+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:03.730067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:03.730067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":962,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267826+0000","last_change":"2024-04-02T04:56:02.040168+0000","last_active":"2024-04-02T04:56:19.267826+0000","last_peered":"2024-04-02T04:56:19.267826+0000","last_clean":"2024-04-02T04:56:19.267826+0000","last_became_active":"2024-04-02T04:56:02.039832+0000","last_became_peered":"2024-04-02T04:56:02.039832+0000","last_unstale":"2024-04-02T04:56:19.267826+0000","last_undegraded":"2024-04-02T04:56:19.267826+0000","last_fullsized":"2024-04-02T04:56:19.267826+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:13.114440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:13.114440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:30.761938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":434,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915787+0000","last_change":"2024-04-02T04:56:02.089568+0000","last_active":"2024-04-02T04:56:19.915787+0000","last_peered":"2024-04-02T04:56:19.915787+0000","last_clean":"2024-04-02T04:56:19.915787+0000","last_became_active":"2024-04-02T04:56:02.031136+0000","last_became_peered":"2024-04-02T04:56:02.031136+0000","last_unstale":"2024-04-02T04:56:19.915787+0000","last_undegraded":"2024-04-02T04:56:19.915787+0000","last_fullsized":"2024-04-02T04:56:19.915787+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:54:00.147830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.147830+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":700,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265875+0000","last_change":"2024-04-02T04:54:55.990492+0000","last_active":"2024-04-02T04:56:19.265875+0000","last_peered":"2024-04-02T04:56:19.265875+0000","last_clean":"2024-04-02T04:56:19.265875+0000","last_became_active":"2024-04-02T04:54:55.989605+0000","last_became_peered":"2024-04-02T04:54:55.989605+0000","last_unstale":"2024-04-02T04:56:19.265875+0000","last_undegraded":"2024-04-02T04:56:19.265875+0000","last_fullsized":"2024-04-02T04:56:19.265875+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.540354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.540354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"402'21","reported_seq":788,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.265134+0000","last_change":"2024-04-02T04:56:02.135135+0000","last_active":"2024-04-02T04:56:19.265134+0000","last_peered":"2024-04-02T04:56:19.265134+0000","last_clean":"2024-04-02T04:56:19.265134+0000","last_became_active":"2024-04-02T04:56:02.038390+0000","last_became_peered":"2024-04-02T04:56:02.038390+0000","last_unstale":"2024-04-02T04:56:19.265134+0000","last_undegraded":"2024-04-02T04:56:19.265134+0000","last_fullsized":"2024-04-02T04:56:19.265134+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:58.806324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:58.806324+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":115,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.918670+0000","last_change":"2024-04-02T04:56:02.070513+0000","last_active":"2024-04-02T04:56:19.918670+0000","last_peered":"2024-04-02T04:56:19.918670+0000","last_clean":"2024-04-02T04:56:19.918670+0000","last_became_active":"2024-04-02T04:56:02.016282+0000","last_became_peered":"2024-04-02T04:56:02.016282+0000","last_unstale":"2024-04-02T04:56:19.918670+0000","last_undegraded":"2024-04-02T04:56:19.918670+0000","last_fullsized":"2024-04-02T04:56:19.918670+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:54:32.960556+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:54:32.960556+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"451'12","reported_seq":1117,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928601+0000","last_change":"2024-04-02T04:54:56.025228+0000","last_active":"2024-04-02T04:56:19.928601+0000","last_peered":"2024-04-02T04:56:19.928601+0000","last_clean":"2024-04-02T04:56:19.928601+0000","last_became_active":"2024-04-02T04:54:56.006792+0000","last_became_peered":"2024-04-02T04:54:56.006792+0000","last_unstale":"2024-04-02T04:56:19.928601+0000","last_undegraded":"2024-04-02T04:56:19.928601+0000","last_fullsized":"2024-04-02T04:56:19.928601+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:54:18.879221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:18.879221+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":949,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451981+0000","last_change":"2024-04-02T04:55:11.232633+0000","last_active":"2024-04-02T04:56:27.451981+0000","last_peered":"2024-04-02T04:56:27.451981+0000","last_clean":"2024-04-02T04:56:27.451981+0000","last_became_active":"2024-04-02T04:54:55.979124+0000","last_became_peered":"2024-04-02T04:54:55.979124+0000","last_unstale":"2024-04-02T04:56:27.451981+0000","last_undegraded":"2024-04-02T04:56:27.451981+0000","last_fullsized":"2024-04-02T04:56:27.451981+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:11.232532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:11.232532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:27.541964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":813,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921153+0000","last_change":"2024-04-02T04:55:24.890462+0000","last_active":"2024-04-02T04:56:19.921153+0000","last_peered":"2024-04-02T04:56:19.921153+0000","last_clean":"2024-04-02T04:56:19.921153+0000","last_became_active":"2024-04-02T04:54:55.972361+0000","last_became_peered":"2024-04-02T04:54:55.972361+0000","last_unstale":"2024-04-02T04:56:19.921153+0000","last_undegraded":"2024-04-02T04:56:19.921153+0000","last_fullsized":"2024-04-02T04:56:19.921153+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:24.890319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.890319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:29.338404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"402'16","reported_seq":979,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.266424+0000","last_change":"2024-04-02T04:54:27.408260+0000","last_active":"2024-04-02T04:56:19.266424+0000","last_peered":"2024-04-02T04:56:19.266424+0000","last_clean":"2024-04-02T04:56:19.266424+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T04:56:19.266424+0000","last_undegraded":"2024-04-02T04:56:19.266424+0000","last_fullsized":"2024-04-02T04:56:19.266424+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'16","last_scrub_stamp":"2024-04-02T04:54:27.408029+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:54:27.408029+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":960,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.921513+0000","last_change":"2024-04-02T04:54:10.943919+0000","last_active":"2024-04-02T04:56:19.921513+0000","last_peered":"2024-04-02T04:56:19.921513+0000","last_clean":"2024-04-02T04:56:19.921513+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T04:56:19.921513+0000","last_undegraded":"2024-04-02T04:56:19.921513+0000","last_fullsized":"2024-04-02T04:56:19.921513+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:10.943768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:54:10.943768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":967,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.546684+0000","last_change":"2024-04-02T04:56:02.043027+0000","last_active":"2024-04-02T04:56:27.546684+0000","last_peered":"2024-04-02T04:56:27.546684+0000","last_clean":"2024-04-02T04:56:27.546684+0000","last_became_active":"2024-04-02T04:56:02.042100+0000","last_became_peered":"2024-04-02T04:56:02.042100+0000","last_unstale":"2024-04-02T04:56:27.546684+0000","last_undegraded":"2024-04-02T04:56:27.546684+0000","last_fullsized":"2024-04-02T04:56:27.546684+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.792837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.792837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:36.541279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":958,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452325+0000","last_change":"2024-04-02T04:56:02.000801+0000","last_active":"2024-04-02T04:56:27.452325+0000","last_peered":"2024-04-02T04:56:27.452325+0000","last_clean":"2024-04-02T04:56:27.452325+0000","last_became_active":"2024-04-02T04:56:01.999893+0000","last_became_peered":"2024-04-02T04:56:01.999893+0000","last_unstale":"2024-04-02T04:56:27.452325+0000","last_undegraded":"2024-04-02T04:56:27.452325+0000","last_fullsized":"2024-04-02T04:56:27.452325+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:24.804934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.804934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:50.138840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":814,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.450893+0000","last_change":"2024-04-02T04:54:21.209523+0000","last_active":"2024-04-02T04:56:27.450893+0000","last_peered":"2024-04-02T04:56:27.450893+0000","last_clean":"2024-04-02T04:56:27.450893+0000","last_became_active":"2024-04-02T04:53:54.942584+0000","last_became_peered":"2024-04-02T04:53:54.942584+0000","last_unstale":"2024-04-02T04:56:27.450893+0000","last_undegraded":"2024-04-02T04:56:27.450893+0000","last_fullsized":"2024-04-02T04:56:27.450893+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:21.209411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:21.209411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"402'35","reported_seq":1010,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449095+0000","last_change":"2024-04-02T04:55:32.581265+0000","last_active":"2024-04-02T04:56:27.449095+0000","last_peered":"2024-04-02T04:56:27.449095+0000","last_clean":"2024-04-02T04:56:27.449095+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:56:27.449095+0000","last_undegraded":"2024-04-02T04:56:27.449095+0000","last_fullsized":"2024-04-02T04:56:27.449095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T04:55:32.581158+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:55:32.581158+0000","objects_scrubbed":6,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:46.768522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":722,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.920453+0000","last_change":"2024-04-02T04:56:02.008497+0000","last_active":"2024-04-02T04:56:19.920453+0000","last_peered":"2024-04-02T04:56:19.920453+0000","last_clean":"2024-04-02T04:56:19.920453+0000","last_became_active":"2024-04-02T04:56:02.008124+0000","last_became_peered":"2024-04-02T04:56:02.008124+0000","last_unstale":"2024-04-02T04:56:19.920453+0000","last_undegraded":"2024-04-02T04:56:19.920453+0000","last_fullsized":"2024-04-02T04:56:19.920453+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:09.153487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:09.153487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":44,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.451851+0000","last_change":"2024-04-02T04:56:02.036944+0000","last_active":"2024-04-02T04:56:27.451851+0000","last_peered":"2024-04-02T04:56:27.451851+0000","last_clean":"2024-04-02T04:56:27.451851+0000","last_became_active":"2024-04-02T04:56:02.036349+0000","last_became_peered":"2024-04-02T04:56:02.036349+0000","last_unstale":"2024-04-02T04:56:27.451851+0000","last_undegraded":"2024-04-02T04:56:27.451851+0000","last_fullsized":"2024-04-02T04:56:27.451851+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:14.837465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:53:14.837465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":815,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.928714+0000","last_change":"2024-04-02T04:56:02.021723+0000","last_active":"2024-04-02T04:56:19.928714+0000","last_peered":"2024-04-02T04:56:19.928714+0000","last_clean":"2024-04-02T04:56:19.928714+0000","last_became_active":"2024-04-02T04:56:02.020753+0000","last_became_peered":"2024-04-02T04:56:02.020753+0000","last_unstale":"2024-04-02T04:56:19.928714+0000","last_undegraded":"2024-04-02T04:56:19.928714+0000","last_fullsized":"2024-04-02T04:56:19.928714+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:15.974122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:15.974122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:25.611204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":813,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449625+0000","last_change":"2024-04-02T04:55:17.392334+0000","last_active":"2024-04-02T04:56:27.449625+0000","last_peered":"2024-04-02T04:56:27.449625+0000","last_clean":"2024-04-02T04:56:27.449625+0000","last_became_active":"2024-04-02T04:49:26.290644+0000","last_became_peered":"2024-04-02T04:49:26.290644+0000","last_unstale":"2024-04-02T04:56:27.449625+0000","last_undegraded":"2024-04-02T04:56:27.449625+0000","last_fullsized":"2024-04-02T04:56:27.449625+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.392235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.392235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:33.822751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"402'24","reported_seq":926,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448582+0000","last_change":"2024-04-02T04:54:56.051234+0000","last_active":"2024-04-02T04:56:27.448582+0000","last_peered":"2024-04-02T04:56:27.448582+0000","last_clean":"2024-04-02T04:56:27.448582+0000","last_became_active":"2024-04-02T04:54:56.014703+0000","last_became_peered":"2024-04-02T04:54:56.014703+0000","last_unstale":"2024-04-02T04:56:27.448582+0000","last_undegraded":"2024-04-02T04:56:27.448582+0000","last_fullsized":"2024-04-02T04:56:27.448582+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'18","last_scrub_stamp":"2024-04-02T04:52:48.727589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:48.727589+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":444,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.915218+0000","last_change":"2024-04-02T04:54:55.991236+0000","last_active":"2024-04-02T04:56:19.915218+0000","last_peered":"2024-04-02T04:56:19.915218+0000","last_clean":"2024-04-02T04:56:19.915218+0000","last_became_active":"2024-04-02T04:54:55.971317+0000","last_became_peered":"2024-04-02T04:54:55.971317+0000","last_unstale":"2024-04-02T04:56:19.915218+0000","last_undegraded":"2024-04-02T04:56:19.915218+0000","last_fullsized":"2024-04-02T04:56:19.915218+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:54:14.246467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:14.246467+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":946,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449250+0000","last_change":"2024-04-02T04:56:02.010795+0000","last_active":"2024-04-02T04:56:27.449250+0000","last_peered":"2024-04-02T04:56:27.449250+0000","last_clean":"2024-04-02T04:56:27.449250+0000","last_became_active":"2024-04-02T04:56:02.010320+0000","last_became_peered":"2024-04-02T04:56:02.010320+0000","last_unstale":"2024-04-02T04:56:27.449250+0000","last_undegraded":"2024-04-02T04:56:27.449250+0000","last_fullsized":"2024-04-02T04:56:27.449250+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:48.311616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:51:48.311616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":943,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.264883+0000","last_change":"2024-04-02T04:54:24.386420+0000","last_active":"2024-04-02T04:56:19.264883+0000","last_peered":"2024-04-02T04:56:19.264883+0000","last_clean":"2024-04-02T04:56:19.264883+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T04:56:19.264883+0000","last_undegraded":"2024-04-02T04:56:19.264883+0000","last_fullsized":"2024-04-02T04:56:19.264883+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:24.386320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.386320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":685,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.452067+0000","last_change":"2024-04-02T04:56:02.053486+0000","last_active":"2024-04-02T04:56:27.452067+0000","last_peered":"2024-04-02T04:56:27.452067+0000","last_clean":"2024-04-02T04:56:27.452067+0000","last_became_active":"2024-04-02T04:56:02.053173+0000","last_became_peered":"2024-04-02T04:56:02.053173+0000","last_unstale":"2024-04-02T04:56:27.452067+0000","last_undegraded":"2024-04-02T04:56:27.452067+0000","last_fullsized":"2024-04-02T04:56:27.452067+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:16.955150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:16.955150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:28.643400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"402'27","reported_seq":928,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448402+0000","last_change":"2024-04-02T04:54:56.036522+0000","last_active":"2024-04-02T04:56:27.448402+0000","last_peered":"2024-04-02T04:56:27.448402+0000","last_clean":"2024-04-02T04:56:27.448402+0000","last_became_active":"2024-04-02T04:54:56.011842+0000","last_became_peered":"2024-04-02T04:54:56.011842+0000","last_unstale":"2024-04-02T04:56:27.448402+0000","last_undegraded":"2024-04-02T04:56:27.448402+0000","last_fullsized":"2024-04-02T04:56:27.448402+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-02T04:52:53.802714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:53.802714+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":447,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.922209+0000","last_change":"2024-04-02T04:56:18.940836+0000","last_active":"2024-04-02T04:56:19.922209+0000","last_peered":"2024-04-02T04:56:19.922209+0000","last_clean":"2024-04-02T04:56:19.922209+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T04:56:19.922209+0000","last_undegraded":"2024-04-02T04:56:19.922209+0000","last_fullsized":"2024-04-02T04:56:19.922209+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:18.940739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.940739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:29.271077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":954,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449437+0000","last_change":"2024-04-02T04:56:02.029393+0000","last_active":"2024-04-02T04:56:27.449437+0000","last_peered":"2024-04-02T04:56:27.449437+0000","last_clean":"2024-04-02T04:56:27.449437+0000","last_became_active":"2024-04-02T04:56:02.028857+0000","last_became_peered":"2024-04-02T04:56:02.028857+0000","last_unstale":"2024-04-02T04:56:27.449437+0000","last_undegraded":"2024-04-02T04:56:27.449437+0000","last_fullsized":"2024-04-02T04:56:27.449437+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:22.882047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:22.882047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:56:28.945252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":180,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.449180+0000","last_change":"2024-04-02T04:56:02.052702+0000","last_active":"2024-04-02T04:56:27.449180+0000","last_peered":"2024-04-02T04:56:27.449180+0000","last_clean":"2024-04-02T04:56:27.449180+0000","last_became_active":"2024-04-02T04:56:02.052349+0000","last_became_peered":"2024-04-02T04:56:02.052349+0000","last_unstale":"2024-04-02T04:56:27.449180+0000","last_undegraded":"2024-04-02T04:56:27.449180+0000","last_fullsized":"2024-04-02T04:56:27.449180+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:47.730634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:52:47.730634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":804,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.924752+0000","last_change":"2024-04-02T04:54:56.014628+0000","last_active":"2024-04-02T04:56:19.924752+0000","last_peered":"2024-04-02T04:56:19.924752+0000","last_clean":"2024-04-02T04:56:19.924752+0000","last_became_active":"2024-04-02T04:54:56.014277+0000","last_became_peered":"2024-04-02T04:54:56.014277+0000","last_unstale":"2024-04-02T04:56:19.924752+0000","last_undegraded":"2024-04-02T04:56:19.924752+0000","last_fullsized":"2024-04-02T04:56:19.924752+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:49.488717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:52:49.488717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"402'25","reported_seq":1014,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267787+0000","last_change":"2024-04-02T04:54:56.049199+0000","last_active":"2024-04-02T04:56:19.267787+0000","last_peered":"2024-04-02T04:56:19.267787+0000","last_clean":"2024-04-02T04:56:19.267787+0000","last_became_active":"2024-04-02T04:54:55.991156+0000","last_became_peered":"2024-04-02T04:54:55.991156+0000","last_unstale":"2024-04-02T04:56:19.267787+0000","last_undegraded":"2024-04-02T04:56:19.267787+0000","last_fullsized":"2024-04-02T04:56:19.267787+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-02T04:52:52.302878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:52:52.302878+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":835,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267950+0000","last_change":"2024-04-02T04:56:02.022839+0000","last_active":"2024-04-02T04:56:19.267950+0000","last_peered":"2024-04-02T04:56:19.267950+0000","last_clean":"2024-04-02T04:56:19.267950+0000","last_became_active":"2024-04-02T04:56:02.022496+0000","last_became_peered":"2024-04-02T04:56:02.022496+0000","last_unstale":"2024-04-02T04:56:19.267950+0000","last_undegraded":"2024-04-02T04:56:19.267950+0000","last_fullsized":"2024-04-02T04:56:19.267950+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:01.793655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:53:01.793655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":941,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.444813+0000","last_change":"2024-04-02T04:51:49.302781+0000","last_active":"2024-04-02T04:56:27.444813+0000","last_peered":"2024-04-02T04:56:27.444813+0000","last_clean":"2024-04-02T04:56:27.444813+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:56:27.444813+0000","last_undegraded":"2024-04-02T04:56:27.444813+0000","last_fullsized":"2024-04-02T04:56:27.444813+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:51:49.302549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:51:49.302549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":942,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.448194+0000","last_change":"2024-04-02T04:54:55.988287+0000","last_active":"2024-04-02T04:56:27.448194+0000","last_peered":"2024-04-02T04:56:27.448194+0000","last_clean":"2024-04-02T04:56:27.448194+0000","last_became_active":"2024-04-02T04:54:55.984764+0000","last_became_peered":"2024-04-02T04:54:55.984764+0000","last_unstale":"2024-04-02T04:56:27.448194+0000","last_undegraded":"2024-04-02T04:56:27.448194+0000","last_fullsized":"2024-04-02T04:56:27.448194+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:41.343663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.677740+0000","last_clean_scrub_stamp":"2024-04-02T04:52:41.343663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":819,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.453098+0000","last_change":"2024-04-02T04:56:20.082893+0000","last_active":"2024-04-02T04:56:27.453098+0000","last_peered":"2024-04-02T04:56:27.453098+0000","last_clean":"2024-04-02T04:56:27.453098+0000","last_became_active":"2024-04-02T04:56:02.028603+0000","last_became_peered":"2024-04-02T04:56:02.028603+0000","last_unstale":"2024-04-02T04:56:27.453098+0000","last_undegraded":"2024-04-02T04:56:27.453098+0000","last_fullsized":"2024-04-02T04:56:27.453098+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:20.082793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:20.082793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:35.028718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"402'32","reported_seq":802,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.925022+0000","last_change":"2024-04-02T04:56:02.128663+0000","last_active":"2024-04-02T04:56:19.925022+0000","last_peered":"2024-04-02T04:56:19.925022+0000","last_clean":"2024-04-02T04:56:19.925022+0000","last_became_active":"2024-04-02T04:56:02.032875+0000","last_became_peered":"2024-04-02T04:56:02.032875+0000","last_unstale":"2024-04-02T04:56:19.925022+0000","last_undegraded":"2024-04-02T04:56:19.925022+0000","last_fullsized":"2024-04-02T04:56:19.925022+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'22","last_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_deep_scrub":"342'22","last_deep_scrub_stamp":"2024-04-02T04:53:02.777776+0000","last_clean_scrub_stamp":"2024-04-02T04:53:02.777776+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":955,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.267929+0000","last_change":"2024-04-02T04:54:23.368378+0000","last_active":"2024-04-02T04:56:19.267929+0000","last_peered":"2024-04-02T04:56:19.267929+0000","last_clean":"2024-04-02T04:56:19.267929+0000","last_became_active":"2024-04-02T04:53:54.924695+0000","last_became_peered":"2024-04-02T04:53:54.924695+0000","last_unstale":"2024-04-02T04:56:19.267929+0000","last_undegraded":"2024-04-02T04:56:19.267929+0000","last_fullsized":"2024-04-02T04:56:19.267929+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:23.368272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:23.368272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":44,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-02T04:56:27.545370+0000","last_change":"2024-04-02T04:56:02.058517+0000","last_active":"2024-04-02T04:56:27.545370+0000","last_peered":"2024-04-02T04:56:27.545370+0000","last_clean":"2024-04-02T04:56:27.545370+0000","last_became_active":"2024-04-02T04:56:02.057825+0000","last_became_peered":"2024-04-02T04:56:02.057825+0000","last_unstale":"2024-04-02T04:56:27.545370+0000","last_undegraded":"2024-04-02T04:56:27.545370+0000","last_fullsized":"2024-04-02T04:56:27.545370+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:52:56.823620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:52:56.823620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":702,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-02T04:56:19.916005+0000","last_change":"2024-04-02T04:56:02.007670+0000","last_active":"2024-04-02T04:56:19.916005+0000","last_peered":"2024-04-02T04:56:19.916005+0000","last_clean":"2024-04-02T04:56:19.916005+0000","last_became_active":"2024-04-02T04:56:02.007310+0000","last_became_peered":"2024-04-02T04:56:02.007310+0000","last_unstale":"2024-04-02T04:56:19.916005+0000","last_undegraded":"2024-04-02T04:56:19.916005+0000","last_fullsized":"2024-04-02T04:56:19.916005+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:15.108990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:54:15.108990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":298,"num_pg":32,"stat_sum":{"num_bytes":6,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":18,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":6},{"poolid":102,"num_pg":32,"stat_sum":{"num_bytes":8195842,"num_objects":166,"num_object_clones":5,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":110,"num_read_kb":10,"num_write":617,"num_write_kb":12706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":8002844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34672640,"data_stored":32587843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1920,"ondisk_log_size":1920,"up":96,"acting":96,"num_store_stats":8},{"poolid":43,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":13,"num_pg":11,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":33,"acting":33,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1401,"num_read_kb":1126,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":2382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114656,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54100,"kb_used_data":13452,"kb_used_omap":57,"kb_used_meta":40582,"kb_avail":93691052,"statfs":{"total":95995035648,"available":95939637248,"internally_reserved":0,"allocated":13774848,"data_stored":10301397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":41556878},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103340,"num_pgs":169,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89308,"kb_used_data":17876,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93655844,"statfs":{"total":95995035648,"available":95903584256,"internally_reserved":0,"allocated":18305024,"data_stored":14757515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575071,"num_pgs":159,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88744,"kb_used_data":17300,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93656408,"statfs":{"total":95995035648,"available":95904161792,"internally_reserved":0,"allocated":17715200,"data_stored":14177788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738601,"num_pgs":153,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76412,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93668740,"statfs":{"total":95995035648,"available":95916789760,"internally_reserved":0,"allocated":9359360,"data_stored":5972569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378837,"num_pgs":150,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80928,"kb_used_data":9496,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664224,"statfs":{"total":95995035648,"available":95912165376,"internally_reserved":0,"allocated":9723904,"data_stored":6235828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":124554051787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822849,"num_pgs":162,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67356,"kb_used_data":13972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93677796,"statfs":{"total":95995035648,"available":95926063104,"internally_reserved":0,"allocated":14307328,"data_stored":10759310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":43,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8589312,"data_stored":8195257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4038656,"data_stored":4000107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4513792,"data_stored":4195036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4067328,"data_stored":4000177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8593408,"data_stored":8195255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":102,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4308992,"data_stored":4000723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":298,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":298,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":298,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":298,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":298,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":298,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:56:30.485 INFO:tasks.thrashosds.thrasher:Setting 40.1 to [4, 1, 6] 2024-04-02T04:56:30.485 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '40.1', '4', '1', '6'] 2024-04-02T04:56:30.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 40.1 4 1 6 2024-04-02T04:56:30.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:30.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:29.010506+0000 osd.0 (osd.0) 283 : cluster 0 40.8 deep-scrub starts 2024-04-02T04:56:30.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:29.012151+0000 osd.0 (osd.0) 284 : cluster 0 40.8 deep-scrub ok 2024-04-02T04:56:30.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: audit 2024-04-02T04:56:29.509772+0000 mgr.y (mgr.14538) 501 : audit 0 from='client.26657 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:29.941203+0000 mon.a (mon.0) 2949 : cluster 0 osdmap e531: 8 total, 8 up, 6 in 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.015723+0000 osd.6 (osd.6) 243 : cluster 0 3.a deep-scrub starts 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.019677+0000 osd.2 (osd.2) 236 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.019976+0000 osd.2 (osd.2) 237 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.024643+0000 osd.6 (osd.6) 244 : cluster 0 3.a deep-scrub ok 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.034190+0000 osd.7 (osd.7) 83 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.036670+0000 osd.7 (osd.7) 84 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[21528]: audit 2024-04-02T04:56:30.078057+0000 mon.c (mon.2) 926 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:29.010506+0000 osd.0 (osd.0) 283 : cluster 0 40.8 deep-scrub starts 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:29.012151+0000 osd.0 (osd.0) 284 : cluster 0 40.8 deep-scrub ok 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: audit 2024-04-02T04:56:29.509772+0000 mgr.y (mgr.14538) 501 : audit 0 from='client.26657 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:56:30.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:29.941203+0000 mon.a (mon.0) 2949 : cluster 0 osdmap e531: 8 total, 8 up, 6 in 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.015723+0000 osd.6 (osd.6) 243 : cluster 0 3.a deep-scrub starts 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.019677+0000 osd.2 (osd.2) 236 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.019976+0000 osd.2 (osd.2) 237 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.024643+0000 osd.6 (osd.6) 244 : cluster 0 3.a deep-scrub ok 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.034190+0000 osd.7 (osd.7) 83 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.036670+0000 osd.7 (osd.7) 84 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:56:30.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:30 smithi069 bash[28904]: audit 2024-04-02T04:56:30.078057+0000 mon.c (mon.2) 926 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:30.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:30.636 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:29.010506+0000 osd.0 (osd.0) 283 : cluster 0 40.8 deep-scrub starts 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:29.012151+0000 osd.0 (osd.0) 284 : cluster 0 40.8 deep-scrub ok 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: audit 2024-04-02T04:56:29.509772+0000 mgr.y (mgr.14538) 501 : audit 0 from='client.26657 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:29.941203+0000 mon.a (mon.0) 2949 : cluster 0 osdmap e531: 8 total, 8 up, 6 in 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.015723+0000 osd.6 (osd.6) 243 : cluster 0 3.a deep-scrub starts 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.019677+0000 osd.2 (osd.2) 236 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.019976+0000 osd.2 (osd.2) 237 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.024643+0000 osd.6 (osd.6) 244 : cluster 0 3.a deep-scrub ok 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.034190+0000 osd.7 (osd.7) 83 : cluster 0 4.0 deep-scrub starts 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.036670+0000 osd.7 (osd.7) 84 : cluster 0 4.0 deep-scrub ok 2024-04-02T04:56:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:30 smithi098 bash[23790]: audit 2024-04-02T04:56:30.078057+0000 mon.c (mon.2) 926 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:30.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:30.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:30.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:30.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:56:30.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:31.022 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:31.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:31.145 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:31.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:31.301 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:56:31.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:31.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:31.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:31.528 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:31.631 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:31.657 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.122047+0000 osd.2 (osd.2) 238 : cluster 0 10.18 deep-scrub starts 2024-04-02T04:56:31.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.126634+0000 osd.2 (osd.2) 239 : cluster 0 10.18 deep-scrub ok 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.165545+0000 mgr.y (mgr.14538) 502 : cluster 0 pgmap v783: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.497196+0000 osd.1 (osd.1) 293 : cluster 0 10.2 scrub starts 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.499503+0000 osd.1 (osd.1) 294 : cluster 0 10.2 scrub ok 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.705372+0000 osd.5 (osd.5) 253 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.705792+0000 osd.5 (osd.5) 254 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: cluster 2024-04-02T04:56:30.961664+0000 mon.a (mon.0) 2950 : cluster 0 osdmap e532: 8 total, 8 up, 6 in 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: audit 2024-04-02T04:56:31.079794+0000 mon.c (mon.2) 927 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: audit 2024-04-02T04:56:31.137206+0000 mon.c (mon.2) 928 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: audit 2024-04-02T04:56:31.137655+0000 mon.a (mon.0) 2951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: audit 2024-04-02T04:56:31.138222+0000 mon.c (mon.2) 929 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-71"}]: dispatch 2024-04-02T04:56:31.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:31 smithi098 bash[23790]: audit 2024-04-02T04:56:31.138488+0000 mon.a (mon.0) 2952 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-71"}]: dispatch 2024-04-02T04:56:31.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:31.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.122047+0000 osd.2 (osd.2) 238 : cluster 0 10.18 deep-scrub starts 2024-04-02T04:56:31.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.126634+0000 osd.2 (osd.2) 239 : cluster 0 10.18 deep-scrub ok 2024-04-02T04:56:31.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.165545+0000 mgr.y (mgr.14538) 502 : cluster 0 pgmap v783: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:31.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.497196+0000 osd.1 (osd.1) 293 : cluster 0 10.2 scrub starts 2024-04-02T04:56:31.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.499503+0000 osd.1 (osd.1) 294 : cluster 0 10.2 scrub ok 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.705372+0000 osd.5 (osd.5) 253 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.705792+0000 osd.5 (osd.5) 254 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: cluster 2024-04-02T04:56:30.961664+0000 mon.a (mon.0) 2950 : cluster 0 osdmap e532: 8 total, 8 up, 6 in 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: audit 2024-04-02T04:56:31.079794+0000 mon.c (mon.2) 927 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: audit 2024-04-02T04:56:31.137206+0000 mon.c (mon.2) 928 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: audit 2024-04-02T04:56:31.137655+0000 mon.a (mon.0) 2951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: audit 2024-04-02T04:56:31.138222+0000 mon.c (mon.2) 929 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-71"}]: dispatch 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[21528]: audit 2024-04-02T04:56:31.138488+0000 mon.a (mon.0) 2952 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-71"}]: dispatch 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.122047+0000 osd.2 (osd.2) 238 : cluster 0 10.18 deep-scrub starts 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.126634+0000 osd.2 (osd.2) 239 : cluster 0 10.18 deep-scrub ok 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.165545+0000 mgr.y (mgr.14538) 502 : cluster 0 pgmap v783: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:31.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.497196+0000 osd.1 (osd.1) 293 : cluster 0 10.2 scrub starts 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.499503+0000 osd.1 (osd.1) 294 : cluster 0 10.2 scrub ok 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.705372+0000 osd.5 (osd.5) 253 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.705792+0000 osd.5 (osd.5) 254 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: cluster 2024-04-02T04:56:30.961664+0000 mon.a (mon.0) 2950 : cluster 0 osdmap e532: 8 total, 8 up, 6 in 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: audit 2024-04-02T04:56:31.079794+0000 mon.c (mon.2) 927 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: audit 2024-04-02T04:56:31.137206+0000 mon.c (mon.2) 928 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: audit 2024-04-02T04:56:31.137655+0000 mon.a (mon.0) 2951 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: audit 2024-04-02T04:56:31.138222+0000 mon.c (mon.2) 929 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-71"}]: dispatch 2024-04-02T04:56:31.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:31 smithi069 bash[28904]: audit 2024-04-02T04:56:31.138488+0000 mon.a (mon.0) 2952 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-71"}]: dispatch 2024-04-02T04:56:31.784 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:31.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:31.914 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:31.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:32.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:32.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:32.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:32.311 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:32.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:32.436 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:32.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:32.563 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:32.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:32.676 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[28904]: cluster 2024-04-02T04:56:31.490806+0000 osd.1 (osd.1) 295 : cluster 0 40.12 scrub starts 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[28904]: cluster 2024-04-02T04:56:31.493195+0000 osd.1 (osd.1) 296 : cluster 0 40.12 scrub ok 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[28904]: cluster 2024-04-02T04:56:31.962181+0000 mon.a (mon.0) 2953 : cluster 0 osdmap e533: 8 total, 8 up, 6 in 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[28904]: audit 2024-04-02T04:56:32.081372+0000 mon.c (mon.2) 930 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[21528]: cluster 2024-04-02T04:56:31.490806+0000 osd.1 (osd.1) 295 : cluster 0 40.12 scrub starts 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[21528]: cluster 2024-04-02T04:56:31.493195+0000 osd.1 (osd.1) 296 : cluster 0 40.12 scrub ok 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[21528]: cluster 2024-04-02T04:56:31.962181+0000 mon.a (mon.0) 2953 : cluster 0 osdmap e533: 8 total, 8 up, 6 in 2024-04-02T04:56:32.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:32 smithi069 bash[21528]: audit 2024-04-02T04:56:32.081372+0000 mon.c (mon.2) 930 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:32.692 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:32 smithi098 bash[23790]: cluster 2024-04-02T04:56:31.490806+0000 osd.1 (osd.1) 295 : cluster 0 40.12 scrub starts 2024-04-02T04:56:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:32 smithi098 bash[23790]: cluster 2024-04-02T04:56:31.493195+0000 osd.1 (osd.1) 296 : cluster 0 40.12 scrub ok 2024-04-02T04:56:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:32 smithi098 bash[23790]: cluster 2024-04-02T04:56:31.962181+0000 mon.a (mon.0) 2953 : cluster 0 osdmap e533: 8 total, 8 up, 6 in 2024-04-02T04:56:32.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:32 smithi098 bash[23790]: audit 2024-04-02T04:56:32.081372+0000 mon.c (mon.2) 930 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:32.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:32.821 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:32.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:32.947 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:32.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:33.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:33.068 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:33.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:33.195 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:33.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:33.325 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:33.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:33.457 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:33.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:33.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: cluster 2024-04-02T04:56:32.166436+0000 mgr.y (mgr.14538) 503 : cluster 0 pgmap v786: 281 pgs: 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: cluster 2024-04-02T04:56:32.170437+0000 osd.2 (osd.2) 240 : cluster 0 4.f scrub starts 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: cluster 2024-04-02T04:56:32.172908+0000 osd.2 (osd.2) 241 : cluster 0 4.f scrub ok 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: cluster 2024-04-02T04:56:32.976128+0000 mon.a (mon.0) 2954 : cluster 0 osdmap e534: 8 total, 8 up, 6 in 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: audit 2024-04-02T04:56:32.980275+0000 mon.c (mon.2) 931 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: audit 2024-04-02T04:56:32.982106+0000 mon.a (mon.0) 2955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.017085+0000 osd.6 (osd.6) 245 : cluster 0 40.1e scrub starts 2024-04-02T04:56:33.655 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.018092+0000 osd.6 (osd.6) 246 : cluster 0 40.1e scrub ok 2024-04-02T04:56:33.656 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: audit 2024-04-02T04:56:33.082896+0000 mon.c (mon.2) 932 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:33.656 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: audit 2024-04-02T04:56:33.229892+0000 mon.a (mon.0) 2956 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:33.656 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:33 smithi098 bash[23790]: audit 2024-04-02T04:56:33.230185+0000 mon.b (mon.1) 268 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:33.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:33.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: cluster 2024-04-02T04:56:32.166436+0000 mgr.y (mgr.14538) 503 : cluster 0 pgmap v786: 281 pgs: 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: cluster 2024-04-02T04:56:32.170437+0000 osd.2 (osd.2) 240 : cluster 0 4.f scrub starts 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: cluster 2024-04-02T04:56:32.172908+0000 osd.2 (osd.2) 241 : cluster 0 4.f scrub ok 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: cluster 2024-04-02T04:56:32.976128+0000 mon.a (mon.0) 2954 : cluster 0 osdmap e534: 8 total, 8 up, 6 in 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: audit 2024-04-02T04:56:32.980275+0000 mon.c (mon.2) 931 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: audit 2024-04-02T04:56:32.982106+0000 mon.a (mon.0) 2955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.017085+0000 osd.6 (osd.6) 245 : cluster 0 40.1e scrub starts 2024-04-02T04:56:33.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.018092+0000 osd.6 (osd.6) 246 : cluster 0 40.1e scrub ok 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: audit 2024-04-02T04:56:33.082896+0000 mon.c (mon.2) 932 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: audit 2024-04-02T04:56:33.229892+0000 mon.a (mon.0) 2956 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[21528]: audit 2024-04-02T04:56:33.230185+0000 mon.b (mon.1) 268 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: cluster 2024-04-02T04:56:32.166436+0000 mgr.y (mgr.14538) 503 : cluster 0 pgmap v786: 281 pgs: 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 446 MiB used, 536 GiB / 536 GiB avail; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: cluster 2024-04-02T04:56:32.170437+0000 osd.2 (osd.2) 240 : cluster 0 4.f scrub starts 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: cluster 2024-04-02T04:56:32.172908+0000 osd.2 (osd.2) 241 : cluster 0 4.f scrub ok 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: cluster 2024-04-02T04:56:32.976128+0000 mon.a (mon.0) 2954 : cluster 0 osdmap e534: 8 total, 8 up, 6 in 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: audit 2024-04-02T04:56:32.980275+0000 mon.c (mon.2) 931 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: audit 2024-04-02T04:56:32.982106+0000 mon.a (mon.0) 2955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.017085+0000 osd.6 (osd.6) 245 : cluster 0 40.1e scrub starts 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.018092+0000 osd.6 (osd.6) 246 : cluster 0 40.1e scrub ok 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: audit 2024-04-02T04:56:33.082896+0000 mon.c (mon.2) 932 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: audit 2024-04-02T04:56:33.229892+0000 mon.a (mon.0) 2956 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:33.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:33 smithi069 bash[28904]: audit 2024-04-02T04:56:33.230185+0000 mon.b (mon.1) 268 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:33.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:33.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:33.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:33.987 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:34.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:34.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:34.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:34.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:34.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:34.429 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:34.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:34.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:34.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: cluster 2024-04-02T04:56:32.923650+0000 osd.4 (osd.4) 291 : cluster 0 40.e deep-scrub starts 2024-04-02T04:56:34.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: cluster 2024-04-02T04:56:32.925316+0000 osd.4 (osd.4) 292 : cluster 0 40.e deep-scrub ok 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.148071+0000 osd.2 (osd.2) 242 : cluster 0 2.3 scrub starts 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.150132+0000 osd.2 (osd.2) 243 : cluster 0 2.3 scrub ok 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: audit 2024-04-02T04:56:33.367854+0000 mgr.y (mgr.14538) 504 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: audit 2024-04-02T04:56:33.971234+0000 mon.a (mon.0) 2957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.982984+0000 mon.a (mon.0) 2958 : cluster 0 osdmap e535: 8 total, 8 up, 6 in 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: audit 2024-04-02T04:56:33.985451+0000 mon.c (mon.2) 933 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[21528]: audit 2024-04-02T04:56:34.083800+0000 mon.c (mon.2) 934 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: cluster 2024-04-02T04:56:32.923650+0000 osd.4 (osd.4) 291 : cluster 0 40.e deep-scrub starts 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: cluster 2024-04-02T04:56:32.925316+0000 osd.4 (osd.4) 292 : cluster 0 40.e deep-scrub ok 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.148071+0000 osd.2 (osd.2) 242 : cluster 0 2.3 scrub starts 2024-04-02T04:56:34.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.150132+0000 osd.2 (osd.2) 243 : cluster 0 2.3 scrub ok 2024-04-02T04:56:34.670 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: audit 2024-04-02T04:56:33.367854+0000 mgr.y (mgr.14538) 504 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:34.670 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: audit 2024-04-02T04:56:33.971234+0000 mon.a (mon.0) 2957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:34.670 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.982984+0000 mon.a (mon.0) 2958 : cluster 0 osdmap e535: 8 total, 8 up, 6 in 2024-04-02T04:56:34.670 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: audit 2024-04-02T04:56:33.985451+0000 mon.c (mon.2) 933 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:34.670 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:34 smithi069 bash[28904]: audit 2024-04-02T04:56:34.083800+0000 mon.c (mon.2) 934 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:34.685 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: cluster 2024-04-02T04:56:32.923650+0000 osd.4 (osd.4) 291 : cluster 0 40.e deep-scrub starts 2024-04-02T04:56:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: cluster 2024-04-02T04:56:32.925316+0000 osd.4 (osd.4) 292 : cluster 0 40.e deep-scrub ok 2024-04-02T04:56:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.148071+0000 osd.2 (osd.2) 242 : cluster 0 2.3 scrub starts 2024-04-02T04:56:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.150132+0000 osd.2 (osd.2) 243 : cluster 0 2.3 scrub ok 2024-04-02T04:56:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: audit 2024-04-02T04:56:33.367854+0000 mgr.y (mgr.14538) 504 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: audit 2024-04-02T04:56:33.971234+0000 mon.a (mon.0) 2957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.982984+0000 mon.a (mon.0) 2958 : cluster 0 osdmap e535: 8 total, 8 up, 6 in 2024-04-02T04:56:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: audit 2024-04-02T04:56:33.985451+0000 mon.c (mon.2) 933 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:34 smithi098 bash[23790]: audit 2024-04-02T04:56:34.083800+0000 mon.c (mon.2) 934 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:34.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:34.817 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:34.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:34.948 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:34.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:35.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:35.072 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:35.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:35.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:35.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:35.331 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:35.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:35.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:35.532 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T04:56:35.532 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:56:35.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.893141+0000 osd.4 (osd.4) 293 : cluster 0 4.1a scrub starts 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:33.895592+0000 osd.4 (osd.4) 294 : cluster 0 4.1a scrub ok 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:34.167857+0000 mgr.y (mgr.14538) 505 : cluster 0 pgmap v789: 313 pgs: 2 active, 14 creating+peering, 12 unknown, 1 active+remapped+backfill_toofull, 284 active+clean; 8.4 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:34.197436+0000 osd.2 (osd.2) 244 : cluster 0 43.18 scrub starts 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:34.199596+0000 osd.2 (osd.2) 245 : cluster 0 43.18 scrub ok 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:34.475395+0000 osd.1 (osd.1) 297 : cluster 0 6.19 scrub starts 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:34.477616+0000 osd.1 (osd.1) 298 : cluster 0 6.19 scrub ok 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: cluster 2024-04-02T04:56:34.977552+0000 mon.a (mon.0) 2959 : cluster 0 osdmap e536: 8 total, 8 up, 6 in 2024-04-02T04:56:35.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:35 smithi098 bash[23790]: audit 2024-04-02T04:56:35.084936+0000 mon.c (mon.2) 935 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:35.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:35.718 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:35.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.893141+0000 osd.4 (osd.4) 293 : cluster 0 4.1a scrub starts 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:33.895592+0000 osd.4 (osd.4) 294 : cluster 0 4.1a scrub ok 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:34.167857+0000 mgr.y (mgr.14538) 505 : cluster 0 pgmap v789: 313 pgs: 2 active, 14 creating+peering, 12 unknown, 1 active+remapped+backfill_toofull, 284 active+clean; 8.4 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:34.197436+0000 osd.2 (osd.2) 244 : cluster 0 43.18 scrub starts 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:34.199596+0000 osd.2 (osd.2) 245 : cluster 0 43.18 scrub ok 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:34.475395+0000 osd.1 (osd.1) 297 : cluster 0 6.19 scrub starts 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:34.477616+0000 osd.1 (osd.1) 298 : cluster 0 6.19 scrub ok 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: cluster 2024-04-02T04:56:34.977552+0000 mon.a (mon.0) 2959 : cluster 0 osdmap e536: 8 total, 8 up, 6 in 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[28904]: audit 2024-04-02T04:56:35.084936+0000 mon.c (mon.2) 935 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:35.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.893141+0000 osd.4 (osd.4) 293 : cluster 0 4.1a scrub starts 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:33.895592+0000 osd.4 (osd.4) 294 : cluster 0 4.1a scrub ok 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:34.167857+0000 mgr.y (mgr.14538) 505 : cluster 0 pgmap v789: 313 pgs: 2 active, 14 creating+peering, 12 unknown, 1 active+remapped+backfill_toofull, 284 active+clean; 8.4 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:34.197436+0000 osd.2 (osd.2) 244 : cluster 0 43.18 scrub starts 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:34.199596+0000 osd.2 (osd.2) 245 : cluster 0 43.18 scrub ok 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:34.475395+0000 osd.1 (osd.1) 297 : cluster 0 6.19 scrub starts 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:34.477616+0000 osd.1 (osd.1) 298 : cluster 0 6.19 scrub ok 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: cluster 2024-04-02T04:56:34.977552+0000 mon.a (mon.0) 2959 : cluster 0 osdmap e536: 8 total, 8 up, 6 in 2024-04-02T04:56:35.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:35 smithi069 bash[21528]: audit 2024-04-02T04:56:35.084936+0000 mon.c (mon.2) 935 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:35.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:35.849 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:35.952 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:35.978 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:35.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:36.082 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:36.107 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:36.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:36.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:36.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-02T04:56:36.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:36.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:36.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:36.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:36.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:36 smithi098 bash[23790]: cluster 2024-04-02T04:56:35.012685+0000 osd.0 (osd.0) 285 : cluster 0 6.10 scrub starts 2024-04-02T04:56:36.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:36 smithi098 bash[23790]: cluster 2024-04-02T04:56:35.017051+0000 osd.0 (osd.0) 286 : cluster 0 6.10 scrub ok 2024-04-02T04:56:36.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:36 smithi098 bash[23790]: cluster 2024-04-02T04:56:35.987947+0000 mon.a (mon.0) 2960 : cluster 0 osdmap e537: 8 total, 8 up, 6 in 2024-04-02T04:56:36.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:36 smithi098 bash[23790]: audit 2024-04-02T04:56:36.086137+0000 mon.c (mon.2) 936 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:36.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[21528]: cluster 2024-04-02T04:56:35.012685+0000 osd.0 (osd.0) 285 : cluster 0 6.10 scrub starts 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[21528]: cluster 2024-04-02T04:56:35.017051+0000 osd.0 (osd.0) 286 : cluster 0 6.10 scrub ok 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[21528]: cluster 2024-04-02T04:56:35.987947+0000 mon.a (mon.0) 2960 : cluster 0 osdmap e537: 8 total, 8 up, 6 in 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[21528]: audit 2024-04-02T04:56:36.086137+0000 mon.c (mon.2) 936 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[28904]: cluster 2024-04-02T04:56:35.012685+0000 osd.0 (osd.0) 285 : cluster 0 6.10 scrub starts 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[28904]: cluster 2024-04-02T04:56:35.017051+0000 osd.0 (osd.0) 286 : cluster 0 6.10 scrub ok 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[28904]: cluster 2024-04-02T04:56:35.987947+0000 mon.a (mon.0) 2960 : cluster 0 osdmap e537: 8 total, 8 up, 6 in 2024-04-02T04:56:36.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:36 smithi069 bash[28904]: audit 2024-04-02T04:56:36.086137+0000 mon.c (mon.2) 936 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:36.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:36.879 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:36.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:36.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:37.010 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:37.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:37.134 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:37.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:37.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:37.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:37.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:37.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:37.510 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:37.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:37 smithi098 bash[23790]: cluster 2024-04-02T04:56:36.168461+0000 mgr.y (mgr.14538) 506 : cluster 0 pgmap v792: 313 pgs: 2 active, 14 creating+peering, 12 unknown, 1 active+remapped+backfill_toofull, 284 active+clean; 8.4 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:37 smithi098 bash[23790]: cluster 2024-04-02T04:56:36.265807+0000 osd.2 (osd.2) 246 : cluster 0 6.14 scrub starts 2024-04-02T04:56:37.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:37 smithi098 bash[23790]: cluster 2024-04-02T04:56:36.267627+0000 osd.2 (osd.2) 247 : cluster 0 6.14 scrub ok 2024-04-02T04:56:37.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:37 smithi098 bash[23790]: audit 2024-04-02T04:56:37.087356+0000 mon.c (mon.2) 937 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:37.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:37.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[28904]: cluster 2024-04-02T04:56:36.168461+0000 mgr.y (mgr.14538) 506 : cluster 0 pgmap v792: 313 pgs: 2 active, 14 creating+peering, 12 unknown, 1 active+remapped+backfill_toofull, 284 active+clean; 8.4 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:37.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[28904]: cluster 2024-04-02T04:56:36.265807+0000 osd.2 (osd.2) 246 : cluster 0 6.14 scrub starts 2024-04-02T04:56:37.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[28904]: cluster 2024-04-02T04:56:36.267627+0000 osd.2 (osd.2) 247 : cluster 0 6.14 scrub ok 2024-04-02T04:56:37.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[28904]: audit 2024-04-02T04:56:37.087356+0000 mon.c (mon.2) 937 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:37.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[21528]: cluster 2024-04-02T04:56:36.168461+0000 mgr.y (mgr.14538) 506 : cluster 0 pgmap v792: 313 pgs: 2 active, 14 creating+peering, 12 unknown, 1 active+remapped+backfill_toofull, 284 active+clean; 8.4 MiB data, 447 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1239 objects misplaced (0.161%) 2024-04-02T04:56:37.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[21528]: cluster 2024-04-02T04:56:36.265807+0000 osd.2 (osd.2) 246 : cluster 0 6.14 scrub starts 2024-04-02T04:56:37.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[21528]: cluster 2024-04-02T04:56:36.267627+0000 osd.2 (osd.2) 247 : cluster 0 6.14 scrub ok 2024-04-02T04:56:37.756 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:37 smithi069 bash[21528]: audit 2024-04-02T04:56:37.087356+0000 mon.c (mon.2) 937 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:37.768 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:37.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:37.898 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:37.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:38.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:38.025 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:38.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:38.157 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:38.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:38.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:38.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:38.548 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:38.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:38.680 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: cluster 2024-04-02T04:56:36.866747+0000 osd.4 (osd.4) 295 : cluster 0 43.15 scrub starts 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: cluster 2024-04-02T04:56:36.869116+0000 osd.4 (osd.4) 296 : cluster 0 43.15 scrub ok 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: cluster 2024-04-02T04:56:37.485952+0000 osd.1 (osd.1) 299 : cluster 0 4.12 scrub starts 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: cluster 2024-04-02T04:56:37.487741+0000 osd.1 (osd.1) 300 : cluster 0 4.12 scrub ok 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: cluster 2024-04-02T04:56:38.078185+0000 osd.7 (osd.7) 85 : cluster 0 3.13 scrub starts 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: cluster 2024-04-02T04:56:38.082330+0000 osd.7 (osd.7) 86 : cluster 0 3.13 scrub ok 2024-04-02T04:56:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:38 smithi098 bash[23790]: audit 2024-04-02T04:56:38.088647+0000 mon.c (mon.2) 938 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: cluster 2024-04-02T04:56:36.866747+0000 osd.4 (osd.4) 295 : cluster 0 43.15 scrub starts 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: cluster 2024-04-02T04:56:36.869116+0000 osd.4 (osd.4) 296 : cluster 0 43.15 scrub ok 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: cluster 2024-04-02T04:56:37.485952+0000 osd.1 (osd.1) 299 : cluster 0 4.12 scrub starts 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: cluster 2024-04-02T04:56:37.487741+0000 osd.1 (osd.1) 300 : cluster 0 4.12 scrub ok 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: cluster 2024-04-02T04:56:38.078185+0000 osd.7 (osd.7) 85 : cluster 0 3.13 scrub starts 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: cluster 2024-04-02T04:56:38.082330+0000 osd.7 (osd.7) 86 : cluster 0 3.13 scrub ok 2024-04-02T04:56:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[28904]: audit 2024-04-02T04:56:38.088647+0000 mon.c (mon.2) 938 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: cluster 2024-04-02T04:56:36.866747+0000 osd.4 (osd.4) 295 : cluster 0 43.15 scrub starts 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: cluster 2024-04-02T04:56:36.869116+0000 osd.4 (osd.4) 296 : cluster 0 43.15 scrub ok 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: cluster 2024-04-02T04:56:37.485952+0000 osd.1 (osd.1) 299 : cluster 0 4.12 scrub starts 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: cluster 2024-04-02T04:56:37.487741+0000 osd.1 (osd.1) 300 : cluster 0 4.12 scrub ok 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: cluster 2024-04-02T04:56:38.078185+0000 osd.7 (osd.7) 85 : cluster 0 3.13 scrub starts 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: cluster 2024-04-02T04:56:38.082330+0000 osd.7 (osd.7) 86 : cluster 0 3.13 scrub ok 2024-04-02T04:56:38.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21528]: audit 2024-04-02T04:56:38.088647+0000 mon.c (mon.2) 938 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:38.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:38.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:38.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:38.945 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:38.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:39.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:56:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:56:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:56:39.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:39.077 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:39.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:39.209 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:39.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:39.337 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:39.440 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:39.466 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:39.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:39.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:39.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:39 smithi098 bash[23790]: cluster 2024-04-02T04:56:38.171087+0000 mgr.y (mgr.14538) 507 : cluster 0 pgmap v793: 313 pgs: 1 active+remapped+backfill_toofull, 2 active, 310 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 984 B/s wr, 4 op/s; 2/1254 objects misplaced (0.159%) 2024-04-02T04:56:39.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:39 smithi098 bash[23790]: cluster 2024-04-02T04:56:38.333850+0000 osd.2 (osd.2) 248 : cluster 0 40.b scrub starts 2024-04-02T04:56:39.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:39 smithi098 bash[23790]: cluster 2024-04-02T04:56:38.335853+0000 osd.2 (osd.2) 249 : cluster 0 40.b scrub ok 2024-04-02T04:56:39.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:39 smithi098 bash[23790]: audit 2024-04-02T04:56:39.089866+0000 mon.c (mon.2) 939 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:39.729 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[28904]: cluster 2024-04-02T04:56:38.171087+0000 mgr.y (mgr.14538) 507 : cluster 0 pgmap v793: 313 pgs: 1 active+remapped+backfill_toofull, 2 active, 310 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 984 B/s wr, 4 op/s; 2/1254 objects misplaced (0.159%) 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[28904]: cluster 2024-04-02T04:56:38.333850+0000 osd.2 (osd.2) 248 : cluster 0 40.b scrub starts 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[28904]: cluster 2024-04-02T04:56:38.335853+0000 osd.2 (osd.2) 249 : cluster 0 40.b scrub ok 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[28904]: audit 2024-04-02T04:56:39.089866+0000 mon.c (mon.2) 939 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[21528]: cluster 2024-04-02T04:56:38.171087+0000 mgr.y (mgr.14538) 507 : cluster 0 pgmap v793: 313 pgs: 1 active+remapped+backfill_toofull, 2 active, 310 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 984 B/s wr, 4 op/s; 2/1254 objects misplaced (0.159%) 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[21528]: cluster 2024-04-02T04:56:38.333850+0000 osd.2 (osd.2) 248 : cluster 0 40.b scrub starts 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[21528]: cluster 2024-04-02T04:56:38.335853+0000 osd.2 (osd.2) 249 : cluster 0 40.b scrub ok 2024-04-02T04:56:39.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:39 smithi069 bash[21528]: audit 2024-04-02T04:56:39.089866+0000 mon.c (mon.2) 939 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:39.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:39.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:39.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:39.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:40.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:40.222 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:40.249 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:40.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:40.381 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:40.483 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[28904]: cluster 2024-04-02T04:56:39.047712+0000 osd.0 (osd.0) 287 : cluster 0 4.6 scrub starts 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[28904]: cluster 2024-04-02T04:56:39.050024+0000 osd.0 (osd.0) 288 : cluster 0 4.6 scrub ok 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[28904]: cluster 2024-04-02T04:56:40.023791+0000 osd.6 (osd.6) 247 : cluster 0 40.7 scrub starts 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[28904]: cluster 2024-04-02T04:56:40.026579+0000 osd.6 (osd.6) 248 : cluster 0 40.7 scrub ok 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[28904]: audit 2024-04-02T04:56:40.091296+0000 mon.c (mon.2) 940 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[21528]: cluster 2024-04-02T04:56:39.047712+0000 osd.0 (osd.0) 287 : cluster 0 4.6 scrub starts 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[21528]: cluster 2024-04-02T04:56:39.050024+0000 osd.0 (osd.0) 288 : cluster 0 4.6 scrub ok 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[21528]: cluster 2024-04-02T04:56:40.023791+0000 osd.6 (osd.6) 247 : cluster 0 40.7 scrub starts 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[21528]: cluster 2024-04-02T04:56:40.026579+0000 osd.6 (osd.6) 248 : cluster 0 40.7 scrub ok 2024-04-02T04:56:40.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:40 smithi069 bash[21528]: audit 2024-04-02T04:56:40.091296+0000 mon.c (mon.2) 940 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:40.510 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:40.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:40.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:40.772 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:40.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:40.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:40 smithi098 bash[23790]: cluster 2024-04-02T04:56:39.047712+0000 osd.0 (osd.0) 287 : cluster 0 4.6 scrub starts 2024-04-02T04:56:40.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:40 smithi098 bash[23790]: cluster 2024-04-02T04:56:39.050024+0000 osd.0 (osd.0) 288 : cluster 0 4.6 scrub ok 2024-04-02T04:56:40.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:40 smithi098 bash[23790]: cluster 2024-04-02T04:56:40.023791+0000 osd.6 (osd.6) 247 : cluster 0 40.7 scrub starts 2024-04-02T04:56:40.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:40 smithi098 bash[23790]: cluster 2024-04-02T04:56:40.026579+0000 osd.6 (osd.6) 248 : cluster 0 40.7 scrub ok 2024-04-02T04:56:40.887 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:40 smithi098 bash[23790]: audit 2024-04-02T04:56:40.091296+0000 mon.c (mon.2) 940 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:40.903 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:40.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:41.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:41.034 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:41.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:41.164 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:56:41.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:41.294 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:41.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:41.423 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:41.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:41.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:41.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:41.668 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[28904]: cluster 2024-04-02T04:56:39.852114+0000 osd.4 (osd.4) 297 : cluster 0 4.18 scrub starts 2024-04-02T04:56:41.668 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[28904]: cluster 2024-04-02T04:56:39.854209+0000 osd.4 (osd.4) 298 : cluster 0 4.18 scrub ok 2024-04-02T04:56:41.668 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[28904]: cluster 2024-04-02T04:56:40.172536+0000 mgr.y (mgr.14538) 508 : cluster 0 pgmap v794: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:41.668 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[28904]: audit 2024-04-02T04:56:41.092387+0000 mon.c (mon.2) 941 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:41.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[21528]: cluster 2024-04-02T04:56:39.852114+0000 osd.4 (osd.4) 297 : cluster 0 4.18 scrub starts 2024-04-02T04:56:41.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[21528]: cluster 2024-04-02T04:56:39.854209+0000 osd.4 (osd.4) 298 : cluster 0 4.18 scrub ok 2024-04-02T04:56:41.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[21528]: cluster 2024-04-02T04:56:40.172536+0000 mgr.y (mgr.14538) 508 : cluster 0 pgmap v794: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:41.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:41 smithi069 bash[21528]: audit 2024-04-02T04:56:41.092387+0000 mon.c (mon.2) 941 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:41.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:41.817 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:41.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:41.930 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:41 smithi098 bash[23790]: cluster 2024-04-02T04:56:39.852114+0000 osd.4 (osd.4) 297 : cluster 0 4.18 scrub starts 2024-04-02T04:56:41.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:41 smithi098 bash[23790]: cluster 2024-04-02T04:56:39.854209+0000 osd.4 (osd.4) 298 : cluster 0 4.18 scrub ok 2024-04-02T04:56:41.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:41 smithi098 bash[23790]: cluster 2024-04-02T04:56:40.172536+0000 mgr.y (mgr.14538) 508 : cluster 0 pgmap v794: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:41.931 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:41 smithi098 bash[23790]: audit 2024-04-02T04:56:41.092387+0000 mon.c (mon.2) 941 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:41.946 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:41.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:42.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.077 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:42.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:42.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:42.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:42.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:42.440 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.466 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:42.569 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:42.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:42 smithi069 bash[21528]: cluster 2024-04-02T04:56:40.843066+0000 osd.4 (osd.4) 299 : cluster 0 4.13 deep-scrub starts 2024-04-02T04:56:42.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:42 smithi069 bash[21528]: cluster 2024-04-02T04:56:40.845490+0000 osd.4 (osd.4) 300 : cluster 0 4.13 deep-scrub ok 2024-04-02T04:56:42.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:42 smithi069 bash[21528]: audit 2024-04-02T04:56:42.093384+0000 mon.c (mon.2) 942 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:42.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:42 smithi069 bash[28904]: cluster 2024-04-02T04:56:40.843066+0000 osd.4 (osd.4) 299 : cluster 0 4.13 deep-scrub starts 2024-04-02T04:56:42.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:42 smithi069 bash[28904]: cluster 2024-04-02T04:56:40.845490+0000 osd.4 (osd.4) 300 : cluster 0 4.13 deep-scrub ok 2024-04-02T04:56:42.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:42 smithi069 bash[28904]: audit 2024-04-02T04:56:42.093384+0000 mon.c (mon.2) 942 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:42.597 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:42.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:42 smithi098 bash[23790]: cluster 2024-04-02T04:56:40.843066+0000 osd.4 (osd.4) 299 : cluster 0 4.13 deep-scrub starts 2024-04-02T04:56:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:42 smithi098 bash[23790]: cluster 2024-04-02T04:56:40.845490+0000 osd.4 (osd.4) 300 : cluster 0 4.13 deep-scrub ok 2024-04-02T04:56:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:42 smithi098 bash[23790]: audit 2024-04-02T04:56:42.093384+0000 mon.c (mon.2) 942 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:42.728 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:42.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:42.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.988 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:42.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:43.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:43.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:43.248 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:43.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:43.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- 172.21.15.69:0/2652167263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 msgr2=0x7fed50104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- 172.21.15.69:0/2652167263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed50104930 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed440099c0 tx=0x7fed44009cd0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- 172.21.15.69:0/2652167263 shutdown_connections 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- 172.21.15.69:0/2652167263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed50104e70 0x7fed50111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- 172.21.15.69:0/2652167263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed5010a560 0x7fed5010a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- 172.21.15.69:0/2652167263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed50104930 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- 172.21.15.69:0/2652167263 >> 172.21.15.69:0/2652167263 conn(0x7fed500fffa0 msgr2=0x7fed501023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- 172.21.15.69:0/2652167263 shutdown_connections 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- 172.21.15.69:0/2652167263 wait complete. 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 librados: starting msgr at 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 librados: starting objecter 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 Processor -- start 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- start start 2024-04-02T04:56:43.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 librados: setting wanted keys 2024-04-02T04:56:43.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 librados: calling monclient init 2024-04-02T04:56:43.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed5019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed50104e70 0x7fed5019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5010a560 0x7fed5019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed50113c90 con 0x7fed501044b0 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed50113b10 con 0x7fed5010a560 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.460+0000 7fed57d1d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed50113e10 con 0x7fed50104e70 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed552b8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed50104e70 0x7fed5019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed5019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed552b8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed50104e70 0x7fed5019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47914/0 (socket says 172.21.15.69:47914) 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed552b8700 1 -- 172.21.15.69:0/4262748968 learned_addr learned my addr 172.21.15.69:0/4262748968 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5010a560 0x7fed5019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed50104e70 msgr2=0x7fed5019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed50104e70 0x7fed5019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5010a560 msgr2=0x7fed5019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed5010a560 0x7fed5019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5019f1f0 con 0x7fed501044b0 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed50104e70 0x7fed5019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed5019a1e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fed4000ef10 tx=0x7fed4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 10 client.?.objecter ms_handle_connect 0x7fed501044b0 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4000ca70 con 0x7fed501044b0 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fed40019070 con 0x7fed501044b0 2024-04-02T04:56:43.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5019f480 con 0x7fed501044b0 2024-04-02T04:56:43.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40014480 con 0x7fed501044b0 2024-04-02T04:56:43.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed57d1d700 10 client.16572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:43.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed501a6d60 con 0x7fed501044b0 2024-04-02T04:56:43.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed57d1d700 1 librados: init done 2024-04-02T04:56:43.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fed5004ff00 con 0x7fed501044b0 2024-04-02T04:56:43.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fed40012050 con 0x7fed501044b0 2024-04-02T04:56:43.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed3806f5e0 0x7fed38071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(537..537 src has 74..537) ==== 10551+0+0 (secure 0 0 0) 0x7fed4008f670 con 0x7fed501044b0 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 osd_map(537..537 src has 74..537) 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 3 client.16572.objecter handle_osd_map got epochs [537,537] > 0 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 3 client.16572.objecter handle_osd_map decoding full epoch 537 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed3806f5e0 0x7fed38071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 20 client.16572.objecter dump_active .. 0 homeless 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.464+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v537) ==== 54+0+16 (secure 0 0 0) 0x7fed40010040 con 0x7fed501044b0 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 10 client.16572.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 1 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fed50062e50 0x7fed50063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 1 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 0 1 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 4 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 10 client.16572.objecter _send_command 1 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed501aa0c0 con 0x7fed50062e50 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed3806f5e0 0x7fed38071aa0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fed440037a0 tx=0x7fed44019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed3806f5e0 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fed50062e50 0x7fed50063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fed50062e50 0x7fed50063230 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed50062e50 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501aa0c0 con 0x7fed50062e50 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 1: 0 ) 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 10 client.16572.objecter _finish_command 1 = system:0 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 4 2024-04-02T04:56:43.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.468+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 0 1 2024-04-02T04:56:43.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 10 client.16572.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 2 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 2 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed5019b540 osd=0 3 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 0 2 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 4 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 10 client.16572.objecter _send_command 2 2024-04-02T04:56:43.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.484+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed500628d0 con 0x7fed50062e50 2024-04-02T04:56:43.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed500628d0 con 0x7fed50062e50 2024-04-02T04:56:43.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 10 client.16572.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 4 2024-04-02T04:56:43.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 0 2 2024-04-02T04:56:43.491 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 10 client.16572.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fed5002c9b0 0x7fed5002cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 1 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 4 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 10 client.16572.objecter _send_command 3 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fed5010d160 con 0x7fed5002c9b0 2024-04-02T04:56:43.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fed5002c9b0 0x7fed5002cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed5002c9b0 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fed5002c9b0 0x7fed5002cd90 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed5010d160 con 0x7fed5002c9b0 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 3: 0 ) 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 10 client.16572.objecter _finish_command 3 = system:0 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 4 2024-04-02T04:56:43.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.488+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 1 3 2024-04-02T04:56:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:43.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 10 client.16572.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 4 2024-04-02T04:56:43.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 4 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed5010cae0 osd=1 3 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 1 4 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 4 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 10 client.16572.objecter _send_command 4 2024-04-02T04:56:43.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.508+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5010c560 con 0x7fed5002c9b0 2024-04-02T04:56:43.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed5010c560 con 0x7fed5002c9b0 2024-04-02T04:56:43.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 10 client.16572.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 4 2024-04-02T04:56:43.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 1 4 2024-04-02T04:56:43.513 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 10 client.16572.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 5 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fed501ac280 0x7fed501ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 5 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 2 5 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 4 2024-04-02T04:56:43.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 10 client.16572.objecter _send_command 5 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fed501b14d0 con 0x7fed501ac280 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fed501ac280 0x7fed501ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed501ac280 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fed501ac280 0x7fed501ae670 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501b14d0 con 0x7fed501ac280 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 5: 0 ) 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 10 client.16572.objecter _finish_command 5 = system:0 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 4 2024-04-02T04:56:43.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.512+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 2 5 2024-04-02T04:56:43.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 10 client.16572.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 6 2024-04-02T04:56:43.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 6 2024-04-02T04:56:43.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed5010d310 osd=2 3 2024-04-02T04:56:43.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 2 6 2024-04-02T04:56:43.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 4 2024-04-02T04:56:43.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 10 client.16572.objecter _send_command 6 2024-04-02T04:56:43.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.528+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed5002d520 con 0x7fed501ac280 2024-04-02T04:56:43.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed5002d520 con 0x7fed501ac280 2024-04-02T04:56:43.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 10 client.16572.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 4 2024-04-02T04:56:43.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 2 6 2024-04-02T04:56:43.533 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 10 client.16572.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 7 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fed501afd30 0x7fed501b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 7 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 3 7 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 4 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 10 client.16572.objecter _send_command 7 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fed501a9de0 con 0x7fed501afd30 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fed501afd30 0x7fed501b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed501afd30 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fed501afd30 0x7fed501b0190 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501a9de0 con 0x7fed501afd30 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 7: 0 ) 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 10 client.16572.objecter _finish_command 7 = system:0 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 4 2024-04-02T04:56:43.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.532+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 3 7 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 10 client.16572.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 8 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 8 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501a9710 osd=3 3 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 3 8 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 4 2024-04-02T04:56:43.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 10 client.16572.objecter _send_command 8 2024-04-02T04:56:43.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed501a9de0 con 0x7fed501afd30 2024-04-02T04:56:43.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed501a9de0 con 0x7fed501afd30 2024-04-02T04:56:43.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed4effd700 10 client.16572.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 4 2024-04-02T04:56:43.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 3 8 2024-04-02T04:56:43.555 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:56:43.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 10 client.16572.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 9 2024-04-02T04:56:43.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fed501b0ea0 0x7fed501b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 9 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 4 9 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 4 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 10 client.16572.objecter _send_command 9 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fed501b8500 con 0x7fed501b0ea0 2024-04-02T04:56:43.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.552+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fed501b0ea0 0x7fed501b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed501b0ea0 2024-04-02T04:56:43.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fed501b0ea0 0x7fed501b56a0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501b8500 con 0x7fed501b0ea0 2024-04-02T04:56:43.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 9: 0 ) 2024-04-02T04:56:43.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed4effd700 10 client.16572.objecter _finish_command 9 = system:0 2024-04-02T04:56:43.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 4 2024-04-02T04:56:43.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.556+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 4 9 2024-04-02T04:56:43.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 10 client.16572.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 10 2024-04-02T04:56:43.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 10 2024-04-02T04:56:43.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501a9de0 osd=4 3 2024-04-02T04:56:43.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 4 10 2024-04-02T04:56:43.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 4 2024-04-02T04:56:43.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 10 client.16572.objecter _send_command 10 2024-04-02T04:56:43.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.572+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed501b5be0 con 0x7fed501b0ea0 2024-04-02T04:56:43.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed501b5be0 con 0x7fed501b0ea0 2024-04-02T04:56:43.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed4effd700 10 client.16572.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 4 2024-04-02T04:56:43.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 4 10 2024-04-02T04:56:43.580 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 10 client.16572.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 11 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fed501b6350 0x7fed501b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 11 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 5 11 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 4 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 10 client.16572.objecter _send_command 11 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.576+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fed501bb870 con 0x7fed501b6350 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fed501b6350 0x7fed501b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed501b6350 2024-04-02T04:56:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed552b8700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fed501b6350 0x7fed501b6790 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501bb870 con 0x7fed501b6350 2024-04-02T04:56:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 11: 0 ) 2024-04-02T04:56:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed4effd700 10 client.16572.objecter _finish_command 11 = system:0 2024-04-02T04:56:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 4 2024-04-02T04:56:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.580+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 5 11 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 10 client.16572.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 12 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 12 2024-04-02T04:56:43.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501b5be0 osd=5 3 2024-04-02T04:56:43.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 5 12 2024-04-02T04:56:43.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 4 2024-04-02T04:56:43.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 10 client.16572.objecter _send_command 12 2024-04-02T04:56:43.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.596+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed501ba6f0 con 0x7fed501b6350 2024-04-02T04:56:43.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:43.605 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed501ba6f0 con 0x7fed501b6350 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 10 client.16572.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 4 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 5 12 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 10 client.16572.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 13 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fed501bae10 0x7fed501bb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 13 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 6 13 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 4 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 10 client.16572.objecter _send_command 13 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fed501bec10 con 0x7fed501bae10 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fed501bae10 0x7fed501bb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed501bae10 2024-04-02T04:56:43.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed562ba700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fed501bae10 0x7fed501bb250 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501bec10 con 0x7fed501bae10 2024-04-02T04:56:43.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 13: 0 ) 2024-04-02T04:56:43.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 10 client.16572.objecter _finish_command 13 = system:0 2024-04-02T04:56:43.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 4 2024-04-02T04:56:43.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.604+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 6 13 2024-04-02T04:56:43.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:56:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:56:43.618 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:43 smithi098 bash[23790]: cluster 2024-04-02T04:56:42.173354+0000 mgr.y (mgr.14538) 509 : cluster 0 pgmap v795: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 895 B/s wr, 3 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:43.618 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:43 smithi098 bash[23790]: cluster 2024-04-02T04:56:42.252245+0000 osd.2 (osd.2) 250 : cluster 0 10.3 deep-scrub starts 2024-04-02T04:56:43.618 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:43 smithi098 bash[23790]: cluster 2024-04-02T04:56:42.254211+0000 osd.2 (osd.2) 251 : cluster 0 10.3 deep-scrub ok 2024-04-02T04:56:43.618 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:43 smithi098 bash[23790]: audit 2024-04-02T04:56:43.094650+0000 mon.c (mon.2) 943 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:43.618 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:43 smithi098 bash[23790]: audit 2024-04-02T04:56:43.469992+0000 mon.a (mon.0) 2961 : audit 0 from='client.? 172.21.15.69:0/4262748968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 10 client.16572.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 14 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 14 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501ba6f0 osd=6 3 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 6 14 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 4 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 10 client.16572.objecter _send_command 14 2024-04-02T04:56:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed501b6de0 con 0x7fed501bae10 2024-04-02T04:56:43.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed501b6de0 con 0x7fed501bae10 2024-04-02T04:56:43.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed4effd700 10 client.16572.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 4 2024-04-02T04:56:43.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 6 14 2024-04-02T04:56:43.628 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:56:43.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 10 client.16572.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:43.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 15 2024-04-02T04:56:43.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fed501b7500 0x7fed501b7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 15 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 7 15 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 4 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 10 client.16572.objecter _send_command 15 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.624+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fed501b8000 con 0x7fed501b7500 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fed501b7500 0x7fed501b7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed55ab9700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fed501b7500 0x7fed501b7920 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed4effd700 10 client.16572.objecter ms_handle_connect 0x7fed501b7500 2024-04-02T04:56:43.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed501b8000 con 0x7fed501b7500 2024-04-02T04:56:43.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 15: 0 ) 2024-04-02T04:56:43.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed4effd700 10 client.16572.objecter _finish_command 15 = system:0 2024-04-02T04:56:43.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 4 2024-04-02T04:56:43.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.628+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 7 15 2024-04-02T04:56:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:43.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 10 client.16572.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:56:43.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign -1 16 2024-04-02T04:56:43.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 20 client.16572.objecter _get_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_remove -1 16 2024-04-02T04:56:43.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 20 client.16572.objecter get_session s=0x7fed501b6de0 osd=7 3 2024-04-02T04:56:43.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 15 client.16572.objecter _session_command_op_assign 7 16 2024-04-02T04:56:43.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 4 2024-04-02T04:56:43.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 10 client.16572.objecter _send_command 16 2024-04-02T04:56:43.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.644+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed501b7fc0 con 0x7fed501b7500 2024-04-02T04:56:43.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed4effd700 1 -- 172.21.15.69:0/4262748968 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed501b7fc0 con 0x7fed501b7500 2024-04-02T04:56:43.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed4effd700 10 client.16572.objecter ms_dispatch 0x7fed50001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:43.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed4effd700 10 client.16572.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:43.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed4effd700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 4 2024-04-02T04:56:43.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed4effd700 15 client.16572.objecter _session_command_op_remove 7 16 2024-04-02T04:56:43.650 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 librados: watch_flush enter 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 librados: watch_flush exit 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.0 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fed50062e50 msgr2=0x7fed50063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fed50062e50 0x7fed50063230 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5019b540 osd=0 1 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.1 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fed5002c9b0 msgr2=0x7fed5002cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fed5002c9b0 0x7fed5002cd90 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010cae0 osd=1 1 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.2 2024-04-02T04:56:43.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fed501ac280 msgr2=0x7fed501ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fed501ac280 0x7fed501ae670 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed5010d310 osd=2 1 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.3 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fed501afd30 msgr2=0x7fed501b0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fed501afd30 0x7fed501b0190 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9710 osd=3 1 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.4 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fed501b0ea0 msgr2=0x7fed501b56a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fed501b0ea0 0x7fed501b56a0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501a9de0 osd=4 1 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.5 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fed501b6350 msgr2=0x7fed501b6790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fed501b6350 0x7fed501b6790 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b5be0 osd=5 1 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.6 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fed501bae10 msgr2=0x7fed501bb250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fed501bae10 0x7fed501bb250 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501ba6f0 osd=6 1 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter close_session for osd.7 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fed501b7500 msgr2=0x7fed501b7920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fed501b7500 0x7fed501b7920 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter put_session s=0x7fed501b6de0 osd=7 1 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 20 client.16572.objecter shutdown clearing up homeless session... 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 10 client.16572.objecter successfully canceled tick 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed562ba700 1 -- 172.21.15.69:0/4262748968 reap_dead start 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed562ba700 1 -- 172.21.15.69:0/4262748968 reap_dead start 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed562ba700 1 -- 172.21.15.69:0/4262748968 reap_dead start 2024-04-02T04:56:43.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed562ba700 1 -- 172.21.15.69:0/4262748968 reap_dead start 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed3806f5e0 msgr2=0x7fed38071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed3806f5e0 0x7fed38071aa0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fed440037a0 tx=0x7fed44019040 comp rx=0 tx=0).stop 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed562ba700 1 -- 172.21.15.69:0/4262748968 reap_dead start 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 msgr2=0x7fed5019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed5019a1e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fed4000ef10 tx=0x7fed4000c3a0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 shutdown_connections 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 --2- 172.21.15.69:0/4262748968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed501044b0 0x7fed5019a1e0 secure :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fed4000ef10 tx=0x7fed4000c3a0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 >> 172.21.15.69:0/4262748968 conn(0x7fed500fffa0 msgr2=0x7fed5010b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 shutdown_connections 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 -- 172.21.15.69:0/4262748968 wait complete. 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.648+0000 7fed57d1d700 1 librados: shutdown 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:56:43.655 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:56:43.656 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:56:43.656 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:56:43.656 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:56:43.656 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:56:43.656 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:56:43.656 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[21528]: cluster 2024-04-02T04:56:42.173354+0000 mgr.y (mgr.14538) 509 : cluster 0 pgmap v795: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 895 B/s wr, 3 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[21528]: cluster 2024-04-02T04:56:42.252245+0000 osd.2 (osd.2) 250 : cluster 0 10.3 deep-scrub starts 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[21528]: cluster 2024-04-02T04:56:42.254211+0000 osd.2 (osd.2) 251 : cluster 0 10.3 deep-scrub ok 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[21528]: audit 2024-04-02T04:56:43.094650+0000 mon.c (mon.2) 943 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[21528]: audit 2024-04-02T04:56:43.469992+0000 mon.a (mon.0) 2961 : audit 0 from='client.? 172.21.15.69:0/4262748968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[28904]: cluster 2024-04-02T04:56:42.173354+0000 mgr.y (mgr.14538) 509 : cluster 0 pgmap v795: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 895 B/s wr, 3 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[28904]: cluster 2024-04-02T04:56:42.252245+0000 osd.2 (osd.2) 250 : cluster 0 10.3 deep-scrub starts 2024-04-02T04:56:43.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[28904]: cluster 2024-04-02T04:56:42.254211+0000 osd.2 (osd.2) 251 : cluster 0 10.3 deep-scrub ok 2024-04-02T04:56:43.664 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[28904]: audit 2024-04-02T04:56:43.094650+0000 mon.c (mon.2) 943 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:43.664 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:43 smithi069 bash[28904]: audit 2024-04-02T04:56:43.469992+0000 mon.a (mon.0) 2961 : audit 0 from='client.? 172.21.15.69:0/4262748968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:43.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:43.758 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:43.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- 172.21.15.69:0/2199978420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74102320 msgr2=0x7fba741027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/2199978420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74102320 0x7fba741027a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fba680098f0 tx=0x7fba68009c00 comp rx=0 tx=0).stop 2024-04-02T04:56:43.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- 172.21.15.69:0/2199978420 shutdown_connections 2024-04-02T04:56:43.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/2199978420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/2199978420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba74108390 0x7fba74108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/2199978420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74102320 0x7fba741027a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- 172.21.15.69:0/2199978420 >> 172.21.15.69:0/2199978420 conn(0x7fba74079310 msgr2=0x7fba74079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- 172.21.15.69:0/2199978420 shutdown_connections 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- 172.21.15.69:0/2199978420 wait complete. 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 librados: starting msgr at 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 librados: starting objecter 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 Processor -- start 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- start start 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 librados: setting wanted keys 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 librados: calling monclient init 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba74102320 0x7fba7419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74108390 0x7fba7419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba74111ad0 con 0x7fba74102ce0 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba74111950 con 0x7fba74102320 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7a3e0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba74111c50 con 0x7fba74108390 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba7897d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74108390 0x7fba7419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7419a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38344/0 (socket says 172.21.15.69:38344) 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 -- 172.21.15.69:0/3338395593 learned_addr learned my addr 172.21.15.69:0/3338395593 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba73fff700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba74102320 0x7fba7419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 -- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74108390 msgr2=0x7fba7419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74108390 0x7fba7419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 -- 172.21.15.69:0/3338395593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba74102320 msgr2=0x7fba7419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba74102320 0x7fba7419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 -- 172.21.15.69:0/3338395593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7419f2a0 con 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba737fe700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7419a7d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba680059d0 tx=0x7fba6800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba717fa700 10 client.?.objecter ms_handle_connect 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.804+0000 7fba717fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68027070 con 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fba6800bb60 con 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7419f530 con 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68004930 con 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba7a3e0700 10 client.16578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba741a6e10 con 0x7fba74102ce0 2024-04-02T04:56:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba7a3e0700 1 librados: init done 2024-04-02T04:56:43.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba7404ff00 con 0x7fba74102ce0 2024-04-02T04:56:43.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fba6802a070 con 0x7fba74102ce0 2024-04-02T04:56:43.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba5c06f5e0 0x7fba5c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(537..537 src has 74..537) ==== 10551+0+0 (secure 0 0 0) 0x7fba6808f740 con 0x7fba74102ce0 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 10 client.16578.objecter ms_dispatch 0x7fba74001550 osd_map(537..537 src has 74..537) 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 3 client.16578.objecter handle_osd_map got epochs [537,537] > 0 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 3 client.16578.objecter handle_osd_map decoding full epoch 537 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba73fff700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba5c06f5e0 0x7fba5c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 20 client.16578.objecter dump_active .. 0 homeless 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba73fff700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba5c06f5e0 0x7fba5c071aa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fba64005fd0 tx=0x7fba6400a560 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.808+0000 7fba717fa700 10 client.16578.objecter ms_handle_connect 0x7fba5c06f5e0 2024-04-02T04:56:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.812+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba68013030 con 0x7fba74102ce0 2024-04-02T04:56:43.861 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:43.880 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:43.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:43.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 -- 172.21.15.69:0/1602972849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102520 msgr2=0x7f6e24102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 --2- 172.21.15.69:0/1602972849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102520 0x7f6e24102900 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c0099c0 tx=0x7f6e0c009cd0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 -- 172.21.15.69:0/1602972849 shutdown_connections 2024-04-02T04:56:43.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 --2- 172.21.15.69:0/1602972849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e241075a0 0x7f6e2410f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 --2- 172.21.15.69:0/1602972849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102520 0x7f6e24102900 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 --2- 172.21.15.69:0/1602972849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e24102ed0 0x7f6e24106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 -- 172.21.15.69:0/1602972849 >> 172.21.15.69:0/1602972849 conn(0x7f6e24079360 msgr2=0x7f6e24079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:43.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 -- 172.21.15.69:0/1602972849 shutdown_connections 2024-04-02T04:56:43.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 -- 172.21.15.69:0/1602972849 wait complete. 2024-04-02T04:56:43.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 librados: starting msgr at 2024-04-02T04:56:43.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 librados: starting objecter 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.888+0000 7f6e28d38700 1 Processor -- start 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- start start 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 librados: setting wanted keys 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 librados: calling monclient init 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24102520 0x7f6e24198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102ed0 0x7f6e241985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 0x7f6e2419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24111a70 con 0x7f6e241075a0 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241118f0 con 0x7f6e24102520 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6e24111bf0 con 0x7f6e24102ed0 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e22d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 0x7f6e2419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102ed0 0x7f6e241985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e22d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 0x7f6e2419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38350/0 (socket says 172.21.15.69:38350) 2024-04-02T04:56:43.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e22d9d700 1 -- 172.21.15.69:0/2883876868 learned_addr learned my addr 172.21.15.69:0/2883876868 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e2259c700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24102520 0x7f6e24198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 -- 172.21.15.69:0/2883876868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24102520 msgr2=0x7f6e24198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24102520 0x7f6e24198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 -- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 msgr2=0x7f6e2419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 0x7f6e2419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 -- 172.21.15.69:0/2883876868 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2419d080 con 0x7f6e24102ed0 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e22d9d700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 0x7f6e2419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e2259c700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24102520 0x7f6e24198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e21d9b700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102ed0 0x7f6e241985b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6e14010cd0 tx=0x7f6e1400b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 10 client.?.objecter ms_handle_connect 0x7f6e24102ed0 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400bba0 con 0x7f6e24102ed0 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6e140044a0 con 0x7f6e24102ed0 2024-04-02T04:56:43.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2419d370 con 0x7f6e24102ed0 2024-04-02T04:56:43.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14003e20 con 0x7f6e24102ed0 2024-04-02T04:56:43.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 10 client.26675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:43.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e24074470 con 0x7f6e24102ed0 2024-04-02T04:56:43.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 librados: init done 2024-04-02T04:56:43.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e2404ff00 con 0x7f6e24102ed0 2024-04-02T04:56:43.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6e14019410 con 0x7f6e24102ed0 2024-04-02T04:56:43.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.892+0000 7f6e1b7fe700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6e1006f5e0 0x7f6e10071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:43.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(537..537 src has 74..537) ==== 10551+0+0 (secure 0 0 0) 0x7f6e14092590 con 0x7f6e24102ed0 2024-04-02T04:56:43.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e1b7fe700 10 client.26675.objecter ms_dispatch 0x7f6e24001550 osd_map(537..537 src has 74..537) 2024-04-02T04:56:43.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e1b7fe700 3 client.26675.objecter handle_osd_map got epochs [537,537] > 0 2024-04-02T04:56:43.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e1b7fe700 3 client.26675.objecter handle_osd_map decoding full epoch 537 2024-04-02T04:56:43.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e2259c700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6e1006f5e0 0x7f6e10071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:43.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e1b7fe700 20 client.26675.objecter dump_active .. 0 homeless 2024-04-02T04:56:43.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e1b7fe700 10 client.26675.objecter ms_handle_connect 0x7f6e1006f5e0 2024-04-02T04:56:43.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.896+0000 7f6e2259c700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6e1006f5e0 0x7f6e10071aa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c006010 tx=0x7f6e0c005ab0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:43.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:43.900+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6e1405ef60 con 0x7f6e24102ed0 2024-04-02T04:56:43.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:44.013 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:44.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.040+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fba741a6fe0 con 0x7fba74102ce0 2024-04-02T04:56:44.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:44.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.124+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]} v 0) -- 0x7f6e240746d0 con 0x7f6e24102ed0 2024-04-02T04:56:44.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:44.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:44.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:44.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:44.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:44.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:44.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.496+0000 7fba717fa700 1 -- 172.21.15.69:0/3338395593 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v538) ==== 87+0+0 (secure 0 0 0) 0x7fba6805c190 con 0x7fba74102ce0 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 10 librados: watch_flush enter 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 10 librados: watch_flush exit 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 20 client.16578.objecter shutdown clearing up homeless session... 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 10 client.16578.objecter successfully canceled tick 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba5c06f5e0 msgr2=0x7fba5c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba5c06f5e0 0x7fba5c071aa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fba64005fd0 tx=0x7fba6400a560 comp rx=0 tx=0).stop 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 msgr2=0x7fba7419a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.500+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7419a7d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba680059d0 tx=0x7fba6800fa70 comp rx=0 tx=0).stop 2024-04-02T04:56:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 shutdown_connections 2024-04-02T04:56:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba5c06f5e0 0x7fba5c071aa0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba74108390 0x7fba7419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba74102320 0x7fba7419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 --2- 172.21.15.69:0/3338395593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba74102ce0 0x7fba7419a7d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 >> 172.21.15.69:0/3338395593 conn(0x7fba74079310 msgr2=0x7fba74074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:44.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.508+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 shutdown_connections 2024-04-02T04:56:44.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.512+0000 7fba7a3e0700 1 -- 172.21.15.69:0/3338395593 wait complete. 2024-04-02T04:56:44.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.512+0000 7fba7a3e0700 1 librados: shutdown 2024-04-02T04:56:44.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.512+0000 7f6e1b7fe700 1 -- 172.21.15.69:0/2883876868 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]=0 set 40.1 pg_upmap mapping to [4,1,6] v538) ==== 129+0+0 (secure 0 0 0) 0x7f6e1407e0a0 con 0x7f6e24102ed0 2024-04-02T04:56:44.513 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:56:44.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:44.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 10 librados: watch_flush enter 2024-04-02T04:56:44.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 10 librados: watch_flush exit 2024-04-02T04:56:44.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 20 client.26675.objecter shutdown clearing up homeless session... 2024-04-02T04:56:44.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 10 client.26675.objecter successfully canceled tick 2024-04-02T04:56:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:44.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6e1006f5e0 msgr2=0x7f6e10071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:44.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6e1006f5e0 0x7f6e10071aa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c006010 tx=0x7f6e0c005ab0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102ed0 msgr2=0x7f6e241985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102ed0 0x7f6e241985b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6e14010cd0 tx=0x7f6e1400b2c0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 shutdown_connections 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6e1006f5e0 0x7f6e10071aa0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e241075a0 0x7f6e2419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24102520 0x7f6e24198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 --2- 172.21.15.69:0/2883876868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e24102ed0 0x7f6e241985b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 >> 172.21.15.69:0/2883876868 conn(0x7f6e24079360 msgr2=0x7f6e24100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 shutdown_connections 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 -- 172.21.15.69:0/2883876868 wait complete. 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:44.516+0000 7f6e28d38700 1 librados: shutdown 2024-04-02T04:56:44.520 INFO:teuthology.orchestra.run.smithi069.stderr:set 40.1 pg_upmap mapping to [4,1,6] 2024-04-02T04:56:44.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:44.644 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:44.681 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:56:44.681 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T04:56:44.681 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T04:56:44.681 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T04:56:44.682 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:56:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:44 smithi098 bash[23790]: audit 2024-04-02T04:56:43.376778+0000 mgr.y (mgr.14538) 510 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:44 smithi098 bash[23790]: audit 2024-04-02T04:56:44.044374+0000 mon.a (mon.0) 2962 : audit 1 from='client.? 172.21.15.69:0/3338395593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:56:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:44 smithi098 bash[23790]: audit 2024-04-02T04:56:44.096026+0000 mon.c (mon.2) 944 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:44 smithi098 bash[23790]: audit 2024-04-02T04:56:44.132070+0000 mon.c (mon.2) 945 : audit 1 from='client.? 172.21.15.69:0/2883876868' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]: dispatch 2024-04-02T04:56:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:44 smithi098 bash[23790]: audit 2024-04-02T04:56:44.132783+0000 mon.a (mon.0) 2963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]: dispatch 2024-04-02T04:56:44.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[28904]: audit 2024-04-02T04:56:43.376778+0000 mgr.y (mgr.14538) 510 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:44.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[28904]: audit 2024-04-02T04:56:44.044374+0000 mon.a (mon.0) 2962 : audit 1 from='client.? 172.21.15.69:0/3338395593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:56:44.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[28904]: audit 2024-04-02T04:56:44.096026+0000 mon.c (mon.2) 944 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:44.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[28904]: audit 2024-04-02T04:56:44.132070+0000 mon.c (mon.2) 945 : audit 1 from='client.? 172.21.15.69:0/2883876868' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]: dispatch 2024-04-02T04:56:44.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[28904]: audit 2024-04-02T04:56:44.132783+0000 mon.a (mon.0) 2963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]: dispatch 2024-04-02T04:56:44.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[21528]: audit 2024-04-02T04:56:43.376778+0000 mgr.y (mgr.14538) 510 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:44.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[21528]: audit 2024-04-02T04:56:44.044374+0000 mon.a (mon.0) 2962 : audit 1 from='client.? 172.21.15.69:0/3338395593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:56:44.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[21528]: audit 2024-04-02T04:56:44.096026+0000 mon.c (mon.2) 944 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:44.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[21528]: audit 2024-04-02T04:56:44.132070+0000 mon.c (mon.2) 945 : audit 1 from='client.? 172.21.15.69:0/2883876868' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]: dispatch 2024-04-02T04:56:44.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:44 smithi069 bash[21528]: audit 2024-04-02T04:56:44.132783+0000 mon.a (mon.0) 2963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]: dispatch 2024-04-02T04:56:44.747 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:44.770 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:44.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:44.901 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:44.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:45.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:45.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:45.162 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:45.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:45.288 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:45.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:45.416 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:45.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:45.543 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:45.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-02T04:56:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:45.647 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:45.672 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:45.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:43.551625+0000 osd.1 (osd.1) 301 : cluster 0 4.2 scrub starts 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:43.553055+0000 osd.1 (osd.1) 302 : cluster 0 4.2 scrub ok 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:43.819633+0000 osd.4 (osd.4) 301 : cluster 0 40.13 scrub starts 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:43.822068+0000 osd.4 (osd.4) 302 : cluster 0 40.13 scrub ok 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:44.084270+0000 osd.0 (osd.0) 289 : cluster 0 6.13 scrub starts 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:44.085838+0000 osd.0 (osd.0) 290 : cluster 0 6.13 scrub ok 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:44.175537+0000 mgr.y (mgr.14538) 511 : cluster 0 pgmap v796: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 779 B/s wr, 3 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:45.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:44.490347+0000 mon.a (mon.0) 2964 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: audit 2024-04-02T04:56:44.499957+0000 mon.a (mon.0) 2965 : audit 1 from='client.? 172.21.15.69:0/3338395593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: audit 2024-04-02T04:56:44.500171+0000 mon.a (mon.0) 2966 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]': finished 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:44.508574+0000 mon.a (mon.0) 2967 : cluster 0 osdmap e538: 8 total, 8 up, 6 in 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: cluster 2024-04-02T04:56:44.947477+0000 mon.a (mon.0) 2968 : cluster 0 osdmap e539: 8 total, 8 up, 6 in 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[21528]: audit 2024-04-02T04:56:45.097531+0000 mon.c (mon.2) 946 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:43.551625+0000 osd.1 (osd.1) 301 : cluster 0 4.2 scrub starts 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:43.553055+0000 osd.1 (osd.1) 302 : cluster 0 4.2 scrub ok 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:43.819633+0000 osd.4 (osd.4) 301 : cluster 0 40.13 scrub starts 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:43.822068+0000 osd.4 (osd.4) 302 : cluster 0 40.13 scrub ok 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:44.084270+0000 osd.0 (osd.0) 289 : cluster 0 6.13 scrub starts 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:44.085838+0000 osd.0 (osd.0) 290 : cluster 0 6.13 scrub ok 2024-04-02T04:56:45.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:44.175537+0000 mgr.y (mgr.14538) 511 : cluster 0 pgmap v796: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 779 B/s wr, 3 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:45.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:44.490347+0000 mon.a (mon.0) 2964 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:56:45.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: audit 2024-04-02T04:56:44.499957+0000 mon.a (mon.0) 2965 : audit 1 from='client.? 172.21.15.69:0/3338395593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:56:45.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: audit 2024-04-02T04:56:44.500171+0000 mon.a (mon.0) 2966 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]': finished 2024-04-02T04:56:45.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:44.508574+0000 mon.a (mon.0) 2967 : cluster 0 osdmap e538: 8 total, 8 up, 6 in 2024-04-02T04:56:45.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: cluster 2024-04-02T04:56:44.947477+0000 mon.a (mon.0) 2968 : cluster 0 osdmap e539: 8 total, 8 up, 6 in 2024-04-02T04:56:45.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:45 smithi069 bash[28904]: audit 2024-04-02T04:56:45.097531+0000 mon.c (mon.2) 946 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:43.551625+0000 osd.1 (osd.1) 301 : cluster 0 4.2 scrub starts 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:43.553055+0000 osd.1 (osd.1) 302 : cluster 0 4.2 scrub ok 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:43.819633+0000 osd.4 (osd.4) 301 : cluster 0 40.13 scrub starts 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:43.822068+0000 osd.4 (osd.4) 302 : cluster 0 40.13 scrub ok 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:44.084270+0000 osd.0 (osd.0) 289 : cluster 0 6.13 scrub starts 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:44.085838+0000 osd.0 (osd.0) 290 : cluster 0 6.13 scrub ok 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:44.175537+0000 mgr.y (mgr.14538) 511 : cluster 0 pgmap v796: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 779 B/s wr, 3 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:44.490347+0000 mon.a (mon.0) 2964 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: audit 2024-04-02T04:56:44.499957+0000 mon.a (mon.0) 2965 : audit 1 from='client.? 172.21.15.69:0/3338395593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:56:45.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: audit 2024-04-02T04:56:44.500171+0000 mon.a (mon.0) 2966 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "40.1", "id": [4, 1, 6]}]': finished 2024-04-02T04:56:45.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:44.508574+0000 mon.a (mon.0) 2967 : cluster 0 osdmap e538: 8 total, 8 up, 6 in 2024-04-02T04:56:45.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: cluster 2024-04-02T04:56:44.947477+0000 mon.a (mon.0) 2968 : cluster 0 osdmap e539: 8 total, 8 up, 6 in 2024-04-02T04:56:45.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:45 smithi098 bash[23790]: audit 2024-04-02T04:56:45.097531+0000 mon.c (mon.2) 946 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:45.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:45.924 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:45.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:46.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:46.053 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:46.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:46.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:46.314 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:46.417 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:46.445 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:46.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:46.560 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:46 smithi098 bash[23790]: audit 2024-04-02T04:56:46.073260+0000 mon.c (mon.2) 947 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:46.560 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:46 smithi098 bash[23790]: audit 2024-04-02T04:56:46.073927+0000 mon.a (mon.0) 2969 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:46.560 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:46 smithi098 bash[23790]: audit 2024-04-02T04:56:46.075183+0000 mon.c (mon.2) 948 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-73"}]: dispatch 2024-04-02T04:56:46.560 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:46 smithi098 bash[23790]: audit 2024-04-02T04:56:46.075786+0000 mon.a (mon.0) 2970 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-73"}]: dispatch 2024-04-02T04:56:46.560 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:46 smithi098 bash[23790]: audit 2024-04-02T04:56:46.099063+0000 mon.c (mon.2) 949 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:46.577 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:46.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:56:46.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:46.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[28904]: audit 2024-04-02T04:56:46.073260+0000 mon.c (mon.2) 947 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:46.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[28904]: audit 2024-04-02T04:56:46.073927+0000 mon.a (mon.0) 2969 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:46.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[28904]: audit 2024-04-02T04:56:46.075183+0000 mon.c (mon.2) 948 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-73"}]: dispatch 2024-04-02T04:56:46.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[28904]: audit 2024-04-02T04:56:46.075786+0000 mon.a (mon.0) 2970 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-73"}]: dispatch 2024-04-02T04:56:46.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[28904]: audit 2024-04-02T04:56:46.099063+0000 mon.c (mon.2) 949 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:46.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[21528]: audit 2024-04-02T04:56:46.073260+0000 mon.c (mon.2) 947 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:46.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[21528]: audit 2024-04-02T04:56:46.073927+0000 mon.a (mon.0) 2969 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:46.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[21528]: audit 2024-04-02T04:56:46.075183+0000 mon.c (mon.2) 948 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-73"}]: dispatch 2024-04-02T04:56:46.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[21528]: audit 2024-04-02T04:56:46.075786+0000 mon.a (mon.0) 2970 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-73"}]: dispatch 2024-04-02T04:56:46.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:46 smithi069 bash[21528]: audit 2024-04-02T04:56:46.099063+0000 mon.c (mon.2) 949 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:46.810 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:46.828 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:46.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:46.954 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:46.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:47.056 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:47.080 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:47.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:47.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:47.338 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:47.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:47.469 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:47.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:47.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:47.710 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:47 smithi069 bash[28904]: cluster 2024-04-02T04:56:46.176648+0000 mgr.y (mgr.14538) 512 : cluster 0 pgmap v799: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:47.710 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:47 smithi069 bash[28904]: cluster 2024-04-02T04:56:46.522450+0000 mon.a (mon.0) 2971 : cluster 0 osdmap e540: 8 total, 8 up, 6 in 2024-04-02T04:56:47.710 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:47 smithi069 bash[28904]: audit 2024-04-02T04:56:47.100559+0000 mon.c (mon.2) 950 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:47.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:47 smithi069 bash[21528]: cluster 2024-04-02T04:56:46.176648+0000 mgr.y (mgr.14538) 512 : cluster 0 pgmap v799: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:47.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:47 smithi069 bash[21528]: cluster 2024-04-02T04:56:46.522450+0000 mon.a (mon.0) 2971 : cluster 0 osdmap e540: 8 total, 8 up, 6 in 2024-04-02T04:56:47.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:47 smithi069 bash[21528]: audit 2024-04-02T04:56:47.100559+0000 mon.c (mon.2) 950 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:47.726 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:47.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:56:47.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:47.840 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:47 smithi098 bash[23790]: cluster 2024-04-02T04:56:46.176648+0000 mgr.y (mgr.14538) 512 : cluster 0 pgmap v799: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:47.840 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:47 smithi098 bash[23790]: cluster 2024-04-02T04:56:46.522450+0000 mon.a (mon.0) 2971 : cluster 0 osdmap e540: 8 total, 8 up, 6 in 2024-04-02T04:56:47.840 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:47 smithi098 bash[23790]: audit 2024-04-02T04:56:47.100559+0000 mon.c (mon.2) 950 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:47.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:47.983 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:47.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:48.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:48.113 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:48.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:48.244 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:48.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:48.375 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:48.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:48.608 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:48.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21528]: cluster 2024-04-02T04:56:47.525615+0000 mon.a (mon.0) 2972 : cluster 0 osdmap e541: 8 total, 8 up, 6 in 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21528]: audit 2024-04-02T04:56:47.529071+0000 mon.c (mon.2) 951 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21528]: audit 2024-04-02T04:56:47.529823+0000 mon.a (mon.0) 2973 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21528]: audit 2024-04-02T04:56:48.101783+0000 mon.c (mon.2) 952 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21528]: audit 2024-04-02T04:56:48.257454+0000 mon.b (mon.1) 269 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21528]: audit 2024-04-02T04:56:48.258719+0000 mon.a (mon.0) 2974 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[28904]: cluster 2024-04-02T04:56:47.525615+0000 mon.a (mon.0) 2972 : cluster 0 osdmap e541: 8 total, 8 up, 6 in 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[28904]: audit 2024-04-02T04:56:47.529071+0000 mon.c (mon.2) 951 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[28904]: audit 2024-04-02T04:56:47.529823+0000 mon.a (mon.0) 2973 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[28904]: audit 2024-04-02T04:56:48.101783+0000 mon.c (mon.2) 952 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:48.618 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[28904]: audit 2024-04-02T04:56:48.257454+0000 mon.b (mon.1) 269 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:48.619 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[28904]: audit 2024-04-02T04:56:48.258719+0000 mon.a (mon.0) 2974 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:48.630 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:48.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:48.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:48 smithi098 bash[23790]: cluster 2024-04-02T04:56:47.525615+0000 mon.a (mon.0) 2972 : cluster 0 osdmap e541: 8 total, 8 up, 6 in 2024-04-02T04:56:48.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:48 smithi098 bash[23790]: audit 2024-04-02T04:56:47.529071+0000 mon.c (mon.2) 951 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:48.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:48 smithi098 bash[23790]: audit 2024-04-02T04:56:47.529823+0000 mon.a (mon.0) 2973 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:48.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:48 smithi098 bash[23790]: audit 2024-04-02T04:56:48.101783+0000 mon.c (mon.2) 952 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:48.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:48 smithi098 bash[23790]: audit 2024-04-02T04:56:48.257454+0000 mon.b (mon.1) 269 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:48.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:48 smithi098 bash[23790]: audit 2024-04-02T04:56:48.258719+0000 mon.a (mon.0) 2974 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:56:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:48.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:48.874 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:56:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:56:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:56:48.889 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:48.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:48.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:49.017 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:49.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:49.147 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:49.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:49.278 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:49.380 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:49.402 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:49.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:49.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:49.656 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:49.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:49.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:49 smithi098 bash[23790]: cluster 2024-04-02T04:56:48.179216+0000 mgr.y (mgr.14538) 513 : cluster 0 pgmap v802: 313 pgs: 1 active+remapped+backfill_toofull, 25 creating+peering, 7 unknown, 280 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:49.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:49 smithi098 bash[23790]: cluster 2024-04-02T04:56:48.522301+0000 mon.a (mon.0) 2975 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:49.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:49 smithi098 bash[23790]: audit 2024-04-02T04:56:48.531977+0000 mon.a (mon.0) 2976 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:49.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:49 smithi098 bash[23790]: cluster 2024-04-02T04:56:48.544297+0000 mon.a (mon.0) 2977 : cluster 0 osdmap e542: 8 total, 8 up, 6 in 2024-04-02T04:56:49.772 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:49 smithi098 bash[23790]: audit 2024-04-02T04:56:48.548914+0000 mon.c (mon.2) 953 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:49.772 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:49 smithi098 bash[23790]: audit 2024-04-02T04:56:49.103566+0000 mon.c (mon.2) 954 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[21528]: cluster 2024-04-02T04:56:48.179216+0000 mgr.y (mgr.14538) 513 : cluster 0 pgmap v802: 313 pgs: 1 active+remapped+backfill_toofull, 25 creating+peering, 7 unknown, 280 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[21528]: cluster 2024-04-02T04:56:48.522301+0000 mon.a (mon.0) 2975 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[21528]: audit 2024-04-02T04:56:48.531977+0000 mon.a (mon.0) 2976 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[21528]: cluster 2024-04-02T04:56:48.544297+0000 mon.a (mon.0) 2977 : cluster 0 osdmap e542: 8 total, 8 up, 6 in 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[21528]: audit 2024-04-02T04:56:48.548914+0000 mon.c (mon.2) 953 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[21528]: audit 2024-04-02T04:56:49.103566+0000 mon.c (mon.2) 954 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:49.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[28904]: cluster 2024-04-02T04:56:48.179216+0000 mgr.y (mgr.14538) 513 : cluster 0 pgmap v802: 313 pgs: 1 active+remapped+backfill_toofull, 25 creating+peering, 7 unknown, 280 active+clean; 8.4 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:49.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[28904]: cluster 2024-04-02T04:56:48.522301+0000 mon.a (mon.0) 2975 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:49.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[28904]: audit 2024-04-02T04:56:48.531977+0000 mon.a (mon.0) 2976 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:49.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[28904]: cluster 2024-04-02T04:56:48.544297+0000 mon.a (mon.0) 2977 : cluster 0 osdmap e542: 8 total, 8 up, 6 in 2024-04-02T04:56:49.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[28904]: audit 2024-04-02T04:56:48.548914+0000 mon.c (mon.2) 953 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:49.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:49 smithi069 bash[28904]: audit 2024-04-02T04:56:49.103566+0000 mon.c (mon.2) 954 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:49.786 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:49.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:49.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:49.916 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:49.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:50.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:50.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:50.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:50.358 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T04:56:50.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:50.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:50.560 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:50.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:50.786 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:50 smithi098 bash[23790]: cluster 2024-04-02T04:56:49.550859+0000 mon.a (mon.0) 2978 : cluster 0 osdmap e543: 8 total, 8 up, 6 in 2024-04-02T04:56:50.786 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:50 smithi098 bash[23790]: audit 2024-04-02T04:56:50.105362+0000 mon.c (mon.2) 955 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:50.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:50.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7] out_osds: [5, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:56:50.811 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:56:50.811 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-02T04:56:50.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd test-reweight-by-utilization 110 3.0 2024-04-02T04:56:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:50.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:50 smithi069 bash[28904]: cluster 2024-04-02T04:56:49.550859+0000 mon.a (mon.0) 2978 : cluster 0 osdmap e543: 8 total, 8 up, 6 in 2024-04-02T04:56:50.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:50 smithi069 bash[28904]: audit 2024-04-02T04:56:50.105362+0000 mon.c (mon.2) 955 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:50.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:50 smithi069 bash[21528]: cluster 2024-04-02T04:56:49.550859+0000 mon.a (mon.0) 2978 : cluster 0 osdmap e543: 8 total, 8 up, 6 in 2024-04-02T04:56:50.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:50 smithi069 bash[21528]: audit 2024-04-02T04:56:50.105362+0000 mon.c (mon.2) 955 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:50.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:50.945 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:50.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:51.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:51.180 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:51.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:51.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:51.382 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:51.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:51.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:51.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:51.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: cluster 2024-04-02T04:56:50.179744+0000 mgr.y (mgr.14538) 514 : cluster 0 pgmap v805: 313 pgs: 1 active+remapped+backfill_toofull, 25 creating+peering, 287 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:51.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: cluster 2024-04-02T04:56:50.569896+0000 mon.a (mon.0) 2979 : cluster 0 osdmap e544: 8 total, 8 up, 6 in 2024-04-02T04:56:51.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: audit 2024-04-02T04:56:50.663141+0000 mon.c (mon.2) 956 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: audit 2024-04-02T04:56:50.663836+0000 mon.a (mon.0) 2980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: audit 2024-04-02T04:56:50.664759+0000 mon.c (mon.2) 957 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-75"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: audit 2024-04-02T04:56:50.665230+0000 mon.a (mon.0) 2981 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-75"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[28904]: audit 2024-04-02T04:56:51.107113+0000 mon.c (mon.2) 958 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: cluster 2024-04-02T04:56:50.179744+0000 mgr.y (mgr.14538) 514 : cluster 0 pgmap v805: 313 pgs: 1 active+remapped+backfill_toofull, 25 creating+peering, 287 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: cluster 2024-04-02T04:56:50.569896+0000 mon.a (mon.0) 2979 : cluster 0 osdmap e544: 8 total, 8 up, 6 in 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: audit 2024-04-02T04:56:50.663141+0000 mon.c (mon.2) 956 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: audit 2024-04-02T04:56:50.663836+0000 mon.a (mon.0) 2980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: audit 2024-04-02T04:56:50.664759+0000 mon.c (mon.2) 957 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-75"}]: dispatch 2024-04-02T04:56:51.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: audit 2024-04-02T04:56:50.665230+0000 mon.a (mon.0) 2981 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-75"}]: dispatch 2024-04-02T04:56:51.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:51 smithi069 bash[21528]: audit 2024-04-02T04:56:51.107113+0000 mon.c (mon.2) 958 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: cluster 2024-04-02T04:56:50.179744+0000 mgr.y (mgr.14538) 514 : cluster 0 pgmap v805: 313 pgs: 1 active+remapped+backfill_toofull, 25 creating+peering, 287 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/1251 objects misplaced (0.160%) 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: cluster 2024-04-02T04:56:50.569896+0000 mon.a (mon.0) 2979 : cluster 0 osdmap e544: 8 total, 8 up, 6 in 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: audit 2024-04-02T04:56:50.663141+0000 mon.c (mon.2) 956 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: audit 2024-04-02T04:56:50.663836+0000 mon.a (mon.0) 2980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: audit 2024-04-02T04:56:50.664759+0000 mon.c (mon.2) 957 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-75"}]: dispatch 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: audit 2024-04-02T04:56:50.665230+0000 mon.a (mon.0) 2981 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-75"}]: dispatch 2024-04-02T04:56:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:51 smithi098 bash[23790]: audit 2024-04-02T04:56:51.107113+0000 mon.c (mon.2) 958 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:51.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:51.858 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:51.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:51.983 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:51.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:52.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:52.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:52.240 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:52.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:52.400 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:52.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:52.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:52.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:52.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:52 smithi069 bash[28904]: cluster 2024-04-02T04:56:51.588340+0000 mon.a (mon.0) 2982 : cluster 0 osdmap e545: 8 total, 8 up, 6 in 2024-04-02T04:56:52.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:52 smithi069 bash[28904]: audit 2024-04-02T04:56:52.108814+0000 mon.c (mon.2) 959 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:52.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:52 smithi069 bash[21528]: cluster 2024-04-02T04:56:51.588340+0000 mon.a (mon.0) 2982 : cluster 0 osdmap e545: 8 total, 8 up, 6 in 2024-04-02T04:56:52.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:52 smithi069 bash[21528]: audit 2024-04-02T04:56:52.108814+0000 mon.c (mon.2) 959 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:52.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:52.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:52 smithi098 bash[23790]: cluster 2024-04-02T04:56:51.588340+0000 mon.a (mon.0) 2982 : cluster 0 osdmap e545: 8 total, 8 up, 6 in 2024-04-02T04:56:52.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:52 smithi098 bash[23790]: audit 2024-04-02T04:56:52.108814+0000 mon.c (mon.2) 959 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:52.880 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:52.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:52.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:53.009 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:53.112 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:53.138 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:53.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:53.268 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:53.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:53.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:53.526 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:53.594 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:56:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:56:53.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:53.659 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:53.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:53.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:53 smithi098 bash[23790]: cluster 2024-04-02T04:56:52.180568+0000 mgr.y (mgr.14538) 515 : cluster 0 pgmap v808: 281 pgs: 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:53.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:53 smithi098 bash[23790]: cluster 2024-04-02T04:56:52.602418+0000 mon.a (mon.0) 2983 : cluster 0 osdmap e546: 8 total, 8 up, 6 in 2024-04-02T04:56:53.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:53 smithi098 bash[23790]: audit 2024-04-02T04:56:52.607054+0000 mon.c (mon.2) 960 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:53.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:53 smithi098 bash[23790]: audit 2024-04-02T04:56:52.608339+0000 mon.a (mon.0) 2984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:53.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:53 smithi098 bash[23790]: audit 2024-04-02T04:56:53.110439+0000 mon.c (mon.2) 961 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:53.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[21528]: cluster 2024-04-02T04:56:52.180568+0000 mgr.y (mgr.14538) 515 : cluster 0 pgmap v808: 281 pgs: 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[21528]: cluster 2024-04-02T04:56:52.602418+0000 mon.a (mon.0) 2983 : cluster 0 osdmap e546: 8 total, 8 up, 6 in 2024-04-02T04:56:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[21528]: audit 2024-04-02T04:56:52.607054+0000 mon.c (mon.2) 960 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[21528]: audit 2024-04-02T04:56:52.608339+0000 mon.a (mon.0) 2984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:53.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[21528]: audit 2024-04-02T04:56:53.110439+0000 mon.c (mon.2) 961 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:53.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[28904]: cluster 2024-04-02T04:56:52.180568+0000 mgr.y (mgr.14538) 515 : cluster 0 pgmap v808: 281 pgs: 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 2/1248 objects misplaced (0.160%) 2024-04-02T04:56:53.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[28904]: cluster 2024-04-02T04:56:52.602418+0000 mon.a (mon.0) 2983 : cluster 0 osdmap e546: 8 total, 8 up, 6 in 2024-04-02T04:56:53.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[28904]: audit 2024-04-02T04:56:52.607054+0000 mon.c (mon.2) 960 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:53.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[28904]: audit 2024-04-02T04:56:52.608339+0000 mon.a (mon.0) 2984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:53.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:53 smithi069 bash[28904]: audit 2024-04-02T04:56:53.110439+0000 mon.c (mon.2) 961 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:53.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:53.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:54.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:54.040 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:54.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:54.167 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:54.261 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "pg_update_log_missing(4.14 epoch 395/394 rep_tid 993 entries 395'14 (0'0) error 4:2b04a3e9:::notify.6:head by client.14415.0:2499 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:54.932320+0000", 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.324375334, 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078416639999999999, 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932320+0000", 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932320+0000", 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932320+0000", 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2300000000000001e-07 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932329+0000", 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8489999999999995e-06 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932345+0000", 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5594e-05 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932362+0000", 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7847999999999999e-05 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.935886+0000", 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035239780000000001 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.940161+0000", 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042748719999999999 2024-04-02T04:56:54.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(102.11 395/394 [PushOp(102:8b5a2dca:test-rados-api-smithi069-72349-12::bar:2, version: 179'2, data_included: [0~8], data_size: 8, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(102:8b5a2dca:test-rados-api-smithi069-72349-12::bar:2@179'2, size: 8, copy_subset: [(0, 8)], clone_subset: {}, snapset: 2=[]:{2: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:54.941304+0000", 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.31539090300001, 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060379550000000002, 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.941304+0000", 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.941304+0000", 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.941305+0000", 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6499999999999998e-07 2024-04-02T04:56:54.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.941397+0000", 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2272e-05 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.941416+0000", 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9287e-05 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.941442+0000", 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5650999999999999e-05 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.946218+0000", 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047756539999999998 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.946227+0000", 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5820000000000008e-06 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.947342+0000", 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011146439999999999 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.6,QREQUEST,e:398,from:3,reservation_nonce:3)", 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:58.130328+0000", 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 176.126366491, 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009765213, 2024-04-02T04:56:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130328+0000", 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130328+0000", 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130329+0000", 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6700000000000003e-07 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130336+0000", 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7279999999999998e-06 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130347+0000", 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1447000000000001e-05 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130356+0000", 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8459999999999997e-06 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130458+0000", 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101839 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130468+0000", 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5170000000000005e-06 2024-04-02T04:56:54.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.139993+0000", 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095249900000000005 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.140023+0000", 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0608999999999998e-05 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.140094+0000", 2024-04-02T04:56:54.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0469999999999994e-05 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2562 3.f 3:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:01.893914+0000", 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.36278064499999, 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044473919999999997, 2024-04-02T04:56:54.275 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2562 2024-04-02T04:56:54.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.893914+0000", 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.893914+0000", 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.893919+0000", 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7079999999999996e-06 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.893943+0000", 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4283000000000001e-05 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.893987+0000", 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3278999999999997e-05 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.894011+0000", 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4612000000000001e-05 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.894216+0000", 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000205138 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.894598+0000", 2024-04-02T04:56:54.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00038165699999999998 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.895165+0000", 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056697300000000002 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.896407+0000", 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001241577 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.896885+0000", 2024-04-02T04:56:54.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00047890900000000002 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.898197+0000", 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001311689 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.898271+0000", 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3438999999999999e-05 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.898362+0000", 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1127999999999999e-05 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2565 3.e 3:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:01.904153+0000", 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.35254204699999, 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011711485000000001, 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2565 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904153+0000", 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904153+0000", 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904154+0000", 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.406e-06 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904166+0000", 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2153e-05 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.909907+0000", 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057403050000000002 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.909922+0000", 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5135e-05 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910011+0000", 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9221999999999994e-05 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910127+0000", 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115573 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910294+0000", 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000167171 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910707+0000", 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041359600000000002 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.911193+0000", 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048597799999999997 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.915775+0000", 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045810590000000002 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.915824+0000", 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9920999999999999e-05 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.915864+0000", 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9966000000000003e-05 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2567 3.e 3:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:01.918137+0000", 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.33855765000001, 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077450569999999996, 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2567 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918137+0000", 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918137+0000", 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918138+0000", 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.124e-06 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918145+0000", 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.477e-06 2024-04-02T04:56:54.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918159+0000", 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4569999999999999e-05 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918166+0000", 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8979999999999999e-06 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918226+0000", 2024-04-02T04:56:54.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0010999999999999e-05 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918330+0000", 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103717 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918554+0000", 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000223923 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:56:54.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.919108+0000", 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055412099999999998 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.919531+0000", 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042319299999999998 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.925816+0000", 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062843250000000003 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.925853+0000", 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7339000000000003e-05 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.925882+0000", 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9359000000000001e-05 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.295 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2850 3.f 3:f6792a7a:::obj_delete_at_hint.0000000097:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:02.343024+0000", 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 171.91367102800001, 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040362130000000003, 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2850 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343024+0000", 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343024+0000", 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343026+0000", 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8029999999999999e-06 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343032+0000", 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4749999999999998e-06 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343046+0000", 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4195999999999999e-05 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343054+0000", 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2760000000000003e-06 2024-04-02T04:56:54.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343114+0000", 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0374000000000002e-05 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343261+0000", 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014702399999999999 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343437+0000", 2024-04-02T04:56:54.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017575300000000001 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.343787+0000", 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034968 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.346943+0000", 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031561139999999998 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.346986+0000", 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3278999999999997e-05 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.347024+0000", 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8365999999999998e-05 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:02.347060+0000", 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5873e-05 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:5120 280.f 280:f0000000:%01:::head [pgnls start_epoch 404] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e404)", 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:03.963023+0000", 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 170.29367157300001, 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062735100000000004, 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5120 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963023+0000", 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963023+0000", 2024-04-02T04:56:54.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963026+0000", 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3379999999999999e-06 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963037+0000", 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1173e-05 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963062+0000", 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4992999999999999e-05 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963072+0000", 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0332999999999999e-05 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963141+0000", 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8725000000000004e-05 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.963146+0000", 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9699999999999998e-06 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.969056+0000", 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059099039999999997 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.969099+0000", 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3001999999999999e-05 2024-04-02T04:56:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:03.969297+0000", 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019807200000000001 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.2.0:2575 102.11 e404/394 102:8a7c15c8:test-rados-api-smithi069-72349-14::foo28:4 v 404'67, mlcod=402'63)", 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:05.025556+0000", 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 169.23113873599999, 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054021269999999996, 2024-04-02T04:56:54.307 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025556+0000", 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025556+0000", 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025557+0000", 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.37e-07 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025568+0000", 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.134e-05 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025594+0000", 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5435999999999998e-05 2024-04-02T04:56:54.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025616+0000", 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.228e-05 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.026159+0000", 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054261000000000001 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.026228+0000", 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9319000000000005e-05 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.030175+0000", 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039471590000000004 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.030958+0000", 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00078324600000000003 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.2.0:2576 102.11 e404/394 102:8d89ec23:test-rados-api-smithi069-72349-14::foo44:4 v 404'69, mlcod=402'63)", 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:05.025968+0000", 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 169.23072656599999, 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050810849999999999, 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025968+0000", 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025968+0000", 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025970+0000", 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6330000000000001e-06 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.025990+0000", 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9956e-05 2024-04-02T04:56:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.026010+0000", 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0302999999999999e-05 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.026048+0000", 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7586000000000002e-05 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.026944+0000", 2024-04-02T04:56:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089641700000000005 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.027002+0000", 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8046e-05 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.030967+0000", 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039647060000000001 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:05.031049+0000", 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2437999999999999e-05 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25807.0:1 284.5s0 284:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:07.910149+0000", 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 166.346546152, 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0227512809999999, 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25807", 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2058437918", 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910149+0000", 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910149+0000", 2024-04-02T04:56:54.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910152+0000", 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1329999999999999e-06 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910168+0000", 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5869e-05 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910196+0000", 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.798e-05 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910246+0000", 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9818999999999999e-05 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910420+0000", 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017426400000000001 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910427+0000", 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4429999999999997e-06 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.930403+0000", 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.019975705 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.930592+0000", 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018934099999999999 2024-04-02T04:56:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.931423+0000", 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00083051700000000002 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.931945+0000", 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052247299999999997 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.932847+0000", 2024-04-02T04:56:54.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00090180299999999996 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.932900+0000", 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2933999999999998e-05 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:5692 287.c 287:30000000:%01:::head [pgnls start_epoch 437] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e437)", 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:44.920753+0000", 2024-04-02T04:56:54.322 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 129.33594217300001, 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018551937000000001, 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5692 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920753+0000", 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920753+0000", 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920756+0000", 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9960000000000002e-06 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920771+0000", 2024-04-02T04:56:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4922999999999999e-05 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920804+0000", 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3676000000000001e-05 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920822+0000", 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7952999999999999e-05 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920994+0000", 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017200399999999999 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.921015+0000", 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.936426+0000", 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015410902000000001 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.936486+0000", 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9821000000000002e-05 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.939305+0000", 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028187400000000001 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(3.12,QREQUEST,e:467,from:0,reservation_nonce:3)", 2024-04-02T04:56:54.327 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:15.086079+0000", 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 99.170616015999997, 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051896160000000002, 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086079+0000", 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086079+0000", 2024-04-02T04:56:54.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086081+0000", 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.77e-06 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086096+0000", 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5424e-05 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086124+0000", 2024-04-02T04:56:54.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7835e-05 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086139+0000", 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5440000000000001e-05 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086305+0000", 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016566700000000001 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.086339+0000", 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3716e-05 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.091087+0000", 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047482000000000002 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.091141+0000", 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3672999999999999e-05 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.091269+0000", 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012789100000000001 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "replica_scrub(pg: 3.12,from:0'0,to:402'15,epoch:467/394,start:3:48000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:15.094728+0000", 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 99.161966532999998, 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060995219999999996, 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.094728+0000", 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.094728+0000", 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.094730+0000", 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.573e-06 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.094742+0000", 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2447e-05 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.096166+0000", 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014238129999999999 2024-04-02T04:56:54.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.096527+0000", 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036063000000000002 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.100621+0000", 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040941830000000004 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.100659+0000", 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7901999999999997e-05 2024-04-02T04:56:54.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:15.100828+0000", 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016897400000000001 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(41.15,QREQUEST,e:470,from:0,reservation_nonce:3)", 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:18.032436+0000", 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 96.224259067000006, 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01257341, 2024-04-02T04:56:54.336 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032436+0000", 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032436+0000", 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032438+0000", 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032447+0000", 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7960000000000007e-06 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032475+0000", 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7319999999999999e-05 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032491+0000", 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6685999999999999e-05 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032667+0000", 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017570699999999999 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032679+0000", 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.216e-05 2024-04-02T04:56:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.044882+0000", 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012202697 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.044914+0000", 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1596e-05 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.045009+0000", 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5665e-05 2024-04-02T04:56:54.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(41.0,QREQUEST,e:472,from:4,reservation_nonce:3)", 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:20.027942+0000", 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.228752510999996, 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021636908, 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027942+0000", 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027942+0000", 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027944+0000", 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027955+0000", 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0936e-05 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027992+0000", 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6973e-05 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.028013+0000", 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0862000000000001e-05 2024-04-02T04:56:54.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.028153+0000", 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000140667 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.028163+0000", 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8400000000000007e-06 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.049430+0000", 2024-04-02T04:56:54.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021266835000000001 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.049472+0000", 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2120000000000003e-05 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.049579+0000", 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107091 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(43.1d,QREQUEST,e:473,from:4,reservation_nonce:3)", 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:20.973101+0000", 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 93.283594441999995, 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062073639999999999, 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973101+0000", 2024-04-02T04:56:54.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973101+0000", 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973102+0000", 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3170000000000001e-06 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973109+0000", 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1740000000000003e-06 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973128+0000", 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9006999999999999e-05 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973137+0000", 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0180000000000007e-06 2024-04-02T04:56:54.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973219+0000", 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1639999999999998e-05 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.973226+0000", 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1799999999999999e-06 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.979204+0000", 2024-04-02T04:56:54.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059782710000000003 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.979227+0000", 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2682000000000001e-05 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.979308+0000", 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1075000000000006e-05 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:6481 295.c 295:3019b403:test-rados-api-smithi069-72349-66::bar:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e497)", 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:44.983792+0000", 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 69.272902931999994, 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045936550000000003, 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T04:56:54.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6481 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.983792+0000", 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.983792+0000", 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.983796+0000", 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3460000000000004e-06 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.983818+0000", 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1846999999999999e-05 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.983873+0000", 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4802000000000001e-05 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.983901+0000", 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8136e-05 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.984102+0000", 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000200843 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.984441+0000", 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033868100000000002 2024-04-02T04:56:54.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,6", 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.984795+0000", 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035393399999999998 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.986242+0000", 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001447704 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.986282+0000", 2024-04-02T04:56:54.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9752000000000002e-05 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.988193+0000", 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019105459999999999 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.988283+0000", 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0044999999999996e-05 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.988386+0000", 2024-04-02T04:56:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103019 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.25442.0:6844 102.13 e522/394 102:cc426ad7:test-rados-api-smithi069-72349-72::foo:head v 522'101, mlcod=515'99)", 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:56:10.004223+0000", 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 44.252471991999997, 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049928380000000003, 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004223+0000", 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004223+0000", 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004224+0000", 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2699999999999999e-07 2024-04-02T04:56:54.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004236+0000", 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2169e-05 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004261+0000", 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5449999999999999e-05 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004285+0000", 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3538999999999999e-05 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004444+0000", 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000159128 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.004508+0000", 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4496e-05 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.009097+0000", 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045884150000000002 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:10.009216+0000", 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011891399999999999 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.361 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:72 300.14 300:294f4083:test-rados-api-smithi069-72349-76::chunk4:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e544)", 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:56:50.586901+0000", 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.6697944319999998, 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042071130000000002, 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.586901+0000", 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.586901+0000", 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.586905+0000", 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0300000000000004e-06 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.586927+0000", 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2203000000000001e-05 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.586978+0000", 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1572e-05 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.587006+0000", 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7657e-05 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.587730+0000", 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072450900000000003 2024-04-02T04:56:54.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.588131+0000", 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040058299999999998 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.588756+0000", 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00062524499999999997 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.589862+0000", 2024-04-02T04:56:54.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011059100000000001 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.589941+0000", 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9167999999999998e-05 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.590830+0000", 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088902000000000002 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.591000+0000", 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016960400000000001 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:56:50.591108+0000", 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107612 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T04:56:54.367 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T04:56:54.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:54.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:54.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:54.661 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:54.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:54.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: audit 2024-04-02T04:56:53.384828+0000 mgr.y (mgr.14538) 516 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:54.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: audit 2024-04-02T04:56:53.605334+0000 mon.a (mon.0) 2985 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:54.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: cluster 2024-04-02T04:56:53.609270+0000 mon.a (mon.0) 2986 : cluster 0 osdmap e547: 8 total, 8 up, 6 in 2024-04-02T04:56:54.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: audit 2024-04-02T04:56:53.615315+0000 mon.c (mon.2) 962 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:54.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: cluster 2024-04-02T04:56:53.819839+0000 osd.0 (osd.0) 291 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: cluster 2024-04-02T04:56:53.820019+0000 osd.0 (osd.0) 292 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[28904]: audit 2024-04-02T04:56:54.112060+0000 mon.c (mon.2) 963 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: audit 2024-04-02T04:56:53.384828+0000 mgr.y (mgr.14538) 516 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: audit 2024-04-02T04:56:53.605334+0000 mon.a (mon.0) 2985 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: cluster 2024-04-02T04:56:53.609270+0000 mon.a (mon.0) 2986 : cluster 0 osdmap e547: 8 total, 8 up, 6 in 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: audit 2024-04-02T04:56:53.615315+0000 mon.c (mon.2) 962 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: cluster 2024-04-02T04:56:53.819839+0000 osd.0 (osd.0) 291 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: cluster 2024-04-02T04:56:53.820019+0000 osd.0 (osd.0) 292 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:54.779 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:54 smithi069 bash[21528]: audit 2024-04-02T04:56:54.112060+0000 mon.c (mon.2) 963 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:54.793 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:54.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:54.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:55.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:55.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:55.181 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: audit 2024-04-02T04:56:53.384828+0000 mgr.y (mgr.14538) 516 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: audit 2024-04-02T04:56:53.605334+0000 mon.a (mon.0) 2985 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: cluster 2024-04-02T04:56:53.609270+0000 mon.a (mon.0) 2986 : cluster 0 osdmap e547: 8 total, 8 up, 6 in 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: audit 2024-04-02T04:56:53.615315+0000 mon.c (mon.2) 962 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: cluster 2024-04-02T04:56:53.819839+0000 osd.0 (osd.0) 291 : cluster 0 purged_snaps scrub starts 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: cluster 2024-04-02T04:56:53.820019+0000 osd.0 (osd.0) 292 : cluster 0 purged_snaps scrub ok 2024-04-02T04:56:55.222 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:54 smithi098 bash[23790]: audit 2024-04-02T04:56:54.112060+0000 mon.c (mon.2) 963 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:55.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-02T04:56:55.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:55.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:55.433 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:55.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:55.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:55.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:55.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: cluster 2024-04-02T04:56:54.181529+0000 mgr.y (mgr.14538) 517 : cluster 0 pgmap v811: 313 pgs: 1 active+remapped+backfill_toofull, 27 creating+peering, 285 active+clean; 8.4 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:55.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: cluster 2024-04-02T04:56:54.630488+0000 mon.a (mon.0) 2987 : cluster 0 osdmap e548: 8 total, 8 up, 6 in 2024-04-02T04:56:55.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: audit 2024-04-02T04:56:54.720290+0000 mon.c (mon.2) 964 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:55.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: audit 2024-04-02T04:56:54.721235+0000 mon.a (mon.0) 2988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: audit 2024-04-02T04:56:54.722578+0000 mon.c (mon.2) 965 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-77"}]: dispatch 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: audit 2024-04-02T04:56:54.723321+0000 mon.a (mon.0) 2989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-77"}]: dispatch 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: cluster 2024-04-02T04:56:54.941959+0000 mon.a (mon.0) 2990 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: cluster 2024-04-02T04:56:54.947717+0000 mon.a (mon.0) 2991 : cluster 0 osdmap e549: 8 total, 8 up, 6 in 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[28904]: audit 2024-04-02T04:56:55.113277+0000 mon.c (mon.2) 966 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: cluster 2024-04-02T04:56:54.181529+0000 mgr.y (mgr.14538) 517 : cluster 0 pgmap v811: 313 pgs: 1 active+remapped+backfill_toofull, 27 creating+peering, 285 active+clean; 8.4 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: cluster 2024-04-02T04:56:54.630488+0000 mon.a (mon.0) 2987 : cluster 0 osdmap e548: 8 total, 8 up, 6 in 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: audit 2024-04-02T04:56:54.720290+0000 mon.c (mon.2) 964 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: audit 2024-04-02T04:56:54.721235+0000 mon.a (mon.0) 2988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:55.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: audit 2024-04-02T04:56:54.722578+0000 mon.c (mon.2) 965 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-77"}]: dispatch 2024-04-02T04:56:55.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: audit 2024-04-02T04:56:54.723321+0000 mon.a (mon.0) 2989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-77"}]: dispatch 2024-04-02T04:56:55.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: cluster 2024-04-02T04:56:54.941959+0000 mon.a (mon.0) 2990 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:55.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: cluster 2024-04-02T04:56:54.947717+0000 mon.a (mon.0) 2991 : cluster 0 osdmap e549: 8 total, 8 up, 6 in 2024-04-02T04:56:55.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:55 smithi069 bash[21528]: audit 2024-04-02T04:56:55.113277+0000 mon.c (mon.2) 966 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:55.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:55.807 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: cluster 2024-04-02T04:56:54.181529+0000 mgr.y (mgr.14538) 517 : cluster 0 pgmap v811: 313 pgs: 1 active+remapped+backfill_toofull, 27 creating+peering, 285 active+clean; 8.4 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 2/1260 objects misplaced (0.159%) 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: cluster 2024-04-02T04:56:54.630488+0000 mon.a (mon.0) 2987 : cluster 0 osdmap e548: 8 total, 8 up, 6 in 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: audit 2024-04-02T04:56:54.720290+0000 mon.c (mon.2) 964 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: audit 2024-04-02T04:56:54.721235+0000 mon.a (mon.0) 2988 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: audit 2024-04-02T04:56:54.722578+0000 mon.c (mon.2) 965 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-77"}]: dispatch 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: audit 2024-04-02T04:56:54.723321+0000 mon.a (mon.0) 2989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-77"}]: dispatch 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: cluster 2024-04-02T04:56:54.941959+0000 mon.a (mon.0) 2990 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: cluster 2024-04-02T04:56:54.947717+0000 mon.a (mon.0) 2991 : cluster 0 osdmap e549: 8 total, 8 up, 6 in 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:55 smithi098 bash[23790]: audit 2024-04-02T04:56:55.113277+0000 mon.c (mon.2) 966 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:55.823 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:55.926 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:55.965 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:55.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:56.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:56.096 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:56.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:56.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:56.357 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:56.451 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:56:56.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:56.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:56.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:56.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:56.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:56.881 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:56.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:56.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:57.001 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:56 smithi098 bash[23790]: cluster 2024-04-02T04:56:55.953389+0000 mon.a (mon.0) 2992 : cluster 0 osdmap e550: 8 total, 8 up, 6 in 2024-04-02T04:56:57.002 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:56 smithi098 bash[23790]: audit 2024-04-02T04:56:55.958287+0000 mon.c (mon.2) 967 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:57.002 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:56 smithi098 bash[23790]: audit 2024-04-02T04:56:55.961066+0000 mon.a (mon.0) 2993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:57.002 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:56 smithi098 bash[23790]: audit 2024-04-02T04:56:56.114892+0000 mon.c (mon.2) 968 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:57.002 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:56 smithi098 bash[23790]: cluster 2024-04-02T04:56:56.182535+0000 mgr.y (mgr.14538) 518 : cluster 0 pgmap v815: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 856 B/s wr, 3 op/s; 2/1257 objects misplaced (0.159%) 2024-04-02T04:56:57.017 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:56:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:56:57.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:57.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:57.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[21528]: cluster 2024-04-02T04:56:55.953389+0000 mon.a (mon.0) 2992 : cluster 0 osdmap e550: 8 total, 8 up, 6 in 2024-04-02T04:56:57.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[21528]: audit 2024-04-02T04:56:55.958287+0000 mon.c (mon.2) 967 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:57.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[21528]: audit 2024-04-02T04:56:55.961066+0000 mon.a (mon.0) 2993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[21528]: audit 2024-04-02T04:56:56.114892+0000 mon.c (mon.2) 968 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[21528]: cluster 2024-04-02T04:56:56.182535+0000 mgr.y (mgr.14538) 518 : cluster 0 pgmap v815: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 856 B/s wr, 3 op/s; 2/1257 objects misplaced (0.159%) 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[28904]: cluster 2024-04-02T04:56:55.953389+0000 mon.a (mon.0) 2992 : cluster 0 osdmap e550: 8 total, 8 up, 6 in 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[28904]: audit 2024-04-02T04:56:55.958287+0000 mon.c (mon.2) 967 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[28904]: audit 2024-04-02T04:56:55.961066+0000 mon.a (mon.0) 2993 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[28904]: audit 2024-04-02T04:56:56.114892+0000 mon.c (mon.2) 968 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:57.257 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:56 smithi069 bash[28904]: cluster 2024-04-02T04:56:56.182535+0000 mgr.y (mgr.14538) 518 : cluster 0 pgmap v815: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 856 B/s wr, 3 op/s; 2/1257 objects misplaced (0.159%) 2024-04-02T04:56:57.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:57.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:57.403 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:57.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:57.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:57.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:57.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:57.763 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:57.790 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:57.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:57.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:58.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:58.048 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:58.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:58.174 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:58.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:57 smithi098 bash[23790]: audit 2024-04-02T04:56:56.959608+0000 mon.a (mon.0) 2994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:58.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:57 smithi098 bash[23790]: cluster 2024-04-02T04:56:56.967439+0000 mon.a (mon.0) 2995 : cluster 0 osdmap e551: 8 total, 8 up, 6 in 2024-04-02T04:56:58.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:57 smithi098 bash[23790]: audit 2024-04-02T04:56:56.970863+0000 mon.c (mon.2) 969 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:58.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:57 smithi098 bash[23790]: audit 2024-04-02T04:56:57.116612+0000 mon.c (mon.2) 970 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:58.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:58.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[21528]: audit 2024-04-02T04:56:56.959608+0000 mon.a (mon.0) 2994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:58.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[21528]: cluster 2024-04-02T04:56:56.967439+0000 mon.a (mon.0) 2995 : cluster 0 osdmap e551: 8 total, 8 up, 6 in 2024-04-02T04:56:58.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[21528]: audit 2024-04-02T04:56:56.970863+0000 mon.c (mon.2) 969 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:58.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[21528]: audit 2024-04-02T04:56:57.116612+0000 mon.c (mon.2) 970 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[28904]: audit 2024-04-02T04:56:56.959608+0000 mon.a (mon.0) 2994 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:56:58.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[28904]: cluster 2024-04-02T04:56:56.967439+0000 mon.a (mon.0) 2995 : cluster 0 osdmap e551: 8 total, 8 up, 6 in 2024-04-02T04:56:58.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[28904]: audit 2024-04-02T04:56:56.970863+0000 mon.c (mon.2) 969 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:56:58.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:57 smithi069 bash[28904]: audit 2024-04-02T04:56:57.116612+0000 mon.c (mon.2) 970 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:58.302 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:58.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:58.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 -- 172.21.15.69:0/805901831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f4107620 msgr2=0x7f78f410f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 --2- 172.21.15.69:0/805901831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f4107620 0x7f78f410f1a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009fc0 tx=0x7f78ec00b330 comp rx=0 tx=0).stop 2024-04-02T04:56:58.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 -- 172.21.15.69:0/805901831 shutdown_connections 2024-04-02T04:56:58.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 --2- 172.21.15.69:0/805901831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f4107620 0x7f78f410f1a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 --2- 172.21.15.69:0/805901831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f406a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 --2- 172.21.15.69:0/805901831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f406ab80 0x7f78f4106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 -- 172.21.15.69:0/805901831 >> 172.21.15.69:0/805901831 conn(0x7f78f40771e0 msgr2=0x7f78f40775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 -- 172.21.15.69:0/805901831 shutdown_connections 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 -- 172.21.15.69:0/805901831 wait complete. 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 librados: starting msgr at 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 librados: starting objecter 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 Processor -- start 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 -- start start 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 librados: setting wanted keys 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.491+0000 7f78fa549700 1 librados: calling monclient init 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f406ab80 0x7f78f4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f4107620 0x7f78f419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78f4111ae0 con 0x7f78f406ab80 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78f4111960 con 0x7f78f4107620 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f78f4111c60 con 0x7f78f406a1d0 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42848/0 (socket says 172.21.15.69:42848) 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f406ab80 0x7f78f4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 -- 172.21.15.69:0/2441023811 learned_addr learned my addr 172.21.15.69:0/2441023811 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f4107620 0x7f78f419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f4107620 msgr2=0x7f78f419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f4107620 0x7f78f419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f406ab80 msgr2=0x7f78f4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f406ab80 0x7f78f4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f419d010 con 0x7f78f406a1d0 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78f406ab80 0x7f78f4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f4107620 0x7f78f419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:56:58.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f4198000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f78e400ec10 tx=0x7f78e400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 10 client.?.objecter ms_handle_connect 0x7f78f406a1d0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e400cd40 con 0x7f78f406a1d0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f78e40044a0 con 0x7f78f406a1d0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f419d300 con 0x7f78f406a1d0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4010430 con 0x7f78f406a1d0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 10 client.26681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f41a4b80 con 0x7f78f406a1d0 2024-04-02T04:56:58.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 librados: init done 2024-04-02T04:56:58.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f78e0005130 con 0x7f78f406a1d0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f78e4004000 con 0x7f78f406a1d0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78dc06f5e0 0x7f78dc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(552..552 src has 74..552) ==== 10584+0+0 (secure 0 0 0) 0x7f78e408f050 con 0x7f78f406a1d0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 osd_map(552..552 src has 74..552) 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 3 client.26681.objecter handle_osd_map got epochs [552,552] > 0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 3 client.26681.objecter handle_osd_map decoding full epoch 552 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78dc06f5e0 0x7f78dc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 20 client.26681.objecter dump_active .. 0 homeless 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v552) ==== 54+0+16 (secure 0 0 0) 0x7f78e4091670 con 0x7f78f406a1d0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78dc06f5e0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78dc06f5e0 0x7f78dc071aa0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f78f4107280 tx=0x7f78e800a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 10 client.26681.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 1 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f78e0004760 0x7f78e0004ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 1 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 0 1 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 4 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 10 client.26681.objecter _send_command 1 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.495+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78e0008ba0 con 0x7f78e0004760 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f78e0004760 0x7f78e0004ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e0004760 2024-04-02T04:56:58.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f78e0004760 0x7f78e0004ba0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e0008ba0 con 0x7f78e0004760 2024-04-02T04:56:58.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 1: 0 ) 2024-04-02T04:56:58.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f17fa700 10 client.26681.objecter _finish_command 1 = system:0 2024-04-02T04:56:58.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 4 2024-04-02T04:56:58.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.499+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 0 1 2024-04-02T04:56:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 10 client.26681.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 2 2024-04-02T04:56:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 2 2024-04-02T04:56:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0005410 osd=0 3 2024-04-02T04:56:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 0 2 2024-04-02T04:56:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 4 2024-04-02T04:56:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 10 client.26681.objecter _send_command 2 2024-04-02T04:56:58.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.515+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e00042d0 con 0x7f78e0004760 2024-04-02T04:56:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e00042d0 con 0x7f78e0004760 2024-04-02T04:56:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 10 client.26681.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 4 2024-04-02T04:56:58.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 0 2 2024-04-02T04:56:58.518 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:56:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 10 client.26681.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 3 2024-04-02T04:56:58.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f78e0002780 0x7f78e0002c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 1 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 4 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 10 client.26681.objecter _send_command 3 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f78e0003140 con 0x7f78e0002780 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f78e0002780 0x7f78e0002c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f78e0002780 0x7f78e0002c00 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e0002780 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e0003140 con 0x7f78e0002780 2024-04-02T04:56:58.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 3: 0 ) 2024-04-02T04:56:58.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 10 client.26681.objecter _finish_command 3 = system:0 2024-04-02T04:56:58.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 4 2024-04-02T04:56:58.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.519+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 1 3 2024-04-02T04:56:58.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 10 client.26681.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 4 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 4 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0004560 osd=1 3 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 1 4 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 4 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 10 client.26681.objecter _send_command 4 2024-04-02T04:56:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e0003b20 con 0x7f78e0002780 2024-04-02T04:56:58.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e0003b20 con 0x7f78e0002780 2024-04-02T04:56:58.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f17fa700 10 client.26681.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 4 2024-04-02T04:56:58.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 1 4 2024-04-02T04:56:58.539 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 10 client.26681.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 5 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f78e000b270 0x7f78e000d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 5 2024-04-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 2 5 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 4 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 10 client.26681.objecter _send_command 5 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f78e0010550 con 0x7f78e000b270 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f78e000b270 0x7f78e000d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e000b270 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.539+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f78e000b270 0x7f78e000d660 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.543+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e0010550 con 0x7f78e000b270 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.543+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 5: 0 ) 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.543+0000 7f78f17fa700 10 client.26681.objecter _finish_command 5 = system:0 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.543+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 4 2024-04-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.543+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 2 5 2024-04-02T04:56:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:56:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 10 client.26681.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 6 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 6 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0003d80 osd=2 3 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 2 6 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 4 2024-04-02T04:56:58.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 10 client.26681.objecter _send_command 6 2024-04-02T04:56:58.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.559+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e0003600 con 0x7f78e000b270 2024-04-02T04:56:58.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e0003600 con 0x7f78e000b270 2024-04-02T04:56:58.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 10 client.26681.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 4 2024-04-02T04:56:58.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 2 6 2024-04-02T04:56:58.561 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 10 client.26681.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 7 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f78e000ed20 0x7f78e000f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 7 2024-04-02T04:56:58.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 3 7 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 4 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 10 client.26681.objecter _send_command 7 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f78e000f710 con 0x7f78e000ed20 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f78e000ed20 0x7f78e000f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e000ed20 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f78e000ed20 0x7f78e000f1d0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e000f710 con 0x7f78e000ed20 2024-04-02T04:56:58.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 7: 0 ) 2024-04-02T04:56:58.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 10 client.26681.objecter _finish_command 7 = system:0 2024-04-02T04:56:58.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 4 2024-04-02T04:56:58.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.563+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 3 7 2024-04-02T04:56:58.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 10 client.26681.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 8 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 8 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e00038c0 osd=3 3 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 3 8 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 4 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 10 client.26681.objecter _send_command 8 2024-04-02T04:56:58.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e000f710 con 0x7f78e000ed20 2024-04-02T04:56:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e000f710 con 0x7f78e000ed20 2024-04-02T04:56:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f17fa700 10 client.26681.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 4 2024-04-02T04:56:58.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 3 8 2024-04-02T04:56:58.583 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:56:58.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 10 client.26681.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 9 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f78e0012740 0x7f78e0014b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 9 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 4 9 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 4 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 10 client.26681.objecter _send_command 9 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f78e000fe80 con 0x7f78e0012740 2024-04-02T04:56:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f78e0012740 0x7f78e0014b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e0012740 2024-04-02T04:56:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.583+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f78e0012740 0x7f78e0014b30 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.587+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e000fe80 con 0x7f78e0012740 2024-04-02T04:56:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.587+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 9: 0 ) 2024-04-02T04:56:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.587+0000 7f78f17fa700 10 client.26681.objecter _finish_command 9 = system:0 2024-04-02T04:56:58.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.587+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 4 2024-04-02T04:56:58.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.587+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 4 9 2024-04-02T04:56:58.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 10 client.26681.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 10 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 10 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e000f710 osd=4 3 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 4 10 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 4 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 10 client.26681.objecter _send_command 10 2024-04-02T04:56:58.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.607+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e000fee0 con 0x7f78e0012740 2024-04-02T04:56:58.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e000fee0 con 0x7f78e0012740 2024-04-02T04:56:58.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78f17fa700 10 client.26681.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 4 2024-04-02T04:56:58.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 4 10 2024-04-02T04:56:58.611 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 10 client.26681.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 11 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f78e00155e0 0x7f78e0015a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 11 2024-04-02T04:56:58.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 5 11 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 4 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 10 client.26681.objecter _send_command 11 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f78e001efe0 con 0x7f78e00155e0 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.611+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f78e00155e0 0x7f78e0015a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e00155e0 2024-04-02T04:56:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f37fe700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f78e00155e0 0x7f78e0015a20 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e001efe0 con 0x7f78e00155e0 2024-04-02T04:56:58.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 11: 0 ) 2024-04-02T04:56:58.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f17fa700 10 client.26681.objecter _finish_command 11 = system:0 2024-04-02T04:56:58.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 4 2024-04-02T04:56:58.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.615+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 5 11 2024-04-02T04:56:58.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 10 client.26681.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 12 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 12 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e000fee0 osd=5 3 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 5 12 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 4 2024-04-02T04:56:58.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 10 client.26681.objecter _send_command 12 2024-04-02T04:56:58.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.651+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e0016070 con 0x7f78e00155e0 2024-04-02T04:56:58.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e0016070 con 0x7f78e00155e0 2024-04-02T04:56:58.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78f17fa700 10 client.26681.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 4 2024-04-02T04:56:58.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 5 12 2024-04-02T04:56:58.656 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:56:58.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 10 client.26681.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 13 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e0016790 0x7f78e0016bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 13 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 6 13 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 4 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 10 client.26681.objecter _send_command 13 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.655+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f78e00172b0 con 0x7f78e0016790 2024-04-02T04:56:58.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e0016790 0x7f78e0016bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f8ae6700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e0016790 0x7f78e0016bd0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e0016790 2024-04-02T04:56:58.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e00172b0 con 0x7f78e0016790 2024-04-02T04:56:58.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 13: 0 ) 2024-04-02T04:56:58.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f17fa700 10 client.26681.objecter _finish_command 13 = system:0 2024-04-02T04:56:58.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 4 2024-04-02T04:56:58.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.659+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 6 13 2024-04-02T04:56:58.672 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 10 client.26681.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 14 2024-04-02T04:56:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 14 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e0016070 osd=6 3 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 6 14 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 4 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 10 client.26681.objecter _send_command 14 2024-04-02T04:56:58.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.679+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e00172b0 con 0x7f78e0016790 2024-04-02T04:56:58.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.683+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e00172b0 con 0x7f78e0016790 2024-04-02T04:56:58.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.683+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.683+0000 7f78f17fa700 10 client.26681.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.683+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 4 2024-04-02T04:56:58.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.683+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 6 14 2024-04-02T04:56:58.684 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 10 client.26681.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 15 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f78e0017420 0x7f78e0022430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 15 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 7 15 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 4 2024-04-02T04:56:58.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 10 client.26681.objecter _send_command 15 2024-04-02T04:56:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f78e00254b0 con 0x7f78e0017420 2024-04-02T04:56:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f78e0017420 0x7f78e0022430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f17fa700 10 client.26681.objecter ms_handle_connect 0x7f78e0017420 2024-04-02T04:56:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f3fff700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f78e0017420 0x7f78e0022430 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78e00254b0 con 0x7f78e0017420 2024-04-02T04:56:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 15: 0 ) 2024-04-02T04:56:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f17fa700 10 client.26681.objecter _finish_command 15 = system:0 2024-04-02T04:56:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 4 2024-04-02T04:56:58.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.687+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 7 15 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 10 client.26681.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign -1 16 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 20 client.26681.objecter _get_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 15 client.26681.objecter _session_command_op_remove -1 16 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 20 client.26681.objecter get_session s=0x7f78e00172b0 osd=7 3 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 15 client.26681.objecter _session_command_op_assign 7 16 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 4 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 10 client.26681.objecter _send_command 16 2024-04-02T04:56:58.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.707+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f78e0025510 con 0x7f78e0017420 2024-04-02T04:56:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f17fa700 1 -- 172.21.15.69:0/2441023811 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f78e0025510 con 0x7f78e0017420 2024-04-02T04:56:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f17fa700 10 client.26681.objecter ms_dispatch 0x7f78f4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:56:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f17fa700 10 client.26681.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:56:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f17fa700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 4 2024-04-02T04:56:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f17fa700 15 client.26681.objecter _session_command_op_remove 7 16 2024-04-02T04:56:58.710 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 librados: watch_flush enter 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 librados: watch_flush exit 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.0 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f78e0004760 msgr2=0x7f78e0004ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f78e0004760 0x7f78e0004ba0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0005410 osd=0 1 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.1 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f78e0002780 msgr2=0x7f78e0002c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f78e0002780 0x7f78e0002c00 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0004560 osd=1 1 2024-04-02T04:56:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.2 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f78e000b270 msgr2=0x7f78e000d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f78e000b270 0x7f78e000d660 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0003d80 osd=2 1 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.3 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f78e000ed20 msgr2=0x7f78e000f1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f78e000ed20 0x7f78e000f1d0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00038c0 osd=3 1 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.4 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f78e0012740 msgr2=0x7f78e0014b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f78e0012740 0x7f78e0014b30 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000f710 osd=4 1 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.5 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f78e00155e0 msgr2=0x7f78e0015a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f78e00155e0 0x7f78e0015a20 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f8ae6700 1 -- 172.21.15.69:0/2441023811 reap_dead start 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e000fee0 osd=5 1 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.6 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e0016790 msgr2=0x7f78e0016bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f78e0016790 0x7f78e0016bd0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f8ae6700 1 -- 172.21.15.69:0/2441023811 reap_dead start 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e0016070 osd=6 1 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter close_session for osd.7 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f78e0017420 msgr2=0x7f78e0022430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f78e0017420 0x7f78e0022430 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter put_session s=0x7f78e00172b0 osd=7 1 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 20 client.26681.objecter shutdown clearing up homeless session... 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 10 client.26681.objecter successfully canceled tick 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f8ae6700 1 -- 172.21.15.69:0/2441023811 reap_dead start 2024-04-02T04:56:58.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78f8ae6700 1 -- 172.21.15.69:0/2441023811 reap_dead start 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78dc06f5e0 msgr2=0x7f78dc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78dc06f5e0 0x7f78dc071aa0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f78f4107280 tx=0x7f78e800a380 comp rx=0 tx=0).stop 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 msgr2=0x7f78f4198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f4198000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f78e400ec10 tx=0x7f78e400ef20 comp rx=0 tx=0).stop 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 shutdown_connections 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78f406a1d0 0x7f78f4198000 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 --2- 172.21.15.69:0/2441023811 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f78dc06f5e0 0x7f78dc071aa0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 >> 172.21.15.69:0/2441023811 conn(0x7f78f40771e0 msgr2=0x7f78f40ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 shutdown_connections 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 -- 172.21.15.69:0/2441023811 wait complete. 2024-04-02T04:56:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.711+0000 7f78fa549700 1 librados: shutdown 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:56:58.715 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:56:58.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 -- 172.21.15.69:0/4214591469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1012a0 msgr2=0x7f905c10d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 --2- 172.21.15.69:0/4214591469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1012a0 0x7f905c10d7b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9054009f90 tx=0x7f905400b330 comp rx=0 tx=0).stop 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 -- 172.21.15.69:0/4214591469 shutdown_connections 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 --2- 172.21.15.69:0/4214591469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1012a0 0x7f905c10d7b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 --2- 172.21.15.69:0/4214591469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1068e0 0x7f905c106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 --2- 172.21.15.69:0/4214591469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 0x7f905c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 -- 172.21.15.69:0/4214591469 >> 172.21.15.69:0/4214591469 conn(0x7f905c077740 msgr2=0x7f905c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 -- 172.21.15.69:0/4214591469 shutdown_connections 2024-04-02T04:56:58.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 -- 172.21.15.69:0/4214591469 wait complete. 2024-04-02T04:56:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 librados: starting msgr at 2024-04-02T04:56:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.735+0000 7f9061b19700 1 librados: starting objecter 2024-04-02T04:56:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 Processor -- start 2024-04-02T04:56:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- start start 2024-04-02T04:56:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 librados: setting wanted keys 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 librados: calling monclient init 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 0x7f905c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1012a0 0x7f905c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 0x7f905c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10f910 con 0x7f905c1068e0 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10f790 con 0x7f905c1012a0 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f905c10fa90 con 0x7f905c1008e0 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 0x7f905c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 0x7f905c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44912/0 (socket says 172.21.15.69:44912) 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905bfff700 1 -- 172.21.15.69:0/1641747466 learned_addr learned my addr 172.21.15.69:0/1641747466 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 0x7f905c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905affd700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1012a0 0x7f905c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 -- 172.21.15.69:0/1641747466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1012a0 msgr2=0x7f905c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1012a0 0x7f905c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 -- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 msgr2=0x7f905c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 0x7f905c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 -- 172.21.15.69:0/1641747466 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c19cf20 con 0x7f905c1008e0 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905bfff700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 0x7f905c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905affd700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1012a0 0x7f905c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f905b7fe700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 0x7f905c197f10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f904c00c970 tx=0x7f904c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 10 client.?.objecter ms_handle_connect 0x7f905c1008e0 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c014070 con 0x7f905c1008e0 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f904c004440 con 0x7f905c1008e0 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c19d210 con 0x7f905c1008e0 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 10 client.26687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:56:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c0039f0 con 0x7f905c1008e0 2024-04-02T04:56:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a4a90 con 0x7f905c1008e0 2024-04-02T04:56:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 librados: init done 2024-04-02T04:56:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f905c04ff00 con 0x7f905c1008e0 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f904c017070 con 0x7f905c1008e0 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f904406f5e0 0x7f9044071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(552..552 src has 74..552) ==== 10584+0+0 (secure 0 0 0) 0x7f904c08e570 con 0x7f905c1008e0 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 10 client.26687.objecter ms_dispatch 0x7f905c001550 osd_map(552..552 src has 74..552) 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 3 client.26687.objecter handle_osd_map got epochs [552,552] > 0 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.739+0000 7f9058ff9700 3 client.26687.objecter handle_osd_map decoding full epoch 552 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.743+0000 7f905affd700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f904406f5e0 0x7f9044071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.743+0000 7f9058ff9700 20 client.26687.objecter dump_active .. 0 homeless 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.743+0000 7f9058ff9700 10 client.26687.objecter ms_handle_connect 0x7f904406f5e0 2024-04-02T04:56:58.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.743+0000 7f905affd700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f904406f5e0 0x7f9044071aa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f905c1050d0 tx=0x7f905000a420 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:56:58.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.747+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f904c01e030 con 0x7f905c1008e0 2024-04-02T04:56:58.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:58.798 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:58.799 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:56:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:56:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:58.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:58.928 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:58.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:58.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:58.971+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f905c02db30 con 0x7f905c1008e0 2024-04-02T04:56:59.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:59.057 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:56:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:56:59.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:59.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[21528]: cluster 2024-04-02T04:56:57.968733+0000 mon.a (mon.0) 2996 : cluster 0 osdmap e552: 8 total, 8 up, 6 in 2024-04-02T04:56:59.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[21528]: audit 2024-04-02T04:56:58.118087+0000 mon.c (mon.2) 971 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:59.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[21528]: cluster 2024-04-02T04:56:58.183628+0000 mgr.y (mgr.14538) 519 : cluster 0 pgmap v818: 313 pgs: 1 active+remapped+backfill_toofull, 6 unknown, 306 active+clean; 8.4 MiB data, 448 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 2/1272 objects misplaced (0.157%) 2024-04-02T04:56:59.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[21528]: audit 2024-04-02T04:56:58.498774+0000 mon.c (mon.2) 972 : audit 0 from='client.? 172.21.15.69:0/2441023811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:59.171 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[28904]: cluster 2024-04-02T04:56:57.968733+0000 mon.a (mon.0) 2996 : cluster 0 osdmap e552: 8 total, 8 up, 6 in 2024-04-02T04:56:59.171 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[28904]: audit 2024-04-02T04:56:58.118087+0000 mon.c (mon.2) 971 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:59.171 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[28904]: cluster 2024-04-02T04:56:58.183628+0000 mgr.y (mgr.14538) 519 : cluster 0 pgmap v818: 313 pgs: 1 active+remapped+backfill_toofull, 6 unknown, 306 active+clean; 8.4 MiB data, 448 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 2/1272 objects misplaced (0.157%) 2024-04-02T04:56:59.171 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:58 smithi069 bash[28904]: audit 2024-04-02T04:56:58.498774+0000 mon.c (mon.2) 972 : audit 0 from='client.? 172.21.15.69:0/2441023811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:58 smithi098 bash[23790]: cluster 2024-04-02T04:56:57.968733+0000 mon.a (mon.0) 2996 : cluster 0 osdmap e552: 8 total, 8 up, 6 in 2024-04-02T04:56:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:58 smithi098 bash[23790]: audit 2024-04-02T04:56:58.118087+0000 mon.c (mon.2) 971 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:56:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:58 smithi098 bash[23790]: cluster 2024-04-02T04:56:58.183628+0000 mgr.y (mgr.14538) 519 : cluster 0 pgmap v818: 313 pgs: 1 active+remapped+backfill_toofull, 6 unknown, 306 active+clean; 8.4 MiB data, 448 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 2/1272 objects misplaced (0.157%) 2024-04-02T04:56:59.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:58 smithi098 bash[23790]: audit 2024-04-02T04:56:58.498774+0000 mon.c (mon.2) 972 : audit 0 from='client.? 172.21.15.69:0/2441023811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:56:59.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:59.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:59.441 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:56:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:56:59.545 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:59.573 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:56:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:56:59.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:56:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:56:59.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:59.834 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:56:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:56:59.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:56:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:56:59.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:56:59.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.991+0000 7f9058ff9700 1 -- 172.21.15.69:0/1641747466 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v554) ==== 97+0+0 (secure 0 0 0) 0x7f904c05af20 con 0x7f905c1008e0 2024-04-02T04:56:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 10 librados: watch_flush enter 2024-04-02T04:56:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 10 librados: watch_flush exit 2024-04-02T04:56:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 20 client.26687.objecter shutdown clearing up homeless session... 2024-04-02T04:56:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 10 client.26687.objecter successfully canceled tick 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f904406f5e0 msgr2=0x7f9044071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f904406f5e0 0x7f9044071aa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f905c1050d0 tx=0x7f905000a420 comp rx=0 tx=0).stop 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 msgr2=0x7f905c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 0x7f905c197f10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f904c00c970 tx=0x7f904c00cc80 comp rx=0 tx=0).stop 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 shutdown_connections 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f904406f5e0 0x7f9044071aa0 secure :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f905c1050d0 tx=0x7f905000a420 comp rx=0 tx=0).stop 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f905c1068e0 0x7f905c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f905c1008e0 0x7f905c197f10 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f904c00c970 tx=0x7f904c00cc80 comp rx=0 tx=0).stop 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 --2- 172.21.15.69:0/1641747466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f905c1012a0 0x7f905c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.995+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 >> 172.21.15.69:0/1641747466 conn(0x7f905c077740 msgr2=0x7f905c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:56:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.999+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 shutdown_connections 2024-04-02T04:56:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.999+0000 7f9061b19700 1 -- 172.21.15.69:0/1641747466 wait complete. 2024-04-02T04:56:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:56:59.999+0000 7f9061b19700 1 librados: shutdown 2024-04-02T04:56:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:57:00.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:00.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[21528]: audit 2024-04-02T04:56:58.977311+0000 mon.c (mon.2) 973 : audit 1 from='client.? 172.21.15.69:0/1641747466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[21528]: audit 2024-04-02T04:56:58.977655+0000 mon.a (mon.0) 2997 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[21528]: cluster 2024-04-02T04:56:58.983295+0000 mon.a (mon.0) 2998 : cluster 0 osdmap e553: 8 total, 8 up, 6 in 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[21528]: audit 2024-04-02T04:56:59.019543+0000 mon.c (mon.2) 974 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "302.7"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[21528]: audit 2024-04-02T04:56:59.019749+0000 mgr.y (mgr.14538) 520 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "302.7"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[21528]: audit 2024-04-02T04:56:59.119341+0000 mon.c (mon.2) 975 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[28904]: audit 2024-04-02T04:56:58.977311+0000 mon.c (mon.2) 973 : audit 1 from='client.? 172.21.15.69:0/1641747466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[28904]: audit 2024-04-02T04:56:58.977655+0000 mon.a (mon.0) 2997 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:00.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[28904]: cluster 2024-04-02T04:56:58.983295+0000 mon.a (mon.0) 2998 : cluster 0 osdmap e553: 8 total, 8 up, 6 in 2024-04-02T04:57:00.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[28904]: audit 2024-04-02T04:56:59.019543+0000 mon.c (mon.2) 974 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "302.7"}]: dispatch 2024-04-02T04:57:00.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[28904]: audit 2024-04-02T04:56:59.019749+0000 mgr.y (mgr.14538) 520 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "302.7"}]: dispatch 2024-04-02T04:57:00.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:56:59 smithi069 bash[28904]: audit 2024-04-02T04:56:59.119341+0000 mon.c (mon.2) 975 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:00.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:00.194 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:00.220 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:00.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:00.349 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:00.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:59 smithi098 bash[23790]: audit 2024-04-02T04:56:58.977311+0000 mon.c (mon.2) 973 : audit 1 from='client.? 172.21.15.69:0/1641747466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:59 smithi098 bash[23790]: audit 2024-04-02T04:56:58.977655+0000 mon.a (mon.0) 2997 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:00.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:59 smithi098 bash[23790]: cluster 2024-04-02T04:56:58.983295+0000 mon.a (mon.0) 2998 : cluster 0 osdmap e553: 8 total, 8 up, 6 in 2024-04-02T04:57:00.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:59 smithi098 bash[23790]: audit 2024-04-02T04:56:59.019543+0000 mon.c (mon.2) 974 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "302.7"}]: dispatch 2024-04-02T04:57:00.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:59 smithi098 bash[23790]: audit 2024-04-02T04:56:59.019749+0000 mgr.y (mgr.14538) 520 : audit 0 from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "302.7"}]: dispatch 2024-04-02T04:57:00.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:56:59 smithi098 bash[23790]: audit 2024-04-02T04:56:59.119341+0000 mon.c (mon.2) 975 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:00.475 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:00.577 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:00.606 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:00.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:00.733 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:00.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:00.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:00.995 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:00.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:01.098 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:01.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: cluster 2024-04-02T04:56:59.781249+0000 osd.1 (osd.1) 303 : cluster 0 302.7 deep-scrub starts 2024-04-02T04:57:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: cluster 2024-04-02T04:56:59.786003+0000 osd.1 (osd.1) 304 : cluster 0 302.7 deep-scrub ok 2024-04-02T04:57:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: cluster 2024-04-02T04:56:59.978714+0000 mon.a (mon.0) 2999 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:01.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: audit 2024-04-02T04:56:59.980655+0000 mon.a (mon.0) 3000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:57:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: cluster 2024-04-02T04:56:59.984058+0000 mon.a (mon.0) 3001 : cluster 0 osdmap e554: 8 total, 8 up, 6 in 2024-04-02T04:57:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: audit 2024-04-02T04:57:00.120969+0000 mon.c (mon.2) 976 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:01.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:01 smithi098 bash[23790]: cluster 2024-04-02T04:57:00.184326+0000 mgr.y (mgr.14538) 521 : cluster 0 pgmap v821: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 2/1275 objects misplaced (0.157%) 2024-04-02T04:57:01.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:01.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:01.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: cluster 2024-04-02T04:56:59.781249+0000 osd.1 (osd.1) 303 : cluster 0 302.7 deep-scrub starts 2024-04-02T04:57:01.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: cluster 2024-04-02T04:56:59.786003+0000 osd.1 (osd.1) 304 : cluster 0 302.7 deep-scrub ok 2024-04-02T04:57:01.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: cluster 2024-04-02T04:56:59.978714+0000 mon.a (mon.0) 2999 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:01.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: audit 2024-04-02T04:56:59.980655+0000 mon.a (mon.0) 3000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:57:01.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: cluster 2024-04-02T04:56:59.984058+0000 mon.a (mon.0) 3001 : cluster 0 osdmap e554: 8 total, 8 up, 6 in 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: audit 2024-04-02T04:57:00.120969+0000 mon.c (mon.2) 976 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[21528]: cluster 2024-04-02T04:57:00.184326+0000 mgr.y (mgr.14538) 521 : cluster 0 pgmap v821: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 2/1275 objects misplaced (0.157%) 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: cluster 2024-04-02T04:56:59.781249+0000 osd.1 (osd.1) 303 : cluster 0 302.7 deep-scrub starts 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: cluster 2024-04-02T04:56:59.786003+0000 osd.1 (osd.1) 304 : cluster 0 302.7 deep-scrub ok 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: cluster 2024-04-02T04:56:59.978714+0000 mon.a (mon.0) 2999 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: audit 2024-04-02T04:56:59.980655+0000 mon.a (mon.0) 3000 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: cluster 2024-04-02T04:56:59.984058+0000 mon.a (mon.0) 3001 : cluster 0 osdmap e554: 8 total, 8 up, 6 in 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: audit 2024-04-02T04:57:00.120969+0000 mon.c (mon.2) 976 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:01.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:01 smithi069 bash[28904]: cluster 2024-04-02T04:57:00.184326+0000 mgr.y (mgr.14538) 521 : cluster 0 pgmap v821: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 2/1275 objects misplaced (0.157%) 2024-04-02T04:57:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:01.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:01.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:01.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:01.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:01.651 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:01.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:01.782 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:01.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:01.913 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:01.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:02.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:02.027 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:02 smithi098 bash[23790]: audit 2024-04-02T04:57:01.122056+0000 mon.c (mon.2) 977 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:02.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:02.174 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:02.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:02.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:02 smithi069 bash[21528]: audit 2024-04-02T04:57:01.122056+0000 mon.c (mon.2) 977 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:02.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:02 smithi069 bash[28904]: audit 2024-04-02T04:57:01.122056+0000 mon.c (mon.2) 977 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:02.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:02.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:57:02.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:02.563 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:02.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:02.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:02.822 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:02.926 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:02.953 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:02.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:03.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:03.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:03.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:03.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:03 smithi069 bash[21528]: audit 2024-04-02T04:57:02.123357+0000 mon.c (mon.2) 978 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:03.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:03 smithi069 bash[21528]: cluster 2024-04-02T04:57:02.184856+0000 mgr.y (mgr.14538) 522 : cluster 0 pgmap v822: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 2/1275 objects misplaced (0.157%) 2024-04-02T04:57:03.195 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:03 smithi069 bash[28904]: audit 2024-04-02T04:57:02.123357+0000 mon.c (mon.2) 978 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:03.195 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:03 smithi069 bash[28904]: cluster 2024-04-02T04:57:02.184856+0000 mgr.y (mgr.14538) 522 : cluster 0 pgmap v822: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 2/1275 objects misplaced (0.157%) 2024-04-02T04:57:03.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:03.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:03.334 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:03.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- 172.21.15.69:0/727959265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c108390 msgr2=0x7f4d1c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:03.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- 172.21.15.69:0/727959265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c108390 0x7f4d1c108770 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009990 tx=0x7f4d18009ca0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- 172.21.15.69:0/727959265 shutdown_connections 2024-04-02T04:57:03.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- 172.21.15.69:0/727959265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102ce0 0x7f4d1c10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- 172.21.15.69:0/727959265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c108390 0x7f4d1c108770 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- 172.21.15.69:0/727959265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d1c102320 0x7f4d1c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- 172.21.15.69:0/727959265 >> 172.21.15.69:0/727959265 conn(0x7f4d1c079310 msgr2=0x7f4d1c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- 172.21.15.69:0/727959265 shutdown_connections 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- 172.21.15.69:0/727959265 wait complete. 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 librados: starting msgr at 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 librados: starting objecter 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 Processor -- start 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- start start 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 librados: setting wanted keys 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 librados: calling monclient init 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 0x7f4d1c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:03.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c102ce0 0x7f4d1c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d1c108390 0x7f4d1c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1c111aa0 con 0x7f4d1c102ce0 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1c111920 con 0x7f4d1c108390 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d24551700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4d1c111c20 con 0x7f4d1c102320 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 0x7f4d1c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d21aec700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c102ce0 0x7f4d1c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 0x7f4d1c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:42896/0 (socket says 172.21.15.69:42896) 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 -- 172.21.15.69:0/1376761955 learned_addr learned my addr 172.21.15.69:0/1376761955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 -- 172.21.15.69:0/1376761955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d1c108390 msgr2=0x7f4d1c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d1c108390 0x7f4d1c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 -- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c102ce0 msgr2=0x7f4d1c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c102ce0 0x7f4d1c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 -- 172.21.15.69:0/1376761955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1c19cfd0 con 0x7f4d1c102320 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d21aec700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c102ce0 0x7f4d1c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d222ed700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 0x7f4d1c197fc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4d18000c00 tx=0x7f4d180047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d137fe700 10 client.?.objecter ms_handle_connect 0x7f4d1c102320 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d137fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.355+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18027070 con 0x7f4d1c102320 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4d18022470 con 0x7f4d1c102320 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1c19d260 con 0x7f4d1c102320 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18017650 con 0x7f4d1c102320 2024-04-02T04:57:03.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d24551700 10 client.26693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:03.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1c1a4b40 con 0x7f4d1c102320 2024-04-02T04:57:03.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d24551700 1 librados: init done 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d1c1039b0 con 0x7f4d1c102320 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4d1802a070 con 0x7f4d1c102320 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d0806f5e0 0x7f4d08071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(554..554 src has 74..554) ==== 10584+0+0 (secure 0 0 0) 0x7f4d1808fb90 con 0x7f4d1c102320 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 10 client.26693.objecter ms_dispatch 0x7f4d1c001550 osd_map(554..554 src has 74..554) 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 3 client.26693.objecter handle_osd_map got epochs [554,554] > 0 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 3 client.26693.objecter handle_osd_map decoding full epoch 554 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d21aec700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d0806f5e0 0x7f4d08071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 20 client.26693.objecter dump_active .. 0 homeless 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d137fe700 10 client.26693.objecter ms_handle_connect 0x7f4d0806f5e0 2024-04-02T04:57:03.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.359+0000 7f4d21aec700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d0806f5e0 0x7f4d08071aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c079ad0 tx=0x7f4d0c00a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:03.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.363+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4d18013070 con 0x7f4d1c102320 2024-04-02T04:57:03.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:03 smithi098 bash[23790]: audit 2024-04-02T04:57:02.123357+0000 mon.c (mon.2) 978 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:03.392 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:03 smithi098 bash[23790]: cluster 2024-04-02T04:57:02.184856+0000 mgr.y (mgr.14538) 522 : cluster 0 pgmap v822: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 2/1275 objects misplaced (0.157%) 2024-04-02T04:57:03.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:03.465 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:03.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:03.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.579+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) -- 0x7f4d1c1075d0 con 0x7f4d0806f5e0 2024-04-02T04:57:03.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.583+0000 7f4d137fe700 1 -- 172.21.15.69:0/1376761955 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 2 no change) ==== 17+0+364 (secure 0 0 0) 0x7f4d1c1075d0 con 0x7f4d0806f5e0 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout:moved 48 / 939 (5.11182%) 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout:avg 156.5 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout:stddev 11.6154 -> 9.86154 (expected baseline 11.42) 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout:min osd.7 with 140 -> 151 pgs (0.894569 -> 0.964856 * mean) 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout:max osd.1 with 169 -> 147 pgs (1.07987 -> 0.939297 * mean) 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:57:03.583 INFO:teuthology.orchestra.run.smithi069.stdout:oload 110 2024-04-02T04:57:03.584 INFO:teuthology.orchestra.run.smithi069.stdout:max_change 3 2024-04-02T04:57:03.584 INFO:teuthology.orchestra.run.smithi069.stdout:max_change_osds 4 2024-04-02T04:57:03.584 INFO:teuthology.orchestra.run.smithi069.stdout:average_utilization 0.0008 2024-04-02T04:57:03.584 INFO:teuthology.orchestra.run.smithi069.stdout:overload_utilization 0.0009 2024-04-02T04:57:03.584 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 weight 1.0000 -> 0.8155 2024-04-02T04:57:03.584 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 weight 1.0000 -> 0.8588 2024-04-02T04:57:03.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 10 librados: watch_flush enter 2024-04-02T04:57:03.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 10 librados: watch_flush exit 2024-04-02T04:57:03.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 20 client.26693.objecter shutdown clearing up homeless session... 2024-04-02T04:57:03.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 10 client.26693.objecter successfully canceled tick 2024-04-02T04:57:03.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d0806f5e0 msgr2=0x7f4d08071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d0806f5e0 0x7f4d08071aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c079ad0 tx=0x7f4d0c00a380 comp rx=0 tx=0).stop 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 msgr2=0x7f4d1c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 0x7f4d1c197fc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4d18000c00 tx=0x7f4d180047b0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 shutdown_connections 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4d0806f5e0 0x7f4d08071aa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d1c108390 0x7f4d1c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d1c102320 0x7f4d1c197fc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 --2- 172.21.15.69:0/1376761955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d1c102ce0 0x7f4d1c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 >> 172.21.15.69:0/1376761955 conn(0x7f4d1c079310 msgr2=0x7f4d1c105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 shutdown_connections 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 -- 172.21.15.69:0/1376761955 wait complete. 2024-04-02T04:57:03.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:03.587+0000 7f4d24551700 1 librados: shutdown 2024-04-02T04:57:03.587 INFO:teuthology.orchestra.run.smithi069.stderr:no change 2024-04-02T04:57:03.595 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:03.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:03.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:57:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:57:03.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:57:03.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:03.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:03.950 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:03 smithi069 bash[21782]: debug 2024-04-02T04:57:03.583+0000 7f42d9757700 -1 mgr.server reply reply (2) No such file or directory no change 2024-04-02T04:57:03.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:03.978 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:03.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:04.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:04.106 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:04.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:04.237 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[21528]: audit 2024-04-02T04:57:03.125062+0000 mon.c (mon.2) 979 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[21528]: audit 2024-04-02T04:57:03.294686+0000 mon.b (mon.1) 270 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[21528]: audit 2024-04-02T04:57:03.394487+0000 mgr.y (mgr.14538) 523 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[21528]: audit 2024-04-02T04:57:03.580701+0000 mgr.y (mgr.14538) 524 : audit 0 from='client.26693 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[28904]: audit 2024-04-02T04:57:03.125062+0000 mon.c (mon.2) 979 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[28904]: audit 2024-04-02T04:57:03.294686+0000 mon.b (mon.1) 270 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:04.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[28904]: audit 2024-04-02T04:57:03.394487+0000 mgr.y (mgr.14538) 523 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:04.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:04 smithi069 bash[28904]: audit 2024-04-02T04:57:03.580701+0000 mgr.y (mgr.14538) 524 : audit 0 from='client.26693 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:04.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:04.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:04 smithi098 bash[23790]: audit 2024-04-02T04:57:03.125062+0000 mon.c (mon.2) 979 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:04 smithi098 bash[23790]: audit 2024-04-02T04:57:03.294686+0000 mon.b (mon.1) 270 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:04 smithi098 bash[23790]: audit 2024-04-02T04:57:03.394487+0000 mgr.y (mgr.14538) 523 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:04.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:04 smithi098 bash[23790]: audit 2024-04-02T04:57:03.580701+0000 mgr.y (mgr.14538) 524 : audit 0 from='client.26693 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:04.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:04.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:04.620 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:04.647 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:04.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:04.779 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:04.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:04.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:05.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:05.143 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:05.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:05 smithi069 bash[28904]: audit 2024-04-02T04:57:04.126699+0000 mon.c (mon.2) 980 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:05.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:05 smithi069 bash[28904]: cluster 2024-04-02T04:57:04.187770+0000 mgr.y (mgr.14538) 525 : cluster 0 pgmap v823: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 987 B/s wr, 3 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:05.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:05 smithi069 bash[28904]: cluster 2024-04-02T04:57:04.958798+0000 mon.a (mon.0) 3002 : cluster 0 osdmap e555: 8 total, 8 up, 6 in 2024-04-02T04:57:05.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:05 smithi069 bash[21528]: audit 2024-04-02T04:57:04.126699+0000 mon.c (mon.2) 980 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:05.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:05 smithi069 bash[21528]: cluster 2024-04-02T04:57:04.187770+0000 mgr.y (mgr.14538) 525 : cluster 0 pgmap v823: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 987 B/s wr, 3 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:05.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:05 smithi069 bash[21528]: cluster 2024-04-02T04:57:04.958798+0000 mon.a (mon.0) 3002 : cluster 0 osdmap e555: 8 total, 8 up, 6 in 2024-04-02T04:57:05.169 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:05 smithi098 bash[23790]: audit 2024-04-02T04:57:04.126699+0000 mon.c (mon.2) 980 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:05 smithi098 bash[23790]: cluster 2024-04-02T04:57:04.187770+0000 mgr.y (mgr.14538) 525 : cluster 0 pgmap v823: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 987 B/s wr, 3 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:05 smithi098 bash[23790]: cluster 2024-04-02T04:57:04.958798+0000 mon.a (mon.0) 3002 : cluster 0 osdmap e555: 8 total, 8 up, 6 in 2024-04-02T04:57:05.271 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-02T04:57:05.271 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-04-02T04:57:05.271 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:05.294 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:05.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:05.421 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:05.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:05.543 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:05.645 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:05.672 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:05.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:05.802 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:05.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:05.933 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:05.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:06.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:06.059 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:06.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:06.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:06 smithi069 bash[21528]: audit 2024-04-02T04:57:05.128407+0000 mon.c (mon.2) 981 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:06.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:06 smithi069 bash[28904]: audit 2024-04-02T04:57:05.128407+0000 mon.c (mon.2) 981 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:06.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:06.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:06 smithi098 bash[23790]: audit 2024-04-02T04:57:05.128407+0000 mon.c (mon.2) 981 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:06.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:06.449 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:06.552 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:06.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:06.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:06.710 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:06.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:06.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:06.863 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:57:06.863 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:57:06.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:06.965 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:06.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:07.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.093 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:07.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:07 smithi069 bash[28904]: audit 2024-04-02T04:57:06.130157+0000 mon.c (mon.2) 982 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:07.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:07 smithi069 bash[28904]: cluster 2024-04-02T04:57:06.188698+0000 mgr.y (mgr.14538) 526 : cluster 0 pgmap v825: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 710 B/s rd, 284 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:07.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:07 smithi069 bash[21528]: audit 2024-04-02T04:57:06.130157+0000 mon.c (mon.2) 982 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:07.208 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:07 smithi069 bash[21528]: cluster 2024-04-02T04:57:06.188698+0000 mgr.y (mgr.14538) 526 : cluster 0 pgmap v825: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 710 B/s rd, 284 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:07.223 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:07.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.352 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:07.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:07.467 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:07 smithi098 bash[23790]: audit 2024-04-02T04:57:06.130157+0000 mon.c (mon.2) 982 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:07.467 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:07 smithi098 bash[23790]: cluster 2024-04-02T04:57:06.188698+0000 mgr.y (mgr.14538) 526 : cluster 0 pgmap v825: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 710 B/s rd, 284 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:07.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.613 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:07.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:07.742 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:07.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:07.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:07.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-02T04:57:07.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:07.999 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:08.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:08.130 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:08.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:08.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:08 smithi069 bash[21528]: audit 2024-04-02T04:57:07.131642+0000 mon.c (mon.2) 983 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:08.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:08 smithi069 bash[28904]: audit 2024-04-02T04:57:07.131642+0000 mon.c (mon.2) 983 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:08.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:08.355 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:08.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:08.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:08.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:08.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:08 smithi098 bash[23790]: audit 2024-04-02T04:57:07.131642+0000 mon.c (mon.2) 983 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:08.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:08.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:08.643 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:08.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:08.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:08.903 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:08.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:09.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:09.029 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:57:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:57:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:09.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:09.161 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout:api_misc: started 0x55c99fccdae0 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd5f0 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd740 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd890 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccdc10 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce100 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce230 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce7f0 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccdd60 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccde70 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccdfc0 2024-04-02T04:57:09.229 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccd140 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce340 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce490 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce5a0 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcce960 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccee50 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccef60 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccea90 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccebe0 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcced30 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf070 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf590 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf6e0 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccfca0 2024-04-02T04:57:09.230 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf180 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf2b0 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf400 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccfdd0 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf7f0 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccf940 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccfa90 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fccff00 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0420 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd05a0 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0b80 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0030 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0160 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0290 2024-04-02T04:57:09.231 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0cb0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd06d0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0800 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0930 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0a60 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd12b0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0e00 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd0f50 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1080 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd13e0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd18b0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd19e0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd14f0 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1640 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1190 2024-04-02T04:57:09.232 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1770 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1fc0 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1b10 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1c60 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd1db0 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2110 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd25e0 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2710 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2cf0 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2240 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2370 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd24a0 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2e20 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2840 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2990 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2aa0 2024-04-02T04:57:09.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2bd0 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3400 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd2f50 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd30a0 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd31d0 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3550 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3a20 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3ba0 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4180 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3680 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd37b0 2024-04-02T04:57:09.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd38e0 2024-04-02T04:57:09.264 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:09.289 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:09.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:09.404 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:09 smithi069 bash[28904]: audit 2024-04-02T04:57:08.132595+0000 mon.c (mon.2) 984 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:09.404 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:09 smithi069 bash[28904]: cluster 2024-04-02T04:57:08.191341+0000 mgr.y (mgr.14538) 527 : cluster 0 pgmap v826: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:09.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:09 smithi069 bash[21528]: audit 2024-04-02T04:57:08.132595+0000 mon.c (mon.2) 984 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:09.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:09 smithi069 bash[21528]: cluster 2024-04-02T04:57:08.191341+0000 mgr.y (mgr.14538) 527 : cluster 0 pgmap v826: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:09.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:09 smithi098 bash[23790]: audit 2024-04-02T04:57:08.132595+0000 mon.c (mon.2) 984 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:09.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:09 smithi098 bash[23790]: cluster 2024-04-02T04:57:08.191341+0000 mgr.y (mgr.14538) 527 : cluster 0 pgmap v826: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:09.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:09.547 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:09.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:09.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:09.809 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:09.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:09.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:10.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:10.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:10.186 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:10 smithi098 bash[23790]: audit 2024-04-02T04:57:09.133831+0000 mon.c (mon.2) 985 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:10.202 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:10.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:10.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:10 smithi069 bash[21528]: audit 2024-04-02T04:57:09.133831+0000 mon.c (mon.2) 985 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:10.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:10 smithi069 bash[28904]: audit 2024-04-02T04:57:09.133831+0000 mon.c (mon.2) 985 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:10.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:10.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:10.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:10.595 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:10.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:10.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:10.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:10.872 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:10.873 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:10.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:10.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:11.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:11.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:11 smithi098 bash[23790]: audit 2024-04-02T04:57:10.135452+0000 mon.c (mon.2) 986 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:11.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:11 smithi098 bash[23790]: cluster 2024-04-02T04:57:10.192712+0000 mgr.y (mgr.14538) 528 : cluster 0 pgmap v827: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:11.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:11 smithi069 bash[21528]: audit 2024-04-02T04:57:10.135452+0000 mon.c (mon.2) 986 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:11.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:11 smithi069 bash[21528]: cluster 2024-04-02T04:57:10.192712+0000 mgr.y (mgr.14538) 528 : cluster 0 pgmap v827: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:11 smithi069 bash[28904]: audit 2024-04-02T04:57:10.135452+0000 mon.c (mon.2) 986 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:11.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:11 smithi069 bash[28904]: cluster 2024-04-02T04:57:10.192712+0000 mgr.y (mgr.14538) 528 : cluster 0 pgmap v827: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:11.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:11.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:11.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:11.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:11.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:11.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:11.992 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:12.022 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:12.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:12.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:12.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:12 smithi069 bash[21528]: audit 2024-04-02T04:57:11.136805+0000 mon.c (mon.2) 987 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:12.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:12 smithi069 bash[28904]: audit 2024-04-02T04:57:11.136805+0000 mon.c (mon.2) 987 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:12.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:12.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:12 smithi098 bash[23790]: audit 2024-04-02T04:57:11.136805+0000 mon.c (mon.2) 987 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:12.431 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:12.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:12.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/2220974929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006a1d0 msgr2=0x7fba4006a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- 172.21.15.69:0/2220974929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006a1d0 0x7fba4006a5b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba340099c0 tx=0x7fba34009cd0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/2220974929 shutdown_connections 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- 172.21.15.69:0/2220974929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba40107620 0x7fba4010f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- 172.21.15.69:0/2220974929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006a1d0 0x7fba4006a5b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- 172.21.15.69:0/2220974929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006ab80 0x7fba40106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/2220974929 >> 172.21.15.69:0/2220974929 conn(0x7fba400771e0 msgr2=0x7fba400775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/2220974929 shutdown_connections 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/2220974929 wait complete. 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 librados: starting msgr at 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 librados: starting objecter 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 Processor -- start 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- start start 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 librados: setting wanted keys 2024-04-02T04:57:12.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 librados: calling monclient init 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006a1d0 0x7fba40101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006ab80 0x7fba401017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba40107620 0x7fba40101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba40111a80 con 0x7fba4006a1d0 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba40111900 con 0x7fba40107620 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba40111c00 con 0x7fba4006ab80 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006ab80 0x7fba401017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006a1d0 0x7fba40101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006ab80 0x7fba401017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:35956/0 (socket says 172.21.15.69:35956) 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3ffff700 1 -- 172.21.15.69:0/1175360927 learned_addr learned my addr 172.21.15.69:0/1175360927 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:12.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba40107620 0x7fba40101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006ab80 msgr2=0x7fba401017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006ab80 0x7fba401017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba40107620 msgr2=0x7fba40101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba40107620 0x7fba40101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba401a6580 con 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4006ab80 0x7fba401017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006a1d0 0x7fba40101270 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fba340099c0 tx=0x7fba3400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3dffb700 10 client.?.objecter ms_handle_connect 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34009e60 con 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba401a6810 con 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fba34005550 con 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3400bea0 con 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 10 client.16608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba401a6cc0 con 0x7fba4006a1d0 2024-04-02T04:57:12.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.547+0000 7fba46fa3700 1 librados: init done 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fba4004ff00 con 0x7fba4006a1d0 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fba34003a80 con 0x7fba4006a1d0 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3006f5e0 0x7fba30071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(555..555 src has 74..555) ==== 10584+0+0 (secure 0 0 0) 0x7fba3408fae0 con 0x7fba4006a1d0 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 osd_map(555..555 src has 74..555) 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 3 client.16608.objecter handle_osd_map got epochs [555,555] > 0 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 3 client.16608.objecter handle_osd_map decoding full epoch 555 2024-04-02T04:57:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3006f5e0 0x7fba30071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 20 client.16608.objecter dump_active .. 0 homeless 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v555) ==== 54+0+16 (secure 0 0 0) 0x7fba34013070 con 0x7fba4006a1d0 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 10 client.16608.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 1 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fba400ffae0 0x7fba4002c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 1 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 0 1 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 4 2024-04-02T04:57:12.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 10 client.16608.objecter _send_command 1 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba40063360 con 0x7fba400ffae0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3006f5e0 0x7fba30071aa0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fba40108ab0 tx=0x7fba2c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba3006f5e0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fba400ffae0 0x7fba4002c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba400ffae0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fba400ffae0 0x7fba4002c9b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba40063360 con 0x7fba400ffae0 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 1: 0 ) 2024-04-02T04:57:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 10 client.16608.objecter _finish_command 1 = system:0 2024-04-02T04:57:12.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 4 2024-04-02T04:57:12.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.551+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 0 1 2024-04-02T04:57:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 10 client.16608.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 2 2024-04-02T04:57:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 2 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba400ff970 osd=0 3 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 0 2 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 4 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 10 client.16608.objecter _send_command 2 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba4002cef0 con 0x7fba400ffae0 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba4002cef0 con 0x7fba400ffae0 2024-04-02T04:57:12.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 10 client.16608.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 4 2024-04-02T04:57:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 0 2 2024-04-02T04:57:12.571 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 10 client.16608.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fba40062a40 0x7fba401aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 1 3 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 4 2024-04-02T04:57:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 10 client.16608.objecter _send_command 3 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fba4002dde0 con 0x7fba40062a40 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fba40062a40 0x7fba401aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba40062a40 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fba40062a40 0x7fba401aaf40 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba4002dde0 con 0x7fba40062a40 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 3: 0 ) 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 10 client.16608.objecter _finish_command 3 = system:0 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 4 2024-04-02T04:57:12.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.571+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 1 3 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 10 client.16608.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 4 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 4 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba400628d0 osd=1 3 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 1 4 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 4 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 10 client.16608.objecter _send_command 4 2024-04-02T04:57:12.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba4002d350 con 0x7fba40062a40 2024-04-02T04:57:12.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba4002d350 con 0x7fba40062a40 2024-04-02T04:57:12.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3dffb700 10 client.16608.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 4 2024-04-02T04:57:12.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 1 4 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 10 client.16608.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 5 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fba40109d30 0x7fba401ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 5 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 2 5 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 4 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 10 client.16608.objecter _send_command 5 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fba401b1470 con 0x7fba40109d30 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fba40109d30 0x7fba401ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fba40109d30 0x7fba401ae610 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.591+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba40109d30 2024-04-02T04:57:12.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.595+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba401b1470 con 0x7fba40109d30 2024-04-02T04:57:12.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.595+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 5: 0 ) 2024-04-02T04:57:12.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.595+0000 7fba3dffb700 10 client.16608.objecter _finish_command 5 = system:0 2024-04-02T04:57:12.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.595+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 4 2024-04-02T04:57:12.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.595+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 2 5 2024-04-02T04:57:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 10 client.16608.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 6 2024-04-02T04:57:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 6 2024-04-02T04:57:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba4002d750 osd=2 3 2024-04-02T04:57:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 2 6 2024-04-02T04:57:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 4 2024-04-02T04:57:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 10 client.16608.objecter _send_command 6 2024-04-02T04:57:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba401092a0 con 0x7fba40109d30 2024-04-02T04:57:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba401092a0 con 0x7fba40109d30 2024-04-02T04:57:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba3dffb700 10 client.16608.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 4 2024-04-02T04:57:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 2 6 2024-04-02T04:57:12.612 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:57:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 10 client.16608.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 7 2024-04-02T04:57:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fba401afcd0 0x7fba401b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 7 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 3 7 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 4 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 10 client.16608.objecter _send_command 7 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fba401ac400 con 0x7fba401afcd0 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.611+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fba401afcd0 0x7fba401b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba401afcd0 2024-04-02T04:57:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fba401afcd0 0x7fba401b00f0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba401ac400 con 0x7fba401afcd0 2024-04-02T04:57:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 7: 0 ) 2024-04-02T04:57:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba3dffb700 10 client.16608.objecter _finish_command 7 = system:0 2024-04-02T04:57:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 4 2024-04-02T04:57:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.615+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 3 7 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 10 client.16608.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 8 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 8 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401abc50 osd=3 3 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 3 8 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 4 2024-04-02T04:57:12.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 10 client.16608.objecter _send_command 8 2024-04-02T04:57:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.631+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba401ab6d0 con 0x7fba401afcd0 2024-04-02T04:57:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba401ab6d0 con 0x7fba401afcd0 2024-04-02T04:57:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 10 client.16608.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 4 2024-04-02T04:57:12.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 3 8 2024-04-02T04:57:12.633 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:57:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 10 client.16608.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 9 2024-04-02T04:57:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fba401ab840 0x7fba401b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 9 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 4 9 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 4 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 10 client.16608.objecter _send_command 9 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fba401b85e0 con 0x7fba401ab840 2024-04-02T04:57:12.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fba401ab840 0x7fba401b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba401ab840 2024-04-02T04:57:12.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fba401ab840 0x7fba401b5640 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba401b85e0 con 0x7fba401ab840 2024-04-02T04:57:12.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 9: 0 ) 2024-04-02T04:57:12.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 10 client.16608.objecter _finish_command 9 = system:0 2024-04-02T04:57:12.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 4 2024-04-02T04:57:12.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.635+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 4 9 2024-04-02T04:57:12.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 10 client.16608.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 10 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 10 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401ab6d0 osd=4 3 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 4 10 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 4 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 10 client.16608.objecter _send_command 10 2024-04-02T04:57:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.655+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba401ab4e0 con 0x7fba401ab840 2024-04-02T04:57:12.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba401ab4e0 con 0x7fba401ab840 2024-04-02T04:57:12.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3dffb700 10 client.16608.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 4 2024-04-02T04:57:12.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 4 10 2024-04-02T04:57:12.658 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:57:12.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 10 client.16608.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 11 2024-04-02T04:57:12.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fba401b6140 0x7fba401b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 11 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 5 11 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 4 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 10 client.16608.objecter _send_command 11 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fba401bb9b0 con 0x7fba401b6140 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fba401b6140 0x7fba401b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba401b6140 2024-04-02T04:57:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.659+0000 7fba3ffff700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fba401b6140 0x7fba401b65a0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.663+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba401bb9b0 con 0x7fba401b6140 2024-04-02T04:57:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.663+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 11: 0 ) 2024-04-02T04:57:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.663+0000 7fba3dffb700 10 client.16608.objecter _finish_command 11 = system:0 2024-04-02T04:57:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.663+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 4 2024-04-02T04:57:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.663+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 5 11 2024-04-02T04:57:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 10 client.16608.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 12 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 12 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401ab4e0 osd=5 3 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 5 12 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 4 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 10 client.16608.objecter _send_command 12 2024-04-02T04:57:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.679+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba401b6bf0 con 0x7fba401b6140 2024-04-02T04:57:12.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba401b6bf0 con 0x7fba401b6140 2024-04-02T04:57:12.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba3dffb700 10 client.16608.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 4 2024-04-02T04:57:12.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 5 12 2024-04-02T04:57:12.683 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:57:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 10 client.16608.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 13 2024-04-02T04:57:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fba401b7100 0x7fba401b7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 13 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 6 13 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 4 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 10 client.16608.objecter _send_command 13 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fba401b7ac0 con 0x7fba401b7100 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.683+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fba401b7100 0x7fba401b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba401b7100 2024-04-02T04:57:12.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba45540700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fba401b7100 0x7fba401b7580 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba401b7ac0 con 0x7fba401b7100 2024-04-02T04:57:12.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 13: 0 ) 2024-04-02T04:57:12.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba3dffb700 10 client.16608.objecter _finish_command 13 = system:0 2024-04-02T04:57:12.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 4 2024-04-02T04:57:12.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.687+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 6 13 2024-04-02T04:57:12.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 10 client.16608.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 14 2024-04-02T04:57:12.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 14 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401b7f50 osd=6 3 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 6 14 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 4 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 10 client.16608.objecter _send_command 14 2024-04-02T04:57:12.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.703+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba401b7ac0 con 0x7fba401b7100 2024-04-02T04:57:12.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba401b7ac0 con 0x7fba401b7100 2024-04-02T04:57:12.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba3dffb700 10 client.16608.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 4 2024-04-02T04:57:12.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 6 14 2024-04-02T04:57:12.708 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 10 client.16608.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 15 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fba401bcd50 0x7fba401bf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 15 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 7 15 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 4 2024-04-02T04:57:12.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 10 client.16608.objecter _send_command 15 2024-04-02T04:57:12.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.707+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fba401c1fa0 con 0x7fba401bcd50 2024-04-02T04:57:12.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fba401bcd50 0x7fba401bf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:12.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba3dffb700 10 client.16608.objecter ms_handle_connect 0x7fba401bcd50 2024-04-02T04:57:12.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba44d3f700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fba401bcd50 0x7fba401bf140 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:12.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba401c1fa0 con 0x7fba401bcd50 2024-04-02T04:57:12.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 15: 0 ) 2024-04-02T04:57:12.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba3dffb700 10 client.16608.objecter _finish_command 15 = system:0 2024-04-02T04:57:12.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 4 2024-04-02T04:57:12.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.711+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 7 15 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 10 client.16608.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign -1 16 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 20 client.16608.objecter _get_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_remove -1 16 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 20 client.16608.objecter get_session s=0x7fba401b7ac0 osd=7 3 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 15 client.16608.objecter _session_command_op_assign 7 16 2024-04-02T04:57:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 4 2024-04-02T04:57:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 10 client.16608.objecter _send_command 16 2024-04-02T04:57:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.727+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba401bf680 con 0x7fba401bcd50 2024-04-02T04:57:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.731+0000 7fba3dffb700 1 -- 172.21.15.69:0/1175360927 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba401bf680 con 0x7fba401bcd50 2024-04-02T04:57:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.731+0000 7fba3dffb700 10 client.16608.objecter ms_dispatch 0x7fba40001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.731+0000 7fba3dffb700 10 client.16608.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.731+0000 7fba3dffb700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 4 2024-04-02T04:57:12.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.731+0000 7fba3dffb700 15 client.16608.objecter _session_command_op_remove 7 16 2024-04-02T04:57:12.732 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 librados: watch_flush enter 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 librados: watch_flush exit 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.0 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fba400ffae0 msgr2=0x7fba4002c9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fba400ffae0 0x7fba4002c9b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400ff970 osd=0 1 2024-04-02T04:57:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.1 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fba40062a40 msgr2=0x7fba401aaf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fba40062a40 0x7fba401aaf40 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba400628d0 osd=1 1 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.2 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fba40109d30 msgr2=0x7fba401ae610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fba40109d30 0x7fba401ae610 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba4002d750 osd=2 1 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.3 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fba401afcd0 msgr2=0x7fba401b00f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fba401afcd0 0x7fba401b00f0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401abc50 osd=3 1 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba45540700 1 -- 172.21.15.69:0/1175360927 reap_dead start 2024-04-02T04:57:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.4 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fba401ab840 msgr2=0x7fba401b5640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fba401ab840 0x7fba401b5640 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba45540700 1 -- 172.21.15.69:0/1175360927 reap_dead start 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab6d0 osd=4 1 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.5 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fba401b6140 msgr2=0x7fba401b65a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fba401b6140 0x7fba401b65a0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401ab4e0 osd=5 1 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.6 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fba401b7100 msgr2=0x7fba401b7580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fba401b7100 0x7fba401b7580 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7f50 osd=6 1 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter close_session for osd.7 2024-04-02T04:57:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fba401bcd50 msgr2=0x7fba401bf140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fba401bcd50 0x7fba401bf140 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter put_session s=0x7fba401b7ac0 osd=7 1 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 20 client.16608.objecter shutdown clearing up homeless session... 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 10 client.16608.objecter successfully canceled tick 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3006f5e0 msgr2=0x7fba30071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3006f5e0 0x7fba30071aa0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fba40108ab0 tx=0x7fba2c009260 comp rx=0 tx=0).stop 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba45540700 1 -- 172.21.15.69:0/1175360927 reap_dead start 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006a1d0 msgr2=0x7fba40101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006a1d0 0x7fba40101270 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fba340099c0 tx=0x7fba3400fa00 comp rx=0 tx=0).stop 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 shutdown_connections 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 --2- 172.21.15.69:0/1175360927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4006a1d0 0x7fba40101270 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 >> 172.21.15.69:0/1175360927 conn(0x7fba400771e0 msgr2=0x7fba400ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 shutdown_connections 2024-04-02T04:57:12.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 -- 172.21.15.69:0/1175360927 wait complete. 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:12.735+0000 7fba46fa3700 1 librados: shutdown 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:57:12.738 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:57:12.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:12.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:12.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:12.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:13.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:13.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:13.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:13.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:13.303 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:13 smithi069 bash[28904]: audit 2024-04-02T04:57:12.138646+0000 mon.c (mon.2) 988 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:13.303 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:13 smithi069 bash[28904]: cluster 2024-04-02T04:57:12.193604+0000 mgr.y (mgr.14538) 529 : cluster 0 pgmap v828: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:13.303 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:13 smithi069 bash[28904]: audit 2024-04-02T04:57:12.552965+0000 mon.a (mon.0) 3003 : audit 0 from='client.? 172.21.15.69:0/1175360927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:13.304 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:13 smithi069 bash[21528]: audit 2024-04-02T04:57:12.138646+0000 mon.c (mon.2) 988 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:13.304 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:13 smithi069 bash[21528]: cluster 2024-04-02T04:57:12.193604+0000 mgr.y (mgr.14538) 529 : cluster 0 pgmap v828: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:13.304 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:13 smithi069 bash[21528]: audit 2024-04-02T04:57:12.552965+0000 mon.a (mon.0) 3003 : audit 0 from='client.? 172.21.15.69:0/1175360927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:13.319 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:13.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:13 smithi098 bash[23790]: audit 2024-04-02T04:57:12.138646+0000 mon.c (mon.2) 988 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:13.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:13 smithi098 bash[23790]: cluster 2024-04-02T04:57:12.193604+0000 mgr.y (mgr.14538) 529 : cluster 0 pgmap v828: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:13.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:13 smithi098 bash[23790]: audit 2024-04-02T04:57:12.552965+0000 mon.a (mon.0) 3003 : audit 0 from='client.? 172.21.15.69:0/1175360927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:13.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:13.449 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:13.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:57:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:57:13.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:13.579 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:13.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:13.706 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:13.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:13.940 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:13.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:14.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:14.096 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:14.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[28904]: audit 2024-04-02T04:57:13.140174+0000 mon.c (mon.2) 989 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[28904]: audit 2024-04-02T04:57:13.405884+0000 mgr.y (mgr.14538) 530 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[28904]: cluster 2024-04-02T04:57:13.441711+0000 osd.6 (osd.6) 249 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[28904]: cluster 2024-04-02T04:57:13.442106+0000 osd.6 (osd.6) 250 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[21528]: audit 2024-04-02T04:57:13.140174+0000 mon.c (mon.2) 989 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[21528]: audit 2024-04-02T04:57:13.405884+0000 mgr.y (mgr.14538) 530 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[21528]: cluster 2024-04-02T04:57:13.441711+0000 osd.6 (osd.6) 249 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:14.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:14 smithi069 bash[21528]: cluster 2024-04-02T04:57:13.442106+0000 osd.6 (osd.6) 250 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:14.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:14.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:14.353 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:14.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:14 smithi098 bash[23790]: audit 2024-04-02T04:57:13.140174+0000 mon.c (mon.2) 989 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:14.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:14 smithi098 bash[23790]: audit 2024-04-02T04:57:13.405884+0000 mgr.y (mgr.14538) 530 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:14.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:14 smithi098 bash[23790]: cluster 2024-04-02T04:57:13.441711+0000 osd.6 (osd.6) 249 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:14.355 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:14 smithi098 bash[23790]: cluster 2024-04-02T04:57:13.442106+0000 osd.6 (osd.6) 250 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:14.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:14.485 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:14.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:14.614 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:14.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:14.745 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:14.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:14.877 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:14.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:14.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:15.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:15.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:15.266 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:15.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[21528]: cluster 2024-04-02T04:57:14.110986+0000 mon.a (mon.0) 3004 : cluster 0 osdmap e556: 8 total, 8 up, 6 in 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[21528]: audit 2024-04-02T04:57:14.141386+0000 mon.c (mon.2) 990 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[21528]: cluster 2024-04-02T04:57:14.194317+0000 mgr.y (mgr.14538) 531 : cluster 0 pgmap v830: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[21528]: cluster 2024-04-02T04:57:14.959589+0000 mon.a (mon.0) 3005 : cluster 0 osdmap e557: 8 total, 8 up, 6 in 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[28904]: cluster 2024-04-02T04:57:14.110986+0000 mon.a (mon.0) 3004 : cluster 0 osdmap e556: 8 total, 8 up, 6 in 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[28904]: audit 2024-04-02T04:57:14.141386+0000 mon.c (mon.2) 990 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:15.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[28904]: cluster 2024-04-02T04:57:14.194317+0000 mgr.y (mgr.14538) 531 : cluster 0 pgmap v830: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:15.383 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:15 smithi069 bash[28904]: cluster 2024-04-02T04:57:14.959589+0000 mon.a (mon.0) 3005 : cluster 0 osdmap e557: 8 total, 8 up, 6 in 2024-04-02T04:57:15.396 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:15.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:15 smithi098 bash[23790]: cluster 2024-04-02T04:57:14.110986+0000 mon.a (mon.0) 3004 : cluster 0 osdmap e556: 8 total, 8 up, 6 in 2024-04-02T04:57:15.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:15 smithi098 bash[23790]: audit 2024-04-02T04:57:14.141386+0000 mon.c (mon.2) 990 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:15.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:15 smithi098 bash[23790]: cluster 2024-04-02T04:57:14.194317+0000 mgr.y (mgr.14538) 531 : cluster 0 pgmap v830: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:15.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:15 smithi098 bash[23790]: cluster 2024-04-02T04:57:14.959589+0000 mon.a (mon.0) 3005 : cluster 0 osdmap e557: 8 total, 8 up, 6 in 2024-04-02T04:57:15.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:15.526 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:15.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:15.653 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:15.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:15.782 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:15.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:15.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:16.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:16.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:16.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:16.167 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:16.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:16.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:16.411 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:16 smithi098 bash[23790]: audit 2024-04-02T04:57:15.252637+0000 mon.c (mon.2) 991 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:16.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:16.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:16 smithi069 bash[21528]: audit 2024-04-02T04:57:15.252637+0000 mon.c (mon.2) 991 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:16.529 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:16 smithi069 bash[28904]: audit 2024-04-02T04:57:15.252637+0000 mon.c (mon.2) 991 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:16.552 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:16.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:16.682 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:16.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:16.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:16.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:16.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:16.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:17.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:17.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:17.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:17.205 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:17 smithi098 bash[23790]: cluster 2024-04-02T04:57:16.194869+0000 mgr.y (mgr.14538) 532 : cluster 0 pgmap v832: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:17.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:17 smithi098 bash[23790]: audit 2024-04-02T04:57:16.254416+0000 mon.c (mon.2) 992 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:17.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:17.345 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:17 smithi069 bash[28904]: cluster 2024-04-02T04:57:16.194869+0000 mgr.y (mgr.14538) 532 : cluster 0 pgmap v832: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:17.345 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:17 smithi069 bash[28904]: audit 2024-04-02T04:57:16.254416+0000 mon.c (mon.2) 992 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:17.345 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:17 smithi069 bash[21528]: cluster 2024-04-02T04:57:16.194869+0000 mgr.y (mgr.14538) 532 : cluster 0 pgmap v832: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:17.345 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:17 smithi069 bash[21528]: audit 2024-04-02T04:57:16.254416+0000 mon.c (mon.2) 992 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:17.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:17.568 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:17.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:17.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:17.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:17.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:17.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:17.984 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:17.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:18.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:57:18.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:18.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:18.271 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:18.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:18 smithi069 bash[21528]: audit 2024-04-02T04:57:17.256099+0000 mon.c (mon.2) 993 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:18.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:18 smithi069 bash[28904]: audit 2024-04-02T04:57:17.256099+0000 mon.c (mon.2) 993 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:18.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:18.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:18 smithi098 bash[23790]: audit 2024-04-02T04:57:17.256099+0000 mon.c (mon.2) 993 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:18.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:18.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:18.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:18.661 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:18.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:18.789 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:18.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:18.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 -- 172.21.15.69:0/528689617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 msgr2=0x7f2174106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 --2- 172.21.15.69:0/528689617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 0x7f2174106fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2168009990 tx=0x7f2168009ca0 comp rx=0 tx=0).stop 2024-04-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 -- 172.21.15.69:0/528689617 shutdown_connections 2024-04-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 --2- 172.21.15.69:0/528689617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2174107680 0x7f217410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 --2- 172.21.15.69:0/528689617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f217406ab00 0x7f217406aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 --2- 172.21.15.69:0/528689617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 0x7f2174106fa0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 -- 172.21.15.69:0/528689617 >> 172.21.15.69:0/528689617 conn(0x7f2174078240 msgr2=0x7f2174078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 -- 172.21.15.69:0/528689617 shutdown_connections 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 -- 172.21.15.69:0/528689617 wait complete. 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 librados: starting msgr at 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 librados: starting objecter 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 Processor -- start 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 -- start start 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 librados: setting wanted keys 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.939+0000 7f21791fb700 1 librados: calling monclient init 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 0x7f21741011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 0x7f2174101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2174107680 0x7f2174101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2174111ac0 con 0x7f217406b4b0 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2174111940 con 0x7f2174107680 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2174111c40 con 0x7f217406ab00 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 0x7f21741011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 0x7f21741011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57702/0 (socket says 172.21.15.69:57702) 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 -- 172.21.15.69:0/1896750063 learned_addr learned my addr 172.21.15.69:0/1896750063 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f217259c700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 0x7f2174101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f217359e700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2174107680 0x7f2174101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 -- 172.21.15.69:0/1896750063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2174107680 msgr2=0x7f2174101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2174107680 0x7f2174101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 -- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 msgr2=0x7f2174101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 0x7f2174101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 -- 172.21.15.69:0/1896750063 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21741a6570 con 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f2172d9d700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 0x7f21741011f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f216400e9a0 tx=0x7f216400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 10 client.?.objecter ms_handle_connect 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216400cbd0 con 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f216400cd70 con 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21741a6860 con 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164010e60 con 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 10 client.26705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21741a6d70 con 0x7f217406ab00 2024-04-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 librados: init done 2024-04-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2160005120 con 0x7f217406ab00 2024-04-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f2164010430 con 0x7f217406ab00 2024-04-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f215c06f5e0 0x7f215c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(557..557 src has 74..557) ==== 10584+0+0 (secure 0 0 0) 0x7f216408ee10 con 0x7f217406ab00 2024-04-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 10 client.26705.objecter ms_dispatch 0x7f2174001550 osd_map(557..557 src has 74..557) 2024-04-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 3 client.26705.objecter handle_osd_map got epochs [557,557] > 0 2024-04-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 3 client.26705.objecter handle_osd_map decoding full epoch 557 2024-04-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f217259c700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f215c06f5e0 0x7f215c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.943+0000 7f215bfff700 20 client.26705.objecter dump_active .. 0 homeless 2024-04-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.947+0000 7f215bfff700 10 client.26705.objecter ms_handle_connect 0x7f215c06f5e0 2024-04-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.947+0000 7f217259c700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f215c06f5e0 0x7f215c071aa0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f216800b2e0 tx=0x7f2168005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:18.946 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:57:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:57:18.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:18.951+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f216405b7c0 con 0x7f217406ab00 2024-04-02T04:57:19.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:19.047 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:19.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:19.182 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:19.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.195+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f2160005410 con 0x7f217406ab00 2024-04-02T04:57:19.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:19.296 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[37262]: debug 2024-04-02T04:57:19.043+0000 7fa4c0352700 -1 snap_mapper.add_oid found existing snaps mapped on 302:e0871dfe:test-rados-api-smithi069-72349-80::foo:2, removing 2024-04-02T04:57:19.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: cluster 2024-04-02T04:57:18.196048+0000 mgr.y (mgr.14538) 533 : cluster 0 pgmap v833: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:19.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: audit 2024-04-02T04:57:18.257321+0000 mon.c (mon.2) 994 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: audit 2024-04-02T04:57:18.326500+0000 mon.b (mon.1) 271 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: audit 2024-04-02T04:57:19.049223+0000 mon.c (mon.2) 995 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: audit 2024-04-02T04:57:19.049816+0000 mon.a (mon.0) 3006 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: audit 2024-04-02T04:57:19.050809+0000 mon.c (mon.2) 996 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-79"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[21528]: audit 2024-04-02T04:57:19.051600+0000 mon.a (mon.0) 3007 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-79"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: cluster 2024-04-02T04:57:18.196048+0000 mgr.y (mgr.14538) 533 : cluster 0 pgmap v833: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: audit 2024-04-02T04:57:18.257321+0000 mon.c (mon.2) 994 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: audit 2024-04-02T04:57:18.326500+0000 mon.b (mon.1) 271 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:19.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: audit 2024-04-02T04:57:19.049223+0000 mon.c (mon.2) 995 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:19.298 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: audit 2024-04-02T04:57:19.049816+0000 mon.a (mon.0) 3006 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:19.298 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: audit 2024-04-02T04:57:19.050809+0000 mon.c (mon.2) 996 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-79"}]: dispatch 2024-04-02T04:57:19.298 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:19 smithi069 bash[28904]: audit 2024-04-02T04:57:19.051600+0000 mon.a (mon.0) 3007 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-79"}]: dispatch 2024-04-02T04:57:19.309 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:19.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:19.426 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[27343]: debug 2024-04-02T04:57:19.038+0000 7fde7a39f700 -1 snap_mapper.add_oid found existing snaps mapped on 302:e0871dfe:test-rados-api-smithi069-72349-80::foo:2, removing 2024-04-02T04:57:19.427 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[69630]: debug 2024-04-02T04:57:19.038+0000 7f5adfee5700 -1 snap_mapper.add_oid found existing snaps mapped on 302:e0871dfe:test-rados-api-smithi069-72349-80::foo:2, removing 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: cluster 2024-04-02T04:57:18.196048+0000 mgr.y (mgr.14538) 533 : cluster 0 pgmap v833: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/1278 objects misplaced (0.156%) 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: audit 2024-04-02T04:57:18.257321+0000 mon.c (mon.2) 994 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: audit 2024-04-02T04:57:18.326500+0000 mon.b (mon.1) 271 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: audit 2024-04-02T04:57:19.049223+0000 mon.c (mon.2) 995 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: audit 2024-04-02T04:57:19.049816+0000 mon.a (mon.0) 3006 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: audit 2024-04-02T04:57:19.050809+0000 mon.c (mon.2) 996 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-79"}]: dispatch 2024-04-02T04:57:19.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:19 smithi098 bash[23790]: audit 2024-04-02T04:57:19.051600+0000 mon.a (mon.0) 3007 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-79"}]: dispatch 2024-04-02T04:57:19.443 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:19.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:19.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:19.700 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:19.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:19.829 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:19.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:19.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.963+0000 7f215bfff700 1 -- 172.21.15.69:0/1896750063 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v559) ==== 91+0+0 (secure 0 0 0) 0x7f216405f210 con 0x7f217406ab00 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 10 librados: watch_flush enter 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 10 librados: watch_flush exit 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 20 client.26705.objecter shutdown clearing up homeless session... 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 10 client.26705.objecter successfully canceled tick 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f215c06f5e0 msgr2=0x7f215c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f215c06f5e0 0x7f215c071aa0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f216800b2e0 tx=0x7f2168005960 comp rx=0 tx=0).stop 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 msgr2=0x7f21741011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 0x7f21741011f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f216400e9a0 tx=0x7f216400ecb0 comp rx=0 tx=0).stop 2024-04-02T04:57:19.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 shutdown_connections 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f215c06f5e0 0x7f215c071aa0 secure :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f216800b2e0 tx=0x7f2168005960 comp rx=0 tx=0).stop 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2174107680 0x7f2174101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f217406ab00 0x7f21741011f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 --2- 172.21.15.69:0/1896750063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217406b4b0 0x7f2174101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 >> 172.21.15.69:0/1896750063 conn(0x7f2174078240 msgr2=0x7f21740fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 shutdown_connections 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 -- 172.21.15.69:0/1896750063 wait complete. 2024-04-02T04:57:19.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:19.967+0000 7f21791fb700 1 librados: shutdown 2024-04-02T04:57:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:19.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:19.973 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:57:20.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:20.100 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:20.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: cluster 2024-04-02T04:57:19.153953+0000 mon.a (mon.0) 3008 : cluster 0 osdmap e558: 8 total, 8 up, 6 in 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: audit 2024-04-02T04:57:19.197611+0000 mon.c (mon.2) 997 : audit 1 from='client.? 172.21.15.69:0/1896750063' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: audit 2024-04-02T04:57:19.258430+0000 mon.c (mon.2) 998 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: audit 2024-04-02T04:57:19.604884+0000 mon.a (mon.0) 3009 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: cluster 2024-04-02T04:57:19.955062+0000 mon.a (mon.0) 3010 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: audit 2024-04-02T04:57:19.958321+0000 mon.a (mon.0) 3011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: cluster 2024-04-02T04:57:19.965953+0000 mon.a (mon.0) 3012 : cluster 0 osdmap e559: 8 total, 8 up, 6 in 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: audit 2024-04-02T04:57:19.972819+0000 mon.c (mon.2) 999 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:20.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[28904]: audit 2024-04-02T04:57:19.978572+0000 mon.a (mon.0) 3013 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: cluster 2024-04-02T04:57:19.153953+0000 mon.a (mon.0) 3008 : cluster 0 osdmap e558: 8 total, 8 up, 6 in 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: audit 2024-04-02T04:57:19.197611+0000 mon.c (mon.2) 997 : audit 1 from='client.? 172.21.15.69:0/1896750063' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: audit 2024-04-02T04:57:19.258430+0000 mon.c (mon.2) 998 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: audit 2024-04-02T04:57:19.604884+0000 mon.a (mon.0) 3009 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: cluster 2024-04-02T04:57:19.955062+0000 mon.a (mon.0) 3010 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: audit 2024-04-02T04:57:19.958321+0000 mon.a (mon.0) 3011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: cluster 2024-04-02T04:57:19.965953+0000 mon.a (mon.0) 3012 : cluster 0 osdmap e559: 8 total, 8 up, 6 in 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: audit 2024-04-02T04:57:19.972819+0000 mon.c (mon.2) 999 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:20.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:20 smithi069 bash[21528]: audit 2024-04-02T04:57:19.978572+0000 mon.a (mon.0) 3013 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:20.230 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:20.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: cluster 2024-04-02T04:57:19.153953+0000 mon.a (mon.0) 3008 : cluster 0 osdmap e558: 8 total, 8 up, 6 in 2024-04-02T04:57:20.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: audit 2024-04-02T04:57:19.197611+0000 mon.c (mon.2) 997 : audit 1 from='client.? 172.21.15.69:0/1896750063' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: audit 2024-04-02T04:57:19.258430+0000 mon.c (mon.2) 998 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: audit 2024-04-02T04:57:19.604884+0000 mon.a (mon.0) 3009 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: cluster 2024-04-02T04:57:19.955062+0000 mon.a (mon.0) 3010 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: audit 2024-04-02T04:57:19.958321+0000 mon.a (mon.0) 3011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: cluster 2024-04-02T04:57:19.965953+0000 mon.a (mon.0) 3012 : cluster 0 osdmap e559: 8 total, 8 up, 6 in 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: audit 2024-04-02T04:57:19.972819+0000 mon.c (mon.2) 999 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:20.345 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:20 smithi098 bash[23790]: audit 2024-04-02T04:57:19.978572+0000 mon.a (mon.0) 3013 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:20.361 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:20.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:20.492 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:20.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:20.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:20.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:20.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:20.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:20.984 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:21.010 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:21.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- 172.21.15.69:0/740885124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa28102ed0 msgr2=0x7ffa28106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- 172.21.15.69:0/740885124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa28102ed0 0x7ffa28106f00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0098f0 tx=0x7ffa1c009c00 comp rx=0 tx=0).stop 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- 172.21.15.69:0/740885124 shutdown_connections 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- 172.21.15.69:0/740885124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa281075e0 0x7ffa2810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- 172.21.15.69:0/740885124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102520 0x7ffa28102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- 172.21.15.69:0/740885124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa28102ed0 0x7ffa28106f00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- 172.21.15.69:0/740885124 >> 172.21.15.69:0/740885124 conn(0x7ffa280793f0 msgr2=0x7ffa28079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:21.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- 172.21.15.69:0/740885124 shutdown_connections 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- 172.21.15.69:0/740885124 wait complete. 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 librados: starting msgr at 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 librados: starting objecter 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 Processor -- start 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- start start 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 librados: setting wanted keys 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 librados: calling monclient init 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa28102520 0x7ffa281980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102ed0 0x7ffa28198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 0x7ffa2819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28111a90 con 0x7ffa281075e0 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28111910 con 0x7ffa28102ed0 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2e389700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ffa28111c10 con 0x7ffa28102520 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 0x7ffa2819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:21.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 0x7ffa2819c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36026/0 (socket says 172.21.15.69:36026) 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 -- 172.21.15.69:0/1160263717 learned_addr learned my addr 172.21.15.69:0/1160263717 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa27fff700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa28102520 0x7ffa281980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa277fe700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102ed0 0x7ffa28198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 -- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa28102520 msgr2=0x7ffa281980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa28102520 0x7ffa281980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 -- 172.21.15.69:0/1160263717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102ed0 msgr2=0x7ffa28198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102ed0 0x7ffa28198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 -- 172.21.15.69:0/1160263717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2819d100 con 0x7ffa281075e0 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa277fe700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102ed0 0x7ffa28198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa2c926700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 0x7ffa2819c9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000c340 tx=0x7ffa2000c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa257fa700 10 client.?.objecter ms_handle_connect 0x7ffa281075e0 2024-04-02T04:57:21.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.135+0000 7ffa257fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa200090d0 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ffa20009270 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2819d3f0 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20007cf0 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa2e389700 10 client.16620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa281a4c70 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa2e389700 1 librados: init done 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa2804ff00 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ffa20012070 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa1006f5e0 0x7ffa10071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(560..560 src has 74..560) ==== 10584+0+0 (secure 0 0 0) 0x7ffa2008e7e0 con 0x7ffa281075e0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 10 client.16620.objecter ms_dispatch 0x7ffa28001550 osd_map(560..560 src has 74..560) 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 3 client.16620.objecter handle_osd_map got epochs [560,560] > 0 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 3 client.16620.objecter handle_osd_map decoding full epoch 560 2024-04-02T04:57:21.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa27fff700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa1006f5e0 0x7ffa10071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:21.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 20 client.16620.objecter dump_active .. 0 homeless 2024-04-02T04:57:21.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa257fa700 10 client.16620.objecter ms_handle_connect 0x7ffa1006f5e0 2024-04-02T04:57:21.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.139+0000 7ffa27fff700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa1006f5e0 0x7ffa10071aa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005950 tx=0x7ffa1800b410 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:21.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.143+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffa20010040 con 0x7ffa281075e0 2024-04-02T04:57:21.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: cluster 2024-04-02T04:57:20.178841+0000 osd.1 (osd.1) 305 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: cluster 2024-04-02T04:57:20.179019+0000 osd.1 (osd.1) 306 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: cluster 2024-04-02T04:57:20.196612+0000 mgr.y (mgr.14538) 534 : cluster 0 pgmap v836: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: audit 2024-04-02T04:57:20.259705+0000 mon.c (mon.2) 1000 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: audit 2024-04-02T04:57:20.960376+0000 mon.a (mon.0) 3014 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: cluster 2024-04-02T04:57:20.966766+0000 mon.a (mon.0) 3015 : cluster 0 osdmap e560: 8 total, 8 up, 6 in 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: audit 2024-04-02T04:57:20.968983+0000 mon.c (mon.2) 1001 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:21.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: audit 2024-04-02T04:57:20.979513+0000 mon.c (mon.2) 1002 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[28904]: audit 2024-04-02T04:57:20.983055+0000 mon.a (mon.0) 3016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: cluster 2024-04-02T04:57:20.178841+0000 osd.1 (osd.1) 305 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: cluster 2024-04-02T04:57:20.179019+0000 osd.1 (osd.1) 306 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: cluster 2024-04-02T04:57:20.196612+0000 mgr.y (mgr.14538) 534 : cluster 0 pgmap v836: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: audit 2024-04-02T04:57:20.259705+0000 mon.c (mon.2) 1000 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: audit 2024-04-02T04:57:20.960376+0000 mon.a (mon.0) 3014 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: cluster 2024-04-02T04:57:20.966766+0000 mon.a (mon.0) 3015 : cluster 0 osdmap e560: 8 total, 8 up, 6 in 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: audit 2024-04-02T04:57:20.968983+0000 mon.c (mon.2) 1001 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:21.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: audit 2024-04-02T04:57:20.979513+0000 mon.c (mon.2) 1002 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:21.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:21 smithi069 bash[21528]: audit 2024-04-02T04:57:20.983055+0000 mon.a (mon.0) 3016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:21.266 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:21.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.359+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}) -- 0x7ffa28100840 con 0x7ffa1006f5e0 2024-04-02T04:57:21.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.363+0000 7ffa257fa700 1 -- 172.21.15.69:0/1160263717 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+297 (secure 0 0 0) 0x7ffa28100840 con 0x7ffa1006f5e0 2024-04-02T04:57:21.362 INFO:teuthology.orchestra.run.smithi069.stdout:moved 0 / 939 (0%) 2024-04-02T04:57:21.362 INFO:teuthology.orchestra.run.smithi069.stdout:avg 156.5 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:stddev 8.67468 -> 8.67468 (expected baseline 11.42) 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:min osd.7 with 142 -> 142 pgs (0.907348 -> 0.907348 * mean) 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:max osd.6 with 170 -> 170 pgs (1.08626 -> 1.08626 * mean) 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:oload 1000 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:max_change 1 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:max_change_osds 4 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:average_utilization 0.0008 2024-04-02T04:57:21.363 INFO:teuthology.orchestra.run.smithi069.stdout:overload_utilization 0.0082 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.363+0000 7ffa2e389700 10 librados: watch_flush enter 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 10 librados: watch_flush exit 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 20 client.16620.objecter shutdown clearing up homeless session... 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 10 client.16620.objecter successfully canceled tick 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa1006f5e0 msgr2=0x7ffa10071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa1006f5e0 0x7ffa10071aa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005950 tx=0x7ffa1800b410 comp rx=0 tx=0).stop 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 msgr2=0x7ffa2819c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 0x7ffa2819c9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000c340 tx=0x7ffa2000c650 comp rx=0 tx=0).stop 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 shutdown_connections 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ffa1006f5e0 0x7ffa10071aa0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa281075e0 0x7ffa2819c9c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa28102520 0x7ffa281980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 --2- 172.21.15.69:0/1160263717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa28102ed0 0x7ffa28198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:21.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 >> 172.21.15.69:0/1160263717 conn(0x7ffa280793f0 msgr2=0x7ffa28100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:21.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 shutdown_connections 2024-04-02T04:57:21.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 -- 172.21.15.69:0/1160263717 wait complete. 2024-04-02T04:57:21.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:21.367+0000 7ffa2e389700 1 librados: shutdown 2024-04-02T04:57:21.366 INFO:teuthology.orchestra.run.smithi069.stderr:no change 2024-04-02T04:57:21.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:21.391 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: cluster 2024-04-02T04:57:20.178841+0000 osd.1 (osd.1) 305 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: cluster 2024-04-02T04:57:20.179019+0000 osd.1 (osd.1) 306 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: cluster 2024-04-02T04:57:20.196612+0000 mgr.y (mgr.14538) 534 : cluster 0 pgmap v836: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: audit 2024-04-02T04:57:20.259705+0000 mon.c (mon.2) 1000 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: audit 2024-04-02T04:57:20.960376+0000 mon.a (mon.0) 3014 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: cluster 2024-04-02T04:57:20.966766+0000 mon.a (mon.0) 3015 : cluster 0 osdmap e560: 8 total, 8 up, 6 in 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: audit 2024-04-02T04:57:20.968983+0000 mon.c (mon.2) 1001 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: audit 2024-04-02T04:57:20.979513+0000 mon.c (mon.2) 1002 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:21.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:21 smithi098 bash[23790]: audit 2024-04-02T04:57:20.983055+0000 mon.a (mon.0) 3016 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:21.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:21.519 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:21.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:21.649 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:21.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:21.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:21.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:21.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:22.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:22.036 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:22.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:22.160 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:22.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: cluster 2024-04-02T04:57:21.147321+0000 mon.a (mon.0) 3017 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: audit 2024-04-02T04:57:21.260650+0000 mon.c (mon.2) 1003 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: cluster 2024-04-02T04:57:21.325781+0000 osd.2 (osd.2) 252 : cluster 0 102.9 scrub starts 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: cluster 2024-04-02T04:57:21.328454+0000 osd.2 (osd.2) 253 : cluster 0 102.9 scrub ok 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: audit 2024-04-02T04:57:21.360774+0000 mgr.y (mgr.14538) 535 : audit 0 from='client.16620 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: audit 2024-04-02T04:57:21.708800+0000 mon.b (mon.1) 272 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: audit 2024-04-02T04:57:21.963049+0000 mon.a (mon.0) 3018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: cluster 2024-04-02T04:57:21.969130+0000 mon.a (mon.0) 3019 : cluster 0 osdmap e561: 8 total, 8 up, 6 in 2024-04-02T04:57:22.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: audit 2024-04-02T04:57:21.970540+0000 mon.c (mon.2) 1004 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[21528]: audit 2024-04-02T04:57:21.971155+0000 mon.a (mon.0) 3020 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: cluster 2024-04-02T04:57:21.147321+0000 mon.a (mon.0) 3017 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: audit 2024-04-02T04:57:21.260650+0000 mon.c (mon.2) 1003 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: cluster 2024-04-02T04:57:21.325781+0000 osd.2 (osd.2) 252 : cluster 0 102.9 scrub starts 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: cluster 2024-04-02T04:57:21.328454+0000 osd.2 (osd.2) 253 : cluster 0 102.9 scrub ok 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: audit 2024-04-02T04:57:21.360774+0000 mgr.y (mgr.14538) 535 : audit 0 from='client.16620 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: audit 2024-04-02T04:57:21.708800+0000 mon.b (mon.1) 272 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: audit 2024-04-02T04:57:21.963049+0000 mon.a (mon.0) 3018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: cluster 2024-04-02T04:57:21.969130+0000 mon.a (mon.0) 3019 : cluster 0 osdmap e561: 8 total, 8 up, 6 in 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: audit 2024-04-02T04:57:21.970540+0000 mon.c (mon.2) 1004 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:22.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:22 smithi069 bash[28904]: audit 2024-04-02T04:57:21.971155+0000 mon.a (mon.0) 3020 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:22.283 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:22.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:22.412 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: cluster 2024-04-02T04:57:21.147321+0000 mon.a (mon.0) 3017 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: audit 2024-04-02T04:57:21.260650+0000 mon.c (mon.2) 1003 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: cluster 2024-04-02T04:57:21.325781+0000 osd.2 (osd.2) 252 : cluster 0 102.9 scrub starts 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: cluster 2024-04-02T04:57:21.328454+0000 osd.2 (osd.2) 253 : cluster 0 102.9 scrub ok 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: audit 2024-04-02T04:57:21.360774+0000 mgr.y (mgr.14538) 535 : audit 0 from='client.16620 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: audit 2024-04-02T04:57:21.708800+0000 mon.b (mon.1) 272 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: audit 2024-04-02T04:57:21.963049+0000 mon.a (mon.0) 3018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: cluster 2024-04-02T04:57:21.969130+0000 mon.a (mon.0) 3019 : cluster 0 osdmap e561: 8 total, 8 up, 6 in 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: audit 2024-04-02T04:57:21.970540+0000 mon.c (mon.2) 1004 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:22.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:22 smithi098 bash[23790]: audit 2024-04-02T04:57:21.971155+0000 mon.a (mon.0) 3020 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:22.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:22.541 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:22.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:22.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:22.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:22.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:22.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:23.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:23.057 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:23.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:23.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: cluster 2024-04-02T04:57:22.004654+0000 osd.1 (osd.1) 307 : cluster 0 40.a scrub starts 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: cluster 2024-04-02T04:57:22.005975+0000 osd.1 (osd.1) 308 : cluster 0 40.a scrub ok 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: cluster 2024-04-02T04:57:22.197059+0000 mgr.y (mgr.14538) 536 : cluster 0 pgmap v839: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.261612+0000 mon.c (mon.2) 1005 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.390293+0000 mon.b (mon.1) 273 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.392189+0000 mon.b (mon.1) 274 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.406983+0000 mon.a (mon.0) 3021 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.965688+0000 mon.a (mon.0) 3022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:23.301 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: cluster 2024-04-02T04:57:22.973014+0000 mon.a (mon.0) 3023 : cluster 0 osdmap e562: 8 total, 8 up, 6 in 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.977507+0000 mon.c (mon.2) 1006 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[28904]: audit 2024-04-02T04:57:22.978462+0000 mon.a (mon.0) 3024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: cluster 2024-04-02T04:57:22.004654+0000 osd.1 (osd.1) 307 : cluster 0 40.a scrub starts 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: cluster 2024-04-02T04:57:22.005975+0000 osd.1 (osd.1) 308 : cluster 0 40.a scrub ok 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: cluster 2024-04-02T04:57:22.197059+0000 mgr.y (mgr.14538) 536 : cluster 0 pgmap v839: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.261612+0000 mon.c (mon.2) 1005 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.390293+0000 mon.b (mon.1) 273 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.392189+0000 mon.b (mon.1) 274 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.406983+0000 mon.a (mon.0) 3021 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.965688+0000 mon.a (mon.0) 3022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:23.302 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: cluster 2024-04-02T04:57:22.973014+0000 mon.a (mon.0) 3023 : cluster 0 osdmap e562: 8 total, 8 up, 6 in 2024-04-02T04:57:23.303 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.977507+0000 mon.c (mon.2) 1006 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:23.303 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:23 smithi069 bash[21528]: audit 2024-04-02T04:57:22.978462+0000 mon.a (mon.0) 3024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:23.317 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: cluster 2024-04-02T04:57:22.004654+0000 osd.1 (osd.1) 307 : cluster 0 40.a scrub starts 2024-04-02T04:57:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: cluster 2024-04-02T04:57:22.005975+0000 osd.1 (osd.1) 308 : cluster 0 40.a scrub ok 2024-04-02T04:57:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: cluster 2024-04-02T04:57:22.197059+0000 mgr.y (mgr.14538) 536 : cluster 0 pgmap v839: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.261612+0000 mon.c (mon.2) 1005 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.390293+0000 mon.b (mon.1) 273 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:57:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.392189+0000 mon.b (mon.1) 274 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:57:23.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.406983+0000 mon.a (mon.0) 3021 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:57:23.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.965688+0000 mon.a (mon.0) 3022 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:23.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: cluster 2024-04-02T04:57:22.973014+0000 mon.a (mon.0) 3023 : cluster 0 osdmap e562: 8 total, 8 up, 6 in 2024-04-02T04:57:23.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.977507+0000 mon.c (mon.2) 1006 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:23.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[23790]: audit 2024-04-02T04:57:22.978462+0000 mon.a (mon.0) 3024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:23.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:23.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:23.576 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-02T04:57:23.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 110 3.0 2024-04-02T04:57:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:23.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:23.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:57:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:57:23.711 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:23.746 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:57:23.746 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:57:23.746 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:57:23.746 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:57:23.746 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:57:23.813 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:23.832 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:23.935 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:23.958 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:23.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:24.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:24.084 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:24.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: audit 2024-04-02T04:57:23.262906+0000 mon.c (mon.2) 1007 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: cluster 2024-04-02T04:57:23.341752+0000 osd.0 (osd.0) 293 : cluster 0 102.b scrub starts 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: cluster 2024-04-02T04:57:23.346746+0000 osd.0 (osd.0) 294 : cluster 0 102.b scrub ok 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: audit 2024-04-02T04:57:23.417236+0000 mgr.y (mgr.14538) 537 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: audit 2024-04-02T04:57:23.967516+0000 mon.a (mon.0) 3025 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: cluster 2024-04-02T04:57:23.973012+0000 mon.a (mon.0) 3026 : cluster 0 osdmap e563: 8 total, 8 up, 6 in 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: audit 2024-04-02T04:57:23.973231+0000 mon.c (mon.2) 1008 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[28904]: audit 2024-04-02T04:57:23.974092+0000 mon.a (mon.0) 3027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:24.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: audit 2024-04-02T04:57:23.262906+0000 mon.c (mon.2) 1007 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: cluster 2024-04-02T04:57:23.341752+0000 osd.0 (osd.0) 293 : cluster 0 102.b scrub starts 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: cluster 2024-04-02T04:57:23.346746+0000 osd.0 (osd.0) 294 : cluster 0 102.b scrub ok 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: audit 2024-04-02T04:57:23.417236+0000 mgr.y (mgr.14538) 537 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: audit 2024-04-02T04:57:23.967516+0000 mon.a (mon.0) 3025 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: cluster 2024-04-02T04:57:23.973012+0000 mon.a (mon.0) 3026 : cluster 0 osdmap e563: 8 total, 8 up, 6 in 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: audit 2024-04-02T04:57:23.973231+0000 mon.c (mon.2) 1008 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:24.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:24 smithi069 bash[21528]: audit 2024-04-02T04:57:23.974092+0000 mon.a (mon.0) 3027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:24.210 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:24.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:24.339 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:24.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: audit 2024-04-02T04:57:23.262906+0000 mon.c (mon.2) 1007 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: cluster 2024-04-02T04:57:23.341752+0000 osd.0 (osd.0) 293 : cluster 0 102.b scrub starts 2024-04-02T04:57:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: cluster 2024-04-02T04:57:23.346746+0000 osd.0 (osd.0) 294 : cluster 0 102.b scrub ok 2024-04-02T04:57:24.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: audit 2024-04-02T04:57:23.417236+0000 mgr.y (mgr.14538) 537 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: audit 2024-04-02T04:57:23.967516+0000 mon.a (mon.0) 3025 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: cluster 2024-04-02T04:57:23.973012+0000 mon.a (mon.0) 3026 : cluster 0 osdmap e563: 8 total, 8 up, 6 in 2024-04-02T04:57:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: audit 2024-04-02T04:57:23.973231+0000 mon.c (mon.2) 1008 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:24.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:24 smithi098 bash[23790]: audit 2024-04-02T04:57:23.974092+0000 mon.a (mon.0) 3027 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:24.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:57:24.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:24.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-02T04:57:24.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:24.758 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:24.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:24.889 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:24.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:24.992 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:25.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:25.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:25.147 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:25.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:25.278 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.198329+0000 mgr.y (mgr.14538) 538 : cluster 0 pgmap v842: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: audit 2024-04-02T04:57:24.264652+0000 mon.c (mon.2) 1009 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.290169+0000 osd.2 (osd.2) 254 : cluster 0 102.1c scrub starts 2024-04-02T04:57:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.293521+0000 osd.2 (osd.2) 255 : cluster 0 102.1c scrub ok 2024-04-02T04:57:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: audit 2024-04-02T04:57:24.970569+0000 mon.a (mon.0) 3028 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:25.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.977748+0000 mon.a (mon.0) 3029 : cluster 0 osdmap e564: 8 total, 8 up, 6 in 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: audit 2024-04-02T04:57:25.050569+0000 mon.c (mon.2) 1010 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[21528]: audit 2024-04-02T04:57:25.050992+0000 mon.a (mon.0) 3030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.198329+0000 mgr.y (mgr.14538) 538 : cluster 0 pgmap v842: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: audit 2024-04-02T04:57:24.264652+0000 mon.c (mon.2) 1009 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.290169+0000 osd.2 (osd.2) 254 : cluster 0 102.1c scrub starts 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.293521+0000 osd.2 (osd.2) 255 : cluster 0 102.1c scrub ok 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: audit 2024-04-02T04:57:24.970569+0000 mon.a (mon.0) 3028 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.977748+0000 mon.a (mon.0) 3029 : cluster 0 osdmap e564: 8 total, 8 up, 6 in 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: audit 2024-04-02T04:57:25.050569+0000 mon.c (mon.2) 1010 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-02T04:57:25.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:25 smithi069 bash[28904]: audit 2024-04-02T04:57:25.050992+0000 mon.a (mon.0) 3030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-02T04:57:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:25.382 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:25.409 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:25.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.198329+0000 mgr.y (mgr.14538) 538 : cluster 0 pgmap v842: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: audit 2024-04-02T04:57:24.264652+0000 mon.c (mon.2) 1009 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.290169+0000 osd.2 (osd.2) 254 : cluster 0 102.1c scrub starts 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.293521+0000 osd.2 (osd.2) 255 : cluster 0 102.1c scrub ok 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: audit 2024-04-02T04:57:24.970569+0000 mon.a (mon.0) 3028 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.977748+0000 mon.a (mon.0) 3029 : cluster 0 osdmap e564: 8 total, 8 up, 6 in 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: audit 2024-04-02T04:57:25.050569+0000 mon.c (mon.2) 1010 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-02T04:57:25.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:25 smithi098 bash[23790]: audit 2024-04-02T04:57:25.050992+0000 mon.a (mon.0) 3030 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-02T04:57:25.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:25.539 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:25.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:25.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:25.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:25.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:25.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:25.931 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:25.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:26.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:26.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:26.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:26.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:26.210 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:26.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:26.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:26.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:26.429 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.827386+0000 osd.4 (osd.4) 303 : cluster 0 40.9 scrub starts 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.829416+0000 osd.4 (osd.4) 304 : cluster 0 40.9 scrub ok 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.982758+0000 osd.7 (osd.7) 87 : cluster 0 2.b scrub starts 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: cluster 2024-04-02T04:57:24.984774+0000 osd.7 (osd.7) 88 : cluster 0 2.b scrub ok 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: audit 2024-04-02T04:57:25.266328+0000 mon.c (mon.2) 1011 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: audit 2024-04-02T04:57:25.975370+0000 mon.a (mon.0) 3031 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: cluster 2024-04-02T04:57:25.982469+0000 mon.a (mon.0) 3032 : cluster 0 osdmap e565: 8 total, 8 up, 6 in 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: audit 2024-04-02T04:57:26.026768+0000 mon.c (mon.2) 1012 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[21528]: audit 2024-04-02T04:57:26.027835+0000 mon.a (mon.0) 3033 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-02T04:57:26.430 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.827386+0000 osd.4 (osd.4) 303 : cluster 0 40.9 scrub starts 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.829416+0000 osd.4 (osd.4) 304 : cluster 0 40.9 scrub ok 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.982758+0000 osd.7 (osd.7) 87 : cluster 0 2.b scrub starts 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: cluster 2024-04-02T04:57:24.984774+0000 osd.7 (osd.7) 88 : cluster 0 2.b scrub ok 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: audit 2024-04-02T04:57:25.266328+0000 mon.c (mon.2) 1011 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: audit 2024-04-02T04:57:25.975370+0000 mon.a (mon.0) 3031 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: cluster 2024-04-02T04:57:25.982469+0000 mon.a (mon.0) 3032 : cluster 0 osdmap e565: 8 total, 8 up, 6 in 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: audit 2024-04-02T04:57:26.026768+0000 mon.c (mon.2) 1012 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-02T04:57:26.431 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:26 smithi069 bash[28904]: audit 2024-04-02T04:57:26.027835+0000 mon.a (mon.0) 3033 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-02T04:57:26.442 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:26.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.827386+0000 osd.4 (osd.4) 303 : cluster 0 40.9 scrub starts 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.829416+0000 osd.4 (osd.4) 304 : cluster 0 40.9 scrub ok 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.982758+0000 osd.7 (osd.7) 87 : cluster 0 2.b scrub starts 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: cluster 2024-04-02T04:57:24.984774+0000 osd.7 (osd.7) 88 : cluster 0 2.b scrub ok 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: audit 2024-04-02T04:57:25.266328+0000 mon.c (mon.2) 1011 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: audit 2024-04-02T04:57:25.975370+0000 mon.a (mon.0) 3031 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: cluster 2024-04-02T04:57:25.982469+0000 mon.a (mon.0) 3032 : cluster 0 osdmap e565: 8 total, 8 up, 6 in 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: audit 2024-04-02T04:57:26.026768+0000 mon.c (mon.2) 1012 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-02T04:57:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:26 smithi098 bash[23790]: audit 2024-04-02T04:57:26.027835+0000 mon.a (mon.0) 3033 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-02T04:57:26.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:26.574 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:26.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:26.705 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:26.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:26.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:26.966 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:26.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:27.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:27.095 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:27.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: cluster 2024-04-02T04:57:25.854902+0000 osd.4 (osd.4) 305 : cluster 0 40.6 scrub starts 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: cluster 2024-04-02T04:57:25.857030+0000 osd.4 (osd.4) 306 : cluster 0 40.6 scrub ok 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: cluster 2024-04-02T04:57:26.199069+0000 mgr.y (mgr.14538) 539 : cluster 0 pgmap v845: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: audit 2024-04-02T04:57:26.267773+0000 mon.c (mon.2) 1013 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: audit 2024-04-02T04:57:26.977616+0000 mon.a (mon.0) 3034 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: cluster 2024-04-02T04:57:26.985794+0000 mon.a (mon.0) 3035 : cluster 0 osdmap e566: 8 total, 8 up, 6 in 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: audit 2024-04-02T04:57:27.010932+0000 mon.c (mon.2) 1014 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:27.210 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:27 smithi098 bash[23790]: audit 2024-04-02T04:57:27.012220+0000 mon.a (mon.0) 3036 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:27.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:27.328 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: cluster 2024-04-02T04:57:25.854902+0000 osd.4 (osd.4) 305 : cluster 0 40.6 scrub starts 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: cluster 2024-04-02T04:57:25.857030+0000 osd.4 (osd.4) 306 : cluster 0 40.6 scrub ok 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: cluster 2024-04-02T04:57:26.199069+0000 mgr.y (mgr.14538) 539 : cluster 0 pgmap v845: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: audit 2024-04-02T04:57:26.267773+0000 mon.c (mon.2) 1013 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: audit 2024-04-02T04:57:26.977616+0000 mon.a (mon.0) 3034 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: cluster 2024-04-02T04:57:26.985794+0000 mon.a (mon.0) 3035 : cluster 0 osdmap e566: 8 total, 8 up, 6 in 2024-04-02T04:57:27.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: audit 2024-04-02T04:57:27.010932+0000 mon.c (mon.2) 1014 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[28904]: audit 2024-04-02T04:57:27.012220+0000 mon.a (mon.0) 3036 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: cluster 2024-04-02T04:57:25.854902+0000 osd.4 (osd.4) 305 : cluster 0 40.6 scrub starts 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: cluster 2024-04-02T04:57:25.857030+0000 osd.4 (osd.4) 306 : cluster 0 40.6 scrub ok 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: cluster 2024-04-02T04:57:26.199069+0000 mgr.y (mgr.14538) 539 : cluster 0 pgmap v845: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 449 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: audit 2024-04-02T04:57:26.267773+0000 mon.c (mon.2) 1013 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: audit 2024-04-02T04:57:26.977616+0000 mon.a (mon.0) 3034 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: cluster 2024-04-02T04:57:26.985794+0000 mon.a (mon.0) 3035 : cluster 0 osdmap e566: 8 total, 8 up, 6 in 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: audit 2024-04-02T04:57:27.010932+0000 mon.c (mon.2) 1014 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:27.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:27 smithi069 bash[21528]: audit 2024-04-02T04:57:27.012220+0000 mon.a (mon.0) 3036 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:27.354 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:27.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:27.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:27.613 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:27.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:27.742 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:27.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:27.873 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:27.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:27.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:28.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:28.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:28.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: cluster 2024-04-02T04:57:26.803931+0000 osd.6 (osd.6) 251 : cluster 0 43.1b scrub starts 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: cluster 2024-04-02T04:57:26.806660+0000 osd.6 (osd.6) 252 : cluster 0 43.1b scrub ok 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: cluster 2024-04-02T04:57:27.149491+0000 osd.3 (osd.3) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: cluster 2024-04-02T04:57:27.149738+0000 osd.3 (osd.3) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: audit 2024-04-02T04:57:27.268894+0000 mon.c (mon.2) 1015 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: audit 2024-04-02T04:57:27.980628+0000 mon.a (mon.0) 3037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: cluster 2024-04-02T04:57:27.987438+0000 mon.a (mon.0) 3038 : cluster 0 osdmap e567: 8 total, 8 up, 6 in 2024-04-02T04:57:28.255 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: audit 2024-04-02T04:57:28.057402+0000 mon.c (mon.2) 1016 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:28.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: audit 2024-04-02T04:57:28.057900+0000 mon.a (mon.0) 3039 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:28.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: audit 2024-04-02T04:57:28.058806+0000 mon.c (mon.2) 1017 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-81"}]: dispatch 2024-04-02T04:57:28.256 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:28 smithi098 bash[23790]: audit 2024-04-02T04:57:28.059233+0000 mon.a (mon.0) 3040 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-81"}]: dispatch 2024-04-02T04:57:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:28.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:28.383 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: cluster 2024-04-02T04:57:26.803931+0000 osd.6 (osd.6) 251 : cluster 0 43.1b scrub starts 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: cluster 2024-04-02T04:57:26.806660+0000 osd.6 (osd.6) 252 : cluster 0 43.1b scrub ok 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: cluster 2024-04-02T04:57:27.149491+0000 osd.3 (osd.3) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: cluster 2024-04-02T04:57:27.149738+0000 osd.3 (osd.3) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: audit 2024-04-02T04:57:27.268894+0000 mon.c (mon.2) 1015 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: audit 2024-04-02T04:57:27.980628+0000 mon.a (mon.0) 3037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: cluster 2024-04-02T04:57:27.987438+0000 mon.a (mon.0) 3038 : cluster 0 osdmap e567: 8 total, 8 up, 6 in 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: audit 2024-04-02T04:57:28.057402+0000 mon.c (mon.2) 1016 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: audit 2024-04-02T04:57:28.057900+0000 mon.a (mon.0) 3039 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: audit 2024-04-02T04:57:28.058806+0000 mon.c (mon.2) 1017 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-81"}]: dispatch 2024-04-02T04:57:28.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[28904]: audit 2024-04-02T04:57:28.059233+0000 mon.a (mon.0) 3040 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-81"}]: dispatch 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: cluster 2024-04-02T04:57:26.803931+0000 osd.6 (osd.6) 251 : cluster 0 43.1b scrub starts 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: cluster 2024-04-02T04:57:26.806660+0000 osd.6 (osd.6) 252 : cluster 0 43.1b scrub ok 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: cluster 2024-04-02T04:57:27.149491+0000 osd.3 (osd.3) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: cluster 2024-04-02T04:57:27.149738+0000 osd.3 (osd.3) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: audit 2024-04-02T04:57:27.268894+0000 mon.c (mon.2) 1015 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: audit 2024-04-02T04:57:27.980628+0000 mon.a (mon.0) 3037 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: cluster 2024-04-02T04:57:27.987438+0000 mon.a (mon.0) 3038 : cluster 0 osdmap e567: 8 total, 8 up, 6 in 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: audit 2024-04-02T04:57:28.057402+0000 mon.c (mon.2) 1016 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: audit 2024-04-02T04:57:28.057900+0000 mon.a (mon.0) 3039 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: audit 2024-04-02T04:57:28.058806+0000 mon.c (mon.2) 1017 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-81"}]: dispatch 2024-04-02T04:57:28.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21528]: audit 2024-04-02T04:57:28.059233+0000 mon.a (mon.0) 3040 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-81"}]: dispatch 2024-04-02T04:57:28.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:28.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:28.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:28.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:28.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:28.776 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:57:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:57:28.793 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:28.881 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:28.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:28.926 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:28.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:29.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:29.052 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:29.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:29.181 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:29.274 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:29.274 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:29.284 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:29.304 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:29.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:29.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:29.565 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:29.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:29.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: cluster 2024-04-02T04:57:27.784496+0000 osd.6 (osd.6) 253 : cluster 0 102.16 scrub starts 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: cluster 2024-04-02T04:57:27.786168+0000 osd.6 (osd.6) 254 : cluster 0 102.16 scrub ok 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.201163+0000 mgr.y (mgr.14538) 540 : cluster 0 pgmap v848: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 10 KiB/s wr, 25 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.233924+0000 osd.0 (osd.0) 295 : cluster 0 102.a scrub starts 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.236068+0000 osd.0 (osd.0) 296 : cluster 0 102.a scrub ok 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: audit 2024-04-02T04:57:28.270030+0000 mon.c (mon.2) 1018 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.990439+0000 mon.a (mon.0) 3041 : cluster 0 osdmap e568: 8 total, 8 up, 6 in 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: cluster 2024-04-02T04:57:27.784496+0000 osd.6 (osd.6) 253 : cluster 0 102.16 scrub starts 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: cluster 2024-04-02T04:57:27.786168+0000 osd.6 (osd.6) 254 : cluster 0 102.16 scrub ok 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.201163+0000 mgr.y (mgr.14538) 540 : cluster 0 pgmap v848: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 10 KiB/s wr, 25 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:29.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.233924+0000 osd.0 (osd.0) 295 : cluster 0 102.a scrub starts 2024-04-02T04:57:29.681 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.236068+0000 osd.0 (osd.0) 296 : cluster 0 102.a scrub ok 2024-04-02T04:57:29.681 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: audit 2024-04-02T04:57:28.270030+0000 mon.c (mon.2) 1018 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:29.681 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:29 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.990439+0000 mon.a (mon.0) 3041 : cluster 0 osdmap e568: 8 total, 8 up, 6 in 2024-04-02T04:57:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: cluster 2024-04-02T04:57:27.784496+0000 osd.6 (osd.6) 253 : cluster 0 102.16 scrub starts 2024-04-02T04:57:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: cluster 2024-04-02T04:57:27.786168+0000 osd.6 (osd.6) 254 : cluster 0 102.16 scrub ok 2024-04-02T04:57:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.201163+0000 mgr.y (mgr.14538) 540 : cluster 0 pgmap v848: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 10 KiB/s wr, 25 op/s; 2/1269 objects misplaced (0.158%) 2024-04-02T04:57:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.233924+0000 osd.0 (osd.0) 295 : cluster 0 102.a scrub starts 2024-04-02T04:57:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.236068+0000 osd.0 (osd.0) 296 : cluster 0 102.a scrub ok 2024-04-02T04:57:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: audit 2024-04-02T04:57:28.270030+0000 mon.c (mon.2) 1018 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:29.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:29 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.990439+0000 mon.a (mon.0) 3041 : cluster 0 osdmap e568: 8 total, 8 up, 6 in 2024-04-02T04:57:29.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:29.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:29.957 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:29.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:30.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:30.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:30.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:30.345 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:30.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:30.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.217737+0000 osd.2 (osd.2) 256 : cluster 0 102.1a scrub starts 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.222288+0000 osd.2 (osd.2) 257 : cluster 0 102.1a scrub ok 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.807366+0000 osd.6 (osd.6) 255 : cluster 0 43.11 scrub starts 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:28.809585+0000 osd.6 (osd.6) 256 : cluster 0 43.11 scrub ok 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:29.177488+0000 osd.2 (osd.2) 258 : cluster 0 2.e scrub starts 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:29.179752+0000 osd.2 (osd.2) 259 : cluster 0 2.e scrub ok 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:29.239469+0000 osd.0 (osd.0) 297 : cluster 0 102.7 scrub starts 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:29.243879+0000 osd.0 (osd.0) 298 : cluster 0 102.7 scrub ok 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: audit 2024-04-02T04:57:29.271473+0000 mon.c (mon.2) 1019 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:30.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:29.959879+0000 mon.a (mon.0) 3042 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:30.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: cluster 2024-04-02T04:57:29.969054+0000 mon.a (mon.0) 3043 : cluster 0 osdmap e569: 8 total, 8 up, 6 in 2024-04-02T04:57:30.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: audit 2024-04-02T04:57:29.970439+0000 mon.c (mon.2) 1020 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:30.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: audit 2024-04-02T04:57:29.970924+0000 mon.a (mon.0) 3044 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:30.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:30 smithi098 bash[23790]: audit 2024-04-02T04:57:30.272879+0000 mon.c (mon.2) 1021 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:30.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:30.735 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.217737+0000 osd.2 (osd.2) 256 : cluster 0 102.1a scrub starts 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.222288+0000 osd.2 (osd.2) 257 : cluster 0 102.1a scrub ok 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.807366+0000 osd.6 (osd.6) 255 : cluster 0 43.11 scrub starts 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:28.809585+0000 osd.6 (osd.6) 256 : cluster 0 43.11 scrub ok 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:29.177488+0000 osd.2 (osd.2) 258 : cluster 0 2.e scrub starts 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:29.179752+0000 osd.2 (osd.2) 259 : cluster 0 2.e scrub ok 2024-04-02T04:57:30.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:29.239469+0000 osd.0 (osd.0) 297 : cluster 0 102.7 scrub starts 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:29.243879+0000 osd.0 (osd.0) 298 : cluster 0 102.7 scrub ok 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: audit 2024-04-02T04:57:29.271473+0000 mon.c (mon.2) 1019 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:29.959879+0000 mon.a (mon.0) 3042 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: cluster 2024-04-02T04:57:29.969054+0000 mon.a (mon.0) 3043 : cluster 0 osdmap e569: 8 total, 8 up, 6 in 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: audit 2024-04-02T04:57:29.970439+0000 mon.c (mon.2) 1020 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: audit 2024-04-02T04:57:29.970924+0000 mon.a (mon.0) 3044 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[21528]: audit 2024-04-02T04:57:30.272879+0000 mon.c (mon.2) 1021 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.217737+0000 osd.2 (osd.2) 256 : cluster 0 102.1a scrub starts 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.222288+0000 osd.2 (osd.2) 257 : cluster 0 102.1a scrub ok 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.807366+0000 osd.6 (osd.6) 255 : cluster 0 43.11 scrub starts 2024-04-02T04:57:30.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:28.809585+0000 osd.6 (osd.6) 256 : cluster 0 43.11 scrub ok 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:29.177488+0000 osd.2 (osd.2) 258 : cluster 0 2.e scrub starts 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:29.179752+0000 osd.2 (osd.2) 259 : cluster 0 2.e scrub ok 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:29.239469+0000 osd.0 (osd.0) 297 : cluster 0 102.7 scrub starts 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:29.243879+0000 osd.0 (osd.0) 298 : cluster 0 102.7 scrub ok 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: audit 2024-04-02T04:57:29.271473+0000 mon.c (mon.2) 1019 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:29.959879+0000 mon.a (mon.0) 3042 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: cluster 2024-04-02T04:57:29.969054+0000 mon.a (mon.0) 3043 : cluster 0 osdmap e569: 8 total, 8 up, 6 in 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: audit 2024-04-02T04:57:29.970439+0000 mon.c (mon.2) 1020 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: audit 2024-04-02T04:57:29.970924+0000 mon.a (mon.0) 3044 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:30.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:30 smithi069 bash[28904]: audit 2024-04-02T04:57:30.272879+0000 mon.c (mon.2) 1021 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:30.838 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:30.866 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:30.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:30.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:31.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:31.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:31.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:31.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:31.400 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 -- 172.21.15.69:0/1026699249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc101b60 msgr2=0x7fa3cc10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 --2- 172.21.15.69:0/1026699249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc101b60 0x7fa3cc10dfd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009f90 tx=0x7fa3c400b330 comp rx=0 tx=0).stop 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 -- 172.21.15.69:0/1026699249 shutdown_connections 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 --2- 172.21.15.69:0/1026699249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc101b60 0x7fa3cc10dfd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 --2- 172.21.15.69:0/1026699249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc107220 0x7fa3cc107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 --2- 172.21.15.69:0/1026699249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc1011a0 0x7fa3cc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 -- 172.21.15.69:0/1026699249 >> 172.21.15.69:0/1026699249 conn(0x7fa3cc0771e0 msgr2=0x7fa3cc0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 -- 172.21.15.69:0/1026699249 shutdown_connections 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 -- 172.21.15.69:0/1026699249 wait complete. 2024-04-02T04:57:31.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 librados: starting msgr at 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 librados: starting objecter 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 Processor -- start 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 -- start start 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 librados: setting wanted keys 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.438+0000 7fa3d2521700 1 librados: calling monclient init 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc1011a0 0x7fa3cc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc101b60 0x7fa3cc198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc107220 0x7fa3cc19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc110970 con 0x7fa3cc107220 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1107f0 con 0x7fa3cc1011a0 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc110af0 con 0x7fa3cc101b60 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc101b60 0x7fa3cc198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d0abe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc107220 0x7fa3cc19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc101b60 0x7fa3cc198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41582/0 (socket says 172.21.15.69:41582) 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 -- 172.21.15.69:0/2704904123 learned_addr learned my addr 172.21.15.69:0/2704904123 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc1011a0 0x7fa3cc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc1011a0 msgr2=0x7fa3cc197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3cc1011a0 0x7fa3cc197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc107220 msgr2=0x7fa3cc19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc107220 0x7fa3cc19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc19cfa0 con 0x7fa3cc101b60 2024-04-02T04:57:31.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3cc107220 0x7fa3cc19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:57:31.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc101b60 0x7fa3cc198520 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000ec10 tx=0x7fa3c000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 10 client.?.objecter ms_handle_connect 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000cc90 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa3c000ce30 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc19d290 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0010430 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 10 client.26717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3cc1a4b10 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 librados: init done 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa3cc04ff00 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa3c0003ed0 con 0x7fa3cc101b60 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa3b406f5e0 0x7fa3b4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(570..570 src has 74..570) ==== 10584+0+0 (secure 0 0 0) 0x7fa3c008f260 con 0x7fa3cc101b60 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa3b406f5e0 0x7fa3b4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 osd_map(570..570 src has 74..570) 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 3 client.26717.objecter handle_osd_map got epochs [570,570] > 0 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 3 client.26717.objecter handle_osd_map decoding full epoch 570 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 20 client.26717.objecter dump_active .. 0 homeless 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v570) ==== 54+0+16 (secure 0 0 0) 0x7fa3c008f6b0 con 0x7fa3cc101b60 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3b406f5e0 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa3b406f5e0 0x7fa3b4071aa0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009a10 tx=0x7fa3bc009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 10 client.26717.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 1 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa3cc062e50 0x7fa3cc063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 1 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 0 1 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 4 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 10 client.26717.objecter _send_command 1 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.442+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1a7fc0 con 0x7fa3cc062e50 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa3cc062e50 0x7fa3cc063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa3cc062e50 0x7fa3cc063230 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc062e50 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1a7fc0 con 0x7fa3cc062e50 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 1: 0 ) 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 1 = system:0 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 4 2024-04-02T04:57:31.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.446+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 0 1 2024-04-02T04:57:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 10 client.26717.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 2 2024-04-02T04:57:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 2 2024-04-02T04:57:31.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1992f0 osd=0 3 2024-04-02T04:57:31.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 0 2 2024-04-02T04:57:31.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 4 2024-04-02T04:57:31.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 10 client.26717.objecter _send_command 2 2024-04-02T04:57:31.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc0628d0 con 0x7fa3cc062e50 2024-04-02T04:57:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc0628d0 con 0x7fa3cc062e50 2024-04-02T04:57:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 4 2024-04-02T04:57:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 0 2 2024-04-02T04:57:31.468 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:57:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 10 client.26717.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 3 2024-04-02T04:57:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa3cc10ac00 0x7fa3cc02c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 1 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 4 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 10 client.26717.objecter _send_command 3 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1ab920 con 0x7fa3cc10ac00 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa3cc10ac00 0x7fa3cc02c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc10ac00 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.466+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa3cc10ac00 0x7fa3cc02c9b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.470+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1ab920 con 0x7fa3cc10ac00 2024-04-02T04:57:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.470+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 3: 0 ) 2024-04-02T04:57:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.470+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 3 = system:0 2024-04-02T04:57:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.470+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 4 2024-04-02T04:57:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.470+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 1 3 2024-04-02T04:57:31.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 10 client.26717.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 4 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 4 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc062cd0 osd=1 3 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 1 4 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 4 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 10 client.26717.objecter _send_command 4 2024-04-02T04:57:31.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc10a170 con 0x7fa3cc10ac00 2024-04-02T04:57:31.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc10a170 con 0x7fa3cc10ac00 2024-04-02T04:57:31.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 4 2024-04-02T04:57:31.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.486+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 1 4 2024-04-02T04:57:31.488 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:57:31.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 10 client.26717.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 5 2024-04-02T04:57:31.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa3cc1aa180 0x7fa3cc1aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 5 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 2 5 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 4 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 10 client.26717.objecter _send_command 5 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1a7c90 con 0x7fa3cc1aa180 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa3cc1aa180 0x7fa3cc1aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc1aa180 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa3cc1aa180 0x7fa3cc1aa5a0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1a7c90 con 0x7fa3cc1aa180 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 5: 0 ) 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 5 = system:0 2024-04-02T04:57:31.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 4 2024-04-02T04:57:31.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.490+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 2 5 2024-04-02T04:57:31.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 10 client.26717.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 6 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 6 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1a7610 osd=2 3 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 2 6 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 4 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 10 client.26717.objecter _send_command 6 2024-04-02T04:57:31.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.506+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc1a7090 con 0x7fa3cc1aa180 2024-04-02T04:57:31.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc1a7090 con 0x7fa3cc1aa180 2024-04-02T04:57:31.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 4 2024-04-02T04:57:31.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 2 6 2024-04-02T04:57:31.509 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 10 client.26717.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 7 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa3cc102d50 0x7fa3cc103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 7 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 3 7 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 4 2024-04-02T04:57:31.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 10 client.26717.objecter _send_command 7 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1b5080 con 0x7fa3cc102d50 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa3cc102d50 0x7fa3cc103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa3cc102d50 0x7fa3cc103190 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc102d50 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1b5080 con 0x7fa3cc102d50 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 7: 0 ) 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 7 = system:0 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 4 2024-04-02T04:57:31.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.510+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 3 7 2024-04-02T04:57:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:31.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 10 client.26717.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 8 2024-04-02T04:57:31.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 8 2024-04-02T04:57:31.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1a7090 osd=3 3 2024-04-02T04:57:31.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 3 8 2024-04-02T04:57:31.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 4 2024-04-02T04:57:31.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 10 client.26717.objecter _send_command 8 2024-04-02T04:57:31.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc1aad30 con 0x7fa3cc102d50 2024-04-02T04:57:31.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc1aad30 con 0x7fa3cc102d50 2024-04-02T04:57:31.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 4 2024-04-02T04:57:31.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.530+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 3 8 2024-04-02T04:57:31.533 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:57:31.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 10 client.26717.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 9 2024-04-02T04:57:31.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa3cc1aaea0 0x7fa3cc02d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 9 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 4 9 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 4 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 10 client.26717.objecter _send_command 9 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc02dc00 con 0x7fa3cc1aaea0 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa3cc1aaea0 0x7fa3cc02d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc1aaea0 2024-04-02T04:57:31.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa3cc1aaea0 0x7fa3cc02d6c0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc02dc00 con 0x7fa3cc1aaea0 2024-04-02T04:57:31.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 9: 0 ) 2024-04-02T04:57:31.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 9 = system:0 2024-04-02T04:57:31.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 4 2024-04-02T04:57:31.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.534+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 4 9 2024-04-02T04:57:31.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 10 client.26717.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 10 2024-04-02T04:57:31.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 10 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1aad30 osd=4 3 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 4 10 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 4 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 10 client.26717.objecter _send_command 10 2024-04-02T04:57:31.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.554+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc02dc00 con 0x7fa3cc1aaea0 2024-04-02T04:57:31.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc02dc00 con 0x7fa3cc1aaea0 2024-04-02T04:57:31.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 4 2024-04-02T04:57:31.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 4 10 2024-04-02T04:57:31.558 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 10 client.26717.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 11 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa3cc1b6420 0x7fa3cc1b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 11 2024-04-02T04:57:31.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 5 11 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 4 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 10 client.26717.objecter _send_command 11 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1bb670 con 0x7fa3cc1b6420 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa3cc1b6420 0x7fa3cc1b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc1b6420 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.558+0000 7fa3cb7fe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa3cc1b6420 0x7fa3cc1b8810 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.562+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1bb670 con 0x7fa3cc1b6420 2024-04-02T04:57:31.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.562+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 11: 0 ) 2024-04-02T04:57:31.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.562+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 11 = system:0 2024-04-02T04:57:31.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.562+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 4 2024-04-02T04:57:31.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.562+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 5 11 2024-04-02T04:57:31.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.202138+0000 mgr.y (mgr.14538) 541 : cluster 0 pgmap v851: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 7.7 KiB/s rd, 9.0 KiB/s wr, 29 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:31.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.216318+0000 osd.0 (osd.0) 299 : cluster 0 3.1 scrub starts 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.220228+0000 osd.0 (osd.0) 300 : cluster 0 3.1 scrub ok 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.749121+0000 osd.6 (osd.6) 257 : cluster 0 102.14 scrub starts 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.752664+0000 osd.6 (osd.6) 258 : cluster 0 102.14 scrub ok 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.842912+0000 osd.7 (osd.7) 89 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.843229+0000 osd.7 (osd.7) 90 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: audit 2024-04-02T04:57:30.974184+0000 mon.a (mon.0) 3045 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: cluster 2024-04-02T04:57:30.979510+0000 mon.a (mon.0) 3046 : cluster 0 osdmap e570: 8 total, 8 up, 6 in 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: audit 2024-04-02T04:57:30.982032+0000 mon.c (mon.2) 1022 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: audit 2024-04-02T04:57:30.995491+0000 mon.c (mon.2) 1023 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: audit 2024-04-02T04:57:30.997900+0000 mon.a (mon.0) 3047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:31.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: audit 2024-04-02T04:57:31.274483+0000 mon.c (mon.2) 1024 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[21528]: audit 2024-04-02T04:57:31.446106+0000 mon.c (mon.2) 1025 : audit 0 from='client.? 172.21.15.69:0/2704904123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.202138+0000 mgr.y (mgr.14538) 541 : cluster 0 pgmap v851: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 7.7 KiB/s rd, 9.0 KiB/s wr, 29 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.216318+0000 osd.0 (osd.0) 299 : cluster 0 3.1 scrub starts 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.220228+0000 osd.0 (osd.0) 300 : cluster 0 3.1 scrub ok 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.749121+0000 osd.6 (osd.6) 257 : cluster 0 102.14 scrub starts 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.752664+0000 osd.6 (osd.6) 258 : cluster 0 102.14 scrub ok 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.842912+0000 osd.7 (osd.7) 89 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.843229+0000 osd.7 (osd.7) 90 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: audit 2024-04-02T04:57:30.974184+0000 mon.a (mon.0) 3045 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: cluster 2024-04-02T04:57:30.979510+0000 mon.a (mon.0) 3046 : cluster 0 osdmap e570: 8 total, 8 up, 6 in 2024-04-02T04:57:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: audit 2024-04-02T04:57:30.982032+0000 mon.c (mon.2) 1022 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: audit 2024-04-02T04:57:30.995491+0000 mon.c (mon.2) 1023 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: audit 2024-04-02T04:57:30.997900+0000 mon.a (mon.0) 3047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: audit 2024-04-02T04:57:31.274483+0000 mon.c (mon.2) 1024 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:31 smithi069 bash[28904]: audit 2024-04-02T04:57:31.446106+0000 mon.c (mon.2) 1025 : audit 0 from='client.? 172.21.15.69:0/2704904123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 10 client.26717.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 12 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 12 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc02dc00 osd=5 3 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 5 12 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 4 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 10 client.26717.objecter _send_command 12 2024-04-02T04:57:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.578+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc1b8d50 con 0x7fa3cc1b6420 2024-04-02T04:57:31.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc1b8d50 con 0x7fa3cc1b6420 2024-04-02T04:57:31.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 4 2024-04-02T04:57:31.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 5 12 2024-04-02T04:57:31.585 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:57:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 10 client.26717.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 13 2024-04-02T04:57:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa3cc1b93d0 0x7fa3cc1bb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 13 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 6 13 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 4 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 10 client.26717.objecter _send_command 13 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.582+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1be8a0 con 0x7fa3cc1b93d0 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa3cc1b93d0 0x7fa3cc1bb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc1b93d0 2024-04-02T04:57:31.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3d0abe700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa3cc1b93d0 0x7fa3cc1bb7e0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1be8a0 con 0x7fa3cc1b93d0 2024-04-02T04:57:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 13: 0 ) 2024-04-02T04:57:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 13 = system:0 2024-04-02T04:57:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 4 2024-04-02T04:57:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.586+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 6 13 2024-04-02T04:57:31.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 10 client.26717.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 14 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 14 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1b8d50 osd=6 3 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 6 14 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 4 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 10 client.26717.objecter _send_command 14 2024-04-02T04:57:31.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.606+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc1bd630 con 0x7fa3cc1b93d0 2024-04-02T04:57:31.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc1bd630 con 0x7fa3cc1b93d0 2024-04-02T04:57:31.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 4 2024-04-02T04:57:31.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 6 14 2024-04-02T04:57:31.611 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 10 client.26717.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 15 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa3cc1bdd50 0x7fa3cc1be190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 15 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 7 15 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 4 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 10 client.26717.objecter _send_command 15 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa3cc1c1c70 con 0x7fa3cc1bdd50 2024-04-02T04:57:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa3cc1bdd50 0x7fa3cc1be190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:31.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3c97fa700 10 client.26717.objecter ms_handle_connect 0x7fa3cc1bdd50 2024-04-02T04:57:31.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.610+0000 7fa3cbfff700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa3cc1bdd50 0x7fa3cc1be190 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:31.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.614+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3cc1c1c70 con 0x7fa3cc1bdd50 2024-04-02T04:57:31.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.614+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 15: 0 ) 2024-04-02T04:57:31.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.614+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 15 = system:0 2024-04-02T04:57:31.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.614+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 4 2024-04-02T04:57:31.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.614+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 7 15 2024-04-02T04:57:31.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:31.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 10 client.26717.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:57:31.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign -1 16 2024-04-02T04:57:31.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 20 client.26717.objecter _get_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_remove -1 16 2024-04-02T04:57:31.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 20 client.26717.objecter get_session s=0x7fa3cc1aaba0 osd=7 3 2024-04-02T04:57:31.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 15 client.26717.objecter _session_command_op_assign 7 16 2024-04-02T04:57:31.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 4 2024-04-02T04:57:31.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 10 client.26717.objecter _send_command 16 2024-04-02T04:57:31.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.630+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa3cc1bbe60 con 0x7fa3cc1bdd50 2024-04-02T04:57:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3c97fa700 1 -- 172.21.15.69:0/2704904123 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa3cc1bbe60 con 0x7fa3cc1bdd50 2024-04-02T04:57:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3c97fa700 10 client.26717.objecter ms_dispatch 0x7fa3cc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3c97fa700 10 client.26717.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3c97fa700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 4 2024-04-02T04:57:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3c97fa700 15 client.26717.objecter _session_command_op_remove 7 16 2024-04-02T04:57:31.637 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:57:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 librados: watch_flush enter 2024-04-02T04:57:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 librados: watch_flush exit 2024-04-02T04:57:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.0 2024-04-02T04:57:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa3cc062e50 msgr2=0x7fa3cc063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa3cc062e50 0x7fa3cc063230 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1992f0 osd=0 1 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.1 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa3cc10ac00 msgr2=0x7fa3cc02c9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa3cc10ac00 0x7fa3cc02c9b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc062cd0 osd=1 1 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.2 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa3cc1aa180 msgr2=0x7fa3cc1aa5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa3cc1aa180 0x7fa3cc1aa5a0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7610 osd=2 1 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.3 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa3cc102d50 msgr2=0x7fa3cc103190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa3cc102d50 0x7fa3cc103190 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1a7090 osd=3 1 2024-04-02T04:57:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.4 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa3cc1aaea0 msgr2=0x7fa3cc02d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa3cc1aaea0 0x7fa3cc02d6c0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aad30 osd=4 1 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.5 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa3cc1b6420 msgr2=0x7fa3cc1b8810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa3cc1b6420 0x7fa3cc1b8810 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc02dc00 osd=5 1 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.6 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa3cc1b93d0 msgr2=0x7fa3cc1bb7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa3cc1b93d0 0x7fa3cc1bb7e0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1b8d50 osd=6 1 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter close_session for osd.7 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa3cc1bdd50 msgr2=0x7fa3cc1be190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa3cc1bdd50 0x7fa3cc1be190 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter put_session s=0x7fa3cc1aaba0 osd=7 1 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 20 client.26717.objecter shutdown clearing up homeless session... 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 10 client.26717.objecter successfully canceled tick 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa3b406f5e0 msgr2=0x7fa3b4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa3b406f5e0 0x7fa3b4071aa0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009a10 tx=0x7fa3bc009190 comp rx=0 tx=0).stop 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d0abe700 1 -- 172.21.15.69:0/2704904123 reap_dead start 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d0abe700 1 -- 172.21.15.69:0/2704904123 reap_dead start 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc101b60 msgr2=0x7fa3cc198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 --2- 172.21.15.69:0/2704904123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3cc101b60 0x7fa3cc198520 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000ec10 tx=0x7fa3c000ef20 comp rx=0 tx=0).stop 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d0abe700 1 -- 172.21.15.69:0/2704904123 reap_dead start 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d0abe700 1 -- 172.21.15.69:0/2704904123 reap_dead start 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d0abe700 1 -- 172.21.15.69:0/2704904123 reap_dead start 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 shutdown_connections 2024-04-02T04:57:31.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 >> 172.21.15.69:0/2704904123 conn(0x7fa3cc0771e0 msgr2=0x7fa3cc0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 shutdown_connections 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 -- 172.21.15.69:0/2704904123 wait complete. 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:31.638+0000 7fa3d2521700 1 librados: shutdown 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:57:31.642 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:57:31.643 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.202138+0000 mgr.y (mgr.14538) 541 : cluster 0 pgmap v851: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 7.7 KiB/s rd, 9.0 KiB/s wr, 29 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.216318+0000 osd.0 (osd.0) 299 : cluster 0 3.1 scrub starts 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.220228+0000 osd.0 (osd.0) 300 : cluster 0 3.1 scrub ok 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.749121+0000 osd.6 (osd.6) 257 : cluster 0 102.14 scrub starts 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.752664+0000 osd.6 (osd.6) 258 : cluster 0 102.14 scrub ok 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.842912+0000 osd.7 (osd.7) 89 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.843229+0000 osd.7 (osd.7) 90 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: audit 2024-04-02T04:57:30.974184+0000 mon.a (mon.0) 3045 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:31.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: cluster 2024-04-02T04:57:30.979510+0000 mon.a (mon.0) 3046 : cluster 0 osdmap e570: 8 total, 8 up, 6 in 2024-04-02T04:57:31.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: audit 2024-04-02T04:57:30.982032+0000 mon.c (mon.2) 1022 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:31.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: audit 2024-04-02T04:57:30.995491+0000 mon.c (mon.2) 1023 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:31.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: audit 2024-04-02T04:57:30.997900+0000 mon.a (mon.0) 3047 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:31.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: audit 2024-04-02T04:57:31.274483+0000 mon.c (mon.2) 1024 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:31.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:31 smithi098 bash[23790]: audit 2024-04-02T04:57:31.446106+0000 mon.c (mon.2) 1025 : audit 0 from='client.? 172.21.15.69:0/2704904123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:31.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:31.779 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:31.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:31.910 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:31.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:32.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.034 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:32.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.162 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:32.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:32.293 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:32.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.427 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:32.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:32.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:32.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:32.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.817 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:32.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.941 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:32.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:33.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:33.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: cluster 2024-04-02T04:57:31.945901+0000 osd.4 (osd.4) 307 : cluster 0 3.1c scrub starts 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: cluster 2024-04-02T04:57:31.948723+0000 osd.4 (osd.4) 308 : cluster 0 3.1c scrub ok 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: audit 2024-04-02T04:57:31.977310+0000 mon.a (mon.0) 3048 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: cluster 2024-04-02T04:57:31.985321+0000 mon.a (mon.0) 3049 : cluster 0 osdmap e571: 8 total, 8 up, 6 in 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: audit 2024-04-02T04:57:31.988355+0000 mon.c (mon.2) 1026 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: audit 2024-04-02T04:57:31.989035+0000 mon.a (mon.0) 3050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.202801+0000 mgr.y (mgr.14538) 542 : cluster 0 pgmap v854: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 3 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:33.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[21528]: audit 2024-04-02T04:57:32.276520+0000 mon.c (mon.2) 1027 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: cluster 2024-04-02T04:57:31.945901+0000 osd.4 (osd.4) 307 : cluster 0 3.1c scrub starts 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: cluster 2024-04-02T04:57:31.948723+0000 osd.4 (osd.4) 308 : cluster 0 3.1c scrub ok 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: audit 2024-04-02T04:57:31.977310+0000 mon.a (mon.0) 3048 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: cluster 2024-04-02T04:57:31.985321+0000 mon.a (mon.0) 3049 : cluster 0 osdmap e571: 8 total, 8 up, 6 in 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: audit 2024-04-02T04:57:31.988355+0000 mon.c (mon.2) 1026 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: audit 2024-04-02T04:57:31.989035+0000 mon.a (mon.0) 3050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.202801+0000 mgr.y (mgr.14538) 542 : cluster 0 pgmap v854: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 3 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:33.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:33 smithi069 bash[28904]: audit 2024-04-02T04:57:32.276520+0000 mon.c (mon.2) 1027 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:33.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:33.326 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: cluster 2024-04-02T04:57:31.945901+0000 osd.4 (osd.4) 307 : cluster 0 3.1c scrub starts 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: cluster 2024-04-02T04:57:31.948723+0000 osd.4 (osd.4) 308 : cluster 0 3.1c scrub ok 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: audit 2024-04-02T04:57:31.977310+0000 mon.a (mon.0) 3048 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: cluster 2024-04-02T04:57:31.985321+0000 mon.a (mon.0) 3049 : cluster 0 osdmap e571: 8 total, 8 up, 6 in 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: audit 2024-04-02T04:57:31.988355+0000 mon.c (mon.2) 1026 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: audit 2024-04-02T04:57:31.989035+0000 mon.a (mon.0) 3050 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:33.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.202801+0000 mgr.y (mgr.14538) 542 : cluster 0 pgmap v854: 313 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 280 active+clean; 8.4 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 3 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:33.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:32 smithi098 bash[23790]: audit 2024-04-02T04:57:32.276520+0000 mon.c (mon.2) 1027 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:33.429 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:33.457 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:33.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:33.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.702 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:57:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:57:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:33.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.849 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:33.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.979 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:33.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:34.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:34.094 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.167985+0000 osd.2 (osd.2) 260 : cluster 0 102.17 scrub starts 2024-04-02T04:57:34.094 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.219334+0000 osd.2 (osd.2) 261 : cluster 0 102.17 scrub ok 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.701674+0000 osd.6 (osd.6) 259 : cluster 0 10.0 scrub starts 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.703370+0000 osd.6 (osd.6) 260 : cluster 0 10.0 scrub ok 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.965032+0000 osd.1 (osd.1) 309 : cluster 0 4.7 scrub starts 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.966981+0000 osd.1 (osd.1) 310 : cluster 0 4.7 scrub ok 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: audit 2024-04-02T04:57:32.985376+0000 mon.a (mon.0) 3051 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: cluster 2024-04-02T04:57:32.991489+0000 mon.a (mon.0) 3052 : cluster 0 osdmap e572: 8 total, 8 up, 6 in 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: audit 2024-04-02T04:57:32.994037+0000 mon.c (mon.2) 1028 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: audit 2024-04-02T04:57:32.994656+0000 mon.a (mon.0) 3053 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: audit 2024-04-02T04:57:33.278289+0000 mon.c (mon.2) 1029 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: audit 2024-04-02T04:57:33.362644+0000 mon.b (mon.1) 275 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:34.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:34 smithi098 bash[23790]: audit 2024-04-02T04:57:33.428634+0000 mgr.y (mgr.14538) 543 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:34.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:34.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:34.240 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:34.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.167985+0000 osd.2 (osd.2) 260 : cluster 0 102.17 scrub starts 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.219334+0000 osd.2 (osd.2) 261 : cluster 0 102.17 scrub ok 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.701674+0000 osd.6 (osd.6) 259 : cluster 0 10.0 scrub starts 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.703370+0000 osd.6 (osd.6) 260 : cluster 0 10.0 scrub ok 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.965032+0000 osd.1 (osd.1) 309 : cluster 0 4.7 scrub starts 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.966981+0000 osd.1 (osd.1) 310 : cluster 0 4.7 scrub ok 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: audit 2024-04-02T04:57:32.985376+0000 mon.a (mon.0) 3051 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: cluster 2024-04-02T04:57:32.991489+0000 mon.a (mon.0) 3052 : cluster 0 osdmap e572: 8 total, 8 up, 6 in 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: audit 2024-04-02T04:57:32.994037+0000 mon.c (mon.2) 1028 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: audit 2024-04-02T04:57:32.994656+0000 mon.a (mon.0) 3053 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: audit 2024-04-02T04:57:33.278289+0000 mon.c (mon.2) 1029 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: audit 2024-04-02T04:57:33.362644+0000 mon.b (mon.1) 275 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:34.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[28904]: audit 2024-04-02T04:57:33.428634+0000 mgr.y (mgr.14538) 543 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.167985+0000 osd.2 (osd.2) 260 : cluster 0 102.17 scrub starts 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.219334+0000 osd.2 (osd.2) 261 : cluster 0 102.17 scrub ok 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.701674+0000 osd.6 (osd.6) 259 : cluster 0 10.0 scrub starts 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.703370+0000 osd.6 (osd.6) 260 : cluster 0 10.0 scrub ok 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.965032+0000 osd.1 (osd.1) 309 : cluster 0 4.7 scrub starts 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.966981+0000 osd.1 (osd.1) 310 : cluster 0 4.7 scrub ok 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: audit 2024-04-02T04:57:32.985376+0000 mon.a (mon.0) 3051 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: cluster 2024-04-02T04:57:32.991489+0000 mon.a (mon.0) 3052 : cluster 0 osdmap e572: 8 total, 8 up, 6 in 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: audit 2024-04-02T04:57:32.994037+0000 mon.c (mon.2) 1028 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: audit 2024-04-02T04:57:32.994656+0000 mon.a (mon.0) 3053 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: audit 2024-04-02T04:57:33.278289+0000 mon.c (mon.2) 1029 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: audit 2024-04-02T04:57:33.362644+0000 mon.b (mon.1) 275 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:34.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:34 smithi069 bash[21528]: audit 2024-04-02T04:57:33.428634+0000 mgr.y (mgr.14538) 543 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:34.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:34.369 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:34.472 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:34.497 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:34.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:34.627 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:34.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:34.757 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:34.861 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:34.885 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:34.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:34.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:35.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:35.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:35.210 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:35.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:33.965052+0000 osd.1 (osd.1) 311 : cluster 0 3.4 scrub starts 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:33.970710+0000 osd.1 (osd.1) 312 : cluster 0 3.4 scrub ok 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: audit 2024-04-02T04:57:33.988209+0000 mon.a (mon.0) 3054 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:33.995923+0000 mon.a (mon.0) 3055 : cluster 0 osdmap e573: 8 total, 8 up, 6 in 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: audit 2024-04-02T04:57:33.999599+0000 mon.c (mon.2) 1030 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: audit 2024-04-02T04:57:34.020881+0000 mon.a (mon.0) 3056 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:34.027860+0000 osd.4 (osd.4) 309 : cluster 0 40.17 scrub starts 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:34.029243+0000 osd.4 (osd.4) 310 : cluster 0 40.17 scrub ok 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:34.204086+0000 mgr.y (mgr.14538) 544 : cluster 0 pgmap v857: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: audit 2024-04-02T04:57:34.280017+0000 mon.c (mon.2) 1031 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: audit 2024-04-02T04:57:34.990266+0000 mon.a (mon.0) 3057 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:35.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[28904]: cluster 2024-04-02T04:57:34.994482+0000 mon.a (mon.0) 3058 : cluster 0 osdmap e574: 8 total, 8 up, 6 in 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:33.965052+0000 osd.1 (osd.1) 311 : cluster 0 3.4 scrub starts 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:33.970710+0000 osd.1 (osd.1) 312 : cluster 0 3.4 scrub ok 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: audit 2024-04-02T04:57:33.988209+0000 mon.a (mon.0) 3054 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:33.995923+0000 mon.a (mon.0) 3055 : cluster 0 osdmap e573: 8 total, 8 up, 6 in 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: audit 2024-04-02T04:57:33.999599+0000 mon.c (mon.2) 1030 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: audit 2024-04-02T04:57:34.020881+0000 mon.a (mon.0) 3056 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:34.027860+0000 osd.4 (osd.4) 309 : cluster 0 40.17 scrub starts 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:34.029243+0000 osd.4 (osd.4) 310 : cluster 0 40.17 scrub ok 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:34.204086+0000 mgr.y (mgr.14538) 544 : cluster 0 pgmap v857: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: audit 2024-04-02T04:57:34.280017+0000 mon.c (mon.2) 1031 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: audit 2024-04-02T04:57:34.990266+0000 mon.a (mon.0) 3057 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:35.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:35 smithi069 bash[21528]: cluster 2024-04-02T04:57:34.994482+0000 mon.a (mon.0) 3058 : cluster 0 osdmap e574: 8 total, 8 up, 6 in 2024-04-02T04:57:35.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:35.342 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:35.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:35.468 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:35.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:33.965052+0000 osd.1 (osd.1) 311 : cluster 0 3.4 scrub starts 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:33.970710+0000 osd.1 (osd.1) 312 : cluster 0 3.4 scrub ok 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: audit 2024-04-02T04:57:33.988209+0000 mon.a (mon.0) 3054 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:33.995923+0000 mon.a (mon.0) 3055 : cluster 0 osdmap e573: 8 total, 8 up, 6 in 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: audit 2024-04-02T04:57:33.999599+0000 mon.c (mon.2) 1030 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: audit 2024-04-02T04:57:34.020881+0000 mon.a (mon.0) 3056 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:34.027860+0000 osd.4 (osd.4) 309 : cluster 0 40.17 scrub starts 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:34.029243+0000 osd.4 (osd.4) 310 : cluster 0 40.17 scrub ok 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:34.204086+0000 mgr.y (mgr.14538) 544 : cluster 0 pgmap v857: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: audit 2024-04-02T04:57:34.280017+0000 mon.c (mon.2) 1031 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: audit 2024-04-02T04:57:34.990266+0000 mon.a (mon.0) 3057 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:35.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:35 smithi098 bash[23790]: cluster 2024-04-02T04:57:34.994482+0000 mon.a (mon.0) 3058 : cluster 0 osdmap e574: 8 total, 8 up, 6 in 2024-04-02T04:57:35.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:35.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:35.730 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:35.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:35.860 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:35.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:35.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:36.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:36.112 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:36.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- 172.21.15.69:0/1780911349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 msgr2=0x7f3c54108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/1780911349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c54108770 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009990 tx=0x7f3c48009ca0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- 172.21.15.69:0/1780911349 shutdown_connections 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/1780911349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102ce0 0x7f3c5410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/1780911349 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c54108770 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/1780911349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102320 0x7f3c541027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- 172.21.15.69:0/1780911349 >> 172.21.15.69:0/1780911349 conn(0x7f3c54079310 msgr2=0x7f3c54079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:36.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- 172.21.15.69:0/1780911349 shutdown_connections 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- 172.21.15.69:0/1780911349 wait complete. 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 librados: starting msgr at 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 librados: starting objecter 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 Processor -- start 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- start start 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 librados: setting wanted keys 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 librados: calling monclient init 2024-04-02T04:57:36.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102320 0x7f3c54198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102ce0 0x7f3c54198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c5419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c54111a80 con 0x7f3c54102ce0 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c54111900 con 0x7f3c54102320 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.182+0000 7f3c5afb1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3c54111c00 con 0x7f3c54108390 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c5419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c53fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102ce0 0x7f3c54198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c5419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41604/0 (socket says 172.21.15.69:41604) 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 -- 172.21.15.69:0/535276967 learned_addr learned my addr 172.21.15.69:0/535276967 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c58d4d700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102320 0x7f3c54198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 -- 172.21.15.69:0/535276967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102320 msgr2=0x7f3c54198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102320 0x7f3c54198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 -- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102ce0 msgr2=0x7f3c54198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102ce0 0x7f3c54198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 -- 172.21.15.69:0/535276967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5419d050 con 0x7f3c54108390 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c58d4d700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102320 0x7f3c54198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:57:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c53fff700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102ce0 0x7f3c54198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5954e700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c5419c910 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00ada0 tx=0x7f3c4c00c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 10 client.?.objecter ms_handle_connect 0x7f3c54108390 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c017020 con 0x7f3c54108390 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3c4c00ce00 con 0x7f3c54108390 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5419d340 con 0x7f3c54108390 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c005400 con 0x7f3c54108390 2024-04-02T04:57:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5afb1700 10 client.26723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c541a4bc0 con 0x7f3c54108390 2024-04-02T04:57:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5afb1700 1 librados: init done 2024-04-02T04:57:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5404ff00 con 0x7f3c54108390 2024-04-02T04:57:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3c4c00c980 con 0x7f3c54108390 2024-04-02T04:57:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c4406f5e0 0x7f3c44071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(575..575 src has 74..575) ==== 10618+0+0 (secure 0 0 0) 0x7f3c4c08e400 con 0x7f3c54108390 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 10 client.26723.objecter ms_dispatch 0x7f3c54001550 osd_map(575..575 src has 74..575) 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 3 client.26723.objecter handle_osd_map got epochs [575,575] > 0 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 3 client.26723.objecter handle_osd_map decoding full epoch 575 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c58d4d700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c4406f5e0 0x7f3c44071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 20 client.26723.objecter dump_active .. 0 homeless 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c58d4d700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c4406f5e0 0x7f3c44071aa0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800b7f0 tx=0x7f3c48005fb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.186+0000 7f3c51ffb700 10 client.26723.objecter ms_handle_connect 0x7f3c4406f5e0 2024-04-02T04:57:36.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.194+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3c4c021020 con 0x7f3c54108390 2024-04-02T04:57:36.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:36 smithi098 bash[23790]: cluster 2024-04-02T04:57:35.110713+0000 osd.7 (osd.7) 91 : cluster 0 6.5 scrub starts 2024-04-02T04:57:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:36 smithi098 bash[23790]: cluster 2024-04-02T04:57:35.112495+0000 osd.7 (osd.7) 92 : cluster 0 6.5 scrub ok 2024-04-02T04:57:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:36 smithi098 bash[23790]: audit 2024-04-02T04:57:35.281671+0000 mon.c (mon.2) 1032 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:36.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:36 smithi069 bash[21528]: cluster 2024-04-02T04:57:35.110713+0000 osd.7 (osd.7) 91 : cluster 0 6.5 scrub starts 2024-04-02T04:57:36.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:36 smithi069 bash[21528]: cluster 2024-04-02T04:57:35.112495+0000 osd.7 (osd.7) 92 : cluster 0 6.5 scrub ok 2024-04-02T04:57:36.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:36 smithi069 bash[21528]: audit 2024-04-02T04:57:35.281671+0000 mon.c (mon.2) 1032 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:36.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:36 smithi069 bash[28904]: cluster 2024-04-02T04:57:35.110713+0000 osd.7 (osd.7) 91 : cluster 0 6.5 scrub starts 2024-04-02T04:57:36.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:36 smithi069 bash[28904]: cluster 2024-04-02T04:57:35.112495+0000 osd.7 (osd.7) 92 : cluster 0 6.5 scrub ok 2024-04-02T04:57:36.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:36 smithi069 bash[28904]: audit 2024-04-02T04:57:35.281671+0000 mon.c (mon.2) 1032 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:36.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:36.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:36.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.422+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f3c541a4d90 con 0x7f3c54108390 2024-04-02T04:57:36.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- 172.21.15.69:0/932262706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406ab80 msgr2=0x7fcd44106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- 172.21.15.69:0/932262706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406ab80 0x7fcd44106ef0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcd400099c0 tx=0x7fcd40009cd0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- 172.21.15.69:0/932262706 shutdown_connections 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- 172.21.15.69:0/932262706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd44107620 0x7fcd4410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- 172.21.15.69:0/932262706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4406a1d0 0x7fcd4406a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- 172.21.15.69:0/932262706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406ab80 0x7fcd44106ef0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- 172.21.15.69:0/932262706 >> 172.21.15.69:0/932262706 conn(0x7fcd440771e0 msgr2=0x7fcd440775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- 172.21.15.69:0/932262706 shutdown_connections 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- 172.21.15.69:0/932262706 wait complete. 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 librados: starting msgr at 2024-04-02T04:57:36.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 librados: starting objecter 2024-04-02T04:57:36.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 Processor -- start 2024-04-02T04:57:36.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- start start 2024-04-02T04:57:36.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 librados: setting wanted keys 2024-04-02T04:57:36.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 librados: calling monclient init 2024-04-02T04:57:36.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 0x7fcd44198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4406ab80 0x7fcd441985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd44107620 0x7fcd4419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd44111b30 con 0x7fcd4406a1d0 2024-04-02T04:57:36.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd441119b0 con 0x7fcd44107620 2024-04-02T04:57:36.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd4bb57700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fcd44111cb0 con 0x7fcd4406ab80 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 0x7fcd44198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd490f2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4406ab80 0x7fcd441985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 0x7fcd44198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34796/0 (socket says 172.21.15.69:34796) 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 -- 172.21.15.69:0/2299369953 learned_addr learned my addr 172.21.15.69:0/2299369953 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 -- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4406ab80 msgr2=0x7fcd441985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4406ab80 0x7fcd441985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 -- 172.21.15.69:0/2299369953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd44107620 msgr2=0x7fcd4419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd44107620 0x7fcd4419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 -- 172.21.15.69:0/2299369953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4419d070 con 0x7fcd4406a1d0 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd498f3700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 0x7fcd44198060 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400e9a0 tx=0x7fcd3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd3affd700 10 client.?.objecter ms_handle_connect 0x7fcd4406a1d0 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.474+0000 7fcd3affd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3400cbd0 con 0x7fcd4406a1d0 2024-04-02T04:57:36.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fcd3400cd70 con 0x7fcd4406a1d0 2024-04-02T04:57:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34010e60 con 0x7fcd4406a1d0 2024-04-02T04:57:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd4bb57700 1 -- 172.21.15.69:0/2299369953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4419d360 con 0x7fcd4406a1d0 2024-04-02T04:57:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd4bb57700 10 client.16638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd4bb57700 1 -- 172.21.15.69:0/2299369953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd441011a0 con 0x7fcd4406a1d0 2024-04-02T04:57:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd4bb57700 1 librados: init done 2024-04-02T04:57:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd2c005120 con 0x7fcd4406a1d0 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fcd34010430 con 0x7fcd4406a1d0 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcd3006f5e0 0x7fcd30071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd490f2700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcd3006f5e0 0x7fcd30071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(575..575 src has 74..575) ==== 10618+0+0 (secure 0 0 0) 0x7fcd3408f330 con 0x7fcd4406a1d0 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 10 client.16638.objecter ms_dispatch 0x7fcd44001550 osd_map(575..575 src has 74..575) 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 3 client.16638.objecter handle_osd_map got epochs [575,575] > 0 2024-04-02T04:57:36.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 3 client.16638.objecter handle_osd_map decoding full epoch 575 2024-04-02T04:57:36.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 20 client.16638.objecter dump_active .. 0 homeless 2024-04-02T04:57:36.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd490f2700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcd3006f5e0 0x7fcd30071aa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009990 tx=0x7fcd40005040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:36.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.478+0000 7fcd3affd700 10 client.16638.objecter ms_handle_connect 0x7fcd3006f5e0 2024-04-02T04:57:36.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.482+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd3405bdc0 con 0x7fcd4406a1d0 2024-04-02T04:57:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:36.519 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:57:36.520 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:57:36.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:36.623 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:36.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:36.690+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) -- 0x7fcd2c000bc0 con 0x7fcd3006f5e0 2024-04-02T04:57:36.710 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:57:36.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:36.752 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:36.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:36.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:36.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:37.009 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:37.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.046+0000 7f3c51ffb700 1 -- 172.21.15.69:0/535276967 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v576) ==== 101+0+0 (secure 0 0 0) 0x7f3c4c05ae10 con 0x7f3c54108390 2024-04-02T04:57:37.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 10 librados: watch_flush enter 2024-04-02T04:57:37.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 10 librados: watch_flush exit 2024-04-02T04:57:37.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 20 client.26723.objecter shutdown clearing up homeless session... 2024-04-02T04:57:37.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 10 client.26723.objecter successfully canceled tick 2024-04-02T04:57:37.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c4406f5e0 msgr2=0x7f3c44071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:37.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c4406f5e0 0x7f3c44071aa0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800b7f0 tx=0x7f3c48005fb0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 msgr2=0x7f3c5419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:37.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.050+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c5419c910 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00ada0 tx=0x7f3c4c00c330 comp rx=0 tx=0).stop 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 shutdown_connections 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3c4406f5e0 0x7f3c44071aa0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c54108390 0x7f3c5419c910 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c54102320 0x7f3c54198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 --2- 172.21.15.69:0/535276967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c54102ce0 0x7f3c54198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 >> 172.21.15.69:0/535276967 conn(0x7f3c54079310 msgr2=0x7f3c54109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 shutdown_connections 2024-04-02T04:57:37.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.054+0000 7f3c5afb1700 1 -- 172.21.15.69:0/535276967 wait complete. 2024-04-02T04:57:37.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.058+0000 7fcd3affd700 1 -- 172.21.15.69:0/2299369953 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+393 (secure 0 0 0) 0x7fcd2c000bc0 con 0x7fcd3006f5e0 2024-04-02T04:57:37.059 INFO:teuthology.orchestra.run.smithi069.stdout:moved 60 / 939 (6.38978%) 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:avg 156.5 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:stddev 7.22842 -> 12.5133 (expected baseline 11.42) 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:min osd.7 with 147 -> 163 pgs (0.939297 -> 1.04153 * mean) 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:max osd.6 with 167 -> 151 pgs (1.06709 -> 0.964856 * mean) 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:oload 110 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:max_change 3 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:max_change_osds 4 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:average_utilization 0.0008 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:overload_utilization 0.0009 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 weight 1.0000 -> 0.8233 2024-04-02T04:57:37.060 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 weight 1.0000 -> 0.8671 2024-04-02T04:57:37.061 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 weight 1.0000 -> 0.9038 2024-04-02T04:57:37.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.058+0000 7f3c5afb1700 1 librados: shutdown 2024-04-02T04:57:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 10 librados: watch_flush enter 2024-04-02T04:57:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 10 librados: watch_flush exit 2024-04-02T04:57:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 20 client.16638.objecter shutdown clearing up homeless session... 2024-04-02T04:57:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 10 client.16638.objecter successfully canceled tick 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcd3006f5e0 msgr2=0x7fcd30071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcd3006f5e0 0x7fcd30071aa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009990 tx=0x7fcd40005040 comp rx=0 tx=0).stop 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 msgr2=0x7fcd44198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 0x7fcd44198060 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400e9a0 tx=0x7fcd3400ecb0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 shutdown_connections 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fcd3006f5e0 0x7fcd30071aa0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd44107620 0x7fcd4419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406a1d0 0x7fcd44198060 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 --2- 172.21.15.69:0/2299369953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4406ab80 0x7fcd441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.062+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 >> 172.21.15.69:0/2299369953 conn(0x7fcd440771e0 msgr2=0x7fcd440ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:57:37.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.070+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 shutdown_connections 2024-04-02T04:57:37.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.070+0000 7fcd38ff9700 1 -- 172.21.15.69:0/2299369953 wait complete. 2024-04-02T04:57:37.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:37.070+0000 7fcd38ff9700 1 librados: shutdown 2024-04-02T04:57:37.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:37.130 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:37.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:35.985549+0000 osd.1 (osd.1) 313 : cluster 0 2.d scrub starts 2024-04-02T04:57:37.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:35.986964+0000 osd.4 (osd.4) 311 : cluster 0 13.6 scrub starts 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:35.987305+0000 osd.1 (osd.1) 314 : cluster 0 2.d scrub ok 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:35.988988+0000 osd.4 (osd.4) 312 : cluster 0 13.6 scrub ok 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:36.033374+0000 mon.a (mon.0) 3059 : cluster 0 osdmap e575: 8 total, 8 up, 6 in 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:36.114014+0000 osd.7 (osd.7) 93 : cluster 0 6.1c scrub starts 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:36.116974+0000 osd.7 (osd.7) 94 : cluster 0 6.1c scrub ok 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:36.142181+0000 osd.0 (osd.0) 301 : cluster 0 4.b scrub starts 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:36.143757+0000 osd.0 (osd.0) 302 : cluster 0 4.b scrub ok 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: cluster 2024-04-02T04:57:36.204630+0000 mgr.y (mgr.14538) 545 : cluster 0 pgmap v860: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: audit 2024-04-02T04:57:36.283188+0000 mon.c (mon.2) 1033 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: audit 2024-04-02T04:57:36.425479+0000 mon.c (mon.2) 1034 : audit 1 from='client.? 172.21.15.69:0/535276967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: audit 2024-04-02T04:57:36.425867+0000 mon.a (mon.0) 3060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: audit 2024-04-02T04:57:36.691614+0000 mon.b (mon.1) 276 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]: dispatch 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: audit 2024-04-02T04:57:36.693520+0000 mgr.y (mgr.14538) 546 : audit 0 from='client.16638 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:37.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[21528]: audit 2024-04-02T04:57:36.696462+0000 mon.a (mon.0) 3061 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]: dispatch 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:35.985549+0000 osd.1 (osd.1) 313 : cluster 0 2.d scrub starts 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:35.986964+0000 osd.4 (osd.4) 311 : cluster 0 13.6 scrub starts 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:35.987305+0000 osd.1 (osd.1) 314 : cluster 0 2.d scrub ok 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:35.988988+0000 osd.4 (osd.4) 312 : cluster 0 13.6 scrub ok 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:36.033374+0000 mon.a (mon.0) 3059 : cluster 0 osdmap e575: 8 total, 8 up, 6 in 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:36.114014+0000 osd.7 (osd.7) 93 : cluster 0 6.1c scrub starts 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:36.116974+0000 osd.7 (osd.7) 94 : cluster 0 6.1c scrub ok 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:36.142181+0000 osd.0 (osd.0) 301 : cluster 0 4.b scrub starts 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:36.143757+0000 osd.0 (osd.0) 302 : cluster 0 4.b scrub ok 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: cluster 2024-04-02T04:57:36.204630+0000 mgr.y (mgr.14538) 545 : cluster 0 pgmap v860: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: audit 2024-04-02T04:57:36.283188+0000 mon.c (mon.2) 1033 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: audit 2024-04-02T04:57:36.425479+0000 mon.c (mon.2) 1034 : audit 1 from='client.? 172.21.15.69:0/535276967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: audit 2024-04-02T04:57:36.425867+0000 mon.a (mon.0) 3060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: audit 2024-04-02T04:57:36.691614+0000 mon.b (mon.1) 276 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]: dispatch 2024-04-02T04:57:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: audit 2024-04-02T04:57:36.693520+0000 mgr.y (mgr.14538) 546 : audit 0 from='client.16638 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:37.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:37 smithi069 bash[28904]: audit 2024-04-02T04:57:36.696462+0000 mon.a (mon.0) 3061 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]: dispatch 2024-04-02T04:57:37.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:35.985549+0000 osd.1 (osd.1) 313 : cluster 0 2.d scrub starts 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:35.986964+0000 osd.4 (osd.4) 311 : cluster 0 13.6 scrub starts 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:35.987305+0000 osd.1 (osd.1) 314 : cluster 0 2.d scrub ok 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:35.988988+0000 osd.4 (osd.4) 312 : cluster 0 13.6 scrub ok 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:36.033374+0000 mon.a (mon.0) 3059 : cluster 0 osdmap e575: 8 total, 8 up, 6 in 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:36.114014+0000 osd.7 (osd.7) 93 : cluster 0 6.1c scrub starts 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:36.116974+0000 osd.7 (osd.7) 94 : cluster 0 6.1c scrub ok 2024-04-02T04:57:37.376 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:36.142181+0000 osd.0 (osd.0) 301 : cluster 0 4.b scrub starts 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:36.143757+0000 osd.0 (osd.0) 302 : cluster 0 4.b scrub ok 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: cluster 2024-04-02T04:57:36.204630+0000 mgr.y (mgr.14538) 545 : cluster 0 pgmap v860: 313 pgs: 1 active+remapped+backfill_toofull, 312 active+clean; 8.4 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1266 objects misplaced (0.158%) 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: audit 2024-04-02T04:57:36.283188+0000 mon.c (mon.2) 1033 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: audit 2024-04-02T04:57:36.425479+0000 mon.c (mon.2) 1034 : audit 1 from='client.? 172.21.15.69:0/535276967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: audit 2024-04-02T04:57:36.425867+0000 mon.a (mon.0) 3060 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: audit 2024-04-02T04:57:36.691614+0000 mon.b (mon.1) 276 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]: dispatch 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: audit 2024-04-02T04:57:36.693520+0000 mgr.y (mgr.14538) 546 : audit 0 from='client.16638 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:37.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:37 smithi098 bash[23790]: audit 2024-04-02T04:57:36.696462+0000 mon.a (mon.0) 3061 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]: dispatch 2024-04-02T04:57:37.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:37.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:37.521 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:37.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-02T04:57:37.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:37.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:37.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:37.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:37.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:38.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:38.038 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:38.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:38.169 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:38.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:38.298 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:38.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:38.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.038516+0000 mon.a (mon.0) 3062 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: audit 2024-04-02T04:57:37.045093+0000 mon.a (mon.0) 3063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: audit 2024-04-02T04:57:37.045169+0000 mon.a (mon.0) 3064 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]': finished 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.049453+0000 mon.a (mon.0) 3065 : cluster 0 osdmap e576: 8 total, 8 up, 6 in 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.100029+0000 osd.0 (osd.0) 303 : cluster 0 3.18 scrub starts 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.101600+0000 osd.0 (osd.0) 304 : cluster 0 3.18 scrub ok 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.147021+0000 osd.7 (osd.7) 95 : cluster 0 2.12 scrub starts 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.148893+0000 osd.7 (osd.7) 96 : cluster 0 2.12 scrub ok 2024-04-02T04:57:38.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:38 smithi098 bash[23790]: audit 2024-04-02T04:57:37.284245+0000 mon.c (mon.2) 1035 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:38.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.038516+0000 mon.a (mon.0) 3062 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:57:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: audit 2024-04-02T04:57:37.045093+0000 mon.a (mon.0) 3063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:57:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: audit 2024-04-02T04:57:37.045169+0000 mon.a (mon.0) 3064 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]': finished 2024-04-02T04:57:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.049453+0000 mon.a (mon.0) 3065 : cluster 0 osdmap e576: 8 total, 8 up, 6 in 2024-04-02T04:57:38.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.100029+0000 osd.0 (osd.0) 303 : cluster 0 3.18 scrub starts 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.101600+0000 osd.0 (osd.0) 304 : cluster 0 3.18 scrub ok 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.147021+0000 osd.7 (osd.7) 95 : cluster 0 2.12 scrub starts 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.148893+0000 osd.7 (osd.7) 96 : cluster 0 2.12 scrub ok 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21528]: audit 2024-04-02T04:57:37.284245+0000 mon.c (mon.2) 1035 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.038516+0000 mon.a (mon.0) 3062 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: audit 2024-04-02T04:57:37.045093+0000 mon.a (mon.0) 3063 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: audit 2024-04-02T04:57:37.045169+0000 mon.a (mon.0) 3064 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'56823','2':'59231','6':'53959'}"}]': finished 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.049453+0000 mon.a (mon.0) 3065 : cluster 0 osdmap e576: 8 total, 8 up, 6 in 2024-04-02T04:57:38.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.100029+0000 osd.0 (osd.0) 303 : cluster 0 3.18 scrub starts 2024-04-02T04:57:38.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.101600+0000 osd.0 (osd.0) 304 : cluster 0 3.18 scrub ok 2024-04-02T04:57:38.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.147021+0000 osd.7 (osd.7) 95 : cluster 0 2.12 scrub starts 2024-04-02T04:57:38.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.148893+0000 osd.7 (osd.7) 96 : cluster 0 2.12 scrub ok 2024-04-02T04:57:38.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[28904]: audit 2024-04-02T04:57:37.284245+0000 mon.c (mon.2) 1035 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:38.553 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:38.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:38.682 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:38.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:38.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:38.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:38.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:38.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:39.029 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:57:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:57:39.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:39.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:39.204 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:39.227 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-02T04:57:39.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd test-reweight-by-utilization 110 3.0 2024-04-02T04:57:39.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:39.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:39.346 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.168141+0000 osd.2 (osd.2) 262 : cluster 0 43.2 scrub starts 2024-04-02T04:57:39.346 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: cluster 2024-04-02T04:57:37.169773+0000 osd.2 (osd.2) 263 : cluster 0 43.2 scrub ok 2024-04-02T04:57:39.346 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.012703+0000 osd.1 (osd.1) 315 : cluster 0 4.1d scrub starts 2024-04-02T04:57:39.346 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.014587+0000 osd.1 (osd.1) 316 : cluster 0 4.1d scrub ok 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.060501+0000 mon.a (mon.0) 3066 : cluster 0 osdmap e577: 8 total, 8 up, 6 in 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: audit 2024-04-02T04:57:38.177020+0000 mon.c (mon.2) 1036 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: audit 2024-04-02T04:57:38.177500+0000 mon.a (mon.0) 3067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: audit 2024-04-02T04:57:38.178110+0000 mon.c (mon.2) 1037 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-83"}]: dispatch 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: audit 2024-04-02T04:57:38.178461+0000 mon.a (mon.0) 3068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-83"}]: dispatch 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.205312+0000 mgr.y (mgr.14538) 547 : cluster 0 pgmap v863: 313 pgs: 47 peering, 3 unknown, 263 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[21528]: audit 2024-04-02T04:57:38.285297+0000 mon.c (mon.2) 1038 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.168141+0000 osd.2 (osd.2) 262 : cluster 0 43.2 scrub starts 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: cluster 2024-04-02T04:57:37.169773+0000 osd.2 (osd.2) 263 : cluster 0 43.2 scrub ok 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.012703+0000 osd.1 (osd.1) 315 : cluster 0 4.1d scrub starts 2024-04-02T04:57:39.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.014587+0000 osd.1 (osd.1) 316 : cluster 0 4.1d scrub ok 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.060501+0000 mon.a (mon.0) 3066 : cluster 0 osdmap e577: 8 total, 8 up, 6 in 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: audit 2024-04-02T04:57:38.177020+0000 mon.c (mon.2) 1036 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: audit 2024-04-02T04:57:38.177500+0000 mon.a (mon.0) 3067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: audit 2024-04-02T04:57:38.178110+0000 mon.c (mon.2) 1037 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-83"}]: dispatch 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: audit 2024-04-02T04:57:38.178461+0000 mon.a (mon.0) 3068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-83"}]: dispatch 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.205312+0000 mgr.y (mgr.14538) 547 : cluster 0 pgmap v863: 313 pgs: 47 peering, 3 unknown, 263 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:57:39.348 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:39 smithi069 bash[28904]: audit 2024-04-02T04:57:38.285297+0000 mon.c (mon.2) 1038 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:39.361 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:39.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.168141+0000 osd.2 (osd.2) 262 : cluster 0 43.2 scrub starts 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: cluster 2024-04-02T04:57:37.169773+0000 osd.2 (osd.2) 263 : cluster 0 43.2 scrub ok 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.012703+0000 osd.1 (osd.1) 315 : cluster 0 4.1d scrub starts 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.014587+0000 osd.1 (osd.1) 316 : cluster 0 4.1d scrub ok 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.060501+0000 mon.a (mon.0) 3066 : cluster 0 osdmap e577: 8 total, 8 up, 6 in 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: audit 2024-04-02T04:57:38.177020+0000 mon.c (mon.2) 1036 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: audit 2024-04-02T04:57:38.177500+0000 mon.a (mon.0) 3067 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: audit 2024-04-02T04:57:38.178110+0000 mon.c (mon.2) 1037 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-83"}]: dispatch 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: audit 2024-04-02T04:57:38.178461+0000 mon.a (mon.0) 3068 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-83"}]: dispatch 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.205312+0000 mgr.y (mgr.14538) 547 : cluster 0 pgmap v863: 313 pgs: 47 peering, 3 unknown, 263 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-04-02T04:57:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:39 smithi098 bash[23790]: audit 2024-04-02T04:57:38.285297+0000 mon.c (mon.2) 1038 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:39.484 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:39.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:39.613 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:39.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:39.741 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:39.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:39.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:39.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:39.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:40.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.138694+0000 osd.2 (osd.2) 264 : cluster 0 43.13 scrub starts 2024-04-02T04:57:40.120 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.143261+0000 osd.2 (osd.2) 265 : cluster 0 43.13 scrub ok 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.676266+0000 osd.6 (osd.6) 261 : cluster 0 102.6 scrub starts 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:38.679406+0000 osd.6 (osd.6) 262 : cluster 0 102.6 scrub ok 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.054616+0000 mon.a (mon.0) 3069 : cluster 3 Health check failed: Reduced data availability: 14 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.054685+0000 mon.a (mon.0) 3070 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.071554+0000 osd.4 (osd.4) 313 : cluster 0 13.5 starting backfill to osd.1 from (0'0,0'0] MAX to 478'11 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.073211+0000 mon.a (mon.0) 3071 : cluster 0 osdmap e578: 8 total, 8 up, 6 in 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.077887+0000 osd.2 (osd.2) 266 : cluster 0 13.4 starting backfill to osd.0 from (0'0,0'0] MAX to 138'2 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: audit 2024-04-02T04:57:39.286602+0000 mon.c (mon.2) 1039 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.330955+0000 osd.4 (osd.4) 314 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.331166+0000 osd.4 (osd.4) 315 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: cluster 2024-04-02T04:57:39.974500+0000 mon.a (mon.0) 3072 : cluster 0 osdmap e579: 8 total, 8 up, 6 in 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: audit 2024-04-02T04:57:39.981026+0000 mon.c (mon.2) 1040 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:40.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[28904]: audit 2024-04-02T04:57:40.005327+0000 mon.a (mon.0) 3073 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.138694+0000 osd.2 (osd.2) 264 : cluster 0 43.13 scrub starts 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.143261+0000 osd.2 (osd.2) 265 : cluster 0 43.13 scrub ok 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.676266+0000 osd.6 (osd.6) 261 : cluster 0 102.6 scrub starts 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:38.679406+0000 osd.6 (osd.6) 262 : cluster 0 102.6 scrub ok 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.054616+0000 mon.a (mon.0) 3069 : cluster 3 Health check failed: Reduced data availability: 14 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.054685+0000 mon.a (mon.0) 3070 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.071554+0000 osd.4 (osd.4) 313 : cluster 0 13.5 starting backfill to osd.1 from (0'0,0'0] MAX to 478'11 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.073211+0000 mon.a (mon.0) 3071 : cluster 0 osdmap e578: 8 total, 8 up, 6 in 2024-04-02T04:57:40.122 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.077887+0000 osd.2 (osd.2) 266 : cluster 0 13.4 starting backfill to osd.0 from (0'0,0'0] MAX to 138'2 2024-04-02T04:57:40.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: audit 2024-04-02T04:57:39.286602+0000 mon.c (mon.2) 1039 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:40.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.330955+0000 osd.4 (osd.4) 314 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:40.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.331166+0000 osd.4 (osd.4) 315 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:40.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: cluster 2024-04-02T04:57:39.974500+0000 mon.a (mon.0) 3072 : cluster 0 osdmap e579: 8 total, 8 up, 6 in 2024-04-02T04:57:40.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: audit 2024-04-02T04:57:39.981026+0000 mon.c (mon.2) 1040 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:40.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:40 smithi069 bash[21528]: audit 2024-04-02T04:57:40.005327+0000 mon.a (mon.0) 3073 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:40.133 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:40.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:40.264 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:40.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.138694+0000 osd.2 (osd.2) 264 : cluster 0 43.13 scrub starts 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.143261+0000 osd.2 (osd.2) 265 : cluster 0 43.13 scrub ok 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.676266+0000 osd.6 (osd.6) 261 : cluster 0 102.6 scrub starts 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:38.679406+0000 osd.6 (osd.6) 262 : cluster 0 102.6 scrub ok 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.054616+0000 mon.a (mon.0) 3069 : cluster 3 Health check failed: Reduced data availability: 14 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.054685+0000 mon.a (mon.0) 3070 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.071554+0000 osd.4 (osd.4) 313 : cluster 0 13.5 starting backfill to osd.1 from (0'0,0'0] MAX to 478'11 2024-04-02T04:57:40.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.073211+0000 mon.a (mon.0) 3071 : cluster 0 osdmap e578: 8 total, 8 up, 6 in 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.077887+0000 osd.2 (osd.2) 266 : cluster 0 13.4 starting backfill to osd.0 from (0'0,0'0] MAX to 138'2 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: audit 2024-04-02T04:57:39.286602+0000 mon.c (mon.2) 1039 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.330955+0000 osd.4 (osd.4) 314 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.331166+0000 osd.4 (osd.4) 315 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: cluster 2024-04-02T04:57:39.974500+0000 mon.a (mon.0) 3072 : cluster 0 osdmap e579: 8 total, 8 up, 6 in 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: audit 2024-04-02T04:57:39.981026+0000 mon.c (mon.2) 1040 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:40.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:40 smithi098 bash[23790]: audit 2024-04-02T04:57:40.005327+0000 mon.a (mon.0) 3073 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:40.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:40.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:40.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:40.773 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:40.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:40.903 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:40.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:41.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:41.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:41.170 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: cluster 2024-04-02T04:57:40.175829+0000 osd.7 (osd.7) 97 : cluster 0 40.0 scrub starts 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: cluster 2024-04-02T04:57:40.178165+0000 osd.7 (osd.7) 98 : cluster 0 40.0 scrub ok 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: cluster 2024-04-02T04:57:40.206010+0000 mgr.y (mgr.14538) 548 : cluster 0 pgmap v866: 313 pgs: 43 peering, 35 unknown, 235 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 282 B/s, 0 objects/s recovering 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: audit 2024-04-02T04:57:40.287699+0000 mon.c (mon.2) 1041 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: audit 2024-04-02T04:57:40.989158+0000 mon.a (mon.0) 3074 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: cluster 2024-04-02T04:57:40.997404+0000 mon.a (mon.0) 3075 : cluster 0 osdmap e580: 8 total, 8 up, 6 in 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: audit 2024-04-02T04:57:41.002787+0000 mon.c (mon.2) 1042 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: audit 2024-04-02T04:57:41.023898+0000 mon.c (mon.2) 1043 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:41.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:41 smithi098 bash[23790]: audit 2024-04-02T04:57:41.025096+0000 mon.a (mon.0) 3076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:41.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:57:41.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:41.301 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:41.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:41.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:41.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: cluster 2024-04-02T04:57:40.175829+0000 osd.7 (osd.7) 97 : cluster 0 40.0 scrub starts 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: cluster 2024-04-02T04:57:40.178165+0000 osd.7 (osd.7) 98 : cluster 0 40.0 scrub ok 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: cluster 2024-04-02T04:57:40.206010+0000 mgr.y (mgr.14538) 548 : cluster 0 pgmap v866: 313 pgs: 43 peering, 35 unknown, 235 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 282 B/s, 0 objects/s recovering 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: audit 2024-04-02T04:57:40.287699+0000 mon.c (mon.2) 1041 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: audit 2024-04-02T04:57:40.989158+0000 mon.a (mon.0) 3074 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: cluster 2024-04-02T04:57:40.997404+0000 mon.a (mon.0) 3075 : cluster 0 osdmap e580: 8 total, 8 up, 6 in 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: audit 2024-04-02T04:57:41.002787+0000 mon.c (mon.2) 1042 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: audit 2024-04-02T04:57:41.023898+0000 mon.c (mon.2) 1043 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[21528]: audit 2024-04-02T04:57:41.025096+0000 mon.a (mon.0) 3076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: cluster 2024-04-02T04:57:40.175829+0000 osd.7 (osd.7) 97 : cluster 0 40.0 scrub starts 2024-04-02T04:57:41.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: cluster 2024-04-02T04:57:40.178165+0000 osd.7 (osd.7) 98 : cluster 0 40.0 scrub ok 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: cluster 2024-04-02T04:57:40.206010+0000 mgr.y (mgr.14538) 548 : cluster 0 pgmap v866: 313 pgs: 43 peering, 35 unknown, 235 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 282 B/s, 0 objects/s recovering 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: audit 2024-04-02T04:57:40.287699+0000 mon.c (mon.2) 1041 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: audit 2024-04-02T04:57:40.989158+0000 mon.a (mon.0) 3074 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: cluster 2024-04-02T04:57:40.997404+0000 mon.a (mon.0) 3075 : cluster 0 osdmap e580: 8 total, 8 up, 6 in 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: audit 2024-04-02T04:57:41.002787+0000 mon.c (mon.2) 1042 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: audit 2024-04-02T04:57:41.023898+0000 mon.c (mon.2) 1043 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:41.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:41 smithi069 bash[28904]: audit 2024-04-02T04:57:41.025096+0000 mon.a (mon.0) 3076 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:41.533 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:41.556 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:41.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:41.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:41.814 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:41.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:41.944 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:41.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:42.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:42.075 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:42.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:42.205 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:42.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: cluster 2024-04-02T04:57:41.055266+0000 osd.4 (osd.4) 316 : cluster 0 6.1a deep-scrub starts 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: cluster 2024-04-02T04:57:41.058747+0000 osd.4 (osd.4) 317 : cluster 0 6.1a deep-scrub ok 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: cluster 2024-04-02T04:57:41.069266+0000 mon.a (mon.0) 3077 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: cluster 2024-04-02T04:57:41.150687+0000 osd.7 (osd.7) 99 : cluster 0 4.14 scrub starts 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: cluster 2024-04-02T04:57:41.153635+0000 osd.7 (osd.7) 100 : cluster 0 4.14 scrub ok 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: audit 2024-04-02T04:57:41.288674+0000 mon.c (mon.2) 1044 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: audit 2024-04-02T04:57:41.991977+0000 mon.a (mon.0) 3078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: audit 2024-04-02T04:57:41.997468+0000 mon.c (mon.2) 1045 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.000371+0000 mon.a (mon.0) 3079 : cluster 0 osdmap e581: 8 total, 8 up, 6 in 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[28904]: audit 2024-04-02T04:57:42.002838+0000 mon.a (mon.0) 3080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:42.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: cluster 2024-04-02T04:57:41.055266+0000 osd.4 (osd.4) 316 : cluster 0 6.1a deep-scrub starts 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: cluster 2024-04-02T04:57:41.058747+0000 osd.4 (osd.4) 317 : cluster 0 6.1a deep-scrub ok 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: cluster 2024-04-02T04:57:41.069266+0000 mon.a (mon.0) 3077 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: cluster 2024-04-02T04:57:41.150687+0000 osd.7 (osd.7) 99 : cluster 0 4.14 scrub starts 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: cluster 2024-04-02T04:57:41.153635+0000 osd.7 (osd.7) 100 : cluster 0 4.14 scrub ok 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: audit 2024-04-02T04:57:41.288674+0000 mon.c (mon.2) 1044 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: audit 2024-04-02T04:57:41.991977+0000 mon.a (mon.0) 3078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: audit 2024-04-02T04:57:41.997468+0000 mon.c (mon.2) 1045 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.000371+0000 mon.a (mon.0) 3079 : cluster 0 osdmap e581: 8 total, 8 up, 6 in 2024-04-02T04:57:42.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:42 smithi069 bash[21528]: audit 2024-04-02T04:57:42.002838+0000 mon.a (mon.0) 3080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:42.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:42.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:42.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: cluster 2024-04-02T04:57:41.055266+0000 osd.4 (osd.4) 316 : cluster 0 6.1a deep-scrub starts 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: cluster 2024-04-02T04:57:41.058747+0000 osd.4 (osd.4) 317 : cluster 0 6.1a deep-scrub ok 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: cluster 2024-04-02T04:57:41.069266+0000 mon.a (mon.0) 3077 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: cluster 2024-04-02T04:57:41.150687+0000 osd.7 (osd.7) 99 : cluster 0 4.14 scrub starts 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: cluster 2024-04-02T04:57:41.153635+0000 osd.7 (osd.7) 100 : cluster 0 4.14 scrub ok 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: audit 2024-04-02T04:57:41.288674+0000 mon.c (mon.2) 1044 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: audit 2024-04-02T04:57:41.991977+0000 mon.a (mon.0) 3078 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: audit 2024-04-02T04:57:41.997468+0000 mon.c (mon.2) 1045 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.000371+0000 mon.a (mon.0) 3079 : cluster 0 osdmap e581: 8 total, 8 up, 6 in 2024-04-02T04:57:42.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:42 smithi098 bash[23790]: audit 2024-04-02T04:57:42.002838+0000 mon.a (mon.0) 3080 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:42.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:42.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:42.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:42.759 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:42.760 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:42.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:42.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:42.950 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:42.977 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:42.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:43.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:43.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.063564+0000 osd.4 (osd.4) 318 : cluster 0 43.1a deep-scrub starts 2024-04-02T04:57:43.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.068267+0000 osd.4 (osd.4) 319 : cluster 0 43.1a deep-scrub ok 2024-04-02T04:57:43.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.118175+0000 osd.7 (osd.7) 101 : cluster 0 6.15 scrub starts 2024-04-02T04:57:43.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.120551+0000 osd.7 (osd.7) 102 : cluster 0 6.15 scrub ok 2024-04-02T04:57:43.091 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.206660+0000 mgr.y (mgr.14538) 549 : cluster 0 pgmap v869: 313 pgs: 43 peering, 35 unknown, 235 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s; 218 B/s, 0 objects/s recovering 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: audit 2024-04-02T04:57:42.289797+0000 mon.c (mon.2) 1046 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: audit 2024-04-02T04:57:42.993806+0000 mon.a (mon.0) 3081 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.997518+0000 mon.a (mon.0) 3082 : cluster 0 osdmap e582: 8 total, 8 up, 6 in 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: audit 2024-04-02T04:57:42.999934+0000 mon.c (mon.2) 1047 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[21528]: audit 2024-04-02T04:57:43.001443+0000 mon.a (mon.0) 3083 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.063564+0000 osd.4 (osd.4) 318 : cluster 0 43.1a deep-scrub starts 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.068267+0000 osd.4 (osd.4) 319 : cluster 0 43.1a deep-scrub ok 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.118175+0000 osd.7 (osd.7) 101 : cluster 0 6.15 scrub starts 2024-04-02T04:57:43.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.120551+0000 osd.7 (osd.7) 102 : cluster 0 6.15 scrub ok 2024-04-02T04:57:43.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.206660+0000 mgr.y (mgr.14538) 549 : cluster 0 pgmap v869: 313 pgs: 43 peering, 35 unknown, 235 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s; 218 B/s, 0 objects/s recovering 2024-04-02T04:57:43.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: audit 2024-04-02T04:57:42.289797+0000 mon.c (mon.2) 1046 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:43.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: audit 2024-04-02T04:57:42.993806+0000 mon.a (mon.0) 3081 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:43.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.997518+0000 mon.a (mon.0) 3082 : cluster 0 osdmap e582: 8 total, 8 up, 6 in 2024-04-02T04:57:43.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: audit 2024-04-02T04:57:42.999934+0000 mon.c (mon.2) 1047 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:43.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:43 smithi069 bash[28904]: audit 2024-04-02T04:57:43.001443+0000 mon.a (mon.0) 3083 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:43.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:43.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:43.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:43.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.063564+0000 osd.4 (osd.4) 318 : cluster 0 43.1a deep-scrub starts 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.068267+0000 osd.4 (osd.4) 319 : cluster 0 43.1a deep-scrub ok 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.118175+0000 osd.7 (osd.7) 101 : cluster 0 6.15 scrub starts 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.120551+0000 osd.7 (osd.7) 102 : cluster 0 6.15 scrub ok 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.206660+0000 mgr.y (mgr.14538) 549 : cluster 0 pgmap v869: 313 pgs: 43 peering, 35 unknown, 235 active+clean; 8.4 MiB data, 455 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s; 218 B/s, 0 objects/s recovering 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: audit 2024-04-02T04:57:42.289797+0000 mon.c (mon.2) 1046 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: audit 2024-04-02T04:57:42.993806+0000 mon.a (mon.0) 3081 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_tier","val": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:57:43.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.997518+0000 mon.a (mon.0) 3082 : cluster 0 osdmap e582: 8 total, 8 up, 6 in 2024-04-02T04:57:43.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: audit 2024-04-02T04:57:42.999934+0000 mon.c (mon.2) 1047 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:43.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[23790]: audit 2024-04-02T04:57:43.001443+0000 mon.a (mon.0) 3083 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:43.362 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:43.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:43.492 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:43.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:43.622 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:43.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:43.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:57:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:57:43.752 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:43.774 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:43.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:43.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:43.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:44.013 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:44.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:44.138 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.151359+0000 osd.2 (osd.2) 267 : cluster 0 10.9 scrub starts 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: cluster 2024-04-02T04:57:42.154580+0000 osd.2 (osd.2) 268 : cluster 0 10.9 scrub ok 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: cluster 2024-04-02T04:57:43.168798+0000 osd.7 (osd.7) 103 : cluster 0 4.17 scrub starts 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: cluster 2024-04-02T04:57:43.172446+0000 osd.7 (osd.7) 104 : cluster 0 4.17 scrub ok 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: audit 2024-04-02T04:57:43.290907+0000 mon.c (mon.2) 1048 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: audit 2024-04-02T04:57:43.439417+0000 mgr.y (mgr.14538) 550 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: audit 2024-04-02T04:57:43.997733+0000 mon.a (mon.0) 3084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: cluster 2024-04-02T04:57:44.005912+0000 mon.a (mon.0) 3085 : cluster 0 osdmap e583: 8 total, 8 up, 6 in 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: audit 2024-04-02T04:57:44.012516+0000 mon.c (mon.2) 1049 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:44.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:44 smithi098 bash[23790]: audit 2024-04-02T04:57:44.013292+0000 mon.a (mon.0) 3086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:44.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:44.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:44.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:44.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:44.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.151359+0000 osd.2 (osd.2) 267 : cluster 0 10.9 scrub starts 2024-04-02T04:57:44.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: cluster 2024-04-02T04:57:42.154580+0000 osd.2 (osd.2) 268 : cluster 0 10.9 scrub ok 2024-04-02T04:57:44.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: cluster 2024-04-02T04:57:43.168798+0000 osd.7 (osd.7) 103 : cluster 0 4.17 scrub starts 2024-04-02T04:57:44.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: cluster 2024-04-02T04:57:43.172446+0000 osd.7 (osd.7) 104 : cluster 0 4.17 scrub ok 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: audit 2024-04-02T04:57:43.290907+0000 mon.c (mon.2) 1048 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: audit 2024-04-02T04:57:43.439417+0000 mgr.y (mgr.14538) 550 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: audit 2024-04-02T04:57:43.997733+0000 mon.a (mon.0) 3084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: cluster 2024-04-02T04:57:44.005912+0000 mon.a (mon.0) 3085 : cluster 0 osdmap e583: 8 total, 8 up, 6 in 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: audit 2024-04-02T04:57:44.012516+0000 mon.c (mon.2) 1049 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[28904]: audit 2024-04-02T04:57:44.013292+0000 mon.a (mon.0) 3086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.151359+0000 osd.2 (osd.2) 267 : cluster 0 10.9 scrub starts 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: cluster 2024-04-02T04:57:42.154580+0000 osd.2 (osd.2) 268 : cluster 0 10.9 scrub ok 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: cluster 2024-04-02T04:57:43.168798+0000 osd.7 (osd.7) 103 : cluster 0 4.17 scrub starts 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: cluster 2024-04-02T04:57:43.172446+0000 osd.7 (osd.7) 104 : cluster 0 4.17 scrub ok 2024-04-02T04:57:44.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: audit 2024-04-02T04:57:43.290907+0000 mon.c (mon.2) 1048 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:44.514 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: audit 2024-04-02T04:57:43.439417+0000 mgr.y (mgr.14538) 550 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:44.514 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: audit 2024-04-02T04:57:43.997733+0000 mon.a (mon.0) 3084 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:44.514 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: cluster 2024-04-02T04:57:44.005912+0000 mon.a (mon.0) 3085 : cluster 0 osdmap e583: 8 total, 8 up, 6 in 2024-04-02T04:57:44.514 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: audit 2024-04-02T04:57:44.012516+0000 mon.c (mon.2) 1049 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:44.514 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:44 smithi069 bash[21528]: audit 2024-04-02T04:57:44.013292+0000 mon.a (mon.0) 3086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:44.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:44.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:44.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:44.788 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:44.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:44.919 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:44.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:45.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:45.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:45.180 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[21528]: cluster 2024-04-02T04:57:43.200273+0000 osd.2 (osd.2) 269 : cluster 0 6.f scrub starts 2024-04-02T04:57:45.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[21528]: cluster 2024-04-02T04:57:43.202090+0000 osd.2 (osd.2) 270 : cluster 0 6.f scrub ok 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[21528]: cluster 2024-04-02T04:57:44.208064+0000 mgr.y (mgr.14538) 551 : cluster 0 pgmap v872: 313 pgs: 313 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 64 KiB/s, 39 objects/s recovering 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[21528]: audit 2024-04-02T04:57:44.291906+0000 mon.c (mon.2) 1050 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[21528]: audit 2024-04-02T04:57:45.000032+0000 mon.a (mon.0) 3087 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.007512+0000 mon.a (mon.0) 3088 : cluster 0 osdmap e584: 8 total, 8 up, 6 in 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[28904]: cluster 2024-04-02T04:57:43.200273+0000 osd.2 (osd.2) 269 : cluster 0 6.f scrub starts 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[28904]: cluster 2024-04-02T04:57:43.202090+0000 osd.2 (osd.2) 270 : cluster 0 6.f scrub ok 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[28904]: cluster 2024-04-02T04:57:44.208064+0000 mgr.y (mgr.14538) 551 : cluster 0 pgmap v872: 313 pgs: 313 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 64 KiB/s, 39 objects/s recovering 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[28904]: audit 2024-04-02T04:57:44.291906+0000 mon.c (mon.2) 1050 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[28904]: audit 2024-04-02T04:57:45.000032+0000 mon.a (mon.0) 3087 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:45 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.007512+0000 mon.a (mon.0) 3088 : cluster 0 osdmap e584: 8 total, 8 up, 6 in 2024-04-02T04:57:45.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:45.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:45 smithi098 bash[23790]: cluster 2024-04-02T04:57:43.200273+0000 osd.2 (osd.2) 269 : cluster 0 6.f scrub starts 2024-04-02T04:57:45.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:45 smithi098 bash[23790]: cluster 2024-04-02T04:57:43.202090+0000 osd.2 (osd.2) 270 : cluster 0 6.f scrub ok 2024-04-02T04:57:45.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:45 smithi098 bash[23790]: cluster 2024-04-02T04:57:44.208064+0000 mgr.y (mgr.14538) 551 : cluster 0 pgmap v872: 313 pgs: 313 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 64 KiB/s, 39 objects/s recovering 2024-04-02T04:57:45.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:45 smithi098 bash[23790]: audit 2024-04-02T04:57:44.291906+0000 mon.c (mon.2) 1050 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:45.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:45 smithi098 bash[23790]: audit 2024-04-02T04:57:45.000032+0000 mon.a (mon.0) 3087 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:45.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:45 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.007512+0000 mon.a (mon.0) 3088 : cluster 0 osdmap e584: 8 total, 8 up, 6 in 2024-04-02T04:57:45.310 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:45.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:45.441 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:45.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:45.544 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:45.564 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:45.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:45.686 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:45.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:45.814 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:57:45.817 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:45.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:45.951 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:45.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:46.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:46.078 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:46.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:46.198 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: cluster 2024-04-02T04:57:44.193040+0000 osd.7 (osd.7) 105 : cluster 0 6.8 scrub starts 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: cluster 2024-04-02T04:57:44.193896+0000 osd.2 (osd.2) 271 : cluster 0 43.16 scrub starts 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: cluster 2024-04-02T04:57:44.195387+0000 osd.7 (osd.7) 106 : cluster 0 6.8 scrub ok 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: cluster 2024-04-02T04:57:44.197318+0000 osd.2 (osd.2) 272 : cluster 0 43.16 scrub ok 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.084687+0000 mon.a (mon.0) 3089 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 15 pgs peering) 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:45.293310+0000 mon.c (mon.2) 1051 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:45.294820+0000 mon.c (mon.2) 1052 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:45.295718+0000 mon.a (mon.0) 3090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:46.002352+0000 mon.a (mon.0) 3091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]': finished 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:46.009121+0000 mon.c (mon.2) 1053 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:46.010055+0000 mon.c (mon.2) 1054 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:46.010627+0000 mon.c (mon.2) 1055 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.011320+0000 mon.a (mon.0) 3092 : cluster 0 osdmap e585: 8 total, 8 up, 6 in 2024-04-02T04:57:46.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:46 smithi098 bash[23790]: audit 2024-04-02T04:57:46.015977+0000 mon.a (mon.0) 3093 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:57:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:46.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:46.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: cluster 2024-04-02T04:57:44.193040+0000 osd.7 (osd.7) 105 : cluster 0 6.8 scrub starts 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: cluster 2024-04-02T04:57:44.193896+0000 osd.2 (osd.2) 271 : cluster 0 43.16 scrub starts 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: cluster 2024-04-02T04:57:44.195387+0000 osd.7 (osd.7) 106 : cluster 0 6.8 scrub ok 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: cluster 2024-04-02T04:57:44.197318+0000 osd.2 (osd.2) 272 : cluster 0 43.16 scrub ok 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.084687+0000 mon.a (mon.0) 3089 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 15 pgs peering) 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:45.293310+0000 mon.c (mon.2) 1051 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:45.294820+0000 mon.c (mon.2) 1052 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:45.295718+0000 mon.a (mon.0) 3090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:46.002352+0000 mon.a (mon.0) 3091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]': finished 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:46.009121+0000 mon.c (mon.2) 1053 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:46.010055+0000 mon.c (mon.2) 1054 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:46.010627+0000 mon.c (mon.2) 1055 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.011320+0000 mon.a (mon.0) 3092 : cluster 0 osdmap e585: 8 total, 8 up, 6 in 2024-04-02T04:57:46.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[21528]: audit 2024-04-02T04:57:46.015977+0000 mon.a (mon.0) 3093 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: cluster 2024-04-02T04:57:44.193040+0000 osd.7 (osd.7) 105 : cluster 0 6.8 scrub starts 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: cluster 2024-04-02T04:57:44.193896+0000 osd.2 (osd.2) 271 : cluster 0 43.16 scrub starts 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: cluster 2024-04-02T04:57:44.195387+0000 osd.7 (osd.7) 106 : cluster 0 6.8 scrub ok 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: cluster 2024-04-02T04:57:44.197318+0000 osd.2 (osd.2) 272 : cluster 0 43.16 scrub ok 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.084687+0000 mon.a (mon.0) 3089 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 15 pgs peering) 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:45.293310+0000 mon.c (mon.2) 1051 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:45.294820+0000 mon.c (mon.2) 1052 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:45.295718+0000 mon.a (mon.0) 3090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:46.002352+0000 mon.a (mon.0) 3091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pg_num","val":"11"}]': finished 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:46.009121+0000 mon.c (mon.2) 1053 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:46.010055+0000 mon.c (mon.2) 1054 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:46.010627+0000 mon.c (mon.2) 1055 : audit 1 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:57:46.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.011320+0000 mon.a (mon.0) 3092 : cluster 0 osdmap e585: 8 total, 8 up, 6 in 2024-04-02T04:57:46.453 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:46 smithi069 bash[28904]: audit 2024-04-02T04:57:46.015977+0000 mon.a (mon.0) 3093 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-02T04:57:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:46.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:46.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:46.726 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:46.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:46.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:46.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:46.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:47.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.225152+0000 osd.7 (osd.7) 107 : cluster 0 6.3 scrub starts 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.226946+0000 osd.7 (osd.7) 108 : cluster 0 6.3 scrub ok 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.763249+0000 osd.6 (osd.6) 263 : cluster 0 43.14 scrub starts 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.765654+0000 osd.6 (osd.6) 264 : cluster 0 43.14 scrub ok 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.941519+0000 osd.4 (osd.4) 320 : cluster 0 3.1d scrub starts 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:45.943934+0000 osd.4 (osd.4) 321 : cluster 0 3.1d scrub ok 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.019853+0000 osd.0 (osd.0) 305 : cluster 0 102.2 deep-scrub starts 2024-04-02T04:57:47.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.039568+0000 osd.0 (osd.0) 306 : cluster 0 102.2 deep-scrub ok 2024-04-02T04:57:47.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:47.241 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:47.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:47.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.225152+0000 osd.7 (osd.7) 107 : cluster 0 6.3 scrub starts 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.226946+0000 osd.7 (osd.7) 108 : cluster 0 6.3 scrub ok 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.763249+0000 osd.6 (osd.6) 263 : cluster 0 43.14 scrub starts 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.765654+0000 osd.6 (osd.6) 264 : cluster 0 43.14 scrub ok 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.941519+0000 osd.4 (osd.4) 320 : cluster 0 3.1d scrub starts 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:45.943934+0000 osd.4 (osd.4) 321 : cluster 0 3.1d scrub ok 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.019853+0000 osd.0 (osd.0) 305 : cluster 0 102.2 deep-scrub starts 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.039568+0000 osd.0 (osd.0) 306 : cluster 0 102.2 deep-scrub ok 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.208964+0000 mgr.y (mgr.14538) 552 : cluster 0 pgmap v875: 313 pgs: 313 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 64 KiB/s, 39 objects/s recovering 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: audit 2024-04-02T04:57:47.004142+0000 mon.a (mon.0) 3094 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]': finished 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: cluster 2024-04-02T04:57:47.009993+0000 mon.a (mon.0) 3095 : cluster 0 osdmap e586: 8 total, 8 up, 6 in 2024-04-02T04:57:47.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: audit 2024-04-02T04:57:47.013397+0000 mon.c (mon.2) 1056 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:47.358 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: audit 2024-04-02T04:57:47.085937+0000 mon.c (mon.2) 1057 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:47.358 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: audit 2024-04-02T04:57:47.086360+0000 mon.a (mon.0) 3096 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:47.358 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: audit 2024-04-02T04:57:47.086957+0000 mon.c (mon.2) 1058 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-85"}]: dispatch 2024-04-02T04:57:47.358 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:47 smithi098 bash[23790]: audit 2024-04-02T04:57:47.087230+0000 mon.a (mon.0) 3097 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-85"}]: dispatch 2024-04-02T04:57:47.373 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:47.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:47.505 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:47.558 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.208964+0000 mgr.y (mgr.14538) 552 : cluster 0 pgmap v875: 313 pgs: 313 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 64 KiB/s, 39 objects/s recovering 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: audit 2024-04-02T04:57:47.004142+0000 mon.a (mon.0) 3094 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]': finished 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: cluster 2024-04-02T04:57:47.009993+0000 mon.a (mon.0) 3095 : cluster 0 osdmap e586: 8 total, 8 up, 6 in 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: audit 2024-04-02T04:57:47.013397+0000 mon.c (mon.2) 1056 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: audit 2024-04-02T04:57:47.085937+0000 mon.c (mon.2) 1057 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: audit 2024-04-02T04:57:47.086360+0000 mon.a (mon.0) 3096 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: audit 2024-04-02T04:57:47.086957+0000 mon.c (mon.2) 1058 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-85"}]: dispatch 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[21528]: audit 2024-04-02T04:57:47.087230+0000 mon.a (mon.0) 3097 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-85"}]: dispatch 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.225152+0000 osd.7 (osd.7) 107 : cluster 0 6.3 scrub starts 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.226946+0000 osd.7 (osd.7) 108 : cluster 0 6.3 scrub ok 2024-04-02T04:57:47.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.763249+0000 osd.6 (osd.6) 263 : cluster 0 43.14 scrub starts 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.765654+0000 osd.6 (osd.6) 264 : cluster 0 43.14 scrub ok 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.941519+0000 osd.4 (osd.4) 320 : cluster 0 3.1d scrub starts 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:45.943934+0000 osd.4 (osd.4) 321 : cluster 0 3.1d scrub ok 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.019853+0000 osd.0 (osd.0) 305 : cluster 0 102.2 deep-scrub starts 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.039568+0000 osd.0 (osd.0) 306 : cluster 0 102.2 deep-scrub ok 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.208964+0000 mgr.y (mgr.14538) 552 : cluster 0 pgmap v875: 313 pgs: 313 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 64 KiB/s, 39 objects/s recovering 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: audit 2024-04-02T04:57:47.004142+0000 mon.a (mon.0) 3094 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-72185-1","var":"pgp_num","val":"11"}]': finished 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: cluster 2024-04-02T04:57:47.009993+0000 mon.a (mon.0) 3095 : cluster 0 osdmap e586: 8 total, 8 up, 6 in 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: audit 2024-04-02T04:57:47.013397+0000 mon.c (mon.2) 1056 : audit 0 from='client.? 172.21.15.69:0/2312672872' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: audit 2024-04-02T04:57:47.085937+0000 mon.c (mon.2) 1057 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: audit 2024-04-02T04:57:47.086360+0000 mon.a (mon.0) 3096 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: audit 2024-04-02T04:57:47.086957+0000 mon.c (mon.2) 1058 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-85"}]: dispatch 2024-04-02T04:57:47.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:47 smithi069 bash[28904]: audit 2024-04-02T04:57:47.087230+0000 mon.a (mon.0) 3097 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-85"}]: dispatch 2024-04-02T04:57:47.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:47.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:47.765 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:47.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:47.894 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:47.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:47.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:48.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:48.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:48.279 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:48.382 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:48.403 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:48.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:48.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.196310+0000 osd.2 (osd.2) 273 : cluster 0 102.11 scrub starts 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.201818+0000 osd.2 (osd.2) 274 : cluster 0 102.11 scrub ok 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.969223+0000 osd.1 (osd.1) 317 : cluster 0 6.d scrub starts 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[21528]: cluster 2024-04-02T04:57:46.972783+0000 osd.1 (osd.1) 318 : cluster 0 6.d scrub ok 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[21528]: cluster 2024-04-02T04:57:48.014044+0000 mon.a (mon.0) 3098 : cluster 0 osdmap e587: 8 total, 8 up, 6 in 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.196310+0000 osd.2 (osd.2) 273 : cluster 0 102.11 scrub starts 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.201818+0000 osd.2 (osd.2) 274 : cluster 0 102.11 scrub ok 2024-04-02T04:57:48.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.969223+0000 osd.1 (osd.1) 317 : cluster 0 6.d scrub starts 2024-04-02T04:57:48.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[28904]: cluster 2024-04-02T04:57:46.972783+0000 osd.1 (osd.1) 318 : cluster 0 6.d scrub ok 2024-04-02T04:57:48.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[28904]: cluster 2024-04-02T04:57:48.014044+0000 mon.a (mon.0) 3098 : cluster 0 osdmap e587: 8 total, 8 up, 6 in 2024-04-02T04:57:48.661 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:48.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:48.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:48 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.196310+0000 osd.2 (osd.2) 273 : cluster 0 102.11 scrub starts 2024-04-02T04:57:48.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:48 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.201818+0000 osd.2 (osd.2) 274 : cluster 0 102.11 scrub ok 2024-04-02T04:57:48.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:48 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.969223+0000 osd.1 (osd.1) 317 : cluster 0 6.d scrub starts 2024-04-02T04:57:48.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:48 smithi098 bash[23790]: cluster 2024-04-02T04:57:46.972783+0000 osd.1 (osd.1) 318 : cluster 0 6.d scrub ok 2024-04-02T04:57:48.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:48 smithi098 bash[23790]: cluster 2024-04-02T04:57:48.014044+0000 mon.a (mon.0) 3098 : cluster 0 osdmap e587: 8 total, 8 up, 6 in 2024-04-02T04:57:48.779 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:57:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:57:48.791 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:48.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:48.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:48.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:49.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:49.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:49.177 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:49.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:49.305 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:49.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:49.435 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:49.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: cluster 2024-04-02T04:57:47.179357+0000 osd.2 (osd.2) 275 : cluster 0 102.18 scrub starts 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: cluster 2024-04-02T04:57:47.182802+0000 osd.2 (osd.2) 276 : cluster 0 102.18 scrub ok 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.071651+0000 mon.c (mon.2) 1059 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: cluster 2024-04-02T04:57:48.210330+0000 mgr.y (mgr.14538) 553 : cluster 0 pgmap v878: 270 pgs: 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.475980+0000 mon.a (mon.0) 3099 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.476761+0000 mon.c (mon.2) 1060 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.483274+0000 mon.a (mon.0) 3100 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.487083+0000 mon.c (mon.2) 1061 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.494474+0000 mon.b (mon.1) 277 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.494808+0000 mon.a (mon.0) 3101 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:48.497267+0000 mon.a (mon.0) 3102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:49.018255+0000 mon.a (mon.0) 3103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.021421+0000 mon.a (mon.0) 3104 : cluster 0 osdmap e588: 8 total, 8 up, 6 in 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:49.023759+0000 mon.c (mon.2) 1062 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:49.024435+0000 mon.a (mon.0) 3105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:49.029244+0000 mon.c (mon.2) 1063 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:49.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:49 smithi098 bash[23790]: audit 2024-04-02T04:57:49.029739+0000 mon.a (mon.0) 3106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:49.565 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:49.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:49.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: cluster 2024-04-02T04:57:47.179357+0000 osd.2 (osd.2) 275 : cluster 0 102.18 scrub starts 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: cluster 2024-04-02T04:57:47.182802+0000 osd.2 (osd.2) 276 : cluster 0 102.18 scrub ok 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.071651+0000 mon.c (mon.2) 1059 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: cluster 2024-04-02T04:57:48.210330+0000 mgr.y (mgr.14538) 553 : cluster 0 pgmap v878: 270 pgs: 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.475980+0000 mon.a (mon.0) 3099 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.476761+0000 mon.c (mon.2) 1060 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.483274+0000 mon.a (mon.0) 3100 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.487083+0000 mon.c (mon.2) 1061 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.494474+0000 mon.b (mon.1) 277 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.494808+0000 mon.a (mon.0) 3101 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:57:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:48.497267+0000 mon.a (mon.0) 3102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:49.018255+0000 mon.a (mon.0) 3103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.021421+0000 mon.a (mon.0) 3104 : cluster 0 osdmap e588: 8 total, 8 up, 6 in 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:49.023759+0000 mon.c (mon.2) 1062 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:49.024435+0000 mon.a (mon.0) 3105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:49.029244+0000 mon.c (mon.2) 1063 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[21528]: audit 2024-04-02T04:57:49.029739+0000 mon.a (mon.0) 3106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: cluster 2024-04-02T04:57:47.179357+0000 osd.2 (osd.2) 275 : cluster 0 102.18 scrub starts 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: cluster 2024-04-02T04:57:47.182802+0000 osd.2 (osd.2) 276 : cluster 0 102.18 scrub ok 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.071651+0000 mon.c (mon.2) 1059 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: cluster 2024-04-02T04:57:48.210330+0000 mgr.y (mgr.14538) 553 : cluster 0 pgmap v878: 270 pgs: 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-02T04:57:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.475980+0000 mon.a (mon.0) 3099 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.476761+0000 mon.c (mon.2) 1060 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.483274+0000 mon.a (mon.0) 3100 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.487083+0000 mon.c (mon.2) 1061 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:57:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.494474+0000 mon.b (mon.1) 277 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.494808+0000 mon.a (mon.0) 3101 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:57:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:48.497267+0000 mon.a (mon.0) 3102 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:57:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:49.018255+0000 mon.a (mon.0) 3103 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-72185-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:57:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.021421+0000 mon.a (mon.0) 3104 : cluster 0 osdmap e588: 8 total, 8 up, 6 in 2024-04-02T04:57:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:49.023759+0000 mon.c (mon.2) 1062 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:49.024435+0000 mon.a (mon.0) 3105 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:49.029244+0000 mon.c (mon.2) 1063 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:49.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:49 smithi069 bash[28904]: audit 2024-04-02T04:57:49.029739+0000 mon.a (mon.0) 3106 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:49.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:49.955 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:49.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18029 ms) 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23036 ms) 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3074 ms) 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2889 ms) 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2987 ms) 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-02T04:57:50.014 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3050 ms) 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4062 ms) 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39954 ms) 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17005 ms) 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4975 ms) 2024-04-02T04:57:50.015 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3002 ms) 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25022 ms) 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14557 ms) 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5068 ms) 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3355 ms) 2024-04-02T04:57:50.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24207 ms) 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9840 ms) 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10091 ms) 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8931 ms) 2024-04-02T04:57:50.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-04-02T04:57:50.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:50.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:50.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:50.343 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:50.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:50.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:50.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:50.730 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:50.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:50.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:50.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:50 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.000222+0000 osd.4 (osd.4) 322 : cluster 0 40.1d scrub starts 2024-04-02T04:57:50.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:50 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.005781+0000 osd.4 (osd.4) 323 : cluster 0 40.1d scrub ok 2024-04-02T04:57:50.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:50 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.969549+0000 mon.a (mon.0) 3107 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:50.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:50 smithi098 bash[23790]: audit 2024-04-02T04:57:49.976169+0000 mon.a (mon.0) 3108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:50.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:50 smithi098 bash[23790]: audit 2024-04-02T04:57:49.996428+0000 mon.c (mon.2) 1064 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:50.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:50 smithi098 bash[23790]: cluster 2024-04-02T04:57:50.451349+0000 mon.a (mon.0) 3109 : cluster 0 osdmap e589: 8 total, 8 up, 6 in 2024-04-02T04:57:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:50.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:51.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.000222+0000 osd.4 (osd.4) 322 : cluster 0 40.1d scrub starts 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.005781+0000 osd.4 (osd.4) 323 : cluster 0 40.1d scrub ok 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.969549+0000 mon.a (mon.0) 3107 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[21528]: audit 2024-04-02T04:57:49.976169+0000 mon.a (mon.0) 3108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[21528]: audit 2024-04-02T04:57:49.996428+0000 mon.c (mon.2) 1064 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[21528]: cluster 2024-04-02T04:57:50.451349+0000 mon.a (mon.0) 3109 : cluster 0 osdmap e589: 8 total, 8 up, 6 in 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.000222+0000 osd.4 (osd.4) 322 : cluster 0 40.1d scrub starts 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.005781+0000 osd.4 (osd.4) 323 : cluster 0 40.1d scrub ok 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.969549+0000 mon.a (mon.0) 3107 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:51.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[28904]: audit 2024-04-02T04:57:49.976169+0000 mon.a (mon.0) 3108 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:51.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[28904]: audit 2024-04-02T04:57:49.996428+0000 mon.c (mon.2) 1064 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:57:51.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:50 smithi069 bash[28904]: cluster 2024-04-02T04:57:50.451349+0000 mon.a (mon.0) 3109 : cluster 0 osdmap e589: 8 total, 8 up, 6 in 2024-04-02T04:57:51.124 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:51.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:51.250 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:51.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:51.382 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:51.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:51.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:51.746 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:51.774 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:51.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:51.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:52.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:52.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.193651+0000 osd.7 (osd.7) 109 : cluster 0 6.b scrub starts 2024-04-02T04:57:52.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.195894+0000 osd.7 (osd.7) 110 : cluster 0 6.b scrub ok 2024-04-02T04:57:52.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.794666+0000 osd.6 (osd.6) 265 : cluster 0 102.1d scrub starts 2024-04-02T04:57:52.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:49.799501+0000 osd.6 (osd.6) 266 : cluster 0 102.1d scrub ok 2024-04-02T04:57:52.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:50.041774+0000 osd.0 (osd.0) 307 : cluster 0 2.1e deep-scrub starts 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:50.043589+0000 osd.0 (osd.0) 308 : cluster 0 2.1e deep-scrub ok 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:50.211167+0000 mgr.y (mgr.14538) 554 : cluster 0 pgmap v881: 302 pgs: 8 creating+peering, 24 unknown, 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: audit 2024-04-02T04:57:50.979884+0000 mon.a (mon.0) 3110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: cluster 2024-04-02T04:57:50.990620+0000 mon.a (mon.0) 3111 : cluster 0 osdmap e590: 8 total, 8 up, 6 in 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: audit 2024-04-02T04:57:50.995272+0000 mon.c (mon.2) 1065 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[21528]: audit 2024-04-02T04:57:50.998573+0000 mon.a (mon.0) 3112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.193651+0000 osd.7 (osd.7) 109 : cluster 0 6.b scrub starts 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.195894+0000 osd.7 (osd.7) 110 : cluster 0 6.b scrub ok 2024-04-02T04:57:52.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.794666+0000 osd.6 (osd.6) 265 : cluster 0 102.1d scrub starts 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:49.799501+0000 osd.6 (osd.6) 266 : cluster 0 102.1d scrub ok 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:50.041774+0000 osd.0 (osd.0) 307 : cluster 0 2.1e deep-scrub starts 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:50.043589+0000 osd.0 (osd.0) 308 : cluster 0 2.1e deep-scrub ok 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:50.211167+0000 mgr.y (mgr.14538) 554 : cluster 0 pgmap v881: 302 pgs: 8 creating+peering, 24 unknown, 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: audit 2024-04-02T04:57:50.979884+0000 mon.a (mon.0) 3110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: cluster 2024-04-02T04:57:50.990620+0000 mon.a (mon.0) 3111 : cluster 0 osdmap e590: 8 total, 8 up, 6 in 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: audit 2024-04-02T04:57:50.995272+0000 mon.c (mon.2) 1065 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:52.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:51 smithi069 bash[28904]: audit 2024-04-02T04:57:50.998573+0000 mon.a (mon.0) 3112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:52.043 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:52.103 INFO:tasks.workunit.client.0.smithi069.stdout: try=15 expected=15 2024-04-02T04:57:52.103 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=0 expected=0 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=12 expected=12 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=4 expected=4 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-04-02T04:57:52.104 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=3 expected=3 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=6 expected=6 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=14 expected=14 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=7 expected=7 2024-04-02T04:57:52.105 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 13:566253c9:::13:head 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=13 expected=13 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (645 ms) 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (358438 ms) 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (141118 ms) 2024-04-02T04:57:52.106 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList (501061 ms total) 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1075 ms) 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2024-04-02T04:57:52.107 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2024-04-02T04:57:52.108 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (41 ms) 2024-04-02T04:57:52.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:52.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.193651+0000 osd.7 (osd.7) 109 : cluster 0 6.b scrub starts 2024-04-02T04:57:52.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.195894+0000 osd.7 (osd.7) 110 : cluster 0 6.b scrub ok 2024-04-02T04:57:52.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.794666+0000 osd.6 (osd.6) 265 : cluster 0 102.1d scrub starts 2024-04-02T04:57:52.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:49.799501+0000 osd.6 (osd.6) 266 : cluster 0 102.1d scrub ok 2024-04-02T04:57:52.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:50.041774+0000 osd.0 (osd.0) 307 : cluster 0 2.1e deep-scrub starts 2024-04-02T04:57:52.158 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:50.043589+0000 osd.0 (osd.0) 308 : cluster 0 2.1e deep-scrub ok 2024-04-02T04:57:52.159 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:50.211167+0000 mgr.y (mgr.14538) 554 : cluster 0 pgmap v881: 302 pgs: 8 creating+peering, 24 unknown, 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-02T04:57:52.159 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: audit 2024-04-02T04:57:50.979884+0000 mon.a (mon.0) 3110 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-72185-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:52.159 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: cluster 2024-04-02T04:57:50.990620+0000 mon.a (mon.0) 3111 : cluster 0 osdmap e590: 8 total, 8 up, 6 in 2024-04-02T04:57:52.159 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: audit 2024-04-02T04:57:50.995272+0000 mon.c (mon.2) 1065 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:52.159 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:51 smithi098 bash[23790]: audit 2024-04-02T04:57:50.998573+0000 mon.a (mon.0) 3112 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:57:52.172 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:52.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:52.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:52.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:52.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- 172.21.15.69:0/2160041034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 msgr2=0x7f60ac106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/2160041034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac106ea0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f60a80099c0 tx=0x7f60a8009cd0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- 172.21.15.69:0/2160041034 shutdown_connections 2024-04-02T04:57:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/2160041034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac107580 0x7f60ac10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/2160041034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac06a1d0 0x7f60ac06a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/2160041034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac106ea0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- 172.21.15.69:0/2160041034 >> 172.21.15.69:0/2160041034 conn(0x7f60ac0771e0 msgr2=0x7f60ac0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- 172.21.15.69:0/2160041034 shutdown_connections 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- 172.21.15.69:0/2160041034 wait complete. 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 librados: starting msgr at 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 librados: starting objecter 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 Processor -- start 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- start start 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 librados: setting wanted keys 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 librados: calling monclient init 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac06a1d0 0x7f60ac197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac107580 0x7f60ac19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac111a20 con 0x7f60ac107580 2024-04-02T04:57:52.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1118a0 con 0x7f60ac06a1d0 2024-04-02T04:57:52.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b3d1f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f60ac111ba0 con 0x7f60ac06ab80 2024-04-02T04:57:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b22bc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac107580 0x7f60ac19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b22bc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac107580 0x7f60ac19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36284/0 (socket says 172.21.15.69:36284) 2024-04-02T04:57:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b12ba700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b22bc700 1 -- 172.21.15.69:0/3851398435 learned_addr learned my addr 172.21.15.69:0/3851398435 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.682+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac06a1d0 0x7f60ac197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b12ba700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac06a1d0 msgr2=0x7f60ac197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac06a1d0 0x7f60ac197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b12ba700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac107580 msgr2=0x7f60ac19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac107580 0x7f60ac19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b12ba700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac19cfb0 con 0x7f60ac06ab80 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ac107580 0x7f60ac19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac1984e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f60ac069d90 tx=0x7f60a800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 10 client.?.objecter ms_handle_connect 0x7f60ac06ab80 2024-04-02T04:57:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8027070 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f60a800fc30 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ac19d240 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 10 client.26741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8022be0 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ac1a4b20 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 1 librados: init done 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f60ac103980 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f60a802a070 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f609806f4b0 0x7f6098071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(591..591 src has 74..591) ==== 11419+0+0 (secure 0 0 0) 0x7f60a808f9a0 con 0x7f60ac06ab80 2024-04-02T04:57:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 osd_map(591..591 src has 74..591) 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 3 client.26741.objecter handle_osd_map got epochs [591,591] > 0 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 3 client.26741.objecter handle_osd_map decoding full epoch 591 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f609806f4b0 0x7f6098071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 20 client.26741.objecter dump_active .. 0 homeless 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v591) ==== 54+0+16 (secure 0 0 0) 0x7f60a8013070 con 0x7f60ac06ab80 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 1 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f60ac062e50 0x7f60ac063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 1 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 0 1 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 4 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 10 client.26741.objecter _send_command 1 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60ac199300 con 0x7f60ac062e50 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f60ac062e50 0x7f60ac063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f609806f4b0 0x7f6098071970 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f609c009c60 tx=0x7f609c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.686+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f609806f4b0 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac062e50 2024-04-02T04:57:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f60ac062e50 0x7f60ac063230 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac199300 con 0x7f60ac062e50 2024-04-02T04:57:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 1: 0 ) 2024-04-02T04:57:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 1 = system:0 2024-04-02T04:57:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 4 2024-04-02T04:57:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.690+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 0 1 2024-04-02T04:57:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:52.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 2 2024-04-02T04:57:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 2 2024-04-02T04:57:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac04ff00 osd=0 3 2024-04-02T04:57:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 0 2 2024-04-02T04:57:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 4 2024-04-02T04:57:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 10 client.26741.objecter _send_command 2 2024-04-02T04:57:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.706+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac0628d0 con 0x7f60ac062e50 2024-04-02T04:57:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac0628d0 con 0x7f60ac062e50 2024-04-02T04:57:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 4 2024-04-02T04:57:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 0 2 2024-04-02T04:57:52.709 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 3 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f60ac02c9b0 0x7f60ac02cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 3 2024-04-02T04:57:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 1 3 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 4 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 10 client.26741.objecter _send_command 3 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f60ac1031b0 con 0x7f60ac02c9b0 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f60ac02c9b0 0x7f60ac02cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac02c9b0 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f60ac02c9b0 0x7f60ac02cd90 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac1031b0 con 0x7f60ac02c9b0 2024-04-02T04:57:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 3: 0 ) 2024-04-02T04:57:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 3 = system:0 2024-04-02T04:57:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 4 2024-04-02T04:57:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.710+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 1 3 2024-04-02T04:57:52.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 4 2024-04-02T04:57:52.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 4 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac109870 osd=1 3 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 1 4 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 4 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 10 client.26741.objecter _send_command 4 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.726+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac02d2d0 con 0x7f60ac02c9b0 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac02d2d0 con 0x7f60ac02c9b0 2024-04-02T04:57:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 4 2024-04-02T04:57:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 1 4 2024-04-02T04:57:52.731 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:57:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 5 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f60ac1ac290 0x7f60ac1ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 5 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 2 5 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 4 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 10 client.26741.objecter _send_command 5 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f60ac02ddc0 con 0x7f60ac1ac290 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f60ac1ac290 0x7f60ac1ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f60ac1ac290 0x7f60ac1ae680 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac1ac290 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac02ddc0 con 0x7f60ac1ac290 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 5: 0 ) 2024-04-02T04:57:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 5 = system:0 2024-04-02T04:57:52.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 4 2024-04-02T04:57:52.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.730+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 2 5 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 6 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 6 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac02dc50 osd=2 3 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 2 6 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 4 2024-04-02T04:57:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 10 client.26741.objecter _send_command 6 2024-04-02T04:57:52.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac02d6d0 con 0x7f60ac1ac290 2024-04-02T04:57:52.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac02d6d0 con 0x7f60ac1ac290 2024-04-02T04:57:52.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 4 2024-04-02T04:57:52.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 2 6 2024-04-02T04:57:52.752 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 7 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f60ac1a79e0 0x7f60ac1b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 7 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 3 7 2024-04-02T04:57:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 4 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 10 client.26741.objecter _send_command 7 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f60ac1b4bb0 con 0x7f60ac1a79e0 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f60ac1a79e0 0x7f60ac1b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac1a79e0 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.750+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f60ac1a79e0 0x7f60ac1b1d50 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.754+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac1b4bb0 con 0x7f60ac1a79e0 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.754+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 7: 0 ) 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.754+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 7 = system:0 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.754+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 4 2024-04-02T04:57:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.754+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 3 7 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- 172.21.15.69:0/459135832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 msgr2=0x7f02c81027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/459135832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c81027a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0098f0 tx=0x7f02bc009c00 comp rx=0 tx=0).stop 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- 172.21.15.69:0/459135832 shutdown_connections 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/459135832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8102ce0 0x7f02c810f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/459135832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8108390 0x7f02c8108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/459135832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c81027a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- 172.21.15.69:0/459135832 >> 172.21.15.69:0/459135832 conn(0x7f02c8079310 msgr2=0x7f02c8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:52.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- 172.21.15.69:0/459135832 shutdown_connections 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- 172.21.15.69:0/459135832 wait complete. 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 librados: starting msgr at 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 librados: starting objecter 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 Processor -- start 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- start start 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 librados: setting wanted keys 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 librados: calling monclient init 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8102ce0 0x7f02c819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 0x7f02c819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8111b00 con 0x7f02c8108390 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8111980 con 0x7f02c8102ce0 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02ce1f2700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f02c8111c80 con 0x7f02c8102320 2024-04-02T04:57:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 0x7f02c819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 0x7f02c819ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36294/0 (socket says 172.21.15.69:36294) 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55788/0 (socket says 172.21.15.69:55788) 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 -- 172.21.15.69:0/2183288011 learned_addr learned my addr 172.21.15.69:0/2183288011 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c6ffd700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8102ce0 0x7f02c819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 -- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 msgr2=0x7f02c819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 -- 172.21.15.69:0/2183288011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8102ce0 msgr2=0x7f02c819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8102ce0 0x7f02c819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 -- 172.21.15.69:0/2183288011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c819f370 con 0x7f02c8108390 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c6ffd700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8102ce0 0x7f02c819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c77fe700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:57:52.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c7fff700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 0x7f02c819ec30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f02c000e3c0 tx=0x7f02c000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c4ff9700 10 client.?.objecter ms_handle_connect 0x7f02c8108390 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.762+0000 7f02c4ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c00092d0 con 0x7f02c8108390 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c819f660 con 0x7f02c8108390 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f02c000ee70 con 0x7f02c8108390 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0007d40 con 0x7f02c8108390 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02ce1f2700 10 client.16656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c81a6ee0 con 0x7f02c8108390 2024-04-02T04:57:52.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02ce1f2700 1 librados: init done 2024-04-02T04:57:52.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f02c0012050 con 0x7f02c8108390 2024-04-02T04:57:52.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c804ff00 con 0x7f02c8108390 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f02b006f5e0 0x7f02b0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(591..591 src has 74..591) ==== 11419+0+0 (secure 0 0 0) 0x7f02c008e9b0 con 0x7f02c8108390 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c77fe700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f02b006f5e0 0x7f02b0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 10 client.16656.objecter ms_dispatch 0x7f02c8001550 osd_map(591..591 src has 74..591) 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 3 client.16656.objecter handle_osd_map got epochs [591,591] > 0 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 3 client.16656.objecter handle_osd_map decoding full epoch 591 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 20 client.16656.objecter dump_active .. 0 homeless 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c4ff9700 10 client.16656.objecter ms_handle_connect 0x7f02b006f5e0 2024-04-02T04:57:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.766+0000 7f02c77fe700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f02b006f5e0 0x7f02b0071aa0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005950 tx=0x7f02b800a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.770+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f02c0010040 con 0x7f02c8108390 2024-04-02T04:57:52.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 8 2024-04-02T04:57:52.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 8 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac02dad0 osd=3 3 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 3 8 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 4 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 10 client.26741.objecter _send_command 8 2024-04-02T04:57:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac1a7010 con 0x7f60ac1a79e0 2024-04-02T04:57:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac1a7010 con 0x7f60ac1a79e0 2024-04-02T04:57:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 4 2024-04-02T04:57:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 3 8 2024-04-02T04:57:52.776 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:57:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 9 2024-04-02T04:57:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f60ac1af8a0 0x7f60ac1b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 9 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 4 9 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 4 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 10 client.26741.objecter _send_command 9 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f60ac1b8280 con 0x7f60ac1af8a0 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.774+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f60ac1af8a0 0x7f60ac1b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f60ac1af8a0 0x7f60ac1b5420 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac1af8a0 2024-04-02T04:57:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac1b8280 con 0x7f60ac1af8a0 2024-04-02T04:57:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 9: 0 ) 2024-04-02T04:57:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 9 = system:0 2024-04-02T04:57:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 4 2024-04-02T04:57:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.778+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 4 9 2024-04-02T04:57:52.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:52.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 10 2024-04-02T04:57:52.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 10 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1a6dc0 osd=4 3 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 4 10 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 4 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 10 client.26741.objecter _send_command 10 2024-04-02T04:57:52.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.798+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac1aee10 con 0x7f60ac1af8a0 2024-04-02T04:57:52.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac1aee10 con 0x7f60ac1af8a0 2024-04-02T04:57:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 4 2024-04-02T04:57:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 4 10 2024-04-02T04:57:52.801 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 11 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f60ac1aef80 0x7f60ac1b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 11 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 5 11 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 4 2024-04-02T04:57:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 10 client.26741.objecter _send_command 11 2024-04-02T04:57:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f60ac1b31e0 con 0x7f60ac1aef80 2024-04-02T04:57:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f60ac1aef80 0x7f60ac1b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac1aef80 2024-04-02T04:57:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60b12ba700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f60ac1aef80 0x7f60ac1b8af0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac1b31e0 con 0x7f60ac1aef80 2024-04-02T04:57:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 11: 0 ) 2024-04-02T04:57:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 11 = system:0 2024-04-02T04:57:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 4 2024-04-02T04:57:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.802+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 5 11 2024-04-02T04:57:52.814 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 12 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 12 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1aee10 osd=5 3 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 5 12 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 4 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 10 client.26741.objecter _send_command 12 2024-04-02T04:57:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.822+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac1b3240 con 0x7f60ac1aef80 2024-04-02T04:57:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac1b3240 con 0x7f60ac1aef80 2024-04-02T04:57:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 4 2024-04-02T04:57:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 5 12 2024-04-02T04:57:52.827 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 13 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f60ac1bad90 0x7f60ac1bb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 13 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 6 13 2024-04-02T04:57:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 4 2024-04-02T04:57:52.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 10 client.26741.objecter _send_command 13 2024-04-02T04:57:52.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f60ac1beb10 con 0x7f60ac1bad90 2024-04-02T04:57:52.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f60ac1bad90 0x7f60ac1bb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac1bad90 2024-04-02T04:57:52.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.826+0000 7f60b22bc700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f60ac1bad90 0x7f60ac1bb1d0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.830+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac1beb10 con 0x7f60ac1bad90 2024-04-02T04:57:52.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.830+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 13: 0 ) 2024-04-02T04:57:52.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.830+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 13 = system:0 2024-04-02T04:57:52.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.830+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 4 2024-04-02T04:57:52.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.830+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 6 13 2024-04-02T04:57:52.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 14 2024-04-02T04:57:52.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 14 2024-04-02T04:57:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1b3240 osd=6 3 2024-04-02T04:57:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 6 14 2024-04-02T04:57:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 4 2024-04-02T04:57:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 10 client.26741.objecter _send_command 14 2024-04-02T04:57:52.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.858+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac1bd990 con 0x7f60ac1bad90 2024-04-02T04:57:52.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac1bd990 con 0x7f60ac1bad90 2024-04-02T04:57:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 4 2024-04-02T04:57:52.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 6 14 2024-04-02T04:57:52.865 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 15 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f60ac1be0b0 0x7f60ac1be510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 15 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 7 15 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 4 2024-04-02T04:57:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 10 client.26741.objecter _send_command 15 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f60ac1c1ed0 con 0x7f60ac1be0b0 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.862+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f60ac1be0b0 0x7f60ac1be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60a2ffd700 10 client.26741.objecter ms_handle_connect 0x7f60ac1be0b0 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60b1abb700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f60ac1be0b0 0x7f60ac1be510 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60ac1c1ed0 con 0x7f60ac1be0b0 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 15: 0 ) 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 15 = system:0 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 4 2024-04-02T04:57:52.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.866+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 7 15 2024-04-02T04:57:52.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 10 client.26741.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:57:52.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign -1 16 2024-04-02T04:57:52.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 20 client.26741.objecter _get_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_remove -1 16 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 20 client.26741.objecter get_session s=0x7f60ac1bd990 osd=7 3 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 15 client.26741.objecter _session_command_op_assign 7 16 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 4 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 10 client.26741.objecter _send_command 16 2024-04-02T04:57:52.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.902+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f60ac1b9030 con 0x7f60ac1be0b0 2024-04-02T04:57:52.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60a2ffd700 1 -- 172.21.15.69:0/3851398435 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f60ac1b9030 con 0x7f60ac1be0b0 2024-04-02T04:57:52.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60a2ffd700 10 client.26741.objecter ms_dispatch 0x7f60ac001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:57:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60a2ffd700 10 client.26741.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:57:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60a2ffd700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 4 2024-04-02T04:57:52.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60a2ffd700 15 client.26741.objecter _session_command_op_remove 7 16 2024-04-02T04:57:52.906 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:57:52.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 librados: watch_flush enter 2024-04-02T04:57:52.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 librados: watch_flush exit 2024-04-02T04:57:52.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.0 2024-04-02T04:57:52.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f60ac062e50 msgr2=0x7f60ac063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f60ac062e50 0x7f60ac063230 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac04ff00 osd=0 1 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.1 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f60ac02c9b0 msgr2=0x7f60ac02cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f60ac02c9b0 0x7f60ac02cd90 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac109870 osd=1 1 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.2 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f60ac1ac290 msgr2=0x7f60ac1ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f60ac1ac290 0x7f60ac1ae680 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dc50 osd=2 1 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.3 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f60ac1a79e0 msgr2=0x7f60ac1b1d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f60ac1a79e0 0x7f60ac1b1d50 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac02dad0 osd=3 1 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.4 2024-04-02T04:57:52.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f60ac1af8a0 msgr2=0x7f60ac1b5420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f60ac1af8a0 0x7f60ac1b5420 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b22bc700 1 -- 172.21.15.69:0/3851398435 reap_dead start 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1a6dc0 osd=4 1 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.5 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f60ac1aef80 msgr2=0x7f60ac1b8af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b22bc700 1 -- 172.21.15.69:0/3851398435 reap_dead start 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f60ac1aef80 0x7f60ac1b8af0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1aee10 osd=5 1 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.6 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f60ac1bad90 msgr2=0x7f60ac1bb1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b22bc700 1 -- 172.21.15.69:0/3851398435 reap_dead start 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f60ac1bad90 0x7f60ac1bb1d0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1b3240 osd=6 1 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter close_session for osd.7 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f60ac1be0b0 msgr2=0x7f60ac1be510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f60ac1be0b0 0x7f60ac1be510 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter put_session s=0x7f60ac1bd990 osd=7 1 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 20 client.26741.objecter shutdown clearing up homeless session... 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 10 client.26741.objecter successfully canceled tick 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f609806f4b0 msgr2=0x7f6098071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f609806f4b0 0x7f6098071970 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f609c009c60 tx=0x7f609c009260 comp rx=0 tx=0).stop 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 msgr2=0x7f60ac1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac1984e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f60ac069d90 tx=0x7f60a800fa00 comp rx=0 tx=0).stop 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 shutdown_connections 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f60ac1be0b0 0x7f60ac1be510 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f60ac1bad90 0x7f60ac1bb1d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ac06ab80 0x7f60ac1984e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 --2- 172.21.15.69:0/3851398435 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f609806f4b0 0x7f6098071970 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 >> 172.21.15.69:0/3851398435 conn(0x7f60ac0771e0 msgr2=0x7f60ac0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 shutdown_connections 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 -- 172.21.15.69:0/3851398435 wait complete. 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:52.906+0000 7f60b3d1f700 1 librados: shutdown 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:57:52.911 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:57:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:57:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:57:52.912 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:57:52.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:52.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:53.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.026+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) -- 0x7f02c8075b90 con 0x7f02b006f5e0 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02c4ff9700 1 -- 172.21.15.69:0/2183288011 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 2 no change) ==== 17+0+362 (secure 0 0 0) 0x7f02c8075b90 con 0x7f02b006f5e0 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout:moved 52 / 1002 (5.18962%) 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout:avg 167 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout:stddev 15.843 -> 28.396 (expected baseline 11.7969) 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout:min osd.2 with 144 -> 154 pgs (0.862275 -> 0.922156 * mean) 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout:max osd.4 with 191 -> 198 pgs (1.14371 -> 1.18563 * mean) 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:57:53.030 INFO:teuthology.orchestra.run.smithi069.stdout:oload 110 2024-04-02T04:57:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:max_change 3 2024-04-02T04:57:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:max_change_osds 4 2024-04-02T04:57:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:average_utilization 0.0008 2024-04-02T04:57:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:overload_utilization 0.0009 2024-04-02T04:57:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 weight 0.8233 -> 0.6865 2024-04-02T04:57:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 weight 0.8671 -> 0.7259 2024-04-02T04:57:53.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 10 librados: watch_flush enter 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 10 librados: watch_flush exit 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 20 client.16656.objecter shutdown clearing up homeless session... 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 10 client.16656.objecter successfully canceled tick 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f02b006f5e0 msgr2=0x7f02b0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f02b006f5e0 0x7f02b0071aa0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005950 tx=0x7f02b800a300 comp rx=0 tx=0).stop 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 msgr2=0x7f02c819ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 0x7f02c819ec30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f02c000e3c0 tx=0x7f02c000e6d0 comp rx=0 tx=0).stop 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 shutdown_connections 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f02b006f5e0 0x7f02b0071aa0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02c8108390 0x7f02c819ec30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:53.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02c8102320 0x7f02c819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 --2- 172.21.15.69:0/2183288011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02c8102ce0 0x7f02c819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.030+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 >> 172.21.15.69:0/2183288011 conn(0x7f02c8079310 msgr2=0x7f02c8074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.034+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 shutdown_connections 2024-04-02T04:57:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.034+0000 7f02ce1f2700 1 -- 172.21.15.69:0/2183288011 wait complete. 2024-04-02T04:57:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:53.034+0000 7f02ce1f2700 1 librados: shutdown 2024-04-02T04:57:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:no change 2024-04-02T04:57:53.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:53.065 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:53 smithi069 bash[21782]: debug 2024-04-02T04:57:53.030+0000 7f42d9757700 -1 mgr.server reply reply (2) No such file or directory no change 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: cluster 2024-04-02T04:57:51.290136+0000 osd.2 (osd.2) 277 : cluster 0 40.10 scrub starts 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: cluster 2024-04-02T04:57:51.292507+0000 osd.2 (osd.2) 278 : cluster 0 40.10 scrub ok 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: cluster 2024-04-02T04:57:51.962913+0000 osd.5 (osd.5) 255 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: cluster 2024-04-02T04:57:51.963286+0000 osd.5 (osd.5) 256 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: audit 2024-04-02T04:57:51.983480+0000 mon.a (mon.0) 3113 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:53.116 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: cluster 2024-04-02T04:57:51.992004+0000 mon.a (mon.0) 3114 : cluster 0 osdmap e591: 8 total, 8 up, 6 in 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: audit 2024-04-02T04:57:52.006612+0000 mon.c (mon.2) 1066 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]: dispatch 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: audit 2024-04-02T04:57:52.008162+0000 mon.a (mon.0) 3115 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]: dispatch 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: cluster 2024-04-02T04:57:52.212262+0000 mgr.y (mgr.14538) 555 : cluster 0 pgmap v884: 342 pgs: 8 creating+peering, 64 unknown, 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[28904]: audit 2024-04-02T04:57:52.690427+0000 mon.c (mon.2) 1067 : audit 0 from='client.? 172.21.15.69:0/3851398435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: cluster 2024-04-02T04:57:51.290136+0000 osd.2 (osd.2) 277 : cluster 0 40.10 scrub starts 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: cluster 2024-04-02T04:57:51.292507+0000 osd.2 (osd.2) 278 : cluster 0 40.10 scrub ok 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: cluster 2024-04-02T04:57:51.962913+0000 osd.5 (osd.5) 255 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: cluster 2024-04-02T04:57:51.963286+0000 osd.5 (osd.5) 256 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: audit 2024-04-02T04:57:51.983480+0000 mon.a (mon.0) 3113 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: cluster 2024-04-02T04:57:51.992004+0000 mon.a (mon.0) 3114 : cluster 0 osdmap e591: 8 total, 8 up, 6 in 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: audit 2024-04-02T04:57:52.006612+0000 mon.c (mon.2) 1066 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]: dispatch 2024-04-02T04:57:53.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: audit 2024-04-02T04:57:52.008162+0000 mon.a (mon.0) 3115 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]: dispatch 2024-04-02T04:57:53.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: cluster 2024-04-02T04:57:52.212262+0000 mgr.y (mgr.14538) 555 : cluster 0 pgmap v884: 342 pgs: 8 creating+peering, 64 unknown, 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:57:53.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:52 smithi069 bash[21528]: audit 2024-04-02T04:57:52.690427+0000 mon.c (mon.2) 1067 : audit 0 from='client.? 172.21.15.69:0/3851398435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:53.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:53.187 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: cluster 2024-04-02T04:57:51.290136+0000 osd.2 (osd.2) 277 : cluster 0 40.10 scrub starts 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: cluster 2024-04-02T04:57:51.292507+0000 osd.2 (osd.2) 278 : cluster 0 40.10 scrub ok 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: cluster 2024-04-02T04:57:51.962913+0000 osd.5 (osd.5) 255 : cluster 0 purged_snaps scrub starts 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: cluster 2024-04-02T04:57:51.963286+0000 osd.5 (osd.5) 256 : cluster 0 purged_snaps scrub ok 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: audit 2024-04-02T04:57:51.983480+0000 mon.a (mon.0) 3113 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: cluster 2024-04-02T04:57:51.992004+0000 mon.a (mon.0) 3114 : cluster 0 osdmap e591: 8 total, 8 up, 6 in 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: audit 2024-04-02T04:57:52.006612+0000 mon.c (mon.2) 1066 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]: dispatch 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: audit 2024-04-02T04:57:52.008162+0000 mon.a (mon.0) 3115 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]: dispatch 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: cluster 2024-04-02T04:57:52.212262+0000 mgr.y (mgr.14538) 555 : cluster 0 pgmap v884: 342 pgs: 8 creating+peering, 64 unknown, 270 active+clean; 8.4 MiB data, 460 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:57:53.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:52 smithi098 bash[23790]: audit 2024-04-02T04:57:52.690427+0000 mon.c (mon.2) 1067 : audit 0 from='client.? 172.21.15.69:0/3851398435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:57:53.267 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:57:53.267 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:57:53.267 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2024-04-02T04:57:53.267 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:57:53.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:53.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:53.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:53.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:53.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:57:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:57:53.707 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:53.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:53.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:53.967 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:53.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:54.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:54.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:54.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.221 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:52.991431+0000 mon.a (mon.0) 3116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]': finished 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: cluster 2024-04-02T04:57:53.001224+0000 mon.a (mon.0) 3117 : cluster 0 osdmap e592: 8 total, 8 up, 6 in 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:53.002141+0000 mon.c (mon.2) 1068 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:53.002874+0000 mon.c (mon.2) 1069 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:53.005121+0000 mon.a (mon.0) 3118 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:53.005391+0000 mon.a (mon.0) 3119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: cluster 2024-04-02T04:57:53.019320+0000 osd.0 (osd.0) 309 : cluster 0 10.8 scrub starts 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: cluster 2024-04-02T04:57:53.023658+0000 osd.0 (osd.0) 310 : cluster 0 10.8 scrub ok 2024-04-02T04:57:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:53.030997+0000 mgr.y (mgr.14538) 556 : audit 0 from='client.16656 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[21528]: audit 2024-04-02T04:57:53.446610+0000 mgr.y (mgr.14538) 557 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:52.991431+0000 mon.a (mon.0) 3116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]': finished 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: cluster 2024-04-02T04:57:53.001224+0000 mon.a (mon.0) 3117 : cluster 0 osdmap e592: 8 total, 8 up, 6 in 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:53.002141+0000 mon.c (mon.2) 1068 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:53.002874+0000 mon.c (mon.2) 1069 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:53.005121+0000 mon.a (mon.0) 3118 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:53.005391+0000 mon.a (mon.0) 3119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: cluster 2024-04-02T04:57:53.019320+0000 osd.0 (osd.0) 309 : cluster 0 10.8 scrub starts 2024-04-02T04:57:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: cluster 2024-04-02T04:57:53.023658+0000 osd.0 (osd.0) 310 : cluster 0 10.8 scrub ok 2024-04-02T04:57:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:53.030997+0000 mgr.y (mgr.14538) 556 : audit 0 from='client.16656 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:54 smithi069 bash[28904]: audit 2024-04-02T04:57:53.446610+0000 mgr.y (mgr.14538) 557 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:54.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:54.351 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:54.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:52.991431+0000 mon.a (mon.0) 3116 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_tier","val": "test-rados-api-smithi069-72349-89-test-flush"}]': finished 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: cluster 2024-04-02T04:57:53.001224+0000 mon.a (mon.0) 3117 : cluster 0 osdmap e592: 8 total, 8 up, 6 in 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:53.002141+0000 mon.c (mon.2) 1068 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:53.002874+0000 mon.c (mon.2) 1069 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:53.005121+0000 mon.a (mon.0) 3118 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:53.005391+0000 mon.a (mon.0) 3119 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: cluster 2024-04-02T04:57:53.019320+0000 osd.0 (osd.0) 309 : cluster 0 10.8 scrub starts 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: cluster 2024-04-02T04:57:53.023658+0000 osd.0 (osd.0) 310 : cluster 0 10.8 scrub ok 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:53.030997+0000 mgr.y (mgr.14538) 556 : audit 0 from='client.16656 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:54.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:54 smithi098 bash[23790]: audit 2024-04-02T04:57:53.446610+0000 mgr.y (mgr.14538) 557 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:57:54.481 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:54.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:54.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.741 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:54.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:54.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:54.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:54.997 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:55.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:55.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: cluster 2024-04-02T04:57:53.310361+0000 osd.2 (osd.2) 279 : cluster 0 43.7 scrub starts 2024-04-02T04:57:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: cluster 2024-04-02T04:57:53.311487+0000 osd.2 (osd.2) 280 : cluster 0 43.7 scrub ok 2024-04-02T04:57:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:53.993559+0000 mon.a (mon.0) 3120 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:53.993634+0000 mon.a (mon.0) 3121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:53.999321+0000 mon.c (mon.2) 1070 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:53.999469+0000 mon.c (mon.2) 1071 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: cluster 2024-04-02T04:57:54.002458+0000 mon.a (mon.0) 3122 : cluster 0 osdmap e593: 8 total, 8 up, 6 in 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:54.003870+0000 mon.a (mon.0) 3123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:54.004193+0000 mon.a (mon.0) 3124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: cluster 2024-04-02T04:57:54.213754+0000 mgr.y (mgr.14538) 558 : cluster 0 pgmap v887: 334 pgs: 334 active+clean; 8.4 MiB data, 469 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: cluster 2024-04-02T04:57:54.975221+0000 mon.a (mon.0) 3125 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:54.997236+0000 mon.a (mon.0) 3126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: audit 2024-04-02T04:57:54.997381+0000 mon.a (mon.0) 3127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.002091+0000 mon.a (mon.0) 3128 : cluster 0 osdmap e594: 8 total, 8 up, 6 in 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: cluster 2024-04-02T04:57:53.310361+0000 osd.2 (osd.2) 279 : cluster 0 43.7 scrub starts 2024-04-02T04:57:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: cluster 2024-04-02T04:57:53.311487+0000 osd.2 (osd.2) 280 : cluster 0 43.7 scrub ok 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:53.993559+0000 mon.a (mon.0) 3120 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:53.993634+0000 mon.a (mon.0) 3121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:53.999321+0000 mon.c (mon.2) 1070 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:53.999469+0000 mon.c (mon.2) 1071 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: cluster 2024-04-02T04:57:54.002458+0000 mon.a (mon.0) 3122 : cluster 0 osdmap e593: 8 total, 8 up, 6 in 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:54.003870+0000 mon.a (mon.0) 3123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:54.004193+0000 mon.a (mon.0) 3124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: cluster 2024-04-02T04:57:54.213754+0000 mgr.y (mgr.14538) 558 : cluster 0 pgmap v887: 334 pgs: 334 active+clean; 8.4 MiB data, 469 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: cluster 2024-04-02T04:57:54.975221+0000 mon.a (mon.0) 3125 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:54.997236+0000 mon.a (mon.0) 3126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: audit 2024-04-02T04:57:54.997381+0000 mon.a (mon.0) 3127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:55 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.002091+0000 mon.a (mon.0) 3128 : cluster 0 osdmap e594: 8 total, 8 up, 6 in 2024-04-02T04:57:55.251 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:55.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:55.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: cluster 2024-04-02T04:57:53.310361+0000 osd.2 (osd.2) 279 : cluster 0 43.7 scrub starts 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: cluster 2024-04-02T04:57:53.311487+0000 osd.2 (osd.2) 280 : cluster 0 43.7 scrub ok 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:53.993559+0000 mon.a (mon.0) 3120 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:53.993634+0000 mon.a (mon.0) 3121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:53.999321+0000 mon.c (mon.2) 1070 : audit 1 from='client.? 172.21.15.69:0/2642992390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:53.999469+0000 mon.c (mon.2) 1071 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: cluster 2024-04-02T04:57:54.002458+0000 mon.a (mon.0) 3122 : cluster 0 osdmap e593: 8 total, 8 up, 6 in 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:54.003870+0000 mon.a (mon.0) 3123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]: dispatch 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:54.004193+0000 mon.a (mon.0) 3124 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: cluster 2024-04-02T04:57:54.213754+0000 mgr.y (mgr.14538) 558 : cluster 0 pgmap v887: 334 pgs: 334 active+clean; 8.4 MiB data, 469 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: cluster 2024-04-02T04:57:54.975221+0000 mon.a (mon.0) 3125 : cluster 3 Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:54.997236+0000 mon.a (mon.0) 3126 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-72185-2"}]': finished 2024-04-02T04:57:55.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: audit 2024-04-02T04:57:54.997381+0000 mon.a (mon.0) 3127 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:57:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:55 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.002091+0000 mon.a (mon.0) 3128 : cluster 0 osdmap e594: 8 total, 8 up, 6 in 2024-04-02T04:57:55.378 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:55.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:55.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:55.638 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:55.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:55.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:55.898 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:55.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:56.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:56.028 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:56.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:56.156 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:56.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:56.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[28904]: cluster 2024-04-02T04:57:54.366269+0000 osd.7 (osd.7) 111 : cluster 0 2.1b scrub starts 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[28904]: cluster 2024-04-02T04:57:54.389730+0000 osd.7 (osd.7) 112 : cluster 0 2.1b scrub ok 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.041020+0000 osd.4 (osd.4) 324 : cluster 0 43.6 scrub starts 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.042626+0000 osd.4 (osd.4) 325 : cluster 0 43.6 scrub ok 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[21528]: cluster 2024-04-02T04:57:54.366269+0000 osd.7 (osd.7) 111 : cluster 0 2.1b scrub starts 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[21528]: cluster 2024-04-02T04:57:54.389730+0000 osd.7 (osd.7) 112 : cluster 0 2.1b scrub ok 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.041020+0000 osd.4 (osd.4) 324 : cluster 0 43.6 scrub starts 2024-04-02T04:57:56.271 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:56 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.042626+0000 osd.4 (osd.4) 325 : cluster 0 43.6 scrub ok 2024-04-02T04:57:56.286 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:56.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:56.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:56 smithi098 bash[23790]: cluster 2024-04-02T04:57:54.366269+0000 osd.7 (osd.7) 111 : cluster 0 2.1b scrub starts 2024-04-02T04:57:56.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:56 smithi098 bash[23790]: cluster 2024-04-02T04:57:54.389730+0000 osd.7 (osd.7) 112 : cluster 0 2.1b scrub ok 2024-04-02T04:57:56.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:56 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.041020+0000 osd.4 (osd.4) 324 : cluster 0 43.6 scrub starts 2024-04-02T04:57:56.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:56 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.042626+0000 osd.4 (osd.4) 325 : cluster 0 43.6 scrub ok 2024-04-02T04:57:56.415 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:56.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:56.544 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:56.647 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:56.672 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:56.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:56.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:56.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:57.032 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:57.045 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-02T04:57:57.045 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 1000 0.05 2024-04-02T04:57:57.075 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:57.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-02T04:57:57.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:57.209 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 -- 172.21.15.69:0/2960409505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8101dd0 msgr2=0x7f0ff8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 --2- 172.21.15.69:0/2960409505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8101dd0 0x7f0ff8105e20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80099c0 tx=0x7f0fe8009cd0 comp rx=0 tx=0).stop 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 -- 172.21.15.69:0/2960409505 shutdown_connections 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 --2- 172.21.15.69:0/2960409505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff8106500 0x7f0ff810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 --2- 172.21.15.69:0/2960409505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff8101420 0x7f0ff8101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 --2- 172.21.15.69:0/2960409505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8101dd0 0x7f0ff8105e20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 -- 172.21.15.69:0/2960409505 >> 172.21.15.69:0/2960409505 conn(0x7f0ff8078240 msgr2=0x7f0ff8078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 -- 172.21.15.69:0/2960409505 shutdown_connections 2024-04-02T04:57:57.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.238+0000 7f0ffeb48700 1 -- 172.21.15.69:0/2960409505 wait complete. 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 librados: starting msgr at 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 librados: starting objecter 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 Processor -- start 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 -- start start 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 librados: setting wanted keys 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 librados: calling monclient init 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff8101420 0x7f0ff8196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff8101dd0 0x7f0ff8197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 0x7f0ff8197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff8110a20 con 0x7f0ff8101dd0 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81108a0 con 0x7f0ff8101420 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff8110ba0 con 0x7f0ff8106500 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 0x7f0ff8197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:57.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 0x7f0ff8197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55830/0 (socket says 172.21.15.69:55830) 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 -- 172.21.15.69:0/3265644026 learned_addr learned my addr 172.21.15.69:0/3265644026 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ff7fff700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff8101dd0 0x7f0ff8197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffc8e4700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff8101420 0x7f0ff8196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 -- 172.21.15.69:0/3265644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff8101420 msgr2=0x7f0ff8196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff8101420 0x7f0ff8196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 -- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff8101dd0 msgr2=0x7f0ff8197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff8101dd0 0x7f0ff8197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 -- 172.21.15.69:0/3265644026 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff81a6550 con 0x7f0ff8106500 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ff5ffb700 10 client.?.objecter ms_handle_connect 0x7f0ff8106500 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffd0e5700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 0x7f0ff8197760 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0007d70 tx=0x7f0ff000a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ff5ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0016020 con 0x7f0ff8106500 2024-04-02T04:57:57.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 -- 172.21.15.69:0/3265644026 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff81a6840 con 0x7f0ff8106500 2024-04-02T04:57:57.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 10 client.26759.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:57:57.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 -- 172.21.15.69:0/3265644026 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff81a6d50 con 0x7f0ff8106500 2024-04-02T04:57:57.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ffeb48700 1 librados: init done 2024-04-02T04:57:57.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0ff000ac90 con 0x7f0ff8106500 2024-04-02T04:57:57.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0011380 con 0x7f0ff8106500 2024-04-02T04:57:57.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.242+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd8005120 con 0x7f0ff8106500 2024-04-02T04:57:57.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0ff0019020 con 0x7f0ff8106500 2024-04-02T04:57:57.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0fe006f5e0 0x7f0fe0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:57:57.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ffc8e4700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0fe006f5e0 0x7f0fe0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ffc8e4700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0fe006f5e0 0x7f0fe0071aa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0fec009c60 tx=0x7f0fec009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(596..596 src has 74..596) ==== 10168+0+0 (secure 0 0 0) 0x7f0ff008ecd0 con 0x7f0ff8106500 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 10 client.26759.objecter ms_dispatch 0x7f0ff8001550 osd_map(596..596 src has 74..596) 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 3 client.26759.objecter handle_osd_map got epochs [596,596] > 0 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 3 client.26759.objecter handle_osd_map decoding full epoch 596 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 20 client.26759.objecter dump_active .. 0 homeless 2024-04-02T04:57:57.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.246+0000 7f0ff5ffb700 10 client.26759.objecter ms_handle_connect 0x7f0fe006f5e0 2024-04-02T04:57:57.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.258+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ff0020020 con 0x7f0ff8106500 2024-04-02T04:57:57.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:57.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.367066+0000 osd.2 (osd.2) 281 : cluster 0 1.8 scrub starts 2024-04-02T04:57:57.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.369064+0000 osd.2 (osd.2) 282 : cluster 0 1.8 scrub ok 2024-04-02T04:57:57.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.971451+0000 osd.0 (osd.0) 311 : cluster 0 3.12 scrub starts 2024-04-02T04:57:57.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:55.973848+0000 osd.0 (osd.0) 312 : cluster 0 3.12 scrub ok 2024-04-02T04:57:57.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:56.004042+0000 osd.1 (osd.1) 319 : cluster 0 2.2 scrub starts 2024-04-02T04:57:57.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:56.006215+0000 osd.1 (osd.1) 320 : cluster 0 2.2 scrub ok 2024-04-02T04:57:57.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:56.037088+0000 mon.a (mon.0) 3129 : cluster 0 osdmap e595: 8 total, 8 up, 6 in 2024-04-02T04:57:57.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: audit 2024-04-02T04:57:56.044749+0000 mon.c (mon.2) 1072 : audit 1 from='client.? 172.21.15.69:0/3612900418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:57.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: audit 2024-04-02T04:57:56.045591+0000 mon.a (mon.0) 3130 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:57.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:56.072514+0000 osd.4 (osd.4) 326 : cluster 0 43.19 scrub starts 2024-04-02T04:57:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:56.073484+0000 osd.4 (osd.4) 327 : cluster 0 43.19 scrub ok 2024-04-02T04:57:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: audit 2024-04-02T04:57:56.085967+0000 mon.c (mon.2) 1073 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: audit 2024-04-02T04:57:56.086359+0000 mon.a (mon.0) 3131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: audit 2024-04-02T04:57:56.086857+0000 mon.c (mon.2) 1074 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-87"}]: dispatch 2024-04-02T04:57:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: audit 2024-04-02T04:57:56.087113+0000 mon.a (mon.0) 3132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-87"}]: dispatch 2024-04-02T04:57:57.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[21528]: cluster 2024-04-02T04:57:56.214781+0000 mgr.y (mgr.14538) 559 : cluster 0 pgmap v890: 334 pgs: 32 unknown, 302 active+clean; 8.4 MiB data, 469 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:57:57.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.367066+0000 osd.2 (osd.2) 281 : cluster 0 1.8 scrub starts 2024-04-02T04:57:57.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.369064+0000 osd.2 (osd.2) 282 : cluster 0 1.8 scrub ok 2024-04-02T04:57:57.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.971451+0000 osd.0 (osd.0) 311 : cluster 0 3.12 scrub starts 2024-04-02T04:57:57.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:55.973848+0000 osd.0 (osd.0) 312 : cluster 0 3.12 scrub ok 2024-04-02T04:57:57.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:56.004042+0000 osd.1 (osd.1) 319 : cluster 0 2.2 scrub starts 2024-04-02T04:57:57.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:56.006215+0000 osd.1 (osd.1) 320 : cluster 0 2.2 scrub ok 2024-04-02T04:57:57.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:56.037088+0000 mon.a (mon.0) 3129 : cluster 0 osdmap e595: 8 total, 8 up, 6 in 2024-04-02T04:57:57.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: audit 2024-04-02T04:57:56.044749+0000 mon.c (mon.2) 1072 : audit 1 from='client.? 172.21.15.69:0/3612900418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:57.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: audit 2024-04-02T04:57:56.045591+0000 mon.a (mon.0) 3130 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:57.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:56.072514+0000 osd.4 (osd.4) 326 : cluster 0 43.19 scrub starts 2024-04-02T04:57:57.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:56.073484+0000 osd.4 (osd.4) 327 : cluster 0 43.19 scrub ok 2024-04-02T04:57:57.327 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: audit 2024-04-02T04:57:56.085967+0000 mon.c (mon.2) 1073 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:57.327 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: audit 2024-04-02T04:57:56.086359+0000 mon.a (mon.0) 3131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:57.327 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: audit 2024-04-02T04:57:56.086857+0000 mon.c (mon.2) 1074 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-87"}]: dispatch 2024-04-02T04:57:57.327 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: audit 2024-04-02T04:57:56.087113+0000 mon.a (mon.0) 3132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-87"}]: dispatch 2024-04-02T04:57:57.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:57 smithi069 bash[28904]: cluster 2024-04-02T04:57:56.214781+0000 mgr.y (mgr.14538) 559 : cluster 0 pgmap v890: 334 pgs: 32 unknown, 302 active+clean; 8.4 MiB data, 469 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:57:57.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:57.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:57.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.367066+0000 osd.2 (osd.2) 281 : cluster 0 1.8 scrub starts 2024-04-02T04:57:57.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.369064+0000 osd.2 (osd.2) 282 : cluster 0 1.8 scrub ok 2024-04-02T04:57:57.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.971451+0000 osd.0 (osd.0) 311 : cluster 0 3.12 scrub starts 2024-04-02T04:57:57.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:55.973848+0000 osd.0 (osd.0) 312 : cluster 0 3.12 scrub ok 2024-04-02T04:57:57.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:56.004042+0000 osd.1 (osd.1) 319 : cluster 0 2.2 scrub starts 2024-04-02T04:57:57.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:56.006215+0000 osd.1 (osd.1) 320 : cluster 0 2.2 scrub ok 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:56.037088+0000 mon.a (mon.0) 3129 : cluster 0 osdmap e595: 8 total, 8 up, 6 in 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: audit 2024-04-02T04:57:56.044749+0000 mon.c (mon.2) 1072 : audit 1 from='client.? 172.21.15.69:0/3612900418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: audit 2024-04-02T04:57:56.045591+0000 mon.a (mon.0) 3130 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:56.072514+0000 osd.4 (osd.4) 326 : cluster 0 43.19 scrub starts 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:56.073484+0000 osd.4 (osd.4) 327 : cluster 0 43.19 scrub ok 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: audit 2024-04-02T04:57:56.085967+0000 mon.c (mon.2) 1073 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: audit 2024-04-02T04:57:56.086359+0000 mon.a (mon.0) 3131 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: audit 2024-04-02T04:57:56.086857+0000 mon.c (mon.2) 1074 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-87"}]: dispatch 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: audit 2024-04-02T04:57:56.087113+0000 mon.a (mon.0) 3132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-87"}]: dispatch 2024-04-02T04:57:57.454 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:57 smithi098 bash[23790]: cluster 2024-04-02T04:57:56.214781+0000 mgr.y (mgr.14538) 559 : cluster 0 pgmap v890: 334 pgs: 32 unknown, 302 active+clean; 8.4 MiB data, 469 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:57:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:57.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:57.494+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f0fd80056e0 con 0x7f0ff8106500 2024-04-02T04:57:57.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:57.592 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:57:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:57:57.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:57.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:57.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:57:57.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:57.979 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:57.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:58.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:57:58.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.066+0000 7f0ff5ffb700 1 -- 172.21.15.69:0/3265644026 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v597) ==== 87+0+0 (secure 0 0 0) 0x7f0ff005b920 con 0x7f0ff8106500 2024-04-02T04:57:58.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.074+0000 7f0fdf7fe700 10 librados: watch_flush enter 2024-04-02T04:57:58.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2024-04-02T04:57:58.077 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (187 ms) 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC (1303 ms total) 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3065 ms) 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP (3065 ms total) 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment tear-down 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] 11 tests from 3 test suites ran. (528084 ms total) 2024-04-02T04:57:58.078 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ PASSED ] 11 tests. 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.078+0000 7f0fdf7fe700 10 librados: watch_flush exit 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 20 client.26759.objecter shutdown clearing up homeless session... 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 10 client.26759.objecter successfully canceled tick 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0fe006f5e0 msgr2=0x7f0fe0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0fe006f5e0 0x7f0fe0071aa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0fec009c60 tx=0x7f0fec009260 comp rx=0 tx=0).stop 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 msgr2=0x7f0ff8197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:57:58.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 0x7f0ff8197760 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0007d70 tx=0x7f0ff000a340 comp rx=0 tx=0).stop 2024-04-02T04:57:58.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 shutdown_connections 2024-04-02T04:57:58.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0fe006f5e0 0x7f0fe0071aa0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff8106500 0x7f0ff8197760 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff8101420 0x7f0ff8196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 --2- 172.21.15.69:0/3265644026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff8101dd0 0x7f0ff8197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 >> 172.21.15.69:0/3265644026 conn(0x7f0ff8078240 msgr2=0x7f0ff80fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 shutdown_connections 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 -- 172.21.15.69:0/3265644026 wait complete. 2024-04-02T04:57:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:57:58.082+0000 7f0fdf7fe700 1 librados: shutdown 2024-04-02T04:57:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:57:58.097 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:58 smithi098 bash[23790]: audit 2024-04-02T04:57:57.041239+0000 mon.a (mon.0) 3133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:58.097 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:58 smithi098 bash[23790]: cluster 2024-04-02T04:57:57.058638+0000 mon.a (mon.0) 3134 : cluster 0 osdmap e596: 8 total, 8 up, 6 in 2024-04-02T04:57:58.097 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:58 smithi098 bash[23790]: audit 2024-04-02T04:57:57.114809+0000 mon.c (mon.2) 1075 : audit 1 from='client.? 172.21.15.69:0/3612900418' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-02T04:57:58.097 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:58 smithi098 bash[23790]: audit 2024-04-02T04:57:57.499360+0000 mon.c (mon.2) 1076 : audit 1 from='client.? 172.21.15.69:0/3265644026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:57:58.098 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:58 smithi098 bash[23790]: audit 2024-04-02T04:57:57.522021+0000 mon.a (mon.0) 3135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-02T04:57:58.098 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:58 smithi098 bash[23790]: audit 2024-04-02T04:57:57.522729+0000 mon.a (mon.0) 3136 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:57:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:58.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:58.243 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:58.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:58.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21528]: audit 2024-04-02T04:57:57.041239+0000 mon.a (mon.0) 3133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21528]: cluster 2024-04-02T04:57:57.058638+0000 mon.a (mon.0) 3134 : cluster 0 osdmap e596: 8 total, 8 up, 6 in 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21528]: audit 2024-04-02T04:57:57.114809+0000 mon.c (mon.2) 1075 : audit 1 from='client.? 172.21.15.69:0/3612900418' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21528]: audit 2024-04-02T04:57:57.499360+0000 mon.c (mon.2) 1076 : audit 1 from='client.? 172.21.15.69:0/3265644026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21528]: audit 2024-04-02T04:57:57.522021+0000 mon.a (mon.0) 3135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21528]: audit 2024-04-02T04:57:57.522729+0000 mon.a (mon.0) 3136 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[28904]: audit 2024-04-02T04:57:57.041239+0000 mon.a (mon.0) 3133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72185-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[28904]: cluster 2024-04-02T04:57:57.058638+0000 mon.a (mon.0) 3134 : cluster 0 osdmap e596: 8 total, 8 up, 6 in 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[28904]: audit 2024-04-02T04:57:57.114809+0000 mon.c (mon.2) 1075 : audit 1 from='client.? 172.21.15.69:0/3612900418' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[28904]: audit 2024-04-02T04:57:57.499360+0000 mon.c (mon.2) 1076 : audit 1 from='client.? 172.21.15.69:0/3265644026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[28904]: audit 2024-04-02T04:57:57.522021+0000 mon.a (mon.0) 3135 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-02T04:57:58.355 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[28904]: audit 2024-04-02T04:57:57.522729+0000 mon.a (mon.0) 3136 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:57:58.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:58.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:58.493 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:58.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:58.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:58.753 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:58.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:58.883 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:57:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:58.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:57:58.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:58.997 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:57:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:57:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:57:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:57:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:57:59.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: cluster 2024-04-02T04:57:57.931318+0000 osd.0 (osd.0) 313 : cluster 0 40.1a scrub starts 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: cluster 2024-04-02T04:57:57.933829+0000 osd.0 (osd.0) 314 : cluster 0 40.1a scrub ok 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: cluster 2024-04-02T04:57:58.044150+0000 mon.a (mon.0) 3137 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: audit 2024-04-02T04:57:58.056134+0000 mon.a (mon.0) 3138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: audit 2024-04-02T04:57:58.056366+0000 mon.a (mon.0) 3139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: cluster 2024-04-02T04:57:58.064587+0000 mon.a (mon.0) 3140 : cluster 0 osdmap e597: 8 total, 8 up, 6 in 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: audit 2024-04-02T04:57:58.068380+0000 mon.c (mon.2) 1077 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: audit 2024-04-02T04:57:58.069978+0000 mon.a (mon.0) 3141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:59.130 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:57:59 smithi098 bash[23790]: cluster 2024-04-02T04:57:58.215632+0000 mgr.y (mgr.14538) 560 : cluster 0 pgmap v893: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T04:57:59.141 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:59.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:59.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: cluster 2024-04-02T04:57:57.931318+0000 osd.0 (osd.0) 313 : cluster 0 40.1a scrub starts 2024-04-02T04:57:59.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: cluster 2024-04-02T04:57:57.933829+0000 osd.0 (osd.0) 314 : cluster 0 40.1a scrub ok 2024-04-02T04:57:59.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: cluster 2024-04-02T04:57:58.044150+0000 mon.a (mon.0) 3137 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:59.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: audit 2024-04-02T04:57:58.056134+0000 mon.a (mon.0) 3138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: audit 2024-04-02T04:57:58.056366+0000 mon.a (mon.0) 3139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: cluster 2024-04-02T04:57:58.064587+0000 mon.a (mon.0) 3140 : cluster 0 osdmap e597: 8 total, 8 up, 6 in 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: audit 2024-04-02T04:57:58.068380+0000 mon.c (mon.2) 1077 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: audit 2024-04-02T04:57:58.069978+0000 mon.a (mon.0) 3141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[21528]: cluster 2024-04-02T04:57:58.215632+0000 mgr.y (mgr.14538) 560 : cluster 0 pgmap v893: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: cluster 2024-04-02T04:57:57.931318+0000 osd.0 (osd.0) 313 : cluster 0 40.1a scrub starts 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: cluster 2024-04-02T04:57:57.933829+0000 osd.0 (osd.0) 314 : cluster 0 40.1a scrub ok 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: cluster 2024-04-02T04:57:58.044150+0000 mon.a (mon.0) 3137 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: audit 2024-04-02T04:57:58.056134+0000 mon.a (mon.0) 3138 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-72185-3","pool2":"test-rados-api-smithi069-72185-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: audit 2024-04-02T04:57:58.056366+0000 mon.a (mon.0) 3139 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:57:59.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: cluster 2024-04-02T04:57:58.064587+0000 mon.a (mon.0) 3140 : cluster 0 osdmap e597: 8 total, 8 up, 6 in 2024-04-02T04:57:59.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: audit 2024-04-02T04:57:58.068380+0000 mon.c (mon.2) 1077 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:59.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: audit 2024-04-02T04:57:58.069978+0000 mon.a (mon.0) 3141 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:57:59.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:57:59 smithi069 bash[28904]: cluster 2024-04-02T04:57:58.215632+0000 mgr.y (mgr.14538) 560 : cluster 0 pgmap v893: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T04:57:59.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:57:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:57:59.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:59.402 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:57:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:57:59.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:59.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:59.666 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:57:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:57:59.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:59.795 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:57:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:57:59.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:57:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:57:59.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:00.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:00.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:00.190 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:00.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:00.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: audit 2024-04-02T04:57:59.064913+0000 mon.a (mon.0) 3142 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: cluster 2024-04-02T04:57:59.069128+0000 mon.a (mon.0) 3143 : cluster 0 osdmap e598: 8 total, 8 up, 6 in 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: audit 2024-04-02T04:57:59.071576+0000 mon.c (mon.2) 1078 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: audit 2024-04-02T04:57:59.080604+0000 mon.c (mon.2) 1079 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: audit 2024-04-02T04:57:59.081124+0000 mon.a (mon.0) 3144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: cluster 2024-04-02T04:57:59.977148+0000 mon.a (mon.0) 3145 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: audit 2024-04-02T04:57:59.980900+0000 mon.a (mon.0) 3146 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:58:00.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:00 smithi098 bash[23790]: cluster 2024-04-02T04:57:59.987980+0000 mon.a (mon.0) 3147 : cluster 0 osdmap e599: 8 total, 8 up, 6 in 2024-04-02T04:58:00.319 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:00.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: audit 2024-04-02T04:57:59.064913+0000 mon.a (mon.0) 3142 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: cluster 2024-04-02T04:57:59.069128+0000 mon.a (mon.0) 3143 : cluster 0 osdmap e598: 8 total, 8 up, 6 in 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: audit 2024-04-02T04:57:59.071576+0000 mon.c (mon.2) 1078 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: audit 2024-04-02T04:57:59.080604+0000 mon.c (mon.2) 1079 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: audit 2024-04-02T04:57:59.081124+0000 mon.a (mon.0) 3144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: cluster 2024-04-02T04:57:59.977148+0000 mon.a (mon.0) 3145 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: audit 2024-04-02T04:57:59.980900+0000 mon.a (mon.0) 3146 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[21528]: cluster 2024-04-02T04:57:59.987980+0000 mon.a (mon.0) 3147 : cluster 0 osdmap e599: 8 total, 8 up, 6 in 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: audit 2024-04-02T04:57:59.064913+0000 mon.a (mon.0) 3142 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: cluster 2024-04-02T04:57:59.069128+0000 mon.a (mon.0) 3143 : cluster 0 osdmap e598: 8 total, 8 up, 6 in 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: audit 2024-04-02T04:57:59.071576+0000 mon.c (mon.2) 1078 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: audit 2024-04-02T04:57:59.080604+0000 mon.c (mon.2) 1079 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: audit 2024-04-02T04:57:59.081124+0000 mon.a (mon.0) 3144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: cluster 2024-04-02T04:57:59.977148+0000 mon.a (mon.0) 3145 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: audit 2024-04-02T04:57:59.980900+0000 mon.a (mon.0) 3146 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:00 smithi069 bash[28904]: cluster 2024-04-02T04:57:59.987980+0000 mon.a (mon.0) 3147 : cluster 0 osdmap e599: 8 total, 8 up, 6 in 2024-04-02T04:58:00.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:00.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:00.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:00.710 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:00.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:00.839 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:00.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:00.969 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:00.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:01.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:01.087 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:01 smithi098 bash[23790]: cluster 2024-04-02T04:57:59.505883+0000 osd.2 (osd.2) 283 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:01.087 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:01 smithi098 bash[23790]: cluster 2024-04-02T04:57:59.506139+0000 osd.2 (osd.2) 284 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:01.087 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:01 smithi098 bash[23790]: cluster 2024-04-02T04:58:00.216554+0000 mgr.y (mgr.14538) 561 : cluster 0 pgmap v896: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T04:58:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:01.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:01.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:01.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:58:01.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:01.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:01.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:01.472 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:01 smithi069 bash[21528]: cluster 2024-04-02T04:57:59.505883+0000 osd.2 (osd.2) 283 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:01.472 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:01 smithi069 bash[21528]: cluster 2024-04-02T04:57:59.506139+0000 osd.2 (osd.2) 284 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:01.472 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:01 smithi069 bash[21528]: cluster 2024-04-02T04:58:00.216554+0000 mgr.y (mgr.14538) 561 : cluster 0 pgmap v896: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T04:58:01.472 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:01 smithi069 bash[28904]: cluster 2024-04-02T04:57:59.505883+0000 osd.2 (osd.2) 283 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:01.472 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:01 smithi069 bash[28904]: cluster 2024-04-02T04:57:59.506139+0000 osd.2 (osd.2) 284 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:01.472 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:01 smithi069 bash[28904]: cluster 2024-04-02T04:58:00.216554+0000 mgr.y (mgr.14538) 561 : cluster 0 pgmap v896: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T04:58:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:01.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:01.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:01.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:01.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:01.871 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:01.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:01.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:02.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:02.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:02 smithi098 bash[23790]: cluster 2024-04-02T04:58:01.081711+0000 mon.a (mon.0) 3148 : cluster 0 osdmap e600: 8 total, 8 up, 6 in 2024-04-02T04:58:02.138 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:02.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:02.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:02 smithi069 bash[21528]: cluster 2024-04-02T04:58:01.081711+0000 mon.a (mon.0) 3148 : cluster 0 osdmap e600: 8 total, 8 up, 6 in 2024-04-02T04:58:02.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:02 smithi069 bash[28904]: cluster 2024-04-02T04:58:01.081711+0000 mon.a (mon.0) 3148 : cluster 0 osdmap e600: 8 total, 8 up, 6 in 2024-04-02T04:58:02.267 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:02.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:02.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:02.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:02.529 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:02.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:02.658 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:02.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:02.790 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:02.840 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:02.842 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:02.842 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:02.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:02.920 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:02.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:03.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:03.049 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:03.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:03.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[21528]: cluster 2024-04-02T04:58:02.099108+0000 mon.a (mon.0) 3149 : cluster 0 osdmap e601: 8 total, 8 up, 6 in 2024-04-02T04:58:03.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[21528]: audit 2024-04-02T04:58:02.173291+0000 mon.c (mon.2) 1080 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:03.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[21528]: audit 2024-04-02T04:58:02.173970+0000 mon.a (mon.0) 3150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:03.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[21528]: audit 2024-04-02T04:58:02.174903+0000 mon.c (mon.2) 1081 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-90"}]: dispatch 2024-04-02T04:58:03.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[21528]: audit 2024-04-02T04:58:02.175386+0000 mon.a (mon.0) 3151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-90"}]: dispatch 2024-04-02T04:58:03.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[21528]: cluster 2024-04-02T04:58:02.217132+0000 mgr.y (mgr.14538) 562 : cluster 0 pgmap v899: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:03.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[28904]: cluster 2024-04-02T04:58:02.099108+0000 mon.a (mon.0) 3149 : cluster 0 osdmap e601: 8 total, 8 up, 6 in 2024-04-02T04:58:03.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[28904]: audit 2024-04-02T04:58:02.173291+0000 mon.c (mon.2) 1080 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:03.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[28904]: audit 2024-04-02T04:58:02.173970+0000 mon.a (mon.0) 3150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:03.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[28904]: audit 2024-04-02T04:58:02.174903+0000 mon.c (mon.2) 1081 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-90"}]: dispatch 2024-04-02T04:58:03.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[28904]: audit 2024-04-02T04:58:02.175386+0000 mon.a (mon.0) 3151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-90"}]: dispatch 2024-04-02T04:58:03.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:03 smithi069 bash[28904]: cluster 2024-04-02T04:58:02.217132+0000 mgr.y (mgr.14538) 562 : cluster 0 pgmap v899: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:03.174 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[23790]: cluster 2024-04-02T04:58:02.099108+0000 mon.a (mon.0) 3149 : cluster 0 osdmap e601: 8 total, 8 up, 6 in 2024-04-02T04:58:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[23790]: audit 2024-04-02T04:58:02.173291+0000 mon.c (mon.2) 1080 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[23790]: audit 2024-04-02T04:58:02.173970+0000 mon.a (mon.0) 3150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[23790]: audit 2024-04-02T04:58:02.174903+0000 mon.c (mon.2) 1081 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-90"}]: dispatch 2024-04-02T04:58:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[23790]: audit 2024-04-02T04:58:02.175386+0000 mon.a (mon.0) 3151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-90"}]: dispatch 2024-04-02T04:58:03.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[23790]: cluster 2024-04-02T04:58:02.217132+0000 mgr.y (mgr.14538) 562 : cluster 0 pgmap v899: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 473 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:03.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:03.305 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:03.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:03.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:03.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:03.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:58:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:58:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:03.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:03.693 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:03.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:03.821 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:03.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:03.949 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:03.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:04.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:04.078 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:04.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[28904]: cluster 2024-04-02T04:58:03.101967+0000 mon.a (mon.0) 3152 : cluster 0 osdmap e602: 8 total, 8 up, 6 in 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[28904]: audit 2024-04-02T04:58:03.457736+0000 mgr.y (mgr.14538) 563 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[28904]: audit 2024-04-02T04:58:03.559087+0000 mon.b (mon.1) 278 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[28904]: audit 2024-04-02T04:58:03.561305+0000 mon.a (mon.0) 3153 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[21528]: cluster 2024-04-02T04:58:03.101967+0000 mon.a (mon.0) 3152 : cluster 0 osdmap e602: 8 total, 8 up, 6 in 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[21528]: audit 2024-04-02T04:58:03.457736+0000 mgr.y (mgr.14538) 563 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[21528]: audit 2024-04-02T04:58:03.559087+0000 mon.b (mon.1) 278 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:04.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:04 smithi069 bash[21528]: audit 2024-04-02T04:58:03.561305+0000 mon.a (mon.0) 3153 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:58:04.203 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:04.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:04.334 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:04.437 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:04.465 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:04 smithi098 bash[23790]: cluster 2024-04-02T04:58:03.101967+0000 mon.a (mon.0) 3152 : cluster 0 osdmap e602: 8 total, 8 up, 6 in 2024-04-02T04:58:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:04 smithi098 bash[23790]: audit 2024-04-02T04:58:03.457736+0000 mgr.y (mgr.14538) 563 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:04.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:04 smithi098 bash[23790]: audit 2024-04-02T04:58:03.559087+0000 mon.b (mon.1) 278 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:04.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:04 smithi098 bash[23790]: audit 2024-04-02T04:58:03.561305+0000 mon.a (mon.0) 3153 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:58:04.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:04.595 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:04.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:04.726 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:04.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:04.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:04.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:04.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:05.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:05.219 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:05.242 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:05.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:05.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:05 smithi098 bash[23790]: cluster 2024-04-02T04:58:04.119014+0000 mon.a (mon.0) 3154 : cluster 0 osdmap e603: 8 total, 8 up, 6 in 2024-04-02T04:58:05.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:05 smithi098 bash[23790]: audit 2024-04-02T04:58:04.121654+0000 mon.c (mon.2) 1082 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:05.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:05 smithi098 bash[23790]: audit 2024-04-02T04:58:04.123387+0000 mon.a (mon.0) 3155 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:05.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:05 smithi098 bash[23790]: cluster 2024-04-02T04:58:04.217701+0000 mgr.y (mgr.14538) 564 : cluster 0 pgmap v902: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-02T04:58:05.357 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:05 smithi098 bash[23790]: cluster 2024-04-02T04:58:04.980064+0000 mon.a (mon.0) 3156 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:05.373 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:05.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:05.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[21528]: cluster 2024-04-02T04:58:04.119014+0000 mon.a (mon.0) 3154 : cluster 0 osdmap e603: 8 total, 8 up, 6 in 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[21528]: audit 2024-04-02T04:58:04.121654+0000 mon.c (mon.2) 1082 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[21528]: audit 2024-04-02T04:58:04.123387+0000 mon.a (mon.0) 3155 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[21528]: cluster 2024-04-02T04:58:04.217701+0000 mgr.y (mgr.14538) 564 : cluster 0 pgmap v902: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[21528]: cluster 2024-04-02T04:58:04.980064+0000 mon.a (mon.0) 3156 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[28904]: cluster 2024-04-02T04:58:04.119014+0000 mon.a (mon.0) 3154 : cluster 0 osdmap e603: 8 total, 8 up, 6 in 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[28904]: audit 2024-04-02T04:58:04.121654+0000 mon.c (mon.2) 1082 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[28904]: audit 2024-04-02T04:58:04.123387+0000 mon.a (mon.0) 3155 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[28904]: cluster 2024-04-02T04:58:04.217701+0000 mgr.y (mgr.14538) 564 : cluster 0 pgmap v902: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-02T04:58:05.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:05 smithi069 bash[28904]: cluster 2024-04-02T04:58:04.980064+0000 mon.a (mon.0) 3156 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:05.498 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:05.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:05.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:05.762 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:05.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:05.890 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:05.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:05.899 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:05.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:06.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:06.155 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:06.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:06.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:06 smithi069 bash[21528]: audit 2024-04-02T04:58:05.124659+0000 mon.a (mon.0) 3157 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:06.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:06 smithi069 bash[21528]: cluster 2024-04-02T04:58:05.136695+0000 mon.a (mon.0) 3158 : cluster 0 osdmap e604: 8 total, 8 up, 6 in 2024-04-02T04:58:06.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:06 smithi069 bash[21528]: audit 2024-04-02T04:58:05.147878+0000 mon.c (mon.2) 1083 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:06.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:06 smithi069 bash[28904]: audit 2024-04-02T04:58:05.124659+0000 mon.a (mon.0) 3157 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:06.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:06 smithi069 bash[28904]: cluster 2024-04-02T04:58:05.136695+0000 mon.a (mon.0) 3158 : cluster 0 osdmap e604: 8 total, 8 up, 6 in 2024-04-02T04:58:06.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:06 smithi069 bash[28904]: audit 2024-04-02T04:58:05.147878+0000 mon.c (mon.2) 1083 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:06.284 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:06.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:06.414 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:06 smithi098 bash[23790]: audit 2024-04-02T04:58:05.124659+0000 mon.a (mon.0) 3157 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:06 smithi098 bash[23790]: cluster 2024-04-02T04:58:05.136695+0000 mon.a (mon.0) 3158 : cluster 0 osdmap e604: 8 total, 8 up, 6 in 2024-04-02T04:58:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:06 smithi098 bash[23790]: audit 2024-04-02T04:58:05.147878+0000 mon.c (mon.2) 1083 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:06.517 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:06.648 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:06.675 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:06.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:06.806 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:06.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:06.937 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:06.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:07.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:07.067 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:07.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:07.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:07 smithi069 bash[21528]: cluster 2024-04-02T04:58:06.135524+0000 mon.a (mon.0) 3159 : cluster 0 osdmap e605: 8 total, 8 up, 6 in 2024-04-02T04:58:07.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:07 smithi069 bash[21528]: cluster 2024-04-02T04:58:06.218414+0000 mgr.y (mgr.14538) 565 : cluster 0 pgmap v905: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-02T04:58:07.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:07 smithi069 bash[28904]: cluster 2024-04-02T04:58:06.135524+0000 mon.a (mon.0) 3159 : cluster 0 osdmap e605: 8 total, 8 up, 6 in 2024-04-02T04:58:07.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:07 smithi069 bash[28904]: cluster 2024-04-02T04:58:06.218414+0000 mgr.y (mgr.14538) 565 : cluster 0 pgmap v905: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-02T04:58:07.198 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:07.302 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:07.329 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:07.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:07.443 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:07 smithi098 bash[23790]: cluster 2024-04-02T04:58:06.135524+0000 mon.a (mon.0) 3159 : cluster 0 osdmap e605: 8 total, 8 up, 6 in 2024-04-02T04:58:07.444 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:07 smithi098 bash[23790]: cluster 2024-04-02T04:58:06.218414+0000 mgr.y (mgr.14538) 565 : cluster 0 pgmap v905: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-02T04:58:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:07.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:07.588 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:07.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:07.717 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:07.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:07.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:07.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:08.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:08.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:08.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:08 smithi098 bash[23790]: cluster 2024-04-02T04:58:07.160802+0000 mon.a (mon.0) 3160 : cluster 0 osdmap e606: 8 total, 8 up, 6 in 2024-04-02T04:58:08.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:08 smithi098 bash[23790]: audit 2024-04-02T04:58:07.248344+0000 mon.c (mon.2) 1084 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:08.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:08 smithi098 bash[23790]: audit 2024-04-02T04:58:07.248921+0000 mon.a (mon.0) 3161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:08.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:08 smithi098 bash[23790]: audit 2024-04-02T04:58:07.249655+0000 mon.c (mon.2) 1085 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-92"}]: dispatch 2024-04-02T04:58:08.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:08 smithi098 bash[23790]: audit 2024-04-02T04:58:07.250120+0000 mon.a (mon.0) 3162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-92"}]: dispatch 2024-04-02T04:58:08.234 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:08.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:08.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[21528]: cluster 2024-04-02T04:58:07.160802+0000 mon.a (mon.0) 3160 : cluster 0 osdmap e606: 8 total, 8 up, 6 in 2024-04-02T04:58:08.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[21528]: audit 2024-04-02T04:58:07.248344+0000 mon.c (mon.2) 1084 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[21528]: audit 2024-04-02T04:58:07.248921+0000 mon.a (mon.0) 3161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[21528]: audit 2024-04-02T04:58:07.249655+0000 mon.c (mon.2) 1085 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-92"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[21528]: audit 2024-04-02T04:58:07.250120+0000 mon.a (mon.0) 3162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-92"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[28904]: cluster 2024-04-02T04:58:07.160802+0000 mon.a (mon.0) 3160 : cluster 0 osdmap e606: 8 total, 8 up, 6 in 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[28904]: audit 2024-04-02T04:58:07.248344+0000 mon.c (mon.2) 1084 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[28904]: audit 2024-04-02T04:58:07.248921+0000 mon.a (mon.0) 3161 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[28904]: audit 2024-04-02T04:58:07.249655+0000 mon.c (mon.2) 1085 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-92"}]: dispatch 2024-04-02T04:58:08.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[28904]: audit 2024-04-02T04:58:07.250120+0000 mon.a (mon.0) 3162 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-92"}]: dispatch 2024-04-02T04:58:08.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:08.625 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:08.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:08.754 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:08.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:08.880 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:08.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:08.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:09.029 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:58:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:58:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:58:09.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:09.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:09.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:09 smithi098 bash[23790]: cluster 2024-04-02T04:58:08.165965+0000 mon.a (mon.0) 3163 : cluster 0 osdmap e607: 8 total, 8 up, 6 in 2024-04-02T04:58:09.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:09 smithi098 bash[23790]: cluster 2024-04-02T04:58:08.219250+0000 mgr.y (mgr.14538) 566 : cluster 0 pgmap v908: 270 pgs: 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-02T04:58:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:09.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:09.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:09 smithi069 bash[28904]: cluster 2024-04-02T04:58:08.165965+0000 mon.a (mon.0) 3163 : cluster 0 osdmap e607: 8 total, 8 up, 6 in 2024-04-02T04:58:09.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:09 smithi069 bash[28904]: cluster 2024-04-02T04:58:08.219250+0000 mgr.y (mgr.14538) 566 : cluster 0 pgmap v908: 270 pgs: 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-02T04:58:09.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:09 smithi069 bash[21528]: cluster 2024-04-02T04:58:08.165965+0000 mon.a (mon.0) 3163 : cluster 0 osdmap e607: 8 total, 8 up, 6 in 2024-04-02T04:58:09.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:09 smithi069 bash[21528]: cluster 2024-04-02T04:58:08.219250+0000 mgr.y (mgr.14538) 566 : cluster 0 pgmap v908: 270 pgs: 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-02T04:58:09.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:09.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:09.530 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:09.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:09.660 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:09.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:09.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:09.921 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:09.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:10.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:10.049 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:10.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:10.176 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:10.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:10.307 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:10.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:10.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[21528]: cluster 2024-04-02T04:58:09.189269+0000 mon.a (mon.0) 3164 : cluster 0 osdmap e608: 8 total, 8 up, 6 in 2024-04-02T04:58:10.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[21528]: audit 2024-04-02T04:58:09.203619+0000 mon.c (mon.2) 1086 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:10.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[21528]: audit 2024-04-02T04:58:09.204125+0000 mon.a (mon.0) 3165 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:10.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[21528]: audit 2024-04-02T04:58:09.986764+0000 mon.a (mon.0) 3166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:10.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[21528]: cluster 2024-04-02T04:58:09.996153+0000 mon.a (mon.0) 3167 : cluster 0 osdmap e609: 8 total, 8 up, 6 in 2024-04-02T04:58:10.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[21528]: audit 2024-04-02T04:58:09.998981+0000 mon.c (mon.2) 1087 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:10.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[28904]: cluster 2024-04-02T04:58:09.189269+0000 mon.a (mon.0) 3164 : cluster 0 osdmap e608: 8 total, 8 up, 6 in 2024-04-02T04:58:10.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[28904]: audit 2024-04-02T04:58:09.203619+0000 mon.c (mon.2) 1086 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:10.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[28904]: audit 2024-04-02T04:58:09.204125+0000 mon.a (mon.0) 3165 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:10.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[28904]: audit 2024-04-02T04:58:09.986764+0000 mon.a (mon.0) 3166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:10.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[28904]: cluster 2024-04-02T04:58:09.996153+0000 mon.a (mon.0) 3167 : cluster 0 osdmap e609: 8 total, 8 up, 6 in 2024-04-02T04:58:10.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:10 smithi069 bash[28904]: audit 2024-04-02T04:58:09.998981+0000 mon.c (mon.2) 1087 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:10.437 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:10 smithi098 bash[23790]: cluster 2024-04-02T04:58:09.189269+0000 mon.a (mon.0) 3164 : cluster 0 osdmap e608: 8 total, 8 up, 6 in 2024-04-02T04:58:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:10 smithi098 bash[23790]: audit 2024-04-02T04:58:09.203619+0000 mon.c (mon.2) 1086 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:10 smithi098 bash[23790]: audit 2024-04-02T04:58:09.204125+0000 mon.a (mon.0) 3165 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:10 smithi098 bash[23790]: audit 2024-04-02T04:58:09.986764+0000 mon.a (mon.0) 3166 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:10 smithi098 bash[23790]: cluster 2024-04-02T04:58:09.996153+0000 mon.a (mon.0) 3167 : cluster 0 osdmap e609: 8 total, 8 up, 6 in 2024-04-02T04:58:10.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:10 smithi098 bash[23790]: audit 2024-04-02T04:58:09.998981+0000 mon.c (mon.2) 1087 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:10.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:10.567 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:10.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:10.697 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:10.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:10.828 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:10.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:10.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:11.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:11.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:11.222 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:11.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:11.352 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:11.416 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:11 smithi069 bash[21528]: cluster 2024-04-02T04:58:10.220061+0000 mgr.y (mgr.14538) 567 : cluster 0 pgmap v911: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-02T04:58:11.417 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:11 smithi069 bash[21528]: cluster 2024-04-02T04:58:10.998217+0000 mon.a (mon.0) 3168 : cluster 0 osdmap e610: 8 total, 8 up, 6 in 2024-04-02T04:58:11.417 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:11 smithi069 bash[28904]: cluster 2024-04-02T04:58:10.220061+0000 mgr.y (mgr.14538) 567 : cluster 0 pgmap v911: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-02T04:58:11.417 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:11 smithi069 bash[28904]: cluster 2024-04-02T04:58:10.998217+0000 mon.a (mon.0) 3168 : cluster 0 osdmap e610: 8 total, 8 up, 6 in 2024-04-02T04:58:11.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:11.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:11 smithi098 bash[23790]: cluster 2024-04-02T04:58:10.220061+0000 mgr.y (mgr.14538) 567 : cluster 0 pgmap v911: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-02T04:58:11.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:11 smithi098 bash[23790]: cluster 2024-04-02T04:58:10.998217+0000 mon.a (mon.0) 3168 : cluster 0 osdmap e610: 8 total, 8 up, 6 in 2024-04-02T04:58:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:11.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:11.610 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:11.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:11.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:11.873 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:11.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:11.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:12.004 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:12.094 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:58:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:58:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T04:58:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T04:58:12.095 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:58:12.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:12.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:12.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:12.254 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:12.357 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:12.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:12.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:12.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:12.745 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:12.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:12.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:12.906 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:12.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:13.009 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:13.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:13 smithi069 bash[21528]: cluster 2024-04-02T04:58:11.995162+0000 mon.a (mon.0) 3169 : cluster 0 osdmap e611: 8 total, 8 up, 6 in 2024-04-02T04:58:13.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:13 smithi069 bash[21528]: cluster 2024-04-02T04:58:12.220826+0000 mgr.y (mgr.14538) 568 : cluster 0 pgmap v914: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:13.020 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:13 smithi069 bash[28904]: cluster 2024-04-02T04:58:11.995162+0000 mon.a (mon.0) 3169 : cluster 0 osdmap e611: 8 total, 8 up, 6 in 2024-04-02T04:58:13.020 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:13 smithi069 bash[28904]: cluster 2024-04-02T04:58:12.220826+0000 mgr.y (mgr.14538) 568 : cluster 0 pgmap v914: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:13.034 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:13.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:13.165 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:13.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:13.280 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:12 smithi098 bash[23790]: cluster 2024-04-02T04:58:11.995162+0000 mon.a (mon.0) 3169 : cluster 0 osdmap e611: 8 total, 8 up, 6 in 2024-04-02T04:58:13.280 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:12 smithi098 bash[23790]: cluster 2024-04-02T04:58:12.220826+0000 mgr.y (mgr.14538) 568 : cluster 0 pgmap v914: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 480 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:13.398 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:13.426 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:13.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:13.556 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:13.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:13.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:13.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:58:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:58:13.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:13.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 -- 172.21.15.69:0/2180463667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 msgr2=0x7f6978108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 --2- 172.21.15.69:0/2180463667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 0x7f6978108cf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6974009990 tx=0x7f6974009ca0 comp rx=0 tx=0).stop 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 -- 172.21.15.69:0/2180463667 shutdown_connections 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 --2- 172.21.15.69:0/2180463667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6978109230 0x7f6978115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 --2- 172.21.15.69:0/2180463667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f697810e870 0x7f697810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 --2- 172.21.15.69:0/2180463667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 0x7f6978108cf0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 -- 172.21.15.69:0/2180463667 >> 172.21.15.69:0/2180463667 conn(0x7f6978075a10 msgr2=0x7f6978077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 -- 172.21.15.69:0/2180463667 shutdown_connections 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 -- 172.21.15.69:0/2180463667 wait complete. 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 librados: starting msgr at 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 librados: starting objecter 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 Processor -- start 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 -- start start 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 librados: setting wanted keys 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.857+0000 7f697fe25700 1 librados: calling monclient init 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 0x7f697819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978109230 0x7f697819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 0x7f69781a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f69781180e0 con 0x7f6978108870 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6978117f60 con 0x7f6978109230 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6978118260 con 0x7f697810e870 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 0x7f69781a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 0x7f69781a2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:56150/0 (socket says 172.21.15.69:56150) 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 learned_addr learned my addr 172.21.15.69:0/2275525496 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697dbc1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 0x7f697819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978109230 0x7f697819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978109230 msgr2=0x7f697819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978109230 0x7f697819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 msgr2=0x7f697819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6978108870 0x7f697819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69781a3650 con 0x7f697810e870 2024-04-02T04:58:13.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6978109230 0x7f697819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 0x7f69781a2f10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f697000ef30 tx=0x7f697000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 10 client.?.objecter ms_handle_connect 0x7f697810e870 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69700092e0 con 0x7f697810e870 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69781a38e0 con 0x7f697810e870 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6970004bb0 con 0x7f697810e870 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970004020 con 0x7f697810e870 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 10 client.26765.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69781ab1c0 con 0x7f697810e870 2024-04-02T04:58:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 librados: init done 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f697804ff00 con 0x7f697810e870 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f697001b030 con 0x7f697810e870 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f696406f5e0 0x7f6964071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(611..611 src has 74..611) ==== 10169+0+0 (secure 0 0 0) 0x7f697008e690 con 0x7f697810e870 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 osd_map(611..611 src has 74..611) 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 3 client.26765.objecter handle_osd_map got epochs [611,611] > 0 2024-04-02T04:58:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 3 client.26765.objecter handle_osd_map decoding full epoch 611 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f696406f5e0 0x7f6964071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 20 client.26765.objecter dump_active .. 0 homeless 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.861+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v611) ==== 54+0+16 (secure 0 0 0) 0x7f6970022020 con 0x7f697810e870 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 10 client.26765.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 1 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69780630a0 0x7f697810f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f696406f5e0 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 1 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f696406f5e0 0x7f6964071aa0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6968005950 tx=0x7f696800a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 0 1 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 4 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 10 client.26765.objecter _send_command 1 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6978110db0 con 0x7f69780630a0 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69780630a0 0x7f697810f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f69780630a0 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69780630a0 0x7f697810f6f0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6978110db0 con 0x7f69780630a0 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 1: 0 ) 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 10 client.26765.objecter _finish_command 1 = system:0 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 4 2024-04-02T04:58:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.865+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 0 1 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 10 client.26765.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 2 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 2 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f697819f9a0 osd=0 3 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 0 2 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 4 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 10 client.26765.objecter _send_command 2 2024-04-02T04:58:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.881+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f697810fc30 con 0x7f69780630a0 2024-04-02T04:58:13.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f697810fc30 con 0x7f69780630a0 2024-04-02T04:58:13.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:13.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 10 client.26765.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:13.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 4 2024-04-02T04:58:13.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 0 2 2024-04-02T04:58:13.886 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:58:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 10 client.26765.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 3 2024-04-02T04:58:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6978062c90 0x7f69781af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 3 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 1 3 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 4 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 10 client.26765.objecter _send_command 3 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6978110b20 con 0x7f6978062c90 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6978062c90 0x7f69781af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f6978062c90 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6978062c90 0x7f69781af3b0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6978110b20 con 0x7f6978062c90 2024-04-02T04:58:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 3: 0 ) 2024-04-02T04:58:13.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 10 client.26765.objecter _finish_command 3 = system:0 2024-04-02T04:58:13.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 4 2024-04-02T04:58:13.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.885+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 1 3 2024-04-02T04:58:13.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 10 client.26765.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:13.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 4 2024-04-02T04:58:13.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 4 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f6978062b20 osd=1 3 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 1 4 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 4 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 10 client.26765.objecter _send_command 4 2024-04-02T04:58:13.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.901+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6978110090 con 0x7f6978062c90 2024-04-02T04:58:13.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6978110090 con 0x7f6978062c90 2024-04-02T04:58:13.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:13.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 10 client.26765.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:13.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 4 2024-04-02T04:58:13.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 1 4 2024-04-02T04:58:13.907 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:58:13.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 10 client.26765.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 5 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f697802d690 0x7f69781b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 5 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 2 5 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 4 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 10 client.26765.objecter _send_command 5 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f69781b58e0 con 0x7f697802d690 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f697802d690 0x7f69781b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f697802d690 0x7f69781b2a80 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f697802d690 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69781b58e0 con 0x7f697802d690 2024-04-02T04:58:13.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 5: 0 ) 2024-04-02T04:58:13.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 10 client.26765.objecter _finish_command 5 = system:0 2024-04-02T04:58:13.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 4 2024-04-02T04:58:13.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.905+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 2 5 2024-04-02T04:58:13.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:13.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 10 client.26765.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:13.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 6 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 6 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f6978110490 osd=2 3 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 2 6 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 4 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 10 client.26765.objecter _send_command 6 2024-04-02T04:58:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f697802cc00 con 0x7f697802d690 2024-04-02T04:58:13.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f697802cc00 con 0x7f697802d690 2024-04-02T04:58:13.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:13.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 10 client.26765.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:13.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 4 2024-04-02T04:58:13.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 2 6 2024-04-02T04:58:13.927 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:58:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 10 client.26765.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 7 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69781b4140 0x7f69781b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 7 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 3 7 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 4 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 10 client.26765.objecter _send_command 7 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f69781b0820 con 0x7f69781b4140 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69781b4140 0x7f69781b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f69781b4140 2024-04-02T04:58:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69781b4140 0x7f69781b4560 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69781b0820 con 0x7f69781b4140 2024-04-02T04:58:13.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 7: 0 ) 2024-04-02T04:58:13.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 10 client.26765.objecter _finish_command 7 = system:0 2024-04-02T04:58:13.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 4 2024-04-02T04:58:13.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.925+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 3 7 2024-04-02T04:58:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:13.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:13.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 10 client.26765.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 8 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 8 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781b00c0 osd=3 3 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 3 8 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 4 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 10 client.26765.objecter _send_command 8 2024-04-02T04:58:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69781afba0 con 0x7f69781b4140 2024-04-02T04:58:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69781afba0 con 0x7f69781b4140 2024-04-02T04:58:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f696effd700 10 client.26765.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 4 2024-04-02T04:58:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 3 8 2024-04-02T04:58:13.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:58:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 10 client.26765.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 9 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69781b7aa0 0x7f69781b9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 9 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 4 9 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 4 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 10 client.26765.objecter _send_command 9 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f69781afb20 con 0x7f69781b7aa0 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.945+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69781b7aa0 0x7f69781b9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f69781b7aa0 2024-04-02T04:58:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69781b7aa0 0x7f69781b9e90 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69781afb20 con 0x7f69781b7aa0 2024-04-02T04:58:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 9: 0 ) 2024-04-02T04:58:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f696effd700 10 client.26765.objecter _finish_command 9 = system:0 2024-04-02T04:58:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 4 2024-04-02T04:58:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.949+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 4 9 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 10 client.26765.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 10 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 10 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781af8f0 osd=4 3 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 4 10 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 4 2024-04-02T04:58:13.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 10 client.26765.objecter _send_command 10 2024-04-02T04:58:13.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.965+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69781ba3d0 con 0x7f69781b7aa0 2024-04-02T04:58:13.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69781ba3d0 con 0x7f69781b7aa0 2024-04-02T04:58:13.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 10 client.26765.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 4 2024-04-02T04:58:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 4 10 2024-04-02T04:58:13.970 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 10 client.26765.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 11 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69781baa50 0x7f69781bce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 11 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 5 11 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 4 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 10 client.26765.objecter _send_command 11 2024-04-02T04:58:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f69781bfdd0 con 0x7f69781baa50 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69781baa50 0x7f69781bce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f69781baa50 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f697d3c0700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69781baa50 0x7f69781bce40 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69781bfdd0 con 0x7f69781baa50 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 11: 0 ) 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 10 client.26765.objecter _finish_command 11 = system:0 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 4 2024-04-02T04:58:13.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.969+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 5 11 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 10 client.26765.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 12 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 12 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781ba3d0 osd=5 3 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 5 12 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 4 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 10 client.26765.objecter _send_command 12 2024-04-02T04:58:13.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69781bfe30 con 0x7f69781baa50 2024-04-02T04:58:13.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69781bfe30 con 0x7f69781baa50 2024-04-02T04:58:13.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:13.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f696effd700 10 client.26765.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:13.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 4 2024-04-02T04:58:13.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 5 12 2024-04-02T04:58:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 10 client.26765.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 13 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69781bf160 0x7f69781bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 13 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 6 13 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 4 2024-04-02T04:58:13.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 10 client.26765.objecter _send_command 13 2024-04-02T04:58:13.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f69781c31a0 con 0x7f69781bf160 2024-04-02T04:58:13.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69781bf160 0x7f69781bf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:13.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f69781bf160 2024-04-02T04:58:13.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.989+0000 7f697e3c2700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69781bf160 0x7f69781bf5c0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:13.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.993+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69781c31a0 con 0x7f69781bf160 2024-04-02T04:58:13.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.993+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 13: 0 ) 2024-04-02T04:58:13.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.993+0000 7f696effd700 10 client.26765.objecter _finish_command 13 = system:0 2024-04-02T04:58:13.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.993+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 4 2024-04-02T04:58:13.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:13.993+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 6 13 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.009+0000 7f697fe25700 10 client.26765.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 14 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 14 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781bfe30 osd=6 3 2024-04-02T04:58:14.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 6 14 2024-04-02T04:58:14.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 4 2024-04-02T04:58:14.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 10 client.26765.objecter _send_command 14 2024-04-02T04:58:14.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.013+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69781c2020 con 0x7f69781bf160 2024-04-02T04:58:14.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69781c2020 con 0x7f69781bf160 2024-04-02T04:58:14.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:14.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 10 client.26765.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:14.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 4 2024-04-02T04:58:14.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 6 14 2024-04-02T04:58:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 10 client.26765.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 15 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f69781c2740 0x7f69781c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 15 2024-04-02T04:58:14.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 7 15 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 4 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 10 client.26765.objecter _send_command 15 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f69781c6510 con 0x7f69781c2740 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f69781c2740 0x7f69781c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 10 client.26765.objecter ms_handle_connect 0x7f69781c2740 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f697dbc1700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f69781c2740 0x7f69781c2b80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69781c6510 con 0x7f69781c2740 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 15: 0 ) 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 10 client.26765.objecter _finish_command 15 = system:0 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 4 2024-04-02T04:58:14.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.017+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 7 15 2024-04-02T04:58:14.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 10 client.26765.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:14.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign -1 16 2024-04-02T04:58:14.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 20 client.26765.objecter _get_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 15 client.26765.objecter _session_command_op_remove -1 16 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 20 client.26765.objecter get_session s=0x7f69781c2020 osd=7 3 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 15 client.26765.objecter _session_command_op_assign 7 16 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 4 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 10 client.26765.objecter _send_command 16 2024-04-02T04:58:14.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.033+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69781bd3d0 con 0x7f69781c2740 2024-04-02T04:58:14.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f696effd700 1 -- 172.21.15.69:0/2275525496 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f69781bd3d0 con 0x7f69781c2740 2024-04-02T04:58:14.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f696effd700 10 client.26765.objecter ms_dispatch 0x7f6978001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:14.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f696effd700 10 client.26765.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:14.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f696effd700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 4 2024-04-02T04:58:14.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f696effd700 15 client.26765.objecter _session_command_op_remove 7 16 2024-04-02T04:58:14.041 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:58:14.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 librados: watch_flush enter 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 librados: watch_flush exit 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.0 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69780630a0 msgr2=0x7f697810f6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f69780630a0 0x7f697810f6f0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f697819f9a0 osd=0 1 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.1 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6978062c90 msgr2=0x7f69781af3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6978062c90 0x7f69781af3b0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978062b20 osd=1 1 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.2 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f697802d690 msgr2=0x7f69781b2a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f697802d690 0x7f69781b2a80 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f6978110490 osd=2 1 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.3 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69781b4140 msgr2=0x7f69781b4560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f69781b4140 0x7f69781b4560 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781b00c0 osd=3 1 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.4 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69781b7aa0 msgr2=0x7f69781b9e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f69781b7aa0 0x7f69781b9e90 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781af8f0 osd=4 1 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.5 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69781baa50 msgr2=0x7f69781bce40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f69781baa50 0x7f69781bce40 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781ba3d0 osd=5 1 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 reap_dead start 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.6 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69781bf160 msgr2=0x7f69781bf5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69781bf160 0x7f69781bf5c0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 reap_dead start 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 reap_dead start 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697e3c2700 1 -- 172.21.15.69:0/2275525496 reap_dead start 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781bfe30 osd=6 1 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter close_session for osd.7 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f69781c2740 msgr2=0x7f69781c2b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f69781c2740 0x7f69781c2b80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter put_session s=0x7f69781c2020 osd=7 1 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 20 client.26765.objecter shutdown clearing up homeless session... 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 10 client.26765.objecter successfully canceled tick 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f696406f5e0 msgr2=0x7f6964071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f696406f5e0 0x7f6964071aa0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6968005950 tx=0x7f696800a300 comp rx=0 tx=0).stop 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 msgr2=0x7f69781a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:14.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 0x7f69781a2f10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f697000ef30 tx=0x7f697000c3a0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 shutdown_connections 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f69781c2740 0x7f69781c2b80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f69781bf160 0x7f69781bf5c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f697810e870 0x7f69781a2f10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 --2- 172.21.15.69:0/2275525496 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f696406f5e0 0x7f6964071aa0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 >> 172.21.15.69:0/2275525496 conn(0x7f6978075a10 msgr2=0x7f6978076340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 shutdown_connections 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 -- 172.21.15.69:0/2275525496 wait complete. 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:14.041+0000 7f697fe25700 1 librados: shutdown 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:58:14.047 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:58:14.048 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:58:14.048 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:58:14.048 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:58:14.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:14.067 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:14.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:14.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:14.181 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:13 smithi098 bash[23790]: audit 2024-04-02T04:58:13.469094+0000 mgr.y (mgr.14538) 569 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:14.181 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:13 smithi098 bash[23790]: audit 2024-04-02T04:58:13.867059+0000 mon.c (mon.2) 1088 : audit 0 from='client.? 172.21.15.69:0/2275525496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:14.198 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:14 smithi069 bash[28904]: audit 2024-04-02T04:58:13.469094+0000 mgr.y (mgr.14538) 569 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:14.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:14 smithi069 bash[28904]: audit 2024-04-02T04:58:13.867059+0000 mon.c (mon.2) 1088 : audit 0 from='client.? 172.21.15.69:0/2275525496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:14.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:14 smithi069 bash[21528]: audit 2024-04-02T04:58:13.469094+0000 mgr.y (mgr.14538) 569 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:14.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:14 smithi069 bash[21528]: audit 2024-04-02T04:58:13.867059+0000 mon.c (mon.2) 1088 : audit 0 from='client.? 172.21.15.69:0/2275525496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:14.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:14.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:14.458 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:14.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:14.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:14.718 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:14.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:14.846 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:14.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:14.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:15.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:15.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:15 smithi098 bash[23790]: cluster 2024-04-02T04:58:14.223715+0000 mgr.y (mgr.14538) 570 : cluster 0 pgmap v915: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-02T04:58:15.108 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 -- 172.21.15.69:0/3913757080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43481051d0 msgr2=0x7f4348105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 --2- 172.21.15.69:0/3913757080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43481051d0 0x7f4348105650 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f433c0099c0 tx=0x7f433c009cd0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 -- 172.21.15.69:0/3913757080 shutdown_connections 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 --2- 172.21.15.69:0/3913757080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f434806a9a0 0x7f434806ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 --2- 172.21.15.69:0/3913757080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4348107200 0x7f4348104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 --2- 172.21.15.69:0/3913757080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43481051d0 0x7f4348105650 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.137+0000 7f434d217700 1 -- 172.21.15.69:0/3913757080 >> 172.21.15.69:0/3913757080 conn(0x7f43480781d0 msgr2=0x7f43480785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- 172.21.15.69:0/3913757080 shutdown_connections 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- 172.21.15.69:0/3913757080 wait complete. 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 librados: starting msgr at 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 librados: starting objecter 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 Processor -- start 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- start start 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 librados: setting wanted keys 2024-04-02T04:58:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 librados: calling monclient init 2024-04-02T04:58:15.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434806a9a0 0x7f434819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43481051d0 0x7f434819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 0x7f43481a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43481180b0 con 0x7f4348107200 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4348117f30 con 0x7f43481051d0 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4348118230 con 0x7f434806a9a0 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434759e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 0x7f43481a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434806a9a0 0x7f434819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434759e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 0x7f43481a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43540/0 (socket says 172.21.15.69:43540) 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434759e700 1 -- 172.21.15.69:0/198454554 learned_addr learned my addr 172.21.15.69:0/198454554 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434659c700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43481051d0 0x7f434819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 -- 172.21.15.69:0/198454554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43481051d0 msgr2=0x7f434819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43481051d0 0x7f434819eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 -- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 msgr2=0x7f43481a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:15.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 0x7f43481a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 -- 172.21.15.69:0/198454554 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43481a3560 con 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434759e700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 0x7f43481a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f4346d9d700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434806a9a0 0x7f434819e5a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f433800e9f0 tx=0x7f433800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f432ffff700 10 client.?.objecter ms_handle_connect 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f432ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433800cb70 con 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f43380044a0 con 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43481a3850 con 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4338003cb0 con 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 10 client.26771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f43481ab0d0 con 0x7f434806a9a0 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 librados: init done 2024-04-02T04:58:15.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f434804ff00 con 0x7f434806a9a0 2024-04-02T04:58:15.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.141+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4338003e70 con 0x7f434806a9a0 2024-04-02T04:58:15.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f433006f5e0 0x7f4330071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(611..611 src has 74..611) ==== 10169+0+0 (secure 0 0 0) 0x7f433808ecd0 con 0x7f434806a9a0 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 10 client.26771.objecter ms_dispatch 0x7f4348001550 osd_map(611..611 src has 74..611) 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 3 client.26771.objecter handle_osd_map got epochs [611,611] > 0 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 3 client.26771.objecter handle_osd_map decoding full epoch 611 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f434659c700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f433006f5e0 0x7f4330071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 20 client.26771.objecter dump_active .. 0 homeless 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f432ffff700 10 client.26771.objecter ms_handle_connect 0x7f433006f5e0 2024-04-02T04:58:15.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.145+0000 7f434659c700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f433006f5e0 0x7f4330071aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f433c006010 tx=0x7f433c019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:15.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.149+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f433805b820 con 0x7f434806a9a0 2024-04-02T04:58:15.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:15.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:15 smithi069 bash[21528]: cluster 2024-04-02T04:58:14.223715+0000 mgr.y (mgr.14538) 570 : cluster 0 pgmap v915: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-02T04:58:15.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:15 smithi069 bash[28904]: cluster 2024-04-02T04:58:14.223715+0000 mgr.y (mgr.14538) 570 : cluster 0 pgmap v915: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-02T04:58:15.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:15.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.345+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7f43480ff6a0 con 0x7f433006f5e0 2024-04-02T04:58:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.349+0000 7f432ffff700 1 -- 172.21.15.69:0/198454554 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+298 (secure 0 0 0) 0x7f43480ff6a0 con 0x7f433006f5e0 2024-04-02T04:58:15.352 INFO:teuthology.orchestra.run.smithi069.stdout:moved 0 / 906 (0%) 2024-04-02T04:58:15.352 INFO:teuthology.orchestra.run.smithi069.stdout:avg 151 2024-04-02T04:58:15.352 INFO:teuthology.orchestra.run.smithi069.stdout:stddev 12.8452 -> 12.8452 (expected baseline 11.2175) 2024-04-02T04:58:15.352 INFO:teuthology.orchestra.run.smithi069.stdout:min osd.2 with 133 -> 133 pgs (0.880795 -> 0.880795 * mean) 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout:max osd.4 with 173 -> 173 pgs (1.1457 -> 1.1457 * mean) 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout:oload 1000 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout:max_change 0.05 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout:max_change_osds 4 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout:average_utilization 0.0009 2024-04-02T04:58:15.353 INFO:teuthology.orchestra.run.smithi069.stdout:overload_utilization 0.0091 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 10 librados: watch_flush enter 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 10 librados: watch_flush exit 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 20 client.26771.objecter shutdown clearing up homeless session... 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 10 client.26771.objecter successfully canceled tick 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f433006f5e0 msgr2=0x7f4330071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f433006f5e0 0x7f4330071aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f433c006010 tx=0x7f433c019040 comp rx=0 tx=0).stop 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434806a9a0 msgr2=0x7f434819e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434806a9a0 0x7f434819e5a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f433800e9f0 tx=0x7f433800ed00 comp rx=0 tx=0).stop 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 shutdown_connections 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f433006f5e0 0x7f4330071aa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4348107200 0x7f43481a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434806a9a0 0x7f434819e5a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 --2- 172.21.15.69:0/198454554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43481051d0 0x7f434819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 >> 172.21.15.69:0/198454554 conn(0x7f43480781d0 msgr2=0x7f43480fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 shutdown_connections 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 -- 172.21.15.69:0/198454554 wait complete. 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:15.353+0000 7f434d217700 1 librados: shutdown 2024-04-02T04:58:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:no change 2024-04-02T04:58:15.370 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:15.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:15.491 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:15.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:15.621 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:15.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:15.750 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:15.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:15.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:15.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:16.044 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:16.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:16.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-02T04:58:16.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:16 smithi098 bash[23790]: audit 2024-04-02T04:58:15.350302+0000 mgr.y (mgr.14538) 571 : audit 0 from='client.26771 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:16.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:16.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:16 smithi069 bash[21528]: audit 2024-04-02T04:58:15.350302+0000 mgr.y (mgr.14538) 571 : audit 0 from='client.26771 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:16.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:16 smithi069 bash[28904]: audit 2024-04-02T04:58:15.350302+0000 mgr.y (mgr.14538) 571 : audit 0 from='client.26771 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:16.302 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:16.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:16.434 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:16.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:16.565 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:16.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:16.695 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:16.798 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:16.823 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:16.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:16.954 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:16.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:17.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:17.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:17.196 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:17 smithi098 bash[23790]: cluster 2024-04-02T04:58:16.224342+0000 mgr.y (mgr.14538) 572 : cluster 0 pgmap v916: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-02T04:58:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:17.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:17 smithi069 bash[21528]: cluster 2024-04-02T04:58:16.224342+0000 mgr.y (mgr.14538) 572 : cluster 0 pgmap v916: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-02T04:58:17.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:17 smithi069 bash[28904]: cluster 2024-04-02T04:58:16.224342+0000 mgr.y (mgr.14538) 572 : cluster 0 pgmap v916: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-02T04:58:17.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:17.339 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:17.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:17.462 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:17.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 -- 172.21.15.69:0/727874176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8102520 msgr2=0x7f2fb8102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/727874176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8102520 0x7f2fb8102900 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80098f0 tx=0x7f2fa8009c00 comp rx=0 tx=0).stop 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 -- 172.21.15.69:0/727874176 shutdown_connections 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/727874176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb81075e0 0x7f2fb810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/727874176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8102520 0x7f2fb8102900 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/727874176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102ed0 0x7f2fb8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 -- 172.21.15.69:0/727874176 >> 172.21.15.69:0/727874176 conn(0x7f2fb80793f0 msgr2=0x7f2fb8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 -- 172.21.15.69:0/727874176 shutdown_connections 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 -- 172.21.15.69:0/727874176 wait complete. 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.545+0000 7f2fbd25c700 1 librados: starting msgr at 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 librados: starting objecter 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 Processor -- start 2024-04-02T04:58:17.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- start start 2024-04-02T04:58:17.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 librados: setting wanted keys 2024-04-02T04:58:17.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 librados: calling monclient init 2024-04-02T04:58:17.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 0x7f2fb8198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:17.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb8102ed0 0x7f2fb8198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:17.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb81075e0 0x7f2fb819c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb8111af0 con 0x7f2fb8102520 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb8111970 con 0x7f2fb8102ed0 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb8111c70 con 0x7f2fb81075e0 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 0x7f2fb8198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 0x7f2fb8198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48286/0 (socket says 172.21.15.69:48286) 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 -- 172.21.15.69:0/1493467106 learned_addr learned my addr 172.21.15.69:0/1493467106 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb759e700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb81075e0 0x7f2fb819c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb659c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb8102ed0 0x7f2fb8198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 -- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb81075e0 msgr2=0x7f2fb819c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb81075e0 0x7f2fb819c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 -- 172.21.15.69:0/1493467106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb8102ed0 msgr2=0x7f2fb8198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:17.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb8102ed0 0x7f2fb8198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 -- 172.21.15.69:0/1493467106 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb819d110 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb759e700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb81075e0 0x7f2fb819c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb6d9d700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 0x7f2fb8198100 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80059d0 tx=0x7f2fa800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 10 client.?.objecter ms_handle_connect 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8027070 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb819d3a0 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f2fa8005040 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8004910 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 10 client.16683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb81a4c80 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 librados: init done 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb804ff00 con 0x7f2fb8102520 2024-04-02T04:58:17.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f2fa802a070 con 0x7f2fb8102520 2024-04-02T04:58:17.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2fa006f5e0 0x7f2fa0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:17.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(611..611 src has 74..611) ==== 10169+0+0 (secure 0 0 0) 0x7f2fa808f640 con 0x7f2fb8102520 2024-04-02T04:58:17.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 10 client.16683.objecter ms_dispatch 0x7f2fb8001550 osd_map(611..611 src has 74..611) 2024-04-02T04:58:17.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 3 client.16683.objecter handle_osd_map got epochs [611,611] > 0 2024-04-02T04:58:17.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 3 client.16683.objecter handle_osd_map decoding full epoch 611 2024-04-02T04:58:17.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2fb659c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2fa006f5e0 0x7f2fa0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:17.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.549+0000 7f2f9ffff700 20 client.16683.objecter dump_active .. 0 homeless 2024-04-02T04:58:17.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.553+0000 7f2fb659c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2fa006f5e0 0x7f2fa0071aa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009750 tx=0x7f2fac009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:17.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.553+0000 7f2f9ffff700 10 client.16683.objecter ms_handle_connect 0x7f2fa006f5e0 2024-04-02T04:58:17.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.557+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2fa8013030 con 0x7f2fb8102520 2024-04-02T04:58:17.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:17.599 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:17.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:17.728 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:17.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:17.761+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f2fb81a4e50 con 0x7f2fb8102520 2024-04-02T04:58:17.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:17.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:17.986 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:17.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:18.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.037+0000 7f2f9ffff700 1 -- 172.21.15.69:0/1493467106 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v612) ==== 97+0+0 (secure 0 0 0) 0x7f2fa805c210 con 0x7f2fb8102520 2024-04-02T04:58:18.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 10 librados: watch_flush enter 2024-04-02T04:58:18.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 10 librados: watch_flush exit 2024-04-02T04:58:18.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 20 client.16683.objecter shutdown clearing up homeless session... 2024-04-02T04:58:18.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 10 client.16683.objecter successfully canceled tick 2024-04-02T04:58:18.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2fa006f5e0 msgr2=0x7f2fa0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:18.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2fa006f5e0 0x7f2fa0071aa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009750 tx=0x7f2fac009310 comp rx=0 tx=0).stop 2024-04-02T04:58:18.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 msgr2=0x7f2fb8198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 0x7f2fb8198100 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80059d0 tx=0x7f2fa800fb20 comp rx=0 tx=0).stop 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 shutdown_connections 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2fa006f5e0 0x7f2fa0071aa0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb81075e0 0x7f2fb819c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8102520 0x7f2fb8198100 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 --2- 172.21.15.69:0/1493467106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb8102ed0 0x7f2fb8198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 >> 172.21.15.69:0/1493467106 conn(0x7f2fb80793f0 msgr2=0x7f2fb8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 shutdown_connections 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 -- 172.21.15.69:0/1493467106 wait complete. 2024-04-02T04:58:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:18.045+0000 7f2fbd25c700 1 librados: shutdown 2024-04-02T04:58:18.048 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:58:18.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:18.099 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:18 smithi098 bash[23790]: audit 2024-04-02T04:58:17.766642+0000 mon.a (mon.0) 3170 : audit 1 from='client.? 172.21.15.69:0/1493467106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:58:18.116 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:18.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:18.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:18 smithi069 bash[21528]: audit 2024-04-02T04:58:17.766642+0000 mon.a (mon.0) 3170 : audit 1 from='client.? 172.21.15.69:0/1493467106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:58:18.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:18 smithi069 bash[28904]: audit 2024-04-02T04:58:17.766642+0000 mon.a (mon.0) 3170 : audit 1 from='client.? 172.21.15.69:0/1493467106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:58:18.244 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:18.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:18.374 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:18.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:18.503 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:18.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:18.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:58:18.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:18.762 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:58:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:58:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:58:18.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:18.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:18.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:19.005 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:19.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:19.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:19.264 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: cluster 2024-04-02T04:58:18.027624+0000 mon.a (mon.0) 3171 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: audit 2024-04-02T04:58:18.041893+0000 mon.a (mon.0) 3172 : audit 1 from='client.? 172.21.15.69:0/1493467106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: cluster 2024-04-02T04:58:18.046316+0000 mon.a (mon.0) 3173 : cluster 0 osdmap e612: 8 total, 8 up, 6 in 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: cluster 2024-04-02T04:58:18.225151+0000 mgr.y (mgr.14538) 573 : cluster 0 pgmap v918: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: cluster 2024-04-02T04:58:18.482089+0000 osd.0 (osd.0) 315 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: cluster 2024-04-02T04:58:18.482320+0000 osd.0 (osd.0) 316 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:19.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[28904]: audit 2024-04-02T04:58:18.579000+0000 mon.b (mon.1) 279 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: cluster 2024-04-02T04:58:18.027624+0000 mon.a (mon.0) 3171 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: audit 2024-04-02T04:58:18.041893+0000 mon.a (mon.0) 3172 : audit 1 from='client.? 172.21.15.69:0/1493467106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: cluster 2024-04-02T04:58:18.046316+0000 mon.a (mon.0) 3173 : cluster 0 osdmap e612: 8 total, 8 up, 6 in 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: cluster 2024-04-02T04:58:18.225151+0000 mgr.y (mgr.14538) 573 : cluster 0 pgmap v918: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: cluster 2024-04-02T04:58:18.482089+0000 osd.0 (osd.0) 315 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: cluster 2024-04-02T04:58:18.482320+0000 osd.0 (osd.0) 316 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:19.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:19 smithi069 bash[21528]: audit 2024-04-02T04:58:18.579000+0000 mon.b (mon.1) 279 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:19.292 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:19.395 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: cluster 2024-04-02T04:58:18.027624+0000 mon.a (mon.0) 3171 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: audit 2024-04-02T04:58:18.041893+0000 mon.a (mon.0) 3172 : audit 1 from='client.? 172.21.15.69:0/1493467106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: cluster 2024-04-02T04:58:18.046316+0000 mon.a (mon.0) 3173 : cluster 0 osdmap e612: 8 total, 8 up, 6 in 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: cluster 2024-04-02T04:58:18.225151+0000 mgr.y (mgr.14538) 573 : cluster 0 pgmap v918: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: cluster 2024-04-02T04:58:18.482089+0000 osd.0 (osd.0) 315 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: cluster 2024-04-02T04:58:18.482320+0000 osd.0 (osd.0) 316 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:19.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:19 smithi098 bash[23790]: audit 2024-04-02T04:58:18.579000+0000 mon.b (mon.1) 279 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:19.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:19.550 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:19.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:19.679 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:19.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:19.810 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:19.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:19.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:20.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:20.069 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:20.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:20.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:20 smithi069 bash[21528]: cluster 2024-04-02T04:58:19.051452+0000 mon.a (mon.0) 3174 : cluster 0 osdmap e613: 8 total, 8 up, 6 in 2024-04-02T04:58:20.184 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:20 smithi069 bash[28904]: cluster 2024-04-02T04:58:19.051452+0000 mon.a (mon.0) 3174 : cluster 0 osdmap e613: 8 total, 8 up, 6 in 2024-04-02T04:58:20.198 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:20.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:20.329 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:20.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:20.459 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:20.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:20 smithi098 bash[23790]: cluster 2024-04-02T04:58:19.051452+0000 mon.a (mon.0) 3174 : cluster 0 osdmap e613: 8 total, 8 up, 6 in 2024-04-02T04:58:20.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:20.591 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:20.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:20.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:20.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:20.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:58:20.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:20.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:21.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:21.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:21 smithi098 bash[23790]: cluster 2024-04-02T04:58:20.225996+0000 mgr.y (mgr.14538) 574 : cluster 0 pgmap v920: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-02T04:58:21.242 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:21.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:21 smithi069 bash[21528]: cluster 2024-04-02T04:58:20.225996+0000 mgr.y (mgr.14538) 574 : cluster 0 pgmap v920: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-02T04:58:21.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:21 smithi069 bash[28904]: cluster 2024-04-02T04:58:20.225996+0000 mgr.y (mgr.14538) 574 : cluster 0 pgmap v920: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-02T04:58:21.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:21.447 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:21.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:21.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:21.634 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:21.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.761 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:21.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:21.891 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:21.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:21.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:22.022 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:22.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:22.152 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:22.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:22.282 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:22.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:22.402 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7] out_osds: [5, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:58:22.402 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:58:22.402 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.9054127156913607 2024-04-02T04:58:22.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 0.9054127156913607 2024-04-02T04:58:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:22.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:22.539 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:22.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:22.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:22.791 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:22.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:22.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:23.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:23.042 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:23.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:23.173 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:23.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:23.303 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:23.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:23 smithi098 bash[23790]: cluster 2024-04-02T04:58:22.066419+0000 mon.a (mon.0) 3175 : cluster 0 osdmap e614: 8 total, 8 up, 6 in 2024-04-02T04:58:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:23 smithi098 bash[23790]: cluster 2024-04-02T04:58:22.227064+0000 mgr.y (mgr.14538) 575 : cluster 0 pgmap v922: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T04:58:23.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:23 smithi098 bash[23790]: audit 2024-04-02T04:58:22.456139+0000 mon.b (mon.1) 280 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:58:23.433 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:23.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:23 smithi069 bash[21528]: cluster 2024-04-02T04:58:22.066419+0000 mon.a (mon.0) 3175 : cluster 0 osdmap e614: 8 total, 8 up, 6 in 2024-04-02T04:58:23.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:23 smithi069 bash[21528]: cluster 2024-04-02T04:58:22.227064+0000 mgr.y (mgr.14538) 575 : cluster 0 pgmap v922: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T04:58:23.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:23 smithi069 bash[21528]: audit 2024-04-02T04:58:22.456139+0000 mon.b (mon.1) 280 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:58:23.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:23 smithi069 bash[28904]: cluster 2024-04-02T04:58:22.066419+0000 mon.a (mon.0) 3175 : cluster 0 osdmap e614: 8 total, 8 up, 6 in 2024-04-02T04:58:23.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:23 smithi069 bash[28904]: cluster 2024-04-02T04:58:22.227064+0000 mgr.y (mgr.14538) 575 : cluster 0 pgmap v922: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T04:58:23.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:23 smithi069 bash[28904]: audit 2024-04-02T04:58:22.456139+0000 mon.b (mon.1) 280 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:58:23.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:23.562 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:23.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:23.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:58:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:58:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:23.797 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:23.825 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:23.897 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:23.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:23.955 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:23.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:24.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:24.084 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:24.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:24.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:24 smithi098 bash[23790]: audit 2024-04-02T04:58:23.141035+0000 mon.b (mon.1) 281 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:58:24.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:24 smithi098 bash[23790]: audit 2024-04-02T04:58:23.142990+0000 mon.b (mon.1) 282 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:58:24.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:24 smithi098 bash[23790]: audit 2024-04-02T04:58:23.157750+0000 mon.a (mon.0) 3176 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:58:24.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:24 smithi098 bash[23790]: audit 2024-04-02T04:58:23.476939+0000 mgr.y (mgr.14538) 576 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:24.214 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[28904]: audit 2024-04-02T04:58:23.141035+0000 mon.b (mon.1) 281 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[28904]: audit 2024-04-02T04:58:23.142990+0000 mon.b (mon.1) 282 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[28904]: audit 2024-04-02T04:58:23.157750+0000 mon.a (mon.0) 3176 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[28904]: audit 2024-04-02T04:58:23.476939+0000 mgr.y (mgr.14538) 576 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[21528]: audit 2024-04-02T04:58:23.141035+0000 mon.b (mon.1) 281 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[21528]: audit 2024-04-02T04:58:23.142990+0000 mon.b (mon.1) 282 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[21528]: audit 2024-04-02T04:58:23.157750+0000 mon.a (mon.0) 3176 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:58:24.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:24 smithi069 bash[21528]: audit 2024-04-02T04:58:23.476939+0000 mgr.y (mgr.14538) 576 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:24.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:24.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:24.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:24.477 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:24.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:24.601 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:24.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:24.731 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:24.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:24.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:24.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:24.992 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:24.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:25.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:25.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:25 smithi098 bash[23790]: cluster 2024-04-02T04:58:24.229980+0000 mgr.y (mgr.14538) 577 : cluster 0 pgmap v923: 302 pgs: 1 active+clean+snaptrim, 301 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:58:25.106 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:25 smithi098 bash[23790]: cluster 2024-04-02T04:58:24.996543+0000 mon.a (mon.0) 3177 : cluster 0 osdmap e615: 8 total, 8 up, 6 in 2024-04-02T04:58:25.121 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:25.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:25.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:25 smithi069 bash[28904]: cluster 2024-04-02T04:58:24.229980+0000 mgr.y (mgr.14538) 577 : cluster 0 pgmap v923: 302 pgs: 1 active+clean+snaptrim, 301 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:58:25.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:25 smithi069 bash[28904]: cluster 2024-04-02T04:58:24.996543+0000 mon.a (mon.0) 3177 : cluster 0 osdmap e615: 8 total, 8 up, 6 in 2024-04-02T04:58:25.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:25 smithi069 bash[21528]: cluster 2024-04-02T04:58:24.229980+0000 mgr.y (mgr.14538) 577 : cluster 0 pgmap v923: 302 pgs: 1 active+clean+snaptrim, 301 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T04:58:25.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:25 smithi069 bash[21528]: cluster 2024-04-02T04:58:24.996543+0000 mon.a (mon.0) 3177 : cluster 0 osdmap e615: 8 total, 8 up, 6 in 2024-04-02T04:58:25.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:25.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:25.377 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:25.480 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:25.502 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:25.503 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:25.572 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:58:25.572 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:58:25.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:25.633 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:25.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:25.757 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:25.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:25.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:25.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:26.015 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:26.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:26.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:26.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:26.271 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:26.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:26.400 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:26.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:26.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:26.662 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:26.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:26.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:26.921 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:26.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:27.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:27.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:27.182 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:27.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:27.311 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:27.337 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:27 smithi069 bash[28904]: cluster 2024-04-02T04:58:26.230857+0000 mgr.y (mgr.14538) 578 : cluster 0 pgmap v925: 302 pgs: 1 active+clean+snaptrim, 301 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 998 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:58:27.338 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:27 smithi069 bash[21528]: cluster 2024-04-02T04:58:26.230857+0000 mgr.y (mgr.14538) 578 : cluster 0 pgmap v925: 302 pgs: 1 active+clean+snaptrim, 301 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 998 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:58:27.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:27.442 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:27.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:27.556 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:27 smithi098 bash[23790]: cluster 2024-04-02T04:58:26.230857+0000 mgr.y (mgr.14538) 578 : cluster 0 pgmap v925: 302 pgs: 1 active+clean+snaptrim, 301 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 998 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:58:27.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:27.672 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:27.700 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:27.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:27.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:27.963 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:27.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:28.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:28.090 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:28.106 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-02T04:58:28.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:28.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:28.351 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:28.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:28.481 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:28.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:28.643 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:28.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:28.779 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:58:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:58:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:58:28.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:28.870 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:28.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:28.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:29.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:29.132 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:29.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:29.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:29.390 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:29.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:29 smithi098 bash[23790]: cluster 2024-04-02T04:58:28.232126+0000 mgr.y (mgr.14538) 579 : cluster 0 pgmap v926: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:58:29.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:29.504 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:29 smithi069 bash[21528]: cluster 2024-04-02T04:58:28.232126+0000 mgr.y (mgr.14538) 579 : cluster 0 pgmap v926: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:58:29.504 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:29 smithi069 bash[28904]: cluster 2024-04-02T04:58:28.232126+0000 mgr.y (mgr.14538) 579 : cluster 0 pgmap v926: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T04:58:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:29.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:29.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:29.782 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:29.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:29.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:29.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:30.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:30.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:30.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:30.302 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:30.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:30.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:30 smithi098 bash[23790]: cluster 2024-04-02T04:58:28.643045+0000 osd.6 (osd.6) 267 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:30.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:30 smithi098 bash[23790]: cluster 2024-04-02T04:58:28.643297+0000 osd.6 (osd.6) 268 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:30.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:30 smithi098 bash[23790]: cluster 2024-04-02T04:58:29.315536+0000 mon.a (mon.0) 3178 : cluster 0 osdmap e616: 8 total, 8 up, 6 in 2024-04-02T04:58:30.417 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:30 smithi098 bash[23790]: cluster 2024-04-02T04:58:30.001170+0000 mon.a (mon.0) 3179 : cluster 0 osdmap e617: 8 total, 8 up, 6 in 2024-04-02T04:58:30.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[21528]: cluster 2024-04-02T04:58:28.643045+0000 osd.6 (osd.6) 267 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[21528]: cluster 2024-04-02T04:58:28.643297+0000 osd.6 (osd.6) 268 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[21528]: cluster 2024-04-02T04:58:29.315536+0000 mon.a (mon.0) 3178 : cluster 0 osdmap e616: 8 total, 8 up, 6 in 2024-04-02T04:58:30.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[21528]: cluster 2024-04-02T04:58:30.001170+0000 mon.a (mon.0) 3179 : cluster 0 osdmap e617: 8 total, 8 up, 6 in 2024-04-02T04:58:30.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[28904]: cluster 2024-04-02T04:58:28.643045+0000 osd.6 (osd.6) 267 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[28904]: cluster 2024-04-02T04:58:28.643297+0000 osd.6 (osd.6) 268 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[28904]: cluster 2024-04-02T04:58:29.315536+0000 mon.a (mon.0) 3178 : cluster 0 osdmap e616: 8 total, 8 up, 6 in 2024-04-02T04:58:30.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:30 smithi069 bash[28904]: cluster 2024-04-02T04:58:30.001170+0000 mon.a (mon.0) 3179 : cluster 0 osdmap e617: 8 total, 8 up, 6 in 2024-04-02T04:58:30.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:30.558 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:30.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:30.689 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:30.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:30.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:30.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:31.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:31.078 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:31.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:31.206 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:31.309 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:31.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:31 smithi098 bash[23790]: cluster 2024-04-02T04:58:30.232925+0000 mgr.y (mgr.14538) 580 : cluster 0 pgmap v929: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:58:31.516 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:31.542 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:31.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:31 smithi069 bash[21528]: cluster 2024-04-02T04:58:30.232925+0000 mgr.y (mgr.14538) 580 : cluster 0 pgmap v929: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:58:31.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:31 smithi069 bash[28904]: cluster 2024-04-02T04:58:30.232925+0000 mgr.y (mgr.14538) 580 : cluster 0 pgmap v929: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T04:58:31.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:31.669 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/1739100680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 msgr2=0x7f3150106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- 172.21.15.69:0/1739100680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150106eb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f31400099c0 tx=0x7f3140009cd0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/1739100680 shutdown_connections 2024-04-02T04:58:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- 172.21.15.69:0/1739100680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3150107590 0x7f315010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- 172.21.15.69:0/1739100680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31501024b0 0x7f3150102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- 172.21.15.69:0/1739100680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150106eb0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/1739100680 >> 172.21.15.69:0/1739100680 conn(0x7f3150079610 msgr2=0x7f3150079a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/1739100680 shutdown_connections 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/1739100680 wait complete. 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 librados: starting msgr at 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 librados: starting objecter 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 Processor -- start 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- start start 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 librados: setting wanted keys 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 librados: calling monclient init 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31501024b0 0x7f3150198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3150107590 0x7f315019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3150111aa0 con 0x7f31501024b0 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3150111920 con 0x7f3150107590 2024-04-02T04:58:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3150111c20 con 0x7f3150102e60 2024-04-02T04:58:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314effd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31501024b0 0x7f3150198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36984/0 (socket says 172.21.15.69:36984) 2024-04-02T04:58:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 -- 172.21.15.69:0/2329908426 learned_addr learned my addr 172.21.15.69:0/2329908426 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3150107590 0x7f315019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3150107590 msgr2=0x7f315019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3150107590 0x7f315019ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31501024b0 msgr2=0x7f3150198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31501024b0 0x7f3150198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315019d160 con 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31501024b0 0x7f3150198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150198690 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f314000fa30 tx=0x7f314000fa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f3137fff700 10 client.?.objecter ms_handle_connect 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f3137fff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140009e60 con 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f314000fd70 con 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315019d3f0 con 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314000bea0 con 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 10 client.26783.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31501a4cd0 con 0x7f3150102e60 2024-04-02T04:58:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 librados: init done 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.753+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3150103990 con 0x7f3150102e60 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f31400050d0 con 0x7f3150102e60 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f313806f5e0 0x7f3138071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(617..617 src has 74..617) ==== 10197+0+0 (secure 0 0 0) 0x7f314008fb50 con 0x7f3150102e60 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 osd_map(617..617 src has 74..617) 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 3 client.26783.objecter handle_osd_map got epochs [617,617] > 0 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 3 client.26783.objecter handle_osd_map decoding full epoch 617 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f313806f5e0 0x7f3138071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 20 client.26783.objecter dump_active .. 0 homeless 2024-04-02T04:58:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v617) ==== 54+0+16 (secure 0 0 0) 0x7f3140013070 con 0x7f3150102e60 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 10 client.26783.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 1 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f313806f5e0 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f313806f5e0 0x7f3138071aa0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3144009de0 tx=0x7f3144009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f31501098e0 0x7f3150109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 1 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 0 1 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 4 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 10 client.26783.objecter _send_command 1 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31501994b0 con 0x7f31501098e0 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f31501098e0 0x7f3150109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f31501098e0 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f31501098e0 0x7f3150109cc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31501994b0 con 0x7f31501098e0 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 1: 0 ) 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 10 client.26783.objecter _finish_command 1 = system:0 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 4 2024-04-02T04:58:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.757+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 0 1 2024-04-02T04:58:31.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 10 client.26783.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 2 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 2 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f315004ff00 osd=0 3 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 0 2 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 4 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 10 client.26783.objecter _send_command 2 2024-04-02T04:58:31.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3150109360 con 0x7f31501098e0 2024-04-02T04:58:31.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3150109360 con 0x7f31501098e0 2024-04-02T04:58:31.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 10 client.26783.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 4 2024-04-02T04:58:31.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 0 2 2024-04-02T04:58:31.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 10 client.26783.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3150063360 0x7f31501ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 1 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 4 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 10 client.26783.objecter _send_command 3 2024-04-02T04:58:31.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3150076480 con 0x7f3150063360 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3150063360 0x7f31501ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f3150063360 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3150063360 0x7f31501ab2c0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3150076480 con 0x7f3150063360 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 3: 0 ) 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 10 client.26783.objecter _finish_command 3 = system:0 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 4 2024-04-02T04:58:31.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.777+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 1 3 2024-04-02T04:58:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:31.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 10 client.26783.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 4 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 4 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f315002db30 osd=1 3 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 1 4 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 4 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 10 client.26783.objecter _send_command 4 2024-04-02T04:58:31.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.801+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f315002def0 con 0x7f3150063360 2024-04-02T04:58:31.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f315002def0 con 0x7f3150063360 2024-04-02T04:58:31.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 10 client.26783.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 4 2024-04-02T04:58:31.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 1 4 2024-04-02T04:58:31.806 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 10 client.26783.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 5 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3150062a40 0x7f31501ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 5 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 2 5 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 4 2024-04-02T04:58:31.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 10 client.26783.objecter _send_command 5 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f315002d690 con 0x7f3150062a40 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3150062a40 0x7f31501ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f3150062a40 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3150062a40 0x7f31501ae990 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f315002d690 con 0x7f3150062a40 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 5: 0 ) 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 10 client.26783.objecter _finish_command 5 = system:0 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 4 2024-04-02T04:58:31.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.805+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 2 5 2024-04-02T04:58:31.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 10 client.26783.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 6 2024-04-02T04:58:31.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 6 2024-04-02T04:58:31.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31500628d0 osd=2 3 2024-04-02T04:58:31.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 2 6 2024-04-02T04:58:31.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 4 2024-04-02T04:58:31.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 10 client.26783.objecter _send_command 6 2024-04-02T04:58:31.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.821+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f31501aeed0 con 0x7f3150062a40 2024-04-02T04:58:31.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f31501aeed0 con 0x7f3150062a40 2024-04-02T04:58:31.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 10 client.26783.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 4 2024-04-02T04:58:31.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 2 6 2024-04-02T04:58:31.826 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:58:31.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 10 client.26783.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 7 2024-04-02T04:58:31.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f31501abfd0 0x7f31501ac410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 7 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 3 7 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 4 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 10 client.26783.objecter _send_command 7 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f31501b4c50 con 0x7f31501abfd0 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f31501abfd0 0x7f31501ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f31501abfd0 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f31501abfd0 0x7f31501ac410 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31501b4c50 con 0x7f31501abfd0 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 7: 0 ) 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 10 client.26783.objecter _finish_command 7 = system:0 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 4 2024-04-02T04:58:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.825+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 3 7 2024-04-02T04:58:31.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 10 client.26783.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 8 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 8 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501aeed0 osd=3 3 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 3 8 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 4 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 10 client.26783.objecter _send_command 8 2024-04-02T04:58:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f31501aba50 con 0x7f31501abfd0 2024-04-02T04:58:31.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f31501aba50 con 0x7f31501abfd0 2024-04-02T04:58:31.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f3137fff700 10 client.26783.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 4 2024-04-02T04:58:31.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 3 8 2024-04-02T04:58:31.847 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 10 client.26783.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 9 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f31501afdc0 0x7f31501b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 9 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 4 9 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 4 2024-04-02T04:58:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 10 client.26783.objecter _send_command 9 2024-04-02T04:58:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f31501ab970 con 0x7f31501afdc0 2024-04-02T04:58:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f31501afdc0 0x7f31501b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f31501afdc0 2024-04-02T04:58:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.845+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f31501afdc0 0x7f31501b3440 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.849+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31501ab970 con 0x7f31501afdc0 2024-04-02T04:58:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.849+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 9: 0 ) 2024-04-02T04:58:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.849+0000 7f3137fff700 10 client.26783.objecter _finish_command 9 = system:0 2024-04-02T04:58:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.849+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 4 2024-04-02T04:58:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.849+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 4 9 2024-04-02T04:58:31.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 10 client.26783.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 10 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 10 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501ab800 osd=4 3 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 4 10 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 4 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 10 client.26783.objecter _send_command 10 2024-04-02T04:58:31.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.865+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f31501ab970 con 0x7f31501afdc0 2024-04-02T04:58:31.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f31501ab970 con 0x7f31501afdc0 2024-04-02T04:58:31.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f3137fff700 10 client.26783.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 4 2024-04-02T04:58:31.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 4 10 2024-04-02T04:58:31.871 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 10 client.26783.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 11 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f31501b3e90 0x7f31501b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 11 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 5 11 2024-04-02T04:58:31.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 4 2024-04-02T04:58:31.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 10 client.26783.objecter _send_command 11 2024-04-02T04:58:31.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f31501b4980 con 0x7f31501b3e90 2024-04-02T04:58:31.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f31501b3e90 0x7f31501b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f314e7fc700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f31501b3e90 0x7f31501b42d0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.869+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f31501b3e90 2024-04-02T04:58:31.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.873+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31501b4980 con 0x7f31501b3e90 2024-04-02T04:58:31.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.873+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 11: 0 ) 2024-04-02T04:58:31.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.873+0000 7f3137fff700 10 client.26783.objecter _finish_command 11 = system:0 2024-04-02T04:58:31.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.873+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 4 2024-04-02T04:58:31.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.873+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 5 11 2024-04-02T04:58:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 10 client.26783.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 12 2024-04-02T04:58:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 12 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501ab970 osd=5 3 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 5 12 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 4 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 10 client.26783.objecter _send_command 12 2024-04-02T04:58:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.889+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f31501b49e0 con 0x7f31501b3e90 2024-04-02T04:58:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f31501b49e0 con 0x7f31501b3e90 2024-04-02T04:58:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f3137fff700 10 client.26783.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 4 2024-04-02T04:58:31.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 5 12 2024-04-02T04:58:31.896 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:58:31.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 10 client.26783.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 13 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f31501bab60 0x7f31501baf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 13 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 6 13 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 4 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 10 client.26783.objecter _send_command 13 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f31501be9e0 con 0x7f31501bab60 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.893+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f31501bab60 0x7f31501baf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f31501bab60 2024-04-02T04:58:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f314f7fe700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f31501bab60 0x7f31501baf80 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31501be9e0 con 0x7f31501bab60 2024-04-02T04:58:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 13: 0 ) 2024-04-02T04:58:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f3137fff700 10 client.26783.objecter _finish_command 13 = system:0 2024-04-02T04:58:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 4 2024-04-02T04:58:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.897+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 6 13 2024-04-02T04:58:31.920 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:31.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:31.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 10 client.26783.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 14 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 14 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501b49e0 osd=6 3 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 6 14 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 4 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 10 client.26783.objecter _send_command 14 2024-04-02T04:58:31.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.921+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f31501b8e80 con 0x7f31501bab60 2024-04-02T04:58:31.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f31501b8e80 con 0x7f31501bab60 2024-04-02T04:58:31.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f3137fff700 10 client.26783.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 4 2024-04-02T04:58:31.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 6 14 2024-04-02T04:58:31.927 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 10 client.26783.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 15 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f31501b95a0 0x7f31501b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 15 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 7 15 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 4 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 10 client.26783.objecter _send_command 15 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f31501ba070 con 0x7f31501b95a0 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f31501b95a0 0x7f31501b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f3137fff700 10 client.26783.objecter ms_handle_connect 0x7f31501b95a0 2024-04-02T04:58:31.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.925+0000 7f314effd700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f31501b95a0 0x7f31501b99a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:31.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.929+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31501ba070 con 0x7f31501b95a0 2024-04-02T04:58:31.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.929+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 15: 0 ) 2024-04-02T04:58:31.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.929+0000 7f3137fff700 10 client.26783.objecter _finish_command 15 = system:0 2024-04-02T04:58:31.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.929+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 4 2024-04-02T04:58:31.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.929+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 7 15 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 10 client.26783.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign -1 16 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 20 client.26783.objecter _get_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 15 client.26783.objecter _session_command_op_remove -1 16 2024-04-02T04:58:31.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 20 client.26783.objecter get_session s=0x7f31501b8e80 osd=7 3 2024-04-02T04:58:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 15 client.26783.objecter _session_command_op_assign 7 16 2024-04-02T04:58:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 4 2024-04-02T04:58:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 10 client.26783.objecter _send_command 16 2024-04-02T04:58:31.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.945+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f31501ba030 con 0x7f31501b95a0 2024-04-02T04:58:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f3137fff700 1 -- 172.21.15.69:0/2329908426 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f31501ba030 con 0x7f31501b95a0 2024-04-02T04:58:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f3137fff700 10 client.26783.objecter ms_dispatch 0x7f3150001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f3137fff700 10 client.26783.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f3137fff700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 4 2024-04-02T04:58:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f3137fff700 15 client.26783.objecter _session_command_op_remove 7 16 2024-04-02T04:58:31.952 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 librados: watch_flush enter 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 librados: watch_flush exit 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.0 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f31501098e0 msgr2=0x7f3150109cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f31501098e0 0x7f3150109cc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315004ff00 osd=0 1 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.1 2024-04-02T04:58:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3150063360 msgr2=0x7f31501ab2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3150063360 0x7f31501ab2c0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f315002db30 osd=1 1 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.2 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3150062a40 msgr2=0x7f31501ae990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3150062a40 0x7f31501ae990 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31500628d0 osd=2 1 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.3 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f31501abfd0 msgr2=0x7f31501ac410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f31501abfd0 0x7f31501ac410 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501aeed0 osd=3 1 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.4 2024-04-02T04:58:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f31501afdc0 msgr2=0x7f31501b3440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f31501afdc0 0x7f31501b3440 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab800 osd=4 1 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.5 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f31501b3e90 msgr2=0x7f31501b42d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f31501b3e90 0x7f31501b42d0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501ab970 osd=5 1 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.6 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f31501bab60 msgr2=0x7f31501baf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f31501bab60 0x7f31501baf80 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b49e0 osd=6 1 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter close_session for osd.7 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f31501b95a0 msgr2=0x7f31501b99a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f31501b95a0 0x7f31501b99a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter put_session s=0x7f31501b8e80 osd=7 1 2024-04-02T04:58:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 20 client.26783.objecter shutdown clearing up homeless session... 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 10 client.26783.objecter successfully canceled tick 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f314f7fe700 1 -- 172.21.15.69:0/2329908426 reap_dead start 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f314f7fe700 1 -- 172.21.15.69:0/2329908426 reap_dead start 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f313806f5e0 msgr2=0x7f3138071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f313806f5e0 0x7f3138071aa0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3144009de0 tx=0x7f3144009190 comp rx=0 tx=0).stop 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f314f7fe700 1 -- 172.21.15.69:0/2329908426 reap_dead start 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f314f7fe700 1 -- 172.21.15.69:0/2329908426 reap_dead start 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 msgr2=0x7f3150198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 --2- 172.21.15.69:0/2329908426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3150102e60 0x7f3150198690 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f314000fa30 tx=0x7f314000fa60 comp rx=0 tx=0).stop 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f314f7fe700 1 -- 172.21.15.69:0/2329908426 reap_dead start 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 shutdown_connections 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.949+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 >> 172.21.15.69:0/2329908426 conn(0x7f3150079610 msgr2=0x7f31500ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.953+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 shutdown_connections 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.953+0000 7f315564c700 1 -- 172.21.15.69:0/2329908426 wait complete. 2024-04-02T04:58:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:31.953+0000 7f315564c700 1 librados: shutdown 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:58:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:58:32.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:32.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:32.172 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:32.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:32.306 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:32.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:32.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[28904]: audit 2024-04-02T04:58:31.760096+0000 mon.c (mon.2) 1089 : audit 0 from='client.? 172.21.15.69:0/2329908426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:32.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[28904]: audit 2024-04-02T04:58:32.090469+0000 mon.c (mon.2) 1090 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:32.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[28904]: audit 2024-04-02T04:58:32.090860+0000 mon.a (mon.0) 3180 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:32.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[28904]: audit 2024-04-02T04:58:32.091383+0000 mon.c (mon.2) 1091 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-94"}]: dispatch 2024-04-02T04:58:32.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[28904]: audit 2024-04-02T04:58:32.091653+0000 mon.a (mon.0) 3181 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-94"}]: dispatch 2024-04-02T04:58:32.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[21528]: audit 2024-04-02T04:58:31.760096+0000 mon.c (mon.2) 1089 : audit 0 from='client.? 172.21.15.69:0/2329908426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:32.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[21528]: audit 2024-04-02T04:58:32.090469+0000 mon.c (mon.2) 1090 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:32.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[21528]: audit 2024-04-02T04:58:32.090860+0000 mon.a (mon.0) 3180 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:32.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[21528]: audit 2024-04-02T04:58:32.091383+0000 mon.c (mon.2) 1091 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-94"}]: dispatch 2024-04-02T04:58:32.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:32 smithi069 bash[21528]: audit 2024-04-02T04:58:32.091653+0000 mon.a (mon.0) 3181 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-94"}]: dispatch 2024-04-02T04:58:32.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:32.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:32.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:32.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:32.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:32 smithi098 bash[23790]: audit 2024-04-02T04:58:31.760096+0000 mon.c (mon.2) 1089 : audit 0 from='client.? 172.21.15.69:0/2329908426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:32.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:32 smithi098 bash[23790]: audit 2024-04-02T04:58:32.090469+0000 mon.c (mon.2) 1090 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:32.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:32 smithi098 bash[23790]: audit 2024-04-02T04:58:32.090860+0000 mon.a (mon.0) 3180 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:32.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:32 smithi098 bash[23790]: audit 2024-04-02T04:58:32.091383+0000 mon.c (mon.2) 1091 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-94"}]: dispatch 2024-04-02T04:58:32.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:32 smithi098 bash[23790]: audit 2024-04-02T04:58:32.091653+0000 mon.a (mon.0) 3181 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-94"}]: dispatch 2024-04-02T04:58:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:32.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:32.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:32.945 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:32.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:33.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:33.075 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:33.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:33.206 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:33.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:33.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:33.455 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[21528]: cluster 2024-04-02T04:58:32.233541+0000 mgr.y (mgr.14538) 581 : cluster 0 pgmap v930: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 707 B/s rd, 0 op/s 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[21528]: cluster 2024-04-02T04:58:32.322288+0000 mon.a (mon.0) 3182 : cluster 0 osdmap e618: 8 total, 8 up, 6 in 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[21528]: cluster 2024-04-02T04:58:32.656949+0000 osd.7 (osd.7) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[21528]: cluster 2024-04-02T04:58:32.657319+0000 osd.7 (osd.7) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[28904]: cluster 2024-04-02T04:58:32.233541+0000 mgr.y (mgr.14538) 581 : cluster 0 pgmap v930: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 707 B/s rd, 0 op/s 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[28904]: cluster 2024-04-02T04:58:32.322288+0000 mon.a (mon.0) 3182 : cluster 0 osdmap e618: 8 total, 8 up, 6 in 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[28904]: cluster 2024-04-02T04:58:32.656949+0000 osd.7 (osd.7) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:33.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:33 smithi069 bash[28904]: cluster 2024-04-02T04:58:32.657319+0000 osd.7 (osd.7) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:33.472 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:33.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:33.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:58:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:58:33.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:33 smithi098 bash[23790]: cluster 2024-04-02T04:58:32.233541+0000 mgr.y (mgr.14538) 581 : cluster 0 pgmap v930: 302 pgs: 302 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail; 707 B/s rd, 0 op/s 2024-04-02T04:58:33.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:33 smithi098 bash[23790]: cluster 2024-04-02T04:58:32.322288+0000 mon.a (mon.0) 3182 : cluster 0 osdmap e618: 8 total, 8 up, 6 in 2024-04-02T04:58:33.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:33 smithi098 bash[23790]: cluster 2024-04-02T04:58:32.656949+0000 osd.7 (osd.7) 113 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:33.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:33 smithi098 bash[23790]: cluster 2024-04-02T04:58:32.657319+0000 osd.7 (osd.7) 114 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:33.705 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:33.729 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:33.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:33.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:33.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:34.093 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:34.121 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:34.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:34.249 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:34.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:34.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:34.511 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: cluster 2024-04-02T04:58:33.335477+0000 mon.a (mon.0) 3183 : cluster 0 osdmap e619: 8 total, 8 up, 6 in 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: audit 2024-04-02T04:58:33.337420+0000 mon.c (mon.2) 1092 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: audit 2024-04-02T04:58:33.340361+0000 mon.a (mon.0) 3184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: audit 2024-04-02T04:58:33.488371+0000 mgr.y (mgr.14538) 582 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: audit 2024-04-02T04:58:33.617310+0000 mon.b (mon.1) 283 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: audit 2024-04-02T04:58:34.323937+0000 mon.a (mon.0) 3185 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: cluster 2024-04-02T04:58:34.330217+0000 mon.a (mon.0) 3186 : cluster 0 osdmap e620: 8 total, 8 up, 6 in 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[21528]: audit 2024-04-02T04:58:34.334819+0000 mon.c (mon.2) 1093 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:34.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: cluster 2024-04-02T04:58:33.335477+0000 mon.a (mon.0) 3183 : cluster 0 osdmap e619: 8 total, 8 up, 6 in 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: audit 2024-04-02T04:58:33.337420+0000 mon.c (mon.2) 1092 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: audit 2024-04-02T04:58:33.340361+0000 mon.a (mon.0) 3184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: audit 2024-04-02T04:58:33.488371+0000 mgr.y (mgr.14538) 582 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: audit 2024-04-02T04:58:33.617310+0000 mon.b (mon.1) 283 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: audit 2024-04-02T04:58:34.323937+0000 mon.a (mon.0) 3185 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: cluster 2024-04-02T04:58:34.330217+0000 mon.a (mon.0) 3186 : cluster 0 osdmap e620: 8 total, 8 up, 6 in 2024-04-02T04:58:34.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:34 smithi069 bash[28904]: audit 2024-04-02T04:58:34.334819+0000 mon.c (mon.2) 1093 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:34.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:34.642 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: cluster 2024-04-02T04:58:33.335477+0000 mon.a (mon.0) 3183 : cluster 0 osdmap e619: 8 total, 8 up, 6 in 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: audit 2024-04-02T04:58:33.337420+0000 mon.c (mon.2) 1092 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: audit 2024-04-02T04:58:33.340361+0000 mon.a (mon.0) 3184 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: audit 2024-04-02T04:58:33.488371+0000 mgr.y (mgr.14538) 582 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: audit 2024-04-02T04:58:33.617310+0000 mon.b (mon.1) 283 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: audit 2024-04-02T04:58:34.323937+0000 mon.a (mon.0) 3185 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: cluster 2024-04-02T04:58:34.330217+0000 mon.a (mon.0) 3186 : cluster 0 osdmap e620: 8 total, 8 up, 6 in 2024-04-02T04:58:34.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:34 smithi098 bash[23790]: audit 2024-04-02T04:58:34.334819+0000 mon.c (mon.2) 1093 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:58:34.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:34.773 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:34.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:34.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:34.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:35.005 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:35.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.165 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/4112724076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0108390 msgr2=0x7fb0e0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/4112724076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0108390 0x7fb0e0108770 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80098f0 tx=0x7fb0d8009c00 comp rx=0 tx=0).stop 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/4112724076 shutdown_connections 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/4112724076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 0x7fb0e010f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/4112724076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0108390 0x7fb0e0108770 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/4112724076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e0102320 0x7fb0e01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:35.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/4112724076 >> 172.21.15.69:0/4112724076 conn(0x7fb0e0079310 msgr2=0x7fb0e0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:35.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/4112724076 shutdown_connections 2024-04-02T04:58:35.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/4112724076 wait complete. 2024-04-02T04:58:35.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 librados: starting msgr at 2024-04-02T04:58:35.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 librados: starting objecter 2024-04-02T04:58:35.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 Processor -- start 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- start start 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 librados: setting wanted keys 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 librados: calling monclient init 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0102320 0x7fb0e0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 0x7fb0e0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e0108390 0x7fb0e019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111af0 con 0x7fb0e0102ce0 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111970 con 0x7fb0e0108390 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e0111c70 con 0x7fb0e0102320 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0de59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0102320 0x7fb0e0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 0x7fb0e0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0de59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0102320 0x7fb0e0198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36994/0 (socket says 172.21.15.69:36994) 2024-04-02T04:58:35.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0de59c700 1 -- 172.21.15.69:0/501634531 learned_addr learned my addr 172.21.15.69:0/501634531 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ded9d700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e0108390 0x7fb0e019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 -- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0102320 msgr2=0x7fb0e0198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0102320 0x7fb0e0198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 -- 172.21.15.69:0/501634531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e0108390 msgr2=0x7fb0e019ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e0108390 0x7fb0e019ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 -- 172.21.15.69:0/501634531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e019d150 con 0x7fb0e0102ce0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0ddd9b700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 0x7fb0e0198680 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800bd70 tx=0x7fb0c800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0d77fe700 10 client.?.objecter ms_handle_connect 0x7fb0e0102ce0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0d77fe700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8019070 con 0x7fb0e0102ce0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb0c800cc90 con 0x7fb0e0102ce0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e019d440 con 0x7fb0e0102ce0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80125a0 con 0x7fb0e0102ce0 2024-04-02T04:58:35.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 10 client.16695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:35.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a4cc0 con 0x7fb0e0102ce0 2024-04-02T04:58:35.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.173+0000 7fb0e4e2c700 1 librados: init done 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e004ff00 con 0x7fb0e0102ce0 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb0c801c070 con 0x7fb0e0102ce0 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb0cc06f5e0 0x7fb0cc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(621..621 src has 74..621) ==== 10169+0+0 (secure 0 0 0) 0x7fb0c808e320 con 0x7fb0e0102ce0 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 10 client.16695.objecter ms_dispatch 0x7fb0e0001550 osd_map(621..621 src has 74..621) 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 3 client.16695.objecter handle_osd_map got epochs [621,621] > 0 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 3 client.16695.objecter handle_osd_map decoding full epoch 621 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0de59c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb0cc06f5e0 0x7fb0cc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 20 client.16695.objecter dump_active .. 0 homeless 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0d77fe700 10 client.16695.objecter ms_handle_connect 0x7fb0cc06f5e0 2024-04-02T04:58:35.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.177+0000 7fb0de59c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb0cc06f5e0 0x7fb0cc071aa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009680 tx=0x7fb0d8019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:35.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.181+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb0c805ae70 con 0x7fb0e0102ce0 2024-04-02T04:58:35.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.288 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:35.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:35.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:35.409+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fb0e01a4ef0 con 0x7fb0e0102ce0 2024-04-02T04:58:35.419 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:35.451 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:35.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:35.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:35 smithi069 bash[21528]: cluster 2024-04-02T04:58:34.234785+0000 mgr.y (mgr.14538) 583 : cluster 0 pgmap v933: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:35.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:35 smithi069 bash[21528]: cluster 2024-04-02T04:58:34.342167+0000 mon.a (mon.0) 3187 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:35.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:35 smithi069 bash[21528]: cluster 2024-04-02T04:58:35.003368+0000 mon.a (mon.0) 3188 : cluster 0 osdmap e621: 8 total, 8 up, 6 in 2024-04-02T04:58:35.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:35 smithi069 bash[28904]: cluster 2024-04-02T04:58:34.234785+0000 mgr.y (mgr.14538) 583 : cluster 0 pgmap v933: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:35.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:35 smithi069 bash[28904]: cluster 2024-04-02T04:58:34.342167+0000 mon.a (mon.0) 3187 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:35.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:35 smithi069 bash[28904]: cluster 2024-04-02T04:58:35.003368+0000 mon.a (mon.0) 3188 : cluster 0 osdmap e621: 8 total, 8 up, 6 in 2024-04-02T04:58:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:35.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:35.678 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:35.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:35 smithi098 bash[23790]: cluster 2024-04-02T04:58:34.234785+0000 mgr.y (mgr.14538) 583 : cluster 0 pgmap v933: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:35.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:35 smithi098 bash[23790]: cluster 2024-04-02T04:58:34.342167+0000 mon.a (mon.0) 3187 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:35.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:35 smithi098 bash[23790]: cluster 2024-04-02T04:58:35.003368+0000 mon.a (mon.0) 3188 : cluster 0 osdmap e621: 8 total, 8 up, 6 in 2024-04-02T04:58:35.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:35.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:35.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:36.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:36.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:36.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:36.331 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:36.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.353+0000 7fb0d77fe700 1 -- 172.21.15.69:0/501634531 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v622) ==== 91+0+0 (secure 0 0 0) 0x7fb0c805e8c0 con 0x7fb0e0102ce0 2024-04-02T04:58:36.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.357+0000 7fb0e4e2c700 10 librados: watch_flush enter 2024-04-02T04:58:36.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 10 librados: watch_flush exit 2024-04-02T04:58:36.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 20 client.16695.objecter shutdown clearing up homeless session... 2024-04-02T04:58:36.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 10 client.16695.objecter successfully canceled tick 2024-04-02T04:58:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb0cc06f5e0 msgr2=0x7fb0cc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb0cc06f5e0 0x7fb0cc071aa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009680 tx=0x7fb0d8019040 comp rx=0 tx=0).stop 2024-04-02T04:58:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 msgr2=0x7fb0e0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 0x7fb0e0198680 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800bd70 tx=0x7fb0c800bda0 comp rx=0 tx=0).stop 2024-04-02T04:58:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 shutdown_connections 2024-04-02T04:58:36.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb0cc06f5e0 0x7fb0cc071aa0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e0108390 0x7fb0e019ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0e0102320 0x7fb0e0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 --2- 172.21.15.69:0/501634531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0e0102ce0 0x7fb0e0198680 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 >> 172.21.15.69:0/501634531 conn(0x7fb0e0079310 msgr2=0x7fb0e0109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.361+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 shutdown_connections 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.365+0000 7fb0e4e2c700 1 -- 172.21.15.69:0/501634531 wait complete. 2024-04-02T04:58:36.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:36.365+0000 7fb0e4e2c700 1 librados: shutdown 2024-04-02T04:58:36.368 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:58:36.416 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:58:36.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:36.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:36 smithi098 bash[23790]: audit 2024-04-02T04:58:35.413120+0000 mon.a (mon.0) 3189 : audit 1 from='client.? 172.21.15.69:0/501634531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:58:36.483 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:36.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:36.607 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:36.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:36 smithi069 bash[21528]: audit 2024-04-02T04:58:35.413120+0000 mon.a (mon.0) 3189 : audit 1 from='client.? 172.21.15.69:0/501634531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:58:36.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:36 smithi069 bash[28904]: audit 2024-04-02T04:58:35.413120+0000 mon.a (mon.0) 3189 : audit 1 from='client.? 172.21.15.69:0/501634531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:58:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:36.710 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:36.736 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:36.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:36.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:36.996 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:36.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:37.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:37.127 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:37.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd42b0 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3cd0 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3e20 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd3f50 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd43e0 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd48b0 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd49e0 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4fc0 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4510 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4640 2024-04-02T04:58:37.245 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4770 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd50f0 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4b10 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4c60 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4d90 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5220 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd56f0 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5820 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd4ea0 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5950 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5a80 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5330 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5440 2024-04-02T04:58:37.246 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5590 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5bb0 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd60f0 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd6240 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd6800 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5d00 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5e30 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd5f80 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd6950 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd6350 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd64a0 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcd65d0 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcacab0 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac600 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac730 2024-04-02T04:58:37.247 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac860 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcac990 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcad090 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcacbe0 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcacd10 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcace20 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcacf70 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcad670 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcad1c0 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcad310 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcad420 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcad7a0 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55c99fcadc70 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.248 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.249 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.250 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.251 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:58:37.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:37.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:37.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:37.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:37 smithi098 bash[23790]: cluster 2024-04-02T04:58:36.235620+0000 mgr.y (mgr.14538) 584 : cluster 0 pgmap v936: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:37.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:37 smithi098 bash[23790]: cluster 2024-04-02T04:58:36.349708+0000 mon.a (mon.0) 3190 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:58:37.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:37 smithi098 bash[23790]: audit 2024-04-02T04:58:36.359320+0000 mon.a (mon.0) 3191 : audit 1 from='client.? 172.21.15.69:0/501634531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:58:37.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:37 smithi098 bash[23790]: cluster 2024-04-02T04:58:36.365717+0000 mon.a (mon.0) 3192 : cluster 0 osdmap e622: 8 total, 8 up, 6 in 2024-04-02T04:58:37.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:37.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[21528]: cluster 2024-04-02T04:58:36.235620+0000 mgr.y (mgr.14538) 584 : cluster 0 pgmap v936: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[21528]: cluster 2024-04-02T04:58:36.349708+0000 mon.a (mon.0) 3190 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[21528]: audit 2024-04-02T04:58:36.359320+0000 mon.a (mon.0) 3191 : audit 1 from='client.? 172.21.15.69:0/501634531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[21528]: cluster 2024-04-02T04:58:36.365717+0000 mon.a (mon.0) 3192 : cluster 0 osdmap e622: 8 total, 8 up, 6 in 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[28904]: cluster 2024-04-02T04:58:36.235620+0000 mgr.y (mgr.14538) 584 : cluster 0 pgmap v936: 302 pgs: 5 creating+peering, 27 unknown, 270 active+clean; 8.4 MiB data, 498 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[28904]: cluster 2024-04-02T04:58:36.349708+0000 mon.a (mon.0) 3190 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[28904]: audit 2024-04-02T04:58:36.359320+0000 mon.a (mon.0) 3191 : audit 1 from='client.? 172.21.15.69:0/501634531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:58:37.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:37 smithi069 bash[28904]: cluster 2024-04-02T04:58:36.365717+0000 mon.a (mon.0) 3192 : cluster 0 osdmap e622: 8 total, 8 up, 6 in 2024-04-02T04:58:37.516 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:37.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:37.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:37.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:37.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/841412491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48102f30 msgr2=0x7fdf48106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/841412491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48102f30 0x7fdf48106ed0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdf380099c0 tx=0x7fdf38009cd0 comp rx=0 tx=0).stop 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/841412491 shutdown_connections 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/841412491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf481075b0 0x7fdf4810f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/841412491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf48102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/841412491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48102f30 0x7fdf48106ed0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/841412491 >> 172.21.15.69:0/841412491 conn(0x7fdf480793a0 msgr2=0x7fdf480797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/841412491 shutdown_connections 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/841412491 wait complete. 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 librados: starting msgr at 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 librados: starting objecter 2024-04-02T04:58:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 Processor -- start 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- start start 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 librados: setting wanted keys 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 librados: calling monclient init 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf481980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48102f30 0x7fdf481985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf481075b0 0x7fdf4819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48111ae0 con 0x7fdf481075b0 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48111960 con 0x7fdf48102f30 2024-04-02T04:58:37.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fdf48111c60 con 0x7fdf48102580 2024-04-02T04:58:37.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf481980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf481980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:39912/0 (socket says 172.21.15.69:39912) 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4e498700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf481075b0 0x7fdf4819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 -- 172.21.15.69:0/3447516934 learned_addr learned my addr 172.21.15.69:0/3447516934 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4d496700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48102f30 0x7fdf481985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 -- 172.21.15.69:0/3447516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48102f30 msgr2=0x7fdf481985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48102f30 0x7fdf481985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 -- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf481075b0 msgr2=0x7fdf4819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf481075b0 0x7fdf4819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 -- 172.21.15.69:0/3447516934 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4819d0c0 con 0x7fdf48102580 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4e498700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf481075b0 0x7fdf4819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4dc97700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf481980b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400eb30 tx=0x7fdf4400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf3effd700 10 client.?.objecter ms_handle_connect 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf3effd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400cc90 con 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fdf4400ce30 con 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4819d3b0 con 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf44003df0 con 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 10 client.26795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481a4c30 con 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.905+0000 7fdf4fefb700 1 librados: init done 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4804ff00 con 0x7fdf48102580 2024-04-02T04:58:37.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fdf44003f90 con 0x7fdf48102580 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf3406f5e0 0x7fdf34071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(622..622 src has 74..622) ==== 10169+0+0 (secure 0 0 0) 0x7fdf4408ed30 con 0x7fdf48102580 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 10 client.26795.objecter ms_dispatch 0x7fdf48001550 osd_map(622..622 src has 74..622) 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 3 client.26795.objecter handle_osd_map got epochs [622,622] > 0 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 3 client.26795.objecter handle_osd_map decoding full epoch 622 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf4d496700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf3406f5e0 0x7fdf34071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 20 client.26795.objecter dump_active .. 0 homeless 2024-04-02T04:58:37.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf3effd700 10 client.26795.objecter ms_handle_connect 0x7fdf3406f5e0 2024-04-02T04:58:37.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.909+0000 7fdf4d496700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf3406f5e0 0x7fdf34071aa0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdf481089c0 tx=0x7fdf38005fb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:37.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:37.913+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf4405b900 con 0x7fdf48102580 2024-04-02T04:58:38.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:38.034 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:38.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:38.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.145+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607} v 0) -- 0x7fdf481a4ec0 con 0x7fdf48102580 2024-04-02T04:58:38.164 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:38.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:38.294 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:38.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.365+0000 7fdf3effd700 1 -- 172.21.15.69:0/3447516934 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]=0 reweighted osd.1 to 0.905413 (e7c9) v623) ==== 134+0+0 (secure 0 0 0) 0x7fdf4405f350 con 0x7fdf48102580 2024-04-02T04:58:38.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.373+0000 7fdf4fefb700 10 librados: watch_flush enter 2024-04-02T04:58:38.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.373+0000 7fdf4fefb700 10 librados: watch_flush exit 2024-04-02T04:58:38.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.373+0000 7fdf4fefb700 20 client.26795.objecter shutdown clearing up homeless session... 2024-04-02T04:58:38.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.373+0000 7fdf4fefb700 10 client.26795.objecter successfully canceled tick 2024-04-02T04:58:38.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf3406f5e0 msgr2=0x7fdf34071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf3406f5e0 0x7fdf34071aa0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdf481089c0 tx=0x7fdf38005fb0 comp rx=0 tx=0).stop 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 msgr2=0x7fdf481980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf481980b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400eb30 tx=0x7fdf4400ee40 comp rx=0 tx=0).stop 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 shutdown_connections 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fdf3406f5e0 0x7fdf34071aa0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf481075b0 0x7fdf4819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf48102580 0x7fdf481980b0 secure :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400eb30 tx=0x7fdf4400ee40 comp rx=0 tx=0).stop 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 --2- 172.21.15.69:0/3447516934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48102f30 0x7fdf481985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:38.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 >> 172.21.15.69:0/3447516934 conn(0x7fdf480793a0 msgr2=0x7fdf48100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:38.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 shutdown_connections 2024-04-02T04:58:38.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.377+0000 7fdf4fefb700 1 -- 172.21.15.69:0/3447516934 wait complete. 2024-04-02T04:58:38.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:38.381+0000 7fdf4fefb700 1 librados: shutdown 2024-04-02T04:58:38.382 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.1 to 0.905413 (e7c9) 2024-04-02T04:58:38.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:38.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:38.531 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:38 smithi098 bash[23790]: audit 2024-04-02T04:58:38.149370+0000 mon.c (mon.2) 1094 : audit 1 from='client.? 172.21.15.69:0/3447516934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]: dispatch 2024-04-02T04:58:38.531 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:38 smithi098 bash[23790]: audit 2024-04-02T04:58:38.149928+0000 mon.a (mon.0) 3193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]: dispatch 2024-04-02T04:58:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:38.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:38.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:38.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:38 smithi069 bash[21528]: audit 2024-04-02T04:58:38.149370+0000 mon.c (mon.2) 1094 : audit 1 from='client.? 172.21.15.69:0/3447516934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]: dispatch 2024-04-02T04:58:38.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:38 smithi069 bash[21528]: audit 2024-04-02T04:58:38.149928+0000 mon.a (mon.0) 3193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]: dispatch 2024-04-02T04:58:38.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:38 smithi069 bash[28904]: audit 2024-04-02T04:58:38.149370+0000 mon.c (mon.2) 1094 : audit 1 from='client.? 172.21.15.69:0/3447516934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]: dispatch 2024-04-02T04:58:38.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:38 smithi069 bash[28904]: audit 2024-04-02T04:58:38.149928+0000 mon.a (mon.0) 3193 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]: dispatch 2024-04-02T04:58:38.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:38.806 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:38.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:38.929 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:38.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:39.029 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:58:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:58:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:58:39.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:39.058 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:39.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:39.187 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:39.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:39.318 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:39.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:39.443 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:39.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.007118+0000 osd.6 (osd.6) 269 : cluster 0 102.f scrub starts 2024-04-02T04:58:39.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.014461+0000 osd.6 (osd.6) 270 : cluster 0 102.f scrub ok 2024-04-02T04:58:39.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.091639+0000 osd.4 (osd.4) 328 : cluster 0 4.9 scrub starts 2024-04-02T04:58:39.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.093849+0000 osd.4 (osd.4) 329 : cluster 0 4.9 scrub ok 2024-04-02T04:58:39.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.238042+0000 mgr.y (mgr.14538) 585 : cluster 0 pgmap v938: 302 pgs: 5 creating+peering, 297 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-04-02T04:58:39.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.264856+0000 osd.2 (osd.2) 285 : cluster 0 2.1 scrub starts 2024-04-02T04:58:39.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.267115+0000 osd.2 (osd.2) 286 : cluster 0 2.1 scrub ok 2024-04-02T04:58:39.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: audit 2024-04-02T04:58:38.358599+0000 mon.a (mon.0) 3194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]': finished 2024-04-02T04:58:39.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.365358+0000 mon.a (mon.0) 3195 : cluster 0 osdmap e623: 8 total, 8 up, 6 in 2024-04-02T04:58:39.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.408610+0000 osd.7 (osd.7) 115 : cluster 0 2.b scrub starts 2024-04-02T04:58:39.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:39.575 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:39.678 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:39.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:38.410903+0000 osd.7 (osd.7) 116 : cluster 0 2.b scrub ok 2024-04-02T04:58:39.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.366998+0000 mon.a (mon.0) 3196 : cluster 0 osdmap e624: 8 total, 8 up, 6 in 2024-04-02T04:58:39.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.007118+0000 osd.6 (osd.6) 269 : cluster 0 102.f scrub starts 2024-04-02T04:58:39.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.014461+0000 osd.6 (osd.6) 270 : cluster 0 102.f scrub ok 2024-04-02T04:58:39.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.091639+0000 osd.4 (osd.4) 328 : cluster 0 4.9 scrub starts 2024-04-02T04:58:39.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.093849+0000 osd.4 (osd.4) 329 : cluster 0 4.9 scrub ok 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.238042+0000 mgr.y (mgr.14538) 585 : cluster 0 pgmap v938: 302 pgs: 5 creating+peering, 297 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.264856+0000 osd.2 (osd.2) 285 : cluster 0 2.1 scrub starts 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.267115+0000 osd.2 (osd.2) 286 : cluster 0 2.1 scrub ok 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: audit 2024-04-02T04:58:38.358599+0000 mon.a (mon.0) 3194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]': finished 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.365358+0000 mon.a (mon.0) 3195 : cluster 0 osdmap e623: 8 total, 8 up, 6 in 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.408610+0000 osd.7 (osd.7) 115 : cluster 0 2.b scrub starts 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:38.410903+0000 osd.7 (osd.7) 116 : cluster 0 2.b scrub ok 2024-04-02T04:58:39.691 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:39 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.366998+0000 mon.a (mon.0) 3196 : cluster 0 osdmap e624: 8 total, 8 up, 6 in 2024-04-02T04:58:39.706 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.007118+0000 osd.6 (osd.6) 269 : cluster 0 102.f scrub starts 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.014461+0000 osd.6 (osd.6) 270 : cluster 0 102.f scrub ok 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.091639+0000 osd.4 (osd.4) 328 : cluster 0 4.9 scrub starts 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.093849+0000 osd.4 (osd.4) 329 : cluster 0 4.9 scrub ok 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.238042+0000 mgr.y (mgr.14538) 585 : cluster 0 pgmap v938: 302 pgs: 5 creating+peering, 297 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.264856+0000 osd.2 (osd.2) 285 : cluster 0 2.1 scrub starts 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.267115+0000 osd.2 (osd.2) 286 : cluster 0 2.1 scrub ok 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: audit 2024-04-02T04:58:38.358599+0000 mon.a (mon.0) 3194 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.9054127156913607}]': finished 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.365358+0000 mon.a (mon.0) 3195 : cluster 0 osdmap e623: 8 total, 8 up, 6 in 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.408610+0000 osd.7 (osd.7) 115 : cluster 0 2.b scrub starts 2024-04-02T04:58:39.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:38.410903+0000 osd.7 (osd.7) 116 : cluster 0 2.b scrub ok 2024-04-02T04:58:39.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:39 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.366998+0000 mon.a (mon.0) 3196 : cluster 0 osdmap e624: 8 total, 8 up, 6 in 2024-04-02T04:58:39.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:58:39.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:39.836 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:39.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:39.958 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:39.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:40.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:40.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:40.211 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:40.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:40.342 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:40.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:40.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.001391+0000 osd.6 (osd.6) 271 : cluster 0 10.7 scrub starts 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.004080+0000 osd.6 (osd.6) 272 : cluster 0 10.7 scrub ok 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.995377+0000 mon.a (mon.0) 3197 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[21528]: cluster 2024-04-02T04:58:40.001709+0000 mon.a (mon.0) 3198 : cluster 0 osdmap e625: 8 total, 8 up, 6 in 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.001391+0000 osd.6 (osd.6) 271 : cluster 0 10.7 scrub starts 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.004080+0000 osd.6 (osd.6) 272 : cluster 0 10.7 scrub ok 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.995377+0000 mon.a (mon.0) 3197 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:40.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:40 smithi069 bash[28904]: cluster 2024-04-02T04:58:40.001709+0000 mon.a (mon.0) 3198 : cluster 0 osdmap e625: 8 total, 8 up, 6 in 2024-04-02T04:58:40.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:40.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:40 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.001391+0000 osd.6 (osd.6) 271 : cluster 0 10.7 scrub starts 2024-04-02T04:58:40.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:40 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.004080+0000 osd.6 (osd.6) 272 : cluster 0 10.7 scrub ok 2024-04-02T04:58:40.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:40 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.995377+0000 mon.a (mon.0) 3197 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:58:40.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:40 smithi098 bash[23790]: cluster 2024-04-02T04:58:40.001709+0000 mon.a (mon.0) 3198 : cluster 0 osdmap e625: 8 total, 8 up, 6 in 2024-04-02T04:58:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:40.705 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:40.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:40.785 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:58:40.785 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T04:58:40.785 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2024-04-02T04:58:40.785 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:58:40.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:40.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:40.951 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:40.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:40.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:41.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:41.104 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:41.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:41.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:41.233 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:41.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:41.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:41.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.973581+0000 osd.6 (osd.6) 273 : cluster 0 40.5 scrub starts 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.975832+0000 osd.6 (osd.6) 274 : cluster 0 40.5 scrub ok 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.985496+0000 osd.1 (osd.1) 321 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.985670+0000 osd.1 (osd.1) 322 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.994677+0000 osd.0 (osd.0) 317 : cluster 0 40.18 scrub starts 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:39.996112+0000 osd.0 (osd.0) 318 : cluster 0 40.18 scrub ok 2024-04-02T04:58:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:41 smithi098 bash[23790]: cluster 2024-04-02T04:58:40.238807+0000 mgr.y (mgr.14538) 586 : cluster 0 pgmap v942: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:58:41.491 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:41.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:41.606 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.973581+0000 osd.6 (osd.6) 273 : cluster 0 40.5 scrub starts 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.975832+0000 osd.6 (osd.6) 274 : cluster 0 40.5 scrub ok 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.985496+0000 osd.1 (osd.1) 321 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.985670+0000 osd.1 (osd.1) 322 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.994677+0000 osd.0 (osd.0) 317 : cluster 0 40.18 scrub starts 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:39.996112+0000 osd.0 (osd.0) 318 : cluster 0 40.18 scrub ok 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[21528]: cluster 2024-04-02T04:58:40.238807+0000 mgr.y (mgr.14538) 586 : cluster 0 pgmap v942: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.973581+0000 osd.6 (osd.6) 273 : cluster 0 40.5 scrub starts 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.975832+0000 osd.6 (osd.6) 274 : cluster 0 40.5 scrub ok 2024-04-02T04:58:41.607 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.985496+0000 osd.1 (osd.1) 321 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:41.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.985670+0000 osd.1 (osd.1) 322 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:41.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.994677+0000 osd.0 (osd.0) 317 : cluster 0 40.18 scrub starts 2024-04-02T04:58:41.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:39.996112+0000 osd.0 (osd.0) 318 : cluster 0 40.18 scrub ok 2024-04-02T04:58:41.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:41 smithi069 bash[28904]: cluster 2024-04-02T04:58:40.238807+0000 mgr.y (mgr.14538) 586 : cluster 0 pgmap v942: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:58:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:41.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-02T04:58:41.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:41.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:41.882 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:41.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:41.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:42.012 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:42.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:42.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:42.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:42.269 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:42.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:42.398 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:42.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:42.527 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:42.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:42.657 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:42 smithi098 bash[23790]: cluster 2024-04-02T04:58:41.395562+0000 mon.a (mon.0) 3199 : cluster 0 osdmap e626: 8 total, 8 up, 6 in 2024-04-02T04:58:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:42.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:42.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:42 smithi069 bash[21528]: cluster 2024-04-02T04:58:41.395562+0000 mon.a (mon.0) 3199 : cluster 0 osdmap e626: 8 total, 8 up, 6 in 2024-04-02T04:58:42.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:42 smithi069 bash[28904]: cluster 2024-04-02T04:58:41.395562+0000 mon.a (mon.0) 3199 : cluster 0 osdmap e626: 8 total, 8 up, 6 in 2024-04-02T04:58:42.784 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:42.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:42.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:43.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:43.041 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:43.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:43.209 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:43.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:43.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:43.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.061100+0000 osd.4 (osd.4) 330 : cluster 0 2.18 scrub starts 2024-04-02T04:58:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.062472+0000 osd.4 (osd.4) 331 : cluster 0 2.18 scrub ok 2024-04-02T04:58:43.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.225480+0000 osd.2 (osd.2) 287 : cluster 0 2.10 scrub starts 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.230235+0000 osd.2 (osd.2) 288 : cluster 0 2.10 scrub ok 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.239600+0000 mgr.y (mgr.14538) 587 : cluster 0 pgmap v944: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.409952+0000 mon.a (mon.0) 3200 : cluster 0 osdmap e627: 8 total, 8 up, 6 in 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.481524+0000 osd.7 (osd.7) 117 : cluster 0 3.16 scrub starts 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.485983+0000 osd.7 (osd.7) 118 : cluster 0 3.16 scrub ok 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.061100+0000 osd.4 (osd.4) 330 : cluster 0 2.18 scrub starts 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.062472+0000 osd.4 (osd.4) 331 : cluster 0 2.18 scrub ok 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.225480+0000 osd.2 (osd.2) 287 : cluster 0 2.10 scrub starts 2024-04-02T04:58:43.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.230235+0000 osd.2 (osd.2) 288 : cluster 0 2.10 scrub ok 2024-04-02T04:58:43.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.239600+0000 mgr.y (mgr.14538) 587 : cluster 0 pgmap v944: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:43.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.409952+0000 mon.a (mon.0) 3200 : cluster 0 osdmap e627: 8 total, 8 up, 6 in 2024-04-02T04:58:43.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.481524+0000 osd.7 (osd.7) 117 : cluster 0 3.16 scrub starts 2024-04-02T04:58:43.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:43 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.485983+0000 osd.7 (osd.7) 118 : cluster 0 3.16 scrub ok 2024-04-02T04:58:43.536 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:43.560 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:43.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:43.689 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:43.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.061100+0000 osd.4 (osd.4) 330 : cluster 0 2.18 scrub starts 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.062472+0000 osd.4 (osd.4) 331 : cluster 0 2.18 scrub ok 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.225480+0000 osd.2 (osd.2) 287 : cluster 0 2.10 scrub starts 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.230235+0000 osd.2 (osd.2) 288 : cluster 0 2.10 scrub ok 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.239600+0000 mgr.y (mgr.14538) 587 : cluster 0 pgmap v944: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.409952+0000 mon.a (mon.0) 3200 : cluster 0 osdmap e627: 8 total, 8 up, 6 in 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.481524+0000 osd.7 (osd.7) 117 : cluster 0 3.16 scrub starts 2024-04-02T04:58:43.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:43 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.485983+0000 osd.7 (osd.7) 118 : cluster 0 3.16 scrub ok 2024-04-02T04:58:43.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:43.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:43.950 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:43.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:44.053 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:44.080 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:44.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:44.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:44.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:44.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:44.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:44 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.898125+0000 osd.1 (osd.1) 323 : cluster 0 2.15 scrub starts 2024-04-02T04:58:44.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:44 smithi098 bash[23790]: cluster 2024-04-02T04:58:42.899848+0000 osd.1 (osd.1) 324 : cluster 0 2.15 scrub ok 2024-04-02T04:58:44.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:44 smithi098 bash[23790]: cluster 2024-04-02T04:58:43.231478+0000 osd.2 (osd.2) 289 : cluster 0 43.9 scrub starts 2024-04-02T04:58:44.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:44 smithi098 bash[23790]: cluster 2024-04-02T04:58:43.232904+0000 osd.2 (osd.2) 290 : cluster 0 43.9 scrub ok 2024-04-02T04:58:44.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:44 smithi098 bash[23790]: audit 2024-04-02T04:58:43.499863+0000 mgr.y (mgr.14538) 588 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:44.465 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:44.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:44.600 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:44.630 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.898125+0000 osd.1 (osd.1) 323 : cluster 0 2.15 scrub starts 2024-04-02T04:58:44.630 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[21528]: cluster 2024-04-02T04:58:42.899848+0000 osd.1 (osd.1) 324 : cluster 0 2.15 scrub ok 2024-04-02T04:58:44.630 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[21528]: cluster 2024-04-02T04:58:43.231478+0000 osd.2 (osd.2) 289 : cluster 0 43.9 scrub starts 2024-04-02T04:58:44.630 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[21528]: cluster 2024-04-02T04:58:43.232904+0000 osd.2 (osd.2) 290 : cluster 0 43.9 scrub ok 2024-04-02T04:58:44.630 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[21528]: audit 2024-04-02T04:58:43.499863+0000 mgr.y (mgr.14538) 588 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:44.631 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.898125+0000 osd.1 (osd.1) 323 : cluster 0 2.15 scrub starts 2024-04-02T04:58:44.631 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[28904]: cluster 2024-04-02T04:58:42.899848+0000 osd.1 (osd.1) 324 : cluster 0 2.15 scrub ok 2024-04-02T04:58:44.631 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[28904]: cluster 2024-04-02T04:58:43.231478+0000 osd.2 (osd.2) 289 : cluster 0 43.9 scrub starts 2024-04-02T04:58:44.631 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[28904]: cluster 2024-04-02T04:58:43.232904+0000 osd.2 (osd.2) 290 : cluster 0 43.9 scrub ok 2024-04-02T04:58:44.631 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:44 smithi069 bash[28904]: audit 2024-04-02T04:58:43.499863+0000 mgr.y (mgr.14538) 588 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:44.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:44.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:44.862 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:44.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:44.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:45.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:45.121 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:45.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7] out_osds: [5, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:58:45.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:58:45.207 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-02T04:58:45.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd in 3 2024-04-02T04:58:45.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:45.234 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[38371]: debug 2024-04-02T04:58:45.052+0000 7ff2d11de700 -1 snap_mapper.add_oid found existing snaps mapped on 102:d1ddb9c0:test-rados-api-smithi069-72349-97::foo:21, removing 2024-04-02T04:58:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:45.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:45.415 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[42900]: debug 2024-04-02T04:58:45.060+0000 7f8c3607d700 -1 snap_mapper.add_oid found existing snaps mapped on 102:d1ddb9c0:test-rados-api-smithi069-72349-97::foo:21, removing 2024-04-02T04:58:45.415 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[31935]: debug 2024-04-02T04:58:45.060+0000 7f46f7483700 -1 snap_mapper.add_oid found existing snaps mapped on 102:d1ddb9c0:test-rados-api-smithi069-72349-97::foo:21, removing 2024-04-02T04:58:45.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: cluster 2024-04-02T04:58:44.241326+0000 mgr.y (mgr.14538) 589 : cluster 0 pgmap v946: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 146 B/s, 8 objects/s recovering 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: cluster 2024-04-02T04:58:44.269958+0000 osd.2 (osd.2) 291 : cluster 0 1.a scrub starts 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: cluster 2024-04-02T04:58:44.271254+0000 osd.2 (osd.2) 292 : cluster 0 1.a scrub ok 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: audit 2024-04-02T04:58:45.102616+0000 mon.c (mon.2) 1095 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: audit 2024-04-02T04:58:45.103379+0000 mon.a (mon.0) 3201 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: audit 2024-04-02T04:58:45.104332+0000 mon.c (mon.2) 1096 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-96"}]: dispatch 2024-04-02T04:58:45.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:45 smithi098 bash[23790]: audit 2024-04-02T04:58:45.104893+0000 mon.a (mon.0) 3202 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-96"}]: dispatch 2024-04-02T04:58:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:45.529 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: cluster 2024-04-02T04:58:44.241326+0000 mgr.y (mgr.14538) 589 : cluster 0 pgmap v946: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 146 B/s, 8 objects/s recovering 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: cluster 2024-04-02T04:58:44.269958+0000 osd.2 (osd.2) 291 : cluster 0 1.a scrub starts 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: cluster 2024-04-02T04:58:44.271254+0000 osd.2 (osd.2) 292 : cluster 0 1.a scrub ok 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: audit 2024-04-02T04:58:45.102616+0000 mon.c (mon.2) 1095 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: audit 2024-04-02T04:58:45.103379+0000 mon.a (mon.0) 3201 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: audit 2024-04-02T04:58:45.104332+0000 mon.c (mon.2) 1096 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-96"}]: dispatch 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[21528]: audit 2024-04-02T04:58:45.104893+0000 mon.a (mon.0) 3202 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-96"}]: dispatch 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: cluster 2024-04-02T04:58:44.241326+0000 mgr.y (mgr.14538) 589 : cluster 0 pgmap v946: 302 pgs: 302 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 146 B/s, 8 objects/s recovering 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: cluster 2024-04-02T04:58:44.269958+0000 osd.2 (osd.2) 291 : cluster 0 1.a scrub starts 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: cluster 2024-04-02T04:58:44.271254+0000 osd.2 (osd.2) 292 : cluster 0 1.a scrub ok 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: audit 2024-04-02T04:58:45.102616+0000 mon.c (mon.2) 1095 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:45.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: audit 2024-04-02T04:58:45.103379+0000 mon.a (mon.0) 3201 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:45.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: audit 2024-04-02T04:58:45.104332+0000 mon.c (mon.2) 1096 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-96"}]: dispatch 2024-04-02T04:58:45.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:45 smithi069 bash[28904]: audit 2024-04-02T04:58:45.104893+0000 mon.a (mon.0) 3202 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-96"}]: dispatch 2024-04-02T04:58:45.613 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:45.641 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:45.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:45.771 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:45.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:45.878 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:45.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:46.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:46.033 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:46.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:46.263 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:46.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:46.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:46.416 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:46.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:46.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:46 smithi069 bash[28904]: cluster 2024-04-02T04:58:45.056206+0000 osd.0 (osd.0) 319 : cluster 0 2.13 scrub starts 2024-04-02T04:58:46.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:46 smithi069 bash[28904]: cluster 2024-04-02T04:58:45.058971+0000 osd.0 (osd.0) 320 : cluster 0 2.13 scrub ok 2024-04-02T04:58:46.528 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:46 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.009940+0000 mon.a (mon.0) 3203 : cluster 0 osdmap e628: 8 total, 8 up, 6 in 2024-04-02T04:58:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:46 smithi069 bash[21528]: cluster 2024-04-02T04:58:45.056206+0000 osd.0 (osd.0) 319 : cluster 0 2.13 scrub starts 2024-04-02T04:58:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:46 smithi069 bash[21528]: cluster 2024-04-02T04:58:45.058971+0000 osd.0 (osd.0) 320 : cluster 0 2.13 scrub ok 2024-04-02T04:58:46.528 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:46 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.009940+0000 mon.a (mon.0) 3203 : cluster 0 osdmap e628: 8 total, 8 up, 6 in 2024-04-02T04:58:46.541 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:46.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:46.670 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:46 smithi098 bash[23790]: cluster 2024-04-02T04:58:45.056206+0000 osd.0 (osd.0) 319 : cluster 0 2.13 scrub starts 2024-04-02T04:58:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:46 smithi098 bash[23790]: cluster 2024-04-02T04:58:45.058971+0000 osd.0 (osd.0) 320 : cluster 0 2.13 scrub ok 2024-04-02T04:58:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:46 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.009940+0000 mon.a (mon.0) 3203 : cluster 0 osdmap e628: 8 total, 8 up, 6 in 2024-04-02T04:58:46.774 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:46.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:46.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:46.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:47.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:47.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:47.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:47.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:47.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:47.313 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:47.416 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:47.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:47.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:45.276384+0000 osd.2 (osd.2) 293 : cluster 0 2.c scrub starts 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:45.278344+0000 osd.2 (osd.2) 294 : cluster 0 2.c scrub ok 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:45.935409+0000 osd.1 (osd.1) 325 : cluster 0 6.1 scrub starts 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:45.936949+0000 osd.1 (osd.1) 326 : cluster 0 6.1 scrub ok 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.242123+0000 mgr.y (mgr.14538) 590 : cluster 0 pgmap v948: 270 pgs: 270 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 119 B/s, 6 objects/s recovering 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.270975+0000 osd.2 (osd.2) 295 : cluster 0 3.5 scrub starts 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.273853+0000 osd.2 (osd.2) 296 : cluster 0 3.5 scrub ok 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.815025+0000 osd.6 (osd.6) 275 : cluster 0 40.4 scrub starts 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.817835+0000 osd.6 (osd.6) 276 : cluster 0 40.4 scrub ok 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: cluster 2024-04-02T04:58:47.023490+0000 mon.a (mon.0) 3204 : cluster 0 osdmap e629: 8 total, 8 up, 6 in 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: audit 2024-04-02T04:58:47.026208+0000 mon.c (mon.2) 1097 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:47 smithi098 bash[23790]: audit 2024-04-02T04:58:47.027405+0000 mon.a (mon.0) 3205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:45.276384+0000 osd.2 (osd.2) 293 : cluster 0 2.c scrub starts 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:45.278344+0000 osd.2 (osd.2) 294 : cluster 0 2.c scrub ok 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:45.935409+0000 osd.1 (osd.1) 325 : cluster 0 6.1 scrub starts 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:45.936949+0000 osd.1 (osd.1) 326 : cluster 0 6.1 scrub ok 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.242123+0000 mgr.y (mgr.14538) 590 : cluster 0 pgmap v948: 270 pgs: 270 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 119 B/s, 6 objects/s recovering 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.270975+0000 osd.2 (osd.2) 295 : cluster 0 3.5 scrub starts 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.273853+0000 osd.2 (osd.2) 296 : cluster 0 3.5 scrub ok 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.815025+0000 osd.6 (osd.6) 275 : cluster 0 40.4 scrub starts 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.817835+0000 osd.6 (osd.6) 276 : cluster 0 40.4 scrub ok 2024-04-02T04:58:47.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: cluster 2024-04-02T04:58:47.023490+0000 mon.a (mon.0) 3204 : cluster 0 osdmap e629: 8 total, 8 up, 6 in 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: audit 2024-04-02T04:58:47.026208+0000 mon.c (mon.2) 1097 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[21528]: audit 2024-04-02T04:58:47.027405+0000 mon.a (mon.0) 3205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:45.276384+0000 osd.2 (osd.2) 293 : cluster 0 2.c scrub starts 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:45.278344+0000 osd.2 (osd.2) 294 : cluster 0 2.c scrub ok 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:45.935409+0000 osd.1 (osd.1) 325 : cluster 0 6.1 scrub starts 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:45.936949+0000 osd.1 (osd.1) 326 : cluster 0 6.1 scrub ok 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.242123+0000 mgr.y (mgr.14538) 590 : cluster 0 pgmap v948: 270 pgs: 270 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 119 B/s, 6 objects/s recovering 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.270975+0000 osd.2 (osd.2) 295 : cluster 0 3.5 scrub starts 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.273853+0000 osd.2 (osd.2) 296 : cluster 0 3.5 scrub ok 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.815025+0000 osd.6 (osd.6) 275 : cluster 0 40.4 scrub starts 2024-04-02T04:58:47.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.817835+0000 osd.6 (osd.6) 276 : cluster 0 40.4 scrub ok 2024-04-02T04:58:47.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: cluster 2024-04-02T04:58:47.023490+0000 mon.a (mon.0) 3204 : cluster 0 osdmap e629: 8 total, 8 up, 6 in 2024-04-02T04:58:47.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: audit 2024-04-02T04:58:47.026208+0000 mon.c (mon.2) 1097 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:47.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:47 smithi069 bash[28904]: audit 2024-04-02T04:58:47.027405+0000 mon.a (mon.0) 3205 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:47.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:47.827 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:47.930 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:47.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:47.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:48.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:48.191 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:48.217 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:48.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:48.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:48.345 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:48.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.471 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:48.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- 172.21.15.69:0/893224230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406b4b0 msgr2=0x7f5384106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- 172.21.15.69:0/893224230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406b4b0 0x7f5384106fa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5374009990 tx=0x7f5374009ca0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- 172.21.15.69:0/893224230 shutdown_connections 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- 172.21.15.69:0/893224230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5384107680 0x7f538410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- 172.21.15.69:0/893224230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406ab00 0x7f538406aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- 172.21.15.69:0/893224230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406b4b0 0x7f5384106fa0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- 172.21.15.69:0/893224230 >> 172.21.15.69:0/893224230 conn(0x7f5384078240 msgr2=0x7f5384078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- 172.21.15.69:0/893224230 shutdown_connections 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- 172.21.15.69:0/893224230 wait complete. 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 librados: starting msgr at 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 librados: starting objecter 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 Processor -- start 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- start start 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 librados: setting wanted keys 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 librados: calling monclient init 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406ab00 0x7f53841011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 0x7f5384101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5384107680 0x7f5384101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5384111ac0 con 0x7f538406b4b0 2024-04-02T04:58:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5384111940 con 0x7f5384107680 2024-04-02T04:58:48.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5389383700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5384111c40 con 0x7f538406ab00 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f53827fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 0x7f5384101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f53827fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 0x7f5384101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48394/0 (socket says 172.21.15.69:48394) 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f53827fc700 1 -- 172.21.15.69:0/1595313771 learned_addr learned my addr 172.21.15.69:0/1595313771 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.548+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406ab00 0x7f53841011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5384107680 0x7f5384101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53827fc700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406ab00 msgr2=0x7f53841011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406ab00 0x7f53841011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53827fc700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5384107680 msgr2=0x7f5384101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5384107680 0x7f5384101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53827fc700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53841a6550 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f538406ab00 0x7f53841011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5384107680 0x7f5384101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 0x7f5384101730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5374000c00 tx=0x7f53740047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 10 client.?.objecter ms_handle_connect 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374027070 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f5374022470 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53841a67e0 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 10 client.16707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53841a6c30 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 1 librados: init done 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374017650 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5364005130 con 0x7f538406b4b0 2024-04-02T04:58:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f537402a070 con 0x7f538406b4b0 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f537006f5e0 0x7f5370071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(630..630 src has 74..630) ==== 10169+0+0 (secure 0 0 0) 0x7f5374090070 con 0x7f538406b4b0 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 osd_map(630..630 src has 74..630) 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 3 client.16707.objecter handle_osd_map got epochs [630,630] > 0 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 3 client.16707.objecter handle_osd_map decoding full epoch 630 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f537006f5e0 0x7f5370071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 20 client.16707.objecter dump_active .. 0 homeless 2024-04-02T04:58:48.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v630) ==== 54+0+16 (secure 0 0 0) 0x7f5374031070 con 0x7f538406b4b0 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f537006f5e0 0x7f5370071aa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f536c011490 tx=0x7f536c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f537006f5e0 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 10 client.16707.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 1 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5364004760 0x7f5364004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 1 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 0 1 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 4 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 10 client.16707.objecter _send_command 1 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5364008ba0 con 0x7f5364004760 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.552+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5364004760 0x7f5364004b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5364004760 0x7f5364004b80 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f5364004760 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5364008ba0 con 0x7f5364004760 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 1: 0 ) 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f537bfff700 10 client.16707.objecter _finish_command 1 = system:0 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 4 2024-04-02T04:58:48.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.556+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 0 1 2024-04-02T04:58:48.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:48.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 10 client.16707.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 2 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 2 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364005410 osd=0 3 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 0 2 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 4 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 10 client.16707.objecter _send_command 2 2024-04-02T04:58:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.572+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5364004210 con 0x7f5364004760 2024-04-02T04:58:48.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5364004210 con 0x7f5364004760 2024-04-02T04:58:48.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 10 client.16707.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 4 2024-04-02T04:58:48.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 0 2 2024-04-02T04:58:48.578 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 10 client.16707.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f5364002780 0x7f5364002bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 1 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 4 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 10 client.16707.objecter _send_command 3 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5364003100 con 0x7f5364002780 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f5364002780 0x7f5364002bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f5364002780 2024-04-02T04:58:48.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f5364002780 0x7f5364002bc0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5364003100 con 0x7f5364002780 2024-04-02T04:58:48.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 3: 0 ) 2024-04-02T04:58:48.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 10 client.16707.objecter _finish_command 3 = system:0 2024-04-02T04:58:48.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 4 2024-04-02T04:58:48.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.576+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 1 3 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 10 client.16707.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 4 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 4 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f53640081f0 osd=1 3 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 1 4 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 4 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 10 client.16707.objecter _send_command 4 2024-04-02T04:58:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5364003b10 con 0x7f5364002780 2024-04-02T04:58:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5364003b10 con 0x7f5364002780 2024-04-02T04:58:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 10 client.16707.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 4 2024-04-02T04:58:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 1 4 2024-04-02T04:58:48.599 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 10 client.16707.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 5 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f536400b270 0x7f536400d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 5 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 2 5 2024-04-02T04:58:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 4 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 10 client.16707.objecter _send_command 5 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f53640105e0 con 0x7f536400b270 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f536400b270 0x7f536400d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f536400b270 0x7f536400d660 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f536400b270 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53640105e0 con 0x7f536400b270 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 5: 0 ) 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 10 client.16707.objecter _finish_command 5 = system:0 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 4 2024-04-02T04:58:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.596+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 2 5 2024-04-02T04:58:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 10 client.16707.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 6 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 6 2024-04-02T04:58:48.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364007ec0 osd=2 3 2024-04-02T04:58:48.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 2 6 2024-04-02T04:58:48.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 4 2024-04-02T04:58:48.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 10 client.16707.objecter _send_command 6 2024-04-02T04:58:48.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53640035d0 con 0x7f536400b270 2024-04-02T04:58:48.619 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:58:48.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53640035d0 con 0x7f536400b270 2024-04-02T04:58:48.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f537bfff700 10 client.16707.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 4 2024-04-02T04:58:48.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 2 6 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 10 client.16707.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 7 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f536400ed20 0x7f536400f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 7 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 3 7 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 4 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 10 client.16707.objecter _send_command 7 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f536400f710 con 0x7f536400ed20 2024-04-02T04:58:48.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f536400ed20 0x7f536400f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f536400ed20 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.616+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f536400ed20 0x7f536400f1d0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.620+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f536400f710 con 0x7f536400ed20 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.620+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 7: 0 ) 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.620+0000 7f537bfff700 10 client.16707.objecter _finish_command 7 = system:0 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.620+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 4 2024-04-02T04:58:48.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.620+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 3 7 2024-04-02T04:58:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 10 client.16707.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 8 2024-04-02T04:58:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 8 2024-04-02T04:58:48.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364001b80 osd=3 3 2024-04-02T04:58:48.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 3 8 2024-04-02T04:58:48.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 4 2024-04-02T04:58:48.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 10 client.16707.objecter _send_command 8 2024-04-02T04:58:48.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.636+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f536400f710 con 0x7f536400ed20 2024-04-02T04:58:48.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f536400f710 con 0x7f536400ed20 2024-04-02T04:58:48.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 10 client.16707.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 4 2024-04-02T04:58:48.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 3 8 2024-04-02T04:58:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:58:48.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 10 client.16707.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 9 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f53640127a0 0x7f5364014b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 9 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 4 9 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 4 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 10 client.16707.objecter _send_command 9 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5364010170 con 0x7f53640127a0 2024-04-02T04:58:48.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f53640127a0 0x7f5364014b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f53640127a0 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f53640127a0 0x7f5364014b90 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5364010170 con 0x7f53640127a0 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 9: 0 ) 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 10 client.16707.objecter _finish_command 9 = system:0 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 4 2024-04-02T04:58:48.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.640+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 4 9 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.930539+0000 osd.1 (osd.1) 327 : cluster 0 2.a scrub starts 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: cluster 2024-04-02T04:58:46.934529+0000 osd.1 (osd.1) 328 : cluster 0 2.a scrub ok 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: cluster 2024-04-02T04:58:47.475550+0000 osd.7 (osd.7) 119 : cluster 0 2.8 scrub starts 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: cluster 2024-04-02T04:58:47.477849+0000 osd.7 (osd.7) 120 : cluster 0 2.8 scrub ok 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: audit 2024-04-02T04:58:48.017956+0000 mon.a (mon.0) 3206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.031215+0000 mon.a (mon.0) 3207 : cluster 0 osdmap e630: 8 total, 8 up, 6 in 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: audit 2024-04-02T04:58:48.043409+0000 mon.c (mon.2) 1098 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21528]: audit 2024-04-02T04:58:48.044724+0000 mon.a (mon.0) 3208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:58:48.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.930539+0000 osd.1 (osd.1) 327 : cluster 0 2.a scrub starts 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: cluster 2024-04-02T04:58:46.934529+0000 osd.1 (osd.1) 328 : cluster 0 2.a scrub ok 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: cluster 2024-04-02T04:58:47.475550+0000 osd.7 (osd.7) 119 : cluster 0 2.8 scrub starts 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: cluster 2024-04-02T04:58:47.477849+0000 osd.7 (osd.7) 120 : cluster 0 2.8 scrub ok 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: audit 2024-04-02T04:58:48.017956+0000 mon.a (mon.0) 3206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.031215+0000 mon.a (mon.0) 3207 : cluster 0 osdmap e630: 8 total, 8 up, 6 in 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: audit 2024-04-02T04:58:48.043409+0000 mon.c (mon.2) 1098 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:58:48.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[28904]: audit 2024-04-02T04:58:48.044724+0000 mon.a (mon.0) 3208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:58:48.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 10 client.16707.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 10 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 10 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f536400f710 osd=4 3 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 4 10 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 4 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 10 client.16707.objecter _send_command 10 2024-04-02T04:58:48.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.660+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5364010170 con 0x7f53640127a0 2024-04-02T04:58:48.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5364010170 con 0x7f53640127a0 2024-04-02T04:58:48.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f537bfff700 10 client.16707.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 4 2024-04-02T04:58:48.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 4 10 2024-04-02T04:58:48.668 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:58:48.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 10 client.16707.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 11 2024-04-02T04:58:48.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f5364015640 0x7f5364017a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 11 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 5 11 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 4 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 10 client.16707.objecter _send_command 11 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f536401aa80 con 0x7f5364015640 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.664+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f5364015640 0x7f5364017a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f5364015640 2024-04-02T04:58:48.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f53827fc700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f5364015640 0x7f5364017a30 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f536401aa80 con 0x7f5364015640 2024-04-02T04:58:48.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 11: 0 ) 2024-04-02T04:58:48.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f537bfff700 10 client.16707.objecter _finish_command 11 = system:0 2024-04-02T04:58:48.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 4 2024-04-02T04:58:48.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.668+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 5 11 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 10 client.16707.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 12 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 12 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364010170 osd=5 3 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 5 12 2024-04-02T04:58:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 4 2024-04-02T04:58:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 10 client.16707.objecter _send_command 12 2024-04-02T04:58:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.688+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5364019850 con 0x7f5364015640 2024-04-02T04:58:48.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5364019850 con 0x7f5364015640 2024-04-02T04:58:48.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f537bfff700 10 client.16707.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 4 2024-04-02T04:58:48.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 5 12 2024-04-02T04:58:48.694 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 10 client.16707.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 13 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f5364019f70 0x7f536401a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 13 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 6 13 2024-04-02T04:58:48.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 4 2024-04-02T04:58:48.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 10 client.16707.objecter _send_command 13 2024-04-02T04:58:48.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f536401de20 con 0x7f5364019f70 2024-04-02T04:58:48.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f5364019f70 0x7f536401a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f5364019f70 2024-04-02T04:58:48.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.692+0000 7f53837fe700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f5364019f70 0x7f536401a3d0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.696+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f536401de20 con 0x7f5364019f70 2024-04-02T04:58:48.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.696+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 13: 0 ) 2024-04-02T04:58:48.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.696+0000 7f537bfff700 10 client.16707.objecter _finish_command 13 = system:0 2024-04-02T04:58:48.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.696+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 4 2024-04-02T04:58:48.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.696+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 6 13 2024-04-02T04:58:48.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 10 client.16707.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 14 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 14 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f5364019850 osd=6 3 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 6 14 2024-04-02T04:58:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 4 2024-04-02T04:58:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 10 client.16707.objecter _send_command 14 2024-04-02T04:58:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.712+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f536401cca0 con 0x7f5364019f70 2024-04-02T04:58:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f536401cca0 con 0x7f5364019f70 2024-04-02T04:58:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 10 client.16707.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 4 2024-04-02T04:58:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 6 14 2024-04-02T04:58:48.721 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 10 client.16707.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 15 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f536401d3c0 0x7f536401d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 15 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 7 15 2024-04-02T04:58:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 4 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 10 client.16707.objecter _send_command 15 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5364021190 con 0x7f536401d3c0 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f536401d3c0 0x7f536401d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 10 client.16707.objecter ms_handle_connect 0x7f536401d3c0 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f5382ffd700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f536401d3c0 0x7f536401d800 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5364021190 con 0x7f536401d3c0 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 15: 0 ) 2024-04-02T04:58:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 10 client.16707.objecter _finish_command 15 = system:0 2024-04-02T04:58:48.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 4 2024-04-02T04:58:48.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.720+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 7 15 2024-04-02T04:58:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.930539+0000 osd.1 (osd.1) 327 : cluster 0 2.a scrub starts 2024-04-02T04:58:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: cluster 2024-04-02T04:58:46.934529+0000 osd.1 (osd.1) 328 : cluster 0 2.a scrub ok 2024-04-02T04:58:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: cluster 2024-04-02T04:58:47.475550+0000 osd.7 (osd.7) 119 : cluster 0 2.8 scrub starts 2024-04-02T04:58:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: cluster 2024-04-02T04:58:47.477849+0000 osd.7 (osd.7) 120 : cluster 0 2.8 scrub ok 2024-04-02T04:58:48.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: audit 2024-04-02T04:58:48.017956+0000 mon.a (mon.0) 3206 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:58:48.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.031215+0000 mon.a (mon.0) 3207 : cluster 0 osdmap e630: 8 total, 8 up, 6 in 2024-04-02T04:58:48.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: audit 2024-04-02T04:58:48.043409+0000 mon.c (mon.2) 1098 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:58:48.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:48 smithi098 bash[23790]: audit 2024-04-02T04:58:48.044724+0000 mon.a (mon.0) 3208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:58:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:48.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 10 client.16707.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:58:48.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign -1 16 2024-04-02T04:58:48.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 20 client.16707.objecter _get_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 15 client.16707.objecter _session_command_op_remove -1 16 2024-04-02T04:58:48.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 20 client.16707.objecter get_session s=0x7f536401cca0 osd=7 3 2024-04-02T04:58:48.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 15 client.16707.objecter _session_command_op_assign 7 16 2024-04-02T04:58:48.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 4 2024-04-02T04:58:48.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 10 client.16707.objecter _send_command 16 2024-04-02T04:58:48.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5364018080 con 0x7f536401d3c0 2024-04-02T04:58:48.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f537bfff700 1 -- 172.21.15.69:0/1595313771 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5364018080 con 0x7f536401d3c0 2024-04-02T04:58:48.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f537bfff700 10 client.16707.objecter ms_dispatch 0x7f5384001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:58:48.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f537bfff700 10 client.16707.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:58:48.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f537bfff700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 4 2024-04-02T04:58:48.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.740+0000 7f537bfff700 15 client.16707.objecter _session_command_op_remove 7 16 2024-04-02T04:58:48.745 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:58:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 librados: watch_flush enter 2024-04-02T04:58:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 librados: watch_flush exit 2024-04-02T04:58:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.0 2024-04-02T04:58:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5364004760 msgr2=0x7f5364004b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f5364004760 0x7f5364004b80 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364005410 osd=0 1 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.1 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f5364002780 msgr2=0x7f5364002bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f5364002780 0x7f5364002bc0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f53640081f0 osd=1 1 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.2 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f536400b270 msgr2=0x7f536400d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f536400b270 0x7f536400d660 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364007ec0 osd=2 1 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.3 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f536400ed20 msgr2=0x7f536400f1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f536400ed20 0x7f536400f1d0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364001b80 osd=3 1 2024-04-02T04:58:48.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.4 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f53640127a0 msgr2=0x7f5364014b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f53640127a0 0x7f5364014b90 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536400f710 osd=4 1 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.5 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f5364015640 msgr2=0x7f5364017a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f5364015640 0x7f5364017a30 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f53837fe700 1 -- 172.21.15.69:0/1595313771 reap_dead start 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f53837fe700 1 -- 172.21.15.69:0/1595313771 reap_dead start 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f53837fe700 1 -- 172.21.15.69:0/1595313771 reap_dead start 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f53837fe700 1 -- 172.21.15.69:0/1595313771 reap_dead start 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364010170 osd=5 1 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.6 2024-04-02T04:58:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f5364019f70 msgr2=0x7f536401a3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f5364019f70 0x7f536401a3d0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f5364019850 osd=6 1 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter close_session for osd.7 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f536401d3c0 msgr2=0x7f536401d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f536401d3c0 0x7f536401d800 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter put_session s=0x7f536401cca0 osd=7 1 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 20 client.16707.objecter shutdown clearing up homeless session... 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 10 client.16707.objecter successfully canceled tick 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f537006f5e0 msgr2=0x7f5370071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f537006f5e0 0x7f5370071aa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f536c011490 tx=0x7f536c009260 comp rx=0 tx=0).stop 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 msgr2=0x7f5384101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 0x7f5384101730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5374000c00 tx=0x7f53740047b0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 shutdown_connections 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f536401d3c0 0x7f536401d800 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f5364019f70 0x7f536401a3d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f538406b4b0 0x7f5384101730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 --2- 172.21.15.69:0/1595313771 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f537006f5e0 0x7f5370071aa0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 >> 172.21.15.69:0/1595313771 conn(0x7f5384078240 msgr2=0x7f53840fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 shutdown_connections 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 -- 172.21.15.69:0/1595313771 wait complete. 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:48.744+0000 7f5389383700 1 librados: shutdown 2024-04-02T04:58:48.750 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:58:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:58:48.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:48.852 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:48.915 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:58:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:58:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:58:48.955 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:48.977 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:48.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:49.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:49.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:49.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.237 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:49.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:49.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:47.886219+0000 osd.1 (osd.1) 329 : cluster 0 2.4 scrub starts 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:47.888326+0000 osd.1 (osd.1) 330 : cluster 0 2.4 scrub ok 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.113377+0000 osd.4 (osd.4) 332 : cluster 0 3.14 scrub starts 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.115763+0000 osd.4 (osd.4) 333 : cluster 0 3.14 scrub ok 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.243021+0000 mgr.y (mgr.14538) 591 : cluster 0 pgmap v951: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 6 unknown, 272 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1 op/s; 122 B/s, 6 objects/s recovering 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.289419+0000 osd.2 (osd.2) 297 : cluster 0 4.19 scrub starts 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.290602+0000 osd.2 (osd.2) 298 : cluster 0 4.19 scrub ok 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: audit 2024-04-02T04:58:48.557798+0000 mon.a (mon.0) 3209 : audit 0 from='client.? 172.21.15.69:0/1595313771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: audit 2024-04-02T04:58:48.655549+0000 mon.b (mon.1) 284 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:49.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:49.626 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: audit 2024-04-02T04:58:49.026595+0000 mon.a (mon.0) 3210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:58:49.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: cluster 2024-04-02T04:58:49.032800+0000 mon.a (mon.0) 3211 : cluster 0 osdmap e631: 8 total, 8 up, 6 in 2024-04-02T04:58:49.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: audit 2024-04-02T04:58:49.034262+0000 mon.c (mon.2) 1099 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]: dispatch 2024-04-02T04:58:49.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:49 smithi098 bash[23790]: audit 2024-04-02T04:58:49.034808+0000 mon.a (mon.0) 3212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]: dispatch 2024-04-02T04:58:49.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:49.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:47.886219+0000 osd.1 (osd.1) 329 : cluster 0 2.4 scrub starts 2024-04-02T04:58:49.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:47.888326+0000 osd.1 (osd.1) 330 : cluster 0 2.4 scrub ok 2024-04-02T04:58:49.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.113377+0000 osd.4 (osd.4) 332 : cluster 0 3.14 scrub starts 2024-04-02T04:58:49.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.115763+0000 osd.4 (osd.4) 333 : cluster 0 3.14 scrub ok 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.243021+0000 mgr.y (mgr.14538) 591 : cluster 0 pgmap v951: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 6 unknown, 272 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1 op/s; 122 B/s, 6 objects/s recovering 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.289419+0000 osd.2 (osd.2) 297 : cluster 0 4.19 scrub starts 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.290602+0000 osd.2 (osd.2) 298 : cluster 0 4.19 scrub ok 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: audit 2024-04-02T04:58:48.557798+0000 mon.a (mon.0) 3209 : audit 0 from='client.? 172.21.15.69:0/1595313771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: audit 2024-04-02T04:58:48.655549+0000 mon.b (mon.1) 284 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: audit 2024-04-02T04:58:49.026595+0000 mon.a (mon.0) 3210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: cluster 2024-04-02T04:58:49.032800+0000 mon.a (mon.0) 3211 : cluster 0 osdmap e631: 8 total, 8 up, 6 in 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: audit 2024-04-02T04:58:49.034262+0000 mon.c (mon.2) 1099 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]: dispatch 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[21528]: audit 2024-04-02T04:58:49.034808+0000 mon.a (mon.0) 3212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]: dispatch 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:47.886219+0000 osd.1 (osd.1) 329 : cluster 0 2.4 scrub starts 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:47.888326+0000 osd.1 (osd.1) 330 : cluster 0 2.4 scrub ok 2024-04-02T04:58:49.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.113377+0000 osd.4 (osd.4) 332 : cluster 0 3.14 scrub starts 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.115763+0000 osd.4 (osd.4) 333 : cluster 0 3.14 scrub ok 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.243021+0000 mgr.y (mgr.14538) 591 : cluster 0 pgmap v951: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 6 unknown, 272 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1 op/s; 122 B/s, 6 objects/s recovering 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.289419+0000 osd.2 (osd.2) 297 : cluster 0 4.19 scrub starts 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.290602+0000 osd.2 (osd.2) 298 : cluster 0 4.19 scrub ok 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: audit 2024-04-02T04:58:48.557798+0000 mon.a (mon.0) 3209 : audit 0 from='client.? 172.21.15.69:0/1595313771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: audit 2024-04-02T04:58:48.655549+0000 mon.b (mon.1) 284 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: audit 2024-04-02T04:58:49.026595+0000 mon.a (mon.0) 3210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: cluster 2024-04-02T04:58:49.032800+0000 mon.a (mon.0) 3211 : cluster 0 osdmap e631: 8 total, 8 up, 6 in 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: audit 2024-04-02T04:58:49.034262+0000 mon.c (mon.2) 1099 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]: dispatch 2024-04-02T04:58:49.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:49 smithi069 bash[28904]: audit 2024-04-02T04:58:49.034808+0000 mon.a (mon.0) 3212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]: dispatch 2024-04-02T04:58:49.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.886 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:49.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:49.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:50.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:50.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:50.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:50.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.874263+0000 osd.1 (osd.1) 331 : cluster 0 3.8 scrub starts 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: cluster 2024-04-02T04:58:48.877779+0000 osd.1 (osd.1) 332 : cluster 0 3.8 scrub ok 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: cluster 2024-04-02T04:58:49.180792+0000 osd.3 (osd.3) 319 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: cluster 2024-04-02T04:58:49.181152+0000 osd.3 (osd.3) 320 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.026870+0000 mon.a (mon.0) 3213 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: audit 2024-04-02T04:58:50.028674+0000 mon.a (mon.0) 3214 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]': finished 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.034392+0000 mon.a (mon.0) 3215 : cluster 0 osdmap e632: 8 total, 8 up, 6 in 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: audit 2024-04-02T04:58:50.036343+0000 mon.c (mon.2) 1100 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[21528]: audit 2024-04-02T04:58:50.036724+0000 mon.a (mon.0) 3216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:50.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.874263+0000 osd.1 (osd.1) 331 : cluster 0 3.8 scrub starts 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: cluster 2024-04-02T04:58:48.877779+0000 osd.1 (osd.1) 332 : cluster 0 3.8 scrub ok 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: cluster 2024-04-02T04:58:49.180792+0000 osd.3 (osd.3) 319 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: cluster 2024-04-02T04:58:49.181152+0000 osd.3 (osd.3) 320 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.026870+0000 mon.a (mon.0) 3213 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: audit 2024-04-02T04:58:50.028674+0000 mon.a (mon.0) 3214 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]': finished 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.034392+0000 mon.a (mon.0) 3215 : cluster 0 osdmap e632: 8 total, 8 up, 6 in 2024-04-02T04:58:50.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: audit 2024-04-02T04:58:50.036343+0000 mon.c (mon.2) 1100 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:50.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:50 smithi069 bash[28904]: audit 2024-04-02T04:58:50.036724+0000 mon.a (mon.0) 3216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:50.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.874263+0000 osd.1 (osd.1) 331 : cluster 0 3.8 scrub starts 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: cluster 2024-04-02T04:58:48.877779+0000 osd.1 (osd.1) 332 : cluster 0 3.8 scrub ok 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: cluster 2024-04-02T04:58:49.180792+0000 osd.3 (osd.3) 319 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: cluster 2024-04-02T04:58:49.181152+0000 osd.3 (osd.3) 320 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.026870+0000 mon.a (mon.0) 3213 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: audit 2024-04-02T04:58:50.028674+0000 mon.a (mon.0) 3214 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-98", "mode": "writeback"}]': finished 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.034392+0000 mon.a (mon.0) 3215 : cluster 0 osdmap e632: 8 total, 8 up, 6 in 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: audit 2024-04-02T04:58:50.036343+0000 mon.c (mon.2) 1100 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:50 smithi098 bash[23790]: audit 2024-04-02T04:58:50.036724+0000 mon.a (mon.0) 3216 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:50.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:50.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:50.947 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:50.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:51.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:51.073 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:51.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:51.197 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:51.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:51.325 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:51.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:51.455 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:51.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:51.611 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:58:51.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.028710+0000 osd.4 (osd.4) 334 : cluster 0 2.f scrub starts 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.033426+0000 osd.4 (osd.4) 335 : cluster 0 2.f scrub ok 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.243927+0000 mgr.y (mgr.14538) 592 : cluster 0 pgmap v954: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 278 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.376197+0000 osd.2 (osd.2) 299 : cluster 0 2.3 scrub starts 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.380066+0000 osd.2 (osd.2) 300 : cluster 0 2.3 scrub ok 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.494610+0000 osd.7 (osd.7) 121 : cluster 0 4.15 scrub starts 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:50.497322+0000 osd.7 (osd.7) 122 : cluster 0 4.15 scrub ok 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: audit 2024-04-02T04:58:51.031777+0000 mon.a (mon.0) 3217 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]': finished 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.035986+0000 mon.a (mon.0) 3218 : cluster 0 osdmap e633: 8 total, 8 up, 6 in 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: audit 2024-04-02T04:58:51.037486+0000 mon.c (mon.2) 1101 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[21528]: audit 2024-04-02T04:58:51.046966+0000 mon.a (mon.0) 3219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:58:51.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.028710+0000 osd.4 (osd.4) 334 : cluster 0 2.f scrub starts 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.033426+0000 osd.4 (osd.4) 335 : cluster 0 2.f scrub ok 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.243927+0000 mgr.y (mgr.14538) 592 : cluster 0 pgmap v954: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 278 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.376197+0000 osd.2 (osd.2) 299 : cluster 0 2.3 scrub starts 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.380066+0000 osd.2 (osd.2) 300 : cluster 0 2.3 scrub ok 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.494610+0000 osd.7 (osd.7) 121 : cluster 0 4.15 scrub starts 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:50.497322+0000 osd.7 (osd.7) 122 : cluster 0 4.15 scrub ok 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: audit 2024-04-02T04:58:51.031777+0000 mon.a (mon.0) 3217 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]': finished 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.035986+0000 mon.a (mon.0) 3218 : cluster 0 osdmap e633: 8 total, 8 up, 6 in 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: audit 2024-04-02T04:58:51.037486+0000 mon.c (mon.2) 1101 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:58:51.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:51 smithi069 bash[28904]: audit 2024-04-02T04:58:51.046966+0000 mon.a (mon.0) 3219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:58:51.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:51.718 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:51.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.028710+0000 osd.4 (osd.4) 334 : cluster 0 2.f scrub starts 2024-04-02T04:58:51.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.033426+0000 osd.4 (osd.4) 335 : cluster 0 2.f scrub ok 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.243927+0000 mgr.y (mgr.14538) 592 : cluster 0 pgmap v954: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 278 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.376197+0000 osd.2 (osd.2) 299 : cluster 0 2.3 scrub starts 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.380066+0000 osd.2 (osd.2) 300 : cluster 0 2.3 scrub ok 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.494610+0000 osd.7 (osd.7) 121 : cluster 0 4.15 scrub starts 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:50.497322+0000 osd.7 (osd.7) 122 : cluster 0 4.15 scrub ok 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: audit 2024-04-02T04:58:51.031777+0000 mon.a (mon.0) 3217 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-98"}]': finished 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.035986+0000 mon.a (mon.0) 3218 : cluster 0 osdmap e633: 8 total, 8 up, 6 in 2024-04-02T04:58:51.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: audit 2024-04-02T04:58:51.037486+0000 mon.c (mon.2) 1101 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:58:51.728 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:51 smithi098 bash[23790]: audit 2024-04-02T04:58:51.046966+0000 mon.a (mon.0) 3219 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T04:58:51.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:51.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:51.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:51.979 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:51.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:52.082 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:52.109 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:52.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:52.239 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:52.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:52.371 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:52.474 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:52.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:52.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:52.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.006925+0000 osd.0 (osd.0) 321 : cluster 0 3.b scrub starts 2024-04-02T04:58:52.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.009678+0000 osd.0 (osd.0) 322 : cluster 0 3.b scrub ok 2024-04-02T04:58:52.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.042707+0000 mon.a (mon.0) 3220 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:58:52.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: audit 2024-04-02T04:58:52.044542+0000 mon.a (mon.0) 3221 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:58:52.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.048862+0000 mon.a (mon.0) 3222 : cluster 0 osdmap e634: 8 total, 8 up, 6 in 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: audit 2024-04-02T04:58:52.053518+0000 mon.c (mon.2) 1102 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[21528]: audit 2024-04-02T04:58:52.054540+0000 mon.a (mon.0) 3223 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.006925+0000 osd.0 (osd.0) 321 : cluster 0 3.b scrub starts 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.009678+0000 osd.0 (osd.0) 322 : cluster 0 3.b scrub ok 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.042707+0000 mon.a (mon.0) 3220 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: audit 2024-04-02T04:58:52.044542+0000 mon.a (mon.0) 3221 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.048862+0000 mon.a (mon.0) 3222 : cluster 0 osdmap e634: 8 total, 8 up, 6 in 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: audit 2024-04-02T04:58:52.053518+0000 mon.c (mon.2) 1102 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-02T04:58:52.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:52 smithi069 bash[28904]: audit 2024-04-02T04:58:52.054540+0000 mon.a (mon.0) 3223 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-02T04:58:52.630 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:52.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.006925+0000 osd.0 (osd.0) 321 : cluster 0 3.b scrub starts 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.009678+0000 osd.0 (osd.0) 322 : cluster 0 3.b scrub ok 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.042707+0000 mon.a (mon.0) 3220 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: audit 2024-04-02T04:58:52.044542+0000 mon.a (mon.0) 3221 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.048862+0000 mon.a (mon.0) 3222 : cluster 0 osdmap e634: 8 total, 8 up, 6 in 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: audit 2024-04-02T04:58:52.053518+0000 mon.c (mon.2) 1102 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-02T04:58:52.745 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:52 smithi098 bash[23790]: audit 2024-04-02T04:58:52.054540+0000 mon.a (mon.0) 3223 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-02T04:58:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:52.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:52.891 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:52.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:52.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:53.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:58:53.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:53.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:53.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:53.413 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:53.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:53.527 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.860931+0000 osd.1 (osd.1) 333 : cluster 0 10.13 scrub starts 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.862862+0000 osd.1 (osd.1) 334 : cluster 0 10.13 scrub ok 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.988024+0000 osd.0 (osd.0) 323 : cluster 0 6.0 scrub starts 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:51.989382+0000 osd.0 (osd.0) 324 : cluster 0 6.0 scrub ok 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.244784+0000 mgr.y (mgr.14538) 593 : cluster 0 pgmap v957: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 278 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.442302+0000 osd.2 (osd.2) 301 : cluster 0 40.b scrub starts 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.446636+0000 osd.2 (osd.2) 302 : cluster 0 40.b scrub ok 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.468453+0000 osd.7 (osd.7) 123 : cluster 0 6.1b scrub starts 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.472647+0000 osd.7 (osd.7) 124 : cluster 0 6.1b scrub ok 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: audit 2024-04-02T04:58:53.053860+0000 mon.a (mon.0) 3224 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]': finished 2024-04-02T04:58:53.528 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: cluster 2024-04-02T04:58:53.061782+0000 mon.a (mon.0) 3225 : cluster 0 osdmap e635: 8 total, 8 up, 6 in 2024-04-02T04:58:53.529 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: audit 2024-04-02T04:58:53.062535+0000 mon.c (mon.2) 1103 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:58:53.529 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:53 smithi098 bash[23790]: audit 2024-04-02T04:58:53.066125+0000 mon.a (mon.0) 3226 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:58:53.541 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:53.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:53.669 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:53.731 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.860931+0000 osd.1 (osd.1) 333 : cluster 0 10.13 scrub starts 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.862862+0000 osd.1 (osd.1) 334 : cluster 0 10.13 scrub ok 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.988024+0000 osd.0 (osd.0) 323 : cluster 0 6.0 scrub starts 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:51.989382+0000 osd.0 (osd.0) 324 : cluster 0 6.0 scrub ok 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.244784+0000 mgr.y (mgr.14538) 593 : cluster 0 pgmap v957: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 278 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.442302+0000 osd.2 (osd.2) 301 : cluster 0 40.b scrub starts 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.446636+0000 osd.2 (osd.2) 302 : cluster 0 40.b scrub ok 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.468453+0000 osd.7 (osd.7) 123 : cluster 0 6.1b scrub starts 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.472647+0000 osd.7 (osd.7) 124 : cluster 0 6.1b scrub ok 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: audit 2024-04-02T04:58:53.053860+0000 mon.a (mon.0) 3224 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]': finished 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: cluster 2024-04-02T04:58:53.061782+0000 mon.a (mon.0) 3225 : cluster 0 osdmap e635: 8 total, 8 up, 6 in 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: audit 2024-04-02T04:58:53.062535+0000 mon.c (mon.2) 1103 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:58:53.732 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[21528]: audit 2024-04-02T04:58:53.066125+0000 mon.a (mon.0) 3226 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.860931+0000 osd.1 (osd.1) 333 : cluster 0 10.13 scrub starts 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.862862+0000 osd.1 (osd.1) 334 : cluster 0 10.13 scrub ok 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.988024+0000 osd.0 (osd.0) 323 : cluster 0 6.0 scrub starts 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:51.989382+0000 osd.0 (osd.0) 324 : cluster 0 6.0 scrub ok 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.244784+0000 mgr.y (mgr.14538) 593 : cluster 0 pgmap v957: 302 pgs: 1 active, 8 creating+activating, 15 creating+peering, 278 active+clean; 8.4 MiB data, 487 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.442302+0000 osd.2 (osd.2) 301 : cluster 0 40.b scrub starts 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.446636+0000 osd.2 (osd.2) 302 : cluster 0 40.b scrub ok 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.468453+0000 osd.7 (osd.7) 123 : cluster 0 6.1b scrub starts 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.472647+0000 osd.7 (osd.7) 124 : cluster 0 6.1b scrub ok 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: audit 2024-04-02T04:58:53.053860+0000 mon.a (mon.0) 3224 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_count","val": "1"}]': finished 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: cluster 2024-04-02T04:58:53.061782+0000 mon.a (mon.0) 3225 : cluster 0 osdmap e635: 8 total, 8 up, 6 in 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: audit 2024-04-02T04:58:53.062535+0000 mon.c (mon.2) 1103 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:58:53.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:53 smithi069 bash[28904]: audit 2024-04-02T04:58:53.066125+0000 mon.a (mon.0) 3226 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:58:53.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:53.801 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:53.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/2490121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 msgr2=0x7fba4c102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- 172.21.15.69:0/2490121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c102710 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fba480099c0 tx=0x7fba48009cd0 comp rx=0 tx=0).stop 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/2490121 shutdown_connections 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- 172.21.15.69:0/2490121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c104f10 0x7fba4c1053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- 172.21.15.69:0/2490121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c1073a0 0x7fba4c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- 172.21.15.69:0/2490121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c102710 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/2490121 >> 172.21.15.69:0/2490121 conn(0x7fba4c0771e0 msgr2=0x7fba4c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/2490121 shutdown_connections 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/2490121 wait complete. 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 librados: starting msgr at 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 librados: starting objecter 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 Processor -- start 2024-04-02T04:58:53.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- start start 2024-04-02T04:58:53.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 librados: setting wanted keys 2024-04-02T04:58:53.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 librados: calling monclient init 2024-04-02T04:58:53.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c1a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:53.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c104f10 0x7fba4c1a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c1073a0 0x7fba4c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1182d0 con 0x7fba4c102290 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c118150 con 0x7fba4c1073a0 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fba4c118450 con 0x7fba4c104f10 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c1a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba5114f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c104f10 0x7fba4c1a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c1a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48410/0 (socket says 172.21.15.69:48410) 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 -- 172.21.15.69:0/4085551155 learned_addr learned my addr 172.21.15.69:0/4085551155 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba52151700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c1073a0 0x7fba4c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 -- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c104f10 msgr2=0x7fba4c1a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c104f10 0x7fba4c1a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 -- 172.21.15.69:0/4085551155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c1073a0 msgr2=0x7fba4c19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c1073a0 0x7fba4c19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:53.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 -- 172.21.15.69:0/4085551155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c19ef90 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba5114f700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c104f10 0x7fba4c1a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba51950700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c1a45e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d950 tx=0x7fba3c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba42ffd700 10 client.?.objecter ms_handle_connect 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba42ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c016070 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c19f280 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fba3c009dc0 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00f3c0 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 10 client.16713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c1ab560 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 librados: init done 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.876+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba30005120 con 0x7fba4c102290 2024-04-02T04:58:53.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fba3c019070 con 0x7fba4c102290 2024-04-02T04:58:53.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3806f5e0 0x7fba38071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:53.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(635..635 src has 74..635) ==== 10203+0+0 (secure 0 0 0) 0x7fba3c08e930 con 0x7fba4c102290 2024-04-02T04:58:53.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 10 client.16713.objecter ms_dispatch 0x7fba4c001550 osd_map(635..635 src has 74..635) 2024-04-02T04:58:53.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 3 client.16713.objecter handle_osd_map got epochs [635,635] > 0 2024-04-02T04:58:53.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 3 client.16713.objecter handle_osd_map decoding full epoch 635 2024-04-02T04:58:53.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba5114f700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3806f5e0 0x7fba38071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:53.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 20 client.16713.objecter dump_active .. 0 homeless 2024-04-02T04:58:53.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba42ffd700 10 client.16713.objecter ms_handle_connect 0x7fba3806f5e0 2024-04-02T04:58:53.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.880+0000 7fba5114f700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3806f5e0 0x7fba38071aa0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba48006010 tx=0x7fba48005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:53.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:53.884+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba3c05b460 con 0x7fba4c102290 2024-04-02T04:58:53.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:53.923 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:53.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:54.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:54.044 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:54.128+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fba30005410 con 0x7fba4c102290 2024-04-02T04:58:54.146 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:58:54.147 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:58:54.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:54.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:54.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:54.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:54.428 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:54.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:54.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:54.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.966118+0000 osd.0 (osd.0) 325 : cluster 0 10.1e scrub starts 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: cluster 2024-04-02T04:58:52.967741+0000 osd.0 (osd.0) 326 : cluster 0 10.1e scrub ok 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: cluster 2024-04-02T04:58:53.478477+0000 osd.2 (osd.2) 303 : cluster 0 43.18 scrub starts 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: cluster 2024-04-02T04:58:53.481834+0000 osd.2 (osd.2) 304 : cluster 0 43.18 scrub ok 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: audit 2024-04-02T04:58:53.511294+0000 mgr.y (mgr.14538) 594 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: audit 2024-04-02T04:58:54.055482+0000 mon.a (mon.0) 3227 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: cluster 2024-04-02T04:58:54.059134+0000 mon.a (mon.0) 3228 : cluster 0 osdmap e636: 8 total, 8 up, 6 in 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: audit 2024-04-02T04:58:54.059881+0000 mon.c (mon.2) 1104 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: audit 2024-04-02T04:58:54.061117+0000 mon.a (mon.0) 3229 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-02T04:58:54.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[21528]: audit 2024-04-02T04:58:54.133267+0000 mon.a (mon.0) 3230 : audit 1 from='client.? 172.21.15.69:0/4085551155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.966118+0000 osd.0 (osd.0) 325 : cluster 0 10.1e scrub starts 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: cluster 2024-04-02T04:58:52.967741+0000 osd.0 (osd.0) 326 : cluster 0 10.1e scrub ok 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: cluster 2024-04-02T04:58:53.478477+0000 osd.2 (osd.2) 303 : cluster 0 43.18 scrub starts 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: cluster 2024-04-02T04:58:53.481834+0000 osd.2 (osd.2) 304 : cluster 0 43.18 scrub ok 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: audit 2024-04-02T04:58:53.511294+0000 mgr.y (mgr.14538) 594 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: audit 2024-04-02T04:58:54.055482+0000 mon.a (mon.0) 3227 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: cluster 2024-04-02T04:58:54.059134+0000 mon.a (mon.0) 3228 : cluster 0 osdmap e636: 8 total, 8 up, 6 in 2024-04-02T04:58:54.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: audit 2024-04-02T04:58:54.059881+0000 mon.c (mon.2) 1104 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-02T04:58:54.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: audit 2024-04-02T04:58:54.061117+0000 mon.a (mon.0) 3229 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-02T04:58:54.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:54 smithi069 bash[28904]: audit 2024-04-02T04:58:54.133267+0000 mon.a (mon.0) 3230 : audit 1 from='client.? 172.21.15.69:0/4085551155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:58:54.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:54.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:54.801 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.966118+0000 osd.0 (osd.0) 325 : cluster 0 10.1e scrub starts 2024-04-02T04:58:54.801 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: cluster 2024-04-02T04:58:52.967741+0000 osd.0 (osd.0) 326 : cluster 0 10.1e scrub ok 2024-04-02T04:58:54.801 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: cluster 2024-04-02T04:58:53.478477+0000 osd.2 (osd.2) 303 : cluster 0 43.18 scrub starts 2024-04-02T04:58:54.801 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: cluster 2024-04-02T04:58:53.481834+0000 osd.2 (osd.2) 304 : cluster 0 43.18 scrub ok 2024-04-02T04:58:54.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: audit 2024-04-02T04:58:53.511294+0000 mgr.y (mgr.14538) 594 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:58:54.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: audit 2024-04-02T04:58:54.055482+0000 mon.a (mon.0) 3227 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:58:54.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: cluster 2024-04-02T04:58:54.059134+0000 mon.a (mon.0) 3228 : cluster 0 osdmap e636: 8 total, 8 up, 6 in 2024-04-02T04:58:54.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: audit 2024-04-02T04:58:54.059881+0000 mon.c (mon.2) 1104 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-02T04:58:54.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: audit 2024-04-02T04:58:54.061117+0000 mon.a (mon.0) 3229 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-02T04:58:54.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:54 smithi098 bash[23790]: audit 2024-04-02T04:58:54.133267+0000 mon.a (mon.0) 3230 : audit 1 from='client.? 172.21.15.69:0/4085551155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:58:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:54.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:54.947 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:54.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:55.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.004+0000 7fba42ffd700 1 -- 172.21.15.69:0/4085551155 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v637) ==== 101+0+0 (secure 0 0 0) 0x7fba3c05eeb0 con 0x7fba4c102290 2024-04-02T04:58:55.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.008+0000 7fba53bb4700 10 librados: watch_flush enter 2024-04-02T04:58:55.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.008+0000 7fba53bb4700 10 librados: watch_flush exit 2024-04-02T04:58:55.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.008+0000 7fba53bb4700 20 client.16713.objecter shutdown clearing up homeless session... 2024-04-02T04:58:55.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.008+0000 7fba53bb4700 10 client.16713.objecter successfully canceled tick 2024-04-02T04:58:55.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3806f5e0 msgr2=0x7fba38071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:55.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3806f5e0 0x7fba38071aa0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba48006010 tx=0x7fba48005960 comp rx=0 tx=0).stop 2024-04-02T04:58:55.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 msgr2=0x7fba4c1a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:55.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c1a45e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d950 tx=0x7fba3c00dc60 comp rx=0 tx=0).stop 2024-04-02T04:58:55.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 shutdown_connections 2024-04-02T04:58:55.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fba3806f5e0 0x7fba38071aa0 secure :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba48006010 tx=0x7fba48005960 comp rx=0 tx=0).stop 2024-04-02T04:58:55.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba4c1073a0 0x7fba4c19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:55.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c102290 0x7fba4c1a45e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:55.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 --2- 172.21.15.69:0/4085551155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba4c104f10 0x7fba4c1a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:55.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.012+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 >> 172.21.15.69:0/4085551155 conn(0x7fba4c0771e0 msgr2=0x7fba4c100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:55.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.020+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 shutdown_connections 2024-04-02T04:58:55.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.020+0000 7fba53bb4700 1 -- 172.21.15.69:0/4085551155 wait complete. 2024-04-02T04:58:55.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:55.020+0000 7fba53bb4700 1 librados: shutdown 2024-04-02T04:58:55.022 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T04:58:55.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:55.073 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:55.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:55.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:55.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:55.329 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:55.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:55.459 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:55.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: cluster 2024-04-02T04:58:54.245488+0000 mgr.y (mgr.14538) 595 : cluster 0 pgmap v960: 302 pgs: 302 active+clean; 8.4 MiB data, 488 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: cluster 2024-04-02T04:58:54.466052+0000 osd.7 (osd.7) 125 : cluster 0 4.11 scrub starts 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: cluster 2024-04-02T04:58:54.470277+0000 osd.7 (osd.7) 126 : cluster 0 4.11 scrub ok 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.003022+0000 mon.a (mon.0) 3231 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: audit 2024-04-02T04:58:55.007309+0000 mon.a (mon.0) 3232 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]': finished 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: audit 2024-04-02T04:58:55.007503+0000 mon.a (mon.0) 3233 : audit 1 from='client.? 172.21.15.69:0/4085551155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.017113+0000 mon.a (mon.0) 3234 : cluster 0 osdmap e637: 8 total, 8 up, 6 in 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: audit 2024-04-02T04:58:55.096867+0000 mon.c (mon.2) 1105 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[21528]: audit 2024-04-02T04:58:55.097181+0000 mon.a (mon.0) 3235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: cluster 2024-04-02T04:58:54.245488+0000 mgr.y (mgr.14538) 595 : cluster 0 pgmap v960: 302 pgs: 302 active+clean; 8.4 MiB data, 488 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: cluster 2024-04-02T04:58:54.466052+0000 osd.7 (osd.7) 125 : cluster 0 4.11 scrub starts 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: cluster 2024-04-02T04:58:54.470277+0000 osd.7 (osd.7) 126 : cluster 0 4.11 scrub ok 2024-04-02T04:58:55.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.003022+0000 mon.a (mon.0) 3231 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:58:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: audit 2024-04-02T04:58:55.007309+0000 mon.a (mon.0) 3232 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]': finished 2024-04-02T04:58:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: audit 2024-04-02T04:58:55.007503+0000 mon.a (mon.0) 3233 : audit 1 from='client.? 172.21.15.69:0/4085551155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:58:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.017113+0000 mon.a (mon.0) 3234 : cluster 0 osdmap e637: 8 total, 8 up, 6 in 2024-04-02T04:58:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: audit 2024-04-02T04:58:55.096867+0000 mon.c (mon.2) 1105 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:55 smithi069 bash[28904]: audit 2024-04-02T04:58:55.097181+0000 mon.a (mon.0) 3235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:55.589 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:55.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:55.719 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:55.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:55.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:55.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: cluster 2024-04-02T04:58:54.245488+0000 mgr.y (mgr.14538) 595 : cluster 0 pgmap v960: 302 pgs: 302 active+clean; 8.4 MiB data, 488 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: cluster 2024-04-02T04:58:54.466052+0000 osd.7 (osd.7) 125 : cluster 0 4.11 scrub starts 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: cluster 2024-04-02T04:58:54.470277+0000 osd.7 (osd.7) 126 : cluster 0 4.11 scrub ok 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.003022+0000 mon.a (mon.0) 3231 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: audit 2024-04-02T04:58:55.007309+0000 mon.a (mon.0) 3232 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-98","var": "target_max_objects","val": "250"}]': finished 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: audit 2024-04-02T04:58:55.007503+0000 mon.a (mon.0) 3233 : audit 1 from='client.? 172.21.15.69:0/4085551155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T04:58:55.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.017113+0000 mon.a (mon.0) 3234 : cluster 0 osdmap e637: 8 total, 8 up, 6 in 2024-04-02T04:58:55.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: audit 2024-04-02T04:58:55.096867+0000 mon.c (mon.2) 1105 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:55.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:55 smithi098 bash[23790]: audit 2024-04-02T04:58:55.097181+0000 mon.a (mon.0) 3235 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:58:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:55.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:56.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:56.108 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:56.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:56.237 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:56.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:56.370 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:56.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:56.501 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:56.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:56.613 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:54.933486+0000 osd.0 (osd.0) 327 : cluster 0 3.6 scrub starts 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:54.937941+0000 osd.0 (osd.0) 328 : cluster 0 3.6 scrub ok 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.188516+0000 osd.4 (osd.4) 336 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.188805+0000 osd.4 (osd.4) 337 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.431036+0000 osd.7 (osd.7) 127 : cluster 0 2.5 scrub starts 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.433230+0000 osd.7 (osd.7) 128 : cluster 0 2.5 scrub ok 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: audit 2024-04-02T04:58:56.083320+0000 mon.a (mon.0) 3236 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.087597+0000 mon.a (mon.0) 3237 : cluster 0 osdmap e638: 8 total, 8 up, 6 in 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: audit 2024-04-02T04:58:56.092309+0000 mon.c (mon.2) 1106 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[28904]: audit 2024-04-02T04:58:56.093330+0000 mon.a (mon.0) 3238 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:56.614 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:54.933486+0000 osd.0 (osd.0) 327 : cluster 0 3.6 scrub starts 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:54.937941+0000 osd.0 (osd.0) 328 : cluster 0 3.6 scrub ok 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.188516+0000 osd.4 (osd.4) 336 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.188805+0000 osd.4 (osd.4) 337 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.431036+0000 osd.7 (osd.7) 127 : cluster 0 2.5 scrub starts 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.433230+0000 osd.7 (osd.7) 128 : cluster 0 2.5 scrub ok 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: audit 2024-04-02T04:58:56.083320+0000 mon.a (mon.0) 3236 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.087597+0000 mon.a (mon.0) 3237 : cluster 0 osdmap e638: 8 total, 8 up, 6 in 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: audit 2024-04-02T04:58:56.092309+0000 mon.c (mon.2) 1106 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:56.615 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:56 smithi069 bash[21528]: audit 2024-04-02T04:58:56.093330+0000 mon.a (mon.0) 3238 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:56.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-02T04:58:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:56.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:56.756 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:56.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:56.879 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:56.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:56.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:54.933486+0000 osd.0 (osd.0) 327 : cluster 0 3.6 scrub starts 2024-04-02T04:58:56.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:54.937941+0000 osd.0 (osd.0) 328 : cluster 0 3.6 scrub ok 2024-04-02T04:58:56.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.188516+0000 osd.4 (osd.4) 336 : cluster 0 purged_snaps scrub starts 2024-04-02T04:58:56.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.188805+0000 osd.4 (osd.4) 337 : cluster 0 purged_snaps scrub ok 2024-04-02T04:58:56.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.431036+0000 osd.7 (osd.7) 127 : cluster 0 2.5 scrub starts 2024-04-02T04:58:56.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.433230+0000 osd.7 (osd.7) 128 : cluster 0 2.5 scrub ok 2024-04-02T04:58:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: audit 2024-04-02T04:58:56.083320+0000 mon.a (mon.0) 3236 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:58:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.087597+0000 mon.a (mon.0) 3237 : cluster 0 osdmap e638: 8 total, 8 up, 6 in 2024-04-02T04:58:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: audit 2024-04-02T04:58:56.092309+0000 mon.c (mon.2) 1106 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:56.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:56 smithi098 bash[23790]: audit 2024-04-02T04:58:56.093330+0000 mon.a (mon.0) 3238 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]: dispatch 2024-04-02T04:58:56.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:57.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:57.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:57.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:57.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:57.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:58:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:58:57.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:57.526 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:57.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:57.640 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.934808+0000 osd.1 (osd.1) 335 : cluster 0 102.13 scrub starts 2024-04-02T04:58:57.640 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:55.938954+0000 osd.1 (osd.1) 336 : cluster 0 102.13 scrub ok 2024-04-02T04:58:57.640 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.013908+0000 osd.4 (osd.4) 338 : cluster 0 40.1b scrub starts 2024-04-02T04:58:57.641 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.015375+0000 osd.4 (osd.4) 339 : cluster 0 40.1b scrub ok 2024-04-02T04:58:57.641 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.246425+0000 mgr.y (mgr.14538) 596 : cluster 0 pgmap v963: 302 pgs: 302 active+clean; 8.4 MiB data, 488 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:57.641 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.404384+0000 osd.7 (osd.7) 129 : cluster 0 6.c deep-scrub starts 2024-04-02T04:58:57.641 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.418312+0000 osd.7 (osd.7) 130 : cluster 0 6.c deep-scrub ok 2024-04-02T04:58:57.641 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: audit 2024-04-02T04:58:57.094116+0000 mon.a (mon.0) 3239 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]': finished 2024-04-02T04:58:57.641 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:57 smithi098 bash[23790]: cluster 2024-04-02T04:58:57.101116+0000 mon.a (mon.0) 3240 : cluster 0 osdmap e639: 8 total, 8 up, 6 in 2024-04-02T04:58:57.655 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:57.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.934808+0000 osd.1 (osd.1) 335 : cluster 0 102.13 scrub starts 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:55.938954+0000 osd.1 (osd.1) 336 : cluster 0 102.13 scrub ok 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.013908+0000 osd.4 (osd.4) 338 : cluster 0 40.1b scrub starts 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.015375+0000 osd.4 (osd.4) 339 : cluster 0 40.1b scrub ok 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.246425+0000 mgr.y (mgr.14538) 596 : cluster 0 pgmap v963: 302 pgs: 302 active+clean; 8.4 MiB data, 488 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.404384+0000 osd.7 (osd.7) 129 : cluster 0 6.c deep-scrub starts 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.418312+0000 osd.7 (osd.7) 130 : cluster 0 6.c deep-scrub ok 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: audit 2024-04-02T04:58:57.094116+0000 mon.a (mon.0) 3239 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]': finished 2024-04-02T04:58:57.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[21528]: cluster 2024-04-02T04:58:57.101116+0000 mon.a (mon.0) 3240 : cluster 0 osdmap e639: 8 total, 8 up, 6 in 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.934808+0000 osd.1 (osd.1) 335 : cluster 0 102.13 scrub starts 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:55.938954+0000 osd.1 (osd.1) 336 : cluster 0 102.13 scrub ok 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.013908+0000 osd.4 (osd.4) 338 : cluster 0 40.1b scrub starts 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.015375+0000 osd.4 (osd.4) 339 : cluster 0 40.1b scrub ok 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.246425+0000 mgr.y (mgr.14538) 596 : cluster 0 pgmap v963: 302 pgs: 302 active+clean; 8.4 MiB data, 488 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.404384+0000 osd.7 (osd.7) 129 : cluster 0 6.c deep-scrub starts 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.418312+0000 osd.7 (osd.7) 130 : cluster 0 6.c deep-scrub ok 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: audit 2024-04-02T04:58:57.094116+0000 mon.a (mon.0) 3239 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-98"}]': finished 2024-04-02T04:58:57.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:57 smithi069 bash[28904]: cluster 2024-04-02T04:58:57.101116+0000 mon.a (mon.0) 3240 : cluster 0 osdmap e639: 8 total, 8 up, 6 in 2024-04-02T04:58:57.785 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:57.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:57.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:58.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:58:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:58:58.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:58.176 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- 172.21.15.69:0/1909510653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030101d10 msgr2=0x7ff030102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- 172.21.15.69:0/1909510653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030101d10 0x7ff030102190 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff018009990 tx=0x7ff018009ca0 comp rx=0 tx=0).stop 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- 172.21.15.69:0/1909510653 shutdown_connections 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- 172.21.15.69:0/1909510653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0301026d0 0x7ff03010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- 172.21.15.69:0/1909510653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff030107430 0x7ff030107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- 172.21.15.69:0/1909510653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030101d10 0x7ff030102190 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- 172.21.15.69:0/1909510653 >> 172.21.15.69:0/1909510653 conn(0x7ff030076460 msgr2=0x7ff030076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- 172.21.15.69:0/1909510653 shutdown_connections 2024-04-02T04:58:58.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- 172.21.15.69:0/1909510653 wait complete. 2024-04-02T04:58:58.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 librados: starting msgr at 2024-04-02T04:58:58.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 librados: starting objecter 2024-04-02T04:58:58.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 Processor -- start 2024-04-02T04:58:58.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- start start 2024-04-02T04:58:58.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 librados: setting wanted keys 2024-04-02T04:58:58.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 librados: calling monclient init 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 0x7ff030102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0301026d0 0x7ff0301033d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 0x7ff030103910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff030110a90 con 0x7ff030101d10 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff030110910 con 0x7ff0301026d0 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff036f62700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff030110c10 con 0x7ff030107430 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff034cfe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 0x7ff030102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff0354ff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 0x7ff030103910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff034cfe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 0x7ff030102e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43998/0 (socket says 172.21.15.69:43998) 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff0354ff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 0x7ff030103910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40822/0 (socket says 172.21.15.69:40822) 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.272+0000 7ff034cfe700 1 -- 172.21.15.69:0/4092408130 learned_addr learned my addr 172.21.15.69:0/4092408130 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02ffff700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0301026d0 0x7ff0301033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:58.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff0354ff700 1 -- 172.21.15.69:0/4092408130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0301026d0 msgr2=0x7ff0301033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff0354ff700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0301026d0 0x7ff0301033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff0354ff700 1 -- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 msgr2=0x7ff030102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff0354ff700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 0x7ff030102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff0354ff700 1 -- 172.21.15.69:0/4092408130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0301a6590 con 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff034cfe700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 0x7ff030102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff0354ff700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 0x7ff030103910 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff02800ef90 tx=0x7ff02800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02dffb700 10 client.?.objecter ms_handle_connect 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0280092e0 con 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff028004bb0 con 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028004020 con 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0301a6820 con 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff036f62700 10 client.26813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0301a6d10 con 0x7ff030107430 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff036f62700 1 librados: init done 2024-04-02T04:58:58.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03004ff00 con 0x7ff030107430 2024-04-02T04:58:58.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.276+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff02801b030 con 0x7ff030107430 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.280+0000 7ff02dffb700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff01c06f5e0 0x7ff01c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.280+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(640..640 src has 74..640) ==== 9784+0+0 (secure 0 0 0) 0x7ff02808e5d0 con 0x7ff030107430 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.280+0000 7ff02dffb700 10 client.26813.objecter ms_dispatch 0x7ff030001550 osd_map(640..640 src has 74..640) 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.280+0000 7ff02dffb700 3 client.26813.objecter handle_osd_map got epochs [640,640] > 0 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.280+0000 7ff02dffb700 3 client.26813.objecter handle_osd_map decoding full epoch 640 2024-04-02T04:58:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.280+0000 7ff02dffb700 20 client.26813.objecter dump_active .. 0 homeless 2024-04-02T04:58:58.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.284+0000 7ff034cfe700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff01c06f5e0 0x7ff01c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:58:58.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.284+0000 7ff02dffb700 10 client.26813.objecter ms_handle_connect 0x7ff01c06f5e0 2024-04-02T04:58:58.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.284+0000 7ff034cfe700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff01c06f5e0 0x7ff01c071aa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff02000f4d0 tx=0x7ff020005f90 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:58:58.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.284+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff028022020 con 0x7ff030107430 2024-04-02T04:58:58.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:58.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:58.433 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:58.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:58.512+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd in", "ids": ["3"]} v 0) -- 0x7ff0301a7030 con 0x7ff030107430 2024-04-02T04:58:58.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:58.560 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:58:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:58:58.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:58.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:58.818 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:58:58.820 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:58:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:58:58.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[21528]: debug 2024-04-02T04:58:58.552+0000 7f02343d7700 -1 mon.a@0(leader).osd e640 definitely_dead 0 2024-04-02T04:58:58.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.990798+0000 osd.4 (osd.4) 340 : cluster 0 10.10 scrub starts 2024-04-02T04:58:58.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[21528]: cluster 2024-04-02T04:58:56.993342+0000 osd.4 (osd.4) 341 : cluster 0 10.10 scrub ok 2024-04-02T04:58:58.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.108655+0000 mon.a (mon.0) 3241 : cluster 0 osdmap e640: 8 total, 8 up, 6 in 2024-04-02T04:58:58.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.990798+0000 osd.4 (osd.4) 340 : cluster 0 10.10 scrub starts 2024-04-02T04:58:58.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[28904]: cluster 2024-04-02T04:58:56.993342+0000 osd.4 (osd.4) 341 : cluster 0 10.10 scrub ok 2024-04-02T04:58:58.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:58 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.108655+0000 mon.a (mon.0) 3241 : cluster 0 osdmap e640: 8 total, 8 up, 6 in 2024-04-02T04:58:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:58:58.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:58.949 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:58:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:58.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:58:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:58 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.990798+0000 osd.4 (osd.4) 340 : cluster 0 10.10 scrub starts 2024-04-02T04:58:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:58 smithi098 bash[23790]: cluster 2024-04-02T04:58:56.993342+0000 osd.4 (osd.4) 341 : cluster 0 10.10 scrub ok 2024-04-02T04:58:58.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:58 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.108655+0000 mon.a (mon.0) 3241 : cluster 0 osdmap e640: 8 total, 8 up, 6 in 2024-04-02T04:58:59.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.078 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:59.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.108+0000 7ff02dffb700 1 -- 172.21.15.69:0/4092408130 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["3"]}]=0 marked in osd.3. v641) ==== 85+0+0 (secure 0 0 0) 0x7ff028010040 con 0x7ff030107430 2024-04-02T04:58:59.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 10 librados: watch_flush enter 2024-04-02T04:58:59.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 10 librados: watch_flush exit 2024-04-02T04:58:59.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 20 client.26813.objecter shutdown clearing up homeless session... 2024-04-02T04:58:59.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 10 client.26813.objecter successfully canceled tick 2024-04-02T04:58:59.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff01c06f5e0 msgr2=0x7ff01c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:59.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff01c06f5e0 0x7ff01c071aa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff02000f4d0 tx=0x7ff020005f90 comp rx=0 tx=0).stop 2024-04-02T04:58:59.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 msgr2=0x7ff030103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:58:59.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.112+0000 7ff036f62700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 0x7ff030103910 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff02800ef90 tx=0x7ff02800c3a0 comp rx=0 tx=0).stop 2024-04-02T04:58:59.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 shutdown_connections 2024-04-02T04:58:59.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff01c06f5e0 0x7ff01c071aa0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:59.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff030107430 0x7ff030103910 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:59.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff030101d10 0x7ff030102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:59.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 --2- 172.21.15.69:0/4092408130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0301026d0 0x7ff0301033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:58:59.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 >> 172.21.15.69:0/4092408130 conn(0x7ff030076460 msgr2=0x7ff030100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:58:59.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 shutdown_connections 2024-04-02T04:58:59.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.116+0000 7ff036f62700 1 -- 172.21.15.69:0/4092408130 wait complete. 2024-04-02T04:58:59.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:58:59.120+0000 7ff036f62700 1 librados: shutdown 2024-04-02T04:58:59.122 INFO:teuthology.orchestra.run.smithi069.stderr:marked in osd.3. 2024-04-02T04:58:59.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.200 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:59.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:59.330 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:58:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:58:59.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T04:58:59.459 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:59.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:59.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:57.451795+0000 osd.7 (osd.7) 131 : cluster 0 3.d scrub starts 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:57.453994+0000 osd.7 (osd.7) 132 : cluster 0 3.d scrub ok 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:57.898577+0000 osd.6 (osd.6) 277 : cluster 0 10.16 scrub starts 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:57.900908+0000 osd.6 (osd.6) 278 : cluster 0 10.16 scrub ok 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.037608+0000 osd.4 (osd.4) 342 : cluster 0 10.1 scrub starts 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.039245+0000 osd.4 (osd.4) 343 : cluster 0 10.1 scrub ok 2024-04-02T04:58:59.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.247452+0000 mgr.y (mgr.14538) 597 : cluster 0 pgmap v966: 270 pgs: 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: audit 2024-04-02T04:58:58.517267+0000 mon.c (mon.2) 1107 : audit 0 from='client.? 172.21.15.69:0/4092408130' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: audit 2024-04-02T04:58:58.558960+0000 mon.a (mon.0) 3242 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: audit 2024-04-02T04:58:59.106774+0000 mon.a (mon.0) 3243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: cluster 2024-04-02T04:58:59.110926+0000 mon.a (mon.0) 3244 : cluster 0 osdmap e641: 8 total, 8 up, 7 in 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: audit 2024-04-02T04:58:59.117719+0000 mon.c (mon.2) 1108 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[21528]: audit 2024-04-02T04:58:59.119101+0000 mon.a (mon.0) 3245 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:57.451795+0000 osd.7 (osd.7) 131 : cluster 0 3.d scrub starts 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:57.453994+0000 osd.7 (osd.7) 132 : cluster 0 3.d scrub ok 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:57.898577+0000 osd.6 (osd.6) 277 : cluster 0 10.16 scrub starts 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:57.900908+0000 osd.6 (osd.6) 278 : cluster 0 10.16 scrub ok 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.037608+0000 osd.4 (osd.4) 342 : cluster 0 10.1 scrub starts 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.039245+0000 osd.4 (osd.4) 343 : cluster 0 10.1 scrub ok 2024-04-02T04:58:59.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.247452+0000 mgr.y (mgr.14538) 597 : cluster 0 pgmap v966: 270 pgs: 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:58:59.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: audit 2024-04-02T04:58:58.517267+0000 mon.c (mon.2) 1107 : audit 0 from='client.? 172.21.15.69:0/4092408130' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-02T04:58:59.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: audit 2024-04-02T04:58:58.558960+0000 mon.a (mon.0) 3242 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-02T04:58:59.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: audit 2024-04-02T04:58:59.106774+0000 mon.a (mon.0) 3243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-02T04:58:59.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: cluster 2024-04-02T04:58:59.110926+0000 mon.a (mon.0) 3244 : cluster 0 osdmap e641: 8 total, 8 up, 7 in 2024-04-02T04:58:59.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: audit 2024-04-02T04:58:59.117719+0000 mon.c (mon.2) 1108 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:59.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:58:59 smithi069 bash[28904]: audit 2024-04-02T04:58:59.119101+0000 mon.a (mon.0) 3245 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:59.708 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:57.451795+0000 osd.7 (osd.7) 131 : cluster 0 3.d scrub starts 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:57.453994+0000 osd.7 (osd.7) 132 : cluster 0 3.d scrub ok 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:57.898577+0000 osd.6 (osd.6) 277 : cluster 0 10.16 scrub starts 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:57.900908+0000 osd.6 (osd.6) 278 : cluster 0 10.16 scrub ok 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.037608+0000 osd.4 (osd.4) 342 : cluster 0 10.1 scrub starts 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.039245+0000 osd.4 (osd.4) 343 : cluster 0 10.1 scrub ok 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.247452+0000 mgr.y (mgr.14538) 597 : cluster 0 pgmap v966: 270 pgs: 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: audit 2024-04-02T04:58:58.517267+0000 mon.c (mon.2) 1107 : audit 0 from='client.? 172.21.15.69:0/4092408130' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-02T04:58:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: audit 2024-04-02T04:58:58.558960+0000 mon.a (mon.0) 3242 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-02T04:58:59.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: audit 2024-04-02T04:58:59.106774+0000 mon.a (mon.0) 3243 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-02T04:58:59.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: cluster 2024-04-02T04:58:59.110926+0000 mon.a (mon.0) 3244 : cluster 0 osdmap e641: 8 total, 8 up, 7 in 2024-04-02T04:58:59.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: audit 2024-04-02T04:58:59.117719+0000 mon.c (mon.2) 1108 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:59.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:58:59 smithi098 bash[23790]: audit 2024-04-02T04:58:59.119101+0000 mon.a (mon.0) 3245 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:58:59.810 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:59.836 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:58:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:58:59.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:58:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:58:59.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:00.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:00.095 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-02T04:59:00.105 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:00.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:00.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:00.236 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:00.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:00.360 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:00.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:00.487 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:00.499 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:00.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:00.609 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:00.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:00.738 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.950808+0000 osd.1 (osd.1) 337 : cluster 0 102.4 scrub starts 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.953597+0000 osd.1 (osd.1) 338 : cluster 0 102.4 scrub ok 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.966042+0000 osd.0 (osd.0) 329 : cluster 0 3.1f scrub starts 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:58:58.968754+0000 osd.0 (osd.0) 330 : cluster 0 3.1f scrub ok 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:58:59.896279+0000 osd.6 (osd.6) 279 : cluster 0 6.a deep-scrub starts 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:58:59.898488+0000 osd.6 (osd.6) 280 : cluster 0 6.a deep-scrub ok 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: audit 2024-04-02T04:59:00.009367+0000 mon.a (mon.0) 3246 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: audit 2024-04-02T04:59:00.024662+0000 mon.c (mon.2) 1109 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:00.027144+0000 mon.a (mon.0) 3247 : cluster 0 osdmap e642: 8 total, 8 up, 7 in 2024-04-02T04:59:00.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[21528]: audit 2024-04-02T04:59:00.030173+0000 mon.a (mon.0) 3248 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:00.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.950808+0000 osd.1 (osd.1) 337 : cluster 0 102.4 scrub starts 2024-04-02T04:59:00.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.953597+0000 osd.1 (osd.1) 338 : cluster 0 102.4 scrub ok 2024-04-02T04:59:00.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.966042+0000 osd.0 (osd.0) 329 : cluster 0 3.1f scrub starts 2024-04-02T04:59:00.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:58:58.968754+0000 osd.0 (osd.0) 330 : cluster 0 3.1f scrub ok 2024-04-02T04:59:00.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:58:59.896279+0000 osd.6 (osd.6) 279 : cluster 0 6.a deep-scrub starts 2024-04-02T04:59:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:58:59.898488+0000 osd.6 (osd.6) 280 : cluster 0 6.a deep-scrub ok 2024-04-02T04:59:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: audit 2024-04-02T04:59:00.009367+0000 mon.a (mon.0) 3246 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: audit 2024-04-02T04:59:00.024662+0000 mon.c (mon.2) 1109 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:00.027144+0000 mon.a (mon.0) 3247 : cluster 0 osdmap e642: 8 total, 8 up, 7 in 2024-04-02T04:59:00.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:00 smithi069 bash[28904]: audit 2024-04-02T04:59:00.030173+0000 mon.a (mon.0) 3248 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:00.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:00.870 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:00.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:00.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.950808+0000 osd.1 (osd.1) 337 : cluster 0 102.4 scrub starts 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.953597+0000 osd.1 (osd.1) 338 : cluster 0 102.4 scrub ok 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.966042+0000 osd.0 (osd.0) 329 : cluster 0 3.1f scrub starts 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:58:58.968754+0000 osd.0 (osd.0) 330 : cluster 0 3.1f scrub ok 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:58:59.896279+0000 osd.6 (osd.6) 279 : cluster 0 6.a deep-scrub starts 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:58:59.898488+0000 osd.6 (osd.6) 280 : cluster 0 6.a deep-scrub ok 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: audit 2024-04-02T04:59:00.009367+0000 mon.a (mon.0) 3246 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: audit 2024-04-02T04:59:00.024662+0000 mon.c (mon.2) 1109 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:00.027144+0000 mon.a (mon.0) 3247 : cluster 0 osdmap e642: 8 total, 8 up, 7 in 2024-04-02T04:59:00.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:00 smithi098 bash[23790]: audit 2024-04-02T04:59:00.030173+0000 mon.a (mon.0) 3248 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:00.999 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:01.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:01.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:01.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:01.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:01.386 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:01.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:01.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:01.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:01.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:01.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:58:59.980887+0000 osd.1 (osd.1) 339 : cluster 0 43.5 deep-scrub starts 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:58:59.982677+0000 osd.1 (osd.1) 340 : cluster 0 43.5 deep-scrub ok 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:00.247969+0000 mgr.y (mgr.14538) 598 : cluster 0 pgmap v969: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:00.396710+0000 osd.7 (osd.7) 133 : cluster 0 6.12 scrub starts 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:00.398872+0000 osd.7 (osd.7) 134 : cluster 0 6.12 scrub ok 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:00.558188+0000 osd.2 (osd.2) 305 : cluster 0 10.5 scrub starts 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:00.559341+0000 osd.2 (osd.2) 306 : cluster 0 10.5 scrub ok 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: audit 2024-04-02T04:59:01.012724+0000 mon.a (mon.0) 3249 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:59:01.761 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:01.020703+0000 mon.a (mon.0) 3250 : cluster 0 osdmap e643: 8 total, 8 up, 7 in 2024-04-02T04:59:01.762 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: audit 2024-04-02T04:59:01.031199+0000 mon.c (mon.2) 1110 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:01.762 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:01 smithi098 bash[23790]: audit 2024-04-02T04:59:01.036838+0000 mon.a (mon.0) 3251 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:01.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:58:59.980887+0000 osd.1 (osd.1) 339 : cluster 0 43.5 deep-scrub starts 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:58:59.982677+0000 osd.1 (osd.1) 340 : cluster 0 43.5 deep-scrub ok 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:00.247969+0000 mgr.y (mgr.14538) 598 : cluster 0 pgmap v969: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:00.396710+0000 osd.7 (osd.7) 133 : cluster 0 6.12 scrub starts 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:00.398872+0000 osd.7 (osd.7) 134 : cluster 0 6.12 scrub ok 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:00.558188+0000 osd.2 (osd.2) 305 : cluster 0 10.5 scrub starts 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:00.559341+0000 osd.2 (osd.2) 306 : cluster 0 10.5 scrub ok 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: audit 2024-04-02T04:59:01.012724+0000 mon.a (mon.0) 3249 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:01.020703+0000 mon.a (mon.0) 3250 : cluster 0 osdmap e643: 8 total, 8 up, 7 in 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: audit 2024-04-02T04:59:01.031199+0000 mon.c (mon.2) 1110 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[28904]: audit 2024-04-02T04:59:01.036838+0000 mon.a (mon.0) 3251 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:01.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:58:59.980887+0000 osd.1 (osd.1) 339 : cluster 0 43.5 deep-scrub starts 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:58:59.982677+0000 osd.1 (osd.1) 340 : cluster 0 43.5 deep-scrub ok 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:00.247969+0000 mgr.y (mgr.14538) 598 : cluster 0 pgmap v969: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:00.396710+0000 osd.7 (osd.7) 133 : cluster 0 6.12 scrub starts 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:00.398872+0000 osd.7 (osd.7) 134 : cluster 0 6.12 scrub ok 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:00.558188+0000 osd.2 (osd.2) 305 : cluster 0 10.5 scrub starts 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:00.559341+0000 osd.2 (osd.2) 306 : cluster 0 10.5 scrub ok 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: audit 2024-04-02T04:59:01.012724+0000 mon.a (mon.0) 3249 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:01.020703+0000 mon.a (mon.0) 3250 : cluster 0 osdmap e643: 8 total, 8 up, 7 in 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: audit 2024-04-02T04:59:01.031199+0000 mon.c (mon.2) 1110 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:01.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:01 smithi069 bash[21528]: audit 2024-04-02T04:59:01.036838+0000 mon.a (mon.0) 3251 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:01.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:02.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:02.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:02.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:02.298 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:02.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:02.533 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:02.561 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:02.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:02.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:02 smithi098 bash[23790]: cluster 2024-04-02T04:59:01.052947+0000 osd.3 (osd.3) 321 : cluster 0 3.c scrub starts 2024-04-02T04:59:02.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:02 smithi098 bash[23790]: cluster 2024-04-02T04:59:01.058789+0000 osd.3 (osd.3) 322 : cluster 0 3.c scrub ok 2024-04-02T04:59:02.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:02 smithi098 bash[23790]: audit 2024-04-02T04:59:02.014973+0000 mon.a (mon.0) 3252 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]': finished 2024-04-02T04:59:02.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:02 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.019320+0000 mon.a (mon.0) 3253 : cluster 0 osdmap e644: 8 total, 8 up, 7 in 2024-04-02T04:59:02.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:02 smithi098 bash[23790]: audit 2024-04-02T04:59:02.020656+0000 mon.c (mon.2) 1111 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]: dispatch 2024-04-02T04:59:02.676 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:02 smithi098 bash[23790]: audit 2024-04-02T04:59:02.022612+0000 mon.a (mon.0) 3254 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]: dispatch 2024-04-02T04:59:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:02.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:02.822 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:02.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[28904]: cluster 2024-04-02T04:59:01.052947+0000 osd.3 (osd.3) 321 : cluster 0 3.c scrub starts 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[28904]: cluster 2024-04-02T04:59:01.058789+0000 osd.3 (osd.3) 322 : cluster 0 3.c scrub ok 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[28904]: audit 2024-04-02T04:59:02.014973+0000 mon.a (mon.0) 3252 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]': finished 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.019320+0000 mon.a (mon.0) 3253 : cluster 0 osdmap e644: 8 total, 8 up, 7 in 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[28904]: audit 2024-04-02T04:59:02.020656+0000 mon.c (mon.2) 1111 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]: dispatch 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[28904]: audit 2024-04-02T04:59:02.022612+0000 mon.a (mon.0) 3254 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]: dispatch 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[21528]: cluster 2024-04-02T04:59:01.052947+0000 osd.3 (osd.3) 321 : cluster 0 3.c scrub starts 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[21528]: cluster 2024-04-02T04:59:01.058789+0000 osd.3 (osd.3) 322 : cluster 0 3.c scrub ok 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[21528]: audit 2024-04-02T04:59:02.014973+0000 mon.a (mon.0) 3252 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-6", "overlaypool": "test-rados-api-smithi069-72349-100"}]': finished 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.019320+0000 mon.a (mon.0) 3253 : cluster 0 osdmap e644: 8 total, 8 up, 7 in 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[21528]: audit 2024-04-02T04:59:02.020656+0000 mon.c (mon.2) 1111 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]: dispatch 2024-04-02T04:59:02.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:02 smithi069 bash[21528]: audit 2024-04-02T04:59:02.022612+0000 mon.a (mon.0) 3254 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]: dispatch 2024-04-02T04:59:02.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:02.950 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:02.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:03.053 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:03.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:03.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:03.205 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:03.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:03.320 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:03.334 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:03.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:03.463 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:03.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:03.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:03.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:03.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:59:03.709 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.248779+0000 mgr.y (mgr.14538) 599 : cluster 0 pgmap v972: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:59:03.709 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.698475+0000 osd.2 (osd.2) 307 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.698803+0000 osd.2 (osd.2) 308 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.928441+0000 osd.6 (osd.6) 281 : cluster 0 43.b scrub starts 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.931027+0000 osd.6 (osd.6) 282 : cluster 0 43.b scrub ok 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.015403+0000 mon.a (mon.0) 3255 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: audit 2024-04-02T04:59:03.017355+0000 mon.a (mon.0) 3256 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]': finished 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.027865+0000 mon.a (mon.0) 3257 : cluster 0 osdmap e645: 8 total, 8 up, 7 in 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: audit 2024-04-02T04:59:03.031666+0000 mon.c (mon.2) 1112 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:03.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:03 smithi098 bash[23790]: audit 2024-04-02T04:59:03.041228+0000 mon.a (mon.0) 3258 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:03.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:03.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.248779+0000 mgr.y (mgr.14538) 599 : cluster 0 pgmap v972: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:59:03.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.698475+0000 osd.2 (osd.2) 307 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:03.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.698803+0000 osd.2 (osd.2) 308 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.928441+0000 osd.6 (osd.6) 281 : cluster 0 43.b scrub starts 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.931027+0000 osd.6 (osd.6) 282 : cluster 0 43.b scrub ok 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.015403+0000 mon.a (mon.0) 3255 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: audit 2024-04-02T04:59:03.017355+0000 mon.a (mon.0) 3256 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]': finished 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.027865+0000 mon.a (mon.0) 3257 : cluster 0 osdmap e645: 8 total, 8 up, 7 in 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: audit 2024-04-02T04:59:03.031666+0000 mon.c (mon.2) 1112 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[28904]: audit 2024-04-02T04:59:03.041228+0000 mon.a (mon.0) 3258 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.248779+0000 mgr.y (mgr.14538) 599 : cluster 0 pgmap v972: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 472 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.698475+0000 osd.2 (osd.2) 307 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.698803+0000 osd.2 (osd.2) 308 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:03.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.928441+0000 osd.6 (osd.6) 281 : cluster 0 43.b scrub starts 2024-04-02T04:59:03.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.931027+0000 osd.6 (osd.6) 282 : cluster 0 43.b scrub ok 2024-04-02T04:59:03.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.015403+0000 mon.a (mon.0) 3255 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:03.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: audit 2024-04-02T04:59:03.017355+0000 mon.a (mon.0) 3256 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-100", "mode": "writeback"}]': finished 2024-04-02T04:59:03.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.027865+0000 mon.a (mon.0) 3257 : cluster 0 osdmap e645: 8 total, 8 up, 7 in 2024-04-02T04:59:03.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: audit 2024-04-02T04:59:03.031666+0000 mon.c (mon.2) 1112 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:03.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:03 smithi069 bash[21528]: audit 2024-04-02T04:59:03.041228+0000 mon.a (mon.0) 3258 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:03.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:03.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:04.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:04.111 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:04.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:04.242 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:04.345 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:04.374 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:04.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:04.504 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:04.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:04.635 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:04.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:04.749 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.948887+0000 osd.3 (osd.3) 323 : cluster 0 3.1e deep-scrub starts 2024-04-02T04:59:04.749 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:02.951744+0000 osd.3 (osd.3) 324 : cluster 0 3.1e deep-scrub ok 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.453277+0000 osd.7 (osd.7) 135 : cluster 0 4.8 scrub starts 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.454559+0000 osd.7 (osd.7) 136 : cluster 0 4.8 scrub ok 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: audit 2024-04-02T04:59:03.520596+0000 mgr.y (mgr.14538) 600 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: audit 2024-04-02T04:59:03.683486+0000 mon.b (mon.1) 285 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: audit 2024-04-02T04:59:03.684468+0000 mon.a (mon.0) 3259 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.949685+0000 osd.6 (osd.6) 283 : cluster 0 40.14 scrub starts 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.952322+0000 osd.6 (osd.6) 284 : cluster 0 40.14 scrub ok 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.965073+0000 osd.1 (osd.1) 341 : cluster 0 43.f scrub starts 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.968420+0000 osd.1 (osd.1) 342 : cluster 0 43.f scrub ok 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: audit 2024-04-02T04:59:04.036867+0000 mon.a (mon.0) 3260 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: cluster 2024-04-02T04:59:04.044748+0000 mon.a (mon.0) 3261 : cluster 0 osdmap e646: 8 total, 8 up, 7 in 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: audit 2024-04-02T04:59:04.058028+0000 mon.c (mon.2) 1113 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:04.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:04 smithi098 bash[23790]: audit 2024-04-02T04:59:04.059230+0000 mon.a (mon.0) 3262 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:04.765 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.948887+0000 osd.3 (osd.3) 323 : cluster 0 3.1e deep-scrub starts 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:02.951744+0000 osd.3 (osd.3) 324 : cluster 0 3.1e deep-scrub ok 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.453277+0000 osd.7 (osd.7) 135 : cluster 0 4.8 scrub starts 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.454559+0000 osd.7 (osd.7) 136 : cluster 0 4.8 scrub ok 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: audit 2024-04-02T04:59:03.520596+0000 mgr.y (mgr.14538) 600 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: audit 2024-04-02T04:59:03.683486+0000 mon.b (mon.1) 285 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: audit 2024-04-02T04:59:03.684468+0000 mon.a (mon.0) 3259 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.949685+0000 osd.6 (osd.6) 283 : cluster 0 40.14 scrub starts 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.952322+0000 osd.6 (osd.6) 284 : cluster 0 40.14 scrub ok 2024-04-02T04:59:04.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.965073+0000 osd.1 (osd.1) 341 : cluster 0 43.f scrub starts 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.968420+0000 osd.1 (osd.1) 342 : cluster 0 43.f scrub ok 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: audit 2024-04-02T04:59:04.036867+0000 mon.a (mon.0) 3260 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: cluster 2024-04-02T04:59:04.044748+0000 mon.a (mon.0) 3261 : cluster 0 osdmap e646: 8 total, 8 up, 7 in 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: audit 2024-04-02T04:59:04.058028+0000 mon.c (mon.2) 1113 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[21528]: audit 2024-04-02T04:59:04.059230+0000 mon.a (mon.0) 3262 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.948887+0000 osd.3 (osd.3) 323 : cluster 0 3.1e deep-scrub starts 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:02.951744+0000 osd.3 (osd.3) 324 : cluster 0 3.1e deep-scrub ok 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.453277+0000 osd.7 (osd.7) 135 : cluster 0 4.8 scrub starts 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.454559+0000 osd.7 (osd.7) 136 : cluster 0 4.8 scrub ok 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: audit 2024-04-02T04:59:03.520596+0000 mgr.y (mgr.14538) 600 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: audit 2024-04-02T04:59:03.683486+0000 mon.b (mon.1) 285 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: audit 2024-04-02T04:59:03.684468+0000 mon.a (mon.0) 3259 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:04.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.949685+0000 osd.6 (osd.6) 283 : cluster 0 40.14 scrub starts 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.952322+0000 osd.6 (osd.6) 284 : cluster 0 40.14 scrub ok 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.965073+0000 osd.1 (osd.1) 341 : cluster 0 43.f scrub starts 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.968420+0000 osd.1 (osd.1) 342 : cluster 0 43.f scrub ok 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: audit 2024-04-02T04:59:04.036867+0000 mon.a (mon.0) 3260 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: cluster 2024-04-02T04:59:04.044748+0000 mon.a (mon.0) 3261 : cluster 0 osdmap e646: 8 total, 8 up, 7 in 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: audit 2024-04-02T04:59:04.058028+0000 mon.c (mon.2) 1113 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:04.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:04 smithi069 bash[28904]: audit 2024-04-02T04:59:04.059230+0000 mon.a (mon.0) 3262 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:04.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:04.897 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:04.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:05.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:05.026 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:05.096 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:59:05.096 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:59:05.096 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.5665245850577482 2024-04-02T04:59:05.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 0.5665245850577482 2024-04-02T04:59:05.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:05.157 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:05.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:05.286 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:05.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/4248440764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 msgr2=0x7f9de810e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- 172.21.15.69:0/4248440764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de810e810 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009990 tx=0x7f9ddc009ca0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/4248440764 shutdown_connections 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- 172.21.15.69:0/4248440764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de810ed50 0x7f9de8111140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- 172.21.15.69:0/4248440764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9de8069f70 0x7f9de806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- 172.21.15.69:0/4248440764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de810e810 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/4248440764 >> 172.21.15.69:0/4248440764 conn(0x7f9de80771e0 msgr2=0x7f9de80775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/4248440764 shutdown_connections 2024-04-02T04:59:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/4248440764 wait complete. 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 librados: starting msgr at 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 librados: starting objecter 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 Processor -- start 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- start start 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 librados: setting wanted keys 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 librados: calling monclient init 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de8069f70 0x7f9de819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9de810ed50 0x7f9de819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8113cf0 con 0x7f9de806a890 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8113b70 con 0x7f9de810ed50 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9de8113e70 con 0x7f9de8069f70 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de8069f70 0x7f9de819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de8069f70 0x7f9de819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40850/0 (socket says 172.21.15.69:40850) 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44014/0 (socket says 172.21.15.69:44014) 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 -- 172.21.15.69:0/3725599478 learned_addr learned my addr 172.21.15.69:0/3725599478 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9de810ed50 0x7f9de819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9de810ed50 msgr2=0x7f9de819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9de810ed50 0x7f9de819eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 msgr2=0x7f9de819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de819f2c0 con 0x7f9de8069f70 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9de806a890 0x7f9de819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:59:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de8069f70 0x7f9de819a2b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400c920 tx=0x7f9dd400cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de5ffb700 10 client.?.objecter ms_handle_connect 0x7f9de8069f70 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de5ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4007b00 con 0x7f9de8069f70 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9dd4004cb0 con 0x7f9de8069f70 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de819f5b0 con 0x7f9de8069f70 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40054a0 con 0x7f9de8069f70 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 10 client.26819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de81a6e30 con 0x7f9de8069f70 2024-04-02T04:59:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 librados: init done 2024-04-02T04:59:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.396+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9de804ff00 con 0x7f9de8069f70 2024-04-02T04:59:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f9dd4004750 con 0x7f9de8069f70 2024-04-02T04:59:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9dd806f5e0 0x7f9dd8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(647..647 src has 74..647) ==== 10178+0+0 (secure 0 0 0) 0x7f9dd408eb60 con 0x7f9de8069f70 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 osd_map(647..647 src has 74..647) 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 3 client.26819.objecter handle_osd_map got epochs [647,647] > 0 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 3 client.26819.objecter handle_osd_map decoding full epoch 647 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9dd806f5e0 0x7f9dd8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 20 client.26819.objecter dump_active .. 0 homeless 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v647) ==== 54+0+16 (secure 0 0 0) 0x7f9dd4014030 con 0x7f9de8069f70 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9dd806f5e0 0x7f9dd8071aa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00b7f0 tx=0x7f9ddc01a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.400+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9dd806f5e0 2024-04-02T04:59:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 10 client.26819.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 1 2024-04-02T04:59:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9de8062e50 0x7f9de8063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 1 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 0 1 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 4 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 10 client.26819.objecter _send_command 1 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9de81aa1f0 con 0x7f9de8062e50 2024-04-02T04:59:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9de8062e50 0x7f9de8063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9de8062e50 0x7f9de8063230 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de8062e50 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81aa1f0 con 0x7f9de8062e50 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 1: 0 ) 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 1 = system:0 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 4 2024-04-02T04:59:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.404+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 0 1 2024-04-02T04:59:05.410 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 10 client.26819.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 2 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 2 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de819b610 osd=0 3 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 0 2 2024-04-02T04:59:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 4 2024-04-02T04:59:05.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 10 client.26819.objecter _send_command 2 2024-04-02T04:59:05.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de80628d0 con 0x7f9de8062e50 2024-04-02T04:59:05.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de80628d0 con 0x7f9de8062e50 2024-04-02T04:59:05.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 4 2024-04-02T04:59:05.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 0 2 2024-04-02T04:59:05.427 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 10 client.26819.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 3 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9de802c9b0 0x7f9de802cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 3 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 1 3 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 4 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 10 client.26819.objecter _send_command 3 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9de8109620 con 0x7f9de802c9b0 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9de802c9b0 0x7f9de802cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de802c9b0 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.424+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9de802c9b0 0x7f9de802cdd0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.428+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de8109620 con 0x7f9de802c9b0 2024-04-02T04:59:05.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.428+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 3: 0 ) 2024-04-02T04:59:05.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.428+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 3 = system:0 2024-04-02T04:59:05.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.428+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 4 2024-04-02T04:59:05.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.428+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 1 3 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 10 client.26819.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 4 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 4 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de8108fa0 osd=1 3 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 1 4 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 4 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 10 client.26819.objecter _send_command 4 2024-04-02T04:59:05.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de8108a20 con 0x7f9de802c9b0 2024-04-02T04:59:05.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de8108a20 con 0x7f9de802c9b0 2024-04-02T04:59:05.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 4 2024-04-02T04:59:05.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.444+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 1 4 2024-04-02T04:59:05.449 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 10 client.26819.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 5 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f9de81ac3b0 0x7f9de81ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 5 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 2 5 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 4 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 10 client.26819.objecter _send_command 5 2024-04-02T04:59:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9de81b1600 con 0x7f9de81ac3b0 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f9de81ac3b0 0x7f9de81ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de81ac3b0 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f9de81ac3b0 0x7f9de81ae7a0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81b1600 con 0x7f9de81ac3b0 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 5: 0 ) 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 5 = system:0 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 4 2024-04-02T04:59:05.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.448+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 2 5 2024-04-02T04:59:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 10 client.26819.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 6 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 6 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de8108e20 osd=2 3 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 2 6 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 4 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 10 client.26819.objecter _send_command 6 2024-04-02T04:59:05.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de802d560 con 0x7f9de81ac3b0 2024-04-02T04:59:05.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de802d560 con 0x7f9de81ac3b0 2024-04-02T04:59:05.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 4 2024-04-02T04:59:05.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 2 6 2024-04-02T04:59:05.473 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 10 client.26819.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 7 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f9de81afe60 0x7f9de81b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 7 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 3 7 2024-04-02T04:59:05.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 4 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 10 client.26819.objecter _send_command 7 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9de81a9f10 con 0x7f9de81afe60 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.468+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f9de81afe60 0x7f9de81b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de81afe60 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f9de81afe60 0x7f9de81b02a0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81a9f10 con 0x7f9de81afe60 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 7: 0 ) 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 7 = system:0 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 4 2024-04-02T04:59:05.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.472+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 3 7 2024-04-02T04:59:05.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 10 client.26819.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 8 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 8 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81a9840 osd=3 3 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 3 8 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 4 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 10 client.26819.objecter _send_command 8 2024-04-02T04:59:05.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.488+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de81a9f10 con 0x7f9de81afe60 2024-04-02T04:59:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de81a9f10 con 0x7f9de81afe60 2024-04-02T04:59:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 4 2024-04-02T04:59:05.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 3 8 2024-04-02T04:59:05.494 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 10 client.26819.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 9 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f9de81b0fb0 0x7f9de81b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 9 2024-04-02T04:59:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 4 9 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 4 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 10 client.26819.objecter _send_command 9 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9de81b8630 con 0x7f9de81b0fb0 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f9de81b0fb0 0x7f9de81b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de81b0fb0 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.492+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f9de81b0fb0 0x7f9de81b57d0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.496+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81b8630 con 0x7f9de81b0fb0 2024-04-02T04:59:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.496+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 9: 0 ) 2024-04-02T04:59:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.496+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 9 = system:0 2024-04-02T04:59:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.496+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 4 2024-04-02T04:59:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.496+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 4 9 2024-04-02T04:59:05.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:05.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 10 client.26819.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 10 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 10 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81a9f10 osd=4 3 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 4 10 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 4 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 10 client.26819.objecter _send_command 10 2024-04-02T04:59:05.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.512+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de81b5d10 con 0x7f9de81b0fb0 2024-04-02T04:59:05.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de81b5d10 con 0x7f9de81b0fb0 2024-04-02T04:59:05.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 4 2024-04-02T04:59:05.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 4 10 2024-04-02T04:59:05.519 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 10 client.26819.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 11 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f9de81b6480 0x7f9de81b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 11 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 5 11 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 4 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 10 client.26819.objecter _send_command 11 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9de81bba20 con 0x7f9de81b6480 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f9de81b6480 0x7f9de81b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de81b6480 2024-04-02T04:59:05.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.516+0000 7f9de7fff700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f9de81b6480 0x7f9de81b68e0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.520+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81bba20 con 0x7f9de81b6480 2024-04-02T04:59:05.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.520+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 11: 0 ) 2024-04-02T04:59:05.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.520+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 11 = system:0 2024-04-02T04:59:05.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.520+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 4 2024-04-02T04:59:05.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.520+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 5 11 2024-04-02T04:59:05.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:05.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 10 client.26819.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 12 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 12 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81b5d10 osd=5 3 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 5 12 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 4 2024-04-02T04:59:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 10 client.26819.objecter _send_command 12 2024-04-02T04:59:05.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.540+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de81b6f30 con 0x7f9de81b6480 2024-04-02T04:59:05.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de81b6f30 con 0x7f9de81b6480 2024-04-02T04:59:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 4 2024-04-02T04:59:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 5 12 2024-04-02T04:59:05.546 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:59:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 10 client.26819.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 13 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f9de81b7650 0x7f9de81b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 13 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 6 13 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 4 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 10 client.26819.objecter _send_command 13 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9de81b8120 con 0x7f9de81b7650 2024-04-02T04:59:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f9de81b7650 0x7f9de81b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de81b7650 2024-04-02T04:59:05.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.544+0000 7f9ded66e700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f9de81b7650 0x7f9de81b7a90 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.548+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81b8120 con 0x7f9de81b7650 2024-04-02T04:59:05.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.548+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 13: 0 ) 2024-04-02T04:59:05.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.548+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 13 = system:0 2024-04-02T04:59:05.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.548+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 4 2024-04-02T04:59:05.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.548+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 6 13 2024-04-02T04:59:05.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 10 client.26819.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 14 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 14 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81b6f30 osd=6 3 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 6 14 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 4 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 10 client.26819.objecter _send_command 14 2024-04-02T04:59:05.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.564+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de81b8120 con 0x7f9de81b7650 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de81b8120 con 0x7f9de81b7650 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 4 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 6 14 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 10 client.26819.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:05.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 15 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f9de81b8290 0x7f9de81bee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 15 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 7 15 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 4 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 10 client.26819.objecter _send_command 15 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9de81c1ed0 con 0x7f9de81b8290 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f9de81b8290 0x7f9de81bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:05.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9de5ffb700 10 client.26819.objecter ms_handle_connect 0x7f9de81b8290 2024-04-02T04:59:05.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.568+0000 7f9dece6d700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f9de81b8290 0x7f9de81bee10 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:05.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.572+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9de81c1ed0 con 0x7f9de81b8290 2024-04-02T04:59:05.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.572+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 15: 0 ) 2024-04-02T04:59:05.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.572+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 15 = system:0 2024-04-02T04:59:05.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.572+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 4 2024-04-02T04:59:05.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.572+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 7 15 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 10 client.26819.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign -1 16 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 20 client.26819.objecter _get_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_remove -1 16 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 20 client.26819.objecter get_session s=0x7f9de81b8120 osd=7 3 2024-04-02T04:59:05.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 15 client.26819.objecter _session_command_op_assign 7 16 2024-04-02T04:59:05.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 4 2024-04-02T04:59:05.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 10 client.26819.objecter _send_command 16 2024-04-02T04:59:05.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.592+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9de81c0c60 con 0x7f9de81b8290 2024-04-02T04:59:05.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9de5ffb700 1 -- 172.21.15.69:0/3725599478 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9de81c0c60 con 0x7f9de81b8290 2024-04-02T04:59:05.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9de5ffb700 10 client.26819.objecter ms_dispatch 0x7f9de8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:05.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9de5ffb700 10 client.26819.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:05.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9de5ffb700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 4 2024-04-02T04:59:05.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9de5ffb700 15 client.26819.objecter _session_command_op_remove 7 16 2024-04-02T04:59:05.598 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:59:05.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 librados: watch_flush enter 2024-04-02T04:59:05.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 librados: watch_flush exit 2024-04-02T04:59:05.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.0 2024-04-02T04:59:05.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9de8062e50 msgr2=0x7f9de8063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f9de8062e50 0x7f9de8063230 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de819b610 osd=0 1 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.1 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9de802c9b0 msgr2=0x7f9de802cdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f9de802c9b0 0x7f9de802cdd0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108fa0 osd=1 1 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.2 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f9de81ac3b0 msgr2=0x7f9de81ae7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f9de81ac3b0 0x7f9de81ae7a0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de8108e20 osd=2 1 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.3 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f9de81afe60 msgr2=0x7f9de81b02a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f9de81afe60 0x7f9de81b02a0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9840 osd=3 1 2024-04-02T04:59:05.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.4 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f9de81b0fb0 msgr2=0x7f9de81b57d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9ded66e700 1 -- 172.21.15.69:0/3725599478 reap_dead start 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f9de81b0fb0 0x7f9de81b57d0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9ded66e700 1 -- 172.21.15.69:0/3725599478 reap_dead start 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81a9f10 osd=4 1 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.5 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f9de81b6480 msgr2=0x7f9de81b68e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f9de81b6480 0x7f9de81b68e0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b5d10 osd=5 1 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.6 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f9de81b7650 msgr2=0x7f9de81b7a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f9de81b7650 0x7f9de81b7a90 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b6f30 osd=6 1 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter close_session for osd.7 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f9de81b8290 msgr2=0x7f9de81bee10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f9de81b8290 0x7f9de81bee10 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter put_session s=0x7f9de81b8120 osd=7 1 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 20 client.26819.objecter shutdown clearing up homeless session... 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 10 client.26819.objecter successfully canceled tick 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9dd806f5e0 msgr2=0x7f9dd8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9dd806f5e0 0x7f9dd8071aa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00b7f0 tx=0x7f9ddc01a040 comp rx=0 tx=0).stop 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de8069f70 msgr2=0x7f9de819a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 --2- 172.21.15.69:0/3725599478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9de8069f70 0x7f9de819a2b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400c920 tx=0x7f9dd400cc30 comp rx=0 tx=0).stop 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9ded66e700 1 -- 172.21.15.69:0/3725599478 reap_dead start 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 shutdown_connections 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 >> 172.21.15.69:0/3725599478 conn(0x7f9de80771e0 msgr2=0x7f9de810ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:05.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 shutdown_connections 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 -- 172.21.15.69:0/3725599478 wait complete. 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:05.596+0000 7f9def0d1700 1 librados: shutdown 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:59:05.603 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:59:05.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:05.654 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.986080+0000 osd.3 (osd.3) 325 : cluster 0 1.4 scrub starts 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: cluster 2024-04-02T04:59:03.987617+0000 osd.3 (osd.3) 326 : cluster 0 1.4 scrub ok 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: cluster 2024-04-02T04:59:04.250050+0000 mgr.y (mgr.14538) 601 : cluster 0 pgmap v975: 302 pgs: 1 activating, 4 peering, 297 active+clean; 8.1 MiB data, 583 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s wr, 3 op/s; 1.1 MiB/s, 69 objects/s recovering 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: cluster 2024-04-02T04:59:04.479338+0000 osd.7 (osd.7) 137 : cluster 0 2.0 scrub starts 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: cluster 2024-04-02T04:59:04.490507+0000 osd.7 (osd.7) 138 : cluster 0 2.0 scrub ok 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: audit 2024-04-02T04:59:05.010074+0000 mon.a (mon.0) 3263 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:59:05.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: cluster 2024-04-02T04:59:05.019231+0000 mon.a (mon.0) 3264 : cluster 0 osdmap e647: 8 total, 8 up, 7 in 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: audit 2024-04-02T04:59:05.023462+0000 mon.c (mon.2) 1114 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: audit 2024-04-02T04:59:05.034821+0000 mon.a (mon.0) 3265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[21528]: audit 2024-04-02T04:59:05.404120+0000 mon.c (mon.2) 1115 : audit 0 from='client.? 172.21.15.69:0/3725599478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.986080+0000 osd.3 (osd.3) 325 : cluster 0 1.4 scrub starts 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: cluster 2024-04-02T04:59:03.987617+0000 osd.3 (osd.3) 326 : cluster 0 1.4 scrub ok 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: cluster 2024-04-02T04:59:04.250050+0000 mgr.y (mgr.14538) 601 : cluster 0 pgmap v975: 302 pgs: 1 activating, 4 peering, 297 active+clean; 8.1 MiB data, 583 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s wr, 3 op/s; 1.1 MiB/s, 69 objects/s recovering 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: cluster 2024-04-02T04:59:04.479338+0000 osd.7 (osd.7) 137 : cluster 0 2.0 scrub starts 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: cluster 2024-04-02T04:59:04.490507+0000 osd.7 (osd.7) 138 : cluster 0 2.0 scrub ok 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: audit 2024-04-02T04:59:05.010074+0000 mon.a (mon.0) 3263 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: cluster 2024-04-02T04:59:05.019231+0000 mon.a (mon.0) 3264 : cluster 0 osdmap e647: 8 total, 8 up, 7 in 2024-04-02T04:59:05.680 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: audit 2024-04-02T04:59:05.023462+0000 mon.c (mon.2) 1114 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:05.681 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: audit 2024-04-02T04:59:05.034821+0000 mon.a (mon.0) 3265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:05.681 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:05 smithi069 bash[28904]: audit 2024-04-02T04:59:05.404120+0000 mon.c (mon.2) 1115 : audit 0 from='client.? 172.21.15.69:0/3725599478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:05.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:05.782 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:05.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:05.915 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:05.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.986080+0000 osd.3 (osd.3) 325 : cluster 0 1.4 scrub starts 2024-04-02T04:59:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: cluster 2024-04-02T04:59:03.987617+0000 osd.3 (osd.3) 326 : cluster 0 1.4 scrub ok 2024-04-02T04:59:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: cluster 2024-04-02T04:59:04.250050+0000 mgr.y (mgr.14538) 601 : cluster 0 pgmap v975: 302 pgs: 1 activating, 4 peering, 297 active+clean; 8.1 MiB data, 583 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s wr, 3 op/s; 1.1 MiB/s, 69 objects/s recovering 2024-04-02T04:59:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: cluster 2024-04-02T04:59:04.479338+0000 osd.7 (osd.7) 137 : cluster 0 2.0 scrub starts 2024-04-02T04:59:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: cluster 2024-04-02T04:59:04.490507+0000 osd.7 (osd.7) 138 : cluster 0 2.0 scrub ok 2024-04-02T04:59:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: audit 2024-04-02T04:59:05.010074+0000 mon.a (mon.0) 3263 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:59:05.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: cluster 2024-04-02T04:59:05.019231+0000 mon.a (mon.0) 3264 : cluster 0 osdmap e647: 8 total, 8 up, 7 in 2024-04-02T04:59:05.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: audit 2024-04-02T04:59:05.023462+0000 mon.c (mon.2) 1114 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:05.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: audit 2024-04-02T04:59:05.034821+0000 mon.a (mon.0) 3265 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:05.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:05 smithi098 bash[23790]: audit 2024-04-02T04:59:05.404120+0000 mon.c (mon.2) 1115 : audit 0 from='client.? 172.21.15.69:0/3725599478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:06.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:06.048 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:06.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:06.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:06.306 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:06.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:06.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:06.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:06.695 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:06.798 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:06.826 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: cluster 2024-04-02T04:59:04.964958+0000 osd.3 (osd.3) 327 : cluster 0 6.6 deep-scrub starts 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: cluster 2024-04-02T04:59:04.967875+0000 osd.3 (osd.3) 328 : cluster 0 6.6 deep-scrub ok 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.024705+0000 mon.a (mon.0) 3266 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: audit 2024-04-02T04:59:06.026977+0000 mon.a (mon.0) 3267 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.034642+0000 mon.a (mon.0) 3268 : cluster 0 osdmap e648: 8 total, 8 up, 7 in 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: audit 2024-04-02T04:59:06.041249+0000 mon.c (mon.2) 1116 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[28904]: audit 2024-04-02T04:59:06.042970+0000 mon.a (mon.0) 3269 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-02T04:59:06.828 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: cluster 2024-04-02T04:59:04.964958+0000 osd.3 (osd.3) 327 : cluster 0 6.6 deep-scrub starts 2024-04-02T04:59:06.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: cluster 2024-04-02T04:59:04.967875+0000 osd.3 (osd.3) 328 : cluster 0 6.6 deep-scrub ok 2024-04-02T04:59:06.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.024705+0000 mon.a (mon.0) 3266 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:06.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: audit 2024-04-02T04:59:06.026977+0000 mon.a (mon.0) 3267 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:59:06.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.034642+0000 mon.a (mon.0) 3268 : cluster 0 osdmap e648: 8 total, 8 up, 7 in 2024-04-02T04:59:06.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: audit 2024-04-02T04:59:06.041249+0000 mon.c (mon.2) 1116 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-02T04:59:06.829 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:06 smithi069 bash[21528]: audit 2024-04-02T04:59:06.042970+0000 mon.a (mon.0) 3269 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-02T04:59:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:06.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:06.958 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:06.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: cluster 2024-04-02T04:59:04.964958+0000 osd.3 (osd.3) 327 : cluster 0 6.6 deep-scrub starts 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: cluster 2024-04-02T04:59:04.967875+0000 osd.3 (osd.3) 328 : cluster 0 6.6 deep-scrub ok 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.024705+0000 mon.a (mon.0) 3266 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: audit 2024-04-02T04:59:06.026977+0000 mon.a (mon.0) 3267 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.034642+0000 mon.a (mon.0) 3268 : cluster 0 osdmap e648: 8 total, 8 up, 7 in 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: audit 2024-04-02T04:59:06.041249+0000 mon.c (mon.2) 1116 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-02T04:59:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:06 smithi098 bash[23790]: audit 2024-04-02T04:59:06.042970+0000 mon.a (mon.0) 3269 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-02T04:59:07.062 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:07.086 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:07.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:07.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:07.216 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:07.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:07.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:07.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:07.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:07.582 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:05.947381+0000 osd.3 (osd.3) 329 : cluster 0 3.7 scrub starts 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:05.949854+0000 osd.3 (osd.3) 330 : cluster 0 3.7 scrub ok 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:05.995077+0000 osd.0 (osd.0) 331 : cluster 0 10.4 scrub starts 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:05.997200+0000 osd.0 (osd.0) 332 : cluster 0 10.4 scrub ok 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.250936+0000 mgr.y (mgr.14538) 602 : cluster 0 pgmap v978: 302 pgs: 1 activating, 4 peering, 297 active+clean; 8.1 MiB data, 583 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s wr, 3 op/s; 1.1 MiB/s, 69 objects/s recovering 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.430814+0000 osd.7 (osd.7) 139 : cluster 0 2.1c scrub starts 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.432887+0000 osd.7 (osd.7) 140 : cluster 0 2.1c scrub ok 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: audit 2024-04-02T04:59:07.029102+0000 mon.a (mon.0) 3270 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.033537+0000 mon.a (mon.0) 3271 : cluster 0 osdmap e649: 8 total, 8 up, 7 in 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: audit 2024-04-02T04:59:07.109059+0000 mon.c (mon.2) 1117 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:07.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[21528]: audit 2024-04-02T04:59:07.109448+0000 mon.a (mon.0) 3272 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:05.947381+0000 osd.3 (osd.3) 329 : cluster 0 3.7 scrub starts 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:05.949854+0000 osd.3 (osd.3) 330 : cluster 0 3.7 scrub ok 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:05.995077+0000 osd.0 (osd.0) 331 : cluster 0 10.4 scrub starts 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:05.997200+0000 osd.0 (osd.0) 332 : cluster 0 10.4 scrub ok 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.250936+0000 mgr.y (mgr.14538) 602 : cluster 0 pgmap v978: 302 pgs: 1 activating, 4 peering, 297 active+clean; 8.1 MiB data, 583 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s wr, 3 op/s; 1.1 MiB/s, 69 objects/s recovering 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.430814+0000 osd.7 (osd.7) 139 : cluster 0 2.1c scrub starts 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.432887+0000 osd.7 (osd.7) 140 : cluster 0 2.1c scrub ok 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: audit 2024-04-02T04:59:07.029102+0000 mon.a (mon.0) 3270 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.033537+0000 mon.a (mon.0) 3271 : cluster 0 osdmap e649: 8 total, 8 up, 7 in 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: audit 2024-04-02T04:59:07.109059+0000 mon.c (mon.2) 1117 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:07.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:07 smithi069 bash[28904]: audit 2024-04-02T04:59:07.109448+0000 mon.a (mon.0) 3272 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:07.610 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:07.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:05.947381+0000 osd.3 (osd.3) 329 : cluster 0 3.7 scrub starts 2024-04-02T04:59:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:05.949854+0000 osd.3 (osd.3) 330 : cluster 0 3.7 scrub ok 2024-04-02T04:59:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:05.995077+0000 osd.0 (osd.0) 331 : cluster 0 10.4 scrub starts 2024-04-02T04:59:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:05.997200+0000 osd.0 (osd.0) 332 : cluster 0 10.4 scrub ok 2024-04-02T04:59:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.250936+0000 mgr.y (mgr.14538) 602 : cluster 0 pgmap v978: 302 pgs: 1 activating, 4 peering, 297 active+clean; 8.1 MiB data, 583 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s wr, 3 op/s; 1.1 MiB/s, 69 objects/s recovering 2024-04-02T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.430814+0000 osd.7 (osd.7) 139 : cluster 0 2.1c scrub starts 2024-04-02T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.432887+0000 osd.7 (osd.7) 140 : cluster 0 2.1c scrub ok 2024-04-02T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: audit 2024-04-02T04:59:07.029102+0000 mon.a (mon.0) 3270 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-02T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.033537+0000 mon.a (mon.0) 3271 : cluster 0 osdmap e649: 8 total, 8 up, 7 in 2024-04-02T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: audit 2024-04-02T04:59:07.109059+0000 mon.c (mon.2) 1117 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:07 smithi098 bash[23790]: audit 2024-04-02T04:59:07.109448+0000 mon.a (mon.0) 3272 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:07.740 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:07.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:07.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:07.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:08.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:08.128 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:08.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:08.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:08.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:08.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:08.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:08.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:08.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.973704+0000 osd.3 (osd.3) 331 : cluster 0 10.1b scrub starts 2024-04-02T04:59:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:06.976082+0000 osd.3 (osd.3) 332 : cluster 0 10.1b scrub ok 2024-04-02T04:59:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.011153+0000 osd.0 (osd.0) 333 : cluster 0 2.1f scrub starts 2024-04-02T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.012979+0000 osd.0 (osd.0) 334 : cluster 0 2.1f scrub ok 2024-04-02T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.460567+0000 osd.7 (osd.7) 141 : cluster 0 40.19 scrub starts 2024-04-02T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.462489+0000 osd.7 (osd.7) 142 : cluster 0 40.19 scrub ok 2024-04-02T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.983326+0000 osd.1 (osd.1) 343 : cluster 0 3.19 scrub starts 2024-04-02T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:07.986814+0000 osd.1 (osd.1) 344 : cluster 0 3.19 scrub ok 2024-04-02T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: audit 2024-04-02T04:59:08.041432+0000 mon.a (mon.0) 3273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:59:08.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.047460+0000 mon.a (mon.0) 3274 : cluster 0 osdmap e650: 8 total, 8 up, 7 in 2024-04-02T04:59:08.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: audit 2024-04-02T04:59:08.050325+0000 mon.c (mon.2) 1118 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:08.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:08 smithi098 bash[23790]: audit 2024-04-02T04:59:08.051600+0000 mon.a (mon.0) 3275 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:08.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:08.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.973704+0000 osd.3 (osd.3) 331 : cluster 0 10.1b scrub starts 2024-04-02T04:59:08.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:06.976082+0000 osd.3 (osd.3) 332 : cluster 0 10.1b scrub ok 2024-04-02T04:59:08.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.011153+0000 osd.0 (osd.0) 333 : cluster 0 2.1f scrub starts 2024-04-02T04:59:08.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.012979+0000 osd.0 (osd.0) 334 : cluster 0 2.1f scrub ok 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.460567+0000 osd.7 (osd.7) 141 : cluster 0 40.19 scrub starts 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.462489+0000 osd.7 (osd.7) 142 : cluster 0 40.19 scrub ok 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.983326+0000 osd.1 (osd.1) 343 : cluster 0 3.19 scrub starts 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:07.986814+0000 osd.1 (osd.1) 344 : cluster 0 3.19 scrub ok 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: audit 2024-04-02T04:59:08.041432+0000 mon.a (mon.0) 3273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.047460+0000 mon.a (mon.0) 3274 : cluster 0 osdmap e650: 8 total, 8 up, 7 in 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: audit 2024-04-02T04:59:08.050325+0000 mon.c (mon.2) 1118 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:08.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21528]: audit 2024-04-02T04:59:08.051600+0000 mon.a (mon.0) 3275 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.973704+0000 osd.3 (osd.3) 331 : cluster 0 10.1b scrub starts 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:06.976082+0000 osd.3 (osd.3) 332 : cluster 0 10.1b scrub ok 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.011153+0000 osd.0 (osd.0) 333 : cluster 0 2.1f scrub starts 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.012979+0000 osd.0 (osd.0) 334 : cluster 0 2.1f scrub ok 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.460567+0000 osd.7 (osd.7) 141 : cluster 0 40.19 scrub starts 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.462489+0000 osd.7 (osd.7) 142 : cluster 0 40.19 scrub ok 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.983326+0000 osd.1 (osd.1) 343 : cluster 0 3.19 scrub starts 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:07.986814+0000 osd.1 (osd.1) 344 : cluster 0 3.19 scrub ok 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: audit 2024-04-02T04:59:08.041432+0000 mon.a (mon.0) 3273 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]': finished 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.047460+0000 mon.a (mon.0) 3274 : cluster 0 osdmap e650: 8 total, 8 up, 7 in 2024-04-02T04:59:08.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: audit 2024-04-02T04:59:08.050325+0000 mon.c (mon.2) 1118 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:08.841 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[28904]: audit 2024-04-02T04:59:08.051600+0000 mon.a (mon.0) 3275 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]: dispatch 2024-04-02T04:59:08.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:08.906 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:08.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:09.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:09.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:09.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:09.162 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:59:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:59:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:59:09.166 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:09.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:09.293 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:09.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:09.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:09.549 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:09.652 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:09.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:09.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:09.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.252020+0000 mgr.y (mgr.14538) 603 : cluster 0 pgmap v981: 302 pgs: 302 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.485512+0000 osd.2 (osd.2) 309 : cluster 0 43.3 scrub starts 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.487901+0000 osd.2 (osd.2) 310 : cluster 0 43.3 scrub ok 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.947113+0000 osd.1 (osd.1) 345 : cluster 0 6.1d scrub starts 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.948564+0000 osd.1 (osd.1) 346 : cluster 0 6.1d scrub ok 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: audit 2024-04-02T04:59:09.052473+0000 mon.a (mon.0) 3276 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]': finished 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.059937+0000 mon.a (mon.0) 3277 : cluster 0 osdmap e651: 8 total, 8 up, 7 in 2024-04-02T04:59:09.922 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.252020+0000 mgr.y (mgr.14538) 603 : cluster 0 pgmap v981: 302 pgs: 302 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:09.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.485512+0000 osd.2 (osd.2) 309 : cluster 0 43.3 scrub starts 2024-04-02T04:59:09.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.487901+0000 osd.2 (osd.2) 310 : cluster 0 43.3 scrub ok 2024-04-02T04:59:09.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.947113+0000 osd.1 (osd.1) 345 : cluster 0 6.1d scrub starts 2024-04-02T04:59:09.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.948564+0000 osd.1 (osd.1) 346 : cluster 0 6.1d scrub ok 2024-04-02T04:59:09.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: audit 2024-04-02T04:59:09.052473+0000 mon.a (mon.0) 3276 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]': finished 2024-04-02T04:59:09.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:09 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.059937+0000 mon.a (mon.0) 3277 : cluster 0 osdmap e651: 8 total, 8 up, 7 in 2024-04-02T04:59:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:09.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:09.952 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:59:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.252020+0000 mgr.y (mgr.14538) 603 : cluster 0 pgmap v981: 302 pgs: 302 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.485512+0000 osd.2 (osd.2) 309 : cluster 0 43.3 scrub starts 2024-04-02T04:59:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.487901+0000 osd.2 (osd.2) 310 : cluster 0 43.3 scrub ok 2024-04-02T04:59:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.947113+0000 osd.1 (osd.1) 345 : cluster 0 6.1d scrub starts 2024-04-02T04:59:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.948564+0000 osd.1 (osd.1) 346 : cluster 0 6.1d scrub ok 2024-04-02T04:59:09.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: audit 2024-04-02T04:59:09.052473+0000 mon.a (mon.0) 3276 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-100"}]': finished 2024-04-02T04:59:09.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:09 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.059937+0000 mon.a (mon.0) 3277 : cluster 0 osdmap e651: 8 total, 8 up, 7 in 2024-04-02T04:59:10.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:10.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:10.198 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:10.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:10.320 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14964.0:2 24.1b 24:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-02T04:59:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:24.961827+0000", 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 585.37837931900003, 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.46321732, 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14964", 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1323876815", 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961827+0000", 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961827+0000", 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961829+0000", 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6270000000000001e-06 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961836+0000", 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0600000000000002e-06 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961850+0000", 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.361e-05 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961861+0000", 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1433999999999999e-05 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961942+0000", 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0846999999999994e-05 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:24.961953+0000", 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0645999999999999e-05 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.332234+0000", 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3702814640000001 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.332313+0000", 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8484000000000003e-05 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 0,5", 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.332396+0000", 2024-04-02T04:59:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3684999999999996e-05 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.348531+0000", 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016134681000000001 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.386005+0000", 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.037473943000000003 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.424196+0000", 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.038191267000000001 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.424253+0000", 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7077000000000003e-05 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.425045+0000", 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079149500000000002 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24911.0:2 33.6 33:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:25.087363+0000", 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 585.25284417199998, 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4477140770000001, 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24911", 2024-04-02T04:59:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1160901141", 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087363+0000", 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087363+0000", 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087364+0000", 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5099999999999999e-06 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087370+0000", 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3330000000000002e-06 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087384+0000", 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.332e-05 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087395+0000", 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1314999999999999e-05 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087474+0000", 2024-04-02T04:59:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8588e-05 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:25.087483+0000", 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1099999999999992e-06 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.416854+0000", 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.329371428 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.416923+0000", 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8962999999999997e-05 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 3,7", 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.416999+0000", 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6223999999999994e-05 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.418994+0000", 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0019949299999999998 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.429425+0000", 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010430457000000001 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.534982+0000", 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.105557161 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.535033+0000", 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1378000000000002e-05 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.535077+0000", 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3359999999999998e-05 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.24917.0:2 37.1a e138/137 37:5c6776db:::0.obj:head v 138'1, mlcod=0'0)", 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:26.322430+0000", 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 584.01777699800004, 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.082681258999999993, 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322430+0000", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322430+0000", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322430+0000", 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1799999999999999e-07 2024-04-02T04:59:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322438+0000", 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5750000000000003e-06 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322443+0000", 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5459999999999999e-06 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.322449+0000", 2024-04-02T04:59:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7780000000000004e-06 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.364560+0000", 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.042110598999999999 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.384278+0000", 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019718907000000001 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.405081+0000", 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.020802646000000001 2024-04-02T04:59:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:26.405111+0000", 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9989999999999999e-05 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24758.0:1987 15.8 15:1569c6cd:test-rados-api-smithi069-72304-15::foo:head [setxattr a969 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:28.610822+0000", 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 581.729384321, 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.10405993700000001, 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24758", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/50059295", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1987 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610822+0000", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610822+0000", 2024-04-02T04:59:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610824+0000", 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.203e-06 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610829+0000", 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7540000000000003e-06 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610843+0000", 2024-04-02T04:59:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3385e-05 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610852+0000", 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0240000000000003e-06 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610908+0000", 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5968999999999999e-05 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.610971+0000", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2914000000000001e-05 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 0,3", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.611089+0000", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011844400000000001 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.618507+0000", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074181940000000004 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.632030+0000", 2024-04-02T04:59:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013522624 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.714782+0000", 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.082751530000000004 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.714840+0000", 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8190999999999999e-05 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.714882+0000", 2024-04-02T04:59:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2704999999999999e-05 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15204.0:3 22.16 22:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:28.633192+0000", 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 581.70701525499999, 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.090455912999999999, 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15204", 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/230979790", 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633192+0000", 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633192+0000", 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633194+0000", 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.962e-06 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633200+0000", 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3840000000000002e-06 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633214+0000", 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4236000000000001e-05 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633223+0000", 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1889999999999996e-06 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633283+0000", 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9862999999999999e-05 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633370+0000", 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6494000000000002e-05 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 5,7", 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.633527+0000", 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015719599999999999 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.635231+0000", 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001704344 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.650992+0000", 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015761112000000001 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.723556+0000", 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.072563427999999999 2024-04-02T04:59:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.723606+0000", 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9824999999999998e-05 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:28.723647+0000", 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1879999999999999e-05 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15162.0:2 48.0 48:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:29.113513+0000", 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 581.22669398599999, 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.469572318, 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15162", 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2993715898", 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113513+0000", 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113513+0000", 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113515+0000", 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.95e-06 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113521+0000", 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5420000000000002e-06 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113532+0000", 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1094000000000001e-05 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113556+0000", 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3907999999999999e-05 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113621+0000", 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5018999999999995e-05 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.113630+0000", 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1370000000000008e-06 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.574806+0000", 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4611755199999998 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.574905+0000", 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8536000000000007e-05 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,4", 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.575035+0000", 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013076200000000001 2024-04-02T04:59:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.576451+0000", 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0014158059999999999 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.576765+0000", 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00031353799999999998 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.582981+0000", 2024-04-02T04:59:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062163979999999997 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.583042+0000", 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0656999999999999e-05 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:31.583085+0000", 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3451000000000001e-05 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.24758.0:2074 15.7 e140/135 15:e36060b3:test-rados-api-smithi069-72304-17::big3:head v 140'3, mlcod=140'1)", 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:29.591620+0000", 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 580.74858713900005, 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.090489785000000003, 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.591620+0000", 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.591620+0000", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.591622+0000", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6340000000000001e-06 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.617162+0000", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.025540041999999999 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.617196+0000", 2024-04-02T04:59:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3318999999999997e-05 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.617211+0000", 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5058000000000001e-05 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.617330+0000", 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000119181 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.617373+0000", 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2814e-05 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.682047+0000", 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.064673839999999996 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.682109+0000", 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2897000000000004e-05 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24794.0:1 49.2s0 49:5a4cc9bb:test-rados-api-smithi069-72439-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:29.755679+0000", 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 580.58452748800005, 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7159205530000001, 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24794", 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2962558957", 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.755679+0000", 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.755679+0000", 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.755681+0000", 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8610000000000001e-06 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.755690+0000", 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0459999999999994e-06 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.755703+0000", 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2925000000000001e-05 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.755715+0000", 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1980999999999999e-05 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.816122+0000", 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.060407108000000001 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:29.816127+0000", 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3089999999999997e-06 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:32.457849+0000", 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6417225909999997 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:32.457920+0000", 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0732e-05 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:32.458229+0000", 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030917599999999998 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:32.461484+0000", 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0032548389999999998 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:32.471529+0000", 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010044730999999999 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:32.471600+0000", 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1254000000000006e-05 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25024.0:2 65.15 65:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:49:37.377652+0000", 2024-04-02T04:59:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 572.96255521399996, 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.27391043199999998, 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25024", 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1321485616", 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377652+0000", 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377652+0000", 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377653+0000", 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8419999999999999e-06 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377661+0000", 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3599999999999998e-06 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377679+0000", 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7938e-05 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377697+0000", 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8159e-05 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377778+0000", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1341000000000006e-05 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.377788+0000", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0169999999999999e-05 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.604219+0000", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.22643058699999999 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.604225+0000", 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9519999999999999e-06 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.645726+0000", 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.041501478000000001 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.647003+0000", 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0012764300000000001 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,7", 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.647102+0000", 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9160000000000006e-05 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.649049+0000", 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001946894 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.649087+0000", 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7821999999999998e-05 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.651472+0000", 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002385388 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.651522+0000", 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9418e-05 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:49:37.651562+0000", 2024-04-02T04:59:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0493e-05 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15462.0:452 115.5s0 115:a5cfcc41:::451:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:04.281255+0000", 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 546.05895223300001, 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.42493462799999998, 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15462", 2024-04-02T04:59:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1621702341", 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 452 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281255+0000", 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281255+0000", 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281255+0000", 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.09e-07 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281259+0000", 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1169999999999997e-06 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281269+0000", 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6290000000000005e-06 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281278+0000", 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1230000000000006e-06 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281333+0000", 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4752e-05 2024-04-02T04:59:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281432+0000", 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9147000000000005e-05 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.281706+0000", 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00027407899999999999 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.282010+0000", 2024-04-02T04:59:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030401700000000001 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.706133+0000", 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.42412256399999998 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:04.706189+0000", 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6391000000000003e-05 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "pg_update_log_missing(125.1e epoch 168/167 rep_tid 552 entries 168'3 (0'0) error 125:79d83b4f:::test_97:head by client.25553.0:104 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 168'1)", 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:06.199042+0000", 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 544.14116506200003, 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.24450266000000001, 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.199042+0000", 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.199042+0000", 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.199042+0000", 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6000000000000003e-07 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.199050+0000", 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9419999999999995e-06 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.199065+0000", 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5116999999999999e-05 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.199074+0000", 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.5539999999999998e-06 2024-04-02T04:59:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.442625+0000", 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.24355088499999999 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.443544+0000", 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00091950199999999999 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25568.0:346 128.b 128:d29485c3:::test_339:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e169)", 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:07.938045+0000", 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 542.40216197999996, 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.32665124400000001, 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25568", 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3776732179", 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 346 2024-04-02T04:59:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.938045+0000", 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.938045+0000", 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.938046+0000", 2024-04-02T04:59:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.85e-07 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.938050+0000", 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3309999999999997e-06 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.938059+0000", 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7050000000000005e-06 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.938064+0000", 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3329999999999996e-06 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.262500+0000", 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.32443627800000002 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.262666+0000", 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000165446 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.264696+0000", 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002030366 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25604.0:2 134.7 134:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:10.419038+0000", 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 539.92116900300005, 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.092686536999999999, 2024-04-02T04:59:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25604", 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1233792253", 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419038+0000", 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419038+0000", 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419039+0000", 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6390000000000001e-06 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419047+0000", 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4429999999999997e-06 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419065+0000", 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7948000000000002e-05 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419078+0000", 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2695e-05 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.414 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419141+0000", 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3245999999999997e-05 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.419150+0000", 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3789999999999995e-06 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.415 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.436738+0000", 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017587375999999998 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.436860+0000", 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000122805 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4", 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.437006+0000", 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014594799999999999 2024-04-02T04:59:10.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.455896+0000", 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0188901 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.511581+0000", 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.055684422999999997 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.417 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.511640+0000", 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9311000000000002e-05 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:10.511724+0000", 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4223999999999999e-05 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15615.0:2 148.0 148:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-04-02T04:59:10.418 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:17.807321+0000", 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 532.53288602800001, 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.673734897, 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15615", 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3893530562", 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.419 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807321+0000", 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807321+0000", 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807323+0000", 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8080000000000001e-06 2024-04-02T04:59:10.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807330+0000", 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.43e-06 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807343+0000", 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3115999999999999e-05 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807355+0000", 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1416999999999999e-05 2024-04-02T04:59:10.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807423+0000", 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8720999999999995e-05 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:17.807434+0000", 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1187e-05 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.422 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.474776+0000", 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.66734140900000005 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.474844+0000", 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.7933000000000002e-05 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 5", 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.475007+0000", 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016278199999999999 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.423 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.480396+0000", 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053895699999999998 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.480970+0000", 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00057374699999999999 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.481002+0000", 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1832999999999998e-05 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.424 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:18.481056+0000", 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3943999999999997e-05 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15585.0:9 145.2s0 145:5afd9e38:test-rados-api-smithi069-72378-13::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:22.464696+0000", 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 527.87551031299995, 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.085786662999999999, 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.425 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15585", 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1438082132", 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 9 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.464696+0000", 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.426 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.464696+0000", 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.464698+0000", 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9099999999999999e-06 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.464707+0000", 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9269999999999994e-06 2024-04-02T04:59:10.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.464729+0000", 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2118000000000001e-05 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.464746+0000", 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6892e-05 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.428 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.465013+0000", 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026651300000000001 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.465021+0000", 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0499999999999992e-06 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.530357+0000", 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.065335602000000007 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.429 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.530956+0000", 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00059980899999999998 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.531386+0000", 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000430121 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.538956+0000", 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0075697840000000004 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.550273+0000", 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011316366 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:22.550483+0000", 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000210571 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.431 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15849.0:1 196.7s0 196:ed3440d3:test-rados-api-smithi069-72389-17::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:51.752770+0000", 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 498.587436776, 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.027444099, 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15849", 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1653232639", 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.432 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.752770+0000", 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.752770+0000", 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.752772+0000", 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7090000000000001e-06 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.752779+0000", 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8549999999999996e-06 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.752789+0000", 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.075e-05 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.752799+0000", 2024-04-02T04:59:10.434 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0156000000000001e-05 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.759576+0000", 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006776252 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:51.759581+0000", 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9219999999999996e-06 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:52.773889+0000", 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.014308537 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:52.774129+0000", 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000239297 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:52.774782+0000", 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00065347300000000001 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.436 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:52.775585+0000", 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080323799999999996 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:52.780112+0000", 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0045271520000000004 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:52.780214+0000", 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101758 2024-04-02T04:59:10.437 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.26093.0:1 226.6s1 226:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:51:33.780579+0000", 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 456.55962758999999, 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.96143569699999998, 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.438 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.26093", 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1657029577", 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780579+0000", 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780579+0000", 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780581+0000", 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5400000000000001e-06 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780587+0000", 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6279999999999999e-06 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.440 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780597+0000", 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8160000000000005e-06 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780608+0000", 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0924000000000001e-05 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780696+0000", 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7546000000000005e-05 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.441 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:33.780699+0000", 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5389999999999999e-06 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:34.731180+0000", 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.95048082899999997 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:34.731346+0000", 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016606100000000001 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.442 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:34.731783+0000", 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00043644499999999999 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:34.741576+0000", 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0097932980000000006 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:34.741651+0000", 2024-04-02T04:59:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5630999999999995e-05 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:34.742015+0000", 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00036343999999999998 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.26309.0:1 258.3s1 258:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-04-02T04:59:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:52:37.177884+0000", 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 393.16232308899998, 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0395663660000001, 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.26309", 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1225121733", 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177884+0000", 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177884+0000", 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177885+0000", 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3549999999999999e-06 2024-04-02T04:59:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177891+0000", 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5990000000000001e-06 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177901+0000", 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0574000000000001e-05 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177909+0000", 2024-04-02T04:59:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2980000000000003e-06 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177963+0000", 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4012999999999999e-05 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.177966+0000", 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5049999999999999e-06 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.214387+0000", 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0364213470000001 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.214492+0000", 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000104845 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.214787+0000", 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00029472599999999998 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.215401+0000", 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00061359499999999998 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.217340+0000", 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0019392999999999999 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.217450+0000", 2024-04-02T04:59:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000110209 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.26411.0:1 273.5s0 273:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e362)", 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:21.908879+0000", 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 348.43132787500002, 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.022950612, 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T04:59:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.26411", 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3582086137", 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.908879+0000", 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.908879+0000", 2024-04-02T04:59:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.908881+0000", 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1780000000000002e-06 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.908888+0000", 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2429999999999999e-06 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.908902+0000", 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3331e-05 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.908918+0000", 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6334999999999999e-05 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.910380+0000", 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0014618669999999999 2024-04-02T04:59:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:21.910385+0000", 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9139999999999999e-06 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:22.931607+0000", 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.021222015 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:22.931830+0000", 2024-04-02T04:59:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022272899999999999 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.16386.0:1 282.2s0 282:5a7a12bd:::oid_SimpleStatPPNS_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:54:00.912917+0000", 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 309.42728979600003, 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0475820650000001, 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T04:59:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.16386", 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/778228701", 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.912917+0000", 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T04:59:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.912917+0000", 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.912919+0000", 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.663e-06 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.912925+0000", 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3060000000000003e-06 2024-04-02T04:59:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.912935+0000", 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0071999999999999e-05 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.912947+0000", 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1953000000000001e-05 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.913071+0000", 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012448299999999999 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:00.913078+0000", 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5860000000000001e-06 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:01.952248+0000", 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0391702789999999 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T04:59:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:01.953128+0000", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00087930699999999996 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:01.954491+0000", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001363834 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:01.956029+0000", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001537131 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:01.960352+0000", 2024-04-02T04:59:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043233710000000003 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:54:01.960499+0000", 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014708000000000001 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:10.462 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:59:10.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:10.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:10.615 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:10.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:10.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.920495+0000 osd.0 (osd.0) 335 : cluster 0 3.1b scrub starts 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:08.922762+0000 osd.0 (osd.0) 336 : cluster 0 3.1b scrub ok 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.135784+0000 osd.4 (osd.4) 344 : cluster 0 3.9 scrub starts 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.140572+0000 osd.4 (osd.4) 345 : cluster 0 3.9 scrub ok 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.511310+0000 osd.7 (osd.7) 143 : cluster 0 3.f scrub starts 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.515117+0000 osd.7 (osd.7) 144 : cluster 0 3.f scrub ok 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.937273+0000 osd.1 (osd.1) 347 : cluster 0 2.6 scrub starts 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.939777+0000 osd.1 (osd.1) 348 : cluster 0 2.6 scrub ok 2024-04-02T04:59:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:10 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.060847+0000 mon.a (mon.0) 3278 : cluster 0 osdmap e652: 8 total, 8 up, 7 in 2024-04-02T04:59:10.746 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.920495+0000 osd.0 (osd.0) 335 : cluster 0 3.1b scrub starts 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:08.922762+0000 osd.0 (osd.0) 336 : cluster 0 3.1b scrub ok 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.135784+0000 osd.4 (osd.4) 344 : cluster 0 3.9 scrub starts 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.140572+0000 osd.4 (osd.4) 345 : cluster 0 3.9 scrub ok 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.511310+0000 osd.7 (osd.7) 143 : cluster 0 3.f scrub starts 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.515117+0000 osd.7 (osd.7) 144 : cluster 0 3.f scrub ok 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.937273+0000 osd.1 (osd.1) 347 : cluster 0 2.6 scrub starts 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.939777+0000 osd.1 (osd.1) 348 : cluster 0 2.6 scrub ok 2024-04-02T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.060847+0000 mon.a (mon.0) 3278 : cluster 0 osdmap e652: 8 total, 8 up, 7 in 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.920495+0000 osd.0 (osd.0) 335 : cluster 0 3.1b scrub starts 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:08.922762+0000 osd.0 (osd.0) 336 : cluster 0 3.1b scrub ok 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.135784+0000 osd.4 (osd.4) 344 : cluster 0 3.9 scrub starts 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.140572+0000 osd.4 (osd.4) 345 : cluster 0 3.9 scrub ok 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.511310+0000 osd.7 (osd.7) 143 : cluster 0 3.f scrub starts 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.515117+0000 osd.7 (osd.7) 144 : cluster 0 3.f scrub ok 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.937273+0000 osd.1 (osd.1) 347 : cluster 0 2.6 scrub starts 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.939777+0000 osd.1 (osd.1) 348 : cluster 0 2.6 scrub ok 2024-04-02T04:59:10.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:10 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.060847+0000 mon.a (mon.0) 3278 : cluster 0 osdmap e652: 8 total, 8 up, 7 in 2024-04-02T04:59:10.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:10.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:10.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:10.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:11.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:11.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:11.265 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:11.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:11.394 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:11.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:11.523 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:11.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:11.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.981435+0000 osd.3 (osd.3) 333 : cluster 0 2.19 scrub starts 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:09.983298+0000 osd.3 (osd.3) 334 : cluster 0 2.19 scrub ok 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.129088+0000 osd.4 (osd.4) 346 : cluster 0 5.1 deep-scrub starts 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.151685+0000 osd.4 (osd.4) 347 : cluster 0 5.1 deep-scrub ok 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.252862+0000 mgr.y (mgr.14538) 604 : cluster 0 pgmap v984: 270 pgs: 270 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.458981+0000 osd.2 (osd.2) 311 : cluster 0 4.f scrub starts 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.460368+0000 osd.2 (osd.2) 312 : cluster 0 4.f scrub ok 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.881724+0000 osd.6 (osd.6) 285 : cluster 0 43.0 scrub starts 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:10.883632+0000 osd.6 (osd.6) 286 : cluster 0 43.0 scrub ok 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: cluster 2024-04-02T04:59:11.071471+0000 mon.a (mon.0) 3279 : cluster 0 osdmap e653: 8 total, 8 up, 7 in 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: audit 2024-04-02T04:59:11.076921+0000 mon.c (mon.2) 1119 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:11.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:11 smithi098 bash[23790]: audit 2024-04-02T04:59:11.089023+0000 mon.a (mon.0) 3280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:11.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.981435+0000 osd.3 (osd.3) 333 : cluster 0 2.19 scrub starts 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:09.983298+0000 osd.3 (osd.3) 334 : cluster 0 2.19 scrub ok 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.129088+0000 osd.4 (osd.4) 346 : cluster 0 5.1 deep-scrub starts 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.151685+0000 osd.4 (osd.4) 347 : cluster 0 5.1 deep-scrub ok 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.252862+0000 mgr.y (mgr.14538) 604 : cluster 0 pgmap v984: 270 pgs: 270 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.458981+0000 osd.2 (osd.2) 311 : cluster 0 4.f scrub starts 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.460368+0000 osd.2 (osd.2) 312 : cluster 0 4.f scrub ok 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.881724+0000 osd.6 (osd.6) 285 : cluster 0 43.0 scrub starts 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:10.883632+0000 osd.6 (osd.6) 286 : cluster 0 43.0 scrub ok 2024-04-02T04:59:11.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: cluster 2024-04-02T04:59:11.071471+0000 mon.a (mon.0) 3279 : cluster 0 osdmap e653: 8 total, 8 up, 7 in 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: audit 2024-04-02T04:59:11.076921+0000 mon.c (mon.2) 1119 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[21528]: audit 2024-04-02T04:59:11.089023+0000 mon.a (mon.0) 3280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.981435+0000 osd.3 (osd.3) 333 : cluster 0 2.19 scrub starts 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:09.983298+0000 osd.3 (osd.3) 334 : cluster 0 2.19 scrub ok 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.129088+0000 osd.4 (osd.4) 346 : cluster 0 5.1 deep-scrub starts 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.151685+0000 osd.4 (osd.4) 347 : cluster 0 5.1 deep-scrub ok 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.252862+0000 mgr.y (mgr.14538) 604 : cluster 0 pgmap v984: 270 pgs: 270 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.458981+0000 osd.2 (osd.2) 311 : cluster 0 4.f scrub starts 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.460368+0000 osd.2 (osd.2) 312 : cluster 0 4.f scrub ok 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.881724+0000 osd.6 (osd.6) 285 : cluster 0 43.0 scrub starts 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:10.883632+0000 osd.6 (osd.6) 286 : cluster 0 43.0 scrub ok 2024-04-02T04:59:11.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: cluster 2024-04-02T04:59:11.071471+0000 mon.a (mon.0) 3279 : cluster 0 osdmap e653: 8 total, 8 up, 7 in 2024-04-02T04:59:11.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: audit 2024-04-02T04:59:11.076921+0000 mon.c (mon.2) 1119 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:11.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:11 smithi069 bash[28904]: audit 2024-04-02T04:59:11.089023+0000 mon.a (mon.0) 3280 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:11.783 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:11.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:11.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:12.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:12.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:12.173 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:12.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:12.301 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:12.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:12.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:12.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:12.612 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-02T04:59:12.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:12.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:12.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:12.911 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:12.934 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:12.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:13.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:13.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:13.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:13.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: cluster 2024-04-02T04:59:11.931396+0000 osd.1 (osd.1) 349 : cluster 0 4.1c scrub starts 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: cluster 2024-04-02T04:59:11.933841+0000 osd.1 (osd.1) 350 : cluster 0 4.1c scrub ok 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: cluster 2024-04-02T04:59:11.993618+0000 osd.3 (osd.3) 335 : cluster 0 102.1e scrub starts 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: cluster 2024-04-02T04:59:11.996484+0000 osd.3 (osd.3) 336 : cluster 0 102.1e scrub ok 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: audit 2024-04-02T04:59:12.067395+0000 mon.a (mon.0) 3281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: cluster 2024-04-02T04:59:12.079834+0000 mon.a (mon.0) 3282 : cluster 0 osdmap e654: 8 total, 8 up, 7 in 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: audit 2024-04-02T04:59:12.082385+0000 mon.c (mon.2) 1120 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:59:13.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: audit 2024-04-02T04:59:12.104359+0000 mon.c (mon.2) 1121 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: audit 2024-04-02T04:59:12.111022+0000 mon.a (mon.0) 3283 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[28904]: cluster 2024-04-02T04:59:12.253479+0000 mgr.y (mgr.14538) 605 : cluster 0 pgmap v987: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: cluster 2024-04-02T04:59:11.931396+0000 osd.1 (osd.1) 349 : cluster 0 4.1c scrub starts 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: cluster 2024-04-02T04:59:11.933841+0000 osd.1 (osd.1) 350 : cluster 0 4.1c scrub ok 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: cluster 2024-04-02T04:59:11.993618+0000 osd.3 (osd.3) 335 : cluster 0 102.1e scrub starts 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: cluster 2024-04-02T04:59:11.996484+0000 osd.3 (osd.3) 336 : cluster 0 102.1e scrub ok 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: audit 2024-04-02T04:59:12.067395+0000 mon.a (mon.0) 3281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: cluster 2024-04-02T04:59:12.079834+0000 mon.a (mon.0) 3282 : cluster 0 osdmap e654: 8 total, 8 up, 7 in 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: audit 2024-04-02T04:59:12.082385+0000 mon.c (mon.2) 1120 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: audit 2024-04-02T04:59:12.104359+0000 mon.c (mon.2) 1121 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: audit 2024-04-02T04:59:12.111022+0000 mon.a (mon.0) 3283 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:59:13.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:13 smithi069 bash[21528]: cluster 2024-04-02T04:59:12.253479+0000 mgr.y (mgr.14538) 605 : cluster 0 pgmap v987: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:13.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:13.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: cluster 2024-04-02T04:59:11.931396+0000 osd.1 (osd.1) 349 : cluster 0 4.1c scrub starts 2024-04-02T04:59:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: cluster 2024-04-02T04:59:11.933841+0000 osd.1 (osd.1) 350 : cluster 0 4.1c scrub ok 2024-04-02T04:59:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: cluster 2024-04-02T04:59:11.993618+0000 osd.3 (osd.3) 335 : cluster 0 102.1e scrub starts 2024-04-02T04:59:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: cluster 2024-04-02T04:59:11.996484+0000 osd.3 (osd.3) 336 : cluster 0 102.1e scrub ok 2024-04-02T04:59:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: audit 2024-04-02T04:59:12.067395+0000 mon.a (mon.0) 3281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:13.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: cluster 2024-04-02T04:59:12.079834+0000 mon.a (mon.0) 3282 : cluster 0 osdmap e654: 8 total, 8 up, 7 in 2024-04-02T04:59:13.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: audit 2024-04-02T04:59:12.082385+0000 mon.c (mon.2) 1120 : audit 0 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-02T04:59:13.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: audit 2024-04-02T04:59:12.104359+0000 mon.c (mon.2) 1121 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:59:13.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: audit 2024-04-02T04:59:12.111022+0000 mon.a (mon.0) 3283 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-02T04:59:13.435 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[23790]: cluster 2024-04-02T04:59:12.253479+0000 mgr.y (mgr.14538) 605 : cluster 0 pgmap v987: 302 pgs: 32 unknown, 270 active+clean; 8.4 MiB data, 583 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:13.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:13.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:13.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:13.697 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:59:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:59:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:13.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:13.845 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:13.948 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:13.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:14.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:14.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:14.219 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: cluster 2024-04-02T04:59:12.431419+0000 osd.7 (osd.7) 145 : cluster 0 3.e scrub starts 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: cluster 2024-04-02T04:59:12.433579+0000 osd.7 (osd.7) 146 : cluster 0 3.e scrub ok 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: cluster 2024-04-02T04:59:12.806905+0000 osd.6 (osd.6) 287 : cluster 0 43.e deep-scrub starts 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: cluster 2024-04-02T04:59:12.808822+0000 osd.6 (osd.6) 288 : cluster 0 43.e deep-scrub ok 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: audit 2024-04-02T04:59:13.075347+0000 mon.a (mon.0) 3284 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: cluster 2024-04-02T04:59:13.081254+0000 mon.a (mon.0) 3285 : cluster 0 osdmap e655: 8 total, 8 up, 7 in 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: audit 2024-04-02T04:59:13.084117+0000 mon.c (mon.2) 1122 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: audit 2024-04-02T04:59:13.084668+0000 mon.a (mon.0) 3286 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[28904]: audit 2024-04-02T04:59:13.528583+0000 mgr.y (mgr.14538) 606 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: cluster 2024-04-02T04:59:12.431419+0000 osd.7 (osd.7) 145 : cluster 0 3.e scrub starts 2024-04-02T04:59:14.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: cluster 2024-04-02T04:59:12.433579+0000 osd.7 (osd.7) 146 : cluster 0 3.e scrub ok 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: cluster 2024-04-02T04:59:12.806905+0000 osd.6 (osd.6) 287 : cluster 0 43.e deep-scrub starts 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: cluster 2024-04-02T04:59:12.808822+0000 osd.6 (osd.6) 288 : cluster 0 43.e deep-scrub ok 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: audit 2024-04-02T04:59:13.075347+0000 mon.a (mon.0) 3284 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: cluster 2024-04-02T04:59:13.081254+0000 mon.a (mon.0) 3285 : cluster 0 osdmap e655: 8 total, 8 up, 7 in 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: audit 2024-04-02T04:59:13.084117+0000 mon.c (mon.2) 1122 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: audit 2024-04-02T04:59:13.084668+0000 mon.a (mon.0) 3286 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:59:14.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:14 smithi069 bash[21528]: audit 2024-04-02T04:59:13.528583+0000 mgr.y (mgr.14538) 606 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:14.234 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:14.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:14.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: cluster 2024-04-02T04:59:12.431419+0000 osd.7 (osd.7) 145 : cluster 0 3.e scrub starts 2024-04-02T04:59:14.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: cluster 2024-04-02T04:59:12.433579+0000 osd.7 (osd.7) 146 : cluster 0 3.e scrub ok 2024-04-02T04:59:14.348 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: cluster 2024-04-02T04:59:12.806905+0000 osd.6 (osd.6) 287 : cluster 0 43.e deep-scrub starts 2024-04-02T04:59:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: cluster 2024-04-02T04:59:12.808822+0000 osd.6 (osd.6) 288 : cluster 0 43.e deep-scrub ok 2024-04-02T04:59:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: audit 2024-04-02T04:59:13.075347+0000 mon.a (mon.0) 3284 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-02T04:59:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: cluster 2024-04-02T04:59:13.081254+0000 mon.a (mon.0) 3285 : cluster 0 osdmap e655: 8 total, 8 up, 7 in 2024-04-02T04:59:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: audit 2024-04-02T04:59:13.084117+0000 mon.c (mon.2) 1122 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:59:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: audit 2024-04-02T04:59:13.084668+0000 mon.a (mon.0) 3286 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-02T04:59:14.349 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:14 smithi098 bash[23790]: audit 2024-04-02T04:59:13.528583+0000 mgr.y (mgr.14538) 606 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:14.467 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:14.495 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:14.598 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:14.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:14.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:14.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:14.886 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:14.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:14.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:15.025 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:15.128 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:15.150 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:13.460655+0000 osd.2 (osd.2) 313 : cluster 0 6.14 scrub starts 2024-04-02T04:59:15.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:13.463542+0000 osd.2 (osd.2) 314 : cluster 0 6.14 scrub ok 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.002637+0000 osd.3 (osd.3) 337 : cluster 0 6.11 scrub starts 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.003864+0000 osd.3 (osd.3) 338 : cluster 0 6.11 scrub ok 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:14.077605+0000 mon.a (mon.0) 3287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.082730+0000 mon.a (mon.0) 3288 : cluster 0 osdmap e656: 8 total, 8 up, 7 in 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:14.083959+0000 mon.c (mon.2) 1123 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:14.155797+0000 mon.a (mon.0) 3289 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.190700+0000 osd.4 (osd.4) 348 : cluster 0 43.1e scrub starts 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.194282+0000 osd.4 (osd.4) 349 : cluster 0 43.1e scrub ok 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.254002+0000 mgr.y (mgr.14538) 607 : cluster 0 pgmap v990: 302 pgs: 302 active+clean; 8.4 MiB data, 593 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:15.013185+0000 mon.a (mon.0) 3290 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.019133+0000 mon.a (mon.0) 3291 : cluster 0 osdmap e657: 8 total, 8 up, 7 in 2024-04-02T04:59:15.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:15.070177+0000 mon.c (mon.2) 1124 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:15.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:15.070555+0000 mon.a (mon.0) 3292 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:15.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:15.071029+0000 mon.c (mon.2) 1125 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-102"}]: dispatch 2024-04-02T04:59:15.227 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:15 smithi098 bash[23790]: audit 2024-04-02T04:59:15.071317+0000 mon.a (mon.0) 3293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-102"}]: dispatch 2024-04-02T04:59:15.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:13.460655+0000 osd.2 (osd.2) 313 : cluster 0 6.14 scrub starts 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:13.463542+0000 osd.2 (osd.2) 314 : cluster 0 6.14 scrub ok 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.002637+0000 osd.3 (osd.3) 337 : cluster 0 6.11 scrub starts 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.003864+0000 osd.3 (osd.3) 338 : cluster 0 6.11 scrub ok 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:14.077605+0000 mon.a (mon.0) 3287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.082730+0000 mon.a (mon.0) 3288 : cluster 0 osdmap e656: 8 total, 8 up, 7 in 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:14.083959+0000 mon.c (mon.2) 1123 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:59:15.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:14.155797+0000 mon.a (mon.0) 3289 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:59:15.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.190700+0000 osd.4 (osd.4) 348 : cluster 0 43.1e scrub starts 2024-04-02T04:59:15.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.194282+0000 osd.4 (osd.4) 349 : cluster 0 43.1e scrub ok 2024-04-02T04:59:15.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.254002+0000 mgr.y (mgr.14538) 607 : cluster 0 pgmap v990: 302 pgs: 302 active+clean; 8.4 MiB data, 593 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:15.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:15.013185+0000 mon.a (mon.0) 3290 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:59:15.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.019133+0000 mon.a (mon.0) 3291 : cluster 0 osdmap e657: 8 total, 8 up, 7 in 2024-04-02T04:59:15.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:15.070177+0000 mon.c (mon.2) 1124 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:15.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:15.070555+0000 mon.a (mon.0) 3292 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:15.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:15.071029+0000 mon.c (mon.2) 1125 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-102"}]: dispatch 2024-04-02T04:59:15.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[28904]: audit 2024-04-02T04:59:15.071317+0000 mon.a (mon.0) 3293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-102"}]: dispatch 2024-04-02T04:59:15.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:13.460655+0000 osd.2 (osd.2) 313 : cluster 0 6.14 scrub starts 2024-04-02T04:59:15.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:13.463542+0000 osd.2 (osd.2) 314 : cluster 0 6.14 scrub ok 2024-04-02T04:59:15.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.002637+0000 osd.3 (osd.3) 337 : cluster 0 6.11 scrub starts 2024-04-02T04:59:15.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.003864+0000 osd.3 (osd.3) 338 : cluster 0 6.11 scrub ok 2024-04-02T04:59:15.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:14.077605+0000 mon.a (mon.0) 3287 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.082730+0000 mon.a (mon.0) 3288 : cluster 0 osdmap e656: 8 total, 8 up, 7 in 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:14.083959+0000 mon.c (mon.2) 1123 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:14.155797+0000 mon.a (mon.0) 3289 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.190700+0000 osd.4 (osd.4) 348 : cluster 0 43.1e scrub starts 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.194282+0000 osd.4 (osd.4) 349 : cluster 0 43.1e scrub ok 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.254002+0000 mgr.y (mgr.14538) 607 : cluster 0 pgmap v990: 302 pgs: 302 active+clean; 8.4 MiB data, 593 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:15.013185+0000 mon.a (mon.0) 3290 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.019133+0000 mon.a (mon.0) 3291 : cluster 0 osdmap e657: 8 total, 8 up, 7 in 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:15.070177+0000 mon.c (mon.2) 1124 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:15.070555+0000 mon.a (mon.0) 3292 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-6"}]: dispatch 2024-04-02T04:59:15.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:15.071029+0000 mon.c (mon.2) 1125 : audit 1 from='client.? 172.21.15.69:0/1136356552' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-102"}]: dispatch 2024-04-02T04:59:15.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:15 smithi069 bash[21528]: audit 2024-04-02T04:59:15.071317+0000 mon.a (mon.0) 3293 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-6", "tierpool": "test-rados-api-smithi069-72349-102"}]: dispatch 2024-04-02T04:59:15.281 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:15.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:15.411 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:15.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/599067606 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 msgr2=0x7f5d38100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/599067606 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38100c10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009920 tx=0x7f5d28009c30 comp rx=0 tx=0).stop 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/599067606 shutdown_connections 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/599067606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38101150 0x7f5d3810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/599067606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38107260 0x7f5d38107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/599067606 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38100c10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/599067606 >> 172.21.15.69:0/599067606 conn(0x7f5d38077740 msgr2=0x7f5d380feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/599067606 shutdown_connections 2024-04-02T04:59:15.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/599067606 wait complete. 2024-04-02T04:59:15.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 librados: starting msgr at 2024-04-02T04:59:15.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 librados: starting objecter 2024-04-02T04:59:15.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 Processor -- start 2024-04-02T04:59:15.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- start start 2024-04-02T04:59:15.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 librados: setting wanted keys 2024-04-02T04:59:15.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 librados: calling monclient init 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38101150 0x7f5d381986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38107260 0x7f5d3819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d38111b60 con 0x7f5d38107260 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d381119e0 con 0x7f5d38101150 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5d38111ce0 con 0x7f5d38100790 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:58708/0 (socket says 172.21.15.69:58708) 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 -- 172.21.15.69:0/3634899326 learned_addr learned my addr 172.21.15.69:0/3634899326 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d37fff700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38107260 0x7f5d3819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 -- 172.21.15.69:0/3634899326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38101150 msgr2=0x7f5d381986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d36ffd700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38101150 0x7f5d381986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:15.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38101150 0x7f5d381986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 -- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38107260 msgr2=0x7f5d3819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38107260 0x7f5d3819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 -- 172.21.15.69:0/3634899326 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3819d190 con 0x7f5d38100790 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d36ffd700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38101150 0x7f5d381986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d377fe700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38198180 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00d950 tx=0x7f5d2c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 10 client.?.objecter ms_handle_connect 0x7f5d38100790 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c016070 con 0x7f5d38100790 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f5d2c009dc0 con 0x7f5d38100790 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3819d480 con 0x7f5d38100790 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 10 client.26825.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00f3c0 con 0x7f5d38100790 2024-04-02T04:59:15.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d381a4d00 con 0x7f5d38100790 2024-04-02T04:59:15.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 librados: init done 2024-04-02T04:59:15.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3804ff00 con 0x7f5d38100790 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5d2c019070 con 0x7f5d38100790 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5d2006f5e0 0x7f5d20071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(657..657 src has 74..657) ==== 10221+0+0 (secure 0 0 0) 0x7f5d2c08e9f0 con 0x7f5d38100790 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 10 client.26825.objecter ms_dispatch 0x7f5d38001550 osd_map(657..657 src has 74..657) 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d36ffd700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5d2006f5e0 0x7f5d20071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 3 client.26825.objecter handle_osd_map got epochs [657,657] > 0 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.547+0000 7f5d34ff9700 3 client.26825.objecter handle_osd_map decoding full epoch 657 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.551+0000 7f5d34ff9700 20 client.26825.objecter dump_active .. 0 homeless 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.551+0000 7f5d34ff9700 10 client.26825.objecter ms_handle_connect 0x7f5d2006f5e0 2024-04-02T04:59:15.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.551+0000 7f5d36ffd700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5d2006f5e0 0x7f5d20071aa0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009680 tx=0x7f5d28005b40 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:15.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.555+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5d2c05b590 con 0x7f5d38100790 2024-04-02T04:59:15.641 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:15.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:15.779+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f5d381a4f60 con 0x7f5d38100790 2024-04-02T04:59:15.792 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:15.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:15.920 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:15.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:16.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:16.047 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:16.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.095+0000 7f5d34ff9700 1 -- 172.21.15.69:0/3634899326 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v658) ==== 87+0+0 (secure 0 0 0) 0x7f5d2c05efe0 con 0x7f5d38100790 2024-04-02T04:59:16.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 10 librados: watch_flush enter 2024-04-02T04:59:16.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 10 librados: watch_flush exit 2024-04-02T04:59:16.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 20 client.26825.objecter shutdown clearing up homeless session... 2024-04-02T04:59:16.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 10 client.26825.objecter successfully canceled tick 2024-04-02T04:59:16.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5d2006f5e0 msgr2=0x7f5d20071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:16.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5d2006f5e0 0x7f5d20071aa0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009680 tx=0x7f5d28005b40 comp rx=0 tx=0).stop 2024-04-02T04:59:16.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 msgr2=0x7f5d38198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:16.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.111+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38198180 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00d950 tx=0x7f5d2c00dc60 comp rx=0 tx=0).stop 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 shutdown_connections 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5d2006f5e0 0x7f5d20071aa0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38107260 0x7f5d3819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d38100790 0x7f5d38198180 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 --2- 172.21.15.69:0/3634899326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d38101150 0x7f5d381986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 >> 172.21.15.69:0/3634899326 conn(0x7f5d38077740 msgr2=0x7f5d38104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 shutdown_connections 2024-04-02T04:59:16.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 -- 172.21.15.69:0/3634899326 wait complete. 2024-04-02T04:59:16.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:16.119+0000 7f5d3dbb8700 1 librados: shutdown 2024-04-02T04:59:16.123 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T04:59:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:16.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:16.177 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:16.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.442155+0000 osd.7 (osd.7) 147 : cluster 0 10.1f deep-scrub starts 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.445733+0000 osd.7 (osd.7) 148 : cluster 0 10.1f deep-scrub ok 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.935880+0000 osd.0 (osd.0) 337 : cluster 0 6.16 scrub starts 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.937920+0000 osd.0 (osd.0) 338 : cluster 0 6.16 scrub ok 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.945638+0000 osd.5 (osd.5) 257 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:14.945849+0000 osd.5 (osd.5) 258 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.182242+0000 osd.4 (osd.4) 350 : cluster 0 10.e deep-scrub starts 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.184446+0000 osd.4 (osd.4) 351 : cluster 0 10.e deep-scrub ok 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: audit 2024-04-02T04:59:15.787375+0000 mon.c (mon.2) 1126 : audit 1 from='client.? 172.21.15.69:0/3634899326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:59:16.291 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[21528]: audit 2024-04-02T04:59:15.787871+0000 mon.a (mon.0) 3294 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.442155+0000 osd.7 (osd.7) 147 : cluster 0 10.1f deep-scrub starts 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.445733+0000 osd.7 (osd.7) 148 : cluster 0 10.1f deep-scrub ok 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.935880+0000 osd.0 (osd.0) 337 : cluster 0 6.16 scrub starts 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.937920+0000 osd.0 (osd.0) 338 : cluster 0 6.16 scrub ok 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.945638+0000 osd.5 (osd.5) 257 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:14.945849+0000 osd.5 (osd.5) 258 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.182242+0000 osd.4 (osd.4) 350 : cluster 0 10.e deep-scrub starts 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.184446+0000 osd.4 (osd.4) 351 : cluster 0 10.e deep-scrub ok 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: audit 2024-04-02T04:59:15.787375+0000 mon.c (mon.2) 1126 : audit 1 from='client.? 172.21.15.69:0/3634899326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:59:16.292 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:16 smithi069 bash[28904]: audit 2024-04-02T04:59:15.787871+0000 mon.a (mon.0) 3294 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:59:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:16.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:16.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.442155+0000 osd.7 (osd.7) 147 : cluster 0 10.1f deep-scrub starts 2024-04-02T04:59:16.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.445733+0000 osd.7 (osd.7) 148 : cluster 0 10.1f deep-scrub ok 2024-04-02T04:59:16.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.935880+0000 osd.0 (osd.0) 337 : cluster 0 6.16 scrub starts 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.937920+0000 osd.0 (osd.0) 338 : cluster 0 6.16 scrub ok 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.945638+0000 osd.5 (osd.5) 257 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:14.945849+0000 osd.5 (osd.5) 258 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.182242+0000 osd.4 (osd.4) 350 : cluster 0 10.e deep-scrub starts 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.184446+0000 osd.4 (osd.4) 351 : cluster 0 10.e deep-scrub ok 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: audit 2024-04-02T04:59:15.787375+0000 mon.c (mon.2) 1126 : audit 1 from='client.? 172.21.15.69:0/3634899326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:59:16.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:16 smithi098 bash[23790]: audit 2024-04-02T04:59:15.787871+0000 mon.a (mon.0) 3294 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T04:59:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:16.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:16.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:16.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:16.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:16.831 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:16.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:16.959 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:16.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:17.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:17.087 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:17.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.467870+0000 osd.7 (osd.7) 149 : cluster 0 3.11 scrub starts 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.470084+0000 osd.7 (osd.7) 150 : cluster 0 3.11 scrub ok 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.848549+0000 osd.6 (osd.6) 289 : cluster 0 40.f deep-scrub starts 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.850536+0000 osd.6 (osd.6) 290 : cluster 0 40.f deep-scrub ok 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.922445+0000 osd.0 (osd.0) 339 : cluster 0 10.11 deep-scrub starts 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:15.923773+0000 osd.0 (osd.0) 340 : cluster 0 10.11 deep-scrub ok 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:16.094255+0000 mon.a (mon.0) 3295 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: audit 2024-04-02T04:59:16.095876+0000 mon.a (mon.0) 3296 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:16.104326+0000 mon.a (mon.0) 3297 : cluster 0 osdmap e658: 8 total, 8 up, 7 in 2024-04-02T04:59:17.218 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[28904]: cluster 2024-04-02T04:59:16.254911+0000 mgr.y (mgr.14538) 608 : cluster 0 pgmap v993: 270 pgs: 270 active+clean; 8.4 MiB data, 593 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.467870+0000 osd.7 (osd.7) 149 : cluster 0 3.11 scrub starts 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.470084+0000 osd.7 (osd.7) 150 : cluster 0 3.11 scrub ok 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.848549+0000 osd.6 (osd.6) 289 : cluster 0 40.f deep-scrub starts 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.850536+0000 osd.6 (osd.6) 290 : cluster 0 40.f deep-scrub ok 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.922445+0000 osd.0 (osd.0) 339 : cluster 0 10.11 deep-scrub starts 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:15.923773+0000 osd.0 (osd.0) 340 : cluster 0 10.11 deep-scrub ok 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:16.094255+0000 mon.a (mon.0) 3295 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: audit 2024-04-02T04:59:16.095876+0000 mon.a (mon.0) 3296 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:16.104326+0000 mon.a (mon.0) 3297 : cluster 0 osdmap e658: 8 total, 8 up, 7 in 2024-04-02T04:59:17.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:17 smithi069 bash[21528]: cluster 2024-04-02T04:59:16.254911+0000 mgr.y (mgr.14538) 608 : cluster 0 pgmap v993: 270 pgs: 270 active+clean; 8.4 MiB data, 593 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:17.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:17.347 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:17.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:17.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.467870+0000 osd.7 (osd.7) 149 : cluster 0 3.11 scrub starts 2024-04-02T04:59:17.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.470084+0000 osd.7 (osd.7) 150 : cluster 0 3.11 scrub ok 2024-04-02T04:59:17.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.848549+0000 osd.6 (osd.6) 289 : cluster 0 40.f deep-scrub starts 2024-04-02T04:59:17.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.850536+0000 osd.6 (osd.6) 290 : cluster 0 40.f deep-scrub ok 2024-04-02T04:59:17.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.922445+0000 osd.0 (osd.0) 339 : cluster 0 10.11 deep-scrub starts 2024-04-02T04:59:17.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:15.923773+0000 osd.0 (osd.0) 340 : cluster 0 10.11 deep-scrub ok 2024-04-02T04:59:17.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:16.094255+0000 mon.a (mon.0) 3295 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:17.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: audit 2024-04-02T04:59:16.095876+0000 mon.a (mon.0) 3296 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T04:59:17.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:16.104326+0000 mon.a (mon.0) 3297 : cluster 0 osdmap e658: 8 total, 8 up, 7 in 2024-04-02T04:59:17.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:17 smithi098 bash[23790]: cluster 2024-04-02T04:59:16.254911+0000 mgr.y (mgr.14538) 608 : cluster 0 pgmap v993: 270 pgs: 270 active+clean; 8.4 MiB data, 593 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:17.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:17.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:17.607 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:17.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:17.739 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:17.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:17.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:17.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:17.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:18.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:18.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:18.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:18.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:18.367 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[28904]: cluster 2024-04-02T04:59:17.127506+0000 mon.a (mon.0) 3298 : cluster 0 osdmap e659: 8 total, 8 up, 7 in 2024-04-02T04:59:18.367 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[28904]: audit 2024-04-02T04:59:17.157918+0000 mon.a (mon.0) 3299 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:18.367 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[28904]: audit 2024-04-02T04:59:17.159085+0000 mon.a (mon.0) 3300 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:18.367 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[28904]: audit 2024-04-02T04:59:17.159442+0000 mon.a (mon.0) 3301 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:18.368 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[21528]: cluster 2024-04-02T04:59:17.127506+0000 mon.a (mon.0) 3298 : cluster 0 osdmap e659: 8 total, 8 up, 7 in 2024-04-02T04:59:18.368 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[21528]: audit 2024-04-02T04:59:17.157918+0000 mon.a (mon.0) 3299 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:18.368 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[21528]: audit 2024-04-02T04:59:17.159085+0000 mon.a (mon.0) 3300 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:18.368 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[21528]: audit 2024-04-02T04:59:17.159442+0000 mon.a (mon.0) 3301 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:18.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:18.383 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:18 smithi098 bash[23790]: cluster 2024-04-02T04:59:17.127506+0000 mon.a (mon.0) 3298 : cluster 0 osdmap e659: 8 total, 8 up, 7 in 2024-04-02T04:59:18.383 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:18 smithi098 bash[23790]: audit 2024-04-02T04:59:17.157918+0000 mon.a (mon.0) 3299 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:18.383 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:18 smithi098 bash[23790]: audit 2024-04-02T04:59:17.159085+0000 mon.a (mon.0) 3300 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:18.383 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:18 smithi098 bash[23790]: audit 2024-04-02T04:59:17.159442+0000 mon.a (mon.0) 3301 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:18.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:18.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:18.520 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:18.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/2065928303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58107220 msgr2=0x7f0f58107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:18.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/2065928303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58107220 0x7f0f58107600 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009990 tx=0x7f0f48009ca0 comp rx=0 tx=0).stop 2024-04-02T04:59:18.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/2065928303 shutdown_connections 2024-04-02T04:59:18.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/2065928303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f58101b60 0x7f0f5810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:18.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/2065928303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58107220 0x7f0f58107600 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/2065928303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f581011a0 0x7f0f58101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/2065928303 >> 172.21.15.69:0/2065928303 conn(0x7f0f580771e0 msgr2=0x7f0f580775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/2065928303 shutdown_connections 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/2065928303 wait complete. 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 librados: starting msgr at 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 librados: starting objecter 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 Processor -- start 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- start start 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 librados: setting wanted keys 2024-04-02T04:59:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 librados: calling monclient init 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f581011a0 0x7f0f58197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 0x7f0f58198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f58107220 0x7f0f5819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f58110970 con 0x7f0f58107220 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f581107f0 con 0x7f0f581011a0 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0f58110af0 con 0x7f0f58101b60 2024-04-02T04:59:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 0x7f0f58198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 0x7f0f58198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33076/0 (socket says 172.21.15.69:33076) 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 -- 172.21.15.69:0/996647192 learned_addr learned my addr 172.21.15.69:0/996647192 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f57fff700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f58107220 0x7f0f5819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f577fe700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f581011a0 0x7f0f58197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 -- 172.21.15.69:0/996647192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f581011a0 msgr2=0x7f0f58197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f581011a0 0x7f0f58197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 -- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f58107220 msgr2=0x7f0f5819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f58107220 0x7f0f5819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 -- 172.21.15.69:0/996647192 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f5819cfa0 con 0x7f0f58101b60 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f577fe700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f581011a0 0x7f0f58197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f57fff700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f58107220 0x7f0f5819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f56ffd700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 0x7f0f58198520 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00eb30 tx=0x7f0f4c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f54ff9700 10 client.?.objecter ms_handle_connect 0x7f0f58101b60 2024-04-02T04:59:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f54ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c00cc90 con 0x7f0f58101b60 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f5819d290 con 0x7f0f58101b60 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0f4c00ce30 con 0x7f0f58101b60 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c003df0 con 0x7f0f58101b60 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 10 client.26837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f581a4b10 con 0x7f0f58101b60 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.539+0000 7f0f5dd37700 1 librados: init done 2024-04-02T04:59:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f5804ff00 con 0x7f0f58101b60 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0f4c004000 con 0x7f0f58101b60 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0f4006f5e0 0x7f0f40071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(660..660 src has 74..660) ==== 9716+0+0 (secure 0 0 0) 0x7f0f4c08ebf0 con 0x7f0f58101b60 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 10 client.26837.objecter ms_dispatch 0x7f0f58001550 osd_map(660..660 src has 74..660) 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 3 client.26837.objecter handle_osd_map got epochs [660,660] > 0 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 3 client.26837.objecter handle_osd_map decoding full epoch 660 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f577fe700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0f4006f5e0 0x7f0f40071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 20 client.26837.objecter dump_active .. 0 homeless 2024-04-02T04:59:18.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f54ff9700 10 client.26837.objecter ms_handle_connect 0x7f0f4006f5e0 2024-04-02T04:59:18.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.543+0000 7f0f577fe700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0f4006f5e0 0x7f0f40071aa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800b2e0 tx=0x7f0f48005e80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:18.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.547+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0f4c016330 con 0x7f0f58101b60 2024-04-02T04:59:18.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:18.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:18.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:18.775 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:18.779 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:59:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:59:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:59:18.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:18.783+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482} v 0) -- 0x7f0f581a4dd0 con 0x7f0f58101b60 2024-04-02T04:59:18.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:18.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:19.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:19.035 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:19.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.139+0000 7f0f54ff9700 1 -- 172.21.15.69:0/996647192 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]=0 reweighted osd.4 to 0.566525 (9107) v661) ==== 134+0+0 (secure 0 0 0) 0x7f0f4c05b900 con 0x7f0f58101b60 2024-04-02T04:59:19.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 10 librados: watch_flush enter 2024-04-02T04:59:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 10 librados: watch_flush exit 2024-04-02T04:59:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 20 client.26837.objecter shutdown clearing up homeless session... 2024-04-02T04:59:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 10 client.26837.objecter successfully canceled tick 2024-04-02T04:59:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0f4006f5e0 msgr2=0x7f0f40071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0f4006f5e0 0x7f0f40071aa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800b2e0 tx=0x7f0f48005e80 comp rx=0 tx=0).stop 2024-04-02T04:59:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 msgr2=0x7f0f58198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:19.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 0x7f0f58198520 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00eb30 tx=0x7f0f4c00ee40 comp rx=0 tx=0).stop 2024-04-02T04:59:19.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 shutdown_connections 2024-04-02T04:59:19.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0f4006f5e0 0x7f0f40071aa0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f58107220 0x7f0f5819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f581011a0 0x7f0f58197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 --2- 172.21.15.69:0/996647192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f58101b60 0x7f0f58198520 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.151+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 >> 172.21.15.69:0/996647192 conn(0x7f0f580771e0 msgr2=0x7f0f58069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.155+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 shutdown_connections 2024-04-02T04:59:19.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.155+0000 7f0f5dd37700 1 -- 172.21.15.69:0/996647192 wait complete. 2024-04-02T04:59:19.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:19.155+0000 7f0f5dd37700 1 librados: shutdown 2024-04-02T04:59:19.156 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.4 to 0.566525 (9107) 2024-04-02T04:59:19.183 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:19.286 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:19.306 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:19.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:19.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:19.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:19.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:19.690 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:19.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: audit 2024-04-02T04:59:18.130323+0000 mon.a (mon.0) 3302 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: cluster 2024-04-02T04:59:18.134186+0000 mon.a (mon.0) 3303 : cluster 0 osdmap e660: 8 total, 8 up, 7 in 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: audit 2024-04-02T04:59:18.137197+0000 mon.a (mon.0) 3304 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-72349-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: cluster 2024-04-02T04:59:18.256111+0000 mgr.y (mgr.14538) 609 : cluster 0 pgmap v996: 238 pgs: 238 active+clean; 582 KiB data, 585 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: audit 2024-04-02T04:59:18.728367+0000 mon.a (mon.0) 3305 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: audit 2024-04-02T04:59:18.729652+0000 mon.b (mon.1) 286 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: audit 2024-04-02T04:59:18.787060+0000 mon.c (mon.2) 1127 : audit 1 from='client.? 172.21.15.69:0/996647192' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]: dispatch 2024-04-02T04:59:19.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[21528]: audit 2024-04-02T04:59:18.787450+0000 mon.a (mon.0) 3306 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]: dispatch 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: audit 2024-04-02T04:59:18.130323+0000 mon.a (mon.0) 3302 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: cluster 2024-04-02T04:59:18.134186+0000 mon.a (mon.0) 3303 : cluster 0 osdmap e660: 8 total, 8 up, 7 in 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: audit 2024-04-02T04:59:18.137197+0000 mon.a (mon.0) 3304 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-72349-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: cluster 2024-04-02T04:59:18.256111+0000 mgr.y (mgr.14538) 609 : cluster 0 pgmap v996: 238 pgs: 238 active+clean; 582 KiB data, 585 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: audit 2024-04-02T04:59:18.728367+0000 mon.a (mon.0) 3305 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: audit 2024-04-02T04:59:18.729652+0000 mon.b (mon.1) 286 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: audit 2024-04-02T04:59:18.787060+0000 mon.c (mon.2) 1127 : audit 1 from='client.? 172.21.15.69:0/996647192' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]: dispatch 2024-04-02T04:59:19.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:19 smithi069 bash[28904]: audit 2024-04-02T04:59:18.787450+0000 mon.a (mon.0) 3306 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]: dispatch 2024-04-02T04:59:19.817 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:19.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:19.947 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:19.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: audit 2024-04-02T04:59:18.130323+0000 mon.a (mon.0) 3302 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: cluster 2024-04-02T04:59:18.134186+0000 mon.a (mon.0) 3303 : cluster 0 osdmap e660: 8 total, 8 up, 7 in 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: audit 2024-04-02T04:59:18.137197+0000 mon.a (mon.0) 3304 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-72349-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: cluster 2024-04-02T04:59:18.256111+0000 mgr.y (mgr.14538) 609 : cluster 0 pgmap v996: 238 pgs: 238 active+clean; 582 KiB data, 585 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: audit 2024-04-02T04:59:18.728367+0000 mon.a (mon.0) 3305 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: audit 2024-04-02T04:59:18.729652+0000 mon.b (mon.1) 286 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: audit 2024-04-02T04:59:18.787060+0000 mon.c (mon.2) 1127 : audit 1 from='client.? 172.21.15.69:0/996647192' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]: dispatch 2024-04-02T04:59:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:19 smithi098 bash[23790]: audit 2024-04-02T04:59:18.787450+0000 mon.a (mon.0) 3306 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]: dispatch 2024-04-02T04:59:20.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:20.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T04:59:20.087 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:20.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:20.209 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:20.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:20.339 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:20.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:20.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:20.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:20.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[21528]: audit 2024-04-02T04:59:19.134467+0000 mon.a (mon.0) 3307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]': finished 2024-04-02T04:59:20.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[21528]: cluster 2024-04-02T04:59:19.592642+0000 mon.a (mon.0) 3308 : cluster 0 osdmap e661: 8 total, 8 up, 7 in 2024-04-02T04:59:20.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[21528]: audit 2024-04-02T04:59:20.015788+0000 mon.a (mon.0) 3309 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-72349-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:20.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[21528]: cluster 2024-04-02T04:59:20.027855+0000 mon.a (mon.0) 3310 : cluster 0 osdmap e662: 8 total, 8 up, 7 in 2024-04-02T04:59:20.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[28904]: audit 2024-04-02T04:59:19.134467+0000 mon.a (mon.0) 3307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]': finished 2024-04-02T04:59:20.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[28904]: cluster 2024-04-02T04:59:19.592642+0000 mon.a (mon.0) 3308 : cluster 0 osdmap e661: 8 total, 8 up, 7 in 2024-04-02T04:59:20.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[28904]: audit 2024-04-02T04:59:20.015788+0000 mon.a (mon.0) 3309 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-72349-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:20.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:20 smithi069 bash[28904]: cluster 2024-04-02T04:59:20.027855+0000 mon.a (mon.0) 3310 : cluster 0 osdmap e662: 8 total, 8 up, 7 in 2024-04-02T04:59:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:20.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:20.859 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:20.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:20.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:20 smithi098 bash[23790]: audit 2024-04-02T04:59:19.134467+0000 mon.a (mon.0) 3307 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.5665245850577482}]': finished 2024-04-02T04:59:20.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:20 smithi098 bash[23790]: cluster 2024-04-02T04:59:19.592642+0000 mon.a (mon.0) 3308 : cluster 0 osdmap e661: 8 total, 8 up, 7 in 2024-04-02T04:59:20.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:20 smithi098 bash[23790]: audit 2024-04-02T04:59:20.015788+0000 mon.a (mon.0) 3309 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-72349-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:20.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:20 smithi098 bash[23790]: cluster 2024-04-02T04:59:20.027855+0000 mon.a (mon.0) 3310 : cluster 0 osdmap e662: 8 total, 8 up, 7 in 2024-04-02T04:59:20.989 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:20.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:21.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:21.115 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:21.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:21.246 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:21.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:21.373 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:21.441 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:21.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:21.497 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:21.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:21.627 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:21.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:21.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:21 smithi069 bash[21528]: cluster 2024-04-02T04:59:20.256743+0000 mgr.y (mgr.14538) 610 : cluster 0 pgmap v999: 246 pgs: 8 unknown, 238 active+clean; 582 KiB data, 581 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:21.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:21 smithi069 bash[21528]: cluster 2024-04-02T04:59:21.026120+0000 mon.a (mon.0) 3311 : cluster 0 osdmap e663: 8 total, 8 up, 7 in 2024-04-02T04:59:21.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:21 smithi069 bash[21528]: cluster 2024-04-02T04:59:21.047412+0000 mon.a (mon.0) 3312 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:21.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:21 smithi069 bash[28904]: cluster 2024-04-02T04:59:20.256743+0000 mgr.y (mgr.14538) 610 : cluster 0 pgmap v999: 246 pgs: 8 unknown, 238 active+clean; 582 KiB data, 581 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:21.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:21 smithi069 bash[28904]: cluster 2024-04-02T04:59:21.026120+0000 mon.a (mon.0) 3311 : cluster 0 osdmap e663: 8 total, 8 up, 7 in 2024-04-02T04:59:21.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:21 smithi069 bash[28904]: cluster 2024-04-02T04:59:21.047412+0000 mon.a (mon.0) 3312 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:21.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:21.887 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:21.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:21 smithi098 bash[23790]: cluster 2024-04-02T04:59:20.256743+0000 mgr.y (mgr.14538) 610 : cluster 0 pgmap v999: 246 pgs: 8 unknown, 238 active+clean; 582 KiB data, 581 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:21 smithi098 bash[23790]: cluster 2024-04-02T04:59:21.026120+0000 mon.a (mon.0) 3311 : cluster 0 osdmap e663: 8 total, 8 up, 7 in 2024-04-02T04:59:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:21 smithi098 bash[23790]: cluster 2024-04-02T04:59:21.047412+0000 mon.a (mon.0) 3312 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:21.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:22.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:22.149 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:22.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:22.276 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:22.379 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:22.404 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:22.508 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:22.535 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:22.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:22.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:22.795 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:22.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:22.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:23.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.067 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:23.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:23.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:23.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:23 smithi098 bash[23790]: cluster 2024-04-02T04:59:22.029774+0000 mon.a (mon.0) 3313 : cluster 0 osdmap e664: 8 total, 8 up, 7 in 2024-04-02T04:59:23.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:23 smithi098 bash[23790]: audit 2024-04-02T04:59:22.041112+0000 mon.c (mon.2) 1128 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:23.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:23 smithi098 bash[23790]: audit 2024-04-02T04:59:22.041917+0000 mon.a (mon.0) 3314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:23.179 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:23 smithi098 bash[23790]: cluster 2024-04-02T04:59:22.257305+0000 mgr.y (mgr.14538) 611 : cluster 0 pgmap v1002: 278 pgs: 40 unknown, 238 active+clean; 582 KiB data, 581 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:23.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1894879255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 msgr2=0x7f4fc410e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1894879255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 0x7f4fc410e0d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00b0d0 tx=0x7f4fbc00b3e0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1894879255 shutdown_connections 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1894879255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 0x7f4fc410e0d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1894879255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fc4101420 0x7f4fc4101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1894879255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc4101dd0 0x7f4fc4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1894879255 >> 172.21.15.69:0/1894879255 conn(0x7f4fc4078240 msgr2=0x7f4fc4078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1894879255 shutdown_connections 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1894879255 wait complete. 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 librados: starting msgr at 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 librados: starting objecter 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 Processor -- start 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- start start 2024-04-02T04:59:23.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 librados: setting wanted keys 2024-04-02T04:59:23.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 librados: calling monclient init 2024-04-02T04:59:23.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc4101420 0x7f4fc4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fc4101dd0 0x7f4fc41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 0x7f4fc419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41109f0 con 0x7f4fc4101420 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4110870 con 0x7f4fc4101dd0 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc4110b70 con 0x7f4fc4106500 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc4101420 0x7f4fc4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 0x7f4fc419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc4101420 0x7f4fc4198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53542/0 (socket says 172.21.15.69:53542) 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc2ffd700 1 -- 172.21.15.69:0/1467377700 learned_addr learned my addr 172.21.15.69:0/1467377700 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fc4101dd0 0x7f4fc41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fc4101dd0 msgr2=0x7f4fc41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fc4101dd0 0x7f4fc41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc4101420 msgr2=0x7f4fc4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc4101420 0x7f4fc4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc419d0a0 con 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fc4101dd0 0x7f4fc41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 0x7f4fc419c960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00b770 tx=0x7f4fbc009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc0ff9700 10 client.?.objecter ms_handle_connect 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc0ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc022070 con 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4fbc020030 con 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc419d330 con 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc005070 con 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 10 client.26849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc4102df0 con 0x7f4fc4106500 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.255+0000 7f4fc94b4700 1 librados: init done 2024-04-02T04:59:23.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4fc404ff00 con 0x7f4fc4106500 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4fbc01c6c0 con 0x7f4fc4106500 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4fb006f5e0 0x7f4fb0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(665..665 src has 74..665) ==== 10637+0+0 (secure 0 0 0) 0x7f4fbc00f030 con 0x7f4fc4106500 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 osd_map(665..665 src has 74..665) 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 3 client.26849.objecter handle_osd_map got epochs [665,665] > 0 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 3 client.26849.objecter handle_osd_map decoding full epoch 665 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4fb006f5e0 0x7f4fb0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 20 client.26849.objecter dump_active .. 0 homeless 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v665) ==== 54+0+16 (secure 0 0 0) 0x7f4fbc02b060 con 0x7f4fc4106500 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 1 2024-04-02T04:59:23.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4fc4062e50 0x7f4fc4063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 1 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 0 1 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 4 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 10 client.26849.objecter _send_command 1 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fc41041d0 con 0x7f4fc4062e50 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4fb006f5e0 0x7f4fb0071aa0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009d60 tx=0x7f4fb4009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.259+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fb006f5e0 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4fc4062e50 0x7f4fc4063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc4062e50 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4fc4062e50 0x7f4fc4063230 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc41041d0 con 0x7f4fc4062e50 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 1: 0 ) 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 1 = system:0 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 4 2024-04-02T04:59:23.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.263+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 0 1 2024-04-02T04:59:23.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 2 2024-04-02T04:59:23.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 2 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41993f0 osd=0 3 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 0 2 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 4 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 10 client.26849.objecter _send_command 2 2024-04-02T04:59:23.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc4104bd0 con 0x7f4fc4062e50 2024-04-02T04:59:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc4104bd0 con 0x7f4fc4062e50 2024-04-02T04:59:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 4 2024-04-02T04:59:23.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 0 2 2024-04-02T04:59:23.285 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:59:23.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4fc4103560 0x7f4fc4103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 1 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 4 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 10 client.26849.objecter _send_command 3 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4fc4103ec0 con 0x7f4fc4103560 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.283+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4fc4103560 0x7f4fc4103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc4103560 2024-04-02T04:59:23.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4fc4103560 0x7f4fc4103980 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc4103ec0 con 0x7f4fc4103560 2024-04-02T04:59:23.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 3: 0 ) 2024-04-02T04:59:23.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 3 = system:0 2024-04-02T04:59:23.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 4 2024-04-02T04:59:23.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.287+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 1 3 2024-04-02T04:59:23.296 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[21528]: cluster 2024-04-02T04:59:22.029774+0000 mon.a (mon.0) 3313 : cluster 0 osdmap e664: 8 total, 8 up, 7 in 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[21528]: audit 2024-04-02T04:59:22.041112+0000 mon.c (mon.2) 1128 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[21528]: audit 2024-04-02T04:59:22.041917+0000 mon.a (mon.0) 3314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[21528]: cluster 2024-04-02T04:59:22.257305+0000 mgr.y (mgr.14538) 611 : cluster 0 pgmap v1002: 278 pgs: 40 unknown, 238 active+clean; 582 KiB data, 581 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[28904]: cluster 2024-04-02T04:59:22.029774+0000 mon.a (mon.0) 3313 : cluster 0 osdmap e664: 8 total, 8 up, 7 in 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[28904]: audit 2024-04-02T04:59:22.041112+0000 mon.c (mon.2) 1128 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[28904]: audit 2024-04-02T04:59:22.041917+0000 mon.a (mon.0) 3314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:23.297 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:23 smithi069 bash[28904]: cluster 2024-04-02T04:59:22.257305+0000 mgr.y (mgr.14538) 611 : cluster 0 pgmap v1002: 278 pgs: 40 unknown, 238 active+clean; 582 KiB data, 581 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:23.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:23.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 4 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 4 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc4104de0 osd=1 3 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 1 4 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 4 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 10 client.26849.objecter _send_command 4 2024-04-02T04:59:23.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc402c9b0 con 0x7f4fc4103560 2024-04-02T04:59:23.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc402c9b0 con 0x7f4fc4103560 2024-04-02T04:59:23.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 4 2024-04-02T04:59:23.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 1 4 2024-04-02T04:59:23.313 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:59:23.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 5 2024-04-02T04:59:23.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.311+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4fc4109860 0x7f4fc4109cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4fc4109860 0x7f4fc4109cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc4109860 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4fc4109860 0x7f4fc4109cd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 5 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 2 5 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 4 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 10 client.26849.objecter _send_command 5 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4fc402dee0 con 0x7f4fc4109860 2024-04-02T04:59:23.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc402dee0 con 0x7f4fc4109860 2024-04-02T04:59:23.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 5: 0 ) 2024-04-02T04:59:23.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 5 = system:0 2024-04-02T04:59:23.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 4 2024-04-02T04:59:23.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.315+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 2 5 2024-04-02T04:59:23.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 6 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 6 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc402dd70 osd=2 3 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 2 6 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 4 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 10 client.26849.objecter _send_command 6 2024-04-02T04:59:23.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.335+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc41092e0 con 0x7f4fc4109860 2024-04-02T04:59:23.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc41092e0 con 0x7f4fc4109860 2024-04-02T04:59:23.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 4 2024-04-02T04:59:23.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 2 6 2024-04-02T04:59:23.340 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:59:23.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 7 2024-04-02T04:59:23.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4fc402d840 0x7f4fc41addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 7 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 3 7 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 4 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 10 client.26849.objecter _send_command 7 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4fc41096a0 con 0x7f4fc402d840 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4fc402d840 0x7f4fc41addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc402d840 2024-04-02T04:59:23.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.339+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4fc402d840 0x7f4fc41addd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.343+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fbc099020 con 0x7f4fc402d840 2024-04-02T04:59:23.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.343+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 7: 0 ) 2024-04-02T04:59:23.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.343+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 7 = system:0 2024-04-02T04:59:23.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.343+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 4 2024-04-02T04:59:23.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.343+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 3 7 2024-04-02T04:59:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 8 2024-04-02T04:59:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 8 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc4109530 osd=3 3 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 3 8 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 4 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 10 client.26849.objecter _send_command 8 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.359+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc402cdb0 con 0x7f4fc402d840 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc402cdb0 con 0x7f4fc402d840 2024-04-02T04:59:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 4 2024-04-02T04:59:23.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 3 8 2024-04-02T04:59:23.362 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:59:23.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 9 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4fc402cf20 0x7f4fc41b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 9 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 4 9 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 4 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 10 client.26849.objecter _send_command 9 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4fc41abab0 con 0x7f4fc402cf20 2024-04-02T04:59:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4fc402cf20 0x7f4fc41b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc402cf20 2024-04-02T04:59:23.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.363+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4fc402cf20 0x7f4fc41b14a0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.367+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc41abab0 con 0x7f4fc402cf20 2024-04-02T04:59:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.367+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 9: 0 ) 2024-04-02T04:59:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.367+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 9 = system:0 2024-04-02T04:59:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.367+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 4 2024-04-02T04:59:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.367+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 4 9 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 10 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 10 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc402cdb0 osd=4 3 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 4 10 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 4 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 10 client.26849.objecter _send_command 10 2024-04-02T04:59:23.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.383+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc41abab0 con 0x7f4fc402cf20 2024-04-02T04:59:23.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc41abab0 con 0x7f4fc402cf20 2024-04-02T04:59:23.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 4 2024-04-02T04:59:23.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 4 10 2024-04-02T04:59:23.387 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 11 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4fc41b1f50 0x7f4fc41b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 11 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 5 11 2024-04-02T04:59:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 4 2024-04-02T04:59:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 10 client.26849.objecter _send_command 11 2024-04-02T04:59:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4fc41b7520 con 0x7f4fc41b1f50 2024-04-02T04:59:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4fc41b1f50 0x7f4fc41b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fbbfff700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4fc41b1f50 0x7f4fc41b23b0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.387+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc41b1f50 2024-04-02T04:59:23.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.391+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc41b7520 con 0x7f4fc41b1f50 2024-04-02T04:59:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.391+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 11: 0 ) 2024-04-02T04:59:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.391+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 11 = system:0 2024-04-02T04:59:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.391+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 4 2024-04-02T04:59:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.391+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 5 11 2024-04-02T04:59:23.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 12 2024-04-02T04:59:23.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 12 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41abab0 osd=5 3 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 5 12 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 4 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 10 client.26849.objecter _send_command 12 2024-04-02T04:59:23.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.407+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc41b2a00 con 0x7f4fc41b1f50 2024-04-02T04:59:23.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc41b2a00 con 0x7f4fc41b1f50 2024-04-02T04:59:23.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 4 2024-04-02T04:59:23.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 5 12 2024-04-02T04:59:23.413 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:59:23.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 13 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4fc41b3120 0x7f4fc41b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 13 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 6 13 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 4 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 10 client.26849.objecter _send_command 13 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.411+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4fc41b3c00 con 0x7f4fc41b3120 2024-04-02T04:59:23.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4fc41b3120 0x7f4fc41b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc37fe700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4fc41b3120 0x7f4fc41b3560 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc41b3120 2024-04-02T04:59:23.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc41b3c00 con 0x7f4fc41b3120 2024-04-02T04:59:23.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 13: 0 ) 2024-04-02T04:59:23.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 13 = system:0 2024-04-02T04:59:23.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 4 2024-04-02T04:59:23.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.415+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 6 13 2024-04-02T04:59:23.425 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 14 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 14 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41b2a00 osd=6 3 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 6 14 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 4 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 10 client.26849.objecter _send_command 14 2024-04-02T04:59:23.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.435+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc41b3c00 con 0x7f4fc41b3120 2024-04-02T04:59:23.441 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:59:23.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc41b3c00 con 0x7f4fc41b3120 2024-04-02T04:59:23.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 4 2024-04-02T04:59:23.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 6 14 2024-04-02T04:59:23.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:23.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 15 2024-04-02T04:59:23.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4fc41b3d70 0x7f4fc41ba8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 15 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 7 15 2024-04-02T04:59:23.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 4 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 10 client.26849.objecter _send_command 15 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4fc41bd950 con 0x7f4fc41b3d70 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4fc41b3d70 0x7f4fc41ba8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc2ffd700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4fc41b3d70 0x7f4fc41ba8d0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 10 client.26849.objecter ms_handle_connect 0x7f4fc41b3d70 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fc41bd950 con 0x7f4fc41b3d70 2024-04-02T04:59:23.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 15: 0 ) 2024-04-02T04:59:23.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.439+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 15 = system:0 2024-04-02T04:59:23.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.443+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 4 2024-04-02T04:59:23.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.443+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 7 15 2024-04-02T04:59:23.453 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:23.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 10 client.26849.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign -1 16 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 20 client.26849.objecter _get_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_remove -1 16 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 20 client.26849.objecter get_session s=0x7f4fc41b3c00 osd=7 3 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 15 client.26849.objecter _session_command_op_assign 7 16 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 4 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 10 client.26849.objecter _send_command 16 2024-04-02T04:59:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.463+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4fc41bc700 con 0x7f4fc41b3d70 2024-04-02T04:59:23.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc0ff9700 1 -- 172.21.15.69:0/1467377700 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4fc41bc700 con 0x7f4fc41b3d70 2024-04-02T04:59:23.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc0ff9700 10 client.26849.objecter ms_dispatch 0x7f4fc4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:23.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc0ff9700 10 client.26849.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:23.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc0ff9700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 4 2024-04-02T04:59:23.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc0ff9700 15 client.26849.objecter _session_command_op_remove 7 16 2024-04-02T04:59:23.467 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 librados: watch_flush enter 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 librados: watch_flush exit 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.0 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4fc4062e50 msgr2=0x7f4fc4063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4fc4062e50 0x7f4fc4063230 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41993f0 osd=0 1 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.1 2024-04-02T04:59:23.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4fc4103560 msgr2=0x7f4fc4103980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4fc4103560 0x7f4fc4103980 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4104de0 osd=1 1 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.2 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4fc4109860 msgr2=0x7f4fc4109cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4fc4109860 0x7f4fc4109cd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402dd70 osd=2 1 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.3 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4fc402d840 msgr2=0x7f4fc41addd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4fc402d840 0x7f4fc41addd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc4109530 osd=3 1 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.4 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4fc402cf20 msgr2=0x7f4fc41b14a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4fc402cf20 0x7f4fc41b14a0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc402cdb0 osd=4 1 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.5 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4fc41b1f50 msgr2=0x7f4fc41b23b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4fc41b1f50 0x7f4fc41b23b0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41abab0 osd=5 1 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.6 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4fc41b3120 msgr2=0x7f4fc41b3560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4fc41b3120 0x7f4fc41b3560 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b2a00 osd=6 1 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter close_session for osd.7 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4fc41b3d70 msgr2=0x7f4fc41ba8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4fc41b3d70 0x7f4fc41ba8d0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:23.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter put_session s=0x7f4fc41b3c00 osd=7 1 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 20 client.26849.objecter shutdown clearing up homeless session... 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 10 client.26849.objecter successfully canceled tick 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 reap_dead start 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 reap_dead start 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4fb006f5e0 msgr2=0x7f4fb0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4fb006f5e0 0x7f4fb0071aa0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009d60 tx=0x7f4fb4009260 comp rx=0 tx=0).stop 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 msgr2=0x7f4fc419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 --2- 172.21.15.69:0/1467377700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fc4106500 0x7f4fc419c960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00b770 tx=0x7f4fbc009c70 comp rx=0 tx=0).stop 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 reap_dead start 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 reap_dead start 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc37fe700 1 -- 172.21.15.69:0/1467377700 reap_dead start 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 shutdown_connections 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 >> 172.21.15.69:0/1467377700 conn(0x7f4fc4078240 msgr2=0x7f4fc406a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 shutdown_connections 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 -- 172.21.15.69:0/1467377700 wait complete. 2024-04-02T04:59:23.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:23.467+0000 7f4fc94b4700 1 librados: shutdown 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:59:23.472 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:59:23.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:23.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:23.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:23.702 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:23.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:23.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:59:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:59:23.827 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:23.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:23.951 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:23.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:24.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:24.070 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.032761+0000 mon.a (mon.0) 3315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: cluster 2024-04-02T04:59:23.044506+0000 mon.a (mon.0) 3316 : cluster 0 osdmap e665: 8 total, 8 up, 7 in 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.233969+0000 mon.b (mon.1) 287 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.262097+0000 mon.c (mon.2) 1129 : audit 0 from='client.? 172.21.15.69:0/1467377700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.536209+0000 mgr.y (mgr.14538) 612 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.918942+0000 mon.b (mon.1) 288 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.921015+0000 mon.b (mon.1) 289 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:59:24.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:24 smithi098 bash[23790]: audit 2024-04-02T04:59:23.936225+0000 mon.a (mon.0) 3317 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:24.082 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:24.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:24.211 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:24.271 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.271 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.272 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.273 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.274 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.278 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.278 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.278 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.278 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 1 sec 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 512 aios 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99f970300 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fabfdc0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fac07b0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fac0db0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fac13b0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fac19f0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fac1540 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca81f0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca86c0 2024-04-02T04:59:24.279 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca87d0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fac12a0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca88e0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca89f0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8fb0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8b00 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8c80 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8e00 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca90c0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9220 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9bf0 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9800 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9950 2024-04-02T04:59:24.280 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9330 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8320 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8430 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca8540 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa1b0 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9d00 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9440 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9550 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9660 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa770 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa2c0 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9e10 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9f20 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa030 2024-04-02T04:59:24.281 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaad30 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa880 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa3d0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fca9ab0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa4e0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa5f0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaae40 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaaf50 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab510 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab060 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaa990 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaaaa0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcaabb0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcabad0 2024-04-02T04:59:24.282 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab620 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab770 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab880 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab990 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac090 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac1e0 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac310 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac480 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb4e10 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcabbe0 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcabd30 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcabe40 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcabf70 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb53d0 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb4f20 2024-04-02T04:59:24.283 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5070 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5180 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5290 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb59b0 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5500 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5650 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5780 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5ae0 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5fe0 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb60f0 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb66d0 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5bf0 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5d20 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb5e50 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6820 2024-04-02T04:59:24.284 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6220 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab170 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab2c0 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcab3f0 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6e00 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6950 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6330 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6460 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6590 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7400 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6f50 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb70a0 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb71b0 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb72c0 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7a00 2024-04-02T04:59:24.285 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7550 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb76a0 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb77b0 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7b50 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7ca0 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb78e0 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8190 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7df0 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb7f00 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8050 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8750 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb82a0 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6a60 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6b90 2024-04-02T04:59:24.286 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb6cc0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8d50 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb88a0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb89d0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8ae0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8c30 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9330 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8e80 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8fd0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb90e0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9460 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9950 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9ab0 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba090 2024-04-02T04:59:24.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb95b0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9700 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9830 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba1c0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9be0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9d30 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb9e60 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba2f0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba7c0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba8f0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbaed0 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba420 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba530 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcba660 2024-04-02T04:59:24.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb020 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbaa20 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb83b0 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb84e0 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcb8630 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb620 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb170 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb2a0 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb3d0 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb500 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbbc00 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb750 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb880 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbb990 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbbac0 2024-04-02T04:59:24.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc1e0 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbbd30 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbbe60 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbbf90 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc0a0 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc7e0 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc330 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc460 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc590 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc930 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbce20 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbcf50 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbca60 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbcbb0 2024-04-02T04:59:24.290 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbc6c0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbcce0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd550 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd0a0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd1f0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd300 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd430 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbdb50 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd6a0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd7d0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbd8e0 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbda10 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe130 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbdc80 2024-04-02T04:59:24.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbddd0 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbdf40 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe260 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe750 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe880 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe390 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe4e0 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe610 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbe9b0 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbee80 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbefb0 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbeac0 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbec10 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbed60 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf0c0 2024-04-02T04:59:24.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf5b0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf6e0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbfca0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf210 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf360 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf470 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbfdd0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf7f0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbf960 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbfad0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbff00 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc03d0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0500 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0ae0 2024-04-02T04:59:24.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0030 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0140 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0290 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0c30 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0630 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0780 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc08d0 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0d60 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1210 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1320 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1920 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0e70 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc0fc0 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc10d0 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1a30 2024-04-02T04:59:24.294 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1470 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbab30 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbac80 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcbadb0 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2010 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1b60 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1c90 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1da0 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1ed0 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2610 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2160 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc22b0 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc23e0 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2760 2024-04-02T04:59:24.295 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2c50 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2da0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2890 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc29e0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2b10 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2ed0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3380 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc34b0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3a70 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc2fe0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3130 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc35c0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1580 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3260 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc16d0 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc1800 2024-04-02T04:59:24.296 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3bc0 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3d10 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4310 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3e60 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3f90 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc40e0 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4420 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc48f0 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4a20 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5000 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4550 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4680 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc47d0 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5150 2024-04-02T04:59:24.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4b50 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4c80 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4db0 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc4ee0 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5730 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5280 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc36d0 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3820 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc3930 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5d30 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5880 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc59d0 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5b00 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5e80 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6330 2024-04-02T04:59:24.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6460 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6a40 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5f90 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc60e0 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc61f0 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6b90 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6590 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc66c0 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc67f0 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6920 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7190 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6ce0 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6e10 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc6f20 2024-04-02T04:59:24.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7070 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7770 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc72c0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc73f0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7520 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc78c0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7d70 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7ea0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7650 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7ff0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8140 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8700 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc5390 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc54c0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc55f0 2024-04-02T04:59:24.300 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8850 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8250 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc83a0 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc84d0 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8980 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8e50 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8f80 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc85e0 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc90b0 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9200 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8a90 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8bc0 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc8d10 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9330 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9820 2024-04-02T04:59:24.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9950 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9f30 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9480 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc95d0 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9a80 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9bb0 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9cc0 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9df0 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca530 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca080 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca1b0 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc9700 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca2c0 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca410 2024-04-02T04:59:24.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca660 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccab10 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccac40 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb240 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca770 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca8c0 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccad90 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccaee0 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb010 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb390 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb860 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb990 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccbf70 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb4c0 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcca9f0 2024-04-02T04:59:24.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb610 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccb740 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccbac0 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc79d0 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7b20 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcc7c50 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc550 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc0a0 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc210 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc340 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc6a0 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcccb50 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcccc80 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd260 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc7b0 2024-04-02T04:59:24.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccc8e0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccca10 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd3b0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcccdb0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcccee0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd030 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd4e0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd9b0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccdae0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd5f0 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd740 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd890 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccdc10 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce100 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce230 2024-04-02T04:59:24.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce7f0 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccdd60 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccde70 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccdfc0 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccd140 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce340 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce490 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce5a0 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcce960 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccee50 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccef60 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccea90 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccebe0 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcced30 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf070 2024-04-02T04:59:24.306 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf590 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf6e0 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccfca0 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf180 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf2b0 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf400 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccfdd0 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf7f0 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccf940 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccfa90 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fccff00 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0420 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd05a0 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0b80 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0030 2024-04-02T04:59:24.307 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0160 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0290 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0cb0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd06d0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0800 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0930 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0a60 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd12b0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0e00 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd0f50 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1080 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd13e0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd18b0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd19e0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd14f0 2024-04-02T04:59:24.308 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1640 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1190 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1770 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1fc0 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1b10 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1c60 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd1db0 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2110 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd25e0 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2710 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2cf0 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2240 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2370 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd24a0 2024-04-02T04:59:24.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2e20 2024-04-02T04:59:24.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2840 2024-04-02T04:59:24.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2990 2024-04-02T04:59:24.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2aa0 2024-04-02T04:59:24.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2bd0 2024-04-02T04:59:24.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3400 2024-04-02T04:59:24.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd2f50 2024-04-02T04:59:24.310 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.032761+0000 mon.a (mon.0) 3315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: cluster 2024-04-02T04:59:23.044506+0000 mon.a (mon.0) 3316 : cluster 0 osdmap e665: 8 total, 8 up, 7 in 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.233969+0000 mon.b (mon.1) 287 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.262097+0000 mon.c (mon.2) 1129 : audit 0 from='client.? 172.21.15.69:0/1467377700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.536209+0000 mgr.y (mgr.14538) 612 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.918942+0000 mon.b (mon.1) 288 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.921015+0000 mon.b (mon.1) 289 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[21528]: audit 2024-04-02T04:59:23.936225+0000 mon.a (mon.0) 3317 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.032761+0000 mon.a (mon.0) 3315 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: cluster 2024-04-02T04:59:23.044506+0000 mon.a (mon.0) 3316 : cluster 0 osdmap e665: 8 total, 8 up, 7 in 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.233969+0000 mon.b (mon.1) 287 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:59:24.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.262097+0000 mon.c (mon.2) 1129 : audit 0 from='client.? 172.21.15.69:0/1467377700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:24.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.536209+0000 mgr.y (mgr.14538) 612 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:24.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.918942+0000 mon.b (mon.1) 288 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:59:24.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.921015+0000 mon.b (mon.1) 289 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:59:24.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:24 smithi069 bash[28904]: audit 2024-04-02T04:59:23.936225+0000 mon.a (mon.0) 3317 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T04:59:24.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:24.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:24.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:24.471 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:24.574 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:24.603 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:24.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:24.732 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:24.835 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:24.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:24.986 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:24.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:25.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:25.111 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:25.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:25.225 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: cluster 2024-04-02T04:59:24.042639+0000 mon.a (mon.0) 3318 : cluster 0 osdmap e666: 8 total, 8 up, 7 in 2024-04-02T04:59:25.225 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: audit 2024-04-02T04:59:24.049276+0000 mon.c (mon.2) 1130 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: audit 2024-04-02T04:59:24.061634+0000 mon.a (mon.0) 3319 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: cluster 2024-04-02T04:59:24.258576+0000 mgr.y (mgr.14538) 613 : cluster 0 pgmap v1005: 310 pgs: 6 creating+activating, 1 peering, 22 creating+peering, 25 unknown, 256 active+clean; 635 KiB data, 605 MiB used, 625 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 39 KiB/s, 11 objects/s recovering 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: audit 2024-04-02T04:59:25.017602+0000 mon.a (mon.0) 3320 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: cluster 2024-04-02T04:59:25.024650+0000 mon.a (mon.0) 3321 : cluster 0 osdmap e667: 8 total, 8 up, 7 in 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: audit 2024-04-02T04:59:25.036916+0000 mon.c (mon.2) 1131 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[21528]: audit 2024-04-02T04:59:25.039610+0000 mon.a (mon.0) 3322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: cluster 2024-04-02T04:59:24.042639+0000 mon.a (mon.0) 3318 : cluster 0 osdmap e666: 8 total, 8 up, 7 in 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: audit 2024-04-02T04:59:24.049276+0000 mon.c (mon.2) 1130 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: audit 2024-04-02T04:59:24.061634+0000 mon.a (mon.0) 3319 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: cluster 2024-04-02T04:59:24.258576+0000 mgr.y (mgr.14538) 613 : cluster 0 pgmap v1005: 310 pgs: 6 creating+activating, 1 peering, 22 creating+peering, 25 unknown, 256 active+clean; 635 KiB data, 605 MiB used, 625 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 39 KiB/s, 11 objects/s recovering 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: audit 2024-04-02T04:59:25.017602+0000 mon.a (mon.0) 3320 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:25.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: cluster 2024-04-02T04:59:25.024650+0000 mon.a (mon.0) 3321 : cluster 0 osdmap e667: 8 total, 8 up, 7 in 2024-04-02T04:59:25.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: audit 2024-04-02T04:59:25.036916+0000 mon.c (mon.2) 1131 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:25.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:25 smithi069 bash[28904]: audit 2024-04-02T04:59:25.039610+0000 mon.a (mon.0) 3322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:25.241 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:25.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: cluster 2024-04-02T04:59:24.042639+0000 mon.a (mon.0) 3318 : cluster 0 osdmap e666: 8 total, 8 up, 7 in 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: audit 2024-04-02T04:59:24.049276+0000 mon.c (mon.2) 1130 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: audit 2024-04-02T04:59:24.061634+0000 mon.a (mon.0) 3319 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: cluster 2024-04-02T04:59:24.258576+0000 mgr.y (mgr.14538) 613 : cluster 0 pgmap v1005: 310 pgs: 6 creating+activating, 1 peering, 22 creating+peering, 25 unknown, 256 active+clean; 635 KiB data, 605 MiB used, 625 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 39 KiB/s, 11 objects/s recovering 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: audit 2024-04-02T04:59:25.017602+0000 mon.a (mon.0) 3320 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: cluster 2024-04-02T04:59:25.024650+0000 mon.a (mon.0) 3321 : cluster 0 osdmap e667: 8 total, 8 up, 7 in 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: audit 2024-04-02T04:59:25.036916+0000 mon.c (mon.2) 1131 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:25.356 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:25 smithi098 bash[23790]: audit 2024-04-02T04:59:25.039610+0000 mon.a (mon.0) 3322 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:25.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:25.496 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:25.598 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:25.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:25.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:25.754 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:25.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:25.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:25.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:26.015 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:26.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:26.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:26.272 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:26.322 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:59:26.323 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:59:26.338 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T04:59:26.338 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T04:59:26.338 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-02T04:59:26.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-02T04:59:26.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:26.404 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:26.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:26.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:26.664 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:26.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:26.794 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:26.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:26.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:26.971 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T04:59:27.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:27.158 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-02T04:59:27.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:27.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[28904]: audit 2024-04-02T04:59:26.020158+0000 mon.a (mon.0) 3323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[28904]: cluster 2024-04-02T04:59:26.028690+0000 mon.a (mon.0) 3324 : cluster 0 osdmap e668: 8 total, 8 up, 7 in 2024-04-02T04:59:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[28904]: audit 2024-04-02T04:59:26.031847+0000 mon.c (mon.2) 1132 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[28904]: audit 2024-04-02T04:59:26.039258+0000 mon.a (mon.0) 3325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:27.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[28904]: cluster 2024-04-02T04:59:26.259403+0000 mgr.y (mgr.14538) 614 : cluster 0 pgmap v1008: 246 pgs: 6 creating+activating, 1 peering, 22 creating+peering, 25 unknown, 192 active+clean; 635 KiB data, 605 MiB used, 625 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 39 KiB/s, 11 objects/s recovering 2024-04-02T04:59:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[21528]: audit 2024-04-02T04:59:26.020158+0000 mon.a (mon.0) 3323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[21528]: cluster 2024-04-02T04:59:26.028690+0000 mon.a (mon.0) 3324 : cluster 0 osdmap e668: 8 total, 8 up, 7 in 2024-04-02T04:59:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[21528]: audit 2024-04-02T04:59:26.031847+0000 mon.c (mon.2) 1132 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[21528]: audit 2024-04-02T04:59:26.039258+0000 mon.a (mon.0) 3325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:27.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:27 smithi069 bash[21528]: cluster 2024-04-02T04:59:26.259403+0000 mgr.y (mgr.14538) 614 : cluster 0 pgmap v1008: 246 pgs: 6 creating+activating, 1 peering, 22 creating+peering, 25 unknown, 192 active+clean; 635 KiB data, 605 MiB used, 625 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 39 KiB/s, 11 objects/s recovering 2024-04-02T04:59:27.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:27.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:27 smithi098 bash[23790]: audit 2024-04-02T04:59:26.020158+0000 mon.a (mon.0) 3323 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:27.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:27 smithi098 bash[23790]: cluster 2024-04-02T04:59:26.028690+0000 mon.a (mon.0) 3324 : cluster 0 osdmap e668: 8 total, 8 up, 7 in 2024-04-02T04:59:27.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:27 smithi098 bash[23790]: audit 2024-04-02T04:59:26.031847+0000 mon.c (mon.2) 1132 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:27.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:27 smithi098 bash[23790]: audit 2024-04-02T04:59:26.039258+0000 mon.a (mon.0) 3325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:27.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:27 smithi098 bash[23790]: cluster 2024-04-02T04:59:26.259403+0000 mgr.y (mgr.14538) 614 : cluster 0 pgmap v1008: 246 pgs: 6 creating+activating, 1 peering, 22 creating+peering, 25 unknown, 192 active+clean; 635 KiB data, 605 MiB used, 625 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 39 KiB/s, 11 objects/s recovering 2024-04-02T04:59:27.308 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:27.411 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:27.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:27.566 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:27.638 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:27.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:27.695 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:27.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:27.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:27.963 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:27.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:28.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:28.093 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:28.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: cluster 2024-04-02T04:59:27.020141+0000 mon.a (mon.0) 3326 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: audit 2024-04-02T04:59:27.031849+0000 mon.a (mon.0) 3327 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: cluster 2024-04-02T04:59:27.037421+0000 mon.a (mon.0) 3328 : cluster 0 osdmap e669: 8 total, 8 up, 7 in 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: audit 2024-04-02T04:59:27.048064+0000 mon.c (mon.2) 1133 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: audit 2024-04-02T04:59:27.050746+0000 mon.c (mon.2) 1134 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: audit 2024-04-02T04:59:27.051515+0000 mon.a (mon.0) 3329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:28.209 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:28 smithi098 bash[23790]: audit 2024-04-02T04:59:27.052159+0000 mon.a (mon.0) 3330 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:28.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:28.354 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:28.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: cluster 2024-04-02T04:59:27.020141+0000 mon.a (mon.0) 3326 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: audit 2024-04-02T04:59:27.031849+0000 mon.a (mon.0) 3327 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: cluster 2024-04-02T04:59:27.037421+0000 mon.a (mon.0) 3328 : cluster 0 osdmap e669: 8 total, 8 up, 7 in 2024-04-02T04:59:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: audit 2024-04-02T04:59:27.048064+0000 mon.c (mon.2) 1133 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:28.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: audit 2024-04-02T04:59:27.050746+0000 mon.c (mon.2) 1134 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: audit 2024-04-02T04:59:27.051515+0000 mon.a (mon.0) 3329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21528]: audit 2024-04-02T04:59:27.052159+0000 mon.a (mon.0) 3330 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: cluster 2024-04-02T04:59:27.020141+0000 mon.a (mon.0) 3326 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: audit 2024-04-02T04:59:27.031849+0000 mon.a (mon.0) 3327 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-107", "overlaypool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: cluster 2024-04-02T04:59:27.037421+0000 mon.a (mon.0) 3328 : cluster 0 osdmap e669: 8 total, 8 up, 7 in 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: audit 2024-04-02T04:59:27.048064+0000 mon.c (mon.2) 1133 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: audit 2024-04-02T04:59:27.050746+0000 mon.c (mon.2) 1134 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: audit 2024-04-02T04:59:27.051515+0000 mon.a (mon.0) 3329 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:28.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[28904]: audit 2024-04-02T04:59:27.052159+0000 mon.a (mon.0) 3330 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:28.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:28.614 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:28.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:28.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:28.851 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:59:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:59:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:59:28.864 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:28.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:28.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:29.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:29.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: cluster 2024-04-02T04:59:28.033720+0000 mon.a (mon.0) 3331 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.045795+0000 mon.a (mon.0) 3332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]': finished 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.046044+0000 mon.a (mon.0) 3333 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: cluster 2024-04-02T04:59:28.056017+0000 mon.a (mon.0) 3334 : cluster 0 osdmap e670: 8 total, 8 up, 7 in 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.111163+0000 mon.c (mon.2) 1135 : audit 0 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.113949+0000 mon.c (mon.2) 1136 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]: dispatch 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.114246+0000 mon.c (mon.2) 1137 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.114504+0000 mon.a (mon.0) 3335 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]: dispatch 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: audit 2024-04-02T04:59:28.114818+0000 mon.a (mon.0) 3336 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:29.225 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[28904]: cluster 2024-04-02T04:59:28.261193+0000 mgr.y (mgr.14538) 615 : cluster 0 pgmap v1011: 278 pgs: 10 creating+activating, 13 creating+peering, 5 unknown, 250 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: cluster 2024-04-02T04:59:28.033720+0000 mon.a (mon.0) 3331 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.045795+0000 mon.a (mon.0) 3332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]': finished 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.046044+0000 mon.a (mon.0) 3333 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: cluster 2024-04-02T04:59:28.056017+0000 mon.a (mon.0) 3334 : cluster 0 osdmap e670: 8 total, 8 up, 7 in 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.111163+0000 mon.c (mon.2) 1135 : audit 0 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.113949+0000 mon.c (mon.2) 1136 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]: dispatch 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.114246+0000 mon.c (mon.2) 1137 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.114504+0000 mon.a (mon.0) 3335 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]: dispatch 2024-04-02T04:59:29.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: audit 2024-04-02T04:59:28.114818+0000 mon.a (mon.0) 3336 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:29.227 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:29 smithi069 bash[21528]: cluster 2024-04-02T04:59:28.261193+0000 mgr.y (mgr.14538) 615 : cluster 0 pgmap v1011: 278 pgs: 10 creating+activating, 13 creating+peering, 5 unknown, 250 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:59:29.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:29.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:29.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: cluster 2024-04-02T04:59:28.033720+0000 mon.a (mon.0) 3331 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:29.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.045795+0000 mon.a (mon.0) 3332 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-107-cache", "mode": "writeback"}]': finished 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.046044+0000 mon.a (mon.0) 3333 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: cluster 2024-04-02T04:59:28.056017+0000 mon.a (mon.0) 3334 : cluster 0 osdmap e670: 8 total, 8 up, 7 in 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.111163+0000 mon.c (mon.2) 1135 : audit 0 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.113949+0000 mon.c (mon.2) 1136 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]: dispatch 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.114246+0000 mon.c (mon.2) 1137 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.114504+0000 mon.a (mon.0) 3335 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]: dispatch 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: audit 2024-04-02T04:59:28.114818+0000 mon.a (mon.0) 3336 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:29.371 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:29 smithi098 bash[23790]: cluster 2024-04-02T04:59:28.261193+0000 mgr.y (mgr.14538) 615 : cluster 0 pgmap v1011: 278 pgs: 10 creating+activating, 13 creating+peering, 5 unknown, 250 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:59:29.385 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:29.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:29.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:29.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:29.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:29.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:29.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:29.908 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:29.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:29.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:30.011 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:30.037 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:30.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:30.162 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:30.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.052672+0000 mon.a (mon.0) 3337 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]': finished 2024-04-02T04:59:30.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.052759+0000 mon.a (mon.0) 3338 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:30.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: cluster 2024-04-02T04:59:29.056110+0000 mon.a (mon.0) 3339 : cluster 0 osdmap e671: 8 total, 8 up, 7 in 2024-04-02T04:59:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.056957+0000 mon.c (mon.2) 1138 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.057466+0000 mon.a (mon.0) 3340 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.058105+0000 mon.c (mon.2) 1139 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2"}]: dispatch 2024-04-02T04:59:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.059105+0000 mon.c (mon.2) 1140 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:30.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:30 smithi098 bash[23790]: audit 2024-04-02T04:59:29.059583+0000 mon.a (mon.0) 3341 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:30.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.052672+0000 mon.a (mon.0) 3337 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]': finished 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.052759+0000 mon.a (mon.0) 3338 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: cluster 2024-04-02T04:59:29.056110+0000 mon.a (mon.0) 3339 : cluster 0 osdmap e671: 8 total, 8 up, 7 in 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.056957+0000 mon.c (mon.2) 1138 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.057466+0000 mon.a (mon.0) 3340 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.058105+0000 mon.c (mon.2) 1139 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2"}]: dispatch 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.059105+0000 mon.c (mon.2) 1140 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[21528]: audit 2024-04-02T04:59:29.059583+0000 mon.a (mon.0) 3341 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.052672+0000 mon.a (mon.0) 3337 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-107"}]': finished 2024-04-02T04:59:30.277 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.052759+0000 mon.a (mon.0) 3338 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: cluster 2024-04-02T04:59:29.056110+0000 mon.a (mon.0) 3339 : cluster 0 osdmap e671: 8 total, 8 up, 7 in 2024-04-02T04:59:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.056957+0000 mon.c (mon.2) 1138 : audit 1 from='client.? 172.21.15.69:0/3385845597' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.057466+0000 mon.a (mon.0) 3340 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]: dispatch 2024-04-02T04:59:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.058105+0000 mon.c (mon.2) 1139 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2"}]: dispatch 2024-04-02T04:59:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.059105+0000 mon.c (mon.2) 1140 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:30.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:30 smithi069 bash[28904]: audit 2024-04-02T04:59:29.059583+0000 mon.a (mon.0) 3341 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:30.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:30.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:30.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:30.549 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:30.652 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:30.675 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:30.679 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:30.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:30.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:30.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:31.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.072 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:31.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:31.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:31.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: cluster 2024-04-02T04:59:30.052946+0000 mon.a (mon.0) 3342 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:31.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: audit 2024-04-02T04:59:30.054684+0000 mon.a (mon.0) 3343 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:31.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: audit 2024-04-02T04:59:30.054787+0000 mon.a (mon.0) 3344 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:31.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: cluster 2024-04-02T04:59:30.058132+0000 mon.a (mon.0) 3345 : cluster 0 osdmap e672: 8 total, 8 up, 7 in 2024-04-02T04:59:31.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: audit 2024-04-02T04:59:30.072969+0000 mon.c (mon.2) 1141 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-02T04:59:31.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: audit 2024-04-02T04:59:30.073376+0000 mon.c (mon.2) 1142 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:59:31.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: audit 2024-04-02T04:59:30.073746+0000 mon.a (mon.0) 3346 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:59:31.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[28904]: cluster 2024-04-02T04:59:30.262053+0000 mgr.y (mgr.14538) 616 : cluster 0 pgmap v1014: 278 pgs: 10 creating+activating, 6 creating+peering, 262 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: cluster 2024-04-02T04:59:30.052946+0000 mon.a (mon.0) 3342 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: audit 2024-04-02T04:59:30.054684+0000 mon.a (mon.0) 3343 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: audit 2024-04-02T04:59:30.054787+0000 mon.a (mon.0) 3344 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: cluster 2024-04-02T04:59:30.058132+0000 mon.a (mon.0) 3345 : cluster 0 osdmap e672: 8 total, 8 up, 7 in 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: audit 2024-04-02T04:59:30.072969+0000 mon.c (mon.2) 1141 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: audit 2024-04-02T04:59:30.073376+0000 mon.c (mon.2) 1142 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:59:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: audit 2024-04-02T04:59:30.073746+0000 mon.a (mon.0) 3346 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:59:31.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:31 smithi069 bash[21528]: cluster 2024-04-02T04:59:30.262053+0000 mgr.y (mgr.14538) 616 : cluster 0 pgmap v1014: 278 pgs: 10 creating+activating, 6 creating+peering, 262 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:59:31.331 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:31.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:31.458 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: cluster 2024-04-02T04:59:30.052946+0000 mon.a (mon.0) 3342 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: audit 2024-04-02T04:59:30.054684+0000 mon.a (mon.0) 3343 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-107", "tierpool": "test-rados-api-smithi069-72349-107-cache"}]': finished 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: audit 2024-04-02T04:59:30.054787+0000 mon.a (mon.0) 3344 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-72272-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: cluster 2024-04-02T04:59:30.058132+0000 mon.a (mon.0) 3345 : cluster 0 osdmap e672: 8 total, 8 up, 7 in 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: audit 2024-04-02T04:59:30.072969+0000 mon.c (mon.2) 1141 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: audit 2024-04-02T04:59:30.073376+0000 mon.c (mon.2) 1142 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: audit 2024-04-02T04:59:30.073746+0000 mon.a (mon.0) 3346 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-02T04:59:31.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:31 smithi098 bash[23790]: cluster 2024-04-02T04:59:30.262053+0000 mgr.y (mgr.14538) 616 : cluster 0 pgmap v1014: 278 pgs: 10 creating+activating, 6 creating+peering, 262 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T04:59:31.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:31.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:31.716 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:31.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 -- 172.21.15.69:0/2519465229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe23410a540 msgr2=0x7fe23410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 --2- 172.21.15.69:0/2519465229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe23410a540 0x7fe23410a920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe22c009f90 tx=0x7fe22c00b330 comp rx=0 tx=0).stop 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 -- 172.21.15.69:0/2519465229 shutdown_connections 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 --2- 172.21.15.69:0/2519465229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe234104e50 0x7fe234111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 --2- 172.21.15.69:0/2519465229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe23410a540 0x7fe23410a920 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 --2- 172.21.15.69:0/2519465229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104490 0x7fe234104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 -- 172.21.15.69:0/2519465229 >> 172.21.15.69:0/2519465229 conn(0x7fe2340fff60 msgr2=0x7fe234102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.727+0000 7fe2392fb700 1 -- 172.21.15.69:0/2519465229 shutdown_connections 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- 172.21.15.69:0/2519465229 wait complete. 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 librados: starting msgr at 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 librados: starting objecter 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 Processor -- start 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- start start 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 librados: setting wanted keys 2024-04-02T04:59:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 librados: calling monclient init 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 0x7fe23419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104e50 0x7fe23419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23410a540 0x7fe23419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe234113c40 con 0x7fe234104e50 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe234113ac0 con 0x7fe23410a540 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe234113dc0 con 0x7fe234104490 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 0x7fe23419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 0x7fe23419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55268/0 (socket says 172.21.15.69:55268) 2024-04-02T04:59:31.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 -- 172.21.15.69:0/3353671421 learned_addr learned my addr 172.21.15.69:0/3353671421 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2327fc700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104e50 0x7fe23419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2337fe700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23410a540 0x7fe23419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 -- 172.21.15.69:0/3353671421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23410a540 msgr2=0x7fe23419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23410a540 0x7fe23419eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 -- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104e50 msgr2=0x7fe23419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104e50 0x7fe23419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 -- 172.21.15.69:0/3353671421 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe23419f220 con 0x7fe234104490 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2327fc700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104e50 0x7fe23419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe232ffd700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 0x7fe23419a210 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe22c004fd0 tx=0x7fe22c00cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe21bfff700 10 client.?.objecter ms_handle_connect 0x7fe234104490 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe21bfff700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22c021070 con 0x7fe234104490 2024-04-02T04:59:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe22c01d450 con 0x7fe234104490 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22c0052a0 con 0x7fe234104490 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe23419f4b0 con 0x7fe234104490 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 10 client.26861.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2341a6d90 con 0x7fe234104490 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 librados: init done 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe234105bb0 con 0x7fe234104490 2024-04-02T04:59:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.731+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe22c024070 con 0x7fe234104490 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe21c06f5e0 0x7fe21c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe2327fc700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe21c06f5e0 0x7fe21c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(673..673 src has 74..673) ==== 10312+0+0 (secure 0 0 0) 0x7fe22c00f020 con 0x7fe234104490 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 10 client.26861.objecter ms_dispatch 0x7fe234001550 osd_map(673..673 src has 74..673) 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 3 client.26861.objecter handle_osd_map got epochs [673,673] > 0 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 3 client.26861.objecter handle_osd_map decoding full epoch 673 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 20 client.26861.objecter dump_active .. 0 homeless 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe21bfff700 10 client.26861.objecter ms_handle_connect 0x7fe21c06f5e0 2024-04-02T04:59:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.735+0000 7fe2327fc700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe21c06f5e0 0x7fe21c071aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe228009750 tx=0x7fe228009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:31.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.739+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe22c02b020 con 0x7fe234104490 2024-04-02T04:59:31.818 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:31.839 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:31.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:31.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:31.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:31.971+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe23419b570 con 0x7fe234104490 2024-04-02T04:59:32.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:32.091 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:32.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.163+0000 7fe21bfff700 1 -- 172.21.15.69:0/3353671421 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v674) ==== 97+0+0 (secure 0 0 0) 0x7fe22c05cb10 con 0x7fe234104490 2024-04-02T04:59:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 10 librados: watch_flush enter 2024-04-02T04:59:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 10 librados: watch_flush exit 2024-04-02T04:59:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 20 client.26861.objecter shutdown clearing up homeless session... 2024-04-02T04:59:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 10 client.26861.objecter successfully canceled tick 2024-04-02T04:59:32.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe21c06f5e0 msgr2=0x7fe21c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:32.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe21c06f5e0 0x7fe21c071aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe228009750 tx=0x7fe228009310 comp rx=0 tx=0).stop 2024-04-02T04:59:32.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 msgr2=0x7fe23419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:32.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.171+0000 7fe2392fb700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 0x7fe23419a210 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe22c004fd0 tx=0x7fe22c00cd80 comp rx=0 tx=0).stop 2024-04-02T04:59:32.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 shutdown_connections 2024-04-02T04:59:32.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe21c06f5e0 0x7fe21c071aa0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:32.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23410a540 0x7fe23419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:32.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe234104490 0x7fe23419a210 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:32.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 --2- 172.21.15.69:0/3353671421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe234104e50 0x7fe23419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:32.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 >> 172.21.15.69:0/3353671421 conn(0x7fe2340fff60 msgr2=0x7fe234108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:32.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 shutdown_connections 2024-04-02T04:59:32.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 -- 172.21.15.69:0/3353671421 wait complete. 2024-04-02T04:59:32.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:32.179+0000 7fe2392fb700 1 librados: shutdown 2024-04-02T04:59:32.185 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T04:59:32.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:32.219 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:32.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:32.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: audit 2024-04-02T04:59:31.155957+0000 mon.a (mon.0) 3347 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-02T04:59:32.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: cluster 2024-04-02T04:59:31.162700+0000 mon.a (mon.0) 3348 : cluster 0 osdmap e673: 8 total, 8 up, 7 in 2024-04-02T04:59:32.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: audit 2024-04-02T04:59:31.170423+0000 mon.c (mon.2) 1143 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:59:32.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: audit 2024-04-02T04:59:31.170993+0000 mon.a (mon.0) 3349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:59:32.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: cluster 2024-04-02T04:59:31.295793+0000 osd.6 (osd.6) 291 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:32.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: cluster 2024-04-02T04:59:31.296476+0000 osd.6 (osd.6) 292 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:32.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: audit 2024-04-02T04:59:31.973642+0000 mon.c (mon.2) 1144 : audit 1 from='client.? 172.21.15.69:0/3353671421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:59:32.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:32 smithi098 bash[23790]: audit 2024-04-02T04:59:31.973989+0000 mon.a (mon.0) 3350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:59:32.350 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:32.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: audit 2024-04-02T04:59:31.155957+0000 mon.a (mon.0) 3347 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-02T04:59:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: cluster 2024-04-02T04:59:31.162700+0000 mon.a (mon.0) 3348 : cluster 0 osdmap e673: 8 total, 8 up, 7 in 2024-04-02T04:59:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: audit 2024-04-02T04:59:31.170423+0000 mon.c (mon.2) 1143 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:59:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: audit 2024-04-02T04:59:31.170993+0000 mon.a (mon.0) 3349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:59:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: cluster 2024-04-02T04:59:31.295793+0000 osd.6 (osd.6) 291 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:32.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: cluster 2024-04-02T04:59:31.296476+0000 osd.6 (osd.6) 292 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: audit 2024-04-02T04:59:31.973642+0000 mon.c (mon.2) 1144 : audit 1 from='client.? 172.21.15.69:0/3353671421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[21528]: audit 2024-04-02T04:59:31.973989+0000 mon.a (mon.0) 3350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: audit 2024-04-02T04:59:31.155957+0000 mon.a (mon.0) 3347 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: cluster 2024-04-02T04:59:31.162700+0000 mon.a (mon.0) 3348 : cluster 0 osdmap e673: 8 total, 8 up, 7 in 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: audit 2024-04-02T04:59:31.170423+0000 mon.c (mon.2) 1143 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: audit 2024-04-02T04:59:31.170993+0000 mon.a (mon.0) 3349 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: cluster 2024-04-02T04:59:31.295793+0000 osd.6 (osd.6) 291 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: cluster 2024-04-02T04:59:31.296476+0000 osd.6 (osd.6) 292 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: audit 2024-04-02T04:59:31.973642+0000 mon.c (mon.2) 1144 : audit 1 from='client.? 172.21.15.69:0/3353671421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:59:32.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:32 smithi069 bash[28904]: audit 2024-04-02T04:59:31.973989+0000 mon.a (mon.0) 3350 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T04:59:32.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:32.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:32.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:32.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:32.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:33.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:33.127 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:33.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd30a0 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd31d0 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3550 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3a20 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3ba0 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4180 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3680 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd37b0 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd38e0 2024-04-02T04:59:33.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd42b0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3cd0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3e20 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd3f50 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd43e0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd48b0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd49e0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4fc0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4510 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4640 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4770 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd50f0 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4b10 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4c60 2024-04-02T04:59:33.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4d90 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5220 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd56f0 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5820 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd4ea0 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5950 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5a80 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5330 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5440 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5590 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5bb0 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd60f0 2024-04-02T04:59:33.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd6240 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd6800 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5d00 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5e30 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd5f80 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd6950 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd6350 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd64a0 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcd65d0 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcacab0 2024-04-02T04:59:33.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac600 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac730 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac860 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcac990 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcad090 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcacbe0 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcacd10 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcace20 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcacf70 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcad670 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcad1c0 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcad310 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcad420 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcad7a0 2024-04-02T04:59:33.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55c99fcadc70 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: done. 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (613099 ms) 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (613100 ms total) 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Exec (38 ms) 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (7 ms) 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-04-02T04:59:33.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-02T04:59:33.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5064 ms) 2024-04-02T04:59:33.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-02T04:59:33.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:33.258 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:33.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: cluster 2024-04-02T04:59:31.974527+0000 osd.0 (osd.0) 341 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:33.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: cluster 2024-04-02T04:59:31.974695+0000 osd.0 (osd.0) 342 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:33.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: cluster 2024-04-02T04:59:32.156400+0000 mon.a (mon.0) 3351 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:33.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.158697+0000 mon.a (mon.0) 3352 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-02T04:59:33.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.158900+0000 mon.a (mon.0) 3353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:59:33.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: cluster 2024-04-02T04:59:32.166319+0000 mon.a (mon.0) 3354 : cluster 0 osdmap e674: 8 total, 8 up, 7 in 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.170779+0000 mon.c (mon.2) 1145 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.176143+0000 mon.a (mon.0) 3355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.205550+0000 mon.a (mon.0) 3356 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.206976+0000 mon.a (mon.0) 3357 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:32.207559+0000 mon.a (mon.0) 3358 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: cluster 2024-04-02T04:59:32.262689+0000 mgr.y (mgr.14538) 617 : cluster 0 pgmap v1017: 214 pgs: 10 creating+activating, 6 creating+peering, 198 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:33.161458+0000 mon.a (mon.0) 3359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]': finished 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: audit 2024-04-02T04:59:33.161728+0000 mon.a (mon.0) 3360 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[21528]: cluster 2024-04-02T04:59:33.172919+0000 mon.a (mon.0) 3361 : cluster 0 osdmap e675: 8 total, 8 up, 7 in 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: cluster 2024-04-02T04:59:31.974527+0000 osd.0 (osd.0) 341 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:33.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: cluster 2024-04-02T04:59:31.974695+0000 osd.0 (osd.0) 342 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: cluster 2024-04-02T04:59:32.156400+0000 mon.a (mon.0) 3351 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.158697+0000 mon.a (mon.0) 3352 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.158900+0000 mon.a (mon.0) 3353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: cluster 2024-04-02T04:59:32.166319+0000 mon.a (mon.0) 3354 : cluster 0 osdmap e674: 8 total, 8 up, 7 in 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.170779+0000 mon.c (mon.2) 1145 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.176143+0000 mon.a (mon.0) 3355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.205550+0000 mon.a (mon.0) 3356 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.206976+0000 mon.a (mon.0) 3357 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:32.207559+0000 mon.a (mon.0) 3358 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: cluster 2024-04-02T04:59:32.262689+0000 mgr.y (mgr.14538) 617 : cluster 0 pgmap v1017: 214 pgs: 10 creating+activating, 6 creating+peering, 198 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:33.161458+0000 mon.a (mon.0) 3359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]': finished 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: audit 2024-04-02T04:59:33.161728+0000 mon.a (mon.0) 3360 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:33.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:33 smithi069 bash[28904]: cluster 2024-04-02T04:59:33.172919+0000 mon.a (mon.0) 3361 : cluster 0 osdmap e675: 8 total, 8 up, 7 in 2024-04-02T04:59:33.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: cluster 2024-04-02T04:59:31.974527+0000 osd.0 (osd.0) 341 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: cluster 2024-04-02T04:59:31.974695+0000 osd.0 (osd.0) 342 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: cluster 2024-04-02T04:59:32.156400+0000 mon.a (mon.0) 3351 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.158697+0000 mon.a (mon.0) 3352 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-02T04:59:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.158900+0000 mon.a (mon.0) 3353 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T04:59:33.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: cluster 2024-04-02T04:59:32.166319+0000 mon.a (mon.0) 3354 : cluster 0 osdmap e674: 8 total, 8 up, 7 in 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.170779+0000 mon.c (mon.2) 1145 : audit 1 from='client.? 172.21.15.69:0/353879912' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.176143+0000 mon.a (mon.0) 3355 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]: dispatch 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.205550+0000 mon.a (mon.0) 3356 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.206976+0000 mon.a (mon.0) 3357 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:32.207559+0000 mon.a (mon.0) 3358 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: cluster 2024-04-02T04:59:32.262689+0000 mgr.y (mgr.14538) 617 : cluster 0 pgmap v1017: 214 pgs: 10 creating+activating, 6 creating+peering, 198 active+clean; 635 KiB data, 606 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:33.161458+0000 mon.a (mon.0) 3359 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-72272-1","app":"app1","key":"key1"}]': finished 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: audit 2024-04-02T04:59:33.161728+0000 mon.a (mon.0) 3360 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:33.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[23790]: cluster 2024-04-02T04:59:33.172919+0000 mon.a (mon.0) 3361 : cluster 0 osdmap e675: 8 total, 8 up, 7 in 2024-04-02T04:59:33.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:33.538 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:33.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:33.670 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:33.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:33.805 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:33.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:33.936 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:33.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:33.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:59:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:59:34.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:34.178 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:34.180 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:34.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:34.231 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:34.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:34.391 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:34.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:34.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:34.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:34.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:34 smithi069 bash[21528]: audit 2024-04-02T04:59:33.177612+0000 mon.a (mon.0) 3362 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:34.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:34 smithi069 bash[21528]: audit 2024-04-02T04:59:33.547182+0000 mgr.y (mgr.14538) 618 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:34.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:34 smithi069 bash[21528]: audit 2024-04-02T04:59:33.749727+0000 mon.b (mon.1) 290 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:34.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:34.760 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:34 smithi098 bash[23790]: audit 2024-04-02T04:59:33.177612+0000 mon.a (mon.0) 3362 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:34.760 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:34 smithi098 bash[23790]: audit 2024-04-02T04:59:33.547182+0000 mgr.y (mgr.14538) 618 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:34.760 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:34 smithi098 bash[23790]: audit 2024-04-02T04:59:33.749727+0000 mon.b (mon.1) 290 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:34.773 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:34.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:34.901 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:34.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:35.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:35.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:34 smithi069 bash[28904]: audit 2024-04-02T04:59:33.177612+0000 mon.a (mon.0) 3362 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:35.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:34 smithi069 bash[28904]: audit 2024-04-02T04:59:33.547182+0000 mgr.y (mgr.14538) 618 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:35.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:34 smithi069 bash[28904]: audit 2024-04-02T04:59:33.749727+0000 mon.b (mon.1) 290 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:35.056 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:35.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:35.224 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:35.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:35.413 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:35.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:35.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:35.715 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:35.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:35.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:35 smithi098 bash[23790]: cluster 2024-04-02T04:59:34.263153+0000 mgr.y (mgr.14538) 619 : cluster 0 pgmap v1020: 214 pgs: 214 active+clean; 635 KiB data, 628 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:59:35.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:35 smithi098 bash[23790]: cluster 2024-04-02T04:59:34.680579+0000 mon.a (mon.0) 3363 : cluster 0 osdmap e676: 8 total, 8 up, 7 in 2024-04-02T04:59:35.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:35 smithi098 bash[23790]: audit 2024-04-02T04:59:35.020213+0000 mon.a (mon.0) 3364 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:35.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:35 smithi098 bash[23790]: cluster 2024-04-02T04:59:35.024212+0000 mon.a (mon.0) 3365 : cluster 0 osdmap e677: 8 total, 8 up, 7 in 2024-04-02T04:59:35.847 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:35.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:35.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:36.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[21528]: cluster 2024-04-02T04:59:34.263153+0000 mgr.y (mgr.14538) 619 : cluster 0 pgmap v1020: 214 pgs: 214 active+clean; 635 KiB data, 628 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:59:36.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[21528]: cluster 2024-04-02T04:59:34.680579+0000 mon.a (mon.0) 3363 : cluster 0 osdmap e676: 8 total, 8 up, 7 in 2024-04-02T04:59:36.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[21528]: audit 2024-04-02T04:59:35.020213+0000 mon.a (mon.0) 3364 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:36.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[21528]: cluster 2024-04-02T04:59:35.024212+0000 mon.a (mon.0) 3365 : cluster 0 osdmap e677: 8 total, 8 up, 7 in 2024-04-02T04:59:36.048 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[28904]: cluster 2024-04-02T04:59:34.263153+0000 mgr.y (mgr.14538) 619 : cluster 0 pgmap v1020: 214 pgs: 214 active+clean; 635 KiB data, 628 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:59:36.048 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[28904]: cluster 2024-04-02T04:59:34.680579+0000 mon.a (mon.0) 3363 : cluster 0 osdmap e676: 8 total, 8 up, 7 in 2024-04-02T04:59:36.048 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[28904]: audit 2024-04-02T04:59:35.020213+0000 mon.a (mon.0) 3364 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:36.048 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:35 smithi069 bash[28904]: cluster 2024-04-02T04:59:35.024212+0000 mon.a (mon.0) 3365 : cluster 0 osdmap e677: 8 total, 8 up, 7 in 2024-04-02T04:59:36.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:36.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T04:59:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:36.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:36.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:36.380 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:36.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:36.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:36.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:36.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.522+0000 7fd33a0f5700 1 -- 172.21.15.69:0/1485553372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd33410ec60 msgr2=0x7fd33410f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:36.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.522+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/1485553372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd33410ec60 0x7fd33410f0e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd32c01c0d0 tx=0x7fd32c01c3e0 comp rx=0 tx=0).stop 2024-04-02T04:59:36.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 -- 172.21.15.69:0/1485553372 shutdown_connections 2024-04-02T04:59:36.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/1485553372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd334109240 0x7fd3341096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:36.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/1485553372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3341109e0 0x7fd33410e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:36.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/1485553372 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd33410ec60 0x7fd33410f0e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:36.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 -- 172.21.15.69:0/1485553372 >> 172.21.15.69:0/1485553372 conn(0x7fd33406d3c0 msgr2=0x7fd33406d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:36.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 -- 172.21.15.69:0/1485553372 shutdown_connections 2024-04-02T04:59:36.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.526+0000 7fd33a0f5700 1 -- 172.21.15.69:0/1485553372 wait complete. 2024-04-02T04:59:36.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 librados: starting msgr at 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 librados: starting objecter 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 Processor -- start 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 -- start start 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 librados: setting wanted keys 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 librados: calling monclient init 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd334109240 0x7fd3341a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 0x7fd3341a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3341109e0 0x7fd33419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd334112270 con 0x7fd33410ec60 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3341120f0 con 0x7fd334109240 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd33a0f5700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd3341123f0 con 0x7fd3341109e0 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 0x7fd3341a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 0x7fd3341a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44364/0 (socket says 172.21.15.69:44364) 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 -- 172.21.15.69:0/3767429614 learned_addr learned my addr 172.21.15.69:0/3767429614 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:36.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd333fff700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3341109e0 0x7fd33419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:36.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 -- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3341109e0 msgr2=0x7fd33419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:36.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3341109e0 0x7fd33419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:36.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 -- 172.21.15.69:0/3767429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd334109240 msgr2=0x7fd3341a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:36.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd3337fe700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd334109240 0x7fd3341a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:36.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd334109240 0x7fd3341a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:36.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 -- 172.21.15.69:0/3767429614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33419f020 con 0x7fd33410ec60 2024-04-02T04:59:36.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd333fff700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3341109e0 0x7fd33419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T04:59:36.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd332ffd700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 0x7fd3341a4b60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd32c003fa0 tx=0x7fd32c01ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd330ff9700 10 client.?.objecter ms_handle_connect 0x7fd33410ec60 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd330ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.530+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c030030 con 0x7fd33410ec60 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33419f2b0 con 0x7fd33410ec60 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd33a0f5700 10 client.18741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3341ab3d0 con 0x7fd33410ec60 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd33a0f5700 1 librados: init done 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd32c0050c0 con 0x7fd33410ec60 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c02c690 con 0x7fd33410ec60 2024-04-02T04:59:36.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.534+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd320005120 con 0x7fd33410ec60 2024-04-02T04:59:36.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd32c033030 con 0x7fd33410ec60 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd31c06f6f0 0x7fd31c071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(678..678 src has 74..678) ==== 11164+0+0 (secure 0 0 0) 0x7fd32c0a0870 con 0x7fd33410ec60 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 10 client.18741.objecter ms_dispatch 0x7fd334001550 osd_map(678..678 src has 74..678) 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 3 client.18741.objecter handle_osd_map got epochs [678,678] > 0 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 3 client.18741.objecter handle_osd_map decoding full epoch 678 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 20 client.18741.objecter dump_active .. 0 homeless 2024-04-02T04:59:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd3337fe700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd31c06f6f0 0x7fd31c071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:36.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd3337fe700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd31c06f6f0 0x7fd31c071bb0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd324005950 tx=0x7fd324005cf0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:36.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.538+0000 7fd330ff9700 10 client.18741.objecter ms_handle_connect 0x7fd31c06f6f0 2024-04-02T04:59:36.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.550+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd32c03a030 con 0x7fd33410ec60 2024-04-02T04:59:36.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:36.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:36.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:36.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:36.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:36.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:36.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.918+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fd320005410 con 0x7fd33410ec60 2024-04-02T04:59:36.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.926+0000 7fd330ff9700 1 -- 172.21.15.69:0/3767429614 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v678) ==== 81+0+65498 (secure 0 0 0) 0x7fd32c06df50 con 0x7fd33410ec60 2024-04-02T04:59:36.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 678, 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "fsid": "f978400e-f0a9-11ee-b647-cb9ed24678a4", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "created": "2024-04-02T04:33:43.348385+0000", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "modified": "2024-04-02T04:59:36.020133+0000", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "last_up_change": "2024-04-02T04:53:53.913436+0000", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "last_in_change": "2024-04-02T04:58:59.098608+0000", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_num": 5806080, 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_set": [ 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "nodeep-scrub", 2024-04-02T04:59:36.935 INFO:teuthology.orchestra.run.smithi069.stdout: "noscrub", 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "pglog_hardlimit", 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "purged_snapdirs", 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_deletes", 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "sortbitwise" 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_version": 110, 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "full_ratio": 0.94999998807907104, 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_snapshot": "", 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_max": 322, 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd": 8, 2024-04-02T04:59:36.936 INFO:teuthology.orchestra.run.smithi069.stdout: "require_min_compat_client": "luminous", 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "min_compat_client": "luminous", 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "require_osd_release": "squid", 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_crimson": false, 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "pools": [ 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 1, 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".mgr", 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:38:57.671714+0000", 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.937 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 11, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 11, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 11, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 11, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 11, 2024-04-02T04:59:36.938 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "481", 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "479", 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.939 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.940 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.941 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_max": 32, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_min": 1 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": {} 2024-04-02T04:59:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.1700000762939453, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.1700000762939453, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9099999666213989, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9099999666213989, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.943 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 2, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": ".rgw.root", 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:52.801763+0000", 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.945 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "79", 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.946 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.947 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 3, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.log", 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:54.772439+0000", 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.950 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.951 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "81", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.952 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.953 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.954 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.9900000095367432, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.9900000095367432, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.75, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.75, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.955 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 4, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.control", 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:56.696111+0000", 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.956 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.957 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "83", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.958 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.959 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.960 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.2400000095367432, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.2400000095367432, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.961 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 5, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "datapool", 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:57.254108+0000", 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 8193, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.962 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 3, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 3, 2024-04-02T04:59:36.963 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 3, 2024-04-02T04:59:36.965 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 3, 2024-04-02T04:59:36.965 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 3, 2024-04-02T04:59:36.965 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "89", 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.966 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 2, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 89, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.967 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.968 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd": {} 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.6500000953674316, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.6500000953674316, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.969 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 6, 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "default.rgw.meta", 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:43:58.846023+0000", 2024-04-02T04:59:36.970 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:36.971 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "85", 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.972 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.973 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.974 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "options": { 2024-04-02T04:59:36.975 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_bias": 4 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": {} 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.4900000095367432, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.4900000095367432, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 2.190000057220459, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 2.190000057220459, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.976 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 10, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:49:11.754451+0000", 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.977 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.978 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "144", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.979 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.980 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.981 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.2400000095367432, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.2400000095367432, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-02T04:59:36.982 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 317, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosTierECPP_smithi069-72349-104", 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:59:19.592593+0000", 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 3, 2024-04-02T04:59:36.983 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 1, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 8, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 8, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 8, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 8, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 8, 2024-04-02T04:59:36.984 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "664", 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.985 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.986 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "testprofile-LibRadosTierECPP_smithi069-72349-104", 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.987 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 8192, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {} 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.988 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 320, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "LibRadosMisc_smithi069-72272-1", 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:59:26.534592+0000", 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool", 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.989 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.990 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "675", 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.991 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.992 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:36.993 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": { 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "app1": { 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "key2": "value2" 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "app2": {}, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "rados": {} 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 2.2400000095367432, 2024-04-02T04:59:36.994 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 2.2400000095367432, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 321, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-109", 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:59:34.680522+0000", 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:59:36.995 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 3, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 2, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 8, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 8, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 8, 2024-04-02T04:59:36.996 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 8, 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 8, 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "677", 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:36.997 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:36.998 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi069-72349-109", 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:36.999 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 8192, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {} 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 322, 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": "test-rados-api-smithi069-72349-109-cache", 2024-04-02T04:59:37.000 INFO:teuthology.orchestra.run.smithi069.stdout: "create_time": "2024-04-02T04:59:35.571958+0000", 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 32769, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "flags_names": "hashpspool,creating", 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "type": 1, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 3, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": 2, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_rule": 0, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_count": 0, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_target": 0, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_barrier": 0, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "object_hash": 2, 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscale_mode": "off", 2024-04-02T04:59:37.001 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": 32, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num": 32, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_placement_num_target": 32, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_target": 32, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num_pending": 32, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "last_pg_merge_meta": { 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "source_pgid": "0.0", 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "ready_epoch": 0, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_started": 0, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "last_epoch_clean": 0, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "source_version": "0'0", 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "target_version": "0'0" 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.002 INFO:teuthology.orchestra.run.smithi069.stdout: "last_change": "678", 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend": "0", 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "last_force_op_resend_preluminous": "0", 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "auid": 0, 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_mode": "selfmanaged", 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_seq": 0, 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_epoch": 0, 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_snaps": [], 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps": "[]", 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_bytes": 0, 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "quota_max_objects": 0, 2024-04-02T04:59:37.003 INFO:teuthology.orchestra.run.smithi069.stdout: "tiers": [], 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_of": -1, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "read_tier": -1, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "write_tier": -1, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_mode": "none", 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_bytes": 0, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "target_max_objects": 0, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_flush_age": 0, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cache_min_evict_age": 0, 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profile": "", 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_params": { 2024-04-02T04:59:37.004 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "none" 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_period": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_count": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "use_gmt_hitset": true, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "min_read_recency_for_promote": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "min_write_recency_for_promote": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_grade_decay_rate": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "hit_set_search_last_n": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "grade_table": [], 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "stripe_width": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "expected_num_objects": 0, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "fast_read": false, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "options": {}, 2024-04-02T04:59:37.005 INFO:teuthology.orchestra.run.smithi069.stdout: "application_metadata": {}, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "read_balance": { 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "score_type": "Fair distribution", 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "score_acting": 1.7400000095367432, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "score_stable": 1.7400000095367432, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "optimal_score": 0.87999999523162842, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity_weighted": 1, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity": 1, 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: "average_primary_affinity_weighted": 1 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.006 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "osds": [ 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "6b13e248-930e-40ca-9bb0-e8c5ce16bd0b", 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 8, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.007 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6802", 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6803", 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6804", 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6805", 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.009 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6808", 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6809", 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.010 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6806", 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6807", 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1087500595 2024-04-02T04:59:37.011 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6803/1087500595", 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6805/1087500595", 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6809/1087500595", 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6807/1087500595", 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:59:37.012 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "afb71182-d4f7-416d-ae44-90f28eab4b7a", 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0.9054107666015625, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 13, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.013 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6810", 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6811", 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.014 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6812", 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6813", 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.015 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6816", 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6817", 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6814", 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6815", 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2651666015 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6811/2651666015", 2024-04-02T04:59:37.017 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6813/2651666015", 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6817/2651666015", 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6815/2651666015", 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "523bcbd7-d389-4b8f-aca8-88b71ce61de7", 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.018 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0.9037933349609375, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 19, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6818", 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.019 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6819", 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6820", 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6821", 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6824", 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.021 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6825", 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6822", 2024-04-02T04:59:37.022 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6823", 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 619998764 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6819/619998764", 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6821/619998764", 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6825/619998764", 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6823/619998764", 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.023 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "20e4526d-2707-44d0-b13a-af36019c42ed", 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 29, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.024 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6826", 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6827", 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.025 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6828", 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6829", 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.026 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6832", 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6833", 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.027 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6830", 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6831", 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 207593208 2024-04-02T04:59:37.028 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.69:6827/207593208", 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.69:6829/207593208", 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.69:6833/207593208", 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.69:6831/207593208", 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:59:37.029 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "aa3bc4f3-4476-4ba1-b033-428e5edf5852", 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0.5665130615234375, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 36, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.030 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6800", 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6801", 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.031 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6802", 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6803", 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.032 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6806", 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6807", 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.033 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6804", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6805", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2065947829 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6801/2065947829", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6803/2065947829", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6807/2065947829", 2024-04-02T04:59:37.034 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6805/2065947829", 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "902326af-4a40-47c8-a3b8-d5d5536f90fe", 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 0, 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0, 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.035 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 47, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 445, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6808", 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.036 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6809", 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6810", 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.037 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6811", 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6814", 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.038 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6815", 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6812", 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6813", 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 219578185 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6809/219578185", 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6811/219578185", 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6815/219578185", 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6813/219578185", 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.040 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "8c73e4b0-cb6b-486a-8b55-a0fe4d97a727", 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 0.8233489990234375, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 0, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 0, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 58, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 0, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.041 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6816", 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6817", 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.042 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6818", 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6819", 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.043 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6822", 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6823", 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.044 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6820", 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6821", 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1226172499 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.045 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6817/1226172499", 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6819/1226172499", 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6823/1226172499", 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6821/1226172499", 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "uuid": "4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3", 2024-04-02T04:59:37.046 INFO:teuthology.orchestra.run.smithi069.stdout: "up": 1, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "in": 1, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "weight": 1, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_affinity": 1, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_begin": 69, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "last_clean_end": 152, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "up_from": 394, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "up_thru": 677, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "down_at": 152, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "lost_at": 0, 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addrs": { 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.047 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6824", 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6825", 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addrs": { 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.048 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6826", 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6827", 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addrs": { 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.049 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6830", 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6831", 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addrs": { 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-04-02T04:59:37.050 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6828", 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.98:6829", 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 575416324 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "public_addr": "172.21.15.98:6825/575416324", 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_addr": "172.21.15.98:6827/575416324", 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_back_addr": "172.21.15.98:6831/575416324", 2024-04-02T04:59:37.051 INFO:teuthology.orchestra.run.smithi069.stdout: "heartbeat_front_addr": "172.21.15.98:6829/575416324", 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "state": [ 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "exists", 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "up" 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_xinfo": [ 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 0, 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.052 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:59:31.974698+0000", 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 1, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:58:39.985672+0000", 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 2, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:59:02.698808+0000", 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 3, 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.054 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:58:49.181159+0000", 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 4, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.055 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:58:55.188810+0000", 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 5, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 65536, 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:59:14.945853+0000", 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 6, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:59:31.296482+0000", 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 0 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": 7, 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "down_stamp": "0.000000", 2024-04-02T04:59:37.057 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_probability": 0, 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "laggy_interval": 0, 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "features": 4540701547738038271, 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "old_weight": 0, 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "last_purged_snaps_scrub": "2024-04-02T04:58:32.657325+0000", 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "dead_epoch": 150 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap": [], 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_items": [ 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "2.9", 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:59:37.058 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 7, 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.f", 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "pgid": "4.17", 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "mappings": [ 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "from": 1, 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "to": 2 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_upmap_primaries": [], 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_temp": [], 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "primary_temp": [], 2024-04-02T04:59:37.060 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklist": { 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2738867930": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1798147282": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/52352690": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2928651991": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/2442918057": "2024-04-03T04:34:13.853990+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/2763215770": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/3657409442": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/879855771": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1025808168": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1168952113": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:59:37.061 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6800/578732137": "2024-04-03T04:34:47.037764+0000", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/1892956550": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:6801/1560062353": "2024-04-03T04:44:51.699278+0000", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "172.21.15.69:0/54578327": "2024-04-03T04:44:51.699278+0000" 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "range_blocklist": {}, 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "erasure_code_profiles": { 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "default": { 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van" 2024-04-02T04:59:37.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "testprofile-LibRadosTierECPP_smithi069-72349-104": { 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-device-class": "", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-num-failure-domains": "0", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-osds-per-failure-domain": "0", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-root": "default", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van", 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "w": "8" 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.063 INFO:teuthology.orchestra.run.smithi069.stdout: "testprofile-test-rados-api-smithi069-72349-109": { 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-device-class": "", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-failure-domain": "osd", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-num-failure-domains": "0", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-osds-per-failure-domain": "0", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-root": "default", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "k": "2", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "m": "1", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "plugin": "jerasure", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "technique": "reed_sol_van", 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "w": "8" 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.064 INFO:teuthology.orchestra.run.smithi069.stdout: "removed_snaps_queue": [ 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 17, 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 36, 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 38, 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.066 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 66, 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 101, 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.067 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.068 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 103, 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.069 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 112, 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 9 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.070 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 120, 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 137, 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.071 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 145, 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 146, 2024-04-02T04:59:37.072 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 154, 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.073 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 164, 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 3 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 165, 2024-04-02T04:59:37.074 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 173, 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.075 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 176, 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 188, 2024-04-02T04:59:37.076 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 3, 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 189, 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.077 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 196, 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 1, 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 4 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 209, 2024-04-02T04:59:37.078 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 2, 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 6 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 214, 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 7, 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.079 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 234, 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 8, 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 245, 2024-04-02T04:59:37.080 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 10, 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 251, 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 11, 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 2 2024-04-02T04:59:37.081 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 262, 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: "snaps": [ 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: "begin": 14, 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: "length": 1 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-02T04:59:37.082 INFO:teuthology.orchestra.run.smithi069.stdout: "new_removed_snaps": [], 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "new_purged_snaps": [], 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_node_flags": {}, 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "device_class_flags": {}, 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode": { 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_enabled": false, 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_bucket_count": 0, 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "degraded_stretch_mode": 0, 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "recovering_stretch_mode": 0, 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: "stretch_mode_bucket": 0 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T04:59:37.083 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 10 librados: watch_flush enter 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 10 librados: watch_flush exit 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 20 client.18741.objecter shutdown clearing up homeless session... 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 10 client.18741.objecter successfully canceled tick 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd31c06f6f0 msgr2=0x7fd31c071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd31c06f6f0 0x7fd31c071bb0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd324005950 tx=0x7fd324005cf0 comp rx=0 tx=0).stop 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 msgr2=0x7fd3341a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.934+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 0x7fd3341a4b60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd32c003fa0 tx=0x7fd32c01ce90 comp rx=0 tx=0).stop 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.938+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 shutdown_connections 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.938+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd31c06f6f0 0x7fd31c071bb0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.938+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3341109e0 0x7fd33419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.938+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd334109240 0x7fd3341a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.938+0000 7fd33a0f5700 1 --2- 172.21.15.69:0/3767429614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd33410ec60 0x7fd3341a4b60 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.938+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 >> 172.21.15.69:0/3767429614 conn(0x7fd33406d3c0 msgr2=0x7fd334072db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.946+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 shutdown_connections 2024-04-02T04:59:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.946+0000 7fd33a0f5700 1 -- 172.21.15.69:0/3767429614 wait complete. 2024-04-02T04:59:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:36.946+0000 7fd33a0f5700 1 librados: shutdown 2024-04-02T04:59:37.130 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:37.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:37.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:37.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[21528]: cluster 2024-04-02T04:59:36.027125+0000 mon.a (mon.0) 3366 : cluster 0 osdmap e678: 8 total, 8 up, 7 in 2024-04-02T04:59:37.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[21528]: audit 2024-04-02T04:59:36.052172+0000 mon.a (mon.0) 3367 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:37.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[21528]: cluster 2024-04-02T04:59:36.263741+0000 mgr.y (mgr.14538) 620 : cluster 0 pgmap v1023: 254 pgs: 40 unknown, 214 active+clean; 635 KiB data, 628 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:59:37.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[21528]: audit 2024-04-02T04:59:36.925804+0000 mon.a (mon.0) 3368 : audit 0 from='client.? 172.21.15.69:0/3767429614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:59:37.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[28904]: cluster 2024-04-02T04:59:36.027125+0000 mon.a (mon.0) 3366 : cluster 0 osdmap e678: 8 total, 8 up, 7 in 2024-04-02T04:59:37.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[28904]: audit 2024-04-02T04:59:36.052172+0000 mon.a (mon.0) 3367 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:37.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[28904]: cluster 2024-04-02T04:59:36.263741+0000 mgr.y (mgr.14538) 620 : cluster 0 pgmap v1023: 254 pgs: 40 unknown, 214 active+clean; 635 KiB data, 628 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:59:37.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:37 smithi069 bash[28904]: audit 2024-04-02T04:59:36.925804+0000 mon.a (mon.0) 3368 : audit 0 from='client.? 172.21.15.69:0/3767429614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:59:37.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:37.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:37.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:37.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:37 smithi098 bash[23790]: cluster 2024-04-02T04:59:36.027125+0000 mon.a (mon.0) 3366 : cluster 0 osdmap e678: 8 total, 8 up, 7 in 2024-04-02T04:59:37.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:37 smithi098 bash[23790]: audit 2024-04-02T04:59:36.052172+0000 mon.a (mon.0) 3367 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:37.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:37 smithi098 bash[23790]: cluster 2024-04-02T04:59:36.263741+0000 mgr.y (mgr.14538) 620 : cluster 0 pgmap v1023: 254 pgs: 40 unknown, 214 active+clean; 635 KiB data, 628 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-02T04:59:37.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:37 smithi098 bash[23790]: audit 2024-04-02T04:59:36.925804+0000 mon.a (mon.0) 3368 : audit 0 from='client.? 172.21.15.69:0/3767429614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-02T04:59:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:37.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:37.574 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:37.608 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:37.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:37.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:37.736 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:37.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:37.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:37.886 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:37.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:37.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:38.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:38.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:38.257 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:38 smithi098 bash[23790]: cluster 2024-04-02T04:59:37.060112+0000 mon.a (mon.0) 3369 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:38.258 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:38 smithi098 bash[23790]: audit 2024-04-02T04:59:37.062794+0000 mon.a (mon.0) 3370 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:38.258 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:38 smithi098 bash[23790]: cluster 2024-04-02T04:59:37.067669+0000 mon.a (mon.0) 3371 : cluster 0 osdmap e679: 8 total, 8 up, 7 in 2024-04-02T04:59:38.258 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:38 smithi098 bash[23790]: audit 2024-04-02T04:59:37.116528+0000 mon.a (mon.0) 3372 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:38.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[21528]: cluster 2024-04-02T04:59:37.060112+0000 mon.a (mon.0) 3369 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:38.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[21528]: audit 2024-04-02T04:59:37.062794+0000 mon.a (mon.0) 3370 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:38.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[21528]: cluster 2024-04-02T04:59:37.067669+0000 mon.a (mon.0) 3371 : cluster 0 osdmap e679: 8 total, 8 up, 7 in 2024-04-02T04:59:38.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[21528]: audit 2024-04-02T04:59:37.116528+0000 mon.a (mon.0) 3372 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[28904]: cluster 2024-04-02T04:59:37.060112+0000 mon.a (mon.0) 3369 : cluster 3 Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[28904]: audit 2024-04-02T04:59:37.062794+0000 mon.a (mon.0) 3370 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[28904]: cluster 2024-04-02T04:59:37.067669+0000 mon.a (mon.0) 3371 : cluster 0 osdmap e679: 8 total, 8 up, 7 in 2024-04-02T04:59:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[28904]: audit 2024-04-02T04:59:37.116528+0000 mon.a (mon.0) 3372 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:38.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:38.398 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:38.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:38.684 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:38.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:38.835 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:59:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:59:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:59:38.852 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:38.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:38.996 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:38.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:39.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:39.133 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:39.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:39.377 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:39.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[21528]: audit 2024-04-02T04:59:38.105617+0000 mon.a (mon.0) 3373 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:39.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[21528]: cluster 2024-04-02T04:59:38.109584+0000 mon.a (mon.0) 3374 : cluster 0 osdmap e680: 8 total, 8 up, 7 in 2024-04-02T04:59:39.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[21528]: audit 2024-04-02T04:59:38.111389+0000 mon.a (mon.0) 3375 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-109", "overlaypool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:39.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[21528]: cluster 2024-04-02T04:59:38.264486+0000 mgr.y (mgr.14538) 621 : cluster 0 pgmap v1026: 254 pgs: 5 unknown, 249 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:39.420 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[28904]: audit 2024-04-02T04:59:38.105617+0000 mon.a (mon.0) 3373 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:39.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[28904]: cluster 2024-04-02T04:59:38.109584+0000 mon.a (mon.0) 3374 : cluster 0 osdmap e680: 8 total, 8 up, 7 in 2024-04-02T04:59:39.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[28904]: audit 2024-04-02T04:59:38.111389+0000 mon.a (mon.0) 3375 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-109", "overlaypool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:39.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:39 smithi069 bash[28904]: cluster 2024-04-02T04:59:38.264486+0000 mgr.y (mgr.14538) 621 : cluster 0 pgmap v1026: 254 pgs: 5 unknown, 249 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:39.460 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:39 smithi098 bash[23790]: audit 2024-04-02T04:59:38.105617+0000 mon.a (mon.0) 3373 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:39 smithi098 bash[23790]: cluster 2024-04-02T04:59:38.109584+0000 mon.a (mon.0) 3374 : cluster 0 osdmap e680: 8 total, 8 up, 7 in 2024-04-02T04:59:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:39 smithi098 bash[23790]: audit 2024-04-02T04:59:38.111389+0000 mon.a (mon.0) 3375 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-109", "overlaypool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:39.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:39 smithi098 bash[23790]: cluster 2024-04-02T04:59:38.264486+0000 mgr.y (mgr.14538) 621 : cluster 0 pgmap v1026: 254 pgs: 5 unknown, 249 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:39.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:39.591 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:39.671 INFO:tasks.thrashosds.thrasher:j is {'epoch': 678, 'fsid': 'f978400e-f0a9-11ee-b647-cb9ed24678a4', 'created': '2024-04-02T04:33:43.348385+0000', 'modified': '2024-04-02T04:59:36.020133+0000', 'last_up_change': '2024-04-02T04:53:53.913436+0000', 'last_in_change': '2024-04-02T04:58:59.098608+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 110, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 322, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.671714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '481', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '479', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-02T04:43:52.801763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-02T04:43:54.772439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-02T04:43:56.696111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-04-02T04:43:57.254108+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-02T04:43:58.846023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-02T04:49:11.754451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 317, 'pool_name': 'LibRadosTierECPP_smithi069-72349-104', 'create_time': '2024-04-02T04:59:19.592593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '664', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosTierECPP_smithi069-72349-104', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 320, 'pool_name': 'LibRadosMisc_smithi069-72272-1', 'create_time': '2024-04-02T04:59:26.534592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '675', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'app1': {'key2': 'value2'}, 'app2': {}, 'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 321, 'pool_name': 'test-rados-api-smithi069-72349-109', 'create_time': '2024-04-02T04:59:34.680522+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 2, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '677', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi069-72349-109', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 322, 'pool_name': 'test-rados-api-smithi069-72349-109-cache', 'create_time': '2024-04-02T04:59:35.571958+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '678', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6b13e248-930e-40ca-9bb0-e8c5ce16bd0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1087500595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1087500595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 1087500595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1087500595}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1087500595}]}, 'public_addr': '172.21.15.69:6803/1087500595', 'cluster_addr': '172.21.15.69:6805/1087500595', 'heartbeat_back_addr': '172.21.15.69:6809/1087500595', 'heartbeat_front_addr': '172.21.15.69:6807/1087500595', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'afb71182-d4f7-416d-ae44-90f28eab4b7a', 'up': 1, 'in': 1, 'weight': 0.9054107666015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6811', 'nonce': 2651666015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6812', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2651666015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6816', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6817', 'nonce': 2651666015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 2651666015}, {'type': 'v1', 'addr': '172.21.15.69:6815', 'nonce': 2651666015}]}, 'public_addr': '172.21.15.69:6811/2651666015', 'cluster_addr': '172.21.15.69:6813/2651666015', 'heartbeat_back_addr': '172.21.15.69:6817/2651666015', 'heartbeat_front_addr': '172.21.15.69:6815/2651666015', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '523bcbd7-d389-4b8f-aca8-88b71ce61de7', 'up': 1, 'in': 1, 'weight': 0.9037933349609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6818', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6819', 'nonce': 619998764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6820', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6821', 'nonce': 619998764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6825', 'nonce': 619998764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6822', 'nonce': 619998764}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 619998764}]}, 'public_addr': '172.21.15.69:6819/619998764', 'cluster_addr': '172.21.15.69:6821/619998764', 'heartbeat_back_addr': '172.21.15.69:6825/619998764', 'heartbeat_front_addr': '172.21.15.69:6823/619998764', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '20e4526d-2707-44d0-b13a-af36019c42ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6826', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 207593208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 207593208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 207593208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 207593208}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 207593208}]}, 'public_addr': '172.21.15.69:6827/207593208', 'cluster_addr': '172.21.15.69:6829/207593208', 'heartbeat_back_addr': '172.21.15.69:6833/207593208', 'heartbeat_front_addr': '172.21.15.69:6831/207593208', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa3bc4f3-4476-4ba1-b033-428e5edf5852', 'up': 1, 'in': 1, 'weight': 0.5665130615234375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 2065947829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 2065947829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 2065947829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 2065947829}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 2065947829}]}, 'public_addr': '172.21.15.98:6801/2065947829', 'cluster_addr': '172.21.15.98:6803/2065947829', 'heartbeat_back_addr': '172.21.15.98:6807/2065947829', 'heartbeat_front_addr': '172.21.15.98:6805/2065947829', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '902326af-4a40-47c8-a3b8-d5d5536f90fe', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 445, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 219578185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 219578185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6814', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 219578185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 219578185}, {'type': 'v1', 'addr': '172.21.15.98:6813', 'nonce': 219578185}]}, 'public_addr': '172.21.15.98:6809/219578185', 'cluster_addr': '172.21.15.98:6811/219578185', 'heartbeat_back_addr': '172.21.15.98:6815/219578185', 'heartbeat_front_addr': '172.21.15.98:6813/219578185', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '8c73e4b0-cb6b-486a-8b55-a0fe4d97a727', 'up': 1, 'in': 1, 'weight': 0.8233489990234375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 1226172499}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 1226172499}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 1226172499}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 1226172499}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 1226172499}]}, 'public_addr': '172.21.15.98:6817/1226172499', 'cluster_addr': '172.21.15.98:6819/1226172499', 'heartbeat_back_addr': '172.21.15.98:6823/1226172499', 'heartbeat_front_addr': '172.21.15.98:6821/1226172499', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 69, 'last_clean_end': 152, 'up_from': 394, 'up_thru': 677, 'down_at': 152, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 575416324}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 575416324}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 575416324}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 575416324}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 575416324}]}, 'public_addr': '172.21.15.98:6825/575416324', 'cluster_addr': '172.21.15.98:6827/575416324', 'heartbeat_back_addr': '172.21.15.98:6831/575416324', 'heartbeat_front_addr': '172.21.15.98:6829/575416324', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:59:31.974698+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:58:39.985672+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:59:02.698808+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:58:49.181159+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:58:55.188810+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-02T04:59:14.945853+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:59:31.296482+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-02T04:58:32.657325+0000', 'dead_epoch': 150}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.69:0/2738867930': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/1798147282': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/52352690': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:0/2928651991': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:6800/2442918057': '2024-04-03T04:34:13.853990+0000', '172.21.15.69:6801/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/2763215770': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/3657409442': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/879855771': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1025808168': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/1168952113': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6800/578732137': '2024-04-03T04:34:47.037764+0000', '172.21.15.69:0/1892956550': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:6801/1560062353': '2024-04-03T04:44:51.699278+0000', '172.21.15.69:0/54578327': '2024-04-03T04:44:51.699278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosTierECPP_smithi069-72349-104': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-test-rados-api-smithi069-72349-109': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 17, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 36, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 38, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 66, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 101, 'snaps': [{'begin': 2, 'length': 4}, {'begin': 7, 'length': 2}]}, {'pool': 103, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 2, 'length': 9}, {'begin': 14, 'length': 1}]}, {'pool': 120, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 137, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 145, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 146, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 164, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 173, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 196, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 209, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 251, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 262, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-02T04:59:39.671 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:59:39.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:39.717 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:39.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:39.850 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:39.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:39.982 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:39.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:40.085 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:40.168 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:40.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:40 smithi098 bash[23790]: cluster 2024-04-02T04:59:38.907336+0000 osd.7 (osd.7) 151 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:40 smithi098 bash[23790]: cluster 2024-04-02T04:59:38.907774+0000 osd.7 (osd.7) 152 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:40 smithi098 bash[23790]: audit 2024-04-02T04:59:39.107924+0000 mon.a (mon.0) 3376 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-109", "overlaypool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:40 smithi098 bash[23790]: cluster 2024-04-02T04:59:39.111520+0000 mon.a (mon.0) 3377 : cluster 0 osdmap e681: 8 total, 8 up, 7 in 2024-04-02T04:59:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:40 smithi098 bash[23790]: audit 2024-04-02T04:59:39.112891+0000 mon.a (mon.0) 3378 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-109-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:40 smithi098 bash[23790]: cluster 2024-04-02T04:59:40.110093+0000 mon.a (mon.0) 3379 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:40.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:40.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:40.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[21528]: cluster 2024-04-02T04:59:38.907336+0000 osd.7 (osd.7) 151 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:40.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[21528]: cluster 2024-04-02T04:59:38.907774+0000 osd.7 (osd.7) 152 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:40.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[21528]: audit 2024-04-02T04:59:39.107924+0000 mon.a (mon.0) 3376 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-109", "overlaypool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:40.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[21528]: cluster 2024-04-02T04:59:39.111520+0000 mon.a (mon.0) 3377 : cluster 0 osdmap e681: 8 total, 8 up, 7 in 2024-04-02T04:59:40.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[21528]: audit 2024-04-02T04:59:39.112891+0000 mon.a (mon.0) 3378 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-109-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:40.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[21528]: cluster 2024-04-02T04:59:40.110093+0000 mon.a (mon.0) 3379 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:40.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[28904]: cluster 2024-04-02T04:59:38.907336+0000 osd.7 (osd.7) 151 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:40.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[28904]: cluster 2024-04-02T04:59:38.907774+0000 osd.7 (osd.7) 152 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:40.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[28904]: audit 2024-04-02T04:59:39.107924+0000 mon.a (mon.0) 3376 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-109", "overlaypool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:40.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[28904]: cluster 2024-04-02T04:59:39.111520+0000 mon.a (mon.0) 3377 : cluster 0 osdmap e681: 8 total, 8 up, 7 in 2024-04-02T04:59:40.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[28904]: audit 2024-04-02T04:59:39.112891+0000 mon.a (mon.0) 3378 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-109-cache", "mode": "writeback"}]: dispatch 2024-04-02T04:59:40.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:40 smithi069 bash[28904]: cluster 2024-04-02T04:59:40.110093+0000 mon.a (mon.0) 3379 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T04:59:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:40.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:40.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:40.601 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:40.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:40.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:40.802 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:40.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:40.915 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:40.984 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:40.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:41.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:41.154 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:41.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:41.284 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: audit 2024-04-02T04:59:40.120187+0000 mon.a (mon.0) 3380 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-109-cache", "mode": "writeback"}]': finished 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: cluster 2024-04-02T04:59:40.124346+0000 mon.a (mon.0) 3381 : cluster 0 osdmap e682: 8 total, 8 up, 7 in 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: audit 2024-04-02T04:59:40.127922+0000 mon.a (mon.0) 3382 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: cluster 2024-04-02T04:59:40.265034+0000 mgr.y (mgr.14538) 622 : cluster 0 pgmap v1029: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: audit 2024-04-02T04:59:41.122395+0000 mon.a (mon.0) 3383 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: cluster 2024-04-02T04:59:41.130324+0000 mon.a (mon.0) 3384 : cluster 0 osdmap e683: 8 total, 8 up, 7 in 2024-04-02T04:59:41.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[21528]: audit 2024-04-02T04:59:41.130584+0000 mon.a (mon.0) 3385 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: audit 2024-04-02T04:59:40.120187+0000 mon.a (mon.0) 3380 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-109-cache", "mode": "writeback"}]': finished 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: cluster 2024-04-02T04:59:40.124346+0000 mon.a (mon.0) 3381 : cluster 0 osdmap e682: 8 total, 8 up, 7 in 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: audit 2024-04-02T04:59:40.127922+0000 mon.a (mon.0) 3382 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: cluster 2024-04-02T04:59:40.265034+0000 mgr.y (mgr.14538) 622 : cluster 0 pgmap v1029: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: audit 2024-04-02T04:59:41.122395+0000 mon.a (mon.0) 3383 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: cluster 2024-04-02T04:59:41.130324+0000 mon.a (mon.0) 3384 : cluster 0 osdmap e683: 8 total, 8 up, 7 in 2024-04-02T04:59:41.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:41 smithi069 bash[28904]: audit 2024-04-02T04:59:41.130584+0000 mon.a (mon.0) 3385 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:41.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: audit 2024-04-02T04:59:40.120187+0000 mon.a (mon.0) 3380 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-109-cache", "mode": "writeback"}]': finished 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: cluster 2024-04-02T04:59:40.124346+0000 mon.a (mon.0) 3381 : cluster 0 osdmap e682: 8 total, 8 up, 7 in 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: audit 2024-04-02T04:59:40.127922+0000 mon.a (mon.0) 3382 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: cluster 2024-04-02T04:59:40.265034+0000 mgr.y (mgr.14538) 622 : cluster 0 pgmap v1029: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: audit 2024-04-02T04:59:41.122395+0000 mon.a (mon.0) 3383 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: cluster 2024-04-02T04:59:41.130324+0000 mon.a (mon.0) 3384 : cluster 0 osdmap e683: 8 total, 8 up, 7 in 2024-04-02T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:41 smithi098 bash[23790]: audit 2024-04-02T04:59:41.130584+0000 mon.a (mon.0) 3385 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T04:59:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:41.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:41.539 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:41.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:41.701 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:41.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:41.929 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T04:59:41.929 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T04:59:41.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- 172.21.15.69:0/3899059167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225c109dd0 msgr2=0x7f225c11da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- 172.21.15.69:0/3899059167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225c109dd0 0x7f225c11da30 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f2250009990 tx=0x7f2250009ca0 comp rx=0 tx=0).stop 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- 172.21.15.69:0/3899059167 shutdown_connections 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- 172.21.15.69:0/3899059167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225c109dd0 0x7f225c11da30 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- 172.21.15.69:0/3899059167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f225c110830 0x7f225c110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- 172.21.15.69:0/3899059167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:41.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- 172.21.15.69:0/3899059167 >> 172.21.15.69:0/3899059167 conn(0x7f225c06d190 msgr2=0x7f225c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- 172.21.15.69:0/3899059167 shutdown_connections 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- 172.21.15.69:0/3899059167 wait complete. 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 librados: starting msgr at 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 librados: starting objecter 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 Processor -- start 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- start start 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 librados: setting wanted keys 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 librados: calling monclient init 2024-04-02T04:59:41.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f225c110830 0x7f225c1acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225c1a6820 0x7f225c1a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f225c120760 con 0x7f225c109240 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f225c1205e0 con 0x7f225c110830 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f226105e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f225c1208e0 con 0x7f225c1a6820 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f225bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f225bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1ac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50706/0 (socket says 172.21.15.69:50706) 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.942+0000 7f225bfff700 1 -- 172.21.15.69:0/2974067208 learned_addr learned my addr 172.21.15.69:0/2974067208 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f225c110830 0x7f225c1acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225bfff700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225c1a6820 msgr2=0x7f225c1a6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f225c1a6820 0x7f225c1a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225bfff700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f225c110830 msgr2=0x7f225c1acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:41.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f225c110830 0x7f225c1acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:41.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225bfff700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225c1a73e0 con 0x7f225c109240 2024-04-02T04:59:41.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1ac6a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f225400cd70 tx=0x7f2254007560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f22597fa700 10 client.?.objecter ms_handle_connect 0x7f225c109240 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f22597fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254007df0 con 0x7f225c109240 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f226105e700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225c1b3500 con 0x7f225c109240 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f226105e700 10 client.23052.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f226105e700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f225c1b3920 con 0x7f225c109240 2024-04-02T04:59:41.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f226105e700 1 librados: init done 2024-04-02T04:59:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f22540158b0 con 0x7f225c109240 2024-04-02T04:59:41.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225400d820 con 0x7f225c109240 2024-04-02T04:59:41.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.946+0000 7f226105e700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f225c06e500 con 0x7f225c109240 2024-04-02T04:59:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f225400daa0 con 0x7f225c109240 2024-04-02T04:59:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f224406f6e0 0x7f2244071ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(683..683 src has 74..683) ==== 11172+0+0 (secure 0 0 0) 0x7f225408ef30 con 0x7f225c109240 2024-04-02T04:59:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 osd_map(683..683 src has 74..683) 2024-04-02T04:59:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 3 client.23052.objecter handle_osd_map got epochs [683,683] > 0 2024-04-02T04:59:41.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 3 client.23052.objecter handle_osd_map decoding full epoch 683 2024-04-02T04:59:41.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 20 client.23052.objecter dump_active .. 0 homeless 2024-04-02T04:59:41.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f224406f6e0 0x7f2244071ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:41.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.950+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v683) ==== 54+0+16 (secure 0 0 0) 0x7f225405c580 con 0x7f225c109240 2024-04-02T04:59:41.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.954+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f224406f6e0 0x7f2244071ba0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f225c06d9b0 tx=0x7f224c00a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:41.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.954+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f224406f6e0 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 10 client.23052.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 1 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f225c02d180 0x7f225c02d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f226085d700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f225c02d180 0x7f225c02d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f226085d700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f225c02d180 0x7f225c02d560 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c02d180 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 1 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 0 1 2024-04-02T04:59:41.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 4 2024-04-02T04:59:41.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 10 client.23052.objecter _send_command 1 2024-04-02T04:59:41.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.958+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f225c1a7d50 con 0x7f225c02d180 2024-04-02T04:59:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.962+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1a7d50 con 0x7f225c02d180 2024-04-02T04:59:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.962+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 1: 0 ) 2024-04-02T04:59:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.962+0000 7f22597fa700 10 client.23052.objecter _finish_command 1 = system:0 2024-04-02T04:59:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.962+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 4 2024-04-02T04:59:41.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.962+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 0 1 2024-04-02T04:59:41.988 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:41.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 10 client.23052.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 2 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 2 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c10d1b0 osd=0 3 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 0 2 2024-04-02T04:59:41.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 4 2024-04-02T04:59:42.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 10 client.23052.objecter _send_command 2 2024-04-02T04:59:42.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c050750 con 0x7f225c02d180 2024-04-02T04:59:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c050750 con 0x7f225c02d180 2024-04-02T04:59:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f22597fa700 10 client.23052.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 4 2024-04-02T04:59:42.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:41.998+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 0 2 2024-04-02T04:59:42.002 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 10 client.23052.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 3 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f225c1b72e0 0x7f225c1b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 3 2024-04-02T04:59:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 1 3 2024-04-02T04:59:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 4 2024-04-02T04:59:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 10 client.23052.objecter _send_command 3 2024-04-02T04:59:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f225c02deb0 con 0x7f225c1b72e0 2024-04-02T04:59:42.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f225c1b72e0 0x7f225c1b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f225c1b72e0 0x7f225c1b96d0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.002+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1b72e0 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.010+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c02deb0 con 0x7f225c1b72e0 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.010+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 3: 0 ) 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.010+0000 7f22597fa700 10 client.23052.objecter _finish_command 3 = system:0 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.010+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 4 2024-04-02T04:59:42.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.010+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 1 3 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 10 client.23052.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 4 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 4 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c02dcf0 osd=1 3 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 1 4 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 4 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 10 client.23052.objecter _send_command 4 2024-04-02T04:59:42.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.042+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c02cc00 con 0x7f225c1b72e0 2024-04-02T04:59:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c02cc00 con 0x7f225c1b72e0 2024-04-02T04:59:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f22597fa700 10 client.23052.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 4 2024-04-02T04:59:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 1 4 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 10 client.23052.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 5 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f225c1bad90 0x7f225c1bd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 5 2024-04-02T04:59:42.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 2 5 2024-04-02T04:59:42.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 4 2024-04-02T04:59:42.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 10 client.23052.objecter _send_command 5 2024-04-02T04:59:42.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f225c1bffe0 con 0x7f225c1bad90 2024-04-02T04:59:42.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.046+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f225c1bad90 0x7f225c1bd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f225c1bad90 0x7f225c1bd180 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1bad90 2024-04-02T04:59:42.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1bffe0 con 0x7f225c1bad90 2024-04-02T04:59:42.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 5: 0 ) 2024-04-02T04:59:42.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f22597fa700 10 client.23052.objecter _finish_command 5 = system:0 2024-04-02T04:59:42.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 4 2024-04-02T04:59:42.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.050+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 2 5 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 10 client.23052.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 6 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 6 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1b4410 osd=2 3 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 2 6 2024-04-02T04:59:42.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 4 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 10 client.23052.objecter _send_command 6 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c1b3f50 con 0x7f225c1bad90 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c1b3f50 con 0x7f225c1bad90 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f22597fa700 10 client.23052.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 4 2024-04-02T04:59:42.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.090+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 2 6 2024-04-02T04:59:42.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:42.098 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T04:59:42.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.094+0000 7f2242ffd700 10 client.23052.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.094+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 7 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f225c1be840 0x7f225c1beca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f226085d700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f225c1be840 0x7f225c1beca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 7 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 3 7 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1be840 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f226085d700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f225c1be840 0x7f225c1beca0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 4 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 10 client.23052.objecter _send_command 7 2024-04-02T04:59:42.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f225c1bf1e0 con 0x7f225c1be840 2024-04-02T04:59:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1bf1e0 con 0x7f225c1be840 2024-04-02T04:59:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 7: 0 ) 2024-04-02T04:59:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f22597fa700 10 client.23052.objecter _finish_command 7 = system:0 2024-04-02T04:59:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 4 2024-04-02T04:59:42.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.098+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 3 7 2024-04-02T04:59:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:42.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 10 client.23052.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 8 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 8 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1b41e0 osd=3 3 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 3 8 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 4 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 10 client.23052.objecter _send_command 8 2024-04-02T04:59:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.130+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c1bf1e0 con 0x7f225c1be840 2024-04-02T04:59:42.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.134+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c1bf1e0 con 0x7f225c1be840 2024-04-02T04:59:42.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.134+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.134+0000 7f22597fa700 10 client.23052.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.134+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 4 2024-04-02T04:59:42.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.134+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 3 8 2024-04-02T04:59:42.137 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 10 client.23052.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 9 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f225c1bfbf0 0x7f225c1c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 9 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 4 9 2024-04-02T04:59:42.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 4 2024-04-02T04:59:42.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 10 client.23052.objecter _send_command 9 2024-04-02T04:59:42.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f225c1c71e0 con 0x7f225c1bfbf0 2024-04-02T04:59:42.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f225c1bfbf0 0x7f225c1c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f225c1bfbf0 0x7f225c1c41b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.138+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1bfbf0 2024-04-02T04:59:42.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.142+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1c71e0 con 0x7f225c1bfbf0 2024-04-02T04:59:42.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.142+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 9: 0 ) 2024-04-02T04:59:42.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.142+0000 7f22597fa700 10 client.23052.objecter _finish_command 9 = system:0 2024-04-02T04:59:42.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.142+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 4 2024-04-02T04:59:42.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.142+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 4 9 2024-04-02T04:59:42.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 10 client.23052.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 10 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 10 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1bf1e0 osd=4 3 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 4 10 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 4 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 10 client.23052.objecter _send_command 10 2024-04-02T04:59:42.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.170+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c1c71e0 con 0x7f225c1bfbf0 2024-04-02T04:59:42.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c1c71e0 con 0x7f225c1bfbf0 2024-04-02T04:59:42.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f22597fa700 10 client.23052.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 4 2024-04-02T04:59:42.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 4 10 2024-04-02T04:59:42.177 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 10 client.23052.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 11 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f225c1c6550 0x7f225c1c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 11 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 5 11 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 4 2024-04-02T04:59:42.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 10 client.23052.objecter _send_command 11 2024-04-02T04:59:42.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.174+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f225c1ca580 con 0x7f225c1c6550 2024-04-02T04:59:42.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f225c1c6550 0x7f225c1c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f225b7fe700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f225c1c6550 0x7f225c1c69d0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1c6550 2024-04-02T04:59:42.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1ca580 con 0x7f225c1c6550 2024-04-02T04:59:42.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 11: 0 ) 2024-04-02T04:59:42.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f22597fa700 10 client.23052.objecter _finish_command 11 = system:0 2024-04-02T04:59:42.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 4 2024-04-02T04:59:42.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.178+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 5 11 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 10 client.23052.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 12 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 12 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1c7220 osd=5 3 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 5 12 2024-04-02T04:59:42.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 4 2024-04-02T04:59:42.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 10 client.23052.objecter _send_command 12 2024-04-02T04:59:42.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.206+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c1ca580 con 0x7f225c1c6550 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c1ca580 con 0x7f225c1c6550 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f22597fa700 10 client.23052.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 4 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 5 12 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 10 client.23052.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 13 2024-04-02T04:59:42.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f225c1bde90 0x7f225c1be2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 13 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 6 13 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 4 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 10 client.23052.objecter _send_command 13 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f225c1cd8f0 con 0x7f225c1bde90 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f226085d700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f225c1bde90 0x7f225c1be2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f226085d700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f225c1bde90 0x7f225c1be2b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.210+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1bde90 2024-04-02T04:59:42.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.214+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1cd8f0 con 0x7f225c1bde90 2024-04-02T04:59:42.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.214+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 13: 0 ) 2024-04-02T04:59:42.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.214+0000 7f22597fa700 10 client.23052.objecter _finish_command 13 = system:0 2024-04-02T04:59:42.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.214+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 4 2024-04-02T04:59:42.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.214+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 6 13 2024-04-02T04:59:42.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:42.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 10 client.23052.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 14 2024-04-02T04:59:42.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 14 2024-04-02T04:59:42.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1ca580 osd=6 3 2024-04-02T04:59:42.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 6 14 2024-04-02T04:59:42.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 4 2024-04-02T04:59:42.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 10 client.23052.objecter _send_command 14 2024-04-02T04:59:42.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.238+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c1cc770 con 0x7f225c1bde90 2024-04-02T04:59:42.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c1cc770 con 0x7f225c1bde90 2024-04-02T04:59:42.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f22597fa700 10 client.23052.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 4 2024-04-02T04:59:42.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 6 14 2024-04-02T04:59:42.250 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 10 client.23052.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 15 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f225c1ccee0 0x7f225c1cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 15 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 7 15 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 4 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 10 client.23052.objecter _send_command 15 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f225c1d0c60 con 0x7f225c1ccee0 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f225c1ccee0 0x7f225c1cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f225bfff700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f225c1ccee0 0x7f225c1cd320 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.250+0000 7f22597fa700 10 client.23052.objecter ms_handle_connect 0x7f225c1ccee0 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.254+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f225c1d0c60 con 0x7f225c1ccee0 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.254+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 15: 0 ) 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.254+0000 7f22597fa700 10 client.23052.objecter _finish_command 15 = system:0 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.254+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 4 2024-04-02T04:59:42.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.254+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 7 15 2024-04-02T04:59:42.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[28904]: audit 2024-04-02T04:59:41.954391+0000 mon.a (mon.0) 3386 : audit 0 from='client.? 172.21.15.69:0/2974067208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:42.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[28904]: audit 2024-04-02T04:59:42.124561+0000 mon.a (mon.0) 3387 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:59:42.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[28904]: cluster 2024-04-02T04:59:42.130646+0000 mon.a (mon.0) 3388 : cluster 0 osdmap e684: 8 total, 8 up, 7 in 2024-04-02T04:59:42.278 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[28904]: audit 2024-04-02T04:59:42.134277+0000 mon.a (mon.0) 3389 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:42.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[21528]: audit 2024-04-02T04:59:41.954391+0000 mon.a (mon.0) 3386 : audit 0 from='client.? 172.21.15.69:0/2974067208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:42.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[21528]: audit 2024-04-02T04:59:42.124561+0000 mon.a (mon.0) 3387 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:59:42.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[21528]: cluster 2024-04-02T04:59:42.130646+0000 mon.a (mon.0) 3388 : cluster 0 osdmap e684: 8 total, 8 up, 7 in 2024-04-02T04:59:42.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:42 smithi069 bash[21528]: audit 2024-04-02T04:59:42.134277+0000 mon.a (mon.0) 3389 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 10 client.23052.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign -1 16 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 20 client.23052.objecter _get_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_remove -1 16 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 20 client.23052.objecter get_session s=0x7f225c1cc770 osd=7 3 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 15 client.23052.objecter _session_command_op_assign 7 16 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 4 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 10 client.23052.objecter _send_command 16 2024-04-02T04:59:42.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.282+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f225c1c47e0 con 0x7f225c1ccee0 2024-04-02T04:59:42.290 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T04:59:42.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.290+0000 7f22597fa700 1 -- 172.21.15.69:0/2974067208 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f225c1c47e0 con 0x7f225c1ccee0 2024-04-02T04:59:42.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.290+0000 7f22597fa700 10 client.23052.objecter ms_dispatch 0x7f225c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T04:59:42.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.290+0000 7f22597fa700 10 client.23052.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T04:59:42.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.290+0000 7f22597fa700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 4 2024-04-02T04:59:42.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.290+0000 7f22597fa700 15 client.23052.objecter _session_command_op_remove 7 16 2024-04-02T04:59:42.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.290+0000 7f2242ffd700 10 librados: watch_flush enter 2024-04-02T04:59:42.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 librados: watch_flush exit 2024-04-02T04:59:42.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.0 2024-04-02T04:59:42.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f225c02d180 msgr2=0x7f225c02d560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f225c02d180 0x7f225c02d560 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c10d1b0 osd=0 1 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.1 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f225c1b72e0 msgr2=0x7f225c1b96d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f225c1b72e0 0x7f225c1b96d0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c02dcf0 osd=1 1 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.2 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f225c1bad90 msgr2=0x7f225c1bd180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f225c1bad90 0x7f225c1bd180 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f226085d700 1 -- 172.21.15.69:0/2974067208 reap_dead start 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b4410 osd=2 1 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.3 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f225c1be840 msgr2=0x7f225c1beca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f225c1be840 0x7f225c1beca0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1b41e0 osd=3 1 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.4 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f225c1bfbf0 msgr2=0x7f225c1c41b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f225c1bfbf0 0x7f225c1c41b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1bf1e0 osd=4 1 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.5 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f225c1c6550 msgr2=0x7f225c1c69d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f225c1c6550 0x7f225c1c69d0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1c7220 osd=5 1 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.6 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f225c1bde90 msgr2=0x7f225c1be2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f225c1bde90 0x7f225c1be2b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1ca580 osd=6 1 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter close_session for osd.7 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f225c1ccee0 msgr2=0x7f225c1cd320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f225c1ccee0 0x7f225c1cd320 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f226085d700 1 -- 172.21.15.69:0/2974067208 reap_dead start 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter put_session s=0x7f225c1cc770 osd=7 1 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 20 client.23052.objecter shutdown clearing up homeless session... 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 10 client.23052.objecter successfully canceled tick 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f224406f6e0 msgr2=0x7f2244071ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f224406f6e0 0x7f2244071ba0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f225c06d9b0 tx=0x7f224c00a300 comp rx=0 tx=0).stop 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 msgr2=0x7f225c1ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1ac6a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f225400cd70 tx=0x7f2254007560 comp rx=0 tx=0).stop 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 shutdown_connections 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f225c109240 0x7f225c1ac6a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 --2- 172.21.15.69:0/2974067208 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f224406f6e0 0x7f2244071ba0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 >> 172.21.15.69:0/2974067208 conn(0x7f225c06d190 msgr2=0x7f225c10a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 shutdown_connections 2024-04-02T04:59:42.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 -- 172.21.15.69:0/2974067208 wait complete. 2024-04-02T04:59:42.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:42.294+0000 7f2242ffd700 1 librados: shutdown 2024-04-02T04:59:42.305 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T04:59:42.305 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T04:59:42.305 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T04:59:42.305 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T04:59:42.305 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T04:59:42.306 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T04:59:42.306 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T04:59:42.306 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T04:59:42.346 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:42.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:42.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:42.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:42 smithi098 bash[23790]: audit 2024-04-02T04:59:41.954391+0000 mon.a (mon.0) 3386 : audit 0 from='client.? 172.21.15.69:0/2974067208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T04:59:42.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:42 smithi098 bash[23790]: audit 2024-04-02T04:59:42.124561+0000 mon.a (mon.0) 3387 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-02T04:59:42.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:42 smithi098 bash[23790]: cluster 2024-04-02T04:59:42.130646+0000 mon.a (mon.0) 3388 : cluster 0 osdmap e684: 8 total, 8 up, 7 in 2024-04-02T04:59:42.459 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:42 smithi098 bash[23790]: audit 2024-04-02T04:59:42.134277+0000 mon.a (mon.0) 3389 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T04:59:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:42.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:42.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:42.600 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:42.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:42.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:42.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:42.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:42.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:42.912 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:42.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:43.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:43.057 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:43.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:43.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:43.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-02T04:59:43.187 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:43.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[28904]: cluster 2024-04-02T04:59:42.265565+0000 mgr.y (mgr.14538) 623 : cluster 0 pgmap v1032: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[28904]: cluster 2024-04-02T04:59:43.124938+0000 mon.a (mon.0) 3390 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[28904]: audit 2024-04-02T04:59:43.128971+0000 mon.a (mon.0) 3391 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:59:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[28904]: cluster 2024-04-02T04:59:43.132131+0000 mon.a (mon.0) 3392 : cluster 0 osdmap e685: 8 total, 8 up, 7 in 2024-04-02T04:59:43.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[28904]: audit 2024-04-02T04:59:43.139780+0000 mon.a (mon.0) 3393 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-02T04:59:43.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[21528]: cluster 2024-04-02T04:59:42.265565+0000 mgr.y (mgr.14538) 623 : cluster 0 pgmap v1032: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:43.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[21528]: cluster 2024-04-02T04:59:43.124938+0000 mon.a (mon.0) 3390 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:43.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[21528]: audit 2024-04-02T04:59:43.128971+0000 mon.a (mon.0) 3391 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:59:43.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[21528]: cluster 2024-04-02T04:59:43.132131+0000 mon.a (mon.0) 3392 : cluster 0 osdmap e685: 8 total, 8 up, 7 in 2024-04-02T04:59:43.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:43 smithi069 bash[21528]: audit 2024-04-02T04:59:43.139780+0000 mon.a (mon.0) 3393 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-02T04:59:43.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:43.394 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:43.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:43 smithi098 bash[23790]: cluster 2024-04-02T04:59:42.265565+0000 mgr.y (mgr.14538) 623 : cluster 0 pgmap v1032: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:43 smithi098 bash[23790]: cluster 2024-04-02T04:59:43.124938+0000 mon.a (mon.0) 3390 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T04:59:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:43 smithi098 bash[23790]: audit 2024-04-02T04:59:43.128971+0000 mon.a (mon.0) 3391 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T04:59:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:43 smithi098 bash[23790]: cluster 2024-04-02T04:59:43.132131+0000 mon.a (mon.0) 3392 : cluster 0 osdmap e685: 8 total, 8 up, 7 in 2024-04-02T04:59:43.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:43 smithi098 bash[23790]: audit 2024-04-02T04:59:43.139780+0000 mon.a (mon.0) 3393 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-02T04:59:43.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:43.529 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:43.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:43.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:43.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:43.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:43.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:43.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:43.918 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:43.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:59:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:59:44.158 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:44.183 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:44.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:44.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:44.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:44.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:44.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:44 smithi098 bash[23790]: audit 2024-04-02T04:59:43.555819+0000 mgr.y (mgr.14538) 624 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:44 smithi098 bash[23790]: audit 2024-04-02T04:59:44.130950+0000 mon.a (mon.0) 3394 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-02T04:59:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:44 smithi098 bash[23790]: cluster 2024-04-02T04:59:44.134401+0000 mon.a (mon.0) 3395 : cluster 0 osdmap e686: 8 total, 8 up, 7 in 2024-04-02T04:59:44.497 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:44.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:44.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:44 smithi069 bash[21528]: audit 2024-04-02T04:59:43.555819+0000 mgr.y (mgr.14538) 624 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:44.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:44 smithi069 bash[21528]: audit 2024-04-02T04:59:44.130950+0000 mon.a (mon.0) 3394 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-02T04:59:44.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:44 smithi069 bash[21528]: cluster 2024-04-02T04:59:44.134401+0000 mon.a (mon.0) 3395 : cluster 0 osdmap e686: 8 total, 8 up, 7 in 2024-04-02T04:59:44.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:44 smithi069 bash[28904]: audit 2024-04-02T04:59:43.555819+0000 mgr.y (mgr.14538) 624 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:44.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:44 smithi069 bash[28904]: audit 2024-04-02T04:59:44.130950+0000 mon.a (mon.0) 3394 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-02T04:59:44.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:44 smithi069 bash[28904]: cluster 2024-04-02T04:59:44.134401+0000 mon.a (mon.0) 3395 : cluster 0 osdmap e686: 8 total, 8 up, 7 in 2024-04-02T04:59:44.600 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:44.654 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:44.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:44.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:44.782 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:44.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:44.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:44.912 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:44.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:45.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:45.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:45.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:45.204 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:45.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:45.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:45.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:45 smithi098 bash[23790]: cluster 2024-04-02T04:59:44.266204+0000 mgr.y (mgr.14538) 625 : cluster 0 pgmap v1035: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:45.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:45 smithi098 bash[23790]: audit 2024-04-02T04:59:44.266666+0000 mon.a (mon.0) 3396 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:45.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:45 smithi098 bash[23790]: cluster 2024-04-02T04:59:45.021174+0000 mon.a (mon.0) 3397 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:45.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:45 smithi098 bash[23790]: audit 2024-04-02T04:59:45.024263+0000 mon.a (mon.0) 3398 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:45.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:45 smithi098 bash[23790]: cluster 2024-04-02T04:59:45.028681+0000 mon.a (mon.0) 3399 : cluster 0 osdmap e687: 8 total, 8 up, 7 in 2024-04-02T04:59:45.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:45 smithi098 bash[23790]: audit 2024-04-02T04:59:45.029915+0000 mon.a (mon.0) 3400 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:45.330 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:45.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:45.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[28904]: cluster 2024-04-02T04:59:44.266204+0000 mgr.y (mgr.14538) 625 : cluster 0 pgmap v1035: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[28904]: audit 2024-04-02T04:59:44.266666+0000 mon.a (mon.0) 3396 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[28904]: cluster 2024-04-02T04:59:45.021174+0000 mon.a (mon.0) 3397 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[28904]: audit 2024-04-02T04:59:45.024263+0000 mon.a (mon.0) 3398 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[28904]: cluster 2024-04-02T04:59:45.028681+0000 mon.a (mon.0) 3399 : cluster 0 osdmap e687: 8 total, 8 up, 7 in 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[28904]: audit 2024-04-02T04:59:45.029915+0000 mon.a (mon.0) 3400 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:45.456 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[21528]: cluster 2024-04-02T04:59:44.266204+0000 mgr.y (mgr.14538) 625 : cluster 0 pgmap v1035: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:45.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[21528]: audit 2024-04-02T04:59:44.266666+0000 mon.a (mon.0) 3396 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:45.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[21528]: cluster 2024-04-02T04:59:45.021174+0000 mon.a (mon.0) 3397 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:45.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[21528]: audit 2024-04-02T04:59:45.024263+0000 mon.a (mon.0) 3398 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:45.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[21528]: cluster 2024-04-02T04:59:45.028681+0000 mon.a (mon.0) 3399 : cluster 0 osdmap e687: 8 total, 8 up, 7 in 2024-04-02T04:59:45.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:45 smithi069 bash[21528]: audit 2024-04-02T04:59:45.029915+0000 mon.a (mon.0) 3400 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]: dispatch 2024-04-02T04:59:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:45.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:45.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:45.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T04:59:45.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:45.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:45.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:45.924 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:45.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:46.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:46.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:46.193 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:46.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:46.348 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:46.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:46.495 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:46.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:46.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:46.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:46.907 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:46.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:47.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:47.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.148 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:47 smithi098 bash[23790]: audit 2024-04-02T04:59:46.026090+0000 mon.a (mon.0) 3401 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:47.148 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:47 smithi098 bash[23790]: cluster 2024-04-02T04:59:46.031390+0000 mon.a (mon.0) 3402 : cluster 0 osdmap e688: 8 total, 8 up, 7 in 2024-04-02T04:59:47.148 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:47 smithi098 bash[23790]: cluster 2024-04-02T04:59:46.266751+0000 mgr.y (mgr.14538) 626 : cluster 0 pgmap v1038: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:47.161 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:47.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.288 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:47.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.416 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:47.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:47.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:47 smithi069 bash[28904]: audit 2024-04-02T04:59:46.026090+0000 mon.a (mon.0) 3401 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:47 smithi069 bash[28904]: cluster 2024-04-02T04:59:46.031390+0000 mon.a (mon.0) 3402 : cluster 0 osdmap e688: 8 total, 8 up, 7 in 2024-04-02T04:59:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:47 smithi069 bash[28904]: cluster 2024-04-02T04:59:46.266751+0000 mgr.y (mgr.14538) 626 : cluster 0 pgmap v1038: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:47.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:47 smithi069 bash[21528]: audit 2024-04-02T04:59:46.026090+0000 mon.a (mon.0) 3401 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-109", "tierpool": "test-rados-api-smithi069-72349-109-cache"}]': finished 2024-04-02T04:59:47.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:47 smithi069 bash[21528]: cluster 2024-04-02T04:59:46.031390+0000 mon.a (mon.0) 3402 : cluster 0 osdmap e688: 8 total, 8 up, 7 in 2024-04-02T04:59:47.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:47 smithi069 bash[21528]: cluster 2024-04-02T04:59:46.266751+0000 mgr.y (mgr.14538) 626 : cluster 0 pgmap v1038: 254 pgs: 254 active+clean; 635 KiB data, 629 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:47.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:47.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.803 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:47.906 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:47.981 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:47.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:48.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:48.156 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:48.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:48.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:48 smithi069 bash[21528]: cluster 2024-04-02T04:59:47.037616+0000 mon.a (mon.0) 3403 : cluster 0 osdmap e689: 8 total, 8 up, 7 in 2024-04-02T04:59:48.226 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:48 smithi069 bash[28904]: cluster 2024-04-02T04:59:47.037616+0000 mon.a (mon.0) 3403 : cluster 0 osdmap e689: 8 total, 8 up, 7 in 2024-04-02T04:59:48.252 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:48.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:48.305 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:48.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:48.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:48.416 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:48 smithi098 bash[23790]: cluster 2024-04-02T04:59:47.037616+0000 mon.a (mon.0) 3403 : cluster 0 osdmap e689: 8 total, 8 up, 7 in 2024-04-02T04:59:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:48.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:48.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:48.586 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:48.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:48.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:48.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:48.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:48.896 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:48.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:48.908 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:59:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:59:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:59:48.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:49.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:49.158 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:49.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:49.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: cluster 2024-04-02T04:59:48.050575+0000 mon.a (mon.0) 3404 : cluster 0 osdmap e690: 8 total, 8 up, 7 in 2024-04-02T04:59:49.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: audit 2024-04-02T04:59:48.052731+0000 mon.a (mon.0) 3405 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:49.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: cluster 2024-04-02T04:59:48.267668+0000 mgr.y (mgr.14538) 627 : cluster 0 pgmap v1041: 214 pgs: 214 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:49.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: audit 2024-04-02T04:59:48.768345+0000 mon.b (mon.1) 291 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:49.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: audit 2024-04-02T04:59:49.036396+0000 mon.a (mon.0) 3406 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:49.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: cluster 2024-04-02T04:59:49.042669+0000 mon.a (mon.0) 3407 : cluster 0 osdmap e691: 8 total, 8 up, 7 in 2024-04-02T04:59:49.271 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:49 smithi098 bash[23790]: audit 2024-04-02T04:59:49.043311+0000 mon.a (mon.0) 3408 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: cluster 2024-04-02T04:59:48.050575+0000 mon.a (mon.0) 3404 : cluster 0 osdmap e690: 8 total, 8 up, 7 in 2024-04-02T04:59:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: audit 2024-04-02T04:59:48.052731+0000 mon.a (mon.0) 3405 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: cluster 2024-04-02T04:59:48.267668+0000 mgr.y (mgr.14538) 627 : cluster 0 pgmap v1041: 214 pgs: 214 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: audit 2024-04-02T04:59:48.768345+0000 mon.b (mon.1) 291 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:49.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: audit 2024-04-02T04:59:49.036396+0000 mon.a (mon.0) 3406 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: cluster 2024-04-02T04:59:49.042669+0000 mon.a (mon.0) 3407 : cluster 0 osdmap e691: 8 total, 8 up, 7 in 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[21528]: audit 2024-04-02T04:59:49.043311+0000 mon.a (mon.0) 3408 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: cluster 2024-04-02T04:59:48.050575+0000 mon.a (mon.0) 3404 : cluster 0 osdmap e690: 8 total, 8 up, 7 in 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: audit 2024-04-02T04:59:48.052731+0000 mon.a (mon.0) 3405 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: cluster 2024-04-02T04:59:48.267668+0000 mgr.y (mgr.14538) 627 : cluster 0 pgmap v1041: 214 pgs: 214 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: audit 2024-04-02T04:59:48.768345+0000 mon.b (mon.1) 291 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: audit 2024-04-02T04:59:49.036396+0000 mon.a (mon.0) 3406 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: cluster 2024-04-02T04:59:49.042669+0000 mon.a (mon.0) 3407 : cluster 0 osdmap e691: 8 total, 8 up, 7 in 2024-04-02T04:59:49.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:49 smithi069 bash[28904]: audit 2024-04-02T04:59:49.043311+0000 mon.a (mon.0) 3408 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]: dispatch 2024-04-02T04:59:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:49.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.441 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:49.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:49.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:49.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:49.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:49.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:50.011 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:50.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:50.345 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:50.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:50 smithi069 bash[21528]: cluster 2024-04-02T04:59:50.022957+0000 mon.a (mon.0) 3409 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:50.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:50 smithi069 bash[21528]: audit 2024-04-02T04:59:50.053591+0000 mon.a (mon.0) 3410 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:50.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:50 smithi069 bash[21528]: cluster 2024-04-02T04:59:50.057270+0000 mon.a (mon.0) 3411 : cluster 0 osdmap e692: 8 total, 8 up, 7 in 2024-04-02T04:59:50.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:50 smithi069 bash[28904]: cluster 2024-04-02T04:59:50.022957+0000 mon.a (mon.0) 3409 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:50.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:50 smithi069 bash[28904]: audit 2024-04-02T04:59:50.053591+0000 mon.a (mon.0) 3410 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:50.361 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:50 smithi069 bash[28904]: cluster 2024-04-02T04:59:50.057270+0000 mon.a (mon.0) 3411 : cluster 0 osdmap e692: 8 total, 8 up, 7 in 2024-04-02T04:59:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:50.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:50 smithi098 bash[23790]: cluster 2024-04-02T04:59:50.022957+0000 mon.a (mon.0) 3409 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:50 smithi098 bash[23790]: audit 2024-04-02T04:59:50.053591+0000 mon.a (mon.0) 3410 : audit 1 from='client.? 172.21.15.69:0/903817344' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-109"}]': finished 2024-04-02T04:59:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:50 smithi098 bash[23790]: cluster 2024-04-02T04:59:50.057270+0000 mon.a (mon.0) 3411 : cluster 0 osdmap e692: 8 total, 8 up, 7 in 2024-04-02T04:59:50.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:50.526 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:50.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:50.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:50.667 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:50.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T04:59:50.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:50.902 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:50.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:51.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:51.081 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:51.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:51.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:51.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:51 smithi069 bash[21528]: cluster 2024-04-02T04:59:50.268243+0000 mgr.y (mgr.14538) 628 : cluster 0 pgmap v1044: 214 pgs: 214 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:51.280 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:51 smithi069 bash[28904]: cluster 2024-04-02T04:59:50.268243+0000 mgr.y (mgr.14538) 628 : cluster 0 pgmap v1044: 214 pgs: 214 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:51.416 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:51.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:51 smithi098 bash[23790]: cluster 2024-04-02T04:59:50.268243+0000 mgr.y (mgr.14538) 628 : cluster 0 pgmap v1044: 214 pgs: 214 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T04:59:51.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:51.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.654+0000 7f1c51646700 1 -- 172.21.15.69:0/3982453526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c1086a0 msgr2=0x7f1c4c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.654+0000 7f1c51646700 1 --2- 172.21.15.69:0/3982453526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c1086a0 0x7f1c4c108b20 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009990 tx=0x7f1c3c009ca0 comp rx=0 tx=0).stop 2024-04-02T04:59:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 -- 172.21.15.69:0/3982453526 shutdown_connections 2024-04-02T04:59:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 --2- 172.21.15.69:0/3982453526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4c109060 0x7f1c4c115570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 --2- 172.21.15.69:0/3982453526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c10e6a0 0x7f1c4c10ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 --2- 172.21.15.69:0/3982453526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c1086a0 0x7f1c4c108b20 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 -- 172.21.15.69:0/3982453526 >> 172.21.15.69:0/3982453526 conn(0x7f1c4c077cd0 msgr2=0x7f1c4c07a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 -- 172.21.15.69:0/3982453526 shutdown_connections 2024-04-02T04:59:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 -- 172.21.15.69:0/3982453526 wait complete. 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 librados: starting msgr at 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 librados: starting objecter 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 Processor -- start 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 -- start start 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 librados: setting wanted keys 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.658+0000 7f1c51646700 1 librados: calling monclient init 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.662+0000 7f1c51646700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4c1086a0 0x7f1c4c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.662+0000 7f1c51646700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c109060 0x7f1c4c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.662+0000 7f1c51646700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 0x7f1c4c1a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.662+0000 7f1c51646700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c118010 con 0x7f1c4c1086a0 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.662+0000 7f1c51646700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c117e90 con 0x7f1c4c109060 2024-04-02T04:59:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.662+0000 7f1c51646700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1c4c118190 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c4b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c109060 0x7f1c4c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 0x7f1c4c1a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 0x7f1c4c1a2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55482/0 (socket says 172.21.15.69:55482) 2024-04-02T04:59:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 -- 172.21.15.69:0/801355365 learned_addr learned my addr 172.21.15.69:0/801355365 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 -- 172.21.15.69:0/801355365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c109060 msgr2=0x7f1c4c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c109060 0x7f1c4c19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 -- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4c1086a0 msgr2=0x7f1c4c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4c1086a0 0x7f1c4c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 -- 172.21.15.69:0/801355365 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4c1a3680 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c50e45700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 0x7f1c4c1a2f40 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000b550 tx=0x7f1c4000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c497fa700 10 client.?.objecter ms_handle_connect 0x7f1c4c10e6a0 2024-04-02T04:59:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.666+0000 7f1c497fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40015020 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4c1a3970 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c51646700 10 client.41069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4c1ab1f0 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c51646700 1 librados: init done 2024-04-02T04:59:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f1c40011cb0 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40010c00 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f1c40018020 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.674+0000 7f1c497fa700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1c3406f670 0x7f1c34071b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.670+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c38005120 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(693..693 src has 74..693) ==== 9503+0+0 (secure 0 0 0) 0x7f1c4008de70 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c497fa700 10 client.41069.objecter ms_dispatch 0x7f1c4c001550 osd_map(693..693 src has 74..693) 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c497fa700 3 client.41069.objecter handle_osd_map got epochs [693,693] > 0 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c497fa700 3 client.41069.objecter handle_osd_map decoding full epoch 693 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c4bfff700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1c3406f670 0x7f1c34071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:51.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c497fa700 20 client.41069.objecter dump_active .. 0 homeless 2024-04-02T04:59:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.678+0000 7f1c4bfff700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1c3406f670 0x7f1c34071b30 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400ac00 tx=0x7f1c44009250 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.682+0000 7f1c497fa700 10 client.41069.objecter ms_handle_connect 0x7f1c3406f670 2024-04-02T04:59:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:51.682+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1c4001f020 con 0x7f1c4c10e6a0 2024-04-02T04:59:51.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:51.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:51.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:51.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:52.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:52.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.058+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f1c380056e0 con 0x7f1c4c10e6a0 2024-04-02T04:59:52.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:52.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.122+0000 7f1c497fa700 1 -- 172.21.15.69:0/801355365 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v694) ==== 91+0+0 (secure 0 0 0) 0x7f1c4005ad60 con 0x7f1c4c10e6a0 2024-04-02T04:59:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 10 librados: watch_flush enter 2024-04-02T04:59:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 10 librados: watch_flush exit 2024-04-02T04:59:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 20 client.41069.objecter shutdown clearing up homeless session... 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 10 client.41069.objecter successfully canceled tick 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1c3406f670 msgr2=0x7f1c34071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1c3406f670 0x7f1c34071b30 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400ac00 tx=0x7f1c44009250 comp rx=0 tx=0).stop 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 msgr2=0x7f1c4c1a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 0x7f1c4c1a2f40 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000b550 tx=0x7f1c4000b910 comp rx=0 tx=0).stop 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 shutdown_connections 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1c3406f670 0x7f1c34071b30 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4c10e6a0 0x7f1c4c1a2f40 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4c1086a0 0x7f1c4c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 --2- 172.21.15.69:0/801355365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c4c109060 0x7f1c4c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 >> 172.21.15.69:0/801355365 conn(0x7f1c4c077cd0 msgr2=0x7f1c4c0780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 shutdown_connections 2024-04-02T04:59:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 -- 172.21.15.69:0/801355365 wait complete. 2024-04-02T04:59:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:52.130+0000 7f1c51646700 1 librados: shutdown 2024-04-02T04:59:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T04:59:52.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:52.243 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:52.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:52.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[21528]: cluster 2024-04-02T04:59:51.109747+0000 mon.a (mon.0) 3412 : cluster 0 osdmap e693: 8 total, 8 up, 7 in 2024-04-02T04:59:52.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[21528]: audit 2024-04-02T04:59:51.122194+0000 mon.a (mon.0) 3413 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:52.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[21528]: audit 2024-04-02T04:59:52.063447+0000 mon.c (mon.2) 1146 : audit 1 from='client.? 172.21.15.69:0/801355365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:59:52.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[21528]: audit 2024-04-02T04:59:52.064794+0000 mon.a (mon.0) 3414 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:59:52.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[28904]: cluster 2024-04-02T04:59:51.109747+0000 mon.a (mon.0) 3412 : cluster 0 osdmap e693: 8 total, 8 up, 7 in 2024-04-02T04:59:52.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[28904]: audit 2024-04-02T04:59:51.122194+0000 mon.a (mon.0) 3413 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:52.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[28904]: audit 2024-04-02T04:59:52.063447+0000 mon.c (mon.2) 1146 : audit 1 from='client.? 172.21.15.69:0/801355365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:59:52.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:52 smithi069 bash[28904]: audit 2024-04-02T04:59:52.064794+0000 mon.a (mon.0) 3414 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:59:52.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:52.370 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:52.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:52.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:52 smithi098 bash[23790]: cluster 2024-04-02T04:59:51.109747+0000 mon.a (mon.0) 3412 : cluster 0 osdmap e693: 8 total, 8 up, 7 in 2024-04-02T04:59:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:52 smithi098 bash[23790]: audit 2024-04-02T04:59:51.122194+0000 mon.a (mon.0) 3413 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:52 smithi098 bash[23790]: audit 2024-04-02T04:59:52.063447+0000 mon.c (mon.2) 1146 : audit 1 from='client.? 172.21.15.69:0/801355365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:59:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:52 smithi098 bash[23790]: audit 2024-04-02T04:59:52.064794+0000 mon.a (mon.0) 3414 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T04:59:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:52.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:52.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:52.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:52.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:52.846 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:52.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:52.949 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:53.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:53.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:53.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:53.147 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:53.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:53.250 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:53.331 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:53.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:53.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:53.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: cluster 2024-04-02T04:59:52.103589+0000 mon.a (mon.0) 3415 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:53.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: audit 2024-04-02T04:59:52.107728+0000 mon.a (mon.0) 3416 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:53.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: audit 2024-04-02T04:59:52.107801+0000 mon.a (mon.0) 3417 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:59:53.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: cluster 2024-04-02T04:59:52.113616+0000 mon.a (mon.0) 3418 : cluster 0 osdmap e694: 8 total, 8 up, 7 in 2024-04-02T04:59:53.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: audit 2024-04-02T04:59:52.120180+0000 mon.a (mon.0) 3419 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:53.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: cluster 2024-04-02T04:59:52.269490+0000 mgr.y (mgr.14538) 629 : cluster 0 pgmap v1047: 206 pgs: 206 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:53.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: audit 2024-04-02T04:59:53.110024+0000 mon.a (mon.0) 3420 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:53.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.115564+0000 mon.a (mon.0) 3421 : cluster 0 osdmap e695: 8 total, 8 up, 7 in 2024-04-02T04:59:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:53.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:53.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: cluster 2024-04-02T04:59:52.103589+0000 mon.a (mon.0) 3415 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:53.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: audit 2024-04-02T04:59:52.107728+0000 mon.a (mon.0) 3416 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:53.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: audit 2024-04-02T04:59:52.107801+0000 mon.a (mon.0) 3417 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:59:53.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: cluster 2024-04-02T04:59:52.113616+0000 mon.a (mon.0) 3418 : cluster 0 osdmap e694: 8 total, 8 up, 7 in 2024-04-02T04:59:53.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: audit 2024-04-02T04:59:52.120180+0000 mon.a (mon.0) 3419 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:53.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: cluster 2024-04-02T04:59:52.269490+0000 mgr.y (mgr.14538) 629 : cluster 0 pgmap v1047: 206 pgs: 206 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: audit 2024-04-02T04:59:53.110024+0000 mon.a (mon.0) 3420 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.115564+0000 mon.a (mon.0) 3421 : cluster 0 osdmap e695: 8 total, 8 up, 7 in 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: cluster 2024-04-02T04:59:52.103589+0000 mon.a (mon.0) 3415 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: audit 2024-04-02T04:59:52.107728+0000 mon.a (mon.0) 3416 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: audit 2024-04-02T04:59:52.107801+0000 mon.a (mon.0) 3417 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: cluster 2024-04-02T04:59:52.113616+0000 mon.a (mon.0) 3418 : cluster 0 osdmap e694: 8 total, 8 up, 7 in 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: audit 2024-04-02T04:59:52.120180+0000 mon.a (mon.0) 3419 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]: dispatch 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: cluster 2024-04-02T04:59:52.269490+0000 mgr.y (mgr.14538) 629 : cluster 0 pgmap v1047: 206 pgs: 206 active+clean; 635 KiB data, 630 MiB used, 625 GiB / 626 GiB avail 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: audit 2024-04-02T04:59:53.110024+0000 mon.a (mon.0) 3420 : audit 1 from='client.? 172.21.15.69:0/3364239127' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-72349-104"}]': finished 2024-04-02T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:53 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.115564+0000 mon.a (mon.0) 3421 : cluster 0 osdmap e695: 8 total, 8 up, 7 in 2024-04-02T04:59:53.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:53.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:53.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:53.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:53.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 04:59:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T04:59:53.798 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:53.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:53.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:54.003 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:54.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:54.136 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:54.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: cluster 2024-04-02T04:59:52.861665+0000 osd.6 (osd.6) 293 : cluster 0 10.7 scrub starts 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: cluster 2024-04-02T04:59:52.868870+0000 osd.6 (osd.6) 294 : cluster 0 10.7 scrub ok 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.179231+0000 mon.b (mon.1) 292 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.179524+0000 mon.b (mon.1) 293 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]: dispatch 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.179751+0000 mon.b (mon.1) 294 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]: dispatch 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.183036+0000 mon.a (mon.0) 3422 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-02T04:59:54.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.183433+0000 mon.a (mon.0) 3423 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.183945+0000 mon.a (mon.0) 3424 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.193098+0000 mon.a (mon.0) 3425 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.196805+0000 mon.a (mon.0) 3426 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.197336+0000 mon.a (mon.0) 3427 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: audit 2024-04-02T04:59:53.570939+0000 mgr.y (mgr.14538) 630 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.670321+0000 osd.3 (osd.3) 339 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.670661+0000 osd.3 (osd.3) 340 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.762836+0000 osd.0 (osd.0) 343 : cluster 0 6.9 scrub starts 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.764565+0000 osd.0 (osd.0) 344 : cluster 0 6.9 scrub ok 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: cluster 2024-04-02T04:59:52.861665+0000 osd.6 (osd.6) 293 : cluster 0 10.7 scrub starts 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: cluster 2024-04-02T04:59:52.868870+0000 osd.6 (osd.6) 294 : cluster 0 10.7 scrub ok 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.179231+0000 mon.b (mon.1) 292 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-02T04:59:54.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.179524+0000 mon.b (mon.1) 293 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.179751+0000 mon.b (mon.1) 294 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.183036+0000 mon.a (mon.0) 3422 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.183433+0000 mon.a (mon.0) 3423 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.183945+0000 mon.a (mon.0) 3424 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.193098+0000 mon.a (mon.0) 3425 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.196805+0000 mon.a (mon.0) 3426 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.197336+0000 mon.a (mon.0) 3427 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: audit 2024-04-02T04:59:53.570939+0000 mgr.y (mgr.14538) 630 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.670321+0000 osd.3 (osd.3) 339 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.670661+0000 osd.3 (osd.3) 340 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.762836+0000 osd.0 (osd.0) 343 : cluster 0 6.9 scrub starts 2024-04-02T04:59:54.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:54 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.764565+0000 osd.0 (osd.0) 344 : cluster 0 6.9 scrub ok 2024-04-02T04:59:54.309 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:54.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: cluster 2024-04-02T04:59:52.861665+0000 osd.6 (osd.6) 293 : cluster 0 10.7 scrub starts 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: cluster 2024-04-02T04:59:52.868870+0000 osd.6 (osd.6) 294 : cluster 0 10.7 scrub ok 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.179231+0000 mon.b (mon.1) 292 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.179524+0000 mon.b (mon.1) 293 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]: dispatch 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.179751+0000 mon.b (mon.1) 294 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]: dispatch 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.183036+0000 mon.a (mon.0) 3422 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.183433+0000 mon.a (mon.0) 3423 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]: dispatch 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.183945+0000 mon.a (mon.0) 3424 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]: dispatch 2024-04-02T04:59:54.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.193098+0000 mon.a (mon.0) 3425 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.196805+0000 mon.a (mon.0) 3426 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.197336+0000 mon.a (mon.0) 3427 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: audit 2024-04-02T04:59:53.570939+0000 mgr.y (mgr.14538) 630 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.670321+0000 osd.3 (osd.3) 339 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.670661+0000 osd.3 (osd.3) 340 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.762836+0000 osd.0 (osd.0) 343 : cluster 0 6.9 scrub starts 2024-04-02T04:59:54.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:54 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.764565+0000 osd.0 (osd.0) 344 : cluster 0 6.9 scrub ok 2024-04-02T04:59:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:54.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:54.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:54.697 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:54.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:54.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:54.943 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:54.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:55.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:55.075 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:55.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:55.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:55.328 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:55.431 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:55.452 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:55.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-02T04:59:55.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-02T04:59:55.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-02T04:59:55.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-02T04:59:55.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21678 ms) 2024-04-02T04:59:55.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc (26793 ms total) 2024-04-02T04:59:55.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-04-02T04:59:55.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment tear-down 2024-04-02T04:59:55.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (645438 ms total) 2024-04-02T04:59:55.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-02T04:59:55.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:55.584 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:55.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:55.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.716994+0000 osd.3 (osd.3) 341 : cluster 0 4.1b scrub starts 2024-04-02T04:59:55.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:53.718256+0000 osd.3 (osd.3) 342 : cluster 0 4.1b scrub ok 2024-04-02T04:59:55.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:54.090860+0000 osd.1 (osd.1) 351 : cluster 0 10.19 scrub starts 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:54.093757+0000 osd.1 (osd.1) 352 : cluster 0 10.19 scrub ok 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: audit 2024-04-02T04:59:54.160764+0000 mon.a (mon.0) 3428 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: audit 2024-04-02T04:59:54.160853+0000 mon.a (mon.0) 3429 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]': finished 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: audit 2024-04-02T04:59:54.160937+0000 mon.a (mon.0) 3430 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]': finished 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: audit 2024-04-02T04:59:54.161021+0000 mon.a (mon.0) 3431 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:54.165394+0000 mon.a (mon.0) 3432 : cluster 0 osdmap e696: 8 total, 8 up, 7 in 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: audit 2024-04-02T04:59:54.165747+0000 mon.a (mon.0) 3433 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:54.270084+0000 mgr.y (mgr.14538) 631 : cluster 0 pgmap v1050: 206 pgs: 206 active+clean; 635 KiB data, 652 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.024169+0000 mon.a (mon.0) 3434 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.716994+0000 osd.3 (osd.3) 341 : cluster 0 4.1b scrub starts 2024-04-02T04:59:55.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:53.718256+0000 osd.3 (osd.3) 342 : cluster 0 4.1b scrub ok 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:54.090860+0000 osd.1 (osd.1) 351 : cluster 0 10.19 scrub starts 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:54.093757+0000 osd.1 (osd.1) 352 : cluster 0 10.19 scrub ok 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: audit 2024-04-02T04:59:54.160764+0000 mon.a (mon.0) 3428 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: audit 2024-04-02T04:59:54.160853+0000 mon.a (mon.0) 3429 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]': finished 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: audit 2024-04-02T04:59:54.160937+0000 mon.a (mon.0) 3430 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]': finished 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: audit 2024-04-02T04:59:54.161021+0000 mon.a (mon.0) 3431 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:54.165394+0000 mon.a (mon.0) 3432 : cluster 0 osdmap e696: 8 total, 8 up, 7 in 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: audit 2024-04-02T04:59:54.165747+0000 mon.a (mon.0) 3433 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:54.270084+0000 mgr.y (mgr.14538) 631 : cluster 0 pgmap v1050: 206 pgs: 206 active+clean; 635 KiB data, 652 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-02T04:59:55.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:55 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.024169+0000 mon.a (mon.0) 3434 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:55.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.716994+0000 osd.3 (osd.3) 341 : cluster 0 4.1b scrub starts 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:53.718256+0000 osd.3 (osd.3) 342 : cluster 0 4.1b scrub ok 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:54.090860+0000 osd.1 (osd.1) 351 : cluster 0 10.19 scrub starts 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:54.093757+0000 osd.1 (osd.1) 352 : cluster 0 10.19 scrub ok 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: audit 2024-04-02T04:59:54.160764+0000 mon.a (mon.0) 3428 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: audit 2024-04-02T04:59:54.160853+0000 mon.a (mon.0) 3429 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [2, 1]}]': finished 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: audit 2024-04-02T04:59:54.160937+0000 mon.a (mon.0) 3430 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [2, 0]}]': finished 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: audit 2024-04-02T04:59:54.161021+0000 mon.a (mon.0) 3431 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-72349-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:54.165394+0000 mon.a (mon.0) 3432 : cluster 0 osdmap e696: 8 total, 8 up, 7 in 2024-04-02T04:59:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: audit 2024-04-02T04:59:54.165747+0000 mon.a (mon.0) 3433 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T04:59:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:54.270084+0000 mgr.y (mgr.14538) 631 : cluster 0 pgmap v1050: 206 pgs: 206 active+clean; 635 KiB data, 652 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-02T04:59:55.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:55 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.024169+0000 mon.a (mon.0) 3434 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T04:59:55.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:55.840 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:55.943 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:55.970 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:55.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:56.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:56.093 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:56.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:56.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:56.355 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:56.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 -- 172.21.15.69:0/3484856240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35441082e0 msgr2=0x7f35441086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- 172.21.15.69:0/3484856240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35441082e0 0x7f35441086c0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f3534009990 tx=0x7f3534009ca0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 -- 172.21.15.69:0/3484856240 shutdown_connections 2024-04-02T04:59:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- 172.21.15.69:0/3484856240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f354406b2a0 0x7f354410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- 172.21.15.69:0/3484856240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35441082e0 0x7f35441086c0 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- 172.21.15.69:0/3484856240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f354406a8e0 0x7f354406ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 -- 172.21.15.69:0/3484856240 >> 172.21.15.69:0/3484856240 conn(0x7f3544078110 msgr2=0x7f3544078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 -- 172.21.15.69:0/3484856240 shutdown_connections 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 -- 172.21.15.69:0/3484856240 wait complete. 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 librados: starting msgr at 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 librados: starting objecter 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 Processor -- start 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 -- start start 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 librados: setting wanted keys 2024-04-02T04:59:56.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 librados: calling monclient init 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 0x7f3544198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f354406b2a0 0x7f3544198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.458+0000 7f354a96d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35441082e0 0x7f354419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111a90 con 0x7f354406a8e0 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111910 con 0x7f354406b2a0 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3544111c10 con 0x7f35441082e0 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3543fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 0x7f3544198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35441082e0 0x7f354419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3543fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 0x7f3544198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34132/0 (socket says 172.21.15.69:34132) 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3543fff700 1 -- 172.21.15.69:0/3638894306 learned_addr learned my addr 172.21.15.69:0/3638894306 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35437fe700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f354406b2a0 0x7f3544198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 -- 172.21.15.69:0/3638894306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f354406b2a0 msgr2=0x7f3544198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f354406b2a0 0x7f3544198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 -- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 msgr2=0x7f3544198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 0x7f3544198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 -- 172.21.15.69:0/3638894306 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354419d140 con 0x7f35441082e0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3543fff700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 0x7f3544198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3548f0a700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35441082e0 0x7f354419ca00 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7f353c007f10 tx=0x7f353c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 10 client.?.objecter ms_handle_connect 0x7f35441082e0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 10 client.?.objecter resend_mon_ops 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c00ddd0 con 0x7f35441082e0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354419d430 con 0x7f35441082e0 2024-04-02T04:59:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f353c009e20 con 0x7f35441082e0 2024-04-02T04:59:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c015400 con 0x7f35441082e0 2024-04-02T04:59:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 10 client.43643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T04:59:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441a4cb0 con 0x7f35441082e0 2024-04-02T04:59:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 librados: init done 2024-04-02T04:59:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3544103990 con 0x7f35441082e0 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f353c004ad0 con 0x7f35441082e0 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f352c06f5e0 0x7f352c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(698..698 src has 74..698) ==== 9581+0+0 (secure 0 0 0) 0x7f353c08e230 con 0x7f35441082e0 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 10 client.43643.objecter ms_dispatch 0x7f3544001550 osd_map(698..698 src has 74..698) 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 3 client.43643.objecter handle_osd_map got epochs [698,698] > 0 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 3 client.43643.objecter handle_osd_map decoding full epoch 698 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f3543fff700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f352c06f5e0 0x7f352c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.462+0000 7f35417fa700 20 client.43643.objecter dump_active .. 0 homeless 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.466+0000 7f35417fa700 10 client.43643.objecter ms_handle_connect 0x7f352c06f5e0 2024-04-02T04:59:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.466+0000 7f3543fff700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f352c06f5e0 0x7f352c071aa0 secure :-1 s=READY pgs=3987 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f3534005c00 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T04:59:56.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.470+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f353c010070 con 0x7f35441082e0 2024-04-02T04:59:56.479 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:56.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.666+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3544199490 con 0x7f352c06f5e0 2024-04-02T04:59:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.690+0000 7f35417fa700 1 -- 172.21.15.69:0/3638894306 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+471455 (secure 0 0 0) 0x7f3544199490 con 0x7f352c06f5e0 2024-04-02T04:59:56.694 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T04:59:56.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 10 librados: watch_flush enter 2024-04-02T04:59:56.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 10 librados: watch_flush exit 2024-04-02T04:59:56.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 20 client.43643.objecter shutdown clearing up homeless session... 2024-04-02T04:59:56.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 10 client.43643.objecter successfully canceled tick 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f352c06f5e0 msgr2=0x7f352c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f352c06f5e0 0x7f352c071aa0 secure :-1 s=READY pgs=3987 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f3534005c00 comp rx=0 tx=0).stop 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35441082e0 msgr2=0x7f354419ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35441082e0 0x7f354419ca00 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7f353c007f10 tx=0x7f353c00d3b0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 shutdown_connections 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f352c06f5e0 0x7f352c071aa0 unknown :-1 s=CLOSED pgs=3987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35441082e0 0x7f354419ca00 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f354406a8e0 0x7f3544198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 --2- 172.21.15.69:0/3638894306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f354406b2a0 0x7f3544198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 >> 172.21.15.69:0/3638894306 conn(0x7f3544078110 msgr2=0x7f35440ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T04:59:56.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 shutdown_connections 2024-04-02T04:59:56.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 -- 172.21.15.69:0/3638894306 wait complete. 2024-04-02T04:59:56.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T04:59:56.698+0000 7f354a96d700 1 librados: shutdown 2024-04-02T04:59:56.706 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T04:59:56.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.123475+0000 osd.1 (osd.1) 353 : cluster 0 10.14 scrub starts 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.124633+0000 osd.1 (osd.1) 354 : cluster 0 10.14 scrub ok 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.298939+0000 osd.4 (osd.4) 352 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.299290+0000 osd.4 (osd.4) 353 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.343098+0000 osd.7 (osd.7) 153 : cluster 0 10.c scrub starts 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.345449+0000 osd.7 (osd.7) 154 : cluster 0 10.c scrub ok 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.471402+0000 mon.a (mon.0) 3435 : cluster 0 osdmap e697: 8 total, 8 up, 7 in 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: audit 2024-04-02T04:59:56.029461+0000 mon.a (mon.0) 3436 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T04:59:56.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[21528]: cluster 2024-04-02T04:59:56.033857+0000 mon.a (mon.0) 3437 : cluster 0 osdmap e698: 8 total, 8 up, 7 in 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.123475+0000 osd.1 (osd.1) 353 : cluster 0 10.14 scrub starts 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.124633+0000 osd.1 (osd.1) 354 : cluster 0 10.14 scrub ok 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.298939+0000 osd.4 (osd.4) 352 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.299290+0000 osd.4 (osd.4) 353 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.343098+0000 osd.7 (osd.7) 153 : cluster 0 10.c scrub starts 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.345449+0000 osd.7 (osd.7) 154 : cluster 0 10.c scrub ok 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.471402+0000 mon.a (mon.0) 3435 : cluster 0 osdmap e697: 8 total, 8 up, 7 in 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: audit 2024-04-02T04:59:56.029461+0000 mon.a (mon.0) 3436 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T04:59:56.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:56 smithi069 bash[28904]: cluster 2024-04-02T04:59:56.033857+0000 mon.a (mon.0) 3437 : cluster 0 osdmap e698: 8 total, 8 up, 7 in 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.123475+0000 osd.1 (osd.1) 353 : cluster 0 10.14 scrub starts 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.124633+0000 osd.1 (osd.1) 354 : cluster 0 10.14 scrub ok 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.298939+0000 osd.4 (osd.4) 352 : cluster 0 purged_snaps scrub starts 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.299290+0000 osd.4 (osd.4) 353 : cluster 0 purged_snaps scrub ok 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.343098+0000 osd.7 (osd.7) 153 : cluster 0 10.c scrub starts 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.345449+0000 osd.7 (osd.7) 154 : cluster 0 10.c scrub ok 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.471402+0000 mon.a (mon.0) 3435 : cluster 0 osdmap e697: 8 total, 8 up, 7 in 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: audit 2024-04-02T04:59:56.029461+0000 mon.a (mon.0) 3436 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-72349-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T04:59:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:56 smithi098 bash[23790]: cluster 2024-04-02T04:59:56.033857+0000 mon.a (mon.0) 3437 : cluster 0 osdmap e698: 8 total, 8 up, 7 in 2024-04-02T04:59:56.733 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:56.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:56.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T04:59:56.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:56.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:56.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:57.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:57.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:57.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T04:59:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T04:59:57.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:57.503 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:57.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.742383+0000 osd.3 (osd.3) 343 : cluster 0 5.0 scrub starts 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:55.744207+0000 osd.3 (osd.3) 344 : cluster 0 5.0 scrub ok 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:56.270748+0000 mgr.y (mgr.14538) 632 : cluster 0 pgmap v1053: 182 pgs: 8 unknown, 174 active+clean; 635 KiB data, 652 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:56.357073+0000 osd.7 (osd.7) 155 : cluster 0 3.15 scrub starts 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:56.359322+0000 osd.7 (osd.7) 156 : cluster 0 3.15 scrub ok 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: audit 2024-04-02T04:59:56.669763+0000 mgr.y (mgr.14538) 633 : audit 0 from='client.43643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:59:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:56.947350+0000 osd.6 (osd.6) 295 : cluster 0 2.16 scrub starts 2024-04-02T04:59:57.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:56.948896+0000 osd.6 (osd.6) 296 : cluster 0 2.16 scrub ok 2024-04-02T04:59:57.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: cluster 2024-04-02T04:59:57.040563+0000 mon.a (mon.0) 3438 : cluster 0 osdmap e699: 8 total, 8 up, 7 in 2024-04-02T04:59:57.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:57 smithi098 bash[23790]: audit 2024-04-02T04:59:57.059081+0000 mon.a (mon.0) 3439 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:57.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:57.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.742383+0000 osd.3 (osd.3) 343 : cluster 0 5.0 scrub starts 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:55.744207+0000 osd.3 (osd.3) 344 : cluster 0 5.0 scrub ok 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:56.270748+0000 mgr.y (mgr.14538) 632 : cluster 0 pgmap v1053: 182 pgs: 8 unknown, 174 active+clean; 635 KiB data, 652 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:56.357073+0000 osd.7 (osd.7) 155 : cluster 0 3.15 scrub starts 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:56.359322+0000 osd.7 (osd.7) 156 : cluster 0 3.15 scrub ok 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: audit 2024-04-02T04:59:56.669763+0000 mgr.y (mgr.14538) 633 : audit 0 from='client.43643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:56.947350+0000 osd.6 (osd.6) 295 : cluster 0 2.16 scrub starts 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:56.948896+0000 osd.6 (osd.6) 296 : cluster 0 2.16 scrub ok 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: cluster 2024-04-02T04:59:57.040563+0000 mon.a (mon.0) 3438 : cluster 0 osdmap e699: 8 total, 8 up, 7 in 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[21528]: audit 2024-04-02T04:59:57.059081+0000 mon.a (mon.0) 3439 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.742383+0000 osd.3 (osd.3) 343 : cluster 0 5.0 scrub starts 2024-04-02T04:59:57.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:55.744207+0000 osd.3 (osd.3) 344 : cluster 0 5.0 scrub ok 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:56.270748+0000 mgr.y (mgr.14538) 632 : cluster 0 pgmap v1053: 182 pgs: 8 unknown, 174 active+clean; 635 KiB data, 652 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:56.357073+0000 osd.7 (osd.7) 155 : cluster 0 3.15 scrub starts 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:56.359322+0000 osd.7 (osd.7) 156 : cluster 0 3.15 scrub ok 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: audit 2024-04-02T04:59:56.669763+0000 mgr.y (mgr.14538) 633 : audit 0 from='client.43643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:56.947350+0000 osd.6 (osd.6) 295 : cluster 0 2.16 scrub starts 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:56.948896+0000 osd.6 (osd.6) 296 : cluster 0 2.16 scrub ok 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: cluster 2024-04-02T04:59:57.040563+0000 mon.a (mon.0) 3438 : cluster 0 osdmap e699: 8 total, 8 up, 7 in 2024-04-02T04:59:57.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:57 smithi069 bash[28904]: audit 2024-04-02T04:59:57.059081+0000 mon.a (mon.0) 3439 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:59:57.766 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:57.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:57.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:57.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:58.027 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:58.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:58.152 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:58.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.282 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:58.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.455 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:58.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T04:59:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T04:59:58.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:58.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T04:59:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T04:59:58.946 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.957 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 04:59:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:04:59:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T04:59:58.974 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:58.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:59.077 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:59.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:59.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: cluster 2024-04-02T04:59:57.834838+0000 osd.4 (osd.4) 354 : cluster 0 3.1a scrub starts 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: cluster 2024-04-02T04:59:57.839461+0000 osd.4 (osd.4) 355 : cluster 0 3.1a scrub ok 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.017483+0000 osd.2 (osd.2) 315 : cluster 0 1.5 scrub starts 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.019389+0000 osd.2 (osd.2) 316 : cluster 0 1.5 scrub ok 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: audit 2024-04-02T04:59:58.039766+0000 mon.a (mon.0) 3440 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.044734+0000 mon.a (mon.0) 3441 : cluster 0 osdmap e700: 8 total, 8 up, 7 in 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: audit 2024-04-02T04:59:58.089326+0000 mon.a (mon.0) 3442 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 04:59:59 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.271755+0000 mgr.y (mgr.14538) 634 : cluster 0 pgmap v1056: 214 pgs: 14 creating+peering, 3 unknown, 197 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-02T04:59:59.225 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: cluster 2024-04-02T04:59:57.834838+0000 osd.4 (osd.4) 354 : cluster 0 3.1a scrub starts 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: cluster 2024-04-02T04:59:57.839461+0000 osd.4 (osd.4) 355 : cluster 0 3.1a scrub ok 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.017483+0000 osd.2 (osd.2) 315 : cluster 0 1.5 scrub starts 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.019389+0000 osd.2 (osd.2) 316 : cluster 0 1.5 scrub ok 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: audit 2024-04-02T04:59:58.039766+0000 mon.a (mon.0) 3440 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.044734+0000 mon.a (mon.0) 3441 : cluster 0 osdmap e700: 8 total, 8 up, 7 in 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: audit 2024-04-02T04:59:58.089326+0000 mon.a (mon.0) 3442 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:59.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.271755+0000 mgr.y (mgr.14538) 634 : cluster 0 pgmap v1056: 214 pgs: 14 creating+peering, 3 unknown, 197 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-02T04:59:59.235 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T04:59:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T04:59:59.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:59.364 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T04:59:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T04:59:59.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: cluster 2024-04-02T04:59:57.834838+0000 osd.4 (osd.4) 354 : cluster 0 3.1a scrub starts 2024-04-02T04:59:59.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: cluster 2024-04-02T04:59:57.839461+0000 osd.4 (osd.4) 355 : cluster 0 3.1a scrub ok 2024-04-02T04:59:59.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.017483+0000 osd.2 (osd.2) 315 : cluster 0 1.5 scrub starts 2024-04-02T04:59:59.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.019389+0000 osd.2 (osd.2) 316 : cluster 0 1.5 scrub ok 2024-04-02T04:59:59.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: audit 2024-04-02T04:59:58.039766+0000 mon.a (mon.0) 3440 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T04:59:59.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.044734+0000 mon.a (mon.0) 3441 : cluster 0 osdmap e700: 8 total, 8 up, 7 in 2024-04-02T04:59:59.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: audit 2024-04-02T04:59:58.089326+0000 mon.a (mon.0) 3442 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T04:59:59.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 04:59:59 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.271755+0000 mgr.y (mgr.14538) 634 : cluster 0 pgmap v1056: 214 pgs: 14 creating+peering, 3 unknown, 197 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-02T04:59:59.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:59.493 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:59.596 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:59.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T04:59:59.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T04:59:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T04:59:59.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:59.886 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T04:59:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:59:59.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T04:59:59.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T04:59:59.996 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1053,"stamp":"2024-04-02T04:59:56.270280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649993,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":1506,"num_read_kb":1235,"num_write":937,"num_write_kb":272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":956715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2502,"ondisk_log_size":2502,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":645,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":667820,"kb_used_data":83136,"kb_used_omap":66,"kb_used_meta":584381,"kb_avail":655548244,"statfs":{"total":671965249536,"available":671281401856,"internally_reserved":0,"allocated":85131264,"data_stored":56384658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68049,"internal_metadata":598406703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-651,"num_objects":-3,"num_object_clones":0,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":-5,"num_write_kb":-5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001098"},"pg_stats":[{"pgid":"323.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"323.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T04:59:56.027374+0000","last_change":"2024-04-02T04:59:56.027374+0000","last_active":"2024-04-02T04:59:56.027374+0000","last_peered":"2024-04-02T04:59:56.027374+0000","last_clean":"2024-04-02T04:59:56.027374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T04:59:56.027374+0000","last_undegraded":"2024-04-02T04:59:56.027374+0000","last_fullsized":"2024-04-02T04:59:56.027374+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":505,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.593691+0000","last_change":"2024-04-02T04:54:56.017671+0000","last_active":"2024-04-02T04:59:52.593691+0000","last_peered":"2024-04-02T04:59:52.593691+0000","last_clean":"2024-04-02T04:59:52.593691+0000","last_became_active":"2024-04-02T04:54:56.017309+0000","last_became_peered":"2024-04-02T04:54:56.017309+0000","last_unstale":"2024-04-02T04:59:52.593691+0000","last_undegraded":"2024-04-02T04:59:52.593691+0000","last_fullsized":"2024-04-02T04:59:52.593691+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:35.773829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:35.773829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1180,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.335669+0000","last_change":"2024-04-02T04:59:03.454678+0000","last_active":"2024-04-02T04:59:53.335669+0000","last_peered":"2024-04-02T04:59:53.335669+0000","last_clean":"2024-04-02T04:59:53.335669+0000","last_became_active":"2024-04-02T04:54:55.990081+0000","last_became_peered":"2024-04-02T04:54:55.990081+0000","last_unstale":"2024-04-02T04:59:53.335669+0000","last_undegraded":"2024-04-02T04:59:53.335669+0000","last_fullsized":"2024-04-02T04:59:53.335669+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:03.454565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:03.454565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:11.894457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"415'28","reported_seq":804,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.339095+0000","last_change":"2024-04-02T04:59:09.515424+0000","last_active":"2024-04-02T04:59:53.339095+0000","last_peered":"2024-04-02T04:59:53.339095+0000","last_clean":"2024-04-02T04:59:53.339095+0000","last_became_active":"2024-04-02T04:53:54.938166+0000","last_became_peered":"2024-04-02T04:53:54.938166+0000","last_unstale":"2024-04-02T04:59:53.339095+0000","last_undegraded":"2024-04-02T04:59:53.339095+0000","last_fullsized":"2024-04-02T04:59:53.339095+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'28","last_scrub_stamp":"2024-04-02T04:59:09.515130+0000","last_deep_scrub":"415'28","last_deep_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.515130+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:25.469041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1298,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588369+0000","last_change":"2024-04-02T04:59:00.088938+0000","last_active":"2024-04-02T04:59:52.588369+0000","last_peered":"2024-04-02T04:59:52.588369+0000","last_clean":"2024-04-02T04:59:52.588369+0000","last_became_active":"2024-04-02T04:59:00.088594+0000","last_became_peered":"2024-04-02T04:59:00.088594+0000","last_unstale":"2024-04-02T04:59:52.588369+0000","last_undegraded":"2024-04-02T04:59:52.588369+0000","last_fullsized":"2024-04-02T04:59:52.588369+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:29.179769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:29.179769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:56.102446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":508,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.111122+0000","last_change":"2024-04-02T04:58:59.898789+0000","last_active":"2024-04-02T04:59:52.111122+0000","last_peered":"2024-04-02T04:59:52.111122+0000","last_clean":"2024-04-02T04:59:52.111122+0000","last_became_active":"2024-04-02T04:54:56.002149+0000","last_became_peered":"2024-04-02T04:54:56.002149+0000","last_unstale":"2024-04-02T04:59:52.111122+0000","last_undegraded":"2024-04-02T04:59:52.111122+0000","last_fullsized":"2024-04-02T04:59:52.111122+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_clean_scrub_stamp":"2024-04-02T04:58:59.898499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:00.219199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1163,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588229+0000","last_change":"2024-04-02T04:59:00.559427+0000","last_active":"2024-04-02T04:59:52.588229+0000","last_peered":"2024-04-02T04:59:52.588229+0000","last_clean":"2024-04-02T04:59:52.588229+0000","last_became_active":"2024-04-02T04:59:00.080466+0000","last_became_peered":"2024-04-02T04:59:00.080466+0000","last_unstale":"2024-04-02T04:59:52.588229+0000","last_undegraded":"2024-04-02T04:59:52.588229+0000","last_fullsized":"2024-04-02T04:59:52.588229+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:00.559345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:00.559345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:02.335336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1290,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.592756+0000","last_change":"2024-04-02T04:57:36.143862+0000","last_active":"2024-04-02T04:59:52.592756+0000","last_peered":"2024-04-02T04:59:52.592756+0000","last_clean":"2024-04-02T04:59:52.592756+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T04:59:52.592756+0000","last_undegraded":"2024-04-02T04:59:52.592756+0000","last_fullsized":"2024-04-02T04:59:52.592756+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:36.143762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:57:36.143762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"402'25","reported_seq":135,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.126653+0000","last_change":"2024-04-02T04:59:01.058923+0000","last_active":"2024-04-02T04:59:52.126653+0000","last_peered":"2024-04-02T04:59:52.126653+0000","last_clean":"2024-04-02T04:59:52.126653+0000","last_became_active":"2024-04-02T04:59:00.107041+0000","last_became_peered":"2024-04-02T04:59:00.107041+0000","last_unstale":"2024-04-02T04:59:52.126653+0000","last_undegraded":"2024-04-02T04:59:52.126653+0000","last_fullsized":"2024-04-02T04:59:52.126653+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:59:01.058796+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:59:01.058796+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:18.466320+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1320,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123765+0000","last_change":"2024-04-02T04:59:20.068275+0000","last_active":"2024-04-02T04:59:53.123765+0000","last_peered":"2024-04-02T04:59:53.123765+0000","last_clean":"2024-04-02T04:59:53.123765+0000","last_became_active":"2024-04-02T04:59:20.067761+0000","last_became_peered":"2024-04-02T04:59:20.067761+0000","last_unstale":"2024-04-02T04:59:53.123765+0000","last_undegraded":"2024-04-02T04:59:53.123765+0000","last_fullsized":"2024-04-02T04:59:53.123765+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:35.987316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T04:57:35.987316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1263,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.591656+0000","last_change":"2024-04-02T04:54:19.360031+0000","last_active":"2024-04-02T04:59:52.591656+0000","last_peered":"2024-04-02T04:59:52.591656+0000","last_clean":"2024-04-02T04:59:52.591656+0000","last_became_active":"2024-04-02T04:53:54.935778+0000","last_became_peered":"2024-04-02T04:53:54.935778+0000","last_unstale":"2024-04-02T04:59:52.591656+0000","last_undegraded":"2024-04-02T04:59:52.591656+0000","last_fullsized":"2024-04-02T04:59:52.591656+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:19.359786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:19.359786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1159,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.593343+0000","last_change":"2024-04-02T04:59:05.997307+0000","last_active":"2024-04-02T04:59:52.593343+0000","last_peered":"2024-04-02T04:59:52.593343+0000","last_clean":"2024-04-02T04:59:52.593343+0000","last_became_active":"2024-04-02T04:59:00.102255+0000","last_became_peered":"2024-04-02T04:59:00.102255+0000","last_unstale":"2024-04-02T04:59:52.593343+0000","last_undegraded":"2024-04-02T04:59:52.593343+0000","last_fullsized":"2024-04-02T04:59:52.593343+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:05.997205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:05.997205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:30.765941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1279,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.115408+0000","last_change":"2024-04-02T04:54:01.950572+0000","last_active":"2024-04-02T04:59:52.115408+0000","last_peered":"2024-04-02T04:59:52.115408+0000","last_clean":"2024-04-02T04:59:52.115408+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T04:59:52.115408+0000","last_undegraded":"2024-04-02T04:59:52.115408+0000","last_fullsized":"2024-04-02T04:59:52.115408+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:01.950469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:01.950469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809741+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"402'31","reported_seq":887,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.339293+0000","last_change":"2024-04-02T04:59:00.203951+0000","last_active":"2024-04-02T04:59:53.339293+0000","last_peered":"2024-04-02T04:59:53.339293+0000","last_clean":"2024-04-02T04:59:53.339293+0000","last_became_active":"2024-04-02T04:59:00.137536+0000","last_became_peered":"2024-04-02T04:59:00.137536+0000","last_unstale":"2024-04-02T04:59:53.339293+0000","last_undegraded":"2024-04-02T04:59:53.339293+0000","last_fullsized":"2024-04-02T04:59:53.339293+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'31","last_scrub_stamp":"2024-04-02T04:58:57.454000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:57.454000+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:58.362502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1299,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589854+0000","last_change":"2024-04-02T04:58:45.278588+0000","last_active":"2024-04-02T04:59:52.589854+0000","last_peered":"2024-04-02T04:59:52.589854+0000","last_clean":"2024-04-02T04:59:52.589854+0000","last_became_active":"2024-04-02T04:54:55.951772+0000","last_became_peered":"2024-04-02T04:54:55.951772+0000","last_unstale":"2024-04-02T04:59:52.589854+0000","last_undegraded":"2024-04-02T04:59:52.589854+0000","last_fullsized":"2024-04-02T04:59:52.589854+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:45.278356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:58:45.278356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:13.244046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":778,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.339343+0000","last_change":"2024-04-02T04:59:00.134390+0000","last_active":"2024-04-02T04:59:53.339343+0000","last_peered":"2024-04-02T04:59:53.339343+0000","last_clean":"2024-04-02T04:59:53.339343+0000","last_became_active":"2024-04-02T04:59:00.134066+0000","last_became_peered":"2024-04-02T04:59:00.134066+0000","last_unstale":"2024-04-02T04:59:53.339343+0000","last_undegraded":"2024-04-02T04:59:53.339343+0000","last_fullsized":"2024-04-02T04:59:53.339343+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:44.195397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:44.195397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"101'128","reported_seq":1281,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588286+0000","last_change":"2024-04-02T04:59:03.045156+0000","last_active":"2024-04-02T04:59:52.588286+0000","last_peered":"2024-04-02T04:59:52.588286+0000","last_clean":"2024-04-02T04:59:52.588286+0000","last_became_active":"2024-04-02T04:59:03.044786+0000","last_became_peered":"2024-04-02T04:59:03.044786+0000","last_unstale":"2024-04-02T04:59:52.588286+0000","last_undegraded":"2024-04-02T04:59:52.588286+0000","last_fullsized":"2024-04-02T04:59:52.588286+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:57:55.369068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:57:55.369068+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:56.102482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":121,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138683+0000","last_change":"2024-04-02T04:59:20.068040+0000","last_active":"2024-04-02T04:59:52.138683+0000","last_peered":"2024-04-02T04:59:52.138683+0000","last_clean":"2024-04-02T04:59:52.138683+0000","last_became_active":"2024-04-02T04:59:20.067247+0000","last_became_peered":"2024-04-02T04:59:20.067247+0000","last_unstale":"2024-04-02T04:59:52.138683+0000","last_undegraded":"2024-04-02T04:59:52.138683+0000","last_fullsized":"2024-04-02T04:59:52.138683+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:42.254216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:42.254216+0000","last_clean_scrub_stamp":"2024-04-02T04:56:42.254216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"531'52","reported_seq":1416,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.115499+0000","last_change":"2024-04-02T04:58:39.418912+0000","last_active":"2024-04-02T04:59:52.115499+0000","last_peered":"2024-04-02T04:59:52.115499+0000","last_clean":"2024-04-02T04:59:52.115499+0000","last_became_active":"2024-04-02T04:58:39.391176+0000","last_became_peered":"2024-04-02T04:58:39.391176+0000","last_unstale":"2024-04-02T04:59:52.115499+0000","last_undegraded":"2024-04-02T04:59:52.115499+0000","last_fullsized":"2024-04-02T04:59:52.115499+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'52","last_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_deep_scrub":"531'52","last_deep_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_clean_scrub_stamp":"2024-04-02T04:56:30.024650+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810527+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":796,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.336413+0000","last_change":"2024-04-02T04:59:20.063523+0000","last_active":"2024-04-02T04:59:53.336413+0000","last_peered":"2024-04-02T04:59:53.336413+0000","last_clean":"2024-04-02T04:59:53.336413+0000","last_became_active":"2024-04-02T04:59:20.057768+0000","last_became_peered":"2024-04-02T04:59:20.057768+0000","last_unstale":"2024-04-02T04:59:53.336413+0000","last_undegraded":"2024-04-02T04:59:53.336413+0000","last_fullsized":"2024-04-02T04:59:53.336413+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:38.410916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:38.410916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:04.593363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"662'13","reported_seq":252,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.587546+0000","last_change":"2024-04-02T04:59:20.099753+0000","last_active":"2024-04-02T04:59:52.587546+0000","last_peered":"2024-04-02T04:59:52.587546+0000","last_clean":"2024-04-02T04:59:52.587546+0000","last_became_active":"2024-04-02T04:59:20.070441+0000","last_became_peered":"2024-04-02T04:59:20.070441+0000","last_unstale":"2024-04-02T04:59:52.587546+0000","last_undegraded":"2024-04-02T04:59:52.587546+0000","last_fullsized":"2024-04-02T04:59:52.587546+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'12","last_scrub_stamp":"2024-04-02T04:55:34.654306+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:55:34.654306+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.654306+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:56.102385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1301,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589037+0000","last_change":"2024-04-02T04:59:00.101020+0000","last_active":"2024-04-02T04:59:52.589037+0000","last_peered":"2024-04-02T04:59:52.589037+0000","last_clean":"2024-04-02T04:59:52.589037+0000","last_became_active":"2024-04-02T04:59:00.100630+0000","last_became_peered":"2024-04-02T04:59:00.100630+0000","last_unstale":"2024-04-02T04:59:52.589037+0000","last_undegraded":"2024-04-02T04:59:52.589037+0000","last_fullsized":"2024-04-02T04:59:52.589037+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:43.202097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:43.202097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:56.102513+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'128","reported_seq":1063,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.335788+0000","last_change":"2024-04-02T04:59:03.040116+0000","last_active":"2024-04-02T04:59:53.335788+0000","last_peered":"2024-04-02T04:59:53.335788+0000","last_clean":"2024-04-02T04:59:53.335788+0000","last_became_active":"2024-04-02T04:59:03.039047+0000","last_became_peered":"2024-04-02T04:59:03.039047+0000","last_unstale":"2024-04-02T04:59:53.335788+0000","last_undegraded":"2024-04-02T04:59:53.335788+0000","last_fullsized":"2024-04-02T04:59:53.335788+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.263513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.263513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1143,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.120875+0000","last_change":"2024-04-02T04:56:30.499726+0000","last_active":"2024-04-02T04:59:53.120875+0000","last_peered":"2024-04-02T04:59:53.120875+0000","last_clean":"2024-04-02T04:59:53.120875+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T04:59:53.120875+0000","last_undegraded":"2024-04-02T04:59:53.120875+0000","last_fullsized":"2024-04-02T04:59:53.120875+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:30.499514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:30.499514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1291,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859605+0000","last_change":"2024-04-02T04:59:00.139716+0000","last_active":"2024-04-02T04:59:52.859605+0000","last_peered":"2024-04-02T04:59:52.859605+0000","last_clean":"2024-04-02T04:59:52.859605+0000","last_became_active":"2024-04-02T04:59:00.138222+0000","last_became_peered":"2024-04-02T04:59:00.138222+0000","last_unstale":"2024-04-02T04:59:52.859605+0000","last_undegraded":"2024-04-02T04:59:52.859605+0000","last_fullsized":"2024-04-02T04:59:52.859605+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:15.169187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T04:56:15.169187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"402'26","reported_seq":127,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138076+0000","last_change":"2024-04-02T04:59:00.230647+0000","last_active":"2024-04-02T04:59:52.138076+0000","last_peered":"2024-04-02T04:59:52.138076+0000","last_clean":"2024-04-02T04:59:52.138076+0000","last_became_active":"2024-04-02T04:59:00.135793+0000","last_became_peered":"2024-04-02T04:59:00.135793+0000","last_unstale":"2024-04-02T04:59:52.138076+0000","last_undegraded":"2024-04-02T04:59:52.138076+0000","last_fullsized":"2024-04-02T04:59:52.138076+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T04:58:51.009683+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:58:51.009683+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:05.793324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1315,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.121002+0000","last_change":"2024-04-02T04:59:00.106966+0000","last_active":"2024-04-02T04:59:53.121002+0000","last_peered":"2024-04-02T04:59:53.121002+0000","last_clean":"2024-04-02T04:59:53.121002+0000","last_became_active":"2024-04-02T04:59:00.106616+0000","last_became_peered":"2024-04-02T04:59:00.106616+0000","last_unstale":"2024-04-02T04:59:53.121002+0000","last_undegraded":"2024-04-02T04:59:53.121002+0000","last_fullsized":"2024-04-02T04:59:53.121002+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:46.934539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:58:46.934539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:55.094759+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":77,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.124485+0000","last_change":"2024-04-02T04:59:20.084064+0000","last_active":"2024-04-02T04:59:53.124485+0000","last_peered":"2024-04-02T04:59:53.124485+0000","last_clean":"2024-04-02T04:59:53.124485+0000","last_became_active":"2024-04-02T04:59:20.083304+0000","last_became_peered":"2024-04-02T04:59:20.083304+0000","last_unstale":"2024-04-02T04:59:53.124485+0000","last_undegraded":"2024-04-02T04:59:53.124485+0000","last_fullsized":"2024-04-02T04:59:53.124485+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:14.151489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:14.151489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"101'128","reported_seq":1242,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.591317+0000","last_change":"2024-04-02T04:58:44.271422+0000","last_active":"2024-04-02T04:59:52.591317+0000","last_peered":"2024-04-02T04:59:52.591317+0000","last_clean":"2024-04-02T04:59:52.591317+0000","last_became_active":"2024-04-02T04:55:32.987580+0000","last_became_peered":"2024-04-02T04:55:32.987580+0000","last_unstale":"2024-04-02T04:59:52.591317+0000","last_undegraded":"2024-04-02T04:59:52.591317+0000","last_fullsized":"2024-04-02T04:59:52.591317+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:58:44.271259+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:34.875105+0000","last_clean_scrub_stamp":"2024-04-02T04:58:44.271259+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:10.348060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":104,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.336181+0000","last_change":"2024-04-02T04:59:20.087407+0000","last_active":"2024-04-02T04:59:53.336181+0000","last_peered":"2024-04-02T04:59:53.336181+0000","last_clean":"2024-04-02T04:59:53.336181+0000","last_became_active":"2024-04-02T04:59:20.070326+0000","last_became_peered":"2024-04-02T04:59:20.070326+0000","last_unstale":"2024-04-02T04:59:53.336181+0000","last_undegraded":"2024-04-02T04:59:53.336181+0000","last_fullsized":"2024-04-02T04:59:53.336181+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T04:58:58.039256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:58.039256+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:11.393816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"402'48","reported_seq":135,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.137340+0000","last_change":"2024-04-02T04:59:00.209092+0000","last_active":"2024-04-02T04:59:52.137340+0000","last_peered":"2024-04-02T04:59:52.137340+0000","last_clean":"2024-04-02T04:59:52.137340+0000","last_became_active":"2024-04-02T04:59:00.122446+0000","last_became_peered":"2024-04-02T04:59:00.122446+0000","last_unstale":"2024-04-02T04:59:52.137340+0000","last_undegraded":"2024-04-02T04:59:52.137340+0000","last_fullsized":"2024-04-02T04:59:52.137340+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'48","last_scrub_stamp":"2024-04-02T04:58:48.877784+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:58:48.877784+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:02.831929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1312,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.122601+0000","last_change":"2024-04-02T04:59:00.108374+0000","last_active":"2024-04-02T04:59:53.122601+0000","last_peered":"2024-04-02T04:59:53.122601+0000","last_clean":"2024-04-02T04:59:53.122601+0000","last_became_active":"2024-04-02T04:59:00.107474+0000","last_became_peered":"2024-04-02T04:59:00.107474+0000","last_unstale":"2024-04-02T04:59:53.122601+0000","last_undegraded":"2024-04-02T04:59:53.122601+0000","last_fullsized":"2024-04-02T04:59:53.122601+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:26.486667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:26.486667+0000","last_clean_scrub_stamp":"2024-04-02T04:55:26.486667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1246,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589123+0000","last_change":"2024-04-02T04:59:10.460466+0000","last_active":"2024-04-02T04:59:52.589123+0000","last_peered":"2024-04-02T04:59:52.589123+0000","last_clean":"2024-04-02T04:59:52.589123+0000","last_became_active":"2024-04-02T04:59:00.111699+0000","last_became_peered":"2024-04-02T04:59:00.111699+0000","last_unstale":"2024-04-02T04:59:52.589123+0000","last_undegraded":"2024-04-02T04:59:52.589123+0000","last_fullsized":"2024-04-02T04:59:52.589123+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:10.460372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:10.460372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:24.278025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":512,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123893+0000","last_change":"2024-04-02T04:57:46.972993+0000","last_active":"2024-04-02T04:59:53.123893+0000","last_peered":"2024-04-02T04:59:53.123893+0000","last_clean":"2024-04-02T04:59:53.123893+0000","last_became_active":"2024-04-02T04:54:55.956870+0000","last_became_peered":"2024-04-02T04:54:55.956870+0000","last_unstale":"2024-04-02T04:59:53.123893+0000","last_undegraded":"2024-04-02T04:59:53.123893+0000","last_fullsized":"2024-04-02T04:59:53.123893+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:46.972794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:57:46.972794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1140,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.109454+0000","last_change":"2024-04-02T04:59:20.039821+0000","last_active":"2024-04-02T04:59:52.109454+0000","last_peered":"2024-04-02T04:59:52.109454+0000","last_clean":"2024-04-02T04:59:52.109454+0000","last_became_active":"2024-04-02T04:59:20.037443+0000","last_became_peered":"2024-04-02T04:59:20.037443+0000","last_unstale":"2024-04-02T04:59:52.109454+0000","last_undegraded":"2024-04-02T04:59:52.109454+0000","last_fullsized":"2024-04-02T04:59:52.109454+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:32.703374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:57:32.703374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"402'30","reported_seq":833,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.590515+0000","last_change":"2024-04-02T04:59:20.101270+0000","last_active":"2024-04-02T04:59:52.590515+0000","last_peered":"2024-04-02T04:59:52.590515+0000","last_clean":"2024-04-02T04:59:52.590515+0000","last_became_active":"2024-04-02T04:59:20.065524+0000","last_became_peered":"2024-04-02T04:59:20.065524+0000","last_unstale":"2024-04-02T04:59:52.590515+0000","last_undegraded":"2024-04-02T04:59:52.590515+0000","last_fullsized":"2024-04-02T04:59:52.590515+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:59:09.140586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.140586+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:37.757375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":780,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337912+0000","last_change":"2024-04-02T04:58:47.478098+0000","last_active":"2024-04-02T04:59:53.337912+0000","last_peered":"2024-04-02T04:59:53.337912+0000","last_clean":"2024-04-02T04:59:53.337912+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T04:59:53.337912+0000","last_undegraded":"2024-04-02T04:59:53.337912+0000","last_fullsized":"2024-04-02T04:59:53.337912+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:47.477862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T04:58:47.477862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.616788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"642'13","reported_seq":1502,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.860881+0000","last_change":"2024-04-02T04:59:00.191481+0000","last_active":"2024-04-02T04:59:52.860881+0000","last_peered":"2024-04-02T04:59:52.860881+0000","last_clean":"2024-04-02T04:59:52.860881+0000","last_became_active":"2024-04-02T04:59:00.136703+0000","last_became_peered":"2024-04-02T04:59:00.136703+0000","last_unstale":"2024-04-02T04:59:52.860881+0000","last_undegraded":"2024-04-02T04:59:52.860881+0000","last_fullsized":"2024-04-02T04:59:52.860881+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:55:36.612122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:36.612122+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990005+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.126474+0000","last_change":"2024-04-02T04:59:00.133941+0000","last_active":"2024-04-02T04:59:52.126474+0000","last_peered":"2024-04-02T04:59:52.126474+0000","last_clean":"2024-04-02T04:59:52.126474+0000","last_became_active":"2024-04-02T04:59:00.125089+0000","last_became_peered":"2024-04-02T04:59:00.125089+0000","last_unstale":"2024-04-02T04:59:52.126474+0000","last_undegraded":"2024-04-02T04:59:52.126474+0000","last_fullsized":"2024-04-02T04:59:52.126474+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:56.418322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:56.418322+0000","last_clean_scrub_stamp":"2024-04-02T04:58:56.418322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:12.377867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'128","reported_seq":723,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.126272+0000","last_change":"2024-04-02T04:59:03.987730+0000","last_active":"2024-04-02T04:59:52.126272+0000","last_peered":"2024-04-02T04:59:52.126272+0000","last_clean":"2024-04-02T04:59:52.126272+0000","last_became_active":"2024-04-02T04:59:03.050648+0000","last_became_peered":"2024-04-02T04:59:03.050648+0000","last_unstale":"2024-04-02T04:59:52.126272+0000","last_undegraded":"2024-04-02T04:59:52.126272+0000","last_fullsized":"2024-04-02T04:59:52.126272+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:59:03.987621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:59:03.987621+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:31.584277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1145,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.116838+0000","last_change":"2024-04-02T04:59:00.106527+0000","last_active":"2024-04-02T04:59:52.116838+0000","last_peered":"2024-04-02T04:59:52.116838+0000","last_clean":"2024-04-02T04:59:52.116838+0000","last_became_active":"2024-04-02T04:59:00.105786+0000","last_became_peered":"2024-04-02T04:59:00.105786+0000","last_unstale":"2024-04-02T04:59:52.116838+0000","last_undegraded":"2024-04-02T04:59:52.116838+0000","last_fullsized":"2024-04-02T04:59:52.116838+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.941615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809926+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":849,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.109048+0000","last_change":"2024-04-02T04:59:20.079551+0000","last_active":"2024-04-02T04:59:52.109048+0000","last_peered":"2024-04-02T04:59:52.109048+0000","last_clean":"2024-04-02T04:59:52.109048+0000","last_became_active":"2024-04-02T04:59:20.074938+0000","last_became_peered":"2024-04-02T04:59:20.074938+0000","last_unstale":"2024-04-02T04:59:52.109048+0000","last_undegraded":"2024-04-02T04:59:52.109048+0000","last_fullsized":"2024-04-02T04:59:52.109048+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:25.081828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:56:25.081828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"402'30","reported_seq":1328,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.593791+0000","last_change":"2024-04-02T04:58:54.938309+0000","last_active":"2024-04-02T04:59:52.593791+0000","last_peered":"2024-04-02T04:59:52.593791+0000","last_clean":"2024-04-02T04:59:52.593791+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T04:59:52.593791+0000","last_undegraded":"2024-04-02T04:59:52.593791+0000","last_fullsized":"2024-04-02T04:59:52.593791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:58:54.937951+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:58:54.937951+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:06.808291+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1287,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.116960+0000","last_change":"2024-04-02T04:54:08.978978+0000","last_active":"2024-04-02T04:59:52.116960+0000","last_peered":"2024-04-02T04:59:52.116960+0000","last_clean":"2024-04-02T04:59:52.116960+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T04:59:52.116960+0000","last_undegraded":"2024-04-02T04:59:52.116960+0000","last_fullsized":"2024-04-02T04:59:52.116960+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:08.978775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:54:08.978775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":172,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.137559+0000","last_change":"2024-04-02T04:59:00.198481+0000","last_active":"2024-04-02T04:59:52.137559+0000","last_peered":"2024-04-02T04:59:52.137559+0000","last_clean":"2024-04-02T04:59:52.137559+0000","last_became_active":"2024-04-02T04:59:00.119844+0000","last_became_peered":"2024-04-02T04:59:00.119844+0000","last_unstale":"2024-04-02T04:59:52.137559+0000","last_undegraded":"2024-04-02T04:59:52.137559+0000","last_fullsized":"2024-04-02T04:59:52.137559+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:55:13.047925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:13.047925+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":733,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337463+0000","last_change":"2024-04-02T04:57:45.227053+0000","last_active":"2024-04-02T04:59:53.337463+0000","last_peered":"2024-04-02T04:59:53.337463+0000","last_clean":"2024-04-02T04:59:53.337463+0000","last_became_active":"2024-04-02T04:53:54.936665+0000","last_became_peered":"2024-04-02T04:53:54.936665+0000","last_unstale":"2024-04-02T04:59:53.337463+0000","last_undegraded":"2024-04-02T04:59:53.337463+0000","last_fullsized":"2024-04-02T04:59:53.337463+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:45.226951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:57:45.226951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'128","reported_seq":877,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.587881+0000","last_change":"2024-04-02T04:59:23.075974+0000","last_active":"2024-04-02T04:59:52.587881+0000","last_peered":"2024-04-02T04:59:52.587881+0000","last_clean":"2024-04-02T04:59:52.587881+0000","last_became_active":"2024-04-02T04:59:23.074106+0000","last_became_peered":"2024-04-02T04:59:23.074106+0000","last_unstale":"2024-04-02T04:59:52.587881+0000","last_undegraded":"2024-04-02T04:59:52.587881+0000","last_fullsized":"2024-04-02T04:59:52.587881+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":665,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.603255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.603255+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:57.075974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1143,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.860366+0000","last_change":"2024-04-02T04:59:15.184662+0000","last_active":"2024-04-02T04:59:52.860366+0000","last_peered":"2024-04-02T04:59:52.860366+0000","last_clean":"2024-04-02T04:59:52.860366+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T04:59:52.860366+0000","last_undegraded":"2024-04-02T04:59:52.860366+0000","last_fullsized":"2024-04-02T04:59:52.860366+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:15.184457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:15.184457+0000","last_clean_scrub_stamp":"2024-04-02T04:59:15.184457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:43.523241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"642'16","reported_seq":130,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.137624+0000","last_change":"2024-04-02T04:59:00.221695+0000","last_active":"2024-04-02T04:59:52.137624+0000","last_peered":"2024-04-02T04:59:52.137624+0000","last_clean":"2024-04-02T04:59:52.137624+0000","last_became_active":"2024-04-02T04:59:00.134138+0000","last_became_peered":"2024-04-02T04:59:00.134138+0000","last_unstale":"2024-04-02T04:59:52.137624+0000","last_undegraded":"2024-04-02T04:59:52.137624+0000","last_fullsized":"2024-04-02T04:59:52.137624+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'15","last_scrub_stamp":"2024-04-02T04:56:30.036674+0000","last_deep_scrub":"516'15","last_deep_scrub_stamp":"2024-04-02T04:56:30.036674+0000","last_clean_scrub_stamp":"2024-04-02T04:56:30.036674+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"402'23","reported_seq":137,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.127923+0000","last_change":"2024-04-02T04:59:05.949973+0000","last_active":"2024-04-02T04:59:52.127923+0000","last_peered":"2024-04-02T04:59:52.127923+0000","last_clean":"2024-04-02T04:59:52.127923+0000","last_became_active":"2024-04-02T04:59:00.134091+0000","last_became_peered":"2024-04-02T04:59:00.134091+0000","last_unstale":"2024-04-02T04:59:52.127923+0000","last_undegraded":"2024-04-02T04:59:52.127923+0000","last_fullsized":"2024-04-02T04:59:52.127923+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'23","last_scrub_stamp":"2024-04-02T04:59:05.949860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:05.949860+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:22.771600+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":1304,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.124688+0000","last_change":"2024-04-02T04:59:09.939926+0000","last_active":"2024-04-02T04:59:53.124688+0000","last_peered":"2024-04-02T04:59:53.124688+0000","last_clean":"2024-04-02T04:59:53.124688+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T04:59:53.124688+0000","last_undegraded":"2024-04-02T04:59:53.124688+0000","last_fullsized":"2024-04-02T04:59:53.124688+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:59:09.939787+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.939787+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:37.387693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1307,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.858707+0000","last_change":"2024-04-02T04:59:10.152080+0000","last_active":"2024-04-02T04:59:52.858707+0000","last_peered":"2024-04-02T04:59:52.858707+0000","last_clean":"2024-04-02T04:59:52.858707+0000","last_became_active":"2024-04-02T04:59:00.135841+0000","last_became_peered":"2024-04-02T04:59:00.135841+0000","last_unstale":"2024-04-02T04:59:52.858707+0000","last_undegraded":"2024-04-02T04:59:52.858707+0000","last_fullsized":"2024-04-02T04:59:52.858707+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:10.151703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:10.151703+0000","last_clean_scrub_stamp":"2024-04-02T04:59:10.151703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:23.573648+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1250,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.127168+0000","last_change":"2024-04-02T04:59:20.038467+0000","last_active":"2024-04-02T04:59:52.127168+0000","last_peered":"2024-04-02T04:59:52.127168+0000","last_clean":"2024-04-02T04:59:52.127168+0000","last_became_active":"2024-04-02T04:59:20.037960+0000","last_became_peered":"2024-04-02T04:59:20.037960+0000","last_unstale":"2024-04-02T04:59:52.127168+0000","last_undegraded":"2024-04-02T04:59:52.127168+0000","last_fullsized":"2024-04-02T04:59:52.127168+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:16.123504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.123504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"101'128","reported_seq":772,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.116472+0000","last_change":"2024-04-02T04:59:03.045870+0000","last_active":"2024-04-02T04:59:52.116472+0000","last_peered":"2024-04-02T04:59:52.116472+0000","last_clean":"2024-04-02T04:59:52.116472+0000","last_became_active":"2024-04-02T04:59:03.045003+0000","last_became_peered":"2024-04-02T04:59:03.045003+0000","last_unstale":"2024-04-02T04:59:52.116472+0000","last_undegraded":"2024-04-02T04:59:52.116472+0000","last_fullsized":"2024-04-02T04:59:52.116472+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.026882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.026882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1133,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.112663+0000","last_change":"2024-04-02T04:54:26.868758+0000","last_active":"2024-04-02T04:59:52.112663+0000","last_peered":"2024-04-02T04:59:52.112663+0000","last_clean":"2024-04-02T04:59:52.112663+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T04:59:52.112663+0000","last_undegraded":"2024-04-02T04:59:52.112663+0000","last_fullsized":"2024-04-02T04:59:52.112663+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.868528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.868528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"451'15","reported_seq":1498,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588956+0000","last_change":"2024-04-02T04:55:27.022983+0000","last_active":"2024-04-02T04:59:52.588956+0000","last_peered":"2024-04-02T04:59:52.588956+0000","last_clean":"2024-04-02T04:59:52.588956+0000","last_became_active":"2024-04-02T04:54:55.961735+0000","last_became_peered":"2024-04-02T04:54:55.961735+0000","last_unstale":"2024-04-02T04:59:52.588956+0000","last_undegraded":"2024-04-02T04:59:52.588956+0000","last_fullsized":"2024-04-02T04:59:52.588956+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'15","last_scrub_stamp":"2024-04-02T04:55:27.022873+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T04:55:27.022873+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"451'60","reported_seq":1611,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123399+0000","last_change":"2024-04-02T04:57:33.971003+0000","last_active":"2024-04-02T04:59:53.123399+0000","last_peered":"2024-04-02T04:59:53.123399+0000","last_clean":"2024-04-02T04:59:53.123399+0000","last_became_active":"2024-04-02T04:54:55.955778+0000","last_became_peered":"2024-04-02T04:54:55.955778+0000","last_unstale":"2024-04-02T04:59:53.123399+0000","last_undegraded":"2024-04-02T04:59:53.123399+0000","last_fullsized":"2024-04-02T04:59:53.123399+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'60","last_scrub_stamp":"2024-04-02T04:57:33.970725+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:57:33.970725+0000","objects_scrubbed":11,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":768,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337852+0000","last_change":"2024-04-02T04:58:55.433444+0000","last_active":"2024-04-02T04:59:53.337852+0000","last_peered":"2024-04-02T04:59:53.337852+0000","last_clean":"2024-04-02T04:59:53.337852+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T04:59:53.337852+0000","last_undegraded":"2024-04-02T04:59:53.337852+0000","last_fullsized":"2024-04-02T04:59:53.337852+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:55.433242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:55.433242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:58.540801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":1297,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.108677+0000","last_change":"2024-04-02T04:56:27.048142+0000","last_active":"2024-04-02T04:59:52.108677+0000","last_peered":"2024-04-02T04:59:52.108677+0000","last_clean":"2024-04-02T04:59:52.108677+0000","last_became_active":"2024-04-02T04:54:56.004769+0000","last_became_peered":"2024-04-02T04:54:56.004769+0000","last_unstale":"2024-04-02T04:59:52.108677+0000","last_undegraded":"2024-04-02T04:59:52.108677+0000","last_fullsized":"2024-04-02T04:59:52.108677+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:56:27.047860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:27.047860+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1282,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.122785+0000","last_change":"2024-04-02T04:58:45.937050+0000","last_active":"2024-04-02T04:59:53.122785+0000","last_peered":"2024-04-02T04:59:53.122785+0000","last_clean":"2024-04-02T04:59:53.122785+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T04:59:53.122785+0000","last_undegraded":"2024-04-02T04:59:53.122785+0000","last_fullsized":"2024-04-02T04:59:53.122785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:45.936953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:58:45.936953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:57.424927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'128","reported_seq":1258,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.121257+0000","last_change":"2024-04-02T04:59:23.087044+0000","last_active":"2024-04-02T04:59:53.121257+0000","last_peered":"2024-04-02T04:59:53.121257+0000","last_clean":"2024-04-02T04:59:53.121257+0000","last_became_active":"2024-04-02T04:59:23.086202+0000","last_became_peered":"2024-04-02T04:59:23.086202+0000","last_unstale":"2024-04-02T04:59:53.121257+0000","last_undegraded":"2024-04-02T04:59:53.121257+0000","last_fullsized":"2024-04-02T04:59:53.121257+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":665,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.448768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.448768+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1147,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.590420+0000","last_change":"2024-04-02T04:57:53.023737+0000","last_active":"2024-04-02T04:59:52.590420+0000","last_peered":"2024-04-02T04:59:52.590420+0000","last_clean":"2024-04-02T04:59:52.590420+0000","last_became_active":"2024-04-02T04:53:54.957413+0000","last_became_peered":"2024-04-02T04:53:54.957413+0000","last_unstale":"2024-04-02T04:59:52.590420+0000","last_undegraded":"2024-04-02T04:59:52.590420+0000","last_fullsized":"2024-04-02T04:59:52.590420+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:53.023662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:57:53.023662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742466+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1294,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589387+0000","last_change":"2024-04-02T04:56:39.050317+0000","last_active":"2024-04-02T04:59:52.589387+0000","last_peered":"2024-04-02T04:59:52.589387+0000","last_clean":"2024-04-02T04:59:52.589387+0000","last_became_active":"2024-04-02T04:54:55.950537+0000","last_became_peered":"2024-04-02T04:54:55.950537+0000","last_unstale":"2024-04-02T04:59:52.589387+0000","last_undegraded":"2024-04-02T04:59:52.589387+0000","last_fullsized":"2024-04-02T04:59:52.589387+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:39.050034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:56:39.050034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":767,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.339271+0000","last_change":"2024-04-02T04:59:04.490663+0000","last_active":"2024-04-02T04:59:53.339271+0000","last_peered":"2024-04-02T04:59:53.339271+0000","last_clean":"2024-04-02T04:59:53.339271+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T04:59:53.339271+0000","last_undegraded":"2024-04-02T04:59:53.339271+0000","last_fullsized":"2024-04-02T04:59:53.339271+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:04.490513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:59:04.490513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:26.491224+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"402'34","reported_seq":1412,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.587897+0000","last_change":"2024-04-02T04:59:20.119285+0000","last_active":"2024-04-02T04:59:52.587897+0000","last_peered":"2024-04-02T04:59:52.587897+0000","last_clean":"2024-04-02T04:59:52.587897+0000","last_became_active":"2024-04-02T04:59:20.079497+0000","last_became_peered":"2024-04-02T04:59:20.079497+0000","last_unstale":"2024-04-02T04:59:52.587897+0000","last_undegraded":"2024-04-02T04:59:52.587897+0000","last_fullsized":"2024-04-02T04:59:52.587897+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'34","last_scrub_stamp":"2024-04-02T04:57:30.220239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:30.220239+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742384+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.126566+0000","last_change":"2024-04-02T04:59:00.121219+0000","last_active":"2024-04-02T04:59:52.126566+0000","last_peered":"2024-04-02T04:59:52.126566+0000","last_clean":"2024-04-02T04:59:52.126566+0000","last_became_active":"2024-04-02T04:59:00.119618+0000","last_became_peered":"2024-04-02T04:59:00.119618+0000","last_unstale":"2024-04-02T04:59:52.126566+0000","last_undegraded":"2024-04-02T04:59:52.126566+0000","last_fullsized":"2024-04-02T04:59:52.126566+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:21.446624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:21.446624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1133,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.110768+0000","last_change":"2024-04-02T04:54:20.899430+0000","last_active":"2024-04-02T04:59:52.110768+0000","last_peered":"2024-04-02T04:59:52.110768+0000","last_clean":"2024-04-02T04:59:52.110768+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T04:59:52.110768+0000","last_undegraded":"2024-04-02T04:59:52.110768+0000","last_fullsized":"2024-04-02T04:59:52.110768+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:20.898842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:20.898842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809789+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1285,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.108795+0000","last_change":"2024-04-02T04:59:20.070969+0000","last_active":"2024-04-02T04:59:52.108795+0000","last_peered":"2024-04-02T04:59:52.108795+0000","last_clean":"2024-04-02T04:59:52.108795+0000","last_became_active":"2024-04-02T04:59:20.070708+0000","last_became_peered":"2024-04-02T04:59:20.070708+0000","last_unstale":"2024-04-02T04:59:52.108795+0000","last_undegraded":"2024-04-02T04:59:52.108795+0000","last_fullsized":"2024-04-02T04:59:52.108795+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:00.909025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:00.909025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":1038,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.338949+0000","last_change":"2024-04-02T04:55:29.149388+0000","last_active":"2024-04-02T04:59:53.338949+0000","last_peered":"2024-04-02T04:59:53.338949+0000","last_clean":"2024-04-02T04:59:53.338949+0000","last_became_active":"2024-04-02T04:55:27.978786+0000","last_became_peered":"2024-04-02T04:55:27.978786+0000","last_unstale":"2024-04-02T04:59:53.338949+0000","last_undegraded":"2024-04-02T04:59:53.338949+0000","last_fullsized":"2024-04-02T04:59:53.338949+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:29.149138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:29.149138+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1196,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588119+0000","last_change":"2024-04-02T04:59:00.082727+0000","last_active":"2024-04-02T04:59:52.588119+0000","last_peered":"2024-04-02T04:59:52.588119+0000","last_clean":"2024-04-02T04:59:52.588119+0000","last_became_active":"2024-04-02T04:59:00.082361+0000","last_became_peered":"2024-04-02T04:59:00.082361+0000","last_unstale":"2024-04-02T04:59:52.588119+0000","last_undegraded":"2024-04-02T04:59:52.588119+0000","last_fullsized":"2024-04-02T04:59:52.588119+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:50.380070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:58:50.380070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:10.633803+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"402'26","reported_seq":129,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.127556+0000","last_change":"2024-04-02T04:59:00.160834+0000","last_active":"2024-04-02T04:59:52.127556+0000","last_peered":"2024-04-02T04:59:52.127556+0000","last_clean":"2024-04-02T04:59:52.127556+0000","last_became_active":"2024-04-02T04:59:00.118923+0000","last_became_peered":"2024-04-02T04:59:00.118923+0000","last_unstale":"2024-04-02T04:59:52.127556+0000","last_undegraded":"2024-04-02T04:59:52.127556+0000","last_fullsized":"2024-04-02T04:59:52.127556+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T04:56:18.130852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.130852+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.128178+0000","last_change":"2024-04-02T04:59:00.129115+0000","last_active":"2024-04-02T04:59:52.128178+0000","last_peered":"2024-04-02T04:59:52.128178+0000","last_clean":"2024-04-02T04:59:52.128178+0000","last_became_active":"2024-04-02T04:59:00.126748+0000","last_became_peered":"2024-04-02T04:59:00.126748+0000","last_unstale":"2024-04-02T04:59:52.128178+0000","last_undegraded":"2024-04-02T04:59:52.128178+0000","last_fullsized":"2024-04-02T04:59:52.128178+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:35.117776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T04:55:35.117776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"101'128","reported_seq":1054,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337035+0000","last_change":"2024-04-02T04:59:23.081517+0000","last_active":"2024-04-02T04:59:53.337035+0000","last_peered":"2024-04-02T04:59:53.337035+0000","last_clean":"2024-04-02T04:59:53.337035+0000","last_became_active":"2024-04-02T04:59:23.081091+0000","last_became_peered":"2024-04-02T04:59:23.081091+0000","last_unstale":"2024-04-02T04:59:53.337035+0000","last_undegraded":"2024-04-02T04:59:53.337035+0000","last_fullsized":"2024-04-02T04:59:53.337035+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":665,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:34.217242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.217242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1034,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.590572+0000","last_change":"2024-04-02T04:57:42.154782+0000","last_active":"2024-04-02T04:59:52.590572+0000","last_peered":"2024-04-02T04:59:52.590572+0000","last_clean":"2024-04-02T04:59:52.590572+0000","last_became_active":"2024-04-02T04:54:55.946927+0000","last_became_peered":"2024-04-02T04:54:55.946927+0000","last_unstale":"2024-04-02T04:59:52.590572+0000","last_undegraded":"2024-04-02T04:59:52.590572+0000","last_fullsized":"2024-04-02T04:59:52.590572+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:42.154585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:57:42.154585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:56.102417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1032,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.595184+0000","last_change":"2024-04-02T04:59:00.138446+0000","last_active":"2024-04-02T04:59:52.595184+0000","last_peered":"2024-04-02T04:59:52.595184+0000","last_clean":"2024-04-02T04:59:52.595184+0000","last_became_active":"2024-04-02T04:59:00.138221+0000","last_became_peered":"2024-04-02T04:59:00.138221+0000","last_unstale":"2024-04-02T04:59:52.595184+0000","last_undegraded":"2024-04-02T04:59:52.595184+0000","last_fullsized":"2024-04-02T04:59:52.595184+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:32.966986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:57:32.966986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1313,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.590621+0000","last_change":"2024-04-02T04:59:00.082692+0000","last_active":"2024-04-02T04:59:52.590621+0000","last_peered":"2024-04-02T04:59:52.590621+0000","last_clean":"2024-04-02T04:59:52.590621+0000","last_became_active":"2024-04-02T04:59:00.081588+0000","last_became_peered":"2024-04-02T04:59:00.081588+0000","last_unstale":"2024-04-02T04:59:52.590621+0000","last_undegraded":"2024-04-02T04:59:52.590621+0000","last_fullsized":"2024-04-02T04:59:52.590621+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:38.267128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:58:38.267128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:07.027534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"402'42","reported_seq":1364,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.121921+0000","last_change":"2024-04-02T04:56:26.536234+0000","last_active":"2024-04-02T04:59:53.121921+0000","last_peered":"2024-04-02T04:59:53.121921+0000","last_clean":"2024-04-02T04:59:53.121921+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T04:59:53.121921+0000","last_undegraded":"2024-04-02T04:59:53.121921+0000","last_fullsized":"2024-04-02T04:59:53.121921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'42","last_scrub_stamp":"2024-04-02T04:56:26.536132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:26.536132+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":751,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.338868+0000","last_change":"2024-04-02T04:59:00.138177+0000","last_active":"2024-04-02T04:59:53.338868+0000","last_peered":"2024-04-02T04:59:53.338868+0000","last_clean":"2024-04-02T04:59:53.338868+0000","last_became_active":"2024-04-02T04:59:00.137237+0000","last_became_peered":"2024-04-02T04:59:00.137237+0000","last_unstale":"2024-04-02T04:59:53.338868+0000","last_undegraded":"2024-04-02T04:59:53.338868+0000","last_fullsized":"2024-04-02T04:59:53.338868+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:35.112504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:35.112504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'128","reported_seq":1243,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.591162+0000","last_change":"2024-04-02T04:55:33.954742+0000","last_active":"2024-04-02T04:59:52.591162+0000","last_peered":"2024-04-02T04:59:52.591162+0000","last_clean":"2024-04-02T04:59:52.591162+0000","last_became_active":"2024-04-02T04:55:32.984097+0000","last_became_peered":"2024-04-02T04:55:32.984097+0000","last_unstale":"2024-04-02T04:59:52.591162+0000","last_undegraded":"2024-04-02T04:59:52.591162+0000","last_fullsized":"2024-04-02T04:59:52.591162+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.954638+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:28.185555+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.954638+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1139,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.110584+0000","last_change":"2024-04-02T04:54:56.005135+0000","last_active":"2024-04-02T04:59:52.110584+0000","last_peered":"2024-04-02T04:59:52.110584+0000","last_clean":"2024-04-02T04:59:52.110584+0000","last_became_active":"2024-04-02T04:54:56.004256+0000","last_became_peered":"2024-04-02T04:54:56.004256+0000","last_unstale":"2024-04-02T04:59:52.110584+0000","last_undegraded":"2024-04-02T04:59:52.110584+0000","last_fullsized":"2024-04-02T04:59:52.110584+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.854283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1304,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123154+0000","last_change":"2024-04-02T04:59:00.109813+0000","last_active":"2024-04-02T04:59:53.123154+0000","last_peered":"2024-04-02T04:59:53.123154+0000","last_clean":"2024-04-02T04:59:53.123154+0000","last_became_active":"2024-04-02T04:59:00.109471+0000","last_became_peered":"2024-04-02T04:59:00.109471+0000","last_unstale":"2024-04-02T04:59:53.123154+0000","last_undegraded":"2024-04-02T04:59:53.123154+0000","last_fullsized":"2024-04-02T04:59:53.123154+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:15.453139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T04:56:15.453139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'35","reported_seq":1346,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859638+0000","last_change":"2024-04-02T04:55:27.706686+0000","last_active":"2024-04-02T04:59:52.859638+0000","last_peered":"2024-04-02T04:59:52.859638+0000","last_clean":"2024-04-02T04:59:52.859638+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T04:59:52.859638+0000","last_undegraded":"2024-04-02T04:59:52.859638+0000","last_fullsized":"2024-04-02T04:59:52.859638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T04:55:27.706548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:27.706548+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.989899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":513,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.120793+0000","last_change":"2024-04-02T04:57:56.006438+0000","last_active":"2024-04-02T04:59:53.120793+0000","last_peered":"2024-04-02T04:59:53.120793+0000","last_clean":"2024-04-02T04:59:53.120793+0000","last_became_active":"2024-04-02T04:54:55.969352+0000","last_became_peered":"2024-04-02T04:54:55.969352+0000","last_unstale":"2024-04-02T04:59:53.120793+0000","last_undegraded":"2024-04-02T04:59:53.120793+0000","last_fullsized":"2024-04-02T04:59:53.120793+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:56.006224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:56.006224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":127,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.129776+0000","last_change":"2024-04-02T04:59:04.968028+0000","last_active":"2024-04-02T04:59:52.129776+0000","last_peered":"2024-04-02T04:59:52.129776+0000","last_clean":"2024-04-02T04:59:52.129776+0000","last_became_active":"2024-04-02T04:59:00.128819+0000","last_became_peered":"2024-04-02T04:59:00.128819+0000","last_unstale":"2024-04-02T04:59:52.129776+0000","last_undegraded":"2024-04-02T04:59:52.129776+0000","last_fullsized":"2024-04-02T04:59:52.129776+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_clean_scrub_stamp":"2024-04-02T04:59:04.967881+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:16.397539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'128","reported_seq":790,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.115832+0000","last_change":"2024-04-02T04:59:03.050796+0000","last_active":"2024-04-02T04:59:52.115832+0000","last_peered":"2024-04-02T04:59:52.115832+0000","last_clean":"2024-04-02T04:59:52.115832+0000","last_became_active":"2024-04-02T04:59:03.048483+0000","last_became_peered":"2024-04-02T04:59:03.048483+0000","last_unstale":"2024-04-02T04:59:52.115832+0000","last_undegraded":"2024-04-02T04:59:52.115832+0000","last_fullsized":"2024-04-02T04:59:52.115832+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.982932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.982932+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":629,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.336975+0000","last_change":"2024-04-02T04:55:19.317421+0000","last_active":"2024-04-02T04:59:53.336975+0000","last_peered":"2024-04-02T04:59:53.336975+0000","last_clean":"2024-04-02T04:59:53.336975+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T04:59:53.336975+0000","last_undegraded":"2024-04-02T04:59:53.336975+0000","last_fullsized":"2024-04-02T04:59:53.336975+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:19.317110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:19.317110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":859,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588491+0000","last_change":"2024-04-02T04:59:20.058465+0000","last_active":"2024-04-02T04:59:52.588491+0000","last_peered":"2024-04-02T04:59:52.588491+0000","last_clean":"2024-04-02T04:59:52.588491+0000","last_became_active":"2024-04-02T04:59:20.057847+0000","last_became_peered":"2024-04-02T04:59:20.057847+0000","last_unstale":"2024-04-02T04:59:52.588491+0000","last_undegraded":"2024-04-02T04:59:52.588491+0000","last_fullsized":"2024-04-02T04:59:52.588491+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:43.553059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:43.553059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"642'31","reported_seq":144,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.136565+0000","last_change":"2024-04-02T04:59:00.171687+0000","last_active":"2024-04-02T04:59:52.136565+0000","last_peered":"2024-04-02T04:59:52.136565+0000","last_clean":"2024-04-02T04:59:52.136565+0000","last_became_active":"2024-04-02T04:59:00.112450+0000","last_became_peered":"2024-04-02T04:59:00.112450+0000","last_unstale":"2024-04-02T04:59:52.136565+0000","last_undegraded":"2024-04-02T04:59:52.136565+0000","last_fullsized":"2024-04-02T04:59:52.136565+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-04-02T04:58:46.273857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:46.273857+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:55.723346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":1311,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123242+0000","last_change":"2024-04-02T04:58:47.888437+0000","last_active":"2024-04-02T04:59:53.123242+0000","last_peered":"2024-04-02T04:59:53.123242+0000","last_clean":"2024-04-02T04:59:53.123242+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T04:59:53.123242+0000","last_undegraded":"2024-04-02T04:59:53.123242+0000","last_fullsized":"2024-04-02T04:59:53.123242+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:58:47.888333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:47.888333+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:11.940130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1298,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.590095+0000","last_change":"2024-04-02T04:59:00.105845+0000","last_active":"2024-04-02T04:59:52.590095+0000","last_peered":"2024-04-02T04:59:52.590095+0000","last_clean":"2024-04-02T04:59:52.590095+0000","last_became_active":"2024-04-02T04:59:00.103740+0000","last_became_peered":"2024-04-02T04:59:00.103740+0000","last_unstale":"2024-04-02T04:59:52.590095+0000","last_undegraded":"2024-04-02T04:59:52.590095+0000","last_fullsized":"2024-04-02T04:59:52.590095+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:51.989387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:58:51.989387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:10.044812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1165,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.869103+0000","last_change":"2024-04-02T04:59:52.869208+0000","last_active":"2024-04-02T04:59:52.869103+0000","last_peered":"2024-04-02T04:59:52.869103+0000","last_clean":"2024-04-02T04:59:52.869103+0000","last_became_active":"2024-04-02T04:59:00.105159+0000","last_became_peered":"2024-04-02T04:59:00.105159+0000","last_unstale":"2024-04-02T04:59:52.869103+0000","last_undegraded":"2024-04-02T04:59:52.869103+0000","last_fullsized":"2024-04-02T04:59:52.869103+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:52.868879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:52.868879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:59.251842+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"642'14","reported_seq":1510,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.858954+0000","last_change":"2024-04-02T04:59:00.204978+0000","last_active":"2024-04-02T04:59:52.858954+0000","last_peered":"2024-04-02T04:59:52.858954+0000","last_clean":"2024-04-02T04:59:52.858954+0000","last_became_active":"2024-04-02T04:59:00.137930+0000","last_became_peered":"2024-04-02T04:59:00.137930+0000","last_unstale":"2024-04-02T04:59:52.858954+0000","last_undegraded":"2024-04-02T04:59:52.858954+0000","last_fullsized":"2024-04-02T04:59:52.858954+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'13","last_scrub_stamp":"2024-04-02T04:58:38.093860+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:58:38.093860+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"402'25","reported_seq":837,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337704+0000","last_change":"2024-04-02T04:59:20.099182+0000","last_active":"2024-04-02T04:59:53.337704+0000","last_peered":"2024-04-02T04:59:53.337704+0000","last_clean":"2024-04-02T04:59:53.337704+0000","last_became_active":"2024-04-02T04:59:20.080550+0000","last_became_peered":"2024-04-02T04:59:20.080550+0000","last_unstale":"2024-04-02T04:59:53.337704+0000","last_undegraded":"2024-04-02T04:59:53.337704+0000","last_fullsized":"2024-04-02T04:59:53.337704+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:59:12.433584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:12.433584+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:33.741206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":1320,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.860590+0000","last_change":"2024-04-02T04:58:50.033861+0000","last_active":"2024-04-02T04:59:52.860590+0000","last_peered":"2024-04-02T04:59:52.860590+0000","last_clean":"2024-04-02T04:59:52.860590+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T04:59:52.860590+0000","last_undegraded":"2024-04-02T04:59:52.860590+0000","last_fullsized":"2024-04-02T04:59:52.860590+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:58:50.033443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:50.033443+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:09.781287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138457+0000","last_change":"2024-04-02T04:59:00.124915+0000","last_active":"2024-04-02T04:59:52.138457+0000","last_peered":"2024-04-02T04:59:52.138457+0000","last_clean":"2024-04-02T04:59:52.138457+0000","last_became_active":"2024-04-02T04:59:00.122883+0000","last_became_peered":"2024-04-02T04:59:00.122883+0000","last_unstale":"2024-04-02T04:59:52.138457+0000","last_undegraded":"2024-04-02T04:59:52.138457+0000","last_fullsized":"2024-04-02T04:59:52.138457+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:49.195904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:49.195904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1028,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.122002+0000","last_change":"2024-04-02T04:59:00.106367+0000","last_active":"2024-04-02T04:59:53.122002+0000","last_peered":"2024-04-02T04:59:53.122002+0000","last_clean":"2024-04-02T04:59:53.122002+0000","last_became_active":"2024-04-02T04:59:00.106070+0000","last_became_peered":"2024-04-02T04:59:00.106070+0000","last_unstale":"2024-04-02T04:59:53.122002+0000","last_undegraded":"2024-04-02T04:59:53.122002+0000","last_fullsized":"2024-04-02T04:59:53.122002+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:30.126652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:30.126652+0000","last_clean_scrub_stamp":"2024-04-02T04:56:30.126652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"402'25","reported_seq":807,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337795+0000","last_change":"2024-04-02T04:59:15.470244+0000","last_active":"2024-04-02T04:59:53.337795+0000","last_peered":"2024-04-02T04:59:53.337795+0000","last_clean":"2024-04-02T04:59:53.337795+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T04:59:53.337795+0000","last_undegraded":"2024-04-02T04:59:53.337795+0000","last_fullsized":"2024-04-02T04:59:53.337795+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:59:15.470090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:15.470090+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:22.020058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1300,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588363+0000","last_change":"2024-04-02T04:58:42.230390+0000","last_active":"2024-04-02T04:59:52.588363+0000","last_peered":"2024-04-02T04:59:52.588363+0000","last_clean":"2024-04-02T04:59:52.588363+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T04:59:52.588363+0000","last_undegraded":"2024-04-02T04:59:52.588363+0000","last_fullsized":"2024-04-02T04:59:52.588363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:42.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.230240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:05.495103+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1286,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.594115+0000","last_change":"2024-04-02T04:59:00.139090+0000","last_active":"2024-04-02T04:59:52.594115+0000","last_peered":"2024-04-02T04:59:52.594115+0000","last_clean":"2024-04-02T04:59:52.594115+0000","last_became_active":"2024-04-02T04:59:00.138663+0000","last_became_peered":"2024-04-02T04:59:00.138663+0000","last_unstale":"2024-04-02T04:59:52.594115+0000","last_undegraded":"2024-04-02T04:59:52.594115+0000","last_fullsized":"2024-04-02T04:59:52.594115+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_clean_scrub_stamp":"2024-04-02T04:54:35.398676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1281,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.590207+0000","last_change":"2024-04-02T04:59:13.463965+0000","last_active":"2024-04-02T04:59:52.590207+0000","last_peered":"2024-04-02T04:59:52.590207+0000","last_clean":"2024-04-02T04:59:52.590207+0000","last_became_active":"2024-04-02T04:53:54.925132+0000","last_became_peered":"2024-04-02T04:53:54.925132+0000","last_unstale":"2024-04-02T04:59:52.590207+0000","last_undegraded":"2024-04-02T04:59:52.590207+0000","last_fullsized":"2024-04-02T04:59:52.590207+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:13.463559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:59:13.463559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:26.762938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1154,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.122435+0000","last_change":"2024-04-02T04:59:00.106276+0000","last_active":"2024-04-02T04:59:53.122435+0000","last_peered":"2024-04-02T04:59:53.122435+0000","last_clean":"2024-04-02T04:59:53.122435+0000","last_became_active":"2024-04-02T04:59:00.104872+0000","last_became_peered":"2024-04-02T04:59:00.104872+0000","last_unstale":"2024-04-02T04:59:53.122435+0000","last_undegraded":"2024-04-02T04:59:53.122435+0000","last_fullsized":"2024-04-02T04:59:53.122435+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:31.495522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:31.495522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"402'8","reported_seq":1321,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.112584+0000","last_change":"2024-04-02T04:59:00.164109+0000","last_active":"2024-04-02T04:59:52.112584+0000","last_peered":"2024-04-02T04:59:52.112584+0000","last_clean":"2024-04-02T04:59:52.112584+0000","last_became_active":"2024-04-02T04:59:00.105124+0000","last_became_peered":"2024-04-02T04:59:00.105124+0000","last_unstale":"2024-04-02T04:59:52.112584+0000","last_undegraded":"2024-04-02T04:59:52.112584+0000","last_fullsized":"2024-04-02T04:59:52.112584+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'8","last_scrub_stamp":"2024-04-02T04:54:17.927334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.927334+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1291,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.110099+0000","last_change":"2024-04-02T04:59:20.079468+0000","last_active":"2024-04-02T04:59:52.110099+0000","last_peered":"2024-04-02T04:59:52.110099+0000","last_clean":"2024-04-02T04:59:52.110099+0000","last_became_active":"2024-04-02T04:59:20.074948+0000","last_became_peered":"2024-04-02T04:59:20.074948+0000","last_unstale":"2024-04-02T04:59:52.110099+0000","last_undegraded":"2024-04-02T04:59:52.110099+0000","last_fullsized":"2024-04-02T04:59:52.110099+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:16.144273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.144273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.128014+0000","last_change":"2024-04-02T04:59:00.128791+0000","last_active":"2024-04-02T04:59:52.128014+0000","last_peered":"2024-04-02T04:59:52.128014+0000","last_clean":"2024-04-02T04:59:52.128014+0000","last_became_active":"2024-04-02T04:59:00.122643+0000","last_became_peered":"2024-04-02T04:59:00.122643+0000","last_unstale":"2024-04-02T04:59:52.128014+0000","last_undegraded":"2024-04-02T04:59:52.128014+0000","last_fullsized":"2024-04-02T04:59:52.128014+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:43.172463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:57:43.172463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":762,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337540+0000","last_change":"2024-04-02T04:59:20.069882+0000","last_active":"2024-04-02T04:59:53.337540+0000","last_peered":"2024-04-02T04:59:53.337540+0000","last_clean":"2024-04-02T04:59:53.337540+0000","last_became_active":"2024-04-02T04:59:20.069216+0000","last_became_peered":"2024-04-02T04:59:20.069216+0000","last_unstale":"2024-04-02T04:59:53.337540+0000","last_undegraded":"2024-04-02T04:59:53.337540+0000","last_fullsized":"2024-04-02T04:59:53.337540+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:42.120564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:42.120564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":876,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.594058+0000","last_change":"2024-04-02T04:57:38.080227+0000","last_active":"2024-04-02T04:59:52.594058+0000","last_peered":"2024-04-02T04:59:52.594058+0000","last_clean":"2024-04-02T04:59:52.594058+0000","last_became_active":"2024-04-02T04:57:38.079839+0000","last_became_peered":"2024-04-02T04:57:38.079839+0000","last_unstale":"2024-04-02T04:59:52.594058+0000","last_undegraded":"2024-04-02T04:59:52.594058+0000","last_fullsized":"2024-04-02T04:59:52.594058+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:26.900131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T04:55:26.900131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"402'19","reported_seq":772,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.338032+0000","last_change":"2024-04-02T04:56:38.082686+0000","last_active":"2024-04-02T04:59:53.338032+0000","last_peered":"2024-04-02T04:59:53.338032+0000","last_clean":"2024-04-02T04:59:53.338032+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T04:59:53.338032+0000","last_undegraded":"2024-04-02T04:59:53.338032+0000","last_fullsized":"2024-04-02T04:59:53.338032+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'19","last_scrub_stamp":"2024-04-02T04:56:38.082345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:38.082345+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.127468+0000","last_change":"2024-04-02T04:59:00.135183+0000","last_active":"2024-04-02T04:59:52.127468+0000","last_peered":"2024-04-02T04:59:52.127468+0000","last_clean":"2024-04-02T04:59:52.127468+0000","last_became_active":"2024-04-02T04:59:00.134529+0000","last_became_peered":"2024-04-02T04:59:00.134529+0000","last_unstale":"2024-04-02T04:59:52.127468+0000","last_undegraded":"2024-04-02T04:59:52.127468+0000","last_fullsized":"2024-04-02T04:59:52.127468+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:37.148900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:37.148900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"642'17","reported_seq":130,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.126379+0000","last_change":"2024-04-02T04:59:00.220320+0000","last_active":"2024-04-02T04:59:52.126379+0000","last_peered":"2024-04-02T04:59:52.126379+0000","last_clean":"2024-04-02T04:59:52.126379+0000","last_became_active":"2024-04-02T04:59:00.133162+0000","last_became_peered":"2024-04-02T04:59:00.133162+0000","last_unstale":"2024-04-02T04:59:52.126379+0000","last_undegraded":"2024-04-02T04:59:52.126379+0000","last_fullsized":"2024-04-02T04:59:52.126379+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'16","last_scrub_stamp":"2024-04-02T04:57:41.153649+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:57:41.153649+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1290,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.594022+0000","last_change":"2024-04-02T04:59:14.938140+0000","last_active":"2024-04-02T04:59:52.594022+0000","last_peered":"2024-04-02T04:59:52.594022+0000","last_clean":"2024-04-02T04:59:52.594022+0000","last_became_active":"2024-04-02T04:59:00.138302+0000","last_became_peered":"2024-04-02T04:59:00.138302+0000","last_unstale":"2024-04-02T04:59:52.594022+0000","last_undegraded":"2024-04-02T04:59:52.594022+0000","last_fullsized":"2024-04-02T04:59:52.594022+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:14.937933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:26.393399+0000","last_clean_scrub_stamp":"2024-04-02T04:59:14.937933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:37.572208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":129,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138212+0000","last_change":"2024-04-02T04:59:20.067666+0000","last_active":"2024-04-02T04:59:52.138212+0000","last_peered":"2024-04-02T04:59:52.138212+0000","last_clean":"2024-04-02T04:59:52.138212+0000","last_became_active":"2024-04-02T04:59:20.067119+0000","last_became_peered":"2024-04-02T04:59:20.067119+0000","last_unstale":"2024-04-02T04:59:52.138212+0000","last_undegraded":"2024-04-02T04:59:52.138212+0000","last_fullsized":"2024-04-02T04:59:52.138212+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:06.976087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:06.976087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:18.177694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"402'15","reported_seq":1351,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.592259+0000","last_change":"2024-04-02T04:59:00.214953+0000","last_active":"2024-04-02T04:59:52.592259+0000","last_peered":"2024-04-02T04:59:52.592259+0000","last_clean":"2024-04-02T04:59:52.592259+0000","last_became_active":"2024-04-02T04:59:00.139853+0000","last_became_peered":"2024-04-02T04:59:00.139853+0000","last_unstale":"2024-04-02T04:59:52.592259+0000","last_undegraded":"2024-04-02T04:59:52.592259+0000","last_fullsized":"2024-04-02T04:59:52.592259+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'15","last_scrub_stamp":"2024-04-02T04:57:55.973853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:55.973853+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1301,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.592538+0000","last_change":"2024-04-02T04:59:00.104438+0000","last_active":"2024-04-02T04:59:52.592538+0000","last_peered":"2024-04-02T04:59:52.592538+0000","last_clean":"2024-04-02T04:59:52.592538+0000","last_became_active":"2024-04-02T04:59:00.104064+0000","last_became_peered":"2024-04-02T04:59:00.104064+0000","last_unstale":"2024-04-02T04:59:52.592538+0000","last_undegraded":"2024-04-02T04:59:52.592538+0000","last_fullsized":"2024-04-02T04:59:52.592538+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:45.058981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:45.058981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:11.000084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"642'16","reported_seq":1390,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.339723+0000","last_change":"2024-04-02T04:59:00.193480+0000","last_active":"2024-04-02T04:59:53.339723+0000","last_peered":"2024-04-02T04:59:53.339723+0000","last_clean":"2024-04-02T04:59:53.339723+0000","last_became_active":"2024-04-02T04:59:00.136907+0000","last_became_peered":"2024-04-02T04:59:00.136907+0000","last_unstale":"2024-04-02T04:59:53.339723+0000","last_undegraded":"2024-04-02T04:59:53.339723+0000","last_fullsized":"2024-04-02T04:59:53.339723+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'15","last_scrub_stamp":"2024-04-02T04:58:50.497333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:58:50.497333+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:18.250864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1280,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.860909+0000","last_change":"2024-04-02T04:56:17.099070+0000","last_active":"2024-04-02T04:59:52.860909+0000","last_peered":"2024-04-02T04:59:52.860909+0000","last_clean":"2024-04-02T04:59:52.860909+0000","last_became_active":"2024-04-02T04:54:56.000394+0000","last_became_peered":"2024-04-02T04:54:56.000394+0000","last_unstale":"2024-04-02T04:59:52.860909+0000","last_undegraded":"2024-04-02T04:59:52.860909+0000","last_fullsized":"2024-04-02T04:59:52.860909+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:17.098961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:17.098961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990357+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1133,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.111855+0000","last_change":"2024-04-02T04:55:23.873415+0000","last_active":"2024-04-02T04:59:52.111855+0000","last_peered":"2024-04-02T04:59:52.111855+0000","last_clean":"2024-04-02T04:59:52.111855+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T04:59:52.111855+0000","last_undegraded":"2024-04-02T04:59:52.111855+0000","last_fullsized":"2024-04-02T04:59:52.111855+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:23.873192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:23.873192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"402'21","reported_seq":845,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.339563+0000","last_change":"2024-04-02T04:59:00.221766+0000","last_active":"2024-04-02T04:59:53.339563+0000","last_peered":"2024-04-02T04:59:53.339563+0000","last_clean":"2024-04-02T04:59:53.339563+0000","last_became_active":"2024-04-02T04:59:00.137738+0000","last_became_peered":"2024-04-02T04:59:00.137738+0000","last_unstale":"2024-04-02T04:59:53.339563+0000","last_undegraded":"2024-04-02T04:59:53.339563+0000","last_fullsized":"2024-04-02T04:59:53.339563+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T04:55:20.279092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:55:20.279092+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1295,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.112261+0000","last_change":"2024-04-02T04:59:00.105775+0000","last_active":"2024-04-02T04:59:52.112261+0000","last_peered":"2024-04-02T04:59:52.112261+0000","last_clean":"2024-04-02T04:59:52.112261+0000","last_became_active":"2024-04-02T04:59:00.105424+0000","last_became_peered":"2024-04-02T04:59:00.105424+0000","last_unstale":"2024-04-02T04:59:52.112261+0000","last_undegraded":"2024-04-02T04:59:52.112261+0000","last_fullsized":"2024-04-02T04:59:52.112261+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.859624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.859624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":242,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.112589+0000","last_change":"2024-04-02T04:57:38.085399+0000","last_active":"2024-04-02T04:59:52.112589+0000","last_peered":"2024-04-02T04:59:52.112589+0000","last_clean":"2024-04-02T04:59:52.112589+0000","last_became_active":"2024-04-02T04:57:38.083311+0000","last_became_peered":"2024-04-02T04:57:38.083311+0000","last_unstale":"2024-04-02T04:59:52.112589+0000","last_undegraded":"2024-04-02T04:59:52.112589+0000","last_fullsized":"2024-04-02T04:59:52.112589+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:37.487754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:37.487754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1283,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589999+0000","last_change":"2024-04-02T04:56:35.017181+0000","last_active":"2024-04-02T04:59:52.589999+0000","last_peered":"2024-04-02T04:59:52.589999+0000","last_clean":"2024-04-02T04:59:52.589999+0000","last_became_active":"2024-04-02T04:54:56.005128+0000","last_became_peered":"2024-04-02T04:54:56.005128+0000","last_unstale":"2024-04-02T04:59:52.589999+0000","last_undegraded":"2024-04-02T04:59:52.589999+0000","last_fullsized":"2024-04-02T04:59:52.589999+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:35.017057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:35.017057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1143,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859964+0000","last_change":"2024-04-02T04:56:18.079213+0000","last_active":"2024-04-02T04:59:52.859964+0000","last_peered":"2024-04-02T04:59:52.859964+0000","last_clean":"2024-04-02T04:59:52.859964+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T04:59:52.859964+0000","last_undegraded":"2024-04-02T04:59:52.859964+0000","last_fullsized":"2024-04-02T04:59:52.859964+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:18.079082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.079082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"402'32","reported_seq":842,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.336581+0000","last_change":"2024-04-02T04:59:20.097757+0000","last_active":"2024-04-02T04:59:53.336581+0000","last_peered":"2024-04-02T04:59:53.336581+0000","last_clean":"2024-04-02T04:59:53.336581+0000","last_became_active":"2024-04-02T04:59:20.057918+0000","last_became_peered":"2024-04-02T04:59:20.057918+0000","last_unstale":"2024-04-02T04:59:53.336581+0000","last_undegraded":"2024-04-02T04:59:53.336581+0000","last_fullsized":"2024-04-02T04:59:53.336581+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'32","last_scrub_stamp":"2024-04-02T04:58:48.115768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:48.115768+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:17.839609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1299,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123636+0000","last_change":"2024-04-02T04:58:42.899985+0000","last_active":"2024-04-02T04:59:53.123636+0000","last_peered":"2024-04-02T04:59:53.123636+0000","last_clean":"2024-04-02T04:59:53.123636+0000","last_became_active":"2024-04-02T04:54:55.987373+0000","last_became_peered":"2024-04-02T04:54:55.987373+0000","last_unstale":"2024-04-02T04:59:53.123636+0000","last_undegraded":"2024-04-02T04:59:53.123636+0000","last_fullsized":"2024-04-02T04:59:53.123636+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:42.899853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.899853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:55.094727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1286,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859183+0000","last_change":"2024-04-02T04:56:40.845726+0000","last_active":"2024-04-02T04:59:52.859183+0000","last_peered":"2024-04-02T04:59:52.859183+0000","last_clean":"2024-04-02T04:59:52.859183+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T04:59:52.859183+0000","last_undegraded":"2024-04-02T04:59:52.859183+0000","last_fullsized":"2024-04-02T04:59:52.859183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_clean_scrub_stamp":"2024-04-02T04:56:40.845504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":123,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.128834+0000","last_change":"2024-04-02T04:59:14.003983+0000","last_active":"2024-04-02T04:59:52.128834+0000","last_peered":"2024-04-02T04:59:52.128834+0000","last_clean":"2024-04-02T04:59:52.128834+0000","last_became_active":"2024-04-02T04:59:00.136143+0000","last_became_peered":"2024-04-02T04:59:00.136143+0000","last_unstale":"2024-04-02T04:59:52.128834+0000","last_undegraded":"2024-04-02T04:59:52.128834+0000","last_fullsized":"2024-04-02T04:59:52.128834+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:14.003869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:59:14.003869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:24.320187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1153,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589497+0000","last_change":"2024-04-02T04:58:52.967879+0000","last_active":"2024-04-02T04:59:52.589497+0000","last_peered":"2024-04-02T04:59:52.589497+0000","last_clean":"2024-04-02T04:59:52.589497+0000","last_became_active":"2024-04-02T04:54:56.017129+0000","last_became_peered":"2024-04-02T04:54:56.017129+0000","last_unstale":"2024-04-02T04:59:52.589497+0000","last_undegraded":"2024-04-02T04:59:52.589497+0000","last_fullsized":"2024-04-02T04:59:52.589497+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:52.967749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:58:52.967749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:17.991280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"402'18","reported_seq":1371,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.595303+0000","last_change":"2024-04-02T04:59:00.231555+0000","last_active":"2024-04-02T04:59:52.595303+0000","last_peered":"2024-04-02T04:59:52.595303+0000","last_clean":"2024-04-02T04:59:52.595303+0000","last_became_active":"2024-04-02T04:59:00.139298+0000","last_became_peered":"2024-04-02T04:59:00.139298+0000","last_unstale":"2024-04-02T04:59:52.595303+0000","last_undegraded":"2024-04-02T04:59:52.595303+0000","last_fullsized":"2024-04-02T04:59:52.595303+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'18","last_scrub_stamp":"2024-04-02T04:56:17.986089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:56:17.986089+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":501,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.109200+0000","last_change":"2024-04-02T04:54:55.973248+0000","last_active":"2024-04-02T04:59:52.109200+0000","last_peered":"2024-04-02T04:59:52.109200+0000","last_clean":"2024-04-02T04:59:52.109200+0000","last_became_active":"2024-04-02T04:54:55.972841+0000","last_became_peered":"2024-04-02T04:54:55.972841+0000","last_unstale":"2024-04-02T04:59:52.109200+0000","last_undegraded":"2024-04-02T04:59:52.109200+0000","last_fullsized":"2024-04-02T04:59:52.109200+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:53:12.624937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:53:12.624937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.137252+0000","last_change":"2024-04-02T04:59:00.102477+0000","last_active":"2024-04-02T04:59:52.137252+0000","last_peered":"2024-04-02T04:59:52.137252+0000","last_clean":"2024-04-02T04:59:52.137252+0000","last_became_active":"2024-04-02T04:59:00.102092+0000","last_became_peered":"2024-04-02T04:59:00.102092+0000","last_unstale":"2024-04-02T04:59:52.137252+0000","last_undegraded":"2024-04-02T04:59:52.137252+0000","last_fullsized":"2024-04-02T04:59:52.137252+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:34.146107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:34.146107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":758,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337443+0000","last_change":"2024-04-02T04:59:00.399088+0000","last_active":"2024-04-02T04:59:53.337443+0000","last_peered":"2024-04-02T04:59:53.337443+0000","last_clean":"2024-04-02T04:59:53.337443+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T04:59:53.337443+0000","last_undegraded":"2024-04-02T04:59:53.337443+0000","last_fullsized":"2024-04-02T04:59:53.337443+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:00.398882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:59:00.398882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:29.239316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":637,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.338959+0000","last_change":"2024-04-02T04:59:14.445982+0000","last_active":"2024-04-02T04:59:53.338959+0000","last_peered":"2024-04-02T04:59:53.338959+0000","last_clean":"2024-04-02T04:59:53.338959+0000","last_became_active":"2024-04-02T04:53:54.927422+0000","last_became_peered":"2024-04-02T04:53:54.927422+0000","last_unstale":"2024-04-02T04:59:53.338959+0000","last_undegraded":"2024-04-02T04:59:53.338959+0000","last_fullsized":"2024-04-02T04:59:53.338959+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:14.445746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:14.445746+0000","last_clean_scrub_stamp":"2024-04-02T04:59:14.445746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:24.979872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"402'27","reported_seq":1308,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.338187+0000","last_change":"2024-04-02T04:59:00.201939+0000","last_active":"2024-04-02T04:59:53.338187+0000","last_peered":"2024-04-02T04:59:53.338187+0000","last_clean":"2024-04-02T04:59:53.338187+0000","last_became_active":"2024-04-02T04:59:00.136183+0000","last_became_peered":"2024-04-02T04:59:00.136183+0000","last_unstale":"2024-04-02T04:59:53.338187+0000","last_undegraded":"2024-04-02T04:59:53.338187+0000","last_fullsized":"2024-04-02T04:59:53.338187+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T04:58:42.485994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.485994+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1291,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.109329+0000","last_change":"2024-04-02T04:54:55.979542+0000","last_active":"2024-04-02T04:59:52.109329+0000","last_peered":"2024-04-02T04:59:52.109329+0000","last_clean":"2024-04-02T04:59:52.109329+0000","last_became_active":"2024-04-02T04:54:55.978936+0000","last_became_peered":"2024-04-02T04:54:55.978936+0000","last_unstale":"2024-04-02T04:59:52.109329+0000","last_undegraded":"2024-04-02T04:59:52.109329+0000","last_fullsized":"2024-04-02T04:59:52.109329+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:34.979136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.979136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.129096+0000","last_change":"2024-04-02T04:59:00.124514+0000","last_active":"2024-04-02T04:59:52.129096+0000","last_peered":"2024-04-02T04:59:52.129096+0000","last_clean":"2024-04-02T04:59:52.129096+0000","last_became_active":"2024-04-02T04:59:00.122631+0000","last_became_peered":"2024-04-02T04:59:00.122631+0000","last_unstale":"2024-04-02T04:59:52.129096+0000","last_undegraded":"2024-04-02T04:59:52.129096+0000","last_fullsized":"2024-04-02T04:59:52.129096+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:54.470294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:58:54.470294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:57.640745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138540+0000","last_change":"2024-04-02T04:59:00.137226+0000","last_active":"2024-04-02T04:59:52.138540+0000","last_peered":"2024-04-02T04:59:52.138540+0000","last_clean":"2024-04-02T04:59:52.138540+0000","last_became_active":"2024-04-02T04:59:00.136243+0000","last_became_peered":"2024-04-02T04:59:00.136243+0000","last_unstale":"2024-04-02T04:59:52.138540+0000","last_undegraded":"2024-04-02T04:59:52.138540+0000","last_fullsized":"2024-04-02T04:59:52.138540+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:44.085844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:44.085844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":603,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.338657+0000","last_change":"2024-04-02T04:59:20.055671+0000","last_active":"2024-04-02T04:59:53.338657+0000","last_peered":"2024-04-02T04:59:53.338657+0000","last_clean":"2024-04-02T04:59:53.338657+0000","last_became_active":"2024-04-02T04:59:20.054470+0000","last_became_peered":"2024-04-02T04:59:20.054470+0000","last_unstale":"2024-04-02T04:59:53.338657+0000","last_undegraded":"2024-04-02T04:59:53.338657+0000","last_fullsized":"2024-04-02T04:59:53.338657+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:56.993348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:56.993348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:58.748796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"402'62","reported_seq":1481,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.122511+0000","last_change":"2024-04-02T04:59:20.119598+0000","last_active":"2024-04-02T04:59:53.122511+0000","last_peered":"2024-04-02T04:59:53.122511+0000","last_clean":"2024-04-02T04:59:53.122511+0000","last_became_active":"2024-04-02T04:59:20.053373+0000","last_became_peered":"2024-04-02T04:59:20.053373+0000","last_unstale":"2024-04-02T04:59:53.122511+0000","last_undegraded":"2024-04-02T04:59:53.122511+0000","last_fullsized":"2024-04-02T04:59:53.122511+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'62","last_scrub_stamp":"2024-04-02T04:59:07.986820+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:59:07.986820+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:28.959117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1191,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.858653+0000","last_change":"2024-04-02T04:58:42.062580+0000","last_active":"2024-04-02T04:59:52.858653+0000","last_peered":"2024-04-02T04:59:52.858653+0000","last_clean":"2024-04-02T04:59:52.858653+0000","last_became_active":"2024-04-02T04:54:56.008703+0000","last_became_peered":"2024-04-02T04:54:56.008703+0000","last_unstale":"2024-04-02T04:59:52.858653+0000","last_undegraded":"2024-04-02T04:59:52.858653+0000","last_fullsized":"2024-04-02T04:59:52.858653+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:42.062477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.062477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1295,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.593878+0000","last_change":"2024-04-02T04:59:00.138885+0000","last_active":"2024-04-02T04:59:52.593878+0000","last_peered":"2024-04-02T04:59:52.593878+0000","last_clean":"2024-04-02T04:59:52.593878+0000","last_became_active":"2024-04-02T04:59:00.138125+0000","last_became_peered":"2024-04-02T04:59:00.138125+0000","last_unstale":"2024-04-02T04:59:52.593878+0000","last_undegraded":"2024-04-02T04:59:52.593878+0000","last_fullsized":"2024-04-02T04:59:52.593878+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:13.114440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:13.114440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742155+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":781,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.336895+0000","last_change":"2024-04-02T04:59:00.218566+0000","last_active":"2024-04-02T04:59:53.336895+0000","last_peered":"2024-04-02T04:59:53.336895+0000","last_clean":"2024-04-02T04:59:53.336895+0000","last_became_active":"2024-04-02T04:59:00.138970+0000","last_became_peered":"2024-04-02T04:59:00.138970+0000","last_unstale":"2024-04-02T04:59:53.336895+0000","last_undegraded":"2024-04-02T04:59:53.336895+0000","last_fullsized":"2024-04-02T04:59:53.336895+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T04:57:36.116986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:36.116986+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1036,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.588063+0000","last_change":"2024-04-02T04:59:15.923872+0000","last_active":"2024-04-02T04:59:52.588063+0000","last_peered":"2024-04-02T04:59:52.588063+0000","last_clean":"2024-04-02T04:59:52.588063+0000","last_became_active":"2024-04-02T04:54:55.989605+0000","last_became_peered":"2024-04-02T04:54:55.989605+0000","last_unstale":"2024-04-02T04:59:52.588063+0000","last_undegraded":"2024-04-02T04:59:52.588063+0000","last_fullsized":"2024-04-02T04:59:52.588063+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:15.923777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:15.923777+0000","last_clean_scrub_stamp":"2024-04-02T04:59:15.923777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:28.058172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"402'21","reported_seq":127,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.137430+0000","last_change":"2024-04-02T04:59:00.221341+0000","last_active":"2024-04-02T04:59:52.137430+0000","last_peered":"2024-04-02T04:59:52.137430+0000","last_clean":"2024-04-02T04:59:52.137430+0000","last_became_active":"2024-04-02T04:59:00.133276+0000","last_became_peered":"2024-04-02T04:59:00.133276+0000","last_unstale":"2024-04-02T04:59:52.137430+0000","last_undegraded":"2024-04-02T04:59:52.137430+0000","last_fullsized":"2024-04-02T04:59:52.137430+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T04:57:37.101609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:37.101609+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":127,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.128922+0000","last_change":"2024-04-02T04:59:09.983405+0000","last_active":"2024-04-02T04:59:52.128922+0000","last_peered":"2024-04-02T04:59:52.128922+0000","last_clean":"2024-04-02T04:59:52.128922+0000","last_became_active":"2024-04-02T04:59:00.135857+0000","last_became_peered":"2024-04-02T04:59:00.135857+0000","last_unstale":"2024-04-02T04:59:52.128922+0000","last_undegraded":"2024-04-02T04:59:52.128922+0000","last_fullsized":"2024-04-02T04:59:52.128922+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:59:09.983303+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.983303+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:32.925090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"451'12","reported_seq":1485,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.119145+0000","last_change":"2024-04-02T04:54:56.025228+0000","last_active":"2024-04-02T04:59:52.119145+0000","last_peered":"2024-04-02T04:59:52.119145+0000","last_clean":"2024-04-02T04:59:52.119145+0000","last_became_active":"2024-04-02T04:54:56.006792+0000","last_became_peered":"2024-04-02T04:54:56.006792+0000","last_unstale":"2024-04-02T04:59:52.119145+0000","last_undegraded":"2024-04-02T04:59:52.119145+0000","last_fullsized":"2024-04-02T04:59:52.119145+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-02T04:54:18.879221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:54:18.879221+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1293,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123986+0000","last_change":"2024-04-02T04:59:20.079038+0000","last_active":"2024-04-02T04:59:53.123986+0000","last_peered":"2024-04-02T04:59:53.123986+0000","last_clean":"2024-04-02T04:59:53.123986+0000","last_became_active":"2024-04-02T04:59:20.078674+0000","last_became_peered":"2024-04-02T04:59:20.078674+0000","last_unstale":"2024-04-02T04:59:53.123986+0000","last_undegraded":"2024-04-02T04:59:53.123986+0000","last_fullsized":"2024-04-02T04:59:53.123986+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:08.948569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:59:08.948569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:35.814608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1139,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.110037+0000","last_change":"2024-04-02T04:55:24.890462+0000","last_active":"2024-04-02T04:59:52.110037+0000","last_peered":"2024-04-02T04:59:52.110037+0000","last_clean":"2024-04-02T04:59:52.110037+0000","last_became_active":"2024-04-02T04:54:55.972361+0000","last_became_peered":"2024-04-02T04:54:55.972361+0000","last_unstale":"2024-04-02T04:59:52.110037+0000","last_undegraded":"2024-04-02T04:59:52.110037+0000","last_fullsized":"2024-04-02T04:59:52.110037+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:24.890319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.890319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"402'16","reported_seq":1314,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.592624+0000","last_change":"2024-04-02T04:59:08.922889+0000","last_active":"2024-04-02T04:59:52.592624+0000","last_peered":"2024-04-02T04:59:52.592624+0000","last_clean":"2024-04-02T04:59:52.592624+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T04:59:52.592624+0000","last_undegraded":"2024-04-02T04:59:52.592624+0000","last_fullsized":"2024-04-02T04:59:52.592624+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'16","last_scrub_stamp":"2024-04-02T04:59:08.922768+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:59:08.922768+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:23.066709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1286,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.115301+0000","last_change":"2024-04-02T04:54:10.943919+0000","last_active":"2024-04-02T04:59:52.115301+0000","last_peered":"2024-04-02T04:59:52.115301+0000","last_clean":"2024-04-02T04:59:52.115301+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T04:59:52.115301+0000","last_undegraded":"2024-04-02T04:59:52.115301+0000","last_fullsized":"2024-04-02T04:59:52.115301+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:10.943768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:54:10.943768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.809834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1186,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123537+0000","last_change":"2024-04-02T04:59:11.934062+0000","last_active":"2024-04-02T04:59:53.123537+0000","last_peered":"2024-04-02T04:59:53.123537+0000","last_clean":"2024-04-02T04:59:53.123537+0000","last_became_active":"2024-04-02T04:59:00.106341+0000","last_became_peered":"2024-04-02T04:59:00.106341+0000","last_unstale":"2024-04-02T04:59:53.123537+0000","last_undegraded":"2024-04-02T04:59:53.123537+0000","last_fullsized":"2024-04-02T04:59:53.123537+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:11.933852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:11.933852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:25.734157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1255,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.109019+0000","last_change":"2024-04-02T04:59:20.063789+0000","last_active":"2024-04-02T04:59:52.109019+0000","last_peered":"2024-04-02T04:59:52.109019+0000","last_clean":"2024-04-02T04:59:52.109019+0000","last_became_active":"2024-04-02T04:59:20.063395+0000","last_became_peered":"2024-04-02T04:59:20.063395+0000","last_unstale":"2024-04-02T04:59:52.109019+0000","last_undegraded":"2024-04-02T04:59:52.109019+0000","last_fullsized":"2024-04-02T04:59:52.109019+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:24.804934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.804934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.810283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1151,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.121421+0000","last_change":"2024-04-02T04:58:51.862964+0000","last_active":"2024-04-02T04:59:53.121421+0000","last_peered":"2024-04-02T04:59:53.121421+0000","last_clean":"2024-04-02T04:59:53.121421+0000","last_became_active":"2024-04-02T04:53:54.942584+0000","last_became_peered":"2024-04-02T04:53:54.942584+0000","last_unstale":"2024-04-02T04:59:53.121421+0000","last_undegraded":"2024-04-02T04:59:53.121421+0000","last_fullsized":"2024-04-02T04:59:53.121421+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:51.862866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:51.862866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:20.755327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"402'35","reported_seq":1339,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859208+0000","last_change":"2024-04-02T04:55:32.581265+0000","last_active":"2024-04-02T04:59:52.859208+0000","last_peered":"2024-04-02T04:59:52.859208+0000","last_clean":"2024-04-02T04:59:52.859208+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T04:59:52.859208+0000","last_undegraded":"2024-04-02T04:59:52.859208+0000","last_fullsized":"2024-04-02T04:59:52.859208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T04:55:32.581158+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:55:32.581158+0000","objects_scrubbed":6,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.989786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138378+0000","last_change":"2024-04-02T04:59:00.125332+0000","last_active":"2024-04-02T04:59:52.138378+0000","last_peered":"2024-04-02T04:59:52.138378+0000","last_clean":"2024-04-02T04:59:52.138378+0000","last_became_active":"2024-04-02T04:59:00.122388+0000","last_became_peered":"2024-04-02T04:59:00.122388+0000","last_unstale":"2024-04-02T04:59:52.138378+0000","last_undegraded":"2024-04-02T04:59:52.138378+0000","last_fullsized":"2024-04-02T04:59:52.138378+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:54.389749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:54.389749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.136695+0000","last_change":"2024-04-02T04:59:00.131623+0000","last_active":"2024-04-02T04:59:52.136695+0000","last_peered":"2024-04-02T04:59:52.136695+0000","last_clean":"2024-04-02T04:59:52.136695+0000","last_became_active":"2024-04-02T04:59:00.131269+0000","last_became_peered":"2024-04-02T04:59:00.131269+0000","last_unstale":"2024-04-02T04:59:52.136695+0000","last_undegraded":"2024-04-02T04:59:52.136695+0000","last_fullsized":"2024-04-02T04:59:52.136695+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:38.014594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:57:38.014594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.128093+0000","last_change":"2024-04-02T04:59:00.098858+0000","last_active":"2024-04-02T04:59:52.128093+0000","last_peered":"2024-04-02T04:59:52.128093+0000","last_clean":"2024-04-02T04:59:52.128093+0000","last_became_active":"2024-04-02T04:59:00.098535+0000","last_became_peered":"2024-04-02T04:59:00.098535+0000","last_unstale":"2024-04-02T04:59:52.128093+0000","last_undegraded":"2024-04-02T04:59:52.128093+0000","last_fullsized":"2024-04-02T04:59:52.128093+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:15.974122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:15.974122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1148,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.123067+0000","last_change":"2024-04-02T04:59:20.083002+0000","last_active":"2024-04-02T04:59:53.123067+0000","last_peered":"2024-04-02T04:59:53.123067+0000","last_clean":"2024-04-02T04:59:53.123067+0000","last_became_active":"2024-04-02T04:59:20.081346+0000","last_became_peered":"2024-04-02T04:59:20.081346+0000","last_unstale":"2024-04-02T04:59:53.123067+0000","last_undegraded":"2024-04-02T04:59:53.123067+0000","last_fullsized":"2024-04-02T04:59:53.123067+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.392235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.392235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"402'24","reported_seq":1263,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.860541+0000","last_change":"2024-04-02T04:57:45.944117+0000","last_active":"2024-04-02T04:59:52.860541+0000","last_peered":"2024-04-02T04:59:52.860541+0000","last_clean":"2024-04-02T04:59:52.860541+0000","last_became_active":"2024-04-02T04:54:56.014703+0000","last_became_peered":"2024-04-02T04:54:56.014703+0000","last_unstale":"2024-04-02T04:59:52.860541+0000","last_undegraded":"2024-04-02T04:59:52.860541+0000","last_fullsized":"2024-04-02T04:59:52.860541+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'24","last_scrub_stamp":"2024-04-02T04:57:45.943939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:45.943939+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":791,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.336591+0000","last_change":"2024-04-02T04:59:06.433001+0000","last_active":"2024-04-02T04:59:53.336591+0000","last_peered":"2024-04-02T04:59:53.336591+0000","last_clean":"2024-04-02T04:59:53.336591+0000","last_became_active":"2024-04-02T04:57:38.129831+0000","last_became_peered":"2024-04-02T04:57:38.129831+0000","last_unstale":"2024-04-02T04:59:53.336591+0000","last_undegraded":"2024-04-02T04:59:53.336591+0000","last_fullsized":"2024-04-02T04:59:53.336591+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:59:06.432895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:59:06.432895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:30.540454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1252,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.127286+0000","last_change":"2024-04-02T04:59:20.066645+0000","last_active":"2024-04-02T04:59:52.127286+0000","last_peered":"2024-04-02T04:59:52.127286+0000","last_clean":"2024-04-02T04:59:52.127286+0000","last_became_active":"2024-04-02T04:59:20.066268+0000","last_became_peered":"2024-04-02T04:59:20.066268+0000","last_unstale":"2024-04-02T04:59:52.127286+0000","last_undegraded":"2024-04-02T04:59:52.127286+0000","last_fullsized":"2024-04-02T04:59:52.127286+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:33.895601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:33.895601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1270,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.589899+0000","last_change":"2024-04-02T04:54:24.386420+0000","last_active":"2024-04-02T04:59:52.589899+0000","last_peered":"2024-04-02T04:59:52.589899+0000","last_clean":"2024-04-02T04:59:52.589899+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T04:59:52.589899+0000","last_undegraded":"2024-04-02T04:59:52.589899+0000","last_fullsized":"2024-04-02T04:59:52.589899+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:24.386320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:54:24.386320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:53.742062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.130467+0000","last_change":"2024-04-02T04:59:00.134461+0000","last_active":"2024-04-02T04:59:52.130467+0000","last_peered":"2024-04-02T04:59:52.130467+0000","last_clean":"2024-04-02T04:59:52.130467+0000","last_became_active":"2024-04-02T04:59:00.134110+0000","last_became_peered":"2024-04-02T04:59:00.134110+0000","last_unstale":"2024-04-02T04:59:52.130467+0000","last_undegraded":"2024-04-02T04:59:52.130467+0000","last_fullsized":"2024-04-02T04:59:52.130467+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:16.955150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:16.955150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"402'27","reported_seq":1265,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859256+0000","last_change":"2024-04-02T04:57:31.948865+0000","last_active":"2024-04-02T04:59:52.859256+0000","last_peered":"2024-04-02T04:59:52.859256+0000","last_clean":"2024-04-02T04:59:52.859256+0000","last_became_active":"2024-04-02T04:54:56.011842+0000","last_became_peered":"2024-04-02T04:54:56.011842+0000","last_unstale":"2024-04-02T04:59:52.859256+0000","last_undegraded":"2024-04-02T04:59:52.859256+0000","last_fullsized":"2024-04-02T04:59:52.859256+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T04:57:31.948728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:31.948728+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":776,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.337239+0000","last_change":"2024-04-02T04:56:18.940836+0000","last_active":"2024-04-02T04:59:53.337239+0000","last_peered":"2024-04-02T04:59:53.337239+0000","last_clean":"2024-04-02T04:59:53.337239+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T04:59:53.337239+0000","last_undegraded":"2024-04-02T04:59:53.337239+0000","last_fullsized":"2024-04-02T04:59:53.337239+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:18.940739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:56:18.940739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1264,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.137937+0000","last_change":"2024-04-02T04:59:20.067706+0000","last_active":"2024-04-02T04:59:52.137937+0000","last_peered":"2024-04-02T04:59:52.137937+0000","last_clean":"2024-04-02T04:59:52.137937+0000","last_became_active":"2024-04-02T04:59:20.066444+0000","last_became_peered":"2024-04-02T04:59:20.066444+0000","last_unstale":"2024-04-02T04:59:52.137937+0000","last_undegraded":"2024-04-02T04:59:52.137937+0000","last_fullsized":"2024-04-02T04:59:52.137937+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:22.882047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:55:22.882047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":524,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-04-02T04:59:53.122734+0000","last_change":"2024-04-02T04:59:00.105228+0000","last_active":"2024-04-02T04:59:53.122734+0000","last_peered":"2024-04-02T04:59:53.122734+0000","last_clean":"2024-04-02T04:59:53.122734+0000","last_became_active":"2024-04-02T04:59:00.104874+0000","last_became_peered":"2024-04-02T04:59:00.104874+0000","last_unstale":"2024-04-02T04:59:53.122734+0000","last_undegraded":"2024-04-02T04:59:53.122734+0000","last_fullsized":"2024-04-02T04:59:53.122734+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:34.477630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:34.477630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1145,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.109201+0000","last_change":"2024-04-02T04:59:20.052845+0000","last_active":"2024-04-02T04:59:52.109201+0000","last_peered":"2024-04-02T04:59:52.109201+0000","last_clean":"2024-04-02T04:59:52.109201+0000","last_became_active":"2024-04-02T04:59:20.052388+0000","last_became_peered":"2024-04-02T04:59:20.052388+0000","last_unstale":"2024-04-02T04:59:52.109201+0000","last_undegraded":"2024-04-02T04:59:52.109201+0000","last_fullsized":"2024-04-02T04:59:52.109201+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:57.900914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:57.900914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:03.376906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"402'25","reported_seq":1389,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.587919+0000","last_change":"2024-04-02T04:59:20.098776+0000","last_active":"2024-04-02T04:59:52.587919+0000","last_peered":"2024-04-02T04:59:52.587919+0000","last_clean":"2024-04-02T04:59:52.587919+0000","last_became_active":"2024-04-02T04:59:20.078754+0000","last_became_peered":"2024-04-02T04:59:20.078754+0000","last_unstale":"2024-04-02T04:59:52.587919+0000","last_undegraded":"2024-04-02T04:59:52.587919+0000","last_fullsized":"2024-04-02T04:59:52.587919+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:58:58.968760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:58.968760+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:08.688700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.129957+0000","last_change":"2024-04-02T04:59:00.136252+0000","last_active":"2024-04-02T04:59:52.129957+0000","last_peered":"2024-04-02T04:59:52.129957+0000","last_clean":"2024-04-02T04:59:52.129957+0000","last_became_active":"2024-04-02T04:59:00.135892+0000","last_became_peered":"2024-04-02T04:59:00.135892+0000","last_unstale":"2024-04-02T04:59:52.129957+0000","last_undegraded":"2024-04-02T04:59:52.129957+0000","last_fullsized":"2024-04-02T04:59:52.129957+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_clean_scrub_stamp":"2024-04-02T04:57:50.043598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1278,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.858345+0000","last_change":"2024-04-02T04:56:39.854503+0000","last_active":"2024-04-02T04:59:52.858345+0000","last_peered":"2024-04-02T04:59:52.858345+0000","last_clean":"2024-04-02T04:59:52.858345+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T04:59:52.858345+0000","last_undegraded":"2024-04-02T04:59:52.858345+0000","last_fullsized":"2024-04-02T04:59:52.858345+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:39.854219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T04:56:39.854219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1280,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.859034+0000","last_change":"2024-04-02T04:57:41.058839+0000","last_active":"2024-04-02T04:59:52.859034+0000","last_peered":"2024-04-02T04:59:52.859034+0000","last_clean":"2024-04-02T04:59:52.859034+0000","last_became_active":"2024-04-02T04:54:55.984764+0000","last_became_peered":"2024-04-02T04:54:55.984764+0000","last_unstale":"2024-04-02T04:59:52.859034+0000","last_undegraded":"2024-04-02T04:59:52.859034+0000","last_fullsized":"2024-04-02T04:59:52.859034+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_clean_scrub_stamp":"2024-04-02T04:57:41.058751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1155,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.860757+0000","last_change":"2024-04-02T04:59:00.135877+0000","last_active":"2024-04-02T04:59:52.860757+0000","last_peered":"2024-04-02T04:59:52.860757+0000","last_clean":"2024-04-02T04:59:52.860757+0000","last_became_active":"2024-04-02T04:59:00.135538+0000","last_became_peered":"2024-04-02T04:59:00.135538+0000","last_unstale":"2024-04-02T04:59:52.860757+0000","last_undegraded":"2024-04-02T04:59:52.860757+0000","last_fullsized":"2024-04-02T04:59:52.860757+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:20.082793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:20.082793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:54.990095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"402'32","reported_seq":137,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.129870+0000","last_change":"2024-04-02T04:59:02.951909+0000","last_active":"2024-04-02T04:59:52.129870+0000","last_peered":"2024-04-02T04:59:52.129870+0000","last_clean":"2024-04-02T04:59:52.129870+0000","last_became_active":"2024-04-02T04:59:00.099386+0000","last_became_peered":"2024-04-02T04:59:00.099386+0000","last_unstale":"2024-04-02T04:59:52.129870+0000","last_undegraded":"2024-04-02T04:59:52.129870+0000","last_fullsized":"2024-04-02T04:59:52.129870+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'32","last_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_deep_scrub":"402'32","last_deep_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_clean_scrub_stamp":"2024-04-02T04:59:02.951751+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:12.818664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1296,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.587748+0000","last_change":"2024-04-02T04:59:20.079425+0000","last_active":"2024-04-02T04:59:52.587748+0000","last_peered":"2024-04-02T04:59:52.587748+0000","last_clean":"2024-04-02T04:59:52.587748+0000","last_became_active":"2024-04-02T04:59:20.077694+0000","last_became_peered":"2024-04-02T04:59:20.077694+0000","last_unstale":"2024-04-02T04:59:52.587748+0000","last_undegraded":"2024-04-02T04:59:52.587748+0000","last_fullsized":"2024-04-02T04:59:52.587748+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:07.012983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:59:07.012983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:31.362765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.129175+0000","last_change":"2024-04-02T04:59:00.138239+0000","last_active":"2024-04-02T04:59:52.129175+0000","last_peered":"2024-04-02T04:59:52.129175+0000","last_clean":"2024-04-02T04:59:52.129175+0000","last_became_active":"2024-04-02T04:59:00.137920+0000","last_became_peered":"2024-04-02T04:59:00.137920+0000","last_unstale":"2024-04-02T04:59:52.129175+0000","last_undegraded":"2024-04-02T04:59:52.129175+0000","last_fullsized":"2024-04-02T04:59:52.129175+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:48.290606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:58:48.290606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:58.767258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":115,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-04-02T04:59:52.138294+0000","last_change":"2024-04-02T04:59:00.124528+0000","last_active":"2024-04-02T04:59:52.138294+0000","last_peered":"2024-04-02T04:59:52.138294+0000","last_clean":"2024-04-02T04:59:52.138294+0000","last_became_active":"2024-04-02T04:59:00.122322+0000","last_became_peered":"2024-04-02T04:59:00.122322+0000","last_unstale":"2024-04-02T04:59:52.138294+0000","last_undegraded":"2024-04-02T04:59:52.138294+0000","last_fullsized":"2024-04-02T04:59:52.138294+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:52.472669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:58:52.472669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:07.267635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":949074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":4296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":953,"ondisk_log_size":953,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114698,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48492,"kb_used_data":11980,"kb_used_omap":57,"kb_used_meta":36422,"kb_avail":93696660,"statfs":{"total":95995035648,"available":95945379840,"internally_reserved":0,"allocated":12267520,"data_stored":8142178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":37297038},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103381,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115340,"kb_used_data":11948,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93629812,"statfs":{"total":95995035648,"available":95876927488,"internally_reserved":0,"allocated":12234752,"data_stored":8130361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92836,"kb_used_data":11640,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93652316,"statfs":{"total":95995035648,"available":95899971584,"internally_reserved":0,"allocated":11919360,"data_stored":7803946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738642,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110852,"kb_used_data":11632,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93634300,"statfs":{"total":95995035648,"available":95881523200,"internally_reserved":0,"allocated":11911168,"data_stored":7793399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378878,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115240,"kb_used_data":11848,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93629912,"statfs":{"total":95995035648,"available":95877029888,"internally_reserved":0,"allocated":12132352,"data_stored":8056036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051828,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87828,"kb_used_data":12200,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657324,"statfs":{"total":95995035648,"available":95905099776,"internally_reserved":0,"allocated":12492800,"data_stored":8393123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822890,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97232,"kb_used_data":11888,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93647920,"statfs":{"total":95995035648,"available":95895470080,"internally_reserved":0,"allocated":12173312,"data_stored":8065615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463134,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T05:00:00.004 INFO:tasks.thrashosds.thrasher:Setting 10.8 to [5, 1, 7] 2024-04-02T05:00:00.004 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '10.8', '5', '1', '7'] 2024-04-02T05:00:00.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 10.8 5 1 7 2024-04-02T05:00:00.016 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:00.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.408286+0000 osd.7 (osd.7) 157 : cluster 0 1.2 scrub starts 2024-04-02T05:00:00.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.410663+0000 osd.7 (osd.7) 158 : cluster 0 1.2 scrub ok 2024-04-02T05:00:00.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:00.141 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:00.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:00.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.408286+0000 osd.7 (osd.7) 157 : cluster 0 1.2 scrub starts 2024-04-02T05:00:00.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.410663+0000 osd.7 (osd.7) 158 : cluster 0 1.2 scrub ok 2024-04-02T05:00:00.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.813222+0000 osd.3 (osd.3) 345 : cluster 0 6.1f scrub starts 2024-04-02T05:00:00.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.815416+0000 osd.3 (osd.3) 346 : cluster 0 6.1f scrub ok 2024-04-02T05:00:00.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.878095+0000 osd.4 (osd.4) 356 : cluster 0 3.3 scrub starts 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.882220+0000 osd.6 (osd.6) 297 : cluster 0 4.5 scrub starts 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.883650+0000 osd.6 (osd.6) 298 : cluster 0 4.5 scrub ok 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:58.883926+0000 osd.4 (osd.4) 357 : cluster 0 3.3 scrub ok 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:59.021864+0000 osd.2 (osd.2) 317 : cluster 0 4.d scrub starts 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:59.024004+0000 osd.2 (osd.2) 318 : cluster 0 4.d scrub ok 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: audit 2024-04-02T04:59:59.042073+0000 mon.a (mon.0) 3443 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T04:59:59.045525+0000 mon.a (mon.0) 3444 : cluster 0 osdmap e701: 8 total, 8 up, 7 in 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: audit 2024-04-02T04:59:59.046257+0000 mon.a (mon.0) 3445 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-112"}]: dispatch 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.000145+0000 mon.a (mon.0) 3446 : cluster 3 Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 pool(s) do not have an application enabled 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.000212+0000 mon.a (mon.0) 3447 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.000235+0000 mon.a (mon.0) 3448 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.000256+0000 mon.a (mon.0) 3449 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.000298+0000 mon.a (mon.0) 3450 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72349-111' 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.000319+0000 mon.a (mon.0) 3451 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-02T05:00:00.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.026396+0000 mon.a (mon.0) 3452 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[21528]: audit 2024-04-02T05:00:00.043936+0000 mon.a (mon.0) 3453 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-112"}]': finished 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.408286+0000 osd.7 (osd.7) 157 : cluster 0 1.2 scrub starts 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.410663+0000 osd.7 (osd.7) 158 : cluster 0 1.2 scrub ok 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.813222+0000 osd.3 (osd.3) 345 : cluster 0 6.1f scrub starts 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.815416+0000 osd.3 (osd.3) 346 : cluster 0 6.1f scrub ok 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.878095+0000 osd.4 (osd.4) 356 : cluster 0 3.3 scrub starts 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.882220+0000 osd.6 (osd.6) 297 : cluster 0 4.5 scrub starts 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.883650+0000 osd.6 (osd.6) 298 : cluster 0 4.5 scrub ok 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:58.883926+0000 osd.4 (osd.4) 357 : cluster 0 3.3 scrub ok 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:59.021864+0000 osd.2 (osd.2) 317 : cluster 0 4.d scrub starts 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:59.024004+0000 osd.2 (osd.2) 318 : cluster 0 4.d scrub ok 2024-04-02T05:00:00.253 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: audit 2024-04-02T04:59:59.042073+0000 mon.a (mon.0) 3443 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T04:59:59.045525+0000 mon.a (mon.0) 3444 : cluster 0 osdmap e701: 8 total, 8 up, 7 in 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: audit 2024-04-02T04:59:59.046257+0000 mon.a (mon.0) 3445 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-112"}]: dispatch 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.000145+0000 mon.a (mon.0) 3446 : cluster 3 Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 pool(s) do not have an application enabled 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.000212+0000 mon.a (mon.0) 3447 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.000235+0000 mon.a (mon.0) 3448 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.000256+0000 mon.a (mon.0) 3449 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.000298+0000 mon.a (mon.0) 3450 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72349-111' 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.000319+0000 mon.a (mon.0) 3451 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.026396+0000 mon.a (mon.0) 3452 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:00.254 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:00 smithi069 bash[28904]: audit 2024-04-02T05:00:00.043936+0000 mon.a (mon.0) 3453 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-112"}]': finished 2024-04-02T05:00:00.262 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25240.0:6 120.0 120:0384ab8e:::oid:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:05.985013+0000", 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 594.29344905899995, 2024-04-02T05:00:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.059777596000000002, 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25240", 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/485445834", 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 6 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985013+0000", 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985013+0000", 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985014+0000", 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2580000000000001e-06 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985018+0000", 2024-04-02T05:00:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2640000000000002e-06 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985027+0000", 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9570000000000008e-06 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985037+0000", 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.024e-05 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985207+0000", 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016968500000000001 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985303+0000", 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5766000000000004e-05 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1", 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.985451+0000", 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014827800000000001 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:05.990202+0000", 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047510699999999996 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.044636+0000", 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.054434187000000002 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.044698+0000", 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1958999999999994e-05 2024-04-02T05:00:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:06.044790+0000", 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1932e-05 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25586.0:2 131.1b 131:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:07.423280+0000", 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 592.85518164400003, 2024-04-02T05:00:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.069137781, 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25586", 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/3965586338", 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.423280+0000", 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.423280+0000", 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.423281+0000", 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.545e-06 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.423287+0000", 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3920000000000002e-06 2024-04-02T05:00:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.423296+0000", 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5040000000000008e-06 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.423307+0000", 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1031e-05 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.438282+0000", 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014974397 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:07.438291+0000", 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1789999999999997e-06 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.472968+0000", 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0346771910000001 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.473088+0000", 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000119425 2024-04-02T05:00:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,5", 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.473219+0000", 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013146699999999999 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.478879+0000", 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005659664 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.489211+0000", 2024-04-02T05:00:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010332533 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.492321+0000", 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003109573 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.492370+0000", 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8690999999999998e-05 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:08.492418+0000", 2024-04-02T05:00:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8189000000000002e-05 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.25210.0:35 112.10 e174/162 112:0bb6dc9d:test-rados-api-smithi069-72389-10::foo:head v 174'7, mlcod=173'4)", 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:12.753080+0000", 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 587.52538185799995, 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.06159804, 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.753080+0000", 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.753080+0000", 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.753080+0000", 2024-04-02T05:00:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0900000000000002e-07 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.753088+0000", 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0749999999999998e-06 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.753098+0000", 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0059e-05 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.753109+0000", 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0321e-05 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.807951+0000", 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.054842296999999998 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.807975+0000", 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4130000000000001e-05 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.814632+0000", 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066566480000000003 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:12.814678+0000", 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6001000000000002e-05 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.25525.0:1 202.5s0 202:abfdb2a5:test-rados-api-smithi069-72304-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-04-02T05:00:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:50:57.713637+0000", 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 542.56482507299995, 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.051457812, 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.25525", 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2157606784", 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713637+0000", 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713637+0000", 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713638+0000", 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4810000000000001e-06 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713645+0000", 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.596e-06 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713656+0000", 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1051e-05 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713667+0000", 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1076e-05 2024-04-02T05:00:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713726+0000", 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9505e-05 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:57.713729+0000", 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4650000000000001e-06 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:58.761910+0000", 2024-04-02T05:00:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.048180975 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:58.762043+0000", 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013333299999999999 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:58.762359+0000", 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00031584400000000002 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:58.762940+0000", 2024-04-02T05:00:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058072199999999999 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:58.765013+0000", 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0020738369999999998 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:50:58.765094+0000", 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0927000000000006e-05 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.15999.0:1 222.6s0 222:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:51:26.723445+0000", 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 513.55501665899999, 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.029328011, 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.15999", 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2876630749", 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723445+0000", 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723445+0000", 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723447+0000", 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.092e-06 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723455+0000", 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2120000000000003e-06 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723469+0000", 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4245e-05 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723481+0000", 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1639e-05 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723541+0000", 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9930999999999999e-05 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:26.723546+0000", 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5220000000000001e-06 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:27.737628+0000", 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.014081963 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:27.737804+0000", 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000176269 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:27.738315+0000", 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00051079299999999995 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:27.741569+0000", 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0032549100000000002 2024-04-02T05:00:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:27.752649+0000", 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011079333 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:51:27.752773+0000", 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012410200000000001 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.26306.0:1 257.6s0 257:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:52:37.179621+0000", 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 443.09884022400001, 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.033196279, 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.26306", 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2861385427", 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179621+0000", 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179621+0000", 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179623+0000", 2024-04-02T05:00:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.451e-06 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179631+0000", 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2500000000000006e-06 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179641+0000", 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8439999999999993e-06 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179651+0000", 2024-04-02T05:00:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0560000000000001e-05 2024-04-02T05:00:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179707+0000", 2024-04-02T05:00:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5263999999999998e-05 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:37.179710+0000", 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8250000000000001e-06 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.195797+0000", 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0160877049999999 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.195955+0000", 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000158164 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.196397+0000", 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00044185200000000001 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.197457+0000", 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001059335 2024-04-02T05:00:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.212742+0000", 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015285798 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:38.212818+0000", 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5230999999999999e-05 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.16236.0:1 261.6s0 261:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:52:44.188680+0000", 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 436.08978120699999, 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.70075011899999995, 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.16236", 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/845592075", 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188680+0000", 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188680+0000", 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188682+0000", 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8300000000000001e-06 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188689+0000", 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8329999999999996e-06 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188705+0000", 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6379e-05 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188719+0000", 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3363000000000001e-05 2024-04-02T05:00:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188785+0000", 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5767999999999998e-05 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.188789+0000", 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1810000000000003e-06 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.859890+0000", 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.67110143 2024-04-02T05:00:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.860036+0000", 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014553600000000001 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.860618+0000", 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058197000000000001 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.862187+0000", 2024-04-02T05:00:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001569427 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.889343+0000", 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.027156079 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:52:44.889430+0000", 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7323000000000004e-05 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.16368.0:1 281.2s0 281:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:52.940159+0000", 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 367.33830266400003, 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9962394400000001, 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.16368", 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2108977557", 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1 2024-04-02T05:00:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940159+0000", 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940159+0000", 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940160+0000", 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4810000000000001e-06 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940166+0000", 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0170000000000002e-06 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940178+0000", 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1746000000000001e-05 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940191+0000", 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2737e-05 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940261+0000", 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0535000000000001e-05 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:52.940265+0000", 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3840000000000001e-06 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:53.925675+0000", 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.98541034500000002 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:53.925682+0000", 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.9500000000000004e-06 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.915100+0000", 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.98941800400000002 2024-04-02T05:00:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.915252+0000", 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015156300000000001 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.915728+0000", 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047595599999999998 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.933813+0000", 2024-04-02T05:00:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.018085615999999999 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.936319+0000", 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0025061900000000002 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.936398+0000", 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8916000000000001e-05 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:2489 4.0 4:05bf5b68:::notify.1:head [watch ping cookie 94452570226560 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:54.776153+0000", 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 365.50230895700003, 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.16137082999999999, 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2489 2024-04-02T05:00:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776153+0000", 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776153+0000", 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776156+0000", 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9560000000000002e-06 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776168+0000", 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2136999999999999e-05 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776225+0000", 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7055999999999997e-05 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776252+0000", 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7676000000000001e-05 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776668+0000", 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00041559099999999999 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776710+0000", 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1468999999999999e-05 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.937270+0000", 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.160560703 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.937432+0000", 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000162294 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.937523+0000", 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0947999999999997e-05 2024-04-02T05:00:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:2490 4.14 4:2b04a3e9:::notify.6:head [watch ping cookie 94452570220800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:54.776351+0000", 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 365.502110786, 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.14946004399999999, 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2490 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776351+0000", 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776351+0000", 2024-04-02T05:00:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776353+0000", 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7829999999999999e-06 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776363+0000", 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0669999999999999e-05 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776387+0000", 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3748999999999999e-05 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776406+0000", 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8712999999999999e-05 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776726+0000", 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000320332 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776759+0000", 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2709999999999997e-05 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.925527+0000", 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.148767817 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.925729+0000", 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020243299999999999 2024-04-02T05:00:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.925811+0000", 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1836999999999996e-05 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:2498 4.0 4:05bf5b68:::notify.1:head [watch reconnect cookie 94452570226560 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:54.779393+0000", 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 365.49906899699999, 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.17156611999999999, 2024-04-02T05:00:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2498 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779393+0000", 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779393+0000", 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779395+0000", 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.283e-06 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779405+0000", 2024-04-02T05:00:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0507e-05 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779436+0000", 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0552999999999997e-05 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779468+0000", 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2143e-05 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779663+0000", 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019468099999999999 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779699+0000", 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5812000000000002e-05 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.937638+0000", 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.157939098 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.937939+0000", 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030102699999999999 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.950959+0000", 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013020016000000001 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:2499 4.14 4:2b04a3e9:::notify.6:head [watch reconnect cookie 94452570220800 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:54.779518+0000", 2024-04-02T05:00:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 365.49894315500001, 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.16742304899999999, 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2499 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779518+0000", 2024-04-02T05:00:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779518+0000", 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779521+0000", 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3970000000000001e-06 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779530+0000", 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6730000000000004e-06 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779557+0000", 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6956000000000001e-05 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779576+0000", 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8960000000000001e-05 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779752+0000", 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017562 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779783+0000", 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0859999999999999e-05 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.925839+0000", 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.146056571 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.925914+0000", 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4692999999999998e-05 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.946941+0000", 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.021027318999999999 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.1.0:1552 322.6 e686/678 322:602f83fe:::foo:head v 686'1, mlcod=0'0)", 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.186019+0000", 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.092442784999999, 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00088855599999999998, 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186019+0000", 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186019+0000", 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186019+0000", 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9999999999999999e-07 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186027+0000", 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3240000000000003e-06 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186046+0000", 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8482e-05 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186056+0000", 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0283e-05 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186194+0000", 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013816499999999999 2024-04-02T05:00:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186215+0000", 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0842000000000001e-05 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186861+0000", 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00064547499999999996 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186907+0000", 2024-04-02T05:00:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6684999999999998e-05 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.1.0:1555 322.6 e686/678 322:602f83fe:::foo:head v 686'3, mlcod=686'1)", 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.204431+0000", 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.074030602000001, 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016531698000000001, 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204431+0000", 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204431+0000", 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204432+0000", 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.471e-06 2024-04-02T05:00:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204444+0000", 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1749999999999999e-05 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204466+0000", 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1466000000000001e-05 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204477+0000", 2024-04-02T05:00:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-05 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204565+0000", 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8807000000000005e-05 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204587+0000", 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1959000000000001e-05 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.220910+0000", 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016322782000000001 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.220963+0000", 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2533e-05 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.16773.0:13 322.1c 322:38000000::::head [pgnls start_epoch 686] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e686)", 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.236121+0000", 2024-04-02T05:00:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.042340716999998, 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00091704700000000004, 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.16773", 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/903817344", 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 13 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236121+0000", 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236121+0000", 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236122+0000", 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5799999999999998e-07 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236128+0000", 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8300000000000001e-06 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236139+0000", 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1803e-05 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236148+0000", 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3240000000000003e-06 2024-04-02T05:00:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236939+0000", 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079081500000000005 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.236967+0000", 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8136e-05 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.237038+0000", 2024-04-02T05:00:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1180999999999995e-05 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3655 4.0 4:05bf5b68:::notify.1:head [watch ping cookie 94452570226560 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e682)", 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:45.045877+0000", 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.232584502, 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00072586199999999997, 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3655 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045877+0000", 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045877+0000", 2024-04-02T05:00:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045879+0000", 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5290000000000001e-06 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045883+0000", 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.634e-06 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045895+0000", 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1266e-05 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045903+0000", 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4139999999999994e-06 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.046443+0000", 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00053968699999999996 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.046511+0000", 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8435999999999993e-05 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.046603+0000", 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1896000000000002e-05 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3657 3.5 3:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94452576124928 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e682)", 2024-04-02T05:00:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:45.045941+0000", 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.232520129999999, 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00085910499999999998, 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3657 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045941+0000", 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045941+0000", 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045942+0000", 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4700000000000002e-07 2024-04-02T05:00:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045944+0000", 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4490000000000002e-06 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045950+0000", 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4530000000000001e-06 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.045955+0000", 2024-04-02T05:00:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4380000000000003e-06 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.046639+0000", 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000683963 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.046710+0000", 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1335999999999996e-05 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:45.046801+0000", 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0118999999999996e-05 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3675 4.0 4:05bf5b68:::notify.1:head [watch ping cookie 94452570226560 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e691)", 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:55.054392+0000", 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.2240693760000001, 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00067111000000000004, 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3675 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054392+0000", 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054392+0000", 2024-04-02T05:00:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054394+0000", 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9539999999999998e-06 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054403+0000", 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.054e-06 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054424+0000", 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0377000000000002e-05 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054440+0000", 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6320999999999998e-05 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054918+0000", 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047812199999999999 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.054991+0000", 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3446000000000001e-05 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:55.055063+0000", 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1835999999999995e-05 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:3 324.9 324:91bcb6b0:test-rados-api-smithi069-72349-113::foo:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-04-02T05:00:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:58.084894+0000", 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 2.1935677440000001, 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040239569999999999, 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084894+0000", 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084894+0000", 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084896+0000", 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.917e-06 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084903+0000", 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5440000000000001e-06 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084918+0000", 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4592e-05 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084929+0000", 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1243000000000001e-05 2024-04-02T05:00:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084986+0000", 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7083999999999998e-05 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.085119+0000", 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000132364 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.085229+0000", 2024-04-02T05:00:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011078000000000001 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.085664+0000", 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00043415800000000003 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.087286+0000", 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016220239999999999 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.088854+0000", 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001568617 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.088886+0000", 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2045999999999998e-05 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.088918+0000", 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1588000000000003e-05 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.24418.0:933 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e699)", 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:58.247694+0000", 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 2.0307674580000001, 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00062344400000000002, 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.24418", 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.98:0/4085133895", 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 933 2024-04-02T05:00:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247694+0000", 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247694+0000", 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247697+0000", 2024-04-02T05:00:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2169999999999999e-06 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247715+0000", 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8048999999999999e-05 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247753+0000", 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7669000000000001e-05 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247773+0000", 2024-04-02T05:00:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9891999999999999e-05 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.247977+0000", 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020397199999999999 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.248133+0000", 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000155978 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:00:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.248318+0000", 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018466700000000001 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:00:00.387 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:00:00.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:00.399 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.813222+0000 osd.3 (osd.3) 345 : cluster 0 6.1f scrub starts 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.815416+0000 osd.3 (osd.3) 346 : cluster 0 6.1f scrub ok 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.878095+0000 osd.4 (osd.4) 356 : cluster 0 3.3 scrub starts 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.882220+0000 osd.6 (osd.6) 297 : cluster 0 4.5 scrub starts 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.883650+0000 osd.6 (osd.6) 298 : cluster 0 4.5 scrub ok 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:58.883926+0000 osd.4 (osd.4) 357 : cluster 0 3.3 scrub ok 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:59.021864+0000 osd.2 (osd.2) 317 : cluster 0 4.d scrub starts 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:59.024004+0000 osd.2 (osd.2) 318 : cluster 0 4.d scrub ok 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: audit 2024-04-02T04:59:59.042073+0000 mon.a (mon.0) 3443 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T04:59:59.045525+0000 mon.a (mon.0) 3444 : cluster 0 osdmap e701: 8 total, 8 up, 7 in 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: audit 2024-04-02T04:59:59.046257+0000 mon.a (mon.0) 3445 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-112"}]: dispatch 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.000145+0000 mon.a (mon.0) 3446 : cluster 3 Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 pool(s) do not have an application enabled 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.000212+0000 mon.a (mon.0) 3447 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.000235+0000 mon.a (mon.0) 3448 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-04-02T05:00:00.400 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.000256+0000 mon.a (mon.0) 3449 : cluster 3 application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-02T05:00:00.401 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.000298+0000 mon.a (mon.0) 3450 : cluster 3 application not enabled on pool 'test-rados-api-smithi069-72349-111' 2024-04-02T05:00:00.401 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.000319+0000 mon.a (mon.0) 3451 : cluster 3 use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-02T05:00:00.401 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.026396+0000 mon.a (mon.0) 3452 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:00.401 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:00 smithi098 bash[23790]: audit 2024-04-02T05:00:00.043936+0000 mon.a (mon.0) 3453 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-112"}]': finished 2024-04-02T05:00:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:00.517 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:00.540 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:00.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:00.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:00.802 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:00.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:00.932 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:00.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:01.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:01.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:01.191 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:01.295 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:01.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:01.425 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:01.450 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:59.394280+0000 osd.7 (osd.7) 159 : cluster 0 1.9 scrub starts 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: cluster 2024-04-02T04:59:59.396000+0000 osd.7 (osd.7) 160 : cluster 0 1.9 scrub ok 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.048109+0000 mon.a (mon.0) 3454 : cluster 0 osdmap e702: 8 total, 8 up, 7 in 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: audit 2024-04-02T05:00:00.119714+0000 mon.a (mon.0) 3455 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.272219+0000 mgr.y (mgr.14538) 635 : cluster 0 pgmap v1059: 214 pgs: 14 creating+peering, 200 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 27 B/s, 1 objects/s recovering 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.684711+0000 osd.0 (osd.0) 345 : cluster 0 6.18 scrub starts 2024-04-02T05:00:01.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:01 smithi098 bash[23790]: cluster 2024-04-02T05:00:00.687126+0000 osd.0 (osd.0) 346 : cluster 0 6.18 scrub ok 2024-04-02T05:00:01.554 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:01.580 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:59.394280+0000 osd.7 (osd.7) 159 : cluster 0 1.9 scrub starts 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: cluster 2024-04-02T04:59:59.396000+0000 osd.7 (osd.7) 160 : cluster 0 1.9 scrub ok 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.048109+0000 mon.a (mon.0) 3454 : cluster 0 osdmap e702: 8 total, 8 up, 7 in 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: audit 2024-04-02T05:00:00.119714+0000 mon.a (mon.0) 3455 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.272219+0000 mgr.y (mgr.14538) 635 : cluster 0 pgmap v1059: 214 pgs: 14 creating+peering, 200 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 27 B/s, 1 objects/s recovering 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.684711+0000 osd.0 (osd.0) 345 : cluster 0 6.18 scrub starts 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[21528]: cluster 2024-04-02T05:00:00.687126+0000 osd.0 (osd.0) 346 : cluster 0 6.18 scrub ok 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:59.394280+0000 osd.7 (osd.7) 159 : cluster 0 1.9 scrub starts 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: cluster 2024-04-02T04:59:59.396000+0000 osd.7 (osd.7) 160 : cluster 0 1.9 scrub ok 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.048109+0000 mon.a (mon.0) 3454 : cluster 0 osdmap e702: 8 total, 8 up, 7 in 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: audit 2024-04-02T05:00:00.119714+0000 mon.a (mon.0) 3455 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:01.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.272219+0000 mgr.y (mgr.14538) 635 : cluster 0 pgmap v1059: 214 pgs: 14 creating+peering, 200 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 27 B/s, 1 objects/s recovering 2024-04-02T05:00:01.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.684711+0000 osd.0 (osd.0) 345 : cluster 0 6.18 scrub starts 2024-04-02T05:00:01.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:01 smithi069 bash[28904]: cluster 2024-04-02T05:00:00.687126+0000 osd.0 (osd.0) 346 : cluster 0 6.18 scrub ok 2024-04-02T05:00:01.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:01.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:01.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:01.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-02T05:00:01.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:01.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- 172.21.15.69:0/3044675055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c101dd0 msgr2=0x7f008c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:01.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- 172.21.15.69:0/3044675055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c101dd0 0x7f008c105e20 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f00800099c0 tx=0x7f0080009cd0 comp rx=0 tx=0).stop 2024-04-02T05:00:01.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- 172.21.15.69:0/3044675055 shutdown_connections 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- 172.21.15.69:0/3044675055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f008c106500 0x7f008c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- 172.21.15.69:0/3044675055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f008c101420 0x7f008c101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- 172.21.15.69:0/3044675055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c101dd0 0x7f008c105e20 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- 172.21.15.69:0/3044675055 >> 172.21.15.69:0/3044675055 conn(0x7f008c078240 msgr2=0x7f008c078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- 172.21.15.69:0/3044675055 shutdown_connections 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- 172.21.15.69:0/3044675055 wait complete. 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 librados: starting msgr at 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 librados: starting objecter 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 Processor -- start 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- start start 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 librados: setting wanted keys 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 librados: calling monclient init 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f008c101420 0x7f008c10eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f008c101dd0 0x7f008c10eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:01.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c106500 0x7f008c10f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f008c111b10 con 0x7f008c101dd0 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f008c111990 con 0x7f008c101420 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0090808700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f008c111c90 con 0x7f008c106500 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0089d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f008c101dd0 0x7f008c10eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f008ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c106500 0x7f008c10f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0089d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f008c101dd0 0x7f008c10eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38162/0 (socket says 172.21.15.69:38162) 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f0089d9b700 1 -- 172.21.15.69:0/3437685324 learned_addr learned my addr 172.21.15.69:0/3437685324 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.953+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f008c101420 0x7f008c10eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:01.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f008c101420 msgr2=0x7f008c10eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:01.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f008c101420 0x7f008c10eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:01.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f008c101dd0 msgr2=0x7f008c10eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:01.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f008c101dd0 0x7f008c10eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c1a65f0 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c106500 0x7f008c10f530 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f0084007f10 tx=0x7f008400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 10 client.?.objecter ms_handle_connect 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084004db0 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0084009e20 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008c1a68e0 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084015400 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 10 client.43649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f008c1a6da0 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 1 librados: init done 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f008c04ff00 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f008400db20 con 0x7f008c106500 2024-04-02T05:00:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f007406f5e0 0x7f0074071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(703..703 src has 74..703) ==== 9962+0+0 (secure 0 0 0) 0x7f008408e300 con 0x7f008c106500 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 osd_map(703..703 src has 74..703) 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 3 client.43649.objecter handle_osd_map got epochs [703,703] > 0 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 3 client.43649.objecter handle_osd_map decoding full epoch 703 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f007406f5e0 0x7f0074071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 20 client.43649.objecter dump_active .. 0 homeless 2024-04-02T05:00:01.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v703) ==== 54+0+16 (secure 0 0 0) 0x7f0084010070 con 0x7f008c106500 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 10 client.43649.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 1 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f007406f5e0 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f007406f5e0 0x7f0074071aa0 secure :-1 s=READY pgs=3988 cs=0 l=1 rev1=1 crypto rx=0x7f007c009680 tx=0x7f007c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f008c062e50 0x7f008c063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 1 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 0 1 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 4 2024-04-02T05:00:01.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 10 client.43649.objecter _send_command 1 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f008c1aa160 con 0x7f008c062e50 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0089d9b700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f008c062e50 0x7f008c063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f0089d9b700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f008c062e50 0x7f008c063230 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.957+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c062e50 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.961+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c1aa160 con 0x7f008c062e50 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.961+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 1: 0 ) 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.961+0000 7f007b7fe700 10 client.43649.objecter _finish_command 1 = system:0 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.961+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 4 2024-04-02T05:00:01.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.961+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 0 1 2024-04-02T05:00:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:01.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 10 client.43649.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 2 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 2 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c10ffe0 osd=0 3 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 0 2 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 4 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 10 client.43649.objecter _send_command 2 2024-04-02T05:00:01.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c0628d0 con 0x7f008c062e50 2024-04-02T05:00:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c0628d0 con 0x7f008c062e50 2024-04-02T05:00:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:01.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f007b7fe700 10 client.43649.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 4 2024-04-02T05:00:01.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.977+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 0 2 2024-04-02T05:00:01.982 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 10 client.43649.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f008c02c9b0 0x7f008c02cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 1 3 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 4 2024-04-02T05:00:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 10 client.43649.objecter _send_command 3 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f008c108e70 con 0x7f008c02c9b0 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f008c02c9b0 0x7f008c02cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c02c9b0 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f008c02c9b0 0x7f008c02cd90 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c108e70 con 0x7f008c02c9b0 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 3: 0 ) 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f007b7fe700 10 client.43649.objecter _finish_command 3 = system:0 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 4 2024-04-02T05:00:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.981+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 1 3 2024-04-02T05:00:02.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 10 client.43649.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 4 2024-04-02T05:00:02.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:02.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 4 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1087f0 osd=1 3 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 1 4 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 4 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 10 client.43649.objecter _send_command 4 2024-04-02T05:00:02.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:01.997+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c108270 con 0x7f008c02c9b0 2024-04-02T05:00:02.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c108270 con 0x7f008c02c9b0 2024-04-02T05:00:02.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 10 client.43649.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 4 2024-04-02T05:00:02.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 1 4 2024-04-02T05:00:02.003 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:00:02.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 10 client.43649.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:02.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 5 2024-04-02T05:00:02.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f008c1ac320 0x7f008c1ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:02.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 5 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 2 5 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 4 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 10 client.43649.objecter _send_command 5 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f008c02dfb0 con 0x7f008c1ac320 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0089d9b700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f008c1ac320 0x7f008c1ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c1ac320 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f0089d9b700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f008c1ac320 0x7f008c1ae710 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c02dfb0 con 0x7f008c1ac320 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 5: 0 ) 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 10 client.43649.objecter _finish_command 5 = system:0 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 4 2024-04-02T05:00:02.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.001+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 2 5 2024-04-02T05:00:02.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 10 client.43649.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 6 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 6 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c108670 osd=2 3 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 2 6 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 4 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 10 client.43649.objecter _send_command 6 2024-04-02T05:00:02.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.017+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c02d5e0 con 0x7f008c1ac320 2024-04-02T05:00:02.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c02d5e0 con 0x7f008c1ac320 2024-04-02T05:00:02.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 10 client.43649.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 4 2024-04-02T05:00:02.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 2 6 2024-04-02T05:00:02.023 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:00:02.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 10 client.43649.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 7 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f008c1afdd0 0x7f008c1b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 7 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 3 7 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 4 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 10 client.43649.objecter _send_command 7 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f008c1b5020 con 0x7f008c1afdd0 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f008c1afdd0 0x7f008c1b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c1afdd0 2024-04-02T05:00:02.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f008c1afdd0 0x7f008c1b21c0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:02.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c1b5020 con 0x7f008c1afdd0 2024-04-02T05:00:02.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 7: 0 ) 2024-04-02T05:00:02.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 10 client.43649.objecter _finish_command 7 = system:0 2024-04-02T05:00:02.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 4 2024-04-02T05:00:02.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.021+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 3 7 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 10 client.43649.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 8 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 8 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1a97b0 osd=3 3 2024-04-02T05:00:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 3 8 2024-04-02T05:00:02.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 4 2024-04-02T05:00:02.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 10 client.43649.objecter _send_command 8 2024-04-02T05:00:02.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.037+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c1a9230 con 0x7f008c1afdd0 2024-04-02T05:00:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c1a9230 con 0x7f008c1afdd0 2024-04-02T05:00:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f007b7fe700 10 client.43649.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 4 2024-04-02T05:00:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 3 8 2024-04-02T05:00:02.044 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:00:02.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 10 client.43649.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:02.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 9 2024-04-02T05:00:02.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f008c1af930 0x7f008c1b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:02.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 9 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 4 9 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 4 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 10 client.43649.objecter _send_command 9 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f008c1b3ed0 con 0x7f008c1af930 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f008c1af930 0x7f008c1b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c1af930 2024-04-02T05:00:02.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.041+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f008c1af930 0x7f008c1b3880 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:02.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.045+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c1b3ed0 con 0x7f008c1af930 2024-04-02T05:00:02.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.045+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 9: 0 ) 2024-04-02T05:00:02.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.045+0000 7f007b7fe700 10 client.43649.objecter _finish_command 9 = system:0 2024-04-02T05:00:02.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.045+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 4 2024-04-02T05:00:02.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.045+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 4 9 2024-04-02T05:00:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 10 client.43649.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 10 2024-04-02T05:00:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 10 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1a9230 osd=4 3 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 4 10 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 4 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 10 client.43649.objecter _send_command 10 2024-04-02T05:00:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c1b3ed0 con 0x7f008c1af930 2024-04-02T05:00:02.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c1b3ed0 con 0x7f008c1af930 2024-04-02T05:00:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f007b7fe700 10 client.43649.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 4 2024-04-02T05:00:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 4 10 2024-04-02T05:00:02.065 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:00:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 10 client.43649.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 11 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f008c1b4620 0x7f008c1b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 11 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 5 11 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 4 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 10 client.43649.objecter _send_command 11 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f008c1bbcb0 con 0x7f008c1b4620 2024-04-02T05:00:02.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.061+0000 7f0089d9b700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f008c1b4620 0x7f008c1b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c1b4620 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f0089d9b700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f008c1b4620 0x7f008c1b4a80 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0080034a50 con 0x7f008c1b4620 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 11: 0 ) 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f007b7fe700 10 client.43649.objecter _finish_command 11 = system:0 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 4 2024-04-02T05:00:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.065+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 5 11 2024-04-02T05:00:02.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:02.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:02 smithi098 bash[23790]: cluster 2024-04-02T05:00:01.011857+0000 osd.2 (osd.2) 319 : cluster 0 10.9 scrub starts 2024-04-02T05:00:02.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:02 smithi098 bash[23790]: cluster 2024-04-02T05:00:01.015770+0000 osd.2 (osd.2) 320 : cluster 0 10.9 scrub ok 2024-04-02T05:00:02.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:02 smithi098 bash[23790]: audit 2024-04-02T05:00:01.066937+0000 mon.a (mon.0) 3456 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:02.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:02 smithi098 bash[23790]: cluster 2024-04-02T05:00:01.225729+0000 mon.a (mon.0) 3457 : cluster 0 osdmap e703: 8 total, 8 up, 7 in 2024-04-02T05:00:02.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:02 smithi098 bash[23790]: audit 2024-04-02T05:00:01.226388+0000 mon.a (mon.0) 3458 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112"}]: dispatch 2024-04-02T05:00:02.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:02 smithi098 bash[23790]: audit 2024-04-02T05:00:01.962319+0000 mon.c (mon.2) 1147 : audit 0 from='client.? 172.21.15.69:0/3437685324' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:02.092 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 10 client.43649.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 12 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 12 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1b3ed0 osd=5 3 2024-04-02T05:00:02.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 5 12 2024-04-02T05:00:02.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 4 2024-04-02T05:00:02.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 10 client.43649.objecter _send_command 12 2024-04-02T05:00:02.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.093+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c1b92e0 con 0x7f008c1b4620 2024-04-02T05:00:02.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c1b92e0 con 0x7f008c1b4620 2024-04-02T05:00:02.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 10 client.43649.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 4 2024-04-02T05:00:02.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 5 12 2024-04-02T05:00:02.098 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 10 client.43649.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 13 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f008c1bb240 0x7f008c1bb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 13 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 6 13 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 4 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 10 client.43649.objecter _send_command 13 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f008c1ba7d0 con 0x7f008c1bb240 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f008c1bb240 0x7f008c1bb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c1bb240 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f008ad9d700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f008c1bb240 0x7f008c1bb6a0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c1ba7d0 con 0x7f008c1bb240 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 13: 0 ) 2024-04-02T05:00:02.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 10 client.43649.objecter _finish_command 13 = system:0 2024-04-02T05:00:02.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 4 2024-04-02T05:00:02.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.097+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 6 13 2024-04-02T05:00:02.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 10 client.43649.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 14 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 14 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1b92e0 osd=6 3 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 6 14 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 4 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 10 client.43649.objecter _send_command 14 2024-04-02T05:00:02.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c1ba7d0 con 0x7f008c1bb240 2024-04-02T05:00:02.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c1ba7d0 con 0x7f008c1bb240 2024-04-02T05:00:02.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f007b7fe700 10 client.43649.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 4 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 6 14 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 10 client.43649.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 15 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f008c1b9b10 0x7f008c1b9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:02.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 15 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 7 15 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 4 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 10 client.43649.objecter _send_command 15 2024-04-02T05:00:02.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.117+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f008c1c2800 con 0x7f008c1b9b10 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f008c1b9b10 0x7f008c1b9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f007b7fe700 10 client.43649.objecter ms_handle_connect 0x7f008c1b9b10 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f008a59c700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f008c1b9b10 0x7f008c1b9f70 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f008c1c2800 con 0x7f008c1b9b10 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 15: 0 ) 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f007b7fe700 10 client.43649.objecter _finish_command 15 = system:0 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 4 2024-04-02T05:00:02.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.121+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 7 15 2024-04-02T05:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 10 client.43649.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign -1 16 2024-04-02T05:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 20 client.43649.objecter _get_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 15 client.43649.objecter _session_command_op_remove -1 16 2024-04-02T05:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 20 client.43649.objecter get_session s=0x7f008c1ba7d0 osd=7 3 2024-04-02T05:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 15 client.43649.objecter _session_command_op_assign 7 16 2024-04-02T05:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 4 2024-04-02T05:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 10 client.43649.objecter _send_command 16 2024-04-02T05:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.137+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f008c1bfee0 con 0x7f008c1b9b10 2024-04-02T05:00:02.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f007b7fe700 1 -- 172.21.15.69:0/3437685324 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f008c1bfee0 con 0x7f008c1b9b10 2024-04-02T05:00:02.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f007b7fe700 10 client.43649.objecter ms_dispatch 0x7f008c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:02.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f007b7fe700 10 client.43649.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:02.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f007b7fe700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 4 2024-04-02T05:00:02.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f007b7fe700 15 client.43649.objecter _session_command_op_remove 7 16 2024-04-02T05:00:02.143 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 librados: watch_flush enter 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 librados: watch_flush exit 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.0 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f008c062e50 msgr2=0x7f008c063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f008c062e50 0x7f008c063230 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c10ffe0 osd=0 1 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.1 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f008c02c9b0 msgr2=0x7f008c02cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f008c02c9b0 0x7f008c02cd90 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1087f0 osd=1 1 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.2 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f008c1ac320 msgr2=0x7f008c1ae710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f008c1ac320 0x7f008c1ae710 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c108670 osd=2 1 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.3 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f008c1afdd0 msgr2=0x7f008c1b21c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f008c1afdd0 0x7f008c1b21c0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a97b0 osd=3 1 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.4 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f008c1af930 msgr2=0x7f008c1b3880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f008c1af930 0x7f008c1b3880 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1a9230 osd=4 1 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.5 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f008c1b4620 msgr2=0x7f008c1b4a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f008c1b4620 0x7f008c1b4a80 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b3ed0 osd=5 1 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.6 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 reap_dead start 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f008c1bb240 msgr2=0x7f008c1bb6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f008c1bb240 0x7f008c1bb6a0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 reap_dead start 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1b92e0 osd=6 1 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter close_session for osd.7 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f008c1b9b10 msgr2=0x7f008c1b9f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f008c1b9b10 0x7f008c1b9f70 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter put_session s=0x7f008c1ba7d0 osd=7 1 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 20 client.43649.objecter shutdown clearing up homeless session... 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 10 client.43649.objecter successfully canceled tick 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 reap_dead start 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f008ad9d700 1 -- 172.21.15.69:0/3437685324 reap_dead start 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f007406f5e0 msgr2=0x7f0074071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f007406f5e0 0x7f0074071aa0 secure :-1 s=READY pgs=3988 cs=0 l=1 rev1=1 crypto rx=0x7f007c009680 tx=0x7f007c009260 comp rx=0 tx=0).stop 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c106500 msgr2=0x7f008c10f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c106500 0x7f008c10f530 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f0084007f10 tx=0x7f008400d3b0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 shutdown_connections 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f008c106500 0x7f008c10f530 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 --2- 172.21.15.69:0/3437685324 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f007406f5e0 0x7f0074071aa0 unknown :-1 s=CLOSED pgs=3988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 >> 172.21.15.69:0/3437685324 conn(0x7f008c078240 msgr2=0x7f008c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 shutdown_connections 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 -- 172.21.15.69:0/3437685324 wait complete. 2024-04-02T05:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:02.141+0000 7f0090808700 1 librados: shutdown 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:00:02.148 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:00:02.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[21528]: cluster 2024-04-02T05:00:01.011857+0000 osd.2 (osd.2) 319 : cluster 0 10.9 scrub starts 2024-04-02T05:00:02.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[21528]: cluster 2024-04-02T05:00:01.015770+0000 osd.2 (osd.2) 320 : cluster 0 10.9 scrub ok 2024-04-02T05:00:02.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[21528]: audit 2024-04-02T05:00:01.066937+0000 mon.a (mon.0) 3456 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:02.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[21528]: cluster 2024-04-02T05:00:01.225729+0000 mon.a (mon.0) 3457 : cluster 0 osdmap e703: 8 total, 8 up, 7 in 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[21528]: audit 2024-04-02T05:00:01.226388+0000 mon.a (mon.0) 3458 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112"}]: dispatch 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[21528]: audit 2024-04-02T05:00:01.962319+0000 mon.c (mon.2) 1147 : audit 0 from='client.? 172.21.15.69:0/3437685324' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[28904]: cluster 2024-04-02T05:00:01.011857+0000 osd.2 (osd.2) 319 : cluster 0 10.9 scrub starts 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[28904]: cluster 2024-04-02T05:00:01.015770+0000 osd.2 (osd.2) 320 : cluster 0 10.9 scrub ok 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[28904]: audit 2024-04-02T05:00:01.066937+0000 mon.a (mon.0) 3456 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[28904]: cluster 2024-04-02T05:00:01.225729+0000 mon.a (mon.0) 3457 : cluster 0 osdmap e703: 8 total, 8 up, 7 in 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[28904]: audit 2024-04-02T05:00:01.226388+0000 mon.a (mon.0) 3458 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112"}]: dispatch 2024-04-02T05:00:02.156 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:02 smithi069 bash[28904]: audit 2024-04-02T05:00:01.962319+0000 mon.c (mon.2) 1147 : audit 0 from='client.? 172.21.15.69:0/3437685324' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:02.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:02.220 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:02.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:02.350 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:02.453 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:02.479 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:02.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:02.607 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:02.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:02.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:02.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:02.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:03.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: cluster 2024-04-02T05:00:01.805918+0000 osd.3 (osd.3) 347 : cluster 0 6.4 scrub starts 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: cluster 2024-04-02T05:00:01.808292+0000 osd.3 (osd.3) 348 : cluster 0 6.4 scrub ok 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: audit 2024-04-02T05:00:02.068287+0000 mon.a (mon.0) 3459 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112"}]': finished 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: cluster 2024-04-02T05:00:02.071659+0000 mon.a (mon.0) 3460 : cluster 0 osdmap e704: 8 total, 8 up, 7 in 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: cluster 2024-04-02T05:00:02.123152+0000 osd.1 (osd.1) 355 : cluster 0 1.3 scrub starts 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: cluster 2024-04-02T05:00:02.124430+0000 osd.1 (osd.1) 356 : cluster 0 1.3 scrub ok 2024-04-02T05:00:03.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[23790]: cluster 2024-04-02T05:00:02.272971+0000 mgr.y (mgr.14538) 636 : cluster 0 pgmap v1062: 214 pgs: 14 creating+peering, 200 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 255 B/s wr, 0 op/s; 27 B/s, 1 objects/s recovering 2024-04-02T05:00:03.124 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:03.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:03.255 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:03.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:03.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: cluster 2024-04-02T05:00:01.805918+0000 osd.3 (osd.3) 347 : cluster 0 6.4 scrub starts 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: cluster 2024-04-02T05:00:01.808292+0000 osd.3 (osd.3) 348 : cluster 0 6.4 scrub ok 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: audit 2024-04-02T05:00:02.068287+0000 mon.a (mon.0) 3459 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112"}]': finished 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: cluster 2024-04-02T05:00:02.071659+0000 mon.a (mon.0) 3460 : cluster 0 osdmap e704: 8 total, 8 up, 7 in 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: cluster 2024-04-02T05:00:02.123152+0000 osd.1 (osd.1) 355 : cluster 0 1.3 scrub starts 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: cluster 2024-04-02T05:00:02.124430+0000 osd.1 (osd.1) 356 : cluster 0 1.3 scrub ok 2024-04-02T05:00:03.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[21528]: cluster 2024-04-02T05:00:02.272971+0000 mgr.y (mgr.14538) 636 : cluster 0 pgmap v1062: 214 pgs: 14 creating+peering, 200 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 255 B/s wr, 0 op/s; 27 B/s, 1 objects/s recovering 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: cluster 2024-04-02T05:00:01.805918+0000 osd.3 (osd.3) 347 : cluster 0 6.4 scrub starts 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: cluster 2024-04-02T05:00:01.808292+0000 osd.3 (osd.3) 348 : cluster 0 6.4 scrub ok 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: audit 2024-04-02T05:00:02.068287+0000 mon.a (mon.0) 3459 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-112"}]': finished 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: cluster 2024-04-02T05:00:02.071659+0000 mon.a (mon.0) 3460 : cluster 0 osdmap e704: 8 total, 8 up, 7 in 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: cluster 2024-04-02T05:00:02.123152+0000 osd.1 (osd.1) 355 : cluster 0 1.3 scrub starts 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: cluster 2024-04-02T05:00:02.124430+0000 osd.1 (osd.1) 356 : cluster 0 1.3 scrub ok 2024-04-02T05:00:03.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:03 smithi069 bash[28904]: cluster 2024-04-02T05:00:02.272971+0000 mgr.y (mgr.14538) 636 : cluster 0 pgmap v1062: 214 pgs: 14 creating+peering, 200 active+clean; 635 KiB data, 674 MiB used, 625 GiB / 626 GiB avail; 255 B/s wr, 0 op/s; 27 B/s, 1 objects/s recovering 2024-04-02T05:00:03.381 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:03.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:03.510 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:03.610 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:03.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:03.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:03.755 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:00:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:00:03.770 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:03.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:03.900 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:03.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:04.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:04.026 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:04.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:04.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[21528]: cluster 2024-04-02T05:00:02.390632+0000 osd.7 (osd.7) 161 : cluster 0 1.0 scrub starts 2024-04-02T05:00:04.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[21528]: cluster 2024-04-02T05:00:02.393124+0000 osd.7 (osd.7) 162 : cluster 0 1.0 scrub ok 2024-04-02T05:00:04.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[21528]: cluster 2024-04-02T05:00:03.091547+0000 mon.a (mon.0) 3461 : cluster 0 osdmap e705: 8 total, 8 up, 7 in 2024-04-02T05:00:04.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[21528]: audit 2024-04-02T05:00:03.582060+0000 mgr.y (mgr.14538) 637 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:04.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[21528]: audit 2024-04-02T05:00:03.794396+0000 mon.b (mon.1) 295 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:04.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[28904]: cluster 2024-04-02T05:00:02.390632+0000 osd.7 (osd.7) 161 : cluster 0 1.0 scrub starts 2024-04-02T05:00:04.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[28904]: cluster 2024-04-02T05:00:02.393124+0000 osd.7 (osd.7) 162 : cluster 0 1.0 scrub ok 2024-04-02T05:00:04.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[28904]: cluster 2024-04-02T05:00:03.091547+0000 mon.a (mon.0) 3461 : cluster 0 osdmap e705: 8 total, 8 up, 7 in 2024-04-02T05:00:04.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[28904]: audit 2024-04-02T05:00:03.582060+0000 mgr.y (mgr.14538) 637 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:04.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:04 smithi069 bash[28904]: audit 2024-04-02T05:00:03.794396+0000 mon.b (mon.1) 295 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:04.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:04.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:04.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:04 smithi098 bash[23790]: cluster 2024-04-02T05:00:02.390632+0000 osd.7 (osd.7) 161 : cluster 0 1.0 scrub starts 2024-04-02T05:00:04.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:04 smithi098 bash[23790]: cluster 2024-04-02T05:00:02.393124+0000 osd.7 (osd.7) 162 : cluster 0 1.0 scrub ok 2024-04-02T05:00:04.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:04 smithi098 bash[23790]: cluster 2024-04-02T05:00:03.091547+0000 mon.a (mon.0) 3461 : cluster 0 osdmap e705: 8 total, 8 up, 7 in 2024-04-02T05:00:04.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:04 smithi098 bash[23790]: audit 2024-04-02T05:00:03.582060+0000 mgr.y (mgr.14538) 637 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:04.396 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:04 smithi098 bash[23790]: audit 2024-04-02T05:00:03.794396+0000 mon.b (mon.1) 295 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:04.411 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:04.514 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:04.542 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:04.645 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:04.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:04.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:04.927 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:04.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:05.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:05.088 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:05.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:00:05.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:05.218 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:05.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:05.321 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:05.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:05.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:05.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:03.812697+0000 osd.4 (osd.4) 358 : cluster 0 4.e scrub starts 2024-04-02T05:00:05.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:03.815112+0000 osd.4 (osd.4) 359 : cluster 0 4.e scrub ok 2024-04-02T05:00:05.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:04.100665+0000 mon.a (mon.0) 3462 : cluster 0 osdmap e706: 8 total, 8 up, 7 in 2024-04-02T05:00:05.455 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: audit 2024-04-02T05:00:04.104433+0000 mon.a (mon.0) 3463 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:04.118520+0000 osd.2 (osd.2) 321 : cluster 0 2.e scrub starts 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:04.123263+0000 osd.2 (osd.2) 322 : cluster 0 2.e scrub ok 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:04.273826+0000 mgr.y (mgr.14538) 638 : cluster 0 pgmap v1065: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 675 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:04.741363+0000 osd.0 (osd.0) 347 : cluster 0 4.16 scrub starts 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:04.742827+0000 osd.0 (osd.0) 348 : cluster 0 4.16 scrub ok 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.027757+0000 mon.a (mon.0) 3464 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: audit 2024-04-02T05:00:05.032424+0000 mon.a (mon.0) 3465 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.041401+0000 mon.a (mon.0) 3466 : cluster 0 osdmap e707: 8 total, 8 up, 7 in 2024-04-02T05:00:05.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:05 smithi098 bash[23790]: audit 2024-04-02T05:00:05.081378+0000 mon.a (mon.0) 3467 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:05.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:05.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:03.812697+0000 osd.4 (osd.4) 358 : cluster 0 4.e scrub starts 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:03.815112+0000 osd.4 (osd.4) 359 : cluster 0 4.e scrub ok 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:04.100665+0000 mon.a (mon.0) 3462 : cluster 0 osdmap e706: 8 total, 8 up, 7 in 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: audit 2024-04-02T05:00:04.104433+0000 mon.a (mon.0) 3463 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:04.118520+0000 osd.2 (osd.2) 321 : cluster 0 2.e scrub starts 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:04.123263+0000 osd.2 (osd.2) 322 : cluster 0 2.e scrub ok 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:04.273826+0000 mgr.y (mgr.14538) 638 : cluster 0 pgmap v1065: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 675 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:05.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:04.741363+0000 osd.0 (osd.0) 347 : cluster 0 4.16 scrub starts 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:04.742827+0000 osd.0 (osd.0) 348 : cluster 0 4.16 scrub ok 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.027757+0000 mon.a (mon.0) 3464 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: audit 2024-04-02T05:00:05.032424+0000 mon.a (mon.0) 3465 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.041401+0000 mon.a (mon.0) 3466 : cluster 0 osdmap e707: 8 total, 8 up, 7 in 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[21528]: audit 2024-04-02T05:00:05.081378+0000 mon.a (mon.0) 3467 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:03.812697+0000 osd.4 (osd.4) 358 : cluster 0 4.e scrub starts 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:03.815112+0000 osd.4 (osd.4) 359 : cluster 0 4.e scrub ok 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:04.100665+0000 mon.a (mon.0) 3462 : cluster 0 osdmap e706: 8 total, 8 up, 7 in 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: audit 2024-04-02T05:00:04.104433+0000 mon.a (mon.0) 3463 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:04.118520+0000 osd.2 (osd.2) 321 : cluster 0 2.e scrub starts 2024-04-02T05:00:05.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:04.123263+0000 osd.2 (osd.2) 322 : cluster 0 2.e scrub ok 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:04.273826+0000 mgr.y (mgr.14538) 638 : cluster 0 pgmap v1065: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 675 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:04.741363+0000 osd.0 (osd.0) 347 : cluster 0 4.16 scrub starts 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:04.742827+0000 osd.0 (osd.0) 348 : cluster 0 4.16 scrub ok 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.027757+0000 mon.a (mon.0) 3464 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: audit 2024-04-02T05:00:05.032424+0000 mon.a (mon.0) 3465 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.041401+0000 mon.a (mon.0) 3466 : cluster 0 osdmap e707: 8 total, 8 up, 7 in 2024-04-02T05:00:05.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:05 smithi069 bash[28904]: audit 2024-04-02T05:00:05.081378+0000 mon.a (mon.0) 3467 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:05.728 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:05.831 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:05.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:05.988 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:05.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:06.091 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:06.117 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:06.127 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:06.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:06.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.155444+0000 osd.2 (osd.2) 323 : cluster 0 1.8 scrub starts 2024-04-02T05:00:06.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.157289+0000 osd.2 (osd.2) 324 : cluster 0 1.8 scrub ok 2024-04-02T05:00:06.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.184068+0000 osd.1 (osd.1) 357 : cluster 0 2.9 scrub starts 2024-04-02T05:00:06.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.185371+0000 osd.1 (osd.1) 358 : cluster 0 2.9 scrub ok 2024-04-02T05:00:06.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: audit 2024-04-02T05:00:06.053705+0000 mon.a (mon.0) 3468 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:06.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: cluster 2024-04-02T05:00:06.057505+0000 mon.a (mon.0) 3469 : cluster 0 osdmap e708: 8 total, 8 up, 7 in 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[28904]: audit 2024-04-02T05:00:06.061322+0000 mon.a (mon.0) 3470 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-114"}]: dispatch 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.155444+0000 osd.2 (osd.2) 323 : cluster 0 1.8 scrub starts 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.157289+0000 osd.2 (osd.2) 324 : cluster 0 1.8 scrub ok 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.184068+0000 osd.1 (osd.1) 357 : cluster 0 2.9 scrub starts 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.185371+0000 osd.1 (osd.1) 358 : cluster 0 2.9 scrub ok 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: audit 2024-04-02T05:00:06.053705+0000 mon.a (mon.0) 3468 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: cluster 2024-04-02T05:00:06.057505+0000 mon.a (mon.0) 3469 : cluster 0 osdmap e708: 8 total, 8 up, 7 in 2024-04-02T05:00:06.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:06 smithi069 bash[21528]: audit 2024-04-02T05:00:06.061322+0000 mon.a (mon.0) 3470 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-114"}]: dispatch 2024-04-02T05:00:06.246 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:06.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.155444+0000 osd.2 (osd.2) 323 : cluster 0 1.8 scrub starts 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.157289+0000 osd.2 (osd.2) 324 : cluster 0 1.8 scrub ok 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.184068+0000 osd.1 (osd.1) 357 : cluster 0 2.9 scrub starts 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.185371+0000 osd.1 (osd.1) 358 : cluster 0 2.9 scrub ok 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: audit 2024-04-02T05:00:06.053705+0000 mon.a (mon.0) 3468 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: cluster 2024-04-02T05:00:06.057505+0000 mon.a (mon.0) 3469 : cluster 0 osdmap e708: 8 total, 8 up, 7 in 2024-04-02T05:00:06.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:06 smithi098 bash[23790]: audit 2024-04-02T05:00:06.061322+0000 mon.a (mon.0) 3470 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-114"}]: dispatch 2024-04-02T05:00:06.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:06.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:06.633 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:06.684 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:00:06.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:06.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:06.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:06.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:07.016 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:07.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:07.145 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:07.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:07.273 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:07.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.457340+0000 osd.7 (osd.7) 163 : cluster 0 2.1d scrub starts 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.459271+0000 osd.7 (osd.7) 164 : cluster 0 2.1d scrub ok 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.816111+0000 osd.4 (osd.4) 360 : cluster 0 10.17 scrub starts 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: cluster 2024-04-02T05:00:05.822023+0000 osd.4 (osd.4) 361 : cluster 0 10.17 scrub ok 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: cluster 2024-04-02T05:00:06.274504+0000 mgr.y (mgr.14538) 639 : cluster 0 pgmap v1068: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 675 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: audit 2024-04-02T05:00:07.056475+0000 mon.a (mon.0) 3471 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-114"}]': finished 2024-04-02T05:00:07.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: cluster 2024-04-02T05:00:07.063718+0000 mon.a (mon.0) 3472 : cluster 0 osdmap e709: 8 total, 8 up, 7 in 2024-04-02T05:00:07.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:07 smithi098 bash[23790]: audit 2024-04-02T05:00:07.065350+0000 mon.a (mon.0) 3473 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-114", "mode": "writeback"}]: dispatch 2024-04-02T05:00:07.403 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:07.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:07.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.457340+0000 osd.7 (osd.7) 163 : cluster 0 2.1d scrub starts 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.459271+0000 osd.7 (osd.7) 164 : cluster 0 2.1d scrub ok 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.816111+0000 osd.4 (osd.4) 360 : cluster 0 10.17 scrub starts 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: cluster 2024-04-02T05:00:05.822023+0000 osd.4 (osd.4) 361 : cluster 0 10.17 scrub ok 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: cluster 2024-04-02T05:00:06.274504+0000 mgr.y (mgr.14538) 639 : cluster 0 pgmap v1068: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 675 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: audit 2024-04-02T05:00:07.056475+0000 mon.a (mon.0) 3471 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-114"}]': finished 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: cluster 2024-04-02T05:00:07.063718+0000 mon.a (mon.0) 3472 : cluster 0 osdmap e709: 8 total, 8 up, 7 in 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[21528]: audit 2024-04-02T05:00:07.065350+0000 mon.a (mon.0) 3473 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-114", "mode": "writeback"}]: dispatch 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.457340+0000 osd.7 (osd.7) 163 : cluster 0 2.1d scrub starts 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.459271+0000 osd.7 (osd.7) 164 : cluster 0 2.1d scrub ok 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.816111+0000 osd.4 (osd.4) 360 : cluster 0 10.17 scrub starts 2024-04-02T05:00:07.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: cluster 2024-04-02T05:00:05.822023+0000 osd.4 (osd.4) 361 : cluster 0 10.17 scrub ok 2024-04-02T05:00:07.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: cluster 2024-04-02T05:00:06.274504+0000 mgr.y (mgr.14538) 639 : cluster 0 pgmap v1068: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 675 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:07.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: audit 2024-04-02T05:00:07.056475+0000 mon.a (mon.0) 3471 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-114"}]': finished 2024-04-02T05:00:07.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: cluster 2024-04-02T05:00:07.063718+0000 mon.a (mon.0) 3472 : cluster 0 osdmap e709: 8 total, 8 up, 7 in 2024-04-02T05:00:07.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:07 smithi069 bash[28904]: audit 2024-04-02T05:00:07.065350+0000 mon.a (mon.0) 3473 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-114", "mode": "writeback"}]: dispatch 2024-04-02T05:00:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:07.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:07.657 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- 172.21.15.69:0/137927765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 msgr2=0x7fa87810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- 172.21.15.69:0/137927765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa87810dfd0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7fa870009f90 tx=0x7fa87000b330 comp rx=0 tx=0).stop 2024-04-02T05:00:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- 172.21.15.69:0/137927765 shutdown_connections 2024-04-02T05:00:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- 172.21.15.69:0/137927765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa87810dfd0 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- 172.21.15.69:0/137927765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 0x7fa878107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:07.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- 172.21.15.69:0/137927765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8781011a0 0x7fa878101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- 172.21.15.69:0/137927765 >> 172.21.15.69:0/137927765 conn(0x7fa8780771e0 msgr2=0x7fa8780775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- 172.21.15.69:0/137927765 shutdown_connections 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- 172.21.15.69:0/137927765 wait complete. 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 librados: starting msgr at 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 librados: starting objecter 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 Processor -- start 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- start start 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 librados: setting wanted keys 2024-04-02T05:00:07.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 librados: calling monclient init 2024-04-02T05:00:07.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8781011a0 0x7fa8780739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa878073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 0x7fa87806dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa87806c400 con 0x7fa878107220 2024-04-02T05:00:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa87806c280 con 0x7fa8781011a0 2024-04-02T05:00:07.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87ce01700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa87806c580 con 0x7fa878101b60 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa875d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa878073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 0x7fa87806dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa875d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa878073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:43592/0 (socket says 172.21.15.69:43592) 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa875d9b700 1 -- 172.21.15.69:0/3723718007 learned_addr learned my addr 172.21.15.69:0/3723718007 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa87659c700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8781011a0 0x7fa8780739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 -- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 msgr2=0x7fa878073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa878073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 -- 172.21.15.69:0/3723718007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8781011a0 msgr2=0x7fa8780739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8781011a0 0x7fa8780739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 -- 172.21.15.69:0/3723718007 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87806e550 con 0x7fa878107220 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa875d9b700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa878073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa876d9d700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 0x7fa87806dc50 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fa870009790 tx=0x7fa87000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa8677fe700 10 client.?.objecter ms_handle_connect 0x7fa878107220 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa8677fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:07.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.757+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870021070 con 0x7fa878107220 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87806e7e0 con 0x7fa878107220 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa87001d450 con 0x7fa878107220 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8700052a0 con 0x7fa878107220 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa87ce01700 10 client.33591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87806a1f0 con 0x7fa878107220 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa87ce01700 1 librados: init done 2024-04-02T05:00:07.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa878102860 con 0x7fa878107220 2024-04-02T05:00:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa870024070 con 0x7fa878107220 2024-04-02T05:00:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa86006f5e0 0x7fa860071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(709..709 src has 74..709) ==== 9962+0+0 (secure 0 0 0) 0x7fa87000f020 con 0x7fa878107220 2024-04-02T05:00:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 10 client.33591.objecter ms_dispatch 0x7fa878001550 osd_map(709..709 src has 74..709) 2024-04-02T05:00:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 3 client.33591.objecter handle_osd_map got epochs [709,709] > 0 2024-04-02T05:00:07.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 3 client.33591.objecter handle_osd_map decoding full epoch 709 2024-04-02T05:00:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa87659c700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa86006f5e0 0x7fa860071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.761+0000 7fa8677fe700 20 client.33591.objecter dump_active .. 0 homeless 2024-04-02T05:00:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.765+0000 7fa8677fe700 10 client.33591.objecter ms_handle_connect 0x7fa86006f5e0 2024-04-02T05:00:07.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.765+0000 7fa87659c700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa86006f5e0 0x7fa860071aa0 secure :-1 s=READY pgs=3989 cs=0 l=1 rev1=1 crypto rx=0x7fa868009750 tx=0x7fa868009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:07.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:07.769+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa87005cc50 con 0x7fa878107220 2024-04-02T05:00:07.791 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:07.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:07.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:07.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:08.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.001+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fa87806a4e0 con 0x7fa878107220 2024-04-02T05:00:08.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:08.039 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:08.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.057+0000 7fa8677fe700 1 -- 172.21.15.69:0/3723718007 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v710) ==== 101+0+0 (secure 0 0 0) 0x7fa87001d600 con 0x7fa878107220 2024-04-02T05:00:08.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.057+0000 7fa87ce01700 10 librados: watch_flush enter 2024-04-02T05:00:08.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.057+0000 7fa87ce01700 10 librados: watch_flush exit 2024-04-02T05:00:08.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.057+0000 7fa87ce01700 20 client.33591.objecter shutdown clearing up homeless session... 2024-04-02T05:00:08.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.057+0000 7fa87ce01700 10 client.33591.objecter successfully canceled tick 2024-04-02T05:00:08.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.061+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa86006f5e0 msgr2=0x7fa860071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:08.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.061+0000 7fa87ce01700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa86006f5e0 0x7fa860071aa0 secure :-1 s=READY pgs=3989 cs=0 l=1 rev1=1 crypto rx=0x7fa868009750 tx=0x7fa868009310 comp rx=0 tx=0).stop 2024-04-02T05:00:08.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.061+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 msgr2=0x7fa87806dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:08.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.061+0000 7fa87ce01700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 0x7fa87806dc50 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fa870009790 tx=0x7fa87000ccb0 comp rx=0 tx=0).stop 2024-04-02T05:00:08.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 shutdown_connections 2024-04-02T05:00:08.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa86006f5e0 0x7fa860071aa0 unknown :-1 s=CLOSED pgs=3989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:08.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa878107220 0x7fa87806dc50 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:08.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8781011a0 0x7fa8780739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:08.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 --2- 172.21.15.69:0/3723718007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa878101b60 0x7fa878073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:08.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 >> 172.21.15.69:0/3723718007 conn(0x7fa8780771e0 msgr2=0x7fa878107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 shutdown_connections 2024-04-02T05:00:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 -- 172.21.15.69:0/3723718007 wait complete. 2024-04-02T05:00:08.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:08.065+0000 7fa87ce01700 1 librados: shutdown 2024-04-02T05:00:08.068 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T05:00:08.142 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:08.159 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:08.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: cluster 2024-04-02T05:00:06.955870+0000 osd.6 (osd.6) 299 : cluster 0 2.7 scrub starts 2024-04-02T05:00:08.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: cluster 2024-04-02T05:00:06.958512+0000 osd.6 (osd.6) 300 : cluster 0 2.7 scrub ok 2024-04-02T05:00:08.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: audit 2024-04-02T05:00:08.004284+0000 mon.a (mon.0) 3474 : audit 1 from='client.? 172.21.15.69:0/3723718007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:00:08.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.056505+0000 mon.a (mon.0) 3475 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:08.160 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.056533+0000 mon.a (mon.0) 3476 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: audit 2024-04-02T05:00:08.059899+0000 mon.a (mon.0) 3477 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-114", "mode": "writeback"}]': finished 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: audit 2024-04-02T05:00:08.059977+0000 mon.a (mon.0) 3478 : audit 1 from='client.? 172.21.15.69:0/3723718007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.065917+0000 mon.a (mon.0) 3479 : cluster 0 osdmap e710: 8 total, 8 up, 7 in 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21528]: audit 2024-04-02T05:00:08.128906+0000 mon.a (mon.0) 3480 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: cluster 2024-04-02T05:00:06.955870+0000 osd.6 (osd.6) 299 : cluster 0 2.7 scrub starts 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: cluster 2024-04-02T05:00:06.958512+0000 osd.6 (osd.6) 300 : cluster 0 2.7 scrub ok 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: audit 2024-04-02T05:00:08.004284+0000 mon.a (mon.0) 3474 : audit 1 from='client.? 172.21.15.69:0/3723718007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.056505+0000 mon.a (mon.0) 3475 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.056533+0000 mon.a (mon.0) 3476 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:00:08.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: audit 2024-04-02T05:00:08.059899+0000 mon.a (mon.0) 3477 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-114", "mode": "writeback"}]': finished 2024-04-02T05:00:08.162 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: audit 2024-04-02T05:00:08.059977+0000 mon.a (mon.0) 3478 : audit 1 from='client.? 172.21.15.69:0/3723718007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:00:08.162 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.065917+0000 mon.a (mon.0) 3479 : cluster 0 osdmap e710: 8 total, 8 up, 7 in 2024-04-02T05:00:08.162 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[28904]: audit 2024-04-02T05:00:08.128906+0000 mon.a (mon.0) 3480 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:08.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:08.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:08.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:08.470 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: cluster 2024-04-02T05:00:06.955870+0000 osd.6 (osd.6) 299 : cluster 0 2.7 scrub starts 2024-04-02T05:00:08.470 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: cluster 2024-04-02T05:00:06.958512+0000 osd.6 (osd.6) 300 : cluster 0 2.7 scrub ok 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: audit 2024-04-02T05:00:08.004284+0000 mon.a (mon.0) 3474 : audit 1 from='client.? 172.21.15.69:0/3723718007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.056505+0000 mon.a (mon.0) 3475 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.056533+0000 mon.a (mon.0) 3476 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: audit 2024-04-02T05:00:08.059899+0000 mon.a (mon.0) 3477 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-114", "mode": "writeback"}]': finished 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: audit 2024-04-02T05:00:08.059977+0000 mon.a (mon.0) 3478 : audit 1 from='client.? 172.21.15.69:0/3723718007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.065917+0000 mon.a (mon.0) 3479 : cluster 0 osdmap e710: 8 total, 8 up, 7 in 2024-04-02T05:00:08.471 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:08 smithi098 bash[23790]: audit 2024-04-02T05:00:08.128906+0000 mon.a (mon.0) 3480 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:08.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:08.549 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:08.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:08.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:08.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:08.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:08.920 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:00:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:00:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:00:08.932 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:08.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:09.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:09.063 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:09.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:09.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:09.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:09.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:09 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.275932+0000 mgr.y (mgr.14538) 640 : cluster 0 pgmap v1071: 214 pgs: 7 unknown, 207 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T05:00:09.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:09 smithi098 bash[23790]: audit 2024-04-02T05:00:09.062922+0000 mon.a (mon.0) 3481 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:09.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:09 smithi098 bash[23790]: cluster 2024-04-02T05:00:09.076451+0000 mon.a (mon.0) 3482 : cluster 0 osdmap e711: 8 total, 8 up, 7 in 2024-04-02T05:00:09.377 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:09 smithi098 bash[23790]: audit 2024-04-02T05:00:09.078739+0000 mon.a (mon.0) 3483 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114"}]: dispatch 2024-04-02T05:00:09.425 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.275932+0000 mgr.y (mgr.14538) 640 : cluster 0 pgmap v1071: 214 pgs: 7 unknown, 207 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[28904]: audit 2024-04-02T05:00:09.062922+0000 mon.a (mon.0) 3481 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[28904]: cluster 2024-04-02T05:00:09.076451+0000 mon.a (mon.0) 3482 : cluster 0 osdmap e711: 8 total, 8 up, 7 in 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[28904]: audit 2024-04-02T05:00:09.078739+0000 mon.a (mon.0) 3483 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114"}]: dispatch 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.275932+0000 mgr.y (mgr.14538) 640 : cluster 0 pgmap v1071: 214 pgs: 7 unknown, 207 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[21528]: audit 2024-04-02T05:00:09.062922+0000 mon.a (mon.0) 3481 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[21528]: cluster 2024-04-02T05:00:09.076451+0000 mon.a (mon.0) 3482 : cluster 0 osdmap e711: 8 total, 8 up, 7 in 2024-04-02T05:00:09.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:09 smithi069 bash[21528]: audit 2024-04-02T05:00:09.078739+0000 mon.a (mon.0) 3483 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114"}]: dispatch 2024-04-02T05:00:09.451 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:09.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:09.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:09.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:09.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:09.840 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:09.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:09.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:10.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:10.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.517995+0000 osd.7 (osd.7) 165 : cluster 0 3.13 scrub starts 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.520546+0000 osd.7 (osd.7) 166 : cluster 0 3.13 scrub ok 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.900312+0000 osd.4 (osd.4) 362 : cluster 0 4.c scrub starts 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:08.902486+0000 osd.4 (osd.4) 363 : cluster 0 4.c scrub ok 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.031501+0000 mon.a (mon.0) 3484 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.063218+0000 mon.a (mon.0) 3485 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: audit 2024-04-02T05:00:10.067041+0000 mon.a (mon.0) 3486 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114"}]': finished 2024-04-02T05:00:10.216 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:10 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.073773+0000 mon.a (mon.0) 3487 : cluster 0 osdmap e712: 8 total, 8 up, 7 in 2024-04-02T05:00:10.230 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:10.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 -- 172.21.15.69:0/2869288116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8103f60 msgr2=0x7f70c81043e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 --2- 172.21.15.69:0/2869288116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8103f60 0x7f70c81043e0 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009990 tx=0x7f70b8009ca0 comp rx=0 tx=0).stop 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 -- 172.21.15.69:0/2869288116 shutdown_connections 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 --2- 172.21.15.69:0/2869288116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8104920 0x7f70c810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 --2- 172.21.15.69:0/2869288116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c806ac50 0x7f70c806b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 --2- 172.21.15.69:0/2869288116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8103f60 0x7f70c81043e0 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 -- 172.21.15.69:0/2869288116 >> 172.21.15.69:0/2869288116 conn(0x7f70c80771e0 msgr2=0x7f70c80775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:10.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 -- 172.21.15.69:0/2869288116 shutdown_connections 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 -- 172.21.15.69:0/2869288116 wait complete. 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 librados: starting msgr at 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 librados: starting objecter 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.453+0000 7f70cd93c700 1 Processor -- start 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- start start 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 librados: setting wanted keys 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 librados: calling monclient init 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c806ac50 0x7f70c819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8103f60 0x7f70c819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8104920 0x7f70c819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8111c40 con 0x7f70c8104920 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8111ac0 con 0x7f70c806ac50 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f70c8111dc0 con 0x7f70c8103f60 2024-04-02T05:00:10.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8104920 0x7f70c819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8103f60 0x7f70c819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8104920 0x7f70c819ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42454/0 (socket says 172.21.15.69:42454) 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c77fe700 1 -- 172.21.15.69:0/153164161 learned_addr learned my addr 172.21.15.69:0/153164161 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c6ffd700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c806ac50 0x7f70c819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 -- 172.21.15.69:0/153164161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c806ac50 msgr2=0x7f70c819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c806ac50 0x7f70c819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 -- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8104920 msgr2=0x7f70c819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8104920 0x7f70c819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 -- 172.21.15.69:0/153164161 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c819f370 con 0x7f70c8103f60 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c67fc700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8103f60 0x7f70c819a8a0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f70b8006010 tx=0x7f70b80057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 10 client.?.objecter ms_handle_connect 0x7f70c8103f60 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8027070 con 0x7f70c8103f60 2024-04-02T05:00:10.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c819f600 con 0x7f70c8103f60 2024-04-02T05:00:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f70b8022470 con 0x7f70c8103f60 2024-04-02T05:00:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8017650 con 0x7f70c8103f60 2024-04-02T05:00:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 10 client.43661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c81a6ee0 con 0x7f70c8103f60 2024-04-02T05:00:10.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 librados: init done 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c804ff00 con 0x7f70c8103f60 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f70b802a070 con 0x7f70c8103f60 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f70b406f5e0 0x7f70b4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(712..712 src has 74..712) ==== 9954+0+0 (secure 0 0 0) 0x7f70b808f890 con 0x7f70c8103f60 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70c6ffd700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f70b406f5e0 0x7f70b4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 10 client.43661.objecter ms_dispatch 0x7f70c8001550 osd_map(712..712 src has 74..712) 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 3 client.43661.objecter handle_osd_map got epochs [712,712] > 0 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 3 client.43661.objecter handle_osd_map decoding full epoch 712 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.457+0000 7f70cc93a700 20 client.43661.objecter dump_active .. 0 homeless 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.461+0000 7f70cc93a700 10 client.43661.objecter ms_handle_connect 0x7f70b406f5e0 2024-04-02T05:00:10.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.461+0000 7f70c6ffd700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f70b406f5e0 0x7f70b4071aa0 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7f70b0005950 tx=0x7f70b000b410 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:10.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:10.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.465+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f70b8013070 con 0x7f70c8103f60 2024-04-02T05:00:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:10.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.517995+0000 osd.7 (osd.7) 165 : cluster 0 3.13 scrub starts 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.520546+0000 osd.7 (osd.7) 166 : cluster 0 3.13 scrub ok 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.900312+0000 osd.4 (osd.4) 362 : cluster 0 4.c scrub starts 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:08.902486+0000 osd.4 (osd.4) 363 : cluster 0 4.c scrub ok 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.031501+0000 mon.a (mon.0) 3484 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.063218+0000 mon.a (mon.0) 3485 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: audit 2024-04-02T05:00:10.067041+0000 mon.a (mon.0) 3486 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114"}]': finished 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.073773+0000 mon.a (mon.0) 3487 : cluster 0 osdmap e712: 8 total, 8 up, 7 in 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.517995+0000 osd.7 (osd.7) 165 : cluster 0 3.13 scrub starts 2024-04-02T05:00:10.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.520546+0000 osd.7 (osd.7) 166 : cluster 0 3.13 scrub ok 2024-04-02T05:00:10.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.900312+0000 osd.4 (osd.4) 362 : cluster 0 4.c scrub starts 2024-04-02T05:00:10.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:08.902486+0000 osd.4 (osd.4) 363 : cluster 0 4.c scrub ok 2024-04-02T05:00:10.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.031501+0000 mon.a (mon.0) 3484 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:10.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.063218+0000 mon.a (mon.0) 3485 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:10.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: audit 2024-04-02T05:00:10.067041+0000 mon.a (mon.0) 3486 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-114"}]': finished 2024-04-02T05:00:10.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:10 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.073773+0000 mon.a (mon.0) 3487 : cluster 0 osdmap e712: 8 total, 8 up, 7 in 2024-04-02T05:00:10.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:10.609 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:10.626 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T05:00:10.626 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:00:10.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:10.689+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]} v 0) -- 0x7f70c81a70b0 con 0x7f70c8103f60 2024-04-02T05:00:10.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:10.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:10.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:10.992 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:10.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:11.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.077+0000 7f70cc93a700 1 -- 172.21.15.69:0/153164161 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]=0 set 10.8 pg_upmap mapping to [5,1,7] v713) ==== 129+0+0 (secure 0 0 0) 0x7f70b805c4b0 con 0x7f70c8103f60 2024-04-02T05:00:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.081+0000 7f70cd93c700 10 librados: watch_flush enter 2024-04-02T05:00:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.081+0000 7f70cd93c700 10 librados: watch_flush exit 2024-04-02T05:00:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.081+0000 7f70cd93c700 20 client.43661.objecter shutdown clearing up homeless session... 2024-04-02T05:00:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.081+0000 7f70cd93c700 10 client.43661.objecter successfully canceled tick 2024-04-02T05:00:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f70b406f5e0 msgr2=0x7f70b4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f70b406f5e0 0x7f70b4071aa0 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7f70b0005950 tx=0x7f70b000b410 comp rx=0 tx=0).stop 2024-04-02T05:00:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8103f60 msgr2=0x7f70c819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8103f60 0x7f70c819a8a0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f70b8006010 tx=0x7f70b80057c0 comp rx=0 tx=0).stop 2024-04-02T05:00:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 shutdown_connections 2024-04-02T05:00:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f70b406f5e0 0x7f70b4071aa0 secure :-1 s=CLOSED pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7f70b0005950 tx=0x7f70b000b410 comp rx=0 tx=0).stop 2024-04-02T05:00:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8104920 0x7f70c819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c806ac50 0x7f70c819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 --2- 172.21.15.69:0/153164161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c8103f60 0x7f70c819a8a0 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.085+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 >> 172.21.15.69:0/153164161 conn(0x7f70c80771e0 msgr2=0x7f70c810f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.089+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 shutdown_connections 2024-04-02T05:00:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.089+0000 7f70cd93c700 1 -- 172.21.15.69:0/153164161 wait complete. 2024-04-02T05:00:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:11.089+0000 7f70cd93c700 1 librados: shutdown 2024-04-02T05:00:11.091 INFO:teuthology.orchestra.run.smithi069.stderr:set 10.8 pg_upmap mapping to [5,1,7] 2024-04-02T05:00:11.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:11.118 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:11.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:11.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:11.375 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:11.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:11.502 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:11.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-02T05:00:11.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:11.632 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:11.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:09.833461+0000 osd.3 (osd.3) 349 : cluster 0 6.7 scrub starts 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:09.835144+0000 osd.3 (osd.3) 350 : cluster 0 6.7 scrub ok 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:09.871305+0000 osd.4 (osd.4) 364 : cluster 0 10.1d scrub starts 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:09.877274+0000 osd.4 (osd.4) 365 : cluster 0 10.1d scrub ok 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.276479+0000 mgr.y (mgr.14538) 641 : cluster 0 pgmap v1074: 214 pgs: 214 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.317064+0000 osd.1 (osd.1) 359 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.317329+0000 osd.1 (osd.1) 360 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: audit 2024-04-02T05:00:10.694328+0000 mon.c (mon.2) 1148 : audit 1 from='client.? 172.21.15.69:0/153164161' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]: dispatch 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: audit 2024-04-02T05:00:10.694804+0000 mon.a (mon.0) 3488 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]: dispatch 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: audit 2024-04-02T05:00:11.069695+0000 mon.a (mon.0) 3489 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]': finished 2024-04-02T05:00:11.744 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:11 smithi098 bash[23790]: cluster 2024-04-02T05:00:11.073878+0000 mon.a (mon.0) 3490 : cluster 0 osdmap e713: 8 total, 8 up, 7 in 2024-04-02T05:00:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:11.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:11.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:09.833461+0000 osd.3 (osd.3) 349 : cluster 0 6.7 scrub starts 2024-04-02T05:00:11.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:09.835144+0000 osd.3 (osd.3) 350 : cluster 0 6.7 scrub ok 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:09.871305+0000 osd.4 (osd.4) 364 : cluster 0 10.1d scrub starts 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:09.877274+0000 osd.4 (osd.4) 365 : cluster 0 10.1d scrub ok 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.276479+0000 mgr.y (mgr.14538) 641 : cluster 0 pgmap v1074: 214 pgs: 214 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.317064+0000 osd.1 (osd.1) 359 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.317329+0000 osd.1 (osd.1) 360 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: audit 2024-04-02T05:00:10.694328+0000 mon.c (mon.2) 1148 : audit 1 from='client.? 172.21.15.69:0/153164161' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]: dispatch 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: audit 2024-04-02T05:00:10.694804+0000 mon.a (mon.0) 3488 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]: dispatch 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: audit 2024-04-02T05:00:11.069695+0000 mon.a (mon.0) 3489 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]': finished 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[21528]: cluster 2024-04-02T05:00:11.073878+0000 mon.a (mon.0) 3490 : cluster 0 osdmap e713: 8 total, 8 up, 7 in 2024-04-02T05:00:11.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:09.833461+0000 osd.3 (osd.3) 349 : cluster 0 6.7 scrub starts 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:09.835144+0000 osd.3 (osd.3) 350 : cluster 0 6.7 scrub ok 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:09.871305+0000 osd.4 (osd.4) 364 : cluster 0 10.1d scrub starts 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:09.877274+0000 osd.4 (osd.4) 365 : cluster 0 10.1d scrub ok 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.276479+0000 mgr.y (mgr.14538) 641 : cluster 0 pgmap v1074: 214 pgs: 214 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.317064+0000 osd.1 (osd.1) 359 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.317329+0000 osd.1 (osd.1) 360 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: audit 2024-04-02T05:00:10.694328+0000 mon.c (mon.2) 1148 : audit 1 from='client.? 172.21.15.69:0/153164161' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]: dispatch 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: audit 2024-04-02T05:00:10.694804+0000 mon.a (mon.0) 3488 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]: dispatch 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: audit 2024-04-02T05:00:11.069695+0000 mon.a (mon.0) 3489 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "10.8", "id": [5, 1, 7]}]': finished 2024-04-02T05:00:11.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:11 smithi069 bash[28904]: cluster 2024-04-02T05:00:11.073878+0000 mon.a (mon.0) 3490 : cluster 0 osdmap e713: 8 total, 8 up, 7 in 2024-04-02T05:00:11.884 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:11.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:11.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T05:00:11.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:12.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:12.131 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:12.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:12.253 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:12.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:12.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:12.519 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:12.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:12.641 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.262181+0000 osd.2 (osd.2) 325 : cluster 0 6.f scrub starts 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:10.264844+0000 osd.2 (osd.2) 326 : cluster 0 6.f scrub ok 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:11.174833+0000 osd.1 (osd.1) 361 : cluster 0 4.4 scrub starts 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:11.176067+0000 osd.1 (osd.1) 362 : cluster 0 4.4 scrub ok 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:11.929486+0000 osd.6 (osd.6) 301 : cluster 0 4.1f scrub starts 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:11.932019+0000 osd.6 (osd.6) 302 : cluster 0 4.1f scrub ok 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.076389+0000 mon.a (mon.0) 3491 : cluster 0 osdmap e714: 8 total, 8 up, 7 in 2024-04-02T05:00:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:12 smithi098 bash[23790]: audit 2024-04-02T05:00:12.078599+0000 mon.a (mon.0) 3492 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:12.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:12.771 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:12.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:12.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:13.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.262181+0000 osd.2 (osd.2) 325 : cluster 0 6.f scrub starts 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:10.264844+0000 osd.2 (osd.2) 326 : cluster 0 6.f scrub ok 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:11.174833+0000 osd.1 (osd.1) 361 : cluster 0 4.4 scrub starts 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:11.176067+0000 osd.1 (osd.1) 362 : cluster 0 4.4 scrub ok 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:11.929486+0000 osd.6 (osd.6) 301 : cluster 0 4.1f scrub starts 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:11.932019+0000 osd.6 (osd.6) 302 : cluster 0 4.1f scrub ok 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.076389+0000 mon.a (mon.0) 3491 : cluster 0 osdmap e714: 8 total, 8 up, 7 in 2024-04-02T05:00:13.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[28904]: audit 2024-04-02T05:00:12.078599+0000 mon.a (mon.0) 3492 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.262181+0000 osd.2 (osd.2) 325 : cluster 0 6.f scrub starts 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:10.264844+0000 osd.2 (osd.2) 326 : cluster 0 6.f scrub ok 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:11.174833+0000 osd.1 (osd.1) 361 : cluster 0 4.4 scrub starts 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:11.176067+0000 osd.1 (osd.1) 362 : cluster 0 4.4 scrub ok 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:11.929486+0000 osd.6 (osd.6) 301 : cluster 0 4.1f scrub starts 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:11.932019+0000 osd.6 (osd.6) 302 : cluster 0 4.1f scrub ok 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.076389+0000 mon.a (mon.0) 3491 : cluster 0 osdmap e714: 8 total, 8 up, 7 in 2024-04-02T05:00:13.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:12 smithi069 bash[21528]: audit 2024-04-02T05:00:12.078599+0000 mon.a (mon.0) 3492 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:13.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:13.159 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:13.262 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:13.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:13.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:13.548 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:13.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:13.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.277115+0000 mgr.y (mgr.14538) 642 : cluster 0 pgmap v1077: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.540213+0000 osd.7 (osd.7) 167 : cluster 0 6.1c scrub starts 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.541928+0000 osd.7 (osd.7) 168 : cluster 0 6.1c scrub ok 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.829034+0000 osd.0 (osd.0) 349 : cluster 0 10.6 scrub starts 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.831232+0000 osd.0 (osd.0) 350 : cluster 0 10.6 scrub ok 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: audit 2024-04-02T05:00:13.082949+0000 mon.a (mon.0) 3493 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:13.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:13 smithi098 bash[23790]: cluster 2024-04-02T05:00:13.087127+0000 mon.a (mon.0) 3494 : cluster 0 osdmap e715: 8 total, 8 up, 7 in 2024-04-02T05:00:13.678 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:13.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.277115+0000 mgr.y (mgr.14538) 642 : cluster 0 pgmap v1077: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.540213+0000 osd.7 (osd.7) 167 : cluster 0 6.1c scrub starts 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.541928+0000 osd.7 (osd.7) 168 : cluster 0 6.1c scrub ok 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.829034+0000 osd.0 (osd.0) 349 : cluster 0 10.6 scrub starts 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.831232+0000 osd.0 (osd.0) 350 : cluster 0 10.6 scrub ok 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: audit 2024-04-02T05:00:13.082949+0000 mon.a (mon.0) 3493 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[28904]: cluster 2024-04-02T05:00:13.087127+0000 mon.a (mon.0) 3494 : cluster 0 osdmap e715: 8 total, 8 up, 7 in 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.277115+0000 mgr.y (mgr.14538) 642 : cluster 0 pgmap v1077: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.540213+0000 osd.7 (osd.7) 167 : cluster 0 6.1c scrub starts 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.541928+0000 osd.7 (osd.7) 168 : cluster 0 6.1c scrub ok 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.829034+0000 osd.0 (osd.0) 349 : cluster 0 10.6 scrub starts 2024-04-02T05:00:13.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.831232+0000 osd.0 (osd.0) 350 : cluster 0 10.6 scrub ok 2024-04-02T05:00:13.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: audit 2024-04-02T05:00:13.082949+0000 mon.a (mon.0) 3493 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:13.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:13 smithi069 bash[21528]: cluster 2024-04-02T05:00:13.087127+0000 mon.a (mon.0) 3494 : cluster 0 osdmap e715: 8 total, 8 up, 7 in 2024-04-02T05:00:13.806 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:13.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:13.929 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:13.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:14.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:14.060 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:14.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:14.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:14.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:14.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:14.678 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:14.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.800323+0000 osd.3 (osd.3) 351 : cluster 0 4.10 deep-scrub starts 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: cluster 2024-04-02T05:00:12.801831+0000 osd.3 (osd.3) 352 : cluster 0 4.10 deep-scrub ok 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: cluster 2024-04-02T05:00:13.213278+0000 osd.1 (osd.1) 363 : cluster 0 6.e scrub starts 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: cluster 2024-04-02T05:00:13.215939+0000 osd.1 (osd.1) 364 : cluster 0 6.e scrub ok 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: audit 2024-04-02T05:00:13.591622+0000 mgr.y (mgr.14538) 643 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: cluster 2024-04-02T05:00:14.096337+0000 mon.a (mon.0) 3495 : cluster 0 osdmap e716: 8 total, 8 up, 7 in 2024-04-02T05:00:14.820 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:14 smithi098 bash[23790]: audit 2024-04-02T05:00:14.129188+0000 mon.a (mon.0) 3496 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:14.834 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:14.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:14.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.800323+0000 osd.3 (osd.3) 351 : cluster 0 4.10 deep-scrub starts 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: cluster 2024-04-02T05:00:12.801831+0000 osd.3 (osd.3) 352 : cluster 0 4.10 deep-scrub ok 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: cluster 2024-04-02T05:00:13.213278+0000 osd.1 (osd.1) 363 : cluster 0 6.e scrub starts 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: cluster 2024-04-02T05:00:13.215939+0000 osd.1 (osd.1) 364 : cluster 0 6.e scrub ok 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: audit 2024-04-02T05:00:13.591622+0000 mgr.y (mgr.14538) 643 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: cluster 2024-04-02T05:00:14.096337+0000 mon.a (mon.0) 3495 : cluster 0 osdmap e716: 8 total, 8 up, 7 in 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[28904]: audit 2024-04-02T05:00:14.129188+0000 mon.a (mon.0) 3496 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.800323+0000 osd.3 (osd.3) 351 : cluster 0 4.10 deep-scrub starts 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: cluster 2024-04-02T05:00:12.801831+0000 osd.3 (osd.3) 352 : cluster 0 4.10 deep-scrub ok 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: cluster 2024-04-02T05:00:13.213278+0000 osd.1 (osd.1) 363 : cluster 0 6.e scrub starts 2024-04-02T05:00:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: cluster 2024-04-02T05:00:13.215939+0000 osd.1 (osd.1) 364 : cluster 0 6.e scrub ok 2024-04-02T05:00:14.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: audit 2024-04-02T05:00:13.591622+0000 mgr.y (mgr.14538) 643 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:14.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: cluster 2024-04-02T05:00:14.096337+0000 mon.a (mon.0) 3495 : cluster 0 osdmap e716: 8 total, 8 up, 7 in 2024-04-02T05:00:14.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:14 smithi069 bash[21528]: audit 2024-04-02T05:00:14.129188+0000 mon.a (mon.0) 3496 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:14.964 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:14.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:15.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:15.091 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:15.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:15.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:15.350 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:15.453 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:15.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:15.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:15.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:15.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:13.769964+0000 osd.3 (osd.3) 353 : cluster 0 10.15 scrub starts 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:13.773994+0000 osd.3 (osd.3) 354 : cluster 0 10.15 scrub ok 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:14.277905+0000 mgr.y (mgr.14538) 644 : cluster 0 pgmap v1080: 214 pgs: 6 creating+activating, 17 creating+peering, 191 active+clean; 582 KiB data, 678 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:14.894286+0000 osd.0 (osd.0) 351 : cluster 0 4.1e scrub starts 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:14.896331+0000 osd.0 (osd.0) 352 : cluster 0 4.1e scrub ok 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.032697+0000 mon.a (mon.0) 3497 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: audit 2024-04-02T05:00:15.035523+0000 mon.a (mon.0) 3498 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.043525+0000 mon.a (mon.0) 3499 : cluster 0 osdmap e717: 8 total, 8 up, 7 in 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[28904]: audit 2024-04-02T05:00:15.047178+0000 mon.a (mon.0) 3500 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-116"}]: dispatch 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:13.769964+0000 osd.3 (osd.3) 353 : cluster 0 10.15 scrub starts 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:13.773994+0000 osd.3 (osd.3) 354 : cluster 0 10.15 scrub ok 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:14.277905+0000 mgr.y (mgr.14538) 644 : cluster 0 pgmap v1080: 214 pgs: 6 creating+activating, 17 creating+peering, 191 active+clean; 582 KiB data, 678 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-04-02T05:00:15.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:14.894286+0000 osd.0 (osd.0) 351 : cluster 0 4.1e scrub starts 2024-04-02T05:00:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:14.896331+0000 osd.0 (osd.0) 352 : cluster 0 4.1e scrub ok 2024-04-02T05:00:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.032697+0000 mon.a (mon.0) 3497 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: audit 2024-04-02T05:00:15.035523+0000 mon.a (mon.0) 3498 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.043525+0000 mon.a (mon.0) 3499 : cluster 0 osdmap e717: 8 total, 8 up, 7 in 2024-04-02T05:00:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:15 smithi069 bash[21528]: audit 2024-04-02T05:00:15.047178+0000 mon.a (mon.0) 3500 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-116"}]: dispatch 2024-04-02T05:00:15.739 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:15.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:15.867 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:15.971 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:13.769964+0000 osd.3 (osd.3) 353 : cluster 0 10.15 scrub starts 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:13.773994+0000 osd.3 (osd.3) 354 : cluster 0 10.15 scrub ok 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:14.277905+0000 mgr.y (mgr.14538) 644 : cluster 0 pgmap v1080: 214 pgs: 6 creating+activating, 17 creating+peering, 191 active+clean; 582 KiB data, 678 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:14.894286+0000 osd.0 (osd.0) 351 : cluster 0 4.1e scrub starts 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:14.896331+0000 osd.0 (osd.0) 352 : cluster 0 4.1e scrub ok 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.032697+0000 mon.a (mon.0) 3497 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: audit 2024-04-02T05:00:15.035523+0000 mon.a (mon.0) 3498 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.043525+0000 mon.a (mon.0) 3499 : cluster 0 osdmap e717: 8 total, 8 up, 7 in 2024-04-02T05:00:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:15 smithi098 bash[23790]: audit 2024-04-02T05:00:15.047178+0000 mon.a (mon.0) 3500 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-116"}]: dispatch 2024-04-02T05:00:15.991 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:15.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:16.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- 172.21.15.69:0/649467936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 msgr2=0x7fa0c010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- 172.21.15.69:0/649467936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 0x7fa0c010dfd0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009f90 tx=0x7fa0b800b330 comp rx=0 tx=0).stop 2024-04-02T05:00:16.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- 172.21.15.69:0/649467936 shutdown_connections 2024-04-02T05:00:16.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- 172.21.15.69:0/649467936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 0x7fa0c010dfd0 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- 172.21.15.69:0/649467936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c0107220 0x7fa0c0107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- 172.21.15.69:0/649467936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c01011a0 0x7fa0c0101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- 172.21.15.69:0/649467936 >> 172.21.15.69:0/649467936 conn(0x7fa0c00771e0 msgr2=0x7fa0c00775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:16.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- 172.21.15.69:0/649467936 shutdown_connections 2024-04-02T05:00:16.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- 172.21.15.69:0/649467936 wait complete. 2024-04-02T05:00:16.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 librados: starting msgr at 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 librados: starting objecter 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 Processor -- start 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- start start 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 librados: setting wanted keys 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 librados: calling monclient init 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c01011a0 0x7fa0c0196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 0x7fa0c01971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c0107220 0x7fa0c0197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c01109d0 con 0x7fa0c01011a0 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c0110850 con 0x7fa0c0107220 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0c5081700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c0110b50 con 0x7fa0c0101b60 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0bed9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c01011a0 0x7fa0c0196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 0x7fa0c01971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0bed9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c01011a0 0x7fa0c0196c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42480/0 (socket says 172.21.15.69:42480) 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0bed9d700 1 -- 172.21.15.69:0/3131350509 learned_addr learned my addr 172.21.15.69:0/3131350509 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c0107220 0x7fa0c0197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0be59c700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c0107220 msgr2=0x7fa0c0197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c0107220 0x7fa0c0197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0be59c700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c01011a0 msgr2=0x7fa0c0196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c01011a0 0x7fa0c0196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0be59c700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c01a64a0 con 0x7fa0c0101b60 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.069+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0c01011a0 0x7fa0c0196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 0x7fa0c01971d0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400b480 tx=0x7fa0b400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 10 client.?.objecter ms_handle_connect 0x7fa0c0101b60 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4015020 con 0x7fa0c0101b60 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c01a6790 con 0x7fa0c0101b60 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa0b4011cb0 con 0x7fa0c0101b60 2024-04-02T05:00:16.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4010bd0 con 0x7fa0c0101b60 2024-04-02T05:00:16.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 10 client.43667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:16.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c01a6d50 con 0x7fa0c0101b60 2024-04-02T05:00:16.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 1 librados: init done 2024-04-02T05:00:16.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa0c004ff00 con 0x7fa0c0101b60 2024-04-02T05:00:16.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa0b4018020 con 0x7fa0c0101b60 2024-04-02T05:00:16.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0a806f5e0 0x7fa0a8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(718..718 src has 74..718) ==== 9995+0+0 (secure 0 0 0) 0x7fa0b408e0b0 con 0x7fa0c0101b60 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 osd_map(718..718 src has 74..718) 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0a806f5e0 0x7fa0a8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 3 client.43667.objecter handle_osd_map got epochs [718,718] > 0 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 3 client.43667.objecter handle_osd_map decoding full epoch 718 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 20 client.43667.objecter dump_active .. 0 homeless 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v718) ==== 54+0+16 (secure 0 0 0) 0x7fa0b401f020 con 0x7fa0c0101b60 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 10 client.43667.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 1 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa0c0062e50 0x7fa0c0063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 1 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 0 1 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 4 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 10 client.43667.objecter _send_command 1 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01aa1a0 con 0x7fa0c0062e50 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0a806f5e0 0x7fa0a8071aa0 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009750 tx=0x7fa0b0009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0a806f5e0 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.073+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa0c0062e50 0x7fa0c0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa0c0062e50 0x7fa0c0063230 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c0062e50 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01aa1a0 con 0x7fa0c0062e50 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 1: 0 ) 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 1 = system:0 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 4 2024-04-02T05:00:16.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.077+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 0 1 2024-04-02T05:00:16.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 10 client.43667.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 2 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 2 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c006a510 osd=0 3 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 0 2 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 4 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 10 client.43667.objecter _send_command 2 2024-04-02T05:00:16.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.093+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c00628d0 con 0x7fa0c0062e50 2024-04-02T05:00:16.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c00628d0 con 0x7fa0c0062e50 2024-04-02T05:00:16.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 4 2024-04-02T05:00:16.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 0 2 2024-04-02T05:00:16.099 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 10 client.43667.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa0c002c9b0 0x7fa0c002cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 1 3 2024-04-02T05:00:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 4 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 10 client.43667.objecter _send_command 3 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa0c0109e00 con 0x7fa0c002c9b0 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa0c002c9b0 0x7fa0c002cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c002c9b0 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa0c002c9b0 0x7fa0c002cd90 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c0109e00 con 0x7fa0c002c9b0 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 3: 0 ) 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 3 = system:0 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 4 2024-04-02T05:00:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.097+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 1 3 2024-04-02T05:00:16.115 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 10 client.43667.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 4 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 4 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c0109780 osd=1 3 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 1 4 2024-04-02T05:00:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 4 2024-04-02T05:00:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 10 client.43667.objecter _send_command 4 2024-04-02T05:00:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.121+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c0109200 con 0x7fa0c002c9b0 2024-04-02T05:00:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c0109200 con 0x7fa0c002c9b0 2024-04-02T05:00:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 4 2024-04-02T05:00:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 1 4 2024-04-02T05:00:16.127 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 10 client.43667.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 5 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa0c01ac360 0x7fa0c01ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 5 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 2 5 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 4 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 10 client.43667.objecter _send_command 5 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01b15b0 con 0x7fa0c01ac360 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa0c01ac360 0x7fa0c01ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c01ac360 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa0c01ac360 0x7fa0c01ae750 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01b15b0 con 0x7fa0c01ac360 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 5: 0 ) 2024-04-02T05:00:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 5 = system:0 2024-04-02T05:00:16.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 4 2024-04-02T05:00:16.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.125+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 2 5 2024-04-02T05:00:16.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 10 client.43667.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 6 2024-04-02T05:00:16.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 6 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c0109600 osd=2 3 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 2 6 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 4 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 10 client.43667.objecter _send_command 6 2024-04-02T05:00:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c002d520 con 0x7fa0c01ac360 2024-04-02T05:00:16.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c002d520 con 0x7fa0c01ac360 2024-04-02T05:00:16.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 4 2024-04-02T05:00:16.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 2 6 2024-04-02T05:00:16.148 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:00:16.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 10 client.43667.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 7 2024-04-02T05:00:16.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa0c01afe10 0x7fa0c01b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 7 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 3 7 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 4 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 10 client.43667.objecter _send_command 7 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01a9f20 con 0x7fa0c01afe10 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa0c01afe10 0x7fa0c01b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c01afe10 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa0c01afe10 0x7fa0c01b0250 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01a9f20 con 0x7fa0c01afe10 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 7: 0 ) 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 7 = system:0 2024-04-02T05:00:16.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 4 2024-04-02T05:00:16.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.145+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 3 7 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 10 client.43667.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 8 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 8 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01a97f0 osd=3 3 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 3 8 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 4 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 10 client.43667.objecter _send_command 8 2024-04-02T05:00:16.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c01a9270 con 0x7fa0c01afe10 2024-04-02T05:00:16.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c01a9270 con 0x7fa0c01afe10 2024-04-02T05:00:16.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 4 2024-04-02T05:00:16.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.165+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 3 8 2024-04-02T05:00:16.170 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:00:16.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 10 client.43667.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 9 2024-04-02T05:00:16.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa0c01b3770 0x7fa0c01b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 9 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 4 9 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 4 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 10 client.43667.objecter _send_command 9 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01a9190 con 0x7fa0c01b3770 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa0c01b3770 0x7fa0c01b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c01b3770 2024-04-02T05:00:16.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa0c01b3770 0x7fa0c01b5b60 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01a9190 con 0x7fa0c01b3770 2024-04-02T05:00:16.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 9: 0 ) 2024-04-02T05:00:16.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 9 = system:0 2024-04-02T05:00:16.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 4 2024-04-02T05:00:16.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.169+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 4 9 2024-04-02T05:00:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 10 client.43667.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 10 2024-04-02T05:00:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 10 2024-04-02T05:00:16.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01a9020 osd=4 3 2024-04-02T05:00:16.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 4 10 2024-04-02T05:00:16.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 4 2024-04-02T05:00:16.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 10 client.43667.objecter _send_command 10 2024-04-02T05:00:16.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.189+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c01a9190 con 0x7fa0c01b3770 2024-04-02T05:00:16.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c01a9190 con 0x7fa0c01b3770 2024-04-02T05:00:16.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 4 2024-04-02T05:00:16.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 4 10 2024-04-02T05:00:16.195 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:00:16.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 10 client.43667.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 11 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa0c01b65b0 0x7fa0c01b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 11 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 5 11 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 4 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 10 client.43667.objecter _send_command 11 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01bba30 con 0x7fa0c01b65b0 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa0c01b65b0 0x7fa0c01b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c01b65b0 2024-04-02T05:00:16.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0be59c700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa0c01b65b0 0x7fa0c01b89e0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01bba30 con 0x7fa0c01b65b0 2024-04-02T05:00:16.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 11: 0 ) 2024-04-02T05:00:16.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 11 = system:0 2024-04-02T05:00:16.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 4 2024-04-02T05:00:16.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.193+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 5 11 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 10 client.43667.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 12 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 12 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01a9190 osd=5 3 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 5 12 2024-04-02T05:00:16.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 4 2024-04-02T05:00:16.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 10 client.43667.objecter _send_command 12 2024-04-02T05:00:16.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.213+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c01bba30 con 0x7fa0c01b65b0 2024-04-02T05:00:16.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:16.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c01bba30 con 0x7fa0c01b65b0 2024-04-02T05:00:16.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 4 2024-04-02T05:00:16.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 5 12 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 10 client.43667.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 13 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa0c01badc0 0x7fa0c01bb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 13 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 6 13 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 4 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 10 client.43667.objecter _send_command 13 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01bedd0 con 0x7fa0c01badc0 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa0c01badc0 0x7fa0c01bb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0bf59e700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa0c01badc0 0x7fa0c01bb220 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.217+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c01badc0 2024-04-02T05:00:16.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.221+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01bedd0 con 0x7fa0c01badc0 2024-04-02T05:00:16.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.221+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 13: 0 ) 2024-04-02T05:00:16.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.221+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 13 = system:0 2024-04-02T05:00:16.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.221+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 4 2024-04-02T05:00:16.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.221+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 6 13 2024-04-02T05:00:16.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 10 client.43667.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 14 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 14 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01bba30 osd=6 3 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 6 14 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 4 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 10 client.43667.objecter _send_command 14 2024-04-02T05:00:16.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.237+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c01b9030 con 0x7fa0c01badc0 2024-04-02T05:00:16.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c01b9030 con 0x7fa0c01badc0 2024-04-02T05:00:16.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 4 2024-04-02T05:00:16.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 6 14 2024-04-02T05:00:16.243 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:00:16.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 10 client.43667.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:16.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 15 2024-04-02T05:00:16.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa0c01b9540 0x7fa0c01b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 15 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 7 15 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 4 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 10 client.43667.objecter _send_command 15 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa0c01b9ee0 con 0x7fa0c01b9540 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa0c01b9540 0x7fa0c01b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 10 client.43667.objecter ms_handle_connect 0x7fa0c01b9540 2024-04-02T05:00:16.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0bed9d700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa0c01b9540 0x7fa0c01b99a0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:16.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c01b9ee0 con 0x7fa0c01b9540 2024-04-02T05:00:16.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 15: 0 ) 2024-04-02T05:00:16.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 15 = system:0 2024-04-02T05:00:16.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 4 2024-04-02T05:00:16.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.241+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 7 15 2024-04-02T05:00:16.247 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:16.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 10 client.43667.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign -1 16 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 20 client.43667.objecter _get_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_remove -1 16 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 20 client.43667.objecter get_session s=0x7fa0c01ba390 osd=7 3 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 15 client.43667.objecter _session_command_op_assign 7 16 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 4 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 10 client.43667.objecter _send_command 16 2024-04-02T05:00:16.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.257+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0c01b9ee0 con 0x7fa0c01b9540 2024-04-02T05:00:16.265 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.261+0000 7fa0a7fff700 1 -- 172.21.15.69:0/3131350509 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa0c01b9ee0 con 0x7fa0c01b9540 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.261+0000 7fa0a7fff700 10 client.43667.objecter ms_dispatch 0x7fa0c0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.261+0000 7fa0a7fff700 10 client.43667.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.261+0000 7fa0a7fff700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 4 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.261+0000 7fa0a7fff700 15 client.43667.objecter _session_command_op_remove 7 16 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 librados: watch_flush enter 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 librados: watch_flush exit 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.0 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa0c0062e50 msgr2=0x7fa0c0063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fa0c0062e50 0x7fa0c0063230 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c006a510 osd=0 1 2024-04-02T05:00:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.1 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa0c002c9b0 msgr2=0x7fa0c002cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fa0c002c9b0 0x7fa0c002cd90 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109780 osd=1 1 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.2 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa0c01ac360 msgr2=0x7fa0c01ae750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fa0c01ac360 0x7fa0c01ae750 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c0109600 osd=2 1 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.3 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa0c01afe10 msgr2=0x7fa0c01b0250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fa0c01afe10 0x7fa0c01b0250 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a97f0 osd=3 1 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.4 2024-04-02T05:00:16.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa0c01b3770 msgr2=0x7fa0c01b5b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fa0c01b3770 0x7fa0c01b5b60 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9020 osd=4 1 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.5 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa0c01b65b0 msgr2=0x7fa0c01b89e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fa0c01b65b0 0x7fa0c01b89e0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01a9190 osd=5 1 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.6 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa0c01badc0 msgr2=0x7fa0c01bb220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa0c01badc0 0x7fa0c01bb220 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01bba30 osd=6 1 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter close_session for osd.7 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa0c01b9540 msgr2=0x7fa0c01b99a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fa0c01b9540 0x7fa0c01b99a0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:16.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter put_session s=0x7fa0c01ba390 osd=7 1 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 20 client.43667.objecter shutdown clearing up homeless session... 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 10 client.43667.objecter successfully canceled tick 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0bf59e700 1 -- 172.21.15.69:0/3131350509 reap_dead start 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0bf59e700 1 -- 172.21.15.69:0/3131350509 reap_dead start 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0a806f5e0 msgr2=0x7fa0a8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa0a806f5e0 0x7fa0a8071aa0 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009750 tx=0x7fa0b0009310 comp rx=0 tx=0).stop 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 msgr2=0x7fa0c01971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 --2- 172.21.15.69:0/3131350509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0c0101b60 0x7fa0c01971d0 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400b480 tx=0x7fa0b400b790 comp rx=0 tx=0).stop 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0bf59e700 1 -- 172.21.15.69:0/3131350509 reap_dead start 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0bf59e700 1 -- 172.21.15.69:0/3131350509 reap_dead start 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0bf59e700 1 -- 172.21.15.69:0/3131350509 reap_dead start 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 shutdown_connections 2024-04-02T05:00:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 >> 172.21.15.69:0/3131350509 conn(0x7fa0c00771e0 msgr2=0x7fa0c0107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 shutdown_connections 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 -- 172.21.15.69:0/3131350509 wait complete. 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:16.265+0000 7fa0c5081700 1 librados: shutdown 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:00:16.271 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:00:16.272 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:00:16.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:16.351 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:00:16.372 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:16.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:16.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:16.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:16.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:14.813693+0000 osd.3 (osd.3) 355 : cluster 0 3.2 scrub starts 2024-04-02T05:00:16.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:14.816449+0000 osd.3 (osd.3) 356 : cluster 0 3.2 scrub ok 2024-04-02T05:00:16.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.154769+0000 osd.1 (osd.1) 365 : cluster 0 10.18 scrub starts 2024-04-02T05:00:16.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.158500+0000 osd.1 (osd.1) 366 : cluster 0 10.18 scrub ok 2024-04-02T05:00:16.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.490480+0000 osd.7 (osd.7) 169 : cluster 0 6.8 scrub starts 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.498295+0000 osd.7 (osd.7) 170 : cluster 0 6.8 scrub ok 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: audit 2024-04-02T05:00:16.037136+0000 mon.a (mon.0) 3501 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-116"}]': finished 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.040817+0000 mon.a (mon.0) 3502 : cluster 0 osdmap e718: 8 total, 8 up, 7 in 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: audit 2024-04-02T05:00:16.043123+0000 mon.a (mon.0) 3503 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-116", "mode": "writeback"}]: dispatch 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: audit 2024-04-02T05:00:16.078179+0000 mon.c (mon.2) 1149 : audit 0 from='client.? 172.21.15.69:0/3131350509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.137453+0000 osd.2 (osd.2) 327 : cluster 0 10.5 scrub starts 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.139348+0000 osd.2 (osd.2) 328 : cluster 0 10.5 scrub ok 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:14.813693+0000 osd.3 (osd.3) 355 : cluster 0 3.2 scrub starts 2024-04-02T05:00:16.616 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:14.816449+0000 osd.3 (osd.3) 356 : cluster 0 3.2 scrub ok 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.154769+0000 osd.1 (osd.1) 365 : cluster 0 10.18 scrub starts 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.158500+0000 osd.1 (osd.1) 366 : cluster 0 10.18 scrub ok 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.490480+0000 osd.7 (osd.7) 169 : cluster 0 6.8 scrub starts 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.498295+0000 osd.7 (osd.7) 170 : cluster 0 6.8 scrub ok 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: audit 2024-04-02T05:00:16.037136+0000 mon.a (mon.0) 3501 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-116"}]': finished 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.040817+0000 mon.a (mon.0) 3502 : cluster 0 osdmap e718: 8 total, 8 up, 7 in 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: audit 2024-04-02T05:00:16.043123+0000 mon.a (mon.0) 3503 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-116", "mode": "writeback"}]: dispatch 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: audit 2024-04-02T05:00:16.078179+0000 mon.c (mon.2) 1149 : audit 0 from='client.? 172.21.15.69:0/3131350509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.137453+0000 osd.2 (osd.2) 327 : cluster 0 10.5 scrub starts 2024-04-02T05:00:16.617 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:16 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.139348+0000 osd.2 (osd.2) 328 : cluster 0 10.5 scrub ok 2024-04-02T05:00:16.628 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:14.813693+0000 osd.3 (osd.3) 355 : cluster 0 3.2 scrub starts 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:14.816449+0000 osd.3 (osd.3) 356 : cluster 0 3.2 scrub ok 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.154769+0000 osd.1 (osd.1) 365 : cluster 0 10.18 scrub starts 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.158500+0000 osd.1 (osd.1) 366 : cluster 0 10.18 scrub ok 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.490480+0000 osd.7 (osd.7) 169 : cluster 0 6.8 scrub starts 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.498295+0000 osd.7 (osd.7) 170 : cluster 0 6.8 scrub ok 2024-04-02T05:00:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: audit 2024-04-02T05:00:16.037136+0000 mon.a (mon.0) 3501 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-116"}]': finished 2024-04-02T05:00:16.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.040817+0000 mon.a (mon.0) 3502 : cluster 0 osdmap e718: 8 total, 8 up, 7 in 2024-04-02T05:00:16.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: audit 2024-04-02T05:00:16.043123+0000 mon.a (mon.0) 3503 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-116", "mode": "writeback"}]: dispatch 2024-04-02T05:00:16.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: audit 2024-04-02T05:00:16.078179+0000 mon.c (mon.2) 1149 : audit 0 from='client.? 172.21.15.69:0/3131350509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:16.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.137453+0000 osd.2 (osd.2) 327 : cluster 0 10.5 scrub starts 2024-04-02T05:00:16.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:16 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.139348+0000 osd.2 (osd.2) 328 : cluster 0 10.5 scrub ok 2024-04-02T05:00:16.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:16.758 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:16.861 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:16.888 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:16.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:16.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:17.115 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T05:00:17.115 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T05:00:17.115 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-02T05:00:17.115 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T05:00:17.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:17.186 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:17.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:17.310 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:17.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:17.440 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:17.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:17.572 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:17.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.832954+0000 osd.3 (osd.3) 357 : cluster 0 4.1a scrub starts 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.835217+0000 osd.3 (osd.3) 358 : cluster 0 4.1a scrub ok 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.925737+0000 osd.0 (osd.0) 353 : cluster 0 10.1a scrub starts 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:15.927390+0000 osd.0 (osd.0) 354 : cluster 0 10.1a scrub ok 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.168404+0000 osd.1 (osd.1) 367 : cluster 0 3.0 scrub starts 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.171884+0000 osd.1 (osd.1) 368 : cluster 0 3.0 scrub ok 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.278845+0000 mgr.y (mgr.14538) 645 : cluster 0 pgmap v1083: 214 pgs: 6 creating+activating, 17 creating+peering, 191 active+clean; 582 KiB data, 678 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-02T05:00:17.686 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.446291+0000 osd.7 (osd.7) 171 : cluster 0 6.5 scrub starts 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.448373+0000 osd.7 (osd.7) 172 : cluster 0 6.5 scrub ok 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.018266+0000 osd.6 (osd.6) 303 : cluster 0 4.a scrub starts 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.020246+0000 osd.6 (osd.6) 304 : cluster 0 4.a scrub ok 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.037998+0000 mon.a (mon.0) 3504 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: audit 2024-04-02T05:00:17.045038+0000 mon.a (mon.0) 3505 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-116", "mode": "writeback"}]': finished 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.048440+0000 mon.a (mon.0) 3506 : cluster 0 osdmap e719: 8 total, 8 up, 7 in 2024-04-02T05:00:17.687 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:17 smithi098 bash[23790]: audit 2024-04-02T05:00:17.068490+0000 mon.a (mon.0) 3507 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-02T05:00:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:17.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.832954+0000 osd.3 (osd.3) 357 : cluster 0 4.1a scrub starts 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.835217+0000 osd.3 (osd.3) 358 : cluster 0 4.1a scrub ok 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.925737+0000 osd.0 (osd.0) 353 : cluster 0 10.1a scrub starts 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:15.927390+0000 osd.0 (osd.0) 354 : cluster 0 10.1a scrub ok 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.168404+0000 osd.1 (osd.1) 367 : cluster 0 3.0 scrub starts 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.171884+0000 osd.1 (osd.1) 368 : cluster 0 3.0 scrub ok 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.278845+0000 mgr.y (mgr.14538) 645 : cluster 0 pgmap v1083: 214 pgs: 6 creating+activating, 17 creating+peering, 191 active+clean; 582 KiB data, 678 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.446291+0000 osd.7 (osd.7) 171 : cluster 0 6.5 scrub starts 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.448373+0000 osd.7 (osd.7) 172 : cluster 0 6.5 scrub ok 2024-04-02T05:00:17.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.018266+0000 osd.6 (osd.6) 303 : cluster 0 4.a scrub starts 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.020246+0000 osd.6 (osd.6) 304 : cluster 0 4.a scrub ok 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.037998+0000 mon.a (mon.0) 3504 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: audit 2024-04-02T05:00:17.045038+0000 mon.a (mon.0) 3505 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-116", "mode": "writeback"}]': finished 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.048440+0000 mon.a (mon.0) 3506 : cluster 0 osdmap e719: 8 total, 8 up, 7 in 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[21528]: audit 2024-04-02T05:00:17.068490+0000 mon.a (mon.0) 3507 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.832954+0000 osd.3 (osd.3) 357 : cluster 0 4.1a scrub starts 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.835217+0000 osd.3 (osd.3) 358 : cluster 0 4.1a scrub ok 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.925737+0000 osd.0 (osd.0) 353 : cluster 0 10.1a scrub starts 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:15.927390+0000 osd.0 (osd.0) 354 : cluster 0 10.1a scrub ok 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.168404+0000 osd.1 (osd.1) 367 : cluster 0 3.0 scrub starts 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.171884+0000 osd.1 (osd.1) 368 : cluster 0 3.0 scrub ok 2024-04-02T05:00:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.278845+0000 mgr.y (mgr.14538) 645 : cluster 0 pgmap v1083: 214 pgs: 6 creating+activating, 17 creating+peering, 191 active+clean; 582 KiB data, 678 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.446291+0000 osd.7 (osd.7) 171 : cluster 0 6.5 scrub starts 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.448373+0000 osd.7 (osd.7) 172 : cluster 0 6.5 scrub ok 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.018266+0000 osd.6 (osd.6) 303 : cluster 0 4.a scrub starts 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.020246+0000 osd.6 (osd.6) 304 : cluster 0 4.a scrub ok 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.037998+0000 mon.a (mon.0) 3504 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: audit 2024-04-02T05:00:17.045038+0000 mon.a (mon.0) 3505 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-116", "mode": "writeback"}]': finished 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.048440+0000 mon.a (mon.0) 3506 : cluster 0 osdmap e719: 8 total, 8 up, 7 in 2024-04-02T05:00:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:17 smithi069 bash[28904]: audit 2024-04-02T05:00:17.068490+0000 mon.a (mon.0) 3507 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-02T05:00:17.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:17.825 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:17.830 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:17.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:17.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:18.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:18.082 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:18.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:18.213 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:18.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:18.446 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:18.474 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:18.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:18.603 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:18.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.827831+0000 osd.4 (osd.4) 366 : cluster 0 6.17 scrub starts 2024-04-02T05:00:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.829017+0000 osd.3 (osd.3) 359 : cluster 0 6.2 scrub starts 2024-04-02T05:00:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.829945+0000 osd.4 (osd.4) 367 : cluster 0 6.17 scrub ok 2024-04-02T05:00:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[28904]: cluster 2024-04-02T05:00:16.830861+0000 osd.3 (osd.3) 360 : cluster 0 6.2 scrub ok 2024-04-02T05:00:18.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[28904]: audit 2024-04-02T05:00:17.068719+0000 mgr.y (mgr.14538) 646 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-02T05:00:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.827831+0000 osd.4 (osd.4) 366 : cluster 0 6.17 scrub starts 2024-04-02T05:00:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.829017+0000 osd.3 (osd.3) 359 : cluster 0 6.2 scrub starts 2024-04-02T05:00:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.829945+0000 osd.4 (osd.4) 367 : cluster 0 6.17 scrub ok 2024-04-02T05:00:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[21528]: cluster 2024-04-02T05:00:16.830861+0000 osd.3 (osd.3) 360 : cluster 0 6.2 scrub ok 2024-04-02T05:00:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[21528]: audit 2024-04-02T05:00:17.068719+0000 mgr.y (mgr.14538) 646 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-02T05:00:18.734 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:18.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:18.847 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:18 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.827831+0000 osd.4 (osd.4) 366 : cluster 0 6.17 scrub starts 2024-04-02T05:00:18.847 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:18 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.829017+0000 osd.3 (osd.3) 359 : cluster 0 6.2 scrub starts 2024-04-02T05:00:18.847 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:18 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.829945+0000 osd.4 (osd.4) 367 : cluster 0 6.17 scrub ok 2024-04-02T05:00:18.847 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:18 smithi098 bash[23790]: cluster 2024-04-02T05:00:16.830861+0000 osd.3 (osd.3) 360 : cluster 0 6.2 scrub ok 2024-04-02T05:00:18.847 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:18 smithi098 bash[23790]: audit 2024-04-02T05:00:17.068719+0000 mgr.y (mgr.14538) 646 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-02T05:00:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:18.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:18.975 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:00:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:00:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:00:18.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:18.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:19.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:19.119 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:19.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:19.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:19.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:19.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T05:00:19.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:19.504 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:19.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:19.626 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.871472+0000 osd.4 (osd.4) 368 : cluster 0 326.6 scrub starts 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.872963+0000 osd.4 (osd.4) 369 : cluster 0 326.6 scrub ok 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.929574+0000 osd.0 (osd.0) 355 : cluster 0 1.1 scrub starts 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:17.931884+0000 osd.0 (osd.0) 356 : cluster 0 1.1 scrub ok 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.165160+0000 osd.1 (osd.1) 369 : cluster 0 6.19 scrub starts 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.167315+0000 osd.1 (osd.1) 370 : cluster 0 6.19 scrub ok 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.280980+0000 mgr.y (mgr.14538) 647 : cluster 0 pgmap v1085: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 488 B/s wr, 2 op/s 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: audit 2024-04-02T05:00:18.835746+0000 mon.b (mon.1) 296 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:19.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:19 smithi098 bash[23790]: audit 2024-04-02T05:00:18.836651+0000 mon.a (mon.0) 3508 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:19.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:19.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:19.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.871472+0000 osd.4 (osd.4) 368 : cluster 0 326.6 scrub starts 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.872963+0000 osd.4 (osd.4) 369 : cluster 0 326.6 scrub ok 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.929574+0000 osd.0 (osd.0) 355 : cluster 0 1.1 scrub starts 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:17.931884+0000 osd.0 (osd.0) 356 : cluster 0 1.1 scrub ok 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.165160+0000 osd.1 (osd.1) 369 : cluster 0 6.19 scrub starts 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.167315+0000 osd.1 (osd.1) 370 : cluster 0 6.19 scrub ok 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.280980+0000 mgr.y (mgr.14538) 647 : cluster 0 pgmap v1085: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 488 B/s wr, 2 op/s 2024-04-02T05:00:19.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: audit 2024-04-02T05:00:18.835746+0000 mon.b (mon.1) 296 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[21528]: audit 2024-04-02T05:00:18.836651+0000 mon.a (mon.0) 3508 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.871472+0000 osd.4 (osd.4) 368 : cluster 0 326.6 scrub starts 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.872963+0000 osd.4 (osd.4) 369 : cluster 0 326.6 scrub ok 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.929574+0000 osd.0 (osd.0) 355 : cluster 0 1.1 scrub starts 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:17.931884+0000 osd.0 (osd.0) 356 : cluster 0 1.1 scrub ok 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.165160+0000 osd.1 (osd.1) 369 : cluster 0 6.19 scrub starts 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.167315+0000 osd.1 (osd.1) 370 : cluster 0 6.19 scrub ok 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.280980+0000 mgr.y (mgr.14538) 647 : cluster 0 pgmap v1085: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 488 B/s wr, 2 op/s 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: audit 2024-04-02T05:00:18.835746+0000 mon.b (mon.1) 296 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:19.871 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:19 smithi069 bash[28904]: audit 2024-04-02T05:00:18.836651+0000 mon.a (mon.0) 3508 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:19.886 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:19.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:19.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:20.012 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:20.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:20.139 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:20.184 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:00:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:00:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T05:00:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T05:00:20.184 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:00:20.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:20.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:20.393 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:20.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:20.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:20.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:20.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.798389+0000 osd.3 (osd.3) 361 : cluster 0 6.13 scrub starts 2024-04-02T05:00:20.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.800034+0000 osd.3 (osd.3) 362 : cluster 0 6.13 scrub ok 2024-04-02T05:00:20.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.915991+0000 osd.0 (osd.0) 357 : cluster 0 4.3 scrub starts 2024-04-02T05:00:20.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:18.918795+0000 osd.0 (osd.0) 358 : cluster 0 4.3 scrub ok 2024-04-02T05:00:20.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.603582+0000 osd.2 (osd.2) 329 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:20.766 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.603745+0000 osd.2 (osd.2) 330 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:20.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.905174+0000 osd.6 (osd.6) 305 : cluster 0 10.b scrub starts 2024-04-02T05:00:20.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.909313+0000 osd.6 (osd.6) 306 : cluster 0 10.b scrub ok 2024-04-02T05:00:20.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:20 smithi098 bash[23790]: cluster 2024-04-02T05:00:20.034427+0000 mon.a (mon.0) 3509 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.798389+0000 osd.3 (osd.3) 361 : cluster 0 6.13 scrub starts 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.800034+0000 osd.3 (osd.3) 362 : cluster 0 6.13 scrub ok 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.915991+0000 osd.0 (osd.0) 357 : cluster 0 4.3 scrub starts 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:18.918795+0000 osd.0 (osd.0) 358 : cluster 0 4.3 scrub ok 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.603582+0000 osd.2 (osd.2) 329 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.603745+0000 osd.2 (osd.2) 330 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.905174+0000 osd.6 (osd.6) 305 : cluster 0 10.b scrub starts 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.909313+0000 osd.6 (osd.6) 306 : cluster 0 10.b scrub ok 2024-04-02T05:00:20.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[21528]: cluster 2024-04-02T05:00:20.034427+0000 mon.a (mon.0) 3509 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.798389+0000 osd.3 (osd.3) 361 : cluster 0 6.13 scrub starts 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.800034+0000 osd.3 (osd.3) 362 : cluster 0 6.13 scrub ok 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.915991+0000 osd.0 (osd.0) 357 : cluster 0 4.3 scrub starts 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:18.918795+0000 osd.0 (osd.0) 358 : cluster 0 4.3 scrub ok 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.603582+0000 osd.2 (osd.2) 329 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.603745+0000 osd.2 (osd.2) 330 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.905174+0000 osd.6 (osd.6) 305 : cluster 0 10.b scrub starts 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.909313+0000 osd.6 (osd.6) 306 : cluster 0 10.b scrub ok 2024-04-02T05:00:20.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:20 smithi069 bash[28904]: cluster 2024-04-02T05:00:20.034427+0000 mon.a (mon.0) 3509 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:20.783 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:20.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:20.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:20.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:21.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:21.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-02T05:00:21.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:21.167 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:21.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:21.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4148679681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0810a540 msgr2=0x7efd0810a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:21.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4148679681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0810a540 0x7efd0810a920 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7efcf80098f0 tx=0x7efcf8009c00 comp rx=0 tx=0).stop 2024-04-02T05:00:21.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4148679681 shutdown_connections 2024-04-02T05:00:21.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4148679681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd08104e50 0x7efd08111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:21.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4148679681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0810a540 0x7efd0810a920 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:21.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4148679681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd08104490 0x7efd08104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4148679681 >> 172.21.15.69:0/4148679681 conn(0x7efd080fff60 msgr2=0x7efd08102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4148679681 shutdown_connections 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4148679681 wait complete. 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 librados: starting msgr at 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 librados: starting objecter 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 Processor -- start 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- start start 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 librados: setting wanted keys 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 librados: calling monclient init 2024-04-02T05:00:21.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 0x7efd0819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd08104e50 0x7efd0819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0810a540 0x7efd0819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efd08113c50 con 0x7efd0810a540 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd08113ad0 con 0x7efd08104e50 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efd08113dd0 con 0x7efd08104490 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 0x7efd0819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 0x7efd0819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:59004/0 (socket says 172.21.15.69:59004) 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 -- 172.21.15.69:0/4201138409 learned_addr learned my addr 172.21.15.69:0/4201138409 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd06d9d700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0810a540 0x7efd0819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:21.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd05d9b700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd08104e50 0x7efd0819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:21.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 -- 172.21.15.69:0/4201138409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd08104e50 msgr2=0x7efd0819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:21.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd08104e50 0x7efd0819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:21.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 -- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0810a540 msgr2=0x7efd0819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:21.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0810a540 0x7efd0819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:21.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 -- 172.21.15.69:0/4201138409 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0819f200 con 0x7efd08104490 2024-04-02T05:00:21.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0659c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 0x7efd0819a1f0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7efcf800fa00 tx=0x7efcf800fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efcf77fe700 10 client.?.objecter ms_handle_connect 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efcf77fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8027070 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7efcf800ba60 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0819f490 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8004930 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 10 client.43673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd081a6d70 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 librados: init done 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.281+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd08105bb0 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7efcf802a070 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efcf006f5e0 0x7efcf0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(720..720 src has 74..720) ==== 9995+0+0 (secure 0 0 0) 0x7efcf8096350 con 0x7efd08104490 2024-04-02T05:00:21.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 10 client.43673.objecter ms_dispatch 0x7efd08001550 osd_map(720..720 src has 74..720) 2024-04-02T05:00:21.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efd05d9b700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efcf006f5e0 0x7efcf0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:21.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 3 client.43673.objecter handle_osd_map got epochs [720,720] > 0 2024-04-02T05:00:21.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 3 client.43673.objecter handle_osd_map decoding full epoch 720 2024-04-02T05:00:21.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 20 client.43673.objecter dump_active .. 0 homeless 2024-04-02T05:00:21.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efcf77fe700 10 client.43673.objecter ms_handle_connect 0x7efcf006f5e0 2024-04-02T05:00:21.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.285+0000 7efd05d9b700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efcf006f5e0 0x7efcf0071aa0 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009750 tx=0x7efcfc009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:21.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.289+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efcf8013030 con 0x7efd08104490 2024-04-02T05:00:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:21.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:21.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:21.517+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7efd0819b550 con 0x7efd08104490 2024-04-02T05:00:21.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:21.545 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:21.647 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:21.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.763827+0000 osd.3 (osd.3) 363 : cluster 0 4.0 scrub starts 2024-04-02T05:00:21.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.766735+0000 osd.3 (osd.3) 364 : cluster 0 4.0 scrub ok 2024-04-02T05:00:21.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.867002+0000 osd.4 (osd.4) 370 : cluster 0 4.18 scrub starts 2024-04-02T05:00:21.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:19.869303+0000 osd.4 (osd.4) 371 : cluster 0 4.18 scrub ok 2024-04-02T05:00:21.658 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:20.282509+0000 mgr.y (mgr.14538) 648 : cluster 0 pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:20.587899+0000 mon.a (mon.0) 3510 : cluster 0 osdmap e720: 8 total, 8 up, 7 in 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:21.141198+0000 osd.2 (osd.2) 331 : cluster 0 2.10 scrub starts 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: cluster 2024-04-02T05:00:21.143334+0000 osd.2 (osd.2) 332 : cluster 0 2.10 scrub ok 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: audit 2024-04-02T05:00:21.520804+0000 mon.c (mon.2) 1150 : audit 1 from='client.? 172.21.15.69:0/4201138409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[21528]: audit 2024-04-02T05:00:21.521145+0000 mon.a (mon.0) 3511 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.763827+0000 osd.3 (osd.3) 363 : cluster 0 4.0 scrub starts 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.766735+0000 osd.3 (osd.3) 364 : cluster 0 4.0 scrub ok 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.867002+0000 osd.4 (osd.4) 370 : cluster 0 4.18 scrub starts 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:19.869303+0000 osd.4 (osd.4) 371 : cluster 0 4.18 scrub ok 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:20.282509+0000 mgr.y (mgr.14538) 648 : cluster 0 pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-02T05:00:21.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:20.587899+0000 mon.a (mon.0) 3510 : cluster 0 osdmap e720: 8 total, 8 up, 7 in 2024-04-02T05:00:21.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:21.141198+0000 osd.2 (osd.2) 331 : cluster 0 2.10 scrub starts 2024-04-02T05:00:21.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: cluster 2024-04-02T05:00:21.143334+0000 osd.2 (osd.2) 332 : cluster 0 2.10 scrub ok 2024-04-02T05:00:21.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: audit 2024-04-02T05:00:21.520804+0000 mon.c (mon.2) 1150 : audit 1 from='client.? 172.21.15.69:0/4201138409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:00:21.660 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:21 smithi069 bash[28904]: audit 2024-04-02T05:00:21.521145+0000 mon.a (mon.0) 3511 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:00:21.671 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:21.774 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:21.798 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:21.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:21.928 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:21.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.763827+0000 osd.3 (osd.3) 363 : cluster 0 4.0 scrub starts 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.766735+0000 osd.3 (osd.3) 364 : cluster 0 4.0 scrub ok 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.867002+0000 osd.4 (osd.4) 370 : cluster 0 4.18 scrub starts 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:19.869303+0000 osd.4 (osd.4) 371 : cluster 0 4.18 scrub ok 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:20.282509+0000 mgr.y (mgr.14538) 648 : cluster 0 pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:20.587899+0000 mon.a (mon.0) 3510 : cluster 0 osdmap e720: 8 total, 8 up, 7 in 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:21.141198+0000 osd.2 (osd.2) 331 : cluster 0 2.10 scrub starts 2024-04-02T05:00:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: cluster 2024-04-02T05:00:21.143334+0000 osd.2 (osd.2) 332 : cluster 0 2.10 scrub ok 2024-04-02T05:00:21.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: audit 2024-04-02T05:00:21.520804+0000 mon.c (mon.2) 1150 : audit 1 from='client.? 172.21.15.69:0/4201138409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:00:21.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:21 smithi098 bash[23790]: audit 2024-04-02T05:00:21.521145+0000 mon.a (mon.0) 3511 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:00:22.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.053+0000 7efcf77fe700 1 -- 172.21.15.69:0/4201138409 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v721) ==== 87+0+0 (secure 0 0 0) 0x7efcf805be10 con 0x7efd08104490 2024-04-02T05:00:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 10 librados: watch_flush enter 2024-04-02T05:00:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 10 librados: watch_flush exit 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 20 client.43673.objecter shutdown clearing up homeless session... 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 10 client.43673.objecter successfully canceled tick 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efcf006f5e0 msgr2=0x7efcf0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efcf006f5e0 0x7efcf0071aa0 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009750 tx=0x7efcfc009310 comp rx=0 tx=0).stop 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 msgr2=0x7efd0819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 0x7efd0819a1f0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7efcf800fa00 tx=0x7efcf800fa30 comp rx=0 tx=0).stop 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 shutdown_connections 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efcf006f5e0 0x7efcf0071aa0 unknown :-1 s=CLOSED pgs=3992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0810a540 0x7efd0819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd08104490 0x7efd0819a1f0 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 --2- 172.21.15.69:0/4201138409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd08104e50 0x7efd0819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 >> 172.21.15.69:0/4201138409 conn(0x7efd080fff60 msgr2=0x7efd08108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 shutdown_connections 2024-04-02T05:00:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 -- 172.21.15.69:0/4201138409 wait complete. 2024-04-02T05:00:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:22.057+0000 7efd0c99c700 1 librados: shutdown 2024-04-02T05:00:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T05:00:22.062 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:22.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:22.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:22.417 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:22.443 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:22.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:22.572 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:22.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:22.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[28904]: cluster 2024-04-02T05:00:21.860211+0000 osd.0 (osd.0) 359 : cluster 0 3.17 scrub starts 2024-04-02T05:00:22.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[28904]: cluster 2024-04-02T05:00:21.862988+0000 osd.0 (osd.0) 360 : cluster 0 3.17 scrub ok 2024-04-02T05:00:22.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[28904]: cluster 2024-04-02T05:00:22.044230+0000 mon.a (mon.0) 3512 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:22.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[28904]: audit 2024-04-02T05:00:22.046560+0000 mon.a (mon.0) 3513 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:00:22.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[28904]: cluster 2024-04-02T05:00:22.055560+0000 mon.a (mon.0) 3514 : cluster 0 osdmap e721: 8 total, 8 up, 7 in 2024-04-02T05:00:22.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[21528]: cluster 2024-04-02T05:00:21.860211+0000 osd.0 (osd.0) 359 : cluster 0 3.17 scrub starts 2024-04-02T05:00:22.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[21528]: cluster 2024-04-02T05:00:21.862988+0000 osd.0 (osd.0) 360 : cluster 0 3.17 scrub ok 2024-04-02T05:00:22.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[21528]: cluster 2024-04-02T05:00:22.044230+0000 mon.a (mon.0) 3512 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:22.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[21528]: audit 2024-04-02T05:00:22.046560+0000 mon.a (mon.0) 3513 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:00:22.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:22 smithi069 bash[21528]: cluster 2024-04-02T05:00:22.055560+0000 mon.a (mon.0) 3514 : cluster 0 osdmap e721: 8 total, 8 up, 7 in 2024-04-02T05:00:22.702 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:22.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:22.831 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:22.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:22.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:22 smithi098 bash[23790]: cluster 2024-04-02T05:00:21.860211+0000 osd.0 (osd.0) 359 : cluster 0 3.17 scrub starts 2024-04-02T05:00:22.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:22 smithi098 bash[23790]: cluster 2024-04-02T05:00:21.862988+0000 osd.0 (osd.0) 360 : cluster 0 3.17 scrub ok 2024-04-02T05:00:22.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:22 smithi098 bash[23790]: cluster 2024-04-02T05:00:22.044230+0000 mon.a (mon.0) 3512 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:22.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:22 smithi098 bash[23790]: audit 2024-04-02T05:00:22.046560+0000 mon.a (mon.0) 3513 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:00:22.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:22 smithi098 bash[23790]: cluster 2024-04-02T05:00:22.055560+0000 mon.a (mon.0) 3514 : cluster 0 osdmap e721: 8 total, 8 up, 7 in 2024-04-02T05:00:22.962 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:22.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:23.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:23.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:23.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:23.351 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:23.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:23.484 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:23.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:23.712 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:23.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:00:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:00:23.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:23 smithi098 bash[23790]: cluster 2024-04-02T05:00:21.698480+0000 osd.3 (osd.3) 365 : cluster 0 10.3 scrub starts 2024-04-02T05:00:23.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:23 smithi098 bash[23790]: cluster 2024-04-02T05:00:21.700482+0000 osd.3 (osd.3) 366 : cluster 0 10.3 scrub ok 2024-04-02T05:00:23.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:23 smithi098 bash[23790]: cluster 2024-04-02T05:00:22.283343+0000 mgr.y (mgr.14538) 649 : cluster 0 pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-02T05:00:23.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:23 smithi069 bash[21528]: cluster 2024-04-02T05:00:21.698480+0000 osd.3 (osd.3) 365 : cluster 0 10.3 scrub starts 2024-04-02T05:00:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:23 smithi069 bash[21528]: cluster 2024-04-02T05:00:21.700482+0000 osd.3 (osd.3) 366 : cluster 0 10.3 scrub ok 2024-04-02T05:00:23.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:23 smithi069 bash[21528]: cluster 2024-04-02T05:00:22.283343+0000 mgr.y (mgr.14538) 649 : cluster 0 pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-02T05:00:23.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:23 smithi069 bash[28904]: cluster 2024-04-02T05:00:21.698480+0000 osd.3 (osd.3) 365 : cluster 0 10.3 scrub starts 2024-04-02T05:00:23.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:23 smithi069 bash[28904]: cluster 2024-04-02T05:00:21.700482+0000 osd.3 (osd.3) 366 : cluster 0 10.3 scrub ok 2024-04-02T05:00:23.781 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:23 smithi069 bash[28904]: cluster 2024-04-02T05:00:22.283343+0000 mgr.y (mgr.14538) 649 : cluster 0 pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-02T05:00:23.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:23.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:23.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:24.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:24.203 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:24.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:24.354 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:24.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:24.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:24.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:24.603 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:24.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:24.731 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:24.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:24.851 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:24.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:24.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:24 smithi069 bash[21528]: audit 2024-04-02T05:00:23.599610+0000 mgr.y (mgr.14538) 650 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:24.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:24 smithi069 bash[21528]: audit 2024-04-02T05:00:23.998131+0000 mon.b (mon.1) 297 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:00:24.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:24 smithi069 bash[28904]: audit 2024-04-02T05:00:23.599610+0000 mgr.y (mgr.14538) 650 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:24.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:24 smithi069 bash[28904]: audit 2024-04-02T05:00:23.998131+0000 mon.b (mon.1) 297 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:00:24.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:24 smithi098 bash[23790]: audit 2024-04-02T05:00:23.599610+0000 mgr.y (mgr.14538) 650 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:24.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:24 smithi098 bash[23790]: audit 2024-04-02T05:00:23.998131+0000 mon.b (mon.1) 297 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:00:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:24.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:25.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T05:00:25.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:25.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:25.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:25.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:25.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:25.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:25.625 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:25.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:25.739 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:25 smithi098 bash[23790]: cluster 2024-04-02T05:00:24.285500+0000 mgr.y (mgr.14538) 651 : cluster 0 pgmap v1090: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 565 B/s wr, 2 op/s 2024-04-02T05:00:25.739 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:25 smithi098 bash[23790]: audit 2024-04-02T05:00:24.732789+0000 mon.b (mon.1) 298 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:00:25.740 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:25 smithi098 bash[23790]: audit 2024-04-02T05:00:24.733805+0000 mon.b (mon.1) 299 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:00:25.740 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:25 smithi098 bash[23790]: audit 2024-04-02T05:00:24.744937+0000 mon.a (mon.0) 3515 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:25.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:25.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:25.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[21528]: cluster 2024-04-02T05:00:24.285500+0000 mgr.y (mgr.14538) 651 : cluster 0 pgmap v1090: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 565 B/s wr, 2 op/s 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[21528]: audit 2024-04-02T05:00:24.732789+0000 mon.b (mon.1) 298 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[21528]: audit 2024-04-02T05:00:24.733805+0000 mon.b (mon.1) 299 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[21528]: audit 2024-04-02T05:00:24.744937+0000 mon.a (mon.0) 3515 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[28904]: cluster 2024-04-02T05:00:24.285500+0000 mgr.y (mgr.14538) 651 : cluster 0 pgmap v1090: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 565 B/s wr, 2 op/s 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[28904]: audit 2024-04-02T05:00:24.732789+0000 mon.b (mon.1) 298 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[28904]: audit 2024-04-02T05:00:24.733805+0000 mon.b (mon.1) 299 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:00:25.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:25 smithi069 bash[28904]: audit 2024-04-02T05:00:24.744937+0000 mon.a (mon.0) 3515 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:25.874 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:00:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:25.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:25.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:26.014 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:26.073 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:26.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:26.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:26.273 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:26.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:26.404 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:26.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:26.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:26.666 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:26.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:26.795 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:26.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:26.925 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:26.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:27.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:27.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:27.176 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:27.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- 172.21.15.69:0/4277662585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae81075a0 msgr2=0x7f9ae810f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- 172.21.15.69:0/4277662585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae81075a0 0x7f9ae810f170 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009fc0 tx=0x7f9ae000b330 comp rx=0 tx=0).stop 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- 172.21.15.69:0/4277662585 shutdown_connections 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- 172.21.15.69:0/4277662585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae81075a0 0x7f9ae810f170 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- 172.21.15.69:0/4277662585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 0x7f9ae8102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- 172.21.15.69:0/4277662585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae8102ed0 0x7f9ae8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- 172.21.15.69:0/4277662585 >> 172.21.15.69:0/4277662585 conn(0x7f9ae8079360 msgr2=0x7f9ae8079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- 172.21.15.69:0/4277662585 shutdown_connections 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- 172.21.15.69:0/4277662585 wait complete. 2024-04-02T05:00:27.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 librados: starting msgr at 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 librados: starting objecter 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 Processor -- start 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- start start 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 librados: setting wanted keys 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 librados: calling monclient init 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 0x7f9ae81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae8102ed0 0x7f9ae81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 0x7f9ae819c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8111ad0 con 0x7f9ae81075a0 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8111950 con 0x7f9ae8102520 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aee548700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae8111c50 con 0x7f9ae8102ed0 2024-04-02T05:00:27.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aecae5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 0x7f9ae819c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:27.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9ae77fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae8102ed0 0x7f9ae81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:27.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aecae5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 0x7f9ae819c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33308/0 (socket says 172.21.15.69:33308) 2024-04-02T05:00:27.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.213+0000 7f9aecae5700 1 -- 172.21.15.69:0/2711150521 learned_addr learned my addr 172.21.15.69:0/2711150521 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae7fff700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 0x7f9ae81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae77fe700 1 -- 172.21.15.69:0/2711150521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 msgr2=0x7f9ae81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae77fe700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 0x7f9ae81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae77fe700 1 -- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 msgr2=0x7f9ae819c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae77fe700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 0x7f9ae819c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae77fe700 1 -- 172.21.15.69:0/2711150521 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae819d0b0 con 0x7f9ae8102ed0 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae7fff700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 0x7f9ae81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9aecae5700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 0x7f9ae819c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae77fe700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae8102ed0 0x7f9ae81985e0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0005fd0 tx=0x7f9ad000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 10 client.?.objecter ms_handle_connect 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000cc90 con 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae819d3a0 con 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9aee548700 10 client.43679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae81a4c20 con 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9aee548700 1 librados: init done 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9ad000ce30 con 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0010430 con 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae804ff00 con 0x7f9ae8102ed0 2024-04-02T05:00:27.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f9ad0003f10 con 0x7f9ae8102ed0 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9ad406f5e0 0x7f9ad4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(721..721 src has 74..721) ==== 9995+0+0 (secure 0 0 0) 0x7f9ad008ec20 con 0x7f9ae8102ed0 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 10 client.43679.objecter ms_dispatch 0x7f9ae8001550 osd_map(721..721 src has 74..721) 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 3 client.43679.objecter handle_osd_map got epochs [721,721] > 0 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 3 client.43679.objecter handle_osd_map decoding full epoch 721 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae7fff700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9ad406f5e0 0x7f9ad4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 20 client.43679.objecter dump_active .. 0 homeless 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae57fa700 10 client.43679.objecter ms_handle_connect 0x7f9ad406f5e0 2024-04-02T05:00:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.217+0000 7f9ae7fff700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9ad406f5e0 0x7f9ad4071aa0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8007900 tx=0x7f9ad8008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:27.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.221+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9ad005b8a0 con 0x7f9ae8102ed0 2024-04-02T05:00:27.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:27.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:27.423 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:27.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.429+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ae8100840 con 0x7f9ad406f5e0 2024-04-02T05:00:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.445+0000 7f9ae57fa700 1 -- 172.21.15.69:0/2711150521 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+552969 (secure 0 0 0) 0x7f9ae8100840 con 0x7f9ad406f5e0 2024-04-02T05:00:27.454 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T05:00:27.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 10 librados: watch_flush enter 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 10 librados: watch_flush exit 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 20 client.43679.objecter shutdown clearing up homeless session... 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 10 client.43679.objecter successfully canceled tick 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9ad406f5e0 msgr2=0x7f9ad4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9ad406f5e0 0x7f9ad4071aa0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8007900 tx=0x7f9ad8008040 comp rx=0 tx=0).stop 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae8102ed0 msgr2=0x7f9ae81985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae8102ed0 0x7f9ae81985e0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0005fd0 tx=0x7f9ad000ee70 comp rx=0 tx=0).stop 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 shutdown_connections 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9ad406f5e0 0x7f9ad4071aa0 unknown :-1 s=CLOSED pgs=3993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae81075a0 0x7f9ae819c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8102520 0x7f9ae81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 --2- 172.21.15.69:0/2711150521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae8102ed0 0x7f9ae81985e0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 >> 172.21.15.69:0/2711150521 conn(0x7f9ae8079360 msgr2=0x7f9ae8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 shutdown_connections 2024-04-02T05:00:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 -- 172.21.15.69:0/2711150521 wait complete. 2024-04-02T05:00:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:27.449+0000 7f9aee548700 1 librados: shutdown 2024-04-02T05:00:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T05:00:27.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:27.544 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:27.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:27.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:27.802 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:27.814 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:27 smithi069 bash[21528]: cluster 2024-04-02T05:00:26.286470+0000 mgr.y (mgr.14538) 652 : cluster 0 pgmap v1091: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 255 B/s wr, 0 op/s 2024-04-02T05:00:27.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:27.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:27 smithi098 bash[23790]: cluster 2024-04-02T05:00:26.286470+0000 mgr.y (mgr.14538) 652 : cluster 0 pgmap v1091: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 255 B/s wr, 0 op/s 2024-04-02T05:00:27.932 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:27.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:28.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:27 smithi069 bash[28904]: cluster 2024-04-02T05:00:26.286470+0000 mgr.y (mgr.14538) 652 : cluster 0 pgmap v1091: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 255 B/s wr, 0 op/s 2024-04-02T05:00:28.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:28.062 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:28.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:28.190 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:28.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:28.320 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:28.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:28.445 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:28.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:28.574 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:28.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:28.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:28.832 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:28.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:28.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:28 smithi098 bash[23790]: audit 2024-04-02T05:00:27.433172+0000 mgr.y (mgr.14538) 653 : audit 0 from='client.43679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:00:28.961 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:28.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:29.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:28 smithi069 bash[21528]: audit 2024-04-02T05:00:27.433172+0000 mgr.y (mgr.14538) 653 : audit 0 from='client.43679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:00:29.031 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:00:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:00:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:00:29.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:28 smithi069 bash[28904]: audit 2024-04-02T05:00:27.433172+0000 mgr.y (mgr.14538) 653 : audit 0 from='client.43679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:00:29.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:29.087 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:29.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:29.210 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.15807.0:2 191.2 191:43af9add:test-rados-api-smithi069-72304-28::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:47.700587+0000", 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 581.59755027799997, 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.052179333000000001, 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.15807", 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1470643160", 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.700587+0000", 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.700587+0000", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.700590+0000", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1499999999999999e-06 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.700617+0000", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7373e-05 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.700647+0000", 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9462999999999999e-05 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.700672+0000", 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4983000000000001e-05 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.703245+0000", 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025733050000000001 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.703268+0000", 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2690000000000001e-05 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.743096+0000", 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.039828073999999998 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.743283+0000", 2024-04-02T05:00:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018735600000000001 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.743449+0000", 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016537999999999999 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.749805+0000", 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063567729999999996 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.752222+0000", 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002417088 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.752309+0000", 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6692000000000003e-05 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:47.752766+0000", 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045700599999999998 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25928.0:2 197.13 197:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:50:52.731551+0000", 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 576.56658595800002, 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.060042685999999998, 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25928", 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2010422105", 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731551+0000", 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731551+0000", 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731552+0000", 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2219999999999999e-06 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731558+0000", 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7010000000000001e-06 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731567+0000", 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6929999999999996e-06 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731576+0000", 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1169999999999993e-06 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731632+0000", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6555999999999999e-05 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.731640+0000", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0479999999999999e-06 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.779313+0000", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.047672895999999999 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.779440+0000", 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012715400000000001 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,3", 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.779651+0000", 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021031199999999999 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.780422+0000", 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00077147200000000004 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.785370+0000", 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049480360000000003 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.791461+0000", 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006090229 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.791520+0000", 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9036999999999997e-05 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:50:52.791594+0000", 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4213000000000002e-05 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16083.0:9 237.c 237:3197012c:::foo_rmxattr:head [rmxattr attr2 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:01.868723+0000", 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 507.42941360200001, 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.047108524999999998, 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16083", 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/3925174490", 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.868723+0000", 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.868723+0000", 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.868726+0000", 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.868744+0000", 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7580000000000001e-05 2024-04-02T05:00:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.868775+0000", 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1146999999999998e-05 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.868800+0000", 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5429999999999999e-05 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.871001+0000", 2024-04-02T05:00:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022001450000000001 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.871223+0000", 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000222835 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.915832+0000", 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.044608595000000001 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16122.0:1 244.5s0 244:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e282)", 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:01.884722+0000", 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 507.413415278, 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.023826379, 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16122", 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/3023083818", 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T05:00:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.884722+0000", 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.884722+0000", 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.884724+0000", 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3329999999999999e-06 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.884734+0000", 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4099999999999997e-06 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.884758+0000", 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.423e-05 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.884785+0000", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6849999999999999e-05 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.902512+0000", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017727634999999999 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:01.902519+0000", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2659999999999998e-06 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:02.908279+0000", 2024-04-02T05:00:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0057605270000001 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:02.908548+0000", 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026912799999999999 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.24938.0:192 40.17 40:ee53464d:::PoolCreationRaceObj:head [read 0~100] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e318)", 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:38.162227+0000", 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 471.13590990699998, 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046160304999999999, 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.24938", 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1735804308", 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 192 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.162227+0000", 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.162227+0000", 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.162230+0000", 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6800000000000002e-06 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.162243+0000", 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.309e-05 2024-04-02T05:00:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.162278+0000", 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4756999999999999e-05 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.162307+0000", 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8887e-05 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.164571+0000", 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022648070000000002 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.164590+0000", 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9006999999999999e-05 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.208221+0000", 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043630549999999997 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:38.208387+0000", 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000166527 2024-04-02T05:00:29.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16230.0:1 260.5s1 260:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:44.186225+0000", 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 465.11191208299999, 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.688184511, 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16230", 2024-04-02T05:00:29.330 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1129903165", 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.186225+0000", 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.186225+0000", 2024-04-02T05:00:29.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.186227+0000", 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.753e-06 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.186235+0000", 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8939999999999993e-06 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.186265+0000", 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0644e-05 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.186278+0000", 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2824e-05 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.187566+0000", 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001288001 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.187570+0000", 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2509999999999996e-06 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.864192+0000", 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.67662177499999998 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.864506+0000", 2024-04-02T05:00:29.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031393099999999999 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.865385+0000", 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00087885100000000005 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.873462+0000", 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0080772180000000006 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.873611+0000", 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014907599999999999 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:44.874409+0000", 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00079829300000000005 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.26339.0:2 263.13 263:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:49.862409+0000", 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 459.43572837400001, 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16111682599999999, 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.26339", 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/225957613", 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.862409+0000", 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.862409+0000", 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.862410+0000", 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5600000000000001e-06 2024-04-02T05:00:29.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.862419+0000", 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9900000000000003e-06 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.862432+0000", 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2782000000000001e-05 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.862445+0000", 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2942000000000001e-05 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.863032+0000", 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058717899999999996 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:49.863052+0000", 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0047999999999998e-05 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.017868+0000", 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15481578700000001 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.018196+0000", 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032787799999999998 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6", 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.018587+0000", 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039141300000000001 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:00:29.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.019898+0000", 2024-04-02T05:00:29.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013110859999999999 2024-04-02T05:00:29.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.023092+0000", 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031932039999999998 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.023259+0000", 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016718899999999999 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.023526+0000", 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026676800000000002 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16242.0:1 264.6s0 264:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:50.884115+0000", 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 458.41402199100003, 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.011933215, 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16242", 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2522793461", 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.884115+0000", 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.884115+0000", 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.884118+0000", 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4930000000000001e-06 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.884132+0000", 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4100000000000001e-05 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.884164+0000", 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2279e-05 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.884188+0000", 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3770000000000001e-05 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.899231+0000", 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015043393 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:50.899237+0000", 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7230000000000001e-06 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:51.886217+0000", 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.98698037100000002 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:51.886498+0000", 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028065799999999998 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:51.887590+0000", 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010924870000000001 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:51.889142+0000", 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015518140000000001 2024-04-02T05:00:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:51.895912+0000", 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067701949999999997 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:51.896048+0000", 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013593200000000001 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16737.0:7 317.3s0 317:de000000::::head [pg-hitset-ls] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e663)", 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:50.089043+0000", 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 39.209094045999997, 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026114139999999998, 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16737", 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/3364239127", 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089043+0000", 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089043+0000", 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089047+0000", 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.143e-06 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089063+0000", 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5607000000000002e-05 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089101+0000", 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8569999999999998e-05 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089144+0000", 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3092e-05 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.091551+0000", 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024064630000000002 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.091631+0000", 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9798999999999999e-05 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.091654+0000", 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.374e-05 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(10.7,QREQUEST,e:694,from:6,reservation_nonce:3)", 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:52.855180+0000", 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.442956592000002, 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059501500000000004, 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855180+0000", 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855180+0000", 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855183+0000", 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0959999999999999e-06 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855193+0000", 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.084e-05 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855222+0000", 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8099999999999999e-05 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855236+0000", 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4752999999999999e-05 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855380+0000", 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014421799999999999 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855393+0000", 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2403e-05 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.861003+0000", 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056103050000000003 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.861032+0000", 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8949e-05 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.861131+0000", 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8486000000000005e-05 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3678 4.e 4:7759931f:::notify.3:head [watch ping cookie 94452570228864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e691)", 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:55.054230+0000", 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.243906825000003, 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050602090000000004, 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3678 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054230+0000", 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054230+0000", 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054233+0000", 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2399999999999999e-06 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054248+0000", 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4413000000000001e-05 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054293+0000", 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.545e-05 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054319+0000", 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5363000000000001e-05 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.058860+0000", 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045414670000000004 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.059057+0000", 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000196615 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.059290+0000", 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000233661 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3688 4.e 4:7759931f:::notify.3:head [watch ping cookie 94452570228864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-04-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:00.053462+0000", 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.244675421, 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019937190000000001, 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3688 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053462+0000", 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053462+0000", 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053464+0000", 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9069999999999999e-06 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053474+0000", 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0783e-05 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053496+0000", 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1860000000000001e-05 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053520+0000", 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3765999999999999e-05 2024-04-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053758+0000", 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000238013 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053985+0000", 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000227421 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055455+0000", 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001469969 2024-04-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3689 4.9 4:93e5b521:::notify.7:head [watch ping cookie 94452570219648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:00.053591+0000", 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.244545992999999, 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023898980000000001, 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3689 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053591+0000", 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053591+0000", 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.053593+0000", 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4869999999999999e-06 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055516+0000", 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019232990000000001 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055545+0000", 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9064000000000002e-05 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055566+0000", 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1073e-05 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055681+0000", 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115031 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055819+0000", 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013787399999999999 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.055981+0000", 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016207 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:34 324.1b 324:d8000000:%01:::head [pgnls start_epoch 702] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e702)", 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:00.108363+0000", 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.189773998, 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005403349, 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 34 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108363+0000", 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108363+0000", 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108365+0000", 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.779e-06 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108372+0000", 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7050000000000002e-06 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108385+0000", 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3147e-05 2024-04-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108395+0000", 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0002e-05 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108554+0000", 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015898600000000001 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108564+0000", 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0566e-05 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.113427+0000", 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004862936 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.113479+0000", 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1322000000000001e-05 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.113766+0000", 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028790599999999999 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.4,QREQUEST,e:703,from:3,reservation_nonce:2)", 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:01.789592+0000", 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.508544832999998, 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056095809999999998, 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789592+0000", 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789592+0000", 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789595+0000", 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6630000000000002e-06 2024-04-02T05:00:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789609+0000", 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3986e-05 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789642+0000", 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2750000000000003e-05 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789659+0000", 2024-04-02T05:00:29.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6976000000000001e-05 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789861+0000", 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020217100000000001 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789874+0000", 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.278e-05 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795023+0000", 2024-04-02T05:00:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051492539999999998 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795074+0000", 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1669000000000003e-05 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795202+0000", 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012733199999999999 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.17,GRANT,e:718,from:2,reservation_nonce:0)", 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:16.827606+0000", 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.470530624, 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044158799999999997, 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827606+0000", 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827606+0000", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827608+0000", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2300000000000001e-06 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827615+0000", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.43e-06 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827637+0000", 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1633000000000001e-05 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827650+0000", 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3464e-05 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827747+0000", 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6892000000000007e-05 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.827773+0000", 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6308999999999998e-05 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:16.828048+0000", 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027462999999999999 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(4.18,GRANT,e:719,from:6,reservation_nonce:0)", 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:19.866737+0000", 2024-04-02T05:00:29.385 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.4313998059999999, 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045629399999999999, 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866737+0000", 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866737+0000", 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866739+0000", 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6300000000000001e-06 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866748+0000", 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4739999999999994e-06 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866775+0000", 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6386000000000001e-05 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866789+0000", 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.383e-05 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866915+0000", 2024-04-02T05:00:29.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126088 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.866949+0000", 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4266e-05 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:19.867194+0000", 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024462000000000003 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3729 4.e 4:7759931f:::notify.3:head [watch ping cookie 94452570228864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:20.069107+0000", 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.2290304059999997, 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036736690000000001, 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3729 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.390 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069107+0000", 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069107+0000", 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069109+0000", 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119999999999999e-06 2024-04-02T05:00:29.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069126+0000", 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6566999999999999e-05 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069154+0000", 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8282000000000001e-05 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069167+0000", 2024-04-02T05:00:29.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.295e-05 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072552+0000", 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033851459999999999 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072651+0000", 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9261999999999999e-05 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072780+0000", 2024-04-02T05:00:29.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012904999999999999 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3739 4.e 4:7759931f:::notify.3:head [watch ping cookie 94452570228864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e719)", 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:25.073550+0000", 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.224587294, 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010435990000000001, 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3739 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073550+0000", 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073550+0000", 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073554+0000", 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8299999999999998e-06 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073579+0000", 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5066e-05 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073625+0000", 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6609e-05 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073650+0000", 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5094000000000001e-05 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.074024+0000", 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037382499999999999 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.074259+0000", 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023457299999999999 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.074593+0000", 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033460199999999998 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3740 4.9 4:93e5b521:::notify.7:head [watch ping cookie 94452570219648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e719)", 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:25.073718+0000", 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.2244194960000003, 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00078441800000000005, 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:29.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3740 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073718+0000", 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:29.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073718+0000", 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073719+0000", 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.536e-06 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073728+0000", 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5339999999999999e-06 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:29.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073797+0000", 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9275000000000003e-05 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.073818+0000", 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0800000000000001e-05 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.074059+0000", 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024079900000000001 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.074253+0000", 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019435700000000001 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:25.074502+0000", 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024911699999999998 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:29.402 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:00:29.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:29.430 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:29.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:29.559 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:29.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:29.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:29 smithi069 bash[21528]: cluster 2024-04-02T05:00:28.288515+0000 mgr.y (mgr.14538) 654 : cluster 0 pgmap v1092: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:00:29.673 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:29 smithi069 bash[28904]: cluster 2024-04-02T05:00:28.288515+0000 mgr.y (mgr.14538) 654 : cluster 0 pgmap v1092: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:00:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:29.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:29.709 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1091,"stamp":"2024-04-02T05:00:26.285854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":243,"num_object_clones":4,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":1,"num_read":1524,"num_read_kb":1241,"num_write":931,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":795815,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2518,"ondisk_log_size":2518,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":642,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":698724,"kb_used_data":86324,"kb_used_omap":66,"kb_used_meta":612157,"kb_avail":655517340,"statfs":{"total":671965249536,"available":671249756160,"internally_reserved":0,"allocated":88395776,"data_stored":58489811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68049,"internal_metadata":626849327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006688"},"pg_stats":[{"pgid":"326.1f","version":"0'0","reported_seq":24,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896879+0000","last_change":"2024-04-02T05:00:13.105147+0000","last_active":"2024-04-02T05:00:20.896879+0000","last_peered":"2024-04-02T05:00:20.896879+0000","last_clean":"2024-04-02T05:00:20.896879+0000","last_became_active":"2024-04-02T05:00:13.104805+0000","last_became_peered":"2024-04-02T05:00:13.104805+0000","last_unstale":"2024-04-02T05:00:20.896879+0000","last_undegraded":"2024-04-02T05:00:20.896879+0000","last_fullsized":"2024-04-02T05:00:20.896879+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.362306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"326.1e","version":"719'1","reported_seq":28,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056287+0000","last_change":"2024-04-02T05:00:13.124302+0000","last_active":"2024-04-02T05:00:22.056287+0000","last_peered":"2024-04-02T05:00:22.056287+0000","last_clean":"2024-04-02T05:00:22.056287+0000","last_became_active":"2024-04-02T05:00:13.122793+0000","last_became_peered":"2024-04-02T05:00:13.122793+0000","last_unstale":"2024-04-02T05:00:22.056287+0000","last_undegraded":"2024-04-02T05:00:22.056287+0000","last_fullsized":"2024-04-02T05:00:22.056287+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:17.979966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.1d","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051998+0000","last_change":"2024-04-02T05:00:13.121673+0000","last_active":"2024-04-02T05:00:22.051998+0000","last_peered":"2024-04-02T05:00:22.051998+0000","last_clean":"2024-04-02T05:00:22.051998+0000","last_became_active":"2024-04-02T05:00:13.120233+0000","last_became_peered":"2024-04-02T05:00:13.120233+0000","last_unstale":"2024-04-02T05:00:22.051998+0000","last_undegraded":"2024-04-02T05:00:22.051998+0000","last_fullsized":"2024-04-02T05:00:22.051998+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:25.301126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"326.1c","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058079+0000","last_change":"2024-04-02T05:00:13.116918+0000","last_active":"2024-04-02T05:00:22.058079+0000","last_peered":"2024-04-02T05:00:22.058079+0000","last_clean":"2024-04-02T05:00:22.058079+0000","last_became_active":"2024-04-02T05:00:13.116644+0000","last_became_peered":"2024-04-02T05:00:13.116644+0000","last_unstale":"2024-04-02T05:00:22.058079+0000","last_undegraded":"2024-04-02T05:00:22.058079+0000","last_fullsized":"2024-04-02T05:00:22.058079+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.273892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.1b","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049539+0000","last_change":"2024-04-02T05:00:13.100660+0000","last_active":"2024-04-02T05:00:22.049539+0000","last_peered":"2024-04-02T05:00:22.049539+0000","last_clean":"2024-04-02T05:00:22.049539+0000","last_became_active":"2024-04-02T05:00:13.099806+0000","last_became_peered":"2024-04-02T05:00:13.099806+0000","last_unstale":"2024-04-02T05:00:22.049539+0000","last_undegraded":"2024-04-02T05:00:22.049539+0000","last_fullsized":"2024-04-02T05:00:22.049539+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:40.917723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"326.1a","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868319+0000","last_change":"2024-04-02T05:00:13.109845+0000","last_active":"2024-04-02T05:00:22.868319+0000","last_peered":"2024-04-02T05:00:22.868319+0000","last_clean":"2024-04-02T05:00:22.868319+0000","last_became_active":"2024-04-02T05:00:13.109481+0000","last_became_peered":"2024-04-02T05:00:13.109481+0000","last_unstale":"2024-04-02T05:00:22.868319+0000","last_undegraded":"2024-04-02T05:00:22.868319+0000","last_fullsized":"2024-04-02T05:00:22.868319+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:37.969743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.19","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107563+0000","last_change":"2024-04-02T05:00:13.124914+0000","last_active":"2024-04-02T05:00:23.107563+0000","last_peered":"2024-04-02T05:00:23.107563+0000","last_clean":"2024-04-02T05:00:23.107563+0000","last_became_active":"2024-04-02T05:00:13.123448+0000","last_became_peered":"2024-04-02T05:00:13.123448+0000","last_unstale":"2024-04-02T05:00:23.107563+0000","last_undegraded":"2024-04-02T05:00:23.107563+0000","last_fullsized":"2024-04-02T05:00:23.107563+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.700612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.18","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.106653+0000","last_change":"2024-04-02T05:00:13.125161+0000","last_active":"2024-04-02T05:00:23.106653+0000","last_peered":"2024-04-02T05:00:23.106653+0000","last_clean":"2024-04-02T05:00:23.106653+0000","last_became_active":"2024-04-02T05:00:13.123833+0000","last_became_peered":"2024-04-02T05:00:13.123833+0000","last_unstale":"2024-04-02T05:00:23.106653+0000","last_undegraded":"2024-04-02T05:00:23.106653+0000","last_fullsized":"2024-04-02T05:00:23.106653+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:28.334737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.17","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.052966+0000","last_change":"2024-04-02T05:00:13.103901+0000","last_active":"2024-04-02T05:00:22.052966+0000","last_peered":"2024-04-02T05:00:22.052966+0000","last_clean":"2024-04-02T05:00:22.052966+0000","last_became_active":"2024-04-02T05:00:13.102865+0000","last_became_peered":"2024-04-02T05:00:13.102865+0000","last_unstale":"2024-04-02T05:00:22.052966+0000","last_undegraded":"2024-04-02T05:00:22.052966+0000","last_fullsized":"2024-04-02T05:00:22.052966+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:31.428049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"326.16","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871482+0000","last_change":"2024-04-02T05:00:13.109067+0000","last_active":"2024-04-02T05:00:22.871482+0000","last_peered":"2024-04-02T05:00:22.871482+0000","last_clean":"2024-04-02T05:00:22.871482+0000","last_became_active":"2024-04-02T05:00:13.108722+0000","last_became_peered":"2024-04-02T05:00:13.108722+0000","last_unstale":"2024-04-02T05:00:22.871482+0000","last_undegraded":"2024-04-02T05:00:22.871482+0000","last_fullsized":"2024-04-02T05:00:22.871482+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:22.835514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.15","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108248+0000","last_change":"2024-04-02T05:00:13.126811+0000","last_active":"2024-04-02T05:00:23.108248+0000","last_peered":"2024-04-02T05:00:23.108248+0000","last_clean":"2024-04-02T05:00:23.108248+0000","last_became_active":"2024-04-02T05:00:13.126059+0000","last_became_peered":"2024-04-02T05:00:13.126059+0000","last_unstale":"2024-04-02T05:00:23.108248+0000","last_undegraded":"2024-04-02T05:00:23.108248+0000","last_fullsized":"2024-04-02T05:00:23.108248+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:12.533388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.14","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051553+0000","last_change":"2024-04-02T05:00:13.104275+0000","last_active":"2024-04-02T05:00:22.051553+0000","last_peered":"2024-04-02T05:00:22.051553+0000","last_clean":"2024-04-02T05:00:22.051553+0000","last_became_active":"2024-04-02T05:00:13.103288+0000","last_became_peered":"2024-04-02T05:00:13.103288+0000","last_unstale":"2024-04-02T05:00:22.051553+0000","last_undegraded":"2024-04-02T05:00:22.051553+0000","last_fullsized":"2024-04-02T05:00:22.051553+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:16.645097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"326.13","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868716+0000","last_change":"2024-04-02T05:00:13.110069+0000","last_active":"2024-04-02T05:00:22.868716+0000","last_peered":"2024-04-02T05:00:22.868716+0000","last_clean":"2024-04-02T05:00:22.868716+0000","last_became_active":"2024-04-02T05:00:13.109743+0000","last_became_peered":"2024-04-02T05:00:13.109743+0000","last_unstale":"2024-04-02T05:00:22.868716+0000","last_undegraded":"2024-04-02T05:00:22.868716+0000","last_fullsized":"2024-04-02T05:00:22.868716+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:12.478181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.12","version":"0'0","reported_seq":24,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896102+0000","last_change":"2024-04-02T05:00:13.107579+0000","last_active":"2024-04-02T05:00:20.896102+0000","last_peered":"2024-04-02T05:00:20.896102+0000","last_clean":"2024-04-02T05:00:20.896102+0000","last_became_active":"2024-04-02T05:00:13.107280+0000","last_became_peered":"2024-04-02T05:00:13.107280+0000","last_unstale":"2024-04-02T05:00:20.896102+0000","last_undegraded":"2024-04-02T05:00:20.896102+0000","last_fullsized":"2024-04-02T05:00:20.896102+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:17.117344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"326.11","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057219+0000","last_change":"2024-04-02T05:00:13.125175+0000","last_active":"2024-04-02T05:00:22.057219+0000","last_peered":"2024-04-02T05:00:22.057219+0000","last_clean":"2024-04-02T05:00:22.057219+0000","last_became_active":"2024-04-02T05:00:13.124752+0000","last_became_peered":"2024-04-02T05:00:13.124752+0000","last_unstale":"2024-04-02T05:00:22.057219+0000","last_undegraded":"2024-04-02T05:00:22.057219+0000","last_fullsized":"2024-04-02T05:00:22.057219+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.936102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.10","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053586+0000","last_change":"2024-04-02T05:00:13.110148+0000","last_active":"2024-04-02T05:00:22.053586+0000","last_peered":"2024-04-02T05:00:22.053586+0000","last_clean":"2024-04-02T05:00:22.053586+0000","last_became_active":"2024-04-02T05:00:13.109837+0000","last_became_peered":"2024-04-02T05:00:13.109837+0000","last_unstale":"2024-04-02T05:00:22.053586+0000","last_undegraded":"2024-04-02T05:00:22.053586+0000","last_fullsized":"2024-04-02T05:00:22.053586+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:12.138646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.f","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055485+0000","last_change":"2024-04-02T05:00:13.123528+0000","last_active":"2024-04-02T05:00:22.055485+0000","last_peered":"2024-04-02T05:00:22.055485+0000","last_clean":"2024-04-02T05:00:22.055485+0000","last_became_active":"2024-04-02T05:00:13.122786+0000","last_became_peered":"2024-04-02T05:00:13.122786+0000","last_unstale":"2024-04-02T05:00:22.055485+0000","last_undegraded":"2024-04-02T05:00:22.055485+0000","last_fullsized":"2024-04-02T05:00:22.055485+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.369735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.e","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051370+0000","last_change":"2024-04-02T05:00:13.098715+0000","last_active":"2024-04-02T05:00:22.051370+0000","last_peered":"2024-04-02T05:00:22.051370+0000","last_clean":"2024-04-02T05:00:22.051370+0000","last_became_active":"2024-04-02T05:00:13.097577+0000","last_became_peered":"2024-04-02T05:00:13.097577+0000","last_unstale":"2024-04-02T05:00:22.051370+0000","last_undegraded":"2024-04-02T05:00:22.051370+0000","last_fullsized":"2024-04-02T05:00:22.051370+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:14.567872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"326.d","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058613+0000","last_change":"2024-04-02T05:00:13.115422+0000","last_active":"2024-04-02T05:00:22.058613+0000","last_peered":"2024-04-02T05:00:22.058613+0000","last_clean":"2024-04-02T05:00:22.058613+0000","last_became_active":"2024-04-02T05:00:13.115029+0000","last_became_peered":"2024-04-02T05:00:13.115029+0000","last_unstale":"2024-04-02T05:00:22.058613+0000","last_undegraded":"2024-04-02T05:00:22.058613+0000","last_fullsized":"2024-04-02T05:00:22.058613+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:20.196224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.c","version":"0'0","reported_seq":24,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896244+0000","last_change":"2024-04-02T05:00:13.105066+0000","last_active":"2024-04-02T05:00:20.896244+0000","last_peered":"2024-04-02T05:00:20.896244+0000","last_clean":"2024-04-02T05:00:20.896244+0000","last_became_active":"2024-04-02T05:00:13.104708+0000","last_became_peered":"2024-04-02T05:00:13.104708+0000","last_unstale":"2024-04-02T05:00:20.896244+0000","last_undegraded":"2024-04-02T05:00:20.896244+0000","last_fullsized":"2024-04-02T05:00:20.896244+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:12.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"326.b","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542382+0000","last_change":"2024-04-02T05:00:13.101746+0000","last_active":"2024-04-02T05:00:22.542382+0000","last_peered":"2024-04-02T05:00:22.542382+0000","last_clean":"2024-04-02T05:00:22.542382+0000","last_became_active":"2024-04-02T05:00:13.100730+0000","last_became_peered":"2024-04-02T05:00:13.100730+0000","last_unstale":"2024-04-02T05:00:22.542382+0000","last_undegraded":"2024-04-02T05:00:22.542382+0000","last_fullsized":"2024-04-02T05:00:22.542382+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:36.124001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.a","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871000+0000","last_change":"2024-04-02T05:00:13.109812+0000","last_active":"2024-04-02T05:00:22.871000+0000","last_peered":"2024-04-02T05:00:22.871000+0000","last_clean":"2024-04-02T05:00:22.871000+0000","last_became_active":"2024-04-02T05:00:13.109423+0000","last_became_peered":"2024-04-02T05:00:13.109423+0000","last_unstale":"2024-04-02T05:00:22.871000+0000","last_undegraded":"2024-04-02T05:00:22.871000+0000","last_fullsized":"2024-04-02T05:00:22.871000+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:21.093829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.9","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.541968+0000","last_change":"2024-04-02T05:00:13.111432+0000","last_active":"2024-04-02T05:00:22.541968+0000","last_peered":"2024-04-02T05:00:22.541968+0000","last_clean":"2024-04-02T05:00:22.541968+0000","last_became_active":"2024-04-02T05:00:13.110552+0000","last_became_peered":"2024-04-02T05:00:13.110552+0000","last_unstale":"2024-04-02T05:00:22.541968+0000","last_undegraded":"2024-04-02T05:00:22.541968+0000","last_fullsized":"2024-04-02T05:00:22.541968+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:33.835534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.8","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.106695+0000","last_change":"2024-04-02T05:00:13.125699+0000","last_active":"2024-04-02T05:00:23.106695+0000","last_peered":"2024-04-02T05:00:23.106695+0000","last_clean":"2024-04-02T05:00:23.106695+0000","last_became_active":"2024-04-02T05:00:13.124475+0000","last_became_peered":"2024-04-02T05:00:13.124475+0000","last_unstale":"2024-04-02T05:00:23.106695+0000","last_undegraded":"2024-04-02T05:00:23.106695+0000","last_fullsized":"2024-04-02T05:00:23.106695+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:14.951636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.2","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107597+0000","last_change":"2024-04-02T05:00:13.125130+0000","last_active":"2024-04-02T05:00:23.107597+0000","last_peered":"2024-04-02T05:00:23.107597+0000","last_clean":"2024-04-02T05:00:23.107597+0000","last_became_active":"2024-04-02T05:00:13.123474+0000","last_became_peered":"2024-04-02T05:00:13.123474+0000","last_unstale":"2024-04-02T05:00:23.107597+0000","last_undegraded":"2024-04-02T05:00:23.107597+0000","last_fullsized":"2024-04-02T05:00:23.107597+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:34.995461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.7","version":"0'0","reported_seq":63,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108592+0000","last_change":"2024-04-02T04:59:57.068679+0000","last_active":"2024-04-02T05:00:23.108592+0000","last_peered":"2024-04-02T05:00:23.108592+0000","last_clean":"2024-04-02T05:00:23.108592+0000","last_became_active":"2024-04-02T04:59:57.067691+0000","last_became_peered":"2024-04-02T04:59:57.067691+0000","last_unstale":"2024-04-02T05:00:23.108592+0000","last_undegraded":"2024-04-02T05:00:23.108592+0000","last_fullsized":"2024-04-02T05:00:23.108592+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:56.590992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.3","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058539+0000","last_change":"2024-04-02T05:00:13.118849+0000","last_active":"2024-04-02T05:00:22.058539+0000","last_peered":"2024-04-02T05:00:22.058539+0000","last_clean":"2024-04-02T05:00:22.058539+0000","last_became_active":"2024-04-02T05:00:13.118662+0000","last_became_peered":"2024-04-02T05:00:13.118662+0000","last_unstale":"2024-04-02T05:00:22.058539+0000","last_undegraded":"2024-04-02T05:00:22.058539+0000","last_fullsized":"2024-04-02T05:00:22.058539+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:24.498425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.6","version":"716'3","reported_seq":67,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055256+0000","last_change":"2024-04-02T04:59:57.064095+0000","last_active":"2024-04-02T05:00:22.055256+0000","last_peered":"2024-04-02T05:00:22.055256+0000","last_clean":"2024-04-02T05:00:22.055256+0000","last_became_active":"2024-04-02T04:59:57.063760+0000","last_became_peered":"2024-04-02T04:59:57.063760+0000","last_unstale":"2024-04-02T05:00:22.055256+0000","last_undegraded":"2024-04-02T05:00:22.055256+0000","last_fullsized":"2024-04-02T05:00:22.055256+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:23.656207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.0","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868801+0000","last_change":"2024-04-02T05:00:13.103258+0000","last_active":"2024-04-02T05:00:22.868801+0000","last_peered":"2024-04-02T05:00:22.868801+0000","last_clean":"2024-04-02T05:00:22.868801+0000","last_became_active":"2024-04-02T05:00:13.102828+0000","last_became_peered":"2024-04-02T05:00:13.102828+0000","last_unstale":"2024-04-02T05:00:22.868801+0000","last_undegraded":"2024-04-02T05:00:22.868801+0000","last_fullsized":"2024-04-02T05:00:22.868801+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:36.215003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.5","version":"716'3","reported_seq":65,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869793+0000","last_change":"2024-04-02T04:59:57.057724+0000","last_active":"2024-04-02T05:00:22.869793+0000","last_peered":"2024-04-02T05:00:22.869793+0000","last_clean":"2024-04-02T05:00:22.869793+0000","last_became_active":"2024-04-02T04:59:57.057395+0000","last_became_peered":"2024-04-02T04:59:57.057395+0000","last_unstale":"2024-04-02T05:00:22.869793+0000","last_undegraded":"2024-04-02T05:00:22.869793+0000","last_fullsized":"2024-04-02T05:00:22.869793+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.277281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.1","version":"719'1","reported_seq":28,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868681+0000","last_change":"2024-04-02T05:00:13.108500+0000","last_active":"2024-04-02T05:00:22.868681+0000","last_peered":"2024-04-02T05:00:22.868681+0000","last_clean":"2024-04-02T05:00:22.868681+0000","last_became_active":"2024-04-02T05:00:13.106024+0000","last_became_peered":"2024-04-02T05:00:13.106024+0000","last_unstale":"2024-04-02T05:00:22.868681+0000","last_undegraded":"2024-04-02T05:00:22.868681+0000","last_fullsized":"2024-04-02T05:00:22.868681+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:24.540901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.4","version":"707'1","reported_seq":66,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869621+0000","last_change":"2024-04-02T04:59:57.057843+0000","last_active":"2024-04-02T05:00:22.869621+0000","last_peered":"2024-04-02T05:00:22.869621+0000","last_clean":"2024-04-02T05:00:22.869621+0000","last_became_active":"2024-04-02T04:59:57.057575+0000","last_became_peered":"2024-04-02T04:59:57.057575+0000","last_unstale":"2024-04-02T05:00:22.869621+0000","last_undegraded":"2024-04-02T05:00:22.869621+0000","last_fullsized":"2024-04-02T05:00:22.869621+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:14.953143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.6","version":"0'0","reported_seq":35,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.543282+0000","last_change":"2024-04-02T05:00:17.873073+0000","last_active":"2024-04-02T05:00:22.543282+0000","last_peered":"2024-04-02T05:00:22.543282+0000","last_clean":"2024-04-02T05:00:22.543282+0000","last_became_active":"2024-04-02T05:00:13.106032+0000","last_became_peered":"2024-04-02T05:00:13.106032+0000","last_unstale":"2024-04-02T05:00:22.543282+0000","last_undegraded":"2024-04-02T05:00:22.543282+0000","last_fullsized":"2024-04-02T05:00:22.543282+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:17.872967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:17.872967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:36.628619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.3","version":"0'0","reported_seq":66,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057897+0000","last_change":"2024-04-02T04:59:57.053639+0000","last_active":"2024-04-02T05:00:22.057897+0000","last_peered":"2024-04-02T05:00:22.057897+0000","last_clean":"2024-04-02T05:00:22.057897+0000","last_became_active":"2024-04-02T04:59:57.053322+0000","last_became_peered":"2024-04-02T04:59:57.053322+0000","last_unstale":"2024-04-02T05:00:22.057897+0000","last_undegraded":"2024-04-02T05:00:22.057897+0000","last_fullsized":"2024-04-02T05:00:22.057897+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:08.528776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.7","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055591+0000","last_change":"2024-04-02T05:00:13.118242+0000","last_active":"2024-04-02T05:00:22.055591+0000","last_peered":"2024-04-02T05:00:22.055591+0000","last_clean":"2024-04-02T05:00:22.055591+0000","last_became_active":"2024-04-02T05:00:13.117858+0000","last_became_peered":"2024-04-02T05:00:13.117858+0000","last_unstale":"2024-04-02T05:00:22.055591+0000","last_undegraded":"2024-04-02T05:00:22.055591+0000","last_fullsized":"2024-04-02T05:00:22.055591+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.826903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.2","version":"0'0","reported_seq":63,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053887+0000","last_change":"2024-04-02T04:59:57.064033+0000","last_active":"2024-04-02T05:00:22.053887+0000","last_peered":"2024-04-02T05:00:22.053887+0000","last_clean":"2024-04-02T05:00:22.053887+0000","last_became_active":"2024-04-02T04:59:57.062555+0000","last_became_peered":"2024-04-02T04:59:57.062555+0000","last_unstale":"2024-04-02T05:00:22.053887+0000","last_undegraded":"2024-04-02T05:00:22.053887+0000","last_fullsized":"2024-04-02T05:00:22.053887+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:03.495282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.4","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050908+0000","last_change":"2024-04-02T05:00:13.110284+0000","last_active":"2024-04-02T05:00:22.050908+0000","last_peered":"2024-04-02T05:00:22.050908+0000","last_clean":"2024-04-02T05:00:22.050908+0000","last_became_active":"2024-04-02T05:00:13.109380+0000","last_became_peered":"2024-04-02T05:00:13.109380+0000","last_unstale":"2024-04-02T05:00:22.050908+0000","last_undegraded":"2024-04-02T05:00:22.050908+0000","last_fullsized":"2024-04-02T05:00:22.050908+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:16.873224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"323.1","version":"716'7","reported_seq":71,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.106805+0000","last_change":"2024-04-02T04:59:57.061855+0000","last_active":"2024-04-02T05:00:23.106805+0000","last_peered":"2024-04-02T05:00:23.106805+0000","last_clean":"2024-04-02T05:00:23.106805+0000","last_became_active":"2024-04-02T04:59:57.060728+0000","last_became_peered":"2024-04-02T04:59:57.060728+0000","last_unstale":"2024-04-02T05:00:23.106805+0000","last_undegraded":"2024-04-02T05:00:23.106805+0000","last_fullsized":"2024-04-02T05:00:23.106805+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:10.133245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.5","version":"0'0","reported_seq":26,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109001+0000","last_change":"2024-04-02T05:00:13.126688+0000","last_active":"2024-04-02T05:00:23.109001+0000","last_peered":"2024-04-02T05:00:23.109001+0000","last_clean":"2024-04-02T05:00:23.109001+0000","last_became_active":"2024-04-02T05:00:13.126130+0000","last_became_peered":"2024-04-02T05:00:13.126130+0000","last_unstale":"2024-04-02T05:00:23.109001+0000","last_undegraded":"2024-04-02T05:00:23.109001+0000","last_fullsized":"2024-04-02T05:00:23.109001+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":714,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.069607+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.069607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:24.705473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.0","version":"0'0","reported_seq":63,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053585+0000","last_change":"2024-04-02T04:59:57.064023+0000","last_active":"2024-04-02T05:00:22.053585+0000","last_peered":"2024-04-02T05:00:22.053585+0000","last_clean":"2024-04-02T05:00:22.053585+0000","last_became_active":"2024-04-02T04:59:57.063730+0000","last_became_peered":"2024-04-02T04:59:57.063730+0000","last_unstale":"2024-04-02T05:00:22.053585+0000","last_undegraded":"2024-04-02T05:00:22.053585+0000","last_fullsized":"2024-04-02T05:00:22.053585+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.027374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:18.010995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":567,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871309+0000","last_change":"2024-04-02T05:00:12.831470+0000","last_active":"2024-04-02T05:00:22.871309+0000","last_peered":"2024-04-02T05:00:22.871309+0000","last_clean":"2024-04-02T05:00:22.871309+0000","last_became_active":"2024-04-02T04:54:56.017309+0000","last_became_peered":"2024-04-02T04:54:56.017309+0000","last_unstale":"2024-04-02T05:00:22.871309+0000","last_undegraded":"2024-04-02T05:00:22.871309+0000","last_fullsized":"2024-04-02T05:00:22.871309+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.831241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.831241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:15.427607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1230,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.106785+0000","last_change":"2024-04-02T04:59:03.454678+0000","last_active":"2024-04-02T05:00:23.106785+0000","last_peered":"2024-04-02T05:00:23.106785+0000","last_clean":"2024-04-02T05:00:23.106785+0000","last_became_active":"2024-04-02T04:54:55.990081+0000","last_became_peered":"2024-04-02T04:54:55.990081+0000","last_unstale":"2024-04-02T05:00:23.106785+0000","last_undegraded":"2024-04-02T05:00:23.106785+0000","last_fullsized":"2024-04-02T05:00:23.106785+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:03.454565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:03.454565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"415'28","reported_seq":876,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109003+0000","last_change":"2024-04-02T04:59:55.084090+0000","last_active":"2024-04-02T05:00:23.109003+0000","last_peered":"2024-04-02T05:00:23.109003+0000","last_clean":"2024-04-02T05:00:23.109003+0000","last_became_active":"2024-04-02T04:59:55.068785+0000","last_became_peered":"2024-04-02T04:59:55.068785+0000","last_unstale":"2024-04-02T05:00:23.109003+0000","last_undegraded":"2024-04-02T05:00:23.109003+0000","last_fullsized":"2024-04-02T05:00:23.109003+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'28","last_scrub_stamp":"2024-04-02T04:59:09.515130+0000","last_deep_scrub":"415'28","last_deep_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.515130+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1358,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896978+0000","last_change":"2024-04-02T05:00:04.123410+0000","last_active":"2024-04-02T05:00:20.896978+0000","last_peered":"2024-04-02T05:00:20.896978+0000","last_clean":"2024-04-02T05:00:20.896978+0000","last_became_active":"2024-04-02T04:59:00.088594+0000","last_became_peered":"2024-04-02T04:59:00.088594+0000","last_unstale":"2024-04-02T05:00:20.896978+0000","last_undegraded":"2024-04-02T05:00:20.896978+0000","last_fullsized":"2024-04-02T05:00:20.896978+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:04.123271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:00:04.123271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:07.968180+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":562,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049635+0000","last_change":"2024-04-02T04:58:59.898789+0000","last_active":"2024-04-02T05:00:22.049635+0000","last_peered":"2024-04-02T05:00:22.049635+0000","last_clean":"2024-04-02T05:00:22.049635+0000","last_became_active":"2024-04-02T04:54:56.002149+0000","last_became_peered":"2024-04-02T04:54:56.002149+0000","last_unstale":"2024-04-02T05:00:22.049635+0000","last_undegraded":"2024-04-02T05:00:22.049635+0000","last_fullsized":"2024-04-02T05:00:22.049635+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_clean_scrub_stamp":"2024-04-02T04:58:59.898499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1223,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896532+0000","last_change":"2024-04-02T05:00:16.139470+0000","last_active":"2024-04-02T05:00:20.896532+0000","last_peered":"2024-04-02T05:00:20.896532+0000","last_clean":"2024-04-02T05:00:20.896532+0000","last_became_active":"2024-04-02T04:59:00.080466+0000","last_became_peered":"2024-04-02T04:59:00.080466+0000","last_unstale":"2024-04-02T05:00:20.896532+0000","last_undegraded":"2024-04-02T05:00:20.896532+0000","last_fullsized":"2024-04-02T05:00:20.896532+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:16.139352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.139352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:36.444907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1344,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.867310+0000","last_change":"2024-04-02T04:57:36.143862+0000","last_active":"2024-04-02T05:00:22.867310+0000","last_peered":"2024-04-02T05:00:22.867310+0000","last_clean":"2024-04-02T05:00:22.867310+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T05:00:22.867310+0000","last_undegraded":"2024-04-02T05:00:22.867310+0000","last_fullsized":"2024-04-02T05:00:22.867310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:36.143762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T04:57:36.143762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"402'25","reported_seq":189,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058873+0000","last_change":"2024-04-02T04:59:01.058923+0000","last_active":"2024-04-02T05:00:22.058873+0000","last_peered":"2024-04-02T05:00:22.058873+0000","last_clean":"2024-04-02T05:00:22.058873+0000","last_became_active":"2024-04-02T04:59:00.107041+0000","last_became_peered":"2024-04-02T04:59:00.107041+0000","last_unstale":"2024-04-02T05:00:22.058873+0000","last_undegraded":"2024-04-02T05:00:22.058873+0000","last_fullsized":"2024-04-02T05:00:22.058873+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:59:01.058796+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T04:59:01.058796+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1370,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055914+0000","last_change":"2024-04-02T04:59:20.068275+0000","last_active":"2024-04-02T05:00:22.055914+0000","last_peered":"2024-04-02T05:00:22.055914+0000","last_clean":"2024-04-02T05:00:22.055914+0000","last_became_active":"2024-04-02T04:59:20.067761+0000","last_became_peered":"2024-04-02T04:59:20.067761+0000","last_unstale":"2024-04-02T05:00:22.055914+0000","last_undegraded":"2024-04-02T05:00:22.055914+0000","last_fullsized":"2024-04-02T05:00:22.055914+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:35.987316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T04:57:35.987316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1325,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869856+0000","last_change":"2024-04-02T04:59:53.764657+0000","last_active":"2024-04-02T05:00:22.869856+0000","last_peered":"2024-04-02T05:00:22.869856+0000","last_clean":"2024-04-02T05:00:22.869856+0000","last_became_active":"2024-04-02T04:53:54.935778+0000","last_became_peered":"2024-04-02T04:53:54.935778+0000","last_unstale":"2024-04-02T05:00:22.869856+0000","last_undegraded":"2024-04-02T05:00:22.869856+0000","last_fullsized":"2024-04-02T05:00:22.869856+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:53.764569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:59:53.764569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:20.141472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1213,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871385+0000","last_change":"2024-04-02T04:59:05.997307+0000","last_active":"2024-04-02T05:00:22.871385+0000","last_peered":"2024-04-02T05:00:22.871385+0000","last_clean":"2024-04-02T05:00:22.871385+0000","last_became_active":"2024-04-02T04:59:00.102255+0000","last_became_peered":"2024-04-02T04:59:00.102255+0000","last_unstale":"2024-04-02T05:00:22.871385+0000","last_undegraded":"2024-04-02T05:00:22.871385+0000","last_fullsized":"2024-04-02T05:00:22.871385+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:05.997205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:05.997205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:30.765941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1341,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050247+0000","last_change":"2024-04-02T05:00:17.020486+0000","last_active":"2024-04-02T05:00:22.050247+0000","last_peered":"2024-04-02T05:00:22.050247+0000","last_clean":"2024-04-02T05:00:22.050247+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T05:00:22.050247+0000","last_undegraded":"2024-04-02T05:00:22.050247+0000","last_fullsized":"2024-04-02T05:00:22.050247+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:17.020256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:17.020256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:42.370204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"402'31","reported_seq":937,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109142+0000","last_change":"2024-04-02T04:59:00.203951+0000","last_active":"2024-04-02T05:00:23.109142+0000","last_peered":"2024-04-02T05:00:23.109142+0000","last_clean":"2024-04-02T05:00:23.109142+0000","last_became_active":"2024-04-02T04:59:00.137536+0000","last_became_peered":"2024-04-02T04:59:00.137536+0000","last_unstale":"2024-04-02T05:00:23.109142+0000","last_undegraded":"2024-04-02T05:00:23.109142+0000","last_fullsized":"2024-04-02T05:00:23.109142+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'31","last_scrub_stamp":"2024-04-02T04:58:57.454000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:57.454000+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1351,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896377+0000","last_change":"2024-04-02T04:58:45.278588+0000","last_active":"2024-04-02T05:00:20.896377+0000","last_peered":"2024-04-02T05:00:20.896377+0000","last_clean":"2024-04-02T05:00:20.896377+0000","last_became_active":"2024-04-02T04:54:55.951772+0000","last_became_peered":"2024-04-02T04:54:55.951772+0000","last_unstale":"2024-04-02T05:00:20.896377+0000","last_undegraded":"2024-04-02T05:00:20.896377+0000","last_fullsized":"2024-04-02T05:00:20.896377+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:45.278356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T04:58:45.278356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":837,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107768+0000","last_change":"2024-04-02T05:00:15.498619+0000","last_active":"2024-04-02T05:00:23.107768+0000","last_peered":"2024-04-02T05:00:23.107768+0000","last_clean":"2024-04-02T05:00:23.107768+0000","last_became_active":"2024-04-02T04:59:00.134066+0000","last_became_peered":"2024-04-02T04:59:00.134066+0000","last_unstale":"2024-04-02T05:00:23.107768+0000","last_undegraded":"2024-04-02T05:00:23.107768+0000","last_fullsized":"2024-04-02T05:00:23.107768+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:15.498312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:15.498312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:42.971115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"101'128","reported_seq":1341,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.894906+0000","last_change":"2024-04-02T05:00:05.157395+0000","last_active":"2024-04-02T05:00:20.894906+0000","last_peered":"2024-04-02T05:00:20.894906+0000","last_clean":"2024-04-02T05:00:20.894906+0000","last_became_active":"2024-04-02T04:59:03.044786+0000","last_became_peered":"2024-04-02T04:59:03.044786+0000","last_unstale":"2024-04-02T05:00:20.894906+0000","last_undegraded":"2024-04-02T05:00:20.894906+0000","last_fullsized":"2024-04-02T05:00:20.894906+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:05.157293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:05.157293+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:11.902949+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":183,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058642+0000","last_change":"2024-04-02T05:00:21.700581+0000","last_active":"2024-04-02T05:00:22.058642+0000","last_peered":"2024-04-02T05:00:22.058642+0000","last_clean":"2024-04-02T05:00:22.058642+0000","last_became_active":"2024-04-02T04:59:20.067247+0000","last_became_peered":"2024-04-02T04:59:20.067247+0000","last_unstale":"2024-04-02T05:00:22.058642+0000","last_undegraded":"2024-04-02T05:00:22.058642+0000","last_fullsized":"2024-04-02T05:00:22.058642+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:21.700487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:42.254216+0000","last_clean_scrub_stamp":"2024-04-02T05:00:21.700487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"531'52","reported_seq":1470,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050671+0000","last_change":"2024-04-02T04:58:39.418912+0000","last_active":"2024-04-02T05:00:22.050671+0000","last_peered":"2024-04-02T05:00:22.050671+0000","last_clean":"2024-04-02T05:00:22.050671+0000","last_became_active":"2024-04-02T04:58:39.391176+0000","last_became_peered":"2024-04-02T04:58:39.391176+0000","last_unstale":"2024-04-02T05:00:22.050671+0000","last_undegraded":"2024-04-02T05:00:22.050671+0000","last_fullsized":"2024-04-02T05:00:22.050671+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'52","last_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_deep_scrub":"531'52","last_deep_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_clean_scrub_stamp":"2024-04-02T04:56:30.024650+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":846,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107988+0000","last_change":"2024-04-02T04:59:20.063523+0000","last_active":"2024-04-02T05:00:23.107988+0000","last_peered":"2024-04-02T05:00:23.107988+0000","last_clean":"2024-04-02T05:00:23.107988+0000","last_became_active":"2024-04-02T04:59:20.057768+0000","last_became_peered":"2024-04-02T04:59:20.057768+0000","last_unstale":"2024-04-02T05:00:23.107988+0000","last_undegraded":"2024-04-02T05:00:23.107988+0000","last_fullsized":"2024-04-02T05:00:23.107988+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:38.410916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:38.410916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"662'13","reported_seq":318,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896329+0000","last_change":"2024-04-02T04:59:59.024108+0000","last_active":"2024-04-02T05:00:20.896329+0000","last_peered":"2024-04-02T05:00:20.896329+0000","last_clean":"2024-04-02T05:00:20.896329+0000","last_became_active":"2024-04-02T04:59:20.070441+0000","last_became_peered":"2024-04-02T04:59:20.070441+0000","last_unstale":"2024-04-02T05:00:20.896329+0000","last_undegraded":"2024-04-02T05:00:20.896329+0000","last_fullsized":"2024-04-02T05:00:20.896329+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'13","last_scrub_stamp":"2024-04-02T04:59:59.024010+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:55:34.654306+0000","last_clean_scrub_stamp":"2024-04-02T04:59:59.024010+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:25.648046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1361,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.897453+0000","last_change":"2024-04-02T05:00:10.265065+0000","last_active":"2024-04-02T05:00:20.897453+0000","last_peered":"2024-04-02T05:00:20.897453+0000","last_clean":"2024-04-02T05:00:20.897453+0000","last_became_active":"2024-04-02T04:59:00.100630+0000","last_became_peered":"2024-04-02T04:59:00.100630+0000","last_unstale":"2024-04-02T05:00:20.897453+0000","last_undegraded":"2024-04-02T05:00:20.897453+0000","last_fullsized":"2024-04-02T05:00:20.897453+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:10.264855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:10.264855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:35.503976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'128","reported_seq":1121,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.106546+0000","last_change":"2024-04-02T04:59:59.396132+0000","last_active":"2024-04-02T05:00:23.106546+0000","last_peered":"2024-04-02T05:00:23.106546+0000","last_clean":"2024-04-02T05:00:23.106546+0000","last_became_active":"2024-04-02T04:59:03.039047+0000","last_became_peered":"2024-04-02T04:59:03.039047+0000","last_unstale":"2024-04-02T05:00:23.106546+0000","last_undegraded":"2024-04-02T05:00:23.106546+0000","last_fullsized":"2024-04-02T05:00:23.106546+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:59:59.396006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:59:59.396006+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.279921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1193,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054596+0000","last_change":"2024-04-02T04:56:30.499726+0000","last_active":"2024-04-02T05:00:22.054596+0000","last_peered":"2024-04-02T05:00:22.054596+0000","last_clean":"2024-04-02T05:00:22.054596+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T05:00:22.054596+0000","last_undegraded":"2024-04-02T05:00:22.054596+0000","last_fullsized":"2024-04-02T05:00:22.054596+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:30.499514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:56:30.499514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1353,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542523+0000","last_change":"2024-04-02T05:00:08.902712+0000","last_active":"2024-04-02T05:00:22.542523+0000","last_peered":"2024-04-02T05:00:22.542523+0000","last_clean":"2024-04-02T05:00:22.542523+0000","last_became_active":"2024-04-02T04:59:00.138222+0000","last_became_peered":"2024-04-02T04:59:00.138222+0000","last_unstale":"2024-04-02T05:00:22.542523+0000","last_undegraded":"2024-04-02T05:00:22.542523+0000","last_fullsized":"2024-04-02T05:00:22.542523+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:08.902496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T05:00:08.902496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:25.583748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"402'26","reported_seq":181,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058076+0000","last_change":"2024-04-02T04:59:00.230647+0000","last_active":"2024-04-02T05:00:22.058076+0000","last_peered":"2024-04-02T05:00:22.058076+0000","last_clean":"2024-04-02T05:00:22.058076+0000","last_became_active":"2024-04-02T04:59:00.135793+0000","last_became_peered":"2024-04-02T04:59:00.135793+0000","last_unstale":"2024-04-02T05:00:22.058076+0000","last_undegraded":"2024-04-02T05:00:22.058076+0000","last_fullsized":"2024-04-02T05:00:22.058076+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T04:58:51.009683+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T04:58:51.009683+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1365,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054773+0000","last_change":"2024-04-02T04:59:00.106966+0000","last_active":"2024-04-02T05:00:22.054773+0000","last_peered":"2024-04-02T05:00:22.054773+0000","last_clean":"2024-04-02T05:00:22.054773+0000","last_became_active":"2024-04-02T04:59:00.106616+0000","last_became_peered":"2024-04-02T04:59:00.106616+0000","last_unstale":"2024-04-02T05:00:22.054773+0000","last_undegraded":"2024-04-02T05:00:22.054773+0000","last_fullsized":"2024-04-02T05:00:22.054773+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:46.934539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T04:58:46.934539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":135,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056027+0000","last_change":"2024-04-02T05:00:13.216154+0000","last_active":"2024-04-02T05:00:22.056027+0000","last_peered":"2024-04-02T05:00:22.056027+0000","last_clean":"2024-04-02T05:00:22.056027+0000","last_became_active":"2024-04-02T04:59:20.083304+0000","last_became_peered":"2024-04-02T04:59:20.083304+0000","last_unstale":"2024-04-02T05:00:22.056027+0000","last_undegraded":"2024-04-02T05:00:22.056027+0000","last_fullsized":"2024-04-02T05:00:22.056027+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:13.215949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:13.215949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:30.543069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"101'128","reported_seq":1294,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.895541+0000","last_change":"2024-04-02T04:58:44.271422+0000","last_active":"2024-04-02T05:00:20.895541+0000","last_peered":"2024-04-02T05:00:20.895541+0000","last_clean":"2024-04-02T05:00:20.895541+0000","last_became_active":"2024-04-02T04:55:32.987580+0000","last_became_peered":"2024-04-02T04:55:32.987580+0000","last_unstale":"2024-04-02T05:00:20.895541+0000","last_undegraded":"2024-04-02T05:00:20.895541+0000","last_fullsized":"2024-04-02T05:00:20.895541+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:58:44.271259+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:34.875105+0000","last_clean_scrub_stamp":"2024-04-02T04:58:44.271259+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":154,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107121+0000","last_change":"2024-04-02T04:59:20.087407+0000","last_active":"2024-04-02T05:00:23.107121+0000","last_peered":"2024-04-02T05:00:23.107121+0000","last_clean":"2024-04-02T05:00:23.107121+0000","last_became_active":"2024-04-02T04:59:20.070326+0000","last_became_peered":"2024-04-02T04:59:20.070326+0000","last_unstale":"2024-04-02T05:00:23.107121+0000","last_undegraded":"2024-04-02T05:00:23.107121+0000","last_fullsized":"2024-04-02T05:00:23.107121+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T04:58:58.039256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:58.039256+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"402'48","reported_seq":189,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056832+0000","last_change":"2024-04-02T04:59:00.209092+0000","last_active":"2024-04-02T05:00:22.056832+0000","last_peered":"2024-04-02T05:00:22.056832+0000","last_clean":"2024-04-02T05:00:22.056832+0000","last_became_active":"2024-04-02T04:59:00.122446+0000","last_became_peered":"2024-04-02T04:59:00.122446+0000","last_unstale":"2024-04-02T05:00:22.056832+0000","last_undegraded":"2024-04-02T05:00:22.056832+0000","last_fullsized":"2024-04-02T05:00:22.056832+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'48","last_scrub_stamp":"2024-04-02T04:58:48.877784+0000","last_deep_scrub":"103'36","last_deep_scrub_stamp":"2024-04-02T04:45:00.578762+0000","last_clean_scrub_stamp":"2024-04-02T04:58:48.877784+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1370,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054707+0000","last_change":"2024-04-02T05:00:05.185460+0000","last_active":"2024-04-02T05:00:22.054707+0000","last_peered":"2024-04-02T05:00:22.054707+0000","last_clean":"2024-04-02T05:00:22.054707+0000","last_became_active":"2024-04-02T04:59:00.107474+0000","last_became_peered":"2024-04-02T04:59:00.107474+0000","last_unstale":"2024-04-02T05:00:22.054707+0000","last_undegraded":"2024-04-02T05:00:22.054707+0000","last_fullsized":"2024-04-02T05:00:22.054707+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:05.185374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:26.486667+0000","last_clean_scrub_stamp":"2024-04-02T05:00:05.185374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:31.389428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1298,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.897550+0000","last_change":"2024-04-02T04:59:10.460466+0000","last_active":"2024-04-02T05:00:20.897550+0000","last_peered":"2024-04-02T05:00:20.897550+0000","last_clean":"2024-04-02T05:00:20.897550+0000","last_became_active":"2024-04-02T04:59:00.111699+0000","last_became_peered":"2024-04-02T04:59:00.111699+0000","last_unstale":"2024-04-02T05:00:20.897550+0000","last_undegraded":"2024-04-02T05:00:20.897550+0000","last_fullsized":"2024-04-02T05:00:20.897550+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:10.460372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:10.460372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:24.278025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":562,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056778+0000","last_change":"2024-04-02T04:57:46.972993+0000","last_active":"2024-04-02T05:00:22.056778+0000","last_peered":"2024-04-02T05:00:22.056778+0000","last_clean":"2024-04-02T05:00:22.056778+0000","last_became_active":"2024-04-02T04:54:55.956870+0000","last_became_peered":"2024-04-02T04:54:55.956870+0000","last_unstale":"2024-04-02T05:00:22.056778+0000","last_undegraded":"2024-04-02T05:00:22.056778+0000","last_fullsized":"2024-04-02T05:00:22.056778+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:46.972794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T04:57:46.972794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1194,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049179+0000","last_change":"2024-04-02T04:59:20.039821+0000","last_active":"2024-04-02T05:00:22.049179+0000","last_peered":"2024-04-02T05:00:22.049179+0000","last_clean":"2024-04-02T05:00:22.049179+0000","last_became_active":"2024-04-02T04:59:20.037443+0000","last_became_peered":"2024-04-02T04:59:20.037443+0000","last_unstale":"2024-04-02T05:00:22.049179+0000","last_undegraded":"2024-04-02T05:00:22.049179+0000","last_fullsized":"2024-04-02T05:00:22.049179+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:32.703374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:57:32.703374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"402'30","reported_seq":885,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.895192+0000","last_change":"2024-04-02T04:59:20.101270+0000","last_active":"2024-04-02T05:00:20.895192+0000","last_peered":"2024-04-02T05:00:20.895192+0000","last_clean":"2024-04-02T05:00:20.895192+0000","last_became_active":"2024-04-02T04:59:20.065524+0000","last_became_peered":"2024-04-02T04:59:20.065524+0000","last_unstale":"2024-04-02T05:00:20.895192+0000","last_undegraded":"2024-04-02T05:00:20.895192+0000","last_fullsized":"2024-04-02T05:00:20.895192+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:59:09.140586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.140586+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:37.757375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":831,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107051+0000","last_change":"2024-04-02T04:58:47.478098+0000","last_active":"2024-04-02T05:00:23.107051+0000","last_peered":"2024-04-02T05:00:23.107051+0000","last_clean":"2024-04-02T05:00:23.107051+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T05:00:23.107051+0000","last_undegraded":"2024-04-02T05:00:23.107051+0000","last_fullsized":"2024-04-02T05:00:23.107051+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:47.477862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T04:58:47.477862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"642'13","reported_seq":1570,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.543629+0000","last_change":"2024-04-02T05:00:03.815253+0000","last_active":"2024-04-02T05:00:22.543629+0000","last_peered":"2024-04-02T05:00:22.543629+0000","last_clean":"2024-04-02T05:00:22.543629+0000","last_became_active":"2024-04-02T04:59:00.136703+0000","last_became_peered":"2024-04-02T04:59:00.136703+0000","last_unstale":"2024-04-02T05:00:22.543629+0000","last_undegraded":"2024-04-02T05:00:22.543629+0000","last_fullsized":"2024-04-02T05:00:22.543629+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'13","last_scrub_stamp":"2024-04-02T05:00:03.815118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:03.815118+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:24.694940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058747+0000","last_change":"2024-04-02T04:59:00.133941+0000","last_active":"2024-04-02T05:00:22.058747+0000","last_peered":"2024-04-02T05:00:22.058747+0000","last_clean":"2024-04-02T05:00:22.058747+0000","last_became_active":"2024-04-02T04:59:00.125089+0000","last_became_peered":"2024-04-02T04:59:00.125089+0000","last_unstale":"2024-04-02T05:00:22.058747+0000","last_undegraded":"2024-04-02T05:00:22.058747+0000","last_fullsized":"2024-04-02T05:00:22.058747+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:56.418322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:56.418322+0000","last_clean_scrub_stamp":"2024-04-02T04:58:56.418322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'128","reported_seq":777,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058595+0000","last_change":"2024-04-02T04:59:03.987730+0000","last_active":"2024-04-02T05:00:22.058595+0000","last_peered":"2024-04-02T05:00:22.058595+0000","last_clean":"2024-04-02T05:00:22.058595+0000","last_became_active":"2024-04-02T04:59:03.050648+0000","last_became_peered":"2024-04-02T04:59:03.050648+0000","last_unstale":"2024-04-02T05:00:22.058595+0000","last_undegraded":"2024-04-02T05:00:22.058595+0000","last_fullsized":"2024-04-02T05:00:22.058595+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:59:03.987621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:59:03.987621+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:31.584277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1199,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053292+0000","last_change":"2024-04-02T04:59:00.106527+0000","last_active":"2024-04-02T05:00:22.053292+0000","last_peered":"2024-04-02T05:00:22.053292+0000","last_clean":"2024-04-02T05:00:22.053292+0000","last_became_active":"2024-04-02T04:59:00.105786+0000","last_became_peered":"2024-04-02T04:59:00.105786+0000","last_unstale":"2024-04-02T05:00:22.053292+0000","last_undegraded":"2024-04-02T05:00:22.053292+0000","last_fullsized":"2024-04-02T05:00:22.053292+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T04:54:29.941615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":903,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049489+0000","last_change":"2024-04-02T04:59:20.079551+0000","last_active":"2024-04-02T05:00:22.049489+0000","last_peered":"2024-04-02T05:00:22.049489+0000","last_clean":"2024-04-02T05:00:22.049489+0000","last_became_active":"2024-04-02T04:59:20.074938+0000","last_became_peered":"2024-04-02T04:59:20.074938+0000","last_unstale":"2024-04-02T05:00:22.049489+0000","last_undegraded":"2024-04-02T05:00:22.049489+0000","last_fullsized":"2024-04-02T05:00:22.049489+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:25.081828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T04:56:25.081828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"402'30","reported_seq":1382,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.872134+0000","last_change":"2024-04-02T04:58:54.938309+0000","last_active":"2024-04-02T05:00:22.872134+0000","last_peered":"2024-04-02T05:00:22.872134+0000","last_clean":"2024-04-02T05:00:22.872134+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T05:00:22.872134+0000","last_undegraded":"2024-04-02T05:00:22.872134+0000","last_fullsized":"2024-04-02T05:00:22.872134+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T04:58:54.937951+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T04:58:54.937951+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1349,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053295+0000","last_change":"2024-04-02T05:00:06.958748+0000","last_active":"2024-04-02T05:00:22.053295+0000","last_peered":"2024-04-02T05:00:22.053295+0000","last_clean":"2024-04-02T05:00:22.053295+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T05:00:22.053295+0000","last_undegraded":"2024-04-02T05:00:22.053295+0000","last_fullsized":"2024-04-02T05:00:22.053295+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:06.958525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:00:06.958525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:33.054452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"92'5","reported_seq":264,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057101+0000","last_change":"2024-04-02T04:59:55.744309+0000","last_active":"2024-04-02T05:00:22.057101+0000","last_peered":"2024-04-02T05:00:22.057101+0000","last_clean":"2024-04-02T05:00:22.057101+0000","last_became_active":"2024-04-02T04:59:00.119844+0000","last_became_peered":"2024-04-02T04:59:00.119844+0000","last_unstale":"2024-04-02T05:00:22.057101+0000","last_undegraded":"2024-04-02T05:00:22.057101+0000","last_fullsized":"2024-04-02T05:00:22.057101+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'5","last_scrub_stamp":"2024-04-02T04:59:55.744211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:55.744211+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:57.653346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":784,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108731+0000","last_change":"2024-04-02T04:57:45.227053+0000","last_active":"2024-04-02T05:00:23.108731+0000","last_peered":"2024-04-02T05:00:23.108731+0000","last_clean":"2024-04-02T05:00:23.108731+0000","last_became_active":"2024-04-02T04:53:54.936665+0000","last_became_peered":"2024-04-02T04:53:54.936665+0000","last_unstale":"2024-04-02T05:00:23.108731+0000","last_undegraded":"2024-04-02T05:00:23.108731+0000","last_fullsized":"2024-04-02T05:00:23.108731+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:45.226951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T04:57:45.226951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'128","reported_seq":937,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896141+0000","last_change":"2024-04-02T04:59:58.019547+0000","last_active":"2024-04-02T05:00:20.896141+0000","last_peered":"2024-04-02T05:00:20.896141+0000","last_clean":"2024-04-02T05:00:20.896141+0000","last_became_active":"2024-04-02T04:59:23.074106+0000","last_became_peered":"2024-04-02T04:59:23.074106+0000","last_unstale":"2024-04-02T05:00:20.896141+0000","last_undegraded":"2024-04-02T05:00:20.896141+0000","last_fullsized":"2024-04-02T05:00:20.896141+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":665,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:59:58.019395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:59:58.019395+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:02.128448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1197,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.540796+0000","last_change":"2024-04-02T04:59:15.184662+0000","last_active":"2024-04-02T05:00:22.540796+0000","last_peered":"2024-04-02T05:00:22.540796+0000","last_clean":"2024-04-02T05:00:22.540796+0000","last_became_active":"2024-04-02T04:49:26.251135+0000","last_became_peered":"2024-04-02T04:49:26.251135+0000","last_unstale":"2024-04-02T05:00:22.540796+0000","last_undegraded":"2024-04-02T05:00:22.540796+0000","last_fullsized":"2024-04-02T05:00:22.540796+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:15.184457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:15.184457+0000","last_clean_scrub_stamp":"2024-04-02T04:59:15.184457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:43.523241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"642'16","reported_seq":198,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057173+0000","last_change":"2024-04-02T05:00:19.766987+0000","last_active":"2024-04-02T05:00:22.057173+0000","last_peered":"2024-04-02T05:00:22.057173+0000","last_clean":"2024-04-02T05:00:22.057173+0000","last_became_active":"2024-04-02T04:59:00.134138+0000","last_became_peered":"2024-04-02T04:59:00.134138+0000","last_unstale":"2024-04-02T05:00:22.057173+0000","last_undegraded":"2024-04-02T05:00:22.057173+0000","last_fullsized":"2024-04-02T05:00:22.057173+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'16","last_scrub_stamp":"2024-04-02T05:00:19.766749+0000","last_deep_scrub":"516'15","last_deep_scrub_stamp":"2024-04-02T04:56:30.036674+0000","last_clean_scrub_stamp":"2024-04-02T05:00:19.766749+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:20.822255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"402'23","reported_seq":191,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059833+0000","last_change":"2024-04-02T04:59:05.949973+0000","last_active":"2024-04-02T05:00:22.059833+0000","last_peered":"2024-04-02T05:00:22.059833+0000","last_clean":"2024-04-02T05:00:22.059833+0000","last_became_active":"2024-04-02T04:59:00.134091+0000","last_became_peered":"2024-04-02T04:59:00.134091+0000","last_unstale":"2024-04-02T05:00:22.059833+0000","last_undegraded":"2024-04-02T05:00:22.059833+0000","last_fullsized":"2024-04-02T05:00:22.059833+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'23","last_scrub_stamp":"2024-04-02T04:59:05.949860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:05.949860+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:22.771600+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":1354,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056224+0000","last_change":"2024-04-02T04:59:09.939926+0000","last_active":"2024-04-02T05:00:22.056224+0000","last_peered":"2024-04-02T05:00:22.056224+0000","last_clean":"2024-04-02T05:00:22.056224+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T05:00:22.056224+0000","last_undegraded":"2024-04-02T05:00:22.056224+0000","last_fullsized":"2024-04-02T05:00:22.056224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T04:59:09.939787+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.939787+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:37.387693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1361,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.541686+0000","last_change":"2024-04-02T04:59:10.152080+0000","last_active":"2024-04-02T05:00:22.541686+0000","last_peered":"2024-04-02T05:00:22.541686+0000","last_clean":"2024-04-02T05:00:22.541686+0000","last_became_active":"2024-04-02T04:59:00.135841+0000","last_became_peered":"2024-04-02T04:59:00.135841+0000","last_unstale":"2024-04-02T05:00:22.541686+0000","last_undegraded":"2024-04-02T05:00:22.541686+0000","last_fullsized":"2024-04-02T05:00:22.541686+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:10.151703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:10.151703+0000","last_clean_scrub_stamp":"2024-04-02T04:59:10.151703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:23.573648+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1312,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057576+0000","last_change":"2024-04-02T05:00:16.830983+0000","last_active":"2024-04-02T05:00:22.057576+0000","last_peered":"2024-04-02T05:00:22.057576+0000","last_clean":"2024-04-02T05:00:22.057576+0000","last_became_active":"2024-04-02T04:59:20.037960+0000","last_became_peered":"2024-04-02T04:59:20.037960+0000","last_unstale":"2024-04-02T05:00:22.057576+0000","last_undegraded":"2024-04-02T05:00:22.057576+0000","last_fullsized":"2024-04-02T05:00:22.057576+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:16.830867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.830867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:20.378328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"101'128","reported_seq":826,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051464+0000","last_change":"2024-04-02T04:59:03.045870+0000","last_active":"2024-04-02T05:00:22.051464+0000","last_peered":"2024-04-02T05:00:22.051464+0000","last_clean":"2024-04-02T05:00:22.051464+0000","last_became_active":"2024-04-02T04:59:03.045003+0000","last_became_peered":"2024-04-02T04:59:03.045003+0000","last_unstale":"2024-04-02T05:00:22.051464+0000","last_undegraded":"2024-04-02T05:00:22.051464+0000","last_fullsized":"2024-04-02T05:00:22.051464+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.026882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.026882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1187,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051647+0000","last_change":"2024-04-02T04:54:26.868758+0000","last_active":"2024-04-02T05:00:22.051647+0000","last_peered":"2024-04-02T05:00:22.051647+0000","last_clean":"2024-04-02T05:00:22.051647+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T05:00:22.051647+0000","last_undegraded":"2024-04-02T05:00:22.051647+0000","last_fullsized":"2024-04-02T05:00:22.051647+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:26.868528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:54:26.868528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"451'15","reported_seq":1566,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871638+0000","last_change":"2024-04-02T05:00:18.919124+0000","last_active":"2024-04-02T05:00:22.871638+0000","last_peered":"2024-04-02T05:00:22.871638+0000","last_clean":"2024-04-02T05:00:22.871638+0000","last_became_active":"2024-04-02T04:54:55.961735+0000","last_became_peered":"2024-04-02T04:54:55.961735+0000","last_unstale":"2024-04-02T05:00:22.871638+0000","last_undegraded":"2024-04-02T05:00:22.871638+0000","last_fullsized":"2024-04-02T05:00:22.871638+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'15","last_scrub_stamp":"2024-04-02T05:00:18.918806+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T05:00:18.918806+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:45.797032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"451'60","reported_seq":1667,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055382+0000","last_change":"2024-04-02T04:57:33.971003+0000","last_active":"2024-04-02T05:00:22.055382+0000","last_peered":"2024-04-02T05:00:22.055382+0000","last_clean":"2024-04-02T05:00:22.055382+0000","last_became_active":"2024-04-02T04:54:55.955778+0000","last_became_peered":"2024-04-02T04:54:55.955778+0000","last_unstale":"2024-04-02T05:00:22.055382+0000","last_undegraded":"2024-04-02T05:00:22.055382+0000","last_fullsized":"2024-04-02T05:00:22.055382+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'60","last_scrub_stamp":"2024-04-02T04:57:33.970725+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T04:57:33.970725+0000","objects_scrubbed":11,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":819,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109004+0000","last_change":"2024-04-02T04:58:55.433444+0000","last_active":"2024-04-02T05:00:23.109004+0000","last_peered":"2024-04-02T05:00:23.109004+0000","last_clean":"2024-04-02T05:00:23.109004+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T05:00:23.109004+0000","last_undegraded":"2024-04-02T05:00:23.109004+0000","last_fullsized":"2024-04-02T05:00:23.109004+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:55.433242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:55.433242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":1351,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050583+0000","last_change":"2024-04-02T04:56:27.048142+0000","last_active":"2024-04-02T05:00:22.050583+0000","last_peered":"2024-04-02T05:00:22.050583+0000","last_clean":"2024-04-02T05:00:22.050583+0000","last_became_active":"2024-04-02T04:54:56.004769+0000","last_became_peered":"2024-04-02T04:54:56.004769+0000","last_unstale":"2024-04-02T05:00:22.050583+0000","last_undegraded":"2024-04-02T05:00:22.050583+0000","last_fullsized":"2024-04-02T05:00:22.050583+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T04:56:27.047860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:27.047860+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1332,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055994+0000","last_change":"2024-04-02T04:58:45.937050+0000","last_active":"2024-04-02T05:00:22.055994+0000","last_peered":"2024-04-02T05:00:22.055994+0000","last_clean":"2024-04-02T05:00:22.055994+0000","last_became_active":"2024-04-02T04:44:00.344136+0000","last_became_peered":"2024-04-02T04:44:00.344136+0000","last_unstale":"2024-04-02T05:00:22.055994+0000","last_undegraded":"2024-04-02T05:00:22.055994+0000","last_fullsized":"2024-04-02T05:00:22.055994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:45.936953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T04:58:45.936953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'128","reported_seq":1316,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054716+0000","last_change":"2024-04-02T05:00:02.124591+0000","last_active":"2024-04-02T05:00:22.054716+0000","last_peered":"2024-04-02T05:00:22.054716+0000","last_clean":"2024-04-02T05:00:22.054716+0000","last_became_active":"2024-04-02T04:59:23.086202+0000","last_became_peered":"2024-04-02T04:59:23.086202+0000","last_unstale":"2024-04-02T05:00:22.054716+0000","last_undegraded":"2024-04-02T05:00:22.054716+0000","last_fullsized":"2024-04-02T05:00:22.054716+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":665,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:02.124439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:02.124439+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:04.321410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1202,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869120+0000","last_change":"2024-04-02T04:57:53.023737+0000","last_active":"2024-04-02T05:00:22.869120+0000","last_peered":"2024-04-02T05:00:22.869120+0000","last_clean":"2024-04-02T05:00:22.869120+0000","last_became_active":"2024-04-02T04:53:54.957413+0000","last_became_peered":"2024-04-02T04:53:54.957413+0000","last_unstale":"2024-04-02T05:00:22.869120+0000","last_undegraded":"2024-04-02T05:00:22.869120+0000","last_fullsized":"2024-04-02T05:00:22.869120+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:53.023662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:57:53.023662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1348,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.872438+0000","last_change":"2024-04-02T04:56:39.050317+0000","last_active":"2024-04-02T05:00:22.872438+0000","last_peered":"2024-04-02T05:00:22.872438+0000","last_clean":"2024-04-02T05:00:22.872438+0000","last_became_active":"2024-04-02T04:54:55.950537+0000","last_became_peered":"2024-04-02T04:54:55.950537+0000","last_unstale":"2024-04-02T05:00:22.872438+0000","last_undegraded":"2024-04-02T05:00:22.872438+0000","last_fullsized":"2024-04-02T05:00:22.872438+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:39.050034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T04:56:39.050034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":817,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107868+0000","last_change":"2024-04-02T04:59:04.490663+0000","last_active":"2024-04-02T05:00:23.107868+0000","last_peered":"2024-04-02T05:00:23.107868+0000","last_clean":"2024-04-02T05:00:23.107868+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T05:00:23.107868+0000","last_undegraded":"2024-04-02T05:00:23.107868+0000","last_fullsized":"2024-04-02T05:00:23.107868+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:04.490513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:59:04.490513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:26.491224+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"402'34","reported_seq":1466,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869555+0000","last_change":"2024-04-02T04:59:20.119285+0000","last_active":"2024-04-02T05:00:22.869555+0000","last_peered":"2024-04-02T05:00:22.869555+0000","last_clean":"2024-04-02T05:00:22.869555+0000","last_became_active":"2024-04-02T04:59:20.079497+0000","last_became_peered":"2024-04-02T04:59:20.079497+0000","last_unstale":"2024-04-02T05:00:22.869555+0000","last_undegraded":"2024-04-02T05:00:22.869555+0000","last_fullsized":"2024-04-02T05:00:22.869555+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'34","last_scrub_stamp":"2024-04-02T04:57:30.220239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:30.220239+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058815+0000","last_change":"2024-04-02T05:00:01.808440+0000","last_active":"2024-04-02T05:00:22.058815+0000","last_peered":"2024-04-02T05:00:22.058815+0000","last_clean":"2024-04-02T05:00:22.058815+0000","last_became_active":"2024-04-02T04:59:00.119618+0000","last_became_peered":"2024-04-02T04:59:00.119618+0000","last_unstale":"2024-04-02T05:00:22.058815+0000","last_undegraded":"2024-04-02T05:00:22.058815+0000","last_fullsized":"2024-04-02T05:00:22.058815+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:01.808297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:01.808297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:27.632568+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1195,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050476+0000","last_change":"2024-04-02T05:00:19.909534+0000","last_active":"2024-04-02T05:00:22.050476+0000","last_peered":"2024-04-02T05:00:22.050476+0000","last_clean":"2024-04-02T05:00:22.050476+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T05:00:22.050476+0000","last_undegraded":"2024-04-02T05:00:22.050476+0000","last_fullsized":"2024-04-02T05:00:22.050476+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:19.909325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:19.909325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:46.485013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1347,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.052765+0000","last_change":"2024-04-02T04:59:58.883758+0000","last_active":"2024-04-02T05:00:22.052765+0000","last_peered":"2024-04-02T05:00:22.052765+0000","last_clean":"2024-04-02T05:00:22.052765+0000","last_became_active":"2024-04-02T04:59:20.070708+0000","last_became_peered":"2024-04-02T04:59:20.070708+0000","last_unstale":"2024-04-02T05:00:22.052765+0000","last_undegraded":"2024-04-02T05:00:22.052765+0000","last_fullsized":"2024-04-02T05:00:22.052765+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:58.883657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:58.883657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:28.844709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":1096,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107526+0000","last_change":"2024-04-02T05:00:02.393377+0000","last_active":"2024-04-02T05:00:23.107526+0000","last_peered":"2024-04-02T05:00:23.107526+0000","last_clean":"2024-04-02T05:00:23.107526+0000","last_became_active":"2024-04-02T04:55:27.978786+0000","last_became_peered":"2024-04-02T04:55:27.978786+0000","last_unstale":"2024-04-02T05:00:23.107526+0000","last_undegraded":"2024-04-02T05:00:23.107526+0000","last_fullsized":"2024-04-02T05:00:23.107526+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:02.393135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:02.393135+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:21.656307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1248,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896411+0000","last_change":"2024-04-02T04:59:00.082727+0000","last_active":"2024-04-02T05:00:20.896411+0000","last_peered":"2024-04-02T05:00:20.896411+0000","last_clean":"2024-04-02T05:00:20.896411+0000","last_became_active":"2024-04-02T04:59:00.082361+0000","last_became_peered":"2024-04-02T04:59:00.082361+0000","last_unstale":"2024-04-02T05:00:20.896411+0000","last_undegraded":"2024-04-02T05:00:20.896411+0000","last_fullsized":"2024-04-02T05:00:20.896411+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:50.380070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T04:58:50.380070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"402'26","reported_seq":191,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057804+0000","last_change":"2024-04-02T05:00:14.816581+0000","last_active":"2024-04-02T05:00:22.057804+0000","last_peered":"2024-04-02T05:00:22.057804+0000","last_clean":"2024-04-02T05:00:22.057804+0000","last_became_active":"2024-04-02T04:59:00.118923+0000","last_became_peered":"2024-04-02T04:59:00.118923+0000","last_unstale":"2024-04-02T05:00:22.057804+0000","last_undegraded":"2024-04-02T05:00:22.057804+0000","last_fullsized":"2024-04-02T05:00:22.057804+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T05:00:14.816455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:14.816455+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:24.957616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.060020+0000","last_change":"2024-04-02T05:00:09.835276+0000","last_active":"2024-04-02T05:00:22.060020+0000","last_peered":"2024-04-02T05:00:22.060020+0000","last_clean":"2024-04-02T05:00:22.060020+0000","last_became_active":"2024-04-02T04:59:00.126748+0000","last_became_peered":"2024-04-02T04:59:00.126748+0000","last_unstale":"2024-04-02T05:00:22.060020+0000","last_undegraded":"2024-04-02T05:00:22.060020+0000","last_fullsized":"2024-04-02T05:00:22.060020+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:09.835148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T05:00:09.835148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:22.647727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"101'128","reported_seq":1112,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107440+0000","last_change":"2024-04-02T04:59:58.410923+0000","last_active":"2024-04-02T05:00:23.107440+0000","last_peered":"2024-04-02T05:00:23.107440+0000","last_clean":"2024-04-02T05:00:23.107440+0000","last_became_active":"2024-04-02T04:59:23.081091+0000","last_became_peered":"2024-04-02T04:59:23.081091+0000","last_unstale":"2024-04-02T05:00:23.107440+0000","last_undegraded":"2024-04-02T05:00:23.107440+0000","last_fullsized":"2024-04-02T05:00:23.107440+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":665,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:59:58.410678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:59:58.410678+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:05.881254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1094,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.895418+0000","last_change":"2024-04-02T05:00:01.015948+0000","last_active":"2024-04-02T05:00:20.895418+0000","last_peered":"2024-04-02T05:00:20.895418+0000","last_clean":"2024-04-02T05:00:20.895418+0000","last_became_active":"2024-04-02T04:54:55.946927+0000","last_became_peered":"2024-04-02T04:54:55.946927+0000","last_unstale":"2024-04-02T05:00:20.895418+0000","last_undegraded":"2024-04-02T05:00:20.895418+0000","last_fullsized":"2024-04-02T05:00:20.895418+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:01.015774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:01.015774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:13.022150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1086,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871358+0000","last_change":"2024-04-02T04:59:00.138446+0000","last_active":"2024-04-02T05:00:22.871358+0000","last_peered":"2024-04-02T05:00:22.871358+0000","last_clean":"2024-04-02T05:00:22.871358+0000","last_became_active":"2024-04-02T04:59:00.138221+0000","last_became_peered":"2024-04-02T04:59:00.138221+0000","last_unstale":"2024-04-02T05:00:22.871358+0000","last_undegraded":"2024-04-02T05:00:22.871358+0000","last_fullsized":"2024-04-02T05:00:22.871358+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:32.966986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:57:32.966986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1365,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.895393+0000","last_change":"2024-04-02T04:59:00.082692+0000","last_active":"2024-04-02T05:00:20.895393+0000","last_peered":"2024-04-02T05:00:20.895393+0000","last_clean":"2024-04-02T05:00:20.895393+0000","last_became_active":"2024-04-02T04:59:00.081588+0000","last_became_peered":"2024-04-02T04:59:00.081588+0000","last_unstale":"2024-04-02T05:00:20.895393+0000","last_undegraded":"2024-04-02T05:00:20.895393+0000","last_fullsized":"2024-04-02T05:00:20.895393+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:38.267128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T04:58:38.267128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"402'42","reported_seq":1423,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053863+0000","last_change":"2024-04-02T05:00:16.172056+0000","last_active":"2024-04-02T05:00:22.053863+0000","last_peered":"2024-04-02T05:00:22.053863+0000","last_clean":"2024-04-02T05:00:22.053863+0000","last_became_active":"2024-04-02T04:43:56.240137+0000","last_became_peered":"2024-04-02T04:43:56.240137+0000","last_unstale":"2024-04-02T05:00:22.053863+0000","last_undegraded":"2024-04-02T05:00:22.053863+0000","last_fullsized":"2024-04-02T05:00:22.053863+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'42","last_scrub_stamp":"2024-04-02T05:00:16.171889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.171889+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:24.439881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":810,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109130+0000","last_change":"2024-04-02T05:00:16.448607+0000","last_active":"2024-04-02T05:00:23.109130+0000","last_peered":"2024-04-02T05:00:23.109130+0000","last_clean":"2024-04-02T05:00:23.109130+0000","last_became_active":"2024-04-02T04:59:00.137237+0000","last_became_peered":"2024-04-02T04:59:00.137237+0000","last_unstale":"2024-04-02T05:00:23.109130+0000","last_undegraded":"2024-04-02T05:00:23.109130+0000","last_fullsized":"2024-04-02T05:00:23.109130+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:16.448386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.448386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.359476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'128","reported_seq":1305,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868334+0000","last_change":"2024-04-02T05:00:17.932082+0000","last_active":"2024-04-02T05:00:22.868334+0000","last_peered":"2024-04-02T05:00:22.868334+0000","last_clean":"2024-04-02T05:00:22.868334+0000","last_became_active":"2024-04-02T04:55:32.984097+0000","last_became_peered":"2024-04-02T04:55:32.984097+0000","last_unstale":"2024-04-02T05:00:22.868334+0000","last_undegraded":"2024-04-02T05:00:22.868334+0000","last_fullsized":"2024-04-02T05:00:22.868334+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:17.931893+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:28.185555+0000","last_clean_scrub_stamp":"2024-04-02T05:00:17.931893+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.838449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1194,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049952+0000","last_change":"2024-04-02T04:54:56.005135+0000","last_active":"2024-04-02T05:00:22.049952+0000","last_peered":"2024-04-02T05:00:22.049952+0000","last_clean":"2024-04-02T05:00:22.049952+0000","last_became_active":"2024-04-02T04:54:56.004256+0000","last_became_peered":"2024-04-02T04:54:56.004256+0000","last_unstale":"2024-04-02T05:00:22.049952+0000","last_undegraded":"2024-04-02T05:00:22.049952+0000","last_fullsized":"2024-04-02T05:00:22.049952+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_clean_scrub_stamp":"2024-04-02T04:54:28.854283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1362,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055066+0000","last_change":"2024-04-02T05:00:11.176158+0000","last_active":"2024-04-02T05:00:22.055066+0000","last_peered":"2024-04-02T05:00:22.055066+0000","last_clean":"2024-04-02T05:00:22.055066+0000","last_became_active":"2024-04-02T04:59:00.109471+0000","last_became_peered":"2024-04-02T04:59:00.109471+0000","last_unstale":"2024-04-02T05:00:22.055066+0000","last_undegraded":"2024-04-02T05:00:22.055066+0000","last_fullsized":"2024-04-02T05:00:22.055066+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:11.176073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T05:00:11.176073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.820892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'35","reported_seq":1408,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542673+0000","last_change":"2024-04-02T04:59:58.884149+0000","last_active":"2024-04-02T05:00:22.542673+0000","last_peered":"2024-04-02T05:00:22.542673+0000","last_clean":"2024-04-02T05:00:22.542673+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T05:00:22.542673+0000","last_undegraded":"2024-04-02T05:00:22.542673+0000","last_fullsized":"2024-04-02T05:00:22.542673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T04:59:58.883932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:58.883932+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:09.596658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":563,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055189+0000","last_change":"2024-04-02T04:57:56.006438+0000","last_active":"2024-04-02T05:00:22.055189+0000","last_peered":"2024-04-02T05:00:22.055189+0000","last_clean":"2024-04-02T05:00:22.055189+0000","last_became_active":"2024-04-02T04:54:55.969352+0000","last_became_peered":"2024-04-02T04:54:55.969352+0000","last_unstale":"2024-04-02T05:00:22.055189+0000","last_undegraded":"2024-04-02T05:00:22.055189+0000","last_fullsized":"2024-04-02T05:00:22.055189+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:56.006224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:56.006224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":181,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059607+0000","last_change":"2024-04-02T04:59:04.968028+0000","last_active":"2024-04-02T05:00:22.059607+0000","last_peered":"2024-04-02T05:00:22.059607+0000","last_clean":"2024-04-02T05:00:22.059607+0000","last_became_active":"2024-04-02T04:59:00.128819+0000","last_became_peered":"2024-04-02T04:59:00.128819+0000","last_unstale":"2024-04-02T05:00:22.059607+0000","last_undegraded":"2024-04-02T05:00:22.059607+0000","last_fullsized":"2024-04-02T05:00:22.059607+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_clean_scrub_stamp":"2024-04-02T04:59:04.967881+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'128","reported_seq":844,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051881+0000","last_change":"2024-04-02T04:59:03.050796+0000","last_active":"2024-04-02T05:00:22.051881+0000","last_peered":"2024-04-02T05:00:22.051881+0000","last_clean":"2024-04-02T05:00:22.051881+0000","last_became_active":"2024-04-02T04:59:03.048483+0000","last_became_peered":"2024-04-02T04:59:03.048483+0000","last_unstale":"2024-04-02T05:00:22.051881+0000","last_undegraded":"2024-04-02T05:00:22.051881+0000","last_fullsized":"2024-04-02T05:00:22.051881+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T04:55:33.982932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T04:55:33.982932+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":687,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108551+0000","last_change":"2024-04-02T04:59:55.345761+0000","last_active":"2024-04-02T05:00:23.108551+0000","last_peered":"2024-04-02T05:00:23.108551+0000","last_clean":"2024-04-02T05:00:23.108551+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T05:00:23.108551+0000","last_undegraded":"2024-04-02T05:00:23.108551+0000","last_fullsized":"2024-04-02T05:00:23.108551+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:55.345460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:55.345460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.186175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":913,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868662+0000","last_change":"2024-04-02T04:59:20.058465+0000","last_active":"2024-04-02T05:00:22.868662+0000","last_peered":"2024-04-02T05:00:22.868662+0000","last_clean":"2024-04-02T05:00:22.868662+0000","last_became_active":"2024-04-02T04:59:20.057847+0000","last_became_peered":"2024-04-02T04:59:20.057847+0000","last_unstale":"2024-04-02T05:00:22.868662+0000","last_undegraded":"2024-04-02T05:00:22.868662+0000","last_fullsized":"2024-04-02T05:00:22.868662+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:43.553059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:43.553059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"642'31","reported_seq":204,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058992+0000","last_change":"2024-04-02T04:59:00.171687+0000","last_active":"2024-04-02T05:00:22.058992+0000","last_peered":"2024-04-02T05:00:22.058992+0000","last_clean":"2024-04-02T05:00:22.058992+0000","last_became_active":"2024-04-02T04:59:00.112450+0000","last_became_peered":"2024-04-02T04:59:00.112450+0000","last_unstale":"2024-04-02T05:00:22.058992+0000","last_undegraded":"2024-04-02T05:00:22.058992+0000","last_fullsized":"2024-04-02T05:00:22.058992+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-04-02T04:58:46.273857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:46.273857+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":1361,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055182+0000","last_change":"2024-04-02T04:58:47.888437+0000","last_active":"2024-04-02T05:00:22.055182+0000","last_peered":"2024-04-02T05:00:22.055182+0000","last_clean":"2024-04-02T05:00:22.055182+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T05:00:22.055182+0000","last_undegraded":"2024-04-02T05:00:22.055182+0000","last_fullsized":"2024-04-02T05:00:22.055182+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:58:47.888333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:47.888333+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1359,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.868089+0000","last_change":"2024-04-02T04:59:55.057850+0000","last_active":"2024-04-02T05:00:22.868089+0000","last_peered":"2024-04-02T05:00:22.868089+0000","last_clean":"2024-04-02T05:00:22.868089+0000","last_became_active":"2024-04-02T04:59:55.057518+0000","last_became_peered":"2024-04-02T04:59:55.057518+0000","last_unstale":"2024-04-02T05:00:22.868089+0000","last_undegraded":"2024-04-02T05:00:22.868089+0000","last_fullsized":"2024-04-02T05:00:22.868089+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:51.989387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:58:51.989387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1220,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053012+0000","last_change":"2024-04-02T04:59:52.869208+0000","last_active":"2024-04-02T05:00:22.053012+0000","last_peered":"2024-04-02T05:00:22.053012+0000","last_clean":"2024-04-02T05:00:22.053012+0000","last_became_active":"2024-04-02T04:59:00.105159+0000","last_became_peered":"2024-04-02T04:59:00.105159+0000","last_unstale":"2024-04-02T05:00:22.053012+0000","last_undegraded":"2024-04-02T05:00:22.053012+0000","last_fullsized":"2024-04-02T05:00:22.053012+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:52.868879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:52.868879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:59.251842+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"642'14","reported_seq":1570,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.541892+0000","last_change":"2024-04-02T04:59:00.204978+0000","last_active":"2024-04-02T05:00:22.541892+0000","last_peered":"2024-04-02T05:00:22.541892+0000","last_clean":"2024-04-02T05:00:22.541892+0000","last_became_active":"2024-04-02T04:59:00.137930+0000","last_became_peered":"2024-04-02T04:59:00.137930+0000","last_unstale":"2024-04-02T05:00:22.541892+0000","last_undegraded":"2024-04-02T05:00:22.541892+0000","last_fullsized":"2024-04-02T05:00:22.541892+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'13","last_scrub_stamp":"2024-04-02T04:58:38.093860+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T04:58:38.093860+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"402'25","reported_seq":887,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108627+0000","last_change":"2024-04-02T04:59:20.099182+0000","last_active":"2024-04-02T05:00:23.108627+0000","last_peered":"2024-04-02T05:00:23.108627+0000","last_clean":"2024-04-02T05:00:23.108627+0000","last_became_active":"2024-04-02T04:59:20.080550+0000","last_became_peered":"2024-04-02T04:59:20.080550+0000","last_unstale":"2024-04-02T05:00:23.108627+0000","last_undegraded":"2024-04-02T05:00:23.108627+0000","last_fullsized":"2024-04-02T05:00:23.108627+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:59:12.433584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:12.433584+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:33.741206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":1374,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.543275+0000","last_change":"2024-04-02T04:58:50.033861+0000","last_active":"2024-04-02T05:00:22.543275+0000","last_peered":"2024-04-02T05:00:22.543275+0000","last_clean":"2024-04-02T05:00:22.543275+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T05:00:22.543275+0000","last_undegraded":"2024-04-02T05:00:22.543275+0000","last_fullsized":"2024-04-02T05:00:22.543275+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T04:58:50.033443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:50.033443+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058421+0000","last_change":"2024-04-02T04:59:00.124915+0000","last_active":"2024-04-02T05:00:22.058421+0000","last_peered":"2024-04-02T05:00:22.058421+0000","last_clean":"2024-04-02T05:00:22.058421+0000","last_became_active":"2024-04-02T04:59:00.122883+0000","last_became_peered":"2024-04-02T04:59:00.122883+0000","last_unstale":"2024-04-02T05:00:22.058421+0000","last_undegraded":"2024-04-02T05:00:22.058421+0000","last_fullsized":"2024-04-02T05:00:22.058421+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:49.195904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:49.195904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1087,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053978+0000","last_change":"2024-04-02T05:00:15.158620+0000","last_active":"2024-04-02T05:00:22.053978+0000","last_peered":"2024-04-02T05:00:22.053978+0000","last_clean":"2024-04-02T05:00:22.053978+0000","last_became_active":"2024-04-02T04:59:00.106070+0000","last_became_peered":"2024-04-02T04:59:00.106070+0000","last_unstale":"2024-04-02T05:00:22.053978+0000","last_undegraded":"2024-04-02T05:00:22.053978+0000","last_fullsized":"2024-04-02T05:00:22.053978+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:15.158507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:30.126652+0000","last_clean_scrub_stamp":"2024-04-02T05:00:15.158507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:33.194727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"402'25","reported_seq":857,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107648+0000","last_change":"2024-04-02T04:59:15.470244+0000","last_active":"2024-04-02T05:00:23.107648+0000","last_peered":"2024-04-02T05:00:23.107648+0000","last_clean":"2024-04-02T05:00:23.107648+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T05:00:23.107648+0000","last_undegraded":"2024-04-02T05:00:23.107648+0000","last_fullsized":"2024-04-02T05:00:23.107648+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:59:15.470090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:15.470090+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1361,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:21.143471+0000","last_change":"2024-04-02T05:00:21.143519+0000","last_active":"2024-04-02T05:00:21.143471+0000","last_peered":"2024-04-02T05:00:21.143471+0000","last_clean":"2024-04-02T05:00:21.143471+0000","last_became_active":"2024-04-02T04:43:54.222359+0000","last_became_peered":"2024-04-02T04:43:54.222359+0000","last_unstale":"2024-04-02T05:00:21.143471+0000","last_undegraded":"2024-04-02T05:00:21.143471+0000","last_fullsized":"2024-04-02T05:00:21.143471+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:21.143342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T05:00:21.143342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:27.602352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1348,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.872583+0000","last_change":"2024-04-02T05:00:04.742935+0000","last_active":"2024-04-02T05:00:22.872583+0000","last_peered":"2024-04-02T05:00:22.872583+0000","last_clean":"2024-04-02T05:00:22.872583+0000","last_became_active":"2024-04-02T04:59:00.138663+0000","last_became_peered":"2024-04-02T04:59:00.138663+0000","last_unstale":"2024-04-02T05:00:22.872583+0000","last_undegraded":"2024-04-02T05:00:22.872583+0000","last_fullsized":"2024-04-02T05:00:22.872583+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:04.742833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_clean_scrub_stamp":"2024-04-02T05:00:04.742833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:07.674766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1334,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-04-02T05:00:20.896976+0000","last_change":"2024-04-02T04:59:13.463965+0000","last_active":"2024-04-02T05:00:20.896976+0000","last_peered":"2024-04-02T05:00:20.896976+0000","last_clean":"2024-04-02T05:00:20.896976+0000","last_became_active":"2024-04-02T04:53:54.925132+0000","last_became_peered":"2024-04-02T04:53:54.925132+0000","last_unstale":"2024-04-02T05:00:20.896976+0000","last_undegraded":"2024-04-02T05:00:20.896976+0000","last_fullsized":"2024-04-02T05:00:20.896976+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:13.463559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:59:13.463559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:26.762938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1212,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054292+0000","last_change":"2024-04-02T04:59:54.093885+0000","last_active":"2024-04-02T05:00:22.054292+0000","last_peered":"2024-04-02T05:00:22.054292+0000","last_clean":"2024-04-02T05:00:22.054292+0000","last_became_active":"2024-04-02T04:59:00.104872+0000","last_became_peered":"2024-04-02T04:59:00.104872+0000","last_unstale":"2024-04-02T05:00:22.054292+0000","last_undegraded":"2024-04-02T05:00:22.054292+0000","last_fullsized":"2024-04-02T05:00:22.054292+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:54.093763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:54.093763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:19.801875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"402'8","reported_seq":1376,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049201+0000","last_change":"2024-04-02T04:59:00.164109+0000","last_active":"2024-04-02T05:00:22.049201+0000","last_peered":"2024-04-02T05:00:22.049201+0000","last_clean":"2024-04-02T05:00:22.049201+0000","last_became_active":"2024-04-02T04:59:00.105124+0000","last_became_peered":"2024-04-02T04:59:00.105124+0000","last_unstale":"2024-04-02T05:00:22.049201+0000","last_undegraded":"2024-04-02T05:00:22.049201+0000","last_fullsized":"2024-04-02T05:00:22.049201+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'8","last_scrub_stamp":"2024-04-02T04:54:17.927334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:54:17.927334+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1346,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049307+0000","last_change":"2024-04-02T04:59:20.079468+0000","last_active":"2024-04-02T05:00:22.049307+0000","last_peered":"2024-04-02T05:00:22.049307+0000","last_clean":"2024-04-02T05:00:22.049307+0000","last_became_active":"2024-04-02T04:59:20.074948+0000","last_became_peered":"2024-04-02T04:59:20.074948+0000","last_unstale":"2024-04-02T05:00:22.049307+0000","last_undegraded":"2024-04-02T05:00:22.049307+0000","last_fullsized":"2024-04-02T05:00:22.049307+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:16.144273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T04:56:16.144273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059904+0000","last_change":"2024-04-02T04:59:00.128791+0000","last_active":"2024-04-02T05:00:22.059904+0000","last_peered":"2024-04-02T05:00:22.059904+0000","last_clean":"2024-04-02T05:00:22.059904+0000","last_became_active":"2024-04-02T04:59:00.122643+0000","last_became_peered":"2024-04-02T04:59:00.122643+0000","last_unstale":"2024-04-02T05:00:22.059904+0000","last_undegraded":"2024-04-02T05:00:22.059904+0000","last_fullsized":"2024-04-02T05:00:22.059904+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:43.172463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T04:57:43.172463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":812,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108874+0000","last_change":"2024-04-02T04:59:20.069882+0000","last_active":"2024-04-02T05:00:23.108874+0000","last_peered":"2024-04-02T05:00:23.108874+0000","last_clean":"2024-04-02T05:00:23.108874+0000","last_became_active":"2024-04-02T04:59:20.069216+0000","last_became_peered":"2024-04-02T04:59:20.069216+0000","last_unstale":"2024-04-02T05:00:23.108874+0000","last_undegraded":"2024-04-02T05:00:23.108874+0000","last_fullsized":"2024-04-02T05:00:23.108874+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:42.120564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:57:42.120564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":939,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869793+0000","last_change":"2024-04-02T05:00:15.927503+0000","last_active":"2024-04-02T05:00:22.869793+0000","last_peered":"2024-04-02T05:00:22.869793+0000","last_clean":"2024-04-02T05:00:22.869793+0000","last_became_active":"2024-04-02T04:57:38.079839+0000","last_became_peered":"2024-04-02T04:57:38.079839+0000","last_unstale":"2024-04-02T05:00:22.869793+0000","last_undegraded":"2024-04-02T05:00:22.869793+0000","last_fullsized":"2024-04-02T05:00:22.869793+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:15.927397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T05:00:15.927397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:39.782534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"402'19","reported_seq":830,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108255+0000","last_change":"2024-04-02T05:00:08.520683+0000","last_active":"2024-04-02T05:00:23.108255+0000","last_peered":"2024-04-02T05:00:23.108255+0000","last_clean":"2024-04-02T05:00:23.108255+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T05:00:23.108255+0000","last_undegraded":"2024-04-02T05:00:23.108255+0000","last_fullsized":"2024-04-02T05:00:23.108255+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'19","last_scrub_stamp":"2024-04-02T05:00:08.520552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:08.520552+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:31.827237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057805+0000","last_change":"2024-04-02T04:59:00.135183+0000","last_active":"2024-04-02T05:00:22.057805+0000","last_peered":"2024-04-02T05:00:22.057805+0000","last_clean":"2024-04-02T05:00:22.057805+0000","last_became_active":"2024-04-02T04:59:00.134529+0000","last_became_peered":"2024-04-02T04:59:00.134529+0000","last_unstale":"2024-04-02T05:00:22.057805+0000","last_undegraded":"2024-04-02T05:00:22.057805+0000","last_fullsized":"2024-04-02T05:00:22.057805+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:37.148900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:37.148900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"642'17","reported_seq":190,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058681+0000","last_change":"2024-04-02T04:59:00.220320+0000","last_active":"2024-04-02T05:00:22.058681+0000","last_peered":"2024-04-02T05:00:22.058681+0000","last_clean":"2024-04-02T05:00:22.058681+0000","last_became_active":"2024-04-02T04:59:00.133162+0000","last_became_peered":"2024-04-02T04:59:00.133162+0000","last_unstale":"2024-04-02T05:00:22.058681+0000","last_undegraded":"2024-04-02T05:00:22.058681+0000","last_fullsized":"2024-04-02T05:00:22.058681+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'16","last_scrub_stamp":"2024-04-02T04:57:41.153649+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T04:57:41.153649+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1344,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.872545+0000","last_change":"2024-04-02T04:59:14.938140+0000","last_active":"2024-04-02T05:00:22.872545+0000","last_peered":"2024-04-02T05:00:22.872545+0000","last_clean":"2024-04-02T05:00:22.872545+0000","last_became_active":"2024-04-02T04:59:00.138302+0000","last_became_peered":"2024-04-02T04:59:00.138302+0000","last_unstale":"2024-04-02T05:00:22.872545+0000","last_undegraded":"2024-04-02T05:00:22.872545+0000","last_fullsized":"2024-04-02T05:00:22.872545+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:14.937933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:26.393399+0000","last_clean_scrub_stamp":"2024-04-02T04:59:14.937933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:37.572208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":183,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058186+0000","last_change":"2024-04-02T04:59:20.067666+0000","last_active":"2024-04-02T05:00:22.058186+0000","last_peered":"2024-04-02T05:00:22.058186+0000","last_clean":"2024-04-02T05:00:22.058186+0000","last_became_active":"2024-04-02T04:59:20.067119+0000","last_became_peered":"2024-04-02T04:59:20.067119+0000","last_unstale":"2024-04-02T05:00:22.058186+0000","last_undegraded":"2024-04-02T05:00:22.058186+0000","last_fullsized":"2024-04-02T05:00:22.058186+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:06.976087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:59:06.976087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"402'15","reported_seq":1406,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869157+0000","last_change":"2024-04-02T04:59:00.214953+0000","last_active":"2024-04-02T05:00:22.869157+0000","last_peered":"2024-04-02T05:00:22.869157+0000","last_clean":"2024-04-02T05:00:22.869157+0000","last_became_active":"2024-04-02T04:59:00.139853+0000","last_became_peered":"2024-04-02T04:59:00.139853+0000","last_unstale":"2024-04-02T05:00:22.869157+0000","last_undegraded":"2024-04-02T05:00:22.869157+0000","last_fullsized":"2024-04-02T05:00:22.869157+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'15","last_scrub_stamp":"2024-04-02T04:57:55.973853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:55.973853+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1356,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869964+0000","last_change":"2024-04-02T04:59:00.104438+0000","last_active":"2024-04-02T05:00:22.869964+0000","last_peered":"2024-04-02T05:00:22.869964+0000","last_clean":"2024-04-02T05:00:22.869964+0000","last_became_active":"2024-04-02T04:59:00.104064+0000","last_became_peered":"2024-04-02T04:59:00.104064+0000","last_unstale":"2024-04-02T05:00:22.869964+0000","last_undegraded":"2024-04-02T05:00:22.869964+0000","last_fullsized":"2024-04-02T05:00:22.869964+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:45.058981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:45.058981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"642'16","reported_seq":1446,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109260+0000","last_change":"2024-04-02T04:59:00.193480+0000","last_active":"2024-04-02T05:00:23.109260+0000","last_peered":"2024-04-02T05:00:23.109260+0000","last_clean":"2024-04-02T05:00:23.109260+0000","last_became_active":"2024-04-02T04:59:00.136907+0000","last_became_peered":"2024-04-02T04:59:00.136907+0000","last_unstale":"2024-04-02T05:00:23.109260+0000","last_undegraded":"2024-04-02T05:00:23.109260+0000","last_fullsized":"2024-04-02T05:00:23.109260+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'15","last_scrub_stamp":"2024-04-02T04:58:50.497333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:58:50.497333+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1342,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.543443+0000","last_change":"2024-04-02T05:00:16.830173+0000","last_active":"2024-04-02T05:00:22.543443+0000","last_peered":"2024-04-02T05:00:22.543443+0000","last_clean":"2024-04-02T05:00:22.543443+0000","last_became_active":"2024-04-02T04:54:56.000394+0000","last_became_peered":"2024-04-02T04:54:56.000394+0000","last_unstale":"2024-04-02T05:00:22.543443+0000","last_undegraded":"2024-04-02T05:00:22.543443+0000","last_fullsized":"2024-04-02T05:00:22.543443+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:16.829955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.829955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:28.164502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1187,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051519+0000","last_change":"2024-04-02T04:55:23.873415+0000","last_active":"2024-04-02T05:00:22.051519+0000","last_peered":"2024-04-02T05:00:22.051519+0000","last_clean":"2024-04-02T05:00:22.051519+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T05:00:22.051519+0000","last_undegraded":"2024-04-02T05:00:22.051519+0000","last_fullsized":"2024-04-02T05:00:22.051519+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:23.873192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:23.873192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"402'21","reported_seq":903,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.109251+0000","last_change":"2024-04-02T04:59:56.359478+0000","last_active":"2024-04-02T05:00:23.109251+0000","last_peered":"2024-04-02T05:00:23.109251+0000","last_clean":"2024-04-02T05:00:23.109251+0000","last_became_active":"2024-04-02T04:59:00.137738+0000","last_became_peered":"2024-04-02T04:59:00.137738+0000","last_unstale":"2024-04-02T05:00:23.109251+0000","last_undegraded":"2024-04-02T05:00:23.109251+0000","last_fullsized":"2024-04-02T05:00:23.109251+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T04:59:56.359328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.359328+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:14.983629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1350,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051998+0000","last_change":"2024-04-02T04:59:00.105775+0000","last_active":"2024-04-02T05:00:22.051998+0000","last_peered":"2024-04-02T05:00:22.051998+0000","last_clean":"2024-04-02T05:00:22.051998+0000","last_became_active":"2024-04-02T04:59:00.105424+0000","last_became_peered":"2024-04-02T04:59:00.105424+0000","last_unstale":"2024-04-02T05:00:22.051998+0000","last_undegraded":"2024-04-02T05:00:22.051998+0000","last_fullsized":"2024-04-02T05:00:22.051998+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:17.859624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:55:17.859624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":297,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.049734+0000","last_change":"2024-04-02T04:57:38.085399+0000","last_active":"2024-04-02T05:00:22.049734+0000","last_peered":"2024-04-02T05:00:22.049734+0000","last_clean":"2024-04-02T05:00:22.049734+0000","last_became_active":"2024-04-02T04:57:38.083311+0000","last_became_peered":"2024-04-02T04:57:38.083311+0000","last_unstale":"2024-04-02T05:00:22.049734+0000","last_undegraded":"2024-04-02T05:00:22.049734+0000","last_fullsized":"2024-04-02T05:00:22.049734+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:37.487754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:56:37.487754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1338,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.867758+0000","last_change":"2024-04-02T04:56:35.017181+0000","last_active":"2024-04-02T05:00:22.867758+0000","last_peered":"2024-04-02T05:00:22.867758+0000","last_clean":"2024-04-02T05:00:22.867758+0000","last_became_active":"2024-04-02T04:54:56.005128+0000","last_became_peered":"2024-04-02T04:54:56.005128+0000","last_unstale":"2024-04-02T05:00:22.867758+0000","last_undegraded":"2024-04-02T05:00:22.867758+0000","last_fullsized":"2024-04-02T05:00:22.867758+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:35.017057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:56:35.017057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1205,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.540747+0000","last_change":"2024-04-02T05:00:09.877481+0000","last_active":"2024-04-02T05:00:22.540747+0000","last_peered":"2024-04-02T05:00:22.540747+0000","last_clean":"2024-04-02T05:00:22.540747+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T05:00:22.540747+0000","last_undegraded":"2024-04-02T05:00:22.540747+0000","last_fullsized":"2024-04-02T05:00:22.540747+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:09.877288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:09.877288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:11.210502+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"402'32","reported_seq":892,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108376+0000","last_change":"2024-04-02T04:59:20.097757+0000","last_active":"2024-04-02T05:00:23.108376+0000","last_peered":"2024-04-02T05:00:23.108376+0000","last_clean":"2024-04-02T05:00:23.108376+0000","last_became_active":"2024-04-02T04:59:20.057918+0000","last_became_peered":"2024-04-02T04:59:20.057918+0000","last_unstale":"2024-04-02T05:00:23.108376+0000","last_undegraded":"2024-04-02T05:00:23.108376+0000","last_fullsized":"2024-04-02T05:00:23.108376+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'32","last_scrub_stamp":"2024-04-02T04:58:48.115768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:48.115768+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1349,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055593+0000","last_change":"2024-04-02T04:58:42.899985+0000","last_active":"2024-04-02T05:00:22.055593+0000","last_peered":"2024-04-02T05:00:22.055593+0000","last_clean":"2024-04-02T05:00:22.055593+0000","last_became_active":"2024-04-02T04:54:55.987373+0000","last_became_peered":"2024-04-02T04:54:55.987373+0000","last_unstale":"2024-04-02T05:00:22.055593+0000","last_undegraded":"2024-04-02T05:00:22.055593+0000","last_fullsized":"2024-04-02T05:00:22.055593+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:42.899853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.899853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1341,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.540761+0000","last_change":"2024-04-02T04:56:40.845726+0000","last_active":"2024-04-02T05:00:22.540761+0000","last_peered":"2024-04-02T05:00:22.540761+0000","last_clean":"2024-04-02T05:00:22.540761+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T05:00:22.540761+0000","last_undegraded":"2024-04-02T05:00:22.540761+0000","last_fullsized":"2024-04-02T05:00:22.540761+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_clean_scrub_stamp":"2024-04-02T04:56:40.845504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057214+0000","last_change":"2024-04-02T04:59:14.003983+0000","last_active":"2024-04-02T05:00:22.057214+0000","last_peered":"2024-04-02T05:00:22.057214+0000","last_clean":"2024-04-02T05:00:22.057214+0000","last_became_active":"2024-04-02T04:59:00.136143+0000","last_became_peered":"2024-04-02T04:59:00.136143+0000","last_unstale":"2024-04-02T05:00:22.057214+0000","last_undegraded":"2024-04-02T05:00:22.057214+0000","last_fullsized":"2024-04-02T05:00:22.057214+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:14.003869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T04:59:14.003869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:24.320187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1207,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871803+0000","last_change":"2024-04-02T04:58:52.967879+0000","last_active":"2024-04-02T05:00:22.871803+0000","last_peered":"2024-04-02T05:00:22.871803+0000","last_clean":"2024-04-02T05:00:22.871803+0000","last_became_active":"2024-04-02T04:54:56.017129+0000","last_became_peered":"2024-04-02T04:54:56.017129+0000","last_unstale":"2024-04-02T05:00:22.871803+0000","last_undegraded":"2024-04-02T05:00:22.871803+0000","last_fullsized":"2024-04-02T05:00:22.871803+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:52.967749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T04:58:52.967749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"402'18","reported_seq":1433,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871457+0000","last_change":"2024-04-02T05:00:21.863287+0000","last_active":"2024-04-02T05:00:22.871457+0000","last_peered":"2024-04-02T05:00:22.871457+0000","last_clean":"2024-04-02T05:00:22.871457+0000","last_became_active":"2024-04-02T04:59:00.139298+0000","last_became_peered":"2024-04-02T04:59:00.139298+0000","last_unstale":"2024-04-02T05:00:22.871457+0000","last_undegraded":"2024-04-02T05:00:22.871457+0000","last_fullsized":"2024-04-02T05:00:22.871457+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'18","last_scrub_stamp":"2024-04-02T05:00:21.863002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:21.863002+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:45.951023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":563,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051992+0000","last_change":"2024-04-02T04:59:56.948999+0000","last_active":"2024-04-02T05:00:22.051992+0000","last_peered":"2024-04-02T05:00:22.051992+0000","last_clean":"2024-04-02T05:00:22.051992+0000","last_became_active":"2024-04-02T04:54:55.972841+0000","last_became_peered":"2024-04-02T04:54:55.972841+0000","last_unstale":"2024-04-02T05:00:22.051992+0000","last_undegraded":"2024-04-02T05:00:22.051992+0000","last_fullsized":"2024-04-02T05:00:22.051992+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:56.948901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.424575+0000","last_clean_scrub_stamp":"2024-04-02T04:59:56.948901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:00.094728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056669+0000","last_change":"2024-04-02T05:00:12.801986+0000","last_active":"2024-04-02T05:00:22.056669+0000","last_peered":"2024-04-02T05:00:22.056669+0000","last_clean":"2024-04-02T05:00:22.056669+0000","last_became_active":"2024-04-02T04:59:00.102092+0000","last_became_peered":"2024-04-02T04:59:00.102092+0000","last_unstale":"2024-04-02T05:00:22.056669+0000","last_undegraded":"2024-04-02T05:00:22.056669+0000","last_fullsized":"2024-04-02T05:00:22.056669+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:12.801839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.801839+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.801839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:21.351595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":808,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107960+0000","last_change":"2024-04-02T04:59:00.399088+0000","last_active":"2024-04-02T05:00:23.107960+0000","last_peered":"2024-04-02T05:00:23.107960+0000","last_clean":"2024-04-02T05:00:23.107960+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T05:00:23.107960+0000","last_undegraded":"2024-04-02T05:00:23.107960+0000","last_fullsized":"2024-04-02T05:00:23.107960+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:00.398882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T04:59:00.398882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:29.239316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":688,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108953+0000","last_change":"2024-04-02T04:59:14.445982+0000","last_active":"2024-04-02T05:00:23.108953+0000","last_peered":"2024-04-02T05:00:23.108953+0000","last_clean":"2024-04-02T05:00:23.108953+0000","last_became_active":"2024-04-02T04:53:54.927422+0000","last_became_peered":"2024-04-02T04:53:54.927422+0000","last_unstale":"2024-04-02T05:00:23.108953+0000","last_undegraded":"2024-04-02T05:00:23.108953+0000","last_fullsized":"2024-04-02T05:00:23.108953+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:14.445746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:14.445746+0000","last_clean_scrub_stamp":"2024-04-02T04:59:14.445746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:24.979872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"402'27","reported_seq":1358,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108788+0000","last_change":"2024-04-02T04:59:00.201939+0000","last_active":"2024-04-02T05:00:23.108788+0000","last_peered":"2024-04-02T05:00:23.108788+0000","last_clean":"2024-04-02T05:00:23.108788+0000","last_became_active":"2024-04-02T04:59:00.136183+0000","last_became_peered":"2024-04-02T04:59:00.136183+0000","last_unstale":"2024-04-02T05:00:23.108788+0000","last_undegraded":"2024-04-02T05:00:23.108788+0000","last_fullsized":"2024-04-02T05:00:23.108788+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T04:58:42.485994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.485994+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1345,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053123+0000","last_change":"2024-04-02T04:54:55.979542+0000","last_active":"2024-04-02T05:00:22.053123+0000","last_peered":"2024-04-02T05:00:22.053123+0000","last_clean":"2024-04-02T05:00:22.053123+0000","last_became_active":"2024-04-02T04:54:55.978936+0000","last_became_peered":"2024-04-02T04:54:55.978936+0000","last_unstale":"2024-04-02T05:00:22.053123+0000","last_undegraded":"2024-04-02T05:00:22.053123+0000","last_fullsized":"2024-04-02T05:00:22.053123+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:34.979136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T04:54:34.979136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057355+0000","last_change":"2024-04-02T04:59:00.124514+0000","last_active":"2024-04-02T05:00:22.057355+0000","last_peered":"2024-04-02T05:00:22.057355+0000","last_clean":"2024-04-02T05:00:22.057355+0000","last_became_active":"2024-04-02T04:59:00.122631+0000","last_became_peered":"2024-04-02T04:59:00.122631+0000","last_unstale":"2024-04-02T05:00:22.057355+0000","last_undegraded":"2024-04-02T05:00:22.057355+0000","last_fullsized":"2024-04-02T05:00:22.057355+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:54.470294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:58:54.470294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058448+0000","last_change":"2024-04-02T05:00:18.800203+0000","last_active":"2024-04-02T05:00:22.058448+0000","last_peered":"2024-04-02T05:00:22.058448+0000","last_clean":"2024-04-02T05:00:22.058448+0000","last_became_active":"2024-04-02T04:59:00.136243+0000","last_became_peered":"2024-04-02T04:59:00.136243+0000","last_unstale":"2024-04-02T05:00:22.058448+0000","last_undegraded":"2024-04-02T05:00:22.058448+0000","last_fullsized":"2024-04-02T05:00:22.058448+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:18.800043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:18.800043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:39.510999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":653,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.107630+0000","last_change":"2024-04-02T04:59:20.055671+0000","last_active":"2024-04-02T05:00:23.107630+0000","last_peered":"2024-04-02T05:00:23.107630+0000","last_clean":"2024-04-02T05:00:23.107630+0000","last_became_active":"2024-04-02T04:59:20.054470+0000","last_became_peered":"2024-04-02T04:59:20.054470+0000","last_unstale":"2024-04-02T05:00:23.107630+0000","last_undegraded":"2024-04-02T05:00:23.107630+0000","last_fullsized":"2024-04-02T05:00:23.107630+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:56.993348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:56.993348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"402'62","reported_seq":1531,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054449+0000","last_change":"2024-04-02T04:59:20.119598+0000","last_active":"2024-04-02T05:00:22.054449+0000","last_peered":"2024-04-02T05:00:22.054449+0000","last_clean":"2024-04-02T05:00:22.054449+0000","last_became_active":"2024-04-02T04:59:20.053373+0000","last_became_peered":"2024-04-02T04:59:20.053373+0000","last_unstale":"2024-04-02T05:00:22.054449+0000","last_undegraded":"2024-04-02T05:00:22.054449+0000","last_fullsized":"2024-04-02T05:00:22.054449+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'62","last_scrub_stamp":"2024-04-02T04:59:07.986820+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T04:59:07.986820+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:28.959117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1245,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.541449+0000","last_change":"2024-04-02T04:58:42.062580+0000","last_active":"2024-04-02T05:00:22.541449+0000","last_peered":"2024-04-02T05:00:22.541449+0000","last_clean":"2024-04-02T05:00:22.541449+0000","last_became_active":"2024-04-02T04:54:56.008703+0000","last_became_peered":"2024-04-02T04:54:56.008703+0000","last_unstale":"2024-04-02T05:00:22.541449+0000","last_undegraded":"2024-04-02T05:00:22.541449+0000","last_fullsized":"2024-04-02T05:00:22.541449+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:42.062477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:58:42.062477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1358,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.872119+0000","last_change":"2024-04-02T05:00:14.896567+0000","last_active":"2024-04-02T05:00:22.872119+0000","last_peered":"2024-04-02T05:00:22.872119+0000","last_clean":"2024-04-02T05:00:22.872119+0000","last_became_active":"2024-04-02T04:59:00.138125+0000","last_became_peered":"2024-04-02T04:59:00.138125+0000","last_unstale":"2024-04-02T05:00:22.872119+0000","last_undegraded":"2024-04-02T05:00:22.872119+0000","last_fullsized":"2024-04-02T05:00:22.872119+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:14.896340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:14.896340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:39.910614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":839,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108755+0000","last_change":"2024-04-02T05:00:12.542064+0000","last_active":"2024-04-02T05:00:23.108755+0000","last_peered":"2024-04-02T05:00:23.108755+0000","last_clean":"2024-04-02T05:00:23.108755+0000","last_became_active":"2024-04-02T04:59:00.138970+0000","last_became_peered":"2024-04-02T04:59:00.138970+0000","last_unstale":"2024-04-02T05:00:23.108755+0000","last_undegraded":"2024-04-02T05:00:23.108755+0000","last_fullsized":"2024-04-02T05:00:23.108755+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T05:00:12.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:12.541934+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:23.160590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1090,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.869200+0000","last_change":"2024-04-02T04:59:15.923872+0000","last_active":"2024-04-02T05:00:22.869200+0000","last_peered":"2024-04-02T05:00:22.869200+0000","last_clean":"2024-04-02T05:00:22.869200+0000","last_became_active":"2024-04-02T04:54:55.989605+0000","last_became_peered":"2024-04-02T04:54:55.989605+0000","last_unstale":"2024-04-02T05:00:22.869200+0000","last_undegraded":"2024-04-02T05:00:22.869200+0000","last_fullsized":"2024-04-02T05:00:22.869200+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:15.923777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:15.923777+0000","last_clean_scrub_stamp":"2024-04-02T04:59:15.923777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:28.058172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"402'21","reported_seq":181,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056951+0000","last_change":"2024-04-02T04:59:00.221341+0000","last_active":"2024-04-02T05:00:22.056951+0000","last_peered":"2024-04-02T05:00:22.056951+0000","last_clean":"2024-04-02T05:00:22.056951+0000","last_became_active":"2024-04-02T04:59:00.133276+0000","last_became_peered":"2024-04-02T04:59:00.133276+0000","last_unstale":"2024-04-02T05:00:22.056951+0000","last_undegraded":"2024-04-02T05:00:22.056951+0000","last_fullsized":"2024-04-02T05:00:22.056951+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T04:57:37.101609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:37.101609+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":181,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057405+0000","last_change":"2024-04-02T04:59:09.983405+0000","last_active":"2024-04-02T05:00:22.057405+0000","last_peered":"2024-04-02T05:00:22.057405+0000","last_clean":"2024-04-02T05:00:22.057405+0000","last_became_active":"2024-04-02T04:59:00.135857+0000","last_became_peered":"2024-04-02T04:59:00.135857+0000","last_unstale":"2024-04-02T05:00:22.057405+0000","last_undegraded":"2024-04-02T05:00:22.057405+0000","last_fullsized":"2024-04-02T05:00:22.057405+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T04:59:09.983303+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T04:59:09.983303+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:32.925090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"451'12","reported_seq":1553,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.053439+0000","last_change":"2024-04-02T05:00:11.932285+0000","last_active":"2024-04-02T05:00:22.053439+0000","last_peered":"2024-04-02T05:00:22.053439+0000","last_clean":"2024-04-02T05:00:22.053439+0000","last_became_active":"2024-04-02T04:54:56.006792+0000","last_became_peered":"2024-04-02T04:54:56.006792+0000","last_unstale":"2024-04-02T05:00:22.053439+0000","last_undegraded":"2024-04-02T05:00:22.053439+0000","last_fullsized":"2024-04-02T05:00:22.053439+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'12","last_scrub_stamp":"2024-04-02T05:00:11.932062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:11.932062+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:23.340568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1343,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.056917+0000","last_change":"2024-04-02T04:59:20.079038+0000","last_active":"2024-04-02T05:00:22.056917+0000","last_peered":"2024-04-02T05:00:22.056917+0000","last_clean":"2024-04-02T05:00:22.056917+0000","last_became_active":"2024-04-02T04:59:20.078674+0000","last_became_peered":"2024-04-02T04:59:20.078674+0000","last_unstale":"2024-04-02T05:00:22.056917+0000","last_undegraded":"2024-04-02T05:00:22.056917+0000","last_fullsized":"2024-04-02T05:00:22.056917+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:08.948569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:59:08.948569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:35.814608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1193,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050298+0000","last_change":"2024-04-02T04:55:24.890462+0000","last_active":"2024-04-02T05:00:22.050298+0000","last_peered":"2024-04-02T05:00:22.050298+0000","last_clean":"2024-04-02T05:00:22.050298+0000","last_became_active":"2024-04-02T04:54:55.972361+0000","last_became_peered":"2024-04-02T04:54:55.972361+0000","last_unstale":"2024-04-02T05:00:22.050298+0000","last_undegraded":"2024-04-02T05:00:22.050298+0000","last_fullsized":"2024-04-02T05:00:22.050298+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:24.890319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.890319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"402'16","reported_seq":1368,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871310+0000","last_change":"2024-04-02T04:59:08.922889+0000","last_active":"2024-04-02T05:00:22.871310+0000","last_peered":"2024-04-02T05:00:22.871310+0000","last_clean":"2024-04-02T05:00:22.871310+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T05:00:22.871310+0000","last_undegraded":"2024-04-02T05:00:22.871310+0000","last_fullsized":"2024-04-02T05:00:22.871310+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'16","last_scrub_stamp":"2024-04-02T04:59:08.922768+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T04:59:08.922768+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:23.066709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1340,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.050437+0000","last_change":"2024-04-02T04:54:10.943919+0000","last_active":"2024-04-02T05:00:22.050437+0000","last_peered":"2024-04-02T05:00:22.050437+0000","last_clean":"2024-04-02T05:00:22.050437+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T05:00:22.050437+0000","last_undegraded":"2024-04-02T05:00:22.050437+0000","last_fullsized":"2024-04-02T05:00:22.050437+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:54:10.943768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T04:54:10.943768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1237,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055501+0000","last_change":"2024-04-02T04:59:11.934062+0000","last_active":"2024-04-02T05:00:22.055501+0000","last_peered":"2024-04-02T05:00:22.055501+0000","last_clean":"2024-04-02T05:00:22.055501+0000","last_became_active":"2024-04-02T04:59:00.106341+0000","last_became_peered":"2024-04-02T04:59:00.106341+0000","last_unstale":"2024-04-02T05:00:22.055501+0000","last_undegraded":"2024-04-02T05:00:22.055501+0000","last_fullsized":"2024-04-02T05:00:22.055501+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:11.933852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:11.933852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:25.734157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1309,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.052597+0000","last_change":"2024-04-02T04:59:20.063789+0000","last_active":"2024-04-02T05:00:22.052597+0000","last_peered":"2024-04-02T05:00:22.052597+0000","last_clean":"2024-04-02T05:00:22.052597+0000","last_became_active":"2024-04-02T04:59:20.063395+0000","last_became_peered":"2024-04-02T04:59:20.063395+0000","last_unstale":"2024-04-02T05:00:22.052597+0000","last_undegraded":"2024-04-02T05:00:22.052597+0000","last_fullsized":"2024-04-02T05:00:22.052597+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:55:24.804934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:55:24.804934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1201,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.054851+0000","last_change":"2024-04-02T04:58:51.862964+0000","last_active":"2024-04-02T05:00:22.054851+0000","last_peered":"2024-04-02T05:00:22.054851+0000","last_clean":"2024-04-02T05:00:22.054851+0000","last_became_active":"2024-04-02T04:53:54.942584+0000","last_became_peered":"2024-04-02T04:53:54.942584+0000","last_unstale":"2024-04-02T05:00:22.054851+0000","last_undegraded":"2024-04-02T05:00:22.054851+0000","last_fullsized":"2024-04-02T05:00:22.054851+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:51.862866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:51.862866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"402'35","reported_seq":1401,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542100+0000","last_change":"2024-04-02T04:59:57.839728+0000","last_active":"2024-04-02T05:00:22.542100+0000","last_peered":"2024-04-02T05:00:22.542100+0000","last_clean":"2024-04-02T05:00:22.542100+0000","last_became_active":"2024-04-02T04:43:56.220824+0000","last_became_peered":"2024-04-02T04:43:56.220824+0000","last_unstale":"2024-04-02T05:00:22.542100+0000","last_undegraded":"2024-04-02T05:00:22.542100+0000","last_fullsized":"2024-04-02T05:00:22.542100+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T04:59:57.839474+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T04:59:57.839474+0000","objects_scrubbed":6,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:03.782432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058306+0000","last_change":"2024-04-02T04:59:00.125332+0000","last_active":"2024-04-02T05:00:22.058306+0000","last_peered":"2024-04-02T05:00:22.058306+0000","last_clean":"2024-04-02T05:00:22.058306+0000","last_became_active":"2024-04-02T04:59:00.122388+0000","last_became_peered":"2024-04-02T04:59:00.122388+0000","last_unstale":"2024-04-02T05:00:22.058306+0000","last_undegraded":"2024-04-02T05:00:22.058306+0000","last_fullsized":"2024-04-02T05:00:22.058306+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:54.389749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:57:54.389749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059100+0000","last_change":"2024-04-02T04:59:00.131623+0000","last_active":"2024-04-02T05:00:22.059100+0000","last_peered":"2024-04-02T05:00:22.059100+0000","last_clean":"2024-04-02T05:00:22.059100+0000","last_became_active":"2024-04-02T04:59:00.131269+0000","last_became_peered":"2024-04-02T04:59:00.131269+0000","last_unstale":"2024-04-02T05:00:22.059100+0000","last_undegraded":"2024-04-02T05:00:22.059100+0000","last_fullsized":"2024-04-02T05:00:22.059100+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:38.014594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T04:57:38.014594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059944+0000","last_change":"2024-04-02T04:59:58.815630+0000","last_active":"2024-04-02T05:00:22.059944+0000","last_peered":"2024-04-02T05:00:22.059944+0000","last_clean":"2024-04-02T05:00:22.059944+0000","last_became_active":"2024-04-02T04:59:00.098535+0000","last_became_peered":"2024-04-02T04:59:00.098535+0000","last_unstale":"2024-04-02T05:00:22.059944+0000","last_undegraded":"2024-04-02T05:00:22.059944+0000","last_fullsized":"2024-04-02T05:00:22.059944+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:58.815422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T04:59:58.815422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:07.022525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1206,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055049+0000","last_change":"2024-04-02T04:59:55.124723+0000","last_active":"2024-04-02T05:00:22.055049+0000","last_peered":"2024-04-02T05:00:22.055049+0000","last_clean":"2024-04-02T05:00:22.055049+0000","last_became_active":"2024-04-02T04:59:20.081346+0000","last_became_peered":"2024-04-02T04:59:20.081346+0000","last_unstale":"2024-04-02T05:00:22.055049+0000","last_undegraded":"2024-04-02T05:00:22.055049+0000","last_fullsized":"2024-04-02T05:00:22.055049+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:55.124637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T04:59:55.124637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:10.949069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"402'24","reported_seq":1318,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542994+0000","last_change":"2024-04-02T04:57:45.944117+0000","last_active":"2024-04-02T05:00:22.542994+0000","last_peered":"2024-04-02T05:00:22.542994+0000","last_clean":"2024-04-02T05:00:22.542994+0000","last_became_active":"2024-04-02T04:54:56.014703+0000","last_became_peered":"2024-04-02T04:54:56.014703+0000","last_unstale":"2024-04-02T05:00:22.542994+0000","last_undegraded":"2024-04-02T05:00:22.542994+0000","last_fullsized":"2024-04-02T05:00:22.542994+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'24","last_scrub_stamp":"2024-04-02T04:57:45.943939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:45.943939+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":841,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108463+0000","last_change":"2024-04-02T04:59:06.433001+0000","last_active":"2024-04-02T05:00:23.108463+0000","last_peered":"2024-04-02T05:00:23.108463+0000","last_clean":"2024-04-02T05:00:23.108463+0000","last_became_active":"2024-04-02T04:57:38.129831+0000","last_became_peered":"2024-04-02T04:57:38.129831+0000","last_unstale":"2024-04-02T05:00:23.108463+0000","last_undegraded":"2024-04-02T05:00:23.108463+0000","last_fullsized":"2024-04-02T05:00:23.108463+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T04:59:06.432895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:59:06.432895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:30.540454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1314,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057646+0000","last_change":"2024-04-02T05:00:15.835366+0000","last_active":"2024-04-02T05:00:22.057646+0000","last_peered":"2024-04-02T05:00:22.057646+0000","last_clean":"2024-04-02T05:00:22.057646+0000","last_became_active":"2024-04-02T04:59:20.066268+0000","last_became_peered":"2024-04-02T04:59:20.066268+0000","last_unstale":"2024-04-02T05:00:22.057646+0000","last_undegraded":"2024-04-02T05:00:22.057646+0000","last_fullsized":"2024-04-02T05:00:22.057646+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:15.835222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:15.835222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:41.417156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1332,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.867734+0000","last_change":"2024-04-02T05:00:00.687611+0000","last_active":"2024-04-02T05:00:22.867734+0000","last_peered":"2024-04-02T05:00:22.867734+0000","last_clean":"2024-04-02T05:00:22.867734+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T05:00:22.867734+0000","last_undegraded":"2024-04-02T05:00:22.867734+0000","last_fullsized":"2024-04-02T05:00:22.867734+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:00.687145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:00.687145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:03.534864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":177,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058876+0000","last_change":"2024-04-02T05:00:13.774214+0000","last_active":"2024-04-02T05:00:22.058876+0000","last_peered":"2024-04-02T05:00:22.058876+0000","last_clean":"2024-04-02T05:00:22.058876+0000","last_became_active":"2024-04-02T04:59:00.134110+0000","last_became_peered":"2024-04-02T04:59:00.134110+0000","last_unstale":"2024-04-02T05:00:22.058876+0000","last_undegraded":"2024-04-02T05:00:22.058876+0000","last_fullsized":"2024-04-02T05:00:22.058876+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:13.774005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:13.774005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:38.340370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"402'27","reported_seq":1320,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542148+0000","last_change":"2024-04-02T04:57:31.948865+0000","last_active":"2024-04-02T05:00:22.542148+0000","last_peered":"2024-04-02T05:00:22.542148+0000","last_clean":"2024-04-02T05:00:22.542148+0000","last_became_active":"2024-04-02T04:54:56.011842+0000","last_became_peered":"2024-04-02T04:54:56.011842+0000","last_unstale":"2024-04-02T05:00:22.542148+0000","last_undegraded":"2024-04-02T05:00:22.542148+0000","last_fullsized":"2024-04-02T05:00:22.542148+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T04:57:31.948728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:57:31.948728+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":834,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:23.108626+0000","last_change":"2024-04-02T05:00:05.459382+0000","last_active":"2024-04-02T05:00:23.108626+0000","last_peered":"2024-04-02T05:00:23.108626+0000","last_clean":"2024-04-02T05:00:23.108626+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T05:00:23.108626+0000","last_undegraded":"2024-04-02T05:00:23.108626+0000","last_fullsized":"2024-04-02T05:00:23.108626+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:05.459276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:00:05.459276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:23.918677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1326,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058004+0000","last_change":"2024-04-02T04:59:53.718383+0000","last_active":"2024-04-02T05:00:22.058004+0000","last_peered":"2024-04-02T05:00:22.058004+0000","last_clean":"2024-04-02T05:00:22.058004+0000","last_became_active":"2024-04-02T04:59:20.066444+0000","last_became_peered":"2024-04-02T04:59:20.066444+0000","last_unstale":"2024-04-02T05:00:22.058004+0000","last_undegraded":"2024-04-02T05:00:22.058004+0000","last_fullsized":"2024-04-02T05:00:22.058004+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:53.718262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:59:53.718262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:16.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":588,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.055913+0000","last_change":"2024-04-02T05:00:18.167535+0000","last_active":"2024-04-02T05:00:22.055913+0000","last_peered":"2024-04-02T05:00:22.055913+0000","last_clean":"2024-04-02T05:00:22.055913+0000","last_became_active":"2024-04-02T04:59:55.065891+0000","last_became_peered":"2024-04-02T04:59:55.065891+0000","last_unstale":"2024-04-02T05:00:22.055913+0000","last_undegraded":"2024-04-02T05:00:22.055913+0000","last_fullsized":"2024-04-02T05:00:22.055913+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:18.167325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:18.167325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:43.251892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1199,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.051658+0000","last_change":"2024-04-02T04:59:20.052845+0000","last_active":"2024-04-02T05:00:22.051658+0000","last_peered":"2024-04-02T05:00:22.051658+0000","last_clean":"2024-04-02T05:00:22.051658+0000","last_became_active":"2024-04-02T04:59:20.052388+0000","last_became_peered":"2024-04-02T04:59:20.052388+0000","last_unstale":"2024-04-02T05:00:22.051658+0000","last_undegraded":"2024-04-02T05:00:22.051658+0000","last_fullsized":"2024-04-02T05:00:22.051658+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:57.900914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T04:58:57.900914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"402'25","reported_seq":1443,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871618+0000","last_change":"2024-04-02T04:59:20.098776+0000","last_active":"2024-04-02T05:00:22.871618+0000","last_peered":"2024-04-02T05:00:22.871618+0000","last_clean":"2024-04-02T05:00:22.871618+0000","last_became_active":"2024-04-02T04:59:20.078754+0000","last_became_peered":"2024-04-02T04:59:20.078754+0000","last_unstale":"2024-04-02T05:00:22.871618+0000","last_undegraded":"2024-04-02T05:00:22.871618+0000","last_fullsized":"2024-04-02T05:00:22.871618+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T04:58:58.968760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T04:58:58.968760+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059716+0000","last_change":"2024-04-02T04:59:00.136252+0000","last_active":"2024-04-02T05:00:22.059716+0000","last_peered":"2024-04-02T05:00:22.059716+0000","last_clean":"2024-04-02T05:00:22.059716+0000","last_became_active":"2024-04-02T04:59:00.135892+0000","last_became_peered":"2024-04-02T04:59:00.135892+0000","last_unstale":"2024-04-02T05:00:22.059716+0000","last_undegraded":"2024-04-02T05:00:22.059716+0000","last_fullsized":"2024-04-02T05:00:22.059716+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_clean_scrub_stamp":"2024-04-02T04:57:50.043598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1340,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.540503+0000","last_change":"2024-04-02T05:00:19.869547+0000","last_active":"2024-04-02T05:00:22.540503+0000","last_peered":"2024-04-02T05:00:22.540503+0000","last_clean":"2024-04-02T05:00:22.540503+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T05:00:22.540503+0000","last_undegraded":"2024-04-02T05:00:22.540503+0000","last_fullsized":"2024-04-02T05:00:22.540503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:19.869315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T05:00:19.869315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:39.958182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1334,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.542077+0000","last_change":"2024-04-02T04:57:41.058839+0000","last_active":"2024-04-02T05:00:22.542077+0000","last_peered":"2024-04-02T05:00:22.542077+0000","last_clean":"2024-04-02T05:00:22.542077+0000","last_became_active":"2024-04-02T04:54:55.984764+0000","last_became_peered":"2024-04-02T04:54:55.984764+0000","last_unstale":"2024-04-02T05:00:22.542077+0000","last_undegraded":"2024-04-02T05:00:22.542077+0000","last_fullsized":"2024-04-02T05:00:22.542077+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_clean_scrub_stamp":"2024-04-02T04:57:41.058751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1217,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.543284+0000","last_change":"2024-04-02T05:00:05.822351+0000","last_active":"2024-04-02T05:00:22.543284+0000","last_peered":"2024-04-02T05:00:22.543284+0000","last_clean":"2024-04-02T05:00:22.543284+0000","last_became_active":"2024-04-02T04:59:00.135538+0000","last_became_peered":"2024-04-02T04:59:00.135538+0000","last_unstale":"2024-04-02T05:00:22.543284+0000","last_undegraded":"2024-04-02T05:00:22.543284+0000","last_fullsized":"2024-04-02T05:00:22.543284+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:05.822033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:05.822033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:01:32.234843+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"402'32","reported_seq":191,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.059612+0000","last_change":"2024-04-02T04:59:02.951909+0000","last_active":"2024-04-02T05:00:22.059612+0000","last_peered":"2024-04-02T05:00:22.059612+0000","last_clean":"2024-04-02T05:00:22.059612+0000","last_became_active":"2024-04-02T04:59:00.099386+0000","last_became_peered":"2024-04-02T04:59:00.099386+0000","last_unstale":"2024-04-02T05:00:22.059612+0000","last_undegraded":"2024-04-02T05:00:22.059612+0000","last_fullsized":"2024-04-02T05:00:22.059612+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'32","last_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_deep_scrub":"402'32","last_deep_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_clean_scrub_stamp":"2024-04-02T04:59:02.951751+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1350,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.871789+0000","last_change":"2024-04-02T04:59:20.079425+0000","last_active":"2024-04-02T05:00:22.871789+0000","last_peered":"2024-04-02T05:00:22.871789+0000","last_clean":"2024-04-02T05:00:22.871789+0000","last_became_active":"2024-04-02T04:59:20.077694+0000","last_became_peered":"2024-04-02T04:59:20.077694+0000","last_unstale":"2024-04-02T05:00:22.871789+0000","last_undegraded":"2024-04-02T05:00:22.871789+0000","last_fullsized":"2024-04-02T05:00:22.871789+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:59:07.012983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T04:59:07.012983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:00:31.362765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.057469+0000","last_change":"2024-04-02T04:59:00.138239+0000","last_active":"2024-04-02T05:00:22.057469+0000","last_peered":"2024-04-02T05:00:22.057469+0000","last_clean":"2024-04-02T05:00:22.057469+0000","last_became_active":"2024-04-02T04:59:00.137920+0000","last_became_peered":"2024-04-02T04:59:00.137920+0000","last_unstale":"2024-04-02T05:00:22.057469+0000","last_undegraded":"2024-04-02T05:00:22.057469+0000","last_fullsized":"2024-04-02T05:00:22.057469+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:48.290606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T04:58:48.290606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":169,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-04-02T05:00:22.058192+0000","last_change":"2024-04-02T04:59:00.124528+0000","last_active":"2024-04-02T05:00:22.058192+0000","last_peered":"2024-04-02T05:00:22.058192+0000","last_clean":"2024-04-02T05:00:22.058192+0000","last_became_active":"2024-04-02T04:59:00.122322+0000","last_became_peered":"2024-04-02T04:59:00.122322+0000","last_unstale":"2024-04-02T05:00:22.058192+0000","last_undegraded":"2024-04-02T05:00:22.058192+0000","last_fullsized":"2024-04-02T05:00:22.058192+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:58:52.472669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T04:58:52.472669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":59,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":24,"acting":24,"num_store_stats":7},{"poolid":326,"num_pg":32,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":7},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":4406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":953,"ondisk_log_size":953,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114704,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48932,"kb_used_data":12432,"kb_used_omap":57,"kb_used_meta":36422,"kb_avail":93696220,"statfs":{"total":95995035648,"available":95944929280,"internally_reserved":0,"allocated":12730368,"data_stored":8446990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":37297038},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103387,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115772,"kb_used_data":12380,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93629380,"statfs":{"total":95995035648,"available":95876485120,"internally_reserved":0,"allocated":12677120,"data_stored":8416605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575118,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115512,"kb_used_data":12120,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93629640,"statfs":{"total":95995035648,"available":95876751360,"internally_reserved":0,"allocated":12410880,"data_stored":8119523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738648,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111312,"kb_used_data":12092,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93633840,"statfs":{"total":95995035648,"available":95881052160,"internally_reserved":0,"allocated":12382208,"data_stored":8095777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378884,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115708,"kb_used_data":12328,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93629444,"statfs":{"total":95995035648,"available":95876550656,"internally_reserved":0,"allocated":12623872,"data_stored":8375037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051834,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93832,"kb_used_data":12648,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93651320,"statfs":{"total":95995035648,"available":95898951680,"internally_reserved":0,"allocated":12951552,"data_stored":8683991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822896,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97656,"kb_used_data":12324,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93647496,"statfs":{"total":95995035648,"available":95895035904,"internally_reserved":0,"allocated":12619776,"data_stored":8351888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T05:00:29.717 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-02T05:00:29.718 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pg_num 21 2024-04-02T05:00:29.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:29.802 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:29 smithi098 bash[23790]: cluster 2024-04-02T05:00:28.288515+0000 mgr.y (mgr.14538) 654 : cluster 0 pgmap v1092: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:00:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:29.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:29.943 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:29.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:30.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:30.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 -- 172.21.15.69:0/199998846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 msgr2=0x7f598c074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 --2- 172.21.15.69:0/199998846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c074e30 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f597c009990 tx=0x7f597c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 -- 172.21.15.69:0/199998846 shutdown_connections 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 --2- 172.21.15.69:0/199998846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f598c112f60 0x7f598c115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 --2- 172.21.15.69:0/199998846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c076250 0x7f598c074470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 --2- 172.21.15.69:0/199998846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c074e30 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 -- 172.21.15.69:0/199998846 >> 172.21.15.69:0/199998846 conn(0x7f598c079310 msgr2=0x7f598c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 -- 172.21.15.69:0/199998846 shutdown_connections 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 -- 172.21.15.69:0/199998846 wait complete. 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 librados: starting msgr at 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 librados: starting objecter 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 Processor -- start 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 -- start start 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 librados: setting wanted keys 2024-04-02T05:00:30.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.048+0000 7f5991756700 1 librados: calling monclient init 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f598c076250 0x7f598c19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c112f60 0x7f598c1a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f598c1180d0 con 0x7f598c112f60 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f598c117f50 con 0x7f598c076250 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f598c118250 con 0x7f598c0749b0 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38546/0 (socket says 172.21.15.69:38546) 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 -- 172.21.15.69:0/1877194351 learned_addr learned my addr 172.21.15.69:0/1877194351 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c112f60 0x7f598c1a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f598c076250 0x7f598c19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f598c076250 msgr2=0x7f598c19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f598c076250 0x7f598c19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c112f60 msgr2=0x7f598c1a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c112f60 0x7f598c1a2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598c1a36b0 con 0x7f598c0749b0 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c19e6f0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9a0 tx=0x7f598000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 10 client.?.objecter ms_handle_connect 0x7f598c0749b0 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598000cbd0 con 0x7f598c0749b0 2024-04-02T05:00:30.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f598000cd70 con 0x7f598c0749b0 2024-04-02T05:00:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598c1a39a0 con 0x7f598c0749b0 2024-04-02T05:00:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980010e60 con 0x7f598c0749b0 2024-04-02T05:00:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 10 client.43685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f598c1ab220 con 0x7f598c0749b0 2024-04-02T05:00:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 librados: init done 2024-04-02T05:00:30.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f598c04ff00 con 0x7f598c0749b0 2024-04-02T05:00:30.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5980010430 con 0x7f598c0749b0 2024-04-02T05:00:30.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f597406f5e0 0x7f5974071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(721..721 src has 74..721) ==== 9995+0+0 (secure 0 0 0) 0x7f598008f400 con 0x7f598c0749b0 2024-04-02T05:00:30.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 osd_map(721..721 src has 74..721) 2024-04-02T05:00:30.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 3 client.43685.objecter handle_osd_map got epochs [721,721] > 0 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 3 client.43685.objecter handle_osd_map decoding full epoch 721 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f597406f5e0 0x7f5974071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 20 client.43685.objecter dump_active .. 0 homeless 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v721) ==== 54+0+16 (secure 0 0 0) 0x7f598008f850 con 0x7f598c0749b0 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f597406f5e0 0x7f5974071aa0 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b2e0 tx=0x7f597c005e80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f597406f5e0 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 10 client.43685.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 1 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f598c100ae0 0x7f598c100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 1 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 0 1 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 4 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 10 client.43685.objecter _send_command 1 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f598c1ae5e0 con 0x7f598c100ae0 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.052+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f598c100ae0 0x7f598c100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c100ae0 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f598c100ae0 0x7f598c100ec0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1ae5e0 con 0x7f598c100ae0 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 1: 0 ) 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f5973fff700 10 client.43685.objecter _finish_command 1 = system:0 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 4 2024-04-02T05:00:30.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.056+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 0 1 2024-04-02T05:00:30.074 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:30.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 10 client.43685.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 2 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 2 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c19fa00 osd=0 3 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 0 2 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 4 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 10 client.43685.objecter _send_command 2 2024-04-02T05:00:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.072+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c100560 con 0x7f598c100ae0 2024-04-02T05:00:30.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c100560 con 0x7f598c100ae0 2024-04-02T05:00:30.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 10 client.43685.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 4 2024-04-02T05:00:30.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 0 2 2024-04-02T05:00:30.079 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:00:30.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 10 client.43685.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f598c02c9b0 0x7f598c02cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 1 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 4 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 10 client.43685.objecter _send_command 3 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f598c0634d0 con 0x7f598c02c9b0 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f598c02c9b0 0x7f598c02cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c02c9b0 2024-04-02T05:00:30.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f598c02c9b0 0x7f598c02cdb0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c0634d0 con 0x7f598c02c9b0 2024-04-02T05:00:30.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 3: 0 ) 2024-04-02T05:00:30.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 10 client.43685.objecter _finish_command 3 = system:0 2024-04-02T05:00:30.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 4 2024-04-02T05:00:30.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.076+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 1 3 2024-04-02T05:00:30.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 10 client.43685.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 4 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 4 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c062e50 osd=1 3 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 1 4 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 4 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 10 client.43685.objecter _send_command 4 2024-04-02T05:00:30.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.092+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c0628d0 con 0x7f598c02c9b0 2024-04-02T05:00:30.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c0628d0 con 0x7f598c02c9b0 2024-04-02T05:00:30.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5973fff700 10 client.43685.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 4 2024-04-02T05:00:30.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 1 4 2024-04-02T05:00:30.100 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 10 client.43685.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 5 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f598c1b07a0 0x7f598c1b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 5 2024-04-02T05:00:30.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 2 5 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 4 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 10 client.43685.objecter _send_command 5 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f598c1b59f0 con 0x7f598c1b07a0 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.096+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f598c1b07a0 0x7f598c1b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f598c1b07a0 0x7f598c1b2b90 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c1b07a0 2024-04-02T05:00:30.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1b59f0 con 0x7f598c1b07a0 2024-04-02T05:00:30.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 5: 0 ) 2024-04-02T05:00:30.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f5973fff700 10 client.43685.objecter _finish_command 5 = system:0 2024-04-02T05:00:30.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 4 2024-04-02T05:00:30.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.100+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 2 5 2024-04-02T05:00:30.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 10 client.43685.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 6 2024-04-02T05:00:30.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 6 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c062cd0 osd=2 3 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 2 6 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 4 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 10 client.43685.objecter _send_command 6 2024-04-02T05:00:30.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.116+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c02d540 con 0x7f598c1b07a0 2024-04-02T05:00:30.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c02d540 con 0x7f598c1b07a0 2024-04-02T05:00:30.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 10 client.43685.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 4 2024-04-02T05:00:30.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 2 6 2024-04-02T05:00:30.124 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:00:30.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 10 client.43685.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 7 2024-04-02T05:00:30.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f598c1ae140 0x7f598c1b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 7 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 3 7 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 4 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 10 client.43685.objecter _send_command 7 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f598c1b90c0 con 0x7f598c1ae140 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f598c1ae140 0x7f598c1b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c1ae140 2024-04-02T05:00:30.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f598c1ae140 0x7f598c1b6260 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1b90c0 con 0x7f598c1ae140 2024-04-02T05:00:30.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 7: 0 ) 2024-04-02T05:00:30.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 10 client.43685.objecter _finish_command 7 = system:0 2024-04-02T05:00:30.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 4 2024-04-02T05:00:30.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.120+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 3 7 2024-04-02T05:00:30.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 10 client.43685.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 8 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 8 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c02d540 osd=3 3 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 3 8 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 4 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 10 client.43685.objecter _send_command 8 2024-04-02T05:00:30.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c1ad710 con 0x7f598c1ae140 2024-04-02T05:00:30.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c1ad710 con 0x7f598c1ae140 2024-04-02T05:00:30.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5973fff700 10 client.43685.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 4 2024-04-02T05:00:30.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 3 8 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 10 client.43685.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 9 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f598c1b3db0 0x7f598c1b7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 9 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 4 9 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 4 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 10 client.43685.objecter _send_command 9 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f598c1ad880 con 0x7f598c1b3db0 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f598c1b3db0 0x7f598c1b7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c1b3db0 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.140+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f598c1b3db0 0x7f598c1b7920 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.144+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1ad880 con 0x7f598c1b3db0 2024-04-02T05:00:30.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.144+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 9: 0 ) 2024-04-02T05:00:30.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.144+0000 7f5973fff700 10 client.43685.objecter _finish_command 9 = system:0 2024-04-02T05:00:30.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.144+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 4 2024-04-02T05:00:30.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.144+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 4 9 2024-04-02T05:00:30.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 10 client.43685.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 10 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 10 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1ad710 osd=4 3 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 4 10 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 4 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 10 client.43685.objecter _send_command 10 2024-04-02T05:00:30.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c1ad880 con 0x7f598c1b3db0 2024-04-02T05:00:30.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c1ad880 con 0x7f598c1b3db0 2024-04-02T05:00:30.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5973fff700 10 client.43685.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 4 2024-04-02T05:00:30.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 4 10 2024-04-02T05:00:30.165 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 10 client.43685.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 11 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f598c1b8370 0x7f598c1b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 11 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 5 11 2024-04-02T05:00:30.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 4 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 10 client.43685.objecter _send_command 11 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f598c1b8da0 con 0x7f598c1b8370 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.160+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f598c1b8370 0x7f598c1b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c1b8370 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f598a7fc700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f598c1b8370 0x7f598c1b87d0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1b8da0 con 0x7f598c1b8370 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 11: 0 ) 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f5973fff700 10 client.43685.objecter _finish_command 11 = system:0 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 4 2024-04-02T05:00:30.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.164+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 5 11 2024-04-02T05:00:30.168 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:30.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 10 client.43685.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 12 2024-04-02T05:00:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 12 2024-04-02T05:00:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1ad880 osd=5 3 2024-04-02T05:00:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 5 12 2024-04-02T05:00:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 4 2024-04-02T05:00:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 10 client.43685.objecter _send_command 12 2024-04-02T05:00:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.180+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c1b8da0 con 0x7f598c1b8370 2024-04-02T05:00:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c1b8da0 con 0x7f598c1b8370 2024-04-02T05:00:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 10 client.43685.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 4 2024-04-02T05:00:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 5 12 2024-04-02T05:00:30.188 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:00:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 10 client.43685.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 13 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f598c1bd7d0 0x7f598c1bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 13 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 6 13 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 4 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 10 client.43685.objecter _send_command 13 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f598c1c2be0 con 0x7f598c1bd7d0 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f598c1bd7d0 0x7f598c1bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c1bd7d0 2024-04-02T05:00:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f598b7fe700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f598c1bd7d0 0x7f598c1bfbc0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1c2be0 con 0x7f598c1bd7d0 2024-04-02T05:00:30.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 13: 0 ) 2024-04-02T05:00:30.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 10 client.43685.objecter _finish_command 13 = system:0 2024-04-02T05:00:30.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 4 2024-04-02T05:00:30.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.184+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 6 13 2024-04-02T05:00:30.200 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 10 client.43685.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 14 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 14 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1b8da0 osd=6 3 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 6 14 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 4 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 10 client.43685.objecter _send_command 14 2024-04-02T05:00:30.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.208+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c1c1970 con 0x7f598c1bd7d0 2024-04-02T05:00:30.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c1c1970 con 0x7f598c1bd7d0 2024-04-02T05:00:30.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5973fff700 10 client.43685.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 4 2024-04-02T05:00:30.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 6 14 2024-04-02T05:00:30.217 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 10 client.43685.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 15 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f598c1c2090 0x7f598c1c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 15 2024-04-02T05:00:30.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 7 15 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 4 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 10 client.43685.objecter _send_command 15 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f598c1c5fb0 con 0x7f598c1c2090 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.212+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f598c1c2090 0x7f598c1c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f598affd700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f598c1c2090 0x7f598c1c24d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f5973fff700 10 client.43685.objecter ms_handle_connect 0x7f598c1c2090 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f598c1c5fb0 con 0x7f598c1c2090 2024-04-02T05:00:30.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 15: 0 ) 2024-04-02T05:00:30.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f5973fff700 10 client.43685.objecter _finish_command 15 = system:0 2024-04-02T05:00:30.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 4 2024-04-02T05:00:30.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.216+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 7 15 2024-04-02T05:00:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 10 client.43685.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:00:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign -1 16 2024-04-02T05:00:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 20 client.43685.objecter _get_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 15 client.43685.objecter _session_command_op_remove -1 16 2024-04-02T05:00:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 20 client.43685.objecter get_session s=0x7f598c1ad580 osd=7 3 2024-04-02T05:00:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 15 client.43685.objecter _session_command_op_assign 7 16 2024-04-02T05:00:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 4 2024-04-02T05:00:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 10 client.43685.objecter _send_command 16 2024-04-02T05:00:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.232+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f598c1c4e30 con 0x7f598c1c2090 2024-04-02T05:00:30.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5973fff700 1 -- 172.21.15.69:0/1877194351 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f598c1c4e30 con 0x7f598c1c2090 2024-04-02T05:00:30.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5973fff700 10 client.43685.objecter ms_dispatch 0x7f598c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:30.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5973fff700 10 client.43685.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:30.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5973fff700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 4 2024-04-02T05:00:30.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5973fff700 15 client.43685.objecter _session_command_op_remove 7 16 2024-04-02T05:00:30.242 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5991756700 10 librados: watch_flush enter 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.236+0000 7f5991756700 10 librados: watch_flush exit 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.0 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f598c100ae0 msgr2=0x7f598c100ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f598c100ae0 0x7f598c100ec0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c19fa00 osd=0 1 2024-04-02T05:00:30.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.1 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f598c02c9b0 msgr2=0x7f598c02cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f598c02c9b0 0x7f598c02cdb0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062e50 osd=1 1 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.2 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f598c1b07a0 msgr2=0x7f598c1b2b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f598c1b07a0 0x7f598c1b2b90 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c062cd0 osd=2 1 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.3 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f598c1ae140 msgr2=0x7f598c1b6260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f598c1ae140 0x7f598c1b6260 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c02d540 osd=3 1 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f598b7fe700 1 -- 172.21.15.69:0/1877194351 reap_dead start 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.4 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f598c1b3db0 msgr2=0x7f598c1b7920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f598c1b3db0 0x7f598c1b7920 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f598b7fe700 1 -- 172.21.15.69:0/1877194351 reap_dead start 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad710 osd=4 1 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.5 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f598c1b8370 msgr2=0x7f598c1b87d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f598c1b8370 0x7f598c1b87d0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad880 osd=5 1 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.6 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f598c1bd7d0 msgr2=0x7f598c1bfbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f598c1bd7d0 0x7f598c1bfbc0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1b8da0 osd=6 1 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter close_session for osd.7 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f598c1c2090 msgr2=0x7f598c1c24d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f598c1c2090 0x7f598c1c24d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter put_session s=0x7f598c1ad580 osd=7 1 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 20 client.43685.objecter shutdown clearing up homeless session... 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 10 client.43685.objecter successfully canceled tick 2024-04-02T05:00:30.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f597406f5e0 msgr2=0x7f5974071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f597406f5e0 0x7f5974071aa0 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b2e0 tx=0x7f597c005e80 comp rx=0 tx=0).stop 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f598b7fe700 1 -- 172.21.15.69:0/1877194351 reap_dead start 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 msgr2=0x7f598c19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c19e6f0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9a0 tx=0x7f598000ecb0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 shutdown_connections 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 --2- 172.21.15.69:0/1877194351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c0749b0 0x7f598c19e6f0 secure :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9a0 tx=0x7f598000ecb0 comp rx=0 tx=0).stop 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 >> 172.21.15.69:0/1877194351 conn(0x7f598c079310 msgr2=0x7f598c1119c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 shutdown_connections 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 -- 172.21.15.69:0/1877194351 wait complete. 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:30.240+0000 7f5991756700 1 librados: shutdown 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:00:30.246 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:00:30.247 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:00:30.247 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:00:30.247 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:00:30.247 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:00:30.247 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:00:30.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:30.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:30.425 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:30.452 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:30.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:30.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:30.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:30.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:30.826 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:30 smithi098 bash[23790]: audit 2024-04-02T05:00:30.057928+0000 mon.c (mon.2) 1151 : audit 0 from='client.? 172.21.15.69:0/1877194351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:30.844 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:30.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:30.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:31.030 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:30 smithi069 bash[21528]: audit 2024-04-02T05:00:30.057928+0000 mon.c (mon.2) 1151 : audit 0 from='client.? 172.21.15.69:0/1877194351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:31.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:30 smithi069 bash[28904]: audit 2024-04-02T05:00:30.057928+0000 mon.c (mon.2) 1151 : audit 0 from='client.? 172.21.15.69:0/1877194351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:31.077 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:31.102 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:31.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:31.234 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:31.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:31.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:31.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:31.628 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:31.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:31.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:31.874 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:31 smithi098 bash[23790]: cluster 2024-04-02T05:00:30.289279+0000 mgr.y (mgr.14538) 655 : cluster 0 pgmap v1093: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:00:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:31.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:31.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:31 smithi069 bash[21528]: cluster 2024-04-02T05:00:30.289279+0000 mgr.y (mgr.14538) 655 : cluster 0 pgmap v1093: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:00:31.972 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:31 smithi069 bash[28904]: cluster 2024-04-02T05:00:30.289279+0000 mgr.y (mgr.14538) 655 : cluster 0 pgmap v1093: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:00:31.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:32.017 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:32.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:32.143 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:32.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:32.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:32.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:32.397 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:32.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:32.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:32.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:32.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:32.786 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:32.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:32.914 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:32.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:33.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:33.044 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:33.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:33.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:33.305 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:33.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:33.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:33.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:33.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:00:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:00:33.679 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:33 smithi098 bash[23790]: cluster 2024-04-02T05:00:32.289894+0000 mgr.y (mgr.14538) 656 : cluster 0 pgmap v1094: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 999 B/s rd, 0 op/s 2024-04-02T05:00:33.679 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:33 smithi098 bash[23790]: cluster 2024-04-02T05:00:32.656715+0000 mon.a (mon.0) 3516 : cluster 0 osdmap e722: 8 total, 8 up, 7 in 2024-04-02T05:00:33.679 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:33 smithi098 bash[23790]: audit 2024-04-02T05:00:32.723612+0000 mon.a (mon.0) 3517 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:33.797 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:33.823 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:33.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:33.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:33 smithi069 bash[21528]: cluster 2024-04-02T05:00:32.289894+0000 mgr.y (mgr.14538) 656 : cluster 0 pgmap v1094: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 999 B/s rd, 0 op/s 2024-04-02T05:00:33.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:33 smithi069 bash[21528]: cluster 2024-04-02T05:00:32.656715+0000 mon.a (mon.0) 3516 : cluster 0 osdmap e722: 8 total, 8 up, 7 in 2024-04-02T05:00:33.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:33 smithi069 bash[21528]: audit 2024-04-02T05:00:32.723612+0000 mon.a (mon.0) 3517 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:33.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:33 smithi069 bash[28904]: cluster 2024-04-02T05:00:32.289894+0000 mgr.y (mgr.14538) 656 : cluster 0 pgmap v1094: 214 pgs: 214 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 999 B/s rd, 0 op/s 2024-04-02T05:00:33.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:33 smithi069 bash[28904]: cluster 2024-04-02T05:00:32.656715+0000 mon.a (mon.0) 3516 : cluster 0 osdmap e722: 8 total, 8 up, 7 in 2024-04-02T05:00:33.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:33 smithi069 bash[28904]: audit 2024-04-02T05:00:32.723612+0000 mon.a (mon.0) 3517 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:33.954 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:33.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:33.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:00:34.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:34.079 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:34.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/3711394542 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 msgr2=0x7fb3fc106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- 172.21.15.69:0/3711394542 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc106ed0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009920 tx=0x7fb3ec009c30 comp rx=0 tx=0).stop 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/3711394542 shutdown_connections 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- 172.21.15.69:0/3711394542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc1075b0 0x7fb3fc10f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- 172.21.15.69:0/3711394542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc102580 0x7fb3fc102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- 172.21.15.69:0/3711394542 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc106ed0 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/3711394542 >> 172.21.15.69:0/3711394542 conn(0x7fb3fc0793a0 msgr2=0x7fb3fc0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/3711394542 shutdown_connections 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/3711394542 wait complete. 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 librados: starting msgr at 2024-04-02T05:00:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 librados: starting objecter 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 Processor -- start 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- start start 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 librados: setting wanted keys 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 librados: calling monclient init 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc102580 0x7fb3fc198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc1075b0 0x7fb3fc19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc111ac0 con 0x7fb3fc102580 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc111940 con 0x7fb3fc1075b0 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fb3fc111c40 con 0x7fb3fc102f30 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb401c8f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc102580 0x7fb3fc198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb401c8f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38558/0 (socket says 172.21.15.69:38558) 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb401c8f700 1 -- 172.21.15.69:0/1408653071 learned_addr learned my addr 172.21.15.69:0/1408653071 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402c91700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc1075b0 0x7fb3fc19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 -- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 msgr2=0x7fb3fc1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 -- 172.21.15.69:0/1408653071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc1075b0 msgr2=0x7fb3fc19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc1075b0 0x7fb3fc19ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 -- 172.21.15.69:0/1408653071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc19d170 con 0x7fb3fc102580 2024-04-02T05:00:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb401c8f700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb402490700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc102580 0x7fb3fc198160 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800b450 tx=0x7fb3f800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb3f37fe700 10 client.?.objecter ms_handle_connect 0x7fb3fc102580 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb3f37fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8015020 con 0x7fb3fc102580 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3fc19d460 con 0x7fb3fc102580 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fb3f8011cb0 con 0x7fb3fc102580 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 10 client.33627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3fc1a4ce0 con 0x7fb3fc102580 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 librados: init done 2024-04-02T05:00:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8010c40 con 0x7fb3fc102580 2024-04-02T05:00:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.208+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3fc04ff00 con 0x7fb3fc102580 2024-04-02T05:00:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fb3f8018020 con 0x7fb3fc102580 2024-04-02T05:00:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb3e806f5e0 0x7fb3e8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb401c8f700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb3e806f5e0 0x7fb3e8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(723..723 src has 74..723) ==== 10051+0+0 (secure 0 0 0) 0x7fb3f8091060 con 0x7fb3fc102580 2024-04-02T05:00:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 10 client.33627.objecter ms_dispatch 0x7fb3fc001550 osd_map(723..723 src has 74..723) 2024-04-02T05:00:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 3 client.33627.objecter handle_osd_map got epochs [723,723] > 0 2024-04-02T05:00:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 3 client.33627.objecter handle_osd_map decoding full epoch 723 2024-04-02T05:00:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 20 client.33627.objecter dump_active .. 0 homeless 2024-04-02T05:00:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb3f37fe700 10 client.33627.objecter ms_handle_connect 0x7fb3e806f5e0 2024-04-02T05:00:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.212+0000 7fb401c8f700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb3e806f5e0 0x7fb3e8071aa0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009680 tx=0x7fb3ec019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.216+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb3f800e260 con 0x7fb3fc102580 2024-04-02T05:00:34.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:34.325 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:34.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:34.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.436+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb3fc1a4fa0 con 0x7fb3fc102580 2024-04-02T05:00:34.453 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:34.456 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:34.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:34.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.648+0000 7fb3f37fe700 1 -- 172.21.15.69:0/1408653071 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v724) ==== 97+0+0 (secure 0 0 0) 0x7fb3f8059fc0 con 0x7fb3fc102580 2024-04-02T05:00:34.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 10 librados: watch_flush enter 2024-04-02T05:00:34.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 10 librados: watch_flush exit 2024-04-02T05:00:34.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 20 client.33627.objecter shutdown clearing up homeless session... 2024-04-02T05:00:34.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 10 client.33627.objecter successfully canceled tick 2024-04-02T05:00:34.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb3e806f5e0 msgr2=0x7fb3e8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:34.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb3e806f5e0 0x7fb3e8071aa0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009680 tx=0x7fb3ec019040 comp rx=0 tx=0).stop 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc102580 msgr2=0x7fb3fc198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc102580 0x7fb3fc198160 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800b450 tx=0x7fb3f800b760 comp rx=0 tx=0).stop 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 shutdown_connections 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fb3e806f5e0 0x7fb3e8071aa0 secure :-1 s=CLOSED pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009680 tx=0x7fb3ec019040 comp rx=0 tx=0).stop 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc1075b0 0x7fb3fc19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3fc102580 0x7fb3fc198160 secure :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800b450 tx=0x7fb3f800b760 comp rx=0 tx=0).stop 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 --2- 172.21.15.69:0/1408653071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3fc102f30 0x7fb3fc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 >> 172.21.15.69:0/1408653071 conn(0x7fb3fc0793a0 msgr2=0x7fb3fc100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.652+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 shutdown_connections 2024-04-02T05:00:34.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.656+0000 7fb4046f4700 1 -- 172.21.15.69:0/1408653071 wait complete. 2024-04-02T05:00:34.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:34.656+0000 7fb4046f4700 1 librados: shutdown 2024-04-02T05:00:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T05:00:34.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:34.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:33.608186+0000 mgr.y (mgr.14538) 657 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:34.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:33.651788+0000 mon.a (mon.0) 3518 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:34.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: cluster 2024-04-02T05:00:33.659532+0000 mon.a (mon.0) 3519 : cluster 0 osdmap e723: 8 total, 8 up, 7 in 2024-04-02T05:00:34.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:33.661156+0000 mon.a (mon.0) 3520 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116"}]: dispatch 2024-04-02T05:00:34.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:33.862599+0000 mon.a (mon.0) 3521 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:34.697 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:33.864262+0000 mon.b (mon.1) 300 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:34.698 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:34.444190+0000 mon.a (mon.0) 3522 : audit 1 from='client.? 172.21.15.69:0/1408653071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:00:34.698 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: cluster 2024-04-02T05:00:34.652219+0000 mon.a (mon.0) 3523 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:34.698 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: cluster 2024-04-02T05:00:34.652281+0000 mon.a (mon.0) 3524 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:34.698 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:34.654494+0000 mon.a (mon.0) 3525 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116"}]': finished 2024-04-02T05:00:34.698 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: audit 2024-04-02T05:00:34.654694+0000 mon.a (mon.0) 3526 : audit 1 from='client.? 172.21.15.69:0/1408653071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:00:34.698 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:34 smithi098 bash[23790]: cluster 2024-04-02T05:00:34.659166+0000 mon.a (mon.0) 3527 : cluster 0 osdmap e724: 8 total, 8 up, 7 in 2024-04-02T05:00:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:33.608186+0000 mgr.y (mgr.14538) 657 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:33.651788+0000 mon.a (mon.0) 3518 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: cluster 2024-04-02T05:00:33.659532+0000 mon.a (mon.0) 3519 : cluster 0 osdmap e723: 8 total, 8 up, 7 in 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:33.661156+0000 mon.a (mon.0) 3520 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116"}]: dispatch 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:33.862599+0000 mon.a (mon.0) 3521 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:33.864262+0000 mon.b (mon.1) 300 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:34.444190+0000 mon.a (mon.0) 3522 : audit 1 from='client.? 172.21.15.69:0/1408653071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: cluster 2024-04-02T05:00:34.652219+0000 mon.a (mon.0) 3523 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: cluster 2024-04-02T05:00:34.652281+0000 mon.a (mon.0) 3524 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:34.654494+0000 mon.a (mon.0) 3525 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116"}]': finished 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: audit 2024-04-02T05:00:34.654694+0000 mon.a (mon.0) 3526 : audit 1 from='client.? 172.21.15.69:0/1408653071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:00:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[21528]: cluster 2024-04-02T05:00:34.659166+0000 mon.a (mon.0) 3527 : cluster 0 osdmap e724: 8 total, 8 up, 7 in 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:33.608186+0000 mgr.y (mgr.14538) 657 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:33.651788+0000 mon.a (mon.0) 3518 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: cluster 2024-04-02T05:00:33.659532+0000 mon.a (mon.0) 3519 : cluster 0 osdmap e723: 8 total, 8 up, 7 in 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:33.661156+0000 mon.a (mon.0) 3520 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116"}]: dispatch 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:33.862599+0000 mon.a (mon.0) 3521 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:33.864262+0000 mon.b (mon.1) 300 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:34.444190+0000 mon.a (mon.0) 3522 : audit 1 from='client.? 172.21.15.69:0/1408653071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: cluster 2024-04-02T05:00:34.652219+0000 mon.a (mon.0) 3523 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: cluster 2024-04-02T05:00:34.652281+0000 mon.a (mon.0) 3524 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:34.654494+0000 mon.a (mon.0) 3525 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-116"}]': finished 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: audit 2024-04-02T05:00:34.654694+0000 mon.a (mon.0) 3526 : audit 1 from='client.? 172.21.15.69:0/1408653071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:00:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:34 smithi069 bash[28904]: cluster 2024-04-02T05:00:34.659166+0000 mon.a (mon.0) 3527 : cluster 0 osdmap e724: 8 total, 8 up, 7 in 2024-04-02T05:00:34.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:00:34.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:34.972 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:34.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:35.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:35.102 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:35.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:35.230 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:35.333 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:35.358 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:35.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:35.487 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:35.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:35.620 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:35.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:35.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:35 smithi098 bash[23790]: cluster 2024-04-02T05:00:34.290890+0000 mgr.y (mgr.14538) 658 : cluster 0 pgmap v1097: 214 pgs: 4 active+clean+snaptrim, 210 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T05:00:35.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:35 smithi098 bash[23790]: cluster 2024-04-02T05:00:35.044069+0000 mon.a (mon.0) 3528 : cluster 0 osdmap e725: 8 total, 8 up, 7 in 2024-04-02T05:00:35.749 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:35.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:35.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:35.979 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:35.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:35 smithi069 bash[21528]: cluster 2024-04-02T05:00:34.290890+0000 mgr.y (mgr.14538) 658 : cluster 0 pgmap v1097: 214 pgs: 4 active+clean+snaptrim, 210 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T05:00:35.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:35 smithi069 bash[21528]: cluster 2024-04-02T05:00:35.044069+0000 mon.a (mon.0) 3528 : cluster 0 osdmap e725: 8 total, 8 up, 7 in 2024-04-02T05:00:35.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:35 smithi069 bash[28904]: cluster 2024-04-02T05:00:34.290890+0000 mgr.y (mgr.14538) 658 : cluster 0 pgmap v1097: 214 pgs: 4 active+clean+snaptrim, 210 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-02T05:00:35.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:35 smithi069 bash[28904]: cluster 2024-04-02T05:00:35.044069+0000 mon.a (mon.0) 3528 : cluster 0 osdmap e725: 8 total, 8 up, 7 in 2024-04-02T05:00:36.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:36.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:36.132 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:36.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:36.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:36.383 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:36.486 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:36.512 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:36.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:36.639 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:36.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:36.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:36.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:36.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:37.022 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:37.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:37.146 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:37.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:37.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:37 smithi069 bash[21528]: cluster 2024-04-02T05:00:36.049047+0000 mon.a (mon.0) 3529 : cluster 0 osdmap e726: 8 total, 8 up, 7 in 2024-04-02T05:00:37.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:37 smithi069 bash[21528]: audit 2024-04-02T05:00:36.051544+0000 mon.a (mon.0) 3530 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:37.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:37 smithi069 bash[21528]: cluster 2024-04-02T05:00:36.291760+0000 mgr.y (mgr.14538) 659 : cluster 0 pgmap v1101: 214 pgs: 32 unknown, 2 active+clean+snaptrim, 180 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 843 B/s rd, 1 op/s 2024-04-02T05:00:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:37 smithi069 bash[28904]: cluster 2024-04-02T05:00:36.049047+0000 mon.a (mon.0) 3529 : cluster 0 osdmap e726: 8 total, 8 up, 7 in 2024-04-02T05:00:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:37 smithi069 bash[28904]: audit 2024-04-02T05:00:36.051544+0000 mon.a (mon.0) 3530 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:37.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:37 smithi069 bash[28904]: cluster 2024-04-02T05:00:36.291760+0000 mgr.y (mgr.14538) 659 : cluster 0 pgmap v1101: 214 pgs: 32 unknown, 2 active+clean+snaptrim, 180 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 843 B/s rd, 1 op/s 2024-04-02T05:00:37.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:37.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:37 smithi098 bash[23790]: cluster 2024-04-02T05:00:36.049047+0000 mon.a (mon.0) 3529 : cluster 0 osdmap e726: 8 total, 8 up, 7 in 2024-04-02T05:00:37.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:37 smithi098 bash[23790]: audit 2024-04-02T05:00:36.051544+0000 mon.a (mon.0) 3530 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:37.388 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:37 smithi098 bash[23790]: cluster 2024-04-02T05:00:36.291760+0000 mgr.y (mgr.14538) 659 : cluster 0 pgmap v1101: 214 pgs: 32 unknown, 2 active+clean+snaptrim, 180 active+clean; 582 KiB data, 682 MiB used, 625 GiB / 626 GiB avail; 843 B/s rd, 1 op/s 2024-04-02T05:00:37.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:37.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:37.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:37.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:37.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:37.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:38.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:38.054 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:38.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:38.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:38 smithi098 bash[23790]: cluster 2024-04-02T05:00:37.045661+0000 mon.a (mon.0) 3531 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:38.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:38 smithi098 bash[23790]: audit 2024-04-02T05:00:37.047101+0000 mon.a (mon.0) 3532 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:38.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:38 smithi098 bash[23790]: cluster 2024-04-02T05:00:37.053841+0000 mon.a (mon.0) 3533 : cluster 0 osdmap e727: 8 total, 8 up, 7 in 2024-04-02T05:00:38.183 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:38.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[21528]: cluster 2024-04-02T05:00:37.045661+0000 mon.a (mon.0) 3531 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:38.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[21528]: audit 2024-04-02T05:00:37.047101+0000 mon.a (mon.0) 3532 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:38.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[21528]: cluster 2024-04-02T05:00:37.053841+0000 mon.a (mon.0) 3533 : cluster 0 osdmap e727: 8 total, 8 up, 7 in 2024-04-02T05:00:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[28904]: cluster 2024-04-02T05:00:37.045661+0000 mon.a (mon.0) 3531 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[28904]: audit 2024-04-02T05:00:37.047101+0000 mon.a (mon.0) 3532 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:38.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[28904]: cluster 2024-04-02T05:00:37.053841+0000 mon.a (mon.0) 3533 : cluster 0 osdmap e727: 8 total, 8 up, 7 in 2024-04-02T05:00:38.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:38.304 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:38.350 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T05:00:38.351 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:00:38.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:38.428 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:38.531 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:38.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:38.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:38.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:38.688 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:38.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T05:00:38.791 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:38.814 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:38.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:38.944 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:38.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:38.951 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:00:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:00:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:00:39.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:39.069 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- 172.21.15.69:0/3304116709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c41073c0 msgr2=0x7f45c41077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/3304116709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c41073c0 0x7f45c41077a0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009990 tx=0x7f45b4009ca0 comp rx=0 tx=0).stop 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- 172.21.15.69:0/3304116709 shutdown_connections 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/3304116709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 0x7f45c4104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/3304116709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c41073c0 0x7f45c41077a0 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/3304116709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c4100790 0x7f45c4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- 172.21.15.69:0/3304116709 >> 172.21.15.69:0/3304116709 conn(0x7f45c4077740 msgr2=0x7f45c40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- 172.21.15.69:0/3304116709 shutdown_connections 2024-04-02T05:00:39.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- 172.21.15.69:0/3304116709 wait complete. 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 librados: starting msgr at 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 librados: starting objecter 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 Processor -- start 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- start start 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 librados: setting wanted keys 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 librados: calling monclient init 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 0x7f45c41a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 0x7f45c41a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c41073c0 0x7f45c419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45c4118330 con 0x7f45c4101230 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c41181b0 con 0x7f45c41073c0 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c9fa0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f45c41184b0 con 0x7f45c4100790 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 0x7f45c41a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c2ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 0x7f45c41a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:39.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 0x7f45c41a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36572/0 (socket says 172.21.15.69:36572) 2024-04-02T05:00:39.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 -- 172.21.15.69:0/396641493 learned_addr learned my addr 172.21.15.69:0/396641493 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c3fff700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c41073c0 0x7f45c419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 -- 172.21.15.69:0/396641493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c41073c0 msgr2=0x7f45c419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c41073c0 0x7f45c419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 -- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 msgr2=0x7f45c41a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 0x7f45c41a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 -- 172.21.15.69:0/396641493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c419ef50 con 0x7f45c4100790 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c2ffd700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 0x7f45c41a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c37fe700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 0x7f45c41a45a0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f45b4000c00 tx=0x7f45b40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c0ff9700 10 client.?.objecter ms_handle_connect 0x7f45c4100790 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c0ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4027070 con 0x7f45c4100790 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f45b4022470 con 0x7f45c4100790 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.088+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4017650 con 0x7f45c4100790 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c419f1e0 con 0x7f45c4100790 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c9fa0700 10 client.43697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c41ab350 con 0x7f45c4100790 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c9fa0700 1 librados: init done 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c4101a40 con 0x7f45c4100790 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f45b402a070 con 0x7f45c4100790 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ac06f5e0 0x7f45ac071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(729..729 src has 74..729) ==== 10051+0+0 (secure 0 0 0) 0x7f45b408f930 con 0x7f45c4100790 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 10 client.43697.objecter ms_dispatch 0x7f45c4001550 osd_map(729..729 src has 74..729) 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 3 client.43697.objecter handle_osd_map got epochs [729,729] > 0 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 3 client.43697.objecter handle_osd_map decoding full epoch 729 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c2ffd700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ac06f5e0 0x7f45ac071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 20 client.43697.objecter dump_active .. 0 homeless 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c2ffd700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ac06f5e0 0x7f45ac071aa0 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7f45c419fe30 tx=0x7f45b800b3c0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.092+0000 7f45c0ff9700 10 client.43697.objecter ms_handle_connect 0x7f45ac06f5e0 2024-04-02T05:00:39.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.096+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f45b4013070 con 0x7f45c4100790 2024-04-02T05:00:39.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:39.183 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:39 smithi098 bash[23790]: cluster 2024-04-02T05:00:38.079921+0000 mon.a (mon.0) 3534 : cluster 0 osdmap e728: 8 total, 8 up, 7 in 2024-04-02T05:00:39.183 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:39 smithi098 bash[23790]: audit 2024-04-02T05:00:38.094288+0000 mon.a (mon.0) 3535 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:39.183 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:39 smithi098 bash[23790]: cluster 2024-04-02T05:00:38.293327+0000 mgr.y (mgr.14538) 660 : cluster 0 pgmap v1104: 214 pgs: 4 unknown, 1 active+clean+snaptrim, 209 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:39.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-02T05:00:39.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:39.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:39.300+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} v 0) -- 0x7f45c419fba0 con 0x7f45c4100790 2024-04-02T05:00:39.327 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:39.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:39.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:39 smithi069 bash[21528]: cluster 2024-04-02T05:00:38.079921+0000 mon.a (mon.0) 3534 : cluster 0 osdmap e728: 8 total, 8 up, 7 in 2024-04-02T05:00:39.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:39 smithi069 bash[21528]: audit 2024-04-02T05:00:38.094288+0000 mon.a (mon.0) 3535 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:39.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:39 smithi069 bash[21528]: cluster 2024-04-02T05:00:38.293327+0000 mgr.y (mgr.14538) 660 : cluster 0 pgmap v1104: 214 pgs: 4 unknown, 1 active+clean+snaptrim, 209 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:39 smithi069 bash[28904]: cluster 2024-04-02T05:00:38.079921+0000 mon.a (mon.0) 3534 : cluster 0 osdmap e728: 8 total, 8 up, 7 in 2024-04-02T05:00:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:39 smithi069 bash[28904]: audit 2024-04-02T05:00:38.094288+0000 mon.a (mon.0) 3535 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:39.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:39 smithi069 bash[28904]: cluster 2024-04-02T05:00:38.293327+0000 mgr.y (mgr.14538) 660 : cluster 0 pgmap v1104: 214 pgs: 4 unknown, 1 active+clean+snaptrim, 209 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:39.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:39.578 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:39.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:39.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:39.842 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:39.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:39.970 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:39.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:40.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:40.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.096+0000 7f45c0ff9700 1 -- 172.21.15.69:0/396641493 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]=0 set pool 1 pg_num to 21 v730) ==== 129+0+0 (secure 0 0 0) 0x7f45b405c570 con 0x7f45c4100790 2024-04-02T05:00:40.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 10 librados: watch_flush enter 2024-04-02T05:00:40.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 10 librados: watch_flush exit 2024-04-02T05:00:40.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 20 client.43697.objecter shutdown clearing up homeless session... 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 10 client.43697.objecter successfully canceled tick 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ac06f5e0 msgr2=0x7f45ac071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ac06f5e0 0x7f45ac071aa0 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7f45c419fe30 tx=0x7f45b800b3c0 comp rx=0 tx=0).stop 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 msgr2=0x7f45c41a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 0x7f45c41a45a0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f45b4000c00 tx=0x7f45b40047b0 comp rx=0 tx=0).stop 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 shutdown_connections 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f45ac06f5e0 0x7f45ac071aa0 secure :-1 s=CLOSED pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7f45c419fe30 tx=0x7f45b800b3c0 comp rx=0 tx=0).stop 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c41073c0 0x7f45c419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45c4100790 0x7f45c41a45a0 secure :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f45b4000c00 tx=0x7f45b40047b0 comp rx=0 tx=0).stop 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 --2- 172.21.15.69:0/396641493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45c4101230 0x7f45c41a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:40.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.100+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 >> 172.21.15.69:0/396641493 conn(0x7f45c4077740 msgr2=0x7f45c4102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.104+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 shutdown_connections 2024-04-02T05:00:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.104+0000 7f45c9fa0700 1 -- 172.21.15.69:0/396641493 wait complete. 2024-04-02T05:00:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:40.104+0000 7f45c9fa0700 1 librados: shutdown 2024-04-02T05:00:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:40.108 INFO:teuthology.orchestra.run.smithi069.stderr:set pool 1 pg_num to 21 2024-04-02T05:00:40.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: cluster 2024-04-02T05:00:38.732203+0000 osd.5 (osd.5) 259 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: cluster 2024-04-02T05:00:38.732582+0000 osd.5 (osd.5) 260 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: audit 2024-04-02T05:00:39.079285+0000 mon.a (mon.0) 3536 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: cluster 2024-04-02T05:00:39.085757+0000 mon.a (mon.0) 3537 : cluster 0 osdmap e729: 8 total, 8 up, 7 in 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: audit 2024-04-02T05:00:39.088975+0000 mon.a (mon.0) 3538 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-118"}]: dispatch 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: audit 2024-04-02T05:00:39.307480+0000 mon.c (mon.2) 1152 : audit 1 from='client.? 172.21.15.69:0/396641493' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-02T05:00:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:40 smithi098 bash[23790]: audit 2024-04-02T05:00:39.308139+0000 mon.a (mon.0) 3539 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-02T05:00:40.227 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:40.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:40.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: cluster 2024-04-02T05:00:38.732203+0000 osd.5 (osd.5) 259 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: cluster 2024-04-02T05:00:38.732582+0000 osd.5 (osd.5) 260 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: audit 2024-04-02T05:00:39.079285+0000 mon.a (mon.0) 3536 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: cluster 2024-04-02T05:00:39.085757+0000 mon.a (mon.0) 3537 : cluster 0 osdmap e729: 8 total, 8 up, 7 in 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: audit 2024-04-02T05:00:39.088975+0000 mon.a (mon.0) 3538 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-118"}]: dispatch 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: audit 2024-04-02T05:00:39.307480+0000 mon.c (mon.2) 1152 : audit 1 from='client.? 172.21.15.69:0/396641493' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[21528]: audit 2024-04-02T05:00:39.308139+0000 mon.a (mon.0) 3539 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: cluster 2024-04-02T05:00:38.732203+0000 osd.5 (osd.5) 259 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: cluster 2024-04-02T05:00:38.732582+0000 osd.5 (osd.5) 260 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: audit 2024-04-02T05:00:39.079285+0000 mon.a (mon.0) 3536 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: cluster 2024-04-02T05:00:39.085757+0000 mon.a (mon.0) 3537 : cluster 0 osdmap e729: 8 total, 8 up, 7 in 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: audit 2024-04-02T05:00:39.088975+0000 mon.a (mon.0) 3538 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-118"}]: dispatch 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: audit 2024-04-02T05:00:39.307480+0000 mon.c (mon.2) 1152 : audit 1 from='client.? 172.21.15.69:0/396641493' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-02T05:00:40.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:40 smithi069 bash[28904]: audit 2024-04-02T05:00:39.308139+0000 mon.a (mon.0) 3539 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-02T05:00:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:40.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:40.484 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:40.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:40.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:40.748 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:40.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:40.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:40.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:41.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:41.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:41.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:41.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:41.271 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:40.092858+0000 mon.a (mon.0) 3540 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-118"}]': finished 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:40.092974+0000 mon.a (mon.0) 3541 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: cluster 2024-04-02T05:00:40.104951+0000 mon.a (mon.0) 3542 : cluster 0 osdmap e730: 8 total, 8 up, 7 in 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:40.117900+0000 mon.a (mon.0) 3543 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-118", "mode": "writeback"}]: dispatch 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:40.292344+0000 mon.b (mon.1) 301 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: cluster 2024-04-02T05:00:40.294073+0000 mgr.y (mgr.14538) 661 : cluster 0 pgmap v1107: 214 pgs: 214 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:40.296015+0000 mon.a (mon.0) 3544 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: cluster 2024-04-02T05:00:41.093702+0000 mon.a (mon.0) 3545 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:41.096079+0000 mon.a (mon.0) 3546 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-118", "mode": "writeback"}]': finished 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: audit 2024-04-02T05:00:41.096300+0000 mon.a (mon.0) 3547 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[21528]: cluster 2024-04-02T05:00:41.105984+0000 mon.a (mon.0) 3548 : cluster 0 osdmap e731: 8 total, 8 up, 7 in 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:40.092858+0000 mon.a (mon.0) 3540 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-118"}]': finished 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:40.092974+0000 mon.a (mon.0) 3541 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: cluster 2024-04-02T05:00:40.104951+0000 mon.a (mon.0) 3542 : cluster 0 osdmap e730: 8 total, 8 up, 7 in 2024-04-02T05:00:41.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:40.117900+0000 mon.a (mon.0) 3543 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-118", "mode": "writeback"}]: dispatch 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:40.292344+0000 mon.b (mon.1) 301 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: cluster 2024-04-02T05:00:40.294073+0000 mgr.y (mgr.14538) 661 : cluster 0 pgmap v1107: 214 pgs: 214 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:40.296015+0000 mon.a (mon.0) 3544 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: cluster 2024-04-02T05:00:41.093702+0000 mon.a (mon.0) 3545 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:41.096079+0000 mon.a (mon.0) 3546 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-118", "mode": "writeback"}]': finished 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: audit 2024-04-02T05:00:41.096300+0000 mon.a (mon.0) 3547 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-02T05:00:41.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:41 smithi069 bash[28904]: cluster 2024-04-02T05:00:41.105984+0000 mon.a (mon.0) 3548 : cluster 0 osdmap e731: 8 total, 8 up, 7 in 2024-04-02T05:00:41.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:41.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:40.092858+0000 mon.a (mon.0) 3540 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-118"}]': finished 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:40.092974+0000 mon.a (mon.0) 3541 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: cluster 2024-04-02T05:00:40.104951+0000 mon.a (mon.0) 3542 : cluster 0 osdmap e730: 8 total, 8 up, 7 in 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:40.117900+0000 mon.a (mon.0) 3543 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-118", "mode": "writeback"}]: dispatch 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:40.292344+0000 mon.b (mon.1) 301 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: cluster 2024-04-02T05:00:40.294073+0000 mgr.y (mgr.14538) 661 : cluster 0 pgmap v1107: 214 pgs: 214 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:40.296015+0000 mon.a (mon.0) 3544 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: cluster 2024-04-02T05:00:41.093702+0000 mon.a (mon.0) 3545 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:41.096079+0000 mon.a (mon.0) 3546 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-118", "mode": "writeback"}]': finished 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: audit 2024-04-02T05:00:41.096300+0000 mon.a (mon.0) 3547 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-02T05:00:41.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:41 smithi098 bash[23790]: cluster 2024-04-02T05:00:41.105984+0000 mon.a (mon.0) 3548 : cluster 0 osdmap e731: 8 total, 8 up, 7 in 2024-04-02T05:00:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:41.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:41.528 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:41.579 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:41.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:41.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:41.785 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:41.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:41.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:42.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:42.042 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:42.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:42.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:42.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:42.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:42.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:42.798 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:42.819 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:42.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:42.951 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:42.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:43.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:43.083 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:43.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[21528]: cluster 2024-04-02T05:00:42.112482+0000 mon.a (mon.0) 3549 : cluster 0 osdmap e732: 8 total, 8 up, 7 in 2024-04-02T05:00:43.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[21528]: audit 2024-04-02T05:00:42.184937+0000 mon.a (mon.0) 3550 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:43.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[21528]: audit 2024-04-02T05:00:42.293017+0000 mon.b (mon.1) 302 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:43.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[21528]: cluster 2024-04-02T05:00:42.295060+0000 mgr.y (mgr.14538) 662 : cluster 0 pgmap v1110: 224 pgs: 10 unknown, 214 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:43.210 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[21528]: audit 2024-04-02T05:00:42.298050+0000 mon.a (mon.0) 3551 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:43.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[28904]: cluster 2024-04-02T05:00:42.112482+0000 mon.a (mon.0) 3549 : cluster 0 osdmap e732: 8 total, 8 up, 7 in 2024-04-02T05:00:43.211 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[28904]: audit 2024-04-02T05:00:42.184937+0000 mon.a (mon.0) 3550 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:43.211 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[28904]: audit 2024-04-02T05:00:42.293017+0000 mon.b (mon.1) 302 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:43.211 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[28904]: cluster 2024-04-02T05:00:42.295060+0000 mgr.y (mgr.14538) 662 : cluster 0 pgmap v1110: 224 pgs: 10 unknown, 214 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:43.211 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:43 smithi069 bash[28904]: audit 2024-04-02T05:00:42.298050+0000 mon.a (mon.0) 3551 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:43 smithi098 bash[23790]: cluster 2024-04-02T05:00:42.112482+0000 mon.a (mon.0) 3549 : cluster 0 osdmap e732: 8 total, 8 up, 7 in 2024-04-02T05:00:43.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:43 smithi098 bash[23790]: audit 2024-04-02T05:00:42.184937+0000 mon.a (mon.0) 3550 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:43 smithi098 bash[23790]: audit 2024-04-02T05:00:42.293017+0000 mon.b (mon.1) 302 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:43 smithi098 bash[23790]: cluster 2024-04-02T05:00:42.295060+0000 mgr.y (mgr.14538) 662 : cluster 0 pgmap v1110: 224 pgs: 10 unknown, 214 active+clean; 582 KiB data, 683 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:43.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:43 smithi098 bash[23790]: audit 2024-04-02T05:00:42.298050+0000 mon.a (mon.0) 3551 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-02T05:00:43.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:43.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.467 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:43.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:43.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:43.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.856 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:43.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:00:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:00:43.986 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:43.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:00:44.089 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:44.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:44.137 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:44.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:44.266 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: audit 2024-04-02T05:00:43.100487+0000 mon.a (mon.0) 3552 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: audit 2024-04-02T05:00:43.100574+0000 mon.a (mon.0) 3553 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: cluster 2024-04-02T05:00:43.105466+0000 mon.a (mon.0) 3554 : cluster 0 osdmap e733: 8 total, 8 up, 7 in 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: audit 2024-04-02T05:00:43.107339+0000 mon.a (mon.0) 3555 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118"}]: dispatch 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: audit 2024-04-02T05:00:43.615279+0000 mgr.y (mgr.14538) 663 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: cluster 2024-04-02T05:00:44.100780+0000 mon.a (mon.0) 3556 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: audit 2024-04-02T05:00:44.102513+0000 mon.a (mon.0) 3557 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118"}]': finished 2024-04-02T05:00:44.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[21528]: cluster 2024-04-02T05:00:44.112600+0000 mon.a (mon.0) 3558 : cluster 0 osdmap e734: 8 total, 8 up, 7 in 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: audit 2024-04-02T05:00:43.100487+0000 mon.a (mon.0) 3552 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: audit 2024-04-02T05:00:43.100574+0000 mon.a (mon.0) 3553 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: cluster 2024-04-02T05:00:43.105466+0000 mon.a (mon.0) 3554 : cluster 0 osdmap e733: 8 total, 8 up, 7 in 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: audit 2024-04-02T05:00:43.107339+0000 mon.a (mon.0) 3555 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118"}]: dispatch 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: audit 2024-04-02T05:00:43.615279+0000 mgr.y (mgr.14538) 663 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: cluster 2024-04-02T05:00:44.100780+0000 mon.a (mon.0) 3556 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: audit 2024-04-02T05:00:44.102513+0000 mon.a (mon.0) 3557 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118"}]': finished 2024-04-02T05:00:44.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:44 smithi069 bash[28904]: cluster 2024-04-02T05:00:44.112600+0000 mon.a (mon.0) 3558 : cluster 0 osdmap e734: 8 total, 8 up, 7 in 2024-04-02T05:00:44.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:44.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: audit 2024-04-02T05:00:43.100487+0000 mon.a (mon.0) 3552 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:44.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: audit 2024-04-02T05:00:43.100574+0000 mon.a (mon.0) 3553 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-02T05:00:44.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: cluster 2024-04-02T05:00:43.105466+0000 mon.a (mon.0) 3554 : cluster 0 osdmap e733: 8 total, 8 up, 7 in 2024-04-02T05:00:44.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: audit 2024-04-02T05:00:43.107339+0000 mon.a (mon.0) 3555 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118"}]: dispatch 2024-04-02T05:00:44.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: audit 2024-04-02T05:00:43.615279+0000 mgr.y (mgr.14538) 663 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:44.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: cluster 2024-04-02T05:00:44.100780+0000 mon.a (mon.0) 3556 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:44.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: audit 2024-04-02T05:00:44.102513+0000 mon.a (mon.0) 3557 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-118"}]': finished 2024-04-02T05:00:44.381 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:44 smithi098 bash[23790]: cluster 2024-04-02T05:00:44.112600+0000 mon.a (mon.0) 3558 : cluster 0 osdmap e734: 8 total, 8 up, 7 in 2024-04-02T05:00:44.395 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:44.498 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:44.527 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:44.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:44.657 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:44.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:44.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:44.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:45.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:45.047 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:45.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:45.170 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:45.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:45.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:45.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:45.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:45.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:45.551 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:45.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:45 smithi098 bash[23790]: cluster 2024-04-02T05:00:44.295913+0000 mgr.y (mgr.14538) 664 : cluster 0 pgmap v1113: 224 pgs: 1 peering, 1 active+clean+snaptrim, 5 unknown, 217 active+clean; 582 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:45 smithi098 bash[23790]: cluster 2024-04-02T05:00:45.042080+0000 mon.a (mon.0) 3559 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:45.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:45 smithi098 bash[23790]: cluster 2024-04-02T05:00:45.058696+0000 mon.a (mon.0) 3560 : cluster 0 osdmap e735: 8 total, 8 up, 7 in 2024-04-02T05:00:45.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:45.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:45 smithi069 bash[21528]: cluster 2024-04-02T05:00:44.295913+0000 mgr.y (mgr.14538) 664 : cluster 0 pgmap v1113: 224 pgs: 1 peering, 1 active+clean+snaptrim, 5 unknown, 217 active+clean; 582 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:45.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:45 smithi069 bash[21528]: cluster 2024-04-02T05:00:45.042080+0000 mon.a (mon.0) 3559 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:45.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:45 smithi069 bash[21528]: cluster 2024-04-02T05:00:45.058696+0000 mon.a (mon.0) 3560 : cluster 0 osdmap e735: 8 total, 8 up, 7 in 2024-04-02T05:00:45.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:45 smithi069 bash[28904]: cluster 2024-04-02T05:00:44.295913+0000 mgr.y (mgr.14538) 664 : cluster 0 pgmap v1113: 224 pgs: 1 peering, 1 active+clean+snaptrim, 5 unknown, 217 active+clean; 582 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:45.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:45 smithi069 bash[28904]: cluster 2024-04-02T05:00:45.042080+0000 mon.a (mon.0) 3559 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:45.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:45 smithi069 bash[28904]: cluster 2024-04-02T05:00:45.058696+0000 mon.a (mon.0) 3560 : cluster 0 osdmap e735: 8 total, 8 up, 7 in 2024-04-02T05:00:45.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:45.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:45.938 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:45.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:46.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:46.079 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:46.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T05:00:46.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T05:00:46.157 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-02T05:00:46.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 'trim stale osdmaps' 2024-04-02T05:00:46.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:46.225 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:46.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:46.356 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:46.459 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- 172.21.15.69:0/1917896054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806ab80 msgr2=0x7f63e8106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/1917896054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806ab80 0x7f63e8106ef0 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0099c0 tx=0x7f63dc009cd0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- 172.21.15.69:0/1917896054 shutdown_connections 2024-04-02T05:00:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/1917896054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63e8107620 0x7f63e810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/1917896054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e806a1d0 0x7f63e806a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/1917896054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806ab80 0x7f63e8106ef0 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- 172.21.15.69:0/1917896054 >> 172.21.15.69:0/1917896054 conn(0x7f63e80771e0 msgr2=0x7f63e80775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:46.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- 172.21.15.69:0/1917896054 shutdown_connections 2024-04-02T05:00:46.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- 172.21.15.69:0/1917896054 wait complete. 2024-04-02T05:00:46.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 librados: starting msgr at 2024-04-02T05:00:46.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 librados: starting objecter 2024-04-02T05:00:46.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 Processor -- start 2024-04-02T05:00:46.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- start start 2024-04-02T05:00:46.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 librados: setting wanted keys 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 librados: calling monclient init 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806a1d0 0x7f63e81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e806ab80 0x7f63e8198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63e8107620 0x7f63e819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63e8111ab0 con 0x7f63e8107620 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63e8111930 con 0x7f63e806ab80 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63ee4f5700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f63e8111c30 con 0x7f63e806a1d0 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806a1d0 0x7f63e81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63eca92700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63e8107620 0x7f63e819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806a1d0 0x7f63e81980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36588/0 (socket says 172.21.15.69:36588) 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 -- 172.21.15.69:0/3285683465 learned_addr learned my addr 172.21.15.69:0/3285683465 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e806ab80 0x7f63e8198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e806ab80 msgr2=0x7f63e8198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63e806ab80 0x7f63e8198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63e8107620 msgr2=0x7f63e819c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63e8107620 0x7f63e819c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e819d0e0 con 0x7f63e806a1d0 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63e8107620 0x7f63e819c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806a1d0 0x7f63e81980d0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f63d800b530 tx=0x7f63d800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e57fa700 10 client.?.objecter ms_handle_connect 0x7f63e806a1d0 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.548+0000 7f63e57fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:46.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8015020 con 0x7f63e806a1d0 2024-04-02T05:00:46.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f63d8011cb0 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8010d90 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e819d3d0 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 10 client.43703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e81a4c50 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 1 librados: init done 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f63e81039d0 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f63d8018020 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f63d006f5e0 0x7f63d0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f63d006f5e0 0x7f63d0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(736..736 src has 74..736) ==== 10071+0+0 (secure 0 0 0) 0x7f63d808e340 con 0x7f63e806a1d0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 osd_map(736..736 src has 74..736) 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 3 client.43703.objecter handle_osd_map got epochs [736,736] > 0 2024-04-02T05:00:46.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 3 client.43703.objecter handle_osd_map decoding full epoch 736 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 20 client.43703.objecter dump_active .. 0 homeless 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v736) ==== 54+0+16 (secure 0 0 0) 0x7f63d801f020 con 0x7f63e806a1d0 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63d006f5e0 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f63d006f5e0 0x7f63d0071aa0 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009990 tx=0x7f63dc005fd0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 1 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63e8062e50 0x7f63e8063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 1 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 0 1 2024-04-02T05:00:46.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 4 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 10 client.43703.objecter _send_command 1 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63e804ff00 con 0x7f63e8062e50 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63e8062e50 0x7f63e8063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63e8062e50 0x7f63e8063230 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.552+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e8062e50 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.556+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e804ff00 con 0x7f63e8062e50 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.556+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 1: 0 ) 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.556+0000 7f63e57fa700 10 client.43703.objecter _finish_command 1 = system:0 2024-04-02T05:00:46.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.556+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 4 2024-04-02T05:00:46.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.556+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 0 1 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 2 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 2 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e8199430 osd=0 3 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 0 2 2024-04-02T05:00:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 4 2024-04-02T05:00:46.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 10 client.43703.objecter _send_command 2 2024-04-02T05:00:46.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.572+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e80628d0 con 0x7f63e8062e50 2024-04-02T05:00:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e80628d0 con 0x7f63e8062e50 2024-04-02T05:00:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 10 client.43703.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 4 2024-04-02T05:00:46.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 0 2 2024-04-02T05:00:46.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:46.580 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:00:46.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 3 2024-04-02T05:00:46.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f63e8109d50 0x7f63e81ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 3 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 1 3 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 4 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 10 client.43703.objecter _send_command 3 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f63e81031b0 con 0x7f63e8109d50 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f63e8109d50 0x7f63e81ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e8109d50 2024-04-02T05:00:46.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f63e8109d50 0x7f63e81ab2a0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e81031b0 con 0x7f63e8109d50 2024-04-02T05:00:46.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 3: 0 ) 2024-04-02T05:00:46.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 10 client.43703.objecter _finish_command 3 = system:0 2024-04-02T05:00:46.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 4 2024-04-02T05:00:46.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.576+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 1 3 2024-04-02T05:00:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 4 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 4 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e802db30 osd=1 3 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 1 4 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 4 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 10 client.43703.objecter _send_command 4 2024-04-02T05:00:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.596+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e802def0 con 0x7f63e8109d50 2024-04-02T05:00:46.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e802def0 con 0x7f63e8109d50 2024-04-02T05:00:46.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 10 client.43703.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 4 2024-04-02T05:00:46.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 1 4 2024-04-02T05:00:46.603 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 5 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f63e8109430 0x7f63e81ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 5 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 2 5 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 4 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 10 client.43703.objecter _send_command 5 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f63e802d720 con 0x7f63e8109430 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f63e8109430 0x7f63e81ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f63e8109430 0x7f63e81ae970 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e8109430 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e802d720 con 0x7f63e8109430 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 5: 0 ) 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 10 client.43703.objecter _finish_command 5 = system:0 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 4 2024-04-02T05:00:46.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.600+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 2 5 2024-04-02T05:00:46.608 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 6 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 6 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81092c0 osd=2 3 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 2 6 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 4 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 10 client.43703.objecter _send_command 6 2024-04-02T05:00:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.616+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e802cc00 con 0x7f63e8109430 2024-04-02T05:00:46.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e802cc00 con 0x7f63e8109430 2024-04-02T05:00:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 10 client.43703.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 4 2024-04-02T05:00:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 2 6 2024-04-02T05:00:46.624 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 7 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63e81ac4c0 0x7f63e81b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 7 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 3 7 2024-04-02T05:00:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 4 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 10 client.43703.objecter _send_command 7 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f63e81b4fa0 con 0x7f63e81ac4c0 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63e81ac4c0 0x7f63e81b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e81ac4c0 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63e81ac4c0 0x7f63e81b2040 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e81b4fa0 con 0x7f63e81ac4c0 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 7: 0 ) 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 10 client.43703.objecter _finish_command 7 = system:0 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 4 2024-04-02T05:00:46.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.620+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 3 7 2024-04-02T05:00:46.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 8 2024-04-02T05:00:46.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 8 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e802d000 osd=3 3 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 3 8 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 4 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 10 client.43703.objecter _send_command 8 2024-04-02T05:00:46.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e81abb50 con 0x7f63e81ac4c0 2024-04-02T05:00:46.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e81abb50 con 0x7f63e81ac4c0 2024-04-02T05:00:46.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e57fa700 10 client.43703.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 4 2024-04-02T05:00:46.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 3 8 2024-04-02T05:00:46.645 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:00:46.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 9 2024-04-02T05:00:46.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63e81afb90 0x7f63e81b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 9 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 4 9 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 4 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 10 client.43703.objecter _send_command 9 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f63e81b8960 con 0x7f63e81afb90 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63e81afb90 0x7f63e81b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e81afb90 2024-04-02T05:00:46.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.640+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63e81afb90 0x7f63e81b3750 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.644+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e81b8960 con 0x7f63e81afb90 2024-04-02T05:00:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.644+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 9: 0 ) 2024-04-02T05:00:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.644+0000 7f63e57fa700 10 client.43703.objecter _finish_command 9 = system:0 2024-04-02T05:00:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.644+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 4 2024-04-02T05:00:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.644+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 4 9 2024-04-02T05:00:46.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 10 2024-04-02T05:00:46.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 10 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81abb50 osd=4 3 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 4 10 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 4 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 10 client.43703.objecter _send_command 10 2024-04-02T05:00:46.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.660+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e81b3c90 con 0x7f63e81afb90 2024-04-02T05:00:46.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e81b3c90 con 0x7f63e81afb90 2024-04-02T05:00:46.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e57fa700 10 client.43703.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 4 2024-04-02T05:00:46.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 4 10 2024-04-02T05:00:46.668 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 11 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63e81b43b0 0x7f63e81b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 11 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 5 11 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 4 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 10 client.43703.objecter _send_command 11 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f63e81bbb40 con 0x7f63e81b43b0 2024-04-02T05:00:46.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63e81b43b0 0x7f63e81b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e77fe700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63e81b43b0 0x7f63e81b4810 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.664+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e81b43b0 2024-04-02T05:00:46.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.668+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e81bbb40 con 0x7f63e81b43b0 2024-04-02T05:00:46.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.668+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 11: 0 ) 2024-04-02T05:00:46.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.668+0000 7f63e57fa700 10 client.43703.objecter _finish_command 11 = system:0 2024-04-02T05:00:46.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.668+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 4 2024-04-02T05:00:46.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.668+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 5 11 2024-04-02T05:00:46.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 12 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 12 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81b3c90 osd=5 3 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 5 12 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 4 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 10 client.43703.objecter _send_command 12 2024-04-02T05:00:46.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.684+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e81bbb40 con 0x7f63e81b43b0 2024-04-02T05:00:46.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e81bbb40 con 0x7f63e81b43b0 2024-04-02T05:00:46.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63e57fa700 10 client.43703.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 4 2024-04-02T05:00:46.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 5 12 2024-04-02T05:00:46.692 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 13 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63e81b9740 0x7f63e81b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 13 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 6 13 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 4 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 10 client.43703.objecter _send_command 13 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f63e81beee0 con 0x7f63e81b9740 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63e81b9740 0x7f63e81b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e81b9740 2024-04-02T05:00:46.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.688+0000 7f63eca92700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63e81b9740 0x7f63e81b9ba0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.692+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e81beee0 con 0x7f63e81b9740 2024-04-02T05:00:46.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.692+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 13: 0 ) 2024-04-02T05:00:46.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.692+0000 7f63e57fa700 10 client.43703.objecter _finish_command 13 = system:0 2024-04-02T05:00:46.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.692+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 4 2024-04-02T05:00:46.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.692+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 6 13 2024-04-02T05:00:46.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 14 2024-04-02T05:00:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 14 2024-04-02T05:00:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81bbb40 osd=6 3 2024-04-02T05:00:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 6 14 2024-04-02T05:00:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 4 2024-04-02T05:00:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 10 client.43703.objecter _send_command 14 2024-04-02T05:00:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.708+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e81ba1a0 con 0x7f63e81b9740 2024-04-02T05:00:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e81ba1a0 con 0x7f63e81b9740 2024-04-02T05:00:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e57fa700 10 client.43703.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 4 2024-04-02T05:00:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 6 14 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 15 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63e81ba8c0 0x7f63e81bad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 15 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 7 15 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 4 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 10 client.43703.objecter _send_command 15 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f63e81bb390 con 0x7f63e81ba8c0 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63e81ba8c0 0x7f63e81bad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e57fa700 10 client.43703.objecter ms_handle_connect 0x7f63e81ba8c0 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.712+0000 7f63e7fff700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63e81ba8c0 0x7f63e81bad00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.716+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e81bb390 con 0x7f63e81ba8c0 2024-04-02T05:00:46.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.716+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 15: 0 ) 2024-04-02T05:00:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.716+0000 7f63e57fa700 10 client.43703.objecter _finish_command 15 = system:0 2024-04-02T05:00:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.716+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 4 2024-04-02T05:00:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.716+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 7 15 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 10 client.43703.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign -1 16 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 20 client.43703.objecter _get_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_remove -1 16 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 20 client.43703.objecter get_session s=0x7f63e81ba1a0 osd=7 3 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 15 client.43703.objecter _session_command_op_assign 7 16 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 4 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 10 client.43703.objecter _send_command 16 2024-04-02T05:00:46.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.732+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e81bb390 con 0x7f63e81ba8c0 2024-04-02T05:00:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:46.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63e57fa700 1 -- 172.21.15.69:0/3285683465 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e81bb390 con 0x7f63e81ba8c0 2024-04-02T05:00:46.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63e57fa700 10 client.43703.objecter ms_dispatch 0x7f63e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:00:46.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63e57fa700 10 client.43703.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:00:46.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63e57fa700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 4 2024-04-02T05:00:46.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63e57fa700 15 client.43703.objecter _session_command_op_remove 7 16 2024-04-02T05:00:46.739 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:00:46.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 librados: watch_flush enter 2024-04-02T05:00:46.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 librados: watch_flush exit 2024-04-02T05:00:46.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.0 2024-04-02T05:00:46.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63e8062e50 msgr2=0x7f63e8063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f63e8062e50 0x7f63e8063230 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e8199430 osd=0 1 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.1 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f63e8109d50 msgr2=0x7f63e81ab2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f63e8109d50 0x7f63e81ab2a0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802db30 osd=1 1 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.2 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f63e8109430 msgr2=0x7f63e81ae970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f63e8109430 0x7f63e81ae970 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81092c0 osd=2 1 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.3 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63e81ac4c0 msgr2=0x7f63e81b2040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f63e81ac4c0 0x7f63e81b2040 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e802d000 osd=3 1 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.4 2024-04-02T05:00:46.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63e81afb90 msgr2=0x7f63e81b3750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f63e81afb90 0x7f63e81b3750 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81abb50 osd=4 1 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.5 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63e81b43b0 msgr2=0x7f63e81b4810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f63e81b43b0 0x7f63e81b4810 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81b3c90 osd=5 1 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.6 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63e81b9740 msgr2=0x7f63e81b9ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f63e81b9740 0x7f63e81b9ba0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81bbb40 osd=6 1 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter close_session for osd.7 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63e81ba8c0 msgr2=0x7f63e81bad00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f63e81ba8c0 0x7f63e81bad00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter put_session s=0x7f63e81ba1a0 osd=7 1 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 20 client.43703.objecter shutdown clearing up homeless session... 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 10 client.43703.objecter successfully canceled tick 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63eca92700 1 -- 172.21.15.69:0/3285683465 reap_dead start 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63eca92700 1 -- 172.21.15.69:0/3285683465 reap_dead start 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63eca92700 1 -- 172.21.15.69:0/3285683465 reap_dead start 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f63d006f5e0 msgr2=0x7f63d0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f63d006f5e0 0x7f63d0071aa0 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009990 tx=0x7f63dc005fd0 comp rx=0 tx=0).stop 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806a1d0 msgr2=0x7f63e81980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63eca92700 1 -- 172.21.15.69:0/3285683465 reap_dead start 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 --2- 172.21.15.69:0/3285683465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63e806a1d0 0x7f63e81980d0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f63d800b530 tx=0x7f63d800b840 comp rx=0 tx=0).stop 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63eca92700 1 -- 172.21.15.69:0/3285683465 reap_dead start 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 shutdown_connections 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 >> 172.21.15.69:0/3285683465 conn(0x7f63e80771e0 msgr2=0x7f63e80ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 shutdown_connections 2024-04-02T05:00:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 -- 172.21.15.69:0/3285683465 wait complete. 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:46.736+0000 7f63ee4f5700 1 librados: shutdown 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:00:46.744 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:00:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:46.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:46.870 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:46.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:46.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:47.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:47.138 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:47 smithi098 bash[23790]: cluster 2024-04-02T05:00:46.054948+0000 mon.a (mon.0) 3561 : cluster 0 osdmap e736: 8 total, 8 up, 7 in 2024-04-02T05:00:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:47 smithi098 bash[23790]: audit 2024-04-02T05:00:46.062005+0000 mon.a (mon.0) 3562 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:47 smithi098 bash[23790]: cluster 2024-04-02T05:00:46.296655+0000 mgr.y (mgr.14538) 665 : cluster 0 pgmap v1116: 224 pgs: 1 peering, 1 active+clean+snaptrim, 37 unknown, 185 active+clean; 582 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:47 smithi098 bash[23790]: audit 2024-04-02T05:00:46.557626+0000 mon.c (mon.2) 1153 : audit 0 from='client.? 172.21.15.69:0/3285683465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:47 smithi098 bash[23790]: cluster 2024-04-02T05:00:46.843246+0000 osd.0 (osd.0) 361 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:47.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:47 smithi098 bash[23790]: cluster 2024-04-02T05:00:46.843432+0000 osd.0 (osd.0) 362 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:47.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:47.267 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:47.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:47.393 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:47.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:47.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[28904]: cluster 2024-04-02T05:00:46.054948+0000 mon.a (mon.0) 3561 : cluster 0 osdmap e736: 8 total, 8 up, 7 in 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[28904]: audit 2024-04-02T05:00:46.062005+0000 mon.a (mon.0) 3562 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[28904]: cluster 2024-04-02T05:00:46.296655+0000 mgr.y (mgr.14538) 665 : cluster 0 pgmap v1116: 224 pgs: 1 peering, 1 active+clean+snaptrim, 37 unknown, 185 active+clean; 582 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[28904]: audit 2024-04-02T05:00:46.557626+0000 mon.c (mon.2) 1153 : audit 0 from='client.? 172.21.15.69:0/3285683465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[28904]: cluster 2024-04-02T05:00:46.843246+0000 osd.0 (osd.0) 361 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[28904]: cluster 2024-04-02T05:00:46.843432+0000 osd.0 (osd.0) 362 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[21528]: cluster 2024-04-02T05:00:46.054948+0000 mon.a (mon.0) 3561 : cluster 0 osdmap e736: 8 total, 8 up, 7 in 2024-04-02T05:00:47.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[21528]: audit 2024-04-02T05:00:46.062005+0000 mon.a (mon.0) 3562 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:47.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[21528]: cluster 2024-04-02T05:00:46.296655+0000 mgr.y (mgr.14538) 665 : cluster 0 pgmap v1116: 224 pgs: 1 peering, 1 active+clean+snaptrim, 37 unknown, 185 active+clean; 582 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:47.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[21528]: audit 2024-04-02T05:00:46.557626+0000 mon.c (mon.2) 1153 : audit 0 from='client.? 172.21.15.69:0/3285683465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:00:47.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[21528]: cluster 2024-04-02T05:00:46.843246+0000 osd.0 (osd.0) 361 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:47.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:47 smithi069 bash[21528]: cluster 2024-04-02T05:00:46.843432+0000 osd.0 (osd.0) 362 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:47.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:47.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:47.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:00:47.765 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:00:47.766 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T05:00:47.766 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T05:00:47.766 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:00:47.773 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:47.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:47.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:47.899 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:47.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:48.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:48.028 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:48.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:48.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:48.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:48 smithi098 bash[23790]: audit 2024-04-02T05:00:47.060751+0000 mon.a (mon.0) 3563 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:48.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:48 smithi098 bash[23790]: cluster 2024-04-02T05:00:47.070908+0000 mon.a (mon.0) 3564 : cluster 0 osdmap e737: 8 total, 8 up, 7 in 2024-04-02T05:00:48.272 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:48 smithi098 bash[23790]: audit 2024-04-02T05:00:47.131734+0000 mon.a (mon.0) 3565 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:48.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[21528]: audit 2024-04-02T05:00:47.060751+0000 mon.a (mon.0) 3563 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:48.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[21528]: cluster 2024-04-02T05:00:47.070908+0000 mon.a (mon.0) 3564 : cluster 0 osdmap e737: 8 total, 8 up, 7 in 2024-04-02T05:00:48.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[21528]: audit 2024-04-02T05:00:47.131734+0000 mon.a (mon.0) 3565 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:48.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[28904]: audit 2024-04-02T05:00:47.060751+0000 mon.a (mon.0) 3563 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:48.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[28904]: cluster 2024-04-02T05:00:47.070908+0000 mon.a (mon.0) 3564 : cluster 0 osdmap e737: 8 total, 8 up, 7 in 2024-04-02T05:00:48.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[28904]: audit 2024-04-02T05:00:47.131734+0000 mon.a (mon.0) 3565 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:48.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:48.408 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:48.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:48.538 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:48.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-02T05:00:48.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:48.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:48.827 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:48.930 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:48.952 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:48.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:49.031 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:00:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:00:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:00:49.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:49.083 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:49.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:49.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:49.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:49.326 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: cluster 2024-04-02T05:00:47.707309+0000 osd.7 (osd.7) 173 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:49.326 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: cluster 2024-04-02T05:00:47.707716+0000 osd.7 (osd.7) 174 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:49.326 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: audit 2024-04-02T05:00:48.062734+0000 mon.a (mon.0) 3566 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:49.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: cluster 2024-04-02T05:00:48.068294+0000 mon.a (mon.0) 3567 : cluster 0 osdmap e738: 8 total, 8 up, 7 in 2024-04-02T05:00:49.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: audit 2024-04-02T05:00:48.069218+0000 mon.a (mon.0) 3568 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-120"}]: dispatch 2024-04-02T05:00:49.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: cluster 2024-04-02T05:00:48.298288+0000 mgr.y (mgr.14538) 666 : cluster 0 pgmap v1119: 224 pgs: 13 unknown, 211 active+clean; 609 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 4.7 KiB/s rd, 28 KiB/s wr, 5 op/s; 100 KiB/s, 0 objects/s recovering 2024-04-02T05:00:49.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: audit 2024-04-02T05:00:48.891317+0000 mon.a (mon.0) 3569 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:49.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: audit 2024-04-02T05:00:48.891593+0000 mon.b (mon.1) 303 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:49.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:49 smithi098 bash[23790]: audit 2024-04-02T05:00:49.065586+0000 mon.a (mon.0) 3570 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-120"}]': finished 2024-04-02T05:00:49.342 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:49.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:49.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: cluster 2024-04-02T05:00:47.707309+0000 osd.7 (osd.7) 173 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:49.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: cluster 2024-04-02T05:00:47.707716+0000 osd.7 (osd.7) 174 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:49.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: audit 2024-04-02T05:00:48.062734+0000 mon.a (mon.0) 3566 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:49.459 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: cluster 2024-04-02T05:00:48.068294+0000 mon.a (mon.0) 3567 : cluster 0 osdmap e738: 8 total, 8 up, 7 in 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: audit 2024-04-02T05:00:48.069218+0000 mon.a (mon.0) 3568 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-120"}]: dispatch 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: cluster 2024-04-02T05:00:48.298288+0000 mgr.y (mgr.14538) 666 : cluster 0 pgmap v1119: 224 pgs: 13 unknown, 211 active+clean; 609 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 4.7 KiB/s rd, 28 KiB/s wr, 5 op/s; 100 KiB/s, 0 objects/s recovering 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: audit 2024-04-02T05:00:48.891317+0000 mon.a (mon.0) 3569 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: audit 2024-04-02T05:00:48.891593+0000 mon.b (mon.1) 303 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[28904]: audit 2024-04-02T05:00:49.065586+0000 mon.a (mon.0) 3570 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-120"}]': finished 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: cluster 2024-04-02T05:00:47.707309+0000 osd.7 (osd.7) 173 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: cluster 2024-04-02T05:00:47.707716+0000 osd.7 (osd.7) 174 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:49.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: audit 2024-04-02T05:00:48.062734+0000 mon.a (mon.0) 3566 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:49.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: cluster 2024-04-02T05:00:48.068294+0000 mon.a (mon.0) 3567 : cluster 0 osdmap e738: 8 total, 8 up, 7 in 2024-04-02T05:00:49.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: audit 2024-04-02T05:00:48.069218+0000 mon.a (mon.0) 3568 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-120"}]: dispatch 2024-04-02T05:00:49.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: cluster 2024-04-02T05:00:48.298288+0000 mgr.y (mgr.14538) 666 : cluster 0 pgmap v1119: 224 pgs: 13 unknown, 211 active+clean; 609 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 4.7 KiB/s rd, 28 KiB/s wr, 5 op/s; 100 KiB/s, 0 objects/s recovering 2024-04-02T05:00:49.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: audit 2024-04-02T05:00:48.891317+0000 mon.a (mon.0) 3569 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:00:49.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: audit 2024-04-02T05:00:48.891593+0000 mon.b (mon.1) 303 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:00:49.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:49 smithi069 bash[21528]: audit 2024-04-02T05:00:49.065586+0000 mon.a (mon.0) 3570 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-120"}]': finished 2024-04-02T05:00:49.473 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:49.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:49.605 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:49.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:49.735 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:49.838 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:49.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/2176649087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 msgr2=0x7fed2c111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- 172.21.15.69:0/2176649087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c111140 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7fed2400b0a0 tx=0x7fed2400b3b0 comp rx=0 tx=0).stop 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/2176649087 shutdown_connections 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- 172.21.15.69:0/2176649087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c111140 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- 172.21.15.69:0/2176649087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c069f70 0x7fed2c06a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- 172.21.15.69:0/2176649087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed2c06a890 0x7fed2c10e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/2176649087 >> 172.21.15.69:0/2176649087 conn(0x7fed2c0771e0 msgr2=0x7fed2c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/2176649087 shutdown_connections 2024-04-02T05:00:49.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/2176649087 wait complete. 2024-04-02T05:00:49.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 librados: starting msgr at 2024-04-02T05:00:49.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 librados: starting objecter 2024-04-02T05:00:49.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 Processor -- start 2024-04-02T05:00:49.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- start start 2024-04-02T05:00:49.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 librados: setting wanted keys 2024-04-02T05:00:49.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 librados: calling monclient init 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed2c069f70 0x7fed2c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c06a890 0x7fed2c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c113c30 con 0x7fed2c10ed50 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c113ab0 con 0x7fed2c069f70 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fed2c113db0 con 0x7fed2c06a890 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed31009700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c06a890 0x7fed2c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed31009700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c06a890 0x7fed2c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:35476/0 (socket says 172.21.15.69:35476) 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed31009700 1 -- 172.21.15.69:0/808530527 learned_addr learned my addr 172.21.15.69:0/808530527 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3180a700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed2c069f70 0x7fed2c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:49.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 -- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c06a890 msgr2=0x7fed2c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c06a890 0x7fed2c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 -- 172.21.15.69:0/808530527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed2c069f70 msgr2=0x7fed2c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed2c069f70 0x7fed2c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 -- 172.21.15.69:0/808530527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c19f250 con 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed3200b700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c19eb10 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fed2400bfd0 tx=0x7fed24007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed22ffd700 10 client.?.objecter ms_handle_connect 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed22ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed24021020 con 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/808530527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2c19f4e0 con 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fed24004c50 con 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed24005050 con 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 10 client.33645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 -- 172.21.15.69:0/808530527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2c1a6dc0 con 0x7fed2c10ed50 2024-04-02T05:00:49.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed33a6e700 1 librados: init done 2024-04-02T05:00:49.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.840+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2c04ff00 con 0x7fed2c10ed50 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fed24024020 con 0x7fed2c10ed50 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed1806f5e0 0x7fed18071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(739..739 src has 74..739) ==== 10079+0+0 (secure 0 0 0) 0x7fed2400f020 con 0x7fed2c10ed50 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 10 client.33645.objecter ms_dispatch 0x7fed2c001550 osd_map(739..739 src has 74..739) 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 3 client.33645.objecter handle_osd_map got epochs [739,739] > 0 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 3 client.33645.objecter handle_osd_map decoding full epoch 739 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed3180a700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed1806f5e0 0x7fed18071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 20 client.33645.objecter dump_active .. 0 homeless 2024-04-02T05:00:49.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed3180a700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed1806f5e0 0x7fed18071aa0 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7fed28009bf0 tx=0x7fed28009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:49.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.844+0000 7fed22ffd700 10 client.33645.objecter ms_handle_connect 0x7fed1806f5e0 2024-04-02T05:00:49.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:49.848+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed2402b020 con 0x7fed2c10ed50 2024-04-02T05:00:49.860 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:49.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:49.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:50.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:50.098 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[21528]: cluster 2024-04-02T05:00:49.073559+0000 mon.a (mon.0) 3571 : cluster 0 osdmap e739: 8 total, 8 up, 7 in 2024-04-02T05:00:50.098 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[21528]: audit 2024-04-02T05:00:49.078081+0000 mon.a (mon.0) 3572 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-120", "mode": "writeback"}]: dispatch 2024-04-02T05:00:50.098 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[21528]: cluster 2024-04-02T05:00:50.046289+0000 mon.a (mon.0) 3573 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:50.098 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[21528]: audit 2024-04-02T05:00:50.048410+0000 mon.a (mon.0) 3574 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-120", "mode": "writeback"}]': finished 2024-04-02T05:00:50.098 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[21528]: cluster 2024-04-02T05:00:50.052204+0000 mon.a (mon.0) 3575 : cluster 0 osdmap e740: 8 total, 8 up, 7 in 2024-04-02T05:00:50.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[28904]: cluster 2024-04-02T05:00:49.073559+0000 mon.a (mon.0) 3571 : cluster 0 osdmap e739: 8 total, 8 up, 7 in 2024-04-02T05:00:50.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[28904]: audit 2024-04-02T05:00:49.078081+0000 mon.a (mon.0) 3572 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-120", "mode": "writeback"}]: dispatch 2024-04-02T05:00:50.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[28904]: cluster 2024-04-02T05:00:50.046289+0000 mon.a (mon.0) 3573 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:50.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[28904]: audit 2024-04-02T05:00:50.048410+0000 mon.a (mon.0) 3574 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-120", "mode": "writeback"}]': finished 2024-04-02T05:00:50.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:50 smithi069 bash[28904]: cluster 2024-04-02T05:00:50.052204+0000 mon.a (mon.0) 3575 : cluster 0 osdmap e740: 8 total, 8 up, 7 in 2024-04-02T05:00:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:50.096+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fed2c19b5a0 con 0x7fed2c10ed50 2024-04-02T05:00:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:50.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:50.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:50.366 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:50.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T05:00:50.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:50.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:50 smithi098 bash[23790]: cluster 2024-04-02T05:00:49.073559+0000 mon.a (mon.0) 3571 : cluster 0 osdmap e739: 8 total, 8 up, 7 in 2024-04-02T05:00:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:50 smithi098 bash[23790]: audit 2024-04-02T05:00:49.078081+0000 mon.a (mon.0) 3572 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-120", "mode": "writeback"}]: dispatch 2024-04-02T05:00:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:50 smithi098 bash[23790]: cluster 2024-04-02T05:00:50.046289+0000 mon.a (mon.0) 3573 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:50 smithi098 bash[23790]: audit 2024-04-02T05:00:50.048410+0000 mon.a (mon.0) 3574 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-120", "mode": "writeback"}]': finished 2024-04-02T05:00:50.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:50 smithi098 bash[23790]: cluster 2024-04-02T05:00:50.052204+0000 mon.a (mon.0) 3575 : cluster 0 osdmap e740: 8 total, 8 up, 7 in 2024-04-02T05:00:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:50.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:50.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:50.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:50.887 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:50.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:50.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.096+0000 7fed22ffd700 1 -- 172.21.15.69:0/808530527 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v741) ==== 91+0+0 (secure 0 0 0) 0x7fed2405d0f0 con 0x7fed2c10ed50 2024-04-02T05:00:51.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 10 librados: watch_flush enter 2024-04-02T05:00:51.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 10 librados: watch_flush exit 2024-04-02T05:00:51.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 20 client.33645.objecter shutdown clearing up homeless session... 2024-04-02T05:00:51.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 10 client.33645.objecter successfully canceled tick 2024-04-02T05:00:51.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed1806f5e0 msgr2=0x7fed18071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:51.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed1806f5e0 0x7fed18071aa0 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7fed28009bf0 tx=0x7fed28009260 comp rx=0 tx=0).stop 2024-04-02T05:00:51.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 msgr2=0x7fed2c19eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:51.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c19eb10 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fed2400bfd0 tx=0x7fed24007a20 comp rx=0 tx=0).stop 2024-04-02T05:00:51.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 shutdown_connections 2024-04-02T05:00:51.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fed1806f5e0 0x7fed18071aa0 secure :-1 s=CLOSED pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7fed28009bf0 tx=0x7fed28009260 comp rx=0 tx=0).stop 2024-04-02T05:00:51.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c10ed50 0x7fed2c19eb10 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:51.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed2c069f70 0x7fed2c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:51.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 --2- 172.21.15.69:0/808530527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed2c06a890 0x7fed2c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:51.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.100+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 >> 172.21.15.69:0/808530527 conn(0x7fed2c0771e0 msgr2=0x7fed2c100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:51.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.104+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 shutdown_connections 2024-04-02T05:00:51.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.104+0000 7fed20ff9700 1 -- 172.21.15.69:0/808530527 wait complete. 2024-04-02T05:00:51.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:51.104+0000 7fed20ff9700 1 librados: shutdown 2024-04-02T05:00:51.112 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T05:00:51.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:51.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:51 smithi069 bash[21528]: audit 2024-04-02T05:00:50.102512+0000 mon.a (mon.0) 3576 : audit 1 from='client.? 172.21.15.69:0/808530527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:00:51.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:51 smithi069 bash[21528]: audit 2024-04-02T05:00:50.164151+0000 mon.a (mon.0) 3577 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:51.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:51 smithi069 bash[21528]: cluster 2024-04-02T05:00:50.299093+0000 mgr.y (mgr.14538) 667 : cluster 0 pgmap v1122: 224 pgs: 224 active+clean; 661 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 12 KiB/s rd, 83 KiB/s wr, 14 op/s; 139 KiB/s, 0 objects/s recovering 2024-04-02T05:00:51.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:51 smithi069 bash[28904]: audit 2024-04-02T05:00:50.102512+0000 mon.a (mon.0) 3576 : audit 1 from='client.? 172.21.15.69:0/808530527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:00:51.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:51 smithi069 bash[28904]: audit 2024-04-02T05:00:50.164151+0000 mon.a (mon.0) 3577 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:51.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:51 smithi069 bash[28904]: cluster 2024-04-02T05:00:50.299093+0000 mgr.y (mgr.14538) 667 : cluster 0 pgmap v1122: 224 pgs: 224 active+clean; 661 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 12 KiB/s rd, 83 KiB/s wr, 14 op/s; 139 KiB/s, 0 objects/s recovering 2024-04-02T05:00:51.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:51.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:51.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:51 smithi098 bash[23790]: audit 2024-04-02T05:00:50.102512+0000 mon.a (mon.0) 3576 : audit 1 from='client.? 172.21.15.69:0/808530527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:00:51.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:51 smithi098 bash[23790]: audit 2024-04-02T05:00:50.164151+0000 mon.a (mon.0) 3577 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:51.254 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:51 smithi098 bash[23790]: cluster 2024-04-02T05:00:50.299093+0000 mgr.y (mgr.14538) 667 : cluster 0 pgmap v1122: 224 pgs: 224 active+clean; 661 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 12 KiB/s rd, 83 KiB/s wr, 14 op/s; 139 KiB/s, 0 objects/s recovering 2024-04-02T05:00:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:51.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:51.399 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:51.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:51.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:51.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:51.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:51.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:51.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:51.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:52.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:52.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:52.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:52 smithi098 bash[23790]: cluster 2024-04-02T05:00:51.090027+0000 mon.a (mon.0) 3578 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:52 smithi098 bash[23790]: audit 2024-04-02T05:00:51.100141+0000 mon.a (mon.0) 3579 : audit 1 from='client.? 172.21.15.69:0/808530527' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:00:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:52 smithi098 bash[23790]: audit 2024-04-02T05:00:51.100326+0000 mon.a (mon.0) 3580 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:52 smithi098 bash[23790]: cluster 2024-04-02T05:00:51.108441+0000 mon.a (mon.0) 3581 : cluster 0 osdmap e741: 8 total, 8 up, 7 in 2024-04-02T05:00:52.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:52 smithi098 bash[23790]: audit 2024-04-02T05:00:51.110394+0000 mon.a (mon.0) 3582 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120"}]: dispatch 2024-04-02T05:00:52.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:52.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:52.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[28904]: cluster 2024-04-02T05:00:51.090027+0000 mon.a (mon.0) 3578 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:52.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[28904]: audit 2024-04-02T05:00:51.100141+0000 mon.a (mon.0) 3579 : audit 1 from='client.? 172.21.15.69:0/808530527' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:00:52.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[28904]: audit 2024-04-02T05:00:51.100326+0000 mon.a (mon.0) 3580 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:52.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[28904]: cluster 2024-04-02T05:00:51.108441+0000 mon.a (mon.0) 3581 : cluster 0 osdmap e741: 8 total, 8 up, 7 in 2024-04-02T05:00:52.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[28904]: audit 2024-04-02T05:00:51.110394+0000 mon.a (mon.0) 3582 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120"}]: dispatch 2024-04-02T05:00:52.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[21528]: cluster 2024-04-02T05:00:51.090027+0000 mon.a (mon.0) 3578 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:00:52.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[21528]: audit 2024-04-02T05:00:51.100141+0000 mon.a (mon.0) 3579 : audit 1 from='client.? 172.21.15.69:0/808530527' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:00:52.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[21528]: audit 2024-04-02T05:00:51.100326+0000 mon.a (mon.0) 3580 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:00:52.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[21528]: cluster 2024-04-02T05:00:51.108441+0000 mon.a (mon.0) 3581 : cluster 0 osdmap e741: 8 total, 8 up, 7 in 2024-04-02T05:00:52.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:52 smithi069 bash[21528]: audit 2024-04-02T05:00:51.110394+0000 mon.a (mon.0) 3582 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120"}]: dispatch 2024-04-02T05:00:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:52.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:52.572 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:52.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:52.700 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:52.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:52.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:52.963 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:52.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:53.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:53.094 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:53.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: cluster 2024-04-02T05:00:51.972442+0000 osd.6 (osd.6) 307 : cluster 0 2.1a scrub starts 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: cluster 2024-04-02T05:00:51.973715+0000 osd.6 (osd.6) 308 : cluster 0 2.1a scrub ok 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.100566+0000 mon.a (mon.0) 3583 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: audit 2024-04-02T05:00:52.102569+0000 mon.a (mon.0) 3584 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120"}]': finished 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.110639+0000 mon.a (mon.0) 3585 : cluster 0 osdmap e742: 8 total, 8 up, 7 in 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.300032+0000 mgr.y (mgr.14538) 668 : cluster 0 pgmap v1125: 224 pgs: 224 active+clean; 661 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 7.0 KiB/s rd, 55 KiB/s wr, 8 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-02T05:00:53.207 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.679859+0000 osd.3 (osd.3) 367 : cluster 0 1.c scrub starts 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: cluster 2024-04-02T05:00:51.972442+0000 osd.6 (osd.6) 307 : cluster 0 2.1a scrub starts 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: cluster 2024-04-02T05:00:51.973715+0000 osd.6 (osd.6) 308 : cluster 0 2.1a scrub ok 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.100566+0000 mon.a (mon.0) 3583 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: audit 2024-04-02T05:00:52.102569+0000 mon.a (mon.0) 3584 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120"}]': finished 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.110639+0000 mon.a (mon.0) 3585 : cluster 0 osdmap e742: 8 total, 8 up, 7 in 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.300032+0000 mgr.y (mgr.14538) 668 : cluster 0 pgmap v1125: 224 pgs: 224 active+clean; 661 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 7.0 KiB/s rd, 55 KiB/s wr, 8 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-02T05:00:53.208 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:53 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.679859+0000 osd.3 (osd.3) 367 : cluster 0 1.c scrub starts 2024-04-02T05:00:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:53.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:53.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: cluster 2024-04-02T05:00:51.972442+0000 osd.6 (osd.6) 307 : cluster 0 2.1a scrub starts 2024-04-02T05:00:53.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: cluster 2024-04-02T05:00:51.973715+0000 osd.6 (osd.6) 308 : cluster 0 2.1a scrub ok 2024-04-02T05:00:53.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.100566+0000 mon.a (mon.0) 3583 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:00:53.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: audit 2024-04-02T05:00:52.102569+0000 mon.a (mon.0) 3584 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-120"}]': finished 2024-04-02T05:00:53.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.110639+0000 mon.a (mon.0) 3585 : cluster 0 osdmap e742: 8 total, 8 up, 7 in 2024-04-02T05:00:53.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.300032+0000 mgr.y (mgr.14538) 668 : cluster 0 pgmap v1125: 224 pgs: 224 active+clean; 661 KiB data, 692 MiB used, 625 GiB / 626 GiB avail; 7.0 KiB/s rd, 55 KiB/s wr, 8 op/s; 39 KiB/s, 0 objects/s recovering 2024-04-02T05:00:53.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.679859+0000 osd.3 (osd.3) 367 : cluster 0 1.c scrub starts 2024-04-02T05:00:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:53.423 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:00:53.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:53.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:53.602 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:53.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:53.730 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:53.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:53.844 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:00:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:00:53.859 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:53.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:53.988 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:53.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:54.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:54.122 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:54.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T05:00:54.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:54.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.681924+0000 osd.3 (osd.3) 368 : cluster 0 1.c scrub ok 2024-04-02T05:00:54.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.941254+0000 osd.6 (osd.6) 309 : cluster 0 3.10 scrub starts 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:52.943803+0000 osd.6 (osd.6) 310 : cluster 0 3.10 scrub ok 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.102544+0000 mon.a (mon.0) 3586 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.118724+0000 mon.a (mon.0) 3587 : cluster 0 osdmap e743: 8 total, 8 up, 7 in 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.223220+0000 osd.1 (osd.1) 371 : cluster 0 1.12 scrub starts 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.225003+0000 osd.1 (osd.1) 372 : cluster 0 1.12 scrub ok 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: audit 2024-04-02T05:00:53.623191+0000 mgr.y (mgr.14538) 669 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.855043+0000 osd.0 (osd.0) 363 : cluster 0 1.11 scrub starts 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.857424+0000 osd.0 (osd.0) 364 : cluster 0 1.11 scrub ok 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.681924+0000 osd.3 (osd.3) 368 : cluster 0 1.c scrub ok 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.941254+0000 osd.6 (osd.6) 309 : cluster 0 3.10 scrub starts 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:52.943803+0000 osd.6 (osd.6) 310 : cluster 0 3.10 scrub ok 2024-04-02T05:00:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.102544+0000 mon.a (mon.0) 3586 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.118724+0000 mon.a (mon.0) 3587 : cluster 0 osdmap e743: 8 total, 8 up, 7 in 2024-04-02T05:00:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.223220+0000 osd.1 (osd.1) 371 : cluster 0 1.12 scrub starts 2024-04-02T05:00:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.225003+0000 osd.1 (osd.1) 372 : cluster 0 1.12 scrub ok 2024-04-02T05:00:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: audit 2024-04-02T05:00:53.623191+0000 mgr.y (mgr.14538) 669 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.855043+0000 osd.0 (osd.0) 363 : cluster 0 1.11 scrub starts 2024-04-02T05:00:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:54 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.857424+0000 osd.0 (osd.0) 364 : cluster 0 1.11 scrub ok 2024-04-02T05:00:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:54.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:54.382 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.681924+0000 osd.3 (osd.3) 368 : cluster 0 1.c scrub ok 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.941254+0000 osd.6 (osd.6) 309 : cluster 0 3.10 scrub starts 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:52.943803+0000 osd.6 (osd.6) 310 : cluster 0 3.10 scrub ok 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.102544+0000 mon.a (mon.0) 3586 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.118724+0000 mon.a (mon.0) 3587 : cluster 0 osdmap e743: 8 total, 8 up, 7 in 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.223220+0000 osd.1 (osd.1) 371 : cluster 0 1.12 scrub starts 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.225003+0000 osd.1 (osd.1) 372 : cluster 0 1.12 scrub ok 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: audit 2024-04-02T05:00:53.623191+0000 mgr.y (mgr.14538) 669 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.855043+0000 osd.0 (osd.0) 363 : cluster 0 1.11 scrub starts 2024-04-02T05:00:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:54 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.857424+0000 osd.0 (osd.0) 364 : cluster 0 1.11 scrub ok 2024-04-02T05:00:54.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:54.511 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:54.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:54.643 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:54.746 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:54.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:54.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:54.923 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:54.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:55.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:55.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:55.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:55.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.436389+0000 osd.7 (osd.7) 175 : cluster 0 1.10 scrub starts 2024-04-02T05:00:55.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.437605+0000 osd.7 (osd.7) 176 : cluster 0 1.10 scrub ok 2024-04-02T05:00:55.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.695114+0000 osd.3 (osd.3) 369 : cluster 0 1.4 scrub starts 2024-04-02T05:00:55.185 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.698087+0000 osd.3 (osd.3) 370 : cluster 0 1.4 scrub ok 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.959609+0000 osd.2 (osd.2) 333 : cluster 0 1.14 scrub starts 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.962274+0000 osd.2 (osd.2) 334 : cluster 0 1.14 scrub ok 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.970355+0000 osd.6 (osd.6) 311 : cluster 0 1.f scrub starts 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:53.972960+0000 osd.6 (osd.6) 312 : cluster 0 1.f scrub ok 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.119090+0000 mon.a (mon.0) 3588 : cluster 0 osdmap e744: 8 total, 8 up, 7 in 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: audit 2024-04-02T05:00:54.121394+0000 mon.a (mon.0) 3589 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.243155+0000 osd.1 (osd.1) 373 : cluster 0 1.3 scrub starts 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.244306+0000 osd.1 (osd.1) 374 : cluster 0 1.3 scrub ok 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.300904+0000 mgr.y (mgr.14538) 670 : cluster 0 pgmap v1128: 224 pgs: 32 unknown, 192 active+clean; 874 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.810000+0000 osd.0 (osd.0) 365 : cluster 0 6.10 scrub starts 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.811105+0000 osd.0 (osd.0) 366 : cluster 0 6.10 scrub ok 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: audit 2024-04-02T05:00:55.052783+0000 mon.a (mon.0) 3590 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:55.186 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.061405+0000 mon.a (mon.0) 3591 : cluster 0 osdmap e745: 8 total, 8 up, 7 in 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[28904]: audit 2024-04-02T05:00:55.108599+0000 mon.a (mon.0) 3592 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.436389+0000 osd.7 (osd.7) 175 : cluster 0 1.10 scrub starts 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.437605+0000 osd.7 (osd.7) 176 : cluster 0 1.10 scrub ok 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.695114+0000 osd.3 (osd.3) 369 : cluster 0 1.4 scrub starts 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.698087+0000 osd.3 (osd.3) 370 : cluster 0 1.4 scrub ok 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.959609+0000 osd.2 (osd.2) 333 : cluster 0 1.14 scrub starts 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.962274+0000 osd.2 (osd.2) 334 : cluster 0 1.14 scrub ok 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.970355+0000 osd.6 (osd.6) 311 : cluster 0 1.f scrub starts 2024-04-02T05:00:55.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:53.972960+0000 osd.6 (osd.6) 312 : cluster 0 1.f scrub ok 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.119090+0000 mon.a (mon.0) 3588 : cluster 0 osdmap e744: 8 total, 8 up, 7 in 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: audit 2024-04-02T05:00:54.121394+0000 mon.a (mon.0) 3589 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.243155+0000 osd.1 (osd.1) 373 : cluster 0 1.3 scrub starts 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.244306+0000 osd.1 (osd.1) 374 : cluster 0 1.3 scrub ok 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.300904+0000 mgr.y (mgr.14538) 670 : cluster 0 pgmap v1128: 224 pgs: 32 unknown, 192 active+clean; 874 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.810000+0000 osd.0 (osd.0) 365 : cluster 0 6.10 scrub starts 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.811105+0000 osd.0 (osd.0) 366 : cluster 0 6.10 scrub ok 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: audit 2024-04-02T05:00:55.052783+0000 mon.a (mon.0) 3590 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.061405+0000 mon.a (mon.0) 3591 : cluster 0 osdmap e745: 8 total, 8 up, 7 in 2024-04-02T05:00:55.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:55 smithi069 bash[21528]: audit 2024-04-02T05:00:55.108599+0000 mon.a (mon.0) 3592 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:55.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:55.297 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.436389+0000 osd.7 (osd.7) 175 : cluster 0 1.10 scrub starts 2024-04-02T05:00:55.297 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.437605+0000 osd.7 (osd.7) 176 : cluster 0 1.10 scrub ok 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.695114+0000 osd.3 (osd.3) 369 : cluster 0 1.4 scrub starts 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.698087+0000 osd.3 (osd.3) 370 : cluster 0 1.4 scrub ok 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.959609+0000 osd.2 (osd.2) 333 : cluster 0 1.14 scrub starts 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.962274+0000 osd.2 (osd.2) 334 : cluster 0 1.14 scrub ok 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.970355+0000 osd.6 (osd.6) 311 : cluster 0 1.f scrub starts 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:53.972960+0000 osd.6 (osd.6) 312 : cluster 0 1.f scrub ok 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.119090+0000 mon.a (mon.0) 3588 : cluster 0 osdmap e744: 8 total, 8 up, 7 in 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: audit 2024-04-02T05:00:54.121394+0000 mon.a (mon.0) 3589 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.243155+0000 osd.1 (osd.1) 373 : cluster 0 1.3 scrub starts 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.244306+0000 osd.1 (osd.1) 374 : cluster 0 1.3 scrub ok 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.300904+0000 mgr.y (mgr.14538) 670 : cluster 0 pgmap v1128: 224 pgs: 32 unknown, 192 active+clean; 874 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.810000+0000 osd.0 (osd.0) 365 : cluster 0 6.10 scrub starts 2024-04-02T05:00:55.298 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.811105+0000 osd.0 (osd.0) 366 : cluster 0 6.10 scrub ok 2024-04-02T05:00:55.299 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: audit 2024-04-02T05:00:55.052783+0000 mon.a (mon.0) 3590 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:00:55.299 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.061405+0000 mon.a (mon.0) 3591 : cluster 0 osdmap e745: 8 total, 8 up, 7 in 2024-04-02T05:00:55.299 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:55 smithi098 bash[23790]: audit 2024-04-02T05:00:55.108599+0000 mon.a (mon.0) 3592 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:00:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:55.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:55.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:55.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:55.567 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:55.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:55.746 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:55.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:55.817 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:55.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:55.948 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:55.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:56.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:56.081 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:56.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.395892+0000 osd.7 (osd.7) 177 : cluster 0 1.d scrub starts 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.397923+0000 osd.7 (osd.7) 178 : cluster 0 1.d scrub ok 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.689624+0000 osd.3 (osd.3) 371 : cluster 0 3.18 scrub starts 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.692450+0000 osd.3 (osd.3) 372 : cluster 0 3.18 scrub ok 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.982992+0000 osd.2 (osd.2) 335 : cluster 0 1.5 scrub starts 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:54.984443+0000 osd.2 (osd.2) 336 : cluster 0 1.5 scrub ok 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.005795+0000 osd.6 (osd.6) 313 : cluster 0 1.6 scrub starts 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.008113+0000 osd.6 (osd.6) 314 : cluster 0 1.6 scrub ok 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.205854+0000 osd.1 (osd.1) 375 : cluster 0 1.b scrub starts 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.206874+0000 osd.1 (osd.1) 376 : cluster 0 1.b scrub ok 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.848180+0000 osd.0 (osd.0) 367 : cluster 0 1.1 scrub starts 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.849797+0000 osd.0 (osd.0) 368 : cluster 0 1.1 scrub ok 2024-04-02T05:00:56.199 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: audit 2024-04-02T05:00:56.078352+0000 mon.a (mon.0) 3593 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:56.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: cluster 2024-04-02T05:00:56.092797+0000 mon.a (mon.0) 3594 : cluster 0 osdmap e746: 8 total, 8 up, 7 in 2024-04-02T05:00:56.200 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:56 smithi098 bash[23790]: audit 2024-04-02T05:00:56.097544+0000 mon.a (mon.0) 3595 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-122"}]: dispatch 2024-04-02T05:00:56.210 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:56.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:56.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:56.474 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.395892+0000 osd.7 (osd.7) 177 : cluster 0 1.d scrub starts 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.397923+0000 osd.7 (osd.7) 178 : cluster 0 1.d scrub ok 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.689624+0000 osd.3 (osd.3) 371 : cluster 0 3.18 scrub starts 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.692450+0000 osd.3 (osd.3) 372 : cluster 0 3.18 scrub ok 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.982992+0000 osd.2 (osd.2) 335 : cluster 0 1.5 scrub starts 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:54.984443+0000 osd.2 (osd.2) 336 : cluster 0 1.5 scrub ok 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.005795+0000 osd.6 (osd.6) 313 : cluster 0 1.6 scrub starts 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.008113+0000 osd.6 (osd.6) 314 : cluster 0 1.6 scrub ok 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.205854+0000 osd.1 (osd.1) 375 : cluster 0 1.b scrub starts 2024-04-02T05:00:56.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.206874+0000 osd.1 (osd.1) 376 : cluster 0 1.b scrub ok 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.848180+0000 osd.0 (osd.0) 367 : cluster 0 1.1 scrub starts 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.849797+0000 osd.0 (osd.0) 368 : cluster 0 1.1 scrub ok 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: audit 2024-04-02T05:00:56.078352+0000 mon.a (mon.0) 3593 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: cluster 2024-04-02T05:00:56.092797+0000 mon.a (mon.0) 3594 : cluster 0 osdmap e746: 8 total, 8 up, 7 in 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[21528]: audit 2024-04-02T05:00:56.097544+0000 mon.a (mon.0) 3595 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-122"}]: dispatch 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.395892+0000 osd.7 (osd.7) 177 : cluster 0 1.d scrub starts 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.397923+0000 osd.7 (osd.7) 178 : cluster 0 1.d scrub ok 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.689624+0000 osd.3 (osd.3) 371 : cluster 0 3.18 scrub starts 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.692450+0000 osd.3 (osd.3) 372 : cluster 0 3.18 scrub ok 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.982992+0000 osd.2 (osd.2) 335 : cluster 0 1.5 scrub starts 2024-04-02T05:00:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:54.984443+0000 osd.2 (osd.2) 336 : cluster 0 1.5 scrub ok 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.005795+0000 osd.6 (osd.6) 313 : cluster 0 1.6 scrub starts 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.008113+0000 osd.6 (osd.6) 314 : cluster 0 1.6 scrub ok 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.205854+0000 osd.1 (osd.1) 375 : cluster 0 1.b scrub starts 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.206874+0000 osd.1 (osd.1) 376 : cluster 0 1.b scrub ok 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.848180+0000 osd.0 (osd.0) 367 : cluster 0 1.1 scrub starts 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.849797+0000 osd.0 (osd.0) 368 : cluster 0 1.1 scrub ok 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: audit 2024-04-02T05:00:56.078352+0000 mon.a (mon.0) 3593 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: cluster 2024-04-02T05:00:56.092797+0000 mon.a (mon.0) 3594 : cluster 0 osdmap e746: 8 total, 8 up, 7 in 2024-04-02T05:00:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:56 smithi069 bash[28904]: audit 2024-04-02T05:00:56.097544+0000 mon.a (mon.0) 3595 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-122"}]: dispatch 2024-04-02T05:00:56.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:56.707 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:56.731 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:56.834 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:56.911 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:00:56.911 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:00:56.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:00:56.911 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:00:56.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.911 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2494 4.15 4:a93a5511:::notify.2:head [watch ping cookie 94452570227712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:54.772153+0000", 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 422.13723873100002, 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16510186900000001, 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2494 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772153+0000", 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772153+0000", 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772156+0000", 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7730000000000001e-06 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772172+0000", 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5780000000000001e-05 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772237+0000", 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4634999999999994e-05 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772264+0000", 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6999999999999999e-05 2024-04-02T05:00:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772541+0000", 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027754800000000002 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.772562+0000", 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0523e-05 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.936979+0000", 2024-04-02T05:00:56.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.164417063 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.937166+0000", 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000186699 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.937255+0000", 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8848e-05 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2500 4.15 4:a93a5511:::notify.2:head [watch reconnect cookie 94452570227712 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:54.775410+0000", 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 422.13398128199998, 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17615541700000001, 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2500 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775410+0000", 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775410+0000", 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775412+0000", 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6360000000000001e-06 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775419+0000", 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9709999999999998e-06 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775437+0000", 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8317000000000001e-05 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775447+0000", 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8940000000000005e-06 2024-04-02T05:00:56.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775525+0000", 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8459000000000002e-05 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.775540+0000", 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4773999999999999e-05 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.937708+0000", 2024-04-02T05:00:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16216826400000001 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.937788+0000", 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9696000000000004e-05 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.951566+0000", 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013777406000000001 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:00:56.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:00:56.922 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:00:56.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:56.984 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:56.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:57.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:57.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:57.243 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.411643+0000 osd.7 (osd.7) 179 : cluster 0 1.2 scrub starts 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.414254+0000 osd.7 (osd.7) 180 : cluster 0 1.2 scrub ok 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.788149+0000 osd.4 (osd.4) 372 : cluster 0 4.13 scrub starts 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.790520+0000 osd.4 (osd.4) 373 : cluster 0 4.13 scrub ok 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.977056+0000 osd.6 (osd.6) 315 : cluster 0 1.7 scrub starts 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:55.978799+0000 osd.6 (osd.6) 316 : cluster 0 1.7 scrub ok 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:56.170173+0000 osd.1 (osd.1) 377 : cluster 0 1.13 scrub starts 2024-04-02T05:00:57.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:56.175399+0000 osd.1 (osd.1) 378 : cluster 0 1.13 scrub ok 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:56.301648+0000 mgr.y (mgr.14538) 671 : cluster 0 pgmap v1131: 224 pgs: 32 unknown, 192 active+clean; 874 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: audit 2024-04-02T05:00:57.081230+0000 mon.a (mon.0) 3596 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-122"}]': finished 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: cluster 2024-04-02T05:00:57.088563+0000 mon.a (mon.0) 3597 : cluster 0 osdmap e747: 8 total, 8 up, 7 in 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[21528]: audit 2024-04-02T05:00:57.090095+0000 mon.a (mon.0) 3598 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-122", "mode": "writeback"}]: dispatch 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.411643+0000 osd.7 (osd.7) 179 : cluster 0 1.2 scrub starts 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.414254+0000 osd.7 (osd.7) 180 : cluster 0 1.2 scrub ok 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.788149+0000 osd.4 (osd.4) 372 : cluster 0 4.13 scrub starts 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.790520+0000 osd.4 (osd.4) 373 : cluster 0 4.13 scrub ok 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.977056+0000 osd.6 (osd.6) 315 : cluster 0 1.7 scrub starts 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:55.978799+0000 osd.6 (osd.6) 316 : cluster 0 1.7 scrub ok 2024-04-02T05:00:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:56.170173+0000 osd.1 (osd.1) 377 : cluster 0 1.13 scrub starts 2024-04-02T05:00:57.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:56.175399+0000 osd.1 (osd.1) 378 : cluster 0 1.13 scrub ok 2024-04-02T05:00:57.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:56.301648+0000 mgr.y (mgr.14538) 671 : cluster 0 pgmap v1131: 224 pgs: 32 unknown, 192 active+clean; 874 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:57.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: audit 2024-04-02T05:00:57.081230+0000 mon.a (mon.0) 3596 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-122"}]': finished 2024-04-02T05:00:57.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: cluster 2024-04-02T05:00:57.088563+0000 mon.a (mon.0) 3597 : cluster 0 osdmap e747: 8 total, 8 up, 7 in 2024-04-02T05:00:57.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:57 smithi069 bash[28904]: audit 2024-04-02T05:00:57.090095+0000 mon.a (mon.0) 3598 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-122", "mode": "writeback"}]: dispatch 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- 172.21.15.69:0/740801243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 msgr2=0x7f7e2810f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- 172.21.15.69:0/740801243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 0x7f7e2810f1a0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000b0d0 tx=0x7f7e2000b3e0 comp rx=0 tx=0).stop 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- 172.21.15.69:0/740801243 shutdown_connections 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- 172.21.15.69:0/740801243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 0x7f7e2810f1a0 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- 172.21.15.69:0/740801243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e2806a1d0 0x7f7e2806a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- 172.21.15.69:0/740801243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2806ab80 0x7f7e28106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- 172.21.15.69:0/740801243 >> 172.21.15.69:0/740801243 conn(0x7f7e280771e0 msgr2=0x7f7e280775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- 172.21.15.69:0/740801243 shutdown_connections 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- 172.21.15.69:0/740801243 wait complete. 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 librados: starting msgr at 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 librados: starting objecter 2024-04-02T05:00:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 Processor -- start 2024-04-02T05:00:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- start start 2024-04-02T05:00:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 librados: setting wanted keys 2024-04-02T05:00:57.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 librados: calling monclient init 2024-04-02T05:00:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2806a1d0 0x7f7e28101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:57.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e2806ab80 0x7f7e281017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 0x7f7e28101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28111ae0 con 0x7f7e2806a1d0 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28111960 con 0x7f7e2806ab80 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2d822700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7e28111c60 con 0x7f7e28107620 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e277fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 0x7f7e28101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2806a1d0 0x7f7e28101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2806a1d0 0x7f7e28101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51974/0 (socket says 172.21.15.69:51974) 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 -- 172.21.15.69:0/3506153898 learned_addr learned my addr 172.21.15.69:0/3506153898 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e267fc700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e2806ab80 0x7f7e281017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 -- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 msgr2=0x7f7e28101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 0x7f7e28101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 -- 172.21.15.69:0/3506153898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e2806ab80 msgr2=0x7f7e281017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:00:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e2806ab80 0x7f7e281017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 -- 172.21.15.69:0/3506153898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e281a6560 con 0x7f7e2806a1d0 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e277fe700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e28107620 0x7f7e28101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e26ffd700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2806a1d0 0x7f7e28101270 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c920 tx=0x7f7e1800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2c820700 10 client.?.objecter ms_handle_connect 0x7f7e2806a1d0 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2c820700 10 client.?.objecter resend_mon_ops 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.312+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18007b00 con 0x7f7e2806a1d0 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 1 -- 172.21.15.69:0/3506153898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e281a6850 con 0x7f7e2806a1d0 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f7e18004cb0 con 0x7f7e2806a1d0 2024-04-02T05:00:57.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18005400 con 0x7f7e2806a1d0 2024-04-02T05:00:57.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 10 client.33651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:57.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 1 -- 172.21.15.69:0/3506153898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e281a6d90 con 0x7f7e2806a1d0 2024-04-02T05:00:57.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 1 librados: init done 2024-04-02T05:00:57.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 10 client.33651.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:00:57.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 15 client.33651.objecter _session_command_op_assign -1 1 2024-04-02T05:00:57.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 20 client.33651.objecter _get_session osd=-1 returning homeless 2024-04-02T05:00:57.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 10 client.33651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:00:57.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2d822700 1 -- 172.21.15.69:0/3506153898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e281039d0 con 0x7f7e2806a1d0 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7e18004750 con 0x7f7e2806a1d0 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7e1006f5e0 0x7f7e10071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(747..747 src has 74..747) ==== 10051+0+0 (secure 0 0 0) 0x7f7e1808e9e0 con 0x7f7e2806a1d0 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 10 client.33651.objecter ms_dispatch 0x7f7e28001550 osd_map(747..747 src has 74..747) 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 3 client.33651.objecter handle_osd_map got epochs [747,747] > 0 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 3 client.33651.objecter handle_osd_map decoding full epoch 747 2024-04-02T05:00:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e267fc700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7e1006f5e0 0x7f7e10071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 10 client.33651.objecter checking command 1 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7e1007d1a0 0x7f7e1007f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 20 client.33651.objecter _get_session s=0x7f7e1007cad0 osd=0 3 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 20 client.33651.objecter put_session s=0x7f7e1007cad0 osd=0 3 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 15 client.33651.objecter _session_command_op_remove -1 1 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 20 client.33651.objecter _get_session s=0x7f7e1007cad0 osd=0 3 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 20 client.33651.objecter get_session s=0x7f7e1007cad0 osd=0 3 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 15 client.33651.objecter _session_command_op_assign 0 1 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 20 client.33651.objecter put_session s=0x7f7e1007cad0 osd=0 4 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 10 client.33651.objecter _send_command 1 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e1007fd10 con 0x7f7e1007d1a0 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 20 client.33651.objecter dump_active .. 0 homeless 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=747) ==== 24+0+0 (secure 0 0 0) 0x7f7e1808ee30 con 0x7f7e2806a1d0 2024-04-02T05:00:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.316+0000 7f7e277fe700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7e1007d1a0 0x7f7e1007f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 10 client.33651.objecter ms_handle_connect 0x7f7e1006f5e0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e267fc700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f7e1006f5e0 0x7f7e10071aa0 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009680 tx=0x7f7e1c009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 10 client.33651.objecter ms_handle_connect 0x7f7e1007d1a0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e277fe700 1 --2- 172.21.15.69:0/3506153898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7e1007d1a0 0x7f7e1007f640 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 1 -- 172.21.15.69:0/3506153898 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e1007fd10 con 0x7f7e1007d1a0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 10 client.33651.objecter ms_dispatch 0x7f7e28001550 command_reply(tid 1: 0 ) 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 10 client.33651.objecter _finish_command 1 = system:0 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 20 client.33651.objecter put_session s=0x7f7e1007cad0 osd=0 4 2024-04-02T05:00:57.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:00:57.320+0000 7f7e2c820700 15 client.33651.objecter _session_command_op_remove 0 1 2024-04-02T05:00:57.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:57.356 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:00:57.356 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:00:57.356 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:00:57.356 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:00:57.356 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:00:57.357 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:00:57.367 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:57.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.411643+0000 osd.7 (osd.7) 179 : cluster 0 1.2 scrub starts 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.414254+0000 osd.7 (osd.7) 180 : cluster 0 1.2 scrub ok 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.788149+0000 osd.4 (osd.4) 372 : cluster 0 4.13 scrub starts 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.790520+0000 osd.4 (osd.4) 373 : cluster 0 4.13 scrub ok 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.977056+0000 osd.6 (osd.6) 315 : cluster 0 1.7 scrub starts 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:55.978799+0000 osd.6 (osd.6) 316 : cluster 0 1.7 scrub ok 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:56.170173+0000 osd.1 (osd.1) 377 : cluster 0 1.13 scrub starts 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:56.175399+0000 osd.1 (osd.1) 378 : cluster 0 1.13 scrub ok 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:56.301648+0000 mgr.y (mgr.14538) 671 : cluster 0 pgmap v1131: 224 pgs: 32 unknown, 192 active+clean; 874 KiB data, 693 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: audit 2024-04-02T05:00:57.081230+0000 mon.a (mon.0) 3596 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-122"}]': finished 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: cluster 2024-04-02T05:00:57.088563+0000 mon.a (mon.0) 3597 : cluster 0 osdmap e747: 8 total, 8 up, 7 in 2024-04-02T05:00:57.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:57 smithi098 bash[23790]: audit 2024-04-02T05:00:57.090095+0000 mon.a (mon.0) 3598 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-122", "mode": "writeback"}]: dispatch 2024-04-02T05:00:57.490 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:57.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:57.618 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:57.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:57.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-02T05:00:57.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:57.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:57.994 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:00:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:57.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:00:58.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:58.126 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:00:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:00:58.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:58.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: cluster 2024-04-02T05:00:56.460165+0000 osd.7 (osd.7) 181 : cluster 0 1.0 scrub starts 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: cluster 2024-04-02T05:00:56.464697+0000 osd.7 (osd.7) 182 : cluster 0 1.0 scrub ok 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: cluster 2024-04-02T05:00:57.459128+0000 osd.6 (osd.6) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: cluster 2024-04-02T05:00:57.459814+0000 osd.6 (osd.6) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.081542+0000 mon.a (mon.0) 3599 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: audit 2024-04-02T05:00:58.084074+0000 mon.a (mon.0) 3600 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-122", "mode": "writeback"}]': finished 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.091736+0000 mon.a (mon.0) 3601 : cluster 0 osdmap e748: 8 total, 8 up, 7 in 2024-04-02T05:00:58.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: cluster 2024-04-02T05:00:56.460165+0000 osd.7 (osd.7) 181 : cluster 0 1.0 scrub starts 2024-04-02T05:00:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: cluster 2024-04-02T05:00:56.464697+0000 osd.7 (osd.7) 182 : cluster 0 1.0 scrub ok 2024-04-02T05:00:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: cluster 2024-04-02T05:00:57.459128+0000 osd.6 (osd.6) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: cluster 2024-04-02T05:00:57.459814+0000 osd.6 (osd.6) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.081542+0000 mon.a (mon.0) 3599 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: audit 2024-04-02T05:00:58.084074+0000 mon.a (mon.0) 3600 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-122", "mode": "writeback"}]': finished 2024-04-02T05:00:58.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.091736+0000 mon.a (mon.0) 3601 : cluster 0 osdmap e748: 8 total, 8 up, 7 in 2024-04-02T05:00:58.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:58.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: cluster 2024-04-02T05:00:56.460165+0000 osd.7 (osd.7) 181 : cluster 0 1.0 scrub starts 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: cluster 2024-04-02T05:00:56.464697+0000 osd.7 (osd.7) 182 : cluster 0 1.0 scrub ok 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: cluster 2024-04-02T05:00:57.459128+0000 osd.6 (osd.6) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: cluster 2024-04-02T05:00:57.459814+0000 osd.6 (osd.6) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.081542+0000 mon.a (mon.0) 3599 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: audit 2024-04-02T05:00:58.084074+0000 mon.a (mon.0) 3600 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-122", "mode": "writeback"}]': finished 2024-04-02T05:00:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:58 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.091736+0000 mon.a (mon.0) 3601 : cluster 0 osdmap e748: 8 total, 8 up, 7 in 2024-04-02T05:00:58.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:58.515 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:58.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:58.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:58.776 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:58.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:58.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:58.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:59.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:59.031 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:00:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:00:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:00:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:00:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:00:59.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:59.173 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:00:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:00:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:00:59.260 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:00:59.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 'trim stale osdmaps' 2024-04-02T05:00:59.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:00:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:00:59.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:59.439 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:59 smithi098 bash[23790]: cluster 2024-04-02T05:00:57.467362+0000 osd.7 (osd.7) 183 : cluster 0 1.e scrub starts 2024-04-02T05:00:59.439 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:59 smithi098 bash[23790]: cluster 2024-04-02T05:00:57.469311+0000 osd.7 (osd.7) 184 : cluster 0 1.e scrub ok 2024-04-02T05:00:59.439 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:59 smithi098 bash[23790]: audit 2024-04-02T05:00:58.185026+0000 mon.a (mon.0) 3602 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:59.439 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:59 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.302658+0000 mgr.y (mgr.14538) 672 : cluster 0 pgmap v1134: 224 pgs: 5 unknown, 219 active+clean; 635 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:59.439 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:59 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.665862+0000 osd.3 (osd.3) 373 : cluster 0 4.14 scrub starts 2024-04-02T05:00:59.439 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:00:59 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.668701+0000 osd.3 (osd.3) 374 : cluster 0 4.14 scrub ok 2024-04-02T05:00:59.455 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:00:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:00:59.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[21528]: cluster 2024-04-02T05:00:57.467362+0000 osd.7 (osd.7) 183 : cluster 0 1.e scrub starts 2024-04-02T05:00:59.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[21528]: cluster 2024-04-02T05:00:57.469311+0000 osd.7 (osd.7) 184 : cluster 0 1.e scrub ok 2024-04-02T05:00:59.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[21528]: audit 2024-04-02T05:00:58.185026+0000 mon.a (mon.0) 3602 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:59.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.302658+0000 mgr.y (mgr.14538) 672 : cluster 0 pgmap v1134: 224 pgs: 5 unknown, 219 active+clean; 635 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:59.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.665862+0000 osd.3 (osd.3) 373 : cluster 0 4.14 scrub starts 2024-04-02T05:00:59.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.668701+0000 osd.3 (osd.3) 374 : cluster 0 4.14 scrub ok 2024-04-02T05:00:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[28904]: cluster 2024-04-02T05:00:57.467362+0000 osd.7 (osd.7) 183 : cluster 0 1.e scrub starts 2024-04-02T05:00:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[28904]: cluster 2024-04-02T05:00:57.469311+0000 osd.7 (osd.7) 184 : cluster 0 1.e scrub ok 2024-04-02T05:00:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[28904]: audit 2024-04-02T05:00:58.185026+0000 mon.a (mon.0) 3602 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:00:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.302658+0000 mgr.y (mgr.14538) 672 : cluster 0 pgmap v1134: 224 pgs: 5 unknown, 219 active+clean; 635 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:00:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.665862+0000 osd.3 (osd.3) 373 : cluster 0 4.14 scrub starts 2024-04-02T05:00:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:00:59 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.668701+0000 osd.3 (osd.3) 374 : cluster 0 4.14 scrub ok 2024-04-02T05:00:59.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:59.584 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:59.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:59.713 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:00:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:00:59.731 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:00:59.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:59.838 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:00:59.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:00:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:00:59.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:00.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:00.096 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:00.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:00.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.447642+0000 osd.7 (osd.7) 185 : cluster 0 6.3 scrub starts 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:58.450019+0000 osd.7 (osd.7) 186 : cluster 0 6.3 scrub ok 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: audit 2024-04-02T05:00:59.163044+0000 mon.a (mon.0) 3603 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:59.169755+0000 mon.a (mon.0) 3604 : cluster 0 osdmap e749: 8 total, 8 up, 7 in 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: audit 2024-04-02T05:00:59.170959+0000 mon.a (mon.0) 3605 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122"}]: dispatch 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:59.183205+0000 osd.1 (osd.1) 379 : cluster 0 10.2 scrub starts 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:59.185285+0000 osd.1 (osd.1) 380 : cluster 0 10.2 scrub ok 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:59.301723+0000 osd.4 (osd.4) 374 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:00:59.302061+0000 osd.4 (osd.4) 375 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:00.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.051924+0000 mon.a (mon.0) 3606 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.447642+0000 osd.7 (osd.7) 185 : cluster 0 6.3 scrub starts 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:58.450019+0000 osd.7 (osd.7) 186 : cluster 0 6.3 scrub ok 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: audit 2024-04-02T05:00:59.163044+0000 mon.a (mon.0) 3603 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:59.169755+0000 mon.a (mon.0) 3604 : cluster 0 osdmap e749: 8 total, 8 up, 7 in 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: audit 2024-04-02T05:00:59.170959+0000 mon.a (mon.0) 3605 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122"}]: dispatch 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:59.183205+0000 osd.1 (osd.1) 379 : cluster 0 10.2 scrub starts 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:59.185285+0000 osd.1 (osd.1) 380 : cluster 0 10.2 scrub ok 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:59.301723+0000 osd.4 (osd.4) 374 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:00:59.302061+0000 osd.4 (osd.4) 375 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:00.341 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:00 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.051924+0000 mon.a (mon.0) 3606 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:00.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:00.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:00.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.447642+0000 osd.7 (osd.7) 185 : cluster 0 6.3 scrub starts 2024-04-02T05:01:00.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:58.450019+0000 osd.7 (osd.7) 186 : cluster 0 6.3 scrub ok 2024-04-02T05:01:00.468 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: audit 2024-04-02T05:00:59.163044+0000 mon.a (mon.0) 3603 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:59.169755+0000 mon.a (mon.0) 3604 : cluster 0 osdmap e749: 8 total, 8 up, 7 in 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: audit 2024-04-02T05:00:59.170959+0000 mon.a (mon.0) 3605 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122"}]: dispatch 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:59.183205+0000 osd.1 (osd.1) 379 : cluster 0 10.2 scrub starts 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:59.185285+0000 osd.1 (osd.1) 380 : cluster 0 10.2 scrub ok 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:59.301723+0000 osd.4 (osd.4) 374 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:00:59.302061+0000 osd.4 (osd.4) 375 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:00.469 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:00 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.051924+0000 mon.a (mon.0) 3606 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:00.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:00.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:00.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:00.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:00.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:00.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:01.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:01.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:01.134 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:01.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:01.262 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:01.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:01.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:01.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:01.515 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:01.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:01.639 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:01.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/4007465826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4107260 msgr2=0x7f6ca4107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- 172.21.15.69:0/4007465826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4107260 0x7f6ca4107640 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f6c940099c0 tx=0x7f6c94009cd0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/4007465826 shutdown_connections 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- 172.21.15.69:0/4007465826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca4101150 0x7f6ca410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- 172.21.15.69:0/4007465826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4107260 0x7f6ca4107640 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- 172.21.15.69:0/4007465826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ca4100790 0x7f6ca4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/4007465826 >> 172.21.15.69:0/4007465826 conn(0x7f6ca4077740 msgr2=0x7f6ca40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/4007465826 shutdown_connections 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/4007465826 wait complete. 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 librados: starting msgr at 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 librados: starting objecter 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 Processor -- start 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- start start 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 librados: setting wanted keys 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 librados: calling monclient init 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ca4100790 0x7f6ca41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4101150 0x7f6ca4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca4107260 0x7f6ca419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca4111b80 con 0x7f6ca4100790 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca4111a00 con 0x7f6ca4107260 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca4111d00 con 0x7f6ca4101150 2024-04-02T05:01:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4101150 0x7f6ca4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ca4100790 0x7f6ca41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4101150 0x7f6ca4198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44508/0 (socket says 172.21.15.69:44508) 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 -- 172.21.15.69:0/2949566644 learned_addr learned my addr 172.21.15.69:0/2949566644 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca4107260 0x7f6ca419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca4107260 msgr2=0x7f6ca419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ca4107260 0x7f6ca419c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ca4100790 msgr2=0x7f6ca41980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ca4100790 0x7f6ca41980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca419d100 con 0x7f6ca4101150 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4101150 0x7f6ca4198630 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800eb30 tx=0x7f6c9800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca0ff9700 10 client.?.objecter ms_handle_connect 0x7f6ca4101150 2024-04-02T05:01:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca0ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800cc90 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca419d3f0 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6c9800ce30 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003df0 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 10 client.43721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca41018a0 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 librados: init done 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.751+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6ca4199450 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6c98004000 con 0x7f6ca4101150 2024-04-02T05:01:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c8c0739f0 0x7f6c8c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(751..751 src has 74..751) ==== 9670+0+0 (secure 0 0 0) 0x7f6c9808ebc0 con 0x7f6ca4101150 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 osd_map(751..751 src has 74..751) 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 3 client.43721.objecter handle_osd_map got epochs [751,751] > 0 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 3 client.43721.objecter handle_osd_map decoding full epoch 751 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c8c0739f0 0x7f6c8c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 20 client.43721.objecter dump_active .. 0 homeless 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v751) ==== 54+0+16 (secure 0 0 0) 0x7f6c98094210 con 0x7f6ca4101150 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 10 client.43721.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 1 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6c8c0739f0 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c8c0739f0 0x7f6c8c075eb0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f6c940037a0 tx=0x7f6c94019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6ca4062e50 0x7f6ca4063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 1 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 0 1 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 4 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 10 client.43721.objecter _send_command 1 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ca4102ec0 con 0x7f6ca4062e50 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6ca4062e50 0x7f6ca4063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca4062e50 2024-04-02T05:01:01.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6ca4062e50 0x7f6ca4063230 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca4102ec0 con 0x7f6ca4062e50 2024-04-02T05:01:01.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:01.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 1 = system:0 2024-04-02T05:01:01.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 4 2024-04-02T05:01:01.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.755+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 0 1 2024-04-02T05:01:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 10 client.43721.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 2 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 2 2024-04-02T05:01:01.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca4101b60 osd=0 3 2024-04-02T05:01:01.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 0 2 2024-04-02T05:01:01.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 4 2024-04-02T05:01:01.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 10 client.43721.objecter _send_command 2 2024-04-02T05:01:01.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.775+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca4103890 con 0x7f6ca4062e50 2024-04-02T05:01:01.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca4103890 con 0x7f6ca4062e50 2024-04-02T05:01:01.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 4 2024-04-02T05:01:01.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 0 2 2024-04-02T05:01:01.784 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:01:01.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 10 client.43721.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 3 2024-04-02T05:01:01.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6ca4109500 0x7f6ca4109920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 1 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 4 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 10 client.43721.objecter _send_command 3 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6ca4062b20 con 0x7f6ca4109500 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6ca4109500 0x7f6ca4109920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6ca4109500 0x7f6ca4109920 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca4109500 2024-04-02T05:01:01.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca4062b20 con 0x7f6ca4109500 2024-04-02T05:01:01.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 3: 0 ) 2024-04-02T05:01:01.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 3 = system:0 2024-04-02T05:01:01.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 4 2024-04-02T05:01:01.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.779+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 1 3 2024-04-02T05:01:01.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 10 client.43721.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 4 2024-04-02T05:01:01.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 4 2024-04-02T05:01:01.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca4103aa0 osd=1 3 2024-04-02T05:01:01.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 1 4 2024-04-02T05:01:01.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 4 2024-04-02T05:01:01.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 10 client.43721.objecter _send_command 4 2024-04-02T05:01:01.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.795+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca402c9b0 con 0x7f6ca4109500 2024-04-02T05:01:01.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca402c9b0 con 0x7f6ca4109500 2024-04-02T05:01:01.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 4 2024-04-02T05:01:01.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 1 4 2024-04-02T05:01:01.804 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 10 client.43721.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 5 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6ca4102510 0x7f6ca4102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 5 2024-04-02T05:01:01.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 2 5 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 4 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 10 client.43721.objecter _send_command 5 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6ca402deb0 con 0x7f6ca4102510 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6ca4102510 0x7f6ca4102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca4102510 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6ca4102510 0x7f6ca4102930 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca402deb0 con 0x7f6ca4102510 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 5: 0 ) 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 5 = system:0 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 4 2024-04-02T05:01:01.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.799+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 2 5 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 10 client.43721.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 6 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 6 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca402dd40 osd=2 3 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 2 6 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 4 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 10 client.43721.objecter _send_command 6 2024-04-02T05:01:01.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca4101f90 con 0x7f6ca4102510 2024-04-02T05:01:01.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca4101f90 con 0x7f6ca4102510 2024-04-02T05:01:01.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 4 2024-04-02T05:01:01.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 2 6 2024-04-02T05:01:01.825 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:01:01.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 10 client.43721.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 7 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6ca402d840 0x7f6ca41ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 7 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 3 7 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 4 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 10 client.43721.objecter _send_command 7 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6ca41b0c90 con 0x7f6ca402d840 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6ca402d840 0x7f6ca41ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca402d840 2024-04-02T05:01:01.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6ca402d840 0x7f6ca41ade30 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca41b0c90 con 0x7f6ca402d840 2024-04-02T05:01:01.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 7: 0 ) 2024-04-02T05:01:01.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 7 = system:0 2024-04-02T05:01:01.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 4 2024-04-02T05:01:01.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.819+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 3 7 2024-04-02T05:01:01.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.163285+0000 mon.a (mon.0) 3607 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:01.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[21528]: audit 2024-04-02T05:01:00.165246+0000 mon.a (mon.0) 3608 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122"}]': finished 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.169065+0000 mon.a (mon.0) 3609 : cluster 0 osdmap e750: 8 total, 8 up, 7 in 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.303147+0000 mgr.y (mgr.14538) 673 : cluster 0 pgmap v1137: 224 pgs: 224 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.866290+0000 osd.0 (osd.0) 369 : cluster 0 4.6 scrub starts 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.868755+0000 osd.0 (osd.0) 370 : cluster 0 4.6 scrub ok 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.163285+0000 mon.a (mon.0) 3607 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[28904]: audit 2024-04-02T05:01:00.165246+0000 mon.a (mon.0) 3608 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122"}]': finished 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.169065+0000 mon.a (mon.0) 3609 : cluster 0 osdmap e750: 8 total, 8 up, 7 in 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.303147+0000 mgr.y (mgr.14538) 673 : cluster 0 pgmap v1137: 224 pgs: 224 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.866290+0000 osd.0 (osd.0) 369 : cluster 0 4.6 scrub starts 2024-04-02T05:01:01.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.868755+0000 osd.0 (osd.0) 370 : cluster 0 4.6 scrub ok 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 10 client.43721.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 8 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 8 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca4102390 osd=3 3 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 3 8 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 4 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 10 client.43721.objecter _send_command 8 2024-04-02T05:01:01.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.839+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca402ce10 con 0x7f6ca402d840 2024-04-02T05:01:01.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca402ce10 con 0x7f6ca402d840 2024-04-02T05:01:01.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 4 2024-04-02T05:01:01.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 3 8 2024-04-02T05:01:01.849 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 10 client.43721.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 9 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6ca41ab980 0x7f6ca41b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 9 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 4 9 2024-04-02T05:01:01.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 4 2024-04-02T05:01:01.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 10 client.43721.objecter _send_command 9 2024-04-02T05:01:01.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.843+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6ca402ce10 con 0x7f6ca41ab980 2024-04-02T05:01:01.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.847+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6ca41ab980 0x7f6ca41b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.847+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca41ab980 2024-04-02T05:01:01.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.847+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6ca41ab980 0x7f6ca41b1500 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.851+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca402ce10 con 0x7f6ca41ab980 2024-04-02T05:01:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.851+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 9: 0 ) 2024-04-02T05:01:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.851+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 9 = system:0 2024-04-02T05:01:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.851+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 4 2024-04-02T05:01:01.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.851+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 4 9 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 10 client.43721.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 10 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 10 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca402cb60 osd=4 3 2024-04-02T05:01:01.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 4 10 2024-04-02T05:01:01.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 4 2024-04-02T05:01:01.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 10 client.43721.objecter _send_command 10 2024-04-02T05:01:01.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.867+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca41b1a40 con 0x7f6ca41ab980 2024-04-02T05:01:01.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:01.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca41b1a40 con 0x7f6ca41ab980 2024-04-02T05:01:01.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 4 2024-04-02T05:01:01.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 4 10 2024-04-02T05:01:01.877 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:01:01.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 10 client.43721.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 11 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6ca41b20c0 0x7f6ca41b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 11 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 5 11 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 4 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 10 client.43721.objecter _send_command 11 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6ca41b7460 con 0x7f6ca41b20c0 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6ca41b20c0 0x7f6ca41b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca41b20c0 2024-04-02T05:01:01.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.871+0000 7f6ca2ffd700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6ca41b20c0 0x7f6ca41b44b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.875+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca41b7460 con 0x7f6ca41b20c0 2024-04-02T05:01:01.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.875+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 11: 0 ) 2024-04-02T05:01:01.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.875+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 11 = system:0 2024-04-02T05:01:01.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.875+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 4 2024-04-02T05:01:01.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.875+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 5 11 2024-04-02T05:01:01.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 10 client.43721.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 12 2024-04-02T05:01:01.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 12 2024-04-02T05:01:01.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca41b1a40 osd=5 3 2024-04-02T05:01:01.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 5 12 2024-04-02T05:01:01.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 4 2024-04-02T05:01:01.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 10 client.43721.objecter _send_command 12 2024-04-02T05:01:01.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.891+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca41b74c0 con 0x7f6ca41b20c0 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca41b74c0 con 0x7f6ca41b20c0 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 4 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 5 12 2024-04-02T05:01:01.899 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:01:01.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 10 client.43721.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 13 2024-04-02T05:01:01.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6ca41b67f0 0x7f6ca41b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 13 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 6 13 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 4 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 10 client.43721.objecter _send_command 13 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6ca41ba830 con 0x7f6ca41b67f0 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6ca41b67f0 0x7f6ca41b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca41b67f0 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca3fff700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6ca41b67f0 0x7f6ca41b6c50 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca41ba830 con 0x7f6ca41b67f0 2024-04-02T05:01:01.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 13: 0 ) 2024-04-02T05:01:01.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 13 = system:0 2024-04-02T05:01:01.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 4 2024-04-02T05:01:01.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.895+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 6 13 2024-04-02T05:01:01.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.163285+0000 mon.a (mon.0) 3607 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:01.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:01 smithi098 bash[23790]: audit 2024-04-02T05:01:00.165246+0000 mon.a (mon.0) 3608 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-122"}]': finished 2024-04-02T05:01:01.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.169065+0000 mon.a (mon.0) 3609 : cluster 0 osdmap e750: 8 total, 8 up, 7 in 2024-04-02T05:01:01.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.303147+0000 mgr.y (mgr.14538) 673 : cluster 0 pgmap v1137: 224 pgs: 224 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:01.902 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.866290+0000 osd.0 (osd.0) 369 : cluster 0 4.6 scrub starts 2024-04-02T05:01:01.903 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.868755+0000 osd.0 (osd.0) 370 : cluster 0 4.6 scrub ok 2024-04-02T05:01:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:01.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:01.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 10 client.43721.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 14 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 14 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca41b74c0 osd=6 3 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 6 14 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 4 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 10 client.43721.objecter _send_command 14 2024-04-02T05:01:01.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.911+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca41b96b0 con 0x7f6ca41b67f0 2024-04-02T05:01:01.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca41b96b0 con 0x7f6ca41b67f0 2024-04-02T05:01:01.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 4 2024-04-02T05:01:01.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 6 14 2024-04-02T05:01:01.922 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 10 client.43721.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 15 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6ca41b9dd0 0x7f6ca41ba210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 15 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 7 15 2024-04-02T05:01:01.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 4 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 10 client.43721.objecter _send_command 15 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6ca41bdba0 con 0x7f6ca41b9dd0 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6ca41b9dd0 0x7f6ca41ba210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca0ff9700 10 client.43721.objecter ms_handle_connect 0x7f6ca41b9dd0 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.915+0000 7f6ca37fe700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6ca41b9dd0 0x7f6ca41ba210 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.919+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ca41bdba0 con 0x7f6ca41b9dd0 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.919+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 15: 0 ) 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.919+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 15 = system:0 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.919+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 4 2024-04-02T05:01:01.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.919+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 7 15 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 10 client.43721.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign -1 16 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 20 client.43721.objecter _get_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 15 client.43721.objecter _session_command_op_remove -1 16 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 20 client.43721.objecter get_session s=0x7f6ca41b96b0 osd=7 3 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 15 client.43721.objecter _session_command_op_assign 7 16 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 4 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 10 client.43721.objecter _send_command 16 2024-04-02T05:01:01.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.935+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ca41b4a90 con 0x7f6ca41b9dd0 2024-04-02T05:01:01.943 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca0ff9700 1 -- 172.21.15.69:0/2949566644 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ca41b4a90 con 0x7f6ca41b9dd0 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca0ff9700 10 client.43721.objecter ms_dispatch 0x7f6ca4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca0ff9700 10 client.43721.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca0ff9700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 4 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca0ff9700 15 client.43721.objecter _session_command_op_remove 7 16 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 librados: watch_flush enter 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 librados: watch_flush exit 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.0 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6ca4062e50 msgr2=0x7f6ca4063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6ca4062e50 0x7f6ca4063230 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4101b60 osd=0 1 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.1 2024-04-02T05:01:01.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6ca4109500 msgr2=0x7f6ca4109920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6ca4109500 0x7f6ca4109920 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4103aa0 osd=1 1 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.2 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6ca4102510 msgr2=0x7f6ca4102930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6ca4102510 0x7f6ca4102930 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402dd40 osd=2 1 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.3 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6ca402d840 msgr2=0x7f6ca41ade30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6ca402d840 0x7f6ca41ade30 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca4102390 osd=3 1 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.4 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6ca41ab980 msgr2=0x7f6ca41b1500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6ca41ab980 0x7f6ca41b1500 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca402cb60 osd=4 1 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.5 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6ca41b20c0 msgr2=0x7f6ca41b44b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6ca41b20c0 0x7f6ca41b44b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b1a40 osd=5 1 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.6 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6ca41b67f0 msgr2=0x7f6ca41b6c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6ca41b67f0 0x7f6ca41b6c50 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b74c0 osd=6 1 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter close_session for osd.7 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6ca41b9dd0 msgr2=0x7f6ca41ba210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6ca41b9dd0 0x7f6ca41ba210 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter put_session s=0x7f6ca41b96b0 osd=7 1 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 20 client.43721.objecter shutdown clearing up homeless session... 2024-04-02T05:01:01.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 10 client.43721.objecter successfully canceled tick 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c8c0739f0 msgr2=0x7f6c8c075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c8c0739f0 0x7f6c8c075eb0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f6c940037a0 tx=0x7f6c94019040 comp rx=0 tx=0).stop 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4101150 msgr2=0x7f6ca4198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 --2- 172.21.15.69:0/2949566644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ca4101150 0x7f6ca4198630 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800eb30 tx=0x7f6c9800ee40 comp rx=0 tx=0).stop 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca3fff700 1 -- 172.21.15.69:0/2949566644 reap_dead start 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca3fff700 1 -- 172.21.15.69:0/2949566644 reap_dead start 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca3fff700 1 -- 172.21.15.69:0/2949566644 reap_dead start 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca3fff700 1 -- 172.21.15.69:0/2949566644 reap_dead start 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6ca3fff700 1 -- 172.21.15.69:0/2949566644 reap_dead start 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 shutdown_connections 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 >> 172.21.15.69:0/2949566644 conn(0x7f6ca4077740 msgr2=0x7f6ca4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 shutdown_connections 2024-04-02T05:01:01.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 -- 172.21.15.69:0/2949566644 wait complete. 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:01.939+0000 7f6caa114700 1 librados: shutdown 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:01:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:01:02.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:02.023 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:02.126 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:02.150 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:02.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:02.282 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:02.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:02.411 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:02.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:02.541 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:01:02.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:02.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:02.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:02.800 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:02.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:02.915 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.975277+0000 osd.2 (osd.2) 337 : cluster 0 2.1 scrub starts 2024-04-02T05:01:02.915 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:00.977489+0000 osd.2 (osd.2) 338 : cluster 0 2.1 scrub ok 2024-04-02T05:01:02.915 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.063303+0000 osd.6 (osd.6) 319 : cluster 0 10.f scrub starts 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.067812+0000 osd.6 (osd.6) 320 : cluster 0 10.f scrub ok 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.193324+0000 mon.a (mon.0) 3610 : cluster 0 osdmap e751: 8 total, 8 up, 7 in 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.673147+0000 osd.3 (osd.3) 375 : cluster 0 2.12 scrub starts 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.676562+0000 osd.3 (osd.3) 376 : cluster 0 2.12 scrub ok 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: audit 2024-04-02T05:01:01.760062+0000 mon.c (mon.2) 1154 : audit 0 from='client.? 172.21.15.69:0/2949566644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.989569+0000 osd.2 (osd.2) 339 : cluster 0 2.3 scrub starts 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:01.990947+0000 osd.2 (osd.2) 340 : cluster 0 2.3 scrub ok 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: cluster 2024-04-02T05:01:02.190535+0000 mon.a (mon.0) 3611 : cluster 0 osdmap e752: 8 total, 8 up, 7 in 2024-04-02T05:01:02.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[21528]: audit 2024-04-02T05:01:02.200754+0000 mon.a (mon.0) 3612 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:02.917 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.975277+0000 osd.2 (osd.2) 337 : cluster 0 2.1 scrub starts 2024-04-02T05:01:02.917 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:00.977489+0000 osd.2 (osd.2) 338 : cluster 0 2.1 scrub ok 2024-04-02T05:01:02.917 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.063303+0000 osd.6 (osd.6) 319 : cluster 0 10.f scrub starts 2024-04-02T05:01:02.917 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.067812+0000 osd.6 (osd.6) 320 : cluster 0 10.f scrub ok 2024-04-02T05:01:02.917 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.193324+0000 mon.a (mon.0) 3610 : cluster 0 osdmap e751: 8 total, 8 up, 7 in 2024-04-02T05:01:02.917 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.673147+0000 osd.3 (osd.3) 375 : cluster 0 2.12 scrub starts 2024-04-02T05:01:02.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.676562+0000 osd.3 (osd.3) 376 : cluster 0 2.12 scrub ok 2024-04-02T05:01:02.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: audit 2024-04-02T05:01:01.760062+0000 mon.c (mon.2) 1154 : audit 0 from='client.? 172.21.15.69:0/2949566644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:02.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.989569+0000 osd.2 (osd.2) 339 : cluster 0 2.3 scrub starts 2024-04-02T05:01:02.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:01.990947+0000 osd.2 (osd.2) 340 : cluster 0 2.3 scrub ok 2024-04-02T05:01:02.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: cluster 2024-04-02T05:01:02.190535+0000 mon.a (mon.0) 3611 : cluster 0 osdmap e752: 8 total, 8 up, 7 in 2024-04-02T05:01:02.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:02 smithi069 bash[28904]: audit 2024-04-02T05:01:02.200754+0000 mon.a (mon.0) 3612 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:02.931 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:02.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.975277+0000 osd.2 (osd.2) 337 : cluster 0 2.1 scrub starts 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:00.977489+0000 osd.2 (osd.2) 338 : cluster 0 2.1 scrub ok 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.063303+0000 osd.6 (osd.6) 319 : cluster 0 10.f scrub starts 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.067812+0000 osd.6 (osd.6) 320 : cluster 0 10.f scrub ok 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.193324+0000 mon.a (mon.0) 3610 : cluster 0 osdmap e751: 8 total, 8 up, 7 in 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.673147+0000 osd.3 (osd.3) 375 : cluster 0 2.12 scrub starts 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.676562+0000 osd.3 (osd.3) 376 : cluster 0 2.12 scrub ok 2024-04-02T05:01:02.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: audit 2024-04-02T05:01:01.760062+0000 mon.c (mon.2) 1154 : audit 0 from='client.? 172.21.15.69:0/2949566644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:02.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.989569+0000 osd.2 (osd.2) 339 : cluster 0 2.3 scrub starts 2024-04-02T05:01:02.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:01.990947+0000 osd.2 (osd.2) 340 : cluster 0 2.3 scrub ok 2024-04-02T05:01:02.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: cluster 2024-04-02T05:01:02.190535+0000 mon.a (mon.0) 3611 : cluster 0 osdmap e752: 8 total, 8 up, 7 in 2024-04-02T05:01:02.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:02 smithi098 bash[23790]: audit 2024-04-02T05:01:02.200754+0000 mon.a (mon.0) 3612 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:03.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:03.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:03.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:03.194 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:03.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:03.326 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[51598]: logger=sqlstore.transactions t=2024-04-02T05:01:03.059800605Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-02T05:01:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:03.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:03.455 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:03.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:03.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:03.700 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: cluster 2024-04-02T05:01:02.303906+0000 mgr.y (mgr.14538) 674 : cluster 0 pgmap v1140: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:03.700 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: cluster 2024-04-02T05:01:02.621021+0000 osd.3 (osd.3) 377 : cluster 0 4.17 scrub starts 2024-04-02T05:01:03.700 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: cluster 2024-04-02T05:01:02.623378+0000 osd.3 (osd.3) 378 : cluster 0 4.17 scrub ok 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: cluster 2024-04-02T05:01:03.022556+0000 osd.2 (osd.2) 341 : cluster 0 2.c scrub starts 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: cluster 2024-04-02T05:01:03.024948+0000 osd.2 (osd.2) 342 : cluster 0 2.c scrub ok 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: audit 2024-04-02T05:01:03.189207+0000 mon.a (mon.0) 3613 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[21528]: cluster 2024-04-02T05:01:03.195726+0000 mon.a (mon.0) 3614 : cluster 0 osdmap e753: 8 total, 8 up, 7 in 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: cluster 2024-04-02T05:01:02.303906+0000 mgr.y (mgr.14538) 674 : cluster 0 pgmap v1140: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: cluster 2024-04-02T05:01:02.621021+0000 osd.3 (osd.3) 377 : cluster 0 4.17 scrub starts 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: cluster 2024-04-02T05:01:02.623378+0000 osd.3 (osd.3) 378 : cluster 0 4.17 scrub ok 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: cluster 2024-04-02T05:01:03.022556+0000 osd.2 (osd.2) 341 : cluster 0 2.c scrub starts 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: cluster 2024-04-02T05:01:03.024948+0000 osd.2 (osd.2) 342 : cluster 0 2.c scrub ok 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: audit 2024-04-02T05:01:03.189207+0000 mon.a (mon.0) 3613 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:03.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:03 smithi069 bash[28904]: cluster 2024-04-02T05:01:03.195726+0000 mon.a (mon.0) 3614 : cluster 0 osdmap e753: 8 total, 8 up, 7 in 2024-04-02T05:01:03.715 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:03.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:03.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:03.829 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: cluster 2024-04-02T05:01:02.303906+0000 mgr.y (mgr.14538) 674 : cluster 0 pgmap v1140: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 693 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: cluster 2024-04-02T05:01:02.621021+0000 osd.3 (osd.3) 377 : cluster 0 4.17 scrub starts 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: cluster 2024-04-02T05:01:02.623378+0000 osd.3 (osd.3) 378 : cluster 0 4.17 scrub ok 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: cluster 2024-04-02T05:01:03.022556+0000 osd.2 (osd.2) 341 : cluster 0 2.c scrub starts 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: cluster 2024-04-02T05:01:03.024948+0000 osd.2 (osd.2) 342 : cluster 0 2.c scrub ok 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: audit 2024-04-02T05:01:03.189207+0000 mon.a (mon.0) 3613 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:03.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:03 smithi098 bash[23790]: cluster 2024-04-02T05:01:03.195726+0000 mon.a (mon.0) 3614 : cluster 0 osdmap e753: 8 total, 8 up, 7 in 2024-04-02T05:01:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:03.949 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:03.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:04.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:04.098 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:04.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:04.328 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:04.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:04.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:04.484 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:04.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:04.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:04.744 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:04.847 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:04.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:04.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:04.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:05.002 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[28904]: audit 2024-04-02T05:01:03.634310+0000 mgr.y (mgr.14538) 675 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[28904]: audit 2024-04-02T05:01:03.926259+0000 mon.b (mon.1) 304 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[28904]: cluster 2024-04-02T05:01:03.974609+0000 osd.2 (osd.2) 343 : cluster 0 1.a scrub starts 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[28904]: cluster 2024-04-02T05:01:03.976378+0000 osd.2 (osd.2) 344 : cluster 0 1.a scrub ok 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[28904]: cluster 2024-04-02T05:01:04.198647+0000 mon.a (mon.0) 3615 : cluster 0 osdmap e754: 8 total, 8 up, 7 in 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[28904]: audit 2024-04-02T05:01:04.228570+0000 mon.a (mon.0) 3616 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:05.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[21528]: audit 2024-04-02T05:01:03.634310+0000 mgr.y (mgr.14538) 675 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:05.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[21528]: audit 2024-04-02T05:01:03.926259+0000 mon.b (mon.1) 304 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:05.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[21528]: cluster 2024-04-02T05:01:03.974609+0000 osd.2 (osd.2) 343 : cluster 0 1.a scrub starts 2024-04-02T05:01:05.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[21528]: cluster 2024-04-02T05:01:03.976378+0000 osd.2 (osd.2) 344 : cluster 0 1.a scrub ok 2024-04-02T05:01:05.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[21528]: cluster 2024-04-02T05:01:04.198647+0000 mon.a (mon.0) 3615 : cluster 0 osdmap e754: 8 total, 8 up, 7 in 2024-04-02T05:01:05.028 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:04 smithi069 bash[21528]: audit 2024-04-02T05:01:04.228570+0000 mon.a (mon.0) 3616 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:05.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:05.125 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- 172.21.15.69:0/3050012221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0474108870 msgr2=0x7f0474108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- 172.21.15.69:0/3050012221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0474108870 0x7f0474108cf0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f04680098f0 tx=0x7f0468009c00 comp rx=0 tx=0).stop 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- 172.21.15.69:0/3050012221 shutdown_connections 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- 172.21.15.69:0/3050012221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0474109230 0x7f0474115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- 172.21.15.69:0/3050012221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f047410e870 0x7f047410ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- 172.21.15.69:0/3050012221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0474108870 0x7f0474108cf0 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- 172.21.15.69:0/3050012221 >> 172.21.15.69:0/3050012221 conn(0x7f0474075a10 msgr2=0x7f0474077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- 172.21.15.69:0/3050012221 shutdown_connections 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- 172.21.15.69:0/3050012221 wait complete. 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 librados: starting msgr at 2024-04-02T05:01:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 librados: starting objecter 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 Processor -- start 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- start start 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 librados: setting wanted keys 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 librados: calling monclient init 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 0x7f047419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0474109230 0x7f047419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f047410e870 0x7f04741a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04741180c0 con 0x7f047410e870 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0474117f40 con 0x7f0474109230 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0474118240 con 0x7f0474108870 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 0x7f047419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 0x7f047419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44532/0 (socket says 172.21.15.69:44532) 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0479074700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f047410e870 0x7f04741a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 -- 172.21.15.69:0/2939946737 learned_addr learned my addr 172.21.15.69:0/2939946737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0473fff700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0474109230 0x7f047419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 -- 172.21.15.69:0/2939946737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0474109230 msgr2=0x7f047419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0474109230 0x7f047419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 -- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f047410e870 msgr2=0x7f04741a2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f047410e870 0x7f04741a2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 -- 172.21.15.69:0/2939946737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04741a3670 con 0x7f0474108870 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0479074700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f047410e870 0x7f04741a2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0478873700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 0x7f047419e660 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f046400e9c0 tx=0x7f046400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0471ffb700 10 client.?.objecter ms_handle_connect 0x7f0474108870 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0471ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:05.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046400cbd0 con 0x7f0474108870 2024-04-02T05:01:05.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04741a3960 con 0x7f0474108870 2024-04-02T05:01:05.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.211+0000 7f047aad7700 10 client.43727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:05.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04741ab1e0 con 0x7f0474108870 2024-04-02T05:01:05.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f047aad7700 1 librados: init done 2024-04-02T05:01:05.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f046400cd70 con 0x7f0474108870 2024-04-02T05:01:05.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04640054c0 con 0x7f0474108870 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0460005120 con 0x7f0474108870 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0464021440 con 0x7f0474108870 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045c06f5e0 0x7f045c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(755..755 src has 74..755) ==== 10051+0+0 (secure 0 0 0) 0x7f046408ed10 con 0x7f0474108870 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 10 client.43727.objecter ms_dispatch 0x7f0474001550 osd_map(755..755 src has 74..755) 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 3 client.43727.objecter handle_osd_map got epochs [755,755] > 0 2024-04-02T05:01:05.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 3 client.43727.objecter handle_osd_map decoding full epoch 755 2024-04-02T05:01:05.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0473fff700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045c06f5e0 0x7f045c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:05.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 20 client.43727.objecter dump_active .. 0 homeless 2024-04-02T05:01:05.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0473fff700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045c06f5e0 0x7f045c071aa0 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f0468009fd0 tx=0x7f0468019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:05.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.215+0000 7f0471ffb700 10 client.43727.objecter ms_handle_connect 0x7f045c06f5e0 2024-04-02T05:01:05.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.219+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0464059f60 con 0x7f0474108870 2024-04-02T05:01:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:04 smithi098 bash[23790]: audit 2024-04-02T05:01:03.634310+0000 mgr.y (mgr.14538) 675 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:04 smithi098 bash[23790]: audit 2024-04-02T05:01:03.926259+0000 mon.b (mon.1) 304 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:04 smithi098 bash[23790]: cluster 2024-04-02T05:01:03.974609+0000 osd.2 (osd.2) 343 : cluster 0 1.a scrub starts 2024-04-02T05:01:05.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:04 smithi098 bash[23790]: cluster 2024-04-02T05:01:03.976378+0000 osd.2 (osd.2) 344 : cluster 0 1.a scrub ok 2024-04-02T05:01:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:04 smithi098 bash[23790]: cluster 2024-04-02T05:01:04.198647+0000 mon.a (mon.0) 3615 : cluster 0 osdmap e754: 8 total, 8 up, 7 in 2024-04-02T05:01:05.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:04 smithi098 bash[23790]: audit 2024-04-02T05:01:04.228570+0000 mon.a (mon.0) 3616 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:05.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:05.248 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:05.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:05.416 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:05.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:05.443+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f0460005410 con 0x7f0474108870 2024-04-02T05:01:05.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:05.506 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:05.609 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:05.634 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:05.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: cluster 2024-04-02T05:01:04.304724+0000 mgr.y (mgr.14538) 676 : cluster 0 pgmap v1143: 224 pgs: 5 creating+activating, 23 creating+peering, 196 active+clean; 582 KiB data, 694 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: cluster 2024-04-02T05:01:04.658894+0000 osd.4 (osd.4) 376 : cluster 0 3.1c scrub starts 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: cluster 2024-04-02T05:01:04.661468+0000 osd.4 (osd.4) 377 : cluster 0 3.1c scrub ok 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: cluster 2024-04-02T05:01:05.053483+0000 mon.a (mon.0) 3617 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: audit 2024-04-02T05:01:05.056847+0000 mon.a (mon.0) 3618 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: cluster 2024-04-02T05:01:05.062600+0000 mon.a (mon.0) 3619 : cluster 0 osdmap e755: 8 total, 8 up, 7 in 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: audit 2024-04-02T05:01:05.063351+0000 mon.a (mon.0) 3620 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-124"}]: dispatch 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: audit 2024-04-02T05:01:05.446804+0000 mon.c (mon.2) 1155 : audit 1 from='client.? 172.21.15.69:0/2939946737' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[28904]: audit 2024-04-02T05:01:05.447199+0000 mon.a (mon.0) 3621 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:05.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: cluster 2024-04-02T05:01:04.304724+0000 mgr.y (mgr.14538) 676 : cluster 0 pgmap v1143: 224 pgs: 5 creating+activating, 23 creating+peering, 196 active+clean; 582 KiB data, 694 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: cluster 2024-04-02T05:01:04.658894+0000 osd.4 (osd.4) 376 : cluster 0 3.1c scrub starts 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: cluster 2024-04-02T05:01:04.661468+0000 osd.4 (osd.4) 377 : cluster 0 3.1c scrub ok 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: cluster 2024-04-02T05:01:05.053483+0000 mon.a (mon.0) 3617 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: audit 2024-04-02T05:01:05.056847+0000 mon.a (mon.0) 3618 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: cluster 2024-04-02T05:01:05.062600+0000 mon.a (mon.0) 3619 : cluster 0 osdmap e755: 8 total, 8 up, 7 in 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: audit 2024-04-02T05:01:05.063351+0000 mon.a (mon.0) 3620 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-124"}]: dispatch 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: audit 2024-04-02T05:01:05.446804+0000 mon.c (mon.2) 1155 : audit 1 from='client.? 172.21.15.69:0/2939946737' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:05.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:05 smithi069 bash[21528]: audit 2024-04-02T05:01:05.447199+0000 mon.a (mon.0) 3621 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:05.763 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:05.866 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:05.892 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:05.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:05.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: cluster 2024-04-02T05:01:04.304724+0000 mgr.y (mgr.14538) 676 : cluster 0 pgmap v1143: 224 pgs: 5 creating+activating, 23 creating+peering, 196 active+clean; 582 KiB data, 694 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: cluster 2024-04-02T05:01:04.658894+0000 osd.4 (osd.4) 376 : cluster 0 3.1c scrub starts 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: cluster 2024-04-02T05:01:04.661468+0000 osd.4 (osd.4) 377 : cluster 0 3.1c scrub ok 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: cluster 2024-04-02T05:01:05.053483+0000 mon.a (mon.0) 3617 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: audit 2024-04-02T05:01:05.056847+0000 mon.a (mon.0) 3618 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: cluster 2024-04-02T05:01:05.062600+0000 mon.a (mon.0) 3619 : cluster 0 osdmap e755: 8 total, 8 up, 7 in 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: audit 2024-04-02T05:01:05.063351+0000 mon.a (mon.0) 3620 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-124"}]: dispatch 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: audit 2024-04-02T05:01:05.446804+0000 mon.c (mon.2) 1155 : audit 1 from='client.? 172.21.15.69:0/2939946737' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:05.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:05 smithi098 bash[23790]: audit 2024-04-02T05:01:05.447199+0000 mon.a (mon.0) 3621 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:05.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:06.017 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.067+0000 7f0471ffb700 1 -- 172.21.15.69:0/2939946737 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v756) ==== 101+0+0 (secure 0 0 0) 0x7f046408c1d0 con 0x7f0474108870 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 10 librados: watch_flush enter 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 10 librados: watch_flush exit 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 20 client.43727.objecter shutdown clearing up homeless session... 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 10 client.43727.objecter successfully canceled tick 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045c06f5e0 msgr2=0x7f045c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045c06f5e0 0x7f045c071aa0 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f0468009fd0 tx=0x7f0468019040 comp rx=0 tx=0).stop 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 msgr2=0x7f047419e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 0x7f047419e660 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f046400e9c0 tx=0x7f046400ecd0 comp rx=0 tx=0).stop 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 shutdown_connections 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f045c06f5e0 0x7f045c071aa0 unknown :-1 s=CLOSED pgs=4001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f047410e870 0x7f04741a2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0474108870 0x7f047419e660 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 --2- 172.21.15.69:0/2939946737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0474109230 0x7f047419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 >> 172.21.15.69:0/2939946737 conn(0x7f0474075a10 msgr2=0x7f0474076370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 shutdown_connections 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 -- 172.21.15.69:0/2939946737 wait complete. 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:06.071+0000 7f047aad7700 1 librados: shutdown 2024-04-02T05:01:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T05:01:06.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:06.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:06.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:06.264 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:06.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:06.497 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:06.524 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:06.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:06.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:06.759 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T05:01:06.759 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:01:06.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: cluster 2024-04-02T05:01:05.618077+0000 osd.3 (osd.3) 379 : cluster 0 4.1d scrub starts 2024-04-02T05:01:06.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: cluster 2024-04-02T05:01:05.620185+0000 osd.3 (osd.3) 380 : cluster 0 4.1d scrub ok 2024-04-02T05:01:06.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.057202+0000 mon.a (mon.0) 3622 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:01:06.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: audit 2024-04-02T05:01:06.058844+0000 mon.a (mon.0) 3623 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-124"}]': finished 2024-04-02T05:01:06.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: audit 2024-04-02T05:01:06.058922+0000 mon.a (mon.0) 3624 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.064133+0000 mon.a (mon.0) 3625 : cluster 0 osdmap e756: 8 total, 8 up, 7 in 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[28904]: audit 2024-04-02T05:01:06.065156+0000 mon.a (mon.0) 3626 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-124", "mode": "writeback"}]: dispatch 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: cluster 2024-04-02T05:01:05.618077+0000 osd.3 (osd.3) 379 : cluster 0 4.1d scrub starts 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: cluster 2024-04-02T05:01:05.620185+0000 osd.3 (osd.3) 380 : cluster 0 4.1d scrub ok 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.057202+0000 mon.a (mon.0) 3622 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: audit 2024-04-02T05:01:06.058844+0000 mon.a (mon.0) 3623 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-124"}]': finished 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: audit 2024-04-02T05:01:06.058922+0000 mon.a (mon.0) 3624 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.064133+0000 mon.a (mon.0) 3625 : cluster 0 osdmap e756: 8 total, 8 up, 7 in 2024-04-02T05:01:06.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:06 smithi069 bash[21528]: audit 2024-04-02T05:01:06.065156+0000 mon.a (mon.0) 3626 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-124", "mode": "writeback"}]: dispatch 2024-04-02T05:01:06.776 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:06.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:06.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:06.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: cluster 2024-04-02T05:01:05.618077+0000 osd.3 (osd.3) 379 : cluster 0 4.1d scrub starts 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: cluster 2024-04-02T05:01:05.620185+0000 osd.3 (osd.3) 380 : cluster 0 4.1d scrub ok 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.057202+0000 mon.a (mon.0) 3622 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: audit 2024-04-02T05:01:06.058844+0000 mon.a (mon.0) 3623 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-124"}]': finished 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: audit 2024-04-02T05:01:06.058922+0000 mon.a (mon.0) 3624 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.064133+0000 mon.a (mon.0) 3625 : cluster 0 osdmap e756: 8 total, 8 up, 7 in 2024-04-02T05:01:06.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:06 smithi098 bash[23790]: audit 2024-04-02T05:01:06.065156+0000 mon.a (mon.0) 3626 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-124", "mode": "writeback"}]: dispatch 2024-04-02T05:01:07.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:07.037 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:07.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:07.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:07.295 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:07.398 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:07.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:07.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:07.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:07.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-02T05:01:07.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:07.815 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:07.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:05.923573+0000 osd.0 (osd.0) 371 : cluster 0 4.2 scrub starts 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:05.925335+0000 osd.0 (osd.0) 372 : cluster 0 4.2 scrub ok 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.305211+0000 mgr.y (mgr.14538) 677 : cluster 0 pgmap v1146: 224 pgs: 5 creating+activating, 23 creating+peering, 196 active+clean; 582 KiB data, 694 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.473965+0000 osd.7 (osd.7) 187 : cluster 0 6.15 scrub starts 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.475828+0000 osd.7 (osd.7) 188 : cluster 0 6.15 scrub ok 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.665686+0000 osd.3 (osd.3) 381 : cluster 0 2.1e scrub starts 2024-04-02T05:01:07.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.668277+0000 osd.3 (osd.3) 382 : cluster 0 2.1e scrub ok 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.060313+0000 mon.a (mon.0) 3627 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: audit 2024-04-02T05:01:07.069480+0000 mon.a (mon.0) 3628 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-124", "mode": "writeback"}]': finished 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.077412+0000 mon.a (mon.0) 3629 : cluster 0 osdmap e757: 8 total, 8 up, 7 in 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:05.923573+0000 osd.0 (osd.0) 371 : cluster 0 4.2 scrub starts 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:05.925335+0000 osd.0 (osd.0) 372 : cluster 0 4.2 scrub ok 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.305211+0000 mgr.y (mgr.14538) 677 : cluster 0 pgmap v1146: 224 pgs: 5 creating+activating, 23 creating+peering, 196 active+clean; 582 KiB data, 694 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.473965+0000 osd.7 (osd.7) 187 : cluster 0 6.15 scrub starts 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.475828+0000 osd.7 (osd.7) 188 : cluster 0 6.15 scrub ok 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.665686+0000 osd.3 (osd.3) 381 : cluster 0 2.1e scrub starts 2024-04-02T05:01:07.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.668277+0000 osd.3 (osd.3) 382 : cluster 0 2.1e scrub ok 2024-04-02T05:01:07.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.060313+0000 mon.a (mon.0) 3627 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:07.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: audit 2024-04-02T05:01:07.069480+0000 mon.a (mon.0) 3628 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-124", "mode": "writeback"}]': finished 2024-04-02T05:01:07.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:07 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.077412+0000 mon.a (mon.0) 3629 : cluster 0 osdmap e757: 8 total, 8 up, 7 in 2024-04-02T05:01:07.947 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:07.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:05.923573+0000 osd.0 (osd.0) 371 : cluster 0 4.2 scrub starts 2024-04-02T05:01:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:05.925335+0000 osd.0 (osd.0) 372 : cluster 0 4.2 scrub ok 2024-04-02T05:01:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.305211+0000 mgr.y (mgr.14538) 677 : cluster 0 pgmap v1146: 224 pgs: 5 creating+activating, 23 creating+peering, 196 active+clean; 582 KiB data, 694 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.473965+0000 osd.7 (osd.7) 187 : cluster 0 6.15 scrub starts 2024-04-02T05:01:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.475828+0000 osd.7 (osd.7) 188 : cluster 0 6.15 scrub ok 2024-04-02T05:01:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.665686+0000 osd.3 (osd.3) 381 : cluster 0 2.1e scrub starts 2024-04-02T05:01:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.668277+0000 osd.3 (osd.3) 382 : cluster 0 2.1e scrub ok 2024-04-02T05:01:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.060313+0000 mon.a (mon.0) 3627 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: audit 2024-04-02T05:01:07.069480+0000 mon.a (mon.0) 3628 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-124", "mode": "writeback"}]': finished 2024-04-02T05:01:07.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:07 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.077412+0000 mon.a (mon.0) 3629 : cluster 0 osdmap e757: 8 total, 8 up, 7 in 2024-04-02T05:01:08.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:08.076 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:08.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:08.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:08.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:08.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:08.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:08.595 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:08.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.911394+0000 osd.0 (osd.0) 373 : cluster 0 3.1 deep-scrub starts 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:06.918096+0000 osd.0 (osd.0) 374 : cluster 0 3.1 deep-scrub ok 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.520986+0000 osd.7 (osd.7) 189 : cluster 0 3.16 scrub starts 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.525784+0000 osd.7 (osd.7) 190 : cluster 0 3.16 scrub ok 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.619868+0000 osd.3 (osd.3) 383 : cluster 0 2.1b scrub starts 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.622040+0000 osd.3 (osd.3) 384 : cluster 0 2.1b scrub ok 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: cluster 2024-04-02T05:01:08.078788+0000 mon.a (mon.0) 3630 : cluster 0 osdmap e758: 8 total, 8 up, 7 in 2024-04-02T05:01:08.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21528]: audit 2024-04-02T05:01:08.113571+0000 mon.a (mon.0) 3631 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.911394+0000 osd.0 (osd.0) 373 : cluster 0 3.1 deep-scrub starts 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:06.918096+0000 osd.0 (osd.0) 374 : cluster 0 3.1 deep-scrub ok 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.520986+0000 osd.7 (osd.7) 189 : cluster 0 3.16 scrub starts 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.525784+0000 osd.7 (osd.7) 190 : cluster 0 3.16 scrub ok 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.619868+0000 osd.3 (osd.3) 383 : cluster 0 2.1b scrub starts 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.622040+0000 osd.3 (osd.3) 384 : cluster 0 2.1b scrub ok 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: cluster 2024-04-02T05:01:08.078788+0000 mon.a (mon.0) 3630 : cluster 0 osdmap e758: 8 total, 8 up, 7 in 2024-04-02T05:01:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[28904]: audit 2024-04-02T05:01:08.113571+0000 mon.a (mon.0) 3631 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:08.724 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:08.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:08.853 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:08.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:08.966 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:01:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:01:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:01:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.911394+0000 osd.0 (osd.0) 373 : cluster 0 3.1 deep-scrub starts 2024-04-02T05:01:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:06.918096+0000 osd.0 (osd.0) 374 : cluster 0 3.1 deep-scrub ok 2024-04-02T05:01:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.520986+0000 osd.7 (osd.7) 189 : cluster 0 3.16 scrub starts 2024-04-02T05:01:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.525784+0000 osd.7 (osd.7) 190 : cluster 0 3.16 scrub ok 2024-04-02T05:01:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.619868+0000 osd.3 (osd.3) 383 : cluster 0 2.1b scrub starts 2024-04-02T05:01:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.622040+0000 osd.3 (osd.3) 384 : cluster 0 2.1b scrub ok 2024-04-02T05:01:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: cluster 2024-04-02T05:01:08.078788+0000 mon.a (mon.0) 3630 : cluster 0 osdmap e758: 8 total, 8 up, 7 in 2024-04-02T05:01:08.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:08 smithi098 bash[23790]: audit 2024-04-02T05:01:08.113571+0000 mon.a (mon.0) 3631 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:08.982 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:08.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:09.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:09.113 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:09.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:09.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:09.372 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:09.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:09.503 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:09.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:09.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:09.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T05:01:09.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:09.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.932836+0000 osd.0 (osd.0) 375 : cluster 0 4.b scrub starts 2024-04-02T05:01:09.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: cluster 2024-04-02T05:01:07.934248+0000 osd.0 (osd.0) 376 : cluster 0 4.b scrub ok 2024-04-02T05:01:09.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: cluster 2024-04-02T05:01:08.306836+0000 mgr.y (mgr.14538) 678 : cluster 0 pgmap v1149: 224 pgs: 2 active+clean+snaptrim, 222 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-02T05:01:09.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: cluster 2024-04-02T05:01:08.608871+0000 osd.3 (osd.3) 385 : cluster 0 6.b scrub starts 2024-04-02T05:01:09.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: cluster 2024-04-02T05:01:08.610427+0000 osd.3 (osd.3) 386 : cluster 0 6.b scrub ok 2024-04-02T05:01:09.878 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: audit 2024-04-02T05:01:09.085033+0000 mon.a (mon.0) 3632 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:09.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: cluster 2024-04-02T05:01:09.096511+0000 mon.a (mon.0) 3633 : cluster 0 osdmap e759: 8 total, 8 up, 7 in 2024-04-02T05:01:09.879 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:09 smithi098 bash[23790]: audit 2024-04-02T05:01:09.098942+0000 mon.a (mon.0) 3634 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124"}]: dispatch 2024-04-02T05:01:09.893 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:09.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:09.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.932836+0000 osd.0 (osd.0) 375 : cluster 0 4.b scrub starts 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: cluster 2024-04-02T05:01:07.934248+0000 osd.0 (osd.0) 376 : cluster 0 4.b scrub ok 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: cluster 2024-04-02T05:01:08.306836+0000 mgr.y (mgr.14538) 678 : cluster 0 pgmap v1149: 224 pgs: 2 active+clean+snaptrim, 222 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: cluster 2024-04-02T05:01:08.608871+0000 osd.3 (osd.3) 385 : cluster 0 6.b scrub starts 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: cluster 2024-04-02T05:01:08.610427+0000 osd.3 (osd.3) 386 : cluster 0 6.b scrub ok 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: audit 2024-04-02T05:01:09.085033+0000 mon.a (mon.0) 3632 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: cluster 2024-04-02T05:01:09.096511+0000 mon.a (mon.0) 3633 : cluster 0 osdmap e759: 8 total, 8 up, 7 in 2024-04-02T05:01:10.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[28904]: audit 2024-04-02T05:01:09.098942+0000 mon.a (mon.0) 3634 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124"}]: dispatch 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.932836+0000 osd.0 (osd.0) 375 : cluster 0 4.b scrub starts 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: cluster 2024-04-02T05:01:07.934248+0000 osd.0 (osd.0) 376 : cluster 0 4.b scrub ok 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: cluster 2024-04-02T05:01:08.306836+0000 mgr.y (mgr.14538) 678 : cluster 0 pgmap v1149: 224 pgs: 2 active+clean+snaptrim, 222 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: cluster 2024-04-02T05:01:08.608871+0000 osd.3 (osd.3) 385 : cluster 0 6.b scrub starts 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: cluster 2024-04-02T05:01:08.610427+0000 osd.3 (osd.3) 386 : cluster 0 6.b scrub ok 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: audit 2024-04-02T05:01:09.085033+0000 mon.a (mon.0) 3632 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: cluster 2024-04-02T05:01:09.096511+0000 mon.a (mon.0) 3633 : cluster 0 osdmap e759: 8 total, 8 up, 7 in 2024-04-02T05:01:10.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:09 smithi069 bash[21528]: audit 2024-04-02T05:01:09.098942+0000 mon.a (mon.0) 3634 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124"}]: dispatch 2024-04-02T05:01:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:10.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:10.149 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:10.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:10.277 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:10.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 -- 172.21.15.69:0/915922949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 msgr2=0x7fd10c10f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 --2- 172.21.15.69:0/915922949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 0x7fd10c10f1b0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fd10400b0d0 tx=0x7fd10400b3e0 comp rx=0 tx=0).stop 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 -- 172.21.15.69:0/915922949 shutdown_connections 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 --2- 172.21.15.69:0/915922949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 0x7fd10c10f1b0 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 --2- 172.21.15.69:0/915922949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd10c06aba0 0x7fd10c06af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 --2- 172.21.15.69:0/915922949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd10c06b550 0x7fd10c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.495+0000 7fd1109ff700 1 -- 172.21.15.69:0/915922949 >> 172.21.15.69:0/915922949 conn(0x7fd10c077ea0 msgr2=0x7fd10c0782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- 172.21.15.69:0/915922949 shutdown_connections 2024-04-02T05:01:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- 172.21.15.69:0/915922949 wait complete. 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 librados: starting msgr at 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 librados: starting objecter 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 Processor -- start 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- start start 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 librados: setting wanted keys 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 librados: calling monclient init 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd10c06aba0 0x7fd10c101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd10c06b550 0x7fd10c101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 0x7fd10c101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c111ae0 con 0x7fd10c06b550 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c111960 con 0x7fd10c06aba0 2024-04-02T05:01:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fd10c111c60 con 0x7fd10c107610 2024-04-02T05:01:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd109d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd10c06b550 0x7fd10c101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 0x7fd10c101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd109d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd10c06b550 0x7fd10c101a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43426/0 (socket says 172.21.15.69:43426) 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 0x7fd10c101fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44244/0 (socket says 172.21.15.69:44244) 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd109d9b700 1 -- 172.21.15.69:0/2543741817 learned_addr learned my addr 172.21.15.69:0/2543741817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10a59c700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd10c06aba0 0x7fd10c101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 -- 172.21.15.69:0/2543741817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd10c06aba0 msgr2=0x7fd10c101530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd10c06aba0 0x7fd10c101530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 -- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd10c06b550 msgr2=0x7fd10c101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd10c06b550 0x7fd10c101a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 -- 172.21.15.69:0/2543741817 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10c1a6530 con 0x7fd10c107610 2024-04-02T05:01:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd10ad9d700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd10c107610 0x7fd10c101fb0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fd10400b770 tx=0x7fd104007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 10 client.?.objecter ms_handle_connect 0x7fd10c107610 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10401f070 con 0x7fd10c107610 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fd104005070 con 0x7fd10c107610 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- 172.21.15.69:0/2543741817 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10c1a67c0 con 0x7fd10c107610 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10401b6a0 con 0x7fd10c107610 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 10 client.43733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- 172.21.15.69:0/2543741817 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10c1a6c70 con 0x7fd10c107610 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 librados: init done 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 10 client.43733.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 15 client.43733.objecter _session_command_op_assign -1 1 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 20 client.43733.objecter _get_session osd=-1 returning homeless 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 10 client.43733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:10.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd1109ff700 1 -- 172.21.15.69:0/2543741817 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd10c1039c0 con 0x7fd10c107610 2024-04-02T05:01:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fd104022070 con 0x7fd10c107610 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.499+0000 7fd0fb7fe700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd0f406f5e0 0x7fd0f4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(760..760 src has 74..760) ==== 10099+0+0 (secure 0 0 0) 0x7fd10400f020 con 0x7fd10c107610 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter ms_dispatch 0x7fd10c001550 osd_map(760..760 src has 74..760) 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 3 client.43733.objecter handle_osd_map got epochs [760,760] > 0 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 3 client.43733.objecter handle_osd_map decoding full epoch 760 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd10a59c700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd0f406f5e0 0x7fd0f4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter checking command 1 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fd0f407d2f0 0x7fd0f407f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter _get_session s=0x7fd0f407cc20 osd=1 3 2024-04-02T05:01:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter put_session s=0x7fd0f407cc20 osd=1 3 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 15 client.43733.objecter _session_command_op_remove -1 1 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter _get_session s=0x7fd0f407cc20 osd=1 3 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter get_session s=0x7fd0f407cc20 osd=1 3 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 15 client.43733.objecter _session_command_op_assign 1 1 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter put_session s=0x7fd0f407cc20 osd=1 4 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter _send_command 1 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0f407fe60 con 0x7fd0f407d2f0 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter dump_active .. 0 homeless 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=760) ==== 24+0+0 (secure 0 0 0) 0x7fd1040942a0 con 0x7fd10c107610 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd109d9b700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fd0f407d2f0 0x7fd0f407f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd10a59c700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fd0f406f5e0 0x7fd0f4071aa0 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7fd100009680 tx=0x7fd100009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter ms_handle_connect 0x7fd0f406f5e0 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter ms_handle_connect 0x7fd0f407d2f0 2024-04-02T05:01:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd109d9b700 1 --2- 172.21.15.69:0/2543741817 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fd0f407d2f0 0x7fd0f407f790 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 1 -- 172.21.15.69:0/2543741817 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd0f407fe60 con 0x7fd0f407d2f0 2024-04-02T05:01:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter ms_dispatch 0x7fd10c001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 10 client.43733.objecter _finish_command 1 = system:0 2024-04-02T05:01:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 20 client.43733.objecter put_session s=0x7fd0f407cc20 osd=1 4 2024-04-02T05:01:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:10.503+0000 7fd0fb7fe700 15 client.43733.objecter _session_command_op_remove 1 1 2024-04-02T05:01:10.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:01:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:01:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:01:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:01:10.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:10.661 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:10.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:10.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:10.906 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:08.739197+0000 osd.4 (osd.4) 378 : cluster 0 6.1a scrub starts 2024-04-02T05:01:10.906 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:08.741469+0000 osd.4 (osd.4) 379 : cluster 0 6.1a scrub ok 2024-04-02T05:01:10.906 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.047463+0000 osd.2 (osd.2) 345 : cluster 0 4.f scrub starts 2024-04-02T05:01:10.906 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.048911+0000 osd.2 (osd.2) 346 : cluster 0 4.f scrub ok 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.055500+0000 mon.a (mon.0) 3635 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.085383+0000 mon.a (mon.0) 3636 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: audit 2024-04-02T05:01:10.087080+0000 mon.a (mon.0) 3637 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124"}]': finished 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.092740+0000 mon.a (mon.0) 3638 : cluster 0 osdmap e760: 8 total, 8 up, 7 in 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.099110+0000 osd.6 (osd.6) 321 : cluster 0 10.d deep-scrub starts 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.105052+0000 osd.6 (osd.6) 322 : cluster 0 10.d deep-scrub ok 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:08.739197+0000 osd.4 (osd.4) 378 : cluster 0 6.1a scrub starts 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:08.741469+0000 osd.4 (osd.4) 379 : cluster 0 6.1a scrub ok 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.047463+0000 osd.2 (osd.2) 345 : cluster 0 4.f scrub starts 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.048911+0000 osd.2 (osd.2) 346 : cluster 0 4.f scrub ok 2024-04-02T05:01:10.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.055500+0000 mon.a (mon.0) 3635 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:10.908 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.085383+0000 mon.a (mon.0) 3636 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:10.908 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: audit 2024-04-02T05:01:10.087080+0000 mon.a (mon.0) 3637 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124"}]': finished 2024-04-02T05:01:10.908 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.092740+0000 mon.a (mon.0) 3638 : cluster 0 osdmap e760: 8 total, 8 up, 7 in 2024-04-02T05:01:10.908 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.099110+0000 osd.6 (osd.6) 321 : cluster 0 10.d deep-scrub starts 2024-04-02T05:01:10.908 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:10 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.105052+0000 osd.6 (osd.6) 322 : cluster 0 10.d deep-scrub ok 2024-04-02T05:01:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:10.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:08.739197+0000 osd.4 (osd.4) 378 : cluster 0 6.1a scrub starts 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:08.741469+0000 osd.4 (osd.4) 379 : cluster 0 6.1a scrub ok 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.047463+0000 osd.2 (osd.2) 345 : cluster 0 4.f scrub starts 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.048911+0000 osd.2 (osd.2) 346 : cluster 0 4.f scrub ok 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.055500+0000 mon.a (mon.0) 3635 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.085383+0000 mon.a (mon.0) 3636 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: audit 2024-04-02T05:01:10.087080+0000 mon.a (mon.0) 3637 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-124"}]': finished 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.092740+0000 mon.a (mon.0) 3638 : cluster 0 osdmap e760: 8 total, 8 up, 7 in 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.099110+0000 osd.6 (osd.6) 321 : cluster 0 10.d deep-scrub starts 2024-04-02T05:01:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:10 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.105052+0000 osd.6 (osd.6) 322 : cluster 0 10.d deep-scrub ok 2024-04-02T05:01:11.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:11.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:11.183 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:11.286 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:11.310 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:11.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:11.443 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:11.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:01:11.592 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:01:11.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 'trim stale osdmaps' 2024-04-02T05:01:11.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:11.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:11.827 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:11.930 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:11.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.307685+0000 mgr.y (mgr.14538) 679 : cluster 0 pgmap v1152: 224 pgs: 3 active+clean+snaptrim, 221 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-02T05:01:11.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.595337+0000 osd.3 (osd.3) 387 : cluster 0 3.5 scrub starts 2024-04-02T05:01:11.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.597563+0000 osd.3 (osd.3) 388 : cluster 0 3.5 scrub ok 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.072331+0000 osd.2 (osd.2) 347 : cluster 0 6.14 scrub starts 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.074668+0000 osd.2 (osd.2) 348 : cluster 0 6.14 scrub ok 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.096987+0000 mon.a (mon.0) 3639 : cluster 0 osdmap e761: 8 total, 8 up, 7 in 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.156073+0000 osd.1 (osd.1) 381 : cluster 0 3.4 scrub starts 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.166598+0000 osd.1 (osd.1) 382 : cluster 0 3.4 scrub ok 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.307685+0000 mgr.y (mgr.14538) 679 : cluster 0 pgmap v1152: 224 pgs: 3 active+clean+snaptrim, 221 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.595337+0000 osd.3 (osd.3) 387 : cluster 0 3.5 scrub starts 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.597563+0000 osd.3 (osd.3) 388 : cluster 0 3.5 scrub ok 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.072331+0000 osd.2 (osd.2) 347 : cluster 0 6.14 scrub starts 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.074668+0000 osd.2 (osd.2) 348 : cluster 0 6.14 scrub ok 2024-04-02T05:01:11.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.096987+0000 mon.a (mon.0) 3639 : cluster 0 osdmap e761: 8 total, 8 up, 7 in 2024-04-02T05:01:11.944 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.156073+0000 osd.1 (osd.1) 381 : cluster 0 3.4 scrub starts 2024-04-02T05:01:11.944 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:11 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.166598+0000 osd.1 (osd.1) 382 : cluster 0 3.4 scrub ok 2024-04-02T05:01:11.954 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:11.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.307685+0000 mgr.y (mgr.14538) 679 : cluster 0 pgmap v1152: 224 pgs: 3 active+clean+snaptrim, 221 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.595337+0000 osd.3 (osd.3) 387 : cluster 0 3.5 scrub starts 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.597563+0000 osd.3 (osd.3) 388 : cluster 0 3.5 scrub ok 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.072331+0000 osd.2 (osd.2) 347 : cluster 0 6.14 scrub starts 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.074668+0000 osd.2 (osd.2) 348 : cluster 0 6.14 scrub ok 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.096987+0000 mon.a (mon.0) 3639 : cluster 0 osdmap e761: 8 total, 8 up, 7 in 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.156073+0000 osd.1 (osd.1) 381 : cluster 0 3.4 scrub starts 2024-04-02T05:01:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:11 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.166598+0000 osd.1 (osd.1) 382 : cluster 0 3.4 scrub ok 2024-04-02T05:01:12.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:12.082 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:12.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:12.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:12.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:12.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:12.473 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:01:12.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:12.604 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:12.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:12.838 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:12.849 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.704137+0000 osd.4 (osd.4) 380 : cluster 0 3.1d scrub starts 2024-04-02T05:01:12.849 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:10.707370+0000 osd.4 (osd.4) 381 : cluster 0 3.1d scrub ok 2024-04-02T05:01:12.849 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.531502+0000 osd.7 (osd.7) 191 : cluster 0 2.8 scrub starts 2024-04-02T05:01:12.849 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.533572+0000 osd.7 (osd.7) 192 : cluster 0 2.8 scrub ok 2024-04-02T05:01:12.849 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.628439+0000 osd.3 (osd.3) 389 : cluster 0 4.11 scrub starts 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.629790+0000 osd.3 (osd.3) 390 : cluster 0 4.11 scrub ok 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:12.097036+0000 mon.a (mon.0) 3640 : cluster 0 osdmap e762: 8 total, 8 up, 7 in 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: audit 2024-04-02T05:01:12.101262+0000 mon.a (mon.0) 3641 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:12.133823+0000 osd.6 (osd.6) 323 : cluster 0 10.a scrub starts 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[21528]: cluster 2024-04-02T05:01:12.135584+0000 osd.6 (osd.6) 324 : cluster 0 10.a scrub ok 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.704137+0000 osd.4 (osd.4) 380 : cluster 0 3.1d scrub starts 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:10.707370+0000 osd.4 (osd.4) 381 : cluster 0 3.1d scrub ok 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.531502+0000 osd.7 (osd.7) 191 : cluster 0 2.8 scrub starts 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.533572+0000 osd.7 (osd.7) 192 : cluster 0 2.8 scrub ok 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.628439+0000 osd.3 (osd.3) 389 : cluster 0 4.11 scrub starts 2024-04-02T05:01:12.850 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.629790+0000 osd.3 (osd.3) 390 : cluster 0 4.11 scrub ok 2024-04-02T05:01:12.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:12.097036+0000 mon.a (mon.0) 3640 : cluster 0 osdmap e762: 8 total, 8 up, 7 in 2024-04-02T05:01:12.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: audit 2024-04-02T05:01:12.101262+0000 mon.a (mon.0) 3641 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:12.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:12.133823+0000 osd.6 (osd.6) 323 : cluster 0 10.a scrub starts 2024-04-02T05:01:12.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:12 smithi069 bash[28904]: cluster 2024-04-02T05:01:12.135584+0000 osd.6 (osd.6) 324 : cluster 0 10.a scrub ok 2024-04-02T05:01:12.863 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:12.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.704137+0000 osd.4 (osd.4) 380 : cluster 0 3.1d scrub starts 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:10.707370+0000 osd.4 (osd.4) 381 : cluster 0 3.1d scrub ok 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.531502+0000 osd.7 (osd.7) 191 : cluster 0 2.8 scrub starts 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.533572+0000 osd.7 (osd.7) 192 : cluster 0 2.8 scrub ok 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.628439+0000 osd.3 (osd.3) 389 : cluster 0 4.11 scrub starts 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.629790+0000 osd.3 (osd.3) 390 : cluster 0 4.11 scrub ok 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:12.097036+0000 mon.a (mon.0) 3640 : cluster 0 osdmap e762: 8 total, 8 up, 7 in 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: audit 2024-04-02T05:01:12.101262+0000 mon.a (mon.0) 3641 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:12.133823+0000 osd.6 (osd.6) 323 : cluster 0 10.a scrub starts 2024-04-02T05:01:12.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:12 smithi098 bash[23790]: cluster 2024-04-02T05:01:12.135584+0000 osd.6 (osd.6) 324 : cluster 0 10.a scrub ok 2024-04-02T05:01:12.994 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:12.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:13.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.102 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:13.132 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:13.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:13.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:13.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:13.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:13.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:13.764 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:01:13.764 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.887755+0000 osd.3 (osd.3) 391 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:13.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[23790]: cluster 2024-04-02T05:01:11.888112+0000 osd.3 (osd.3) 392 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:13.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[23790]: cluster 2024-04-02T05:01:12.308507+0000 mgr.y (mgr.14538) 680 : cluster 0 pgmap v1155: 224 pgs: 32 unknown, 3 active+clean+snaptrim, 189 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:13.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[23790]: audit 2024-04-02T05:01:13.103889+0000 mon.a (mon.0) 3642 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:13.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[23790]: cluster 2024-04-02T05:01:13.115528+0000 mon.a (mon.0) 3643 : cluster 0 osdmap e763: 8 total, 8 up, 7 in 2024-04-02T05:01:13.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:13 smithi098 bash[23790]: audit 2024-04-02T05:01:13.137518+0000 mon.a (mon.0) 3644 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:13.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.911 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:13.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:14.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:14.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.887755+0000 osd.3 (osd.3) 391 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:14.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[21528]: cluster 2024-04-02T05:01:11.888112+0000 osd.3 (osd.3) 392 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:14.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[21528]: cluster 2024-04-02T05:01:12.308507+0000 mgr.y (mgr.14538) 680 : cluster 0 pgmap v1155: 224 pgs: 32 unknown, 3 active+clean+snaptrim, 189 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[21528]: audit 2024-04-02T05:01:13.103889+0000 mon.a (mon.0) 3642 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[21528]: cluster 2024-04-02T05:01:13.115528+0000 mon.a (mon.0) 3643 : cluster 0 osdmap e763: 8 total, 8 up, 7 in 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[21528]: audit 2024-04-02T05:01:13.137518+0000 mon.a (mon.0) 3644 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.887755+0000 osd.3 (osd.3) 391 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[28904]: cluster 2024-04-02T05:01:11.888112+0000 osd.3 (osd.3) 392 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[28904]: cluster 2024-04-02T05:01:12.308507+0000 mgr.y (mgr.14538) 680 : cluster 0 pgmap v1155: 224 pgs: 32 unknown, 3 active+clean+snaptrim, 189 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[28904]: audit 2024-04-02T05:01:13.103889+0000 mon.a (mon.0) 3642 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[28904]: cluster 2024-04-02T05:01:13.115528+0000 mon.a (mon.0) 3643 : cluster 0 osdmap e763: 8 total, 8 up, 7 in 2024-04-02T05:01:14.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:13 smithi069 bash[28904]: audit 2024-04-02T05:01:13.137518+0000 mon.a (mon.0) 3644 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:14.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:14.167 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:14.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:14.291 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:14.394 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:14.416 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:14.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:14.543 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:14.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:14.672 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: cluster 2024-04-02T05:01:12.988625+0000 osd.0 (osd.0) 377 : cluster 0 4.7 scrub starts 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: cluster 2024-04-02T05:01:12.990803+0000 osd.0 (osd.0) 378 : cluster 0 4.7 scrub ok 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: cluster 2024-04-02T05:01:13.535422+0000 osd.7 (osd.7) 193 : cluster 0 3.d scrub starts 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: cluster 2024-04-02T05:01:13.538232+0000 osd.7 (osd.7) 194 : cluster 0 3.d scrub ok 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: audit 2024-04-02T05:01:13.642905+0000 mgr.y (mgr.14538) 681 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: audit 2024-04-02T05:01:14.106743+0000 mon.a (mon.0) 3645 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: cluster 2024-04-02T05:01:14.114463+0000 mon.a (mon.0) 3646 : cluster 0 osdmap e764: 8 total, 8 up, 7 in 2024-04-02T05:01:14.675 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[21528]: audit 2024-04-02T05:01:14.116475+0000 mon.a (mon.0) 3647 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-126"}]: dispatch 2024-04-02T05:01:14.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:14.799 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:14.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:14.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: cluster 2024-04-02T05:01:12.988625+0000 osd.0 (osd.0) 377 : cluster 0 4.7 scrub starts 2024-04-02T05:01:14.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: cluster 2024-04-02T05:01:12.990803+0000 osd.0 (osd.0) 378 : cluster 0 4.7 scrub ok 2024-04-02T05:01:14.913 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: cluster 2024-04-02T05:01:13.535422+0000 osd.7 (osd.7) 193 : cluster 0 3.d scrub starts 2024-04-02T05:01:14.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: cluster 2024-04-02T05:01:13.538232+0000 osd.7 (osd.7) 194 : cluster 0 3.d scrub ok 2024-04-02T05:01:14.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: audit 2024-04-02T05:01:13.642905+0000 mgr.y (mgr.14538) 681 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:14.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: audit 2024-04-02T05:01:14.106743+0000 mon.a (mon.0) 3645 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:14.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: cluster 2024-04-02T05:01:14.114463+0000 mon.a (mon.0) 3646 : cluster 0 osdmap e764: 8 total, 8 up, 7 in 2024-04-02T05:01:14.914 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:14 smithi069 bash[28904]: audit 2024-04-02T05:01:14.116475+0000 mon.a (mon.0) 3647 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-126"}]: dispatch 2024-04-02T05:01:14.929 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:14.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: cluster 2024-04-02T05:01:12.988625+0000 osd.0 (osd.0) 377 : cluster 0 4.7 scrub starts 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: cluster 2024-04-02T05:01:12.990803+0000 osd.0 (osd.0) 378 : cluster 0 4.7 scrub ok 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: cluster 2024-04-02T05:01:13.535422+0000 osd.7 (osd.7) 193 : cluster 0 3.d scrub starts 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: cluster 2024-04-02T05:01:13.538232+0000 osd.7 (osd.7) 194 : cluster 0 3.d scrub ok 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: audit 2024-04-02T05:01:13.642905+0000 mgr.y (mgr.14538) 681 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: audit 2024-04-02T05:01:14.106743+0000 mon.a (mon.0) 3645 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: cluster 2024-04-02T05:01:14.114463+0000 mon.a (mon.0) 3646 : cluster 0 osdmap e764: 8 total, 8 up, 7 in 2024-04-02T05:01:14.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:14 smithi098 bash[23790]: audit 2024-04-02T05:01:14.116475+0000 mon.a (mon.0) 3647 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-126"}]: dispatch 2024-04-02T05:01:15.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:15.061 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:15.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:15.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:15.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:15.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:15.632 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:15.684 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:15.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:15.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:14.309036+0000 mgr.y (mgr.14538) 682 : cluster 0 pgmap v1158: 224 pgs: 26 creating+peering, 198 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:15.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:14.660796+0000 osd.3 (osd.3) 393 : cluster 0 4.19 deep-scrub starts 2024-04-02T05:01:15.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:14.663335+0000 osd.3 (osd.3) 394 : cluster 0 4.19 deep-scrub ok 2024-04-02T05:01:15.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.056320+0000 mon.a (mon.0) 3648 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: audit 2024-04-02T05:01:15.060576+0000 mon.a (mon.0) 3649 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-126"}]': finished 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.070845+0000 mon.a (mon.0) 3650 : cluster 0 osdmap e765: 8 total, 8 up, 7 in 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: audit 2024-04-02T05:01:15.073371+0000 mon.a (mon.0) 3651 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-126", "mode": "writeback"}]: dispatch 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.097053+0000 osd.6 (osd.6) 325 : cluster 0 2.17 scrub starts 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.104526+0000 osd.6 (osd.6) 326 : cluster 0 2.17 scrub ok 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:14.309036+0000 mgr.y (mgr.14538) 682 : cluster 0 pgmap v1158: 224 pgs: 26 creating+peering, 198 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:14.660796+0000 osd.3 (osd.3) 393 : cluster 0 4.19 deep-scrub starts 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:14.663335+0000 osd.3 (osd.3) 394 : cluster 0 4.19 deep-scrub ok 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.056320+0000 mon.a (mon.0) 3648 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:15.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: audit 2024-04-02T05:01:15.060576+0000 mon.a (mon.0) 3649 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-126"}]': finished 2024-04-02T05:01:15.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.070845+0000 mon.a (mon.0) 3650 : cluster 0 osdmap e765: 8 total, 8 up, 7 in 2024-04-02T05:01:15.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: audit 2024-04-02T05:01:15.073371+0000 mon.a (mon.0) 3651 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-126", "mode": "writeback"}]: dispatch 2024-04-02T05:01:15.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.097053+0000 osd.6 (osd.6) 325 : cluster 0 2.17 scrub starts 2024-04-02T05:01:15.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:15 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.104526+0000 osd.6 (osd.6) 326 : cluster 0 2.17 scrub ok 2024-04-02T05:01:15.760 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:15.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:15.890 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:15.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:14.309036+0000 mgr.y (mgr.14538) 682 : cluster 0 pgmap v1158: 224 pgs: 26 creating+peering, 198 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:14.660796+0000 osd.3 (osd.3) 393 : cluster 0 4.19 deep-scrub starts 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:14.663335+0000 osd.3 (osd.3) 394 : cluster 0 4.19 deep-scrub ok 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.056320+0000 mon.a (mon.0) 3648 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: audit 2024-04-02T05:01:15.060576+0000 mon.a (mon.0) 3649 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-126"}]': finished 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.070845+0000 mon.a (mon.0) 3650 : cluster 0 osdmap e765: 8 total, 8 up, 7 in 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: audit 2024-04-02T05:01:15.073371+0000 mon.a (mon.0) 3651 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-126", "mode": "writeback"}]: dispatch 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.097053+0000 osd.6 (osd.6) 325 : cluster 0 2.17 scrub starts 2024-04-02T05:01:15.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:15 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.104526+0000 osd.6 (osd.6) 326 : cluster 0 2.17 scrub ok 2024-04-02T05:01:15.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:16.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:16.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:16.151 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:16.155 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:16.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T05:01:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:01:16.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:16.402 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:16.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:16.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:16.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:16.664 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:16 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.061031+0000 mon.a (mon.0) 3652 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:16 smithi098 bash[23790]: audit 2024-04-02T05:01:16.065136+0000 mon.a (mon.0) 3653 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-126", "mode": "writeback"}]': finished 2024-04-02T05:01:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:16 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.074879+0000 mon.a (mon.0) 3654 : cluster 0 osdmap e766: 8 total, 8 up, 7 in 2024-04-02T05:01:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:16 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.148993+0000 osd.1 (osd.1) 383 : cluster 0 2.d scrub starts 2024-04-02T05:01:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:16 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.150081+0000 osd.1 (osd.1) 384 : cluster 0 2.d scrub ok 2024-04-02T05:01:16.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:16 smithi098 bash[23790]: audit 2024-04-02T05:01:16.176226+0000 mon.a (mon.0) 3655 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:16.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:16.796 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:16.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:16.926 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:16.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:17.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:17.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-02T05:01:17.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:17.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.061031+0000 mon.a (mon.0) 3652 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[21528]: audit 2024-04-02T05:01:16.065136+0000 mon.a (mon.0) 3653 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-126", "mode": "writeback"}]': finished 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.074879+0000 mon.a (mon.0) 3654 : cluster 0 osdmap e766: 8 total, 8 up, 7 in 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.148993+0000 osd.1 (osd.1) 383 : cluster 0 2.d scrub starts 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.150081+0000 osd.1 (osd.1) 384 : cluster 0 2.d scrub ok 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[21528]: audit 2024-04-02T05:01:16.176226+0000 mon.a (mon.0) 3655 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:17.313 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.061031+0000 mon.a (mon.0) 3652 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:17.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[28904]: audit 2024-04-02T05:01:16.065136+0000 mon.a (mon.0) 3653 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-126", "mode": "writeback"}]': finished 2024-04-02T05:01:17.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.074879+0000 mon.a (mon.0) 3654 : cluster 0 osdmap e766: 8 total, 8 up, 7 in 2024-04-02T05:01:17.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.148993+0000 osd.1 (osd.1) 383 : cluster 0 2.d scrub starts 2024-04-02T05:01:17.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.150081+0000 osd.1 (osd.1) 384 : cluster 0 2.d scrub ok 2024-04-02T05:01:17.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:16 smithi069 bash[28904]: audit 2024-04-02T05:01:16.176226+0000 mon.a (mon.0) 3655 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:17.330 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:17.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:17.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:17.589 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:17.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:17.721 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.689805+0000 osd.4 (osd.4) 382 : cluster 0 2.18 scrub starts 2024-04-02T05:01:17.721 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.692594+0000 osd.4 (osd.4) 383 : cluster 0 2.18 scrub ok 2024-04-02T05:01:17.721 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.697184+0000 osd.3 (osd.3) 395 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:15.701308+0000 osd.3 (osd.3) 396 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.309637+0000 mgr.y (mgr.14538) 683 : cluster 0 pgmap v1161: 224 pgs: 26 creating+peering, 198 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.507090+0000 osd.7 (osd.7) 195 : cluster 0 2.5 scrub starts 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:16.509571+0000 osd.7 (osd.7) 196 : cluster 0 2.5 scrub ok 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: audit 2024-04-02T05:01:17.074834+0000 mon.a (mon.0) 3656 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: cluster 2024-04-02T05:01:17.083801+0000 mon.a (mon.0) 3657 : cluster 0 osdmap e767: 8 total, 8 up, 7 in 2024-04-02T05:01:17.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:17 smithi098 bash[23790]: audit 2024-04-02T05:01:17.086076+0000 mon.a (mon.0) 3658 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126"}]: dispatch 2024-04-02T05:01:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:17.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:17.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.689805+0000 osd.4 (osd.4) 382 : cluster 0 2.18 scrub starts 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.692594+0000 osd.4 (osd.4) 383 : cluster 0 2.18 scrub ok 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.697184+0000 osd.3 (osd.3) 395 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:15.701308+0000 osd.3 (osd.3) 396 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.309637+0000 mgr.y (mgr.14538) 683 : cluster 0 pgmap v1161: 224 pgs: 26 creating+peering, 198 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.507090+0000 osd.7 (osd.7) 195 : cluster 0 2.5 scrub starts 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:16.509571+0000 osd.7 (osd.7) 196 : cluster 0 2.5 scrub ok 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: audit 2024-04-02T05:01:17.074834+0000 mon.a (mon.0) 3656 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: cluster 2024-04-02T05:01:17.083801+0000 mon.a (mon.0) 3657 : cluster 0 osdmap e767: 8 total, 8 up, 7 in 2024-04-02T05:01:17.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[21528]: audit 2024-04-02T05:01:17.086076+0000 mon.a (mon.0) 3658 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126"}]: dispatch 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.689805+0000 osd.4 (osd.4) 382 : cluster 0 2.18 scrub starts 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.692594+0000 osd.4 (osd.4) 383 : cluster 0 2.18 scrub ok 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.697184+0000 osd.3 (osd.3) 395 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:15.701308+0000 osd.3 (osd.3) 396 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.309637+0000 mgr.y (mgr.14538) 683 : cluster 0 pgmap v1161: 224 pgs: 26 creating+peering, 198 active+clean; 582 KiB data, 699 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.507090+0000 osd.7 (osd.7) 195 : cluster 0 2.5 scrub starts 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:16.509571+0000 osd.7 (osd.7) 196 : cluster 0 2.5 scrub ok 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: audit 2024-04-02T05:01:17.074834+0000 mon.a (mon.0) 3656 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: cluster 2024-04-02T05:01:17.083801+0000 mon.a (mon.0) 3657 : cluster 0 osdmap e767: 8 total, 8 up, 7 in 2024-04-02T05:01:17.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:17 smithi069 bash[28904]: audit 2024-04-02T05:01:17.086076+0000 mon.a (mon.0) 3658 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126"}]: dispatch 2024-04-02T05:01:17.980 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:17.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:18.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:18.111 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- 172.21.15.69:0/610148767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 msgr2=0x7f8020104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- 172.21.15.69:0/610148767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f8020104930 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f8014009990 tx=0x7f8014009ca0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- 172.21.15.69:0/610148767 shutdown_connections 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- 172.21.15.69:0/610148767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8020104e70 0x7f8020111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- 172.21.15.69:0/610148767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f802010a560 0x7f802010a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- 172.21.15.69:0/610148767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f8020104930 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- 172.21.15.69:0/610148767 >> 172.21.15.69:0/610148767 conn(0x7f80200fffa0 msgr2=0x7f80201023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- 172.21.15.69:0/610148767 shutdown_connections 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- 172.21.15.69:0/610148767 wait complete. 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 librados: starting msgr at 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 librados: starting objecter 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 Processor -- start 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- start start 2024-04-02T05:01:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 librados: setting wanted keys 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 librados: calling monclient init 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f802019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8020104e70 0x7f802019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f802010a560 0x7f802019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8020113c90 con 0x7f802010a560 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8020113b10 con 0x7f8020104e70 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f80268b8700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8020113e10 con 0x7f80201044b0 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f801ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f802019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f8024e55700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f802010a560 0x7f802019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f801ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f802019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57114/0 (socket says 172.21.15.69:57114) 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f801ffff700 1 -- 172.21.15.69:0/4027582737 learned_addr learned my addr 172.21.15.69:0/4027582737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8020104e70 0x7f802019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f801ffff700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8020104e70 msgr2=0x7f802019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.155+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8020104e70 0x7f802019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801ffff700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f802010a560 msgr2=0x7f802019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f802010a560 0x7f802019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801ffff700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802019f220 con 0x7f80201044b0 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8020104e70 0x7f802019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f802010a560 0x7f802019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f802019a210 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f801000e9a0 tx=0x7f801000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 10 client.?.objecter ms_handle_connect 0x7f80201044b0 2024-04-02T05:01:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801000cbd0 con 0x7f80201044b0 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f801000cd70 con 0x7f80201044b0 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802019f510 con 0x7f80201044b0 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010010e60 con 0x7f80201044b0 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 10 client.43739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80201a6d90 con 0x7f80201044b0 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 1 librados: init done 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f802004ff00 con 0x7f80201044b0 2024-04-02T05:01:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8010010430 con 0x7f80201044b0 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f800806f5e0 0x7f8008071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(768..768 src has 74..768) ==== 10099+0+0 (secure 0 0 0) 0x7f801008eca0 con 0x7f80201044b0 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 osd_map(768..768 src has 74..768) 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 3 client.43739.objecter handle_osd_map got epochs [768,768] > 0 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 3 client.43739.objecter handle_osd_map decoding full epoch 768 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f800806f5e0 0x7f8008071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 20 client.43739.objecter dump_active .. 0 homeless 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v768) ==== 54+0+16 (secure 0 0 0) 0x7f8010094070 con 0x7f80201044b0 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 10 client.43739.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 1 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8020062e50 0x7f8020063250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 1 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 0 1 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 4 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 10 client.43739.objecter _send_command 1 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80201aa150 con 0x7f8020062e50 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f800806f5e0 0x7f8008071aa0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f801400b2e0 tx=0x7f801401a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f800806f5e0 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8020062e50 0x7f8020063250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f8020062e50 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.159+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8020062e50 0x7f8020063250 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.163+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201aa150 con 0x7f8020062e50 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.163+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.163+0000 7f801d7fa700 10 client.43739.objecter _finish_command 1 = system:0 2024-04-02T05:01:18.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.163+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 4 2024-04-02T05:01:18.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.163+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 0 1 2024-04-02T05:01:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 10 client.43739.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 2 2024-04-02T05:01:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 2 2024-04-02T05:01:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f802019b570 osd=0 3 2024-04-02T05:01:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 0 2 2024-04-02T05:01:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 4 2024-04-02T05:01:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 10 client.43739.objecter _send_command 2 2024-04-02T05:01:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.179+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80200628d0 con 0x7f8020062e50 2024-04-02T05:01:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80200628d0 con 0x7f8020062e50 2024-04-02T05:01:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 10 client.43739.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 4 2024-04-02T05:01:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 0 2 2024-04-02T05:01:18.183 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 10 client.43739.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 3 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f802010bf50 0x7f802002c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 3 2024-04-02T05:01:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 1 3 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 4 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 10 client.43739.objecter _send_command 3 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f80201adab0 con 0x7f802010bf50 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f802010bf50 0x7f802002c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f802010bf50 0x7f802002c9b0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f802010bf50 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201adab0 con 0x7f802010bf50 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 3: 0 ) 2024-04-02T05:01:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 10 client.43739.objecter _finish_command 3 = system:0 2024-04-02T05:01:18.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 4 2024-04-02T05:01:18.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.183+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 1 3 2024-04-02T05:01:18.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 10 client.43739.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 4 2024-04-02T05:01:18.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 4 2024-04-02T05:01:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f8020062cd0 osd=1 3 2024-04-02T05:01:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 1 4 2024-04-02T05:01:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 4 2024-04-02T05:01:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 10 client.43739.objecter _send_command 4 2024-04-02T05:01:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.199+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f802010b4c0 con 0x7f802010bf50 2024-04-02T05:01:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f802010b4c0 con 0x7f802010bf50 2024-04-02T05:01:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 10 client.43739.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 4 2024-04-02T05:01:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 1 4 2024-04-02T05:01:18.204 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:01:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 10 client.43739.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 5 2024-04-02T05:01:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f80201a9cb0 0x7f80201ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 5 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 2 5 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 4 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 10 client.43739.objecter _send_command 5 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f80201b1410 con 0x7f80201a9cb0 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f80201a9cb0 0x7f80201ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f80201a9cb0 0x7f80201ac310 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f80201a9cb0 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201b1410 con 0x7f80201a9cb0 2024-04-02T05:01:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 5: 0 ) 2024-04-02T05:01:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 10 client.43739.objecter _finish_command 5 = system:0 2024-04-02T05:01:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 4 2024-04-02T05:01:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.203+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 2 5 2024-04-02T05:01:18.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 10 client.43739.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 6 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 6 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f802010b8c0 osd=2 3 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 2 6 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 4 2024-04-02T05:01:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 10 client.43739.objecter _send_command 6 2024-04-02T05:01:18.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.223+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80201a92e0 con 0x7f80201a9cb0 2024-04-02T05:01:18.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80201a92e0 con 0x7f80201a9cb0 2024-04-02T05:01:18.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 10 client.43739.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 4 2024-04-02T05:01:18.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 2 6 2024-04-02T05:01:18.227 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 10 client.43739.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 7 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f802002dbd0 0x7f80201afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 7 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 3 7 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 4 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 10 client.43739.objecter _send_command 7 2024-04-02T05:01:18.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f80201b4dd0 con 0x7f802002dbd0 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f802002dbd0 0x7f80201afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f802002dbd0 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f802002dbd0 0x7f80201afc70 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201b4dd0 con 0x7f802002dbd0 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 7: 0 ) 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 10 client.43739.objecter _finish_command 7 = system:0 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 4 2024-04-02T05:01:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.227+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 3 7 2024-04-02T05:01:18.233 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 10 client.43739.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 8 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 8 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201a92e0 osd=3 3 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 3 8 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 4 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 10 client.43739.objecter _send_command 8 2024-04-02T05:01:18.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f802002d140 con 0x7f802002dbd0 2024-04-02T05:01:18.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f802002d140 con 0x7f802002dbd0 2024-04-02T05:01:18.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801d7fa700 10 client.43739.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 4 2024-04-02T05:01:18.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 3 8 2024-04-02T05:01:18.249 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:01:18.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 10 client.43739.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 9 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f802002d2b0 0x7f80201b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 9 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 4 9 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 4 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 10 client.43739.objecter _send_command 9 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f80201b4da0 con 0x7f802002d2b0 2024-04-02T05:01:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f802002d2b0 0x7f80201b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f802002d2b0 2024-04-02T05:01:18.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.247+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f802002d2b0 0x7f80201b3630 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.251+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201b4da0 con 0x7f802002d2b0 2024-04-02T05:01:18.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.251+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 9: 0 ) 2024-04-02T05:01:18.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.251+0000 7f801d7fa700 10 client.43739.objecter _finish_command 9 = system:0 2024-04-02T05:01:18.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.251+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 4 2024-04-02T05:01:18.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.251+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 4 9 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 10 client.43739.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 10 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 10 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f802002d140 osd=4 3 2024-04-02T05:01:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 4 10 2024-04-02T05:01:18.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 4 2024-04-02T05:01:18.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 10 client.43739.objecter _send_command 10 2024-04-02T05:01:18.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.267+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80201b4da0 con 0x7f802002d2b0 2024-04-02T05:01:18.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80201b4da0 con 0x7f802002d2b0 2024-04-02T05:01:18.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f801d7fa700 10 client.43739.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 4 2024-04-02T05:01:18.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 4 10 2024-04-02T05:01:18.274 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:01:18.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 10 client.43739.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 11 2024-04-02T05:01:18.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f80201b40e0 0x7f80201b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 11 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 5 11 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 4 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 10 client.43739.objecter _send_command 11 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f80201bb8f0 con 0x7f80201b40e0 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.271+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f80201b40e0 0x7f80201b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f80201b40e0 2024-04-02T05:01:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801f7fe700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f80201b40e0 0x7f80201b4540 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201bb8f0 con 0x7f80201b40e0 2024-04-02T05:01:18.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 11: 0 ) 2024-04-02T05:01:18.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801d7fa700 10 client.43739.objecter _finish_command 11 = system:0 2024-04-02T05:01:18.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 4 2024-04-02T05:01:18.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.275+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 5 11 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 10 client.43739.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 12 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 12 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201b4da0 osd=5 3 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 5 12 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 4 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 10 client.43739.objecter _send_command 12 2024-04-02T05:01:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.291+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80201ad750 con 0x7f80201b40e0 2024-04-02T05:01:18.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.295+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80201ad750 con 0x7f80201b40e0 2024-04-02T05:01:18.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.295+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.295+0000 7f801d7fa700 10 client.43739.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.295+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 4 2024-04-02T05:01:18.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.295+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 5 12 2024-04-02T05:01:18.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:01:18.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 10 client.43739.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 13 2024-04-02T05:01:18.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f80201b94e0 0x7f80201bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 13 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 6 13 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 4 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 10 client.43739.objecter _send_command 13 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f80201be860 con 0x7f80201b94e0 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f80201b94e0 0x7f80201bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f80201b94e0 2024-04-02T05:01:18.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f8024e55700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f80201b94e0 0x7f80201bb8d0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201be860 con 0x7f80201b94e0 2024-04-02T05:01:18.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 13: 0 ) 2024-04-02T05:01:18.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f801d7fa700 10 client.43739.objecter _finish_command 13 = system:0 2024-04-02T05:01:18.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 4 2024-04-02T05:01:18.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.299+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 6 13 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 10 client.43739.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 14 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 14 2024-04-02T05:01:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201ad720 osd=6 3 2024-04-02T05:01:18.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 6 14 2024-04-02T05:01:18.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 4 2024-04-02T05:01:18.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 10 client.43739.objecter _send_command 14 2024-04-02T05:01:18.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.319+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80201be8c0 con 0x7f80201b94e0 2024-04-02T05:01:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80201be8c0 con 0x7f80201b94e0 2024-04-02T05:01:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801d7fa700 10 client.43739.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 4 2024-04-02T05:01:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 6 14 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 10 client.43739.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 15 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f80201bdc40 0x7f80201be0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 15 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 7 15 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 4 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 10 client.43739.objecter _send_command 15 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f80201c1c30 con 0x7f80201bdc40 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f80201bdc40 0x7f80201be0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801d7fa700 10 client.43739.objecter ms_handle_connect 0x7f80201bdc40 2024-04-02T05:01:18.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.323+0000 7f801ffff700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f80201bdc40 0x7f80201be0a0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:18.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.327+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80201c1c30 con 0x7f80201bdc40 2024-04-02T05:01:18.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.327+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 15: 0 ) 2024-04-02T05:01:18.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.327+0000 7f801d7fa700 10 client.43739.objecter _finish_command 15 = system:0 2024-04-02T05:01:18.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.327+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 4 2024-04-02T05:01:18.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.327+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 7 15 2024-04-02T05:01:18.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 10 client.43739.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign -1 16 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 20 client.43739.objecter _get_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 15 client.43739.objecter _session_command_op_remove -1 16 2024-04-02T05:01:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 20 client.43739.objecter get_session s=0x7f80201be8c0 osd=7 3 2024-04-02T05:01:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 15 client.43739.objecter _session_command_op_assign 7 16 2024-04-02T05:01:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 4 2024-04-02T05:01:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 10 client.43739.objecter _send_command 16 2024-04-02T05:01:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.343+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80201c0ab0 con 0x7f80201bdc40 2024-04-02T05:01:18.351 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:01:18.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f801d7fa700 1 -- 172.21.15.69:0/4027582737 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80201c0ab0 con 0x7f80201bdc40 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f801d7fa700 10 client.43739.objecter ms_dispatch 0x7f8020001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f801d7fa700 10 client.43739.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f801d7fa700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 4 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f801d7fa700 15 client.43739.objecter _session_command_op_remove 7 16 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 librados: watch_flush enter 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 librados: watch_flush exit 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.0 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8020062e50 msgr2=0x7f8020063250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f8020062e50 0x7f8020063250 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802019b570 osd=0 1 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.1 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f802010bf50 msgr2=0x7f802002c9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f802010bf50 0x7f802002c9b0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f8020062cd0 osd=1 1 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.2 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f80201a9cb0 msgr2=0x7f80201ac310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f80201a9cb0 0x7f80201ac310 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802010b8c0 osd=2 1 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.3 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f802002dbd0 msgr2=0x7f80201afc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f802002dbd0 0x7f80201afc70 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201a92e0 osd=3 1 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.4 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f802002d2b0 msgr2=0x7f80201b3630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f802002d2b0 0x7f80201b3630 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f8024e55700 1 -- 172.21.15.69:0/4027582737 reap_dead start 2024-04-02T05:01:18.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f8024e55700 1 -- 172.21.15.69:0/4027582737 reap_dead start 2024-04-02T05:01:18.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f802002d140 osd=4 1 2024-04-02T05:01:18.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.5 2024-04-02T05:01:18.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f80201b40e0 msgr2=0x7f80201b4540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f80201b40e0 0x7f80201b4540 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201b4da0 osd=5 1 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.6 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f80201b94e0 msgr2=0x7f80201bb8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f80201b94e0 0x7f80201bb8d0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201ad720 osd=6 1 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter close_session for osd.7 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f80201bdc40 msgr2=0x7f80201be0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f80201bdc40 0x7f80201be0a0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter put_session s=0x7f80201be8c0 osd=7 1 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 20 client.43739.objecter shutdown clearing up homeless session... 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 10 client.43739.objecter successfully canceled tick 2024-04-02T05:01:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f800806f5e0 msgr2=0x7f8008071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f800806f5e0 0x7f8008071aa0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f801400b2e0 tx=0x7f801401a040 comp rx=0 tx=0).stop 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f8024e55700 1 -- 172.21.15.69:0/4027582737 reap_dead start 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 msgr2=0x7f802019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f802019a210 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f801000e9a0 tx=0x7f801000ecb0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 shutdown_connections 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 --2- 172.21.15.69:0/4027582737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80201044b0 0x7f802019a210 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 >> 172.21.15.69:0/4027582737 conn(0x7f80200fffa0 msgr2=0x7f8020108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 shutdown_connections 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 -- 172.21.15.69:0/4027582737 wait complete. 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:18.351+0000 7f80268b8700 1 librados: shutdown 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:01:18.356 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:01:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:01:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:01:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:01:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:01:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:01:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:18.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:18.481 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:18.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:18.614 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:18.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:18.742 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:18.781 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:01:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:01:18.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[21528]: cluster 2024-04-02T05:01:18.075024+0000 mon.a (mon.0) 3659 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[21528]: audit 2024-04-02T05:01:18.076747+0000 mon.a (mon.0) 3660 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126"}]': finished 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[21528]: cluster 2024-04-02T05:01:18.080501+0000 mon.a (mon.0) 3661 : cluster 0 osdmap e768: 8 total, 8 up, 7 in 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[21528]: audit 2024-04-02T05:01:18.162394+0000 mon.c (mon.2) 1156 : audit 0 from='client.? 172.21.15.69:0/4027582737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[28904]: cluster 2024-04-02T05:01:18.075024+0000 mon.a (mon.0) 3659 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[28904]: audit 2024-04-02T05:01:18.076747+0000 mon.a (mon.0) 3660 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126"}]': finished 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[28904]: cluster 2024-04-02T05:01:18.080501+0000 mon.a (mon.0) 3661 : cluster 0 osdmap e768: 8 total, 8 up, 7 in 2024-04-02T05:01:18.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:18 smithi069 bash[28904]: audit 2024-04-02T05:01:18.162394+0000 mon.c (mon.2) 1156 : audit 0 from='client.? 172.21.15.69:0/4027582737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:18.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:18.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:18.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:18 smithi098 bash[23790]: cluster 2024-04-02T05:01:18.075024+0000 mon.a (mon.0) 3659 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:18 smithi098 bash[23790]: audit 2024-04-02T05:01:18.076747+0000 mon.a (mon.0) 3660 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-126"}]': finished 2024-04-02T05:01:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:18 smithi098 bash[23790]: cluster 2024-04-02T05:01:18.080501+0000 mon.a (mon.0) 3661 : cluster 0 osdmap e768: 8 total, 8 up, 7 in 2024-04-02T05:01:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:18 smithi098 bash[23790]: audit 2024-04-02T05:01:18.162394+0000 mon.c (mon.2) 1156 : audit 0 from='client.? 172.21.15.69:0/4027582737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:18.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:19.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T05:01:19.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9043 ms) 2024-04-02T05:01:19.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-02T05:01:19.097 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6045 ms) 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5076 ms) 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (24147 ms) 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13684 ms) 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12100 ms) 2024-04-02T05:01:19.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000012 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11957 ms) 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6041 ms) 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (560897 ms total) 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-04-02T05:01:19.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1042 ms) 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11093 ms) 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17918 ms) 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (38 ms) 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30091 ms total) 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7057 ms) 2024-04-02T05:01:19.100 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7990 ms) 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrub... 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (23962 ms) 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10017 ms) 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8059 ms) 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8074 ms) 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9908 ms) 2024-04-02T05:01:19.101 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-02T05:01:19.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:19.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:19.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:19.394 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:19.498 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:19.522 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:19.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:19.651 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:19.754 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:19.780 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:19.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:19.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[28904]: cluster 2024-04-02T05:01:18.310453+0000 mgr.y (mgr.14538) 684 : cluster 0 pgmap v1164: 224 pgs: 224 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-02T05:01:19.851 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[28904]: audit 2024-04-02T05:01:18.950274+0000 mon.a (mon.0) 3662 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:19.852 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[28904]: audit 2024-04-02T05:01:18.952043+0000 mon.b (mon.1) 305 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:19.852 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[28904]: cluster 2024-04-02T05:01:19.095225+0000 mon.a (mon.0) 3663 : cluster 0 osdmap e769: 8 total, 8 up, 7 in 2024-04-02T05:01:19.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[21528]: cluster 2024-04-02T05:01:18.310453+0000 mgr.y (mgr.14538) 684 : cluster 0 pgmap v1164: 224 pgs: 224 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-02T05:01:19.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[21528]: audit 2024-04-02T05:01:18.950274+0000 mon.a (mon.0) 3662 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:19.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[21528]: audit 2024-04-02T05:01:18.952043+0000 mon.b (mon.1) 305 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:19.852 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:19 smithi069 bash[21528]: cluster 2024-04-02T05:01:19.095225+0000 mon.a (mon.0) 3663 : cluster 0 osdmap e769: 8 total, 8 up, 7 in 2024-04-02T05:01:19.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:19.911 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:19.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:19 smithi098 bash[23790]: cluster 2024-04-02T05:01:18.310453+0000 mgr.y (mgr.14538) 684 : cluster 0 pgmap v1164: 224 pgs: 224 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-02T05:01:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:19 smithi098 bash[23790]: audit 2024-04-02T05:01:18.950274+0000 mon.a (mon.0) 3662 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:19 smithi098 bash[23790]: audit 2024-04-02T05:01:18.952043+0000 mon.b (mon.1) 305 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:19 smithi098 bash[23790]: cluster 2024-04-02T05:01:19.095225+0000 mon.a (mon.0) 3663 : cluster 0 osdmap e769: 8 total, 8 up, 7 in 2024-04-02T05:01:20.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:20.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:20.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:20.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:20.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:20.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:20.431 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:20.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:20.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:20.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:20.824 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:20.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:20.939 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:20 smithi098 bash[23790]: cluster 2024-04-02T05:01:18.717843+0000 osd.3 (osd.3) 397 : cluster 0 3.b scrub starts 2024-04-02T05:01:20.939 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:20 smithi098 bash[23790]: cluster 2024-04-02T05:01:18.723181+0000 osd.3 (osd.3) 398 : cluster 0 3.b scrub ok 2024-04-02T05:01:20.939 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:20 smithi098 bash[23790]: cluster 2024-04-02T05:01:19.495120+0000 osd.7 (osd.7) 197 : cluster 0 10.10 scrub starts 2024-04-02T05:01:20.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:20 smithi098 bash[23790]: cluster 2024-04-02T05:01:19.497395+0000 osd.7 (osd.7) 198 : cluster 0 10.10 scrub ok 2024-04-02T05:01:20.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:20 smithi098 bash[23790]: cluster 2024-04-02T05:01:20.078030+0000 mon.a (mon.0) 3664 : cluster 0 osdmap e770: 8 total, 8 up, 7 in 2024-04-02T05:01:20.940 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:20 smithi098 bash[23790]: audit 2024-04-02T05:01:20.082598+0000 mon.a (mon.0) 3665 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:20.953 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:20.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[21528]: cluster 2024-04-02T05:01:18.717843+0000 osd.3 (osd.3) 397 : cluster 0 3.b scrub starts 2024-04-02T05:01:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[21528]: cluster 2024-04-02T05:01:18.723181+0000 osd.3 (osd.3) 398 : cluster 0 3.b scrub ok 2024-04-02T05:01:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[21528]: cluster 2024-04-02T05:01:19.495120+0000 osd.7 (osd.7) 197 : cluster 0 10.10 scrub starts 2024-04-02T05:01:21.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[21528]: cluster 2024-04-02T05:01:19.497395+0000 osd.7 (osd.7) 198 : cluster 0 10.10 scrub ok 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[21528]: cluster 2024-04-02T05:01:20.078030+0000 mon.a (mon.0) 3664 : cluster 0 osdmap e770: 8 total, 8 up, 7 in 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[21528]: audit 2024-04-02T05:01:20.082598+0000 mon.a (mon.0) 3665 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[28904]: cluster 2024-04-02T05:01:18.717843+0000 osd.3 (osd.3) 397 : cluster 0 3.b scrub starts 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[28904]: cluster 2024-04-02T05:01:18.723181+0000 osd.3 (osd.3) 398 : cluster 0 3.b scrub ok 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[28904]: cluster 2024-04-02T05:01:19.495120+0000 osd.7 (osd.7) 197 : cluster 0 10.10 scrub starts 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[28904]: cluster 2024-04-02T05:01:19.497395+0000 osd.7 (osd.7) 198 : cluster 0 10.10 scrub ok 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[28904]: cluster 2024-04-02T05:01:20.078030+0000 mon.a (mon.0) 3664 : cluster 0 osdmap e770: 8 total, 8 up, 7 in 2024-04-02T05:01:21.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:20 smithi069 bash[28904]: audit 2024-04-02T05:01:20.082598+0000 mon.a (mon.0) 3665 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:21.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:21.091 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:21.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:21.225 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:21.328 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:21.355 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:21.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:21.485 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:21.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:21.614 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:21.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:21.741 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:21.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:19.715128+0000 osd.3 (osd.3) 399 : cluster 0 6.1b scrub starts 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:19.717847+0000 osd.3 (osd.3) 400 : cluster 0 6.1b scrub ok 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:20.311144+0000 mgr.y (mgr.14538) 685 : cluster 0 pgmap v1167: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:20.463733+0000 osd.7 (osd.7) 199 : cluster 0 2.b scrub starts 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:20.466339+0000 osd.7 (osd.7) 200 : cluster 0 2.b scrub ok 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.013992+0000 osd.6 (osd.6) 327 : cluster 0 2.14 scrub starts 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.015840+0000 osd.6 (osd.6) 328 : cluster 0 2.14 scrub ok 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: audit 2024-04-02T05:01:21.070610+0000 mon.a (mon.0) 3666 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:21.857 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.077161+0000 mon.a (mon.0) 3667 : cluster 0 osdmap e771: 8 total, 8 up, 7 in 2024-04-02T05:01:21.858 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: audit 2024-04-02T05:01:21.081727+0000 mon.a (mon.0) 3668 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:21.858 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.129393+0000 osd.1 (osd.1) 385 : cluster 0 6.d scrub starts 2024-04-02T05:01:21.858 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:21 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.130385+0000 osd.1 (osd.1) 386 : cluster 0 6.d scrub ok 2024-04-02T05:01:21.870 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:21.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:21.922 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:01:21.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:21.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:19.715128+0000 osd.3 (osd.3) 399 : cluster 0 6.1b scrub starts 2024-04-02T05:01:21.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:19.717847+0000 osd.3 (osd.3) 400 : cluster 0 6.1b scrub ok 2024-04-02T05:01:21.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:20.311144+0000 mgr.y (mgr.14538) 685 : cluster 0 pgmap v1167: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:21.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:20.463733+0000 osd.7 (osd.7) 199 : cluster 0 2.b scrub starts 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:20.466339+0000 osd.7 (osd.7) 200 : cluster 0 2.b scrub ok 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.013992+0000 osd.6 (osd.6) 327 : cluster 0 2.14 scrub starts 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.015840+0000 osd.6 (osd.6) 328 : cluster 0 2.14 scrub ok 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: audit 2024-04-02T05:01:21.070610+0000 mon.a (mon.0) 3666 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.077161+0000 mon.a (mon.0) 3667 : cluster 0 osdmap e771: 8 total, 8 up, 7 in 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: audit 2024-04-02T05:01:21.081727+0000 mon.a (mon.0) 3668 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.129393+0000 osd.1 (osd.1) 385 : cluster 0 6.d scrub starts 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.130385+0000 osd.1 (osd.1) 386 : cluster 0 6.d scrub ok 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:19.715128+0000 osd.3 (osd.3) 399 : cluster 0 6.1b scrub starts 2024-04-02T05:01:21.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:19.717847+0000 osd.3 (osd.3) 400 : cluster 0 6.1b scrub ok 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:20.311144+0000 mgr.y (mgr.14538) 685 : cluster 0 pgmap v1167: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:20.463733+0000 osd.7 (osd.7) 199 : cluster 0 2.b scrub starts 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:20.466339+0000 osd.7 (osd.7) 200 : cluster 0 2.b scrub ok 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.013992+0000 osd.6 (osd.6) 327 : cluster 0 2.14 scrub starts 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.015840+0000 osd.6 (osd.6) 328 : cluster 0 2.14 scrub ok 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: audit 2024-04-02T05:01:21.070610+0000 mon.a (mon.0) 3666 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.077161+0000 mon.a (mon.0) 3667 : cluster 0 osdmap e771: 8 total, 8 up, 7 in 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: audit 2024-04-02T05:01:21.081727+0000 mon.a (mon.0) 3668 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.129393+0000 osd.1 (osd.1) 385 : cluster 0 6.d scrub starts 2024-04-02T05:01:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:21 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.130385+0000 osd.1 (osd.1) 386 : cluster 0 6.d scrub ok 2024-04-02T05:01:21.998 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:22.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:22.126 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:22.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:22.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:22.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:22.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:22.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:22.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:22.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:22.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T05:01:22.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:22.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.690923+0000 osd.4 (osd.4) 384 : cluster 0 4.9 scrub starts 2024-04-02T05:01:22.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.694371+0000 osd.4 (osd.4) 385 : cluster 0 4.9 scrub ok 2024-04-02T05:01:22.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.781692+0000 osd.3 (osd.3) 401 : cluster 0 6.c scrub starts 2024-04-02T05:01:22.890 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.783650+0000 osd.3 (osd.3) 402 : cluster 0 6.c scrub ok 2024-04-02T05:01:22.891 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.875543+0000 osd.0 (osd.0) 379 : cluster 0 10.8 scrub starts 2024-04-02T05:01:22.891 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:21.877681+0000 osd.0 (osd.0) 380 : cluster 0 10.8 scrub ok 2024-04-02T05:01:22.891 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: audit 2024-04-02T05:01:22.073209+0000 mon.a (mon.0) 3669 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:22.891 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: cluster 2024-04-02T05:01:22.081043+0000 mon.a (mon.0) 3670 : cluster 0 osdmap e772: 8 total, 8 up, 7 in 2024-04-02T05:01:22.891 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:22 smithi098 bash[23790]: audit 2024-04-02T05:01:22.083527+0000 mon.a (mon.0) 3671 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-128"}]: dispatch 2024-04-02T05:01:22.906 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:22.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- 172.21.15.69:0/626377936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806a8e0 msgr2=0x7fc62806ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- 172.21.15.69:0/626377936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806a8e0 0x7fc62806ad60 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009990 tx=0x7fc61c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- 172.21.15.69:0/626377936 shutdown_connections 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- 172.21.15.69:0/626377936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc62806b2a0 0x7fc62810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- 172.21.15.69:0/626377936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6281082e0 0x7fc6281086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- 172.21.15.69:0/626377936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806a8e0 0x7fc62806ad60 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- 172.21.15.69:0/626377936 >> 172.21.15.69:0/626377936 conn(0x7fc628078110 msgr2=0x7fc628078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- 172.21.15.69:0/626377936 shutdown_connections 2024-04-02T05:01:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- 172.21.15.69:0/626377936 wait complete. 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 librados: starting msgr at 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 librados: starting objecter 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 Processor -- start 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- start start 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 librados: setting wanted keys 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 librados: calling monclient init 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc62806a8e0 0x7fc628198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806b2a0 0x7fc6281986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6281082e0 0x7fc62819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc628111a90 con 0x7fc6281082e0 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc628111910 con 0x7fc62806a8e0 2024-04-02T05:01:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc62d36b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc628111c10 con 0x7fc62806b2a0 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6281082e0 0x7fc62819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6267fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806b2a0 0x7fc6281986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6281082e0 0x7fc62819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34352/0 (socket says 172.21.15.69:34352) 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 -- 172.21.15.69:0/3464981984 learned_addr learned my addr 172.21.15.69:0/3464981984 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc626ffd700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc62806a8e0 0x7fc628198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 -- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806b2a0 msgr2=0x7fc6281986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806b2a0 0x7fc6281986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 -- 172.21.15.69:0/3464981984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc62806a8e0 msgr2=0x7fc628198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc62806a8e0 0x7fc628198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 -- 172.21.15.69:0/3464981984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62819d170 con 0x7fc6281082e0 2024-04-02T05:01:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6267fc700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc62806b2a0 0x7fc6281986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:01:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc6277fe700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6281082e0 0x7fc62819ca30 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fc620007ee0 tx=0x7fc62000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc60ffff700 10 client.?.objecter ms_handle_connect 0x7fc6281082e0 2024-04-02T05:01:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc60ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.943+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620004bb0 con 0x7fc6281082e0 2024-04-02T05:01:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 1 -- 172.21.15.69:0/3464981984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62819d460 con 0x7fc6281082e0 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc620004d50 con 0x7fc6281082e0 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620005830 con 0x7fc6281082e0 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 10 client.33681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 1 -- 172.21.15.69:0/3464981984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6281a4ce0 con 0x7fc6281082e0 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 1 librados: init done 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 10 client.33681.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 15 client.33681.objecter _session_command_op_assign -1 1 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 20 client.33681.objecter _get_session osd=-1 returning homeless 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 10 client.33681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc62d36b700 1 -- 172.21.15.69:0/3464981984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc614002920 con 0x7fc6281082e0 2024-04-02T05:01:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc62001a020 con 0x7fc6281082e0 2024-04-02T05:01:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc61006f5e0 0x7fc610071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(772..772 src has 74..772) ==== 10079+0+0 (secure 0 0 0) 0x7fc62008e710 con 0x7fc6281082e0 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 10 client.33681.objecter ms_dispatch 0x7fc628001550 osd_map(772..772 src has 74..772) 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 3 client.33681.objecter handle_osd_map got epochs [772,772] > 0 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 3 client.33681.objecter handle_osd_map decoding full epoch 772 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 10 client.33681.objecter checking command 1 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc626ffd700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc61006f5e0 0x7fc610071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc61007d220 0x7fc61007f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 20 client.33681.objecter _get_session s=0x7fc61007cb50 osd=2 3 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 20 client.33681.objecter put_session s=0x7fc61007cb50 osd=2 3 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 15 client.33681.objecter _session_command_op_remove -1 1 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 20 client.33681.objecter _get_session s=0x7fc61007cb50 osd=2 3 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 20 client.33681.objecter get_session s=0x7fc61007cb50 osd=2 3 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 15 client.33681.objecter _session_command_op_assign 2 1 2024-04-02T05:01:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 20 client.33681.objecter put_session s=0x7fc61007cb50 osd=2 4 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 10 client.33681.objecter _send_command 1 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc62001de50 con 0x7fc61007d220 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 20 client.33681.objecter dump_active .. 0 homeless 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=772) ==== 24+0+0 (secure 0 0 0) 0x7fc62008ebf0 con 0x7fc6281082e0 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc6267fc700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc61007d220 0x7fc61007f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc60ffff700 10 client.33681.objecter ms_handle_connect 0x7fc61006f5e0 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.947+0000 7fc626ffd700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc61006f5e0 0x7fc610071aa0 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7fc618009680 tx=0x7fc618009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc60ffff700 10 client.33681.objecter ms_handle_connect 0x7fc61007d220 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc6267fc700 1 --2- 172.21.15.69:0/3464981984 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc61007d220 0x7fc61007f6c0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc60ffff700 1 -- 172.21.15.69:0/3464981984 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc62001de50 con 0x7fc61007d220 2024-04-02T05:01:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc60ffff700 10 client.33681.objecter ms_dispatch 0x7fc628001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc60ffff700 10 client.33681.objecter _finish_command 1 = system:0 2024-04-02T05:01:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc60ffff700 20 client.33681.objecter put_session s=0x7fc61007cb50 osd=2 4 2024-04-02T05:01:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:22.951+0000 7fc60ffff700 15 client.33681.objecter _session_command_op_remove 2 1 2024-04-02T05:01:22.985 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:01:22.986 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:01:22.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.690923+0000 osd.4 (osd.4) 384 : cluster 0 4.9 scrub starts 2024-04-02T05:01:22.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.694371+0000 osd.4 (osd.4) 385 : cluster 0 4.9 scrub ok 2024-04-02T05:01:22.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.781692+0000 osd.3 (osd.3) 401 : cluster 0 6.c scrub starts 2024-04-02T05:01:22.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.783650+0000 osd.3 (osd.3) 402 : cluster 0 6.c scrub ok 2024-04-02T05:01:22.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.875543+0000 osd.0 (osd.0) 379 : cluster 0 10.8 scrub starts 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:21.877681+0000 osd.0 (osd.0) 380 : cluster 0 10.8 scrub ok 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: audit 2024-04-02T05:01:22.073209+0000 mon.a (mon.0) 3669 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: cluster 2024-04-02T05:01:22.081043+0000 mon.a (mon.0) 3670 : cluster 0 osdmap e772: 8 total, 8 up, 7 in 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[28904]: audit 2024-04-02T05:01:22.083527+0000 mon.a (mon.0) 3671 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-128"}]: dispatch 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.690923+0000 osd.4 (osd.4) 384 : cluster 0 4.9 scrub starts 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.694371+0000 osd.4 (osd.4) 385 : cluster 0 4.9 scrub ok 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.781692+0000 osd.3 (osd.3) 401 : cluster 0 6.c scrub starts 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.783650+0000 osd.3 (osd.3) 402 : cluster 0 6.c scrub ok 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.875543+0000 osd.0 (osd.0) 379 : cluster 0 10.8 scrub starts 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:21.877681+0000 osd.0 (osd.0) 380 : cluster 0 10.8 scrub ok 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: audit 2024-04-02T05:01:22.073209+0000 mon.a (mon.0) 3669 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:22.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: cluster 2024-04-02T05:01:22.081043+0000 mon.a (mon.0) 3670 : cluster 0 osdmap e772: 8 total, 8 up, 7 in 2024-04-02T05:01:22.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:22 smithi069 bash[21528]: audit 2024-04-02T05:01:22.083527+0000 mon.a (mon.0) 3671 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-128"}]: dispatch 2024-04-02T05:01:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2915975494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102320 msgr2=0x7fbb481027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:23.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2915975494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102320 0x7fbb481027a0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c009990 tx=0x7fbb3c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2915975494 shutdown_connections 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2915975494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102ce0 0x7fbb4810f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2915975494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb48108390 0x7fbb48108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2915975494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102320 0x7fbb481027a0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2915975494 >> 172.21.15.69:0/2915975494 conn(0x7fbb48079310 msgr2=0x7fbb48079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2915975494 shutdown_connections 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2915975494 wait complete. 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 librados: starting msgr at 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 librados: starting objecter 2024-04-02T05:01:23.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 Processor -- start 2024-04-02T05:01:23.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- start start 2024-04-02T05:01:23.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 librados: setting wanted keys 2024-04-02T05:01:23.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 librados: calling monclient init 2024-04-02T05:01:23.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 0x7fbb48074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:23.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 0x7fbb48074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:23.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb48108390 0x7fbb48074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:23.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb48111ad0 con 0x7fbb48102320 2024-04-02T05:01:23.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb48111950 con 0x7fbb48108390 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbb48111c50 con 0x7fbb48102ce0 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 0x7fbb48074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb46ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 0x7fbb48074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb46ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 0x7fbb48074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57138/0 (socket says 172.21.15.69:57138) 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb46ffd700 1 -- 172.21.15.69:0/2888435603 learned_addr learned my addr 172.21.15.69:0/2888435603 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 0x7fbb48074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34384/0 (socket says 172.21.15.69:34384) 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb47fff700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb48108390 0x7fbb48074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 -- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 msgr2=0x7fbb48074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 0x7fbb48074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 -- 172.21.15.69:0/2888435603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb48108390 msgr2=0x7fbb48074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb48108390 0x7fbb48074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 -- 172.21.15.69:0/2888435603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb481aa9d0 con 0x7fbb48102320 2024-04-02T05:01:23.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb46ffd700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 0x7fbb48074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb477fe700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 0x7fbb48074540 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800d950 tx=0x7fbb3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb44ff9700 10 client.?.objecter ms_handle_connect 0x7fbb48102320 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb44ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38017070 con 0x7fbb48102320 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb481aac60 con 0x7fbb48102320 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fbb38009db0 con 0x7fbb48102320 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38010d20 con 0x7fbb48102320 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 10 client.33687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb481ab1a0 con 0x7fbb48102320 2024-04-02T05:01:23.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.003+0000 7fbb4df2c700 1 librados: init done 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb34005120 con 0x7fbb48102320 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fbb3801a070 con 0x7fbb48102320 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbb3006f5e0 0x7fbb30071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(772..772 src has 74..772) ==== 10079+0+0 (secure 0 0 0) 0x7fbb3808e260 con 0x7fbb48102320 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 10 client.33687.objecter ms_dispatch 0x7fbb48001550 osd_map(772..772 src has 74..772) 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 3 client.33687.objecter handle_osd_map got epochs [772,772] > 0 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 3 client.33687.objecter handle_osd_map decoding full epoch 772 2024-04-02T05:01:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.007+0000 7fbb44ff9700 20 client.33687.objecter dump_active .. 0 homeless 2024-04-02T05:01:23.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:23.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.011+0000 7fbb46ffd700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbb3006f5e0 0x7fbb30071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:23.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.011+0000 7fbb44ff9700 10 client.33687.objecter ms_handle_connect 0x7fbb3006f5e0 2024-04-02T05:01:23.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.011+0000 7fbb46ffd700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbb3006f5e0 0x7fbb30071aa0 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7fbb48075d00 tx=0x7fbb3c01a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.011+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbb3805ae10 con 0x7fbb48102320 2024-04-02T05:01:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:23.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:23.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:23.247+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fbb34005410 con 0x7fbb48102320 2024-04-02T05:01:23.264 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:23.287 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:23.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:23.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:23.547 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:23.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:23.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:23.776 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:23.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:22.311629+0000 mgr.y (mgr.14538) 686 : cluster 0 pgmap v1170: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:23.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:22.856125+0000 osd.0 (osd.0) 381 : cluster 0 3.12 scrub starts 2024-04-02T05:01:23.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:22.858589+0000 osd.0 (osd.0) 382 : cluster 0 3.12 scrub ok 2024-04-02T05:01:23.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:23.028894+0000 osd.6 (osd.6) 329 : cluster 0 10.12 scrub starts 2024-04-02T05:01:23.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:23.031694+0000 osd.6 (osd.6) 330 : cluster 0 10.12 scrub ok 2024-04-02T05:01:23.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:23.073108+0000 mon.a (mon.0) 3672 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:23.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: audit 2024-04-02T05:01:23.081476+0000 mon.a (mon.0) 3673 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-128"}]': finished 2024-04-02T05:01:23.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:23.085380+0000 mon.a (mon.0) 3674 : cluster 0 osdmap e773: 8 total, 8 up, 7 in 2024-04-02T05:01:23.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: audit 2024-04-02T05:01:23.086993+0000 mon.a (mon.0) 3675 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-128", "mode": "writeback"}]: dispatch 2024-04-02T05:01:23.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:23.191595+0000 osd.1 (osd.1) 387 : cluster 0 2.2 scrub starts 2024-04-02T05:01:23.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: cluster 2024-04-02T05:01:23.193377+0000 osd.1 (osd.1) 388 : cluster 0 2.2 scrub ok 2024-04-02T05:01:23.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[28904]: audit 2024-04-02T05:01:23.251901+0000 mon.a (mon.0) 3676 : audit 1 from='client.? 172.21.15.69:0/2888435603' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:01:23.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:22.311629+0000 mgr.y (mgr.14538) 686 : cluster 0 pgmap v1170: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:23.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:22.856125+0000 osd.0 (osd.0) 381 : cluster 0 3.12 scrub starts 2024-04-02T05:01:23.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:22.858589+0000 osd.0 (osd.0) 382 : cluster 0 3.12 scrub ok 2024-04-02T05:01:23.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:23.028894+0000 osd.6 (osd.6) 329 : cluster 0 10.12 scrub starts 2024-04-02T05:01:23.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:23.031694+0000 osd.6 (osd.6) 330 : cluster 0 10.12 scrub ok 2024-04-02T05:01:23.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:23.073108+0000 mon.a (mon.0) 3672 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:23.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: audit 2024-04-02T05:01:23.081476+0000 mon.a (mon.0) 3673 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-128"}]': finished 2024-04-02T05:01:23.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:23.085380+0000 mon.a (mon.0) 3674 : cluster 0 osdmap e773: 8 total, 8 up, 7 in 2024-04-02T05:01:23.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: audit 2024-04-02T05:01:23.086993+0000 mon.a (mon.0) 3675 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-128", "mode": "writeback"}]: dispatch 2024-04-02T05:01:23.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:23.191595+0000 osd.1 (osd.1) 387 : cluster 0 2.2 scrub starts 2024-04-02T05:01:23.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: cluster 2024-04-02T05:01:23.193377+0000 osd.1 (osd.1) 388 : cluster 0 2.2 scrub ok 2024-04-02T05:01:23.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:23 smithi069 bash[21528]: audit 2024-04-02T05:01:23.251901+0000 mon.a (mon.0) 3676 : audit 1 from='client.? 172.21.15.69:0/2888435603' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:01:23.803 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:23.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:22.311629+0000 mgr.y (mgr.14538) 686 : cluster 0 pgmap v1170: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 721 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:23.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:22.856125+0000 osd.0 (osd.0) 381 : cluster 0 3.12 scrub starts 2024-04-02T05:01:23.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:22.858589+0000 osd.0 (osd.0) 382 : cluster 0 3.12 scrub ok 2024-04-02T05:01:23.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:23.028894+0000 osd.6 (osd.6) 329 : cluster 0 10.12 scrub starts 2024-04-02T05:01:23.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:23.031694+0000 osd.6 (osd.6) 330 : cluster 0 10.12 scrub ok 2024-04-02T05:01:23.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:23.073108+0000 mon.a (mon.0) 3672 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:23.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: audit 2024-04-02T05:01:23.081476+0000 mon.a (mon.0) 3673 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-128"}]': finished 2024-04-02T05:01:23.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:23.085380+0000 mon.a (mon.0) 3674 : cluster 0 osdmap e773: 8 total, 8 up, 7 in 2024-04-02T05:01:23.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: audit 2024-04-02T05:01:23.086993+0000 mon.a (mon.0) 3675 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-128", "mode": "writeback"}]: dispatch 2024-04-02T05:01:23.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:23.191595+0000 osd.1 (osd.1) 387 : cluster 0 2.2 scrub starts 2024-04-02T05:01:23.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: cluster 2024-04-02T05:01:23.193377+0000 osd.1 (osd.1) 388 : cluster 0 2.2 scrub ok 2024-04-02T05:01:23.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:23 smithi098 bash[23790]: audit 2024-04-02T05:01:23.251901+0000 mon.a (mon.0) 3676 : audit 1 from='client.? 172.21.15.69:0/2888435603' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:01:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:01:23.903 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:01:23.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 'trim stale osdmaps' 2024-04-02T05:01:23.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:23.934 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:23.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:24.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:24.066 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:24.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.078+0000 7fbb44ff9700 1 -- 172.21.15.69:0/2888435603 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v774) ==== 87+0+0 (secure 0 0 0) 0x7fbb3805e860 con 0x7fbb48102320 2024-04-02T05:01:24.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.082+0000 7fbb4df2c700 10 librados: watch_flush enter 2024-04-02T05:01:24.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.082+0000 7fbb4df2c700 10 librados: watch_flush exit 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.082+0000 7fbb4df2c700 20 client.33687.objecter shutdown clearing up homeless session... 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.082+0000 7fbb4df2c700 10 client.33687.objecter successfully canceled tick 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbb3006f5e0 msgr2=0x7fbb30071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbb3006f5e0 0x7fbb30071aa0 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7fbb48075d00 tx=0x7fbb3c01a040 comp rx=0 tx=0).stop 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 msgr2=0x7fbb48074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 0x7fbb48074540 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800d950 tx=0x7fbb3800dc60 comp rx=0 tx=0).stop 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 shutdown_connections 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbb3006f5e0 0x7fbb30071aa0 secure :-1 s=CLOSED pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7fbb48075d00 tx=0x7fbb3c01a040 comp rx=0 tx=0).stop 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb48108390 0x7fbb48074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb48102320 0x7fbb48074540 secure :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800d950 tx=0x7fbb3800dc60 comp rx=0 tx=0).stop 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 --2- 172.21.15.69:0/2888435603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb48102ce0 0x7fbb48074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 >> 172.21.15.69:0/2888435603 conn(0x7fbb48079310 msgr2=0x7fbb48105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:24.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 shutdown_connections 2024-04-02T05:01:24.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 -- 172.21.15.69:0/2888435603 wait complete. 2024-04-02T05:01:24.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:24.086+0000 7fbb4df2c700 1 librados: shutdown 2024-04-02T05:01:24.093 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T05:01:24.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:24.189 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:24.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:24.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:24.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:24.550 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:24.577 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:24.680 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:24.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:24.823 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:24 smithi098 bash[23790]: audit 2024-04-02T05:01:23.650828+0000 mgr.y (mgr.14538) 687 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:24.823 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:24 smithi098 bash[23790]: cluster 2024-04-02T05:01:24.081636+0000 mon.a (mon.0) 3677 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:24.823 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:24 smithi098 bash[23790]: cluster 2024-04-02T05:01:24.081670+0000 mon.a (mon.0) 3678 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:24.823 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:24 smithi098 bash[23790]: audit 2024-04-02T05:01:24.083068+0000 mon.a (mon.0) 3679 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-128", "mode": "writeback"}]': finished 2024-04-02T05:01:24.823 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:24 smithi098 bash[23790]: audit 2024-04-02T05:01:24.083152+0000 mon.a (mon.0) 3680 : audit 1 from='client.? 172.21.15.69:0/2888435603' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:01:24.823 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:24 smithi098 bash[23790]: cluster 2024-04-02T05:01:24.087304+0000 mon.a (mon.0) 3681 : cluster 0 osdmap e774: 8 total, 8 up, 7 in 2024-04-02T05:01:24.840 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:24.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:24.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[28904]: audit 2024-04-02T05:01:23.650828+0000 mgr.y (mgr.14538) 687 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:24.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[28904]: cluster 2024-04-02T05:01:24.081636+0000 mon.a (mon.0) 3677 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:24.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[28904]: cluster 2024-04-02T05:01:24.081670+0000 mon.a (mon.0) 3678 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:24.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[28904]: audit 2024-04-02T05:01:24.083068+0000 mon.a (mon.0) 3679 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-128", "mode": "writeback"}]': finished 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[28904]: audit 2024-04-02T05:01:24.083152+0000 mon.a (mon.0) 3680 : audit 1 from='client.? 172.21.15.69:0/2888435603' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[28904]: cluster 2024-04-02T05:01:24.087304+0000 mon.a (mon.0) 3681 : cluster 0 osdmap e774: 8 total, 8 up, 7 in 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[21528]: audit 2024-04-02T05:01:23.650828+0000 mgr.y (mgr.14538) 687 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[21528]: cluster 2024-04-02T05:01:24.081636+0000 mon.a (mon.0) 3677 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[21528]: cluster 2024-04-02T05:01:24.081670+0000 mon.a (mon.0) 3678 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[21528]: audit 2024-04-02T05:01:24.083068+0000 mon.a (mon.0) 3679 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-128", "mode": "writeback"}]': finished 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[21528]: audit 2024-04-02T05:01:24.083152+0000 mon.a (mon.0) 3680 : audit 1 from='client.? 172.21.15.69:0/2888435603' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:01:24.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:24 smithi069 bash[21528]: cluster 2024-04-02T05:01:24.087304+0000 mon.a (mon.0) 3681 : cluster 0 osdmap e774: 8 total, 8 up, 7 in 2024-04-02T05:01:24.967 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:24.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:25.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:25.093 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:25.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:25.221 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:25.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:25.345 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:25.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:25.472 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:25.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:25.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[21528]: cluster 2024-04-02T05:01:24.312506+0000 mgr.y (mgr.14538) 688 : cluster 0 pgmap v1173: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[21528]: audit 2024-04-02T05:01:24.786211+0000 mon.b (mon.1) 306 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[21528]: cluster 2024-04-02T05:01:25.074332+0000 mon.a (mon.0) 3682 : cluster 0 osdmap e775: 8 total, 8 up, 7 in 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[21528]: audit 2024-04-02T05:01:25.520590+0000 mon.b (mon.1) 307 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[21528]: audit 2024-04-02T05:01:25.522241+0000 mon.b (mon.1) 308 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[21528]: audit 2024-04-02T05:01:25.531331+0000 mon.a (mon.0) 3683 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[28904]: cluster 2024-04-02T05:01:24.312506+0000 mgr.y (mgr.14538) 688 : cluster 0 pgmap v1173: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[28904]: audit 2024-04-02T05:01:24.786211+0000 mon.b (mon.1) 306 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[28904]: cluster 2024-04-02T05:01:25.074332+0000 mon.a (mon.0) 3682 : cluster 0 osdmap e775: 8 total, 8 up, 7 in 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[28904]: audit 2024-04-02T05:01:25.520590+0000 mon.b (mon.1) 307 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:01:25.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[28904]: audit 2024-04-02T05:01:25.522241+0000 mon.b (mon.1) 308 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:01:25.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:25 smithi069 bash[28904]: audit 2024-04-02T05:01:25.531331+0000 mon.a (mon.0) 3683 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:25.782 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:2081 102.f 102:f2af07b9:test-rados-api-smithi069-72349-26::foo:head [writefull 0~5] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:51:58.834944+0000", 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 566.944040126, 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046229852000000002, 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T05:01:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2081 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834944+0000", 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834944+0000", 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834947+0000", 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.834967+0000", 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9167000000000001e-05 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835000+0000", 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3853000000000003e-05 2024-04-02T05:01:25.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835027+0000", 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6682999999999998e-05 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835206+0000", 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000178772 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.835225+0000", 2024-04-02T05:01:25.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.895e-05 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.878478+0000", 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043253536000000002 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.878682+0000", 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203518 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.879050+0000", 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036813500000000001 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.880257+0000", 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001207025 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.880408+0000", 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150982 2024-04-02T05:01:25.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.880998+0000", 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058958899999999996 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.881093+0000", 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5283000000000002e-05 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:51:58.881174+0000", 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1268999999999997e-05 2024-04-02T05:01:25.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:2183 102.f 102:f3bc94da:test-rados-api-smithi069-72349-28::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:52:06.827655+0000", 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 558.95132901399995, 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046062289999999999, 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T05:01:25.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2183 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.827655+0000", 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.827655+0000", 2024-04-02T05:01:25.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.827659+0000", 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9169999999999999e-06 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.827679+0000", 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9346000000000001e-05 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.827723+0000", 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4165999999999997e-05 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.827751+0000", 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8453e-05 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.828012+0000", 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026036199999999998 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.828046+0000", 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3822000000000002e-05 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.866006+0000", 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037960379000000002 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.866117+0000", 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011082399999999999 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.866218+0000", 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010089099999999999 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.867240+0000", 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001022027 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.868862+0000", 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001622489 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.873508+0000", 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046456120000000004 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.873629+0000", 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121373 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:52:06.873718+0000", 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8628999999999995e-05 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.796 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.26387.0:1 272.2s0 272:5455dc94:::oid_IsSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:14.872068+0000", 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 490.90691692299998, 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0932115659999999, 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.26387", 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/49085792", 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T05:01:25.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872068+0000", 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872068+0000", 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872071+0000", 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0230000000000001e-06 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872087+0000", 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.668e-05 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872131+0000", 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4023000000000002e-05 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872158+0000", 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872327+0000", 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016880899999999999 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:14.872333+0000", 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:15.962731+0000", 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.090397858 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:15.963040+0000", 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000309301 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:15.963715+0000", 2024-04-02T05:01:25.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067450799999999999 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:15.964738+0000", 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001023611 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:15.965060+0000", 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032210099999999998 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:15.965279+0000", 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021868899999999999 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16320.0:1 275.5s0 275:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:28.874333+0000", 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 476.90465106900001, 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0516995360000001, 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16320", 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/616633040", 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.874333+0000", 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.874333+0000", 2024-04-02T05:01:25.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.874336+0000", 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.88e-06 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.874346+0000", 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5829999999999996e-06 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.874362+0000", 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5696999999999999e-05 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.874379+0000", 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7119e-05 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.883998+0000", 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096191290000000006 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:01:25.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:28.884003+0000", 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.276e-06 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:29.912596+0000", 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.028592986 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:29.912886+0000", 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028948999999999999 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:29.913613+0000", 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072718800000000003 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:29.915522+0000", 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001908828 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:29.925909+0000", 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010387744000000001 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:29.926033+0000", 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123616 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.25442.0:6480 102.15 e497/394 102:acce2558:test-rados-api-smithi069-72349-66::foo:head v 497'70, mlcod=395'67)", 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:44.947231+0000", 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 340.83175328200002, 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.032025084000000002, 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.947231+0000", 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.947231+0000", 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.947232+0000", 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2e-07 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.947241+0000", 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1239999999999994e-06 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.977955+0000", 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030714600000000002 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.977977+0000", 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1206e-05 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.978127+0000", 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014991700000000001 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.978182+0000", 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5896000000000002e-05 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.979143+0000", 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00096039599999999997 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:44.979256+0000", 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011342500000000001 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.813 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDECSubOpWrite(321.6s1 686/677 ECSubWrite(tid=1550, reqid=client.16773.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=686'1))", 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:44.177561+0000", 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 101.601423393, 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002010062, 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.177561+0000", 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.177561+0000", 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.177563+0000", 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9530000000000002e-06 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.177597+0000", 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3726999999999997e-05 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.177947+0000", 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035050800000000003 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.177986+0000", 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8596000000000001e-05 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.178816+0000", 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00083007800000000004 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.178840+0000", 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4363e-05 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.179500+0000", 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00065945000000000001 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.179571+0000", 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1386999999999999e-05 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3680 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94452570225408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e691)", 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:55.051400+0000", 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 90.727584469999996, 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005337545, 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:01:25.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3680 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.051400+0000", 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.051400+0000", 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.051402+0000", 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0770000000000002e-06 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.051413+0000", 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.049e-05 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.051436+0000", 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3450000000000001e-05 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.051462+0000", 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5778999999999998e-05 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.053236+0000", 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017739819999999999 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.053489+0000", 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025328299999999998 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.056738+0000", 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032484839999999998 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.32028.0:3 324.9 e700/699 324:91bcb6b0:test-rados-api-smithi069-72349-113::foo:head v 700'1, mlcod=0'0)", 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:58.082802+0000", 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 87.696182112000002, 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026441799999999999, 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082802+0000", 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082802+0000", 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082803+0000", 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082815+0000", 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1246999999999999e-05 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082835+0000", 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0279999999999999e-05 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082855+0000", 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9715999999999999e-05 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.082964+0000", 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109342 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.083015+0000", 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1045999999999999e-05 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.085349+0000", 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023343370000000001 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.085447+0000", 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7225999999999994e-05 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.826 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:7 324.0 324:00000000:%01:::head [pgnls start_epoch 702] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e702)", 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:00.060634+0000", 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 85.718350236000006, 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050695089999999998, 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060634+0000", 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060634+0000", 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060636+0000", 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.683e-06 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060644+0000", 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6520000000000006e-06 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060659+0000", 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5637999999999998e-05 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060670+0000", 2024-04-02T05:01:25.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1207e-05 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060796+0000", 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012545500000000001 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.060812+0000", 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6337999999999998e-05 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.065209+0000", 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043971909999999999 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.065275+0000", 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5499000000000004e-05 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.065704+0000", 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042884599999999998 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.4,QREQUEST,e:703,from:3,reservation_nonce:3)", 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:01.795935+0000", 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.983049007000005, 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063419150000000001, 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795935+0000", 2024-04-02T05:01:25.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795935+0000", 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795938+0000", 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3700000000000002e-06 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795950+0000", 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1823e-05 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795981+0000", 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1372999999999999e-05 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795996+0000", 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5255999999999999e-05 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.796142+0000", 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014554499999999999 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.796153+0000", 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1192999999999999e-05 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.802074+0000", 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059212680000000004 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.802120+0000", 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6189000000000001e-05 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.802277+0000", 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015689799999999999 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3731 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94452570225408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:20.069170+0000", 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.709814148999996, 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033063400000000001, 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3731 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069170+0000", 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069170+0000", 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069173+0000", 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.774e-06 2024-04-02T05:01:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069190+0000", 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6784999999999999e-05 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069218+0000", 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7858e-05 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069231+0000", 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3665999999999999e-05 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072244+0000", 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00301214 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072354+0000", 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110158 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072477+0000", 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012295900000000001 2024-04-02T05:01:25.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.2.0:2754 328.16 e740/736 328:6f126094:test-rados-api-smithi069-72349-121::bar:head v 740'1, mlcod=0'0)", 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:50.087227+0000", 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.691757150999997, 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023368389999999998, 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087227+0000", 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087227+0000", 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087228+0000", 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9999999999999997e-07 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087240+0000", 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.19e-05 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087285+0000", 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5451999999999998e-05 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087310+0000", 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.427e-05 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087452+0000", 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014226500000000001 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087525+0000", 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3226000000000007e-05 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.089469+0000", 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019435839999999999 2024-04-02T05:01:25.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.089564+0000", 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5441999999999999e-05 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "pg_update_log_missing(328.16 epoch 740/736 rep_tid 2755 entries 740'2 (0'0) error 328:6f126094:test-rados-api-smithi069-72349-121::bar:head by client.32028.0:476 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:50.087366+0000", 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.691618503000001, 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024529280000000001, 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087366+0000", 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087366+0000", 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087367+0000", 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.407e-06 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087388+0000", 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0740000000000001e-05 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087431+0000", 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3108999999999997e-05 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087469+0000", 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7449e-05 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.087911+0000", 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044250499999999999 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.089819+0000", 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019077180000000001 2024-04-02T05:01:25.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:592 328.6 328:60000000::::head [pgnls start_epoch 742] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e742)", 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:52.156156+0000", 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.622828153999997, 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0087676249999999994, 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:25.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 592 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156156+0000", 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156156+0000", 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156160+0000", 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156174+0000", 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4503e-05 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156209+0000", 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4996999999999997e-05 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156228+0000", 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9035999999999999e-05 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156367+0000", 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013865500000000001 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156381+0000", 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3857e-05 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.164717+0000", 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0083357199999999996 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.164769+0000", 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1996000000000002e-05 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.164924+0000", 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015539800000000001 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDECSubOpWrite(323.0s2 745/698 ECSubWrite(tid=1574, reqid=client.32028.0:646, at_version=0'0, trim_to=0'0, roll_forward_to=745'1))", 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:55.105627+0000", 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.673357754000001, 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013194453, 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105627+0000", 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105627+0000", 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105628+0000", 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1399999999999995e-07 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105641+0000", 2024-04-02T05:01:25.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3354e-05 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105655+0000", 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3621999999999999e-05 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105667+0000", 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2690999999999999e-05 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105729+0000", 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1605000000000004e-05 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105738+0000", 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6430000000000001e-06 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.118765+0000", 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013027828999999999 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.118821+0000", 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5794999999999997e-05 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3812 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94452570225408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e745)", 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:00.101976+0000", 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 25.677008499999999, 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014810310000000001, 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3812 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.101976+0000", 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.101976+0000", 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.101978+0000", 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3800000000000001e-06 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.101989+0000", 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0183000000000001e-05 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.102011+0000", 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2056999999999999e-05 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.102036+0000", 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4905999999999998e-05 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.102951+0000", 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00091555499999999997 2024-04-02T05:01:25.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.103168+0000", 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000216467 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.103457+0000", 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028948299999999999 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.864 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:741 329.1 329:80000000::::head [pgnls start_epoch 750] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e750)", 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:00.194445+0000", 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 25.584539187000001, 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054574519999999998, 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 741 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194445+0000", 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194445+0000", 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194448+0000", 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1200000000000002e-06 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194463+0000", 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4843e-05 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194510+0000", 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6285999999999997e-05 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194522+0000", 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2075e-05 2024-04-02T05:01:25.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194662+0000", 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014064000000000001 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.194685+0000", 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2370000000000001e-05 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.199644+0000", 2024-04-02T05:01:25.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049593759999999997 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.199699+0000", 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4827000000000002e-05 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:00.199903+0000", 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020391499999999999 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(2.2,QREQUEST,e:773,from:1,reservation_nonce:5)", 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:23.185681+0000", 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.593303605, 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033841259999999999, 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185681+0000", 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185681+0000", 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185683+0000", 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5560000000000002e-06 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185699+0000", 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5984000000000001e-05 2024-04-02T05:01:25.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185736+0000", 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6312000000000001e-05 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185754+0000", 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8649999999999999e-05 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185900+0000", 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014553100000000001 2024-04-02T05:01:25.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.185940+0000", 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9826999999999997e-05 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.188969+0000", 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030288210000000001 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.188990+0000", 2024-04-02T05:01:25.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1662e-05 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:23.189065+0000", 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4783000000000006e-05 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1101 332.5 332:a7884da2:test-rados-api-smithi069-72349-129::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e774)", 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:24.087247+0000", 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.691737426, 2024-04-02T05:01:25.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074450489999999996, 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1101 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087247+0000", 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087247+0000", 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087250+0000", 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2629999999999999e-06 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087267+0000", 2024-04-02T05:01:25.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7056999999999999e-05 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087294+0000", 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6957999999999998e-05 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087313+0000", 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8916999999999999e-05 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087424+0000", 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011061399999999999 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087435+0000", 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0937e-05 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.092137+0000", 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047020370000000001 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.092381+0000", 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024452300000000002 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.093460+0000", 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010781829999999999 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.093563+0000", 2024-04-02T05:01:25.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010354299999999999 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.093716+0000", 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015263000000000001 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094200+0000", 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048412800000000001 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:25.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094575+0000", 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037488099999999998 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094612+0000", 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7339000000000003e-05 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094660+0000", 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8415000000000003e-05 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094692+0000", 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1624000000000001e-05 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1143 332.5 332:a7884da2:test-rados-api-smithi069-72349-129::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e774)", 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:24.118429+0000", 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.660555389, 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001449224, 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:25.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1143 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118429+0000", 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118429+0000", 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118430+0000", 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2839999999999999e-06 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118437+0000", 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.55e-06 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118452+0000", 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5167000000000001e-05 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118463+0000", 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0961000000000001e-05 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118540+0000", 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6840000000000003e-05 2024-04-02T05:01:25.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.118632+0000", 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2299000000000005e-05 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.119878+0000", 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001246123 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:25.886 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:01:25.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:25.895 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:25 smithi098 bash[23790]: cluster 2024-04-02T05:01:24.312506+0000 mgr.y (mgr.14538) 688 : cluster 0 pgmap v1173: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:25.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:25 smithi098 bash[23790]: audit 2024-04-02T05:01:24.786211+0000 mon.b (mon.1) 306 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:01:25.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:25 smithi098 bash[23790]: cluster 2024-04-02T05:01:25.074332+0000 mon.a (mon.0) 3682 : cluster 0 osdmap e775: 8 total, 8 up, 7 in 2024-04-02T05:01:25.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:25 smithi098 bash[23790]: audit 2024-04-02T05:01:25.520590+0000 mon.b (mon.1) 307 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:01:25.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:25 smithi098 bash[23790]: audit 2024-04-02T05:01:25.522241+0000 mon.b (mon.1) 308 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:01:25.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:25 smithi098 bash[23790]: audit 2024-04-02T05:01:25.531331+0000 mon.a (mon.0) 3683 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:25.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:25.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:26.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:26.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:26.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:26.166 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:26.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:26.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:26.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:26.530 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:26.556 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:26.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:26.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:26.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-02T05:01:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:26.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:26.938 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:26.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:26.952 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T05:01:27.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:27.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:27.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:27.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:27.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:27.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:27.454 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:27.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:27.641 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:27.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:27.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:27.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:27 smithi069 bash[28904]: cluster 2024-04-02T05:01:26.313332+0000 mgr.y (mgr.14538) 689 : cluster 0 pgmap v1175: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:27.827 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:27 smithi069 bash[21528]: cluster 2024-04-02T05:01:26.313332+0000 mgr.y (mgr.14538) 689 : cluster 0 pgmap v1175: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:27.844 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:27.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:27 smithi098 bash[23790]: cluster 2024-04-02T05:01:26.313332+0000 mgr.y (mgr.14538) 689 : cluster 0 pgmap v1175: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:27.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:28.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:28.100 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:28.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:28.231 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:28.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:28.361 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:28.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:28.491 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:28.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:28.619 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:28.658 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:28.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:28.733 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:28 smithi098 bash[23790]: cluster 2024-04-02T05:01:28.491910+0000 osd.1 (osd.1) 389 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:28.733 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:28 smithi098 bash[23790]: cluster 2024-04-02T05:01:28.492331+0000 osd.1 (osd.1) 390 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:28.750 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:28.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:28.864 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:01:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:01:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:01:28.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:28 smithi069 bash[28904]: cluster 2024-04-02T05:01:28.491910+0000 osd.1 (osd.1) 389 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:28.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:28 smithi069 bash[28904]: cluster 2024-04-02T05:01:28.492331+0000 osd.1 (osd.1) 390 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:28.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:28 smithi069 bash[21528]: cluster 2024-04-02T05:01:28.491910+0000 osd.1 (osd.1) 389 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:28.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:28 smithi069 bash[21528]: cluster 2024-04-02T05:01:28.492331+0000 osd.1 (osd.1) 390 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:28.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:28.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:29.010 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:29.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:29.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:29.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:29.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:29.402 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:29.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:29.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:29.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:29.657 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:29.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:29.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:29 smithi098 bash[23790]: cluster 2024-04-02T05:01:28.315427+0000 mgr.y (mgr.14538) 690 : cluster 0 pgmap v1176: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-02T05:01:29.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:29 smithi098 bash[23790]: cluster 2024-04-02T05:01:28.716542+0000 mon.a (mon.0) 3684 : cluster 0 osdmap e776: 8 total, 8 up, 7 in 2024-04-02T05:01:29.771 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:29 smithi098 bash[23790]: audit 2024-04-02T05:01:29.191664+0000 mon.a (mon.0) 3685 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:29.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:29.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:30.005 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:29 smithi069 bash[21528]: cluster 2024-04-02T05:01:28.315427+0000 mgr.y (mgr.14538) 690 : cluster 0 pgmap v1176: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-02T05:01:30.005 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:29 smithi069 bash[21528]: cluster 2024-04-02T05:01:28.716542+0000 mon.a (mon.0) 3684 : cluster 0 osdmap e776: 8 total, 8 up, 7 in 2024-04-02T05:01:30.005 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:29 smithi069 bash[21528]: audit 2024-04-02T05:01:29.191664+0000 mon.a (mon.0) 3685 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:30.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:29 smithi069 bash[28904]: cluster 2024-04-02T05:01:28.315427+0000 mgr.y (mgr.14538) 690 : cluster 0 pgmap v1176: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-02T05:01:30.005 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:29 smithi069 bash[28904]: cluster 2024-04-02T05:01:28.716542+0000 mon.a (mon.0) 3684 : cluster 0 osdmap e776: 8 total, 8 up, 7 in 2024-04-02T05:01:30.006 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:29 smithi069 bash[28904]: audit 2024-04-02T05:01:29.191664+0000 mon.a (mon.0) 3685 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:30.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:30.039 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:30.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:30.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:30.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:30.301 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:30.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:30.431 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:30.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:30.671 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:30.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:30.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:30.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:31.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:30 smithi069 bash[21528]: audit 2024-04-02T05:01:29.719496+0000 mon.a (mon.0) 3686 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:31.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:30 smithi069 bash[21528]: cluster 2024-04-02T05:01:29.724371+0000 mon.a (mon.0) 3687 : cluster 0 osdmap e777: 8 total, 8 up, 7 in 2024-04-02T05:01:31.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:30 smithi069 bash[21528]: audit 2024-04-02T05:01:29.724980+0000 mon.a (mon.0) 3688 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128"}]: dispatch 2024-04-02T05:01:31.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:30 smithi069 bash[28904]: audit 2024-04-02T05:01:29.719496+0000 mon.a (mon.0) 3686 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:31.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:30 smithi069 bash[28904]: cluster 2024-04-02T05:01:29.724371+0000 mon.a (mon.0) 3687 : cluster 0 osdmap e777: 8 total, 8 up, 7 in 2024-04-02T05:01:31.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:30 smithi069 bash[28904]: audit 2024-04-02T05:01:29.724980+0000 mon.a (mon.0) 3688 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128"}]: dispatch 2024-04-02T05:01:31.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:31.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:30 smithi098 bash[23790]: audit 2024-04-02T05:01:29.719496+0000 mon.a (mon.0) 3686 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:31.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:30 smithi098 bash[23790]: cluster 2024-04-02T05:01:29.724371+0000 mon.a (mon.0) 3687 : cluster 0 osdmap e777: 8 total, 8 up, 7 in 2024-04-02T05:01:31.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:30 smithi098 bash[23790]: audit 2024-04-02T05:01:29.724980+0000 mon.a (mon.0) 3688 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128"}]: dispatch 2024-04-02T05:01:31.059 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:31.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:31.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:31.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:31.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:31.558 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:01:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:31.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:31.710 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:31.813 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:31.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:31.963 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:31.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:32.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.091 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:32.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:32.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:32 smithi098 bash[23790]: cluster 2024-04-02T05:01:30.316227+0000 mgr.y (mgr.14538) 691 : cluster 0 pgmap v1179: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-04-02T05:01:32.205 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:32 smithi098 bash[23790]: cluster 2024-04-02T05:01:30.720237+0000 mon.a (mon.0) 3689 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:32.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:32 smithi098 bash[23790]: audit 2024-04-02T05:01:30.728165+0000 mon.a (mon.0) 3690 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128"}]': finished 2024-04-02T05:01:32.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:32 smithi098 bash[23790]: cluster 2024-04-02T05:01:30.740966+0000 mon.a (mon.0) 3691 : cluster 0 osdmap e778: 8 total, 8 up, 7 in 2024-04-02T05:01:32.221 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:32.233 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:32.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:32.351 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:32.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:32.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[21528]: cluster 2024-04-02T05:01:30.316227+0000 mgr.y (mgr.14538) 691 : cluster 0 pgmap v1179: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[21528]: cluster 2024-04-02T05:01:30.720237+0000 mon.a (mon.0) 3689 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[21528]: audit 2024-04-02T05:01:30.728165+0000 mon.a (mon.0) 3690 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128"}]': finished 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[21528]: cluster 2024-04-02T05:01:30.740966+0000 mon.a (mon.0) 3691 : cluster 0 osdmap e778: 8 total, 8 up, 7 in 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[28904]: cluster 2024-04-02T05:01:30.316227+0000 mgr.y (mgr.14538) 691 : cluster 0 pgmap v1179: 224 pgs: 224 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[28904]: cluster 2024-04-02T05:01:30.720237+0000 mon.a (mon.0) 3689 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[28904]: audit 2024-04-02T05:01:30.728165+0000 mon.a (mon.0) 3690 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-128"}]': finished 2024-04-02T05:01:32.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:32 smithi069 bash[28904]: cluster 2024-04-02T05:01:30.740966+0000 mon.a (mon.0) 3691 : cluster 0 osdmap e778: 8 total, 8 up, 7 in 2024-04-02T05:01:32.480 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:32.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:32.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:32.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:32.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:32.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:33.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:33.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:33.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:33.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:33 smithi098 bash[23790]: cluster 2024-04-02T05:01:31.734940+0000 mon.a (mon.0) 3692 : cluster 0 osdmap e779: 8 total, 8 up, 7 in 2024-04-02T05:01:33.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:33 smithi098 bash[23790]: cluster 2024-04-02T05:01:32.316684+0000 mgr.y (mgr.14538) 692 : cluster 0 pgmap v1182: 192 pgs: 192 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:33.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:33 smithi098 bash[23790]: cluster 2024-04-02T05:01:32.744956+0000 mon.a (mon.0) 3693 : cluster 0 osdmap e780: 8 total, 8 up, 7 in 2024-04-02T05:01:33.362 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:33 smithi098 bash[23790]: audit 2024-04-02T05:01:32.774569+0000 mon.a (mon.0) 3694 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:33.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:33.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[21528]: cluster 2024-04-02T05:01:31.734940+0000 mon.a (mon.0) 3692 : cluster 0 osdmap e779: 8 total, 8 up, 7 in 2024-04-02T05:01:33.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[21528]: cluster 2024-04-02T05:01:32.316684+0000 mgr.y (mgr.14538) 692 : cluster 0 pgmap v1182: 192 pgs: 192 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:33.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[21528]: cluster 2024-04-02T05:01:32.744956+0000 mon.a (mon.0) 3693 : cluster 0 osdmap e780: 8 total, 8 up, 7 in 2024-04-02T05:01:33.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[21528]: audit 2024-04-02T05:01:32.774569+0000 mon.a (mon.0) 3694 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:33.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[28904]: cluster 2024-04-02T05:01:31.734940+0000 mon.a (mon.0) 3692 : cluster 0 osdmap e779: 8 total, 8 up, 7 in 2024-04-02T05:01:33.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[28904]: cluster 2024-04-02T05:01:32.316684+0000 mgr.y (mgr.14538) 692 : cluster 0 pgmap v1182: 192 pgs: 192 active+clean; 582 KiB data, 722 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:33.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[28904]: cluster 2024-04-02T05:01:32.744956+0000 mon.a (mon.0) 3693 : cluster 0 osdmap e780: 8 total, 8 up, 7 in 2024-04-02T05:01:33.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:33 smithi069 bash[28904]: audit 2024-04-02T05:01:32.774569+0000 mon.a (mon.0) 3694 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:33.508 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:33.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:33.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:33.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:33.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:01:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:01:33.906 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:33.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 -- 172.21.15.69:0/2959111297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 msgr2=0x7f3bc0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 --2- 172.21.15.69:0/2959111297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0101620 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009990 tx=0x7f3bb4009ca0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 -- 172.21.15.69:0/2959111297 shutdown_connections 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 --2- 172.21.15.69:0/2959111297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bc0101b60 0x7f3bc010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 --2- 172.21.15.69:0/2959111297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc0107220 0x7f3bc0107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 --2- 172.21.15.69:0/2959111297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0101620 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 -- 172.21.15.69:0/2959111297 >> 172.21.15.69:0/2959111297 conn(0x7f3bc00771e0 msgr2=0x7f3bc00775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 -- 172.21.15.69:0/2959111297 shutdown_connections 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.002+0000 7f3bc6406700 1 -- 172.21.15.69:0/2959111297 wait complete. 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 librados: starting msgr at 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 librados: starting objecter 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 Processor -- start 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- start start 2024-04-02T05:01:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 librados: setting wanted keys 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 librados: calling monclient init 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc0101b60 0x7f3bc0108520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bc0107220 0x7f3bc0108a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc0110930 con 0x7f3bc0107220 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01107b0 con 0x7f3bc0101b60 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc0110ab0 con 0x7f3bc01011a0 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60266/0 (socket says 172.21.15.69:60266) 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 -- 172.21.15.69:0/3011143923 learned_addr learned my addr 172.21.15.69:0/3011143923 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bc0107220 0x7f3bc0108a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc0101b60 0x7f3bc0108520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:34.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc0101b60 msgr2=0x7f3bc0108520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc0101b60 0x7f3bc0108520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bc0107220 msgr2=0x7f3bc0108a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bc0107220 0x7f3bc0108a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc01aa910 con 0x7f3bc01011a0 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bc0101b60 0x7f3bc0108520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0107fe0 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000e9f0 tx=0x7f3bb000edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 10 client.?.objecter ms_handle_connect 0x7f3bc01011a0 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb000cbd0 con 0x7f3bc01011a0 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3bb000cd70 con 0x7f3bc01011a0 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc01aac00 con 0x7f3bc01011a0 2024-04-02T05:01:34.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0005520 con 0x7f3bc01011a0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 10 client.43757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc01ab140 con 0x7f3bc01011a0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 librados: init done 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3bc0102860 con 0x7f3bc01011a0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3bb0004750 con 0x7f3bc01011a0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.006+0000 7f3bbd7fa700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3ba806f5e0 0x7f3ba8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(781..781 src has 74..781) ==== 10084+0+0 (secure 0 0 0) 0x7f3bb008ebf0 con 0x7f3bc01011a0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 osd_map(781..781 src has 74..781) 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 3 client.43757.objecter handle_osd_map got epochs [781,781] > 0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 3 client.43757.objecter handle_osd_map decoding full epoch 781 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3ba806f5e0 0x7f3ba8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 20 client.43757.objecter dump_active .. 0 homeless 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v781) ==== 54+0+16 (secure 0 0 0) 0x7f3bb0094240 con 0x7f3bc01011a0 2024-04-02T05:01:34.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3ba806f5e0 0x7f3ba8071aa0 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400b2e0 tx=0x7f3bb401a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3ba806f5e0 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 10 client.43757.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 1 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3bc0062e50 0x7f3bc0063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 1 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 0 1 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 4 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 10 client.43757.objecter _send_command 1 2024-04-02T05:01:34.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bc0109510 con 0x7f3bc0062e50 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3bc0062e50 0x7f3bc0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc0062e50 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3bc0062e50 0x7f3bc0063230 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc0109510 con 0x7f3bc0062e50 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 1 = system:0 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 4 2024-04-02T05:01:34.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.010+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 0 1 2024-04-02T05:01:34.030 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 10 client.43757.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 2 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 2 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc004ff00 osd=0 3 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 0 2 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 4 2024-04-02T05:01:34.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 10 client.43757.objecter _send_command 2 2024-04-02T05:01:34.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc00628d0 con 0x7f3bc0062e50 2024-04-02T05:01:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:34.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc00628d0 con 0x7f3bc0062e50 2024-04-02T05:01:34.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 4 2024-04-02T05:01:34.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 0 2 2024-04-02T05:01:34.034 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:01:34.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 10 client.43757.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 3 2024-04-02T05:01:34.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3bc0104a40 0x7f3bc01af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 1 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 4 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 10 client.43757.objecter _send_command 3 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3bc002dca0 con 0x7f3bc0104a40 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3bc0104a40 0x7f3bc01af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc0104a40 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3bc0104a40 0x7f3bc01af390 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc002dca0 con 0x7f3bc0104a40 2024-04-02T05:01:34.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 3: 0 ) 2024-04-02T05:01:34.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 3 = system:0 2024-04-02T05:01:34.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 4 2024-04-02T05:01:34.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.030+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 1 3 2024-04-02T05:01:34.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 10 client.43757.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 4 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 4 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc002db30 osd=1 3 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 1 4 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 4 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 10 client.43757.objecter _send_command 4 2024-04-02T05:01:34.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc01af8d0 con 0x7f3bc0104a40 2024-04-02T05:01:34.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc01af8d0 con 0x7f3bc0104a40 2024-04-02T05:01:34.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 4 2024-04-02T05:01:34.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.050+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 1 4 2024-04-02T05:01:34.055 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:01:34.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 10 client.43757.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 5 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3bc0104120 0x7f3bc01b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 5 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 2 5 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 4 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 10 client.43757.objecter _send_command 5 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3bc002d860 con 0x7f3bc0104120 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3bc0104120 0x7f3bc01b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc0104120 2024-04-02T05:01:34.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3bc0104120 0x7f3bc01b2c10 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc002d860 con 0x7f3bc0104120 2024-04-02T05:01:34.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 5: 0 ) 2024-04-02T05:01:34.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 5 = system:0 2024-04-02T05:01:34.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 4 2024-04-02T05:01:34.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.054+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 2 5 2024-04-02T05:01:34.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 10 client.43757.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 6 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 6 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc0103fb0 osd=2 3 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 2 6 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 4 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 10 client.43757.objecter _send_command 6 2024-04-02T05:01:34.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc002cc00 con 0x7f3bc0104120 2024-04-02T05:01:34.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc002cc00 con 0x7f3bc0104120 2024-04-02T05:01:34.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 4 2024-04-02T05:01:34.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 2 6 2024-04-02T05:01:34.079 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:01:34.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 10 client.43757.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 7 2024-04-02T05:01:34.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3bc01b42d0 0x7f3bc01b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 7 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 3 7 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 4 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 10 client.43757.objecter _send_command 7 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.074+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3bc01b9520 con 0x7f3bc01b42d0 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3bc01b42d0 0x7f3bc01b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc01b42d0 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3bc01b42d0 0x7f3bc01b66c0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc01b9520 con 0x7f3bc01b42d0 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 7: 0 ) 2024-04-02T05:01:34.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 7 = system:0 2024-04-02T05:01:34.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 4 2024-04-02T05:01:34.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.078+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 3 7 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.094+0000 7f3bc6406700 10 client.43757.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 8 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 8 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01b0250 osd=3 3 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 3 8 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 4 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 10 client.43757.objecter _send_command 8 2024-04-02T05:01:34.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc01afcd0 con 0x7f3bc01b42d0 2024-04-02T05:01:34.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc01afcd0 con 0x7f3bc01b42d0 2024-04-02T05:01:34.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 4 2024-04-02T05:01:34.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 3 8 2024-04-02T05:01:34.102 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:01:34.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 10 client.43757.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 9 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3bc01b3e30 0x7f3bc01b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 9 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 4 9 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 4 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 10 client.43757.objecter _send_command 9 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.098+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3bc01afbf0 con 0x7f3bc01b3e30 2024-04-02T05:01:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3bc01b3e30 0x7f3bc01b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3bc01b3e30 0x7f3bc01b7d80 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc01b3e30 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc01afbf0 con 0x7f3bc01b3e30 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 9: 0 ) 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 9 = system:0 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 4 2024-04-02T05:01:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.102+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 4 9 2024-04-02T05:01:34.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 10 client.43757.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 10 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 10 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01afa80 osd=4 3 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 4 10 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 4 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 10 client.43757.objecter _send_command 10 2024-04-02T05:01:34.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.118+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc01afbf0 con 0x7f3bc01b3e30 2024-04-02T05:01:34.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc01afbf0 con 0x7f3bc01b3e30 2024-04-02T05:01:34.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 4 2024-04-02T05:01:34.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 4 10 2024-04-02T05:01:34.126 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 10 client.43757.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 11 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3bc01b87d0 0x7f3bc01b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 11 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 5 11 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 4 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 10 client.43757.objecter _send_command 11 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3bc01c0010 con 0x7f3bc01b87d0 2024-04-02T05:01:34.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.122+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3bc01b87d0 0x7f3bc01b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc01b87d0 2024-04-02T05:01:34.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbf7fe700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3bc01b87d0 0x7f3bc01b8c50 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc01c0010 con 0x7f3bc01b87d0 2024-04-02T05:01:34.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 11: 0 ) 2024-04-02T05:01:34.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 11 = system:0 2024-04-02T05:01:34.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 4 2024-04-02T05:01:34.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.126+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 5 11 2024-04-02T05:01:34.133 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:34.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 10 client.43757.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 12 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 12 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01afbf0 osd=5 3 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 5 12 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 4 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 10 client.43757.objecter _send_command 12 2024-04-02T05:01:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.146+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc01bd6f0 con 0x7f3bc01b87d0 2024-04-02T05:01:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc01bd6f0 con 0x7f3bc01b87d0 2024-04-02T05:01:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 4 2024-04-02T05:01:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 5 12 2024-04-02T05:01:34.154 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:01:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 10 client.43757.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 13 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3bc01bdd70 0x7f3bc01c0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 13 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 6 13 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 4 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 10 client.43757.objecter _send_command 13 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.150+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3bc01c31a0 con 0x7f3bc01bdd70 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3bc01bdd70 0x7f3bc01c0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bc49a3700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3bc01bdd70 0x7f3bc01c0160 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc01bdd70 2024-04-02T05:01:34.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc01c31a0 con 0x7f3bc01bdd70 2024-04-02T05:01:34.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 13: 0 ) 2024-04-02T05:01:34.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 13 = system:0 2024-04-02T05:01:34.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 4 2024-04-02T05:01:34.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.154+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 6 13 2024-04-02T05:01:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 10 client.43757.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 14 2024-04-02T05:01:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 14 2024-04-02T05:01:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01bd6f0 osd=6 3 2024-04-02T05:01:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 6 14 2024-04-02T05:01:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 4 2024-04-02T05:01:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 10 client.43757.objecter _send_command 14 2024-04-02T05:01:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.174+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc01c31c0 con 0x7f3bc01bdd70 2024-04-02T05:01:34.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bb801d2b0 con 0x7f3bc01bdd70 2024-04-02T05:01:34.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 4 2024-04-02T05:01:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 6 14 2024-04-02T05:01:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 10 client.43757.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 15 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f3bc01c2790 0x7f3bc01c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 15 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 7 15 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 4 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 10 client.43757.objecter _send_command 15 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3bc01c2c50 con 0x7f3bc01c2790 2024-04-02T05:01:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f3bc01c2790 0x7f3bc01c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbd7fa700 10 client.43757.objecter ms_handle_connect 0x7f3bc01c2790 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.178+0000 7f3bbffff700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f3bc01c2790 0x7f3bc01c07f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.182+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bc01c2c50 con 0x7f3bc01c2790 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.182+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 15: 0 ) 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.182+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 15 = system:0 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.182+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 4 2024-04-02T05:01:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.182+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 7 15 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 10 client.43757.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign -1 16 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 20 client.43757.objecter _get_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_remove -1 16 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 20 client.43757.objecter get_session s=0x7f3bc01afee0 osd=7 3 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 15 client.43757.objecter _session_command_op_assign 7 16 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 4 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 10 client.43757.objecter _send_command 16 2024-04-02T05:01:34.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.198+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3bc01c2c50 con 0x7f3bc01c2790 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- 172.21.15.69:0/4010921435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff438102cb0 msgr2=0x7ff43810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- 172.21.15.69:0/4010921435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff438102cb0 0x7ff43810f120 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7ff43000b0d0 tx=0x7ff43000b3e0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- 172.21.15.69:0/4010921435 shutdown_connections 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- 172.21.15.69:0/4010921435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff438102cb0 0x7ff43810f120 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- 172.21.15.69:0/4010921435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff438108300 0x7ff4381086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- 172.21.15.69:0/4010921435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4381022f0 0x7ff438102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- 172.21.15.69:0/4010921435 >> 172.21.15.69:0/4010921435 conn(0x7ff4380793a0 msgr2=0x7ff4380797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- 172.21.15.69:0/4010921435 shutdown_connections 2024-04-02T05:01:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- 172.21.15.69:0/4010921435 wait complete. 2024-04-02T05:01:34.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bbd7fa700 1 -- 172.21.15.69:0/3011143923 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3bc01c2c50 con 0x7f3bc01c2790 2024-04-02T05:01:34.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bbd7fa700 10 client.43757.objecter ms_dispatch 0x7f3bc0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:34.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bbd7fa700 10 client.43757.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:34.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bbd7fa700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 4 2024-04-02T05:01:34.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bbd7fa700 15 client.43757.objecter _session_command_op_remove 7 16 2024-04-02T05:01:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:01:34.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 librados: watch_flush enter 2024-04-02T05:01:34.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 librados: starting msgr at 2024-04-02T05:01:34.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 librados: starting objecter 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 librados: watch_flush exit 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.0 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3bc0062e50 msgr2=0x7f3bc0063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3bc0062e50 0x7f3bc0063230 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc004ff00 osd=0 1 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.1 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3bc0104a40 msgr2=0x7f3bc01af390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f3bc0104a40 0x7f3bc01af390 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc002db30 osd=1 1 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.2 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3bc0104120 msgr2=0x7f3bc01b2c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f3bc0104120 0x7f3bc01b2c10 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc0103fb0 osd=2 1 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.3 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3bc01b42d0 msgr2=0x7f3bc01b66c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f3bc01b42d0 0x7f3bc01b66c0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01b0250 osd=3 1 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.4 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3bc01b3e30 msgr2=0x7f3bc01b7d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f3bc01b3e30 0x7f3bc01b7d80 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afa80 osd=4 1 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.5 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3bc01b87d0 msgr2=0x7f3bc01b8c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f3bc01b87d0 0x7f3bc01b8c50 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc49a3700 1 -- 172.21.15.69:0/3011143923 reap_dead start 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc49a3700 1 -- 172.21.15.69:0/3011143923 reap_dead start 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afbf0 osd=5 1 2024-04-02T05:01:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.6 2024-04-02T05:01:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 Processor -- start 2024-04-02T05:01:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- start start 2024-04-02T05:01:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 librados: setting wanted keys 2024-04-02T05:01:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 librados: calling monclient init 2024-04-02T05:01:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3bc01bdd70 msgr2=0x7f3bc01c0160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc49a3700 1 -- 172.21.15.69:0/3011143923 reap_dead start 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3bc01bdd70 0x7f3bc01c0160 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01bd6f0 osd=6 1 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter close_session for osd.7 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f3bc01c2790 msgr2=0x7f3bc01c07f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f3bc01c2790 0x7f3bc01c07f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter put_session s=0x7f3bc01afee0 osd=7 1 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 20 client.43757.objecter shutdown clearing up homeless session... 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 10 client.43757.objecter successfully canceled tick 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3ba806f5e0 msgr2=0x7f3ba8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3ba806f5e0 0x7f3ba8071aa0 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400b2e0 tx=0x7f3bb401a040 comp rx=0 tx=0).stop 2024-04-02T05:01:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4381022f0 0x7ff43806c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438102cb0 0x7ff43806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 msgr2=0x7f3bc0107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0107fe0 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000e9f0 tx=0x7f3bb000edb0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff438108300 0x7ff4381ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff438111ae0 con 0x7ff4381022f0 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff438111960 con 0x7ff438108300 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43e2a1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff438111c60 con 0x7ff438102cb0 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 shutdown_connections 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f3bc01c2790 0x7f3bc01c07f0 crc :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f3bc01bdd70 0x7f3bc01c0160 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bc01011a0 0x7f3bc0107fe0 secure :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000e9f0 tx=0x7f3bb000edb0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 --2- 172.21.15.69:0/3011143923 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f3ba806f5e0 0x7f3ba8071aa0 unknown :-1 s=CLOSED pgs=4006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 >> 172.21.15.69:0/3011143923 conn(0x7f3bc00771e0 msgr2=0x7f3bc00fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff4377fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438102cb0 0x7ff43806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff4377fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438102cb0 0x7ff43806cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60296/0 (socket says 172.21.15.69:60296) 2024-04-02T05:01:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff4377fe700 1 -- 172.21.15.69:0/2463802501 learned_addr learned my addr 172.21.15.69:0/2463802501 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff437fff700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4381022f0 0x7ff43806c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7ff43c83e700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff438108300 0x7ff4381ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.202+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 shutdown_connections 2024-04-02T05:01:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7f3bc6406700 1 -- 172.21.15.69:0/3011143923 wait complete. 2024-04-02T05:01:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7f3bc6406700 1 librados: shutdown 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4377fe700 1 -- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff438108300 msgr2=0x7ff4381ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4377fe700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff438108300 0x7ff4381ac510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4377fe700 1 -- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4381022f0 msgr2=0x7ff43806c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4377fe700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4381022f0 0x7ff43806c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4377fe700 1 -- 172.21.15.69:0/2463802501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4381acad0 con 0x7ff438102cb0 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff437fff700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4381022f0 0x7ff43806c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43c83e700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff438108300 0x7ff4381ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4377fe700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438102cb0 0x7ff43806cc40 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7ff42800e9c0 tx=0x7ff42800ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 10 client.?.objecter ms_handle_connect 0x7ff438102cb0 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:34.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42800cbd0 con 0x7ff438102cb0 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff42800cd70 con 0x7ff438102cb0 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4280055e0 con 0x7ff438102cb0 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 1 -- 172.21.15.69:0/2463802501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4380744d0 con 0x7ff438102cb0 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 10 client.43763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 1 -- 172.21.15.69:0/2463802501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4380749c0 con 0x7ff438102cb0 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 1 librados: init done 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:01:34.217 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:01:34.218 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:01:34.218 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:01:34.218 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 10 client.43763.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 15 client.43763.objecter _session_command_op_assign -1 1 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 20 client.43763.objecter _get_session osd=-1 returning homeless 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 10 client.43763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43e2a1700 1 -- 172.21.15.69:0/2463802501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4381039b0 con 0x7ff438102cb0 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff428004750 con 0x7ff438102cb0 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff42006f5e0 0x7ff420071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(781..781 src has 74..781) ==== 10084+0+0 (secure 0 0 0) 0x7ff42808f360 con 0x7ff438102cb0 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 10 client.43763.objecter ms_dispatch 0x7ff438001550 osd_map(781..781 src has 74..781) 2024-04-02T05:01:34.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 3 client.43763.objecter handle_osd_map got epochs [781,781] > 0 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 3 client.43763.objecter handle_osd_map decoding full epoch 781 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff437fff700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff42006f5e0 0x7ff420071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 10 client.43763.objecter checking command 1 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7ff42007d2d0 0x7ff42007f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 20 client.43763.objecter _get_session s=0x7ff42007cc00 osd=4 3 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 20 client.43763.objecter put_session s=0x7ff42007cc00 osd=4 3 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 15 client.43763.objecter _session_command_op_remove -1 1 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 20 client.43763.objecter _get_session s=0x7ff42007cc00 osd=4 3 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 20 client.43763.objecter get_session s=0x7ff42007cc00 osd=4 3 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 15 client.43763.objecter _session_command_op_assign 4 1 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 20 client.43763.objecter put_session s=0x7ff42007cc00 osd=4 4 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 10 client.43763.objecter _send_command 1 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff428004500 con 0x7ff42007d2d0 2024-04-02T05:01:34.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 20 client.43763.objecter dump_active .. 0 homeless 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=781) ==== 24+0+0 (secure 0 0 0) 0x7ff42808f7b0 con 0x7ff438102cb0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff4357fa700 10 client.43763.objecter ms_handle_connect 0x7ff42006f5e0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff437fff700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff42006f5e0 0x7ff420071aa0 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0099d0 tx=0x7ff42c008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.206+0000 7ff43c83e700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7ff42007d2d0 0x7ff42007f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff4357fa700 10 client.43763.objecter ms_handle_connect 0x7ff42007d2d0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff43c83e700 1 --2- 172.21.15.69:0/2463802501 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7ff42007d2d0 0x7ff42007f770 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff4357fa700 1 -- 172.21.15.69:0/2463802501 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff428004500 con 0x7ff42007d2d0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff4357fa700 10 client.43763.objecter ms_dispatch 0x7ff438001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff4357fa700 10 client.43763.objecter _finish_command 1 = system:0 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff4357fa700 20 client.43763.objecter put_session s=0x7ff42007cc00 osd=4 4 2024-04-02T05:01:34.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:34.210+0000 7ff4357fa700 15 client.43763.objecter _session_command_op_remove 4 1 2024-04-02T05:01:34.247 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:01:34.247 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:01:34.248 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:01:34.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:34.282 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:34.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:34.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:34.414 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:34.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:34.544 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:34.647 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:34.778 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: audit 2024-04-02T05:01:33.658794+0000 mgr.y (mgr.14538) 693 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:34.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: audit 2024-04-02T05:01:33.737777+0000 mon.a (mon.0) 3695 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: cluster 2024-04-02T05:01:33.746876+0000 mon.a (mon.0) 3696 : cluster 0 osdmap e781: 8 total, 8 up, 7 in 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: audit 2024-04-02T05:01:33.761374+0000 mon.a (mon.0) 3697 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: audit 2024-04-02T05:01:33.985387+0000 mon.a (mon.0) 3698 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: audit 2024-04-02T05:01:33.985472+0000 mon.b (mon.1) 309 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[21528]: audit 2024-04-02T05:01:34.011370+0000 mon.c (mon.2) 1157 : audit 0 from='client.? 172.21.15.69:0/3011143923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: audit 2024-04-02T05:01:33.658794+0000 mgr.y (mgr.14538) 693 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: audit 2024-04-02T05:01:33.737777+0000 mon.a (mon.0) 3695 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: cluster 2024-04-02T05:01:33.746876+0000 mon.a (mon.0) 3696 : cluster 0 osdmap e781: 8 total, 8 up, 7 in 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: audit 2024-04-02T05:01:33.761374+0000 mon.a (mon.0) 3697 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: audit 2024-04-02T05:01:33.985387+0000 mon.a (mon.0) 3698 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:34.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: audit 2024-04-02T05:01:33.985472+0000 mon.b (mon.1) 309 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:34.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:34 smithi069 bash[28904]: audit 2024-04-02T05:01:34.011370+0000 mon.c (mon.2) 1157 : audit 0 from='client.? 172.21.15.69:0/3011143923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:34.806 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:34.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:34.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:35.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: audit 2024-04-02T05:01:33.658794+0000 mgr.y (mgr.14538) 693 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: audit 2024-04-02T05:01:33.737777+0000 mon.a (mon.0) 3695 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: cluster 2024-04-02T05:01:33.746876+0000 mon.a (mon.0) 3696 : cluster 0 osdmap e781: 8 total, 8 up, 7 in 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: audit 2024-04-02T05:01:33.761374+0000 mon.a (mon.0) 3697 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: audit 2024-04-02T05:01:33.985387+0000 mon.a (mon.0) 3698 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: audit 2024-04-02T05:01:33.985472+0000 mon.b (mon.1) 309 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:35.050 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:34 smithi098 bash[23790]: audit 2024-04-02T05:01:34.011370+0000 mon.c (mon.2) 1157 : audit 0 from='client.? 172.21.15.69:0/3011143923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:35.063 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:35.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:35.193 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:35.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:35.315 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:35.385 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T05:01:35.386 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:01:35.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:35.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:35.672 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:35.700 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:35.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:35.833 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:01:35.915 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:01:35.915 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 'trim stale osdmaps' 2024-04-02T05:01:35.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:35.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: cluster 2024-04-02T05:01:34.317360+0000 mgr.y (mgr.14538) 694 : cluster 0 pgmap v1185: 224 pgs: 27 creating+peering, 197 active+clean; 582 KiB data, 727 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: audit 2024-04-02T05:01:34.740157+0000 mon.a (mon.0) 3699 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: cluster 2024-04-02T05:01:34.748863+0000 mon.a (mon.0) 3700 : cluster 0 osdmap e782: 8 total, 8 up, 7 in 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: audit 2024-04-02T05:01:34.753008+0000 mon.a (mon.0) 3701 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-130"}]: dispatch 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: audit 2024-04-02T05:01:35.072411+0000 mon.a (mon.0) 3702 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-130"}]': finished 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: cluster 2024-04-02T05:01:35.082485+0000 mon.a (mon.0) 3703 : cluster 0 osdmap e783: 8 total, 8 up, 7 in 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[21528]: audit 2024-04-02T05:01:35.087376+0000 mon.a (mon.0) 3704 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-130", "mode": "writeback"}]: dispatch 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: cluster 2024-04-02T05:01:34.317360+0000 mgr.y (mgr.14538) 694 : cluster 0 pgmap v1185: 224 pgs: 27 creating+peering, 197 active+clean; 582 KiB data, 727 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: audit 2024-04-02T05:01:34.740157+0000 mon.a (mon.0) 3699 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: cluster 2024-04-02T05:01:34.748863+0000 mon.a (mon.0) 3700 : cluster 0 osdmap e782: 8 total, 8 up, 7 in 2024-04-02T05:01:35.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: audit 2024-04-02T05:01:34.753008+0000 mon.a (mon.0) 3701 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-130"}]: dispatch 2024-04-02T05:01:35.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: audit 2024-04-02T05:01:35.072411+0000 mon.a (mon.0) 3702 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-130"}]': finished 2024-04-02T05:01:35.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: cluster 2024-04-02T05:01:35.082485+0000 mon.a (mon.0) 3703 : cluster 0 osdmap e783: 8 total, 8 up, 7 in 2024-04-02T05:01:35.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:35 smithi069 bash[28904]: audit 2024-04-02T05:01:35.087376+0000 mon.a (mon.0) 3704 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-130", "mode": "writeback"}]: dispatch 2024-04-02T05:01:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:35.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:36.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:36.097 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:36.200 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:36.223 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: cluster 2024-04-02T05:01:34.317360+0000 mgr.y (mgr.14538) 694 : cluster 0 pgmap v1185: 224 pgs: 27 creating+peering, 197 active+clean; 582 KiB data, 727 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: audit 2024-04-02T05:01:34.740157+0000 mon.a (mon.0) 3699 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: cluster 2024-04-02T05:01:34.748863+0000 mon.a (mon.0) 3700 : cluster 0 osdmap e782: 8 total, 8 up, 7 in 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: audit 2024-04-02T05:01:34.753008+0000 mon.a (mon.0) 3701 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-130"}]: dispatch 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: audit 2024-04-02T05:01:35.072411+0000 mon.a (mon.0) 3702 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-130"}]': finished 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: cluster 2024-04-02T05:01:35.082485+0000 mon.a (mon.0) 3703 : cluster 0 osdmap e783: 8 total, 8 up, 7 in 2024-04-02T05:01:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:35 smithi098 bash[23790]: audit 2024-04-02T05:01:35.087376+0000 mon.a (mon.0) 3704 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-130", "mode": "writeback"}]: dispatch 2024-04-02T05:01:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:36.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-02T05:01:36.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:36.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:36.484 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:36.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:36.614 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:36.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:36.745 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:36.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:36.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:36.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:36.990 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:36 smithi098 bash[23790]: cluster 2024-04-02T05:01:36.072769+0000 mon.a (mon.0) 3705 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:36.990 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:36 smithi098 bash[23790]: audit 2024-04-02T05:01:36.076077+0000 mon.a (mon.0) 3706 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-130", "mode": "writeback"}]': finished 2024-04-02T05:01:36.990 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:36 smithi098 bash[23790]: cluster 2024-04-02T05:01:36.086998+0000 mon.a (mon.0) 3707 : cluster 0 osdmap e784: 8 total, 8 up, 7 in 2024-04-02T05:01:36.990 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:36 smithi098 bash[23790]: audit 2024-04-02T05:01:36.217993+0000 mon.a (mon.0) 3708 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:37.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:37.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[21528]: cluster 2024-04-02T05:01:36.072769+0000 mon.a (mon.0) 3705 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[21528]: audit 2024-04-02T05:01:36.076077+0000 mon.a (mon.0) 3706 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-130", "mode": "writeback"}]': finished 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[21528]: cluster 2024-04-02T05:01:36.086998+0000 mon.a (mon.0) 3707 : cluster 0 osdmap e784: 8 total, 8 up, 7 in 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[21528]: audit 2024-04-02T05:01:36.217993+0000 mon.a (mon.0) 3708 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[28904]: cluster 2024-04-02T05:01:36.072769+0000 mon.a (mon.0) 3705 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[28904]: audit 2024-04-02T05:01:36.076077+0000 mon.a (mon.0) 3706 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-130", "mode": "writeback"}]': finished 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[28904]: cluster 2024-04-02T05:01:36.086998+0000 mon.a (mon.0) 3707 : cluster 0 osdmap e784: 8 total, 8 up, 7 in 2024-04-02T05:01:37.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:36 smithi069 bash[28904]: audit 2024-04-02T05:01:36.217993+0000 mon.a (mon.0) 3708 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:37.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:37.135 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:37.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- 172.21.15.69:0/1701256979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8101810 msgr2=0x7fc4e8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- 172.21.15.69:0/1701256979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8101810 0x7fc4e8106f50 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009990 tx=0x7fc4d8009ca0 comp rx=0 tx=0).stop 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- 172.21.15.69:0/1701256979 shutdown_connections 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- 172.21.15.69:0/1701256979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8107680 0x7fc4e810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- 172.21.15.69:0/1701256979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e8100e60 0x7fc4e8101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- 172.21.15.69:0/1701256979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8101810 0x7fc4e8106f50 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:37.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- 172.21.15.69:0/1701256979 >> 172.21.15.69:0/1701256979 conn(0x7fc4e8076460 msgr2=0x7fc4e8076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- 172.21.15.69:0/1701256979 shutdown_connections 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- 172.21.15.69:0/1701256979 wait complete. 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 librados: starting msgr at 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 librados: starting objecter 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 Processor -- start 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- start start 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 librados: setting wanted keys 2024-04-02T05:01:37.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 librados: calling monclient init 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e8100e60 0x7fc4e8101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 0x7fc4e8102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 0x7fc4e8102860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8111ae0 con 0x7fc4e8101810 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8111960 con 0x7fc4e8100e60 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.246+0000 7fc4edc16700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e8111c60 con 0x7fc4e8107680 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 0x7fc4e8102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 0x7fc4e8102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 0x7fc4e8102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60312/0 (socket says 172.21.15.69:60312) 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 0x7fc4e8102320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52840/0 (socket says 172.21.15.69:52840) 2024-04-02T05:01:37.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 -- 172.21.15.69:0/2572235856 learned_addr learned my addr 172.21.15.69:0/2572235856 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e77fe700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e8100e60 0x7fc4e8101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 -- 172.21.15.69:0/2572235856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e8100e60 msgr2=0x7fc4e8101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e8100e60 0x7fc4e8101de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 -- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 msgr2=0x7fc4e8102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 0x7fc4e8102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 -- 172.21.15.69:0/2572235856 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e81a65b0 con 0x7fc4e8107680 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e6ffd700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 0x7fc4e8102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e7fff700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 0x7fc4e8102860 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000c370 tx=0x7fc4e000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 10 client.?.objecter ms_handle_connect 0x7fc4e8107680 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e000ce20 con 0x7fc4e8107680 2024-04-02T05:01:37.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc4e00090d0 con 0x7fc4e8107680 2024-04-02T05:01:37.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e81a6840 con 0x7fc4e8107680 2024-04-02T05:01:37.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00046b0 con 0x7fc4e8107680 2024-04-02T05:01:37.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4edc16700 10 client.43769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:37.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e81a6d80 con 0x7fc4e8107680 2024-04-02T05:01:37.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4edc16700 1 librados: init done 2024-04-02T05:01:37.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e804ff00 con 0x7fc4e8107680 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc4e001a020 con 0x7fc4e8107680 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc4d006f5e0 0x7fc4d0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e77fe700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc4d006f5e0 0x7fc4d0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(785..785 src has 74..785) ==== 10079+0+0 (secure 0 0 0) 0x7fc4e008e3d0 con 0x7fc4e8107680 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 10 client.43769.objecter ms_dispatch 0x7fc4e8001550 osd_map(785..785 src has 74..785) 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 3 client.43769.objecter handle_osd_map got epochs [785,785] > 0 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 3 client.43769.objecter handle_osd_map decoding full epoch 785 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 20 client.43769.objecter dump_active .. 0 homeless 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e4ff9700 10 client.43769.objecter ms_handle_connect 0x7fc4d006f5e0 2024-04-02T05:01:37.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.250+0000 7fc4e77fe700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc4d006f5e0 0x7fc4d0071aa0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc005ea0 tx=0x7fc4dc005e30 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:37.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.258+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc4e0010040 con 0x7fc4e8107680 2024-04-02T05:01:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:37.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:37.394 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:37.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:37.478+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fc4e81a6fe0 con 0x7fc4e8107680 2024-04-02T05:01:37.497 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:37.519 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:37.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:37.647 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:37.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:37.777 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:37.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:37.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:37.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:01:38.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:38.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[21528]: cluster 2024-04-02T05:01:36.318170+0000 mgr.y (mgr.14538) 695 : cluster 0 pgmap v1189: 224 pgs: 27 creating+peering, 197 active+clean; 582 KiB data, 727 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T05:01:38.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[21528]: audit 2024-04-02T05:01:37.086365+0000 mon.a (mon.0) 3709 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:38.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[21528]: cluster 2024-04-02T05:01:37.091379+0000 mon.a (mon.0) 3710 : cluster 0 osdmap e785: 8 total, 8 up, 7 in 2024-04-02T05:01:38.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[21528]: audit 2024-04-02T05:01:37.091794+0000 mon.a (mon.0) 3711 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130"}]: dispatch 2024-04-02T05:01:38.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[21528]: audit 2024-04-02T05:01:37.483753+0000 mon.c (mon.2) 1158 : audit 1 from='client.? 172.21.15.69:0/2572235856' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:38.021 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[21528]: audit 2024-04-02T05:01:37.484288+0000 mon.a (mon.0) 3712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:38.022 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[28904]: cluster 2024-04-02T05:01:36.318170+0000 mgr.y (mgr.14538) 695 : cluster 0 pgmap v1189: 224 pgs: 27 creating+peering, 197 active+clean; 582 KiB data, 727 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T05:01:38.022 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[28904]: audit 2024-04-02T05:01:37.086365+0000 mon.a (mon.0) 3709 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:38.022 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[28904]: cluster 2024-04-02T05:01:37.091379+0000 mon.a (mon.0) 3710 : cluster 0 osdmap e785: 8 total, 8 up, 7 in 2024-04-02T05:01:38.022 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[28904]: audit 2024-04-02T05:01:37.091794+0000 mon.a (mon.0) 3711 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130"}]: dispatch 2024-04-02T05:01:38.022 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[28904]: audit 2024-04-02T05:01:37.483753+0000 mon.c (mon.2) 1158 : audit 1 from='client.? 172.21.15.69:0/2572235856' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:38.022 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:37 smithi069 bash[28904]: audit 2024-04-02T05:01:37.484288+0000 mon.a (mon.0) 3712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:38.034 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.090+0000 7fc4e4ff9700 1 -- 172.21.15.69:0/2572235856 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v786) ==== 97+0+0 (secure 0 0 0) 0x7fc4e005afa0 con 0x7fc4e8107680 2024-04-02T05:01:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 10 librados: watch_flush enter 2024-04-02T05:01:38.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 10 librados: watch_flush exit 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 20 client.43769.objecter shutdown clearing up homeless session... 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 10 client.43769.objecter successfully canceled tick 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc4d006f5e0 msgr2=0x7fc4d0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc4d006f5e0 0x7fc4d0071aa0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc005ea0 tx=0x7fc4dc005e30 comp rx=0 tx=0).stop 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 msgr2=0x7fc4e8102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 0x7fc4e8102860 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000c370 tx=0x7fc4e000c680 comp rx=0 tx=0).stop 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 shutdown_connections 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc4d006f5e0 0x7fc4d0071aa0 unknown :-1 s=CLOSED pgs=4008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4e8107680 0x7fc4e8102860 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e8100e60 0x7fc4e8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 --2- 172.21.15.69:0/2572235856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e8101810 0x7fc4e8102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 >> 172.21.15.69:0/2572235856 conn(0x7fc4e8076460 msgr2=0x7fc4e80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:38.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 shutdown_connections 2024-04-02T05:01:38.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 -- 172.21.15.69:0/2572235856 wait complete. 2024-04-02T05:01:38.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:38.094+0000 7fc4edc16700 1 librados: shutdown 2024-04-02T05:01:38.099 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T05:01:38.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:38.159 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:37 smithi098 bash[23790]: cluster 2024-04-02T05:01:36.318170+0000 mgr.y (mgr.14538) 695 : cluster 0 pgmap v1189: 224 pgs: 27 creating+peering, 197 active+clean; 582 KiB data, 727 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T05:01:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:37 smithi098 bash[23790]: audit 2024-04-02T05:01:37.086365+0000 mon.a (mon.0) 3709 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:37 smithi098 bash[23790]: cluster 2024-04-02T05:01:37.091379+0000 mon.a (mon.0) 3710 : cluster 0 osdmap e785: 8 total, 8 up, 7 in 2024-04-02T05:01:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:37 smithi098 bash[23790]: audit 2024-04-02T05:01:37.091794+0000 mon.a (mon.0) 3711 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130"}]: dispatch 2024-04-02T05:01:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:37 smithi098 bash[23790]: audit 2024-04-02T05:01:37.483753+0000 mon.c (mon.2) 1158 : audit 1 from='client.? 172.21.15.69:0/2572235856' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:37 smithi098 bash[23790]: audit 2024-04-02T05:01:37.484288+0000 mon.a (mon.0) 3712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:01:38.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:38.290 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:38.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:38.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:38.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:38.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:38.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:38.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:38 smithi098 bash[23790]: cluster 2024-04-02T05:01:38.086582+0000 mon.a (mon.0) 3713 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:38.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:38 smithi098 bash[23790]: cluster 2024-04-02T05:01:38.086612+0000 mon.a (mon.0) 3714 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:38.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:38 smithi098 bash[23790]: audit 2024-04-02T05:01:38.088226+0000 mon.a (mon.0) 3715 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130"}]': finished 2024-04-02T05:01:38.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:38 smithi098 bash[23790]: audit 2024-04-02T05:01:38.088289+0000 mon.a (mon.0) 3716 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:01:38.797 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:38 smithi098 bash[23790]: cluster 2024-04-02T05:01:38.092124+0000 mon.a (mon.0) 3717 : cluster 0 osdmap e786: 8 total, 8 up, 7 in 2024-04-02T05:01:38.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:38.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:38.944 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:38.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:39.031 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:01:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:01:39.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[21528]: cluster 2024-04-02T05:01:38.086582+0000 mon.a (mon.0) 3713 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:39.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[21528]: cluster 2024-04-02T05:01:38.086612+0000 mon.a (mon.0) 3714 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:39.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[21528]: audit 2024-04-02T05:01:38.088226+0000 mon.a (mon.0) 3715 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130"}]': finished 2024-04-02T05:01:39.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[21528]: audit 2024-04-02T05:01:38.088289+0000 mon.a (mon.0) 3716 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:01:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[21528]: cluster 2024-04-02T05:01:38.092124+0000 mon.a (mon.0) 3717 : cluster 0 osdmap e786: 8 total, 8 up, 7 in 2024-04-02T05:01:39.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[28904]: cluster 2024-04-02T05:01:38.086582+0000 mon.a (mon.0) 3713 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:39.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[28904]: cluster 2024-04-02T05:01:38.086612+0000 mon.a (mon.0) 3714 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:39.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[28904]: audit 2024-04-02T05:01:38.088226+0000 mon.a (mon.0) 3715 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-130"}]': finished 2024-04-02T05:01:39.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[28904]: audit 2024-04-02T05:01:38.088289+0000 mon.a (mon.0) 3716 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:01:39.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:38 smithi069 bash[28904]: cluster 2024-04-02T05:01:38.092124+0000 mon.a (mon.0) 3717 : cluster 0 osdmap e786: 8 total, 8 up, 7 in 2024-04-02T05:01:39.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:39.068 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:39.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:39.195 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:39.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:39.322 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:39.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:39.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:39.581 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:39.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:39.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:39.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:39.841 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:39.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:39.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:39 smithi069 bash[28904]: cluster 2024-04-02T05:01:38.319900+0000 mgr.y (mgr.14538) 696 : cluster 0 pgmap v1192: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 573 B/s wr, 4 op/s 2024-04-02T05:01:39.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:39 smithi069 bash[28904]: cluster 2024-04-02T05:01:39.089101+0000 mon.a (mon.0) 3718 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:39.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:39 smithi069 bash[28904]: cluster 2024-04-02T05:01:39.093594+0000 mon.a (mon.0) 3719 : cluster 0 osdmap e787: 8 total, 8 up, 7 in 2024-04-02T05:01:39.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:39 smithi069 bash[21528]: cluster 2024-04-02T05:01:38.319900+0000 mgr.y (mgr.14538) 696 : cluster 0 pgmap v1192: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 573 B/s wr, 4 op/s 2024-04-02T05:01:39.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:39 smithi069 bash[21528]: cluster 2024-04-02T05:01:39.089101+0000 mon.a (mon.0) 3718 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:39.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:39 smithi069 bash[21528]: cluster 2024-04-02T05:01:39.093594+0000 mon.a (mon.0) 3719 : cluster 0 osdmap e787: 8 total, 8 up, 7 in 2024-04-02T05:01:39.971 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:39.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:40.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:40.085 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:39 smithi098 bash[23790]: cluster 2024-04-02T05:01:38.319900+0000 mgr.y (mgr.14538) 696 : cluster 0 pgmap v1192: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 573 B/s wr, 4 op/s 2024-04-02T05:01:40.086 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:39 smithi098 bash[23790]: cluster 2024-04-02T05:01:39.089101+0000 mon.a (mon.0) 3718 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:40.086 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:39 smithi098 bash[23790]: cluster 2024-04-02T05:01:39.093594+0000 mon.a (mon.0) 3719 : cluster 0 osdmap e787: 8 total, 8 up, 7 in 2024-04-02T05:01:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:40.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:40.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:40.361 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:40.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:40.492 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:40.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:40.624 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:40.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:40.753 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:40.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:40.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:40.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:41.043 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:01:41.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T05:01:41.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:41.125 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:41 smithi098 bash[23790]: cluster 2024-04-02T05:01:40.105909+0000 mon.a (mon.0) 3720 : cluster 0 osdmap e788: 8 total, 8 up, 7 in 2024-04-02T05:01:41.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:41 smithi098 bash[23790]: audit 2024-04-02T05:01:40.124581+0000 mon.a (mon.0) 3721 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:41.126 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:41 smithi098 bash[23790]: cluster 2024-04-02T05:01:40.320767+0000 mgr.y (mgr.14538) 697 : cluster 0 pgmap v1195: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:41.139 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:41.242 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:41.264 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:41.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:41.394 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:41.497 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:41 smithi069 bash[21528]: cluster 2024-04-02T05:01:40.105909+0000 mon.a (mon.0) 3720 : cluster 0 osdmap e788: 8 total, 8 up, 7 in 2024-04-02T05:01:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:41 smithi069 bash[21528]: audit 2024-04-02T05:01:40.124581+0000 mon.a (mon.0) 3721 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:41 smithi069 bash[21528]: cluster 2024-04-02T05:01:40.320767+0000 mgr.y (mgr.14538) 697 : cluster 0 pgmap v1195: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:41.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:41 smithi069 bash[28904]: cluster 2024-04-02T05:01:40.105909+0000 mon.a (mon.0) 3720 : cluster 0 osdmap e788: 8 total, 8 up, 7 in 2024-04-02T05:01:41.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:41 smithi069 bash[28904]: audit 2024-04-02T05:01:40.124581+0000 mon.a (mon.0) 3721 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:41.510 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:41 smithi069 bash[28904]: cluster 2024-04-02T05:01:40.320767+0000 mgr.y (mgr.14538) 697 : cluster 0 pgmap v1195: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-02T05:01:41.524 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:41.582 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:41.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:41.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:41.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:41.778 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:41.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:41.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:42.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:42.040 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:42.143 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:42.170 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:42 smithi098 bash[23790]: audit 2024-04-02T05:01:41.096997+0000 mon.a (mon.0) 3722 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:42.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:42 smithi098 bash[23790]: cluster 2024-04-02T05:01:41.102640+0000 mon.a (mon.0) 3723 : cluster 0 osdmap e789: 8 total, 8 up, 7 in 2024-04-02T05:01:42.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:42 smithi098 bash[23790]: audit 2024-04-02T05:01:41.106757+0000 mon.a (mon.0) 3724 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:42.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:42 smithi098 bash[23790]: audit 2024-04-02T05:01:42.099704+0000 mon.a (mon.0) 3725 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:42.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:42.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[21528]: audit 2024-04-02T05:01:41.096997+0000 mon.a (mon.0) 3722 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[21528]: cluster 2024-04-02T05:01:41.102640+0000 mon.a (mon.0) 3723 : cluster 0 osdmap e789: 8 total, 8 up, 7 in 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[21528]: audit 2024-04-02T05:01:41.106757+0000 mon.a (mon.0) 3724 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[21528]: audit 2024-04-02T05:01:42.099704+0000 mon.a (mon.0) 3725 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[28904]: audit 2024-04-02T05:01:41.096997+0000 mon.a (mon.0) 3722 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[28904]: cluster 2024-04-02T05:01:41.102640+0000 mon.a (mon.0) 3723 : cluster 0 osdmap e789: 8 total, 8 up, 7 in 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[28904]: audit 2024-04-02T05:01:41.106757+0000 mon.a (mon.0) 3724 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:42.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:42 smithi069 bash[28904]: audit 2024-04-02T05:01:42.099704+0000 mon.a (mon.0) 3725 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:42.299 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:42.402 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:42.429 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:42.532 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:42.561 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:42.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:42.691 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:42.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:42.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:42.954 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:42.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:43.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:43.079 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:43.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:43.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:43.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[21528]: cluster 2024-04-02T05:01:42.107362+0000 mon.a (mon.0) 3726 : cluster 0 osdmap e790: 8 total, 8 up, 7 in 2024-04-02T05:01:43.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[21528]: audit 2024-04-02T05:01:42.111208+0000 mon.a (mon.0) 3727 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[21528]: cluster 2024-04-02T05:01:42.321488+0000 mgr.y (mgr.14538) 698 : cluster 0 pgmap v1198: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[21528]: audit 2024-04-02T05:01:43.101914+0000 mon.a (mon.0) 3728 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[21528]: cluster 2024-04-02T05:01:43.105767+0000 mon.a (mon.0) 3729 : cluster 0 osdmap e791: 8 total, 8 up, 7 in 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[21528]: audit 2024-04-02T05:01:43.106980+0000 mon.a (mon.0) 3730 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-132", "mode": "writeback"}]: dispatch 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[28904]: cluster 2024-04-02T05:01:42.107362+0000 mon.a (mon.0) 3726 : cluster 0 osdmap e790: 8 total, 8 up, 7 in 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[28904]: audit 2024-04-02T05:01:42.111208+0000 mon.a (mon.0) 3727 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[28904]: cluster 2024-04-02T05:01:42.321488+0000 mgr.y (mgr.14538) 698 : cluster 0 pgmap v1198: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[28904]: audit 2024-04-02T05:01:43.101914+0000 mon.a (mon.0) 3728 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[28904]: cluster 2024-04-02T05:01:43.105767+0000 mon.a (mon.0) 3729 : cluster 0 osdmap e791: 8 total, 8 up, 7 in 2024-04-02T05:01:43.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:43 smithi069 bash[28904]: audit 2024-04-02T05:01:43.106980+0000 mon.a (mon.0) 3730 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-132", "mode": "writeback"}]: dispatch 2024-04-02T05:01:43.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:43.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[23790]: cluster 2024-04-02T05:01:42.107362+0000 mon.a (mon.0) 3726 : cluster 0 osdmap e790: 8 total, 8 up, 7 in 2024-04-02T05:01:43.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[23790]: audit 2024-04-02T05:01:42.111208+0000 mon.a (mon.0) 3727 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:43.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[23790]: cluster 2024-04-02T05:01:42.321488+0000 mgr.y (mgr.14538) 698 : cluster 0 pgmap v1198: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:43.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[23790]: audit 2024-04-02T05:01:43.101914+0000 mon.a (mon.0) 3728 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:43.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[23790]: cluster 2024-04-02T05:01:43.105767+0000 mon.a (mon.0) 3729 : cluster 0 osdmap e791: 8 total, 8 up, 7 in 2024-04-02T05:01:43.387 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[23790]: audit 2024-04-02T05:01:43.106980+0000 mon.a (mon.0) 3730 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-132", "mode": "writeback"}]: dispatch 2024-04-02T05:01:43.431 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:43.456 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:43.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:43.629 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:43.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:43.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:43.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:43.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:01:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:01:43.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:43.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:43.971 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:43.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:44.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:44.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:44.229 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:44.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:44.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:44 smithi098 bash[23790]: audit 2024-04-02T05:01:43.669939+0000 mgr.y (mgr.14538) 699 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:44.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:44 smithi098 bash[23790]: cluster 2024-04-02T05:01:44.102454+0000 mon.a (mon.0) 3731 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:44.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:44 smithi098 bash[23790]: audit 2024-04-02T05:01:44.104635+0000 mon.a (mon.0) 3732 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-132", "mode": "writeback"}]': finished 2024-04-02T05:01:44.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:44 smithi098 bash[23790]: cluster 2024-04-02T05:01:44.112727+0000 mon.a (mon.0) 3733 : cluster 0 osdmap e792: 8 total, 8 up, 7 in 2024-04-02T05:01:44.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:44.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:44.483 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:44.531 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[28904]: audit 2024-04-02T05:01:43.669939+0000 mgr.y (mgr.14538) 699 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:44.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[28904]: cluster 2024-04-02T05:01:44.102454+0000 mon.a (mon.0) 3731 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:44.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[28904]: audit 2024-04-02T05:01:44.104635+0000 mon.a (mon.0) 3732 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-132", "mode": "writeback"}]': finished 2024-04-02T05:01:44.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[28904]: cluster 2024-04-02T05:01:44.112727+0000 mon.a (mon.0) 3733 : cluster 0 osdmap e792: 8 total, 8 up, 7 in 2024-04-02T05:01:44.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:44.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[21528]: audit 2024-04-02T05:01:43.669939+0000 mgr.y (mgr.14538) 699 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:44.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[21528]: cluster 2024-04-02T05:01:44.102454+0000 mon.a (mon.0) 3731 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:44.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[21528]: audit 2024-04-02T05:01:44.104635+0000 mon.a (mon.0) 3732 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-132", "mode": "writeback"}]': finished 2024-04-02T05:01:44.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:44 smithi069 bash[21528]: cluster 2024-04-02T05:01:44.112727+0000 mon.a (mon.0) 3733 : cluster 0 osdmap e792: 8 total, 8 up, 7 in 2024-04-02T05:01:44.655 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-04-02T05:01:44.655 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:01:44.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:44.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:44.859 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:44.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:44.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:45.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:45.120 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:45.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:45.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:45.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:45.375 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:45 smithi098 bash[23790]: cluster 2024-04-02T05:01:44.322388+0000 mgr.y (mgr.14538) 700 : cluster 0 pgmap v1201: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:45.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:45 smithi098 bash[23790]: cluster 2024-04-02T05:01:45.073628+0000 mon.a (mon.0) 3734 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:45.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:45 smithi098 bash[23790]: cluster 2024-04-02T05:01:45.086215+0000 mon.a (mon.0) 3735 : cluster 0 osdmap e793: 8 total, 8 up, 7 in 2024-04-02T05:01:45.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:45.514 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:45 smithi069 bash[21528]: cluster 2024-04-02T05:01:44.322388+0000 mgr.y (mgr.14538) 700 : cluster 0 pgmap v1201: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:45.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:45 smithi069 bash[21528]: cluster 2024-04-02T05:01:45.073628+0000 mon.a (mon.0) 3734 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:45.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:45 smithi069 bash[21528]: cluster 2024-04-02T05:01:45.086215+0000 mon.a (mon.0) 3735 : cluster 0 osdmap e793: 8 total, 8 up, 7 in 2024-04-02T05:01:45.515 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:45 smithi069 bash[28904]: cluster 2024-04-02T05:01:44.322388+0000 mgr.y (mgr.14538) 700 : cluster 0 pgmap v1201: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:45.515 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:45 smithi069 bash[28904]: cluster 2024-04-02T05:01:45.073628+0000 mon.a (mon.0) 3734 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:01:45.515 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:45 smithi069 bash[28904]: cluster 2024-04-02T05:01:45.086215+0000 mon.a (mon.0) 3735 : cluster 0 osdmap e793: 8 total, 8 up, 7 in 2024-04-02T05:01:45.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-02T05:01:45.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:45.647 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:45.688 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:45.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:45.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:45.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:46.005 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:46.033 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:46.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:46.161 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:46.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:46.289 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:46.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:46.421 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:46.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:46.551 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:46.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:46.680 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:46.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:46.811 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:46.914 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:46.941 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:46.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:47.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:47.065 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:47.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:47.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:47.191 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:47.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:47.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:47.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:47 smithi098 bash[23790]: cluster 2024-04-02T05:01:46.083986+0000 mon.a (mon.0) 3736 : cluster 0 osdmap e794: 8 total, 8 up, 7 in 2024-04-02T05:01:47.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:47 smithi098 bash[23790]: audit 2024-04-02T05:01:46.160939+0000 mon.a (mon.0) 3737 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:47.437 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:47 smithi098 bash[23790]: cluster 2024-04-02T05:01:46.323308+0000 mgr.y (mgr.14538) 701 : cluster 0 pgmap v1204: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:47.470 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:47 smithi069 bash[21528]: cluster 2024-04-02T05:01:46.083986+0000 mon.a (mon.0) 3736 : cluster 0 osdmap e794: 8 total, 8 up, 7 in 2024-04-02T05:01:47.470 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:47 smithi069 bash[21528]: audit 2024-04-02T05:01:46.160939+0000 mon.a (mon.0) 3737 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:47.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:47 smithi069 bash[21528]: cluster 2024-04-02T05:01:46.323308+0000 mgr.y (mgr.14538) 701 : cluster 0 pgmap v1204: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:47.471 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:47 smithi069 bash[28904]: cluster 2024-04-02T05:01:46.083986+0000 mon.a (mon.0) 3736 : cluster 0 osdmap e794: 8 total, 8 up, 7 in 2024-04-02T05:01:47.471 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:47 smithi069 bash[28904]: audit 2024-04-02T05:01:46.160939+0000 mon.a (mon.0) 3737 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:47.471 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:47 smithi069 bash[28904]: cluster 2024-04-02T05:01:46.323308+0000 mgr.y (mgr.14538) 701 : cluster 0 pgmap v1204: 224 pgs: 224 active+clean; 582 KiB data, 728 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:47.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:47.577 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 -- 172.21.15.69:0/704969077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa570101b60 msgr2=0x7fa57010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 --2- 172.21.15.69:0/704969077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa570101b60 0x7fa57010dfd0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fa568009f90 tx=0x7fa56800b330 comp rx=0 tx=0).stop 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 -- 172.21.15.69:0/704969077 shutdown_connections 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 --2- 172.21.15.69:0/704969077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa570101b60 0x7fa57010dfd0 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 --2- 172.21.15.69:0/704969077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa570107220 0x7fa570107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 --2- 172.21.15.69:0/704969077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5701011a0 0x7fa570101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 -- 172.21.15.69:0/704969077 >> 172.21.15.69:0/704969077 conn(0x7fa5700771e0 msgr2=0x7fa5700775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 -- 172.21.15.69:0/704969077 shutdown_connections 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.602+0000 7fa574aa0700 1 -- 172.21.15.69:0/704969077 wait complete. 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 librados: starting msgr at 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 librados: starting objecter 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 Processor -- start 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- start start 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 librados: setting wanted keys 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 librados: calling monclient init 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5701011a0 0x7fa570197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa570101b60 0x7fa5701984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa570107220 0x7fa57019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa570110970 con 0x7fa570107220 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5701107f0 con 0x7fa570101b60 2024-04-02T05:01:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa570110af0 con 0x7fa5701011a0 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5701011a0 0x7fa570197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56ed9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa570107220 0x7fa57019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5701011a0 0x7fa570197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41908/0 (socket says 172.21.15.69:41908) 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 -- 172.21.15.69:0/4264141632 learned_addr learned my addr 172.21.15.69:0/4264141632 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56dd9b700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa570101b60 0x7fa5701984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 -- 172.21.15.69:0/4264141632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa570101b60 msgr2=0x7fa5701984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa570101b60 0x7fa5701984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 -- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa570107220 msgr2=0x7fa57019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa570107220 0x7fa57019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 -- 172.21.15.69:0/4264141632 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57019cf50 con 0x7fa5701011a0 2024-04-02T05:01:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56ed9d700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa570107220 0x7fa57019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:01:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa56e59c700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5701011a0 0x7fa570197f90 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fa56000ec10 tx=0x7fa56000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 10 client.?.objecter ms_handle_connect 0x7fa5701011a0 2024-04-02T05:01:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56000cd40 con 0x7fa5701011a0 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa5600044a0 con 0x7fa5701011a0 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- 172.21.15.69:0/4264141632 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57019d240 con 0x7fa5701011a0 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560010430 con 0x7fa5701011a0 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 10 client.43775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- 172.21.15.69:0/4264141632 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5701a4ac0 con 0x7fa5701011a0 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 librados: init done 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 10 client.43775.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 15 client.43775.objecter _session_command_op_assign -1 1 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 20 client.43775.objecter _get_session osd=-1 returning homeless 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 10 client.43775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa574aa0700 1 -- 172.21.15.69:0/4264141632 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa570102860 con 0x7fa5701011a0 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa560004000 con 0x7fa5701011a0 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa55806f5e0 0x7fa558071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(795..795 src has 74..795) ==== 10079+0+0 (secure 0 0 0) 0x7fa56008ee00 con 0x7fa5701011a0 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 10 client.43775.objecter ms_dispatch 0x7fa570001550 osd_map(795..795 src has 74..795) 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 3 client.43775.objecter handle_osd_map got epochs [795,795] > 0 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.606+0000 7fa55f7fe700 3 client.43775.objecter handle_osd_map decoding full epoch 795 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa56dd9b700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa55806f5e0 0x7fa558071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 10 client.43775.objecter checking command 1 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa55807d2a0 0x7fa55807f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter _get_session s=0x7fa55807cbd0 osd=6 3 2024-04-02T05:01:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter put_session s=0x7fa55807cbd0 osd=6 3 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 15 client.43775.objecter _session_command_op_remove -1 1 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter _get_session s=0x7fa55807cbd0 osd=6 3 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter get_session s=0x7fa55807cbd0 osd=6 3 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 15 client.43775.objecter _session_command_op_assign 6 1 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter put_session s=0x7fa55807cbd0 osd=6 4 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 10 client.43775.objecter _send_command 1 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5600042f0 con 0x7fa55807d2a0 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter dump_active .. 0 homeless 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=795) ==== 24+0+0 (secure 0 0 0) 0x7fa5600944d0 con 0x7fa5701011a0 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 10 client.43775.objecter ms_handle_connect 0x7fa55806f5e0 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa56dd9b700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa55806f5e0 0x7fa558071aa0 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7fa570077920 tx=0x7fa564005cf0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa56ed9d700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa55807d2a0 0x7fa55807f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 10 client.43775.objecter ms_handle_connect 0x7fa55807d2a0 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa56ed9d700 1 --2- 172.21.15.69:0/4264141632 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fa55807d2a0 0x7fa55807f740 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 1 -- 172.21.15.69:0/4264141632 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5600042f0 con 0x7fa55807d2a0 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 10 client.43775.objecter ms_dispatch 0x7fa570001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 10 client.43775.objecter _finish_command 1 = system:0 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 20 client.43775.objecter put_session s=0x7fa55807cbd0 osd=6 4 2024-04-02T05:01:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:47.610+0000 7fa55f7fe700 15 client.43775.objecter _session_command_op_remove 6 1 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:01:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:01:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:01:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:01:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:01:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:01:47.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:47.707 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:47.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:47.838 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:47.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:47.963 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:47.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:48.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:48.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:48.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[21528]: audit 2024-04-02T05:01:47.090241+0000 mon.a (mon.0) 3738 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[21528]: cluster 2024-04-02T05:01:47.094533+0000 mon.a (mon.0) 3739 : cluster 0 osdmap e795: 8 total, 8 up, 7 in 2024-04-02T05:01:48.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[21528]: audit 2024-04-02T05:01:47.102666+0000 mon.a (mon.0) 3740 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:48.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[28904]: audit 2024-04-02T05:01:47.090241+0000 mon.a (mon.0) 3738 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:48.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[28904]: cluster 2024-04-02T05:01:47.094533+0000 mon.a (mon.0) 3739 : cluster 0 osdmap e795: 8 total, 8 up, 7 in 2024-04-02T05:01:48.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[28904]: audit 2024-04-02T05:01:47.102666+0000 mon.a (mon.0) 3740 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:48.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:48.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:48 smithi098 bash[23790]: audit 2024-04-02T05:01:47.090241+0000 mon.a (mon.0) 3738 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:48.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:48 smithi098 bash[23790]: cluster 2024-04-02T05:01:47.094533+0000 mon.a (mon.0) 3739 : cluster 0 osdmap e795: 8 total, 8 up, 7 in 2024-04-02T05:01:48.337 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:48 smithi098 bash[23790]: audit 2024-04-02T05:01:47.102666+0000 mon.a (mon.0) 3740 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:48.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:48.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:48.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:48.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:48.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:48.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:48.871 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:48.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:48.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:49.001 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:49.052 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:01:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:01:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:01:49.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:49.134 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:49.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:49.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:49.376 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[21528]: audit 2024-04-02T05:01:48.101551+0000 mon.a (mon.0) 3741 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[21528]: cluster 2024-04-02T05:01:48.110007+0000 mon.a (mon.0) 3742 : cluster 0 osdmap e796: 8 total, 8 up, 7 in 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[21528]: cluster 2024-04-02T05:01:48.324249+0000 mgr.y (mgr.14538) 702 : cluster 0 pgmap v1207: 224 pgs: 224 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[21528]: audit 2024-04-02T05:01:49.010633+0000 mon.a (mon.0) 3743 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[21528]: audit 2024-04-02T05:01:49.013000+0000 mon.b (mon.1) 310 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[28904]: audit 2024-04-02T05:01:48.101551+0000 mon.a (mon.0) 3741 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[28904]: cluster 2024-04-02T05:01:48.110007+0000 mon.a (mon.0) 3742 : cluster 0 osdmap e796: 8 total, 8 up, 7 in 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[28904]: cluster 2024-04-02T05:01:48.324249+0000 mgr.y (mgr.14538) 702 : cluster 0 pgmap v1207: 224 pgs: 224 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[28904]: audit 2024-04-02T05:01:49.010633+0000 mon.a (mon.0) 3743 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:49.377 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:49 smithi069 bash[28904]: audit 2024-04-02T05:01:49.013000+0000 mon.b (mon.1) 310 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:49 smithi098 bash[23790]: audit 2024-04-02T05:01:48.101551+0000 mon.a (mon.0) 3741 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:49 smithi098 bash[23790]: cluster 2024-04-02T05:01:48.110007+0000 mon.a (mon.0) 3742 : cluster 0 osdmap e796: 8 total, 8 up, 7 in 2024-04-02T05:01:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:49 smithi098 bash[23790]: cluster 2024-04-02T05:01:48.324249+0000 mgr.y (mgr.14538) 702 : cluster 0 pgmap v1207: 224 pgs: 224 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-02T05:01:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:49 smithi098 bash[23790]: audit 2024-04-02T05:01:49.010633+0000 mon.a (mon.0) 3743 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:01:49.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:49 smithi098 bash[23790]: audit 2024-04-02T05:01:49.013000+0000 mon.b (mon.1) 310 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:01:49.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:49.520 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:49.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:49.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:49.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:49.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:49.906 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:49.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:50.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:50.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:50.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:50.166 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:50.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:50.281 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:50 smithi098 bash[23790]: cluster 2024-04-02T05:01:49.124871+0000 mon.a (mon.0) 3744 : cluster 0 osdmap e797: 8 total, 8 up, 7 in 2024-04-02T05:01:50.281 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:50 smithi098 bash[23790]: audit 2024-04-02T05:01:49.184674+0000 mon.a (mon.0) 3745 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:50.297 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:50.359 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:01:50.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:50.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:50.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:50 smithi069 bash[21528]: cluster 2024-04-02T05:01:49.124871+0000 mon.a (mon.0) 3744 : cluster 0 osdmap e797: 8 total, 8 up, 7 in 2024-04-02T05:01:50.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:50 smithi069 bash[21528]: audit 2024-04-02T05:01:49.184674+0000 mon.a (mon.0) 3745 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:50.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:50 smithi069 bash[28904]: cluster 2024-04-02T05:01:49.124871+0000 mon.a (mon.0) 3744 : cluster 0 osdmap e797: 8 total, 8 up, 7 in 2024-04-02T05:01:50.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:50 smithi069 bash[28904]: audit 2024-04-02T05:01:49.184674+0000 mon.a (mon.0) 3745 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:50.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:50.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:50.683 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:50.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:01:50.867 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:01:50.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 'trim stale osdmaps' 2024-04-02T05:01:50.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:50.938 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:50.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:51.042 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:51.140 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:51.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:51.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:51.383 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:49.554144+0000 osd.2 (osd.2) 349 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:49.554465+0000 osd.2 (osd.2) 350 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: audit 2024-04-02T05:01:50.113670+0000 mon.a (mon.0) 3746 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:50.121050+0000 mon.a (mon.0) 3747 : cluster 0 osdmap e798: 8 total, 8 up, 7 in 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: audit 2024-04-02T05:01:50.123161+0000 mon.a (mon.0) 3748 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:50.324999+0000 mgr.y (mgr.14538) 703 : cluster 0 pgmap v1210: 224 pgs: 224 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:50.660165+0000 osd.0 (osd.0) 383 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:50.660422+0000 osd.0 (osd.0) 384 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:51.113899+0000 mon.a (mon.0) 3749 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: audit 2024-04-02T05:01:51.115552+0000 mon.a (mon.0) 3750 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[28904]: cluster 2024-04-02T05:01:51.119373+0000 mon.a (mon.0) 3751 : cluster 0 osdmap e799: 8 total, 8 up, 7 in 2024-04-02T05:01:51.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:49.554144+0000 osd.2 (osd.2) 349 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:49.554465+0000 osd.2 (osd.2) 350 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: audit 2024-04-02T05:01:50.113670+0000 mon.a (mon.0) 3746 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:50.121050+0000 mon.a (mon.0) 3747 : cluster 0 osdmap e798: 8 total, 8 up, 7 in 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: audit 2024-04-02T05:01:50.123161+0000 mon.a (mon.0) 3748 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:50.324999+0000 mgr.y (mgr.14538) 703 : cluster 0 pgmap v1210: 224 pgs: 224 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:50.660165+0000 osd.0 (osd.0) 383 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:50.660422+0000 osd.0 (osd.0) 384 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:51.113899+0000 mon.a (mon.0) 3749 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: audit 2024-04-02T05:01:51.115552+0000 mon.a (mon.0) 3750 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:51.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:51 smithi069 bash[21528]: cluster 2024-04-02T05:01:51.119373+0000 mon.a (mon.0) 3751 : cluster 0 osdmap e799: 8 total, 8 up, 7 in 2024-04-02T05:01:51.397 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:49.554144+0000 osd.2 (osd.2) 349 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:49.554465+0000 osd.2 (osd.2) 350 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: audit 2024-04-02T05:01:50.113670+0000 mon.a (mon.0) 3746 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:50.121050+0000 mon.a (mon.0) 3747 : cluster 0 osdmap e798: 8 total, 8 up, 7 in 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: audit 2024-04-02T05:01:50.123161+0000 mon.a (mon.0) 3748 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132"}]: dispatch 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:50.324999+0000 mgr.y (mgr.14538) 703 : cluster 0 pgmap v1210: 224 pgs: 224 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:50.660165+0000 osd.0 (osd.0) 383 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:50.660422+0000 osd.0 (osd.0) 384 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:51.113899+0000 mon.a (mon.0) 3749 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: audit 2024-04-02T05:01:51.115552+0000 mon.a (mon.0) 3750 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-132"}]': finished 2024-04-02T05:01:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:51 smithi098 bash[23790]: cluster 2024-04-02T05:01:51.119373+0000 mon.a (mon.0) 3751 : cluster 0 osdmap e799: 8 total, 8 up, 7 in 2024-04-02T05:01:51.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:51.528 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:51.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:51.652 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:51.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:51.783 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:51.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:51.914 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:51.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:52.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:52.042 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:52.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:52.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:52.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/584185767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b48102cb0 msgr2=0x7f4b4810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/584185767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b48102cb0 0x7f4b4810f120 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000b0a0 tx=0x7f4b4000b3b0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/584185767 shutdown_connections 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/584185767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b48102cb0 0x7f4b4810f120 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/584185767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b48108300 0x7f4b481086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/584185767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b481022f0 0x7f4b48102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/584185767 >> 172.21.15.69:0/584185767 conn(0x7f4b480793a0 msgr2=0x7f4b480797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/584185767 shutdown_connections 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/584185767 wait complete. 2024-04-02T05:01:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 librados: starting msgr at 2024-04-02T05:01:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 librados: starting objecter 2024-04-02T05:01:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 Processor -- start 2024-04-02T05:01:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- start start 2024-04-02T05:01:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 librados: setting wanted keys 2024-04-02T05:01:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 librados: calling monclient init 2024-04-02T05:01:52.431 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 0x7f4b48197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b48102cb0 0x7f4b48198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b48108300 0x7f4b4819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b48111ac0 con 0x7f4b48108300 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b48111940 con 0x7f4b48102cb0 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ecd6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4b48111c40 con 0x7f4b481022f0 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 0x7f4b48197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 0x7f4b48197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41928/0 (socket says 172.21.15.69:41928) 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 -- 172.21.15.69:0/2543859676 learned_addr learned my addr 172.21.15.69:0/2543859676 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b48108300 0x7f4b4819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b48102cb0 msgr2=0x7f4b48198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b48102cb0 0x7f4b48198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b48102cb0 0x7f4b48198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b48108300 msgr2=0x7f4b4819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.425+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b48108300 0x7f4b4819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ca72700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4819cf10 con 0x7f4b481022f0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b48102cb0 0x7f4b48198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 0x7f4b48197f00 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00d950 tx=0x7f4b3c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 10 client.?.objecter ms_handle_connect 0x7f4b481022f0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c016070 con 0x7f4b481022f0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4b3c009dc0 con 0x7f4b481022f0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00f3c0 con 0x7f4b481022f0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4819d200 con 0x7f4b481022f0 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 10 client.43781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b481a4a80 con 0x7f4b481022f0 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 1 librados: init done 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4b4804ff00 con 0x7f4b481022f0 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4b3c019070 con 0x7f4b481022f0 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4b3806f5e0 0x7f4b38071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(800..800 src has 74..800) ==== 9754+0+0 (secure 0 0 0) 0x7f4b3c08e420 con 0x7f4b481022f0 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 osd_map(800..800 src has 74..800) 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 3 client.43781.objecter handle_osd_map got epochs [800,800] > 0 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 3 client.43781.objecter handle_osd_map decoding full epoch 800 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4b3806f5e0 0x7f4b38071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 20 client.43781.objecter dump_active .. 0 homeless 2024-04-02T05:01:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v800) ==== 54+0+16 (secure 0 0 0) 0x7f4b3c090a40 con 0x7f4b481022f0 2024-04-02T05:01:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 1 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4b48062e50 0x7f4b48063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 1 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 0 1 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 4 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 1 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b481a7de0 con 0x7f4b48062e50 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4b48062e50 0x7f4b48063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4b3806f5e0 0x7f4b38071aa0 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7f4b48079ac0 tx=0x7f4b34009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b3806f5e0 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b48062e50 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.429+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4b48062e50 0x7f4b48063230 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.433+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b481a7de0 con 0x7f4b48062e50 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.433+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 1: 0 ) 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.433+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 1 = system:0 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.433+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 4 2024-04-02T05:01:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.433+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 0 1 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 2 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 2 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b48199260 osd=0 3 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 0 2 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 4 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 2 2024-04-02T05:01:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.449+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b480628d0 con 0x7f4b48062e50 2024-04-02T05:01:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b480628d0 con 0x7f4b48062e50 2024-04-02T05:01:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 4 2024-04-02T05:01:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 0 2 2024-04-02T05:01:52.457 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:01:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4b48109d40 0x7f4b4802c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 1 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 4 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 3 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4b481ab740 con 0x7f4b48109d40 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4b48109d40 0x7f4b4802c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b48109d40 2024-04-02T05:01:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4b48109d40 0x7f4b4802c9b0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b481ab740 con 0x7f4b48109d40 2024-04-02T05:01:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 3: 0 ) 2024-04-02T05:01:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 3 = system:0 2024-04-02T05:01:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 4 2024-04-02T05:01:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.453+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 1 3 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 4 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 4 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b48062cd0 osd=1 3 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 1 4 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 4 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 4 2024-04-02T05:01:52.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481092b0 con 0x7f4b48109d40 2024-04-02T05:01:52.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481092b0 con 0x7f4b48109d40 2024-04-02T05:01:52.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 4 2024-04-02T05:01:52.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 1 4 2024-04-02T05:01:52.478 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 5 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4b481a9fa0 0x7f4b481aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 5 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 2 5 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 4 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 5 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4b48076480 con 0x7f4b481a9fa0 2024-04-02T05:01:52.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4b481a9fa0 0x7f4b481aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b481a9fa0 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.473+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4b481a9fa0 0x7f4b481aa3c0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.477+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b48076480 con 0x7f4b481a9fa0 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.477+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 5: 0 ) 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.477+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 5 = system:0 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.477+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 4 2024-04-02T05:01:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.477+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 2 5 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 6 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 6 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481a7430 osd=2 3 2024-04-02T05:01:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 2 6 2024-04-02T05:01:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 4 2024-04-02T05:01:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 6 2024-04-02T05:01:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.505+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481aa900 con 0x7f4b481a9fa0 2024-04-02T05:01:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481aa900 con 0x7f4b481a9fa0 2024-04-02T05:01:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 4 2024-04-02T05:01:52.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 2 6 2024-04-02T05:01:52.513 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 7 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4b481afd00 0x7f4b481b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 7 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 3 7 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 4 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 7 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4b481b4f50 con 0x7f4b481afd00 2024-04-02T05:01:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4b481afd00 0x7f4b481b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b481afd00 2024-04-02T05:01:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4b481afd00 0x7f4b481b20f0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b481b4f50 con 0x7f4b481afd00 2024-04-02T05:01:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 7: 0 ) 2024-04-02T05:01:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 7 = system:0 2024-04-02T05:01:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 4 2024-04-02T05:01:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.509+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 3 7 2024-04-02T05:01:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 8 2024-04-02T05:01:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 8 2024-04-02T05:01:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481ab280 osd=3 3 2024-04-02T05:01:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 3 8 2024-04-02T05:01:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 4 2024-04-02T05:01:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 8 2024-04-02T05:01:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481aad00 con 0x7f4b481afd00 2024-04-02T05:01:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481aad00 con 0x7f4b481afd00 2024-04-02T05:01:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 4 2024-04-02T05:01:52.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 3 8 2024-04-02T05:01:52.534 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:01:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 9 2024-04-02T05:01:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4b4802dbd0 0x7f4b481b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 9 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 4 9 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 4 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 9 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4b481aac60 con 0x7f4b4802dbd0 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.529+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4b4802dbd0 0x7f4b481b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b4802dbd0 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4b4802dbd0 0x7f4b481b57c0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b481aac60 con 0x7f4b4802dbd0 2024-04-02T05:01:52.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 9: 0 ) 2024-04-02T05:01:52.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 9 = system:0 2024-04-02T05:01:52.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 4 2024-04-02T05:01:52.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.533+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 4 9 2024-04-02T05:01:52.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:52.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 10 2024-04-02T05:01:52.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 10 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481aaab0 osd=4 3 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 4 10 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 4 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 10 2024-04-02T05:01:52.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.549+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481aacc0 con 0x7f4b4802dbd0 2024-04-02T05:01:52.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481aacc0 con 0x7f4b4802dbd0 2024-04-02T05:01:52.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 4 2024-04-02T05:01:52.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 4 10 2024-04-02T05:01:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:01:52.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 11 2024-04-02T05:01:52.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4b481b6270 0x7f4b481b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 11 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 5 11 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 4 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 11 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4b481bb860 con 0x7f4b481b6270 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4b481b6270 0x7f4b481b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b47fff700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4b481b6270 0x7f4b481b66b0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b481b6270 2024-04-02T05:01:52.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b481bb860 con 0x7f4b481b6270 2024-04-02T05:01:52.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 11: 0 ) 2024-04-02T05:01:52.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 11 = system:0 2024-04-02T05:01:52.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 4 2024-04-02T05:01:52.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.553+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 5 11 2024-04-02T05:01:52.559 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:52.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 12 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 12 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481aacc0 osd=5 3 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 5 12 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 4 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 12 2024-04-02T05:01:52.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.569+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481b6d00 con 0x7f4b481b6270 2024-04-02T05:01:52.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481b6d00 con 0x7f4b481b6270 2024-04-02T05:01:52.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 4 2024-04-02T05:01:52.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 5 12 2024-04-02T05:01:52.577 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 13 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4b481b7420 0x7f4b481b7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 13 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 6 13 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 4 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 13 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4b481b7f10 con 0x7f4b481b7420 2024-04-02T05:01:52.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4b481b7420 0x7f4b481b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b481b7420 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b4d273700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4b481b7420 0x7f4b481b7880 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b481b7f10 con 0x7f4b481b7420 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 13: 0 ) 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 13 = system:0 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 4 2024-04-02T05:01:52.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.573+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 6 13 2024-04-02T05:01:52.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 14 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 14 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481b6d00 osd=6 3 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 6 14 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 4 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 14 2024-04-02T05:01:52.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481b7f10 con 0x7f4b481b7420 2024-04-02T05:01:52.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481b7f10 con 0x7f4b481b7420 2024-04-02T05:01:52.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 4 2024-04-02T05:01:52.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 6 14 2024-04-02T05:01:52.598 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:01:52.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:01:52.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 15 2024-04-02T05:01:52.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4b481bcc00 0x7f4b481beff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 15 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 7 15 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 4 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 15 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4b481b8310 con 0x7f4b481bcc00 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4b481bcc00 0x7f4b481beff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b45ffb700 10 client.43781.objecter ms_handle_connect 0x7f4b481bcc00 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.593+0000 7f4b4ca72700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4b481bcc00 0x7f4b481beff0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.597+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b3c063410 con 0x7f4b481bcc00 2024-04-02T05:01:52.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.597+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 15: 0 ) 2024-04-02T05:01:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.597+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 15 = system:0 2024-04-02T05:01:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.597+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 4 2024-04-02T05:01:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.597+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 7 15 2024-04-02T05:01:52.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 10 client.43781.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:01:52.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign -1 16 2024-04-02T05:01:52.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 20 client.43781.objecter _get_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_remove -1 16 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 20 client.43781.objecter get_session s=0x7f4b481b7f10 osd=7 3 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 15 client.43781.objecter _session_command_op_assign 7 16 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 4 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 10 client.43781.objecter _send_command 16 2024-04-02T05:01:52.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.613+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4b481aaf70 con 0x7f4b481bcc00 2024-04-02T05:01:52.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b45ffb700 1 -- 172.21.15.69:0/2543859676 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4b481aaf70 con 0x7f4b481bcc00 2024-04-02T05:01:52.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b45ffb700 10 client.43781.objecter ms_dispatch 0x7f4b48001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:01:52.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b45ffb700 10 client.43781.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:01:52.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b45ffb700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 4 2024-04-02T05:01:52.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b45ffb700 15 client.43781.objecter _session_command_op_remove 7 16 2024-04-02T05:01:52.622 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:01:52.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 librados: watch_flush enter 2024-04-02T05:01:52.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 librados: watch_flush exit 2024-04-02T05:01:52.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.0 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4b48062e50 msgr2=0x7f4b48063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f4b48062e50 0x7f4b48063230 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48199260 osd=0 1 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.1 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4b48109d40 msgr2=0x7f4b4802c9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f4b48109d40 0x7f4b4802c9b0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b48062cd0 osd=1 1 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.2 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4b481a9fa0 msgr2=0x7f4b481aa3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f4b481a9fa0 0x7f4b481aa3c0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481a7430 osd=2 1 2024-04-02T05:01:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.3 2024-04-02T05:01:52.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4b481afd00 msgr2=0x7f4b481b20f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f4b481afd00 0x7f4b481b20f0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481ab280 osd=3 1 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.4 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4b4802dbd0 msgr2=0x7f4b481b57c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f4b4802dbd0 0x7f4b481b57c0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aaab0 osd=4 1 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.5 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4b481b6270 msgr2=0x7f4b481b66b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f4b481b6270 0x7f4b481b66b0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481aacc0 osd=5 1 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.6 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4b481b7420 msgr2=0x7f4b481b7880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f4b481b7420 0x7f4b481b7880 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b6d00 osd=6 1 2024-04-02T05:01:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4d273700 1 -- 172.21.15.69:0/2543859676 reap_dead start 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter close_session for osd.7 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4b481bcc00 msgr2=0x7f4b481beff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f4b481bcc00 0x7f4b481beff0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4d273700 1 -- 172.21.15.69:0/2543859676 reap_dead start 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter put_session s=0x7f4b481b7f10 osd=7 1 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 20 client.43781.objecter shutdown clearing up homeless session... 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 10 client.43781.objecter successfully canceled tick 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4b3806f5e0 msgr2=0x7f4b38071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f4b3806f5e0 0x7f4b38071aa0 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7f4b48079ac0 tx=0x7f4b34009260 comp rx=0 tx=0).stop 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 msgr2=0x7f4b48197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 0x7f4b48197f00 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00d950 tx=0x7f4b3c00dc60 comp rx=0 tx=0).stop 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4d273700 1 -- 172.21.15.69:0/2543859676 reap_dead start 2024-04-02T05:01:52.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4d273700 1 -- 172.21.15.69:0/2543859676 reap_dead start 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 shutdown_connections 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 --2- 172.21.15.69:0/2543859676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b481022f0 0x7f4b48197f00 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 >> 172.21.15.69:0/2543859676 conn(0x7f4b480793a0 msgr2=0x7f4b48105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 shutdown_connections 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 -- 172.21.15.69:0/2543859676 wait complete. 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.617+0000 7f4b4ecd6700 1 librados: shutdown 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:01:52.628 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:01:52.629 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:01:52.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:52.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- 172.21.15.69:0/794274918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e2406ab80 msgr2=0x7f9e24106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- 172.21.15.69:0/794274918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e2406ab80 0x7f9e24106ef0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f9e140099c0 tx=0x7f9e14009cd0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- 172.21.15.69:0/794274918 shutdown_connections 2024-04-02T05:01:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- 172.21.15.69:0/794274918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e24107620 0x7f9e2410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- 172.21.15.69:0/794274918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2406a1d0 0x7f9e2406a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- 172.21.15.69:0/794274918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e2406ab80 0x7f9e24106ef0 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- 172.21.15.69:0/794274918 >> 172.21.15.69:0/794274918 conn(0x7f9e240771e0 msgr2=0x7f9e240775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- 172.21.15.69:0/794274918 shutdown_connections 2024-04-02T05:01:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- 172.21.15.69:0/794274918 wait complete. 2024-04-02T05:01:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 librados: starting msgr at 2024-04-02T05:01:52.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 librados: starting objecter 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 Processor -- start 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- start start 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 librados: setting wanted keys 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 librados: calling monclient init 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 0x7f9e24101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2406ab80 0x7f9e241017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e24107620 0x7f9e24101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9e24111ab0 con 0x7f9e2406a1d0 2024-04-02T05:01:52.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e24111930 con 0x7f9e2406ab80 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e2b410700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9e24111c30 con 0x7f9e24107620 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e291ac700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 0x7f9e24101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e291ac700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 0x7f9e24101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52386/0 (socket says 172.21.15.69:52386) 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e299ad700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e24107620 0x7f9e24101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e291ac700 1 -- 172.21.15.69:0/3850849776 learned_addr learned my addr 172.21.15.69:0/3850849776 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.793+0000 7f9e289ab700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2406ab80 0x7f9e241017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e291ac700 1 -- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e24107620 msgr2=0x7f9e24101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e291ac700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e24107620 0x7f9e24101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e291ac700 1 -- 172.21.15.69:0/3850849776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2406ab80 msgr2=0x7f9e241017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e291ac700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2406ab80 0x7f9e241017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e291ac700 1 -- 172.21.15.69:0/3850849776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e241a65c0 con 0x7f9e2406a1d0 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e291ac700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 0x7f9e24101270 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c9b0 tx=0x7f9e2000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 10 client.?.objecter ms_handle_connect 0x7f9e2406a1d0 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20007b00 con 0x7f9e2406a1d0 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e241a68b0 con 0x7f9e2406a1d0 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e2b410700 10 client.33723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e241a6e10 con 0x7f9e2406a1d0 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9e20007ca0 con 0x7f9e2406a1d0 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e2b410700 1 librados: init done 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20018660 con 0x7f9e2406a1d0 2024-04-02T05:01:52.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e2404ff00 con 0x7f9e2406a1d0 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f9e2000f450 con 0x7f9e2406a1d0 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9e1006f5e0 0x7f9e10071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(800..800 src has 74..800) ==== 9754+0+0 (secure 0 0 0) 0x7f9e2008ec10 con 0x7f9e2406a1d0 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 10 client.33723.objecter ms_dispatch 0x7f9e24001550 osd_map(800..800 src has 74..800) 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 3 client.33723.objecter handle_osd_map got epochs [800,800] > 0 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 3 client.33723.objecter handle_osd_map decoding full epoch 800 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e289ab700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9e1006f5e0 0x7f9e10071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:01:52.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 20 client.33723.objecter dump_active .. 0 homeless 2024-04-02T05:01:52.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e289ab700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9e1006f5e0 0x7f9e10071aa0 secure :-1 s=READY pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b580 tx=0x7f9e14005fd0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:01:52.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.797+0000 7f9e1a7fc700 10 client.33723.objecter ms_handle_connect 0x7f9e1006f5e0 2024-04-02T05:01:52.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:52.801+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9e20014030 con 0x7f9e2406a1d0 2024-04-02T05:01:52.813 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:52.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:52.941 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:52.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:53.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.017+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f9e241a70a0 con 0x7f9e2406a1d0 2024-04-02T05:01:53.044 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:53.065 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:53.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.141+0000 7f9e1a7fc700 1 -- 172.21.15.69:0/3850849776 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v801) ==== 91+0+0 (secure 0 0 0) 0x7f9e2005f450 con 0x7f9e2406a1d0 2024-04-02T05:01:53.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 10 librados: watch_flush enter 2024-04-02T05:01:53.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 10 librados: watch_flush exit 2024-04-02T05:01:53.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 20 client.33723.objecter shutdown clearing up homeless session... 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 10 client.33723.objecter successfully canceled tick 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9e1006f5e0 msgr2=0x7f9e10071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9e1006f5e0 0x7f9e10071aa0 secure :-1 s=READY pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b580 tx=0x7f9e14005fd0 comp rx=0 tx=0).stop 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 msgr2=0x7f9e24101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 0x7f9e24101270 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c9b0 tx=0x7f9e2000cd70 comp rx=0 tx=0).stop 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 shutdown_connections 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f9e1006f5e0 0x7f9e10071aa0 secure :-1 s=CLOSED pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b580 tx=0x7f9e14005fd0 comp rx=0 tx=0).stop 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e24107620 0x7f9e24101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e2406a1d0 0x7f9e24101270 secure :-1 s=CLOSED pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c9b0 tx=0x7f9e2000cd70 comp rx=0 tx=0).stop 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 --2- 172.21.15.69:0/3850849776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2406ab80 0x7f9e241017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:01:53.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.145+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 >> 172.21.15.69:0/3850849776 conn(0x7f9e240771e0 msgr2=0x7f9e240ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:01:53.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.149+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 shutdown_connections 2024-04-02T05:01:53.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.149+0000 7f9e2b410700 1 -- 172.21.15.69:0/3850849776 wait complete. 2024-04-02T05:01:53.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:01:53.149+0000 7f9e2b410700 1 librados: shutdown 2024-04-02T05:01:53.163 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T05:01:53.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:53.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:53.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:53.313 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:53.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[28904]: cluster 2024-04-02T05:01:51.653329+0000 osd.7 (osd.7) 201 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:53.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[28904]: cluster 2024-04-02T05:01:51.653537+0000 osd.7 (osd.7) 202 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:53.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[28904]: cluster 2024-04-02T05:01:52.148345+0000 mon.a (mon.0) 3752 : cluster 0 osdmap e800: 8 total, 8 up, 7 in 2024-04-02T05:01:53.314 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[28904]: cluster 2024-04-02T05:01:52.325406+0000 mgr.y (mgr.14538) 704 : cluster 0 pgmap v1213: 192 pgs: 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[28904]: audit 2024-04-02T05:01:52.434227+0000 mon.c (mon.2) 1159 : audit 0 from='client.? 172.21.15.69:0/2543859676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[28904]: audit 2024-04-02T05:01:53.023492+0000 mon.a (mon.0) 3753 : audit 1 from='client.? 172.21.15.69:0/3850849776' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[21528]: cluster 2024-04-02T05:01:51.653329+0000 osd.7 (osd.7) 201 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[21528]: cluster 2024-04-02T05:01:51.653537+0000 osd.7 (osd.7) 202 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[21528]: cluster 2024-04-02T05:01:52.148345+0000 mon.a (mon.0) 3752 : cluster 0 osdmap e800: 8 total, 8 up, 7 in 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[21528]: cluster 2024-04-02T05:01:52.325406+0000 mgr.y (mgr.14538) 704 : cluster 0 pgmap v1213: 192 pgs: 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[21528]: audit 2024-04-02T05:01:52.434227+0000 mon.c (mon.2) 1159 : audit 0 from='client.? 172.21.15.69:0/2543859676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:53.315 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:53 smithi069 bash[21528]: audit 2024-04-02T05:01:53.023492+0000 mon.a (mon.0) 3753 : audit 1 from='client.? 172.21.15.69:0/3850849776' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:01:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:53.417 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[23790]: cluster 2024-04-02T05:01:51.653329+0000 osd.7 (osd.7) 201 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:53.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[23790]: cluster 2024-04-02T05:01:51.653537+0000 osd.7 (osd.7) 202 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:53.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[23790]: cluster 2024-04-02T05:01:52.148345+0000 mon.a (mon.0) 3752 : cluster 0 osdmap e800: 8 total, 8 up, 7 in 2024-04-02T05:01:53.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[23790]: cluster 2024-04-02T05:01:52.325406+0000 mgr.y (mgr.14538) 704 : cluster 0 pgmap v1213: 192 pgs: 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:01:53.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[23790]: audit 2024-04-02T05:01:52.434227+0000 mon.c (mon.2) 1159 : audit 0 from='client.? 172.21.15.69:0/2543859676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:01:53.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[23790]: audit 2024-04-02T05:01:53.023492+0000 mon.a (mon.0) 3753 : audit 1 from='client.? 172.21.15.69:0/3850849776' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:01:53.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:53.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:53.696 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:53.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "pg_update_log_missing(4.14 epoch 395/394 rep_tid 993 entries 395'14 (0'0) error 4:2b04a3e9:::notify.6:head by client.14415.0:2499 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:54.932320+0000", 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 478.87490464400003, 2024-04-02T05:01:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078416639999999999, 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932320+0000", 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932320+0000", 2024-04-02T05:01:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932320+0000", 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2300000000000001e-07 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932329+0000", 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8489999999999995e-06 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932345+0000", 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5594e-05 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.932362+0000", 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7847999999999999e-05 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.935886+0000", 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035239780000000001 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:54.940161+0000", 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042748719999999999 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.6,QREQUEST,e:398,from:3,reservation_nonce:3)", 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:53:58.130328+0000", 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 475.676895801, 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009765213, 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130328+0000", 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130328+0000", 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130329+0000", 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6700000000000003e-07 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130336+0000", 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7279999999999998e-06 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130347+0000", 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1447000000000001e-05 2024-04-02T05:01:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130356+0000", 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8459999999999997e-06 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130458+0000", 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101839 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.130468+0000", 2024-04-02T05:01:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5170000000000005e-06 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.139993+0000", 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095249900000000005 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.140023+0000", 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0608999999999998e-05 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:53:58.140094+0000", 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0469999999999994e-05 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2565 3.e 3:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:01.904153+0000", 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 471.90307135699999, 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011711485000000001, 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2565 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904153+0000", 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904153+0000", 2024-04-02T05:01:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904154+0000", 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.406e-06 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.904166+0000", 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2153e-05 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.909907+0000", 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057403050000000002 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.909922+0000", 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5135e-05 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910011+0000", 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9221999999999994e-05 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910127+0000", 2024-04-02T05:01:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115573 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910294+0000", 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000167171 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.910707+0000", 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041359600000000002 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.911193+0000", 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048597799999999997 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.915775+0000", 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045810590000000002 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.915824+0000", 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9920999999999999e-05 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.915864+0000", 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9966000000000003e-05 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:2567 3.e 3:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-04-02T05:01:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:01.918137+0000", 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 471.88908695999999, 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077450569999999996, 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2567 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918137+0000", 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918137+0000", 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918138+0000", 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.124e-06 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918145+0000", 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.477e-06 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918159+0000", 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4569999999999999e-05 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918166+0000", 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8979999999999999e-06 2024-04-02T05:01:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918226+0000", 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0010999999999999e-05 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918330+0000", 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103717 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.918554+0000", 2024-04-02T05:01:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000223923 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.919108+0000", 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055412099999999998 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.919531+0000", 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042319299999999998 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.925816+0000", 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062843250000000003 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.925853+0000", 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7339000000000003e-05 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:01.925882+0000", 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9359000000000001e-05 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25807.0:1 284.5s0 284:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:07.910149+0000", 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 465.89707546199998, 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0227512809999999, 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25807", 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2058437918", 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1 2024-04-02T05:01:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910149+0000", 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910149+0000", 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910152+0000", 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1329999999999999e-06 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910168+0000", 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5869e-05 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910196+0000", 2024-04-02T05:01:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.798e-05 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910246+0000", 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9818999999999999e-05 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910420+0000", 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017426400000000001 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:07.910427+0000", 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4429999999999997e-06 2024-04-02T05:01:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.930403+0000", 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.019975705 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.930592+0000", 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018934099999999999 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.931423+0000", 2024-04-02T05:01:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00083051700000000002 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.931945+0000", 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052247299999999997 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.932847+0000", 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00090180299999999996 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:08.932900+0000", 2024-04-02T05:01:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2933999999999998e-05 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.25442.0:5692 287.c 287:30000000:%01:::head [pgnls start_epoch 437] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e437)", 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:54:44.920753+0000", 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 428.88647148299998, 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018551937000000001, 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.25442", 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1136356552", 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5692 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920753+0000", 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920753+0000", 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920756+0000", 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9960000000000002e-06 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920771+0000", 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4922999999999999e-05 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920804+0000", 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3676000000000001e-05 2024-04-02T05:01:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920822+0000", 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7952999999999999e-05 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.920994+0000", 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017200399999999999 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.921015+0000", 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.936426+0000", 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015410902000000001 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.936486+0000", 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9821000000000002e-05 2024-04-02T05:01:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:54:44.939305+0000", 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028187400000000001 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(41.15,QREQUEST,e:470,from:0,reservation_nonce:3)", 2024-04-02T05:01:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:18.032436+0000", 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 395.77478837699999, 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01257341, 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032436+0000", 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032436+0000", 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032438+0000", 2024-04-02T05:01:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032447+0000", 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7960000000000007e-06 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032475+0000", 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7319999999999999e-05 2024-04-02T05:01:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032491+0000", 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6685999999999999e-05 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032667+0000", 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017570699999999999 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.032679+0000", 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.216e-05 2024-04-02T05:01:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.044882+0000", 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012202697 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.044914+0000", 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1596e-05 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:18.045009+0000", 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5665e-05 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(41.0,QREQUEST,e:472,from:4,reservation_nonce:3)", 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:55:20.027942+0000", 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 393.77928182099998, 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021636908, 2024-04-02T05:01:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027942+0000", 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027942+0000", 2024-04-02T05:01:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027944+0000", 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027955+0000", 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0936e-05 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.027992+0000", 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6973e-05 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.028013+0000", 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0862000000000001e-05 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.028153+0000", 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000140667 2024-04-02T05:01:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.028163+0000", 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8400000000000007e-06 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.049430+0000", 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021266835000000001 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.049472+0000", 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2120000000000003e-05 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:55:20.049579+0000", 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107091 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:92 102.19 102:9e3d24f0:test-rados-api-smithi069-72349-82::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:25.994122+0000", 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 267.81310227400002, 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014748196, 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994122+0000", 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994122+0000", 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994124+0000", 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.48e-06 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994137+0000", 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2853e-05 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994166+0000", 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8206000000000001e-05 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994184+0000", 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8482999999999999e-05 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994273+0000", 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8955999999999993e-05 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994281+0000", 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5120000000000008e-06 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000734+0000", 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064523369999999998 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000854+0000", 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011993300000000001 2024-04-02T05:01:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.001046+0000", 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000192413 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006085+0000", 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050385450000000002 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006149+0000", 2024-04-02T05:01:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4776000000000005e-05 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008780+0000", 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026304789999999998 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008825+0000", 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4566e-05 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008870+0000", 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5657e-05 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:97 102.19 102:9be20e43:test-rados-api-smithi069-72349-82::b74d4df232b43241f65ea39108ac0a45561a59cf:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:25.999239+0000", 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 267.80798521399998, 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010862856000000001, 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999239+0000", 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999239+0000", 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999242+0000", 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7659999999999999e-06 2024-04-02T05:01:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999257+0000", 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5497000000000001e-05 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999292+0000", 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5036000000000001e-05 2024-04-02T05:01:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999317+0000", 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5069999999999999e-05 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.005436+0000", 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061181559999999996 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.005584+0000", 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014880500000000001 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.005769+0000", 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000184861 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008764+0000", 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029949669999999999 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008882+0000", 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117718 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.009832+0000", 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094983899999999996 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.009988+0000", 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156033 2024-04-02T05:01:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.010102+0000", 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114108 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:98 102.1 102:84b9727a:test-rados-api-smithi069-72349-82::060e72eaff4fb3562782e1de4fe785e9648c08af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:25.999354+0000", 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 267.80787068299998, 2024-04-02T05:01:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012512808, 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999354+0000", 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999354+0000", 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999355+0000", 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5069999999999999e-06 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999364+0000", 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3859999999999996e-06 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999388+0000", 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3589999999999999e-05 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999406+0000", 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8312999999999999e-05 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006131+0000", 2024-04-02T05:01:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067249609999999998 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006264+0000", 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000132965 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,6", 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006476+0000", 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000211931 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008887+0000", 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024108649999999999 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.009800+0000", 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00091312000000000001 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011752+0000", 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001951997 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011823+0000", 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1167999999999994e-05 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011866+0000", 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3004999999999999e-05 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:100 102.1 102:83dfbecf:test-rados-api-smithi069-72349-82::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:26.000007+0000", 2024-04-02T05:01:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 267.80721705399998, 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012134123, 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000007+0000", 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000007+0000", 2024-04-02T05:01:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000009+0000", 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3170000000000001e-06 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000020+0000", 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1249e-05 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000057+0000", 2024-04-02T05:01:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6869000000000002e-05 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000068+0000", 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1333e-05 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006827+0000", 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067592370000000004 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006982+0000", 2024-04-02T05:01:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000155265 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,6", 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.007206+0000", 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022388400000000001 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008891+0000", 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016850789999999999 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011921+0000", 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030295209999999999 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.012052+0000", 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001315 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.012103+0000", 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0899000000000002e-05 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.012141+0000", 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7969999999999997e-05 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(10.8,QREQUEST,e:592,from:0,reservation_nonce:5)", 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:53.008120+0000", 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 240.79910452600001, 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066560220000000002, 2024-04-02T05:01:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008120+0000", 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008120+0000", 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008122+0000", 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5440000000000001e-06 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008136+0000", 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3766e-05 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008180+0000", 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3503e-05 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008196+0000", 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6786000000000002e-05 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008295+0000", 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8335e-05 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.008302+0000", 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3610000000000003e-06 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.014605+0000", 2024-04-02T05:01:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063030430000000004 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.014659+0000", 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4326000000000001e-05 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:53.014776+0000", 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116358 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.2.19:133 310.1a 310:5d12752b:test-rados-api-smithi069-72349-91::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e601)", 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:58:02.126813+0000", 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 231.68041132499999, 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094835509999999998, 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126813+0000", 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126813+0000", 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126815+0000", 2024-04-02T05:01:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6160000000000001e-06 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126821+0000", 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.144e-06 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126832+0000", 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1532000000000001e-05 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126842+0000", 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.363e-06 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126907+0000", 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5173999999999996e-05 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126915+0000", 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8169999999999999e-06 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.133860+0000", 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0069455849999999998 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.133954+0000", 2024-04-02T05:01:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3366e-05 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.134098+0000", 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.135599+0000", 2024-04-02T05:01:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001501198 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.135925+0000", 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032581400000000001 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.136219+0000", 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029453299999999998 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.136269+0000", 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9823000000000001e-05 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.136296+0000", 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7518e-05 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(3.7,QREQUEST,e:647,from:3,reservation_nonce:3)", 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:05.930835+0000", 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 167.876388908, 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012097088000000001, 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930835+0000", 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930835+0000", 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930837+0000", 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0030000000000001e-06 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930847+0000", 2024-04-02T05:01:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.005e-05 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930877+0000", 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9856e-05 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930893+0000", 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6192999999999999e-05 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.931053+0000", 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015919099999999999 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.931081+0000", 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7883000000000001e-05 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.942828+0000", 2024-04-02T05:01:53.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011747857 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.942860+0000", 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1136000000000001e-05 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.942932+0000", 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2918999999999998e-05 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16773.0:14 322.2 322:40000000::::head [pgnls start_epoch 686] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e686)", 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:44.234719+0000", 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 129.57250511300001, 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0076262120000000003, 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16773", 2024-04-02T05:01:53.900 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/903817344", 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234719+0000", 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234719+0000", 2024-04-02T05:01:53.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234721+0000", 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0449999999999999e-06 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234729+0000", 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5920000000000003e-06 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234742+0000", 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3077e-05 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234756+0000", 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3913999999999999e-05 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234906+0000", 2024-04-02T05:01:53.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015000300000000001 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.234920+0000", 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3747000000000001e-05 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.241923+0000", 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070036029999999997 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.242017+0000", 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3622000000000006e-05 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:44.242345+0000", 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000328609 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:2 323.1s0 323:91bcb6b0:test-rados-api-smithi069-72349-113::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:58.049814+0000", 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 115.757410601, 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030785092999999999, 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049814+0000", 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049814+0000", 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049816+0000", 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1670000000000002e-06 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049827+0000", 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0856000000000001e-05 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049849+0000", 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.198e-05 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049873+0000", 2024-04-02T05:01:53.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4510000000000001e-05 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.050076+0000", 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020266699999999999 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.050095+0000", 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9403e-05 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.071101+0000", 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021005280000000001 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.073818+0000", 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027173950000000001 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.074995+0000", 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001177261 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.076333+0000", 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001338234 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.080443+0000", 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004109522 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.080599+0000", 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000155818 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:475 328.1b 328:df8c41d3:test-rados-api-smithi069-72349-121::foo:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:50.055035+0000", 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 63.752188859999997, 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028280998000000002, 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 475 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055035+0000", 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055035+0000", 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055038+0000", 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0340000000000001e-06 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055053+0000", 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4163e-05 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055092+0000", 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9128999999999998e-05 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055111+0000", 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9680000000000001e-05 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055283+0000", 2024-04-02T05:01:53.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017113499999999999 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055296+0000", 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3883000000000001e-05 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.068291+0000", 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012994309000000001 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.068851+0000", 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055996599999999996 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.073683+0000", 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048323730000000001 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.073948+0000", 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026499599999999998 2024-04-02T05:01:53.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.074377+0000", 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042873800000000001 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.075272+0000", 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089567200000000003 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.081520+0000", 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062475730000000002 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.083031+0000", 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00151052 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.083194+0000", 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016390799999999999 2024-04-02T05:01:53.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.083316+0000", 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121919 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(2.12,QREQUEST,e:751,from:3,reservation_nonce:3)", 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:01.656229+0000", 2024-04-02T05:01:53.920 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.150995061000003, 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01403385, 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656229+0000", 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656229+0000", 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656232+0000", 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7829999999999999e-06 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656248+0000", 2024-04-02T05:01:53.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6062999999999999e-05 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656285+0000", 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6579999999999999e-05 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656303+0000", 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8185999999999998e-05 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656443+0000", 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014039799999999999 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656455+0000", 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.221e-05 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.670100+0000", 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013644269000000001 2024-04-02T05:01:53.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.670145+0000", 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5291999999999999e-05 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.670263+0000", 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118069 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1243 332.1e 332:78000000::::head [pgnls start_epoch 778] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:30.768520+0000", 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.038703777999999, 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095984439999999994, 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1243 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.926 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768520+0000", 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768520+0000", 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768524+0000", 2024-04-02T05:01:53.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0230000000000001e-06 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768540+0000", 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6286e-05 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768579+0000", 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9515999999999999e-05 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:01:53.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768598+0000", 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8474e-05 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768748+0000", 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015029099999999999 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768759+0000", 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0943e-05 2024-04-02T05:01:53.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.777709+0000", 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089495300000000007 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.777820+0000", 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011117500000000001 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:01:53.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.778119+0000", 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029920599999999999 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:01:53.931 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:01:53.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:01:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:01:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:53.932 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:53.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:54.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:54.061 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:01:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:01:54.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:54.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.327 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:54.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:54.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:54 smithi098 bash[23790]: cluster 2024-04-02T05:01:53.136514+0000 mon.a (mon.0) 3754 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:54.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:54 smithi098 bash[23790]: audit 2024-04-02T05:01:53.145397+0000 mon.a (mon.0) 3755 : audit 1 from='client.? 172.21.15.69:0/3850849776' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:01:54.379 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:54 smithi098 bash[23790]: cluster 2024-04-02T05:01:53.154958+0000 mon.a (mon.0) 3756 : cluster 0 osdmap e801: 8 total, 8 up, 7 in 2024-04-02T05:01:54.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:54 smithi098 bash[23790]: audit 2024-04-02T05:01:53.161420+0000 mon.a (mon.0) 3757 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:54.380 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:54 smithi098 bash[23790]: audit 2024-04-02T05:01:53.679900+0000 mgr.y (mgr.14538) 705 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:54.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[28904]: cluster 2024-04-02T05:01:53.136514+0000 mon.a (mon.0) 3754 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[28904]: audit 2024-04-02T05:01:53.145397+0000 mon.a (mon.0) 3755 : audit 1 from='client.? 172.21.15.69:0/3850849776' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[28904]: cluster 2024-04-02T05:01:53.154958+0000 mon.a (mon.0) 3756 : cluster 0 osdmap e801: 8 total, 8 up, 7 in 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[28904]: audit 2024-04-02T05:01:53.161420+0000 mon.a (mon.0) 3757 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[28904]: audit 2024-04-02T05:01:53.679900+0000 mgr.y (mgr.14538) 705 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[21528]: cluster 2024-04-02T05:01:53.136514+0000 mon.a (mon.0) 3754 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[21528]: audit 2024-04-02T05:01:53.145397+0000 mon.a (mon.0) 3755 : audit 1 from='client.? 172.21.15.69:0/3850849776' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[21528]: cluster 2024-04-02T05:01:53.154958+0000 mon.a (mon.0) 3756 : cluster 0 osdmap e801: 8 total, 8 up, 7 in 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[21528]: audit 2024-04-02T05:01:53.161420+0000 mon.a (mon.0) 3757 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:01:54.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:54 smithi069 bash[21528]: audit 2024-04-02T05:01:53.679900+0000 mgr.y (mgr.14538) 705 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:01:54.455 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:54.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:54.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:54.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:54.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-02T05:01:54.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:54.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:54.972 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:54.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:55.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:55.106 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:55.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:55.236 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:55.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:55.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: audit 2024-04-02T05:01:54.148740+0000 mon.a (mon.0) 3758 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: cluster 2024-04-02T05:01:54.157760+0000 mon.a (mon.0) 3759 : cluster 0 osdmap e802: 8 total, 8 up, 7 in 2024-04-02T05:01:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: audit 2024-04-02T05:01:54.174816+0000 mon.a (mon.0) 3760 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: cluster 2024-04-02T05:01:54.325914+0000 mgr.y (mgr.14538) 706 : cluster 0 pgmap v1216: 224 pgs: 11 creating+peering, 21 unknown, 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:01:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: audit 2024-04-02T05:01:55.083107+0000 mon.a (mon.0) 3761 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: cluster 2024-04-02T05:01:55.091767+0000 mon.a (mon.0) 3762 : cluster 0 osdmap e803: 8 total, 8 up, 7 in 2024-04-02T05:01:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:55 smithi098 bash[23790]: audit 2024-04-02T05:01:55.094705+0000 mon.a (mon.0) 3763 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-134"}]: dispatch 2024-04-02T05:01:55.480 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: audit 2024-04-02T05:01:54.148740+0000 mon.a (mon.0) 3758 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: cluster 2024-04-02T05:01:54.157760+0000 mon.a (mon.0) 3759 : cluster 0 osdmap e802: 8 total, 8 up, 7 in 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: audit 2024-04-02T05:01:54.174816+0000 mon.a (mon.0) 3760 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: cluster 2024-04-02T05:01:54.325914+0000 mgr.y (mgr.14538) 706 : cluster 0 pgmap v1216: 224 pgs: 11 creating+peering, 21 unknown, 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: audit 2024-04-02T05:01:55.083107+0000 mon.a (mon.0) 3761 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: cluster 2024-04-02T05:01:55.091767+0000 mon.a (mon.0) 3762 : cluster 0 osdmap e803: 8 total, 8 up, 7 in 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[21528]: audit 2024-04-02T05:01:55.094705+0000 mon.a (mon.0) 3763 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-134"}]: dispatch 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: audit 2024-04-02T05:01:54.148740+0000 mon.a (mon.0) 3758 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: cluster 2024-04-02T05:01:54.157760+0000 mon.a (mon.0) 3759 : cluster 0 osdmap e802: 8 total, 8 up, 7 in 2024-04-02T05:01:55.481 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: audit 2024-04-02T05:01:54.174816+0000 mon.a (mon.0) 3760 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:01:55.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: cluster 2024-04-02T05:01:54.325914+0000 mgr.y (mgr.14538) 706 : cluster 0 pgmap v1216: 224 pgs: 11 creating+peering, 21 unknown, 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:01:55.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: audit 2024-04-02T05:01:55.083107+0000 mon.a (mon.0) 3761 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:01:55.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: cluster 2024-04-02T05:01:55.091767+0000 mon.a (mon.0) 3762 : cluster 0 osdmap e803: 8 total, 8 up, 7 in 2024-04-02T05:01:55.482 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:55 smithi069 bash[28904]: audit 2024-04-02T05:01:55.094705+0000 mon.a (mon.0) 3763 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-134"}]: dispatch 2024-04-02T05:01:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:55.599 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:55.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T05:01:55.626 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:55.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:55.854 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:55.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:55.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:55.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:56.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:56.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:56.138 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:56.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: cluster 2024-04-02T05:01:55.108066+0000 osd.0 (osd.0) 385 : cluster 0 6.0 scrub starts 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: cluster 2024-04-02T05:01:55.111498+0000 osd.0 (osd.0) 386 : cluster 0 6.0 scrub ok 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: cluster 2024-04-02T05:01:55.392839+0000 osd.4 (osd.4) 386 : cluster 0 2.f scrub starts 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: cluster 2024-04-02T05:01:55.395233+0000 osd.4 (osd.4) 387 : cluster 0 2.f scrub ok 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: audit 2024-04-02T05:01:56.086032+0000 mon.a (mon.0) 3764 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-134"}]': finished 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.093590+0000 mon.a (mon.0) 3765 : cluster 0 osdmap e804: 8 total, 8 up, 7 in 2024-04-02T05:01:56.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:56 smithi098 bash[23790]: audit 2024-04-02T05:01:56.097167+0000 mon.a (mon.0) 3766 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-134", "mode": "writeback"}]: dispatch 2024-04-02T05:01:56.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T05:01:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:56.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: cluster 2024-04-02T05:01:55.108066+0000 osd.0 (osd.0) 385 : cluster 0 6.0 scrub starts 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: cluster 2024-04-02T05:01:55.111498+0000 osd.0 (osd.0) 386 : cluster 0 6.0 scrub ok 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: cluster 2024-04-02T05:01:55.392839+0000 osd.4 (osd.4) 386 : cluster 0 2.f scrub starts 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: cluster 2024-04-02T05:01:55.395233+0000 osd.4 (osd.4) 387 : cluster 0 2.f scrub ok 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: audit 2024-04-02T05:01:56.086032+0000 mon.a (mon.0) 3764 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-134"}]': finished 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.093590+0000 mon.a (mon.0) 3765 : cluster 0 osdmap e804: 8 total, 8 up, 7 in 2024-04-02T05:01:56.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[21528]: audit 2024-04-02T05:01:56.097167+0000 mon.a (mon.0) 3766 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-134", "mode": "writeback"}]: dispatch 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: cluster 2024-04-02T05:01:55.108066+0000 osd.0 (osd.0) 385 : cluster 0 6.0 scrub starts 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: cluster 2024-04-02T05:01:55.111498+0000 osd.0 (osd.0) 386 : cluster 0 6.0 scrub ok 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: cluster 2024-04-02T05:01:55.392839+0000 osd.4 (osd.4) 386 : cluster 0 2.f scrub starts 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: cluster 2024-04-02T05:01:55.395233+0000 osd.4 (osd.4) 387 : cluster 0 2.f scrub ok 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: audit 2024-04-02T05:01:56.086032+0000 mon.a (mon.0) 3764 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-134"}]': finished 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.093590+0000 mon.a (mon.0) 3765 : cluster 0 osdmap e804: 8 total, 8 up, 7 in 2024-04-02T05:01:56.385 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:56 smithi069 bash[28904]: audit 2024-04-02T05:01:56.097167+0000 mon.a (mon.0) 3766 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-134", "mode": "writeback"}]: dispatch 2024-04-02T05:01:56.398 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:56.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:56.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:56.651 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:56.754 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.783 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:56.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:56.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:56.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:57.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:57.041 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:57.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:57.247 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:57.274 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:57.293 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:57.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:57.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:57.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:57 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.132627+0000 osd.0 (osd.0) 387 : cluster 0 3.6 scrub starts 2024-04-02T05:01:57.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:57 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.137150+0000 osd.0 (osd.0) 388 : cluster 0 3.6 scrub ok 2024-04-02T05:01:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:57 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.326718+0000 mgr.y (mgr.14538) 707 : cluster 0 pgmap v1219: 224 pgs: 11 creating+peering, 21 unknown, 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:57 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.086632+0000 mon.a (mon.0) 3767 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:57 smithi098 bash[23790]: audit 2024-04-02T05:01:57.088944+0000 mon.a (mon.0) 3768 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-134", "mode": "writeback"}]': finished 2024-04-02T05:01:57.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:57 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.097307+0000 mon.a (mon.0) 3769 : cluster 0 osdmap e805: 8 total, 8 up, 7 in 2024-04-02T05:01:57.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.132627+0000 osd.0 (osd.0) 387 : cluster 0 3.6 scrub starts 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.137150+0000 osd.0 (osd.0) 388 : cluster 0 3.6 scrub ok 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.326718+0000 mgr.y (mgr.14538) 707 : cluster 0 pgmap v1219: 224 pgs: 11 creating+peering, 21 unknown, 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.086632+0000 mon.a (mon.0) 3767 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[28904]: audit 2024-04-02T05:01:57.088944+0000 mon.a (mon.0) 3768 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-134", "mode": "writeback"}]': finished 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.097307+0000 mon.a (mon.0) 3769 : cluster 0 osdmap e805: 8 total, 8 up, 7 in 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.132627+0000 osd.0 (osd.0) 387 : cluster 0 3.6 scrub starts 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.137150+0000 osd.0 (osd.0) 388 : cluster 0 3.6 scrub ok 2024-04-02T05:01:57.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.326718+0000 mgr.y (mgr.14538) 707 : cluster 0 pgmap v1219: 224 pgs: 11 creating+peering, 21 unknown, 192 active+clean; 582 KiB data, 733 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:01:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.086632+0000 mon.a (mon.0) 3767 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:01:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[21528]: audit 2024-04-02T05:01:57.088944+0000 mon.a (mon.0) 3768 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-134", "mode": "writeback"}]': finished 2024-04-02T05:01:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:57 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.097307+0000 mon.a (mon.0) 3769 : cluster 0 osdmap e805: 8 total, 8 up, 7 in 2024-04-02T05:01:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:57.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:57.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:57.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.943 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:57.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:58.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:58.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:58.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:58.326 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:01:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:01:58.429 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.349148+0000 osd.7 (osd.7) 203 : cluster 0 10.1 scrub starts 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.352238+0000 osd.7 (osd.7) 204 : cluster 0 10.1 scrub ok 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.834056+0000 osd.3 (osd.3) 403 : cluster 0 3.1e scrub starts 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:56.836907+0000 osd.3 (osd.3) 404 : cluster 0 3.1e scrub ok 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.160471+0000 osd.5 (osd.5) 261 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.160719+0000 osd.5 (osd.5) 262 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: audit 2024-04-02T05:01:57.201855+0000 mon.a (mon.0) 3770 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.329378+0000 osd.4 (osd.4) 388 : cluster 0 5.1 scrub starts 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.331506+0000 osd.4 (osd.4) 389 : cluster 0 5.1 scrub ok 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: audit 2024-04-02T05:01:58.091853+0000 mon.a (mon.0) 3771 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: cluster 2024-04-02T05:01:58.099716+0000 mon.a (mon.0) 3772 : cluster 0 osdmap e806: 8 total, 8 up, 7 in 2024-04-02T05:01:58.441 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[28904]: audit 2024-04-02T05:01:58.101281+0000 mon.a (mon.0) 3773 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134"}]: dispatch 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.349148+0000 osd.7 (osd.7) 203 : cluster 0 10.1 scrub starts 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.352238+0000 osd.7 (osd.7) 204 : cluster 0 10.1 scrub ok 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.834056+0000 osd.3 (osd.3) 403 : cluster 0 3.1e scrub starts 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:56.836907+0000 osd.3 (osd.3) 404 : cluster 0 3.1e scrub ok 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.160471+0000 osd.5 (osd.5) 261 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.160719+0000 osd.5 (osd.5) 262 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: audit 2024-04-02T05:01:57.201855+0000 mon.a (mon.0) 3770 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.329378+0000 osd.4 (osd.4) 388 : cluster 0 5.1 scrub starts 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.331506+0000 osd.4 (osd.4) 389 : cluster 0 5.1 scrub ok 2024-04-02T05:01:58.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: audit 2024-04-02T05:01:58.091853+0000 mon.a (mon.0) 3771 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:58.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: cluster 2024-04-02T05:01:58.099716+0000 mon.a (mon.0) 3772 : cluster 0 osdmap e806: 8 total, 8 up, 7 in 2024-04-02T05:01:58.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21528]: audit 2024-04-02T05:01:58.101281+0000 mon.a (mon.0) 3773 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134"}]: dispatch 2024-04-02T05:01:58.454 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:01:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:01:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.349148+0000 osd.7 (osd.7) 203 : cluster 0 10.1 scrub starts 2024-04-02T05:01:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.352238+0000 osd.7 (osd.7) 204 : cluster 0 10.1 scrub ok 2024-04-02T05:01:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.834056+0000 osd.3 (osd.3) 403 : cluster 0 3.1e scrub starts 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:56.836907+0000 osd.3 (osd.3) 404 : cluster 0 3.1e scrub ok 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.160471+0000 osd.5 (osd.5) 261 : cluster 0 purged_snaps scrub starts 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.160719+0000 osd.5 (osd.5) 262 : cluster 0 purged_snaps scrub ok 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: audit 2024-04-02T05:01:57.201855+0000 mon.a (mon.0) 3770 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.329378+0000 osd.4 (osd.4) 388 : cluster 0 5.1 scrub starts 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.331506+0000 osd.4 (osd.4) 389 : cluster 0 5.1 scrub ok 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: audit 2024-04-02T05:01:58.091853+0000 mon.a (mon.0) 3771 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: cluster 2024-04-02T05:01:58.099716+0000 mon.a (mon.0) 3772 : cluster 0 osdmap e806: 8 total, 8 up, 7 in 2024-04-02T05:01:58.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:58 smithi098 bash[23790]: audit 2024-04-02T05:01:58.101281+0000 mon.a (mon.0) 3773 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134"}]: dispatch 2024-04-02T05:01:58.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:58.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:58.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:01:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:01:58.742 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:01:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:01:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:01:58.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:58.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:58.965 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:58.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:59.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.095 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:59.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.568348+0000 osd.6 (osd.6) 331 : cluster 0 3.10 scrub starts 2024-04-02T05:01:59.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:59.223 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:01:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:01:59.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[21528]: cluster 2024-04-02T05:01:57.570779+0000 osd.6 (osd.6) 332 : cluster 0 3.10 scrub ok 2024-04-02T05:01:59.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[21528]: cluster 2024-04-02T05:01:58.327761+0000 mgr.y (mgr.14538) 708 : cluster 0 pgmap v1222: 224 pgs: 5 creating+peering, 219 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-02T05:01:59.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[21528]: cluster 2024-04-02T05:01:59.092420+0000 mon.a (mon.0) 3774 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[21528]: audit 2024-04-02T05:01:59.094662+0000 mon.a (mon.0) 3775 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134"}]': finished 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[21528]: cluster 2024-04-02T05:01:59.102086+0000 mon.a (mon.0) 3776 : cluster 0 osdmap e807: 8 total, 8 up, 7 in 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.568348+0000 osd.6 (osd.6) 331 : cluster 0 3.10 scrub starts 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[28904]: cluster 2024-04-02T05:01:57.570779+0000 osd.6 (osd.6) 332 : cluster 0 3.10 scrub ok 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[28904]: cluster 2024-04-02T05:01:58.327761+0000 mgr.y (mgr.14538) 708 : cluster 0 pgmap v1222: 224 pgs: 5 creating+peering, 219 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[28904]: cluster 2024-04-02T05:01:59.092420+0000 mon.a (mon.0) 3774 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[28904]: audit 2024-04-02T05:01:59.094662+0000 mon.a (mon.0) 3775 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134"}]': finished 2024-04-02T05:01:59.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:01:59 smithi069 bash[28904]: cluster 2024-04-02T05:01:59.102086+0000 mon.a (mon.0) 3776 : cluster 0 osdmap e807: 8 total, 8 up, 7 in 2024-04-02T05:01:59.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:59.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:59 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.568348+0000 osd.6 (osd.6) 331 : cluster 0 3.10 scrub starts 2024-04-02T05:01:59.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:59 smithi098 bash[23790]: cluster 2024-04-02T05:01:57.570779+0000 osd.6 (osd.6) 332 : cluster 0 3.10 scrub ok 2024-04-02T05:01:59.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:59 smithi098 bash[23790]: cluster 2024-04-02T05:01:58.327761+0000 mgr.y (mgr.14538) 708 : cluster 0 pgmap v1222: 224 pgs: 5 creating+peering, 219 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-02T05:01:59.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:59 smithi098 bash[23790]: cluster 2024-04-02T05:01:59.092420+0000 mon.a (mon.0) 3774 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:01:59.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:59 smithi098 bash[23790]: audit 2024-04-02T05:01:59.094662+0000 mon.a (mon.0) 3775 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-134"}]': finished 2024-04-02T05:01:59.338 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:01:59 smithi098 bash[23790]: cluster 2024-04-02T05:01:59.102086+0000 mon.a (mon.0) 3776 : cluster 0 osdmap e807: 8 total, 8 up, 7 in 2024-04-02T05:01:59.354 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:01:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:01:59.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:59.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:59.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:59.746 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:01:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:01:59.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.878 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:01:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:01:59.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:01:59.904 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:01:59.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:00.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:00.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:00.268 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:00.292 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:00.293 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:00.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:00.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:00.527 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:00.630 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:00.637 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[21528]: cluster 2024-04-02T05:01:58.746743+0000 osd.3 (osd.3) 405 : cluster 0 6.6 scrub starts 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[21528]: cluster 2024-04-02T05:01:58.749121+0000 osd.3 (osd.3) 406 : cluster 0 6.6 scrub ok 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[21528]: cluster 2024-04-02T05:01:59.052846+0000 osd.1 (osd.1) 391 : cluster 0 2.15 scrub starts 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[21528]: cluster 2024-04-02T05:01:59.055164+0000 osd.1 (osd.1) 392 : cluster 0 2.15 scrub ok 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.093921+0000 mon.a (mon.0) 3777 : cluster 0 osdmap e808: 8 total, 8 up, 7 in 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[28904]: cluster 2024-04-02T05:01:58.746743+0000 osd.3 (osd.3) 405 : cluster 0 6.6 scrub starts 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[28904]: cluster 2024-04-02T05:01:58.749121+0000 osd.3 (osd.3) 406 : cluster 0 6.6 scrub ok 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[28904]: cluster 2024-04-02T05:01:59.052846+0000 osd.1 (osd.1) 391 : cluster 0 2.15 scrub starts 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[28904]: cluster 2024-04-02T05:01:59.055164+0000 osd.1 (osd.1) 392 : cluster 0 2.15 scrub ok 2024-04-02T05:02:00.638 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:00 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.093921+0000 mon.a (mon.0) 3777 : cluster 0 osdmap e808: 8 total, 8 up, 7 in 2024-04-02T05:02:00.652 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:00.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:00.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:00 smithi098 bash[23790]: cluster 2024-04-02T05:01:58.746743+0000 osd.3 (osd.3) 405 : cluster 0 6.6 scrub starts 2024-04-02T05:02:00.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:00 smithi098 bash[23790]: cluster 2024-04-02T05:01:58.749121+0000 osd.3 (osd.3) 406 : cluster 0 6.6 scrub ok 2024-04-02T05:02:00.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:00 smithi098 bash[23790]: cluster 2024-04-02T05:01:59.052846+0000 osd.1 (osd.1) 391 : cluster 0 2.15 scrub starts 2024-04-02T05:02:00.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:00 smithi098 bash[23790]: cluster 2024-04-02T05:01:59.055164+0000 osd.1 (osd.1) 392 : cluster 0 2.15 scrub ok 2024-04-02T05:02:00.767 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:00 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.093921+0000 mon.a (mon.0) 3777 : cluster 0 osdmap e808: 8 total, 8 up, 7 in 2024-04-02T05:02:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:00.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:00.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:01.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:01.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:01.169 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:01.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:01.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:01.403 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:01.425 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:01.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:01.554 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:01.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:01.686 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:01.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:59.661378+0000 osd.6 (osd.6) 333 : cluster 0 10.1c scrub starts 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:01:59.663035+0000 osd.6 (osd.6) 334 : cluster 0 10.1c scrub ok 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.098272+0000 osd.1 (osd.1) 393 : cluster 0 2.a scrub starts 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.101581+0000 osd.1 (osd.1) 394 : cluster 0 2.a scrub ok 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.191537+0000 osd.0 (osd.0) 389 : cluster 0 3.1f scrub starts 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.195582+0000 osd.0 (osd.0) 390 : cluster 0 3.1f scrub ok 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.328221+0000 mgr.y (mgr.14538) 709 : cluster 0 pgmap v1225: 192 pgs: 192 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.346716+0000 osd.4 (osd.4) 390 : cluster 0 10.e scrub starts 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.349225+0000 osd.4 (osd.4) 391 : cluster 0 10.e scrub ok 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.689296+0000 osd.6 (osd.6) 335 : cluster 0 6.1e scrub starts 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:00.691402+0000 osd.6 (osd.6) 336 : cluster 0 6.1e scrub ok 2024-04-02T05:02:01.800 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.109357+0000 mon.a (mon.0) 3778 : cluster 0 osdmap e809: 8 total, 8 up, 7 in 2024-04-02T05:02:01.801 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:01 smithi098 bash[23790]: audit 2024-04-02T05:02:01.119290+0000 mon.a (mon.0) 3779 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:01.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:01.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:59.661378+0000 osd.6 (osd.6) 333 : cluster 0 10.1c scrub starts 2024-04-02T05:02:01.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:01:59.663035+0000 osd.6 (osd.6) 334 : cluster 0 10.1c scrub ok 2024-04-02T05:02:01.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.098272+0000 osd.1 (osd.1) 393 : cluster 0 2.a scrub starts 2024-04-02T05:02:01.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.101581+0000 osd.1 (osd.1) 394 : cluster 0 2.a scrub ok 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.191537+0000 osd.0 (osd.0) 389 : cluster 0 3.1f scrub starts 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.195582+0000 osd.0 (osd.0) 390 : cluster 0 3.1f scrub ok 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.328221+0000 mgr.y (mgr.14538) 709 : cluster 0 pgmap v1225: 192 pgs: 192 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.346716+0000 osd.4 (osd.4) 390 : cluster 0 10.e scrub starts 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.349225+0000 osd.4 (osd.4) 391 : cluster 0 10.e scrub ok 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.689296+0000 osd.6 (osd.6) 335 : cluster 0 6.1e scrub starts 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:00.691402+0000 osd.6 (osd.6) 336 : cluster 0 6.1e scrub ok 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.109357+0000 mon.a (mon.0) 3778 : cluster 0 osdmap e809: 8 total, 8 up, 7 in 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[21528]: audit 2024-04-02T05:02:01.119290+0000 mon.a (mon.0) 3779 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:59.661378+0000 osd.6 (osd.6) 333 : cluster 0 10.1c scrub starts 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:01:59.663035+0000 osd.6 (osd.6) 334 : cluster 0 10.1c scrub ok 2024-04-02T05:02:01.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.098272+0000 osd.1 (osd.1) 393 : cluster 0 2.a scrub starts 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.101581+0000 osd.1 (osd.1) 394 : cluster 0 2.a scrub ok 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.191537+0000 osd.0 (osd.0) 389 : cluster 0 3.1f scrub starts 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.195582+0000 osd.0 (osd.0) 390 : cluster 0 3.1f scrub ok 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.328221+0000 mgr.y (mgr.14538) 709 : cluster 0 pgmap v1225: 192 pgs: 192 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.346716+0000 osd.4 (osd.4) 390 : cluster 0 10.e scrub starts 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.349225+0000 osd.4 (osd.4) 391 : cluster 0 10.e scrub ok 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.689296+0000 osd.6 (osd.6) 335 : cluster 0 6.1e scrub starts 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:00.691402+0000 osd.6 (osd.6) 336 : cluster 0 6.1e scrub ok 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.109357+0000 mon.a (mon.0) 3778 : cluster 0 osdmap e809: 8 total, 8 up, 7 in 2024-04-02T05:02:01.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:01 smithi069 bash[28904]: audit 2024-04-02T05:02:01.119290+0000 mon.a (mon.0) 3779 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:01.944 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:01.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:02.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:02.074 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:02.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:02.201 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:02.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:02.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:02.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:02.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:02.596 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:02.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.366571+0000 osd.4 (osd.4) 392 : cluster 0 3.1a scrub starts 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.372760+0000 osd.4 (osd.4) 393 : cluster 0 3.1a scrub ok 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.696834+0000 osd.6 (osd.6) 337 : cluster 0 2.11 scrub starts 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.698794+0000 osd.6 (osd.6) 338 : cluster 0 2.11 scrub ok 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.799900+0000 osd.2 (osd.2) 351 : cluster 0 3.9 scrub starts 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:01.804342+0000 osd.2 (osd.2) 352 : cluster 0 3.9 scrub ok 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: audit 2024-04-02T05:02:02.093009+0000 mon.a (mon.0) 3780 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: cluster 2024-04-02T05:02:02.103289+0000 mon.a (mon.0) 3781 : cluster 0 osdmap e810: 8 total, 8 up, 7 in 2024-04-02T05:02:02.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[21528]: audit 2024-04-02T05:02:02.128087+0000 mon.a (mon.0) 3782 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.366571+0000 osd.4 (osd.4) 392 : cluster 0 3.1a scrub starts 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.372760+0000 osd.4 (osd.4) 393 : cluster 0 3.1a scrub ok 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.696834+0000 osd.6 (osd.6) 337 : cluster 0 2.11 scrub starts 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.698794+0000 osd.6 (osd.6) 338 : cluster 0 2.11 scrub ok 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.799900+0000 osd.2 (osd.2) 351 : cluster 0 3.9 scrub starts 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:01.804342+0000 osd.2 (osd.2) 352 : cluster 0 3.9 scrub ok 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: audit 2024-04-02T05:02:02.093009+0000 mon.a (mon.0) 3780 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: cluster 2024-04-02T05:02:02.103289+0000 mon.a (mon.0) 3781 : cluster 0 osdmap e810: 8 total, 8 up, 7 in 2024-04-02T05:02:02.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:02 smithi069 bash[28904]: audit 2024-04-02T05:02:02.128087+0000 mon.a (mon.0) 3782 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.723 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:02.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:02.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:02.847 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 -- 172.21.15.69:0/2966072046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 msgr2=0x7f110c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 --2- 172.21.15.69:0/2966072046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 0x7f110c106ec0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009920 tx=0x7f10f8009c30 comp rx=0 tx=0).stop 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 -- 172.21.15.69:0/2966072046 shutdown_connections 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 --2- 172.21.15.69:0/2966072046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c1075a0 0x7f110c10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 --2- 172.21.15.69:0/2966072046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f110c102520 0x7f110c102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 --2- 172.21.15.69:0/2966072046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 0x7f110c106ec0 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 -- 172.21.15.69:0/2966072046 >> 172.21.15.69:0/2966072046 conn(0x7f110c079360 msgr2=0x7f110c079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 -- 172.21.15.69:0/2966072046 shutdown_connections 2024-04-02T05:02:02.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.853+0000 7f1112a71700 1 -- 172.21.15.69:0/2966072046 wait complete. 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 librados: starting msgr at 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 librados: starting objecter 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 Processor -- start 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- start start 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 librados: setting wanted keys 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 librados: calling monclient init 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c102520 0x7f110c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 0x7f110c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f110c1075a0 0x7f110c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f110c111af0 con 0x7f110c102520 2024-04-02T05:02:02.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f110c111970 con 0x7f110c1075a0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f110c111c70 con 0x7f110c102ed0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 0x7f110c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f111080d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c102520 0x7f110c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 0x7f110c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37838/0 (socket says 172.21.15.69:37838) 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 -- 172.21.15.69:0/4089340474 learned_addr learned my addr 172.21.15.69:0/4089340474 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 -- 172.21.15.69:0/4089340474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f110c1075a0 msgr2=0x7f110c19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f110c1075a0 0x7f110c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 -- 172.21.15.69:0/4089340474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c102520 msgr2=0x7f110c1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c102520 0x7f110c1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 -- 172.21.15.69:0/4089340474 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c19d1e0 con 0x7f110c102ed0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f110bfff700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c102ed0 0x7f110c198710 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7f10f8000c00 tx=0x7f10f800fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 10 client.?.objecter ms_handle_connect 0x7f110c102ed0 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:02.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f800b710 con 0x7f110c102ed0 2024-04-02T05:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- 172.21.15.69:0/4089340474 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110c19d470 con 0x7f110c102ed0 2024-04-02T05:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f10f8005040 con 0x7f110c102ed0 2024-04-02T05:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8003990 con 0x7f110c102ed0 2024-04-02T05:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 10 client.43793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- 172.21.15.69:0/4089340474 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f110c1a4d50 con 0x7f110c102ed0 2024-04-02T05:02:02.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 librados: init done 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 10 client.43793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 15 client.43793.objecter _session_command_op_assign -1 1 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 20 client.43793.objecter _get_session osd=-1 returning homeless 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 10 client.43793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1112a71700 1 -- 172.21.15.69:0/4089340474 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f110c1039a0 con 0x7f110c102ed0 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f10f8005540 con 0x7f110c102ed0 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f10fc06f5e0 0x7f10fc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:02.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(810..810 src has 74..810) ==== 10112+0+0 (secure 0 0 0) 0x7f10f808fb90 con 0x7f110c102ed0 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 10 client.43793.objecter ms_dispatch 0x7f110c001550 osd_map(810..810 src has 74..810) 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 3 client.43793.objecter handle_osd_map got epochs [810,810] > 0 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f1109ffb700 3 client.43793.objecter handle_osd_map decoding full epoch 810 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.857+0000 7f111080d700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f10fc06f5e0 0x7f10fc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 10 client.43793.objecter checking command 1 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f10fc07d2f0 0x7f10fc07f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter _get_session s=0x7f10fc07cc20 osd=7 3 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter put_session s=0x7f10fc07cc20 osd=7 3 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 15 client.43793.objecter _session_command_op_remove -1 1 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter _get_session s=0x7f10fc07cc20 osd=7 3 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter get_session s=0x7f10fc07cc20 osd=7 3 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 15 client.43793.objecter _session_command_op_assign 7 1 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter put_session s=0x7f10fc07cc20 osd=7 4 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 10 client.43793.objecter _send_command 1 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10f8009080 con 0x7f10fc07d2f0 2024-04-02T05:02:02.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter dump_active .. 0 homeless 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=810) ==== 24+0+0 (secure 0 0 0) 0x7f10f80942a0 con 0x7f110c102ed0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f111080d700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f10fc06f5e0 0x7f10fc071aa0 secure :-1 s=READY pgs=4012 cs=0 l=1 rev1=1 crypto rx=0x7f1100009a10 tx=0x7f1100009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 10 client.43793.objecter ms_handle_connect 0x7f10fc06f5e0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f111100e700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f10fc07d2f0 0x7f10fc07f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f111100e700 1 --2- 172.21.15.69:0/4089340474 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f10fc07d2f0 0x7f10fc07f790 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 10 client.43793.objecter ms_handle_connect 0x7f10fc07d2f0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 1 -- 172.21.15.69:0/4089340474 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10f8009080 con 0x7f10fc07d2f0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 10 client.43793.objecter ms_dispatch 0x7f110c001550 command_reply(tid 1: 0 ) 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 10 client.43793.objecter _finish_command 1 = system:0 2024-04-02T05:02:02.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 20 client.43793.objecter put_session s=0x7f10fc07cc20 osd=7 4 2024-04-02T05:02:02.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:02.861+0000 7f1109ffb700 15 client.43793.objecter _session_command_op_remove 7 1 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:02:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:02:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:02:02.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.366571+0000 osd.4 (osd.4) 392 : cluster 0 3.1a scrub starts 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.372760+0000 osd.4 (osd.4) 393 : cluster 0 3.1a scrub ok 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.696834+0000 osd.6 (osd.6) 337 : cluster 0 2.11 scrub starts 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.698794+0000 osd.6 (osd.6) 338 : cluster 0 2.11 scrub ok 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.799900+0000 osd.2 (osd.2) 351 : cluster 0 3.9 scrub starts 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:01.804342+0000 osd.2 (osd.2) 352 : cluster 0 3.9 scrub ok 2024-04-02T05:02:02.962 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: audit 2024-04-02T05:02:02.093009+0000 mon.a (mon.0) 3780 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:02.963 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: cluster 2024-04-02T05:02:02.103289+0000 mon.a (mon.0) 3781 : cluster 0 osdmap e810: 8 total, 8 up, 7 in 2024-04-02T05:02:02.963 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:02 smithi098 bash[23790]: audit 2024-04-02T05:02:02.128087+0000 mon.a (mon.0) 3782 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:02.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:03.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:03.109 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:03.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:03.345 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:03.371 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:03.474 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:03.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:03.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:03.632 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[28904]: cluster 2024-04-02T05:02:02.328746+0000 mgr.y (mgr.14538) 710 : cluster 0 pgmap v1228: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[28904]: cluster 2024-04-02T05:02:02.351951+0000 osd.4 (osd.4) 394 : cluster 0 3.3 scrub starts 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[28904]: cluster 2024-04-02T05:02:02.356776+0000 osd.4 (osd.4) 395 : cluster 0 3.3 scrub ok 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[28904]: audit 2024-04-02T05:02:03.100789+0000 mon.a (mon.0) 3783 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[28904]: cluster 2024-04-02T05:02:03.104896+0000 mon.a (mon.0) 3784 : cluster 0 osdmap e811: 8 total, 8 up, 7 in 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[28904]: audit 2024-04-02T05:02:03.106432+0000 mon.a (mon.0) 3785 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-136"}]: dispatch 2024-04-02T05:02:03.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[21528]: cluster 2024-04-02T05:02:02.328746+0000 mgr.y (mgr.14538) 710 : cluster 0 pgmap v1228: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:03.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[21528]: cluster 2024-04-02T05:02:02.351951+0000 osd.4 (osd.4) 394 : cluster 0 3.3 scrub starts 2024-04-02T05:02:03.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[21528]: cluster 2024-04-02T05:02:02.356776+0000 osd.4 (osd.4) 395 : cluster 0 3.3 scrub ok 2024-04-02T05:02:03.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[21528]: audit 2024-04-02T05:02:03.100789+0000 mon.a (mon.0) 3783 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:03.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[21528]: cluster 2024-04-02T05:02:03.104896+0000 mon.a (mon.0) 3784 : cluster 0 osdmap e811: 8 total, 8 up, 7 in 2024-04-02T05:02:03.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:03 smithi069 bash[21528]: audit 2024-04-02T05:02:03.106432+0000 mon.a (mon.0) 3785 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-136"}]: dispatch 2024-04-02T05:02:03.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:03.761 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:03.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:03.892 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:03.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:03.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:02:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[23790]: cluster 2024-04-02T05:02:02.328746+0000 mgr.y (mgr.14538) 710 : cluster 0 pgmap v1228: 224 pgs: 32 unknown, 192 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[23790]: cluster 2024-04-02T05:02:02.351951+0000 osd.4 (osd.4) 394 : cluster 0 3.3 scrub starts 2024-04-02T05:02:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[23790]: cluster 2024-04-02T05:02:02.356776+0000 osd.4 (osd.4) 395 : cluster 0 3.3 scrub ok 2024-04-02T05:02:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[23790]: audit 2024-04-02T05:02:03.100789+0000 mon.a (mon.0) 3783 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[23790]: cluster 2024-04-02T05:02:03.104896+0000 mon.a (mon.0) 3784 : cluster 0 osdmap e811: 8 total, 8 up, 7 in 2024-04-02T05:02:03.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:03 smithi098 bash[23790]: audit 2024-04-02T05:02:03.106432+0000 mon.a (mon.0) 3785 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-136"}]: dispatch 2024-04-02T05:02:03.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:04.021 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:04.124 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:04.152 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:04.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:04.280 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:04.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:04.410 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:04.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:04.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:04.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: cluster 2024-04-02T05:02:03.176256+0000 osd.0 (osd.0) 391 : cluster 0 2.13 scrub starts 2024-04-02T05:02:04.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: cluster 2024-04-02T05:02:03.179488+0000 osd.0 (osd.0) 392 : cluster 0 2.13 scrub ok 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: audit 2024-04-02T05:02:03.690884+0000 mgr.y (mgr.14538) 711 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: audit 2024-04-02T05:02:04.035471+0000 mon.a (mon.0) 3786 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: audit 2024-04-02T05:02:04.038338+0000 mon.b (mon.1) 311 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: audit 2024-04-02T05:02:04.103113+0000 mon.a (mon.0) 3787 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-136"}]': finished 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: cluster 2024-04-02T05:02:04.111453+0000 mon.a (mon.0) 3788 : cluster 0 osdmap e812: 8 total, 8 up, 7 in 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[21528]: audit 2024-04-02T05:02:04.112984+0000 mon.a (mon.0) 3789 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-136", "mode": "writeback"}]: dispatch 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: cluster 2024-04-02T05:02:03.176256+0000 osd.0 (osd.0) 391 : cluster 0 2.13 scrub starts 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: cluster 2024-04-02T05:02:03.179488+0000 osd.0 (osd.0) 392 : cluster 0 2.13 scrub ok 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: audit 2024-04-02T05:02:03.690884+0000 mgr.y (mgr.14538) 711 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: audit 2024-04-02T05:02:04.035471+0000 mon.a (mon.0) 3786 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:04.655 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: audit 2024-04-02T05:02:04.038338+0000 mon.b (mon.1) 311 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:04.656 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: audit 2024-04-02T05:02:04.103113+0000 mon.a (mon.0) 3787 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-136"}]': finished 2024-04-02T05:02:04.656 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: cluster 2024-04-02T05:02:04.111453+0000 mon.a (mon.0) 3788 : cluster 0 osdmap e812: 8 total, 8 up, 7 in 2024-04-02T05:02:04.656 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:04 smithi069 bash[28904]: audit 2024-04-02T05:02:04.112984+0000 mon.a (mon.0) 3789 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-136", "mode": "writeback"}]: dispatch 2024-04-02T05:02:04.669 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: cluster 2024-04-02T05:02:03.176256+0000 osd.0 (osd.0) 391 : cluster 0 2.13 scrub starts 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: cluster 2024-04-02T05:02:03.179488+0000 osd.0 (osd.0) 392 : cluster 0 2.13 scrub ok 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: audit 2024-04-02T05:02:03.690884+0000 mgr.y (mgr.14538) 711 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: audit 2024-04-02T05:02:04.035471+0000 mon.a (mon.0) 3786 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: audit 2024-04-02T05:02:04.038338+0000 mon.b (mon.1) 311 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: audit 2024-04-02T05:02:04.103113+0000 mon.a (mon.0) 3787 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-136"}]': finished 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: cluster 2024-04-02T05:02:04.111453+0000 mon.a (mon.0) 3788 : cluster 0 osdmap e812: 8 total, 8 up, 7 in 2024-04-02T05:02:04.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:04 smithi098 bash[23790]: audit 2024-04-02T05:02:04.112984+0000 mon.a (mon.0) 3789 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-136", "mode": "writeback"}]: dispatch 2024-04-02T05:02:04.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:04.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:04.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:05.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:05.053 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:05.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:02:05.204 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:02:05.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 'trim stale osdmaps' 2024-04-02T05:02:05.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:05.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:05.541 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:05.569 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:05.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:05.699 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:05.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:05.830 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:05.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:04.175740+0000 osd.0 (osd.0) 393 : cluster 0 10.1e scrub starts 2024-04-02T05:02:05.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:04.177787+0000 osd.0 (osd.0) 394 : cluster 0 10.1e scrub ok 2024-04-02T05:02:05.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:04.329510+0000 mgr.y (mgr.14538) 712 : cluster 0 pgmap v1231: 224 pgs: 224 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:05.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:05.086396+0000 mon.a (mon.0) 3790 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: audit 2024-04-02T05:02:05.088286+0000 mon.a (mon.0) 3791 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-136", "mode": "writeback"}]': finished 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:05.092068+0000 mon.a (mon.0) 3792 : cluster 0 osdmap e813: 8 total, 8 up, 7 in 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: audit 2024-04-02T05:02:05.199036+0000 mon.a (mon.0) 3793 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:05.313575+0000 osd.6 (osd.6) 339 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[28904]: cluster 2024-04-02T05:02:05.314161+0000 osd.6 (osd.6) 340 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:04.175740+0000 osd.0 (osd.0) 393 : cluster 0 10.1e scrub starts 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:04.177787+0000 osd.0 (osd.0) 394 : cluster 0 10.1e scrub ok 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:04.329510+0000 mgr.y (mgr.14538) 712 : cluster 0 pgmap v1231: 224 pgs: 224 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:05.086396+0000 mon.a (mon.0) 3790 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:05.885 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: audit 2024-04-02T05:02:05.088286+0000 mon.a (mon.0) 3791 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-136", "mode": "writeback"}]': finished 2024-04-02T05:02:05.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:05.092068+0000 mon.a (mon.0) 3792 : cluster 0 osdmap e813: 8 total, 8 up, 7 in 2024-04-02T05:02:05.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: audit 2024-04-02T05:02:05.199036+0000 mon.a (mon.0) 3793 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:05.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:05.313575+0000 osd.6 (osd.6) 339 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:05.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:05 smithi069 bash[21528]: cluster 2024-04-02T05:02:05.314161+0000 osd.6 (osd.6) 340 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:05.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:04.175740+0000 osd.0 (osd.0) 393 : cluster 0 10.1e scrub starts 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:04.177787+0000 osd.0 (osd.0) 394 : cluster 0 10.1e scrub ok 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:04.329510+0000 mgr.y (mgr.14538) 712 : cluster 0 pgmap v1231: 224 pgs: 224 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:05.086396+0000 mon.a (mon.0) 3790 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: audit 2024-04-02T05:02:05.088286+0000 mon.a (mon.0) 3791 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-136", "mode": "writeback"}]': finished 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:05.092068+0000 mon.a (mon.0) 3792 : cluster 0 osdmap e813: 8 total, 8 up, 7 in 2024-04-02T05:02:05.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: audit 2024-04-02T05:02:05.199036+0000 mon.a (mon.0) 3793 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:05.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:05.313575+0000 osd.6 (osd.6) 339 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:05.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:05 smithi098 bash[23790]: cluster 2024-04-02T05:02:05.314161+0000 osd.6 (osd.6) 340 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:05.960 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:05.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:06.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:06.090 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:06.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:06.220 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:06.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:06.345 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:06.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:06.468 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:06.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:06.599 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:06.665 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: cluster 2024-04-02T05:02:04.705089+0000 osd.3 (osd.3) 407 : cluster 0 10.1b scrub starts 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: cluster 2024-04-02T05:02:04.708315+0000 osd.3 (osd.3) 408 : cluster 0 10.1b scrub ok 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: cluster 2024-04-02T05:02:05.183217+0000 osd.1 (osd.1) 395 : cluster 0 6.1 scrub starts 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: cluster 2024-04-02T05:02:05.184315+0000 osd.1 (osd.1) 396 : cluster 0 6.1 scrub ok 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: audit 2024-04-02T05:02:06.109285+0000 mon.a (mon.0) 3794 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: cluster 2024-04-02T05:02:06.116707+0000 mon.a (mon.0) 3795 : cluster 0 osdmap e814: 8 total, 8 up, 7 in 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[21528]: audit 2024-04-02T05:02:06.118888+0000 mon.a (mon.0) 3796 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136"}]: dispatch 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: cluster 2024-04-02T05:02:04.705089+0000 osd.3 (osd.3) 407 : cluster 0 10.1b scrub starts 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: cluster 2024-04-02T05:02:04.708315+0000 osd.3 (osd.3) 408 : cluster 0 10.1b scrub ok 2024-04-02T05:02:06.666 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: cluster 2024-04-02T05:02:05.183217+0000 osd.1 (osd.1) 395 : cluster 0 6.1 scrub starts 2024-04-02T05:02:06.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: cluster 2024-04-02T05:02:05.184315+0000 osd.1 (osd.1) 396 : cluster 0 6.1 scrub ok 2024-04-02T05:02:06.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: audit 2024-04-02T05:02:06.109285+0000 mon.a (mon.0) 3794 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:02:06.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: cluster 2024-04-02T05:02:06.116707+0000 mon.a (mon.0) 3795 : cluster 0 osdmap e814: 8 total, 8 up, 7 in 2024-04-02T05:02:06.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:06 smithi069 bash[28904]: audit 2024-04-02T05:02:06.118888+0000 mon.a (mon.0) 3796 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136"}]: dispatch 2024-04-02T05:02:06.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:06.730 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:06.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: cluster 2024-04-02T05:02:04.705089+0000 osd.3 (osd.3) 407 : cluster 0 10.1b scrub starts 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: cluster 2024-04-02T05:02:04.708315+0000 osd.3 (osd.3) 408 : cluster 0 10.1b scrub ok 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: cluster 2024-04-02T05:02:05.183217+0000 osd.1 (osd.1) 395 : cluster 0 6.1 scrub starts 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: cluster 2024-04-02T05:02:05.184315+0000 osd.1 (osd.1) 396 : cluster 0 6.1 scrub ok 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: audit 2024-04-02T05:02:06.109285+0000 mon.a (mon.0) 3794 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: cluster 2024-04-02T05:02:06.116707+0000 mon.a (mon.0) 3795 : cluster 0 osdmap e814: 8 total, 8 up, 7 in 2024-04-02T05:02:06.845 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:06 smithi098 bash[23790]: audit 2024-04-02T05:02:06.118888+0000 mon.a (mon.0) 3796 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136"}]: dispatch 2024-04-02T05:02:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:06.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:06.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:07.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:07.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:07.255 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:07.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:07.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:07.515 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:07.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:07.643 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:06.330130+0000 mgr.y (mgr.14538) 713 : cluster 0 pgmap v1234: 224 pgs: 224 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:06.485446+0000 osd.7 (osd.7) 205 : cluster 0 4.8 scrub starts 2024-04-02T05:02:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:06.490902+0000 osd.7 (osd.7) 206 : cluster 0 4.8 scrub ok 2024-04-02T05:02:07.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:06.665758+0000 osd.6 (osd.6) 341 : cluster 0 5.2 scrub starts 2024-04-02T05:02:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:06.668594+0000 osd.6 (osd.6) 342 : cluster 0 5.2 scrub ok 2024-04-02T05:02:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:07.110683+0000 mon.a (mon.0) 3797 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:02:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: audit 2024-04-02T05:02:07.118234+0000 mon.a (mon.0) 3798 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136"}]': finished 2024-04-02T05:02:07.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:07 smithi098 bash[23790]: cluster 2024-04-02T05:02:07.126698+0000 mon.a (mon.0) 3799 : cluster 0 osdmap e815: 8 total, 8 up, 7 in 2024-04-02T05:02:07.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:07.774 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:07.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:07.905 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:07.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:08.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:08.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:06.330130+0000 mgr.y (mgr.14538) 713 : cluster 0 pgmap v1234: 224 pgs: 224 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:08.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:06.485446+0000 osd.7 (osd.7) 205 : cluster 0 4.8 scrub starts 2024-04-02T05:02:08.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:06.490902+0000 osd.7 (osd.7) 206 : cluster 0 4.8 scrub ok 2024-04-02T05:02:08.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:06.665758+0000 osd.6 (osd.6) 341 : cluster 0 5.2 scrub starts 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:06.668594+0000 osd.6 (osd.6) 342 : cluster 0 5.2 scrub ok 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:07.110683+0000 mon.a (mon.0) 3797 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: audit 2024-04-02T05:02:07.118234+0000 mon.a (mon.0) 3798 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136"}]': finished 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[21528]: cluster 2024-04-02T05:02:07.126698+0000 mon.a (mon.0) 3799 : cluster 0 osdmap e815: 8 total, 8 up, 7 in 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:06.330130+0000 mgr.y (mgr.14538) 713 : cluster 0 pgmap v1234: 224 pgs: 224 active+clean; 582 KiB data, 734 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:06.485446+0000 osd.7 (osd.7) 205 : cluster 0 4.8 scrub starts 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:06.490902+0000 osd.7 (osd.7) 206 : cluster 0 4.8 scrub ok 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:06.665758+0000 osd.6 (osd.6) 341 : cluster 0 5.2 scrub starts 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:06.668594+0000 osd.6 (osd.6) 342 : cluster 0 5.2 scrub ok 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:07.110683+0000 mon.a (mon.0) 3797 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: audit 2024-04-02T05:02:07.118234+0000 mon.a (mon.0) 3798 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-136"}]': finished 2024-04-02T05:02:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:07 smithi069 bash[28904]: cluster 2024-04-02T05:02:07.126698+0000 mon.a (mon.0) 3799 : cluster 0 osdmap e815: 8 total, 8 up, 7 in 2024-04-02T05:02:08.034 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:08.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:08.162 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:08.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:08.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:08.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:08.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:08.542 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:08.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[28904]: cluster 2024-04-02T05:02:07.340795+0000 osd.4 (osd.4) 396 : cluster 0 10.1d scrub starts 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[28904]: cluster 2024-04-02T05:02:07.344987+0000 osd.4 (osd.4) 397 : cluster 0 10.1d scrub ok 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[28904]: cluster 2024-04-02T05:02:07.481592+0000 osd.4 (osd.4) 398 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[28904]: cluster 2024-04-02T05:02:07.481905+0000 osd.4 (osd.4) 399 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.128568+0000 mon.a (mon.0) 3800 : cluster 0 osdmap e816: 8 total, 8 up, 7 in 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[21528]: cluster 2024-04-02T05:02:07.340795+0000 osd.4 (osd.4) 396 : cluster 0 10.1d scrub starts 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[21528]: cluster 2024-04-02T05:02:07.344987+0000 osd.4 (osd.4) 397 : cluster 0 10.1d scrub ok 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[21528]: cluster 2024-04-02T05:02:07.481592+0000 osd.4 (osd.4) 398 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[21528]: cluster 2024-04-02T05:02:07.481905+0000 osd.4 (osd.4) 399 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:08.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.128568+0000 mon.a (mon.0) 3800 : cluster 0 osdmap e816: 8 total, 8 up, 7 in 2024-04-02T05:02:08.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:08.800 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:08.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:08.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:08 smithi098 bash[23790]: cluster 2024-04-02T05:02:07.340795+0000 osd.4 (osd.4) 396 : cluster 0 10.1d scrub starts 2024-04-02T05:02:08.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:08 smithi098 bash[23790]: cluster 2024-04-02T05:02:07.344987+0000 osd.4 (osd.4) 397 : cluster 0 10.1d scrub ok 2024-04-02T05:02:08.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:08 smithi098 bash[23790]: cluster 2024-04-02T05:02:07.481592+0000 osd.4 (osd.4) 398 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:08.914 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:08 smithi098 bash[23790]: cluster 2024-04-02T05:02:07.481905+0000 osd.4 (osd.4) 399 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:08.915 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:08 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.128568+0000 mon.a (mon.0) 3800 : cluster 0 osdmap e816: 8 total, 8 up, 7 in 2024-04-02T05:02:08.930 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:08.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:09.032 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:02:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:02:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:02:09.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:09.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:09.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:09.186 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:09.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:09.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:09.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:09.575 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:09.678 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.196778+0000 osd.1 (osd.1) 397 : cluster 0 2.4 scrub starts 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.199532+0000 osd.1 (osd.1) 398 : cluster 0 2.4 scrub ok 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.303780+0000 osd.4 (osd.4) 400 : cluster 0 4.e scrub starts 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.306839+0000 osd.4 (osd.4) 401 : cluster 0 4.e scrub ok 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.331027+0000 mgr.y (mgr.14538) 714 : cluster 0 pgmap v1237: 192 pgs: 192 active+clean; 4.4 MiB data, 743 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.447986+0000 osd.7 (osd.7) 207 : cluster 0 3.14 scrub starts 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.452207+0000 osd.7 (osd.7) 208 : cluster 0 3.14 scrub ok 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.659355+0000 osd.6 (osd.6) 343 : cluster 0 3.a scrub starts 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:08.663187+0000 osd.6 (osd.6) 344 : cluster 0 3.a scrub ok 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: cluster 2024-04-02T05:02:09.139177+0000 mon.a (mon.0) 3801 : cluster 0 osdmap e817: 8 total, 8 up, 7 in 2024-04-02T05:02:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:09 smithi098 bash[23790]: audit 2024-04-02T05:02:09.142702+0000 mon.a (mon.0) 3802 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:09.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:09.829 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:09.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.196778+0000 osd.1 (osd.1) 397 : cluster 0 2.4 scrub starts 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.199532+0000 osd.1 (osd.1) 398 : cluster 0 2.4 scrub ok 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.303780+0000 osd.4 (osd.4) 400 : cluster 0 4.e scrub starts 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.306839+0000 osd.4 (osd.4) 401 : cluster 0 4.e scrub ok 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.331027+0000 mgr.y (mgr.14538) 714 : cluster 0 pgmap v1237: 192 pgs: 192 active+clean; 4.4 MiB data, 743 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.447986+0000 osd.7 (osd.7) 207 : cluster 0 3.14 scrub starts 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.452207+0000 osd.7 (osd.7) 208 : cluster 0 3.14 scrub ok 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.659355+0000 osd.6 (osd.6) 343 : cluster 0 3.a scrub starts 2024-04-02T05:02:09.945 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:08.663187+0000 osd.6 (osd.6) 344 : cluster 0 3.a scrub ok 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: cluster 2024-04-02T05:02:09.139177+0000 mon.a (mon.0) 3801 : cluster 0 osdmap e817: 8 total, 8 up, 7 in 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[28904]: audit 2024-04-02T05:02:09.142702+0000 mon.a (mon.0) 3802 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.196778+0000 osd.1 (osd.1) 397 : cluster 0 2.4 scrub starts 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.199532+0000 osd.1 (osd.1) 398 : cluster 0 2.4 scrub ok 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.303780+0000 osd.4 (osd.4) 400 : cluster 0 4.e scrub starts 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.306839+0000 osd.4 (osd.4) 401 : cluster 0 4.e scrub ok 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.331027+0000 mgr.y (mgr.14538) 714 : cluster 0 pgmap v1237: 192 pgs: 192 active+clean; 4.4 MiB data, 743 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.447986+0000 osd.7 (osd.7) 207 : cluster 0 3.14 scrub starts 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.452207+0000 osd.7 (osd.7) 208 : cluster 0 3.14 scrub ok 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.659355+0000 osd.6 (osd.6) 343 : cluster 0 3.a scrub starts 2024-04-02T05:02:09.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:08.663187+0000 osd.6 (osd.6) 344 : cluster 0 3.a scrub ok 2024-04-02T05:02:09.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: cluster 2024-04-02T05:02:09.139177+0000 mon.a (mon.0) 3801 : cluster 0 osdmap e817: 8 total, 8 up, 7 in 2024-04-02T05:02:09.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:09 smithi069 bash[21528]: audit 2024-04-02T05:02:09.142702+0000 mon.a (mon.0) 3802 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:09.960 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:09.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:10.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:10.088 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:10.191 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:10.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:10.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:10.338 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:10.440 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:10.467 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:10.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:10.598 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:10.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:10.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[21528]: cluster 2024-04-02T05:02:09.253834+0000 osd.0 (osd.0) 395 : cluster 0 3.1b scrub starts 2024-04-02T05:02:10.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[21528]: cluster 2024-04-02T05:02:09.256574+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub ok 2024-04-02T05:02:10.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[21528]: audit 2024-04-02T05:02:10.133137+0000 mon.a (mon.0) 3803 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:10.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[21528]: cluster 2024-04-02T05:02:10.137185+0000 mon.a (mon.0) 3804 : cluster 0 osdmap e818: 8 total, 8 up, 7 in 2024-04-02T05:02:10.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[21528]: audit 2024-04-02T05:02:10.143897+0000 mon.a (mon.0) 3805 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[28904]: cluster 2024-04-02T05:02:09.253834+0000 osd.0 (osd.0) 395 : cluster 0 3.1b scrub starts 2024-04-02T05:02:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[28904]: cluster 2024-04-02T05:02:09.256574+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub ok 2024-04-02T05:02:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[28904]: audit 2024-04-02T05:02:10.133137+0000 mon.a (mon.0) 3803 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[28904]: cluster 2024-04-02T05:02:10.137185+0000 mon.a (mon.0) 3804 : cluster 0 osdmap e818: 8 total, 8 up, 7 in 2024-04-02T05:02:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:10 smithi069 bash[28904]: audit 2024-04-02T05:02:10.143897+0000 mon.a (mon.0) 3805 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:10.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:10.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:10 smithi098 bash[23790]: cluster 2024-04-02T05:02:09.253834+0000 osd.0 (osd.0) 395 : cluster 0 3.1b scrub starts 2024-04-02T05:02:10.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:10 smithi098 bash[23790]: cluster 2024-04-02T05:02:09.256574+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub ok 2024-04-02T05:02:10.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:10 smithi098 bash[23790]: audit 2024-04-02T05:02:10.133137+0000 mon.a (mon.0) 3803 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:10.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:10 smithi098 bash[23790]: cluster 2024-04-02T05:02:10.137185+0000 mon.a (mon.0) 3804 : cluster 0 osdmap e818: 8 total, 8 up, 7 in 2024-04-02T05:02:10.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:10 smithi098 bash[23790]: audit 2024-04-02T05:02:10.143897+0000 mon.a (mon.0) 3805 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:10.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:10.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:10.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:11.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:11.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:11.248 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:11.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:11.373 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:11.428 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:02:11.428 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T05:02:11.428 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2024-04-02T05:02:11.428 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:02:11.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:11.497 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:11.600 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:11.627 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:11.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:10.249060+0000 osd.0 (osd.0) 397 : cluster 0 10.11 scrub starts 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:10.250633+0000 osd.0 (osd.0) 398 : cluster 0 10.11 scrub ok 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:10.314882+0000 osd.4 (osd.4) 402 : cluster 0 4.c scrub starts 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:10.317019+0000 osd.4 (osd.4) 403 : cluster 0 4.c scrub ok 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:10.331546+0000 mgr.y (mgr.14538) 715 : cluster 0 pgmap v1240: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:11.131195+0000 osd.1 (osd.1) 399 : cluster 0 10.13 scrub starts 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:11.136850+0000 osd.1 (osd.1) 400 : cluster 0 10.13 scrub ok 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: audit 2024-04-02T05:02:11.145609+0000 mon.a (mon.0) 3806 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: cluster 2024-04-02T05:02:11.154364+0000 mon.a (mon.0) 3807 : cluster 0 osdmap e819: 8 total, 8 up, 7 in 2024-04-02T05:02:11.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[21528]: audit 2024-04-02T05:02:11.156065+0000 mon.a (mon.0) 3808 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:10.249060+0000 osd.0 (osd.0) 397 : cluster 0 10.11 scrub starts 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:10.250633+0000 osd.0 (osd.0) 398 : cluster 0 10.11 scrub ok 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:10.314882+0000 osd.4 (osd.4) 402 : cluster 0 4.c scrub starts 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:10.317019+0000 osd.4 (osd.4) 403 : cluster 0 4.c scrub ok 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:10.331546+0000 mgr.y (mgr.14538) 715 : cluster 0 pgmap v1240: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:11.131195+0000 osd.1 (osd.1) 399 : cluster 0 10.13 scrub starts 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:11.136850+0000 osd.1 (osd.1) 400 : cluster 0 10.13 scrub ok 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: audit 2024-04-02T05:02:11.145609+0000 mon.a (mon.0) 3806 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: cluster 2024-04-02T05:02:11.154364+0000 mon.a (mon.0) 3807 : cluster 0 osdmap e819: 8 total, 8 up, 7 in 2024-04-02T05:02:11.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:11 smithi069 bash[28904]: audit 2024-04-02T05:02:11.156065+0000 mon.a (mon.0) 3808 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:02:11.756 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:11.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:11.885 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:11.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:10.249060+0000 osd.0 (osd.0) 397 : cluster 0 10.11 scrub starts 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:10.250633+0000 osd.0 (osd.0) 398 : cluster 0 10.11 scrub ok 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:10.314882+0000 osd.4 (osd.4) 402 : cluster 0 4.c scrub starts 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:10.317019+0000 osd.4 (osd.4) 403 : cluster 0 4.c scrub ok 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:10.331546+0000 mgr.y (mgr.14538) 715 : cluster 0 pgmap v1240: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:11.131195+0000 osd.1 (osd.1) 399 : cluster 0 10.13 scrub starts 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:11.136850+0000 osd.1 (osd.1) 400 : cluster 0 10.13 scrub ok 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: audit 2024-04-02T05:02:11.145609+0000 mon.a (mon.0) 3806 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: cluster 2024-04-02T05:02:11.154364+0000 mon.a (mon.0) 3807 : cluster 0 osdmap e819: 8 total, 8 up, 7 in 2024-04-02T05:02:11.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:11 smithi098 bash[23790]: audit 2024-04-02T05:02:11.156065+0000 mon.a (mon.0) 3808 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:02:11.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:12.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:12.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- 172.21.15.69:0/3373082276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102520 msgr2=0x7f7468102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- 172.21.15.69:0/3373082276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102520 0x7f7468102900 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f7458009990 tx=0x7f7458009ca0 comp rx=0 tx=0).stop 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- 172.21.15.69:0/3373082276 shutdown_connections 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- 172.21.15.69:0/3373082276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74681075e0 0x7f746810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- 172.21.15.69:0/3373082276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102520 0x7f7468102900 unknown :-1 s=CLOSED pgs=3286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- 172.21.15.69:0/3373082276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7468102ed0 0x7f7468106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- 172.21.15.69:0/3373082276 >> 172.21.15.69:0/3373082276 conn(0x7f74680793f0 msgr2=0x7f7468079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- 172.21.15.69:0/3373082276 shutdown_connections 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- 172.21.15.69:0/3373082276 wait complete. 2024-04-02T05:02:12.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 librados: starting msgr at 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 librados: starting objecter 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 Processor -- start 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- start start 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 librados: setting wanted keys 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 librados: calling monclient init 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7468102520 0x7f74681980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102ed0 0x7f7468198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74681075e0 0x7f746819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7468111af0 con 0x7f7468102520 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7468111970 con 0x7f74681075e0 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.313+0000 7f746e109700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7468111c70 con 0x7f7468102ed0 2024-04-02T05:02:12.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f74677fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7468102520 0x7f74681980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102ed0 0x7f7468198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f74677fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7468102520 0x7f74681980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49194/0 (socket says 172.21.15.69:49194) 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f74677fe700 1 -- 172.21.15.69:0/3067972721 learned_addr learned my addr 172.21.15.69:0/3067972721 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7467fff700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74681075e0 0x7f746819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 -- 172.21.15.69:0/3067972721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74681075e0 msgr2=0x7f746819c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74681075e0 0x7f746819c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 -- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7468102520 msgr2=0x7f74681980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7468102520 0x7f74681980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 -- 172.21.15.69:0/3067972721 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746819d0e0 con 0x7f7468102ed0 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7466ffd700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102ed0 0x7f7468198610 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f746806b9f0 tx=0x7f745c00b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 10 client.?.objecter ms_handle_connect 0x7f7468102ed0 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:12.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c015020 con 0x7f7468102ed0 2024-04-02T05:02:12.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746819d3d0 con 0x7f7468102ed0 2024-04-02T05:02:12.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f745c011cb0 con 0x7f7468102ed0 2024-04-02T05:02:12.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c010bb0 con 0x7f7468102ed0 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f746e109700 10 client.43799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74681a4c50 con 0x7f7468102ed0 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f746e109700 1 librados: init done 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746804ff00 con 0x7f7468102ed0 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f745c018020 con 0x7f7468102ed0 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f745006f5e0 0x7f7450071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(820..820 src has 74..820) ==== 10107+0+0 (secure 0 0 0) 0x7f745c08e6f0 con 0x7f7468102ed0 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 10 client.43799.objecter ms_dispatch 0x7f7468001550 osd_map(820..820 src has 74..820) 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 3 client.43799.objecter handle_osd_map got epochs [820,820] > 0 2024-04-02T05:02:12.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 3 client.43799.objecter handle_osd_map decoding full epoch 820 2024-04-02T05:02:12.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f74677fe700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f745006f5e0 0x7f7450071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:12.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 20 client.43799.objecter dump_active .. 0 homeless 2024-04-02T05:02:12.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f74677fe700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f745006f5e0 0x7f7450071aa0 secure :-1 s=READY pgs=4013 cs=0 l=1 rev1=1 crypto rx=0x7f7458006010 tx=0x7f7458009590 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:12.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.317+0000 7f7464ff9700 10 client.43799.objecter ms_handle_connect 0x7f745006f5e0 2024-04-02T05:02:12.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.321+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f745c01f020 con 0x7f7468102ed0 2024-04-02T05:02:12.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:12.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:12.522 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:12.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:12.541+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f74681a4e80 con 0x7f7468102ed0 2024-04-02T05:02:12.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:12.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:12.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:12.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:12.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:13.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:13.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:13.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:13.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.173+0000 7f7464ff9700 1 -- 172.21.15.69:0/3067972721 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v821) ==== 101+0+0 (secure 0 0 0) 0x7f745c05b300 con 0x7f7468102ed0 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 10 librados: watch_flush enter 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 10 librados: watch_flush exit 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 20 client.43799.objecter shutdown clearing up homeless session... 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 10 client.43799.objecter successfully canceled tick 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f745006f5e0 msgr2=0x7f7450071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f745006f5e0 0x7f7450071aa0 secure :-1 s=READY pgs=4013 cs=0 l=1 rev1=1 crypto rx=0x7f7458006010 tx=0x7f7458009590 comp rx=0 tx=0).stop 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102ed0 msgr2=0x7f7468198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102ed0 0x7f7468198610 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f746806b9f0 tx=0x7f745c00b730 comp rx=0 tx=0).stop 2024-04-02T05:02:13.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 shutdown_connections 2024-04-02T05:02:13.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f745006f5e0 0x7f7450071aa0 unknown :-1 s=CLOSED pgs=4013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74681075e0 0x7f746819c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7468102520 0x7f74681980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 --2- 172.21.15.69:0/3067972721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7468102ed0 0x7f7468198610 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 >> 172.21.15.69:0/3067972721 conn(0x7f74680793f0 msgr2=0x7f7468100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:13.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 shutdown_connections 2024-04-02T05:02:13.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 -- 172.21.15.69:0/3067972721 wait complete. 2024-04-02T05:02:13.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.177+0000 7f746e109700 1 librados: shutdown 2024-04-02T05:02:13.184 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T05:02:13.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[28904]: audit 2024-04-02T05:02:12.148228+0000 mon.a (mon.0) 3809 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:02:13.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[28904]: cluster 2024-04-02T05:02:12.152545+0000 mon.a (mon.0) 3810 : cluster 0 osdmap e820: 8 total, 8 up, 7 in 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[28904]: audit 2024-04-02T05:02:12.153269+0000 mon.a (mon.0) 3811 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[28904]: cluster 2024-04-02T05:02:12.332062+0000 mgr.y (mgr.14538) 716 : cluster 0 pgmap v1243: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[28904]: audit 2024-04-02T05:02:12.546119+0000 mon.c (mon.2) 1160 : audit 1 from='client.? 172.21.15.69:0/3067972721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[28904]: audit 2024-04-02T05:02:12.546452+0000 mon.a (mon.0) 3812 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[21528]: audit 2024-04-02T05:02:12.148228+0000 mon.a (mon.0) 3809 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[21528]: cluster 2024-04-02T05:02:12.152545+0000 mon.a (mon.0) 3810 : cluster 0 osdmap e820: 8 total, 8 up, 7 in 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[21528]: audit 2024-04-02T05:02:12.153269+0000 mon.a (mon.0) 3811 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[21528]: cluster 2024-04-02T05:02:12.332062+0000 mgr.y (mgr.14538) 716 : cluster 0 pgmap v1243: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[21528]: audit 2024-04-02T05:02:12.546119+0000 mon.c (mon.2) 1160 : audit 1 from='client.? 172.21.15.69:0/3067972721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:13.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:13 smithi069 bash[21528]: audit 2024-04-02T05:02:12.546452+0000 mon.a (mon.0) 3812 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:13.294 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:13.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.424 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:13.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[23790]: audit 2024-04-02T05:02:12.148228+0000 mon.a (mon.0) 3809 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:02:13.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[23790]: cluster 2024-04-02T05:02:12.152545+0000 mon.a (mon.0) 3810 : cluster 0 osdmap e820: 8 total, 8 up, 7 in 2024-04-02T05:02:13.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[23790]: audit 2024-04-02T05:02:12.153269+0000 mon.a (mon.0) 3811 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:02:13.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[23790]: cluster 2024-04-02T05:02:12.332062+0000 mgr.y (mgr.14538) 716 : cluster 0 pgmap v1243: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:13.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[23790]: audit 2024-04-02T05:02:12.546119+0000 mon.c (mon.2) 1160 : audit 1 from='client.? 172.21.15.69:0/3067972721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:13.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[23790]: audit 2024-04-02T05:02:12.546452+0000 mon.a (mon.0) 3812 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:13.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.552 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:13.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.680 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:13.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-02T05:02:13.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- 172.21.15.69:0/1779439707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 msgr2=0x7f722010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- 172.21.15.69:0/1779439707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f722010dfd0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f7210009f90 tx=0x7f721000b330 comp rx=0 tx=0).stop 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- 172.21.15.69:0/1779439707 shutdown_connections 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- 172.21.15.69:0/1779439707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f722010dfd0 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- 172.21.15.69:0/1779439707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7220106150 0x7f7220106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- 172.21.15.69:0/1779439707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7220101900 0x7f7220101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- 172.21.15.69:0/1779439707 >> 172.21.15.69:0/1779439707 conn(0x7f7220077740 msgr2=0x7f72200feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- 172.21.15.69:0/1779439707 shutdown_connections 2024-04-02T05:02:13.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- 172.21.15.69:0/1779439707 wait complete. 2024-04-02T05:02:13.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 librados: starting msgr at 2024-04-02T05:02:13.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 librados: starting objecter 2024-04-02T05:02:13.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 Processor -- start 2024-04-02T05:02:13.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- start start 2024-04-02T05:02:13.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 librados: setting wanted keys 2024-04-02T05:02:13.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 librados: calling monclient init 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7220101900 0x7f7220198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f72201985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7220106150 0x7f722019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7220110a80 con 0x7f7220106150 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7220110900 con 0x7f7220101900 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7228093700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7220110c00 con 0x7f72201022c0 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f722562e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f72201985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f722562e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f72201985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:53860/0 (socket says 172.21.15.69:53860) 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f722562e700 1 -- 172.21.15.69:0/1290076307 learned_addr learned my addr 172.21.15.69:0/1290076307 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7220106150 0x7f722019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.785+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7220101900 0x7f7220198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f722562e700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7220101900 msgr2=0x7f7220198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7220101900 0x7f7220198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f722562e700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7220106150 msgr2=0x7f722019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7220106150 0x7f722019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f722562e700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722019d090 con 0x7f72201022c0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7220101900 0x7f7220198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7220106150 0x7f722019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f72201985c0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f721400ec10 tx=0x7f721400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 10 client.?.objecter ms_handle_connect 0x7f72201022c0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721400cde0 con 0x7f72201022c0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722019d380 con 0x7f72201022c0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f72140044a0 con 0x7f72201022c0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 10 client.43805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214010430 con 0x7f72201022c0 2024-04-02T05:02:13.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72201a4c00 con 0x7f72201022c0 2024-04-02T05:02:13.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 1 librados: init done 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f722004ff00 con 0x7f72201022c0 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7214003fe0 con 0x7f72201022c0 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f720806f5e0 0x7f7208071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(821..821 src has 74..821) ==== 10107+0+0 (secure 0 0 0) 0x7f721408ef30 con 0x7f72201022c0 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 osd_map(821..821 src has 74..821) 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 3 client.43805.objecter handle_osd_map got epochs [821,821] > 0 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 3 client.43805.objecter handle_osd_map decoding full epoch 821 2024-04-02T05:02:13.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f720806f5e0 0x7f7208071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 20 client.43805.objecter dump_active .. 0 homeless 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v821) ==== 54+0+16 (secure 0 0 0) 0x7f7214091550 con 0x7f72201022c0 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 10 client.43805.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 1 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7220101300 0x7f722002c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 1 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 0 1 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 4 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 10 client.43805.objecter _send_command 1 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f720806f5e0 0x7f7208071aa0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 crypto rx=0x7f7218005fd0 tx=0x7f7218005dc0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f720806f5e0 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7220063360 con 0x7f7220101300 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7220101300 0x7f722002c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f7220101300 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.789+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7220101300 0x7f722002c9b0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.793+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7220063360 con 0x7f7220101300 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.793+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 1: 0 ) 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.793+0000 7f721effd700 10 client.43805.objecter _finish_command 1 = system:0 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.793+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 4 2024-04-02T05:02:13.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.793+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 0 1 2024-04-02T05:02:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:13.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 10 client.43805.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 2 2024-04-02T05:02:13.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 2 2024-04-02T05:02:13.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f7220101190 osd=0 3 2024-04-02T05:02:13.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 0 2 2024-04-02T05:02:13.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 4 2024-04-02T05:02:13.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 10 client.43805.objecter _send_command 2 2024-04-02T05:02:13.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f722002cef0 con 0x7f7220101300 2024-04-02T05:02:13.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f722002cef0 con 0x7f7220101300 2024-04-02T05:02:13.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f721effd700 10 client.43805.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 4 2024-04-02T05:02:13.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.809+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 0 2 2024-04-02T05:02:13.816 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:02:13.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 10 client.43805.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7220062a40 0x7f72201a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 1 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 4 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 10 client.43805.objecter _send_command 3 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f722002dde0 con 0x7f7220062a40 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7220062a40 0x7f72201a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f7220062a40 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7220062a40 0x7f72201a8eb0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f722002dde0 con 0x7f7220062a40 2024-04-02T05:02:13.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 3: 0 ) 2024-04-02T05:02:13.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f721effd700 10 client.43805.objecter _finish_command 3 = system:0 2024-04-02T05:02:13.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 4 2024-04-02T05:02:13.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.813+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 1 3 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 10 client.43805.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 4 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 4 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72200628d0 osd=1 3 2024-04-02T05:02:13.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 1 4 2024-04-02T05:02:13.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 4 2024-04-02T05:02:13.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 10 client.43805.objecter _send_command 4 2024-04-02T05:02:13.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.829+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f722002df90 con 0x7f7220062a40 2024-04-02T05:02:13.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f722002df90 con 0x7f7220062a40 2024-04-02T05:02:13.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 10 client.43805.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 4 2024-04-02T05:02:13.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 1 4 2024-04-02T05:02:13.837 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:02:13.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 10 client.43805.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 5 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7220109bb0 0x7f72201ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 5 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 2 5 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 4 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 10 client.43805.objecter _send_command 5 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f722002d710 con 0x7f7220109bb0 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7220109bb0 0x7f72201ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f7220109bb0 2024-04-02T05:02:13.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7220109bb0 0x7f72201ac580 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7214003810 con 0x7f7220109bb0 2024-04-02T05:02:13.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 5: 0 ) 2024-04-02T05:02:13.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 10 client.43805.objecter _finish_command 5 = system:0 2024-04-02T05:02:13.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 4 2024-04-02T05:02:13.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.833+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 2 5 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 10 client.43805.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 6 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 6 2024-04-02T05:02:13.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f722002d5a0 osd=2 3 2024-04-02T05:02:13.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 2 6 2024-04-02T05:02:13.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 4 2024-04-02T05:02:13.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 10 client.43805.objecter _send_command 6 2024-04-02T05:02:13.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.849+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72201091e0 con 0x7f7220109bb0 2024-04-02T05:02:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f72201091e0 con 0x7f7220109bb0 2024-04-02T05:02:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 10 client.43805.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 4 2024-04-02T05:02:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 2 6 2024-04-02T05:02:13.856 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 10 client.43805.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 7 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f72200fdc30 0x7f72201aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 7 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 3 7 2024-04-02T05:02:13.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 4 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 10 client.43805.objecter _send_command 7 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f72201af010 con 0x7f72200fdc30 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f72200fdc30 0x7f72201aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f72200fdc30 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f72200fdc30 0x7f72201aead0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72201af010 con 0x7f72200fdc30 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 7: 0 ) 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 10 client.43805.objecter _finish_command 7 = system:0 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 4 2024-04-02T05:02:13.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.853+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 3 7 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 10 client.43805.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 8 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 8 2024-04-02T05:02:13.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201a9bc0 osd=3 3 2024-04-02T05:02:13.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 3 8 2024-04-02T05:02:13.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 4 2024-04-02T05:02:13.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 10 client.43805.objecter _send_command 8 2024-04-02T05:02:13.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.869+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72200fe840 con 0x7f72200fdc30 2024-04-02T05:02:13.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f72200fe840 con 0x7f72200fdc30 2024-04-02T05:02:13.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 10 client.43805.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 4 2024-04-02T05:02:13.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 3 8 2024-04-02T05:02:13.876 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 10 client.43805.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 9 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f72200fe840 0x7f72201b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 9 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 4 9 2024-04-02T05:02:13.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 4 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 10 client.43805.objecter _send_command 9 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f72201b8660 con 0x7f72200fe840 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f72200fe840 0x7f72201b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f72200fe840 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f72200fe840 0x7f72201b5800 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72201b8660 con 0x7f72200fe840 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 9: 0 ) 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 10 client.43805.objecter _finish_command 9 = system:0 2024-04-02T05:02:13.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 4 2024-04-02T05:02:13.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.873+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 4 9 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 10 client.43805.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 10 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 10 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201aa3f0 osd=4 3 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 4 10 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 4 2024-04-02T05:02:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 10 client.43805.objecter _send_command 10 2024-04-02T05:02:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.889+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72201a9450 con 0x7f72200fe840 2024-04-02T05:02:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f72201a9450 con 0x7f72200fe840 2024-04-02T05:02:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f721effd700 10 client.43805.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 4 2024-04-02T05:02:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 4 10 2024-04-02T05:02:13.897 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:02:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 10 client.43805.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 11 2024-04-02T05:02:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f72200fd280 0x7f72200fd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 11 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 5 11 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 4 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 10 client.43805.objecter _send_command 11 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f72201bba00 con 0x7f72200fd280 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f72200fd280 0x7f72200fd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f72200fd280 2024-04-02T05:02:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.893+0000 7f722562e700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f72200fd280 0x7f72200fd6e0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.897+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72201bba00 con 0x7f72200fd280 2024-04-02T05:02:13.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.897+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 11: 0 ) 2024-04-02T05:02:13.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.897+0000 7f721effd700 10 client.43805.objecter _finish_command 11 = system:0 2024-04-02T05:02:13.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.897+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 4 2024-04-02T05:02:13.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.897+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 5 11 2024-04-02T05:02:13.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 10 client.43805.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 12 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 12 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201b8690 osd=5 3 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 5 12 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 4 2024-04-02T05:02:13.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 10 client.43805.objecter _send_command 12 2024-04-02T05:02:13.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.917+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72201bba00 con 0x7f72200fd280 2024-04-02T05:02:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f72201bba00 con 0x7f72200fd280 2024-04-02T05:02:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f721effd700 10 client.43805.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 4 2024-04-02T05:02:13.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 5 12 2024-04-02T05:02:13.926 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:02:13.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 10 client.43805.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 13 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f72201b6250 0x7f72201b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 13 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 6 13 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 4 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 10 client.43805.objecter _send_command 13 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f72201bee20 con 0x7f72201b6250 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f72201b6250 0x7f72201b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f72201b6250 2024-04-02T05:02:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.921+0000 7f7226630700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f72201b6250 0x7f72201b6690 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.925+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72201bee20 con 0x7f72201b6250 2024-04-02T05:02:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.925+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 13: 0 ) 2024-04-02T05:02:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.925+0000 7f721effd700 10 client.43805.objecter _finish_command 13 = system:0 2024-04-02T05:02:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.925+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 4 2024-04-02T05:02:13.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.925+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 6 13 2024-04-02T05:02:13.931 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:13.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 10 client.43805.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 14 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 14 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201bba00 osd=6 3 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 6 14 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 4 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 10 client.43805.objecter _send_command 14 2024-04-02T05:02:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.941+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72201b6ce0 con 0x7f72201b6250 2024-04-02T05:02:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f72201b6ce0 con 0x7f72201b6250 2024-04-02T05:02:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f721effd700 10 client.43805.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 4 2024-04-02T05:02:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 6 14 2024-04-02T05:02:13.951 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:02:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 10 client.43805.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 15 2024-04-02T05:02:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f72201b71f0 0x7f72201b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 15 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 7 15 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 4 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 10 client.43805.objecter _send_command 15 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f72201b7b70 con 0x7f72201b71f0 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.945+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f72201b71f0 0x7f72201b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f721effd700 10 client.43805.objecter ms_handle_connect 0x7f72201b71f0 2024-04-02T05:02:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f7225e2f700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f72201b71f0 0x7f72201b7630 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72201b7b70 con 0x7f72201b71f0 2024-04-02T05:02:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 15: 0 ) 2024-04-02T05:02:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f721effd700 10 client.43805.objecter _finish_command 15 = system:0 2024-04-02T05:02:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 4 2024-04-02T05:02:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.949+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 7 15 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 10 client.43805.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign -1 16 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 20 client.43805.objecter _get_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 15 client.43805.objecter _session_command_op_remove -1 16 2024-04-02T05:02:13.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 20 client.43805.objecter get_session s=0x7f72201b8060 osd=7 3 2024-04-02T05:02:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 15 client.43805.objecter _session_command_op_assign 7 16 2024-04-02T05:02:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 4 2024-04-02T05:02:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 10 client.43805.objecter _send_command 16 2024-04-02T05:02:13.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.965+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72201b7b70 con 0x7f72201b71f0 2024-04-02T05:02:13.974 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f721effd700 1 -- 172.21.15.69:0/1290076307 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f72201b7b70 con 0x7f72201b71f0 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f721effd700 10 client.43805.objecter ms_dispatch 0x7f7220001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f721effd700 10 client.43805.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f721effd700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 4 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f721effd700 15 client.43805.objecter _session_command_op_remove 7 16 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 librados: watch_flush enter 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 librados: watch_flush exit 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.0 2024-04-02T05:02:13.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7220101300 msgr2=0x7f722002c9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f7220101300 0x7f722002c9b0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f7220101190 osd=0 1 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.1 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7220062a40 msgr2=0x7f72201a8eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f7220062a40 0x7f72201a8eb0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72200628d0 osd=1 1 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.2 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7220109bb0 msgr2=0x7f72201ac580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f7220109bb0 0x7f72201ac580 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f722002d5a0 osd=2 1 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.3 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f72200fdc30 msgr2=0x7f72201aead0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7226630700 1 -- 172.21.15.69:0/1290076307 reap_dead start 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f72200fdc30 0x7f72201aead0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201a9bc0 osd=3 1 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.4 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f72200fe840 msgr2=0x7f72201b5800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f72200fe840 0x7f72201b5800 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201aa3f0 osd=4 1 2024-04-02T05:02:13.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.5 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f72200fd280 msgr2=0x7f72200fd6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f72200fd280 0x7f72200fd6e0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8690 osd=5 1 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.6 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f72201b6250 msgr2=0x7f72201b6690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f72201b6250 0x7f72201b6690 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201bba00 osd=6 1 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter close_session for osd.7 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f72201b71f0 msgr2=0x7f72201b7630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f72201b71f0 0x7f72201b7630 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter put_session s=0x7f72201b8060 osd=7 1 2024-04-02T05:02:13.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 20 client.43805.objecter shutdown clearing up homeless session... 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 10 client.43805.objecter successfully canceled tick 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7226630700 1 -- 172.21.15.69:0/1290076307 reap_dead start 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f720806f5e0 msgr2=0x7f7208071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f720806f5e0 0x7f7208071aa0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 crypto rx=0x7f7218005fd0 tx=0x7f7218005dc0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 msgr2=0x7f72201985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f72201985c0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f721400ec10 tx=0x7f721400c2d0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 shutdown_connections 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72201022c0 0x7f72201985c0 unknown :-1 s=CLOSED pgs=3289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 --2- 172.21.15.69:0/1290076307 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f720806f5e0 0x7f7208071aa0 unknown :-1 s=CLOSED pgs=4014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 >> 172.21.15.69:0/1290076307 conn(0x7f7220077740 msgr2=0x7f7220100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 shutdown_connections 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 -- 172.21.15.69:0/1290076307 wait complete. 2024-04-02T05:02:13.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:13.969+0000 7f7228093700 1 librados: shutdown 2024-04-02T05:02:13.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:02:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:02:13.980 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:02:13.980 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:02:13.980 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:02:13.980 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:02:13.980 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:02:13.980 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:02:13.981 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:02:13.981 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:02:14.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:14.053 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:14.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:12.363744+0000 osd.7 (osd.7) 209 : cluster 0 4.15 scrub starts 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:12.368346+0000 osd.7 (osd.7) 210 : cluster 0 4.15 scrub ok 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:12.685797+0000 osd.6 (osd.6) 345 : cluster 0 4.12 scrub starts 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:12.687844+0000 osd.6 (osd.6) 346 : cluster 0 4.12 scrub ok 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:13.149611+0000 mon.a (mon.0) 3813 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: audit 2024-04-02T05:02:13.159751+0000 mon.a (mon.0) 3814 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: audit 2024-04-02T05:02:13.159982+0000 mon.a (mon.0) 3815 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:13.174645+0000 mon.a (mon.0) 3816 : cluster 0 osdmap e821: 8 total, 8 up, 7 in 2024-04-02T05:02:14.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: audit 2024-04-02T05:02:13.176617+0000 mon.a (mon.0) 3817 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:13.200588+0000 osd.0 (osd.0) 399 : cluster 0 10.4 scrub starts 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: cluster 2024-04-02T05:02:13.203337+0000 osd.0 (osd.0) 400 : cluster 0 10.4 scrub ok 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: audit 2024-04-02T05:02:13.702034+0000 mgr.y (mgr.14538) 717 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[21528]: audit 2024-04-02T05:02:13.794192+0000 mon.c (mon.2) 1161 : audit 0 from='client.? 172.21.15.69:0/1290076307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:12.363744+0000 osd.7 (osd.7) 209 : cluster 0 4.15 scrub starts 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:12.368346+0000 osd.7 (osd.7) 210 : cluster 0 4.15 scrub ok 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:12.685797+0000 osd.6 (osd.6) 345 : cluster 0 4.12 scrub starts 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:12.687844+0000 osd.6 (osd.6) 346 : cluster 0 4.12 scrub ok 2024-04-02T05:02:14.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:13.149611+0000 mon.a (mon.0) 3813 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: audit 2024-04-02T05:02:13.159751+0000 mon.a (mon.0) 3814 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: audit 2024-04-02T05:02:13.159982+0000 mon.a (mon.0) 3815 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:13.174645+0000 mon.a (mon.0) 3816 : cluster 0 osdmap e821: 8 total, 8 up, 7 in 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: audit 2024-04-02T05:02:13.176617+0000 mon.a (mon.0) 3817 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:13.200588+0000 osd.0 (osd.0) 399 : cluster 0 10.4 scrub starts 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: cluster 2024-04-02T05:02:13.203337+0000 osd.0 (osd.0) 400 : cluster 0 10.4 scrub ok 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: audit 2024-04-02T05:02:13.702034+0000 mgr.y (mgr.14538) 717 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:14.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:14 smithi069 bash[28904]: audit 2024-04-02T05:02:13.794192+0000 mon.c (mon.2) 1161 : audit 0 from='client.? 172.21.15.69:0/1290076307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:14.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:14.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:14.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:12.363744+0000 osd.7 (osd.7) 209 : cluster 0 4.15 scrub starts 2024-04-02T05:02:14.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:12.368346+0000 osd.7 (osd.7) 210 : cluster 0 4.15 scrub ok 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:12.685797+0000 osd.6 (osd.6) 345 : cluster 0 4.12 scrub starts 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:12.687844+0000 osd.6 (osd.6) 346 : cluster 0 4.12 scrub ok 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:13.149611+0000 mon.a (mon.0) 3813 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: audit 2024-04-02T05:02:13.159751+0000 mon.a (mon.0) 3814 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: audit 2024-04-02T05:02:13.159982+0000 mon.a (mon.0) 3815 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:13.174645+0000 mon.a (mon.0) 3816 : cluster 0 osdmap e821: 8 total, 8 up, 7 in 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: audit 2024-04-02T05:02:13.176617+0000 mon.a (mon.0) 3817 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:13.200588+0000 osd.0 (osd.0) 399 : cluster 0 10.4 scrub starts 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: cluster 2024-04-02T05:02:13.203337+0000 osd.0 (osd.0) 400 : cluster 0 10.4 scrub ok 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: audit 2024-04-02T05:02:13.702034+0000 mgr.y (mgr.14538) 717 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:14.428 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:14 smithi098 bash[23790]: audit 2024-04-02T05:02:13.794192+0000 mon.c (mon.2) 1161 : audit 0 from='client.? 172.21.15.69:0/1290076307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:14.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:14.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:14.709 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:14.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:14.839 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:14.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:14.967 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:14.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:15.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:15.099 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:15.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:15.213 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:13.641208+0000 osd.2 (osd.2) 353 : cluster 0 2.e scrub starts 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:13.642892+0000 osd.2 (osd.2) 354 : cluster 0 2.e scrub ok 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: audit 2024-04-02T05:02:14.172334+0000 mon.a (mon.0) 3818 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.182332+0000 mon.a (mon.0) 3819 : cluster 0 osdmap e822: 8 total, 8 up, 7 in 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.332602+0000 mgr.y (mgr.14538) 718 : cluster 0 pgmap v1246: 224 pgs: 224 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.393975+0000 osd.4 (osd.4) 404 : cluster 0 6.17 deep-scrub starts 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.397829+0000 osd.4 (osd.4) 405 : cluster 0 6.17 deep-scrub ok 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.098326+0000 mon.a (mon.0) 3820 : cluster 0 osdmap e823: 8 total, 8 up, 7 in 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:13.641208+0000 osd.2 (osd.2) 353 : cluster 0 2.e scrub starts 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:13.642892+0000 osd.2 (osd.2) 354 : cluster 0 2.e scrub ok 2024-04-02T05:02:15.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: audit 2024-04-02T05:02:14.172334+0000 mon.a (mon.0) 3818 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T05:02:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.182332+0000 mon.a (mon.0) 3819 : cluster 0 osdmap e822: 8 total, 8 up, 7 in 2024-04-02T05:02:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.332602+0000 mgr.y (mgr.14538) 718 : cluster 0 pgmap v1246: 224 pgs: 224 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.393975+0000 osd.4 (osd.4) 404 : cluster 0 6.17 deep-scrub starts 2024-04-02T05:02:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.397829+0000 osd.4 (osd.4) 405 : cluster 0 6.17 deep-scrub ok 2024-04-02T05:02:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:15 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.098326+0000 mon.a (mon.0) 3820 : cluster 0 osdmap e823: 8 total, 8 up, 7 in 2024-04-02T05:02:15.229 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:15.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:13.641208+0000 osd.2 (osd.2) 353 : cluster 0 2.e scrub starts 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:13.642892+0000 osd.2 (osd.2) 354 : cluster 0 2.e scrub ok 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: audit 2024-04-02T05:02:14.172334+0000 mon.a (mon.0) 3818 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.182332+0000 mon.a (mon.0) 3819 : cluster 0 osdmap e822: 8 total, 8 up, 7 in 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.332602+0000 mgr.y (mgr.14538) 718 : cluster 0 pgmap v1246: 224 pgs: 224 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.393975+0000 osd.4 (osd.4) 404 : cluster 0 6.17 deep-scrub starts 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.397829+0000 osd.4 (osd.4) 405 : cluster 0 6.17 deep-scrub ok 2024-04-02T05:02:15.344 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:15 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.098326+0000 mon.a (mon.0) 3820 : cluster 0 osdmap e823: 8 total, 8 up, 7 in 2024-04-02T05:02:15.358 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:15.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:15.487 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:15.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:15.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:15.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:15.748 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:15.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:15.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:15.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:15.991 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:16.012 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:16.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:16.143 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:16.247 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:16.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.305861+0000 osd.7 (osd.7) 211 : cluster 0 3.f scrub starts 2024-04-02T05:02:16.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.312817+0000 osd.7 (osd.7) 212 : cluster 0 3.f scrub ok 2024-04-02T05:02:16.258 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.760855+0000 osd.3 (osd.3) 409 : cluster 0 3.c scrub starts 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.764266+0000 osd.3 (osd.3) 410 : cluster 0 3.c scrub ok 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.775900+0000 osd.6 (osd.6) 347 : cluster 0 4.1 scrub starts 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:14.778073+0000 osd.6 (osd.6) 348 : cluster 0 4.1 scrub ok 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.111702+0000 osd.1 (osd.1) 401 : cluster 0 4.1c scrub starts 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.114842+0000 osd.1 (osd.1) 402 : cluster 0 4.1c scrub ok 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: audit 2024-04-02T05:02:15.215903+0000 mon.a (mon.0) 3821 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: audit 2024-04-02T05:02:15.216535+0000 mon.a (mon.0) 3822 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138"}]: dispatch 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.226824+0000 osd.0 (osd.0) 401 : cluster 0 2.1f scrub starts 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.228467+0000 osd.0 (osd.0) 402 : cluster 0 2.1f scrub ok 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.387724+0000 osd.4 (osd.4) 406 : cluster 0 10.17 scrub starts 2024-04-02T05:02:16.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[21528]: cluster 2024-04-02T05:02:15.392086+0000 osd.4 (osd.4) 407 : cluster 0 10.17 scrub ok 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.305861+0000 osd.7 (osd.7) 211 : cluster 0 3.f scrub starts 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.312817+0000 osd.7 (osd.7) 212 : cluster 0 3.f scrub ok 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.760855+0000 osd.3 (osd.3) 409 : cluster 0 3.c scrub starts 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.764266+0000 osd.3 (osd.3) 410 : cluster 0 3.c scrub ok 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.775900+0000 osd.6 (osd.6) 347 : cluster 0 4.1 scrub starts 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:14.778073+0000 osd.6 (osd.6) 348 : cluster 0 4.1 scrub ok 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.111702+0000 osd.1 (osd.1) 401 : cluster 0 4.1c scrub starts 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.114842+0000 osd.1 (osd.1) 402 : cluster 0 4.1c scrub ok 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: audit 2024-04-02T05:02:15.215903+0000 mon.a (mon.0) 3821 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: audit 2024-04-02T05:02:15.216535+0000 mon.a (mon.0) 3822 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138"}]: dispatch 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.226824+0000 osd.0 (osd.0) 401 : cluster 0 2.1f scrub starts 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.228467+0000 osd.0 (osd.0) 402 : cluster 0 2.1f scrub ok 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.387724+0000 osd.4 (osd.4) 406 : cluster 0 10.17 scrub starts 2024-04-02T05:02:16.260 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:16 smithi069 bash[28904]: cluster 2024-04-02T05:02:15.392086+0000 osd.4 (osd.4) 407 : cluster 0 10.17 scrub ok 2024-04-02T05:02:16.274 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:16.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.305861+0000 osd.7 (osd.7) 211 : cluster 0 3.f scrub starts 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.312817+0000 osd.7 (osd.7) 212 : cluster 0 3.f scrub ok 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.760855+0000 osd.3 (osd.3) 409 : cluster 0 3.c scrub starts 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.764266+0000 osd.3 (osd.3) 410 : cluster 0 3.c scrub ok 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.775900+0000 osd.6 (osd.6) 347 : cluster 0 4.1 scrub starts 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:14.778073+0000 osd.6 (osd.6) 348 : cluster 0 4.1 scrub ok 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.111702+0000 osd.1 (osd.1) 401 : cluster 0 4.1c scrub starts 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.114842+0000 osd.1 (osd.1) 402 : cluster 0 4.1c scrub ok 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: audit 2024-04-02T05:02:15.215903+0000 mon.a (mon.0) 3821 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: audit 2024-04-02T05:02:15.216535+0000 mon.a (mon.0) 3822 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138"}]: dispatch 2024-04-02T05:02:16.389 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.226824+0000 osd.0 (osd.0) 401 : cluster 0 2.1f scrub starts 2024-04-02T05:02:16.390 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.228467+0000 osd.0 (osd.0) 402 : cluster 0 2.1f scrub ok 2024-04-02T05:02:16.390 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.387724+0000 osd.4 (osd.4) 406 : cluster 0 10.17 scrub starts 2024-04-02T05:02:16.390 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:16 smithi098 bash[23790]: cluster 2024-04-02T05:02:15.392086+0000 osd.4 (osd.4) 407 : cluster 0 10.17 scrub ok 2024-04-02T05:02:16.405 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:16.508 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:16.532 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:16.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:16.664 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:16.767 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:16.797 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:16.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:16.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T05:02:16.926 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:16.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:17.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:17.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:17.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:17.181 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:17.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:17.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:17 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.062923+0000 osd.1 (osd.1) 403 : cluster 0 3.19 scrub starts 2024-04-02T05:02:17.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:17 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.066415+0000 osd.1 (osd.1) 404 : cluster 0 3.19 scrub ok 2024-04-02T05:02:17.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:17 smithi098 bash[23790]: audit 2024-04-02T05:02:16.186687+0000 mon.a (mon.0) 3823 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138"}]': finished 2024-04-02T05:02:17.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:17 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.194169+0000 mon.a (mon.0) 3824 : cluster 0 osdmap e824: 8 total, 8 up, 7 in 2024-04-02T05:02:17.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:17 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.333192+0000 mgr.y (mgr.14538) 719 : cluster 0 pgmap v1249: 224 pgs: 224 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:17.313 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:17.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:17.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.062923+0000 osd.1 (osd.1) 403 : cluster 0 3.19 scrub starts 2024-04-02T05:02:17.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.066415+0000 osd.1 (osd.1) 404 : cluster 0 3.19 scrub ok 2024-04-02T05:02:17.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[28904]: audit 2024-04-02T05:02:16.186687+0000 mon.a (mon.0) 3823 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138"}]': finished 2024-04-02T05:02:17.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.194169+0000 mon.a (mon.0) 3824 : cluster 0 osdmap e824: 8 total, 8 up, 7 in 2024-04-02T05:02:17.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.333192+0000 mgr.y (mgr.14538) 719 : cluster 0 pgmap v1249: 224 pgs: 224 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:17.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.062923+0000 osd.1 (osd.1) 403 : cluster 0 3.19 scrub starts 2024-04-02T05:02:17.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.066415+0000 osd.1 (osd.1) 404 : cluster 0 3.19 scrub ok 2024-04-02T05:02:17.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[21528]: audit 2024-04-02T05:02:16.186687+0000 mon.a (mon.0) 3823 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-138"}]': finished 2024-04-02T05:02:17.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.194169+0000 mon.a (mon.0) 3824 : cluster 0 osdmap e824: 8 total, 8 up, 7 in 2024-04-02T05:02:17.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:17 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.333192+0000 mgr.y (mgr.14538) 719 : cluster 0 pgmap v1249: 224 pgs: 224 active+clean; 4.4 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:17.438 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:17.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:02:17.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:17.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:17.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:17.830 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:17.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:17.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:18.062 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:18.089 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:18.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:18.219 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:18.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:18.349 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:18.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.722731+0000 osd.2 (osd.2) 355 : cluster 0 10.9 deep-scrub starts 2024-04-02T05:02:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.726887+0000 osd.2 (osd.2) 356 : cluster 0 10.9 deep-scrub ok 2024-04-02T05:02:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.767966+0000 osd.3 (osd.3) 411 : cluster 0 3.7 deep-scrub starts 2024-04-02T05:02:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:16.780198+0000 osd.3 (osd.3) 412 : cluster 0 3.7 deep-scrub ok 2024-04-02T05:02:18.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.201008+0000 mon.a (mon.0) 3825 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:18.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.211574+0000 osd.0 (osd.0) 403 : cluster 0 6.16 deep-scrub starts 2024-04-02T05:02:18.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.214548+0000 osd.0 (osd.0) 404 : cluster 0 6.16 deep-scrub ok 2024-04-02T05:02:18.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.216122+0000 mon.a (mon.0) 3826 : cluster 0 osdmap e825: 8 total, 8 up, 7 in 2024-04-02T05:02:18.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.351808+0000 osd.4 (osd.4) 408 : cluster 0 4.18 scrub starts 2024-04-02T05:02:18.465 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:18 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.353283+0000 osd.4 (osd.4) 409 : cluster 0 4.18 scrub ok 2024-04-02T05:02:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.722731+0000 osd.2 (osd.2) 355 : cluster 0 10.9 deep-scrub starts 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.726887+0000 osd.2 (osd.2) 356 : cluster 0 10.9 deep-scrub ok 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.767966+0000 osd.3 (osd.3) 411 : cluster 0 3.7 deep-scrub starts 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:16.780198+0000 osd.3 (osd.3) 412 : cluster 0 3.7 deep-scrub ok 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.201008+0000 mon.a (mon.0) 3825 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.211574+0000 osd.0 (osd.0) 403 : cluster 0 6.16 deep-scrub starts 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.214548+0000 osd.0 (osd.0) 404 : cluster 0 6.16 deep-scrub ok 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.216122+0000 mon.a (mon.0) 3826 : cluster 0 osdmap e825: 8 total, 8 up, 7 in 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.351808+0000 osd.4 (osd.4) 408 : cluster 0 4.18 scrub starts 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.353283+0000 osd.4 (osd.4) 409 : cluster 0 4.18 scrub ok 2024-04-02T05:02:18.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.722731+0000 osd.2 (osd.2) 355 : cluster 0 10.9 deep-scrub starts 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.726887+0000 osd.2 (osd.2) 356 : cluster 0 10.9 deep-scrub ok 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.767966+0000 osd.3 (osd.3) 411 : cluster 0 3.7 deep-scrub starts 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:16.780198+0000 osd.3 (osd.3) 412 : cluster 0 3.7 deep-scrub ok 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.201008+0000 mon.a (mon.0) 3825 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.211574+0000 osd.0 (osd.0) 403 : cluster 0 6.16 deep-scrub starts 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.214548+0000 osd.0 (osd.0) 404 : cluster 0 6.16 deep-scrub ok 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.216122+0000 mon.a (mon.0) 3826 : cluster 0 osdmap e825: 8 total, 8 up, 7 in 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.351808+0000 osd.4 (osd.4) 408 : cluster 0 4.18 scrub starts 2024-04-02T05:02:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.353283+0000 osd.4 (osd.4) 409 : cluster 0 4.18 scrub ok 2024-04-02T05:02:18.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:18.610 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:18.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:18.740 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:18.741 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:02:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:02:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:02:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:18.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:18.871 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:18.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:18.916 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:18.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:19.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:19.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:19.270 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:19.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.385 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.420943+0000 osd.7 (osd.7) 213 : cluster 0 3.11 scrub starts 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.429890+0000 osd.7 (osd.7) 214 : cluster 0 3.11 scrub ok 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.777256+0000 osd.6 (osd.6) 349 : cluster 0 10.0 scrub starts 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: cluster 2024-04-02T05:02:17.779526+0000 osd.6 (osd.6) 350 : cluster 0 10.0 scrub ok 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: cluster 2024-04-02T05:02:18.233273+0000 mon.a (mon.0) 3827 : cluster 0 osdmap e826: 8 total, 8 up, 7 in 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: audit 2024-04-02T05:02:18.236126+0000 mon.a (mon.0) 3828 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: cluster 2024-04-02T05:02:18.333981+0000 mgr.y (mgr.14538) 720 : cluster 0 pgmap v1252: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: audit 2024-04-02T05:02:19.065542+0000 mon.a (mon.0) 3829 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:19.386 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:19 smithi098 bash[23790]: audit 2024-04-02T05:02:19.067131+0000 mon.b (mon.1) 312 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:19.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.531 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.420943+0000 osd.7 (osd.7) 213 : cluster 0 3.11 scrub starts 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.429890+0000 osd.7 (osd.7) 214 : cluster 0 3.11 scrub ok 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.777256+0000 osd.6 (osd.6) 349 : cluster 0 10.0 scrub starts 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: cluster 2024-04-02T05:02:17.779526+0000 osd.6 (osd.6) 350 : cluster 0 10.0 scrub ok 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: cluster 2024-04-02T05:02:18.233273+0000 mon.a (mon.0) 3827 : cluster 0 osdmap e826: 8 total, 8 up, 7 in 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: audit 2024-04-02T05:02:18.236126+0000 mon.a (mon.0) 3828 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: cluster 2024-04-02T05:02:18.333981+0000 mgr.y (mgr.14538) 720 : cluster 0 pgmap v1252: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: audit 2024-04-02T05:02:19.065542+0000 mon.a (mon.0) 3829 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:19.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[21528]: audit 2024-04-02T05:02:19.067131+0000 mon.b (mon.1) 312 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.420943+0000 osd.7 (osd.7) 213 : cluster 0 3.11 scrub starts 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.429890+0000 osd.7 (osd.7) 214 : cluster 0 3.11 scrub ok 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.777256+0000 osd.6 (osd.6) 349 : cluster 0 10.0 scrub starts 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: cluster 2024-04-02T05:02:17.779526+0000 osd.6 (osd.6) 350 : cluster 0 10.0 scrub ok 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: cluster 2024-04-02T05:02:18.233273+0000 mon.a (mon.0) 3827 : cluster 0 osdmap e826: 8 total, 8 up, 7 in 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: audit 2024-04-02T05:02:18.236126+0000 mon.a (mon.0) 3828 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: cluster 2024-04-02T05:02:18.333981+0000 mgr.y (mgr.14538) 720 : cluster 0 pgmap v1252: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: audit 2024-04-02T05:02:19.065542+0000 mon.a (mon.0) 3829 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:19.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:19 smithi069 bash[28904]: audit 2024-04-02T05:02:19.067131+0000 mon.b (mon.1) 312 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:19.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.663 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:19.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:19.793 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:19.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:19.923 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:19.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:20.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:20.054 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:20.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:20.283 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:20.306 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:20.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:20.421 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:20 smithi098 bash[23790]: cluster 2024-04-02T05:02:18.712670+0000 osd.2 (osd.2) 357 : cluster 0 1.8 scrub starts 2024-04-02T05:02:20.421 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:20 smithi098 bash[23790]: cluster 2024-04-02T05:02:18.715794+0000 osd.2 (osd.2) 358 : cluster 0 1.8 scrub ok 2024-04-02T05:02:20.421 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:20 smithi098 bash[23790]: audit 2024-04-02T05:02:19.236345+0000 mon.a (mon.0) 3830 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:20.421 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:20 smithi098 bash[23790]: cluster 2024-04-02T05:02:19.246009+0000 mon.a (mon.0) 3831 : cluster 0 osdmap e827: 8 total, 8 up, 7 in 2024-04-02T05:02:20.421 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:20 smithi098 bash[23790]: audit 2024-04-02T05:02:19.248226+0000 mon.a (mon.0) 3832 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:20.438 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:20.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[21528]: cluster 2024-04-02T05:02:18.712670+0000 osd.2 (osd.2) 357 : cluster 0 1.8 scrub starts 2024-04-02T05:02:20.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[21528]: cluster 2024-04-02T05:02:18.715794+0000 osd.2 (osd.2) 358 : cluster 0 1.8 scrub ok 2024-04-02T05:02:20.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[21528]: audit 2024-04-02T05:02:19.236345+0000 mon.a (mon.0) 3830 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:20.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[21528]: cluster 2024-04-02T05:02:19.246009+0000 mon.a (mon.0) 3831 : cluster 0 osdmap e827: 8 total, 8 up, 7 in 2024-04-02T05:02:20.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[21528]: audit 2024-04-02T05:02:19.248226+0000 mon.a (mon.0) 3832 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:20.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[28904]: cluster 2024-04-02T05:02:18.712670+0000 osd.2 (osd.2) 357 : cluster 0 1.8 scrub starts 2024-04-02T05:02:20.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[28904]: cluster 2024-04-02T05:02:18.715794+0000 osd.2 (osd.2) 358 : cluster 0 1.8 scrub ok 2024-04-02T05:02:20.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[28904]: audit 2024-04-02T05:02:19.236345+0000 mon.a (mon.0) 3830 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:20.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[28904]: cluster 2024-04-02T05:02:19.246009+0000 mon.a (mon.0) 3831 : cluster 0 osdmap e827: 8 total, 8 up, 7 in 2024-04-02T05:02:20.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:20 smithi069 bash[28904]: audit 2024-04-02T05:02:19.248226+0000 mon.a (mon.0) 3832 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:20.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:20.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:20.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:20.696 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:20.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 -- 172.21.15.69:0/2918548294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806a890 msgr2=0x7f81e810e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 --2- 172.21.15.69:0/2918548294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806a890 0x7f81e810e810 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f81d4009990 tx=0x7f81d4009ca0 comp rx=0 tx=0).stop 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 -- 172.21.15.69:0/2918548294 shutdown_connections 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 --2- 172.21.15.69:0/2918548294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e810ed50 0x7f81e8111140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 --2- 172.21.15.69:0/2918548294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e8069f70 0x7f81e806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 --2- 172.21.15.69:0/2918548294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e806a890 0x7f81e810e810 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 -- 172.21.15.69:0/2918548294 >> 172.21.15.69:0/2918548294 conn(0x7f81e80771e0 msgr2=0x7f81e80775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 -- 172.21.15.69:0/2918548294 shutdown_connections 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 -- 172.21.15.69:0/2918548294 wait complete. 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 librados: starting msgr at 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 librados: starting objecter 2024-04-02T05:02:20.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.864+0000 7f81ef156700 1 Processor -- start 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- start start 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 librados: setting wanted keys 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 librados: calling monclient init 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e8069f70 0x7f81e819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e806a890 0x7f81e819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e810ed50 0x7f81e819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8113ca0 con 0x7f81e806a890 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8113b20 con 0x7f81e810ed50 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f81e8113e20 con 0x7f81e8069f70 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e8069f70 0x7f81e819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e8069f70 0x7f81e819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41720/0 (socket says 172.21.15.69:41720) 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 -- 172.21.15.69:0/2066675334 learned_addr learned my addr 172.21.15.69:0/2066675334 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e7fff700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e806a890 0x7f81e819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:20.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ed6f3700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e810ed50 0x7f81e819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 -- 172.21.15.69:0/2066675334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e810ed50 msgr2=0x7f81e819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e810ed50 0x7f81e819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 -- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e806a890 msgr2=0x7f81e819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:20.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81e806a890 0x7f81e819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 -- 172.21.15.69:0/2066675334 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e819f2e0 con 0x7f81e8069f70 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ed6f3700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e810ed50 0x7f81e819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ecef2700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81e8069f70 0x7f81e819a2d0 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00eb30 tx=0x7f81dc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 10 client.?.objecter ms_handle_connect 0x7f81e8069f70 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00cc90 con 0x7f81e8069f70 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- 172.21.15.69:0/2066675334 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e819f5d0 con 0x7f81e8069f70 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f81dc00ce30 con 0x7f81e8069f70 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc003df0 con 0x7f81e8069f70 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 10 client.43811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:20.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- 172.21.15.69:0/2066675334 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81a6e50 con 0x7f81e8069f70 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 librados: init done 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 10 client.43811.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 15 client.43811.objecter _session_command_op_assign -1 1 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 20 client.43811.objecter _get_session osd=-1 returning homeless 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 10 client.43811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ef156700 1 -- 172.21.15.69:0/2066675334 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81e8101780 con 0x7f81e8069f70 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f81dc004000 con 0x7f81e8069f70 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f81d806f5e0 0x7f81d8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(828..828 src has 74..828) ==== 10107+0+0 (secure 0 0 0) 0x7f81dc08ed30 con 0x7f81e8069f70 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e7fff700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f81d806f5e0 0x7f81d8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 10 client.43811.objecter ms_dispatch 0x7f81e8001550 osd_map(828..828 src has 74..828) 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 3 client.43811.objecter handle_osd_map got epochs [828,828] > 0 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 3 client.43811.objecter handle_osd_map decoding full epoch 828 2024-04-02T05:02:20.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 10 client.43811.objecter checking command 1 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f81d807d2a0 0x7f81d807f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 20 client.43811.objecter _get_session s=0x7f81d807cbd0 osd=3 3 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 20 client.43811.objecter put_session s=0x7f81d807cbd0 osd=3 3 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 15 client.43811.objecter _session_command_op_remove -1 1 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 20 client.43811.objecter _get_session s=0x7f81d807cbd0 osd=3 3 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 20 client.43811.objecter get_session s=0x7f81d807cbd0 osd=3 3 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 15 client.43811.objecter _session_command_op_assign 3 1 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 20 client.43811.objecter put_session s=0x7f81d807cbd0 osd=3 4 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 10 client.43811.objecter _send_command 1 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81d807fe10 con 0x7f81d807d2a0 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 20 client.43811.objecter dump_active .. 0 homeless 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=828) ==== 24+0+0 (secure 0 0 0) 0x7f81dc094400 con 0x7f81e8069f70 2024-04-02T05:02:20.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.868+0000 7f81ed6f3700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f81d807d2a0 0x7f81d807f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:20.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 10 client.43811.objecter ms_handle_connect 0x7f81d806f5e0 2024-04-02T05:02:20.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e7fff700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f81d806f5e0 0x7f81d8071aa0 secure :-1 s=READY pgs=4015 cs=0 l=1 rev1=1 crypto rx=0x7f81d400b2e0 tx=0x7f81d4005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:20.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 10 client.43811.objecter ms_handle_connect 0x7f81d807d2a0 2024-04-02T05:02:20.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81ed6f3700 1 --2- 172.21.15.69:0/2066675334 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f81d807d2a0 0x7f81d807f740 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:20.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 1 -- 172.21.15.69:0/2066675334 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81d807fe10 con 0x7f81d807d2a0 2024-04-02T05:02:20.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 10 client.43811.objecter ms_dispatch 0x7f81e8001550 command_reply(tid 1: 0 ) 2024-04-02T05:02:20.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 10 client.43811.objecter _finish_command 1 = system:0 2024-04-02T05:02:20.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 20 client.43811.objecter put_session s=0x7f81d807cbd0 osd=3 4 2024-04-02T05:02:20.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:20.872+0000 7f81e5ffb700 15 client.43811.objecter _session_command_op_remove 3 1 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:no valid command found; 10 closest matches: 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:0 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:1 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:abort 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:assert 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:bench [] [] [] [] 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs debug_inject_read_zeros 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs files list 2024-04-02T05:02:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:bluefs stats 2024-04-02T05:02:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:bluestore allocator dump block 2024-04-02T05:02:20.911 INFO:teuthology.orchestra.run.smithi069.stderr:Error EINVAL: invalid command 2024-04-02T05:02:20.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:20.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:21.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:21.083 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:21.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:21.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:21.212 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:21.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:21.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:21.342 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:21.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:21.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:21.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.036257+0000 osd.1 (osd.1) 405 : cluster 0 6.1d deep-scrub starts 2024-04-02T05:02:21.456 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.038574+0000 osd.1 (osd.1) 406 : cluster 0 6.1d deep-scrub ok 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.260951+0000 osd.4 (osd.4) 410 : cluster 0 4.13 scrub starts 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.263270+0000 osd.4 (osd.4) 411 : cluster 0 4.13 scrub ok 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: audit 2024-04-02T05:02:20.272890+0000 mon.a (mon.0) 3833 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.276848+0000 mon.a (mon.0) 3834 : cluster 0 osdmap e828: 8 total, 8 up, 7 in 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: audit 2024-04-02T05:02:20.277398+0000 mon.a (mon.0) 3835 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.334575+0000 mgr.y (mgr.14538) 721 : cluster 0 pgmap v1255: 224 pgs: 3 creating+peering, 29 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.676372+0000 osd.3 (osd.3) 413 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[28904]: cluster 2024-04-02T05:02:20.676697+0000 osd.3 (osd.3) 414 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.036257+0000 osd.1 (osd.1) 405 : cluster 0 6.1d deep-scrub starts 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.038574+0000 osd.1 (osd.1) 406 : cluster 0 6.1d deep-scrub ok 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.260951+0000 osd.4 (osd.4) 410 : cluster 0 4.13 scrub starts 2024-04-02T05:02:21.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.263270+0000 osd.4 (osd.4) 411 : cluster 0 4.13 scrub ok 2024-04-02T05:02:21.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: audit 2024-04-02T05:02:20.272890+0000 mon.a (mon.0) 3833 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:21.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.276848+0000 mon.a (mon.0) 3834 : cluster 0 osdmap e828: 8 total, 8 up, 7 in 2024-04-02T05:02:21.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: audit 2024-04-02T05:02:20.277398+0000 mon.a (mon.0) 3835 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T05:02:21.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.334575+0000 mgr.y (mgr.14538) 721 : cluster 0 pgmap v1255: 224 pgs: 3 creating+peering, 29 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:21.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.676372+0000 osd.3 (osd.3) 413 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:21.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:21 smithi069 bash[21528]: cluster 2024-04-02T05:02:20.676697+0000 osd.3 (osd.3) 414 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:21.472 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:21.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:21.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:21.601 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:21.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:21.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.036257+0000 osd.1 (osd.1) 405 : cluster 0 6.1d deep-scrub starts 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.038574+0000 osd.1 (osd.1) 406 : cluster 0 6.1d deep-scrub ok 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.260951+0000 osd.4 (osd.4) 410 : cluster 0 4.13 scrub starts 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.263270+0000 osd.4 (osd.4) 411 : cluster 0 4.13 scrub ok 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: audit 2024-04-02T05:02:20.272890+0000 mon.a (mon.0) 3833 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.276848+0000 mon.a (mon.0) 3834 : cluster 0 osdmap e828: 8 total, 8 up, 7 in 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: audit 2024-04-02T05:02:20.277398+0000 mon.a (mon.0) 3835 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-02T05:02:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.334575+0000 mgr.y (mgr.14538) 721 : cluster 0 pgmap v1255: 224 pgs: 3 creating+peering, 29 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.676372+0000 osd.3 (osd.3) 413 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:21.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:21 smithi098 bash[23790]: cluster 2024-04-02T05:02:20.676697+0000 osd.3 (osd.3) 414 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:21.730 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:21.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:21.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:22.112 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:22.141 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:22.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:22.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:22.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.519 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:22 smithi098 bash[23790]: audit 2024-04-02T05:02:21.281859+0000 mon.a (mon.0) 3836 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_count","val": "3"}]': finished 2024-04-02T05:02:22.519 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:22 smithi098 bash[23790]: cluster 2024-04-02T05:02:21.287595+0000 mon.a (mon.0) 3837 : cluster 0 osdmap e829: 8 total, 8 up, 7 in 2024-04-02T05:02:22.519 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:22 smithi098 bash[23790]: audit 2024-04-02T05:02:21.292749+0000 mon.a (mon.0) 3838 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T05:02:22.534 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:22.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:22.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:22 smithi069 bash[21528]: audit 2024-04-02T05:02:21.281859+0000 mon.a (mon.0) 3836 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_count","val": "3"}]': finished 2024-04-02T05:02:22.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:22 smithi069 bash[21528]: cluster 2024-04-02T05:02:21.287595+0000 mon.a (mon.0) 3837 : cluster 0 osdmap e829: 8 total, 8 up, 7 in 2024-04-02T05:02:22.650 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:22 smithi069 bash[21528]: audit 2024-04-02T05:02:21.292749+0000 mon.a (mon.0) 3838 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T05:02:22.650 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:22 smithi069 bash[28904]: audit 2024-04-02T05:02:21.281859+0000 mon.a (mon.0) 3836 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_count","val": "3"}]': finished 2024-04-02T05:02:22.650 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:22 smithi069 bash[28904]: cluster 2024-04-02T05:02:21.287595+0000 mon.a (mon.0) 3837 : cluster 0 osdmap e829: 8 total, 8 up, 7 in 2024-04-02T05:02:22.650 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:22 smithi069 bash[28904]: audit 2024-04-02T05:02:21.292749+0000 mon.a (mon.0) 3838 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-02T05:02:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T05:02:22.651 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-02T05:02:22.662 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:22.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:22.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.793 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:22.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.922 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:22.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:23.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:23.052 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:23.123 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:23.124 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:23.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:23.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:23.305 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:23.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:23.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:21.791852+0000 osd.3 (osd.3) 415 : cluster 0 6.11 scrub starts 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:21.794006+0000 osd.3 (osd.3) 416 : cluster 0 6.11 scrub ok 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:21.998975+0000 osd.1 (osd.1) 407 : cluster 0 2.6 scrub starts 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.143272+0000 osd.1 (osd.1) 408 : cluster 0 2.6 scrub ok 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: audit 2024-04-02T05:02:22.284396+0000 mon.a (mon.0) 3839 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_period","val": "3"}]': finished 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.296125+0000 mon.a (mon.0) 3840 : cluster 0 osdmap e830: 8 total, 8 up, 7 in 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: audit 2024-04-02T05:02:22.300482+0000 mon.a (mon.0) 3841 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.335371+0000 mgr.y (mgr.14538) 722 : cluster 0 pgmap v1258: 224 pgs: 3 creating+peering, 29 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.716095+0000 osd.2 (osd.2) 359 : cluster 0 4.d deep-scrub starts 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.718254+0000 osd.2 (osd.2) 360 : cluster 0 4.d deep-scrub ok 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: audit 2024-04-02T05:02:23.287213+0000 mon.a (mon.0) 3842 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: cluster 2024-04-02T05:02:23.294095+0000 mon.a (mon.0) 3843 : cluster 0 osdmap e831: 8 total, 8 up, 7 in 2024-04-02T05:02:23.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[21528]: audit 2024-04-02T05:02:23.297479+0000 mon.a (mon.0) 3844 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:21.791852+0000 osd.3 (osd.3) 415 : cluster 0 6.11 scrub starts 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:21.794006+0000 osd.3 (osd.3) 416 : cluster 0 6.11 scrub ok 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:21.998975+0000 osd.1 (osd.1) 407 : cluster 0 2.6 scrub starts 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.143272+0000 osd.1 (osd.1) 408 : cluster 0 2.6 scrub ok 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: audit 2024-04-02T05:02:22.284396+0000 mon.a (mon.0) 3839 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_period","val": "3"}]': finished 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.296125+0000 mon.a (mon.0) 3840 : cluster 0 osdmap e830: 8 total, 8 up, 7 in 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: audit 2024-04-02T05:02:22.300482+0000 mon.a (mon.0) 3841 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.335371+0000 mgr.y (mgr.14538) 722 : cluster 0 pgmap v1258: 224 pgs: 3 creating+peering, 29 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.716095+0000 osd.2 (osd.2) 359 : cluster 0 4.d deep-scrub starts 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.718254+0000 osd.2 (osd.2) 360 : cluster 0 4.d deep-scrub ok 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: audit 2024-04-02T05:02:23.287213+0000 mon.a (mon.0) 3842 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:02:23.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: cluster 2024-04-02T05:02:23.294095+0000 mon.a (mon.0) 3843 : cluster 0 osdmap e831: 8 total, 8 up, 7 in 2024-04-02T05:02:23.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:23 smithi069 bash[28904]: audit 2024-04-02T05:02:23.297479+0000 mon.a (mon.0) 3844 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T05:02:23.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:23.564 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:23.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:23.694 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:23.712 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:21.791852+0000 osd.3 (osd.3) 415 : cluster 0 6.11 scrub starts 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:21.794006+0000 osd.3 (osd.3) 416 : cluster 0 6.11 scrub ok 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:21.998975+0000 osd.1 (osd.1) 407 : cluster 0 2.6 scrub starts 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.143272+0000 osd.1 (osd.1) 408 : cluster 0 2.6 scrub ok 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: audit 2024-04-02T05:02:22.284396+0000 mon.a (mon.0) 3839 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_period","val": "3"}]': finished 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.296125+0000 mon.a (mon.0) 3840 : cluster 0 osdmap e830: 8 total, 8 up, 7 in 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: audit 2024-04-02T05:02:22.300482+0000 mon.a (mon.0) 3841 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.335371+0000 mgr.y (mgr.14538) 722 : cluster 0 pgmap v1258: 224 pgs: 3 creating+peering, 29 unknown, 192 active+clean; 4.4 MiB data, 750 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.716095+0000 osd.2 (osd.2) 359 : cluster 0 4.d deep-scrub starts 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.718254+0000 osd.2 (osd.2) 360 : cluster 0 4.d deep-scrub ok 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: audit 2024-04-02T05:02:23.287213+0000 mon.a (mon.0) 3842 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: cluster 2024-04-02T05:02:23.294095+0000 mon.a (mon.0) 3843 : cluster 0 osdmap e831: 8 total, 8 up, 7 in 2024-04-02T05:02:23.713 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[23790]: audit 2024-04-02T05:02:23.297479+0000 mon.a (mon.0) 3844 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-02T05:02:23.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:23.826 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:23.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:23.956 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:23.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:23.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:02:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:02:24.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:24.089 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:24.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:24.217 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:24.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:24.348 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:24.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:24.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.842543+0000 osd.3 (osd.3) 417 : cluster 0 2.19 scrub starts 2024-04-02T05:02:24.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.847360+0000 osd.3 (osd.3) 418 : cluster 0 2.19 scrub ok 2024-04-02T05:02:24.462 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.860433+0000 osd.6 (osd.6) 351 : cluster 0 6.a scrub starts 2024-04-02T05:02:24.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: cluster 2024-04-02T05:02:22.864303+0000 osd.6 (osd.6) 352 : cluster 0 6.a scrub ok 2024-04-02T05:02:24.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: audit 2024-04-02T05:02:23.711034+0000 mgr.y (mgr.14538) 723 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:24.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: audit 2024-04-02T05:02:24.289511+0000 mon.a (mon.0) 3845 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-02T05:02:24.463 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:24 smithi098 bash[23790]: cluster 2024-04-02T05:02:24.297957+0000 mon.a (mon.0) 3846 : cluster 0 osdmap e832: 8 total, 8 up, 7 in 2024-04-02T05:02:24.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:24.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:24.608 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:24.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:24.738 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:24.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.842543+0000 osd.3 (osd.3) 417 : cluster 0 2.19 scrub starts 2024-04-02T05:02:24.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.847360+0000 osd.3 (osd.3) 418 : cluster 0 2.19 scrub ok 2024-04-02T05:02:24.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.860433+0000 osd.6 (osd.6) 351 : cluster 0 6.a scrub starts 2024-04-02T05:02:24.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: cluster 2024-04-02T05:02:22.864303+0000 osd.6 (osd.6) 352 : cluster 0 6.a scrub ok 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: audit 2024-04-02T05:02:23.711034+0000 mgr.y (mgr.14538) 723 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: audit 2024-04-02T05:02:24.289511+0000 mon.a (mon.0) 3845 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[21528]: cluster 2024-04-02T05:02:24.297957+0000 mon.a (mon.0) 3846 : cluster 0 osdmap e832: 8 total, 8 up, 7 in 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.842543+0000 osd.3 (osd.3) 417 : cluster 0 2.19 scrub starts 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.847360+0000 osd.3 (osd.3) 418 : cluster 0 2.19 scrub ok 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.860433+0000 osd.6 (osd.6) 351 : cluster 0 6.a scrub starts 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: cluster 2024-04-02T05:02:22.864303+0000 osd.6 (osd.6) 352 : cluster 0 6.a scrub ok 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: audit 2024-04-02T05:02:23.711034+0000 mgr.y (mgr.14538) 723 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: audit 2024-04-02T05:02:24.289511+0000 mon.a (mon.0) 3845 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-02T05:02:24.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:24 smithi069 bash[28904]: cluster 2024-04-02T05:02:24.297957+0000 mon.a (mon.0) 3846 : cluster 0 osdmap e832: 8 total, 8 up, 7 in 2024-04-02T05:02:24.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:24.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:24.997 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:24.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:25.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:25.128 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:25.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:25.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:25.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:25.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:25.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:25.519 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:25.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:25.635 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:23.834960+0000 osd.6 (osd.6) 353 : cluster 0 10.16 scrub starts 2024-04-02T05:02:25.635 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:23.842686+0000 osd.6 (osd.6) 354 : cluster 0 10.16 scrub ok 2024-04-02T05:02:25.635 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:23.848316+0000 osd.3 (osd.3) 419 : cluster 0 5.0 deep-scrub starts 2024-04-02T05:02:25.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:23.853858+0000 osd.3 (osd.3) 420 : cluster 0 5.0 deep-scrub ok 2024-04-02T05:02:25.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:24.317388+0000 osd.7 (osd.7) 215 : cluster 0 10.1f deep-scrub starts 2024-04-02T05:02:25.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:24.327563+0000 osd.7 (osd.7) 216 : cluster 0 10.1f deep-scrub ok 2024-04-02T05:02:25.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:24.336192+0000 mgr.y (mgr.14538) 724 : cluster 0 pgmap v1261: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:25.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.093679+0000 mon.a (mon.0) 3847 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:25.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:25 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.105487+0000 mon.a (mon.0) 3848 : cluster 0 osdmap e833: 8 total, 8 up, 7 in 2024-04-02T05:02:25.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:23.834960+0000 osd.6 (osd.6) 353 : cluster 0 10.16 scrub starts 2024-04-02T05:02:25.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:23.842686+0000 osd.6 (osd.6) 354 : cluster 0 10.16 scrub ok 2024-04-02T05:02:25.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:23.848316+0000 osd.3 (osd.3) 419 : cluster 0 5.0 deep-scrub starts 2024-04-02T05:02:25.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:23.853858+0000 osd.3 (osd.3) 420 : cluster 0 5.0 deep-scrub ok 2024-04-02T05:02:25.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:24.317388+0000 osd.7 (osd.7) 215 : cluster 0 10.1f deep-scrub starts 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:24.327563+0000 osd.7 (osd.7) 216 : cluster 0 10.1f deep-scrub ok 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:24.336192+0000 mgr.y (mgr.14538) 724 : cluster 0 pgmap v1261: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.093679+0000 mon.a (mon.0) 3847 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.105487+0000 mon.a (mon.0) 3848 : cluster 0 osdmap e833: 8 total, 8 up, 7 in 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:23.834960+0000 osd.6 (osd.6) 353 : cluster 0 10.16 scrub starts 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:23.842686+0000 osd.6 (osd.6) 354 : cluster 0 10.16 scrub ok 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:23.848316+0000 osd.3 (osd.3) 419 : cluster 0 5.0 deep-scrub starts 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:23.853858+0000 osd.3 (osd.3) 420 : cluster 0 5.0 deep-scrub ok 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:24.317388+0000 osd.7 (osd.7) 215 : cluster 0 10.1f deep-scrub starts 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:24.327563+0000 osd.7 (osd.7) 216 : cluster 0 10.1f deep-scrub ok 2024-04-02T05:02:25.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:24.336192+0000 mgr.y (mgr.14538) 724 : cluster 0 pgmap v1261: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:25.640 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.093679+0000 mon.a (mon.0) 3847 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:25.640 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:25 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.105487+0000 mon.a (mon.0) 3848 : cluster 0 osdmap e833: 8 total, 8 up, 7 in 2024-04-02T05:02:25.649 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:25.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:25.777 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:25.826 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T05:02:25.826 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:02:25.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:25.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:25.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:26.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:26.023 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:26.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:26.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:26.380 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:26.402 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: cluster 2024-04-02T05:02:24.972114+0000 osd.1 (osd.1) 409 : cluster 0 323.6s0 deep-scrub starts 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: cluster 2024-04-02T05:02:24.974690+0000 osd.1 (osd.1) 410 : cluster 0 323.6 deep-scrub ok 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.383019+0000 osd.0 (osd.0) 405 : cluster 0 6.18 deep-scrub starts 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.384493+0000 osd.0 (osd.0) 406 : cluster 0 6.18 deep-scrub ok 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: audit 2024-04-02T05:02:25.592619+0000 mon.b (mon.1) 313 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: audit 2024-04-02T05:02:26.268441+0000 mon.b (mon.1) 314 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: audit 2024-04-02T05:02:26.270058+0000 mon.b (mon.1) 315 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:02:26.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:26 smithi098 bash[23790]: audit 2024-04-02T05:02:26.283475+0000 mon.a (mon.0) 3849 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:26.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:26.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:26.660 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:26.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: cluster 2024-04-02T05:02:24.972114+0000 osd.1 (osd.1) 409 : cluster 0 323.6s0 deep-scrub starts 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: cluster 2024-04-02T05:02:24.974690+0000 osd.1 (osd.1) 410 : cluster 0 323.6 deep-scrub ok 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.383019+0000 osd.0 (osd.0) 405 : cluster 0 6.18 deep-scrub starts 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.384493+0000 osd.0 (osd.0) 406 : cluster 0 6.18 deep-scrub ok 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: audit 2024-04-02T05:02:25.592619+0000 mon.b (mon.1) 313 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: audit 2024-04-02T05:02:26.268441+0000 mon.b (mon.1) 314 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: audit 2024-04-02T05:02:26.270058+0000 mon.b (mon.1) 315 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[21528]: audit 2024-04-02T05:02:26.283475+0000 mon.a (mon.0) 3849 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: cluster 2024-04-02T05:02:24.972114+0000 osd.1 (osd.1) 409 : cluster 0 323.6s0 deep-scrub starts 2024-04-02T05:02:26.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: cluster 2024-04-02T05:02:24.974690+0000 osd.1 (osd.1) 410 : cluster 0 323.6 deep-scrub ok 2024-04-02T05:02:26.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.383019+0000 osd.0 (osd.0) 405 : cluster 0 6.18 deep-scrub starts 2024-04-02T05:02:26.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.384493+0000 osd.0 (osd.0) 406 : cluster 0 6.18 deep-scrub ok 2024-04-02T05:02:26.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: audit 2024-04-02T05:02:25.592619+0000 mon.b (mon.1) 313 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:02:26.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: audit 2024-04-02T05:02:26.268441+0000 mon.b (mon.1) 314 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:02:26.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: audit 2024-04-02T05:02:26.270058+0000 mon.b (mon.1) 315 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:02:26.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:26 smithi069 bash[28904]: audit 2024-04-02T05:02:26.283475+0000 mon.a (mon.0) 3849 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:26.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:26.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:26.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:26.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:27.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:27.049 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:27.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:27.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:27.307 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:27.411 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:27.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:27.551 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.775036+0000 osd.3 (osd.3) 421 : cluster 0 323.3s0 deep-scrub starts 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.777470+0000 osd.3 (osd.3) 422 : cluster 0 323.3 deep-scrub ok 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.854807+0000 osd.6 (osd.6) 355 : cluster 0 10.7 scrub starts 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.859224+0000 osd.6 (osd.6) 356 : cluster 0 10.7 scrub ok 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.941858+0000 osd.1 (osd.1) 411 : cluster 0 10.14 scrub starts 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:25.942827+0000 osd.1 (osd.1) 412 : cluster 0 10.14 scrub ok 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:26.300757+0000 osd.7 (osd.7) 217 : cluster 0 2.0 scrub starts 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:26.301963+0000 osd.7 (osd.7) 218 : cluster 0 2.0 scrub ok 2024-04-02T05:02:27.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:27 smithi098 bash[23790]: cluster 2024-04-02T05:02:26.336921+0000 mgr.y (mgr.14538) 725 : cluster 0 pgmap v1263: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:27.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:27.653 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T05:02:27.653 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T05:02:27.653 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-02T05:02:27.653 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.692343 2024-04-02T05:02:27.654 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 0 0.6923426559459699 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.775036+0000 osd.3 (osd.3) 421 : cluster 0 323.3s0 deep-scrub starts 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.777470+0000 osd.3 (osd.3) 422 : cluster 0 323.3 deep-scrub ok 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.854807+0000 osd.6 (osd.6) 355 : cluster 0 10.7 scrub starts 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.859224+0000 osd.6 (osd.6) 356 : cluster 0 10.7 scrub ok 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.941858+0000 osd.1 (osd.1) 411 : cluster 0 10.14 scrub starts 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:25.942827+0000 osd.1 (osd.1) 412 : cluster 0 10.14 scrub ok 2024-04-02T05:02:27.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:26.300757+0000 osd.7 (osd.7) 217 : cluster 0 2.0 scrub starts 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:26.301963+0000 osd.7 (osd.7) 218 : cluster 0 2.0 scrub ok 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[21528]: cluster 2024-04-02T05:02:26.336921+0000 mgr.y (mgr.14538) 725 : cluster 0 pgmap v1263: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.775036+0000 osd.3 (osd.3) 421 : cluster 0 323.3s0 deep-scrub starts 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.777470+0000 osd.3 (osd.3) 422 : cluster 0 323.3 deep-scrub ok 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.854807+0000 osd.6 (osd.6) 355 : cluster 0 10.7 scrub starts 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.859224+0000 osd.6 (osd.6) 356 : cluster 0 10.7 scrub ok 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.941858+0000 osd.1 (osd.1) 411 : cluster 0 10.14 scrub starts 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:25.942827+0000 osd.1 (osd.1) 412 : cluster 0 10.14 scrub ok 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:26.300757+0000 osd.7 (osd.7) 217 : cluster 0 2.0 scrub starts 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:26.301963+0000 osd.7 (osd.7) 218 : cluster 0 2.0 scrub ok 2024-04-02T05:02:27.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:27 smithi069 bash[28904]: cluster 2024-04-02T05:02:26.336921+0000 mgr.y (mgr.14538) 725 : cluster 0 pgmap v1263: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:27.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:27.693 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:27.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-02T05:02:27.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:27.821 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:27.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:27.943 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:27.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:28.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:28.074 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:28.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:28.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:28.333 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:28.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:28.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[21528]: cluster 2024-04-02T05:02:27.198588+0000 osd.4 (osd.4) 412 : cluster 0 6.1a scrub starts 2024-04-02T05:02:28.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[21528]: cluster 2024-04-02T05:02:27.200125+0000 osd.4 (osd.4) 413 : cluster 0 6.1a scrub ok 2024-04-02T05:02:28.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[21528]: cluster 2024-04-02T05:02:27.266034+0000 osd.7 (osd.7) 219 : cluster 0 6.12 scrub starts 2024-04-02T05:02:28.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[21528]: cluster 2024-04-02T05:02:27.268304+0000 osd.7 (osd.7) 220 : cluster 0 6.12 scrub ok 2024-04-02T05:02:28.448 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[28904]: cluster 2024-04-02T05:02:27.198588+0000 osd.4 (osd.4) 412 : cluster 0 6.1a scrub starts 2024-04-02T05:02:28.448 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[28904]: cluster 2024-04-02T05:02:27.200125+0000 osd.4 (osd.4) 413 : cluster 0 6.1a scrub ok 2024-04-02T05:02:28.448 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[28904]: cluster 2024-04-02T05:02:27.266034+0000 osd.7 (osd.7) 219 : cluster 0 6.12 scrub starts 2024-04-02T05:02:28.448 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[28904]: cluster 2024-04-02T05:02:27.268304+0000 osd.7 (osd.7) 220 : cluster 0 6.12 scrub ok 2024-04-02T05:02:28.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:28.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:28.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:28.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:28 smithi098 bash[23790]: cluster 2024-04-02T05:02:27.198588+0000 osd.4 (osd.4) 412 : cluster 0 6.1a scrub starts 2024-04-02T05:02:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:28 smithi098 bash[23790]: cluster 2024-04-02T05:02:27.200125+0000 osd.4 (osd.4) 413 : cluster 0 6.1a scrub ok 2024-04-02T05:02:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:28 smithi098 bash[23790]: cluster 2024-04-02T05:02:27.266034+0000 osd.7 (osd.7) 219 : cluster 0 6.12 scrub starts 2024-04-02T05:02:28.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:28 smithi098 bash[23790]: cluster 2024-04-02T05:02:27.268304+0000 osd.7 (osd.7) 220 : cluster 0 6.12 scrub ok 2024-04-02T05:02:28.725 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:28.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:28.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:28.986 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:28.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:29.032 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:02:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:02:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:02:29.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:29.115 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:29.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:29.349 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:29.376 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:29.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:29.501 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:29.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:29.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:29 smithi098 bash[23790]: cluster 2024-04-02T05:02:27.893369+0000 osd.6 (osd.6) 357 : cluster 0 2.16 deep-scrub starts 2024-04-02T05:02:29.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:29 smithi098 bash[23790]: cluster 2024-04-02T05:02:27.895588+0000 osd.6 (osd.6) 358 : cluster 0 2.16 deep-scrub ok 2024-04-02T05:02:29.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:29 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.338396+0000 mgr.y (mgr.14538) 726 : cluster 0 pgmap v1264: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 2 op/s 2024-04-02T05:02:29.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:29 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.393796+0000 osd.0 (osd.0) 407 : cluster 0 4.16 scrub starts 2024-04-02T05:02:29.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:29 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.395507+0000 osd.0 (osd.0) 408 : cluster 0 4.16 scrub ok 2024-04-02T05:02:29.630 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2565598813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102520 msgr2=0x7f227c102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- 172.21.15.69:0/2565598813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102520 0x7f227c102900 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7f22740099c0 tx=0x7f2274009cd0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2565598813 shutdown_connections 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- 172.21.15.69:0/2565598813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 0x7f227c10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- 172.21.15.69:0/2565598813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102520 0x7f227c102900 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- 172.21.15.69:0/2565598813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c102ed0 0x7f227c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2565598813 >> 172.21.15.69:0/2565598813 conn(0x7f227c079360 msgr2=0x7f227c079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2565598813 shutdown_connections 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2565598813 wait complete. 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 librados: starting msgr at 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 librados: starting objecter 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 Processor -- start 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- start start 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 librados: setting wanted keys 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 librados: calling monclient init 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c102520 0x7f227c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 0x7f227c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 0x7f227c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f227c111a70 con 0x7f227c102ed0 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1118f0 con 0x7f227c102520 2024-04-02T05:02:29.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f227c111bf0 con 0x7f227c1075a0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f227b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 0x7f227c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 0x7f227c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f227b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 0x7f227c074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52510/0 (socket says 172.21.15.69:52510) 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f227b7fe700 1 -- 172.21.15.69:0/2816744428 learned_addr learned my addr 172.21.15.69:0/2816744428 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f227bfff700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c102520 0x7f227c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 -- 172.21.15.69:0/2816744428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c102520 msgr2=0x7f227c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c102520 0x7f227c074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 -- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 msgr2=0x7f227c074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 0x7f227c074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 -- 172.21.15.69:0/2816744428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c1a65b0 con 0x7f227c1075a0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f227b7fe700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 0x7f227c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f2280f56700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 0x7f227c074fc0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f226c00efc0 tx=0x7f226c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22797fa700 10 client.?.objecter ms_handle_connect 0x7f227c1075a0 2024-04-02T05:02:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22797fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0092e0 con 0x7f227c1075a0 2024-04-02T05:02:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f226c004bb0 con 0x7f227c1075a0 2024-04-02T05:02:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c1a6840 con 0x7f227c1075a0 2024-04-02T05:02:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 10 client.43817.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c1a6d80 con 0x7f227c1075a0 2024-04-02T05:02:29.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.672+0000 7f22829b9700 1 librados: init done 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c004020 con 0x7f227c1075a0 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227c1039a0 con 0x7f227c1075a0 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f226c01b030 con 0x7f227c1075a0 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f226406f4b0 0x7f2264071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(833..833 src has 74..833) ==== 10133+0+0 (secure 0 0 0) 0x7f226c08e5a0 con 0x7f227c1075a0 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 10 client.43817.objecter ms_dispatch 0x7f227c001550 osd_map(833..833 src has 74..833) 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 3 client.43817.objecter handle_osd_map got epochs [833,833] > 0 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 3 client.43817.objecter handle_osd_map decoding full epoch 833 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f227bfff700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f226406f4b0 0x7f2264071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 20 client.43817.objecter dump_active .. 0 homeless 2024-04-02T05:02:29.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f22797fa700 10 client.43817.objecter ms_handle_connect 0x7f226406f4b0 2024-04-02T05:02:29.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.676+0000 7f227bfff700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f226406f4b0 0x7f2264071970 secure :-1 s=READY pgs=4016 cs=0 l=1 rev1=1 crypto rx=0x7f22740059d0 tx=0x7f2274005940 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.680+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f226c022020 con 0x7f227c1075a0 2024-04-02T05:02:29.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- 172.21.15.69:0/2292429137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06b4b0 msgr2=0x7f73bc106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- 172.21.15.69:0/2292429137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06b4b0 0x7f73bc106fa0 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0098f0 tx=0x7f73ac009c00 comp rx=0 tx=0).stop 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- 172.21.15.69:0/2292429137 shutdown_connections 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- 172.21.15.69:0/2292429137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73bc107680 0x7f73bc10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- 172.21.15.69:0/2292429137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06ab00 0x7f73bc06aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- 172.21.15.69:0/2292429137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06b4b0 0x7f73bc106fa0 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- 172.21.15.69:0/2292429137 >> 172.21.15.69:0/2292429137 conn(0x7f73bc078240 msgr2=0x7f73bc078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- 172.21.15.69:0/2292429137 shutdown_connections 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- 172.21.15.69:0/2292429137 wait complete. 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 librados: starting msgr at 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 librados: starting objecter 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 Processor -- start 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- start start 2024-04-02T05:02:29.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 librados: setting wanted keys 2024-04-02T05:02:29.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 librados: calling monclient init 2024-04-02T05:02:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06ab00 0x7f73bc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 0x7f73bc19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73bc107680 0x7f73bc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc111ac0 con 0x7f73bc06b4b0 2024-04-02T05:02:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc111940 con 0x7f73bc107680 2024-04-02T05:02:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.692+0000 7f73c1169700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc111c40 con 0x7f73bc06ab00 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 0x7f73bc19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73bad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06ab00 0x7f73bc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 0x7f73bc19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52530/0 (socket says 172.21.15.69:52530) 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 -- 172.21.15.69:0/822335303 learned_addr learned my addr 172.21.15.69:0/822335303 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73bc107680 0x7f73bc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06ab00 msgr2=0x7f73bc19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73bc06ab00 0x7f73bc19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73bc107680 msgr2=0x7f73bc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73bc107680 0x7f73bc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc19f2a0 con 0x7f73bc06b4b0 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73bc107680 0x7f73bc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 0x7f73bc19a820 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009230 tx=0x7f73ac00fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 10 client.?.objecter ms_handle_connect 0x7f73bc06b4b0 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00b6e0 con 0x7f73bc06b4b0 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f73ac005040 con 0x7f73bc06b4b0 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac003990 con 0x7f73bc06b4b0 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc19f530 con 0x7f73bc06b4b0 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73c1169700 10 client.33759.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc1a6e10 con 0x7f73bc06b4b0 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73c1169700 1 librados: init done 2024-04-02T05:02:29.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f73bc04ff00 con 0x7f73bc06b4b0 2024-04-02T05:02:29.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f73ac00b880 con 0x7f73bc06b4b0 2024-04-02T05:02:29.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f73a406f5e0 0x7f73a4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(833..833 src has 74..833) ==== 10133+0+0 (secure 0 0 0) 0x7f73ac0901a0 con 0x7f73bc06b4b0 2024-04-02T05:02:29.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 osd_map(833..833 src has 74..833) 2024-04-02T05:02:29.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 3 client.33759.objecter handle_osd_map got epochs [833,833] > 0 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73a3fff700 3 client.33759.objecter handle_osd_map decoding full epoch 833 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.696+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f73a406f5e0 0x7f73a4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 20 client.33759.objecter dump_active .. 0 homeless 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v833) ==== 54+0+16 (secure 0 0 0) 0x7f73ac031020 con 0x7f73bc06b4b0 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f73a406f5e0 0x7f73a4071aa0 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009750 tx=0x7f73b0009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 10 client.33759.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 1 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73a406f5e0 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f73bc062e50 0x7f73bc063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 1 2024-04-02T05:02:29.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 0 1 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 4 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 10 client.33759.objecter _send_command 1 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73bc1aa170 con 0x7f73bc062e50 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f73bc062e50 0x7f73bc063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc062e50 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f73bc062e50 0x7f73bc063230 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc1aa170 con 0x7f73bc062e50 2024-04-02T05:02:29.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 1: 0 ) 2024-04-02T05:02:29.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 10 client.33759.objecter _finish_command 1 = system:0 2024-04-02T05:02:29.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 4 2024-04-02T05:02:29.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.700+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 0 1 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 10 client.33759.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 2 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 2 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc19b5f0 osd=0 3 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 0 2 2024-04-02T05:02:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 4 2024-04-02T05:02:29.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 10 client.33759.objecter _send_command 2 2024-04-02T05:02:29.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.716+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc0628d0 con 0x7f73bc062e50 2024-04-02T05:02:29.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc0628d0 con 0x7f73bc062e50 2024-04-02T05:02:29.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 10 client.33759.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 4 2024-04-02T05:02:29.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 0 2 2024-04-02T05:02:29.725 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 10 client.33759.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f73bc02c9b0 0x7f73bc02cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 1 3 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 4 2024-04-02T05:02:29.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 10 client.33759.objecter _send_command 3 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f73bc0ffde0 con 0x7f73bc02c9b0 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f73bc02c9b0 0x7f73bc02cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc02c9b0 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f73bc02c9b0 0x7f73bc02cdb0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc0ffde0 con 0x7f73bc02c9b0 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 3: 0 ) 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 10 client.33759.objecter _finish_command 3 = system:0 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 4 2024-04-02T05:02:29.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.720+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 1 3 2024-04-02T05:02:29.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:29.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[21528]: cluster 2024-04-02T05:02:27.893369+0000 osd.6 (osd.6) 357 : cluster 0 2.16 deep-scrub starts 2024-04-02T05:02:29.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[21528]: cluster 2024-04-02T05:02:27.895588+0000 osd.6 (osd.6) 358 : cluster 0 2.16 deep-scrub ok 2024-04-02T05:02:29.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.338396+0000 mgr.y (mgr.14538) 726 : cluster 0 pgmap v1264: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 2 op/s 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.393796+0000 osd.0 (osd.0) 407 : cluster 0 4.16 scrub starts 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.395507+0000 osd.0 (osd.0) 408 : cluster 0 4.16 scrub ok 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[28904]: cluster 2024-04-02T05:02:27.893369+0000 osd.6 (osd.6) 357 : cluster 0 2.16 deep-scrub starts 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[28904]: cluster 2024-04-02T05:02:27.895588+0000 osd.6 (osd.6) 358 : cluster 0 2.16 deep-scrub ok 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.338396+0000 mgr.y (mgr.14538) 726 : cluster 0 pgmap v1264: 224 pgs: 224 active+clean; 4.4 MiB data, 755 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 2 op/s 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.393796+0000 osd.0 (osd.0) 407 : cluster 0 4.16 scrub starts 2024-04-02T05:02:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:29 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.395507+0000 osd.0 (osd.0) 408 : cluster 0 4.16 scrub ok 2024-04-02T05:02:29.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 10 client.33759.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 4 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 4 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc0ff760 osd=1 3 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 1 4 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 4 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 10 client.33759.objecter _send_command 4 2024-04-02T05:02:29.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc0ff1e0 con 0x7f73bc02c9b0 2024-04-02T05:02:29.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc0ff1e0 con 0x7f73bc02c9b0 2024-04-02T05:02:29.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73a3fff700 10 client.33759.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 4 2024-04-02T05:02:29.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 1 4 2024-04-02T05:02:29.747 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:02:29.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 10 client.33759.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 5 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f73bc1ac330 0x7f73bc1ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 5 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 2 5 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 4 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 10 client.33759.objecter _send_command 5 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f73bc1b1580 con 0x7f73bc1ac330 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.740+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f73bc1ac330 0x7f73bc1ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc1ac330 2024-04-02T05:02:29.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f73bc1ac330 0x7f73bc1ae720 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc1b1580 con 0x7f73bc1ac330 2024-04-02T05:02:29.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 5: 0 ) 2024-04-02T05:02:29.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73a3fff700 10 client.33759.objecter _finish_command 5 = system:0 2024-04-02T05:02:29.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 4 2024-04-02T05:02:29.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.744+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 2 5 2024-04-02T05:02:29.752 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 10 client.33759.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 6 2024-04-02T05:02:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 6 2024-04-02T05:02:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc0ff5e0 osd=2 3 2024-04-02T05:02:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 2 6 2024-04-02T05:02:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 4 2024-04-02T05:02:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 10 client.33759.objecter _send_command 6 2024-04-02T05:02:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.760+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc02d540 con 0x7f73bc1ac330 2024-04-02T05:02:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc02d540 con 0x7f73bc1ac330 2024-04-02T05:02:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 10 client.33759.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 4 2024-04-02T05:02:29.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 2 6 2024-04-02T05:02:29.770 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:02:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 10 client.33759.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 7 2024-04-02T05:02:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f73bc1a9cd0 0x7f73bc1b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 7 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 3 7 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 4 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 10 client.33759.objecter _send_command 7 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f73bc02d880 con 0x7f73bc1a9cd0 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f73bc1a9cd0 0x7f73bc1b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc1a9cd0 2024-04-02T05:02:29.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f73bc1a9cd0 0x7f73bc1b1df0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc02d880 con 0x7f73bc1a9cd0 2024-04-02T05:02:29.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 7: 0 ) 2024-04-02T05:02:29.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 10 client.33759.objecter _finish_command 7 = system:0 2024-04-02T05:02:29.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 4 2024-04-02T05:02:29.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.764+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 3 7 2024-04-02T05:02:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 10 client.33759.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 8 2024-04-02T05:02:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 8 2024-04-02T05:02:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc02dfd0 osd=3 3 2024-04-02T05:02:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 3 8 2024-04-02T05:02:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 4 2024-04-02T05:02:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 10 client.33759.objecter _send_command 8 2024-04-02T05:02:29.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc1a9240 con 0x7f73bc1a9cd0 2024-04-02T05:02:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc1a9240 con 0x7f73bc1a9cd0 2024-04-02T05:02:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73a3fff700 10 client.33759.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 4 2024-04-02T05:02:29.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 3 8 2024-04-02T05:02:29.790 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 10 client.33759.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 9 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f73bc1a93b0 0x7f73bc1b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 9 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 4 9 2024-04-02T05:02:29.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 4 2024-04-02T05:02:29.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 10 client.33759.objecter _send_command 9 2024-04-02T05:02:29.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f73bc1afb10 con 0x7f73bc1a93b0 2024-04-02T05:02:29.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.784+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f73bc1a93b0 0x7f73bc1b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc1a93b0 2024-04-02T05:02:29.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f73bc1a93b0 0x7f73bc1b54c0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc1afb10 con 0x7f73bc1a93b0 2024-04-02T05:02:29.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 9: 0 ) 2024-04-02T05:02:29.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73a3fff700 10 client.33759.objecter _finish_command 9 = system:0 2024-04-02T05:02:29.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 4 2024-04-02T05:02:29.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.788+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 4 9 2024-04-02T05:02:29.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 10 client.33759.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 10 2024-04-02T05:02:29.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 10 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1a9240 osd=4 3 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 4 10 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 4 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 10 client.33759.objecter _send_command 10 2024-04-02T05:02:29.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.804+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc1b5a00 con 0x7f73bc1a93b0 2024-04-02T05:02:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc1b5a00 con 0x7f73bc1a93b0 2024-04-02T05:02:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73a3fff700 10 client.33759.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 4 2024-04-02T05:02:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 4 10 2024-04-02T05:02:29.816 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:02:29.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 10 client.33759.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 11 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f73bc1b6080 0x7f73bc1b8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 11 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 5 11 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 4 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 10 client.33759.objecter _send_command 11 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.808+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f73bc1bb3e0 con 0x7f73bc1b6080 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f73bc1b6080 0x7f73bc1b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc1b6080 2024-04-02T05:02:29.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73ba59c700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f73bc1b6080 0x7f73bc1b8470 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc1bb3e0 con 0x7f73bc1b6080 2024-04-02T05:02:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 11: 0 ) 2024-04-02T05:02:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73a3fff700 10 client.33759.objecter _finish_command 11 = system:0 2024-04-02T05:02:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 4 2024-04-02T05:02:29.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.812+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 5 11 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 10 client.33759.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 12 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 12 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1b5a00 osd=5 3 2024-04-02T05:02:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 5 12 2024-04-02T05:02:29.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 4 2024-04-02T05:02:29.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 10 client.33759.objecter _send_command 12 2024-04-02T05:02:29.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.828+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc1ba1d0 con 0x7f73bc1b6080 2024-04-02T05:02:29.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.832+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc1ba1d0 con 0x7f73bc1b6080 2024-04-02T05:02:29.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.832+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.832+0000 7f73a3fff700 10 client.33759.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.832+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 4 2024-04-02T05:02:29.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.832+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 5 12 2024-04-02T05:02:29.840 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 10 client.33759.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 13 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f73bc1ba8f0 0x7f73bc1bad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 13 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 6 13 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 4 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 10 client.33759.objecter _send_command 13 2024-04-02T05:02:29.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f73bc1be840 con 0x7f73bc1ba8f0 2024-04-02T05:02:29.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f73bc1ba8f0 0x7f73bc1bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc1ba8f0 2024-04-02T05:02:29.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73bb59e700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f73bc1ba8f0 0x7f73bc1bad30 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc1be840 con 0x7f73bc1ba8f0 2024-04-02T05:02:29.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 13: 0 ) 2024-04-02T05:02:29.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73a3fff700 10 client.33759.objecter _finish_command 13 = system:0 2024-04-02T05:02:29.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 4 2024-04-02T05:02:29.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.836+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 6 13 2024-04-02T05:02:29.855 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 10 client.33759.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 14 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 14 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1ba1d0 osd=6 3 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 6 14 2024-04-02T05:02:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 4 2024-04-02T05:02:29.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 10 client.33759.objecter _send_command 14 2024-04-02T05:02:29.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.856+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc1be840 con 0x7f73bc1ba8f0 2024-04-02T05:02:29.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc1be840 con 0x7f73bc1ba8f0 2024-04-02T05:02:29.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73a3fff700 10 client.33759.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 4 2024-04-02T05:02:29.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 6 14 2024-04-02T05:02:29.864 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:02:29.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 10 client.33759.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:29.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 15 2024-04-02T05:02:29.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f73bc1bdb40 0x7f73bc1bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:29.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 15 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 7 15 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 4 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 10 client.33759.objecter _send_command 15 2024-04-02T05:02:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f73bc1c1c30 con 0x7f73bc1bdb40 2024-04-02T05:02:29.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f73bc1bdb40 0x7f73bc1bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:29.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73bad9d700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f73bc1bdb40 0x7f73bc1bdfa0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:29.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.860+0000 7f73a3fff700 10 client.33759.objecter ms_handle_connect 0x7f73bc1bdb40 2024-04-02T05:02:29.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.864+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73bc1c1c30 con 0x7f73bc1bdb40 2024-04-02T05:02:29.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.864+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 15: 0 ) 2024-04-02T05:02:29.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.864+0000 7f73a3fff700 10 client.33759.objecter _finish_command 15 = system:0 2024-04-02T05:02:29.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.864+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 4 2024-04-02T05:02:29.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.864+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 7 15 2024-04-02T05:02:29.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:29.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 10 client.33759.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign -1 16 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 20 client.33759.objecter _get_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 15 client.33759.objecter _session_command_op_remove -1 16 2024-04-02T05:02:29.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 20 client.33759.objecter get_session s=0x7f73bc1be840 osd=7 3 2024-04-02T05:02:29.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 15 client.33759.objecter _session_command_op_assign 7 16 2024-04-02T05:02:29.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 4 2024-04-02T05:02:29.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 10 client.33759.objecter _send_command 16 2024-04-02T05:02:29.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.884+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f73bc1b8a00 con 0x7f73bc1bdb40 2024-04-02T05:02:29.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73a3fff700 1 -- 172.21.15.69:0/822335303 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f73bc1b8a00 con 0x7f73bc1bdb40 2024-04-02T05:02:29.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73a3fff700 10 client.33759.objecter ms_dispatch 0x7f73bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:29.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73a3fff700 10 client.33759.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:29.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73a3fff700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 4 2024-04-02T05:02:29.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73a3fff700 15 client.33759.objecter _session_command_op_remove 7 16 2024-04-02T05:02:29.894 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:02:29.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 librados: watch_flush enter 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 librados: watch_flush exit 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.0 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f73bc062e50 msgr2=0x7f73bc063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f73bc062e50 0x7f73bc063230 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc19b5f0 osd=0 1 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.1 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f73bc02c9b0 msgr2=0x7f73bc02cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f73bc02c9b0 0x7f73bc02cdb0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff760 osd=1 1 2024-04-02T05:02:29.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.2 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f73bc1ac330 msgr2=0x7f73bc1ae720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f73bc1ac330 0x7f73bc1ae720 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc0ff5e0 osd=2 1 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.3 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f73bc1a9cd0 msgr2=0x7f73bc1b1df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f73bc1a9cd0 0x7f73bc1b1df0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc02dfd0 osd=3 1 2024-04-02T05:02:29.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.4 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f73bc1a93b0 msgr2=0x7f73bc1b54c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f73bc1a93b0 0x7f73bc1b54c0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1a9240 osd=4 1 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.5 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f73bc1b6080 msgr2=0x7f73bc1b8470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f73bc1b6080 0x7f73bc1b8470 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1b5a00 osd=5 1 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.6 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f73bc1ba8f0 msgr2=0x7f73bc1bad30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f73bc1ba8f0 0x7f73bc1bad30 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1ba1d0 osd=6 1 2024-04-02T05:02:29.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter close_session for osd.7 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f73bc1bdb40 msgr2=0x7f73bc1bdfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f73bc1bdb40 0x7f73bc1bdfa0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter put_session s=0x7f73bc1be840 osd=7 1 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 20 client.33759.objecter shutdown clearing up homeless session... 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 10 client.33759.objecter successfully canceled tick 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73bb59e700 1 -- 172.21.15.69:0/822335303 reap_dead start 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73bb59e700 1 -- 172.21.15.69:0/822335303 reap_dead start 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f73a406f5e0 msgr2=0x7f73a4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73bb59e700 1 -- 172.21.15.69:0/822335303 reap_dead start 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f73a406f5e0 0x7f73a4071aa0 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009750 tx=0x7f73b0009310 comp rx=0 tx=0).stop 2024-04-02T05:02:29.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73bb59e700 1 -- 172.21.15.69:0/822335303 reap_dead start 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 msgr2=0x7f73bc19a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 0x7f73bc19a820 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009230 tx=0x7f73ac00fa70 comp rx=0 tx=0).stop 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73bb59e700 1 -- 172.21.15.69:0/822335303 reap_dead start 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 shutdown_connections 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 --2- 172.21.15.69:0/822335303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73bc06b4b0 0x7f73bc19a820 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 >> 172.21.15.69:0/822335303 conn(0x7f73bc078240 msgr2=0x7f73bc10d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 shutdown_connections 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 -- 172.21.15.69:0/822335303 wait complete. 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.888+0000 7f73c1169700 1 librados: shutdown 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:02:29.900 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:02:29.901 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:02:29.901 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:02:29.901 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:02:29.901 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:02:29.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:29.916+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f227c075a70 con 0x7f227c1075a0 2024-04-02T05:02:29.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:30.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:30.133 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:30.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:30.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:30.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.352+0000 7f22797fa700 1 -- 172.21.15.69:0/2816744428 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v834) ==== 87+0+0 (secure 0 0 0) 0x7f226c010040 con 0x7f227c1075a0 2024-04-02T05:02:30.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:30.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 10 librados: watch_flush enter 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 10 librados: watch_flush exit 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 20 client.43817.objecter shutdown clearing up homeless session... 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 10 client.43817.objecter successfully canceled tick 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f226406f4b0 msgr2=0x7f2264071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f226406f4b0 0x7f2264071970 secure :-1 s=READY pgs=4016 cs=0 l=1 rev1=1 crypto rx=0x7f22740059d0 tx=0x7f2274005940 comp rx=0 tx=0).stop 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 msgr2=0x7f227c074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 0x7f227c074fc0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f226c00efc0 tx=0x7f226c00c3a0 comp rx=0 tx=0).stop 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 shutdown_connections 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f226406f4b0 0x7f2264071970 unknown :-1 s=CLOSED pgs=4016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f227c1075a0 0x7f227c074fc0 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c102520 0x7f227c074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 --2- 172.21.15.69:0/2816744428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f227c102ed0 0x7f227c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:30.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 >> 172.21.15.69:0/2816744428 conn(0x7f227c079360 msgr2=0x7f227c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 shutdown_connections 2024-04-02T05:02:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 -- 172.21.15.69:0/2816744428 wait complete. 2024-04-02T05:02:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:30.356+0000 7f22829b9700 1 librados: shutdown 2024-04-02T05:02:30.362 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T05:02:30.384 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:30.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:30.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.758449+0000 osd.3 (osd.3) 423 : cluster 0 6.1f scrub starts 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.760107+0000 osd.3 (osd.3) 424 : cluster 0 6.1f scrub ok 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.930430+0000 osd.1 (osd.1) 413 : cluster 0 323.2s0 scrub starts 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: cluster 2024-04-02T05:02:28.932921+0000 osd.1 (osd.1) 414 : cluster 0 323.2 scrub ok 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: audit 2024-04-02T05:02:29.702455+0000 mon.a (mon.0) 3850 : audit 0 from='client.? 172.21.15.69:0/822335303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: audit 2024-04-02T05:02:29.923801+0000 mon.c (mon.2) 1162 : audit 1 from='client.? 172.21.15.69:0/2816744428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:02:30.626 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:30 smithi098 bash[23790]: audit 2024-04-02T05:02:29.924133+0000 mon.a (mon.0) 3851 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:02:30.641 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:30.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:30.773 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:30.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:30.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.758449+0000 osd.3 (osd.3) 423 : cluster 0 6.1f scrub starts 2024-04-02T05:02:30.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.760107+0000 osd.3 (osd.3) 424 : cluster 0 6.1f scrub ok 2024-04-02T05:02:30.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.930430+0000 osd.1 (osd.1) 413 : cluster 0 323.2s0 scrub starts 2024-04-02T05:02:30.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: cluster 2024-04-02T05:02:28.932921+0000 osd.1 (osd.1) 414 : cluster 0 323.2 scrub ok 2024-04-02T05:02:30.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: audit 2024-04-02T05:02:29.702455+0000 mon.a (mon.0) 3850 : audit 0 from='client.? 172.21.15.69:0/822335303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: audit 2024-04-02T05:02:29.923801+0000 mon.c (mon.2) 1162 : audit 1 from='client.? 172.21.15.69:0/2816744428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[21528]: audit 2024-04-02T05:02:29.924133+0000 mon.a (mon.0) 3851 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.758449+0000 osd.3 (osd.3) 423 : cluster 0 6.1f scrub starts 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.760107+0000 osd.3 (osd.3) 424 : cluster 0 6.1f scrub ok 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.930430+0000 osd.1 (osd.1) 413 : cluster 0 323.2s0 scrub starts 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: cluster 2024-04-02T05:02:28.932921+0000 osd.1 (osd.1) 414 : cluster 0 323.2 scrub ok 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: audit 2024-04-02T05:02:29.702455+0000 mon.a (mon.0) 3850 : audit 0 from='client.? 172.21.15.69:0/822335303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: audit 2024-04-02T05:02:29.923801+0000 mon.c (mon.2) 1162 : audit 1 from='client.? 172.21.15.69:0/2816744428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:02:30.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:30 smithi069 bash[28904]: audit 2024-04-02T05:02:29.924133+0000 mon.a (mon.0) 3851 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:02:30.902 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:30.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:31.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:31.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:31.159 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:31.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:31.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[21528]: cluster 2024-04-02T05:02:30.339050+0000 mgr.y (mgr.14538) 727 : cluster 0 pgmap v1265: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 726 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[21528]: cluster 2024-04-02T05:02:30.346490+0000 mon.a (mon.0) 3852 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[21528]: audit 2024-04-02T05:02:30.348536+0000 mon.a (mon.0) 3853 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[21528]: cluster 2024-04-02T05:02:30.351919+0000 mon.a (mon.0) 3854 : cluster 0 osdmap e834: 8 total, 8 up, 7 in 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[21528]: cluster 2024-04-02T05:02:30.741972+0000 osd.2 (osd.2) 361 : cluster 0 2.10 scrub starts 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[28904]: cluster 2024-04-02T05:02:30.339050+0000 mgr.y (mgr.14538) 727 : cluster 0 pgmap v1265: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 726 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[28904]: cluster 2024-04-02T05:02:30.346490+0000 mon.a (mon.0) 3852 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:02:31.405 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[28904]: audit 2024-04-02T05:02:30.348536+0000 mon.a (mon.0) 3853 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:02:31.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[28904]: cluster 2024-04-02T05:02:30.351919+0000 mon.a (mon.0) 3854 : cluster 0 osdmap e834: 8 total, 8 up, 7 in 2024-04-02T05:02:31.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:31 smithi069 bash[28904]: cluster 2024-04-02T05:02:30.741972+0000 osd.2 (osd.2) 361 : cluster 0 2.10 scrub starts 2024-04-02T05:02:31.420 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:31.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:31.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:31 smithi098 bash[23790]: cluster 2024-04-02T05:02:30.339050+0000 mgr.y (mgr.14538) 727 : cluster 0 pgmap v1265: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 726 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-04-02T05:02:31.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:31 smithi098 bash[23790]: cluster 2024-04-02T05:02:30.346490+0000 mon.a (mon.0) 3852 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:02:31.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:31 smithi098 bash[23790]: audit 2024-04-02T05:02:30.348536+0000 mon.a (mon.0) 3853 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:02:31.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:31 smithi098 bash[23790]: cluster 2024-04-02T05:02:30.351919+0000 mon.a (mon.0) 3854 : cluster 0 osdmap e834: 8 total, 8 up, 7 in 2024-04-02T05:02:31.535 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:31 smithi098 bash[23790]: cluster 2024-04-02T05:02:30.741972+0000 osd.2 (osd.2) 361 : cluster 0 2.10 scrub starts 2024-04-02T05:02:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:31.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:31.681 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:31.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:31.811 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:31.914 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:31.939 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:31.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:32.042 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:32.070 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:32.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:32.198 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:32.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:32.330 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:32.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:32.460 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:32.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:32.574 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:32 smithi098 bash[23790]: cluster 2024-04-02T05:02:30.914987+0000 osd.6 (osd.6) 359 : cluster 0 4.1f scrub starts 2024-04-02T05:02:32.575 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:32 smithi098 bash[23790]: cluster 2024-04-02T05:02:30.918018+0000 osd.6 (osd.6) 360 : cluster 0 4.1f scrub ok 2024-04-02T05:02:32.589 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:32.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:32.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:32 smithi069 bash[21528]: cluster 2024-04-02T05:02:30.914987+0000 osd.6 (osd.6) 359 : cluster 0 4.1f scrub starts 2024-04-02T05:02:32.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:32 smithi069 bash[21528]: cluster 2024-04-02T05:02:30.918018+0000 osd.6 (osd.6) 360 : cluster 0 4.1f scrub ok 2024-04-02T05:02:32.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:32 smithi069 bash[28904]: cluster 2024-04-02T05:02:30.914987+0000 osd.6 (osd.6) 359 : cluster 0 4.1f scrub starts 2024-04-02T05:02:32.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:32 smithi069 bash[28904]: cluster 2024-04-02T05:02:30.918018+0000 osd.6 (osd.6) 360 : cluster 0 4.1f scrub ok 2024-04-02T05:02:32.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:32.849 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:32.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:32.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:33.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:33.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:33.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T05:02:33.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:33.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:33.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:33.485 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:33 smithi098 bash[23790]: cluster 2024-04-02T05:02:32.339576+0000 mgr.y (mgr.14538) 728 : cluster 0 pgmap v1267: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-04-02T05:02:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:33.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:33 smithi069 bash[21528]: cluster 2024-04-02T05:02:32.339576+0000 mgr.y (mgr.14538) 728 : cluster 0 pgmap v1267: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-04-02T05:02:33.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:33 smithi069 bash[28904]: cluster 2024-04-02T05:02:32.339576+0000 mgr.y (mgr.14538) 728 : cluster 0 pgmap v1267: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-04-02T05:02:33.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:33.633 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:33.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:33.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:02:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:02:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:33.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:33.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-02T05:02:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:33.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:33.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:34.020 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:34.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:34.151 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:34.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:34.282 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:34.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:34.379 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:34.379 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:34.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:34.407 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:34.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:34.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:34 smithi069 bash[28904]: audit 2024-04-02T05:02:33.719107+0000 mgr.y (mgr.14538) 729 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:34.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:34 smithi069 bash[28904]: audit 2024-04-02T05:02:34.110615+0000 mon.a (mon.0) 3855 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:34.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:34 smithi069 bash[28904]: audit 2024-04-02T05:02:34.112669+0000 mon.b (mon.1) 316 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:34.521 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:34 smithi069 bash[21528]: audit 2024-04-02T05:02:33.719107+0000 mgr.y (mgr.14538) 729 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:34.521 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:34 smithi069 bash[21528]: audit 2024-04-02T05:02:34.110615+0000 mon.a (mon.0) 3855 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:34.521 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:34 smithi069 bash[21528]: audit 2024-04-02T05:02:34.112669+0000 mon.b (mon.1) 316 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:34.535 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:34.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:34.666 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:34 smithi098 bash[23790]: audit 2024-04-02T05:02:33.719107+0000 mgr.y (mgr.14538) 729 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:34 smithi098 bash[23790]: audit 2024-04-02T05:02:34.110615+0000 mon.a (mon.0) 3855 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:02:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:34 smithi098 bash[23790]: audit 2024-04-02T05:02:34.112669+0000 mon.b (mon.1) 316 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:34.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:34.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:34.924 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:34.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:35.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:35.054 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:35.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:35.184 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:35.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:35.312 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:35.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:35.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:35 smithi069 bash[21528]: cluster 2024-04-02T05:02:34.341937+0000 mgr.y (mgr.14538) 730 : cluster 0 pgmap v1268: 224 pgs: 224 active+clean; 4.5 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 8.0 KiB/s wr, 2 op/s 2024-04-02T05:02:35.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:35 smithi069 bash[21528]: cluster 2024-04-02T05:02:35.111800+0000 mon.a (mon.0) 3856 : cluster 0 osdmap e835: 8 total, 8 up, 7 in 2024-04-02T05:02:35.424 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:35 smithi069 bash[28904]: cluster 2024-04-02T05:02:34.341937+0000 mgr.y (mgr.14538) 730 : cluster 0 pgmap v1268: 224 pgs: 224 active+clean; 4.5 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 8.0 KiB/s wr, 2 op/s 2024-04-02T05:02:35.424 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:35 smithi069 bash[28904]: cluster 2024-04-02T05:02:35.111800+0000 mon.a (mon.0) 3856 : cluster 0 osdmap e835: 8 total, 8 up, 7 in 2024-04-02T05:02:35.435 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:35.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:35.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:35.681 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:35 smithi098 bash[23790]: cluster 2024-04-02T05:02:34.341937+0000 mgr.y (mgr.14538) 730 : cluster 0 pgmap v1268: 224 pgs: 224 active+clean; 4.5 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 8.0 KiB/s wr, 2 op/s 2024-04-02T05:02:35.682 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:35 smithi098 bash[23790]: cluster 2024-04-02T05:02:35.111800+0000 mon.a (mon.0) 3856 : cluster 0 osdmap e835: 8 total, 8 up, 7 in 2024-04-02T05:02:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:35.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:35.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:35.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:36.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:36.079 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:36.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:36.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:36.341 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:36.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:36.465 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:36.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:36.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:36.728 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:36.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:36.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:36.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:37.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:37.118 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:37.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:37.247 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:37.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:37.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:37.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:37 smithi069 bash[21528]: cluster 2024-04-02T05:02:36.342495+0000 mgr.y (mgr.14538) 731 : cluster 0 pgmap v1270: 224 pgs: 224 active+clean; 4.5 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 5.1 KiB/s wr, 1 op/s 2024-04-02T05:02:37.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:37 smithi069 bash[21528]: audit 2024-04-02T05:02:36.383798+0000 mon.a (mon.0) 3857 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:37.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:37 smithi069 bash[21528]: audit 2024-04-02T05:02:36.384267+0000 mon.a (mon.0) 3858 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140"}]: dispatch 2024-04-02T05:02:37.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:37 smithi069 bash[28904]: cluster 2024-04-02T05:02:36.342495+0000 mgr.y (mgr.14538) 731 : cluster 0 pgmap v1270: 224 pgs: 224 active+clean; 4.5 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 5.1 KiB/s wr, 1 op/s 2024-04-02T05:02:37.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:37 smithi069 bash[28904]: audit 2024-04-02T05:02:36.383798+0000 mon.a (mon.0) 3857 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:37.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:37 smithi069 bash[28904]: audit 2024-04-02T05:02:36.384267+0000 mon.a (mon.0) 3858 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140"}]: dispatch 2024-04-02T05:02:37.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:37.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:37 smithi098 bash[23790]: cluster 2024-04-02T05:02:36.342495+0000 mgr.y (mgr.14538) 731 : cluster 0 pgmap v1270: 224 pgs: 224 active+clean; 4.5 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 5.1 KiB/s wr, 1 op/s 2024-04-02T05:02:37.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:37 smithi098 bash[23790]: audit 2024-04-02T05:02:36.383798+0000 mon.a (mon.0) 3857 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:37.623 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:37 smithi098 bash[23790]: audit 2024-04-02T05:02:36.384267+0000 mon.a (mon.0) 3858 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140"}]: dispatch 2024-04-02T05:02:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:37.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:37.765 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:37.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:37.895 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:37.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:37.955 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:37.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:38.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:38.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:38.150 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:38.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:38.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:38.448 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:38.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:38.549 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:38.651 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:38 smithi069 bash[28904]: audit 2024-04-02T05:02:37.379337+0000 mon.a (mon.0) 3859 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140"}]': finished 2024-04-02T05:02:38.651 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:38 smithi069 bash[28904]: cluster 2024-04-02T05:02:37.389616+0000 mon.a (mon.0) 3860 : cluster 0 osdmap e836: 8 total, 8 up, 7 in 2024-04-02T05:02:38.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:38.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:38 smithi098 bash[23790]: audit 2024-04-02T05:02:37.379337+0000 mon.a (mon.0) 3859 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140"}]': finished 2024-04-02T05:02:38.662 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:38 smithi098 bash[23790]: cluster 2024-04-02T05:02:37.389616+0000 mon.a (mon.0) 3860 : cluster 0 osdmap e836: 8 total, 8 up, 7 in 2024-04-02T05:02:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:38.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:38.809 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:38.812 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:02:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:02:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:02:38.812 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:38 smithi069 bash[21528]: audit 2024-04-02T05:02:37.379337+0000 mon.a (mon.0) 3859 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-140"}]': finished 2024-04-02T05:02:38.812 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:38 smithi069 bash[21528]: cluster 2024-04-02T05:02:37.389616+0000 mon.a (mon.0) 3860 : cluster 0 osdmap e836: 8 total, 8 up, 7 in 2024-04-02T05:02:38.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:38.937 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:38.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:39.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:39.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:39.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:39.329 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:39.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:39.463 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:39 smithi098 bash[23790]: cluster 2024-04-02T05:02:38.344452+0000 mgr.y (mgr.14538) 732 : cluster 0 pgmap v1272: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 9.4 KiB/s wr, 2 op/s 2024-04-02T05:02:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:39 smithi098 bash[23790]: cluster 2024-04-02T05:02:38.394868+0000 mon.a (mon.0) 3861 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:39.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:39 smithi098 bash[23790]: cluster 2024-04-02T05:02:38.412086+0000 mon.a (mon.0) 3862 : cluster 0 osdmap e837: 8 total, 8 up, 7 in 2024-04-02T05:02:39.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:39.592 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:39.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:39.722 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:39.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:39 smithi069 bash[21528]: cluster 2024-04-02T05:02:38.344452+0000 mgr.y (mgr.14538) 732 : cluster 0 pgmap v1272: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 9.4 KiB/s wr, 2 op/s 2024-04-02T05:02:39.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:39 smithi069 bash[21528]: cluster 2024-04-02T05:02:38.394868+0000 mon.a (mon.0) 3861 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:39.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:39 smithi069 bash[21528]: cluster 2024-04-02T05:02:38.412086+0000 mon.a (mon.0) 3862 : cluster 0 osdmap e837: 8 total, 8 up, 7 in 2024-04-02T05:02:39.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:39 smithi069 bash[28904]: cluster 2024-04-02T05:02:38.344452+0000 mgr.y (mgr.14538) 732 : cluster 0 pgmap v1272: 224 pgs: 224 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 9.4 KiB/s wr, 2 op/s 2024-04-02T05:02:39.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:39 smithi069 bash[28904]: cluster 2024-04-02T05:02:38.394868+0000 mon.a (mon.0) 3861 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:39.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:39 smithi069 bash[28904]: cluster 2024-04-02T05:02:38.412086+0000 mon.a (mon.0) 3862 : cluster 0 osdmap e837: 8 total, 8 up, 7 in 2024-04-02T05:02:39.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:39.852 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:39.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:39.983 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:39.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:40.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:40.119 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:40.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:40.250 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:40.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:40.381 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:40.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:40.511 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:40.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:40.637 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:40.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:40 smithi098 bash[23790]: cluster 2024-04-02T05:02:39.433177+0000 mon.a (mon.0) 3863 : cluster 0 osdmap e838: 8 total, 8 up, 7 in 2024-04-02T05:02:40.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:40 smithi098 bash[23790]: audit 2024-04-02T05:02:39.442887+0000 mon.a (mon.0) 3864 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:40.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:40 smithi098 bash[23790]: audit 2024-04-02T05:02:40.105836+0000 mon.a (mon.0) 3865 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:40.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:40 smithi098 bash[23790]: cluster 2024-04-02T05:02:40.111542+0000 mon.a (mon.0) 3866 : cluster 0 osdmap e839: 8 total, 8 up, 7 in 2024-04-02T05:02:40.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:40 smithi098 bash[23790]: audit 2024-04-02T05:02:40.233166+0000 mon.a (mon.0) 3867 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:40.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:40.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[28904]: cluster 2024-04-02T05:02:39.433177+0000 mon.a (mon.0) 3863 : cluster 0 osdmap e838: 8 total, 8 up, 7 in 2024-04-02T05:02:40.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[28904]: audit 2024-04-02T05:02:39.442887+0000 mon.a (mon.0) 3864 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:40.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[28904]: audit 2024-04-02T05:02:40.105836+0000 mon.a (mon.0) 3865 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[28904]: cluster 2024-04-02T05:02:40.111542+0000 mon.a (mon.0) 3866 : cluster 0 osdmap e839: 8 total, 8 up, 7 in 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[28904]: audit 2024-04-02T05:02:40.233166+0000 mon.a (mon.0) 3867 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[21528]: cluster 2024-04-02T05:02:39.433177+0000 mon.a (mon.0) 3863 : cluster 0 osdmap e838: 8 total, 8 up, 7 in 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[21528]: audit 2024-04-02T05:02:39.442887+0000 mon.a (mon.0) 3864 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[21528]: audit 2024-04-02T05:02:40.105836+0000 mon.a (mon.0) 3865 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[21528]: cluster 2024-04-02T05:02:40.111542+0000 mon.a (mon.0) 3866 : cluster 0 osdmap e839: 8 total, 8 up, 7 in 2024-04-02T05:02:40.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:40 smithi069 bash[21528]: audit 2024-04-02T05:02:40.233166+0000 mon.a (mon.0) 3867 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:40.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- 172.21.15.69:0/1647367741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 msgr2=0x7f607010f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:40.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- 172.21.15.69:0/1647367741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f607010f130 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7f6068009990 tx=0x7f6068009ca0 comp rx=0 tx=0).stop 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- 172.21.15.69:0/1647367741 shutdown_connections 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- 172.21.15.69:0/1647367741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f607010f130 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- 172.21.15.69:0/1647367741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60701082f0 0x7f60701086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- 172.21.15.69:0/1647367741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f607006a8c0 0x7f607006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- 172.21.15.69:0/1647367741 >> 172.21.15.69:0/1647367741 conn(0x7f6070078140 msgr2=0x7f6070078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- 172.21.15.69:0/1647367741 shutdown_connections 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- 172.21.15.69:0/1647367741 wait complete. 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 librados: starting msgr at 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 librados: starting objecter 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 Processor -- start 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- start start 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 librados: setting wanted keys 2024-04-02T05:02:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 librados: calling monclient init 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 0x7f6070197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f6070198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60701082f0 0x7f607019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6070111a40 con 0x7f607006b260 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60701118c0 con 0x7f60701082f0 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6076ece700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6070111bc0 con 0x7f607006a8c0 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6074c6a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 0x7f6070197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6074c6a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 0x7f6070197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57054/0 (socket says 172.21.15.69:57054) 2024-04-02T05:02:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f6074c6a700 1 -- 172.21.15.69:0/956044879 learned_addr learned my addr 172.21.15.69:0/956044879 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f606ffff700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f6070198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.796+0000 7f607546b700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60701082f0 0x7f607019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606ffff700 1 -- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 msgr2=0x7f6070197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606ffff700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 0x7f6070197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606ffff700 1 -- 172.21.15.69:0/956044879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60701082f0 msgr2=0x7f607019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606ffff700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60701082f0 0x7f607019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606ffff700 1 -- 172.21.15.69:0/956044879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607019cfe0 con 0x7f607006b260 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6074c6a700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 0x7f6070197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606ffff700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f6070198510 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f605800e9f0 tx=0x7f605800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 10 client.?.objecter ms_handle_connect 0x7f607006b260 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605800c9c0 con 0x7f607006b260 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6058010430 con 0x7f607006b260 2024-04-02T05:02:40.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607019d2d0 con 0x7f607006b260 2024-04-02T05:02:40.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058018680 con 0x7f607006b260 2024-04-02T05:02:40.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6076ece700 10 client.33765.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:40.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60701a4b50 con 0x7f607006b260 2024-04-02T05:02:40.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6076ece700 1 librados: init done 2024-04-02T05:02:40.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60701039a0 con 0x7f607006b260 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f60580105e0 con 0x7f607006b260 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f605c06f5e0 0x7f605c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6074c6a700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f605c06f5e0 0x7f605c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(839..839 src has 74..839) ==== 10112+0+0 (secure 0 0 0) 0x7f605808eb50 con 0x7f607006b260 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 10 client.33765.objecter ms_dispatch 0x7f6070001550 osd_map(839..839 src has 74..839) 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 3 client.33765.objecter handle_osd_map got epochs [839,839] > 0 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 3 client.33765.objecter handle_osd_map decoding full epoch 839 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 20 client.33765.objecter dump_active .. 0 homeless 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f6074c6a700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f605c06f5e0 0x7f605c071aa0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7f6060009480 tx=0x7f606000b040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:40.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.800+0000 7f606dffb700 10 client.33765.objecter ms_handle_connect 0x7f605c06f5e0 2024-04-02T05:02:40.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:40.808+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f605805b6e0 con 0x7f607006b260 2024-04-02T05:02:40.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:40.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:40.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:41.019 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.044+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699} v 0) -- 0x7f60701a4d20 con 0x7f607006b260 2024-04-02T05:02:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.112+0000 7f606dffb700 1 -- 172.21.15.69:0/956044879 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]=0 set osd.0 primary-affinity to 0.692343 (8453732) v840) ==== 155+0+0 (secure 0 0 0) 0x7f605805f130 con 0x7f607006b260 2024-04-02T05:02:41.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 10 librados: watch_flush enter 2024-04-02T05:02:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 10 librados: watch_flush exit 2024-04-02T05:02:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 20 client.33765.objecter shutdown clearing up homeless session... 2024-04-02T05:02:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 10 client.33765.objecter successfully canceled tick 2024-04-02T05:02:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f605c06f5e0 msgr2=0x7f605c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f605c06f5e0 0x7f605c071aa0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7f6060009480 tx=0x7f606000b040 comp rx=0 tx=0).stop 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 msgr2=0x7f6070198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f6070198510 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f605800e9f0 tx=0x7f605800ed00 comp rx=0 tx=0).stop 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 shutdown_connections 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f605c06f5e0 0x7f605c071aa0 secure :-1 s=CLOSED pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7f6060009480 tx=0x7f606000b040 comp rx=0 tx=0).stop 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60701082f0 0x7f607019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f607006a8c0 0x7f6070197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 --2- 172.21.15.69:0/956044879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607006b260 0x7f6070198510 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.116+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 >> 172.21.15.69:0/956044879 conn(0x7f6070078140 msgr2=0x7f60700ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:41.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.120+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 shutdown_connections 2024-04-02T05:02:41.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.120+0000 7f6076ece700 1 -- 172.21.15.69:0/956044879 wait complete. 2024-04-02T05:02:41.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:41.124+0000 7f6076ece700 1 librados: shutdown 2024-04-02T05:02:41.140 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.0 primary-affinity to 0.692343 (8453732) 2024-04-02T05:02:41.153 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:41.244 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:02:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T05:02:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T05:02:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T05:02:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:2487 4.3 4:c609908c:::notify.5:head [watch ping cookie 94452570221952 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:54.776064+0000", 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 526.46458238900004, 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.16532492500000001, 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2487 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776064+0000", 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776064+0000", 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776067+0000", 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7299999999999999e-06 2024-04-02T05:02:41.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776085+0000", 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7428999999999999e-05 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776134+0000", 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8832999999999998e-05 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776166+0000", 2024-04-02T05:02:41.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2836000000000002e-05 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776439+0000", 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000273033 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.776481+0000", 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2002999999999997e-05 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.941135+0000", 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.16465365000000001 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.941307+0000", 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017156199999999999 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.941389+0000", 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1848999999999995e-05 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.249 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:2497 4.3 4:c609908c:::notify.5:head [watch reconnect cookie 94452570221952 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:53:54.779969+0000", 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 526.46067707199995, 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.178588097, 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:02:41.250 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2497 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779969+0000", 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779969+0000", 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779971+0000", 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2699999999999999e-06 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.779984+0000", 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2480000000000001e-05 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.780022+0000", 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7908999999999999e-05 2024-04-02T05:02:41.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.780047+0000", 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5507000000000002e-05 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.780212+0000", 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000164714 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for peered", 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.780234+0000", 2024-04-02T05:02:41.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2696000000000001e-05 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.946573+0000", 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.16633883099999999 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.946698+0000", 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012425199999999999 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:53:54.958557+0000", 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011859438 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(41.0,QREQUEST,e:472,from:4,reservation_nonce:2)", 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:55:19.968104+0000", 2024-04-02T05:02:41.256 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 441.27254148399999, 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.050792392999999998, 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968104+0000", 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968104+0000", 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968107+0000", 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0140000000000001e-06 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968121+0000", 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4511e-05 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968153+0000", 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2274999999999998e-05 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968175+0000", 2024-04-02T05:02:41.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1685e-05 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968334+0000", 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015892400000000001 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:19.968349+0000", 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4891e-05 2024-04-02T05:02:41.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:20.018811+0000", 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.050461747000000001 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:20.018832+0000", 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1537999999999999e-05 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:55:20.018897+0000", 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4807999999999993e-05 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:17 324.a 324:50000000:%01:::head [pgnls start_epoch 702] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e702)", 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:00.077960+0000", 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 161.162685872, 2024-04-02T05:02:41.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.025442811999999999, 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 17 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.077960+0000", 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.077960+0000", 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.077962+0000", 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6440000000000001e-06 2024-04-02T05:02:41.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.077967+0000", 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9740000000000001e-06 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.077975+0000", 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1669999999999999e-06 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.265 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.077982+0000", 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0550000000000002e-06 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.078045+0000", 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2910000000000006e-05 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.078050+0000", 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2279999999999998e-06 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.103214+0000", 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.025164405000000001 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.103263+0000", 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8934000000000003e-05 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:00.103403+0000", 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013949500000000001 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:256 326.1 326:84fc9ed6:test-rados-api-smithi069-72349-117::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e719)", 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:17.060194+0000", 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 144.180451616, 2024-04-02T05:02:41.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0078960290000000006, 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 256 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060194+0000", 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060194+0000", 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060196+0000", 2024-04-02T05:02:41.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4330000000000001e-06 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060202+0000", 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5660000000000003e-06 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060213+0000", 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0484e-05 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060219+0000", 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.348e-06 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060261+0000", 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2250999999999999e-05 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.060413+0000", 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015109800000000001 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.065495+0000", 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050819749999999999 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.065597+0000", 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010227500000000001 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.066021+0000", 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000424268 2024-04-02T05:02:41.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.066079+0000", 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8387999999999997e-05 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:17.068090+0000", 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0020109429999999998 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.274 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.32028.0:475 328.1b e740/736 328:df8c41d3:test-rados-api-smithi069-72349-121::foo:head v 740'2, mlcod=0'0)", 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:50.077973+0000", 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 111.16267342, 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074568489999999998, 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.275 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.077973+0000", 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.077973+0000", 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.077974+0000", 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0899999999999999e-06 2024-04-02T05:02:41.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.077981+0000", 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1339999999999998e-06 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.077993+0000", 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2105e-05 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.078002+0000", 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6260000000000001e-06 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.078085+0000", 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3048999999999997e-05 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.078106+0000", 2024-04-02T05:02:41.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1537999999999999e-05 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.083984+0000", 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058781140000000003 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.085429+0000", 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001445193 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(2.12,QREQUEST,e:751,from:3,reservation_nonce:2)", 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:01.647943+0000", 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 99.592702513000006, 2024-04-02T05:02:41.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067375270000000001, 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.647943+0000", 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.647943+0000", 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.647945+0000", 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7090000000000001e-06 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.647953+0000", 2024-04-02T05:02:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2889999999999998e-06 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.647970+0000", 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6316000000000001e-05 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.647983+0000", 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3224000000000001e-05 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.648060+0000", 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7072999999999999e-05 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.648067+0000", 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3810000000000001e-06 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.654641+0000", 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065738990000000002 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.654653+0000", 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1817999999999999e-05 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:01.654681+0000", 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7818e-05 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.285 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(3.b,QREQUEST,e:768,from:3,reservation_nonce:2)", 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:18.707422+0000", 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 82.533224168999993, 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0089861290000000007, 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707422+0000", 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707422+0000", 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707425+0000", 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.773e-06 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707439+0000", 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.49e-05 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707475+0000", 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5185000000000003e-05 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707493+0000", 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8113000000000001e-05 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707619+0000", 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012652099999999999 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.707633+0000", 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3360000000000001e-05 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.716298+0000", 2024-04-02T05:02:41.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086655450000000002 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.716329+0000", 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0911999999999998e-05 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:18.716408+0000", 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8819999999999994e-05 2024-04-02T05:02:41.290 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(2.2,QREQUEST,e:773,from:1,reservation_nonce:4)", 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:23.180620+0000", 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 78.060026148999995, 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059025500000000003, 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180620+0000", 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180620+0000", 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180622+0000", 2024-04-02T05:02:41.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2230000000000001e-06 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180635+0000", 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2622999999999999e-05 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180673+0000", 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7938000000000002e-05 2024-04-02T05:02:41.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180693+0000", 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0573000000000001e-05 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180838+0000", 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014516099999999999 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.180853+0000", 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4786e-05 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.186420+0000", 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0055673279999999999 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.186438+0000", 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7895000000000001e-05 2024-04-02T05:02:41.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:23.186522+0000", 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4023000000000005e-05 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:1146 323.4s0 323:20000000:test-rados-api-smithi069-72349-129:::head [pgnls start_epoch 776] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e776)", 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:29.123648+0000", 2024-04-02T05:02:41.296 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 72.116998402999997, 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082138330000000002, 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1146 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123648+0000", 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123648+0000", 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123651+0000", 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0230000000000001e-06 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123664+0000", 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2999e-05 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123697+0000", 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3275999999999998e-05 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123714+0000", 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7431999999999998e-05 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123900+0000", 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018526999999999999 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.123912+0000", 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2359e-05 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.131482+0000", 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0075697830000000001 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.131536+0000", 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4750000000000003e-05 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:29.131861+0000", 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00032494099999999998 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:1247 332.19 332:98000000::::head [pgnls start_epoch 778] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:30.783666+0000", 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 70.456980329000004, 2024-04-02T05:02:41.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063950830000000002, 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1247 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783666+0000", 2024-04-02T05:02:41.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783666+0000", 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783669+0000", 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.27e-06 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783686+0000", 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6979000000000001e-05 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783724+0000", 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8458999999999999e-05 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783743+0000", 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8678999999999999e-05 2024-04-02T05:02:41.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783896+0000", 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000153017 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.783907+0000", 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1056999999999999e-05 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.789632+0000", 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057251120000000001 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.789720+0000", 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7613000000000003e-05 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:30.790061+0000", 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000340897 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.307 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.6.58:154 323.5s0 323:b062609c:test-rados-api-smithi069-72349-135::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:57.107896+0000", 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 44.132750100000003, 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011157279000000001, 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T05:02:41.308 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.107896+0000", 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.107896+0000", 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.107901+0000", 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7099999999999998e-06 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.107922+0000", 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1903e-05 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.107974+0000", 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1490999999999999e-05 2024-04-02T05:02:41.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108006+0000", 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1702000000000002e-05 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108203+0000", 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019733000000000001 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108226+0000", 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2671999999999999e-05 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.118793+0000", 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010566829999999999 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.119053+0000", 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026064099999999999 2024-04-02T05:02:41.312 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.6.58:155 323.5s0 323:b062609c:test-rados-api-smithi069-72349-135::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:57.108064+0000", 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 44.132582176, 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011260229, 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T05:02:41.313 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108064+0000", 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108064+0000", 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108065+0000", 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.637e-06 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108077+0000", 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1336e-05 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108099+0000", 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2201999999999999e-05 2024-04-02T05:02:41.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108121+0000", 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.228e-05 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108375+0000", 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025411699999999999 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting_for_map not empty", 2024-04-02T05:02:41.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.108397+0000", 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2022e-05 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.119110+0000", 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010712886 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.119324+0000", 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000213749 2024-04-02T05:02:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.6.58:156 323.5s0 323:b062609c:test-rados-api-smithi069-72349-135::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:57.123854+0000", 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 44.116791597999999, 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040402449999999996, 2024-04-02T05:02:41.318 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.123854+0000", 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.123854+0000", 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.123857+0000", 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7269999999999999e-06 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.123872+0000", 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4623e-05 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.123903+0000", 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1171999999999999e-05 2024-04-02T05:02:41.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.123920+0000", 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.747e-05 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.124041+0000", 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000120315 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.124270+0000", 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000229502 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.126261+0000", 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00199066 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.126927+0000", 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00066595499999999998 2024-04-02T05:02:41.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.127710+0000", 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00078351500000000004 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.127895+0000", 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018430599999999999 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.323 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.6.58:159 323.5s0 323:b062609c:test-rados-api-smithi069-72349-135::foo:head [copy-from ver 6] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:57.140517+0000", 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 44.100129250999998, 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074181769999999998, 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140517+0000", 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140517+0000", 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140519+0000", 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1299999999999999e-06 2024-04-02T05:02:41.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140530+0000", 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0753e-05 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140552+0000", 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2714999999999999e-05 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140571+0000", 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8283000000000001e-05 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.140950+0000", 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00037960000000000001 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.141054+0000", 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000103386 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.143092+0000", 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0020387920000000002 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.143375+0000", 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000282346 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:02:41.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.146245+0000", 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028704410000000001 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.147098+0000", 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00085327500000000002 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.147791+0000", 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00069224800000000002 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.147935+0000", 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000144208 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.6.58:160 323.5s0 323:b062609c:test-rados-api-smithi069-72349-135::foo:head [copy-from ver 8] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:57.152887+0000", 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 44.087759200000001, 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046994539999999996, 2024-04-02T05:02:41.330 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.152887+0000", 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.152887+0000", 2024-04-02T05:02:41.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.152889+0000", 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1469999999999999e-06 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.152901+0000", 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2529e-05 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.152932+0000", 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0821999999999997e-05 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.152949+0000", 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6535999999999999e-05 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.153112+0000", 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016269799999999999 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.153398+0000", 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028691200000000001 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.155872+0000", 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024739530000000001 2024-04-02T05:02:41.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.156667+0000", 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079478700000000003 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.157455+0000", 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00078805800000000001 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:57.157586+0000", 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013101200000000001 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.3.29:94 323.4s0 323:2c2c3553:test-rados-api-smithi069-72349-137::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e813)", 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.132191+0000", 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 36.108454729999998, 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.038358504000000002, 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.337 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132191+0000", 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132191+0000", 2024-04-02T05:02:41.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132194+0000", 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2620000000000001e-06 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132204+0000", 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-05 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132227+0000", 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2271000000000001e-05 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132247+0000", 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0211e-05 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132345+0000", 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7878e-05 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132566+0000", 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022169299999999999 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.144947+0000", 2024-04-02T05:02:41.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012380700999999999 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.170185+0000", 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.025237691 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.170239+0000", 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4592000000000002e-05 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.170550+0000", 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00031027499999999999 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:1834 336.4 336:20000000::::head [pgnls start_epoch 815] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e815)", 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:07.139751+0000", 2024-04-02T05:02:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 34.100895420000001, 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0085711710000000007, 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1834 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139751+0000", 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139751+0000", 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139754+0000", 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9560000000000002e-06 2024-04-02T05:02:41.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139767+0000", 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3076e-05 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139796+0000", 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9122e-05 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139845+0000", 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8952000000000002e-05 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139970+0000", 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000125606 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:02:41.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.139981+0000", 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1066e-05 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.148016+0000", 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0080350479999999995 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.148104+0000", 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7990000000000003e-05 2024-04-02T05:02:41.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:07.148322+0000", 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000217355 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:815,from:1,reservation_nonce:6)", 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:08.141840+0000", 2024-04-02T05:02:41.349 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 33.098805874999996, 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.052951993000000003, 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.141840+0000", 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.141840+0000", 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.141843+0000", 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8289999999999999e-06 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.141856+0000", 2024-04-02T05:02:41.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.271e-05 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.192354+0000", 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.050498589000000003 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.192367+0000", 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2405999999999999e-05 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.194631+0000", 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002264734 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.194651+0000", 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0085000000000002e-05 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:08.194792+0000", 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014064000000000001 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4007 4.3 4:c609908c:::notify.5:head [watch ping cookie 94452570221952 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:40.192387+0000", 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 1.0482586869999999, 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0021488140000000002, 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:02:41.354 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4007 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.192387+0000", 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.192387+0000", 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.192389+0000", 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4649999999999999e-06 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:02:41.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.192393+0000", 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1810000000000003e-06 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.192403+0000", 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7860000000000008e-06 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.192412+0000", 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2010000000000005e-06 2024-04-02T05:02:41.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.193993+0000", 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001581547 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194094+0000", 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000100529 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:02:41.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194536+0000", 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00044210499999999998 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:02:41.359 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:02:41.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:41.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:41.516 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:41.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:41.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:41.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:41 smithi098 bash[23790]: cluster 2024-04-02T05:02:40.345024+0000 mgr.y (mgr.14538) 733 : cluster 0 pgmap v1276: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:41.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:41 smithi098 bash[23790]: audit 2024-04-02T05:02:41.048939+0000 mon.a (mon.0) 3868 : audit 1 from='client.? 172.21.15.69:0/956044879' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]: dispatch 2024-04-02T05:02:41.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:41 smithi098 bash[23790]: audit 2024-04-02T05:02:41.118375+0000 mon.a (mon.0) 3869 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:41 smithi098 bash[23790]: audit 2024-04-02T05:02:41.118450+0000 mon.a (mon.0) 3870 : audit 1 from='client.? 172.21.15.69:0/956044879' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]': finished 2024-04-02T05:02:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:41 smithi098 bash[23790]: cluster 2024-04-02T05:02:41.122266+0000 mon.a (mon.0) 3871 : cluster 0 osdmap e840: 8 total, 8 up, 7 in 2024-04-02T05:02:41.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:41 smithi098 bash[23790]: audit 2024-04-02T05:02:41.124047+0000 mon.a (mon.0) 3872 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:41.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[21528]: cluster 2024-04-02T05:02:40.345024+0000 mgr.y (mgr.14538) 733 : cluster 0 pgmap v1276: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[21528]: audit 2024-04-02T05:02:41.048939+0000 mon.a (mon.0) 3868 : audit 1 from='client.? 172.21.15.69:0/956044879' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]: dispatch 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[21528]: audit 2024-04-02T05:02:41.118375+0000 mon.a (mon.0) 3869 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[21528]: audit 2024-04-02T05:02:41.118450+0000 mon.a (mon.0) 3870 : audit 1 from='client.? 172.21.15.69:0/956044879' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]': finished 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[21528]: cluster 2024-04-02T05:02:41.122266+0000 mon.a (mon.0) 3871 : cluster 0 osdmap e840: 8 total, 8 up, 7 in 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[21528]: audit 2024-04-02T05:02:41.124047+0000 mon.a (mon.0) 3872 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[28904]: cluster 2024-04-02T05:02:40.345024+0000 mgr.y (mgr.14538) 733 : cluster 0 pgmap v1276: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:41.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[28904]: audit 2024-04-02T05:02:41.048939+0000 mon.a (mon.0) 3868 : audit 1 from='client.? 172.21.15.69:0/956044879' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]: dispatch 2024-04-02T05:02:41.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[28904]: audit 2024-04-02T05:02:41.118375+0000 mon.a (mon.0) 3869 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:41.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[28904]: audit 2024-04-02T05:02:41.118450+0000 mon.a (mon.0) 3870 : audit 1 from='client.? 172.21.15.69:0/956044879' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6923426559459699}]': finished 2024-04-02T05:02:41.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[28904]: cluster 2024-04-02T05:02:41.122266+0000 mon.a (mon.0) 3871 : cluster 0 osdmap e840: 8 total, 8 up, 7 in 2024-04-02T05:02:41.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:41 smithi069 bash[28904]: audit 2024-04-02T05:02:41.124047+0000 mon.a (mon.0) 3872 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:41.774 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:41.877 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:41.904 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:41.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:42.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:42.035 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:42.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:42.165 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:42.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:42.296 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:42.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:42.426 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:42.531 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:42.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:42.688 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:42.792 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:42.818 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:42.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:42.948 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:42.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:43.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:43.078 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:43.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:43.208 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:43.311 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[28904]: audit 2024-04-02T05:02:42.121650+0000 mon.a (mon.0) 3873 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-142"}]': finished 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[28904]: cluster 2024-04-02T05:02:42.129920+0000 mon.a (mon.0) 3874 : cluster 0 osdmap e841: 8 total, 8 up, 7 in 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[28904]: audit 2024-04-02T05:02:42.132572+0000 mon.a (mon.0) 3875 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-142", "mode": "writeback"}]: dispatch 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[28904]: cluster 2024-04-02T05:02:42.345530+0000 mgr.y (mgr.14538) 734 : cluster 0 pgmap v1279: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[21528]: audit 2024-04-02T05:02:42.121650+0000 mon.a (mon.0) 3873 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-142"}]': finished 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[21528]: cluster 2024-04-02T05:02:42.129920+0000 mon.a (mon.0) 3874 : cluster 0 osdmap e841: 8 total, 8 up, 7 in 2024-04-02T05:02:43.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[21528]: audit 2024-04-02T05:02:42.132572+0000 mon.a (mon.0) 3875 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-142", "mode": "writeback"}]: dispatch 2024-04-02T05:02:43.324 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:43 smithi069 bash[21528]: cluster 2024-04-02T05:02:42.345530+0000 mgr.y (mgr.14538) 734 : cluster 0 pgmap v1279: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:43.338 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:43.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:43.469 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:43 smithi098 bash[23790]: audit 2024-04-02T05:02:42.121650+0000 mon.a (mon.0) 3873 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-142"}]': finished 2024-04-02T05:02:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:43 smithi098 bash[23790]: cluster 2024-04-02T05:02:42.129920+0000 mon.a (mon.0) 3874 : cluster 0 osdmap e841: 8 total, 8 up, 7 in 2024-04-02T05:02:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:43 smithi098 bash[23790]: audit 2024-04-02T05:02:42.132572+0000 mon.a (mon.0) 3875 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-142", "mode": "writeback"}]: dispatch 2024-04-02T05:02:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:43 smithi098 bash[23790]: cluster 2024-04-02T05:02:42.345530+0000 mgr.y (mgr.14538) 734 : cluster 0 pgmap v1279: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 739 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:43.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:43.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:43.729 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:43.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:43.857 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:43.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:43.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:02:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:02:43.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:43.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:44.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:44.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[28904]: cluster 2024-04-02T05:02:43.122253+0000 mon.a (mon.0) 3876 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:44.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[28904]: audit 2024-04-02T05:02:43.131915+0000 mon.a (mon.0) 3877 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-142", "mode": "writeback"}]': finished 2024-04-02T05:02:44.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[28904]: cluster 2024-04-02T05:02:43.141227+0000 mon.a (mon.0) 3878 : cluster 0 osdmap e842: 8 total, 8 up, 7 in 2024-04-02T05:02:44.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[28904]: audit 2024-04-02T05:02:43.145028+0000 mon.a (mon.0) 3879 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:02:44.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[28904]: audit 2024-04-02T05:02:43.729718+0000 mgr.y (mgr.14538) 735 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:44.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:44.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:44.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:44 smithi098 bash[23790]: cluster 2024-04-02T05:02:43.122253+0000 mon.a (mon.0) 3876 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:44.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:44 smithi098 bash[23790]: audit 2024-04-02T05:02:43.131915+0000 mon.a (mon.0) 3877 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-142", "mode": "writeback"}]': finished 2024-04-02T05:02:44.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:44 smithi098 bash[23790]: cluster 2024-04-02T05:02:43.141227+0000 mon.a (mon.0) 3878 : cluster 0 osdmap e842: 8 total, 8 up, 7 in 2024-04-02T05:02:44.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:44 smithi098 bash[23790]: audit 2024-04-02T05:02:43.145028+0000 mon.a (mon.0) 3879 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:02:44.361 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:44 smithi098 bash[23790]: audit 2024-04-02T05:02:43.729718+0000 mgr.y (mgr.14538) 735 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:44.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:44.507 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:44.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[21528]: cluster 2024-04-02T05:02:43.122253+0000 mon.a (mon.0) 3876 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:44.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[21528]: audit 2024-04-02T05:02:43.131915+0000 mon.a (mon.0) 3877 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-142", "mode": "writeback"}]': finished 2024-04-02T05:02:44.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[21528]: cluster 2024-04-02T05:02:43.141227+0000 mon.a (mon.0) 3878 : cluster 0 osdmap e842: 8 total, 8 up, 7 in 2024-04-02T05:02:44.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[21528]: audit 2024-04-02T05:02:43.145028+0000 mon.a (mon.0) 3879 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:02:44.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:44 smithi069 bash[21528]: audit 2024-04-02T05:02:43.729718+0000 mgr.y (mgr.14538) 735 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:44.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:44.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:44.766 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:44.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:44.897 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:44.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:45.000 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:45.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:45.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:45.159 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:45.262 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:45.273 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[28904]: audit 2024-04-02T05:02:44.147296+0000 mon.a (mon.0) 3880 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[28904]: cluster 2024-04-02T05:02:44.156858+0000 mon.a (mon.0) 3881 : cluster 0 osdmap e843: 8 total, 8 up, 7 in 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[28904]: audit 2024-04-02T05:02:44.160897+0000 mon.a (mon.0) 3882 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[28904]: cluster 2024-04-02T05:02:44.346286+0000 mgr.y (mgr.14538) 736 : cluster 0 pgmap v1282: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 5 op/s 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[28904]: cluster 2024-04-02T05:02:45.105367+0000 mon.a (mon.0) 3883 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[21528]: audit 2024-04-02T05:02:44.147296+0000 mon.a (mon.0) 3880 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[21528]: cluster 2024-04-02T05:02:44.156858+0000 mon.a (mon.0) 3881 : cluster 0 osdmap e843: 8 total, 8 up, 7 in 2024-04-02T05:02:45.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[21528]: audit 2024-04-02T05:02:44.160897+0000 mon.a (mon.0) 3882 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:02:45.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[21528]: cluster 2024-04-02T05:02:44.346286+0000 mgr.y (mgr.14538) 736 : cluster 0 pgmap v1282: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 5 op/s 2024-04-02T05:02:45.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:45 smithi069 bash[21528]: cluster 2024-04-02T05:02:45.105367+0000 mon.a (mon.0) 3883 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:45.289 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:45.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:45.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:45 smithi098 bash[23790]: audit 2024-04-02T05:02:44.147296+0000 mon.a (mon.0) 3880 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:02:45.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:45 smithi098 bash[23790]: cluster 2024-04-02T05:02:44.156858+0000 mon.a (mon.0) 3881 : cluster 0 osdmap e843: 8 total, 8 up, 7 in 2024-04-02T05:02:45.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:45 smithi098 bash[23790]: audit 2024-04-02T05:02:44.160897+0000 mon.a (mon.0) 3882 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:02:45.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:45 smithi098 bash[23790]: cluster 2024-04-02T05:02:44.346286+0000 mgr.y (mgr.14538) 736 : cluster 0 pgmap v1282: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 5 op/s 2024-04-02T05:02:45.404 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:45 smithi098 bash[23790]: cluster 2024-04-02T05:02:45.105367+0000 mon.a (mon.0) 3883 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:45.418 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:45.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:45.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-02T05:02:45.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:45.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:45.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:45.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:45.931 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:45.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:46.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:46.059 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:46.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:46.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:46.318 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:46.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:46.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: audit 2024-04-02T05:02:45.151273+0000 mon.a (mon.0) 3884 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:02:46.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: cluster 2024-04-02T05:02:45.159166+0000 mon.a (mon.0) 3885 : cluster 0 osdmap e844: 8 total, 8 up, 7 in 2024-04-02T05:02:46.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: audit 2024-04-02T05:02:45.184271+0000 mon.a (mon.0) 3886 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:02:46.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: cluster 2024-04-02T05:02:46.151896+0000 mon.a (mon.0) 3887 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:02:46.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: audit 2024-04-02T05:02:46.154250+0000 mon.a (mon.0) 3888 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:02:46.432 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: cluster 2024-04-02T05:02:46.164045+0000 mon.a (mon.0) 3889 : cluster 0 osdmap e845: 8 total, 8 up, 7 in 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[28904]: audit 2024-04-02T05:02:46.165914+0000 mon.a (mon.0) 3890 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: audit 2024-04-02T05:02:45.151273+0000 mon.a (mon.0) 3884 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: cluster 2024-04-02T05:02:45.159166+0000 mon.a (mon.0) 3885 : cluster 0 osdmap e844: 8 total, 8 up, 7 in 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: audit 2024-04-02T05:02:45.184271+0000 mon.a (mon.0) 3886 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: cluster 2024-04-02T05:02:46.151896+0000 mon.a (mon.0) 3887 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: audit 2024-04-02T05:02:46.154250+0000 mon.a (mon.0) 3888 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: cluster 2024-04-02T05:02:46.164045+0000 mon.a (mon.0) 3889 : cluster 0 osdmap e845: 8 total, 8 up, 7 in 2024-04-02T05:02:46.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:46 smithi069 bash[21528]: audit 2024-04-02T05:02:46.165914+0000 mon.a (mon.0) 3890 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T05:02:46.449 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: audit 2024-04-02T05:02:45.151273+0000 mon.a (mon.0) 3884 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:02:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: cluster 2024-04-02T05:02:45.159166+0000 mon.a (mon.0) 3885 : cluster 0 osdmap e844: 8 total, 8 up, 7 in 2024-04-02T05:02:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: audit 2024-04-02T05:02:45.184271+0000 mon.a (mon.0) 3886 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:02:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: cluster 2024-04-02T05:02:46.151896+0000 mon.a (mon.0) 3887 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:02:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: audit 2024-04-02T05:02:46.154250+0000 mon.a (mon.0) 3888 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:02:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: cluster 2024-04-02T05:02:46.164045+0000 mon.a (mon.0) 3889 : cluster 0 osdmap e845: 8 total, 8 up, 7 in 2024-04-02T05:02:46.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:46 smithi098 bash[23790]: audit 2024-04-02T05:02:46.165914+0000 mon.a (mon.0) 3890 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T05:02:46.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:46.577 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:46.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:46.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:46.835 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:46.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:46.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:47.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:47.101 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:47.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:47.229 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:47.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:47.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[21528]: cluster 2024-04-02T05:02:46.347073+0000 mgr.y (mgr.14538) 737 : cluster 0 pgmap v1285: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 5 op/s 2024-04-02T05:02:47.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[21528]: audit 2024-04-02T05:02:47.156317+0000 mon.a (mon.0) 3891 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T05:02:47.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[21528]: cluster 2024-04-02T05:02:47.160891+0000 mon.a (mon.0) 3892 : cluster 0 osdmap e846: 8 total, 8 up, 7 in 2024-04-02T05:02:47.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[21528]: audit 2024-04-02T05:02:47.161454+0000 mon.a (mon.0) 3893 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T05:02:47.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[28904]: cluster 2024-04-02T05:02:46.347073+0000 mgr.y (mgr.14538) 737 : cluster 0 pgmap v1285: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 5 op/s 2024-04-02T05:02:47.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[28904]: audit 2024-04-02T05:02:47.156317+0000 mon.a (mon.0) 3891 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T05:02:47.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[28904]: cluster 2024-04-02T05:02:47.160891+0000 mon.a (mon.0) 3892 : cluster 0 osdmap e846: 8 total, 8 up, 7 in 2024-04-02T05:02:47.342 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:47 smithi069 bash[28904]: audit 2024-04-02T05:02:47.161454+0000 mon.a (mon.0) 3893 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T05:02:47.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/30495251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102ed0 msgr2=0x7f6bc8106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/30495251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102ed0 0x7f6bc8106ec0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0099c0 tx=0x7f6bbc009cd0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/30495251 shutdown_connections 2024-04-02T05:02:47.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/30495251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc81075a0 0x7f6bc810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/30495251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bc8102520 0x7f6bc8102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/30495251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102ed0 0x7f6bc8106ec0 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/30495251 >> 172.21.15.69:0/30495251 conn(0x7f6bc8079360 msgr2=0x7f6bc8079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/30495251 shutdown_connections 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/30495251 wait complete. 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 librados: starting msgr at 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 librados: starting objecter 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 Processor -- start 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- start start 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 librados: setting wanted keys 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 librados: calling monclient init 2024-04-02T05:02:47.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102520 0x7f6bc8074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc8102ed0 0x7f6bc8074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bc81075a0 0x7f6bc8074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc8111a80 con 0x7f6bc8102520 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc8111900 con 0x7f6bc8102ed0 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc8111c00 con 0x7f6bc81075a0 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102520 0x7f6bc8074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bc81075a0 0x7f6bc8074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102520 0x7f6bc8074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47518/0 (socket says 172.21.15.69:47518) 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 -- 172.21.15.69:0/2580562572 learned_addr learned my addr 172.21.15.69:0/2580562572 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc8102ed0 0x7f6bc8074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bc81075a0 msgr2=0x7f6bc8074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bc81075a0 0x7f6bc8074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc8102ed0 msgr2=0x7f6bc8074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc8102ed0 0x7f6bc8074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc81a6630 con 0x7f6bc8102520 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bc81075a0 0x7f6bc8074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc8102ed0 0x7f6bc8074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102520 0x7f6bc8074540 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800eb30 tx=0x7f6bb800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc4ff9700 10 client.?.objecter ms_handle_connect 0x7f6bc8102520 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc4ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb800cc90 con 0x7f6bc8102520 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6bb800ce30 con 0x7f6bc8102520 2024-04-02T05:02:47.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8003e20 con 0x7f6bc8102520 2024-04-02T05:02:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc81a6920 con 0x7f6bc8102520 2024-04-02T05:02:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 10 client.33771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc81a6de0 con 0x7f6bc8102520 2024-04-02T05:02:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.439+0000 7f6bcdc1d700 1 librados: init done 2024-04-02T05:02:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6bc804ff00 con 0x7f6bc8102520 2024-04-02T05:02:47.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6bb8003fc0 con 0x7f6bc8102520 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6bb006f5e0 0x7f6bb0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(846..846 src has 74..846) ==== 10165+0+0 (secure 0 0 0) 0x7f6bb808ee10 con 0x7f6bc8102520 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 osd_map(846..846 src has 74..846) 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 3 client.33771.objecter handle_osd_map got epochs [846,846] > 0 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 3 client.33771.objecter handle_osd_map decoding full epoch 846 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6bb006f5e0 0x7f6bb0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 20 client.33771.objecter dump_active .. 0 homeless 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v846) ==== 54+0+16 (secure 0 0 0) 0x7f6bb8094460 con 0x7f6bc8102520 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 1 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6bc8062e50 0x7f6bc8063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 1 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 0 1 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 4 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 1 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bc81aa1d0 con 0x7f6bc8062e50 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6bb006f5e0 0x7f6bb0071aa0 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc000c00 tx=0x7f6bbc005ea0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bb006f5e0 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6bc8062e50 0x7f6bc8063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc8062e50 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6bc8062e50 0x7f6bc8063230 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc81aa1d0 con 0x7f6bc8062e50 2024-04-02T05:02:47.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 1: 0 ) 2024-04-02T05:02:47.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 1 = system:0 2024-04-02T05:02:47.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 4 2024-04-02T05:02:47.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.443+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 0 1 2024-04-02T05:02:47.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:47.470 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:47 smithi098 bash[23790]: cluster 2024-04-02T05:02:46.347073+0000 mgr.y (mgr.14538) 737 : cluster 0 pgmap v1285: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 5 op/s 2024-04-02T05:02:47.470 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:47 smithi098 bash[23790]: audit 2024-04-02T05:02:47.156317+0000 mon.a (mon.0) 3891 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T05:02:47.470 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:47 smithi098 bash[23790]: cluster 2024-04-02T05:02:47.160891+0000 mon.a (mon.0) 3892 : cluster 0 osdmap e846: 8 total, 8 up, 7 in 2024-04-02T05:02:47.470 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:47 smithi098 bash[23790]: audit 2024-04-02T05:02:47.161454+0000 mon.a (mon.0) 3893 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-02T05:02:47.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 2 2024-04-02T05:02:47.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 2 2024-04-02T05:02:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc8075a70 osd=0 3 2024-04-02T05:02:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 0 2 2024-04-02T05:02:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 4 2024-04-02T05:02:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 2 2024-04-02T05:02:47.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc80628d0 con 0x7f6bc8062e50 2024-04-02T05:02:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc80628d0 con 0x7f6bc8062e50 2024-04-02T05:02:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 4 2024-04-02T05:02:47.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 0 2 2024-04-02T05:02:47.476 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:02:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 3 2024-04-02T05:02:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6bc802c9b0 0x7f6bc802cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 1 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 4 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 3 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.467+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6bc8109ec0 con 0x7f6bc802c9b0 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6bc802c9b0 0x7f6bc802cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc802c9b0 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6bc802c9b0 0x7f6bc802cdb0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc8109ec0 con 0x7f6bc802c9b0 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 3: 0 ) 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 3 = system:0 2024-04-02T05:02:47.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 4 2024-04-02T05:02:47.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.471+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 1 3 2024-04-02T05:02:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 4 2024-04-02T05:02:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 4 2024-04-02T05:02:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc8109840 osd=1 3 2024-04-02T05:02:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 1 4 2024-04-02T05:02:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 4 2024-04-02T05:02:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 4 2024-04-02T05:02:47.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.487+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc81092c0 con 0x7f6bc802c9b0 2024-04-02T05:02:47.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc81092c0 con 0x7f6bc802c9b0 2024-04-02T05:02:47.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 4 2024-04-02T05:02:47.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 1 4 2024-04-02T05:02:47.496 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:02:47.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 5 2024-04-02T05:02:47.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6bc81ac390 0x7f6bc81ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 5 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 2 5 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 4 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 5 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6bc802dfd0 con 0x7f6bc81ac390 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6bc81ac390 0x7f6bc81ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc81ac390 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6bc81ac390 0x7f6bc81ae780 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc802dfd0 con 0x7f6bc81ac390 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 5: 0 ) 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 5 = system:0 2024-04-02T05:02:47.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 4 2024-04-02T05:02:47.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.491+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 2 5 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 6 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 6 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81096c0 osd=2 3 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 2 6 2024-04-02T05:02:47.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 4 2024-04-02T05:02:47.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 6 2024-04-02T05:02:47.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.507+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc802d600 con 0x7f6bc81ac390 2024-04-02T05:02:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc802d600 con 0x7f6bc81ac390 2024-04-02T05:02:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 4 2024-04-02T05:02:47.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 2 6 2024-04-02T05:02:47.517 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 7 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6bc81afe40 0x7f6bc81b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 7 2024-04-02T05:02:47.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 3 7 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 4 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 7 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6bc81b07e0 con 0x7f6bc81afe40 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6bc81afe40 0x7f6bc81b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc81afe40 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6bc81afe40 0x7f6bc81b02a0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc81b07e0 con 0x7f6bc81afe40 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 7: 0 ) 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 7 = system:0 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 4 2024-04-02T05:02:47.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.511+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 3 7 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 8 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 8 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc802d890 osd=3 3 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 3 8 2024-04-02T05:02:47.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 4 2024-04-02T05:02:47.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 8 2024-04-02T05:02:47.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc81b07e0 con 0x7f6bc81afe40 2024-04-02T05:02:47.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc81b07e0 con 0x7f6bc81afe40 2024-04-02T05:02:47.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 4 2024-04-02T05:02:47.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.531+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 3 8 2024-04-02T05:02:47.540 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:02:47.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 9 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6bc81b3620 0x7f6bc81b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 9 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 4 9 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 4 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 9 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6bc81b0f50 con 0x7f6bc81b3620 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6bc81b3620 0x7f6bc81b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc81b3620 2024-04-02T05:02:47.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6bc81b3620 0x7f6bc81b5a10 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc81b0f50 con 0x7f6bc81b3620 2024-04-02T05:02:47.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 9: 0 ) 2024-04-02T05:02:47.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 9 = system:0 2024-04-02T05:02:47.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 4 2024-04-02T05:02:47.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.535+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 4 9 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 10 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 10 2024-04-02T05:02:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81b07e0 osd=4 3 2024-04-02T05:02:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 4 10 2024-04-02T05:02:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 4 2024-04-02T05:02:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 10 2024-04-02T05:02:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.551+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc81b0f50 con 0x7f6bc81b3620 2024-04-02T05:02:47.562 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:02:47.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc81b0f50 con 0x7f6bc81b3620 2024-04-02T05:02:47.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 4 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 4 10 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 11 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6bc81b6460 0x7f6bc81b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 11 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 5 11 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 4 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 11 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6bc81bb860 con 0x7f6bc81b6460 2024-04-02T05:02:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6bc81b6460 0x7f6bc81b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc81b6460 2024-04-02T05:02:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.555+0000 7f6bc6ffd700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6bc81b6460 0x7f6bc81b8850 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.559+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc81bb860 con 0x7f6bc81b6460 2024-04-02T05:02:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.559+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 11: 0 ) 2024-04-02T05:02:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.559+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 11 = system:0 2024-04-02T05:02:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.559+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 4 2024-04-02T05:02:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.559+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 5 11 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 12 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 12 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81b0f50 osd=5 3 2024-04-02T05:02:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 5 12 2024-04-02T05:02:47.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 4 2024-04-02T05:02:47.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 12 2024-04-02T05:02:47.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.579+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc81ba650 con 0x7f6bc81b6460 2024-04-02T05:02:47.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc81ba650 con 0x7f6bc81b6460 2024-04-02T05:02:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 4 2024-04-02T05:02:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 5 12 2024-04-02T05:02:47.590 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 13 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6bc81bad70 0x7f6bc81bb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 13 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 6 13 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 4 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 13 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6bc81bec80 con 0x7f6bc81bad70 2024-04-02T05:02:47.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6bc81bad70 0x7f6bc81bb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc81bad70 2024-04-02T05:02:47.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.583+0000 7f6bc7fff700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6bc81bad70 0x7f6bc81bb1b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.587+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc81bec80 con 0x7f6bc81bad70 2024-04-02T05:02:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.587+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 13: 0 ) 2024-04-02T05:02:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.587+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 13 = system:0 2024-04-02T05:02:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.587+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 4 2024-04-02T05:02:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.587+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 6 13 2024-04-02T05:02:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 14 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 14 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81ba650 osd=6 3 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 6 14 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 4 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 14 2024-04-02T05:02:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc81bdab0 con 0x7f6bc81bad70 2024-04-02T05:02:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc81bdab0 con 0x7f6bc81bad70 2024-04-02T05:02:47.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 4 2024-04-02T05:02:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 6 14 2024-04-02T05:02:47.615 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 15 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6bc81be1d0 0x7f6bc81be630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 15 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 7 15 2024-04-02T05:02:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 4 2024-04-02T05:02:47.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 15 2024-04-02T05:02:47.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6bc81c1ff0 con 0x7f6bc81be1d0 2024-04-02T05:02:47.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.607+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6bc81be1d0 0x7f6bc81be630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc4ff9700 10 client.33771.objecter ms_handle_connect 0x7f6bc81be1d0 2024-04-02T05:02:47.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc77fe700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6bc81be1d0 0x7f6bc81be630 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bc81c1ff0 con 0x7f6bc81be1d0 2024-04-02T05:02:47.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 15: 0 ) 2024-04-02T05:02:47.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 15 = system:0 2024-04-02T05:02:47.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 4 2024-04-02T05:02:47.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.611+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 7 15 2024-04-02T05:02:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:47.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 10 client.33771.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign -1 16 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 20 client.33771.objecter _get_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_remove -1 16 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 20 client.33771.objecter get_session s=0x7f6bc81bdab0 osd=7 3 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 15 client.33771.objecter _session_command_op_assign 7 16 2024-04-02T05:02:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 4 2024-04-02T05:02:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 10 client.33771.objecter _send_command 16 2024-04-02T05:02:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.643+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6bc81b8e80 con 0x7f6bc81be1d0 2024-04-02T05:02:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc4ff9700 1 -- 172.21.15.69:0/2580562572 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6bc81b8e80 con 0x7f6bc81be1d0 2024-04-02T05:02:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc4ff9700 10 client.33771.objecter ms_dispatch 0x7f6bc8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:02:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc4ff9700 10 client.33771.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:02:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc4ff9700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 4 2024-04-02T05:02:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc4ff9700 15 client.33771.objecter _session_command_op_remove 7 16 2024-04-02T05:02:47.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:02:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 librados: watch_flush enter 2024-04-02T05:02:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 librados: watch_flush exit 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.0 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6bc8062e50 msgr2=0x7f6bc8063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f6bc8062e50 0x7f6bc8063230 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8075a70 osd=0 1 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.1 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6bc802c9b0 msgr2=0x7f6bc802cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f6bc802c9b0 0x7f6bc802cdb0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc8109840 osd=1 1 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.2 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6bc81ac390 msgr2=0x7f6bc81ae780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f6bc81ac390 0x7f6bc81ae780 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81096c0 osd=2 1 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.3 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6bc81afe40 msgr2=0x7f6bc81b02a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f6bc81afe40 0x7f6bc81b02a0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc802d890 osd=3 1 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.4 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6bc81b3620 msgr2=0x7f6bc81b5a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f6bc81b3620 0x7f6bc81b5a10 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b07e0 osd=4 1 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc7fff700 1 -- 172.21.15.69:0/2580562572 reap_dead start 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.5 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6bc81b6460 msgr2=0x7f6bc81b8850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f6bc81b6460 0x7f6bc81b8850 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc7fff700 1 -- 172.21.15.69:0/2580562572 reap_dead start 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc7fff700 1 -- 172.21.15.69:0/2580562572 reap_dead start 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81b0f50 osd=5 1 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.6 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6bc81bad70 msgr2=0x7f6bc81bb1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f6bc81bad70 0x7f6bc81bb1b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81ba650 osd=6 1 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter close_session for osd.7 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6bc81be1d0 msgr2=0x7f6bc81be630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f6bc81be1d0 0x7f6bc81be630 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter put_session s=0x7f6bc81bdab0 osd=7 1 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 20 client.33771.objecter shutdown clearing up homeless session... 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 10 client.33771.objecter successfully canceled tick 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6bb006f5e0 msgr2=0x7f6bb0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6bb006f5e0 0x7f6bb0071aa0 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc000c00 tx=0x7f6bbc005ea0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102520 msgr2=0x7f6bc8074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 --2- 172.21.15.69:0/2580562572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bc8102520 0x7f6bc8074540 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800eb30 tx=0x7f6bb800ee40 comp rx=0 tx=0).stop 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bc7fff700 1 -- 172.21.15.69:0/2580562572 reap_dead start 2024-04-02T05:02:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 shutdown_connections 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 >> 172.21.15.69:0/2580562572 conn(0x7f6bc8079360 msgr2=0x7f6bc8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 shutdown_connections 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 -- 172.21.15.69:0/2580562572 wait complete. 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.647+0000 7f6bcdc1d700 1 librados: shutdown 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:02:47.659 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:02:47.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:47.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3833148690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c1011a0 msgr2=0x7f6c3c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3833148690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c1011a0 0x7f6c3c101620 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009990 tx=0x7f6c2c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3833148690 shutdown_connections 2024-04-02T05:02:47.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3833148690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3833148690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c107220 0x7f6c3c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3833148690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c1011a0 0x7f6c3c101620 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3833148690 >> 172.21.15.69:0/3833148690 conn(0x7f6c3c0771e0 msgr2=0x7f6c3c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:47.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3833148690 shutdown_connections 2024-04-02T05:02:47.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3833148690 wait complete. 2024-04-02T05:02:47.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 librados: starting msgr at 2024-04-02T05:02:47.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 librados: starting objecter 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 Processor -- start 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- start start 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 librados: setting wanted keys 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 librados: calling monclient init 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c1011a0 0x7f6c3c102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c103360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c107220 0x7f6c3c1038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c110970 con 0x7f6c3c101b60 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1107f0 con 0x7f6c3c1011a0 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c110af0 con 0x7f6c3c107220 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c103360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c107220 0x7f6c3c1038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c103360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47538/0 (socket says 172.21.15.69:47538) 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 -- 172.21.15.69:0/3521190899 learned_addr learned my addr 172.21.15.69:0/3521190899 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3affd700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c1011a0 0x7f6c3c102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 -- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c107220 msgr2=0x7f6c3c1038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c107220 0x7f6c3c1038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 -- 172.21.15.69:0/3521190899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c1011a0 msgr2=0x7f6c3c102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c1011a0 0x7f6c3c102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 -- 172.21.15.69:0/3521190899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c1a6560 con 0x7f6c3c101b60 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3affd700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c1011a0 0x7f6c3c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c3a7fc700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c103360 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0057f0 tx=0x7f6c2c00bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c23fff700 10 client.?.objecter ms_handle_connect 0x7f6c3c101b60 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c23fff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c026070 con 0x7f6c3c101b60 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c1a67f0 con 0x7f6c3c101b60 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 10 client.33777.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:02:47.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c1a6d20 con 0x7f6c3c101b60 2024-04-02T05:02:47.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 librados: init done 2024-04-02T05:02:47.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6c2c021410 con 0x7f6c3c101b60 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c017650 con 0x7f6c3c101b60 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.723+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3c102860 con 0x7f6c3c101b60 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6c2c029070 con 0x7f6c3c101b60 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c2406f4b0 0x7f6c24071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(846..846 src has 74..846) ==== 10165+0+0 (secure 0 0 0) 0x7f6c2c08fad0 con 0x7f6c3c101b60 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 10 client.33777.objecter ms_dispatch 0x7f6c3c001550 osd_map(846..846 src has 74..846) 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 3 client.33777.objecter handle_osd_map got epochs [846,846] > 0 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 3 client.33777.objecter handle_osd_map decoding full epoch 846 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c3affd700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c2406f4b0 0x7f6c24071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:02:47.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 20 client.33777.objecter dump_active .. 0 homeless 2024-04-02T05:02:47.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c3affd700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c2406f4b0 0x7f6c24071970 secure :-1 s=READY pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007900 tx=0x7f6c30008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:02:47.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.727+0000 7f6c23fff700 10 client.33777.objecter ms_handle_connect 0x7f6c2406f4b0 2024-04-02T05:02:47.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.731+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c2c013070 con 0x7f6c3c101b60 2024-04-02T05:02:47.741 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:47.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:47.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:47.955+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f6c3c104350 con 0x7f6c3c101b60 2024-04-02T05:02:47.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:47.997 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:47.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:48.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:48.123 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c23fff700 1 -- 172.21.15.69:0/3521190899 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v847) ==== 97+0+0 (secure 0 0 0) 0x7f6c2c05c720 con 0x7f6c3c101b60 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 10 librados: watch_flush enter 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 10 librados: watch_flush exit 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 20 client.33777.objecter shutdown clearing up homeless session... 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 10 client.33777.objecter successfully canceled tick 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c2406f4b0 msgr2=0x7f6c24071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c2406f4b0 0x7f6c24071970 secure :-1 s=READY pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007900 tx=0x7f6c30008040 comp rx=0 tx=0).stop 2024-04-02T05:02:48.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 msgr2=0x7f6c3c103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:02:48.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c103360 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0057f0 tx=0x7f6c2c00bfa0 comp rx=0 tx=0).stop 2024-04-02T05:02:48.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 shutdown_connections 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6c2406f4b0 0x7f6c24071970 secure :-1 s=CLOSED pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007900 tx=0x7f6c30008040 comp rx=0 tx=0).stop 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c107220 0x7f6c3c1038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c3c1011a0 0x7f6c3c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 --2- 172.21.15.69:0/3521190899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c101b60 0x7f6c3c103360 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 >> 172.21.15.69:0/3521190899 conn(0x7f6c3c0771e0 msgr2=0x7f6c3c0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 shutdown_connections 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 -- 172.21.15.69:0/3521190899 wait complete. 2024-04-02T05:02:48.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:02:48.155+0000 7f6c417f3700 1 librados: shutdown 2024-04-02T05:02:48.166 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is set 2024-04-02T05:02:48.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:48.243 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:48.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: audit 2024-04-02T05:02:47.448945+0000 mon.a (mon.0) 3894 : audit 0 from='client.? 172.21.15.69:0/2580562572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:48.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: audit 2024-04-02T05:02:47.962332+0000 mon.a (mon.0) 3895 : audit 1 from='client.? 172.21.15.69:0/3521190899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:48.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: cluster 2024-04-02T05:02:48.156491+0000 mon.a (mon.0) 3896 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:02:48.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: audit 2024-04-02T05:02:48.158079+0000 mon.a (mon.0) 3897 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: audit 2024-04-02T05:02:48.158158+0000 mon.a (mon.0) 3898 : audit 1 from='client.? 172.21.15.69:0/3521190899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: cluster 2024-04-02T05:02:48.161766+0000 mon.a (mon.0) 3899 : cluster 0 osdmap e847: 8 total, 8 up, 7 in 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[28904]: audit 2024-04-02T05:02:48.162734+0000 mon.a (mon.0) 3900 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: audit 2024-04-02T05:02:47.448945+0000 mon.a (mon.0) 3894 : audit 0 from='client.? 172.21.15.69:0/2580562572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: audit 2024-04-02T05:02:47.962332+0000 mon.a (mon.0) 3895 : audit 1 from='client.? 172.21.15.69:0/3521190899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: cluster 2024-04-02T05:02:48.156491+0000 mon.a (mon.0) 3896 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: audit 2024-04-02T05:02:48.158079+0000 mon.a (mon.0) 3897 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: audit 2024-04-02T05:02:48.158158+0000 mon.a (mon.0) 3898 : audit 1 from='client.? 172.21.15.69:0/3521190899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: cluster 2024-04-02T05:02:48.161766+0000 mon.a (mon.0) 3899 : cluster 0 osdmap e847: 8 total, 8 up, 7 in 2024-04-02T05:02:48.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21528]: audit 2024-04-02T05:02:48.162734+0000 mon.a (mon.0) 3900 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T05:02:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:48.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:48.367 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:48.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: audit 2024-04-02T05:02:47.448945+0000 mon.a (mon.0) 3894 : audit 0 from='client.? 172.21.15.69:0/2580562572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:02:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: audit 2024-04-02T05:02:47.962332+0000 mon.a (mon.0) 3895 : audit 1 from='client.? 172.21.15.69:0/3521190899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:02:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: cluster 2024-04-02T05:02:48.156491+0000 mon.a (mon.0) 3896 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:02:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: audit 2024-04-02T05:02:48.158079+0000 mon.a (mon.0) 3897 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-02T05:02:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: audit 2024-04-02T05:02:48.158158+0000 mon.a (mon.0) 3898 : audit 1 from='client.? 172.21.15.69:0/3521190899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-02T05:02:48.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: cluster 2024-04-02T05:02:48.161766+0000 mon.a (mon.0) 3899 : cluster 0 osdmap e847: 8 total, 8 up, 7 in 2024-04-02T05:02:48.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:48 smithi098 bash[23790]: audit 2024-04-02T05:02:48.162734+0000 mon.a (mon.0) 3900 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-02T05:02:48.494 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:48.597 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:48.623 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:48.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:48.746 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:48.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:48.871 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T05:02:48.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T05:02:48.871 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.3040669151245585 2024-04-02T05:02:48.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 0.3040669151245585 2024-04-02T05:02:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:48.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:48.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:49.009 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:49.028 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:02:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:02:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:02:49.112 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:49.134 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:49.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:49.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[28904]: cluster 2024-04-02T05:02:48.348174+0000 mgr.y (mgr.14538) 738 : cluster 0 pgmap v1288: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:49.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[28904]: audit 2024-04-02T05:02:49.128422+0000 mon.b (mon.1) 317 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:49.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[28904]: audit 2024-04-02T05:02:49.160194+0000 mon.a (mon.0) 3901 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-02T05:02:49.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[28904]: cluster 2024-04-02T05:02:49.163329+0000 mon.a (mon.0) 3902 : cluster 0 osdmap e848: 8 total, 8 up, 7 in 2024-04-02T05:02:49.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[21528]: cluster 2024-04-02T05:02:48.348174+0000 mgr.y (mgr.14538) 738 : cluster 0 pgmap v1288: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:49.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[21528]: audit 2024-04-02T05:02:49.128422+0000 mon.b (mon.1) 317 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:49.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[21528]: audit 2024-04-02T05:02:49.160194+0000 mon.a (mon.0) 3901 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-02T05:02:49.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:49 smithi069 bash[21528]: cluster 2024-04-02T05:02:49.163329+0000 mon.a (mon.0) 3902 : cluster 0 osdmap e848: 8 total, 8 up, 7 in 2024-04-02T05:02:49.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:49.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:49 smithi098 bash[23790]: cluster 2024-04-02T05:02:48.348174+0000 mgr.y (mgr.14538) 738 : cluster 0 pgmap v1288: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:49.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:49 smithi098 bash[23790]: audit 2024-04-02T05:02:49.128422+0000 mon.b (mon.1) 317 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:02:49.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:49 smithi098 bash[23790]: audit 2024-04-02T05:02:49.160194+0000 mon.a (mon.0) 3901 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-02T05:02:49.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:49 smithi098 bash[23790]: cluster 2024-04-02T05:02:49.163329+0000 mon.a (mon.0) 3902 : cluster 0 osdmap e848: 8 total, 8 up, 7 in 2024-04-02T05:02:49.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:49.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:49.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:49.646 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:49.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:49.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:49.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:50.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:50.032 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:50.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:50.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:50.156 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:50.260 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:50.283 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:50.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:50.411 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:50.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:50.543 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:50.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:02:50.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:50.810 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:50.831 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:50.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:50.960 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:50.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:51.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:51.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:51.202 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:51 smithi069 bash[28904]: cluster 2024-04-02T05:02:50.118392+0000 mon.a (mon.0) 3903 : cluster 0 osdmap e849: 8 total, 8 up, 7 in 2024-04-02T05:02:51.202 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:51 smithi069 bash[28904]: audit 2024-04-02T05:02:50.284632+0000 mon.a (mon.0) 3904 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:51.202 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:51 smithi069 bash[28904]: cluster 2024-04-02T05:02:50.348973+0000 mgr.y (mgr.14538) 739 : cluster 0 pgmap v1291: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:51.202 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:51 smithi069 bash[21528]: cluster 2024-04-02T05:02:50.118392+0000 mon.a (mon.0) 3903 : cluster 0 osdmap e849: 8 total, 8 up, 7 in 2024-04-02T05:02:51.202 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:51 smithi069 bash[21528]: audit 2024-04-02T05:02:50.284632+0000 mon.a (mon.0) 3904 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:51.202 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:51 smithi069 bash[21528]: cluster 2024-04-02T05:02:50.348973+0000 mgr.y (mgr.14538) 739 : cluster 0 pgmap v1291: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:51.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:51.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:51 smithi098 bash[23790]: cluster 2024-04-02T05:02:50.118392+0000 mon.a (mon.0) 3903 : cluster 0 osdmap e849: 8 total, 8 up, 7 in 2024-04-02T05:02:51.328 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:51 smithi098 bash[23790]: audit 2024-04-02T05:02:50.284632+0000 mon.a (mon.0) 3904 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:51.328 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:51 smithi098 bash[23790]: cluster 2024-04-02T05:02:50.348973+0000 mgr.y (mgr.14538) 739 : cluster 0 pgmap v1291: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:51.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T05:02:51.446 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:51.505 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:51.608 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:51.628 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:51.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:51.756 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:51.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:51.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:51.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:52.015 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:52.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:52.149 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:52.178 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:52.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:52.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:52 smithi069 bash[21528]: audit 2024-04-02T05:02:51.121021+0000 mon.a (mon.0) 3905 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:02:52.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:52 smithi069 bash[21528]: cluster 2024-04-02T05:02:51.132382+0000 mon.a (mon.0) 3906 : cluster 0 osdmap e850: 8 total, 8 up, 7 in 2024-04-02T05:02:52.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:52 smithi069 bash[21528]: audit 2024-04-02T05:02:51.133694+0000 mon.a (mon.0) 3907 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:52.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:52 smithi069 bash[28904]: audit 2024-04-02T05:02:51.121021+0000 mon.a (mon.0) 3905 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:02:52.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:52 smithi069 bash[28904]: cluster 2024-04-02T05:02:51.132382+0000 mon.a (mon.0) 3906 : cluster 0 osdmap e850: 8 total, 8 up, 7 in 2024-04-02T05:02:52.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:52 smithi069 bash[28904]: audit 2024-04-02T05:02:51.133694+0000 mon.a (mon.0) 3907 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:52.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:52.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:52 smithi098 bash[23790]: audit 2024-04-02T05:02:51.121021+0000 mon.a (mon.0) 3905 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:02:52.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:52 smithi098 bash[23790]: cluster 2024-04-02T05:02:51.132382+0000 mon.a (mon.0) 3906 : cluster 0 osdmap e850: 8 total, 8 up, 7 in 2024-04-02T05:02:52.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:52 smithi098 bash[23790]: audit 2024-04-02T05:02:51.133694+0000 mon.a (mon.0) 3907 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:52.409 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:52.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:52.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:52.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:52.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:52.877 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:52.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:52.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:53.006 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:53.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:53.138 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:53.195 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:53.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:53.273 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:53.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:53.400 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:53.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:53.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:53.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:53.644 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[28904]: audit 2024-04-02T05:02:52.133431+0000 mon.a (mon.0) 3908 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]': finished 2024-04-02T05:02:53.644 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[28904]: cluster 2024-04-02T05:02:52.141700+0000 mon.a (mon.0) 3909 : cluster 0 osdmap e851: 8 total, 8 up, 7 in 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[28904]: audit 2024-04-02T05:02:52.197743+0000 mon.a (mon.0) 3910 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[28904]: audit 2024-04-02T05:02:52.198336+0000 mon.a (mon.0) 3911 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[28904]: cluster 2024-04-02T05:02:52.349774+0000 mgr.y (mgr.14538) 740 : cluster 0 pgmap v1294: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[21528]: audit 2024-04-02T05:02:52.133431+0000 mon.a (mon.0) 3908 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]': finished 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[21528]: cluster 2024-04-02T05:02:52.141700+0000 mon.a (mon.0) 3909 : cluster 0 osdmap e851: 8 total, 8 up, 7 in 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[21528]: audit 2024-04-02T05:02:52.197743+0000 mon.a (mon.0) 3910 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[21528]: audit 2024-04-02T05:02:52.198336+0000 mon.a (mon.0) 3911 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:53.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:53 smithi069 bash[21528]: cluster 2024-04-02T05:02:52.349774+0000 mgr.y (mgr.14538) 740 : cluster 0 pgmap v1294: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:53.659 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:53.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:53.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:53.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:02:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:02:53.774 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:53 smithi098 bash[23790]: audit 2024-04-02T05:02:52.133431+0000 mon.a (mon.0) 3908 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]': finished 2024-04-02T05:02:53.774 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:53 smithi098 bash[23790]: cluster 2024-04-02T05:02:52.141700+0000 mon.a (mon.0) 3909 : cluster 0 osdmap e851: 8 total, 8 up, 7 in 2024-04-02T05:02:53.774 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:53 smithi098 bash[23790]: audit 2024-04-02T05:02:52.197743+0000 mon.a (mon.0) 3910 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:02:53.774 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:53 smithi098 bash[23790]: audit 2024-04-02T05:02:52.198336+0000 mon.a (mon.0) 3911 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-142"}]: dispatch 2024-04-02T05:02:53.774 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:53 smithi098 bash[23790]: cluster 2024-04-02T05:02:52.349774+0000 mgr.y (mgr.14538) 740 : cluster 0 pgmap v1294: 224 pgs: 224 active+clean; 4.4 MiB data, 740 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:02:53.790 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:53.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:53.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:54.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:54.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:54.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:54.323 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:54.426 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:54.453 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:54.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:54.584 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:54.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:54.817 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:54.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[21528]: cluster 2024-04-02T05:02:53.133947+0000 mon.a (mon.0) 3912 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:54.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[21528]: cluster 2024-04-02T05:02:53.141464+0000 mon.a (mon.0) 3913 : cluster 0 osdmap e852: 8 total, 8 up, 7 in 2024-04-02T05:02:54.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[21528]: audit 2024-04-02T05:02:53.740786+0000 mgr.y (mgr.14538) 741 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:54.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[21528]: cluster 2024-04-02T05:02:54.147492+0000 mon.a (mon.0) 3914 : cluster 0 osdmap e853: 8 total, 8 up, 7 in 2024-04-02T05:02:54.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[21528]: audit 2024-04-02T05:02:54.163423+0000 mon.a (mon.0) 3915 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:54.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[28904]: cluster 2024-04-02T05:02:53.133947+0000 mon.a (mon.0) 3912 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:54.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[28904]: cluster 2024-04-02T05:02:53.141464+0000 mon.a (mon.0) 3913 : cluster 0 osdmap e852: 8 total, 8 up, 7 in 2024-04-02T05:02:54.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[28904]: audit 2024-04-02T05:02:53.740786+0000 mgr.y (mgr.14538) 741 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:54.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[28904]: cluster 2024-04-02T05:02:54.147492+0000 mon.a (mon.0) 3914 : cluster 0 osdmap e853: 8 total, 8 up, 7 in 2024-04-02T05:02:54.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:54 smithi069 bash[28904]: audit 2024-04-02T05:02:54.163423+0000 mon.a (mon.0) 3915 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:54.838 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:54.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:54.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:54 smithi098 bash[23790]: cluster 2024-04-02T05:02:53.133947+0000 mon.a (mon.0) 3912 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:02:54.953 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:54 smithi098 bash[23790]: cluster 2024-04-02T05:02:53.141464+0000 mon.a (mon.0) 3913 : cluster 0 osdmap e852: 8 total, 8 up, 7 in 2024-04-02T05:02:54.953 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:54 smithi098 bash[23790]: audit 2024-04-02T05:02:53.740786+0000 mgr.y (mgr.14538) 741 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:02:54.953 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:54 smithi098 bash[23790]: cluster 2024-04-02T05:02:54.147492+0000 mon.a (mon.0) 3914 : cluster 0 osdmap e853: 8 total, 8 up, 7 in 2024-04-02T05:02:54.953 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:54 smithi098 bash[23790]: audit 2024-04-02T05:02:54.163423+0000 mon.a (mon.0) 3915 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:02:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:54.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:55.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:55.098 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:55.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:55.219 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:55.220 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:02:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:55.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:55.353 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:55.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:55.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:55.616 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:55.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:55 smithi098 bash[23790]: cluster 2024-04-02T05:02:54.350283+0000 mgr.y (mgr.14538) 742 : cluster 0 pgmap v1297: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:02:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:55 smithi098 bash[23790]: cluster 2024-04-02T05:02:54.437753+0000 osd.1 (osd.1) 415 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:55 smithi098 bash[23790]: cluster 2024-04-02T05:02:54.438177+0000 osd.1 (osd.1) 416 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:55 smithi098 bash[23790]: audit 2024-04-02T05:02:55.144969+0000 mon.a (mon.0) 3916 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:55 smithi098 bash[23790]: cluster 2024-04-02T05:02:55.149640+0000 mon.a (mon.0) 3917 : cluster 0 osdmap e854: 8 total, 8 up, 7 in 2024-04-02T05:02:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:55 smithi098 bash[23790]: audit 2024-04-02T05:02:55.184487+0000 mon.a (mon.0) 3918 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:55.851 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:55.862 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[21528]: cluster 2024-04-02T05:02:54.350283+0000 mgr.y (mgr.14538) 742 : cluster 0 pgmap v1297: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[21528]: cluster 2024-04-02T05:02:54.437753+0000 osd.1 (osd.1) 415 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[21528]: cluster 2024-04-02T05:02:54.438177+0000 osd.1 (osd.1) 416 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[21528]: audit 2024-04-02T05:02:55.144969+0000 mon.a (mon.0) 3916 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[21528]: cluster 2024-04-02T05:02:55.149640+0000 mon.a (mon.0) 3917 : cluster 0 osdmap e854: 8 total, 8 up, 7 in 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[21528]: audit 2024-04-02T05:02:55.184487+0000 mon.a (mon.0) 3918 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[28904]: cluster 2024-04-02T05:02:54.350283+0000 mgr.y (mgr.14538) 742 : cluster 0 pgmap v1297: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[28904]: cluster 2024-04-02T05:02:54.437753+0000 osd.1 (osd.1) 415 : cluster 0 purged_snaps scrub starts 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[28904]: cluster 2024-04-02T05:02:54.438177+0000 osd.1 (osd.1) 416 : cluster 0 purged_snaps scrub ok 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[28904]: audit 2024-04-02T05:02:55.144969+0000 mon.a (mon.0) 3916 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[28904]: cluster 2024-04-02T05:02:55.149640+0000 mon.a (mon.0) 3917 : cluster 0 osdmap e854: 8 total, 8 up, 7 in 2024-04-02T05:02:55.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:55 smithi069 bash[28904]: audit 2024-04-02T05:02:55.184487+0000 mon.a (mon.0) 3918 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:02:55.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:55.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:55.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:56.006 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:56.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:56.130 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:56.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:56.261 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:56.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:56.387 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:56.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:56.514 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:56.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:56.645 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:56.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:56.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:56.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:57.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:57.037 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:57.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:57.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[28904]: audit 2024-04-02T05:02:56.147115+0000 mon.a (mon.0) 3919 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[28904]: cluster 2024-04-02T05:02:56.151405+0000 mon.a (mon.0) 3920 : cluster 0 osdmap e855: 8 total, 8 up, 7 in 2024-04-02T05:02:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[28904]: audit 2024-04-02T05:02:56.153840+0000 mon.a (mon.0) 3921 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:02:57.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[28904]: cluster 2024-04-02T05:02:56.351039+0000 mgr.y (mgr.14538) 743 : cluster 0 pgmap v1300: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:02:57.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[21528]: audit 2024-04-02T05:02:56.147115+0000 mon.a (mon.0) 3919 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:57.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[21528]: cluster 2024-04-02T05:02:56.151405+0000 mon.a (mon.0) 3920 : cluster 0 osdmap e855: 8 total, 8 up, 7 in 2024-04-02T05:02:57.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[21528]: audit 2024-04-02T05:02:56.153840+0000 mon.a (mon.0) 3921 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:02:57.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:57 smithi069 bash[21528]: cluster 2024-04-02T05:02:56.351039+0000 mgr.y (mgr.14538) 743 : cluster 0 pgmap v1300: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:02:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:57.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:57.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:57 smithi098 bash[23790]: audit 2024-04-02T05:02:56.147115+0000 mon.a (mon.0) 3919 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:02:57.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:57 smithi098 bash[23790]: cluster 2024-04-02T05:02:56.151405+0000 mon.a (mon.0) 3920 : cluster 0 osdmap e855: 8 total, 8 up, 7 in 2024-04-02T05:02:57.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:57 smithi098 bash[23790]: audit 2024-04-02T05:02:56.153840+0000 mon.a (mon.0) 3921 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:02:57.413 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:57 smithi098 bash[23790]: cluster 2024-04-02T05:02:56.351039+0000 mgr.y (mgr.14538) 743 : cluster 0 pgmap v1300: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:02:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:57.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:57.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:57.690 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:57.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:57.821 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:57.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:57.949 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:57.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:58.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:58.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:58.210 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21528]: audit 2024-04-02T05:02:57.149766+0000 mon.a (mon.0) 3922 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-144"}]': finished 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21528]: cluster 2024-04-02T05:02:57.157412+0000 mon.a (mon.0) 3923 : cluster 0 osdmap e856: 8 total, 8 up, 7 in 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21528]: audit 2024-04-02T05:02:57.161089+0000 mon.a (mon.0) 3924 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-144", "mode": "readproxy"}]: dispatch 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21528]: cluster 2024-04-02T05:02:58.150462+0000 mon.a (mon.0) 3925 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21528]: audit 2024-04-02T05:02:58.153045+0000 mon.a (mon.0) 3926 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-144", "mode": "readproxy"}]': finished 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21528]: cluster 2024-04-02T05:02:58.160500+0000 mon.a (mon.0) 3927 : cluster 0 osdmap e857: 8 total, 8 up, 7 in 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[28904]: audit 2024-04-02T05:02:57.149766+0000 mon.a (mon.0) 3922 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-144"}]': finished 2024-04-02T05:02:58.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[28904]: cluster 2024-04-02T05:02:57.157412+0000 mon.a (mon.0) 3923 : cluster 0 osdmap e856: 8 total, 8 up, 7 in 2024-04-02T05:02:58.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[28904]: audit 2024-04-02T05:02:57.161089+0000 mon.a (mon.0) 3924 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-144", "mode": "readproxy"}]: dispatch 2024-04-02T05:02:58.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[28904]: cluster 2024-04-02T05:02:58.150462+0000 mon.a (mon.0) 3925 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:58.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[28904]: audit 2024-04-02T05:02:58.153045+0000 mon.a (mon.0) 3926 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-144", "mode": "readproxy"}]': finished 2024-04-02T05:02:58.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[28904]: cluster 2024-04-02T05:02:58.160500+0000 mon.a (mon.0) 3927 : cluster 0 osdmap e857: 8 total, 8 up, 7 in 2024-04-02T05:02:58.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:58.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:58 smithi098 bash[23790]: audit 2024-04-02T05:02:57.149766+0000 mon.a (mon.0) 3922 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-144"}]': finished 2024-04-02T05:02:58.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:58 smithi098 bash[23790]: cluster 2024-04-02T05:02:57.157412+0000 mon.a (mon.0) 3923 : cluster 0 osdmap e856: 8 total, 8 up, 7 in 2024-04-02T05:02:58.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:58 smithi098 bash[23790]: audit 2024-04-02T05:02:57.161089+0000 mon.a (mon.0) 3924 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-144", "mode": "readproxy"}]: dispatch 2024-04-02T05:02:58.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:58 smithi098 bash[23790]: cluster 2024-04-02T05:02:58.150462+0000 mon.a (mon.0) 3925 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:02:58.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:58 smithi098 bash[23790]: audit 2024-04-02T05:02:58.153045+0000 mon.a (mon.0) 3926 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-144", "mode": "readproxy"}]': finished 2024-04-02T05:02:58.324 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:58 smithi098 bash[23790]: cluster 2024-04-02T05:02:58.160500+0000 mon.a (mon.0) 3927 : cluster 0 osdmap e857: 8 total, 8 up, 7 in 2024-04-02T05:02:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:58.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:58.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:58.602 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:02:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:02:58.705 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:58.734 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:02:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:02:58.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:02:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:02:58.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:58.995 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:58.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:59.032 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:02:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:02:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:02:59.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:59.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:59.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:02:59.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:59.370 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:02:59 smithi098 bash[23790]: cluster 2024-04-02T05:02:58.352219+0000 mgr.y (mgr.14538) 744 : cluster 0 pgmap v1303: 224 pgs: 5 unknown, 219 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:59.386 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:02:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:02:59.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:02:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:02:59.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:02:59 smithi069 bash[21528]: cluster 2024-04-02T05:02:58.352219+0000 mgr.y (mgr.14538) 744 : cluster 0 pgmap v1303: 224 pgs: 5 unknown, 219 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:59.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:02:59 smithi069 bash[28904]: cluster 2024-04-02T05:02:58.352219+0000 mgr.y (mgr.14538) 744 : cluster 0 pgmap v1303: 224 pgs: 5 unknown, 219 active+clean; 4.4 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:02:59.620 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:59.647 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:02:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:02:59.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:59.778 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:02:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:02:59.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:02:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:02:59.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:00.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:00.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:00.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:00.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:00.283 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:00 smithi098 bash[23790]: cluster 2024-04-02T05:03:00.110475+0000 mon.a (mon.0) 3928 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:00.296 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:00.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:00.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:00.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:00 smithi069 bash[21528]: cluster 2024-04-02T05:03:00.110475+0000 mon.a (mon.0) 3928 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:00.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:00 smithi069 bash[28904]: cluster 2024-04-02T05:03:00.110475+0000 mon.a (mon.0) 3928 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:00.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:00.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:00.688 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:00.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:00.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:00.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:01.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:01.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:01.204 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:01.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.317 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:01 smithi098 bash[23790]: cluster 2024-04-02T05:03:00.353261+0000 mgr.y (mgr.14538) 745 : cluster 0 pgmap v1304: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 196 B/s wr, 1 op/s 2024-04-02T05:03:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:01.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.461 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:01.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:01 smithi069 bash[21528]: cluster 2024-04-02T05:03:00.353261+0000 mgr.y (mgr.14538) 745 : cluster 0 pgmap v1304: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 196 B/s wr, 1 op/s 2024-04-02T05:03:01.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:01 smithi069 bash[28904]: cluster 2024-04-02T05:03:00.353261+0000 mgr.y (mgr.14538) 745 : cluster 0 pgmap v1304: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 196 B/s wr, 1 op/s 2024-04-02T05:03:01.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:01.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:01.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:01.849 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:01.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:01.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:02.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:02.110 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:02.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:02.241 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:02.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:02.467 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:02.494 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:02.597 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:02.617 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:02.706 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T05:03:02.706 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:03:02.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:02.742 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:02.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:02.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:02.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:02.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:03.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:03.132 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:03.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:03.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:03.391 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:03.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:03.505 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:03 smithi069 bash[28904]: cluster 2024-04-02T05:03:02.353784+0000 mgr.y (mgr.14538) 746 : cluster 0 pgmap v1305: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 990 B/s rd, 165 B/s wr, 1 op/s 2024-04-02T05:03:03.505 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:03 smithi069 bash[28904]: cluster 2024-04-02T05:03:03.071307+0000 osd.2 (osd.2) 362 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:03.505 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:03 smithi069 bash[28904]: cluster 2024-04-02T05:03:03.071775+0000 osd.2 (osd.2) 363 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:03.505 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:03 smithi069 bash[21528]: cluster 2024-04-02T05:03:02.353784+0000 mgr.y (mgr.14538) 746 : cluster 0 pgmap v1305: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 990 B/s rd, 165 B/s wr, 1 op/s 2024-04-02T05:03:03.505 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:03 smithi069 bash[21528]: cluster 2024-04-02T05:03:03.071307+0000 osd.2 (osd.2) 362 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:03.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:03 smithi069 bash[21528]: cluster 2024-04-02T05:03:03.071775+0000 osd.2 (osd.2) 363 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:03.521 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:03.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:03.635 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:03 smithi098 bash[23790]: cluster 2024-04-02T05:03:02.353784+0000 mgr.y (mgr.14538) 746 : cluster 0 pgmap v1305: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 990 B/s rd, 165 B/s wr, 1 op/s 2024-04-02T05:03:03.635 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:03 smithi098 bash[23790]: cluster 2024-04-02T05:03:03.071307+0000 osd.2 (osd.2) 362 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:03.636 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:03 smithi098 bash[23790]: cluster 2024-04-02T05:03:03.071775+0000 osd.2 (osd.2) 363 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:03.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:03.775 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:03.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:03.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:03:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:03:03.905 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:03.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:04.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:04.038 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:04.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:04.170 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:04.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:04.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:04.429 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:04.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:04.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:04 smithi098 bash[23790]: cluster 2024-04-02T05:03:03.422015+0000 mon.a (mon.0) 3929 : cluster 0 osdmap e858: 8 total, 8 up, 7 in 2024-04-02T05:03:04.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:04 smithi098 bash[23790]: audit 2024-04-02T05:03:03.751552+0000 mgr.y (mgr.14538) 747 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:04.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:04 smithi098 bash[23790]: audit 2024-04-02T05:03:04.156963+0000 mon.a (mon.0) 3930 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:04.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:04 smithi098 bash[23790]: audit 2024-04-02T05:03:04.159356+0000 mon.b (mon.1) 318 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:04.561 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:04.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:04.690 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:04.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[21528]: cluster 2024-04-02T05:03:03.422015+0000 mon.a (mon.0) 3929 : cluster 0 osdmap e858: 8 total, 8 up, 7 in 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[21528]: audit 2024-04-02T05:03:03.751552+0000 mgr.y (mgr.14538) 747 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[21528]: audit 2024-04-02T05:03:04.156963+0000 mon.a (mon.0) 3930 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[21528]: audit 2024-04-02T05:03:04.159356+0000 mon.b (mon.1) 318 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[28904]: cluster 2024-04-02T05:03:03.422015+0000 mon.a (mon.0) 3929 : cluster 0 osdmap e858: 8 total, 8 up, 7 in 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[28904]: audit 2024-04-02T05:03:03.751552+0000 mgr.y (mgr.14538) 747 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[28904]: audit 2024-04-02T05:03:04.156963+0000 mon.a (mon.0) 3930 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:04 smithi069 bash[28904]: audit 2024-04-02T05:03:04.159356+0000 mon.b (mon.1) 318 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:04.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:04.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:04.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:05.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:05.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:05.202 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/3849987699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc10a560 msgr2=0x7f1afc10a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- 172.21.15.69:0/3849987699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc10a560 0x7f1afc10a940 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7f1af800b0a0 tx=0x7f1af800b3b0 comp rx=0 tx=0).stop 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/3849987699 shutdown_connections 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- 172.21.15.69:0/3849987699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1afc104e70 0x7f1afc111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- 172.21.15.69:0/3849987699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc10a560 0x7f1afc10a940 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- 172.21.15.69:0/3849987699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc1044b0 0x7f1afc104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/3849987699 >> 172.21.15.69:0/3849987699 conn(0x7f1afc0fffa0 msgr2=0x7f1afc1023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/3849987699 shutdown_connections 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/3849987699 wait complete. 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 librados: starting msgr at 2024-04-02T05:03:05.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 librados: starting objecter 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 Processor -- start 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- start start 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 librados: setting wanted keys 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 librados: calling monclient init 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 0x7f1afc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1afc104e70 0x7f1afc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc10a560 0x7f1afc19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc113ca0 con 0x7f1afc10a560 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc113b20 con 0x7f1afc104e70 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1afc113e20 con 0x7f1afc1044b0 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 0x7f1afc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01989700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc10a560 0x7f1afc19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 0x7f1afc19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:55136/0 (socket says 172.21.15.69:55136) 2024-04-02T05:03:05.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 -- 172.21.15.69:0/1037220943 learned_addr learned my addr 172.21.15.69:0/1037220943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 -- 172.21.15.69:0/1037220943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1afc104e70 msgr2=0x7f1afc19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:05.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1afc104e70 0x7f1afc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 -- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc10a560 msgr2=0x7f1afc19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc10a560 0x7f1afc19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 -- 172.21.15.69:0/1037220943 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc19f1a0 con 0x7f1afc1044b0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01989700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc10a560 0x7f1afc19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b01188700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 0x7f1afc19a190 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7f1af8004d00 tx=0x7f1af8004d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1af27fc700 10 client.?.objecter ms_handle_connect 0x7f1afc1044b0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1af27fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8021020 con 0x7f1afc1044b0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1afc19f430 con 0x7f1afc1044b0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f1af8007b80 con 0x7f1afc1044b0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af801b690 con 0x7f1afc1044b0 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 10 client.43847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:05.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1afc1a6d10 con 0x7f1afc1044b0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 librados: init done 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.279+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1afc04ff00 con 0x7f1afc1044b0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f1af8024020 con 0x7f1afc1044b0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1ae806f5e0 0x7f1ae8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(859..859 src has 74..859) ==== 10139+0+0 (secure 0 0 0) 0x7f1af800f020 con 0x7f1afc1044b0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 10 client.43847.objecter ms_dispatch 0x7f1afc001550 osd_map(859..859 src has 74..859) 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 3 client.43847.objecter handle_osd_map got epochs [859,859] > 0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 3 client.43847.objecter handle_osd_map decoding full epoch 859 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1b00987700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1ae806f5e0 0x7f1ae8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 20 client.43847.objecter dump_active .. 0 homeless 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1af27fc700 10 client.43847.objecter ms_handle_connect 0x7f1ae806f5e0 2024-04-02T05:03:05.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.283+0000 7f1b00987700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1ae806f5e0 0x7f1ae8071aa0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7f1aec005d90 tx=0x7f1aec005ce0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:05.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.287+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1af805c3e0 con 0x7f1afc1044b0 2024-04-02T05:03:05.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:05.324 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:05.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:05.450 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:05.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:05 smithi069 bash[21528]: cluster 2024-04-02T05:03:04.354999+0000 mgr.y (mgr.14538) 748 : cluster 0 pgmap v1307: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 142 B/s wr, 1 op/s 2024-04-02T05:03:05.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:05 smithi069 bash[21528]: cluster 2024-04-02T05:03:05.119619+0000 mon.a (mon.0) 3931 : cluster 0 osdmap e859: 8 total, 8 up, 7 in 2024-04-02T05:03:05.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:05 smithi069 bash[28904]: cluster 2024-04-02T05:03:04.354999+0000 mgr.y (mgr.14538) 748 : cluster 0 pgmap v1307: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 142 B/s wr, 1 op/s 2024-04-02T05:03:05.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:05 smithi069 bash[28904]: cluster 2024-04-02T05:03:05.119619+0000 mon.a (mon.0) 3931 : cluster 0 osdmap e859: 8 total, 8 up, 7 in 2024-04-02T05:03:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:05.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:05.515+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585} v 0) -- 0x7f1afc1a6ee0 con 0x7f1afc1044b0 2024-04-02T05:03:05.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:05.582 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:05.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:05.712 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:05 smithi098 bash[23790]: cluster 2024-04-02T05:03:04.354999+0000 mgr.y (mgr.14538) 748 : cluster 0 pgmap v1307: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 142 B/s wr, 1 op/s 2024-04-02T05:03:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:05 smithi098 bash[23790]: cluster 2024-04-02T05:03:05.119619+0000 mon.a (mon.0) 3931 : cluster 0 osdmap e859: 8 total, 8 up, 7 in 2024-04-02T05:03:05.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:05.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:05.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:05.971 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:05.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:06.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:06.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:06.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:06.231 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:06.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-02T05:03:06.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:06.362 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:06.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1af27fc700 1 -- 172.21.15.69:0/1037220943 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]=0 reweighted osd.2 to 0.304067 (4dd7) v860) ==== 134+0+0 (secure 0 0 0) 0x7f1af80074e0 con 0x7f1afc1044b0 2024-04-02T05:03:06.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 10 librados: watch_flush enter 2024-04-02T05:03:06.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 10 librados: watch_flush exit 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 20 client.43847.objecter shutdown clearing up homeless session... 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 10 client.43847.objecter successfully canceled tick 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1ae806f5e0 msgr2=0x7f1ae8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1ae806f5e0 0x7f1ae8071aa0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7f1aec005d90 tx=0x7f1aec005ce0 comp rx=0 tx=0).stop 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.431+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 msgr2=0x7f1afc19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 0x7f1afc19a190 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7f1af8004d00 tx=0x7f1af8004d30 comp rx=0 tx=0).stop 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 shutdown_connections 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1ae806f5e0 0x7f1ae8071aa0 unknown :-1 s=CLOSED pgs=4021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1afc10a560 0x7f1afc19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1afc1044b0 0x7f1afc19a190 secure :-1 s=CLOSED pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7f1af8004d00 tx=0x7f1af8004d30 comp rx=0 tx=0).stop 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 --2- 172.21.15.69:0/1037220943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1afc104e70 0x7f1afc19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:06.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 >> 172.21.15.69:0/1037220943 conn(0x7f1afc0fffa0 msgr2=0x7f1afc10b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:06.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 shutdown_connections 2024-04-02T05:03:06.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 -- 172.21.15.69:0/1037220943 wait complete. 2024-04-02T05:03:06.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:06.435+0000 7f1b033ec700 1 librados: shutdown 2024-04-02T05:03:06.438 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.2 to 0.304067 (4dd7) 2024-04-02T05:03:06.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:06.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:06 smithi069 bash[21528]: audit 2024-04-02T05:03:05.517551+0000 mon.c (mon.2) 1163 : audit 1 from='client.? 172.21.15.69:0/1037220943' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]: dispatch 2024-04-02T05:03:06.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:06 smithi069 bash[21528]: audit 2024-04-02T05:03:05.518047+0000 mon.a (mon.0) 3932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]: dispatch 2024-04-02T05:03:06.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:06 smithi069 bash[28904]: audit 2024-04-02T05:03:05.517551+0000 mon.c (mon.2) 1163 : audit 1 from='client.? 172.21.15.69:0/1037220943' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]: dispatch 2024-04-02T05:03:06.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:06 smithi069 bash[28904]: audit 2024-04-02T05:03:05.518047+0000 mon.a (mon.0) 3932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]: dispatch 2024-04-02T05:03:06.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:06.605 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:06 smithi098 bash[23790]: audit 2024-04-02T05:03:05.517551+0000 mon.c (mon.2) 1163 : audit 1 from='client.? 172.21.15.69:0/1037220943' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]: dispatch 2024-04-02T05:03:06.605 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:06 smithi098 bash[23790]: audit 2024-04-02T05:03:05.518047+0000 mon.a (mon.0) 3932 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]: dispatch 2024-04-02T05:03:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:06.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:06.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:06.878 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:06.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:06.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:07.005 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:07.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:07.135 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:07.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:07.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:07.396 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:07.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:07.508 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:07 smithi098 bash[23790]: cluster 2024-04-02T05:03:06.355789+0000 mgr.y (mgr.14538) 749 : cluster 0 pgmap v1309: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 127 B/s wr, 0 op/s 2024-04-02T05:03:07.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:07 smithi098 bash[23790]: audit 2024-04-02T05:03:06.424126+0000 mon.a (mon.0) 3933 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]': finished 2024-04-02T05:03:07.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:07 smithi098 bash[23790]: cluster 2024-04-02T05:03:06.432387+0000 mon.a (mon.0) 3934 : cluster 0 osdmap e860: 8 total, 8 up, 7 in 2024-04-02T05:03:07.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:07 smithi098 bash[23790]: cluster 2024-04-02T05:03:06.685335+0000 osd.0 (osd.0) 409 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:07.509 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:07 smithi098 bash[23790]: cluster 2024-04-02T05:03:06.685599+0000 osd.0 (osd.0) 410 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:07.521 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:07.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/1698610092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 msgr2=0x7febe4107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- 172.21.15.69:0/1698610092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe4107600 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7febe0009990 tx=0x7febe0009ca0 comp rx=0 tx=0).stop 2024-04-02T05:03:07.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/1698610092 shutdown_connections 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- 172.21.15.69:0/1698610092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- 172.21.15.69:0/1698610092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe4107600 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- 172.21.15.69:0/1698610092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe41011a0 0x7febe4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/1698610092 >> 172.21.15.69:0/1698610092 conn(0x7febe40771e0 msgr2=0x7febe40775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/1698610092 shutdown_connections 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/1698610092 wait complete. 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 librados: starting msgr at 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 librados: starting objecter 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 Processor -- start 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- start start 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 librados: setting wanted keys 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 librados: calling monclient init 2024-04-02T05:03:07.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe41011a0 0x7febe410e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe410f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febe4111af0 con 0x7febe4107220 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7febe4111970 con 0x7febe41011a0 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7febe4111c70 con 0x7febe4101b60 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe8e81700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe410f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe8e81700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44196/0 (socket says 172.21.15.69:44196) 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe410f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37850/0 (socket says 172.21.15.69:37850) 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe8e81700 1 -- 172.21.15.69:0/678096101 learned_addr learned my addr 172.21.15.69:0/678096101 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9682700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe41011a0 0x7febe410e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 -- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 msgr2=0x7febe410ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:07.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 -- 172.21.15.69:0/678096101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe41011a0 msgr2=0x7febe410e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe41011a0 0x7febe410e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 -- 172.21.15.69:0/678096101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe41a65f0 con 0x7febe4107220 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe8e81700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febe9e83700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe410f360 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7febdc00efc0 tx=0x7febdc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febda7fc700 10 client.?.objecter ms_handle_connect 0x7febe4107220 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febda7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc0092e0 con 0x7febe4107220 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7febdc004bb0 con 0x7febe4107220 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe41a6880 con 0x7febe4107220 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 10 client.33789.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe41a6e10 con 0x7febe4107220 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 librados: init done 2024-04-02T05:03:07.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc004020 con 0x7febe4107220 2024-04-02T05:03:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.715+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc8005120 con 0x7febe4107220 2024-04-02T05:03:07.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7febdc01b030 con 0x7febe4107220 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7febd006f5e0 0x7febd0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(861..861 src has 74..861) ==== 10284+0+0 (secure 0 0 0) 0x7febdc08ef70 con 0x7febe4107220 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 10 client.33789.objecter ms_dispatch 0x7febe4001550 osd_map(861..861 src has 74..861) 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 3 client.33789.objecter handle_osd_map got epochs [861,861] > 0 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 3 client.33789.objecter handle_osd_map decoding full epoch 861 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febe9682700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7febd006f5e0 0x7febd0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 20 client.33789.objecter dump_active .. 0 homeless 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febe9682700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7febd006f5e0 0x7febd0071aa0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7febe0006010 tx=0x7febe001a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:07.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.719+0000 7febda7fc700 10 client.33789.objecter ms_handle_connect 0x7febd006f5e0 2024-04-02T05:03:07.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.723+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7febdc022020 con 0x7febe4107220 2024-04-02T05:03:07.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:07.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[28904]: cluster 2024-04-02T05:03:06.355789+0000 mgr.y (mgr.14538) 749 : cluster 0 pgmap v1309: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 127 B/s wr, 0 op/s 2024-04-02T05:03:07.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[28904]: audit 2024-04-02T05:03:06.424126+0000 mon.a (mon.0) 3933 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]': finished 2024-04-02T05:03:07.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[28904]: cluster 2024-04-02T05:03:06.432387+0000 mon.a (mon.0) 3934 : cluster 0 osdmap e860: 8 total, 8 up, 7 in 2024-04-02T05:03:07.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[28904]: cluster 2024-04-02T05:03:06.685335+0000 osd.0 (osd.0) 409 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:07.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[28904]: cluster 2024-04-02T05:03:06.685599+0000 osd.0 (osd.0) 410 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[21528]: cluster 2024-04-02T05:03:06.355789+0000 mgr.y (mgr.14538) 749 : cluster 0 pgmap v1309: 224 pgs: 224 active+clean; 4.4 MiB data, 742 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 127 B/s wr, 0 op/s 2024-04-02T05:03:07.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[21528]: audit 2024-04-02T05:03:06.424126+0000 mon.a (mon.0) 3933 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.3040669151245585}]': finished 2024-04-02T05:03:07.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[21528]: cluster 2024-04-02T05:03:06.432387+0000 mon.a (mon.0) 3934 : cluster 0 osdmap e860: 8 total, 8 up, 7 in 2024-04-02T05:03:07.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[21528]: cluster 2024-04-02T05:03:06.685335+0000 osd.0 (osd.0) 409 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:07.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:07 smithi069 bash[21528]: cluster 2024-04-02T05:03:06.685599+0000 osd.0 (osd.0) 410 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:07.771 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:07.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:07.900 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:07.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:07.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:07.947+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7febc80056e0 con 0x7febe4107220 2024-04-02T05:03:08.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:08.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:08.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:08.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:08.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:08.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.427+0000 7febda7fc700 1 -- 172.21.15.69:0/678096101 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v862) ==== 91+0+0 (secure 0 0 0) 0x7febdc010040 con 0x7febe4107220 2024-04-02T05:03:08.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 10 librados: watch_flush enter 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 10 librados: watch_flush exit 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 20 client.33789.objecter shutdown clearing up homeless session... 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 10 client.33789.objecter successfully canceled tick 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7febd006f5e0 msgr2=0x7febd0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7febd006f5e0 0x7febd0071aa0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7febe0006010 tx=0x7febe001a040 comp rx=0 tx=0).stop 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 msgr2=0x7febe410f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe410f360 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7febdc00efc0 tx=0x7febdc00c3a0 comp rx=0 tx=0).stop 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 shutdown_connections 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7febd006f5e0 0x7febd0071aa0 secure :-1 s=CLOSED pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7febe0006010 tx=0x7febe001a040 comp rx=0 tx=0).stop 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febe4107220 0x7febe410f360 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febe41011a0 0x7febe410e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 --2- 172.21.15.69:0/678096101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febe4101b60 0x7febe410ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:08.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.431+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 >> 172.21.15.69:0/678096101 conn(0x7febe40771e0 msgr2=0x7febe4107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:08.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.435+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 shutdown_connections 2024-04-02T05:03:08.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.435+0000 7febeb8e6700 1 -- 172.21.15.69:0/678096101 wait complete. 2024-04-02T05:03:08.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:08.439+0000 7febeb8e6700 1 librados: shutdown 2024-04-02T05:03:08.443 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T05:03:08.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:08.541 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:08.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[28904]: cluster 2024-04-02T05:03:07.434610+0000 mon.a (mon.0) 3935 : cluster 0 osdmap e861: 8 total, 8 up, 7 in 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[28904]: audit 2024-04-02T05:03:07.950036+0000 mon.a (mon.0) 3936 : audit 1 from='client.? 172.21.15.69:0/678096101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[28904]: cluster 2024-04-02T05:03:08.427356+0000 mon.a (mon.0) 3937 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[28904]: audit 2024-04-02T05:03:08.429423+0000 mon.a (mon.0) 3938 : audit 1 from='client.? 172.21.15.69:0/678096101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[28904]: cluster 2024-04-02T05:03:08.435911+0000 mon.a (mon.0) 3939 : cluster 0 osdmap e862: 8 total, 8 up, 7 in 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[28904]: cluster 2024-04-02T05:03:08.451684+0000 mon.a (mon.0) 3940 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21528]: cluster 2024-04-02T05:03:07.434610+0000 mon.a (mon.0) 3935 : cluster 0 osdmap e861: 8 total, 8 up, 7 in 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21528]: audit 2024-04-02T05:03:07.950036+0000 mon.a (mon.0) 3936 : audit 1 from='client.? 172.21.15.69:0/678096101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21528]: cluster 2024-04-02T05:03:08.427356+0000 mon.a (mon.0) 3937 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21528]: audit 2024-04-02T05:03:08.429423+0000 mon.a (mon.0) 3938 : audit 1 from='client.? 172.21.15.69:0/678096101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21528]: cluster 2024-04-02T05:03:08.435911+0000 mon.a (mon.0) 3939 : cluster 0 osdmap e862: 8 total, 8 up, 7 in 2024-04-02T05:03:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21528]: cluster 2024-04-02T05:03:08.451684+0000 mon.a (mon.0) 3940 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T05:03:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:08 smithi098 bash[23790]: cluster 2024-04-02T05:03:07.434610+0000 mon.a (mon.0) 3935 : cluster 0 osdmap e861: 8 total, 8 up, 7 in 2024-04-02T05:03:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:08 smithi098 bash[23790]: audit 2024-04-02T05:03:07.950036+0000 mon.a (mon.0) 3936 : audit 1 from='client.? 172.21.15.69:0/678096101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:03:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:08 smithi098 bash[23790]: cluster 2024-04-02T05:03:08.427356+0000 mon.a (mon.0) 3937 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:03:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:08 smithi098 bash[23790]: audit 2024-04-02T05:03:08.429423+0000 mon.a (mon.0) 3938 : audit 1 from='client.? 172.21.15.69:0/678096101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:03:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:08 smithi098 bash[23790]: cluster 2024-04-02T05:03:08.435911+0000 mon.a (mon.0) 3939 : cluster 0 osdmap e862: 8 total, 8 up, 7 in 2024-04-02T05:03:08.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:08 smithi098 bash[23790]: cluster 2024-04-02T05:03:08.451684+0000 mon.a (mon.0) 3940 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T05:03:08.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:08.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:08.912 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:03:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:03:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:03:08.926 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:08.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:09.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:09.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.187 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:09.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:09.314 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:09.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:09.554 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:09.580 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:09.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:09.710 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: cluster 2024-04-02T05:03:08.357809+0000 mgr.y (mgr.14538) 750 : cluster 0 pgmap v1312: 224 pgs: 3 remapped+peering, 221 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 4 op/s; 16 KiB/s, 8 objects/s recovering 2024-04-02T05:03:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: audit 2024-04-02T05:03:08.583061+0000 mon.a (mon.0) 3941 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: cluster 2024-04-02T05:03:08.638889+0000 osd.0 (osd.0) 411 : cluster 0 2.c scrub starts 2024-04-02T05:03:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: cluster 2024-04-02T05:03:08.640479+0000 osd.0 (osd.0) 412 : cluster 0 2.c scrub ok 2024-04-02T05:03:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.195289+0000 osd.7 (osd.7) 221 : cluster 0 323.1s0 scrub starts 2024-04-02T05:03:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.197885+0000 osd.7 (osd.7) 222 : cluster 0 323.1 scrub ok 2024-04-02T05:03:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: audit 2024-04-02T05:03:09.432221+0000 mon.a (mon.0) 3942 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.440813+0000 mon.a (mon.0) 3943 : cluster 0 osdmap e863: 8 total, 8 up, 7 in 2024-04-02T05:03:09.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:09 smithi098 bash[23790]: audit 2024-04-02T05:03:09.443854+0000 mon.a (mon.0) 3944 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:03:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: cluster 2024-04-02T05:03:08.357809+0000 mgr.y (mgr.14538) 750 : cluster 0 pgmap v1312: 224 pgs: 3 remapped+peering, 221 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 4 op/s; 16 KiB/s, 8 objects/s recovering 2024-04-02T05:03:09.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: audit 2024-04-02T05:03:08.583061+0000 mon.a (mon.0) 3941 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: cluster 2024-04-02T05:03:08.638889+0000 osd.0 (osd.0) 411 : cluster 0 2.c scrub starts 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: cluster 2024-04-02T05:03:08.640479+0000 osd.0 (osd.0) 412 : cluster 0 2.c scrub ok 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.195289+0000 osd.7 (osd.7) 221 : cluster 0 323.1s0 scrub starts 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.197885+0000 osd.7 (osd.7) 222 : cluster 0 323.1 scrub ok 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: audit 2024-04-02T05:03:09.432221+0000 mon.a (mon.0) 3942 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.440813+0000 mon.a (mon.0) 3943 : cluster 0 osdmap e863: 8 total, 8 up, 7 in 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[21528]: audit 2024-04-02T05:03:09.443854+0000 mon.a (mon.0) 3944 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: cluster 2024-04-02T05:03:08.357809+0000 mgr.y (mgr.14538) 750 : cluster 0 pgmap v1312: 224 pgs: 3 remapped+peering, 221 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 4 op/s; 16 KiB/s, 8 objects/s recovering 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: audit 2024-04-02T05:03:08.583061+0000 mon.a (mon.0) 3941 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:09.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: cluster 2024-04-02T05:03:08.638889+0000 osd.0 (osd.0) 411 : cluster 0 2.c scrub starts 2024-04-02T05:03:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: cluster 2024-04-02T05:03:08.640479+0000 osd.0 (osd.0) 412 : cluster 0 2.c scrub ok 2024-04-02T05:03:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.195289+0000 osd.7 (osd.7) 221 : cluster 0 323.1s0 scrub starts 2024-04-02T05:03:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.197885+0000 osd.7 (osd.7) 222 : cluster 0 323.1 scrub ok 2024-04-02T05:03:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: audit 2024-04-02T05:03:09.432221+0000 mon.a (mon.0) 3942 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.440813+0000 mon.a (mon.0) 3943 : cluster 0 osdmap e863: 8 total, 8 up, 7 in 2024-04-02T05:03:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:09 smithi069 bash[28904]: audit 2024-04-02T05:03:09.443854+0000 mon.a (mon.0) 3944 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:03:09.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.841 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:09.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:09.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:10.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 -- 172.21.15.69:0/2227047088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06ab80 msgr2=0x7f75cc106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 --2- 172.21.15.69:0/2227047088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06ab80 0x7f75cc106ef0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0099c0 tx=0x7f75bc009cd0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 -- 172.21.15.69:0/2227047088 shutdown_connections 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 --2- 172.21.15.69:0/2227047088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75cc107620 0x7f75cc10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 --2- 172.21.15.69:0/2227047088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc06a1d0 0x7f75cc06a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 --2- 172.21.15.69:0/2227047088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06ab80 0x7f75cc106ef0 unknown :-1 s=CLOSED pgs=3297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 -- 172.21.15.69:0/2227047088 >> 172.21.15.69:0/2227047088 conn(0x7f75cc0771e0 msgr2=0x7f75cc0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:10.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 -- 172.21.15.69:0/2227047088 shutdown_connections 2024-04-02T05:03:10.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 -- 172.21.15.69:0/2227047088 wait complete. 2024-04-02T05:03:10.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 librados: starting msgr at 2024-04-02T05:03:10.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 librados: starting objecter 2024-04-02T05:03:10.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.091+0000 7f75d2605700 1 Processor -- start 2024-04-02T05:03:10.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- start start 2024-04-02T05:03:10.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 librados: setting wanted keys 2024-04-02T05:03:10.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 librados: calling monclient init 2024-04-02T05:03:10.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06a1d0 0x7f75cc101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc06ab80 0x7f75cc1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06a1d0 0x7f75cc101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06a1d0 0x7f75cc101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:44220/0 (socket says 172.21.15.69:44220) 2024-04-02T05:03:10.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75cc107620 0x7f75cc101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 -- 172.21.15.69:0/1966444977 learned_addr learned my addr 172.21.15.69:0/1966444977 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:10.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc06ab80 0x7f75cc1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc111b00 con 0x7f75cc06ab80 2024-04-02T05:03:10.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc111980 con 0x7f75cc107620 2024-04-02T05:03:10.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f75cc111c80 con 0x7f75cc06a1d0 2024-04-02T05:03:10.100 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:10.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75cc107620 0x7f75cc101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75cc107620 msgr2=0x7f75cc101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75cc107620 0x7f75cc101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc06ab80 msgr2=0x7f75cc1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc06ab80 0x7f75cc1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75cc1a6630 con 0x7f75cc06a1d0 2024-04-02T05:03:10.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06a1d0 0x7f75cc101270 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7f75c000e9a0 tx=0x7f75c000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 10 client.?.objecter ms_handle_connect 0x7f75cc06a1d0 2024-04-02T05:03:10.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:10.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c000cbd0 con 0x7f75cc06a1d0 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75cc1a6920 con 0x7f75cc06a1d0 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f75c000cd70 con 0x7f75cc06a1d0 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0010e60 con 0x7f75cc06a1d0 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 10 client.43859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75cc1a6dc0 con 0x7f75cc06a1d0 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 librados: init done 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f75cc04ff00 con 0x7f75cc06a1d0 2024-04-02T05:03:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:10.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f75c0010430 con 0x7f75cc06a1d0 2024-04-02T05:03:10.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f75b406f5e0 0x7f75b4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(863..863 src has 74..863) ==== 10110+0+0 (secure 0 0 0) 0x7f75c008ec80 con 0x7f75cc06a1d0 2024-04-02T05:03:10.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 osd_map(863..863 src has 74..863) 2024-04-02T05:03:10.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 3 client.43859.objecter handle_osd_map got epochs [863,863] > 0 2024-04-02T05:03:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 3 client.43859.objecter handle_osd_map decoding full epoch 863 2024-04-02T05:03:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f75b406f5e0 0x7f75b4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 20 client.43859.objecter dump_active .. 0 homeless 2024-04-02T05:03:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.095+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v863) ==== 54+0+16 (secure 0 0 0) 0x7f75c0091220 con 0x7f75cc06a1d0 2024-04-02T05:03:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 10 client.43859.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 1 2024-04-02T05:03:10.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f75cc062e50 0x7f75cc063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 1 2024-04-02T05:03:10.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 0 1 2024-04-02T05:03:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 4 2024-04-02T05:03:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 10 client.43859.objecter _send_command 1 2024-04-02T05:03:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1aa1d0 con 0x7f75cc062e50 2024-04-02T05:03:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f75cc062e50 0x7f75cc063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f75b406f5e0 0x7f75b4071aa0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7f75bc000c00 tx=0x7f75bc019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75b406f5e0 2024-04-02T05:03:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc062e50 2024-04-02T05:03:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f75cc062e50 0x7f75cc063230 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1aa1d0 con 0x7f75cc062e50 2024-04-02T05:03:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 1: 0 ) 2024-04-02T05:03:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 10 client.43859.objecter _finish_command 1 = system:0 2024-04-02T05:03:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 4 2024-04-02T05:03:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.099+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 0 1 2024-04-02T05:03:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 10 client.43859.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 2 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 2 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1027a0 osd=0 3 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 0 2 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 4 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 10 client.43859.objecter _send_command 2 2024-04-02T05:03:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.115+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc0628d0 con 0x7f75cc062e50 2024-04-02T05:03:10.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc0628d0 con 0x7f75cc062e50 2024-04-02T05:03:10.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 10 client.43859.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 4 2024-04-02T05:03:10.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 0 2 2024-04-02T05:03:10.121 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:03:10.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 10 client.43859.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f75cc02c9b0 0x7f75cc02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 1 3 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 4 2024-04-02T05:03:10.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 10 client.43859.objecter _send_command 3 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f75cc109ec0 con 0x7f75cc02c9b0 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f75cc02c9b0 0x7f75cc02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc02c9b0 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f75cc02c9b0 0x7f75cc02cdf0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc109ec0 con 0x7f75cc02c9b0 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 3: 0 ) 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 10 client.43859.objecter _finish_command 3 = system:0 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 4 2024-04-02T05:03:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.119+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 1 3 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 10 client.43859.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 4 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 4 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc109840 osd=1 3 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.135+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 1 4 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 4 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 10 client.43859.objecter _send_command 4 2024-04-02T05:03:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc1092c0 con 0x7f75cc02c9b0 2024-04-02T05:03:10.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc1092c0 con 0x7f75cc02c9b0 2024-04-02T05:03:10.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 10 client.43859.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 4 2024-04-02T05:03:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 1 4 2024-04-02T05:03:10.142 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 10 client.43859.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 5 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f75cc1ac390 0x7f75cc1ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 5 2024-04-02T05:03:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 2 5 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 4 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 10 client.43859.objecter _send_command 5 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1b15e0 con 0x7f75cc1ac390 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f75cc1ac390 0x7f75cc1ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc1ac390 2024-04-02T05:03:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f75cc1ac390 0x7f75cc1ae780 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1b15e0 con 0x7f75cc1ac390 2024-04-02T05:03:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 5: 0 ) 2024-04-02T05:03:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 10 client.43859.objecter _finish_command 5 = system:0 2024-04-02T05:03:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 4 2024-04-02T05:03:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.139+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 2 5 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 10 client.43859.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 6 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 6 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1096c0 osd=2 3 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 2 6 2024-04-02T05:03:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 4 2024-04-02T05:03:10.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 10 client.43859.objecter _send_command 6 2024-04-02T05:03:10.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc02d580 con 0x7f75cc1ac390 2024-04-02T05:03:10.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc02d580 con 0x7f75cc1ac390 2024-04-02T05:03:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 10 client.43859.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 4 2024-04-02T05:03:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 2 6 2024-04-02T05:03:10.162 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:03:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 10 client.43859.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 7 2024-04-02T05:03:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f75cc1afe40 0x7f75cc1b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 7 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 3 7 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 4 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 10 client.43859.objecter _send_command 7 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1a9f90 con 0x7f75cc1afe40 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f75cc1afe40 0x7f75cc1b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc1afe40 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f75cc1afe40 0x7f75cc1b0280 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1a9f90 con 0x7f75cc1afe40 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 7: 0 ) 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 10 client.43859.objecter _finish_command 7 = system:0 2024-04-02T05:03:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 4 2024-04-02T05:03:10.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.159+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 3 7 2024-04-02T05:03:10.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 10 client.43859.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 8 2024-04-02T05:03:10.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 8 2024-04-02T05:03:10.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1a9820 osd=3 3 2024-04-02T05:03:10.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 3 8 2024-04-02T05:03:10.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 4 2024-04-02T05:03:10.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 10 client.43859.objecter _send_command 8 2024-04-02T05:03:10.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc1a92a0 con 0x7f75cc1afe40 2024-04-02T05:03:10.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc1a92a0 con 0x7f75cc1afe40 2024-04-02T05:03:10.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75c97fa700 10 client.43859.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 4 2024-04-02T05:03:10.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 3 8 2024-04-02T05:03:10.184 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:03:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 10 client.43859.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 9 2024-04-02T05:03:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f75cc1a9410 0x7f75cc1b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 9 2024-04-02T05:03:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 4 9 2024-04-02T05:03:10.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 4 2024-04-02T05:03:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 10 client.43859.objecter _send_command 9 2024-04-02T05:03:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1b8660 con 0x7f75cc1a9410 2024-04-02T05:03:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.179+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f75cc1a9410 0x7f75cc1b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc1a9410 2024-04-02T05:03:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f75cc1a9410 0x7f75cc1b57b0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1b8660 con 0x7f75cc1a9410 2024-04-02T05:03:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 9: 0 ) 2024-04-02T05:03:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75c97fa700 10 client.43859.objecter _finish_command 9 = system:0 2024-04-02T05:03:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 4 2024-04-02T05:03:10.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.183+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 4 9 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 10 client.43859.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 10 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 10 2024-04-02T05:03:10.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1a92a0 osd=4 3 2024-04-02T05:03:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 4 10 2024-04-02T05:03:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 4 2024-04-02T05:03:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 10 client.43859.objecter _send_command 10 2024-04-02T05:03:10.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.203+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc1b86c0 con 0x7f75cc1a9410 2024-04-02T05:03:10.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc1b86c0 con 0x7f75cc1a9410 2024-04-02T05:03:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 10 client.43859.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 4 2024-04-02T05:03:10.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 4 10 2024-04-02T05:03:10.209 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 10 client.43859.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 11 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f75cc1b7a50 0x7f75cc1b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 11 2024-04-02T05:03:10.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 5 11 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 4 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 10 client.43859.objecter _send_command 11 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1bba30 con 0x7f75cc1b7a50 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f75cc1b7a50 0x7f75cc1b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75cb7fe700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f75cc1b7a50 0x7f75cc1b7e70 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc1b7a50 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1bba30 con 0x7f75cc1b7a50 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 11: 0 ) 2024-04-02T05:03:10.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 10 client.43859.objecter _finish_command 11 = system:0 2024-04-02T05:03:10.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 4 2024-04-02T05:03:10.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.207+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 5 11 2024-04-02T05:03:10.225 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:10.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 10 client.43859.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 12 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 12 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1b86c0 osd=5 3 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 5 12 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 4 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 10 client.43859.objecter _send_command 12 2024-04-02T05:03:10.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.227+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc1b5d50 con 0x7f75cc1b7a50 2024-04-02T05:03:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc1b5d50 con 0x7f75cc1b7a50 2024-04-02T05:03:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75c97fa700 10 client.43859.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 4 2024-04-02T05:03:10.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 5 12 2024-04-02T05:03:10.235 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:03:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 10 client.43859.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 13 2024-04-02T05:03:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f75cc1b6470 0x7f75cc1b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 13 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 6 13 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 4 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 10 client.43859.objecter _send_command 13 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1b6fd0 con 0x7f75cc1b6470 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f75cc1b6470 0x7f75cc1b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc1b6470 2024-04-02T05:03:10.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.231+0000 7f75d0ba2700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f75cc1b6470 0x7f75cc1b68d0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.235+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1b6fd0 con 0x7f75cc1b6470 2024-04-02T05:03:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.235+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 13: 0 ) 2024-04-02T05:03:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.235+0000 7f75c97fa700 10 client.43859.objecter _finish_command 13 = system:0 2024-04-02T05:03:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.235+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 4 2024-04-02T05:03:10.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.235+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 6 13 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 10 client.43859.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 14 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 14 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1b5d50 osd=6 3 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 6 14 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 4 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 10 client.43859.objecter _send_command 14 2024-04-02T05:03:10.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.251+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc1b6fd0 con 0x7f75cc1b6470 2024-04-02T05:03:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc1b6fd0 con 0x7f75cc1b6470 2024-04-02T05:03:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75c97fa700 10 client.43859.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 4 2024-04-02T05:03:10.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 6 14 2024-04-02T05:03:10.259 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 10 client.43859.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 15 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f75cc1bcdd0 0x7f75cc1bf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 15 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 7 15 2024-04-02T05:03:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 4 2024-04-02T05:03:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 10 client.43859.objecter _send_command 15 2024-04-02T05:03:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.255+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f75cc1c2020 con 0x7f75cc1bcdd0 2024-04-02T05:03:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f75cc1bcdd0 0x7f75cc1bf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75c97fa700 10 client.43859.objecter ms_handle_connect 0x7f75cc1bcdd0 2024-04-02T05:03:10.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75cbfff700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f75cc1bcdd0 0x7f75cc1bf1c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:10.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75cc1c2020 con 0x7f75cc1bcdd0 2024-04-02T05:03:10.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 15: 0 ) 2024-04-02T05:03:10.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75c97fa700 10 client.43859.objecter _finish_command 15 = system:0 2024-04-02T05:03:10.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 4 2024-04-02T05:03:10.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.259+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 7 15 2024-04-02T05:03:10.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 10 client.43859.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:10.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign -1 16 2024-04-02T05:03:10.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 20 client.43859.objecter _get_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 15 client.43859.objecter _session_command_op_remove -1 16 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 20 client.43859.objecter get_session s=0x7f75cc1b6fd0 osd=7 3 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 15 client.43859.objecter _session_command_op_assign 7 16 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 4 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 10 client.43859.objecter _send_command 16 2024-04-02T05:03:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.275+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75cc1bf700 con 0x7f75cc1bcdd0 2024-04-02T05:03:10.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75c97fa700 1 -- 172.21.15.69:0/1966444977 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75cc1bf700 con 0x7f75cc1bcdd0 2024-04-02T05:03:10.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75c97fa700 10 client.43859.objecter ms_dispatch 0x7f75cc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:10.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75c97fa700 10 client.43859.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:10.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75c97fa700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 4 2024-04-02T05:03:10.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75c97fa700 15 client.43859.objecter _session_command_op_remove 7 16 2024-04-02T05:03:10.284 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:03:10.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 librados: watch_flush enter 2024-04-02T05:03:10.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 librados: watch_flush exit 2024-04-02T05:03:10.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.0 2024-04-02T05:03:10.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f75cc062e50 msgr2=0x7f75cc063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f75cc062e50 0x7f75cc063230 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1027a0 osd=0 1 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.1 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f75cc02c9b0 msgr2=0x7f75cc02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f75cc02c9b0 0x7f75cc02cdf0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc109840 osd=1 1 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.2 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f75cc1ac390 msgr2=0x7f75cc1ae780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f75cc1ac390 0x7f75cc1ae780 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1096c0 osd=2 1 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.3 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f75cc1afe40 msgr2=0x7f75cc1b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f75cc1afe40 0x7f75cc1b0280 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a9820 osd=3 1 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.4 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f75cc1a9410 msgr2=0x7f75cc1b57b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f75cc1a9410 0x7f75cc1b57b0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1a92a0 osd=4 1 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.5 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f75cc1b7a50 msgr2=0x7f75cc1b7e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f75cc1b7a50 0x7f75cc1b7e70 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b86c0 osd=5 1 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.6 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f75cc1b6470 msgr2=0x7f75cc1b68d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f75cc1b6470 0x7f75cc1b68d0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b5d50 osd=6 1 2024-04-02T05:03:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter close_session for osd.7 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f75cc1bcdd0 msgr2=0x7f75cc1bf1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f75cc1bcdd0 0x7f75cc1bf1c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter put_session s=0x7f75cc1b6fd0 osd=7 1 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d0ba2700 1 -- 172.21.15.69:0/1966444977 reap_dead start 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 20 client.43859.objecter shutdown clearing up homeless session... 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 10 client.43859.objecter successfully canceled tick 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d0ba2700 1 -- 172.21.15.69:0/1966444977 reap_dead start 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f75b406f5e0 msgr2=0x7f75b4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f75b406f5e0 0x7f75b4071aa0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7f75bc000c00 tx=0x7f75bc019040 comp rx=0 tx=0).stop 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d0ba2700 1 -- 172.21.15.69:0/1966444977 reap_dead start 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d0ba2700 1 -- 172.21.15.69:0/1966444977 reap_dead start 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06a1d0 msgr2=0x7f75cc101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d0ba2700 1 -- 172.21.15.69:0/1966444977 reap_dead start 2024-04-02T05:03:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 --2- 172.21.15.69:0/1966444977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc06a1d0 0x7f75cc101270 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7f75c000e9a0 tx=0x7f75c000ecb0 comp rx=0 tx=0).stop 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d0ba2700 1 -- 172.21.15.69:0/1966444977 reap_dead start 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 shutdown_connections 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 >> 172.21.15.69:0/1966444977 conn(0x7f75cc0771e0 msgr2=0x7f75cc0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 shutdown_connections 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 -- 172.21.15.69:0/1966444977 wait complete. 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:10.283+0000 7f75d2605700 1 librados: shutdown 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:03:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:03:10.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:10.348 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:10.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:10.482 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:10.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.188770+0000 osd.1 (osd.1) 417 : cluster 0 323.0s0 scrub starts 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.192626+0000 osd.1 (osd.1) 418 : cluster 0 323.0 scrub ok 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.525240+0000 osd.5 (osd.5) 263 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.525492+0000 osd.5 (osd.5) 264 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: audit 2024-04-02T05:03:10.099118+0000 mon.c (mon.2) 1164 : audit 0 from='client.? 172.21.15.69:0/1966444977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.432419+0000 mon.a (mon.0) 3945 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: audit 2024-04-02T05:03:10.434594+0000 mon.a (mon.0) 3946 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]': finished 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.442941+0000 mon.a (mon.0) 3947 : cluster 0 osdmap e864: 8 total, 8 up, 7 in 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.188770+0000 osd.1 (osd.1) 417 : cluster 0 323.0s0 scrub starts 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.192626+0000 osd.1 (osd.1) 418 : cluster 0 323.0 scrub ok 2024-04-02T05:03:10.596 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.525240+0000 osd.5 (osd.5) 263 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:10.597 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.525492+0000 osd.5 (osd.5) 264 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:10.597 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: audit 2024-04-02T05:03:10.099118+0000 mon.c (mon.2) 1164 : audit 0 from='client.? 172.21.15.69:0/1966444977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:10.597 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.432419+0000 mon.a (mon.0) 3945 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:03:10.597 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: audit 2024-04-02T05:03:10.434594+0000 mon.a (mon.0) 3946 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]': finished 2024-04-02T05:03:10.597 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:10 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.442941+0000 mon.a (mon.0) 3947 : cluster 0 osdmap e864: 8 total, 8 up, 7 in 2024-04-02T05:03:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:10.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:10.737 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:10.840 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:10.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:10.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:10.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.188770+0000 osd.1 (osd.1) 417 : cluster 0 323.0s0 scrub starts 2024-04-02T05:03:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.192626+0000 osd.1 (osd.1) 418 : cluster 0 323.0 scrub ok 2024-04-02T05:03:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.525240+0000 osd.5 (osd.5) 263 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.525492+0000 osd.5 (osd.5) 264 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: audit 2024-04-02T05:03:10.099118+0000 mon.c (mon.2) 1164 : audit 0 from='client.? 172.21.15.69:0/1966444977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.432419+0000 mon.a (mon.0) 3945 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:03:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: audit 2024-04-02T05:03:10.434594+0000 mon.a (mon.0) 3946 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]': finished 2024-04-02T05:03:10.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:10 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.442941+0000 mon.a (mon.0) 3947 : cluster 0 osdmap e864: 8 total, 8 up, 7 in 2024-04-02T05:03:10.998 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:11.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:11.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:11.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:11.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:11.385 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:11.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:11.513 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:11.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:11.640 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:11.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:11.768 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:11.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:11.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:11.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T05:03:12.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:12.027 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:12.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:12.142 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.937290+0000 osd.6 (osd.6) 361 : cluster 0 2.17 scrub starts 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:09.938872+0000 osd.6 (osd.6) 362 : cluster 0 2.17 scrub ok 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.158602+0000 osd.1 (osd.1) 419 : cluster 0 6.e scrub starts 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.160252+0000 osd.1 (osd.1) 420 : cluster 0 6.e scrub ok 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.358310+0000 mgr.y (mgr.14538) 751 : cluster 0 pgmap v1315: 224 pgs: 3 active+undersized+remapped, 3 remapped+peering, 218 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 4.5 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 12 objects/s recovering 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: audit 2024-04-02T05:03:10.490428+0000 mon.a (mon.0) 3948 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: audit 2024-04-02T05:03:10.490895+0000 mon.a (mon.0) 3949 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.507651+0000 osd.3 (osd.3) 425 : cluster 0 4.1b scrub starts 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.508981+0000 osd.3 (osd.3) 426 : cluster 0 4.1b scrub ok 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.594059+0000 osd.0 (osd.0) 413 : cluster 0 323.4s0 scrub starts 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.597089+0000 osd.0 (osd.0) 414 : cluster 0 323.4 scrub ok 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.729987+0000 osd.7 (osd.7) 223 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.730315+0000 osd.7 (osd.7) 224 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:11.213246+0000 osd.7 (osd.7) 225 : cluster 0 2.1c scrub starts 2024-04-02T05:03:12.143 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:11 smithi098 bash[23790]: cluster 2024-04-02T05:03:11.217306+0000 osd.7 (osd.7) 226 : cluster 0 2.1c scrub ok 2024-04-02T05:03:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:12.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:12.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.937290+0000 osd.6 (osd.6) 361 : cluster 0 2.17 scrub starts 2024-04-02T05:03:12.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:09.938872+0000 osd.6 (osd.6) 362 : cluster 0 2.17 scrub ok 2024-04-02T05:03:12.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.158602+0000 osd.1 (osd.1) 419 : cluster 0 6.e scrub starts 2024-04-02T05:03:12.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.160252+0000 osd.1 (osd.1) 420 : cluster 0 6.e scrub ok 2024-04-02T05:03:12.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.358310+0000 mgr.y (mgr.14538) 751 : cluster 0 pgmap v1315: 224 pgs: 3 active+undersized+remapped, 3 remapped+peering, 218 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 4.5 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 12 objects/s recovering 2024-04-02T05:03:12.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: audit 2024-04-02T05:03:10.490428+0000 mon.a (mon.0) 3948 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: audit 2024-04-02T05:03:10.490895+0000 mon.a (mon.0) 3949 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.507651+0000 osd.3 (osd.3) 425 : cluster 0 4.1b scrub starts 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.508981+0000 osd.3 (osd.3) 426 : cluster 0 4.1b scrub ok 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.594059+0000 osd.0 (osd.0) 413 : cluster 0 323.4s0 scrub starts 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.597089+0000 osd.0 (osd.0) 414 : cluster 0 323.4 scrub ok 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.729987+0000 osd.7 (osd.7) 223 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.730315+0000 osd.7 (osd.7) 224 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:11.213246+0000 osd.7 (osd.7) 225 : cluster 0 2.1c scrub starts 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[21528]: cluster 2024-04-02T05:03:11.217306+0000 osd.7 (osd.7) 226 : cluster 0 2.1c scrub ok 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.937290+0000 osd.6 (osd.6) 361 : cluster 0 2.17 scrub starts 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:09.938872+0000 osd.6 (osd.6) 362 : cluster 0 2.17 scrub ok 2024-04-02T05:03:12.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.158602+0000 osd.1 (osd.1) 419 : cluster 0 6.e scrub starts 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.160252+0000 osd.1 (osd.1) 420 : cluster 0 6.e scrub ok 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.358310+0000 mgr.y (mgr.14538) 751 : cluster 0 pgmap v1315: 224 pgs: 3 active+undersized+remapped, 3 remapped+peering, 218 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 4.5 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 12 objects/s recovering 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: audit 2024-04-02T05:03:10.490428+0000 mon.a (mon.0) 3948 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: audit 2024-04-02T05:03:10.490895+0000 mon.a (mon.0) 3949 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-144"}]: dispatch 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.507651+0000 osd.3 (osd.3) 425 : cluster 0 4.1b scrub starts 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.508981+0000 osd.3 (osd.3) 426 : cluster 0 4.1b scrub ok 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.594059+0000 osd.0 (osd.0) 413 : cluster 0 323.4s0 scrub starts 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.597089+0000 osd.0 (osd.0) 414 : cluster 0 323.4 scrub ok 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.729987+0000 osd.7 (osd.7) 223 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.730315+0000 osd.7 (osd.7) 224 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:11.213246+0000 osd.7 (osd.7) 225 : cluster 0 2.1c scrub starts 2024-04-02T05:03:12.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:11 smithi069 bash[28904]: cluster 2024-04-02T05:03:11.217306+0000 osd.7 (osd.7) 226 : cluster 0 2.1c scrub ok 2024-04-02T05:03:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:12.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:12.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:12.518 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:12.541 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:12.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:12.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:12.801 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:12.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:12.930 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:12.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:13.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:13.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.938440+0000 osd.6 (osd.6) 363 : cluster 0 1.5 scrub starts 2024-04-02T05:03:13.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:10.940824+0000 osd.6 (osd.6) 364 : cluster 0 1.5 scrub ok 2024-04-02T05:03:13.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:11.475544+0000 osd.3 (osd.3) 427 : cluster 0 1.11 scrub starts 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:11.477069+0000 osd.3 (osd.3) 428 : cluster 0 1.11 scrub ok 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:11.482118+0000 mon.a (mon.0) 3950 : cluster 0 osdmap e865: 8 total, 8 up, 7 in 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:12.359083+0000 mgr.y (mgr.14538) 752 : cluster 0 pgmap v1318: 192 pgs: 3 active+undersized+remapped, 3 remapped+peering, 186 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 9 B/s, 1 objects/s recovering 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: cluster 2024-04-02T05:03:12.487715+0000 mon.a (mon.0) 3951 : cluster 0 osdmap e866: 8 total, 8 up, 7 in 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[28904]: audit 2024-04-02T05:03:12.492187+0000 mon.a (mon.0) 3952 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.938440+0000 osd.6 (osd.6) 363 : cluster 0 1.5 scrub starts 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:10.940824+0000 osd.6 (osd.6) 364 : cluster 0 1.5 scrub ok 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:11.475544+0000 osd.3 (osd.3) 427 : cluster 0 1.11 scrub starts 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:11.477069+0000 osd.3 (osd.3) 428 : cluster 0 1.11 scrub ok 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:11.482118+0000 mon.a (mon.0) 3950 : cluster 0 osdmap e865: 8 total, 8 up, 7 in 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:12.359083+0000 mgr.y (mgr.14538) 752 : cluster 0 pgmap v1318: 192 pgs: 3 active+undersized+remapped, 3 remapped+peering, 186 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 9 B/s, 1 objects/s recovering 2024-04-02T05:03:13.044 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: cluster 2024-04-02T05:03:12.487715+0000 mon.a (mon.0) 3951 : cluster 0 osdmap e866: 8 total, 8 up, 7 in 2024-04-02T05:03:13.045 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:12 smithi069 bash[21528]: audit 2024-04-02T05:03:12.492187+0000 mon.a (mon.0) 3952 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:13.057 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:13.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:13.187 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.938440+0000 osd.6 (osd.6) 363 : cluster 0 1.5 scrub starts 2024-04-02T05:03:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:10.940824+0000 osd.6 (osd.6) 364 : cluster 0 1.5 scrub ok 2024-04-02T05:03:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:11.475544+0000 osd.3 (osd.3) 427 : cluster 0 1.11 scrub starts 2024-04-02T05:03:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:11.477069+0000 osd.3 (osd.3) 428 : cluster 0 1.11 scrub ok 2024-04-02T05:03:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:11.482118+0000 mon.a (mon.0) 3950 : cluster 0 osdmap e865: 8 total, 8 up, 7 in 2024-04-02T05:03:13.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:12.359083+0000 mgr.y (mgr.14538) 752 : cluster 0 pgmap v1318: 192 pgs: 3 active+undersized+remapped, 3 remapped+peering, 186 active+clean; 4.4 MiB data, 746 MiB used, 625 GiB / 626 GiB avail; 9 B/s, 1 objects/s recovering 2024-04-02T05:03:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: cluster 2024-04-02T05:03:12.487715+0000 mon.a (mon.0) 3951 : cluster 0 osdmap e866: 8 total, 8 up, 7 in 2024-04-02T05:03:13.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:12 smithi098 bash[23790]: audit 2024-04-02T05:03:12.492187+0000 mon.a (mon.0) 3952 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:13.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:13.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:13.517 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-02T05:03:13.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:13.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:13.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:13.829 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:13.924 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T05:03:13.924 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T05:03:13.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T05:03:13.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:13.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:03:13.993 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:13.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:12.250154+0000 osd.7 (osd.7) 227 : cluster 0 3.e scrub starts 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:12.265335+0000 osd.7 (osd.7) 228 : cluster 0 3.e scrub ok 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:13.046269+0000 osd.4 (osd.4) 414 : cluster 0 3.1c scrub starts 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:13.050526+0000 osd.4 (osd.4) 415 : cluster 0 3.1c scrub ok 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:13.433812+0000 osd.3 (osd.3) 429 : cluster 0 4.0 scrub starts 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:13.436698+0000 osd.3 (osd.3) 430 : cluster 0 4.0 scrub ok 2024-04-02T05:03:14.092 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: audit 2024-04-02T05:03:13.484258+0000 mon.a (mon.0) 3953 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: cluster 2024-04-02T05:03:13.495964+0000 mon.a (mon.0) 3954 : cluster 0 osdmap e867: 8 total, 8 up, 7 in 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: audit 2024-04-02T05:03:13.548406+0000 mon.a (mon.0) 3955 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[21528]: audit 2024-04-02T05:03:13.762854+0000 mgr.y (mgr.14538) 753 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:12.250154+0000 osd.7 (osd.7) 227 : cluster 0 3.e scrub starts 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:12.265335+0000 osd.7 (osd.7) 228 : cluster 0 3.e scrub ok 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:13.046269+0000 osd.4 (osd.4) 414 : cluster 0 3.1c scrub starts 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:13.050526+0000 osd.4 (osd.4) 415 : cluster 0 3.1c scrub ok 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:13.433812+0000 osd.3 (osd.3) 429 : cluster 0 4.0 scrub starts 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:13.436698+0000 osd.3 (osd.3) 430 : cluster 0 4.0 scrub ok 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: audit 2024-04-02T05:03:13.484258+0000 mon.a (mon.0) 3953 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: cluster 2024-04-02T05:03:13.495964+0000 mon.a (mon.0) 3954 : cluster 0 osdmap e867: 8 total, 8 up, 7 in 2024-04-02T05:03:14.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: audit 2024-04-02T05:03:13.548406+0000 mon.a (mon.0) 3955 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:03:14.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:13 smithi069 bash[28904]: audit 2024-04-02T05:03:13.762854+0000 mgr.y (mgr.14538) 753 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:14.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:14.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:12.250154+0000 osd.7 (osd.7) 227 : cluster 0 3.e scrub starts 2024-04-02T05:03:14.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:12.265335+0000 osd.7 (osd.7) 228 : cluster 0 3.e scrub ok 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:13.046269+0000 osd.4 (osd.4) 414 : cluster 0 3.1c scrub starts 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:13.050526+0000 osd.4 (osd.4) 415 : cluster 0 3.1c scrub ok 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:13.433812+0000 osd.3 (osd.3) 429 : cluster 0 4.0 scrub starts 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:13.436698+0000 osd.3 (osd.3) 430 : cluster 0 4.0 scrub ok 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: audit 2024-04-02T05:03:13.484258+0000 mon.a (mon.0) 3953 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: cluster 2024-04-02T05:03:13.495964+0000 mon.a (mon.0) 3954 : cluster 0 osdmap e867: 8 total, 8 up, 7 in 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: audit 2024-04-02T05:03:13.548406+0000 mon.a (mon.0) 3955 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:03:14.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:13 smithi098 bash[23790]: audit 2024-04-02T05:03:13.762854+0000 mgr.y (mgr.14538) 753 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:14.125 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:14.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:14.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:14.382 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:14.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:14.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:14.642 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:14.745 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:14.772 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:14.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:14.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:15.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:15.028 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:15.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:15.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:15.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:15.417 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:15.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:15.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.011143+0000 osd.4 (osd.4) 416 : cluster 0 2.18 scrub starts 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.013615+0000 osd.4 (osd.4) 417 : cluster 0 2.18 scrub ok 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.360274+0000 mgr.y (mgr.14538) 754 : cluster 0 pgmap v1321: 224 pgs: 27 creating+peering, 197 active+clean; 4.4 MiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: audit 2024-04-02T05:03:14.487018+0000 mon.a (mon.0) 3956 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.494817+0000 mon.a (mon.0) 3957 : cluster 0 osdmap e868: 8 total, 8 up, 7 in 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: audit 2024-04-02T05:03:14.496396+0000 mon.a (mon.0) 3958 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.875725+0000 osd.2 (osd.2) 364 : cluster 0 10.5 scrub starts 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.881592+0000 osd.2 (osd.2) 365 : cluster 0 10.5 scrub ok 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.942166+0000 mon.a (mon.0) 3959 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: audit 2024-04-02T05:03:15.118643+0000 mon.a (mon.0) 3960 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-146"}]': finished 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:15.123483+0000 mon.a (mon.0) 3961 : cluster 0 osdmap e869: 8 total, 8 up, 7 in 2024-04-02T05:03:15.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: audit 2024-04-02T05:03:15.125250+0000 mon.a (mon.0) 3962 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-146", "mode": "writeback"}]: dispatch 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:15.217009+0000 osd.7 (osd.7) 229 : cluster 0 3.15 scrub starts 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[21528]: cluster 2024-04-02T05:03:15.223791+0000 osd.7 (osd.7) 230 : cluster 0 3.15 scrub ok 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.011143+0000 osd.4 (osd.4) 416 : cluster 0 2.18 scrub starts 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.013615+0000 osd.4 (osd.4) 417 : cluster 0 2.18 scrub ok 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.360274+0000 mgr.y (mgr.14538) 754 : cluster 0 pgmap v1321: 224 pgs: 27 creating+peering, 197 active+clean; 4.4 MiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: audit 2024-04-02T05:03:14.487018+0000 mon.a (mon.0) 3956 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.494817+0000 mon.a (mon.0) 3957 : cluster 0 osdmap e868: 8 total, 8 up, 7 in 2024-04-02T05:03:15.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: audit 2024-04-02T05:03:14.496396+0000 mon.a (mon.0) 3958 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.875725+0000 osd.2 (osd.2) 364 : cluster 0 10.5 scrub starts 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.881592+0000 osd.2 (osd.2) 365 : cluster 0 10.5 scrub ok 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.942166+0000 mon.a (mon.0) 3959 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: audit 2024-04-02T05:03:15.118643+0000 mon.a (mon.0) 3960 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-146"}]': finished 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:15.123483+0000 mon.a (mon.0) 3961 : cluster 0 osdmap e869: 8 total, 8 up, 7 in 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: audit 2024-04-02T05:03:15.125250+0000 mon.a (mon.0) 3962 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-146", "mode": "writeback"}]: dispatch 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:15.217009+0000 osd.7 (osd.7) 229 : cluster 0 3.15 scrub starts 2024-04-02T05:03:15.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:15 smithi069 bash[28904]: cluster 2024-04-02T05:03:15.223791+0000 osd.7 (osd.7) 230 : cluster 0 3.15 scrub ok 2024-04-02T05:03:15.673 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:15.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:15.788 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.011143+0000 osd.4 (osd.4) 416 : cluster 0 2.18 scrub starts 2024-04-02T05:03:15.788 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.013615+0000 osd.4 (osd.4) 417 : cluster 0 2.18 scrub ok 2024-04-02T05:03:15.788 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.360274+0000 mgr.y (mgr.14538) 754 : cluster 0 pgmap v1321: 224 pgs: 27 creating+peering, 197 active+clean; 4.4 MiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: audit 2024-04-02T05:03:14.487018+0000 mon.a (mon.0) 3956 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.494817+0000 mon.a (mon.0) 3957 : cluster 0 osdmap e868: 8 total, 8 up, 7 in 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: audit 2024-04-02T05:03:14.496396+0000 mon.a (mon.0) 3958 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.875725+0000 osd.2 (osd.2) 364 : cluster 0 10.5 scrub starts 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.881592+0000 osd.2 (osd.2) 365 : cluster 0 10.5 scrub ok 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.942166+0000 mon.a (mon.0) 3959 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: audit 2024-04-02T05:03:15.118643+0000 mon.a (mon.0) 3960 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-72349-111", "overlaypool": "test-rados-api-smithi069-72349-146"}]': finished 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:15.123483+0000 mon.a (mon.0) 3961 : cluster 0 osdmap e869: 8 total, 8 up, 7 in 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: audit 2024-04-02T05:03:15.125250+0000 mon.a (mon.0) 3962 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-146", "mode": "writeback"}]: dispatch 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:15.217009+0000 osd.7 (osd.7) 229 : cluster 0 3.15 scrub starts 2024-04-02T05:03:15.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:15 smithi098 bash[23790]: cluster 2024-04-02T05:03:15.223791+0000 osd.7 (osd.7) 230 : cluster 0 3.15 scrub ok 2024-04-02T05:03:15.803 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:15.906 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:15.933 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:15.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:16.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:16.062 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:16.166 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:16.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:16.314 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:16.333 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:03:16.333 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T05:03:16.333 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T05:03:16.333 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T05:03:16.333 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:03:16.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:16.438 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:16.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.570 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:16.584 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:16.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.700 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:16.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.979764+0000 osd.4 (osd.4) 418 : cluster 0 4.9 scrub starts 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: cluster 2024-04-02T05:03:14.982732+0000 osd.4 (osd.4) 419 : cluster 0 4.9 scrub ok 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: cluster 2024-04-02T05:03:15.843496+0000 osd.2 (osd.2) 366 : cluster 0 2.3 scrub starts 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: cluster 2024-04-02T05:03:15.845669+0000 osd.2 (osd.2) 367 : cluster 0 2.3 scrub ok 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.119043+0000 mon.a (mon.0) 3963 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: audit 2024-04-02T05:03:16.121098+0000 mon.a (mon.0) 3964 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-146", "mode": "writeback"}]': finished 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.128284+0000 mon.a (mon.0) 3965 : cluster 0 osdmap e870: 8 total, 8 up, 7 in 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[21528]: audit 2024-04-02T05:03:16.201198+0000 mon.a (mon.0) 3966 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.979764+0000 osd.4 (osd.4) 418 : cluster 0 4.9 scrub starts 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: cluster 2024-04-02T05:03:14.982732+0000 osd.4 (osd.4) 419 : cluster 0 4.9 scrub ok 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: cluster 2024-04-02T05:03:15.843496+0000 osd.2 (osd.2) 366 : cluster 0 2.3 scrub starts 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: cluster 2024-04-02T05:03:15.845669+0000 osd.2 (osd.2) 367 : cluster 0 2.3 scrub ok 2024-04-02T05:03:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.119043+0000 mon.a (mon.0) 3963 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:03:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: audit 2024-04-02T05:03:16.121098+0000 mon.a (mon.0) 3964 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-146", "mode": "writeback"}]': finished 2024-04-02T05:03:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.128284+0000 mon.a (mon.0) 3965 : cluster 0 osdmap e870: 8 total, 8 up, 7 in 2024-04-02T05:03:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:16 smithi069 bash[28904]: audit 2024-04-02T05:03:16.201198+0000 mon.a (mon.0) 3966 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:03:16.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.979764+0000 osd.4 (osd.4) 418 : cluster 0 4.9 scrub starts 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: cluster 2024-04-02T05:03:14.982732+0000 osd.4 (osd.4) 419 : cluster 0 4.9 scrub ok 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: cluster 2024-04-02T05:03:15.843496+0000 osd.2 (osd.2) 366 : cluster 0 2.3 scrub starts 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: cluster 2024-04-02T05:03:15.845669+0000 osd.2 (osd.2) 367 : cluster 0 2.3 scrub ok 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.119043+0000 mon.a (mon.0) 3963 : cluster 3 Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: audit 2024-04-02T05:03:16.121098+0000 mon.a (mon.0) 3964 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-72349-146", "mode": "writeback"}]': finished 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.128284+0000 mon.a (mon.0) 3965 : cluster 0 osdmap e870: 8 total, 8 up, 7 in 2024-04-02T05:03:16.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:16 smithi098 bash[23790]: audit 2024-04-02T05:03:16.201198+0000 mon.a (mon.0) 3966 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-02T05:03:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:16.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.961 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:16.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:17.064 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:17.094 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:17.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:17.220 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:17.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:17.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:17.482 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:17.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:17.610 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:17.712 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.173435+0000 osd.1 (osd.1) 421 : cluster 0 10.19 scrub starts 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.175751+0000 osd.1 (osd.1) 422 : cluster 0 10.19 scrub ok 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.360742+0000 mgr.y (mgr.14538) 755 : cluster 0 pgmap v1325: 224 pgs: 27 creating+peering, 197 active+clean; 4.4 MiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.485277+0000 osd.0 (osd.0) 415 : cluster 0 323.5s0 scrub starts 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: cluster 2024-04-02T05:03:16.488406+0000 osd.0 (osd.0) 416 : cluster 0 323.5 scrub ok 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: audit 2024-04-02T05:03:17.129376+0000 mon.a (mon.0) 3967 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.134250+0000 mon.a (mon.0) 3968 : cluster 0 osdmap e871: 8 total, 8 up, 7 in 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[21528]: audit 2024-04-02T05:03:17.136077+0000 mon.a (mon.0) 3969 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.173435+0000 osd.1 (osd.1) 421 : cluster 0 10.19 scrub starts 2024-04-02T05:03:17.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.175751+0000 osd.1 (osd.1) 422 : cluster 0 10.19 scrub ok 2024-04-02T05:03:17.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.360742+0000 mgr.y (mgr.14538) 755 : cluster 0 pgmap v1325: 224 pgs: 27 creating+peering, 197 active+clean; 4.4 MiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:17.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.485277+0000 osd.0 (osd.0) 415 : cluster 0 323.5s0 scrub starts 2024-04-02T05:03:17.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: cluster 2024-04-02T05:03:16.488406+0000 osd.0 (osd.0) 416 : cluster 0 323.5 scrub ok 2024-04-02T05:03:17.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: audit 2024-04-02T05:03:17.129376+0000 mon.a (mon.0) 3967 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:03:17.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.134250+0000 mon.a (mon.0) 3968 : cluster 0 osdmap e871: 8 total, 8 up, 7 in 2024-04-02T05:03:17.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:17 smithi069 bash[28904]: audit 2024-04-02T05:03:17.136077+0000 mon.a (mon.0) 3969 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:03:17.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.173435+0000 osd.1 (osd.1) 421 : cluster 0 10.19 scrub starts 2024-04-02T05:03:17.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.175751+0000 osd.1 (osd.1) 422 : cluster 0 10.19 scrub ok 2024-04-02T05:03:17.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.360742+0000 mgr.y (mgr.14538) 755 : cluster 0 pgmap v1325: 224 pgs: 27 creating+peering, 197 active+clean; 4.4 MiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:17.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.485277+0000 osd.0 (osd.0) 415 : cluster 0 323.5s0 scrub starts 2024-04-02T05:03:17.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: cluster 2024-04-02T05:03:16.488406+0000 osd.0 (osd.0) 416 : cluster 0 323.5 scrub ok 2024-04-02T05:03:17.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: audit 2024-04-02T05:03:17.129376+0000 mon.a (mon.0) 3967 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_count","val": "2"}]': finished 2024-04-02T05:03:17.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.134250+0000 mon.a (mon.0) 3968 : cluster 0 osdmap e871: 8 total, 8 up, 7 in 2024-04-02T05:03:17.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:17 smithi098 bash[23790]: audit 2024-04-02T05:03:17.136077+0000 mon.a (mon.0) 3969 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-02T05:03:17.738 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:17.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:17.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:17.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:17.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:17.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:18.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:18.127 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:18.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:18.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:18.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:18.383 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:18.422 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:18.486 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:18.513 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:18.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-02T05:03:18.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:18.645 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.160238+0000 osd.1 (osd.1) 423 : cluster 0 4.4 scrub starts 2024-04-02T05:03:18.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.164901+0000 osd.1 (osd.1) 424 : cluster 0 4.4 scrub ok 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.278058+0000 osd.7 (osd.7) 231 : cluster 0 323.7s0 scrub starts 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.281275+0000 osd.7 (osd.7) 232 : cluster 0 323.7 scrub ok 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.402688+0000 osd.3 (osd.3) 431 : cluster 0 4.1a scrub starts 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.404867+0000 osd.3 (osd.3) 432 : cluster 0 4.1a scrub ok 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.893219+0000 osd.2 (osd.2) 368 : cluster 0 4.f scrub starts 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.894893+0000 osd.2 (osd.2) 369 : cluster 0 4.f scrub ok 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: audit 2024-04-02T05:03:18.131930+0000 mon.a (mon.0) 3970 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.139454+0000 mon.a (mon.0) 3971 : cluster 0 osdmap e872: 8 total, 8 up, 7 in 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21528]: audit 2024-04-02T05:03:18.142879+0000 mon.a (mon.0) 3972 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.160238+0000 osd.1 (osd.1) 423 : cluster 0 4.4 scrub starts 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.164901+0000 osd.1 (osd.1) 424 : cluster 0 4.4 scrub ok 2024-04-02T05:03:18.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.278058+0000 osd.7 (osd.7) 231 : cluster 0 323.7s0 scrub starts 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.281275+0000 osd.7 (osd.7) 232 : cluster 0 323.7 scrub ok 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.402688+0000 osd.3 (osd.3) 431 : cluster 0 4.1a scrub starts 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.404867+0000 osd.3 (osd.3) 432 : cluster 0 4.1a scrub ok 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.893219+0000 osd.2 (osd.2) 368 : cluster 0 4.f scrub starts 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.894893+0000 osd.2 (osd.2) 369 : cluster 0 4.f scrub ok 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: audit 2024-04-02T05:03:18.131930+0000 mon.a (mon.0) 3970 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.139454+0000 mon.a (mon.0) 3971 : cluster 0 osdmap e872: 8 total, 8 up, 7 in 2024-04-02T05:03:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[28904]: audit 2024-04-02T05:03:18.142879+0000 mon.a (mon.0) 3972 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:03:18.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:18.756 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:03:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:03:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:03:18.769 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:18.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.160238+0000 osd.1 (osd.1) 423 : cluster 0 4.4 scrub starts 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.164901+0000 osd.1 (osd.1) 424 : cluster 0 4.4 scrub ok 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.278058+0000 osd.7 (osd.7) 231 : cluster 0 323.7s0 scrub starts 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.281275+0000 osd.7 (osd.7) 232 : cluster 0 323.7 scrub ok 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.402688+0000 osd.3 (osd.3) 431 : cluster 0 4.1a scrub starts 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.404867+0000 osd.3 (osd.3) 432 : cluster 0 4.1a scrub ok 2024-04-02T05:03:18.884 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.893219+0000 osd.2 (osd.2) 368 : cluster 0 4.f scrub starts 2024-04-02T05:03:18.885 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.894893+0000 osd.2 (osd.2) 369 : cluster 0 4.f scrub ok 2024-04-02T05:03:18.885 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: audit 2024-04-02T05:03:18.131930+0000 mon.a (mon.0) 3970 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_period","val": "600"}]': finished 2024-04-02T05:03:18.885 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.139454+0000 mon.a (mon.0) 3971 : cluster 0 osdmap e872: 8 total, 8 up, 7 in 2024-04-02T05:03:18.885 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:18 smithi098 bash[23790]: audit 2024-04-02T05:03:18.142879+0000 mon.a (mon.0) 3972 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-02T05:03:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:18.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:19.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:19.025 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:19.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:19.158 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:19.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:19.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:19.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:19.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:19.548 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:19.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.911880+0000 osd.4 (osd.4) 420 : cluster 0 3.1d scrub starts 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:17.916090+0000 osd.4 (osd.4) 421 : cluster 0 3.1d scrub ok 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.202799+0000 osd.1 (osd.1) 425 : cluster 0 2.9 scrub starts 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.208091+0000 osd.1 (osd.1) 426 : cluster 0 2.9 scrub ok 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.361692+0000 mgr.y (mgr.14538) 756 : cluster 0 pgmap v1328: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 2.1 KiB/s rd, 793 B/s wr, 4 op/s 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.457782+0000 osd.3 (osd.3) 433 : cluster 0 4.10 scrub starts 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.461999+0000 osd.3 (osd.3) 434 : cluster 0 4.10 scrub ok 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:19.133037+0000 mon.a (mon.0) 3973 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: audit 2024-04-02T05:03:19.143152+0000 mon.a (mon.0) 3974 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: cluster 2024-04-02T05:03:19.148044+0000 mon.a (mon.0) 3975 : cluster 0 osdmap e873: 8 total, 8 up, 7 in 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: audit 2024-04-02T05:03:19.149071+0000 mon.a (mon.0) 3976 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T05:03:19.666 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: audit 2024-04-02T05:03:19.185095+0000 mon.a (mon.0) 3977 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:19.667 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:19 smithi098 bash[23790]: audit 2024-04-02T05:03:19.187911+0000 mon.b (mon.1) 319 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:19.677 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:19.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.911880+0000 osd.4 (osd.4) 420 : cluster 0 3.1d scrub starts 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:17.916090+0000 osd.4 (osd.4) 421 : cluster 0 3.1d scrub ok 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.202799+0000 osd.1 (osd.1) 425 : cluster 0 2.9 scrub starts 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.208091+0000 osd.1 (osd.1) 426 : cluster 0 2.9 scrub ok 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.361692+0000 mgr.y (mgr.14538) 756 : cluster 0 pgmap v1328: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 2.1 KiB/s rd, 793 B/s wr, 4 op/s 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.457782+0000 osd.3 (osd.3) 433 : cluster 0 4.10 scrub starts 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.461999+0000 osd.3 (osd.3) 434 : cluster 0 4.10 scrub ok 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:19.133037+0000 mon.a (mon.0) 3973 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: audit 2024-04-02T05:03:19.143152+0000 mon.a (mon.0) 3974 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: cluster 2024-04-02T05:03:19.148044+0000 mon.a (mon.0) 3975 : cluster 0 osdmap e873: 8 total, 8 up, 7 in 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: audit 2024-04-02T05:03:19.149071+0000 mon.a (mon.0) 3976 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: audit 2024-04-02T05:03:19.185095+0000 mon.a (mon.0) 3977 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[21528]: audit 2024-04-02T05:03:19.187911+0000 mon.b (mon.1) 319 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.911880+0000 osd.4 (osd.4) 420 : cluster 0 3.1d scrub starts 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:17.916090+0000 osd.4 (osd.4) 421 : cluster 0 3.1d scrub ok 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.202799+0000 osd.1 (osd.1) 425 : cluster 0 2.9 scrub starts 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.208091+0000 osd.1 (osd.1) 426 : cluster 0 2.9 scrub ok 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.361692+0000 mgr.y (mgr.14538) 756 : cluster 0 pgmap v1328: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 2.1 KiB/s rd, 793 B/s wr, 4 op/s 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.457782+0000 osd.3 (osd.3) 433 : cluster 0 4.10 scrub starts 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.461999+0000 osd.3 (osd.3) 434 : cluster 0 4.10 scrub ok 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:19.133037+0000 mon.a (mon.0) 3973 : cluster 1 Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: audit 2024-04-02T05:03:19.143152+0000 mon.a (mon.0) 3974 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: cluster 2024-04-02T05:03:19.148044+0000 mon.a (mon.0) 3975 : cluster 0 osdmap e873: 8 total, 8 up, 7 in 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: audit 2024-04-02T05:03:19.149071+0000 mon.a (mon.0) 3976 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: audit 2024-04-02T05:03:19.185095+0000 mon.a (mon.0) 3977 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:19 smithi069 bash[28904]: audit 2024-04-02T05:03:19.187911+0000 mon.b (mon.1) 319 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:19.807 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:19.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:19.937 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:19.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:20.028 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:20.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:20.061 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:20.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:20.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:20.320 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:20.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:20.448 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:20.552 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:20.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:20.693 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.912580+0000 osd.6 (osd.6) 365 : cluster 0 4.5 scrub starts 2024-04-02T05:03:20.693 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: cluster 2024-04-02T05:03:18.914228+0000 osd.6 (osd.6) 366 : cluster 0 4.5 scrub ok 2024-04-02T05:03:20.693 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: cluster 2024-04-02T05:03:19.288589+0000 osd.7 (osd.7) 233 : cluster 0 6.9 scrub starts 2024-04-02T05:03:20.693 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: cluster 2024-04-02T05:03:19.291766+0000 osd.7 (osd.7) 234 : cluster 0 6.9 scrub ok 2024-04-02T05:03:20.693 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: audit 2024-04-02T05:03:20.145267+0000 mon.a (mon.0) 3978 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T05:03:20.694 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: cluster 2024-04-02T05:03:20.153227+0000 mon.a (mon.0) 3979 : cluster 0 osdmap e874: 8 total, 8 up, 7 in 2024-04-02T05:03:20.694 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:20 smithi098 bash[23790]: audit 2024-04-02T05:03:20.153736+0000 mon.a (mon.0) 3980 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T05:03:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.912580+0000 osd.6 (osd.6) 365 : cluster 0 4.5 scrub starts 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: cluster 2024-04-02T05:03:18.914228+0000 osd.6 (osd.6) 366 : cluster 0 4.5 scrub ok 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: cluster 2024-04-02T05:03:19.288589+0000 osd.7 (osd.7) 233 : cluster 0 6.9 scrub starts 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: cluster 2024-04-02T05:03:19.291766+0000 osd.7 (osd.7) 234 : cluster 0 6.9 scrub ok 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: audit 2024-04-02T05:03:20.145267+0000 mon.a (mon.0) 3978 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: cluster 2024-04-02T05:03:20.153227+0000 mon.a (mon.0) 3979 : cluster 0 osdmap e874: 8 total, 8 up, 7 in 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[21528]: audit 2024-04-02T05:03:20.153736+0000 mon.a (mon.0) 3980 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.912580+0000 osd.6 (osd.6) 365 : cluster 0 4.5 scrub starts 2024-04-02T05:03:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: cluster 2024-04-02T05:03:18.914228+0000 osd.6 (osd.6) 366 : cluster 0 4.5 scrub ok 2024-04-02T05:03:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: cluster 2024-04-02T05:03:19.288589+0000 osd.7 (osd.7) 233 : cluster 0 6.9 scrub starts 2024-04-02T05:03:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: cluster 2024-04-02T05:03:19.291766+0000 osd.7 (osd.7) 234 : cluster 0 6.9 scrub ok 2024-04-02T05:03:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: audit 2024-04-02T05:03:20.145267+0000 mon.a (mon.0) 3978 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-02T05:03:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: cluster 2024-04-02T05:03:20.153227+0000 mon.a (mon.0) 3979 : cluster 0 osdmap e874: 8 total, 8 up, 7 in 2024-04-02T05:03:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:20 smithi069 bash[28904]: audit 2024-04-02T05:03:20.153736+0000 mon.a (mon.0) 3980 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-02T05:03:20.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:20.838 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:20.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:20.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:21.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:21.092 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:21.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:21.222 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:21.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:21.353 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:21.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:21.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:21.613 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:21.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: cluster 2024-04-02T05:03:20.227999+0000 osd.1 (osd.1) 427 : cluster 0 10.18 scrub starts 2024-04-02T05:03:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: cluster 2024-04-02T05:03:20.231161+0000 osd.1 (osd.1) 428 : cluster 0 10.18 scrub ok 2024-04-02T05:03:21.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: cluster 2024-04-02T05:03:20.304466+0000 osd.7 (osd.7) 235 : cluster 0 10.c scrub starts 2024-04-02T05:03:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: cluster 2024-04-02T05:03:20.307536+0000 osd.7 (osd.7) 236 : cluster 0 10.c scrub ok 2024-04-02T05:03:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: cluster 2024-04-02T05:03:20.362537+0000 mgr.y (mgr.14538) 757 : cluster 0 pgmap v1331: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: audit 2024-04-02T05:03:21.147598+0000 mon.a (mon.0) 3981 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "target_max_objects","val": "1"}]': finished 2024-04-02T05:03:21.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:21 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.153434+0000 mon.a (mon.0) 3982 : cluster 0 osdmap e875: 8 total, 8 up, 7 in 2024-04-02T05:03:21.744 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:21.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: cluster 2024-04-02T05:03:20.227999+0000 osd.1 (osd.1) 427 : cluster 0 10.18 scrub starts 2024-04-02T05:03:21.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: cluster 2024-04-02T05:03:20.231161+0000 osd.1 (osd.1) 428 : cluster 0 10.18 scrub ok 2024-04-02T05:03:21.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: cluster 2024-04-02T05:03:20.304466+0000 osd.7 (osd.7) 235 : cluster 0 10.c scrub starts 2024-04-02T05:03:21.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: cluster 2024-04-02T05:03:20.307536+0000 osd.7 (osd.7) 236 : cluster 0 10.c scrub ok 2024-04-02T05:03:21.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: cluster 2024-04-02T05:03:20.362537+0000 mgr.y (mgr.14538) 757 : cluster 0 pgmap v1331: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:21.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: audit 2024-04-02T05:03:21.147598+0000 mon.a (mon.0) 3981 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "target_max_objects","val": "1"}]': finished 2024-04-02T05:03:21.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.153434+0000 mon.a (mon.0) 3982 : cluster 0 osdmap e875: 8 total, 8 up, 7 in 2024-04-02T05:03:21.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: cluster 2024-04-02T05:03:20.227999+0000 osd.1 (osd.1) 427 : cluster 0 10.18 scrub starts 2024-04-02T05:03:21.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: cluster 2024-04-02T05:03:20.231161+0000 osd.1 (osd.1) 428 : cluster 0 10.18 scrub ok 2024-04-02T05:03:21.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: cluster 2024-04-02T05:03:20.304466+0000 osd.7 (osd.7) 235 : cluster 0 10.c scrub starts 2024-04-02T05:03:21.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: cluster 2024-04-02T05:03:20.307536+0000 osd.7 (osd.7) 236 : cluster 0 10.c scrub ok 2024-04-02T05:03:21.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: cluster 2024-04-02T05:03:20.362537+0000 mgr.y (mgr.14538) 757 : cluster 0 pgmap v1331: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:21.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: audit 2024-04-02T05:03:21.147598+0000 mon.a (mon.0) 3981 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-146","var": "target_max_objects","val": "1"}]': finished 2024-04-02T05:03:21.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:21 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.153434+0000 mon.a (mon.0) 3982 : cluster 0 osdmap e875: 8 total, 8 up, 7 in 2024-04-02T05:03:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:21.850 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:21.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:21.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:22.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:22.138 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:22.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:22.263 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:22.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:22.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:22.525 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:22.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:22.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:22 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.233308+0000 osd.1 (osd.1) 429 : cluster 0 2.10 scrub starts 2024-04-02T05:03:22.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:22 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.247295+0000 osd.1 (osd.1) 430 : cluster 0 2.10 scrub ok 2024-04-02T05:03:22.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:22 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.291298+0000 osd.7 (osd.7) 237 : cluster 0 1.9 scrub starts 2024-04-02T05:03:22.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:22 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.294390+0000 osd.7 (osd.7) 238 : cluster 0 1.9 scrub ok 2024-04-02T05:03:22.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:22 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.490443+0000 osd.0 (osd.0) 417 : cluster 0 10.6 scrub starts 2024-04-02T05:03:22.639 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:22 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.491666+0000 osd.0 (osd.0) 418 : cluster 0 10.6 scrub ok 2024-04-02T05:03:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:22.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:22.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.233308+0000 osd.1 (osd.1) 429 : cluster 0 2.10 scrub starts 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.247295+0000 osd.1 (osd.1) 430 : cluster 0 2.10 scrub ok 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.291298+0000 osd.7 (osd.7) 237 : cluster 0 1.9 scrub starts 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.294390+0000 osd.7 (osd.7) 238 : cluster 0 1.9 scrub ok 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.490443+0000 osd.0 (osd.0) 417 : cluster 0 10.6 scrub starts 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.491666+0000 osd.0 (osd.0) 418 : cluster 0 10.6 scrub ok 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.233308+0000 osd.1 (osd.1) 429 : cluster 0 2.10 scrub starts 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.247295+0000 osd.1 (osd.1) 430 : cluster 0 2.10 scrub ok 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.291298+0000 osd.7 (osd.7) 237 : cluster 0 1.9 scrub starts 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.294390+0000 osd.7 (osd.7) 238 : cluster 0 1.9 scrub ok 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.490443+0000 osd.0 (osd.0) 417 : cluster 0 10.6 scrub starts 2024-04-02T05:03:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:22 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.491666+0000 osd.0 (osd.0) 418 : cluster 0 10.6 scrub ok 2024-04-02T05:03:22.785 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:22.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:22.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:23.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:23.046 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:23.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.177 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:23.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.362 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:23.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:23.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.847767+0000 osd.4 (osd.4) 422 : cluster 0 6.14 scrub starts 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:21.850120+0000 osd.4 (osd.4) 423 : cluster 0 6.14 scrub ok 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.277221+0000 osd.1 (osd.1) 431 : cluster 0 6.19 scrub starts 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.278313+0000 osd.1 (osd.1) 432 : cluster 0 6.19 scrub ok 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.280875+0000 osd.7 (osd.7) 239 : cluster 0 6.1c scrub starts 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.283415+0000 osd.7 (osd.7) 240 : cluster 0 6.1c scrub ok 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.363330+0000 mgr.y (mgr.14538) 758 : cluster 0 pgmap v1333: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 484 B/s rd, 726 B/s wr, 2 op/s 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.501638+0000 mon.a (mon.0) 3983 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-02T05:03:23.608 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.862085+0000 osd.6 (osd.6) 367 : cluster 0 2.7 scrub starts 2024-04-02T05:03:23.609 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.864008+0000 osd.6 (osd.6) 368 : cluster 0 2.7 scrub ok 2024-04-02T05:03:23.609 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:23.239765+0000 osd.7 (osd.7) 241 : cluster 0 2.1d scrub starts 2024-04-02T05:03:23.609 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[23790]: cluster 2024-04-02T05:03:23.241585+0000 osd.7 (osd.7) 242 : cluster 0 2.1d scrub ok 2024-04-02T05:03:23.620 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:23.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:23.750 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:23.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.847767+0000 osd.4 (osd.4) 422 : cluster 0 6.14 scrub starts 2024-04-02T05:03:23.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:21.850120+0000 osd.4 (osd.4) 423 : cluster 0 6.14 scrub ok 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.277221+0000 osd.1 (osd.1) 431 : cluster 0 6.19 scrub starts 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.278313+0000 osd.1 (osd.1) 432 : cluster 0 6.19 scrub ok 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.280875+0000 osd.7 (osd.7) 239 : cluster 0 6.1c scrub starts 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.283415+0000 osd.7 (osd.7) 240 : cluster 0 6.1c scrub ok 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.363330+0000 mgr.y (mgr.14538) 758 : cluster 0 pgmap v1333: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 484 B/s rd, 726 B/s wr, 2 op/s 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.501638+0000 mon.a (mon.0) 3983 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.862085+0000 osd.6 (osd.6) 367 : cluster 0 2.7 scrub starts 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.864008+0000 osd.6 (osd.6) 368 : cluster 0 2.7 scrub ok 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:23.239765+0000 osd.7 (osd.7) 241 : cluster 0 2.1d scrub starts 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[21528]: cluster 2024-04-02T05:03:23.241585+0000 osd.7 (osd.7) 242 : cluster 0 2.1d scrub ok 2024-04-02T05:03:23.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.847767+0000 osd.4 (osd.4) 422 : cluster 0 6.14 scrub starts 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:21.850120+0000 osd.4 (osd.4) 423 : cluster 0 6.14 scrub ok 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.277221+0000 osd.1 (osd.1) 431 : cluster 0 6.19 scrub starts 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.278313+0000 osd.1 (osd.1) 432 : cluster 0 6.19 scrub ok 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.280875+0000 osd.7 (osd.7) 239 : cluster 0 6.1c scrub starts 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.283415+0000 osd.7 (osd.7) 240 : cluster 0 6.1c scrub ok 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.363330+0000 mgr.y (mgr.14538) 758 : cluster 0 pgmap v1333: 224 pgs: 224 active+clean; 4.4 MiB data, 751 MiB used, 625 GiB / 626 GiB avail; 484 B/s rd, 726 B/s wr, 2 op/s 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.501638+0000 mon.a (mon.0) 3983 : cluster 3 Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.862085+0000 osd.6 (osd.6) 367 : cluster 0 2.7 scrub starts 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.864008+0000 osd.6 (osd.6) 368 : cluster 0 2.7 scrub ok 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:23.239765+0000 osd.7 (osd.7) 241 : cluster 0 2.1d scrub starts 2024-04-02T05:03:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:23 smithi069 bash[28904]: cluster 2024-04-02T05:03:23.241585+0000 osd.7 (osd.7) 242 : cluster 0 2.1d scrub ok 2024-04-02T05:03:23.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.866 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:03:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:03:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:23.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:23.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:24.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:24.139 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:24.242 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:24.267 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:24.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:24.397 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:24.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:24.530 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:24.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:24.661 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.889277+0000 osd.4 (osd.4) 424 : cluster 0 2.f scrub starts 2024-04-02T05:03:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: cluster 2024-04-02T05:03:22.892809+0000 osd.4 (osd.4) 425 : cluster 0 2.f scrub ok 2024-04-02T05:03:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: cluster 2024-04-02T05:03:23.277896+0000 osd.1 (osd.1) 433 : cluster 0 3.0 scrub starts 2024-04-02T05:03:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: cluster 2024-04-02T05:03:23.281082+0000 osd.1 (osd.1) 434 : cluster 0 3.0 scrub ok 2024-04-02T05:03:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: audit 2024-04-02T05:03:23.772192+0000 mgr.y (mgr.14538) 759 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: cluster 2024-04-02T05:03:23.973984+0000 osd.2 (osd.2) 370 : cluster 0 1.14 scrub starts 2024-04-02T05:03:24.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:24 smithi098 bash[23790]: cluster 2024-04-02T05:03:23.976474+0000 osd.2 (osd.2) 371 : cluster 0 1.14 scrub ok 2024-04-02T05:03:24.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:24.792 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:24.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:24.921 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:24.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:25.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:25.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.889277+0000 osd.4 (osd.4) 424 : cluster 0 2.f scrub starts 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: cluster 2024-04-02T05:03:22.892809+0000 osd.4 (osd.4) 425 : cluster 0 2.f scrub ok 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: cluster 2024-04-02T05:03:23.277896+0000 osd.1 (osd.1) 433 : cluster 0 3.0 scrub starts 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: cluster 2024-04-02T05:03:23.281082+0000 osd.1 (osd.1) 434 : cluster 0 3.0 scrub ok 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: audit 2024-04-02T05:03:23.772192+0000 mgr.y (mgr.14538) 759 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: cluster 2024-04-02T05:03:23.973984+0000 osd.2 (osd.2) 370 : cluster 0 1.14 scrub starts 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[21528]: cluster 2024-04-02T05:03:23.976474+0000 osd.2 (osd.2) 371 : cluster 0 1.14 scrub ok 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.889277+0000 osd.4 (osd.4) 424 : cluster 0 2.f scrub starts 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: cluster 2024-04-02T05:03:22.892809+0000 osd.4 (osd.4) 425 : cluster 0 2.f scrub ok 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: cluster 2024-04-02T05:03:23.277896+0000 osd.1 (osd.1) 433 : cluster 0 3.0 scrub starts 2024-04-02T05:03:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: cluster 2024-04-02T05:03:23.281082+0000 osd.1 (osd.1) 434 : cluster 0 3.0 scrub ok 2024-04-02T05:03:25.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: audit 2024-04-02T05:03:23.772192+0000 mgr.y (mgr.14538) 759 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:25.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: cluster 2024-04-02T05:03:23.973984+0000 osd.2 (osd.2) 370 : cluster 0 1.14 scrub starts 2024-04-02T05:03:25.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:24 smithi069 bash[28904]: cluster 2024-04-02T05:03:23.976474+0000 osd.2 (osd.2) 371 : cluster 0 1.14 scrub ok 2024-04-02T05:03:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:25.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:25.178 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:25.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:25.310 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:25.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:25.441 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:25.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:25.674 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:25.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:25 smithi069 bash[28904]: cluster 2024-04-02T05:03:24.233972+0000 osd.1 (osd.1) 435 : cluster 0 1.3 scrub starts 2024-04-02T05:03:25.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:25 smithi069 bash[28904]: cluster 2024-04-02T05:03:24.236382+0000 osd.1 (osd.1) 436 : cluster 0 1.3 scrub ok 2024-04-02T05:03:25.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:25 smithi069 bash[28904]: cluster 2024-04-02T05:03:24.365698+0000 mgr.y (mgr.14538) 760 : cluster 0 pgmap v1334: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:25 smithi069 bash[21528]: cluster 2024-04-02T05:03:24.233972+0000 osd.1 (osd.1) 435 : cluster 0 1.3 scrub starts 2024-04-02T05:03:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:25 smithi069 bash[21528]: cluster 2024-04-02T05:03:24.236382+0000 osd.1 (osd.1) 436 : cluster 0 1.3 scrub ok 2024-04-02T05:03:25.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:25 smithi069 bash[21528]: cluster 2024-04-02T05:03:24.365698+0000 mgr.y (mgr.14538) 760 : cluster 0 pgmap v1334: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:25.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:25.829 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:25.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:25.957 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:25.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:25.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:25 smithi098 bash[23790]: cluster 2024-04-02T05:03:24.233972+0000 osd.1 (osd.1) 435 : cluster 0 1.3 scrub starts 2024-04-02T05:03:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:25 smithi098 bash[23790]: cluster 2024-04-02T05:03:24.236382+0000 osd.1 (osd.1) 436 : cluster 0 1.3 scrub ok 2024-04-02T05:03:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:25 smithi098 bash[23790]: cluster 2024-04-02T05:03:24.365698+0000 mgr.y (mgr.14538) 760 : cluster 0 pgmap v1334: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:26.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:26.087 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:26.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:26.215 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:26.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:26.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:26.474 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:26.577 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:26.600 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:26.651 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:26.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:26 smithi098 bash[23790]: cluster 2024-04-02T05:03:25.222979+0000 osd.1 (osd.1) 437 : cluster 0 1.12 scrub starts 2024-04-02T05:03:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:26 smithi098 bash[23790]: cluster 2024-04-02T05:03:25.225034+0000 osd.1 (osd.1) 438 : cluster 0 1.12 scrub ok 2024-04-02T05:03:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:26 smithi098 bash[23790]: cluster 2024-04-02T05:03:25.325769+0000 osd.3 (osd.3) 435 : cluster 0 6.7 scrub starts 2024-04-02T05:03:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:26 smithi098 bash[23790]: cluster 2024-04-02T05:03:25.327783+0000 osd.3 (osd.3) 436 : cluster 0 6.7 scrub ok 2024-04-02T05:03:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:26 smithi098 bash[23790]: audit 2024-04-02T05:03:26.329020+0000 mon.b (mon.1) 320 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:03:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:26.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[21528]: cluster 2024-04-02T05:03:25.222979+0000 osd.1 (osd.1) 437 : cluster 0 1.12 scrub starts 2024-04-02T05:03:26.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[21528]: cluster 2024-04-02T05:03:25.225034+0000 osd.1 (osd.1) 438 : cluster 0 1.12 scrub ok 2024-04-02T05:03:26.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[21528]: cluster 2024-04-02T05:03:25.325769+0000 osd.3 (osd.3) 435 : cluster 0 6.7 scrub starts 2024-04-02T05:03:26.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[21528]: cluster 2024-04-02T05:03:25.327783+0000 osd.3 (osd.3) 436 : cluster 0 6.7 scrub ok 2024-04-02T05:03:26.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[21528]: audit 2024-04-02T05:03:26.329020+0000 mon.b (mon.1) 320 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:03:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[28904]: cluster 2024-04-02T05:03:25.222979+0000 osd.1 (osd.1) 437 : cluster 0 1.12 scrub starts 2024-04-02T05:03:26.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[28904]: cluster 2024-04-02T05:03:25.225034+0000 osd.1 (osd.1) 438 : cluster 0 1.12 scrub ok 2024-04-02T05:03:26.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[28904]: cluster 2024-04-02T05:03:25.325769+0000 osd.3 (osd.3) 435 : cluster 0 6.7 scrub starts 2024-04-02T05:03:26.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[28904]: cluster 2024-04-02T05:03:25.327783+0000 osd.3 (osd.3) 436 : cluster 0 6.7 scrub ok 2024-04-02T05:03:26.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:26 smithi069 bash[28904]: audit 2024-04-02T05:03:26.329020+0000 mon.b (mon.1) 320 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:03:26.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:26.848 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:26.951 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:26.971 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:26.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:27.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:27.096 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:27.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:27.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:27.350 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:27.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:27.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:27.608 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:27.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:27.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:27 smithi098 bash[23790]: cluster 2024-04-02T05:03:26.366192+0000 mgr.y (mgr.14538) 761 : cluster 0 pgmap v1335: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 709 B/s rd, 0 B/s wr, 0 op/s 2024-04-02T05:03:27.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:27 smithi098 bash[23790]: cluster 2024-04-02T05:03:26.527109+0000 osd.0 (osd.0) 419 : cluster 0 10.1a scrub starts 2024-04-02T05:03:27.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:27 smithi098 bash[23790]: cluster 2024-04-02T05:03:26.530028+0000 osd.0 (osd.0) 420 : cluster 0 10.1a scrub ok 2024-04-02T05:03:27.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:27 smithi098 bash[23790]: audit 2024-04-02T05:03:26.999160+0000 mon.b (mon.1) 321 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:03:27.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:27 smithi098 bash[23790]: audit 2024-04-02T05:03:27.000349+0000 mon.b (mon.1) 322 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:03:27.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:27 smithi098 bash[23790]: audit 2024-04-02T05:03:27.006515+0000 mon.a (mon.0) 3984 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:27.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:27.868 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:27.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:27.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[21528]: cluster 2024-04-02T05:03:26.366192+0000 mgr.y (mgr.14538) 761 : cluster 0 pgmap v1335: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 709 B/s rd, 0 B/s wr, 0 op/s 2024-04-02T05:03:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[21528]: cluster 2024-04-02T05:03:26.527109+0000 osd.0 (osd.0) 419 : cluster 0 10.1a scrub starts 2024-04-02T05:03:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[21528]: cluster 2024-04-02T05:03:26.530028+0000 osd.0 (osd.0) 420 : cluster 0 10.1a scrub ok 2024-04-02T05:03:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[21528]: audit 2024-04-02T05:03:26.999160+0000 mon.b (mon.1) 321 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:03:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[21528]: audit 2024-04-02T05:03:27.000349+0000 mon.b (mon.1) 322 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:03:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[21528]: audit 2024-04-02T05:03:27.006515+0000 mon.a (mon.0) 3984 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[28904]: cluster 2024-04-02T05:03:26.366192+0000 mgr.y (mgr.14538) 761 : cluster 0 pgmap v1335: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 709 B/s rd, 0 B/s wr, 0 op/s 2024-04-02T05:03:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[28904]: cluster 2024-04-02T05:03:26.527109+0000 osd.0 (osd.0) 419 : cluster 0 10.1a scrub starts 2024-04-02T05:03:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[28904]: cluster 2024-04-02T05:03:26.530028+0000 osd.0 (osd.0) 420 : cluster 0 10.1a scrub ok 2024-04-02T05:03:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[28904]: audit 2024-04-02T05:03:26.999160+0000 mon.b (mon.1) 321 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:03:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[28904]: audit 2024-04-02T05:03:27.000349+0000 mon.b (mon.1) 322 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:03:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:27 smithi069 bash[28904]: audit 2024-04-02T05:03:27.006515+0000 mon.a (mon.0) 3984 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:28.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:28.128 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:28.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:28.258 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:28.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:28.386 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:28.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:28.517 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:28.620 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:28.633 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:28 smithi069 bash[28904]: cluster 2024-04-02T05:03:27.359558+0000 osd.3 (osd.3) 437 : cluster 0 3.2 scrub starts 2024-04-02T05:03:28.633 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:28 smithi069 bash[28904]: cluster 2024-04-02T05:03:27.362777+0000 osd.3 (osd.3) 438 : cluster 0 3.2 scrub ok 2024-04-02T05:03:28.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:28 smithi069 bash[21528]: cluster 2024-04-02T05:03:27.359558+0000 osd.3 (osd.3) 437 : cluster 0 3.2 scrub starts 2024-04-02T05:03:28.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:28 smithi069 bash[21528]: cluster 2024-04-02T05:03:27.362777+0000 osd.3 (osd.3) 438 : cluster 0 3.2 scrub ok 2024-04-02T05:03:28.649 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:28.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:28.779 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:28.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:28.892 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:03:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:03:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:03:28.907 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:28.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:28 smithi098 bash[23790]: cluster 2024-04-02T05:03:27.359558+0000 osd.3 (osd.3) 437 : cluster 0 3.2 scrub starts 2024-04-02T05:03:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:28 smithi098 bash[23790]: cluster 2024-04-02T05:03:27.362777+0000 osd.3 (osd.3) 438 : cluster 0 3.2 scrub ok 2024-04-02T05:03:29.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:29.035 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:29.138 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:29.165 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:29.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:29.296 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:29.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:29.427 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/989187379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e806ab80 msgr2=0x7fc0e8106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- 172.21.15.69:0/989187379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e806ab80 0x7fc0e8106ef0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0099c0 tx=0x7fc0dc009cd0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/989187379 shutdown_connections 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- 172.21.15.69:0/989187379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e8107620 0x7fc0e810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- 172.21.15.69:0/989187379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e806a1d0 0x7fc0e806a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- 172.21.15.69:0/989187379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e806ab80 0x7fc0e8106ef0 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/989187379 >> 172.21.15.69:0/989187379 conn(0x7fc0e80771e0 msgr2=0x7fc0e80775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/989187379 shutdown_connections 2024-04-02T05:03:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/989187379 wait complete. 2024-04-02T05:03:29.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 librados: starting msgr at 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 librados: starting objecter 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 Processor -- start 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- start start 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 librados: setting wanted keys 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 librados: calling monclient init 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e806a1d0 0x7fc0e8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e806ab80 0x7fc0e81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e8107620 0x7fc0e819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e8111ab0 con 0x7fc0e806a1d0 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e8111930 con 0x7fc0e806ab80 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e8111c30 con 0x7fc0e8107620 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e806a1d0 0x7fc0e8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e8107620 0x7fc0e819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e806a1d0 0x7fc0e8198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46096/0 (socket says 172.21.15.69:46096) 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e659c700 1 -- 172.21.15.69:0/2849712394 learned_addr learned my addr 172.21.15.69:0/2849712394 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e806ab80 0x7fc0e81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e806ab80 msgr2=0x7fc0e81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0e806ab80 0x7fc0e81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e806a1d0 msgr2=0x7fc0e8198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e806a1d0 0x7fc0e8198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e819d090 con 0x7fc0e8107620 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e8107620 0x7fc0e819c950 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000efb0 tx=0x7fc0e000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0d77fe700 10 client.?.objecter ms_handle_connect 0x7fc0e8107620 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0d77fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0003a00 con 0x7fc0e8107620 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc0e000ce20 con 0x7fc0e8107620 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e819d380 con 0x7fc0e8107620 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0004670 con 0x7fc0e8107620 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 10 client.43865.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e81011a0 con 0x7fc0e8107620 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 librados: init done 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.446+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc0e81039d0 con 0x7fc0e8107620 2024-04-02T05:03:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc0e0012050 con 0x7fc0e8107620 2024-04-02T05:03:29.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc0d006f5e0 0x7fc0d0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(875..875 src has 74..875) ==== 10136+0+0 (secure 0 0 0) 0x7fc0e008e950 con 0x7fc0e8107620 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 osd_map(875..875 src has 74..875) 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 3 client.43865.objecter handle_osd_map got epochs [875,875] > 0 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 3 client.43865.objecter handle_osd_map decoding full epoch 875 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc0d006f5e0 0x7fc0d0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 20 client.43865.objecter dump_active .. 0 homeless 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v875) ==== 54+0+16 (secure 0 0 0) 0x7fc0e0022060 con 0x7fc0e8107620 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 10 client.43865.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 1 2024-04-02T05:03:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fc0e8062e50 0x7fc0e8063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 1 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 0 1 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 4 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 10 client.43865.objecter _send_command 1 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0e81993e0 con 0x7fc0e8062e50 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fc0e8062e50 0x7fc0e8063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc0d006f5e0 0x7fc0d0071aa0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800bd80 tx=0x7fc0d800b480 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0d006f5e0 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e8062e50 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fc0e8062e50 0x7fc0e8063230 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e81993e0 con 0x7fc0e8062e50 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 1: 0 ) 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 1 = system:0 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 4 2024-04-02T05:03:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.450+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 0 1 2024-04-02T05:03:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 10 client.43865.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 2 2024-04-02T05:03:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 2 2024-04-02T05:03:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e804ff00 osd=0 3 2024-04-02T05:03:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 0 2 2024-04-02T05:03:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 4 2024-04-02T05:03:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 10 client.43865.objecter _send_command 2 2024-04-02T05:03:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e80628d0 con 0x7fc0e8062e50 2024-04-02T05:03:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e80628d0 con 0x7fc0e8062e50 2024-04-02T05:03:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 4 2024-04-02T05:03:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 0 2 2024-04-02T05:03:29.476 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:03:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 10 client.43865.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 3 2024-04-02T05:03:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fc0e8109d50 0x7fc0e8101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 1 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 4 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 10 client.43865.objecter _send_command 3 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.470+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc0e81030e0 con 0x7fc0e8109d50 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fc0e8109d50 0x7fc0e8101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fc0e8109d50 0x7fc0e8101400 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e8109d50 2024-04-02T05:03:29.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e81030e0 con 0x7fc0e8109d50 2024-04-02T05:03:29.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 3: 0 ) 2024-04-02T05:03:29.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 3 = system:0 2024-04-02T05:03:29.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 4 2024-04-02T05:03:29.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.474+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 1 3 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 10 client.43865.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 4 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 4 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e802db30 osd=1 3 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 1 4 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 4 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 10 client.43865.objecter _send_command 4 2024-04-02T05:03:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.490+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e802def0 con 0x7fc0e8109d50 2024-04-02T05:03:29.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e802def0 con 0x7fc0e8109d50 2024-04-02T05:03:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 4 2024-04-02T05:03:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 1 4 2024-04-02T05:03:29.497 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 10 client.43865.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 5 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc0e8109430 0x7fc0e802d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 5 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 2 5 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 4 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 10 client.43865.objecter _send_command 5 2024-04-02T05:03:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc0e802d6c0 con 0x7fc0e8109430 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc0e8109430 0x7fc0e802d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e8109430 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc0e8109430 0x7fc0e802d180 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e802d6c0 con 0x7fc0e8109430 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 5: 0 ) 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 5 = system:0 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 4 2024-04-02T05:03:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.494+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 2 5 2024-04-02T05:03:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- 172.21.15.69:0/172307731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c1082f0 msgr2=0x7f023c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- 172.21.15.69:0/172307731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c1082f0 0x7f023c1086d0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f022c009990 tx=0x7f022c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- 172.21.15.69:0/172307731 shutdown_connections 2024-04-02T05:03:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- 172.21.15.69:0/172307731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c06a910 0x7f023c10f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- 172.21.15.69:0/172307731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c1082f0 0x7f023c1086d0 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- 172.21.15.69:0/172307731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f023c069f70 0x7f023c06a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- 172.21.15.69:0/172307731 >> 172.21.15.69:0/172307731 conn(0x7f023c0771e0 msgr2=0x7f023c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:29.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- 172.21.15.69:0/172307731 shutdown_connections 2024-04-02T05:03:29.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- 172.21.15.69:0/172307731 wait complete. 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 librados: starting msgr at 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 librados: starting objecter 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 Processor -- start 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- start start 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 librados: setting wanted keys 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 librados: calling monclient init 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 0x7f023c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c06a910 0x7f023c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f023c1082f0 0x7f023c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f023c111a20 con 0x7f023c06a910 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f023c1118a0 con 0x7f023c1082f0 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0240f90700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f023c111ba0 con 0x7f023c069f70 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023a59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 0x7f023c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023a59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 0x7f023c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60442/0 (socket says 172.21.15.69:60442) 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023a59c700 1 -- 172.21.15.69:0/1762919814 learned_addr learned my addr 172.21.15.69:0/1762919814 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f0239d9b700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c06a910 0x7f023c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023ad9d700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f023c1082f0 0x7f023c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023a59c700 1 -- 172.21.15.69:0/1762919814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f023c1082f0 msgr2=0x7f023c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023a59c700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f023c1082f0 0x7f023c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.506+0000 7f023a59c700 1 -- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c06a910 msgr2=0x7f023c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f023a59c700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c06a910 0x7f023c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f023a59c700 1 -- 172.21.15.69:0/1762919814 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c19d000 con 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f0239d9b700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c06a910 0x7f023c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f023a59c700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 0x7f023c197ff0 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7f022c000c00 tx=0x7f022c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 10 client.?.objecter ms_handle_connect 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c027070 con 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f022c022470 con 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f0240f90700 1 -- 172.21.15.69:0/1762919814 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c19d290 con 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f0240f90700 10 client.43871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f0240f90700 1 -- 172.21.15.69:0/1762919814 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c19d700 con 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c017650 con 0x7f023c069f70 2024-04-02T05:03:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f0240f90700 1 librados: init done 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0220005120 con 0x7f023c069f70 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f022c02a070 con 0x7f023c069f70 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f022406f5e0 0x7f0224071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(875..875 src has 74..875) ==== 10136+0+0 (secure 0 0 0) 0x7f022c08fdc0 con 0x7f023c069f70 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 10 client.43871.objecter ms_dispatch 0x7f023c001550 osd_map(875..875 src has 74..875) 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 3 client.43871.objecter handle_osd_map got epochs [875,875] > 0 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 3 client.43871.objecter handle_osd_map decoding full epoch 875 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.510+0000 7f022b7fe700 20 client.43871.objecter dump_active .. 0 homeless 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7f0239d9b700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f022406f5e0 0x7f0224071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7f0239d9b700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f022406f5e0 0x7f0224071aa0 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f023c077b30 tx=0x7f023000a380 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7f022b7fe700 10 client.43871.objecter ms_handle_connect 0x7f022406f5e0 2024-04-02T05:03:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 10 client.43865.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 6 2024-04-02T05:03:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 6 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81092c0 osd=2 3 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 2 6 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 4 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 10 client.43865.objecter _send_command 6 2024-04-02T05:03:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.514+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e81af040 con 0x7fc0e8109430 2024-04-02T05:03:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e81af040 con 0x7fc0e8109430 2024-04-02T05:03:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 4 2024-04-02T05:03:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 2 6 2024-04-02T05:03:29.522 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:03:29.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 10 client.43865.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 7 2024-04-02T05:03:29.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fc0e802cd70 0x7fc0e81b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f022c013070 con 0x7f023c069f70 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 7 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 3 7 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 4 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 10 client.43865.objecter _send_command 7 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc0e8102780 con 0x7fc0e802cd70 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fc0e802cd70 0x7fc0e81b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e802cd70 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fc0e802cd70 0x7fc0e81b23e0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e8102780 con 0x7fc0e802cd70 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 7: 0 ) 2024-04-02T05:03:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 7 = system:0 2024-04-02T05:03:29.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 4 2024-04-02T05:03:29.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.518+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 3 7 2024-04-02T05:03:29.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 10 client.43865.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 8 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 8 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e802cc00 osd=3 3 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 3 8 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 4 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 10 client.43865.objecter _send_command 8 2024-04-02T05:03:29.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.538+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e8101b90 con 0x7fc0e802cd70 2024-04-02T05:03:29.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e8101b90 con 0x7fc0e802cd70 2024-04-02T05:03:29.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 4 2024-04-02T05:03:29.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 3 8 2024-04-02T05:03:29.547 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:03:29.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 10 client.43865.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 9 2024-04-02T05:03:29.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fc0e81afed0 0x7fc0e81b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 9 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 4 9 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 4 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 10 client.43865.objecter _send_command 9 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc0e81b8910 con 0x7fc0e81afed0 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fc0e81afed0 0x7fc0e81b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e81afed0 2024-04-02T05:03:29.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.542+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fc0e81afed0 0x7fc0e81b5ab0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.546+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e81b8910 con 0x7fc0e81afed0 2024-04-02T05:03:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.546+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 9: 0 ) 2024-04-02T05:03:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.546+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 9 = system:0 2024-04-02T05:03:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.546+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 4 2024-04-02T05:03:29.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.546+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 4 9 2024-04-02T05:03:29.557 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 10 client.43865.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 10 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 10 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e8101de0 osd=4 3 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 4 10 2024-04-02T05:03:29.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 4 2024-04-02T05:03:29.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 10 client.43865.objecter _send_command 10 2024-04-02T05:03:29.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.562+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e81b5ff0 con 0x7fc0e81afed0 2024-04-02T05:03:29.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e81b5ff0 con 0x7fc0e81afed0 2024-04-02T05:03:29.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 4 2024-04-02T05:03:29.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 4 10 2024-04-02T05:03:29.571 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 10 client.43865.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 11 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fc0e81b6710 0x7fc0e81b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 11 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 5 11 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 4 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 10 client.43865.objecter _send_command 11 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc0e81bbb30 con 0x7fc0e81b6710 2024-04-02T05:03:29.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.566+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fc0e81b6710 0x7fc0e81b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0e5d9b700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fc0e81b6710 0x7fc0e81b8b00 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e81b6710 2024-04-02T05:03:29.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e81bbb30 con 0x7fc0e81b6710 2024-04-02T05:03:29.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 11: 0 ) 2024-04-02T05:03:29.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 11 = system:0 2024-04-02T05:03:29.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 4 2024-04-02T05:03:29.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.570+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 5 11 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 10 client.43865.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 12 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 12 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81b5ff0 osd=5 3 2024-04-02T05:03:29.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 5 12 2024-04-02T05:03:29.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 4 2024-04-02T05:03:29.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 10 client.43865.objecter _send_command 12 2024-04-02T05:03:29.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.590+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e81bbb30 con 0x7fc0e81b6710 2024-04-02T05:03:29.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e81bbb30 con 0x7fc0e81b6710 2024-04-02T05:03:29.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 4 2024-04-02T05:03:29.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 5 12 2024-04-02T05:03:29.597 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:03:29.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 10 client.43865.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 13 2024-04-02T05:03:29.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fc0e81baec0 0x7fc0e81bb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 13 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 6 13 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 4 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 10 client.43865.objecter _send_command 13 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc0e81beed0 con 0x7fc0e81baec0 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fc0e81baec0 0x7fc0e81bb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e81baec0 2024-04-02T05:03:29.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0e6d9d700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fc0e81baec0 0x7fc0e81bb320 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e81beed0 con 0x7fc0e81baec0 2024-04-02T05:03:29.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 13: 0 ) 2024-04-02T05:03:29.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 13 = system:0 2024-04-02T05:03:29.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 4 2024-04-02T05:03:29.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.594+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 6 13 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 10 client.43865.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 14 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 14 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81bbb30 osd=6 3 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 6 14 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 4 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 10 client.43865.objecter _send_command 14 2024-04-02T05:03:29.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e81b9130 con 0x7fc0e81baec0 2024-04-02T05:03:29.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e81b9130 con 0x7fc0e81baec0 2024-04-02T05:03:29.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 4 2024-04-02T05:03:29.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 6 14 2024-04-02T05:03:29.619 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 10 client.43865.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 15 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fc0e81b9640 0x7fc0e81b9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 15 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 7 15 2024-04-02T05:03:29.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 4 2024-04-02T05:03:29.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 10 client.43865.objecter _send_command 15 2024-04-02T05:03:29.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.614+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc0e81b9fe0 con 0x7fc0e81b9640 2024-04-02T05:03:29.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fc0e81b9640 0x7fc0e81b9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0d77fe700 10 client.43865.objecter ms_handle_connect 0x7fc0e81b9640 2024-04-02T05:03:29.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0e659c700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fc0e81b9640 0x7fc0e81b9aa0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e81b9fe0 con 0x7fc0e81b9640 2024-04-02T05:03:29.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 15: 0 ) 2024-04-02T05:03:29.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 15 = system:0 2024-04-02T05:03:29.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 4 2024-04-02T05:03:29.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.618+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 7 15 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 10 client.43865.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign -1 16 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 20 client.43865.objecter _get_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_remove -1 16 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 20 client.43865.objecter get_session s=0x7fc0e81ba490 osd=7 3 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 15 client.43865.objecter _session_command_op_assign 7 16 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 4 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 10 client.43865.objecter _send_command 16 2024-04-02T05:03:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.638+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc0e81b9fe0 con 0x7fc0e81b9640 2024-04-02T05:03:29.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0d77fe700 1 -- 172.21.15.69:0/2849712394 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc0e81b9fe0 con 0x7fc0e81b9640 2024-04-02T05:03:29.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0d77fe700 10 client.43865.objecter ms_dispatch 0x7fc0e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:29.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0d77fe700 10 client.43865.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:29.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0d77fe700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 4 2024-04-02T05:03:29.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0d77fe700 15 client.43865.objecter _session_command_op_remove 7 16 2024-04-02T05:03:29.647 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:03:29.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 librados: watch_flush enter 2024-04-02T05:03:29.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 librados: watch_flush exit 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.0 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fc0e8062e50 msgr2=0x7fc0e8063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fc0e8062e50 0x7fc0e8063230 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e804ff00 osd=0 1 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.1 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fc0e8109d50 msgr2=0x7fc0e8101400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7fc0e8109d50 0x7fc0e8101400 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802db30 osd=1 1 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.2 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc0e8109430 msgr2=0x7fc0e802d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7fc0e8109430 0x7fc0e802d180 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0e6d9d700 1 -- 172.21.15.69:0/2849712394 reap_dead start 2024-04-02T05:03:29.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81092c0 osd=2 1 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.3 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fc0e802cd70 msgr2=0x7fc0e81b23e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7fc0e802cd70 0x7fc0e81b23e0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e802cc00 osd=3 1 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.4 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fc0e81afed0 msgr2=0x7fc0e81b5ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7fc0e81afed0 0x7fc0e81b5ab0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e8101de0 osd=4 1 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.5 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fc0e81b6710 msgr2=0x7fc0e81b8b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7fc0e81b6710 0x7fc0e81b8b00 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81b5ff0 osd=5 1 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.6 2024-04-02T05:03:29.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fc0e81baec0 msgr2=0x7fc0e81bb320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7fc0e81baec0 0x7fc0e81bb320 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81bbb30 osd=6 1 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter close_session for osd.7 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fc0e81b9640 msgr2=0x7fc0e81b9aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7fc0e81b9640 0x7fc0e81b9aa0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter put_session s=0x7fc0e81ba490 osd=7 1 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 20 client.43865.objecter shutdown clearing up homeless session... 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 10 client.43865.objecter successfully canceled tick 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0e6d9d700 1 -- 172.21.15.69:0/2849712394 reap_dead start 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.642+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc0d006f5e0 msgr2=0x7fc0d0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc0d006f5e0 0x7fc0d0071aa0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800bd80 tx=0x7fc0d800b480 comp rx=0 tx=0).stop 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e8107620 msgr2=0x7fc0e819c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e8107620 0x7fc0e819c950 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000efb0 tx=0x7fc0e000c3a0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 shutdown_connections 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e8107620 0x7fc0e819c950 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 --2- 172.21.15.69:0/2849712394 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc0d006f5e0 0x7fc0d0071aa0 unknown :-1 s=CLOSED pgs=4024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 >> 172.21.15.69:0/2849712394 conn(0x7fc0e80771e0 msgr2=0x7fc0e80ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 shutdown_connections 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 -- 172.21.15.69:0/2849712394 wait complete. 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.646+0000 7fc0ec846700 1 librados: shutdown 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:03:29.652 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:03:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:03:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:03:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:03:29.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:29.681 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- 172.21.15.69:0/4189628782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24100c10 msgr2=0x7fad24112ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- 172.21.15.69:0/4189628782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24100c10 0x7fad24112ad0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7fad18009990 tx=0x7fad18009ca0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- 172.21.15.69:0/4189628782 shutdown_connections 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- 172.21.15.69:0/4189628782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad24113010 0x7fad24115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- 172.21.15.69:0/4189628782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad241002f0 0x7fad241006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- 172.21.15.69:0/4189628782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24100c10 0x7fad24112ad0 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- 172.21.15.69:0/4189628782 >> 172.21.15.69:0/4189628782 conn(0x7fad240782b0 msgr2=0x7fad240fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:29.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- 172.21.15.69:0/4189628782 shutdown_connections 2024-04-02T05:03:29.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- 172.21.15.69:0/4189628782 wait complete. 2024-04-02T05:03:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 librados: starting msgr at 2024-04-02T05:03:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 librados: starting objecter 2024-04-02T05:03:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 Processor -- start 2024-04-02T05:03:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- start start 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 librados: setting wanted keys 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 librados: calling monclient init 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad241002f0 0x7fad2419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24100c10 0x7fad2419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 0x7fad241a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad241180c0 con 0x7fad241002f0 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad24117f40 con 0x7fad24100c10 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fad24118240 con 0x7fad24113010 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 0x7fad241a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 0x7fad241a2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:60446/0 (socket says 172.21.15.69:60446) 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 -- 172.21.15.69:0/1517226117 learned_addr learned my addr 172.21.15.69:0/1517226117 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad2259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad241002f0 0x7fad2419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 -- 172.21.15.69:0/1517226117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24100c10 msgr2=0x7fad2419ec00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24100c10 0x7fad2419ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 -- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad241002f0 msgr2=0x7fad2419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad241002f0 0x7fad2419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 -- 172.21.15.69:0/1517226117 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad241a36d0 con 0x7fad24113010 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad137fe700 10 client.?.objecter ms_handle_connect 0x7fad24113010 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad137fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad22d9d700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 0x7fad241a2f90 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c110 tx=0x7fad1c00c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0092e0 con 0x7fad24113010 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fad1c003a00 con 0x7fad24113010 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad241a3960 con 0x7fad24113010 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad28d19700 10 client.43877.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.710+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0045b0 con 0x7fad24113010 2024-04-02T05:03:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.714+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241ab240 con 0x7fad24113010 2024-04-02T05:03:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.714+0000 7fad28d19700 1 librados: init done 2024-04-02T05:03:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.714+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad2404ff00 con 0x7fad24113010 2024-04-02T05:03:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.714+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fad1c01a020 con 0x7fad24113010 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fad0c06f5e0 0x7fad0c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(875..875 src has 74..875) ==== 10136+0+0 (secure 0 0 0) 0x7fad1c090420 con 0x7fad24113010 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 10 client.43877.objecter ms_dispatch 0x7fad24001550 osd_map(875..875 src has 74..875) 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 3 client.43877.objecter handle_osd_map got epochs [875,875] > 0 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 3 client.43877.objecter handle_osd_map decoding full epoch 875 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 20 client.43877.objecter dump_active .. 0 homeless 2024-04-02T05:03:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad2259c700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fad0c06f5e0 0x7fad0c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:29.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 10 client.43877.objecter ms_handle_connect 0x7fad0c06f5e0 2024-04-02T05:03:29.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad2259c700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fad0c06f5e0 0x7fad0c071aa0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7fad14005950 tx=0x7fad1400a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.718+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fad1c010040 con 0x7fad24113010 2024-04-02T05:03:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:29 smithi098 bash[23790]: cluster 2024-04-02T05:03:27.933682+0000 osd.4 (osd.4) 426 : cluster 0 10.e scrub starts 2024-04-02T05:03:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:29 smithi098 bash[23790]: cluster 2024-04-02T05:03:27.935802+0000 osd.4 (osd.4) 427 : cluster 0 10.e scrub ok 2024-04-02T05:03:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:29 smithi098 bash[23790]: cluster 2024-04-02T05:03:28.368186+0000 mgr.y (mgr.14538) 762 : cluster 0 pgmap v1336: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:29 smithi098 bash[23790]: cluster 2024-04-02T05:03:28.958428+0000 osd.2 (osd.2) 372 : cluster 0 10.8 scrub starts 2024-04-02T05:03:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:29 smithi098 bash[23790]: cluster 2024-04-02T05:03:28.960723+0000 osd.2 (osd.2) 373 : cluster 0 10.8 scrub ok 2024-04-02T05:03:29.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:29 smithi098 bash[23790]: audit 2024-04-02T05:03:29.451852+0000 mon.c (mon.2) 1165 : audit 0 from='client.? 172.21.15.69:0/2849712394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:29.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.762+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f0220005410 con 0x7f023c069f70 2024-04-02T05:03:29.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:29.805 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:29.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[21528]: cluster 2024-04-02T05:03:27.933682+0000 osd.4 (osd.4) 426 : cluster 0 10.e scrub starts 2024-04-02T05:03:29.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[21528]: cluster 2024-04-02T05:03:27.935802+0000 osd.4 (osd.4) 427 : cluster 0 10.e scrub ok 2024-04-02T05:03:29.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[21528]: cluster 2024-04-02T05:03:28.368186+0000 mgr.y (mgr.14538) 762 : cluster 0 pgmap v1336: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:29.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[21528]: cluster 2024-04-02T05:03:28.958428+0000 osd.2 (osd.2) 372 : cluster 0 10.8 scrub starts 2024-04-02T05:03:29.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[21528]: cluster 2024-04-02T05:03:28.960723+0000 osd.2 (osd.2) 373 : cluster 0 10.8 scrub ok 2024-04-02T05:03:29.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[21528]: audit 2024-04-02T05:03:29.451852+0000 mon.c (mon.2) 1165 : audit 0 from='client.? 172.21.15.69:0/2849712394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:29.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[28904]: cluster 2024-04-02T05:03:27.933682+0000 osd.4 (osd.4) 426 : cluster 0 10.e scrub starts 2024-04-02T05:03:29.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[28904]: cluster 2024-04-02T05:03:27.935802+0000 osd.4 (osd.4) 427 : cluster 0 10.e scrub ok 2024-04-02T05:03:29.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[28904]: cluster 2024-04-02T05:03:28.368186+0000 mgr.y (mgr.14538) 762 : cluster 0 pgmap v1336: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:29.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[28904]: cluster 2024-04-02T05:03:28.958428+0000 osd.2 (osd.2) 372 : cluster 0 10.8 scrub starts 2024-04-02T05:03:29.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[28904]: cluster 2024-04-02T05:03:28.960723+0000 osd.2 (osd.2) 373 : cluster 0 10.8 scrub ok 2024-04-02T05:03:29.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:29 smithi069 bash[28904]: audit 2024-04-02T05:03:29.451852+0000 mon.c (mon.2) 1165 : audit 0 from='client.? 172.21.15.69:0/2849712394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:29.909 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:29.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.926+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fad241114e0 con 0x7fad0c06f5e0 2024-04-02T05:03:29.929 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:29.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:29.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.946+0000 7fad137fe700 1 -- 172.21.15.69:0/1517226117 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+577425 (secure 0 0 0) 0x7fad241114e0 con 0x7fad0c06f5e0 2024-04-02T05:03:29.952 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T05:03:29.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 10 librados: watch_flush enter 2024-04-02T05:03:29.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 10 librados: watch_flush exit 2024-04-02T05:03:29.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 20 client.43877.objecter shutdown clearing up homeless session... 2024-04-02T05:03:29.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 10 client.43877.objecter successfully canceled tick 2024-04-02T05:03:29.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fad0c06f5e0 msgr2=0x7fad0c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fad0c06f5e0 0x7fad0c071aa0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7fad14005950 tx=0x7fad1400a300 comp rx=0 tx=0).stop 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 msgr2=0x7fad241a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 0x7fad241a2f90 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c110 tx=0x7fad1c00c420 comp rx=0 tx=0).stop 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 shutdown_connections 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fad0c06f5e0 0x7fad0c071aa0 unknown :-1 s=CLOSED pgs=4026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad24113010 0x7fad241a2f90 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad241002f0 0x7fad2419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 --2- 172.21.15.69:0/1517226117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24100c10 0x7fad2419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 >> 172.21.15.69:0/1517226117 conn(0x7fad240782b0 msgr2=0x7fad24110dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 shutdown_connections 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 -- 172.21.15.69:0/1517226117 wait complete. 2024-04-02T05:03:29.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:29.950+0000 7fad28d19700 1 librados: shutdown 2024-04-02T05:03:29.963 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T05:03:30.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:30.058 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:30.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:30.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:30.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:30.552 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:30.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.562+0000 7f022b7fe700 1 -- 172.21.15.69:0/1762919814 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v876) ==== 101+0+0 (secure 0 0 0) 0x7f022c05c950 con 0x7f023c069f70 2024-04-02T05:03:30.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 10 librados: watch_flush enter 2024-04-02T05:03:30.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 10 librados: watch_flush exit 2024-04-02T05:03:30.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 20 client.43871.objecter shutdown clearing up homeless session... 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 10 client.43871.objecter successfully canceled tick 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f022406f5e0 msgr2=0x7f0224071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f022406f5e0 0x7f0224071aa0 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f023c077b30 tx=0x7f023000a380 comp rx=0 tx=0).stop 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 msgr2=0x7f023c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 0x7f023c197ff0 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7f022c000c00 tx=0x7f022c0047b0 comp rx=0 tx=0).stop 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 shutdown_connections 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f022406f5e0 0x7f0224071aa0 unknown :-1 s=CLOSED pgs=4025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f023c1082f0 0x7f023c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f023c069f70 0x7f023c197ff0 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 --2- 172.21.15.69:0/1762919814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f023c06a910 0x7f023c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 >> 172.21.15.69:0/1762919814 conn(0x7f023c0771e0 msgr2=0x7f023c105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 shutdown_connections 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 -- 172.21.15.69:0/1762919814 wait complete. 2024-04-02T05:03:30.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:30.566+0000 7f02297fa700 1 librados: shutdown 2024-04-02T05:03:30.572 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T05:03:30.579 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:30.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:30.705 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: cluster 2024-04-02T05:03:29.270853+0000 osd.7 (osd.7) 243 : cluster 0 6.5 scrub starts 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: cluster 2024-04-02T05:03:29.273227+0000 osd.7 (osd.7) 244 : cluster 0 6.5 scrub ok 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: cluster 2024-04-02T05:03:29.606476+0000 osd.0 (osd.0) 421 : cluster 0 4.1e scrub starts 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: cluster 2024-04-02T05:03:29.607916+0000 osd.0 (osd.0) 422 : cluster 0 4.1e scrub ok 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: audit 2024-04-02T05:03:29.767493+0000 mon.c (mon.2) 1166 : audit 1 from='client.? 172.21.15.69:0/1762919814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: audit 2024-04-02T05:03:29.767872+0000 mon.a (mon.0) 3985 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:03:30.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: cluster 2024-04-02T05:03:29.790613+0000 osd.6 (osd.6) 369 : cluster 0 4.a scrub starts 2024-04-02T05:03:30.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:30 smithi098 bash[23790]: cluster 2024-04-02T05:03:29.793324+0000 osd.6 (osd.6) 370 : cluster 0 4.a scrub ok 2024-04-02T05:03:30.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: cluster 2024-04-02T05:03:29.270853+0000 osd.7 (osd.7) 243 : cluster 0 6.5 scrub starts 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: cluster 2024-04-02T05:03:29.273227+0000 osd.7 (osd.7) 244 : cluster 0 6.5 scrub ok 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: cluster 2024-04-02T05:03:29.606476+0000 osd.0 (osd.0) 421 : cluster 0 4.1e scrub starts 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: cluster 2024-04-02T05:03:29.607916+0000 osd.0 (osd.0) 422 : cluster 0 4.1e scrub ok 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: audit 2024-04-02T05:03:29.767493+0000 mon.c (mon.2) 1166 : audit 1 from='client.? 172.21.15.69:0/1762919814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: audit 2024-04-02T05:03:29.767872+0000 mon.a (mon.0) 3985 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: cluster 2024-04-02T05:03:29.790613+0000 osd.6 (osd.6) 369 : cluster 0 4.a scrub starts 2024-04-02T05:03:30.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[28904]: cluster 2024-04-02T05:03:29.793324+0000 osd.6 (osd.6) 370 : cluster 0 4.a scrub ok 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: cluster 2024-04-02T05:03:29.270853+0000 osd.7 (osd.7) 243 : cluster 0 6.5 scrub starts 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: cluster 2024-04-02T05:03:29.273227+0000 osd.7 (osd.7) 244 : cluster 0 6.5 scrub ok 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: cluster 2024-04-02T05:03:29.606476+0000 osd.0 (osd.0) 421 : cluster 0 4.1e scrub starts 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: cluster 2024-04-02T05:03:29.607916+0000 osd.0 (osd.0) 422 : cluster 0 4.1e scrub ok 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: audit 2024-04-02T05:03:29.767493+0000 mon.c (mon.2) 1166 : audit 1 from='client.? 172.21.15.69:0/1762919814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: audit 2024-04-02T05:03:29.767872+0000 mon.a (mon.0) 3985 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: cluster 2024-04-02T05:03:29.790613+0000 osd.6 (osd.6) 369 : cluster 0 4.a scrub starts 2024-04-02T05:03:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:30 smithi069 bash[21528]: cluster 2024-04-02T05:03:29.793324+0000 osd.6 (osd.6) 370 : cluster 0 4.a scrub ok 2024-04-02T05:03:30.833 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:30.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:30.963 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:30.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:31.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:31.095 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:31.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:31.226 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:31.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:31.356 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:31.459 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:31.485 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:31.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:31.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:31.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: audit 2024-04-02T05:03:29.928905+0000 mgr.y (mgr.14538) 763 : audit 0 from='client.43877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:03:31.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.317775+0000 osd.7 (osd.7) 245 : cluster 0 4.3 scrub starts 2024-04-02T05:03:31.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.320062+0000 osd.7 (osd.7) 246 : cluster 0 4.3 scrub ok 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.346859+0000 osd.3 (osd.3) 439 : cluster 0 6.4 scrub starts 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.348758+0000 osd.3 (osd.3) 440 : cluster 0 6.4 scrub ok 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.369491+0000 mgr.y (mgr.14538) 764 : cluster 0 pgmap v1337: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.546205+0000 mon.a (mon.0) 3986 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: audit 2024-04-02T05:03:30.559105+0000 mon.a (mon.0) 3987 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: cluster 2024-04-02T05:03:30.568406+0000 mon.a (mon.0) 3988 : cluster 0 osdmap e876: 8 total, 8 up, 7 in 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[21528]: audit 2024-04-02T05:03:31.178821+0000 mon.a (mon.0) 3989 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: audit 2024-04-02T05:03:29.928905+0000 mgr.y (mgr.14538) 763 : audit 0 from='client.43877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.317775+0000 osd.7 (osd.7) 245 : cluster 0 4.3 scrub starts 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.320062+0000 osd.7 (osd.7) 246 : cluster 0 4.3 scrub ok 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.346859+0000 osd.3 (osd.3) 439 : cluster 0 6.4 scrub starts 2024-04-02T05:03:31.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.348758+0000 osd.3 (osd.3) 440 : cluster 0 6.4 scrub ok 2024-04-02T05:03:31.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.369491+0000 mgr.y (mgr.14538) 764 : cluster 0 pgmap v1337: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:31.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.546205+0000 mon.a (mon.0) 3986 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:03:31.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: audit 2024-04-02T05:03:30.559105+0000 mon.a (mon.0) 3987 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:03:31.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: cluster 2024-04-02T05:03:30.568406+0000 mon.a (mon.0) 3988 : cluster 0 osdmap e876: 8 total, 8 up, 7 in 2024-04-02T05:03:31.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:31 smithi069 bash[28904]: audit 2024-04-02T05:03:31.178821+0000 mon.a (mon.0) 3989 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:31.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:31.877 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:31.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: audit 2024-04-02T05:03:29.928905+0000 mgr.y (mgr.14538) 763 : audit 0 from='client.43877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.317775+0000 osd.7 (osd.7) 245 : cluster 0 4.3 scrub starts 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.320062+0000 osd.7 (osd.7) 246 : cluster 0 4.3 scrub ok 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.346859+0000 osd.3 (osd.3) 439 : cluster 0 6.4 scrub starts 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.348758+0000 osd.3 (osd.3) 440 : cluster 0 6.4 scrub ok 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.369491+0000 mgr.y (mgr.14538) 764 : cluster 0 pgmap v1337: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.546205+0000 mon.a (mon.0) 3986 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: audit 2024-04-02T05:03:30.559105+0000 mon.a (mon.0) 3987 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: cluster 2024-04-02T05:03:30.568406+0000 mon.a (mon.0) 3988 : cluster 0 osdmap e876: 8 total, 8 up, 7 in 2024-04-02T05:03:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:31 smithi098 bash[23790]: audit 2024-04-02T05:03:31.178821+0000 mon.a (mon.0) 3989 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:31.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:32.007 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:32.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:32.138 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:32.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:32.268 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:32.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:32.396 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:32.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:32.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:32.526 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:32.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.329379+0000 osd.3 (osd.3) 441 : cluster 0 10.3 scrub starts 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.330494+0000 osd.1 (osd.1) 439 : cluster 0 1.13 scrub starts 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.333542+0000 osd.1 (osd.1) 440 : cluster 0 1.13 scrub ok 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.333600+0000 osd.3 (osd.3) 442 : cluster 0 10.3 scrub ok 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: audit 2024-04-02T05:03:31.561015+0000 mon.a (mon.0) 3990 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.565341+0000 mon.a (mon.0) 3991 : cluster 0 osdmap e877: 8 total, 8 up, 7 in 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: audit 2024-04-02T05:03:31.567181+0000 mon.a (mon.0) 3992 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:32.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.646301+0000 osd.0 (osd.0) 423 : cluster 0 6.10 scrub starts 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.648585+0000 osd.0 (osd.0) 424 : cluster 0 6.10 scrub ok 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.984708+0000 osd.2 (osd.2) 374 : cluster 0 1.a scrub starts 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[21528]: cluster 2024-04-02T05:03:31.987301+0000 osd.2 (osd.2) 375 : cluster 0 1.a scrub ok 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.329379+0000 osd.3 (osd.3) 441 : cluster 0 10.3 scrub starts 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.330494+0000 osd.1 (osd.1) 439 : cluster 0 1.13 scrub starts 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.333542+0000 osd.1 (osd.1) 440 : cluster 0 1.13 scrub ok 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.333600+0000 osd.3 (osd.3) 442 : cluster 0 10.3 scrub ok 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: audit 2024-04-02T05:03:31.561015+0000 mon.a (mon.0) 3990 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.565341+0000 mon.a (mon.0) 3991 : cluster 0 osdmap e877: 8 total, 8 up, 7 in 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: audit 2024-04-02T05:03:31.567181+0000 mon.a (mon.0) 3992 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:32.645 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.646301+0000 osd.0 (osd.0) 423 : cluster 0 6.10 scrub starts 2024-04-02T05:03:32.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.648585+0000 osd.0 (osd.0) 424 : cluster 0 6.10 scrub ok 2024-04-02T05:03:32.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.984708+0000 osd.2 (osd.2) 374 : cluster 0 1.a scrub starts 2024-04-02T05:03:32.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:32 smithi069 bash[28904]: cluster 2024-04-02T05:03:31.987301+0000 osd.2 (osd.2) 375 : cluster 0 1.a scrub ok 2024-04-02T05:03:32.657 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:32.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:32.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:32.917 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:32.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.329379+0000 osd.3 (osd.3) 441 : cluster 0 10.3 scrub starts 2024-04-02T05:03:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.330494+0000 osd.1 (osd.1) 439 : cluster 0 1.13 scrub starts 2024-04-02T05:03:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.333542+0000 osd.1 (osd.1) 440 : cluster 0 1.13 scrub ok 2024-04-02T05:03:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.333600+0000 osd.3 (osd.3) 442 : cluster 0 10.3 scrub ok 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: audit 2024-04-02T05:03:31.561015+0000 mon.a (mon.0) 3990 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.565341+0000 mon.a (mon.0) 3991 : cluster 0 osdmap e877: 8 total, 8 up, 7 in 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: audit 2024-04-02T05:03:31.567181+0000 mon.a (mon.0) 3992 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.646301+0000 osd.0 (osd.0) 423 : cluster 0 6.10 scrub starts 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.648585+0000 osd.0 (osd.0) 424 : cluster 0 6.10 scrub ok 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.984708+0000 osd.2 (osd.2) 374 : cluster 0 1.a scrub starts 2024-04-02T05:03:32.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:32 smithi098 bash[23790]: cluster 2024-04-02T05:03:31.987301+0000 osd.2 (osd.2) 375 : cluster 0 1.a scrub ok 2024-04-02T05:03:33.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:33.046 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:33.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:33.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:33.304 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:33.308 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1336,"stamp":"2024-04-02T05:03:28.366394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596628,"num_objects":275,"num_object_clones":1,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":1730,"num_read_kb":1433,"num_write":987,"num_write_kb":4016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":796390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3903,"ondisk_log_size":3903,"up":672,"acting":672,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":672,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":769712,"kb_used_data":110264,"kb_used_omap":66,"kb_used_meta":659325,"kb_avail":655446352,"statfs":{"total":671965249536,"available":671177064448,"internally_reserved":0,"allocated":112910336,"data_stored":76524585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68049,"internal_metadata":675149359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":0,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.219256"},"pg_stats":[{"pgid":"341.1f","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160187+0000","last_change":"2024-04-02T05:03:13.528667+0000","last_active":"2024-04-02T05:03:21.160187+0000","last_peered":"2024-04-02T05:03:21.160187+0000","last_clean":"2024-04-02T05:03:21.160187+0000","last_became_active":"2024-04-02T05:03:13.527578+0000","last_became_peered":"2024-04-02T05:03:13.527578+0000","last_unstale":"2024-04-02T05:03:21.160187+0000","last_undegraded":"2024-04-02T05:03:21.160187+0000","last_fullsized":"2024-04-02T05:03:21.160187+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:21.897187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"341.1e","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156724+0000","last_change":"2024-04-02T05:03:13.538310+0000","last_active":"2024-04-02T05:03:21.156724+0000","last_peered":"2024-04-02T05:03:21.156724+0000","last_clean":"2024-04-02T05:03:21.156724+0000","last_became_active":"2024-04-02T05:03:13.537455+0000","last_became_peered":"2024-04-02T05:03:13.537455+0000","last_unstale":"2024-04-02T05:03:21.156724+0000","last_undegraded":"2024-04-02T05:03:21.156724+0000","last_fullsized":"2024-04-02T05:03:21.156724+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:14.153446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.1d","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.248215+0000","last_change":"2024-04-02T05:03:13.524553+0000","last_active":"2024-04-02T05:03:21.248215+0000","last_peered":"2024-04-02T05:03:21.248215+0000","last_clean":"2024-04-02T05:03:21.248215+0000","last_became_active":"2024-04-02T05:03:13.522424+0000","last_became_peered":"2024-04-02T05:03:13.522424+0000","last_unstale":"2024-04-02T05:03:21.248215+0000","last_undegraded":"2024-04-02T05:03:21.248215+0000","last_fullsized":"2024-04-02T05:03:21.248215+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:37.966727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.1c","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156488+0000","last_change":"2024-04-02T05:03:13.678913+0000","last_active":"2024-04-02T05:03:21.156488+0000","last_peered":"2024-04-02T05:03:21.156488+0000","last_clean":"2024-04-02T05:03:21.156488+0000","last_became_active":"2024-04-02T05:03:13.677987+0000","last_became_peered":"2024-04-02T05:03:13.677987+0000","last_unstale":"2024-04-02T05:03:21.156488+0000","last_undegraded":"2024-04-02T05:03:21.156488+0000","last_fullsized":"2024-04-02T05:03:21.156488+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:14.067336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.1b","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.246873+0000","last_change":"2024-04-02T05:03:13.503799+0000","last_active":"2024-04-02T05:03:21.246873+0000","last_peered":"2024-04-02T05:03:21.246873+0000","last_clean":"2024-04-02T05:03:21.246873+0000","last_became_active":"2024-04-02T05:03:13.501051+0000","last_became_peered":"2024-04-02T05:03:13.501051+0000","last_unstale":"2024-04-02T05:03:21.246873+0000","last_undegraded":"2024-04-02T05:03:21.246873+0000","last_fullsized":"2024-04-02T05:03:21.246873+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:35.126850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.1a","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301407+0000","last_change":"2024-04-02T05:03:13.515594+0000","last_active":"2024-04-02T05:03:21.301407+0000","last_peered":"2024-04-02T05:03:21.301407+0000","last_clean":"2024-04-02T05:03:21.301407+0000","last_became_active":"2024-04-02T05:03:13.515237+0000","last_became_peered":"2024-04-02T05:03:13.515237+0000","last_unstale":"2024-04-02T05:03:21.301407+0000","last_undegraded":"2024-04-02T05:03:21.301407+0000","last_fullsized":"2024-04-02T05:03:21.301407+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:27.388055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"341.19","version":"870'2","reported_seq":37,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.154803+0000","last_change":"2024-04-02T05:03:13.537965+0000","last_active":"2024-04-02T05:03:21.154803+0000","last_peered":"2024-04-02T05:03:21.154803+0000","last_clean":"2024-04-02T05:03:21.154803+0000","last_became_active":"2024-04-02T05:03:13.537600+0000","last_became_peered":"2024-04-02T05:03:13.537600+0000","last_unstale":"2024-04-02T05:03:21.154803+0000","last_undegraded":"2024-04-02T05:03:21.154803+0000","last_fullsized":"2024-04-02T05:03:21.154803+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:38.573304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.18","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250392+0000","last_change":"2024-04-02T05:03:13.674123+0000","last_active":"2024-04-02T05:03:21.250392+0000","last_peered":"2024-04-02T05:03:21.250392+0000","last_clean":"2024-04-02T05:03:21.250392+0000","last_became_active":"2024-04-02T05:03:13.673381+0000","last_became_peered":"2024-04-02T05:03:13.673381+0000","last_unstale":"2024-04-02T05:03:21.250392+0000","last_undegraded":"2024-04-02T05:03:21.250392+0000","last_fullsized":"2024-04-02T05:03:21.250392+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:31.182875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"341.f","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253940+0000","last_change":"2024-04-02T05:03:13.529067+0000","last_active":"2024-04-02T05:03:21.253940+0000","last_peered":"2024-04-02T05:03:21.253940+0000","last_clean":"2024-04-02T05:03:21.253940+0000","last_became_active":"2024-04-02T05:03:13.528645+0000","last_became_peered":"2024-04-02T05:03:13.528645+0000","last_unstale":"2024-04-02T05:03:21.253940+0000","last_undegraded":"2024-04-02T05:03:21.253940+0000","last_fullsized":"2024-04-02T05:03:21.253940+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:31.475806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"341.e","version":"875'2","reported_seq":37,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298263+0000","last_change":"2024-04-02T05:03:13.540471+0000","last_active":"2024-04-02T05:03:21.298263+0000","last_peered":"2024-04-02T05:03:21.298263+0000","last_clean":"2024-04-02T05:03:21.298263+0000","last_became_active":"2024-04-02T05:03:13.539962+0000","last_became_peered":"2024-04-02T05:03:13.539962+0000","last_unstale":"2024-04-02T05:03:21.298263+0000","last_undegraded":"2024-04-02T05:03:21.298263+0000","last_fullsized":"2024-04-02T05:03:21.298263+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:21.034601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"341.d","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.155940+0000","last_change":"2024-04-02T05:03:13.677026+0000","last_active":"2024-04-02T05:03:21.155940+0000","last_peered":"2024-04-02T05:03:21.155940+0000","last_clean":"2024-04-02T05:03:21.155940+0000","last_became_active":"2024-04-02T05:03:13.676242+0000","last_became_peered":"2024-04-02T05:03:13.676242+0000","last_unstale":"2024-04-02T05:03:21.155940+0000","last_undegraded":"2024-04-02T05:03:21.155940+0000","last_fullsized":"2024-04-02T05:03:21.155940+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:13.409473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.c","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302638+0000","last_change":"2024-04-02T05:03:13.677622+0000","last_active":"2024-04-02T05:03:21.302638+0000","last_peered":"2024-04-02T05:03:21.302638+0000","last_clean":"2024-04-02T05:03:21.302638+0000","last_became_active":"2024-04-02T05:03:13.676772+0000","last_became_peered":"2024-04-02T05:03:13.676772+0000","last_unstale":"2024-04-02T05:03:21.302638+0000","last_undegraded":"2024-04-02T05:03:21.302638+0000","last_fullsized":"2024-04-02T05:03:21.302638+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:12.701856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"341.b","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.244440+0000","last_change":"2024-04-02T05:03:13.533662+0000","last_active":"2024-04-02T05:03:21.244440+0000","last_peered":"2024-04-02T05:03:21.244440+0000","last_clean":"2024-04-02T05:03:21.244440+0000","last_became_active":"2024-04-02T05:03:13.532865+0000","last_became_peered":"2024-04-02T05:03:13.532865+0000","last_unstale":"2024-04-02T05:03:21.244440+0000","last_undegraded":"2024-04-02T05:03:21.244440+0000","last_fullsized":"2024-04-02T05:03:21.244440+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:26.598195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.a","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250836+0000","last_change":"2024-04-02T05:03:13.533546+0000","last_active":"2024-04-02T05:03:21.250836+0000","last_peered":"2024-04-02T05:03:21.250836+0000","last_clean":"2024-04-02T05:03:21.250836+0000","last_became_active":"2024-04-02T05:03:13.533230+0000","last_became_peered":"2024-04-02T05:03:13.533230+0000","last_unstale":"2024-04-02T05:03:21.250836+0000","last_undegraded":"2024-04-02T05:03:21.250836+0000","last_fullsized":"2024-04-02T05:03:21.250836+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:24.762350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"341.9","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294898+0000","last_change":"2024-04-02T05:03:13.533097+0000","last_active":"2024-04-02T05:03:21.294898+0000","last_peered":"2024-04-02T05:03:21.294898+0000","last_clean":"2024-04-02T05:03:21.294898+0000","last_became_active":"2024-04-02T05:03:13.532570+0000","last_became_peered":"2024-04-02T05:03:13.532570+0000","last_unstale":"2024-04-02T05:03:21.294898+0000","last_undegraded":"2024-04-02T05:03:21.294898+0000","last_fullsized":"2024-04-02T05:03:21.294898+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:35.139962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"341.8","version":"875'2","reported_seq":37,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250512+0000","last_change":"2024-04-02T05:03:13.531760+0000","last_active":"2024-04-02T05:03:21.250512+0000","last_peered":"2024-04-02T05:03:21.250512+0000","last_clean":"2024-04-02T05:03:21.250512+0000","last_became_active":"2024-04-02T05:03:13.531361+0000","last_became_peered":"2024-04-02T05:03:13.531361+0000","last_unstale":"2024-04-02T05:03:21.250512+0000","last_undegraded":"2024-04-02T05:03:21.250512+0000","last_fullsized":"2024-04-02T05:03:21.250512+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:15.278783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"341.7","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.159847+0000","last_change":"2024-04-02T05:03:13.532343+0000","last_active":"2024-04-02T05:03:21.159847+0000","last_peered":"2024-04-02T05:03:21.159847+0000","last_clean":"2024-04-02T05:03:21.159847+0000","last_became_active":"2024-04-02T05:03:13.530969+0000","last_became_peered":"2024-04-02T05:03:13.530969+0000","last_unstale":"2024-04-02T05:03:21.159847+0000","last_undegraded":"2024-04-02T05:03:21.159847+0000","last_fullsized":"2024-04-02T05:03:21.159847+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:26.180854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"341.6","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302819+0000","last_change":"2024-04-02T05:03:13.503107+0000","last_active":"2024-04-02T05:03:21.302819+0000","last_peered":"2024-04-02T05:03:21.302819+0000","last_clean":"2024-04-02T05:03:21.302819+0000","last_became_active":"2024-04-02T05:03:13.502775+0000","last_became_peered":"2024-04-02T05:03:13.502775+0000","last_unstale":"2024-04-02T05:03:21.302819+0000","last_undegraded":"2024-04-02T05:03:21.302819+0000","last_fullsized":"2024-04-02T05:03:21.302819+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:39.408668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"341.5","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.295324+0000","last_change":"2024-04-02T05:03:13.540543+0000","last_active":"2024-04-02T05:03:21.295324+0000","last_peered":"2024-04-02T05:03:21.295324+0000","last_clean":"2024-04-02T05:03:21.295324+0000","last_became_active":"2024-04-02T05:03:13.540135+0000","last_became_peered":"2024-04-02T05:03:13.540135+0000","last_unstale":"2024-04-02T05:03:21.295324+0000","last_undegraded":"2024-04-02T05:03:21.295324+0000","last_fullsized":"2024-04-02T05:03:21.295324+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:39.804944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"341.4","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.159283+0000","last_change":"2024-04-02T05:03:13.537634+0000","last_active":"2024-04-02T05:03:21.159283+0000","last_peered":"2024-04-02T05:03:21.159283+0000","last_clean":"2024-04-02T05:03:21.159283+0000","last_became_active":"2024-04-02T05:03:13.536983+0000","last_became_peered":"2024-04-02T05:03:13.536983+0000","last_unstale":"2024-04-02T05:03:21.159283+0000","last_undegraded":"2024-04-02T05:03:21.159283+0000","last_fullsized":"2024-04-02T05:03:21.159283+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:29.465999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"341.3","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.155456+0000","last_change":"2024-04-02T05:03:13.538802+0000","last_active":"2024-04-02T05:03:21.155456+0000","last_peered":"2024-04-02T05:03:21.155456+0000","last_clean":"2024-04-02T05:03:21.155456+0000","last_became_active":"2024-04-02T05:03:13.538204+0000","last_became_peered":"2024-04-02T05:03:13.538204+0000","last_unstale":"2024-04-02T05:03:21.155456+0000","last_undegraded":"2024-04-02T05:03:21.155456+0000","last_fullsized":"2024-04-02T05:03:21.155456+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:21.093597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.2","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.244021+0000","last_change":"2024-04-02T05:03:13.519975+0000","last_active":"2024-04-02T05:03:21.244021+0000","last_peered":"2024-04-02T05:03:21.244021+0000","last_clean":"2024-04-02T05:03:21.244021+0000","last_became_active":"2024-04-02T05:03:13.517977+0000","last_became_peered":"2024-04-02T05:03:13.517977+0000","last_unstale":"2024-04-02T05:03:21.244021+0000","last_undegraded":"2024-04-02T05:03:21.244021+0000","last_fullsized":"2024-04-02T05:03:21.244021+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:20.703615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.1","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301388+0000","last_change":"2024-04-02T05:03:13.515778+0000","last_active":"2024-04-02T05:03:21.301388+0000","last_peered":"2024-04-02T05:03:21.301388+0000","last_clean":"2024-04-02T05:03:21.301388+0000","last_became_active":"2024-04-02T05:03:13.515429+0000","last_became_peered":"2024-04-02T05:03:13.515429+0000","last_unstale":"2024-04-02T05:03:21.301388+0000","last_undegraded":"2024-04-02T05:03:21.301388+0000","last_fullsized":"2024-04-02T05:03:21.301388+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:35.804707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"341.0","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.243186+0000","last_change":"2024-04-02T05:03:13.523458+0000","last_active":"2024-04-02T05:03:21.243186+0000","last_peered":"2024-04-02T05:03:21.243186+0000","last_clean":"2024-04-02T05:03:21.243186+0000","last_became_active":"2024-04-02T05:03:13.522979+0000","last_became_peered":"2024-04-02T05:03:13.522979+0000","last_unstale":"2024-04-02T05:03:21.243186+0000","last_undegraded":"2024-04-02T05:03:21.243186+0000","last_fullsized":"2024-04-02T05:03:21.243186+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:41.830746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.11","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.154540+0000","last_change":"2024-04-02T05:03:13.527436+0000","last_active":"2024-04-02T05:03:21.154540+0000","last_peered":"2024-04-02T05:03:21.154540+0000","last_clean":"2024-04-02T05:03:21.154540+0000","last_became_active":"2024-04-02T05:03:13.526970+0000","last_became_peered":"2024-04-02T05:03:13.526970+0000","last_unstale":"2024-04-02T05:03:21.154540+0000","last_undegraded":"2024-04-02T05:03:21.154540+0000","last_fullsized":"2024-04-02T05:03:21.154540+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:29.671035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.7","version":"839'6","reported_seq":462,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298286+0000","last_change":"2024-04-02T05:03:17.281519+0000","last_active":"2024-04-02T05:03:21.298286+0000","last_peered":"2024-04-02T05:03:21.298286+0000","last_clean":"2024-04-02T05:03:21.298286+0000","last_became_active":"2024-04-02T04:59:57.067691+0000","last_became_peered":"2024-04-02T04:59:57.067691+0000","last_unstale":"2024-04-02T05:03:21.298286+0000","last_undegraded":"2024-04-02T05:03:21.298286+0000","last_fullsized":"2024-04-02T05:03:21.298286+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'6","last_scrub_stamp":"2024-04-02T05:03:17.281285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T05:03:17.281285+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.223581+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0015143509999999999,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"341.10","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.292892+0000","last_change":"2024-04-02T05:03:13.533053+0000","last_active":"2024-04-02T05:03:21.292892+0000","last_peered":"2024-04-02T05:03:21.292892+0000","last_clean":"2024-04-02T05:03:21.292892+0000","last_became_active":"2024-04-02T05:03:13.532578+0000","last_became_peered":"2024-04-02T05:03:13.532578+0000","last_unstale":"2024-04-02T05:03:21.292892+0000","last_undegraded":"2024-04-02T05:03:21.292892+0000","last_fullsized":"2024-04-02T05:03:21.292892+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.013107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.6","version":"867'9","reported_seq":487,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251317+0000","last_change":"2024-04-02T05:03:07.584717+0000","last_active":"2024-04-02T05:03:21.251317+0000","last_peered":"2024-04-02T05:03:21.251317+0000","last_clean":"2024-04-02T05:03:21.251317+0000","last_became_active":"2024-04-02T05:03:07.561672+0000","last_became_peered":"2024-04-02T05:03:07.561672+0000","last_unstale":"2024-04-02T05:03:21.251317+0000","last_undegraded":"2024-04-02T05:03:21.251317+0000","last_fullsized":"2024-04-02T05:03:21.251317+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'5","last_scrub_stamp":"2024-04-02T05:02:24.974695+0000","last_deep_scrub":"722'5","last_deep_scrub_stamp":"2024-04-02T05:02:24.974695+0000","last_clean_scrub_stamp":"2024-04-02T05:02:24.974695+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:40.931930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031820910000000002,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.13","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.157859+0000","last_change":"2024-04-02T05:03:13.538715+0000","last_active":"2024-04-02T05:03:21.157859+0000","last_peered":"2024-04-02T05:03:21.157859+0000","last_clean":"2024-04-02T05:03:21.157859+0000","last_became_active":"2024-04-02T05:03:13.538287+0000","last_became_peered":"2024-04-02T05:03:13.538287+0000","last_unstale":"2024-04-02T05:03:21.157859+0000","last_undegraded":"2024-04-02T05:03:21.157859+0000","last_fullsized":"2024-04-02T05:03:21.157859+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:26.700646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.5","version":"839'23","reported_seq":480,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.155880+0000","last_change":"2024-04-02T05:03:16.488532+0000","last_active":"2024-04-02T05:03:21.155880+0000","last_peered":"2024-04-02T05:03:21.155880+0000","last_clean":"2024-04-02T05:03:21.155880+0000","last_became_active":"2024-04-02T04:59:57.057395+0000","last_became_peered":"2024-04-02T04:59:57.057395+0000","last_unstale":"2024-04-02T05:03:21.155880+0000","last_undegraded":"2024-04-02T05:03:21.155880+0000","last_fullsized":"2024-04-02T05:03:21.155880+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'23","last_scrub_stamp":"2024-04-02T05:03:16.488411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T05:03:16.488411+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.516640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.033063621000000001,"stat_sum":{"num_bytes":34,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":14,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.12","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158057+0000","last_change":"2024-04-02T05:03:13.539418+0000","last_active":"2024-04-02T05:03:21.158057+0000","last_peered":"2024-04-02T05:03:21.158057+0000","last_clean":"2024-04-02T05:03:21.158057+0000","last_became_active":"2024-04-02T05:03:13.538837+0000","last_became_peered":"2024-04-02T05:03:13.538837+0000","last_unstale":"2024-04-02T05:03:21.158057+0000","last_undegraded":"2024-04-02T05:03:21.158057+0000","last_fullsized":"2024-04-02T05:03:21.158057+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:22.437702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"323.4","version":"839'13","reported_seq":474,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156660+0000","last_change":"2024-04-02T05:03:10.597233+0000","last_active":"2024-04-02T05:03:21.156660+0000","last_peered":"2024-04-02T05:03:21.156660+0000","last_clean":"2024-04-02T05:03:21.156660+0000","last_became_active":"2024-04-02T04:59:57.057575+0000","last_became_peered":"2024-04-02T04:59:57.057575+0000","last_unstale":"2024-04-02T05:03:21.156660+0000","last_undegraded":"2024-04-02T05:03:21.156660+0000","last_fullsized":"2024-04-02T05:03:21.156660+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'13","last_scrub_stamp":"2024-04-02T05:03:10.597096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T05:03:10.597096+0000","objects_scrubbed":7,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:31.769286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094995799999999999,"stat_sum":{"num_bytes":4000053,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":10,"num_write_kb":3915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"341.15","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.247771+0000","last_change":"2024-04-02T05:03:13.525289+0000","last_active":"2024-04-02T05:03:21.247771+0000","last_peered":"2024-04-02T05:03:21.247771+0000","last_clean":"2024-04-02T05:03:21.247771+0000","last_became_active":"2024-04-02T05:03:13.523575+0000","last_became_peered":"2024-04-02T05:03:13.523575+0000","last_unstale":"2024-04-02T05:03:21.247771+0000","last_undegraded":"2024-04-02T05:03:21.247771+0000","last_fullsized":"2024-04-02T05:03:21.247771+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:28.040109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.3","version":"839'21","reported_seq":489,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158306+0000","last_change":"2024-04-02T05:02:25.777580+0000","last_active":"2024-04-02T05:03:21.158306+0000","last_peered":"2024-04-02T05:03:21.158306+0000","last_clean":"2024-04-02T05:03:21.158306+0000","last_became_active":"2024-04-02T04:59:57.053322+0000","last_became_peered":"2024-04-02T04:59:57.053322+0000","last_unstale":"2024-04-02T05:03:21.158306+0000","last_undegraded":"2024-04-02T05:03:21.158306+0000","last_fullsized":"2024-04-02T05:03:21.158306+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"758'17","last_scrub_stamp":"2024-04-02T05:02:25.777476+0000","last_deep_scrub":"758'17","last_deep_scrub_stamp":"2024-04-02T05:02:25.777476+0000","last_clean_scrub_stamp":"2024-04-02T05:02:25.777476+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:28.137847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013697640000000001,"stat_sum":{"num_bytes":50,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"341.14","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301829+0000","last_change":"2024-04-02T05:03:13.523164+0000","last_active":"2024-04-02T05:03:21.301829+0000","last_peered":"2024-04-02T05:03:21.301829+0000","last_clean":"2024-04-02T05:03:21.301829+0000","last_became_active":"2024-04-02T05:03:13.522655+0000","last_became_peered":"2024-04-02T05:03:13.522655+0000","last_unstale":"2024-04-02T05:03:21.301829+0000","last_undegraded":"2024-04-02T05:03:21.301829+0000","last_fullsized":"2024-04-02T05:03:21.301829+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:35.035458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.2","version":"839'3","reported_seq":473,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.244433+0000","last_change":"2024-04-02T05:03:07.579149+0000","last_active":"2024-04-02T05:03:21.244433+0000","last_peered":"2024-04-02T05:03:21.244433+0000","last_clean":"2024-04-02T05:03:21.244433+0000","last_became_active":"2024-04-02T05:03:07.556121+0000","last_became_peered":"2024-04-02T05:03:07.556121+0000","last_unstale":"2024-04-02T05:03:21.244433+0000","last_undegraded":"2024-04-02T05:03:21.244433+0000","last_fullsized":"2024-04-02T05:03:21.244433+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:28.932934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T05:02:28.932934+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:31.972835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040707649999999996,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"341.17","version":"0'0","reported_seq":34,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.299147+0000","last_change":"2024-04-02T05:03:13.530445+0000","last_active":"2024-04-02T05:03:21.299147+0000","last_peered":"2024-04-02T05:03:21.299147+0000","last_clean":"2024-04-02T05:03:21.299147+0000","last_became_active":"2024-04-02T05:03:13.530046+0000","last_became_peered":"2024-04-02T05:03:13.530046+0000","last_unstale":"2024-04-02T05:03:21.299147+0000","last_undegraded":"2024-04-02T05:03:21.299147+0000","last_fullsized":"2024-04-02T05:03:21.299147+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.572221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.1","version":"867'15","reported_seq":499,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293172+0000","last_change":"2024-04-02T05:03:09.198004+0000","last_active":"2024-04-02T05:03:21.293172+0000","last_peered":"2024-04-02T05:03:21.293172+0000","last_clean":"2024-04-02T05:03:21.293172+0000","last_became_active":"2024-04-02T05:03:07.470119+0000","last_became_peered":"2024-04-02T05:03:07.470119+0000","last_unstale":"2024-04-02T05:03:21.293172+0000","last_undegraded":"2024-04-02T05:03:21.293172+0000","last_fullsized":"2024-04-02T05:03:21.293172+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"854'14","last_scrub_stamp":"2024-04-02T05:03:09.197890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T05:03:09.197890+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:33.857006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001090347,"stat_sum":{"num_bytes":46,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"341.16","version":"870'2","reported_seq":37,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160567+0000","last_change":"2024-04-02T05:03:13.530872+0000","last_active":"2024-04-02T05:03:21.160567+0000","last_peered":"2024-04-02T05:03:21.160567+0000","last_clean":"2024-04-02T05:03:21.160567+0000","last_became_active":"2024-04-02T05:03:13.530501+0000","last_became_peered":"2024-04-02T05:03:13.530501+0000","last_unstale":"2024-04-02T05:03:21.160567+0000","last_undegraded":"2024-04-02T05:03:21.160567+0000","last_fullsized":"2024-04-02T05:03:21.160567+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":866,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:12.476333+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.476333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.125813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.0","version":"867'5","reported_seq":483,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.242732+0000","last_change":"2024-04-02T05:03:09.192872+0000","last_active":"2024-04-02T05:03:21.242732+0000","last_peered":"2024-04-02T05:03:21.242732+0000","last_clean":"2024-04-02T05:03:21.242732+0000","last_became_active":"2024-04-02T05:03:07.561097+0000","last_became_peered":"2024-04-02T05:03:07.561097+0000","last_unstale":"2024-04-02T05:03:21.242732+0000","last_undegraded":"2024-04-02T05:03:21.242732+0000","last_fullsized":"2024-04-02T05:03:21.242732+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'4","last_scrub_stamp":"2024-04-02T05:03:09.192638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:56.027374+0000","last_clean_scrub_stamp":"2024-04-02T05:03:09.192638+0000","objects_scrubbed":3,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:14.039819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0084358820000000004,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"101'128","reported_seq":1105,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.295487+0000","last_change":"2024-04-02T05:03:10.462487+0000","last_active":"2024-04-02T05:03:21.295487+0000","last_peered":"2024-04-02T05:03:21.295487+0000","last_clean":"2024-04-02T05:03:21.295487+0000","last_became_active":"2024-04-02T05:03:10.461570+0000","last_became_peered":"2024-04-02T05:03:10.461570+0000","last_unstale":"2024-04-02T05:03:21.295487+0000","last_undegraded":"2024-04-02T05:03:21.295487+0000","last_fullsized":"2024-04-02T05:03:21.295487+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:54.397932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:54.397932+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":879,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.491754+0000","last_change":"2024-04-02T05:03:21.491782+0000","last_active":"2024-04-02T05:03:21.491754+0000","last_peered":"2024-04-02T05:03:21.491754+0000","last_clean":"2024-04-02T05:03:21.491754+0000","last_became_active":"2024-04-02T04:54:56.017309+0000","last_became_peered":"2024-04-02T04:54:56.017309+0000","last_unstale":"2024-04-02T05:03:21.491754+0000","last_undegraded":"2024-04-02T05:03:21.491754+0000","last_fullsized":"2024-04-02T05:03:21.491754+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:21.491671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:03:21.491671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:23.772626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1541,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.292908+0000","last_change":"2024-04-02T05:02:06.494033+0000","last_active":"2024-04-02T05:03:21.292908+0000","last_peered":"2024-04-02T05:03:21.292908+0000","last_clean":"2024-04-02T05:03:21.292908+0000","last_became_active":"2024-04-02T04:54:55.990081+0000","last_became_peered":"2024-04-02T04:54:55.990081+0000","last_unstale":"2024-04-02T05:03:21.292908+0000","last_undegraded":"2024-04-02T05:03:21.292908+0000","last_fullsized":"2024-04-02T05:03:21.292908+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:06.490919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:06.490919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"415'28","reported_seq":1188,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298941+0000","last_change":"2024-04-02T05:02:14.313068+0000","last_active":"2024-04-02T05:03:21.298941+0000","last_peered":"2024-04-02T05:03:21.298941+0000","last_clean":"2024-04-02T05:03:21.298941+0000","last_became_active":"2024-04-02T04:59:55.068785+0000","last_became_peered":"2024-04-02T04:59:55.068785+0000","last_unstale":"2024-04-02T05:03:21.298941+0000","last_undegraded":"2024-04-02T05:03:21.298941+0000","last_fullsized":"2024-04-02T05:03:21.298941+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'28","last_scrub_stamp":"2024-04-02T05:02:14.312828+0000","last_deep_scrub":"415'28","last_deep_scrub_stamp":"2024-04-02T04:54:15.261679+0000","last_clean_scrub_stamp":"2024-04-02T05:02:14.312828+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1228,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163236+0000","last_change":"2024-04-02T05:03:07.500144+0000","last_active":"2024-04-02T05:03:21.163236+0000","last_peered":"2024-04-02T05:03:21.163236+0000","last_clean":"2024-04-02T05:03:21.163236+0000","last_became_active":"2024-04-02T05:03:07.498576+0000","last_became_peered":"2024-04-02T05:03:07.498576+0000","last_unstale":"2024-04-02T05:03:21.163236+0000","last_undegraded":"2024-04-02T05:03:21.163236+0000","last_fullsized":"2024-04-02T05:03:21.163236+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:13.642898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:02:13.642898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:42.888363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":880,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251003+0000","last_change":"2024-04-02T05:03:07.495051+0000","last_active":"2024-04-02T05:03:21.251003+0000","last_peered":"2024-04-02T05:03:21.251003+0000","last_clean":"2024-04-02T05:03:21.251003+0000","last_became_active":"2024-04-02T05:03:07.494566+0000","last_became_peered":"2024-04-02T05:03:07.494566+0000","last_unstale":"2024-04-02T05:03:21.251003+0000","last_undegraded":"2024-04-02T05:03:21.251003+0000","last_fullsized":"2024-04-02T05:03:21.251003+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:22.864309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_clean_scrub_stamp":"2024-04-02T05:02:22.864309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:30.403965+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"101'128","reported_seq":1091,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.295702+0000","last_change":"2024-04-02T05:00:57.469548+0000","last_active":"2024-04-02T05:03:21.295702+0000","last_peered":"2024-04-02T05:03:21.295702+0000","last_clean":"2024-04-02T05:03:21.295702+0000","last_became_active":"2024-04-02T05:00:47.105882+0000","last_became_peered":"2024-04-02T05:00:47.105882+0000","last_unstale":"2024-04-02T05:03:21.295702+0000","last_undegraded":"2024-04-02T05:03:21.295702+0000","last_fullsized":"2024-04-02T05:03:21.295702+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:57.469321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:57.469321+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1540,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484724+0000","last_change":"2024-04-02T05:03:14.881830+0000","last_active":"2024-04-02T05:03:21.484724+0000","last_peered":"2024-04-02T05:03:21.484724+0000","last_clean":"2024-04-02T05:03:21.484724+0000","last_became_active":"2024-04-02T04:59:00.080466+0000","last_became_peered":"2024-04-02T04:59:00.080466+0000","last_unstale":"2024-04-02T05:03:21.484724+0000","last_undegraded":"2024-04-02T05:03:21.484724+0000","last_fullsized":"2024-04-02T05:03:21.484724+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:14.881601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:03:14.881601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:42.803331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1656,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160647+0000","last_change":"2024-04-02T05:01:07.934420+0000","last_active":"2024-04-02T05:03:21.160647+0000","last_peered":"2024-04-02T05:03:21.160647+0000","last_clean":"2024-04-02T05:03:21.160647+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T05:03:21.160647+0000","last_undegraded":"2024-04-02T05:03:21.160647+0000","last_fullsized":"2024-04-02T05:03:21.160647+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:07.934254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:12.993519+0000","last_clean_scrub_stamp":"2024-04-02T05:01:07.934254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"402'25","reported_seq":506,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.164170+0000","last_change":"2024-04-02T05:02:14.764391+0000","last_active":"2024-04-02T05:03:21.164170+0000","last_peered":"2024-04-02T05:03:21.164170+0000","last_clean":"2024-04-02T05:03:21.164170+0000","last_became_active":"2024-04-02T04:59:00.107041+0000","last_became_peered":"2024-04-02T04:59:00.107041+0000","last_unstale":"2024-04-02T05:03:21.164170+0000","last_undegraded":"2024-04-02T05:03:21.164170+0000","last_fullsized":"2024-04-02T05:03:21.164170+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T05:02:14.764272+0000","last_deep_scrub":"105'15","last_deep_scrub_stamp":"2024-04-02T04:46:37.302252+0000","last_clean_scrub_stamp":"2024-04-02T05:02:14.764272+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:36.334516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1689,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251504+0000","last_change":"2024-04-02T05:03:07.561188+0000","last_active":"2024-04-02T05:03:21.251504+0000","last_peered":"2024-04-02T05:03:21.251504+0000","last_clean":"2024-04-02T05:03:21.251504+0000","last_became_active":"2024-04-02T05:03:07.556923+0000","last_became_peered":"2024-04-02T05:03:07.556923+0000","last_unstale":"2024-04-02T05:03:21.251504+0000","last_undegraded":"2024-04-02T05:03:21.251504+0000","last_fullsized":"2024-04-02T05:03:21.251504+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:16.150088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T05:01:16.150088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":742,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293941+0000","last_change":"2024-04-02T05:03:19.291912+0000","last_active":"2024-04-02T05:03:21.293941+0000","last_peered":"2024-04-02T05:03:21.293941+0000","last_clean":"2024-04-02T05:03:21.293941+0000","last_became_active":"2024-04-02T05:03:07.546087+0000","last_became_peered":"2024-04-02T05:03:07.546087+0000","last_unstale":"2024-04-02T05:03:21.293941+0000","last_undegraded":"2024-04-02T05:03:21.293941+0000","last_fullsized":"2024-04-02T05:03:21.293941+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:19.291771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:03:19.291771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:22.151249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"101'128","reported_seq":1181,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254307+0000","last_change":"2024-04-02T05:03:10.468024+0000","last_active":"2024-04-02T05:03:21.254307+0000","last_peered":"2024-04-02T05:03:21.254307+0000","last_clean":"2024-04-02T05:03:21.254307+0000","last_became_active":"2024-04-02T05:03:10.466845+0000","last_became_peered":"2024-04-02T05:03:10.466845+0000","last_unstale":"2024-04-02T05:03:21.254307+0000","last_undegraded":"2024-04-02T05:03:21.254307+0000","last_fullsized":"2024-04-02T05:03:21.254307+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:53.972971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:53.972971+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1525,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.157299+0000","last_change":"2024-04-02T05:02:13.203456+0000","last_active":"2024-04-02T05:03:21.157299+0000","last_peered":"2024-04-02T05:03:21.157299+0000","last_clean":"2024-04-02T05:03:21.157299+0000","last_became_active":"2024-04-02T04:59:00.102255+0000","last_became_peered":"2024-04-02T04:59:00.102255+0000","last_unstale":"2024-04-02T05:03:21.157299+0000","last_undegraded":"2024-04-02T05:03:21.157299+0000","last_fullsized":"2024-04-02T05:03:21.157299+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:13.203342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:13.203342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:27.224846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1645,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251388+0000","last_change":"2024-04-02T05:00:17.020486+0000","last_active":"2024-04-02T05:03:21.251388+0000","last_peered":"2024-04-02T05:03:21.251388+0000","last_clean":"2024-04-02T05:03:21.251388+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T05:03:21.251388+0000","last_undegraded":"2024-04-02T05:03:21.251388+0000","last_fullsized":"2024-04-02T05:03:21.251388+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:17.020256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:17.020256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"402'31","reported_seq":1249,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298445+0000","last_change":"2024-04-02T05:01:13.538363+0000","last_active":"2024-04-02T05:03:21.298445+0000","last_peered":"2024-04-02T05:03:21.298445+0000","last_clean":"2024-04-02T05:03:21.298445+0000","last_became_active":"2024-04-02T04:59:00.137536+0000","last_became_peered":"2024-04-02T04:59:00.137536+0000","last_unstale":"2024-04-02T05:03:21.298445+0000","last_undegraded":"2024-04-02T05:03:21.298445+0000","last_fullsized":"2024-04-02T05:03:21.298445+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'31","last_scrub_stamp":"2024-04-02T05:01:13.538239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:01:13.538239+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":834,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.157733+0000","last_change":"2024-04-02T05:03:08.640626+0000","last_active":"2024-04-02T05:03:21.157733+0000","last_peered":"2024-04-02T05:03:21.157733+0000","last_clean":"2024-04-02T05:03:21.157733+0000","last_became_active":"2024-04-02T05:03:07.563503+0000","last_became_peered":"2024-04-02T05:03:07.563503+0000","last_unstale":"2024-04-02T05:03:21.157733+0000","last_undegraded":"2024-04-02T05:03:21.157733+0000","last_fullsized":"2024-04-02T05:03:21.157733+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:08.640484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T05:03:08.640484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.068097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1146,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293410+0000","last_change":"2024-04-02T05:03:07.477413+0000","last_active":"2024-04-02T05:03:21.293410+0000","last_peered":"2024-04-02T05:03:21.293410+0000","last_clean":"2024-04-02T05:03:21.293410+0000","last_became_active":"2024-04-02T05:03:07.475762+0000","last_became_peered":"2024-04-02T05:03:07.475762+0000","last_unstale":"2024-04-02T05:03:21.293410+0000","last_undegraded":"2024-04-02T05:03:21.293410+0000","last_fullsized":"2024-04-02T05:03:21.293410+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:15.498312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:15.498312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"101'128","reported_seq":1661,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484030+0000","last_change":"2024-04-02T05:02:18.716185+0000","last_active":"2024-04-02T05:03:21.484030+0000","last_peered":"2024-04-02T05:03:21.484030+0000","last_clean":"2024-04-02T05:03:21.484030+0000","last_became_active":"2024-04-02T04:59:03.044786+0000","last_became_peered":"2024-04-02T04:59:03.044786+0000","last_unstale":"2024-04-02T05:03:21.484030+0000","last_undegraded":"2024-04-02T05:03:21.484030+0000","last_fullsized":"2024-04-02T05:03:21.484030+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:02:18.715835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:02:18.715835+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:48.260886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":493,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.159317+0000","last_change":"2024-04-02T05:00:21.700581+0000","last_active":"2024-04-02T05:03:21.159317+0000","last_peered":"2024-04-02T05:03:21.159317+0000","last_clean":"2024-04-02T05:03:21.159317+0000","last_became_active":"2024-04-02T04:59:20.067247+0000","last_became_peered":"2024-04-02T04:59:20.067247+0000","last_unstale":"2024-04-02T05:03:21.159317+0000","last_undegraded":"2024-04-02T05:03:21.159317+0000","last_fullsized":"2024-04-02T05:03:21.159317+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:21.700487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:42.254216+0000","last_clean_scrub_stamp":"2024-04-02T05:00:21.700487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"531'52","reported_seq":1783,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251789+0000","last_change":"2024-04-02T05:02:08.663373+0000","last_active":"2024-04-02T05:03:21.251789+0000","last_peered":"2024-04-02T05:03:21.251789+0000","last_clean":"2024-04-02T05:03:21.251789+0000","last_became_active":"2024-04-02T04:58:39.391176+0000","last_became_peered":"2024-04-02T04:58:39.391176+0000","last_unstale":"2024-04-02T05:03:21.251789+0000","last_undegraded":"2024-04-02T05:03:21.251789+0000","last_fullsized":"2024-04-02T05:03:21.251789+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'52","last_scrub_stamp":"2024-04-02T05:02:08.663194+0000","last_deep_scrub":"531'52","last_deep_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_clean_scrub_stamp":"2024-04-02T05:02:08.663194+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1158,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293902+0000","last_change":"2024-04-02T05:01:20.466568+0000","last_active":"2024-04-02T05:03:21.293902+0000","last_peered":"2024-04-02T05:03:21.293902+0000","last_clean":"2024-04-02T05:03:21.293902+0000","last_became_active":"2024-04-02T04:59:20.057768+0000","last_became_peered":"2024-04-02T04:59:20.057768+0000","last_unstale":"2024-04-02T05:03:21.293902+0000","last_undegraded":"2024-04-02T05:03:21.293902+0000","last_fullsized":"2024-04-02T05:03:21.293902+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:20.466353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:20.466353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"863'14","reported_seq":215,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251329+0000","last_change":"2024-04-02T05:03:07.581594+0000","last_active":"2024-04-02T05:03:21.251329+0000","last_peered":"2024-04-02T05:03:21.251329+0000","last_clean":"2024-04-02T05:03:21.251329+0000","last_became_active":"2024-04-02T05:03:07.561581+0000","last_became_peered":"2024-04-02T05:03:07.561581+0000","last_unstale":"2024-04-02T05:03:21.251329+0000","last_undegraded":"2024-04-02T05:03:21.251329+0000","last_fullsized":"2024-04-02T05:03:21.251329+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'13","last_scrub_stamp":"2024-04-02T05:02:22.718259+0000","last_deep_scrub":"662'13","last_deep_scrub_stamp":"2024-04-02T05:02:22.718259+0000","last_clean_scrub_stamp":"2024-04-02T05:02:22.718259+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1231,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.162333+0000","last_change":"2024-04-02T05:03:07.486083+0000","last_active":"2024-04-02T05:03:21.162333+0000","last_peered":"2024-04-02T05:03:21.162333+0000","last_clean":"2024-04-02T05:03:21.162333+0000","last_became_active":"2024-04-02T05:03:07.485267+0000","last_became_peered":"2024-04-02T05:03:07.485267+0000","last_unstale":"2024-04-02T05:03:21.162333+0000","last_undegraded":"2024-04-02T05:03:21.162333+0000","last_fullsized":"2024-04-02T05:03:21.162333+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:10.264855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:10.264855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"101'128","reported_seq":1444,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294519+0000","last_change":"2024-04-02T05:03:21.294569+0000","last_active":"2024-04-02T05:03:21.294519+0000","last_peered":"2024-04-02T05:03:21.294519+0000","last_clean":"2024-04-02T05:03:21.294519+0000","last_became_active":"2024-04-02T05:03:10.457317+0000","last_became_peered":"2024-04-02T05:03:10.457317+0000","last_unstale":"2024-04-02T05:03:21.294519+0000","last_undegraded":"2024-04-02T05:03:21.294519+0000","last_fullsized":"2024-04-02T05:03:21.294519+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:03:21.294397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:03:21.294397+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:43.255766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1506,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.246851+0000","last_change":"2024-04-02T05:00:59.185543+0000","last_active":"2024-04-02T05:03:21.246851+0000","last_peered":"2024-04-02T05:03:21.246851+0000","last_clean":"2024-04-02T05:03:21.246851+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T05:03:21.246851+0000","last_undegraded":"2024-04-02T05:03:21.246851+0000","last_fullsized":"2024-04-02T05:03:21.246851+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:59.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:59.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1669,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302249+0000","last_change":"2024-04-02T05:02:10.317230+0000","last_active":"2024-04-02T05:03:21.302249+0000","last_peered":"2024-04-02T05:03:21.302249+0000","last_clean":"2024-04-02T05:03:21.302249+0000","last_became_active":"2024-04-02T04:59:00.138222+0000","last_became_peered":"2024-04-02T04:59:00.138222+0000","last_unstale":"2024-04-02T05:03:21.302249+0000","last_undegraded":"2024-04-02T05:03:21.302249+0000","last_fullsized":"2024-04-02T05:03:21.302249+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:10.317032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T05:02:10.317032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:36.133714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"402'26","reported_seq":498,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158550+0000","last_change":"2024-04-02T05:01:18.723453+0000","last_active":"2024-04-02T05:03:21.158550+0000","last_peered":"2024-04-02T05:03:21.158550+0000","last_clean":"2024-04-02T05:03:21.158550+0000","last_became_active":"2024-04-02T04:59:00.135793+0000","last_became_peered":"2024-04-02T04:59:00.135793+0000","last_unstale":"2024-04-02T05:03:21.158550+0000","last_undegraded":"2024-04-02T05:03:21.158550+0000","last_fullsized":"2024-04-02T05:03:21.158550+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T05:01:18.723192+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T05:01:18.723192+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1679,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.247319+0000","last_change":"2024-04-02T05:02:00.101731+0000","last_active":"2024-04-02T05:03:21.247319+0000","last_peered":"2024-04-02T05:03:21.247319+0000","last_clean":"2024-04-02T05:03:21.247319+0000","last_became_active":"2024-04-02T04:59:00.106616+0000","last_became_peered":"2024-04-02T04:59:00.106616+0000","last_unstale":"2024-04-02T05:03:21.247319+0000","last_undegraded":"2024-04-02T05:03:21.247319+0000","last_fullsized":"2024-04-02T05:03:21.247319+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:00.101591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T05:02:00.101591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":454,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251573+0000","last_change":"2024-04-02T05:03:10.160405+0000","last_active":"2024-04-02T05:03:21.251573+0000","last_peered":"2024-04-02T05:03:21.251573+0000","last_clean":"2024-04-02T05:03:21.251573+0000","last_became_active":"2024-04-02T05:03:07.548782+0000","last_became_peered":"2024-04-02T05:03:07.548782+0000","last_unstale":"2024-04-02T05:03:21.251573+0000","last_undegraded":"2024-04-02T05:03:21.251573+0000","last_fullsized":"2024-04-02T05:03:21.251573+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:10.160260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:03:10.160260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:24.704204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"101'128","reported_seq":1614,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484199+0000","last_change":"2024-04-02T05:01:03.976473+0000","last_active":"2024-04-02T05:03:21.484199+0000","last_peered":"2024-04-02T05:03:21.484199+0000","last_clean":"2024-04-02T05:03:21.484199+0000","last_became_active":"2024-04-02T04:55:32.987580+0000","last_became_peered":"2024-04-02T04:55:32.987580+0000","last_unstale":"2024-04-02T05:03:21.484199+0000","last_undegraded":"2024-04-02T05:03:21.484199+0000","last_fullsized":"2024-04-02T05:03:21.484199+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:01:03.976383+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:34.875105+0000","last_clean_scrub_stamp":"2024-04-02T05:01:03.976383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":465,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293627+0000","last_change":"2024-04-02T05:01:56.352568+0000","last_active":"2024-04-02T05:03:21.293627+0000","last_peered":"2024-04-02T05:03:21.293627+0000","last_clean":"2024-04-02T05:03:21.293627+0000","last_became_active":"2024-04-02T04:59:20.070326+0000","last_became_peered":"2024-04-02T04:59:20.070326+0000","last_unstale":"2024-04-02T05:03:21.293627+0000","last_undegraded":"2024-04-02T05:03:21.293627+0000","last_fullsized":"2024-04-02T05:03:21.293627+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T05:01:56.352250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:01:56.352250+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:25.110809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"402'48","reported_seq":507,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.153077+0000","last_change":"2024-04-02T05:01:15.701477+0000","last_active":"2024-04-02T05:03:21.153077+0000","last_peered":"2024-04-02T05:03:21.153077+0000","last_clean":"2024-04-02T05:03:21.153077+0000","last_became_active":"2024-04-02T04:59:00.122446+0000","last_became_peered":"2024-04-02T04:59:00.122446+0000","last_unstale":"2024-04-02T05:03:21.153077+0000","last_undegraded":"2024-04-02T05:03:21.153077+0000","last_fullsized":"2024-04-02T05:03:21.153077+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'48","last_scrub_stamp":"2024-04-02T05:01:15.701314+0000","last_deep_scrub":"402'48","last_deep_scrub_stamp":"2024-04-02T05:01:15.701314+0000","last_clean_scrub_stamp":"2024-04-02T05:01:15.701314+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1684,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.243792+0000","last_change":"2024-04-02T05:03:18.209369+0000","last_active":"2024-04-02T05:03:21.243792+0000","last_peered":"2024-04-02T05:03:21.243792+0000","last_clean":"2024-04-02T05:03:21.243792+0000","last_became_active":"2024-04-02T04:59:00.107474+0000","last_became_peered":"2024-04-02T04:59:00.107474+0000","last_unstale":"2024-04-02T05:03:21.243792+0000","last_undegraded":"2024-04-02T05:03:21.243792+0000","last_fullsized":"2024-04-02T05:03:21.243792+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:18.208108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:55:26.486667+0000","last_clean_scrub_stamp":"2024-04-02T05:03:18.208108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:46.706872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1623,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484957+0000","last_change":"2024-04-02T05:03:17.895035+0000","last_active":"2024-04-02T05:03:21.484957+0000","last_peered":"2024-04-02T05:03:21.484957+0000","last_clean":"2024-04-02T05:03:21.484957+0000","last_became_active":"2024-04-02T04:59:00.111699+0000","last_became_peered":"2024-04-02T04:59:00.111699+0000","last_unstale":"2024-04-02T05:03:21.484957+0000","last_undegraded":"2024-04-02T05:03:21.484957+0000","last_fullsized":"2024-04-02T05:03:21.484957+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:17.894901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:03:17.894901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:39.614105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":875,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251671+0000","last_change":"2024-04-02T05:01:21.130476+0000","last_active":"2024-04-02T05:03:21.251671+0000","last_peered":"2024-04-02T05:03:21.251671+0000","last_clean":"2024-04-02T05:03:21.251671+0000","last_became_active":"2024-04-02T04:54:55.956870+0000","last_became_peered":"2024-04-02T04:54:55.956870+0000","last_unstale":"2024-04-02T05:03:21.251671+0000","last_undegraded":"2024-04-02T05:03:21.251671+0000","last_fullsized":"2024-04-02T05:03:21.251671+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:21.130389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:37.405673+0000","last_clean_scrub_stamp":"2024-04-02T05:01:21.130389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"101'128","reported_seq":1654,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.247214+0000","last_change":"2024-04-02T05:03:10.467875+0000","last_active":"2024-04-02T05:03:21.247214+0000","last_peered":"2024-04-02T05:03:21.247214+0000","last_clean":"2024-04-02T05:03:21.247214+0000","last_became_active":"2024-04-02T05:03:10.467301+0000","last_became_peered":"2024-04-02T05:03:10.467301+0000","last_unstale":"2024-04-02T05:03:21.247214+0000","last_undegraded":"2024-04-02T05:03:21.247214+0000","last_fullsized":"2024-04-02T05:03:21.247214+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:55.206878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:55.206878+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1507,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250813+0000","last_change":"2024-04-02T05:02:17.779772+0000","last_active":"2024-04-02T05:03:21.250813+0000","last_peered":"2024-04-02T05:03:21.250813+0000","last_clean":"2024-04-02T05:03:21.250813+0000","last_became_active":"2024-04-02T04:59:20.037443+0000","last_became_peered":"2024-04-02T04:59:20.037443+0000","last_unstale":"2024-04-02T05:03:21.250813+0000","last_undegraded":"2024-04-02T05:03:21.250813+0000","last_fullsized":"2024-04-02T05:03:21.250813+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:17.779539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:17.779539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:24.606189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"402'30","reported_seq":1202,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484090+0000","last_change":"2024-04-02T05:02:01.804695+0000","last_active":"2024-04-02T05:03:21.484090+0000","last_peered":"2024-04-02T05:03:21.484090+0000","last_clean":"2024-04-02T05:03:21.484090+0000","last_became_active":"2024-04-02T04:59:20.065524+0000","last_became_peered":"2024-04-02T04:59:20.065524+0000","last_unstale":"2024-04-02T05:03:21.484090+0000","last_undegraded":"2024-04-02T05:03:21.484090+0000","last_fullsized":"2024-04-02T05:03:21.484090+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T05:02:01.804358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:02:01.804358+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:30.829514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1142,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293078+0000","last_change":"2024-04-02T05:01:11.533782+0000","last_active":"2024-04-02T05:03:21.293078+0000","last_peered":"2024-04-02T05:03:21.293078+0000","last_clean":"2024-04-02T05:03:21.293078+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T05:03:21.293078+0000","last_undegraded":"2024-04-02T05:03:21.293078+0000","last_fullsized":"2024-04-02T05:03:21.293078+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:11.533582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T05:01:11.533582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"642'13","reported_seq":1923,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.303167+0000","last_change":"2024-04-02T05:02:08.306983+0000","last_active":"2024-04-02T05:03:21.303167+0000","last_peered":"2024-04-02T05:03:21.303167+0000","last_clean":"2024-04-02T05:03:21.303167+0000","last_became_active":"2024-04-02T04:59:00.136703+0000","last_became_peered":"2024-04-02T04:59:00.136703+0000","last_unstale":"2024-04-02T05:03:21.303167+0000","last_undegraded":"2024-04-02T05:03:21.303167+0000","last_fullsized":"2024-04-02T05:03:21.303167+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'13","last_scrub_stamp":"2024-04-02T05:02:08.306847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:08.306847+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.164041+0000","last_change":"2024-04-02T05:01:21.783910+0000","last_active":"2024-04-02T05:03:21.164041+0000","last_peered":"2024-04-02T05:03:21.164041+0000","last_clean":"2024-04-02T05:03:21.164041+0000","last_became_active":"2024-04-02T04:59:00.125089+0000","last_became_peered":"2024-04-02T04:59:00.125089+0000","last_unstale":"2024-04-02T05:03:21.164041+0000","last_undegraded":"2024-04-02T05:03:21.164041+0000","last_fullsized":"2024-04-02T05:03:21.164041+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:21.783662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:56.418322+0000","last_clean_scrub_stamp":"2024-04-02T05:01:21.783662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'128","reported_seq":1103,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163954+0000","last_change":"2024-04-02T05:00:53.698323+0000","last_active":"2024-04-02T05:03:21.163954+0000","last_peered":"2024-04-02T05:03:21.163954+0000","last_clean":"2024-04-02T05:03:21.163954+0000","last_became_active":"2024-04-02T05:00:42.137295+0000","last_became_peered":"2024-04-02T05:00:42.137295+0000","last_unstale":"2024-04-02T05:03:21.163954+0000","last_undegraded":"2024-04-02T05:03:21.163954+0000","last_fullsized":"2024-04-02T05:03:21.163954+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:53.698099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:53.698099+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1512,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254631+0000","last_change":"2024-04-02T05:01:01.067946+0000","last_active":"2024-04-02T05:03:21.254631+0000","last_peered":"2024-04-02T05:03:21.254631+0000","last_clean":"2024-04-02T05:03:21.254631+0000","last_became_active":"2024-04-02T04:59:00.105786+0000","last_became_peered":"2024-04-02T04:59:00.105786+0000","last_unstale":"2024-04-02T05:03:21.254631+0000","last_undegraded":"2024-04-02T05:03:21.254631+0000","last_fullsized":"2024-04-02T05:03:21.254631+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:01.067820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T05:01:01.067820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1215,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250798+0000","last_change":"2024-04-02T05:02:14.778298+0000","last_active":"2024-04-02T05:03:21.250798+0000","last_peered":"2024-04-02T05:03:21.250798+0000","last_clean":"2024-04-02T05:03:21.250798+0000","last_became_active":"2024-04-02T04:59:20.074938+0000","last_became_peered":"2024-04-02T04:59:20.074938+0000","last_unstale":"2024-04-02T05:03:21.250798+0000","last_undegraded":"2024-04-02T05:03:21.250798+0000","last_fullsized":"2024-04-02T05:03:21.250798+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:14.778086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:40.385759+0000","last_clean_scrub_stamp":"2024-04-02T05:02:14.778086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:27.211830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"402'30","reported_seq":1695,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156846+0000","last_change":"2024-04-02T05:01:56.137272+0000","last_active":"2024-04-02T05:03:21.156846+0000","last_peered":"2024-04-02T05:03:21.156846+0000","last_clean":"2024-04-02T05:03:21.156846+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T05:03:21.156846+0000","last_undegraded":"2024-04-02T05:03:21.156846+0000","last_fullsized":"2024-04-02T05:03:21.156846+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'30","last_scrub_stamp":"2024-04-02T05:01:56.137155+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T05:01:56.137155+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1662,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:22.864199+0000","last_change":"2024-04-02T05:03:22.864199+0000","last_active":"2024-04-02T05:03:22.864199+0000","last_peered":"2024-04-02T05:03:22.864199+0000","last_clean":"2024-04-02T05:03:22.864199+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T05:03:22.864199+0000","last_undegraded":"2024-04-02T05:03:22.864199+0000","last_fullsized":"2024-04-02T05:03:22.864199+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:22.864014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:03:22.864014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:36.365077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"831'6","reported_seq":768,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:26.686340+0000","last_change":"2024-04-02T05:02:23.854083+0000","last_active":"2024-04-02T05:03:26.686340+0000","last_peered":"2024-04-02T05:03:26.686340+0000","last_clean":"2024-04-02T05:03:26.686340+0000","last_became_active":"2024-04-02T04:59:00.119844+0000","last_became_peered":"2024-04-02T04:59:00.119844+0000","last_unstale":"2024-04-02T05:03:26.686340+0000","last_undegraded":"2024-04-02T05:03:26.686340+0000","last_fullsized":"2024-04-02T05:03:26.686340+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'6","last_scrub_stamp":"2024-04-02T05:02:23.853869+0000","last_deep_scrub":"831'6","last_deep_scrub_stamp":"2024-04-02T05:02:23.853869+0000","last_clean_scrub_stamp":"2024-04-02T05:02:23.853869+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:47.813818+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":274,"num_read_kb":269,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1101,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294613+0000","last_change":"2024-04-02T05:03:07.460358+0000","last_active":"2024-04-02T05:03:21.294613+0000","last_peered":"2024-04-02T05:03:21.294613+0000","last_clean":"2024-04-02T05:03:21.294613+0000","last_became_active":"2024-04-02T05:03:07.459977+0000","last_became_peered":"2024-04-02T05:03:07.459977+0000","last_unstale":"2024-04-02T05:03:21.294613+0000","last_undegraded":"2024-04-02T05:03:21.294613+0000","last_fullsized":"2024-04-02T05:03:21.294613+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:58.450029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T05:00:58.450029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'128","reported_seq":361,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253980+0000","last_change":"2024-04-02T05:03:10.941058+0000","last_active":"2024-04-02T05:03:21.253980+0000","last_peered":"2024-04-02T05:03:21.253980+0000","last_clean":"2024-04-02T05:03:21.253980+0000","last_became_active":"2024-04-02T05:03:10.458729+0000","last_became_peered":"2024-04-02T05:03:10.458729+0000","last_unstale":"2024-04-02T05:03:21.253980+0000","last_undegraded":"2024-04-02T05:03:21.253980+0000","last_fullsized":"2024-04-02T05:03:21.253980+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:03:10.940834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:03:10.940834+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:37.497042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1528,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:27.935971+0000","last_change":"2024-04-02T05:03:27.936051+0000","last_active":"2024-04-02T05:03:27.935971+0000","last_peered":"2024-04-02T05:03:27.935971+0000","last_clean":"2024-04-02T05:03:27.935971+0000","last_became_active":"2024-04-02T05:03:07.473682+0000","last_became_peered":"2024-04-02T05:03:07.473682+0000","last_unstale":"2024-04-02T05:03:27.935971+0000","last_undegraded":"2024-04-02T05:03:27.935971+0000","last_fullsized":"2024-04-02T05:03:27.935971+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:27.935813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:15.184457+0000","last_clean_scrub_stamp":"2024-04-02T05:03:27.935813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:48.438722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"642'16","reported_seq":553,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.153274+0000","last_change":"2024-04-02T05:03:13.436932+0000","last_active":"2024-04-02T05:03:21.153274+0000","last_peered":"2024-04-02T05:03:21.153274+0000","last_clean":"2024-04-02T05:03:21.153274+0000","last_became_active":"2024-04-02T04:59:00.134138+0000","last_became_peered":"2024-04-02T04:59:00.134138+0000","last_unstale":"2024-04-02T05:03:21.153274+0000","last_undegraded":"2024-04-02T05:03:21.153274+0000","last_fullsized":"2024-04-02T05:03:21.153274+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'16","last_scrub_stamp":"2024-04-02T05:03:13.436708+0000","last_deep_scrub":"516'15","last_deep_scrub_stamp":"2024-04-02T04:56:30.036674+0000","last_clean_scrub_stamp":"2024-04-02T05:03:13.436708+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:19.725369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"824'29","reported_seq":515,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.162516+0000","last_change":"2024-04-02T05:02:16.780459+0000","last_active":"2024-04-02T05:03:21.162516+0000","last_peered":"2024-04-02T05:03:21.162516+0000","last_clean":"2024-04-02T05:03:21.162516+0000","last_became_active":"2024-04-02T04:59:00.134091+0000","last_became_peered":"2024-04-02T04:59:00.134091+0000","last_unstale":"2024-04-02T05:03:21.162516+0000","last_undegraded":"2024-04-02T05:03:21.162516+0000","last_fullsized":"2024-04-02T05:03:21.162516+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'29","last_scrub_stamp":"2024-04-02T05:02:16.780210+0000","last_deep_scrub":"824'29","last_deep_scrub_stamp":"2024-04-02T05:02:16.780210+0000","last_clean_scrub_stamp":"2024-04-02T05:02:16.780210+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:34.406014+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":1668,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.252051+0000","last_change":"2024-04-02T05:02:22.143568+0000","last_active":"2024-04-02T05:03:21.252051+0000","last_peered":"2024-04-02T05:03:21.252051+0000","last_clean":"2024-04-02T05:03:21.252051+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T05:03:21.252051+0000","last_undegraded":"2024-04-02T05:03:21.252051+0000","last_fullsized":"2024-04-02T05:03:21.252051+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T05:02:22.143285+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T04:47:58.111561+0000","last_clean_scrub_stamp":"2024-04-02T05:02:22.143285+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:34.256885+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1678,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301943+0000","last_change":"2024-04-02T05:01:57.331715+0000","last_active":"2024-04-02T05:03:21.301943+0000","last_peered":"2024-04-02T05:03:21.301943+0000","last_clean":"2024-04-02T05:03:21.301943+0000","last_became_active":"2024-04-02T04:59:00.135841+0000","last_became_peered":"2024-04-02T04:59:00.135841+0000","last_unstale":"2024-04-02T05:03:21.301943+0000","last_undegraded":"2024-04-02T05:03:21.301943+0000","last_fullsized":"2024-04-02T05:03:21.301943+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:57.331518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:10.151703+0000","last_clean_scrub_stamp":"2024-04-02T05:01:57.331518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1627,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160006+0000","last_change":"2024-04-02T05:03:07.494162+0000","last_active":"2024-04-02T05:03:21.160006+0000","last_peered":"2024-04-02T05:03:21.160006+0000","last_clean":"2024-04-02T05:03:21.160006+0000","last_became_active":"2024-04-02T05:03:07.493328+0000","last_became_peered":"2024-04-02T05:03:07.493328+0000","last_unstale":"2024-04-02T05:03:21.160006+0000","last_undegraded":"2024-04-02T05:03:21.160006+0000","last_fullsized":"2024-04-02T05:03:21.160006+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:16.830867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.830867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"101'128","reported_seq":1146,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253569+0000","last_change":"2024-04-02T05:00:55.008343+0000","last_active":"2024-04-02T05:03:21.253569+0000","last_peered":"2024-04-02T05:03:21.253569+0000","last_clean":"2024-04-02T05:03:21.253569+0000","last_became_active":"2024-04-02T05:00:42.158023+0000","last_became_peered":"2024-04-02T05:00:42.158023+0000","last_unstale":"2024-04-02T05:03:21.253569+0000","last_undegraded":"2024-04-02T05:03:21.253569+0000","last_fullsized":"2024-04-02T05:03:21.253569+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:55.008124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:55.008124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1500,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254157+0000","last_change":"2024-04-02T05:01:10.109672+0000","last_active":"2024-04-02T05:03:21.254157+0000","last_peered":"2024-04-02T05:03:21.254157+0000","last_clean":"2024-04-02T05:03:21.254157+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T05:03:21.254157+0000","last_undegraded":"2024-04-02T05:03:21.254157+0000","last_fullsized":"2024-04-02T05:03:21.254157+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:10.105075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:01:10.105075+0000","last_clean_scrub_stamp":"2024-04-02T05:01:10.105075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"841'16","reported_seq":890,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294521+0000","last_change":"2024-04-02T05:02:42.151418+0000","last_active":"2024-04-02T05:03:21.294521+0000","last_peered":"2024-04-02T05:03:21.294521+0000","last_clean":"2024-04-02T05:03:21.294521+0000","last_became_active":"2024-04-02T05:02:42.150790+0000","last_became_peered":"2024-04-02T05:02:42.150790+0000","last_unstale":"2024-04-02T05:03:21.294521+0000","last_undegraded":"2024-04-02T05:03:21.294521+0000","last_fullsized":"2024-04-02T05:03:21.294521+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'15","last_scrub_stamp":"2024-04-02T05:00:18.918806+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:40.137127+0000","last_clean_scrub_stamp":"2024-04-02T05:00:18.918806+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"863'61","reported_seq":2048,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251942+0000","last_change":"2024-04-02T05:03:07.600595+0000","last_active":"2024-04-02T05:03:21.251942+0000","last_peered":"2024-04-02T05:03:21.251942+0000","last_clean":"2024-04-02T05:03:21.251942+0000","last_became_active":"2024-04-02T05:03:07.568146+0000","last_became_peered":"2024-04-02T05:03:07.568146+0000","last_unstale":"2024-04-02T05:03:21.251942+0000","last_undegraded":"2024-04-02T05:03:21.251942+0000","last_fullsized":"2024-04-02T05:03:21.251942+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'60","last_scrub_stamp":"2024-04-02T05:01:11.166607+0000","last_deep_scrub":"121'47","last_deep_scrub_stamp":"2024-04-02T04:49:18.095890+0000","last_clean_scrub_stamp":"2024-04-02T05:01:11.166607+0000","objects_scrubbed":11,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1130,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298375+0000","last_change":"2024-04-02T05:01:16.509820+0000","last_active":"2024-04-02T05:03:21.298375+0000","last_peered":"2024-04-02T05:03:21.298375+0000","last_clean":"2024-04-02T05:03:21.298375+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T05:03:21.298375+0000","last_undegraded":"2024-04-02T05:03:21.298375+0000","last_fullsized":"2024-04-02T05:03:21.298375+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:16.509584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:16.509584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"89'2","reported_seq":1664,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251535+0000","last_change":"2024-04-02T05:02:06.668832+0000","last_active":"2024-04-02T05:03:21.251535+0000","last_peered":"2024-04-02T05:03:21.251535+0000","last_clean":"2024-04-02T05:03:21.251535+0000","last_became_active":"2024-04-02T04:54:56.004769+0000","last_became_peered":"2024-04-02T04:54:56.004769+0000","last_unstale":"2024-04-02T05:03:21.251535+0000","last_undegraded":"2024-04-02T05:03:21.251535+0000","last_fullsized":"2024-04-02T05:03:21.251535+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-02T05:02:06.668608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:06.668608+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1652,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.244140+0000","last_change":"2024-04-02T05:03:07.548586+0000","last_active":"2024-04-02T05:03:21.244140+0000","last_peered":"2024-04-02T05:03:21.244140+0000","last_clean":"2024-04-02T05:03:21.244140+0000","last_became_active":"2024-04-02T05:03:07.548217+0000","last_became_peered":"2024-04-02T05:03:07.548217+0000","last_unstale":"2024-04-02T05:03:21.244140+0000","last_undegraded":"2024-04-02T05:03:21.244140+0000","last_fullsized":"2024-04-02T05:03:21.244140+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:05.184319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T05:02:05.184319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:31.983484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'128","reported_seq":1645,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:24.236555+0000","last_change":"2024-04-02T05:03:24.236618+0000","last_active":"2024-04-02T05:03:24.236555+0000","last_peered":"2024-04-02T05:03:24.236555+0000","last_clean":"2024-04-02T05:03:24.236555+0000","last_became_active":"2024-04-02T05:00:42.161888+0000","last_became_peered":"2024-04-02T05:00:42.161888+0000","last_unstale":"2024-04-02T05:03:24.236555+0000","last_undegraded":"2024-04-02T05:03:24.236555+0000","last_fullsized":"2024-04-02T05:03:24.236555+0000","mapping_epoch":664,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:03:24.236401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:03:24.236401+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:46.678608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":604,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484119+0000","last_change":"2024-04-02T05:02:42.159347+0000","last_active":"2024-04-02T05:03:21.484119+0000","last_peered":"2024-04-02T05:03:21.484119+0000","last_clean":"2024-04-02T05:03:21.484119+0000","last_became_active":"2024-04-02T05:02:42.158490+0000","last_became_peered":"2024-04-02T05:02:42.158490+0000","last_unstale":"2024-04-02T05:03:21.484119+0000","last_undegraded":"2024-04-02T05:03:21.484119+0000","last_fullsized":"2024-04-02T05:03:21.484119+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:21.877690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:01:21.877690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1667,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.157365+0000","last_change":"2024-04-02T05:03:07.563531+0000","last_active":"2024-04-02T05:03:21.157365+0000","last_peered":"2024-04-02T05:03:21.157365+0000","last_clean":"2024-04-02T05:03:21.157365+0000","last_became_active":"2024-04-02T05:03:07.563104+0000","last_became_peered":"2024-04-02T05:03:07.563104+0000","last_unstale":"2024-04-02T05:03:21.157365+0000","last_undegraded":"2024-04-02T05:03:21.157365+0000","last_fullsized":"2024-04-02T05:03:21.157365+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:00.868767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:50.400390+0000","last_clean_scrub_stamp":"2024-04-02T05:01:00.868767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1128,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293477+0000","last_change":"2024-04-02T05:02:26.302101+0000","last_active":"2024-04-02T05:03:21.293477+0000","last_peered":"2024-04-02T05:03:21.293477+0000","last_clean":"2024-04-02T05:03:21.293477+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T05:03:21.293477+0000","last_undegraded":"2024-04-02T05:03:21.293477+0000","last_fullsized":"2024-04-02T05:03:21.293477+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:26.301970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:02:26.301970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:47.605458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"756'42","reported_seq":1787,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.155482+0000","last_change":"2024-04-02T05:01:06.918255+0000","last_active":"2024-04-02T05:03:21.155482+0000","last_peered":"2024-04-02T05:03:21.155482+0000","last_clean":"2024-04-02T05:03:21.155482+0000","last_became_active":"2024-04-02T04:59:20.079497+0000","last_became_peered":"2024-04-02T04:59:20.079497+0000","last_unstale":"2024-04-02T05:03:21.155482+0000","last_undegraded":"2024-04-02T05:03:21.155482+0000","last_fullsized":"2024-04-02T05:03:21.155482+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"756'42","last_scrub_stamp":"2024-04-02T05:01:06.918104+0000","last_deep_scrub":"756'42","last_deep_scrub_stamp":"2024-04-02T05:01:06.918104+0000","last_clean_scrub_stamp":"2024-04-02T05:01:06.918104+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":487,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.164109+0000","last_change":"2024-04-02T05:00:01.808440+0000","last_active":"2024-04-02T05:03:21.164109+0000","last_peered":"2024-04-02T05:03:21.164109+0000","last_clean":"2024-04-02T05:03:21.164109+0000","last_became_active":"2024-04-02T04:59:00.119618+0000","last_became_peered":"2024-04-02T04:59:00.119618+0000","last_unstale":"2024-04-02T05:03:21.164109+0000","last_undegraded":"2024-04-02T05:03:21.164109+0000","last_fullsized":"2024-04-02T05:03:21.164109+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:01.808297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:01.808297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1500,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251272+0000","last_change":"2024-04-02T05:00:19.909534+0000","last_active":"2024-04-02T05:03:21.251272+0000","last_peered":"2024-04-02T05:03:21.251272+0000","last_clean":"2024-04-02T05:03:21.251272+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T05:03:21.251272+0000","last_undegraded":"2024-04-02T05:03:21.251272+0000","last_fullsized":"2024-04-02T05:03:21.251272+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:19.909325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:19.909325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1660,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254264+0000","last_change":"2024-04-02T05:03:18.914377+0000","last_active":"2024-04-02T05:03:21.254264+0000","last_peered":"2024-04-02T05:03:21.254264+0000","last_clean":"2024-04-02T05:03:21.254264+0000","last_became_active":"2024-04-02T04:59:20.070708+0000","last_became_peered":"2024-04-02T04:59:20.070708+0000","last_unstale":"2024-04-02T05:03:21.254264+0000","last_undegraded":"2024-04-02T05:03:21.254264+0000","last_fullsized":"2024-04-02T05:03:21.254264+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:18.914237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:03:18.914237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:19.656035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":1415,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293242+0000","last_change":"2024-04-02T05:00:56.464859+0000","last_active":"2024-04-02T05:03:21.293242+0000","last_peered":"2024-04-02T05:03:21.293242+0000","last_clean":"2024-04-02T05:03:21.293242+0000","last_became_active":"2024-04-02T05:00:42.145480+0000","last_became_peered":"2024-04-02T05:00:42.145480+0000","last_unstale":"2024-04-02T05:03:21.293242+0000","last_undegraded":"2024-04-02T05:03:21.293242+0000","last_fullsized":"2024-04-02T05:03:21.293242+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:56.464705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:56.464705+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1574,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.484299+0000","last_change":"2024-04-02T05:03:15.845898+0000","last_active":"2024-04-02T05:03:21.484299+0000","last_peered":"2024-04-02T05:03:21.484299+0000","last_clean":"2024-04-02T05:03:21.484299+0000","last_became_active":"2024-04-02T04:59:00.082361+0000","last_became_peered":"2024-04-02T04:59:00.082361+0000","last_unstale":"2024-04-02T05:03:21.484299+0000","last_undegraded":"2024-04-02T05:03:21.484299+0000","last_fullsized":"2024-04-02T05:03:21.484299+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:15.845683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T05:03:15.845683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:39.272069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"402'26","reported_seq":509,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:27.362872+0000","last_change":"2024-04-02T05:03:27.362916+0000","last_active":"2024-04-02T05:03:27.362872+0000","last_peered":"2024-04-02T05:03:27.362872+0000","last_clean":"2024-04-02T05:03:27.362872+0000","last_became_active":"2024-04-02T04:59:00.118923+0000","last_became_peered":"2024-04-02T04:59:00.118923+0000","last_unstale":"2024-04-02T05:03:27.362872+0000","last_undegraded":"2024-04-02T05:03:27.362872+0000","last_fullsized":"2024-04-02T05:03:27.362872+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'26","last_scrub_stamp":"2024-04-02T05:03:27.362783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:03:27.362783+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:37.141977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":495,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:25.327940+0000","last_change":"2024-04-02T05:03:25.327976+0000","last_active":"2024-04-02T05:03:25.327940+0000","last_peered":"2024-04-02T05:03:25.327940+0000","last_clean":"2024-04-02T05:03:25.327940+0000","last_became_active":"2024-04-02T04:59:00.126748+0000","last_became_peered":"2024-04-02T04:59:00.126748+0000","last_unstale":"2024-04-02T05:03:25.327940+0000","last_undegraded":"2024-04-02T05:03:25.327940+0000","last_fullsized":"2024-04-02T05:03:25.327940+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:25.327789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T05:03:25.327789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:34.062772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"101'128","reported_seq":1440,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293413+0000","last_change":"2024-04-02T05:03:10.458065+0000","last_active":"2024-04-02T05:03:21.293413+0000","last_peered":"2024-04-02T05:03:21.293413+0000","last_clean":"2024-04-02T05:03:21.293413+0000","last_became_active":"2024-04-02T05:03:10.457462+0000","last_became_peered":"2024-04-02T05:03:10.457462+0000","last_unstale":"2024-04-02T05:03:21.293413+0000","last_undegraded":"2024-04-02T05:03:21.293413+0000","last_fullsized":"2024-04-02T05:03:21.293413+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:55.414259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:55.414259+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":569,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293862+0000","last_change":"2024-04-02T05:03:07.481512+0000","last_active":"2024-04-02T05:03:21.293862+0000","last_peered":"2024-04-02T05:03:21.293862+0000","last_clean":"2024-04-02T05:03:21.293862+0000","last_became_active":"2024-04-02T05:03:07.480995+0000","last_became_peered":"2024-04-02T05:03:07.480995+0000","last_unstale":"2024-04-02T05:03:21.293862+0000","last_undegraded":"2024-04-02T05:03:21.293862+0000","last_fullsized":"2024-04-02T05:03:21.293862+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:16.726893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:16.726893+0000","last_clean_scrub_stamp":"2024-04-02T05:02:16.726893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:46.393823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1000,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298692+0000","last_change":"2024-04-02T05:02:42.158537+0000","last_active":"2024-04-02T05:03:21.298692+0000","last_peered":"2024-04-02T05:03:21.298692+0000","last_clean":"2024-04-02T05:03:21.298692+0000","last_became_active":"2024-04-02T05:02:42.158096+0000","last_became_peered":"2024-04-02T05:02:42.158096+0000","last_unstale":"2024-04-02T05:03:21.298692+0000","last_undegraded":"2024-04-02T05:03:21.298692+0000","last_fullsized":"2024-04-02T05:03:21.298692+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:12.990813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:01:12.990813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1244,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156434+0000","last_change":"2024-04-02T05:03:07.551472+0000","last_active":"2024-04-02T05:03:21.156434+0000","last_peered":"2024-04-02T05:03:21.156434+0000","last_clean":"2024-04-02T05:03:21.156434+0000","last_became_active":"2024-04-02T05:03:07.550905+0000","last_became_peered":"2024-04-02T05:03:07.550905+0000","last_unstale":"2024-04-02T05:03:21.156434+0000","last_undegraded":"2024-04-02T05:03:21.156434+0000","last_fullsized":"2024-04-02T05:03:21.156434+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:00.977502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T05:01:00.977502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"402'42","reported_seq":1764,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:23.281182+0000","last_change":"2024-04-02T05:03:23.281246+0000","last_active":"2024-04-02T05:03:23.281182+0000","last_peered":"2024-04-02T05:03:23.281182+0000","last_clean":"2024-04-02T05:03:23.281182+0000","last_became_active":"2024-04-02T05:03:07.566352+0000","last_became_peered":"2024-04-02T05:03:07.566352+0000","last_unstale":"2024-04-02T05:03:23.281182+0000","last_undegraded":"2024-04-02T05:03:23.281182+0000","last_fullsized":"2024-04-02T05:03:23.281182+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'42","last_scrub_stamp":"2024-04-02T05:03:23.281090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:03:23.281090+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:29.957239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1119,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298310+0000","last_change":"2024-04-02T05:03:07.481522+0000","last_active":"2024-04-02T05:03:21.298310+0000","last_peered":"2024-04-02T05:03:21.298310+0000","last_clean":"2024-04-02T05:03:21.298310+0000","last_became_active":"2024-04-02T05:03:07.481015+0000","last_became_peered":"2024-04-02T05:03:07.481015+0000","last_unstale":"2024-04-02T05:03:21.298310+0000","last_undegraded":"2024-04-02T05:03:21.298310+0000","last_fullsized":"2024-04-02T05:03:21.298310+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:16.448386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:16.448386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'128","reported_seq":1625,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.154856+0000","last_change":"2024-04-02T05:00:55.849914+0000","last_active":"2024-04-02T05:03:21.154856+0000","last_peered":"2024-04-02T05:03:21.154856+0000","last_clean":"2024-04-02T05:03:21.154856+0000","last_became_active":"2024-04-02T05:00:42.152456+0000","last_became_peered":"2024-04-02T05:00:42.152456+0000","last_unstale":"2024-04-02T05:03:21.154856+0000","last_undegraded":"2024-04-02T05:03:21.154856+0000","last_fullsized":"2024-04-02T05:03:21.154856+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:55.849803+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:28.185555+0000","last_clean_scrub_stamp":"2024-04-02T05:00:55.849803+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1506,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251179+0000","last_change":"2024-04-02T05:01:12.135691+0000","last_active":"2024-04-02T05:03:21.251179+0000","last_peered":"2024-04-02T05:03:21.251179+0000","last_clean":"2024-04-02T05:03:21.251179+0000","last_became_active":"2024-04-02T04:54:56.004256+0000","last_became_peered":"2024-04-02T04:54:56.004256+0000","last_unstale":"2024-04-02T05:03:21.251179+0000","last_undegraded":"2024-04-02T05:03:21.251179+0000","last_fullsized":"2024-04-02T05:03:21.251179+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:12.135589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:28.854283+0000","last_clean_scrub_stamp":"2024-04-02T05:01:12.135589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1682,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.249148+0000","last_change":"2024-04-02T05:03:17.165037+0000","last_active":"2024-04-02T05:03:21.249148+0000","last_peered":"2024-04-02T05:03:21.249148+0000","last_clean":"2024-04-02T05:03:21.249148+0000","last_became_active":"2024-04-02T05:03:07.559525+0000","last_became_peered":"2024-04-02T05:03:07.559525+0000","last_unstale":"2024-04-02T05:03:21.249148+0000","last_undegraded":"2024-04-02T05:03:21.249148+0000","last_fullsized":"2024-04-02T05:03:21.249148+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:17.164907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T05:03:17.164907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:24.883314+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'35","reported_seq":1725,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302480+0000","last_change":"2024-04-02T05:02:02.357043+0000","last_active":"2024-04-02T05:03:21.302480+0000","last_peered":"2024-04-02T05:03:21.302480+0000","last_clean":"2024-04-02T05:03:21.302480+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T05:03:21.302480+0000","last_undegraded":"2024-04-02T05:03:21.302480+0000","last_fullsized":"2024-04-02T05:03:21.302480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T05:02:02.356791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:02:02.356791+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":877,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.246856+0000","last_change":"2024-04-02T05:01:23.193527+0000","last_active":"2024-04-02T05:03:21.246856+0000","last_peered":"2024-04-02T05:03:21.246856+0000","last_clean":"2024-04-02T05:03:21.246856+0000","last_became_active":"2024-04-02T04:54:55.969352+0000","last_became_peered":"2024-04-02T04:54:55.969352+0000","last_unstale":"2024-04-02T05:03:21.246856+0000","last_undegraded":"2024-04-02T05:03:21.246856+0000","last_fullsized":"2024-04-02T05:03:21.246856+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:23.193384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:23.193384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":499,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163639+0000","last_change":"2024-04-02T05:01:58.749243+0000","last_active":"2024-04-02T05:03:21.163639+0000","last_peered":"2024-04-02T05:03:21.163639+0000","last_clean":"2024-04-02T05:03:21.163639+0000","last_became_active":"2024-04-02T04:59:00.128819+0000","last_became_peered":"2024-04-02T04:59:00.128819+0000","last_unstale":"2024-04-02T05:03:21.163639+0000","last_undegraded":"2024-04-02T05:03:21.163639+0000","last_fullsized":"2024-04-02T05:03:21.163639+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T05:01:58.749127+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_clean_scrub_stamp":"2024-04-02T05:01:58.749127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'128","reported_seq":1164,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254308+0000","last_change":"2024-04-02T05:00:55.978927+0000","last_active":"2024-04-02T05:03:21.254308+0000","last_peered":"2024-04-02T05:03:21.254308+0000","last_clean":"2024-04-02T05:03:21.254308+0000","last_became_active":"2024-04-02T05:00:42.122155+0000","last_became_peered":"2024-04-02T05:00:42.122155+0000","last_unstale":"2024-04-02T05:03:21.254308+0000","last_undegraded":"2024-04-02T05:03:21.254308+0000","last_fullsized":"2024-04-02T05:03:21.254308+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:55.978807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:55.978807+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":999,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.295194+0000","last_change":"2024-04-02T05:03:20.307888+0000","last_active":"2024-04-02T05:03:21.295194+0000","last_peered":"2024-04-02T05:03:21.295194+0000","last_clean":"2024-04-02T05:03:21.295194+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T05:03:21.295194+0000","last_undegraded":"2024-04-02T05:03:21.295194+0000","last_fullsized":"2024-04-02T05:03:21.295194+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:20.307553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:03:20.307553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:49.111091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1231,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.155261+0000","last_change":"2024-04-02T05:03:07.552553+0000","last_active":"2024-04-02T05:03:21.155261+0000","last_peered":"2024-04-02T05:03:21.155261+0000","last_clean":"2024-04-02T05:03:21.155261+0000","last_became_active":"2024-04-02T05:03:07.552163+0000","last_became_peered":"2024-04-02T05:03:07.552163+0000","last_unstale":"2024-04-02T05:03:21.155261+0000","last_undegraded":"2024-04-02T05:03:21.155261+0000","last_fullsized":"2024-04-02T05:03:21.155261+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:05.925341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:01:05.925341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"642'31","reported_seq":559,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163390+0000","last_change":"2024-04-02T05:01:10.597720+0000","last_active":"2024-04-02T05:03:21.163390+0000","last_peered":"2024-04-02T05:03:21.163390+0000","last_clean":"2024-04-02T05:03:21.163390+0000","last_became_active":"2024-04-02T04:59:00.112450+0000","last_became_peered":"2024-04-02T04:59:00.112450+0000","last_unstale":"2024-04-02T05:03:21.163390+0000","last_undegraded":"2024-04-02T05:03:21.163390+0000","last_fullsized":"2024-04-02T05:03:21.163390+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'31","last_scrub_stamp":"2024-04-02T05:01:10.597569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:01:10.597569+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":1675,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251475+0000","last_change":"2024-04-02T05:02:08.199932+0000","last_active":"2024-04-02T05:03:21.251475+0000","last_peered":"2024-04-02T05:03:21.251475+0000","last_clean":"2024-04-02T05:03:21.251475+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T05:03:21.251475+0000","last_undegraded":"2024-04-02T05:03:21.251475+0000","last_fullsized":"2024-04-02T05:03:21.251475+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T05:02:08.199544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:02:08.199544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:27.513023+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1671,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.154304+0000","last_change":"2024-04-02T05:01:55.111675+0000","last_active":"2024-04-02T05:03:21.154304+0000","last_peered":"2024-04-02T05:03:21.154304+0000","last_clean":"2024-04-02T05:03:21.154304+0000","last_became_active":"2024-04-02T04:59:55.057518+0000","last_became_peered":"2024-04-02T04:59:55.057518+0000","last_unstale":"2024-04-02T05:03:21.154304+0000","last_undegraded":"2024-04-02T05:03:21.154304+0000","last_fullsized":"2024-04-02T05:03:21.154304+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:55.111507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:01:55.111507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"101'128","reported_seq":1115,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163615+0000","last_change":"2024-04-02T05:00:52.682032+0000","last_active":"2024-04-02T05:03:21.163615+0000","last_peered":"2024-04-02T05:03:21.163615+0000","last_clean":"2024-04-02T05:03:21.163615+0000","last_became_active":"2024-04-02T05:00:47.116787+0000","last_became_peered":"2024-04-02T05:00:47.116787+0000","last_unstale":"2024-04-02T05:03:21.163615+0000","last_undegraded":"2024-04-02T05:03:21.163615+0000","last_fullsized":"2024-04-02T05:03:21.163615+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:52.681929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:52.681929+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1532,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254491+0000","last_change":"2024-04-02T05:02:25.859474+0000","last_active":"2024-04-02T05:03:21.254491+0000","last_peered":"2024-04-02T05:03:21.254491+0000","last_clean":"2024-04-02T05:03:21.254491+0000","last_became_active":"2024-04-02T04:59:00.105159+0000","last_became_peered":"2024-04-02T04:59:00.105159+0000","last_unstale":"2024-04-02T05:03:21.254491+0000","last_undegraded":"2024-04-02T05:03:21.254491+0000","last_fullsized":"2024-04-02T05:03:21.254491+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:25.859237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:25.859237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:42.013082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"642'14","reported_seq":1931,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301968+0000","last_change":"2024-04-02T05:03:14.982990+0000","last_active":"2024-04-02T05:03:21.301968+0000","last_peered":"2024-04-02T05:03:21.301968+0000","last_clean":"2024-04-02T05:03:21.301968+0000","last_became_active":"2024-04-02T04:59:00.137930+0000","last_became_peered":"2024-04-02T04:59:00.137930+0000","last_unstale":"2024-04-02T05:03:21.301968+0000","last_undegraded":"2024-04-02T05:03:21.301968+0000","last_fullsized":"2024-04-02T05:03:21.301968+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-04-02T05:03:14.982752+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:47:42.831447+0000","last_clean_scrub_stamp":"2024-04-02T05:03:14.982752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:15.703338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"402'25","reported_seq":1199,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298442+0000","last_change":"2024-04-02T05:03:12.266857+0000","last_active":"2024-04-02T05:03:21.298442+0000","last_peered":"2024-04-02T05:03:21.298442+0000","last_clean":"2024-04-02T05:03:21.298442+0000","last_became_active":"2024-04-02T04:59:20.080550+0000","last_became_peered":"2024-04-02T04:59:20.080550+0000","last_unstale":"2024-04-02T05:03:21.298442+0000","last_undegraded":"2024-04-02T05:03:21.298442+0000","last_fullsized":"2024-04-02T05:03:21.298442+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T05:03:12.265344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:03:12.265344+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:28.843895+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":1698,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:22.892970+0000","last_change":"2024-04-02T05:03:22.893045+0000","last_active":"2024-04-02T05:03:22.892970+0000","last_peered":"2024-04-02T05:03:22.892970+0000","last_clean":"2024-04-02T05:03:22.892970+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T05:03:22.892970+0000","last_undegraded":"2024-04-02T05:03:22.892970+0000","last_fullsized":"2024-04-02T05:03:22.892970+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T05:03:22.892819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:03:22.892819+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:52.469041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.159086+0000","last_change":"2024-04-02T05:01:08.610530+0000","last_active":"2024-04-02T05:03:21.159086+0000","last_peered":"2024-04-02T05:03:21.159086+0000","last_clean":"2024-04-02T05:03:21.159086+0000","last_became_active":"2024-04-02T04:59:00.122883+0000","last_became_peered":"2024-04-02T04:59:00.122883+0000","last_unstale":"2024-04-02T05:03:21.159086+0000","last_undegraded":"2024-04-02T05:03:21.159086+0000","last_fullsized":"2024-04-02T05:03:21.159086+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:08.610432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:01:08.610432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"101'128","reported_seq":1642,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.244809+0000","last_change":"2024-04-02T05:00:56.177719+0000","last_active":"2024-04-02T05:03:21.244809+0000","last_peered":"2024-04-02T05:03:21.244809+0000","last_clean":"2024-04-02T05:03:21.244809+0000","last_became_active":"2024-04-02T05:00:44.136883+0000","last_became_peered":"2024-04-02T05:00:44.136883+0000","last_unstale":"2024-04-02T05:03:21.244809+0000","last_undegraded":"2024-04-02T05:03:21.244809+0000","last_fullsized":"2024-04-02T05:03:21.244809+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:56.175413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:56.175413+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1400,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.244500+0000","last_change":"2024-04-02T05:03:20.231248+0000","last_active":"2024-04-02T05:03:21.244500+0000","last_peered":"2024-04-02T05:03:21.244500+0000","last_clean":"2024-04-02T05:03:21.244500+0000","last_became_active":"2024-04-02T04:59:00.106070+0000","last_became_peered":"2024-04-02T04:59:00.106070+0000","last_unstale":"2024-04-02T05:03:21.244500+0000","last_undegraded":"2024-04-02T05:03:21.244500+0000","last_fullsized":"2024-04-02T05:03:21.244500+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:20.231166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:30.126652+0000","last_clean_scrub_stamp":"2024-04-02T05:03:20.231166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:33.178589+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"402'25","reported_seq":1169,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294007+0000","last_change":"2024-04-02T05:02:17.430287+0000","last_active":"2024-04-02T05:03:21.294007+0000","last_peered":"2024-04-02T05:03:21.294007+0000","last_clean":"2024-04-02T05:03:21.294007+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T05:03:21.294007+0000","last_undegraded":"2024-04-02T05:03:21.294007+0000","last_fullsized":"2024-04-02T05:03:21.294007+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T05:02:17.429910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:02:17.429910+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:23.855086+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":48,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.247529+0000","last_change":"2024-04-02T05:03:21.247639+0000","last_active":"2024-04-02T05:03:21.247529+0000","last_peered":"2024-04-02T05:03:21.247529+0000","last_clean":"2024-04-02T05:03:21.247529+0000","last_became_active":"2024-04-02T05:03:07.565981+0000","last_became_peered":"2024-04-02T05:03:07.565981+0000","last_unstale":"2024-04-02T05:03:21.247529+0000","last_undegraded":"2024-04-02T05:03:21.247529+0000","last_fullsized":"2024-04-02T05:03:21.247529+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:21.247321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:49.839000+0000","last_clean_scrub_stamp":"2024-04-02T05:03:21.247321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:50.498400+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1661,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.157776+0000","last_change":"2024-04-02T05:02:28.395643+0000","last_active":"2024-04-02T05:03:21.157776+0000","last_peered":"2024-04-02T05:03:21.157776+0000","last_clean":"2024-04-02T05:03:21.157776+0000","last_became_active":"2024-04-02T04:59:00.138663+0000","last_became_peered":"2024-04-02T04:59:00.138663+0000","last_unstale":"2024-04-02T05:03:21.157776+0000","last_undegraded":"2024-04-02T05:03:21.157776+0000","last_fullsized":"2024-04-02T05:03:21.157776+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:28.395512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:35.398676+0000","last_clean_scrub_stamp":"2024-04-02T05:02:28.395512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:32.672958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":703,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.850284+0000","last_change":"2024-04-02T05:03:21.850342+0000","last_active":"2024-04-02T05:03:21.850284+0000","last_peered":"2024-04-02T05:03:21.850284+0000","last_clean":"2024-04-02T05:03:21.850284+0000","last_became_active":"2024-04-02T05:03:07.481815+0000","last_became_peered":"2024-04-02T05:03:07.481815+0000","last_unstale":"2024-04-02T05:03:21.850284+0000","last_undegraded":"2024-04-02T05:03:21.850284+0000","last_fullsized":"2024-04-02T05:03:21.850284+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:21.850129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:03:21.850129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:48.722044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"101'128","reported_seq":1101,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:25.225141+0000","last_change":"2024-04-02T05:03:25.225192+0000","last_active":"2024-04-02T05:03:25.225141+0000","last_peered":"2024-04-02T05:03:25.225141+0000","last_clean":"2024-04-02T05:03:25.225141+0000","last_became_active":"2024-04-02T05:00:47.112033+0000","last_became_peered":"2024-04-02T05:00:47.112033+0000","last_unstale":"2024-04-02T05:03:25.225141+0000","last_undegraded":"2024-04-02T05:03:25.225141+0000","last_fullsized":"2024-04-02T05:03:25.225141+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:03:25.225041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:03:25.225041+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:29.265532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1525,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.243084+0000","last_change":"2024-04-02T05:03:16.175888+0000","last_active":"2024-04-02T05:03:21.243084+0000","last_peered":"2024-04-02T05:03:21.243084+0000","last_clean":"2024-04-02T05:03:21.243084+0000","last_became_active":"2024-04-02T04:59:00.104872+0000","last_became_peered":"2024-04-02T04:59:00.104872+0000","last_unstale":"2024-04-02T05:03:21.243084+0000","last_undegraded":"2024-04-02T05:03:21.243084+0000","last_fullsized":"2024-04-02T05:03:21.243084+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:16.175756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:03:16.175756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:44.640668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"402'8","reported_seq":1696,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250821+0000","last_change":"2024-04-02T05:01:57.570903+0000","last_active":"2024-04-02T05:03:21.250821+0000","last_peered":"2024-04-02T05:03:21.250821+0000","last_clean":"2024-04-02T05:03:21.250821+0000","last_became_active":"2024-04-02T04:59:00.105124+0000","last_became_peered":"2024-04-02T04:59:00.105124+0000","last_unstale":"2024-04-02T05:03:21.250821+0000","last_undegraded":"2024-04-02T05:03:21.250821+0000","last_fullsized":"2024-04-02T05:03:21.250821+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'8","last_scrub_stamp":"2024-04-02T05:01:57.570785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:01:57.570785+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1658,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.250626+0000","last_change":"2024-04-02T05:02:01.699019+0000","last_active":"2024-04-02T05:03:21.250626+0000","last_peered":"2024-04-02T05:03:21.250626+0000","last_clean":"2024-04-02T05:03:21.250626+0000","last_became_active":"2024-04-02T04:59:20.074948+0000","last_became_peered":"2024-04-02T04:59:20.074948+0000","last_unstale":"2024-04-02T05:03:21.250626+0000","last_undegraded":"2024-04-02T05:03:21.250626+0000","last_fullsized":"2024-04-02T05:03:21.250626+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:01.698808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:07.675606+0000","last_clean_scrub_stamp":"2024-04-02T05:02:01.698808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.162625+0000","last_change":"2024-04-02T05:01:02.623605+0000","last_active":"2024-04-02T05:03:21.162625+0000","last_peered":"2024-04-02T05:03:21.162625+0000","last_clean":"2024-04-02T05:03:21.162625+0000","last_became_active":"2024-04-02T04:59:00.122643+0000","last_became_peered":"2024-04-02T04:59:00.122643+0000","last_unstale":"2024-04-02T05:03:21.162625+0000","last_undegraded":"2024-04-02T05:03:21.162625+0000","last_fullsized":"2024-04-02T05:03:21.162625+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:02.623388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T05:01:02.623388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1123,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.295712+0000","last_change":"2024-04-02T05:01:06.475976+0000","last_active":"2024-04-02T05:03:21.295712+0000","last_peered":"2024-04-02T05:03:21.295712+0000","last_clean":"2024-04-02T05:03:21.295712+0000","last_became_active":"2024-04-02T04:59:20.069216+0000","last_became_peered":"2024-04-02T04:59:20.069216+0000","last_unstale":"2024-04-02T05:03:21.295712+0000","last_undegraded":"2024-04-02T05:03:21.295712+0000","last_fullsized":"2024-04-02T05:03:21.295712+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:06.475834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:01:06.475834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"101'128","reported_seq":1423,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.153622+0000","last_change":"2024-04-02T05:03:11.477191+0000","last_active":"2024-04-02T05:03:21.153622+0000","last_peered":"2024-04-02T05:03:21.153622+0000","last_clean":"2024-04-02T05:03:21.153622+0000","last_became_active":"2024-04-02T05:02:42.156808+0000","last_became_peered":"2024-04-02T05:02:42.156808+0000","last_unstale":"2024-04-02T05:03:21.153622+0000","last_undegraded":"2024-04-02T05:03:21.153622+0000","last_fullsized":"2024-04-02T05:03:21.153622+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:03:11.477074+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T04:55:28.185555+0000","last_clean_scrub_stamp":"2024-04-02T05:03:11.477074+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:26.677644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1251,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:26.530150+0000","last_change":"2024-04-02T05:03:26.530191+0000","last_active":"2024-04-02T05:03:26.530150+0000","last_peered":"2024-04-02T05:03:26.530150+0000","last_clean":"2024-04-02T05:03:26.530150+0000","last_became_active":"2024-04-02T04:57:38.079839+0000","last_became_peered":"2024-04-02T04:57:38.079839+0000","last_unstale":"2024-04-02T05:03:26.530150+0000","last_undegraded":"2024-04-02T05:03:26.530150+0000","last_fullsized":"2024-04-02T05:03:26.530150+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:26.530036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T05:03:26.530036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:50.925080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"402'19","reported_seq":1134,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294249+0000","last_change":"2024-04-02T05:00:08.520683+0000","last_active":"2024-04-02T05:03:21.294249+0000","last_peered":"2024-04-02T05:03:21.294249+0000","last_clean":"2024-04-02T05:03:21.294249+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T05:03:21.294249+0000","last_undegraded":"2024-04-02T05:03:21.294249+0000","last_fullsized":"2024-04-02T05:03:21.294249+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'19","last_scrub_stamp":"2024-04-02T05:00:08.520552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:08.520552+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160823+0000","last_change":"2024-04-02T05:01:01.676655+0000","last_active":"2024-04-02T05:03:21.160823+0000","last_peered":"2024-04-02T05:03:21.160823+0000","last_clean":"2024-04-02T05:03:21.160823+0000","last_became_active":"2024-04-02T04:59:00.134529+0000","last_became_peered":"2024-04-02T04:59:00.134529+0000","last_unstale":"2024-04-02T05:03:21.160823+0000","last_undegraded":"2024-04-02T05:03:21.160823+0000","last_fullsized":"2024-04-02T05:03:21.160823+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:01.676566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:01.676566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"642'17","reported_seq":544,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.164154+0000","last_change":"2024-04-02T05:00:58.668882+0000","last_active":"2024-04-02T05:03:21.164154+0000","last_peered":"2024-04-02T05:03:21.164154+0000","last_clean":"2024-04-02T05:03:21.164154+0000","last_became_active":"2024-04-02T04:59:00.133162+0000","last_became_peered":"2024-04-02T04:59:00.133162+0000","last_unstale":"2024-04-02T05:03:21.164154+0000","last_undegraded":"2024-04-02T05:03:21.164154+0000","last_fullsized":"2024-04-02T05:03:21.164154+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'17","last_scrub_stamp":"2024-04-02T05:00:58.668709+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T05:00:58.668709+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1251,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298507+0000","last_change":"2024-04-02T05:02:42.158803+0000","last_active":"2024-04-02T05:03:21.298507+0000","last_peered":"2024-04-02T05:03:21.298507+0000","last_clean":"2024-04-02T05:03:21.298507+0000","last_became_active":"2024-04-02T05:02:42.158370+0000","last_became_peered":"2024-04-02T05:02:42.158370+0000","last_unstale":"2024-04-02T05:03:21.298507+0000","last_undegraded":"2024-04-02T05:03:21.298507+0000","last_fullsized":"2024-04-02T05:03:21.298507+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:17.214556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:17.214556+0000","last_clean_scrub_stamp":"2024-04-02T05:02:17.214556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:28.046752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"101'128","reported_seq":1423,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.292787+0000","last_change":"2024-04-02T05:00:53.437707+0000","last_active":"2024-04-02T05:03:21.292787+0000","last_peered":"2024-04-02T05:03:21.292787+0000","last_clean":"2024-04-02T05:03:21.292787+0000","last_became_active":"2024-04-02T05:00:47.084122+0000","last_became_peered":"2024-04-02T05:00:47.084122+0000","last_unstale":"2024-04-02T05:03:21.292787+0000","last_undegraded":"2024-04-02T05:03:21.292787+0000","last_fullsized":"2024-04-02T05:03:21.292787+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:00:53.437610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:00:53.437610+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":500,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158685+0000","last_change":"2024-04-02T05:02:04.708433+0000","last_active":"2024-04-02T05:03:21.158685+0000","last_peered":"2024-04-02T05:03:21.158685+0000","last_clean":"2024-04-02T05:03:21.158685+0000","last_became_active":"2024-04-02T04:59:20.067119+0000","last_became_peered":"2024-04-02T04:59:20.067119+0000","last_unstale":"2024-04-02T05:03:21.158685+0000","last_undegraded":"2024-04-02T05:03:21.158685+0000","last_fullsized":"2024-04-02T05:03:21.158685+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:04.708320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:04.708320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"402'15","reported_seq":798,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293763+0000","last_change":"2024-04-02T05:02:42.159033+0000","last_active":"2024-04-02T05:03:21.293763+0000","last_peered":"2024-04-02T05:03:21.293763+0000","last_clean":"2024-04-02T05:03:21.293763+0000","last_became_active":"2024-04-02T05:02:42.158229+0000","last_became_peered":"2024-04-02T05:02:42.158229+0000","last_unstale":"2024-04-02T05:03:21.293763+0000","last_undegraded":"2024-04-02T05:03:21.293763+0000","last_fullsized":"2024-04-02T05:03:21.293763+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'15","last_scrub_stamp":"2024-04-02T05:01:22.858597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:01:22.858597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1270,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302159+0000","last_change":"2024-04-02T05:02:42.158563+0000","last_active":"2024-04-02T05:03:21.302159+0000","last_peered":"2024-04-02T05:03:21.302159+0000","last_clean":"2024-04-02T05:03:21.302159+0000","last_became_active":"2024-04-02T05:02:42.158179+0000","last_became_peered":"2024-04-02T05:02:42.158179+0000","last_unstale":"2024-04-02T05:03:21.302159+0000","last_undegraded":"2024-04-02T05:03:21.302159+0000","last_fullsized":"2024-04-02T05:03:21.302159+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:03.179496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:02:03.179496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:27.112398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"642'16","reported_seq":1793,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298730+0000","last_change":"2024-04-02T05:02:12.368607+0000","last_active":"2024-04-02T05:03:21.298730+0000","last_peered":"2024-04-02T05:03:21.298730+0000","last_clean":"2024-04-02T05:03:21.298730+0000","last_became_active":"2024-04-02T04:59:00.136907+0000","last_became_peered":"2024-04-02T04:59:00.136907+0000","last_unstale":"2024-04-02T05:03:21.298730+0000","last_undegraded":"2024-04-02T05:03:21.298730+0000","last_fullsized":"2024-04-02T05:03:21.298730+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'16","last_scrub_stamp":"2024-04-02T05:02:12.368369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:12.368369+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:42.222045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1659,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.303262+0000","last_change":"2024-04-02T05:02:14.398047+0000","last_active":"2024-04-02T05:03:21.303262+0000","last_peered":"2024-04-02T05:03:21.303262+0000","last_clean":"2024-04-02T05:03:21.303262+0000","last_became_active":"2024-04-02T04:54:56.000394+0000","last_became_peered":"2024-04-02T04:54:56.000394+0000","last_unstale":"2024-04-02T05:03:21.303262+0000","last_undegraded":"2024-04-02T05:03:21.303262+0000","last_fullsized":"2024-04-02T05:03:21.303262+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:14.397839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:14.397839+0000","last_clean_scrub_stamp":"2024-04-02T05:02:14.397839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:32.621983+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1499,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253733+0000","last_change":"2024-04-02T05:01:59.663154+0000","last_active":"2024-04-02T05:03:21.253733+0000","last_peered":"2024-04-02T05:03:21.253733+0000","last_clean":"2024-04-02T05:03:21.253733+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T05:03:21.253733+0000","last_undegraded":"2024-04-02T05:03:21.253733+0000","last_fullsized":"2024-04-02T05:03:21.253733+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:59.663039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:01:59.663039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"402'21","reported_seq":1214,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298573+0000","last_change":"2024-04-02T05:03:15.224169+0000","last_active":"2024-04-02T05:03:21.298573+0000","last_peered":"2024-04-02T05:03:21.298573+0000","last_clean":"2024-04-02T05:03:21.298573+0000","last_became_active":"2024-04-02T04:59:00.137738+0000","last_became_peered":"2024-04-02T04:59:00.137738+0000","last_unstale":"2024-04-02T05:03:21.298573+0000","last_undegraded":"2024-04-02T05:03:21.298573+0000","last_fullsized":"2024-04-02T05:03:21.298573+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T05:03:15.223814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:03:15.223814+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:38.423589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1662,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253742+0000","last_change":"2024-04-02T05:01:21.015956+0000","last_active":"2024-04-02T05:03:21.253742+0000","last_peered":"2024-04-02T05:03:21.253742+0000","last_clean":"2024-04-02T05:03:21.253742+0000","last_became_active":"2024-04-02T04:59:00.105424+0000","last_became_peered":"2024-04-02T04:59:00.105424+0000","last_unstale":"2024-04-02T05:03:21.253742+0000","last_undegraded":"2024-04-02T05:03:21.253742+0000","last_fullsized":"2024-04-02T05:03:21.253742+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:21.015847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:21.015847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":609,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251033+0000","last_change":"2024-04-02T05:02:12.688097+0000","last_active":"2024-04-02T05:03:21.251033+0000","last_peered":"2024-04-02T05:03:21.251033+0000","last_clean":"2024-04-02T05:03:21.251033+0000","last_became_active":"2024-04-02T04:57:38.083311+0000","last_became_peered":"2024-04-02T04:57:38.083311+0000","last_unstale":"2024-04-02T05:03:21.251033+0000","last_undegraded":"2024-04-02T05:03:21.251033+0000","last_fullsized":"2024-04-02T05:03:21.251033+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:12.687855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:12.687855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:25.915508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1650,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.153949+0000","last_change":"2024-04-02T05:00:54.811201+0000","last_active":"2024-04-02T05:03:21.153949+0000","last_peered":"2024-04-02T05:03:21.153949+0000","last_clean":"2024-04-02T05:03:21.153949+0000","last_became_active":"2024-04-02T04:54:56.005128+0000","last_became_peered":"2024-04-02T04:54:56.005128+0000","last_unstale":"2024-04-02T05:03:21.153949+0000","last_undegraded":"2024-04-02T05:03:21.153949+0000","last_fullsized":"2024-04-02T05:03:21.153949+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:54.811109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:54.811109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1521,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302109+0000","last_change":"2024-04-02T05:02:07.345214+0000","last_active":"2024-04-02T05:03:21.302109+0000","last_peered":"2024-04-02T05:03:21.302109+0000","last_clean":"2024-04-02T05:03:21.302109+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T05:03:21.302109+0000","last_undegraded":"2024-04-02T05:03:21.302109+0000","last_fullsized":"2024-04-02T05:03:21.302109+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:07.345000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:07.345000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"402'32","reported_seq":1203,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.294986+0000","last_change":"2024-04-02T05:02:08.452482+0000","last_active":"2024-04-02T05:03:21.294986+0000","last_peered":"2024-04-02T05:03:21.294986+0000","last_clean":"2024-04-02T05:03:21.294986+0000","last_became_active":"2024-04-02T04:59:20.057918+0000","last_became_peered":"2024-04-02T04:59:20.057918+0000","last_unstale":"2024-04-02T05:03:21.294986+0000","last_undegraded":"2024-04-02T05:03:21.294986+0000","last_fullsized":"2024-04-02T05:03:21.294986+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'32","last_scrub_stamp":"2024-04-02T05:02:08.452218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:02:08.452218+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1663,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.248930+0000","last_change":"2024-04-02T05:01:59.055393+0000","last_active":"2024-04-02T05:03:21.248930+0000","last_peered":"2024-04-02T05:03:21.248930+0000","last_clean":"2024-04-02T05:03:21.248930+0000","last_became_active":"2024-04-02T04:54:55.987373+0000","last_became_peered":"2024-04-02T04:54:55.987373+0000","last_unstale":"2024-04-02T05:03:21.248930+0000","last_undegraded":"2024-04-02T05:03:21.248930+0000","last_fullsized":"2024-04-02T05:03:21.248930+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:59.055175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:59.055175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1665,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301637+0000","last_change":"2024-04-02T05:02:20.263512+0000","last_active":"2024-04-02T05:03:21.301637+0000","last_peered":"2024-04-02T05:03:21.301637+0000","last_clean":"2024-04-02T05:03:21.301637+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T05:03:21.301637+0000","last_undegraded":"2024-04-02T05:03:21.301637+0000","last_fullsized":"2024-04-02T05:03:21.301637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:20.263282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_clean_scrub_stamp":"2024-04-02T05:02:20.263282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:48.589692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":500,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156699+0000","last_change":"2024-04-02T05:03:07.553336+0000","last_active":"2024-04-02T05:03:21.156699+0000","last_peered":"2024-04-02T05:03:21.156699+0000","last_clean":"2024-04-02T05:03:21.156699+0000","last_became_active":"2024-04-02T05:03:07.551168+0000","last_became_peered":"2024-04-02T05:03:07.551168+0000","last_unstale":"2024-04-02T05:03:21.156699+0000","last_undegraded":"2024-04-02T05:03:21.156699+0000","last_fullsized":"2024-04-02T05:03:21.156699+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:21.794018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T05:02:21.794018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:33.259150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1520,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156668+0000","last_change":"2024-04-02T05:02:04.177883+0000","last_active":"2024-04-02T05:03:21.156668+0000","last_peered":"2024-04-02T05:03:21.156668+0000","last_clean":"2024-04-02T05:03:21.156668+0000","last_became_active":"2024-04-02T04:54:56.017129+0000","last_became_peered":"2024-04-02T04:54:56.017129+0000","last_unstale":"2024-04-02T05:03:21.156668+0000","last_undegraded":"2024-04-02T05:03:21.156668+0000","last_fullsized":"2024-04-02T05:03:21.156668+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:04.177791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T05:02:04.177791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:32.455552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"402'18","reported_seq":1331,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160675+0000","last_change":"2024-04-02T05:02:42.160657+0000","last_active":"2024-04-02T05:03:21.160675+0000","last_peered":"2024-04-02T05:03:21.160675+0000","last_clean":"2024-04-02T05:03:21.160675+0000","last_became_active":"2024-04-02T05:02:42.159557+0000","last_became_peered":"2024-04-02T05:02:42.159557+0000","last_unstale":"2024-04-02T05:03:21.160675+0000","last_undegraded":"2024-04-02T05:03:21.160675+0000","last_fullsized":"2024-04-02T05:03:21.160675+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'18","last_scrub_stamp":"2024-04-02T05:00:21.863002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:21.863002+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":881,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253747+0000","last_change":"2024-04-02T05:03:07.493400+0000","last_active":"2024-04-02T05:03:21.253747+0000","last_peered":"2024-04-02T05:03:21.253747+0000","last_clean":"2024-04-02T05:03:21.253747+0000","last_became_active":"2024-04-02T05:03:07.493017+0000","last_became_peered":"2024-04-02T05:03:07.493017+0000","last_unstale":"2024-04-02T05:03:21.253747+0000","last_undegraded":"2024-04-02T05:03:21.253747+0000","last_fullsized":"2024-04-02T05:03:21.253747+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:27.895598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:27.895598+0000","last_clean_scrub_stamp":"2024-04-02T05:02:27.895598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:29.751057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":494,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.152978+0000","last_change":"2024-04-02T05:03:18.462127+0000","last_active":"2024-04-02T05:03:21.152978+0000","last_peered":"2024-04-02T05:03:21.152978+0000","last_clean":"2024-04-02T05:03:21.152978+0000","last_became_active":"2024-04-02T04:59:00.102092+0000","last_became_peered":"2024-04-02T04:59:00.102092+0000","last_unstale":"2024-04-02T05:03:21.152978+0000","last_undegraded":"2024-04-02T05:03:21.152978+0000","last_fullsized":"2024-04-02T05:03:21.152978+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:18.462004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.801839+0000","last_clean_scrub_stamp":"2024-04-02T05:03:18.462004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:46.106012+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1120,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293797+0000","last_change":"2024-04-02T05:02:27.268504+0000","last_active":"2024-04-02T05:03:21.293797+0000","last_peered":"2024-04-02T05:03:21.293797+0000","last_clean":"2024-04-02T05:03:21.293797+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T05:03:21.293797+0000","last_undegraded":"2024-04-02T05:03:21.293797+0000","last_fullsized":"2024-04-02T05:03:21.293797+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:27.268316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T05:02:27.268316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:55.614137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"101'128","reported_seq":1111,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:23.976641+0000","last_change":"2024-04-02T05:03:23.976703+0000","last_active":"2024-04-02T05:03:23.976641+0000","last_peered":"2024-04-02T05:03:23.976641+0000","last_clean":"2024-04-02T05:03:23.976641+0000","last_became_active":"2024-04-02T05:00:47.101836+0000","last_became_peered":"2024-04-02T05:00:47.101836+0000","last_unstale":"2024-04-02T05:03:23.976641+0000","last_undegraded":"2024-04-02T05:03:23.976641+0000","last_fullsized":"2024-04-02T05:03:23.976641+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:03:23.976487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:03:23.976487+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:36.551064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1005,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298832+0000","last_change":"2024-04-02T05:03:07.470705+0000","last_active":"2024-04-02T05:03:21.298832+0000","last_peered":"2024-04-02T05:03:21.298832+0000","last_clean":"2024-04-02T05:03:21.298832+0000","last_became_active":"2024-04-02T05:03:07.469959+0000","last_became_peered":"2024-04-02T05:03:07.469959+0000","last_unstale":"2024-04-02T05:03:21.298832+0000","last_undegraded":"2024-04-02T05:03:21.298832+0000","last_fullsized":"2024-04-02T05:03:21.298832+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:24.327579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:24.327579+0000","last_clean_scrub_stamp":"2024-04-02T05:02:24.327579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:51.237605+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"402'27","reported_seq":1669,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.298580+0000","last_change":"2024-04-02T05:01:07.526039+0000","last_active":"2024-04-02T05:03:21.298580+0000","last_peered":"2024-04-02T05:03:21.298580+0000","last_clean":"2024-04-02T05:03:21.298580+0000","last_became_active":"2024-04-02T04:59:00.136183+0000","last_became_peered":"2024-04-02T04:59:00.136183+0000","last_unstale":"2024-04-02T05:03:21.298580+0000","last_undegraded":"2024-04-02T05:03:21.298580+0000","last_fullsized":"2024-04-02T05:03:21.298580+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T05:01:07.525794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:01:07.525794+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1672,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254577+0000","last_change":"2024-04-02T05:03:09.938981+0000","last_active":"2024-04-02T05:03:21.254577+0000","last_peered":"2024-04-02T05:03:21.254577+0000","last_clean":"2024-04-02T05:03:21.254577+0000","last_became_active":"2024-04-02T05:03:07.494565+0000","last_became_peered":"2024-04-02T05:03:07.494565+0000","last_unstale":"2024-04-02T05:03:21.254577+0000","last_undegraded":"2024-04-02T05:03:21.254577+0000","last_fullsized":"2024-04-02T05:03:21.254577+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:09.938876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:58.795150+0000","last_clean_scrub_stamp":"2024-04-02T05:03:09.938876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:38.054088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.161919+0000","last_change":"2024-04-02T05:01:11.629900+0000","last_active":"2024-04-02T05:03:21.161919+0000","last_peered":"2024-04-02T05:03:21.161919+0000","last_clean":"2024-04-02T05:03:21.161919+0000","last_became_active":"2024-04-02T04:59:00.122631+0000","last_became_peered":"2024-04-02T04:59:00.122631+0000","last_unstale":"2024-04-02T05:03:21.161919+0000","last_undegraded":"2024-04-02T05:03:21.161919+0000","last_fullsized":"2024-04-02T05:03:21.161919+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:11.629796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:01:11.629796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":492,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.159068+0000","last_change":"2024-04-02T05:03:07.557524+0000","last_active":"2024-04-02T05:03:21.159068+0000","last_peered":"2024-04-02T05:03:21.159068+0000","last_clean":"2024-04-02T05:03:21.159068+0000","last_became_active":"2024-04-02T05:03:07.556925+0000","last_became_peered":"2024-04-02T05:03:07.556925+0000","last_unstale":"2024-04-02T05:03:21.159068+0000","last_undegraded":"2024-04-02T05:03:21.159068+0000","last_fullsized":"2024-04-02T05:03:21.159068+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:18.800043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:00:18.800043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":964,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.293241+0000","last_change":"2024-04-02T05:01:19.497650+0000","last_active":"2024-04-02T05:03:21.293241+0000","last_peered":"2024-04-02T05:03:21.293241+0000","last_clean":"2024-04-02T05:03:21.293241+0000","last_became_active":"2024-04-02T04:59:20.054470+0000","last_became_peered":"2024-04-02T04:59:20.054470+0000","last_unstale":"2024-04-02T05:03:21.293241+0000","last_undegraded":"2024-04-02T05:03:21.293241+0000","last_fullsized":"2024-04-02T05:03:21.293241+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:19.497409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:01:19.497409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"402'62","reported_seq":1845,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.243590+0000","last_change":"2024-04-02T05:02:16.066537+0000","last_active":"2024-04-02T05:03:21.243590+0000","last_peered":"2024-04-02T05:03:21.243590+0000","last_clean":"2024-04-02T05:03:21.243590+0000","last_became_active":"2024-04-02T04:59:20.053373+0000","last_became_peered":"2024-04-02T04:59:20.053373+0000","last_unstale":"2024-04-02T05:03:21.243590+0000","last_undegraded":"2024-04-02T05:03:21.243590+0000","last_fullsized":"2024-04-02T05:03:21.243590+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'62","last_scrub_stamp":"2024-04-02T05:02:16.066420+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T05:02:16.066420+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:40.078994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1569,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301372+0000","last_change":"2024-04-02T05:03:14.013839+0000","last_active":"2024-04-02T05:03:21.301372+0000","last_peered":"2024-04-02T05:03:21.301372+0000","last_clean":"2024-04-02T05:03:21.301372+0000","last_became_active":"2024-04-02T04:54:56.008703+0000","last_became_peered":"2024-04-02T04:54:56.008703+0000","last_unstale":"2024-04-02T05:03:21.301372+0000","last_undegraded":"2024-04-02T05:03:21.301372+0000","last_fullsized":"2024-04-02T05:03:21.301372+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:14.013633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:03:14.013633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:25.796580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1662,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.156920+0000","last_change":"2024-04-02T05:00:14.896567+0000","last_active":"2024-04-02T05:03:21.156920+0000","last_peered":"2024-04-02T05:03:21.156920+0000","last_clean":"2024-04-02T05:03:21.156920+0000","last_became_active":"2024-04-02T04:59:00.138125+0000","last_became_peered":"2024-04-02T04:59:00.138125+0000","last_unstale":"2024-04-02T05:03:21.156920+0000","last_undegraded":"2024-04-02T05:03:21.156920+0000","last_fullsized":"2024-04-02T05:03:21.156920+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:14.896340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:00:14.896340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":1150,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:22.283600+0000","last_change":"2024-04-02T05:03:22.283667+0000","last_active":"2024-04-02T05:03:22.283600+0000","last_peered":"2024-04-02T05:03:22.283600+0000","last_clean":"2024-04-02T05:03:22.283600+0000","last_became_active":"2024-04-02T04:59:00.138970+0000","last_became_peered":"2024-04-02T04:59:00.138970+0000","last_unstale":"2024-04-02T05:03:22.283600+0000","last_undegraded":"2024-04-02T05:03:22.283600+0000","last_fullsized":"2024-04-02T05:03:22.283600+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-02T05:03:22.283436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:03:22.283436+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:45.593920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1402,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.155219+0000","last_change":"2024-04-02T05:02:10.250735+0000","last_active":"2024-04-02T05:03:21.155219+0000","last_peered":"2024-04-02T05:03:21.155219+0000","last_clean":"2024-04-02T05:03:21.155219+0000","last_became_active":"2024-04-02T04:54:55.989605+0000","last_became_peered":"2024-04-02T04:54:55.989605+0000","last_unstale":"2024-04-02T05:03:21.155219+0000","last_undegraded":"2024-04-02T05:03:21.155219+0000","last_fullsized":"2024-04-02T05:03:21.155219+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:10.250638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:59:15.923777+0000","last_clean_scrub_stamp":"2024-04-02T05:02:10.250638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:31.883441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"402'21","reported_seq":498,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.153113+0000","last_change":"2024-04-02T05:00:54.692563+0000","last_active":"2024-04-02T05:03:21.153113+0000","last_peered":"2024-04-02T05:03:21.153113+0000","last_clean":"2024-04-02T05:03:21.153113+0000","last_became_active":"2024-04-02T04:59:00.133276+0000","last_became_peered":"2024-04-02T04:59:00.133276+0000","last_unstale":"2024-04-02T05:03:21.153113+0000","last_undegraded":"2024-04-02T05:03:21.153113+0000","last_fullsized":"2024-04-02T05:03:21.153113+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'21","last_scrub_stamp":"2024-04-02T05:00:54.692455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:00:54.692455+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":498,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158244+0000","last_change":"2024-04-02T05:02:22.847586+0000","last_active":"2024-04-02T05:03:21.158244+0000","last_peered":"2024-04-02T05:03:21.158244+0000","last_clean":"2024-04-02T05:03:21.158244+0000","last_became_active":"2024-04-02T04:59:00.135857+0000","last_became_peered":"2024-04-02T04:59:00.135857+0000","last_unstale":"2024-04-02T05:03:21.158244+0000","last_undegraded":"2024-04-02T05:03:21.158244+0000","last_fullsized":"2024-04-02T05:03:21.158244+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T05:02:22.847370+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T05:02:22.847370+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:26.424734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"451'12","reported_seq":1903,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254739+0000","last_change":"2024-04-02T05:02:30.918905+0000","last_active":"2024-04-02T05:03:21.254739+0000","last_peered":"2024-04-02T05:03:21.254739+0000","last_clean":"2024-04-02T05:03:21.254739+0000","last_became_active":"2024-04-02T04:54:56.006792+0000","last_became_peered":"2024-04-02T04:54:56.006792+0000","last_unstale":"2024-04-02T05:03:21.254739+0000","last_undegraded":"2024-04-02T05:03:21.254739+0000","last_fullsized":"2024-04-02T05:03:21.254739+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'12","last_scrub_stamp":"2024-04-02T05:02:30.918037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:30.918037+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:57.530773+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1656,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251771+0000","last_change":"2024-04-02T05:02:20.038782+0000","last_active":"2024-04-02T05:03:21.251771+0000","last_peered":"2024-04-02T05:03:21.251771+0000","last_clean":"2024-04-02T05:03:21.251771+0000","last_became_active":"2024-04-02T04:59:20.078674+0000","last_became_peered":"2024-04-02T04:59:20.078674+0000","last_unstale":"2024-04-02T05:03:21.251771+0000","last_undegraded":"2024-04-02T05:03:21.251771+0000","last_fullsized":"2024-04-02T05:03:21.251771+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:20.038583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:20.038583+0000","last_clean_scrub_stamp":"2024-04-02T05:02:20.038583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:26.523916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1512,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251401+0000","last_change":"2024-04-02T05:03:07.476845+0000","last_active":"2024-04-02T05:03:21.251401+0000","last_peered":"2024-04-02T05:03:21.251401+0000","last_clean":"2024-04-02T05:03:21.251401+0000","last_became_active":"2024-04-02T05:03:07.476404+0000","last_became_peered":"2024-04-02T05:03:07.476404+0000","last_unstale":"2024-04-02T05:03:21.251401+0000","last_undegraded":"2024-04-02T05:03:21.251401+0000","last_fullsized":"2024-04-02T05:03:21.251401+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:23.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:01:23.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"402'16","reported_seq":1680,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.159925+0000","last_change":"2024-04-02T05:02:09.256690+0000","last_active":"2024-04-02T05:03:21.159925+0000","last_peered":"2024-04-02T05:03:21.159925+0000","last_clean":"2024-04-02T05:03:21.159925+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T05:03:21.159925+0000","last_undegraded":"2024-04-02T05:03:21.159925+0000","last_fullsized":"2024-04-02T05:03:21.159925+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'16","last_scrub_stamp":"2024-04-02T05:02:09.256581+0000","last_deep_scrub":"116'14","last_deep_scrub_stamp":"2024-04-02T04:47:13.347670+0000","last_clean_scrub_stamp":"2024-04-02T05:02:09.256581+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1652,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251527+0000","last_change":"2024-04-02T05:00:51.973822+0000","last_active":"2024-04-02T05:03:21.251527+0000","last_peered":"2024-04-02T05:03:21.251527+0000","last_clean":"2024-04-02T05:03:21.251527+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T05:03:21.251527+0000","last_undegraded":"2024-04-02T05:03:21.251527+0000","last_fullsized":"2024-04-02T05:03:21.251527+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:51.973722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T05:00:51.973722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1550,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.251835+0000","last_change":"2024-04-02T05:02:15.115206+0000","last_active":"2024-04-02T05:03:21.251835+0000","last_peered":"2024-04-02T05:03:21.251835+0000","last_clean":"2024-04-02T05:03:21.251835+0000","last_became_active":"2024-04-02T04:59:00.106341+0000","last_became_peered":"2024-04-02T04:59:00.106341+0000","last_unstale":"2024-04-02T05:03:21.251835+0000","last_undegraded":"2024-04-02T05:03:21.251835+0000","last_fullsized":"2024-04-02T05:03:21.251835+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:15.114860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:02:15.114860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:24.216502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1622,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.254115+0000","last_change":"2024-04-02T05:02:00.691645+0000","last_active":"2024-04-02T05:03:21.254115+0000","last_peered":"2024-04-02T05:03:21.254115+0000","last_clean":"2024-04-02T05:03:21.254115+0000","last_became_active":"2024-04-02T04:59:20.063395+0000","last_became_peered":"2024-04-02T04:59:20.063395+0000","last_unstale":"2024-04-02T05:03:21.254115+0000","last_undegraded":"2024-04-02T05:03:21.254115+0000","last_fullsized":"2024-04-02T05:03:21.254115+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:00.691411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:02:00.691411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1521,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.247730+0000","last_change":"2024-04-02T05:03:07.547453+0000","last_active":"2024-04-02T05:03:21.247730+0000","last_peered":"2024-04-02T05:03:21.247730+0000","last_clean":"2024-04-02T05:03:21.247730+0000","last_became_active":"2024-04-02T05:03:07.545958+0000","last_became_peered":"2024-04-02T05:03:07.545958+0000","last_unstale":"2024-04-02T05:03:21.247730+0000","last_undegraded":"2024-04-02T05:03:21.247730+0000","last_fullsized":"2024-04-02T05:03:21.247730+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:11.136861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:11.136861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:22.251854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"402'35","reported_seq":1737,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301661+0000","last_change":"2024-04-02T05:03:07.570185+0000","last_active":"2024-04-02T05:03:21.301661+0000","last_peered":"2024-04-02T05:03:21.301661+0000","last_clean":"2024-04-02T05:03:21.301661+0000","last_became_active":"2024-04-02T05:03:07.483047+0000","last_became_peered":"2024-04-02T05:03:07.483047+0000","last_unstale":"2024-04-02T05:03:21.301661+0000","last_undegraded":"2024-04-02T05:03:21.301661+0000","last_fullsized":"2024-04-02T05:03:21.301661+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'35","last_scrub_stamp":"2024-04-02T05:02:01.372783+0000","last_deep_scrub":"121'29","last_deep_scrub_stamp":"2024-04-02T04:47:40.838617+0000","last_clean_scrub_stamp":"2024-04-02T05:02:01.372783+0000","objects_scrubbed":6,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158904+0000","last_change":"2024-04-02T05:01:07.622262+0000","last_active":"2024-04-02T05:03:21.158904+0000","last_peered":"2024-04-02T05:03:21.158904+0000","last_clean":"2024-04-02T05:03:21.158904+0000","last_became_active":"2024-04-02T04:59:00.122388+0000","last_became_peered":"2024-04-02T04:59:00.122388+0000","last_unstale":"2024-04-02T05:03:21.158904+0000","last_undegraded":"2024-04-02T05:03:21.158904+0000","last_fullsized":"2024-04-02T05:03:21.158904+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:07.622054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:01:07.622054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163338+0000","last_change":"2024-04-02T05:01:05.620400+0000","last_active":"2024-04-02T05:03:21.163338+0000","last_peered":"2024-04-02T05:03:21.163338+0000","last_clean":"2024-04-02T05:03:21.163338+0000","last_became_active":"2024-04-02T04:59:00.131269+0000","last_became_peered":"2024-04-02T04:59:00.131269+0000","last_unstale":"2024-04-02T05:03:21.163338+0000","last_undegraded":"2024-04-02T05:03:21.163338+0000","last_fullsized":"2024-04-02T05:03:21.163338+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:05.620195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:29.283405+0000","last_clean_scrub_stamp":"2024-04-02T05:01:05.620195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":501,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.162639+0000","last_change":"2024-04-02T05:03:07.552010+0000","last_active":"2024-04-02T05:03:21.162639+0000","last_peered":"2024-04-02T05:03:21.162639+0000","last_clean":"2024-04-02T05:03:21.162639+0000","last_became_active":"2024-04-02T05:03:07.550894+0000","last_became_peered":"2024-04-02T05:03:07.550894+0000","last_unstale":"2024-04-02T05:03:21.162639+0000","last_undegraded":"2024-04-02T05:03:21.162639+0000","last_fullsized":"2024-04-02T05:03:21.162639+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:28.760112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:02:28.760112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:42.095241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1520,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.249220+0000","last_change":"2024-04-02T05:02:25.942936+0000","last_active":"2024-04-02T05:03:21.249220+0000","last_peered":"2024-04-02T05:03:21.249220+0000","last_clean":"2024-04-02T05:03:21.249220+0000","last_became_active":"2024-04-02T04:59:20.081346+0000","last_became_peered":"2024-04-02T04:59:20.081346+0000","last_unstale":"2024-04-02T05:03:21.249220+0000","last_undegraded":"2024-04-02T05:03:21.249220+0000","last_fullsized":"2024-04-02T05:03:21.249220+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:25.942831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T05:02:25.942831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:30.291771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"402'24","reported_seq":1642,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302492+0000","last_change":"2024-04-02T05:03:17.916353+0000","last_active":"2024-04-02T05:03:21.302492+0000","last_peered":"2024-04-02T05:03:21.302492+0000","last_clean":"2024-04-02T05:03:21.302492+0000","last_became_active":"2024-04-02T04:54:56.014703+0000","last_became_peered":"2024-04-02T04:54:56.014703+0000","last_unstale":"2024-04-02T05:03:21.302492+0000","last_undegraded":"2024-04-02T05:03:21.302492+0000","last_fullsized":"2024-04-02T05:03:21.302492+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'24","last_scrub_stamp":"2024-04-02T05:03:17.916102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:03:17.916102+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:22.435159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'1","reported_seq":1152,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.295088+0000","last_change":"2024-04-02T05:03:11.217686+0000","last_active":"2024-04-02T05:03:21.295088+0000","last_peered":"2024-04-02T05:03:21.295088+0000","last_clean":"2024-04-02T05:03:21.295088+0000","last_became_active":"2024-04-02T04:57:38.129831+0000","last_became_peered":"2024-04-02T04:57:38.129831+0000","last_unstale":"2024-04-02T05:03:21.295088+0000","last_undegraded":"2024-04-02T05:03:21.295088+0000","last_fullsized":"2024-04-02T05:03:21.295088+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T05:03:11.217331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:03:11.217331+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:11.248782+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1637,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.160311+0000","last_change":"2024-04-02T05:03:17.405095+0000","last_active":"2024-04-02T05:03:21.160311+0000","last_peered":"2024-04-02T05:03:21.160311+0000","last_clean":"2024-04-02T05:03:21.160311+0000","last_became_active":"2024-04-02T05:03:07.551304+0000","last_became_peered":"2024-04-02T05:03:07.551304+0000","last_unstale":"2024-04-02T05:03:21.160311+0000","last_undegraded":"2024-04-02T05:03:21.160311+0000","last_fullsized":"2024-04-02T05:03:21.160311+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:17.404878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:03:17.404878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:31.820049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1645,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.154118+0000","last_change":"2024-04-02T05:02:25.384595+0000","last_active":"2024-04-02T05:03:21.154118+0000","last_peered":"2024-04-02T05:03:21.154118+0000","last_clean":"2024-04-02T05:03:21.154118+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T05:03:21.154118+0000","last_undegraded":"2024-04-02T05:03:21.154118+0000","last_fullsized":"2024-04-02T05:03:21.154118+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:25.384498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:25.384498+0000","last_clean_scrub_stamp":"2024-04-02T05:02:25.384498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:28.837013+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163075+0000","last_change":"2024-04-02T05:00:13.774214+0000","last_active":"2024-04-02T05:03:21.163075+0000","last_peered":"2024-04-02T05:03:21.163075+0000","last_clean":"2024-04-02T05:03:21.163075+0000","last_became_active":"2024-04-02T04:59:00.134110+0000","last_became_peered":"2024-04-02T04:59:00.134110+0000","last_unstale":"2024-04-02T05:03:21.163075+0000","last_undegraded":"2024-04-02T05:03:21.163075+0000","last_fullsized":"2024-04-02T05:03:21.163075+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:00:13.774005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:00:13.774005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"402'27","reported_seq":1644,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.302111+0000","last_change":"2024-04-02T05:03:13.050902+0000","last_active":"2024-04-02T05:03:21.302111+0000","last_peered":"2024-04-02T05:03:21.302111+0000","last_clean":"2024-04-02T05:03:21.302111+0000","last_became_active":"2024-04-02T04:54:56.011842+0000","last_became_peered":"2024-04-02T04:54:56.011842+0000","last_unstale":"2024-04-02T05:03:21.302111+0000","last_undegraded":"2024-04-02T05:03:21.302111+0000","last_fullsized":"2024-04-02T05:03:21.302111+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'27","last_scrub_stamp":"2024-04-02T05:03:13.050537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:03:13.050537+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:19.404642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1145,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:23.241683+0000","last_change":"2024-04-02T05:03:23.241707+0000","last_active":"2024-04-02T05:03:23.241683+0000","last_peered":"2024-04-02T05:03:23.241683+0000","last_clean":"2024-04-02T05:03:23.241683+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T05:03:23.241683+0000","last_undegraded":"2024-04-02T05:03:23.241683+0000","last_fullsized":"2024-04-02T05:03:23.241683+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:23.241589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:03:23.241589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:50.442513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1649,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158460+0000","last_change":"2024-04-02T05:03:10.509100+0000","last_active":"2024-04-02T05:03:21.158460+0000","last_peered":"2024-04-02T05:03:21.158460+0000","last_clean":"2024-04-02T05:03:21.158460+0000","last_became_active":"2024-04-02T05:03:07.556485+0000","last_became_peered":"2024-04-02T05:03:07.556485+0000","last_unstale":"2024-04-02T05:03:21.158460+0000","last_undegraded":"2024-04-02T05:03:21.158460+0000","last_fullsized":"2024-04-02T05:03:21.158460+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:10.508990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:03:10.508990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:24.756850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":901,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:22.278413+0000","last_change":"2024-04-02T05:03:22.278442+0000","last_active":"2024-04-02T05:03:22.278413+0000","last_peered":"2024-04-02T05:03:22.278413+0000","last_clean":"2024-04-02T05:03:22.278413+0000","last_became_active":"2024-04-02T04:59:55.065891+0000","last_became_peered":"2024-04-02T04:59:55.065891+0000","last_unstale":"2024-04-02T05:03:22.278413+0000","last_undegraded":"2024-04-02T05:03:22.278413+0000","last_fullsized":"2024-04-02T05:03:22.278413+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:03:22.278320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:03:22.278320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:04:45.560503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1511,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.253612+0000","last_change":"2024-04-02T05:02:23.842820+0000","last_active":"2024-04-02T05:03:21.253612+0000","last_peered":"2024-04-02T05:03:21.253612+0000","last_clean":"2024-04-02T05:03:21.253612+0000","last_became_active":"2024-04-02T04:59:20.052388+0000","last_became_peered":"2024-04-02T04:59:20.052388+0000","last_unstale":"2024-04-02T05:03:21.253612+0000","last_undegraded":"2024-04-02T05:03:21.253612+0000","last_fullsized":"2024-04-02T05:03:21.253612+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:23.842690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:23.842690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:28.555731+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"402'25","reported_seq":1756,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.157083+0000","last_change":"2024-04-02T05:02:00.195942+0000","last_active":"2024-04-02T05:03:21.157083+0000","last_peered":"2024-04-02T05:03:21.157083+0000","last_clean":"2024-04-02T05:03:21.157083+0000","last_became_active":"2024-04-02T04:59:20.078754+0000","last_became_peered":"2024-04-02T04:59:20.078754+0000","last_unstale":"2024-04-02T05:03:21.157083+0000","last_undegraded":"2024-04-02T05:03:21.157083+0000","last_fullsized":"2024-04-02T05:03:21.157083+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'25","last_scrub_stamp":"2024-04-02T05:02:00.195593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:02:00.195593+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:27.063800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":493,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.164147+0000","last_change":"2024-04-02T05:03:07.550825+0000","last_active":"2024-04-02T05:03:21.164147+0000","last_peered":"2024-04-02T05:03:21.164147+0000","last_clean":"2024-04-02T05:03:21.164147+0000","last_became_active":"2024-04-02T05:03:07.550195+0000","last_became_peered":"2024-04-02T05:03:07.550195+0000","last_unstale":"2024-04-02T05:03:21.164147+0000","last_undegraded":"2024-04-02T05:03:21.164147+0000","last_fullsized":"2024-04-02T05:03:21.164147+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:06.668287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_clean_scrub_stamp":"2024-04-02T05:01:06.668287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1657,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301220+0000","last_change":"2024-04-02T05:02:17.353404+0000","last_active":"2024-04-02T05:03:21.301220+0000","last_peered":"2024-04-02T05:03:21.301220+0000","last_clean":"2024-04-02T05:03:21.301220+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T05:03:21.301220+0000","last_undegraded":"2024-04-02T05:03:21.301220+0000","last_fullsized":"2024-04-02T05:03:21.301220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:17.353289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T05:02:17.353289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:47.268443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1658,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.301690+0000","last_change":"2024-04-02T05:02:27.200240+0000","last_active":"2024-04-02T05:03:21.301690+0000","last_peered":"2024-04-02T05:03:21.301690+0000","last_clean":"2024-04-02T05:03:21.301690+0000","last_became_active":"2024-04-02T04:54:55.984764+0000","last_became_peered":"2024-04-02T04:54:55.984764+0000","last_unstale":"2024-04-02T05:03:21.301690+0000","last_undegraded":"2024-04-02T05:03:21.301690+0000","last_fullsized":"2024-04-02T05:03:21.301690+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:27.200130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_clean_scrub_stamp":"2024-04-02T05:02:27.200130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:42.813303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1533,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.303107+0000","last_change":"2024-04-02T05:02:15.392207+0000","last_active":"2024-04-02T05:03:21.303107+0000","last_peered":"2024-04-02T05:03:21.303107+0000","last_clean":"2024-04-02T05:03:21.303107+0000","last_became_active":"2024-04-02T04:59:00.135538+0000","last_became_peered":"2024-04-02T04:59:00.135538+0000","last_unstale":"2024-04-02T05:03:21.303107+0000","last_undegraded":"2024-04-02T05:03:21.303107+0000","last_fullsized":"2024-04-02T05:03:21.303107+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:15.392091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:02:15.392091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"402'32","reported_seq":529,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.163795+0000","last_change":"2024-04-02T05:03:07.580733+0000","last_active":"2024-04-02T05:03:21.163795+0000","last_peered":"2024-04-02T05:03:21.163795+0000","last_clean":"2024-04-02T05:03:21.163795+0000","last_became_active":"2024-04-02T05:03:07.556658+0000","last_became_peered":"2024-04-02T05:03:07.556658+0000","last_unstale":"2024-04-02T05:03:21.163795+0000","last_undegraded":"2024-04-02T05:03:21.163795+0000","last_fullsized":"2024-04-02T05:03:21.163795+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'32","last_scrub_stamp":"2024-04-02T05:01:56.836914+0000","last_deep_scrub":"402'32","last_deep_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_clean_scrub_stamp":"2024-04-02T05:01:56.836914+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1305,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.299313+0000","last_change":"2024-04-02T05:02:42.884355+0000","last_active":"2024-04-02T05:03:21.299313+0000","last_peered":"2024-04-02T05:03:21.299313+0000","last_clean":"2024-04-02T05:03:21.299313+0000","last_became_active":"2024-04-02T05:02:42.883303+0000","last_became_peered":"2024-04-02T05:02:42.883303+0000","last_unstale":"2024-04-02T05:03:21.299313+0000","last_undegraded":"2024-04-02T05:03:21.299313+0000","last_fullsized":"2024-04-02T05:03:21.299313+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:02:15.228475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:02:15.228475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:03:22.669098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":492,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.162034+0000","last_change":"2024-04-02T05:03:07.558167+0000","last_active":"2024-04-02T05:03:21.162034+0000","last_peered":"2024-04-02T05:03:21.162034+0000","last_clean":"2024-04-02T05:03:21.162034+0000","last_became_active":"2024-04-02T05:03:07.557827+0000","last_became_peered":"2024-04-02T05:03:07.557827+0000","last_unstale":"2024-04-02T05:03:21.162034+0000","last_undegraded":"2024-04-02T05:03:21.162034+0000","last_fullsized":"2024-04-02T05:03:21.162034+0000","mapping_epoch":860,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:14.663348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:01:14.663348+0000","last_clean_scrub_stamp":"2024-04-02T05:01:14.663348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":486,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-02T05:03:21.158709+0000","last_change":"2024-04-02T05:01:19.718065+0000","last_active":"2024-04-02T05:03:21.158709+0000","last_peered":"2024-04-02T05:03:21.158709+0000","last_clean":"2024-04-02T05:03:21.158709+0000","last_became_active":"2024-04-02T04:59:00.122322+0000","last_became_peered":"2024-04-02T04:59:00.122322+0000","last_unstale":"2024-04-02T05:03:21.158709+0000","last_undegraded":"2024-04-02T05:03:21.158709+0000","last_fullsized":"2024-04-02T05:03:21.158709+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:01:19.717856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:14.337178+0000","last_clean_scrub_stamp":"2024-04-02T05:01:19.717856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":341,"num_pg":32,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":48,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":7},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":4000289,"num_objects":42,"num_object_clones":1,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":24,"num_read_kb":7,"num_write":62,"num_write_kb":3964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6500352,"data_stored":6500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":24,"acting":24,"num_store_stats":7},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1398,"num_read_kb":1123,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":4874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":968,"ondisk_log_size":968,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":274,"num_read_kb":269,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114740,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44812,"kb_used_data":16644,"kb_used_omap":57,"kb_used_meta":28102,"kb_avail":93700340,"statfs":{"total":95995035648,"available":95949148160,"internally_reserved":0,"allocated":17043456,"data_stored":11840251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":28777358},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103424,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126676,"kb_used_data":14988,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93618476,"statfs":{"total":95995035648,"available":95865319424,"internally_reserved":0,"allocated":15347712,"data_stored":10143165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122224,"kb_used_data":14696,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93622928,"statfs":{"total":95995035648,"available":95869878272,"internally_reserved":0,"allocated":15048704,"data_stored":9837523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738685,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124500,"kb_used_data":16972,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93620652,"statfs":{"total":95995035648,"available":95867547648,"internally_reserved":0,"allocated":17379328,"data_stored":12150838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378921,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124316,"kb_used_data":16788,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93620836,"statfs":{"total":95995035648,"available":95867736064,"internally_reserved":0,"allocated":17190912,"data_stored":12039954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051871,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122796,"kb_used_data":15268,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93622356,"statfs":{"total":95995035648,"available":95869292544,"internally_reserved":0,"allocated":15634432,"data_stored":10442956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822933,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104388,"kb_used_data":14908,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93640764,"statfs":{"total":95995035648,"available":95888142336,"internally_reserved":0,"allocated":15265792,"data_stored":10069898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463177,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2072576,"data_stored":2072576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":2027520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2088960,"data_stored":2088960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":341,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T05:03:33.402 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-02T05:03:33.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:33.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:33.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:33.679 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.364996+0000 osd.1 (osd.1) 441 : cluster 0 3.4 scrub starts 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.370056+0000 mgr.y (mgr.14538) 765 : cluster 0 pgmap v1340: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.370564+0000 osd.1 (osd.1) 442 : cluster 0 3.4 scrub ok 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: audit 2024-04-02T05:03:32.563428+0000 mon.a (mon.0) 3993 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]': finished 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.571472+0000 mon.a (mon.0) 3994 : cluster 0 osdmap e878: 8 total, 8 up, 7 in 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: audit 2024-04-02T05:03:32.627624+0000 mon.a (mon.0) 3995 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: audit 2024-04-02T05:03:32.628457+0000 mon.a (mon.0) 3996 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.633163+0000 osd.0 (osd.0) 425 : cluster 0 1.1 scrub starts 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.634892+0000 osd.0 (osd.0) 426 : cluster 0 1.1 scrub ok 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.860845+0000 osd.6 (osd.6) 371 : cluster 0 10.b scrub starts 2024-04-02T05:03:33.680 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[23790]: cluster 2024-04-02T05:03:32.866178+0000 osd.6 (osd.6) 372 : cluster 0 10.b scrub ok 2024-04-02T05:03:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:33.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:33.823 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:33.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:03:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:03:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:33.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.364996+0000 osd.1 (osd.1) 441 : cluster 0 3.4 scrub starts 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.370056+0000 mgr.y (mgr.14538) 765 : cluster 0 pgmap v1340: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.370564+0000 osd.1 (osd.1) 442 : cluster 0 3.4 scrub ok 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: audit 2024-04-02T05:03:32.563428+0000 mon.a (mon.0) 3993 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]': finished 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.571472+0000 mon.a (mon.0) 3994 : cluster 0 osdmap e878: 8 total, 8 up, 7 in 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: audit 2024-04-02T05:03:32.627624+0000 mon.a (mon.0) 3995 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: audit 2024-04-02T05:03:32.628457+0000 mon.a (mon.0) 3996 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.633163+0000 osd.0 (osd.0) 425 : cluster 0 1.1 scrub starts 2024-04-02T05:03:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.634892+0000 osd.0 (osd.0) 426 : cluster 0 1.1 scrub ok 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.860845+0000 osd.6 (osd.6) 371 : cluster 0 10.b scrub starts 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[21528]: cluster 2024-04-02T05:03:32.866178+0000 osd.6 (osd.6) 372 : cluster 0 10.b scrub ok 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.364996+0000 osd.1 (osd.1) 441 : cluster 0 3.4 scrub starts 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.370056+0000 mgr.y (mgr.14538) 765 : cluster 0 pgmap v1340: 224 pgs: 224 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.370564+0000 osd.1 (osd.1) 442 : cluster 0 3.4 scrub ok 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: audit 2024-04-02T05:03:32.563428+0000 mon.a (mon.0) 3993 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]': finished 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.571472+0000 mon.a (mon.0) 3994 : cluster 0 osdmap e878: 8 total, 8 up, 7 in 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: audit 2024-04-02T05:03:32.627624+0000 mon.a (mon.0) 3995 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: audit 2024-04-02T05:03:32.628457+0000 mon.a (mon.0) 3996 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-146"}]: dispatch 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.633163+0000 osd.0 (osd.0) 425 : cluster 0 1.1 scrub starts 2024-04-02T05:03:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.634892+0000 osd.0 (osd.0) 426 : cluster 0 1.1 scrub ok 2024-04-02T05:03:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.860845+0000 osd.6 (osd.6) 371 : cluster 0 10.b scrub starts 2024-04-02T05:03:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:33 smithi069 bash[28904]: cluster 2024-04-02T05:03:32.866178+0000 osd.6 (osd.6) 372 : cluster 0 10.b scrub ok 2024-04-02T05:03:34.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:34.076 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:34.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:34.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:34.334 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:34.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:34.464 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:34.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:34.594 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:34.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:34.720 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:34 smithi098 bash[23790]: cluster 2024-04-02T05:03:33.584344+0000 mon.a (mon.0) 3997 : cluster 0 osdmap e879: 8 total, 8 up, 7 in 2024-04-02T05:03:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:34 smithi098 bash[23790]: audit 2024-04-02T05:03:33.774679+0000 mgr.y (mgr.14538) 766 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:34 smithi098 bash[23790]: audit 2024-04-02T05:03:34.209777+0000 mon.a (mon.0) 3998 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:34.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:34 smithi098 bash[23790]: audit 2024-04-02T05:03:34.213185+0000 mon.b (mon.1) 323 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:34.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:34.876 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2024-04-02T05:03:34.876 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[21528]: cluster 2024-04-02T05:03:33.584344+0000 mon.a (mon.0) 3997 : cluster 0 osdmap e879: 8 total, 8 up, 7 in 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[21528]: audit 2024-04-02T05:03:33.774679+0000 mgr.y (mgr.14538) 766 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[21528]: audit 2024-04-02T05:03:34.209777+0000 mon.a (mon.0) 3998 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[21528]: audit 2024-04-02T05:03:34.213185+0000 mon.b (mon.1) 323 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[28904]: cluster 2024-04-02T05:03:33.584344+0000 mon.a (mon.0) 3997 : cluster 0 osdmap e879: 8 total, 8 up, 7 in 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[28904]: audit 2024-04-02T05:03:33.774679+0000 mgr.y (mgr.14538) 766 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[28904]: audit 2024-04-02T05:03:34.209777+0000 mon.a (mon.0) 3998 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:34.888 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:34 smithi069 bash[28904]: audit 2024-04-02T05:03:34.213185+0000 mon.b (mon.1) 323 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:34.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:34.972 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:34.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:35.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:35.101 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:35.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:35.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:35.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-04-02T05:03:35.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:35.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:35.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:35.616 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:35.718 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:35.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.353709+0000 osd.7 (osd.7) 247 : cluster 0 3.13 scrub starts 2024-04-02T05:03:35.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.357644+0000 osd.7 (osd.7) 248 : cluster 0 3.13 scrub ok 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.370913+0000 mgr.y (mgr.14538) 767 : cluster 0 pgmap v1343: 192 pgs: 192 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.438601+0000 osd.1 (osd.1) 443 : cluster 0 1.b scrub starts 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.441416+0000 osd.1 (osd.1) 444 : cluster 0 1.b scrub ok 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.599254+0000 mon.a (mon.0) 3999 : cluster 0 osdmap e880: 8 total, 8 up, 7 in 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: audit 2024-04-02T05:03:34.602207+0000 mon.a (mon.0) 4000 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.852522+0000 osd.6 (osd.6) 373 : cluster 0 10.f scrub starts 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.856778+0000 osd.6 (osd.6) 374 : cluster 0 10.f scrub ok 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: audit 2024-04-02T05:03:35.126329+0000 mon.a (mon.0) 4001 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.136402+0000 mon.a (mon.0) 4002 : cluster 0 osdmap e881: 8 total, 8 up, 7 in 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.207439+0000 mon.a (mon.0) 4003 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[21528]: audit 2024-04-02T05:03:35.235468+0000 mon.a (mon.0) 4004 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:03:35.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.353709+0000 osd.7 (osd.7) 247 : cluster 0 3.13 scrub starts 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.357644+0000 osd.7 (osd.7) 248 : cluster 0 3.13 scrub ok 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.370913+0000 mgr.y (mgr.14538) 767 : cluster 0 pgmap v1343: 192 pgs: 192 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.438601+0000 osd.1 (osd.1) 443 : cluster 0 1.b scrub starts 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.441416+0000 osd.1 (osd.1) 444 : cluster 0 1.b scrub ok 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.599254+0000 mon.a (mon.0) 3999 : cluster 0 osdmap e880: 8 total, 8 up, 7 in 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: audit 2024-04-02T05:03:34.602207+0000 mon.a (mon.0) 4000 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.852522+0000 osd.6 (osd.6) 373 : cluster 0 10.f scrub starts 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.856778+0000 osd.6 (osd.6) 374 : cluster 0 10.f scrub ok 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: audit 2024-04-02T05:03:35.126329+0000 mon.a (mon.0) 4001 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.136402+0000 mon.a (mon.0) 4002 : cluster 0 osdmap e881: 8 total, 8 up, 7 in 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.207439+0000 mon.a (mon.0) 4003 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-02T05:03:35.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:35 smithi069 bash[28904]: audit 2024-04-02T05:03:35.235468+0000 mon.a (mon.0) 4004 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:03:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:35.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:35.872 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:35.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:35.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-02T05:03:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.353709+0000 osd.7 (osd.7) 247 : cluster 0 3.13 scrub starts 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.357644+0000 osd.7 (osd.7) 248 : cluster 0 3.13 scrub ok 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.370913+0000 mgr.y (mgr.14538) 767 : cluster 0 pgmap v1343: 192 pgs: 192 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.438601+0000 osd.1 (osd.1) 443 : cluster 0 1.b scrub starts 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.441416+0000 osd.1 (osd.1) 444 : cluster 0 1.b scrub ok 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.599254+0000 mon.a (mon.0) 3999 : cluster 0 osdmap e880: 8 total, 8 up, 7 in 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: audit 2024-04-02T05:03:34.602207+0000 mon.a (mon.0) 4000 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.852522+0000 osd.6 (osd.6) 373 : cluster 0 10.f scrub starts 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.856778+0000 osd.6 (osd.6) 374 : cluster 0 10.f scrub ok 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: audit 2024-04-02T05:03:35.126329+0000 mon.a (mon.0) 4001 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.136402+0000 mon.a (mon.0) 4002 : cluster 0 osdmap e881: 8 total, 8 up, 7 in 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.207439+0000 mon.a (mon.0) 4003 : cluster 1 Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-02T05:03:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:35 smithi098 bash[23790]: audit 2024-04-02T05:03:35.235468+0000 mon.a (mon.0) 4004 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-02T05:03:35.976 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:36.008 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:36.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:36.129 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:36.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:36.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:36.386 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:36.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:36.517 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:36.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:36.632 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.966294+0000 osd.4 (osd.4) 428 : cluster 0 5.1 scrub starts 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:34.968560+0000 osd.4 (osd.4) 429 : cluster 0 5.1 scrub ok 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.546024+0000 osd.6 (osd.6) 375 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.546513+0000 osd.6 (osd.6) 376 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.861005+0000 osd.6 (osd.6) 377 : cluster 0 1.6 scrub starts 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.863540+0000 osd.6 (osd.6) 378 : cluster 0 1.6 scrub ok 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: audit 2024-04-02T05:03:36.141488+0000 mon.a (mon.0) 4005 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.145542+0000 mon.a (mon.0) 4006 : cluster 0 osdmap e882: 8 total, 8 up, 7 in 2024-04-02T05:03:36.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:36 smithi098 bash[23790]: audit 2024-04-02T05:03:36.155238+0000 mon.a (mon.0) 4007 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]: dispatch 2024-04-02T05:03:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:36.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:36.780 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:36.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:36.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:37.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.966294+0000 osd.4 (osd.4) 428 : cluster 0 5.1 scrub starts 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:34.968560+0000 osd.4 (osd.4) 429 : cluster 0 5.1 scrub ok 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.546024+0000 osd.6 (osd.6) 375 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.546513+0000 osd.6 (osd.6) 376 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.861005+0000 osd.6 (osd.6) 377 : cluster 0 1.6 scrub starts 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.863540+0000 osd.6 (osd.6) 378 : cluster 0 1.6 scrub ok 2024-04-02T05:03:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: audit 2024-04-02T05:03:36.141488+0000 mon.a (mon.0) 4005 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.145542+0000 mon.a (mon.0) 4006 : cluster 0 osdmap e882: 8 total, 8 up, 7 in 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[21528]: audit 2024-04-02T05:03:36.155238+0000 mon.a (mon.0) 4007 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]: dispatch 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.966294+0000 osd.4 (osd.4) 428 : cluster 0 5.1 scrub starts 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:34.968560+0000 osd.4 (osd.4) 429 : cluster 0 5.1 scrub ok 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.546024+0000 osd.6 (osd.6) 375 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.546513+0000 osd.6 (osd.6) 376 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.861005+0000 osd.6 (osd.6) 377 : cluster 0 1.6 scrub starts 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.863540+0000 osd.6 (osd.6) 378 : cluster 0 1.6 scrub ok 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: audit 2024-04-02T05:03:36.141488+0000 mon.a (mon.0) 4005 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.145542+0000 mon.a (mon.0) 4006 : cluster 0 osdmap e882: 8 total, 8 up, 7 in 2024-04-02T05:03:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:36 smithi069 bash[28904]: audit 2024-04-02T05:03:36.155238+0000 mon.a (mon.0) 4007 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]: dispatch 2024-04-02T05:03:37.041 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:37.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:37.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:37.300 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:37.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:37.429 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:37.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:37.558 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:37.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:37.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.973123+0000 osd.4 (osd.4) 430 : cluster 0 3.3 scrub starts 2024-04-02T05:03:37.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:35.978383+0000 osd.4 (osd.4) 431 : cluster 0 3.3 scrub ok 2024-04-02T05:03:37.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.369048+0000 osd.7 (osd.7) 249 : cluster 0 6.8 scrub starts 2024-04-02T05:03:37.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.371790+0000 osd.7 (osd.7) 250 : cluster 0 6.8 scrub ok 2024-04-02T05:03:37.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.371850+0000 mgr.y (mgr.14538) 768 : cluster 0 pgmap v1347: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 269 B/s rd, 0 op/s 2024-04-02T05:03:37.672 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.383961+0000 osd.1 (osd.1) 445 : cluster 0 10.2 scrub starts 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.385484+0000 osd.1 (osd.1) 446 : cluster 0 10.2 scrub ok 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.461219+0000 osd.3 (osd.3) 443 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.461493+0000 osd.3 (osd.3) 444 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.526728+0000 osd.0 (osd.0) 427 : cluster 0 4.b deep-scrub starts 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.528980+0000 osd.0 (osd.0) 428 : cluster 0 4.b deep-scrub ok 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.870853+0000 osd.6 (osd.6) 379 : cluster 0 2.1a scrub starts 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:36.873398+0000 osd.6 (osd.6) 380 : cluster 0 2.1a scrub ok 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: audit 2024-04-02T05:03:37.144456+0000 mon.a (mon.0) 4008 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]': finished 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.156111+0000 mon.a (mon.0) 4009 : cluster 0 osdmap e883: 8 total, 8 up, 7 in 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: audit 2024-04-02T05:03:37.211516+0000 mon.a (mon.0) 4010 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:37.673 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:37 smithi098 bash[23790]: audit 2024-04-02T05:03:37.212491+0000 mon.a (mon.0) 4011 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]: dispatch 2024-04-02T05:03:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:37.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:37.815 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:37.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:37.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.973123+0000 osd.4 (osd.4) 430 : cluster 0 3.3 scrub starts 2024-04-02T05:03:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:35.978383+0000 osd.4 (osd.4) 431 : cluster 0 3.3 scrub ok 2024-04-02T05:03:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.369048+0000 osd.7 (osd.7) 249 : cluster 0 6.8 scrub starts 2024-04-02T05:03:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.371790+0000 osd.7 (osd.7) 250 : cluster 0 6.8 scrub ok 2024-04-02T05:03:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.371850+0000 mgr.y (mgr.14538) 768 : cluster 0 pgmap v1347: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 269 B/s rd, 0 op/s 2024-04-02T05:03:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.383961+0000 osd.1 (osd.1) 445 : cluster 0 10.2 scrub starts 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.385484+0000 osd.1 (osd.1) 446 : cluster 0 10.2 scrub ok 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.461219+0000 osd.3 (osd.3) 443 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.461493+0000 osd.3 (osd.3) 444 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.526728+0000 osd.0 (osd.0) 427 : cluster 0 4.b deep-scrub starts 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.528980+0000 osd.0 (osd.0) 428 : cluster 0 4.b deep-scrub ok 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.870853+0000 osd.6 (osd.6) 379 : cluster 0 2.1a scrub starts 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:36.873398+0000 osd.6 (osd.6) 380 : cluster 0 2.1a scrub ok 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: audit 2024-04-02T05:03:37.144456+0000 mon.a (mon.0) 4008 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]': finished 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.156111+0000 mon.a (mon.0) 4009 : cluster 0 osdmap e883: 8 total, 8 up, 7 in 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: audit 2024-04-02T05:03:37.211516+0000 mon.a (mon.0) 4010 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[21528]: audit 2024-04-02T05:03:37.212491+0000 mon.a (mon.0) 4011 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]: dispatch 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.973123+0000 osd.4 (osd.4) 430 : cluster 0 3.3 scrub starts 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:35.978383+0000 osd.4 (osd.4) 431 : cluster 0 3.3 scrub ok 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.369048+0000 osd.7 (osd.7) 249 : cluster 0 6.8 scrub starts 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.371790+0000 osd.7 (osd.7) 250 : cluster 0 6.8 scrub ok 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.371850+0000 mgr.y (mgr.14538) 768 : cluster 0 pgmap v1347: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 752 MiB used, 625 GiB / 626 GiB avail; 269 B/s rd, 0 op/s 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.383961+0000 osd.1 (osd.1) 445 : cluster 0 10.2 scrub starts 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.385484+0000 osd.1 (osd.1) 446 : cluster 0 10.2 scrub ok 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.461219+0000 osd.3 (osd.3) 443 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.461493+0000 osd.3 (osd.3) 444 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.526728+0000 osd.0 (osd.0) 427 : cluster 0 4.b deep-scrub starts 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.528980+0000 osd.0 (osd.0) 428 : cluster 0 4.b deep-scrub ok 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.870853+0000 osd.6 (osd.6) 379 : cluster 0 2.1a scrub starts 2024-04-02T05:03:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:36.873398+0000 osd.6 (osd.6) 380 : cluster 0 2.1a scrub ok 2024-04-02T05:03:38.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: audit 2024-04-02T05:03:37.144456+0000 mon.a (mon.0) 4008 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]': finished 2024-04-02T05:03:38.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.156111+0000 mon.a (mon.0) 4009 : cluster 0 osdmap e883: 8 total, 8 up, 7 in 2024-04-02T05:03:38.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: audit 2024-04-02T05:03:37.211516+0000 mon.a (mon.0) 4010 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:38.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:37 smithi069 bash[28904]: audit 2024-04-02T05:03:37.212491+0000 mon.a (mon.0) 4011 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-148"}]: dispatch 2024-04-02T05:03:38.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:38.077 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:38.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:38.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:38.319 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-02T05:03:38.319 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-02T05:03:38.319 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-02T05:03:38.319 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-02T05:03:38.319 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-02T05:03:38.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-04-02T05:03:38.336 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:38.426 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:38.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:38.568 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:38.586 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:38.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:38.711 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:38 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.362025+0000 osd.3 (osd.3) 445 : cluster 0 10.15 scrub starts 2024-04-02T05:03:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:38 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.364247+0000 osd.3 (osd.3) 446 : cluster 0 10.15 scrub ok 2024-04-02T05:03:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:38 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.573414+0000 osd.0 (osd.0) 429 : cluster 0 4.2 scrub starts 2024-04-02T05:03:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:38 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.575543+0000 osd.0 (osd.0) 430 : cluster 0 4.2 scrub ok 2024-04-02T05:03:38.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:38 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.179975+0000 mon.a (mon.0) 4012 : cluster 0 osdmap e884: 8 total, 8 up, 7 in 2024-04-02T05:03:38.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:38.843 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.362025+0000 osd.3 (osd.3) 445 : cluster 0 10.15 scrub starts 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.364247+0000 osd.3 (osd.3) 446 : cluster 0 10.15 scrub ok 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.573414+0000 osd.0 (osd.0) 429 : cluster 0 4.2 scrub starts 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.575543+0000 osd.0 (osd.0) 430 : cluster 0 4.2 scrub ok 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.179975+0000 mon.a (mon.0) 4012 : cluster 0 osdmap e884: 8 total, 8 up, 7 in 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.362025+0000 osd.3 (osd.3) 445 : cluster 0 10.15 scrub starts 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.364247+0000 osd.3 (osd.3) 446 : cluster 0 10.15 scrub ok 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.573414+0000 osd.0 (osd.0) 429 : cluster 0 4.2 scrub starts 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.575543+0000 osd.0 (osd.0) 430 : cluster 0 4.2 scrub ok 2024-04-02T05:03:38.940 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:38 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.179975+0000 mon.a (mon.0) 4012 : cluster 0 osdmap e884: 8 total, 8 up, 7 in 2024-04-02T05:03:38.946 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:38.972 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:38.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:39.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:39.102 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:39.204 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:39.213 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:03:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:03:39.225 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:39.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:39.355 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:39.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:39.486 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:39.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:39.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:39.747 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:39.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.963505+0000 osd.4 (osd.4) 432 : cluster 0 10.1d scrub starts 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:37.967064+0000 osd.4 (osd.4) 433 : cluster 0 10.1d scrub ok 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.077152+0000 osd.4 (osd.4) 434 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.077445+0000 osd.4 (osd.4) 435 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.372478+0000 mgr.y (mgr.14538) 769 : cluster 0 pgmap v1350: 192 pgs: 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 542 B/s wr, 1 op/s 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.547731+0000 osd.0 (osd.0) 431 : cluster 0 4.6 scrub starts 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.553187+0000 osd.0 (osd.0) 432 : cluster 0 4.6 scrub ok 2024-04-02T05:03:39.862 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.872273+0000 osd.6 (osd.6) 381 : cluster 0 1.7 scrub starts 2024-04-02T05:03:39.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:38.878100+0000 osd.6 (osd.6) 382 : cluster 0 1.7 scrub ok 2024-04-02T05:03:39.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.184770+0000 mon.a (mon.0) 4013 : cluster 0 osdmap e885: 8 total, 8 up, 7 in 2024-04-02T05:03:39.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:39 smithi098 bash[23790]: audit 2024-04-02T05:03:39.194962+0000 mon.a (mon.0) 4014 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:39.878 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:39.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:39.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:39.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.963505+0000 osd.4 (osd.4) 432 : cluster 0 10.1d scrub starts 2024-04-02T05:03:39.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:37.967064+0000 osd.4 (osd.4) 433 : cluster 0 10.1d scrub ok 2024-04-02T05:03:39.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.077152+0000 osd.4 (osd.4) 434 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:39.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.077445+0000 osd.4 (osd.4) 435 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.372478+0000 mgr.y (mgr.14538) 769 : cluster 0 pgmap v1350: 192 pgs: 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 542 B/s wr, 1 op/s 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.547731+0000 osd.0 (osd.0) 431 : cluster 0 4.6 scrub starts 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.553187+0000 osd.0 (osd.0) 432 : cluster 0 4.6 scrub ok 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.872273+0000 osd.6 (osd.6) 381 : cluster 0 1.7 scrub starts 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:38.878100+0000 osd.6 (osd.6) 382 : cluster 0 1.7 scrub ok 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.184770+0000 mon.a (mon.0) 4013 : cluster 0 osdmap e885: 8 total, 8 up, 7 in 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[21528]: audit 2024-04-02T05:03:39.194962+0000 mon.a (mon.0) 4014 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.963505+0000 osd.4 (osd.4) 432 : cluster 0 10.1d scrub starts 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:37.967064+0000 osd.4 (osd.4) 433 : cluster 0 10.1d scrub ok 2024-04-02T05:03:39.983 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.077152+0000 osd.4 (osd.4) 434 : cluster 0 purged_snaps scrub starts 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.077445+0000 osd.4 (osd.4) 435 : cluster 0 purged_snaps scrub ok 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.372478+0000 mgr.y (mgr.14538) 769 : cluster 0 pgmap v1350: 192 pgs: 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 542 B/s wr, 1 op/s 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.547731+0000 osd.0 (osd.0) 431 : cluster 0 4.6 scrub starts 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.553187+0000 osd.0 (osd.0) 432 : cluster 0 4.6 scrub ok 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.872273+0000 osd.6 (osd.6) 381 : cluster 0 1.7 scrub starts 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:38.878100+0000 osd.6 (osd.6) 382 : cluster 0 1.7 scrub ok 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.184770+0000 mon.a (mon.0) 4013 : cluster 0 osdmap e885: 8 total, 8 up, 7 in 2024-04-02T05:03:39.984 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:39 smithi069 bash[28904]: audit 2024-04-02T05:03:39.194962+0000 mon.a (mon.0) 4014 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:40.005 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:40.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:40.131 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:40.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:40.257 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:40.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:40.472 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:40.473 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:40.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:40.512 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:40.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:40.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:40.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.348346+0000 osd.3 (osd.3) 447 : cluster 0 6.f scrub starts 2024-04-02T05:03:40.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.350306+0000 osd.3 (osd.3) 448 : cluster 0 6.f scrub ok 2024-04-02T05:03:40.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.465210+0000 osd.1 (osd.1) 447 : cluster 0 2.2 scrub starts 2024-04-02T05:03:40.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.467410+0000 osd.1 (osd.1) 448 : cluster 0 2.2 scrub ok 2024-04-02T05:03:40.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: audit 2024-04-02T05:03:40.194549+0000 mon.a (mon.0) 4015 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: cluster 2024-04-02T05:03:40.198934+0000 mon.a (mon.0) 4016 : cluster 0 osdmap e886: 8 total, 8 up, 7 in 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: audit 2024-04-02T05:03:40.264103+0000 mon.a (mon.0) 4017 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[21528]: audit 2024-04-02T05:03:40.264489+0000 mon.a (mon.0) 4018 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-150"}]: dispatch 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.348346+0000 osd.3 (osd.3) 447 : cluster 0 6.f scrub starts 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.350306+0000 osd.3 (osd.3) 448 : cluster 0 6.f scrub ok 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.465210+0000 osd.1 (osd.1) 447 : cluster 0 2.2 scrub starts 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.467410+0000 osd.1 (osd.1) 448 : cluster 0 2.2 scrub ok 2024-04-02T05:03:40.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: audit 2024-04-02T05:03:40.194549+0000 mon.a (mon.0) 4015 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:40.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: cluster 2024-04-02T05:03:40.198934+0000 mon.a (mon.0) 4016 : cluster 0 osdmap e886: 8 total, 8 up, 7 in 2024-04-02T05:03:40.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: audit 2024-04-02T05:03:40.264103+0000 mon.a (mon.0) 4017 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:40.764 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:40 smithi069 bash[28904]: audit 2024-04-02T05:03:40.264489+0000 mon.a (mon.0) 4018 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-150"}]: dispatch 2024-04-02T05:03:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:40.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:40.901 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:40.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.348346+0000 osd.3 (osd.3) 447 : cluster 0 6.f scrub starts 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.350306+0000 osd.3 (osd.3) 448 : cluster 0 6.f scrub ok 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.465210+0000 osd.1 (osd.1) 447 : cluster 0 2.2 scrub starts 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.467410+0000 osd.1 (osd.1) 448 : cluster 0 2.2 scrub ok 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: audit 2024-04-02T05:03:40.194549+0000 mon.a (mon.0) 4015 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: cluster 2024-04-02T05:03:40.198934+0000 mon.a (mon.0) 4016 : cluster 0 osdmap e886: 8 total, 8 up, 7 in 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: audit 2024-04-02T05:03:40.264103+0000 mon.a (mon.0) 4017 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:40 smithi098 bash[23790]: audit 2024-04-02T05:03:40.264489+0000 mon.a (mon.0) 4018 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-150"}]: dispatch 2024-04-02T05:03:41.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:41.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:41.133 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:41.160 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:41.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:41.290 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:41.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:41.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:41.552 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:41.655 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:41.680 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:41.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:41.808 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:41.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:41.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.987345+0000 osd.4 (osd.4) 436 : cluster 0 10.17 scrub starts 2024-04-02T05:03:41.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[21528]: cluster 2024-04-02T05:03:39.994007+0000 osd.4 (osd.4) 437 : cluster 0 10.17 scrub ok 2024-04-02T05:03:41.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[21528]: cluster 2024-04-02T05:03:40.319010+0000 osd.3 (osd.3) 449 : cluster 0 6.2 scrub starts 2024-04-02T05:03:41.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[21528]: cluster 2024-04-02T05:03:40.320574+0000 osd.3 (osd.3) 450 : cluster 0 6.2 scrub ok 2024-04-02T05:03:41.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[21528]: cluster 2024-04-02T05:03:40.373126+0000 mgr.y (mgr.14538) 770 : cluster 0 pgmap v1353: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.220980+0000 mon.a (mon.0) 4019 : cluster 0 osdmap e887: 8 total, 8 up, 7 in 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.987345+0000 osd.4 (osd.4) 436 : cluster 0 10.17 scrub starts 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[28904]: cluster 2024-04-02T05:03:39.994007+0000 osd.4 (osd.4) 437 : cluster 0 10.17 scrub ok 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[28904]: cluster 2024-04-02T05:03:40.319010+0000 osd.3 (osd.3) 449 : cluster 0 6.2 scrub starts 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[28904]: cluster 2024-04-02T05:03:40.320574+0000 osd.3 (osd.3) 450 : cluster 0 6.2 scrub ok 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[28904]: cluster 2024-04-02T05:03:40.373126+0000 mgr.y (mgr.14538) 770 : cluster 0 pgmap v1353: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-02T05:03:41.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:41 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.220980+0000 mon.a (mon.0) 4019 : cluster 0 osdmap e887: 8 total, 8 up, 7 in 2024-04-02T05:03:41.938 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:41.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:41.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:41 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.987345+0000 osd.4 (osd.4) 436 : cluster 0 10.17 scrub starts 2024-04-02T05:03:41.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:41 smithi098 bash[23790]: cluster 2024-04-02T05:03:39.994007+0000 osd.4 (osd.4) 437 : cluster 0 10.17 scrub ok 2024-04-02T05:03:41.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:41 smithi098 bash[23790]: cluster 2024-04-02T05:03:40.319010+0000 osd.3 (osd.3) 449 : cluster 0 6.2 scrub starts 2024-04-02T05:03:41.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:41 smithi098 bash[23790]: cluster 2024-04-02T05:03:40.320574+0000 osd.3 (osd.3) 450 : cluster 0 6.2 scrub ok 2024-04-02T05:03:41.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:41 smithi098 bash[23790]: cluster 2024-04-02T05:03:40.373126+0000 mgr.y (mgr.14538) 770 : cluster 0 pgmap v1353: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-02T05:03:41.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:41 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.220980+0000 mon.a (mon.0) 4019 : cluster 0 osdmap e887: 8 total, 8 up, 7 in 2024-04-02T05:03:42.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.064 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:42.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:42.188 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:42.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.315 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:42.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.444 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:42.544 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:03:42.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:42.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:42.702 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:42.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.830 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:42.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:42.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:40.978791+0000 osd.2 (osd.2) 376 : cluster 0 3.9 scrub starts 2024-04-02T05:03:42.945 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:40.981703+0000 osd.2 (osd.2) 377 : cluster 0 3.9 scrub ok 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.335623+0000 osd.3 (osd.3) 451 : cluster 0 6.13 scrub starts 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.337066+0000 osd.3 (osd.3) 452 : cluster 0 6.13 scrub ok 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.589842+0000 osd.0 (osd.0) 433 : cluster 0 3.1 scrub starts 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.594468+0000 osd.0 (osd.0) 434 : cluster 0 3.1 scrub ok 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.900061+0000 osd.6 (osd.6) 383 : cluster 0 1.f scrub starts 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.902362+0000 osd.6 (osd.6) 384 : cluster 0 1.f scrub ok 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: cluster 2024-04-02T05:03:42.214217+0000 mon.a (mon.0) 4020 : cluster 0 osdmap e888: 8 total, 8 up, 7 in 2024-04-02T05:03:42.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:42 smithi098 bash[23790]: audit 2024-04-02T05:03:42.217914+0000 mon.a (mon.0) 4021 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:42.962 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:42.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:43.065 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:43.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:40.978791+0000 osd.2 (osd.2) 376 : cluster 0 3.9 scrub starts 2024-04-02T05:03:43.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:40.981703+0000 osd.2 (osd.2) 377 : cluster 0 3.9 scrub ok 2024-04-02T05:03:43.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.335623+0000 osd.3 (osd.3) 451 : cluster 0 6.13 scrub starts 2024-04-02T05:03:43.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.337066+0000 osd.3 (osd.3) 452 : cluster 0 6.13 scrub ok 2024-04-02T05:03:43.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.589842+0000 osd.0 (osd.0) 433 : cluster 0 3.1 scrub starts 2024-04-02T05:03:43.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.594468+0000 osd.0 (osd.0) 434 : cluster 0 3.1 scrub ok 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.900061+0000 osd.6 (osd.6) 383 : cluster 0 1.f scrub starts 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.902362+0000 osd.6 (osd.6) 384 : cluster 0 1.f scrub ok 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: cluster 2024-04-02T05:03:42.214217+0000 mon.a (mon.0) 4020 : cluster 0 osdmap e888: 8 total, 8 up, 7 in 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[28904]: audit 2024-04-02T05:03:42.217914+0000 mon.a (mon.0) 4021 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:40.978791+0000 osd.2 (osd.2) 376 : cluster 0 3.9 scrub starts 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:40.981703+0000 osd.2 (osd.2) 377 : cluster 0 3.9 scrub ok 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.335623+0000 osd.3 (osd.3) 451 : cluster 0 6.13 scrub starts 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.337066+0000 osd.3 (osd.3) 452 : cluster 0 6.13 scrub ok 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.589842+0000 osd.0 (osd.0) 433 : cluster 0 3.1 scrub starts 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.594468+0000 osd.0 (osd.0) 434 : cluster 0 3.1 scrub ok 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.900061+0000 osd.6 (osd.6) 383 : cluster 0 1.f scrub starts 2024-04-02T05:03:43.077 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.902362+0000 osd.6 (osd.6) 384 : cluster 0 1.f scrub ok 2024-04-02T05:03:43.078 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: cluster 2024-04-02T05:03:42.214217+0000 mon.a (mon.0) 4020 : cluster 0 osdmap e888: 8 total, 8 up, 7 in 2024-04-02T05:03:43.078 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:42 smithi069 bash[21528]: audit 2024-04-02T05:03:42.217914+0000 mon.a (mon.0) 4021 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:43.092 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:43.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:43.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:43.353 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:43.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:43.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:43.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:43.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.974900+0000 osd.4 (osd.4) 438 : cluster 0 4.e scrub starts 2024-04-02T05:03:43.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:41.977657+0000 osd.4 (osd.4) 439 : cluster 0 4.e scrub ok 2024-04-02T05:03:43.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:42.326409+0000 osd.3 (osd.3) 453 : cluster 0 3.17 scrub starts 2024-04-02T05:03:43.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:42.328030+0000 osd.3 (osd.3) 454 : cluster 0 3.17 scrub ok 2024-04-02T05:03:43.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:42.373640+0000 mgr.y (mgr.14538) 771 : cluster 0 pgmap v1356: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:43.212185+0000 mon.a (mon.0) 4022 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: audit 2024-04-02T05:03:43.213967+0000 mon.a (mon.0) 4023 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:43.225744+0000 mon.a (mon.0) 4024 : cluster 0 osdmap e889: 8 total, 8 up, 7 in 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: audit 2024-04-02T05:03:43.315919+0000 mon.a (mon.0) 4025 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: audit 2024-04-02T05:03:43.316513+0000 mon.a (mon.0) 4026 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-152"}]: dispatch 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:43.349583+0000 osd.1 (osd.1) 449 : cluster 0 2.d scrub starts 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[21528]: cluster 2024-04-02T05:03:43.350926+0000 osd.1 (osd.1) 450 : cluster 0 2.d scrub ok 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.974900+0000 osd.4 (osd.4) 438 : cluster 0 4.e scrub starts 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:41.977657+0000 osd.4 (osd.4) 439 : cluster 0 4.e scrub ok 2024-04-02T05:03:43.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:42.326409+0000 osd.3 (osd.3) 453 : cluster 0 3.17 scrub starts 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:42.328030+0000 osd.3 (osd.3) 454 : cluster 0 3.17 scrub ok 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:42.373640+0000 mgr.y (mgr.14538) 771 : cluster 0 pgmap v1356: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:43.212185+0000 mon.a (mon.0) 4022 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: audit 2024-04-02T05:03:43.213967+0000 mon.a (mon.0) 4023 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:43.225744+0000 mon.a (mon.0) 4024 : cluster 0 osdmap e889: 8 total, 8 up, 7 in 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: audit 2024-04-02T05:03:43.315919+0000 mon.a (mon.0) 4025 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: audit 2024-04-02T05:03:43.316513+0000 mon.a (mon.0) 4026 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-152"}]: dispatch 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:43.349583+0000 osd.1 (osd.1) 449 : cluster 0 2.d scrub starts 2024-04-02T05:03:43.730 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:43 smithi069 bash[28904]: cluster 2024-04-02T05:03:43.350926+0000 osd.1 (osd.1) 450 : cluster 0 2.d scrub ok 2024-04-02T05:03:43.745 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:43.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:43.874 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:43.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:43.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:03:43.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.974900+0000 osd.4 (osd.4) 438 : cluster 0 4.e scrub starts 2024-04-02T05:03:43.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:41.977657+0000 osd.4 (osd.4) 439 : cluster 0 4.e scrub ok 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:42.326409+0000 osd.3 (osd.3) 453 : cluster 0 3.17 scrub starts 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:42.328030+0000 osd.3 (osd.3) 454 : cluster 0 3.17 scrub ok 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:42.373640+0000 mgr.y (mgr.14538) 771 : cluster 0 pgmap v1356: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 753 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:43.212185+0000 mon.a (mon.0) 4022 : cluster 3 Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: audit 2024-04-02T05:03:43.213967+0000 mon.a (mon.0) 4023 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:43.225744+0000 mon.a (mon.0) 4024 : cluster 0 osdmap e889: 8 total, 8 up, 7 in 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: audit 2024-04-02T05:03:43.315919+0000 mon.a (mon.0) 4025 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: audit 2024-04-02T05:03:43.316513+0000 mon.a (mon.0) 4026 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-152"}]: dispatch 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:43.349583+0000 osd.1 (osd.1) 449 : cluster 0 2.d scrub starts 2024-04-02T05:03:43.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:43 smithi098 bash[23790]: cluster 2024-04-02T05:03:43.350926+0000 osd.1 (osd.1) 450 : cluster 0 2.d scrub ok 2024-04-02T05:03:43.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:44.002 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:44.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:44.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:44.260 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:44.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:44.389 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:44.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:44.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:44.650 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:44.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:44.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:44 smithi098 bash[23790]: audit 2024-04-02T05:03:43.777794+0000 mgr.y (mgr.14538) 772 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:44.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:44 smithi098 bash[23790]: cluster 2024-04-02T05:03:44.243390+0000 mon.a (mon.0) 4027 : cluster 0 osdmap e890: 8 total, 8 up, 7 in 2024-04-02T05:03:44.781 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:44.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:44.909 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:44.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:44.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:44 smithi069 bash[21528]: audit 2024-04-02T05:03:43.777794+0000 mgr.y (mgr.14538) 772 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:44.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:44 smithi069 bash[21528]: cluster 2024-04-02T05:03:44.243390+0000 mon.a (mon.0) 4027 : cluster 0 osdmap e890: 8 total, 8 up, 7 in 2024-04-02T05:03:44.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:44 smithi069 bash[28904]: audit 2024-04-02T05:03:43.777794+0000 mgr.y (mgr.14538) 772 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:44.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:44 smithi069 bash[28904]: cluster 2024-04-02T05:03:44.243390+0000 mon.a (mon.0) 4027 : cluster 0 osdmap e890: 8 total, 8 up, 7 in 2024-04-02T05:03:45.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:45.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:45.179 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:45.282 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:45.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:45.437 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:45.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:45.565 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:45.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:45.694 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:45.797 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:45.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[21528]: cluster 2024-04-02T05:03:44.374501+0000 mgr.y (mgr.14538) 773 : cluster 0 pgmap v1359: 192 pgs: 192 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:45.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[21528]: cluster 2024-04-02T05:03:44.570266+0000 osd.0 (osd.0) 435 : cluster 0 3.1b scrub starts 2024-04-02T05:03:45.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[21528]: cluster 2024-04-02T05:03:44.573407+0000 osd.0 (osd.0) 436 : cluster 0 3.1b scrub ok 2024-04-02T05:03:45.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.139475+0000 mon.a (mon.0) 4028 : cluster 0 osdmap e891: 8 total, 8 up, 7 in 2024-04-02T05:03:45.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[21528]: audit 2024-04-02T05:03:45.143558+0000 mon.a (mon.0) 4029 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:45.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[28904]: cluster 2024-04-02T05:03:44.374501+0000 mgr.y (mgr.14538) 773 : cluster 0 pgmap v1359: 192 pgs: 192 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:45.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[28904]: cluster 2024-04-02T05:03:44.570266+0000 osd.0 (osd.0) 435 : cluster 0 3.1b scrub starts 2024-04-02T05:03:45.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[28904]: cluster 2024-04-02T05:03:44.573407+0000 osd.0 (osd.0) 436 : cluster 0 3.1b scrub ok 2024-04-02T05:03:45.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.139475+0000 mon.a (mon.0) 4028 : cluster 0 osdmap e891: 8 total, 8 up, 7 in 2024-04-02T05:03:45.807 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:45 smithi069 bash[28904]: audit 2024-04-02T05:03:45.143558+0000 mon.a (mon.0) 4029 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:45.819 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:45.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:45.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:45 smithi098 bash[23790]: cluster 2024-04-02T05:03:44.374501+0000 mgr.y (mgr.14538) 773 : cluster 0 pgmap v1359: 192 pgs: 192 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:45.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:45 smithi098 bash[23790]: cluster 2024-04-02T05:03:44.570266+0000 osd.0 (osd.0) 435 : cluster 0 3.1b scrub starts 2024-04-02T05:03:45.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:45 smithi098 bash[23790]: cluster 2024-04-02T05:03:44.573407+0000 osd.0 (osd.0) 436 : cluster 0 3.1b scrub ok 2024-04-02T05:03:45.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:45 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.139475+0000 mon.a (mon.0) 4028 : cluster 0 osdmap e891: 8 total, 8 up, 7 in 2024-04-02T05:03:45.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:45 smithi098 bash[23790]: audit 2024-04-02T05:03:45.143558+0000 mon.a (mon.0) 4029 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-02T05:03:45.950 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:45.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:46.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:46.081 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:46.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:46.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:46.341 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:46.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:46.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:46.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.310983+0000 osd.3 (osd.3) 455 : cluster 0 1.c scrub starts 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.313028+0000 osd.3 (osd.3) 456 : cluster 0 1.c scrub ok 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.608756+0000 osd.0 (osd.0) 437 : cluster 0 6.0 deep-scrub starts 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.610336+0000 osd.0 (osd.0) 438 : cluster 0 6.0 deep-scrub ok 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.907344+0000 osd.6 (osd.6) 385 : cluster 0 10.a scrub starts 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:45.911288+0000 osd.6 (osd.6) 386 : cluster 0 10.a scrub ok 2024-04-02T05:03:46.713 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: audit 2024-04-02T05:03:46.136975+0000 mon.a (mon.0) 4030 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.145628+0000 mon.a (mon.0) 4031 : cluster 0 osdmap e892: 8 total, 8 up, 7 in 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: audit 2024-04-02T05:03:46.152883+0000 mon.a (mon.0) 4032 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-111","var": "dedup_tier","val": "test-rados-api-smithi069-72349-154"}]: dispatch 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: audit 2024-04-02T05:03:46.211304+0000 mon.a (mon.0) 4033 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: audit 2024-04-02T05:03:46.211950+0000 mon.a (mon.0) 4034 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-154"}]: dispatch 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.432252+0000 osd.1 (osd.1) 451 : cluster 0 6.d scrub starts 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.434505+0000 osd.1 (osd.1) 452 : cluster 0 6.d scrub ok 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.310983+0000 osd.3 (osd.3) 455 : cluster 0 1.c scrub starts 2024-04-02T05:03:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.313028+0000 osd.3 (osd.3) 456 : cluster 0 1.c scrub ok 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.608756+0000 osd.0 (osd.0) 437 : cluster 0 6.0 deep-scrub starts 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.610336+0000 osd.0 (osd.0) 438 : cluster 0 6.0 deep-scrub ok 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.907344+0000 osd.6 (osd.6) 385 : cluster 0 10.a scrub starts 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:45.911288+0000 osd.6 (osd.6) 386 : cluster 0 10.a scrub ok 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: audit 2024-04-02T05:03:46.136975+0000 mon.a (mon.0) 4030 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.145628+0000 mon.a (mon.0) 4031 : cluster 0 osdmap e892: 8 total, 8 up, 7 in 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: audit 2024-04-02T05:03:46.152883+0000 mon.a (mon.0) 4032 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-111","var": "dedup_tier","val": "test-rados-api-smithi069-72349-154"}]: dispatch 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: audit 2024-04-02T05:03:46.211304+0000 mon.a (mon.0) 4033 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: audit 2024-04-02T05:03:46.211950+0000 mon.a (mon.0) 4034 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-154"}]: dispatch 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.432252+0000 osd.1 (osd.1) 451 : cluster 0 6.d scrub starts 2024-04-02T05:03:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:46 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.434505+0000 osd.1 (osd.1) 452 : cluster 0 6.d scrub ok 2024-04-02T05:03:46.729 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:46.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:46.857 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:46.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:46.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.310983+0000 osd.3 (osd.3) 455 : cluster 0 1.c scrub starts 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.313028+0000 osd.3 (osd.3) 456 : cluster 0 1.c scrub ok 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.608756+0000 osd.0 (osd.0) 437 : cluster 0 6.0 deep-scrub starts 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.610336+0000 osd.0 (osd.0) 438 : cluster 0 6.0 deep-scrub ok 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.907344+0000 osd.6 (osd.6) 385 : cluster 0 10.a scrub starts 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:45.911288+0000 osd.6 (osd.6) 386 : cluster 0 10.a scrub ok 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: audit 2024-04-02T05:03:46.136975+0000 mon.a (mon.0) 4030 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-72349-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.145628+0000 mon.a (mon.0) 4031 : cluster 0 osdmap e892: 8 total, 8 up, 7 in 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: audit 2024-04-02T05:03:46.152883+0000 mon.a (mon.0) 4032 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-72349-111","var": "dedup_tier","val": "test-rados-api-smithi069-72349-154"}]: dispatch 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: audit 2024-04-02T05:03:46.211304+0000 mon.a (mon.0) 4033 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: audit 2024-04-02T05:03:46.211950+0000 mon.a (mon.0) 4034 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-72349-111", "tierpool": "test-rados-api-smithi069-72349-154"}]: dispatch 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.432252+0000 osd.1 (osd.1) 451 : cluster 0 6.d scrub starts 2024-04-02T05:03:46.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:46 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.434505+0000 osd.1 (osd.1) 452 : cluster 0 6.d scrub ok 2024-04-02T05:03:46.989 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:46.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:47.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:47.118 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:47.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:47.249 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:47.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:47.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:47.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:47.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:47.753 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:47 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.346913+0000 osd.3 (osd.3) 457 : cluster 0 1.4 deep-scrub starts 2024-04-02T05:03:47.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:47 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.349013+0000 osd.3 (osd.3) 458 : cluster 0 1.4 deep-scrub ok 2024-04-02T05:03:47.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:47 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.375301+0000 mgr.y (mgr.14538) 774 : cluster 0 pgmap v1362: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:47.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:47 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.584019+0000 osd.0 (osd.0) 439 : cluster 0 3.6 scrub starts 2024-04-02T05:03:47.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:47 smithi098 bash[23790]: cluster 2024-04-02T05:03:46.587783+0000 osd.0 (osd.0) 440 : cluster 0 3.6 scrub ok 2024-04-02T05:03:47.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:47 smithi098 bash[23790]: cluster 2024-04-02T05:03:47.153049+0000 mon.a (mon.0) 4035 : cluster 0 osdmap e893: 8 total, 8 up, 7 in 2024-04-02T05:03:47.770 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:47.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:47.899 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:47.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:48.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.346913+0000 osd.3 (osd.3) 457 : cluster 0 1.4 deep-scrub starts 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.349013+0000 osd.3 (osd.3) 458 : cluster 0 1.4 deep-scrub ok 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.375301+0000 mgr.y (mgr.14538) 774 : cluster 0 pgmap v1362: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.584019+0000 osd.0 (osd.0) 439 : cluster 0 3.6 scrub starts 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[21528]: cluster 2024-04-02T05:03:46.587783+0000 osd.0 (osd.0) 440 : cluster 0 3.6 scrub ok 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[21528]: cluster 2024-04-02T05:03:47.153049+0000 mon.a (mon.0) 4035 : cluster 0 osdmap e893: 8 total, 8 up, 7 in 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.346913+0000 osd.3 (osd.3) 457 : cluster 0 1.4 deep-scrub starts 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.349013+0000 osd.3 (osd.3) 458 : cluster 0 1.4 deep-scrub ok 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.375301+0000 mgr.y (mgr.14538) 774 : cluster 0 pgmap v1362: 224 pgs: 32 unknown, 192 active+clean; 4.4 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-02T05:03:48.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.584019+0000 osd.0 (osd.0) 439 : cluster 0 3.6 scrub starts 2024-04-02T05:03:48.015 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[28904]: cluster 2024-04-02T05:03:46.587783+0000 osd.0 (osd.0) 440 : cluster 0 3.6 scrub ok 2024-04-02T05:03:48.015 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:47 smithi069 bash[28904]: cluster 2024-04-02T05:03:47.153049+0000 mon.a (mon.0) 4035 : cluster 0 osdmap e893: 8 total, 8 up, 7 in 2024-04-02T05:03:48.029 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:48.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:48.166 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:48.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:48.297 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:48.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:48.530 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:48.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:48.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:48.685 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:48.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:48.815 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:48.837 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:03:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:03:48.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[21528]: cluster 2024-04-02T05:03:47.359330+0000 osd.3 (osd.3) 459 : cluster 0 3.18 scrub starts 2024-04-02T05:03:48.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[21528]: cluster 2024-04-02T05:03:47.365323+0000 osd.3 (osd.3) 460 : cluster 0 3.18 scrub ok 2024-04-02T05:03:48.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[21528]: cluster 2024-04-02T05:03:48.155066+0000 mon.a (mon.0) 4036 : cluster 0 osdmap e894: 8 total, 8 up, 7 in 2024-04-02T05:03:48.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[21528]: audit 2024-04-02T05:03:48.156908+0000 mon.a (mon.0) 4037 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:48.837 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[28904]: cluster 2024-04-02T05:03:47.359330+0000 osd.3 (osd.3) 459 : cluster 0 3.18 scrub starts 2024-04-02T05:03:48.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[28904]: cluster 2024-04-02T05:03:47.365323+0000 osd.3 (osd.3) 460 : cluster 0 3.18 scrub ok 2024-04-02T05:03:48.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[28904]: cluster 2024-04-02T05:03:48.155066+0000 mon.a (mon.0) 4036 : cluster 0 osdmap e894: 8 total, 8 up, 7 in 2024-04-02T05:03:48.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:48 smithi069 bash[28904]: audit 2024-04-02T05:03:48.156908+0000 mon.a (mon.0) 4037 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:48.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:48.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:48 smithi098 bash[23790]: cluster 2024-04-02T05:03:47.359330+0000 osd.3 (osd.3) 459 : cluster 0 3.18 scrub starts 2024-04-02T05:03:48.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:48 smithi098 bash[23790]: cluster 2024-04-02T05:03:47.365323+0000 osd.3 (osd.3) 460 : cluster 0 3.18 scrub ok 2024-04-02T05:03:48.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:48 smithi098 bash[23790]: cluster 2024-04-02T05:03:48.155066+0000 mon.a (mon.0) 4036 : cluster 0 osdmap e894: 8 total, 8 up, 7 in 2024-04-02T05:03:48.946 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:48 smithi098 bash[23790]: audit 2024-04-02T05:03:48.156908+0000 mon.a (mon.0) 4037 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:48.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:49.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:49.074 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:49.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:49.206 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:49.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:49.336 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:49.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:49.459 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3 2024-04-02T05:03:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.3' 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/1072786608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514107220 msgr2=0x7f3514107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- 172.21.15.69:0/1072786608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514107220 0x7f3514107600 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7f3504009990 tx=0x7f3504009ca0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/1072786608 shutdown_connections 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- 172.21.15.69:0/1072786608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3514101b60 0x7f351410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- 172.21.15.69:0/1072786608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514107220 0x7f3514107600 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- 172.21.15.69:0/1072786608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35141011a0 0x7f3514101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/1072786608 >> 172.21.15.69:0/1072786608 conn(0x7f35140771e0 msgr2=0x7f35140775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/1072786608 shutdown_connections 2024-04-02T05:03:49.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/1072786608 wait complete. 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 librados: starting msgr at 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 librados: starting objecter 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 Processor -- start 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- start start 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 librados: setting wanted keys 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 librados: calling monclient init 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 0x7f3514198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514101b60 0x7f3514198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3514107220 0x7f351419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3514110970 con 0x7f35141011a0 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35141107f0 con 0x7f3514107220 2024-04-02T05:03:49.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3514110af0 con 0x7f3514101b60 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 0x7f3514198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 0x7f3514198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54500/0 (socket says 172.21.15.69:54500) 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 -- 172.21.15.69:0/2304563852 learned_addr learned my addr 172.21.15.69:0/2304563852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514101b60 0x7f3514198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514101b60 msgr2=0x7f3514198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514101b60 0x7f3514198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3514107220 0x7f351419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3514107220 msgr2=0x7f351419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3514107220 0x7f351419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351419d050 con 0x7f35141011a0 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3514101b60 0x7f3514198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 0x7f3514198040 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f35040057f0 tx=0x7f350400bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3510ff9700 10 client.?.objecter ms_handle_connect 0x7f35141011a0 2024-04-02T05:03:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3510ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504026070 con 0x7f35141011a0 2024-04-02T05:03:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3504021410 con 0x7f35141011a0 2024-04-02T05:03:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351419d2e0 con 0x7f35141011a0 2024-04-02T05:03:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504017650 con 0x7f35141011a0 2024-04-02T05:03:49.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 10 client.33819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:49.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35141a4bc0 con 0x7f35141011a0 2024-04-02T05:03:49.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 librados: init done 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.541+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f351404ff00 con 0x7f35141011a0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3504029070 con 0x7f35141011a0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f34fc06f5e0 0x7f34fc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(895..895 src has 74..895) ==== 8980+0+0 (secure 0 0 0) 0x7f350408f970 con 0x7f35141011a0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 osd_map(895..895 src has 74..895) 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 3 client.33819.objecter handle_osd_map got epochs [895,895] > 0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 3 client.33819.objecter handle_osd_map decoding full epoch 895 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f34fc06f5e0 0x7f34fc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 20 client.33819.objecter dump_active .. 0 homeless 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v895) ==== 54+0+16 (secure 0 0 0) 0x7f3504030020 con 0x7f35141011a0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f34fc06f5e0 0x7f34fc071aa0 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f3514077990 tx=0x7f3508009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f34fc06f5e0 2024-04-02T05:03:49.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 10 client.33819.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 1 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3514062e50 0x7f3514063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 1 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 0 1 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 4 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 10 client.33819.objecter _send_command 1 2024-04-02T05:03:49.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.545+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35141a7fe0 con 0x7f3514062e50 2024-04-02T05:03:49.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:49.581 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:49.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.589+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3514062e50 0x7f3514063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f3514062e50 2024-04-02T05:03:49.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3514062e50 0x7f3514063230 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141a7fe0 con 0x7f3514062e50 2024-04-02T05:03:49.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 1: 0 ) 2024-04-02T05:03:49.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3510ff9700 10 client.33819.objecter _finish_command 1 = system:0 2024-04-02T05:03:49.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 4 2024-04-02T05:03:49.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.593+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 0 1 2024-04-02T05:03:49.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 10 client.33819.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 2 2024-04-02T05:03:49.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 2 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141993a0 osd=0 3 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 0 2 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 4 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 10 client.33819.objecter _send_command 2 2024-04-02T05:03:49.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35140628d0 con 0x7f3514062e50 2024-04-02T05:03:49.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35140628d0 con 0x7f3514062e50 2024-04-02T05:03:49.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 10 client.33819.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 4 2024-04-02T05:03:49.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 0 2 2024-04-02T05:03:49.618 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0: {} 2024-04-02T05:03:49.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 10 client.33819.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 3 2024-04-02T05:03:49.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f351402c9b0 0x7f351402cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 3 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 1 3 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 4 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 10 client.33819.objecter _send_command 3 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f351410ad70 con 0x7f351402c9b0 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f351402c9b0 0x7f351402cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f351402c9b0 0x7f351402cd90 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f351402c9b0 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f351410ad70 con 0x7f351402c9b0 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 3: 0 ) 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 10 client.33819.objecter _finish_command 3 = system:0 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 4 2024-04-02T05:03:49.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.613+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 1 3 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 10 client.33819.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 4 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 4 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f351410a6f0 osd=1 3 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 1 4 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 4 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 10 client.33819.objecter _send_command 4 2024-04-02T05:03:49.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f351410a170 con 0x7f351402c9b0 2024-04-02T05:03:49.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f351410a170 con 0x7f351402c9b0 2024-04-02T05:03:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3510ff9700 10 client.33819.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 4 2024-04-02T05:03:49.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 1 4 2024-04-02T05:03:49.639 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1: {} 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 10 client.33819.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 5 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35141aa1a0 0x7f35141ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 5 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 2 5 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 4 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 10 client.33819.objecter _send_command 5 2024-04-02T05:03:49.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f35141af3f0 con 0x7f35141aa1a0 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35141aa1a0 0x7f35141ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35141aa1a0 0x7f35141ac590 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.633+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f35141aa1a0 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.637+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141af3f0 con 0x7f35141aa1a0 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.637+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 5: 0 ) 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.637+0000 7f3510ff9700 10 client.33819.objecter _finish_command 5 = system:0 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.637+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 4 2024-04-02T05:03:49.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.637+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 2 5 2024-04-02T05:03:49.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 10 client.33819.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 6 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 6 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f351402daa0 osd=2 3 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 2 6 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 4 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 10 client.33819.objecter _send_command 6 2024-04-02T05:03:49.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f351402d520 con 0x7f35141aa1a0 2024-04-02T05:03:49.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f351402d520 con 0x7f35141aa1a0 2024-04-02T05:03:49.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3510ff9700 10 client.33819.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 4 2024-04-02T05:03:49.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 2 6 2024-04-02T05:03:49.660 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2: {} 2024-04-02T05:03:49.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 10 client.33819.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 7 2024-04-02T05:03:49.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35141adc50 0x7f35141ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 7 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 3 7 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 4 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 10 client.33819.objecter _send_command 7 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.653+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f35141a7cb0 con 0x7f35141adc50 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35141adc50 0x7f35141ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f35141adc50 2024-04-02T05:03:49.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35141adc50 0x7f35141ae070 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141a7cb0 con 0x7f35141adc50 2024-04-02T05:03:49.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 7: 0 ) 2024-04-02T05:03:49.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3510ff9700 10 client.33819.objecter _finish_command 7 = system:0 2024-04-02T05:03:49.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 4 2024-04-02T05:03:49.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.657+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 3 7 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 10 client.33819.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 8 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 8 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141a7630 osd=3 3 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 3 8 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 4 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 10 client.33819.objecter _send_command 8 2024-04-02T05:03:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.673+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35141a7cb0 con 0x7f35141adc50 2024-04-02T05:03:49.680 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3: {} 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35141a7cb0 con 0x7f35141adc50 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3510ff9700 10 client.33819.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 4 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 3 8 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 10 client.33819.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 9 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35141aed80 0x7f35141b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 9 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 4 9 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 4 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 10 client.33819.objecter _send_command 9 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f35141af1e0 con 0x7f35141aed80 2024-04-02T05:03:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35141aed80 0x7f35141b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f35141aed80 2024-04-02T05:03:49.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.677+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35141aed80 0x7f35141b59c0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:49.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.681+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141af1e0 con 0x7f35141aed80 2024-04-02T05:03:49.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.681+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 9: 0 ) 2024-04-02T05:03:49.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.681+0000 7f3510ff9700 10 client.33819.objecter _finish_command 9 = system:0 2024-04-02T05:03:49.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.681+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 4 2024-04-02T05:03:49.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.681+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 4 9 2024-04-02T05:03:49.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 10 client.33819.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 10 2024-04-02T05:03:49.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 10 2024-04-02T05:03:49.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141a7cb0 osd=4 3 2024-04-02T05:03:49.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 4 10 2024-04-02T05:03:49.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 4 2024-04-02T05:03:49.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 10 client.33819.objecter _send_command 10 2024-04-02T05:03:49.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.697+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35141af1e0 con 0x7f35141aed80 2024-04-02T05:03:49.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35141af1e0 con 0x7f35141aed80 2024-04-02T05:03:49.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 10 client.33819.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 4 2024-04-02T05:03:49.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 4 10 2024-04-02T05:03:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4: {} 2024-04-02T05:03:49.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 10 client.33819.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 11 2024-04-02T05:03:49.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35141b6410 0x7f35141b8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 11 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 5 11 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 4 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 10 client.33819.objecter _send_command 11 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f35141bb800 con 0x7f35141b6410 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35141b6410 0x7f35141b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f35141b6410 2024-04-02T05:03:49.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3512ffd700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35141b6410 0x7f35141b8800 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141bb800 con 0x7f35141b6410 2024-04-02T05:03:49.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 11: 0 ) 2024-04-02T05:03:49.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 10 client.33819.objecter _finish_command 11 = system:0 2024-04-02T05:03:49.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 4 2024-04-02T05:03:49.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.701+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 5 11 2024-04-02T05:03:49.711 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:49.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 10 client.33819.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 12 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 12 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141af1e0 osd=5 3 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 5 12 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 4 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 10 client.33819.objecter _send_command 12 2024-04-02T05:03:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.717+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35141b8e30 con 0x7f35141b6410 2024-04-02T05:03:49.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35141b8e30 con 0x7f35141b6410 2024-04-02T05:03:49.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 10 client.33819.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 4 2024-04-02T05:03:49.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 5 12 2024-04-02T05:03:49.726 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5: {} 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 10 client.33819.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 13 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35141b9550 0x7f35141b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 13 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 6 13 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 4 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 10 client.33819.objecter _send_command 13 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f35141bec80 con 0x7f35141b9550 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35141b9550 0x7f35141b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f35141b9550 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3513fff700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35141b9550 0x7f35141b99b0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.721+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141bec80 con 0x7f35141b9550 2024-04-02T05:03:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.725+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 13: 0 ) 2024-04-02T05:03:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.725+0000 7f3510ff9700 10 client.33819.objecter _finish_command 13 = system:0 2024-04-02T05:03:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.725+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 4 2024-04-02T05:03:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.725+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 6 13 2024-04-02T05:03:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 10 client.33819.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 14 2024-04-02T05:03:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 14 2024-04-02T05:03:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141b8e30 osd=6 3 2024-04-02T05:03:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 6 14 2024-04-02T05:03:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 4 2024-04-02T05:03:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 10 client.33819.objecter _send_command 14 2024-04-02T05:03:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35141bec80 con 0x7f35141b9550 2024-04-02T05:03:49.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35141bec80 con 0x7f35141b9550 2024-04-02T05:03:49.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3510ff9700 10 client.33819.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 4 2024-04-02T05:03:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 6 14 2024-04-02T05:03:49.748 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6: {} 2024-04-02T05:03:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 10 client.33819.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:03:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 15 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f35141bdf60 0x7f35141be3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 15 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 7 15 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 4 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 10 client.33819.objecter _send_command 15 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.741+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f35141bb170 con 0x7f35141bdf60 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f35141bdf60 0x7f35141be3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f35137fe700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f35141bdf60 0x7f35141be3e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:49.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f3510ff9700 10 client.33819.objecter ms_handle_connect 0x7f35141bdf60 2024-04-02T05:03:49.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35141bb170 con 0x7f35141bdf60 2024-04-02T05:03:49.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 15: 0 ) 2024-04-02T05:03:49.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f3510ff9700 10 client.33819.objecter _finish_command 15 = system:0 2024-04-02T05:03:49.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 4 2024-04-02T05:03:49.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.745+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 7 15 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 10 client.33819.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign -1 16 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 20 client.33819.objecter _get_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 15 client.33819.objecter _session_command_op_remove -1 16 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 20 client.33819.objecter get_session s=0x7f35141bec80 osd=7 3 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 15 client.33819.objecter _session_command_op_assign 7 16 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 4 2024-04-02T05:03:49.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 10 client.33819.objecter _send_command 16 2024-04-02T05:03:49.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.761+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f35141bb380 con 0x7f35141bdf60 2024-04-02T05:03:49.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3510ff9700 1 -- 172.21.15.69:0/2304563852 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f35141bb380 con 0x7f35141bdf60 2024-04-02T05:03:49.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3510ff9700 10 client.33819.objecter ms_dispatch 0x7f3514001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-02T05:03:49.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3510ff9700 10 client.33819.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-02T05:03:49.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3510ff9700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 4 2024-04-02T05:03:49.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3510ff9700 15 client.33819.objecter _session_command_op_remove 7 16 2024-04-02T05:03:49.771 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7: {} 2024-04-02T05:03:49.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 librados: watch_flush enter 2024-04-02T05:03:49.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 librados: watch_flush exit 2024-04-02T05:03:49.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.0 2024-04-02T05:03:49.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3514062e50 msgr2=0x7f3514063230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7f3514062e50 0x7f3514063230 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141993a0 osd=0 1 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.1 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f351402c9b0 msgr2=0x7f351402cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f351402c9b0 0x7f351402cd90 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351410a6f0 osd=1 1 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.2 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35141aa1a0 msgr2=0x7f35141ac590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f35141aa1a0 0x7f35141ac590 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f351402daa0 osd=2 1 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.3 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35141adc50 msgr2=0x7f35141ae070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f35141adc50 0x7f35141ae070 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7630 osd=3 1 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.4 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35141aed80 msgr2=0x7f35141b59c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f35141aed80 0x7f35141b59c0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141a7cb0 osd=4 1 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.5 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3513fff700 1 -- 172.21.15.69:0/2304563852 reap_dead start 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35141b6410 msgr2=0x7f35141b8800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f35141b6410 0x7f35141b8800 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3513fff700 1 -- 172.21.15.69:0/2304563852 reap_dead start 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141af1e0 osd=5 1 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.6 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3513fff700 1 -- 172.21.15.69:0/2304563852 reap_dead start 2024-04-02T05:03:49.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35141b9550 msgr2=0x7f35141b99b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35141b9550 0x7f35141b99b0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141b8e30 osd=6 1 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter close_session for osd.7 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f35141bdf60 msgr2=0x7f35141be3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f35141bdf60 0x7f35141be3e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter put_session s=0x7f35141bec80 osd=7 1 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 20 client.33819.objecter shutdown clearing up homeless session... 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 10 client.33819.objecter successfully canceled tick 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f34fc06f5e0 msgr2=0x7f34fc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f34fc06f5e0 0x7f34fc071aa0 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f3514077990 tx=0x7f3508009190 comp rx=0 tx=0).stop 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 msgr2=0x7f3514198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 0x7f3514198040 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f35040057f0 tx=0x7f350400bfa0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 shutdown_connections 2024-04-02T05:03:49.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f35141bdf60 0x7f35141be3e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f35141b9550 0x7f35141b99b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35141011a0 0x7f3514198040 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 --2- 172.21.15.69:0/2304563852 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f34fc06f5e0 0x7f34fc071aa0 unknown :-1 s=CLOSED pgs=4027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.765+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 >> 172.21.15.69:0/2304563852 conn(0x7f35140771e0 msgr2=0x7f3514069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.769+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 shutdown_connections 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.769+0000 7f3519a84700 1 -- 172.21.15.69:0/2304563852 wait complete. 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:49.769+0000 7f3519a84700 1 librados: shutdown 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-02T05:03:49.776 INFO:teuthology.orchestra.run.smithi069.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-02T05:03:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-02T05:03:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-02T05:03:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-02T05:03:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-02T05:03:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-02T05:03:49.777 INFO:teuthology.orchestra.run.smithi069.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-02T05:03:49.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: cluster 2024-04-02T05:03:48.346588+0000 osd.3 (osd.3) 461 : cluster 0 3.8 scrub starts 2024-04-02T05:03:49.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: cluster 2024-04-02T05:03:48.354551+0000 osd.3 (osd.3) 462 : cluster 0 3.8 scrub ok 2024-04-02T05:03:49.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: cluster 2024-04-02T05:03:48.376061+0000 mgr.y (mgr.14538) 775 : cluster 0 pgmap v1365: 184 pgs: 184 active+clean; 582 KiB data, 766 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:49.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: cluster 2024-04-02T05:03:49.150987+0000 mon.a (mon.0) 4038 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:49.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: audit 2024-04-02T05:03:49.152659+0000 mon.a (mon.0) 4039 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:49.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: cluster 2024-04-02T05:03:49.161412+0000 mon.a (mon.0) 4040 : cluster 0 osdmap e895: 8 total, 8 up, 7 in 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: audit 2024-04-02T05:03:49.164416+0000 mon.a (mon.0) 4041 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: audit 2024-04-02T05:03:49.238550+0000 mon.a (mon.0) 4042 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: audit 2024-04-02T05:03:49.241045+0000 mon.b (mon.1) 324 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[28904]: audit 2024-04-02T05:03:49.547936+0000 mon.a (mon.0) 4043 : audit 0 from='client.? 172.21.15.69:0/2304563852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: cluster 2024-04-02T05:03:48.346588+0000 osd.3 (osd.3) 461 : cluster 0 3.8 scrub starts 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: cluster 2024-04-02T05:03:48.354551+0000 osd.3 (osd.3) 462 : cluster 0 3.8 scrub ok 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: cluster 2024-04-02T05:03:48.376061+0000 mgr.y (mgr.14538) 775 : cluster 0 pgmap v1365: 184 pgs: 184 active+clean; 582 KiB data, 766 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: cluster 2024-04-02T05:03:49.150987+0000 mon.a (mon.0) 4038 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: audit 2024-04-02T05:03:49.152659+0000 mon.a (mon.0) 4039 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: cluster 2024-04-02T05:03:49.161412+0000 mon.a (mon.0) 4040 : cluster 0 osdmap e895: 8 total, 8 up, 7 in 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: audit 2024-04-02T05:03:49.164416+0000 mon.a (mon.0) 4041 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: audit 2024-04-02T05:03:49.238550+0000 mon.a (mon.0) 4042 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:49.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: audit 2024-04-02T05:03:49.241045+0000 mon.b (mon.1) 324 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:49.810 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:49 smithi069 bash[21528]: audit 2024-04-02T05:03:49.547936+0000 mon.a (mon.0) 4043 : audit 0 from='client.? 172.21.15.69:0/2304563852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:49.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:49.838 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2 2024-04-02T05:03:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.2' 2024-04-02T05:03:49.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: cluster 2024-04-02T05:03:48.346588+0000 osd.3 (osd.3) 461 : cluster 0 3.8 scrub starts 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: cluster 2024-04-02T05:03:48.354551+0000 osd.3 (osd.3) 462 : cluster 0 3.8 scrub ok 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: cluster 2024-04-02T05:03:48.376061+0000 mgr.y (mgr.14538) 775 : cluster 0 pgmap v1365: 184 pgs: 184 active+clean; 582 KiB data, 766 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: cluster 2024-04-02T05:03:49.150987+0000 mon.a (mon.0) 4038 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: audit 2024-04-02T05:03:49.152659+0000 mon.a (mon.0) 4039 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: cluster 2024-04-02T05:03:49.161412+0000 mon.a (mon.0) 4040 : cluster 0 osdmap e895: 8 total, 8 up, 7 in 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: audit 2024-04-02T05:03:49.164416+0000 mon.a (mon.0) 4041 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]: dispatch 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: audit 2024-04-02T05:03:49.238550+0000 mon.a (mon.0) 4042 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:03:49.952 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: audit 2024-04-02T05:03:49.241045+0000 mon.b (mon.1) 324 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:03:49.953 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:49 smithi098 bash[23790]: audit 2024-04-02T05:03:49.547936+0000 mon.a (mon.0) 4043 : audit 0 from='client.? 172.21.15.69:0/2304563852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-02T05:03:49.967 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5 2024-04-02T05:03:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:49.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.5' 2024-04-02T05:03:50.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:50.098 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7993 ms) 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12640 ms) 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7358 ms) 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-02T05:03:50.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13057 ms) 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7952 ms) 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8028 ms) 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 337:602f83fe:::foo:head 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9085 ms) 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,0 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1712034145 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,0 2024-04-02T05:03:50.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,1712034150,1712034151,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,1712034150,1712034151,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,1712034150,1712034151,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,1712034150,1712034151,1712034153,1712034154,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,1712034150,1712034151,1712034153,1712034154,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034145,1712034147,1712034148,1712034150,1712034151,1712034153,1712034154,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1712034148,1712034150,1712034151,1712034153,1712034154,1712034156,1712034157,0 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1712034148, trimmed 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21197 ms) 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-02T05:03:50.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14729 ms) 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18343 ms) 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22100 ms) 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4595 ms) 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3042 ms) 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3021 ms) 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-02T05:03:50.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (2910 ms) 2024-04-02T05:03:50.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (231118 ms total) 2024-04-02T05:03:50.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-04-02T05:03:50.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-02T05:03:50.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (880076 ms total) 2024-04-02T05:03:50.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-02T05:03:50.170 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.170 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72172 2024-04-02T05:03:50.170 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72172 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72373 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72373 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72716 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72716 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72490 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72490 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72413 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72413 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.171 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72245 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72245 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72434 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72434 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72876 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72876 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72914 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72914 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72299 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72299 2024-04-02T05:03:50.172 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72764 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72764 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72647 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72647 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72156 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72156 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72188 2024-04-02T05:03:50.177 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72188 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72657 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72657 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72148 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72148 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72164 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72164 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72958 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72958 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.178 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72383 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72383 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73008 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 73008 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72422 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72422 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72844 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72844 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73029 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 73029 2024-04-02T05:03:50.179 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72405 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72405 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72669 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72669 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72944 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72944 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72821 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72821 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72992 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72992 2024-04-02T05:03:50.180 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72634 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72634 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72356 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72356 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73018 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 73018 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72548 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72548 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72140 2024-04-02T05:03:50.181 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72140 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72180 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72180 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72468 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72468 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72201 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72201 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72563 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72563 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.182 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72585 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72585 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=72936 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 72936 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=73043 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 73043 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ exit 0 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ cleanup 2024-04-02T05:03:50.183 INFO:tasks.workunit.client.0.smithi069.stderr:+ pkill -P 72134 2024-04-02T05:03:50.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:50.330 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2024-04-02T05:03:50.330 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-02T05:03:50.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-02T05:03:50.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:50.346 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-04-02T05:03:50.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-02T05:03:50.356 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0 2024-04-02T05:03:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.0' 2024-04-02T05:03:50.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7 2024-04-02T05:03:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.7' 2024-04-02T05:03:50.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6 2024-04-02T05:03:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.6' 2024-04-02T05:03:50.698 DEBUG:teuthology.parallel:result is None 2024-04-02T05:03:50.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-02T05:03:50.707 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-02T05:03:50.707 DEBUG:teuthology.orchestra.run.smithi069:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-02T05:03:50.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:50.741 INFO:teuthology.orchestra.run.smithi098.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4 2024-04-02T05:03:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.4' 2024-04-02T05:03:50.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:50.866 INFO:teuthology.orchestra.run.smithi069.stderr:Error response from daemon: Cannot kill container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1 2024-04-02T05:03:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:50.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi069 with status 1: 'sudo docker kill -s 1 ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd.1' 2024-04-02T05:03:50.871 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-02T05:03:50.871 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-02T05:03:50.880 INFO:tasks.thrashosds:joining thrashosds 2024-04-02T05:03:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.004994+0000 osd.4 (osd.4) 440 : cluster 0 3.1a scrub starts 2024-04-02T05:03:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.008379+0000 osd.4 (osd.4) 441 : cluster 0 3.1a scrub ok 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.052052+0000 osd.2 (osd.2) 378 : cluster 0 1.8 scrub starts 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.055096+0000 osd.2 (osd.2) 379 : cluster 0 1.8 scrub ok 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: audit 2024-04-02T05:03:50.154223+0000 mon.a (mon.0) 4044 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.157173+0000 mon.a (mon.0) 4045 : cluster 0 osdmap e896: 8 total, 8 up, 7 in 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.376490+0000 mgr.y (mgr.14538) 776 : cluster 0 pgmap v1368: 184 pgs: 184 active+clean; 582 KiB data, 764 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.633103+0000 osd.0 (osd.0) 441 : cluster 0 3.1f scrub starts 2024-04-02T05:03:51.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:51 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.634906+0000 osd.0 (osd.0) 442 : cluster 0 3.1f scrub ok 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.004994+0000 osd.4 (osd.4) 440 : cluster 0 3.1a scrub starts 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.008379+0000 osd.4 (osd.4) 441 : cluster 0 3.1a scrub ok 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.052052+0000 osd.2 (osd.2) 378 : cluster 0 1.8 scrub starts 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.055096+0000 osd.2 (osd.2) 379 : cluster 0 1.8 scrub ok 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: audit 2024-04-02T05:03:50.154223+0000 mon.a (mon.0) 4044 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.157173+0000 mon.a (mon.0) 4045 : cluster 0 osdmap e896: 8 total, 8 up, 7 in 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.376490+0000 mgr.y (mgr.14538) 776 : cluster 0 pgmap v1368: 184 pgs: 184 active+clean; 582 KiB data, 764 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.633103+0000 osd.0 (osd.0) 441 : cluster 0 3.1f scrub starts 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.634906+0000 osd.0 (osd.0) 442 : cluster 0 3.1f scrub ok 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.004994+0000 osd.4 (osd.4) 440 : cluster 0 3.1a scrub starts 2024-04-02T05:03:51.479 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.008379+0000 osd.4 (osd.4) 441 : cluster 0 3.1a scrub ok 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.052052+0000 osd.2 (osd.2) 378 : cluster 0 1.8 scrub starts 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.055096+0000 osd.2 (osd.2) 379 : cluster 0 1.8 scrub ok 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: audit 2024-04-02T05:03:50.154223+0000 mon.a (mon.0) 4044 : audit 1 from='client.? 172.21.15.69:0/2099179437' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-72349-111"}]': finished 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.157173+0000 mon.a (mon.0) 4045 : cluster 0 osdmap e896: 8 total, 8 up, 7 in 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.376490+0000 mgr.y (mgr.14538) 776 : cluster 0 pgmap v1368: 184 pgs: 184 active+clean; 582 KiB data, 764 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.633103+0000 osd.0 (osd.0) 441 : cluster 0 3.1f scrub starts 2024-04-02T05:03:51.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:51 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.634906+0000 osd.0 (osd.0) 442 : cluster 0 3.1f scrub ok 2024-04-02T05:03:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:52 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.356832+0000 osd.7 (osd.7) 251 : cluster 0 1.10 scrub starts 2024-04-02T05:03:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:52 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.363294+0000 osd.7 (osd.7) 252 : cluster 0 1.10 scrub ok 2024-04-02T05:03:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:52 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.920517+0000 osd.6 (osd.6) 387 : cluster 0 10.12 scrub starts 2024-04-02T05:03:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:52 smithi098 bash[23790]: cluster 2024-04-02T05:03:50.922891+0000 osd.6 (osd.6) 388 : cluster 0 10.12 scrub ok 2024-04-02T05:03:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:52 smithi098 bash[23790]: cluster 2024-04-02T05:03:51.372492+0000 osd.1 (osd.1) 453 : cluster 0 2.15 deep-scrub starts 2024-04-02T05:03:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:52 smithi098 bash[23790]: cluster 2024-04-02T05:03:51.374762+0000 osd.1 (osd.1) 454 : cluster 0 2.15 deep-scrub ok 2024-04-02T05:03:52.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.356832+0000 osd.7 (osd.7) 251 : cluster 0 1.10 scrub starts 2024-04-02T05:03:52.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.363294+0000 osd.7 (osd.7) 252 : cluster 0 1.10 scrub ok 2024-04-02T05:03:52.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.920517+0000 osd.6 (osd.6) 387 : cluster 0 10.12 scrub starts 2024-04-02T05:03:52.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[21528]: cluster 2024-04-02T05:03:50.922891+0000 osd.6 (osd.6) 388 : cluster 0 10.12 scrub ok 2024-04-02T05:03:52.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[21528]: cluster 2024-04-02T05:03:51.372492+0000 osd.1 (osd.1) 453 : cluster 0 2.15 deep-scrub starts 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[21528]: cluster 2024-04-02T05:03:51.374762+0000 osd.1 (osd.1) 454 : cluster 0 2.15 deep-scrub ok 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.356832+0000 osd.7 (osd.7) 251 : cluster 0 1.10 scrub starts 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.363294+0000 osd.7 (osd.7) 252 : cluster 0 1.10 scrub ok 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.920517+0000 osd.6 (osd.6) 387 : cluster 0 10.12 scrub starts 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[28904]: cluster 2024-04-02T05:03:50.922891+0000 osd.6 (osd.6) 388 : cluster 0 10.12 scrub ok 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[28904]: cluster 2024-04-02T05:03:51.372492+0000 osd.1 (osd.1) 453 : cluster 0 2.15 deep-scrub starts 2024-04-02T05:03:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:52 smithi069 bash[28904]: cluster 2024-04-02T05:03:51.374762+0000 osd.1 (osd.1) 454 : cluster 0 2.15 deep-scrub ok 2024-04-02T05:03:53.363 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:03:53.363 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T05:03:53.363 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2024-04-02T05:03:53.363 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 -- 172.21.15.69:0/1947480167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6006ab80 msgr2=0x7f6a60106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 --2- 172.21.15.69:0/1947480167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6006ab80 0x7f6a60106ef0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7f6a500099c0 tx=0x7f6a50009cd0 comp rx=0 tx=0).stop 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 -- 172.21.15.69:0/1947480167 shutdown_connections 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 --2- 172.21.15.69:0/1947480167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a60107620 0x7f6a6010f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 --2- 172.21.15.69:0/1947480167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6006a1d0 0x7f6a6006a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 --2- 172.21.15.69:0/1947480167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6006ab80 0x7f6a60106ef0 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:53.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 -- 172.21.15.69:0/1947480167 >> 172.21.15.69:0/1947480167 conn(0x7f6a600771e0 msgr2=0x7f6a600775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 -- 172.21.15.69:0/1947480167 shutdown_connections 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.401+0000 7f6a67296700 1 -- 172.21.15.69:0/1947480167 wait complete. 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 librados: starting msgr at 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 librados: starting objecter 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 Processor -- start 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- start start 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 librados: setting wanted keys 2024-04-02T05:03:53.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 librados: calling monclient init 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a6006a1d0 0x7f6a601980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6006ab80 0x7f6a601985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a60107620 0x7f6a6019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60111ab0 con 0x7f6a6006a1d0 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60111930 con 0x7f6a6006ab80 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f6a60111c30 con 0x7f6a60107620 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65032700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a6006a1d0 0x7f6a601980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65032700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a6006a1d0 0x7f6a601980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54516/0 (socket says 172.21.15.69:54516) 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65032700 1 -- 172.21.15.69:0/3694514296 learned_addr learned my addr 172.21.15.69:0/3694514296 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:03:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a60107620 0x7f6a6019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a64831700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6006ab80 0x7f6a601985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 -- 172.21.15.69:0/3694514296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6006ab80 msgr2=0x7f6a601985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6006ab80 0x7f6a601985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 -- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a6006a1d0 msgr2=0x7f6a601980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a6006a1d0 0x7f6a601980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 -- 172.21.15.69:0/3694514296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6019d0c0 con 0x7f6a60107620 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65833700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a60107620 0x7f6a6019c980 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800efb0 tx=0x7f6a5800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 10 client.?.objecter ms_handle_connect 0x7f6a60107620 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800ca50 con 0x7f6a60107620 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f6a5800cbf0 con 0x7f6a60107620 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6019d3b0 con 0x7f6a60107620 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58004740 con 0x7f6a60107620 2024-04-02T05:03:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 10 client.43889.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:03:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a601a4c30 con 0x7f6a60107620 2024-04-02T05:03:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 librados: init done 2024-04-02T05:03:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6004ff00 con 0x7f6a60107620 2024-04-02T05:03:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f6a58012050 con 0x7f6a60107620 2024-04-02T05:03:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a4c06f5e0 0x7f6a4c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(896..896 src has 574..896) ==== 8870+0+0 (secure 0 0 0) 0x7f6a5808def0 con 0x7f6a60107620 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a65032700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a4c06f5e0 0x7f6a4c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 10 client.43889.objecter ms_dispatch 0x7f6a60001550 osd_map(896..896 src has 574..896) 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 3 client.43889.objecter handle_osd_map got epochs [896,896] > 0 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.405+0000 7f6a567fc700 3 client.43889.objecter handle_osd_map decoding full epoch 896 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.409+0000 7f6a567fc700 20 client.43889.objecter dump_active .. 0 homeless 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.409+0000 7f6a567fc700 10 client.43889.objecter ms_handle_connect 0x7f6a4c06f5e0 2024-04-02T05:03:53.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.409+0000 7f6a65032700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a4c06f5e0 0x7f6a4c071aa0 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00bd30 tx=0x7f6a5c00b480 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:03:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.413+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6a58022060 con 0x7f6a60107620 2024-04-02T05:03:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:53 smithi098 bash[23790]: cluster 2024-04-02T05:03:52.376956+0000 mgr.y (mgr.14538) 777 : cluster 0 pgmap v1369: 184 pgs: 184 active+clean; 582 KiB data, 764 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:53 smithi098 bash[23790]: cluster 2024-04-02T05:03:52.678738+0000 osd.0 (osd.0) 443 : cluster 0 10.4 scrub starts 2024-04-02T05:03:53.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:53 smithi098 bash[23790]: cluster 2024-04-02T05:03:52.681231+0000 osd.0 (osd.0) 444 : cluster 0 10.4 scrub ok 2024-04-02T05:03:53.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:53 smithi069 bash[21528]: cluster 2024-04-02T05:03:52.376956+0000 mgr.y (mgr.14538) 777 : cluster 0 pgmap v1369: 184 pgs: 184 active+clean; 582 KiB data, 764 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:53 smithi069 bash[21528]: cluster 2024-04-02T05:03:52.678738+0000 osd.0 (osd.0) 443 : cluster 0 10.4 scrub starts 2024-04-02T05:03:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:53 smithi069 bash[21528]: cluster 2024-04-02T05:03:52.681231+0000 osd.0 (osd.0) 444 : cluster 0 10.4 scrub ok 2024-04-02T05:03:53.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:53 smithi069 bash[28904]: cluster 2024-04-02T05:03:52.376956+0000 mgr.y (mgr.14538) 777 : cluster 0 pgmap v1369: 184 pgs: 184 active+clean; 582 KiB data, 764 MiB used, 625 GiB / 626 GiB avail 2024-04-02T05:03:53.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:53 smithi069 bash[28904]: cluster 2024-04-02T05:03:52.678738+0000 osd.0 (osd.0) 443 : cluster 0 10.4 scrub starts 2024-04-02T05:03:53.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:53 smithi069 bash[28904]: cluster 2024-04-02T05:03:52.681231+0000 osd.0 (osd.0) 444 : cluster 0 10.4 scrub ok 2024-04-02T05:03:53.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:53.633+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f6a601a4e60 con 0x7f6a60107620 2024-04-02T05:03:54.178 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:03:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:03:54.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.237+0000 7f6a567fc700 1 -- 172.21.15.69:0/3694514296 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v897) ==== 87+0+0 (secure 0 0 0) 0x7f6a58010040 con 0x7f6a60107620 2024-04-02T05:03:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 10 librados: watch_flush enter 2024-04-02T05:03:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 10 librados: watch_flush exit 2024-04-02T05:03:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 20 client.43889.objecter shutdown clearing up homeless session... 2024-04-02T05:03:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 10 client.43889.objecter successfully canceled tick 2024-04-02T05:03:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a4c06f5e0 msgr2=0x7f6a4c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a4c06f5e0 0x7f6a4c071aa0 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00bd30 tx=0x7f6a5c00b480 comp rx=0 tx=0).stop 2024-04-02T05:03:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a60107620 msgr2=0x7f6a6019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:03:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a60107620 0x7f6a6019c980 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800efb0 tx=0x7f6a5800c3a0 comp rx=0 tx=0).stop 2024-04-02T05:03:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 shutdown_connections 2024-04-02T05:03:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f6a4c06f5e0 0x7f6a4c071aa0 unknown :-1 s=CLOSED pgs=4028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a60107620 0x7f6a6019c980 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a6006a1d0 0x7f6a601980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 --2- 172.21.15.69:0/3694514296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6006ab80 0x7f6a601985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:03:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 >> 172.21.15.69:0/3694514296 conn(0x7f6a600771e0 msgr2=0x7f6a600ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:03:54.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 shutdown_connections 2024-04-02T05:03:54.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.241+0000 7f6a67296700 1 -- 172.21.15.69:0/3694514296 wait complete. 2024-04-02T05:03:54.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:03:54.245+0000 7f6a67296700 1 librados: shutdown 2024-04-02T05:03:54.250 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is set 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: cluster 2024-04-02T05:03:52.344429+0000 osd.7 (osd.7) 253 : cluster 0 6.15 scrub starts 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: cluster 2024-04-02T05:03:52.346736+0000 osd.7 (osd.7) 254 : cluster 0 6.15 scrub ok 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: cluster 2024-04-02T05:03:53.323075+0000 osd.3 (osd.3) 463 : cluster 0 4.11 scrub starts 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: cluster 2024-04-02T05:03:53.324739+0000 osd.3 (osd.3) 464 : cluster 0 4.11 scrub ok 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: cluster 2024-04-02T05:03:53.403832+0000 osd.1 (osd.1) 455 : cluster 0 2.a scrub starts 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: cluster 2024-04-02T05:03:53.405290+0000 osd.1 (osd.1) 456 : cluster 0 2.a scrub ok 2024-04-02T05:03:54.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: audit 2024-04-02T05:03:53.638352+0000 mon.c (mon.2) 1167 : audit 1 from='client.? 172.21.15.69:0/3694514296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: audit 2024-04-02T05:03:53.638898+0000 mon.a (mon.0) 4046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[21528]: audit 2024-04-02T05:03:53.782406+0000 mgr.y (mgr.14538) 778 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: cluster 2024-04-02T05:03:52.344429+0000 osd.7 (osd.7) 253 : cluster 0 6.15 scrub starts 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: cluster 2024-04-02T05:03:52.346736+0000 osd.7 (osd.7) 254 : cluster 0 6.15 scrub ok 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: cluster 2024-04-02T05:03:53.323075+0000 osd.3 (osd.3) 463 : cluster 0 4.11 scrub starts 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: cluster 2024-04-02T05:03:53.324739+0000 osd.3 (osd.3) 464 : cluster 0 4.11 scrub ok 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: cluster 2024-04-02T05:03:53.403832+0000 osd.1 (osd.1) 455 : cluster 0 2.a scrub starts 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: cluster 2024-04-02T05:03:53.405290+0000 osd.1 (osd.1) 456 : cluster 0 2.a scrub ok 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: audit 2024-04-02T05:03:53.638352+0000 mon.c (mon.2) 1167 : audit 1 from='client.? 172.21.15.69:0/3694514296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: audit 2024-04-02T05:03:53.638898+0000 mon.a (mon.0) 4046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:03:54.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:54 smithi069 bash[28904]: audit 2024-04-02T05:03:53.782406+0000 mgr.y (mgr.14538) 778 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: cluster 2024-04-02T05:03:52.344429+0000 osd.7 (osd.7) 253 : cluster 0 6.15 scrub starts 2024-04-02T05:03:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: cluster 2024-04-02T05:03:52.346736+0000 osd.7 (osd.7) 254 : cluster 0 6.15 scrub ok 2024-04-02T05:03:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: cluster 2024-04-02T05:03:53.323075+0000 osd.3 (osd.3) 463 : cluster 0 4.11 scrub starts 2024-04-02T05:03:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: cluster 2024-04-02T05:03:53.324739+0000 osd.3 (osd.3) 464 : cluster 0 4.11 scrub ok 2024-04-02T05:03:54.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: cluster 2024-04-02T05:03:53.403832+0000 osd.1 (osd.1) 455 : cluster 0 2.a scrub starts 2024-04-02T05:03:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: cluster 2024-04-02T05:03:53.405290+0000 osd.1 (osd.1) 456 : cluster 0 2.a scrub ok 2024-04-02T05:03:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: audit 2024-04-02T05:03:53.638352+0000 mon.c (mon.2) 1167 : audit 1 from='client.? 172.21.15.69:0/3694514296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:03:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: audit 2024-04-02T05:03:53.638898+0000 mon.a (mon.0) 4046 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-02T05:03:54.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:54 smithi098 bash[23790]: audit 2024-04-02T05:03:53.782406+0000 mgr.y (mgr.14538) 778 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:03:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: cluster 2024-04-02T05:03:54.019735+0000 osd.4 (osd.4) 442 : cluster 0 2.13 scrub starts 2024-04-02T05:03:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: cluster 2024-04-02T05:03:54.022509+0000 osd.4 (osd.4) 443 : cluster 0 2.13 scrub ok 2024-04-02T05:03:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: cluster 2024-04-02T05:03:54.175091+0000 mon.a (mon.0) 4047 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:03:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: audit 2024-04-02T05:03:54.231052+0000 mon.a (mon.0) 4048 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:03:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: cluster 2024-04-02T05:03:54.238738+0000 mon.a (mon.0) 4049 : cluster 0 osdmap e897: 8 total, 8 up, 7 in 2024-04-02T05:03:55.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: audit 2024-04-02T05:03:54.328214+0000 mon.b (mon.1) 325 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]: dispatch 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: audit 2024-04-02T05:03:54.329309+0000 mon.a (mon.0) 4050 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]: dispatch 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[21528]: cluster 2024-04-02T05:03:54.377612+0000 mgr.y (mgr.14538) 779 : cluster 0 pgmap v1371: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: cluster 2024-04-02T05:03:54.019735+0000 osd.4 (osd.4) 442 : cluster 0 2.13 scrub starts 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: cluster 2024-04-02T05:03:54.022509+0000 osd.4 (osd.4) 443 : cluster 0 2.13 scrub ok 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: cluster 2024-04-02T05:03:54.175091+0000 mon.a (mon.0) 4047 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: audit 2024-04-02T05:03:54.231052+0000 mon.a (mon.0) 4048 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: cluster 2024-04-02T05:03:54.238738+0000 mon.a (mon.0) 4049 : cluster 0 osdmap e897: 8 total, 8 up, 7 in 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: audit 2024-04-02T05:03:54.328214+0000 mon.b (mon.1) 325 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]: dispatch 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: audit 2024-04-02T05:03:54.329309+0000 mon.a (mon.0) 4050 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]: dispatch 2024-04-02T05:03:55.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:55 smithi069 bash[28904]: cluster 2024-04-02T05:03:54.377612+0000 mgr.y (mgr.14538) 779 : cluster 0 pgmap v1371: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T05:03:55.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-02T05:03:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: cluster 2024-04-02T05:03:54.019735+0000 osd.4 (osd.4) 442 : cluster 0 2.13 scrub starts 2024-04-02T05:03:55.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: cluster 2024-04-02T05:03:54.022509+0000 osd.4 (osd.4) 443 : cluster 0 2.13 scrub ok 2024-04-02T05:03:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: cluster 2024-04-02T05:03:54.175091+0000 mon.a (mon.0) 4047 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-02T05:03:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: audit 2024-04-02T05:03:54.231052+0000 mon.a (mon.0) 4048 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-02T05:03:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: cluster 2024-04-02T05:03:54.238738+0000 mon.a (mon.0) 4049 : cluster 0 osdmap e897: 8 total, 8 up, 7 in 2024-04-02T05:03:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: audit 2024-04-02T05:03:54.328214+0000 mon.b (mon.1) 325 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]: dispatch 2024-04-02T05:03:55.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: audit 2024-04-02T05:03:54.329309+0000 mon.a (mon.0) 4050 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]: dispatch 2024-04-02T05:03:55.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:55 smithi098 bash[23790]: cluster 2024-04-02T05:03:54.377612+0000 mgr.y (mgr.14538) 779 : cluster 0 pgmap v1371: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T05:03:55.827 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:03:55.827 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-02T05:03:55.827 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:03:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:56 smithi069 bash[21528]: audit 2024-04-02T05:03:55.240943+0000 mon.a (mon.0) 4051 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]': finished 2024-04-02T05:03:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:56 smithi069 bash[21528]: cluster 2024-04-02T05:03:55.244725+0000 mon.a (mon.0) 4052 : cluster 0 osdmap e898: 8 total, 8 up, 7 in 2024-04-02T05:03:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:56 smithi069 bash[28904]: audit 2024-04-02T05:03:55.240943+0000 mon.a (mon.0) 4051 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]': finished 2024-04-02T05:03:56.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:56 smithi069 bash[28904]: cluster 2024-04-02T05:03:55.244725+0000 mon.a (mon.0) 4052 : cluster 0 osdmap e898: 8 total, 8 up, 7 in 2024-04-02T05:03:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:56 smithi098 bash[23790]: audit 2024-04-02T05:03:55.240943+0000 mon.a (mon.0) 4051 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "10.e", "id": [4, 7]}]': finished 2024-04-02T05:03:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:56 smithi098 bash[23790]: cluster 2024-04-02T05:03:55.244725+0000 mon.a (mon.0) 4052 : cluster 0 osdmap e898: 8 total, 8 up, 7 in 2024-04-02T05:03:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:57 smithi069 bash[21528]: cluster 2024-04-02T05:03:56.251049+0000 mon.a (mon.0) 4053 : cluster 0 osdmap e899: 8 total, 8 up, 7 in 2024-04-02T05:03:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:57 smithi069 bash[21528]: cluster 2024-04-02T05:03:56.378340+0000 mgr.y (mgr.14538) 780 : cluster 0 pgmap v1374: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T05:03:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:57 smithi069 bash[28904]: cluster 2024-04-02T05:03:56.251049+0000 mon.a (mon.0) 4053 : cluster 0 osdmap e899: 8 total, 8 up, 7 in 2024-04-02T05:03:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:57 smithi069 bash[28904]: cluster 2024-04-02T05:03:56.378340+0000 mgr.y (mgr.14538) 780 : cluster 0 pgmap v1374: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T05:03:57.613 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-02T05:03:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:57 smithi098 bash[23790]: cluster 2024-04-02T05:03:56.251049+0000 mon.a (mon.0) 4053 : cluster 0 osdmap e899: 8 total, 8 up, 7 in 2024-04-02T05:03:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:57 smithi098 bash[23790]: cluster 2024-04-02T05:03:56.378340+0000 mgr.y (mgr.14538) 780 : cluster 0 pgmap v1374: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-02T05:03:59.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:03:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:03:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:03:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:03:59 smithi098 bash[23790]: cluster 2024-04-02T05:03:58.380090+0000 mgr.y (mgr.14538) 781 : cluster 0 pgmap v1375: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:03:59.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:03:59 smithi069 bash[21528]: cluster 2024-04-02T05:03:58.380090+0000 mgr.y (mgr.14538) 781 : cluster 0 pgmap v1375: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:03:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:03:59 smithi069 bash[28904]: cluster 2024-04-02T05:03:58.380090+0000 mgr.y (mgr.14538) 781 : cluster 0 pgmap v1375: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:04:00.275 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:01.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:00.381170+0000 mgr.y (mgr.14538) 782 : cluster 0 pgmap v1376: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 832 B/s rd, 0 op/s 2024-04-02T05:04:01.501 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:00.381170+0000 mgr.y (mgr.14538) 782 : cluster 0 pgmap v1376: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 832 B/s rd, 0 op/s 2024-04-02T05:04:01.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:00.381170+0000 mgr.y (mgr.14538) 782 : cluster 0 pgmap v1376: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 832 B/s rd, 0 op/s 2024-04-02T05:04:01.736 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-02T05:04:01.736 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-02T05:04:01.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd in 5 2024-04-02T05:04:02.682 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:03 smithi098 bash[23790]: cluster 2024-04-02T05:04:02.381960+0000 mgr.y (mgr.14538) 783 : cluster 0 pgmap v1377: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:03.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:03 smithi069 bash[21528]: cluster 2024-04-02T05:04:02.381960+0000 mgr.y (mgr.14538) 783 : cluster 0 pgmap v1377: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:03.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:03 smithi069 bash[28904]: cluster 2024-04-02T05:04:02.381960+0000 mgr.y (mgr.14538) 783 : cluster 0 pgmap v1377: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:03.948 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:04:03.948 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T05:04:03.948 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T05:04:03.948 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T05:04:03.948 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:04:04.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:04:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:04:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:04 smithi098 bash[23790]: audit 2024-04-02T05:04:03.787671+0000 mgr.y (mgr.14538) 784 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:04 smithi098 bash[23790]: audit 2024-04-02T05:04:04.284228+0000 mon.a (mon.0) 4054 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:04 smithi098 bash[23790]: audit 2024-04-02T05:04:04.287168+0000 mon.b (mon.1) 326 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:04 smithi069 bash[21528]: audit 2024-04-02T05:04:03.787671+0000 mgr.y (mgr.14538) 784 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:04 smithi069 bash[21528]: audit 2024-04-02T05:04:04.284228+0000 mon.a (mon.0) 4054 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:04 smithi069 bash[21528]: audit 2024-04-02T05:04:04.287168+0000 mon.b (mon.1) 326 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:04 smithi069 bash[28904]: audit 2024-04-02T05:04:03.787671+0000 mgr.y (mgr.14538) 784 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:04 smithi069 bash[28904]: audit 2024-04-02T05:04:04.284228+0000 mon.a (mon.0) 4054 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:04.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:04 smithi069 bash[28904]: audit 2024-04-02T05:04:04.287168+0000 mon.b (mon.1) 326 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:05.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-02T05:04:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:05 smithi098 bash[23790]: cluster 2024-04-02T05:04:04.383100+0000 mgr.y (mgr.14538) 785 : cluster 0 pgmap v1378: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-04-02T05:04:05.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:05 smithi069 bash[21528]: cluster 2024-04-02T05:04:04.383100+0000 mgr.y (mgr.14538) 785 : cluster 0 pgmap v1378: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-04-02T05:04:05.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:05 smithi069 bash[28904]: cluster 2024-04-02T05:04:04.383100+0000 mgr.y (mgr.14538) 785 : cluster 0 pgmap v1378: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 -- 172.21.15.69:0/2244661498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986006ab60 msgr2=0x7f986006af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 --2- 172.21.15.69:0/2244661498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986006ab60 0x7f986006af40 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f9850009920 tx=0x7f9850009c30 comp rx=0 tx=0).stop 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 -- 172.21.15.69:0/2244661498 shutdown_connections 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 --2- 172.21.15.69:0/2244661498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9860107610 0x7f986010f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 --2- 172.21.15.69:0/2244661498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986006ab60 0x7f986006af40 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 --2- 172.21.15.69:0/2244661498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006b510 0x7f9860106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 -- 172.21.15.69:0/2244661498 >> 172.21.15.69:0/2244661498 conn(0x7f9860077ea0 msgr2=0x7f98600782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:06.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 -- 172.21.15.69:0/2244661498 shutdown_connections 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.325+0000 7f9865c95700 1 -- 172.21.15.69:0/2244661498 wait complete. 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 librados: starting msgr at 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 librados: starting objecter 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 Processor -- start 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- start start 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 librados: setting wanted keys 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 librados: calling monclient init 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006ab60 0x7f98601981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 0x7f9860198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:06.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9860107610 0x7f986019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9860111b30 con 0x7f986006b510 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98601119b0 con 0x7f986006ab60 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9860111cb0 con 0x7f9860107610 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 0x7f9860198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9860107610 0x7f986019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 0x7f9860198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42858/0 (socket says 172.21.15.69:42858) 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 -- 172.21.15.69:0/222087158 learned_addr learned my addr 172.21.15.69:0/222087158 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985f7fe700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006ab60 0x7f98601981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 -- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9860107610 msgr2=0x7f986019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9860107610 0x7f986019caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 -- 172.21.15.69:0/222087158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006ab60 msgr2=0x7f98601981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006ab60 0x7f98601981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 -- 172.21.15.69:0/222087158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986019d1e0 con 0x7f986006b510 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985f7fe700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006ab60 0x7f98601981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985ffff700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9860107610 0x7f986019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985effd700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 0x7f9860198710 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f984800b4d0 tx=0x7f984800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 10 client.?.objecter ms_handle_connect 0x7f986006b510 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848015020 con 0x7f986006b510 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986019d4d0 con 0x7f986006b510 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f9848011cb0 con 0x7f986006b510 2024-04-02T05:04:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848010c20 con 0x7f986006b510 2024-04-02T05:04:06.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 10 client.33831.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:06.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98601a4d50 con 0x7f986006b510 2024-04-02T05:04:06.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 librados: init done 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f986004ff00 con 0x7f986006b510 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f9848018020 con 0x7f986006b510 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.329+0000 7f985cff9700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f984c06f5e0 0x7f984c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(899..899 src has 795..899) ==== 8899+0+0 (secure 0 0 0) 0x7f984808dcb0 con 0x7f986006b510 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985cff9700 10 client.33831.objecter ms_dispatch 0x7f9860001550 osd_map(899..899 src has 795..899) 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985f7fe700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f984c06f5e0 0x7f984c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985cff9700 3 client.33831.objecter handle_osd_map got epochs [899,899] > 0 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985cff9700 3 client.33831.objecter handle_osd_map decoding full epoch 899 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985cff9700 20 client.33831.objecter dump_active .. 0 homeless 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985cff9700 10 client.33831.objecter ms_handle_connect 0x7f984c06f5e0 2024-04-02T05:04:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.333+0000 7f985f7fe700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f984c06f5e0 0x7f984c071aa0 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f9850009680 tx=0x7f9850009f80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:06.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.337+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f984801f020 con 0x7f986006b510 2024-04-02T05:04:06.392 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:06.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:06.553+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f98601a4f80 con 0x7f986006b510 2024-04-02T05:04:07.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.449+0000 7f985cff9700 1 -- 172.21.15.69:0/222087158 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v900) ==== 91+0+0 (secure 0 0 0) 0x7f984805acf0 con 0x7f986006b510 2024-04-02T05:04:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 10 librados: watch_flush enter 2024-04-02T05:04:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 10 librados: watch_flush exit 2024-04-02T05:04:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 20 client.33831.objecter shutdown clearing up homeless session... 2024-04-02T05:04:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 10 client.33831.objecter successfully canceled tick 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f984c06f5e0 msgr2=0x7f984c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f984c06f5e0 0x7f984c071aa0 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f9850009680 tx=0x7f9850009f80 comp rx=0 tx=0).stop 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 msgr2=0x7f9860198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 0x7f9860198710 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f984800b4d0 tx=0x7f984800b7e0 comp rx=0 tx=0).stop 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 shutdown_connections 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f984c06f5e0 0x7f984c071aa0 unknown :-1 s=CLOSED pgs=4029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9860107610 0x7f986019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f986006ab60 0x7f98601981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:07.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 --2- 172.21.15.69:0/222087158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f986006b510 0x7f9860198710 secure :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f984800b4d0 tx=0x7f984800b7e0 comp rx=0 tx=0).stop 2024-04-02T05:04:07.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.453+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 >> 172.21.15.69:0/222087158 conn(0x7f9860077ea0 msgr2=0x7f98600ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:07.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.457+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 shutdown_connections 2024-04-02T05:04:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.457+0000 7f9865c95700 1 -- 172.21.15.69:0/222087158 wait complete. 2024-04-02T05:04:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:07.457+0000 7f9865c95700 1 librados: shutdown 2024-04-02T05:04:07.463 INFO:teuthology.orchestra.run.smithi069.stderr:noscrub is unset 2024-04-02T05:04:07.477 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:07 smithi069 bash[21528]: cluster 2024-04-02T05:04:06.383531+0000 mgr.y (mgr.14538) 786 : cluster 0 pgmap v1379: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-04-02T05:04:07.477 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:07 smithi069 bash[21528]: audit 2024-04-02T05:04:06.560088+0000 mon.a (mon.0) 4055 : audit 1 from='client.? 172.21.15.69:0/222087158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:04:07.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:07 smithi069 bash[28904]: cluster 2024-04-02T05:04:06.383531+0000 mgr.y (mgr.14538) 786 : cluster 0 pgmap v1379: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-04-02T05:04:07.477 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:07 smithi069 bash[28904]: audit 2024-04-02T05:04:06.560088+0000 mon.a (mon.0) 4055 : audit 1 from='client.? 172.21.15.69:0/222087158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:04:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:07 smithi098 bash[23790]: cluster 2024-04-02T05:04:06.383531+0000 mgr.y (mgr.14538) 786 : cluster 0 pgmap v1379: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-04-02T05:04:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:07 smithi098 bash[23790]: audit 2024-04-02T05:04:06.560088+0000 mon.a (mon.0) 4055 : audit 1 from='client.? 172.21.15.69:0/222087158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-02T05:04:08.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-02T05:04:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:08 smithi098 bash[23790]: cluster 2024-04-02T05:04:07.444910+0000 mon.a (mon.0) 4056 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-04-02T05:04:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:08 smithi098 bash[23790]: audit 2024-04-02T05:04:07.453122+0000 mon.a (mon.0) 4057 : audit 1 from='client.? 172.21.15.69:0/222087158' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:04:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:08 smithi098 bash[23790]: cluster 2024-04-02T05:04:07.458761+0000 mon.a (mon.0) 4058 : cluster 0 osdmap e900: 8 total, 8 up, 7 in 2024-04-02T05:04:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[21528]: cluster 2024-04-02T05:04:07.444910+0000 mon.a (mon.0) 4056 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-04-02T05:04:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[21528]: audit 2024-04-02T05:04:07.453122+0000 mon.a (mon.0) 4057 : audit 1 from='client.? 172.21.15.69:0/222087158' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:04:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[21528]: cluster 2024-04-02T05:04:07.458761+0000 mon.a (mon.0) 4058 : cluster 0 osdmap e900: 8 total, 8 up, 7 in 2024-04-02T05:04:08.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[28904]: cluster 2024-04-02T05:04:07.444910+0000 mon.a (mon.0) 4056 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-04-02T05:04:08.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[28904]: audit 2024-04-02T05:04:07.453122+0000 mon.a (mon.0) 4057 : audit 1 from='client.? 172.21.15.69:0/222087158' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-02T05:04:08.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[28904]: cluster 2024-04-02T05:04:07.458761+0000 mon.a (mon.0) 4058 : cluster 0 osdmap e900: 8 total, 8 up, 7 in 2024-04-02T05:04:09.032 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:04:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:04:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:04:09.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:09 smithi069 bash[21528]: cluster 2024-04-02T05:04:08.385252+0000 mgr.y (mgr.14538) 787 : cluster 0 pgmap v1381: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-04-02T05:04:09.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:09 smithi069 bash[28904]: cluster 2024-04-02T05:04:08.385252+0000 mgr.y (mgr.14538) 787 : cluster 0 pgmap v1381: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-04-02T05:04:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:09 smithi098 bash[23790]: cluster 2024-04-02T05:04:08.385252+0000 mgr.y (mgr.14538) 787 : cluster 0 pgmap v1381: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 -- 172.21.15.69:0/1109967900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7058104e50 msgr2=0x7f7058111310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 --2- 172.21.15.69:0/1109967900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7058104e50 0x7f7058111310 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f705000b0a0 tx=0x7f705000b3b0 comp rx=0 tx=0).stop 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 -- 172.21.15.69:0/1109967900 shutdown_connections 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 --2- 172.21.15.69:0/1109967900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7058104e50 0x7f7058111310 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 --2- 172.21.15.69:0/1109967900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705810a540 0x7f705810a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 --2- 172.21.15.69:0/1109967900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7058104490 0x7f7058104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 -- 172.21.15.69:0/1109967900 >> 172.21.15.69:0/1109967900 conn(0x7f70580fff60 msgr2=0x7f7058102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 -- 172.21.15.69:0/1109967900 shutdown_connections 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 -- 172.21.15.69:0/1109967900 wait complete. 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 librados: starting msgr at 2024-04-02T05:04:09.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 librados: starting objecter 2024-04-02T05:04:09.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 Processor -- start 2024-04-02T05:04:09.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 -- start start 2024-04-02T05:04:09.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 librados: setting wanted keys 2024-04-02T05:04:09.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.857+0000 7f705d297700 1 librados: calling monclient init 2024-04-02T05:04:09.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 0x7f705819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:09.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7058104e50 0x7f705819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f705810a540 0x7f705819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7058113d10 con 0x7f7058104490 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7058113b90 con 0x7f7058104e50 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f7058113e90 con 0x7f705810a540 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 0x7f705819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 0x7f705819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57586/0 (socket says 172.21.15.69:57586) 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f70577fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f705810a540 0x7f705819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 -- 172.21.15.69:0/894227585 learned_addr learned my addr 172.21.15.69:0/894227585 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 -- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f705810a540 msgr2=0x7f705819ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f705810a540 0x7f705819ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 -- 172.21.15.69:0/894227585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7058104e50 msgr2=0x7f705819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7058104e50 0x7f705819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 -- 172.21.15.69:0/894227585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705819f1b0 con 0x7f7058104490 2024-04-02T05:04:09.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f70577fe700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f705810a540 0x7f705819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f7056ffd700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 0x7f705819a1a0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f704800eb30 tx=0x7f704800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 10 client.?.objecter ms_handle_connect 0x7f7058104490 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704800cc90 con 0x7f7058104490 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f704800ce30 con 0x7f7058104490 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705819f4a0 con 0x7f7058104490 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048003df0 con 0x7f7058104490 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 10 client.33837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70581a6d20 con 0x7f7058104490 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 librados: init done 2024-04-02T05:04:09.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f705804ff00 con 0x7f7058104490 2024-04-02T05:04:09.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f7048003f90 con 0x7f7058104490 2024-04-02T05:04:09.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f704406f5e0 0x7f7044071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:09.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(900..900 src has 795..900) ==== 8899+0+0 (secure 0 0 0) 0x7f704808e8b0 con 0x7f7058104490 2024-04-02T05:04:09.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 10 client.33837.objecter ms_dispatch 0x7f7058001550 osd_map(900..900 src has 795..900) 2024-04-02T05:04:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 3 client.33837.objecter handle_osd_map got epochs [900,900] > 0 2024-04-02T05:04:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 3 client.33837.objecter handle_osd_map decoding full epoch 900 2024-04-02T05:04:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f70567fc700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f704406f5e0 0x7f7044071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.861+0000 7f704ffff700 20 client.33837.objecter dump_active .. 0 homeless 2024-04-02T05:04:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.865+0000 7f704ffff700 10 client.33837.objecter ms_handle_connect 0x7f704406f5e0 2024-04-02T05:04:09.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.865+0000 7f70567fc700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f704406f5e0 0x7f7044071aa0 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7f7058103630 tx=0x7f7040006d20 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:09.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:09.865+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f704805b9f0 con 0x7f7058104490 2024-04-02T05:04:10.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.089+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd in", "ids": ["5"]} v 0) -- 0x7f705819b500 con 0x7f7058104490 2024-04-02T05:04:10.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.153+0000 7f704ffff700 1 -- 172.21.15.69:0/894227585 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["5"]}]=0 marked in osd.5. v901) ==== 85+0+0 (secure 0 0 0) 0x7f705819b500 con 0x7f7058104490 2024-04-02T05:04:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 10 librados: watch_flush enter 2024-04-02T05:04:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 10 librados: watch_flush exit 2024-04-02T05:04:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 20 client.33837.objecter shutdown clearing up homeless session... 2024-04-02T05:04:10.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 10 client.33837.objecter successfully canceled tick 2024-04-02T05:04:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f704406f5e0 msgr2=0x7f7044071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f704406f5e0 0x7f7044071aa0 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7f7058103630 tx=0x7f7040006d20 comp rx=0 tx=0).stop 2024-04-02T05:04:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 msgr2=0x7f705819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.157+0000 7f705d297700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 0x7f705819a1a0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f704800eb30 tx=0x7f704800ee40 comp rx=0 tx=0).stop 2024-04-02T05:04:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 shutdown_connections 2024-04-02T05:04:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f704406f5e0 0x7f7044071aa0 secure :-1 s=CLOSED pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7f7058103630 tx=0x7f7040006d20 comp rx=0 tx=0).stop 2024-04-02T05:04:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f705810a540 0x7f705819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7058104490 0x7f705819a1a0 secure :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f704800eb30 tx=0x7f704800ee40 comp rx=0 tx=0).stop 2024-04-02T05:04:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 --2- 172.21.15.69:0/894227585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7058104e50 0x7f705819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 >> 172.21.15.69:0/894227585 conn(0x7f70580fff60 msgr2=0x7f705810b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:10.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 shutdown_connections 2024-04-02T05:04:10.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.161+0000 7f705d297700 1 -- 172.21.15.69:0/894227585 wait complete. 2024-04-02T05:04:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:10.185+0000 7f705d297700 1 librados: shutdown 2024-04-02T05:04:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:marked in osd.5. 2024-04-02T05:04:10.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[21528]: debug 2024-04-02T05:04:10.089+0000 7f02343d7700 -1 mon.a@0(leader).osd e900 definitely_dead 0 2024-04-02T05:04:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:10 smithi098 bash[23790]: cluster 2024-04-02T05:04:09.521733+0000 osd.0 (osd.0) 445 : cluster 0 6.18 scrub starts 2024-04-02T05:04:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:10 smithi098 bash[23790]: cluster 2024-04-02T05:04:09.524219+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub ok 2024-04-02T05:04:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:10 smithi098 bash[23790]: audit 2024-04-02T05:04:10.093041+0000 mon.a (mon.0) 4059 : audit 0 from='client.? 172.21.15.69:0/894227585' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-02T05:04:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:10 smithi098 bash[23790]: audit 2024-04-02T05:04:10.156623+0000 mon.a (mon.0) 4060 : audit 1 from='client.? 172.21.15.69:0/894227585' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-02T05:04:10.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:10 smithi098 bash[23790]: cluster 2024-04-02T05:04:10.161909+0000 mon.a (mon.0) 4061 : cluster 0 osdmap e901: 8 total, 8 up, 8 in 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[21528]: cluster 2024-04-02T05:04:09.521733+0000 osd.0 (osd.0) 445 : cluster 0 6.18 scrub starts 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[21528]: cluster 2024-04-02T05:04:09.524219+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub ok 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[21528]: audit 2024-04-02T05:04:10.093041+0000 mon.a (mon.0) 4059 : audit 0 from='client.? 172.21.15.69:0/894227585' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[21528]: audit 2024-04-02T05:04:10.156623+0000 mon.a (mon.0) 4060 : audit 1 from='client.? 172.21.15.69:0/894227585' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[21528]: cluster 2024-04-02T05:04:10.161909+0000 mon.a (mon.0) 4061 : cluster 0 osdmap e901: 8 total, 8 up, 8 in 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[28904]: cluster 2024-04-02T05:04:09.521733+0000 osd.0 (osd.0) 445 : cluster 0 6.18 scrub starts 2024-04-02T05:04:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[28904]: cluster 2024-04-02T05:04:09.524219+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub ok 2024-04-02T05:04:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[28904]: audit 2024-04-02T05:04:10.093041+0000 mon.a (mon.0) 4059 : audit 0 from='client.? 172.21.15.69:0/894227585' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-02T05:04:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[28904]: audit 2024-04-02T05:04:10.156623+0000 mon.a (mon.0) 4060 : audit 1 from='client.? 172.21.15.69:0/894227585' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-02T05:04:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:10 smithi069 bash[28904]: cluster 2024-04-02T05:04:10.161909+0000 mon.a (mon.0) 4061 : cluster 0 osdmap e901: 8 total, 8 up, 8 in 2024-04-02T05:04:10.993 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-02T05:04:10.993 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-02T05:04:10.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 21 2024-04-02T05:04:11.408 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:11 smithi098 bash[23790]: cluster 2024-04-02T05:04:09.966933+0000 osd.4 (osd.4) 444 : cluster 0 6.17 scrub starts 2024-04-02T05:04:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:11 smithi098 bash[23790]: cluster 2024-04-02T05:04:09.970282+0000 osd.4 (osd.4) 445 : cluster 0 6.17 scrub ok 2024-04-02T05:04:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:11 smithi098 bash[23790]: cluster 2024-04-02T05:04:10.385909+0000 mgr.y (mgr.14538) 788 : cluster 0 pgmap v1383: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-02T05:04:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:11 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.163255+0000 mon.a (mon.0) 4062 : cluster 0 osdmap e902: 8 total, 8 up, 8 in 2024-04-02T05:04:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[21528]: cluster 2024-04-02T05:04:09.966933+0000 osd.4 (osd.4) 444 : cluster 0 6.17 scrub starts 2024-04-02T05:04:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[21528]: cluster 2024-04-02T05:04:09.970282+0000 osd.4 (osd.4) 445 : cluster 0 6.17 scrub ok 2024-04-02T05:04:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[21528]: cluster 2024-04-02T05:04:10.385909+0000 mgr.y (mgr.14538) 788 : cluster 0 pgmap v1383: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-02T05:04:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.163255+0000 mon.a (mon.0) 4062 : cluster 0 osdmap e902: 8 total, 8 up, 8 in 2024-04-02T05:04:11.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[28904]: cluster 2024-04-02T05:04:09.966933+0000 osd.4 (osd.4) 444 : cluster 0 6.17 scrub starts 2024-04-02T05:04:11.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[28904]: cluster 2024-04-02T05:04:09.970282+0000 osd.4 (osd.4) 445 : cluster 0 6.17 scrub ok 2024-04-02T05:04:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[28904]: cluster 2024-04-02T05:04:10.385909+0000 mgr.y (mgr.14538) 788 : cluster 0 pgmap v1383: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-02T05:04:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:11 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.163255+0000 mon.a (mon.0) 4062 : cluster 0 osdmap e902: 8 total, 8 up, 8 in 2024-04-02T05:04:12.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.501506+0000 osd.3 (osd.3) 465 : cluster 0 4.19 scrub starts 2024-04-02T05:04:12.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.503111+0000 osd.3 (osd.3) 466 : cluster 0 4.19 scrub ok 2024-04-02T05:04:12.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.547967+0000 osd.7 (osd.7) 255 : cluster 0 2.8 scrub starts 2024-04-02T05:04:12.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.550786+0000 osd.7 (osd.7) 256 : cluster 0 2.8 scrub ok 2024-04-02T05:04:12.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.584713+0000 osd.0 (osd.0) 447 : cluster 0 10.1e scrub starts 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.589564+0000 osd.0 (osd.0) 448 : cluster 0 10.1e scrub ok 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.697236+0000 osd.5 (osd.5) 265 : cluster 0 2.12 scrub starts 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.699594+0000 osd.5 (osd.5) 266 : cluster 0 2.12 scrub ok 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.740767+0000 osd.5 (osd.5) 267 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:11.741118+0000 osd.5 (osd.5) 268 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[21528]: cluster 2024-04-02T05:04:12.169749+0000 mon.a (mon.0) 4063 : cluster 0 osdmap e903: 8 total, 8 up, 8 in 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.501506+0000 osd.3 (osd.3) 465 : cluster 0 4.19 scrub starts 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.503111+0000 osd.3 (osd.3) 466 : cluster 0 4.19 scrub ok 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.547967+0000 osd.7 (osd.7) 255 : cluster 0 2.8 scrub starts 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.550786+0000 osd.7 (osd.7) 256 : cluster 0 2.8 scrub ok 2024-04-02T05:04:12.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.584713+0000 osd.0 (osd.0) 447 : cluster 0 10.1e scrub starts 2024-04-02T05:04:12.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.589564+0000 osd.0 (osd.0) 448 : cluster 0 10.1e scrub ok 2024-04-02T05:04:12.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.697236+0000 osd.5 (osd.5) 265 : cluster 0 2.12 scrub starts 2024-04-02T05:04:12.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.699594+0000 osd.5 (osd.5) 266 : cluster 0 2.12 scrub ok 2024-04-02T05:04:12.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.740767+0000 osd.5 (osd.5) 267 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:12.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:11.741118+0000 osd.5 (osd.5) 268 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:12.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:12 smithi069 bash[28904]: cluster 2024-04-02T05:04:12.169749+0000 mon.a (mon.0) 4063 : cluster 0 osdmap e903: 8 total, 8 up, 8 in 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.501506+0000 osd.3 (osd.3) 465 : cluster 0 4.19 scrub starts 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.503111+0000 osd.3 (osd.3) 466 : cluster 0 4.19 scrub ok 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.547967+0000 osd.7 (osd.7) 255 : cluster 0 2.8 scrub starts 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.550786+0000 osd.7 (osd.7) 256 : cluster 0 2.8 scrub ok 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.584713+0000 osd.0 (osd.0) 447 : cluster 0 10.1e scrub starts 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.589564+0000 osd.0 (osd.0) 448 : cluster 0 10.1e scrub ok 2024-04-02T05:04:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.697236+0000 osd.5 (osd.5) 265 : cluster 0 2.12 scrub starts 2024-04-02T05:04:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.699594+0000 osd.5 (osd.5) 266 : cluster 0 2.12 scrub ok 2024-04-02T05:04:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.740767+0000 osd.5 (osd.5) 267 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:11.741118+0000 osd.5 (osd.5) 268 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:12.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:12 smithi098 bash[23790]: cluster 2024-04-02T05:04:12.169749+0000 mon.a (mon.0) 4063 : cluster 0 osdmap e903: 8 total, 8 up, 8 in 2024-04-02T05:04:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[23790]: cluster 2024-04-02T05:04:12.386651+0000 mgr.y (mgr.14538) 789 : cluster 0 pgmap v1386: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-02T05:04:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[23790]: cluster 2024-04-02T05:04:12.440272+0000 osd.1 (osd.1) 457 : cluster 0 10.13 scrub starts 2024-04-02T05:04:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[23790]: cluster 2024-04-02T05:04:12.445291+0000 osd.1 (osd.1) 458 : cluster 0 10.13 scrub ok 2024-04-02T05:04:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[23790]: cluster 2024-04-02T05:04:12.536908+0000 osd.7 (osd.7) 257 : cluster 0 2.5 scrub starts 2024-04-02T05:04:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[23790]: cluster 2024-04-02T05:04:12.539263+0000 osd.7 (osd.7) 258 : cluster 0 2.5 scrub ok 2024-04-02T05:04:13.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[23790]: cluster 2024-04-02T05:04:13.171340+0000 mon.a (mon.0) 4064 : cluster 0 osdmap e904: 8 total, 8 up, 8 in 2024-04-02T05:04:13.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[28904]: cluster 2024-04-02T05:04:12.386651+0000 mgr.y (mgr.14538) 789 : cluster 0 pgmap v1386: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-02T05:04:13.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[28904]: cluster 2024-04-02T05:04:12.440272+0000 osd.1 (osd.1) 457 : cluster 0 10.13 scrub starts 2024-04-02T05:04:13.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[28904]: cluster 2024-04-02T05:04:12.445291+0000 osd.1 (osd.1) 458 : cluster 0 10.13 scrub ok 2024-04-02T05:04:13.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[28904]: cluster 2024-04-02T05:04:12.536908+0000 osd.7 (osd.7) 257 : cluster 0 2.5 scrub starts 2024-04-02T05:04:13.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[28904]: cluster 2024-04-02T05:04:12.539263+0000 osd.7 (osd.7) 258 : cluster 0 2.5 scrub ok 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[28904]: cluster 2024-04-02T05:04:13.171340+0000 mon.a (mon.0) 4064 : cluster 0 osdmap e904: 8 total, 8 up, 8 in 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[21528]: cluster 2024-04-02T05:04:12.386651+0000 mgr.y (mgr.14538) 789 : cluster 0 pgmap v1386: 184 pgs: 184 active+clean; 582 KiB data, 756 MiB used, 625 GiB / 626 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[21528]: cluster 2024-04-02T05:04:12.440272+0000 osd.1 (osd.1) 457 : cluster 0 10.13 scrub starts 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[21528]: cluster 2024-04-02T05:04:12.445291+0000 osd.1 (osd.1) 458 : cluster 0 10.13 scrub ok 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[21528]: cluster 2024-04-02T05:04:12.536908+0000 osd.7 (osd.7) 257 : cluster 0 2.5 scrub starts 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[21528]: cluster 2024-04-02T05:04:12.539263+0000 osd.7 (osd.7) 258 : cluster 0 2.5 scrub ok 2024-04-02T05:04:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:13 smithi069 bash[21528]: cluster 2024-04-02T05:04:13.171340+0000 mon.a (mon.0) 4064 : cluster 0 osdmap e904: 8 total, 8 up, 8 in 2024-04-02T05:04:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:14.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:04:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:04:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:14 smithi098 bash[23790]: cluster 2024-04-02T05:04:13.461529+0000 osd.1 (osd.1) 459 : cluster 0 4.d scrub starts 2024-04-02T05:04:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:14 smithi098 bash[23790]: cluster 2024-04-02T05:04:13.463601+0000 osd.1 (osd.1) 460 : cluster 0 4.d scrub ok 2024-04-02T05:04:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:14 smithi098 bash[23790]: audit 2024-04-02T05:04:13.796815+0000 mgr.y (mgr.14538) 790 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:14 smithi098 bash[23790]: cluster 2024-04-02T05:04:14.173975+0000 mon.a (mon.0) 4065 : cluster 0 osdmap e905: 8 total, 8 up, 8 in 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[21528]: cluster 2024-04-02T05:04:13.461529+0000 osd.1 (osd.1) 459 : cluster 0 4.d scrub starts 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[21528]: cluster 2024-04-02T05:04:13.463601+0000 osd.1 (osd.1) 460 : cluster 0 4.d scrub ok 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[21528]: audit 2024-04-02T05:04:13.796815+0000 mgr.y (mgr.14538) 790 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[21528]: cluster 2024-04-02T05:04:14.173975+0000 mon.a (mon.0) 4065 : cluster 0 osdmap e905: 8 total, 8 up, 8 in 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[28904]: cluster 2024-04-02T05:04:13.461529+0000 osd.1 (osd.1) 459 : cluster 0 4.d scrub starts 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[28904]: cluster 2024-04-02T05:04:13.463601+0000 osd.1 (osd.1) 460 : cluster 0 4.d scrub ok 2024-04-02T05:04:14.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[28904]: audit 2024-04-02T05:04:13.796815+0000 mgr.y (mgr.14538) 790 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:14 smithi069 bash[28904]: cluster 2024-04-02T05:04:14.173975+0000 mon.a (mon.0) 4065 : cluster 0 osdmap e905: 8 total, 8 up, 8 in 2024-04-02T05:04:15.154 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:04:15.154 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T05:04:15.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.1.0:1552 322.6 e686/678 322:602f83fe:::foo:head v 686'1, mlcod=0'0)", 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.186165+0000", 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 270.965368474, 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004381869, 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186165+0000", 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186165+0000", 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186167+0000", 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3340000000000001e-06 2024-04-02T05:04:15.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186177+0000", 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7710000000000001e-06 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186197+0000", 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9993e-05 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186207+0000", 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0526000000000001e-05 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186382+0000", 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017489799999999999 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.186401+0000", 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9358e-05 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.190468+0000", 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040667569999999998 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.190547+0000", 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9232000000000003e-05 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.159 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.16773.0:4 322.6 e686/678 322:602f83fe:::foo:head v 686'2, mlcod=0'0)", 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.187774+0000", 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 270.96376023699997, 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0029527279999999999, 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187774+0000", 2024-04-02T05:04:15.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187774+0000", 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187774+0000", 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3200000000000004e-07 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187782+0000", 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9270000000000005e-06 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187799+0000", 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7096e-05 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187819+0000", 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9870000000000001e-05 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187899+0000", 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9246999999999995e-05 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.187919+0000", 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0088000000000001e-05 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.190692+0000", 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002773649 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.190726+0000", 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4119000000000003e-05 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.1.0:1555 322.6 e686/678 322:602f83fe:::foo:head v 686'3, mlcod=686'1)", 2024-04-02T05:04:15.164 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.204686+0000", 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 270.94684809900002, 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.020012549000000001, 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204686+0000", 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204686+0000", 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204687+0000", 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1689999999999999e-06 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204697+0000", 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0360000000000001e-05 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204719+0000", 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1268e-05 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204729+0000", 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0200000000000001e-05 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204809+0000", 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0628000000000001e-05 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204830+0000", 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0832999999999999e-05 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.224637+0000", 2024-04-02T05:04:15.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019806391999999999 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.224698+0000", 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1699000000000007e-05 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.169 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:476 328.16 328:6f126094:test-rados-api-smithi069-72349-121::bar:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:50.086184+0000", 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 205.065350201, 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0076562820000000004, 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:15.170 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 476 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086184+0000", 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086184+0000", 2024-04-02T05:04:15.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086185+0000", 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.331e-06 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086191+0000", 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8819999999999998e-06 2024-04-02T05:04:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086202+0000", 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.133e-05 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086214+0000", 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1877e-05 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086285+0000", 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0946999999999997e-05 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.086529+0000", 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000243661 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.089324+0000", 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0027950269999999998 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.089423+0000", 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9139000000000001e-05 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.093840+0000", 2024-04-02T05:04:15.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044170879999999996 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:580 328.0 328:00000000::::head [pgnls start_epoch 742] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e742)", 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:52.117229+0000", 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 203.03430453300001, 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086072419999999993, 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.176 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 580 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117229+0000", 2024-04-02T05:04:15.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117229+0000", 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117232+0000", 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7970000000000001e-06 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117246+0000", 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4266e-05 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117276+0000", 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9666e-05 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117303+0000", 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7195999999999998e-05 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117474+0000", 2024-04-02T05:04:15.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017082100000000001 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.117492+0000", 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7952999999999999e-05 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.125400+0000", 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079080130000000002 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.125495+0000", 2024-04-02T05:04:15.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4924000000000001e-05 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:52.125837+0000", 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000341606 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(323.0s1 745/698 ECSubWrite(tid=1573, reqid=client.32028.0:646, at_version=745'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:55.095498+0000", 2024-04-02T05:04:15.182 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 200.05603594900001, 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012086728999999999, 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.095498+0000", 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.095498+0000", 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.095499+0000", 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0699999999999996e-07 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.095550+0000", 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1544e-05 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.095573+0000", 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3130999999999999e-05 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.095598+0000", 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4304999999999999e-05 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.096730+0000", 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0011323800000000001 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.186 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.096748+0000", 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8453000000000001e-05 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.107504+0000", 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010755430999999999 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.107585+0000", 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0778000000000005e-05 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.187 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3806 4.d 4:b4812045:::notify.4:head [watch ping cookie 94452570230016 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e745)", 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:00.104119+0000", 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 195.047414876, 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0030860449999999999, 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.188 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3806 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.104119+0000", 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.104119+0000", 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.104121+0000", 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2330000000000001e-06 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.104131+0000", 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2620000000000004e-06 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.104160+0000", 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9391e-05 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.104175+0000", 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5175e-05 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.107064+0000", 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028890959999999999 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.107129+0000", 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5227000000000004e-05 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.107205+0000", 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5661000000000006e-05 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:734 329.12 329:48000000::::head [pgnls start_epoch 750] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e750)", 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:00.189854+0000", 2024-04-02T05:04:15.192 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 194.96167984499999, 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0032450360000000002, 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 734 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189854+0000", 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189854+0000", 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189857+0000", 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5500000000000001e-06 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189867+0000", 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0322e-05 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189887+0000", 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0191999999999999e-05 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189896+0000", 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9199999999999993e-06 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189973+0000", 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7002000000000006e-05 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.189980+0000", 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0140000000000001e-06 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.192955+0000", 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002974867 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.193001+0000", 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6190999999999999e-05 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:00.193099+0000", 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7978000000000003e-05 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:1732 335.4 335:20000000::::head [pgnls start_epoch 807] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e807)", 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:01:59.122600+0000", 2024-04-02T05:04:15.197 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 136.02893414600001, 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067839040000000003, 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1732 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122600+0000", 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122600+0000", 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122603+0000", 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7630000000000001e-06 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122615+0000", 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2819999999999999e-05 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122647+0000", 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1628999999999998e-05 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122672+0000", 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4885999999999999e-05 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122796+0000", 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012382499999999999 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.122806+0000", 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0716e-05 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.129124+0000", 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063176789999999997 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.129219+0000", 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4518999999999994e-05 2024-04-02T05:04:15.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:01:59.129384+0000", 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000165067 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.3.0:1093 336.14 e813/809 336:2c2c3553:test-rados-api-smithi069-72349-137::foo:head v 813'1, mlcod=0'0)", 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.106249+0000", 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 130.045285006, 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052124739999999999, 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.202 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106249+0000", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106249+0000", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106249+0000", 2024-04-02T05:04:15.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5100000000000001e-07 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106256+0000", 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2080000000000003e-06 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106275+0000", 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.861e-05 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106286+0000", 2024-04-02T05:04:15.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0815999999999999e-05 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106378+0000", 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2155000000000001e-05 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106413+0000", 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5423000000000003e-05 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.111386+0000", 2024-04-02T05:04:15.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049729450000000003 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.111461+0000", 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4966e-05 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.32028.0:1794 336.14 e813/809 336:2c2c3553:test-rados-api-smithi069-72349-137::foo:head v 813'2, mlcod=0'0)", 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.108617+0000", 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 130.042917131, 2024-04-02T05:04:15.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0098411320000000007, 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.108617+0000", 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.108617+0000", 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.108619+0000", 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.934e-06 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.112079+0000", 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034605579999999999 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.112108+0000", 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8668999999999999e-05 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.112123+0000", 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5204999999999999e-05 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.112253+0000", 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013010499999999999 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.112285+0000", 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1282000000000003e-05 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.118405+0000", 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061207479999999996 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.118458+0000", 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2630999999999998e-05 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(323.4s2 813/698 ECSubWrite(tid=1285, reqid=osd.3.29:94, at_version=813'10, trim_to=0'0, roll_forward_to=784'9))", 2024-04-02T05:04:15.211 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.145009+0000", 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 130.00652485099999, 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.024935654000000002, 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.145009+0000", 2024-04-02T05:04:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.145009+0000", 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.145010+0000", 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3399999999999998e-07 2024-04-02T05:04:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.146047+0000", 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010373439999999999 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.146068+0000", 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0577e-05 2024-04-02T05:04:15.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.146081+0000", 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3553000000000001e-05 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.146202+0000", 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000120272 2024-04-02T05:04:15.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.146216+0000", 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4406000000000001e-05 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.169870+0000", 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023654082 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.169945+0000", 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4585999999999994e-05 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3938 4.d 4:b4812045:::notify.4:head [watch ping cookie 94452570230016 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e808)", 2024-04-02T05:04:15.217 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.152170+0000", 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 129.999363778, 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0045220809999999998, 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3938 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.218 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.152170+0000", 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.152170+0000", 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.152173+0000", 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6079999999999998e-06 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.152183+0000", 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0575e-05 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.152209+0000", 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6072e-05 2024-04-02T05:04:15.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.152224+0000", 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4368e-05 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.156440+0000", 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042159060000000002 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.156609+0000", 2024-04-02T05:04:15.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016922000000000001 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.156692+0000", 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3331999999999994e-05 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:2064 338.c 338:30000000::::head [pgnls start_epoch 836] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e836)", 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:37.406730+0000", 2024-04-02T05:04:15.222 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 97.744804350999999, 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010247143, 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2064 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.406730+0000", 2024-04-02T05:04:15.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.406730+0000", 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.406734+0000", 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2259999999999999e-06 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.406754+0000", 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0423000000000001e-05 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.406806+0000", 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1484999999999999e-05 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.406835+0000", 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9805999999999999e-05 2024-04-02T05:04:15.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.407029+0000", 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019348600000000001 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.407047+0000", 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7717e-05 2024-04-02T05:04:15.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.416652+0000", 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0096052110000000007 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.416746+0000", 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3759999999999997e-05 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:37.416977+0000", 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000231029 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4009 4.d 4:b4812045:::notify.4:head [watch ping cookie 94452570230016 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:40.180095+0000", 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 94.971438585000001, 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072534280000000001, 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.228 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4009 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180095+0000", 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180095+0000", 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180098+0000", 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5009999999999999e-06 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180113+0000", 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5036e-05 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.186973+0000", 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0068604290000000004 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.186988+0000", 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4625e-05 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.187097+0000", 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000109348 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.187195+0000", 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7619000000000002e-05 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.187349+0000", 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015386999999999999 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(323.0s1 839/698 ECSubWrite(tid=1596, reqid=client.32028.0:2125, at_version=0'0, trim_to=0'0, roll_forward_to=839'3))", 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:40.194261+0000", 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 94.957272806999995, 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0036173139999999999, 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194261+0000", 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194261+0000", 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194261+0000", 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3099999999999999e-07 2024-04-02T05:04:15.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194279+0000", 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8070000000000001e-05 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194291+0000", 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1955e-05 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.194307+0000", 2024-04-02T05:04:15.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5199e-05 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.195918+0000", 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016112819999999999 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.195930+0000", 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2541000000000001e-05 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.197842+0000", 2024-04-02T05:04:15.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001911077 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.197878+0000", 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6859000000000001e-05 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4020 4.d 4:b4812045:::notify.4:head [watch ping cookie 94452570230016 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e841)", 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:45.185181+0000", 2024-04-02T05:04:15.236 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 89.966352985, 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041924880000000003, 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4020 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.185181+0000", 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.185181+0000", 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.185182+0000", 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3909999999999999e-06 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.185190+0000", 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6580000000000002e-06 2024-04-02T05:04:15.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.185204+0000", 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4394e-05 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.185221+0000", 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6566999999999999e-05 2024-04-02T05:04:15.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.188968+0000", 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0037467210000000002 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.189139+0000", 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017108 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:45.189373+0000", 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000234677 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:2677 340.a 340:50000000:%01:::head [pgnls start_epoch 864] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e864)", 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:03:10.470643+0000", 2024-04-02T05:04:15.241 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 64.680891113000001, 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067257660000000002, 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2677 2024-04-02T05:04:15.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470643+0000", 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470643+0000", 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470645+0000", 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0499999999999999e-06 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470653+0000", 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8089999999999993e-06 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470677+0000", 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4091e-05 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470691+0000", 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4134999999999999e-05 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470770+0000", 2024-04-02T05:04:15.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8750000000000003e-05 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.470779+0000", 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7019999999999996e-06 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.477180+0000", 2024-04-02T05:04:15.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006400808 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.477221+0000", 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0513999999999999e-05 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:10.477369+0000", 2024-04-02T05:04:15.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014790699999999999 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4290 3.9 3:9268b73e:::obj_delete_at_hint.0000000019:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:02.587996+0000", 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 12.563537539, 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024944049999999999, 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4290 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.587996+0000", 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.587996+0000", 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.587998+0000", 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.897e-06 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.250 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.588009+0000", 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0393e-05 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.588035+0000", 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6202000000000001e-05 2024-04-02T05:04:15.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.588049+0000", 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4049999999999999e-05 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.588177+0000", 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000127906 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.255 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.588419+0000", 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024201999999999999 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,7", 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.588910+0000", 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00049142200000000004 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.589718+0000", 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080769399999999997 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.256 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.589949+0000", 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023064400000000001 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.590427+0000", 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047806200000000002 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.590463+0000", 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5911000000000002e-05 2024-04-02T05:04:15.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.590491+0000", 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8204e-05 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4312 3.9 3:9195808b:::obj_delete_at_hint.0000000027:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:02.635142+0000", 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 12.516391461, 2024-04-02T05:04:15.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024529320000000001, 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4312 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635142+0000", 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635142+0000", 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635145+0000", 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.993e-06 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635159+0000", 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3869999999999999e-05 2024-04-02T05:04:15.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635193+0000", 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3708999999999999e-05 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635210+0000", 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6708999999999999e-05 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635330+0000", 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000120393 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635490+0000", 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016024600000000001 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,7", 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.635778+0000", 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028731500000000001 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.636516+0000", 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00073789500000000002 2024-04-02T05:04:15.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.637310+0000", 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079456399999999999 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.637387+0000", 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7037000000000001e-05 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:15.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.637482+0000", 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4438000000000006e-05 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.637595+0000", 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000113763 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:15.264 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:04:15.533 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:15 smithi098 bash[23790]: cluster 2024-04-02T05:04:14.387600+0000 mgr.y (mgr.14538) 791 : cluster 0 pgmap v1389: 184 pgs: 4 peering, 7 active+undersized+remapped, 1 active+remapped, 172 active+clean; 582 KiB data, 820 MiB used, 714 GiB / 715 GiB avail; 65 KiB/s, 20 objects/s recovering 2024-04-02T05:04:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:15 smithi098 bash[23790]: cluster 2024-04-02T05:04:14.830632+0000 osd.5 (osd.5) 269 : cluster 0 1.10 scrub starts 2024-04-02T05:04:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:15 smithi098 bash[23790]: cluster 2024-04-02T05:04:14.833042+0000 osd.5 (osd.5) 270 : cluster 0 1.10 scrub ok 2024-04-02T05:04:15.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:15 smithi069 bash[21528]: cluster 2024-04-02T05:04:14.387600+0000 mgr.y (mgr.14538) 791 : cluster 0 pgmap v1389: 184 pgs: 4 peering, 7 active+undersized+remapped, 1 active+remapped, 172 active+clean; 582 KiB data, 820 MiB used, 714 GiB / 715 GiB avail; 65 KiB/s, 20 objects/s recovering 2024-04-02T05:04:15.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:15 smithi069 bash[21528]: cluster 2024-04-02T05:04:14.830632+0000 osd.5 (osd.5) 269 : cluster 0 1.10 scrub starts 2024-04-02T05:04:15.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:15 smithi069 bash[21528]: cluster 2024-04-02T05:04:14.833042+0000 osd.5 (osd.5) 270 : cluster 0 1.10 scrub ok 2024-04-02T05:04:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:15 smithi069 bash[28904]: cluster 2024-04-02T05:04:14.387600+0000 mgr.y (mgr.14538) 791 : cluster 0 pgmap v1389: 184 pgs: 4 peering, 7 active+undersized+remapped, 1 active+remapped, 172 active+clean; 582 KiB data, 820 MiB used, 714 GiB / 715 GiB avail; 65 KiB/s, 20 objects/s recovering 2024-04-02T05:04:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:15 smithi069 bash[28904]: cluster 2024-04-02T05:04:14.830632+0000 osd.5 (osd.5) 269 : cluster 0 1.10 scrub starts 2024-04-02T05:04:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:15 smithi069 bash[28904]: cluster 2024-04-02T05:04:14.833042+0000 osd.5 (osd.5) 270 : cluster 0 1.10 scrub ok 2024-04-02T05:04:16.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[21528]: cluster 2024-04-02T05:04:14.923458+0000 osd.6 (osd.6) 389 : cluster 0 2.14 scrub starts 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[21528]: cluster 2024-04-02T05:04:14.925660+0000 osd.6 (osd.6) 390 : cluster 0 2.14 scrub ok 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[21528]: cluster 2024-04-02T05:04:15.398597+0000 osd.1 (osd.1) 461 : cluster 0 4.1c scrub starts 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[21528]: cluster 2024-04-02T05:04:15.401210+0000 osd.1 (osd.1) 462 : cluster 0 4.1c scrub ok 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[21528]: cluster 2024-04-02T05:04:15.797098+0000 osd.5 (osd.5) 271 : cluster 0 3.16 deep-scrub starts 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[21528]: cluster 2024-04-02T05:04:15.805559+0000 osd.5 (osd.5) 272 : cluster 0 3.16 deep-scrub ok 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[28904]: cluster 2024-04-02T05:04:14.923458+0000 osd.6 (osd.6) 389 : cluster 0 2.14 scrub starts 2024-04-02T05:04:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[28904]: cluster 2024-04-02T05:04:14.925660+0000 osd.6 (osd.6) 390 : cluster 0 2.14 scrub ok 2024-04-02T05:04:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[28904]: cluster 2024-04-02T05:04:15.398597+0000 osd.1 (osd.1) 461 : cluster 0 4.1c scrub starts 2024-04-02T05:04:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[28904]: cluster 2024-04-02T05:04:15.401210+0000 osd.1 (osd.1) 462 : cluster 0 4.1c scrub ok 2024-04-02T05:04:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[28904]: cluster 2024-04-02T05:04:15.797098+0000 osd.5 (osd.5) 271 : cluster 0 3.16 deep-scrub starts 2024-04-02T05:04:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:16 smithi069 bash[28904]: cluster 2024-04-02T05:04:15.805559+0000 osd.5 (osd.5) 272 : cluster 0 3.16 deep-scrub ok 2024-04-02T05:04:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:16 smithi098 bash[23790]: cluster 2024-04-02T05:04:14.923458+0000 osd.6 (osd.6) 389 : cluster 0 2.14 scrub starts 2024-04-02T05:04:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:16 smithi098 bash[23790]: cluster 2024-04-02T05:04:14.925660+0000 osd.6 (osd.6) 390 : cluster 0 2.14 scrub ok 2024-04-02T05:04:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:16 smithi098 bash[23790]: cluster 2024-04-02T05:04:15.398597+0000 osd.1 (osd.1) 461 : cluster 0 4.1c scrub starts 2024-04-02T05:04:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:16 smithi098 bash[23790]: cluster 2024-04-02T05:04:15.401210+0000 osd.1 (osd.1) 462 : cluster 0 4.1c scrub ok 2024-04-02T05:04:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:16 smithi098 bash[23790]: cluster 2024-04-02T05:04:15.797098+0000 osd.5 (osd.5) 271 : cluster 0 3.16 deep-scrub starts 2024-04-02T05:04:16.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:16 smithi098 bash[23790]: cluster 2024-04-02T05:04:15.805559+0000 osd.5 (osd.5) 272 : cluster 0 3.16 deep-scrub ok 2024-04-02T05:04:17.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.353038+0000 osd.1 (osd.1) 463 : cluster 0 6.1d scrub starts 2024-04-02T05:04:17.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.357135+0000 osd.1 (osd.1) 464 : cluster 0 6.1d scrub ok 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.388464+0000 mgr.y (mgr.14538) 792 : cluster 0 pgmap v1390: 184 pgs: 4 peering, 7 active+undersized+remapped, 1 active+remapped, 172 active+clean; 582 KiB data, 820 MiB used, 714 GiB / 715 GiB avail; 50 KiB/s, 16 objects/s recovering 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.519084+0000 osd.7 (osd.7) 259 : cluster 0 1.2 scrub starts 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.521526+0000 osd.7 (osd.7) 260 : cluster 0 1.2 scrub ok 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.806889+0000 osd.5 (osd.5) 273 : cluster 0 4.7 scrub starts 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.809159+0000 osd.5 (osd.5) 274 : cluster 0 4.7 scrub ok 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.353038+0000 osd.1 (osd.1) 463 : cluster 0 6.1d scrub starts 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.357135+0000 osd.1 (osd.1) 464 : cluster 0 6.1d scrub ok 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.388464+0000 mgr.y (mgr.14538) 792 : cluster 0 pgmap v1390: 184 pgs: 4 peering, 7 active+undersized+remapped, 1 active+remapped, 172 active+clean; 582 KiB data, 820 MiB used, 714 GiB / 715 GiB avail; 50 KiB/s, 16 objects/s recovering 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.519084+0000 osd.7 (osd.7) 259 : cluster 0 1.2 scrub starts 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.521526+0000 osd.7 (osd.7) 260 : cluster 0 1.2 scrub ok 2024-04-02T05:04:17.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.806889+0000 osd.5 (osd.5) 273 : cluster 0 4.7 scrub starts 2024-04-02T05:04:17.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:17 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.809159+0000 osd.5 (osd.5) 274 : cluster 0 4.7 scrub ok 2024-04-02T05:04:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.353038+0000 osd.1 (osd.1) 463 : cluster 0 6.1d scrub starts 2024-04-02T05:04:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.357135+0000 osd.1 (osd.1) 464 : cluster 0 6.1d scrub ok 2024-04-02T05:04:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.388464+0000 mgr.y (mgr.14538) 792 : cluster 0 pgmap v1390: 184 pgs: 4 peering, 7 active+undersized+remapped, 1 active+remapped, 172 active+clean; 582 KiB data, 820 MiB used, 714 GiB / 715 GiB avail; 50 KiB/s, 16 objects/s recovering 2024-04-02T05:04:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.519084+0000 osd.7 (osd.7) 259 : cluster 0 1.2 scrub starts 2024-04-02T05:04:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.521526+0000 osd.7 (osd.7) 260 : cluster 0 1.2 scrub ok 2024-04-02T05:04:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.806889+0000 osd.5 (osd.5) 273 : cluster 0 4.7 scrub starts 2024-04-02T05:04:17.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:17 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.809159+0000 osd.5 (osd.5) 274 : cluster 0 4.7 scrub ok 2024-04-02T05:04:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.144+0000 7f568899b700 1 -- 172.21.15.69:0/3392625862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684102320 msgr2=0x7f56841027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.144+0000 7f568899b700 1 --2- 172.21.15.69:0/3392625862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684102320 0x7f56841027a0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f5674009990 tx=0x7f5674009ca0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/3392625862 shutdown_connections 2024-04-02T05:04:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 --2- 172.21.15.69:0/3392625862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 0x7f568410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 --2- 172.21.15.69:0/3392625862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684108390 0x7f5684108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 --2- 172.21.15.69:0/3392625862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684102320 0x7f56841027a0 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/3392625862 >> 172.21.15.69:0/3392625862 conn(0x7f5684079310 msgr2=0x7f5684079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/3392625862 shutdown_connections 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/3392625862 wait complete. 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 librados: starting msgr at 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 librados: starting objecter 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 Processor -- start 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- start start 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 librados: setting wanted keys 2024-04-02T05:04:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 librados: calling monclient init 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 0x7f5684074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 0x7f5684074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684108390 0x7f5684074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5684111ad0 con 0x7f5684108390 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5684111950 con 0x7f5684102ce0 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5684111c50 con 0x7f5684102320 2024-04-02T05:04:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 0x7f5684074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 0x7f5684074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:43930/0 (socket says 172.21.15.69:43930) 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 -- 172.21.15.69:0/450182380 learned_addr learned my addr 172.21.15.69:0/450182380 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f5682d9d700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684108390 0x7f5684074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f5681d9b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 0x7f5684074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 -- 172.21.15.69:0/450182380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 msgr2=0x7f5684074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 0x7f5684074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 -- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684108390 msgr2=0x7f5684074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684108390 0x7f5684074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 -- 172.21.15.69:0/450182380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56841a6580 con 0x7f5684102320 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f5681d9b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 0x7f5684074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568259c700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 0x7f5684074540 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f567800e9a0 tx=0x7f567800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f56737fe700 10 client.?.objecter ms_handle_connect 0x7f5684102320 2024-04-02T05:04:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f56737fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567800cbd0 con 0x7f5684102320 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56841a6870 con 0x7f5684102320 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f567800cd70 con 0x7f5684102320 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678010e60 con 0x7f5684102320 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 10 client.43907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56841a6e00 con 0x7f5684102320 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 librados: init done 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.148+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568404ff00 con 0x7f5684102320 2024-04-02T05:04:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5678010430 con 0x7f5684102320 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566c06f5e0 0x7f566c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(905..905 src has 795..905) ==== 8870+0+0 (secure 0 0 0) 0x7f567808e750 con 0x7f5684102320 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 10 client.43907.objecter ms_dispatch 0x7f5684001550 osd_map(905..905 src has 795..905) 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f5681d9b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566c06f5e0 0x7f566c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 3 client.43907.objecter handle_osd_map got epochs [905,905] > 0 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 3 client.43907.objecter handle_osd_map decoding full epoch 905 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 20 client.43907.objecter dump_active .. 0 homeless 2024-04-02T05:04:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f56737fe700 10 client.43907.objecter ms_handle_connect 0x7f566c06f5e0 2024-04-02T05:04:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.152+0000 7f5681d9b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566c06f5e0 0x7f566c071aa0 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7f567400b2e0 tx=0x7f5674005e80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.156+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f567805b830 con 0x7f5684102320 2024-04-02T05:04:18.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.368+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f56841a7090 con 0x7f5684102320 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.496+0000 7f56737fe700 1 -- 172.21.15.69:0/450182380 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v906) ==== 101+0+0 (secure 0 0 0) 0x7f567805f280 con 0x7f5684102320 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.496+0000 7f568899b700 10 librados: watch_flush enter 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 10 librados: watch_flush exit 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 20 client.43907.objecter shutdown clearing up homeless session... 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 10 client.43907.objecter successfully canceled tick 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566c06f5e0 msgr2=0x7f566c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566c06f5e0 0x7f566c071aa0 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7f567400b2e0 tx=0x7f5674005e80 comp rx=0 tx=0).stop 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 msgr2=0x7f5684074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 0x7f5684074540 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f567800e9a0 tx=0x7f567800ecb0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 shutdown_connections 2024-04-02T05:04:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f566c06f5e0 0x7f566c071aa0 unknown :-1 s=CLOSED pgs=4031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684108390 0x7f5684074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5684102320 0x7f5684074540 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 --2- 172.21.15.69:0/450182380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5684102ce0 0x7f5684074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 >> 172.21.15.69:0/450182380 conn(0x7f5684079310 msgr2=0x7f5684109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 shutdown_connections 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 -- 172.21.15.69:0/450182380 wait complete. 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:18.500+0000 7f568899b700 1 librados: shutdown 2024-04-02T05:04:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:nodeep-scrub is unset 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:04:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.914609+0000 osd.6 (osd.6) 391 : cluster 0 10.d deep-scrub starts 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.918623+0000 osd.6 (osd.6) 392 : cluster 0 10.d deep-scrub ok 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.921373+0000 osd.4 (osd.4) 446 : cluster 0 4.c scrub starts 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:16.923467+0000 osd.4 (osd.4) 447 : cluster 0 4.c scrub ok 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:17.347753+0000 osd.1 (osd.1) 465 : cluster 0 2.4 scrub starts 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:17.350786+0000 osd.1 (osd.1) 466 : cluster 0 2.4 scrub ok 2024-04-02T05:04:18.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:17.470634+0000 osd.7 (osd.7) 261 : cluster 0 1.d scrub starts 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: cluster 2024-04-02T05:04:17.472927+0000 osd.7 (osd.7) 262 : cluster 0 1.d scrub ok 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: audit 2024-04-02T05:04:18.373696+0000 mon.c (mon.2) 1168 : audit 1 from='client.? 172.21.15.69:0/450182380' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[28904]: audit 2024-04-02T05:04:18.374125+0000 mon.a (mon.0) 4066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.914609+0000 osd.6 (osd.6) 391 : cluster 0 10.d deep-scrub starts 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.918623+0000 osd.6 (osd.6) 392 : cluster 0 10.d deep-scrub ok 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.921373+0000 osd.4 (osd.4) 446 : cluster 0 4.c scrub starts 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:16.923467+0000 osd.4 (osd.4) 447 : cluster 0 4.c scrub ok 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:17.347753+0000 osd.1 (osd.1) 465 : cluster 0 2.4 scrub starts 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:17.350786+0000 osd.1 (osd.1) 466 : cluster 0 2.4 scrub ok 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:17.470634+0000 osd.7 (osd.7) 261 : cluster 0 1.d scrub starts 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: cluster 2024-04-02T05:04:17.472927+0000 osd.7 (osd.7) 262 : cluster 0 1.d scrub ok 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: audit 2024-04-02T05:04:18.373696+0000 mon.c (mon.2) 1168 : audit 1 from='client.? 172.21.15.69:0/450182380' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:04:18.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:18 smithi069 bash[21528]: audit 2024-04-02T05:04:18.374125+0000 mon.a (mon.0) 4066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:04:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.914609+0000 osd.6 (osd.6) 391 : cluster 0 10.d deep-scrub starts 2024-04-02T05:04:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.918623+0000 osd.6 (osd.6) 392 : cluster 0 10.d deep-scrub ok 2024-04-02T05:04:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.921373+0000 osd.4 (osd.4) 446 : cluster 0 4.c scrub starts 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:16.923467+0000 osd.4 (osd.4) 447 : cluster 0 4.c scrub ok 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:17.347753+0000 osd.1 (osd.1) 465 : cluster 0 2.4 scrub starts 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:17.350786+0000 osd.1 (osd.1) 466 : cluster 0 2.4 scrub ok 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:17.470634+0000 osd.7 (osd.7) 261 : cluster 0 1.d scrub starts 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: cluster 2024-04-02T05:04:17.472927+0000 osd.7 (osd.7) 262 : cluster 0 1.d scrub ok 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: audit 2024-04-02T05:04:18.373696+0000 mon.c (mon.2) 1168 : audit 1 from='client.? 172.21.15.69:0/450182380' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:04:18.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:18 smithi098 bash[23790]: audit 2024-04-02T05:04:18.374125+0000 mon.a (mon.0) 4066 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.376311+0000 osd.1 (osd.1) 467 : cluster 0 10.14 scrub starts 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.378074+0000 osd.1 (osd.1) 468 : cluster 0 10.14 scrub ok 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.389573+0000 mgr.y (mgr.14538) 793 : cluster 0 pgmap v1391: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail; 657 B/s rd, 0 op/s; 42 KiB/s, 13 objects/s recovering 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.463429+0000 osd.3 (osd.3) 467 : cluster 0 4.14 scrub starts 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.466210+0000 osd.3 (osd.3) 468 : cluster 0 4.14 scrub ok 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: audit 2024-04-02T05:04:18.488977+0000 mon.a (mon.0) 4067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.498489+0000 mon.a (mon.0) 4068 : cluster 0 osdmap e906: 8 total, 8 up, 8 in 2024-04-02T05:04:19.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.511839+0000 osd.7 (osd.7) 263 : cluster 0 6.3 scrub starts 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.514127+0000 osd.7 (osd.7) 264 : cluster 0 6.3 scrub ok 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.584221+0000 osd.0 (osd.0) 449 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.584388+0000 osd.0 (osd.0) 450 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.617667+0000 osd.0 (osd.0) 451 : cluster 0 4.16 scrub starts 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.619113+0000 osd.0 (osd.0) 452 : cluster 0 4.16 scrub ok 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.840384+0000 osd.5 (osd.5) 275 : cluster 0 3.5 scrub starts 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: cluster 2024-04-02T05:04:18.845555+0000 osd.5 (osd.5) 276 : cluster 0 3.5 scrub ok 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: audit 2024-04-02T05:04:19.328485+0000 mon.a (mon.0) 4069 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[28904]: audit 2024-04-02T05:04:19.331481+0000 mon.b (mon.1) 327 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.376311+0000 osd.1 (osd.1) 467 : cluster 0 10.14 scrub starts 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.378074+0000 osd.1 (osd.1) 468 : cluster 0 10.14 scrub ok 2024-04-02T05:04:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.389573+0000 mgr.y (mgr.14538) 793 : cluster 0 pgmap v1391: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail; 657 B/s rd, 0 op/s; 42 KiB/s, 13 objects/s recovering 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.463429+0000 osd.3 (osd.3) 467 : cluster 0 4.14 scrub starts 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.466210+0000 osd.3 (osd.3) 468 : cluster 0 4.14 scrub ok 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: audit 2024-04-02T05:04:18.488977+0000 mon.a (mon.0) 4067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.498489+0000 mon.a (mon.0) 4068 : cluster 0 osdmap e906: 8 total, 8 up, 8 in 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.511839+0000 osd.7 (osd.7) 263 : cluster 0 6.3 scrub starts 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.514127+0000 osd.7 (osd.7) 264 : cluster 0 6.3 scrub ok 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.584221+0000 osd.0 (osd.0) 449 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.584388+0000 osd.0 (osd.0) 450 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.617667+0000 osd.0 (osd.0) 451 : cluster 0 4.16 scrub starts 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.619113+0000 osd.0 (osd.0) 452 : cluster 0 4.16 scrub ok 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.840384+0000 osd.5 (osd.5) 275 : cluster 0 3.5 scrub starts 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: cluster 2024-04-02T05:04:18.845555+0000 osd.5 (osd.5) 276 : cluster 0 3.5 scrub ok 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: audit 2024-04-02T05:04:19.328485+0000 mon.a (mon.0) 4069 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:19.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:19 smithi069 bash[21528]: audit 2024-04-02T05:04:19.331481+0000 mon.b (mon.1) 327 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.376311+0000 osd.1 (osd.1) 467 : cluster 0 10.14 scrub starts 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.378074+0000 osd.1 (osd.1) 468 : cluster 0 10.14 scrub ok 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.389573+0000 mgr.y (mgr.14538) 793 : cluster 0 pgmap v1391: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail; 657 B/s rd, 0 op/s; 42 KiB/s, 13 objects/s recovering 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.463429+0000 osd.3 (osd.3) 467 : cluster 0 4.14 scrub starts 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.466210+0000 osd.3 (osd.3) 468 : cluster 0 4.14 scrub ok 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: audit 2024-04-02T05:04:18.488977+0000 mon.a (mon.0) 4067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.498489+0000 mon.a (mon.0) 4068 : cluster 0 osdmap e906: 8 total, 8 up, 8 in 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.511839+0000 osd.7 (osd.7) 263 : cluster 0 6.3 scrub starts 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.514127+0000 osd.7 (osd.7) 264 : cluster 0 6.3 scrub ok 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.584221+0000 osd.0 (osd.0) 449 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.584388+0000 osd.0 (osd.0) 450 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.617667+0000 osd.0 (osd.0) 451 : cluster 0 4.16 scrub starts 2024-04-02T05:04:19.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.619113+0000 osd.0 (osd.0) 452 : cluster 0 4.16 scrub ok 2024-04-02T05:04:19.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.840384+0000 osd.5 (osd.5) 275 : cluster 0 3.5 scrub starts 2024-04-02T05:04:19.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: cluster 2024-04-02T05:04:18.845555+0000 osd.5 (osd.5) 276 : cluster 0 3.5 scrub ok 2024-04-02T05:04:19.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: audit 2024-04-02T05:04:19.328485+0000 mon.a (mon.0) 4069 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:19.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:19 smithi098 bash[23790]: audit 2024-04-02T05:04:19.331481+0000 mon.b (mon.1) 327 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:20.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 -- 172.21.15.69:0/3081276002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a0104e60 msgr2=0x7ff1a0111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/3081276002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a0104e60 0x7ff1a0111320 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7ff198009f90 tx=0x7ff19800b330 comp rx=0 tx=0).stop 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 -- 172.21.15.69:0/3081276002 shutdown_connections 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/3081276002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a0104e60 0x7ff1a0111320 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/3081276002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a010a550 0x7ff1a010a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/3081276002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a01044a0 0x7ff1a0104920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 -- 172.21.15.69:0/3081276002 >> 172.21.15.69:0/3081276002 conn(0x7ff1a00fff70 msgr2=0x7ff1a0102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 -- 172.21.15.69:0/3081276002 shutdown_connections 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.104+0000 7ff1a77f4700 1 -- 172.21.15.69:0/3081276002 wait complete. 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 librados: starting msgr at 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 librados: starting objecter 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 Processor -- start 2024-04-02T05:04:20.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- start start 2024-04-02T05:04:20.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 librados: setting wanted keys 2024-04-02T05:04:20.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 librados: calling monclient init 2024-04-02T05:04:20.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a01044a0 0x7ff1a019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:20.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 0x7ff1a019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:20.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a010a550 0x7ff1a019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:20.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a0113cc0 con 0x7ff1a0104e60 2024-04-02T05:04:20.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a0113b40 con 0x7ff1a01044a0 2024-04-02T05:04:20.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a0113e40 con 0x7ff1a010a550 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 0x7ff1a019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a5d91700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a010a550 0x7ff1a019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 0x7ff1a019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40886/0 (socket says 172.21.15.69:40886) 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 -- 172.21.15.69:0/1566378695 learned_addr learned my addr 172.21.15.69:0/1566378695 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a5590700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a01044a0 0x7ff1a019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 -- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a010a550 msgr2=0x7ff1a019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a010a550 0x7ff1a019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 -- 172.21.15.69:0/1566378695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a01044a0 msgr2=0x7ff1a019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a01044a0 0x7ff1a019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 -- 172.21.15.69:0/1566378695 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a019f260 con 0x7ff1a0104e60 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a4d8f700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 0x7ff1a019a790 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7ff19400e9c0 tx=0x7ff19400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 10 client.?.objecter ms_handle_connect 0x7ff1a0104e60 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:20.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19400cbd0 con 0x7ff1a0104e60 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a019f550 con 0x7ff1a0104e60 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff19400cd70 con 0x7ff1a0104e60 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1940054c0 con 0x7ff1a0104e60 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 10 client.33849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a01a6dd0 con 0x7ff1a0104e60 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 librados: init done 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a004ff00 con 0x7ff1a0104e60 2024-04-02T05:04:20.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff194004750 con 0x7ff1a0104e60 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff18c06f5e0 0x7ff18c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(907..907 src has 795..907) ==== 8870+0+0 (secure 0 0 0) 0x7ff19408e720 con 0x7ff1a0104e60 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 10 client.33849.objecter ms_dispatch 0x7ff1a0001550 osd_map(907..907 src has 795..907) 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 3 client.33849.objecter handle_osd_map got epochs [907,907] > 0 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1927fc700 3 client.33849.objecter handle_osd_map decoding full epoch 907 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.108+0000 7ff1a5590700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff18c06f5e0 0x7ff18c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.112+0000 7ff1927fc700 20 client.33849.objecter dump_active .. 0 homeless 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.112+0000 7ff1a5590700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff18c06f5e0 0x7ff18c071aa0 secure :-1 s=READY pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006fd0 tx=0x7ff19c008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:20.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.112+0000 7ff1927fc700 10 client.33849.objecter ms_handle_connect 0x7ff18c06f5e0 2024-04-02T05:04:20.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.116+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff19405b780 con 0x7ff1a0104e60 2024-04-02T05:04:20.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:20.336+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} v 0) -- 0x7ff1a01a6fa0 con 0x7ff1a0104e60 2024-04-02T05:04:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:20 smithi069 bash[28904]: cluster 2024-04-02T05:04:19.498941+0000 mon.a (mon.0) 4070 : cluster 0 osdmap e907: 8 total, 8 up, 8 in 2024-04-02T05:04:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:20 smithi069 bash[28904]: audit 2024-04-02T05:04:20.343095+0000 mon.a (mon.0) 4071 : audit 1 from='client.? 172.21.15.69:0/1566378695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-04-02T05:04:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:20 smithi069 bash[21528]: cluster 2024-04-02T05:04:19.498941+0000 mon.a (mon.0) 4070 : cluster 0 osdmap e907: 8 total, 8 up, 8 in 2024-04-02T05:04:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:20 smithi069 bash[21528]: audit 2024-04-02T05:04:20.343095+0000 mon.a (mon.0) 4071 : audit 1 from='client.? 172.21.15.69:0/1566378695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-04-02T05:04:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:20 smithi098 bash[23790]: cluster 2024-04-02T05:04:19.498941+0000 mon.a (mon.0) 4070 : cluster 0 osdmap e907: 8 total, 8 up, 8 in 2024-04-02T05:04:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:20 smithi098 bash[23790]: audit 2024-04-02T05:04:20.343095+0000 mon.a (mon.0) 4071 : audit 1 from='client.? 172.21.15.69:0/1566378695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-04-02T05:04:21.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.164+0000 7ff1927fc700 1 -- 172.21.15.69:0/1566378695 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]=0 set pool 1 pgp_num to 21 v908) ==== 131+0+0 (secure 0 0 0) 0x7ff19405f1d0 con 0x7ff1a0104e60 2024-04-02T05:04:21.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.168+0000 7ff1a77f4700 10 librados: watch_flush enter 2024-04-02T05:04:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.172+0000 7ff1a77f4700 10 librados: watch_flush exit 2024-04-02T05:04:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.172+0000 7ff1a77f4700 20 client.33849.objecter shutdown clearing up homeless session... 2024-04-02T05:04:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.172+0000 7ff1a77f4700 10 client.33849.objecter successfully canceled tick 2024-04-02T05:04:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff18c06f5e0 msgr2=0x7ff18c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff18c06f5e0 0x7ff18c071aa0 secure :-1 s=READY pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006fd0 tx=0x7ff19c008040 comp rx=0 tx=0).stop 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 msgr2=0x7ff1a019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 0x7ff1a019a790 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7ff19400e9c0 tx=0x7ff19400ecd0 comp rx=0 tx=0).stop 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 shutdown_connections 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff18c06f5e0 0x7ff18c071aa0 secure :-1 s=CLOSED pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006fd0 tx=0x7ff19c008040 comp rx=0 tx=0).stop 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a010a550 0x7ff1a019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a01044a0 0x7ff1a019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 --2- 172.21.15.69:0/1566378695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a0104e60 0x7ff1a019a790 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 >> 172.21.15.69:0/1566378695 conn(0x7ff1a00fff70 msgr2=0x7ff1a0108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 shutdown_connections 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 -- 172.21.15.69:0/1566378695 wait complete. 2024-04-02T05:04:21.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:21.176+0000 7ff1a77f4700 1 librados: shutdown 2024-04-02T05:04:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:set pool 1 pgp_num to 21 2024-04-02T05:04:21.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:19.966407+0000 osd.4 (osd.4) 448 : cluster 0 4.18 scrub starts 2024-04-02T05:04:21.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:19.969178+0000 osd.4 (osd.4) 449 : cluster 0 4.18 scrub ok 2024-04-02T05:04:21.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.333108+0000 osd.2 (osd.2) 380 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.333557+0000 osd.2 (osd.2) 381 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.390289+0000 mgr.y (mgr.14538) 794 : cluster 0 pgmap v1394: 184 pgs: 184 active+clean; 582 KiB data, 812 MiB used, 714 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.501934+0000 osd.3 (osd.3) 469 : cluster 0 2.1 scrub starts 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.504602+0000 osd.3 (osd.3) 470 : cluster 0 2.1 scrub ok 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.533929+0000 osd.7 (osd.7) 265 : cluster 0 1.e scrub starts 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.536681+0000 osd.7 (osd.7) 266 : cluster 0 1.e scrub ok 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: audit 2024-04-02T05:04:21.170396+0000 mon.a (mon.0) 4072 : audit 1 from='client.? 172.21.15.69:0/1566378695' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[21528]: cluster 2024-04-02T05:04:21.178619+0000 mon.a (mon.0) 4073 : cluster 0 osdmap e908: 8 total, 8 up, 8 in 2024-04-02T05:04:21.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:19.966407+0000 osd.4 (osd.4) 448 : cluster 0 4.18 scrub starts 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:19.969178+0000 osd.4 (osd.4) 449 : cluster 0 4.18 scrub ok 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.333108+0000 osd.2 (osd.2) 380 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.333557+0000 osd.2 (osd.2) 381 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.390289+0000 mgr.y (mgr.14538) 794 : cluster 0 pgmap v1394: 184 pgs: 184 active+clean; 582 KiB data, 812 MiB used, 714 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.501934+0000 osd.3 (osd.3) 469 : cluster 0 2.1 scrub starts 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.504602+0000 osd.3 (osd.3) 470 : cluster 0 2.1 scrub ok 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.533929+0000 osd.7 (osd.7) 265 : cluster 0 1.e scrub starts 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.536681+0000 osd.7 (osd.7) 266 : cluster 0 1.e scrub ok 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: audit 2024-04-02T05:04:21.170396+0000 mon.a (mon.0) 4072 : audit 1 from='client.? 172.21.15.69:0/1566378695' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-04-02T05:04:21.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:21 smithi069 bash[28904]: cluster 2024-04-02T05:04:21.178619+0000 mon.a (mon.0) 4073 : cluster 0 osdmap e908: 8 total, 8 up, 8 in 2024-04-02T05:04:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:19.966407+0000 osd.4 (osd.4) 448 : cluster 0 4.18 scrub starts 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:19.969178+0000 osd.4 (osd.4) 449 : cluster 0 4.18 scrub ok 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.333108+0000 osd.2 (osd.2) 380 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.333557+0000 osd.2 (osd.2) 381 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.390289+0000 mgr.y (mgr.14538) 794 : cluster 0 pgmap v1394: 184 pgs: 184 active+clean; 582 KiB data, 812 MiB used, 714 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.501934+0000 osd.3 (osd.3) 469 : cluster 0 2.1 scrub starts 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.504602+0000 osd.3 (osd.3) 470 : cluster 0 2.1 scrub ok 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.533929+0000 osd.7 (osd.7) 265 : cluster 0 1.e scrub starts 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.536681+0000 osd.7 (osd.7) 266 : cluster 0 1.e scrub ok 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: audit 2024-04-02T05:04:21.170396+0000 mon.a (mon.0) 4072 : audit 1 from='client.? 172.21.15.69:0/1566378695' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-04-02T05:04:21.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:21 smithi098 bash[23790]: cluster 2024-04-02T05:04:21.178619+0000 mon.a (mon.0) 4073 : cluster 0 osdmap e908: 8 total, 8 up, 8 in 2024-04-02T05:04:22.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-02T05:04:22.600 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.860467+0000 osd.6 (osd.6) 393 : cluster 0 10.1c deep-scrub starts 2024-04-02T05:04:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[21528]: cluster 2024-04-02T05:04:20.862834+0000 osd.6 (osd.6) 394 : cluster 0 10.1c deep-scrub ok 2024-04-02T05:04:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[21528]: cluster 2024-04-02T05:04:21.349466+0000 osd.1 (osd.1) 469 : cluster 0 6.1 scrub starts 2024-04-02T05:04:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[21528]: cluster 2024-04-02T05:04:21.354406+0000 osd.1 (osd.1) 470 : cluster 0 6.1 scrub ok 2024-04-02T05:04:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.860467+0000 osd.6 (osd.6) 393 : cluster 0 10.1c deep-scrub starts 2024-04-02T05:04:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[28904]: cluster 2024-04-02T05:04:20.862834+0000 osd.6 (osd.6) 394 : cluster 0 10.1c deep-scrub ok 2024-04-02T05:04:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[28904]: cluster 2024-04-02T05:04:21.349466+0000 osd.1 (osd.1) 469 : cluster 0 6.1 scrub starts 2024-04-02T05:04:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:22 smithi069 bash[28904]: cluster 2024-04-02T05:04:21.354406+0000 osd.1 (osd.1) 470 : cluster 0 6.1 scrub ok 2024-04-02T05:04:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:22 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.860467+0000 osd.6 (osd.6) 393 : cluster 0 10.1c deep-scrub starts 2024-04-02T05:04:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:22 smithi098 bash[23790]: cluster 2024-04-02T05:04:20.862834+0000 osd.6 (osd.6) 394 : cluster 0 10.1c deep-scrub ok 2024-04-02T05:04:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:22 smithi098 bash[23790]: cluster 2024-04-02T05:04:21.349466+0000 osd.1 (osd.1) 469 : cluster 0 6.1 scrub starts 2024-04-02T05:04:22.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:22 smithi098 bash[23790]: cluster 2024-04-02T05:04:21.354406+0000 osd.1 (osd.1) 470 : cluster 0 6.1 scrub ok 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:21.996343+0000 osd.4 (osd.4) 450 : cluster 0 6.1a scrub starts 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:21.998677+0000 osd.4 (osd.4) 451 : cluster 0 6.1a scrub ok 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.224155+0000 osd.1 (osd.1) 471 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.224568+0000 osd.1 (osd.1) 472 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.307626+0000 osd.1 (osd.1) 473 : cluster 0 2.6 deep-scrub starts 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.311031+0000 osd.1 (osd.1) 474 : cluster 0 2.6 deep-scrub ok 2024-04-02T05:04:23.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.391043+0000 mgr.y (mgr.14538) 795 : cluster 0 pgmap v1396: 184 pgs: 184 active+clean; 582 KiB data, 812 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.526213+0000 mon.a (mon.0) 4074 : cluster 0 osdmap e909: 8 total, 8 up, 8 in 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.563833+0000 osd.3 (osd.3) 471 : cluster 0 6.1b scrub starts 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.566255+0000 osd.3 (osd.3) 472 : cluster 0 6.1b scrub ok 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.815555+0000 osd.6 (osd.6) 395 : cluster 0 2.11 deep-scrub starts 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.817916+0000 osd.6 (osd.6) 396 : cluster 0 2.11 deep-scrub ok 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.889528+0000 osd.5 (osd.5) 277 : cluster 0 4.15 scrub starts 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.893109+0000 osd.5 (osd.5) 278 : cluster 0 4.15 scrub ok 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:21.996343+0000 osd.4 (osd.4) 450 : cluster 0 6.1a scrub starts 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:21.998677+0000 osd.4 (osd.4) 451 : cluster 0 6.1a scrub ok 2024-04-02T05:04:23.773 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.224155+0000 osd.1 (osd.1) 471 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.224568+0000 osd.1 (osd.1) 472 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.307626+0000 osd.1 (osd.1) 473 : cluster 0 2.6 deep-scrub starts 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.311031+0000 osd.1 (osd.1) 474 : cluster 0 2.6 deep-scrub ok 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.391043+0000 mgr.y (mgr.14538) 795 : cluster 0 pgmap v1396: 184 pgs: 184 active+clean; 582 KiB data, 812 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.526213+0000 mon.a (mon.0) 4074 : cluster 0 osdmap e909: 8 total, 8 up, 8 in 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.563833+0000 osd.3 (osd.3) 471 : cluster 0 6.1b scrub starts 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.566255+0000 osd.3 (osd.3) 472 : cluster 0 6.1b scrub ok 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.815555+0000 osd.6 (osd.6) 395 : cluster 0 2.11 deep-scrub starts 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.817916+0000 osd.6 (osd.6) 396 : cluster 0 2.11 deep-scrub ok 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.889528+0000 osd.5 (osd.5) 277 : cluster 0 4.15 scrub starts 2024-04-02T05:04:23.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:23 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.893109+0000 osd.5 (osd.5) 278 : cluster 0 4.15 scrub ok 2024-04-02T05:04:23.810 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:21.996343+0000 osd.4 (osd.4) 450 : cluster 0 6.1a scrub starts 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:21.998677+0000 osd.4 (osd.4) 451 : cluster 0 6.1a scrub ok 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.224155+0000 osd.1 (osd.1) 471 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.224568+0000 osd.1 (osd.1) 472 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.307626+0000 osd.1 (osd.1) 473 : cluster 0 2.6 deep-scrub starts 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.311031+0000 osd.1 (osd.1) 474 : cluster 0 2.6 deep-scrub ok 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.391043+0000 mgr.y (mgr.14538) 795 : cluster 0 pgmap v1396: 184 pgs: 184 active+clean; 582 KiB data, 812 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.526213+0000 mon.a (mon.0) 4074 : cluster 0 osdmap e909: 8 total, 8 up, 8 in 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.563833+0000 osd.3 (osd.3) 471 : cluster 0 6.1b scrub starts 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.566255+0000 osd.3 (osd.3) 472 : cluster 0 6.1b scrub ok 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.815555+0000 osd.6 (osd.6) 395 : cluster 0 2.11 deep-scrub starts 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.817916+0000 osd.6 (osd.6) 396 : cluster 0 2.11 deep-scrub ok 2024-04-02T05:04:23.811 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.889528+0000 osd.5 (osd.5) 277 : cluster 0 4.15 scrub starts 2024-04-02T05:04:23.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.893109+0000 osd.5 (osd.5) 278 : cluster 0 4.15 scrub ok 2024-04-02T05:04:24.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:04:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:04:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.992450+0000 osd.4 (osd.4) 452 : cluster 0 4.13 scrub starts 2024-04-02T05:04:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[21528]: cluster 2024-04-02T05:04:22.994708+0000 osd.4 (osd.4) 453 : cluster 0 4.13 scrub ok 2024-04-02T05:04:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[21528]: cluster 2024-04-02T05:04:23.349217+0000 osd.1 (osd.1) 475 : cluster 0 3.19 scrub starts 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[21528]: cluster 2024-04-02T05:04:23.354659+0000 osd.1 (osd.1) 476 : cluster 0 3.19 scrub ok 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[21528]: audit 2024-04-02T05:04:23.808191+0000 mgr.y (mgr.14538) 796 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.992450+0000 osd.4 (osd.4) 452 : cluster 0 4.13 scrub starts 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[28904]: cluster 2024-04-02T05:04:22.994708+0000 osd.4 (osd.4) 453 : cluster 0 4.13 scrub ok 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[28904]: cluster 2024-04-02T05:04:23.349217+0000 osd.1 (osd.1) 475 : cluster 0 3.19 scrub starts 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[28904]: cluster 2024-04-02T05:04:23.354659+0000 osd.1 (osd.1) 476 : cluster 0 3.19 scrub ok 2024-04-02T05:04:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:24 smithi069 bash[28904]: audit 2024-04-02T05:04:23.808191+0000 mgr.y (mgr.14538) 796 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:24 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.992450+0000 osd.4 (osd.4) 452 : cluster 0 4.13 scrub starts 2024-04-02T05:04:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:24 smithi098 bash[23790]: cluster 2024-04-02T05:04:22.994708+0000 osd.4 (osd.4) 453 : cluster 0 4.13 scrub ok 2024-04-02T05:04:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:24 smithi098 bash[23790]: cluster 2024-04-02T05:04:23.349217+0000 osd.1 (osd.1) 475 : cluster 0 3.19 scrub starts 2024-04-02T05:04:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:24 smithi098 bash[23790]: cluster 2024-04-02T05:04:23.354659+0000 osd.1 (osd.1) 476 : cluster 0 3.19 scrub ok 2024-04-02T05:04:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:24 smithi098 bash[23790]: audit 2024-04-02T05:04:23.808191+0000 mgr.y (mgr.14538) 796 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:25.534 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:04:25.534 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T05:04:25.535 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2024-04-02T05:04:25.535 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:04:25.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:25 smithi069 bash[21528]: cluster 2024-04-02T05:04:24.393395+0000 mgr.y (mgr.14538) 797 : cluster 0 pgmap v1398: 184 pgs: 184 active+clean; 582 KiB data, 808 MiB used, 714 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-04-02T05:04:25.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:25 smithi069 bash[28904]: cluster 2024-04-02T05:04:24.393395+0000 mgr.y (mgr.14538) 797 : cluster 0 pgmap v1398: 184 pgs: 184 active+clean; 582 KiB data, 808 MiB used, 714 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-04-02T05:04:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:25 smithi098 bash[23790]: cluster 2024-04-02T05:04:24.393395+0000 mgr.y (mgr.14538) 797 : cluster 0 pgmap v1398: 184 pgs: 184 active+clean; 582 KiB data, 808 MiB used, 714 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-04-02T05:04:26.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-02T05:04:26.742 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:24.973805+0000 osd.4 (osd.4) 454 : cluster 0 4.9 scrub starts 2024-04-02T05:04:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:24.976388+0000 osd.4 (osd.4) 455 : cluster 0 4.9 scrub ok 2024-04-02T05:04:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:25.277314+0000 osd.1 (osd.1) 477 : cluster 0 4.4 scrub starts 2024-04-02T05:04:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:25.279786+0000 osd.1 (osd.1) 478 : cluster 0 4.4 scrub ok 2024-04-02T05:04:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:25.817882+0000 osd.6 (osd.6) 397 : cluster 0 3.a scrub starts 2024-04-02T05:04:26.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:25.823567+0000 osd.6 (osd.6) 398 : cluster 0 3.a scrub ok 2024-04-02T05:04:26.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:25.875297+0000 osd.5 (osd.5) 279 : cluster 0 3.c scrub starts 2024-04-02T05:04:26.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:26 smithi098 bash[23790]: cluster 2024-04-02T05:04:25.879571+0000 osd.5 (osd.5) 280 : cluster 0 3.c scrub ok 2024-04-02T05:04:27.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:24.973805+0000 osd.4 (osd.4) 454 : cluster 0 4.9 scrub starts 2024-04-02T05:04:27.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:24.976388+0000 osd.4 (osd.4) 455 : cluster 0 4.9 scrub ok 2024-04-02T05:04:27.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:25.277314+0000 osd.1 (osd.1) 477 : cluster 0 4.4 scrub starts 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:25.279786+0000 osd.1 (osd.1) 478 : cluster 0 4.4 scrub ok 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:25.817882+0000 osd.6 (osd.6) 397 : cluster 0 3.a scrub starts 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:25.823567+0000 osd.6 (osd.6) 398 : cluster 0 3.a scrub ok 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:25.875297+0000 osd.5 (osd.5) 279 : cluster 0 3.c scrub starts 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[21528]: cluster 2024-04-02T05:04:25.879571+0000 osd.5 (osd.5) 280 : cluster 0 3.c scrub ok 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:24.973805+0000 osd.4 (osd.4) 454 : cluster 0 4.9 scrub starts 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:24.976388+0000 osd.4 (osd.4) 455 : cluster 0 4.9 scrub ok 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:25.277314+0000 osd.1 (osd.1) 477 : cluster 0 4.4 scrub starts 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:25.279786+0000 osd.1 (osd.1) 478 : cluster 0 4.4 scrub ok 2024-04-02T05:04:27.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:25.817882+0000 osd.6 (osd.6) 397 : cluster 0 3.a scrub starts 2024-04-02T05:04:27.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:25.823567+0000 osd.6 (osd.6) 398 : cluster 0 3.a scrub ok 2024-04-02T05:04:27.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:25.875297+0000 osd.5 (osd.5) 279 : cluster 0 3.c scrub starts 2024-04-02T05:04:27.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:26 smithi069 bash[28904]: cluster 2024-04-02T05:04:25.879571+0000 osd.5 (osd.5) 280 : cluster 0 3.c scrub ok 2024-04-02T05:04:27.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[21528]: cluster 2024-04-02T05:04:26.394308+0000 mgr.y (mgr.14538) 798 : cluster 0 pgmap v1399: 184 pgs: 184 active+clean; 582 KiB data, 808 MiB used, 714 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-04-02T05:04:27.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[21528]: cluster 2024-04-02T05:04:26.860009+0000 osd.6 (osd.6) 399 : cluster 0 5.2 scrub starts 2024-04-02T05:04:27.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[21528]: cluster 2024-04-02T05:04:26.863152+0000 osd.6 (osd.6) 400 : cluster 0 5.2 scrub ok 2024-04-02T05:04:27.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[21528]: audit 2024-04-02T05:04:27.051962+0000 mon.b (mon.1) 328 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:04:27.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[28904]: cluster 2024-04-02T05:04:26.394308+0000 mgr.y (mgr.14538) 798 : cluster 0 pgmap v1399: 184 pgs: 184 active+clean; 582 KiB data, 808 MiB used, 714 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-04-02T05:04:27.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[28904]: cluster 2024-04-02T05:04:26.860009+0000 osd.6 (osd.6) 399 : cluster 0 5.2 scrub starts 2024-04-02T05:04:27.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[28904]: cluster 2024-04-02T05:04:26.863152+0000 osd.6 (osd.6) 400 : cluster 0 5.2 scrub ok 2024-04-02T05:04:27.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:27 smithi069 bash[28904]: audit 2024-04-02T05:04:27.051962+0000 mon.b (mon.1) 328 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:04:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:27 smithi098 bash[23790]: cluster 2024-04-02T05:04:26.394308+0000 mgr.y (mgr.14538) 798 : cluster 0 pgmap v1399: 184 pgs: 184 active+clean; 582 KiB data, 808 MiB used, 714 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-04-02T05:04:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:27 smithi098 bash[23790]: cluster 2024-04-02T05:04:26.860009+0000 osd.6 (osd.6) 399 : cluster 0 5.2 scrub starts 2024-04-02T05:04:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:27 smithi098 bash[23790]: cluster 2024-04-02T05:04:26.863152+0000 osd.6 (osd.6) 400 : cluster 0 5.2 scrub ok 2024-04-02T05:04:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:27 smithi098 bash[23790]: audit 2024-04-02T05:04:27.051962+0000 mon.b (mon.1) 328 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:04:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:28 smithi098 bash[23790]: cluster 2024-04-02T05:04:27.422891+0000 osd.7 (osd.7) 267 : cluster 0 1.0 deep-scrub starts 2024-04-02T05:04:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:28 smithi098 bash[23790]: cluster 2024-04-02T05:04:27.424838+0000 osd.7 (osd.7) 268 : cluster 0 1.0 deep-scrub ok 2024-04-02T05:04:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:28 smithi098 bash[23790]: audit 2024-04-02T05:04:27.785874+0000 mon.b (mon.1) 329 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:04:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:28 smithi098 bash[23790]: audit 2024-04-02T05:04:27.787273+0000 mon.b (mon.1) 330 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:04:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:28 smithi098 bash[23790]: audit 2024-04-02T05:04:27.793411+0000 mon.a (mon.0) 4075 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:04:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[21528]: cluster 2024-04-02T05:04:27.422891+0000 osd.7 (osd.7) 267 : cluster 0 1.0 deep-scrub starts 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[21528]: cluster 2024-04-02T05:04:27.424838+0000 osd.7 (osd.7) 268 : cluster 0 1.0 deep-scrub ok 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[21528]: audit 2024-04-02T05:04:27.785874+0000 mon.b (mon.1) 329 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[21528]: audit 2024-04-02T05:04:27.787273+0000 mon.b (mon.1) 330 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[21528]: audit 2024-04-02T05:04:27.793411+0000 mon.a (mon.0) 4075 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:29.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[28904]: cluster 2024-04-02T05:04:27.422891+0000 osd.7 (osd.7) 267 : cluster 0 1.0 deep-scrub starts 2024-04-02T05:04:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[28904]: cluster 2024-04-02T05:04:27.424838+0000 osd.7 (osd.7) 268 : cluster 0 1.0 deep-scrub ok 2024-04-02T05:04:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[28904]: audit 2024-04-02T05:04:27.785874+0000 mon.b (mon.1) 329 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:04:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[28904]: audit 2024-04-02T05:04:27.787273+0000 mon.b (mon.1) 330 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:04:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:28 smithi069 bash[28904]: audit 2024-04-02T05:04:27.793411+0000 mon.a (mon.0) 4075 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:04:29.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 -- 172.21.15.69:0/3034632902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505006ac50 msgr2=0x7f505006b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 --2- 172.21.15.69:0/3034632902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505006ac50 0x7f505006b030 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f50400098f0 tx=0x7f5040009c00 comp rx=0 tx=0).stop 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 -- 172.21.15.69:0/3034632902 shutdown_connections 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 --2- 172.21.15.69:0/3034632902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050104920 0x7f505010f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 --2- 172.21.15.69:0/3034632902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505006ac50 0x7f505006b030 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 --2- 172.21.15.69:0/3034632902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5050103f60 0x7f50501043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 -- 172.21.15.69:0/3034632902 >> 172.21.15.69:0/3034632902 conn(0x7f50500771e0 msgr2=0x7f50500775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 -- 172.21.15.69:0/3034632902 shutdown_connections 2024-04-02T05:04:29.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 -- 172.21.15.69:0/3034632902 wait complete. 2024-04-02T05:04:29.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 librados: starting msgr at 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.620+0000 7f50566d9700 1 librados: starting objecter 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 Processor -- start 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- start start 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 librados: setting wanted keys 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 librados: calling monclient init 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f505006ac50 0x7f5050101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5050103f60 0x7f5050101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5050104920 0x7f5050101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5050111c40 con 0x7f505006ac50 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5050111ac0 con 0x7f5050104920 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5050111dc0 con 0x7f5050103f60 2024-04-02T05:04:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f505006ac50 0x7f5050101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f505006ac50 0x7f5050101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38678/0 (socket says 172.21.15.69:38678) 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 -- 172.21.15.69:0/269996160 learned_addr learned my addr 172.21.15.69:0/269996160 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504f7fe700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5050103f60 0x7f5050101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5050103f60 msgr2=0x7f5050101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5050103f60 0x7f5050101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5050104920 msgr2=0x7f5050101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5050104920 0x7f5050101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50501a67d0 con 0x7f505006ac50 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504ffff700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f505006ac50 0x7f5050101310 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f50400059d0 tx=0x7f504000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 10 client.?.objecter ms_handle_connect 0x7f505006ac50 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040027070 con 0x7f505006ac50 2024-04-02T05:04:29.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50501a6a60 con 0x7f505006ac50 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f504000bb60 con 0x7f505006ac50 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040004930 con 0x7f505006ac50 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 10 client.33855.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50501a6f10 con 0x7f505006ac50 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 librados: init done 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f505004ff00 con 0x7f505006ac50 2024-04-02T05:04:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f504002a070 con 0x7f505006ac50 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503c06f5e0 0x7f503c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(909..909 src has 795..909) ==== 8870+0+0 (secure 0 0 0) 0x7f504008f060 con 0x7f505006ac50 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504f7fe700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503c06f5e0 0x7f503c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 10 client.33855.objecter ms_dispatch 0x7f5050001550 osd_map(909..909 src has 795..909) 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 3 client.33855.objecter handle_osd_map got epochs [909,909] > 0 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 3 client.33855.objecter handle_osd_map decoding full epoch 909 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 20 client.33855.objecter dump_active .. 0 homeless 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.624+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1277 (secure 0 0 0) 0x7f5040013030 con 0x7f505006ac50 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 0x7f5050062a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504f7fe700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503c06f5e0 0x7f503c071aa0 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f5050077a70 tx=0x7f5038009190 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5050062fa0 con 0x7f5050062680 2024-04-02T05:04:29.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f503c06f5e0 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 0x7f5050062a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f5050063150 con 0x7f5050062680 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f5050062680 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 0x7f5050062a60 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f504800efb0 tx=0x7f504800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 10 client.33855.objecter resend_mon_ops 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048003a00 con 0x7f5050062680 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5048003a00 con 0x7f5050062680 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 msgr2=0x7f5050062a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 0x7f5050062a60 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f504800efb0 tx=0x7f504800c450 comp rx=0 tx=0).stop 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 msgr2=0x7f5050062a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T05:04:29.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.628+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050062680 0x7f5050062a60 secure :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f504800efb0 tx=0x7f504800c450 comp rx=0 tx=0).stop 2024-04-02T05:04:29.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 0x7f50501ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f50501ab640 con 0x7f5050063360 2024-04-02T05:04:29.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 0x7f50501ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f50501ab7f0 con 0x7f5050063360 2024-04-02T05:04:29.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f5050063360 2024-04-02T05:04:29.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 0x7f50501ab100 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f504800efb0 tx=0x7f5048004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 10 client.33855.objecter resend_mon_ops 2024-04-02T05:04:29.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504802b0a0 con 0x7f5050063360 2024-04-02T05:04:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.0 v2:172.21.15.69:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f504801dc00 con 0x7f5050063360 2024-04-02T05:04:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 msgr2=0x7f50501ab100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 0x7f50501ab100 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f504800efb0 tx=0x7f5048004ad0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 msgr2=0x7f50501ab100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T05:04:29.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5050063360 0x7f50501ab100 secure :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f504800efb0 tx=0x7f5048004ad0 comp rx=0 tx=0).stop 2024-04-02T05:04:29.643 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a: {} 2024-04-02T05:04:29.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 0x7f505002cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f505002d300 con 0x7f505002c9b0 2024-04-02T05:04:29.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.636+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 0x7f505002cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f505002d4b0 con 0x7f505002c9b0 2024-04-02T05:04:29.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 0x7f505002cdc0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f50501ab7c0 tx=0x7f504800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f505002c9b0 2024-04-02T05:04:29.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 10 client.33855.objecter resend_mon_ops 2024-04-02T05:04:29.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504802f040 con 0x7f505002c9b0 2024-04-02T05:04:29.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.1 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5048004bd0 con 0x7f505002c9b0 2024-04-02T05:04:29.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 msgr2=0x7f505002cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 0x7f505002cdc0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f50501ab7c0 tx=0x7f504800b040 comp rx=0 tx=0).stop 2024-04-02T05:04:29.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 msgr2=0x7f505002cdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T05:04:29.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002c9b0 0x7f505002cdc0 secure :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f50501ab7c0 tx=0x7f504800b040 comp rx=0 tx=0).stop 2024-04-02T05:04:29.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.640+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 reap_dead start 2024-04-02T05:04:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 0x7f505002daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50501a5bf0 con 0x7f505002d6c0 2024-04-02T05:04:29.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 0x7f505002daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f50501a5d70 con 0x7f505002d6c0 2024-04-02T05:04:29.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 0x7f505002daa0 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f504800c490 tx=0x7f5048029000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f505002d6c0 2024-04-02T05:04:29.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f504d7fa700 10 client.33855.objecter resend_mon_ops 2024-04-02T05:04:29.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.648+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048042020 con 0x7f505002d6c0 2024-04-02T05:04:29.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.1 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f504803c760 con 0x7f505002d6c0 2024-04-02T05:04:29.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 msgr2=0x7f505002daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 0x7f505002daa0 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f504800c490 tx=0x7f5048029000 comp rx=0 tx=0).stop 2024-04-02T05:04:29.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 msgr2=0x7f505002daa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T05:04:29.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505002d6c0 0x7f505002daa0 secure :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f504800c490 tx=0x7f5048029000 comp rx=0 tx=0).stop 2024-04-02T05:04:29.658 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b: {} 2024-04-02T05:04:29.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 0x7f5050101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5050101d90 con 0x7f50501a5f80 2024-04-02T05:04:29.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 0x7f5050101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f5050101f40 con 0x7f50501a5f80 2024-04-02T05:04:29.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f50501a5f80 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 0x7f5050101850 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f5048009610 tx=0x7f5048023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 10 client.33855.objecter resend_mon_ops 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048042020 con 0x7f50501a5f80 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.2 v2:172.21.15.69:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f504803cd30 con 0x7f50501a5f80 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 msgr2=0x7f5050101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 0x7f5050101850 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f5048009610 tx=0x7f5048023000 comp rx=0 tx=0).stop 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 msgr2=0x7f5050101850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T05:04:29.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.652+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a5f80 0x7f5050101850 secure :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f5048009610 tx=0x7f5048023000 comp rx=0 tx=0).stop 2024-04-02T05:04:29.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 0x7f50501a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:29.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5050102150 con 0x7f50501a4450 2024-04-02T05:04:29.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 0x7f50501a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:29.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f50501b2be0 con 0x7f50501a4450 2024-04-02T05:04:29.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f504d7fa700 10 client.33855.objecter ms_handle_connect 0x7f50501a4450 2024-04-02T05:04:29.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f504d7fa700 10 client.33855.objecter resend_mon_ops 2024-04-02T05:04:29.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f5054c76700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 0x7f50501a4830 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f504800f010 tx=0x7f5048041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:29.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.660+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048042040 con 0x7f50501a4450 2024-04-02T05:04:29.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 <== mon.2 v2:172.21.15.69:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f504804e270 con 0x7f50501a4450 2024-04-02T05:04:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 msgr2=0x7f50501a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 0x7f50501a4830 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f504800f010 tx=0x7f5048041000 comp rx=0 tx=0).stop 2024-04-02T05:04:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f504d7fa700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 msgr2=0x7f50501a4830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-02T05:04:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f504d7fa700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50501a4450 0x7f50501a4830 secure :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f504800f010 tx=0x7f5048041000 comp rx=0 tx=0).stop 2024-04-02T05:04:29.669 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c: {} 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 10 librados: watch_flush enter 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 10 librados: watch_flush exit 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 20 client.33855.objecter shutdown clearing up homeless session... 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 10 client.33855.objecter successfully canceled tick 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503c06f5e0 msgr2=0x7f503c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f503c06f5e0 0x7f503c071aa0 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f5050077a70 tx=0x7f5038009190 comp rx=0 tx=0).stop 2024-04-02T05:04:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f505006ac50 msgr2=0x7f5050101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 --2- 172.21.15.69:0/269996160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f505006ac50 0x7f5050101310 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f50400059d0 tx=0x7f504000fa70 comp rx=0 tx=0).stop 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f5054c76700 1 -- 172.21.15.69:0/269996160 reap_dead start 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 shutdown_connections 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 >> 172.21.15.69:0/269996160 conn(0x7f50500771e0 msgr2=0x7f5050109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 shutdown_connections 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 -- 172.21.15.69:0/269996160 wait complete. 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:29.664+0000 7f50566d9700 1 librados: shutdown 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-02T05:04:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-02T05:04:30.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:29 smithi098 bash[23790]: cluster 2024-04-02T05:04:28.396200+0000 mgr.y (mgr.14538) 799 : cluster 0 pgmap v1400: 184 pgs: 184 active+clean; 582 KiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:30.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:29 smithi098 bash[23790]: cluster 2024-04-02T05:04:28.775503+0000 osd.6 (osd.6) 401 : cluster 0 6.1e scrub starts 2024-04-02T05:04:30.229 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:29 smithi098 bash[23790]: cluster 2024-04-02T05:04:28.777842+0000 osd.6 (osd.6) 402 : cluster 0 6.1e scrub ok 2024-04-02T05:04:30.230 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:29 smithi098 bash[23790]: audit 2024-04-02T05:04:29.630995+0000 mon.a (mon.0) 4076 : audit 0 from='client.? 172.21.15.69:0/269996160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[21528]: cluster 2024-04-02T05:04:28.396200+0000 mgr.y (mgr.14538) 799 : cluster 0 pgmap v1400: 184 pgs: 184 active+clean; 582 KiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[21528]: cluster 2024-04-02T05:04:28.775503+0000 osd.6 (osd.6) 401 : cluster 0 6.1e scrub starts 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[21528]: cluster 2024-04-02T05:04:28.777842+0000 osd.6 (osd.6) 402 : cluster 0 6.1e scrub ok 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[21528]: audit 2024-04-02T05:04:29.630995+0000 mon.a (mon.0) 4076 : audit 0 from='client.? 172.21.15.69:0/269996160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[28904]: cluster 2024-04-02T05:04:28.396200+0000 mgr.y (mgr.14538) 799 : cluster 0 pgmap v1400: 184 pgs: 184 active+clean; 582 KiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[28904]: cluster 2024-04-02T05:04:28.775503+0000 osd.6 (osd.6) 401 : cluster 0 6.1e scrub starts 2024-04-02T05:04:30.290 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[28904]: cluster 2024-04-02T05:04:28.777842+0000 osd.6 (osd.6) 402 : cluster 0 6.1e scrub ok 2024-04-02T05:04:30.291 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:29 smithi069 bash[28904]: audit 2024-04-02T05:04:29.630995+0000 mon.a (mon.0) 4076 : audit 0 from='client.? 172.21.15.69:0/269996160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T05:04:30.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 0 1 2024-04-02T05:04:30.834 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:30 smithi098 bash[23790]: cluster 2024-04-02T05:04:29.975434+0000 osd.5 (osd.5) 281 : cluster 0 4.8 deep-scrub starts 2024-04-02T05:04:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:30 smithi098 bash[23790]: cluster 2024-04-02T05:04:29.977920+0000 osd.5 (osd.5) 282 : cluster 0 4.8 deep-scrub ok 2024-04-02T05:04:31.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:30 smithi069 bash[21528]: cluster 2024-04-02T05:04:29.975434+0000 osd.5 (osd.5) 281 : cluster 0 4.8 deep-scrub starts 2024-04-02T05:04:31.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:30 smithi069 bash[21528]: cluster 2024-04-02T05:04:29.977920+0000 osd.5 (osd.5) 282 : cluster 0 4.8 deep-scrub ok 2024-04-02T05:04:31.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:30 smithi069 bash[28904]: cluster 2024-04-02T05:04:29.975434+0000 osd.5 (osd.5) 281 : cluster 0 4.8 deep-scrub starts 2024-04-02T05:04:31.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:30 smithi069 bash[28904]: cluster 2024-04-02T05:04:29.977920+0000 osd.5 (osd.5) 282 : cluster 0 4.8 deep-scrub ok 2024-04-02T05:04:32.168 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:31 smithi069 bash[21528]: cluster 2024-04-02T05:04:30.396895+0000 mgr.y (mgr.14538) 800 : cluster 0 pgmap v1401: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:04:32.168 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:31 smithi069 bash[21528]: cluster 2024-04-02T05:04:30.794600+0000 osd.6 (osd.6) 403 : cluster 0 10.0 deep-scrub starts 2024-04-02T05:04:32.168 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:31 smithi069 bash[21528]: cluster 2024-04-02T05:04:30.797024+0000 osd.6 (osd.6) 404 : cluster 0 10.0 deep-scrub ok 2024-04-02T05:04:32.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:31 smithi069 bash[28904]: cluster 2024-04-02T05:04:30.396895+0000 mgr.y (mgr.14538) 800 : cluster 0 pgmap v1401: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:04:32.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:31 smithi069 bash[28904]: cluster 2024-04-02T05:04:30.794600+0000 osd.6 (osd.6) 403 : cluster 0 10.0 deep-scrub starts 2024-04-02T05:04:32.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:31 smithi069 bash[28904]: cluster 2024-04-02T05:04:30.797024+0000 osd.6 (osd.6) 404 : cluster 0 10.0 deep-scrub ok 2024-04-02T05:04:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:31 smithi098 bash[23790]: cluster 2024-04-02T05:04:30.396895+0000 mgr.y (mgr.14538) 800 : cluster 0 pgmap v1401: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:04:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:31 smithi098 bash[23790]: cluster 2024-04-02T05:04:30.794600+0000 osd.6 (osd.6) 403 : cluster 0 10.0 deep-scrub starts 2024-04-02T05:04:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:31 smithi098 bash[23790]: cluster 2024-04-02T05:04:30.797024+0000 osd.6 (osd.6) 404 : cluster 0 10.0 deep-scrub ok 2024-04-02T05:04:33.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:32 smithi069 bash[28904]: cluster 2024-04-02T05:04:31.811421+0000 osd.6 (osd.6) 405 : cluster 0 3.10 scrub starts 2024-04-02T05:04:33.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:32 smithi069 bash[28904]: cluster 2024-04-02T05:04:31.814897+0000 osd.6 (osd.6) 406 : cluster 0 3.10 scrub ok 2024-04-02T05:04:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:32 smithi069 bash[21528]: cluster 2024-04-02T05:04:31.811421+0000 osd.6 (osd.6) 405 : cluster 0 3.10 scrub starts 2024-04-02T05:04:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:32 smithi069 bash[21528]: cluster 2024-04-02T05:04:31.814897+0000 osd.6 (osd.6) 406 : cluster 0 3.10 scrub ok 2024-04-02T05:04:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:32 smithi098 bash[23790]: cluster 2024-04-02T05:04:31.811421+0000 osd.6 (osd.6) 405 : cluster 0 3.10 scrub starts 2024-04-02T05:04:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:32 smithi098 bash[23790]: cluster 2024-04-02T05:04:31.814897+0000 osd.6 (osd.6) 406 : cluster 0 3.10 scrub ok 2024-04-02T05:04:33.835 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:04:33.835 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2024-04-02T05:04:33.836 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2024-04-02T05:04:33.836 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2024-04-02T05:04:33.836 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:04:33.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:33 smithi069 bash[21528]: cluster 2024-04-02T05:04:32.397659+0000 mgr.y (mgr.14538) 801 : cluster 0 pgmap v1402: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:33.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:33 smithi069 bash[21528]: cluster 2024-04-02T05:04:33.563841+0000 osd.3 (osd.3) 473 : cluster 0 3.b scrub starts 2024-04-02T05:04:33.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:33 smithi069 bash[21528]: cluster 2024-04-02T05:04:33.569247+0000 osd.3 (osd.3) 474 : cluster 0 3.b scrub ok 2024-04-02T05:04:33.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:33 smithi069 bash[28904]: cluster 2024-04-02T05:04:32.397659+0000 mgr.y (mgr.14538) 801 : cluster 0 pgmap v1402: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:33.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:33 smithi069 bash[28904]: cluster 2024-04-02T05:04:33.563841+0000 osd.3 (osd.3) 473 : cluster 0 3.b scrub starts 2024-04-02T05:04:33.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:33 smithi069 bash[28904]: cluster 2024-04-02T05:04:33.569247+0000 osd.3 (osd.3) 474 : cluster 0 3.b scrub ok 2024-04-02T05:04:34.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:04:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:04:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:33 smithi098 bash[23790]: cluster 2024-04-02T05:04:32.397659+0000 mgr.y (mgr.14538) 801 : cluster 0 pgmap v1402: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:33 smithi098 bash[23790]: cluster 2024-04-02T05:04:33.563841+0000 osd.3 (osd.3) 473 : cluster 0 3.b scrub starts 2024-04-02T05:04:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:33 smithi098 bash[23790]: cluster 2024-04-02T05:04:33.569247+0000 osd.3 (osd.3) 474 : cluster 0 3.b scrub ok 2024-04-02T05:04:34.650 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-02T05:04:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[21528]: cluster 2024-04-02T05:04:33.477502+0000 osd.7 (osd.7) 269 : cluster 0 2.b scrub starts 2024-04-02T05:04:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[21528]: cluster 2024-04-02T05:04:33.479740+0000 osd.7 (osd.7) 270 : cluster 0 2.b scrub ok 2024-04-02T05:04:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[21528]: audit 2024-04-02T05:04:33.819316+0000 mgr.y (mgr.14538) 802 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[21528]: cluster 2024-04-02T05:04:33.822840+0000 osd.6 (osd.6) 407 : cluster 0 4.12 scrub starts 2024-04-02T05:04:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[21528]: cluster 2024-04-02T05:04:33.824985+0000 osd.6 (osd.6) 408 : cluster 0 4.12 scrub ok 2024-04-02T05:04:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[21528]: audit 2024-04-02T05:04:34.365762+0000 mon.b (mon.1) 331 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[28904]: cluster 2024-04-02T05:04:33.477502+0000 osd.7 (osd.7) 269 : cluster 0 2.b scrub starts 2024-04-02T05:04:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[28904]: cluster 2024-04-02T05:04:33.479740+0000 osd.7 (osd.7) 270 : cluster 0 2.b scrub ok 2024-04-02T05:04:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[28904]: audit 2024-04-02T05:04:33.819316+0000 mgr.y (mgr.14538) 802 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[28904]: cluster 2024-04-02T05:04:33.822840+0000 osd.6 (osd.6) 407 : cluster 0 4.12 scrub starts 2024-04-02T05:04:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[28904]: cluster 2024-04-02T05:04:33.824985+0000 osd.6 (osd.6) 408 : cluster 0 4.12 scrub ok 2024-04-02T05:04:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:34 smithi069 bash[28904]: audit 2024-04-02T05:04:34.365762+0000 mon.b (mon.1) 331 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:35.131 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:34 smithi098 bash[23790]: cluster 2024-04-02T05:04:33.477502+0000 osd.7 (osd.7) 269 : cluster 0 2.b scrub starts 2024-04-02T05:04:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:34 smithi098 bash[23790]: cluster 2024-04-02T05:04:33.479740+0000 osd.7 (osd.7) 270 : cluster 0 2.b scrub ok 2024-04-02T05:04:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:34 smithi098 bash[23790]: audit 2024-04-02T05:04:33.819316+0000 mgr.y (mgr.14538) 802 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:34 smithi098 bash[23790]: cluster 2024-04-02T05:04:33.822840+0000 osd.6 (osd.6) 407 : cluster 0 4.12 scrub starts 2024-04-02T05:04:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:34 smithi098 bash[23790]: cluster 2024-04-02T05:04:33.824985+0000 osd.6 (osd.6) 408 : cluster 0 4.12 scrub ok 2024-04-02T05:04:35.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:34 smithi098 bash[23790]: audit 2024-04-02T05:04:34.365762+0000 mon.b (mon.1) 331 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:36.108 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[21528]: cluster 2024-04-02T05:04:34.399681+0000 mgr.y (mgr.14538) 803 : cluster 0 pgmap v1403: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-02T05:04:36.108 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[21528]: cluster 2024-04-02T05:04:34.443664+0000 osd.7 (osd.7) 271 : cluster 0 3.d scrub starts 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[21528]: cluster 2024-04-02T05:04:34.448766+0000 osd.7 (osd.7) 272 : cluster 0 3.d scrub ok 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[21528]: cluster 2024-04-02T05:04:34.995580+0000 osd.5 (osd.5) 283 : cluster 0 10.9 scrub starts 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[21528]: cluster 2024-04-02T05:04:34.998998+0000 osd.5 (osd.5) 284 : cluster 0 10.9 scrub ok 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[28904]: cluster 2024-04-02T05:04:34.399681+0000 mgr.y (mgr.14538) 803 : cluster 0 pgmap v1403: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[28904]: cluster 2024-04-02T05:04:34.443664+0000 osd.7 (osd.7) 271 : cluster 0 3.d scrub starts 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[28904]: cluster 2024-04-02T05:04:34.448766+0000 osd.7 (osd.7) 272 : cluster 0 3.d scrub ok 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[28904]: cluster 2024-04-02T05:04:34.995580+0000 osd.5 (osd.5) 283 : cluster 0 10.9 scrub starts 2024-04-02T05:04:36.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:35 smithi069 bash[28904]: cluster 2024-04-02T05:04:34.998998+0000 osd.5 (osd.5) 284 : cluster 0 10.9 scrub ok 2024-04-02T05:04:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:35 smithi098 bash[23790]: cluster 2024-04-02T05:04:34.399681+0000 mgr.y (mgr.14538) 803 : cluster 0 pgmap v1403: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-02T05:04:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:35 smithi098 bash[23790]: cluster 2024-04-02T05:04:34.443664+0000 osd.7 (osd.7) 271 : cluster 0 3.d scrub starts 2024-04-02T05:04:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:35 smithi098 bash[23790]: cluster 2024-04-02T05:04:34.448766+0000 osd.7 (osd.7) 272 : cluster 0 3.d scrub ok 2024-04-02T05:04:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:35 smithi098 bash[23790]: cluster 2024-04-02T05:04:34.995580+0000 osd.5 (osd.5) 283 : cluster 0 10.9 scrub starts 2024-04-02T05:04:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:35 smithi098 bash[23790]: cluster 2024-04-02T05:04:34.998998+0000 osd.5 (osd.5) 284 : cluster 0 10.9 scrub ok 2024-04-02T05:04:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:36 smithi098 bash[23790]: cluster 2024-04-02T05:04:35.568931+0000 osd.0 (osd.0) 453 : cluster 0 10.1a scrub starts 2024-04-02T05:04:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:36 smithi098 bash[23790]: cluster 2024-04-02T05:04:35.573697+0000 osd.0 (osd.0) 454 : cluster 0 10.1a scrub ok 2024-04-02T05:04:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:36 smithi098 bash[23790]: cluster 2024-04-02T05:04:35.754832+0000 osd.7 (osd.7) 273 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:36 smithi098 bash[23790]: cluster 2024-04-02T05:04:35.755270+0000 osd.7 (osd.7) 274 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:36 smithi098 bash[23790]: cluster 2024-04-02T05:04:36.170553+0000 mon.a (mon.0) 4077 : cluster 0 osdmap e910: 8 total, 8 up, 8 in 2024-04-02T05:04:37.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[21528]: cluster 2024-04-02T05:04:35.568931+0000 osd.0 (osd.0) 453 : cluster 0 10.1a scrub starts 2024-04-02T05:04:37.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[21528]: cluster 2024-04-02T05:04:35.573697+0000 osd.0 (osd.0) 454 : cluster 0 10.1a scrub ok 2024-04-02T05:04:37.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[21528]: cluster 2024-04-02T05:04:35.754832+0000 osd.7 (osd.7) 273 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:37.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[21528]: cluster 2024-04-02T05:04:35.755270+0000 osd.7 (osd.7) 274 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:37.286 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[21528]: cluster 2024-04-02T05:04:36.170553+0000 mon.a (mon.0) 4077 : cluster 0 osdmap e910: 8 total, 8 up, 8 in 2024-04-02T05:04:37.286 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[28904]: cluster 2024-04-02T05:04:35.568931+0000 osd.0 (osd.0) 453 : cluster 0 10.1a scrub starts 2024-04-02T05:04:37.287 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[28904]: cluster 2024-04-02T05:04:35.573697+0000 osd.0 (osd.0) 454 : cluster 0 10.1a scrub ok 2024-04-02T05:04:37.287 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[28904]: cluster 2024-04-02T05:04:35.754832+0000 osd.7 (osd.7) 273 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:37.287 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[28904]: cluster 2024-04-02T05:04:35.755270+0000 osd.7 (osd.7) 274 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:37.287 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:36 smithi069 bash[28904]: cluster 2024-04-02T05:04:36.170553+0000 mon.a (mon.0) 4077 : cluster 0 osdmap e910: 8 total, 8 up, 8 in 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 -- 172.21.15.69:0/1011818678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406b510 msgr2=0x7f87f4106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 --2- 172.21.15.69:0/1011818678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406b510 0x7f87f4106ef0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f87e8009990 tx=0x7f87e8009ca0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 -- 172.21.15.69:0/1011818678 shutdown_connections 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 --2- 172.21.15.69:0/1011818678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f41075d0 0x7f87f410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 --2- 172.21.15.69:0/1011818678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f406ab60 0x7f87f406af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 --2- 172.21.15.69:0/1011818678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406b510 0x7f87f4106ef0 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 -- 172.21.15.69:0/1011818678 >> 172.21.15.69:0/1011818678 conn(0x7f87f4078190 msgr2=0x7f87f40785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:38.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 -- 172.21.15.69:0/1011818678 shutdown_connections 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.032+0000 7f87fb971700 1 -- 172.21.15.69:0/1011818678 wait complete. 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 librados: starting msgr at 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 librados: starting objecter 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 Processor -- start 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- start start 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 librados: setting wanted keys 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 librados: calling monclient init 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 0x7f87f4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f406b510 0x7f87f4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f41075d0 0x7f87f419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87f4111a80 con 0x7f87f41075d0 2024-04-02T05:04:38.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87f4111900 con 0x7f87f406b510 2024-04-02T05:04:38.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f87f4111c00 con 0x7f87f406ab60 2024-04-02T05:04:38.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 0x7f87f4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:38.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 0x7f87f4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38672/0 (socket says 172.21.15.69:38672) 2024-04-02T05:04:38.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 -- 172.21.15.69:0/519243271 learned_addr learned my addr 172.21.15.69:0/519243271 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:38.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f9f0e700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f41075d0 0x7f87f419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f8f0c700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f406b510 0x7f87f4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:38.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 -- 172.21.15.69:0/519243271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f406b510 msgr2=0x7f87f4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f406b510 0x7f87f4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 -- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f41075d0 msgr2=0x7f87f419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f41075d0 0x7f87f419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 -- 172.21.15.69:0/519243271 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f419d060 con 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f970d700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 0x7f87f4198050 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f87f000eb30 tx=0x7f87f000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 10 client.?.objecter ms_handle_connect 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f000cc90 con 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f87f000ce30 con 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f419d350 con 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f0003df0 con 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 10 client.43931.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f41a4bd0 con 0x7f87f406ab60 2024-04-02T05:04:38.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 librados: init done 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f404ff00 con 0x7f87f406ab60 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f87f0003f90 con 0x7f87f406ab60 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f87e006f5e0 0x7f87e0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(910..910 src has 795..910) ==== 8870+0+0 (secure 0 0 0) 0x7f87f008e820 con 0x7f87f406ab60 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 10 client.43931.objecter ms_dispatch 0x7f87f4001550 osd_map(910..910 src has 795..910) 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 3 client.43931.objecter handle_osd_map got epochs [910,910] > 0 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87e67fc700 3 client.43931.objecter handle_osd_map decoding full epoch 910 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.036+0000 7f87f8f0c700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f87e006f5e0 0x7f87e0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.040+0000 7f87e67fc700 20 client.43931.objecter dump_active .. 0 homeless 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.040+0000 7f87f8f0c700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f87e006f5e0 0x7f87e0071aa0 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7f87e8006010 tx=0x7f87e801a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:38.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.040+0000 7f87e67fc700 10 client.43931.objecter ms_handle_connect 0x7f87e006f5e0 2024-04-02T05:04:38.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.044+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f87f005b900 con 0x7f87f406ab60 2024-04-02T05:04:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:36.400411+0000 mgr.y (mgr.14538) 804 : cluster 0 pgmap v1405: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:36.561126+0000 osd.0 (osd.0) 455 : cluster 0 4.1e scrub starts 2024-04-02T05:04:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:36.563496+0000 osd.0 (osd.0) 456 : cluster 0 4.1e scrub ok 2024-04-02T05:04:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:37.053874+0000 osd.5 (osd.5) 285 : cluster 0 4.1 scrub starts 2024-04-02T05:04:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:37.056289+0000 osd.5 (osd.5) 286 : cluster 0 4.1 scrub ok 2024-04-02T05:04:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:37.549443+0000 osd.3 (osd.3) 475 : cluster 0 4.1d scrub starts 2024-04-02T05:04:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:37 smithi098 bash[23790]: cluster 2024-04-02T05:04:37.551829+0000 osd.3 (osd.3) 476 : cluster 0 4.1d scrub ok 2024-04-02T05:04:38.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.272+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f87f41a4e60 con 0x7f87f406ab60 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:36.400411+0000 mgr.y (mgr.14538) 804 : cluster 0 pgmap v1405: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:36.561126+0000 osd.0 (osd.0) 455 : cluster 0 4.1e scrub starts 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:36.563496+0000 osd.0 (osd.0) 456 : cluster 0 4.1e scrub ok 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:37.053874+0000 osd.5 (osd.5) 285 : cluster 0 4.1 scrub starts 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:37.056289+0000 osd.5 (osd.5) 286 : cluster 0 4.1 scrub ok 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:37.549443+0000 osd.3 (osd.3) 475 : cluster 0 4.1d scrub starts 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[28904]: cluster 2024-04-02T05:04:37.551829+0000 osd.3 (osd.3) 476 : cluster 0 4.1d scrub ok 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:36.400411+0000 mgr.y (mgr.14538) 804 : cluster 0 pgmap v1405: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:36.561126+0000 osd.0 (osd.0) 455 : cluster 0 4.1e scrub starts 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:36.563496+0000 osd.0 (osd.0) 456 : cluster 0 4.1e scrub ok 2024-04-02T05:04:38.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:37.053874+0000 osd.5 (osd.5) 285 : cluster 0 4.1 scrub starts 2024-04-02T05:04:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:37.056289+0000 osd.5 (osd.5) 286 : cluster 0 4.1 scrub ok 2024-04-02T05:04:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:37.549443+0000 osd.3 (osd.3) 475 : cluster 0 4.1d scrub starts 2024-04-02T05:04:38.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:37 smithi069 bash[21528]: cluster 2024-04-02T05:04:37.551829+0000 osd.3 (osd.3) 476 : cluster 0 4.1d scrub ok 2024-04-02T05:04:38.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.856+0000 7f87e67fc700 1 -- 172.21.15.69:0/519243271 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v911) ==== 113+0+0 (secure 0 0 0) 0x7f87f005f350 con 0x7f87f406ab60 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 10 librados: watch_flush enter 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 10 librados: watch_flush exit 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 20 client.43931.objecter shutdown clearing up homeless session... 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 10 client.43931.objecter successfully canceled tick 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f87e006f5e0 msgr2=0x7f87e0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f87e006f5e0 0x7f87e0071aa0 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7f87e8006010 tx=0x7f87e801a040 comp rx=0 tx=0).stop 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 msgr2=0x7f87f4198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 0x7f87f4198050 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f87f000eb30 tx=0x7f87f000ee40 comp rx=0 tx=0).stop 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 shutdown_connections 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f87e006f5e0 0x7f87e0071aa0 unknown :-1 s=CLOSED pgs=4034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f41075d0 0x7f87f419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f406ab60 0x7f87f4198050 secure :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f87f000eb30 tx=0x7f87f000ee40 comp rx=0 tx=0).stop 2024-04-02T05:04:38.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 --2- 172.21.15.69:0/519243271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f406b510 0x7f87f4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:38.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 >> 172.21.15.69:0/519243271 conn(0x7f87f4078190 msgr2=0x7f87f40fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:38.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 shutdown_connections 2024-04-02T05:04:38.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 -- 172.21.15.69:0/519243271 wait complete. 2024-04-02T05:04:38.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:38.860+0000 7f87fb971700 1 librados: shutdown 2024-04-02T05:04:38.868 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.0 to 1 (10000) 2024-04-02T05:04:39.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:04:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:04:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[21528]: audit 2024-04-02T05:04:38.277333+0000 mon.c (mon.2) 1169 : audit 1 from='client.? 172.21.15.69:0/519243271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[21528]: audit 2024-04-02T05:04:38.277910+0000 mon.a (mon.0) 4078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[21528]: cluster 2024-04-02T05:04:38.494772+0000 osd.1 (osd.1) 479 : cluster 0 6.e scrub starts 2024-04-02T05:04:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[21528]: cluster 2024-04-02T05:04:38.497295+0000 osd.1 (osd.1) 480 : cluster 0 6.e scrub ok 2024-04-02T05:04:39.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[28904]: audit 2024-04-02T05:04:38.277333+0000 mon.c (mon.2) 1169 : audit 1 from='client.? 172.21.15.69:0/519243271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[28904]: audit 2024-04-02T05:04:38.277910+0000 mon.a (mon.0) 4078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[28904]: cluster 2024-04-02T05:04:38.494772+0000 osd.1 (osd.1) 479 : cluster 0 6.e scrub starts 2024-04-02T05:04:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:38 smithi069 bash[28904]: cluster 2024-04-02T05:04:38.497295+0000 osd.1 (osd.1) 480 : cluster 0 6.e scrub ok 2024-04-02T05:04:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:38 smithi098 bash[23790]: audit 2024-04-02T05:04:38.277333+0000 mon.c (mon.2) 1169 : audit 1 from='client.? 172.21.15.69:0/519243271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:38 smithi098 bash[23790]: audit 2024-04-02T05:04:38.277910+0000 mon.a (mon.0) 4078 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:38 smithi098 bash[23790]: cluster 2024-04-02T05:04:38.494772+0000 osd.1 (osd.1) 479 : cluster 0 6.e scrub starts 2024-04-02T05:04:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:38 smithi098 bash[23790]: cluster 2024-04-02T05:04:38.497295+0000 osd.1 (osd.1) 480 : cluster 0 6.e scrub ok 2024-04-02T05:04:39.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 0 1 2024-04-02T05:04:40.013 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: cluster 2024-04-02T05:04:38.402158+0000 mgr.y (mgr.14538) 805 : cluster 0 pgmap v1406: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: cluster 2024-04-02T05:04:38.760726+0000 osd.6 (osd.6) 409 : cluster 0 10.16 scrub starts 2024-04-02T05:04:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: cluster 2024-04-02T05:04:38.763290+0000 osd.6 (osd.6) 410 : cluster 0 10.16 scrub ok 2024-04-02T05:04:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: audit 2024-04-02T05:04:38.852093+0000 mon.a (mon.0) 4079 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-02T05:04:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: cluster 2024-04-02T05:04:38.860924+0000 mon.a (mon.0) 4080 : cluster 0 osdmap e911: 8 total, 8 up, 8 in 2024-04-02T05:04:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: cluster 2024-04-02T05:04:39.083612+0000 osd.5 (osd.5) 287 : cluster 0 10.11 deep-scrub starts 2024-04-02T05:04:40.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:39 smithi098 bash[23790]: cluster 2024-04-02T05:04:39.087130+0000 osd.5 (osd.5) 288 : cluster 0 10.11 deep-scrub ok 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: cluster 2024-04-02T05:04:38.402158+0000 mgr.y (mgr.14538) 805 : cluster 0 pgmap v1406: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: cluster 2024-04-02T05:04:38.760726+0000 osd.6 (osd.6) 409 : cluster 0 10.16 scrub starts 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: cluster 2024-04-02T05:04:38.763290+0000 osd.6 (osd.6) 410 : cluster 0 10.16 scrub ok 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: audit 2024-04-02T05:04:38.852093+0000 mon.a (mon.0) 4079 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: cluster 2024-04-02T05:04:38.860924+0000 mon.a (mon.0) 4080 : cluster 0 osdmap e911: 8 total, 8 up, 8 in 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: cluster 2024-04-02T05:04:39.083612+0000 osd.5 (osd.5) 287 : cluster 0 10.11 deep-scrub starts 2024-04-02T05:04:40.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[21528]: cluster 2024-04-02T05:04:39.087130+0000 osd.5 (osd.5) 288 : cluster 0 10.11 deep-scrub ok 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: cluster 2024-04-02T05:04:38.402158+0000 mgr.y (mgr.14538) 805 : cluster 0 pgmap v1406: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: cluster 2024-04-02T05:04:38.760726+0000 osd.6 (osd.6) 409 : cluster 0 10.16 scrub starts 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: cluster 2024-04-02T05:04:38.763290+0000 osd.6 (osd.6) 410 : cluster 0 10.16 scrub ok 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: audit 2024-04-02T05:04:38.852093+0000 mon.a (mon.0) 4079 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: cluster 2024-04-02T05:04:38.860924+0000 mon.a (mon.0) 4080 : cluster 0 osdmap e911: 8 total, 8 up, 8 in 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: cluster 2024-04-02T05:04:39.083612+0000 osd.5 (osd.5) 287 : cluster 0 10.11 deep-scrub starts 2024-04-02T05:04:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:39 smithi069 bash[28904]: cluster 2024-04-02T05:04:39.087130+0000 osd.5 (osd.5) 288 : cluster 0 10.11 deep-scrub ok 2024-04-02T05:04:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:39.719132+0000 osd.6 (osd.6) 411 : cluster 0 4.1f scrub starts 2024-04-02T05:04:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:39.721896+0000 osd.6 (osd.6) 412 : cluster 0 4.1f scrub ok 2024-04-02T05:04:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:39.747898+0000 osd.2 (osd.2) 382 : cluster 0 4.f scrub starts 2024-04-02T05:04:41.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:39.749703+0000 osd.2 (osd.2) 383 : cluster 0 4.f scrub ok 2024-04-02T05:04:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:40.403244+0000 mgr.y (mgr.14538) 806 : cluster 0 pgmap v1408: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:40.514709+0000 osd.1 (osd.1) 481 : cluster 0 3.0 scrub starts 2024-04-02T05:04:41.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:41 smithi098 bash[23790]: cluster 2024-04-02T05:04:40.520358+0000 osd.1 (osd.1) 482 : cluster 0 3.0 scrub ok 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:39.719132+0000 osd.6 (osd.6) 411 : cluster 0 4.1f scrub starts 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:39.721896+0000 osd.6 (osd.6) 412 : cluster 0 4.1f scrub ok 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:39.747898+0000 osd.2 (osd.2) 382 : cluster 0 4.f scrub starts 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:39.749703+0000 osd.2 (osd.2) 383 : cluster 0 4.f scrub ok 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:40.403244+0000 mgr.y (mgr.14538) 806 : cluster 0 pgmap v1408: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:40.514709+0000 osd.1 (osd.1) 481 : cluster 0 3.0 scrub starts 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[28904]: cluster 2024-04-02T05:04:40.520358+0000 osd.1 (osd.1) 482 : cluster 0 3.0 scrub ok 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:39.719132+0000 osd.6 (osd.6) 411 : cluster 0 4.1f scrub starts 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:39.721896+0000 osd.6 (osd.6) 412 : cluster 0 4.1f scrub ok 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:39.747898+0000 osd.2 (osd.2) 382 : cluster 0 4.f scrub starts 2024-04-02T05:04:41.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:39.749703+0000 osd.2 (osd.2) 383 : cluster 0 4.f scrub ok 2024-04-02T05:04:41.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:40.403244+0000 mgr.y (mgr.14538) 806 : cluster 0 pgmap v1408: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:41.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:40.514709+0000 osd.1 (osd.1) 481 : cluster 0 3.0 scrub starts 2024-04-02T05:04:41.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:41 smithi069 bash[21528]: cluster 2024-04-02T05:04:40.520358+0000 osd.1 (osd.1) 482 : cluster 0 3.0 scrub ok 2024-04-02T05:04:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:42 smithi098 bash[23790]: cluster 2024-04-02T05:04:41.162592+0000 osd.5 (osd.5) 289 : cluster 0 2.16 scrub starts 2024-04-02T05:04:42.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:42 smithi098 bash[23790]: cluster 2024-04-02T05:04:41.164766+0000 osd.5 (osd.5) 290 : cluster 0 2.16 scrub ok 2024-04-02T05:04:42.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:42 smithi069 bash[21528]: cluster 2024-04-02T05:04:41.162592+0000 osd.5 (osd.5) 289 : cluster 0 2.16 scrub starts 2024-04-02T05:04:42.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:42 smithi069 bash[21528]: cluster 2024-04-02T05:04:41.164766+0000 osd.5 (osd.5) 290 : cluster 0 2.16 scrub ok 2024-04-02T05:04:42.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:42 smithi069 bash[28904]: cluster 2024-04-02T05:04:41.162592+0000 osd.5 (osd.5) 289 : cluster 0 2.16 scrub starts 2024-04-02T05:04:42.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:42 smithi069 bash[28904]: cluster 2024-04-02T05:04:41.164766+0000 osd.5 (osd.5) 290 : cluster 0 2.16 scrub ok 2024-04-02T05:04:43.122 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(osd.1.0:1555 322.6 e686/678 322:602f83fe:::foo:head v 686'3, mlcod=686'1)", 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:44.204431+0000", 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 298.91379004499998, 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016531698000000001, 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.123 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204431+0000", 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204431+0000", 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204432+0000", 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.471e-06 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204444+0000", 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1749999999999999e-05 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204466+0000", 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1466000000000001e-05 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204477+0000", 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-05 2024-04-02T05:04:43.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204565+0000", 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8807000000000005e-05 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.204587+0000", 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1959000000000001e-05 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.220910+0000", 2024-04-02T05:04:43.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016322782000000001 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:44.220963+0000", 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2533e-05 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:3 324.9 324:91bcb6b0:test-rados-api-smithi069-72349-113::foo:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T04:59:58.084894+0000", 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 285.033327187, 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040239569999999999, 2024-04-02T05:04:43.127 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084894+0000", 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.128 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084894+0000", 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084896+0000", 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.917e-06 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084903+0000", 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5440000000000001e-06 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084918+0000", 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4592e-05 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084929+0000", 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1243000000000001e-05 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.084986+0000", 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7083999999999998e-05 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.085119+0000", 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000132364 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.085229+0000", 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011078000000000001 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.085664+0000", 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00043415800000000003 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.087286+0000", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016220239999999999 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.088854+0000", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001568617 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.088886+0000", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2045999999999998e-05 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T04:59:58.088918+0000", 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1588000000000003e-05 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:474 323.3s0 323:df8c41d3:test-rados-api-smithi069-72349-121::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e737)", 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:47.100767+0000", 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 236.01745359500001, 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.02919327, 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.133 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 474 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.100767+0000", 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.134 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.100767+0000", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.100771+0000", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1300000000000001e-06 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.100784+0000", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3091999999999999e-05 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.100811+0000", 2024-04-02T05:04:43.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7696e-05 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.100841+0000", 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0108999999999999e-05 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.108589+0000", 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0077471780000000004 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.108598+0000", 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2269999999999999e-06 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.125779+0000", 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017181609 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.125937+0000", 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015709200000000001 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.126417+0000", 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00048000099999999997 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.129672+0000", 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003255913 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.129866+0000", 2024-04-02T05:04:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019375100000000001 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:47.129961+0000", 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4471999999999999e-05 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.7.394:28 323.3s0 323:df8c41d3:test-rados-api-smithi069-72349-121::foo:head [copy-from ver 8] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e740)", 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:50.156549+0000", 2024-04-02T05:04:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 232.961671577, 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039246009999999998, 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156549+0000", 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156549+0000", 2024-04-02T05:04:43.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156550+0000", 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8000000000000004e-07 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156556+0000", 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2109999999999997e-06 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156566+0000", 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0472000000000001e-05 2024-04-02T05:04:43.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156572+0000", 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9549999999999997e-06 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156619+0000", 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6981000000000003e-05 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.156698+0000", 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8827999999999998e-05 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.158811+0000", 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002112835 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.159156+0000", 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00034587799999999998 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.160413+0000", 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001256285 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:50.160474+0000", 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1276000000000001e-05 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3798 4.14 4:2b04a3e9:::notify.6:head [watch ping cookie 94452570220800 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-04-02T05:04:43.144 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:55.100961+0000", 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 228.01725993599999, 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0036123040000000002, 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3798 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.100961+0000", 2024-04-02T05:04:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.100961+0000", 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.100962+0000", 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6199999999999997e-07 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.146 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.100966+0000", 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3630000000000004e-06 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.100978+0000", 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1915e-05 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.100988+0000", 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8900000000000002e-06 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104305+0000", 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0033170399999999998 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104411+0000", 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000105605 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104573+0000", 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016272899999999999 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.148 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:3799 3.5 3:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94452576124928 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:55.104135+0000", 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 228.014085674, 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059629210000000004, 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 3799 2024-04-02T05:04:43.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104135+0000", 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104135+0000", 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.150 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104137+0000", 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3379999999999999e-06 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104142+0000", 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6579999999999999e-06 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104155+0000", 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2797e-05 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.151 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.104165+0000", 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5000000000000005e-06 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.109868+0000", 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057038560000000002 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.109969+0000", 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000100116 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.152 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:55.110098+0000", 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012965599999999999 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:648 329.1d 329:b9d75f00:test-rados-api-smithi069-72349-123::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e748)", 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:58.122023+0000", 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 224.99619803600001, 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082857190000000004, 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.153 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 648 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122023+0000", 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122023+0000", 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122025+0000", 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1660000000000001e-06 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122035+0000", 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0108999999999999e-05 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122055+0000", 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0024e-05 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122066+0000", 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1216999999999999e-05 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122138+0000", 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1645999999999998e-05 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.122336+0000", 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019737500000000001 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.123214+0000", 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00087831200000000002 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.123331+0000", 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000117407 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.130210+0000", 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0068788850000000004 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.130309+0000", 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8578000000000004e-05 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.158 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:723 329.1d 329:b9d75f00:test-rados-api-smithi069-72349-123::bar:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e748)", 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:00:58.179437+0000", 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 224.938784059, 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034526180000000002, 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 723 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179437+0000", 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179437+0000", 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179438+0000", 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1999999999999998e-07 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179443+0000", 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5310000000000001e-06 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179455+0000", 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1687e-05 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179462+0000", 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5270000000000004e-06 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179511+0000", 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9271999999999999e-05 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.179583+0000", 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1709000000000001e-05 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.182738+0000", 2024-04-02T05:04:43.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0031554529999999999 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.182816+0000", 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7490999999999994e-05 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:00:58.182890+0000", 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4028000000000003e-05 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.163 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:1794 336.14 336:2c2c3553:test-rados-api-smithi069-72349-137::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.099048+0000", 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 158.01917293899999, 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.032422624999999997, 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.164 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1794 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.099048+0000", 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.099048+0000", 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.099052+0000", 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9790000000000004e-06 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.104818+0000", 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057657689999999996 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.104861+0000", 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2840999999999999e-05 2024-04-02T05:04:43.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.104884+0000", 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2903000000000001e-05 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.104970+0000", 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6633999999999998e-05 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.105187+0000", 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000217128 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106614+0000", 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001426484 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106704+0000", 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0507999999999995e-05 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,2", 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.106844+0000", 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014006399999999999 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.115026+0000", 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0081818889999999995 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.118702+0000", 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0036760149999999999 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131327+0000", 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012624402 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131399+0000", 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2285000000000003e-05 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131471+0000", 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1724000000000006e-05 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:1795 336.14 336:2c2c3553:test-rados-api-smithi069-72349-137::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e813)", 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.131748+0000", 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 157.986472964, 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.039335666999999998, 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 1795 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131748+0000", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131748+0000", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131750+0000", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3520000000000001e-06 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131758+0000", 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7209999999999994e-06 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131776+0000", 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7944e-05 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131789+0000", 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2850999999999999e-05 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.131867+0000", 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8084999999999995e-05 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132011+0000", 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014349499999999999 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.170856+0000", 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.038845794000000003 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.170986+0000", 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012981900000000001 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.171084+0000", 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7606e-05 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.175 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.0.8:81 336.14 336:2c2c3553:test-rados-api-smithi069-72349-137::foo:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+flush+map_snap_clone+known_if_redirected+supports_pool_eio e813)", 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:05.132803+0000", 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 157.98541834599999, 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050858429999999996, 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132803+0000", 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.176 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132803+0000", 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132805+0000", 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9350000000000001e-06 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132814+0000", 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4369999999999996e-06 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132836+0000", 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1818999999999999e-05 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132850+0000", 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4453e-05 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.132937+0000", 2024-04-02T05:04:43.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6720999999999999e-05 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.133063+0000", 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012574400000000001 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:05.137889+0000", 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048257339999999999 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:2016 338.6 338:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:33.346530+0000", 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 129.771691027, 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038403080000000002, 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2016 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346530+0000", 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346530+0000", 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346532+0000", 2024-04-02T05:04:43.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1639999999999999e-06 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346550+0000", 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7830999999999999e-05 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346569+0000", 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9151e-05 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346583+0000", 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3995999999999999e-05 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.346698+0000", 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011524299999999999 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.348081+0000", 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001382936 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,6", 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.348496+0000", 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00041469599999999999 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.349385+0000", 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00088886200000000003 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.349590+0000", 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020499 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.350287+0000", 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000697542 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.350332+0000", 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4215999999999998e-05 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:33.350370+0000", 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8680999999999998e-05 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4011 4.14 4:2b04a3e9:::notify.6:head [watch ping cookie 94452570220800 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:40.180198+0000", 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 122.938023057, 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066344630000000002, 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.186 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4011 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180198+0000", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180198+0000", 2024-04-02T05:04:43.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180198+0000", 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4900000000000001e-07 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180201+0000", 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1600000000000001e-06 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180205+0000", 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.442e-06 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180209+0000", 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5279999999999999e-06 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180282+0000", 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3634999999999995e-05 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180399+0000", 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011668299999999999 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.186832+0000", 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064335659999999999 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4012 3.5 3:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94452576124928 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-04-02T05:04:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:02:40.180234+0000", 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 122.937987019, 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065786760000000003, 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4012 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180234+0000", 2024-04-02T05:04:43.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180234+0000", 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180235+0000", 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8800000000000002e-07 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180238+0000", 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.585e-06 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180246+0000", 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2790000000000001e-06 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180252+0000", 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2129999999999996e-06 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180327+0000", 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4833999999999995e-05 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.180461+0000", 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013400099999999999 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:02:40.186813+0000", 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063520759999999999 2024-04-02T05:04:43.194 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.32028.0:2771 341.16 341:6f82276c:test-rados-api-smithi069-72349-147::baz:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e870)", 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:03:16.165740+0000", 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 86.952481457000005, 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059326980000000001, 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.32028", 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:04:43.195 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 2771 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165740+0000", 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165740+0000", 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165741+0000", 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9e-06 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165747+0000", 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0170000000000002e-06 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165762+0000", 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4392999999999999e-05 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165769+0000", 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5719999999999997e-06 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.165839+0000", 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.9524999999999996e-05 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.166099+0000", 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025994499999999998 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.167685+0000", 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001586514 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.167741+0000", 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5321000000000001e-05 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.168837+0000", 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010957670000000001 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.168934+0000", 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7812999999999993e-05 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:03:16.171672+0000", 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0027379309999999999 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.200 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4183 3.17 3:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:01.633237+0000", 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.484983778999997, 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028077309999999999, 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4183 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633237+0000", 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633237+0000", 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633241+0000", 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9260000000000002e-06 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633263+0000", 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1943e-05 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633303+0000", 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9935000000000003e-05 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633323+0000", 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9743000000000001e-05 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633452+0000", 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000129107 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633687+0000", 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000235293 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.633993+0000", 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030578700000000002 2024-04-02T05:04:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.635049+0000", 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010565189999999999 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.635686+0000", 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00063684899999999999 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.635789+0000", 2024-04-02T05:04:43.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010247999999999999 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.635917+0000", 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012808000000000001 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:01.636045+0000", 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000128069 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4233 3.18 3:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:02.461514+0000", 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.656706593000003, 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0027937299999999999, 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4233 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.208 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461514+0000", 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461514+0000", 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461516+0000", 2024-04-02T05:04:43.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6300000000000001e-06 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461526+0000", 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0319999999999999e-05 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461549+0000", 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2512000000000001e-05 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461562+0000", 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2627e-05 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461683+0000", 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000121129 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.461917+0000", 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000234057 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:04:43.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.462425+0000", 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00050817899999999999 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.463367+0000", 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00094189299999999996 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.464059+0000", 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00069242999999999996 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.464152+0000", 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2429999999999994e-05 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.464243+0000", 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0888000000000001e-05 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.464308+0000", 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5635000000000004e-05 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.213 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4240 3.8 3:11d93ab7:::obj_delete_at_hint.0000000003:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:02.479373+0000", 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.638848222, 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028434480000000002, 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.214 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4240 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479373+0000", 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479373+0000", 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479375+0000", 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.063e-06 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479389+0000", 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.377e-05 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479417+0000", 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8671999999999999e-05 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479433+0000", 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6143000000000001e-05 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479585+0000", 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000151452 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.479874+0000", 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028942600000000002 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,7", 2024-04-02T05:04:43.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.480368+0000", 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00049348199999999995 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.481196+0000", 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000828526 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.481921+0000", 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00072433899999999995 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.482060+0000", 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013949899999999999 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.482146+0000", 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6330999999999997e-05 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.482216+0000", 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.9745000000000004e-05 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.219 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4518 3.8 3:116b4168:::obj_delete_at_hint.0000000095:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:02.979693+0000", 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.138528446000002, 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0031673090000000001, 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4518 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979693+0000", 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979693+0000", 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979694+0000", 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5379999999999999e-06 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979703+0000", 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8179999999999993e-06 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979724+0000", 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1299000000000001e-05 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979738+0000", 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3314e-05 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.979915+0000", 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017794499999999999 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.980247+0000", 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00033105 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 1,7", 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.980627+0000", 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00038006900000000002 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.981317+0000", 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00069036599999999998 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.982316+0000", 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00099867599999999999 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.982666+0000", 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00035065600000000003 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.982782+0000", 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011594700000000001 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:02.982860+0000", 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7631000000000003e-05 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.225 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.14415.0:4567 3.2 3:47403372:::obj_delete_at_hint.0000000112:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2024-04-02T05:04:03.077539+0000", 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.040681773000003, 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028631099999999999, 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "continuous": false, 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.14415", 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 4567 2024-04-02T05:04:43.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077539+0000", 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077539+0000", 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2024-04-02T05:04:43.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077542+0000", 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3929999999999998e-06 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077554+0000", 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2413e-05 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077582+0000", 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8266e-05 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2024-04-02T05:04:43.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077598+0000", 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5449000000000001e-05 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077740+0000", 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014216000000000001 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.077998+0000", 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025790699999999998 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:04:43.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.078507+0000", 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00050922800000000002 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op_commit", 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.079299+0000", 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079237999999999997 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.079966+0000", 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00066670399999999995 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.080080+0000", 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000113362 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.080262+0000", 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018262399999999999 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2024-04-02T05:04:03.080402+0000", 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014022399999999999 2024-04-02T05:04:43.231 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.232 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.232 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.232 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-02T05:04:43.232 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-02T05:04:43.232 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-02T05:04:43.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:43 smithi098 bash[23790]: cluster 2024-04-02T05:04:42.404104+0000 mgr.y (mgr.14538) 807 : cluster 0 pgmap v1409: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:43.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:43 smithi069 bash[28904]: cluster 2024-04-02T05:04:42.404104+0000 mgr.y (mgr.14538) 807 : cluster 0 pgmap v1409: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:43 smithi069 bash[21528]: cluster 2024-04-02T05:04:42.404104+0000 mgr.y (mgr.14538) 807 : cluster 0 pgmap v1409: 184 pgs: 184 active+clean; 582 KiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:43.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-02T05:04:44.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:04:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:04:44.396 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:44 smithi098 bash[23790]: cluster 2024-04-02T05:04:43.159672+0000 osd.5 (osd.5) 291 : cluster 0 6.a scrub starts 2024-04-02T05:04:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:44 smithi098 bash[23790]: cluster 2024-04-02T05:04:43.162119+0000 osd.5 (osd.5) 292 : cluster 0 6.a scrub ok 2024-04-02T05:04:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:44 smithi098 bash[23790]: cluster 2024-04-02T05:04:43.508050+0000 osd.3 (osd.3) 477 : cluster 0 6.b scrub starts 2024-04-02T05:04:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:44 smithi098 bash[23790]: cluster 2024-04-02T05:04:43.510406+0000 osd.3 (osd.3) 478 : cluster 0 6.b scrub ok 2024-04-02T05:04:44.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:44 smithi098 bash[23790]: audit 2024-04-02T05:04:43.830423+0000 mgr.y (mgr.14538) 808 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:44.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[28904]: cluster 2024-04-02T05:04:43.159672+0000 osd.5 (osd.5) 291 : cluster 0 6.a scrub starts 2024-04-02T05:04:44.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[28904]: cluster 2024-04-02T05:04:43.162119+0000 osd.5 (osd.5) 292 : cluster 0 6.a scrub ok 2024-04-02T05:04:44.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[28904]: cluster 2024-04-02T05:04:43.508050+0000 osd.3 (osd.3) 477 : cluster 0 6.b scrub starts 2024-04-02T05:04:44.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[28904]: cluster 2024-04-02T05:04:43.510406+0000 osd.3 (osd.3) 478 : cluster 0 6.b scrub ok 2024-04-02T05:04:44.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[28904]: audit 2024-04-02T05:04:43.830423+0000 mgr.y (mgr.14538) 808 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:44.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[21528]: cluster 2024-04-02T05:04:43.159672+0000 osd.5 (osd.5) 291 : cluster 0 6.a scrub starts 2024-04-02T05:04:44.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[21528]: cluster 2024-04-02T05:04:43.162119+0000 osd.5 (osd.5) 292 : cluster 0 6.a scrub ok 2024-04-02T05:04:44.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[21528]: cluster 2024-04-02T05:04:43.508050+0000 osd.3 (osd.3) 477 : cluster 0 6.b scrub starts 2024-04-02T05:04:44.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[21528]: cluster 2024-04-02T05:04:43.510406+0000 osd.3 (osd.3) 478 : cluster 0 6.b scrub ok 2024-04-02T05:04:44.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:44 smithi069 bash[21528]: audit 2024-04-02T05:04:43.830423+0000 mgr.y (mgr.14538) 808 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:45.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.405254+0000 mgr.y (mgr.14538) 809 : cluster 0 pgmap v1410: 184 pgs: 184 active+clean; 582 KiB data, 802 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:45.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.528378+0000 osd.3 (osd.3) 479 : cluster 0 4.17 scrub starts 2024-04-02T05:04:45.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.530812+0000 osd.3 (osd.3) 480 : cluster 0 4.17 scrub ok 2024-04-02T05:04:45.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.641605+0000 osd.1 (osd.1) 483 : cluster 0 10.18 scrub starts 2024-04-02T05:04:45.454 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.643833+0000 osd.1 (osd.1) 484 : cluster 0 10.18 scrub ok 2024-04-02T05:04:45.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.405254+0000 mgr.y (mgr.14538) 809 : cluster 0 pgmap v1410: 184 pgs: 184 active+clean; 582 KiB data, 802 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:45.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.528378+0000 osd.3 (osd.3) 479 : cluster 0 4.17 scrub starts 2024-04-02T05:04:45.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.530812+0000 osd.3 (osd.3) 480 : cluster 0 4.17 scrub ok 2024-04-02T05:04:45.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.641605+0000 osd.1 (osd.1) 483 : cluster 0 10.18 scrub starts 2024-04-02T05:04:45.454 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:45 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.643833+0000 osd.1 (osd.1) 484 : cluster 0 10.18 scrub ok 2024-04-02T05:04:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:45 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.405254+0000 mgr.y (mgr.14538) 809 : cluster 0 pgmap v1410: 184 pgs: 184 active+clean; 582 KiB data, 802 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:45 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.528378+0000 osd.3 (osd.3) 479 : cluster 0 4.17 scrub starts 2024-04-02T05:04:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:45 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.530812+0000 osd.3 (osd.3) 480 : cluster 0 4.17 scrub ok 2024-04-02T05:04:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:45 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.641605+0000 osd.1 (osd.1) 483 : cluster 0 10.18 scrub starts 2024-04-02T05:04:45.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:45 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.643833+0000 osd.1 (osd.1) 484 : cluster 0 10.18 scrub ok 2024-04-02T05:04:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:46 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.596499+0000 osd.6 (osd.6) 413 : cluster 0 10.7 scrub starts 2024-04-02T05:04:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:46 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.602477+0000 osd.6 (osd.6) 414 : cluster 0 10.7 scrub ok 2024-04-02T05:04:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:46 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.769165+0000 osd.2 (osd.2) 384 : cluster 0 10.5 deep-scrub starts 2024-04-02T05:04:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:46 smithi098 bash[23790]: cluster 2024-04-02T05:04:44.775650+0000 osd.2 (osd.2) 385 : cluster 0 10.5 deep-scrub ok 2024-04-02T05:04:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:46 smithi098 bash[23790]: cluster 2024-04-02T05:04:45.178353+0000 osd.5 (osd.5) 293 : cluster 0 2.c scrub starts 2024-04-02T05:04:46.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:46 smithi098 bash[23790]: cluster 2024-04-02T05:04:45.180753+0000 osd.5 (osd.5) 294 : cluster 0 2.c scrub ok 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.596499+0000 osd.6 (osd.6) 413 : cluster 0 10.7 scrub starts 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.602477+0000 osd.6 (osd.6) 414 : cluster 0 10.7 scrub ok 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.769165+0000 osd.2 (osd.2) 384 : cluster 0 10.5 deep-scrub starts 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[21528]: cluster 2024-04-02T05:04:44.775650+0000 osd.2 (osd.2) 385 : cluster 0 10.5 deep-scrub ok 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[21528]: cluster 2024-04-02T05:04:45.178353+0000 osd.5 (osd.5) 293 : cluster 0 2.c scrub starts 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[21528]: cluster 2024-04-02T05:04:45.180753+0000 osd.5 (osd.5) 294 : cluster 0 2.c scrub ok 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.596499+0000 osd.6 (osd.6) 413 : cluster 0 10.7 scrub starts 2024-04-02T05:04:46.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.602477+0000 osd.6 (osd.6) 414 : cluster 0 10.7 scrub ok 2024-04-02T05:04:46.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.769165+0000 osd.2 (osd.2) 384 : cluster 0 10.5 deep-scrub starts 2024-04-02T05:04:46.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[28904]: cluster 2024-04-02T05:04:44.775650+0000 osd.2 (osd.2) 385 : cluster 0 10.5 deep-scrub ok 2024-04-02T05:04:46.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[28904]: cluster 2024-04-02T05:04:45.178353+0000 osd.5 (osd.5) 293 : cluster 0 2.c scrub starts 2024-04-02T05:04:46.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:46 smithi069 bash[28904]: cluster 2024-04-02T05:04:45.180753+0000 osd.5 (osd.5) 294 : cluster 0 2.c scrub ok 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/2141769041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 msgr2=0x7fe13c107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- 172.21.15.69:0/2141769041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c107670 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009990 tx=0x7fe12c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/2141769041 shutdown_connections 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- 172.21.15.69:0/2141769041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 0x7fe13c10f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- 172.21.15.69:0/2141769041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c107670 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- 172.21.15.69:0/2141769041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe13c100310 0x7fe13c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/2141769041 >> 172.21.15.69:0/2141769041 conn(0x7fe13c0782b0 msgr2=0x7fe13c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:47.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/2141769041 shutdown_connections 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/2141769041 wait complete. 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 librados: starting msgr at 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 librados: starting objecter 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 Processor -- start 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- start start 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 librados: setting wanted keys 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 librados: calling monclient init 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe13c100310 0x7fe13c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 0x7fe13c1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c111a40 con 0x7fe13c100cd0 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1118c0 con 0x7fe13c100310 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe13c111bc0 con 0x7fe13c107290 2024-04-02T05:04:47.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe139d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 0x7fe13c1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe139d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 0x7fe13c1022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57448/0 (socket says 172.21.15.69:57448) 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c102800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38702/0 (socket says 172.21.15.69:38702) 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe139d9b700 1 -- 172.21.15.69:0/6260089 learned_addr learned my addr 172.21.15.69:0/6260089 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13a59c700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe13c100310 0x7fe13c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 -- 172.21.15.69:0/6260089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe13c100310 msgr2=0x7fe13c101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe13c100310 0x7fe13c101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 -- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 msgr2=0x7fe13c1022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 0x7fe13c1022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:47.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 -- 172.21.15.69:0/6260089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c1a6540 con 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe13ad9d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c102800 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7fe13400ef70 tx=0x7fe13400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe12b7fe700 10 client.?.objecter ms_handle_connect 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe12b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340092e0 con 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe13400c9f0 con 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c1a67d0 con 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340043e0 con 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 10 client.43937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c1a6d10 con 0x7fe13c107290 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 librados: init done 2024-04-02T05:04:47.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13c04ff00 con 0x7fe13c107290 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.308+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe13401b030 con 0x7fe13c107290 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe12406f5e0 0x7fe124071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(911..911 src has 795..911) ==== 8870+0+0 (secure 0 0 0) 0x7fe13408dcc0 con 0x7fe13c107290 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe13a59c700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe12406f5e0 0x7fe124071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 10 client.43937.objecter ms_dispatch 0x7fe13c001550 osd_map(911..911 src has 795..911) 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 3 client.43937.objecter handle_osd_map got epochs [911,911] > 0 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 3 client.43937.objecter handle_osd_map decoding full epoch 911 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 20 client.43937.objecter dump_active .. 0 homeless 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe12b7fe700 10 client.43937.objecter ms_handle_connect 0x7fe12406f5e0 2024-04-02T05:04:47.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.312+0000 7fe13a59c700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe12406f5e0 0x7fe124071aa0 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7fe12c006010 tx=0x7fe12c01a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:47.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.316+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe134022060 con 0x7fe13c107290 2024-04-02T05:04:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:45.549408+0000 osd.6 (osd.6) 415 : cluster 0 4.5 scrub starts 2024-04-02T05:04:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:45.551903+0000 osd.6 (osd.6) 416 : cluster 0 4.5 scrub ok 2024-04-02T05:04:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.219870+0000 osd.5 (osd.5) 295 : cluster 0 2.18 scrub starts 2024-04-02T05:04:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.222326+0000 osd.5 (osd.5) 296 : cluster 0 2.18 scrub ok 2024-04-02T05:04:47.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.405959+0000 mgr.y (mgr.14538) 810 : cluster 0 pgmap v1411: 184 pgs: 184 active+clean; 582 KiB data, 802 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:47.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.511946+0000 osd.6 (osd.6) 417 : cluster 0 2.17 deep-scrub starts 2024-04-02T05:04:47.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.514365+0000 osd.6 (osd.6) 418 : cluster 0 2.17 deep-scrub ok 2024-04-02T05:04:47.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.564916+0000 osd.3 (osd.3) 481 : cluster 0 2.1b scrub starts 2024-04-02T05:04:47.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:47 smithi098 bash[23790]: cluster 2024-04-02T05:04:46.567043+0000 osd.3 (osd.3) 482 : cluster 0 2.1b scrub ok 2024-04-02T05:04:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:45.549408+0000 osd.6 (osd.6) 415 : cluster 0 4.5 scrub starts 2024-04-02T05:04:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:45.551903+0000 osd.6 (osd.6) 416 : cluster 0 4.5 scrub ok 2024-04-02T05:04:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.219870+0000 osd.5 (osd.5) 295 : cluster 0 2.18 scrub starts 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.222326+0000 osd.5 (osd.5) 296 : cluster 0 2.18 scrub ok 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.405959+0000 mgr.y (mgr.14538) 810 : cluster 0 pgmap v1411: 184 pgs: 184 active+clean; 582 KiB data, 802 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.511946+0000 osd.6 (osd.6) 417 : cluster 0 2.17 deep-scrub starts 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.514365+0000 osd.6 (osd.6) 418 : cluster 0 2.17 deep-scrub ok 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.564916+0000 osd.3 (osd.3) 481 : cluster 0 2.1b scrub starts 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[21528]: cluster 2024-04-02T05:04:46.567043+0000 osd.3 (osd.3) 482 : cluster 0 2.1b scrub ok 2024-04-02T05:04:47.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:45.549408+0000 osd.6 (osd.6) 415 : cluster 0 4.5 scrub starts 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:45.551903+0000 osd.6 (osd.6) 416 : cluster 0 4.5 scrub ok 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.219870+0000 osd.5 (osd.5) 295 : cluster 0 2.18 scrub starts 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.222326+0000 osd.5 (osd.5) 296 : cluster 0 2.18 scrub ok 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.405959+0000 mgr.y (mgr.14538) 810 : cluster 0 pgmap v1411: 184 pgs: 184 active+clean; 582 KiB data, 802 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.511946+0000 osd.6 (osd.6) 417 : cluster 0 2.17 deep-scrub starts 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.514365+0000 osd.6 (osd.6) 418 : cluster 0 2.17 deep-scrub ok 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.564916+0000 osd.3 (osd.3) 481 : cluster 0 2.1b scrub starts 2024-04-02T05:04:47.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:47 smithi069 bash[28904]: cluster 2024-04-02T05:04:46.567043+0000 osd.3 (osd.3) 482 : cluster 0 2.1b scrub ok 2024-04-02T05:04:47.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:47.532+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) -- 0x7fe13c1a6f70 con 0x7fe13c107290 2024-04-02T05:04:48.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.224+0000 7fe12b7fe700 1 -- 172.21.15.69:0/6260089 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v912) ==== 133+0+0 (secure 0 0 0) 0x7fe134010040 con 0x7fe13c107290 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 10 librados: watch_flush enter 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 10 librados: watch_flush exit 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 20 client.43937.objecter shutdown clearing up homeless session... 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 10 client.43937.objecter successfully canceled tick 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe12406f5e0 msgr2=0x7fe124071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe12406f5e0 0x7fe124071aa0 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7fe12c006010 tx=0x7fe12c01a040 comp rx=0 tx=0).stop 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 msgr2=0x7fe13c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:48.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c102800 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7fe13400ef70 tx=0x7fe13400c3a0 comp rx=0 tx=0).stop 2024-04-02T05:04:48.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 shutdown_connections 2024-04-02T05:04:48.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe12406f5e0 0x7fe124071aa0 unknown :-1 s=CLOSED pgs=4035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:48.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe13c107290 0x7fe13c102800 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:48.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe13c100310 0x7fe13c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:48.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 --2- 172.21.15.69:0/6260089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe13c100cd0 0x7fe13c1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:48.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 >> 172.21.15.69:0/6260089 conn(0x7fe13c0782b0 msgr2=0x7fe13c104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:48.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 shutdown_connections 2024-04-02T05:04:48.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 -- 172.21.15.69:0/6260089 wait complete. 2024-04-02T05:04:48.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:48.232+0000 7fe140d0d700 1 librados: shutdown 2024-04-02T05:04:48.239 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-02T05:04:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:48 smithi098 bash[23790]: cluster 2024-04-02T05:04:47.525237+0000 osd.3 (osd.3) 483 : cluster 0 2.1e scrub starts 2024-04-02T05:04:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:48 smithi098 bash[23790]: cluster 2024-04-02T05:04:47.527246+0000 osd.3 (osd.3) 484 : cluster 0 2.1e scrub ok 2024-04-02T05:04:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:48 smithi098 bash[23790]: audit 2024-04-02T05:04:47.540001+0000 mon.c (mon.2) 1170 : audit 1 from='client.? 172.21.15.69:0/6260089' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:48.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:48 smithi098 bash[23790]: audit 2024-04-02T05:04:47.540438+0000 mon.a (mon.0) 4081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:48.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[21528]: cluster 2024-04-02T05:04:47.525237+0000 osd.3 (osd.3) 483 : cluster 0 2.1e scrub starts 2024-04-02T05:04:48.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[21528]: cluster 2024-04-02T05:04:47.527246+0000 osd.3 (osd.3) 484 : cluster 0 2.1e scrub ok 2024-04-02T05:04:48.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[21528]: audit 2024-04-02T05:04:47.540001+0000 mon.c (mon.2) 1170 : audit 1 from='client.? 172.21.15.69:0/6260089' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:48.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[21528]: audit 2024-04-02T05:04:47.540438+0000 mon.a (mon.0) 4081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:48.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[28904]: cluster 2024-04-02T05:04:47.525237+0000 osd.3 (osd.3) 483 : cluster 0 2.1e scrub starts 2024-04-02T05:04:48.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[28904]: cluster 2024-04-02T05:04:47.527246+0000 osd.3 (osd.3) 484 : cluster 0 2.1e scrub ok 2024-04-02T05:04:48.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[28904]: audit 2024-04-02T05:04:47.540001+0000 mon.c (mon.2) 1170 : audit 1 from='client.? 172.21.15.69:0/6260089' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:48.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[28904]: audit 2024-04-02T05:04:47.540438+0000 mon.a (mon.0) 4081 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-02T05:04:48.610 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:04:49.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:04:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:04:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:04:49.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 1 2024-04-02T05:04:49.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:47.668628+0000 osd.1 (osd.1) 485 : cluster 0 10.2 scrub starts 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:47.671153+0000 osd.1 (osd.1) 486 : cluster 0 10.2 scrub ok 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:47.859765+0000 osd.4 (osd.4) 456 : cluster 0 5.1 scrub starts 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:47.862293+0000 osd.4 (osd.4) 457 : cluster 0 5.1 scrub ok 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: audit 2024-04-02T05:04:48.219355+0000 mon.a (mon.0) 4082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.226607+0000 mon.a (mon.0) 4083 : cluster 0 osdmap e912: 8 total, 8 up, 8 in 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.406607+0000 mgr.y (mgr.14538) 811 : cluster 0 pgmap v1413: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.536441+0000 osd.3 (osd.3) 485 : cluster 0 6.c scrub starts 2024-04-02T05:04:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.539088+0000 osd.3 (osd.3) 486 : cluster 0 6.c scrub ok 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:47.668628+0000 osd.1 (osd.1) 485 : cluster 0 10.2 scrub starts 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:47.671153+0000 osd.1 (osd.1) 486 : cluster 0 10.2 scrub ok 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:47.859765+0000 osd.4 (osd.4) 456 : cluster 0 5.1 scrub starts 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:47.862293+0000 osd.4 (osd.4) 457 : cluster 0 5.1 scrub ok 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: audit 2024-04-02T05:04:48.219355+0000 mon.a (mon.0) 4082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.226607+0000 mon.a (mon.0) 4083 : cluster 0 osdmap e912: 8 total, 8 up, 8 in 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.406607+0000 mgr.y (mgr.14538) 811 : cluster 0 pgmap v1413: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.536441+0000 osd.3 (osd.3) 485 : cluster 0 6.c scrub starts 2024-04-02T05:04:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:49 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.539088+0000 osd.3 (osd.3) 486 : cluster 0 6.c scrub ok 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:47.668628+0000 osd.1 (osd.1) 485 : cluster 0 10.2 scrub starts 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:47.671153+0000 osd.1 (osd.1) 486 : cluster 0 10.2 scrub ok 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:47.859765+0000 osd.4 (osd.4) 456 : cluster 0 5.1 scrub starts 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:47.862293+0000 osd.4 (osd.4) 457 : cluster 0 5.1 scrub ok 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: audit 2024-04-02T05:04:48.219355+0000 mon.a (mon.0) 4082 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.226607+0000 mon.a (mon.0) 4083 : cluster 0 osdmap e912: 8 total, 8 up, 8 in 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.406607+0000 mgr.y (mgr.14538) 811 : cluster 0 pgmap v1413: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:04:49.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.536441+0000 osd.3 (osd.3) 485 : cluster 0 6.c scrub starts 2024-04-02T05:04:49.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:49 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.539088+0000 osd.3 (osd.3) 486 : cluster 0 6.c scrub ok 2024-04-02T05:04:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.473941+0000 osd.7 (osd.7) 275 : cluster 0 10.10 scrub starts 2024-04-02T05:04:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.476160+0000 osd.7 (osd.7) 276 : cluster 0 10.10 scrub ok 2024-04-02T05:04:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.713851+0000 osd.1 (osd.1) 487 : cluster 0 2.d scrub starts 2024-04-02T05:04:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:48.716150+0000 osd.1 (osd.1) 488 : cluster 0 2.d scrub ok 2024-04-02T05:04:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.132880+0000 osd.5 (osd.5) 297 : cluster 0 6.19 scrub starts 2024-04-02T05:04:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.135264+0000 osd.5 (osd.5) 298 : cluster 0 6.19 scrub ok 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.225077+0000 mon.a (mon.0) 4084 : cluster 0 osdmap e913: 8 total, 8 up, 8 in 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: audit 2024-04-02T05:04:49.399540+0000 mon.b (mon.1) 332 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.541563+0000 osd.3 (osd.3) 487 : cluster 0 6.6 scrub starts 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.545303+0000 osd.3 (osd.3) 488 : cluster 0 6.6 scrub ok 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.473941+0000 osd.7 (osd.7) 275 : cluster 0 10.10 scrub starts 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.476160+0000 osd.7 (osd.7) 276 : cluster 0 10.10 scrub ok 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.713851+0000 osd.1 (osd.1) 487 : cluster 0 2.d scrub starts 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:48.716150+0000 osd.1 (osd.1) 488 : cluster 0 2.d scrub ok 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.132880+0000 osd.5 (osd.5) 297 : cluster 0 6.19 scrub starts 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.135264+0000 osd.5 (osd.5) 298 : cluster 0 6.19 scrub ok 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.225077+0000 mon.a (mon.0) 4084 : cluster 0 osdmap e913: 8 total, 8 up, 8 in 2024-04-02T05:04:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: audit 2024-04-02T05:04:49.399540+0000 mon.b (mon.1) 332 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.541563+0000 osd.3 (osd.3) 487 : cluster 0 6.6 scrub starts 2024-04-02T05:04:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:50 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.545303+0000 osd.3 (osd.3) 488 : cluster 0 6.6 scrub ok 2024-04-02T05:04:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.473941+0000 osd.7 (osd.7) 275 : cluster 0 10.10 scrub starts 2024-04-02T05:04:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.476160+0000 osd.7 (osd.7) 276 : cluster 0 10.10 scrub ok 2024-04-02T05:04:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.713851+0000 osd.1 (osd.1) 487 : cluster 0 2.d scrub starts 2024-04-02T05:04:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:48.716150+0000 osd.1 (osd.1) 488 : cluster 0 2.d scrub ok 2024-04-02T05:04:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.132880+0000 osd.5 (osd.5) 297 : cluster 0 6.19 scrub starts 2024-04-02T05:04:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.135264+0000 osd.5 (osd.5) 298 : cluster 0 6.19 scrub ok 2024-04-02T05:04:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.225077+0000 mon.a (mon.0) 4084 : cluster 0 osdmap e913: 8 total, 8 up, 8 in 2024-04-02T05:04:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: audit 2024-04-02T05:04:49.399540+0000 mon.b (mon.1) 332 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:04:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.541563+0000 osd.3 (osd.3) 487 : cluster 0 6.6 scrub starts 2024-04-02T05:04:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:50 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.545303+0000 osd.3 (osd.3) 488 : cluster 0 6.6 scrub ok 2024-04-02T05:04:51.466 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:04:51.467 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:04:51.467 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T05:04:51.467 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.526409+0000 osd.7 (osd.7) 277 : cluster 0 3.f scrub starts 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.532482+0000 osd.7 (osd.7) 278 : cluster 0 3.f scrub ok 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.539232+0000 osd.6 (osd.6) 419 : cluster 0 1.5 scrub starts 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:49.542506+0000 osd.6 (osd.6) 420 : cluster 0 1.5 scrub ok 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.407365+0000 mgr.y (mgr.14538) 812 : cluster 0 pgmap v1415: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.508454+0000 osd.0 (osd.0) 457 : cluster 0 3.12 scrub starts 2024-04-02T05:04:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.512240+0000 osd.0 (osd.0) 458 : cluster 0 3.12 scrub ok 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.526409+0000 osd.7 (osd.7) 277 : cluster 0 3.f scrub starts 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.532482+0000 osd.7 (osd.7) 278 : cluster 0 3.f scrub ok 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.539232+0000 osd.6 (osd.6) 419 : cluster 0 1.5 scrub starts 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:49.542506+0000 osd.6 (osd.6) 420 : cluster 0 1.5 scrub ok 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.407365+0000 mgr.y (mgr.14538) 812 : cluster 0 pgmap v1415: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.508454+0000 osd.0 (osd.0) 457 : cluster 0 3.12 scrub starts 2024-04-02T05:04:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:51 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.512240+0000 osd.0 (osd.0) 458 : cluster 0 3.12 scrub ok 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.526409+0000 osd.7 (osd.7) 277 : cluster 0 3.f scrub starts 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.532482+0000 osd.7 (osd.7) 278 : cluster 0 3.f scrub ok 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.539232+0000 osd.6 (osd.6) 419 : cluster 0 1.5 scrub starts 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:49.542506+0000 osd.6 (osd.6) 420 : cluster 0 1.5 scrub ok 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.407365+0000 mgr.y (mgr.14538) 812 : cluster 0 pgmap v1415: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.508454+0000 osd.0 (osd.0) 457 : cluster 0 3.12 scrub starts 2024-04-02T05:04:51.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:51 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.512240+0000 osd.0 (osd.0) 458 : cluster 0 3.12 scrub ok 2024-04-02T05:04:52.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-02T05:04:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.529662+0000 osd.7 (osd.7) 279 : cluster 0 3.14 scrub starts 2024-04-02T05:04:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.536861+0000 osd.7 (osd.7) 280 : cluster 0 3.14 scrub ok 2024-04-02T05:04:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.759461+0000 osd.1 (osd.1) 489 : cluster 0 10.19 scrub starts 2024-04-02T05:04:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.763414+0000 osd.1 (osd.1) 490 : cluster 0 10.19 scrub ok 2024-04-02T05:04:52.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.910844+0000 osd.4 (osd.4) 458 : cluster 0 3.3 scrub starts 2024-04-02T05:04:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:50.915801+0000 osd.4 (osd.4) 459 : cluster 0 3.3 scrub ok 2024-04-02T05:04:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:51.606880+0000 osd.3 (osd.3) 489 : cluster 0 3.1e scrub starts 2024-04-02T05:04:52.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:52 smithi098 bash[23790]: cluster 2024-04-02T05:04:51.612236+0000 osd.3 (osd.3) 490 : cluster 0 3.1e scrub ok 2024-04-02T05:04:52.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.529662+0000 osd.7 (osd.7) 279 : cluster 0 3.14 scrub starts 2024-04-02T05:04:52.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.536861+0000 osd.7 (osd.7) 280 : cluster 0 3.14 scrub ok 2024-04-02T05:04:52.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.759461+0000 osd.1 (osd.1) 489 : cluster 0 10.19 scrub starts 2024-04-02T05:04:52.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.763414+0000 osd.1 (osd.1) 490 : cluster 0 10.19 scrub ok 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.910844+0000 osd.4 (osd.4) 458 : cluster 0 3.3 scrub starts 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:50.915801+0000 osd.4 (osd.4) 459 : cluster 0 3.3 scrub ok 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:51.606880+0000 osd.3 (osd.3) 489 : cluster 0 3.1e scrub starts 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[28904]: cluster 2024-04-02T05:04:51.612236+0000 osd.3 (osd.3) 490 : cluster 0 3.1e scrub ok 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.529662+0000 osd.7 (osd.7) 279 : cluster 0 3.14 scrub starts 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.536861+0000 osd.7 (osd.7) 280 : cluster 0 3.14 scrub ok 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.759461+0000 osd.1 (osd.1) 489 : cluster 0 10.19 scrub starts 2024-04-02T05:04:52.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.763414+0000 osd.1 (osd.1) 490 : cluster 0 10.19 scrub ok 2024-04-02T05:04:52.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.910844+0000 osd.4 (osd.4) 458 : cluster 0 3.3 scrub starts 2024-04-02T05:04:52.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:50.915801+0000 osd.4 (osd.4) 459 : cluster 0 3.3 scrub ok 2024-04-02T05:04:52.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:51.606880+0000 osd.3 (osd.3) 489 : cluster 0 3.1e scrub starts 2024-04-02T05:04:52.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:52 smithi069 bash[21528]: cluster 2024-04-02T05:04:51.612236+0000 osd.3 (osd.3) 490 : cluster 0 3.1e scrub ok 2024-04-02T05:04:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:51.726951+0000 osd.1 (osd.1) 491 : cluster 0 2.9 scrub starts 2024-04-02T05:04:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:51.729378+0000 osd.1 (osd.1) 492 : cluster 0 2.9 scrub ok 2024-04-02T05:04:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.408191+0000 mgr.y (mgr.14538) 813 : cluster 0 pgmap v1416: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.465634+0000 osd.0 (osd.0) 459 : cluster 0 2.1f scrub starts 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.468022+0000 osd.0 (osd.0) 460 : cluster 0 2.1f scrub ok 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.510181+0000 osd.7 (osd.7) 281 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.522890+0000 osd.7 (osd.7) 282 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.858265+0000 osd.2 (osd.2) 386 : cluster 0 1.14 scrub starts 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[21528]: cluster 2024-04-02T05:04:52.860841+0000 osd.2 (osd.2) 387 : cluster 0 1.14 scrub ok 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:51.726951+0000 osd.1 (osd.1) 491 : cluster 0 2.9 scrub starts 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:51.729378+0000 osd.1 (osd.1) 492 : cluster 0 2.9 scrub ok 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.408191+0000 mgr.y (mgr.14538) 813 : cluster 0 pgmap v1416: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.465634+0000 osd.0 (osd.0) 459 : cluster 0 2.1f scrub starts 2024-04-02T05:04:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.468022+0000 osd.0 (osd.0) 460 : cluster 0 2.1f scrub ok 2024-04-02T05:04:53.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.510181+0000 osd.7 (osd.7) 281 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:04:53.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.522890+0000 osd.7 (osd.7) 282 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:04:53.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.858265+0000 osd.2 (osd.2) 386 : cluster 0 1.14 scrub starts 2024-04-02T05:04:53.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:53 smithi069 bash[28904]: cluster 2024-04-02T05:04:52.860841+0000 osd.2 (osd.2) 387 : cluster 0 1.14 scrub ok 2024-04-02T05:04:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:51.726951+0000 osd.1 (osd.1) 491 : cluster 0 2.9 scrub starts 2024-04-02T05:04:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:51.729378+0000 osd.1 (osd.1) 492 : cluster 0 2.9 scrub ok 2024-04-02T05:04:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.408191+0000 mgr.y (mgr.14538) 813 : cluster 0 pgmap v1416: 184 pgs: 184 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:04:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.465634+0000 osd.0 (osd.0) 459 : cluster 0 2.1f scrub starts 2024-04-02T05:04:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.468022+0000 osd.0 (osd.0) 460 : cluster 0 2.1f scrub ok 2024-04-02T05:04:53.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.510181+0000 osd.7 (osd.7) 281 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:04:53.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.522890+0000 osd.7 (osd.7) 282 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:04:53.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.858265+0000 osd.2 (osd.2) 386 : cluster 0 1.14 scrub starts 2024-04-02T05:04:53.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[23790]: cluster 2024-04-02T05:04:52.860841+0000 osd.2 (osd.2) 387 : cluster 0 1.14 scrub ok 2024-04-02T05:04:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:04:54.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:04:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: cluster 2024-04-02T05:04:53.025240+0000 osd.5 (osd.5) 299 : cluster 0 3.1d scrub starts 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: cluster 2024-04-02T05:04:53.029614+0000 osd.5 (osd.5) 300 : cluster 0 3.1d scrub ok 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: cluster 2024-04-02T05:04:53.438919+0000 osd.6 (osd.6) 421 : cluster 0 4.a deep-scrub starts 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: cluster 2024-04-02T05:04:53.441293+0000 osd.6 (osd.6) 422 : cluster 0 4.a deep-scrub ok 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: cluster 2024-04-02T05:04:53.680187+0000 osd.3 (osd.3) 491 : cluster 0 2.19 scrub starts 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: cluster 2024-04-02T05:04:53.683379+0000 osd.3 (osd.3) 492 : cluster 0 2.19 scrub ok 2024-04-02T05:04:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:54 smithi098 bash[23790]: audit 2024-04-02T05:04:53.841689+0000 mgr.y (mgr.14538) 814 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: cluster 2024-04-02T05:04:53.025240+0000 osd.5 (osd.5) 299 : cluster 0 3.1d scrub starts 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: cluster 2024-04-02T05:04:53.029614+0000 osd.5 (osd.5) 300 : cluster 0 3.1d scrub ok 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: cluster 2024-04-02T05:04:53.438919+0000 osd.6 (osd.6) 421 : cluster 0 4.a deep-scrub starts 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: cluster 2024-04-02T05:04:53.441293+0000 osd.6 (osd.6) 422 : cluster 0 4.a deep-scrub ok 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: cluster 2024-04-02T05:04:53.680187+0000 osd.3 (osd.3) 491 : cluster 0 2.19 scrub starts 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: cluster 2024-04-02T05:04:53.683379+0000 osd.3 (osd.3) 492 : cluster 0 2.19 scrub ok 2024-04-02T05:04:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[21528]: audit 2024-04-02T05:04:53.841689+0000 mgr.y (mgr.14538) 814 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: cluster 2024-04-02T05:04:53.025240+0000 osd.5 (osd.5) 299 : cluster 0 3.1d scrub starts 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: cluster 2024-04-02T05:04:53.029614+0000 osd.5 (osd.5) 300 : cluster 0 3.1d scrub ok 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: cluster 2024-04-02T05:04:53.438919+0000 osd.6 (osd.6) 421 : cluster 0 4.a deep-scrub starts 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: cluster 2024-04-02T05:04:53.441293+0000 osd.6 (osd.6) 422 : cluster 0 4.a deep-scrub ok 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: cluster 2024-04-02T05:04:53.680187+0000 osd.3 (osd.3) 491 : cluster 0 2.19 scrub starts 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: cluster 2024-04-02T05:04:53.683379+0000 osd.3 (osd.3) 492 : cluster 0 2.19 scrub ok 2024-04-02T05:04:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:54 smithi069 bash[28904]: audit 2024-04-02T05:04:53.841689+0000 mgr.y (mgr.14538) 814 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:04:55.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:55 smithi069 bash[21528]: cluster 2024-04-02T05:04:54.410183+0000 mgr.y (mgr.14538) 815 : cluster 0 pgmap v1417: 184 pgs: 184 active+clean; 582 KiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:04:55.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:55 smithi069 bash[21528]: cluster 2024-04-02T05:04:54.458926+0000 osd.0 (osd.0) 461 : cluster 0 6.16 scrub starts 2024-04-02T05:04:55.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:55 smithi069 bash[21528]: cluster 2024-04-02T05:04:54.461280+0000 osd.0 (osd.0) 462 : cluster 0 6.16 scrub ok 2024-04-02T05:04:55.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:55 smithi069 bash[28904]: cluster 2024-04-02T05:04:54.410183+0000 mgr.y (mgr.14538) 815 : cluster 0 pgmap v1417: 184 pgs: 184 active+clean; 582 KiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:04:55.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:55 smithi069 bash[28904]: cluster 2024-04-02T05:04:54.458926+0000 osd.0 (osd.0) 461 : cluster 0 6.16 scrub starts 2024-04-02T05:04:55.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:55 smithi069 bash[28904]: cluster 2024-04-02T05:04:54.461280+0000 osd.0 (osd.0) 462 : cluster 0 6.16 scrub ok 2024-04-02T05:04:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:55 smithi098 bash[23790]: cluster 2024-04-02T05:04:54.410183+0000 mgr.y (mgr.14538) 815 : cluster 0 pgmap v1417: 184 pgs: 184 active+clean; 582 KiB data, 797 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:04:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:55 smithi098 bash[23790]: cluster 2024-04-02T05:04:54.458926+0000 osd.0 (osd.0) 461 : cluster 0 6.16 scrub starts 2024-04-02T05:04:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:55 smithi098 bash[23790]: cluster 2024-04-02T05:04:54.461280+0000 osd.0 (osd.0) 462 : cluster 0 6.16 scrub ok 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:54.899899+0000 osd.4 (osd.4) 460 : cluster 0 6.14 scrub starts 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:54.902021+0000 osd.4 (osd.4) 461 : cluster 0 6.14 scrub ok 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.317854+0000 mon.b (mon.1) 333 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.318582+0000 mon.b (mon.1) 334 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.318724+0000 mon.a (mon.0) 4085 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.319278+0000 mon.b (mon.1) 335 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.319932+0000 mon.b (mon.1) 336 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.320136+0000 mon.a (mon.0) 4086 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.320686+0000 mon.b (mon.1) 337 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.320774+0000 mon.a (mon.0) 4087 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]: dispatch 2024-04-02T05:04:56.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.321348+0000 mon.b (mon.1) 338 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]: dispatch 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.321411+0000 mon.a (mon.0) 4088 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.322007+0000 mon.b (mon.1) 339 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.322063+0000 mon.a (mon.0) 4089 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.322757+0000 mon.a (mon.0) 4090 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]: dispatch 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: audit 2024-04-02T05:04:55.323230+0000 mon.a (mon.0) 4091 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.428120+0000 osd.0 (osd.0) 463 : cluster 0 1.11 deep-scrub starts 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.430770+0000 osd.0 (osd.0) 464 : cluster 0 1.11 deep-scrub ok 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.461518+0000 osd.3 (osd.3) 493 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.461873+0000 osd.3 (osd.3) 494 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.859115+0000 osd.2 (osd.2) 388 : cluster 0 3.9 scrub starts 2024-04-02T05:04:56.608 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.863412+0000 osd.2 (osd.2) 389 : cluster 0 3.9 scrub ok 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:54.899899+0000 osd.4 (osd.4) 460 : cluster 0 6.14 scrub starts 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:54.902021+0000 osd.4 (osd.4) 461 : cluster 0 6.14 scrub ok 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.317854+0000 mon.b (mon.1) 333 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.318582+0000 mon.b (mon.1) 334 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.318724+0000 mon.a (mon.0) 4085 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.319278+0000 mon.b (mon.1) 335 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.319932+0000 mon.b (mon.1) 336 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.320136+0000 mon.a (mon.0) 4086 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.320686+0000 mon.b (mon.1) 337 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.320774+0000 mon.a (mon.0) 4087 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.321348+0000 mon.b (mon.1) 338 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]: dispatch 2024-04-02T05:04:56.609 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.321411+0000 mon.a (mon.0) 4088 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.322007+0000 mon.b (mon.1) 339 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.322063+0000 mon.a (mon.0) 4089 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.322757+0000 mon.a (mon.0) 4090 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]: dispatch 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: audit 2024-04-02T05:04:55.323230+0000 mon.a (mon.0) 4091 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.428120+0000 osd.0 (osd.0) 463 : cluster 0 1.11 deep-scrub starts 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.430770+0000 osd.0 (osd.0) 464 : cluster 0 1.11 deep-scrub ok 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.461518+0000 osd.3 (osd.3) 493 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.461873+0000 osd.3 (osd.3) 494 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.859115+0000 osd.2 (osd.2) 388 : cluster 0 3.9 scrub starts 2024-04-02T05:04:56.610 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:56 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.863412+0000 osd.2 (osd.2) 389 : cluster 0 3.9 scrub ok 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:54.899899+0000 osd.4 (osd.4) 460 : cluster 0 6.14 scrub starts 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:54.902021+0000 osd.4 (osd.4) 461 : cluster 0 6.14 scrub ok 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.317854+0000 mon.b (mon.1) 333 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.318582+0000 mon.b (mon.1) 334 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.318724+0000 mon.a (mon.0) 4085 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.319278+0000 mon.b (mon.1) 335 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.319932+0000 mon.b (mon.1) 336 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.320136+0000 mon.a (mon.0) 4086 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.320686+0000 mon.b (mon.1) 337 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.320774+0000 mon.a (mon.0) 4087 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.321348+0000 mon.b (mon.1) 338 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]: dispatch 2024-04-02T05:04:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.321411+0000 mon.a (mon.0) 4088 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.322007+0000 mon.b (mon.1) 339 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.322063+0000 mon.a (mon.0) 4089 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.322757+0000 mon.a (mon.0) 4090 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]: dispatch 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: audit 2024-04-02T05:04:55.323230+0000 mon.a (mon.0) 4091 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]: dispatch 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.428120+0000 osd.0 (osd.0) 463 : cluster 0 1.11 deep-scrub starts 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.430770+0000 osd.0 (osd.0) 464 : cluster 0 1.11 deep-scrub ok 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.461518+0000 osd.3 (osd.3) 493 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.461873+0000 osd.3 (osd.3) 494 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.859115+0000 osd.2 (osd.2) 388 : cluster 0 3.9 scrub starts 2024-04-02T05:04:56.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:56 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.863412+0000 osd.2 (osd.2) 389 : cluster 0 3.9 scrub ok 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 -- 172.21.15.69:0/3794786802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102320 msgr2=0x7fa4941027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 --2- 172.21.15.69:0/3794786802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102320 0x7fa4941027a0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7fa48800b0a0 tx=0x7fa48800b3b0 comp rx=0 tx=0).stop 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 -- 172.21.15.69:0/3794786802 shutdown_connections 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 --2- 172.21.15.69:0/3794786802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494102ce0 0x7fa49410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 --2- 172.21.15.69:0/3794786802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494108390 0x7fa494108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 --2- 172.21.15.69:0/3794786802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102320 0x7fa4941027a0 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 -- 172.21.15.69:0/3794786802 >> 172.21.15.69:0/3794786802 conn(0x7fa494079310 msgr2=0x7fa494079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 -- 172.21.15.69:0/3794786802 shutdown_connections 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.779+0000 7fa49b291700 1 -- 172.21.15.69:0/3794786802 wait complete. 2024-04-02T05:04:56.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 librados: starting msgr at 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 librados: starting objecter 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 Processor -- start 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- start start 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 librados: setting wanted keys 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 librados: calling monclient init 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 0x7fa49419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102ce0 0x7fa49419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494108390 0x7fa49419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa494111b50 con 0x7fa494102320 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4941119d0 con 0x7fa494108390 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fa494111cd0 con 0x7fa494102ce0 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 0x7fa49419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:56.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 0x7fa49419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45492/0 (socket says 172.21.15.69:45492) 2024-04-02T05:04:56.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 -- 172.21.15.69:0/352720913 learned_addr learned my addr 172.21.15.69:0/352720913 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:04:56.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49882c700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102ce0 0x7fa49419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49982e700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494108390 0x7fa49419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:56.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 -- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102ce0 msgr2=0x7fa49419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102ce0 0x7fa49419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 -- 172.21.15.69:0/352720913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494108390 msgr2=0x7fa49419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494108390 0x7fa49419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 -- 172.21.15.69:0/352720913 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49419f2b0 con 0x7fa494102320 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49982e700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494108390 0x7fa49419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49902d700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 0x7fa49419a2a0 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7fa48400e9a0 tx=0x7fa48400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 10 client.?.objecter ms_handle_connect 0x7fa494102320 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48400cbd0 con 0x7fa494102320 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49419f5a0 con 0x7fa494102320 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fa48400cd70 con 0x7fa494102320 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484010e60 con 0x7fa494102320 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 10 client.33879.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:04:56.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4941a6e20 con 0x7fa494102320 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 librados: init done 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa49404ff00 con 0x7fa494102320 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fa484010430 con 0x7fa494102320 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa47c06f5e0 0x7fa47c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(914..914 src has 795..914) ==== 9073+0+0 (secure 0 0 0) 0x7fa48408e8e0 con 0x7fa494102320 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 10 client.33879.objecter ms_dispatch 0x7fa494001550 osd_map(914..914 src has 795..914) 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 3 client.33879.objecter handle_osd_map got epochs [914,914] > 0 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.783+0000 7fa4927fc700 3 client.33879.objecter handle_osd_map decoding full epoch 914 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.787+0000 7fa49882c700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa47c06f5e0 0x7fa47c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.787+0000 7fa4927fc700 20 client.33879.objecter dump_active .. 0 homeless 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.787+0000 7fa4927fc700 10 client.33879.objecter ms_handle_connect 0x7fa47c06f5e0 2024-04-02T05:04:56.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.787+0000 7fa49882c700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa47c06f5e0 0x7fa47c071aa0 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7fa494079c50 tx=0x7fa48800b700 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:04:56.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:56.791+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa48405b870 con 0x7fa494102320 2024-04-02T05:04:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:04:57.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.019+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7fa4941a7050 con 0x7fa494102320 2024-04-02T05:04:57.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.307+0000 7fa4927fc700 1 -- 172.21.15.69:0/352720913 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v915) ==== 113+0+0 (secure 0 0 0) 0x7fa48405f2c0 con 0x7fa494102320 2024-04-02T05:04:57.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 10 librados: watch_flush enter 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 10 librados: watch_flush exit 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 20 client.33879.objecter shutdown clearing up homeless session... 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 10 client.33879.objecter successfully canceled tick 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa47c06f5e0 msgr2=0x7fa47c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa47c06f5e0 0x7fa47c071aa0 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7fa494079c50 tx=0x7fa48800b700 comp rx=0 tx=0).stop 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 msgr2=0x7fa49419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:04:57.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.311+0000 7fa49b291700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 0x7fa49419a2a0 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7fa48400e9a0 tx=0x7fa48400ecb0 comp rx=0 tx=0).stop 2024-04-02T05:04:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.319+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 shutdown_connections 2024-04-02T05:04:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.319+0000 7fa49b291700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fa47c06f5e0 0x7fa47c071aa0 unknown :-1 s=CLOSED pgs=4036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:57.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.319+0000 7fa49b291700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa494108390 0x7fa49419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.319+0000 7fa49b291700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa494102320 0x7fa49419a2a0 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.319+0000 7fa49b291700 1 --2- 172.21.15.69:0/352720913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa494102ce0 0x7fa49419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:04:57.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.319+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 >> 172.21.15.69:0/352720913 conn(0x7fa494079310 msgr2=0x7fa494074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:04:57.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.331+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 shutdown_connections 2024-04-02T05:04:57.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.331+0000 7fa49b291700 1 -- 172.21.15.69:0/352720913 wait complete. 2024-04-02T05:04:57.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:04:57.331+0000 7fa49b291700 1 librados: shutdown 2024-04-02T05:04:57.338 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.1 to 1 (10000) 2024-04-02T05:04:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.772577+0000 osd.1 (osd.1) 493 : cluster 0 2.10 scrub starts 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:55.774182+0000 osd.1 (osd.1) 494 : cluster 0 2.10 scrub ok 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.308496+0000 mon.a (mon.0) 4092 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.308787+0000 mon.a (mon.0) 4093 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.308929+0000 mon.a (mon.0) 4094 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.309058+0000 mon.a (mon.0) 4095 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.309270+0000 mon.a (mon.0) 4096 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.309468+0000 mon.a (mon.0) 4097 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:56.309598+0000 mon.a (mon.0) 4098 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:56.319002+0000 mon.a (mon.0) 4099 : cluster 0 osdmap e914: 8 total, 8 up, 8 in 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:56.410905+0000 mgr.y (mgr.14538) 816 : cluster 0 pgmap v1419: 184 pgs: 184 active+clean; 582 KiB data, 797 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:56.426301+0000 osd.7 (osd.7) 283 : cluster 0 10.1 scrub starts 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:56.429550+0000 osd.7 (osd.7) 284 : cluster 0 10.1 scrub ok 2024-04-02T05:04:57.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:56.699224+0000 osd.4 (osd.4) 462 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:57.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: cluster 2024-04-02T05:04:56.699583+0000 osd.4 (osd.4) 463 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:57.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:57 smithi098 bash[23790]: audit 2024-04-02T05:04:57.025340+0000 mon.a (mon.0) 4100 : audit 1 from='client.? 172.21.15.69:0/352720913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-02T05:04:57.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.772577+0000 osd.1 (osd.1) 493 : cluster 0 2.10 scrub starts 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:55.774182+0000 osd.1 (osd.1) 494 : cluster 0 2.10 scrub ok 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.308496+0000 mon.a (mon.0) 4092 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.308787+0000 mon.a (mon.0) 4093 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.308929+0000 mon.a (mon.0) 4094 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.309058+0000 mon.a (mon.0) 4095 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.309270+0000 mon.a (mon.0) 4096 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.309468+0000 mon.a (mon.0) 4097 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:56.309598+0000 mon.a (mon.0) 4098 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:56.319002+0000 mon.a (mon.0) 4099 : cluster 0 osdmap e914: 8 total, 8 up, 8 in 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:56.410905+0000 mgr.y (mgr.14538) 816 : cluster 0 pgmap v1419: 184 pgs: 184 active+clean; 582 KiB data, 797 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:04:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:56.426301+0000 osd.7 (osd.7) 283 : cluster 0 10.1 scrub starts 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:56.429550+0000 osd.7 (osd.7) 284 : cluster 0 10.1 scrub ok 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:56.699224+0000 osd.4 (osd.4) 462 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: cluster 2024-04-02T05:04:56.699583+0000 osd.4 (osd.4) 463 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[28904]: audit 2024-04-02T05:04:57.025340+0000 mon.a (mon.0) 4100 : audit 1 from='client.? 172.21.15.69:0/352720913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.772577+0000 osd.1 (osd.1) 493 : cluster 0 2.10 scrub starts 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:55.774182+0000 osd.1 (osd.1) 494 : cluster 0 2.10 scrub ok 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.308496+0000 mon.a (mon.0) 4092 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [5, 1]}]': finished 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.308787+0000 mon.a (mon.0) 4093 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.a", "id": [5, 4]}]': finished 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.308929+0000 mon.a (mon.0) 4094 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [6, 4]}]': finished 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.309058+0000 mon.a (mon.0) 4095 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]': finished 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.309270+0000 mon.a (mon.0) 4096 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-04-02T05:04:57.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.309468+0000 mon.a (mon.0) 4097 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 1]}]': finished 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:56.309598+0000 mon.a (mon.0) 4098 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:56.319002+0000 mon.a (mon.0) 4099 : cluster 0 osdmap e914: 8 total, 8 up, 8 in 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:56.410905+0000 mgr.y (mgr.14538) 816 : cluster 0 pgmap v1419: 184 pgs: 184 active+clean; 582 KiB data, 797 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:56.426301+0000 osd.7 (osd.7) 283 : cluster 0 10.1 scrub starts 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:56.429550+0000 osd.7 (osd.7) 284 : cluster 0 10.1 scrub ok 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:56.699224+0000 osd.4 (osd.4) 462 : cluster 0 purged_snaps scrub starts 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: cluster 2024-04-02T05:04:56.699583+0000 osd.4 (osd.4) 463 : cluster 0 purged_snaps scrub ok 2024-04-02T05:04:57.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:57 smithi069 bash[21528]: audit 2024-04-02T05:04:57.025340+0000 mon.a (mon.0) 4100 : audit 1 from='client.? 172.21.15.69:0/352720913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-02T05:04:58.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 1 1 2024-04-02T05:04:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.002343+0000 osd.5 (osd.5) 301 : cluster 0 3.1c scrub starts 2024-04-02T05:04:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.005247+0000 osd.5 (osd.5) 302 : cluster 0 3.1c scrub ok 2024-04-02T05:04:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: audit 2024-04-02T05:04:57.311731+0000 mon.a (mon.0) 4101 : audit 1 from='client.? 172.21.15.69:0/352720913' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-02T05:04:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.319690+0000 mon.a (mon.0) 4102 : cluster 0 osdmap e915: 8 total, 8 up, 8 in 2024-04-02T05:04:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.399052+0000 osd.0 (osd.0) 465 : cluster 0 4.3 scrub starts 2024-04-02T05:04:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.400745+0000 osd.0 (osd.0) 466 : cluster 0 4.3 scrub ok 2024-04-02T05:04:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.503422+0000 osd.6 (osd.6) 423 : cluster 0 2.7 scrub starts 2024-04-02T05:04:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.505824+0000 osd.6 (osd.6) 424 : cluster 0 2.7 scrub ok 2024-04-02T05:04:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:58 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.321637+0000 mon.a (mon.0) 4103 : cluster 0 osdmap e916: 8 total, 8 up, 8 in 2024-04-02T05:04:58.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.002343+0000 osd.5 (osd.5) 301 : cluster 0 3.1c scrub starts 2024-04-02T05:04:58.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.005247+0000 osd.5 (osd.5) 302 : cluster 0 3.1c scrub ok 2024-04-02T05:04:58.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: audit 2024-04-02T05:04:57.311731+0000 mon.a (mon.0) 4101 : audit 1 from='client.? 172.21.15.69:0/352720913' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.319690+0000 mon.a (mon.0) 4102 : cluster 0 osdmap e915: 8 total, 8 up, 8 in 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.399052+0000 osd.0 (osd.0) 465 : cluster 0 4.3 scrub starts 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.400745+0000 osd.0 (osd.0) 466 : cluster 0 4.3 scrub ok 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.503422+0000 osd.6 (osd.6) 423 : cluster 0 2.7 scrub starts 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.505824+0000 osd.6 (osd.6) 424 : cluster 0 2.7 scrub ok 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.321637+0000 mon.a (mon.0) 4103 : cluster 0 osdmap e916: 8 total, 8 up, 8 in 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.002343+0000 osd.5 (osd.5) 301 : cluster 0 3.1c scrub starts 2024-04-02T05:04:58.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.005247+0000 osd.5 (osd.5) 302 : cluster 0 3.1c scrub ok 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: audit 2024-04-02T05:04:57.311731+0000 mon.a (mon.0) 4101 : audit 1 from='client.? 172.21.15.69:0/352720913' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.319690+0000 mon.a (mon.0) 4102 : cluster 0 osdmap e915: 8 total, 8 up, 8 in 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.399052+0000 osd.0 (osd.0) 465 : cluster 0 4.3 scrub starts 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.400745+0000 osd.0 (osd.0) 466 : cluster 0 4.3 scrub ok 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.503422+0000 osd.6 (osd.6) 423 : cluster 0 2.7 scrub starts 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.505824+0000 osd.6 (osd.6) 424 : cluster 0 2.7 scrub ok 2024-04-02T05:04:58.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.321637+0000 mon.a (mon.0) 4103 : cluster 0 osdmap e916: 8 total, 8 up, 8 in 2024-04-02T05:04:59.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:04:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:04:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:04:59.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.836066+0000 osd.1 (osd.1) 495 : cluster 0 3.4 deep-scrub starts 2024-04-02T05:04:59.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.843566+0000 osd.1 (osd.1) 496 : cluster 0 3.4 deep-scrub ok 2024-04-02T05:04:59.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.935859+0000 osd.4 (osd.4) 464 : cluster 0 3.1a scrub starts 2024-04-02T05:04:59.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:57.940854+0000 osd.4 (osd.4) 465 : cluster 0 3.1a scrub ok 2024-04-02T05:04:59.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.407136+0000 osd.0 (osd.0) 467 : cluster 0 6.9 scrub starts 2024-04-02T05:04:59.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.408189+0000 osd.0 (osd.0) 468 : cluster 0 6.9 scrub ok 2024-04-02T05:04:59.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.411533+0000 mgr.y (mgr.14538) 817 : cluster 0 pgmap v1422: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T05:04:59.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.508933+0000 osd.6 (osd.6) 425 : cluster 0 1.f scrub starts 2024-04-02T05:04:59.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:04:59 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.511449+0000 osd.6 (osd.6) 426 : cluster 0 1.f scrub ok 2024-04-02T05:04:59.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.836066+0000 osd.1 (osd.1) 495 : cluster 0 3.4 deep-scrub starts 2024-04-02T05:04:59.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.843566+0000 osd.1 (osd.1) 496 : cluster 0 3.4 deep-scrub ok 2024-04-02T05:04:59.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.935859+0000 osd.4 (osd.4) 464 : cluster 0 3.1a scrub starts 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:57.940854+0000 osd.4 (osd.4) 465 : cluster 0 3.1a scrub ok 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.407136+0000 osd.0 (osd.0) 467 : cluster 0 6.9 scrub starts 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.408189+0000 osd.0 (osd.0) 468 : cluster 0 6.9 scrub ok 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.411533+0000 mgr.y (mgr.14538) 817 : cluster 0 pgmap v1422: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.508933+0000 osd.6 (osd.6) 425 : cluster 0 1.f scrub starts 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.511449+0000 osd.6 (osd.6) 426 : cluster 0 1.f scrub ok 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.836066+0000 osd.1 (osd.1) 495 : cluster 0 3.4 deep-scrub starts 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.843566+0000 osd.1 (osd.1) 496 : cluster 0 3.4 deep-scrub ok 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.935859+0000 osd.4 (osd.4) 464 : cluster 0 3.1a scrub starts 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:57.940854+0000 osd.4 (osd.4) 465 : cluster 0 3.1a scrub ok 2024-04-02T05:04:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.407136+0000 osd.0 (osd.0) 467 : cluster 0 6.9 scrub starts 2024-04-02T05:04:59.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.408189+0000 osd.0 (osd.0) 468 : cluster 0 6.9 scrub ok 2024-04-02T05:04:59.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.411533+0000 mgr.y (mgr.14538) 817 : cluster 0 pgmap v1422: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T05:04:59.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.508933+0000 osd.6 (osd.6) 425 : cluster 0 1.f scrub starts 2024-04-02T05:04:59.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:04:59 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.511449+0000 osd.6 (osd.6) 426 : cluster 0 1.f scrub ok 2024-04-02T05:05:00.334 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:00.334 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:05:00.334 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T05:05:00.335 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T05:05:00.335 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:00 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.760108+0000 osd.3 (osd.3) 495 : cluster 0 10.1b scrub starts 2024-04-02T05:05:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:00 smithi098 bash[23790]: cluster 2024-04-02T05:04:58.765214+0000 osd.3 (osd.3) 496 : cluster 0 10.1b scrub ok 2024-04-02T05:05:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:00 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.336198+0000 mon.a (mon.0) 4104 : cluster 3 Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-02T05:05:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:00 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.344555+0000 mon.a (mon.0) 4105 : cluster 0 osdmap e917: 8 total, 8 up, 8 in 2024-04-02T05:05:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:00 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.478980+0000 osd.7 (osd.7) 285 : cluster 0 2.0 deep-scrub starts 2024-04-02T05:05:00.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:00 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.481950+0000 osd.7 (osd.7) 286 : cluster 0 2.0 deep-scrub ok 2024-04-02T05:05:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.760108+0000 osd.3 (osd.3) 495 : cluster 0 10.1b scrub starts 2024-04-02T05:05:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[28904]: cluster 2024-04-02T05:04:58.765214+0000 osd.3 (osd.3) 496 : cluster 0 10.1b scrub ok 2024-04-02T05:05:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.336198+0000 mon.a (mon.0) 4104 : cluster 3 Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-02T05:05:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.344555+0000 mon.a (mon.0) 4105 : cluster 0 osdmap e917: 8 total, 8 up, 8 in 2024-04-02T05:05:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.478980+0000 osd.7 (osd.7) 285 : cluster 0 2.0 deep-scrub starts 2024-04-02T05:05:00.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.481950+0000 osd.7 (osd.7) 286 : cluster 0 2.0 deep-scrub ok 2024-04-02T05:05:00.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.760108+0000 osd.3 (osd.3) 495 : cluster 0 10.1b scrub starts 2024-04-02T05:05:00.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[21528]: cluster 2024-04-02T05:04:58.765214+0000 osd.3 (osd.3) 496 : cluster 0 10.1b scrub ok 2024-04-02T05:05:00.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.336198+0000 mon.a (mon.0) 4104 : cluster 3 Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-02T05:05:00.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.344555+0000 mon.a (mon.0) 4105 : cluster 0 osdmap e917: 8 total, 8 up, 8 in 2024-04-02T05:05:00.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.478980+0000 osd.7 (osd.7) 285 : cluster 0 2.0 deep-scrub starts 2024-04-02T05:05:00.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:00 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.481950+0000 osd.7 (osd.7) 286 : cluster 0 2.0 deep-scrub ok 2024-04-02T05:05:01.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-02T05:05:01.646 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.757711+0000 osd.3 (osd.3) 497 : cluster 0 3.7 scrub starts 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.762424+0000 osd.3 (osd.3) 498 : cluster 0 3.7 scrub ok 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.858937+0000 osd.4 (osd.4) 466 : cluster 0 2.f scrub starts 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.861561+0000 osd.1 (osd.1) 497 : cluster 0 6.4 scrub starts 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.862000+0000 osd.4 (osd.4) 467 : cluster 0 2.f scrub ok 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:04:59.863656+0000 osd.1 (osd.1) 498 : cluster 0 6.4 scrub ok 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.066769+0000 osd.5 (osd.5) 303 : cluster 0 4.1b scrub starts 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.069137+0000 osd.5 (osd.5) 304 : cluster 0 4.1b scrub ok 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.350620+0000 mon.a (mon.0) 4106 : cluster 0 osdmap e918: 8 total, 8 up, 8 in 2024-04-02T05:05:01.647 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:01 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.412414+0000 mgr.y (mgr.14538) 818 : cluster 0 pgmap v1425: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:01.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.757711+0000 osd.3 (osd.3) 497 : cluster 0 3.7 scrub starts 2024-04-02T05:05:01.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.762424+0000 osd.3 (osd.3) 498 : cluster 0 3.7 scrub ok 2024-04-02T05:05:01.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.858937+0000 osd.4 (osd.4) 466 : cluster 0 2.f scrub starts 2024-04-02T05:05:01.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.861561+0000 osd.1 (osd.1) 497 : cluster 0 6.4 scrub starts 2024-04-02T05:05:01.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.862000+0000 osd.4 (osd.4) 467 : cluster 0 2.f scrub ok 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:04:59.863656+0000 osd.1 (osd.1) 498 : cluster 0 6.4 scrub ok 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.066769+0000 osd.5 (osd.5) 303 : cluster 0 4.1b scrub starts 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.069137+0000 osd.5 (osd.5) 304 : cluster 0 4.1b scrub ok 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.350620+0000 mon.a (mon.0) 4106 : cluster 0 osdmap e918: 8 total, 8 up, 8 in 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.412414+0000 mgr.y (mgr.14538) 818 : cluster 0 pgmap v1425: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.757711+0000 osd.3 (osd.3) 497 : cluster 0 3.7 scrub starts 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.762424+0000 osd.3 (osd.3) 498 : cluster 0 3.7 scrub ok 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.858937+0000 osd.4 (osd.4) 466 : cluster 0 2.f scrub starts 2024-04-02T05:05:01.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.861561+0000 osd.1 (osd.1) 497 : cluster 0 6.4 scrub starts 2024-04-02T05:05:01.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.862000+0000 osd.4 (osd.4) 467 : cluster 0 2.f scrub ok 2024-04-02T05:05:01.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:04:59.863656+0000 osd.1 (osd.1) 498 : cluster 0 6.4 scrub ok 2024-04-02T05:05:01.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.066769+0000 osd.5 (osd.5) 303 : cluster 0 4.1b scrub starts 2024-04-02T05:05:01.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.069137+0000 osd.5 (osd.5) 304 : cluster 0 4.1b scrub ok 2024-04-02T05:05:01.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.350620+0000 mon.a (mon.0) 4106 : cluster 0 osdmap e918: 8 total, 8 up, 8 in 2024-04-02T05:05:01.770 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:01 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.412414+0000 mgr.y (mgr.14538) 818 : cluster 0 pgmap v1425: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.805432+0000 osd.3 (osd.3) 499 : cluster 0 6.1f deep-scrub starts 2024-04-02T05:05:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.807989+0000 osd.3 (osd.3) 500 : cluster 0 6.1f deep-scrub ok 2024-04-02T05:05:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.883055+0000 osd.4 (osd.4) 468 : cluster 0 4.e scrub starts 2024-04-02T05:05:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:00.885937+0000 osd.4 (osd.4) 469 : cluster 0 4.e scrub ok 2024-04-02T05:05:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:01.392774+0000 osd.0 (osd.0) 469 : cluster 0 3.1 scrub starts 2024-04-02T05:05:02.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:01.397692+0000 osd.0 (osd.0) 470 : cluster 0 3.1 scrub ok 2024-04-02T05:05:02.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:01.433046+0000 osd.7 (osd.7) 287 : cluster 0 10.1f scrub starts 2024-04-02T05:05:02.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:02 smithi098 bash[23790]: cluster 2024-04-02T05:05:01.439049+0000 osd.7 (osd.7) 288 : cluster 0 10.1f scrub ok 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.805432+0000 osd.3 (osd.3) 499 : cluster 0 6.1f deep-scrub starts 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.807989+0000 osd.3 (osd.3) 500 : cluster 0 6.1f deep-scrub ok 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.883055+0000 osd.4 (osd.4) 468 : cluster 0 4.e scrub starts 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:00.885937+0000 osd.4 (osd.4) 469 : cluster 0 4.e scrub ok 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:01.392774+0000 osd.0 (osd.0) 469 : cluster 0 3.1 scrub starts 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:01.397692+0000 osd.0 (osd.0) 470 : cluster 0 3.1 scrub ok 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:01.433046+0000 osd.7 (osd.7) 287 : cluster 0 10.1f scrub starts 2024-04-02T05:05:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[21528]: cluster 2024-04-02T05:05:01.439049+0000 osd.7 (osd.7) 288 : cluster 0 10.1f scrub ok 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.805432+0000 osd.3 (osd.3) 499 : cluster 0 6.1f deep-scrub starts 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.807989+0000 osd.3 (osd.3) 500 : cluster 0 6.1f deep-scrub ok 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.883055+0000 osd.4 (osd.4) 468 : cluster 0 4.e scrub starts 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:00.885937+0000 osd.4 (osd.4) 469 : cluster 0 4.e scrub ok 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:01.392774+0000 osd.0 (osd.0) 469 : cluster 0 3.1 scrub starts 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:01.397692+0000 osd.0 (osd.0) 470 : cluster 0 3.1 scrub ok 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:01.433046+0000 osd.7 (osd.7) 287 : cluster 0 10.1f scrub starts 2024-04-02T05:05:02.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:02 smithi069 bash[28904]: cluster 2024-04-02T05:05:01.439049+0000 osd.7 (osd.7) 288 : cluster 0 10.1f scrub ok 2024-04-02T05:05:02.951 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:03 smithi098 bash[23790]: cluster 2024-04-02T05:05:01.908276+0000 osd.4 (osd.4) 470 : cluster 0 1.a scrub starts 2024-04-02T05:05:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:03 smithi098 bash[23790]: cluster 2024-04-02T05:05:01.910910+0000 osd.4 (osd.4) 471 : cluster 0 1.a scrub ok 2024-04-02T05:05:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:03 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.413431+0000 mgr.y (mgr.14538) 819 : cluster 0 pgmap v1426: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:03 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.460331+0000 osd.6 (osd.6) 427 : cluster 0 1.7 deep-scrub starts 2024-04-02T05:05:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:03 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.462786+0000 osd.6 (osd.6) 428 : cluster 0 1.7 deep-scrub ok 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[21528]: cluster 2024-04-02T05:05:01.908276+0000 osd.4 (osd.4) 470 : cluster 0 1.a scrub starts 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[21528]: cluster 2024-04-02T05:05:01.910910+0000 osd.4 (osd.4) 471 : cluster 0 1.a scrub ok 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.413431+0000 mgr.y (mgr.14538) 819 : cluster 0 pgmap v1426: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.460331+0000 osd.6 (osd.6) 427 : cluster 0 1.7 deep-scrub starts 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.462786+0000 osd.6 (osd.6) 428 : cluster 0 1.7 deep-scrub ok 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[28904]: cluster 2024-04-02T05:05:01.908276+0000 osd.4 (osd.4) 470 : cluster 0 1.a scrub starts 2024-04-02T05:05:03.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[28904]: cluster 2024-04-02T05:05:01.910910+0000 osd.4 (osd.4) 471 : cluster 0 1.a scrub ok 2024-04-02T05:05:03.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.413431+0000 mgr.y (mgr.14538) 819 : cluster 0 pgmap v1426: 184 pgs: 6 peering, 1 remapped+peering, 177 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:03.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.460331+0000 osd.6 (osd.6) 427 : cluster 0 1.7 deep-scrub starts 2024-04-02T05:05:03.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:03 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.462786+0000 osd.6 (osd.6) 428 : cluster 0 1.7 deep-scrub ok 2024-04-02T05:05:04.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:05:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.557154+0000 osd.6 (osd.6) 429 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.557830+0000 osd.6 (osd.6) 430 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.886177+0000 osd.4 (osd.4) 472 : cluster 0 10.17 scrub starts 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[21528]: cluster 2024-04-02T05:05:02.892889+0000 osd.4 (osd.4) 473 : cluster 0 10.17 scrub ok 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[21528]: cluster 2024-04-02T05:05:03.390528+0000 mon.a (mon.0) 4107 : cluster 0 osdmap e919: 8 total, 8 up, 8 in 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[21528]: audit 2024-04-02T05:05:03.852350+0000 mgr.y (mgr.14538) 820 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.557154+0000 osd.6 (osd.6) 429 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.557830+0000 osd.6 (osd.6) 430 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.886177+0000 osd.4 (osd.4) 472 : cluster 0 10.17 scrub starts 2024-04-02T05:05:04.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[28904]: cluster 2024-04-02T05:05:02.892889+0000 osd.4 (osd.4) 473 : cluster 0 10.17 scrub ok 2024-04-02T05:05:04.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[28904]: cluster 2024-04-02T05:05:03.390528+0000 mon.a (mon.0) 4107 : cluster 0 osdmap e919: 8 total, 8 up, 8 in 2024-04-02T05:05:04.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:04 smithi069 bash[28904]: audit 2024-04-02T05:05:03.852350+0000 mgr.y (mgr.14538) 820 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:04 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.557154+0000 osd.6 (osd.6) 429 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:04 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.557830+0000 osd.6 (osd.6) 430 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:04 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.886177+0000 osd.4 (osd.4) 472 : cluster 0 10.17 scrub starts 2024-04-02T05:05:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:04 smithi098 bash[23790]: cluster 2024-04-02T05:05:02.892889+0000 osd.4 (osd.4) 473 : cluster 0 10.17 scrub ok 2024-04-02T05:05:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:04 smithi098 bash[23790]: cluster 2024-04-02T05:05:03.390528+0000 mon.a (mon.0) 4107 : cluster 0 osdmap e919: 8 total, 8 up, 8 in 2024-04-02T05:05:04.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:04 smithi098 bash[23790]: audit 2024-04-02T05:05:03.852350+0000 mgr.y (mgr.14538) 820 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:03.806515+0000 osd.3 (osd.3) 501 : cluster 0 6.11 scrub starts 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:03.808897+0000 osd.3 (osd.3) 502 : cluster 0 6.11 scrub ok 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:03.865944+0000 osd.1 (osd.1) 499 : cluster 0 1.3 scrub starts 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:03.868203+0000 osd.1 (osd.1) 500 : cluster 0 1.3 scrub ok 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.414581+0000 mgr.y (mgr.14538) 821 : cluster 0 pgmap v1428: 184 pgs: 184 active+clean; 582 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: audit 2024-04-02T05:05:04.439294+0000 mon.a (mon.0) 4108 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: audit 2024-04-02T05:05:04.442555+0000 mon.b (mon.1) 340 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:05.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.469338+0000 osd.0 (osd.0) 471 : cluster 0 6.10 scrub starts 2024-04-02T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.471068+0000 osd.0 (osd.0) 472 : cluster 0 6.10 scrub ok 2024-04-02T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.495338+0000 osd.6 (osd.6) 431 : cluster 0 10.a scrub starts 2024-04-02T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.497508+0000 osd.6 (osd.6) 432 : cluster 0 10.a scrub ok 2024-04-02T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.945957+0000 osd.5 (osd.5) 305 : cluster 0 1.12 scrub starts 2024-04-02T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:04.948418+0000 osd.5 (osd.5) 306 : cluster 0 1.12 scrub ok 2024-04-02T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:05 smithi098 bash[23790]: cluster 2024-04-02T05:05:05.382994+0000 mon.a (mon.0) 4109 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:03.806515+0000 osd.3 (osd.3) 501 : cluster 0 6.11 scrub starts 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:03.808897+0000 osd.3 (osd.3) 502 : cluster 0 6.11 scrub ok 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:03.865944+0000 osd.1 (osd.1) 499 : cluster 0 1.3 scrub starts 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:03.868203+0000 osd.1 (osd.1) 500 : cluster 0 1.3 scrub ok 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.414581+0000 mgr.y (mgr.14538) 821 : cluster 0 pgmap v1428: 184 pgs: 184 active+clean; 582 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: audit 2024-04-02T05:05:04.439294+0000 mon.a (mon.0) 4108 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: audit 2024-04-02T05:05:04.442555+0000 mon.b (mon.1) 340 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.469338+0000 osd.0 (osd.0) 471 : cluster 0 6.10 scrub starts 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.471068+0000 osd.0 (osd.0) 472 : cluster 0 6.10 scrub ok 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.495338+0000 osd.6 (osd.6) 431 : cluster 0 10.a scrub starts 2024-04-02T05:05:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.497508+0000 osd.6 (osd.6) 432 : cluster 0 10.a scrub ok 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.945957+0000 osd.5 (osd.5) 305 : cluster 0 1.12 scrub starts 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:04.948418+0000 osd.5 (osd.5) 306 : cluster 0 1.12 scrub ok 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[21528]: cluster 2024-04-02T05:05:05.382994+0000 mon.a (mon.0) 4109 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:03.806515+0000 osd.3 (osd.3) 501 : cluster 0 6.11 scrub starts 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:03.808897+0000 osd.3 (osd.3) 502 : cluster 0 6.11 scrub ok 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:03.865944+0000 osd.1 (osd.1) 499 : cluster 0 1.3 scrub starts 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:03.868203+0000 osd.1 (osd.1) 500 : cluster 0 1.3 scrub ok 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.414581+0000 mgr.y (mgr.14538) 821 : cluster 0 pgmap v1428: 184 pgs: 184 active+clean; 582 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: audit 2024-04-02T05:05:04.439294+0000 mon.a (mon.0) 4108 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: audit 2024-04-02T05:05:04.442555+0000 mon.b (mon.1) 340 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.469338+0000 osd.0 (osd.0) 471 : cluster 0 6.10 scrub starts 2024-04-02T05:05:05.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.471068+0000 osd.0 (osd.0) 472 : cluster 0 6.10 scrub ok 2024-04-02T05:05:05.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.495338+0000 osd.6 (osd.6) 431 : cluster 0 10.a scrub starts 2024-04-02T05:05:05.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.497508+0000 osd.6 (osd.6) 432 : cluster 0 10.a scrub ok 2024-04-02T05:05:05.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.945957+0000 osd.5 (osd.5) 305 : cluster 0 1.12 scrub starts 2024-04-02T05:05:05.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:04.948418+0000 osd.5 (osd.5) 306 : cluster 0 1.12 scrub ok 2024-04-02T05:05:05.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:05 smithi069 bash[28904]: cluster 2024-04-02T05:05:05.382994+0000 mon.a (mon.0) 4109 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-02T05:05:05.888 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2417048970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094102320 msgr2=0x7f30941027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2417048970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094102320 0x7f30941027a0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f307c009990 tx=0x7f307c009ca0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2417048970 shutdown_connections 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2417048970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 0x7f309410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2417048970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094108390 0x7f3094108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2417048970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094102320 0x7f30941027a0 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2417048970 >> 172.21.15.69:0/2417048970 conn(0x7f3094079310 msgr2=0x7f3094079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2417048970 shutdown_connections 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2417048970 wait complete. 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 librados: starting msgr at 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 librados: starting objecter 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 Processor -- start 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- start start 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 librados: setting wanted keys 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 librados: calling monclient init 2024-04-02T05:05:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 0x7f30941980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 0x7f3094198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094108390 0x7f309419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3094111b50 con 0x7f3094102320 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30941119d0 con 0x7f3094102ce0 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3094111cd0 con 0x7f3094108390 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 0x7f30941980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3092d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094108390 0x7f309419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 0x7f30941980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55710/0 (socket says 172.21.15.69:55710) 2024-04-02T05:05:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 -- 172.21.15.69:0/2586410567 learned_addr learned my addr 172.21.15.69:0/2586410567 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3091d9b700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 0x7f3094198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 -- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094108390 msgr2=0x7f309419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094108390 0x7f309419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 -- 172.21.15.69:0/2586410567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 msgr2=0x7f3094198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 0x7f3094198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 -- 172.21.15.69:0/2586410567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309419d0e0 con 0x7f3094102320 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3092d9d700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094108390 0x7f309419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3091d9b700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 0x7f3094198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:05:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f309259c700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 0x7f30941980d0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f308400e9a0 tx=0x7f308400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f308b7fe700 10 client.?.objecter ms_handle_connect 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f308b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308400cbd0 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f308400cd70 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309419d3d0 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 10 client.33885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30941a4c50 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 librados: init done 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084010e60 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3074005120 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.095+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3084010430 con 0x7f3094102320 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f308006f5e0 0x7f3080071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(919..919 src has 795..919) ==== 9044+0+0 (secure 0 0 0) 0x7f308408e7b0 con 0x7f3094102320 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f3091d9b700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f308006f5e0 0x7f3080071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 10 client.33885.objecter ms_dispatch 0x7f3094001550 osd_map(919..919 src has 795..919) 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 3 client.33885.objecter handle_osd_map got epochs [919,919] > 0 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 3 client.33885.objecter handle_osd_map decoding full epoch 919 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 20 client.33885.objecter dump_active .. 0 homeless 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f3091d9b700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f308006f5e0 0x7f3080071aa0 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 crypto rx=0x7f307c00b2e0 tx=0x7f307c005fd0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.099+0000 7f308b7fe700 10 client.33885.objecter ms_handle_connect 0x7f308006f5e0 2024-04-02T05:05:06.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.103+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f308405b7e0 con 0x7f3094102320 2024-04-02T05:05:06.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.327+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7f3074005c20 con 0x7f3094102320 2024-04-02T05:05:06.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f308b7fe700 1 -- 172.21.15.69:0/2586410567 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v920) ==== 133+0+0 (secure 0 0 0) 0x7f308405f230 con 0x7f3094102320 2024-04-02T05:05:06.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f3098ebc700 10 librados: watch_flush enter 2024-04-02T05:05:06.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f3098ebc700 10 librados: watch_flush exit 2024-04-02T05:05:06.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f3098ebc700 20 client.33885.objecter shutdown clearing up homeless session... 2024-04-02T05:05:06.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f3098ebc700 10 client.33885.objecter successfully canceled tick 2024-04-02T05:05:06.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f308006f5e0 msgr2=0x7f3080071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:06.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.439+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f308006f5e0 0x7f3080071aa0 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 crypto rx=0x7f307c00b2e0 tx=0x7f307c005fd0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.443+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 msgr2=0x7f30941980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:06.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.443+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 0x7f30941980d0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f308400e9a0 tx=0x7f308400ecb0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 shutdown_connections 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f308006f5e0 0x7f3080071aa0 unknown :-1 s=CLOSED pgs=4037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3094108390 0x7f309419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3094102320 0x7f30941980d0 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 --2- 172.21.15.69:0/2586410567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3094102ce0 0x7f3094198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 >> 172.21.15.69:0/2586410567 conn(0x7f3094079310 msgr2=0x7f3094109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 shutdown_connections 2024-04-02T05:05:06.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 -- 172.21.15.69:0/2586410567 wait complete. 2024-04-02T05:05:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:06.447+0000 7f3098ebc700 1 librados: shutdown 2024-04-02T05:05:06.452 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-02T05:05:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:06 smithi098 bash[23790]: cluster 2024-04-02T05:05:05.396413+0000 osd.7 (osd.7) 289 : cluster 0 6.12 scrub starts 2024-04-02T05:05:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:06 smithi098 bash[23790]: cluster 2024-04-02T05:05:05.399183+0000 osd.7 (osd.7) 290 : cluster 0 6.12 scrub ok 2024-04-02T05:05:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:06 smithi098 bash[23790]: cluster 2024-04-02T05:05:05.936782+0000 osd.5 (osd.5) 307 : cluster 0 1.1 scrub starts 2024-04-02T05:05:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:06 smithi098 bash[23790]: cluster 2024-04-02T05:05:05.938174+0000 osd.5 (osd.5) 308 : cluster 0 1.1 scrub ok 2024-04-02T05:05:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:06 smithi098 bash[23790]: audit 2024-04-02T05:05:06.328381+0000 mon.a (mon.0) 4110 : audit 1 from='client.? 172.21.15.69:0/2586410567' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-02T05:05:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[21528]: cluster 2024-04-02T05:05:05.396413+0000 osd.7 (osd.7) 289 : cluster 0 6.12 scrub starts 2024-04-02T05:05:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[21528]: cluster 2024-04-02T05:05:05.399183+0000 osd.7 (osd.7) 290 : cluster 0 6.12 scrub ok 2024-04-02T05:05:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[21528]: cluster 2024-04-02T05:05:05.936782+0000 osd.5 (osd.5) 307 : cluster 0 1.1 scrub starts 2024-04-02T05:05:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[21528]: cluster 2024-04-02T05:05:05.938174+0000 osd.5 (osd.5) 308 : cluster 0 1.1 scrub ok 2024-04-02T05:05:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[21528]: audit 2024-04-02T05:05:06.328381+0000 mon.a (mon.0) 4110 : audit 1 from='client.? 172.21.15.69:0/2586410567' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-02T05:05:06.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[28904]: cluster 2024-04-02T05:05:05.396413+0000 osd.7 (osd.7) 289 : cluster 0 6.12 scrub starts 2024-04-02T05:05:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[28904]: cluster 2024-04-02T05:05:05.399183+0000 osd.7 (osd.7) 290 : cluster 0 6.12 scrub ok 2024-04-02T05:05:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[28904]: cluster 2024-04-02T05:05:05.936782+0000 osd.5 (osd.5) 307 : cluster 0 1.1 scrub starts 2024-04-02T05:05:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[28904]: cluster 2024-04-02T05:05:05.938174+0000 osd.5 (osd.5) 308 : cluster 0 1.1 scrub ok 2024-04-02T05:05:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:06 smithi069 bash[28904]: audit 2024-04-02T05:05:06.328381+0000 mon.a (mon.0) 4110 : audit 1 from='client.? 172.21.15.69:0/2586410567' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-02T05:05:07.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 1 2024-04-02T05:05:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:07 smithi098 bash[23790]: cluster 2024-04-02T05:05:06.415352+0000 mgr.y (mgr.14538) 822 : cluster 0 pgmap v1429: 184 pgs: 184 active+clean; 582 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 723 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:07 smithi098 bash[23790]: audit 2024-04-02T05:05:06.440800+0000 mon.a (mon.0) 4111 : audit 1 from='client.? 172.21.15.69:0/2586410567' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-02T05:05:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:07 smithi098 bash[23790]: cluster 2024-04-02T05:05:06.449726+0000 mon.a (mon.0) 4112 : cluster 0 osdmap e920: 8 total, 8 up, 8 in 2024-04-02T05:05:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:07 smithi069 bash[21528]: cluster 2024-04-02T05:05:06.415352+0000 mgr.y (mgr.14538) 822 : cluster 0 pgmap v1429: 184 pgs: 184 active+clean; 582 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 723 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:07 smithi069 bash[21528]: audit 2024-04-02T05:05:06.440800+0000 mon.a (mon.0) 4111 : audit 1 from='client.? 172.21.15.69:0/2586410567' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-02T05:05:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:07 smithi069 bash[21528]: cluster 2024-04-02T05:05:06.449726+0000 mon.a (mon.0) 4112 : cluster 0 osdmap e920: 8 total, 8 up, 8 in 2024-04-02T05:05:07.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:07 smithi069 bash[28904]: cluster 2024-04-02T05:05:06.415352+0000 mgr.y (mgr.14538) 822 : cluster 0 pgmap v1429: 184 pgs: 184 active+clean; 582 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 723 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:07 smithi069 bash[28904]: audit 2024-04-02T05:05:06.440800+0000 mon.a (mon.0) 4111 : audit 1 from='client.? 172.21.15.69:0/2586410567' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-02T05:05:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:07 smithi069 bash[28904]: cluster 2024-04-02T05:05:06.449726+0000 mon.a (mon.0) 4112 : cluster 0 osdmap e920: 8 total, 8 up, 8 in 2024-04-02T05:05:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:08 smithi098 bash[23790]: cluster 2024-04-02T05:05:06.832050+0000 osd.4 (osd.4) 474 : cluster 0 10.1d deep-scrub starts 2024-04-02T05:05:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:08 smithi098 bash[23790]: cluster 2024-04-02T05:05:06.838835+0000 osd.4 (osd.4) 475 : cluster 0 10.1d deep-scrub ok 2024-04-02T05:05:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:08 smithi098 bash[23790]: cluster 2024-04-02T05:05:07.427381+0000 osd.0 (osd.0) 473 : cluster 0 4.6 scrub starts 2024-04-02T05:05:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:08 smithi098 bash[23790]: cluster 2024-04-02T05:05:07.429471+0000 osd.0 (osd.0) 474 : cluster 0 4.6 scrub ok 2024-04-02T05:05:08.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:05:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:05:08.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[21528]: cluster 2024-04-02T05:05:06.832050+0000 osd.4 (osd.4) 474 : cluster 0 10.1d deep-scrub starts 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[21528]: cluster 2024-04-02T05:05:06.838835+0000 osd.4 (osd.4) 475 : cluster 0 10.1d deep-scrub ok 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[21528]: cluster 2024-04-02T05:05:07.427381+0000 osd.0 (osd.0) 473 : cluster 0 4.6 scrub starts 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[21528]: cluster 2024-04-02T05:05:07.429471+0000 osd.0 (osd.0) 474 : cluster 0 4.6 scrub ok 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[28904]: cluster 2024-04-02T05:05:06.832050+0000 osd.4 (osd.4) 474 : cluster 0 10.1d deep-scrub starts 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[28904]: cluster 2024-04-02T05:05:06.838835+0000 osd.4 (osd.4) 475 : cluster 0 10.1d deep-scrub ok 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[28904]: cluster 2024-04-02T05:05:07.427381+0000 osd.0 (osd.0) 473 : cluster 0 4.6 scrub starts 2024-04-02T05:05:08.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:08 smithi069 bash[28904]: cluster 2024-04-02T05:05:07.429471+0000 osd.0 (osd.0) 474 : cluster 0 4.6 scrub ok 2024-04-02T05:05:09.038 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.16737.0:7 317.3s0 317:de000000::::head [pg-hitset-ls] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e663)", 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:50.089043+0000", 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 318.94444121499998, 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026114139999999998, 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.16737", 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/3364239127", 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089043+0000", 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089043+0000", 2024-04-02T05:05:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089047+0000", 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.143e-06 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089063+0000", 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5607000000000002e-05 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089101+0000", 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8569999999999998e-05 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.089144+0000", 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3092e-05 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.091551+0000", 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024064630000000002 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.091631+0000", 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9798999999999999e-05 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:50.091654+0000", 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.374e-05 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.043 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(10.7,QREQUEST,e:694,from:6,reservation_nonce:3)", 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:52.855180+0000", 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 316.178303761, 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059501500000000004, 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855180+0000", 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855180+0000", 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855183+0000", 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0959999999999999e-06 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855193+0000", 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.084e-05 2024-04-02T05:05:09.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855222+0000", 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8099999999999999e-05 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855236+0000", 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4752999999999999e-05 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855380+0000", 2024-04-02T05:05:09.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014421799999999999 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.855393+0000", 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2403e-05 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.861003+0000", 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056103050000000003 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.861032+0000", 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8949e-05 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:52.861131+0000", 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8486000000000005e-05 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3678 4.e 4:7759931f:::notify.3:head [watch ping cookie 94452570228864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e691)", 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:55.054230+0000", 2024-04-02T05:05:09.048 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 313.97925399399998, 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050602090000000004, 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3678 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054230+0000", 2024-04-02T05:05:09.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054230+0000", 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054233+0000", 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2399999999999999e-06 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054248+0000", 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4413000000000001e-05 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054293+0000", 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.545e-05 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.054319+0000", 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5363000000000001e-05 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.058860+0000", 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045414670000000004 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.059057+0000", 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000196615 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:55.059290+0000", 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000233661 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:34 324.1b 324:d8000000:%01:::head [pgnls start_epoch 702] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e702)", 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:00.108363+0000", 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 308.92512116699999, 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005403349, 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:05:09.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 34 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108363+0000", 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108363+0000", 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108365+0000", 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.779e-06 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108372+0000", 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7050000000000002e-06 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108385+0000", 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3147e-05 2024-04-02T05:05:09.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108395+0000", 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0002e-05 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108554+0000", 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015898600000000001 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.108564+0000", 2024-04-02T05:05:09.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0566e-05 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.113427+0000", 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004862936 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.113479+0000", 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1322000000000001e-05 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:00.113766+0000", 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028790599999999999 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.4,QREQUEST,e:703,from:3,reservation_nonce:2)", 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:01.789592+0000", 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 307.243892002, 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056095809999999998, 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789592+0000", 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789592+0000", 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789595+0000", 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6630000000000002e-06 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789609+0000", 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3986e-05 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789642+0000", 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2750000000000003e-05 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789659+0000", 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6976000000000001e-05 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789861+0000", 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020217100000000001 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.789874+0000", 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.278e-05 2024-04-02T05:05:09.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795023+0000", 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051492539999999998 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795074+0000", 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1669000000000003e-05 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795202+0000", 2024-04-02T05:05:09.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012733199999999999 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:3729 4.e 4:7759931f:::notify.3:head [watch ping cookie 94452570228864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:20.069107+0000", 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 288.96437757500001, 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036736690000000001, 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 3729 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069107+0000", 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069107+0000", 2024-04-02T05:05:09.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069109+0000", 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119999999999999e-06 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069126+0000", 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6566999999999999e-05 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069154+0000", 2024-04-02T05:05:09.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8282000000000001e-05 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.069167+0000", 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.295e-05 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072552+0000", 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033851459999999999 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072651+0000", 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9261999999999999e-05 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:20.072780+0000", 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012904999999999999 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1576 329.8 e748/744 329:16b7c28f:test-rados-api-smithi069-72349-123::foo:head v 748'1, mlcod=0'0)", 2024-04-02T05:05:09.067 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:58.112402+0000", 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 250.92108234299999, 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027641139999999998, 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112402+0000", 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112402+0000", 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112402+0000", 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0299999999999999e-07 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112411+0000", 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1100000000000003e-06 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112435+0000", 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3937000000000001e-05 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112450+0000", 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5109999999999999e-05 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112582+0000", 2024-04-02T05:05:09.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013190399999999999 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.112637+0000", 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5123000000000001e-05 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.115030+0000", 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023931450000000002 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:58.115166+0000", 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013618199999999999 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1695 335.10 335:08000000:%01:::head [pgnls start_epoch 805] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.158791+0000", 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 191.87469328200001, 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030532039999999999, 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1695 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158791+0000", 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158791+0000", 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158794+0000", 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.672e-06 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158806+0000", 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2221999999999999e-05 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158832+0000", 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6089999999999999e-05 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158843+0000", 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1216999999999999e-05 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158951+0000", 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107629 2024-04-02T05:05:09.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.158962+0000", 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1341e-05 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.161625+0000", 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026623760000000002 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.161667+0000", 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2537000000000003e-05 2024-04-02T05:05:09.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.161844+0000", 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017712000000000001 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1815 336.19 336:98000000:%01:::head [pgnls start_epoch 813] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e813)", 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:02:05.184719+0000", 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.84876567500001, 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043530929999999997, 2024-04-02T05:05:09.077 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1815 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184719+0000", 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184719+0000", 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184722+0000", 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0599999999999999e-06 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184738+0000", 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6146000000000001e-05 2024-04-02T05:05:09.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184776+0000", 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7836999999999997e-05 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184793+0000", 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7776000000000001e-05 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184917+0000", 2024-04-02T05:05:09.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000124022 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.184928+0000", 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0777e-05 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.188589+0000", 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036613079999999998 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.188677+0000", 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7799000000000004e-05 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.189072+0000", 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039436799999999998 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:2061 338.18 338:18000000::::head [pgnls start_epoch 836] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e836)", 2024-04-02T05:05:09.082 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:02:37.399476+0000", 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 151.63400859399999, 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033089920000000002, 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2061 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399476+0000", 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399476+0000", 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399478+0000", 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2220000000000001e-06 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399489+0000", 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1245e-05 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399513+0000", 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3841000000000001e-05 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399525+0000", 2024-04-02T05:05:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2278e-05 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399594+0000", 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8739999999999996e-05 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.399599+0000", 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9389999999999996e-06 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.402635+0000", 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030362060000000001 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.402669+0000", 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3961999999999998e-05 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:37.402785+0000", 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115559 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4195 3.3 3:c3a682d4:reshard::reshard.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:01.661080+0000", 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 67.372404329000005, 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029309639999999999, 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4195 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661080+0000", 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661080+0000", 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661083+0000", 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0340000000000001e-06 2024-04-02T05:05:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661098+0000", 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4616e-05 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661130+0000", 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2815999999999999e-05 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661147+0000", 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-05 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661302+0000", 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015451500000000001 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.661586+0000", 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000284343 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,6", 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.662117+0000", 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053112099999999996 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.663039+0000", 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00092165300000000003 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.663672+0000", 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00063336700000000002 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.663803+0000", 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000130438 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.663897+0000", 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3762000000000001e-05 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.664011+0000", 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114299 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4197 3.3 3:c3a682d4:reshard::reshard.0000000004:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:01.665565+0000", 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 67.367919455999996, 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026300970000000001, 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4197 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665565+0000", 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665565+0000", 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665567+0000", 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.75e-06 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665577+0000", 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0142999999999999e-05 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665599+0000", 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2027000000000001e-05 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665610+0000", 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1739999999999999e-05 2024-04-02T05:05:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665726+0000", 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116048 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.665998+0000", 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027139699999999999 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,6", 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.666487+0000", 2024-04-02T05:05:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048901499999999998 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.667287+0000", 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00080016400000000002 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.667849+0000", 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056195599999999996 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.667970+0000", 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012130599999999999 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.668086+0000", 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011549599999999999 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.668195+0000", 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010905500000000001 2024-04-02T05:05:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4228 3.1a 3:5c7e8a6d:reshard::reshard.0000000015:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:01.736260+0000", 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 67.297224094000001, 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030861450000000002, 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4228 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736260+0000", 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736260+0000", 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736262+0000", 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736275+0000", 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.306e-05 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736301+0000", 2024-04-02T05:05:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5622999999999998e-05 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736316+0000", 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.467e-05 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736461+0000", 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000144745 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.736745+0000", 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028435 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.737268+0000", 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052346399999999996 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.738139+0000", 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000870943 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.739001+0000", 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00086207799999999995 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.739183+0000", 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018149400000000001 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.739275+0000", 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2275999999999995e-05 2024-04-02T05:05:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.739346+0000", 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1179000000000004e-05 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4230 3.1a 3:5c7e8a6d:reshard::reshard.0000000015:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:01.741083+0000", 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 67.292400955999994, 2024-04-02T05:05:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027450360000000002, 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4230 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741083+0000", 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741083+0000", 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741086+0000", 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4339999999999999e-06 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741099+0000", 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3006e-05 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741132+0000", 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3306999999999998e-05 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741148+0000", 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6138999999999999e-05 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741276+0000", 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012766200000000001 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.741539+0000", 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000262811 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.742029+0000", 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049057400000000002 2024-04-02T05:05:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.742907+0000", 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00087741700000000002 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.743527+0000", 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00062004900000000001 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.743659+0000", 2024-04-02T05:05:09.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013206200000000001 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.743757+0000", 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8066000000000006e-05 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:01.743828+0000", 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1508999999999996e-05 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4279 3.1c 3:3b93e7b4:::obj_delete_at_hint.0000000016:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:02.564763+0000", 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 66.468720886, 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027939779999999999, 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4279 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564763+0000", 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564763+0000", 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564766+0000", 2024-04-02T05:05:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3429999999999998e-06 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564779+0000", 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3213e-05 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564810+0000", 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1170000000000001e-05 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564827+0000", 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6725999999999999e-05 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.564981+0000", 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000154258 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.565270+0000", 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000289101 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.565791+0000", 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052052999999999995 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.566679+0000", 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088873700000000001 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.567291+0000", 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061157599999999996 2024-04-02T05:05:09.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.567399+0000", 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010786899999999999 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.567483+0000", 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4395999999999996e-05 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.567557+0000", 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4059000000000004e-05 2024-04-02T05:05:09.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4329 3.1a 3:5ae4284b:::obj_delete_at_hint.0000000032:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:02.669847+0000", 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 66.363636810000003, 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028816900000000001, 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.119 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4329 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.669847+0000", 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.669847+0000", 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.669850+0000", 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5160000000000001e-06 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.669863+0000", 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3003000000000001e-05 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.669896+0000", 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3046000000000002e-05 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.669916+0000", 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0106999999999999e-05 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.670043+0000", 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012694900000000001 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.670301+0000", 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025748999999999998 2024-04-02T05:05:09.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.670812+0000", 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051098099999999998 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.671710+0000", 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089830300000000004 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.672309+0000", 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00059880700000000003 2024-04-02T05:05:09.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.672432+0000", 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012333799999999999 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.672601+0000", 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000168598 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:02.672729+0000", 2024-04-02T05:05:09.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012855200000000001 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4557 3.1d 3:bf899a2e:::obj_delete_at_hint.0000000108:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:03.054496+0000", 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.978988107000006, 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025983640000000001, 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4557 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054496+0000", 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054496+0000", 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054499+0000", 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4269999999999998e-06 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054511+0000", 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2643000000000001e-05 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054542+0000", 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1112999999999998e-05 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054558+0000", 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5840000000000001e-05 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054683+0000", 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012471800000000001 2024-04-02T05:05:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.054943+0000", 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000260201 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.055467+0000", 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052357700000000001 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.056314+0000", 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084760800000000002 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.056722+0000", 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040812500000000001 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.056931+0000", 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020873800000000001 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.057022+0000", 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1038999999999994e-05 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.057094+0000", 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2335000000000005e-05 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4560 3.3 3:c35966f0:::obj_delete_at_hint.0000000109:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:03.061188+0000", 2024-04-02T05:05:09.131 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.972296486000005, 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00268512, 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4560 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061188+0000", 2024-04-02T05:05:09.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061188+0000", 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061190+0000", 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4329999999999998e-06 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061203+0000", 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2643000000000001e-05 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061233+0000", 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0664000000000003e-05 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061249+0000", 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5823000000000001e-05 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061371+0000", 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012190500000000001 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.061598+0000", 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022633399999999999 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,6", 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.062062+0000", 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00046475200000000003 2024-04-02T05:05:09.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.062830+0000", 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00076746799999999999 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.063559+0000", 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072896799999999998 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:05:09.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.063678+0000", 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119165 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.063766+0000", 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8529000000000006e-05 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:03.063873+0000", 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106436 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4731 4.e 4:7759931f:::notify.3:head [watch reconnect cookie 94452570228864 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:56.314188+0000", 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.719295840000001, 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0291011800000001, 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4731 2024-04-02T05:05:09.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314188+0000", 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314188+0000", 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314192+0000", 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5200000000000002e-06 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314210+0000", 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8122999999999999e-05 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314264+0000", 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3813000000000001e-05 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314292+0000", 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8306e-05 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314537+0000", 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000245121 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.314558+0000", 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0429999999999999e-05 2024-04-02T05:05:09.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.327411+0000", 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012853172 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.327441+0000", 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0586000000000002e-05 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.317628+0000", 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99018678500000001 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.317646+0000", 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7994000000000001e-05 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.339452+0000", 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.021805415 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for degraded object", 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.339767+0000", 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031550099999999999 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.341796+0000", 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020283760000000001 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.341860+0000", 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4824000000000001e-05 2024-04-02T05:05:09.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.343290+0000", 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001429214 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4734 4.e 4:7759931f:::notify.3:head [watch reconnect cookie 94452570228864 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e915)", 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:57.318757+0000", 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 11.714727549999999, 2024-04-02T05:05:09.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0247434419999999, 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4734 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318757+0000", 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318757+0000", 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318758+0000", 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5269999999999999e-06 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318764+0000", 2024-04-02T05:05:09.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2330000000000003e-06 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318777+0000", 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3039e-05 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318788+0000", 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0219e-05 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318873+0000", 2024-04-02T05:05:09.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5731999999999998e-05 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.318890+0000", 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6823000000000001e-05 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.340068+0000", 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0211777070000001 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for degraded object", 2024-04-02T05:05:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.340107+0000", 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9144999999999999e-05 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.342306+0000", 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021986639999999999 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.342362+0000", 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5982e-05 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:58.343500+0000", 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001138371 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:09.151 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:09 smithi098 bash[23790]: cluster 2024-04-02T05:05:08.416727+0000 mgr.y (mgr.14538) 823 : cluster 0 pgmap v1431: 184 pgs: 184 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:09 smithi098 bash[23790]: cluster 2024-04-02T05:05:08.465562+0000 osd.6 (osd.6) 433 : cluster 0 2.1a scrub starts 2024-04-02T05:05:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:09 smithi098 bash[23790]: cluster 2024-04-02T05:05:08.467149+0000 osd.6 (osd.6) 434 : cluster 0 2.1a scrub ok 2024-04-02T05:05:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:09 smithi069 bash[21528]: cluster 2024-04-02T05:05:08.416727+0000 mgr.y (mgr.14538) 823 : cluster 0 pgmap v1431: 184 pgs: 184 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:09 smithi069 bash[21528]: cluster 2024-04-02T05:05:08.465562+0000 osd.6 (osd.6) 433 : cluster 0 2.1a scrub starts 2024-04-02T05:05:09.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:09 smithi069 bash[21528]: cluster 2024-04-02T05:05:08.467149+0000 osd.6 (osd.6) 434 : cluster 0 2.1a scrub ok 2024-04-02T05:05:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:09 smithi069 bash[28904]: cluster 2024-04-02T05:05:08.416727+0000 mgr.y (mgr.14538) 823 : cluster 0 pgmap v1431: 184 pgs: 184 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:05:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:09 smithi069 bash[28904]: cluster 2024-04-02T05:05:08.465562+0000 osd.6 (osd.6) 433 : cluster 0 2.1a scrub starts 2024-04-02T05:05:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:09 smithi069 bash[28904]: cluster 2024-04-02T05:05:08.467149+0000 osd.6 (osd.6) 434 : cluster 0 2.1a scrub ok 2024-04-02T05:05:09.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-02T05:05:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:10 smithi098 bash[23790]: cluster 2024-04-02T05:05:08.787728+0000 osd.3 (osd.3) 503 : cluster 0 2.e scrub starts 2024-04-02T05:05:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:10 smithi098 bash[23790]: cluster 2024-04-02T05:05:08.790261+0000 osd.3 (osd.3) 504 : cluster 0 2.e scrub ok 2024-04-02T05:05:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:10 smithi098 bash[23790]: cluster 2024-04-02T05:05:09.462236+0000 osd.7 (osd.7) 291 : cluster 0 2.1c deep-scrub starts 2024-04-02T05:05:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:10 smithi098 bash[23790]: cluster 2024-04-02T05:05:09.467898+0000 osd.7 (osd.7) 292 : cluster 0 2.1c deep-scrub ok 2024-04-02T05:05:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:10 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.184917+0000 mon.a (mon.0) 4113 : cluster 0 osdmap e921: 8 total, 8 up, 8 in 2024-04-02T05:05:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[21528]: cluster 2024-04-02T05:05:08.787728+0000 osd.3 (osd.3) 503 : cluster 0 2.e scrub starts 2024-04-02T05:05:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[21528]: cluster 2024-04-02T05:05:08.790261+0000 osd.3 (osd.3) 504 : cluster 0 2.e scrub ok 2024-04-02T05:05:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[21528]: cluster 2024-04-02T05:05:09.462236+0000 osd.7 (osd.7) 291 : cluster 0 2.1c deep-scrub starts 2024-04-02T05:05:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[21528]: cluster 2024-04-02T05:05:09.467898+0000 osd.7 (osd.7) 292 : cluster 0 2.1c deep-scrub ok 2024-04-02T05:05:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.184917+0000 mon.a (mon.0) 4113 : cluster 0 osdmap e921: 8 total, 8 up, 8 in 2024-04-02T05:05:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[28904]: cluster 2024-04-02T05:05:08.787728+0000 osd.3 (osd.3) 503 : cluster 0 2.e scrub starts 2024-04-02T05:05:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[28904]: cluster 2024-04-02T05:05:08.790261+0000 osd.3 (osd.3) 504 : cluster 0 2.e scrub ok 2024-04-02T05:05:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[28904]: cluster 2024-04-02T05:05:09.462236+0000 osd.7 (osd.7) 291 : cluster 0 2.1c deep-scrub starts 2024-04-02T05:05:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[28904]: cluster 2024-04-02T05:05:09.467898+0000 osd.7 (osd.7) 292 : cluster 0 2.1c deep-scrub ok 2024-04-02T05:05:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:10 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.184917+0000 mon.a (mon.0) 4113 : cluster 0 osdmap e921: 8 total, 8 up, 8 in 2024-04-02T05:05:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:09.869013+0000 osd.1 (osd.1) 501 : cluster 0 1.13 scrub starts 2024-04-02T05:05:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:09.871582+0000 osd.1 (osd.1) 502 : cluster 0 1.13 scrub ok 2024-04-02T05:05:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.417536+0000 mgr.y (mgr.14538) 824 : cluster 0 pgmap v1433: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-04-02T05:05:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.418290+0000 osd.7 (osd.7) 293 : cluster 0 10.e scrub starts 2024-04-02T05:05:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.426047+0000 osd.7 (osd.7) 294 : cluster 0 10.e scrub ok 2024-04-02T05:05:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.993373+0000 osd.5 (osd.5) 309 : cluster 0 10.6 scrub starts 2024-04-02T05:05:11.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:11 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.996208+0000 osd.5 (osd.5) 310 : cluster 0 10.6 scrub ok 2024-04-02T05:05:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:09.869013+0000 osd.1 (osd.1) 501 : cluster 0 1.13 scrub starts 2024-04-02T05:05:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:09.871582+0000 osd.1 (osd.1) 502 : cluster 0 1.13 scrub ok 2024-04-02T05:05:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.417536+0000 mgr.y (mgr.14538) 824 : cluster 0 pgmap v1433: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-04-02T05:05:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.418290+0000 osd.7 (osd.7) 293 : cluster 0 10.e scrub starts 2024-04-02T05:05:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.426047+0000 osd.7 (osd.7) 294 : cluster 0 10.e scrub ok 2024-04-02T05:05:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.993373+0000 osd.5 (osd.5) 309 : cluster 0 10.6 scrub starts 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.996208+0000 osd.5 (osd.5) 310 : cluster 0 10.6 scrub ok 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:09.869013+0000 osd.1 (osd.1) 501 : cluster 0 1.13 scrub starts 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:09.871582+0000 osd.1 (osd.1) 502 : cluster 0 1.13 scrub ok 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.417536+0000 mgr.y (mgr.14538) 824 : cluster 0 pgmap v1433: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.418290+0000 osd.7 (osd.7) 293 : cluster 0 10.e scrub starts 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.426047+0000 osd.7 (osd.7) 294 : cluster 0 10.e scrub ok 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.993373+0000 osd.5 (osd.5) 309 : cluster 0 10.6 scrub starts 2024-04-02T05:05:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:11 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.996208+0000 osd.5 (osd.5) 310 : cluster 0 10.6 scrub ok 2024-04-02T05:05:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:12.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.854380+0000 osd.1 (osd.1) 503 : cluster 0 4.2 scrub starts 2024-04-02T05:05:12.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[21528]: cluster 2024-04-02T05:05:10.856806+0000 osd.1 (osd.1) 504 : cluster 0 4.2 scrub ok 2024-04-02T05:05:12.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[21528]: cluster 2024-04-02T05:05:11.380201+0000 osd.7 (osd.7) 295 : cluster 0 3.e scrub starts 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[21528]: cluster 2024-04-02T05:05:11.384437+0000 osd.7 (osd.7) 296 : cluster 0 3.e scrub ok 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[21528]: cluster 2024-04-02T05:05:11.416367+0000 osd.0 (osd.0) 475 : cluster 0 10.4 scrub starts 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[21528]: cluster 2024-04-02T05:05:11.418936+0000 osd.0 (osd.0) 476 : cluster 0 10.4 scrub ok 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.854380+0000 osd.1 (osd.1) 503 : cluster 0 4.2 scrub starts 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[28904]: cluster 2024-04-02T05:05:10.856806+0000 osd.1 (osd.1) 504 : cluster 0 4.2 scrub ok 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[28904]: cluster 2024-04-02T05:05:11.380201+0000 osd.7 (osd.7) 295 : cluster 0 3.e scrub starts 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[28904]: cluster 2024-04-02T05:05:11.384437+0000 osd.7 (osd.7) 296 : cluster 0 3.e scrub ok 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[28904]: cluster 2024-04-02T05:05:11.416367+0000 osd.0 (osd.0) 475 : cluster 0 10.4 scrub starts 2024-04-02T05:05:12.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:12 smithi069 bash[28904]: cluster 2024-04-02T05:05:11.418936+0000 osd.0 (osd.0) 476 : cluster 0 10.4 scrub ok 2024-04-02T05:05:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:12 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.854380+0000 osd.1 (osd.1) 503 : cluster 0 4.2 scrub starts 2024-04-02T05:05:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:12 smithi098 bash[23790]: cluster 2024-04-02T05:05:10.856806+0000 osd.1 (osd.1) 504 : cluster 0 4.2 scrub ok 2024-04-02T05:05:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:12 smithi098 bash[23790]: cluster 2024-04-02T05:05:11.380201+0000 osd.7 (osd.7) 295 : cluster 0 3.e scrub starts 2024-04-02T05:05:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:12 smithi098 bash[23790]: cluster 2024-04-02T05:05:11.384437+0000 osd.7 (osd.7) 296 : cluster 0 3.e scrub ok 2024-04-02T05:05:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:12 smithi098 bash[23790]: cluster 2024-04-02T05:05:11.416367+0000 osd.0 (osd.0) 475 : cluster 0 10.4 scrub starts 2024-04-02T05:05:12.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:12 smithi098 bash[23790]: cluster 2024-04-02T05:05:11.418936+0000 osd.0 (osd.0) 476 : cluster 0 10.4 scrub ok 2024-04-02T05:05:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[21528]: cluster 2024-04-02T05:05:11.894526+0000 osd.1 (osd.1) 505 : cluster 0 1.b scrub starts 2024-04-02T05:05:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[21528]: cluster 2024-04-02T05:05:11.897219+0000 osd.1 (osd.1) 506 : cluster 0 1.b scrub ok 2024-04-02T05:05:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[21528]: cluster 2024-04-02T05:05:12.418068+0000 mgr.y (mgr.14538) 825 : cluster 0 pgmap v1434: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:05:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[21528]: cluster 2024-04-02T05:05:12.436127+0000 osd.6 (osd.6) 435 : cluster 0 1.6 scrub starts 2024-04-02T05:05:13.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[21528]: cluster 2024-04-02T05:05:12.438264+0000 osd.6 (osd.6) 436 : cluster 0 1.6 scrub ok 2024-04-02T05:05:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[28904]: cluster 2024-04-02T05:05:11.894526+0000 osd.1 (osd.1) 505 : cluster 0 1.b scrub starts 2024-04-02T05:05:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[28904]: cluster 2024-04-02T05:05:11.897219+0000 osd.1 (osd.1) 506 : cluster 0 1.b scrub ok 2024-04-02T05:05:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[28904]: cluster 2024-04-02T05:05:12.418068+0000 mgr.y (mgr.14538) 825 : cluster 0 pgmap v1434: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:05:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[28904]: cluster 2024-04-02T05:05:12.436127+0000 osd.6 (osd.6) 435 : cluster 0 1.6 scrub starts 2024-04-02T05:05:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:13 smithi069 bash[28904]: cluster 2024-04-02T05:05:12.438264+0000 osd.6 (osd.6) 436 : cluster 0 1.6 scrub ok 2024-04-02T05:05:13.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:13 smithi098 bash[23790]: cluster 2024-04-02T05:05:11.894526+0000 osd.1 (osd.1) 505 : cluster 0 1.b scrub starts 2024-04-02T05:05:13.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:13 smithi098 bash[23790]: cluster 2024-04-02T05:05:11.897219+0000 osd.1 (osd.1) 506 : cluster 0 1.b scrub ok 2024-04-02T05:05:13.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:13 smithi098 bash[23790]: cluster 2024-04-02T05:05:12.418068+0000 mgr.y (mgr.14538) 825 : cluster 0 pgmap v1434: 184 pgs: 184 active+clean; 582 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:05:13.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:13 smithi098 bash[23790]: cluster 2024-04-02T05:05:12.436127+0000 osd.6 (osd.6) 435 : cluster 0 1.6 scrub starts 2024-04-02T05:05:13.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:13 smithi098 bash[23790]: cluster 2024-04-02T05:05:12.438264+0000 osd.6 (osd.6) 436 : cluster 0 1.6 scrub ok 2024-04-02T05:05:14.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:05:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:05:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[21528]: cluster 2024-04-02T05:05:12.878871+0000 osd.1 (osd.1) 507 : cluster 0 2.15 scrub starts 2024-04-02T05:05:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[21528]: cluster 2024-04-02T05:05:12.881349+0000 osd.1 (osd.1) 508 : cluster 0 2.15 scrub ok 2024-04-02T05:05:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[21528]: cluster 2024-04-02T05:05:13.371082+0000 osd.7 (osd.7) 297 : cluster 0 6.5 scrub starts 2024-04-02T05:05:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[21528]: cluster 2024-04-02T05:05:13.373316+0000 osd.7 (osd.7) 298 : cluster 0 6.5 scrub ok 2024-04-02T05:05:14.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[28904]: cluster 2024-04-02T05:05:12.878871+0000 osd.1 (osd.1) 507 : cluster 0 2.15 scrub starts 2024-04-02T05:05:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[28904]: cluster 2024-04-02T05:05:12.881349+0000 osd.1 (osd.1) 508 : cluster 0 2.15 scrub ok 2024-04-02T05:05:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[28904]: cluster 2024-04-02T05:05:13.371082+0000 osd.7 (osd.7) 297 : cluster 0 6.5 scrub starts 2024-04-02T05:05:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:14 smithi069 bash[28904]: cluster 2024-04-02T05:05:13.373316+0000 osd.7 (osd.7) 298 : cluster 0 6.5 scrub ok 2024-04-02T05:05:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:14 smithi098 bash[23790]: cluster 2024-04-02T05:05:12.878871+0000 osd.1 (osd.1) 507 : cluster 0 2.15 scrub starts 2024-04-02T05:05:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:14 smithi098 bash[23790]: cluster 2024-04-02T05:05:12.881349+0000 osd.1 (osd.1) 508 : cluster 0 2.15 scrub ok 2024-04-02T05:05:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:14 smithi098 bash[23790]: cluster 2024-04-02T05:05:13.371082+0000 osd.7 (osd.7) 297 : cluster 0 6.5 scrub starts 2024-04-02T05:05:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:14 smithi098 bash[23790]: cluster 2024-04-02T05:05:13.373316+0000 osd.7 (osd.7) 298 : cluster 0 6.5 scrub ok 2024-04-02T05:05:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3718079353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c006ae40 msgr2=0x7f90c006b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3718079353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c006ae40 0x7f90c006b2c0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009990 tx=0x7f90b4009ca0 comp rx=0 tx=0).stop 2024-04-02T05:05:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3718079353 shutdown_connections 2024-04-02T05:05:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3718079353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c010ead0 0x7f90c0110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3718079353 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c0103160 0x7f90c006a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:14.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3718079353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c006ae40 0x7f90c006b2c0 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3718079353 >> 172.21.15.69:0/3718079353 conn(0x7f90c0077ea0 msgr2=0x7f90c00782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3718079353 shutdown_connections 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3718079353 wait complete. 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 librados: starting msgr at 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 librados: starting objecter 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 Processor -- start 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- start start 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 librados: setting wanted keys 2024-04-02T05:05:14.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 librados: calling monclient init 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 0x7f90c019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c0103160 0x7f90c019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c010ead0 0x7f90c019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90c0113c60 con 0x7f90c010ead0 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90c0113ae0 con 0x7f90c0103160 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f90c0113de0 con 0x7f90c006ae40 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 0x7f90c019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:14.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 0x7f90c019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:58690/0 (socket says 172.21.15.69:58690) 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 -- 172.21.15.69:0/3256864938 learned_addr learned my addr 172.21.15.69:0/3256864938 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c6221700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c010ead0 0x7f90c019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c521f700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c0103160 0x7f90c019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 -- 172.21.15.69:0/3256864938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c0103160 msgr2=0x7f90c019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c0103160 0x7f90c019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 -- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c010ead0 msgr2=0x7f90c019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c010ead0 0x7f90c019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 -- 172.21.15.69:0/3256864938 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c019f240 con 0x7f90c006ae40 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c521f700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c0103160 0x7f90c019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c5a20700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 0x7f90c019a280 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7f90b000e9a0 tx=0x7f90b000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90beffd700 10 client.?.objecter ms_handle_connect 0x7f90c006ae40 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90beffd700 10 client.?.objecter resend_mon_ops 2024-04-02T05:05:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b000cbd0 con 0x7f90c006ae40 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f90b000cd70 con 0x7f90c006ae40 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0010e60 con 0x7f90c006ae40 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c019f530 con 0x7f90c006ae40 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 10 client.43955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c01a6db0 con 0x7f90c006ae40 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.975+0000 7f90c7c84700 1 librados: init done 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90c004ff00 con 0x7f90c006ae40 2024-04-02T05:05:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f90b0010430 con 0x7f90c006ae40 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f90a806f5e0 0x7f90a8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(921..921 src has 795..921) ==== 9044+0+0 (secure 0 0 0) 0x7f90b008e8c0 con 0x7f90c006ae40 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 10 client.43955.objecter ms_dispatch 0x7f90c0001550 osd_map(921..921 src has 795..921) 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 3 client.43955.objecter handle_osd_map got epochs [921,921] > 0 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 3 client.43955.objecter handle_osd_map decoding full epoch 921 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90c521f700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f90a806f5e0 0x7f90a8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 20 client.43955.objecter dump_active .. 0 homeless 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90beffd700 10 client.43955.objecter ms_handle_connect 0x7f90a806f5e0 2024-04-02T05:05:14.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.979+0000 7f90c521f700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f90a806f5e0 0x7f90a8071aa0 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b2e0 tx=0x7f90b4005e80 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:14.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:14.983+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f90b005b870 con 0x7f90c006ae40 2024-04-02T05:05:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.207+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7f90c002db30 con 0x7f90c006ae40 2024-04-02T05:05:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.503+0000 7f90beffd700 1 -- 172.21.15.69:0/3256864938 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v922) ==== 113+0+0 (secure 0 0 0) 0x7f90b005f2c0 con 0x7f90c006ae40 2024-04-02T05:05:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 10 librados: watch_flush enter 2024-04-02T05:05:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 10 librados: watch_flush exit 2024-04-02T05:05:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 20 client.43955.objecter shutdown clearing up homeless session... 2024-04-02T05:05:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 10 client.43955.objecter successfully canceled tick 2024-04-02T05:05:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f90a806f5e0 msgr2=0x7f90a8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f90a806f5e0 0x7f90a8071aa0 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b2e0 tx=0x7f90b4005e80 comp rx=0 tx=0).stop 2024-04-02T05:05:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 msgr2=0x7f90c019a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.515+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 0x7f90c019a280 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7f90b000e9a0 tx=0x7f90b000ecb0 comp rx=0 tx=0).stop 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 shutdown_connections 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f90a806f5e0 0x7f90a8071aa0 secure :-1 s=CLOSED pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b2e0 tx=0x7f90b4005e80 comp rx=0 tx=0).stop 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90c010ead0 0x7f90c019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90c006ae40 0x7f90c019a280 unknown :-1 s=CLOSED pgs=3320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 --2- 172.21.15.69:0/3256864938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90c0103160 0x7f90c019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 >> 172.21.15.69:0/3256864938 conn(0x7f90c0077ea0 msgr2=0x7f90c0110650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 shutdown_connections 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 -- 172.21.15.69:0/3256864938 wait complete. 2024-04-02T05:05:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:15.519+0000 7f90c7c84700 1 librados: shutdown 2024-04-02T05:05:15.531 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.2 to 1 (10000) 2024-04-02T05:05:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[28904]: audit 2024-04-02T05:05:13.863435+0000 mgr.y (mgr.14538) 826 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[28904]: cluster 2024-04-02T05:05:13.888763+0000 osd.3 (osd.3) 505 : cluster 0 5.0 scrub starts 2024-04-02T05:05:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[28904]: cluster 2024-04-02T05:05:13.891390+0000 osd.3 (osd.3) 506 : cluster 0 5.0 scrub ok 2024-04-02T05:05:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[28904]: cluster 2024-04-02T05:05:14.420050+0000 mgr.y (mgr.14538) 827 : cluster 0 pgmap v1435: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:15.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[28904]: audit 2024-04-02T05:05:15.209844+0000 mon.c (mon.2) 1171 : audit 1 from='client.? 172.21.15.69:0/3256864938' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[28904]: audit 2024-04-02T05:05:15.210366+0000 mon.a (mon.0) 4114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[21528]: audit 2024-04-02T05:05:13.863435+0000 mgr.y (mgr.14538) 826 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[21528]: cluster 2024-04-02T05:05:13.888763+0000 osd.3 (osd.3) 505 : cluster 0 5.0 scrub starts 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[21528]: cluster 2024-04-02T05:05:13.891390+0000 osd.3 (osd.3) 506 : cluster 0 5.0 scrub ok 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[21528]: cluster 2024-04-02T05:05:14.420050+0000 mgr.y (mgr.14538) 827 : cluster 0 pgmap v1435: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[21528]: audit 2024-04-02T05:05:15.209844+0000 mon.c (mon.2) 1171 : audit 1 from='client.? 172.21.15.69:0/3256864938' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:15 smithi069 bash[21528]: audit 2024-04-02T05:05:15.210366+0000 mon.a (mon.0) 4114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:15.817 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:15 smithi098 bash[23790]: audit 2024-04-02T05:05:13.863435+0000 mgr.y (mgr.14538) 826 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:15.817 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:15 smithi098 bash[23790]: cluster 2024-04-02T05:05:13.888763+0000 osd.3 (osd.3) 505 : cluster 0 5.0 scrub starts 2024-04-02T05:05:15.817 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:15 smithi098 bash[23790]: cluster 2024-04-02T05:05:13.891390+0000 osd.3 (osd.3) 506 : cluster 0 5.0 scrub ok 2024-04-02T05:05:15.817 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:15 smithi098 bash[23790]: cluster 2024-04-02T05:05:14.420050+0000 mgr.y (mgr.14538) 827 : cluster 0 pgmap v1435: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:15.817 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:15 smithi098 bash[23790]: audit 2024-04-02T05:05:15.209844+0000 mon.c (mon.2) 1171 : audit 1 from='client.? 172.21.15.69:0/3256864938' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:15.817 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:15 smithi098 bash[23790]: audit 2024-04-02T05:05:15.210366+0000 mon.a (mon.0) 4114 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:16.349 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 2 1 2024-04-02T05:05:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:16 smithi069 bash[28904]: audit 2024-04-02T05:05:15.496992+0000 mon.a (mon.0) 4115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-02T05:05:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:16 smithi069 bash[28904]: cluster 2024-04-02T05:05:15.505353+0000 mon.a (mon.0) 4116 : cluster 0 osdmap e922: 8 total, 8 up, 8 in 2024-04-02T05:05:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:16 smithi069 bash[28904]: cluster 2024-04-02T05:05:16.508525+0000 mon.a (mon.0) 4117 : cluster 0 osdmap e923: 8 total, 8 up, 8 in 2024-04-02T05:05:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:16 smithi069 bash[21528]: audit 2024-04-02T05:05:15.496992+0000 mon.a (mon.0) 4115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-02T05:05:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:16 smithi069 bash[21528]: cluster 2024-04-02T05:05:15.505353+0000 mon.a (mon.0) 4116 : cluster 0 osdmap e922: 8 total, 8 up, 8 in 2024-04-02T05:05:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:16 smithi069 bash[21528]: cluster 2024-04-02T05:05:16.508525+0000 mon.a (mon.0) 4117 : cluster 0 osdmap e923: 8 total, 8 up, 8 in 2024-04-02T05:05:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:16 smithi098 bash[23790]: audit 2024-04-02T05:05:15.496992+0000 mon.a (mon.0) 4115 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-02T05:05:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:16 smithi098 bash[23790]: cluster 2024-04-02T05:05:15.505353+0000 mon.a (mon.0) 4116 : cluster 0 osdmap e922: 8 total, 8 up, 8 in 2024-04-02T05:05:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:16 smithi098 bash[23790]: cluster 2024-04-02T05:05:16.508525+0000 mon.a (mon.0) 4117 : cluster 0 osdmap e923: 8 total, 8 up, 8 in 2024-04-02T05:05:17.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:05:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T05:05:17.773 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:17.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:17 smithi098 bash[23790]: cluster 2024-04-02T05:05:15.900644+0000 osd.3 (osd.3) 507 : cluster 0 4.0 deep-scrub starts 2024-04-02T05:05:17.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:17 smithi098 bash[23790]: cluster 2024-04-02T05:05:15.903528+0000 osd.3 (osd.3) 508 : cluster 0 4.0 deep-scrub ok 2024-04-02T05:05:17.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:17 smithi098 bash[23790]: cluster 2024-04-02T05:05:16.420765+0000 mgr.y (mgr.14538) 828 : cluster 0 pgmap v1437: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:05:17.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:17 smithi098 bash[23790]: cluster 2024-04-02T05:05:16.887524+0000 osd.5 (osd.5) 311 : cluster 0 2.3 scrub starts 2024-04-02T05:05:17.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:17 smithi098 bash[23790]: cluster 2024-04-02T05:05:16.889881+0000 osd.5 (osd.5) 312 : cluster 0 2.3 scrub ok 2024-04-02T05:05:17.793 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:17 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.509745+0000 mon.a (mon.0) 4118 : cluster 0 osdmap e924: 8 total, 8 up, 8 in 2024-04-02T05:05:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[21528]: cluster 2024-04-02T05:05:15.900644+0000 osd.3 (osd.3) 507 : cluster 0 4.0 deep-scrub starts 2024-04-02T05:05:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[21528]: cluster 2024-04-02T05:05:15.903528+0000 osd.3 (osd.3) 508 : cluster 0 4.0 deep-scrub ok 2024-04-02T05:05:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[21528]: cluster 2024-04-02T05:05:16.420765+0000 mgr.y (mgr.14538) 828 : cluster 0 pgmap v1437: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:05:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[21528]: cluster 2024-04-02T05:05:16.887524+0000 osd.5 (osd.5) 311 : cluster 0 2.3 scrub starts 2024-04-02T05:05:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[21528]: cluster 2024-04-02T05:05:16.889881+0000 osd.5 (osd.5) 312 : cluster 0 2.3 scrub ok 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.509745+0000 mon.a (mon.0) 4118 : cluster 0 osdmap e924: 8 total, 8 up, 8 in 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[28904]: cluster 2024-04-02T05:05:15.900644+0000 osd.3 (osd.3) 507 : cluster 0 4.0 deep-scrub starts 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[28904]: cluster 2024-04-02T05:05:15.903528+0000 osd.3 (osd.3) 508 : cluster 0 4.0 deep-scrub ok 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[28904]: cluster 2024-04-02T05:05:16.420765+0000 mgr.y (mgr.14538) 828 : cluster 0 pgmap v1437: 184 pgs: 184 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[28904]: cluster 2024-04-02T05:05:16.887524+0000 osd.5 (osd.5) 311 : cluster 0 2.3 scrub starts 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[28904]: cluster 2024-04-02T05:05:16.889881+0000 osd.5 (osd.5) 312 : cluster 0 2.3 scrub ok 2024-04-02T05:05:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:17 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.509745+0000 mon.a (mon.0) 4118 : cluster 0 osdmap e924: 8 total, 8 up, 8 in 2024-04-02T05:05:18.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-02T05:05:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:18 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.389743+0000 osd.0 (osd.0) 477 : cluster 0 3.6 scrub starts 2024-04-02T05:05:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:18 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.394329+0000 osd.0 (osd.0) 478 : cluster 0 3.6 scrub ok 2024-04-02T05:05:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:18 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.940083+0000 osd.5 (osd.5) 313 : cluster 0 6.7 scrub starts 2024-04-02T05:05:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:18 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.942248+0000 osd.5 (osd.5) 314 : cluster 0 6.7 scrub ok 2024-04-02T05:05:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:18 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.992009+0000 osd.2 (osd.2) 390 : cluster 0 10.3 deep-scrub starts 2024-04-02T05:05:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:18 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.996908+0000 osd.2 (osd.2) 391 : cluster 0 10.3 deep-scrub ok 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:05:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.389743+0000 osd.0 (osd.0) 477 : cluster 0 3.6 scrub starts 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.394329+0000 osd.0 (osd.0) 478 : cluster 0 3.6 scrub ok 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.940083+0000 osd.5 (osd.5) 313 : cluster 0 6.7 scrub starts 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.942248+0000 osd.5 (osd.5) 314 : cluster 0 6.7 scrub ok 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.992009+0000 osd.2 (osd.2) 390 : cluster 0 10.3 deep-scrub starts 2024-04-02T05:05:19.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.996908+0000 osd.2 (osd.2) 391 : cluster 0 10.3 deep-scrub ok 2024-04-02T05:05:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.389743+0000 osd.0 (osd.0) 477 : cluster 0 3.6 scrub starts 2024-04-02T05:05:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.394329+0000 osd.0 (osd.0) 478 : cluster 0 3.6 scrub ok 2024-04-02T05:05:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.940083+0000 osd.5 (osd.5) 313 : cluster 0 6.7 scrub starts 2024-04-02T05:05:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.942248+0000 osd.5 (osd.5) 314 : cluster 0 6.7 scrub ok 2024-04-02T05:05:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.992009+0000 osd.2 (osd.2) 390 : cluster 0 10.3 deep-scrub starts 2024-04-02T05:05:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:18 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.996908+0000 osd.2 (osd.2) 391 : cluster 0 10.3 deep-scrub ok 2024-04-02T05:05:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.843713+0000 osd.3 (osd.3) 509 : cluster 0 4.1a scrub starts 2024-04-02T05:05:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: cluster 2024-04-02T05:05:17.848861+0000 osd.3 (osd.3) 510 : cluster 0 4.1a scrub ok 2024-04-02T05:05:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: cluster 2024-04-02T05:05:18.422682+0000 mgr.y (mgr.14538) 829 : cluster 0 pgmap v1440: 184 pgs: 1 active+remapped, 183 active+clean; 609 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 18 KiB/s wr, 4 op/s; 13 KiB/s, 6 objects/s recovering 2024-04-02T05:05:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: cluster 2024-04-02T05:05:18.559637+0000 mon.a (mon.0) 4119 : cluster 0 osdmap e925: 8 total, 8 up, 8 in 2024-04-02T05:05:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.010701+0000 osd.2 (osd.2) 392 : cluster 0 6.f scrub starts 2024-04-02T05:05:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.013082+0000 osd.2 (osd.2) 393 : cluster 0 6.f scrub ok 2024-04-02T05:05:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: audit 2024-04-02T05:05:19.475709+0000 mon.a (mon.0) 4120 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:19 smithi098 bash[23790]: audit 2024-04-02T05:05:19.479078+0000 mon.b (mon.1) 341 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:20.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.843713+0000 osd.3 (osd.3) 509 : cluster 0 4.1a scrub starts 2024-04-02T05:05:20.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: cluster 2024-04-02T05:05:17.848861+0000 osd.3 (osd.3) 510 : cluster 0 4.1a scrub ok 2024-04-02T05:05:20.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: cluster 2024-04-02T05:05:18.422682+0000 mgr.y (mgr.14538) 829 : cluster 0 pgmap v1440: 184 pgs: 1 active+remapped, 183 active+clean; 609 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 18 KiB/s wr, 4 op/s; 13 KiB/s, 6 objects/s recovering 2024-04-02T05:05:20.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: cluster 2024-04-02T05:05:18.559637+0000 mon.a (mon.0) 4119 : cluster 0 osdmap e925: 8 total, 8 up, 8 in 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.010701+0000 osd.2 (osd.2) 392 : cluster 0 6.f scrub starts 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.013082+0000 osd.2 (osd.2) 393 : cluster 0 6.f scrub ok 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: audit 2024-04-02T05:05:19.475709+0000 mon.a (mon.0) 4120 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[21528]: audit 2024-04-02T05:05:19.479078+0000 mon.b (mon.1) 341 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.843713+0000 osd.3 (osd.3) 509 : cluster 0 4.1a scrub starts 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: cluster 2024-04-02T05:05:17.848861+0000 osd.3 (osd.3) 510 : cluster 0 4.1a scrub ok 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: cluster 2024-04-02T05:05:18.422682+0000 mgr.y (mgr.14538) 829 : cluster 0 pgmap v1440: 184 pgs: 1 active+remapped, 183 active+clean; 609 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 18 KiB/s wr, 4 op/s; 13 KiB/s, 6 objects/s recovering 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: cluster 2024-04-02T05:05:18.559637+0000 mon.a (mon.0) 4119 : cluster 0 osdmap e925: 8 total, 8 up, 8 in 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.010701+0000 osd.2 (osd.2) 392 : cluster 0 6.f scrub starts 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.013082+0000 osd.2 (osd.2) 393 : cluster 0 6.f scrub ok 2024-04-02T05:05:20.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: audit 2024-04-02T05:05:19.475709+0000 mon.a (mon.0) 4120 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:20.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:19 smithi069 bash[28904]: audit 2024-04-02T05:05:19.479078+0000 mon.b (mon.1) 341 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:18.825588+0000 osd.3 (osd.3) 511 : cluster 0 3.2 scrub starts 2024-04-02T05:05:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:18.830506+0000 osd.3 (osd.3) 512 : cluster 0 3.2 scrub ok 2024-04-02T05:05:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.431089+0000 osd.0 (osd.0) 479 : cluster 0 3.17 scrub starts 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.435185+0000 osd.0 (osd.0) 480 : cluster 0 3.17 scrub ok 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.560762+0000 mon.a (mon.0) 4121 : cluster 0 osdmap e926: 8 total, 8 up, 8 in 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.979008+0000 osd.2 (osd.2) 394 : cluster 0 1.5 scrub starts 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.981865+0000 osd.2 (osd.2) 395 : cluster 0 1.5 scrub ok 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:20.190328+0000 mon.a (mon.0) 4122 : cluster 0 osdmap e927: 8 total, 8 up, 8 in 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:20.451005+0000 osd.6 (osd.6) 437 : cluster 0 10.12 deep-scrub starts 2024-04-02T05:05:20.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:20 smithi098 bash[23790]: cluster 2024-04-02T05:05:20.453969+0000 osd.6 (osd.6) 438 : cluster 0 10.12 deep-scrub ok 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:18.825588+0000 osd.3 (osd.3) 511 : cluster 0 3.2 scrub starts 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:18.830506+0000 osd.3 (osd.3) 512 : cluster 0 3.2 scrub ok 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.431089+0000 osd.0 (osd.0) 479 : cluster 0 3.17 scrub starts 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.435185+0000 osd.0 (osd.0) 480 : cluster 0 3.17 scrub ok 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.560762+0000 mon.a (mon.0) 4121 : cluster 0 osdmap e926: 8 total, 8 up, 8 in 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.979008+0000 osd.2 (osd.2) 394 : cluster 0 1.5 scrub starts 2024-04-02T05:05:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.981865+0000 osd.2 (osd.2) 395 : cluster 0 1.5 scrub ok 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:20.190328+0000 mon.a (mon.0) 4122 : cluster 0 osdmap e927: 8 total, 8 up, 8 in 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:20.451005+0000 osd.6 (osd.6) 437 : cluster 0 10.12 deep-scrub starts 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[21528]: cluster 2024-04-02T05:05:20.453969+0000 osd.6 (osd.6) 438 : cluster 0 10.12 deep-scrub ok 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:18.825588+0000 osd.3 (osd.3) 511 : cluster 0 3.2 scrub starts 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:18.830506+0000 osd.3 (osd.3) 512 : cluster 0 3.2 scrub ok 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.431089+0000 osd.0 (osd.0) 479 : cluster 0 3.17 scrub starts 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.435185+0000 osd.0 (osd.0) 480 : cluster 0 3.17 scrub ok 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.560762+0000 mon.a (mon.0) 4121 : cluster 0 osdmap e926: 8 total, 8 up, 8 in 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.979008+0000 osd.2 (osd.2) 394 : cluster 0 1.5 scrub starts 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.981865+0000 osd.2 (osd.2) 395 : cluster 0 1.5 scrub ok 2024-04-02T05:05:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:20.190328+0000 mon.a (mon.0) 4122 : cluster 0 osdmap e927: 8 total, 8 up, 8 in 2024-04-02T05:05:21.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:20.451005+0000 osd.6 (osd.6) 437 : cluster 0 10.12 deep-scrub starts 2024-04-02T05:05:21.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:20 smithi069 bash[28904]: cluster 2024-04-02T05:05:20.453969+0000 osd.6 (osd.6) 438 : cluster 0 10.12 deep-scrub ok 2024-04-02T05:05:21.153 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:21 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.867874+0000 osd.1 (osd.1) 509 : cluster 0 2.a scrub starts 2024-04-02T05:05:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:21 smithi098 bash[23790]: cluster 2024-04-02T05:05:19.870271+0000 osd.1 (osd.1) 510 : cluster 0 2.a scrub ok 2024-04-02T05:05:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:21 smithi098 bash[23790]: cluster 2024-04-02T05:05:20.423539+0000 mgr.y (mgr.14538) 830 : cluster 0 pgmap v1444: 184 pgs: 3 peering, 1 active+remapped, 180 active+clean; 609 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 28 KiB/s wr, 5 op/s; 20 KiB/s, 9 objects/s recovering 2024-04-02T05:05:22.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:21 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.867874+0000 osd.1 (osd.1) 509 : cluster 0 2.a scrub starts 2024-04-02T05:05:22.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:21 smithi069 bash[21528]: cluster 2024-04-02T05:05:19.870271+0000 osd.1 (osd.1) 510 : cluster 0 2.a scrub ok 2024-04-02T05:05:22.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:21 smithi069 bash[21528]: cluster 2024-04-02T05:05:20.423539+0000 mgr.y (mgr.14538) 830 : cluster 0 pgmap v1444: 184 pgs: 3 peering, 1 active+remapped, 180 active+clean; 609 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 28 KiB/s wr, 5 op/s; 20 KiB/s, 9 objects/s recovering 2024-04-02T05:05:22.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:21 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.867874+0000 osd.1 (osd.1) 509 : cluster 0 2.a scrub starts 2024-04-02T05:05:22.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:21 smithi069 bash[28904]: cluster 2024-04-02T05:05:19.870271+0000 osd.1 (osd.1) 510 : cluster 0 2.a scrub ok 2024-04-02T05:05:22.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:21 smithi069 bash[28904]: cluster 2024-04-02T05:05:20.423539+0000 mgr.y (mgr.14538) 830 : cluster 0 pgmap v1444: 184 pgs: 3 peering, 1 active+remapped, 180 active+clean; 609 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 28 KiB/s wr, 5 op/s; 20 KiB/s, 9 objects/s recovering 2024-04-02T05:05:22.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[21528]: cluster 2024-04-02T05:05:20.882477+0000 osd.3 (osd.3) 513 : cluster 0 6.2 scrub starts 2024-04-02T05:05:22.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[21528]: cluster 2024-04-02T05:05:20.884873+0000 osd.3 (osd.3) 514 : cluster 0 6.2 scrub ok 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[21528]: cluster 2024-04-02T05:05:21.931151+0000 osd.5 (osd.5) 315 : cluster 0 2.2 scrub starts 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[21528]: cluster 2024-04-02T05:05:21.934127+0000 osd.5 (osd.5) 316 : cluster 0 2.2 scrub ok 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[21528]: cluster 2024-04-02T05:05:22.400559+0000 osd.6 (osd.6) 439 : cluster 0 10.b scrub starts 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[21528]: cluster 2024-04-02T05:05:22.406454+0000 osd.6 (osd.6) 440 : cluster 0 10.b scrub ok 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[28904]: cluster 2024-04-02T05:05:20.882477+0000 osd.3 (osd.3) 513 : cluster 0 6.2 scrub starts 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[28904]: cluster 2024-04-02T05:05:20.884873+0000 osd.3 (osd.3) 514 : cluster 0 6.2 scrub ok 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[28904]: cluster 2024-04-02T05:05:21.931151+0000 osd.5 (osd.5) 315 : cluster 0 2.2 scrub starts 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[28904]: cluster 2024-04-02T05:05:21.934127+0000 osd.5 (osd.5) 316 : cluster 0 2.2 scrub ok 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[28904]: cluster 2024-04-02T05:05:22.400559+0000 osd.6 (osd.6) 439 : cluster 0 10.b scrub starts 2024-04-02T05:05:22.729 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:22 smithi069 bash[28904]: cluster 2024-04-02T05:05:22.406454+0000 osd.6 (osd.6) 440 : cluster 0 10.b scrub ok 2024-04-02T05:05:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:22 smithi098 bash[23790]: cluster 2024-04-02T05:05:20.882477+0000 osd.3 (osd.3) 513 : cluster 0 6.2 scrub starts 2024-04-02T05:05:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:22 smithi098 bash[23790]: cluster 2024-04-02T05:05:20.884873+0000 osd.3 (osd.3) 514 : cluster 0 6.2 scrub ok 2024-04-02T05:05:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:22 smithi098 bash[23790]: cluster 2024-04-02T05:05:21.931151+0000 osd.5 (osd.5) 315 : cluster 0 2.2 scrub starts 2024-04-02T05:05:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:22 smithi098 bash[23790]: cluster 2024-04-02T05:05:21.934127+0000 osd.5 (osd.5) 316 : cluster 0 2.2 scrub ok 2024-04-02T05:05:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:22 smithi098 bash[23790]: cluster 2024-04-02T05:05:22.400559+0000 osd.6 (osd.6) 439 : cluster 0 10.b scrub starts 2024-04-02T05:05:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:22 smithi098 bash[23790]: cluster 2024-04-02T05:05:22.406454+0000 osd.6 (osd.6) 440 : cluster 0 10.b scrub ok 2024-04-02T05:05:23.321 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:23 smithi098 bash[23790]: cluster 2024-04-02T05:05:22.350283+0000 osd.0 (osd.0) 481 : cluster 0 3.1b deep-scrub starts 2024-04-02T05:05:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:23 smithi098 bash[23790]: cluster 2024-04-02T05:05:22.354650+0000 osd.0 (osd.0) 482 : cluster 0 3.1b deep-scrub ok 2024-04-02T05:05:23.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:23 smithi098 bash[23790]: cluster 2024-04-02T05:05:22.424077+0000 mgr.y (mgr.14538) 831 : cluster 0 pgmap v1445: 184 pgs: 3 peering, 1 active+remapped, 180 active+clean; 609 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 22 KiB/s wr, 4 op/s; 16 KiB/s, 7 objects/s recovering 2024-04-02T05:05:24.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:23 smithi069 bash[28904]: cluster 2024-04-02T05:05:22.350283+0000 osd.0 (osd.0) 481 : cluster 0 3.1b deep-scrub starts 2024-04-02T05:05:24.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:23 smithi069 bash[28904]: cluster 2024-04-02T05:05:22.354650+0000 osd.0 (osd.0) 482 : cluster 0 3.1b deep-scrub ok 2024-04-02T05:05:24.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:23 smithi069 bash[28904]: cluster 2024-04-02T05:05:22.424077+0000 mgr.y (mgr.14538) 831 : cluster 0 pgmap v1445: 184 pgs: 3 peering, 1 active+remapped, 180 active+clean; 609 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 22 KiB/s wr, 4 op/s; 16 KiB/s, 7 objects/s recovering 2024-04-02T05:05:24.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:23 smithi069 bash[21528]: cluster 2024-04-02T05:05:22.350283+0000 osd.0 (osd.0) 481 : cluster 0 3.1b deep-scrub starts 2024-04-02T05:05:24.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:23 smithi069 bash[21528]: cluster 2024-04-02T05:05:22.354650+0000 osd.0 (osd.0) 482 : cluster 0 3.1b deep-scrub ok 2024-04-02T05:05:24.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:23 smithi069 bash[21528]: cluster 2024-04-02T05:05:22.424077+0000 mgr.y (mgr.14538) 831 : cluster 0 pgmap v1445: 184 pgs: 3 peering, 1 active+remapped, 180 active+clean; 609 KiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 22 KiB/s wr, 4 op/s; 16 KiB/s, 7 objects/s recovering 2024-04-02T05:05:24.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:05:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:05:24.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/1217220420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 msgr2=0x7f3040107610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:24.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- 172.21.15.69:0/1217220420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f3040107610 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f30340098f0 tx=0x7f3034009c00 comp rx=0 tx=0).stop 2024-04-02T05:05:24.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/1217220420 shutdown_connections 2024-04-02T05:05:24.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- 172.21.15.69:0/1217220420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040101b40 0x7f304010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- 172.21.15.69:0/1217220420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f3040107610 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- 172.21.15.69:0/1217220420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/1217220420 >> 172.21.15.69:0/1217220420 conn(0x7f30400780e0 msgr2=0x7f30400784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/1217220420 shutdown_connections 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/1217220420 wait complete. 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 librados: starting msgr at 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 librados: starting objecter 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 Processor -- start 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- start start 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 librados: setting wanted keys 2024-04-02T05:05:24.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 librados: calling monclient init 2024-04-02T05:05:24.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:24.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040101b40 0x7f3040103370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:24.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f30401038b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:24.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3040110910 con 0x7f3040107230 2024-04-02T05:05:24.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3040110790 con 0x7f3040101b40 2024-04-02T05:05:24.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3040110a90 con 0x7f3040101180 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30450e4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f30401038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040102e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40182/0 (socket says 172.21.15.69:40182) 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 -- 172.21.15.69:0/172701592 learned_addr learned my addr 172.21.15.69:0/172701592 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f303ffff700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040101b40 0x7f3040103370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 -- 172.21.15.69:0/172701592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040101b40 msgr2=0x7f3040103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040101b40 0x7f3040103370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 -- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 msgr2=0x7f30401038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f30401038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 -- 172.21.15.69:0/172701592 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30401a6550 con 0x7f3040101180 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30450e4700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f30401038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f30448e3700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040102e30 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f30340059d0 tx=0x7f303400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:24.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f303dffb700 10 client.?.objecter ms_handle_connect 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f303dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034027070 con 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f303400bb60 con 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30401a67e0 con 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034004930 con 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 10 client.43961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f30401a6d10 con 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.283+0000 7f3046b47700 1 librados: init done 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f304004ff00 con 0x7f3040101180 2024-04-02T05:05:24.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f303402a070 con 0x7f3040101180 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f303006f5e0 0x7f3030071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303ffff700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f303006f5e0 0x7f3030071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(927..927 src has 795..927) ==== 9044+0+0 (secure 0 0 0) 0x7f303408f190 con 0x7f3040101180 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 10 client.43961.objecter ms_dispatch 0x7f3040001550 osd_map(927..927 src has 795..927) 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 3 client.43961.objecter handle_osd_map got epochs [927,927] > 0 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 3 client.43961.objecter handle_osd_map decoding full epoch 927 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 20 client.43961.objecter dump_active .. 0 homeless 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303ffff700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f303006f5e0 0x7f3030071aa0 secure :-1 s=READY pgs=4039 cs=0 l=1 rev1=1 crypto rx=0x7f302c009750 tx=0x7f302c009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:24.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.287+0000 7f303dffb700 10 client.43961.objecter ms_handle_connect 0x7f303006f5e0 2024-04-02T05:05:24.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.291+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3034013030 con 0x7f3040101180 2024-04-02T05:05:24.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.519+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7f30401a6fd0 con 0x7f3040101180 2024-04-02T05:05:24.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f303dffb700 1 -- 172.21.15.69:0/172701592 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v928) ==== 133+0+0 (secure 0 0 0) 0x7f303405c1c0 con 0x7f3040101180 2024-04-02T05:05:24.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 10 librados: watch_flush enter 2024-04-02T05:05:24.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 10 librados: watch_flush exit 2024-04-02T05:05:24.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 20 client.43961.objecter shutdown clearing up homeless session... 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 10 client.43961.objecter successfully canceled tick 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f303006f5e0 msgr2=0x7f3030071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f303006f5e0 0x7f3030071aa0 secure :-1 s=READY pgs=4039 cs=0 l=1 rev1=1 crypto rx=0x7f302c009750 tx=0x7f302c009310 comp rx=0 tx=0).stop 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 msgr2=0x7f3040102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040102e30 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f30340059d0 tx=0x7f303400fa70 comp rx=0 tx=0).stop 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 shutdown_connections 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f303006f5e0 0x7f3030071aa0 unknown :-1 s=CLOSED pgs=4039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040107230 0x7f30401038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3040101180 0x7f3040102e30 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 --2- 172.21.15.69:0/172701592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040101b40 0x7f3040103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:24.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 >> 172.21.15.69:0/172701592 conn(0x7f30400780e0 msgr2=0x7f30400ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:24.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 shutdown_connections 2024-04-02T05:05:24.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 -- 172.21.15.69:0/172701592 wait complete. 2024-04-02T05:05:24.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:24.599+0000 7f3046b47700 1 librados: shutdown 2024-04-02T05:05:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-02T05:05:24.807 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:24 smithi098 bash[23790]: cluster 2024-04-02T05:05:23.754506+0000 osd.5 (osd.5) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:24.807 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:24 smithi098 bash[23790]: cluster 2024-04-02T05:05:23.754878+0000 osd.5 (osd.5) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:24.807 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:24 smithi098 bash[23790]: audit 2024-04-02T05:05:24.522605+0000 mon.c (mon.2) 1172 : audit 1 from='client.? 172.21.15.69:0/172701592' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:24.808 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:24 smithi098 bash[23790]: audit 2024-04-02T05:05:24.523287+0000 mon.a (mon.0) 4123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[21528]: cluster 2024-04-02T05:05:23.754506+0000 osd.5 (osd.5) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[21528]: cluster 2024-04-02T05:05:23.754878+0000 osd.5 (osd.5) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[21528]: audit 2024-04-02T05:05:24.522605+0000 mon.c (mon.2) 1172 : audit 1 from='client.? 172.21.15.69:0/172701592' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:25.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[21528]: audit 2024-04-02T05:05:24.523287+0000 mon.a (mon.0) 4123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[28904]: cluster 2024-04-02T05:05:23.754506+0000 osd.5 (osd.5) 317 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:25.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[28904]: cluster 2024-04-02T05:05:23.754878+0000 osd.5 (osd.5) 318 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:25.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[28904]: audit 2024-04-02T05:05:24.522605+0000 mon.c (mon.2) 1172 : audit 1 from='client.? 172.21.15.69:0/172701592' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:25.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:24 smithi069 bash[28904]: audit 2024-04-02T05:05:24.523287+0000 mon.a (mon.0) 4123 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-02T05:05:25.389 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 3 1 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: cluster 2024-04-02T05:05:23.853187+0000 osd.1 (osd.1) 511 : cluster 0 10.13 scrub starts 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: cluster 2024-04-02T05:05:23.858536+0000 osd.1 (osd.1) 512 : cluster 0 10.13 scrub ok 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: audit 2024-04-02T05:05:23.874796+0000 mgr.y (mgr.14538) 832 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: cluster 2024-04-02T05:05:24.425973+0000 mgr.y (mgr.14538) 833 : cluster 0 pgmap v1446: 184 pgs: 184 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: audit 2024-04-02T05:05:24.591475+0000 mon.a (mon.0) 4124 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: cluster 2024-04-02T05:05:24.594617+0000 mon.a (mon.0) 4125 : cluster 0 osdmap e928: 8 total, 8 up, 8 in 2024-04-02T05:05:25.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:25 smithi098 bash[23790]: cluster 2024-04-02T05:05:25.193363+0000 mon.a (mon.0) 4126 : cluster 0 osdmap e929: 8 total, 8 up, 8 in 2024-04-02T05:05:25.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: cluster 2024-04-02T05:05:23.853187+0000 osd.1 (osd.1) 511 : cluster 0 10.13 scrub starts 2024-04-02T05:05:25.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: cluster 2024-04-02T05:05:23.858536+0000 osd.1 (osd.1) 512 : cluster 0 10.13 scrub ok 2024-04-02T05:05:25.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: audit 2024-04-02T05:05:23.874796+0000 mgr.y (mgr.14538) 832 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:25.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: cluster 2024-04-02T05:05:24.425973+0000 mgr.y (mgr.14538) 833 : cluster 0 pgmap v1446: 184 pgs: 184 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: audit 2024-04-02T05:05:24.591475+0000 mon.a (mon.0) 4124 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: cluster 2024-04-02T05:05:24.594617+0000 mon.a (mon.0) 4125 : cluster 0 osdmap e928: 8 total, 8 up, 8 in 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[21528]: cluster 2024-04-02T05:05:25.193363+0000 mon.a (mon.0) 4126 : cluster 0 osdmap e929: 8 total, 8 up, 8 in 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: cluster 2024-04-02T05:05:23.853187+0000 osd.1 (osd.1) 511 : cluster 0 10.13 scrub starts 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: cluster 2024-04-02T05:05:23.858536+0000 osd.1 (osd.1) 512 : cluster 0 10.13 scrub ok 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: audit 2024-04-02T05:05:23.874796+0000 mgr.y (mgr.14538) 832 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: cluster 2024-04-02T05:05:24.425973+0000 mgr.y (mgr.14538) 833 : cluster 0 pgmap v1446: 184 pgs: 184 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: audit 2024-04-02T05:05:24.591475+0000 mon.a (mon.0) 4124 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: cluster 2024-04-02T05:05:24.594617+0000 mon.a (mon.0) 4125 : cluster 0 osdmap e928: 8 total, 8 up, 8 in 2024-04-02T05:05:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:25 smithi069 bash[28904]: cluster 2024-04-02T05:05:25.193363+0000 mon.a (mon.0) 4126 : cluster 0 osdmap e929: 8 total, 8 up, 8 in 2024-04-02T05:05:26.313 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:26.313 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:05:26.313 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T05:05:26.313 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T05:05:26.314 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:26 smithi098 bash[23790]: cluster 2024-04-02T05:05:24.860398+0000 osd.4 (osd.4) 476 : cluster 0 4.18 scrub starts 2024-04-02T05:05:27.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:26 smithi098 bash[23790]: cluster 2024-04-02T05:05:24.862957+0000 osd.4 (osd.4) 477 : cluster 0 4.18 scrub ok 2024-04-02T05:05:27.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:26 smithi098 bash[23790]: cluster 2024-04-02T05:05:24.923359+0000 osd.3 (osd.3) 515 : cluster 0 4.10 scrub starts 2024-04-02T05:05:27.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:26 smithi098 bash[23790]: cluster 2024-04-02T05:05:24.925621+0000 osd.3 (osd.3) 516 : cluster 0 4.10 scrub ok 2024-04-02T05:05:27.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:26 smithi098 bash[23790]: cluster 2024-04-02T05:05:25.373154+0000 osd.7 (osd.7) 299 : cluster 0 1.9 scrub starts 2024-04-02T05:05:27.220 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:26 smithi098 bash[23790]: cluster 2024-04-02T05:05:25.374966+0000 osd.7 (osd.7) 300 : cluster 0 1.9 scrub ok 2024-04-02T05:05:27.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[21528]: cluster 2024-04-02T05:05:24.860398+0000 osd.4 (osd.4) 476 : cluster 0 4.18 scrub starts 2024-04-02T05:05:27.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[21528]: cluster 2024-04-02T05:05:24.862957+0000 osd.4 (osd.4) 477 : cluster 0 4.18 scrub ok 2024-04-02T05:05:27.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[21528]: cluster 2024-04-02T05:05:24.923359+0000 osd.3 (osd.3) 515 : cluster 0 4.10 scrub starts 2024-04-02T05:05:27.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[21528]: cluster 2024-04-02T05:05:24.925621+0000 osd.3 (osd.3) 516 : cluster 0 4.10 scrub ok 2024-04-02T05:05:27.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[21528]: cluster 2024-04-02T05:05:25.373154+0000 osd.7 (osd.7) 299 : cluster 0 1.9 scrub starts 2024-04-02T05:05:27.221 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[21528]: cluster 2024-04-02T05:05:25.374966+0000 osd.7 (osd.7) 300 : cluster 0 1.9 scrub ok 2024-04-02T05:05:27.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[28904]: cluster 2024-04-02T05:05:24.860398+0000 osd.4 (osd.4) 476 : cluster 0 4.18 scrub starts 2024-04-02T05:05:27.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[28904]: cluster 2024-04-02T05:05:24.862957+0000 osd.4 (osd.4) 477 : cluster 0 4.18 scrub ok 2024-04-02T05:05:27.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[28904]: cluster 2024-04-02T05:05:24.923359+0000 osd.3 (osd.3) 515 : cluster 0 4.10 scrub starts 2024-04-02T05:05:27.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[28904]: cluster 2024-04-02T05:05:24.925621+0000 osd.3 (osd.3) 516 : cluster 0 4.10 scrub ok 2024-04-02T05:05:27.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[28904]: cluster 2024-04-02T05:05:25.373154+0000 osd.7 (osd.7) 299 : cluster 0 1.9 scrub starts 2024-04-02T05:05:27.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:26 smithi069 bash[28904]: cluster 2024-04-02T05:05:25.374966+0000 osd.7 (osd.7) 300 : cluster 0 1.9 scrub ok 2024-04-02T05:05:27.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-02T05:05:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:27 smithi098 bash[23790]: cluster 2024-04-02T05:05:26.286953+0000 osd.0 (osd.0) 483 : cluster 0 4.b scrub starts 2024-04-02T05:05:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:27 smithi098 bash[23790]: cluster 2024-04-02T05:05:26.289274+0000 osd.0 (osd.0) 484 : cluster 0 4.b scrub ok 2024-04-02T05:05:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:27 smithi098 bash[23790]: cluster 2024-04-02T05:05:26.426741+0000 mgr.y (mgr.14538) 834 : cluster 0 pgmap v1449: 184 pgs: 184 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-02T05:05:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:27 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.326483+0000 osd.6 (osd.6) 441 : cluster 0 10.f scrub starts 2024-04-02T05:05:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:27 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.333394+0000 osd.6 (osd.6) 442 : cluster 0 10.f scrub ok 2024-04-02T05:05:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[28904]: cluster 2024-04-02T05:05:26.286953+0000 osd.0 (osd.0) 483 : cluster 0 4.b scrub starts 2024-04-02T05:05:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[28904]: cluster 2024-04-02T05:05:26.289274+0000 osd.0 (osd.0) 484 : cluster 0 4.b scrub ok 2024-04-02T05:05:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[28904]: cluster 2024-04-02T05:05:26.426741+0000 mgr.y (mgr.14538) 834 : cluster 0 pgmap v1449: 184 pgs: 184 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-02T05:05:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.326483+0000 osd.6 (osd.6) 441 : cluster 0 10.f scrub starts 2024-04-02T05:05:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.333394+0000 osd.6 (osd.6) 442 : cluster 0 10.f scrub ok 2024-04-02T05:05:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[21528]: cluster 2024-04-02T05:05:26.286953+0000 osd.0 (osd.0) 483 : cluster 0 4.b scrub starts 2024-04-02T05:05:28.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[21528]: cluster 2024-04-02T05:05:26.289274+0000 osd.0 (osd.0) 484 : cluster 0 4.b scrub ok 2024-04-02T05:05:28.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[21528]: cluster 2024-04-02T05:05:26.426741+0000 mgr.y (mgr.14538) 834 : cluster 0 pgmap v1449: 184 pgs: 184 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-02T05:05:28.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.326483+0000 osd.6 (osd.6) 441 : cluster 0 10.f scrub starts 2024-04-02T05:05:28.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:27 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.333394+0000 osd.6 (osd.6) 442 : cluster 0 10.f scrub ok 2024-04-02T05:05:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.018749+0000 osd.1 (osd.1) 513 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.019213+0000 osd.1 (osd.1) 514 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.627236+0000 mon.a (mon.0) 4127 : cluster 0 osdmap e930: 8 total, 8 up, 8 in 2024-04-02T05:05:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: audit 2024-04-02T05:05:27.855533+0000 mon.b (mon.1) 342 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:05:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.874098+0000 osd.5 (osd.5) 319 : cluster 0 10.8 scrub starts 2024-04-02T05:05:28.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.875879+0000 osd.5 (osd.5) 320 : cluster 0 10.8 scrub ok 2024-04-02T05:05:28.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: audit 2024-04-02T05:05:28.381338+0000 mon.a (mon.0) 4128 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:28.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:28 smithi098 bash[23790]: audit 2024-04-02T05:05:28.390907+0000 mon.a (mon.0) 4129 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:29.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:05:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:05:29.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.018749+0000 osd.1 (osd.1) 513 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:29.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.019213+0000 osd.1 (osd.1) 514 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:29.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.627236+0000 mon.a (mon.0) 4127 : cluster 0 osdmap e930: 8 total, 8 up, 8 in 2024-04-02T05:05:29.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: audit 2024-04-02T05:05:27.855533+0000 mon.b (mon.1) 342 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.874098+0000 osd.5 (osd.5) 319 : cluster 0 10.8 scrub starts 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.875879+0000 osd.5 (osd.5) 320 : cluster 0 10.8 scrub ok 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: audit 2024-04-02T05:05:28.381338+0000 mon.a (mon.0) 4128 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[28904]: audit 2024-04-02T05:05:28.390907+0000 mon.a (mon.0) 4129 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.018749+0000 osd.1 (osd.1) 513 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.019213+0000 osd.1 (osd.1) 514 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.627236+0000 mon.a (mon.0) 4127 : cluster 0 osdmap e930: 8 total, 8 up, 8 in 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: audit 2024-04-02T05:05:27.855533+0000 mon.b (mon.1) 342 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.874098+0000 osd.5 (osd.5) 319 : cluster 0 10.8 scrub starts 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.875879+0000 osd.5 (osd.5) 320 : cluster 0 10.8 scrub ok 2024-04-02T05:05:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: audit 2024-04-02T05:05:28.381338+0000 mon.a (mon.0) 4128 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:29.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:28 smithi069 bash[21528]: audit 2024-04-02T05:05:28.390907+0000 mon.a (mon.0) 4129 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.968536+0000 osd.1 (osd.1) 515 : cluster 0 4.7 scrub starts 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: cluster 2024-04-02T05:05:27.970556+0000 osd.1 (osd.1) 516 : cluster 0 4.7 scrub ok 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.262884+0000 osd.0 (osd.0) 485 : cluster 0 3.1f scrub starts 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.268958+0000 osd.0 (osd.0) 486 : cluster 0 3.1f scrub ok 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.428226+0000 mgr.y (mgr.14538) 835 : cluster 0 pgmap v1451: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: audit 2024-04-02T05:05:29.026342+0000 mon.b (mon.1) 343 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: audit 2024-04-02T05:05:29.028385+0000 mon.b (mon.1) 344 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:05:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:29 smithi098 bash[23790]: audit 2024-04-02T05:05:29.040016+0000 mon.a (mon.0) 4130 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:30.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.968536+0000 osd.1 (osd.1) 515 : cluster 0 4.7 scrub starts 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: cluster 2024-04-02T05:05:27.970556+0000 osd.1 (osd.1) 516 : cluster 0 4.7 scrub ok 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.262884+0000 osd.0 (osd.0) 485 : cluster 0 3.1f scrub starts 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.268958+0000 osd.0 (osd.0) 486 : cluster 0 3.1f scrub ok 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.428226+0000 mgr.y (mgr.14538) 835 : cluster 0 pgmap v1451: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: audit 2024-04-02T05:05:29.026342+0000 mon.b (mon.1) 343 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: audit 2024-04-02T05:05:29.028385+0000 mon.b (mon.1) 344 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[21528]: audit 2024-04-02T05:05:29.040016+0000 mon.a (mon.0) 4130 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.968536+0000 osd.1 (osd.1) 515 : cluster 0 4.7 scrub starts 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: cluster 2024-04-02T05:05:27.970556+0000 osd.1 (osd.1) 516 : cluster 0 4.7 scrub ok 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.262884+0000 osd.0 (osd.0) 485 : cluster 0 3.1f scrub starts 2024-04-02T05:05:30.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.268958+0000 osd.0 (osd.0) 486 : cluster 0 3.1f scrub ok 2024-04-02T05:05:30.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.428226+0000 mgr.y (mgr.14538) 835 : cluster 0 pgmap v1451: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-04-02T05:05:30.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: audit 2024-04-02T05:05:29.026342+0000 mon.b (mon.1) 343 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:05:30.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: audit 2024-04-02T05:05:29.028385+0000 mon.b (mon.1) 344 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:05:30.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:29 smithi069 bash[28904]: audit 2024-04-02T05:05:29.040016+0000 mon.a (mon.0) 4130 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:05:30.180 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.890597+0000 osd.4 (osd.4) 478 : cluster 0 4.9 scrub starts 2024-04-02T05:05:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.893098+0000 osd.4 (osd.4) 479 : cluster 0 4.9 scrub ok 2024-04-02T05:05:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.938286+0000 osd.3 (osd.3) 517 : cluster 0 3.18 scrub starts 2024-04-02T05:05:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:28.940453+0000 osd.3 (osd.3) 518 : cluster 0 3.18 scrub ok 2024-04-02T05:05:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:29.319962+0000 osd.7 (osd.7) 301 : cluster 0 3.15 deep-scrub starts 2024-04-02T05:05:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:29.330301+0000 osd.7 (osd.7) 302 : cluster 0 3.15 deep-scrub ok 2024-04-02T05:05:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:29.887152+0000 osd.5 (osd.5) 321 : cluster 0 1.8 scrub starts 2024-04-02T05:05:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:29.890495+0000 osd.5 (osd.5) 322 : cluster 0 1.8 scrub ok 2024-04-02T05:05:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.191191+0000 mon.a (mon.0) 4131 : cluster 0 osdmap e931: 8 total, 8 up, 8 in 2024-04-02T05:05:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.221879+0000 osd.6 (osd.6) 443 : cluster 0 10.d scrub starts 2024-04-02T05:05:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:30 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.228129+0000 osd.6 (osd.6) 444 : cluster 0 10.d scrub ok 2024-04-02T05:05:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.890597+0000 osd.4 (osd.4) 478 : cluster 0 4.9 scrub starts 2024-04-02T05:05:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.893098+0000 osd.4 (osd.4) 479 : cluster 0 4.9 scrub ok 2024-04-02T05:05:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.938286+0000 osd.3 (osd.3) 517 : cluster 0 3.18 scrub starts 2024-04-02T05:05:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:28.940453+0000 osd.3 (osd.3) 518 : cluster 0 3.18 scrub ok 2024-04-02T05:05:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:29.319962+0000 osd.7 (osd.7) 301 : cluster 0 3.15 deep-scrub starts 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:29.330301+0000 osd.7 (osd.7) 302 : cluster 0 3.15 deep-scrub ok 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:29.887152+0000 osd.5 (osd.5) 321 : cluster 0 1.8 scrub starts 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:29.890495+0000 osd.5 (osd.5) 322 : cluster 0 1.8 scrub ok 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.191191+0000 mon.a (mon.0) 4131 : cluster 0 osdmap e931: 8 total, 8 up, 8 in 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.221879+0000 osd.6 (osd.6) 443 : cluster 0 10.d scrub starts 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.228129+0000 osd.6 (osd.6) 444 : cluster 0 10.d scrub ok 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.890597+0000 osd.4 (osd.4) 478 : cluster 0 4.9 scrub starts 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.893098+0000 osd.4 (osd.4) 479 : cluster 0 4.9 scrub ok 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.938286+0000 osd.3 (osd.3) 517 : cluster 0 3.18 scrub starts 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:28.940453+0000 osd.3 (osd.3) 518 : cluster 0 3.18 scrub ok 2024-04-02T05:05:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:29.319962+0000 osd.7 (osd.7) 301 : cluster 0 3.15 deep-scrub starts 2024-04-02T05:05:31.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:29.330301+0000 osd.7 (osd.7) 302 : cluster 0 3.15 deep-scrub ok 2024-04-02T05:05:31.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:29.887152+0000 osd.5 (osd.5) 321 : cluster 0 1.8 scrub starts 2024-04-02T05:05:31.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:29.890495+0000 osd.5 (osd.5) 322 : cluster 0 1.8 scrub ok 2024-04-02T05:05:31.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.191191+0000 mon.a (mon.0) 4131 : cluster 0 osdmap e931: 8 total, 8 up, 8 in 2024-04-02T05:05:31.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.221879+0000 osd.6 (osd.6) 443 : cluster 0 10.d scrub starts 2024-04-02T05:05:31.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:30 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.228129+0000 osd.6 (osd.6) 444 : cluster 0 10.d scrub ok 2024-04-02T05:05:31.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.327658+0000 osd.7 (osd.7) 303 : cluster 0 6.8 scrub starts 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.330112+0000 osd.7 (osd.7) 304 : cluster 0 6.8 scrub ok 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.428941+0000 mgr.y (mgr.14538) 836 : cluster 0 pgmap v1453: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.900421+0000 osd.5 (osd.5) 323 : cluster 0 6.d scrub starts 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.903250+0000 osd.5 (osd.5) 324 : cluster 0 6.d scrub ok 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:31.230737+0000 osd.6 (osd.6) 445 : cluster 0 10.1c deep-scrub starts 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[21528]: cluster 2024-04-02T05:05:31.233044+0000 osd.6 (osd.6) 446 : cluster 0 10.1c deep-scrub ok 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.327658+0000 osd.7 (osd.7) 303 : cluster 0 6.8 scrub starts 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.330112+0000 osd.7 (osd.7) 304 : cluster 0 6.8 scrub ok 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.428941+0000 mgr.y (mgr.14538) 836 : cluster 0 pgmap v1453: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.900421+0000 osd.5 (osd.5) 323 : cluster 0 6.d scrub starts 2024-04-02T05:05:31.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.903250+0000 osd.5 (osd.5) 324 : cluster 0 6.d scrub ok 2024-04-02T05:05:31.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:31.230737+0000 osd.6 (osd.6) 445 : cluster 0 10.1c deep-scrub starts 2024-04-02T05:05:31.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:31 smithi069 bash[28904]: cluster 2024-04-02T05:05:31.233044+0000 osd.6 (osd.6) 446 : cluster 0 10.1c deep-scrub ok 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.327658+0000 osd.7 (osd.7) 303 : cluster 0 6.8 scrub starts 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.330112+0000 osd.7 (osd.7) 304 : cluster 0 6.8 scrub ok 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.428941+0000 mgr.y (mgr.14538) 836 : cluster 0 pgmap v1453: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.900421+0000 osd.5 (osd.5) 323 : cluster 0 6.d scrub starts 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.903250+0000 osd.5 (osd.5) 324 : cluster 0 6.d scrub ok 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:31.230737+0000 osd.6 (osd.6) 445 : cluster 0 10.1c deep-scrub starts 2024-04-02T05:05:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:31 smithi098 bash[23790]: cluster 2024-04-02T05:05:31.233044+0000 osd.6 (osd.6) 446 : cluster 0 10.1c deep-scrub ok 2024-04-02T05:05:32.001 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:32 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.930860+0000 osd.3 (osd.3) 519 : cluster 0 6.13 deep-scrub starts 2024-04-02T05:05:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:32 smithi098 bash[23790]: cluster 2024-04-02T05:05:30.933139+0000 osd.3 (osd.3) 520 : cluster 0 6.13 deep-scrub ok 2024-04-02T05:05:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:32 smithi098 bash[23790]: cluster 2024-04-02T05:05:31.268372+0000 osd.0 (osd.0) 487 : cluster 0 6.0 scrub starts 2024-04-02T05:05:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:32 smithi098 bash[23790]: cluster 2024-04-02T05:05:31.270646+0000 osd.0 (osd.0) 488 : cluster 0 6.0 scrub ok 2024-04-02T05:05:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:32 smithi098 bash[23790]: cluster 2024-04-02T05:05:32.217504+0000 osd.6 (osd.6) 447 : cluster 0 2.14 scrub starts 2024-04-02T05:05:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:32 smithi098 bash[23790]: cluster 2024-04-02T05:05:32.219913+0000 osd.6 (osd.6) 448 : cluster 0 2.14 scrub ok 2024-04-02T05:05:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.930860+0000 osd.3 (osd.3) 519 : cluster 0 6.13 deep-scrub starts 2024-04-02T05:05:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[28904]: cluster 2024-04-02T05:05:30.933139+0000 osd.3 (osd.3) 520 : cluster 0 6.13 deep-scrub ok 2024-04-02T05:05:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[28904]: cluster 2024-04-02T05:05:31.268372+0000 osd.0 (osd.0) 487 : cluster 0 6.0 scrub starts 2024-04-02T05:05:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[28904]: cluster 2024-04-02T05:05:31.270646+0000 osd.0 (osd.0) 488 : cluster 0 6.0 scrub ok 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[28904]: cluster 2024-04-02T05:05:32.217504+0000 osd.6 (osd.6) 447 : cluster 0 2.14 scrub starts 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[28904]: cluster 2024-04-02T05:05:32.219913+0000 osd.6 (osd.6) 448 : cluster 0 2.14 scrub ok 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.930860+0000 osd.3 (osd.3) 519 : cluster 0 6.13 deep-scrub starts 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[21528]: cluster 2024-04-02T05:05:30.933139+0000 osd.3 (osd.3) 520 : cluster 0 6.13 deep-scrub ok 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[21528]: cluster 2024-04-02T05:05:31.268372+0000 osd.0 (osd.0) 487 : cluster 0 6.0 scrub starts 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[21528]: cluster 2024-04-02T05:05:31.270646+0000 osd.0 (osd.0) 488 : cluster 0 6.0 scrub ok 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[21528]: cluster 2024-04-02T05:05:32.217504+0000 osd.6 (osd.6) 447 : cluster 0 2.14 scrub starts 2024-04-02T05:05:33.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:32 smithi069 bash[21528]: cluster 2024-04-02T05:05:32.219913+0000 osd.6 (osd.6) 448 : cluster 0 2.14 scrub ok 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/3843041614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806b260 msgr2=0x7f126810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- 172.21.15.69:0/3843041614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806b260 0x7f126810e7b0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f1264009990 tx=0x7f1264009ca0 comp rx=0 tx=0).stop 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/3843041614 shutdown_connections 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- 172.21.15.69:0/3843041614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 0x7f12681110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- 172.21.15.69:0/3843041614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f126806a940 0x7f126806ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- 172.21.15.69:0/3843041614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806b260 0x7f126810e7b0 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:33.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/3843041614 >> 172.21.15.69:0/3843041614 conn(0x7f12680780e0 msgr2=0x7f12680784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/3843041614 shutdown_connections 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/3843041614 wait complete. 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 librados: starting msgr at 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 librados: starting objecter 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 Processor -- start 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- start start 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 librados: setting wanted keys 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 librados: calling monclient init 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 0x7f126819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f126806b260 0x7f126819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 0x7f126819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1268113c40 con 0x7f126806b260 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1268113ac0 con 0x7f126810ecf0 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1268113dc0 con 0x7f126806a940 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 0x7f126819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:33.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 0x7f126819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47614/0 (socket says 172.21.15.69:47614) 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 -- 172.21.15.69:0/1955386813 learned_addr learned my addr 172.21.15.69:0/1955386813 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126ce61700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f126806b260 0x7f126819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126de63700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 0x7f126819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 -- 172.21.15.69:0/1955386813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 msgr2=0x7f126819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 0x7f126819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 -- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f126806b260 msgr2=0x7f126819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f126806b260 0x7f126819a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 -- 172.21.15.69:0/1955386813 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126819f300 con 0x7f126806a940 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126de63700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 0x7f126819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126d662700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 0x7f126819a2f0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f125c00eb30 tx=0x7f125c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f125a7fc700 10 client.?.objecter ms_handle_connect 0x7f126806a940 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f125a7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c00cc90 con 0x7f126806a940 2024-04-02T05:05:33.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f125c00ce30 con 0x7f126806a940 2024-04-02T05:05:33.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126819f5f0 con 0x7f126806a940 2024-04-02T05:05:33.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c003df0 con 0x7f126806a940 2024-04-02T05:05:33.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 10 client.43967.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:05:33.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12681a6e70 con 0x7f126806a940 2024-04-02T05:05:33.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.602+0000 7f126f8c6700 1 librados: init done 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f126804ff00 con 0x7f126806a940 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f125c003f90 con 0x7f126806a940 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f125406f5e0 0x7f1254071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(931..931 src has 795..931) ==== 9044+0+0 (secure 0 0 0) 0x7f125c08e8c0 con 0x7f126806a940 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f126ce61700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f125406f5e0 0x7f1254071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 10 client.43967.objecter ms_dispatch 0x7f1268001550 osd_map(931..931 src has 795..931) 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 3 client.43967.objecter handle_osd_map got epochs [931,931] > 0 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 3 client.43967.objecter handle_osd_map decoding full epoch 931 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 20 client.43967.objecter dump_active .. 0 homeless 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f125a7fc700 10 client.43967.objecter ms_handle_connect 0x7f125406f5e0 2024-04-02T05:05:33.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.606+0000 7f126ce61700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f125406f5e0 0x7f1254071aa0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7f12640098e0 tx=0x7f1264005960 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:33.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.610+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f125c05b8f0 con 0x7f126806a940 2024-04-02T05:05:33.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:33 smithi069 bash[28904]: cluster 2024-04-02T05:05:32.429712+0000 mgr.y (mgr.14538) 837 : cluster 0 pgmap v1454: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-02T05:05:33.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:33 smithi069 bash[28904]: cluster 2024-04-02T05:05:33.170985+0000 osd.6 (osd.6) 449 : cluster 0 2.11 scrub starts 2024-04-02T05:05:33.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:33 smithi069 bash[28904]: cluster 2024-04-02T05:05:33.173468+0000 osd.6 (osd.6) 450 : cluster 0 2.11 scrub ok 2024-04-02T05:05:33.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:33 smithi069 bash[21528]: cluster 2024-04-02T05:05:32.429712+0000 mgr.y (mgr.14538) 837 : cluster 0 pgmap v1454: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-02T05:05:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:33 smithi069 bash[21528]: cluster 2024-04-02T05:05:33.170985+0000 osd.6 (osd.6) 449 : cluster 0 2.11 scrub starts 2024-04-02T05:05:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:33 smithi069 bash[21528]: cluster 2024-04-02T05:05:33.173468+0000 osd.6 (osd.6) 450 : cluster 0 2.11 scrub ok 2024-04-02T05:05:33.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:33.830+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f12681a7040 con 0x7f126806a940 2024-04-02T05:05:33.888 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:33 smithi098 bash[23790]: cluster 2024-04-02T05:05:32.429712+0000 mgr.y (mgr.14538) 837 : cluster 0 pgmap v1454: 184 pgs: 184 active+clean; 582 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-02T05:05:33.888 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:33 smithi098 bash[23790]: cluster 2024-04-02T05:05:33.170985+0000 osd.6 (osd.6) 449 : cluster 0 2.11 scrub starts 2024-04-02T05:05:33.888 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:33 smithi098 bash[23790]: cluster 2024-04-02T05:05:33.173468+0000 osd.6 (osd.6) 450 : cluster 0 2.11 scrub ok 2024-04-02T05:05:34.190 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:05:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:05:34.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.662+0000 7f125a7fc700 1 -- 172.21.15.69:0/1955386813 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v932) ==== 113+0+0 (secure 0 0 0) 0x7f125c05f340 con 0x7f126806a940 2024-04-02T05:05:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 10 librados: watch_flush enter 2024-04-02T05:05:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 10 librados: watch_flush exit 2024-04-02T05:05:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 20 client.43967.objecter shutdown clearing up homeless session... 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 10 client.43967.objecter successfully canceled tick 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f125406f5e0 msgr2=0x7f1254071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f125406f5e0 0x7f1254071aa0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7f12640098e0 tx=0x7f1264005960 comp rx=0 tx=0).stop 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 msgr2=0x7f126819a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 0x7f126819a2f0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f125c00eb30 tx=0x7f125c00ee40 comp rx=0 tx=0).stop 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 shutdown_connections 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f125406f5e0 0x7f1254071aa0 unknown :-1 s=CLOSED pgs=4040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f126810ecf0 0x7f126819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f126806a940 0x7f126819a2f0 secure :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f125c00eb30 tx=0x7f125c00ee40 comp rx=0 tx=0).stop 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 --2- 172.21.15.69:0/1955386813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f126806b260 0x7f126819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 >> 172.21.15.69:0/1955386813 conn(0x7f12680780e0 msgr2=0x7f1268100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 shutdown_connections 2024-04-02T05:05:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 -- 172.21.15.69:0/1955386813 wait complete. 2024-04-02T05:05:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:34.666+0000 7f126f8c6700 1 librados: shutdown 2024-04-02T05:05:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.3 to 1 (10000) 2024-04-02T05:05:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: cluster 2024-04-02T05:05:33.190714+0000 osd.0 (osd.0) 489 : cluster 0 2.13 scrub starts 2024-04-02T05:05:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: cluster 2024-04-02T05:05:33.192572+0000 osd.0 (osd.0) 490 : cluster 0 2.13 scrub ok 2024-04-02T05:05:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: audit 2024-04-02T05:05:33.832659+0000 mon.c (mon.2) 1173 : audit 1 from='client.? 172.21.15.69:0/1955386813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: audit 2024-04-02T05:05:33.833260+0000 mon.a (mon.0) 4132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.040121+0000 osd.2 (osd.2) 396 : cluster 0 4.d scrub starts 2024-04-02T05:05:34.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.042908+0000 osd.2 (osd.2) 397 : cluster 0 4.d scrub ok 2024-04-02T05:05:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.208294+0000 osd.6 (osd.6) 451 : cluster 0 6.1e scrub starts 2024-04-02T05:05:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.209853+0000 osd.6 (osd.6) 452 : cluster 0 6.1e scrub ok 2024-04-02T05:05:34.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:34 smithi098 bash[23790]: audit 2024-04-02T05:05:34.511867+0000 mon.b (mon.1) 345 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:35.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: cluster 2024-04-02T05:05:33.190714+0000 osd.0 (osd.0) 489 : cluster 0 2.13 scrub starts 2024-04-02T05:05:35.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: cluster 2024-04-02T05:05:33.192572+0000 osd.0 (osd.0) 490 : cluster 0 2.13 scrub ok 2024-04-02T05:05:35.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: audit 2024-04-02T05:05:33.832659+0000 mon.c (mon.2) 1173 : audit 1 from='client.? 172.21.15.69:0/1955386813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: audit 2024-04-02T05:05:33.833260+0000 mon.a (mon.0) 4132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.040121+0000 osd.2 (osd.2) 396 : cluster 0 4.d scrub starts 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.042908+0000 osd.2 (osd.2) 397 : cluster 0 4.d scrub ok 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.208294+0000 osd.6 (osd.6) 451 : cluster 0 6.1e scrub starts 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.209853+0000 osd.6 (osd.6) 452 : cluster 0 6.1e scrub ok 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[28904]: audit 2024-04-02T05:05:34.511867+0000 mon.b (mon.1) 345 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: cluster 2024-04-02T05:05:33.190714+0000 osd.0 (osd.0) 489 : cluster 0 2.13 scrub starts 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: cluster 2024-04-02T05:05:33.192572+0000 osd.0 (osd.0) 490 : cluster 0 2.13 scrub ok 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: audit 2024-04-02T05:05:33.832659+0000 mon.c (mon.2) 1173 : audit 1 from='client.? 172.21.15.69:0/1955386813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: audit 2024-04-02T05:05:33.833260+0000 mon.a (mon.0) 4132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.040121+0000 osd.2 (osd.2) 396 : cluster 0 4.d scrub starts 2024-04-02T05:05:35.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.042908+0000 osd.2 (osd.2) 397 : cluster 0 4.d scrub ok 2024-04-02T05:05:35.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.208294+0000 osd.6 (osd.6) 451 : cluster 0 6.1e scrub starts 2024-04-02T05:05:35.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.209853+0000 osd.6 (osd.6) 452 : cluster 0 6.1e scrub ok 2024-04-02T05:05:35.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:34 smithi069 bash[21528]: audit 2024-04-02T05:05:34.511867+0000 mon.b (mon.1) 345 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4633 3.5 3:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94452576124928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:10.248626+0000", 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 85.047651235000004, 2024-04-02T05:05:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.021456994, 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4633 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248626+0000", 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248626+0000", 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248628+0000", 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.86e-06 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248635+0000", 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8220000000000001e-06 2024-04-02T05:05:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248678+0000", 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2864000000000002e-05 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248694+0000", 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5651e-05 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248759+0000", 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5018000000000007e-05 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248766+0000", 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0090000000000001e-06 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.256453+0000", 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076872699999999 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for missing object", 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.257406+0000", 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00095330000000000002 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.264126+0000", 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006720129 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.264713+0000", 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058655899999999995 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266056+0000", 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001343466 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266170+0000", 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011336999999999999 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.270083+0000", 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039136759999999996 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4635 4.15 4:a93a5511:::notify.2:head [watch reconnect cookie 94452570227712 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:10.248724+0000", 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 85.047554038000001, 2024-04-02T05:05:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0214099569999999, 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4635 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248724+0000", 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248724+0000", 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248724+0000", 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3900000000000004e-07 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248729+0000", 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3000000000000003e-06 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248738+0000", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7420000000000003e-06 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248743+0000", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.93e-06 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248790+0000", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6863999999999997e-05 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248801+0000", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0811e-05 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.257540+0000", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.008738892 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for missing object", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.257993+0000", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045254300000000002 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263832+0000", 2024-04-02T05:05:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058393830000000001 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263933+0000", 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010124399999999999 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.265812+0000", 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018785010000000001 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.265900+0000", 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8399999999999994e-05 2024-04-02T05:05:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.270134+0000", 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042335080000000004 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4636 4.3 4:c609908c:::notify.5:head [watch reconnect cookie 94452570221952 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:10.248757+0000", 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 85.047520766999995, 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0173513970000001, 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4636 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248757+0000", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248757+0000", 2024-04-02T05:05:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248757+0000", 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5900000000000002e-07 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248760+0000", 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1880000000000001e-06 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248765+0000", 2024-04-02T05:05:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.322e-06 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248771+0000", 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1009999999999997e-06 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248821+0000", 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0209e-05 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248827+0000", 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7119999999999997e-06 2024-04-02T05:05:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.250539+0000", 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.001711749 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for missing object", 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.250751+0000", 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021225500000000001 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262563+0000", 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011812357000000001 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263395+0000", 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00083190100000000002 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.264093+0000", 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069782399999999999 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.264187+0000", 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4063999999999999e-05 2024-04-02T05:05:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266108+0000", 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019212560000000001 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(4.3 902/901 [PushOp(4:c609908c:::notify.5:head, version: 121'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:c609908c:::notify.5:head@121'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:11.259376+0000", 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 84.036901310999994, 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031435959999999998, 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.259376+0000", 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.259376+0000", 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.259377+0000", 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4799999999999997e-07 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.259396+0000", 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8998e-05 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.259422+0000", 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5547000000000001e-05 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.259444+0000", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.243e-05 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.261463+0000", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020184019999999999 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.261480+0000", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7751e-05 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262520+0000", 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00103962 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(3.1 902/901 [PushOp(3:8386e9da:lc::lc.16:head, version: 756'38, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8386e9da:lc::lc.16:head@756'38, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:11.260394+0000", 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 84.035883721000005, 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060301230000000001, 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260394+0000", 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260394+0000", 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260394+0000", 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1600000000000002e-07 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260408+0000", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4185999999999999e-05 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260426+0000", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7691000000000001e-05 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260447+0000", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1268e-05 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266076+0000", 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056289540000000002 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266097+0000", 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0676999999999999e-05 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266424+0000", 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032703099999999998 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(4.15 902/901 [PushOp(4:a93a5511:::notify.2:head, version: 85'11, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a93a5511:::notify.2:head@85'11, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:11.262306+0000", 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 84.033971804000004, 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012989799999999999, 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262306+0000", 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262306+0000", 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262307+0000", 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2100000000000001e-06 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262330+0000", 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3110999999999999e-05 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262359+0000", 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8303999999999999e-05 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262388+0000", 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9065999999999999e-05 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262513+0000", 2024-04-02T05:05:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000125072 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262528+0000", 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5235000000000001e-05 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263605+0000", 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001076982 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(3.5 902/901 [PushOp(3:a60f9269:lc::lua_package_allowlist:head, version: 85'4, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a60f9269:lc::lua_package_allowlist:head@85'4, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:11.262491+0000", 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 84.033786359999993, 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00151012, 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262491+0000", 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262491+0000", 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262492+0000", 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262506+0000", 2024-04-02T05:05:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4466e-05 2024-04-02T05:05:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262531+0000", 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4352999999999999e-05 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262557+0000", 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6071000000000002e-05 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263433+0000", 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00087661200000000003 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263533+0000", 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9011000000000005e-05 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.264001+0000", 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00046895800000000001 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(4.1f 902/901 [PushOp(4:f95f44c2:::notify.0:head, version: 85'11, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:f95f44c2:::notify.0:head@85'11, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:11.265469+0000", 2024-04-02T05:05:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 84.030809090999995, 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014137760000000001, 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.265469+0000", 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.265469+0000", 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.265469+0000", 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266027+0000", 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055808799999999999 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266045+0000", 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7623999999999999e-05 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266062+0000", 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7068999999999999e-05 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266719+0000", 2024-04-02T05:05:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000657432 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266736+0000", 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6742999999999999e-05 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.266882+0000", 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014617100000000001 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "pg_update_log_missing(3.4 epoch 902/901 rep_tid 1679 entries 902'74 (0'0) error 3:22d26bf9:::data_loggenerations_metadata:head by client.14415.0:4632 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:11.268772+0000", 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 84.027505439999999, 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035850220000000002, 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.268772+0000", 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.268772+0000", 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.268773+0000", 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5499999999999997e-07 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.268784+0000", 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1282e-05 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.268804+0000", 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9613999999999999e-05 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.268883+0000", 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8798e-05 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.269041+0000", 2024-04-02T05:05:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157823 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.272357+0000", 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00331665 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDPGPush(1.8 903/902 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 23'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@23'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:12.202371+0000", 2024-04-02T05:05:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.093906805000003, 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014164570000000001, 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.202371+0000", 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.202371+0000", 2024-04-02T05:05:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.202372+0000", 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.202895+0000", 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052334899999999999 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.202934+0000", 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8671999999999998e-05 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.202968+0000", 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4230000000000003e-05 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.203155+0000", 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018668600000000001 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.203193+0000", 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8275999999999997e-05 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:12.203787+0000", 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00059407600000000002 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(osd.5.0:1195 3.16 e905/901 ondisk, result = 0)", 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:15.803646+0000", 2024-04-02T05:05:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 79.492631244999998, 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011588340000000001, 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.803646+0000", 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.803646+0000", 2024-04-02T05:05:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.803648+0000", 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0869999999999999e-06 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.803654+0000", 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8800000000000002e-06 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.803689+0000", 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4378999999999997e-05 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.803704+0000", 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5526999999999999e-05 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.804769+0000", 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001064855 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.804781+0000", 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.208e-05 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:15.804805+0000", 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4026e-05 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4654 4.15 4:a93a5511:::notify.2:head [watch ping cookie 94452570227712 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e905)", 2024-04-02T05:05:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:20.256258+0000", 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 75.040019968999999, 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011248040000000001, 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4654 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256258+0000", 2024-04-02T05:05:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256258+0000", 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256261+0000", 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7970000000000001e-06 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256271+0000", 2024-04-02T05:05:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0777e-05 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256306+0000", 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4582000000000002e-05 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256330+0000", 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4496999999999999e-05 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.256858+0000", 2024-04-02T05:05:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052789700000000005 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.257098+0000", 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023929999999999999 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:20.257383+0000", 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028495400000000002 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(2.11,QREQUEST,e:909,from:6,reservation_nonce:3)", 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:22.809149+0000", 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 72.487129041000003, 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058462740000000003, 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809149+0000", 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809149+0000", 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809151+0000", 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2800000000000002e-06 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809163+0000", 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1816e-05 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809192+0000", 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9513e-05 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809207+0000", 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5027e-05 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809357+0000", 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014927199999999999 2024-04-02T05:05:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.809375+0000", 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8762000000000001e-05 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.814643+0000", 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052675350000000003 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.814684+0000", 2024-04-02T05:05:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0948999999999997e-05 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.814995+0000", 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031112000000000002 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4685 4.3 4:c609908c:::notify.5:head [watch ping cookie 94452570221952 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e909)", 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:35.268681+0000", 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 60.027596686000003, 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015465819999999999, 2024-04-02T05:05:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4685 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268681+0000", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268681+0000", 2024-04-02T05:05:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268682+0000", 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.42e-06 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268689+0000", 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9999999999999999e-06 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268709+0000", 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.948e-05 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268725+0000", 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5662e-05 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268845+0000", 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120116 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.269075+0000", 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000229903 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.270228+0000", 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011530010000000001 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4686 3.5 3:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94452576124928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e909)", 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:35.268787+0000", 2024-04-02T05:05:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 60.027490880999999, 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014464440000000001, 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4686 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268787+0000", 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268787+0000", 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268788+0000", 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1969999999999999e-06 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268794+0000", 2024-04-02T05:05:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0909999999999998e-06 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268811+0000", 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6494000000000002e-05 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268824+0000", 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3026e-05 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.268934+0000", 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110497 2024-04-02T05:05:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.269138+0000", 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020344800000000001 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:35.270233+0000", 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001095691 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4732 4.15 4:a93a5511:::notify.2:head [watch reconnect cookie 94452570227712 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-04-02T05:05:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:56.316906+0000", 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.979371704999998, 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0344666810000001, 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4732 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.316906+0000", 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.316906+0000", 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.316910+0000", 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0629999999999999e-06 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.316928+0000", 2024-04-02T05:05:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7873e-05 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.316976+0000", 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.829e-05 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.317004+0000", 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7613000000000001e-05 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.317200+0000", 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019578799999999999 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.317219+0000", 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9606000000000002e-05 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.325229+0000", 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0080096319999999992 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:56.325260+0000", 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1575999999999997e-05 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.347095+0000", 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.02183472 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for degraded object", 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.347908+0000", 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00081263699999999995 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.349567+0000", 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016590109999999999 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.349668+0000", 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000100783 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:57.351373+0000", 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017050889999999999 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.7.0:395 2.1c e920/901 2:3fa705ff:::zone_info.42d89671-35c8-448f-8f76-91e8aab74f30:head v 920'2, mlcod=0'0)", 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:05:09.466035+0000", 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 25.830242431999999, 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001102848, 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466035+0000", 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466035+0000", 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466038+0000", 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.317e-06 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466057+0000", 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9072999999999998e-05 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466099+0000", 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2039999999999997e-05 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466123+0000", 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4153000000000001e-05 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466316+0000", 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019289700000000001 2024-04-02T05:05:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.466383+0000", 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7632999999999994e-05 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.467049+0000", 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00066595599999999999 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:09.467138+0000", 2024-04-02T05:05:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8778999999999998e-05 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4762 4.15 4:a93a5511:::notify.2:head [watch ping cookie 94452570227712 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e919)", 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:05:10.292760+0000", 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 25.003517274, 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011221600000000001, 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4762 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.292760+0000", 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.292760+0000", 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.292764+0000", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4829999999999999e-06 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.292779+0000", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5123000000000001e-05 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.292817+0000", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8279999999999999e-05 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.292837+0000", 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0000000000000002e-05 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.293356+0000", 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051828699999999998 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.293587+0000", 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023189400000000001 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:10.293883+0000", 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029509300000000002 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4805 4.15 4:a93a5511:::notify.2:head [watch ping cookie 94452570227712 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e929)", 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:05:30.309363+0000", 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.9869151790000004, 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011589879999999999, 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4805 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309363+0000", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309363+0000", 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309365+0000", 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.537e-06 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309375+0000", 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0113e-05 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309400+0000", 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5247000000000001e-05 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309424+0000", 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3776000000000001e-05 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309870+0000", 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000446052 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.310126+0000", 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025543400000000002 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.310522+0000", 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039582899999999999 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4806 3.5 3:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94452576124928 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e929)", 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:05:30.309689+0000", 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.9865884620000003, 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016234089999999999, 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:05:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4806 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309689+0000", 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309689+0000", 2024-04-02T05:05:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309691+0000", 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309701+0000", 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7359999999999996e-06 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309725+0000", 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4346e-05 2024-04-02T05:05:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.309745+0000", 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0279999999999999e-05 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.310749+0000", 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001003323 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.311023+0000", 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027455300000000003 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:30.311313+0000", 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028947100000000002 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:05:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:05:35.392 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:35.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 3 1 2024-04-02T05:05:35.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: audit 2024-04-02T05:05:33.886172+0000 mgr.y (mgr.14538) 838 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:35.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.061577+0000 osd.3 (osd.3) 521 : cluster 0 10.15 deep-scrub starts 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.065378+0000 osd.3 (osd.3) 522 : cluster 0 10.15 deep-scrub ok 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.351777+0000 osd.7 (osd.7) 305 : cluster 0 6.1c scrub starts 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.354762+0000 osd.7 (osd.7) 306 : cluster 0 6.1c scrub ok 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.431692+0000 mgr.y (mgr.14538) 839 : cluster 0 pgmap v1455: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: audit 2024-04-02T05:05:34.653915+0000 mon.a (mon.0) 4133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.661055+0000 mon.a (mon.0) 4134 : cluster 0 osdmap e932: 8 total, 8 up, 8 in 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.198992+0000 mon.a (mon.0) 4135 : cluster 0 osdmap e933: 8 total, 8 up, 8 in 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: audit 2024-04-02T05:05:33.886172+0000 mgr.y (mgr.14538) 838 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.061577+0000 osd.3 (osd.3) 521 : cluster 0 10.15 deep-scrub starts 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.065378+0000 osd.3 (osd.3) 522 : cluster 0 10.15 deep-scrub ok 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.351777+0000 osd.7 (osd.7) 305 : cluster 0 6.1c scrub starts 2024-04-02T05:05:35.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.354762+0000 osd.7 (osd.7) 306 : cluster 0 6.1c scrub ok 2024-04-02T05:05:35.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.431692+0000 mgr.y (mgr.14538) 839 : cluster 0 pgmap v1455: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:35.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: audit 2024-04-02T05:05:34.653915+0000 mon.a (mon.0) 4133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-02T05:05:35.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.661055+0000 mon.a (mon.0) 4134 : cluster 0 osdmap e932: 8 total, 8 up, 8 in 2024-04-02T05:05:35.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:35 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.198992+0000 mon.a (mon.0) 4135 : cluster 0 osdmap e933: 8 total, 8 up, 8 in 2024-04-02T05:05:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: audit 2024-04-02T05:05:33.886172+0000 mgr.y (mgr.14538) 838 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.061577+0000 osd.3 (osd.3) 521 : cluster 0 10.15 deep-scrub starts 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.065378+0000 osd.3 (osd.3) 522 : cluster 0 10.15 deep-scrub ok 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.351777+0000 osd.7 (osd.7) 305 : cluster 0 6.1c scrub starts 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.354762+0000 osd.7 (osd.7) 306 : cluster 0 6.1c scrub ok 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.431692+0000 mgr.y (mgr.14538) 839 : cluster 0 pgmap v1455: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: audit 2024-04-02T05:05:34.653915+0000 mon.a (mon.0) 4133 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.661055+0000 mon.a (mon.0) 4134 : cluster 0 osdmap e932: 8 total, 8 up, 8 in 2024-04-02T05:05:35.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:35 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.198992+0000 mon.a (mon.0) 4135 : cluster 0 osdmap e933: 8 total, 8 up, 8 in 2024-04-02T05:05:36.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-02T05:05:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.942486+0000 osd.4 (osd.4) 480 : cluster 0 4.c scrub starts 2024-04-02T05:05:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:34.945160+0000 osd.4 (osd.4) 481 : cluster 0 4.c scrub ok 2024-04-02T05:05:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.055699+0000 osd.3 (osd.3) 523 : cluster 0 1.c scrub starts 2024-04-02T05:05:36.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.059361+0000 osd.3 (osd.3) 524 : cluster 0 1.c scrub ok 2024-04-02T05:05:36.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.224872+0000 osd.0 (osd.0) 491 : cluster 0 4.16 scrub starts 2024-04-02T05:05:36.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.228606+0000 osd.0 (osd.0) 492 : cluster 0 4.16 scrub ok 2024-04-02T05:05:36.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.326705+0000 osd.7 (osd.7) 307 : cluster 0 3.13 scrub starts 2024-04-02T05:05:36.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:36 smithi098 bash[23790]: cluster 2024-04-02T05:05:35.329069+0000 osd.7 (osd.7) 308 : cluster 0 3.13 scrub ok 2024-04-02T05:05:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.942486+0000 osd.4 (osd.4) 480 : cluster 0 4.c scrub starts 2024-04-02T05:05:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:34.945160+0000 osd.4 (osd.4) 481 : cluster 0 4.c scrub ok 2024-04-02T05:05:37.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.055699+0000 osd.3 (osd.3) 523 : cluster 0 1.c scrub starts 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.059361+0000 osd.3 (osd.3) 524 : cluster 0 1.c scrub ok 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.224872+0000 osd.0 (osd.0) 491 : cluster 0 4.16 scrub starts 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.228606+0000 osd.0 (osd.0) 492 : cluster 0 4.16 scrub ok 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.326705+0000 osd.7 (osd.7) 307 : cluster 0 3.13 scrub starts 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[21528]: cluster 2024-04-02T05:05:35.329069+0000 osd.7 (osd.7) 308 : cluster 0 3.13 scrub ok 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.942486+0000 osd.4 (osd.4) 480 : cluster 0 4.c scrub starts 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:34.945160+0000 osd.4 (osd.4) 481 : cluster 0 4.c scrub ok 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.055699+0000 osd.3 (osd.3) 523 : cluster 0 1.c scrub starts 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.059361+0000 osd.3 (osd.3) 524 : cluster 0 1.c scrub ok 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.224872+0000 osd.0 (osd.0) 491 : cluster 0 4.16 scrub starts 2024-04-02T05:05:37.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.228606+0000 osd.0 (osd.0) 492 : cluster 0 4.16 scrub ok 2024-04-02T05:05:37.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.326705+0000 osd.7 (osd.7) 307 : cluster 0 3.13 scrub starts 2024-04-02T05:05:37.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:36 smithi069 bash[28904]: cluster 2024-04-02T05:05:35.329069+0000 osd.7 (osd.7) 308 : cluster 0 3.13 scrub ok 2024-04-02T05:05:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.010727+0000 osd.1 (osd.1) 517 : cluster 0 4.4 scrub starts 2024-04-02T05:05:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.013092+0000 osd.1 (osd.1) 518 : cluster 0 4.4 scrub ok 2024-04-02T05:05:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.103423+0000 osd.3 (osd.3) 525 : cluster 0 1.4 scrub starts 2024-04-02T05:05:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.106417+0000 osd.3 (osd.3) 526 : cluster 0 1.4 scrub ok 2024-04-02T05:05:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.432457+0000 mgr.y (mgr.14538) 840 : cluster 0 pgmap v1458: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-04-02T05:05:37.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.974986+0000 osd.1 (osd.1) 519 : cluster 0 10.14 scrub starts 2024-04-02T05:05:37.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:37 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.978174+0000 osd.1 (osd.1) 520 : cluster 0 10.14 scrub ok 2024-04-02T05:05:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.010727+0000 osd.1 (osd.1) 517 : cluster 0 4.4 scrub starts 2024-04-02T05:05:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.013092+0000 osd.1 (osd.1) 518 : cluster 0 4.4 scrub ok 2024-04-02T05:05:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.103423+0000 osd.3 (osd.3) 525 : cluster 0 1.4 scrub starts 2024-04-02T05:05:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.106417+0000 osd.3 (osd.3) 526 : cluster 0 1.4 scrub ok 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.432457+0000 mgr.y (mgr.14538) 840 : cluster 0 pgmap v1458: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.974986+0000 osd.1 (osd.1) 519 : cluster 0 10.14 scrub starts 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.978174+0000 osd.1 (osd.1) 520 : cluster 0 10.14 scrub ok 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.010727+0000 osd.1 (osd.1) 517 : cluster 0 4.4 scrub starts 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.013092+0000 osd.1 (osd.1) 518 : cluster 0 4.4 scrub ok 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.103423+0000 osd.3 (osd.3) 525 : cluster 0 1.4 scrub starts 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.106417+0000 osd.3 (osd.3) 526 : cluster 0 1.4 scrub ok 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.432457+0000 mgr.y (mgr.14538) 840 : cluster 0 pgmap v1458: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.974986+0000 osd.1 (osd.1) 519 : cluster 0 10.14 scrub starts 2024-04-02T05:05:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:37 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.978174+0000 osd.1 (osd.1) 520 : cluster 0 10.14 scrub ok 2024-04-02T05:05:38.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.981539+0000 osd.4 (osd.4) 482 : cluster 0 6.17 scrub starts 2024-04-02T05:05:38.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:36.983938+0000 osd.4 (osd.4) 483 : cluster 0 6.17 scrub ok 2024-04-02T05:05:38.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.225955+0000 osd.0 (osd.0) 493 : cluster 0 10.1e scrub starts 2024-04-02T05:05:38.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.229596+0000 osd.0 (osd.0) 494 : cluster 0 10.1e scrub ok 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.643889+0000 osd.0 (osd.0) 495 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.644272+0000 osd.0 (osd.0) 496 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.687834+0000 mon.a (mon.0) 4136 : cluster 0 osdmap e934: 8 total, 8 up, 8 in 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.843595+0000 osd.5 (osd.5) 325 : cluster 0 4.11 scrub starts 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.847884+0000 osd.5 (osd.5) 326 : cluster 0 4.11 scrub ok 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.941876+0000 osd.1 (osd.1) 521 : cluster 0 6.1d scrub starts 2024-04-02T05:05:38.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:38 smithi098 bash[23790]: cluster 2024-04-02T05:05:37.944293+0000 osd.1 (osd.1) 522 : cluster 0 6.1d scrub ok 2024-04-02T05:05:39.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:05:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:05:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.981539+0000 osd.4 (osd.4) 482 : cluster 0 6.17 scrub starts 2024-04-02T05:05:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:36.983938+0000 osd.4 (osd.4) 483 : cluster 0 6.17 scrub ok 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.225955+0000 osd.0 (osd.0) 493 : cluster 0 10.1e scrub starts 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.229596+0000 osd.0 (osd.0) 494 : cluster 0 10.1e scrub ok 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.643889+0000 osd.0 (osd.0) 495 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.644272+0000 osd.0 (osd.0) 496 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.687834+0000 mon.a (mon.0) 4136 : cluster 0 osdmap e934: 8 total, 8 up, 8 in 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.843595+0000 osd.5 (osd.5) 325 : cluster 0 4.11 scrub starts 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.847884+0000 osd.5 (osd.5) 326 : cluster 0 4.11 scrub ok 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.941876+0000 osd.1 (osd.1) 521 : cluster 0 6.1d scrub starts 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[21528]: cluster 2024-04-02T05:05:37.944293+0000 osd.1 (osd.1) 522 : cluster 0 6.1d scrub ok 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.981539+0000 osd.4 (osd.4) 482 : cluster 0 6.17 scrub starts 2024-04-02T05:05:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:36.983938+0000 osd.4 (osd.4) 483 : cluster 0 6.17 scrub ok 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.225955+0000 osd.0 (osd.0) 493 : cluster 0 10.1e scrub starts 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.229596+0000 osd.0 (osd.0) 494 : cluster 0 10.1e scrub ok 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.643889+0000 osd.0 (osd.0) 495 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.644272+0000 osd.0 (osd.0) 496 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.687834+0000 mon.a (mon.0) 4136 : cluster 0 osdmap e934: 8 total, 8 up, 8 in 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.843595+0000 osd.5 (osd.5) 325 : cluster 0 4.11 scrub starts 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.847884+0000 osd.5 (osd.5) 326 : cluster 0 4.11 scrub ok 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.941876+0000 osd.1 (osd.1) 521 : cluster 0 6.1d scrub starts 2024-04-02T05:05:39.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:38 smithi069 bash[28904]: cluster 2024-04-02T05:05:37.944293+0000 osd.1 (osd.1) 522 : cluster 0 6.1d scrub ok 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.132107+0000 osd.3 (osd.3) 527 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.138219+0000 osd.3 (osd.3) 528 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.387932+0000 osd.7 (osd.7) 309 : cluster 0 10.c scrub starts 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.390571+0000 osd.7 (osd.7) 310 : cluster 0 10.c scrub ok 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.434252+0000 mgr.y (mgr.14538) 841 : cluster 0 pgmap v1460: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.932236+0000 osd.1 (osd.1) 523 : cluster 0 2.6 scrub starts 2024-04-02T05:05:39.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.935129+0000 osd.1 (osd.1) 524 : cluster 0 2.6 scrub ok 2024-04-02T05:05:39.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:39.167393+0000 osd.6 (osd.6) 453 : cluster 0 3.a scrub starts 2024-04-02T05:05:39.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:39 smithi098 bash[23790]: cluster 2024-04-02T05:05:39.172417+0000 osd.6 (osd.6) 454 : cluster 0 3.a scrub ok 2024-04-02T05:05:40.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.132107+0000 osd.3 (osd.3) 527 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:05:40.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.138219+0000 osd.3 (osd.3) 528 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:05:40.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.387932+0000 osd.7 (osd.7) 309 : cluster 0 10.c scrub starts 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.390571+0000 osd.7 (osd.7) 310 : cluster 0 10.c scrub ok 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.434252+0000 mgr.y (mgr.14538) 841 : cluster 0 pgmap v1460: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.932236+0000 osd.1 (osd.1) 523 : cluster 0 2.6 scrub starts 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.935129+0000 osd.1 (osd.1) 524 : cluster 0 2.6 scrub ok 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:39.167393+0000 osd.6 (osd.6) 453 : cluster 0 3.a scrub starts 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[21528]: cluster 2024-04-02T05:05:39.172417+0000 osd.6 (osd.6) 454 : cluster 0 3.a scrub ok 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.132107+0000 osd.3 (osd.3) 527 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.138219+0000 osd.3 (osd.3) 528 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.387932+0000 osd.7 (osd.7) 309 : cluster 0 10.c scrub starts 2024-04-02T05:05:40.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.390571+0000 osd.7 (osd.7) 310 : cluster 0 10.c scrub ok 2024-04-02T05:05:40.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.434252+0000 mgr.y (mgr.14538) 841 : cluster 0 pgmap v1460: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:05:40.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.932236+0000 osd.1 (osd.1) 523 : cluster 0 2.6 scrub starts 2024-04-02T05:05:40.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.935129+0000 osd.1 (osd.1) 524 : cluster 0 2.6 scrub ok 2024-04-02T05:05:40.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:39.167393+0000 osd.6 (osd.6) 453 : cluster 0 3.a scrub starts 2024-04-02T05:05:40.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:39 smithi069 bash[28904]: cluster 2024-04-02T05:05:39.172417+0000 osd.6 (osd.6) 454 : cluster 0 3.a scrub ok 2024-04-02T05:05:40.278 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.968307+0000 osd.4 (osd.4) 484 : cluster 0 6.1a scrub starts 2024-04-02T05:05:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:38.970586+0000 osd.4 (osd.4) 485 : cluster 0 6.1a scrub ok 2024-04-02T05:05:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:39.427964+0000 osd.7 (osd.7) 311 : cluster 0 2.1d scrub starts 2024-04-02T05:05:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:39.430511+0000 osd.7 (osd.7) 312 : cluster 0 2.1d scrub ok 2024-04-02T05:05:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:39.863299+0000 osd.5 (osd.5) 327 : cluster 0 3.5 scrub starts 2024-04-02T05:05:40.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:39.868628+0000 osd.5 (osd.5) 328 : cluster 0 3.5 scrub ok 2024-04-02T05:05:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.017148+0000 osd.2 (osd.2) 398 : cluster 0 2.1 scrub starts 2024-04-02T05:05:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.019468+0000 osd.2 (osd.2) 399 : cluster 0 2.1 scrub ok 2024-04-02T05:05:40.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:40 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.200218+0000 mon.a (mon.0) 4137 : cluster 0 osdmap e935: 8 total, 8 up, 8 in 2024-04-02T05:05:41.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.968307+0000 osd.4 (osd.4) 484 : cluster 0 6.1a scrub starts 2024-04-02T05:05:41.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:38.970586+0000 osd.4 (osd.4) 485 : cluster 0 6.1a scrub ok 2024-04-02T05:05:41.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:39.427964+0000 osd.7 (osd.7) 311 : cluster 0 2.1d scrub starts 2024-04-02T05:05:41.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:39.430511+0000 osd.7 (osd.7) 312 : cluster 0 2.1d scrub ok 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:39.863299+0000 osd.5 (osd.5) 327 : cluster 0 3.5 scrub starts 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:39.868628+0000 osd.5 (osd.5) 328 : cluster 0 3.5 scrub ok 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.017148+0000 osd.2 (osd.2) 398 : cluster 0 2.1 scrub starts 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.019468+0000 osd.2 (osd.2) 399 : cluster 0 2.1 scrub ok 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.200218+0000 mon.a (mon.0) 4137 : cluster 0 osdmap e935: 8 total, 8 up, 8 in 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.968307+0000 osd.4 (osd.4) 484 : cluster 0 6.1a scrub starts 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:38.970586+0000 osd.4 (osd.4) 485 : cluster 0 6.1a scrub ok 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:39.427964+0000 osd.7 (osd.7) 311 : cluster 0 2.1d scrub starts 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:39.430511+0000 osd.7 (osd.7) 312 : cluster 0 2.1d scrub ok 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:39.863299+0000 osd.5 (osd.5) 327 : cluster 0 3.5 scrub starts 2024-04-02T05:05:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:39.868628+0000 osd.5 (osd.5) 328 : cluster 0 3.5 scrub ok 2024-04-02T05:05:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.017148+0000 osd.2 (osd.2) 398 : cluster 0 2.1 scrub starts 2024-04-02T05:05:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.019468+0000 osd.2 (osd.2) 399 : cluster 0 2.1 scrub ok 2024-04-02T05:05:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:40 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.200218+0000 mon.a (mon.0) 4137 : cluster 0 osdmap e935: 8 total, 8 up, 8 in 2024-04-02T05:05:41.833 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.162914+0000 osd.3 (osd.3) 529 : cluster 0 4.19 scrub starts 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.165128+0000 osd.3 (osd.3) 530 : cluster 0 4.19 scrub ok 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.435007+0000 mgr.y (mgr.14538) 842 : cluster 0 pgmap v1462: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.872808+0000 osd.5 (osd.5) 329 : cluster 0 1.10 scrub starts 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[21528]: cluster 2024-04-02T05:05:40.876197+0000 osd.5 (osd.5) 330 : cluster 0 1.10 scrub ok 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.162914+0000 osd.3 (osd.3) 529 : cluster 0 4.19 scrub starts 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.165128+0000 osd.3 (osd.3) 530 : cluster 0 4.19 scrub ok 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.435007+0000 mgr.y (mgr.14538) 842 : cluster 0 pgmap v1462: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.872808+0000 osd.5 (osd.5) 329 : cluster 0 1.10 scrub starts 2024-04-02T05:05:41.834 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:41 smithi069 bash[28904]: cluster 2024-04-02T05:05:40.876197+0000 osd.5 (osd.5) 330 : cluster 0 1.10 scrub ok 2024-04-02T05:05:41.970 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:41 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.162914+0000 osd.3 (osd.3) 529 : cluster 0 4.19 scrub starts 2024-04-02T05:05:41.970 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:41 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.165128+0000 osd.3 (osd.3) 530 : cluster 0 4.19 scrub ok 2024-04-02T05:05:41.970 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:41 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.435007+0000 mgr.y (mgr.14538) 842 : cluster 0 pgmap v1462: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-04-02T05:05:41.970 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:41 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.872808+0000 osd.5 (osd.5) 329 : cluster 0 1.10 scrub starts 2024-04-02T05:05:41.970 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:41 smithi098 bash[23790]: cluster 2024-04-02T05:05:40.876197+0000 osd.5 (osd.5) 330 : cluster 0 1.10 scrub ok 2024-04-02T05:05:42.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[21528]: cluster 2024-04-02T05:05:41.157452+0000 osd.0 (osd.0) 497 : cluster 0 6.18 scrub starts 2024-04-02T05:05:42.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[21528]: cluster 2024-04-02T05:05:41.159802+0000 osd.0 (osd.0) 498 : cluster 0 6.18 scrub ok 2024-04-02T05:05:42.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[21528]: cluster 2024-04-02T05:05:41.890170+0000 osd.5 (osd.5) 331 : cluster 0 2.12 scrub starts 2024-04-02T05:05:42.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[21528]: cluster 2024-04-02T05:05:41.891768+0000 osd.5 (osd.5) 332 : cluster 0 2.12 scrub ok 2024-04-02T05:05:42.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[28904]: cluster 2024-04-02T05:05:41.157452+0000 osd.0 (osd.0) 497 : cluster 0 6.18 scrub starts 2024-04-02T05:05:42.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[28904]: cluster 2024-04-02T05:05:41.159802+0000 osd.0 (osd.0) 498 : cluster 0 6.18 scrub ok 2024-04-02T05:05:42.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[28904]: cluster 2024-04-02T05:05:41.890170+0000 osd.5 (osd.5) 331 : cluster 0 2.12 scrub starts 2024-04-02T05:05:42.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:42 smithi069 bash[28904]: cluster 2024-04-02T05:05:41.891768+0000 osd.5 (osd.5) 332 : cluster 0 2.12 scrub ok 2024-04-02T05:05:42.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:42 smithi098 bash[23790]: cluster 2024-04-02T05:05:41.157452+0000 osd.0 (osd.0) 497 : cluster 0 6.18 scrub starts 2024-04-02T05:05:42.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:42 smithi098 bash[23790]: cluster 2024-04-02T05:05:41.159802+0000 osd.0 (osd.0) 498 : cluster 0 6.18 scrub ok 2024-04-02T05:05:42.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:42 smithi098 bash[23790]: cluster 2024-04-02T05:05:41.890170+0000 osd.5 (osd.5) 331 : cluster 0 2.12 scrub starts 2024-04-02T05:05:42.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:42 smithi098 bash[23790]: cluster 2024-04-02T05:05:41.891768+0000 osd.5 (osd.5) 332 : cluster 0 2.12 scrub ok 2024-04-02T05:05:43.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1877986615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481024b0 msgr2=0x7f5e48105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1877986615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481024b0 0x7f5e48105fe0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0099c0 tx=0x7f5e3c009cd0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1877986615 shutdown_connections 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1877986615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481066c0 0x7f5e4810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1877986615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e48101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1877986615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481024b0 0x7f5e48105fe0 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1877986615 >> 172.21.15.69:0/1877986615 conn(0x7f5e48077740 msgr2=0x7f5e480feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1877986615 shutdown_connections 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1877986615 wait complete. 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 librados: starting msgr at 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 librados: starting objecter 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 Processor -- start 2024-04-02T05:05:43.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- start start 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 librados: setting wanted keys 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 librados: calling monclient init 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e4810ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481024b0 0x7f5e4810efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481066c0 0x7f5e4810f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48111cc0 con 0x7f5e48101b00 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48111b40 con 0x7f5e481024b0 2024-04-02T05:05:43.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5e48111e40 con 0x7f5e481066c0 2024-04-02T05:05:43.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e4810ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:43.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e4810ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59640/0 (socket says 172.21.15.69:59640) 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 -- 172.21.15.69:0/1704724438 learned_addr learned my addr 172.21.15.69:0/1704724438 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4e239700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481066c0 0x7f5e4810f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4d237700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481024b0 0x7f5e4810efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 -- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481066c0 msgr2=0x7f5e4810f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481066c0 0x7f5e4810f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 -- 172.21.15.69:0/1704724438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481024b0 msgr2=0x7f5e4810efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481024b0 0x7f5e4810efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 -- 172.21.15.69:0/1704724438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e481a67c0 con 0x7f5e48101b00 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4d237700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481024b0 0x7f5e4810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4da38700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e4810ea70 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800eb30 tx=0x7f5e3800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e46ffd700 10 client.?.objecter ms_handle_connect 0x7f5e48101b00 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e46ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800cc90 con 0x7f5e48101b00 2024-04-02T05:05:43.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f5e3800ce30 con 0x7f5e48101b00 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e481a6ab0 con 0x7f5e48101b00 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38003df0 con 0x7f5e48101b00 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 10 client.33909.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e481a6ff0 con 0x7f5e48101b00 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 librados: init done 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.474+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e4804ff00 con 0x7f5e48101b00 2024-04-02T05:05:43.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5e38003f90 con 0x7f5e48101b00 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e3006f5e0 0x7f5e30071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(935..935 src has 795..935) ==== 9044+0+0 (secure 0 0 0) 0x7f5e3808e910 con 0x7f5e48101b00 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 10 client.33909.objecter ms_dispatch 0x7f5e48001550 osd_map(935..935 src has 795..935) 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 3 client.33909.objecter handle_osd_map got epochs [935,935] > 0 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 3 client.33909.objecter handle_osd_map decoding full epoch 935 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e4d237700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e3006f5e0 0x7f5e30071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 20 client.33909.objecter dump_active .. 0 homeless 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e4d237700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e3006f5e0 0x7f5e30071aa0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000c00 tx=0x7f5e3c019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:43.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.478+0000 7f5e46ffd700 10 client.33909.objecter ms_handle_connect 0x7f5e3006f5e0 2024-04-02T05:05:43.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.482+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5e3805b8c0 con 0x7f5e48101b00 2024-04-02T05:05:43.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.706+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7f5e48069d50 con 0x7f5e48101b00 2024-04-02T05:05:43.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.710+0000 7f5e46ffd700 1 -- 172.21.15.69:0/1704724438 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v936) ==== 133+0+0 (secure 0 0 0) 0x7f5e3805f310 con 0x7f5e48101b00 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 10 librados: watch_flush enter 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 10 librados: watch_flush exit 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 20 client.33909.objecter shutdown clearing up homeless session... 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 10 client.33909.objecter successfully canceled tick 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e3006f5e0 msgr2=0x7f5e30071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e3006f5e0 0x7f5e30071aa0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000c00 tx=0x7f5e3c019040 comp rx=0 tx=0).stop 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 msgr2=0x7f5e4810ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e4810ea70 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800eb30 tx=0x7f5e3800ee40 comp rx=0 tx=0).stop 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 shutdown_connections 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5e3006f5e0 0x7f5e30071aa0 unknown :-1 s=CLOSED pgs=4041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e481066c0 0x7f5e4810f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e48101b00 0x7f5e4810ea70 secure :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800eb30 tx=0x7f5e3800ee40 comp rx=0 tx=0).stop 2024-04-02T05:05:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 --2- 172.21.15.69:0/1704724438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481024b0 0x7f5e4810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.714+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 >> 172.21.15.69:0/1704724438 conn(0x7f5e48077740 msgr2=0x7f5e481042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.718+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 shutdown_connections 2024-04-02T05:05:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.718+0000 7f5e4fc9c700 1 -- 172.21.15.69:0/1704724438 wait complete. 2024-04-02T05:05:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:43.718+0000 7f5e4fc9c700 1 librados: shutdown 2024-04-02T05:05:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-02T05:05:43.964 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:43.964 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:05:43.964 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T05:05:43.964 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:43.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:05:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:05:43.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:43 smithi098 bash[23790]: cluster 2024-04-02T05:05:42.435994+0000 mgr.y (mgr.14538) 843 : cluster 0 pgmap v1463: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-02T05:05:43.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:43 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.115223+0000 osd.2 (osd.2) 400 : cluster 0 4.f scrub starts 2024-04-02T05:05:43.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:43 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.116911+0000 osd.2 (osd.2) 401 : cluster 0 4.f scrub ok 2024-04-02T05:05:43.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:43 smithi098 bash[23790]: audit 2024-04-02T05:05:43.711087+0000 mon.a (mon.0) 4138 : audit 1 from='client.? 172.21.15.69:0/1704724438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:44.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[21528]: cluster 2024-04-02T05:05:42.435994+0000 mgr.y (mgr.14538) 843 : cluster 0 pgmap v1463: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-02T05:05:44.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.115223+0000 osd.2 (osd.2) 400 : cluster 0 4.f scrub starts 2024-04-02T05:05:44.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.116911+0000 osd.2 (osd.2) 401 : cluster 0 4.f scrub ok 2024-04-02T05:05:44.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[21528]: audit 2024-04-02T05:05:43.711087+0000 mon.a (mon.0) 4138 : audit 1 from='client.? 172.21.15.69:0/1704724438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:44.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[28904]: cluster 2024-04-02T05:05:42.435994+0000 mgr.y (mgr.14538) 843 : cluster 0 pgmap v1463: 184 pgs: 184 active+clean; 582 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-02T05:05:44.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.115223+0000 osd.2 (osd.2) 400 : cluster 0 4.f scrub starts 2024-04-02T05:05:44.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.116911+0000 osd.2 (osd.2) 401 : cluster 0 4.f scrub ok 2024-04-02T05:05:44.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:43 smithi069 bash[28904]: audit 2024-04-02T05:05:43.711087+0000 mon.a (mon.0) 4138 : audit 1 from='client.? 172.21.15.69:0/1704724438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-02T05:05:44.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 1 2024-04-02T05:05:44.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-02T05:05:44.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.397813+0000 osd.7 (osd.7) 313 : cluster 0 6.15 scrub starts 2024-04-02T05:05:44.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.400118+0000 osd.7 (osd.7) 314 : cluster 0 6.15 scrub ok 2024-04-02T05:05:44.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: audit 2024-04-02T05:05:43.714748+0000 mon.a (mon.0) 4139 : audit 1 from='client.? 172.21.15.69:0/1704724438' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-02T05:05:44.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.722750+0000 mon.a (mon.0) 4140 : cluster 0 osdmap e936: 8 total, 8 up, 8 in 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.875123+0000 osd.5 (osd.5) 333 : cluster 0 3.c deep-scrub starts 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:43.878788+0000 osd.5 (osd.5) 334 : cluster 0 3.c deep-scrub ok 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.113794+0000 osd.2 (osd.2) 402 : cluster 0 4.1c scrub starts 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.119931+0000 osd.2 (osd.2) 403 : cluster 0 4.1c scrub ok 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.181405+0000 osd.6 (osd.6) 455 : cluster 0 5.2 scrub starts 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.184363+0000 osd.6 (osd.6) 456 : cluster 0 5.2 scrub ok 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.397813+0000 osd.7 (osd.7) 313 : cluster 0 6.15 scrub starts 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.400118+0000 osd.7 (osd.7) 314 : cluster 0 6.15 scrub ok 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: audit 2024-04-02T05:05:43.714748+0000 mon.a (mon.0) 4139 : audit 1 from='client.? 172.21.15.69:0/1704724438' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.722750+0000 mon.a (mon.0) 4140 : cluster 0 osdmap e936: 8 total, 8 up, 8 in 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.875123+0000 osd.5 (osd.5) 333 : cluster 0 3.c deep-scrub starts 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:43.878788+0000 osd.5 (osd.5) 334 : cluster 0 3.c deep-scrub ok 2024-04-02T05:05:44.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.113794+0000 osd.2 (osd.2) 402 : cluster 0 4.1c scrub starts 2024-04-02T05:05:44.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.119931+0000 osd.2 (osd.2) 403 : cluster 0 4.1c scrub ok 2024-04-02T05:05:44.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.181405+0000 osd.6 (osd.6) 455 : cluster 0 5.2 scrub starts 2024-04-02T05:05:44.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:44 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.184363+0000 osd.6 (osd.6) 456 : cluster 0 5.2 scrub ok 2024-04-02T05:05:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.397813+0000 osd.7 (osd.7) 313 : cluster 0 6.15 scrub starts 2024-04-02T05:05:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.400118+0000 osd.7 (osd.7) 314 : cluster 0 6.15 scrub ok 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: audit 2024-04-02T05:05:43.714748+0000 mon.a (mon.0) 4139 : audit 1 from='client.? 172.21.15.69:0/1704724438' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.722750+0000 mon.a (mon.0) 4140 : cluster 0 osdmap e936: 8 total, 8 up, 8 in 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.875123+0000 osd.5 (osd.5) 333 : cluster 0 3.c deep-scrub starts 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:43.878788+0000 osd.5 (osd.5) 334 : cluster 0 3.c deep-scrub ok 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.113794+0000 osd.2 (osd.2) 402 : cluster 0 4.1c scrub starts 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.119931+0000 osd.2 (osd.2) 403 : cluster 0 4.1c scrub ok 2024-04-02T05:05:45.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.181405+0000 osd.6 (osd.6) 455 : cluster 0 5.2 scrub starts 2024-04-02T05:05:45.226 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:44 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.184363+0000 osd.6 (osd.6) 456 : cluster 0 5.2 scrub ok 2024-04-02T05:05:46.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[28904]: audit 2024-04-02T05:05:43.896132+0000 mgr.y (mgr.14538) 844 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:46.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.437119+0000 mgr.y (mgr.14538) 845 : cluster 0 pgmap v1465: 184 pgs: 184 active+clean; 582 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T05:05:46.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.996831+0000 osd.1 (osd.1) 525 : cluster 0 2.4 scrub starts 2024-04-02T05:05:46.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[28904]: cluster 2024-04-02T05:05:44.999737+0000 osd.1 (osd.1) 526 : cluster 0 2.4 scrub ok 2024-04-02T05:05:46.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[28904]: cluster 2024-04-02T05:05:45.202446+0000 mon.a (mon.0) 4141 : cluster 0 osdmap e937: 8 total, 8 up, 8 in 2024-04-02T05:05:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[21528]: audit 2024-04-02T05:05:43.896132+0000 mgr.y (mgr.14538) 844 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.437119+0000 mgr.y (mgr.14538) 845 : cluster 0 pgmap v1465: 184 pgs: 184 active+clean; 582 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T05:05:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.996831+0000 osd.1 (osd.1) 525 : cluster 0 2.4 scrub starts 2024-04-02T05:05:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[21528]: cluster 2024-04-02T05:05:44.999737+0000 osd.1 (osd.1) 526 : cluster 0 2.4 scrub ok 2024-04-02T05:05:46.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:45 smithi069 bash[21528]: cluster 2024-04-02T05:05:45.202446+0000 mon.a (mon.0) 4141 : cluster 0 osdmap e937: 8 total, 8 up, 8 in 2024-04-02T05:05:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:45 smithi098 bash[23790]: audit 2024-04-02T05:05:43.896132+0000 mgr.y (mgr.14538) 844 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:45 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.437119+0000 mgr.y (mgr.14538) 845 : cluster 0 pgmap v1465: 184 pgs: 184 active+clean; 582 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-02T05:05:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:45 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.996831+0000 osd.1 (osd.1) 525 : cluster 0 2.4 scrub starts 2024-04-02T05:05:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:45 smithi098 bash[23790]: cluster 2024-04-02T05:05:44.999737+0000 osd.1 (osd.1) 526 : cluster 0 2.4 scrub ok 2024-04-02T05:05:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:45 smithi098 bash[23790]: cluster 2024-04-02T05:05:45.202446+0000 mon.a (mon.0) 4141 : cluster 0 osdmap e937: 8 total, 8 up, 8 in 2024-04-02T05:05:47.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[28904]: cluster 2024-04-02T05:05:45.425387+0000 osd.7 (osd.7) 315 : cluster 0 6.3 scrub starts 2024-04-02T05:05:47.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[28904]: cluster 2024-04-02T05:05:45.430353+0000 osd.7 (osd.7) 316 : cluster 0 6.3 scrub ok 2024-04-02T05:05:47.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[28904]: cluster 2024-04-02T05:05:45.924225+0000 osd.5 (osd.5) 335 : cluster 0 3.16 scrub starts 2024-04-02T05:05:47.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[28904]: cluster 2024-04-02T05:05:45.931214+0000 osd.5 (osd.5) 336 : cluster 0 3.16 scrub ok 2024-04-02T05:05:47.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.015553+0000 osd.3 (osd.3) 531 : cluster 0 4.14 scrub starts 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.018676+0000 osd.3 (osd.3) 532 : cluster 0 4.14 scrub ok 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[21528]: cluster 2024-04-02T05:05:45.425387+0000 osd.7 (osd.7) 315 : cluster 0 6.3 scrub starts 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[21528]: cluster 2024-04-02T05:05:45.430353+0000 osd.7 (osd.7) 316 : cluster 0 6.3 scrub ok 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[21528]: cluster 2024-04-02T05:05:45.924225+0000 osd.5 (osd.5) 335 : cluster 0 3.16 scrub starts 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[21528]: cluster 2024-04-02T05:05:45.931214+0000 osd.5 (osd.5) 336 : cluster 0 3.16 scrub ok 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.015553+0000 osd.3 (osd.3) 531 : cluster 0 4.14 scrub starts 2024-04-02T05:05:47.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:46 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.018676+0000 osd.3 (osd.3) 532 : cluster 0 4.14 scrub ok 2024-04-02T05:05:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:46 smithi098 bash[23790]: cluster 2024-04-02T05:05:45.425387+0000 osd.7 (osd.7) 315 : cluster 0 6.3 scrub starts 2024-04-02T05:05:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:46 smithi098 bash[23790]: cluster 2024-04-02T05:05:45.430353+0000 osd.7 (osd.7) 316 : cluster 0 6.3 scrub ok 2024-04-02T05:05:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:46 smithi098 bash[23790]: cluster 2024-04-02T05:05:45.924225+0000 osd.5 (osd.5) 335 : cluster 0 3.16 scrub starts 2024-04-02T05:05:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:46 smithi098 bash[23790]: cluster 2024-04-02T05:05:45.931214+0000 osd.5 (osd.5) 336 : cluster 0 3.16 scrub ok 2024-04-02T05:05:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:46 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.015553+0000 osd.3 (osd.3) 531 : cluster 0 4.14 scrub starts 2024-04-02T05:05:47.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:46 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.018676+0000 osd.3 (osd.3) 532 : cluster 0 4.14 scrub ok 2024-04-02T05:05:48.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.344710+0000 osd.2 (osd.2) 404 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:48.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.345200+0000 osd.2 (osd.2) 405 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:48.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.437879+0000 mgr.y (mgr.14538) 846 : cluster 0 pgmap v1467: 184 pgs: 184 active+clean; 582 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-04-02T05:05:48.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.469224+0000 osd.7 (osd.7) 317 : cluster 0 1.e scrub starts 2024-04-02T05:05:48.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.471655+0000 osd.7 (osd.7) 318 : cluster 0 1.e scrub ok 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.753005+0000 mon.a (mon.0) 4142 : cluster 0 osdmap e938: 8 total, 8 up, 8 in 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.927271+0000 osd.5 (osd.5) 337 : cluster 0 4.15 deep-scrub starts 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.927509+0000 osd.1 (osd.1) 527 : cluster 0 3.19 scrub starts 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.934471+0000 osd.1 (osd.1) 528 : cluster 0 3.19 scrub ok 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[28904]: cluster 2024-04-02T05:05:46.936734+0000 osd.5 (osd.5) 338 : cluster 0 4.15 deep-scrub ok 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.344710+0000 osd.2 (osd.2) 404 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.345200+0000 osd.2 (osd.2) 405 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.437879+0000 mgr.y (mgr.14538) 846 : cluster 0 pgmap v1467: 184 pgs: 184 active+clean; 582 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.469224+0000 osd.7 (osd.7) 317 : cluster 0 1.e scrub starts 2024-04-02T05:05:48.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.471655+0000 osd.7 (osd.7) 318 : cluster 0 1.e scrub ok 2024-04-02T05:05:48.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.753005+0000 mon.a (mon.0) 4142 : cluster 0 osdmap e938: 8 total, 8 up, 8 in 2024-04-02T05:05:48.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.927271+0000 osd.5 (osd.5) 337 : cluster 0 4.15 deep-scrub starts 2024-04-02T05:05:48.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.927509+0000 osd.1 (osd.1) 527 : cluster 0 3.19 scrub starts 2024-04-02T05:05:48.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.934471+0000 osd.1 (osd.1) 528 : cluster 0 3.19 scrub ok 2024-04-02T05:05:48.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:47 smithi069 bash[21528]: cluster 2024-04-02T05:05:46.936734+0000 osd.5 (osd.5) 338 : cluster 0 4.15 deep-scrub ok 2024-04-02T05:05:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.344710+0000 osd.2 (osd.2) 404 : cluster 0 purged_snaps scrub starts 2024-04-02T05:05:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.345200+0000 osd.2 (osd.2) 405 : cluster 0 purged_snaps scrub ok 2024-04-02T05:05:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.437879+0000 mgr.y (mgr.14538) 846 : cluster 0 pgmap v1467: 184 pgs: 184 active+clean; 582 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-04-02T05:05:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.469224+0000 osd.7 (osd.7) 317 : cluster 0 1.e scrub starts 2024-04-02T05:05:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.471655+0000 osd.7 (osd.7) 318 : cluster 0 1.e scrub ok 2024-04-02T05:05:48.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.753005+0000 mon.a (mon.0) 4142 : cluster 0 osdmap e938: 8 total, 8 up, 8 in 2024-04-02T05:05:48.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.927271+0000 osd.5 (osd.5) 337 : cluster 0 4.15 deep-scrub starts 2024-04-02T05:05:48.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.927509+0000 osd.1 (osd.1) 527 : cluster 0 3.19 scrub starts 2024-04-02T05:05:48.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.934471+0000 osd.1 (osd.1) 528 : cluster 0 3.19 scrub ok 2024-04-02T05:05:48.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:47 smithi098 bash[23790]: cluster 2024-04-02T05:05:46.936734+0000 osd.5 (osd.5) 338 : cluster 0 4.15 deep-scrub ok 2024-04-02T05:05:49.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:05:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:05:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[21528]: cluster 2024-04-02T05:05:47.512714+0000 osd.7 (osd.7) 319 : cluster 0 2.5 scrub starts 2024-04-02T05:05:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[21528]: cluster 2024-04-02T05:05:47.514440+0000 osd.7 (osd.7) 320 : cluster 0 2.5 scrub ok 2024-04-02T05:05:49.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[21528]: cluster 2024-04-02T05:05:47.921753+0000 osd.5 (osd.5) 339 : cluster 0 10.9 scrub starts 2024-04-02T05:05:49.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[21528]: cluster 2024-04-02T05:05:47.925323+0000 osd.5 (osd.5) 340 : cluster 0 10.9 scrub ok 2024-04-02T05:05:49.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[28904]: cluster 2024-04-02T05:05:47.512714+0000 osd.7 (osd.7) 319 : cluster 0 2.5 scrub starts 2024-04-02T05:05:49.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[28904]: cluster 2024-04-02T05:05:47.514440+0000 osd.7 (osd.7) 320 : cluster 0 2.5 scrub ok 2024-04-02T05:05:49.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[28904]: cluster 2024-04-02T05:05:47.921753+0000 osd.5 (osd.5) 339 : cluster 0 10.9 scrub starts 2024-04-02T05:05:49.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:48 smithi069 bash[28904]: cluster 2024-04-02T05:05:47.925323+0000 osd.5 (osd.5) 340 : cluster 0 10.9 scrub ok 2024-04-02T05:05:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:48 smithi098 bash[23790]: cluster 2024-04-02T05:05:47.512714+0000 osd.7 (osd.7) 319 : cluster 0 2.5 scrub starts 2024-04-02T05:05:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:48 smithi098 bash[23790]: cluster 2024-04-02T05:05:47.514440+0000 osd.7 (osd.7) 320 : cluster 0 2.5 scrub ok 2024-04-02T05:05:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:48 smithi098 bash[23790]: cluster 2024-04-02T05:05:47.921753+0000 osd.5 (osd.5) 339 : cluster 0 10.9 scrub starts 2024-04-02T05:05:49.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:48 smithi098 bash[23790]: cluster 2024-04-02T05:05:47.925323+0000 osd.5 (osd.5) 340 : cluster 0 10.9 scrub ok 2024-04-02T05:05:49.296 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:49.560 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:49.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:48.439777+0000 mgr.y (mgr.14538) 847 : cluster 0 pgmap v1469: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:05:49.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:48.941699+0000 osd.1 (osd.1) 529 : cluster 0 3.0 scrub starts 2024-04-02T05:05:49.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:48.947241+0000 osd.1 (osd.1) 530 : cluster 0 3.0 scrub ok 2024-04-02T05:05:49.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:48.963841+0000 osd.5 (osd.5) 341 : cluster 0 4.8 deep-scrub starts 2024-04-02T05:05:49.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:48.966231+0000 osd.5 (osd.5) 342 : cluster 0 4.8 deep-scrub ok 2024-04-02T05:05:49.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:49.236004+0000 osd.6 (osd.6) 457 : cluster 0 10.0 scrub starts 2024-04-02T05:05:49.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: cluster 2024-04-02T05:05:49.238398+0000 osd.6 (osd.6) 458 : cluster 0 10.0 scrub ok 2024-04-02T05:05:49.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:49 smithi098 bash[23790]: audit 2024-04-02T05:05:49.545767+0000 mon.b (mon.1) 346 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:48.439777+0000 mgr.y (mgr.14538) 847 : cluster 0 pgmap v1469: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:48.941699+0000 osd.1 (osd.1) 529 : cluster 0 3.0 scrub starts 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:48.947241+0000 osd.1 (osd.1) 530 : cluster 0 3.0 scrub ok 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:48.963841+0000 osd.5 (osd.5) 341 : cluster 0 4.8 deep-scrub starts 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:48.966231+0000 osd.5 (osd.5) 342 : cluster 0 4.8 deep-scrub ok 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:49.236004+0000 osd.6 (osd.6) 457 : cluster 0 10.0 scrub starts 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: cluster 2024-04-02T05:05:49.238398+0000 osd.6 (osd.6) 458 : cluster 0 10.0 scrub ok 2024-04-02T05:05:50.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[21528]: audit 2024-04-02T05:05:49.545767+0000 mon.b (mon.1) 346 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:48.439777+0000 mgr.y (mgr.14538) 847 : cluster 0 pgmap v1469: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:48.941699+0000 osd.1 (osd.1) 529 : cluster 0 3.0 scrub starts 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:48.947241+0000 osd.1 (osd.1) 530 : cluster 0 3.0 scrub ok 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:48.963841+0000 osd.5 (osd.5) 341 : cluster 0 4.8 deep-scrub starts 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:48.966231+0000 osd.5 (osd.5) 342 : cluster 0 4.8 deep-scrub ok 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:49.236004+0000 osd.6 (osd.6) 457 : cluster 0 10.0 scrub starts 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: cluster 2024-04-02T05:05:49.238398+0000 osd.6 (osd.6) 458 : cluster 0 10.0 scrub ok 2024-04-02T05:05:50.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:49 smithi069 bash[28904]: audit 2024-04-02T05:05:49.545767+0000 mon.b (mon.1) 346 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:05:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:51 smithi098 bash[23790]: cluster 2024-04-02T05:05:49.522318+0000 osd.7 (osd.7) 321 : cluster 0 2.8 scrub starts 2024-04-02T05:05:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:51 smithi098 bash[23790]: cluster 2024-04-02T05:05:49.525112+0000 osd.7 (osd.7) 322 : cluster 0 2.8 scrub ok 2024-04-02T05:05:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:51 smithi098 bash[23790]: cluster 2024-04-02T05:05:49.834158+0000 osd.4 (osd.4) 486 : cluster 0 4.13 scrub starts 2024-04-02T05:05:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:51 smithi098 bash[23790]: cluster 2024-04-02T05:05:49.836309+0000 osd.4 (osd.4) 487 : cluster 0 4.13 scrub ok 2024-04-02T05:05:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:51 smithi098 bash[23790]: cluster 2024-04-02T05:05:50.209146+0000 mon.a (mon.0) 4143 : cluster 0 osdmap e939: 8 total, 8 up, 8 in 2024-04-02T05:05:51.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:51 smithi098 bash[23790]: cluster 2024-04-02T05:05:50.440307+0000 mgr.y (mgr.14538) 848 : cluster 0 pgmap v1471: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:05:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[21528]: cluster 2024-04-02T05:05:49.522318+0000 osd.7 (osd.7) 321 : cluster 0 2.8 scrub starts 2024-04-02T05:05:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[21528]: cluster 2024-04-02T05:05:49.525112+0000 osd.7 (osd.7) 322 : cluster 0 2.8 scrub ok 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[21528]: cluster 2024-04-02T05:05:49.834158+0000 osd.4 (osd.4) 486 : cluster 0 4.13 scrub starts 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[21528]: cluster 2024-04-02T05:05:49.836309+0000 osd.4 (osd.4) 487 : cluster 0 4.13 scrub ok 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[21528]: cluster 2024-04-02T05:05:50.209146+0000 mon.a (mon.0) 4143 : cluster 0 osdmap e939: 8 total, 8 up, 8 in 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[21528]: cluster 2024-04-02T05:05:50.440307+0000 mgr.y (mgr.14538) 848 : cluster 0 pgmap v1471: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[28904]: cluster 2024-04-02T05:05:49.522318+0000 osd.7 (osd.7) 321 : cluster 0 2.8 scrub starts 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[28904]: cluster 2024-04-02T05:05:49.525112+0000 osd.7 (osd.7) 322 : cluster 0 2.8 scrub ok 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[28904]: cluster 2024-04-02T05:05:49.834158+0000 osd.4 (osd.4) 486 : cluster 0 4.13 scrub starts 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[28904]: cluster 2024-04-02T05:05:49.836309+0000 osd.4 (osd.4) 487 : cluster 0 4.13 scrub ok 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[28904]: cluster 2024-04-02T05:05:50.209146+0000 mon.a (mon.0) 4143 : cluster 0 osdmap e939: 8 total, 8 up, 8 in 2024-04-02T05:05:51.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:51 smithi069 bash[28904]: cluster 2024-04-02T05:05:50.440307+0000 mgr.y (mgr.14538) 848 : cluster 0 pgmap v1471: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/228602303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98101e20 msgr2=0x7efe9810dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/228602303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98101e20 0x7efe9810dfa0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7efe90009fc0 tx=0x7efe9000b330 comp rx=0 tx=0).stop 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/228602303 shutdown_connections 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/228602303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98101e20 0x7efe9810dfa0 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/228602303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98107350 0x7efe98107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/228602303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe98101460 0x7efe981018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:52.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/228602303 >> 172.21.15.69:0/228602303 conn(0x7efe98077ea0 msgr2=0x7efe980782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:52.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/228602303 shutdown_connections 2024-04-02T05:05:52.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/228602303 wait complete. 2024-04-02T05:05:52.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 librados: starting msgr at 2024-04-02T05:05:52.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 librados: starting objecter 2024-04-02T05:05:52.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 Processor -- start 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- start start 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 librados: setting wanted keys 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 librados: calling monclient init 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 0x7efe981980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe98101e20 0x7efe981985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98107350 0x7efe9819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efe98110a70 con 0x7efe98101460 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efe981108f0 con 0x7efe98101e20 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7efe98110bf0 con 0x7efe98107350 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 0x7efe981980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 0x7efe981980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44836/0 (socket says 172.21.15.69:44836) 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe977fe700 1 -- 172.21.15.69:0/74764648 learned_addr learned my addr 172.21.15.69:0/74764648 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98107350 0x7efe9819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:52.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe96ffd700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe98101e20 0x7efe981985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 -- 172.21.15.69:0/74764648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe98101e20 msgr2=0x7efe981985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe98101e20 0x7efe981985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 -- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 msgr2=0x7efe981980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 0x7efe981980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 -- 172.21.15.69:0/74764648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9819d060 con 0x7efe98107350 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe977fe700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 0x7efe981980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe97fff700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98107350 0x7efe9819c920 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7efe90007fd0 tx=0x7efe9000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 10 client.?.objecter ms_handle_connect 0x7efe98107350 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90021070 con 0x7efe98107350 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7efe9001d450 con 0x7efe98107350 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe9819d2f0 con 0x7efe98107350 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 10 client.43979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe900052a0 con 0x7efe98107350 2024-04-02T05:05:52.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe981a4bd0 con 0x7efe98107350 2024-04-02T05:05:52.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 librados: init done 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe981028d0 con 0x7efe98107350 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7efe90024070 con 0x7efe98107350 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efe8006f5e0 0x7efe80071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(939..939 src has 795..939) ==== 9044+0+0 (secure 0 0 0) 0x7efe9000f020 con 0x7efe98107350 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 10 client.43979.objecter ms_dispatch 0x7efe98001550 osd_map(939..939 src has 795..939) 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 3 client.43979.objecter handle_osd_map got epochs [939,939] > 0 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.250+0000 7efe94ff9700 3 client.43979.objecter handle_osd_map decoding full epoch 939 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.254+0000 7efe977fe700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efe8006f5e0 0x7efe80071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.254+0000 7efe94ff9700 20 client.43979.objecter dump_active .. 0 homeless 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.254+0000 7efe94ff9700 10 client.43979.objecter ms_handle_connect 0x7efe8006f5e0 2024-04-02T05:05:52.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.254+0000 7efe977fe700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efe8006f5e0 0x7efe80071aa0 secure :-1 s=READY pgs=4042 cs=0 l=1 rev1=1 crypto rx=0x7efe88009750 tx=0x7efe88009310 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:05:52.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.258+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efe9002b020 con 0x7efe98107350 2024-04-02T05:05:52.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:52 smithi098 bash[23790]: cluster 2024-04-02T05:05:51.003076+0000 osd.1 (osd.1) 531 : cluster 0 6.1 scrub starts 2024-04-02T05:05:52.473 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:52 smithi098 bash[23790]: cluster 2024-04-02T05:05:51.005438+0000 osd.1 (osd.1) 532 : cluster 0 6.1 scrub ok 2024-04-02T05:05:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:52.482+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7efe981993b0 con 0x7efe98107350 2024-04-02T05:05:52.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:52 smithi069 bash[21528]: cluster 2024-04-02T05:05:51.003076+0000 osd.1 (osd.1) 531 : cluster 0 6.1 scrub starts 2024-04-02T05:05:52.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:52 smithi069 bash[21528]: cluster 2024-04-02T05:05:51.005438+0000 osd.1 (osd.1) 532 : cluster 0 6.1 scrub ok 2024-04-02T05:05:52.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:52 smithi069 bash[28904]: cluster 2024-04-02T05:05:51.003076+0000 osd.1 (osd.1) 531 : cluster 0 6.1 scrub starts 2024-04-02T05:05:52.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:52 smithi069 bash[28904]: cluster 2024-04-02T05:05:51.005438+0000 osd.1 (osd.1) 532 : cluster 0 6.1 scrub ok 2024-04-02T05:05:52.565 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:05:52.565 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:05:52.565 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T05:05:52.565 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T05:05:52.565 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:05:53.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.230+0000 7efe94ff9700 1 -- 172.21.15.69:0/74764648 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v940) ==== 113+0+0 (secure 0 0 0) 0x7efe9005cac0 con 0x7efe98107350 2024-04-02T05:05:53.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.234+0000 7efe9dcaf700 10 librados: watch_flush enter 2024-04-02T05:05:53.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.234+0000 7efe9dcaf700 10 librados: watch_flush exit 2024-04-02T05:05:53.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.238+0000 7efe9dcaf700 20 client.43979.objecter shutdown clearing up homeless session... 2024-04-02T05:05:53.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.238+0000 7efe9dcaf700 10 client.43979.objecter successfully canceled tick 2024-04-02T05:05:53.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.238+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efe8006f5e0 msgr2=0x7efe80071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:53.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.238+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efe8006f5e0 0x7efe80071aa0 secure :-1 s=READY pgs=4042 cs=0 l=1 rev1=1 crypto rx=0x7efe88009750 tx=0x7efe88009310 comp rx=0 tx=0).stop 2024-04-02T05:05:53.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.238+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98107350 msgr2=0x7efe9819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:05:53.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.238+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98107350 0x7efe9819c920 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7efe90007fd0 tx=0x7efe9000ccb0 comp rx=0 tx=0).stop 2024-04-02T05:05:53.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 shutdown_connections 2024-04-02T05:05:53.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7efe8006f5e0 0x7efe80071aa0 unknown :-1 s=CLOSED pgs=4042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:53.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe98107350 0x7efe9819c920 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:53.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe98101460 0x7efe981980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:53.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.246+0000 7efe9dcaf700 1 --2- 172.21.15.69:0/74764648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe98101e20 0x7efe981985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:05:53.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.246+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 >> 172.21.15.69:0/74764648 conn(0x7efe98077ea0 msgr2=0x7efe9806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:05:53.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.250+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 shutdown_connections 2024-04-02T05:05:53.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.250+0000 7efe9dcaf700 1 -- 172.21.15.69:0/74764648 wait complete. 2024-04-02T05:05:53.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:05:53.250+0000 7efe9dcaf700 1 librados: shutdown 2024-04-02T05:05:53.262 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.4 to 1 (10000) 2024-04-02T05:05:53.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-02T05:05:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: cluster 2024-04-02T05:05:51.918159+0000 osd.5 (osd.5) 343 : cluster 0 10.11 scrub starts 2024-04-02T05:05:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: cluster 2024-04-02T05:05:51.921687+0000 osd.5 (osd.5) 344 : cluster 0 10.11 scrub ok 2024-04-02T05:05:53.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: cluster 2024-04-02T05:05:52.061731+0000 osd.3 (osd.3) 533 : cluster 0 6.1b deep-scrub starts 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: cluster 2024-04-02T05:05:52.064363+0000 osd.3 (osd.3) 534 : cluster 0 6.1b deep-scrub ok 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: cluster 2024-04-02T05:05:52.440971+0000 mgr.y (mgr.14538) 849 : cluster 0 pgmap v1472: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: audit 2024-04-02T05:05:52.487516+0000 mon.c (mon.2) 1174 : audit 1 from='client.? 172.21.15.69:0/74764648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[21528]: audit 2024-04-02T05:05:52.488195+0000 mon.a (mon.0) 4144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: cluster 2024-04-02T05:05:51.918159+0000 osd.5 (osd.5) 343 : cluster 0 10.11 scrub starts 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: cluster 2024-04-02T05:05:51.921687+0000 osd.5 (osd.5) 344 : cluster 0 10.11 scrub ok 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: cluster 2024-04-02T05:05:52.061731+0000 osd.3 (osd.3) 533 : cluster 0 6.1b deep-scrub starts 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: cluster 2024-04-02T05:05:52.064363+0000 osd.3 (osd.3) 534 : cluster 0 6.1b deep-scrub ok 2024-04-02T05:05:53.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: cluster 2024-04-02T05:05:52.440971+0000 mgr.y (mgr.14538) 849 : cluster 0 pgmap v1472: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-02T05:05:53.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: audit 2024-04-02T05:05:52.487516+0000 mon.c (mon.2) 1174 : audit 1 from='client.? 172.21.15.69:0/74764648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:05:53.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:53 smithi069 bash[28904]: audit 2024-04-02T05:05:52.488195+0000 mon.a (mon.0) 4144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: cluster 2024-04-02T05:05:51.918159+0000 osd.5 (osd.5) 343 : cluster 0 10.11 scrub starts 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: cluster 2024-04-02T05:05:51.921687+0000 osd.5 (osd.5) 344 : cluster 0 10.11 scrub ok 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: cluster 2024-04-02T05:05:52.061731+0000 osd.3 (osd.3) 533 : cluster 0 6.1b deep-scrub starts 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: cluster 2024-04-02T05:05:52.064363+0000 osd.3 (osd.3) 534 : cluster 0 6.1b deep-scrub ok 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: cluster 2024-04-02T05:05:52.440971+0000 mgr.y (mgr.14538) 849 : cluster 0 pgmap v1472: 184 pgs: 184 active+clean; 582 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: audit 2024-04-02T05:05:52.487516+0000 mon.c (mon.2) 1174 : audit 1 from='client.? 172.21.15.69:0/74764648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:05:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[23790]: audit 2024-04-02T05:05:52.488195+0000 mon.a (mon.0) 4144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:05:54.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 4 1 2024-04-02T05:05:54.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:05:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:05:54.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:54 smithi069 bash[21528]: audit 2024-04-02T05:05:53.222983+0000 mon.a (mon.0) 4145 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-02T05:05:54.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:54 smithi069 bash[21528]: cluster 2024-04-02T05:05:53.230584+0000 mon.a (mon.0) 4146 : cluster 0 osdmap e940: 8 total, 8 up, 8 in 2024-04-02T05:05:54.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:54 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.229402+0000 mon.a (mon.0) 4147 : cluster 0 osdmap e941: 8 total, 8 up, 8 in 2024-04-02T05:05:54.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:54 smithi069 bash[28904]: audit 2024-04-02T05:05:53.222983+0000 mon.a (mon.0) 4145 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-02T05:05:54.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:54 smithi069 bash[28904]: cluster 2024-04-02T05:05:53.230584+0000 mon.a (mon.0) 4146 : cluster 0 osdmap e940: 8 total, 8 up, 8 in 2024-04-02T05:05:54.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:54 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.229402+0000 mon.a (mon.0) 4147 : cluster 0 osdmap e941: 8 total, 8 up, 8 in 2024-04-02T05:05:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:54 smithi098 bash[23790]: audit 2024-04-02T05:05:53.222983+0000 mon.a (mon.0) 4145 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-02T05:05:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:54 smithi098 bash[23790]: cluster 2024-04-02T05:05:53.230584+0000 mon.a (mon.0) 4146 : cluster 0 osdmap e940: 8 total, 8 up, 8 in 2024-04-02T05:05:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:54 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.229402+0000 mon.a (mon.0) 4147 : cluster 0 osdmap e941: 8 total, 8 up, 8 in 2024-04-02T05:05:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:55 smithi098 bash[23790]: audit 2024-04-02T05:05:53.907219+0000 mgr.y (mgr.14538) 850 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:55 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.050330+0000 osd.3 (osd.3) 535 : cluster 0 4.17 scrub starts 2024-04-02T05:05:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:55 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.052751+0000 osd.3 (osd.3) 536 : cluster 0 4.17 scrub ok 2024-04-02T05:05:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:55 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.441712+0000 mgr.y (mgr.14538) 851 : cluster 0 pgmap v1475: 184 pgs: 11 peering, 1 unknown, 172 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:05:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:55 smithi098 bash[23790]: cluster 2024-04-02T05:05:55.236633+0000 mon.a (mon.0) 4148 : cluster 0 osdmap e942: 8 total, 8 up, 8 in 2024-04-02T05:05:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[21528]: audit 2024-04-02T05:05:53.907219+0000 mgr.y (mgr.14538) 850 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.050330+0000 osd.3 (osd.3) 535 : cluster 0 4.17 scrub starts 2024-04-02T05:05:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.052751+0000 osd.3 (osd.3) 536 : cluster 0 4.17 scrub ok 2024-04-02T05:05:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.441712+0000 mgr.y (mgr.14538) 851 : cluster 0 pgmap v1475: 184 pgs: 11 peering, 1 unknown, 172 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:05:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[21528]: cluster 2024-04-02T05:05:55.236633+0000 mon.a (mon.0) 4148 : cluster 0 osdmap e942: 8 total, 8 up, 8 in 2024-04-02T05:05:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[28904]: audit 2024-04-02T05:05:53.907219+0000 mgr.y (mgr.14538) 850 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:05:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.050330+0000 osd.3 (osd.3) 535 : cluster 0 4.17 scrub starts 2024-04-02T05:05:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.052751+0000 osd.3 (osd.3) 536 : cluster 0 4.17 scrub ok 2024-04-02T05:05:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.441712+0000 mgr.y (mgr.14538) 851 : cluster 0 pgmap v1475: 184 pgs: 11 peering, 1 unknown, 172 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:05:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:55 smithi069 bash[28904]: cluster 2024-04-02T05:05:55.236633+0000 mon.a (mon.0) 4148 : cluster 0 osdmap e942: 8 total, 8 up, 8 in 2024-04-02T05:05:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.593335+0000 osd.7 (osd.7) 323 : cluster 0 2.b deep-scrub starts 2024-04-02T05:05:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.596204+0000 osd.7 (osd.7) 324 : cluster 0 2.b deep-scrub ok 2024-04-02T05:05:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.810879+0000 osd.4 (osd.4) 488 : cluster 0 5.1 scrub starts 2024-04-02T05:05:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.813238+0000 osd.4 (osd.4) 489 : cluster 0 5.1 scrub ok 2024-04-02T05:05:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.981780+0000 osd.5 (osd.5) 345 : cluster 0 2.16 scrub starts 2024-04-02T05:05:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:54.983984+0000 osd.5 (osd.5) 346 : cluster 0 2.16 scrub ok 2024-04-02T05:05:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:55.097530+0000 osd.3 (osd.3) 537 : cluster 0 3.b scrub starts 2024-04-02T05:05:56.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:56 smithi098 bash[23790]: cluster 2024-04-02T05:05:55.101460+0000 osd.3 (osd.3) 538 : cluster 0 3.b scrub ok 2024-04-02T05:05:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.593335+0000 osd.7 (osd.7) 323 : cluster 0 2.b deep-scrub starts 2024-04-02T05:05:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.596204+0000 osd.7 (osd.7) 324 : cluster 0 2.b deep-scrub ok 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.810879+0000 osd.4 (osd.4) 488 : cluster 0 5.1 scrub starts 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.813238+0000 osd.4 (osd.4) 489 : cluster 0 5.1 scrub ok 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.981780+0000 osd.5 (osd.5) 345 : cluster 0 2.16 scrub starts 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:54.983984+0000 osd.5 (osd.5) 346 : cluster 0 2.16 scrub ok 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:55.097530+0000 osd.3 (osd.3) 537 : cluster 0 3.b scrub starts 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[21528]: cluster 2024-04-02T05:05:55.101460+0000 osd.3 (osd.3) 538 : cluster 0 3.b scrub ok 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.593335+0000 osd.7 (osd.7) 323 : cluster 0 2.b deep-scrub starts 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.596204+0000 osd.7 (osd.7) 324 : cluster 0 2.b deep-scrub ok 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.810879+0000 osd.4 (osd.4) 488 : cluster 0 5.1 scrub starts 2024-04-02T05:05:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.813238+0000 osd.4 (osd.4) 489 : cluster 0 5.1 scrub ok 2024-04-02T05:05:56.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.981780+0000 osd.5 (osd.5) 345 : cluster 0 2.16 scrub starts 2024-04-02T05:05:56.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:54.983984+0000 osd.5 (osd.5) 346 : cluster 0 2.16 scrub ok 2024-04-02T05:05:56.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:55.097530+0000 osd.3 (osd.3) 537 : cluster 0 3.b scrub starts 2024-04-02T05:05:56.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:56 smithi069 bash[28904]: cluster 2024-04-02T05:05:55.101460+0000 osd.3 (osd.3) 538 : cluster 0 3.b scrub ok 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.003200+0000 osd.5 (osd.5) 347 : cluster 0 2.18 scrub starts 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.005525+0000 osd.5 (osd.5) 348 : cluster 0 2.18 scrub ok 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.309555+0000 mon.a (mon.0) 4149 : cluster 0 osdmap e943: 8 total, 8 up, 8 in 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.442494+0000 mgr.y (mgr.14538) 852 : cluster 0 pgmap v1478: 184 pgs: 11 peering, 1 unknown, 172 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.549613+0000 osd.7 (osd.7) 325 : cluster 0 3.d scrub starts 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.555043+0000 osd.7 (osd.7) 326 : cluster 0 3.d scrub ok 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.929554+0000 osd.2 (osd.2) 406 : cluster 0 2.10 scrub starts 2024-04-02T05:05:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:57 smithi098 bash[23790]: cluster 2024-04-02T05:05:56.931565+0000 osd.2 (osd.2) 407 : cluster 0 2.10 scrub ok 2024-04-02T05:05:57.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.003200+0000 osd.5 (osd.5) 347 : cluster 0 2.18 scrub starts 2024-04-02T05:05:57.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.005525+0000 osd.5 (osd.5) 348 : cluster 0 2.18 scrub ok 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.309555+0000 mon.a (mon.0) 4149 : cluster 0 osdmap e943: 8 total, 8 up, 8 in 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.442494+0000 mgr.y (mgr.14538) 852 : cluster 0 pgmap v1478: 184 pgs: 11 peering, 1 unknown, 172 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.549613+0000 osd.7 (osd.7) 325 : cluster 0 3.d scrub starts 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.555043+0000 osd.7 (osd.7) 326 : cluster 0 3.d scrub ok 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.929554+0000 osd.2 (osd.2) 406 : cluster 0 2.10 scrub starts 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[21528]: cluster 2024-04-02T05:05:56.931565+0000 osd.2 (osd.2) 407 : cluster 0 2.10 scrub ok 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.003200+0000 osd.5 (osd.5) 347 : cluster 0 2.18 scrub starts 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.005525+0000 osd.5 (osd.5) 348 : cluster 0 2.18 scrub ok 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.309555+0000 mon.a (mon.0) 4149 : cluster 0 osdmap e943: 8 total, 8 up, 8 in 2024-04-02T05:05:57.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.442494+0000 mgr.y (mgr.14538) 852 : cluster 0 pgmap v1478: 184 pgs: 11 peering, 1 unknown, 172 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:05:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.549613+0000 osd.7 (osd.7) 325 : cluster 0 3.d scrub starts 2024-04-02T05:05:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.555043+0000 osd.7 (osd.7) 326 : cluster 0 3.d scrub ok 2024-04-02T05:05:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.929554+0000 osd.2 (osd.2) 406 : cluster 0 2.10 scrub starts 2024-04-02T05:05:57.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:57 smithi069 bash[28904]: cluster 2024-04-02T05:05:56.931565+0000 osd.2 (osd.2) 407 : cluster 0 2.10 scrub ok 2024-04-02T05:05:58.181 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:05:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.070473+0000 osd.3 (osd.3) 539 : cluster 0 4.1d deep-scrub starts 2024-04-02T05:05:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.072987+0000 osd.3 (osd.3) 540 : cluster 0 4.1d deep-scrub ok 2024-04-02T05:05:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.297543+0000 osd.6 (osd.6) 459 : cluster 0 4.5 scrub starts 2024-04-02T05:05:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.299847+0000 osd.6 (osd.6) 460 : cluster 0 4.5 scrub ok 2024-04-02T05:05:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.321816+0000 mon.a (mon.0) 4150 : cluster 0 osdmap e944: 8 total, 8 up, 8 in 2024-04-02T05:05:58.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.507792+0000 osd.7 (osd.7) 327 : cluster 0 1.d scrub starts 2024-04-02T05:05:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.510248+0000 osd.7 (osd.7) 328 : cluster 0 1.d scrub ok 2024-04-02T05:05:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.718009+0000 osd.4 (osd.4) 490 : cluster 0 1.e scrub starts 2024-04-02T05:05:58.475 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:58 smithi098 bash[23790]: cluster 2024-04-02T05:05:57.720472+0000 osd.4 (osd.4) 491 : cluster 0 1.e scrub ok 2024-04-02T05:05:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.070473+0000 osd.3 (osd.3) 539 : cluster 0 4.1d deep-scrub starts 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.072987+0000 osd.3 (osd.3) 540 : cluster 0 4.1d deep-scrub ok 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.297543+0000 osd.6 (osd.6) 459 : cluster 0 4.5 scrub starts 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.299847+0000 osd.6 (osd.6) 460 : cluster 0 4.5 scrub ok 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.321816+0000 mon.a (mon.0) 4150 : cluster 0 osdmap e944: 8 total, 8 up, 8 in 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.507792+0000 osd.7 (osd.7) 327 : cluster 0 1.d scrub starts 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.510248+0000 osd.7 (osd.7) 328 : cluster 0 1.d scrub ok 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.718009+0000 osd.4 (osd.4) 490 : cluster 0 1.e scrub starts 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21528]: cluster 2024-04-02T05:05:57.720472+0000 osd.4 (osd.4) 491 : cluster 0 1.e scrub ok 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.070473+0000 osd.3 (osd.3) 539 : cluster 0 4.1d deep-scrub starts 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.072987+0000 osd.3 (osd.3) 540 : cluster 0 4.1d deep-scrub ok 2024-04-02T05:05:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.297543+0000 osd.6 (osd.6) 459 : cluster 0 4.5 scrub starts 2024-04-02T05:05:58.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.299847+0000 osd.6 (osd.6) 460 : cluster 0 4.5 scrub ok 2024-04-02T05:05:58.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.321816+0000 mon.a (mon.0) 4150 : cluster 0 osdmap e944: 8 total, 8 up, 8 in 2024-04-02T05:05:58.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.507792+0000 osd.7 (osd.7) 327 : cluster 0 1.d scrub starts 2024-04-02T05:05:58.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.510248+0000 osd.7 (osd.7) 328 : cluster 0 1.d scrub ok 2024-04-02T05:05:58.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.718009+0000 osd.4 (osd.4) 490 : cluster 0 1.e scrub starts 2024-04-02T05:05:58.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[28904]: cluster 2024-04-02T05:05:57.720472+0000 osd.4 (osd.4) 491 : cluster 0 1.e scrub ok 2024-04-02T05:05:58.894 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:05:59.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:05:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:05:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:05:59.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:59 smithi098 bash[23790]: cluster 2024-04-02T05:05:58.444277+0000 mgr.y (mgr.14538) 853 : cluster 0 pgmap v1480: 184 pgs: 4 peering, 1 unknown, 179 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 78 B/s, 7 objects/s recovering 2024-04-02T05:05:59.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:59 smithi098 bash[23790]: cluster 2024-04-02T05:05:58.477953+0000 osd.7 (osd.7) 329 : cluster 0 1.2 scrub starts 2024-04-02T05:05:59.637 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:05:59 smithi098 bash[23790]: cluster 2024-04-02T05:05:58.480452+0000 osd.7 (osd.7) 330 : cluster 0 1.2 scrub ok 2024-04-02T05:05:59.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:59 smithi069 bash[28904]: cluster 2024-04-02T05:05:58.444277+0000 mgr.y (mgr.14538) 853 : cluster 0 pgmap v1480: 184 pgs: 4 peering, 1 unknown, 179 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 78 B/s, 7 objects/s recovering 2024-04-02T05:05:59.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:59 smithi069 bash[28904]: cluster 2024-04-02T05:05:58.477953+0000 osd.7 (osd.7) 329 : cluster 0 1.2 scrub starts 2024-04-02T05:05:59.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:05:59 smithi069 bash[28904]: cluster 2024-04-02T05:05:58.480452+0000 osd.7 (osd.7) 330 : cluster 0 1.2 scrub ok 2024-04-02T05:05:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:59 smithi069 bash[21528]: cluster 2024-04-02T05:05:58.444277+0000 mgr.y (mgr.14538) 853 : cluster 0 pgmap v1480: 184 pgs: 4 peering, 1 unknown, 179 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 78 B/s, 7 objects/s recovering 2024-04-02T05:05:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:59 smithi069 bash[21528]: cluster 2024-04-02T05:05:58.477953+0000 osd.7 (osd.7) 329 : cluster 0 1.2 scrub starts 2024-04-02T05:05:59.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:05:59 smithi069 bash[21528]: cluster 2024-04-02T05:05:58.480452+0000 osd.7 (osd.7) 330 : cluster 0 1.2 scrub ok 2024-04-02T05:06:00.413 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:00 smithi069 bash[28904]: cluster 2024-04-02T05:05:59.667026+0000 osd.4 (osd.4) 492 : cluster 0 3.14 scrub starts 2024-04-02T05:06:00.413 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:00 smithi069 bash[28904]: cluster 2024-04-02T05:05:59.669069+0000 osd.4 (osd.4) 493 : cluster 0 3.14 scrub ok 2024-04-02T05:06:00.413 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:00 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.208036+0000 mon.a (mon.0) 4151 : cluster 0 osdmap e945: 8 total, 8 up, 8 in 2024-04-02T05:06:00.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:00 smithi069 bash[21528]: cluster 2024-04-02T05:05:59.667026+0000 osd.4 (osd.4) 492 : cluster 0 3.14 scrub starts 2024-04-02T05:06:00.414 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:00 smithi069 bash[21528]: cluster 2024-04-02T05:05:59.669069+0000 osd.4 (osd.4) 493 : cluster 0 3.14 scrub ok 2024-04-02T05:06:00.414 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:00 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.208036+0000 mon.a (mon.0) 4151 : cluster 0 osdmap e945: 8 total, 8 up, 8 in 2024-04-02T05:06:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:00 smithi098 bash[23790]: cluster 2024-04-02T05:05:59.667026+0000 osd.4 (osd.4) 492 : cluster 0 3.14 scrub starts 2024-04-02T05:06:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:00 smithi098 bash[23790]: cluster 2024-04-02T05:05:59.669069+0000 osd.4 (osd.4) 493 : cluster 0 3.14 scrub ok 2024-04-02T05:06:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:00 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.208036+0000 mon.a (mon.0) 4151 : cluster 0 osdmap e945: 8 total, 8 up, 8 in 2024-04-02T05:06:01.509 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:06:01.509 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.4,QREQUEST,e:703,from:3,reservation_nonce:3)", 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:01.795935+0000", 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 359.71103696900002, 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063419150000000001, 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795935+0000", 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795935+0000", 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795938+0000", 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3700000000000002e-06 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795950+0000", 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1823e-05 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795981+0000", 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1372999999999999e-05 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.795996+0000", 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5255999999999999e-05 2024-04-02T05:06:01.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.796142+0000", 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014554499999999999 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.796153+0000", 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1192999999999999e-05 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.802074+0000", 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059212680000000004 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.802120+0000", 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6189000000000001e-05 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:01.802277+0000", 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015689799999999999 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:592 328.6 328:60000000::::head [pgnls start_epoch 742] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e742)", 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:52.156156+0000", 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 309.35081611599998, 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0087676249999999994, 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 592 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156156+0000", 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156156+0000", 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156160+0000", 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156174+0000", 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4503e-05 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156209+0000", 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4996999999999997e-05 2024-04-02T05:06:01.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156228+0000", 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9035999999999999e-05 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156367+0000", 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013865500000000001 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.156381+0000", 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3857e-05 2024-04-02T05:06:01.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.164717+0000", 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0083357199999999996 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.164769+0000", 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1996000000000002e-05 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:52.164924+0000", 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015539800000000001 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDECSubOpWrite(323.0s2 745/698 ECSubWrite(tid=1574, reqid=client.32028.0:646, at_version=0'0, trim_to=0'0, roll_forward_to=745'1))", 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:55.105627+0000", 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 306.40134571599998, 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013194453, 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105627+0000", 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105627+0000", 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105628+0000", 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1399999999999995e-07 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105641+0000", 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3354e-05 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105655+0000", 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3621999999999999e-05 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105667+0000", 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2690999999999999e-05 2024-04-02T05:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105729+0000", 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1605000000000004e-05 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.105738+0000", 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6430000000000001e-06 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.118765+0000", 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013027828999999999 2024-04-02T05:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:55.118821+0000", 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5794999999999997e-05 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1101 332.5 332:a7884da2:test-rados-api-smithi069-72349-129::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e774)", 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:24.087247+0000", 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 277.41972538800002, 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074450489999999996, 2024-04-02T05:06:01.524 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1101 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087247+0000", 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087247+0000", 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087250+0000", 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2629999999999999e-06 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087267+0000", 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7056999999999999e-05 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087294+0000", 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6957999999999998e-05 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087313+0000", 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8916999999999999e-05 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087424+0000", 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011061399999999999 2024-04-02T05:06:01.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.087435+0000", 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0937e-05 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.092137+0000", 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047020370000000001 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.092381+0000", 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024452300000000002 2024-04-02T05:06:01.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.093460+0000", 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010781829999999999 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.093563+0000", 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010354299999999999 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.093716+0000", 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015263000000000001 2024-04-02T05:06:01.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094200+0000", 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048412800000000001 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094575+0000", 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037488099999999998 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094612+0000", 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7339000000000003e-05 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094660+0000", 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8415000000000003e-05 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:24.094692+0000", 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1624000000000001e-05 2024-04-02T05:06:01.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1153 332.5 332:a7884da2:test-rados-api-smithi069-72349-129::foo:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e776)", 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:29.134681+0000", 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 272.37229136799999, 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067511580000000002, 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1153 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134681+0000", 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134681+0000", 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134684+0000", 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.977e-06 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134696+0000", 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1946e-05 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134728+0000", 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1619999999999999e-05 2024-04-02T05:06:01.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134745+0000", 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7013000000000001e-05 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134910+0000", 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016564499999999999 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.134930+0000", 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9338999999999999e-05 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.138655+0000", 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037251300000000001 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.138941+0000", 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028616699999999999 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.139334+0000", 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039350099999999999 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.140982+0000", 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016480589999999999 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.141046+0000", 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3229000000000001e-05 2024-04-02T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.141148+0000", 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102166 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.141319+0000", 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017131399999999999 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:29.141432+0000", 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113052 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1235 332.1c 332:38000000::::head [pgnls start_epoch 778] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:30.753019+0000", 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 270.75395357500003, 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0090052929999999993, 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1235 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753019+0000", 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753019+0000", 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753022+0000", 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2100000000000002e-06 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753039+0000", 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6546999999999999e-05 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753078+0000", 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9486000000000001e-05 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753098+0000", 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0026000000000001e-05 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753290+0000", 2024-04-02T05:06:01.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000192141 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.753301+0000", 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.081e-05 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.761690+0000", 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0083890869999999999 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.761794+0000", 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104205 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.762024+0000", 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000229781 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1682 335.d 335:b062609c:test-rados-api-smithi069-72349-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:06:01.544 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.101143+0000", 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.405829461, 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020133867999999999, 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1682 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101143+0000", 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101143+0000", 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101145+0000", 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4420000000000001e-06 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101156+0000", 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0512e-05 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101177+0000", 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0959e-05 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101189+0000", 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2140000000000001e-05 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101253+0000", 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4369000000000007e-05 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.101263+0000", 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5289999999999997e-06 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.105826+0000", 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004563157 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.106171+0000", 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034445299999999999 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.118797+0000", 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012626717000000001 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.118984+0000", 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018699600000000001 2024-04-02T05:06:01.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.119243+0000", 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025829100000000001 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.119951+0000", 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00070843800000000004 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:01.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.120831+0000", 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00087984300000000005 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121191+0000", 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035997600000000002 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121240+0000", 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9642999999999999e-05 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121277+0000", 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1683 335.d 335:b062609c:test-rados-api-smithi069-72349-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.121715+0000", 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.38525736599999, 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005608717, 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1683 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121715+0000", 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121715+0000", 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121717+0000", 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0729999999999999e-06 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121730+0000", 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2459999999999999e-05 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121752+0000", 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2087e-05 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121765+0000", 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.36e-05 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121840+0000", 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4951999999999997e-05 2024-04-02T05:06:01.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121989+0000", 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000148558 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.126758+0000", 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047695740000000004 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.127084+0000", 2024-04-02T05:06:01.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032565600000000001 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.127324+0000", 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000239757 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1684 335.d 335:b062609c:test-rados-api-smithi069-72349-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.121794+0000", 2024-04-02T05:06:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.38517891999999, 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00575378, 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1684 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121794+0000", 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121794+0000", 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121795+0000", 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121805+0000", 2024-04-02T05:06:01.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0648999999999999e-05 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121816+0000", 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1080000000000001e-05 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.121825+0000", 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1339999999999993e-06 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.122237+0000", 2024-04-02T05:06:01.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000411801 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.122395+0000", 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000158092 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.127371+0000", 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049754999999999999 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.127459+0000", 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8503000000000003e-05 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.127547+0000", 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8041000000000006e-05 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1690 335.d 335:b062609c:test-rados-api-smithi069-72349-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.138237+0000", 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.368735531, 2024-04-02T05:06:01.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093971279999999994, 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1690 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138237+0000", 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138237+0000", 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138237+0000", 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138240+0000", 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6910000000000002e-06 2024-04-02T05:06:01.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138247+0000", 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5749999999999997e-06 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138256+0000", 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1109999999999997e-06 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138625+0000", 2024-04-02T05:06:01.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036888999999999997 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.138721+0000", 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6655000000000002e-05 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.147207+0000", 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084862129999999994 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.147447+0000", 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023901400000000001 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.147634+0000", 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000187531 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1692 335.d 335:b062609c:test-rados-api-smithi069-72349-135::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.150749+0000", 2024-04-02T05:06:01.567 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.35622332899999, 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064221260000000002, 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1692 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150749+0000", 2024-04-02T05:06:01.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150749+0000", 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150751+0000", 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0999999999999998e-06 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150763+0000", 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1457000000000001e-05 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150794+0000", 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1406999999999999e-05 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150812+0000", 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7521999999999999e-05 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150897+0000", 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5162999999999998e-05 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.151024+0000", 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012759199999999999 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.156983+0000", 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059584199999999999 2024-04-02T05:06:01.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.157080+0000", 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6915000000000003e-05 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.157171+0000", 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1550000000000003e-05 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1693 335.d 335:b062609c:test-rados-api-smithi069-72349-135::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e805)", 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:57.150842+0000", 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.35613030299999, 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065056269999999999, 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1693 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.573 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150842+0000", 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150842+0000", 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150843+0000", 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6300000000000004e-07 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150848+0000", 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6469999999999997e-06 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150859+0000", 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1813e-05 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.150871+0000", 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1399999999999999e-05 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.151232+0000", 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036084499999999999 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.151412+0000", 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000180118 2024-04-02T05:06:01.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.157211+0000", 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057991220000000003 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.157285+0000", 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3795999999999996e-05 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:57.157348+0000", 2024-04-02T05:06:01.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3022999999999996e-05 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1798 336.8 336:10000000:%01:::head [pgnls start_epoch 813] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e813)", 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:02:05.170721+0000", 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 236.336251003, 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064658729999999996, 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.578 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1798 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170721+0000", 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170721+0000", 2024-04-02T05:06:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170724+0000", 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.616e-06 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170738+0000", 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4323e-05 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170771+0000", 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2716000000000003e-05 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170788+0000", 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7260000000000001e-05 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170870+0000", 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1934000000000005e-05 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.170875+0000", 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7559999999999999e-06 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.176828+0000", 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005952822 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.176929+0000", 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010108200000000001 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.177187+0000", 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025836400000000002 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(4.e,QREQUEST,e:816,from:4,reservation_nonce:5)", 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:02:08.297152+0000", 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 233.20982014099999, 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061150129999999999, 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.583 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297152+0000", 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297152+0000", 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297154+0000", 2024-04-02T05:06:01.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5910000000000001e-06 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297162+0000", 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9869999999999999e-06 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297180+0000", 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7706e-05 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297191+0000", 2024-04-02T05:06:01.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1571e-05 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297264+0000", 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3164000000000006e-05 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.297271+0000", 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.764e-06 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.303177+0000", 2024-04-02T05:06:01.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059063910000000004 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.303203+0000", 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5924000000000001e-05 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:08.303267+0000", 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3915e-05 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:2670 340.18 340:18000000:%01:::head [pgnls start_epoch 864] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e864)", 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:03:10.451791+0000", 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 171.05518099599999, 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012656723, 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2670 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.588 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.451791+0000", 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.451791+0000", 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.451795+0000", 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1439999999999999e-06 2024-04-02T05:06:01.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.451812+0000", 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7036000000000001e-05 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.453400+0000", 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001588232 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.453428+0000", 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8286e-05 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.464110+0000", 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010681741999999999 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.464209+0000", 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9355e-05 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:10.464448+0000", 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023892799999999999 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(323.7s1,QREQUEST,e:871,from:7(0),reservation_nonce:3)", 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:03:17.267154+0000", 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 164.239818141, 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010342682000000001, 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267154+0000", 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267154+0000", 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267157+0000", 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5359999999999999e-06 2024-04-02T05:06:01.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267168+0000", 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1204e-05 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267200+0000", 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1888999999999999e-05 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267217+0000", 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7220000000000001e-05 2024-04-02T05:06:01.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267350+0000", 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000132612 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.267362+0000", 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2004999999999999e-05 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.277393+0000", 2024-04-02T05:06:01.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010031727000000001 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.277424+0000", 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0049999999999999e-05 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:17.277497+0000", 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3438999999999999e-05 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(4.10,QREQUEST,e:872,from:3,reservation_nonce:5)", 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:03:18.450146+0000", 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 163.05682642799999, 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064911709999999996, 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450146+0000", 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450146+0000", 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450148+0000", 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.84e-06 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450158+0000", 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9599999999999995e-06 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450186+0000", 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7917000000000001e-05 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450202+0000", 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6606e-05 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450349+0000", 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000146899 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.450361+0000", 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1875999999999999e-05 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.456301+0000", 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059396500000000003 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.456435+0000", 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013393500000000001 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:18.456637+0000", 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020248799999999999 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4629 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94452570225408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e900)", 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:10.247997+0000", 2024-04-02T05:06:01.601 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.25897575, 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.023812964, 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4629 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.247997+0000", 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.247997+0000", 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248000+0000", 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.88e-06 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248024+0000", 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3961e-05 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248051+0000", 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7126000000000001e-05 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248066+0000", 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5119e-05 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248208+0000", 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000142316 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248236+0000", 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8337999999999999e-05 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.260821+0000", 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.012584573 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for degraded object", 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262927+0000", 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002105764 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.271646+0000", 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0087195109999999992 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.271737+0000", 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1036e-05 2024-04-02T05:06:01.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.271810+0000", 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2340000000000002e-05 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14415.0:4634 4.1f 4:f95f44c2:::notify.0:head [watch reconnect cookie 94452570225408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:10.248638+0000", 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.258334643, 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0246447940000001, 2024-04-02T05:06:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14415", 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/1159886337", 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4634 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248638+0000", 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248638+0000", 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248639+0000", 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3300000000000001e-07 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248644+0000", 2024-04-02T05:06:01.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0220000000000004e-06 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248655+0000", 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1816e-05 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248662+0000", 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9310000000000002e-06 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248724+0000", 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2006000000000002e-05 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:10.248734+0000", 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5009999999999993e-06 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.262998+0000", 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.01426422 2024-04-02T05:06:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for degraded object", 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.263036+0000", 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8160000000000001e-05 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.271873+0000", 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088363569999999995 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.271933+0000", 2024-04-02T05:06:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9932999999999997e-05 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:11.273283+0000", 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001350015 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.24446.0:45 5.2 5:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e913)", 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:05:52.043729+0000", 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.4632435879999992, 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013496789999999999, 2024-04-02T05:06:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.24446", 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2252228867", 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 45 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.043729+0000", 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.043729+0000", 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.043732+0000", 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2930000000000001e-06 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.043750+0000", 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.804e-05 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.043788+0000", 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.782e-05 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.043806+0000", 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8379000000000001e-05 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.044794+0000", 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0009880100000000001 2024-04-02T05:06:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:05:52.045079+0000", 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028413700000000001 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:01.617 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:06:01.702 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:01 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.229840+0000 osd.6 (osd.6) 461 : cluster 0 10.16 scrub starts 2024-04-02T05:06:01.702 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:01 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.233602+0000 osd.6 (osd.6) 462 : cluster 0 10.16 scrub ok 2024-04-02T05:06:01.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:01 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.445025+0000 mgr.y (mgr.14538) 854 : cluster 0 pgmap v1482: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 589 B/s rd, 0 op/s; 84 B/s, 7 objects/s recovering 2024-04-02T05:06:01.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:01 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.498125+0000 osd.7 (osd.7) 331 : cluster 0 1.0 scrub starts 2024-04-02T05:06:01.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:01 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.500628+0000 osd.7 (osd.7) 332 : cluster 0 1.0 scrub ok 2024-04-02T05:06:01.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.229840+0000 osd.6 (osd.6) 461 : cluster 0 10.16 scrub starts 2024-04-02T05:06:01.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.233602+0000 osd.6 (osd.6) 462 : cluster 0 10.16 scrub ok 2024-04-02T05:06:01.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.445025+0000 mgr.y (mgr.14538) 854 : cluster 0 pgmap v1482: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 589 B/s rd, 0 op/s; 84 B/s, 7 objects/s recovering 2024-04-02T05:06:01.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.498125+0000 osd.7 (osd.7) 331 : cluster 0 1.0 scrub starts 2024-04-02T05:06:01.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.500628+0000 osd.7 (osd.7) 332 : cluster 0 1.0 scrub ok 2024-04-02T05:06:01.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.229840+0000 osd.6 (osd.6) 461 : cluster 0 10.16 scrub starts 2024-04-02T05:06:01.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.233602+0000 osd.6 (osd.6) 462 : cluster 0 10.16 scrub ok 2024-04-02T05:06:01.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.445025+0000 mgr.y (mgr.14538) 854 : cluster 0 pgmap v1482: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 589 B/s rd, 0 op/s; 84 B/s, 7 objects/s recovering 2024-04-02T05:06:01.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.498125+0000 osd.7 (osd.7) 331 : cluster 0 1.0 scrub starts 2024-04-02T05:06:01.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:01 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.500628+0000 osd.7 (osd.7) 332 : cluster 0 1.0 scrub ok 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- 172.21.15.69:0/2133424471 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 msgr2=0x7f37d0101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/2133424471 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0101800 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0099c0 tx=0x7f37cc009cd0 comp rx=0 tx=0).stop 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- 172.21.15.69:0/2133424471 shutdown_connections 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/2133424471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0106500 0x7f37d010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/2133424471 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0101800 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/2133424471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0101dd0 0x7f37d0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- 172.21.15.69:0/2133424471 >> 172.21.15.69:0/2133424471 conn(0x7f37d0078240 msgr2=0x7f37d0078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:01.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- 172.21.15.69:0/2133424471 shutdown_connections 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- 172.21.15.69:0/2133424471 wait complete. 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 librados: starting msgr at 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 librados: starting objecter 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 Processor -- start 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- start start 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 librados: setting wanted keys 2024-04-02T05:06:01.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 librados: calling monclient init 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0101dd0 0x7f37d0103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0106500 0x7f37d0103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01109d0 con 0x7f37d0106500 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0110850 con 0x7f37d0101dd0 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d7f1b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f37d0110b50 con 0x7f37d0101420 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0102ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34606/0 (socket says 172.21.15.69:34606) 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 -- 172.21.15.69:0/1397324598 learned_addr learned my addr 172.21.15.69:0/1397324598 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d64b8700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0106500 0x7f37d0103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d54b6700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0101dd0 0x7f37d0103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:01.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 -- 172.21.15.69:0/1397324598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0101dd0 msgr2=0x7f37d0103400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0101dd0 0x7f37d0103400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 -- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0106500 msgr2=0x7f37d0103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0106500 0x7f37d0103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 -- 172.21.15.69:0/1397324598 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d01a6640 con 0x7f37d0101420 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d64b8700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0106500 0x7f37d0103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37d5cb7700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0102ec0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005910 tx=0x7f37cc00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37c6ffd700 10 client.?.objecter ms_handle_connect 0x7f37d0101420 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37c6ffd700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.978+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc027070 con 0x7f37d0101420 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f37cc00fc90 con 0x7f37d0101420 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc022c90 con 0x7f37d0101420 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d01a68d0 con 0x7f37d0101420 2024-04-02T05:06:01.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d7f1b700 10 client.43985.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:01.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d01a6d20 con 0x7f37d0101420 2024-04-02T05:06:01.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d7f1b700 1 librados: init done 2024-04-02T05:06:01.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d004ff00 con 0x7f37d0101420 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f37cc02a070 con 0x7f37d0101420 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f37bc06f5e0 0x7f37bc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(945..945 src has 857..945) ==== 9015+0+0 (secure 0 0 0) 0x7f37cc08f400 con 0x7f37d0101420 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 10 client.43985.objecter ms_dispatch 0x7f37d0001550 osd_map(945..945 src has 857..945) 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 3 client.43985.objecter handle_osd_map got epochs [945,945] > 0 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 3 client.43985.objecter handle_osd_map decoding full epoch 945 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d54b6700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f37bc06f5e0 0x7f37bc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 20 client.43985.objecter dump_active .. 0 homeless 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37d54b6700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f37bc06f5e0 0x7f37bc071aa0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009680 tx=0x7f37c0009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:01.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.982+0000 7f37c6ffd700 10 client.43985.objecter ms_handle_connect 0x7f37bc06f5e0 2024-04-02T05:06:01.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:01.986+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f37cc013070 con 0x7f37d0101420 2024-04-02T05:06:02.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.210+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) -- 0x7f37d01a6fe0 con 0x7f37d0101420 2024-04-02T05:06:02.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-02T05:06:02.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.346+0000 7f37c6ffd700 1 -- 172.21.15.69:0/1397324598 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v946) ==== 133+0+0 (secure 0 0 0) 0x7f37cc05c3d0 con 0x7f37d0101420 2024-04-02T05:06:02.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 10 librados: watch_flush enter 2024-04-02T05:06:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 10 librados: watch_flush exit 2024-04-02T05:06:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 20 client.43985.objecter shutdown clearing up homeless session... 2024-04-02T05:06:02.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 10 client.43985.objecter successfully canceled tick 2024-04-02T05:06:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f37bc06f5e0 msgr2=0x7f37bc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f37bc06f5e0 0x7f37bc071aa0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009680 tx=0x7f37c0009260 comp rx=0 tx=0).stop 2024-04-02T05:06:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 msgr2=0x7f37d0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:02.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0102ec0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005910 tx=0x7f37cc00fa00 comp rx=0 tx=0).stop 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 shutdown_connections 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f37bc06f5e0 0x7f37bc071aa0 unknown :-1 s=CLOSED pgs=4043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37d0106500 0x7f37d0103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37d0101420 0x7f37d0102ec0 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 --2- 172.21.15.69:0/1397324598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0101dd0 0x7f37d0103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 >> 172.21.15.69:0/1397324598 conn(0x7f37d0078240 msgr2=0x7f37d006a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.350+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 shutdown_connections 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.354+0000 7f37d7f1b700 1 -- 172.21.15.69:0/1397324598 wait complete. 2024-04-02T05:06:02.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:02.354+0000 7f37d7f1b700 1 librados: shutdown 2024-04-02T05:06:02.358 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.898189+0000 osd.1 (osd.1) 533 : cluster 0 4.12 scrub starts 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[28904]: cluster 2024-04-02T05:06:00.900695+0000 osd.1 (osd.1) 534 : cluster 0 4.12 scrub ok 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[28904]: cluster 2024-04-02T05:06:01.954166+0000 osd.7 (osd.7) 333 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[28904]: cluster 2024-04-02T05:06:01.954598+0000 osd.7 (osd.7) 334 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[28904]: audit 2024-04-02T05:06:02.213336+0000 mon.c (mon.2) 1175 : audit 1 from='client.? 172.21.15.69:0/1397324598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[28904]: audit 2024-04-02T05:06:02.213984+0000 mon.a (mon.0) 4152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:06:02.388 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.898189+0000 osd.1 (osd.1) 533 : cluster 0 4.12 scrub starts 2024-04-02T05:06:02.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[21528]: cluster 2024-04-02T05:06:00.900695+0000 osd.1 (osd.1) 534 : cluster 0 4.12 scrub ok 2024-04-02T05:06:02.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[21528]: cluster 2024-04-02T05:06:01.954166+0000 osd.7 (osd.7) 333 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:02.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[21528]: cluster 2024-04-02T05:06:01.954598+0000 osd.7 (osd.7) 334 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:02.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[21528]: audit 2024-04-02T05:06:02.213336+0000 mon.c (mon.2) 1175 : audit 1 from='client.? 172.21.15.69:0/1397324598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:06:02.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:02 smithi069 bash[21528]: audit 2024-04-02T05:06:02.213984+0000 mon.a (mon.0) 4152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:06:02.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:02 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.898189+0000 osd.1 (osd.1) 533 : cluster 0 4.12 scrub starts 2024-04-02T05:06:02.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:02 smithi098 bash[23790]: cluster 2024-04-02T05:06:00.900695+0000 osd.1 (osd.1) 534 : cluster 0 4.12 scrub ok 2024-04-02T05:06:02.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:02 smithi098 bash[23790]: cluster 2024-04-02T05:06:01.954166+0000 osd.7 (osd.7) 333 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:02.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:02 smithi098 bash[23790]: cluster 2024-04-02T05:06:01.954598+0000 osd.7 (osd.7) 334 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:02.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:02 smithi098 bash[23790]: audit 2024-04-02T05:06:02.213336+0000 mon.c (mon.2) 1175 : audit 1 from='client.? 172.21.15.69:0/1397324598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:06:02.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:02 smithi098 bash[23790]: audit 2024-04-02T05:06:02.213984+0000 mon.a (mon.0) 4152 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-02T05:06:03.176 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 1 2024-04-02T05:06:03.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:01.941496+0000 osd.1 (osd.1) 535 : cluster 0 2.9 deep-scrub starts 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:01.943782+0000 osd.1 (osd.1) 536 : cluster 0 2.9 deep-scrub ok 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.230176+0000 osd.6 (osd.6) 463 : cluster 0 3.10 scrub starts 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.233155+0000 osd.6 (osd.6) 464 : cluster 0 3.10 scrub ok 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: audit 2024-04-02T05:06:02.337191+0000 mon.a (mon.0) 4153 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.344070+0000 mon.a (mon.0) 4154 : cluster 0 osdmap e946: 8 total, 8 up, 8 in 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.445619+0000 mgr.y (mgr.14538) 855 : cluster 0 pgmap v1484: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 73 B/s, 6 objects/s recovering 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.669045+0000 osd.4 (osd.4) 494 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.669357+0000 osd.4 (osd.4) 495 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:01.941496+0000 osd.1 (osd.1) 535 : cluster 0 2.9 deep-scrub starts 2024-04-02T05:06:03.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:01.943782+0000 osd.1 (osd.1) 536 : cluster 0 2.9 deep-scrub ok 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.230176+0000 osd.6 (osd.6) 463 : cluster 0 3.10 scrub starts 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.233155+0000 osd.6 (osd.6) 464 : cluster 0 3.10 scrub ok 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: audit 2024-04-02T05:06:02.337191+0000 mon.a (mon.0) 4153 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.344070+0000 mon.a (mon.0) 4154 : cluster 0 osdmap e946: 8 total, 8 up, 8 in 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.445619+0000 mgr.y (mgr.14538) 855 : cluster 0 pgmap v1484: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 73 B/s, 6 objects/s recovering 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.669045+0000 osd.4 (osd.4) 494 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:03.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:03 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.669357+0000 osd.4 (osd.4) 495 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:01.941496+0000 osd.1 (osd.1) 535 : cluster 0 2.9 deep-scrub starts 2024-04-02T05:06:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:01.943782+0000 osd.1 (osd.1) 536 : cluster 0 2.9 deep-scrub ok 2024-04-02T05:06:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.230176+0000 osd.6 (osd.6) 463 : cluster 0 3.10 scrub starts 2024-04-02T05:06:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.233155+0000 osd.6 (osd.6) 464 : cluster 0 3.10 scrub ok 2024-04-02T05:06:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: audit 2024-04-02T05:06:02.337191+0000 mon.a (mon.0) 4153 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-02T05:06:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.344070+0000 mon.a (mon.0) 4154 : cluster 0 osdmap e946: 8 total, 8 up, 8 in 2024-04-02T05:06:03.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.445619+0000 mgr.y (mgr.14538) 855 : cluster 0 pgmap v1484: 184 pgs: 184 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 73 B/s, 6 objects/s recovering 2024-04-02T05:06:03.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.669045+0000 osd.4 (osd.4) 494 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:03.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.669357+0000 osd.4 (osd.4) 495 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:04.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:06:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:06:04.474 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:06:04 smithi098 bash[51598]: logger=cleanup t=2024-04-02T05:06:04.221131342Z level=info msg="Completed cleanup jobs" duration=55.895218ms 2024-04-02T05:06:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:04 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.981273+0000 osd.1 (osd.1) 537 : cluster 0 10.2 scrub starts 2024-04-02T05:06:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:04 smithi098 bash[23790]: cluster 2024-04-02T05:06:02.984267+0000 osd.1 (osd.1) 538 : cluster 0 10.2 scrub ok 2024-04-02T05:06:04.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:04 smithi098 bash[23790]: cluster 2024-04-02T05:06:03.364168+0000 mon.a (mon.0) 4155 : cluster 0 osdmap e947: 8 total, 8 up, 8 in 2024-04-02T05:06:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:04 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.981273+0000 osd.1 (osd.1) 537 : cluster 0 10.2 scrub starts 2024-04-02T05:06:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:04 smithi069 bash[21528]: cluster 2024-04-02T05:06:02.984267+0000 osd.1 (osd.1) 538 : cluster 0 10.2 scrub ok 2024-04-02T05:06:04.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:04 smithi069 bash[21528]: cluster 2024-04-02T05:06:03.364168+0000 mon.a (mon.0) 4155 : cluster 0 osdmap e947: 8 total, 8 up, 8 in 2024-04-02T05:06:04.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:04 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.981273+0000 osd.1 (osd.1) 537 : cluster 0 10.2 scrub starts 2024-04-02T05:06:04.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:04 smithi069 bash[28904]: cluster 2024-04-02T05:06:02.984267+0000 osd.1 (osd.1) 538 : cluster 0 10.2 scrub ok 2024-04-02T05:06:04.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:04 smithi069 bash[28904]: cluster 2024-04-02T05:06:03.364168+0000 mon.a (mon.0) 4155 : cluster 0 osdmap e947: 8 total, 8 up, 8 in 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: audit 2024-04-02T05:06:03.918709+0000 mgr.y (mgr.14538) 856 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: cluster 2024-04-02T05:06:04.202383+0000 osd.0 (osd.0) 499 : cluster 0 4.1e scrub starts 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: cluster 2024-04-02T05:06:04.206785+0000 osd.0 (osd.0) 500 : cluster 0 4.1e scrub ok 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: cluster 2024-04-02T05:06:04.446592+0000 mgr.y (mgr.14538) 857 : cluster 0 pgmap v1486: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: audit 2024-04-02T05:06:04.578260+0000 mon.b (mon.1) 347 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: cluster 2024-04-02T05:06:04.587288+0000 osd.4 (osd.4) 496 : cluster 0 10.1 scrub starts 2024-04-02T05:06:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:05 smithi098 bash[23790]: cluster 2024-04-02T05:06:04.590556+0000 osd.4 (osd.4) 497 : cluster 0 10.1 scrub ok 2024-04-02T05:06:05.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: audit 2024-04-02T05:06:03.918709+0000 mgr.y (mgr.14538) 856 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: cluster 2024-04-02T05:06:04.202383+0000 osd.0 (osd.0) 499 : cluster 0 4.1e scrub starts 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: cluster 2024-04-02T05:06:04.206785+0000 osd.0 (osd.0) 500 : cluster 0 4.1e scrub ok 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: cluster 2024-04-02T05:06:04.446592+0000 mgr.y (mgr.14538) 857 : cluster 0 pgmap v1486: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: audit 2024-04-02T05:06:04.578260+0000 mon.b (mon.1) 347 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: cluster 2024-04-02T05:06:04.587288+0000 osd.4 (osd.4) 496 : cluster 0 10.1 scrub starts 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[21528]: cluster 2024-04-02T05:06:04.590556+0000 osd.4 (osd.4) 497 : cluster 0 10.1 scrub ok 2024-04-02T05:06:05.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: audit 2024-04-02T05:06:03.918709+0000 mgr.y (mgr.14538) 856 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:05.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: cluster 2024-04-02T05:06:04.202383+0000 osd.0 (osd.0) 499 : cluster 0 4.1e scrub starts 2024-04-02T05:06:05.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: cluster 2024-04-02T05:06:04.206785+0000 osd.0 (osd.0) 500 : cluster 0 4.1e scrub ok 2024-04-02T05:06:05.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: cluster 2024-04-02T05:06:04.446592+0000 mgr.y (mgr.14538) 857 : cluster 0 pgmap v1486: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-04-02T05:06:05.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: audit 2024-04-02T05:06:04.578260+0000 mon.b (mon.1) 347 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:05.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: cluster 2024-04-02T05:06:04.587288+0000 osd.4 (osd.4) 496 : cluster 0 10.1 scrub starts 2024-04-02T05:06:05.788 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:05 smithi069 bash[28904]: cluster 2024-04-02T05:06:04.590556+0000 osd.4 (osd.4) 497 : cluster 0 10.1 scrub ok 2024-04-02T05:06:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:06 smithi098 bash[23790]: cluster 2024-04-02T05:06:05.460337+0000 osd.3 (osd.3) 541 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:06 smithi098 bash[23790]: cluster 2024-04-02T05:06:05.460701+0000 osd.3 (osd.3) 542 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:06 smithi069 bash[21528]: cluster 2024-04-02T05:06:05.460337+0000 osd.3 (osd.3) 541 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:06 smithi069 bash[21528]: cluster 2024-04-02T05:06:05.460701+0000 osd.3 (osd.3) 542 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:06.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:06 smithi069 bash[28904]: cluster 2024-04-02T05:06:05.460337+0000 osd.3 (osd.3) 541 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:06.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:06 smithi069 bash[28904]: cluster 2024-04-02T05:06:05.460701+0000 osd.3 (osd.3) 542 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:07.074 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:06:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:07 smithi098 bash[23790]: cluster 2024-04-02T05:06:06.382438+0000 mon.a (mon.0) 4156 : cluster 0 osdmap e948: 8 total, 8 up, 8 in 2024-04-02T05:06:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:07 smithi098 bash[23790]: cluster 2024-04-02T05:06:06.447375+0000 mgr.y (mgr.14538) 858 : cluster 0 pgmap v1488: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:06:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:07 smithi098 bash[23790]: cluster 2024-04-02T05:06:06.600599+0000 osd.4 (osd.4) 498 : cluster 0 10.10 scrub starts 2024-04-02T05:06:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:07 smithi098 bash[23790]: cluster 2024-04-02T05:06:06.602422+0000 osd.4 (osd.4) 499 : cluster 0 10.10 scrub ok 2024-04-02T05:06:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[21528]: cluster 2024-04-02T05:06:06.382438+0000 mon.a (mon.0) 4156 : cluster 0 osdmap e948: 8 total, 8 up, 8 in 2024-04-02T05:06:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[21528]: cluster 2024-04-02T05:06:06.447375+0000 mgr.y (mgr.14538) 858 : cluster 0 pgmap v1488: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:06:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[21528]: cluster 2024-04-02T05:06:06.600599+0000 osd.4 (osd.4) 498 : cluster 0 10.10 scrub starts 2024-04-02T05:06:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[21528]: cluster 2024-04-02T05:06:06.602422+0000 osd.4 (osd.4) 499 : cluster 0 10.10 scrub ok 2024-04-02T05:06:07.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[28904]: cluster 2024-04-02T05:06:06.382438+0000 mon.a (mon.0) 4156 : cluster 0 osdmap e948: 8 total, 8 up, 8 in 2024-04-02T05:06:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[28904]: cluster 2024-04-02T05:06:06.447375+0000 mgr.y (mgr.14538) 858 : cluster 0 pgmap v1488: 184 pgs: 184 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:06:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[28904]: cluster 2024-04-02T05:06:06.600599+0000 osd.4 (osd.4) 498 : cluster 0 10.10 scrub starts 2024-04-02T05:06:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:07 smithi069 bash[28904]: cluster 2024-04-02T05:06:06.602422+0000 osd.4 (osd.4) 499 : cluster 0 10.10 scrub ok 2024-04-02T05:06:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:06:08.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:08 smithi098 bash[23790]: cluster 2024-04-02T05:06:07.126722+0000 osd.3 (osd.3) 543 : cluster 0 3.1e scrub starts 2024-04-02T05:06:08.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:08 smithi098 bash[23790]: cluster 2024-04-02T05:06:07.130808+0000 osd.3 (osd.3) 544 : cluster 0 3.1e scrub ok 2024-04-02T05:06:08.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:08 smithi098 bash[23790]: cluster 2024-04-02T05:06:07.183281+0000 osd.0 (osd.0) 501 : cluster 0 3.12 scrub starts 2024-04-02T05:06:08.578 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:08 smithi098 bash[23790]: cluster 2024-04-02T05:06:07.187135+0000 osd.0 (osd.0) 502 : cluster 0 3.12 scrub ok 2024-04-02T05:06:08.578 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:08 smithi098 bash[23790]: cluster 2024-04-02T05:06:07.522059+0000 osd.7 (osd.7) 335 : cluster 0 2.0 deep-scrub starts 2024-04-02T05:06:08.578 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:08 smithi098 bash[23790]: cluster 2024-04-02T05:06:07.525252+0000 osd.7 (osd.7) 336 : cluster 0 2.0 deep-scrub ok 2024-04-02T05:06:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21528]: cluster 2024-04-02T05:06:07.126722+0000 osd.3 (osd.3) 543 : cluster 0 3.1e scrub starts 2024-04-02T05:06:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21528]: cluster 2024-04-02T05:06:07.130808+0000 osd.3 (osd.3) 544 : cluster 0 3.1e scrub ok 2024-04-02T05:06:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21528]: cluster 2024-04-02T05:06:07.183281+0000 osd.0 (osd.0) 501 : cluster 0 3.12 scrub starts 2024-04-02T05:06:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21528]: cluster 2024-04-02T05:06:07.187135+0000 osd.0 (osd.0) 502 : cluster 0 3.12 scrub ok 2024-04-02T05:06:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21528]: cluster 2024-04-02T05:06:07.522059+0000 osd.7 (osd.7) 335 : cluster 0 2.0 deep-scrub starts 2024-04-02T05:06:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21528]: cluster 2024-04-02T05:06:07.525252+0000 osd.7 (osd.7) 336 : cluster 0 2.0 deep-scrub ok 2024-04-02T05:06:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[28904]: cluster 2024-04-02T05:06:07.126722+0000 osd.3 (osd.3) 543 : cluster 0 3.1e scrub starts 2024-04-02T05:06:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[28904]: cluster 2024-04-02T05:06:07.130808+0000 osd.3 (osd.3) 544 : cluster 0 3.1e scrub ok 2024-04-02T05:06:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[28904]: cluster 2024-04-02T05:06:07.183281+0000 osd.0 (osd.0) 501 : cluster 0 3.12 scrub starts 2024-04-02T05:06:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[28904]: cluster 2024-04-02T05:06:07.187135+0000 osd.0 (osd.0) 502 : cluster 0 3.12 scrub ok 2024-04-02T05:06:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[28904]: cluster 2024-04-02T05:06:07.522059+0000 osd.7 (osd.7) 335 : cluster 0 2.0 deep-scrub starts 2024-04-02T05:06:08.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[28904]: cluster 2024-04-02T05:06:07.525252+0000 osd.7 (osd.7) 336 : cluster 0 2.0 deep-scrub ok 2024-04-02T05:06:08.985 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:06:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:06:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:06:09.474 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:09 smithi098 bash[23790]: cluster 2024-04-02T05:06:08.449324+0000 mgr.y (mgr.14538) 859 : cluster 0 pgmap v1489: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-02T05:06:09.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:09 smithi069 bash[21528]: cluster 2024-04-02T05:06:08.449324+0000 mgr.y (mgr.14538) 859 : cluster 0 pgmap v1489: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-02T05:06:09.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:09 smithi069 bash[28904]: cluster 2024-04-02T05:06:08.449324+0000 mgr.y (mgr.14538) 859 : cluster 0 pgmap v1489: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-02T05:06:10.207 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:06:10.208 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:06:10.208 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-04-02T05:06:10.208 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:06:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:10 smithi098 bash[23790]: cluster 2024-04-02T05:06:09.048827+0000 osd.6 (osd.6) 465 : cluster 0 4.1f scrub starts 2024-04-02T05:06:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:10 smithi098 bash[23790]: cluster 2024-04-02T05:06:09.051747+0000 osd.6 (osd.6) 466 : cluster 0 4.1f scrub ok 2024-04-02T05:06:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:10 smithi098 bash[23790]: cluster 2024-04-02T05:06:09.218158+0000 osd.3 (osd.3) 545 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:06:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:10 smithi098 bash[23790]: cluster 2024-04-02T05:06:09.220833+0000 osd.3 (osd.3) 546 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:06:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:10 smithi098 bash[23790]: cluster 2024-04-02T05:06:09.983007+0000 osd.2 (osd.2) 408 : cluster 0 2.c scrub starts 2024-04-02T05:06:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:10 smithi098 bash[23790]: cluster 2024-04-02T05:06:09.985240+0000 osd.2 (osd.2) 409 : cluster 0 2.c scrub ok 2024-04-02T05:06:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[28904]: cluster 2024-04-02T05:06:09.048827+0000 osd.6 (osd.6) 465 : cluster 0 4.1f scrub starts 2024-04-02T05:06:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[28904]: cluster 2024-04-02T05:06:09.051747+0000 osd.6 (osd.6) 466 : cluster 0 4.1f scrub ok 2024-04-02T05:06:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[28904]: cluster 2024-04-02T05:06:09.218158+0000 osd.3 (osd.3) 545 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:06:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[28904]: cluster 2024-04-02T05:06:09.220833+0000 osd.3 (osd.3) 546 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:06:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[28904]: cluster 2024-04-02T05:06:09.983007+0000 osd.2 (osd.2) 408 : cluster 0 2.c scrub starts 2024-04-02T05:06:10.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[28904]: cluster 2024-04-02T05:06:09.985240+0000 osd.2 (osd.2) 409 : cluster 0 2.c scrub ok 2024-04-02T05:06:10.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[21528]: cluster 2024-04-02T05:06:09.048827+0000 osd.6 (osd.6) 465 : cluster 0 4.1f scrub starts 2024-04-02T05:06:10.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[21528]: cluster 2024-04-02T05:06:09.051747+0000 osd.6 (osd.6) 466 : cluster 0 4.1f scrub ok 2024-04-02T05:06:10.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[21528]: cluster 2024-04-02T05:06:09.218158+0000 osd.3 (osd.3) 545 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:06:10.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[21528]: cluster 2024-04-02T05:06:09.220833+0000 osd.3 (osd.3) 546 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:06:10.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[21528]: cluster 2024-04-02T05:06:09.983007+0000 osd.2 (osd.2) 408 : cluster 0 2.c scrub starts 2024-04-02T05:06:10.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:10 smithi069 bash[21528]: cluster 2024-04-02T05:06:09.985240+0000 osd.2 (osd.2) 409 : cluster 0 2.c scrub ok 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.937+0000 7fce662f0700 1 -- 172.21.15.69:0/4088209422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60107230 msgr2=0x7fce60107610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- 172.21.15.69:0/4088209422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60107230 0x7fce60107610 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7fce500098f0 tx=0x7fce50009c00 comp rx=0 tx=0).stop 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/4088209422 shutdown_connections 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- 172.21.15.69:0/4088209422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101b40 0x7fce6010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- 172.21.15.69:0/4088209422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60107230 0x7fce60107610 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- 172.21.15.69:0/4088209422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60101180 0x7fce60101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/4088209422 >> 172.21.15.69:0/4088209422 conn(0x7fce600781f0 msgr2=0x7fce60078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/4088209422 shutdown_connections 2024-04-02T05:06:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/4088209422 wait complete. 2024-04-02T05:06:10.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 librados: starting msgr at 2024-04-02T05:06:10.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 librados: starting objecter 2024-04-02T05:06:10.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 Processor -- start 2024-04-02T05:06:10.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- start start 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 librados: setting wanted keys 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 librados: calling monclient init 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101180 0x7fce60198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60101b40 0x7fce60198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 0x7fce6019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fce60111ae0 con 0x7fce60107230 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce60111960 con 0x7fce60101180 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fce60111c60 con 0x7fce60101b40 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 0x7fce6019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 0x7fce6019c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52028/0 (socket says 172.21.15.69:52028) 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 -- 172.21.15.69:0/3100572744 learned_addr learned my addr 172.21.15.69:0/3100572744 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5f7fe700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60101b40 0x7fce60198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5ffff700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101180 0x7fce60198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:10.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 -- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60101b40 msgr2=0x7fce60198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60101b40 0x7fce60198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 -- 172.21.15.69:0/3100572744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101180 msgr2=0x7fce60198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101180 0x7fce60198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 -- 172.21.15.69:0/3100572744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6019d060 con 0x7fce60107230 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5ffff700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101180 0x7fce60198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce6488d700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 0x7fce6019c920 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7fce5800b970 tx=0x7fce5800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5d7fa700 10 client.?.objecter ms_handle_connect 0x7fce60107230 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5d7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58019070 con 0x7fce60107230 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6019d350 con 0x7fce60107230 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fce5800cdb0 con 0x7fce60107230 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 10 client.33927.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58014e40 con 0x7fce60107230 2024-04-02T05:06:10.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce601a4bd0 con 0x7fce60107230 2024-04-02T05:06:10.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 librados: init done 2024-04-02T05:06:10.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce60102870 con 0x7fce60107230 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.941+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fce5801c070 con 0x7fce60107230 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fce4806f4b0 0x7fce48071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5ffff700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fce4806f4b0 0x7fce48071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(948..948 src has 897..948) ==== 9015+0+0 (secure 0 0 0) 0x7fce5808dae0 con 0x7fce60107230 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 10 client.33927.objecter ms_dispatch 0x7fce60001550 osd_map(948..948 src has 897..948) 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 3 client.33927.objecter handle_osd_map got epochs [948,948] > 0 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 3 client.33927.objecter handle_osd_map decoding full epoch 948 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 20 client.33927.objecter dump_active .. 0 homeless 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5d7fa700 10 client.33927.objecter ms_handle_connect 0x7fce4806f4b0 2024-04-02T05:06:10.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.945+0000 7fce5ffff700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fce4806f4b0 0x7fce48071970 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 crypto rx=0x7fce50009fd0 tx=0x7fce50019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:10.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:10.949+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fce5805aab0 con 0x7fce60107230 2024-04-02T05:06:11.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-02T05:06:11.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.169+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7fce6004ff00 con 0x7fce60107230 2024-04-02T05:06:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.409+0000 7fce5d7fa700 1 -- 172.21.15.69:0/3100572744 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v949) ==== 113+0+0 (secure 0 0 0) 0x7fce5805e500 con 0x7fce60107230 2024-04-02T05:06:11.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.413+0000 7fce662f0700 10 librados: watch_flush enter 2024-04-02T05:06:11.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 10 librados: watch_flush exit 2024-04-02T05:06:11.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 20 client.33927.objecter shutdown clearing up homeless session... 2024-04-02T05:06:11.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 10 client.33927.objecter successfully canceled tick 2024-04-02T05:06:11.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fce4806f4b0 msgr2=0x7fce48071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:11.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fce4806f4b0 0x7fce48071970 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 crypto rx=0x7fce50009fd0 tx=0x7fce50019040 comp rx=0 tx=0).stop 2024-04-02T05:06:11.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 msgr2=0x7fce6019c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:11.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.417+0000 7fce662f0700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 0x7fce6019c920 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7fce5800b970 tx=0x7fce5800bc80 comp rx=0 tx=0).stop 2024-04-02T05:06:11.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 shutdown_connections 2024-04-02T05:06:11.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fce4806f4b0 0x7fce48071970 unknown :-1 s=CLOSED pgs=4044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:11.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce60107230 0x7fce6019c920 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:11.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce60101180 0x7fce60198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:11.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 --2- 172.21.15.69:0/3100572744 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce60101b40 0x7fce60198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:11.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 >> 172.21.15.69:0/3100572744 conn(0x7fce600781f0 msgr2=0x7fce60107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:11.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 shutdown_connections 2024-04-02T05:06:11.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 -- 172.21.15.69:0/3100572744 wait complete. 2024-04-02T05:06:11.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:11.421+0000 7fce662f0700 1 librados: shutdown 2024-04-02T05:06:11.428 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.5 to 1 (10000) 2024-04-02T05:06:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:11 smithi098 bash[23790]: cluster 2024-04-02T05:06:10.450464+0000 mgr.y (mgr.14538) 860 : cluster 0 pgmap v1490: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:11 smithi098 bash[23790]: cluster 2024-04-02T05:06:10.937743+0000 osd.2 (osd.2) 410 : cluster 0 1.14 scrub starts 2024-04-02T05:06:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:11 smithi098 bash[23790]: cluster 2024-04-02T05:06:10.939461+0000 osd.2 (osd.2) 411 : cluster 0 1.14 scrub ok 2024-04-02T05:06:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:11 smithi098 bash[23790]: audit 2024-04-02T05:06:11.175300+0000 mon.a (mon.0) 4157 : audit 1 from='client.? 172.21.15.69:0/3100572744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-02T05:06:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[21528]: cluster 2024-04-02T05:06:10.450464+0000 mgr.y (mgr.14538) 860 : cluster 0 pgmap v1490: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[21528]: cluster 2024-04-02T05:06:10.937743+0000 osd.2 (osd.2) 410 : cluster 0 1.14 scrub starts 2024-04-02T05:06:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[21528]: cluster 2024-04-02T05:06:10.939461+0000 osd.2 (osd.2) 411 : cluster 0 1.14 scrub ok 2024-04-02T05:06:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[21528]: audit 2024-04-02T05:06:11.175300+0000 mon.a (mon.0) 4157 : audit 1 from='client.? 172.21.15.69:0/3100572744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-02T05:06:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[28904]: cluster 2024-04-02T05:06:10.450464+0000 mgr.y (mgr.14538) 860 : cluster 0 pgmap v1490: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[28904]: cluster 2024-04-02T05:06:10.937743+0000 osd.2 (osd.2) 410 : cluster 0 1.14 scrub starts 2024-04-02T05:06:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[28904]: cluster 2024-04-02T05:06:10.939461+0000 osd.2 (osd.2) 411 : cluster 0 1.14 scrub ok 2024-04-02T05:06:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:11 smithi069 bash[28904]: audit 2024-04-02T05:06:11.175300+0000 mon.a (mon.0) 4157 : audit 1 from='client.? 172.21.15.69:0/3100572744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-02T05:06:12.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 5 1 2024-04-02T05:06:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[21528]: audit 2024-04-02T05:06:11.411649+0000 mon.a (mon.0) 4158 : audit 1 from='client.? 172.21.15.69:0/3100572744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-02T05:06:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[21528]: cluster 2024-04-02T05:06:11.419878+0000 mon.a (mon.0) 4159 : cluster 0 osdmap e949: 8 total, 8 up, 8 in 2024-04-02T05:06:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[21528]: cluster 2024-04-02T05:06:11.623494+0000 osd.4 (osd.4) 500 : cluster 0 6.e scrub starts 2024-04-02T05:06:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[21528]: cluster 2024-04-02T05:06:11.625965+0000 osd.4 (osd.4) 501 : cluster 0 6.e scrub ok 2024-04-02T05:06:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[28904]: audit 2024-04-02T05:06:11.411649+0000 mon.a (mon.0) 4158 : audit 1 from='client.? 172.21.15.69:0/3100572744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-02T05:06:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[28904]: cluster 2024-04-02T05:06:11.419878+0000 mon.a (mon.0) 4159 : cluster 0 osdmap e949: 8 total, 8 up, 8 in 2024-04-02T05:06:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[28904]: cluster 2024-04-02T05:06:11.623494+0000 osd.4 (osd.4) 500 : cluster 0 6.e scrub starts 2024-04-02T05:06:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:12 smithi069 bash[28904]: cluster 2024-04-02T05:06:11.625965+0000 osd.4 (osd.4) 501 : cluster 0 6.e scrub ok 2024-04-02T05:06:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:12 smithi098 bash[23790]: audit 2024-04-02T05:06:11.411649+0000 mon.a (mon.0) 4158 : audit 1 from='client.? 172.21.15.69:0/3100572744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-02T05:06:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:12 smithi098 bash[23790]: cluster 2024-04-02T05:06:11.419878+0000 mon.a (mon.0) 4159 : cluster 0 osdmap e949: 8 total, 8 up, 8 in 2024-04-02T05:06:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:12 smithi098 bash[23790]: cluster 2024-04-02T05:06:11.623494+0000 osd.4 (osd.4) 500 : cluster 0 6.e scrub starts 2024-04-02T05:06:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:12 smithi098 bash[23790]: cluster 2024-04-02T05:06:11.625965+0000 osd.4 (osd.4) 501 : cluster 0 6.e scrub ok 2024-04-02T05:06:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:13 smithi098 bash[23790]: cluster 2024-04-02T05:06:12.451070+0000 mgr.y (mgr.14538) 861 : cluster 0 pgmap v1492: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:13 smithi098 bash[23790]: cluster 2024-04-02T05:06:12.488794+0000 osd.7 (osd.7) 337 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:06:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:13 smithi098 bash[23790]: cluster 2024-04-02T05:06:12.493867+0000 osd.7 (osd.7) 338 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:06:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:13 smithi098 bash[23790]: cluster 2024-04-02T05:06:12.925925+0000 osd.2 (osd.2) 412 : cluster 0 10.18 scrub starts 2024-04-02T05:06:13.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:13 smithi098 bash[23790]: cluster 2024-04-02T05:06:12.928339+0000 osd.2 (osd.2) 413 : cluster 0 10.18 scrub ok 2024-04-02T05:06:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[21528]: cluster 2024-04-02T05:06:12.451070+0000 mgr.y (mgr.14538) 861 : cluster 0 pgmap v1492: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[21528]: cluster 2024-04-02T05:06:12.488794+0000 osd.7 (osd.7) 337 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:06:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[21528]: cluster 2024-04-02T05:06:12.493867+0000 osd.7 (osd.7) 338 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:06:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[21528]: cluster 2024-04-02T05:06:12.925925+0000 osd.2 (osd.2) 412 : cluster 0 10.18 scrub starts 2024-04-02T05:06:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[21528]: cluster 2024-04-02T05:06:12.928339+0000 osd.2 (osd.2) 413 : cluster 0 10.18 scrub ok 2024-04-02T05:06:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[28904]: cluster 2024-04-02T05:06:12.451070+0000 mgr.y (mgr.14538) 861 : cluster 0 pgmap v1492: 184 pgs: 184 active+clean; 582 KiB data, 763 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[28904]: cluster 2024-04-02T05:06:12.488794+0000 osd.7 (osd.7) 337 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:06:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[28904]: cluster 2024-04-02T05:06:12.493867+0000 osd.7 (osd.7) 338 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:06:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[28904]: cluster 2024-04-02T05:06:12.925925+0000 osd.2 (osd.2) 412 : cluster 0 10.18 scrub starts 2024-04-02T05:06:13.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:13 smithi069 bash[28904]: cluster 2024-04-02T05:06:12.928339+0000 osd.2 (osd.2) 413 : cluster 0 10.18 scrub ok 2024-04-02T05:06:14.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:06:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:06:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:14 smithi098 bash[23790]: cluster 2024-04-02T05:06:13.207062+0000 osd.3 (osd.3) 547 : cluster 0 6.6 scrub starts 2024-04-02T05:06:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:14 smithi098 bash[23790]: cluster 2024-04-02T05:06:13.210706+0000 osd.3 (osd.3) 548 : cluster 0 6.6 scrub ok 2024-04-02T05:06:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:14 smithi098 bash[23790]: cluster 2024-04-02T05:06:13.930869+0000 osd.2 (osd.2) 414 : cluster 0 10.5 scrub starts 2024-04-02T05:06:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:14 smithi098 bash[23790]: cluster 2024-04-02T05:06:13.936495+0000 osd.2 (osd.2) 415 : cluster 0 10.5 scrub ok 2024-04-02T05:06:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[21528]: cluster 2024-04-02T05:06:13.207062+0000 osd.3 (osd.3) 547 : cluster 0 6.6 scrub starts 2024-04-02T05:06:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[21528]: cluster 2024-04-02T05:06:13.210706+0000 osd.3 (osd.3) 548 : cluster 0 6.6 scrub ok 2024-04-02T05:06:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[21528]: cluster 2024-04-02T05:06:13.930869+0000 osd.2 (osd.2) 414 : cluster 0 10.5 scrub starts 2024-04-02T05:06:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[21528]: cluster 2024-04-02T05:06:13.936495+0000 osd.2 (osd.2) 415 : cluster 0 10.5 scrub ok 2024-04-02T05:06:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[28904]: cluster 2024-04-02T05:06:13.207062+0000 osd.3 (osd.3) 547 : cluster 0 6.6 scrub starts 2024-04-02T05:06:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[28904]: cluster 2024-04-02T05:06:13.210706+0000 osd.3 (osd.3) 548 : cluster 0 6.6 scrub ok 2024-04-02T05:06:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[28904]: cluster 2024-04-02T05:06:13.930869+0000 osd.2 (osd.2) 414 : cluster 0 10.5 scrub starts 2024-04-02T05:06:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:14 smithi069 bash[28904]: cluster 2024-04-02T05:06:13.936495+0000 osd.2 (osd.2) 415 : cluster 0 10.5 scrub ok 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:13.867589+0000 osd.5 (osd.5) 349 : cluster 0 1.1 scrub starts 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:13.870555+0000 osd.5 (osd.5) 350 : cluster 0 1.1 scrub ok 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: audit 2024-04-02T05:06:13.930106+0000 mgr.y (mgr.14538) 862 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:14.037120+0000 osd.6 (osd.6) 467 : cluster 0 2.7 scrub starts 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:14.039502+0000 osd.6 (osd.6) 468 : cluster 0 2.7 scrub ok 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:14.453141+0000 mgr.y (mgr.14538) 863 : cluster 0 pgmap v1493: 184 pgs: 184 active+clean; 582 KiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:14.514979+0000 osd.7 (osd.7) 339 : cluster 0 10.e scrub starts 2024-04-02T05:06:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:15 smithi098 bash[23790]: cluster 2024-04-02T05:06:14.518035+0000 osd.7 (osd.7) 340 : cluster 0 10.e scrub ok 2024-04-02T05:06:15.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:13.867589+0000 osd.5 (osd.5) 349 : cluster 0 1.1 scrub starts 2024-04-02T05:06:15.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:13.870555+0000 osd.5 (osd.5) 350 : cluster 0 1.1 scrub ok 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: audit 2024-04-02T05:06:13.930106+0000 mgr.y (mgr.14538) 862 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:14.037120+0000 osd.6 (osd.6) 467 : cluster 0 2.7 scrub starts 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:14.039502+0000 osd.6 (osd.6) 468 : cluster 0 2.7 scrub ok 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:14.453141+0000 mgr.y (mgr.14538) 863 : cluster 0 pgmap v1493: 184 pgs: 184 active+clean; 582 KiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:14.514979+0000 osd.7 (osd.7) 339 : cluster 0 10.e scrub starts 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[21528]: cluster 2024-04-02T05:06:14.518035+0000 osd.7 (osd.7) 340 : cluster 0 10.e scrub ok 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:13.867589+0000 osd.5 (osd.5) 349 : cluster 0 1.1 scrub starts 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:13.870555+0000 osd.5 (osd.5) 350 : cluster 0 1.1 scrub ok 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: audit 2024-04-02T05:06:13.930106+0000 mgr.y (mgr.14538) 862 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:14.037120+0000 osd.6 (osd.6) 467 : cluster 0 2.7 scrub starts 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:14.039502+0000 osd.6 (osd.6) 468 : cluster 0 2.7 scrub ok 2024-04-02T05:06:15.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:14.453141+0000 mgr.y (mgr.14538) 863 : cluster 0 pgmap v1493: 184 pgs: 184 active+clean; 582 KiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:15.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:14.514979+0000 osd.7 (osd.7) 339 : cluster 0 10.e scrub starts 2024-04-02T05:06:15.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:15 smithi069 bash[28904]: cluster 2024-04-02T05:06:14.518035+0000 osd.7 (osd.7) 340 : cluster 0 10.e scrub ok 2024-04-02T05:06:15.796 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:15.259505+0000 osd.0 (osd.0) 503 : cluster 0 10.1a scrub starts 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:15.262481+0000 osd.0 (osd.0) 504 : cluster 0 10.1a scrub ok 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:15.380744+0000 osd.6 (osd.6) 469 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:15.381423+0000 osd.6 (osd.6) 470 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:15.941450+0000 osd.2 (osd.2) 416 : cluster 0 2.e scrub starts 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:15.944208+0000 osd.2 (osd.2) 417 : cluster 0 2.e scrub ok 2024-04-02T05:06:16.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:16 smithi098 bash[23790]: cluster 2024-04-02T05:06:16.224139+0000 mon.a (mon.0) 4160 : cluster 0 osdmap e950: 8 total, 8 up, 8 in 2024-04-02T05:06:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:15.259505+0000 osd.0 (osd.0) 503 : cluster 0 10.1a scrub starts 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:15.262481+0000 osd.0 (osd.0) 504 : cluster 0 10.1a scrub ok 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:15.380744+0000 osd.6 (osd.6) 469 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:15.381423+0000 osd.6 (osd.6) 470 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:15.941450+0000 osd.2 (osd.2) 416 : cluster 0 2.e scrub starts 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:15.944208+0000 osd.2 (osd.2) 417 : cluster 0 2.e scrub ok 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[21528]: cluster 2024-04-02T05:06:16.224139+0000 mon.a (mon.0) 4160 : cluster 0 osdmap e950: 8 total, 8 up, 8 in 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:15.259505+0000 osd.0 (osd.0) 503 : cluster 0 10.1a scrub starts 2024-04-02T05:06:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:15.262481+0000 osd.0 (osd.0) 504 : cluster 0 10.1a scrub ok 2024-04-02T05:06:16.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:15.380744+0000 osd.6 (osd.6) 469 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:16.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:15.381423+0000 osd.6 (osd.6) 470 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:16.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:15.941450+0000 osd.2 (osd.2) 416 : cluster 0 2.e scrub starts 2024-04-02T05:06:16.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:15.944208+0000 osd.2 (osd.2) 417 : cluster 0 2.e scrub ok 2024-04-02T05:06:16.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:16 smithi069 bash[28904]: cluster 2024-04-02T05:06:16.224139+0000 mon.a (mon.0) 4160 : cluster 0 osdmap e950: 8 total, 8 up, 8 in 2024-04-02T05:06:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:06:17.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:17 smithi098 bash[23790]: cluster 2024-04-02T05:06:16.094924+0000 osd.6 (osd.6) 471 : cluster 0 10.7 deep-scrub starts 2024-04-02T05:06:17.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:17 smithi098 bash[23790]: cluster 2024-04-02T05:06:16.100609+0000 osd.6 (osd.6) 472 : cluster 0 10.7 deep-scrub ok 2024-04-02T05:06:17.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:17 smithi098 bash[23790]: cluster 2024-04-02T05:06:16.454024+0000 mgr.y (mgr.14538) 864 : cluster 0 pgmap v1495: 184 pgs: 184 active+clean; 582 KiB data, 761 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:17 smithi069 bash[21528]: cluster 2024-04-02T05:06:16.094924+0000 osd.6 (osd.6) 471 : cluster 0 10.7 deep-scrub starts 2024-04-02T05:06:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:17 smithi069 bash[21528]: cluster 2024-04-02T05:06:16.100609+0000 osd.6 (osd.6) 472 : cluster 0 10.7 deep-scrub ok 2024-04-02T05:06:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:17 smithi069 bash[21528]: cluster 2024-04-02T05:06:16.454024+0000 mgr.y (mgr.14538) 864 : cluster 0 pgmap v1495: 184 pgs: 184 active+clean; 582 KiB data, 761 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:17.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:17 smithi069 bash[28904]: cluster 2024-04-02T05:06:16.094924+0000 osd.6 (osd.6) 471 : cluster 0 10.7 deep-scrub starts 2024-04-02T05:06:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:17 smithi069 bash[28904]: cluster 2024-04-02T05:06:16.100609+0000 osd.6 (osd.6) 472 : cluster 0 10.7 deep-scrub ok 2024-04-02T05:06:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:17 smithi069 bash[28904]: cluster 2024-04-02T05:06:16.454024+0000 mgr.y (mgr.14538) 864 : cluster 0 pgmap v1495: 184 pgs: 184 active+clean; 582 KiB data, 761 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:18.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21528]: cluster 2024-04-02T05:06:17.051145+0000 osd.1 (osd.1) 539 : cluster 0 2.d scrub starts 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21528]: cluster 2024-04-02T05:06:17.053149+0000 osd.1 (osd.1) 540 : cluster 0 2.d scrub ok 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21528]: cluster 2024-04-02T05:06:17.284704+0000 osd.3 (osd.3) 549 : cluster 0 6.b scrub starts 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21528]: cluster 2024-04-02T05:06:17.287426+0000 osd.3 (osd.3) 550 : cluster 0 6.b scrub ok 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21528]: cluster 2024-04-02T05:06:17.998434+0000 osd.5 (osd.5) 351 : cluster 0 6.a scrub starts 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.000723+0000 osd.5 (osd.5) 352 : cluster 0 6.a scrub ok 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[28904]: cluster 2024-04-02T05:06:17.051145+0000 osd.1 (osd.1) 539 : cluster 0 2.d scrub starts 2024-04-02T05:06:18.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[28904]: cluster 2024-04-02T05:06:17.053149+0000 osd.1 (osd.1) 540 : cluster 0 2.d scrub ok 2024-04-02T05:06:18.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[28904]: cluster 2024-04-02T05:06:17.284704+0000 osd.3 (osd.3) 549 : cluster 0 6.b scrub starts 2024-04-02T05:06:18.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[28904]: cluster 2024-04-02T05:06:17.287426+0000 osd.3 (osd.3) 550 : cluster 0 6.b scrub ok 2024-04-02T05:06:18.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[28904]: cluster 2024-04-02T05:06:17.998434+0000 osd.5 (osd.5) 351 : cluster 0 6.a scrub starts 2024-04-02T05:06:18.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.000723+0000 osd.5 (osd.5) 352 : cluster 0 6.a scrub ok 2024-04-02T05:06:18.823 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:06:18.823 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-04-02T05:06:18.823 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-04-02T05:06:18.823 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-04-02T05:06:18.823 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:06:18.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:18 smithi098 bash[23790]: cluster 2024-04-02T05:06:17.051145+0000 osd.1 (osd.1) 539 : cluster 0 2.d scrub starts 2024-04-02T05:06:18.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:18 smithi098 bash[23790]: cluster 2024-04-02T05:06:17.053149+0000 osd.1 (osd.1) 540 : cluster 0 2.d scrub ok 2024-04-02T05:06:18.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:18 smithi098 bash[23790]: cluster 2024-04-02T05:06:17.284704+0000 osd.3 (osd.3) 549 : cluster 0 6.b scrub starts 2024-04-02T05:06:18.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:18 smithi098 bash[23790]: cluster 2024-04-02T05:06:17.287426+0000 osd.3 (osd.3) 550 : cluster 0 6.b scrub ok 2024-04-02T05:06:18.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:18 smithi098 bash[23790]: cluster 2024-04-02T05:06:17.998434+0000 osd.5 (osd.5) 351 : cluster 0 6.a scrub starts 2024-04-02T05:06:18.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:18 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.000723+0000 osd.5 (osd.5) 352 : cluster 0 6.a scrub ok 2024-04-02T05:06:19.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:06:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:06:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:06:19.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.069773+0000 osd.1 (osd.1) 541 : cluster 0 1.3 scrub starts 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.072563+0000 osd.1 (osd.1) 542 : cluster 0 1.3 scrub ok 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.248609+0000 osd.3 (osd.3) 551 : cluster 0 6.1f scrub starts 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.250610+0000 osd.3 (osd.3) 552 : cluster 0 6.1f scrub ok 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.292662+0000 osd.0 (osd.0) 505 : cluster 0 2.1f scrub starts 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.294419+0000 osd.0 (osd.0) 506 : cluster 0 2.1f scrub ok 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.456024+0000 mgr.y (mgr.14538) 865 : cluster 0 pgmap v1496: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.569858+0000 osd.4 (osd.4) 502 : cluster 0 3.1a deep-scrub starts 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.573684+0000 osd.4 (osd.4) 503 : cluster 0 3.1a deep-scrub ok 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.985562+0000 osd.5 (osd.5) 353 : cluster 0 3.1d scrub starts 2024-04-02T05:06:19.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[21528]: cluster 2024-04-02T05:06:18.990468+0000 osd.5 (osd.5) 354 : cluster 0 3.1d scrub ok 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.069773+0000 osd.1 (osd.1) 541 : cluster 0 1.3 scrub starts 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.072563+0000 osd.1 (osd.1) 542 : cluster 0 1.3 scrub ok 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.248609+0000 osd.3 (osd.3) 551 : cluster 0 6.1f scrub starts 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.250610+0000 osd.3 (osd.3) 552 : cluster 0 6.1f scrub ok 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.292662+0000 osd.0 (osd.0) 505 : cluster 0 2.1f scrub starts 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.294419+0000 osd.0 (osd.0) 506 : cluster 0 2.1f scrub ok 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.456024+0000 mgr.y (mgr.14538) 865 : cluster 0 pgmap v1496: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.569858+0000 osd.4 (osd.4) 502 : cluster 0 3.1a deep-scrub starts 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.573684+0000 osd.4 (osd.4) 503 : cluster 0 3.1a deep-scrub ok 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.985562+0000 osd.5 (osd.5) 353 : cluster 0 3.1d scrub starts 2024-04-02T05:06:19.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:19 smithi069 bash[28904]: cluster 2024-04-02T05:06:18.990468+0000 osd.5 (osd.5) 354 : cluster 0 3.1d scrub ok 2024-04-02T05:06:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.069773+0000 osd.1 (osd.1) 541 : cluster 0 1.3 scrub starts 2024-04-02T05:06:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.072563+0000 osd.1 (osd.1) 542 : cluster 0 1.3 scrub ok 2024-04-02T05:06:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.248609+0000 osd.3 (osd.3) 551 : cluster 0 6.1f scrub starts 2024-04-02T05:06:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.250610+0000 osd.3 (osd.3) 552 : cluster 0 6.1f scrub ok 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.292662+0000 osd.0 (osd.0) 505 : cluster 0 2.1f scrub starts 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.294419+0000 osd.0 (osd.0) 506 : cluster 0 2.1f scrub ok 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.456024+0000 mgr.y (mgr.14538) 865 : cluster 0 pgmap v1496: 184 pgs: 184 active+clean; 582 KiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.569858+0000 osd.4 (osd.4) 502 : cluster 0 3.1a deep-scrub starts 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.573684+0000 osd.4 (osd.4) 503 : cluster 0 3.1a deep-scrub ok 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.985562+0000 osd.5 (osd.5) 353 : cluster 0 3.1d scrub starts 2024-04-02T05:06:19.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:19 smithi098 bash[23790]: cluster 2024-04-02T05:06:18.990468+0000 osd.5 (osd.5) 354 : cluster 0 3.1d scrub ok 2024-04-02T05:06:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/152471392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc21010a560 msgr2=0x7fc21010a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- 172.21.15.69:0/152471392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc21010a560 0x7fc21010a940 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7fc200009920 tx=0x7fc200009c30 comp rx=0 tx=0).stop 2024-04-02T05:06:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/152471392 shutdown_connections 2024-04-02T05:06:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- 172.21.15.69:0/152471392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc210104e70 0x7fc210111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- 172.21.15.69:0/152471392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc21010a560 0x7fc21010a940 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- 172.21.15.69:0/152471392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2101044b0 0x7fc210104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/152471392 >> 172.21.15.69:0/152471392 conn(0x7fc2100fffa0 msgr2=0x7fc2101023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/152471392 shutdown_connections 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/152471392 wait complete. 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 librados: starting msgr at 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 librados: starting objecter 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 Processor -- start 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- start start 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 librados: setting wanted keys 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 librados: calling monclient init 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 0x7fc21019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc210104e70 0x7fc21019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc21010a560 0x7fc21019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc210113c60 con 0x7fc2101044b0 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc210113ae0 con 0x7fc210104e70 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc210113de0 con 0x7fc21010a560 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 0x7fc21019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc21010a560 0x7fc21019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 0x7fc21019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42988/0 (socket says 172.21.15.69:42988) 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 -- 172.21.15.69:0/2651483990 learned_addr learned my addr 172.21.15.69:0/2651483990 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20effd700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc210104e70 0x7fc21019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 -- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc21010a560 msgr2=0x7fc21019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc21010a560 0x7fc21019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 -- 172.21.15.69:0/2651483990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc210104e70 msgr2=0x7fc21019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc210104e70 0x7fc21019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 -- 172.21.15.69:0/2651483990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21019f2a0 con 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20ffff700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc21010a560 0x7fc21019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20f7fe700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 0x7fc21019a290 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7fc200009230 tx=0x7fc20000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20cff9700 10 client.?.objecter ms_handle_connect 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20cff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200027070 con 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21019f530 con 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc20000bb60 con 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200004930 con 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 10 client.33933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2101a6e10 con 0x7fc2101044b0 2024-04-02T05:06:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.161+0000 7fc215e42700 1 librados: init done 2024-04-02T05:06:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc210105bd0 con 0x7fc2101044b0 2024-04-02T05:06:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc20002a070 con 0x7fc2101044b0 2024-04-02T05:06:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1f806f5e0 0x7fc1f8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(950..950 src has 897..950) ==== 9015+0+0 (secure 0 0 0) 0x7fc20008f200 con 0x7fc2101044b0 2024-04-02T05:06:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 10 client.33933.objecter ms_dispatch 0x7fc210001550 osd_map(950..950 src has 897..950) 2024-04-02T05:06:20.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 3 client.33933.objecter handle_osd_map got epochs [950,950] > 0 2024-04-02T05:06:20.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 3 client.33933.objecter handle_osd_map decoding full epoch 950 2024-04-02T05:06:20.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20effd700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1f806f5e0 0x7fc1f8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:20.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 20 client.33933.objecter dump_active .. 0 homeless 2024-04-02T05:06:20.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20cff9700 10 client.33933.objecter ms_handle_connect 0x7fc1f806f5e0 2024-04-02T05:06:20.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.165+0000 7fc20effd700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1f806f5e0 0x7fc1f8071aa0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7fc204005950 tx=0x7fc2040058e0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:20.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.169+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc200013030 con 0x7fc2101044b0 2024-04-02T05:06:20.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:20.397+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7fc21019b5f0 con 0x7fc2101044b0 2024-04-02T05:06:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[21528]: audit 2024-04-02T05:06:19.612886+0000 mon.b (mon.1) 348 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[21528]: cluster 2024-04-02T05:06:19.980565+0000 osd.5 (osd.5) 355 : cluster 0 6.19 scrub starts 2024-04-02T05:06:20.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[21528]: cluster 2024-04-02T05:06:19.982879+0000 osd.5 (osd.5) 356 : cluster 0 6.19 scrub ok 2024-04-02T05:06:20.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[21528]: audit 2024-04-02T05:06:20.404070+0000 mon.a (mon.0) 4161 : audit 1 from='client.? 172.21.15.69:0/2651483990' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-02T05:06:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[28904]: audit 2024-04-02T05:06:19.612886+0000 mon.b (mon.1) 348 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[28904]: cluster 2024-04-02T05:06:19.980565+0000 osd.5 (osd.5) 355 : cluster 0 6.19 scrub starts 2024-04-02T05:06:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[28904]: cluster 2024-04-02T05:06:19.982879+0000 osd.5 (osd.5) 356 : cluster 0 6.19 scrub ok 2024-04-02T05:06:20.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:20 smithi069 bash[28904]: audit 2024-04-02T05:06:20.404070+0000 mon.a (mon.0) 4161 : audit 1 from='client.? 172.21.15.69:0/2651483990' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-02T05:06:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:20 smithi098 bash[23790]: audit 2024-04-02T05:06:19.612886+0000 mon.b (mon.1) 348 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:20 smithi098 bash[23790]: cluster 2024-04-02T05:06:19.980565+0000 osd.5 (osd.5) 355 : cluster 0 6.19 scrub starts 2024-04-02T05:06:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:20 smithi098 bash[23790]: cluster 2024-04-02T05:06:19.982879+0000 osd.5 (osd.5) 356 : cluster 0 6.19 scrub ok 2024-04-02T05:06:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:20 smithi098 bash[23790]: audit 2024-04-02T05:06:20.404070+0000 mon.a (mon.0) 4161 : audit 1 from='client.? 172.21.15.69:0/2651483990' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-02T05:06:21.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.213+0000 7fc20cff9700 1 -- 172.21.15.69:0/2651483990 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v951) ==== 133+0+0 (secure 0 0 0) 0x7fc20005c1d0 con 0x7fc2101044b0 2024-04-02T05:06:21.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 10 librados: watch_flush enter 2024-04-02T05:06:21.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 10 librados: watch_flush exit 2024-04-02T05:06:21.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 20 client.33933.objecter shutdown clearing up homeless session... 2024-04-02T05:06:21.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 10 client.33933.objecter successfully canceled tick 2024-04-02T05:06:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1f806f5e0 msgr2=0x7fc1f8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1f806f5e0 0x7fc1f8071aa0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7fc204005950 tx=0x7fc2040058e0 comp rx=0 tx=0).stop 2024-04-02T05:06:21.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 msgr2=0x7fc21019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:21.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.217+0000 7fc215e42700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 0x7fc21019a290 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7fc200009230 tx=0x7fc20000fa70 comp rx=0 tx=0).stop 2024-04-02T05:06:21.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.221+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 shutdown_connections 2024-04-02T05:06:21.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.221+0000 7fc215e42700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1f806f5e0 0x7fc1f8071aa0 unknown :-1 s=CLOSED pgs=4045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:21.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.221+0000 7fc215e42700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc21010a560 0x7fc21019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:21.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.221+0000 7fc215e42700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2101044b0 0x7fc21019a290 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:21.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.221+0000 7fc215e42700 1 --2- 172.21.15.69:0/2651483990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc210104e70 0x7fc21019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:21.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.221+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 >> 172.21.15.69:0/2651483990 conn(0x7fc2100fffa0 msgr2=0x7fc210108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:21.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.225+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 shutdown_connections 2024-04-02T05:06:21.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.225+0000 7fc215e42700 1 -- 172.21.15.69:0/2651483990 wait complete. 2024-04-02T05:06:21.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:21.225+0000 7fc215e42700 1 librados: shutdown 2024-04-02T05:06:21.230 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-02T05:06:21.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: cluster 2024-04-02T05:06:20.003197+0000 osd.6 (osd.6) 473 : cluster 0 10.a scrub starts 2024-04-02T05:06:21.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: cluster 2024-04-02T05:06:20.005868+0000 osd.6 (osd.6) 474 : cluster 0 10.a scrub ok 2024-04-02T05:06:21.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: cluster 2024-04-02T05:06:20.456972+0000 mgr.y (mgr.14538) 866 : cluster 0 pgmap v1497: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: cluster 2024-04-02T05:06:20.493954+0000 osd.7 (osd.7) 341 : cluster 0 3.f deep-scrub starts 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: cluster 2024-04-02T05:06:20.505170+0000 osd.7 (osd.7) 342 : cluster 0 3.f deep-scrub ok 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: audit 2024-04-02T05:06:21.216334+0000 mon.a (mon.0) 4162 : audit 1 from='client.? 172.21.15.69:0/2651483990' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.223192+0000 mon.a (mon.0) 4163 : cluster 0 osdmap e951: 8 total, 8 up, 8 in 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: cluster 2024-04-02T05:06:20.003197+0000 osd.6 (osd.6) 473 : cluster 0 10.a scrub starts 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: cluster 2024-04-02T05:06:20.005868+0000 osd.6 (osd.6) 474 : cluster 0 10.a scrub ok 2024-04-02T05:06:21.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: cluster 2024-04-02T05:06:20.456972+0000 mgr.y (mgr.14538) 866 : cluster 0 pgmap v1497: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:06:21.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: cluster 2024-04-02T05:06:20.493954+0000 osd.7 (osd.7) 341 : cluster 0 3.f deep-scrub starts 2024-04-02T05:06:21.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: cluster 2024-04-02T05:06:20.505170+0000 osd.7 (osd.7) 342 : cluster 0 3.f deep-scrub ok 2024-04-02T05:06:21.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: audit 2024-04-02T05:06:21.216334+0000 mon.a (mon.0) 4162 : audit 1 from='client.? 172.21.15.69:0/2651483990' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-02T05:06:21.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:21 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.223192+0000 mon.a (mon.0) 4163 : cluster 0 osdmap e951: 8 total, 8 up, 8 in 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: cluster 2024-04-02T05:06:20.003197+0000 osd.6 (osd.6) 473 : cluster 0 10.a scrub starts 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: cluster 2024-04-02T05:06:20.005868+0000 osd.6 (osd.6) 474 : cluster 0 10.a scrub ok 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: cluster 2024-04-02T05:06:20.456972+0000 mgr.y (mgr.14538) 866 : cluster 0 pgmap v1497: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: cluster 2024-04-02T05:06:20.493954+0000 osd.7 (osd.7) 341 : cluster 0 3.f deep-scrub starts 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: cluster 2024-04-02T05:06:20.505170+0000 osd.7 (osd.7) 342 : cluster 0 3.f deep-scrub ok 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: audit 2024-04-02T05:06:21.216334+0000 mon.a (mon.0) 4162 : audit 1 from='client.? 172.21.15.69:0/2651483990' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-02T05:06:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:21 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.223192+0000 mon.a (mon.0) 4163 : cluster 0 osdmap e951: 8 total, 8 up, 8 in 2024-04-02T05:06:22.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 6 1 2024-04-02T05:06:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.274485+0000 osd.0 (osd.0) 507 : cluster 0 6.16 scrub starts 2024-04-02T05:06:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.275921+0000 osd.0 (osd.0) 508 : cluster 0 6.16 scrub ok 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.481332+0000 osd.7 (osd.7) 343 : cluster 0 3.e scrub starts 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.486516+0000 osd.7 (osd.7) 344 : cluster 0 3.e scrub ok 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.969867+0000 osd.5 (osd.5) 357 : cluster 0 3.1c scrub starts 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[21528]: cluster 2024-04-02T05:06:21.974529+0000 osd.5 (osd.5) 358 : cluster 0 3.1c scrub ok 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.274485+0000 osd.0 (osd.0) 507 : cluster 0 6.16 scrub starts 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.275921+0000 osd.0 (osd.0) 508 : cluster 0 6.16 scrub ok 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.481332+0000 osd.7 (osd.7) 343 : cluster 0 3.e scrub starts 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.486516+0000 osd.7 (osd.7) 344 : cluster 0 3.e scrub ok 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.969867+0000 osd.5 (osd.5) 357 : cluster 0 3.1c scrub starts 2024-04-02T05:06:22.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:22 smithi069 bash[28904]: cluster 2024-04-02T05:06:21.974529+0000 osd.5 (osd.5) 358 : cluster 0 3.1c scrub ok 2024-04-02T05:06:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:22 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.274485+0000 osd.0 (osd.0) 507 : cluster 0 6.16 scrub starts 2024-04-02T05:06:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:22 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.275921+0000 osd.0 (osd.0) 508 : cluster 0 6.16 scrub ok 2024-04-02T05:06:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:22 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.481332+0000 osd.7 (osd.7) 343 : cluster 0 3.e scrub starts 2024-04-02T05:06:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:22 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.486516+0000 osd.7 (osd.7) 344 : cluster 0 3.e scrub ok 2024-04-02T05:06:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:22 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.969867+0000 osd.5 (osd.5) 357 : cluster 0 3.1c scrub starts 2024-04-02T05:06:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:22 smithi098 bash[23790]: cluster 2024-04-02T05:06:21.974529+0000 osd.5 (osd.5) 358 : cluster 0 3.1c scrub ok 2024-04-02T05:06:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.028220+0000 osd.6 (osd.6) 475 : cluster 0 2.17 scrub starts 2024-04-02T05:06:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.030496+0000 osd.6 (osd.6) 476 : cluster 0 2.17 scrub ok 2024-04-02T05:06:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.235301+0000 osd.0 (osd.0) 509 : cluster 0 3.1 scrub starts 2024-04-02T05:06:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.239896+0000 osd.0 (osd.0) 510 : cluster 0 3.1 scrub ok 2024-04-02T05:06:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.283092+0000 osd.3 (osd.3) 553 : cluster 0 6.11 scrub starts 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.284652+0000 osd.3 (osd.3) 554 : cluster 0 6.11 scrub ok 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.457774+0000 mgr.y (mgr.14538) 867 : cluster 0 pgmap v1499: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.480122+0000 osd.4 (osd.4) 504 : cluster 0 4.1 scrub starts 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[21528]: cluster 2024-04-02T05:06:22.482509+0000 osd.4 (osd.4) 505 : cluster 0 4.1 scrub ok 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.028220+0000 osd.6 (osd.6) 475 : cluster 0 2.17 scrub starts 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.030496+0000 osd.6 (osd.6) 476 : cluster 0 2.17 scrub ok 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.235301+0000 osd.0 (osd.0) 509 : cluster 0 3.1 scrub starts 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.239896+0000 osd.0 (osd.0) 510 : cluster 0 3.1 scrub ok 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.283092+0000 osd.3 (osd.3) 553 : cluster 0 6.11 scrub starts 2024-04-02T05:06:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.284652+0000 osd.3 (osd.3) 554 : cluster 0 6.11 scrub ok 2024-04-02T05:06:23.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.457774+0000 mgr.y (mgr.14538) 867 : cluster 0 pgmap v1499: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:23.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.480122+0000 osd.4 (osd.4) 504 : cluster 0 4.1 scrub starts 2024-04-02T05:06:23.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:23 smithi069 bash[28904]: cluster 2024-04-02T05:06:22.482509+0000 osd.4 (osd.4) 505 : cluster 0 4.1 scrub ok 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.028220+0000 osd.6 (osd.6) 475 : cluster 0 2.17 scrub starts 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.030496+0000 osd.6 (osd.6) 476 : cluster 0 2.17 scrub ok 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.235301+0000 osd.0 (osd.0) 509 : cluster 0 3.1 scrub starts 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.239896+0000 osd.0 (osd.0) 510 : cluster 0 3.1 scrub ok 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.283092+0000 osd.3 (osd.3) 553 : cluster 0 6.11 scrub starts 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.284652+0000 osd.3 (osd.3) 554 : cluster 0 6.11 scrub ok 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.457774+0000 mgr.y (mgr.14538) 867 : cluster 0 pgmap v1499: 184 pgs: 184 active+clean; 582 KiB data, 759 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.480122+0000 osd.4 (osd.4) 504 : cluster 0 4.1 scrub starts 2024-04-02T05:06:23.938 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[23790]: cluster 2024-04-02T05:06:22.482509+0000 osd.4 (osd.4) 505 : cluster 0 4.1 scrub ok 2024-04-02T05:06:24.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:06:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:06:24.343 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T05:06:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:24 smithi098 bash[23790]: cluster 2024-04-02T05:06:23.041540+0000 osd.1 (osd.1) 543 : cluster 0 6.4 scrub starts 2024-04-02T05:06:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:24 smithi098 bash[23790]: cluster 2024-04-02T05:06:23.043944+0000 osd.1 (osd.1) 544 : cluster 0 6.4 scrub ok 2024-04-02T05:06:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:24 smithi098 bash[23790]: cluster 2024-04-02T05:06:24.031780+0000 osd.5 (osd.5) 359 : cluster 0 2.2 scrub starts 2024-04-02T05:06:24.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:24 smithi098 bash[23790]: cluster 2024-04-02T05:06:24.034170+0000 osd.5 (osd.5) 360 : cluster 0 2.2 scrub ok 2024-04-02T05:06:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[21528]: cluster 2024-04-02T05:06:23.041540+0000 osd.1 (osd.1) 543 : cluster 0 6.4 scrub starts 2024-04-02T05:06:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[21528]: cluster 2024-04-02T05:06:23.043944+0000 osd.1 (osd.1) 544 : cluster 0 6.4 scrub ok 2024-04-02T05:06:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[21528]: cluster 2024-04-02T05:06:24.031780+0000 osd.5 (osd.5) 359 : cluster 0 2.2 scrub starts 2024-04-02T05:06:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[21528]: cluster 2024-04-02T05:06:24.034170+0000 osd.5 (osd.5) 360 : cluster 0 2.2 scrub ok 2024-04-02T05:06:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[28904]: cluster 2024-04-02T05:06:23.041540+0000 osd.1 (osd.1) 543 : cluster 0 6.4 scrub starts 2024-04-02T05:06:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[28904]: cluster 2024-04-02T05:06:23.043944+0000 osd.1 (osd.1) 544 : cluster 0 6.4 scrub ok 2024-04-02T05:06:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[28904]: cluster 2024-04-02T05:06:24.031780+0000 osd.5 (osd.5) 359 : cluster 0 2.2 scrub starts 2024-04-02T05:06:24.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:24 smithi069 bash[28904]: cluster 2024-04-02T05:06:24.034170+0000 osd.5 (osd.5) 360 : cluster 0 2.2 scrub ok 2024-04-02T05:06:25.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:25 smithi098 bash[23790]: audit 2024-04-02T05:06:23.935185+0000 mgr.y (mgr.14538) 868 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:25.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:25 smithi098 bash[23790]: cluster 2024-04-02T05:06:24.459741+0000 mgr.y (mgr.14538) 869 : cluster 0 pgmap v1500: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:25.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:25 smithi069 bash[21528]: audit 2024-04-02T05:06:23.935185+0000 mgr.y (mgr.14538) 868 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:25.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:25 smithi069 bash[21528]: cluster 2024-04-02T05:06:24.459741+0000 mgr.y (mgr.14538) 869 : cluster 0 pgmap v1500: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:25 smithi069 bash[28904]: audit 2024-04-02T05:06:23.935185+0000 mgr.y (mgr.14538) 868 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:25.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:25 smithi069 bash[28904]: cluster 2024-04-02T05:06:24.459741+0000 mgr.y (mgr.14538) 869 : cluster 0 pgmap v1500: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:26.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:24.960451+0000 osd.1 (osd.1) 545 : cluster 0 10.19 scrub starts 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:24.965123+0000 osd.1 (osd.1) 546 : cluster 0 10.19 scrub ok 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.182279+0000 osd.0 (osd.0) 511 : cluster 0 6.9 scrub starts 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.184698+0000 osd.0 (osd.0) 512 : cluster 0 6.9 scrub ok 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.335914+0000 osd.3 (osd.3) 555 : cluster 0 3.7 scrub starts 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.340736+0000 osd.3 (osd.3) 556 : cluster 0 3.7 scrub ok 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.770708+0000 osd.2 (osd.2) 418 : cluster 0 6.14 deep-scrub starts 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.773167+0000 osd.2 (osd.2) 419 : cluster 0 6.14 deep-scrub ok 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:26.049210+0000 osd.5 (osd.5) 361 : cluster 0 6.7 scrub starts 2024-04-02T05:06:26.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:26 smithi098 bash[23790]: cluster 2024-04-02T05:06:26.051403+0000 osd.5 (osd.5) 362 : cluster 0 6.7 scrub ok 2024-04-02T05:06:26.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:24.960451+0000 osd.1 (osd.1) 545 : cluster 0 10.19 scrub starts 2024-04-02T05:06:26.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:24.965123+0000 osd.1 (osd.1) 546 : cluster 0 10.19 scrub ok 2024-04-02T05:06:26.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.182279+0000 osd.0 (osd.0) 511 : cluster 0 6.9 scrub starts 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.184698+0000 osd.0 (osd.0) 512 : cluster 0 6.9 scrub ok 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.335914+0000 osd.3 (osd.3) 555 : cluster 0 3.7 scrub starts 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.340736+0000 osd.3 (osd.3) 556 : cluster 0 3.7 scrub ok 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.770708+0000 osd.2 (osd.2) 418 : cluster 0 6.14 deep-scrub starts 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.773167+0000 osd.2 (osd.2) 419 : cluster 0 6.14 deep-scrub ok 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:26.049210+0000 osd.5 (osd.5) 361 : cluster 0 6.7 scrub starts 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[21528]: cluster 2024-04-02T05:06:26.051403+0000 osd.5 (osd.5) 362 : cluster 0 6.7 scrub ok 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:24.960451+0000 osd.1 (osd.1) 545 : cluster 0 10.19 scrub starts 2024-04-02T05:06:26.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:24.965123+0000 osd.1 (osd.1) 546 : cluster 0 10.19 scrub ok 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.182279+0000 osd.0 (osd.0) 511 : cluster 0 6.9 scrub starts 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.184698+0000 osd.0 (osd.0) 512 : cluster 0 6.9 scrub ok 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.335914+0000 osd.3 (osd.3) 555 : cluster 0 3.7 scrub starts 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.340736+0000 osd.3 (osd.3) 556 : cluster 0 3.7 scrub ok 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.770708+0000 osd.2 (osd.2) 418 : cluster 0 6.14 deep-scrub starts 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.773167+0000 osd.2 (osd.2) 419 : cluster 0 6.14 deep-scrub ok 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:26.049210+0000 osd.5 (osd.5) 361 : cluster 0 6.7 scrub starts 2024-04-02T05:06:26.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:26 smithi069 bash[28904]: cluster 2024-04-02T05:06:26.051403+0000 osd.5 (osd.5) 362 : cluster 0 6.7 scrub ok 2024-04-02T05:06:26.883 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:06:27.346 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-02T05:06:27.346 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-04-02T05:06:27.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-04-02T05:06:27.346 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-04-02T05:06:27.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:92 102.19 102:9e3d24f0:test-rados-api-smithi069-72349-82::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:25.994122+0000", 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 541.34926181399999, 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014748196, 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994122+0000", 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994122+0000", 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994124+0000", 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.48e-06 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994137+0000", 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2853e-05 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994166+0000", 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8206000000000001e-05 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994184+0000", 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8482999999999999e-05 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994273+0000", 2024-04-02T05:06:27.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8955999999999993e-05 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.994281+0000", 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5120000000000008e-06 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000734+0000", 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064523369999999998 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000854+0000", 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011993300000000001 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.001046+0000", 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000192413 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006085+0000", 2024-04-02T05:06:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050385450000000002 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006149+0000", 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4776000000000005e-05 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008780+0000", 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026304789999999998 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008825+0000", 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4566e-05 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008870+0000", 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5657e-05 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:97 102.19 102:9be20e43:test-rados-api-smithi069-72349-82::b74d4df232b43241f65ea39108ac0a45561a59cf:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:25.999239+0000", 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 541.34414475400001, 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010862856000000001, 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999239+0000", 2024-04-02T05:06:27.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999239+0000", 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999242+0000", 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7659999999999999e-06 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999257+0000", 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5497000000000001e-05 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999292+0000", 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5036000000000001e-05 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999317+0000", 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5069999999999999e-05 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.005436+0000", 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061181559999999996 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.005584+0000", 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014880500000000001 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4,6", 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.005769+0000", 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000184861 2024-04-02T05:06:27.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008764+0000", 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029949669999999999 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008882+0000", 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117718 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.009832+0000", 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094983899999999996 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.009988+0000", 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156033 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.010102+0000", 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114108 2024-04-02T05:06:27.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:98 102.1 102:84b9727a:test-rados-api-smithi069-72349-82::060e72eaff4fb3562782e1de4fe785e9648c08af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:25.999354+0000", 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 541.344030223, 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012512808, 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999354+0000", 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999354+0000", 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999355+0000", 2024-04-02T05:06:27.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5069999999999999e-06 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999364+0000", 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3859999999999996e-06 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999388+0000", 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3589999999999999e-05 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:25.999406+0000", 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8312999999999999e-05 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006131+0000", 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067249609999999998 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006264+0000", 2024-04-02T05:06:27.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000132965 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,6", 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006476+0000", 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000211931 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008887+0000", 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024108649999999999 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.009800+0000", 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00091312000000000001 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011752+0000", 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001951997 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011823+0000", 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1167999999999994e-05 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011866+0000", 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3004999999999999e-05 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.1.13:100 102.1 102:83dfbecf:test-rados-api-smithi069-72349-82::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e565)", 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:57:26.000007+0000", 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 541.34337659400001, 2024-04-02T05:06:27.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012134123, 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000007+0000", 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000007+0000", 2024-04-02T05:06:27.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000009+0000", 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3170000000000001e-06 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000020+0000", 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1249e-05 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000057+0000", 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6869000000000002e-05 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.000068+0000", 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1333e-05 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006827+0000", 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067592370000000004 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.006982+0000", 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000155265 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,6", 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.007206+0000", 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022388400000000001 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.008891+0000", 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016850789999999999 2024-04-02T05:06:27.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.011921+0000", 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030295209999999999 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.012052+0000", 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001315 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.012103+0000", 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0899000000000002e-05 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:57:26.012141+0000", 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7969999999999997e-05 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.2.19:133 310.1a 310:5d12752b:test-rados-api-smithi069-72349-91::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e601)", 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:58:02.126813+0000", 2024-04-02T05:06:27.372 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 505.21657086499999, 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094835509999999998, 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126813+0000", 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126813+0000", 2024-04-02T05:06:27.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126815+0000", 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6160000000000001e-06 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126821+0000", 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.144e-06 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126832+0000", 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1532000000000001e-05 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126842+0000", 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.363e-06 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126907+0000", 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5173999999999996e-05 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.126915+0000", 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8169999999999999e-06 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.133860+0000", 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0069455849999999998 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.133954+0000", 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3366e-05 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.134098+0000", 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.135599+0000", 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001501198 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.135925+0000", 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032581400000000001 2024-04-02T05:06:27.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.136219+0000", 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029453299999999998 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.136269+0000", 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9823000000000001e-05 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:58:02.136296+0000", 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7518e-05 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(3.7,QREQUEST,e:647,from:3,reservation_nonce:3)", 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:05.930835+0000", 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 441.412548448, 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012097088000000001, 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930835+0000", 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930835+0000", 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930837+0000", 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0030000000000001e-06 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930847+0000", 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.005e-05 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930877+0000", 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9856e-05 2024-04-02T05:06:27.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.930893+0000", 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6192999999999999e-05 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.931053+0000", 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015919099999999999 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.931081+0000", 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7883000000000001e-05 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.942828+0000", 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011747857 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.942860+0000", 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1136000000000001e-05 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:05.942932+0000", 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2918999999999998e-05 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:2 323.1s0 323:91bcb6b0:test-rados-api-smithi069-72349-113::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T04:59:58.049814+0000", 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 389.29357014099998, 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030785092999999999, 2024-04-02T05:06:27.384 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049814+0000", 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049814+0000", 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049816+0000", 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1670000000000002e-06 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049827+0000", 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0856000000000001e-05 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049849+0000", 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.198e-05 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.049873+0000", 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4510000000000001e-05 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.050076+0000", 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020266699999999999 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.050095+0000", 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9403e-05 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.071101+0000", 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021005280000000001 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.073818+0000", 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027173950000000001 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.074995+0000", 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001177261 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.076333+0000", 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001338234 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.080443+0000", 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004109522 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T04:59:58.080599+0000", 2024-04-02T05:06:27.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000155818 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:475 328.1b 328:df8c41d3:test-rados-api-smithi069-72349-121::foo:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:00:50.055035+0000", 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 337.28834840000002, 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028280998000000002, 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 475 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055035+0000", 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055035+0000", 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055038+0000", 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0340000000000001e-06 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055053+0000", 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4163e-05 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055092+0000", 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9128999999999998e-05 2024-04-02T05:06:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055111+0000", 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9680000000000001e-05 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055283+0000", 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017113499999999999 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.055296+0000", 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3883000000000001e-05 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.068291+0000", 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012994309000000001 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.068851+0000", 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055996599999999996 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.073683+0000", 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048323730000000001 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.073948+0000", 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026499599999999998 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.074377+0000", 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042873800000000001 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.075272+0000", 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089567200000000003 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.081520+0000", 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062475730000000002 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.083031+0000", 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00151052 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.083194+0000", 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016390799999999999 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:00:50.083316+0000", 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121919 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(2.12,QREQUEST,e:751,from:3,reservation_nonce:3)", 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:01.656229+0000", 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 325.68715460099997, 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01403385, 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656229+0000", 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656229+0000", 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656232+0000", 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7829999999999999e-06 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656248+0000", 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6062999999999999e-05 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656285+0000", 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6579999999999999e-05 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656303+0000", 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8185999999999998e-05 2024-04-02T05:06:27.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656443+0000", 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014039799999999999 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.656455+0000", 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.221e-05 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.670100+0000", 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013644269000000001 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.670145+0000", 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5291999999999999e-05 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:01.670263+0000", 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118069 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:1243 332.1e 332:78000000::::head [pgnls start_epoch 778] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:01:30.768520+0000", 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 296.57486331799998, 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095984439999999994, 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1243 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.404 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768520+0000", 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768520+0000", 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768524+0000", 2024-04-02T05:06:27.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0230000000000001e-06 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768540+0000", 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6286e-05 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768579+0000", 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9515999999999999e-05 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768598+0000", 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8474e-05 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768748+0000", 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015029099999999999 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.768759+0000", 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0943e-05 2024-04-02T05:06:27.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.777709+0000", 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089495300000000007 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.777820+0000", 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011117500000000001 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:01:30.778119+0000", 2024-04-02T05:06:27.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029920599999999999 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDECSubOpWrite(323.4s1 813/698 ECSubWrite(tid=1285, reqid=osd.3.29:94, at_version=813'10, trim_to=0'0, roll_forward_to=784'9))", 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:02:05.145129+0000", 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 262.19825445700002, 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017199283999999999, 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.145129+0000", 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.145129+0000", 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.145130+0000", 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2399999999999998e-07 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.150105+0000", 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049751689999999998 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.150140+0000", 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5138999999999997e-05 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.150164+0000", 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3689999999999998e-05 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.150303+0000", 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013940900000000001 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.150318+0000", 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4642e-05 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.162287+0000", 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011968837 2024-04-02T05:06:27.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:02:05.162329+0000", 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1873999999999999e-05 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:2765 323.1s0 323:9c215b1d:test-rados-api-smithi069-72349-147::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:03:13.502007+0000", 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 193.841376962, 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030505797000000001, 2024-04-02T05:06:27.413 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2765 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502007+0000", 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502007+0000", 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502009+0000", 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3609999999999999e-06 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502021+0000", 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.137e-05 2024-04-02T05:06:27.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502042+0000", 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1747e-05 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502058+0000", 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6030999999999999e-05 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502152+0000", 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3978e-05 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502313+0000", 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016066600000000001 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_started", 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.502864+0000", 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055124799999999995 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_committed", 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.504781+0000", 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019170420000000001 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.532428+0000", 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027646411999999999 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:13.532513+0000", 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4942000000000001e-05 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.418 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(osd.0.8:84 323.1s0 323:9c215b1d:test-rados-api-smithi069-72349-147::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e870)", 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:03:16.136953+0000", 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 191.20643082000001, 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011485510000000001, 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.136953+0000", 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.136953+0000", 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.136954+0000", 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0300000000000001e-06 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.136960+0000", 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2940000000000002e-06 2024-04-02T05:06:27.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.136972+0000", 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1579999999999999e-05 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.136981+0000", 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2329999999999995e-06 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.137231+0000", 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024997700000000002 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.137248+0000", 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6693000000000001e-05 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.143344+0000", 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006096232 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.143606+0000", 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026172400000000002 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.148438+0000", 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048327470000000001 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.32028.0:2770 341.e 341:73ccdb4e:test-rados-api-smithi069-72349-147::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e870)", 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:03:16.154058+0000", 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 191.189325459, 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010648832, 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.32028", 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.69:0/2099179437", 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2770 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154058+0000", 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154058+0000", 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154061+0000", 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.553e-06 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154073+0000", 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.241e-05 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154101+0000", 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8116e-05 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154125+0000", 2024-04-02T05:06:27.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3269999999999999e-05 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154246+0000", 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121155 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for blocked object", 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.154585+0000", 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033897000000000002 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.159823+0000", 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052382990000000001 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.160104+0000", 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028058299999999999 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.161585+0000", 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001481336 2024-04-02T05:06:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.161724+0000", 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139455 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:03:16.164707+0000", 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029826850000000001 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.1b,QREQUEST,e:909,from:3,reservation_nonce:5)", 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:22.556339+0000", 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 124.78704457000001, 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067634380000000001, 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556339+0000", 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556339+0000", 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556342+0000", 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3769999999999999e-06 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556357+0000", 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5302999999999999e-05 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556398+0000", 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0609999999999999e-05 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556420+0000", 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2243e-05 2024-04-02T05:06:27.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556579+0000", 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015883499999999999 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.556595+0000", 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6872999999999999e-05 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.562923+0000", 2024-04-02T05:06:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006327765 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.562971+0000", 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7573e-05 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:22.563103+0000", 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013185900000000001 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(osd.7.0:389 3.11 e913/394 ondisk, result = 0)", 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:52.516622+0000", 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.826762063000004, 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003896067, 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.516622+0000", 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.516622+0000", 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.516623+0000", 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6029999999999999e-06 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.516636+0000", 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2354000000000001e-05 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.516664+0000", 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8036000000000001e-05 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.516683+0000", 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9233e-05 2024-04-02T05:06:27.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.520447+0000", 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037638480000000002 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.520473+0000", 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6406000000000001e-05 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.520518+0000", 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4586999999999998e-05 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(osd.7.0:389 3.11 e913/394 ondisk, result = 0)", 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:52.517763+0000", 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.825620723, 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031870790000000002, 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.517763+0000", 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.517763+0000", 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.517765+0000", 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8339999999999999e-06 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.517777+0000", 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1753000000000001e-05 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.517814+0000", 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7333000000000003e-05 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.517834+0000", 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9993e-05 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.520612+0000", 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027779660000000002 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.520639+0000", 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6945999999999999e-05 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.520950+0000", 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031125400000000001 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.443 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(osd.7.0:391 3.11 e913/394 ondisk, result = 0)", 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:04:52.518406+0000", 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.824977532999995, 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032536570000000001, 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.518406+0000", 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.518406+0000", 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.518407+0000", 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.057e-06 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.518414+0000", 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6030000000000001e-06 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.518432+0000", 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8442999999999999e-05 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.518447+0000", 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4253000000000001e-05 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.521541+0000", 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030948059999999999 2024-04-02T05:06:27.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.521551+0000", 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8630000000000003e-06 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:04:52.521660+0000", 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108632 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.448 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(10.10,QREQUEST,e:948,from:4,reservation_nonce:3)", 2024-04-02T05:06:27.448 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:06:06.597127+0000", 2024-04-02T05:06:27.448 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.746257105000002, 2024-04-02T05:06:27.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030687129999999998, 2024-04-02T05:06:27.448 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.449 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597127+0000", 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597127+0000", 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597129+0000", 2024-04-02T05:06:27.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9139999999999998e-06 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597140+0000", 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0984e-05 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597168+0000", 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8668000000000001e-05 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597180+0000", 2024-04-02T05:06:27.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1912e-05 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597286+0000", 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010631 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.597299+0000", 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2658e-05 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.600103+0000", 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028036070000000001 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.600122+0000", 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9701e-05 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:06.600195+0000", 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2959000000000004e-05 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "MOSDScrubReserve(6.16,QREQUEST,e:951,from:0,reservation_nonce:5)", 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-04-02T05:06:21.268448+0000", 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.0749354760000003, 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062408380000000003, 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268448+0000", 2024-04-02T05:06:27.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268448+0000", 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268451+0000", 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6299999999999998e-06 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268466+0000", 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4773999999999999e-05 2024-04-02T05:06:27.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268500+0000", 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4165999999999998e-05 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268517+0000", 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7447000000000001e-05 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268687+0000", 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000169185 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.268706+0000", 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9026e-05 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.274589+0000", 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058831040000000001 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.274620+0000", 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0877000000000003e-05 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-02T05:06:27.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-04-02T05:06:21.274689+0000", 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9628999999999994e-05 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-02T05:06:27.458 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-02T05:06:27.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.969380+0000 osd.6 (osd.6) 477 : cluster 0 1.f scrub starts 2024-04-02T05:06:27.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.971713+0000 osd.6 (osd.6) 478 : cluster 0 1.f scrub ok 2024-04-02T05:06:27.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.977767+0000 osd.1 (osd.1) 547 : cluster 0 2.15 scrub starts 2024-04-02T05:06:27.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[21528]: cluster 2024-04-02T05:06:25.980331+0000 osd.1 (osd.1) 548 : cluster 0 2.15 scrub ok 2024-04-02T05:06:27.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[21528]: cluster 2024-04-02T05:06:26.460515+0000 mgr.y (mgr.14538) 870 : cluster 0 pgmap v1501: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:27.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.969380+0000 osd.6 (osd.6) 477 : cluster 0 1.f scrub starts 2024-04-02T05:06:27.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.971713+0000 osd.6 (osd.6) 478 : cluster 0 1.f scrub ok 2024-04-02T05:06:27.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.977767+0000 osd.1 (osd.1) 547 : cluster 0 2.15 scrub starts 2024-04-02T05:06:27.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[28904]: cluster 2024-04-02T05:06:25.980331+0000 osd.1 (osd.1) 548 : cluster 0 2.15 scrub ok 2024-04-02T05:06:27.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:27 smithi069 bash[28904]: cluster 2024-04-02T05:06:26.460515+0000 mgr.y (mgr.14538) 870 : cluster 0 pgmap v1501: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:27 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.969380+0000 osd.6 (osd.6) 477 : cluster 0 1.f scrub starts 2024-04-02T05:06:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:27 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.971713+0000 osd.6 (osd.6) 478 : cluster 0 1.f scrub ok 2024-04-02T05:06:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:27 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.977767+0000 osd.1 (osd.1) 547 : cluster 0 2.15 scrub starts 2024-04-02T05:06:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:27 smithi098 bash[23790]: cluster 2024-04-02T05:06:25.980331+0000 osd.1 (osd.1) 548 : cluster 0 2.15 scrub ok 2024-04-02T05:06:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:27 smithi098 bash[23790]: cluster 2024-04-02T05:06:26.460515+0000 mgr.y (mgr.14538) 870 : cluster 0 pgmap v1501: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:28.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:06:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:06:28.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.011516+0000 osd.1 (osd.1) 549 : cluster 0 4.2 scrub starts 2024-04-02T05:06:28.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.013960+0000 osd.1 (osd.1) 550 : cluster 0 4.2 scrub ok 2024-04-02T05:06:28.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.451140+0000 osd.4 (osd.4) 506 : cluster 0 4.e scrub starts 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.452996+0000 osd.4 (osd.4) 507 : cluster 0 4.e scrub ok 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.798178+0000 osd.2 (osd.2) 420 : cluster 0 6.f scrub starts 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.800146+0000 osd.2 (osd.2) 421 : cluster 0 6.f scrub ok 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.011516+0000 osd.1 (osd.1) 549 : cluster 0 4.2 scrub starts 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.013960+0000 osd.1 (osd.1) 550 : cluster 0 4.2 scrub ok 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.451140+0000 osd.4 (osd.4) 506 : cluster 0 4.e scrub starts 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.452996+0000 osd.4 (osd.4) 507 : cluster 0 4.e scrub ok 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.798178+0000 osd.2 (osd.2) 420 : cluster 0 6.f scrub starts 2024-04-02T05:06:28.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:28 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.800146+0000 osd.2 (osd.2) 421 : cluster 0 6.f scrub ok 2024-04-02T05:06:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:28 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.011516+0000 osd.1 (osd.1) 549 : cluster 0 4.2 scrub starts 2024-04-02T05:06:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:28 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.013960+0000 osd.1 (osd.1) 550 : cluster 0 4.2 scrub ok 2024-04-02T05:06:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:28 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.451140+0000 osd.4 (osd.4) 506 : cluster 0 4.e scrub starts 2024-04-02T05:06:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:28 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.452996+0000 osd.4 (osd.4) 507 : cluster 0 4.e scrub ok 2024-04-02T05:06:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:28 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.798178+0000 osd.2 (osd.2) 420 : cluster 0 6.f scrub starts 2024-04-02T05:06:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:28 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.800146+0000 osd.2 (osd.2) 421 : cluster 0 6.f scrub ok 2024-04-02T05:06:29.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.937998+0000 osd.6 (osd.6) 479 : cluster 0 4.a scrub starts 2024-04-02T05:06:29.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:27.939954+0000 osd.6 (osd.6) 480 : cluster 0 4.a scrub ok 2024-04-02T05:06:29.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:28.403528+0000 osd.3 (osd.3) 557 : cluster 0 2.1e scrub starts 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:28.406018+0000 osd.3 (osd.3) 558 : cluster 0 2.1e scrub ok 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:28.436646+0000 osd.4 (osd.4) 508 : cluster 0 10.17 scrub starts 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:28.443392+0000 osd.4 (osd.4) 509 : cluster 0 10.17 scrub ok 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: cluster 2024-04-02T05:06:28.462333+0000 mgr.y (mgr.14538) 871 : cluster 0 pgmap v1502: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[21528]: audit 2024-04-02T05:06:29.133373+0000 mon.b (mon.1) 349 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.937998+0000 osd.6 (osd.6) 479 : cluster 0 4.a scrub starts 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:27.939954+0000 osd.6 (osd.6) 480 : cluster 0 4.a scrub ok 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:28.403528+0000 osd.3 (osd.3) 557 : cluster 0 2.1e scrub starts 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:28.406018+0000 osd.3 (osd.3) 558 : cluster 0 2.1e scrub ok 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:28.436646+0000 osd.4 (osd.4) 508 : cluster 0 10.17 scrub starts 2024-04-02T05:06:29.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:28.443392+0000 osd.4 (osd.4) 509 : cluster 0 10.17 scrub ok 2024-04-02T05:06:29.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: cluster 2024-04-02T05:06:28.462333+0000 mgr.y (mgr.14538) 871 : cluster 0 pgmap v1502: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:29.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:29 smithi069 bash[28904]: audit 2024-04-02T05:06:29.133373+0000 mon.b (mon.1) 349 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.937998+0000 osd.6 (osd.6) 479 : cluster 0 4.a scrub starts 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:27.939954+0000 osd.6 (osd.6) 480 : cluster 0 4.a scrub ok 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:28.403528+0000 osd.3 (osd.3) 557 : cluster 0 2.1e scrub starts 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:28.406018+0000 osd.3 (osd.3) 558 : cluster 0 2.1e scrub ok 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:28.436646+0000 osd.4 (osd.4) 508 : cluster 0 10.17 scrub starts 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:28.443392+0000 osd.4 (osd.4) 509 : cluster 0 10.17 scrub ok 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: cluster 2024-04-02T05:06:28.462333+0000 mgr.y (mgr.14538) 871 : cluster 0 pgmap v1502: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:29 smithi098 bash[23790]: audit 2024-04-02T05:06:29.133373+0000 mon.b (mon.1) 349 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:06:30.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- 172.21.15.69:0/3823518312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248101b60 msgr2=0x7f324810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:30.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- 172.21.15.69:0/3823518312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248101b60 0x7f324810dfd0 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f324000b0a0 tx=0x7f324000b3b0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- 172.21.15.69:0/3823518312 shutdown_connections 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- 172.21.15.69:0/3823518312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248101b60 0x7f324810dfd0 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- 172.21.15.69:0/3823518312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3248107220 0x7f3248107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- 172.21.15.69:0/3823518312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32481011a0 0x7f3248101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- 172.21.15.69:0/3823518312 >> 172.21.15.69:0/3823518312 conn(0x7f32480771e0 msgr2=0x7f32480775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- 172.21.15.69:0/3823518312 shutdown_connections 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- 172.21.15.69:0/3823518312 wait complete. 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 librados: starting msgr at 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 librados: starting objecter 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 Processor -- start 2024-04-02T05:06:30.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- start start 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 librados: setting wanted keys 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 librados: calling monclient init 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32481011a0 0x7f324819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3248101b60 0x7f324819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 0x7f324819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32481109d0 con 0x7f3248101b60 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3248110850 con 0x7f32481011a0 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f324cce3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3248110b50 con 0x7f3248107220 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3245d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3248101b60 0x7f324819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 0x7f324819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 0x7f324819eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50484/0 (socket says 172.21.15.69:50484) 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3245d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3248101b60 0x7f324819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51400/0 (socket says 172.21.15.69:51400) 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 -- 172.21.15.69:0/3700736561 learned_addr learned my addr 172.21.15.69:0/3700736561 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:30.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 -- 172.21.15.69:0/3700736561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32481011a0 msgr2=0x7f324819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:30.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32481011a0 0x7f324819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 -- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3248101b60 msgr2=0x7f324819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:30.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3248101b60 0x7f324819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 -- 172.21.15.69:0/3700736561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324819f2c0 con 0x7f3248107220 2024-04-02T05:06:30.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f3246d9d700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 0x7f324819eb80 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f3240009850 tx=0x7f32400079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:30.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f32377fe700 10 client.?.objecter ms_handle_connect 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f32377fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.001+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240003c00 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f3240003da0 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324819f550 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324001b6a0 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324cce3700 10 client.44003.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32481a6e30 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324cce3700 1 librados: init done 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f324804ff00 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3240026400 con 0x7f3248107220 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f323006f630 0x7f3230071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:30.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(951..951 src has 897..951) ==== 9015+0+0 (secure 0 0 0) 0x7f324000f020 con 0x7f3248107220 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 10 client.44003.objecter ms_dispatch 0x7f3248001550 osd_map(951..951 src has 897..951) 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 3 client.44003.objecter handle_osd_map got epochs [951,951] > 0 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 3 client.44003.objecter handle_osd_map decoding full epoch 951 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324659c700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f323006f630 0x7f3230071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 20 client.44003.objecter dump_active .. 0 homeless 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f324659c700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f323006f630 0x7f3230071af0 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 crypto rx=0x7f32380097b0 tx=0x7f3238006d20 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:30.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.005+0000 7f32377fe700 10 client.44003.objecter ms_handle_connect 0x7f323006f630 2024-04-02T05:06:30.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.009+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f324002b070 con 0x7f3248107220 2024-04-02T05:06:30.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.237+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f32481a7000 con 0x7f3248107220 2024-04-02T05:06:30.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.677+0000 7f32377fe700 1 -- 172.21.15.69:0/3700736561 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v952) ==== 113+0+0 (secure 0 0 0) 0x7f324005ca20 con 0x7f3248107220 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 10 librados: watch_flush enter 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 10 librados: watch_flush exit 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 20 client.44003.objecter shutdown clearing up homeless session... 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 10 client.44003.objecter successfully canceled tick 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f323006f630 msgr2=0x7f3230071af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f323006f630 0x7f3230071af0 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 crypto rx=0x7f32380097b0 tx=0x7f3238006d20 comp rx=0 tx=0).stop 2024-04-02T05:06:30.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 msgr2=0x7f324819eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:30.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 0x7f324819eb80 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f3240009850 tx=0x7f32400079a0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 shutdown_connections 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f323006f630 0x7f3230071af0 unknown :-1 s=CLOSED pgs=4046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3248107220 0x7f324819eb80 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32481011a0 0x7f324819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 --2- 172.21.15.69:0/3700736561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3248101b60 0x7f324819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 >> 172.21.15.69:0/3700736561 conn(0x7f32480771e0 msgr2=0x7f32480ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 shutdown_connections 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 -- 172.21.15.69:0/3700736561 wait complete. 2024-04-02T05:06:30.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:30.685+0000 7f324cce3700 1 librados: shutdown 2024-04-02T05:06:30.693 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.6 to 1 (10000) 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: cluster 2024-04-02T05:06:29.000415+0000 osd.1 (osd.1) 551 : cluster 0 3.4 scrub starts 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: cluster 2024-04-02T05:06:29.006211+0000 osd.1 (osd.1) 552 : cluster 0 3.4 scrub ok 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: cluster 2024-04-02T05:06:29.438389+0000 osd.3 (osd.3) 559 : cluster 0 2.19 scrub starts 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: cluster 2024-04-02T05:06:29.440083+0000 osd.3 (osd.3) 560 : cluster 0 2.19 scrub ok 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: audit 2024-04-02T05:06:29.664020+0000 mon.a (mon.0) 4164 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: audit 2024-04-02T05:06:29.677094+0000 mon.a (mon.0) 4165 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.137850+0000 osd.5 (osd.5) 363 : cluster 0 1.12 scrub starts 2024-04-02T05:06:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.139329+0000 osd.5 (osd.5) 364 : cluster 0 1.12 scrub ok 2024-04-02T05:06:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: audit 2024-04-02T05:06:30.243964+0000 mon.c (mon.2) 1176 : audit 1 from='client.? 172.21.15.69:0/3700736561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:30.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:30 smithi098 bash[23790]: audit 2024-04-02T05:06:30.244577+0000 mon.a (mon.0) 4166 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: cluster 2024-04-02T05:06:29.000415+0000 osd.1 (osd.1) 551 : cluster 0 3.4 scrub starts 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: cluster 2024-04-02T05:06:29.006211+0000 osd.1 (osd.1) 552 : cluster 0 3.4 scrub ok 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: cluster 2024-04-02T05:06:29.438389+0000 osd.3 (osd.3) 559 : cluster 0 2.19 scrub starts 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: cluster 2024-04-02T05:06:29.440083+0000 osd.3 (osd.3) 560 : cluster 0 2.19 scrub ok 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: audit 2024-04-02T05:06:29.664020+0000 mon.a (mon.0) 4164 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: audit 2024-04-02T05:06:29.677094+0000 mon.a (mon.0) 4165 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.137850+0000 osd.5 (osd.5) 363 : cluster 0 1.12 scrub starts 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.139329+0000 osd.5 (osd.5) 364 : cluster 0 1.12 scrub ok 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: audit 2024-04-02T05:06:30.243964+0000 mon.c (mon.2) 1176 : audit 1 from='client.? 172.21.15.69:0/3700736561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[28904]: audit 2024-04-02T05:06:30.244577+0000 mon.a (mon.0) 4166 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: cluster 2024-04-02T05:06:29.000415+0000 osd.1 (osd.1) 551 : cluster 0 3.4 scrub starts 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: cluster 2024-04-02T05:06:29.006211+0000 osd.1 (osd.1) 552 : cluster 0 3.4 scrub ok 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: cluster 2024-04-02T05:06:29.438389+0000 osd.3 (osd.3) 559 : cluster 0 2.19 scrub starts 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: cluster 2024-04-02T05:06:29.440083+0000 osd.3 (osd.3) 560 : cluster 0 2.19 scrub ok 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: audit 2024-04-02T05:06:29.664020+0000 mon.a (mon.0) 4164 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: audit 2024-04-02T05:06:29.677094+0000 mon.a (mon.0) 4165 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.137850+0000 osd.5 (osd.5) 363 : cluster 0 1.12 scrub starts 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.139329+0000 osd.5 (osd.5) 364 : cluster 0 1.12 scrub ok 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: audit 2024-04-02T05:06:30.243964+0000 mon.c (mon.2) 1176 : audit 1 from='client.? 172.21.15.69:0/3700736561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:31.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:30 smithi069 bash[21528]: audit 2024-04-02T05:06:30.244577+0000 mon.a (mon.0) 4166 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:31.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 6 1 2024-04-02T05:06:31.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.439392+0000 osd.4 (osd.4) 510 : cluster 0 3.3 scrub starts 2024-04-02T05:06:31.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.444435+0000 osd.4 (osd.4) 511 : cluster 0 3.3 scrub ok 2024-04-02T05:06:31.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.455958+0000 osd.7 (osd.7) 345 : cluster 0 10.1f scrub starts 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.456141+0000 osd.3 (osd.3) 561 : cluster 0 6.c deep-scrub starts 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.458764+0000 osd.3 (osd.3) 562 : cluster 0 6.c deep-scrub ok 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.462986+0000 osd.7 (osd.7) 346 : cluster 0 10.1f scrub ok 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.463557+0000 mgr.y (mgr.14538) 872 : cluster 0 pgmap v1503: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: audit 2024-04-02T05:06:30.675764+0000 mon.a (mon.0) 4167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[21528]: cluster 2024-04-02T05:06:30.682331+0000 mon.a (mon.0) 4168 : cluster 0 osdmap e952: 8 total, 8 up, 8 in 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.439392+0000 osd.4 (osd.4) 510 : cluster 0 3.3 scrub starts 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.444435+0000 osd.4 (osd.4) 511 : cluster 0 3.3 scrub ok 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.455958+0000 osd.7 (osd.7) 345 : cluster 0 10.1f scrub starts 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.456141+0000 osd.3 (osd.3) 561 : cluster 0 6.c deep-scrub starts 2024-04-02T05:06:31.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.458764+0000 osd.3 (osd.3) 562 : cluster 0 6.c deep-scrub ok 2024-04-02T05:06:31.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.462986+0000 osd.7 (osd.7) 346 : cluster 0 10.1f scrub ok 2024-04-02T05:06:31.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.463557+0000 mgr.y (mgr.14538) 872 : cluster 0 pgmap v1503: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:31.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: audit 2024-04-02T05:06:30.675764+0000 mon.a (mon.0) 4167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-02T05:06:31.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:31 smithi069 bash[28904]: cluster 2024-04-02T05:06:30.682331+0000 mon.a (mon.0) 4168 : cluster 0 osdmap e952: 8 total, 8 up, 8 in 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.439392+0000 osd.4 (osd.4) 510 : cluster 0 3.3 scrub starts 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.444435+0000 osd.4 (osd.4) 511 : cluster 0 3.3 scrub ok 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.455958+0000 osd.7 (osd.7) 345 : cluster 0 10.1f scrub starts 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.456141+0000 osd.3 (osd.3) 561 : cluster 0 6.c deep-scrub starts 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.458764+0000 osd.3 (osd.3) 562 : cluster 0 6.c deep-scrub ok 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.462986+0000 osd.7 (osd.7) 346 : cluster 0 10.1f scrub ok 2024-04-02T05:06:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.463557+0000 mgr.y (mgr.14538) 872 : cluster 0 pgmap v1503: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: audit 2024-04-02T05:06:30.675764+0000 mon.a (mon.0) 4167 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-02T05:06:31.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:31 smithi098 bash[23790]: cluster 2024-04-02T05:06:30.682331+0000 mon.a (mon.0) 4168 : cluster 0 osdmap e952: 8 total, 8 up, 8 in 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.106560+0000 osd.0 (osd.0) 513 : cluster 0 1.11 scrub starts 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.109033+0000 osd.0 (osd.0) 514 : cluster 0 1.11 scrub ok 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.464087+0000 osd.7 (osd.7) 347 : cluster 0 1.9 scrub starts 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.466446+0000 osd.7 (osd.7) 348 : cluster 0 1.9 scrub ok 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.692347+0000 mon.a (mon.0) 4169 : cluster 0 osdmap e953: 8 total, 8 up, 8 in 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.876248+0000 osd.6 (osd.6) 481 : cluster 0 1.6 scrub starts 2024-04-02T05:06:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:32 smithi098 bash[23790]: cluster 2024-04-02T05:06:31.878499+0000 osd.6 (osd.6) 482 : cluster 0 1.6 scrub ok 2024-04-02T05:06:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.106560+0000 osd.0 (osd.0) 513 : cluster 0 1.11 scrub starts 2024-04-02T05:06:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.109033+0000 osd.0 (osd.0) 514 : cluster 0 1.11 scrub ok 2024-04-02T05:06:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.464087+0000 osd.7 (osd.7) 347 : cluster 0 1.9 scrub starts 2024-04-02T05:06:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.466446+0000 osd.7 (osd.7) 348 : cluster 0 1.9 scrub ok 2024-04-02T05:06:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.692347+0000 mon.a (mon.0) 4169 : cluster 0 osdmap e953: 8 total, 8 up, 8 in 2024-04-02T05:06:33.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.876248+0000 osd.6 (osd.6) 481 : cluster 0 1.6 scrub starts 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[21528]: cluster 2024-04-02T05:06:31.878499+0000 osd.6 (osd.6) 482 : cluster 0 1.6 scrub ok 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.106560+0000 osd.0 (osd.0) 513 : cluster 0 1.11 scrub starts 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.109033+0000 osd.0 (osd.0) 514 : cluster 0 1.11 scrub ok 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.464087+0000 osd.7 (osd.7) 347 : cluster 0 1.9 scrub starts 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.466446+0000 osd.7 (osd.7) 348 : cluster 0 1.9 scrub ok 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.692347+0000 mon.a (mon.0) 4169 : cluster 0 osdmap e953: 8 total, 8 up, 8 in 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.876248+0000 osd.6 (osd.6) 481 : cluster 0 1.6 scrub starts 2024-04-02T05:06:33.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:32 smithi069 bash[28904]: cluster 2024-04-02T05:06:31.878499+0000 osd.6 (osd.6) 482 : cluster 0 1.6 scrub ok 2024-04-02T05:06:34.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.420163+0000 osd.7 (osd.7) 349 : cluster 0 6.12 scrub starts 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.422555+0000 osd.7 (osd.7) 350 : cluster 0 6.12 scrub ok 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.464490+0000 mgr.y (mgr.14538) 873 : cluster 0 pgmap v1506: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.470825+0000 osd.3 (osd.3) 563 : cluster 0 3.2 scrub starts 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.475536+0000 osd.3 (osd.3) 564 : cluster 0 3.2 scrub ok 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.696926+0000 mon.a (mon.0) 4170 : cluster 0 osdmap e954: 8 total, 8 up, 8 in 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.926875+0000 osd.6 (osd.6) 483 : cluster 0 10.b scrub starts 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[21528]: cluster 2024-04-02T05:06:32.932519+0000 osd.6 (osd.6) 484 : cluster 0 10.b scrub ok 2024-04-02T05:06:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.420163+0000 osd.7 (osd.7) 349 : cluster 0 6.12 scrub starts 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.422555+0000 osd.7 (osd.7) 350 : cluster 0 6.12 scrub ok 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.464490+0000 mgr.y (mgr.14538) 873 : cluster 0 pgmap v1506: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.470825+0000 osd.3 (osd.3) 563 : cluster 0 3.2 scrub starts 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.475536+0000 osd.3 (osd.3) 564 : cluster 0 3.2 scrub ok 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.696926+0000 mon.a (mon.0) 4170 : cluster 0 osdmap e954: 8 total, 8 up, 8 in 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.926875+0000 osd.6 (osd.6) 483 : cluster 0 10.b scrub starts 2024-04-02T05:06:34.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:33 smithi069 bash[28904]: cluster 2024-04-02T05:06:32.932519+0000 osd.6 (osd.6) 484 : cluster 0 10.b scrub ok 2024-04-02T05:06:34.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.420163+0000 osd.7 (osd.7) 349 : cluster 0 6.12 scrub starts 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.422555+0000 osd.7 (osd.7) 350 : cluster 0 6.12 scrub ok 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.464490+0000 mgr.y (mgr.14538) 873 : cluster 0 pgmap v1506: 184 pgs: 184 active+clean; 582 KiB data, 758 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.470825+0000 osd.3 (osd.3) 563 : cluster 0 3.2 scrub starts 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.475536+0000 osd.3 (osd.3) 564 : cluster 0 3.2 scrub ok 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.696926+0000 mon.a (mon.0) 4170 : cluster 0 osdmap e954: 8 total, 8 up, 8 in 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.926875+0000 osd.6 (osd.6) 483 : cluster 0 10.b scrub starts 2024-04-02T05:06:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:33 smithi098 bash[23790]: cluster 2024-04-02T05:06:32.932519+0000 osd.6 (osd.6) 484 : cluster 0 10.b scrub ok 2024-04-02T05:06:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: cluster 2024-04-02T05:06:33.086732+0000 osd.0 (osd.0) 515 : cluster 0 4.6 deep-scrub starts 2024-04-02T05:06:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: cluster 2024-04-02T05:06:33.089438+0000 osd.0 (osd.0) 516 : cluster 0 4.6 deep-scrub ok 2024-04-02T05:06:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: cluster 2024-04-02T05:06:33.461716+0000 osd.3 (osd.3) 565 : cluster 0 4.0 scrub starts 2024-04-02T05:06:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: cluster 2024-04-02T05:06:33.464757+0000 osd.3 (osd.3) 566 : cluster 0 4.0 scrub ok 2024-04-02T05:06:35.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: cluster 2024-04-02T05:06:33.713442+0000 mon.a (mon.0) 4171 : cluster 0 osdmap e955: 8 total, 8 up, 8 in 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: audit 2024-04-02T05:06:34.644561+0000 mon.b (mon.1) 350 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[21528]: cluster 2024-04-02T05:06:34.711179+0000 mon.a (mon.0) 4172 : cluster 0 osdmap e956: 8 total, 8 up, 8 in 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: cluster 2024-04-02T05:06:33.086732+0000 osd.0 (osd.0) 515 : cluster 0 4.6 deep-scrub starts 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: cluster 2024-04-02T05:06:33.089438+0000 osd.0 (osd.0) 516 : cluster 0 4.6 deep-scrub ok 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: cluster 2024-04-02T05:06:33.461716+0000 osd.3 (osd.3) 565 : cluster 0 4.0 scrub starts 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: cluster 2024-04-02T05:06:33.464757+0000 osd.3 (osd.3) 566 : cluster 0 4.0 scrub ok 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: cluster 2024-04-02T05:06:33.713442+0000 mon.a (mon.0) 4171 : cluster 0 osdmap e955: 8 total, 8 up, 8 in 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: audit 2024-04-02T05:06:34.644561+0000 mon.b (mon.1) 350 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:35.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:34 smithi069 bash[28904]: cluster 2024-04-02T05:06:34.711179+0000 mon.a (mon.0) 4172 : cluster 0 osdmap e956: 8 total, 8 up, 8 in 2024-04-02T05:06:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: cluster 2024-04-02T05:06:33.086732+0000 osd.0 (osd.0) 515 : cluster 0 4.6 deep-scrub starts 2024-04-02T05:06:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: cluster 2024-04-02T05:06:33.089438+0000 osd.0 (osd.0) 516 : cluster 0 4.6 deep-scrub ok 2024-04-02T05:06:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: cluster 2024-04-02T05:06:33.461716+0000 osd.3 (osd.3) 565 : cluster 0 4.0 scrub starts 2024-04-02T05:06:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: cluster 2024-04-02T05:06:33.464757+0000 osd.3 (osd.3) 566 : cluster 0 4.0 scrub ok 2024-04-02T05:06:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: cluster 2024-04-02T05:06:33.713442+0000 mon.a (mon.0) 4171 : cluster 0 osdmap e955: 8 total, 8 up, 8 in 2024-04-02T05:06:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: audit 2024-04-02T05:06:34.644561+0000 mon.b (mon.1) 350 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:34 smithi098 bash[23790]: cluster 2024-04-02T05:06:34.711179+0000 mon.a (mon.0) 4172 : cluster 0 osdmap e956: 8 total, 8 up, 8 in 2024-04-02T05:06:36.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: audit 2024-04-02T05:06:33.937216+0000 mgr.y (mgr.14538) 874 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: cluster 2024-04-02T05:06:34.465113+0000 mgr.y (mgr.14538) 875 : cluster 0 pgmap v1509: 184 pgs: 1 active+undersized+remapped, 183 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: audit 2024-04-02T05:06:34.825119+0000 mon.a (mon.0) 4173 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: audit 2024-04-02T05:06:34.835657+0000 mon.a (mon.0) 4174 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: cluster 2024-04-02T05:06:35.414547+0000 osd.3 (osd.3) 567 : cluster 0 5.0 deep-scrub starts 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: cluster 2024-04-02T05:06:35.417444+0000 osd.3 (osd.3) 568 : cluster 0 5.0 deep-scrub ok 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: audit 2024-04-02T05:06:35.734892+0000 mon.a (mon.0) 4175 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:35 smithi098 bash[23790]: audit 2024-04-02T05:06:35.750876+0000 mon.a (mon.0) 4176 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.234 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: audit 2024-04-02T05:06:33.937216+0000 mgr.y (mgr.14538) 874 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: cluster 2024-04-02T05:06:34.465113+0000 mgr.y (mgr.14538) 875 : cluster 0 pgmap v1509: 184 pgs: 1 active+undersized+remapped, 183 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: audit 2024-04-02T05:06:34.825119+0000 mon.a (mon.0) 4173 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: audit 2024-04-02T05:06:34.835657+0000 mon.a (mon.0) 4174 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: cluster 2024-04-02T05:06:35.414547+0000 osd.3 (osd.3) 567 : cluster 0 5.0 deep-scrub starts 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: cluster 2024-04-02T05:06:35.417444+0000 osd.3 (osd.3) 568 : cluster 0 5.0 deep-scrub ok 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: audit 2024-04-02T05:06:35.734892+0000 mon.a (mon.0) 4175 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[28904]: audit 2024-04-02T05:06:35.750876+0000 mon.a (mon.0) 4176 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: audit 2024-04-02T05:06:33.937216+0000 mgr.y (mgr.14538) 874 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: cluster 2024-04-02T05:06:34.465113+0000 mgr.y (mgr.14538) 875 : cluster 0 pgmap v1509: 184 pgs: 1 active+undersized+remapped, 183 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: audit 2024-04-02T05:06:34.825119+0000 mon.a (mon.0) 4173 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: audit 2024-04-02T05:06:34.835657+0000 mon.a (mon.0) 4174 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: cluster 2024-04-02T05:06:35.414547+0000 osd.3 (osd.3) 567 : cluster 0 5.0 deep-scrub starts 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: cluster 2024-04-02T05:06:35.417444+0000 osd.3 (osd.3) 568 : cluster 0 5.0 deep-scrub ok 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: audit 2024-04-02T05:06:35.734892+0000 mon.a (mon.0) 4175 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:36.257 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:35 smithi069 bash[21528]: audit 2024-04-02T05:06:35.750876+0000 mon.a (mon.0) 4176 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:36 smithi098 bash[23790]: cluster 2024-04-02T05:06:35.863909+0000 osd.6 (osd.6) 485 : cluster 0 2.1a scrub starts 2024-04-02T05:06:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:36 smithi098 bash[23790]: cluster 2024-04-02T05:06:35.865609+0000 osd.6 (osd.6) 486 : cluster 0 2.1a scrub ok 2024-04-02T05:06:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:36 smithi098 bash[23790]: audit 2024-04-02T05:06:36.378117+0000 mon.b (mon.1) 351 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:06:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:36 smithi098 bash[23790]: audit 2024-04-02T05:06:36.380296+0000 mon.b (mon.1) 352 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:06:37.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:36 smithi098 bash[23790]: audit 2024-04-02T05:06:36.390806+0000 mon.a (mon.0) 4177 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[28904]: cluster 2024-04-02T05:06:35.863909+0000 osd.6 (osd.6) 485 : cluster 0 2.1a scrub starts 2024-04-02T05:06:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[28904]: cluster 2024-04-02T05:06:35.865609+0000 osd.6 (osd.6) 486 : cluster 0 2.1a scrub ok 2024-04-02T05:06:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[28904]: audit 2024-04-02T05:06:36.378117+0000 mon.b (mon.1) 351 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:06:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[28904]: audit 2024-04-02T05:06:36.380296+0000 mon.b (mon.1) 352 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:06:37.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[28904]: audit 2024-04-02T05:06:36.390806+0000 mon.a (mon.0) 4177 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:37.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[21528]: cluster 2024-04-02T05:06:35.863909+0000 osd.6 (osd.6) 485 : cluster 0 2.1a scrub starts 2024-04-02T05:06:37.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[21528]: cluster 2024-04-02T05:06:35.865609+0000 osd.6 (osd.6) 486 : cluster 0 2.1a scrub ok 2024-04-02T05:06:37.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[21528]: audit 2024-04-02T05:06:36.378117+0000 mon.b (mon.1) 351 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:06:37.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[21528]: audit 2024-04-02T05:06:36.380296+0000 mon.b (mon.1) 352 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:06:37.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:36 smithi069 bash[21528]: audit 2024-04-02T05:06:36.390806+0000 mon.a (mon.0) 4177 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:06:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[21528]: cluster 2024-04-02T05:06:36.465885+0000 mgr.y (mgr.14538) 876 : cluster 0 pgmap v1511: 184 pgs: 1 active+undersized+remapped, 183 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:06:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[21528]: cluster 2024-04-02T05:06:36.854056+0000 osd.6 (osd.6) 487 : cluster 0 1.7 scrub starts 2024-04-02T05:06:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[21528]: cluster 2024-04-02T05:06:36.856563+0000 osd.6 (osd.6) 488 : cluster 0 1.7 scrub ok 2024-04-02T05:06:38.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[21528]: cluster 2024-04-02T05:06:37.160974+0000 osd.5 (osd.5) 365 : cluster 0 1.8 deep-scrub starts 2024-04-02T05:06:38.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[21528]: cluster 2024-04-02T05:06:37.167006+0000 osd.5 (osd.5) 366 : cluster 0 1.8 deep-scrub ok 2024-04-02T05:06:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[28904]: cluster 2024-04-02T05:06:36.465885+0000 mgr.y (mgr.14538) 876 : cluster 0 pgmap v1511: 184 pgs: 1 active+undersized+remapped, 183 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:06:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[28904]: cluster 2024-04-02T05:06:36.854056+0000 osd.6 (osd.6) 487 : cluster 0 1.7 scrub starts 2024-04-02T05:06:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[28904]: cluster 2024-04-02T05:06:36.856563+0000 osd.6 (osd.6) 488 : cluster 0 1.7 scrub ok 2024-04-02T05:06:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[28904]: cluster 2024-04-02T05:06:37.160974+0000 osd.5 (osd.5) 365 : cluster 0 1.8 deep-scrub starts 2024-04-02T05:06:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:37 smithi069 bash[28904]: cluster 2024-04-02T05:06:37.167006+0000 osd.5 (osd.5) 366 : cluster 0 1.8 deep-scrub ok 2024-04-02T05:06:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:37 smithi098 bash[23790]: cluster 2024-04-02T05:06:36.465885+0000 mgr.y (mgr.14538) 876 : cluster 0 pgmap v1511: 184 pgs: 1 active+undersized+remapped, 183 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:06:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:37 smithi098 bash[23790]: cluster 2024-04-02T05:06:36.854056+0000 osd.6 (osd.6) 487 : cluster 0 1.7 scrub starts 2024-04-02T05:06:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:37 smithi098 bash[23790]: cluster 2024-04-02T05:06:36.856563+0000 osd.6 (osd.6) 488 : cluster 0 1.7 scrub ok 2024-04-02T05:06:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:37 smithi098 bash[23790]: cluster 2024-04-02T05:06:37.160974+0000 osd.5 (osd.5) 365 : cluster 0 1.8 deep-scrub starts 2024-04-02T05:06:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:37 smithi098 bash[23790]: cluster 2024-04-02T05:06:37.167006+0000 osd.5 (osd.5) 366 : cluster 0 1.8 deep-scrub ok 2024-04-02T05:06:39.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:06:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:06:39.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21528]: cluster 2024-04-02T05:06:37.480220+0000 osd.7 (osd.7) 351 : cluster 0 3.15 scrub starts 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21528]: cluster 2024-04-02T05:06:37.484012+0000 osd.7 (osd.7) 352 : cluster 0 3.15 scrub ok 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21528]: cluster 2024-04-02T05:06:37.737855+0000 osd.2 (osd.2) 422 : cluster 0 10.3 deep-scrub starts 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21528]: cluster 2024-04-02T05:06:37.743660+0000 osd.2 (osd.2) 423 : cluster 0 10.3 deep-scrub ok 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21528]: cluster 2024-04-02T05:06:38.508080+0000 osd.4 (osd.4) 512 : cluster 0 4.1b scrub starts 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[21528]: cluster 2024-04-02T05:06:38.510521+0000 osd.4 (osd.4) 513 : cluster 0 4.1b scrub ok 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[28904]: cluster 2024-04-02T05:06:37.480220+0000 osd.7 (osd.7) 351 : cluster 0 3.15 scrub starts 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[28904]: cluster 2024-04-02T05:06:37.484012+0000 osd.7 (osd.7) 352 : cluster 0 3.15 scrub ok 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[28904]: cluster 2024-04-02T05:06:37.737855+0000 osd.2 (osd.2) 422 : cluster 0 10.3 deep-scrub starts 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[28904]: cluster 2024-04-02T05:06:37.743660+0000 osd.2 (osd.2) 423 : cluster 0 10.3 deep-scrub ok 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[28904]: cluster 2024-04-02T05:06:38.508080+0000 osd.4 (osd.4) 512 : cluster 0 4.1b scrub starts 2024-04-02T05:06:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:38 smithi069 bash[28904]: cluster 2024-04-02T05:06:38.510521+0000 osd.4 (osd.4) 513 : cluster 0 4.1b scrub ok 2024-04-02T05:06:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:38 smithi098 bash[23790]: cluster 2024-04-02T05:06:37.480220+0000 osd.7 (osd.7) 351 : cluster 0 3.15 scrub starts 2024-04-02T05:06:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:38 smithi098 bash[23790]: cluster 2024-04-02T05:06:37.484012+0000 osd.7 (osd.7) 352 : cluster 0 3.15 scrub ok 2024-04-02T05:06:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:38 smithi098 bash[23790]: cluster 2024-04-02T05:06:37.737855+0000 osd.2 (osd.2) 422 : cluster 0 10.3 deep-scrub starts 2024-04-02T05:06:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:38 smithi098 bash[23790]: cluster 2024-04-02T05:06:37.743660+0000 osd.2 (osd.2) 423 : cluster 0 10.3 deep-scrub ok 2024-04-02T05:06:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:38 smithi098 bash[23790]: cluster 2024-04-02T05:06:38.508080+0000 osd.4 (osd.4) 512 : cluster 0 4.1b scrub starts 2024-04-02T05:06:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:38 smithi098 bash[23790]: cluster 2024-04-02T05:06:38.510521+0000 osd.4 (osd.4) 513 : cluster 0 4.1b scrub ok 2024-04-02T05:06:39.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 -- 172.21.15.69:0/2618120754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80104e20 msgr2=0x7f8f80111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:39.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 --2- 172.21.15.69:0/2618120754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80104e20 0x7f8f80111330 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800b0a0 tx=0x7f8f7800b3b0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 -- 172.21.15.69:0/2618120754 shutdown_connections 2024-04-02T05:06:39.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 --2- 172.21.15.69:0/2618120754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80104e20 0x7f8f80111330 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 --2- 172.21.15.69:0/2618120754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f8010a4f0 0x7f8f8010a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 --2- 172.21.15.69:0/2618120754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 0x7f8f801048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 -- 172.21.15.69:0/2618120754 >> 172.21.15.69:0/2618120754 conn(0x7f8f800fff50 msgr2=0x7f8f80102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:39.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 -- 172.21.15.69:0/2618120754 shutdown_connections 2024-04-02T05:06:39.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.405+0000 7f8f86438700 1 -- 172.21.15.69:0/2618120754 wait complete. 2024-04-02T05:06:39.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 librados: starting msgr at 2024-04-02T05:06:39.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 librados: starting objecter 2024-04-02T05:06:39.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 Processor -- start 2024-04-02T05:06:39.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- start start 2024-04-02T05:06:39.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 librados: setting wanted keys 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 librados: calling monclient init 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 0x7f8f8019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f80104e20 0x7f8f8019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 0x7f8f8019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80113c80 con 0x7f8f8010a4f0 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80113b00 con 0x7f8f80104e20 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8f80113e00 con 0x7f8f80104460 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f849d5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 0x7f8f8019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 0x7f8f8019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f849d5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 0x7f8f8019ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39352/0 (socket says 172.21.15.69:39352) 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f849d5700 1 -- 172.21.15.69:0/3398255595 learned_addr learned my addr 172.21.15.69:0/3398255595 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:39.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7f7fe700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f80104e20 0x7f8f8019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 -- 172.21.15.69:0/3398255595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f80104e20 msgr2=0x7f8f8019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f80104e20 0x7f8f8019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 -- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 msgr2=0x7f8f8019ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 0x7f8f8019ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 -- 172.21.15.69:0/3398255595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8019f1b0 con 0x7f8f80104460 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f849d5700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 0x7f8f8019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7ffff700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 0x7f8f8019a1a0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c970 tx=0x7f8f7000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7d7fa700 10 client.?.objecter ms_handle_connect 0x7f8f80104460 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7d7fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7000ce60 con 0x7f8f80104460 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8019f4a0 con 0x7f8f80104460 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8f70004830 con 0x7f8f80104460 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70005310 con 0x7f8f80104460 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 10 client.44009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:39.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f801a6d20 con 0x7f8f80104460 2024-04-02T05:06:39.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 librados: init done 2024-04-02T05:06:39.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8004ff00 con 0x7f8f80104460 2024-04-02T05:06:39.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.409+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8f700054f0 con 0x7f8f80104460 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f6c06f5e0 0x7f8f6c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(956..956 src has 897..956) ==== 9015+0+0 (secure 0 0 0) 0x7f8f7008e1d0 con 0x7f8f80104460 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 10 client.44009.objecter ms_dispatch 0x7f8f80001550 osd_map(956..956 src has 897..956) 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 3 client.44009.objecter handle_osd_map got epochs [956,956] > 0 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 3 client.44009.objecter handle_osd_map decoding full epoch 956 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7f7fe700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f6c06f5e0 0x7f8f6c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 20 client.44009.objecter dump_active .. 0 homeless 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7d7fa700 10 client.44009.objecter ms_handle_connect 0x7f8f6c06f5e0 2024-04-02T05:06:39.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.413+0000 7f8f7f7fe700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f6c06f5e0 0x7f8f6c071aa0 secure :-1 s=READY pgs=4047 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009680 tx=0x7f8f68009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:39.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.417+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8f70014030 con 0x7f8f80104460 2024-04-02T05:06:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.637+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7f8f801a6f50 con 0x7f8f80104460 2024-04-02T05:06:39.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.849+0000 7f8f7d7fa700 1 -- 172.21.15.69:0/3398255595 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v957) ==== 133+0+0 (secure 0 0 0) 0x7f8f7005b1a0 con 0x7f8f80104460 2024-04-02T05:06:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 10 librados: watch_flush enter 2024-04-02T05:06:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 10 librados: watch_flush exit 2024-04-02T05:06:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 20 client.44009.objecter shutdown clearing up homeless session... 2024-04-02T05:06:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 10 client.44009.objecter successfully canceled tick 2024-04-02T05:06:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f6c06f5e0 msgr2=0x7f8f6c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:39.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f6c06f5e0 0x7f8f6c071aa0 secure :-1 s=READY pgs=4047 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009680 tx=0x7f8f68009260 comp rx=0 tx=0).stop 2024-04-02T05:06:39.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.853+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 msgr2=0x7f8f8019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:39.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 0x7f8f8019a1a0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c970 tx=0x7f8f7000cc80 comp rx=0 tx=0).stop 2024-04-02T05:06:39.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 shutdown_connections 2024-04-02T05:06:39.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f8f6c06f5e0 0x7f8f6c071aa0 unknown :-1 s=CLOSED pgs=4047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8010a4f0 0x7f8f8019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80104460 0x7f8f8019a1a0 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 --2- 172.21.15.69:0/3398255595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f80104e20 0x7f8f8019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:39.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 >> 172.21.15.69:0/3398255595 conn(0x7f8f800fff50 msgr2=0x7f8f8010b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:39.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 shutdown_connections 2024-04-02T05:06:39.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 -- 172.21.15.69:0/3398255595 wait complete. 2024-04-02T05:06:39.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:39.857+0000 7f8f86438700 1 librados: shutdown 2024-04-02T05:06:39.864 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:38.467757+0000 mgr.y (mgr.14538) 877 : cluster 0 pgmap v1512: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:38.813933+0000 osd.6 (osd.6) 489 : cluster 0 10.f scrub starts 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:38.823345+0000 osd.6 (osd.6) 490 : cluster 0 10.f scrub ok 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.108770+0000 osd.5 (osd.5) 367 : cluster 0 10.8 deep-scrub starts 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.111140+0000 osd.5 (osd.5) 368 : cluster 0 10.8 deep-scrub ok 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.454155+0000 osd.3 (osd.3) 569 : cluster 0 10.1b scrub starts 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.457907+0000 osd.3 (osd.3) 570 : cluster 0 10.1b scrub ok 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: audit 2024-04-02T05:06:39.643758+0000 mon.c (mon.2) 1177 : audit 1 from='client.? 172.21.15.69:0/3398255595' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:40.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:39 smithi098 bash[23790]: audit 2024-04-02T05:06:39.644432+0000 mon.a (mon.0) 4178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:40.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:38.467757+0000 mgr.y (mgr.14538) 877 : cluster 0 pgmap v1512: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:06:40.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:38.813933+0000 osd.6 (osd.6) 489 : cluster 0 10.f scrub starts 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:38.823345+0000 osd.6 (osd.6) 490 : cluster 0 10.f scrub ok 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.108770+0000 osd.5 (osd.5) 367 : cluster 0 10.8 deep-scrub starts 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.111140+0000 osd.5 (osd.5) 368 : cluster 0 10.8 deep-scrub ok 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.454155+0000 osd.3 (osd.3) 569 : cluster 0 10.1b scrub starts 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.457907+0000 osd.3 (osd.3) 570 : cluster 0 10.1b scrub ok 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: audit 2024-04-02T05:06:39.643758+0000 mon.c (mon.2) 1177 : audit 1 from='client.? 172.21.15.69:0/3398255595' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[28904]: audit 2024-04-02T05:06:39.644432+0000 mon.a (mon.0) 4178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:38.467757+0000 mgr.y (mgr.14538) 877 : cluster 0 pgmap v1512: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:38.813933+0000 osd.6 (osd.6) 489 : cluster 0 10.f scrub starts 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:38.823345+0000 osd.6 (osd.6) 490 : cluster 0 10.f scrub ok 2024-04-02T05:06:40.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.108770+0000 osd.5 (osd.5) 367 : cluster 0 10.8 deep-scrub starts 2024-04-02T05:06:40.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.111140+0000 osd.5 (osd.5) 368 : cluster 0 10.8 deep-scrub ok 2024-04-02T05:06:40.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.454155+0000 osd.3 (osd.3) 569 : cluster 0 10.1b scrub starts 2024-04-02T05:06:40.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.457907+0000 osd.3 (osd.3) 570 : cluster 0 10.1b scrub ok 2024-04-02T05:06:40.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: audit 2024-04-02T05:06:39.643758+0000 mon.c (mon.2) 1177 : audit 1 from='client.? 172.21.15.69:0/3398255595' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:40.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:39 smithi069 bash[21528]: audit 2024-04-02T05:06:39.644432+0000 mon.a (mon.0) 4178 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-02T05:06:40.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd reweight 7 1 2024-04-02T05:06:41.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.052650+0000 osd.0 (osd.0) 517 : cluster 0 4.3 deep-scrub starts 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.055776+0000 osd.0 (osd.0) 518 : cluster 0 4.3 deep-scrub ok 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.810111+0000 osd.1 (osd.1) 553 : cluster 0 1.13 scrub starts 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.812511+0000 osd.1 (osd.1) 554 : cluster 0 1.13 scrub ok 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: audit 2024-04-02T05:06:39.844221+0000 mon.a (mon.0) 4179 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:39.856597+0000 mon.a (mon.0) 4180 : cluster 0 osdmap e957: 8 total, 8 up, 8 in 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.461893+0000 osd.3 (osd.3) 571 : cluster 0 3.18 deep-scrub starts 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.469952+0000 osd.3 (osd.3) 572 : cluster 0 3.18 deep-scrub ok 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.512540+0000 osd.4 (osd.4) 514 : cluster 0 3.9 scrub starts 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.516876+0000 osd.4 (osd.4) 515 : cluster 0 3.9 scrub ok 2024-04-02T05:06:41.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.052650+0000 osd.0 (osd.0) 517 : cluster 0 4.3 deep-scrub starts 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.055776+0000 osd.0 (osd.0) 518 : cluster 0 4.3 deep-scrub ok 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.810111+0000 osd.1 (osd.1) 553 : cluster 0 1.13 scrub starts 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.812511+0000 osd.1 (osd.1) 554 : cluster 0 1.13 scrub ok 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: audit 2024-04-02T05:06:39.844221+0000 mon.a (mon.0) 4179 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:39.856597+0000 mon.a (mon.0) 4180 : cluster 0 osdmap e957: 8 total, 8 up, 8 in 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.461893+0000 osd.3 (osd.3) 571 : cluster 0 3.18 deep-scrub starts 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.469952+0000 osd.3 (osd.3) 572 : cluster 0 3.18 deep-scrub ok 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.512540+0000 osd.4 (osd.4) 514 : cluster 0 3.9 scrub starts 2024-04-02T05:06:41.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:40 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.516876+0000 osd.4 (osd.4) 515 : cluster 0 3.9 scrub ok 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.052650+0000 osd.0 (osd.0) 517 : cluster 0 4.3 deep-scrub starts 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.055776+0000 osd.0 (osd.0) 518 : cluster 0 4.3 deep-scrub ok 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.810111+0000 osd.1 (osd.1) 553 : cluster 0 1.13 scrub starts 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.812511+0000 osd.1 (osd.1) 554 : cluster 0 1.13 scrub ok 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: audit 2024-04-02T05:06:39.844221+0000 mon.a (mon.0) 4179 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:39.856597+0000 mon.a (mon.0) 4180 : cluster 0 osdmap e957: 8 total, 8 up, 8 in 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.461893+0000 osd.3 (osd.3) 571 : cluster 0 3.18 deep-scrub starts 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.469952+0000 osd.3 (osd.3) 572 : cluster 0 3.18 deep-scrub ok 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.512540+0000 osd.4 (osd.4) 514 : cluster 0 3.9 scrub starts 2024-04-02T05:06:41.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:40 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.516876+0000 osd.4 (osd.4) 515 : cluster 0 3.9 scrub ok 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.461957+0000 osd.7 (osd.7) 353 : cluster 0 6.5 scrub starts 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.464272+0000 osd.7 (osd.7) 354 : cluster 0 6.5 scrub ok 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:40.468376+0000 mgr.y (mgr.14538) 878 : cluster 0 pgmap v1514: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 757 B/s rd, 0 op/s 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:41.094621+0000 osd.5 (osd.5) 369 : cluster 0 10.6 scrub starts 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:41.098139+0000 osd.5 (osd.5) 370 : cluster 0 10.6 scrub ok 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:41.486954+0000 osd.4 (osd.4) 516 : cluster 0 1.a deep-scrub starts 2024-04-02T05:06:42.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:41 smithi098 bash[23790]: cluster 2024-04-02T05:06:41.489416+0000 osd.4 (osd.4) 517 : cluster 0 1.a deep-scrub ok 2024-04-02T05:06:42.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.461957+0000 osd.7 (osd.7) 353 : cluster 0 6.5 scrub starts 2024-04-02T05:06:42.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.464272+0000 osd.7 (osd.7) 354 : cluster 0 6.5 scrub ok 2024-04-02T05:06:42.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:40.468376+0000 mgr.y (mgr.14538) 878 : cluster 0 pgmap v1514: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 757 B/s rd, 0 op/s 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:41.094621+0000 osd.5 (osd.5) 369 : cluster 0 10.6 scrub starts 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:41.098139+0000 osd.5 (osd.5) 370 : cluster 0 10.6 scrub ok 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:41.486954+0000 osd.4 (osd.4) 516 : cluster 0 1.a deep-scrub starts 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[28904]: cluster 2024-04-02T05:06:41.489416+0000 osd.4 (osd.4) 517 : cluster 0 1.a deep-scrub ok 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.461957+0000 osd.7 (osd.7) 353 : cluster 0 6.5 scrub starts 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.464272+0000 osd.7 (osd.7) 354 : cluster 0 6.5 scrub ok 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:40.468376+0000 mgr.y (mgr.14538) 878 : cluster 0 pgmap v1514: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 757 B/s rd, 0 op/s 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:41.094621+0000 osd.5 (osd.5) 369 : cluster 0 10.6 scrub starts 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:41.098139+0000 osd.5 (osd.5) 370 : cluster 0 10.6 scrub ok 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:41.486954+0000 osd.4 (osd.4) 516 : cluster 0 1.a deep-scrub starts 2024-04-02T05:06:42.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:41 smithi069 bash[21528]: cluster 2024-04-02T05:06:41.489416+0000 osd.4 (osd.4) 517 : cluster 0 1.a deep-scrub ok 2024-04-02T05:06:44.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:06:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:06:44.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:43 smithi098 bash[23790]: cluster 2024-04-02T05:06:42.457333+0000 osd.7 (osd.7) 355 : cluster 0 2.1c scrub starts 2024-04-02T05:06:44.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:43 smithi098 bash[23790]: cluster 2024-04-02T05:06:42.460576+0000 osd.7 (osd.7) 356 : cluster 0 2.1c scrub ok 2024-04-02T05:06:44.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:43 smithi098 bash[23790]: cluster 2024-04-02T05:06:42.469092+0000 mgr.y (mgr.14538) 879 : cluster 0 pgmap v1515: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:44.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:43 smithi069 bash[21528]: cluster 2024-04-02T05:06:42.457333+0000 osd.7 (osd.7) 355 : cluster 0 2.1c scrub starts 2024-04-02T05:06:44.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:43 smithi069 bash[21528]: cluster 2024-04-02T05:06:42.460576+0000 osd.7 (osd.7) 356 : cluster 0 2.1c scrub ok 2024-04-02T05:06:44.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:43 smithi069 bash[21528]: cluster 2024-04-02T05:06:42.469092+0000 mgr.y (mgr.14538) 879 : cluster 0 pgmap v1515: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:44.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:43 smithi069 bash[28904]: cluster 2024-04-02T05:06:42.457333+0000 osd.7 (osd.7) 355 : cluster 0 2.1c scrub starts 2024-04-02T05:06:44.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:43 smithi069 bash[28904]: cluster 2024-04-02T05:06:42.460576+0000 osd.7 (osd.7) 356 : cluster 0 2.1c scrub ok 2024-04-02T05:06:44.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:43 smithi069 bash[28904]: cluster 2024-04-02T05:06:42.469092+0000 mgr.y (mgr.14538) 879 : cluster 0 pgmap v1515: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:43.061231+0000 osd.0 (osd.0) 519 : cluster 0 6.10 deep-scrub starts 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:43.063641+0000 osd.0 (osd.0) 520 : cluster 0 6.10 deep-scrub ok 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.135456+0000 osd.5 (osd.5) 371 : cluster 0 6.d scrub starts 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.137886+0000 osd.5 (osd.5) 372 : cluster 0 6.d scrub ok 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.396966+0000 osd.3 (osd.3) 573 : cluster 0 1.4 scrub starts 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.399432+0000 osd.3 (osd.3) 574 : cluster 0 1.4 scrub ok 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.525181+0000 osd.4 (osd.4) 518 : cluster 0 2.f scrub starts 2024-04-02T05:06:45.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:44 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.528347+0000 osd.4 (osd.4) 519 : cluster 0 2.f scrub ok 2024-04-02T05:06:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:43.061231+0000 osd.0 (osd.0) 519 : cluster 0 6.10 deep-scrub starts 2024-04-02T05:06:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:43.063641+0000 osd.0 (osd.0) 520 : cluster 0 6.10 deep-scrub ok 2024-04-02T05:06:45.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.135456+0000 osd.5 (osd.5) 371 : cluster 0 6.d scrub starts 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.137886+0000 osd.5 (osd.5) 372 : cluster 0 6.d scrub ok 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.396966+0000 osd.3 (osd.3) 573 : cluster 0 1.4 scrub starts 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.399432+0000 osd.3 (osd.3) 574 : cluster 0 1.4 scrub ok 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.525181+0000 osd.4 (osd.4) 518 : cluster 0 2.f scrub starts 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.528347+0000 osd.4 (osd.4) 519 : cluster 0 2.f scrub ok 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:43.061231+0000 osd.0 (osd.0) 519 : cluster 0 6.10 deep-scrub starts 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:43.063641+0000 osd.0 (osd.0) 520 : cluster 0 6.10 deep-scrub ok 2024-04-02T05:06:45.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.135456+0000 osd.5 (osd.5) 371 : cluster 0 6.d scrub starts 2024-04-02T05:06:45.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.137886+0000 osd.5 (osd.5) 372 : cluster 0 6.d scrub ok 2024-04-02T05:06:45.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.396966+0000 osd.3 (osd.3) 573 : cluster 0 1.4 scrub starts 2024-04-02T05:06:45.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.399432+0000 osd.3 (osd.3) 574 : cluster 0 1.4 scrub ok 2024-04-02T05:06:45.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.525181+0000 osd.4 (osd.4) 518 : cluster 0 2.f scrub starts 2024-04-02T05:06:45.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:44 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.528347+0000 osd.4 (osd.4) 519 : cluster 0 2.f scrub ok 2024-04-02T05:06:45.505 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: audit 2024-04-02T05:06:43.948604+0000 mgr.y (mgr.14538) 880 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.471112+0000 mgr.y (mgr.14538) 881 : cluster 0 pgmap v1516: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.867828+0000 osd.1 (osd.1) 555 : cluster 0 2.a scrub starts 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:44.870880+0000 osd.1 (osd.1) 556 : cluster 0 2.a scrub ok 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:45.236944+0000 osd.5 (osd.5) 373 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:45.237328+0000 osd.5 (osd.5) 374 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:45.565949+0000 osd.4 (osd.4) 520 : cluster 0 1.5 deep-scrub starts 2024-04-02T05:06:46.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:45 smithi098 bash[23790]: cluster 2024-04-02T05:06:45.568377+0000 osd.4 (osd.4) 521 : cluster 0 1.5 deep-scrub ok 2024-04-02T05:06:46.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: audit 2024-04-02T05:06:43.948604+0000 mgr.y (mgr.14538) 880 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:46.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.471112+0000 mgr.y (mgr.14538) 881 : cluster 0 pgmap v1516: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:06:46.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.867828+0000 osd.1 (osd.1) 555 : cluster 0 2.a scrub starts 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:44.870880+0000 osd.1 (osd.1) 556 : cluster 0 2.a scrub ok 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:45.236944+0000 osd.5 (osd.5) 373 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:45.237328+0000 osd.5 (osd.5) 374 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:45.565949+0000 osd.4 (osd.4) 520 : cluster 0 1.5 deep-scrub starts 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[21528]: cluster 2024-04-02T05:06:45.568377+0000 osd.4 (osd.4) 521 : cluster 0 1.5 deep-scrub ok 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: audit 2024-04-02T05:06:43.948604+0000 mgr.y (mgr.14538) 880 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.471112+0000 mgr.y (mgr.14538) 881 : cluster 0 pgmap v1516: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.867828+0000 osd.1 (osd.1) 555 : cluster 0 2.a scrub starts 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:44.870880+0000 osd.1 (osd.1) 556 : cluster 0 2.a scrub ok 2024-04-02T05:06:46.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:45.236944+0000 osd.5 (osd.5) 373 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:46.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:45.237328+0000 osd.5 (osd.5) 374 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:46.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:45.565949+0000 osd.4 (osd.4) 520 : cluster 0 1.5 deep-scrub starts 2024-04-02T05:06:46.285 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:45 smithi069 bash[28904]: cluster 2024-04-02T05:06:45.568377+0000 osd.4 (osd.4) 521 : cluster 0 1.5 deep-scrub ok 2024-04-02T05:06:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[21528]: cluster 2024-04-02T05:06:46.234271+0000 mon.a (mon.0) 4181 : cluster 0 osdmap e958: 8 total, 8 up, 8 in 2024-04-02T05:06:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[21528]: cluster 2024-04-02T05:06:46.354018+0000 osd.3 (osd.3) 575 : cluster 0 4.10 scrub starts 2024-04-02T05:06:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[21528]: cluster 2024-04-02T05:06:46.356338+0000 osd.3 (osd.3) 576 : cluster 0 4.10 scrub ok 2024-04-02T05:06:47.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[21528]: cluster 2024-04-02T05:06:46.471914+0000 mgr.y (mgr.14538) 882 : cluster 0 pgmap v1518: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:47.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[28904]: cluster 2024-04-02T05:06:46.234271+0000 mon.a (mon.0) 4181 : cluster 0 osdmap e958: 8 total, 8 up, 8 in 2024-04-02T05:06:47.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[28904]: cluster 2024-04-02T05:06:46.354018+0000 osd.3 (osd.3) 575 : cluster 0 4.10 scrub starts 2024-04-02T05:06:47.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[28904]: cluster 2024-04-02T05:06:46.356338+0000 osd.3 (osd.3) 576 : cluster 0 4.10 scrub ok 2024-04-02T05:06:47.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:47 smithi069 bash[28904]: cluster 2024-04-02T05:06:46.471914+0000 mgr.y (mgr.14538) 882 : cluster 0 pgmap v1518: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:47.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:47 smithi098 bash[23790]: cluster 2024-04-02T05:06:46.234271+0000 mon.a (mon.0) 4181 : cluster 0 osdmap e958: 8 total, 8 up, 8 in 2024-04-02T05:06:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:47 smithi098 bash[23790]: cluster 2024-04-02T05:06:46.354018+0000 osd.3 (osd.3) 575 : cluster 0 4.10 scrub starts 2024-04-02T05:06:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:47 smithi098 bash[23790]: cluster 2024-04-02T05:06:46.356338+0000 osd.3 (osd.3) 576 : cluster 0 4.10 scrub ok 2024-04-02T05:06:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:47 smithi098 bash[23790]: cluster 2024-04-02T05:06:46.471914+0000 mgr.y (mgr.14538) 882 : cluster 0 pgmap v1518: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:48.508 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[28904]: cluster 2024-04-02T05:06:46.961376+0000 osd.1 (osd.1) 557 : cluster 0 1.b scrub starts 2024-04-02T05:06:48.508 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[28904]: cluster 2024-04-02T05:06:46.962988+0000 osd.1 (osd.1) 558 : cluster 0 1.b scrub ok 2024-04-02T05:06:48.508 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[28904]: cluster 2024-04-02T05:06:47.001742+0000 osd.0 (osd.0) 521 : cluster 0 3.6 scrub starts 2024-04-02T05:06:48.508 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[28904]: cluster 2024-04-02T05:06:47.004350+0000 osd.0 (osd.0) 522 : cluster 0 3.6 scrub ok 2024-04-02T05:06:48.508 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[21528]: cluster 2024-04-02T05:06:46.961376+0000 osd.1 (osd.1) 557 : cluster 0 1.b scrub starts 2024-04-02T05:06:48.508 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[21528]: cluster 2024-04-02T05:06:46.962988+0000 osd.1 (osd.1) 558 : cluster 0 1.b scrub ok 2024-04-02T05:06:48.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[21528]: cluster 2024-04-02T05:06:47.001742+0000 osd.0 (osd.0) 521 : cluster 0 3.6 scrub starts 2024-04-02T05:06:48.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[21528]: cluster 2024-04-02T05:06:47.004350+0000 osd.0 (osd.0) 522 : cluster 0 3.6 scrub ok 2024-04-02T05:06:48.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/1853095667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac0749b0 msgr2=0x7fe5ac074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- 172.21.15.69:0/1853095667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac0749b0 0x7fe5ac074e30 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009990 tx=0x7fe5a0009ca0 comp rx=0 tx=0).stop 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/1853095667 shutdown_connections 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- 172.21.15.69:0/1853095667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac112f60 0x7fe5ac115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- 172.21.15.69:0/1853095667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5ac076250 0x7fe5ac074470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- 172.21.15.69:0/1853095667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac0749b0 0x7fe5ac074e30 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/1853095667 >> 172.21.15.69:0/1853095667 conn(0x7fe5ac079310 msgr2=0x7fe5ac079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/1853095667 shutdown_connections 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/1853095667 wait complete. 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 librados: starting msgr at 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 librados: starting objecter 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 Processor -- start 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- start start 2024-04-02T05:06:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 librados: setting wanted keys 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 librados: calling monclient init 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 0x7fe5ac19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5ac076250 0x7fe5ac19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 0x7fe5ac1a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1180d0 con 0x7fe5ac0749b0 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac117f50 con 0x7fe5ac076250 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ac118250 con 0x7fe5ac112f60 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 0x7fe5ac19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5ab7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 0x7fe5ac1a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 0x7fe5ac19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35762/0 (socket says 172.21.15.69:35762) 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 -- 172.21.15.69:0/499328061 learned_addr learned my addr 172.21.15.69:0/499328061 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:48.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5ab7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 0x7fe5ac1a2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33888/0 (socket says 172.21.15.69:33888) 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aa7fc700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5ac076250 0x7fe5ac19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 -- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 msgr2=0x7fe5ac1a2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 0x7fe5ac1a2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 -- 172.21.15.69:0/499328061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5ac076250 msgr2=0x7fe5ac19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5ac076250 0x7fe5ac19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 -- 172.21.15.69:0/499328061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac1a3700 con 0x7fe5ac0749b0 2024-04-02T05:06:48.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5ab7fe700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 0x7fe5ac1a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5aaffd700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 0x7fe5ac19e6f0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b530 tx=0x7fe59c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b0893700 10 client.?.objecter ms_handle_connect 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b0893700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c015020 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac1a39f0 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe59c011cb0 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c010cb0 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 10 client.33951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac1ab270 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.680+0000 7fe5b1895700 1 librados: init done 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ac04ff00 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe59c018020 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe59406f5e0 0x7fe594071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(958..958 src has 907..958) ==== 9015+0+0 (secure 0 0 0) 0x7fe59c08de30 con 0x7fe5ac0749b0 2024-04-02T05:06:48.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 10 client.33951.objecter ms_dispatch 0x7fe5ac001550 osd_map(958..958 src has 907..958) 2024-04-02T05:06:48.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 3 client.33951.objecter handle_osd_map got epochs [958,958] > 0 2024-04-02T05:06:48.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 3 client.33951.objecter handle_osd_map decoding full epoch 958 2024-04-02T05:06:48.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5aa7fc700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe59406f5e0 0x7fe594071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:48.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 20 client.33951.objecter dump_active .. 0 homeless 2024-04-02T05:06:48.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5aa7fc700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe59406f5e0 0x7fe594071aa0 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000b7f0 tx=0x7fe5a0005fb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:48.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.684+0000 7fe5b0893700 10 client.33951.objecter ms_handle_connect 0x7fe59406f5e0 2024-04-02T05:06:48.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.688+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe59c01f020 con 0x7fe5ac0749b0 2024-04-02T05:06:48.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:48 smithi098 bash[23790]: cluster 2024-04-02T05:06:46.961376+0000 osd.1 (osd.1) 557 : cluster 0 1.b scrub starts 2024-04-02T05:06:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:48 smithi098 bash[23790]: cluster 2024-04-02T05:06:46.962988+0000 osd.1 (osd.1) 558 : cluster 0 1.b scrub ok 2024-04-02T05:06:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:48 smithi098 bash[23790]: cluster 2024-04-02T05:06:47.001742+0000 osd.0 (osd.0) 521 : cluster 0 3.6 scrub starts 2024-04-02T05:06:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:48 smithi098 bash[23790]: cluster 2024-04-02T05:06:47.004350+0000 osd.0 (osd.0) 522 : cluster 0 3.6 scrub ok 2024-04-02T05:06:48.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:06:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:06:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:06:48.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:48.908+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fe5ac1ab500 con 0x7fe5ac0749b0 2024-04-02T05:06:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.240+0000 7fe5b0893700 1 -- 172.21.15.69:0/499328061 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v959) ==== 113+0+0 (secure 0 0 0) 0x7fe59c05ae00 con 0x7fe5ac0749b0 2024-04-02T05:06:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 10 librados: watch_flush enter 2024-04-02T05:06:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 10 librados: watch_flush exit 2024-04-02T05:06:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 20 client.33951.objecter shutdown clearing up homeless session... 2024-04-02T05:06:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 10 client.33951.objecter successfully canceled tick 2024-04-02T05:06:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe59406f5e0 msgr2=0x7fe594071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe59406f5e0 0x7fe594071aa0 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000b7f0 tx=0x7fe5a0005fb0 comp rx=0 tx=0).stop 2024-04-02T05:06:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 msgr2=0x7fe5ac19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 0x7fe5ac19e6f0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b530 tx=0x7fe59c00b8f0 comp rx=0 tx=0).stop 2024-04-02T05:06:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 shutdown_connections 2024-04-02T05:06:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe59406f5e0 0x7fe594071aa0 unknown :-1 s=CLOSED pgs=4048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5ac112f60 0x7fe5ac1a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5ac0749b0 0x7fe5ac19e6f0 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 --2- 172.21.15.69:0/499328061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5ac076250 0x7fe5ac19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 >> 172.21.15.69:0/499328061 conn(0x7fe5ac079310 msgr2=0x7fe5ac111920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 shutdown_connections 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 -- 172.21.15.69:0/499328061 wait complete. 2024-04-02T05:06:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:49.248+0000 7fe5b1895700 1 librados: shutdown 2024-04-02T05:06:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:reweighted osd.7 to 1 (10000) 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:47.918020+0000 osd.1 (osd.1) 559 : cluster 0 4.4 scrub starts 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:47.920378+0000 osd.1 (osd.1) 560 : cluster 0 4.4 scrub ok 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.400525+0000 osd.3 (osd.3) 577 : cluster 0 3.8 scrub starts 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.404019+0000 osd.3 (osd.3) 578 : cluster 0 3.8 scrub ok 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.446553+0000 osd.1 (osd.1) 561 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.446971+0000 osd.1 (osd.1) 562 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.473778+0000 mgr.y (mgr.14538) 883 : cluster 0 pgmap v1519: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[21528]: audit 2024-04-02T05:06:48.915578+0000 mon.a (mon.0) 4182 : audit 1 from='client.? 172.21.15.69:0/499328061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-02T05:06:49.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:47.918020+0000 osd.1 (osd.1) 559 : cluster 0 4.4 scrub starts 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:47.920378+0000 osd.1 (osd.1) 560 : cluster 0 4.4 scrub ok 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.400525+0000 osd.3 (osd.3) 577 : cluster 0 3.8 scrub starts 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.404019+0000 osd.3 (osd.3) 578 : cluster 0 3.8 scrub ok 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.446553+0000 osd.1 (osd.1) 561 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.446971+0000 osd.1 (osd.1) 562 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.473778+0000 mgr.y (mgr.14538) 883 : cluster 0 pgmap v1519: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:49 smithi069 bash[28904]: audit 2024-04-02T05:06:48.915578+0000 mon.a (mon.0) 4182 : audit 1 from='client.? 172.21.15.69:0/499328061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:47.918020+0000 osd.1 (osd.1) 559 : cluster 0 4.4 scrub starts 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:47.920378+0000 osd.1 (osd.1) 560 : cluster 0 4.4 scrub ok 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.400525+0000 osd.3 (osd.3) 577 : cluster 0 3.8 scrub starts 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.404019+0000 osd.3 (osd.3) 578 : cluster 0 3.8 scrub ok 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.446553+0000 osd.1 (osd.1) 561 : cluster 0 purged_snaps scrub starts 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.446971+0000 osd.1 (osd.1) 562 : cluster 0 purged_snaps scrub ok 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.473778+0000 mgr.y (mgr.14538) 883 : cluster 0 pgmap v1519: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:49 smithi098 bash[23790]: audit 2024-04-02T05:06:48.915578+0000 mon.a (mon.0) 4182 : audit 1 from='client.? 172.21.15.69:0/499328061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-02T05:06:50.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 7 1 2024-04-02T05:06:50.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.492183+0000 osd.7 (osd.7) 357 : cluster 0 6.1c deep-scrub starts 2024-04-02T05:06:50.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.497959+0000 osd.7 (osd.7) 358 : cluster 0 6.1c deep-scrub ok 2024-04-02T05:06:50.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.702419+0000 osd.6 (osd.6) 491 : cluster 0 10.12 scrub starts 2024-04-02T05:06:50.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:48.705399+0000 osd.6 (osd.6) 492 : cluster 0 10.12 scrub ok 2024-04-02T05:06:50.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:49.175823+0000 osd.5 (osd.5) 375 : cluster 0 3.5 scrub starts 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:49.180685+0000 osd.5 (osd.5) 376 : cluster 0 3.5 scrub ok 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: audit 2024-04-02T05:06:49.246024+0000 mon.a (mon.0) 4183 : audit 1 from='client.? 172.21.15.69:0/499328061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:49.254042+0000 mon.a (mon.0) 4184 : cluster 0 osdmap e959: 8 total, 8 up, 8 in 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:49.498664+0000 osd.4 (osd.4) 522 : cluster 0 4.1a deep-scrub starts 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: cluster 2024-04-02T05:06:49.501137+0000 osd.4 (osd.4) 523 : cluster 0 4.1a deep-scrub ok 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[28904]: audit 2024-04-02T05:06:49.660108+0000 mon.b (mon.1) 353 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.492183+0000 osd.7 (osd.7) 357 : cluster 0 6.1c deep-scrub starts 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.497959+0000 osd.7 (osd.7) 358 : cluster 0 6.1c deep-scrub ok 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.702419+0000 osd.6 (osd.6) 491 : cluster 0 10.12 scrub starts 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:48.705399+0000 osd.6 (osd.6) 492 : cluster 0 10.12 scrub ok 2024-04-02T05:06:50.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:49.175823+0000 osd.5 (osd.5) 375 : cluster 0 3.5 scrub starts 2024-04-02T05:06:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:49.180685+0000 osd.5 (osd.5) 376 : cluster 0 3.5 scrub ok 2024-04-02T05:06:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: audit 2024-04-02T05:06:49.246024+0000 mon.a (mon.0) 4183 : audit 1 from='client.? 172.21.15.69:0/499328061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-02T05:06:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:49.254042+0000 mon.a (mon.0) 4184 : cluster 0 osdmap e959: 8 total, 8 up, 8 in 2024-04-02T05:06:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:49.498664+0000 osd.4 (osd.4) 522 : cluster 0 4.1a deep-scrub starts 2024-04-02T05:06:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: cluster 2024-04-02T05:06:49.501137+0000 osd.4 (osd.4) 523 : cluster 0 4.1a deep-scrub ok 2024-04-02T05:06:50.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:50 smithi069 bash[21528]: audit 2024-04-02T05:06:49.660108+0000 mon.b (mon.1) 353 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.492183+0000 osd.7 (osd.7) 357 : cluster 0 6.1c deep-scrub starts 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.497959+0000 osd.7 (osd.7) 358 : cluster 0 6.1c deep-scrub ok 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.702419+0000 osd.6 (osd.6) 491 : cluster 0 10.12 scrub starts 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:48.705399+0000 osd.6 (osd.6) 492 : cluster 0 10.12 scrub ok 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:49.175823+0000 osd.5 (osd.5) 375 : cluster 0 3.5 scrub starts 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:49.180685+0000 osd.5 (osd.5) 376 : cluster 0 3.5 scrub ok 2024-04-02T05:06:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: audit 2024-04-02T05:06:49.246024+0000 mon.a (mon.0) 4183 : audit 1 from='client.? 172.21.15.69:0/499328061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-02T05:06:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:49.254042+0000 mon.a (mon.0) 4184 : cluster 0 osdmap e959: 8 total, 8 up, 8 in 2024-04-02T05:06:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:49.498664+0000 osd.4 (osd.4) 522 : cluster 0 4.1a deep-scrub starts 2024-04-02T05:06:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: cluster 2024-04-02T05:06:49.501137+0000 osd.4 (osd.4) 523 : cluster 0 4.1a deep-scrub ok 2024-04-02T05:06:50.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:50 smithi098 bash[23790]: audit 2024-04-02T05:06:49.660108+0000 mon.b (mon.1) 353 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:06:51.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:51 smithi069 bash[21528]: cluster 2024-04-02T05:06:50.474563+0000 mgr.y (mgr.14538) 884 : cluster 0 pgmap v1521: 184 pgs: 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:51.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:51 smithi069 bash[28904]: cluster 2024-04-02T05:06:50.474563+0000 mgr.y (mgr.14538) 884 : cluster 0 pgmap v1521: 184 pgs: 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:51.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:51 smithi098 bash[23790]: cluster 2024-04-02T05:06:50.474563+0000 mgr.y (mgr.14538) 884 : cluster 0 pgmap v1521: 184 pgs: 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:52.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[21528]: cluster 2024-04-02T05:06:50.703940+0000 osd.6 (osd.6) 493 : cluster 0 10.d scrub starts 2024-04-02T05:06:52.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[21528]: cluster 2024-04-02T05:06:50.708553+0000 osd.6 (osd.6) 494 : cluster 0 10.d scrub ok 2024-04-02T05:06:52.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[21528]: cluster 2024-04-02T05:06:51.020830+0000 osd.0 (osd.0) 523 : cluster 0 10.4 deep-scrub starts 2024-04-02T05:06:52.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[21528]: cluster 2024-04-02T05:06:51.023722+0000 osd.0 (osd.0) 524 : cluster 0 10.4 deep-scrub ok 2024-04-02T05:06:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[28904]: cluster 2024-04-02T05:06:50.703940+0000 osd.6 (osd.6) 493 : cluster 0 10.d scrub starts 2024-04-02T05:06:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[28904]: cluster 2024-04-02T05:06:50.708553+0000 osd.6 (osd.6) 494 : cluster 0 10.d scrub ok 2024-04-02T05:06:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[28904]: cluster 2024-04-02T05:06:51.020830+0000 osd.0 (osd.0) 523 : cluster 0 10.4 deep-scrub starts 2024-04-02T05:06:52.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:52 smithi069 bash[28904]: cluster 2024-04-02T05:06:51.023722+0000 osd.0 (osd.0) 524 : cluster 0 10.4 deep-scrub ok 2024-04-02T05:06:52.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:52 smithi098 bash[23790]: cluster 2024-04-02T05:06:50.703940+0000 osd.6 (osd.6) 493 : cluster 0 10.d scrub starts 2024-04-02T05:06:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:52 smithi098 bash[23790]: cluster 2024-04-02T05:06:50.708553+0000 osd.6 (osd.6) 494 : cluster 0 10.d scrub ok 2024-04-02T05:06:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:52 smithi098 bash[23790]: cluster 2024-04-02T05:06:51.020830+0000 osd.0 (osd.0) 523 : cluster 0 10.4 deep-scrub starts 2024-04-02T05:06:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:52 smithi098 bash[23790]: cluster 2024-04-02T05:06:51.023722+0000 osd.0 (osd.0) 524 : cluster 0 10.4 deep-scrub ok 2024-04-02T05:06:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:53 smithi098 bash[23790]: cluster 2024-04-02T05:06:52.475320+0000 mgr.y (mgr.14538) 885 : cluster 0 pgmap v1522: 184 pgs: 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:53.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:53 smithi069 bash[21528]: cluster 2024-04-02T05:06:52.475320+0000 mgr.y (mgr.14538) 885 : cluster 0 pgmap v1522: 184 pgs: 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:53.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:53 smithi069 bash[28904]: cluster 2024-04-02T05:06:52.475320+0000 mgr.y (mgr.14538) 885 : cluster 0 pgmap v1522: 184 pgs: 184 active+clean; 582 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:06:54.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:06:53 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:06:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:54 smithi098 bash[23790]: cluster 2024-04-02T05:06:53.398615+0000 osd.3 (osd.3) 579 : cluster 0 1.c scrub starts 2024-04-02T05:06:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:54 smithi098 bash[23790]: cluster 2024-04-02T05:06:53.401678+0000 osd.3 (osd.3) 580 : cluster 0 1.c scrub ok 2024-04-02T05:06:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:54 smithi098 bash[23790]: cluster 2024-04-02T05:06:53.569217+0000 osd.7 (osd.7) 359 : cluster 0 2.8 scrub starts 2024-04-02T05:06:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:54 smithi098 bash[23790]: cluster 2024-04-02T05:06:53.571814+0000 osd.7 (osd.7) 360 : cluster 0 2.8 scrub ok 2024-04-02T05:06:54.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[28904]: cluster 2024-04-02T05:06:53.398615+0000 osd.3 (osd.3) 579 : cluster 0 1.c scrub starts 2024-04-02T05:06:54.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[28904]: cluster 2024-04-02T05:06:53.401678+0000 osd.3 (osd.3) 580 : cluster 0 1.c scrub ok 2024-04-02T05:06:54.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[28904]: cluster 2024-04-02T05:06:53.569217+0000 osd.7 (osd.7) 359 : cluster 0 2.8 scrub starts 2024-04-02T05:06:54.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[28904]: cluster 2024-04-02T05:06:53.571814+0000 osd.7 (osd.7) 360 : cluster 0 2.8 scrub ok 2024-04-02T05:06:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[21528]: cluster 2024-04-02T05:06:53.398615+0000 osd.3 (osd.3) 579 : cluster 0 1.c scrub starts 2024-04-02T05:06:54.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[21528]: cluster 2024-04-02T05:06:53.401678+0000 osd.3 (osd.3) 580 : cluster 0 1.c scrub ok 2024-04-02T05:06:54.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[21528]: cluster 2024-04-02T05:06:53.569217+0000 osd.7 (osd.7) 359 : cluster 0 2.8 scrub starts 2024-04-02T05:06:54.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:54 smithi069 bash[21528]: cluster 2024-04-02T05:06:53.571814+0000 osd.7 (osd.7) 360 : cluster 0 2.8 scrub ok 2024-04-02T05:06:54.818 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:53.880723+0000 osd.1 (osd.1) 563 : cluster 0 10.14 scrub starts 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:53.883915+0000 osd.1 (osd.1) 564 : cluster 0 10.14 scrub ok 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: audit 2024-04-02T05:06:53.960092+0000 mgr.y (mgr.14538) 886 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.249052+0000 osd.5 (osd.5) 377 : cluster 0 2.3 scrub starts 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.251362+0000 osd.5 (osd.5) 378 : cluster 0 2.3 scrub ok 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.432149+0000 osd.4 (osd.4) 524 : cluster 0 10.1d scrub starts 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.437465+0000 osd.4 (osd.4) 525 : cluster 0 10.1d scrub ok 2024-04-02T05:06:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.477350+0000 mgr.y (mgr.14538) 887 : cluster 0 pgmap v1523: 184 pgs: 184 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.677355+0000 osd.2 (osd.2) 424 : cluster 0 4.1c scrub starts 2024-04-02T05:06:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.679759+0000 osd.2 (osd.2) 425 : cluster 0 4.1c scrub ok 2024-04-02T05:06:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.892537+0000 osd.0 (osd.0) 525 : cluster 0 2.13 deep-scrub starts 2024-04-02T05:06:55.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:55 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.894242+0000 osd.0 (osd.0) 526 : cluster 0 2.13 deep-scrub ok 2024-04-02T05:06:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:53.880723+0000 osd.1 (osd.1) 563 : cluster 0 10.14 scrub starts 2024-04-02T05:06:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:53.883915+0000 osd.1 (osd.1) 564 : cluster 0 10.14 scrub ok 2024-04-02T05:06:55.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: audit 2024-04-02T05:06:53.960092+0000 mgr.y (mgr.14538) 886 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.249052+0000 osd.5 (osd.5) 377 : cluster 0 2.3 scrub starts 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.251362+0000 osd.5 (osd.5) 378 : cluster 0 2.3 scrub ok 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.432149+0000 osd.4 (osd.4) 524 : cluster 0 10.1d scrub starts 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.437465+0000 osd.4 (osd.4) 525 : cluster 0 10.1d scrub ok 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.477350+0000 mgr.y (mgr.14538) 887 : cluster 0 pgmap v1523: 184 pgs: 184 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.677355+0000 osd.2 (osd.2) 424 : cluster 0 4.1c scrub starts 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.679759+0000 osd.2 (osd.2) 425 : cluster 0 4.1c scrub ok 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.892537+0000 osd.0 (osd.0) 525 : cluster 0 2.13 deep-scrub starts 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.894242+0000 osd.0 (osd.0) 526 : cluster 0 2.13 deep-scrub ok 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:53.880723+0000 osd.1 (osd.1) 563 : cluster 0 10.14 scrub starts 2024-04-02T05:06:55.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:53.883915+0000 osd.1 (osd.1) 564 : cluster 0 10.14 scrub ok 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: audit 2024-04-02T05:06:53.960092+0000 mgr.y (mgr.14538) 886 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.249052+0000 osd.5 (osd.5) 377 : cluster 0 2.3 scrub starts 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.251362+0000 osd.5 (osd.5) 378 : cluster 0 2.3 scrub ok 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.432149+0000 osd.4 (osd.4) 524 : cluster 0 10.1d scrub starts 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.437465+0000 osd.4 (osd.4) 525 : cluster 0 10.1d scrub ok 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.477350+0000 mgr.y (mgr.14538) 887 : cluster 0 pgmap v1523: 184 pgs: 184 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.677355+0000 osd.2 (osd.2) 424 : cluster 0 4.1c scrub starts 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.679759+0000 osd.2 (osd.2) 425 : cluster 0 4.1c scrub ok 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.892537+0000 osd.0 (osd.0) 525 : cluster 0 2.13 deep-scrub starts 2024-04-02T05:06:55.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:55 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.894242+0000 osd.0 (osd.0) 526 : cluster 0 2.13 deep-scrub ok 2024-04-02T05:06:56.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.880980+0000 osd.1 (osd.1) 565 : cluster 0 10.13 scrub starts 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:54.885047+0000 osd.1 (osd.1) 566 : cluster 0 10.13 scrub ok 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:55.336927+0000 osd.3 (osd.3) 581 : cluster 0 4.19 scrub starts 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:55.339453+0000 osd.3 (osd.3) 582 : cluster 0 4.19 scrub ok 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:55.552640+0000 osd.7 (osd.7) 361 : cluster 0 3.13 scrub starts 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:55.556792+0000 osd.7 (osd.7) 362 : cluster 0 3.13 scrub ok 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:55.897175+0000 osd.0 (osd.0) 527 : cluster 0 3.17 deep-scrub starts 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[21528]: cluster 2024-04-02T05:06:55.902630+0000 osd.0 (osd.0) 528 : cluster 0 3.17 deep-scrub ok 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.880980+0000 osd.1 (osd.1) 565 : cluster 0 10.13 scrub starts 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:54.885047+0000 osd.1 (osd.1) 566 : cluster 0 10.13 scrub ok 2024-04-02T05:06:56.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:55.336927+0000 osd.3 (osd.3) 581 : cluster 0 4.19 scrub starts 2024-04-02T05:06:56.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:55.339453+0000 osd.3 (osd.3) 582 : cluster 0 4.19 scrub ok 2024-04-02T05:06:56.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:55.552640+0000 osd.7 (osd.7) 361 : cluster 0 3.13 scrub starts 2024-04-02T05:06:56.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:55.556792+0000 osd.7 (osd.7) 362 : cluster 0 3.13 scrub ok 2024-04-02T05:06:56.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:55.897175+0000 osd.0 (osd.0) 527 : cluster 0 3.17 deep-scrub starts 2024-04-02T05:06:56.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:56 smithi069 bash[28904]: cluster 2024-04-02T05:06:55.902630+0000 osd.0 (osd.0) 528 : cluster 0 3.17 deep-scrub ok 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.880980+0000 osd.1 (osd.1) 565 : cluster 0 10.13 scrub starts 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:54.885047+0000 osd.1 (osd.1) 566 : cluster 0 10.13 scrub ok 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:55.336927+0000 osd.3 (osd.3) 581 : cluster 0 4.19 scrub starts 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:55.339453+0000 osd.3 (osd.3) 582 : cluster 0 4.19 scrub ok 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:55.552640+0000 osd.7 (osd.7) 361 : cluster 0 3.13 scrub starts 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:55.556792+0000 osd.7 (osd.7) 362 : cluster 0 3.13 scrub ok 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:55.897175+0000 osd.0 (osd.0) 527 : cluster 0 3.17 deep-scrub starts 2024-04-02T05:06:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:56 smithi098 bash[23790]: cluster 2024-04-02T05:06:55.902630+0000 osd.0 (osd.0) 528 : cluster 0 3.17 deep-scrub ok 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.288412+0000 osd.5 (osd.5) 379 : cluster 0 2.12 scrub starts 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.290824+0000 osd.5 (osd.5) 380 : cluster 0 2.12 scrub ok 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: audit 2024-04-02T05:06:56.463440+0000 mon.b (mon.1) 354 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: audit 2024-04-02T05:06:56.463747+0000 mon.a (mon.0) 4185 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: audit 2024-04-02T05:06:56.464177+0000 mon.b (mon.1) 355 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: audit 2024-04-02T05:06:56.464719+0000 mon.a (mon.0) 4186 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.478128+0000 mgr.y (mgr.14538) 888 : cluster 0 pgmap v1524: 184 pgs: 184 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.486609+0000 osd.4 (osd.4) 526 : cluster 0 4.18 scrub starts 2024-04-02T05:06:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.488785+0000 osd.4 (osd.4) 527 : cluster 0 4.18 scrub ok 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.288412+0000 osd.5 (osd.5) 379 : cluster 0 2.12 scrub starts 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.290824+0000 osd.5 (osd.5) 380 : cluster 0 2.12 scrub ok 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: audit 2024-04-02T05:06:56.463440+0000 mon.b (mon.1) 354 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: audit 2024-04-02T05:06:56.463747+0000 mon.a (mon.0) 4185 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: audit 2024-04-02T05:06:56.464177+0000 mon.b (mon.1) 355 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: audit 2024-04-02T05:06:56.464719+0000 mon.a (mon.0) 4186 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.478128+0000 mgr.y (mgr.14538) 888 : cluster 0 pgmap v1524: 184 pgs: 184 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.486609+0000 osd.4 (osd.4) 526 : cluster 0 4.18 scrub starts 2024-04-02T05:06:57.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:57 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.488785+0000 osd.4 (osd.4) 527 : cluster 0 4.18 scrub ok 2024-04-02T05:06:57.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.288412+0000 osd.5 (osd.5) 379 : cluster 0 2.12 scrub starts 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.290824+0000 osd.5 (osd.5) 380 : cluster 0 2.12 scrub ok 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: audit 2024-04-02T05:06:56.463440+0000 mon.b (mon.1) 354 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: audit 2024-04-02T05:06:56.463747+0000 mon.a (mon.0) 4185 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: audit 2024-04-02T05:06:56.464177+0000 mon.b (mon.1) 355 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: audit 2024-04-02T05:06:56.464719+0000 mon.a (mon.0) 4186 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]: dispatch 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.478128+0000 mgr.y (mgr.14538) 888 : cluster 0 pgmap v1524: 184 pgs: 184 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.486609+0000 osd.4 (osd.4) 526 : cluster 0 4.18 scrub starts 2024-04-02T05:06:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:57 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.488785+0000 osd.4 (osd.4) 527 : cluster 0 4.18 scrub ok 2024-04-02T05:06:57.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- 172.21.15.69:0/3584224457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee006ac50 msgr2=0x7f0ee006b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:57.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- 172.21.15.69:0/3584224457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee006ac50 0x7f0ee006b030 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00098f0 tx=0x7f0ed0009c00 comp rx=0 tx=0).stop 2024-04-02T05:06:57.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- 172.21.15.69:0/3584224457 shutdown_connections 2024-04-02T05:06:57.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- 172.21.15.69:0/3584224457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ee0104920 0x7f0ee010f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:57.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- 172.21.15.69:0/3584224457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee006ac50 0x7f0ee006b030 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- 172.21.15.69:0/3584224457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0103f60 0x7f0ee01043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- 172.21.15.69:0/3584224457 >> 172.21.15.69:0/3584224457 conn(0x7f0ee00771e0 msgr2=0x7f0ee00775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- 172.21.15.69:0/3584224457 shutdown_connections 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- 172.21.15.69:0/3584224457 wait complete. 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 librados: starting msgr at 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 librados: starting objecter 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 Processor -- start 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- start start 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 librados: setting wanted keys 2024-04-02T05:06:57.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 librados: calling monclient init 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ee006ac50 0x7f0ee0198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee0103f60 0x7f0ee0198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 0x7f0ee019cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0111bd0 con 0x7f0ee0103f60 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0111a50 con 0x7f0ee006ac50 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee595a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee0111d50 con 0x7f0ee0104920 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0edf7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 0x7f0ee019cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:57.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee0103f60 0x7f0ee0198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0edf7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 0x7f0ee019cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34546/0 (socket says 172.21.15.69:34546) 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0edf7fe700 1 -- 172.21.15.69:0/680157865 learned_addr learned my addr 172.21.15.69:0/680157865 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0edeffd700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ee006ac50 0x7f0ee0198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 -- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 msgr2=0x7f0ee019cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 0x7f0ee019cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 -- 172.21.15.69:0/680157865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ee006ac50 msgr2=0x7f0ee0198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ee006ac50 0x7f0ee0198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 -- 172.21.15.69:0/680157865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee019d240 con 0x7f0ee0103f60 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0edf7fe700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 0x7f0ee019cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ede7fc700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee0103f60 0x7f0ee0198770 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400e9c0 tx=0x7f0ed400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee4958700 10 client.?.objecter ms_handle_connect 0x7f0ee0103f60 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.832+0000 7f0ee4958700 10 client.?.objecter resend_mon_ops 2024-04-02T05:06:57.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed400c9c0 con 0x7f0ee0103f60 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0ed4010430 con 0x7f0ee0103f60 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee019d530 con 0x7f0ee0103f60 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4018680 con 0x7f0ee0103f60 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee595a700 10 client.33957.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee01a4db0 con 0x7f0ee0103f60 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee595a700 1 librados: init done 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee004ff00 con 0x7f0ee0103f60 2024-04-02T05:06:57.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0ed40105e0 con 0x7f0ee0103f60 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0ec806f5e0 0x7f0ec8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(960..960 src has 907..960) ==== 9015+0+0 (secure 0 0 0) 0x7f0ed408e6b0 con 0x7f0ee0103f60 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 10 client.33957.objecter ms_dispatch 0x7f0ee0001550 osd_map(960..960 src has 907..960) 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 3 client.33957.objecter handle_osd_map got epochs [960,960] > 0 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 3 client.33957.objecter handle_osd_map decoding full epoch 960 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0edeffd700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0ec806f5e0 0x7f0ec8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 20 client.33957.objecter dump_active .. 0 homeless 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0ee4958700 10 client.33957.objecter ms_handle_connect 0x7f0ec806f5e0 2024-04-02T05:06:57.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.836+0000 7f0edeffd700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0ec806f5e0 0x7f0ec8071aa0 secure :-1 s=READY pgs=4049 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009fd0 tx=0x7f0ed0019040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:06:57.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:57.840+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ed405b700 con 0x7f0ee0103f60 2024-04-02T05:06:58.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.060+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) -- 0x7f0ee01a5040 con 0x7f0ee0103f60 2024-04-02T05:06:58.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.320+0000 7f0ee4958700 1 -- 172.21.15.69:0/680157865 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v961) ==== 133+0+0 (secure 0 0 0) 0x7f0ed405f150 con 0x7f0ee0103f60 2024-04-02T05:06:58.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 10 librados: watch_flush enter 2024-04-02T05:06:58.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 10 librados: watch_flush exit 2024-04-02T05:06:58.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 20 client.33957.objecter shutdown clearing up homeless session... 2024-04-02T05:06:58.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 10 client.33957.objecter successfully canceled tick 2024-04-02T05:06:58.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0ec806f5e0 msgr2=0x7f0ec8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:58.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0ec806f5e0 0x7f0ec8071aa0 secure :-1 s=READY pgs=4049 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009fd0 tx=0x7f0ed0019040 comp rx=0 tx=0).stop 2024-04-02T05:06:58.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee0103f60 msgr2=0x7f0ee0198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:06:58.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.324+0000 7f0ee595a700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee0103f60 0x7f0ee0198770 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400e9c0 tx=0x7f0ed400ecd0 comp rx=0 tx=0).stop 2024-04-02T05:06:58.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 shutdown_connections 2024-04-02T05:06:58.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0ec806f5e0 0x7f0ec8071aa0 unknown :-1 s=CLOSED pgs=4049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:58.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ee0104920 0x7f0ee019cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:58.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ee006ac50 0x7f0ee0198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:58.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 --2- 172.21.15.69:0/680157865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee0103f60 0x7f0ee0198770 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:06:58.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 >> 172.21.15.69:0/680157865 conn(0x7f0ee00771e0 msgr2=0x7f0ee0109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:06:58.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 shutdown_connections 2024-04-02T05:06:58.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 -- 172.21.15.69:0/680157865 wait complete. 2024-04-02T05:06:58.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:06:58.328+0000 7f0ee595a700 1 librados: shutdown 2024-04-02T05:06:58.336 INFO:teuthology.orchestra.run.smithi069.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-02T05:06:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.687761+0000 osd.6 (osd.6) 495 : cluster 0 3.a scrub starts 2024-04-02T05:06:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21528]: cluster 2024-04-02T05:06:56.693055+0000 osd.6 (osd.6) 496 : cluster 0 3.a scrub ok 2024-04-02T05:06:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21528]: audit 2024-04-02T05:06:57.311841+0000 mon.a (mon.0) 4187 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-02T05:06:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21528]: audit 2024-04-02T05:06:57.312060+0000 mon.a (mon.0) 4188 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]': finished 2024-04-02T05:06:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21528]: cluster 2024-04-02T05:06:57.326229+0000 mon.a (mon.0) 4189 : cluster 0 osdmap e960: 8 total, 8 up, 8 in 2024-04-02T05:06:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21528]: audit 2024-04-02T05:06:58.068235+0000 mon.a (mon.0) 4190 : audit 1 from='client.? 172.21.15.69:0/680157865' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-02T05:06:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.687761+0000 osd.6 (osd.6) 495 : cluster 0 3.a scrub starts 2024-04-02T05:06:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[28904]: cluster 2024-04-02T05:06:56.693055+0000 osd.6 (osd.6) 496 : cluster 0 3.a scrub ok 2024-04-02T05:06:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[28904]: audit 2024-04-02T05:06:57.311841+0000 mon.a (mon.0) 4187 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-02T05:06:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[28904]: audit 2024-04-02T05:06:57.312060+0000 mon.a (mon.0) 4188 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]': finished 2024-04-02T05:06:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[28904]: cluster 2024-04-02T05:06:57.326229+0000 mon.a (mon.0) 4189 : cluster 0 osdmap e960: 8 total, 8 up, 8 in 2024-04-02T05:06:58.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[28904]: audit 2024-04-02T05:06:58.068235+0000 mon.a (mon.0) 4190 : audit 1 from='client.? 172.21.15.69:0/680157865' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-02T05:06:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:58 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.687761+0000 osd.6 (osd.6) 495 : cluster 0 3.a scrub starts 2024-04-02T05:06:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:58 smithi098 bash[23790]: cluster 2024-04-02T05:06:56.693055+0000 osd.6 (osd.6) 496 : cluster 0 3.a scrub ok 2024-04-02T05:06:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:58 smithi098 bash[23790]: audit 2024-04-02T05:06:57.311841+0000 mon.a (mon.0) 4187 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-02T05:06:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:58 smithi098 bash[23790]: audit 2024-04-02T05:06:57.312060+0000 mon.a (mon.0) 4188 : audit 1 from='mgr.14538 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.15"}]': finished 2024-04-02T05:06:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:58 smithi098 bash[23790]: cluster 2024-04-02T05:06:57.326229+0000 mon.a (mon.0) 4189 : cluster 0 osdmap e960: 8 total, 8 up, 8 in 2024-04-02T05:06:58.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:58 smithi098 bash[23790]: audit 2024-04-02T05:06:58.068235+0000 mon.a (mon.0) 4190 : audit 1 from='client.? 172.21.15.69:0/680157865' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-02T05:06:59.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:06:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:06:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:06:59.194 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-02T05:06:59.194 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-02T05:06:59.194 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-02T05:06:59.194 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-02T05:06:59.195 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-02T05:06:59.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T05:06:59.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:57.707963+0000 osd.6 (osd.6) 497 : cluster 0 10.1c scrub starts 2024-04-02T05:06:59.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:57.710134+0000 osd.6 (osd.6) 498 : cluster 0 10.1c scrub ok 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.248024+0000 osd.5 (osd.5) 381 : cluster 0 4.11 deep-scrub starts 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.250577+0000 osd.5 (osd.5) 382 : cluster 0 4.11 deep-scrub ok 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: audit 2024-04-02T05:06:58.324380+0000 mon.a (mon.0) 4191 : audit 1 from='client.? 172.21.15.69:0/680157865' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.332873+0000 mon.a (mon.0) 4192 : cluster 0 osdmap e961: 8 total, 8 up, 8 in 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.344583+0000 osd.3 (osd.3) 583 : cluster 0 10.15 scrub starts 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.351235+0000 osd.3 (osd.3) 584 : cluster 0 10.15 scrub ok 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.479003+0000 mgr.y (mgr.14538) 889 : cluster 0 pgmap v1527: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.502070+0000 osd.4 (osd.4) 528 : cluster 0 6.2 scrub starts 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[21528]: cluster 2024-04-02T05:06:58.504554+0000 osd.4 (osd.4) 529 : cluster 0 6.2 scrub ok 2024-04-02T05:06:59.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:57.707963+0000 osd.6 (osd.6) 497 : cluster 0 10.1c scrub starts 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:57.710134+0000 osd.6 (osd.6) 498 : cluster 0 10.1c scrub ok 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.248024+0000 osd.5 (osd.5) 381 : cluster 0 4.11 deep-scrub starts 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.250577+0000 osd.5 (osd.5) 382 : cluster 0 4.11 deep-scrub ok 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: audit 2024-04-02T05:06:58.324380+0000 mon.a (mon.0) 4191 : audit 1 from='client.? 172.21.15.69:0/680157865' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.332873+0000 mon.a (mon.0) 4192 : cluster 0 osdmap e961: 8 total, 8 up, 8 in 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.344583+0000 osd.3 (osd.3) 583 : cluster 0 10.15 scrub starts 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.351235+0000 osd.3 (osd.3) 584 : cluster 0 10.15 scrub ok 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.479003+0000 mgr.y (mgr.14538) 889 : cluster 0 pgmap v1527: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.502070+0000 osd.4 (osd.4) 528 : cluster 0 6.2 scrub starts 2024-04-02T05:06:59.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:06:59 smithi069 bash[28904]: cluster 2024-04-02T05:06:58.504554+0000 osd.4 (osd.4) 529 : cluster 0 6.2 scrub ok 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:57.707963+0000 osd.6 (osd.6) 497 : cluster 0 10.1c scrub starts 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:57.710134+0000 osd.6 (osd.6) 498 : cluster 0 10.1c scrub ok 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.248024+0000 osd.5 (osd.5) 381 : cluster 0 4.11 deep-scrub starts 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.250577+0000 osd.5 (osd.5) 382 : cluster 0 4.11 deep-scrub ok 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: audit 2024-04-02T05:06:58.324380+0000 mon.a (mon.0) 4191 : audit 1 from='client.? 172.21.15.69:0/680157865' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.332873+0000 mon.a (mon.0) 4192 : cluster 0 osdmap e961: 8 total, 8 up, 8 in 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.344583+0000 osd.3 (osd.3) 583 : cluster 0 10.15 scrub starts 2024-04-02T05:06:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.351235+0000 osd.3 (osd.3) 584 : cluster 0 10.15 scrub ok 2024-04-02T05:06:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.479003+0000 mgr.y (mgr.14538) 889 : cluster 0 pgmap v1527: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:06:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.502070+0000 osd.4 (osd.4) 528 : cluster 0 6.2 scrub starts 2024-04-02T05:06:59.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:06:59 smithi098 bash[23790]: cluster 2024-04-02T05:06:58.504554+0000 osd.4 (osd.4) 529 : cluster 0 6.2 scrub ok 2024-04-02T05:07:01.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:01 smithi069 bash[21528]: cluster 2024-04-02T05:07:00.364379+0000 osd.3 (osd.3) 585 : cluster 0 6.13 scrub starts 2024-04-02T05:07:01.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:01 smithi069 bash[21528]: cluster 2024-04-02T05:07:00.366330+0000 osd.3 (osd.3) 586 : cluster 0 6.13 scrub ok 2024-04-02T05:07:01.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:01 smithi069 bash[21528]: cluster 2024-04-02T05:07:00.480109+0000 mgr.y (mgr.14538) 890 : cluster 0 pgmap v1528: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:07:01.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:01 smithi069 bash[28904]: cluster 2024-04-02T05:07:00.364379+0000 osd.3 (osd.3) 585 : cluster 0 6.13 scrub starts 2024-04-02T05:07:01.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:01 smithi069 bash[28904]: cluster 2024-04-02T05:07:00.366330+0000 osd.3 (osd.3) 586 : cluster 0 6.13 scrub ok 2024-04-02T05:07:01.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:01 smithi069 bash[28904]: cluster 2024-04-02T05:07:00.480109+0000 mgr.y (mgr.14538) 890 : cluster 0 pgmap v1528: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:07:01.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:01 smithi098 bash[23790]: cluster 2024-04-02T05:07:00.364379+0000 osd.3 (osd.3) 585 : cluster 0 6.13 scrub starts 2024-04-02T05:07:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:01 smithi098 bash[23790]: cluster 2024-04-02T05:07:00.366330+0000 osd.3 (osd.3) 586 : cluster 0 6.13 scrub ok 2024-04-02T05:07:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:01 smithi098 bash[23790]: cluster 2024-04-02T05:07:00.480109+0000 mgr.y (mgr.14538) 890 : cluster 0 pgmap v1528: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:07:02.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:02 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.259467+0000 osd.5 (osd.5) 383 : cluster 0 4.15 scrub starts 2024-04-02T05:07:02.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:02 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.262592+0000 osd.5 (osd.5) 384 : cluster 0 4.15 scrub ok 2024-04-02T05:07:02.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:02 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.259467+0000 osd.5 (osd.5) 383 : cluster 0 4.15 scrub starts 2024-04-02T05:07:02.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:02 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.262592+0000 osd.5 (osd.5) 384 : cluster 0 4.15 scrub ok 2024-04-02T05:07:02.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:02 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.259467+0000 osd.5 (osd.5) 383 : cluster 0 4.15 scrub starts 2024-04-02T05:07:02.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:02 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.262592+0000 osd.5 (osd.5) 384 : cluster 0 4.15 scrub ok 2024-04-02T05:07:03.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:01.747252+0000 osd.6 (osd.6) 499 : cluster 0 10.0 scrub starts 2024-04-02T05:07:03.966 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:01.749585+0000 osd.6 (osd.6) 500 : cluster 0 10.0 scrub ok 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:01.757310+0000 osd.1 (osd.1) 567 : cluster 0 3.0 scrub starts 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:01.762930+0000 osd.1 (osd.1) 568 : cluster 0 3.0 scrub ok 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.480908+0000 mgr.y (mgr.14538) 891 : cluster 0 pgmap v1529: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.877996+0000 osd.7 (osd.7) 363 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.878422+0000 osd.7 (osd.7) 364 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.967520+0000 osd.0 (osd.0) 529 : cluster 0 10.1e scrub starts 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.230035+0000 osd.5 (osd.5) 385 : cluster 0 1.10 deep-scrub starts 2024-04-02T05:07:03.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.232794+0000 osd.5 (osd.5) 386 : cluster 0 1.10 deep-scrub ok 2024-04-02T05:07:03.977 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:03.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:01.747252+0000 osd.6 (osd.6) 499 : cluster 0 10.0 scrub starts 2024-04-02T05:07:03.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:01.749585+0000 osd.6 (osd.6) 500 : cluster 0 10.0 scrub ok 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:01.757310+0000 osd.1 (osd.1) 567 : cluster 0 3.0 scrub starts 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:01.762930+0000 osd.1 (osd.1) 568 : cluster 0 3.0 scrub ok 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.480908+0000 mgr.y (mgr.14538) 891 : cluster 0 pgmap v1529: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.877996+0000 osd.7 (osd.7) 363 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.878422+0000 osd.7 (osd.7) 364 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.967520+0000 osd.0 (osd.0) 529 : cluster 0 10.1e scrub starts 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.230035+0000 osd.5 (osd.5) 385 : cluster 0 1.10 deep-scrub starts 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.232794+0000 osd.5 (osd.5) 386 : cluster 0 1.10 deep-scrub ok 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:01.747252+0000 osd.6 (osd.6) 499 : cluster 0 10.0 scrub starts 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:01.749585+0000 osd.6 (osd.6) 500 : cluster 0 10.0 scrub ok 2024-04-02T05:07:04.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:01.757310+0000 osd.1 (osd.1) 567 : cluster 0 3.0 scrub starts 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:01.762930+0000 osd.1 (osd.1) 568 : cluster 0 3.0 scrub ok 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.480908+0000 mgr.y (mgr.14538) 891 : cluster 0 pgmap v1529: 184 pgs: 2 peering, 182 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.877996+0000 osd.7 (osd.7) 363 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.878422+0000 osd.7 (osd.7) 364 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.967520+0000 osd.0 (osd.0) 529 : cluster 0 10.1e scrub starts 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.230035+0000 osd.5 (osd.5) 385 : cluster 0 1.10 deep-scrub starts 2024-04-02T05:07:04.001 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:03 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.232794+0000 osd.5 (osd.5) 386 : cluster 0 1.10 deep-scrub ok 2024-04-02T05:07:04.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:07:03 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:07:04.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:04 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.708220+0000 osd.6 (osd.6) 501 : cluster 0 2.11 scrub starts 2024-04-02T05:07:04.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:04 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.710517+0000 osd.6 (osd.6) 502 : cluster 0 2.11 scrub ok 2024-04-02T05:07:04.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:04 smithi098 bash[23790]: cluster 2024-04-02T05:07:02.971961+0000 osd.0 (osd.0) 530 : cluster 0 10.1e scrub ok 2024-04-02T05:07:04.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:04 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.536809+0000 osd.4 (osd.4) 530 : cluster 0 4.9 deep-scrub starts 2024-04-02T05:07:04.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:04 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.540121+0000 osd.4 (osd.4) 531 : cluster 0 4.9 deep-scrub ok 2024-04-02T05:07:04.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:04 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.554624+0000 mon.a (mon.0) 4193 : cluster 0 osdmap e962: 8 total, 8 up, 8 in 2024-04-02T05:07:05.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.708220+0000 osd.6 (osd.6) 501 : cluster 0 2.11 scrub starts 2024-04-02T05:07:05.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.710517+0000 osd.6 (osd.6) 502 : cluster 0 2.11 scrub ok 2024-04-02T05:07:05.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[21528]: cluster 2024-04-02T05:07:02.971961+0000 osd.0 (osd.0) 530 : cluster 0 10.1e scrub ok 2024-04-02T05:07:05.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.536809+0000 osd.4 (osd.4) 530 : cluster 0 4.9 deep-scrub starts 2024-04-02T05:07:05.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.540121+0000 osd.4 (osd.4) 531 : cluster 0 4.9 deep-scrub ok 2024-04-02T05:07:05.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.554624+0000 mon.a (mon.0) 4193 : cluster 0 osdmap e962: 8 total, 8 up, 8 in 2024-04-02T05:07:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.708220+0000 osd.6 (osd.6) 501 : cluster 0 2.11 scrub starts 2024-04-02T05:07:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.710517+0000 osd.6 (osd.6) 502 : cluster 0 2.11 scrub ok 2024-04-02T05:07:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[28904]: cluster 2024-04-02T05:07:02.971961+0000 osd.0 (osd.0) 530 : cluster 0 10.1e scrub ok 2024-04-02T05:07:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.536809+0000 osd.4 (osd.4) 530 : cluster 0 4.9 deep-scrub starts 2024-04-02T05:07:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.540121+0000 osd.4 (osd.4) 531 : cluster 0 4.9 deep-scrub ok 2024-04-02T05:07:05.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:04 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.554624+0000 mon.a (mon.0) 4193 : cluster 0 osdmap e962: 8 total, 8 up, 8 in 2024-04-02T05:07:05.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.692148+0000 osd.6 (osd.6) 503 : cluster 0 2.14 deep-scrub starts 2024-04-02T05:07:05.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:03.694631+0000 osd.6 (osd.6) 504 : cluster 0 2.14 deep-scrub ok 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: audit 2024-04-02T05:07:03.964039+0000 mgr.y (mgr.14538) 892 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:04.458251+0000 osd.3 (osd.3) 587 : cluster 0 4.17 scrub starts 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:04.460497+0000 osd.3 (osd.3) 588 : cluster 0 4.17 scrub ok 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:04.481942+0000 mgr.y (mgr.14538) 893 : cluster 0 pgmap v1531: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:04.513124+0000 osd.4 (osd.4) 532 : cluster 0 4.d scrub starts 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: cluster 2024-04-02T05:07:04.516454+0000 osd.4 (osd.4) 533 : cluster 0 4.d scrub ok 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[21528]: audit 2024-04-02T05:07:04.693417+0000 mon.b (mon.1) 356 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.692148+0000 osd.6 (osd.6) 503 : cluster 0 2.14 deep-scrub starts 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:03.694631+0000 osd.6 (osd.6) 504 : cluster 0 2.14 deep-scrub ok 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: audit 2024-04-02T05:07:03.964039+0000 mgr.y (mgr.14538) 892 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:04.458251+0000 osd.3 (osd.3) 587 : cluster 0 4.17 scrub starts 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:04.460497+0000 osd.3 (osd.3) 588 : cluster 0 4.17 scrub ok 2024-04-02T05:07:05.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:04.481942+0000 mgr.y (mgr.14538) 893 : cluster 0 pgmap v1531: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:04.513124+0000 osd.4 (osd.4) 532 : cluster 0 4.d scrub starts 2024-04-02T05:07:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: cluster 2024-04-02T05:07:04.516454+0000 osd.4 (osd.4) 533 : cluster 0 4.d scrub ok 2024-04-02T05:07:05.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:05 smithi069 bash[28904]: audit 2024-04-02T05:07:04.693417+0000 mon.b (mon.1) 356 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.692148+0000 osd.6 (osd.6) 503 : cluster 0 2.14 deep-scrub starts 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:03.694631+0000 osd.6 (osd.6) 504 : cluster 0 2.14 deep-scrub ok 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: audit 2024-04-02T05:07:03.964039+0000 mgr.y (mgr.14538) 892 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:04.458251+0000 osd.3 (osd.3) 587 : cluster 0 4.17 scrub starts 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:04.460497+0000 osd.3 (osd.3) 588 : cluster 0 4.17 scrub ok 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:04.481942+0000 mgr.y (mgr.14538) 893 : cluster 0 pgmap v1531: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:04.513124+0000 osd.4 (osd.4) 532 : cluster 0 4.d scrub starts 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: cluster 2024-04-02T05:07:04.516454+0000 osd.4 (osd.4) 533 : cluster 0 4.d scrub ok 2024-04-02T05:07:05.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:05 smithi098 bash[23790]: audit 2024-04-02T05:07:04.693417+0000 mon.b (mon.1) 356 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:06.896 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[28904]: cluster 2024-04-02T05:07:05.541132+0000 osd.4 (osd.4) 534 : cluster 0 6.1a scrub starts 2024-04-02T05:07:06.896 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[28904]: cluster 2024-04-02T05:07:05.543515+0000 osd.4 (osd.4) 535 : cluster 0 6.1a scrub ok 2024-04-02T05:07:06.896 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[28904]: cluster 2024-04-02T05:07:05.653380+0000 osd.0 (osd.0) 531 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:06.896 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[28904]: cluster 2024-04-02T05:07:05.653738+0000 osd.0 (osd.0) 532 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:06.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[21528]: cluster 2024-04-02T05:07:05.541132+0000 osd.4 (osd.4) 534 : cluster 0 6.1a scrub starts 2024-04-02T05:07:06.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[21528]: cluster 2024-04-02T05:07:05.543515+0000 osd.4 (osd.4) 535 : cluster 0 6.1a scrub ok 2024-04-02T05:07:06.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[21528]: cluster 2024-04-02T05:07:05.653380+0000 osd.0 (osd.0) 531 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:06.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:06 smithi069 bash[21528]: cluster 2024-04-02T05:07:05.653738+0000 osd.0 (osd.0) 532 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:06.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:06 smithi098 bash[23790]: cluster 2024-04-02T05:07:05.541132+0000 osd.4 (osd.4) 534 : cluster 0 6.1a scrub starts 2024-04-02T05:07:06.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:06 smithi098 bash[23790]: cluster 2024-04-02T05:07:05.543515+0000 osd.4 (osd.4) 535 : cluster 0 6.1a scrub ok 2024-04-02T05:07:06.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:06 smithi098 bash[23790]: cluster 2024-04-02T05:07:05.653380+0000 osd.0 (osd.0) 531 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:06.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:06 smithi098 bash[23790]: cluster 2024-04-02T05:07:05.653738+0000 osd.0 (osd.0) 532 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- 172.21.15.69:0/571838473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c100790 msgr2=0x7f114c100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- 172.21.15.69:0/571838473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c100790 0x7f114c100c10 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f11400099c0 tx=0x7f1140009cd0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- 172.21.15.69:0/571838473 shutdown_connections 2024-04-02T05:07:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- 172.21.15.69:0/571838473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f114c101150 0x7f114c10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- 172.21.15.69:0/571838473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c107260 0x7f114c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- 172.21.15.69:0/571838473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c100790 0x7f114c100c10 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- 172.21.15.69:0/571838473 >> 172.21.15.69:0/571838473 conn(0x7f114c077740 msgr2=0x7f114c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- 172.21.15.69:0/571838473 shutdown_connections 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- 172.21.15.69:0/571838473 wait complete. 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 librados: starting msgr at 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 librados: starting objecter 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 Processor -- start 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- start start 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 librados: setting wanted keys 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 librados: calling monclient init 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f114c100790 0x7f114c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 0x7f114c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 0x7f114c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111be0 con 0x7f114c101150 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111a60 con 0x7f114c100790 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1153df9700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f114c111d60 con 0x7f114c107260 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1151394700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 0x7f114c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1152396700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 0x7f114c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1151394700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 0x7f114c198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49590/0 (socket says 172.21.15.69:49590) 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1152396700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 0x7f114c19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34556/0 (socket says 172.21.15.69:34556) 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1151394700 1 -- 172.21.15.69:0/1743352194 learned_addr learned my addr 172.21.15.69:0/1743352194 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.064+0000 7f1151b95700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f114c100790 0x7f114c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151394700 1 -- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 msgr2=0x7f114c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151394700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 0x7f114c19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151394700 1 -- 172.21.15.69:0/1743352194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f114c100790 msgr2=0x7f114c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151394700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f114c100790 0x7f114c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151394700 1 -- 172.21.15.69:0/1743352194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c19d0e0 con 0x7f114c101150 2024-04-02T05:07:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1152396700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 0x7f114c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:07:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151394700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 0x7f114c198610 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f1140005910 tx=0x7f114000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 10 client.?.objecter ms_handle_connect 0x7f114c101150 2024-04-02T05:07:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140027070 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c19d370 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f114000fc90 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140022c90 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1153df9700 10 client.33963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c1a4c50 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1153df9700 1 librados: init done 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f114c04ff00 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f114002a070 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f113406f5e0 0x7f1134071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(963..963 src has 907..963) ==== 9015+0+0 (secure 0 0 0) 0x7f114008f300 con 0x7f114c101150 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 10 client.33963.objecter ms_dispatch 0x7f114c001550 osd_map(963..963 src has 907..963) 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 3 client.33963.objecter handle_osd_map got epochs [963,963] > 0 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 3 client.33963.objecter handle_osd_map decoding full epoch 963 2024-04-02T05:07:07.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151b95700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f113406f5e0 0x7f1134071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 20 client.33963.objecter dump_active .. 0 homeless 2024-04-02T05:07:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f114affd700 10 client.33963.objecter ms_handle_connect 0x7f113406f5e0 2024-04-02T05:07:07.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.068+0000 7f1151b95700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f113406f5e0 0x7f1134071aa0 secure :-1 s=READY pgs=4050 cs=0 l=1 rev1=1 crypto rx=0x7f1144009da0 tx=0x7f114400b040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:07.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.072+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1140013070 con 0x7f114c101150 2024-04-02T05:07:07.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.292+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f114c1a4e20 con 0x7f114c101150 2024-04-02T05:07:07.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.292+0000 7f114affd700 1 -- 172.21.15.69:0/1743352194 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v963) ==== 74+0+26650 (secure 0 0 0) 0x7f114005c3f0 con 0x7f114c101150 2024-04-02T05:07:07.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T05:07:07.300 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":963,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:33:43.348385+0000","modified":"2024-04-02T05:07:06.558740+0000","last_up_change":"2024-04-02T04:53:53.913436+0000","last_in_change":"2024-04-02T05:04:10.153375+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":114,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.671714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"908","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"731","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5199999809265137,"score_stable":1.5199999809265137,"optimal_score":1,"raw_score_acting":1.5199999809265137,"raw_score_stable":1.5199999809265137,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-02T04:43:52.801763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-02T04:43:54.772439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-04-02T04:43:56.696111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-04-02T04:43:57.254108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-02T04:43:58.846023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-02T04:49:11.754451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b13e248-930e-40ca-9bb0-e8c5ce16bd0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":960,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6803","nonce":1087500595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6805","nonce":1087500595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6809","nonce":1087500595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6807","nonce":1087500595}]},"public_addr":"172.21.15.69:6803/1087500595","cluster_addr":"172.21.15.69:6805/1087500595","heartbeat_back_addr":"172.21.15.69:6809/1087500595","heartbeat_front_addr":"172.21.15.69:6807/1087500595","state":["exists","up"]},{"osd":1,"uuid":"afb71182-d4f7-416d-ae44-90f28eab4b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":955,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6811","nonce":2651666015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6813","nonce":2651666015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6817","nonce":2651666015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6815","nonce":2651666015}]},"public_addr":"172.21.15.69:6811/2651666015","cluster_addr":"172.21.15.69:6813/2651666015","heartbeat_back_addr":"172.21.15.69:6817/2651666015","heartbeat_front_addr":"172.21.15.69:6815/2651666015","state":["exists","up"]},{"osd":2,"uuid":"523bcbd7-d389-4b8f-aca8-88b71ce61de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":943,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6819","nonce":619998764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6821","nonce":619998764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6825","nonce":619998764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6823","nonce":619998764}]},"public_addr":"172.21.15.69:6819/619998764","cluster_addr":"172.21.15.69:6821/619998764","heartbeat_back_addr":"172.21.15.69:6825/619998764","heartbeat_front_addr":"172.21.15.69:6823/619998764","state":["exists","up"]},{"osd":3,"uuid":"20e4526d-2707-44d0-b13a-af36019c42ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":952,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6827","nonce":207593208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6829","nonce":207593208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6833","nonce":207593208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6831","nonce":207593208}]},"public_addr":"172.21.15.69:6827/207593208","cluster_addr":"172.21.15.69:6829/207593208","heartbeat_back_addr":"172.21.15.69:6833/207593208","heartbeat_front_addr":"172.21.15.69:6831/207593208","state":["exists","up"]},{"osd":4,"uuid":"aa3bc4f3-4476-4ba1-b033-428e5edf5852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":943,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6801","nonce":2065947829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6803","nonce":2065947829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6807","nonce":2065947829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6805","nonce":2065947829}]},"public_addr":"172.21.15.98:6801/2065947829","cluster_addr":"172.21.15.98:6803/2065947829","heartbeat_back_addr":"172.21.15.98:6807/2065947829","heartbeat_front_addr":"172.21.15.98:6805/2065947829","state":["exists","up"]},{"osd":5,"uuid":"902326af-4a40-47c8-a3b8-d5d5536f90fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":960,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6809","nonce":219578185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6811","nonce":219578185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6815","nonce":219578185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6813","nonce":219578185}]},"public_addr":"172.21.15.98:6809/219578185","cluster_addr":"172.21.15.98:6811/219578185","heartbeat_back_addr":"172.21.15.98:6815/219578185","heartbeat_front_addr":"172.21.15.98:6813/219578185","state":["exists","up"]},{"osd":6,"uuid":"8c73e4b0-cb6b-486a-8b55-a0fe4d97a727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":952,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6817","nonce":1226172499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6819","nonce":1226172499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6823","nonce":1226172499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6821","nonce":1226172499}]},"public_addr":"172.21.15.98:6817/1226172499","cluster_addr":"172.21.15.98:6819/1226172499","heartbeat_back_addr":"172.21.15.98:6823/1226172499","heartbeat_front_addr":"172.21.15.98:6821/1226172499","state":["exists","up"]},{"osd":7,"uuid":"4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":69,"last_clean_end":152,"up_from":394,"up_thru":952,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6825","nonce":575416324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6827","nonce":575416324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6831","nonce":575416324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6829","nonce":575416324}]},"public_addr":"172.21.15.98:6825/575416324","cluster_addr":"172.21.15.98:6827/575416324","heartbeat_back_addr":"172.21.15.98:6831/575416324","heartbeat_front_addr":"172.21.15.98:6829/575416324","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:05.653743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:48.446977+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:05:46.345207+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:06.461474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:02.669368+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:45.237333+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:15.381429+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:02.878427+0000","dead_epoch":150}],"pg_upmap":[{"pgid":"10.8","osds":[5,1,7]}],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":5,"to":1}]},{"pgid":"1.a","mappings":[{"from":5,"to":4}]},{"pgid":"2.9","mappings":[{"from":7,"to":2}]},{"pgid":"3.f","mappings":[{"from":4,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":3,"to":5}]},{"pgid":"10.e","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/2738867930":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/1798147282":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/52352690":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/2928651991":"2024-04-03T04:34:47.037764+0000","172.21.15.69:6800/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/2763215770":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/3657409442":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/879855771":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1025808168":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1168952113":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/1892956550":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6801/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/54578327":"2024-04-03T04:44:51.699278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":3,"length":4}]},{"pool":36,"snaps":[{"begin":3,"length":2}]},{"pool":38,"snaps":[{"begin":1,"length":1}]},{"pool":66,"snaps":[{"begin":1,"length":1}]},{"pool":101,"snaps":[{"begin":2,"length":4},{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":112,"snaps":[{"begin":2,"length":9},{"begin":14,"length":1}]},{"pool":120,"snaps":[{"begin":1,"length":2}]},{"pool":137,"snaps":[{"begin":3,"length":1}]},{"pool":145,"snaps":[{"begin":1,"length":4}]},{"pool":146,"snaps":[{"begin":2,"length":1}]},{"pool":154,"snaps":[{"begin":2,"length":2}]},{"pool":164,"snaps":[{"begin":2,"length":3}]},{"pool":165,"snaps":[{"begin":3,"length":4}]},{"pool":173,"snaps":[{"begin":2,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":2}]},{"pool":188,"snaps":[{"begin":3,"length":1}]},{"pool":189,"snaps":[{"begin":2,"length":1}]},{"pool":196,"snaps":[{"begin":1,"length":4}]},{"pool":209,"snaps":[{"begin":2,"length":6}]},{"pool":214,"snaps":[{"begin":7,"length":1}]},{"pool":234,"snaps":[{"begin":8,"length":1}]},{"pool":245,"snaps":[{"begin":10,"length":1}]},{"pool":251,"snaps":[{"begin":11,"length":2}]},{"pool":262,"snaps":[{"begin":14,"length":1}]},{"pool":326,"snaps":[{"begin":2,"length":1}]},{"pool":327,"snaps":[{"begin":3,"length":1}]},{"pool":330,"snaps":[{"begin":4,"length":1}]},{"pool":334,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T05:07:07.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 10 librados: watch_flush enter 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 10 librados: watch_flush exit 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 20 client.33963.objecter shutdown clearing up homeless session... 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 10 client.33963.objecter successfully canceled tick 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f113406f5e0 msgr2=0x7f1134071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f113406f5e0 0x7f1134071aa0 secure :-1 s=READY pgs=4050 cs=0 l=1 rev1=1 crypto rx=0x7f1144009da0 tx=0x7f114400b040 comp rx=0 tx=0).stop 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 msgr2=0x7f114c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 0x7f114c198610 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f1140005910 tx=0x7f114000fa00 comp rx=0 tx=0).stop 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 shutdown_connections 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f113406f5e0 0x7f1134071aa0 unknown :-1 s=CLOSED pgs=4050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f114c107260 0x7f114c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f114c100790 0x7f114c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 --2- 172.21.15.69:0/1743352194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f114c101150 0x7f114c198610 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 >> 172.21.15.69:0/1743352194 conn(0x7f114c077740 msgr2=0x7f114c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 shutdown_connections 2024-04-02T05:07:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 -- 172.21.15.69:0/1743352194 wait complete. 2024-04-02T05:07:07.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:07.296+0000 7f1153df9700 1 librados: shutdown 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.461113+0000 osd.3 (osd.3) 589 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.461469+0000 osd.3 (osd.3) 590 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.482694+0000 mgr.y (mgr.14538) 894 : cluster 0 pgmap v1532: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 627 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.578158+0000 mon.a (mon.0) 4194 : cluster 0 osdmap e963: 8 total, 8 up, 8 in 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.720393+0000 osd.6 (osd.6) 505 : cluster 0 5.2 deep-scrub starts 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.725606+0000 osd.6 (osd.6) 506 : cluster 0 5.2 deep-scrub ok 2024-04-02T05:07:07.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:07 smithi098 bash[23790]: audit 2024-04-02T05:07:07.296730+0000 mon.a (mon.0) 4195 : audit 0 from='client.? 172.21.15.69:0/1743352194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T05:07:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.461113+0000 osd.3 (osd.3) 589 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.461469+0000 osd.3 (osd.3) 590 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.482694+0000 mgr.y (mgr.14538) 894 : cluster 0 pgmap v1532: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 627 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.578158+0000 mon.a (mon.0) 4194 : cluster 0 osdmap e963: 8 total, 8 up, 8 in 2024-04-02T05:07:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.720393+0000 osd.6 (osd.6) 505 : cluster 0 5.2 deep-scrub starts 2024-04-02T05:07:08.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.725606+0000 osd.6 (osd.6) 506 : cluster 0 5.2 deep-scrub ok 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[28904]: audit 2024-04-02T05:07:07.296730+0000 mon.a (mon.0) 4195 : audit 0 from='client.? 172.21.15.69:0/1743352194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.461113+0000 osd.3 (osd.3) 589 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.461469+0000 osd.3 (osd.3) 590 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.482694+0000 mgr.y (mgr.14538) 894 : cluster 0 pgmap v1532: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 627 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.578158+0000 mon.a (mon.0) 4194 : cluster 0 osdmap e963: 8 total, 8 up, 8 in 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.720393+0000 osd.6 (osd.6) 505 : cluster 0 5.2 deep-scrub starts 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.725606+0000 osd.6 (osd.6) 506 : cluster 0 5.2 deep-scrub ok 2024-04-02T05:07:08.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:07 smithi069 bash[21528]: audit 2024-04-02T05:07:07.296730+0000 mon.a (mon.0) 4195 : audit 0 from='client.? 172.21.15.69:0/1743352194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T05:07:08.150 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-02T05:07:08.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T05:07:08.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:08 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.994701+0000 osd.0 (osd.0) 533 : cluster 0 3.1b scrub starts 2024-04-02T05:07:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:08 smithi098 bash[23790]: cluster 2024-04-02T05:07:06.996887+0000 osd.0 (osd.0) 534 : cluster 0 3.1b scrub ok 2024-04-02T05:07:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:08 smithi098 bash[23790]: cluster 2024-04-02T05:07:07.578028+0000 osd.2 (osd.2) 426 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:08 smithi098 bash[23790]: cluster 2024-04-02T05:07:07.578541+0000 osd.2 (osd.2) 427 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:08 smithi098 bash[23790]: cluster 2024-04-02T05:07:07.749215+0000 osd.2 (osd.2) 428 : cluster 0 2.1 scrub starts 2024-04-02T05:07:08.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:08 smithi098 bash[23790]: cluster 2024-04-02T05:07:07.752124+0000 osd.2 (osd.2) 429 : cluster 0 2.1 scrub ok 2024-04-02T05:07:09.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:07:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:07:09.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.994701+0000 osd.0 (osd.0) 533 : cluster 0 3.1b scrub starts 2024-04-02T05:07:09.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21528]: cluster 2024-04-02T05:07:06.996887+0000 osd.0 (osd.0) 534 : cluster 0 3.1b scrub ok 2024-04-02T05:07:09.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21528]: cluster 2024-04-02T05:07:07.578028+0000 osd.2 (osd.2) 426 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:09.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21528]: cluster 2024-04-02T05:07:07.578541+0000 osd.2 (osd.2) 427 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:09.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21528]: cluster 2024-04-02T05:07:07.749215+0000 osd.2 (osd.2) 428 : cluster 0 2.1 scrub starts 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[21528]: cluster 2024-04-02T05:07:07.752124+0000 osd.2 (osd.2) 429 : cluster 0 2.1 scrub ok 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.994701+0000 osd.0 (osd.0) 533 : cluster 0 3.1b scrub starts 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[28904]: cluster 2024-04-02T05:07:06.996887+0000 osd.0 (osd.0) 534 : cluster 0 3.1b scrub ok 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[28904]: cluster 2024-04-02T05:07:07.578028+0000 osd.2 (osd.2) 426 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[28904]: cluster 2024-04-02T05:07:07.578541+0000 osd.2 (osd.2) 427 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[28904]: cluster 2024-04-02T05:07:07.749215+0000 osd.2 (osd.2) 428 : cluster 0 2.1 scrub starts 2024-04-02T05:07:09.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:08 smithi069 bash[28904]: cluster 2024-04-02T05:07:07.752124+0000 osd.2 (osd.2) 429 : cluster 0 2.1 scrub ok 2024-04-02T05:07:09.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:09 smithi098 bash[23790]: cluster 2024-04-02T05:07:07.747472+0000 osd.1 (osd.1) 569 : cluster 0 4.7 scrub starts 2024-04-02T05:07:09.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:09 smithi098 bash[23790]: cluster 2024-04-02T05:07:07.750503+0000 osd.1 (osd.1) 570 : cluster 0 4.7 scrub ok 2024-04-02T05:07:09.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:09 smithi098 bash[23790]: cluster 2024-04-02T05:07:08.484576+0000 mgr.y (mgr.14538) 895 : cluster 0 pgmap v1534: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:09.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:09 smithi098 bash[23790]: cluster 2024-04-02T05:07:08.506436+0000 osd.3 (osd.3) 591 : cluster 0 3.b scrub starts 2024-04-02T05:07:09.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:09 smithi098 bash[23790]: cluster 2024-04-02T05:07:08.510620+0000 osd.3 (osd.3) 592 : cluster 0 3.b scrub ok 2024-04-02T05:07:09.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:09 smithi098 bash[23790]: cluster 2024-04-02T05:07:08.586991+0000 mon.a (mon.0) 4196 : cluster 0 osdmap e964: 8 total, 8 up, 8 in 2024-04-02T05:07:10.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[21528]: cluster 2024-04-02T05:07:07.747472+0000 osd.1 (osd.1) 569 : cluster 0 4.7 scrub starts 2024-04-02T05:07:10.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[21528]: cluster 2024-04-02T05:07:07.750503+0000 osd.1 (osd.1) 570 : cluster 0 4.7 scrub ok 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[21528]: cluster 2024-04-02T05:07:08.484576+0000 mgr.y (mgr.14538) 895 : cluster 0 pgmap v1534: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[21528]: cluster 2024-04-02T05:07:08.506436+0000 osd.3 (osd.3) 591 : cluster 0 3.b scrub starts 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[21528]: cluster 2024-04-02T05:07:08.510620+0000 osd.3 (osd.3) 592 : cluster 0 3.b scrub ok 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[21528]: cluster 2024-04-02T05:07:08.586991+0000 mon.a (mon.0) 4196 : cluster 0 osdmap e964: 8 total, 8 up, 8 in 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[28904]: cluster 2024-04-02T05:07:07.747472+0000 osd.1 (osd.1) 569 : cluster 0 4.7 scrub starts 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[28904]: cluster 2024-04-02T05:07:07.750503+0000 osd.1 (osd.1) 570 : cluster 0 4.7 scrub ok 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[28904]: cluster 2024-04-02T05:07:08.484576+0000 mgr.y (mgr.14538) 895 : cluster 0 pgmap v1534: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[28904]: cluster 2024-04-02T05:07:08.506436+0000 osd.3 (osd.3) 591 : cluster 0 3.b scrub starts 2024-04-02T05:07:10.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[28904]: cluster 2024-04-02T05:07:08.510620+0000 osd.3 (osd.3) 592 : cluster 0 3.b scrub ok 2024-04-02T05:07:10.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:09 smithi069 bash[28904]: cluster 2024-04-02T05:07:08.586991+0000 mon.a (mon.0) 4196 : cluster 0 osdmap e964: 8 total, 8 up, 8 in 2024-04-02T05:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:10 smithi098 bash[23790]: cluster 2024-04-02T05:07:08.756936+0000 osd.1 (osd.1) 571 : cluster 0 2.6 deep-scrub starts 2024-04-02T05:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:10 smithi098 bash[23790]: cluster 2024-04-02T05:07:08.760657+0000 osd.1 (osd.1) 572 : cluster 0 2.6 deep-scrub ok 2024-04-02T05:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:10 smithi098 bash[23790]: cluster 2024-04-02T05:07:09.905455+0000 osd.0 (osd.0) 535 : cluster 0 3.1f scrub starts 2024-04-02T05:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:10 smithi098 bash[23790]: cluster 2024-04-02T05:07:09.912155+0000 osd.0 (osd.0) 536 : cluster 0 3.1f scrub ok 2024-04-02T05:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:10 smithi098 bash[23790]: cluster 2024-04-02T05:07:10.224517+0000 osd.5 (osd.5) 387 : cluster 0 10.9 scrub starts 2024-04-02T05:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:10 smithi098 bash[23790]: cluster 2024-04-02T05:07:10.228625+0000 osd.5 (osd.5) 388 : cluster 0 10.9 scrub ok 2024-04-02T05:07:11.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[21528]: cluster 2024-04-02T05:07:08.756936+0000 osd.1 (osd.1) 571 : cluster 0 2.6 deep-scrub starts 2024-04-02T05:07:11.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[21528]: cluster 2024-04-02T05:07:08.760657+0000 osd.1 (osd.1) 572 : cluster 0 2.6 deep-scrub ok 2024-04-02T05:07:11.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[21528]: cluster 2024-04-02T05:07:09.905455+0000 osd.0 (osd.0) 535 : cluster 0 3.1f scrub starts 2024-04-02T05:07:11.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[21528]: cluster 2024-04-02T05:07:09.912155+0000 osd.0 (osd.0) 536 : cluster 0 3.1f scrub ok 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[21528]: cluster 2024-04-02T05:07:10.224517+0000 osd.5 (osd.5) 387 : cluster 0 10.9 scrub starts 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[21528]: cluster 2024-04-02T05:07:10.228625+0000 osd.5 (osd.5) 388 : cluster 0 10.9 scrub ok 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[28904]: cluster 2024-04-02T05:07:08.756936+0000 osd.1 (osd.1) 571 : cluster 0 2.6 deep-scrub starts 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[28904]: cluster 2024-04-02T05:07:08.760657+0000 osd.1 (osd.1) 572 : cluster 0 2.6 deep-scrub ok 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[28904]: cluster 2024-04-02T05:07:09.905455+0000 osd.0 (osd.0) 535 : cluster 0 3.1f scrub starts 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[28904]: cluster 2024-04-02T05:07:09.912155+0000 osd.0 (osd.0) 536 : cluster 0 3.1f scrub ok 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[28904]: cluster 2024-04-02T05:07:10.224517+0000 osd.5 (osd.5) 387 : cluster 0 10.9 scrub starts 2024-04-02T05:07:11.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:10 smithi069 bash[28904]: cluster 2024-04-02T05:07:10.228625+0000 osd.5 (osd.5) 388 : cluster 0 10.9 scrub ok 2024-04-02T05:07:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:11 smithi098 bash[23790]: cluster 2024-04-02T05:07:09.785194+0000 osd.1 (osd.1) 573 : cluster 0 6.1 scrub starts 2024-04-02T05:07:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:11 smithi098 bash[23790]: cluster 2024-04-02T05:07:09.788071+0000 osd.1 (osd.1) 574 : cluster 0 6.1 scrub ok 2024-04-02T05:07:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:11 smithi098 bash[23790]: cluster 2024-04-02T05:07:10.485781+0000 mgr.y (mgr.14538) 896 : cluster 0 pgmap v1536: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-02T05:07:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:11 smithi098 bash[23790]: cluster 2024-04-02T05:07:10.537362+0000 osd.7 (osd.7) 365 : cluster 0 6.8 scrub starts 2024-04-02T05:07:11.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:11 smithi098 bash[23790]: cluster 2024-04-02T05:07:10.539866+0000 osd.7 (osd.7) 366 : cluster 0 6.8 scrub ok 2024-04-02T05:07:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[21528]: cluster 2024-04-02T05:07:09.785194+0000 osd.1 (osd.1) 573 : cluster 0 6.1 scrub starts 2024-04-02T05:07:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[21528]: cluster 2024-04-02T05:07:09.788071+0000 osd.1 (osd.1) 574 : cluster 0 6.1 scrub ok 2024-04-02T05:07:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[21528]: cluster 2024-04-02T05:07:10.485781+0000 mgr.y (mgr.14538) 896 : cluster 0 pgmap v1536: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-02T05:07:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[21528]: cluster 2024-04-02T05:07:10.537362+0000 osd.7 (osd.7) 365 : cluster 0 6.8 scrub starts 2024-04-02T05:07:12.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[21528]: cluster 2024-04-02T05:07:10.539866+0000 osd.7 (osd.7) 366 : cluster 0 6.8 scrub ok 2024-04-02T05:07:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[28904]: cluster 2024-04-02T05:07:09.785194+0000 osd.1 (osd.1) 573 : cluster 0 6.1 scrub starts 2024-04-02T05:07:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[28904]: cluster 2024-04-02T05:07:09.788071+0000 osd.1 (osd.1) 574 : cluster 0 6.1 scrub ok 2024-04-02T05:07:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[28904]: cluster 2024-04-02T05:07:10.485781+0000 mgr.y (mgr.14538) 896 : cluster 0 pgmap v1536: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-02T05:07:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[28904]: cluster 2024-04-02T05:07:10.537362+0000 osd.7 (osd.7) 365 : cluster 0 6.8 scrub starts 2024-04-02T05:07:12.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:11 smithi069 bash[28904]: cluster 2024-04-02T05:07:10.539866+0000 osd.7 (osd.7) 366 : cluster 0 6.8 scrub ok 2024-04-02T05:07:12.930 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:12.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:12 smithi069 bash[21528]: cluster 2024-04-02T05:07:11.542817+0000 osd.7 (osd.7) 367 : cluster 0 6.15 scrub starts 2024-04-02T05:07:12.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:12 smithi069 bash[21528]: cluster 2024-04-02T05:07:11.545186+0000 osd.7 (osd.7) 368 : cluster 0 6.15 scrub ok 2024-04-02T05:07:12.951 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:12 smithi069 bash[28904]: cluster 2024-04-02T05:07:11.542817+0000 osd.7 (osd.7) 367 : cluster 0 6.15 scrub starts 2024-04-02T05:07:12.951 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:12 smithi069 bash[28904]: cluster 2024-04-02T05:07:11.545186+0000 osd.7 (osd.7) 368 : cluster 0 6.15 scrub ok 2024-04-02T05:07:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:12 smithi098 bash[23790]: cluster 2024-04-02T05:07:11.542817+0000 osd.7 (osd.7) 367 : cluster 0 6.15 scrub starts 2024-04-02T05:07:12.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:12 smithi098 bash[23790]: cluster 2024-04-02T05:07:11.545186+0000 osd.7 (osd.7) 368 : cluster 0 6.15 scrub ok 2024-04-02T05:07:13.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:13 smithi098 bash[23790]: cluster 2024-04-02T05:07:12.486600+0000 mgr.y (mgr.14538) 897 : cluster 0 pgmap v1537: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:07:13.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:13 smithi069 bash[21528]: cluster 2024-04-02T05:07:12.486600+0000 mgr.y (mgr.14538) 897 : cluster 0 pgmap v1537: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:07:13.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:13 smithi069 bash[28904]: cluster 2024-04-02T05:07:12.486600+0000 mgr.y (mgr.14538) 897 : cluster 0 pgmap v1537: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:07:14.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:07:13 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:07:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[21528]: cluster 2024-04-02T05:07:12.768393+0000 osd.1 (osd.1) 575 : cluster 0 2.4 scrub starts 2024-04-02T05:07:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[21528]: cluster 2024-04-02T05:07:12.771633+0000 osd.1 (osd.1) 576 : cluster 0 2.4 scrub ok 2024-04-02T05:07:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[21528]: cluster 2024-04-02T05:07:13.551506+0000 osd.3 (osd.3) 593 : cluster 0 4.14 scrub starts 2024-04-02T05:07:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[21528]: cluster 2024-04-02T05:07:13.554482+0000 osd.3 (osd.3) 594 : cluster 0 4.14 scrub ok 2024-04-02T05:07:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[21528]: cluster 2024-04-02T05:07:13.724467+0000 osd.2 (osd.2) 430 : cluster 0 4.f scrub starts 2024-04-02T05:07:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[21528]: cluster 2024-04-02T05:07:13.726788+0000 osd.2 (osd.2) 431 : cluster 0 4.f scrub ok 2024-04-02T05:07:14.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[28904]: cluster 2024-04-02T05:07:12.768393+0000 osd.1 (osd.1) 575 : cluster 0 2.4 scrub starts 2024-04-02T05:07:14.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[28904]: cluster 2024-04-02T05:07:12.771633+0000 osd.1 (osd.1) 576 : cluster 0 2.4 scrub ok 2024-04-02T05:07:14.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[28904]: cluster 2024-04-02T05:07:13.551506+0000 osd.3 (osd.3) 593 : cluster 0 4.14 scrub starts 2024-04-02T05:07:14.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[28904]: cluster 2024-04-02T05:07:13.554482+0000 osd.3 (osd.3) 594 : cluster 0 4.14 scrub ok 2024-04-02T05:07:14.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[28904]: cluster 2024-04-02T05:07:13.724467+0000 osd.2 (osd.2) 430 : cluster 0 4.f scrub starts 2024-04-02T05:07:14.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:14 smithi069 bash[28904]: cluster 2024-04-02T05:07:13.726788+0000 osd.2 (osd.2) 431 : cluster 0 4.f scrub ok 2024-04-02T05:07:14.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:14 smithi098 bash[23790]: cluster 2024-04-02T05:07:12.768393+0000 osd.1 (osd.1) 575 : cluster 0 2.4 scrub starts 2024-04-02T05:07:14.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:14 smithi098 bash[23790]: cluster 2024-04-02T05:07:12.771633+0000 osd.1 (osd.1) 576 : cluster 0 2.4 scrub ok 2024-04-02T05:07:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:14 smithi098 bash[23790]: cluster 2024-04-02T05:07:13.551506+0000 osd.3 (osd.3) 593 : cluster 0 4.14 scrub starts 2024-04-02T05:07:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:14 smithi098 bash[23790]: cluster 2024-04-02T05:07:13.554482+0000 osd.3 (osd.3) 594 : cluster 0 4.14 scrub ok 2024-04-02T05:07:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:14 smithi098 bash[23790]: cluster 2024-04-02T05:07:13.724467+0000 osd.2 (osd.2) 430 : cluster 0 4.f scrub starts 2024-04-02T05:07:14.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:14 smithi098 bash[23790]: cluster 2024-04-02T05:07:13.726788+0000 osd.2 (osd.2) 431 : cluster 0 4.f scrub ok 2024-04-02T05:07:15.909 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[28904]: audit 2024-04-02T05:07:13.975499+0000 mgr.y (mgr.14538) 898 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[28904]: cluster 2024-04-02T05:07:14.488680+0000 mgr.y (mgr.14538) 899 : cluster 0 pgmap v1538: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[28904]: cluster 2024-04-02T05:07:14.549644+0000 osd.7 (osd.7) 369 : cluster 0 1.d deep-scrub starts 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[28904]: cluster 2024-04-02T05:07:14.552247+0000 osd.7 (osd.7) 370 : cluster 0 1.d deep-scrub ok 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[21528]: audit 2024-04-02T05:07:13.975499+0000 mgr.y (mgr.14538) 898 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[21528]: cluster 2024-04-02T05:07:14.488680+0000 mgr.y (mgr.14538) 899 : cluster 0 pgmap v1538: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[21528]: cluster 2024-04-02T05:07:14.549644+0000 osd.7 (osd.7) 369 : cluster 0 1.d deep-scrub starts 2024-04-02T05:07:15.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:15 smithi069 bash[21528]: cluster 2024-04-02T05:07:14.552247+0000 osd.7 (osd.7) 370 : cluster 0 1.d deep-scrub ok 2024-04-02T05:07:15.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:15 smithi098 bash[23790]: audit 2024-04-02T05:07:13.975499+0000 mgr.y (mgr.14538) 898 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:15.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:15 smithi098 bash[23790]: cluster 2024-04-02T05:07:14.488680+0000 mgr.y (mgr.14538) 899 : cluster 0 pgmap v1538: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:15.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:15 smithi098 bash[23790]: cluster 2024-04-02T05:07:14.549644+0000 osd.7 (osd.7) 369 : cluster 0 1.d deep-scrub starts 2024-04-02T05:07:15.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:15 smithi098 bash[23790]: cluster 2024-04-02T05:07:14.552247+0000 osd.7 (osd.7) 370 : cluster 0 1.d deep-scrub ok 2024-04-02T05:07:16.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 -- 172.21.15.69:0/217295941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 msgr2=0x7f79dc107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:16.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/217295941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc107600 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f79d400b0a0 tx=0x7f79d400b3b0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 -- 172.21.15.69:0/217295941 shutdown_connections 2024-04-02T05:07:16.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/217295941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc101b60 0x7f79dc10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/217295941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc107600 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/217295941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc1011a0 0x7f79dc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 -- 172.21.15.69:0/217295941 >> 172.21.15.69:0/217295941 conn(0x7f79dc0771e0 msgr2=0x7f79dc0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 -- 172.21.15.69:0/217295941 shutdown_connections 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 -- 172.21.15.69:0/217295941 wait complete. 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 librados: starting msgr at 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 librados: starting objecter 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 Processor -- start 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 -- start start 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 librados: setting wanted keys 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.040+0000 7f79e2ac6700 1 librados: calling monclient init 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 0x7f79dc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:16.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc101b60 0x7f79dc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc110910 con 0x7f79dc1011a0 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc110790 con 0x7f79dc101b60 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc110a90 con 0x7f79dc107220 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 0x7f79dc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 0x7f79dc198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52962/0 (socket says 172.21.15.69:52962) 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 -- 172.21.15.69:0/2098834553 learned_addr learned my addr 172.21.15.69:0/2098834553 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e1063700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79dbfff700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc101b60 0x7f79dc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 -- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 msgr2=0x7f79dc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 -- 172.21.15.69:0/2098834553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc101b60 msgr2=0x7f79dc199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc101b60 0x7f79dc199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 -- 172.21.15.69:0/2098834553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc1a43b0 con 0x7f79dc1011a0 2024-04-02T05:07:16.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e1063700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79dbfff700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc101b60 0x7f79dc199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e0862700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 0x7f79dc198df0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f79d400b070 tx=0x7f79d4007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 10 client.?.objecter ms_handle_connect 0x7f79dc1011a0 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4003c00 con 0x7f79dc1011a0 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f79d4022070 con 0x7f79dc1011a0 2024-04-02T05:07:16.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc1a45e0 con 0x7f79dc1011a0 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d401b6c0 con 0x7f79dc1011a0 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 10 client.33969.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc1a4a30 con 0x7f79dc1011a0 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 librados: init done 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc04ff00 con 0x7f79dc1011a0 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f79d401da10 con 0x7f79dc1011a0 2024-04-02T05:07:16.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79c806f5e0 0x7f79c8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79dbfff700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79c806f5e0 0x7f79c8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(964..964 src has 907..964) ==== 9015+0+0 (secure 0 0 0) 0x7f79d400f020 con 0x7f79dc1011a0 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 10 client.33969.objecter ms_dispatch 0x7f79dc001550 osd_map(964..964 src has 907..964) 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 3 client.33969.objecter handle_osd_map got epochs [964,964] > 0 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 3 client.33969.objecter handle_osd_map decoding full epoch 964 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 20 client.33969.objecter dump_active .. 0 homeless 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79d9ffb700 10 client.33969.objecter ms_handle_connect 0x7f79c806f5e0 2024-04-02T05:07:16.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.044+0000 7f79dbfff700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79c806f5e0 0x7f79c8071aa0 secure :-1 s=READY pgs=4051 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005fd0 tx=0x7f79c4005ee0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:16.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.052+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f79d405c510 con 0x7f79dc1011a0 2024-04-02T05:07:16.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.260+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79dc1a4c60 con 0x7f79dc1011a0 2024-04-02T05:07:16.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.264+0000 7f79d9ffb700 1 -- 172.21.15.69:0/2098834553 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v964) ==== 74+0+26650 (secure 0 0 0) 0x7f79d405ff60 con 0x7f79dc1011a0 2024-04-02T05:07:16.270 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T05:07:16.271 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":964,"fsid":"f978400e-f0a9-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:33:43.348385+0000","modified":"2024-04-02T05:07:08.569603+0000","last_up_change":"2024-04-02T04:53:53.913436+0000","last_in_change":"2024-04-02T05:04:10.153375+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":114,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.671714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"908","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"731","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5199999809265137,"score_stable":1.5199999809265137,"optimal_score":1,"raw_score_acting":1.5199999809265137,"raw_score_stable":1.5199999809265137,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-02T04:43:52.801763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-02T04:43:54.772439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-04-02T04:43:56.696111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-04-02T04:43:57.254108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-02T04:43:58.846023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-02T04:49:11.754451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b13e248-930e-40ca-9bb0-e8c5ce16bd0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":960,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6803","nonce":1087500595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6805","nonce":1087500595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6809","nonce":1087500595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1087500595},{"type":"v1","addr":"172.21.15.69:6807","nonce":1087500595}]},"public_addr":"172.21.15.69:6803/1087500595","cluster_addr":"172.21.15.69:6805/1087500595","heartbeat_back_addr":"172.21.15.69:6809/1087500595","heartbeat_front_addr":"172.21.15.69:6807/1087500595","state":["exists","up"]},{"osd":1,"uuid":"afb71182-d4f7-416d-ae44-90f28eab4b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":955,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6811","nonce":2651666015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6813","nonce":2651666015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6817","nonce":2651666015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2651666015},{"type":"v1","addr":"172.21.15.69:6815","nonce":2651666015}]},"public_addr":"172.21.15.69:6811/2651666015","cluster_addr":"172.21.15.69:6813/2651666015","heartbeat_back_addr":"172.21.15.69:6817/2651666015","heartbeat_front_addr":"172.21.15.69:6815/2651666015","state":["exists","up"]},{"osd":2,"uuid":"523bcbd7-d389-4b8f-aca8-88b71ce61de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":943,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6819","nonce":619998764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6821","nonce":619998764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6825","nonce":619998764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":619998764},{"type":"v1","addr":"172.21.15.69:6823","nonce":619998764}]},"public_addr":"172.21.15.69:6819/619998764","cluster_addr":"172.21.15.69:6821/619998764","heartbeat_back_addr":"172.21.15.69:6825/619998764","heartbeat_front_addr":"172.21.15.69:6823/619998764","state":["exists","up"]},{"osd":3,"uuid":"20e4526d-2707-44d0-b13a-af36019c42ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":952,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6827","nonce":207593208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6829","nonce":207593208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6833","nonce":207593208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":207593208},{"type":"v1","addr":"172.21.15.69:6831","nonce":207593208}]},"public_addr":"172.21.15.69:6827/207593208","cluster_addr":"172.21.15.69:6829/207593208","heartbeat_back_addr":"172.21.15.69:6833/207593208","heartbeat_front_addr":"172.21.15.69:6831/207593208","state":["exists","up"]},{"osd":4,"uuid":"aa3bc4f3-4476-4ba1-b033-428e5edf5852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":943,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6801","nonce":2065947829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6803","nonce":2065947829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6807","nonce":2065947829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2065947829},{"type":"v1","addr":"172.21.15.98:6805","nonce":2065947829}]},"public_addr":"172.21.15.98:6801/2065947829","cluster_addr":"172.21.15.98:6803/2065947829","heartbeat_back_addr":"172.21.15.98:6807/2065947829","heartbeat_front_addr":"172.21.15.98:6805/2065947829","state":["exists","up"]},{"osd":5,"uuid":"902326af-4a40-47c8-a3b8-d5d5536f90fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":960,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6809","nonce":219578185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6811","nonce":219578185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6815","nonce":219578185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":219578185},{"type":"v1","addr":"172.21.15.98:6813","nonce":219578185}]},"public_addr":"172.21.15.98:6809/219578185","cluster_addr":"172.21.15.98:6811/219578185","heartbeat_back_addr":"172.21.15.98:6815/219578185","heartbeat_front_addr":"172.21.15.98:6813/219578185","state":["exists","up"]},{"osd":6,"uuid":"8c73e4b0-cb6b-486a-8b55-a0fe4d97a727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":952,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6817","nonce":1226172499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6819","nonce":1226172499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6823","nonce":1226172499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1226172499},{"type":"v1","addr":"172.21.15.98:6821","nonce":1226172499}]},"public_addr":"172.21.15.98:6817/1226172499","cluster_addr":"172.21.15.98:6819/1226172499","heartbeat_back_addr":"172.21.15.98:6823/1226172499","heartbeat_front_addr":"172.21.15.98:6821/1226172499","state":["exists","up"]},{"osd":7,"uuid":"4cff4de7-9efd-4f87-b87e-94f8ec6d0dc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":69,"last_clean_end":152,"up_from":394,"up_thru":952,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6825","nonce":575416324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6827","nonce":575416324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6831","nonce":575416324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":575416324},{"type":"v1","addr":"172.21.15.98:6829","nonce":575416324}]},"public_addr":"172.21.15.98:6825/575416324","cluster_addr":"172.21.15.98:6827/575416324","heartbeat_back_addr":"172.21.15.98:6831/575416324","heartbeat_front_addr":"172.21.15.98:6829/575416324","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:05.653743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:48.446977+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:07.578549+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:06.461474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:02.669368+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:45.237333+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:06:15.381429+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T05:07:02.878427+0000","dead_epoch":150}],"pg_upmap":[{"pgid":"10.8","osds":[5,1,7]}],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":5,"to":1}]},{"pgid":"1.a","mappings":[{"from":5,"to":4}]},{"pgid":"2.9","mappings":[{"from":7,"to":2}]},{"pgid":"3.f","mappings":[{"from":4,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":3,"to":5}]},{"pgid":"10.e","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/2738867930":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/1798147282":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/52352690":"2024-04-03T04:34:13.853990+0000","172.21.15.69:0/2928651991":"2024-04-03T04:34:47.037764+0000","172.21.15.69:6800/2442918057":"2024-04-03T04:34:13.853990+0000","172.21.15.69:6801/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/2763215770":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/3657409442":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/879855771":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1025808168":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/1168952113":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6800/578732137":"2024-04-03T04:34:47.037764+0000","172.21.15.69:0/1892956550":"2024-04-03T04:44:51.699278+0000","172.21.15.69:6801/1560062353":"2024-04-03T04:44:51.699278+0000","172.21.15.69:0/54578327":"2024-04-03T04:44:51.699278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":3,"length":4}]},{"pool":36,"snaps":[{"begin":3,"length":2}]},{"pool":38,"snaps":[{"begin":1,"length":1}]},{"pool":66,"snaps":[{"begin":1,"length":1}]},{"pool":101,"snaps":[{"begin":2,"length":4},{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":112,"snaps":[{"begin":2,"length":9},{"begin":14,"length":1}]},{"pool":120,"snaps":[{"begin":1,"length":2}]},{"pool":137,"snaps":[{"begin":3,"length":1}]},{"pool":145,"snaps":[{"begin":1,"length":4}]},{"pool":146,"snaps":[{"begin":2,"length":1}]},{"pool":154,"snaps":[{"begin":2,"length":2}]},{"pool":164,"snaps":[{"begin":2,"length":3}]},{"pool":165,"snaps":[{"begin":3,"length":4}]},{"pool":173,"snaps":[{"begin":2,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":2}]},{"pool":188,"snaps":[{"begin":3,"length":1}]},{"pool":189,"snaps":[{"begin":2,"length":1}]},{"pool":196,"snaps":[{"begin":1,"length":4}]},{"pool":209,"snaps":[{"begin":2,"length":6}]},{"pool":214,"snaps":[{"begin":7,"length":1}]},{"pool":234,"snaps":[{"begin":8,"length":1}]},{"pool":245,"snaps":[{"begin":10,"length":1}]},{"pool":251,"snaps":[{"begin":11,"length":2}]},{"pool":262,"snaps":[{"begin":14,"length":1}]},{"pool":326,"snaps":[{"begin":2,"length":1}]},{"pool":327,"snaps":[{"begin":3,"length":1}]},{"pool":330,"snaps":[{"begin":4,"length":1}]},{"pool":334,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.264+0000 7f79e2ac6700 10 librados: watch_flush enter 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.264+0000 7f79e2ac6700 10 librados: watch_flush exit 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.264+0000 7f79e2ac6700 20 client.33969.objecter shutdown clearing up homeless session... 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.264+0000 7f79e2ac6700 10 client.33969.objecter successfully canceled tick 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79c806f5e0 msgr2=0x7f79c8071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79c806f5e0 0x7f79c8071aa0 secure :-1 s=READY pgs=4051 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005fd0 tx=0x7f79c4005ee0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 msgr2=0x7f79dc198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 0x7f79dc198df0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f79d400b070 tx=0x7f79d4007ab0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 shutdown_connections 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f79c806f5e0 0x7f79c8071aa0 unknown :-1 s=CLOSED pgs=4051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc107220 0x7f79dc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc1011a0 0x7f79dc198df0 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 --2- 172.21.15.69:0/2098834553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79dc101b60 0x7f79dc199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:16.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 >> 172.21.15.69:0/2098834553 conn(0x7f79dc0771e0 msgr2=0x7f79dc107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:16.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 shutdown_connections 2024-04-02T05:07:16.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 -- 172.21.15.69:0/2098834553 wait complete. 2024-04-02T05:07:16.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:16.268+0000 7f79e2ac6700 1 librados: shutdown 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: cluster 2024-04-02T05:07:14.778750+0000 osd.1 (osd.1) 577 : cluster 0 2.9 scrub starts 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: cluster 2024-04-02T05:07:14.781355+0000 osd.1 (osd.1) 578 : cluster 0 2.9 scrub ok 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: cluster 2024-04-02T05:07:15.727550+0000 osd.2 (osd.2) 432 : cluster 0 2.10 deep-scrub starts 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: cluster 2024-04-02T05:07:15.729874+0000 osd.2 (osd.2) 433 : cluster 0 2.10 deep-scrub ok 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.205747+0000 osd.5 (osd.5) 389 : cluster 0 2.16 scrub starts 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.208401+0000 osd.5 (osd.5) 390 : cluster 0 2.16 scrub ok 2024-04-02T05:07:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:16 smithi098 bash[23790]: audit 2024-04-02T05:07:16.266971+0000 mon.a (mon.0) 4197 : audit 0 from='client.? 172.21.15.69:0/2098834553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: cluster 2024-04-02T05:07:14.778750+0000 osd.1 (osd.1) 577 : cluster 0 2.9 scrub starts 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: cluster 2024-04-02T05:07:14.781355+0000 osd.1 (osd.1) 578 : cluster 0 2.9 scrub ok 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: cluster 2024-04-02T05:07:15.727550+0000 osd.2 (osd.2) 432 : cluster 0 2.10 deep-scrub starts 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: cluster 2024-04-02T05:07:15.729874+0000 osd.2 (osd.2) 433 : cluster 0 2.10 deep-scrub ok 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.205747+0000 osd.5 (osd.5) 389 : cluster 0 2.16 scrub starts 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.208401+0000 osd.5 (osd.5) 390 : cluster 0 2.16 scrub ok 2024-04-02T05:07:17.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[21528]: audit 2024-04-02T05:07:16.266971+0000 mon.a (mon.0) 4197 : audit 0 from='client.? 172.21.15.69:0/2098834553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: cluster 2024-04-02T05:07:14.778750+0000 osd.1 (osd.1) 577 : cluster 0 2.9 scrub starts 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: cluster 2024-04-02T05:07:14.781355+0000 osd.1 (osd.1) 578 : cluster 0 2.9 scrub ok 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: cluster 2024-04-02T05:07:15.727550+0000 osd.2 (osd.2) 432 : cluster 0 2.10 deep-scrub starts 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: cluster 2024-04-02T05:07:15.729874+0000 osd.2 (osd.2) 433 : cluster 0 2.10 deep-scrub ok 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.205747+0000 osd.5 (osd.5) 389 : cluster 0 2.16 scrub starts 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.208401+0000 osd.5 (osd.5) 390 : cluster 0 2.16 scrub ok 2024-04-02T05:07:17.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:16 smithi069 bash[28904]: audit 2024-04-02T05:07:16.266971+0000 mon.a (mon.0) 4197 : audit 0 from='client.? 172.21.15.69:0/2098834553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T05:07:17.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-02T05:07:17.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-02T05:07:17.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-02T05:07:17.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-02T05:07:17.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-02T05:07:17.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-02T05:07:17.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-02T05:07:17.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-02T05:07:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:17 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.489429+0000 mgr.y (mgr.14538) 900 : cluster 0 pgmap v1539: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:07:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:17 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.754908+0000 osd.6 (osd.6) 507 : cluster 0 10.16 scrub starts 2024-04-02T05:07:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:17 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.755368+0000 osd.2 (osd.2) 434 : cluster 0 10.5 scrub starts 2024-04-02T05:07:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:17 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.758448+0000 osd.6 (osd.6) 508 : cluster 0 10.16 scrub ok 2024-04-02T05:07:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:17 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.761289+0000 osd.2 (osd.2) 435 : cluster 0 10.5 scrub ok 2024-04-02T05:07:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.489429+0000 mgr.y (mgr.14538) 900 : cluster 0 pgmap v1539: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:07:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.754908+0000 osd.6 (osd.6) 507 : cluster 0 10.16 scrub starts 2024-04-02T05:07:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.755368+0000 osd.2 (osd.2) 434 : cluster 0 10.5 scrub starts 2024-04-02T05:07:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.758448+0000 osd.6 (osd.6) 508 : cluster 0 10.16 scrub ok 2024-04-02T05:07:18.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.761289+0000 osd.2 (osd.2) 435 : cluster 0 10.5 scrub ok 2024-04-02T05:07:18.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.489429+0000 mgr.y (mgr.14538) 900 : cluster 0 pgmap v1539: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:07:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.754908+0000 osd.6 (osd.6) 507 : cluster 0 10.16 scrub starts 2024-04-02T05:07:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.755368+0000 osd.2 (osd.2) 434 : cluster 0 10.5 scrub starts 2024-04-02T05:07:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.758448+0000 osd.6 (osd.6) 508 : cluster 0 10.16 scrub ok 2024-04-02T05:07:18.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:17 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.761289+0000 osd.2 (osd.2) 435 : cluster 0 10.5 scrub ok 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.853898+0000 osd.1 (osd.1) 579 : cluster 0 6.1d scrub starts 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:16.856391+0000 osd.1 (osd.1) 580 : cluster 0 6.1d scrub ok 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:17.659196+0000 osd.7 (osd.7) 371 : cluster 0 1.2 deep-scrub starts 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:17.661714+0000 osd.7 (osd.7) 372 : cluster 0 1.2 deep-scrub ok 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:17.675740+0000 osd.3 (osd.3) 595 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:17.678345+0000 osd.3 (osd.3) 596 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:17.719160+0000 osd.6 (osd.6) 509 : cluster 0 4.5 scrub starts 2024-04-02T05:07:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:18 smithi098 bash[23790]: cluster 2024-04-02T05:07:17.721647+0000 osd.6 (osd.6) 510 : cluster 0 4.5 scrub ok 2024-04-02T05:07:19.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:07:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:07:19.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.853898+0000 osd.1 (osd.1) 579 : cluster 0 6.1d scrub starts 2024-04-02T05:07:19.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:16.856391+0000 osd.1 (osd.1) 580 : cluster 0 6.1d scrub ok 2024-04-02T05:07:19.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:17.659196+0000 osd.7 (osd.7) 371 : cluster 0 1.2 deep-scrub starts 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:17.661714+0000 osd.7 (osd.7) 372 : cluster 0 1.2 deep-scrub ok 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:17.675740+0000 osd.3 (osd.3) 595 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:17.678345+0000 osd.3 (osd.3) 596 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:17.719160+0000 osd.6 (osd.6) 509 : cluster 0 4.5 scrub starts 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[21528]: cluster 2024-04-02T05:07:17.721647+0000 osd.6 (osd.6) 510 : cluster 0 4.5 scrub ok 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.853898+0000 osd.1 (osd.1) 579 : cluster 0 6.1d scrub starts 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:16.856391+0000 osd.1 (osd.1) 580 : cluster 0 6.1d scrub ok 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:17.659196+0000 osd.7 (osd.7) 371 : cluster 0 1.2 deep-scrub starts 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:17.661714+0000 osd.7 (osd.7) 372 : cluster 0 1.2 deep-scrub ok 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:17.675740+0000 osd.3 (osd.3) 595 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:17.678345+0000 osd.3 (osd.3) 596 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:07:19.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:17.719160+0000 osd.6 (osd.6) 509 : cluster 0 4.5 scrub starts 2024-04-02T05:07:19.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:18 smithi069 bash[28904]: cluster 2024-04-02T05:07:17.721647+0000 osd.6 (osd.6) 510 : cluster 0 4.5 scrub ok 2024-04-02T05:07:19.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:19 smithi098 bash[23790]: cluster 2024-04-02T05:07:18.491238+0000 mgr.y (mgr.14538) 901 : cluster 0 pgmap v1540: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:19 smithi098 bash[23790]: cluster 2024-04-02T05:07:18.629648+0000 osd.7 (osd.7) 373 : cluster 0 2.1d scrub starts 2024-04-02T05:07:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:19 smithi098 bash[23790]: cluster 2024-04-02T05:07:18.632100+0000 osd.7 (osd.7) 374 : cluster 0 2.1d scrub ok 2024-04-02T05:07:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:19 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.264230+0000 osd.5 (osd.5) 391 : cluster 0 3.c scrub starts 2024-04-02T05:07:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:19 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.268322+0000 osd.5 (osd.5) 392 : cluster 0 3.c scrub ok 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[21528]: cluster 2024-04-02T05:07:18.491238+0000 mgr.y (mgr.14538) 901 : cluster 0 pgmap v1540: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[21528]: cluster 2024-04-02T05:07:18.629648+0000 osd.7 (osd.7) 373 : cluster 0 2.1d scrub starts 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[21528]: cluster 2024-04-02T05:07:18.632100+0000 osd.7 (osd.7) 374 : cluster 0 2.1d scrub ok 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.264230+0000 osd.5 (osd.5) 391 : cluster 0 3.c scrub starts 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.268322+0000 osd.5 (osd.5) 392 : cluster 0 3.c scrub ok 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[28904]: cluster 2024-04-02T05:07:18.491238+0000 mgr.y (mgr.14538) 901 : cluster 0 pgmap v1540: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[28904]: cluster 2024-04-02T05:07:18.629648+0000 osd.7 (osd.7) 373 : cluster 0 2.1d scrub starts 2024-04-02T05:07:20.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[28904]: cluster 2024-04-02T05:07:18.632100+0000 osd.7 (osd.7) 374 : cluster 0 2.1d scrub ok 2024-04-02T05:07:20.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.264230+0000 osd.5 (osd.5) 391 : cluster 0 3.c scrub starts 2024-04-02T05:07:20.036 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:19 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.268322+0000 osd.5 (osd.5) 392 : cluster 0 3.c scrub ok 2024-04-02T05:07:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:20 smithi098 bash[23790]: audit 2024-04-02T05:07:19.726660+0000 mon.b (mon.1) 357 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:20 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.814361+0000 osd.2 (osd.2) 436 : cluster 0 2.c scrub starts 2024-04-02T05:07:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:20 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.816662+0000 osd.2 (osd.2) 437 : cluster 0 2.c scrub ok 2024-04-02T05:07:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:20 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.885577+0000 osd.0 (osd.0) 537 : cluster 0 4.b scrub starts 2024-04-02T05:07:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:20 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.887947+0000 osd.0 (osd.0) 538 : cluster 0 4.b scrub ok 2024-04-02T05:07:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[21528]: audit 2024-04-02T05:07:19.726660+0000 mon.b (mon.1) 357 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.814361+0000 osd.2 (osd.2) 436 : cluster 0 2.c scrub starts 2024-04-02T05:07:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.816662+0000 osd.2 (osd.2) 437 : cluster 0 2.c scrub ok 2024-04-02T05:07:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.885577+0000 osd.0 (osd.0) 537 : cluster 0 4.b scrub starts 2024-04-02T05:07:21.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.887947+0000 osd.0 (osd.0) 538 : cluster 0 4.b scrub ok 2024-04-02T05:07:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[28904]: audit 2024-04-02T05:07:19.726660+0000 mon.b (mon.1) 357 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.814361+0000 osd.2 (osd.2) 436 : cluster 0 2.c scrub starts 2024-04-02T05:07:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.816662+0000 osd.2 (osd.2) 437 : cluster 0 2.c scrub ok 2024-04-02T05:07:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.885577+0000 osd.0 (osd.0) 537 : cluster 0 4.b scrub starts 2024-04-02T05:07:21.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:20 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.887947+0000 osd.0 (osd.0) 538 : cluster 0 4.b scrub ok 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.771024+0000 osd.1 (osd.1) 581 : cluster 0 3.19 scrub starts 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:19.776354+0000 osd.1 (osd.1) 582 : cluster 0 3.19 scrub ok 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:20.492237+0000 mgr.y (mgr.14538) 902 : cluster 0 pgmap v1541: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:20.505513+0000 osd.4 (osd.4) 536 : cluster 0 6.1e deep-scrub starts 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:20.507982+0000 osd.4 (osd.4) 537 : cluster 0 6.1e deep-scrub ok 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.196939+0000 osd.5 (osd.5) 393 : cluster 0 3.16 scrub starts 2024-04-02T05:07:21.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:21 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.201413+0000 osd.5 (osd.5) 394 : cluster 0 3.16 scrub ok 2024-04-02T05:07:21.986 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.986 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.988 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:21.988 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:22.013 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.771024+0000 osd.1 (osd.1) 581 : cluster 0 3.19 scrub starts 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:19.776354+0000 osd.1 (osd.1) 582 : cluster 0 3.19 scrub ok 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:20.492237+0000 mgr.y (mgr.14538) 902 : cluster 0 pgmap v1541: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:20.505513+0000 osd.4 (osd.4) 536 : cluster 0 6.1e deep-scrub starts 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:20.507982+0000 osd.4 (osd.4) 537 : cluster 0 6.1e deep-scrub ok 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.196939+0000 osd.5 (osd.5) 393 : cluster 0 3.16 scrub starts 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.201413+0000 osd.5 (osd.5) 394 : cluster 0 3.16 scrub ok 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.771024+0000 osd.1 (osd.1) 581 : cluster 0 3.19 scrub starts 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:19.776354+0000 osd.1 (osd.1) 582 : cluster 0 3.19 scrub ok 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:20.492237+0000 mgr.y (mgr.14538) 902 : cluster 0 pgmap v1541: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:20.505513+0000 osd.4 (osd.4) 536 : cluster 0 6.1e deep-scrub starts 2024-04-02T05:07:22.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:20.507982+0000 osd.4 (osd.4) 537 : cluster 0 6.1e deep-scrub ok 2024-04-02T05:07:22.015 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.196939+0000 osd.5 (osd.5) 393 : cluster 0 3.16 scrub starts 2024-04-02T05:07:22.015 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:21 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.201413+0000 osd.5 (osd.5) 394 : cluster 0 3.16 scrub ok 2024-04-02T05:07:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:22 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.628231+0000 osd.6 (osd.6) 511 : cluster 0 10.a deep-scrub starts 2024-04-02T05:07:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:22 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.631096+0000 osd.6 (osd.6) 512 : cluster 0 10.a deep-scrub ok 2024-04-02T05:07:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:22 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.671064+0000 osd.6 (osd.6) 513 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:22 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.671722+0000 osd.6 (osd.6) 514 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:22 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.826080+0000 osd.2 (osd.2) 438 : cluster 0 10.18 scrub starts 2024-04-02T05:07:22.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:22 smithi098 bash[23790]: cluster 2024-04-02T05:07:21.828513+0000 osd.2 (osd.2) 439 : cluster 0 10.18 scrub ok 2024-04-02T05:07:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.628231+0000 osd.6 (osd.6) 511 : cluster 0 10.a deep-scrub starts 2024-04-02T05:07:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.631096+0000 osd.6 (osd.6) 512 : cluster 0 10.a deep-scrub ok 2024-04-02T05:07:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.671064+0000 osd.6 (osd.6) 513 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.671722+0000 osd.6 (osd.6) 514 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.826080+0000 osd.2 (osd.2) 438 : cluster 0 10.18 scrub starts 2024-04-02T05:07:23.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[21528]: cluster 2024-04-02T05:07:21.828513+0000 osd.2 (osd.2) 439 : cluster 0 10.18 scrub ok 2024-04-02T05:07:23.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.628231+0000 osd.6 (osd.6) 511 : cluster 0 10.a deep-scrub starts 2024-04-02T05:07:23.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.631096+0000 osd.6 (osd.6) 512 : cluster 0 10.a deep-scrub ok 2024-04-02T05:07:23.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.671064+0000 osd.6 (osd.6) 513 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:23.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.671722+0000 osd.6 (osd.6) 514 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:23.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.826080+0000 osd.2 (osd.2) 438 : cluster 0 10.18 scrub starts 2024-04-02T05:07:23.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:22 smithi069 bash[28904]: cluster 2024-04-02T05:07:21.828513+0000 osd.2 (osd.2) 439 : cluster 0 10.18 scrub ok 2024-04-02T05:07:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.478040+0000 osd.4 (osd.4) 538 : cluster 0 6.17 scrub starts 2024-04-02T05:07:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.480421+0000 osd.4 (osd.4) 539 : cluster 0 6.17 scrub ok 2024-04-02T05:07:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.493035+0000 mgr.y (mgr.14538) 903 : cluster 0 pgmap v1542: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.558823+0000 osd.3 (osd.3) 597 : cluster 0 6.6 scrub starts 2024-04-02T05:07:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.559167+0000 osd.7 (osd.7) 375 : cluster 0 10.c scrub starts 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.561794+0000 osd.7 (osd.7) 376 : cluster 0 10.c scrub ok 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.561863+0000 osd.3 (osd.3) 598 : cluster 0 6.6 scrub ok 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:22.670051+0000 mon.a (mon.0) 4198 : cluster 0 osdmap e965: 8 total, 8 up, 8 in 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:23.221843+0000 osd.5 (osd.5) 395 : cluster 0 10.11 scrub starts 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[21528]: cluster 2024-04-02T05:07:23.225821+0000 osd.5 (osd.5) 396 : cluster 0 10.11 scrub ok 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.478040+0000 osd.4 (osd.4) 538 : cluster 0 6.17 scrub starts 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.480421+0000 osd.4 (osd.4) 539 : cluster 0 6.17 scrub ok 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.493035+0000 mgr.y (mgr.14538) 903 : cluster 0 pgmap v1542: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.558823+0000 osd.3 (osd.3) 597 : cluster 0 6.6 scrub starts 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.559167+0000 osd.7 (osd.7) 375 : cluster 0 10.c scrub starts 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.561794+0000 osd.7 (osd.7) 376 : cluster 0 10.c scrub ok 2024-04-02T05:07:23.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.561863+0000 osd.3 (osd.3) 598 : cluster 0 6.6 scrub ok 2024-04-02T05:07:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:22.670051+0000 mon.a (mon.0) 4198 : cluster 0 osdmap e965: 8 total, 8 up, 8 in 2024-04-02T05:07:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:23.221843+0000 osd.5 (osd.5) 395 : cluster 0 10.11 scrub starts 2024-04-02T05:07:23.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:23 smithi069 bash[28904]: cluster 2024-04-02T05:07:23.225821+0000 osd.5 (osd.5) 396 : cluster 0 10.11 scrub ok 2024-04-02T05:07:23.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.478040+0000 osd.4 (osd.4) 538 : cluster 0 6.17 scrub starts 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.480421+0000 osd.4 (osd.4) 539 : cluster 0 6.17 scrub ok 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.493035+0000 mgr.y (mgr.14538) 903 : cluster 0 pgmap v1542: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.558823+0000 osd.3 (osd.3) 597 : cluster 0 6.6 scrub starts 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.559167+0000 osd.7 (osd.7) 375 : cluster 0 10.c scrub starts 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.561794+0000 osd.7 (osd.7) 376 : cluster 0 10.c scrub ok 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.561863+0000 osd.3 (osd.3) 598 : cluster 0 6.6 scrub ok 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:22.670051+0000 mon.a (mon.0) 4198 : cluster 0 osdmap e965: 8 total, 8 up, 8 in 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:23.221843+0000 osd.5 (osd.5) 395 : cluster 0 10.11 scrub starts 2024-04-02T05:07:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[23790]: cluster 2024-04-02T05:07:23.225821+0000 osd.5 (osd.5) 396 : cluster 0 10.11 scrub ok 2024-04-02T05:07:24.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:07:23 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:07:24.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:24 smithi098 bash[23790]: cluster 2024-04-02T05:07:23.530683+0000 osd.3 (osd.3) 599 : cluster 0 4.1d deep-scrub starts 2024-04-02T05:07:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:24 smithi098 bash[23790]: cluster 2024-04-02T05:07:23.532243+0000 osd.3 (osd.3) 600 : cluster 0 4.1d deep-scrub ok 2024-04-02T05:07:25.085 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:24 smithi069 bash[21528]: cluster 2024-04-02T05:07:23.530683+0000 osd.3 (osd.3) 599 : cluster 0 4.1d deep-scrub starts 2024-04-02T05:07:25.085 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:24 smithi069 bash[21528]: cluster 2024-04-02T05:07:23.532243+0000 osd.3 (osd.3) 600 : cluster 0 4.1d deep-scrub ok 2024-04-02T05:07:25.085 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:24 smithi069 bash[28904]: cluster 2024-04-02T05:07:23.530683+0000 osd.3 (osd.3) 599 : cluster 0 4.1d deep-scrub starts 2024-04-02T05:07:25.086 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:24 smithi069 bash[28904]: cluster 2024-04-02T05:07:23.532243+0000 osd.3 (osd.3) 600 : cluster 0 4.1d deep-scrub ok 2024-04-02T05:07:25.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: audit 2024-04-02T05:07:23.984949+0000 mgr.y (mgr.14538) 904 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:25.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.459401+0000 osd.4 (osd.4) 540 : cluster 0 4.13 scrub starts 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.461792+0000 osd.4 (osd.4) 541 : cluster 0 4.13 scrub ok 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.495003+0000 mgr.y (mgr.14538) 905 : cluster 0 pgmap v1544: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.502699+0000 osd.3 (osd.3) 601 : cluster 0 6.1b scrub starts 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.505109+0000 osd.3 (osd.3) 602 : cluster 0 6.1b scrub ok 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.535012+0000 osd.7 (osd.7) 377 : cluster 0 2.5 scrub starts 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[21528]: cluster 2024-04-02T05:07:24.537031+0000 osd.7 (osd.7) 378 : cluster 0 2.5 scrub ok 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: audit 2024-04-02T05:07:23.984949+0000 mgr.y (mgr.14538) 904 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.459401+0000 osd.4 (osd.4) 540 : cluster 0 4.13 scrub starts 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.461792+0000 osd.4 (osd.4) 541 : cluster 0 4.13 scrub ok 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.495003+0000 mgr.y (mgr.14538) 905 : cluster 0 pgmap v1544: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:25.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.502699+0000 osd.3 (osd.3) 601 : cluster 0 6.1b scrub starts 2024-04-02T05:07:25.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.505109+0000 osd.3 (osd.3) 602 : cluster 0 6.1b scrub ok 2024-04-02T05:07:25.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.535012+0000 osd.7 (osd.7) 377 : cluster 0 2.5 scrub starts 2024-04-02T05:07:25.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:25 smithi069 bash[28904]: cluster 2024-04-02T05:07:24.537031+0000 osd.7 (osd.7) 378 : cluster 0 2.5 scrub ok 2024-04-02T05:07:25.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: audit 2024-04-02T05:07:23.984949+0000 mgr.y (mgr.14538) 904 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:25.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.459401+0000 osd.4 (osd.4) 540 : cluster 0 4.13 scrub starts 2024-04-02T05:07:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.461792+0000 osd.4 (osd.4) 541 : cluster 0 4.13 scrub ok 2024-04-02T05:07:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.495003+0000 mgr.y (mgr.14538) 905 : cluster 0 pgmap v1544: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.502699+0000 osd.3 (osd.3) 601 : cluster 0 6.1b scrub starts 2024-04-02T05:07:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.505109+0000 osd.3 (osd.3) 602 : cluster 0 6.1b scrub ok 2024-04-02T05:07:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.535012+0000 osd.7 (osd.7) 377 : cluster 0 2.5 scrub starts 2024-04-02T05:07:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:25 smithi098 bash[23790]: cluster 2024-04-02T05:07:24.537031+0000 osd.7 (osd.7) 378 : cluster 0 2.5 scrub ok 2024-04-02T05:07:26.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:26 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.210387+0000 osd.5 (osd.5) 397 : cluster 0 4.8 scrub starts 2024-04-02T05:07:26.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:26 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.213045+0000 osd.5 (osd.5) 398 : cluster 0 4.8 scrub ok 2024-04-02T05:07:26.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:26 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.210387+0000 osd.5 (osd.5) 397 : cluster 0 4.8 scrub starts 2024-04-02T05:07:26.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:26 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.213045+0000 osd.5 (osd.5) 398 : cluster 0 4.8 scrub ok 2024-04-02T05:07:26.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:26 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.210387+0000 osd.5 (osd.5) 397 : cluster 0 4.8 scrub starts 2024-04-02T05:07:26.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:26 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.213045+0000 osd.5 (osd.5) 398 : cluster 0 4.8 scrub ok 2024-04-02T05:07:27.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:27 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.495498+0000 mgr.y (mgr.14538) 906 : cluster 0 pgmap v1545: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:27 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.587406+0000 osd.7 (osd.7) 379 : cluster 0 1.0 deep-scrub starts 2024-04-02T05:07:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:27 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.590234+0000 osd.7 (osd.7) 380 : cluster 0 1.0 deep-scrub ok 2024-04-02T05:07:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:27 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.751137+0000 osd.6 (osd.6) 515 : cluster 0 3.10 scrub starts 2024-04-02T05:07:27.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:27 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.754146+0000 osd.6 (osd.6) 516 : cluster 0 3.10 scrub ok 2024-04-02T05:07:28.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/3742267125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487410a540 msgr2=0x7f487410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- 172.21.15.69:0/3742267125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487410a540 0x7f487410a920 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7f4864009990 tx=0x7f4864009ca0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/3742267125 shutdown_connections 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- 172.21.15.69:0/3742267125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4874104e50 0x7f4874111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- 172.21.15.69:0/3742267125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487410a540 0x7f487410a920 unknown :-1 s=CLOSED pgs=3336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- 172.21.15.69:0/3742267125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4874104490 0x7f4874104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/3742267125 >> 172.21.15.69:0/3742267125 conn(0x7f48740fff20 msgr2=0x7f4874102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/3742267125 shutdown_connections 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/3742267125 wait complete. 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 librados: starting msgr at 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 librados: starting objecter 2024-04-02T05:07:28.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 Processor -- start 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- start start 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 librados: setting wanted keys 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 librados: calling monclient init 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4874104490 0x7f487419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4874104e50 0x7f487419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487410a540 0x7f487419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4874113cb0 con 0x7f487410a540 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4874113b30 con 0x7f4874104e50 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f4874113e30 con 0x7f4874104490 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4874104490 0x7f487419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4873fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487410a540 0x7f487419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4874104490 0x7f487419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52716/0 (socket says 172.21.15.69:52716) 2024-04-02T05:07:28.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 -- 172.21.15.69:0/985028674 learned_addr learned my addr 172.21.15.69:0/985028674 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 -- 172.21.15.69:0/985028674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4874104e50 msgr2=0x7f487419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4872ffd700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4874104e50 0x7f487419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4874104e50 0x7f487419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 -- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487410a540 msgr2=0x7f487419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487410a540 0x7f487419eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 -- 172.21.15.69:0/985028674 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487419f220 con 0x7f4874104490 2024-04-02T05:07:28.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4873fff700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487410a540 0x7f487419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f48737fe700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4874104490 0x7f487419a210 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f4864000c00 tx=0x7f486400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4870ff9700 10 client.?.objecter ms_handle_connect 0x7f4874104490 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4870ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864027070 con 0x7f4874104490 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4864022470 con 0x7f4874104490 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487419f4b0 con 0x7f4874104490 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864017650 con 0x7f4874104490 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 10 client.44039.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48741a6d90 con 0x7f4874104490 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 librados: init done 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 10 client.44039.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 15 client.44039.objecter _session_command_op_assign -1 1 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 20 client.44039.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:28.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 10 client.44039.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.008+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4874105bb0 con 0x7f4874104490 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f486402a070 con 0x7f4874104490 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f485c06f5e0 0x7f485c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f486408fd40 con 0x7f4874104490 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter ms_dispatch 0x7f4874001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 3 client.44039.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 3 client.44039.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4872ffd700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f485c06f5e0 0x7f485c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter checking command 1 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f485c07bb20 0x7f485c07dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter _get_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 15 client.44039.objecter _session_command_op_remove -1 1 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter _get_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter get_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 15 client.44039.objecter _session_command_op_assign 2 1 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 4 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter _send_command 1 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f485c07e690 con 0x7f485c07bb20 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter dump_active .. 0 homeless 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f4864090190 con 0x7f4874104490 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4873fff700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f485c07bb20 0x7f485c07dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter ms_handle_connect 0x7f485c06f5e0 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4872ffd700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f485c06f5e0 0x7f485c071aa0 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 crypto rx=0x7f4868006fd0 tx=0x7f4868008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter ms_handle_connect 0x7f485c07bb20 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4873fff700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f485c07bb20 0x7f485c07dfc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== osd.2 v2:172.21.15.69:6818/619998764 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f485c07e690 con 0x7f485c07bb20 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter ms_dispatch 0x7f4874001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 10 client.44039.objecter _finish_command 1 = system:0 2024-04-02T05:07:28.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 4 2024-04-02T05:07:28.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.012+0000 7f4870ff9700 15 client.44039.objecter _session_command_op_remove 2 1 2024-04-02T05:07:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 10 client.44039.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 15 client.44039.objecter _session_command_op_assign -1 2 2024-04-02T05:07:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 20 client.44039.objecter _get_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 20 client.44039.objecter _get_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 15 client.44039.objecter _session_command_op_remove -1 2 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 20 client.44039.objecter get_session s=0x7f485c07b450 osd=2 3 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 15 client.44039.objecter _session_command_op_assign 2 2 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 4 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 10 client.44039.objecter _send_command 2 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.028+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 --> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f487404ff00 con 0x7f485c07bb20 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4870ff9700 1 -- 172.21.15.69:0/985028674 <== osd.2 v2:172.21.15.69:6818/619998764 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f487404ff00 con 0x7f485c07bb20 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4870ff9700 10 client.44039.objecter ms_dispatch 0x7f4874001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4870ff9700 10 client.44039.objecter _finish_command 2 = system:0 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4870ff9700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 4 2024-04-02T05:07:28.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4870ff9700 15 client.44039.objecter _session_command_op_remove 2 2 2024-04-02T05:07:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 10 librados: watch_flush enter 2024-04-02T05:07:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 10 librados: watch_flush exit 2024-04-02T05:07:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 10 client.44039.objecter close_session for osd.2 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f485c07bb20 msgr2=0x7f485c07dfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6818/619998764,v1:172.21.15.69:6819/619998764] conn(0x7f485c07bb20 0x7f485c07dfc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 20 client.44039.objecter put_session s=0x7f485c07b450 osd=2 1 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 20 client.44039.objecter shutdown clearing up homeless session... 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 10 client.44039.objecter successfully canceled tick 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f485c06f5e0 msgr2=0x7f485c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f485c06f5e0 0x7f485c071aa0 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 crypto rx=0x7f4868006fd0 tx=0x7f4868008040 comp rx=0 tx=0).stop 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4874104490 msgr2=0x7f487419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 --2- 172.21.15.69:0/985028674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4874104490 0x7f487419a210 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f4864000c00 tx=0x7f486400bfd0 comp rx=0 tx=0).stop 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4873fff700 1 -- 172.21.15.69:0/985028674 reap_dead start 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 shutdown_connections 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 >> 172.21.15.69:0/985028674 conn(0x7f48740fff20 msgr2=0x7f4874108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 shutdown_connections 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 -- 172.21.15.69:0/985028674 wait complete. 2024-04-02T05:07:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:28.032+0000 7f4879d48700 1 librados: shutdown 2024-04-02T05:07:28.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.495498+0000 mgr.y (mgr.14538) 906 : cluster 0 pgmap v1545: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:28.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.587406+0000 osd.7 (osd.7) 379 : cluster 0 1.0 deep-scrub starts 2024-04-02T05:07:28.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.590234+0000 osd.7 (osd.7) 380 : cluster 0 1.0 deep-scrub ok 2024-04-02T05:07:28.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.751137+0000 osd.6 (osd.6) 515 : cluster 0 3.10 scrub starts 2024-04-02T05:07:28.048 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.754146+0000 osd.6 (osd.6) 516 : cluster 0 3.10 scrub ok 2024-04-02T05:07:28.048 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.495498+0000 mgr.y (mgr.14538) 906 : cluster 0 pgmap v1545: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:07:28.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.587406+0000 osd.7 (osd.7) 379 : cluster 0 1.0 deep-scrub starts 2024-04-02T05:07:28.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.590234+0000 osd.7 (osd.7) 380 : cluster 0 1.0 deep-scrub ok 2024-04-02T05:07:28.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.751137+0000 osd.6 (osd.6) 515 : cluster 0 3.10 scrub starts 2024-04-02T05:07:28.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:27 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.754146+0000 osd.6 (osd.6) 516 : cluster 0 3.10 scrub ok 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.764169+0000 osd.4 (osd.4) 542 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:26.764476+0000 osd.4 (osd.4) 543 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:27.530465+0000 osd.4 (osd.4) 544 : cluster 0 4.c scrub starts 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:27.532772+0000 osd.4 (osd.4) 545 : cluster 0 4.c scrub ok 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:27.696742+0000 mon.a (mon.0) 4199 : cluster 0 osdmap e966: 8 total, 8 up, 8 in 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:27.754915+0000 osd.1 (osd.1) 583 : cluster 0 4.12 scrub starts 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:27.757328+0000 osd.1 (osd.1) 584 : cluster 0 4.12 scrub ok 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.180939+0000 osd.5 (osd.5) 399 : cluster 0 2.18 scrub starts 2024-04-02T05:07:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:28 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.183254+0000 osd.5 (osd.5) 400 : cluster 0 2.18 scrub ok 2024-04-02T05:07:29.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:07:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:07:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.764169+0000 osd.4 (osd.4) 542 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:26.764476+0000 osd.4 (osd.4) 543 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:27.530465+0000 osd.4 (osd.4) 544 : cluster 0 4.c scrub starts 2024-04-02T05:07:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:27.532772+0000 osd.4 (osd.4) 545 : cluster 0 4.c scrub ok 2024-04-02T05:07:29.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:27.696742+0000 mon.a (mon.0) 4199 : cluster 0 osdmap e966: 8 total, 8 up, 8 in 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:27.754915+0000 osd.1 (osd.1) 583 : cluster 0 4.12 scrub starts 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:27.757328+0000 osd.1 (osd.1) 584 : cluster 0 4.12 scrub ok 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.180939+0000 osd.5 (osd.5) 399 : cluster 0 2.18 scrub starts 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.183254+0000 osd.5 (osd.5) 400 : cluster 0 2.18 scrub ok 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.764169+0000 osd.4 (osd.4) 542 : cluster 0 purged_snaps scrub starts 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:26.764476+0000 osd.4 (osd.4) 543 : cluster 0 purged_snaps scrub ok 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:27.530465+0000 osd.4 (osd.4) 544 : cluster 0 4.c scrub starts 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:27.532772+0000 osd.4 (osd.4) 545 : cluster 0 4.c scrub ok 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:27.696742+0000 mon.a (mon.0) 4199 : cluster 0 osdmap e966: 8 total, 8 up, 8 in 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:27.754915+0000 osd.1 (osd.1) 583 : cluster 0 4.12 scrub starts 2024-04-02T05:07:29.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:27.757328+0000 osd.1 (osd.1) 584 : cluster 0 4.12 scrub ok 2024-04-02T05:07:29.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.180939+0000 osd.5 (osd.5) 399 : cluster 0 2.18 scrub starts 2024-04-02T05:07:29.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:28 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.183254+0000 osd.5 (osd.5) 400 : cluster 0 2.18 scrub ok 2024-04-02T05:07:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:29 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.497303+0000 mgr.y (mgr.14538) 907 : cluster 0 pgmap v1547: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:29 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.792022+0000 osd.6 (osd.6) 517 : cluster 0 10.7 scrub starts 2024-04-02T05:07:29.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:29 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.798817+0000 osd.6 (osd.6) 518 : cluster 0 10.7 scrub ok 2024-04-02T05:07:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:29 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.497303+0000 mgr.y (mgr.14538) 907 : cluster 0 pgmap v1547: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:29 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.792022+0000 osd.6 (osd.6) 517 : cluster 0 10.7 scrub starts 2024-04-02T05:07:30.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:29 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.798817+0000 osd.6 (osd.6) 518 : cluster 0 10.7 scrub ok 2024-04-02T05:07:30.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:29 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.497303+0000 mgr.y (mgr.14538) 907 : cluster 0 pgmap v1547: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:30.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:29 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.792022+0000 osd.6 (osd.6) 517 : cluster 0 10.7 scrub starts 2024-04-02T05:07:30.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:29 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.798817+0000 osd.6 (osd.6) 518 : cluster 0 10.7 scrub ok 2024-04-02T05:07:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.973799+0000 osd.0 (osd.0) 539 : cluster 0 4.16 deep-scrub starts 2024-04-02T05:07:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[21528]: cluster 2024-04-02T05:07:28.976174+0000 osd.0 (osd.0) 540 : cluster 0 4.16 deep-scrub ok 2024-04-02T05:07:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[21528]: cluster 2024-04-02T05:07:29.810792+0000 osd.6 (osd.6) 519 : cluster 0 2.7 scrub starts 2024-04-02T05:07:31.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[21528]: cluster 2024-04-02T05:07:29.813165+0000 osd.6 (osd.6) 520 : cluster 0 2.7 scrub ok 2024-04-02T05:07:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.973799+0000 osd.0 (osd.0) 539 : cluster 0 4.16 deep-scrub starts 2024-04-02T05:07:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[28904]: cluster 2024-04-02T05:07:28.976174+0000 osd.0 (osd.0) 540 : cluster 0 4.16 deep-scrub ok 2024-04-02T05:07:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[28904]: cluster 2024-04-02T05:07:29.810792+0000 osd.6 (osd.6) 519 : cluster 0 2.7 scrub starts 2024-04-02T05:07:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:30 smithi069 bash[28904]: cluster 2024-04-02T05:07:29.813165+0000 osd.6 (osd.6) 520 : cluster 0 2.7 scrub ok 2024-04-02T05:07:31.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:30 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.973799+0000 osd.0 (osd.0) 539 : cluster 0 4.16 deep-scrub starts 2024-04-02T05:07:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:30 smithi098 bash[23790]: cluster 2024-04-02T05:07:28.976174+0000 osd.0 (osd.0) 540 : cluster 0 4.16 deep-scrub ok 2024-04-02T05:07:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:30 smithi098 bash[23790]: cluster 2024-04-02T05:07:29.810792+0000 osd.6 (osd.6) 519 : cluster 0 2.7 scrub starts 2024-04-02T05:07:31.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:30 smithi098 bash[23790]: cluster 2024-04-02T05:07:29.813165+0000 osd.6 (osd.6) 520 : cluster 0 2.7 scrub ok 2024-04-02T05:07:31.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[21528]: cluster 2024-04-02T05:07:30.022685+0000 osd.0 (osd.0) 541 : cluster 0 6.18 scrub starts 2024-04-02T05:07:31.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[21528]: cluster 2024-04-02T05:07:30.024246+0000 osd.0 (osd.0) 542 : cluster 0 6.18 scrub ok 2024-04-02T05:07:31.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[21528]: cluster 2024-04-02T05:07:30.498249+0000 mgr.y (mgr.14538) 908 : cluster 0 pgmap v1548: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:31.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[21528]: cluster 2024-04-02T05:07:31.627164+0000 osd.1 (osd.1) 585 : cluster 0 10.2 deep-scrub starts 2024-04-02T05:07:32.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[21528]: cluster 2024-04-02T05:07:31.630036+0000 osd.1 (osd.1) 586 : cluster 0 10.2 deep-scrub ok 2024-04-02T05:07:32.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[28904]: cluster 2024-04-02T05:07:30.022685+0000 osd.0 (osd.0) 541 : cluster 0 6.18 scrub starts 2024-04-02T05:07:32.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[28904]: cluster 2024-04-02T05:07:30.024246+0000 osd.0 (osd.0) 542 : cluster 0 6.18 scrub ok 2024-04-02T05:07:32.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[28904]: cluster 2024-04-02T05:07:30.498249+0000 mgr.y (mgr.14538) 908 : cluster 0 pgmap v1548: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:32.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[28904]: cluster 2024-04-02T05:07:31.627164+0000 osd.1 (osd.1) 585 : cluster 0 10.2 deep-scrub starts 2024-04-02T05:07:32.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:31 smithi069 bash[28904]: cluster 2024-04-02T05:07:31.630036+0000 osd.1 (osd.1) 586 : cluster 0 10.2 deep-scrub ok 2024-04-02T05:07:32.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:31 smithi098 bash[23790]: cluster 2024-04-02T05:07:30.022685+0000 osd.0 (osd.0) 541 : cluster 0 6.18 scrub starts 2024-04-02T05:07:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:31 smithi098 bash[23790]: cluster 2024-04-02T05:07:30.024246+0000 osd.0 (osd.0) 542 : cluster 0 6.18 scrub ok 2024-04-02T05:07:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:31 smithi098 bash[23790]: cluster 2024-04-02T05:07:30.498249+0000 mgr.y (mgr.14538) 908 : cluster 0 pgmap v1548: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:31 smithi098 bash[23790]: cluster 2024-04-02T05:07:31.627164+0000 osd.1 (osd.1) 585 : cluster 0 10.2 deep-scrub starts 2024-04-02T05:07:32.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:31 smithi098 bash[23790]: cluster 2024-04-02T05:07:31.630036+0000 osd.1 (osd.1) 586 : cluster 0 10.2 deep-scrub ok 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[28904]: cluster 2024-04-02T05:07:31.755318+0000 osd.6 (osd.6) 521 : cluster 0 4.1f scrub starts 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[28904]: cluster 2024-04-02T05:07:31.758118+0000 osd.6 (osd.6) 522 : cluster 0 4.1f scrub ok 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[28904]: cluster 2024-04-02T05:07:31.983630+0000 osd.2 (osd.2) 440 : cluster 0 1.14 deep-scrub starts 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[28904]: cluster 2024-04-02T05:07:31.985490+0000 osd.2 (osd.2) 441 : cluster 0 1.14 deep-scrub ok 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.581948+0000 osd.1 (osd.1) 587 : cluster 0 10.19 scrub starts 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.585361+0000 osd.1 (osd.1) 588 : cluster 0 10.19 scrub ok 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[21528]: cluster 2024-04-02T05:07:31.755318+0000 osd.6 (osd.6) 521 : cluster 0 4.1f scrub starts 2024-04-02T05:07:32.916 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[21528]: cluster 2024-04-02T05:07:31.758118+0000 osd.6 (osd.6) 522 : cluster 0 4.1f scrub ok 2024-04-02T05:07:32.917 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[21528]: cluster 2024-04-02T05:07:31.983630+0000 osd.2 (osd.2) 440 : cluster 0 1.14 deep-scrub starts 2024-04-02T05:07:32.917 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[21528]: cluster 2024-04-02T05:07:31.985490+0000 osd.2 (osd.2) 441 : cluster 0 1.14 deep-scrub ok 2024-04-02T05:07:32.917 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.581948+0000 osd.1 (osd.1) 587 : cluster 0 10.19 scrub starts 2024-04-02T05:07:32.917 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:32 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.585361+0000 osd.1 (osd.1) 588 : cluster 0 10.19 scrub ok 2024-04-02T05:07:32.917 INFO:teuthology.orchestra.run.smithi069.stdout:81604378970 2024-04-02T05:07:32.918 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-02T05:07:33.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:32 smithi098 bash[23790]: cluster 2024-04-02T05:07:31.755318+0000 osd.6 (osd.6) 521 : cluster 0 4.1f scrub starts 2024-04-02T05:07:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:32 smithi098 bash[23790]: cluster 2024-04-02T05:07:31.758118+0000 osd.6 (osd.6) 522 : cluster 0 4.1f scrub ok 2024-04-02T05:07:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:32 smithi098 bash[23790]: cluster 2024-04-02T05:07:31.983630+0000 osd.2 (osd.2) 440 : cluster 0 1.14 deep-scrub starts 2024-04-02T05:07:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:32 smithi098 bash[23790]: cluster 2024-04-02T05:07:31.985490+0000 osd.2 (osd.2) 441 : cluster 0 1.14 deep-scrub ok 2024-04-02T05:07:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:32 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.581948+0000 osd.1 (osd.1) 587 : cluster 0 10.19 scrub starts 2024-04-02T05:07:33.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:32 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.585361+0000 osd.1 (osd.1) 588 : cluster 0 10.19 scrub ok 2024-04-02T05:07:34.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.499049+0000 mgr.y (mgr.14538) 909 : cluster 0 pgmap v1549: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.548289+0000 osd.7 (osd.7) 381 : cluster 0 6.3 scrub starts 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.551381+0000 osd.7 (osd.7) 382 : cluster 0 6.3 scrub ok 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.595806+0000 osd.4 (osd.4) 546 : cluster 0 3.14 scrub starts 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[28904]: cluster 2024-04-02T05:07:32.601261+0000 osd.4 (osd.4) 547 : cluster 0 3.14 scrub ok 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.499049+0000 mgr.y (mgr.14538) 909 : cluster 0 pgmap v1549: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.548289+0000 osd.7 (osd.7) 381 : cluster 0 6.3 scrub starts 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.551381+0000 osd.7 (osd.7) 382 : cluster 0 6.3 scrub ok 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.595806+0000 osd.4 (osd.4) 546 : cluster 0 3.14 scrub starts 2024-04-02T05:07:34.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:33 smithi069 bash[21528]: cluster 2024-04-02T05:07:32.601261+0000 osd.4 (osd.4) 547 : cluster 0 3.14 scrub ok 2024-04-02T05:07:34.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:07:33 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:07:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:33 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.499049+0000 mgr.y (mgr.14538) 909 : cluster 0 pgmap v1549: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-02T05:07:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:33 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.548289+0000 osd.7 (osd.7) 381 : cluster 0 6.3 scrub starts 2024-04-02T05:07:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:33 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.551381+0000 osd.7 (osd.7) 382 : cluster 0 6.3 scrub ok 2024-04-02T05:07:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:33 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.595806+0000 osd.4 (osd.4) 546 : cluster 0 3.14 scrub starts 2024-04-02T05:07:34.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:33 smithi098 bash[23790]: cluster 2024-04-02T05:07:32.601261+0000 osd.4 (osd.4) 547 : cluster 0 3.14 scrub ok 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- 172.21.15.69:0/1935234889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 msgr2=0x7f8598111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- 172.21.15.69:0/1935234889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f8598111320 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f859000b0a0 tx=0x7f859000b3b0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- 172.21.15.69:0/1935234889 shutdown_connections 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- 172.21.15.69:0/1935234889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f8598111320 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- 172.21.15.69:0/1935234889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f859810a4e0 0x7f859810a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- 172.21.15.69:0/1935234889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8598104450 0x7f85981048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- 172.21.15.69:0/1935234889 >> 172.21.15.69:0/1935234889 conn(0x7f85980fff20 msgr2=0x7f8598102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:34.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- 172.21.15.69:0/1935234889 shutdown_connections 2024-04-02T05:07:34.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- 172.21.15.69:0/1935234889 wait complete. 2024-04-02T05:07:34.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 librados: starting msgr at 2024-04-02T05:07:34.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 librados: starting objecter 2024-04-02T05:07:34.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 Processor -- start 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- start start 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 librados: setting wanted keys 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 librados: calling monclient init 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598104450 0x7f859819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f859819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f859810a4e0 0x7f859819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8598113c60 con 0x7f8598104e10 2024-04-02T05:07:34.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8598113ae0 con 0x7f8598104450 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859e707700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f8598113de0 con 0x7f859810a4e0 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f85977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f859819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f859cca4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f859810a4e0 0x7f859819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f85977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f859819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45960/0 (socket says 172.21.15.69:45960) 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f85977fe700 1 -- 172.21.15.69:0/1212613996 learned_addr learned my addr 172.21.15.69:0/1212613996 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.299+0000 7f8597fff700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598104450 0x7f859819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859cca4700 1 -- 172.21.15.69:0/1212613996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598104450 msgr2=0x7f859819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859cca4700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8598104450 0x7f859819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859cca4700 1 -- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 msgr2=0x7f859819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859cca4700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f859819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859cca4700 1 -- 172.21.15.69:0/1212613996 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859819f200 con 0x7f859810a4e0 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85977fe700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8598104e10 0x7f859819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859cca4700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f859810a4e0 0x7f859819eac0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f859000bfd0 tx=0x7f8590007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.?.objecter ms_handle_connect 0x7f859810a4e0 2024-04-02T05:07:34.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:34.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590021020 con 0x7f859810a4e0 2024-04-02T05:07:34.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859819f490 con 0x7f859810a4e0 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f8590004c50 con 0x7f859810a4e0 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590005050 con 0x7f859810a4e0 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 10 client.44045.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85981a6d70 con 0x7f859810a4e0 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 1 librados: init done 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 10 client.44045.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 15 client.44045.objecter _session_command_op_assign -1 1 2024-04-02T05:07:34.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 20 client.44045.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 10 client.44045.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8598105b90 con 0x7f859810a4e0 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f8590024020 con 0x7f859810a4e0 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858006f5e0 0x7f8580071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f859000f020 con 0x7f859810a4e0 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.44045.objecter ms_dispatch 0x7f8598001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 3 client.44045.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 3 client.44045.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f8597fff700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858006f5e0 0x7f8580071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.44045.objecter checking command 1 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f858007bb80 0x7f858007e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 20 client.44045.objecter _get_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 15 client.44045.objecter _session_command_op_remove -1 1 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 20 client.44045.objecter _get_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 20 client.44045.objecter get_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 15 client.44045.objecter _session_command_op_assign 3 1 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 4 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.44045.objecter _send_command 1 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8590029e70 con 0x7f858007bb80 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 20 client.44045.objecter dump_active .. 0 homeless 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f8590096050 con 0x7f859810a4e0 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85977fe700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f858007bb80 0x7f858007e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f8597fff700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858006f5e0 0x7f8580071aa0 secure :-1 s=READY pgs=4053 cs=0 l=1 rev1=1 crypto rx=0x7f8588005fd0 tx=0x7f8588005db0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.44045.objecter ms_handle_connect 0x7f858006f5e0 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85977fe700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f858007bb80 0x7f858007e020 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.303+0000 7f85957fa700 10 client.44045.objecter ms_handle_connect 0x7f858007bb80 2024-04-02T05:07:34.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.307+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== osd.3 v2:172.21.15.69:6826/207593208 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8590029e70 con 0x7f858007bb80 2024-04-02T05:07:34.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.307+0000 7f85957fa700 10 client.44045.objecter ms_dispatch 0x7f8598001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:34.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.307+0000 7f85957fa700 10 client.44045.objecter _finish_command 1 = system:0 2024-04-02T05:07:34.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.307+0000 7f85957fa700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 4 2024-04-02T05:07:34.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.307+0000 7f85957fa700 15 client.44045.objecter _session_command_op_remove 3 1 2024-04-02T05:07:34.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 10 client.44045.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:34.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 15 client.44045.objecter _session_command_op_assign -1 2 2024-04-02T05:07:34.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter _get_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter _get_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 15 client.44045.objecter _session_command_op_remove -1 2 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter get_session s=0x7f858007b4b0 osd=3 3 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 15 client.44045.objecter _session_command_op_assign 3 2 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 4 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 10 client.44045.objecter _send_command 2 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 --> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f859804ff00 con 0x7f858007bb80 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f85957fa700 1 -- 172.21.15.69:0/1212613996 <== osd.3 v2:172.21.15.69:6826/207593208 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f859804ff00 con 0x7f858007bb80 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f85957fa700 10 client.44045.objecter ms_dispatch 0x7f8598001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f85957fa700 10 client.44045.objecter _finish_command 2 = system:0 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f85957fa700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 4 2024-04-02T05:07:34.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f85957fa700 15 client.44045.objecter _session_command_op_remove 3 2 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 10 librados: watch_flush enter 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 10 librados: watch_flush exit 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 10 client.44045.objecter close_session for osd.3 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f858007bb80 msgr2=0x7f858007e020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6826/207593208,v1:172.21.15.69:6827/207593208] conn(0x7f858007bb80 0x7f858007e020 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter put_session s=0x7f858007b4b0 osd=3 1 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 20 client.44045.objecter shutdown clearing up homeless session... 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 10 client.44045.objecter successfully canceled tick 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858006f5e0 msgr2=0x7f8580071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f858006f5e0 0x7f8580071aa0 secure :-1 s=READY pgs=4053 cs=0 l=1 rev1=1 crypto rx=0x7f8588005fd0 tx=0x7f8588005db0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f859810a4e0 msgr2=0x7f859819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 --2- 172.21.15.69:0/1212613996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f859810a4e0 0x7f859819eac0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f859000bfd0 tx=0x7f8590007a20 comp rx=0 tx=0).stop 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859cca4700 1 -- 172.21.15.69:0/1212613996 reap_dead start 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 shutdown_connections 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 >> 172.21.15.69:0/1212613996 conn(0x7f85980fff20 msgr2=0x7f8598101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 shutdown_connections 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 -- 172.21.15.69:0/1212613996 wait complete. 2024-04-02T05:07:34.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.323+0000 7f859e707700 1 librados: shutdown 2024-04-02T05:07:34.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 -- 172.21.15.69:0/906115456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 msgr2=0x7f47a410f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 --2- 172.21.15.69:0/906115456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a410f0d0 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f4794009990 tx=0x7f4794009ca0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 -- 172.21.15.69:0/906115456 shutdown_connections 2024-04-02T05:07:34.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 --2- 172.21.15.69:0/906115456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a410f0d0 unknown :-1 s=CLOSED pgs=3339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 --2- 172.21.15.69:0/906115456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47a4108390 0x7f47a4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 --2- 172.21.15.69:0/906115456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47a4102320 0x7f47a41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.451+0000 7f47a8b97700 1 -- 172.21.15.69:0/906115456 >> 172.21.15.69:0/906115456 conn(0x7f47a4079310 msgr2=0x7f47a4079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- 172.21.15.69:0/906115456 shutdown_connections 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- 172.21.15.69:0/906115456 wait complete. 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 librados: starting msgr at 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 librados: starting objecter 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 Processor -- start 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- start start 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 librados: setting wanted keys 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 librados: calling monclient init 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47a4102320 0x7f47a4074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a4074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47a4108390 0x7f47a4074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f47a4111ad0 con 0x7f47a4102320 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47a4111950 con 0x7f47a4108390 2024-04-02T05:07:34.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f47a4111c50 con 0x7f47a4102ce0 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a4074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a4074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52748/0 (socket says 172.21.15.69:52748) 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 -- 172.21.15.69:0/1214549774 learned_addr learned my addr 172.21.15.69:0/1214549774 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a259c700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47a4102320 0x7f47a4074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a2d9d700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47a4108390 0x7f47a4074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 -- 172.21.15.69:0/1214549774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47a4108390 msgr2=0x7f47a4074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47a4108390 0x7f47a4074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 -- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47a4102320 msgr2=0x7f47a4074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47a4102320 0x7f47a4074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 -- 172.21.15.69:0/1214549774 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a41a65b0 con 0x7f47a4102ce0 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a2d9d700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47a4108390 0x7f47a4074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:07:34.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a259c700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47a4102320 0x7f47a4074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a1d9b700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a4074a80 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7f479800c970 tx=0x7f479800cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47937fe700 10 client.?.objecter ms_handle_connect 0x7f47a4102ce0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47937fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798007aa0 con 0x7f47a4102ce0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a41a68a0 con 0x7f47a4102ce0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f4798004cb0 con 0x7f47a4102ce0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798005570 con 0x7f47a4102ce0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 10 client.44051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a41a6de0 con 0x7f47a4102ce0 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 librados: init done 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 10 client.44051.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 15 client.44051.objecter _session_command_op_assign -1 1 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 20 client.44051.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 10 client.44051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:34.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f47a41039b0 con 0x7f47a4102ce0 2024-04-02T05:07:34.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.455+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f4798004750 con 0x7f47a4102ce0 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f478c06f5e0 0x7f478c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47a259c700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f478c06f5e0 0x7f478c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f479808ea00 con 0x7f47a4102ce0 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter ms_dispatch 0x7f47a4001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 3 client.44051.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 3 client.44051.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter checking command 1 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f478c07bb20 0x7f478c07dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter _get_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 15 client.44051.objecter _session_command_op_remove -1 1 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter _get_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter get_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 15 client.44051.objecter _session_command_op_assign 1 1 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 4 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter _send_command 1 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f478c07e690 con 0x7f478c07bb20 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter dump_active .. 0 homeless 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f479808ee50 con 0x7f47a4102ce0 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47a2d9d700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f478c07bb20 0x7f478c07dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter ms_handle_connect 0x7f478c06f5e0 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47a259c700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f478c06f5e0 0x7f478c071aa0 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 crypto rx=0x7f479c009da0 tx=0x7f479c00b040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter ms_handle_connect 0x7f478c07bb20 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47a2d9d700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f478c07bb20 0x7f478c07dfc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== osd.1 v2:172.21.15.69:6810/2651666015 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f478c07e690 con 0x7f478c07bb20 2024-04-02T05:07:34.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter ms_dispatch 0x7f47a4001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:34.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 10 client.44051.objecter _finish_command 1 = system:0 2024-04-02T05:07:34.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 4 2024-04-02T05:07:34.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.459+0000 7f47937fe700 15 client.44051.objecter _session_command_op_remove 1 1 2024-04-02T05:07:34.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 10 client.44051.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:34.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 15 client.44051.objecter _session_command_op_assign -1 2 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 20 client.44051.objecter _get_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 20 client.44051.objecter _get_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 15 client.44051.objecter _session_command_op_remove -1 2 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 20 client.44051.objecter get_session s=0x7f478c07b450 osd=1 3 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 15 client.44051.objecter _session_command_op_assign 1 2 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 4 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 10 client.44051.objecter _send_command 2 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 --> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f47a404ff00 con 0x7f478c07bb20 2024-04-02T05:07:34.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47937fe700 1 -- 172.21.15.69:0/1214549774 <== osd.1 v2:172.21.15.69:6810/2651666015 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f47a404ff00 con 0x7f478c07bb20 2024-04-02T05:07:34.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47937fe700 10 client.44051.objecter ms_dispatch 0x7f47a4001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:34.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47937fe700 10 client.44051.objecter _finish_command 2 = system:0 2024-04-02T05:07:34.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47937fe700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 4 2024-04-02T05:07:34.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47937fe700 15 client.44051.objecter _session_command_op_remove 1 2 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 10 librados: watch_flush enter 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 10 librados: watch_flush exit 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 10 client.44051.objecter close_session for osd.1 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.475+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f478c07bb20 msgr2=0x7f478c07dfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6810/2651666015,v1:172.21.15.69:6811/2651666015] conn(0x7f478c07bb20 0x7f478c07dfc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 20 client.44051.objecter put_session s=0x7f478c07b450 osd=1 1 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 20 client.44051.objecter shutdown clearing up homeless session... 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 10 client.44051.objecter successfully canceled tick 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f478c06f5e0 msgr2=0x7f478c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f478c06f5e0 0x7f478c071aa0 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 crypto rx=0x7f479c009da0 tx=0x7f479c00b040 comp rx=0 tx=0).stop 2024-04-02T05:07:34.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 msgr2=0x7f47a4074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 --2- 172.21.15.69:0/1214549774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47a4102ce0 0x7f47a4074a80 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7f479800c970 tx=0x7f479800cd30 comp rx=0 tx=0).stop 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a2d9d700 1 -- 172.21.15.69:0/1214549774 reap_dead start 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 shutdown_connections 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 >> 172.21.15.69:0/1214549774 conn(0x7f47a4079310 msgr2=0x7f47a4109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 shutdown_connections 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 -- 172.21.15.69:0/1214549774 wait complete. 2024-04-02T05:07:34.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.479+0000 7f47a8b97700 1 librados: shutdown 2024-04-02T05:07:34.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1261300221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec1044a0 msgr2=0x7f2cec104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1261300221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec1044a0 0x7f2cec104920 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00099c0 tx=0x7f2ce0009cd0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1261300221 shutdown_connections 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1261300221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec104e60 0x7f2cec111320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1261300221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2cec10a550 0x7f2cec10a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1261300221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec1044a0 0x7f2cec104920 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1261300221 >> 172.21.15.69:0/1261300221 conn(0x7f2cec0fff70 msgr2=0x7f2cec102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1261300221 shutdown_connections 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1261300221 wait complete. 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 librados: starting msgr at 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.659+0000 7f2cf1bfa700 1 librados: starting objecter 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 Processor -- start 2024-04-02T05:07:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- start start 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 librados: setting wanted keys 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 librados: calling monclient init 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2cec1044a0 0x7f2cec19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec104e60 0x7f2cec19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec10a550 0x7f2cec19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec113c80 con 0x7f2cec104e60 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec113b00 con 0x7f2cec10a550 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f2cec113e00 con 0x7f2cec1044a0 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec104e60 0x7f2cec19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceb7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2cec1044a0 0x7f2cec19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec104e60 0x7f2cec19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46000/0 (socket says 172.21.15.69:46000) 2024-04-02T05:07:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 -- 172.21.15.69:0/1302568346 learned_addr learned my addr 172.21.15.69:0/1302568346 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cebfff700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec10a550 0x7f2cec19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 -- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2cec1044a0 msgr2=0x7f2cec19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2cec1044a0 0x7f2cec19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 -- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec10a550 msgr2=0x7f2cec19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec10a550 0x7f2cec19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 -- 172.21.15.69:0/1302568346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec19f240 con 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceb7fe700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2cec1044a0 0x7f2cec19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cebfff700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cec10a550 0x7f2cec19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceaffd700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec104e60 0x7f2cec19a770 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0006010 tx=0x7f2ce0003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 10 client.?.objecter ms_handle_connect 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0009e60 con 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f2ce000f460 con 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cec19f4d0 con 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 10 client.33993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0017650 con 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cec1a6db0 con 0x7f2cec104e60 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 librados: init done 2024-04-02T05:07:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 10 client.33993.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 15 client.33993.objecter _session_command_op_assign -1 1 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 20 client.33993.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 10 client.33993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2cd8002950 con 0x7f2cec104e60 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f2ce00177f0 con 0x7f2cec104e60 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2cd406f5e0 0x7f2cd4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f2ce008f400 con 0x7f2cec104e60 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceb7fe700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2cd406f5e0 0x7f2cd4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 10 client.33993.objecter ms_dispatch 0x7f2cec001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 3 client.33993.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 3 client.33993.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 10 client.33993.objecter checking command 1 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f2cd407baa0 0x7f2cd407df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 20 client.33993.objecter _get_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 15 client.33993.objecter _session_command_op_remove -1 1 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 20 client.33993.objecter _get_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 20 client.33993.objecter get_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 15 client.33993.objecter _session_command_op_assign 5 1 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 4 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 10 client.33993.objecter _send_command 1 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ce000b790 con 0x7f2cd407baa0 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 20 client.33993.objecter dump_active .. 0 homeless 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f2ce0096050 con 0x7f2cec104e60 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.663+0000 7f2ceb7fe700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2cd406f5e0 0x7f2cd4071aa0 secure :-1 s=READY pgs=4055 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009680 tx=0x7f2cdc009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 10 client.33993.objecter ms_handle_connect 0x7f2cd406f5e0 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2cebfff700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f2cd407baa0 0x7f2cd407df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 10 client.33993.objecter ms_handle_connect 0x7f2cd407baa0 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2cebfff700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f2cd407baa0 0x7f2cd407df40 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== osd.5 v2:172.21.15.98:6808/219578185 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ce000b790 con 0x7f2cd407baa0 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 10 client.33993.objecter ms_dispatch 0x7f2cec001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 10 client.33993.objecter _finish_command 1 = system:0 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 4 2024-04-02T05:07:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.667+0000 7f2ce8ff9700 15 client.33993.objecter _session_command_op_remove 5 1 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 10 client.33993.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 15 client.33993.objecter _session_command_op_assign -1 2 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 20 client.33993.objecter _get_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 20 client.33993.objecter _get_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 15 client.33993.objecter _session_command_op_remove -1 2 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 20 client.33993.objecter get_session s=0x7f2cd407b3d0 osd=5 3 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 15 client.33993.objecter _session_command_op_assign 5 2 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 4 2024-04-02T05:07:34.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 10 client.33993.objecter _send_command 2 2024-04-02T05:07:34.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.683+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 --> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2cd8001250 con 0x7f2cd407baa0 2024-04-02T05:07:34.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2ce8ff9700 1 -- 172.21.15.69:0/1302568346 <== osd.5 v2:172.21.15.98:6808/219578185 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2cd8001250 con 0x7f2cd407baa0 2024-04-02T05:07:34.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2ce8ff9700 10 client.33993.objecter ms_dispatch 0x7f2cec001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:34.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2ce8ff9700 10 client.33993.objecter _finish_command 2 = system:0 2024-04-02T05:07:34.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2ce8ff9700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 4 2024-04-02T05:07:34.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2ce8ff9700 15 client.33993.objecter _session_command_op_remove 5 2 2024-04-02T05:07:34.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 10 librados: watch_flush enter 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 10 librados: watch_flush exit 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 10 client.33993.objecter close_session for osd.5 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f2cd407baa0 msgr2=0x7f2cd407df40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.98:6808/219578185,v1:172.21.15.98:6809/219578185] conn(0x7f2cd407baa0 0x7f2cd407df40 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 20 client.33993.objecter put_session s=0x7f2cd407b3d0 osd=5 1 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 20 client.33993.objecter shutdown clearing up homeless session... 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 10 client.33993.objecter successfully canceled tick 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2cd406f5e0 msgr2=0x7f2cd4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f2cd406f5e0 0x7f2cd4071aa0 secure :-1 s=READY pgs=4055 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009680 tx=0x7f2cdc009260 comp rx=0 tx=0).stop 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec104e60 msgr2=0x7f2cec19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 --2- 172.21.15.69:0/1302568346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cec104e60 0x7f2cec19a770 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0006010 tx=0x7f2ce0003da0 comp rx=0 tx=0).stop 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cebfff700 1 -- 172.21.15.69:0/1302568346 reap_dead start 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 shutdown_connections 2024-04-02T05:07:34.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 >> 172.21.15.69:0/1302568346 conn(0x7f2cec0fff70 msgr2=0x7f2cec10b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:34.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 shutdown_connections 2024-04-02T05:07:34.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 -- 172.21.15.69:0/1302568346 wait complete. 2024-04-02T05:07:34.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:34.687+0000 7f2cf1bfa700 1 librados: shutdown 2024-04-02T05:07:34.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[28904]: cluster 2024-04-02T05:07:33.574341+0000 osd.4 (osd.4) 548 : cluster 0 1.e deep-scrub starts 2024-04-02T05:07:34.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[28904]: cluster 2024-04-02T05:07:33.576752+0000 osd.4 (osd.4) 549 : cluster 0 1.e deep-scrub ok 2024-04-02T05:07:34.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[28904]: cluster 2024-04-02T05:07:33.584131+0000 osd.3 (osd.3) 603 : cluster 0 3.1e scrub starts 2024-04-02T05:07:34.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[28904]: cluster 2024-04-02T05:07:33.588606+0000 osd.3 (osd.3) 604 : cluster 0 3.1e scrub ok 2024-04-02T05:07:34.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[28904]: cluster 2024-04-02T05:07:33.941389+0000 osd.2 (osd.2) 442 : cluster 0 6.14 scrub starts 2024-04-02T05:07:34.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[28904]: cluster 2024-04-02T05:07:33.943912+0000 osd.2 (osd.2) 443 : cluster 0 6.14 scrub ok 2024-04-02T05:07:34.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[21528]: cluster 2024-04-02T05:07:33.574341+0000 osd.4 (osd.4) 548 : cluster 0 1.e deep-scrub starts 2024-04-02T05:07:34.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[21528]: cluster 2024-04-02T05:07:33.576752+0000 osd.4 (osd.4) 549 : cluster 0 1.e deep-scrub ok 2024-04-02T05:07:34.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[21528]: cluster 2024-04-02T05:07:33.584131+0000 osd.3 (osd.3) 603 : cluster 0 3.1e scrub starts 2024-04-02T05:07:34.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[21528]: cluster 2024-04-02T05:07:33.588606+0000 osd.3 (osd.3) 604 : cluster 0 3.1e scrub ok 2024-04-02T05:07:34.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[21528]: cluster 2024-04-02T05:07:33.941389+0000 osd.2 (osd.2) 442 : cluster 0 6.14 scrub starts 2024-04-02T05:07:34.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:34 smithi069 bash[21528]: cluster 2024-04-02T05:07:33.943912+0000 osd.2 (osd.2) 443 : cluster 0 6.14 scrub ok 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- 172.21.15.69:0/3738052488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda810ecd0 msgr2=0x7fbda81110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/3738052488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda810ecd0 0x7fbda81110c0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fbda000b0a0 tx=0x7fbda000b3b0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- 172.21.15.69:0/3738052488 shutdown_connections 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/3738052488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda810ecd0 0x7fbda81110c0 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/3738052488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074470 0x7fbda8074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/3738052488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda8074d90 0x7fbda810e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- 172.21.15.69:0/3738052488 >> 172.21.15.69:0/3738052488 conn(0x7fbda8079310 msgr2=0x7fbda8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:35.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- 172.21.15.69:0/3738052488 shutdown_connections 2024-04-02T05:07:35.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- 172.21.15.69:0/3738052488 wait complete. 2024-04-02T05:07:35.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 librados: starting msgr at 2024-04-02T05:07:35.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 librados: starting objecter 2024-04-02T05:07:35.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 Processor -- start 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- start start 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 librados: setting wanted keys 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 librados: calling monclient init 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda8074470 0x7fbda819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074d90 0x7fbda819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda810ecd0 0x7fbda819ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:35.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8113c40 con 0x7fbda8074470 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8113ac0 con 0x7fbda810ecd0 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdb02aa700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fbda8113dc0 con 0x7fbda8074d90 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.119+0000 7fbdad845700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074d90 0x7fbda819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae046700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda8074470 0x7fbda819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae046700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda8074470 0x7fbda819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46012/0 (socket says 172.21.15.69:46012) 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074d90 0x7fbda819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52792/0 (socket says 172.21.15.69:52792) 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae046700 1 -- 172.21.15.69:0/1143197898 learned_addr learned my addr 172.21.15.69:0/1143197898 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae847700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda810ecd0 0x7fbda819ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 -- 172.21.15.69:0/1143197898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda810ecd0 msgr2=0x7fbda819ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda810ecd0 0x7fbda819ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 -- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda8074470 msgr2=0x7fbda819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda8074470 0x7fbda819a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 -- 172.21.15.69:0/1143197898 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda819f160 con 0x7fbda8074d90 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae847700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbda810ecd0 0x7fbda819ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae046700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbda8074470 0x7fbda819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdad845700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074d90 0x7fbda819a690 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fbda400c970 tx=0x7fbda400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 10 client.?.objecter ms_handle_connect 0x7fbda8074d90 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda4014070 con 0x7fbda8074d90 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda819f450 con 0x7fbda8074d90 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fbda4004440 con 0x7fbda8074d90 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 10 client.44063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda81a6cd0 con 0x7fbda8074d90 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 1 librados: init done 2024-04-02T05:07:35.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda4003a60 con 0x7fbda8074d90 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 10 client.44063.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 15 client.44063.objecter _session_command_op_assign -1 1 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 20 client.44063.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 10 client.44063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbda8075b00 con 0x7fbda8074d90 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fbda4017070 con 0x7fbda8074d90 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbd9406f5e0 0x7fbd94071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:35.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7fbda408e550 con 0x7fbda8074d90 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 10 client.44063.objecter ms_dispatch 0x7fbda8001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 3 client.44063.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 3 client.44063.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae046700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbd9406f5e0 0x7fbd94071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 10 client.44063.objecter checking command 1 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbd9407baa0 0x7fbd9407df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 20 client.44063.objecter _get_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 15 client.44063.objecter _session_command_op_remove -1 1 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 20 client.44063.objecter _get_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 20 client.44063.objecter get_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 15 client.44063.objecter _session_command_op_assign 0 1 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 4 2024-04-02T05:07:35.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 10 client.44063.objecter _send_command 1 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbda4016ec0 con 0x7fbd9407baa0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 20 client.44063.objecter dump_active .. 0 homeless 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7fbda408e9a0 con 0x7fbda8074d90 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae847700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbd9407baa0 0x7fbd9407df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbd9f7fe700 10 client.44063.objecter ms_handle_connect 0x7fbd9406f5e0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.123+0000 7fbdae046700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbd9406f5e0 0x7fbd94071aa0 secure :-1 s=READY pgs=4056 cs=0 l=1 rev1=1 crypto rx=0x7fbd98011630 tx=0x7fbd98009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbd9f7fe700 10 client.44063.objecter ms_handle_connect 0x7fbd9407baa0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbdae847700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbd9407baa0 0x7fbd9407df40 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== osd.0 v2:172.21.15.69:6802/1087500595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbda4016ec0 con 0x7fbd9407baa0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbd9f7fe700 10 client.44063.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbd9f7fe700 10 client.44063.objecter _finish_command 1 = system:0 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbd9f7fe700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 4 2024-04-02T05:07:35.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.127+0000 7fbd9f7fe700 15 client.44063.objecter _session_command_op_remove 0 1 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 10 client.44063.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 15 client.44063.objecter _session_command_op_assign -1 2 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter _get_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter _get_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 15 client.44063.objecter _session_command_op_remove -1 2 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter get_session s=0x7fbd9407b3d0 osd=0 3 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 15 client.44063.objecter _session_command_op_assign 0 2 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 4 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 10 client.44063.objecter _send_command 2 2024-04-02T05:07:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 --> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbda804ff00 con 0x7fbd9407baa0 2024-04-02T05:07:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbd9f7fe700 1 -- 172.21.15.69:0/1143197898 <== osd.0 v2:172.21.15.69:6802/1087500595 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbda804ff00 con 0x7fbd9407baa0 2024-04-02T05:07:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbd9f7fe700 10 client.44063.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbd9f7fe700 10 client.44063.objecter _finish_command 2 = system:0 2024-04-02T05:07:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbd9f7fe700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 4 2024-04-02T05:07:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbd9f7fe700 15 client.44063.objecter _session_command_op_remove 0 2 2024-04-02T05:07:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 10 librados: watch_flush enter 2024-04-02T05:07:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 10 librados: watch_flush exit 2024-04-02T05:07:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 10 client.44063.objecter close_session for osd.0 2024-04-02T05:07:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbd9407baa0 msgr2=0x7fbd9407df40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6802/1087500595,v1:172.21.15.69:6803/1087500595] conn(0x7fbd9407baa0 0x7fbd9407df40 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter put_session s=0x7fbd9407b3d0 osd=0 1 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 20 client.44063.objecter shutdown clearing up homeless session... 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 10 client.44063.objecter successfully canceled tick 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbd9406f5e0 msgr2=0x7fbd94071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fbd9406f5e0 0x7fbd94071aa0 secure :-1 s=READY pgs=4056 cs=0 l=1 rev1=1 crypto rx=0x7fbd98011630 tx=0x7fbd98009260 comp rx=0 tx=0).stop 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074d90 msgr2=0x7fbda819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 --2- 172.21.15.69:0/1143197898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbda8074d90 0x7fbda819a690 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fbda400c970 tx=0x7fbda400cc80 comp rx=0 tx=0).stop 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdae847700 1 -- 172.21.15.69:0/1143197898 reap_dead start 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 shutdown_connections 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 >> 172.21.15.69:0/1143197898 conn(0x7fbda8079310 msgr2=0x7fbda8106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 shutdown_connections 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 -- 172.21.15.69:0/1143197898 wait complete. 2024-04-02T05:07:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:35.143+0000 7fbdb02aa700 1 librados: shutdown 2024-04-02T05:07:35.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:34 smithi098 bash[23790]: cluster 2024-04-02T05:07:33.574341+0000 osd.4 (osd.4) 548 : cluster 0 1.e deep-scrub starts 2024-04-02T05:07:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:34 smithi098 bash[23790]: cluster 2024-04-02T05:07:33.576752+0000 osd.4 (osd.4) 549 : cluster 0 1.e deep-scrub ok 2024-04-02T05:07:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:34 smithi098 bash[23790]: cluster 2024-04-02T05:07:33.584131+0000 osd.3 (osd.3) 603 : cluster 0 3.1e scrub starts 2024-04-02T05:07:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:34 smithi098 bash[23790]: cluster 2024-04-02T05:07:33.588606+0000 osd.3 (osd.3) 604 : cluster 0 3.1e scrub ok 2024-04-02T05:07:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:34 smithi098 bash[23790]: cluster 2024-04-02T05:07:33.941389+0000 osd.2 (osd.2) 442 : cluster 0 6.14 scrub starts 2024-04-02T05:07:35.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:34 smithi098 bash[23790]: cluster 2024-04-02T05:07:33.943912+0000 osd.2 (osd.2) 443 : cluster 0 6.14 scrub ok 2024-04-02T05:07:36.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: audit 2024-04-02T05:07:33.988009+0000 mgr.y (mgr.14538) 910 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:34.500086+0000 mgr.y (mgr.14538) 911 : cluster 0 pgmap v1550: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:34.626774+0000 osd.3 (osd.3) 605 : cluster 0 2.19 scrub starts 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:34.628678+0000 osd.3 (osd.3) 606 : cluster 0 2.19 scrub ok 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:34.730462+0000 osd.6 (osd.6) 523 : cluster 0 10.1a scrub starts 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:34.733211+0000 osd.6 (osd.6) 524 : cluster 0 10.1a scrub ok 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: audit 2024-04-02T05:07:34.758026+0000 mon.b (mon.1) 358 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:35.114432+0000 osd.5 (osd.5) 401 : cluster 0 3.1d scrub starts 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:35.118888+0000 osd.5 (osd.5) 402 : cluster 0 3.1d scrub ok 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:35.607902+0000 osd.4 (osd.4) 550 : cluster 0 5.1 deep-scrub starts 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[21528]: cluster 2024-04-02T05:07:35.610363+0000 osd.4 (osd.4) 551 : cluster 0 5.1 deep-scrub ok 2024-04-02T05:07:36.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: audit 2024-04-02T05:07:33.988009+0000 mgr.y (mgr.14538) 910 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:34.500086+0000 mgr.y (mgr.14538) 911 : cluster 0 pgmap v1550: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:34.626774+0000 osd.3 (osd.3) 605 : cluster 0 2.19 scrub starts 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:34.628678+0000 osd.3 (osd.3) 606 : cluster 0 2.19 scrub ok 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:34.730462+0000 osd.6 (osd.6) 523 : cluster 0 10.1a scrub starts 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:34.733211+0000 osd.6 (osd.6) 524 : cluster 0 10.1a scrub ok 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: audit 2024-04-02T05:07:34.758026+0000 mon.b (mon.1) 358 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:35.114432+0000 osd.5 (osd.5) 401 : cluster 0 3.1d scrub starts 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:35.118888+0000 osd.5 (osd.5) 402 : cluster 0 3.1d scrub ok 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:35.607902+0000 osd.4 (osd.4) 550 : cluster 0 5.1 deep-scrub starts 2024-04-02T05:07:36.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:35 smithi069 bash[28904]: cluster 2024-04-02T05:07:35.610363+0000 osd.4 (osd.4) 551 : cluster 0 5.1 deep-scrub ok 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: audit 2024-04-02T05:07:33.988009+0000 mgr.y (mgr.14538) 910 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:34.500086+0000 mgr.y (mgr.14538) 911 : cluster 0 pgmap v1550: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:34.626774+0000 osd.3 (osd.3) 605 : cluster 0 2.19 scrub starts 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:34.628678+0000 osd.3 (osd.3) 606 : cluster 0 2.19 scrub ok 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:34.730462+0000 osd.6 (osd.6) 523 : cluster 0 10.1a scrub starts 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:34.733211+0000 osd.6 (osd.6) 524 : cluster 0 10.1a scrub ok 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: audit 2024-04-02T05:07:34.758026+0000 mon.b (mon.1) 358 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:35.114432+0000 osd.5 (osd.5) 401 : cluster 0 3.1d scrub starts 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:35.118888+0000 osd.5 (osd.5) 402 : cluster 0 3.1d scrub ok 2024-04-02T05:07:36.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:35.607902+0000 osd.4 (osd.4) 550 : cluster 0 5.1 deep-scrub starts 2024-04-02T05:07:36.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:35 smithi098 bash[23790]: cluster 2024-04-02T05:07:35.610363+0000 osd.4 (osd.4) 551 : cluster 0 5.1 deep-scrub ok 2024-04-02T05:07:36.730 INFO:teuthology.orchestra.run.smithi069.stdout:124554051921 2024-04-02T05:07:36.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-02T05:07:36.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:36 smithi069 bash[28904]: audit 2024-04-02T05:07:36.484230+0000 mon.b (mon.1) 359 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:07:36.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:36 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.600376+0000 osd.4 (osd.4) 552 : cluster 0 10.10 scrub starts 2024-04-02T05:07:36.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:36 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.602400+0000 osd.4 (osd.4) 553 : cluster 0 10.10 scrub ok 2024-04-02T05:07:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:36 smithi069 bash[21528]: audit 2024-04-02T05:07:36.484230+0000 mon.b (mon.1) 359 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:07:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:36 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.600376+0000 osd.4 (osd.4) 552 : cluster 0 10.10 scrub starts 2024-04-02T05:07:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:36 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.602400+0000 osd.4 (osd.4) 553 : cluster 0 10.10 scrub ok 2024-04-02T05:07:37.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 -- 172.21.15.69:0/2774427016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 msgr2=0x7f9464100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 --2- 172.21.15.69:0/2774427016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464100d60 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f94500098f0 tx=0x7f9450009c00 comp rx=0 tx=0).stop 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 -- 172.21.15.69:0/2774427016 shutdown_connections 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 --2- 172.21.15.69:0/2774427016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641012a0 0x7f946410d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 --2- 172.21.15.69:0/2774427016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94641068e0 0x7f9464106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 --2- 172.21.15.69:0/2774427016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464100d60 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 -- 172.21.15.69:0/2774427016 >> 172.21.15.69:0/2774427016 conn(0x7f9464077740 msgr2=0x7f94640feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 -- 172.21.15.69:0/2774427016 shutdown_connections 2024-04-02T05:07:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 -- 172.21.15.69:0/2774427016 wait complete. 2024-04-02T05:07:37.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 librados: starting msgr at 2024-04-02T05:07:37.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 librados: starting objecter 2024-04-02T05:07:37.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 Processor -- start 2024-04-02T05:07:37.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 -- start start 2024-04-02T05:07:37.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 librados: setting wanted keys 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.027+0000 7f946ac2e700 1 librados: calling monclient init 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94641012a0 0x7f9464197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641068e0 0x7f946419b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94689ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94691cb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641068e0 0x7f946419b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94691cb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641068e0 0x7f946419b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52804/0 (socket says 172.21.15.69:52804) 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94689ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464196e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.69:54444/0 (socket says 172.21.15.69:54444) 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94641012a0 0x7f9464197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94691cb700 1 -- 172.21.15.69:0/3544365370 learned_addr learned my addr 172.21.15.69:0/3544365370 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9464110190 con 0x7f94641012a0 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9464110010 con 0x7f94641008e0 2024-04-02T05:07:37.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f9464110310 con 0x7f94641068e0 2024-04-02T05:07:37.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 -- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641068e0 msgr2=0x7f946419b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641068e0 0x7f946419b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 -- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 msgr2=0x7f9464196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946419be50 con 0x7f94641012a0 2024-04-02T05:07:37.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94691cb700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94641068e0 0x7f946419b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9463fff700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94641012a0 0x7f9464197380 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f9450009fd0 tx=0x7f945000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9461ffb700 10 client.?.objecter ms_handle_connect 0x7f94641012a0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9461ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9450025070 con 0x7f94641012a0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f945000ba30 con 0x7f94641012a0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9450003be0 con 0x7f94641012a0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f94689ca700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94641008e0 0x7f9464196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946419c0e0 con 0x7f94641012a0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 10 client.34005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946419c4b0 con 0x7f94641012a0 2024-04-02T05:07:37.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 librados: init done 2024-04-02T05:07:37.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 10 client.34005.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:37.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 15 client.34005.objecter _session_command_op_assign -1 1 2024-04-02T05:07:37.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 20 client.34005.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:37.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 10 client.34005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.031+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9464102020 con 0x7f94641012a0 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f9450028070 con 0x7f94641012a0 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f945406f5e0 0x7f9454071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f945008f290 con 0x7f94641012a0 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 10 client.34005.objecter ms_dispatch 0x7f9464001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 3 client.34005.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:37.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 3 client.34005.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:37.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 10 client.34005.objecter checking command 1 2024-04-02T05:07:37.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f945407bb20 0x7f945407dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 20 client.34005.objecter _get_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 15 client.34005.objecter _session_command_op_remove -1 1 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 20 client.34005.objecter _get_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 20 client.34005.objecter get_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 15 client.34005.objecter _session_command_op_assign 7 1 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 4 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 10 client.34005.objecter _send_command 1 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9450001d30 con 0x7f945407bb20 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 20 client.34005.objecter dump_active .. 0 homeless 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f9450096050 con 0x7f94641012a0 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f94689ca700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f945406f5e0 0x7f9454071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f94691cb700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f945407bb20 0x7f945407dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 10 client.34005.objecter ms_handle_connect 0x7f945406f5e0 2024-04-02T05:07:37.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f94689ca700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f945406f5e0 0x7f9454071aa0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7f9458009ca0 tx=0x7f9458009260 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f9461ffb700 10 client.34005.objecter ms_handle_connect 0x7f945407bb20 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.035+0000 7f94691cb700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f945407bb20 0x7f945407dfc0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.039+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== osd.7 v2:172.21.15.98:6824/575416324 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9450001d30 con 0x7f945407bb20 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.039+0000 7f9461ffb700 10 client.34005.objecter ms_dispatch 0x7f9464001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.039+0000 7f9461ffb700 10 client.34005.objecter _finish_command 1 = system:0 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.039+0000 7f9461ffb700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 4 2024-04-02T05:07:37.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.039+0000 7f9461ffb700 15 client.34005.objecter _session_command_op_remove 7 1 2024-04-02T05:07:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 10 client.34005.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 15 client.34005.objecter _session_command_op_assign -1 2 2024-04-02T05:07:37.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 20 client.34005.objecter _get_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 20 client.34005.objecter _get_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 15 client.34005.objecter _session_command_op_remove -1 2 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 20 client.34005.objecter get_session s=0x7f945407b450 osd=7 3 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 15 client.34005.objecter _session_command_op_assign 7 2 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 4 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 10 client.34005.objecter _send_command 2 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 --> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f946404ff00 con 0x7f945407bb20 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f9461ffb700 1 -- 172.21.15.69:0/3544365370 <== osd.7 v2:172.21.15.98:6824/575416324 2 ==== command_reply(tid 2: 0 ) ==== 8+0+13 (crc 0 0 0) 0x7f946404ff00 con 0x7f945407bb20 2024-04-02T05:07:37.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f9461ffb700 10 client.34005.objecter ms_dispatch 0x7f9464001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:37.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f9461ffb700 10 client.34005.objecter _finish_command 2 = system:0 2024-04-02T05:07:37.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f9461ffb700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 4 2024-04-02T05:07:37.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.055+0000 7f9461ffb700 15 client.34005.objecter _session_command_op_remove 7 2 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 10 librados: watch_flush enter 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 10 librados: watch_flush exit 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 10 client.34005.objecter close_session for osd.7 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f945407bb20 msgr2=0x7f945407dfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.98:6824/575416324,v1:172.21.15.98:6825/575416324] conn(0x7f945407bb20 0x7f945407dfc0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 20 client.34005.objecter put_session s=0x7f945407b450 osd=7 1 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 20 client.34005.objecter shutdown clearing up homeless session... 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 10 client.34005.objecter successfully canceled tick 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f945406f5e0 msgr2=0x7f9454071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f945406f5e0 0x7f9454071aa0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7f9458009ca0 tx=0x7f9458009260 comp rx=0 tx=0).stop 2024-04-02T05:07:37.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94641012a0 msgr2=0x7f9464197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 --2- 172.21.15.69:0/3544365370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94641012a0 0x7f9464197380 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f9450009fd0 tx=0x7f945000b790 comp rx=0 tx=0).stop 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f94691cb700 1 -- 172.21.15.69:0/3544365370 reap_dead start 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 shutdown_connections 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 >> 172.21.15.69:0/3544365370 conn(0x7f9464077740 msgr2=0x7f94640feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 shutdown_connections 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 -- 172.21.15.69:0/3544365370 wait complete. 2024-04-02T05:07:37.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.059+0000 7f946ac2e700 1 librados: shutdown 2024-04-02T05:07:37.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- 172.21.15.69:0/2604315954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a8c104e20 msgr2=0x7f0a8c111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/2604315954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a8c104e20 0x7f0a8c111330 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400b0a0 tx=0x7f0a8400b3b0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- 172.21.15.69:0/2604315954 shutdown_connections 2024-04-02T05:07:37.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/2604315954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a8c104e20 0x7f0a8c111330 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/2604315954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c10a4f0 0x7f0a8c10a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/2604315954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c104460 0x7f0a8c1048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- 172.21.15.69:0/2604315954 >> 172.21.15.69:0/2604315954 conn(0x7f0a8c0fff50 msgr2=0x7f0a8c102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:37.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- 172.21.15.69:0/2604315954 shutdown_connections 2024-04-02T05:07:37.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- 172.21.15.69:0/2604315954 wait complete. 2024-04-02T05:07:37.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 librados: starting msgr at 2024-04-02T05:07:37.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 librados: starting objecter 2024-04-02T05:07:37.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 Processor -- start 2024-04-02T05:07:37.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- start start 2024-04-02T05:07:37.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 librados: setting wanted keys 2024-04-02T05:07:37.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 librados: calling monclient init 2024-04-02T05:07:37.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c104460 0x7f0a8c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a8c104e20 0x7f0a8c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c10a4f0 0x7f0a8c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c113c80 con 0x7f0a8c104e20 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c113b00 con 0x7f0a8c10a4f0 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a90d4e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c113e00 con 0x7f0a8c104460 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c104460 0x7f0a8c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c104460 0x7f0a8c19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52824/0 (socket says 172.21.15.69:52824) 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 -- 172.21.15.69:0/3119373485 learned_addr learned my addr 172.21.15.69:0/3119373485 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8ad9d700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c10a4f0 0x7f0a8c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 -- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c10a4f0 msgr2=0x7f0a8c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a8c10a4f0 0x7f0a8c19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 -- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a8c104e20 msgr2=0x7f0a8c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a8c104e20 0x7f0a8c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.071+0000 7f0a8a59c700 1 -- 172.21.15.69:0/3119373485 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c19f200 con 0x7f0a8c104460 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a8a59c700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c104460 0x7f0a8c19a1f0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00e9a0 tx=0x7f0a7c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 10 client.?.objecter ms_handle_connect 0x7f0a8c104460 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c00cbd0 con 0x7f0a8c104460 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f0a7c00cd70 con 0x7f0a8c104460 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c19f4f0 con 0x7f0a8c104460 2024-04-02T05:07:37.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c018680 con 0x7f0a8c104460 2024-04-02T05:07:37.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 10 client.44075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:37.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1a6d70 con 0x7f0a8c104460 2024-04-02T05:07:37.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 1 librados: init done 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 10 client.44075.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 15 client.44075.objecter _session_command_op_assign -1 1 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 20 client.44075.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 10 client.44075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0a8c105ba0 con 0x7f0a8c104460 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f0a7c018820 con 0x7f0a8c104460 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0a7406f5e0 0x7f0a74071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f0a7c08e8d0 con 0x7f0a8c104460 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 10 client.44075.objecter ms_dispatch 0x7f0a8c001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 3 client.44075.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 3 client.44075.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a89d9b700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0a7406f5e0 0x7f0a74071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 10 client.44075.objecter checking command 1 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f0a7407bb80 0x7f0a7407e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 20 client.44075.objecter _get_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 15 client.44075.objecter _session_command_op_remove -1 1 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 20 client.44075.objecter _get_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 20 client.44075.objecter get_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 15 client.44075.objecter _session_command_op_assign 6 1 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 4 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 10 client.44075.objecter _send_command 1 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a7c01ee50 con 0x7f0a7407bb80 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 20 client.44075.objecter dump_active .. 0 homeless 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f0a7c0932a0 con 0x7f0a8c104460 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a8ad9d700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f0a7407bb80 0x7f0a7407e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a7b7fe700 10 client.44075.objecter ms_handle_connect 0x7f0a7406f5e0 2024-04-02T05:07:37.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.075+0000 7f0a89d9b700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0a7406f5e0 0x7f0a74071aa0 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f0a80005950 tx=0x7f0a800058e0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a8ad9d700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f0a7407bb80 0x7f0a7407e020 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a7b7fe700 10 client.44075.objecter ms_handle_connect 0x7f0a7407bb80 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== osd.6 v2:172.21.15.98:6816/1226172499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0a7c01ee50 con 0x7f0a7407bb80 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a7b7fe700 10 client.44075.objecter ms_dispatch 0x7f0a8c001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a7b7fe700 10 client.44075.objecter _finish_command 1 = system:0 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a7b7fe700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 4 2024-04-02T05:07:37.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.079+0000 7f0a7b7fe700 15 client.44075.objecter _session_command_op_remove 6 1 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 10 client.44075.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 15 client.44075.objecter _session_command_op_assign -1 2 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 20 client.44075.objecter _get_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 20 client.44075.objecter _get_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 15 client.44075.objecter _session_command_op_remove -1 2 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 20 client.44075.objecter get_session s=0x7f0a7407b4b0 osd=6 3 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 15 client.44075.objecter _session_command_op_assign 6 2 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 4 2024-04-02T05:07:37.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 10 client.44075.objecter _send_command 2 2024-04-02T05:07:37.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 --> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0a8c04ff00 con 0x7f0a7407bb80 2024-04-02T05:07:37.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a7b7fe700 1 -- 172.21.15.69:0/3119373485 <== osd.6 v2:172.21.15.98:6816/1226172499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0a8c04ff00 con 0x7f0a7407bb80 2024-04-02T05:07:37.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a7b7fe700 10 client.44075.objecter ms_dispatch 0x7f0a8c001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:37.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a7b7fe700 10 client.44075.objecter _finish_command 2 = system:0 2024-04-02T05:07:37.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a7b7fe700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 4 2024-04-02T05:07:37.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a7b7fe700 15 client.44075.objecter _session_command_op_remove 6 2 2024-04-02T05:07:37.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 10 librados: watch_flush enter 2024-04-02T05:07:37.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.095+0000 7f0a90d4e700 10 librados: watch_flush exit 2024-04-02T05:07:37.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 10 client.44075.objecter close_session for osd.6 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f0a7407bb80 msgr2=0x7f0a7407e020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.98:6816/1226172499,v1:172.21.15.98:6817/1226172499] conn(0x7f0a7407bb80 0x7f0a7407e020 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 20 client.44075.objecter put_session s=0x7f0a7407b4b0 osd=6 1 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 20 client.44075.objecter shutdown clearing up homeless session... 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 10 client.44075.objecter successfully canceled tick 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0a7406f5e0 msgr2=0x7f0a74071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f0a7406f5e0 0x7f0a74071aa0 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f0a80005950 tx=0x7f0a800058e0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c104460 msgr2=0x7f0a8c19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 --2- 172.21.15.69:0/3119373485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a8c104460 0x7f0a8c19a1f0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00e9a0 tx=0x7f0a7c00ed60 comp rx=0 tx=0).stop 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a8ad9d700 1 -- 172.21.15.69:0/3119373485 reap_dead start 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 shutdown_connections 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 >> 172.21.15.69:0/3119373485 conn(0x7f0a8c0fff50 msgr2=0x7f0a8c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 shutdown_connections 2024-04-02T05:07:37.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 -- 172.21.15.69:0/3119373485 wait complete. 2024-04-02T05:07:37.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.099+0000 7f0a90d4e700 1 librados: shutdown 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1327272112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3698101b60 msgr2=0x7f369810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- 172.21.15.69:0/1327272112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3698101b60 0x7f369810dfd0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f3690009f90 tx=0x7f369000b330 comp rx=0 tx=0).stop 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1327272112 shutdown_connections 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- 172.21.15.69:0/1327272112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3698101b60 0x7f369810dfd0 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- 172.21.15.69:0/1327272112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3698107220 0x7f3698107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- 172.21.15.69:0/1327272112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36981011a0 0x7f3698101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1327272112 >> 172.21.15.69:0/1327272112 conn(0x7f36980771e0 msgr2=0x7f36980775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1327272112 shutdown_connections 2024-04-02T05:07:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1327272112 wait complete. 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 librados: starting msgr at 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 librados: starting objecter 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 Processor -- start 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- start start 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 librados: setting wanted keys 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 librados: calling monclient init 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36981011a0 0x7f3698196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3698101b60 0x7f3698197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3698107220 0x7f36981975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3698110930 con 0x7f36981011a0 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36981107b0 con 0x7f3698101b60 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f3698110ab0 con 0x7f3698107220 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36981011a0 0x7f3698196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3697fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3698107220 0x7f36981975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36981011a0 0x7f3698196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46066/0 (socket says 172.21.15.69:46066) 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3697fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3698107220 0x7f36981975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:52828/0 (socket says 172.21.15.69:52828) 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 -- 172.21.15.69:0/1205653716 learned_addr learned my addr 172.21.15.69:0/1205653716 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3696ffd700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3698101b60 0x7f3698197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 -- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3698107220 msgr2=0x7f36981975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3698107220 0x7f36981975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 -- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3698101b60 msgr2=0x7f3698197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3698101b60 0x7f3698197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 -- 172.21.15.69:0/1205653716 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36981a63c0 con 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3696ffd700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3698101b60 0x7f3698197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f36977fe700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36981011a0 0x7f3698196b40 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f368800ec10 tx=0x7f368800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3694ff9700 10 client.?.objecter ms_handle_connect 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3694ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368800cd40 con 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f36880044a0 con 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688010430 con 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36981a66b0 con 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 10 client.34014.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36981a6b70 con 0x7f36981011a0 2024-04-02T05:07:37.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 librados: init done 2024-04-02T05:07:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 10 client.34014.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 15 client.34014.objecter _session_command_op_assign -1 1 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 20 client.34014.objecter _get_session osd=-1 returning homeless 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 10 client.34014.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.139+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3698102860 con 0x7f36981011a0 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f3688003fb0 con 0x7f36981011a0 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f368006f5e0 0x7f3680071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f368808ee50 con 0x7f36981011a0 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 10 client.34014.objecter ms_dispatch 0x7f3698001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 3 client.34014.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 3 client.34014.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3696ffd700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f368006f5e0 0x7f3680071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 10 client.34014.objecter checking command 1 2024-04-02T05:07:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f368007bb20 0x7f368007dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 20 client.34014.objecter _get_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 15 client.34014.objecter _session_command_op_remove -1 1 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 20 client.34014.objecter _get_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 20 client.34014.objecter get_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 15 client.34014.objecter _session_command_op_assign 4 1 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 4 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 10 client.34014.objecter _send_command 1 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3688016ec0 con 0x7f368007bb20 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 20 client.34014.objecter dump_active .. 0 homeless 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=966) ==== 24+0+0 (secure 0 0 0) 0x7f368808f2a0 con 0x7f36981011a0 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3696ffd700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f368006f5e0 0x7f3680071aa0 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7f368c005fd0 tx=0x7f368c005dc0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 10 client.34014.objecter ms_handle_connect 0x7f368006f5e0 2024-04-02T05:07:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3697fff700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f368007bb20 0x7f368007dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3694ff9700 10 client.34014.objecter ms_handle_connect 0x7f368007bb20 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.143+0000 7f3697fff700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f368007bb20 0x7f368007dfc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.147+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== osd.4 v2:172.21.15.98:6800/2065947829 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3688016ec0 con 0x7f368007bb20 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.147+0000 7f3694ff9700 10 client.34014.objecter ms_dispatch 0x7f3698001550 command_reply(tid 1: 0 ) 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.147+0000 7f3694ff9700 10 client.34014.objecter _finish_command 1 = system:0 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.147+0000 7f3694ff9700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 4 2024-04-02T05:07:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.147+0000 7f3694ff9700 15 client.34014.objecter _session_command_op_remove 4 1 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 10 client.34014.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 15 client.34014.objecter _session_command_op_assign -1 2 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter _get_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter _get_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 15 client.34014.objecter _session_command_op_remove -1 2 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter get_session s=0x7f368007b450 osd=4 3 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 15 client.34014.objecter _session_command_op_assign 4 2 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 4 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 10 client.34014.objecter _send_command 2 2024-04-02T05:07:37.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 --> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f369804ff00 con 0x7f368007bb20 2024-04-02T05:07:37.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f3694ff9700 1 -- 172.21.15.69:0/1205653716 <== osd.4 v2:172.21.15.98:6800/2065947829 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f369804ff00 con 0x7f368007bb20 2024-04-02T05:07:37.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f3694ff9700 10 client.34014.objecter ms_dispatch 0x7f3698001550 command_reply(tid 2: 0 ) 2024-04-02T05:07:37.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f3694ff9700 10 client.34014.objecter _finish_command 2 = system:0 2024-04-02T05:07:37.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f3694ff9700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 4 2024-04-02T05:07:37.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f3694ff9700 15 client.34014.objecter _session_command_op_remove 4 2 2024-04-02T05:07:37.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 10 librados: watch_flush enter 2024-04-02T05:07:37.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 10 librados: watch_flush exit 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 10 client.34014.objecter close_session for osd.4 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f368007bb20 msgr2=0x7f368007dfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.98:6800/2065947829,v1:172.21.15.98:6801/2065947829] conn(0x7f368007bb20 0x7f368007dfc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter put_session s=0x7f368007b450 osd=4 1 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 20 client.34014.objecter shutdown clearing up homeless session... 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 10 client.34014.objecter successfully canceled tick 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f368006f5e0 msgr2=0x7f3680071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f368006f5e0 0x7f3680071aa0 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7f368c005fd0 tx=0x7f368c005dc0 comp rx=0 tx=0).stop 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36981011a0 msgr2=0x7f3698196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f369dffb700 1 --2- 172.21.15.69:0/1205653716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36981011a0 0x7f3698196b40 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f368800ec10 tx=0x7f368800ef20 comp rx=0 tx=0).stop 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.163+0000 7f3697fff700 1 -- 172.21.15.69:0/1205653716 reap_dead start 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.167+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 shutdown_connections 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.167+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 >> 172.21.15.69:0/1205653716 conn(0x7f36980771e0 msgr2=0x7f3698107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:37.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.167+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 shutdown_connections 2024-04-02T05:07:37.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.167+0000 7f369dffb700 1 -- 172.21.15.69:0/1205653716 wait complete. 2024-04-02T05:07:37.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:37.167+0000 7f369dffb700 1 librados: shutdown 2024-04-02T05:07:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:36 smithi098 bash[23790]: audit 2024-04-02T05:07:36.484230+0000 mon.b (mon.1) 359 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:07:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:36 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.600376+0000 osd.4 (osd.4) 552 : cluster 0 10.10 scrub starts 2024-04-02T05:07:37.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:36 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.602400+0000 osd.4 (osd.4) 553 : cluster 0 10.10 scrub ok 2024-04-02T05:07:37.728 INFO:teuthology.orchestra.run.smithi069.stdout:55834575205 2024-04-02T05:07:37.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.080506+0000 osd.0 (osd.0) 543 : cluster 0 6.0 scrub starts 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.083140+0000 osd.0 (osd.0) 544 : cluster 0 6.0 scrub ok 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.501135+0000 mgr.y (mgr.14538) 912 : cluster 0 pgmap v1551: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.750680+0000 osd.6 (osd.6) 525 : cluster 0 1.f scrub starts 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:36.752810+0000 osd.6 (osd.6) 526 : cluster 0 1.f scrub ok 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: audit 2024-04-02T05:07:37.222780+0000 mon.b (mon.1) 360 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: audit 2024-04-02T05:07:37.224296+0000 mon.b (mon.1) 361 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: audit 2024-04-02T05:07:37.229781+0000 mon.a (mon.0) 4200 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:37.631060+0000 osd.3 (osd.3) 607 : cluster 0 4.0 scrub starts 2024-04-02T05:07:38.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:37.633362+0000 osd.3 (osd.3) 608 : cluster 0 4.0 scrub ok 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:37.634715+0000 osd.1 (osd.1) 589 : cluster 0 1.3 scrub starts 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[28904]: cluster 2024-04-02T05:07:37.636148+0000 osd.1 (osd.1) 590 : cluster 0 1.3 scrub ok 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.080506+0000 osd.0 (osd.0) 543 : cluster 0 6.0 scrub starts 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.083140+0000 osd.0 (osd.0) 544 : cluster 0 6.0 scrub ok 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.501135+0000 mgr.y (mgr.14538) 912 : cluster 0 pgmap v1551: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.750680+0000 osd.6 (osd.6) 525 : cluster 0 1.f scrub starts 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:36.752810+0000 osd.6 (osd.6) 526 : cluster 0 1.f scrub ok 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: audit 2024-04-02T05:07:37.222780+0000 mon.b (mon.1) 360 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: audit 2024-04-02T05:07:37.224296+0000 mon.b (mon.1) 361 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: audit 2024-04-02T05:07:37.229781+0000 mon.a (mon.0) 4200 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:37.631060+0000 osd.3 (osd.3) 607 : cluster 0 4.0 scrub starts 2024-04-02T05:07:38.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:37.633362+0000 osd.3 (osd.3) 608 : cluster 0 4.0 scrub ok 2024-04-02T05:07:38.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:37.634715+0000 osd.1 (osd.1) 589 : cluster 0 1.3 scrub starts 2024-04-02T05:07:38.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:37 smithi069 bash[21528]: cluster 2024-04-02T05:07:37.636148+0000 osd.1 (osd.1) 590 : cluster 0 1.3 scrub ok 2024-04-02T05:07:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.080506+0000 osd.0 (osd.0) 543 : cluster 0 6.0 scrub starts 2024-04-02T05:07:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.083140+0000 osd.0 (osd.0) 544 : cluster 0 6.0 scrub ok 2024-04-02T05:07:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.501135+0000 mgr.y (mgr.14538) 912 : cluster 0 pgmap v1551: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.750680+0000 osd.6 (osd.6) 525 : cluster 0 1.f scrub starts 2024-04-02T05:07:38.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:36.752810+0000 osd.6 (osd.6) 526 : cluster 0 1.f scrub ok 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: audit 2024-04-02T05:07:37.222780+0000 mon.b (mon.1) 360 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: audit 2024-04-02T05:07:37.224296+0000 mon.b (mon.1) 361 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: audit 2024-04-02T05:07:37.229781+0000 mon.a (mon.0) 4200 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:37.631060+0000 osd.3 (osd.3) 607 : cluster 0 4.0 scrub starts 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:37.633362+0000 osd.3 (osd.3) 608 : cluster 0 4.0 scrub ok 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:37.634715+0000 osd.1 (osd.1) 589 : cluster 0 1.3 scrub starts 2024-04-02T05:07:38.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:37 smithi098 bash[23790]: cluster 2024-04-02T05:07:37.636148+0000 osd.1 (osd.1) 590 : cluster 0 1.3 scrub ok 2024-04-02T05:07:39.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:07:38 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:07:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:07:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:38 smithi069 bash[21528]: cluster 2024-04-02T05:07:38.648037+0000 osd.1 (osd.1) 591 : cluster 0 6.4 scrub starts 2024-04-02T05:07:39.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:38 smithi069 bash[21528]: cluster 2024-04-02T05:07:38.650541+0000 osd.1 (osd.1) 592 : cluster 0 6.4 scrub ok 2024-04-02T05:07:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:38 smithi069 bash[28904]: cluster 2024-04-02T05:07:38.648037+0000 osd.1 (osd.1) 591 : cluster 0 6.4 scrub starts 2024-04-02T05:07:39.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:38 smithi069 bash[28904]: cluster 2024-04-02T05:07:38.650541+0000 osd.1 (osd.1) 592 : cluster 0 6.4 scrub ok 2024-04-02T05:07:39.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:38 smithi098 bash[23790]: cluster 2024-04-02T05:07:38.648037+0000 osd.1 (osd.1) 591 : cluster 0 6.4 scrub starts 2024-04-02T05:07:39.224 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:38 smithi098 bash[23790]: cluster 2024-04-02T05:07:38.650541+0000 osd.1 (osd.1) 592 : cluster 0 6.4 scrub ok 2024-04-02T05:07:39.747 INFO:teuthology.orchestra.run.smithi069.stdout:201863463227 2024-04-02T05:07:39.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-02T05:07:39.751 INFO:teuthology.orchestra.run.smithi069.stdout:34359738735 2024-04-02T05:07:39.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-02T05:07:39.868 INFO:teuthology.orchestra.run.smithi069.stdout:1692217114791 2024-04-02T05:07:39.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-02T05:07:40.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:39 smithi069 bash[21528]: cluster 2024-04-02T05:07:38.503158+0000 mgr.y (mgr.14538) 913 : cluster 0 pgmap v1552: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T05:07:40.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:39 smithi069 bash[28904]: cluster 2024-04-02T05:07:38.503158+0000 mgr.y (mgr.14538) 913 : cluster 0 pgmap v1552: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T05:07:40.223 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:39 smithi098 bash[23790]: cluster 2024-04-02T05:07:38.503158+0000 mgr.y (mgr.14538) 913 : cluster 0 pgmap v1552: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-02T05:07:40.575 INFO:teuthology.orchestra.run.smithi069.stdout:154618822983 2024-04-02T05:07:40.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-02T05:07:40.757 INFO:teuthology.orchestra.run.smithi069.stdout:249108103474 2024-04-02T05:07:40.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-02T05:07:41.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:41 smithi069 bash[28904]: cluster 2024-04-02T05:07:40.504270+0000 mgr.y (mgr.14538) 914 : cluster 0 pgmap v1553: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:41.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:41 smithi069 bash[21528]: cluster 2024-04-02T05:07:40.504270+0000 mgr.y (mgr.14538) 914 : cluster 0 pgmap v1553: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:41.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:41 smithi098 bash[23790]: cluster 2024-04-02T05:07:40.504270+0000 mgr.y (mgr.14538) 914 : cluster 0 pgmap v1553: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:42.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:40.510965+0000 osd.7 (osd.7) 383 : cluster 0 2.b scrub starts 2024-04-02T05:07:42.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:40.513343+0000 osd.7 (osd.7) 384 : cluster 0 2.b scrub ok 2024-04-02T05:07:42.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:41.039034+0000 osd.0 (osd.0) 545 : cluster 0 3.12 scrub starts 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:41.043513+0000 osd.0 (osd.0) 546 : cluster 0 3.12 scrub ok 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:41.616203+0000 osd.3 (osd.3) 609 : cluster 0 6.b scrub starts 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:41.618618+0000 osd.3 (osd.3) 610 : cluster 0 6.b scrub ok 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:41.639469+0000 osd.4 (osd.4) 554 : cluster 0 10.1 scrub starts 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[21528]: cluster 2024-04-02T05:07:41.643866+0000 osd.4 (osd.4) 555 : cluster 0 10.1 scrub ok 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:40.510965+0000 osd.7 (osd.7) 383 : cluster 0 2.b scrub starts 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:40.513343+0000 osd.7 (osd.7) 384 : cluster 0 2.b scrub ok 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:41.039034+0000 osd.0 (osd.0) 545 : cluster 0 3.12 scrub starts 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:41.043513+0000 osd.0 (osd.0) 546 : cluster 0 3.12 scrub ok 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:41.616203+0000 osd.3 (osd.3) 609 : cluster 0 6.b scrub starts 2024-04-02T05:07:42.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:41.618618+0000 osd.3 (osd.3) 610 : cluster 0 6.b scrub ok 2024-04-02T05:07:42.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:41.639469+0000 osd.4 (osd.4) 554 : cluster 0 10.1 scrub starts 2024-04-02T05:07:42.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:42 smithi069 bash[28904]: cluster 2024-04-02T05:07:41.643866+0000 osd.4 (osd.4) 555 : cluster 0 10.1 scrub ok 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:40.510965+0000 osd.7 (osd.7) 383 : cluster 0 2.b scrub starts 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:40.513343+0000 osd.7 (osd.7) 384 : cluster 0 2.b scrub ok 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:41.039034+0000 osd.0 (osd.0) 545 : cluster 0 3.12 scrub starts 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:41.043513+0000 osd.0 (osd.0) 546 : cluster 0 3.12 scrub ok 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:41.616203+0000 osd.3 (osd.3) 609 : cluster 0 6.b scrub starts 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:41.618618+0000 osd.3 (osd.3) 610 : cluster 0 6.b scrub ok 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:41.639469+0000 osd.4 (osd.4) 554 : cluster 0 10.1 scrub starts 2024-04-02T05:07:42.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:42 smithi098 bash[23790]: cluster 2024-04-02T05:07:41.643866+0000 osd.4 (osd.4) 555 : cluster 0 10.1 scrub ok 2024-04-02T05:07:43.283 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.176008+0000 osd.5 (osd.5) 403 : cluster 0 6.19 scrub starts 2024-04-02T05:07:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.178539+0000 osd.5 (osd.5) 404 : cluster 0 6.19 scrub ok 2024-04-02T05:07:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.505209+0000 mgr.y (mgr.14538) 915 : cluster 0 pgmap v1554: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.607226+0000 osd.4 (osd.4) 556 : cluster 0 3.1a scrub starts 2024-04-02T05:07:43.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.611826+0000 osd.4 (osd.4) 557 : cluster 0 3.1a scrub ok 2024-04-02T05:07:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.176008+0000 osd.5 (osd.5) 403 : cluster 0 6.19 scrub starts 2024-04-02T05:07:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.178539+0000 osd.5 (osd.5) 404 : cluster 0 6.19 scrub ok 2024-04-02T05:07:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.505209+0000 mgr.y (mgr.14538) 915 : cluster 0 pgmap v1554: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.607226+0000 osd.4 (osd.4) 556 : cluster 0 3.1a scrub starts 2024-04-02T05:07:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:43 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.611826+0000 osd.4 (osd.4) 557 : cluster 0 3.1a scrub ok 2024-04-02T05:07:43.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:43 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.176008+0000 osd.5 (osd.5) 403 : cluster 0 6.19 scrub starts 2024-04-02T05:07:43.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:43 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.178539+0000 osd.5 (osd.5) 404 : cluster 0 6.19 scrub ok 2024-04-02T05:07:43.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:43 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.505209+0000 mgr.y (mgr.14538) 915 : cluster 0 pgmap v1554: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:43.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:43 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.607226+0000 osd.4 (osd.4) 556 : cluster 0 3.1a scrub starts 2024-04-02T05:07:43.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:43 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.611826+0000 osd.4 (osd.4) 557 : cluster 0 3.1a scrub ok 2024-04-02T05:07:44.265 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:07:43 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:07:44.300 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:44.301 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:44.301 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:44.301 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:44.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.449367+0000 osd.7 (osd.7) 385 : cluster 0 3.d scrub starts 2024-04-02T05:07:44.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.454626+0000 osd.7 (osd.7) 386 : cluster 0 3.d scrub ok 2024-04-02T05:07:44.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.686661+0000 osd.6 (osd.6) 527 : cluster 0 2.17 scrub starts 2024-04-02T05:07:44.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[21528]: cluster 2024-04-02T05:07:42.688953+0000 osd.6 (osd.6) 528 : cluster 0 2.17 scrub ok 2024-04-02T05:07:44.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[21528]: cluster 2024-04-02T05:07:43.110665+0000 osd.0 (osd.0) 547 : cluster 0 2.1f scrub starts 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[21528]: cluster 2024-04-02T05:07:43.113166+0000 osd.0 (osd.0) 548 : cluster 0 2.1f scrub ok 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.449367+0000 osd.7 (osd.7) 385 : cluster 0 3.d scrub starts 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.454626+0000 osd.7 (osd.7) 386 : cluster 0 3.d scrub ok 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.686661+0000 osd.6 (osd.6) 527 : cluster 0 2.17 scrub starts 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[28904]: cluster 2024-04-02T05:07:42.688953+0000 osd.6 (osd.6) 528 : cluster 0 2.17 scrub ok 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[28904]: cluster 2024-04-02T05:07:43.110665+0000 osd.0 (osd.0) 547 : cluster 0 2.1f scrub starts 2024-04-02T05:07:44.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:44 smithi069 bash[28904]: cluster 2024-04-02T05:07:43.113166+0000 osd.0 (osd.0) 548 : cluster 0 2.1f scrub ok 2024-04-02T05:07:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:44 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.449367+0000 osd.7 (osd.7) 385 : cluster 0 3.d scrub starts 2024-04-02T05:07:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:44 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.454626+0000 osd.7 (osd.7) 386 : cluster 0 3.d scrub ok 2024-04-02T05:07:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:44 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.686661+0000 osd.6 (osd.6) 527 : cluster 0 2.17 scrub starts 2024-04-02T05:07:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:44 smithi098 bash[23790]: cluster 2024-04-02T05:07:42.688953+0000 osd.6 (osd.6) 528 : cluster 0 2.17 scrub ok 2024-04-02T05:07:44.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:44 smithi098 bash[23790]: cluster 2024-04-02T05:07:43.110665+0000 osd.0 (osd.0) 547 : cluster 0 2.1f scrub starts 2024-04-02T05:07:44.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:44 smithi098 bash[23790]: cluster 2024-04-02T05:07:43.113166+0000 osd.0 (osd.0) 548 : cluster 0 2.1f scrub ok 2024-04-02T05:07:45.323 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:45.324 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:07:45.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[21528]: audit 2024-04-02T05:07:43.990705+0000 mgr.y (mgr.14538) 916 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:45.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[21528]: cluster 2024-04-02T05:07:44.140130+0000 osd.0 (osd.0) 549 : cluster 0 4.1e deep-scrub starts 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[21528]: cluster 2024-04-02T05:07:44.142478+0000 osd.0 (osd.0) 550 : cluster 0 4.1e deep-scrub ok 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[21528]: cluster 2024-04-02T05:07:44.507386+0000 mgr.y (mgr.14538) 917 : cluster 0 pgmap v1555: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[21528]: cluster 2024-04-02T05:07:44.606865+0000 osd.3 (osd.3) 611 : cluster 0 6.1f scrub starts 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[21528]: cluster 2024-04-02T05:07:44.609283+0000 osd.3 (osd.3) 612 : cluster 0 6.1f scrub ok 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[28904]: audit 2024-04-02T05:07:43.990705+0000 mgr.y (mgr.14538) 916 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[28904]: cluster 2024-04-02T05:07:44.140130+0000 osd.0 (osd.0) 549 : cluster 0 4.1e deep-scrub starts 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[28904]: cluster 2024-04-02T05:07:44.142478+0000 osd.0 (osd.0) 550 : cluster 0 4.1e deep-scrub ok 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[28904]: cluster 2024-04-02T05:07:44.507386+0000 mgr.y (mgr.14538) 917 : cluster 0 pgmap v1555: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[28904]: cluster 2024-04-02T05:07:44.606865+0000 osd.3 (osd.3) 611 : cluster 0 6.1f scrub starts 2024-04-02T05:07:45.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:45 smithi069 bash[28904]: cluster 2024-04-02T05:07:44.609283+0000 osd.3 (osd.3) 612 : cluster 0 6.1f scrub ok 2024-04-02T05:07:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:45 smithi098 bash[23790]: audit 2024-04-02T05:07:43.990705+0000 mgr.y (mgr.14538) 916 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:45 smithi098 bash[23790]: cluster 2024-04-02T05:07:44.140130+0000 osd.0 (osd.0) 549 : cluster 0 4.1e deep-scrub starts 2024-04-02T05:07:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:45 smithi098 bash[23790]: cluster 2024-04-02T05:07:44.142478+0000 osd.0 (osd.0) 550 : cluster 0 4.1e deep-scrub ok 2024-04-02T05:07:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:45 smithi098 bash[23790]: cluster 2024-04-02T05:07:44.507386+0000 mgr.y (mgr.14538) 917 : cluster 0 pgmap v1555: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:45 smithi098 bash[23790]: cluster 2024-04-02T05:07:44.606865+0000 osd.3 (osd.3) 611 : cluster 0 6.1f scrub starts 2024-04-02T05:07:45.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:45 smithi098 bash[23790]: cluster 2024-04-02T05:07:44.609283+0000 osd.3 (osd.3) 612 : cluster 0 6.1f scrub ok 2024-04-02T05:07:46.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[28904]: cluster 2024-04-02T05:07:45.113816+0000 osd.0 (osd.0) 551 : cluster 0 6.16 scrub starts 2024-04-02T05:07:46.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[28904]: cluster 2024-04-02T05:07:45.116023+0000 osd.0 (osd.0) 552 : cluster 0 6.16 scrub ok 2024-04-02T05:07:46.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[28904]: cluster 2024-04-02T05:07:45.116402+0000 osd.5 (osd.5) 405 : cluster 0 6.a scrub starts 2024-04-02T05:07:46.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[28904]: cluster 2024-04-02T05:07:45.118814+0000 osd.5 (osd.5) 406 : cluster 0 6.a scrub ok 2024-04-02T05:07:46.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[21528]: cluster 2024-04-02T05:07:45.113816+0000 osd.0 (osd.0) 551 : cluster 0 6.16 scrub starts 2024-04-02T05:07:46.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[21528]: cluster 2024-04-02T05:07:45.116023+0000 osd.0 (osd.0) 552 : cluster 0 6.16 scrub ok 2024-04-02T05:07:46.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[21528]: cluster 2024-04-02T05:07:45.116402+0000 osd.5 (osd.5) 405 : cluster 0 6.a scrub starts 2024-04-02T05:07:46.535 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:46 smithi069 bash[21528]: cluster 2024-04-02T05:07:45.118814+0000 osd.5 (osd.5) 406 : cluster 0 6.a scrub ok 2024-04-02T05:07:46.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:46 smithi098 bash[23790]: cluster 2024-04-02T05:07:45.113816+0000 osd.0 (osd.0) 551 : cluster 0 6.16 scrub starts 2024-04-02T05:07:46.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:46 smithi098 bash[23790]: cluster 2024-04-02T05:07:45.116023+0000 osd.0 (osd.0) 552 : cluster 0 6.16 scrub ok 2024-04-02T05:07:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:46 smithi098 bash[23790]: cluster 2024-04-02T05:07:45.116402+0000 osd.5 (osd.5) 405 : cluster 0 6.a scrub starts 2024-04-02T05:07:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:46 smithi098 bash[23790]: cluster 2024-04-02T05:07:45.118814+0000 osd.5 (osd.5) 406 : cluster 0 6.a scrub ok 2024-04-02T05:07:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:47 smithi098 bash[23790]: cluster 2024-04-02T05:07:45.438897+0000 osd.7 (osd.7) 387 : cluster 0 3.f scrub starts 2024-04-02T05:07:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:47 smithi098 bash[23790]: cluster 2024-04-02T05:07:45.443304+0000 osd.7 (osd.7) 388 : cluster 0 3.f scrub ok 2024-04-02T05:07:47.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:47 smithi098 bash[23790]: cluster 2024-04-02T05:07:46.508353+0000 mgr.y (mgr.14538) 918 : cluster 0 pgmap v1556: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-02T05:07:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:47 smithi098 bash[23790]: cluster 2024-04-02T05:07:46.626914+0000 osd.1 (osd.1) 593 : cluster 0 4.2 deep-scrub starts 2024-04-02T05:07:47.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:47 smithi098 bash[23790]: cluster 2024-04-02T05:07:46.628535+0000 osd.1 (osd.1) 594 : cluster 0 4.2 deep-scrub ok 2024-04-02T05:07:47.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[21528]: cluster 2024-04-02T05:07:45.438897+0000 osd.7 (osd.7) 387 : cluster 0 3.f scrub starts 2024-04-02T05:07:47.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[21528]: cluster 2024-04-02T05:07:45.443304+0000 osd.7 (osd.7) 388 : cluster 0 3.f scrub ok 2024-04-02T05:07:47.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[21528]: cluster 2024-04-02T05:07:46.508353+0000 mgr.y (mgr.14538) 918 : cluster 0 pgmap v1556: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-02T05:07:47.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[21528]: cluster 2024-04-02T05:07:46.626914+0000 osd.1 (osd.1) 593 : cluster 0 4.2 deep-scrub starts 2024-04-02T05:07:47.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[21528]: cluster 2024-04-02T05:07:46.628535+0000 osd.1 (osd.1) 594 : cluster 0 4.2 deep-scrub ok 2024-04-02T05:07:47.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[28904]: cluster 2024-04-02T05:07:45.438897+0000 osd.7 (osd.7) 387 : cluster 0 3.f scrub starts 2024-04-02T05:07:47.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[28904]: cluster 2024-04-02T05:07:45.443304+0000 osd.7 (osd.7) 388 : cluster 0 3.f scrub ok 2024-04-02T05:07:47.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[28904]: cluster 2024-04-02T05:07:46.508353+0000 mgr.y (mgr.14538) 918 : cluster 0 pgmap v1556: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-02T05:07:47.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[28904]: cluster 2024-04-02T05:07:46.626914+0000 osd.1 (osd.1) 593 : cluster 0 4.2 deep-scrub starts 2024-04-02T05:07:47.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:47 smithi069 bash[28904]: cluster 2024-04-02T05:07:46.628535+0000 osd.1 (osd.1) 594 : cluster 0 4.2 deep-scrub ok 2024-04-02T05:07:48.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[28904]: cluster 2024-04-02T05:07:46.486686+0000 osd.7 (osd.7) 389 : cluster 0 3.e scrub starts 2024-04-02T05:07:48.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[28904]: cluster 2024-04-02T05:07:46.491306+0000 osd.7 (osd.7) 390 : cluster 0 3.e scrub ok 2024-04-02T05:07:48.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[28904]: cluster 2024-04-02T05:07:47.109521+0000 osd.5 (osd.5) 407 : cluster 0 1.1 deep-scrub starts 2024-04-02T05:07:48.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[28904]: cluster 2024-04-02T05:07:47.112001+0000 osd.5 (osd.5) 408 : cluster 0 1.1 deep-scrub ok 2024-04-02T05:07:48.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[21528]: cluster 2024-04-02T05:07:46.486686+0000 osd.7 (osd.7) 389 : cluster 0 3.e scrub starts 2024-04-02T05:07:48.524 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[21528]: cluster 2024-04-02T05:07:46.491306+0000 osd.7 (osd.7) 390 : cluster 0 3.e scrub ok 2024-04-02T05:07:48.525 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[21528]: cluster 2024-04-02T05:07:47.109521+0000 osd.5 (osd.5) 407 : cluster 0 1.1 deep-scrub starts 2024-04-02T05:07:48.525 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[21528]: cluster 2024-04-02T05:07:47.112001+0000 osd.5 (osd.5) 408 : cluster 0 1.1 deep-scrub ok 2024-04-02T05:07:48.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:48 smithi098 bash[23790]: cluster 2024-04-02T05:07:46.486686+0000 osd.7 (osd.7) 389 : cluster 0 3.e scrub starts 2024-04-02T05:07:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:48 smithi098 bash[23790]: cluster 2024-04-02T05:07:46.491306+0000 osd.7 (osd.7) 390 : cluster 0 3.e scrub ok 2024-04-02T05:07:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:48 smithi098 bash[23790]: cluster 2024-04-02T05:07:47.109521+0000 osd.5 (osd.5) 407 : cluster 0 1.1 deep-scrub starts 2024-04-02T05:07:48.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:48 smithi098 bash[23790]: cluster 2024-04-02T05:07:47.112001+0000 osd.5 (osd.5) 408 : cluster 0 1.1 deep-scrub ok 2024-04-02T05:07:48.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:07:48 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:07:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:07:49.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/580893075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 msgr2=0x7f1fe410e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:49.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/580893075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe410e790 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009990 tx=0x7f1fd4009ca0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/580893075 shutdown_connections 2024-04-02T05:07:49.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/580893075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fe410ecd0 0x7f1fe41110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/580893075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe4074470 0x7f1fe4074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/580893075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe410e790 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/580893075 >> 172.21.15.69:0/580893075 conn(0x7f1fe4079310 msgr2=0x7f1fe4079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:49.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/580893075 shutdown_connections 2024-04-02T05:07:49.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/580893075 wait complete. 2024-04-02T05:07:49.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 librados: starting msgr at 2024-04-02T05:07:49.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 librados: starting objecter 2024-04-02T05:07:49.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 Processor -- start 2024-04-02T05:07:49.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- start start 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 librados: setting wanted keys 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 librados: calling monclient init 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fe4074470 0x7f1fe419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe410ecd0 0x7f1fe419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fe4074470 0x7f1fe419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:49.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4113c60 con 0x7f1fe4074470 2024-04-02T05:07:49.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4113ae0 con 0x7f1fe410ecd0 2024-04-02T05:07:49.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe4113de0 con 0x7f1fe4074d90 2024-04-02T05:07:49.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:38388/0 (socket says 172.21.15.69:38388) 2024-04-02T05:07:49.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 -- 172.21.15.69:0/2782689351 learned_addr learned my addr 172.21.15.69:0/2782689351 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:49.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 -- 172.21.15.69:0/2782689351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe410ecd0 msgr2=0x7f1fe419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe410ecd0 0x7f1fe419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 -- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fe4074470 msgr2=0x7f1fe419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fe4074470 0x7f1fe419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 -- 172.21.15.69:0/2782689351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe419f2d0 con 0x7f1fe4074d90 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe1d9b700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe419a800 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4000c00 tx=0x7f1fd40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fd37fe700 10 client.?.objecter ms_handle_connect 0x7f1fe4074d90 2024-04-02T05:07:49.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fd37fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4027070 con 0x7f1fe4074d90 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f1fd4022470 con 0x7f1fe4074d90 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4017650 con 0x7f1fe4074d90 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe419f560 con 0x7f1fe4074d90 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 10 client.44087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe41a6e40 con 0x7f1fe4074d90 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.483+0000 7f1fe8fdf700 1 librados: init done 2024-04-02T05:07:49.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe4075b00 con 0x7f1fe4074d90 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f1fd402a070 con 0x7f1fe4074d90 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fcc06f5e0 0x7f1fcc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fe259c700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fcc06f5e0 0x7f1fcc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f1fd408f520 con 0x7f1fe4074d90 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 10 client.44087.objecter ms_dispatch 0x7f1fe4001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 3 client.44087.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 3 client.44087.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 20 client.44087.objecter dump_active .. 0 homeless 2024-04-02T05:07:49.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fe259c700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fcc06f5e0 0x7f1fcc071aa0 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8005950 tx=0x7f1fd800a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:49.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.487+0000 7f1fd37fe700 10 client.44087.objecter ms_handle_connect 0x7f1fcc06f5e0 2024-04-02T05:07:49.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.491+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1fd4013070 con 0x7f1fe4074d90 2024-04-02T05:07:49.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.699+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1fe419b620 con 0x7f1fe4074d90 2024-04-02T05:07:49.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fd37fe700 1 -- 172.21.15.69:0/2782689351 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1fd405c4f0 con 0x7f1fe4074d90 2024-04-02T05:07:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:81604378974 2024-04-02T05:07:49.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 10 librados: watch_flush enter 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 10 librados: watch_flush exit 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 20 client.44087.objecter shutdown clearing up homeless session... 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 10 client.44087.objecter successfully canceled tick 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fcc06f5e0 msgr2=0x7f1fcc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fcc06f5e0 0x7f1fcc071aa0 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8005950 tx=0x7f1fd800a300 comp rx=0 tx=0).stop 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 msgr2=0x7f1fe419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe419a800 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4000c00 tx=0x7f1fd40047b0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 shutdown_connections 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f1fcc06f5e0 0x7f1fcc071aa0 unknown :-1 s=CLOSED pgs=4060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe410ecd0 0x7f1fe419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fe4074470 0x7f1fe419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 --2- 172.21.15.69:0/2782689351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fe4074d90 0x7f1fe419a800 unknown :-1 s=CLOSED pgs=3344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:49.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 >> 172.21.15.69:0/2782689351 conn(0x7f1fe4079310 msgr2=0x7f1fe4106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:49.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 shutdown_connections 2024-04-02T05:07:49.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 -- 172.21.15.69:0/2782689351 wait complete. 2024-04-02T05:07:49.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:49.703+0000 7f1fe8fdf700 1 librados: shutdown 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:47.782593+0000 osd.6 (osd.6) 529 : cluster 0 10.b deep-scrub starts 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:47.789907+0000 osd.6 (osd.6) 530 : cluster 0 10.b deep-scrub ok 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.115821+0000 osd.5 (osd.5) 409 : cluster 0 2.2 scrub starts 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.117988+0000 osd.5 (osd.5) 410 : cluster 0 2.2 scrub ok 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.509620+0000 mgr.y (mgr.14538) 919 : cluster 0 pgmap v1557: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.659513+0000 osd.1 (osd.1) 595 : cluster 0 2.15 scrub starts 2024-04-02T05:07:49.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.661010+0000 osd.1 (osd.1) 596 : cluster 0 2.15 scrub ok 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.699791+0000 osd.4 (osd.4) 558 : cluster 0 6.e scrub starts 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.701876+0000 osd.4 (osd.4) 559 : cluster 0 6.e scrub ok 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:47.782593+0000 osd.6 (osd.6) 529 : cluster 0 10.b deep-scrub starts 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:47.789907+0000 osd.6 (osd.6) 530 : cluster 0 10.b deep-scrub ok 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.115821+0000 osd.5 (osd.5) 409 : cluster 0 2.2 scrub starts 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.117988+0000 osd.5 (osd.5) 410 : cluster 0 2.2 scrub ok 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.509620+0000 mgr.y (mgr.14538) 919 : cluster 0 pgmap v1557: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.659513+0000 osd.1 (osd.1) 595 : cluster 0 2.15 scrub starts 2024-04-02T05:07:49.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.661010+0000 osd.1 (osd.1) 596 : cluster 0 2.15 scrub ok 2024-04-02T05:07:49.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.699791+0000 osd.4 (osd.4) 558 : cluster 0 6.e scrub starts 2024-04-02T05:07:49.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:49 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.701876+0000 osd.4 (osd.4) 559 : cluster 0 6.e scrub ok 2024-04-02T05:07:49.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:47.782593+0000 osd.6 (osd.6) 529 : cluster 0 10.b deep-scrub starts 2024-04-02T05:07:49.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:47.789907+0000 osd.6 (osd.6) 530 : cluster 0 10.b deep-scrub ok 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.115821+0000 osd.5 (osd.5) 409 : cluster 0 2.2 scrub starts 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.117988+0000 osd.5 (osd.5) 410 : cluster 0 2.2 scrub ok 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.509620+0000 mgr.y (mgr.14538) 919 : cluster 0 pgmap v1557: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.659513+0000 osd.1 (osd.1) 595 : cluster 0 2.15 scrub starts 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.661010+0000 osd.1 (osd.1) 596 : cluster 0 2.15 scrub ok 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.699791+0000 osd.4 (osd.4) 558 : cluster 0 6.e scrub starts 2024-04-02T05:07:49.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:49 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.701876+0000 osd.4 (osd.4) 559 : cluster 0 6.e scrub ok 2024-04-02T05:07:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.502783+0000 osd.7 (osd.7) 391 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:07:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.507895+0000 osd.7 (osd.7) 392 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:07:50.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.820311+0000 osd.6 (osd.6) 531 : cluster 0 1.7 scrub starts 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:48.823121+0000 osd.6 (osd.6) 532 : cluster 0 1.7 scrub ok 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:49.589498+0000 osd.3 (osd.3) 613 : cluster 0 6.11 scrub starts 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:49.591582+0000 osd.3 (osd.3) 614 : cluster 0 6.11 scrub ok 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: audit 2024-04-02T05:07:49.704220+0000 mon.c (mon.2) 1178 : audit 0 from='client.? 172.21.15.69:0/2782689351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:49.733770+0000 osd.4 (osd.4) 560 : cluster 0 4.1 deep-scrub starts 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: cluster 2024-04-02T05:07:49.736180+0000 osd.4 (osd.4) 561 : cluster 0 4.1 deep-scrub ok 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[21528]: audit 2024-04-02T05:07:49.771545+0000 mon.b (mon.1) 362 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.502783+0000 osd.7 (osd.7) 391 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.507895+0000 osd.7 (osd.7) 392 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.820311+0000 osd.6 (osd.6) 531 : cluster 0 1.7 scrub starts 2024-04-02T05:07:50.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:48.823121+0000 osd.6 (osd.6) 532 : cluster 0 1.7 scrub ok 2024-04-02T05:07:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:49.589498+0000 osd.3 (osd.3) 613 : cluster 0 6.11 scrub starts 2024-04-02T05:07:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:49.591582+0000 osd.3 (osd.3) 614 : cluster 0 6.11 scrub ok 2024-04-02T05:07:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: audit 2024-04-02T05:07:49.704220+0000 mon.c (mon.2) 1178 : audit 0 from='client.? 172.21.15.69:0/2782689351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T05:07:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:49.733770+0000 osd.4 (osd.4) 560 : cluster 0 4.1 deep-scrub starts 2024-04-02T05:07:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: cluster 2024-04-02T05:07:49.736180+0000 osd.4 (osd.4) 561 : cluster 0 4.1 deep-scrub ok 2024-04-02T05:07:50.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:50 smithi069 bash[28904]: audit 2024-04-02T05:07:49.771545+0000 mon.b (mon.1) 362 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.502783+0000 osd.7 (osd.7) 391 : cluster 0 3.11 deep-scrub starts 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.507895+0000 osd.7 (osd.7) 392 : cluster 0 3.11 deep-scrub ok 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.820311+0000 osd.6 (osd.6) 531 : cluster 0 1.7 scrub starts 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:48.823121+0000 osd.6 (osd.6) 532 : cluster 0 1.7 scrub ok 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:49.589498+0000 osd.3 (osd.3) 613 : cluster 0 6.11 scrub starts 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:49.591582+0000 osd.3 (osd.3) 614 : cluster 0 6.11 scrub ok 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: audit 2024-04-02T05:07:49.704220+0000 mon.c (mon.2) 1178 : audit 0 from='client.? 172.21.15.69:0/2782689351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:49.733770+0000 osd.4 (osd.4) 560 : cluster 0 4.1 deep-scrub starts 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: cluster 2024-04-02T05:07:49.736180+0000 osd.4 (osd.4) 561 : cluster 0 4.1 deep-scrub ok 2024-04-02T05:07:50.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:50 smithi098 bash[23790]: audit 2024-04-02T05:07:49.771545+0000 mon.b (mon.1) 362 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:07:51.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:51 smithi098 bash[23790]: cluster 2024-04-02T05:07:50.510575+0000 mgr.y (mgr.14538) 920 : cluster 0 pgmap v1558: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:51.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:51 smithi098 bash[23790]: cluster 2024-04-02T05:07:50.689807+0000 osd.4 (osd.4) 562 : cluster 0 10.17 scrub starts 2024-04-02T05:07:51.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:51 smithi098 bash[23790]: cluster 2024-04-02T05:07:50.697595+0000 osd.4 (osd.4) 563 : cluster 0 10.17 scrub ok 2024-04-02T05:07:51.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:51 smithi069 bash[21528]: cluster 2024-04-02T05:07:50.510575+0000 mgr.y (mgr.14538) 920 : cluster 0 pgmap v1558: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:51.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:51 smithi069 bash[21528]: cluster 2024-04-02T05:07:50.689807+0000 osd.4 (osd.4) 562 : cluster 0 10.17 scrub starts 2024-04-02T05:07:51.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:51 smithi069 bash[21528]: cluster 2024-04-02T05:07:50.697595+0000 osd.4 (osd.4) 563 : cluster 0 10.17 scrub ok 2024-04-02T05:07:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:51 smithi069 bash[28904]: cluster 2024-04-02T05:07:50.510575+0000 mgr.y (mgr.14538) 920 : cluster 0 pgmap v1558: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:51 smithi069 bash[28904]: cluster 2024-04-02T05:07:50.689807+0000 osd.4 (osd.4) 562 : cluster 0 10.17 scrub starts 2024-04-02T05:07:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:51 smithi069 bash[28904]: cluster 2024-04-02T05:07:50.697595+0000 osd.4 (osd.4) 563 : cluster 0 10.17 scrub ok 2024-04-02T05:07:52.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[21528]: cluster 2024-04-02T05:07:50.832276+0000 osd.6 (osd.6) 533 : cluster 0 1.6 scrub starts 2024-04-02T05:07:52.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[21528]: cluster 2024-04-02T05:07:50.834595+0000 osd.6 (osd.6) 534 : cluster 0 1.6 scrub ok 2024-04-02T05:07:52.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:52 smithi098 bash[23790]: cluster 2024-04-02T05:07:50.832276+0000 osd.6 (osd.6) 533 : cluster 0 1.6 scrub starts 2024-04-02T05:07:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:52 smithi098 bash[23790]: cluster 2024-04-02T05:07:50.834595+0000 osd.6 (osd.6) 534 : cluster 0 1.6 scrub ok 2024-04-02T05:07:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:52 smithi098 bash[23790]: cluster 2024-04-02T05:07:51.509959+0000 osd.3 (osd.3) 615 : cluster 0 2.1e scrub starts 2024-04-02T05:07:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:52 smithi098 bash[23790]: cluster 2024-04-02T05:07:51.512213+0000 osd.3 (osd.3) 616 : cluster 0 2.1e scrub ok 2024-04-02T05:07:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:52 smithi098 bash[23790]: cluster 2024-04-02T05:07:51.694764+0000 osd.4 (osd.4) 564 : cluster 0 3.9 deep-scrub starts 2024-04-02T05:07:52.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:52 smithi098 bash[23790]: cluster 2024-04-02T05:07:51.708437+0000 osd.4 (osd.4) 565 : cluster 0 3.9 deep-scrub ok 2024-04-02T05:07:52.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[21528]: cluster 2024-04-02T05:07:51.509959+0000 osd.3 (osd.3) 615 : cluster 0 2.1e scrub starts 2024-04-02T05:07:52.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[21528]: cluster 2024-04-02T05:07:51.512213+0000 osd.3 (osd.3) 616 : cluster 0 2.1e scrub ok 2024-04-02T05:07:52.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[21528]: cluster 2024-04-02T05:07:51.694764+0000 osd.4 (osd.4) 564 : cluster 0 3.9 deep-scrub starts 2024-04-02T05:07:52.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[21528]: cluster 2024-04-02T05:07:51.708437+0000 osd.4 (osd.4) 565 : cluster 0 3.9 deep-scrub ok 2024-04-02T05:07:52.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[28904]: cluster 2024-04-02T05:07:50.832276+0000 osd.6 (osd.6) 533 : cluster 0 1.6 scrub starts 2024-04-02T05:07:52.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[28904]: cluster 2024-04-02T05:07:50.834595+0000 osd.6 (osd.6) 534 : cluster 0 1.6 scrub ok 2024-04-02T05:07:52.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[28904]: cluster 2024-04-02T05:07:51.509959+0000 osd.3 (osd.3) 615 : cluster 0 2.1e scrub starts 2024-04-02T05:07:52.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[28904]: cluster 2024-04-02T05:07:51.512213+0000 osd.3 (osd.3) 616 : cluster 0 2.1e scrub ok 2024-04-02T05:07:52.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[28904]: cluster 2024-04-02T05:07:51.694764+0000 osd.4 (osd.4) 564 : cluster 0 3.9 deep-scrub starts 2024-04-02T05:07:52.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:52 smithi069 bash[28904]: cluster 2024-04-02T05:07:51.708437+0000 osd.4 (osd.4) 565 : cluster 0 3.9 deep-scrub ok 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:51.876430+0000 osd.6 (osd.6) 535 : cluster 0 4.a scrub starts 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:51.878674+0000 osd.6 (osd.6) 536 : cluster 0 4.a scrub ok 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.452660+0000 osd.7 (osd.7) 393 : cluster 0 2.0 scrub starts 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.455795+0000 osd.7 (osd.7) 394 : cluster 0 2.0 scrub ok 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.511356+0000 mgr.y (mgr.14538) 921 : cluster 0 pgmap v1559: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.629052+0000 osd.1 (osd.1) 597 : cluster 0 3.4 scrub starts 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.634823+0000 osd.1 (osd.1) 598 : cluster 0 3.4 scrub ok 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.820454+0000 osd.2 (osd.2) 444 : cluster 0 2.e scrub starts 2024-04-02T05:07:53.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:53 smithi098 bash[23790]: cluster 2024-04-02T05:07:52.822701+0000 osd.2 (osd.2) 445 : cluster 0 2.e scrub ok 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:51.876430+0000 osd.6 (osd.6) 535 : cluster 0 4.a scrub starts 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:51.878674+0000 osd.6 (osd.6) 536 : cluster 0 4.a scrub ok 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.452660+0000 osd.7 (osd.7) 393 : cluster 0 2.0 scrub starts 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.455795+0000 osd.7 (osd.7) 394 : cluster 0 2.0 scrub ok 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.511356+0000 mgr.y (mgr.14538) 921 : cluster 0 pgmap v1559: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.629052+0000 osd.1 (osd.1) 597 : cluster 0 3.4 scrub starts 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.634823+0000 osd.1 (osd.1) 598 : cluster 0 3.4 scrub ok 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.820454+0000 osd.2 (osd.2) 444 : cluster 0 2.e scrub starts 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[21528]: cluster 2024-04-02T05:07:52.822701+0000 osd.2 (osd.2) 445 : cluster 0 2.e scrub ok 2024-04-02T05:07:53.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:51.876430+0000 osd.6 (osd.6) 535 : cluster 0 4.a scrub starts 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:51.878674+0000 osd.6 (osd.6) 536 : cluster 0 4.a scrub ok 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.452660+0000 osd.7 (osd.7) 393 : cluster 0 2.0 scrub starts 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.455795+0000 osd.7 (osd.7) 394 : cluster 0 2.0 scrub ok 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.511356+0000 mgr.y (mgr.14538) 921 : cluster 0 pgmap v1559: 184 pgs: 184 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.629052+0000 osd.1 (osd.1) 597 : cluster 0 3.4 scrub starts 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.634823+0000 osd.1 (osd.1) 598 : cluster 0 3.4 scrub ok 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.820454+0000 osd.2 (osd.2) 444 : cluster 0 2.e scrub starts 2024-04-02T05:07:53.805 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:53 smithi069 bash[28904]: cluster 2024-04-02T05:07:52.822701+0000 osd.2 (osd.2) 445 : cluster 0 2.e scrub ok 2024-04-02T05:07:53.964 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378970 got 81604378974 for osd.2 2024-04-02T05:07:53.964 DEBUG:teuthology.parallel:result is None 2024-04-02T05:07:54.325 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:07:54.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.455081+0000 osd.3 (osd.3) 617 : cluster 0 3.7 scrub starts 2024-04-02T05:07:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.458393+0000 osd.3 (osd.3) 618 : cluster 0 3.7 scrub ok 2024-04-02T05:07:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.737932+0000 osd.4 (osd.4) 566 : cluster 0 4.1b scrub starts 2024-04-02T05:07:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.740231+0000 osd.4 (osd.4) 567 : cluster 0 4.1b scrub ok 2024-04-02T05:07:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.861960+0000 osd.2 (osd.2) 446 : cluster 0 6.f scrub starts 2024-04-02T05:07:54.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:54 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.863705+0000 osd.2 (osd.2) 447 : cluster 0 6.f scrub ok 2024-04-02T05:07:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.455081+0000 osd.3 (osd.3) 617 : cluster 0 3.7 scrub starts 2024-04-02T05:07:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.458393+0000 osd.3 (osd.3) 618 : cluster 0 3.7 scrub ok 2024-04-02T05:07:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.737932+0000 osd.4 (osd.4) 566 : cluster 0 4.1b scrub starts 2024-04-02T05:07:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.740231+0000 osd.4 (osd.4) 567 : cluster 0 4.1b scrub ok 2024-04-02T05:07:54.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.861960+0000 osd.2 (osd.2) 446 : cluster 0 6.f scrub starts 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.863705+0000 osd.2 (osd.2) 447 : cluster 0 6.f scrub ok 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.455081+0000 osd.3 (osd.3) 617 : cluster 0 3.7 scrub starts 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.458393+0000 osd.3 (osd.3) 618 : cluster 0 3.7 scrub ok 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.737932+0000 osd.4 (osd.4) 566 : cluster 0 4.1b scrub starts 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.740231+0000 osd.4 (osd.4) 567 : cluster 0 4.1b scrub ok 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.861960+0000 osd.2 (osd.2) 446 : cluster 0 6.f scrub starts 2024-04-02T05:07:54.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:54 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.863705+0000 osd.2 (osd.2) 447 : cluster 0 6.f scrub ok 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.938030+0000 osd.6 (osd.6) 537 : cluster 0 2.1a scrub starts 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:53.940282+0000 osd.6 (osd.6) 538 : cluster 0 2.1a scrub ok 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: audit 2024-04-02T05:07:54.001709+0000 mgr.y (mgr.14538) 922 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.158357+0000 osd.5 (osd.5) 411 : cluster 0 6.7 scrub starts 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.160745+0000 osd.5 (osd.5) 412 : cluster 0 6.7 scrub ok 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.424800+0000 osd.3 (osd.3) 619 : cluster 0 6.c scrub starts 2024-04-02T05:07:55.682 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.427474+0000 osd.3 (osd.3) 620 : cluster 0 6.c scrub ok 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.481475+0000 osd.7 (osd.7) 395 : cluster 0 10.e deep-scrub starts 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.484442+0000 osd.7 (osd.7) 396 : cluster 0 10.e deep-scrub ok 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.513410+0000 mgr.y (mgr.14538) 923 : cluster 0 pgmap v1560: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.854563+0000 osd.2 (osd.2) 448 : cluster 0 10.3 scrub starts 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[28904]: cluster 2024-04-02T05:07:54.858898+0000 osd.2 (osd.2) 449 : cluster 0 10.3 scrub ok 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.938030+0000 osd.6 (osd.6) 537 : cluster 0 2.1a scrub starts 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:53.940282+0000 osd.6 (osd.6) 538 : cluster 0 2.1a scrub ok 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: audit 2024-04-02T05:07:54.001709+0000 mgr.y (mgr.14538) 922 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.158357+0000 osd.5 (osd.5) 411 : cluster 0 6.7 scrub starts 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.160745+0000 osd.5 (osd.5) 412 : cluster 0 6.7 scrub ok 2024-04-02T05:07:55.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.424800+0000 osd.3 (osd.3) 619 : cluster 0 6.c scrub starts 2024-04-02T05:07:55.684 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.427474+0000 osd.3 (osd.3) 620 : cluster 0 6.c scrub ok 2024-04-02T05:07:55.684 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.481475+0000 osd.7 (osd.7) 395 : cluster 0 10.e deep-scrub starts 2024-04-02T05:07:55.684 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.484442+0000 osd.7 (osd.7) 396 : cluster 0 10.e deep-scrub ok 2024-04-02T05:07:55.684 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.513410+0000 mgr.y (mgr.14538) 923 : cluster 0 pgmap v1560: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:55.684 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.854563+0000 osd.2 (osd.2) 448 : cluster 0 10.3 scrub starts 2024-04-02T05:07:55.684 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:55 smithi069 bash[21528]: cluster 2024-04-02T05:07:54.858898+0000 osd.2 (osd.2) 449 : cluster 0 10.3 scrub ok 2024-04-02T05:07:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.938030+0000 osd.6 (osd.6) 537 : cluster 0 2.1a scrub starts 2024-04-02T05:07:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:53.940282+0000 osd.6 (osd.6) 538 : cluster 0 2.1a scrub ok 2024-04-02T05:07:55.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: audit 2024-04-02T05:07:54.001709+0000 mgr.y (mgr.14538) 922 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.158357+0000 osd.5 (osd.5) 411 : cluster 0 6.7 scrub starts 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.160745+0000 osd.5 (osd.5) 412 : cluster 0 6.7 scrub ok 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.424800+0000 osd.3 (osd.3) 619 : cluster 0 6.c scrub starts 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.427474+0000 osd.3 (osd.3) 620 : cluster 0 6.c scrub ok 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.481475+0000 osd.7 (osd.7) 395 : cluster 0 10.e deep-scrub starts 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.484442+0000 osd.7 (osd.7) 396 : cluster 0 10.e deep-scrub ok 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.513410+0000 mgr.y (mgr.14538) 923 : cluster 0 pgmap v1560: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.854563+0000 osd.2 (osd.2) 448 : cluster 0 10.3 scrub starts 2024-04-02T05:07:55.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:55 smithi098 bash[23790]: cluster 2024-04-02T05:07:54.858898+0000 osd.2 (osd.2) 449 : cluster 0 10.3 scrub ok 2024-04-02T05:07:56.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:56 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.082815+0000 osd.0 (osd.0) 553 : cluster 0 6.9 scrub starts 2024-04-02T05:07:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:56 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.085429+0000 osd.0 (osd.0) 554 : cluster 0 6.9 scrub ok 2024-04-02T05:07:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:56 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.206117+0000 osd.5 (osd.5) 413 : cluster 0 1.8 deep-scrub starts 2024-04-02T05:07:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:56 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.210020+0000 osd.5 (osd.5) 414 : cluster 0 1.8 deep-scrub ok 2024-04-02T05:07:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:56 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.684486+0000 osd.4 (osd.4) 568 : cluster 0 3.3 scrub starts 2024-04-02T05:07:56.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:56 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.689746+0000 osd.4 (osd.4) 569 : cluster 0 3.3 scrub ok 2024-04-02T05:07:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.082815+0000 osd.0 (osd.0) 553 : cluster 0 6.9 scrub starts 2024-04-02T05:07:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.085429+0000 osd.0 (osd.0) 554 : cluster 0 6.9 scrub ok 2024-04-02T05:07:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.206117+0000 osd.5 (osd.5) 413 : cluster 0 1.8 deep-scrub starts 2024-04-02T05:07:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.210020+0000 osd.5 (osd.5) 414 : cluster 0 1.8 deep-scrub ok 2024-04-02T05:07:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.684486+0000 osd.4 (osd.4) 568 : cluster 0 3.3 scrub starts 2024-04-02T05:07:56.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.689746+0000 osd.4 (osd.4) 569 : cluster 0 3.3 scrub ok 2024-04-02T05:07:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.082815+0000 osd.0 (osd.0) 553 : cluster 0 6.9 scrub starts 2024-04-02T05:07:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.085429+0000 osd.0 (osd.0) 554 : cluster 0 6.9 scrub ok 2024-04-02T05:07:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.206117+0000 osd.5 (osd.5) 413 : cluster 0 1.8 deep-scrub starts 2024-04-02T05:07:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.210020+0000 osd.5 (osd.5) 414 : cluster 0 1.8 deep-scrub ok 2024-04-02T05:07:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.684486+0000 osd.4 (osd.4) 568 : cluster 0 3.3 scrub starts 2024-04-02T05:07:56.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:56 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.689746+0000 osd.4 (osd.4) 569 : cluster 0 3.3 scrub ok 2024-04-02T05:07:57.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2371653597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 msgr2=0x7ff464076b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:57.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2371653597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff464076b50 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7ff458009990 tx=0x7ff458009ca0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2371653597 shutdown_connections 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2371653597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 0x7ff4641154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2371653597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff464076b50 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2371653597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 0x7ff464112bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2371653597 >> 172.21.15.69:0/2371653597 conn(0x7ff4640fff20 msgr2=0x7ff464102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2371653597 shutdown_connections 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2371653597 wait complete. 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 librados: starting msgr at 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 librados: starting objecter 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.447+0000 7ff46ac2e700 1 Processor -- start 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- start start 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 librados: setting wanted keys 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 librados: calling monclient init 2024-04-02T05:07:57.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff46419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 0x7ff46419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 0x7ff4641a3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff464118110 con 0x7ff4641130f0 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff464117f90 con 0x7ff464077090 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7ff464118290 con 0x7ff464076770 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4689ca700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff46419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 0x7ff4641a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4689ca700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff46419e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37072/0 (socket says 172.21.15.69:37072) 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4689ca700 1 -- 172.21.15.69:0/2210573545 learned_addr learned my addr 172.21.15.69:0/2210573545 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff463fff700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 0x7ff46419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 -- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 msgr2=0x7ff46419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff46419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 -- 172.21.15.69:0/2210573545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 msgr2=0x7ff46419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:57.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 0x7ff46419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 -- 172.21.15.69:0/2210573545 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4641a3750 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff463fff700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 0x7ff46419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4691cb700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 0x7ff4641a3010 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7ff46406ba70 tx=0x7ff45c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 10 client.?.objecter ms_handle_connect 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c00ca20 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7ff45c00cbc0 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4641a3a40 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 10 client.34026.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c004660 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4641ab2c0 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 librados: init done 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff46404ff00 con 0x7ff4641130f0 2024-04-02T05:07:57.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7ff45c012050 con 0x7ff4641130f0 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff45406f5e0 0x7ff454071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7ff45c08dfd0 con 0x7ff4641130f0 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 10 client.34026.objecter ms_dispatch 0x7ff464001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 3 client.34026.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 3 client.34026.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff4689ca700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff45406f5e0 0x7ff454071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.451+0000 7ff461ffb700 20 client.34026.objecter dump_active .. 0 homeless 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.455+0000 7ff461ffb700 10 client.34026.objecter ms_handle_connect 0x7ff45406f5e0 2024-04-02T05:07:57.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.455+0000 7ff4689ca700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff45406f5e0 0x7ff454071aa0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 crypto rx=0x7ff45800b2e0 tx=0x7ff458009590 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:57.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.459+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff45c022060 con 0x7ff4641130f0 2024-04-02T05:07:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.726119+0000 osd.1 (osd.1) 599 : cluster 0 2.d scrub starts 2024-04-02T05:07:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[21528]: cluster 2024-04-02T05:07:55.728427+0000 osd.1 (osd.1) 600 : cluster 0 2.d scrub ok 2024-04-02T05:07:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[21528]: cluster 2024-04-02T05:07:56.241077+0000 osd.5 (osd.5) 415 : cluster 0 3.1c scrub starts 2024-04-02T05:07:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[21528]: cluster 2024-04-02T05:07:56.246137+0000 osd.5 (osd.5) 416 : cluster 0 3.1c scrub ok 2024-04-02T05:07:57.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[21528]: cluster 2024-04-02T05:07:56.514229+0000 mgr.y (mgr.14538) 924 : cluster 0 pgmap v1561: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:57.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.726119+0000 osd.1 (osd.1) 599 : cluster 0 2.d scrub starts 2024-04-02T05:07:57.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[28904]: cluster 2024-04-02T05:07:55.728427+0000 osd.1 (osd.1) 600 : cluster 0 2.d scrub ok 2024-04-02T05:07:57.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[28904]: cluster 2024-04-02T05:07:56.241077+0000 osd.5 (osd.5) 415 : cluster 0 3.1c scrub starts 2024-04-02T05:07:57.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[28904]: cluster 2024-04-02T05:07:56.246137+0000 osd.5 (osd.5) 416 : cluster 0 3.1c scrub ok 2024-04-02T05:07:57.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:57 smithi069 bash[28904]: cluster 2024-04-02T05:07:56.514229+0000 mgr.y (mgr.14538) 924 : cluster 0 pgmap v1561: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.667+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff4641ab4f0 con 0x7ff4641130f0 2024-04-02T05:07:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.667+0000 7ff461ffb700 1 -- 172.21.15.69:0/2210573545 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff45c010040 con 0x7ff4641130f0 2024-04-02T05:07:57.671 INFO:teuthology.orchestra.run.smithi069.stdout:55834575209 2024-04-02T05:07:57.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 10 librados: watch_flush enter 2024-04-02T05:07:57.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 10 librados: watch_flush exit 2024-04-02T05:07:57.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 20 client.34026.objecter shutdown clearing up homeless session... 2024-04-02T05:07:57.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 10 client.34026.objecter successfully canceled tick 2024-04-02T05:07:57.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff45406f5e0 msgr2=0x7ff454071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:57.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff45406f5e0 0x7ff454071aa0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 crypto rx=0x7ff45800b2e0 tx=0x7ff458009590 comp rx=0 tx=0).stop 2024-04-02T05:07:57.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 msgr2=0x7ff4641a3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:57.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 0x7ff4641a3010 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7ff46406ba70 tx=0x7ff45c00c3a0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 shutdown_connections 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7ff45406f5e0 0x7ff454071aa0 unknown :-1 s=CLOSED pgs=4061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4641130f0 0x7ff4641a3010 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464076770 0x7ff46419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 --2- 172.21.15.69:0/2210573545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff464077090 0x7ff46419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 >> 172.21.15.69:0/2210573545 conn(0x7ff4640fff20 msgr2=0x7ff464102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 shutdown_connections 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 -- 172.21.15.69:0/2210573545 wait complete. 2024-04-02T05:07:57.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:57.671+0000 7ff46ac2e700 1 librados: shutdown 2024-04-02T05:07:57.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:57 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.726119+0000 osd.1 (osd.1) 599 : cluster 0 2.d scrub starts 2024-04-02T05:07:57.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:57 smithi098 bash[23790]: cluster 2024-04-02T05:07:55.728427+0000 osd.1 (osd.1) 600 : cluster 0 2.d scrub ok 2024-04-02T05:07:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:57 smithi098 bash[23790]: cluster 2024-04-02T05:07:56.241077+0000 osd.5 (osd.5) 415 : cluster 0 3.1c scrub starts 2024-04-02T05:07:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:57 smithi098 bash[23790]: cluster 2024-04-02T05:07:56.246137+0000 osd.5 (osd.5) 416 : cluster 0 3.1c scrub ok 2024-04-02T05:07:57.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:57 smithi098 bash[23790]: cluster 2024-04-02T05:07:56.514229+0000 mgr.y (mgr.14538) 924 : cluster 0 pgmap v1561: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[28904]: cluster 2024-04-02T05:07:57.088148+0000 osd.0 (osd.0) 555 : cluster 0 3.1 deep-scrub starts 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[28904]: cluster 2024-04-02T05:07:57.092876+0000 osd.0 (osd.0) 556 : cluster 0 3.1 deep-scrub ok 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[28904]: cluster 2024-04-02T05:07:57.210716+0000 osd.5 (osd.5) 417 : cluster 0 1.12 deep-scrub starts 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[28904]: cluster 2024-04-02T05:07:57.213297+0000 osd.5 (osd.5) 418 : cluster 0 1.12 deep-scrub ok 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[28904]: audit 2024-04-02T05:07:57.670533+0000 mon.a (mon.0) 4201 : audit 0 from='client.? 172.21.15.69:0/2210573545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[21528]: cluster 2024-04-02T05:07:57.088148+0000 osd.0 (osd.0) 555 : cluster 0 3.1 deep-scrub starts 2024-04-02T05:07:58.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[21528]: cluster 2024-04-02T05:07:57.092876+0000 osd.0 (osd.0) 556 : cluster 0 3.1 deep-scrub ok 2024-04-02T05:07:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[21528]: cluster 2024-04-02T05:07:57.210716+0000 osd.5 (osd.5) 417 : cluster 0 1.12 deep-scrub starts 2024-04-02T05:07:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[21528]: cluster 2024-04-02T05:07:57.213297+0000 osd.5 (osd.5) 418 : cluster 0 1.12 deep-scrub ok 2024-04-02T05:07:58.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[21528]: audit 2024-04-02T05:07:57.670533+0000 mon.a (mon.0) 4201 : audit 0 from='client.? 172.21.15.69:0/2210573545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T05:07:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:58 smithi098 bash[23790]: cluster 2024-04-02T05:07:57.088148+0000 osd.0 (osd.0) 555 : cluster 0 3.1 deep-scrub starts 2024-04-02T05:07:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:58 smithi098 bash[23790]: cluster 2024-04-02T05:07:57.092876+0000 osd.0 (osd.0) 556 : cluster 0 3.1 deep-scrub ok 2024-04-02T05:07:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:58 smithi098 bash[23790]: cluster 2024-04-02T05:07:57.210716+0000 osd.5 (osd.5) 417 : cluster 0 1.12 deep-scrub starts 2024-04-02T05:07:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:58 smithi098 bash[23790]: cluster 2024-04-02T05:07:57.213297+0000 osd.5 (osd.5) 418 : cluster 0 1.12 deep-scrub ok 2024-04-02T05:07:58.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:58 smithi098 bash[23790]: audit 2024-04-02T05:07:57.670533+0000 mon.a (mon.0) 4201 : audit 0 from='client.? 172.21.15.69:0/2210573545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T05:07:59.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:07:58 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:07:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:07:59.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 -- 172.21.15.69:0/1288235707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef8104e70 msgr2=0x7faef8111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1288235707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef8104e70 0x7faef8111330 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7faef000b0a0 tx=0x7faef000b3b0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 -- 172.21.15.69:0/1288235707 shutdown_connections 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1288235707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef8104e70 0x7faef8111330 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1288235707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef810a560 0x7faef810a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1288235707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef81044b0 0x7faef8104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 -- 172.21.15.69:0/1288235707 >> 172.21.15.69:0/1288235707 conn(0x7faef80fffa0 msgr2=0x7faef81023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 -- 172.21.15.69:0/1288235707 shutdown_connections 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.327+0000 7faefe9d9700 1 -- 172.21.15.69:0/1288235707 wait complete. 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 librados: starting msgr at 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 librados: starting objecter 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 Processor -- start 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- start start 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 librados: setting wanted keys 2024-04-02T05:07:59.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 librados: calling monclient init 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef81044b0 0x7faef819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 0x7faef819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef810a560 0x7faef819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faef8113c90 con 0x7faef8104e70 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef8113b10 con 0x7faef810a560 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7faef8113e10 con 0x7faef81044b0 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef81044b0 0x7faef819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 0x7faef819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 0x7faef819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56844/0 (socket says 172.21.15.69:56844) 2024-04-02T05:07:59.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef81044b0 0x7faef819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37094/0 (socket says 172.21.15.69:37094) 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 -- 172.21.15.69:0/1969778282 learned_addr learned my addr 172.21.15.69:0/1969778282 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefcf76700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef810a560 0x7faef819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 -- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef81044b0 msgr2=0x7faef819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef81044b0 0x7faef819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 -- 172.21.15.69:0/1969778282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef810a560 msgr2=0x7faef819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef810a560 0x7faef819eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 -- 172.21.15.69:0/1969778282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef819f1f0 con 0x7faef8104e70 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faeeffff700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 0x7faef819a720 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7faee000e9a0 tx=0x7faee000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 10 client.?.objecter ms_handle_connect 0x7faef8104e70 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee000cbd0 con 0x7faef8104e70 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7faee000cd70 con 0x7faef8104e70 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0010e60 con 0x7faef8104e70 2024-04-02T05:07:59.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef819f4e0 con 0x7faef8104e70 2024-04-02T05:07:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 10 client.34032.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef81a6d60 con 0x7faef8104e70 2024-04-02T05:07:59.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 librados: init done 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef804ff00 con 0x7faef8104e70 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7faee0010430 con 0x7faef8104e70 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.331+0000 7faef5ffb700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faee406f5e0 0x7faee4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7faee008e820 con 0x7faef8104e70 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef5ffb700 10 client.34032.objecter ms_dispatch 0x7faef8001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef5ffb700 3 client.34032.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef5ffb700 3 client.34032.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef7fff700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faee406f5e0 0x7faee4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef5ffb700 20 client.34032.objecter dump_active .. 0 homeless 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef5ffb700 10 client.34032.objecter ms_handle_connect 0x7faee406f5e0 2024-04-02T05:07:59.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.335+0000 7faef7fff700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faee406f5e0 0x7faee4071aa0 secure :-1 s=READY pgs=4062 cs=0 l=1 rev1=1 crypto rx=0x7faee8005950 tx=0x7faee800a300 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:59.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.339+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faee005b870 con 0x7faef8104e70 2024-04-02T05:07:59.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.119426+0000 osd.0 (osd.0) 557 : cluster 0 1.11 scrub starts 2024-04-02T05:07:59.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.122079+0000 osd.0 (osd.0) 558 : cluster 0 1.11 scrub ok 2024-04-02T05:07:59.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.162478+0000 osd.5 (osd.5) 419 : cluster 0 3.5 scrub starts 2024-04-02T05:07:59.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.167412+0000 osd.5 (osd.5) 420 : cluster 0 3.5 scrub ok 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.463248+0000 osd.3 (osd.3) 621 : cluster 0 10.1b deep-scrub starts 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.469864+0000 osd.3 (osd.3) 622 : cluster 0 10.1b deep-scrub ok 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.516097+0000 mgr.y (mgr.14538) 925 : cluster 0 pgmap v1562: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.525905+0000 osd.7 (osd.7) 397 : cluster 0 3.15 scrub starts 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.529806+0000 osd.7 (osd.7) 398 : cluster 0 3.15 scrub ok 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.119426+0000 osd.0 (osd.0) 557 : cluster 0 1.11 scrub starts 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.122079+0000 osd.0 (osd.0) 558 : cluster 0 1.11 scrub ok 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.162478+0000 osd.5 (osd.5) 419 : cluster 0 3.5 scrub starts 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.167412+0000 osd.5 (osd.5) 420 : cluster 0 3.5 scrub ok 2024-04-02T05:07:59.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.463248+0000 osd.3 (osd.3) 621 : cluster 0 10.1b deep-scrub starts 2024-04-02T05:07:59.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.469864+0000 osd.3 (osd.3) 622 : cluster 0 10.1b deep-scrub ok 2024-04-02T05:07:59.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.516097+0000 mgr.y (mgr.14538) 925 : cluster 0 pgmap v1562: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:59.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.525905+0000 osd.7 (osd.7) 397 : cluster 0 3.15 scrub starts 2024-04-02T05:07:59.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:07:59 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.529806+0000 osd.7 (osd.7) 398 : cluster 0 3.15 scrub ok 2024-04-02T05:07:59.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7faef81a6fc0 con 0x7faef8104e70 2024-04-02T05:07:59.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faef5ffb700 1 -- 172.21.15.69:0/1969778282 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faee005f2c0 con 0x7faef8104e70 2024-04-02T05:07:59.557 INFO:teuthology.orchestra.run.smithi069.stdout:124554051926 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 10 librados: watch_flush enter 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 10 librados: watch_flush exit 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 20 client.34032.objecter shutdown clearing up homeless session... 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 10 client.34032.objecter successfully canceled tick 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faee406f5e0 msgr2=0x7faee4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faee406f5e0 0x7faee4071aa0 secure :-1 s=READY pgs=4062 cs=0 l=1 rev1=1 crypto rx=0x7faee8005950 tx=0x7faee800a300 comp rx=0 tx=0).stop 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 msgr2=0x7faef819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 0x7faef819a720 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7faee000e9a0 tx=0x7faee000ecb0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 shutdown_connections 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7faee406f5e0 0x7faee4071aa0 unknown :-1 s=CLOSED pgs=4062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef810a560 0x7faef819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faef81044b0 0x7faef819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 --2- 172.21.15.69:0/1969778282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef8104e70 0x7faef819a720 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 >> 172.21.15.69:0/1969778282 conn(0x7faef80fffa0 msgr2=0x7faef810b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 shutdown_connections 2024-04-02T05:07:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 -- 172.21.15.69:0/1969778282 wait complete. 2024-04-02T05:07:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.555+0000 7faefe9d9700 1 librados: shutdown 2024-04-02T05:07:59.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.119426+0000 osd.0 (osd.0) 557 : cluster 0 1.11 scrub starts 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.122079+0000 osd.0 (osd.0) 558 : cluster 0 1.11 scrub ok 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.162478+0000 osd.5 (osd.5) 419 : cluster 0 3.5 scrub starts 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.167412+0000 osd.5 (osd.5) 420 : cluster 0 3.5 scrub ok 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.463248+0000 osd.3 (osd.3) 621 : cluster 0 10.1b deep-scrub starts 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.469864+0000 osd.3 (osd.3) 622 : cluster 0 10.1b deep-scrub ok 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.516097+0000 mgr.y (mgr.14538) 925 : cluster 0 pgmap v1562: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.525905+0000 osd.7 (osd.7) 397 : cluster 0 3.15 scrub starts 2024-04-02T05:07:59.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:07:59 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.529806+0000 osd.7 (osd.7) 398 : cluster 0 3.15 scrub ok 2024-04-02T05:07:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 -- 172.21.15.69:0/2518523036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0301044b0 msgr2=0x7fc030104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 --2- 172.21.15.69:0/2518523036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0301044b0 0x7fc030104930 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009920 tx=0x7fc01c009c30 comp rx=0 tx=0).stop 2024-04-02T05:07:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 -- 172.21.15.69:0/2518523036 shutdown_connections 2024-04-02T05:07:59.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 --2- 172.21.15.69:0/2518523036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc030111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 --2- 172.21.15.69:0/2518523036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03010a560 0x7fc03010a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 --2- 172.21.15.69:0/2518523036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0301044b0 0x7fc030104930 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 -- 172.21.15.69:0/2518523036 >> 172.21.15.69:0/2518523036 conn(0x7fc0300fffa0 msgr2=0x7fc0301023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 -- 172.21.15.69:0/2518523036 shutdown_connections 2024-04-02T05:07:59.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 -- 172.21.15.69:0/2518523036 wait complete. 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 librados: starting msgr at 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 librados: starting objecter 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 Processor -- start 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 -- start start 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 librados: setting wanted keys 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 librados: calling monclient init 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.747+0000 7fc036a7c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0301044b0 0x7fc03019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc03019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03010a560 0x7fc03019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc030113cd0 con 0x7fc03010a560 2024-04-02T05:07:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc030113b50 con 0x7fc0301044b0 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc030113e50 con 0x7fc030104e70 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc03019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc03019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37112/0 (socket says 172.21.15.69:37112) 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 -- 172.21.15.69:0/3544051314 learned_addr learned my addr 172.21.15.69:0/3544051314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc035019700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03010a560 0x7fc03019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc034818700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0301044b0 0x7fc03019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 -- 172.21.15.69:0/3544051314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0301044b0 msgr2=0x7fc03019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0301044b0 0x7fc03019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 -- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03010a560 msgr2=0x7fc03019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03010a560 0x7fc03019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 -- 172.21.15.69:0/3544051314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03019f300 con 0x7fc030104e70 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc034818700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0301044b0 0x7fc03019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02ffff700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc03019a830 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009230 tx=0x7fc01c00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 10 client.?.objecter ms_handle_connect 0x7fc030104e70 2024-04-02T05:07:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c027070 con 0x7fc030104e70 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc01c00bb60 con 0x7fc030104e70 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03019f590 con 0x7fc030104e70 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c004930 con 0x7fc030104e70 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 10 client.44105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0301a6e70 con 0x7fc030104e70 2024-04-02T05:07:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 librados: init done 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc03004ff00 con 0x7fc030104e70 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc01c02a070 con 0x7fc030104e70 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc02006f5e0 0x7fc020071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7fc01c08f180 con 0x7fc030104e70 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 10 client.44105.objecter ms_dispatch 0x7fc030001550 osd_map(966..966 src has 907..966) 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 3 client.44105.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 3 client.44105.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:07:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc034818700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc02006f5e0 0x7fc020071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:07:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 20 client.44105.objecter dump_active .. 0 homeless 2024-04-02T05:07:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc034818700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc02006f5e0 0x7fc020071aa0 secure :-1 s=READY pgs=4063 cs=0 l=1 rev1=1 crypto rx=0x7fc024007900 tx=0x7fc024008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:07:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.751+0000 7fc02dffb700 10 client.44105.objecter ms_handle_connect 0x7fc02006f5e0 2024-04-02T05:07:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.759+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc01c013030 con 0x7fc030104e70 2024-04-02T05:07:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.963+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc0301a7040 con 0x7fc030104e70 2024-04-02T05:07:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.963+0000 7fc02dffb700 1 -- 172.21.15.69:0/3544051314 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+14 (secure 0 0 0) 0x7fc01c05c1d0 con 0x7fc030104e70 2024-04-02T05:07:59.967 INFO:teuthology.orchestra.run.smithi069.stdout:1692217114795 2024-04-02T05:07:59.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 10 librados: watch_flush enter 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 10 librados: watch_flush exit 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 20 client.44105.objecter shutdown clearing up homeless session... 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 10 client.44105.objecter successfully canceled tick 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc02006f5e0 msgr2=0x7fc020071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc02006f5e0 0x7fc020071aa0 secure :-1 s=READY pgs=4063 cs=0 l=1 rev1=1 crypto rx=0x7fc024007900 tx=0x7fc024008040 comp rx=0 tx=0).stop 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 msgr2=0x7fc03019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc03019a830 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009230 tx=0x7fc01c00fa70 comp rx=0 tx=0).stop 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 shutdown_connections 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc02006f5e0 0x7fc020071aa0 unknown :-1 s=CLOSED pgs=4063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03010a560 0x7fc03019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0301044b0 0x7fc03019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 --2- 172.21.15.69:0/3544051314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc030104e70 0x7fc03019a830 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:07:59.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 >> 172.21.15.69:0/3544051314 conn(0x7fc0300fffa0 msgr2=0x7fc030108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:07:59.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 shutdown_connections 2024-04-02T05:07:59.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 -- 172.21.15.69:0/3544051314 wait complete. 2024-04-02T05:07:59.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:07:59.967+0000 7fc036a7c700 1 librados: shutdown 2024-04-02T05:08:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 -- 172.21.15.69:0/3453668916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724109240 msgr2=0x7f5724109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 --2- 172.21.15.69:0/3453668916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724109240 0x7f5724109620 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f5720009990 tx=0x7f5720009ca0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 -- 172.21.15.69:0/3453668916 shutdown_connections 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 --2- 172.21.15.69:0/3453668916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724115390 0x7f5724117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 --2- 172.21.15.69:0/3453668916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724109240 0x7f5724109620 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 --2- 172.21.15.69:0/3453668916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5724109b60 0x7f5724114e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 -- 172.21.15.69:0/3453668916 >> 172.21.15.69:0/3453668916 conn(0x7f572406d3c0 msgr2=0x7f572406d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 -- 172.21.15.69:0/3453668916 shutdown_connections 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.359+0000 7f572b616700 1 -- 172.21.15.69:0/3453668916 wait complete. 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 librados: starting msgr at 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 librados: starting objecter 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 Processor -- start 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- start start 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 librados: setting wanted keys 2024-04-02T05:08:00.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 librados: calling monclient init 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 0x7f572410acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5724109b60 0x7f572410b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724115390 0x7f572410b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f572411a3d0 con 0x7f5724115390 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f572411a250 con 0x7f5724109b60 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f572411a550 con 0x7f5724109240 2024-04-02T05:08:00.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f57293b2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 0x7f572410acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724115390 0x7f572410b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f57293b2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 0x7f572410acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37132/0 (socket says 172.21.15.69:37132) 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f57293b2700 1 -- 172.21.15.69:0/3120640574 learned_addr learned my addr 172.21.15.69:0/3120640574 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5728bb1700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5724109b60 0x7f572410b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 -- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 msgr2=0x7f572410acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 0x7f572410acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 -- 172.21.15.69:0/3120640574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5724109b60 msgr2=0x7f572410b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5724109b60 0x7f572410b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 -- 172.21.15.69:0/3120640574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57241aefa0 con 0x7f5724115390 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f57293b2700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 0x7f572410acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f5729bb3700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724115390 0x7f572410b730 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f572406b990 tx=0x7f571c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 10 client.?.objecter ms_handle_connect 0x7f5724115390 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:00.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c0092e0 con 0x7f5724115390 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f571c00ce00 con 0x7f5724115390 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57241af230 con 0x7f5724115390 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c007a80 con 0x7f5724115390 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 10 client.34044.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57241af720 con 0x7f5724115390 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 librados: init done 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572404ff00 con 0x7f5724115390 2024-04-02T05:08:00.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f571c01b020 con 0x7f5724115390 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f571006f5e0 0x7f5710071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(966..966 src has 907..966) ==== 9015+0+0 (secure 0 0 0) 0x7f571c08e230 con 0x7f5724115390 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 10 client.34044.objecter ms_dispatch 0x7f5724001550 osd_map(966..966 src has 907..966) 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 3 client.34044.objecter handle_osd_map got epochs [966,966] > 0 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f57293b2700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f571006f5e0 0x7f5710071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 3 client.34044.objecter handle_osd_map decoding full epoch 966 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.363+0000 7f571a7fc700 20 client.34044.objecter dump_active .. 0 homeless 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.367+0000 7f571a7fc700 10 client.34044.objecter ms_handle_connect 0x7f571006f5e0 2024-04-02T05:08:00.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.367+0000 7f57293b2700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f571006f5e0 0x7f5710071aa0 secure :-1 s=READY pgs=4064 cs=0 l=1 rev1=1 crypto rx=0x7f5720006010 tx=0x7f572000b630 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:00.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.371+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f571c010040 con 0x7f5724115390 2024-04-02T05:08:00.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575205 got 55834575209 for osd.1 2024-04-02T05:08:00.438 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:00.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.579+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f572402db30 con 0x7f5724115390 2024-04-02T05:08:00.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.579+0000 7f571a7fc700 1 -- 172.21.15.69:0/3120640574 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f571c012340 con 0x7f5724115390 2024-04-02T05:08:00.583 INFO:teuthology.orchestra.run.smithi069.stdout:201863463232 2024-04-02T05:08:00.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 10 librados: watch_flush enter 2024-04-02T05:08:00.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 10 librados: watch_flush exit 2024-04-02T05:08:00.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 20 client.34044.objecter shutdown clearing up homeless session... 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 10 client.34044.objecter successfully canceled tick 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f571006f5e0 msgr2=0x7f5710071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f571006f5e0 0x7f5710071aa0 secure :-1 s=READY pgs=4064 cs=0 l=1 rev1=1 crypto rx=0x7f5720006010 tx=0x7f572000b630 comp rx=0 tx=0).stop 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724115390 msgr2=0x7f572410b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724115390 0x7f572410b730 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f572406b990 tx=0x7f571c00ef90 comp rx=0 tx=0).stop 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 shutdown_connections 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f571006f5e0 0x7f5710071aa0 unknown :-1 s=CLOSED pgs=4064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5724115390 0x7f572410b730 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5724109240 0x7f572410acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 --2- 172.21.15.69:0/3120640574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5724109b60 0x7f572410b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 >> 172.21.15.69:0/3120640574 conn(0x7f572406d3c0 msgr2=0x7f572410cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 shutdown_connections 2024-04-02T05:08:00.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 -- 172.21.15.69:0/3120640574 wait complete. 2024-04-02T05:08:00.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:00.583+0000 7f572b616700 1 librados: shutdown 2024-04-02T05:08:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.933986+0000 osd.6 (osd.6) 539 : cluster 0 10.12 scrub starts 2024-04-02T05:08:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[21528]: cluster 2024-04-02T05:07:58.936932+0000 osd.6 (osd.6) 540 : cluster 0 10.12 scrub ok 2024-04-02T05:08:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[21528]: cluster 2024-04-02T05:07:59.426199+0000 osd.3 (osd.3) 623 : cluster 0 3.2 scrub starts 2024-04-02T05:08:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[21528]: cluster 2024-04-02T05:07:59.429916+0000 osd.3 (osd.3) 624 : cluster 0 3.2 scrub ok 2024-04-02T05:08:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[21528]: audit 2024-04-02T05:07:59.556394+0000 mon.a (mon.0) 4202 : audit 0 from='client.? 172.21.15.69:0/1969778282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T05:08:00.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[21528]: audit 2024-04-02T05:07:59.966367+0000 mon.c (mon.2) 1179 : audit 0 from='client.? 172.21.15.69:0/3544051314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T05:08:00.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.933986+0000 osd.6 (osd.6) 539 : cluster 0 10.12 scrub starts 2024-04-02T05:08:00.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[28904]: cluster 2024-04-02T05:07:58.936932+0000 osd.6 (osd.6) 540 : cluster 0 10.12 scrub ok 2024-04-02T05:08:00.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[28904]: cluster 2024-04-02T05:07:59.426199+0000 osd.3 (osd.3) 623 : cluster 0 3.2 scrub starts 2024-04-02T05:08:00.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[28904]: cluster 2024-04-02T05:07:59.429916+0000 osd.3 (osd.3) 624 : cluster 0 3.2 scrub ok 2024-04-02T05:08:00.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[28904]: audit 2024-04-02T05:07:59.556394+0000 mon.a (mon.0) 4202 : audit 0 from='client.? 172.21.15.69:0/1969778282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T05:08:00.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:00 smithi069 bash[28904]: audit 2024-04-02T05:07:59.966367+0000 mon.c (mon.2) 1179 : audit 0 from='client.? 172.21.15.69:0/3544051314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T05:08:00.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:00 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.933986+0000 osd.6 (osd.6) 539 : cluster 0 10.12 scrub starts 2024-04-02T05:08:00.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:00 smithi098 bash[23790]: cluster 2024-04-02T05:07:58.936932+0000 osd.6 (osd.6) 540 : cluster 0 10.12 scrub ok 2024-04-02T05:08:00.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:00 smithi098 bash[23790]: cluster 2024-04-02T05:07:59.426199+0000 osd.3 (osd.3) 623 : cluster 0 3.2 scrub starts 2024-04-02T05:08:00.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:00 smithi098 bash[23790]: cluster 2024-04-02T05:07:59.429916+0000 osd.3 (osd.3) 624 : cluster 0 3.2 scrub ok 2024-04-02T05:08:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:00 smithi098 bash[23790]: audit 2024-04-02T05:07:59.556394+0000 mon.a (mon.0) 4202 : audit 0 from='client.? 172.21.15.69:0/1969778282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T05:08:00.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:00 smithi098 bash[23790]: audit 2024-04-02T05:07:59.966367+0000 mon.c (mon.2) 1179 : audit 0 from='client.? 172.21.15.69:0/3544051314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T05:08:01.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[21528]: cluster 2024-04-02T05:08:00.470962+0000 osd.3 (osd.3) 625 : cluster 0 5.0 scrub starts 2024-04-02T05:08:01.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[21528]: cluster 2024-04-02T05:08:00.472743+0000 osd.3 (osd.3) 626 : cluster 0 5.0 scrub ok 2024-04-02T05:08:01.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[21528]: cluster 2024-04-02T05:08:00.517019+0000 mgr.y (mgr.14538) 926 : cluster 0 pgmap v1563: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:01.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[21528]: cluster 2024-04-02T05:08:00.536233+0000 osd.7 (osd.7) 399 : cluster 0 2.1c scrub starts 2024-04-02T05:08:01.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[21528]: cluster 2024-04-02T05:08:00.541316+0000 osd.7 (osd.7) 400 : cluster 0 2.1c scrub ok 2024-04-02T05:08:01.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[21528]: audit 2024-04-02T05:08:00.582012+0000 mon.a (mon.0) 4203 : audit 0 from='client.? 172.21.15.69:0/3120640574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T05:08:01.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[28904]: cluster 2024-04-02T05:08:00.470962+0000 osd.3 (osd.3) 625 : cluster 0 5.0 scrub starts 2024-04-02T05:08:01.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[28904]: cluster 2024-04-02T05:08:00.472743+0000 osd.3 (osd.3) 626 : cluster 0 5.0 scrub ok 2024-04-02T05:08:01.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[28904]: cluster 2024-04-02T05:08:00.517019+0000 mgr.y (mgr.14538) 926 : cluster 0 pgmap v1563: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:01.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[28904]: cluster 2024-04-02T05:08:00.536233+0000 osd.7 (osd.7) 399 : cluster 0 2.1c scrub starts 2024-04-02T05:08:01.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[28904]: cluster 2024-04-02T05:08:00.541316+0000 osd.7 (osd.7) 400 : cluster 0 2.1c scrub ok 2024-04-02T05:08:01.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:01 smithi069 bash[28904]: audit 2024-04-02T05:08:00.582012+0000 mon.a (mon.0) 4203 : audit 0 from='client.? 172.21.15.69:0/3120640574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T05:08:01.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051921 got 124554051926 for osd.3 2024-04-02T05:08:01.717 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:01.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:01 smithi098 bash[23790]: cluster 2024-04-02T05:08:00.470962+0000 osd.3 (osd.3) 625 : cluster 0 5.0 scrub starts 2024-04-02T05:08:01.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:01 smithi098 bash[23790]: cluster 2024-04-02T05:08:00.472743+0000 osd.3 (osd.3) 626 : cluster 0 5.0 scrub ok 2024-04-02T05:08:01.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:01 smithi098 bash[23790]: cluster 2024-04-02T05:08:00.517019+0000 mgr.y (mgr.14538) 926 : cluster 0 pgmap v1563: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:01.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:01 smithi098 bash[23790]: cluster 2024-04-02T05:08:00.536233+0000 osd.7 (osd.7) 399 : cluster 0 2.1c scrub starts 2024-04-02T05:08:01.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:01 smithi098 bash[23790]: cluster 2024-04-02T05:08:00.541316+0000 osd.7 (osd.7) 400 : cluster 0 2.1c scrub ok 2024-04-02T05:08:01.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:01 smithi098 bash[23790]: audit 2024-04-02T05:08:00.582012+0000 mon.a (mon.0) 4203 : audit 0 from='client.? 172.21.15.69:0/3120640574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T05:08:02.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463227 got 201863463232 for osd.5 2024-04-02T05:08:02.452 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/2901691240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c34101cf0 msgr2=0x7f5c34102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/2901691240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c34101cf0 0x7f5c34102170 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009990 tx=0x7f5c28009ca0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/2901691240 shutdown_connections 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/2901691240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c341026b0 0x7f5c3410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/2901691240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c34107410 0x7f5c341077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/2901691240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c34101cf0 0x7f5c34102170 unknown :-1 s=CLOSED pgs=3348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/2901691240 >> 172.21.15.69:0/2901691240 conn(0x7f5c340782b0 msgr2=0x7f5c340fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/2901691240 shutdown_connections 2024-04-02T05:08:02.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/2901691240 wait complete. 2024-04-02T05:08:02.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 librados: starting msgr at 2024-04-02T05:08:02.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.643+0000 7f5c3aa9a700 1 librados: starting objecter 2024-04-02T05:08:02.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 Processor -- start 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- start start 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 librados: setting wanted keys 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 librados: calling monclient init 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c34101cf0 0x7f5c341981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 0x7f5c341986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 0x7f5c3419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c34110ab0 con 0x7f5c34107410 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c34110930 con 0x7f5c34101cf0 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f5c34110c30 con 0x7f5c341026b0 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 0x7f5c3419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c33fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 0x7f5c341986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 0x7f5c3419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56914/0 (socket says 172.21.15.69:56914) 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c33fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 0x7f5c341986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37150/0 (socket says 172.21.15.69:37150) 2024-04-02T05:08:02.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 -- 172.21.15.69:0/922181116 learned_addr learned my addr 172.21.15.69:0/922181116 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c38836700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c34101cf0 0x7f5c341981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 -- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 msgr2=0x7f5c341986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 0x7f5c341986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 -- 172.21.15.69:0/922181116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c34101cf0 msgr2=0x7f5c341981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c34101cf0 0x7f5c341981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 -- 172.21.15.69:0/922181116 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3419d1b0 con 0x7f5c34107410 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c33fff700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 0x7f5c341986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c39037700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 0x7f5c3419ca70 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00ef90 tx=0x7f5c2c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 10 client.?.objecter ms_handle_connect 0x7f5c34107410 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0092e0 con 0x7f5c34107410 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3419d440 con 0x7f5c34107410 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f5c2c004bb0 con 0x7f5c34107410 2024-04-02T05:08:02.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 10 client.34050.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:02.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c004020 con 0x7f5c34107410 2024-04-02T05:08:02.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c341a4d20 con 0x7f5c34107410 2024-04-02T05:08:02.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 librados: init done 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c34100780 con 0x7f5c34107410 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f5c2c01b030 con 0x7f5c34107410 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.647+0000 7f5c31ffb700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5c2406f4b0 0x7f5c24071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(967..967 src has 907..967) ==== 9015+0+0 (secure 0 0 0) 0x7f5c2c08de80 con 0x7f5c34107410 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c31ffb700 10 client.34050.objecter ms_dispatch 0x7f5c34001550 osd_map(967..967 src has 907..967) 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c31ffb700 3 client.34050.objecter handle_osd_map got epochs [967,967] > 0 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c31ffb700 3 client.34050.objecter handle_osd_map decoding full epoch 967 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c38836700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5c2406f4b0 0x7f5c24071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c31ffb700 20 client.34050.objecter dump_active .. 0 homeless 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c31ffb700 10 client.34050.objecter ms_handle_connect 0x7f5c2406f4b0 2024-04-02T05:08:02.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.651+0000 7f5c38836700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5c2406f4b0 0x7f5c24071970 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 crypto rx=0x7f5c20006fd0 tx=0x7f5c20008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:02.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.655+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5c2c010040 con 0x7f5c34107410 2024-04-02T05:08:02.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:02 smithi098 bash[23790]: cluster 2024-04-02T05:08:01.178186+0000 osd.5 (osd.5) 421 : cluster 0 10.8 scrub starts 2024-04-02T05:08:02.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:02 smithi098 bash[23790]: cluster 2024-04-02T05:08:01.180471+0000 osd.5 (osd.5) 422 : cluster 0 10.8 scrub ok 2024-04-02T05:08:02.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:02 smithi069 bash[21528]: cluster 2024-04-02T05:08:01.178186+0000 osd.5 (osd.5) 421 : cluster 0 10.8 scrub starts 2024-04-02T05:08:02.791 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:02 smithi069 bash[21528]: cluster 2024-04-02T05:08:01.180471+0000 osd.5 (osd.5) 422 : cluster 0 10.8 scrub ok 2024-04-02T05:08:02.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:02 smithi069 bash[28904]: cluster 2024-04-02T05:08:01.178186+0000 osd.5 (osd.5) 421 : cluster 0 10.8 scrub starts 2024-04-02T05:08:02.791 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:02 smithi069 bash[28904]: cluster 2024-04-02T05:08:01.180471+0000 osd.5 (osd.5) 422 : cluster 0 10.8 scrub ok 2024-04-02T05:08:02.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.871+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5c3404ff00 con 0x7f5c34107410 2024-04-02T05:08:02.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.871+0000 7f5c31ffb700 1 -- 172.21.15.69:0/922181116 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5c2c05ae50 con 0x7f5c34107410 2024-04-02T05:08:02.874 INFO:teuthology.orchestra.run.smithi069.stdout:249108103479 2024-04-02T05:08:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 10 librados: watch_flush enter 2024-04-02T05:08:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 10 librados: watch_flush exit 2024-04-02T05:08:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 20 client.34050.objecter shutdown clearing up homeless session... 2024-04-02T05:08:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 10 client.34050.objecter successfully canceled tick 2024-04-02T05:08:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5c2406f4b0 msgr2=0x7f5c24071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:02.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5c2406f4b0 0x7f5c24071970 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 crypto rx=0x7f5c20006fd0 tx=0x7f5c20008040 comp rx=0 tx=0).stop 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 msgr2=0x7f5c3419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 0x7f5c3419ca70 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00ef90 tx=0x7f5c2c00c3a0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 shutdown_connections 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f5c2406f4b0 0x7f5c24071970 unknown :-1 s=CLOSED pgs=4065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c34107410 0x7f5c3419ca70 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c34101cf0 0x7f5c341981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 --2- 172.21.15.69:0/922181116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c341026b0 0x7f5c341986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 >> 172.21.15.69:0/922181116 conn(0x7f5c340782b0 msgr2=0x7f5c34100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 shutdown_connections 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 -- 172.21.15.69:0/922181116 wait complete. 2024-04-02T05:08:02.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:02.875+0000 7f5c3aa9a700 1 librados: shutdown 2024-04-02T05:08:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- 172.21.15.69:0/3300314760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 msgr2=0x7fe35806af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- 172.21.15.69:0/3300314760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe35806af40 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fe3480099c0 tx=0x7fe348009cd0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- 172.21.15.69:0/3300314760 shutdown_connections 2024-04-02T05:08:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- 172.21.15.69:0/3300314760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe358107620 0x7fe35810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- 172.21.15.69:0/3300314760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe35806af40 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- 172.21.15.69:0/3300314760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 0x7fe358106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- 172.21.15.69:0/3300314760 >> 172.21.15.69:0/3300314760 conn(0x7fe358077190 msgr2=0x7fe3580775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- 172.21.15.69:0/3300314760 shutdown_connections 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- 172.21.15.69:0/3300314760 wait complete. 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 librados: starting msgr at 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 librados: starting objecter 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 Processor -- start 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- start start 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 librados: setting wanted keys 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 librados: calling monclient init 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe3581012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 0x7fe3581017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe358107620 0x7fe358101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe358111ad0 con 0x7fe358107620 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe358111950 con 0x7fe35806b510 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe35d35d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fe358111c50 con 0x7fe35806ab60 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe3581012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe3581012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37166/0 (socket says 172.21.15.69:37166) 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 -- 172.21.15.69:0/4037990146 learned_addr learned my addr 172.21.15.69:0/4037990146 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe3577fe700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe358107620 0x7fe358101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe3567fc700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 0x7fe3581017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 -- 172.21.15.69:0/4037990146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 msgr2=0x7fe3581017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 0x7fe3581017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 -- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe358107620 msgr2=0x7fe358101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe358107620 0x7fe358101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe356ffd700 1 -- 172.21.15.69:0/4037990146 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3581a6640 con 0x7fe35806ab60 2024-04-02T05:08:03.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.059+0000 7fe3567fc700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 0x7fe3581017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe356ffd700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe3581012a0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fe3480098e0 tx=0x7fe348003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 10 client.?.objecter ms_handle_connect 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348009e60 con 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fe34800f460 con 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3581a68d0 con 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348017650 con 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe35d35d700 10 client.44123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3581a6d80 con 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe35d35d700 1 librados: init done 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe35804ff00 con 0x7fe35806ab60 2024-04-02T05:08:03.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fe34800f980 con 0x7fe35806ab60 2024-04-02T05:08:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe34006f5e0 0x7fe340071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(967..967 src has 907..967) ==== 9015+0+0 (secure 0 0 0) 0x7fe34808f430 con 0x7fe35806ab60 2024-04-02T05:08:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 10 client.44123.objecter ms_dispatch 0x7fe358001550 osd_map(967..967 src has 907..967) 2024-04-02T05:08:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 3 client.44123.objecter handle_osd_map got epochs [967,967] > 0 2024-04-02T05:08:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 3 client.44123.objecter handle_osd_map decoding full epoch 967 2024-04-02T05:08:03.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe3567fc700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe34006f5e0 0x7fe340071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 20 client.44123.objecter dump_active .. 0 homeless 2024-04-02T05:08:03.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe33ffff700 10 client.44123.objecter ms_handle_connect 0x7fe34006f5e0 2024-04-02T05:08:03.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.063+0000 7fe3567fc700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe34006f5e0 0x7fe340071aa0 secure :-1 s=READY pgs=4066 cs=0 l=1 rev1=1 crypto rx=0x7fe358108b00 tx=0x7fe34c008040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:03.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.067+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe348013070 con 0x7fe35806ab60 2024-04-02T05:08:03.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe3581a7040 con 0x7fe35806ab60 2024-04-02T05:08:03.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe33ffff700 1 -- 172.21.15.69:0/4037990146 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe34805c480 con 0x7fe35806ab60 2024-04-02T05:08:03.281 INFO:teuthology.orchestra.run.smithi069.stdout:154618822988 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 10 librados: watch_flush enter 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 10 librados: watch_flush exit 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 20 client.44123.objecter shutdown clearing up homeless session... 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 10 client.44123.objecter successfully canceled tick 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe34006f5e0 msgr2=0x7fe340071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe34006f5e0 0x7fe340071aa0 secure :-1 s=READY pgs=4066 cs=0 l=1 rev1=1 crypto rx=0x7fe358108b00 tx=0x7fe34c008040 comp rx=0 tx=0).stop 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 msgr2=0x7fe3581012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe3581012a0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fe3480098e0 tx=0x7fe348003e80 comp rx=0 tx=0).stop 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 shutdown_connections 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fe34006f5e0 0x7fe340071aa0 unknown :-1 s=CLOSED pgs=4066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe358107620 0x7fe358101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe35806ab60 0x7fe3581012a0 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 --2- 172.21.15.69:0/4037990146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe35806b510 0x7fe3581017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 >> 172.21.15.69:0/4037990146 conn(0x7fe358077190 msgr2=0x7fe3580ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:03.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 shutdown_connections 2024-04-02T05:08:03.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 -- 172.21.15.69:0/4037990146 wait complete. 2024-04-02T05:08:03.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.279+0000 7fe35d35d700 1 librados: shutdown 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:01.527703+0000 osd.5 (osd.5) 423 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:01.528092+0000 osd.5 (osd.5) 424 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:01.906234+0000 osd.6 (osd.6) 541 : cluster 0 10.d scrub starts 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:01.910847+0000 osd.6 (osd.6) 542 : cluster 0 10.d scrub ok 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:02.039792+0000 osd.0 (osd.0) 559 : cluster 0 4.6 scrub starts 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:02.042183+0000 osd.0 (osd.0) 560 : cluster 0 4.6 scrub ok 2024-04-02T05:08:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:02.386028+0000 mon.a (mon.0) 4204 : cluster 0 osdmap e967: 8 total, 8 up, 8 in 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:02.473682+0000 osd.3 (osd.3) 627 : cluster 0 1.c scrub starts 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:02.475604+0000 osd.3 (osd.3) 628 : cluster 0 1.c scrub ok 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: cluster 2024-04-02T05:08:02.517690+0000 mgr.y (mgr.14538) 927 : cluster 0 pgmap v1565: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: audit 2024-04-02T05:08:02.873368+0000 mon.a (mon.0) 4205 : audit 0 from='client.? 172.21.15.69:0/922181116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[21528]: audit 2024-04-02T05:08:03.280552+0000 mon.c (mon.2) 1180 : audit 0 from='client.? 172.21.15.69:0/4037990146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:01.527703+0000 osd.5 (osd.5) 423 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:01.528092+0000 osd.5 (osd.5) 424 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:01.906234+0000 osd.6 (osd.6) 541 : cluster 0 10.d scrub starts 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:01.910847+0000 osd.6 (osd.6) 542 : cluster 0 10.d scrub ok 2024-04-02T05:08:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:02.039792+0000 osd.0 (osd.0) 559 : cluster 0 4.6 scrub starts 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:02.042183+0000 osd.0 (osd.0) 560 : cluster 0 4.6 scrub ok 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:02.386028+0000 mon.a (mon.0) 4204 : cluster 0 osdmap e967: 8 total, 8 up, 8 in 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:02.473682+0000 osd.3 (osd.3) 627 : cluster 0 1.c scrub starts 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:02.475604+0000 osd.3 (osd.3) 628 : cluster 0 1.c scrub ok 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: cluster 2024-04-02T05:08:02.517690+0000 mgr.y (mgr.14538) 927 : cluster 0 pgmap v1565: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: audit 2024-04-02T05:08:02.873368+0000 mon.a (mon.0) 4205 : audit 0 from='client.? 172.21.15.69:0/922181116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T05:08:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:03 smithi069 bash[28904]: audit 2024-04-02T05:08:03.280552+0000 mon.c (mon.2) 1180 : audit 0 from='client.? 172.21.15.69:0/4037990146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T05:08:03.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:01.527703+0000 osd.5 (osd.5) 423 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:01.528092+0000 osd.5 (osd.5) 424 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:01.906234+0000 osd.6 (osd.6) 541 : cluster 0 10.d scrub starts 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:01.910847+0000 osd.6 (osd.6) 542 : cluster 0 10.d scrub ok 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:02.039792+0000 osd.0 (osd.0) 559 : cluster 0 4.6 scrub starts 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:02.042183+0000 osd.0 (osd.0) 560 : cluster 0 4.6 scrub ok 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:02.386028+0000 mon.a (mon.0) 4204 : cluster 0 osdmap e967: 8 total, 8 up, 8 in 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:02.473682+0000 osd.3 (osd.3) 627 : cluster 0 1.c scrub starts 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:02.475604+0000 osd.3 (osd.3) 628 : cluster 0 1.c scrub ok 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: cluster 2024-04-02T05:08:02.517690+0000 mgr.y (mgr.14538) 927 : cluster 0 pgmap v1565: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: audit 2024-04-02T05:08:02.873368+0000 mon.a (mon.0) 4205 : audit 0 from='client.? 172.21.15.69:0/922181116' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T05:08:03.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:03 smithi098 bash[23790]: audit 2024-04-02T05:08:03.280552+0000 mon.c (mon.2) 1180 : audit 0 from='client.? 172.21.15.69:0/4037990146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 -- 172.21.15.69:0/422278440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 msgr2=0x7f17f8106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 --2- 172.21.15.69:0/422278440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f8106d90 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00aaa0 tx=0x7f17ec00adb0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 -- 172.21.15.69:0/422278440 shutdown_connections 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 --2- 172.21.15.69:0/422278440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8101600 0x7f17f810d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 --2- 172.21.15.69:0/422278440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f8106d90 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 --2- 172.21.15.69:0/422278440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f8100c40 0x7f17f81010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 -- 172.21.15.69:0/422278440 >> 172.21.15.69:0/422278440 conn(0x7f17f80782b0 msgr2=0x7f17f80fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:03.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 -- 172.21.15.69:0/422278440 shutdown_connections 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 -- 172.21.15.69:0/422278440 wait complete. 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 librados: starting msgr at 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.987+0000 7f17fdc80700 1 librados: starting objecter 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 Processor -- start 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- start start 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 librados: setting wanted keys 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 librados: calling monclient init 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8100c40 0x7f17f80fc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f8101600 0x7f17f80fcc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f80fd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17f810f950 con 0x7f17f81069b0 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17f810f7d0 con 0x7f17f8100c40 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f17f810fad0 con 0x7f17f8101600 2024-04-02T05:08:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f6ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f8101600 0x7f17f80fcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f80fd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f80fd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56954/0 (socket says 172.21.15.69:56954) 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 -- 172.21.15.69:0/2434691325 learned_addr learned my addr 172.21.15.69:0/2434691325 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f77fe700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8100c40 0x7f17f80fc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 -- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f8101600 msgr2=0x7f17f80fcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f8101600 0x7f17f80fcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 -- 172.21.15.69:0/2434691325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8100c40 msgr2=0x7f17f80fc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8100c40 0x7f17f80fc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 -- 172.21.15.69:0/2434691325 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f81a65d0 con 0x7f17f81069b0 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f77fe700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8100c40 0x7f17f80fc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f7fff700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f80fd160 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f17f000b970 tx=0x7f17f000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 10 client.?.objecter ms_handle_connect 0x7f17f81069b0 2024-04-02T05:08:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0019070 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f81a68c0 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f17f000cdb0 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0014e40 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 10 client.34062.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f81a6e00 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 librados: init done 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f804ff00 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f17f001c070 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e006f5e0 0x7f17e0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(967..967 src has 907..967) ==== 9015+0+0 (secure 0 0 0) 0x7f17f008d9c0 con 0x7f17f81069b0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 10 client.34062.objecter ms_dispatch 0x7f17f8001550 osd_map(967..967 src has 907..967) 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 3 client.34062.objecter handle_osd_map got epochs [967,967] > 0 2024-04-02T05:08:03.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 3 client.34062.objecter handle_osd_map decoding full epoch 967 2024-04-02T05:08:04.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f77fe700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e006f5e0 0x7f17e0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:04.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.991+0000 7f17f4ff9700 20 client.34062.objecter dump_active .. 0 homeless 2024-04-02T05:08:04.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.995+0000 7f17f4ff9700 10 client.34062.objecter ms_handle_connect 0x7f17e006f5e0 2024-04-02T05:08:04.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.995+0000 7f17f77fe700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e006f5e0 0x7f17e0071aa0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7f17ec000c00 tx=0x7f17ec005fb0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:04.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:03.999+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f17f008fe20 con 0x7f17f81069b0 2024-04-02T05:08:04.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.211+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f17f81a7090 con 0x7f17f81069b0 2024-04-02T05:08:04.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17f4ff9700 1 -- 172.21.15.69:0/2434691325 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f17f005e460 con 0x7f17f81069b0 2024-04-02T05:08:04.217 INFO:teuthology.orchestra.run.smithi069.stdout:34359738740 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 10 librados: watch_flush enter 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 10 librados: watch_flush exit 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 20 client.34062.objecter shutdown clearing up homeless session... 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 10 client.34062.objecter successfully canceled tick 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e006f5e0 msgr2=0x7f17e0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e006f5e0 0x7f17e0071aa0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7f17ec000c00 tx=0x7f17ec005fb0 comp rx=0 tx=0).stop 2024-04-02T05:08:04.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 msgr2=0x7f17f80fd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f80fd160 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f17f000b970 tx=0x7f17f000bc80 comp rx=0 tx=0).stop 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 shutdown_connections 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f17e006f5e0 0x7f17e0071aa0 unknown :-1 s=CLOSED pgs=4067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f81069b0 0x7f17f80fd160 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8100c40 0x7f17f80fc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 --2- 172.21.15.69:0/2434691325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f8101600 0x7f17f80fcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 >> 172.21.15.69:0/2434691325 conn(0x7f17f80782b0 msgr2=0x7f17f8103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 shutdown_connections 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 -- 172.21.15.69:0/2434691325 wait complete. 2024-04-02T05:08:04.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:04.215+0000 7f17fdc80700 1 librados: shutdown 2024-04-02T05:08:04.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:08:04 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:08:04.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:04 smithi069 bash[21528]: cluster 2024-04-02T05:08:03.012231+0000 osd.0 (osd.0) 561 : cluster 0 2.13 scrub starts 2024-04-02T05:08:04.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:04 smithi069 bash[21528]: cluster 2024-04-02T05:08:03.014096+0000 osd.0 (osd.0) 562 : cluster 0 2.13 scrub ok 2024-04-02T05:08:04.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:04 smithi069 bash[21528]: audit 2024-04-02T05:08:04.216158+0000 mon.a (mon.0) 4206 : audit 0 from='client.? 172.21.15.69:0/2434691325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T05:08:04.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:04 smithi069 bash[28904]: cluster 2024-04-02T05:08:03.012231+0000 osd.0 (osd.0) 561 : cluster 0 2.13 scrub starts 2024-04-02T05:08:04.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:04 smithi069 bash[28904]: cluster 2024-04-02T05:08:03.014096+0000 osd.0 (osd.0) 562 : cluster 0 2.13 scrub ok 2024-04-02T05:08:04.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:04 smithi069 bash[28904]: audit 2024-04-02T05:08:04.216158+0000 mon.a (mon.0) 4206 : audit 0 from='client.? 172.21.15.69:0/2434691325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T05:08:04.640 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1692217114791 got 1692217114795 for osd.7 2024-04-02T05:08:04.641 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:04.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:04 smithi098 bash[23790]: cluster 2024-04-02T05:08:03.012231+0000 osd.0 (osd.0) 561 : cluster 0 2.13 scrub starts 2024-04-02T05:08:04.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:04 smithi098 bash[23790]: cluster 2024-04-02T05:08:03.014096+0000 osd.0 (osd.0) 562 : cluster 0 2.13 scrub ok 2024-04-02T05:08:04.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:04 smithi098 bash[23790]: audit 2024-04-02T05:08:04.216158+0000 mon.a (mon.0) 4206 : audit 0 from='client.? 172.21.15.69:0/2434691325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T05:08:05.718 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:03.700292+0000 osd.1 (osd.1) 601 : cluster 0 2.a scrub starts 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:03.702729+0000 osd.1 (osd.1) 602 : cluster 0 2.a scrub ok 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:03.871854+0000 osd.6 (osd.6) 543 : cluster 0 3.a scrub starts 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:03.877285+0000 osd.6 (osd.6) 544 : cluster 0 3.a scrub ok 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: audit 2024-04-02T05:08:04.011682+0000 mgr.y (mgr.14538) 928 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.025830+0000 osd.0 (osd.0) 563 : cluster 0 3.17 scrub starts 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.028133+0000 osd.0 (osd.0) 564 : cluster 0 3.17 scrub ok 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.230831+0000 osd.1 (osd.1) 603 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.231193+0000 osd.1 (osd.1) 604 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.397740+0000 mon.a (mon.0) 4207 : cluster 0 osdmap e968: 8 total, 8 up, 8 in 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.518238+0000 mgr.y (mgr.14538) 929 : cluster 0 pgmap v1567: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.669206+0000 osd.4 (osd.4) 570 : cluster 0 4.e scrub starts 2024-04-02T05:08:05.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.672063+0000 osd.4 (osd.4) 571 : cluster 0 4.e scrub ok 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[21528]: audit 2024-04-02T05:08:04.786059+0000 mon.b (mon.1) 363 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:03.700292+0000 osd.1 (osd.1) 601 : cluster 0 2.a scrub starts 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:03.702729+0000 osd.1 (osd.1) 602 : cluster 0 2.a scrub ok 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:03.871854+0000 osd.6 (osd.6) 543 : cluster 0 3.a scrub starts 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:03.877285+0000 osd.6 (osd.6) 544 : cluster 0 3.a scrub ok 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: audit 2024-04-02T05:08:04.011682+0000 mgr.y (mgr.14538) 928 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.025830+0000 osd.0 (osd.0) 563 : cluster 0 3.17 scrub starts 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.028133+0000 osd.0 (osd.0) 564 : cluster 0 3.17 scrub ok 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.230831+0000 osd.1 (osd.1) 603 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.231193+0000 osd.1 (osd.1) 604 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:05.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.397740+0000 mon.a (mon.0) 4207 : cluster 0 osdmap e968: 8 total, 8 up, 8 in 2024-04-02T05:08:05.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.518238+0000 mgr.y (mgr.14538) 929 : cluster 0 pgmap v1567: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:05.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.669206+0000 osd.4 (osd.4) 570 : cluster 0 4.e scrub starts 2024-04-02T05:08:05.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.672063+0000 osd.4 (osd.4) 571 : cluster 0 4.e scrub ok 2024-04-02T05:08:05.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:05 smithi069 bash[28904]: audit 2024-04-02T05:08:04.786059+0000 mon.b (mon.1) 363 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:08:05.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:03.700292+0000 osd.1 (osd.1) 601 : cluster 0 2.a scrub starts 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:03.702729+0000 osd.1 (osd.1) 602 : cluster 0 2.a scrub ok 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:03.871854+0000 osd.6 (osd.6) 543 : cluster 0 3.a scrub starts 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:03.877285+0000 osd.6 (osd.6) 544 : cluster 0 3.a scrub ok 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: audit 2024-04-02T05:08:04.011682+0000 mgr.y (mgr.14538) 928 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.025830+0000 osd.0 (osd.0) 563 : cluster 0 3.17 scrub starts 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.028133+0000 osd.0 (osd.0) 564 : cluster 0 3.17 scrub ok 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.230831+0000 osd.1 (osd.1) 603 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.231193+0000 osd.1 (osd.1) 604 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.397740+0000 mon.a (mon.0) 4207 : cluster 0 osdmap e968: 8 total, 8 up, 8 in 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.518238+0000 mgr.y (mgr.14538) 929 : cluster 0 pgmap v1567: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.669206+0000 osd.4 (osd.4) 570 : cluster 0 4.e scrub starts 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.672063+0000 osd.4 (osd.4) 571 : cluster 0 4.e scrub ok 2024-04-02T05:08:05.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:05 smithi098 bash[23790]: audit 2024-04-02T05:08:04.786059+0000 mon.b (mon.1) 363 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:08:05.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103474 got 249108103479 for osd.6 2024-04-02T05:08:05.761 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:06.287 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822983 got 154618822988 for osd.4 2024-04-02T05:08:06.287 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:06.330 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738735 got 34359738740 for osd.0 2024-04-02T05:08:06.331 DEBUG:teuthology.parallel:result is None 2024-04-02T05:08:06.331 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-02T05:08:06.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T05:08:06.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:06 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.877105+0000 osd.6 (osd.6) 545 : cluster 0 10.f scrub starts 2024-04-02T05:08:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:06 smithi098 bash[23790]: cluster 2024-04-02T05:08:04.885248+0000 osd.6 (osd.6) 546 : cluster 0 10.f scrub ok 2024-04-02T05:08:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:06 smithi098 bash[23790]: cluster 2024-04-02T05:08:05.101281+0000 osd.5 (osd.5) 425 : cluster 0 2.3 scrub starts 2024-04-02T05:08:06.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:06 smithi098 bash[23790]: cluster 2024-04-02T05:08:05.103545+0000 osd.5 (osd.5) 426 : cluster 0 2.3 scrub ok 2024-04-02T05:08:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.877105+0000 osd.6 (osd.6) 545 : cluster 0 10.f scrub starts 2024-04-02T05:08:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[21528]: cluster 2024-04-02T05:08:04.885248+0000 osd.6 (osd.6) 546 : cluster 0 10.f scrub ok 2024-04-02T05:08:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[21528]: cluster 2024-04-02T05:08:05.101281+0000 osd.5 (osd.5) 425 : cluster 0 2.3 scrub starts 2024-04-02T05:08:06.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[21528]: cluster 2024-04-02T05:08:05.103545+0000 osd.5 (osd.5) 426 : cluster 0 2.3 scrub ok 2024-04-02T05:08:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.877105+0000 osd.6 (osd.6) 545 : cluster 0 10.f scrub starts 2024-04-02T05:08:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[28904]: cluster 2024-04-02T05:08:04.885248+0000 osd.6 (osd.6) 546 : cluster 0 10.f scrub ok 2024-04-02T05:08:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[28904]: cluster 2024-04-02T05:08:05.101281+0000 osd.5 (osd.5) 425 : cluster 0 2.3 scrub starts 2024-04-02T05:08:06.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:06 smithi069 bash[28904]: cluster 2024-04-02T05:08:05.103545+0000 osd.5 (osd.5) 426 : cluster 0 2.3 scrub ok 2024-04-02T05:08:07.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:05.704198+0000 osd.1 (osd.1) 605 : cluster 0 10.14 scrub starts 2024-04-02T05:08:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:05.707339+0000 osd.1 (osd.1) 606 : cluster 0 10.14 scrub ok 2024-04-02T05:08:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:05.980012+0000 osd.0 (osd.0) 565 : cluster 0 4.3 deep-scrub starts 2024-04-02T05:08:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:05.983476+0000 osd.0 (osd.0) 566 : cluster 0 4.3 deep-scrub ok 2024-04-02T05:08:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:06.120089+0000 osd.5 (osd.5) 427 : cluster 0 6.d deep-scrub starts 2024-04-02T05:08:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:06.122461+0000 osd.5 (osd.5) 428 : cluster 0 6.d deep-scrub ok 2024-04-02T05:08:07.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:07 smithi098 bash[23790]: cluster 2024-04-02T05:08:06.518665+0000 mgr.y (mgr.14538) 930 : cluster 0 pgmap v1568: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:08:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:05.704198+0000 osd.1 (osd.1) 605 : cluster 0 10.14 scrub starts 2024-04-02T05:08:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:05.707339+0000 osd.1 (osd.1) 606 : cluster 0 10.14 scrub ok 2024-04-02T05:08:07.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:05.980012+0000 osd.0 (osd.0) 565 : cluster 0 4.3 deep-scrub starts 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:05.983476+0000 osd.0 (osd.0) 566 : cluster 0 4.3 deep-scrub ok 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:06.120089+0000 osd.5 (osd.5) 427 : cluster 0 6.d deep-scrub starts 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:06.122461+0000 osd.5 (osd.5) 428 : cluster 0 6.d deep-scrub ok 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[21528]: cluster 2024-04-02T05:08:06.518665+0000 mgr.y (mgr.14538) 930 : cluster 0 pgmap v1568: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:05.704198+0000 osd.1 (osd.1) 605 : cluster 0 10.14 scrub starts 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:05.707339+0000 osd.1 (osd.1) 606 : cluster 0 10.14 scrub ok 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:05.980012+0000 osd.0 (osd.0) 565 : cluster 0 4.3 deep-scrub starts 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:05.983476+0000 osd.0 (osd.0) 566 : cluster 0 4.3 deep-scrub ok 2024-04-02T05:08:07.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:06.120089+0000 osd.5 (osd.5) 427 : cluster 0 6.d deep-scrub starts 2024-04-02T05:08:07.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:06.122461+0000 osd.5 (osd.5) 428 : cluster 0 6.d deep-scrub ok 2024-04-02T05:08:07.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:07 smithi069 bash[28904]: cluster 2024-04-02T05:08:06.518665+0000 mgr.y (mgr.14538) 930 : cluster 0 pgmap v1568: 184 pgs: 184 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:06.978214+0000 osd.0 (osd.0) 567 : cluster 0 6.10 scrub starts 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:06.980709+0000 osd.0 (osd.0) 568 : cluster 0 6.10 scrub ok 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.158006+0000 osd.5 (osd.5) 429 : cluster 0 10.6 deep-scrub starts 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.161618+0000 osd.5 (osd.5) 430 : cluster 0 10.6 deep-scrub ok 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.406131+0000 osd.3 (osd.3) 629 : cluster 0 1.4 scrub starts 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.408621+0000 osd.3 (osd.3) 630 : cluster 0 1.4 scrub ok 2024-04-02T05:08:08.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.666084+0000 osd.4 (osd.4) 572 : cluster 0 4.1a deep-scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.668658+0000 osd.4 (osd.4) 573 : cluster 0 4.1a deep-scrub ok 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.691150+0000 osd.1 (osd.1) 607 : cluster 0 1.13 scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21528]: cluster 2024-04-02T05:08:07.693600+0000 osd.1 (osd.1) 608 : cluster 0 1.13 scrub ok 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:06.978214+0000 osd.0 (osd.0) 567 : cluster 0 6.10 scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:06.980709+0000 osd.0 (osd.0) 568 : cluster 0 6.10 scrub ok 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.158006+0000 osd.5 (osd.5) 429 : cluster 0 10.6 deep-scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.161618+0000 osd.5 (osd.5) 430 : cluster 0 10.6 deep-scrub ok 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.406131+0000 osd.3 (osd.3) 629 : cluster 0 1.4 scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.408621+0000 osd.3 (osd.3) 630 : cluster 0 1.4 scrub ok 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.666084+0000 osd.4 (osd.4) 572 : cluster 0 4.1a deep-scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.668658+0000 osd.4 (osd.4) 573 : cluster 0 4.1a deep-scrub ok 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.691150+0000 osd.1 (osd.1) 607 : cluster 0 1.13 scrub starts 2024-04-02T05:08:08.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[28904]: cluster 2024-04-02T05:08:07.693600+0000 osd.1 (osd.1) 608 : cluster 0 1.13 scrub ok 2024-04-02T05:08:08.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:06.978214+0000 osd.0 (osd.0) 567 : cluster 0 6.10 scrub starts 2024-04-02T05:08:08.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:06.980709+0000 osd.0 (osd.0) 568 : cluster 0 6.10 scrub ok 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.158006+0000 osd.5 (osd.5) 429 : cluster 0 10.6 deep-scrub starts 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.161618+0000 osd.5 (osd.5) 430 : cluster 0 10.6 deep-scrub ok 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.406131+0000 osd.3 (osd.3) 629 : cluster 0 1.4 scrub starts 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.408621+0000 osd.3 (osd.3) 630 : cluster 0 1.4 scrub ok 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.666084+0000 osd.4 (osd.4) 572 : cluster 0 4.1a deep-scrub starts 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.668658+0000 osd.4 (osd.4) 573 : cluster 0 4.1a deep-scrub ok 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.691150+0000 osd.1 (osd.1) 607 : cluster 0 1.13 scrub starts 2024-04-02T05:08:08.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:08 smithi098 bash[23790]: cluster 2024-04-02T05:08:07.693600+0000 osd.1 (osd.1) 608 : cluster 0 1.13 scrub ok 2024-04-02T05:08:09.033 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:08 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:08:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:08:09.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.369779+0000 osd.3 (osd.3) 631 : cluster 0 3.18 scrub starts 2024-04-02T05:08:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.374191+0000 osd.3 (osd.3) 632 : cluster 0 3.18 scrub ok 2024-04-02T05:08:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.520452+0000 mgr.y (mgr.14538) 931 : cluster 0 pgmap v1569: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.532357+0000 osd.7 (osd.7) 401 : cluster 0 6.12 scrub starts 2024-04-02T05:08:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.534704+0000 osd.7 (osd.7) 402 : cluster 0 6.12 scrub ok 2024-04-02T05:08:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.689666+0000 osd.4 (osd.4) 574 : cluster 0 6.2 scrub starts 2024-04-02T05:08:09.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:09 smithi098 bash[23790]: cluster 2024-04-02T05:08:08.692064+0000 osd.4 (osd.4) 575 : cluster 0 6.2 scrub ok 2024-04-02T05:08:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.369779+0000 osd.3 (osd.3) 631 : cluster 0 3.18 scrub starts 2024-04-02T05:08:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.374191+0000 osd.3 (osd.3) 632 : cluster 0 3.18 scrub ok 2024-04-02T05:08:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.520452+0000 mgr.y (mgr.14538) 931 : cluster 0 pgmap v1569: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.532357+0000 osd.7 (osd.7) 401 : cluster 0 6.12 scrub starts 2024-04-02T05:08:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.534704+0000 osd.7 (osd.7) 402 : cluster 0 6.12 scrub ok 2024-04-02T05:08:09.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.689666+0000 osd.4 (osd.4) 574 : cluster 0 6.2 scrub starts 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[21528]: cluster 2024-04-02T05:08:08.692064+0000 osd.4 (osd.4) 575 : cluster 0 6.2 scrub ok 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.369779+0000 osd.3 (osd.3) 631 : cluster 0 3.18 scrub starts 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.374191+0000 osd.3 (osd.3) 632 : cluster 0 3.18 scrub ok 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.520452+0000 mgr.y (mgr.14538) 931 : cluster 0 pgmap v1569: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.532357+0000 osd.7 (osd.7) 401 : cluster 0 6.12 scrub starts 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.534704+0000 osd.7 (osd.7) 402 : cluster 0 6.12 scrub ok 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.689666+0000 osd.4 (osd.4) 574 : cluster 0 6.2 scrub starts 2024-04-02T05:08:09.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:09 smithi069 bash[28904]: cluster 2024-04-02T05:08:08.692064+0000 osd.4 (osd.4) 575 : cluster 0 6.2 scrub ok 2024-04-02T05:08:10.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:10 smithi098 bash[23790]: cluster 2024-04-02T05:08:09.527158+0000 osd.7 (osd.7) 403 : cluster 0 6.1c scrub starts 2024-04-02T05:08:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:10 smithi098 bash[23790]: cluster 2024-04-02T05:08:09.530118+0000 osd.7 (osd.7) 404 : cluster 0 6.1c scrub ok 2024-04-02T05:08:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:10 smithi098 bash[23790]: cluster 2024-04-02T05:08:09.662586+0000 osd.1 (osd.1) 609 : cluster 0 4.4 scrub starts 2024-04-02T05:08:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:10 smithi098 bash[23790]: cluster 2024-04-02T05:08:09.665288+0000 osd.1 (osd.1) 610 : cluster 0 4.4 scrub ok 2024-04-02T05:08:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:10 smithi098 bash[23790]: cluster 2024-04-02T05:08:09.730382+0000 osd.4 (osd.4) 576 : cluster 0 4.d scrub starts 2024-04-02T05:08:10.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:10 smithi098 bash[23790]: cluster 2024-04-02T05:08:09.733342+0000 osd.4 (osd.4) 577 : cluster 0 4.d scrub ok 2024-04-02T05:08:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[21528]: cluster 2024-04-02T05:08:09.527158+0000 osd.7 (osd.7) 403 : cluster 0 6.1c scrub starts 2024-04-02T05:08:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[21528]: cluster 2024-04-02T05:08:09.530118+0000 osd.7 (osd.7) 404 : cluster 0 6.1c scrub ok 2024-04-02T05:08:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[21528]: cluster 2024-04-02T05:08:09.662586+0000 osd.1 (osd.1) 609 : cluster 0 4.4 scrub starts 2024-04-02T05:08:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[21528]: cluster 2024-04-02T05:08:09.665288+0000 osd.1 (osd.1) 610 : cluster 0 4.4 scrub ok 2024-04-02T05:08:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[21528]: cluster 2024-04-02T05:08:09.730382+0000 osd.4 (osd.4) 576 : cluster 0 4.d scrub starts 2024-04-02T05:08:10.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[21528]: cluster 2024-04-02T05:08:09.733342+0000 osd.4 (osd.4) 577 : cluster 0 4.d scrub ok 2024-04-02T05:08:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[28904]: cluster 2024-04-02T05:08:09.527158+0000 osd.7 (osd.7) 403 : cluster 0 6.1c scrub starts 2024-04-02T05:08:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[28904]: cluster 2024-04-02T05:08:09.530118+0000 osd.7 (osd.7) 404 : cluster 0 6.1c scrub ok 2024-04-02T05:08:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[28904]: cluster 2024-04-02T05:08:09.662586+0000 osd.1 (osd.1) 609 : cluster 0 4.4 scrub starts 2024-04-02T05:08:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[28904]: cluster 2024-04-02T05:08:09.665288+0000 osd.1 (osd.1) 610 : cluster 0 4.4 scrub ok 2024-04-02T05:08:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[28904]: cluster 2024-04-02T05:08:09.730382+0000 osd.4 (osd.4) 576 : cluster 0 4.d scrub starts 2024-04-02T05:08:10.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:10 smithi069 bash[28904]: cluster 2024-04-02T05:08:09.733342+0000 osd.4 (osd.4) 577 : cluster 0 4.d scrub ok 2024-04-02T05:08:11.113 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:08:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:11 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.448957+0000 osd.3 (osd.3) 633 : cluster 0 10.15 scrub starts 2024-04-02T05:08:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:11 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.452994+0000 osd.3 (osd.3) 634 : cluster 0 10.15 scrub ok 2024-04-02T05:08:11.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:11 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.521502+0000 mgr.y (mgr.14538) 932 : cluster 0 pgmap v1570: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:11 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.567486+0000 osd.7 (osd.7) 405 : cluster 0 6.5 scrub starts 2024-04-02T05:08:11.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:11 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.569798+0000 osd.7 (osd.7) 406 : cluster 0 6.5 scrub ok 2024-04-02T05:08:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.448957+0000 osd.3 (osd.3) 633 : cluster 0 10.15 scrub starts 2024-04-02T05:08:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.452994+0000 osd.3 (osd.3) 634 : cluster 0 10.15 scrub ok 2024-04-02T05:08:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.521502+0000 mgr.y (mgr.14538) 932 : cluster 0 pgmap v1570: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.567486+0000 osd.7 (osd.7) 405 : cluster 0 6.5 scrub starts 2024-04-02T05:08:11.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.569798+0000 osd.7 (osd.7) 406 : cluster 0 6.5 scrub ok 2024-04-02T05:08:11.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.448957+0000 osd.3 (osd.3) 633 : cluster 0 10.15 scrub starts 2024-04-02T05:08:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.452994+0000 osd.3 (osd.3) 634 : cluster 0 10.15 scrub ok 2024-04-02T05:08:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.521502+0000 mgr.y (mgr.14538) 932 : cluster 0 pgmap v1570: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.567486+0000 osd.7 (osd.7) 405 : cluster 0 6.5 scrub starts 2024-04-02T05:08:11.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:11 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.569798+0000 osd.7 (osd.7) 406 : cluster 0 6.5 scrub ok 2024-04-02T05:08:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.767237+0000 osd.4 (osd.4) 578 : cluster 0 2.f scrub starts 2024-04-02T05:08:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:10.770378+0000 osd.4 (osd.4) 579 : cluster 0 2.f scrub ok 2024-04-02T05:08:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:11.430674+0000 osd.3 (osd.3) 635 : cluster 0 4.17 scrub starts 2024-04-02T05:08:12.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:11.432965+0000 osd.3 (osd.3) 636 : cluster 0 4.17 scrub ok 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:11.604097+0000 osd.7 (osd.7) 407 : cluster 0 2.8 scrub starts 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:11.606555+0000 osd.7 (osd.7) 408 : cluster 0 2.8 scrub ok 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:11.638637+0000 osd.1 (osd.1) 611 : cluster 0 1.b scrub starts 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[21528]: cluster 2024-04-02T05:08:11.640878+0000 osd.1 (osd.1) 612 : cluster 0 1.b scrub ok 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.767237+0000 osd.4 (osd.4) 578 : cluster 0 2.f scrub starts 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:10.770378+0000 osd.4 (osd.4) 579 : cluster 0 2.f scrub ok 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:11.430674+0000 osd.3 (osd.3) 635 : cluster 0 4.17 scrub starts 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:11.432965+0000 osd.3 (osd.3) 636 : cluster 0 4.17 scrub ok 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:11.604097+0000 osd.7 (osd.7) 407 : cluster 0 2.8 scrub starts 2024-04-02T05:08:12.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:11.606555+0000 osd.7 (osd.7) 408 : cluster 0 2.8 scrub ok 2024-04-02T05:08:12.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:11.638637+0000 osd.1 (osd.1) 611 : cluster 0 1.b scrub starts 2024-04-02T05:08:12.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:12 smithi069 bash[28904]: cluster 2024-04-02T05:08:11.640878+0000 osd.1 (osd.1) 612 : cluster 0 1.b scrub ok 2024-04-02T05:08:12.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.767237+0000 osd.4 (osd.4) 578 : cluster 0 2.f scrub starts 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:10.770378+0000 osd.4 (osd.4) 579 : cluster 0 2.f scrub ok 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:11.430674+0000 osd.3 (osd.3) 635 : cluster 0 4.17 scrub starts 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:11.432965+0000 osd.3 (osd.3) 636 : cluster 0 4.17 scrub ok 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:11.604097+0000 osd.7 (osd.7) 407 : cluster 0 2.8 scrub starts 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:11.606555+0000 osd.7 (osd.7) 408 : cluster 0 2.8 scrub ok 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:11.638637+0000 osd.1 (osd.1) 611 : cluster 0 1.b scrub starts 2024-04-02T05:08:12.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:12 smithi098 bash[23790]: cluster 2024-04-02T05:08:11.640878+0000 osd.1 (osd.1) 612 : cluster 0 1.b scrub ok 2024-04-02T05:08:13.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:13 smithi098 bash[23790]: cluster 2024-04-02T05:08:12.522260+0000 mgr.y (mgr.14538) 933 : cluster 0 pgmap v1571: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:08:13.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:13 smithi069 bash[21528]: cluster 2024-04-02T05:08:12.522260+0000 mgr.y (mgr.14538) 933 : cluster 0 pgmap v1571: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:08:13.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:13 smithi069 bash[28904]: cluster 2024-04-02T05:08:12.522260+0000 mgr.y (mgr.14538) 933 : cluster 0 pgmap v1571: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2106258423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a4104e20 msgr2=0x7f95a4111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2106258423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a4104e20 0x7f95a4111330 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f959c00b0a0 tx=0x7f959c00b3b0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2106258423 shutdown_connections 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2106258423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a4104e20 0x7f95a4111330 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2106258423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a410a4f0 0x7f95a410a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2106258423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104460 0x7f95a41048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2106258423 >> 172.21.15.69:0/2106258423 conn(0x7f95a40fff50 msgr2=0x7f95a4102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2106258423 shutdown_connections 2024-04-02T05:08:14.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2106258423 wait complete. 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 librados: starting msgr at 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 librados: starting objecter 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 Processor -- start 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- start start 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 librados: setting wanted keys 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 librados: calling monclient init 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 0x7f95a419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:14.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104e20 0x7f95a419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a410a4f0 0x7f95a419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95a4113c70 con 0x7f95a4104460 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95a4113af0 con 0x7f95a4104e20 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f95a4113df0 con 0x7f95a410a4f0 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 0x7f95a419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 0x7f95a419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50764/0 (socket says 172.21.15.69:50764) 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 -- 172.21.15.69:0/2249876382 learned_addr learned my addr 172.21.15.69:0/2249876382 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a2d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a410a4f0 0x7f95a419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 -- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a410a4f0 msgr2=0x7f95a419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a410a4f0 0x7f95a419ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 -- 172.21.15.69:0/2249876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104e20 msgr2=0x7f95a419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a1d9b700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104e20 0x7f95a419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104e20 0x7f95a419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 -- 172.21.15.69:0/2249876382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a419f1b0 con 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a2d9d700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a410a4f0 0x7f95a419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a1d9b700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104e20 0x7f95a419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a259c700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 0x7f95a419a1a0 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f958c00d980 tx=0x7f958c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f959b7fe700 10 client.?.objecter ms_handle_connect 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f959b7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c016070 con 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a419f4a0 con 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f958c004830 con 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c0053c0 con 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 10 client.34068.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a41a6d20 con 0x7f95a4104460 2024-04-02T05:08:14.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.175+0000 7f95a8f19700 1 librados: init done 2024-04-02T05:08:14.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a404ff00 con 0x7f95a4104460 2024-04-02T05:08:14.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f958c005560 con 0x7f95a4104460 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f959006f5e0 0x7f9590071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(968..968 src has 918..968) ==== 9015+0+0 (secure 0 0 0) 0x7f958c08e3f0 con 0x7f95a4104460 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 10 client.34068.objecter ms_dispatch 0x7f95a4001550 osd_map(968..968 src has 918..968) 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 3 client.34068.objecter handle_osd_map got epochs [968,968] > 0 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 3 client.34068.objecter handle_osd_map decoding full epoch 968 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f95a1d9b700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f959006f5e0 0x7f9590071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 20 client.34068.objecter dump_active .. 0 homeless 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f95a1d9b700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f959006f5e0 0x7f9590071aa0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 crypto rx=0x7f9594005fd0 tx=0x7f9594005dc0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:14.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.179+0000 7f959b7fe700 10 client.34068.objecter ms_handle_connect 0x7f959006f5e0 2024-04-02T05:08:14.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.183+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f958c01f020 con 0x7f95a4104460 2024-04-02T05:08:14.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.391+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95a4100a40 con 0x7f959006f5e0 2024-04-02T05:08:14.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.439+0000 7f959b7fe700 1 -- 172.21.15.69:0/2249876382 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+478396 (secure 0 0 0) 0x7f95a4100a40 con 0x7f959006f5e0 2024-04-02T05:08:14.449 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T05:08:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 10 librados: watch_flush enter 2024-04-02T05:08:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 10 librados: watch_flush exit 2024-04-02T05:08:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 20 client.34068.objecter shutdown clearing up homeless session... 2024-04-02T05:08:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 10 client.34068.objecter successfully canceled tick 2024-04-02T05:08:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f959006f5e0 msgr2=0x7f9590071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f959006f5e0 0x7f9590071aa0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 crypto rx=0x7f9594005fd0 tx=0x7f9594005dc0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 msgr2=0x7f95a419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 0x7f95a419a1a0 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f958c00d980 tx=0x7f958c00dc90 comp rx=0 tx=0).stop 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 shutdown_connections 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f959006f5e0 0x7f9590071aa0 unknown :-1 s=CLOSED pgs=4068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95a410a4f0 0x7f95a419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a4104460 0x7f95a419a1a0 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 --2- 172.21.15.69:0/2249876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a4104e20 0x7f95a419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 >> 172.21.15.69:0/2249876382 conn(0x7f95a40fff50 msgr2=0x7f95a4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 shutdown_connections 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 -- 172.21.15.69:0/2249876382 wait complete. 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:14.443+0000 7f95a8f19700 1 librados: shutdown 2024-04-02T05:08:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T05:08:14.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:08:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[23790]: cluster 2024-04-02T05:08:12.879289+0000 osd.6 (osd.6) 547 : cluster 0 5.2 scrub starts 2024-04-02T05:08:14.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[23790]: cluster 2024-04-02T05:08:12.882517+0000 osd.6 (osd.6) 548 : cluster 0 5.2 scrub ok 2024-04-02T05:08:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[23790]: cluster 2024-04-02T05:08:13.034770+0000 osd.0 (osd.0) 569 : cluster 0 3.6 scrub starts 2024-04-02T05:08:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[23790]: cluster 2024-04-02T05:08:13.038953+0000 osd.0 (osd.0) 570 : cluster 0 3.6 scrub ok 2024-04-02T05:08:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[23790]: cluster 2024-04-02T05:08:13.690604+0000 osd.7 (osd.7) 409 : cluster 0 1.9 scrub starts 2024-04-02T05:08:14.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:14 smithi098 bash[23790]: cluster 2024-04-02T05:08:13.693343+0000 osd.7 (osd.7) 410 : cluster 0 1.9 scrub ok 2024-04-02T05:08:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[21528]: cluster 2024-04-02T05:08:12.879289+0000 osd.6 (osd.6) 547 : cluster 0 5.2 scrub starts 2024-04-02T05:08:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[21528]: cluster 2024-04-02T05:08:12.882517+0000 osd.6 (osd.6) 548 : cluster 0 5.2 scrub ok 2024-04-02T05:08:14.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[21528]: cluster 2024-04-02T05:08:13.034770+0000 osd.0 (osd.0) 569 : cluster 0 3.6 scrub starts 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[21528]: cluster 2024-04-02T05:08:13.038953+0000 osd.0 (osd.0) 570 : cluster 0 3.6 scrub ok 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[21528]: cluster 2024-04-02T05:08:13.690604+0000 osd.7 (osd.7) 409 : cluster 0 1.9 scrub starts 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[21528]: cluster 2024-04-02T05:08:13.693343+0000 osd.7 (osd.7) 410 : cluster 0 1.9 scrub ok 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[28904]: cluster 2024-04-02T05:08:12.879289+0000 osd.6 (osd.6) 547 : cluster 0 5.2 scrub starts 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[28904]: cluster 2024-04-02T05:08:12.882517+0000 osd.6 (osd.6) 548 : cluster 0 5.2 scrub ok 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[28904]: cluster 2024-04-02T05:08:13.034770+0000 osd.0 (osd.0) 569 : cluster 0 3.6 scrub starts 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[28904]: cluster 2024-04-02T05:08:13.038953+0000 osd.0 (osd.0) 570 : cluster 0 3.6 scrub ok 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[28904]: cluster 2024-04-02T05:08:13.690604+0000 osd.7 (osd.7) 409 : cluster 0 1.9 scrub starts 2024-04-02T05:08:14.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:14 smithi069 bash[28904]: cluster 2024-04-02T05:08:13.693343+0000 osd.7 (osd.7) 410 : cluster 0 1.9 scrub ok 2024-04-02T05:08:15.231 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2024-04-02T05:08:12.521774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":2401,"num_read_kb":2121,"num_write":1207,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":1062645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4142,"ondisk_log_size":4142,"up":552,"acting":552,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":552,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":764468,"kb_used_data":21096,"kb_used_omap":68,"kb_used_meta":743227,"kb_avail":749196748,"statfs":{"total":767960285184,"available":767177469952,"internally_reserved":0,"allocated":21602304,"data_stored":7550617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69644,"internal_metadata":761065460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004508"},"pg_stats":[{"pgid":"1.d","version":"101'128","reported_seq":1336,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006305+0000","last_change":"2024-04-02T05:07:14.552491+0000","last_active":"2024-04-02T05:08:05.006305+0000","last_peered":"2024-04-02T05:08:05.006305+0000","last_clean":"2024-04-02T05:08:05.006305+0000","last_became_active":"2024-04-02T05:05:57.346674+0000","last_became_peered":"2024-04-02T05:05:57.346674+0000","last_unstale":"2024-04-02T05:08:05.006305+0000","last_undegraded":"2024-04-02T05:08:05.006305+0000","last_fullsized":"2024-04-02T05:08:05.006305+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:14.552260+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:14.552260+0000","last_clean_scrub_stamp":"2024-04-02T05:07:14.552260+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:42.782879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.161784+0000","last_change":"2024-04-02T05:08:07.161848+0000","last_active":"2024-04-02T05:08:07.161784+0000","last_peered":"2024-04-02T05:08:07.161784+0000","last_clean":"2024-04-02T05:08:07.161784+0000","last_became_active":"2024-04-02T05:04:11.245164+0000","last_became_peered":"2024-04-02T05:04:11.245164+0000","last_unstale":"2024-04-02T05:08:07.161784+0000","last_undegraded":"2024-04-02T05:08:07.161784+0000","last_fullsized":"2024-04-02T05:08:07.161784+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:07.161628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:07.161628+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.161628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.731217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003006+0000","last_change":"2024-04-02T05:07:26.213276+0000","last_active":"2024-04-02T05:08:05.003006+0000","last_peered":"2024-04-02T05:08:05.003006+0000","last_clean":"2024-04-02T05:08:05.003006+0000","last_became_active":"2024-04-02T05:04:11.244828+0000","last_became_peered":"2024-04-02T05:04:11.244828+0000","last_unstale":"2024-04-02T05:08:05.003006+0000","last_undegraded":"2024-04-02T05:08:05.003006+0000","last_fullsized":"2024-04-02T05:08:05.003006+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:26.213058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:48.966242+0000","last_clean_scrub_stamp":"2024-04-02T05:07:26.213058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.485877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"950'46","reported_seq":1424,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007467+0000","last_change":"2024-04-02T05:07:45.443583+0000","last_active":"2024-04-02T05:08:05.007467+0000","last_peered":"2024-04-02T05:08:05.007467+0000","last_clean":"2024-04-02T05:08:05.007467+0000","last_became_active":"2024-04-02T04:59:55.068785+0000","last_became_peered":"2024-04-02T04:59:55.068785+0000","last_unstale":"2024-04-02T05:08:05.007467+0000","last_undegraded":"2024-04-02T05:08:05.007467+0000","last_fullsized":"2024-04-02T05:08:05.007467+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'46","last_scrub_stamp":"2024-04-02T05:07:45.443316+0000","last_deep_scrub":"950'46","last_deep_scrub_stamp":"2024-04-02T05:06:20.505182+0000","last_clean_scrub_stamp":"2024-04-02T05:07:45.443316+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.045719+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871333+0000","last_change":"2024-04-02T05:07:52.822951+0000","last_active":"2024-04-02T05:08:04.871333+0000","last_peered":"2024-04-02T05:08:04.871333+0000","last_clean":"2024-04-02T05:08:04.871333+0000","last_became_active":"2024-04-02T05:05:16.544433+0000","last_became_peered":"2024-04-02T05:05:16.544433+0000","last_unstale":"2024-04-02T05:08:04.871333+0000","last_undegraded":"2024-04-02T05:08:04.871333+0000","last_fullsized":"2024-04-02T05:08:04.871333+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:52.822725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:52.822725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.887431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004187+0000","last_change":"2024-04-02T05:07:45.119037+0000","last_active":"2024-04-02T05:08:05.004187+0000","last_peered":"2024-04-02T05:08:05.004187+0000","last_clean":"2024-04-02T05:08:05.004187+0000","last_became_active":"2024-04-02T05:04:11.260461+0000","last_became_peered":"2024-04-02T05:04:11.260461+0000","last_unstale":"2024-04-02T05:08:05.004187+0000","last_undegraded":"2024-04-02T05:08:05.004187+0000","last_fullsized":"2024-04-02T05:08:05.004187+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:45.118824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_clean_scrub_stamp":"2024-04-02T05:07:45.118824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.778332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"101'128","reported_seq":879,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397635+0000","last_change":"2024-04-02T05:07:33.577003+0000","last_active":"2024-04-02T05:08:04.397635+0000","last_peered":"2024-04-02T05:08:04.397635+0000","last_clean":"2024-04-02T05:08:04.397635+0000","last_became_active":"2024-04-02T05:05:57.327887+0000","last_became_peered":"2024-04-02T05:05:57.327887+0000","last_unstale":"2024-04-02T05:08:04.397635+0000","last_undegraded":"2024-04-02T05:08:04.397635+0000","last_fullsized":"2024-04-02T05:08:04.397635+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:33.576764+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:33.576764+0000","last_clean_scrub_stamp":"2024-04-02T05:07:33.576764+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:50.402444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1753,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.873697+0000","last_change":"2024-04-02T05:07:16.761520+0000","last_active":"2024-04-02T05:08:04.873697+0000","last_peered":"2024-04-02T05:08:04.873697+0000","last_clean":"2024-04-02T05:08:04.873697+0000","last_became_active":"2024-04-02T04:59:00.080466+0000","last_became_peered":"2024-04-02T04:59:00.080466+0000","last_unstale":"2024-04-02T05:08:04.873697+0000","last_undegraded":"2024-04-02T05:08:04.873697+0000","last_fullsized":"2024-04-02T05:08:04.873697+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.761302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:44.775660+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.761302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:44.143566+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1868,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073581+0000","last_change":"2024-04-02T05:07:19.888171+0000","last_active":"2024-04-02T05:08:05.073581+0000","last_peered":"2024-04-02T05:08:05.073581+0000","last_clean":"2024-04-02T05:08:05.073581+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T05:08:05.073581+0000","last_undegraded":"2024-04-02T05:08:05.073581+0000","last_fullsized":"2024-04-02T05:08:05.073581+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:19.887957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:36.528990+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.887957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.325679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"899'35","reported_seq":180,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003805+0000","last_change":"2024-04-02T05:07:19.268579+0000","last_active":"2024-04-02T05:08:05.003805+0000","last_peered":"2024-04-02T05:08:05.003805+0000","last_clean":"2024-04-02T05:08:05.003805+0000","last_became_active":"2024-04-02T05:04:11.261777+0000","last_became_peered":"2024-04-02T05:04:11.261777+0000","last_unstale":"2024-04-02T05:08:05.003805+0000","last_undegraded":"2024-04-02T05:08:05.003805+0000","last_fullsized":"2024-04-02T05:08:05.003805+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'35","last_scrub_stamp":"2024-04-02T05:07:19.268333+0000","last_deep_scrub":"899'35","last_deep_scrub_stamp":"2024-04-02T05:05:43.878796+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.268333+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.552262+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1921,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401333+0000","last_change":"2024-04-02T05:07:55.728643+0000","last_active":"2024-04-02T05:08:04.401333+0000","last_peered":"2024-04-02T05:08:04.401333+0000","last_clean":"2024-04-02T05:08:04.401333+0000","last_became_active":"2024-04-02T05:05:54.274933+0000","last_became_peered":"2024-04-02T05:05:54.274933+0000","last_unstale":"2024-04-02T05:08:04.401333+0000","last_undegraded":"2024-04-02T05:08:04.401333+0000","last_fullsized":"2024-04-02T05:08:04.401333+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:55.728437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.728437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.735484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1709,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.072786+0000","last_change":"2024-04-02T05:07:55.085752+0000","last_active":"2024-04-02T05:08:05.072786+0000","last_peered":"2024-04-02T05:08:05.072786+0000","last_clean":"2024-04-02T05:08:05.072786+0000","last_became_active":"2024-04-02T05:05:16.537209+0000","last_became_peered":"2024-04-02T05:05:16.537209+0000","last_unstale":"2024-04-02T05:08:05.072786+0000","last_undegraded":"2024-04-02T05:08:05.072786+0000","last_fullsized":"2024-04-02T05:08:05.072786+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:55.085446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.085446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:59.568285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"101'128","reported_seq":1420,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667824+0000","last_change":"2024-04-02T05:07:36.752971+0000","last_active":"2024-04-02T05:08:04.667824+0000","last_peered":"2024-04-02T05:08:04.667824+0000","last_clean":"2024-04-02T05:08:04.667824+0000","last_became_active":"2024-04-02T05:05:57.340555+0000","last_became_peered":"2024-04-02T05:05:57.340555+0000","last_unstale":"2024-04-02T05:08:04.667824+0000","last_undegraded":"2024-04-02T05:08:04.667824+0000","last_fullsized":"2024-04-02T05:08:04.667824+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:36.752818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:07:36.752818+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:59.269988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1738,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.072577+0000","last_change":"2024-04-02T05:06:51.023980+0000","last_active":"2024-04-02T05:08:05.072577+0000","last_peered":"2024-04-02T05:08:05.072577+0000","last_clean":"2024-04-02T05:08:05.072577+0000","last_became_active":"2024-04-02T04:59:00.102255+0000","last_became_peered":"2024-04-02T04:59:00.102255+0000","last_unstale":"2024-04-02T05:08:05.072577+0000","last_undegraded":"2024-04-02T05:08:05.072577+0000","last_fullsized":"2024-04-02T05:08:05.072577+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:51.023733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:51.023733+0000","last_clean_scrub_stamp":"2024-04-02T05:06:51.023733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:16.232899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1866,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667103+0000","last_change":"2024-04-02T05:07:51.878887+0000","last_active":"2024-04-02T05:08:04.667103+0000","last_peered":"2024-04-02T05:08:04.667103+0000","last_clean":"2024-04-02T05:08:04.667103+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T05:08:04.667103+0000","last_undegraded":"2024-04-02T05:08:04.667103+0000","last_fullsized":"2024-04-02T05:08:04.667103+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:51.878683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:53.441303+0000","last_clean_scrub_stamp":"2024-04-02T05:07:51.878683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:10.663762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"899'45","reported_seq":1506,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007145+0000","last_change":"2024-04-02T05:07:42.454887+0000","last_active":"2024-04-02T05:08:05.007145+0000","last_peered":"2024-04-02T05:08:05.007145+0000","last_clean":"2024-04-02T05:08:05.007145+0000","last_became_active":"2024-04-02T05:04:11.240495+0000","last_became_peered":"2024-04-02T05:04:11.240495+0000","last_unstale":"2024-04-02T05:08:05.007145+0000","last_undegraded":"2024-04-02T05:08:05.007145+0000","last_fullsized":"2024-04-02T05:08:05.007145+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'45","last_scrub_stamp":"2024-04-02T05:07:42.454640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.454640+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:50.274109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.872686+0000","last_change":"2024-04-02T05:07:19.816884+0000","last_active":"2024-04-02T05:08:04.872686+0000","last_peered":"2024-04-02T05:08:04.872686+0000","last_clean":"2024-04-02T05:08:04.872686+0000","last_became_active":"2024-04-02T05:05:16.549444+0000","last_became_peered":"2024-04-02T05:05:16.549444+0000","last_unstale":"2024-04-02T05:08:04.872686+0000","last_undegraded":"2024-04-02T05:08:04.872686+0000","last_fullsized":"2024-04-02T05:08:04.872686+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:19.816675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.816675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.424433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1364,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.001601+0000","last_change":"2024-04-02T05:07:10.540148+0000","last_active":"2024-04-02T05:08:05.001601+0000","last_peered":"2024-04-02T05:08:05.001601+0000","last_clean":"2024-04-02T05:08:05.001601+0000","last_became_active":"2024-04-02T05:05:16.545371+0000","last_became_peered":"2024-04-02T05:05:16.545371+0000","last_unstale":"2024-04-02T05:08:05.001601+0000","last_undegraded":"2024-04-02T05:08:05.001601+0000","last_fullsized":"2024-04-02T05:08:05.001601+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:10.539875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:10.539875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:25.735993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"956'129","reported_seq":1886,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003270+0000","last_change":"2024-04-02T05:07:55.210371+0000","last_active":"2024-04-02T05:08:05.003270+0000","last_peered":"2024-04-02T05:08:05.003270+0000","last_clean":"2024-04-02T05:08:05.003270+0000","last_became_active":"2024-04-02T05:04:14.200867+0000","last_became_peered":"2024-04-02T05:04:14.200867+0000","last_unstale":"2024-04-02T05:08:05.003270+0000","last_undegraded":"2024-04-02T05:08:05.003270+0000","last_fullsized":"2024-04-02T05:08:05.003270+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":4,"last_scrub":"956'129","last_scrub_stamp":"2024-04-02T05:07:55.210035+0000","last_deep_scrub":"956'129","last_deep_scrub_stamp":"2024-04-02T05:07:55.210035+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.210035+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.341579+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":126,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870748+0000","last_change":"2024-04-02T05:07:54.859012+0000","last_active":"2024-04-02T05:08:04.870748+0000","last_peered":"2024-04-02T05:08:04.870748+0000","last_clean":"2024-04-02T05:08:04.870748+0000","last_became_active":"2024-04-02T05:05:16.556262+0000","last_became_peered":"2024-04-02T05:05:16.556262+0000","last_unstale":"2024-04-02T05:08:04.870748+0000","last_undegraded":"2024-04-02T05:08:04.870748+0000","last_fullsized":"2024-04-02T05:08:04.870748+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.858904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:37.743671+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.858904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:21.788379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"899'68","reported_seq":2027,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667791+0000","last_change":"2024-04-02T05:08:03.877564+0000","last_active":"2024-04-02T05:08:04.667791+0000","last_peered":"2024-04-02T05:08:04.667791+0000","last_clean":"2024-04-02T05:08:04.667791+0000","last_became_active":"2024-04-02T04:58:39.391176+0000","last_became_peered":"2024-04-02T04:58:39.391176+0000","last_unstale":"2024-04-02T05:08:04.667791+0000","last_undegraded":"2024-04-02T05:08:04.667791+0000","last_fullsized":"2024-04-02T05:08:04.667791+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'68","last_scrub_stamp":"2024-04-02T05:08:03.877297+0000","last_deep_scrub":"531'52","last_deep_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_clean_scrub_stamp":"2024-04-02T05:08:03.877297+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:31.505482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1382,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005914+0000","last_change":"2024-04-02T05:07:40.513562+0000","last_active":"2024-04-02T05:08:05.005914+0000","last_peered":"2024-04-02T05:08:05.005914+0000","last_clean":"2024-04-02T05:08:05.005914+0000","last_became_active":"2024-04-02T05:05:54.268662+0000","last_became_peered":"2024-04-02T05:05:54.268662+0000","last_unstale":"2024-04-02T05:08:05.005914+0000","last_undegraded":"2024-04-02T05:08:05.005914+0000","last_fullsized":"2024-04-02T05:08:05.005914+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:40.513353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:54.596224+0000","last_clean_scrub_stamp":"2024-04-02T05:07:40.513353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.669891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"942'17","reported_seq":419,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398395+0000","last_change":"2024-04-02T05:07:04.516711+0000","last_active":"2024-04-02T05:08:04.398395+0000","last_peered":"2024-04-02T05:08:04.398395+0000","last_clean":"2024-04-02T05:08:04.398395+0000","last_became_active":"2024-04-02T05:05:54.274960+0000","last_became_peered":"2024-04-02T05:05:54.274960+0000","last_unstale":"2024-04-02T05:08:04.398395+0000","last_undegraded":"2024-04-02T05:08:04.398395+0000","last_fullsized":"2024-04-02T05:08:04.398395+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'17","last_scrub_stamp":"2024-04-02T05:07:04.516467+0000","last_deep_scrub":"662'13","last_deep_scrub_stamp":"2024-04-02T05:02:22.718259+0000","last_clean_scrub_stamp":"2024-04-02T05:07:04.516467+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:06.154767+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":126,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871458+0000","last_change":"2024-04-02T05:07:53.863861+0000","last_active":"2024-04-02T05:08:04.871458+0000","last_peered":"2024-04-02T05:08:04.871458+0000","last_clean":"2024-04-02T05:08:04.871458+0000","last_became_active":"2024-04-02T05:05:16.545041+0000","last_became_peered":"2024-04-02T05:05:16.545041+0000","last_unstale":"2024-04-02T05:08:04.871458+0000","last_undegraded":"2024-04-02T05:08:04.871458+0000","last_fullsized":"2024-04-02T05:08:04.871458+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:53.863711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.863711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:05.588023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'128","reported_seq":1667,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002118+0000","last_change":"2024-04-02T05:06:31.466688+0000","last_active":"2024-04-02T05:08:05.002118+0000","last_peered":"2024-04-02T05:08:05.002118+0000","last_clean":"2024-04-02T05:08:05.002118+0000","last_became_active":"2024-04-02T05:05:19.572970+0000","last_became_peered":"2024-04-02T05:05:19.572970+0000","last_unstale":"2024-04-02T05:08:05.002118+0000","last_undegraded":"2024-04-02T05:08:05.002118+0000","last_fullsized":"2024-04-02T05:08:05.002118+0000","mapping_epoch":925,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":926,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:06:31.466458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:06:31.466458+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1727,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401859+0000","last_change":"2024-04-02T05:07:31.630281+0000","last_active":"2024-04-02T05:08:04.401859+0000","last_peered":"2024-04-02T05:08:04.401859+0000","last_clean":"2024-04-02T05:08:04.401859+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T05:08:04.401859+0000","last_undegraded":"2024-04-02T05:08:04.401859+0000","last_fullsized":"2024-04-02T05:08:04.401859+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:31.630051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:31.630051+0000","last_clean_scrub_stamp":"2024-04-02T05:07:31.630051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.538788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1881,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397231+0000","last_change":"2024-04-02T05:07:27.532993+0000","last_active":"2024-04-02T05:08:04.397231+0000","last_peered":"2024-04-02T05:08:04.397231+0000","last_clean":"2024-04-02T05:08:04.397231+0000","last_became_active":"2024-04-02T04:59:00.138222+0000","last_became_peered":"2024-04-02T04:59:00.138222+0000","last_unstale":"2024-04-02T05:08:04.397231+0000","last_undegraded":"2024-04-02T05:08:04.397231+0000","last_fullsized":"2024-04-02T05:08:04.397231+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:27.532784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T05:07:27.532784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:34.311284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"899'32","reported_seq":720,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395994+0000","last_change":"2024-04-02T05:07:08.510901+0000","last_active":"2024-04-02T05:08:04.395994+0000","last_peered":"2024-04-02T05:08:04.395994+0000","last_clean":"2024-04-02T05:08:04.395994+0000","last_became_active":"2024-04-02T04:59:00.135793+0000","last_became_peered":"2024-04-02T04:59:00.135793+0000","last_unstale":"2024-04-02T05:08:04.395994+0000","last_undegraded":"2024-04-02T05:08:04.395994+0000","last_fullsized":"2024-04-02T05:08:04.395994+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'32","last_scrub_stamp":"2024-04-02T05:07:08.510631+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T05:07:08.510631+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:36.256714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1899,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400330+0000","last_change":"2024-04-02T05:08:03.702952+0000","last_active":"2024-04-02T05:08:04.400330+0000","last_peered":"2024-04-02T05:08:04.400330+0000","last_clean":"2024-04-02T05:08:04.400330+0000","last_became_active":"2024-04-02T04:59:00.106616+0000","last_became_peered":"2024-04-02T04:59:00.106616+0000","last_unstale":"2024-04-02T05:08:04.400330+0000","last_undegraded":"2024-04-02T05:08:04.400330+0000","last_fullsized":"2024-04-02T05:08:04.400330+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:03.702739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T05:08:03.702739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:14.771693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":81,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397775+0000","last_change":"2024-04-02T05:07:48.702097+0000","last_active":"2024-04-02T05:08:04.397775+0000","last_peered":"2024-04-02T05:08:04.397775+0000","last_clean":"2024-04-02T05:08:04.397775+0000","last_became_active":"2024-04-02T05:05:54.273784+0000","last_became_peered":"2024-04-02T05:05:54.273784+0000","last_unstale":"2024-04-02T05:08:04.397775+0000","last_undegraded":"2024-04-02T05:08:04.397775+0000","last_fullsized":"2024-04-02T05:08:04.397775+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:48.701885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.701885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:08.687729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"101'128","reported_seq":1812,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396388+0000","last_change":"2024-04-02T05:06:41.489660+0000","last_active":"2024-04-02T05:08:04.396388+0000","last_peered":"2024-04-02T05:08:04.396388+0000","last_clean":"2024-04-02T05:08:04.396388+0000","last_became_active":"2024-04-02T05:05:00.361876+0000","last_became_peered":"2024-04-02T05:05:00.361876+0000","last_unstale":"2024-04-02T05:08:04.396388+0000","last_undegraded":"2024-04-02T05:08:04.396388+0000","last_fullsized":"2024-04-02T05:08:04.396388+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:06:41.489427+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:06:41.489427+0000","last_clean_scrub_stamp":"2024-04-02T05:06:41.489427+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:08.984869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":86,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395793+0000","last_change":"2024-04-02T05:07:41.644212+0000","last_active":"2024-04-02T05:08:04.395793+0000","last_peered":"2024-04-02T05:08:04.395793+0000","last_clean":"2024-04-02T05:08:04.395793+0000","last_became_active":"2024-04-02T05:05:54.284476+0000","last_became_peered":"2024-04-02T05:05:54.284476+0000","last_unstale":"2024-04-02T05:08:04.395793+0000","last_undegraded":"2024-04-02T05:08:04.395793+0000","last_fullsized":"2024-04-02T05:08:04.395793+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T05:07:41.643877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:41.643877+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:59.541105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"899'60","reported_seq":737,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.394741+0000","last_change":"2024-04-02T05:06:48.404172+0000","last_active":"2024-04-02T05:08:04.394741+0000","last_peered":"2024-04-02T05:08:04.394741+0000","last_clean":"2024-04-02T05:08:04.394741+0000","last_became_active":"2024-04-02T04:59:00.122446+0000","last_became_peered":"2024-04-02T04:59:00.122446+0000","last_unstale":"2024-04-02T05:08:04.394741+0000","last_undegraded":"2024-04-02T05:08:04.394741+0000","last_fullsized":"2024-04-02T05:08:04.394741+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'60","last_scrub_stamp":"2024-04-02T05:06:48.404026+0000","last_deep_scrub":"899'60","last_deep_scrub_stamp":"2024-04-02T05:05:38.138230+0000","last_clean_scrub_stamp":"2024-04-02T05:06:48.404026+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:15.847159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1896,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395365+0000","last_change":"2024-04-02T05:07:14.781571+0000","last_active":"2024-04-02T05:08:04.395365+0000","last_peered":"2024-04-02T05:08:04.395365+0000","last_clean":"2024-04-02T05:08:04.395365+0000","last_became_active":"2024-04-02T04:59:00.107474+0000","last_became_peered":"2024-04-02T04:59:00.107474+0000","last_unstale":"2024-04-02T05:08:04.395365+0000","last_undegraded":"2024-04-02T05:08:04.395365+0000","last_fullsized":"2024-04-02T05:08:04.395365+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:14.781365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:01.943795+0000","last_clean_scrub_stamp":"2024-04-02T05:07:14.781365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:27.386708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1835,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.874244+0000","last_change":"2024-04-02T05:07:13.727004+0000","last_active":"2024-04-02T05:08:04.874244+0000","last_peered":"2024-04-02T05:08:04.874244+0000","last_clean":"2024-04-02T05:08:04.874244+0000","last_became_active":"2024-04-02T04:59:00.111699+0000","last_became_peered":"2024-04-02T04:59:00.111699+0000","last_unstale":"2024-04-02T05:08:04.874244+0000","last_undegraded":"2024-04-02T05:08:04.874244+0000","last_fullsized":"2024-04-02T05:08:04.874244+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:13.726798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:13.726798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:43.310957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:06.122666+0000","last_change":"2024-04-02T05:08:06.122776+0000","last_active":"2024-04-02T05:08:06.122666+0000","last_peered":"2024-04-02T05:08:06.122666+0000","last_clean":"2024-04-02T05:08:06.122666+0000","last_became_active":"2024-04-02T05:04:11.236780+0000","last_became_peered":"2024-04-02T05:04:11.236780+0000","last_unstale":"2024-04-02T05:08:06.122666+0000","last_undegraded":"2024-04-02T05:08:06.122666+0000","last_fullsized":"2024-04-02T05:08:06.122666+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:06.122472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:06.122472+0000","last_clean_scrub_stamp":"2024-04-02T05:08:06.122472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:34.519358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"101'128","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396846+0000","last_change":"2024-04-02T05:06:46.963153+0000","last_active":"2024-04-02T05:08:04.396846+0000","last_peered":"2024-04-02T05:08:04.396846+0000","last_clean":"2024-04-02T05:08:04.396846+0000","last_became_active":"2024-04-02T05:04:14.197651+0000","last_became_peered":"2024-04-02T05:04:14.197651+0000","last_unstale":"2024-04-02T05:08:04.396846+0000","last_undegraded":"2024-04-02T05:08:04.396846+0000","last_fullsized":"2024-04-02T05:08:04.396846+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:06:46.962995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:06:46.962995+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:05.704198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1725,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.665789+0000","last_change":"2024-04-02T05:07:01.749829+0000","last_active":"2024-04-02T05:08:04.665789+0000","last_peered":"2024-04-02T05:08:04.665789+0000","last_clean":"2024-04-02T05:08:04.665789+0000","last_became_active":"2024-04-02T05:05:54.276276+0000","last_became_peered":"2024-04-02T05:05:54.276276+0000","last_unstale":"2024-04-02T05:08:04.665789+0000","last_undegraded":"2024-04-02T05:08:04.665789+0000","last_fullsized":"2024-04-02T05:08:04.665789+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:01.749595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:30.797036+0000","last_clean_scrub_stamp":"2024-04-02T05:07:01.749595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:16.396838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"966'43","reported_seq":105,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395998+0000","last_change":"2024-04-02T05:07:51.708709+0000","last_active":"2024-04-02T05:08:04.395998+0000","last_peered":"2024-04-02T05:08:04.395998+0000","last_clean":"2024-04-02T05:08:04.395998+0000","last_became_active":"2024-04-02T05:05:54.280900+0000","last_became_peered":"2024-04-02T05:05:54.280900+0000","last_unstale":"2024-04-02T05:08:04.395998+0000","last_undegraded":"2024-04-02T05:08:04.395998+0000","last_fullsized":"2024-04-02T05:08:04.395998+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'43","last_scrub_stamp":"2024-04-02T05:07:51.708449+0000","last_deep_scrub":"966'43","last_deep_scrub_stamp":"2024-04-02T05:07:51.708449+0000","last_clean_scrub_stamp":"2024-04-02T05:07:51.708449+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:52.505344+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":48,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1354,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003520+0000","last_change":"2024-04-02T05:06:53.572103+0000","last_active":"2024-04-02T05:08:05.003520+0000","last_peered":"2024-04-02T05:08:05.003520+0000","last_clean":"2024-04-02T05:08:05.003520+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T05:08:05.003520+0000","last_undegraded":"2024-04-02T05:08:05.003520+0000","last_fullsized":"2024-04-02T05:08:05.003520+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:53.571825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T05:06:53.571825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"916'15","reported_seq":2210,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.672304+0000","last_change":"2024-04-02T05:08:04.672304+0000","last_active":"2024-04-02T05:08:04.672304+0000","last_peered":"2024-04-02T05:08:04.672304+0000","last_clean":"2024-04-02T05:08:04.672304+0000","last_became_active":"2024-04-02T05:04:58.338999+0000","last_became_peered":"2024-04-02T05:04:58.338999+0000","last_unstale":"2024-04-02T05:08:04.672304+0000","last_undegraded":"2024-04-02T05:08:04.672304+0000","last_fullsized":"2024-04-02T05:08:04.672304+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"916'15","last_scrub_stamp":"2024-04-02T05:08:04.672082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:08:04.672082+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:16.630151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397474+0000","last_change":"2024-04-02T05:07:54.427694+0000","last_active":"2024-04-02T05:08:04.397474+0000","last_peered":"2024-04-02T05:08:04.397474+0000","last_clean":"2024-04-02T05:08:04.397474+0000","last_became_active":"2024-04-02T05:06:31.713530+0000","last_became_peered":"2024-04-02T05:06:31.713530+0000","last_unstale":"2024-04-02T05:08:04.397474+0000","last_undegraded":"2024-04-02T05:08:04.397474+0000","last_fullsized":"2024-04-02T05:08:04.397474+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.427485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:30.458780+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.427485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:24.419377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'128","reported_seq":1342,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.408805+0000","last_change":"2024-04-02T05:08:07.408869+0000","last_active":"2024-04-02T05:08:07.408805+0000","last_peered":"2024-04-02T05:08:07.408805+0000","last_clean":"2024-04-02T05:08:07.408805+0000","last_became_active":"2024-04-02T05:05:00.369659+0000","last_became_peered":"2024-04-02T05:05:00.369659+0000","last_unstale":"2024-04-02T05:08:07.408805+0000","last_undegraded":"2024-04-02T05:08:07.408805+0000","last_fullsized":"2024-04-02T05:08:07.408805+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:07.408631+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:03:46.349028+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.408631+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:18.949607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1732,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.885405+0000","last_change":"2024-04-02T05:08:04.885468+0000","last_active":"2024-04-02T05:08:04.885405+0000","last_peered":"2024-04-02T05:08:04.885405+0000","last_clean":"2024-04-02T05:08:04.885405+0000","last_became_active":"2024-04-02T04:59:00.105786+0000","last_became_peered":"2024-04-02T04:59:00.105786+0000","last_unstale":"2024-04-02T05:08:04.885405+0000","last_undegraded":"2024-04-02T05:08:04.885405+0000","last_fullsized":"2024-04-02T05:08:04.885405+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:04.885258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T05:08:04.885258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:15.597944+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396371+0000","last_change":"2024-04-02T05:07:49.736414+0000","last_active":"2024-04-02T05:08:04.396371+0000","last_peered":"2024-04-02T05:08:04.396371+0000","last_clean":"2024-04-02T05:08:04.396371+0000","last_became_active":"2024-04-02T05:05:54.286532+0000","last_became_peered":"2024-04-02T05:05:54.286532+0000","last_unstale":"2024-04-02T05:08:04.396371+0000","last_undegraded":"2024-04-02T05:08:04.396371+0000","last_fullsized":"2024-04-02T05:08:04.396371+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:49.736191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:49.736191+0000","last_clean_scrub_stamp":"2024-04-02T05:07:49.736191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:52.156024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"899'42","reported_seq":1925,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073717+0000","last_change":"2024-04-02T05:06:47.004490+0000","last_active":"2024-04-02T05:08:05.073717+0000","last_peered":"2024-04-02T05:08:05.073717+0000","last_clean":"2024-04-02T05:08:05.073717+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T05:08:05.073717+0000","last_undegraded":"2024-04-02T05:08:05.073717+0000","last_fullsized":"2024-04-02T05:08:05.073717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'42","last_scrub_stamp":"2024-04-02T05:06:47.004356+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T05:06:47.004356+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:10.897155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1874,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668522+0000","last_change":"2024-04-02T05:07:29.813460+0000","last_active":"2024-04-02T05:08:04.668522+0000","last_peered":"2024-04-02T05:08:04.668522+0000","last_clean":"2024-04-02T05:08:04.668522+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T05:08:04.668522+0000","last_undegraded":"2024-04-02T05:08:04.668522+0000","last_fullsized":"2024-04-02T05:08:04.668522+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:29.813178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:29.813178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.316016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"831'6","reported_seq":1259,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:06.320443+0000","last_change":"2024-04-02T05:08:00.472828+0000","last_active":"2024-04-02T05:08:06.320443+0000","last_peered":"2024-04-02T05:08:06.320443+0000","last_clean":"2024-04-02T05:08:06.320443+0000","last_became_active":"2024-04-02T04:59:00.119844+0000","last_became_peered":"2024-04-02T04:59:00.119844+0000","last_unstale":"2024-04-02T05:08:06.320443+0000","last_undegraded":"2024-04-02T05:08:06.320443+0000","last_fullsized":"2024-04-02T05:08:06.320443+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'6","last_scrub_stamp":"2024-04-02T05:08:00.472747+0000","last_deep_scrub":"831'6","last_deep_scrub_stamp":"2024-04-02T05:06:35.417454+0000","last_clean_scrub_stamp":"2024-04-02T05:08:00.472747+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:02.348273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":553,"num_read_kb":548,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1319,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.008004+0000","last_change":"2024-04-02T05:07:32.551736+0000","last_active":"2024-04-02T05:08:05.008004+0000","last_peered":"2024-04-02T05:08:05.008004+0000","last_clean":"2024-04-02T05:08:05.008004+0000","last_became_active":"2024-04-02T05:05:16.545510+0000","last_became_peered":"2024-04-02T05:05:16.545510+0000","last_unstale":"2024-04-02T05:08:05.008004+0000","last_undegraded":"2024-04-02T05:08:05.008004+0000","last_fullsized":"2024-04-02T05:08:05.008004+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:32.551400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T05:07:32.551400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:33.564796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'128","reported_seq":923,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397303+0000","last_change":"2024-04-02T05:06:45.568625+0000","last_active":"2024-04-02T05:08:04.397303+0000","last_peered":"2024-04-02T05:08:04.397303+0000","last_clean":"2024-04-02T05:08:04.397303+0000","last_became_active":"2024-04-02T05:05:57.336377+0000","last_became_peered":"2024-04-02T05:05:57.336377+0000","last_unstale":"2024-04-02T05:08:04.397303+0000","last_undegraded":"2024-04-02T05:08:04.397303+0000","last_fullsized":"2024-04-02T05:08:04.397303+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:06:45.568389+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:06:45.568389+0000","last_clean_scrub_stamp":"2024-04-02T05:06:45.568389+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:10.544744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":186,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006740+0000","last_change":"2024-04-02T05:07:54.484663+0000","last_active":"2024-04-02T05:08:05.006740+0000","last_peered":"2024-04-02T05:08:05.006740+0000","last_clean":"2024-04-02T05:08:05.006740+0000","last_became_active":"2024-04-02T05:05:16.548018+0000","last_became_peered":"2024-04-02T05:05:16.548018+0000","last_unstale":"2024-04-02T05:08:05.006740+0000","last_undegraded":"2024-04-02T05:08:05.006740+0000","last_fullsized":"2024-04-02T05:08:05.006740+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.484452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:54.484452+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.484452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:24.426306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"642'16","reported_seq":821,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395718+0000","last_change":"2024-04-02T05:07:37.633477+0000","last_active":"2024-04-02T05:08:04.395718+0000","last_peered":"2024-04-02T05:08:04.395718+0000","last_clean":"2024-04-02T05:08:04.395718+0000","last_became_active":"2024-04-02T04:59:00.134138+0000","last_became_peered":"2024-04-02T04:59:00.134138+0000","last_unstale":"2024-04-02T05:08:04.395718+0000","last_undegraded":"2024-04-02T05:08:04.395718+0000","last_fullsized":"2024-04-02T05:08:04.395718+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'16","last_scrub_stamp":"2024-04-02T05:07:37.633367+0000","last_deep_scrub":"642'16","last_deep_scrub_stamp":"2024-04-02T05:05:15.903540+0000","last_clean_scrub_stamp":"2024-04-02T05:07:37.633367+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.483933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"899'33","reported_seq":733,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.403319+0000","last_change":"2024-04-02T05:07:53.458580+0000","last_active":"2024-04-02T05:08:04.403319+0000","last_peered":"2024-04-02T05:08:04.403319+0000","last_clean":"2024-04-02T05:08:04.403319+0000","last_became_active":"2024-04-02T04:59:00.134091+0000","last_became_peered":"2024-04-02T04:59:00.134091+0000","last_unstale":"2024-04-02T05:08:04.403319+0000","last_undegraded":"2024-04-02T05:08:04.403319+0000","last_fullsized":"2024-04-02T05:08:04.403319+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'33","last_scrub_stamp":"2024-04-02T05:07:53.458399+0000","last_deep_scrub":"824'29","last_deep_scrub_stamp":"2024-04-02T05:02:16.780210+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.458399+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.517576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":1880,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401792+0000","last_change":"2024-04-02T05:07:08.760901+0000","last_active":"2024-04-02T05:08:04.401792+0000","last_peered":"2024-04-02T05:08:04.401792+0000","last_clean":"2024-04-02T05:08:04.401792+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T05:08:04.401792+0000","last_undegraded":"2024-04-02T05:08:04.401792+0000","last_fullsized":"2024-04-02T05:08:04.401792+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T05:07:08.760668+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T05:07:08.760668+0000","last_clean_scrub_stamp":"2024-04-02T05:07:08.760668+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:10.417829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1898,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396181+0000","last_change":"2024-04-02T05:07:35.610591+0000","last_active":"2024-04-02T05:08:04.396181+0000","last_peered":"2024-04-02T05:08:04.396181+0000","last_clean":"2024-04-02T05:08:04.396181+0000","last_became_active":"2024-04-02T04:59:00.135841+0000","last_became_peered":"2024-04-02T04:59:00.135841+0000","last_unstale":"2024-04-02T05:08:04.396181+0000","last_undegraded":"2024-04-02T05:08:04.396181+0000","last_fullsized":"2024-04-02T05:08:04.396181+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:35.610373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:35.610373+0000","last_clean_scrub_stamp":"2024-04-02T05:07:35.610373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.495590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":74,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396077+0000","last_change":"2024-04-02T05:06:58.504782+0000","last_active":"2024-04-02T05:08:04.396077+0000","last_peered":"2024-04-02T05:08:04.396077+0000","last_clean":"2024-04-02T05:08:04.396077+0000","last_became_active":"2024-04-02T05:05:54.274830+0000","last_became_peered":"2024-04-02T05:05:54.274830+0000","last_unstale":"2024-04-02T05:08:04.396077+0000","last_undegraded":"2024-04-02T05:08:04.396077+0000","last_fullsized":"2024-04-02T05:08:04.396077+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:58.504567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:06:58.504567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:04.908757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"101'128","reported_seq":1367,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667315+0000","last_change":"2024-04-02T05:07:50.834842+0000","last_active":"2024-04-02T05:08:04.667315+0000","last_peered":"2024-04-02T05:08:04.667315+0000","last_clean":"2024-04-02T05:08:04.667315+0000","last_became_active":"2024-04-02T05:00:42.158023+0000","last_became_peered":"2024-04-02T05:00:42.158023+0000","last_unstale":"2024-04-02T05:08:04.667315+0000","last_undegraded":"2024-04-02T05:08:04.667315+0000","last_fullsized":"2024-04-02T05:08:04.667315+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:50.834610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:07:50.834610+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:19.534776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1720,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667294+0000","last_change":"2024-04-02T05:08:01.911079+0000","last_active":"2024-04-02T05:08:04.667294+0000","last_peered":"2024-04-02T05:08:04.667294+0000","last_clean":"2024-04-02T05:08:04.667294+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T05:08:04.667294+0000","last_undegraded":"2024-04-02T05:08:04.667294+0000","last_fullsized":"2024-04-02T05:08:04.667294+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:01.910857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:16.918634+0000","last_clean_scrub_stamp":"2024-04-02T05:08:01.910857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:23.455489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"913'18","reported_seq":2013,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.983656+0000","last_change":"2024-04-02T05:08:05.983724+0000","last_active":"2024-04-02T05:08:05.983656+0000","last_peered":"2024-04-02T05:08:05.983656+0000","last_clean":"2024-04-02T05:08:05.983656+0000","last_became_active":"2024-04-02T05:04:49.246171+0000","last_became_peered":"2024-04-02T05:04:49.246171+0000","last_unstale":"2024-04-02T05:08:05.983656+0000","last_undegraded":"2024-04-02T05:08:05.983656+0000","last_fullsized":"2024-04-02T05:08:05.983656+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'18","last_scrub_stamp":"2024-04-02T05:08:05.983491+0000","last_deep_scrub":"913'18","last_deep_scrub_stamp":"2024-04-02T05:08:05.983491+0000","last_clean_scrub_stamp":"2024-04-02T05:08:05.983491+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:22.023395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"923'75","reported_seq":2403,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.402494+0000","last_change":"2024-04-02T05:07:52.635084+0000","last_active":"2024-04-02T05:08:04.402494+0000","last_peered":"2024-04-02T05:08:04.402494+0000","last_clean":"2024-04-02T05:08:04.402494+0000","last_became_active":"2024-04-02T05:05:16.541267+0000","last_became_peered":"2024-04-02T05:05:16.541267+0000","last_unstale":"2024-04-02T05:08:04.402494+0000","last_undegraded":"2024-04-02T05:08:04.402494+0000","last_fullsized":"2024-04-02T05:08:04.402494+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'75","last_scrub_stamp":"2024-04-02T05:07:52.634835+0000","last_deep_scrub":"902'74","last_deep_scrub_stamp":"2024-04-02T05:04:57.843581+0000","last_clean_scrub_stamp":"2024-04-02T05:07:52.634835+0000","objects_scrubbed":11,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:04.951665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006867+0000","last_change":"2024-04-02T05:07:24.537256+0000","last_active":"2024-04-02T05:08:05.006867+0000","last_peered":"2024-04-02T05:08:05.006867+0000","last_clean":"2024-04-02T05:08:05.006867+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T05:08:05.006867+0000","last_undegraded":"2024-04-02T05:08:05.006867+0000","last_fullsized":"2024-04-02T05:08:05.006867+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:24.537040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:24.537040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.680818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"963'3","reported_seq":1887,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667455+0000","last_change":"2024-04-02T05:07:06.725851+0000","last_active":"2024-04-02T05:08:04.667455+0000","last_peered":"2024-04-02T05:08:04.667455+0000","last_clean":"2024-04-02T05:08:04.667455+0000","last_became_active":"2024-04-02T05:04:11.248836+0000","last_became_peered":"2024-04-02T05:04:11.248836+0000","last_unstale":"2024-04-02T05:08:04.667455+0000","last_undegraded":"2024-04-02T05:08:04.667455+0000","last_fullsized":"2024-04-02T05:08:04.667455+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'3","last_scrub_stamp":"2024-04-02T05:07:06.725621+0000","last_deep_scrub":"963'3","last_deep_scrub_stamp":"2024-04-02T05:07:06.725621+0000","last_clean_scrub_stamp":"2024-04-02T05:07:06.725621+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:12.700902+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1870,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396003+0000","last_change":"2024-04-02T05:07:09.788379+0000","last_active":"2024-04-02T05:08:04.396003+0000","last_peered":"2024-04-02T05:08:04.396003+0000","last_clean":"2024-04-02T05:08:04.396003+0000","last_became_active":"2024-04-02T05:05:16.534493+0000","last_became_peered":"2024-04-02T05:05:16.534493+0000","last_unstale":"2024-04-02T05:08:04.396003+0000","last_undegraded":"2024-04-02T05:08:04.396003+0000","last_fullsized":"2024-04-02T05:08:04.396003+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:09.788081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T05:07:09.788081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.983066+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'128","reported_seq":1886,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399076+0000","last_change":"2024-04-02T05:07:37.636248+0000","last_active":"2024-04-02T05:08:04.399076+0000","last_peered":"2024-04-02T05:08:04.399076+0000","last_clean":"2024-04-02T05:08:04.399076+0000","last_became_active":"2024-04-02T05:06:34.716887+0000","last_became_peered":"2024-04-02T05:06:34.716887+0000","last_unstale":"2024-04-02T05:08:04.399076+0000","last_undegraded":"2024-04-02T05:08:04.399076+0000","last_fullsized":"2024-04-02T05:08:04.399076+0000","mapping_epoch":955,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":956,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:37.636152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:07:37.636152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:44.889862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002126+0000","last_change":"2024-04-02T05:08:01.180720+0000","last_active":"2024-04-02T05:08:05.002126+0000","last_peered":"2024-04-02T05:08:05.002126+0000","last_clean":"2024-04-02T05:08:05.002126+0000","last_became_active":"2024-04-02T05:04:11.244090+0000","last_became_peered":"2024-04-02T05:04:11.244090+0000","last_unstale":"2024-04-02T05:08:05.002126+0000","last_undegraded":"2024-04-02T05:08:05.002126+0000","last_fullsized":"2024-04-02T05:08:05.002126+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:01.180481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:39.111154+0000","last_clean_scrub_stamp":"2024-04-02T05:08:01.180481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1899,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074187+0000","last_change":"2024-04-02T05:08:02.042395+0000","last_active":"2024-04-02T05:08:05.074187+0000","last_peered":"2024-04-02T05:08:05.074187+0000","last_clean":"2024-04-02T05:08:05.074187+0000","last_became_active":"2024-04-02T05:06:58.342937+0000","last_became_peered":"2024-04-02T05:06:58.342937+0000","last_unstale":"2024-04-02T05:08:05.074187+0000","last_undegraded":"2024-04-02T05:08:05.074187+0000","last_fullsized":"2024-04-02T05:08:05.074187+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:02.042192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:33.089451+0000","last_clean_scrub_stamp":"2024-04-02T05:08:02.042192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:08.179643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1341,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004499+0000","last_change":"2024-04-02T05:07:52.456071+0000","last_active":"2024-04-02T05:08:05.004499+0000","last_peered":"2024-04-02T05:08:05.004499+0000","last_clean":"2024-04-02T05:08:05.004499+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T05:08:05.004499+0000","last_undegraded":"2024-04-02T05:08:05.004499+0000","last_fullsized":"2024-04-02T05:08:05.004499+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:52.455804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:07.525265+0000","last_clean_scrub_stamp":"2024-04-02T05:07:52.455804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.186658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"899'56","reported_seq":2076,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.982441+0000","last_change":"2024-04-02T05:07:57.093142+0000","last_active":"2024-04-02T05:08:04.982441+0000","last_peered":"2024-04-02T05:08:04.982441+0000","last_clean":"2024-04-02T05:08:04.982441+0000","last_became_active":"2024-04-02T05:05:54.272867+0000","last_became_peered":"2024-04-02T05:05:54.272867+0000","last_unstale":"2024-04-02T05:08:04.982441+0000","last_undegraded":"2024-04-02T05:08:04.982441+0000","last_fullsized":"2024-04-02T05:08:04.982441+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'56","last_scrub_stamp":"2024-04-02T05:07:57.092887+0000","last_deep_scrub":"899'56","last_deep_scrub_stamp":"2024-04-02T05:07:57.092887+0000","last_clean_scrub_stamp":"2024-04-02T05:07:57.092887+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:17.555596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":140,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.402434+0000","last_change":"2024-04-02T05:07:38.650816+0000","last_active":"2024-04-02T05:08:04.402434+0000","last_peered":"2024-04-02T05:08:04.402434+0000","last_clean":"2024-04-02T05:08:04.402434+0000","last_became_active":"2024-04-02T05:04:58.366649+0000","last_became_peered":"2024-04-02T05:04:58.366649+0000","last_unstale":"2024-04-02T05:08:04.402434+0000","last_undegraded":"2024-04-02T05:08:04.402434+0000","last_fullsized":"2024-04-02T05:08:04.402434+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:38.650551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:38.650551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:00.510973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1720,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666881+0000","last_change":"2024-04-02T05:07:47.790135+0000","last_active":"2024-04-02T05:08:04.666881+0000","last_peered":"2024-04-02T05:08:04.666881+0000","last_clean":"2024-04-02T05:08:04.666881+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T05:08:04.666881+0000","last_undegraded":"2024-04-02T05:08:04.666881+0000","last_fullsized":"2024-04-02T05:08:04.666881+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:47.789917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:47.789917+0000","last_clean_scrub_stamp":"2024-04-02T05:07:47.789917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:13.222507+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1878,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667677+0000","last_change":"2024-04-02T05:07:17.721865+0000","last_active":"2024-04-02T05:08:04.667677+0000","last_peered":"2024-04-02T05:08:04.667677+0000","last_clean":"2024-04-02T05:08:04.667677+0000","last_became_active":"2024-04-02T05:05:54.277066+0000","last_became_peered":"2024-04-02T05:05:54.277066+0000","last_unstale":"2024-04-02T05:08:04.667677+0000","last_undegraded":"2024-04-02T05:08:04.667677+0000","last_fullsized":"2024-04-02T05:08:04.667677+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:17.721660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:17.721660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:32.313455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":1629,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004301+0000","last_change":"2024-04-02T05:07:26.590476+0000","last_active":"2024-04-02T05:08:05.004301+0000","last_peered":"2024-04-02T05:08:05.004301+0000","last_clean":"2024-04-02T05:08:05.004301+0000","last_became_active":"2024-04-02T05:00:42.145480+0000","last_became_peered":"2024-04-02T05:00:42.145480+0000","last_unstale":"2024-04-02T05:08:05.004301+0000","last_undegraded":"2024-04-02T05:08:05.004301+0000","last_fullsized":"2024-04-02T05:08:05.004301+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:26.590246+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:26.590246+0000","last_clean_scrub_stamp":"2024-04-02T05:07:26.590246+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.687128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.103719+0000","last_change":"2024-04-02T05:08:05.103776+0000","last_active":"2024-04-02T05:08:05.103719+0000","last_peered":"2024-04-02T05:08:05.103719+0000","last_clean":"2024-04-02T05:08:05.103719+0000","last_became_active":"2024-04-02T05:04:11.249101+0000","last_became_peered":"2024-04-02T05:04:11.249101+0000","last_unstale":"2024-04-02T05:08:05.103719+0000","last_undegraded":"2024-04-02T05:08:05.103719+0000","last_fullsized":"2024-04-02T05:08:05.103719+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:05.103555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T05:08:05.103555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:27.513983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"899'36","reported_seq":756,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395491+0000","last_change":"2024-04-02T05:07:59.430078+0000","last_active":"2024-04-02T05:08:04.395491+0000","last_peered":"2024-04-02T05:08:04.395491+0000","last_clean":"2024-04-02T05:08:04.395491+0000","last_became_active":"2024-04-02T05:04:11.244441+0000","last_became_peered":"2024-04-02T05:04:11.244441+0000","last_unstale":"2024-04-02T05:08:04.395491+0000","last_undegraded":"2024-04-02T05:08:04.395491+0000","last_fullsized":"2024-04-02T05:08:04.395491+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'36","last_scrub_stamp":"2024-04-02T05:07:59.429923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:59.429923+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:12.306308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005740+0000","last_change":"2024-04-02T05:07:54.161006+0000","last_active":"2024-04-02T05:08:05.005740+0000","last_peered":"2024-04-02T05:08:05.005740+0000","last_clean":"2024-04-02T05:08:05.005740+0000","last_became_active":"2024-04-02T05:04:11.248679+0000","last_became_peered":"2024-04-02T05:04:11.248679+0000","last_unstale":"2024-04-02T05:08:05.005740+0000","last_undegraded":"2024-04-02T05:08:05.005740+0000","last_fullsized":"2024-04-02T05:08:05.005740+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.160760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.160760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:13.538179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"101'128","reported_seq":1681,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005110+0000","last_change":"2024-04-02T05:07:17.661945+0000","last_active":"2024-04-02T05:08:05.005110+0000","last_peered":"2024-04-02T05:08:05.005110+0000","last_clean":"2024-04-02T05:08:05.005110+0000","last_became_active":"2024-04-02T05:05:57.345300+0000","last_became_peered":"2024-04-02T05:05:57.345300+0000","last_unstale":"2024-04-02T05:08:05.005110+0000","last_undegraded":"2024-04-02T05:08:05.005110+0000","last_fullsized":"2024-04-02T05:08:05.005110+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:17.661725+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:17.661725+0000","last_clean_scrub_stamp":"2024-04-02T05:07:17.661725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:33.567552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":174,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004798+0000","last_change":"2024-04-02T05:07:10.228915+0000","last_active":"2024-04-02T05:08:05.004798+0000","last_peered":"2024-04-02T05:08:05.004798+0000","last_clean":"2024-04-02T05:08:05.004798+0000","last_became_active":"2024-04-02T05:05:16.541976+0000","last_became_peered":"2024-04-02T05:05:16.541976+0000","last_unstale":"2024-04-02T05:08:05.004798+0000","last_undegraded":"2024-04-02T05:08:05.004798+0000","last_fullsized":"2024-04-02T05:08:05.004798+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:10.228639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:16.726893+0000","last_clean_scrub_stamp":"2024-04-02T05:07:10.228639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:38.742421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":132,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401075+0000","last_change":"2024-04-02T05:07:07.750918+0000","last_active":"2024-04-02T05:08:04.401075+0000","last_peered":"2024-04-02T05:08:04.401075+0000","last_clean":"2024-04-02T05:08:04.401075+0000","last_became_active":"2024-04-02T05:04:58.366684+0000","last_became_peered":"2024-04-02T05:04:58.366684+0000","last_unstale":"2024-04-02T05:08:04.401075+0000","last_undegraded":"2024-04-02T05:08:04.401075+0000","last_fullsized":"2024-04-02T05:08:04.401075+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:07.750520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:07.750520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:25.056645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870749+0000","last_change":"2024-04-02T05:07:07.752347+0000","last_active":"2024-04-02T05:08:04.870749+0000","last_peered":"2024-04-02T05:08:04.870749+0000","last_clean":"2024-04-02T05:08:04.870749+0000","last_became_active":"2024-04-02T05:05:16.542650+0000","last_became_peered":"2024-04-02T05:05:16.542650+0000","last_unstale":"2024-04-02T05:08:04.870749+0000","last_undegraded":"2024-04-02T05:08:04.870749+0000","last_fullsized":"2024-04-02T05:08:04.870749+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:07.752137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T05:07:07.752137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:36.052305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"899'54","reported_seq":2023,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397410+0000","last_change":"2024-04-02T05:07:01.763184+0000","last_active":"2024-04-02T05:08:04.397410+0000","last_peered":"2024-04-02T05:08:04.397410+0000","last_clean":"2024-04-02T05:08:04.397410+0000","last_became_active":"2024-04-02T05:05:16.533054+0000","last_became_peered":"2024-04-02T05:05:16.533054+0000","last_unstale":"2024-04-02T05:08:04.397410+0000","last_undegraded":"2024-04-02T05:08:04.397410+0000","last_fullsized":"2024-04-02T05:08:04.397410+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'54","last_scrub_stamp":"2024-04-02T05:07:01.762942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:01.762942+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:17.239990+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1349,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006998+0000","last_change":"2024-04-02T05:06:40.464481+0000","last_active":"2024-04-02T05:08:05.006998+0000","last_peered":"2024-04-02T05:08:05.006998+0000","last_clean":"2024-04-02T05:08:05.006998+0000","last_became_active":"2024-04-02T05:06:31.706429+0000","last_became_peered":"2024-04-02T05:06:31.706429+0000","last_unstale":"2024-04-02T05:08:05.006998+0000","last_undegraded":"2024-04-02T05:08:05.006998+0000","last_fullsized":"2024-04-02T05:08:05.006998+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:40.464282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:06:40.464282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'128","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005358+0000","last_change":"2024-04-02T05:07:47.112277+0000","last_active":"2024-04-02T05:08:05.005358+0000","last_peered":"2024-04-02T05:08:05.005358+0000","last_clean":"2024-04-02T05:08:05.005358+0000","last_became_active":"2024-04-02T05:04:14.205875+0000","last_became_peered":"2024-04-02T05:04:14.205875+0000","last_unstale":"2024-04-02T05:08:05.005358+0000","last_undegraded":"2024-04-02T05:08:05.005358+0000","last_fullsized":"2024-04-02T05:08:05.005358+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:47.112015+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:47.112015+0000","last_clean_scrub_stamp":"2024-04-02T05:07:47.112015+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:57.574668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1732,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666846+0000","last_change":"2024-04-02T05:07:21.631349+0000","last_active":"2024-04-02T05:08:04.666846+0000","last_peered":"2024-04-02T05:08:04.666846+0000","last_clean":"2024-04-02T05:08:04.666846+0000","last_became_active":"2024-04-02T05:04:11.250256+0000","last_became_peered":"2024-04-02T05:04:11.250256+0000","last_unstale":"2024-04-02T05:08:04.666846+0000","last_undegraded":"2024-04-02T05:08:04.666846+0000","last_fullsized":"2024-04-02T05:08:04.666846+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:21.631110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:21.631110+0000","last_clean_scrub_stamp":"2024-04-02T05:07:21.631110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:41.716023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1900,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401058+0000","last_change":"2024-04-02T05:06:47.920598+0000","last_active":"2024-04-02T05:08:04.401058+0000","last_peered":"2024-04-02T05:08:04.401058+0000","last_clean":"2024-04-02T05:08:04.401058+0000","last_became_active":"2024-04-02T05:05:16.531568+0000","last_became_peered":"2024-04-02T05:05:16.531568+0000","last_unstale":"2024-04-02T05:08:04.401058+0000","last_undegraded":"2024-04-02T05:08:04.401058+0000","last_fullsized":"2024-04-02T05:08:04.401058+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:47.920390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T05:06:47.920390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:05.501063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"899'45","reported_seq":1960,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397642+0000","last_change":"2024-04-02T05:07:55.690006+0000","last_active":"2024-04-02T05:08:04.397642+0000","last_peered":"2024-04-02T05:08:04.397642+0000","last_clean":"2024-04-02T05:08:04.397642+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T05:08:04.397642+0000","last_undegraded":"2024-04-02T05:08:04.397642+0000","last_fullsized":"2024-04-02T05:08:04.397642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'45","last_scrub_stamp":"2024-04-02T05:07:55.689760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.689760+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:23.320627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003521+0000","last_change":"2024-04-02T05:07:48.118210+0000","last_active":"2024-04-02T05:08:05.003521+0000","last_peered":"2024-04-02T05:08:05.003521+0000","last_clean":"2024-04-02T05:08:05.003521+0000","last_became_active":"2024-04-02T05:04:11.259878+0000","last_became_peered":"2024-04-02T05:04:11.259878+0000","last_unstale":"2024-04-02T05:08:05.003521+0000","last_undegraded":"2024-04-02T05:08:05.003521+0000","last_fullsized":"2024-04-02T05:08:05.003521+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:48.117999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.117999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:02.211490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400379+0000","last_change":"2024-04-02T05:07:22.562222+0000","last_active":"2024-04-02T05:08:04.400379+0000","last_peered":"2024-04-02T05:08:04.400379+0000","last_clean":"2024-04-02T05:08:04.400379+0000","last_became_active":"2024-04-02T04:59:00.128819+0000","last_became_peered":"2024-04-02T04:59:00.128819+0000","last_unstale":"2024-04-02T05:08:04.400379+0000","last_undegraded":"2024-04-02T05:08:04.400379+0000","last_fullsized":"2024-04-02T05:08:04.400379+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T05:07:22.561886+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_clean_scrub_stamp":"2024-04-02T05:07:22.561886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:29.778515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'128","reported_seq":1385,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668169+0000","last_change":"2024-04-02T05:07:48.823388+0000","last_active":"2024-04-02T05:08:04.668169+0000","last_peered":"2024-04-02T05:08:04.668169+0000","last_clean":"2024-04-02T05:08:04.668169+0000","last_became_active":"2024-04-02T05:00:42.122155+0000","last_became_peered":"2024-04-02T05:00:42.122155+0000","last_unstale":"2024-04-02T05:08:04.668169+0000","last_undegraded":"2024-04-02T05:08:04.668169+0000","last_fullsized":"2024-04-02T05:08:04.668169+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:48.823135+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:05:02.462796+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.823135+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:54.600448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1203,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006591+0000","last_change":"2024-04-02T05:07:22.562164+0000","last_active":"2024-04-02T05:08:05.006591+0000","last_peered":"2024-04-02T05:08:05.006591+0000","last_clean":"2024-04-02T05:08:05.006591+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T05:08:05.006591+0000","last_undegraded":"2024-04-02T05:08:05.006591+0000","last_fullsized":"2024-04-02T05:08:05.006591+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:22.561811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:22.561811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.405822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":146,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398974+0000","last_change":"2024-04-02T05:07:46.628675+0000","last_active":"2024-04-02T05:08:04.398974+0000","last_peered":"2024-04-02T05:08:04.398974+0000","last_clean":"2024-04-02T05:08:04.398974+0000","last_became_active":"2024-04-02T05:05:54.274401+0000","last_became_peered":"2024-04-02T05:05:54.274401+0000","last_unstale":"2024-04-02T05:08:04.398974+0000","last_undegraded":"2024-04-02T05:08:04.398974+0000","last_fullsized":"2024-04-02T05:08:04.398974+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:46.628541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:46.628541+0000","last_clean_scrub_stamp":"2024-04-02T05:07:46.628541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:49.663608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"902'44","reported_seq":242,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004207+0000","last_change":"2024-04-02T05:07:58.167702+0000","last_active":"2024-04-02T05:08:05.004207+0000","last_peered":"2024-04-02T05:08:05.004207+0000","last_clean":"2024-04-02T05:08:05.004207+0000","last_became_active":"2024-04-02T05:04:11.254466+0000","last_became_peered":"2024-04-02T05:04:11.254466+0000","last_unstale":"2024-04-02T05:08:05.004207+0000","last_undegraded":"2024-04-02T05:08:05.004207+0000","last_fullsized":"2024-04-02T05:08:05.004207+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'44","last_scrub_stamp":"2024-04-02T05:07:58.167425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.167425+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:04.286283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":1887,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401311+0000","last_change":"2024-04-02T05:07:12.771912+0000","last_active":"2024-04-02T05:08:04.401311+0000","last_peered":"2024-04-02T05:08:04.401311+0000","last_clean":"2024-04-02T05:08:04.401311+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T05:08:04.401311+0000","last_undegraded":"2024-04-02T05:08:04.401311+0000","last_fullsized":"2024-04-02T05:08:04.401311+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T05:07:12.771646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:12.771646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:22.704784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1896,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.981218+0000","last_change":"2024-04-02T05:07:36.083372+0000","last_active":"2024-04-02T05:08:04.981218+0000","last_peered":"2024-04-02T05:08:04.981218+0000","last_clean":"2024-04-02T05:08:04.981218+0000","last_became_active":"2024-04-02T05:05:16.535656+0000","last_became_peered":"2024-04-02T05:05:16.535656+0000","last_unstale":"2024-04-02T05:08:04.981218+0000","last_undegraded":"2024-04-02T05:08:04.981218+0000","last_fullsized":"2024-04-02T05:08:04.981218+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:36.083150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:45.610342+0000","last_clean_scrub_stamp":"2024-04-02T05:07:36.083150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:00.992666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"101'128","reported_seq":1336,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395857+0000","last_change":"2024-04-02T05:08:02.475711+0000","last_active":"2024-04-02T05:08:04.395857+0000","last_peered":"2024-04-02T05:08:04.395857+0000","last_clean":"2024-04-02T05:08:04.395857+0000","last_became_active":"2024-04-02T05:00:47.116787+0000","last_became_peered":"2024-04-02T05:00:47.116787+0000","last_unstale":"2024-04-02T05:08:04.395857+0000","last_undegraded":"2024-04-02T05:08:04.395857+0000","last_fullsized":"2024-04-02T05:08:04.395857+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:02.475610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:08:02.475610+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.035285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1750,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668296+0000","last_change":"2024-04-02T05:07:28.799135+0000","last_active":"2024-04-02T05:08:04.668296+0000","last_peered":"2024-04-02T05:08:04.668296+0000","last_clean":"2024-04-02T05:08:04.668296+0000","last_became_active":"2024-04-02T05:04:11.250122+0000","last_became_peered":"2024-04-02T05:04:11.250122+0000","last_unstale":"2024-04-02T05:08:04.668296+0000","last_undegraded":"2024-04-02T05:08:04.668296+0000","last_fullsized":"2024-04-02T05:08:04.668296+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:28.798827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:16.100623+0000","last_clean_scrub_stamp":"2024-04-02T05:07:28.798827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:46.169552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"642'14","reported_seq":2200,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396593+0000","last_change":"2024-04-02T05:07:03.540482+0000","last_active":"2024-04-02T05:08:04.396593+0000","last_peered":"2024-04-02T05:08:04.396593+0000","last_clean":"2024-04-02T05:08:04.396593+0000","last_became_active":"2024-04-02T04:59:00.137930+0000","last_became_peered":"2024-04-02T04:59:00.137930+0000","last_unstale":"2024-04-02T05:08:04.396593+0000","last_undegraded":"2024-04-02T05:08:04.396593+0000","last_fullsized":"2024-04-02T05:08:04.396593+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-04-02T05:07:03.540149+0000","last_deep_scrub":"642'14","last_deep_scrub_stamp":"2024-04-02T05:07:03.540149+0000","last_clean_scrub_stamp":"2024-04-02T05:07:03.540149+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:32.888678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"899'33","reported_seq":1443,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007391+0000","last_change":"2024-04-02T05:07:46.491568+0000","last_active":"2024-04-02T05:08:05.007391+0000","last_peered":"2024-04-02T05:08:05.007391+0000","last_clean":"2024-04-02T05:08:05.007391+0000","last_became_active":"2024-04-02T05:05:54.270036+0000","last_became_peered":"2024-04-02T05:05:54.270036+0000","last_unstale":"2024-04-02T05:08:05.007391+0000","last_undegraded":"2024-04-02T05:08:05.007391+0000","last_fullsized":"2024-04-02T05:08:05.007391+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'33","last_scrub_stamp":"2024-04-02T05:07:46.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:46.491320+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:10.683151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":1902,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398547+0000","last_change":"2024-04-02T05:06:44.528582+0000","last_active":"2024-04-02T05:08:04.398547+0000","last_peered":"2024-04-02T05:08:04.398547+0000","last_clean":"2024-04-02T05:08:04.398547+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T05:08:04.398547+0000","last_undegraded":"2024-04-02T05:08:04.398547+0000","last_fullsized":"2024-04-02T05:08:04.398547+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T05:06:44.528358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:06:44.528358+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:07.661918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":699,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397173+0000","last_change":"2024-04-02T05:07:41.618915+0000","last_active":"2024-04-02T05:08:04.397173+0000","last_peered":"2024-04-02T05:08:04.397173+0000","last_clean":"2024-04-02T05:08:04.397173+0000","last_became_active":"2024-04-02T04:59:00.122883+0000","last_became_peered":"2024-04-02T04:59:00.122883+0000","last_unstale":"2024-04-02T05:08:04.397173+0000","last_undegraded":"2024-04-02T05:08:04.397173+0000","last_fullsized":"2024-04-02T05:08:04.397173+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:41.618628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:41.618628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.210559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"101'128","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.693882+0000","last_change":"2024-04-02T05:08:07.694000+0000","last_active":"2024-04-02T05:08:07.693882+0000","last_peered":"2024-04-02T05:08:07.693882+0000","last_clean":"2024-04-02T05:08:07.693882+0000","last_became_active":"2024-04-02T05:05:57.347095+0000","last_became_peered":"2024-04-02T05:05:57.347095+0000","last_unstale":"2024-04-02T05:08:07.693882+0000","last_undegraded":"2024-04-02T05:08:07.693882+0000","last_fullsized":"2024-04-02T05:08:07.693882+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:07.693611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.693611+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:30.323022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870697+0000","last_change":"2024-04-02T05:07:21.828734+0000","last_active":"2024-04-02T05:08:04.870697+0000","last_peered":"2024-04-02T05:08:04.870697+0000","last_clean":"2024-04-02T05:08:04.870697+0000","last_became_active":"2024-04-02T05:05:16.545382+0000","last_became_peered":"2024-04-02T05:05:16.545382+0000","last_unstale":"2024-04-02T05:08:04.870697+0000","last_undegraded":"2024-04-02T05:08:04.870697+0000","last_fullsized":"2024-04-02T05:08:04.870697+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:21.828523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:30.126652+0000","last_clean_scrub_stamp":"2024-04-02T05:07:21.828523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:49.925302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"913'39","reported_seq":1399,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005420+0000","last_change":"2024-04-02T05:07:48.508225+0000","last_active":"2024-04-02T05:08:05.005420+0000","last_peered":"2024-04-02T05:08:05.005420+0000","last_clean":"2024-04-02T05:08:05.005420+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T05:08:05.005420+0000","last_undegraded":"2024-04-02T05:08:05.005420+0000","last_fullsized":"2024-04-02T05:08:05.005420+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'39","last_scrub_stamp":"2024-04-02T05:07:48.507909+0000","last_deep_scrub":"913'39","last_deep_scrub_stamp":"2024-04-02T05:07:48.507909+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.507909+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:03.501904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871079+0000","last_change":"2024-04-02T05:07:15.730014+0000","last_active":"2024-04-02T05:08:04.871079+0000","last_peered":"2024-04-02T05:08:04.871079+0000","last_clean":"2024-04-02T05:08:04.871079+0000","last_became_active":"2024-04-02T05:05:16.538443+0000","last_became_peered":"2024-04-02T05:05:16.538443+0000","last_unstale":"2024-04-02T05:08:04.871079+0000","last_undegraded":"2024-04-02T05:08:04.871079+0000","last_fullsized":"2024-04-02T05:08:04.871079+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:15.729881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:15.729881+0000","last_clean_scrub_stamp":"2024-04-02T05:07:15.729881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.089676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074442+0000","last_change":"2024-04-02T05:07:28.976406+0000","last_active":"2024-04-02T05:08:05.074442+0000","last_peered":"2024-04-02T05:08:05.074442+0000","last_clean":"2024-04-02T05:08:05.074442+0000","last_became_active":"2024-04-02T04:59:00.138663+0000","last_became_peered":"2024-04-02T04:59:00.138663+0000","last_unstale":"2024-04-02T05:08:05.074442+0000","last_undegraded":"2024-04-02T05:08:05.074442+0000","last_fullsized":"2024-04-02T05:08:05.074442+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:28.976184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:28.976184+0000","last_clean_scrub_stamp":"2024-04-02T05:07:28.976184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.759955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870870+0000","last_change":"2024-04-02T05:07:33.944141+0000","last_active":"2024-04-02T05:08:04.870870+0000","last_peered":"2024-04-02T05:08:04.870870+0000","last_clean":"2024-04-02T05:08:04.870870+0000","last_became_active":"2024-04-02T05:05:16.546959+0000","last_became_peered":"2024-04-02T05:05:16.546959+0000","last_unstale":"2024-04-02T05:08:04.870870+0000","last_undegraded":"2024-04-02T05:08:04.870870+0000","last_fullsized":"2024-04-02T05:08:04.870870+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:33.943921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:25.773178+0000","last_clean_scrub_stamp":"2024-04-02T05:07:33.943921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:53.140964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"101'128","reported_seq":1317,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003808+0000","last_change":"2024-04-02T05:07:57.213543+0000","last_active":"2024-04-02T05:08:05.003808+0000","last_peered":"2024-04-02T05:08:05.003808+0000","last_clean":"2024-04-02T05:08:05.003808+0000","last_became_active":"2024-04-02T05:04:14.204099+0000","last_became_peered":"2024-04-02T05:04:14.204099+0000","last_unstale":"2024-04-02T05:08:05.003808+0000","last_undegraded":"2024-04-02T05:08:05.003808+0000","last_fullsized":"2024-04-02T05:08:05.003808+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:57.213308+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:57.213308+0000","last_clean_scrub_stamp":"2024-04-02T05:07:57.213308+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:17.535160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1737,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.394679+0000","last_change":"2024-04-02T05:07:32.585524+0000","last_active":"2024-04-02T05:08:04.394679+0000","last_peered":"2024-04-02T05:08:04.394679+0000","last_clean":"2024-04-02T05:08:04.394679+0000","last_became_active":"2024-04-02T04:59:00.104872+0000","last_became_peered":"2024-04-02T04:59:00.104872+0000","last_unstale":"2024-04-02T05:08:04.394679+0000","last_undegraded":"2024-04-02T05:08:04.394679+0000","last_fullsized":"2024-04-02T05:08:04.394679+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:32.585367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:32.585367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.155974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"899'12","reported_seq":1926,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.665870+0000","last_change":"2024-04-02T05:07:26.754432+0000","last_active":"2024-04-02T05:08:04.665870+0000","last_peered":"2024-04-02T05:08:04.665870+0000","last_clean":"2024-04-02T05:08:04.665870+0000","last_became_active":"2024-04-02T05:04:11.244389+0000","last_became_peered":"2024-04-02T05:04:11.244389+0000","last_unstale":"2024-04-02T05:08:04.665870+0000","last_undegraded":"2024-04-02T05:08:04.665870+0000","last_fullsized":"2024-04-02T05:08:04.665870+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'12","last_scrub_stamp":"2024-04-02T05:07:26.754157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:26.754157+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:33.811607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1882,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666060+0000","last_change":"2024-04-02T05:07:02.710738+0000","last_active":"2024-04-02T05:08:04.666060+0000","last_peered":"2024-04-02T05:08:04.666060+0000","last_clean":"2024-04-02T05:08:04.666060+0000","last_became_active":"2024-04-02T05:05:54.277359+0000","last_became_peered":"2024-04-02T05:05:54.277359+0000","last_unstale":"2024-04-02T05:08:04.666060+0000","last_undegraded":"2024-04-02T05:08:04.666060+0000","last_fullsized":"2024-04-02T05:08:04.666060+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:02.710526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:22.817927+0000","last_clean_scrub_stamp":"2024-04-02T05:07:02.710526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:30.066926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.403490+0000","last_change":"2024-04-02T05:07:04.460731+0000","last_active":"2024-04-02T05:08:04.403490+0000","last_peered":"2024-04-02T05:08:04.403490+0000","last_clean":"2024-04-02T05:08:04.403490+0000","last_became_active":"2024-04-02T04:59:00.122643+0000","last_became_peered":"2024-04-02T04:59:00.122643+0000","last_unstale":"2024-04-02T05:08:04.403490+0000","last_undegraded":"2024-04-02T05:08:04.403490+0000","last_fullsized":"2024-04-02T05:08:04.403490+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:04.460509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T05:07:04.460509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:07.683941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1347,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006718+0000","last_change":"2024-04-02T05:07:11.545505+0000","last_active":"2024-04-02T05:08:05.006718+0000","last_peered":"2024-04-02T05:08:05.006718+0000","last_clean":"2024-04-02T05:08:05.006718+0000","last_became_active":"2024-04-02T05:05:54.284170+0000","last_became_peered":"2024-04-02T05:05:54.284170+0000","last_unstale":"2024-04-02T05:08:05.006718+0000","last_undegraded":"2024-04-02T05:08:05.006718+0000","last_fullsized":"2024-04-02T05:08:05.006718+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:11.545196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:11.545196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:11.794955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"101'128","reported_seq":1714,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.981582+0000","last_change":"2024-04-02T05:07:58.122361+0000","last_active":"2024-04-02T05:08:04.981582+0000","last_peered":"2024-04-02T05:08:04.981582+0000","last_clean":"2024-04-02T05:08:04.981582+0000","last_became_active":"2024-04-02T05:04:49.251034+0000","last_became_peered":"2024-04-02T05:04:49.251034+0000","last_unstale":"2024-04-02T05:08:04.981582+0000","last_undegraded":"2024-04-02T05:08:04.981582+0000","last_fullsized":"2024-04-02T05:08:04.981582+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:58.122090+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:04:55.430781+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.122090+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:15.725450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":49,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666294+0000","last_change":"2024-04-02T05:07:34.733429+0000","last_active":"2024-04-02T05:08:04.666294+0000","last_peered":"2024-04-02T05:08:04.666294+0000","last_clean":"2024-04-02T05:08:04.666294+0000","last_became_active":"2024-04-02T05:06:31.713784+0000","last_became_peered":"2024-04-02T05:06:31.713784+0000","last_unstale":"2024-04-02T05:08:04.666294+0000","last_undegraded":"2024-04-02T05:08:04.666294+0000","last_fullsized":"2024-04-02T05:08:04.666294+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:34.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T05:07:34.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.914233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"899'27","reported_seq":1358,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006193+0000","last_change":"2024-04-02T05:06:55.557040+0000","last_active":"2024-04-02T05:08:05.006193+0000","last_peered":"2024-04-02T05:08:05.006193+0000","last_clean":"2024-04-02T05:08:05.006193+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T05:08:05.006193+0000","last_undegraded":"2024-04-02T05:08:05.006193+0000","last_fullsized":"2024-04-02T05:08:05.006193+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'27","last_scrub_stamp":"2024-04-02T05:06:55.556806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:06:55.556806+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:20.928519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002964+0000","last_change":"2024-04-02T05:06:56.291063+0000","last_active":"2024-04-02T05:08:05.002964+0000","last_peered":"2024-04-02T05:08:05.002964+0000","last_clean":"2024-04-02T05:08:05.002964+0000","last_became_active":"2024-04-02T05:04:11.258328+0000","last_became_peered":"2024-04-02T05:04:11.258328+0000","last_unstale":"2024-04-02T05:08:05.002964+0000","last_undegraded":"2024-04-02T05:08:05.002964+0000","last_fullsized":"2024-04-02T05:08:05.002964+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:56.290838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:06:56.290838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:21.303130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"916'18","reported_seq":823,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397334+0000","last_change":"2024-04-02T05:07:13.554733+0000","last_active":"2024-04-02T05:08:04.397334+0000","last_peered":"2024-04-02T05:08:04.397334+0000","last_clean":"2024-04-02T05:08:04.397334+0000","last_became_active":"2024-04-02T05:04:58.340049+0000","last_became_peered":"2024-04-02T05:04:58.340049+0000","last_unstale":"2024-04-02T05:08:04.397334+0000","last_undegraded":"2024-04-02T05:08:04.397334+0000","last_fullsized":"2024-04-02T05:08:04.397334+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"916'18","last_scrub_stamp":"2024-04-02T05:07:13.554493+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T05:07:13.554493+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:36.245747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1730,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074198+0000","last_change":"2024-04-02T05:07:45.116251+0000","last_active":"2024-04-02T05:08:05.074198+0000","last_peered":"2024-04-02T05:08:05.074198+0000","last_clean":"2024-04-02T05:08:05.074198+0000","last_became_active":"2024-04-02T05:04:49.246021+0000","last_became_peered":"2024-04-02T05:04:49.246021+0000","last_unstale":"2024-04-02T05:08:05.074198+0000","last_undegraded":"2024-04-02T05:08:05.074198+0000","last_fullsized":"2024-04-02T05:08:05.074198+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:45.116034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:17.214556+0000","last_clean_scrub_stamp":"2024-04-02T05:07:45.116034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.999021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"101'128","reported_seq":1294,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002699+0000","last_change":"2024-04-02T05:07:03.233057+0000","last_active":"2024-04-02T05:08:05.002699+0000","last_peered":"2024-04-02T05:08:05.002699+0000","last_clean":"2024-04-02T05:08:05.002699+0000","last_became_active":"2024-04-02T05:04:14.202460+0000","last_became_peered":"2024-04-02T05:04:14.202460+0000","last_unstale":"2024-04-02T05:08:05.002699+0000","last_undegraded":"2024-04-02T05:08:05.002699+0000","last_fullsized":"2024-04-02T05:08:05.002699+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:03.232808+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:03.232808+0000","last_clean_scrub_stamp":"2024-04-02T05:07:03.232808+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:15.699132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":724,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396419+0000","last_change":"2024-04-02T05:07:58.470081+0000","last_active":"2024-04-02T05:08:04.396419+0000","last_peered":"2024-04-02T05:08:04.396419+0000","last_clean":"2024-04-02T05:08:04.396419+0000","last_became_active":"2024-04-02T05:05:54.274382+0000","last_became_peered":"2024-04-02T05:05:54.274382+0000","last_unstale":"2024-04-02T05:08:04.396419+0000","last_undegraded":"2024-04-02T05:08:04.396419+0000","last_fullsized":"2024-04-02T05:08:04.396419+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:58.469874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:58.469874+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.469874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.283378+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"899'21","reported_seq":1000,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.982108+0000","last_change":"2024-04-02T05:07:41.043760+0000","last_active":"2024-04-02T05:08:04.982108+0000","last_peered":"2024-04-02T05:08:04.982108+0000","last_clean":"2024-04-02T05:08:04.982108+0000","last_became_active":"2024-04-02T05:04:49.244686+0000","last_became_peered":"2024-04-02T05:04:49.244686+0000","last_unstale":"2024-04-02T05:08:04.982108+0000","last_undegraded":"2024-04-02T05:08:04.982108+0000","last_fullsized":"2024-04-02T05:08:04.982108+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'21","last_scrub_stamp":"2024-04-02T05:07:41.043524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:41.043524+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.025183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1742,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.982403+0000","last_change":"2024-04-02T05:08:03.014352+0000","last_active":"2024-04-02T05:08:04.982403+0000","last_peered":"2024-04-02T05:08:04.982403+0000","last_clean":"2024-04-02T05:08:04.982403+0000","last_became_active":"2024-04-02T05:04:49.231489+0000","last_became_peered":"2024-04-02T05:04:49.231489+0000","last_unstale":"2024-04-02T05:08:04.982403+0000","last_undegraded":"2024-04-02T05:08:04.982403+0000","last_fullsized":"2024-04-02T05:08:04.982403+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:03.014106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:54.894248+0000","last_clean_scrub_stamp":"2024-04-02T05:08:03.014106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:05.384593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"961'20","reported_seq":243,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004668+0000","last_change":"2024-04-02T05:07:02.262832+0000","last_active":"2024-04-02T05:08:05.004668+0000","last_peered":"2024-04-02T05:08:05.004668+0000","last_clean":"2024-04-02T05:08:05.004668+0000","last_became_active":"2024-04-02T05:06:58.350725+0000","last_became_peered":"2024-04-02T05:06:58.350725+0000","last_unstale":"2024-04-02T05:08:05.004668+0000","last_undegraded":"2024-04-02T05:08:05.004668+0000","last_fullsized":"2024-04-02T05:08:05.004668+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":0,"last_scrub":"961'20","last_scrub_stamp":"2024-04-02T05:07:02.262604+0000","last_deep_scrub":"937'19","last_deep_scrub_stamp":"2024-04-02T05:05:46.936756+0000","last_clean_scrub_stamp":"2024-04-02T05:07:02.262604+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:31.096682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399011+0000","last_change":"2024-04-02T05:07:22.480713+0000","last_active":"2024-04-02T05:08:04.399011+0000","last_peered":"2024-04-02T05:08:04.399011+0000","last_clean":"2024-04-02T05:08:04.399011+0000","last_became_active":"2024-04-02T05:04:11.238138+0000","last_became_peered":"2024-04-02T05:04:11.238138+0000","last_unstale":"2024-04-02T05:08:04.399011+0000","last_undegraded":"2024-04-02T05:08:04.399011+0000","last_fullsized":"2024-04-02T05:08:04.399011+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:22.480435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:14.397839+0000","last_clean_scrub_stamp":"2024-04-02T05:07:22.480435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:28.511395+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666877+0000","last_change":"2024-04-02T05:06:57.710439+0000","last_active":"2024-04-02T05:08:04.666877+0000","last_peered":"2024-04-02T05:08:04.666877+0000","last_clean":"2024-04-02T05:08:04.666877+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T05:08:04.666877+0000","last_undegraded":"2024-04-02T05:08:04.666877+0000","last_fullsized":"2024-04-02T05:08:04.666877+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:57.710144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:31.233061+0000","last_clean_scrub_stamp":"2024-04-02T05:06:57.710144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:22.891060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"930'32","reported_seq":1440,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007283+0000","last_change":"2024-04-02T05:07:58.530087+0000","last_active":"2024-04-02T05:08:05.007283+0000","last_peered":"2024-04-02T05:08:05.007283+0000","last_clean":"2024-04-02T05:08:05.007283+0000","last_became_active":"2024-04-02T04:59:00.137738+0000","last_became_peered":"2024-04-02T04:59:00.137738+0000","last_unstale":"2024-04-02T05:08:05.007283+0000","last_undegraded":"2024-04-02T05:08:05.007283+0000","last_fullsized":"2024-04-02T05:08:05.007283+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'32","last_scrub_stamp":"2024-04-02T05:07:58.529819+0000","last_deep_scrub":"930'32","last_deep_scrub_stamp":"2024-04-02T05:05:29.330313+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.529819+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:25.425552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1880,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667803+0000","last_change":"2024-04-02T05:07:03.694867+0000","last_active":"2024-04-02T05:08:04.667803+0000","last_peered":"2024-04-02T05:08:04.667803+0000","last_clean":"2024-04-02T05:08:04.667803+0000","last_became_active":"2024-04-02T05:04:11.257466+0000","last_became_peered":"2024-04-02T05:04:11.257466+0000","last_unstale":"2024-04-02T05:08:04.667803+0000","last_undegraded":"2024-04-02T05:08:04.667803+0000","last_fullsized":"2024-04-02T05:08:04.667803+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:03.694645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:03.694645+0000","last_clean_scrub_stamp":"2024-04-02T05:07:03.694645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:31.460633+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":132,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399239+0000","last_change":"2024-04-02T05:07:27.757587+0000","last_active":"2024-04-02T05:08:04.399239+0000","last_peered":"2024-04-02T05:08:04.399239+0000","last_clean":"2024-04-02T05:08:04.399239+0000","last_became_active":"2024-04-02T05:04:58.335973+0000","last_became_peered":"2024-04-02T05:04:58.335973+0000","last_unstale":"2024-04-02T05:08:04.399239+0000","last_undegraded":"2024-04-02T05:08:04.399239+0000","last_fullsized":"2024-04-02T05:08:04.399239+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:27.757340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:27.757340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:44.793115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:06.980882+0000","last_change":"2024-04-02T05:08:06.980944+0000","last_active":"2024-04-02T05:08:06.980882+0000","last_peered":"2024-04-02T05:08:06.980882+0000","last_clean":"2024-04-02T05:08:06.980882+0000","last_became_active":"2024-04-02T05:04:11.228488+0000","last_became_peered":"2024-04-02T05:04:11.228488+0000","last_unstale":"2024-04-02T05:08:06.980882+0000","last_undegraded":"2024-04-02T05:08:06.980882+0000","last_fullsized":"2024-04-02T05:08:06.980882+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:06.980719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:43.063652+0000","last_clean_scrub_stamp":"2024-04-02T05:08:06.980719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:18.002283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1733,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399364+0000","last_change":"2024-04-02T05:06:54.437684+0000","last_active":"2024-04-02T05:08:04.399364+0000","last_peered":"2024-04-02T05:08:04.399364+0000","last_clean":"2024-04-02T05:08:04.399364+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T05:08:04.399364+0000","last_undegraded":"2024-04-02T05:08:04.399364+0000","last_fullsized":"2024-04-02T05:08:04.399364+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:54.437476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:06.838854+0000","last_clean_scrub_stamp":"2024-04-02T05:06:54.437476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:09.575894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"899'42","reported_seq":97,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396772+0000","last_change":"2024-04-02T05:07:32.601500+0000","last_active":"2024-04-02T05:08:04.396772+0000","last_peered":"2024-04-02T05:08:04.396772+0000","last_clean":"2024-04-02T05:08:04.396772+0000","last_became_active":"2024-04-02T05:05:54.282114+0000","last_became_peered":"2024-04-02T05:05:54.282114+0000","last_unstale":"2024-04-02T05:08:04.396772+0000","last_undegraded":"2024-04-02T05:08:04.396772+0000","last_fullsized":"2024-04-02T05:08:04.396772+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'42","last_scrub_stamp":"2024-04-02T05:07:32.601274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:32.601274+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:49.240468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1889,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400913+0000","last_change":"2024-04-02T05:07:48.661178+0000","last_active":"2024-04-02T05:08:04.400913+0000","last_peered":"2024-04-02T05:08:04.400913+0000","last_clean":"2024-04-02T05:08:04.400913+0000","last_became_active":"2024-04-02T05:04:11.236464+0000","last_became_peered":"2024-04-02T05:04:11.236464+0000","last_unstale":"2024-04-02T05:08:04.400913+0000","last_undegraded":"2024-04-02T05:08:04.400913+0000","last_fullsized":"2024-04-02T05:08:04.400913+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:48.661017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:51.374778+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.661017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:03.428976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1877,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396959+0000","last_change":"2024-04-02T05:07:24.462014+0000","last_active":"2024-04-02T05:08:04.396959+0000","last_peered":"2024-04-02T05:08:04.396959+0000","last_clean":"2024-04-02T05:08:04.396959+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T05:08:04.396959+0000","last_undegraded":"2024-04-02T05:08:04.396959+0000","last_fullsized":"2024-04-02T05:08:04.396959+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:24.461805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_clean_scrub_stamp":"2024-04-02T05:07:24.461805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:50.752216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":718,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399977+0000","last_change":"2024-04-02T05:07:49.591776+0000","last_active":"2024-04-02T05:08:04.399977+0000","last_peered":"2024-04-02T05:08:04.399977+0000","last_clean":"2024-04-02T05:08:04.399977+0000","last_became_active":"2024-04-02T05:04:11.244750+0000","last_became_peered":"2024-04-02T05:04:11.244750+0000","last_unstale":"2024-04-02T05:08:04.399977+0000","last_undegraded":"2024-04-02T05:08:04.399977+0000","last_fullsized":"2024-04-02T05:08:04.399977+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:49.591591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T05:07:49.591591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:00.965159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1738,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073633+0000","last_change":"2024-04-02T05:07:02.972201+0000","last_active":"2024-04-02T05:08:05.073633+0000","last_peered":"2024-04-02T05:08:05.073633+0000","last_clean":"2024-04-02T05:08:05.073633+0000","last_became_active":"2024-04-02T05:04:11.228518+0000","last_became_peered":"2024-04-02T05:04:11.228518+0000","last_unstale":"2024-04-02T05:08:05.073633+0000","last_undegraded":"2024-04-02T05:08:05.073633+0000","last_fullsized":"2024-04-02T05:08:05.073633+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:02.971975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T05:07:02.971975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:22.016275+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"959'28","reported_seq":1545,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074256+0000","last_change":"2024-04-02T05:08:04.028230+0000","last_active":"2024-04-02T05:08:05.074256+0000","last_peered":"2024-04-02T05:08:05.074256+0000","last_clean":"2024-04-02T05:08:05.074256+0000","last_became_active":"2024-04-02T05:04:49.241105+0000","last_became_peered":"2024-04-02T05:04:49.241105+0000","last_unstale":"2024-04-02T05:08:05.074256+0000","last_undegraded":"2024-04-02T05:08:05.074256+0000","last_fullsized":"2024-04-02T05:08:05.074256+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'28","last_scrub_stamp":"2024-04-02T05:08:04.028138+0000","last_deep_scrub":"959'28","last_deep_scrub_stamp":"2024-04-02T05:06:55.902636+0000","last_clean_scrub_stamp":"2024-04-02T05:08:04.028138+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:27.781946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":174,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003875+0000","last_change":"2024-04-02T05:07:16.208642+0000","last_active":"2024-04-02T05:08:05.003875+0000","last_peered":"2024-04-02T05:08:05.003875+0000","last_clean":"2024-04-02T05:08:05.003875+0000","last_became_active":"2024-04-02T05:05:16.546222+0000","last_became_peered":"2024-04-02T05:05:16.546222+0000","last_unstale":"2024-04-02T05:08:05.003875+0000","last_undegraded":"2024-04-02T05:08:05.003875+0000","last_fullsized":"2024-04-02T05:08:05.003875+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.208414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:27.895598+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.208414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.128614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.394473+0000","last_change":"2024-04-02T05:06:46.356620+0000","last_active":"2024-04-02T05:08:04.394473+0000","last_peered":"2024-04-02T05:08:04.394473+0000","last_clean":"2024-04-02T05:08:04.394473+0000","last_became_active":"2024-04-02T04:59:00.102092+0000","last_became_peered":"2024-04-02T04:59:00.102092+0000","last_unstale":"2024-04-02T05:08:04.394473+0000","last_undegraded":"2024-04-02T05:08:04.394473+0000","last_fullsized":"2024-04-02T05:08:04.394473+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:46.356347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.801839+0000","last_clean_scrub_stamp":"2024-04-02T05:06:46.356347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:11.009981+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1332,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:08.534867+0000","last_change":"2024-04-02T05:08:08.534924+0000","last_active":"2024-04-02T05:08:08.534867+0000","last_peered":"2024-04-02T05:08:08.534867+0000","last_clean":"2024-04-02T05:08:08.534867+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T05:08:08.534867+0000","last_undegraded":"2024-04-02T05:08:08.534867+0000","last_fullsized":"2024-04-02T05:08:08.534867+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:08.534714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T05:08:08.534714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:24.671108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"101'128","reported_seq":1333,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.872970+0000","last_change":"2024-04-02T05:07:31.985648+0000","last_active":"2024-04-02T05:08:04.872970+0000","last_peered":"2024-04-02T05:08:04.872970+0000","last_clean":"2024-04-02T05:08:04.872970+0000","last_became_active":"2024-04-02T05:05:57.337502+0000","last_became_peered":"2024-04-02T05:05:57.337502+0000","last_unstale":"2024-04-02T05:08:04.872970+0000","last_undegraded":"2024-04-02T05:08:04.872970+0000","last_fullsized":"2024-04-02T05:08:04.872970+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:31.985497+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:31.985497+0000","last_clean_scrub_stamp":"2024-04-02T05:07:31.985497+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:38.027767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1215,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002117+0000","last_change":"2024-04-02T05:06:30.463193+0000","last_active":"2024-04-02T05:08:05.002117+0000","last_peered":"2024-04-02T05:08:05.002117+0000","last_clean":"2024-04-02T05:08:05.002117+0000","last_became_active":"2024-04-02T05:05:16.544892+0000","last_became_peered":"2024-04-02T05:05:16.544892+0000","last_unstale":"2024-04-02T05:08:05.002117+0000","last_undegraded":"2024-04-02T05:08:05.002117+0000","last_fullsized":"2024-04-02T05:08:05.002117+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:30.462995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:24.327579+0000","last_clean_scrub_stamp":"2024-04-02T05:06:30.462995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"905'36","reported_seq":185,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003549+0000","last_change":"2024-04-02T05:07:21.201775+0000","last_active":"2024-04-02T05:08:05.003549+0000","last_peered":"2024-04-02T05:08:05.003549+0000","last_clean":"2024-04-02T05:08:05.003549+0000","last_became_active":"2024-04-02T05:04:11.251941+0000","last_became_peered":"2024-04-02T05:04:11.251941+0000","last_unstale":"2024-04-02T05:08:05.003549+0000","last_undegraded":"2024-04-02T05:08:05.003549+0000","last_fullsized":"2024-04-02T05:08:05.003549+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'36","last_scrub_stamp":"2024-04-02T05:07:21.201423+0000","last_deep_scrub":"905'36","last_deep_scrub_stamp":"2024-04-02T05:04:15.805570+0000","last_clean_scrub_stamp":"2024-04-02T05:07:21.201423+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.087735+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1896,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668437+0000","last_change":"2024-04-02T05:07:42.689166+0000","last_active":"2024-04-02T05:08:04.668437+0000","last_peered":"2024-04-02T05:08:04.668437+0000","last_clean":"2024-04-02T05:08:04.668437+0000","last_became_active":"2024-04-02T05:05:16.546077+0000","last_became_peered":"2024-04-02T05:05:16.546077+0000","last_unstale":"2024-04-02T05:08:04.668437+0000","last_undegraded":"2024-04-02T05:08:04.668437+0000","last_fullsized":"2024-04-02T05:08:04.668437+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:42.688963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:46.514376+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.688963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:51.992363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":134,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003917+0000","last_change":"2024-04-02T05:06:58.250803+0000","last_active":"2024-04-02T05:08:05.003917+0000","last_peered":"2024-04-02T05:08:05.003917+0000","last_clean":"2024-04-02T05:08:05.003917+0000","last_became_active":"2024-04-02T05:04:57.347965+0000","last_became_peered":"2024-04-02T05:04:57.347965+0000","last_unstale":"2024-04-02T05:08:05.003917+0000","last_undegraded":"2024-04-02T05:08:05.003917+0000","last_fullsized":"2024-04-02T05:08:05.003917+0000","mapping_epoch":914,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":915,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:58.250588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:58.250588+0000","last_clean_scrub_stamp":"2024-04-02T05:06:58.250588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:08.380443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":722,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398250+0000","last_change":"2024-04-02T05:07:00.366446+0000","last_active":"2024-04-02T05:08:04.398250+0000","last_peered":"2024-04-02T05:08:04.398250+0000","last_clean":"2024-04-02T05:08:04.398250+0000","last_became_active":"2024-04-02T05:06:31.702170+0000","last_became_peered":"2024-04-02T05:06:31.702170+0000","last_unstale":"2024-04-02T05:08:04.398250+0000","last_undegraded":"2024-04-02T05:08:04.398250+0000","last_fullsized":"2024-04-02T05:08:04.398250+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:00.366335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:30.933150+0000","last_clean_scrub_stamp":"2024-04-02T05:07:00.366335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:27.483325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":81,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395599+0000","last_change":"2024-04-02T05:07:36.602643+0000","last_active":"2024-04-02T05:08:04.395599+0000","last_peered":"2024-04-02T05:08:04.395599+0000","last_clean":"2024-04-02T05:08:04.395599+0000","last_became_active":"2024-04-02T05:05:54.288794+0000","last_became_peered":"2024-04-02T05:05:54.288794+0000","last_unstale":"2024-04-02T05:08:04.395599+0000","last_undegraded":"2024-04-02T05:08:04.395599+0000","last_fullsized":"2024-04-02T05:08:04.395599+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:36.602412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:36.602412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:02.228203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"899'74","reported_seq":2104,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395103+0000","last_change":"2024-04-02T05:07:19.776641+0000","last_active":"2024-04-02T05:08:04.395103+0000","last_peered":"2024-04-02T05:08:04.395103+0000","last_clean":"2024-04-02T05:08:04.395103+0000","last_became_active":"2024-04-02T05:05:54.287852+0000","last_became_peered":"2024-04-02T05:05:54.287852+0000","last_unstale":"2024-04-02T05:08:04.395103+0000","last_undegraded":"2024-04-02T05:08:04.395103+0000","last_fullsized":"2024-04-02T05:08:04.395103+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'74","last_scrub_stamp":"2024-04-02T05:07:19.776365+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.776365+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:39.952209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002401+0000","last_change":"2024-04-02T05:07:28.183493+0000","last_active":"2024-04-02T05:08:05.002401+0000","last_peered":"2024-04-02T05:08:05.002401+0000","last_clean":"2024-04-02T05:08:05.002401+0000","last_became_active":"2024-04-02T05:04:11.252720+0000","last_became_peered":"2024-04-02T05:04:11.252720+0000","last_unstale":"2024-04-02T05:08:05.002401+0000","last_undegraded":"2024-04-02T05:08:05.002401+0000","last_fullsized":"2024-04-02T05:08:05.002401+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:28.183264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:28.183264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.041853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1883,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073883+0000","last_change":"2024-04-02T05:07:44.142715+0000","last_active":"2024-04-02T05:08:05.073883+0000","last_peered":"2024-04-02T05:08:05.073883+0000","last_clean":"2024-04-02T05:08:05.073883+0000","last_became_active":"2024-04-02T04:59:00.138125+0000","last_became_peered":"2024-04-02T04:59:00.138125+0000","last_unstale":"2024-04-02T05:08:05.073883+0000","last_undegraded":"2024-04-02T05:08:05.073883+0000","last_fullsized":"2024-04-02T05:08:05.073883+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:44.142492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:44.142492+0000","last_clean_scrub_stamp":"2024-04-02T05:07:44.142492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:05.673220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"958'2","reported_seq":1365,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006909+0000","last_change":"2024-04-02T05:06:48.498344+0000","last_active":"2024-04-02T05:08:05.006909+0000","last_peered":"2024-04-02T05:08:05.006909+0000","last_clean":"2024-04-02T05:08:05.006909+0000","last_became_active":"2024-04-02T05:04:11.237558+0000","last_became_peered":"2024-04-02T05:04:11.237558+0000","last_unstale":"2024-04-02T05:08:05.006909+0000","last_undegraded":"2024-04-02T05:08:05.006909+0000","last_fullsized":"2024-04-02T05:08:05.006909+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'2","last_scrub_stamp":"2024-04-02T05:06:48.497980+0000","last_deep_scrub":"958'2","last_deep_scrub_stamp":"2024-04-02T05:06:48.497980+0000","last_clean_scrub_stamp":"2024-04-02T05:06:48.497980+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004379+0000","last_change":"2024-04-02T05:07:23.226101+0000","last_active":"2024-04-02T05:08:05.004379+0000","last_peered":"2024-04-02T05:08:05.004379+0000","last_clean":"2024-04-02T05:08:05.004379+0000","last_became_active":"2024-04-02T05:04:11.244814+0000","last_became_peered":"2024-04-02T05:04:11.244814+0000","last_unstale":"2024-04-02T05:08:05.004379+0000","last_undegraded":"2024-04-02T05:08:05.004379+0000","last_fullsized":"2024-04-02T05:08:05.004379+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:23.225832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:39.087140+0000","last_clean_scrub_stamp":"2024-04-02T05:07:23.225832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.274991+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"957'32","reported_seq":724,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395016+0000","last_change":"2024-04-02T05:06:40.470123+0000","last_active":"2024-04-02T05:08:04.395016+0000","last_peered":"2024-04-02T05:08:04.395016+0000","last_clean":"2024-04-02T05:08:04.395016+0000","last_became_active":"2024-04-02T04:59:00.133276+0000","last_became_peered":"2024-04-02T04:59:00.133276+0000","last_unstale":"2024-04-02T05:08:04.395016+0000","last_undegraded":"2024-04-02T05:08:04.395016+0000","last_fullsized":"2024-04-02T05:08:04.395016+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'32","last_scrub_stamp":"2024-04-02T05:06:40.469957+0000","last_deep_scrub":"957'32","last_deep_scrub_stamp":"2024-04-02T05:06:40.469957+0000","last_clean_scrub_stamp":"2024-04-02T05:06:40.469957+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:05.415128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":710,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400234+0000","last_change":"2024-04-02T05:07:34.628774+0000","last_active":"2024-04-02T05:08:04.400234+0000","last_peered":"2024-04-02T05:08:04.400234+0000","last_clean":"2024-04-02T05:08:04.400234+0000","last_became_active":"2024-04-02T04:59:00.135857+0000","last_became_peered":"2024-04-02T04:59:00.135857+0000","last_unstale":"2024-04-02T05:08:04.400234+0000","last_undegraded":"2024-04-02T05:08:04.400234+0000","last_fullsized":"2024-04-02T05:08:04.400234+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T05:07:34.628682+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T05:07:34.628682+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:54.427606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"902'13","reported_seq":2181,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668665+0000","last_change":"2024-04-02T05:07:31.758360+0000","last_active":"2024-04-02T05:08:04.668665+0000","last_peered":"2024-04-02T05:08:04.668665+0000","last_clean":"2024-04-02T05:08:04.668665+0000","last_became_active":"2024-04-02T05:04:11.260020+0000","last_became_peered":"2024-04-02T05:04:11.260020+0000","last_unstale":"2024-04-02T05:08:04.668665+0000","last_undegraded":"2024-04-02T05:08:04.668665+0000","last_fullsized":"2024-04-02T05:08:04.668665+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'13","last_scrub_stamp":"2024-04-02T05:07:31.758131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:31.758131+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.441886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1880,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401871+0000","last_change":"2024-04-02T05:07:16.856612+0000","last_active":"2024-04-02T05:08:04.401871+0000","last_peered":"2024-04-02T05:08:04.401871+0000","last_clean":"2024-04-02T05:08:04.401871+0000","last_became_active":"2024-04-02T05:05:54.275152+0000","last_became_peered":"2024-04-02T05:05:54.275152+0000","last_unstale":"2024-04-02T05:08:04.401871+0000","last_undegraded":"2024-04-02T05:08:04.401871+0000","last_fullsized":"2024-04-02T05:08:04.401871+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.856401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:20.038583+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.856401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:43.774861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1744,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667117+0000","last_change":"2024-04-02T05:07:58.937172+0000","last_active":"2024-04-02T05:08:04.667117+0000","last_peered":"2024-04-02T05:08:04.667117+0000","last_clean":"2024-04-02T05:08:04.667117+0000","last_became_active":"2024-04-02T05:05:16.544183+0000","last_became_peered":"2024-04-02T05:05:16.544183+0000","last_unstale":"2024-04-02T05:08:04.667117+0000","last_undegraded":"2024-04-02T05:08:04.667117+0000","last_fullsized":"2024-04-02T05:08:04.667117+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:58.936944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:20.453979+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.936944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.018261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"899'18","reported_seq":1895,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.072966+0000","last_change":"2024-04-02T05:07:06.997018+0000","last_active":"2024-04-02T05:08:05.072966+0000","last_peered":"2024-04-02T05:08:05.072966+0000","last_clean":"2024-04-02T05:08:05.072966+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T05:08:05.072966+0000","last_undegraded":"2024-04-02T05:08:05.072966+0000","last_fullsized":"2024-04-02T05:08:05.072966+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'18","last_scrub_stamp":"2024-04-02T05:07:06.996892+0000","last_deep_scrub":"899'18","last_deep_scrub_stamp":"2024-04-02T05:05:22.354664+0000","last_clean_scrub_stamp":"2024-04-02T05:07:06.996892+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:13.189718+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667426+0000","last_change":"2024-04-02T05:07:53.940505+0000","last_active":"2024-04-02T05:08:04.667426+0000","last_peered":"2024-04-02T05:08:04.667426+0000","last_clean":"2024-04-02T05:08:04.667426+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T05:08:04.667426+0000","last_undegraded":"2024-04-02T05:08:04.667426+0000","last_fullsized":"2024-04-02T05:08:04.667426+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:53.940291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.940291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:11.453458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871122+0000","last_change":"2024-04-02T05:06:54.680012+0000","last_active":"2024-04-02T05:08:04.871122+0000","last_peered":"2024-04-02T05:08:04.871122+0000","last_clean":"2024-04-02T05:08:04.871122+0000","last_became_active":"2024-04-02T05:05:16.542585+0000","last_became_peered":"2024-04-02T05:05:16.542585+0000","last_unstale":"2024-04-02T05:08:04.871122+0000","last_undegraded":"2024-04-02T05:08:04.871122+0000","last_fullsized":"2024-04-02T05:08:04.871122+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:54.679771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:06:54.679771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:20.134874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":74,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398737+0000","last_change":"2024-04-02T05:07:20.508239+0000","last_active":"2024-04-02T05:08:04.398737+0000","last_peered":"2024-04-02T05:08:04.398737+0000","last_clean":"2024-04-02T05:08:04.398737+0000","last_became_active":"2024-04-02T05:05:54.282177+0000","last_became_peered":"2024-04-02T05:05:54.282177+0000","last_unstale":"2024-04-02T05:08:04.398737+0000","last_undegraded":"2024-04-02T05:08:04.398737+0000","last_fullsized":"2024-04-02T05:08:04.398737+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:20.507993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:20.507993+0000","last_clean_scrub_stamp":"2024-04-02T05:07:20.507993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:21.875519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1739,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397654+0000","last_change":"2024-04-02T05:06:54.885190+0000","last_active":"2024-04-02T05:08:04.397654+0000","last_peered":"2024-04-02T05:08:04.397654+0000","last_clean":"2024-04-02T05:08:04.397654+0000","last_became_active":"2024-04-02T05:05:16.540588+0000","last_became_peered":"2024-04-02T05:05:16.540588+0000","last_unstale":"2024-04-02T05:08:04.397654+0000","last_undegraded":"2024-04-02T05:08:04.397654+0000","last_fullsized":"2024-04-02T05:08:04.397654+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:54.885053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:06:54.885053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:12.014983+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"899'41","reported_seq":1986,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397169+0000","last_change":"2024-04-02T05:07:42.612118+0000","last_active":"2024-04-02T05:08:04.397169+0000","last_peered":"2024-04-02T05:08:04.397169+0000","last_clean":"2024-04-02T05:08:04.397169+0000","last_became_active":"2024-04-02T05:05:16.536332+0000","last_became_peered":"2024-04-02T05:05:16.536332+0000","last_unstale":"2024-04-02T05:08:04.397169+0000","last_undegraded":"2024-04-02T05:08:04.397169+0000","last_fullsized":"2024-04-02T05:08:04.397169+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'41","last_scrub_stamp":"2024-04-02T05:07:42.611837+0000","last_deep_scrub":"899'41","last_deep_scrub_stamp":"2024-04-02T05:06:18.573690+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.611837+0000","objects_scrubbed":6,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:11.875159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396939+0000","last_change":"2024-04-02T05:07:17.678571+0000","last_active":"2024-04-02T05:08:04.396939+0000","last_peered":"2024-04-02T05:08:04.396939+0000","last_clean":"2024-04-02T05:08:04.396939+0000","last_became_active":"2024-04-02T04:59:00.122388+0000","last_became_peered":"2024-04-02T04:59:00.122388+0000","last_unstale":"2024-04-02T05:08:04.396939+0000","last_undegraded":"2024-04-02T05:08:04.396939+0000","last_fullsized":"2024-04-02T05:08:04.396939+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:17.678355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:17.678355+0000","last_clean_scrub_stamp":"2024-04-02T05:07:17.678355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:26.045987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399150+0000","last_change":"2024-04-02T05:07:23.532355+0000","last_active":"2024-04-02T05:08:04.399150+0000","last_peered":"2024-04-02T05:08:04.399150+0000","last_clean":"2024-04-02T05:08:04.399150+0000","last_became_active":"2024-04-02T04:59:00.131269+0000","last_became_peered":"2024-04-02T04:59:00.131269+0000","last_unstale":"2024-04-02T05:08:04.399150+0000","last_undegraded":"2024-04-02T05:08:04.399150+0000","last_fullsized":"2024-04-02T05:08:04.399150+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:23.532250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:23.532250+0000","last_clean_scrub_stamp":"2024-04-02T05:07:23.532250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:30.212439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":719,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.403477+0000","last_change":"2024-04-02T05:07:44.609515+0000","last_active":"2024-04-02T05:08:04.403477+0000","last_peered":"2024-04-02T05:08:04.403477+0000","last_clean":"2024-04-02T05:08:04.403477+0000","last_became_active":"2024-04-02T05:04:11.244470+0000","last_became_peered":"2024-04-02T05:04:11.244470+0000","last_unstale":"2024-04-02T05:08:04.403477+0000","last_undegraded":"2024-04-02T05:08:04.403477+0000","last_fullsized":"2024-04-02T05:08:04.403477+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:44.609296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:00.807998+0000","last_clean_scrub_stamp":"2024-04-02T05:07:44.609296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.947311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1746,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.707518+0000","last_change":"2024-04-02T05:08:05.707578+0000","last_active":"2024-04-02T05:08:05.707518+0000","last_peered":"2024-04-02T05:08:05.707518+0000","last_clean":"2024-04-02T05:08:05.707518+0000","last_became_active":"2024-04-02T05:05:54.280088+0000","last_became_peered":"2024-04-02T05:05:54.280088+0000","last_unstale":"2024-04-02T05:08:05.707518+0000","last_undegraded":"2024-04-02T05:08:05.707518+0000","last_fullsized":"2024-04-02T05:08:05.707518+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:05.707349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T05:08:05.707349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:31.428024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"899'30","reported_seq":182,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005137+0000","last_change":"2024-04-02T05:07:35.119158+0000","last_active":"2024-04-02T05:08:05.005137+0000","last_peered":"2024-04-02T05:08:05.005137+0000","last_clean":"2024-04-02T05:08:05.005137+0000","last_became_active":"2024-04-02T05:04:11.242466+0000","last_became_peered":"2024-04-02T05:04:11.242466+0000","last_unstale":"2024-04-02T05:08:05.005137+0000","last_undegraded":"2024-04-02T05:08:05.005137+0000","last_fullsized":"2024-04-02T05:08:05.005137+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'30","last_scrub_stamp":"2024-04-02T05:07:35.118899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:35.118899+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:03.285725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"920'2","reported_seq":1375,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006434+0000","last_change":"2024-04-02T05:08:00.541852+0000","last_active":"2024-04-02T05:08:05.006434+0000","last_peered":"2024-04-02T05:08:05.006434+0000","last_clean":"2024-04-02T05:08:05.006434+0000","last_became_active":"2024-04-02T05:04:11.238352+0000","last_became_peered":"2024-04-02T05:04:11.238352+0000","last_unstale":"2024-04-02T05:08:05.006434+0000","last_undegraded":"2024-04-02T05:08:05.006434+0000","last_fullsized":"2024-04-02T05:08:05.006434+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'2","last_scrub_stamp":"2024-04-02T05:08:00.541348+0000","last_deep_scrub":"920'2","last_deep_scrub_stamp":"2024-04-02T05:05:09.467910+0000","last_clean_scrub_stamp":"2024-04-02T05:08:00.541348+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:29.978854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":81,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.668827+0000","last_change":"2024-04-02T05:08:07.668886+0000","last_active":"2024-04-02T05:08:07.668827+0000","last_peered":"2024-04-02T05:08:07.668827+0000","last_clean":"2024-04-02T05:08:07.668827+0000","last_became_active":"2024-04-02T05:05:54.277955+0000","last_became_peered":"2024-04-02T05:05:54.277955+0000","last_unstale":"2024-04-02T05:08:07.668827+0000","last_undegraded":"2024-04-02T05:08:07.668827+0000","last_fullsized":"2024-04-02T05:08:07.668827+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:07.668668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:07.668668+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.668668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:21.874085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1856,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.981385+0000","last_change":"2024-04-02T05:07:30.024340+0000","last_active":"2024-04-02T05:08:04.981385+0000","last_peered":"2024-04-02T05:08:04.981385+0000","last_clean":"2024-04-02T05:08:04.981385+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T05:08:04.981385+0000","last_undegraded":"2024-04-02T05:08:04.981385+0000","last_fullsized":"2024-04-02T05:08:04.981385+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:30.024251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:25.384498+0000","last_clean_scrub_stamp":"2024-04-02T05:07:30.024251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:53.695979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":704,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395553+0000","last_change":"2024-04-02T05:06:58.351459+0000","last_active":"2024-04-02T05:08:04.395553+0000","last_peered":"2024-04-02T05:08:04.395553+0000","last_clean":"2024-04-02T05:08:04.395553+0000","last_became_active":"2024-04-02T05:04:11.244083+0000","last_became_peered":"2024-04-02T05:04:11.244083+0000","last_unstale":"2024-04-02T05:08:04.395553+0000","last_undegraded":"2024-04-02T05:08:04.395553+0000","last_fullsized":"2024-04-02T05:08:04.395553+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:58.351248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:34.065387+0000","last_clean_scrub_stamp":"2024-04-02T05:06:58.351248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:07.119715+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"899'39","reported_seq":184,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004314+0000","last_change":"2024-04-02T05:07:56.246390+0000","last_active":"2024-04-02T05:08:05.004314+0000","last_peered":"2024-04-02T05:08:05.004314+0000","last_clean":"2024-04-02T05:08:05.004314+0000","last_became_active":"2024-04-02T05:04:11.243362+0000","last_became_peered":"2024-04-02T05:04:11.243362+0000","last_unstale":"2024-04-02T05:08:05.004314+0000","last_undegraded":"2024-04-02T05:08:05.004314+0000","last_fullsized":"2024-04-02T05:08:05.004314+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'39","last_scrub_stamp":"2024-04-02T05:07:56.246150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:56.246150+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:07.331098+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1349,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006439+0000","last_change":"2024-04-02T05:07:18.632363+0000","last_active":"2024-04-02T05:08:05.006439+0000","last_peered":"2024-04-02T05:08:05.006439+0000","last_clean":"2024-04-02T05:08:05.006439+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T05:08:05.006439+0000","last_undegraded":"2024-04-02T05:08:05.006439+0000","last_fullsized":"2024-04-02T05:08:05.006439+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:18.632111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:18.632111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.990446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":81,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396628+0000","last_change":"2024-04-02T05:07:53.740460+0000","last_active":"2024-04-02T05:08:04.396628+0000","last_peered":"2024-04-02T05:08:04.396628+0000","last_clean":"2024-04-02T05:08:04.396628+0000","last_became_active":"2024-04-02T05:05:54.275404+0000","last_became_peered":"2024-04-02T05:05:54.275404+0000","last_unstale":"2024-04-02T05:08:04.396628+0000","last_undegraded":"2024-04-02T05:08:04.396628+0000","last_fullsized":"2024-04-02T05:08:04.396628+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:53.740246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.740246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:10.903946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005078+0000","last_change":"2024-04-02T05:07:42.178758+0000","last_active":"2024-04-02T05:08:05.005078+0000","last_peered":"2024-04-02T05:08:05.005078+0000","last_clean":"2024-04-02T05:08:05.005078+0000","last_became_active":"2024-04-02T05:04:11.225907+0000","last_became_peered":"2024-04-02T05:04:11.225907+0000","last_unstale":"2024-04-02T05:08:05.005078+0000","last_undegraded":"2024-04-02T05:08:05.005078+0000","last_fullsized":"2024-04-02T05:08:05.005078+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:42.178550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.178550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:57.864068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1729,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667699+0000","last_change":"2024-04-02T05:07:16.758675+0000","last_active":"2024-04-02T05:08:04.667699+0000","last_peered":"2024-04-02T05:08:04.667699+0000","last_clean":"2024-04-02T05:08:04.667699+0000","last_became_active":"2024-04-02T05:04:11.247660+0000","last_became_peered":"2024-04-02T05:04:11.247660+0000","last_unstale":"2024-04-02T05:08:04.667699+0000","last_undegraded":"2024-04-02T05:08:04.667699+0000","last_fullsized":"2024-04-02T05:08:04.667699+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.758458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.758458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:27.103181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"899'33","reported_seq":2020,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074275+0000","last_change":"2024-04-02T05:07:09.912538+0000","last_active":"2024-04-02T05:08:05.074275+0000","last_peered":"2024-04-02T05:08:05.074275+0000","last_clean":"2024-04-02T05:08:05.074275+0000","last_became_active":"2024-04-02T05:05:16.543343+0000","last_became_peered":"2024-04-02T05:05:16.543343+0000","last_unstale":"2024-04-02T05:08:05.074275+0000","last_undegraded":"2024-04-02T05:08:05.074275+0000","last_fullsized":"2024-04-02T05:08:05.074275+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'33","last_scrub_stamp":"2024-04-02T05:07:09.912173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:09.912173+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:11.648215+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400951+0000","last_change":"2024-04-02T05:07:51.512534+0000","last_active":"2024-04-02T05:08:04.400951+0000","last_peered":"2024-04-02T05:08:04.400951+0000","last_clean":"2024-04-02T05:08:04.400951+0000","last_became_active":"2024-04-02T05:04:11.244435+0000","last_became_peered":"2024-04-02T05:04:11.244435+0000","last_unstale":"2024-04-02T05:08:04.400951+0000","last_undegraded":"2024-04-02T05:08:04.400951+0000","last_fullsized":"2024-04-02T05:08:04.400951+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:51.512226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_clean_scrub_stamp":"2024-04-02T05:07:51.512226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.909126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1868,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395415+0000","last_change":"2024-04-02T05:06:56.489013+0000","last_active":"2024-04-02T05:08:04.395415+0000","last_peered":"2024-04-02T05:08:04.395415+0000","last_clean":"2024-04-02T05:08:04.395415+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T05:08:04.395415+0000","last_undegraded":"2024-04-02T05:08:04.395415+0000","last_fullsized":"2024-04-02T05:08:04.395415+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:56.488795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T05:06:56.488795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:12.379473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396917+0000","last_change":"2024-04-02T05:07:05.543736+0000","last_active":"2024-04-02T05:08:04.396917+0000","last_peered":"2024-04-02T05:08:04.396917+0000","last_clean":"2024-04-02T05:08:04.396917+0000","last_became_active":"2024-04-02T05:04:11.255981+0000","last_became_peered":"2024-04-02T05:04:11.255981+0000","last_unstale":"2024-04-02T05:08:04.396917+0000","last_undegraded":"2024-04-02T05:08:04.396917+0000","last_fullsized":"2024-04-02T05:08:04.396917+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:05.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_clean_scrub_stamp":"2024-04-02T05:07:05.543525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:31.265009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1753,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399001+0000","last_change":"2024-04-02T05:07:50.697827+0000","last_active":"2024-04-02T05:08:04.399001+0000","last_peered":"2024-04-02T05:08:04.399001+0000","last_clean":"2024-04-02T05:08:04.399001+0000","last_became_active":"2024-04-02T04:59:00.135538+0000","last_became_peered":"2024-04-02T04:59:00.135538+0000","last_unstale":"2024-04-02T05:08:04.399001+0000","last_undegraded":"2024-04-02T05:08:04.399001+0000","last_fullsized":"2024-04-02T05:08:04.399001+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:50.697605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:50.697605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.144527+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"899'42","reported_seq":776,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400678+0000","last_change":"2024-04-02T05:07:33.588845+0000","last_active":"2024-04-02T05:08:04.400678+0000","last_peered":"2024-04-02T05:08:04.400678+0000","last_clean":"2024-04-02T05:08:04.400678+0000","last_became_active":"2024-04-02T05:05:16.544863+0000","last_became_peered":"2024-04-02T05:05:16.544863+0000","last_unstale":"2024-04-02T05:08:04.400678+0000","last_undegraded":"2024-04-02T05:08:04.400678+0000","last_fullsized":"2024-04-02T05:08:04.400678+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'42","last_scrub_stamp":"2024-04-02T05:07:33.588619+0000","last_deep_scrub":"402'32","last_deep_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_clean_scrub_stamp":"2024-04-02T05:07:33.588619+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.716255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1743,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074583+0000","last_change":"2024-04-02T05:07:43.113383+0000","last_active":"2024-04-02T05:08:05.074583+0000","last_peered":"2024-04-02T05:08:05.074583+0000","last_clean":"2024-04-02T05:08:05.074583+0000","last_became_active":"2024-04-02T05:05:54.272364+0000","last_became_peered":"2024-04-02T05:05:54.272364+0000","last_unstale":"2024-04-02T05:08:05.074583+0000","last_undegraded":"2024-04-02T05:08:05.074583+0000","last_fullsized":"2024-04-02T05:08:05.074583+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:43.113176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:43.113176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.599605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":716,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400502+0000","last_change":"2024-04-02T05:06:55.339683+0000","last_active":"2024-04-02T05:08:04.400502+0000","last_peered":"2024-04-02T05:08:04.400502+0000","last_clean":"2024-04-02T05:08:04.400502+0000","last_became_active":"2024-04-02T05:05:16.544759+0000","last_became_peered":"2024-04-02T05:05:16.544759+0000","last_unstale":"2024-04-02T05:08:04.400502+0000","last_undegraded":"2024-04-02T05:08:04.400502+0000","last_fullsized":"2024-04-02T05:08:04.400502+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:55.339468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:01:14.663348+0000","last_clean_scrub_stamp":"2024-04-02T05:06:55.339468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:13.584923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396681+0000","last_change":"2024-04-02T05:07:24.505337+0000","last_active":"2024-04-02T05:08:04.396681+0000","last_peered":"2024-04-02T05:08:04.396681+0000","last_clean":"2024-04-02T05:08:04.396681+0000","last_became_active":"2024-04-02T04:59:00.122322+0000","last_became_peered":"2024-04-02T04:59:00.122322+0000","last_unstale":"2024-04-02T05:08:04.396681+0000","last_undegraded":"2024-04-02T05:08:04.396681+0000","last_fullsized":"2024-04-02T05:08:04.396681+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:24.505123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:52.064373+0000","last_clean_scrub_stamp":"2024-04-02T05:07:24.505123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:39.647086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1818,"num_read_kb":1543,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":7284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1293,"ondisk_log_size":1293,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":553,"num_read_kb":548,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114798,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56976,"kb_used_data":2440,"kb_used_omap":57,"kb_used_meta":54470,"kb_avail":93688176,"statfs":{"total":95995035648,"available":95936692224,"internally_reserved":0,"allocated":2498560,"data_stored":723609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":55778190},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103481,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101936,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93643216,"statfs":{"total":95995035648,"available":95890653184,"internally_reserved":0,"allocated":2789376,"data_stored":1049097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575212,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114108,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93631044,"statfs":{"total":95995035648,"available":95878189056,"internally_reserved":0,"allocated":2478080,"data_stored":722497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738742,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114432,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630720,"statfs":{"total":95995035648,"available":95877857280,"internally_reserved":0,"allocated":2809856,"data_stored":1049766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378978,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114364,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630788,"statfs":{"total":95995035648,"available":95877926912,"internally_reserved":0,"allocated":2736128,"data_stored":985531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051928,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114676,"kb_used_data":2984,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630476,"statfs":{"total":95995035648,"available":95877607424,"internally_reserved":0,"allocated":3055616,"data_stored":1312166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822990,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96084,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649068,"statfs":{"total":95995035648,"available":95896645632,"internally_reserved":0,"allocated":2502656,"data_stored":722621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463234,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51892,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93693260,"statfs":{"total":95995035648,"available":95941898240,"internally_reserved":0,"allocated":2732032,"data_stored":985330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T05:08:15.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T05:08:15.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[21528]: audit 2024-04-02T05:08:14.022910+0000 mgr.y (mgr.14538) 934 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:15.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[21528]: cluster 2024-04-02T05:08:14.032670+0000 osd.0 (osd.0) 571 : cluster 0 3.1f scrub starts 2024-04-02T05:08:15.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[21528]: cluster 2024-04-02T05:08:14.035682+0000 osd.0 (osd.0) 572 : cluster 0 3.1f scrub ok 2024-04-02T05:08:15.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[21528]: audit 2024-04-02T05:08:14.396210+0000 mgr.y (mgr.14538) 935 : audit 0 from='client.34068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:08:15.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[21528]: cluster 2024-04-02T05:08:14.523350+0000 mgr.y (mgr.14538) 936 : cluster 0 pgmap v1572: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-04-02T05:08:15.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[28904]: audit 2024-04-02T05:08:14.022910+0000 mgr.y (mgr.14538) 934 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:15.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[28904]: cluster 2024-04-02T05:08:14.032670+0000 osd.0 (osd.0) 571 : cluster 0 3.1f scrub starts 2024-04-02T05:08:15.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[28904]: cluster 2024-04-02T05:08:14.035682+0000 osd.0 (osd.0) 572 : cluster 0 3.1f scrub ok 2024-04-02T05:08:15.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[28904]: audit 2024-04-02T05:08:14.396210+0000 mgr.y (mgr.14538) 935 : audit 0 from='client.34068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:08:15.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:15 smithi069 bash[28904]: cluster 2024-04-02T05:08:14.523350+0000 mgr.y (mgr.14538) 936 : cluster 0 pgmap v1572: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-04-02T05:08:15.723 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:15 smithi098 bash[23790]: audit 2024-04-02T05:08:14.022910+0000 mgr.y (mgr.14538) 934 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:15 smithi098 bash[23790]: cluster 2024-04-02T05:08:14.032670+0000 osd.0 (osd.0) 571 : cluster 0 3.1f scrub starts 2024-04-02T05:08:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:15 smithi098 bash[23790]: cluster 2024-04-02T05:08:14.035682+0000 osd.0 (osd.0) 572 : cluster 0 3.1f scrub ok 2024-04-02T05:08:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:15 smithi098 bash[23790]: audit 2024-04-02T05:08:14.396210+0000 mgr.y (mgr.14538) 935 : audit 0 from='client.34068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:08:15.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:15 smithi098 bash[23790]: cluster 2024-04-02T05:08:14.523350+0000 mgr.y (mgr.14538) 936 : cluster 0 pgmap v1572: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-04-02T05:08:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[21528]: cluster 2024-04-02T05:08:15.629265+0000 osd.1 (osd.1) 613 : cluster 0 2.6 scrub starts 2024-04-02T05:08:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[21528]: cluster 2024-04-02T05:08:15.632631+0000 osd.1 (osd.1) 614 : cluster 0 2.6 scrub ok 2024-04-02T05:08:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[21528]: cluster 2024-04-02T05:08:15.734676+0000 osd.7 (osd.7) 411 : cluster 0 10.1f deep-scrub starts 2024-04-02T05:08:16.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[21528]: cluster 2024-04-02T05:08:15.742924+0000 osd.7 (osd.7) 412 : cluster 0 10.1f deep-scrub ok 2024-04-02T05:08:16.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[28904]: cluster 2024-04-02T05:08:15.629265+0000 osd.1 (osd.1) 613 : cluster 0 2.6 scrub starts 2024-04-02T05:08:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[28904]: cluster 2024-04-02T05:08:15.632631+0000 osd.1 (osd.1) 614 : cluster 0 2.6 scrub ok 2024-04-02T05:08:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[28904]: cluster 2024-04-02T05:08:15.734676+0000 osd.7 (osd.7) 411 : cluster 0 10.1f deep-scrub starts 2024-04-02T05:08:16.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:16 smithi069 bash[28904]: cluster 2024-04-02T05:08:15.742924+0000 osd.7 (osd.7) 412 : cluster 0 10.1f deep-scrub ok 2024-04-02T05:08:16.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:16 smithi098 bash[23790]: cluster 2024-04-02T05:08:15.629265+0000 osd.1 (osd.1) 613 : cluster 0 2.6 scrub starts 2024-04-02T05:08:16.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:16 smithi098 bash[23790]: cluster 2024-04-02T05:08:15.632631+0000 osd.1 (osd.1) 614 : cluster 0 2.6 scrub ok 2024-04-02T05:08:16.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:16 smithi098 bash[23790]: cluster 2024-04-02T05:08:15.734676+0000 osd.7 (osd.7) 411 : cluster 0 10.1f deep-scrub starts 2024-04-02T05:08:16.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:16 smithi098 bash[23790]: cluster 2024-04-02T05:08:15.742924+0000 osd.7 (osd.7) 412 : cluster 0 10.1f deep-scrub ok 2024-04-02T05:08:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[21528]: cluster 2024-04-02T05:08:16.524133+0000 mgr.y (mgr.14538) 937 : cluster 0 pgmap v1573: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[21528]: cluster 2024-04-02T05:08:16.581742+0000 osd.1 (osd.1) 615 : cluster 0 10.13 scrub starts 2024-04-02T05:08:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[21528]: cluster 2024-04-02T05:08:16.587025+0000 osd.1 (osd.1) 616 : cluster 0 10.13 scrub ok 2024-04-02T05:08:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[21528]: cluster 2024-04-02T05:08:17.023329+0000 osd.5 (osd.5) 431 : cluster 0 4.11 scrub starts 2024-04-02T05:08:17.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[21528]: cluster 2024-04-02T05:08:17.025595+0000 osd.5 (osd.5) 432 : cluster 0 4.11 scrub ok 2024-04-02T05:08:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[28904]: cluster 2024-04-02T05:08:16.524133+0000 mgr.y (mgr.14538) 937 : cluster 0 pgmap v1573: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[28904]: cluster 2024-04-02T05:08:16.581742+0000 osd.1 (osd.1) 615 : cluster 0 10.13 scrub starts 2024-04-02T05:08:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[28904]: cluster 2024-04-02T05:08:16.587025+0000 osd.1 (osd.1) 616 : cluster 0 10.13 scrub ok 2024-04-02T05:08:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[28904]: cluster 2024-04-02T05:08:17.023329+0000 osd.5 (osd.5) 431 : cluster 0 4.11 scrub starts 2024-04-02T05:08:17.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:17 smithi069 bash[28904]: cluster 2024-04-02T05:08:17.025595+0000 osd.5 (osd.5) 432 : cluster 0 4.11 scrub ok 2024-04-02T05:08:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:17 smithi098 bash[23790]: cluster 2024-04-02T05:08:16.524133+0000 mgr.y (mgr.14538) 937 : cluster 0 pgmap v1573: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:17.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:17 smithi098 bash[23790]: cluster 2024-04-02T05:08:16.581742+0000 osd.1 (osd.1) 615 : cluster 0 10.13 scrub starts 2024-04-02T05:08:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:17 smithi098 bash[23790]: cluster 2024-04-02T05:08:16.587025+0000 osd.1 (osd.1) 616 : cluster 0 10.13 scrub ok 2024-04-02T05:08:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:17 smithi098 bash[23790]: cluster 2024-04-02T05:08:17.023329+0000 osd.5 (osd.5) 431 : cluster 0 4.11 scrub starts 2024-04-02T05:08:17.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:17 smithi098 bash[23790]: cluster 2024-04-02T05:08:17.025595+0000 osd.5 (osd.5) 432 : cluster 0 4.11 scrub ok 2024-04-02T05:08:18.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:08:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:08:18.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[21528]: cluster 2024-04-02T05:08:16.876185+0000 osd.6 (osd.6) 549 : cluster 0 10.0 scrub starts 2024-04-02T05:08:18.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[21528]: cluster 2024-04-02T05:08:16.878571+0000 osd.6 (osd.6) 550 : cluster 0 10.0 scrub ok 2024-04-02T05:08:18.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[21528]: cluster 2024-04-02T05:08:17.531712+0000 osd.3 (osd.3) 637 : cluster 0 4.10 scrub starts 2024-04-02T05:08:18.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[21528]: cluster 2024-04-02T05:08:17.534107+0000 osd.3 (osd.3) 638 : cluster 0 4.10 scrub ok 2024-04-02T05:08:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[28904]: cluster 2024-04-02T05:08:16.876185+0000 osd.6 (osd.6) 549 : cluster 0 10.0 scrub starts 2024-04-02T05:08:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[28904]: cluster 2024-04-02T05:08:16.878571+0000 osd.6 (osd.6) 550 : cluster 0 10.0 scrub ok 2024-04-02T05:08:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[28904]: cluster 2024-04-02T05:08:17.531712+0000 osd.3 (osd.3) 637 : cluster 0 4.10 scrub starts 2024-04-02T05:08:18.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:18 smithi069 bash[28904]: cluster 2024-04-02T05:08:17.534107+0000 osd.3 (osd.3) 638 : cluster 0 4.10 scrub ok 2024-04-02T05:08:18.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:18 smithi098 bash[23790]: cluster 2024-04-02T05:08:16.876185+0000 osd.6 (osd.6) 549 : cluster 0 10.0 scrub starts 2024-04-02T05:08:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:18 smithi098 bash[23790]: cluster 2024-04-02T05:08:16.878571+0000 osd.6 (osd.6) 550 : cluster 0 10.0 scrub ok 2024-04-02T05:08:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:18 smithi098 bash[23790]: cluster 2024-04-02T05:08:17.531712+0000 osd.3 (osd.3) 637 : cluster 0 4.10 scrub starts 2024-04-02T05:08:18.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:18 smithi098 bash[23790]: cluster 2024-04-02T05:08:17.534107+0000 osd.3 (osd.3) 638 : cluster 0 4.10 scrub ok 2024-04-02T05:08:19.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[21528]: cluster 2024-04-02T05:08:18.489604+0000 osd.3 (osd.3) 639 : cluster 0 4.19 scrub starts 2024-04-02T05:08:19.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[21528]: cluster 2024-04-02T05:08:18.492683+0000 osd.3 (osd.3) 640 : cluster 0 4.19 scrub ok 2024-04-02T05:08:19.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[21528]: cluster 2024-04-02T05:08:18.525928+0000 mgr.y (mgr.14538) 938 : cluster 0 pgmap v1574: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:19.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[21528]: cluster 2024-04-02T05:08:18.680631+0000 osd.7 (osd.7) 413 : cluster 0 6.15 scrub starts 2024-04-02T05:08:19.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[21528]: cluster 2024-04-02T05:08:18.683145+0000 osd.7 (osd.7) 414 : cluster 0 6.15 scrub ok 2024-04-02T05:08:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[28904]: cluster 2024-04-02T05:08:18.489604+0000 osd.3 (osd.3) 639 : cluster 0 4.19 scrub starts 2024-04-02T05:08:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[28904]: cluster 2024-04-02T05:08:18.492683+0000 osd.3 (osd.3) 640 : cluster 0 4.19 scrub ok 2024-04-02T05:08:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[28904]: cluster 2024-04-02T05:08:18.525928+0000 mgr.y (mgr.14538) 938 : cluster 0 pgmap v1574: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[28904]: cluster 2024-04-02T05:08:18.680631+0000 osd.7 (osd.7) 413 : cluster 0 6.15 scrub starts 2024-04-02T05:08:19.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:19 smithi069 bash[28904]: cluster 2024-04-02T05:08:18.683145+0000 osd.7 (osd.7) 414 : cluster 0 6.15 scrub ok 2024-04-02T05:08:19.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:19 smithi098 bash[23790]: cluster 2024-04-02T05:08:18.489604+0000 osd.3 (osd.3) 639 : cluster 0 4.19 scrub starts 2024-04-02T05:08:19.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:19 smithi098 bash[23790]: cluster 2024-04-02T05:08:18.492683+0000 osd.3 (osd.3) 640 : cluster 0 4.19 scrub ok 2024-04-02T05:08:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:19 smithi098 bash[23790]: cluster 2024-04-02T05:08:18.525928+0000 mgr.y (mgr.14538) 938 : cluster 0 pgmap v1574: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:19 smithi098 bash[23790]: cluster 2024-04-02T05:08:18.680631+0000 osd.7 (osd.7) 413 : cluster 0 6.15 scrub starts 2024-04-02T05:08:19.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:19 smithi098 bash[23790]: cluster 2024-04-02T05:08:18.683145+0000 osd.7 (osd.7) 414 : cluster 0 6.15 scrub ok 2024-04-02T05:08:20.030 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:08:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:20 smithi069 bash[28904]: audit 2024-04-02T05:08:19.818988+0000 mon.b (mon.1) 364 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:08:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:20 smithi069 bash[28904]: cluster 2024-04-02T05:08:20.014735+0000 osd.5 (osd.5) 433 : cluster 0 1.10 deep-scrub starts 2024-04-02T05:08:20.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:20 smithi069 bash[28904]: cluster 2024-04-02T05:08:20.016717+0000 osd.5 (osd.5) 434 : cluster 0 1.10 deep-scrub ok 2024-04-02T05:08:20.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:20 smithi069 bash[21528]: audit 2024-04-02T05:08:19.818988+0000 mon.b (mon.1) 364 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:08:20.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:20 smithi069 bash[21528]: cluster 2024-04-02T05:08:20.014735+0000 osd.5 (osd.5) 433 : cluster 0 1.10 deep-scrub starts 2024-04-02T05:08:20.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:20 smithi069 bash[21528]: cluster 2024-04-02T05:08:20.016717+0000 osd.5 (osd.5) 434 : cluster 0 1.10 deep-scrub ok 2024-04-02T05:08:20.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:20 smithi098 bash[23790]: audit 2024-04-02T05:08:19.818988+0000 mon.b (mon.1) 364 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T05:08:20.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:20 smithi098 bash[23790]: cluster 2024-04-02T05:08:20.014735+0000 osd.5 (osd.5) 433 : cluster 0 1.10 deep-scrub starts 2024-04-02T05:08:20.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:20 smithi098 bash[23790]: cluster 2024-04-02T05:08:20.016717+0000 osd.5 (osd.5) 434 : cluster 0 1.10 deep-scrub ok 2024-04-02T05:08:21.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:21 smithi069 bash[21528]: cluster 2024-04-02T05:08:20.526948+0000 mgr.y (mgr.14538) 939 : cluster 0 pgmap v1575: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:21.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:21 smithi069 bash[28904]: cluster 2024-04-02T05:08:20.526948+0000 mgr.y (mgr.14538) 939 : cluster 0 pgmap v1575: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:21.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:21 smithi098 bash[23790]: cluster 2024-04-02T05:08:20.526948+0000 mgr.y (mgr.14538) 939 : cluster 0 pgmap v1575: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:22 smithi069 bash[21528]: cluster 2024-04-02T05:08:21.956309+0000 osd.5 (osd.5) 435 : cluster 0 2.12 scrub starts 2024-04-02T05:08:22.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:22 smithi069 bash[21528]: cluster 2024-04-02T05:08:21.958774+0000 osd.5 (osd.5) 436 : cluster 0 2.12 scrub ok 2024-04-02T05:08:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:22 smithi069 bash[28904]: cluster 2024-04-02T05:08:21.956309+0000 osd.5 (osd.5) 435 : cluster 0 2.12 scrub starts 2024-04-02T05:08:22.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:22 smithi069 bash[28904]: cluster 2024-04-02T05:08:21.958774+0000 osd.5 (osd.5) 436 : cluster 0 2.12 scrub ok 2024-04-02T05:08:22.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:22 smithi098 bash[23790]: cluster 2024-04-02T05:08:21.956309+0000 osd.5 (osd.5) 435 : cluster 0 2.12 scrub starts 2024-04-02T05:08:22.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:22 smithi098 bash[23790]: cluster 2024-04-02T05:08:21.958774+0000 osd.5 (osd.5) 436 : cluster 0 2.12 scrub ok 2024-04-02T05:08:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4019204253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074470 msgr2=0x7fc1d4074850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4019204253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074470 0x7fc1d4074850 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009990 tx=0x7fc1c4009ca0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4019204253 shutdown_connections 2024-04-02T05:08:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4019204253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d410ecd0 0x7fc1d41110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4019204253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074470 0x7fc1d4074850 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4019204253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4019204253 >> 172.21.15.69:0/4019204253 conn(0x7fc1d4079310 msgr2=0x7fc1d4079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:23.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4019204253 shutdown_connections 2024-04-02T05:08:23.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4019204253 wait complete. 2024-04-02T05:08:23.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 librados: starting msgr at 2024-04-02T05:08:23.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 librados: starting objecter 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 Processor -- start 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- start start 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 librados: setting wanted keys 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 librados: calling monclient init 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d4074470 0x7fc1d419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d410ecd0 0x7fc1d419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d4113cf0 con 0x7fc1d410ecd0 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d4113b70 con 0x7fc1d4074470 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7fc1d4113e70 con 0x7fc1d4074d90 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d410ecd0 0x7fc1d419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:23.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d419a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49506/0 (socket says 172.21.15.69:49506) 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 -- 172.21.15.69:0/4169931030 learned_addr learned my addr 172.21.15.69:0/4169931030 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d2ffd700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d4074470 0x7fc1d419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 -- 172.21.15.69:0/4169931030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d4074470 msgr2=0x7fc1d419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d4074470 0x7fc1d419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 -- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d410ecd0 msgr2=0x7fc1d419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d410ecd0 0x7fc1d419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 -- 172.21.15.69:0/4169931030 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d419f2e0 con 0x7fc1d4074d90 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d37fe700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d410ecd0 0x7fc1d419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d27fc700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d419a860 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00e9a0 tx=0x7fc1bc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1cbfff700 10 client.?.objecter ms_handle_connect 0x7fc1d4074d90 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1cbfff700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00cbd0 con 0x7fc1d4074d90 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d419f5d0 con 0x7fc1d4074d90 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 10 client.44141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:23.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d41a6e50 con 0x7fc1d4074d90 2024-04-02T05:08:23.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1d9510700 1 librados: init done 2024-04-02T05:08:23.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7fc1bc00cd70 con 0x7fc1d4074d90 2024-04-02T05:08:23.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.006+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc010e60 con 0x7fc1d4074d90 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b4005120 con 0x7fc1d4074d90 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7fc1bc018680 con 0x7fc1d4074d90 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1c006f5e0 0x7fc1c0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(968..968 src has 918..968) ==== 9015+0+0 (secure 0 0 0) 0x7fc1bc08e730 con 0x7fc1d4074d90 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 10 client.44141.objecter ms_dispatch 0x7fc1d4001550 osd_map(968..968 src has 918..968) 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 3 client.44141.objecter handle_osd_map got epochs [968,968] > 0 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 3 client.44141.objecter handle_osd_map decoding full epoch 968 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1d2ffd700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1c006f5e0 0x7fc1c0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 20 client.44141.objecter dump_active .. 0 homeless 2024-04-02T05:08:23.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1cbfff700 10 client.44141.objecter ms_handle_connect 0x7fc1c006f5e0 2024-04-02T05:08:23.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.010+0000 7fc1d2ffd700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1c006f5e0 0x7fc1c0071aa0 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4000c00 tx=0x7fc1c401a040 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:23.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.014+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc1bc05b780 con 0x7fc1d4074d90 2024-04-02T05:08:23.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.218+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 --> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc1b4000bc0 con 0x7fc1c006f5e0 2024-04-02T05:08:23.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.238+0000 7fc1cbfff700 1 -- 172.21.15.69:0/4169931030 <== mgr.14538 v2:172.21.15.69:6800/1004417273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+478560 (secure 0 0 0) 0x7fc1b4000bc0 con 0x7fc1c006f5e0 2024-04-02T05:08:23.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-02T05:08:23.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 10 librados: watch_flush enter 2024-04-02T05:08:23.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 10 librados: watch_flush exit 2024-04-02T05:08:23.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 20 client.44141.objecter shutdown clearing up homeless session... 2024-04-02T05:08:23.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 10 client.44141.objecter successfully canceled tick 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1c006f5e0 msgr2=0x7fc1c0071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1c006f5e0 0x7fc1c0071aa0 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4000c00 tx=0x7fc1c401a040 comp rx=0 tx=0).stop 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 msgr2=0x7fc1d419a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d419a860 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00e9a0 tx=0x7fc1bc00ecb0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 shutdown_connections 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7fc1c006f5e0 0x7fc1c0071aa0 unknown :-1 s=CLOSED pgs=4069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d410ecd0 0x7fc1d419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d4074470 0x7fc1d419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 --2- 172.21.15.69:0/4169931030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1d4074d90 0x7fc1d419a860 unknown :-1 s=CLOSED pgs=3352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 >> 172.21.15.69:0/4169931030 conn(0x7fc1d4079310 msgr2=0x7fc1d4106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 shutdown_connections 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 -- 172.21.15.69:0/4169931030 wait complete. 2024-04-02T05:08:23.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:23.242+0000 7fc1d9510700 1 librados: shutdown 2024-04-02T05:08:23.253 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-02T05:08:23.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[28904]: cluster 2024-04-02T05:08:21.780909+0000 osd.4 (osd.4) 580 : cluster 0 1.a scrub starts 2024-04-02T05:08:23.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[28904]: cluster 2024-04-02T05:08:21.783374+0000 osd.4 (osd.4) 581 : cluster 0 1.a scrub ok 2024-04-02T05:08:23.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[28904]: cluster 2024-04-02T05:08:22.159146+0000 osd.0 (osd.0) 573 : cluster 0 3.1b scrub starts 2024-04-02T05:08:23.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[28904]: cluster 2024-04-02T05:08:22.163012+0000 osd.0 (osd.0) 574 : cluster 0 3.1b scrub ok 2024-04-02T05:08:23.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[28904]: cluster 2024-04-02T05:08:22.527695+0000 mgr.y (mgr.14538) 940 : cluster 0 pgmap v1576: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:23.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[21528]: cluster 2024-04-02T05:08:21.780909+0000 osd.4 (osd.4) 580 : cluster 0 1.a scrub starts 2024-04-02T05:08:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[21528]: cluster 2024-04-02T05:08:21.783374+0000 osd.4 (osd.4) 581 : cluster 0 1.a scrub ok 2024-04-02T05:08:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[21528]: cluster 2024-04-02T05:08:22.159146+0000 osd.0 (osd.0) 573 : cluster 0 3.1b scrub starts 2024-04-02T05:08:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[21528]: cluster 2024-04-02T05:08:22.163012+0000 osd.0 (osd.0) 574 : cluster 0 3.1b scrub ok 2024-04-02T05:08:23.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:23 smithi069 bash[21528]: cluster 2024-04-02T05:08:22.527695+0000 mgr.y (mgr.14538) 940 : cluster 0 pgmap v1576: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:23.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:23 smithi098 bash[23790]: cluster 2024-04-02T05:08:21.780909+0000 osd.4 (osd.4) 580 : cluster 0 1.a scrub starts 2024-04-02T05:08:23.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:23 smithi098 bash[23790]: cluster 2024-04-02T05:08:21.783374+0000 osd.4 (osd.4) 581 : cluster 0 1.a scrub ok 2024-04-02T05:08:23.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:23 smithi098 bash[23790]: cluster 2024-04-02T05:08:22.159146+0000 osd.0 (osd.0) 573 : cluster 0 3.1b scrub starts 2024-04-02T05:08:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:23 smithi098 bash[23790]: cluster 2024-04-02T05:08:22.163012+0000 osd.0 (osd.0) 574 : cluster 0 3.1b scrub ok 2024-04-02T05:08:23.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:23 smithi098 bash[23790]: cluster 2024-04-02T05:08:22.527695+0000 mgr.y (mgr.14538) 940 : cluster 0 pgmap v1576: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:24.016 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2024-04-02T05:08:22.527208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":2411,"num_read_kb":2131,"num_write":1207,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":1062645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4142,"ondisk_log_size":4142,"up":552,"acting":552,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":552,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":764456,"kb_used_data":21096,"kb_used_omap":68,"kb_used_meta":743227,"kb_avail":749196760,"statfs":{"total":767960285184,"available":767177482240,"internally_reserved":0,"allocated":21602304,"data_stored":7550617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69644,"internal_metadata":761065460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006429"},"pg_stats":[{"pgid":"1.d","version":"101'128","reported_seq":1336,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006305+0000","last_change":"2024-04-02T05:07:14.552491+0000","last_active":"2024-04-02T05:08:05.006305+0000","last_peered":"2024-04-02T05:08:05.006305+0000","last_clean":"2024-04-02T05:08:05.006305+0000","last_became_active":"2024-04-02T05:05:57.346674+0000","last_became_peered":"2024-04-02T05:05:57.346674+0000","last_unstale":"2024-04-02T05:08:05.006305+0000","last_undegraded":"2024-04-02T05:08:05.006305+0000","last_fullsized":"2024-04-02T05:08:05.006305+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:14.552260+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:14.552260+0000","last_clean_scrub_stamp":"2024-04-02T05:07:14.552260+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:42.782879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.161784+0000","last_change":"2024-04-02T05:08:07.161848+0000","last_active":"2024-04-02T05:08:07.161784+0000","last_peered":"2024-04-02T05:08:07.161784+0000","last_clean":"2024-04-02T05:08:07.161784+0000","last_became_active":"2024-04-02T05:04:11.245164+0000","last_became_peered":"2024-04-02T05:04:11.245164+0000","last_unstale":"2024-04-02T05:08:07.161784+0000","last_undegraded":"2024-04-02T05:08:07.161784+0000","last_fullsized":"2024-04-02T05:08:07.161784+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:07.161628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:07.161628+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.161628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.731217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003006+0000","last_change":"2024-04-02T05:07:26.213276+0000","last_active":"2024-04-02T05:08:05.003006+0000","last_peered":"2024-04-02T05:08:05.003006+0000","last_clean":"2024-04-02T05:08:05.003006+0000","last_became_active":"2024-04-02T05:04:11.244828+0000","last_became_peered":"2024-04-02T05:04:11.244828+0000","last_unstale":"2024-04-02T05:08:05.003006+0000","last_undegraded":"2024-04-02T05:08:05.003006+0000","last_fullsized":"2024-04-02T05:08:05.003006+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:26.213058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:48.966242+0000","last_clean_scrub_stamp":"2024-04-02T05:07:26.213058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.485877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"950'46","reported_seq":1424,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007467+0000","last_change":"2024-04-02T05:07:45.443583+0000","last_active":"2024-04-02T05:08:05.007467+0000","last_peered":"2024-04-02T05:08:05.007467+0000","last_clean":"2024-04-02T05:08:05.007467+0000","last_became_active":"2024-04-02T04:59:55.068785+0000","last_became_peered":"2024-04-02T04:59:55.068785+0000","last_unstale":"2024-04-02T05:08:05.007467+0000","last_undegraded":"2024-04-02T05:08:05.007467+0000","last_fullsized":"2024-04-02T05:08:05.007467+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'46","last_scrub_stamp":"2024-04-02T05:07:45.443316+0000","last_deep_scrub":"950'46","last_deep_scrub_stamp":"2024-04-02T05:06:20.505182+0000","last_clean_scrub_stamp":"2024-04-02T05:07:45.443316+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.045719+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871333+0000","last_change":"2024-04-02T05:07:52.822951+0000","last_active":"2024-04-02T05:08:04.871333+0000","last_peered":"2024-04-02T05:08:04.871333+0000","last_clean":"2024-04-02T05:08:04.871333+0000","last_became_active":"2024-04-02T05:05:16.544433+0000","last_became_peered":"2024-04-02T05:05:16.544433+0000","last_unstale":"2024-04-02T05:08:04.871333+0000","last_undegraded":"2024-04-02T05:08:04.871333+0000","last_fullsized":"2024-04-02T05:08:04.871333+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:52.822725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:52.822725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.887431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004187+0000","last_change":"2024-04-02T05:07:45.119037+0000","last_active":"2024-04-02T05:08:05.004187+0000","last_peered":"2024-04-02T05:08:05.004187+0000","last_clean":"2024-04-02T05:08:05.004187+0000","last_became_active":"2024-04-02T05:04:11.260461+0000","last_became_peered":"2024-04-02T05:04:11.260461+0000","last_unstale":"2024-04-02T05:08:05.004187+0000","last_undegraded":"2024-04-02T05:08:05.004187+0000","last_fullsized":"2024-04-02T05:08:05.004187+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:45.118824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:58:59.898499+0000","last_clean_scrub_stamp":"2024-04-02T05:07:45.118824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.778332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"101'128","reported_seq":880,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397774+0000","last_change":"2024-04-02T05:07:33.577003+0000","last_active":"2024-04-02T05:08:04.397774+0000","last_peered":"2024-04-02T05:08:04.397774+0000","last_clean":"2024-04-02T05:08:04.397774+0000","last_became_active":"2024-04-02T05:05:57.327887+0000","last_became_peered":"2024-04-02T05:05:57.327887+0000","last_unstale":"2024-04-02T05:08:04.397774+0000","last_undegraded":"2024-04-02T05:08:04.397774+0000","last_fullsized":"2024-04-02T05:08:04.397774+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:33.576764+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:33.576764+0000","last_clean_scrub_stamp":"2024-04-02T05:07:33.576764+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:50.402444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1753,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.873697+0000","last_change":"2024-04-02T05:07:16.761520+0000","last_active":"2024-04-02T05:08:04.873697+0000","last_peered":"2024-04-02T05:08:04.873697+0000","last_clean":"2024-04-02T05:08:04.873697+0000","last_became_active":"2024-04-02T04:59:00.080466+0000","last_became_peered":"2024-04-02T04:59:00.080466+0000","last_unstale":"2024-04-02T05:08:04.873697+0000","last_undegraded":"2024-04-02T05:08:04.873697+0000","last_fullsized":"2024-04-02T05:08:04.873697+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.761302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:44.775660+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.761302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:44.143566+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1868,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073581+0000","last_change":"2024-04-02T05:07:19.888171+0000","last_active":"2024-04-02T05:08:05.073581+0000","last_peered":"2024-04-02T05:08:05.073581+0000","last_clean":"2024-04-02T05:08:05.073581+0000","last_became_active":"2024-04-02T04:43:58.311512+0000","last_became_peered":"2024-04-02T04:43:58.311512+0000","last_unstale":"2024-04-02T05:08:05.073581+0000","last_undegraded":"2024-04-02T05:08:05.073581+0000","last_fullsized":"2024-04-02T05:08:05.073581+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:19.887957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:36.528990+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.887957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.325679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"899'35","reported_seq":180,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003805+0000","last_change":"2024-04-02T05:07:19.268579+0000","last_active":"2024-04-02T05:08:05.003805+0000","last_peered":"2024-04-02T05:08:05.003805+0000","last_clean":"2024-04-02T05:08:05.003805+0000","last_became_active":"2024-04-02T05:04:11.261777+0000","last_became_peered":"2024-04-02T05:04:11.261777+0000","last_unstale":"2024-04-02T05:08:05.003805+0000","last_undegraded":"2024-04-02T05:08:05.003805+0000","last_fullsized":"2024-04-02T05:08:05.003805+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'35","last_scrub_stamp":"2024-04-02T05:07:19.268333+0000","last_deep_scrub":"899'35","last_deep_scrub_stamp":"2024-04-02T05:05:43.878796+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.268333+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.552262+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1921,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401333+0000","last_change":"2024-04-02T05:07:55.728643+0000","last_active":"2024-04-02T05:08:04.401333+0000","last_peered":"2024-04-02T05:08:04.401333+0000","last_clean":"2024-04-02T05:08:04.401333+0000","last_became_active":"2024-04-02T05:05:54.274933+0000","last_became_peered":"2024-04-02T05:05:54.274933+0000","last_unstale":"2024-04-02T05:08:04.401333+0000","last_undegraded":"2024-04-02T05:08:04.401333+0000","last_fullsized":"2024-04-02T05:08:04.401333+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:55.728437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:53:10.415766+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.728437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.735484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1709,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.072786+0000","last_change":"2024-04-02T05:07:55.085752+0000","last_active":"2024-04-02T05:08:05.072786+0000","last_peered":"2024-04-02T05:08:05.072786+0000","last_clean":"2024-04-02T05:08:05.072786+0000","last_became_active":"2024-04-02T05:05:16.537209+0000","last_became_peered":"2024-04-02T05:05:16.537209+0000","last_unstale":"2024-04-02T05:08:05.072786+0000","last_undegraded":"2024-04-02T05:08:05.072786+0000","last_fullsized":"2024-04-02T05:08:05.072786+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:55.085446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.085446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:59.568285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"101'128","reported_seq":1420,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667824+0000","last_change":"2024-04-02T05:07:36.752971+0000","last_active":"2024-04-02T05:08:04.667824+0000","last_peered":"2024-04-02T05:08:04.667824+0000","last_clean":"2024-04-02T05:08:04.667824+0000","last_became_active":"2024-04-02T05:05:57.340555+0000","last_became_peered":"2024-04-02T05:05:57.340555+0000","last_unstale":"2024-04-02T05:08:04.667824+0000","last_undegraded":"2024-04-02T05:08:04.667824+0000","last_fullsized":"2024-04-02T05:08:04.667824+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:36.752818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:07:36.752818+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:59.269988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1738,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.072577+0000","last_change":"2024-04-02T05:06:51.023980+0000","last_active":"2024-04-02T05:08:05.072577+0000","last_peered":"2024-04-02T05:08:05.072577+0000","last_clean":"2024-04-02T05:08:05.072577+0000","last_became_active":"2024-04-02T04:59:00.102255+0000","last_became_peered":"2024-04-02T04:59:00.102255+0000","last_unstale":"2024-04-02T05:08:05.072577+0000","last_undegraded":"2024-04-02T05:08:05.072577+0000","last_fullsized":"2024-04-02T05:08:05.072577+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:51.023733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:51.023733+0000","last_clean_scrub_stamp":"2024-04-02T05:06:51.023733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:16.232899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1866,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667103+0000","last_change":"2024-04-02T05:07:51.878887+0000","last_active":"2024-04-02T05:08:04.667103+0000","last_peered":"2024-04-02T05:08:04.667103+0000","last_clean":"2024-04-02T05:08:04.667103+0000","last_became_active":"2024-04-02T04:53:54.904402+0000","last_became_peered":"2024-04-02T04:53:54.904402+0000","last_unstale":"2024-04-02T05:08:04.667103+0000","last_undegraded":"2024-04-02T05:08:04.667103+0000","last_fullsized":"2024-04-02T05:08:04.667103+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:51.878683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:53.441303+0000","last_clean_scrub_stamp":"2024-04-02T05:07:51.878683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:10.663762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"899'45","reported_seq":1506,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007145+0000","last_change":"2024-04-02T05:07:42.454887+0000","last_active":"2024-04-02T05:08:05.007145+0000","last_peered":"2024-04-02T05:08:05.007145+0000","last_clean":"2024-04-02T05:08:05.007145+0000","last_became_active":"2024-04-02T05:04:11.240495+0000","last_became_peered":"2024-04-02T05:04:11.240495+0000","last_unstale":"2024-04-02T05:08:05.007145+0000","last_undegraded":"2024-04-02T05:08:05.007145+0000","last_fullsized":"2024-04-02T05:08:05.007145+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'45","last_scrub_stamp":"2024-04-02T05:07:42.454640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.454640+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:50.274109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.872686+0000","last_change":"2024-04-02T05:07:19.816884+0000","last_active":"2024-04-02T05:08:04.872686+0000","last_peered":"2024-04-02T05:08:04.872686+0000","last_clean":"2024-04-02T05:08:04.872686+0000","last_became_active":"2024-04-02T05:05:16.549444+0000","last_became_peered":"2024-04-02T05:05:16.549444+0000","last_unstale":"2024-04-02T05:08:04.872686+0000","last_undegraded":"2024-04-02T05:08:04.872686+0000","last_fullsized":"2024-04-02T05:08:04.872686+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:19.816675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:23.149595+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.816675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.424433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1364,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.001601+0000","last_change":"2024-04-02T05:07:10.540148+0000","last_active":"2024-04-02T05:08:05.001601+0000","last_peered":"2024-04-02T05:08:05.001601+0000","last_clean":"2024-04-02T05:08:05.001601+0000","last_became_active":"2024-04-02T05:05:16.545371+0000","last_became_peered":"2024-04-02T05:05:16.545371+0000","last_unstale":"2024-04-02T05:08:05.001601+0000","last_undegraded":"2024-04-02T05:08:05.001601+0000","last_fullsized":"2024-04-02T05:08:05.001601+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:10.539875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:10.539875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:25.735993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"956'129","reported_seq":1886,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003270+0000","last_change":"2024-04-02T05:07:55.210371+0000","last_active":"2024-04-02T05:08:05.003270+0000","last_peered":"2024-04-02T05:08:05.003270+0000","last_clean":"2024-04-02T05:08:05.003270+0000","last_became_active":"2024-04-02T05:04:14.200867+0000","last_became_peered":"2024-04-02T05:04:14.200867+0000","last_unstale":"2024-04-02T05:08:05.003270+0000","last_undegraded":"2024-04-02T05:08:05.003270+0000","last_fullsized":"2024-04-02T05:08:05.003270+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":4,"last_scrub":"956'129","last_scrub_stamp":"2024-04-02T05:07:55.210035+0000","last_deep_scrub":"956'129","last_deep_scrub_stamp":"2024-04-02T05:07:55.210035+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.210035+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.341579+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":126,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870748+0000","last_change":"2024-04-02T05:07:54.859012+0000","last_active":"2024-04-02T05:08:04.870748+0000","last_peered":"2024-04-02T05:08:04.870748+0000","last_clean":"2024-04-02T05:08:04.870748+0000","last_became_active":"2024-04-02T05:05:16.556262+0000","last_became_peered":"2024-04-02T05:05:16.556262+0000","last_unstale":"2024-04-02T05:08:04.870748+0000","last_undegraded":"2024-04-02T05:08:04.870748+0000","last_fullsized":"2024-04-02T05:08:04.870748+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.858904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:37.743671+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.858904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:21.788379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"899'68","reported_seq":2027,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667791+0000","last_change":"2024-04-02T05:08:03.877564+0000","last_active":"2024-04-02T05:08:04.667791+0000","last_peered":"2024-04-02T05:08:04.667791+0000","last_clean":"2024-04-02T05:08:04.667791+0000","last_became_active":"2024-04-02T04:58:39.391176+0000","last_became_peered":"2024-04-02T04:58:39.391176+0000","last_unstale":"2024-04-02T05:08:04.667791+0000","last_undegraded":"2024-04-02T05:08:04.667791+0000","last_fullsized":"2024-04-02T05:08:04.667791+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'68","last_scrub_stamp":"2024-04-02T05:08:03.877297+0000","last_deep_scrub":"531'52","last_deep_scrub_stamp":"2024-04-02T04:56:30.024650+0000","last_clean_scrub_stamp":"2024-04-02T05:08:03.877297+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:31.505482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1382,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005914+0000","last_change":"2024-04-02T05:07:40.513562+0000","last_active":"2024-04-02T05:08:05.005914+0000","last_peered":"2024-04-02T05:08:05.005914+0000","last_clean":"2024-04-02T05:08:05.005914+0000","last_became_active":"2024-04-02T05:05:54.268662+0000","last_became_peered":"2024-04-02T05:05:54.268662+0000","last_unstale":"2024-04-02T05:08:05.005914+0000","last_undegraded":"2024-04-02T05:08:05.005914+0000","last_fullsized":"2024-04-02T05:08:05.005914+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:40.513353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:54.596224+0000","last_clean_scrub_stamp":"2024-04-02T05:07:40.513353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.669891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"942'17","reported_seq":430,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:13.065666+0000","last_change":"2024-04-02T05:08:09.733582+0000","last_active":"2024-04-02T05:08:13.065666+0000","last_peered":"2024-04-02T05:08:13.065666+0000","last_clean":"2024-04-02T05:08:13.065666+0000","last_became_active":"2024-04-02T05:05:54.274960+0000","last_became_peered":"2024-04-02T05:05:54.274960+0000","last_unstale":"2024-04-02T05:08:13.065666+0000","last_undegraded":"2024-04-02T05:08:13.065666+0000","last_fullsized":"2024-04-02T05:08:13.065666+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'17","last_scrub_stamp":"2024-04-02T05:08:09.733353+0000","last_deep_scrub":"662'13","last_deep_scrub_stamp":"2024-04-02T05:02:22.718259+0000","last_clean_scrub_stamp":"2024-04-02T05:08:09.733353+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:25.419121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":126,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871458+0000","last_change":"2024-04-02T05:07:53.863861+0000","last_active":"2024-04-02T05:08:04.871458+0000","last_peered":"2024-04-02T05:08:04.871458+0000","last_clean":"2024-04-02T05:08:04.871458+0000","last_became_active":"2024-04-02T05:05:16.545041+0000","last_became_peered":"2024-04-02T05:05:16.545041+0000","last_unstale":"2024-04-02T05:08:04.871458+0000","last_undegraded":"2024-04-02T05:08:04.871458+0000","last_fullsized":"2024-04-02T05:08:04.871458+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:53.863711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.863711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:05.588023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'128","reported_seq":1675,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:13.693515+0000","last_change":"2024-04-02T05:08:13.693579+0000","last_active":"2024-04-02T05:08:13.693515+0000","last_peered":"2024-04-02T05:08:13.693515+0000","last_clean":"2024-04-02T05:08:13.693515+0000","last_became_active":"2024-04-02T05:05:19.572970+0000","last_became_peered":"2024-04-02T05:05:19.572970+0000","last_unstale":"2024-04-02T05:08:13.693515+0000","last_undegraded":"2024-04-02T05:08:13.693515+0000","last_fullsized":"2024-04-02T05:08:13.693515+0000","mapping_epoch":925,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":926,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:13.693354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:08:13.693354+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:29.063460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1727,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401859+0000","last_change":"2024-04-02T05:07:31.630281+0000","last_active":"2024-04-02T05:08:04.401859+0000","last_peered":"2024-04-02T05:08:04.401859+0000","last_clean":"2024-04-02T05:08:04.401859+0000","last_became_active":"2024-04-02T04:49:26.312184+0000","last_became_peered":"2024-04-02T04:49:26.312184+0000","last_unstale":"2024-04-02T05:08:04.401859+0000","last_undegraded":"2024-04-02T05:08:04.401859+0000","last_fullsized":"2024-04-02T05:08:04.401859+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:31.630051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:31.630051+0000","last_clean_scrub_stamp":"2024-04-02T05:07:31.630051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.538788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1881,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397231+0000","last_change":"2024-04-02T05:07:27.532993+0000","last_active":"2024-04-02T05:08:04.397231+0000","last_peered":"2024-04-02T05:08:04.397231+0000","last_clean":"2024-04-02T05:08:04.397231+0000","last_became_active":"2024-04-02T04:59:00.138222+0000","last_became_peered":"2024-04-02T04:59:00.138222+0000","last_unstale":"2024-04-02T05:08:04.397231+0000","last_undegraded":"2024-04-02T05:08:04.397231+0000","last_fullsized":"2024-04-02T05:08:04.397231+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:27.532784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:32.675301+0000","last_clean_scrub_stamp":"2024-04-02T05:07:27.532784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:34.311284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"899'32","reported_seq":720,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395994+0000","last_change":"2024-04-02T05:07:08.510901+0000","last_active":"2024-04-02T05:08:04.395994+0000","last_peered":"2024-04-02T05:08:04.395994+0000","last_clean":"2024-04-02T05:08:04.395994+0000","last_became_active":"2024-04-02T04:59:00.135793+0000","last_became_peered":"2024-04-02T04:59:00.135793+0000","last_unstale":"2024-04-02T05:08:04.395994+0000","last_undegraded":"2024-04-02T05:08:04.395994+0000","last_fullsized":"2024-04-02T05:08:04.395994+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'32","last_scrub_stamp":"2024-04-02T05:07:08.510631+0000","last_deep_scrub":"112'20","last_deep_scrub_stamp":"2024-04-02T04:46:18.673548+0000","last_clean_scrub_stamp":"2024-04-02T05:07:08.510631+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:36.256714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1899,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400330+0000","last_change":"2024-04-02T05:08:03.702952+0000","last_active":"2024-04-02T05:08:04.400330+0000","last_peered":"2024-04-02T05:08:04.400330+0000","last_clean":"2024-04-02T05:08:04.400330+0000","last_became_active":"2024-04-02T04:59:00.106616+0000","last_became_peered":"2024-04-02T04:59:00.106616+0000","last_unstale":"2024-04-02T05:08:04.400330+0000","last_undegraded":"2024-04-02T05:08:04.400330+0000","last_fullsized":"2024-04-02T05:08:04.400330+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:03.702739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:11.071225+0000","last_clean_scrub_stamp":"2024-04-02T05:08:03.702739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:14.771693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":82,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398592+0000","last_change":"2024-04-02T05:07:48.702097+0000","last_active":"2024-04-02T05:08:04.398592+0000","last_peered":"2024-04-02T05:08:04.398592+0000","last_clean":"2024-04-02T05:08:04.398592+0000","last_became_active":"2024-04-02T05:05:54.273784+0000","last_became_peered":"2024-04-02T05:05:54.273784+0000","last_unstale":"2024-04-02T05:08:04.398592+0000","last_undegraded":"2024-04-02T05:08:04.398592+0000","last_fullsized":"2024-04-02T05:08:04.398592+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:48.701885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.701885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:08.687729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"101'128","reported_seq":1812,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396388+0000","last_change":"2024-04-02T05:06:41.489660+0000","last_active":"2024-04-02T05:08:04.396388+0000","last_peered":"2024-04-02T05:08:04.396388+0000","last_clean":"2024-04-02T05:08:04.396388+0000","last_became_active":"2024-04-02T05:05:00.361876+0000","last_became_peered":"2024-04-02T05:05:00.361876+0000","last_unstale":"2024-04-02T05:08:04.396388+0000","last_undegraded":"2024-04-02T05:08:04.396388+0000","last_fullsized":"2024-04-02T05:08:04.396388+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:06:41.489427+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:06:41.489427+0000","last_clean_scrub_stamp":"2024-04-02T05:06:41.489427+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:08.984869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"138'7","reported_seq":86,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395793+0000","last_change":"2024-04-02T05:07:41.644212+0000","last_active":"2024-04-02T05:08:04.395793+0000","last_peered":"2024-04-02T05:08:04.395793+0000","last_clean":"2024-04-02T05:08:04.395793+0000","last_became_active":"2024-04-02T05:05:54.284476+0000","last_became_peered":"2024-04-02T05:05:54.284476+0000","last_unstale":"2024-04-02T05:08:04.395793+0000","last_undegraded":"2024-04-02T05:08:04.395793+0000","last_fullsized":"2024-04-02T05:08:04.395793+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'7","last_scrub_stamp":"2024-04-02T05:07:41.643877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:41.643877+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:59.541105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"899'60","reported_seq":737,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.394741+0000","last_change":"2024-04-02T05:06:48.404172+0000","last_active":"2024-04-02T05:08:04.394741+0000","last_peered":"2024-04-02T05:08:04.394741+0000","last_clean":"2024-04-02T05:08:04.394741+0000","last_became_active":"2024-04-02T04:59:00.122446+0000","last_became_peered":"2024-04-02T04:59:00.122446+0000","last_unstale":"2024-04-02T05:08:04.394741+0000","last_undegraded":"2024-04-02T05:08:04.394741+0000","last_fullsized":"2024-04-02T05:08:04.394741+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'60","last_scrub_stamp":"2024-04-02T05:06:48.404026+0000","last_deep_scrub":"899'60","last_deep_scrub_stamp":"2024-04-02T05:05:38.138230+0000","last_clean_scrub_stamp":"2024-04-02T05:06:48.404026+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:15.847159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1896,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395365+0000","last_change":"2024-04-02T05:07:14.781571+0000","last_active":"2024-04-02T05:08:04.395365+0000","last_peered":"2024-04-02T05:08:04.395365+0000","last_clean":"2024-04-02T05:08:04.395365+0000","last_became_active":"2024-04-02T04:59:00.107474+0000","last_became_peered":"2024-04-02T04:59:00.107474+0000","last_unstale":"2024-04-02T05:08:04.395365+0000","last_undegraded":"2024-04-02T05:08:04.395365+0000","last_fullsized":"2024-04-02T05:08:04.395365+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:14.781365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:01.943795+0000","last_clean_scrub_stamp":"2024-04-02T05:07:14.781365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:27.386708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1835,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.874244+0000","last_change":"2024-04-02T05:07:13.727004+0000","last_active":"2024-04-02T05:08:04.874244+0000","last_peered":"2024-04-02T05:08:04.874244+0000","last_clean":"2024-04-02T05:08:04.874244+0000","last_became_active":"2024-04-02T04:59:00.111699+0000","last_became_peered":"2024-04-02T04:59:00.111699+0000","last_unstale":"2024-04-02T05:08:04.874244+0000","last_undegraded":"2024-04-02T05:08:04.874244+0000","last_fullsized":"2024-04-02T05:08:04.874244+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:13.726798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:13.726798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:43.310957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:06.122666+0000","last_change":"2024-04-02T05:08:06.122776+0000","last_active":"2024-04-02T05:08:06.122666+0000","last_peered":"2024-04-02T05:08:06.122666+0000","last_clean":"2024-04-02T05:08:06.122666+0000","last_became_active":"2024-04-02T05:04:11.236780+0000","last_became_peered":"2024-04-02T05:04:11.236780+0000","last_unstale":"2024-04-02T05:08:06.122666+0000","last_undegraded":"2024-04-02T05:08:06.122666+0000","last_fullsized":"2024-04-02T05:08:06.122666+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:06.122472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:06.122472+0000","last_clean_scrub_stamp":"2024-04-02T05:08:06.122472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:34.519358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"101'128","reported_seq":1884,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:11.641058+0000","last_change":"2024-04-02T05:08:11.641123+0000","last_active":"2024-04-02T05:08:11.641058+0000","last_peered":"2024-04-02T05:08:11.641058+0000","last_clean":"2024-04-02T05:08:11.641058+0000","last_became_active":"2024-04-02T05:04:14.197651+0000","last_became_peered":"2024-04-02T05:04:14.197651+0000","last_unstale":"2024-04-02T05:08:11.641058+0000","last_undegraded":"2024-04-02T05:08:11.641058+0000","last_fullsized":"2024-04-02T05:08:11.641058+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:11.640889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:08:11.640889+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:13.595909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1733,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:16.878773+0000","last_change":"2024-04-02T05:08:16.878831+0000","last_active":"2024-04-02T05:08:16.878773+0000","last_peered":"2024-04-02T05:08:16.878773+0000","last_clean":"2024-04-02T05:08:16.878773+0000","last_became_active":"2024-04-02T05:05:54.276276+0000","last_became_peered":"2024-04-02T05:05:54.276276+0000","last_unstale":"2024-04-02T05:08:16.878773+0000","last_undegraded":"2024-04-02T05:08:16.878773+0000","last_fullsized":"2024-04-02T05:08:16.878773+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:16.878581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:30.797036+0000","last_clean_scrub_stamp":"2024-04-02T05:08:16.878581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:29.384731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"966'43","reported_seq":105,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395998+0000","last_change":"2024-04-02T05:07:51.708709+0000","last_active":"2024-04-02T05:08:04.395998+0000","last_peered":"2024-04-02T05:08:04.395998+0000","last_clean":"2024-04-02T05:08:04.395998+0000","last_became_active":"2024-04-02T05:05:54.280900+0000","last_became_peered":"2024-04-02T05:05:54.280900+0000","last_unstale":"2024-04-02T05:08:04.395998+0000","last_undegraded":"2024-04-02T05:08:04.395998+0000","last_fullsized":"2024-04-02T05:08:04.395998+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'43","last_scrub_stamp":"2024-04-02T05:07:51.708449+0000","last_deep_scrub":"966'43","last_deep_scrub_stamp":"2024-04-02T05:07:51.708449+0000","last_clean_scrub_stamp":"2024-04-02T05:07:51.708449+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:52.505344+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":48,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1362,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:11.606740+0000","last_change":"2024-04-02T05:08:11.606799+0000","last_active":"2024-04-02T05:08:11.606740+0000","last_peered":"2024-04-02T05:08:11.606740+0000","last_clean":"2024-04-02T05:08:11.606740+0000","last_became_active":"2024-04-02T04:53:54.944203+0000","last_became_peered":"2024-04-02T04:53:54.944203+0000","last_unstale":"2024-04-02T05:08:11.606740+0000","last_undegraded":"2024-04-02T05:08:11.606740+0000","last_fullsized":"2024-04-02T05:08:11.606740+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:11.606565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:18.248880+0000","last_clean_scrub_stamp":"2024-04-02T05:08:11.606565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:19.157202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"916'15","reported_seq":2212,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.434057+0000","last_change":"2024-04-02T05:08:04.672304+0000","last_active":"2024-04-02T05:08:10.434057+0000","last_peered":"2024-04-02T05:08:10.434057+0000","last_clean":"2024-04-02T05:08:10.434057+0000","last_became_active":"2024-04-02T05:04:58.338999+0000","last_became_peered":"2024-04-02T05:04:58.338999+0000","last_unstale":"2024-04-02T05:08:10.434057+0000","last_undegraded":"2024-04-02T05:08:10.434057+0000","last_fullsized":"2024-04-02T05:08:10.434057+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"916'15","last_scrub_stamp":"2024-04-02T05:08:04.672082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:08:04.672082+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:16.630151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397474+0000","last_change":"2024-04-02T05:07:54.427694+0000","last_active":"2024-04-02T05:08:04.397474+0000","last_peered":"2024-04-02T05:08:04.397474+0000","last_clean":"2024-04-02T05:08:04.397474+0000","last_became_active":"2024-04-02T05:06:31.713530+0000","last_became_peered":"2024-04-02T05:06:31.713530+0000","last_unstale":"2024-04-02T05:08:04.397474+0000","last_undegraded":"2024-04-02T05:08:04.397474+0000","last_fullsized":"2024-04-02T05:08:04.397474+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.427485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:30.458780+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.427485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:24.419377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'128","reported_seq":1342,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.408805+0000","last_change":"2024-04-02T05:08:07.408869+0000","last_active":"2024-04-02T05:08:07.408805+0000","last_peered":"2024-04-02T05:08:07.408805+0000","last_clean":"2024-04-02T05:08:07.408805+0000","last_became_active":"2024-04-02T05:05:00.369659+0000","last_became_peered":"2024-04-02T05:05:00.369659+0000","last_unstale":"2024-04-02T05:08:07.408805+0000","last_undegraded":"2024-04-02T05:08:07.408805+0000","last_fullsized":"2024-04-02T05:08:07.408805+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:07.408631+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:03:46.349028+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.408631+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:18.949607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1732,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.885405+0000","last_change":"2024-04-02T05:08:04.885468+0000","last_active":"2024-04-02T05:08:04.885405+0000","last_peered":"2024-04-02T05:08:04.885405+0000","last_clean":"2024-04-02T05:08:04.885405+0000","last_became_active":"2024-04-02T04:59:00.105786+0000","last_became_peered":"2024-04-02T04:59:00.105786+0000","last_unstale":"2024-04-02T05:08:04.885405+0000","last_undegraded":"2024-04-02T05:08:04.885405+0000","last_fullsized":"2024-04-02T05:08:04.885405+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:04.885258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:54:29.941615+0000","last_clean_scrub_stamp":"2024-04-02T05:08:04.885258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:15.597944+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396371+0000","last_change":"2024-04-02T05:07:49.736414+0000","last_active":"2024-04-02T05:08:04.396371+0000","last_peered":"2024-04-02T05:08:04.396371+0000","last_clean":"2024-04-02T05:08:04.396371+0000","last_became_active":"2024-04-02T05:05:54.286532+0000","last_became_peered":"2024-04-02T05:05:54.286532+0000","last_unstale":"2024-04-02T05:08:04.396371+0000","last_undegraded":"2024-04-02T05:08:04.396371+0000","last_fullsized":"2024-04-02T05:08:04.396371+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:49.736191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:49.736191+0000","last_clean_scrub_stamp":"2024-04-02T05:07:49.736191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:52.156024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"899'42","reported_seq":1933,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:13.039150+0000","last_change":"2024-04-02T05:08:13.039244+0000","last_active":"2024-04-02T05:08:13.039150+0000","last_peered":"2024-04-02T05:08:13.039150+0000","last_clean":"2024-04-02T05:08:13.039150+0000","last_became_active":"2024-04-02T04:43:56.247267+0000","last_became_peered":"2024-04-02T04:43:56.247267+0000","last_unstale":"2024-04-02T05:08:13.039150+0000","last_undegraded":"2024-04-02T05:08:13.039150+0000","last_fullsized":"2024-04-02T05:08:13.039150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'42","last_scrub_stamp":"2024-04-02T05:08:13.038963+0000","last_deep_scrub":"188'18","last_deep_scrub_stamp":"2024-04-02T04:50:28.030658+0000","last_clean_scrub_stamp":"2024-04-02T05:08:13.038963+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:42.760576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1874,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668522+0000","last_change":"2024-04-02T05:07:29.813460+0000","last_active":"2024-04-02T05:08:04.668522+0000","last_peered":"2024-04-02T05:08:04.668522+0000","last_clean":"2024-04-02T05:08:04.668522+0000","last_became_active":"2024-04-02T04:53:54.913340+0000","last_became_peered":"2024-04-02T04:53:54.913340+0000","last_unstale":"2024-04-02T05:08:04.668522+0000","last_undegraded":"2024-04-02T05:08:04.668522+0000","last_fullsized":"2024-04-02T05:08:04.668522+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:29.813178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:29.813178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.316016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"831'6","reported_seq":1269,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:16.342344+0000","last_change":"2024-04-02T05:08:00.472828+0000","last_active":"2024-04-02T05:08:16.342344+0000","last_peered":"2024-04-02T05:08:16.342344+0000","last_clean":"2024-04-02T05:08:16.342344+0000","last_became_active":"2024-04-02T04:59:00.119844+0000","last_became_peered":"2024-04-02T04:59:00.119844+0000","last_unstale":"2024-04-02T05:08:16.342344+0000","last_undegraded":"2024-04-02T05:08:16.342344+0000","last_fullsized":"2024-04-02T05:08:16.342344+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'6","last_scrub_stamp":"2024-04-02T05:08:00.472747+0000","last_deep_scrub":"831'6","last_deep_scrub_stamp":"2024-04-02T05:06:35.417454+0000","last_clean_scrub_stamp":"2024-04-02T05:08:00.472747+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:02.348273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010829380000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":563,"num_read_kb":558,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1319,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.008004+0000","last_change":"2024-04-02T05:07:32.551736+0000","last_active":"2024-04-02T05:08:05.008004+0000","last_peered":"2024-04-02T05:08:05.008004+0000","last_clean":"2024-04-02T05:08:05.008004+0000","last_became_active":"2024-04-02T05:05:16.545510+0000","last_became_peered":"2024-04-02T05:05:16.545510+0000","last_unstale":"2024-04-02T05:08:05.008004+0000","last_undegraded":"2024-04-02T05:08:05.008004+0000","last_fullsized":"2024-04-02T05:08:05.008004+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:32.551400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:22.438115+0000","last_clean_scrub_stamp":"2024-04-02T05:07:32.551400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:33.564796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'128","reported_seq":923,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397303+0000","last_change":"2024-04-02T05:06:45.568625+0000","last_active":"2024-04-02T05:08:04.397303+0000","last_peered":"2024-04-02T05:08:04.397303+0000","last_clean":"2024-04-02T05:08:04.397303+0000","last_became_active":"2024-04-02T05:05:57.336377+0000","last_became_peered":"2024-04-02T05:05:57.336377+0000","last_unstale":"2024-04-02T05:08:04.397303+0000","last_undegraded":"2024-04-02T05:08:04.397303+0000","last_fullsized":"2024-04-02T05:08:04.397303+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:06:45.568389+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:06:45.568389+0000","last_clean_scrub_stamp":"2024-04-02T05:06:45.568389+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:10.544744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":186,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006740+0000","last_change":"2024-04-02T05:07:54.484663+0000","last_active":"2024-04-02T05:08:05.006740+0000","last_peered":"2024-04-02T05:08:05.006740+0000","last_clean":"2024-04-02T05:08:05.006740+0000","last_became_active":"2024-04-02T05:05:16.548018+0000","last_became_peered":"2024-04-02T05:05:16.548018+0000","last_unstale":"2024-04-02T05:08:05.006740+0000","last_undegraded":"2024-04-02T05:08:05.006740+0000","last_fullsized":"2024-04-02T05:08:05.006740+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.484452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:54.484452+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.484452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:24.426306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"642'16","reported_seq":823,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.433124+0000","last_change":"2024-04-02T05:07:37.633477+0000","last_active":"2024-04-02T05:08:10.433124+0000","last_peered":"2024-04-02T05:08:10.433124+0000","last_clean":"2024-04-02T05:08:10.433124+0000","last_became_active":"2024-04-02T04:59:00.134138+0000","last_became_peered":"2024-04-02T04:59:00.134138+0000","last_unstale":"2024-04-02T05:08:10.433124+0000","last_undegraded":"2024-04-02T05:08:10.433124+0000","last_fullsized":"2024-04-02T05:08:10.433124+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'16","last_scrub_stamp":"2024-04-02T05:07:37.633367+0000","last_deep_scrub":"642'16","last_deep_scrub_stamp":"2024-04-02T05:05:15.903540+0000","last_clean_scrub_stamp":"2024-04-02T05:07:37.633367+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.483933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"899'33","reported_seq":733,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.403319+0000","last_change":"2024-04-02T05:07:53.458580+0000","last_active":"2024-04-02T05:08:04.403319+0000","last_peered":"2024-04-02T05:08:04.403319+0000","last_clean":"2024-04-02T05:08:04.403319+0000","last_became_active":"2024-04-02T04:59:00.134091+0000","last_became_peered":"2024-04-02T04:59:00.134091+0000","last_unstale":"2024-04-02T05:08:04.403319+0000","last_undegraded":"2024-04-02T05:08:04.403319+0000","last_fullsized":"2024-04-02T05:08:04.403319+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'33","last_scrub_stamp":"2024-04-02T05:07:53.458399+0000","last_deep_scrub":"824'29","last_deep_scrub_stamp":"2024-04-02T05:02:16.780210+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.458399+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.517576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"121'2","reported_seq":1888,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:15.632803+0000","last_change":"2024-04-02T05:08:15.632874+0000","last_active":"2024-04-02T05:08:15.632803+0000","last_peered":"2024-04-02T05:08:15.632803+0000","last_clean":"2024-04-02T05:08:15.632803+0000","last_became_active":"2024-04-02T04:43:54.232044+0000","last_became_peered":"2024-04-02T04:43:54.232044+0000","last_unstale":"2024-04-02T05:08:15.632803+0000","last_undegraded":"2024-04-02T05:08:15.632803+0000","last_fullsized":"2024-04-02T05:08:15.632803+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-02T05:08:15.632642+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-02T05:07:08.760668+0000","last_clean_scrub_stamp":"2024-04-02T05:08:15.632642+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:38.658429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1898,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396181+0000","last_change":"2024-04-02T05:07:35.610591+0000","last_active":"2024-04-02T05:08:04.396181+0000","last_peered":"2024-04-02T05:08:04.396181+0000","last_clean":"2024-04-02T05:08:04.396181+0000","last_became_active":"2024-04-02T04:59:00.135841+0000","last_became_peered":"2024-04-02T04:59:00.135841+0000","last_unstale":"2024-04-02T05:08:04.396181+0000","last_undegraded":"2024-04-02T05:08:04.396181+0000","last_fullsized":"2024-04-02T05:08:04.396181+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:35.610373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:35.610373+0000","last_clean_scrub_stamp":"2024-04-02T05:07:35.610373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.495590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017018140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":82,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:08.692227+0000","last_change":"2024-04-02T05:08:08.692283+0000","last_active":"2024-04-02T05:08:08.692227+0000","last_peered":"2024-04-02T05:08:08.692227+0000","last_clean":"2024-04-02T05:08:08.692227+0000","last_became_active":"2024-04-02T05:05:54.274830+0000","last_became_peered":"2024-04-02T05:05:54.274830+0000","last_unstale":"2024-04-02T05:08:08.692227+0000","last_undegraded":"2024-04-02T05:08:08.692227+0000","last_fullsized":"2024-04-02T05:08:08.692227+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:08.692073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:08:08.692073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:27.113749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"101'128","reported_seq":1367,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667315+0000","last_change":"2024-04-02T05:07:50.834842+0000","last_active":"2024-04-02T05:08:04.667315+0000","last_peered":"2024-04-02T05:08:04.667315+0000","last_clean":"2024-04-02T05:08:04.667315+0000","last_became_active":"2024-04-02T05:00:42.158023+0000","last_became_peered":"2024-04-02T05:00:42.158023+0000","last_unstale":"2024-04-02T05:08:04.667315+0000","last_undegraded":"2024-04-02T05:08:04.667315+0000","last_fullsized":"2024-04-02T05:08:04.667315+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:50.834610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:07:50.834610+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:19.534776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1720,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667294+0000","last_change":"2024-04-02T05:08:01.911079+0000","last_active":"2024-04-02T05:08:04.667294+0000","last_peered":"2024-04-02T05:08:04.667294+0000","last_clean":"2024-04-02T05:08:04.667294+0000","last_became_active":"2024-04-02T04:49:26.183760+0000","last_became_peered":"2024-04-02T04:49:26.183760+0000","last_unstale":"2024-04-02T05:08:04.667294+0000","last_undegraded":"2024-04-02T05:08:04.667294+0000","last_fullsized":"2024-04-02T05:08:04.667294+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:01.910857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:16.918634+0000","last_clean_scrub_stamp":"2024-04-02T05:08:01.910857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:23.455489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"913'18","reported_seq":2015,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.433103+0000","last_change":"2024-04-02T05:08:05.983724+0000","last_active":"2024-04-02T05:08:10.433103+0000","last_peered":"2024-04-02T05:08:10.433103+0000","last_clean":"2024-04-02T05:08:10.433103+0000","last_became_active":"2024-04-02T05:04:49.246171+0000","last_became_peered":"2024-04-02T05:04:49.246171+0000","last_unstale":"2024-04-02T05:08:10.433103+0000","last_undegraded":"2024-04-02T05:08:10.433103+0000","last_fullsized":"2024-04-02T05:08:10.433103+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'18","last_scrub_stamp":"2024-04-02T05:08:05.983491+0000","last_deep_scrub":"913'18","last_deep_scrub_stamp":"2024-04-02T05:08:05.983491+0000","last_clean_scrub_stamp":"2024-04-02T05:08:05.983491+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:22.023395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"923'75","reported_seq":2405,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.433256+0000","last_change":"2024-04-02T05:07:52.635084+0000","last_active":"2024-04-02T05:08:10.433256+0000","last_peered":"2024-04-02T05:08:10.433256+0000","last_clean":"2024-04-02T05:08:10.433256+0000","last_became_active":"2024-04-02T05:05:16.541267+0000","last_became_peered":"2024-04-02T05:05:16.541267+0000","last_unstale":"2024-04-02T05:08:10.433256+0000","last_undegraded":"2024-04-02T05:08:10.433256+0000","last_fullsized":"2024-04-02T05:08:10.433256+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'75","last_scrub_stamp":"2024-04-02T05:07:52.634835+0000","last_deep_scrub":"902'74","last_deep_scrub_stamp":"2024-04-02T05:04:57.843581+0000","last_clean_scrub_stamp":"2024-04-02T05:07:52.634835+0000","objects_scrubbed":11,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:04.951665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006867+0000","last_change":"2024-04-02T05:07:24.537256+0000","last_active":"2024-04-02T05:08:05.006867+0000","last_peered":"2024-04-02T05:08:05.006867+0000","last_clean":"2024-04-02T05:08:05.006867+0000","last_became_active":"2024-04-02T04:53:54.941913+0000","last_became_peered":"2024-04-02T04:53:54.941913+0000","last_unstale":"2024-04-02T05:08:05.006867+0000","last_undegraded":"2024-04-02T05:08:05.006867+0000","last_fullsized":"2024-04-02T05:08:05.006867+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:24.537040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:24.537040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.680818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"963'3","reported_seq":1895,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:12.882686+0000","last_change":"2024-04-02T05:08:12.882758+0000","last_active":"2024-04-02T05:08:12.882686+0000","last_peered":"2024-04-02T05:08:12.882686+0000","last_clean":"2024-04-02T05:08:12.882686+0000","last_became_active":"2024-04-02T05:04:11.248836+0000","last_became_peered":"2024-04-02T05:04:11.248836+0000","last_unstale":"2024-04-02T05:08:12.882686+0000","last_undegraded":"2024-04-02T05:08:12.882686+0000","last_fullsized":"2024-04-02T05:08:12.882686+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'3","last_scrub_stamp":"2024-04-02T05:08:12.882527+0000","last_deep_scrub":"963'3","last_deep_scrub_stamp":"2024-04-02T05:07:06.725621+0000","last_clean_scrub_stamp":"2024-04-02T05:08:12.882527+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:26.220639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010497799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1870,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396003+0000","last_change":"2024-04-02T05:07:09.788379+0000","last_active":"2024-04-02T05:08:04.396003+0000","last_peered":"2024-04-02T05:08:04.396003+0000","last_clean":"2024-04-02T05:08:04.396003+0000","last_became_active":"2024-04-02T05:05:16.534493+0000","last_became_peered":"2024-04-02T05:05:16.534493+0000","last_unstale":"2024-04-02T05:08:04.396003+0000","last_undegraded":"2024-04-02T05:08:04.396003+0000","last_fullsized":"2024-04-02T05:08:04.396003+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:09.788081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:48:04.001090+0000","last_clean_scrub_stamp":"2024-04-02T05:07:09.788081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.983066+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'128","reported_seq":1886,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399076+0000","last_change":"2024-04-02T05:07:37.636248+0000","last_active":"2024-04-02T05:08:04.399076+0000","last_peered":"2024-04-02T05:08:04.399076+0000","last_clean":"2024-04-02T05:08:04.399076+0000","last_became_active":"2024-04-02T05:06:34.716887+0000","last_became_peered":"2024-04-02T05:06:34.716887+0000","last_unstale":"2024-04-02T05:08:04.399076+0000","last_undegraded":"2024-04-02T05:08:04.399076+0000","last_fullsized":"2024-04-02T05:08:04.399076+0000","mapping_epoch":955,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":956,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:37.636152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:07:37.636152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:44.889862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002126+0000","last_change":"2024-04-02T05:08:01.180720+0000","last_active":"2024-04-02T05:08:05.002126+0000","last_peered":"2024-04-02T05:08:05.002126+0000","last_clean":"2024-04-02T05:08:05.002126+0000","last_became_active":"2024-04-02T05:04:11.244090+0000","last_became_peered":"2024-04-02T05:04:11.244090+0000","last_unstale":"2024-04-02T05:08:05.002126+0000","last_undegraded":"2024-04-02T05:08:05.002126+0000","last_fullsized":"2024-04-02T05:08:05.002126+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:01.180481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:39.111154+0000","last_clean_scrub_stamp":"2024-04-02T05:08:01.180481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1899,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074187+0000","last_change":"2024-04-02T05:08:02.042395+0000","last_active":"2024-04-02T05:08:05.074187+0000","last_peered":"2024-04-02T05:08:05.074187+0000","last_clean":"2024-04-02T05:08:05.074187+0000","last_became_active":"2024-04-02T05:06:58.342937+0000","last_became_peered":"2024-04-02T05:06:58.342937+0000","last_unstale":"2024-04-02T05:08:05.074187+0000","last_undegraded":"2024-04-02T05:08:05.074187+0000","last_fullsized":"2024-04-02T05:08:05.074187+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:02.042192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:33.089451+0000","last_clean_scrub_stamp":"2024-04-02T05:08:02.042192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:08.179643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1341,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004499+0000","last_change":"2024-04-02T05:07:52.456071+0000","last_active":"2024-04-02T05:08:05.004499+0000","last_peered":"2024-04-02T05:08:05.004499+0000","last_clean":"2024-04-02T05:08:05.004499+0000","last_became_active":"2024-04-02T04:53:54.950261+0000","last_became_peered":"2024-04-02T04:53:54.950261+0000","last_unstale":"2024-04-02T05:08:05.004499+0000","last_undegraded":"2024-04-02T05:08:05.004499+0000","last_fullsized":"2024-04-02T05:08:05.004499+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:52.455804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:07.525265+0000","last_clean_scrub_stamp":"2024-04-02T05:07:52.455804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.186658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"899'56","reported_seq":2076,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.982441+0000","last_change":"2024-04-02T05:07:57.093142+0000","last_active":"2024-04-02T05:08:04.982441+0000","last_peered":"2024-04-02T05:08:04.982441+0000","last_clean":"2024-04-02T05:08:04.982441+0000","last_became_active":"2024-04-02T05:05:54.272867+0000","last_became_peered":"2024-04-02T05:05:54.272867+0000","last_unstale":"2024-04-02T05:08:04.982441+0000","last_undegraded":"2024-04-02T05:08:04.982441+0000","last_fullsized":"2024-04-02T05:08:04.982441+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'56","last_scrub_stamp":"2024-04-02T05:07:57.092887+0000","last_deep_scrub":"899'56","last_deep_scrub_stamp":"2024-04-02T05:07:57.092887+0000","last_clean_scrub_stamp":"2024-04-02T05:07:57.092887+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:17.555596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":140,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.402434+0000","last_change":"2024-04-02T05:07:38.650816+0000","last_active":"2024-04-02T05:08:04.402434+0000","last_peered":"2024-04-02T05:08:04.402434+0000","last_clean":"2024-04-02T05:08:04.402434+0000","last_became_active":"2024-04-02T05:04:58.366649+0000","last_became_peered":"2024-04-02T05:04:58.366649+0000","last_unstale":"2024-04-02T05:08:04.402434+0000","last_undegraded":"2024-04-02T05:08:04.402434+0000","last_fullsized":"2024-04-02T05:08:04.402434+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:38.650551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:38.650551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:00.510973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1720,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666881+0000","last_change":"2024-04-02T05:07:47.790135+0000","last_active":"2024-04-02T05:08:04.666881+0000","last_peered":"2024-04-02T05:08:04.666881+0000","last_clean":"2024-04-02T05:08:04.666881+0000","last_became_active":"2024-04-02T04:49:26.301270+0000","last_became_peered":"2024-04-02T04:49:26.301270+0000","last_unstale":"2024-04-02T05:08:04.666881+0000","last_undegraded":"2024-04-02T05:08:04.666881+0000","last_fullsized":"2024-04-02T05:08:04.666881+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:47.789917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:47.789917+0000","last_clean_scrub_stamp":"2024-04-02T05:07:47.789917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:13.222507+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1878,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667677+0000","last_change":"2024-04-02T05:07:17.721865+0000","last_active":"2024-04-02T05:08:04.667677+0000","last_peered":"2024-04-02T05:08:04.667677+0000","last_clean":"2024-04-02T05:08:04.667677+0000","last_became_active":"2024-04-02T05:05:54.277066+0000","last_became_peered":"2024-04-02T05:05:54.277066+0000","last_unstale":"2024-04-02T05:08:04.667677+0000","last_undegraded":"2024-04-02T05:08:04.667677+0000","last_fullsized":"2024-04-02T05:08:04.667677+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:17.721660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:17.721660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:32.313455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"101'128","reported_seq":1629,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004301+0000","last_change":"2024-04-02T05:07:26.590476+0000","last_active":"2024-04-02T05:08:05.004301+0000","last_peered":"2024-04-02T05:08:05.004301+0000","last_clean":"2024-04-02T05:08:05.004301+0000","last_became_active":"2024-04-02T05:00:42.145480+0000","last_became_peered":"2024-04-02T05:00:42.145480+0000","last_unstale":"2024-04-02T05:08:05.004301+0000","last_undegraded":"2024-04-02T05:08:05.004301+0000","last_fullsized":"2024-04-02T05:08:05.004301+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:26.590246+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:26.590246+0000","last_clean_scrub_stamp":"2024-04-02T05:07:26.590246+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.687128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.103719+0000","last_change":"2024-04-02T05:08:05.103776+0000","last_active":"2024-04-02T05:08:05.103719+0000","last_peered":"2024-04-02T05:08:05.103719+0000","last_clean":"2024-04-02T05:08:05.103719+0000","last_became_active":"2024-04-02T05:04:11.249101+0000","last_became_peered":"2024-04-02T05:04:11.249101+0000","last_unstale":"2024-04-02T05:08:05.103719+0000","last_undegraded":"2024-04-02T05:08:05.103719+0000","last_fullsized":"2024-04-02T05:08:05.103719+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:05.103555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:55.305293+0000","last_clean_scrub_stamp":"2024-04-02T05:08:05.103555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:27.513983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"899'36","reported_seq":756,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395491+0000","last_change":"2024-04-02T05:07:59.430078+0000","last_active":"2024-04-02T05:08:04.395491+0000","last_peered":"2024-04-02T05:08:04.395491+0000","last_clean":"2024-04-02T05:08:04.395491+0000","last_became_active":"2024-04-02T05:04:11.244441+0000","last_became_peered":"2024-04-02T05:04:11.244441+0000","last_unstale":"2024-04-02T05:08:04.395491+0000","last_undegraded":"2024-04-02T05:08:04.395491+0000","last_fullsized":"2024-04-02T05:08:04.395491+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'36","last_scrub_stamp":"2024-04-02T05:07:59.429923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:59.429923+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:12.306308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005740+0000","last_change":"2024-04-02T05:07:54.161006+0000","last_active":"2024-04-02T05:08:05.005740+0000","last_peered":"2024-04-02T05:08:05.005740+0000","last_clean":"2024-04-02T05:08:05.005740+0000","last_became_active":"2024-04-02T05:04:11.248679+0000","last_became_peered":"2024-04-02T05:04:11.248679+0000","last_unstale":"2024-04-02T05:08:05.005740+0000","last_undegraded":"2024-04-02T05:08:05.005740+0000","last_fullsized":"2024-04-02T05:08:05.005740+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:54.160760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:25.573285+0000","last_clean_scrub_stamp":"2024-04-02T05:07:54.160760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:13.538179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"101'128","reported_seq":1681,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005110+0000","last_change":"2024-04-02T05:07:17.661945+0000","last_active":"2024-04-02T05:08:05.005110+0000","last_peered":"2024-04-02T05:08:05.005110+0000","last_clean":"2024-04-02T05:08:05.005110+0000","last_became_active":"2024-04-02T05:05:57.345300+0000","last_became_peered":"2024-04-02T05:05:57.345300+0000","last_unstale":"2024-04-02T05:08:05.005110+0000","last_undegraded":"2024-04-02T05:08:05.005110+0000","last_fullsized":"2024-04-02T05:08:05.005110+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:17.661725+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:17.661725+0000","last_clean_scrub_stamp":"2024-04-02T05:07:17.661725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:33.567552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":174,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004798+0000","last_change":"2024-04-02T05:07:10.228915+0000","last_active":"2024-04-02T05:08:05.004798+0000","last_peered":"2024-04-02T05:08:05.004798+0000","last_clean":"2024-04-02T05:08:05.004798+0000","last_became_active":"2024-04-02T05:05:16.541976+0000","last_became_peered":"2024-04-02T05:05:16.541976+0000","last_unstale":"2024-04-02T05:08:05.004798+0000","last_undegraded":"2024-04-02T05:08:05.004798+0000","last_fullsized":"2024-04-02T05:08:05.004798+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:10.228639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:16.726893+0000","last_clean_scrub_stamp":"2024-04-02T05:07:10.228639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:38.742421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":132,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401075+0000","last_change":"2024-04-02T05:07:07.750918+0000","last_active":"2024-04-02T05:08:04.401075+0000","last_peered":"2024-04-02T05:08:04.401075+0000","last_clean":"2024-04-02T05:08:04.401075+0000","last_became_active":"2024-04-02T05:04:58.366684+0000","last_became_peered":"2024-04-02T05:04:58.366684+0000","last_unstale":"2024-04-02T05:08:04.401075+0000","last_undegraded":"2024-04-02T05:08:04.401075+0000","last_fullsized":"2024-04-02T05:08:04.401075+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:07.750520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:07.750520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:25.056645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870749+0000","last_change":"2024-04-02T05:07:07.752347+0000","last_active":"2024-04-02T05:08:04.870749+0000","last_peered":"2024-04-02T05:08:04.870749+0000","last_clean":"2024-04-02T05:08:04.870749+0000","last_became_active":"2024-04-02T05:05:16.542650+0000","last_became_peered":"2024-04-02T05:05:16.542650+0000","last_unstale":"2024-04-02T05:08:04.870749+0000","last_undegraded":"2024-04-02T05:08:04.870749+0000","last_fullsized":"2024-04-02T05:08:04.870749+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:07.752137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:44:57.985236+0000","last_clean_scrub_stamp":"2024-04-02T05:07:07.752137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:36.052305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"899'54","reported_seq":2023,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397410+0000","last_change":"2024-04-02T05:07:01.763184+0000","last_active":"2024-04-02T05:08:04.397410+0000","last_peered":"2024-04-02T05:08:04.397410+0000","last_clean":"2024-04-02T05:08:04.397410+0000","last_became_active":"2024-04-02T05:05:16.533054+0000","last_became_peered":"2024-04-02T05:05:16.533054+0000","last_unstale":"2024-04-02T05:08:04.397410+0000","last_undegraded":"2024-04-02T05:08:04.397410+0000","last_fullsized":"2024-04-02T05:08:04.397410+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'54","last_scrub_stamp":"2024-04-02T05:07:01.762942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:01.762942+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:17.239990+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1357,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.570026+0000","last_change":"2024-04-02T05:08:10.570109+0000","last_active":"2024-04-02T05:08:10.570026+0000","last_peered":"2024-04-02T05:08:10.570026+0000","last_clean":"2024-04-02T05:08:10.570026+0000","last_became_active":"2024-04-02T05:06:31.706429+0000","last_became_peered":"2024-04-02T05:06:31.706429+0000","last_unstale":"2024-04-02T05:08:10.570026+0000","last_undegraded":"2024-04-02T05:08:10.570026+0000","last_fullsized":"2024-04-02T05:08:10.570026+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:10.569808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:08:10.569808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:39.796267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'128","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005358+0000","last_change":"2024-04-02T05:07:47.112277+0000","last_active":"2024-04-02T05:08:05.005358+0000","last_peered":"2024-04-02T05:08:05.005358+0000","last_clean":"2024-04-02T05:08:05.005358+0000","last_became_active":"2024-04-02T05:04:14.205875+0000","last_became_peered":"2024-04-02T05:04:14.205875+0000","last_unstale":"2024-04-02T05:08:05.005358+0000","last_undegraded":"2024-04-02T05:08:05.005358+0000","last_fullsized":"2024-04-02T05:08:05.005358+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:47.112015+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:47.112015+0000","last_clean_scrub_stamp":"2024-04-02T05:07:47.112015+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:57.574668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1732,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666846+0000","last_change":"2024-04-02T05:07:21.631349+0000","last_active":"2024-04-02T05:08:04.666846+0000","last_peered":"2024-04-02T05:08:04.666846+0000","last_clean":"2024-04-02T05:08:04.666846+0000","last_became_active":"2024-04-02T05:04:11.250256+0000","last_became_peered":"2024-04-02T05:04:11.250256+0000","last_unstale":"2024-04-02T05:08:04.666846+0000","last_undegraded":"2024-04-02T05:08:04.666846+0000","last_fullsized":"2024-04-02T05:08:04.666846+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:21.631110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:21.631110+0000","last_clean_scrub_stamp":"2024-04-02T05:07:21.631110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:41.716023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1908,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:09.665450+0000","last_change":"2024-04-02T05:08:09.665507+0000","last_active":"2024-04-02T05:08:09.665450+0000","last_peered":"2024-04-02T05:08:09.665450+0000","last_clean":"2024-04-02T05:08:09.665450+0000","last_became_active":"2024-04-02T05:05:16.531568+0000","last_became_peered":"2024-04-02T05:05:16.531568+0000","last_unstale":"2024-04-02T05:08:09.665450+0000","last_undegraded":"2024-04-02T05:08:09.665450+0000","last_fullsized":"2024-04-02T05:08:09.665450+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:09.665299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:47:47.106497+0000","last_clean_scrub_stamp":"2024-04-02T05:08:09.665299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:34.366610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"899'45","reported_seq":1960,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397642+0000","last_change":"2024-04-02T05:07:55.690006+0000","last_active":"2024-04-02T05:08:04.397642+0000","last_peered":"2024-04-02T05:08:04.397642+0000","last_clean":"2024-04-02T05:08:04.397642+0000","last_became_active":"2024-04-02T04:43:56.240521+0000","last_became_peered":"2024-04-02T04:43:56.240521+0000","last_unstale":"2024-04-02T05:08:04.397642+0000","last_undegraded":"2024-04-02T05:08:04.397642+0000","last_fullsized":"2024-04-02T05:08:04.397642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'45","last_scrub_stamp":"2024-04-02T05:07:55.689760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:55.689760+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:23.320627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003521+0000","last_change":"2024-04-02T05:07:48.118210+0000","last_active":"2024-04-02T05:08:05.003521+0000","last_peered":"2024-04-02T05:08:05.003521+0000","last_clean":"2024-04-02T05:08:05.003521+0000","last_became_active":"2024-04-02T05:04:11.259878+0000","last_became_peered":"2024-04-02T05:04:11.259878+0000","last_unstale":"2024-04-02T05:08:05.003521+0000","last_undegraded":"2024-04-02T05:08:05.003521+0000","last_fullsized":"2024-04-02T05:08:05.003521+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:48.117999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.117999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:02.211490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"106'2","reported_seq":711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400379+0000","last_change":"2024-04-02T05:07:22.562222+0000","last_active":"2024-04-02T05:08:04.400379+0000","last_peered":"2024-04-02T05:08:04.400379+0000","last_clean":"2024-04-02T05:08:04.400379+0000","last_became_active":"2024-04-02T04:59:00.128819+0000","last_became_peered":"2024-04-02T04:59:00.128819+0000","last_unstale":"2024-04-02T05:08:04.400379+0000","last_undegraded":"2024-04-02T05:08:04.400379+0000","last_fullsized":"2024-04-02T05:08:04.400379+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-04-02T05:07:22.561886+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-04-02T04:59:04.967881+0000","last_clean_scrub_stamp":"2024-04-02T05:07:22.561886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:29.778515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'128","reported_seq":1385,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668169+0000","last_change":"2024-04-02T05:07:48.823388+0000","last_active":"2024-04-02T05:08:04.668169+0000","last_peered":"2024-04-02T05:08:04.668169+0000","last_clean":"2024-04-02T05:08:04.668169+0000","last_became_active":"2024-04-02T05:00:42.122155+0000","last_became_peered":"2024-04-02T05:00:42.122155+0000","last_unstale":"2024-04-02T05:08:04.668169+0000","last_undegraded":"2024-04-02T05:08:04.668169+0000","last_fullsized":"2024-04-02T05:08:04.668169+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":479,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:48.823135+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:05:02.462796+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.823135+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:54.600448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1203,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006591+0000","last_change":"2024-04-02T05:07:22.562164+0000","last_active":"2024-04-02T05:08:05.006591+0000","last_peered":"2024-04-02T05:08:05.006591+0000","last_clean":"2024-04-02T05:08:05.006591+0000","last_became_active":"2024-04-02T04:53:54.941875+0000","last_became_peered":"2024-04-02T04:53:54.941875+0000","last_unstale":"2024-04-02T05:08:05.006591+0000","last_undegraded":"2024-04-02T05:08:05.006591+0000","last_fullsized":"2024-04-02T05:08:05.006591+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:22.561811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:22.561811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.405822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":146,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398974+0000","last_change":"2024-04-02T05:07:46.628675+0000","last_active":"2024-04-02T05:08:04.398974+0000","last_peered":"2024-04-02T05:08:04.398974+0000","last_clean":"2024-04-02T05:08:04.398974+0000","last_became_active":"2024-04-02T05:05:54.274401+0000","last_became_peered":"2024-04-02T05:05:54.274401+0000","last_unstale":"2024-04-02T05:08:04.398974+0000","last_undegraded":"2024-04-02T05:08:04.398974+0000","last_fullsized":"2024-04-02T05:08:04.398974+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:46.628541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:46.628541+0000","last_clean_scrub_stamp":"2024-04-02T05:07:46.628541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:49.663608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"902'44","reported_seq":244,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.434916+0000","last_change":"2024-04-02T05:07:58.167702+0000","last_active":"2024-04-02T05:08:10.434916+0000","last_peered":"2024-04-02T05:08:10.434916+0000","last_clean":"2024-04-02T05:08:10.434916+0000","last_became_active":"2024-04-02T05:04:11.254466+0000","last_became_peered":"2024-04-02T05:04:11.254466+0000","last_unstale":"2024-04-02T05:08:10.434916+0000","last_undegraded":"2024-04-02T05:08:10.434916+0000","last_fullsized":"2024-04-02T05:08:10.434916+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'44","last_scrub_stamp":"2024-04-02T05:07:58.167425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.167425+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:04.286283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":1887,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401311+0000","last_change":"2024-04-02T05:07:12.771912+0000","last_active":"2024-04-02T05:08:04.401311+0000","last_peered":"2024-04-02T05:08:04.401311+0000","last_clean":"2024-04-02T05:08:04.401311+0000","last_became_active":"2024-04-02T04:53:54.941074+0000","last_became_peered":"2024-04-02T04:53:54.941074+0000","last_unstale":"2024-04-02T05:08:04.401311+0000","last_undegraded":"2024-04-02T05:08:04.401311+0000","last_fullsized":"2024-04-02T05:08:04.401311+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-02T05:07:12.771646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:12.771646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:22.704784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1896,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.981218+0000","last_change":"2024-04-02T05:07:36.083372+0000","last_active":"2024-04-02T05:08:04.981218+0000","last_peered":"2024-04-02T05:08:04.981218+0000","last_clean":"2024-04-02T05:08:04.981218+0000","last_became_active":"2024-04-02T05:05:16.535656+0000","last_became_peered":"2024-04-02T05:05:16.535656+0000","last_unstale":"2024-04-02T05:08:04.981218+0000","last_undegraded":"2024-04-02T05:08:04.981218+0000","last_fullsized":"2024-04-02T05:08:04.981218+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:36.083150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:45.610342+0000","last_clean_scrub_stamp":"2024-04-02T05:07:36.083150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:00.992666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"101'128","reported_seq":1336,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395857+0000","last_change":"2024-04-02T05:08:02.475711+0000","last_active":"2024-04-02T05:08:04.395857+0000","last_peered":"2024-04-02T05:08:04.395857+0000","last_clean":"2024-04-02T05:08:04.395857+0000","last_became_active":"2024-04-02T05:00:47.116787+0000","last_became_peered":"2024-04-02T05:00:47.116787+0000","last_unstale":"2024-04-02T05:08:04.395857+0000","last_undegraded":"2024-04-02T05:08:04.395857+0000","last_fullsized":"2024-04-02T05:08:04.395857+0000","mapping_epoch":736,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:02.475610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:08:02.475610+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.035285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1750,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668296+0000","last_change":"2024-04-02T05:07:28.799135+0000","last_active":"2024-04-02T05:08:04.668296+0000","last_peered":"2024-04-02T05:08:04.668296+0000","last_clean":"2024-04-02T05:08:04.668296+0000","last_became_active":"2024-04-02T05:04:11.250122+0000","last_became_peered":"2024-04-02T05:04:11.250122+0000","last_unstale":"2024-04-02T05:08:04.668296+0000","last_undegraded":"2024-04-02T05:08:04.668296+0000","last_fullsized":"2024-04-02T05:08:04.668296+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:28.798827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:16.100623+0000","last_clean_scrub_stamp":"2024-04-02T05:07:28.798827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:46.169552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"642'14","reported_seq":2202,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.434196+0000","last_change":"2024-04-02T05:07:03.540482+0000","last_active":"2024-04-02T05:08:10.434196+0000","last_peered":"2024-04-02T05:08:10.434196+0000","last_clean":"2024-04-02T05:08:10.434196+0000","last_became_active":"2024-04-02T04:59:00.137930+0000","last_became_peered":"2024-04-02T04:59:00.137930+0000","last_unstale":"2024-04-02T05:08:10.434196+0000","last_undegraded":"2024-04-02T05:08:10.434196+0000","last_fullsized":"2024-04-02T05:08:10.434196+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-04-02T05:07:03.540149+0000","last_deep_scrub":"642'14","last_deep_scrub_stamp":"2024-04-02T05:07:03.540149+0000","last_clean_scrub_stamp":"2024-04-02T05:07:03.540149+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:32.888678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"899'33","reported_seq":1443,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007391+0000","last_change":"2024-04-02T05:07:46.491568+0000","last_active":"2024-04-02T05:08:05.007391+0000","last_peered":"2024-04-02T05:08:05.007391+0000","last_clean":"2024-04-02T05:08:05.007391+0000","last_became_active":"2024-04-02T05:05:54.270036+0000","last_became_peered":"2024-04-02T05:05:54.270036+0000","last_unstale":"2024-04-02T05:08:05.007391+0000","last_undegraded":"2024-04-02T05:08:05.007391+0000","last_fullsized":"2024-04-02T05:08:05.007391+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'33","last_scrub_stamp":"2024-04-02T05:07:46.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:46.491320+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:10.683151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":1910,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.770552+0000","last_change":"2024-04-02T05:08:10.770630+0000","last_active":"2024-04-02T05:08:10.770552+0000","last_peered":"2024-04-02T05:08:10.770552+0000","last_clean":"2024-04-02T05:08:10.770552+0000","last_became_active":"2024-04-02T04:53:54.904445+0000","last_became_peered":"2024-04-02T04:53:54.904445+0000","last_unstale":"2024-04-02T05:08:10.770552+0000","last_undegraded":"2024-04-02T05:08:10.770552+0000","last_fullsized":"2024-04-02T05:08:10.770552+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-02T05:08:10.770389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:08:10.770389+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:40.242938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":699,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397173+0000","last_change":"2024-04-02T05:07:41.618915+0000","last_active":"2024-04-02T05:08:04.397173+0000","last_peered":"2024-04-02T05:08:04.397173+0000","last_clean":"2024-04-02T05:08:04.397173+0000","last_became_active":"2024-04-02T04:59:00.122883+0000","last_became_peered":"2024-04-02T04:59:00.122883+0000","last_unstale":"2024-04-02T05:08:04.397173+0000","last_undegraded":"2024-04-02T05:08:04.397173+0000","last_fullsized":"2024-04-02T05:08:04.397173+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:41.618628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:41.618628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.210559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"101'128","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.693882+0000","last_change":"2024-04-02T05:08:07.694000+0000","last_active":"2024-04-02T05:08:07.693882+0000","last_peered":"2024-04-02T05:08:07.693882+0000","last_clean":"2024-04-02T05:08:07.693882+0000","last_became_active":"2024-04-02T05:05:57.347095+0000","last_became_peered":"2024-04-02T05:05:57.347095+0000","last_unstale":"2024-04-02T05:08:07.693882+0000","last_undegraded":"2024-04-02T05:08:07.693882+0000","last_fullsized":"2024-04-02T05:08:07.693882+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:08:07.693611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:57.721978+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.693611+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:30.323022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870697+0000","last_change":"2024-04-02T05:07:21.828734+0000","last_active":"2024-04-02T05:08:04.870697+0000","last_peered":"2024-04-02T05:08:04.870697+0000","last_clean":"2024-04-02T05:08:04.870697+0000","last_became_active":"2024-04-02T05:05:16.545382+0000","last_became_peered":"2024-04-02T05:05:16.545382+0000","last_unstale":"2024-04-02T05:08:04.870697+0000","last_undegraded":"2024-04-02T05:08:04.870697+0000","last_fullsized":"2024-04-02T05:08:04.870697+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:21.828523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:30.126652+0000","last_clean_scrub_stamp":"2024-04-02T05:07:21.828523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:49.925302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"913'39","reported_seq":1399,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005420+0000","last_change":"2024-04-02T05:07:48.508225+0000","last_active":"2024-04-02T05:08:05.005420+0000","last_peered":"2024-04-02T05:08:05.005420+0000","last_clean":"2024-04-02T05:08:05.005420+0000","last_became_active":"2024-04-02T04:53:54.922917+0000","last_became_peered":"2024-04-02T04:53:54.922917+0000","last_unstale":"2024-04-02T05:08:05.005420+0000","last_undegraded":"2024-04-02T05:08:05.005420+0000","last_fullsized":"2024-04-02T05:08:05.005420+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'39","last_scrub_stamp":"2024-04-02T05:07:48.507909+0000","last_deep_scrub":"913'39","last_deep_scrub_stamp":"2024-04-02T05:07:48.507909+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.507909+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:03.501904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871079+0000","last_change":"2024-04-02T05:07:15.730014+0000","last_active":"2024-04-02T05:08:04.871079+0000","last_peered":"2024-04-02T05:08:04.871079+0000","last_clean":"2024-04-02T05:08:04.871079+0000","last_became_active":"2024-04-02T05:05:16.538443+0000","last_became_peered":"2024-04-02T05:05:16.538443+0000","last_unstale":"2024-04-02T05:08:04.871079+0000","last_undegraded":"2024-04-02T05:08:04.871079+0000","last_fullsized":"2024-04-02T05:08:04.871079+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:15.729881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:15.729881+0000","last_clean_scrub_stamp":"2024-04-02T05:07:15.729881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.089676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1873,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074502+0000","last_change":"2024-04-02T05:07:28.976406+0000","last_active":"2024-04-02T05:08:05.074502+0000","last_peered":"2024-04-02T05:08:05.074502+0000","last_clean":"2024-04-02T05:08:05.074502+0000","last_became_active":"2024-04-02T04:59:00.138663+0000","last_became_peered":"2024-04-02T04:59:00.138663+0000","last_unstale":"2024-04-02T05:08:05.074502+0000","last_undegraded":"2024-04-02T05:08:05.074502+0000","last_fullsized":"2024-04-02T05:08:05.074502+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:28.976184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:28.976184+0000","last_clean_scrub_stamp":"2024-04-02T05:07:28.976184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.759955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.870870+0000","last_change":"2024-04-02T05:07:33.944141+0000","last_active":"2024-04-02T05:08:04.870870+0000","last_peered":"2024-04-02T05:08:04.870870+0000","last_clean":"2024-04-02T05:08:04.870870+0000","last_became_active":"2024-04-02T05:05:16.546959+0000","last_became_peered":"2024-04-02T05:05:16.546959+0000","last_unstale":"2024-04-02T05:08:04.870870+0000","last_undegraded":"2024-04-02T05:08:04.870870+0000","last_fullsized":"2024-04-02T05:08:04.870870+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:33.943921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:25.773178+0000","last_clean_scrub_stamp":"2024-04-02T05:07:33.943921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:53.140964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"101'128","reported_seq":1317,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003808+0000","last_change":"2024-04-02T05:07:57.213543+0000","last_active":"2024-04-02T05:08:05.003808+0000","last_peered":"2024-04-02T05:08:05.003808+0000","last_clean":"2024-04-02T05:08:05.003808+0000","last_became_active":"2024-04-02T05:04:14.204099+0000","last_became_peered":"2024-04-02T05:04:14.204099+0000","last_unstale":"2024-04-02T05:08:05.003808+0000","last_undegraded":"2024-04-02T05:08:05.003808+0000","last_fullsized":"2024-04-02T05:08:05.003808+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:57.213308+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:57.213308+0000","last_clean_scrub_stamp":"2024-04-02T05:07:57.213308+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:17.535160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1737,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.394679+0000","last_change":"2024-04-02T05:07:32.585524+0000","last_active":"2024-04-02T05:08:04.394679+0000","last_peered":"2024-04-02T05:08:04.394679+0000","last_clean":"2024-04-02T05:08:04.394679+0000","last_became_active":"2024-04-02T04:59:00.104872+0000","last_became_peered":"2024-04-02T04:59:00.104872+0000","last_unstale":"2024-04-02T05:08:04.394679+0000","last_undegraded":"2024-04-02T05:08:04.394679+0000","last_fullsized":"2024-04-02T05:08:04.394679+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:32.585367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:32.585367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.155974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"899'12","reported_seq":1926,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.665870+0000","last_change":"2024-04-02T05:07:26.754432+0000","last_active":"2024-04-02T05:08:04.665870+0000","last_peered":"2024-04-02T05:08:04.665870+0000","last_clean":"2024-04-02T05:08:04.665870+0000","last_became_active":"2024-04-02T05:04:11.244389+0000","last_became_peered":"2024-04-02T05:04:11.244389+0000","last_unstale":"2024-04-02T05:08:04.665870+0000","last_undegraded":"2024-04-02T05:08:04.665870+0000","last_fullsized":"2024-04-02T05:08:04.665870+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'12","last_scrub_stamp":"2024-04-02T05:07:26.754157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:26.754157+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:33.811607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1882,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666060+0000","last_change":"2024-04-02T05:07:02.710738+0000","last_active":"2024-04-02T05:08:04.666060+0000","last_peered":"2024-04-02T05:08:04.666060+0000","last_clean":"2024-04-02T05:08:04.666060+0000","last_became_active":"2024-04-02T05:05:54.277359+0000","last_became_peered":"2024-04-02T05:05:54.277359+0000","last_unstale":"2024-04-02T05:08:04.666060+0000","last_undegraded":"2024-04-02T05:08:04.666060+0000","last_fullsized":"2024-04-02T05:08:04.666060+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:02.710526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:22.817927+0000","last_clean_scrub_stamp":"2024-04-02T05:07:02.710526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:30.066926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":706,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:11.433141+0000","last_change":"2024-04-02T05:08:11.433272+0000","last_active":"2024-04-02T05:08:11.433141+0000","last_peered":"2024-04-02T05:08:11.433141+0000","last_clean":"2024-04-02T05:08:11.433141+0000","last_became_active":"2024-04-02T04:59:00.122643+0000","last_became_peered":"2024-04-02T04:59:00.122643+0000","last_unstale":"2024-04-02T05:08:11.433141+0000","last_undegraded":"2024-04-02T05:08:11.433141+0000","last_fullsized":"2024-04-02T05:08:11.433141+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:11.432980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:46:45.921164+0000","last_clean_scrub_stamp":"2024-04-02T05:08:11.432980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:39.130127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1355,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:18.683354+0000","last_change":"2024-04-02T05:08:18.683450+0000","last_active":"2024-04-02T05:08:18.683354+0000","last_peered":"2024-04-02T05:08:18.683354+0000","last_clean":"2024-04-02T05:08:18.683354+0000","last_became_active":"2024-04-02T05:05:54.284170+0000","last_became_peered":"2024-04-02T05:05:54.284170+0000","last_unstale":"2024-04-02T05:08:18.683354+0000","last_undegraded":"2024-04-02T05:08:18.683354+0000","last_fullsized":"2024-04-02T05:08:18.683354+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:18.683159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:08:18.683159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:43.488690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"101'128","reported_seq":1714,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.981582+0000","last_change":"2024-04-02T05:07:58.122361+0000","last_active":"2024-04-02T05:08:04.981582+0000","last_peered":"2024-04-02T05:08:04.981582+0000","last_clean":"2024-04-02T05:08:04.981582+0000","last_became_active":"2024-04-02T05:04:49.251034+0000","last_became_peered":"2024-04-02T05:04:49.251034+0000","last_unstale":"2024-04-02T05:08:04.981582+0000","last_undegraded":"2024-04-02T05:08:04.981582+0000","last_fullsized":"2024-04-02T05:08:04.981582+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:58.122090+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:04:55.430781+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.122090+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:15.725450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":49,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666294+0000","last_change":"2024-04-02T05:07:34.733429+0000","last_active":"2024-04-02T05:08:04.666294+0000","last_peered":"2024-04-02T05:08:04.666294+0000","last_clean":"2024-04-02T05:08:04.666294+0000","last_became_active":"2024-04-02T05:06:31.713784+0000","last_became_peered":"2024-04-02T05:06:31.713784+0000","last_unstale":"2024-04-02T05:08:04.666294+0000","last_undegraded":"2024-04-02T05:08:04.666294+0000","last_fullsized":"2024-04-02T05:08:04.666294+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:34.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:51:39.048988+0000","last_clean_scrub_stamp":"2024-04-02T05:07:34.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.914233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"899'27","reported_seq":1358,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006193+0000","last_change":"2024-04-02T05:06:55.557040+0000","last_active":"2024-04-02T05:08:05.006193+0000","last_peered":"2024-04-02T05:08:05.006193+0000","last_clean":"2024-04-02T05:08:05.006193+0000","last_became_active":"2024-04-02T04:53:54.932304+0000","last_became_peered":"2024-04-02T04:53:54.932304+0000","last_unstale":"2024-04-02T05:08:05.006193+0000","last_undegraded":"2024-04-02T05:08:05.006193+0000","last_fullsized":"2024-04-02T05:08:05.006193+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'27","last_scrub_stamp":"2024-04-02T05:06:55.556806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:06:55.556806+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:20.928519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002964+0000","last_change":"2024-04-02T05:06:56.291063+0000","last_active":"2024-04-02T05:08:05.002964+0000","last_peered":"2024-04-02T05:08:05.002964+0000","last_clean":"2024-04-02T05:08:05.002964+0000","last_became_active":"2024-04-02T05:04:11.258328+0000","last_became_peered":"2024-04-02T05:04:11.258328+0000","last_unstale":"2024-04-02T05:08:05.002964+0000","last_undegraded":"2024-04-02T05:08:05.002964+0000","last_fullsized":"2024-04-02T05:08:05.002964+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:56.290838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:06:56.290838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:21.303130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"916'18","reported_seq":825,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.433343+0000","last_change":"2024-04-02T05:07:13.554733+0000","last_active":"2024-04-02T05:08:10.433343+0000","last_peered":"2024-04-02T05:08:10.433343+0000","last_clean":"2024-04-02T05:08:10.433343+0000","last_became_active":"2024-04-02T05:04:58.340049+0000","last_became_peered":"2024-04-02T05:04:58.340049+0000","last_unstale":"2024-04-02T05:08:10.433343+0000","last_undegraded":"2024-04-02T05:08:10.433343+0000","last_fullsized":"2024-04-02T05:08:10.433343+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"916'18","last_scrub_stamp":"2024-04-02T05:07:13.554493+0000","last_deep_scrub":"121'12","last_deep_scrub_stamp":"2024-04-02T04:48:02.448123+0000","last_clean_scrub_stamp":"2024-04-02T05:07:13.554493+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:36.245747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1730,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074198+0000","last_change":"2024-04-02T05:07:45.116251+0000","last_active":"2024-04-02T05:08:05.074198+0000","last_peered":"2024-04-02T05:08:05.074198+0000","last_clean":"2024-04-02T05:08:05.074198+0000","last_became_active":"2024-04-02T05:04:49.246021+0000","last_became_peered":"2024-04-02T05:04:49.246021+0000","last_unstale":"2024-04-02T05:08:05.074198+0000","last_undegraded":"2024-04-02T05:08:05.074198+0000","last_fullsized":"2024-04-02T05:08:05.074198+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:45.116034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:17.214556+0000","last_clean_scrub_stamp":"2024-04-02T05:07:45.116034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.999021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"101'128","reported_seq":1294,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002699+0000","last_change":"2024-04-02T05:07:03.233057+0000","last_active":"2024-04-02T05:08:05.002699+0000","last_peered":"2024-04-02T05:08:05.002699+0000","last_clean":"2024-04-02T05:08:05.002699+0000","last_became_active":"2024-04-02T05:04:14.202460+0000","last_became_peered":"2024-04-02T05:04:14.202460+0000","last_unstale":"2024-04-02T05:08:05.002699+0000","last_undegraded":"2024-04-02T05:08:05.002699+0000","last_fullsized":"2024-04-02T05:08:05.002699+0000","mapping_epoch":904,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":905,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:03.232808+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:03.232808+0000","last_clean_scrub_stamp":"2024-04-02T05:07:03.232808+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:15.699132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":724,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396419+0000","last_change":"2024-04-02T05:07:58.470081+0000","last_active":"2024-04-02T05:08:04.396419+0000","last_peered":"2024-04-02T05:08:04.396419+0000","last_clean":"2024-04-02T05:08:04.396419+0000","last_became_active":"2024-04-02T05:05:54.274382+0000","last_became_peered":"2024-04-02T05:05:54.274382+0000","last_unstale":"2024-04-02T05:08:04.396419+0000","last_undegraded":"2024-04-02T05:08:04.396419+0000","last_fullsized":"2024-04-02T05:08:04.396419+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:58.469874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:58.469874+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.469874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:09.283378+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"899'21","reported_seq":1000,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.982108+0000","last_change":"2024-04-02T05:07:41.043760+0000","last_active":"2024-04-02T05:08:04.982108+0000","last_peered":"2024-04-02T05:08:04.982108+0000","last_clean":"2024-04-02T05:08:04.982108+0000","last_became_active":"2024-04-02T05:04:49.244686+0000","last_became_peered":"2024-04-02T05:04:49.244686+0000","last_unstale":"2024-04-02T05:08:04.982108+0000","last_undegraded":"2024-04-02T05:08:04.982108+0000","last_fullsized":"2024-04-02T05:08:04.982108+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'21","last_scrub_stamp":"2024-04-02T05:07:41.043524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:41.043524+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.025183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1742,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.982403+0000","last_change":"2024-04-02T05:08:03.014352+0000","last_active":"2024-04-02T05:08:04.982403+0000","last_peered":"2024-04-02T05:08:04.982403+0000","last_clean":"2024-04-02T05:08:04.982403+0000","last_became_active":"2024-04-02T05:04:49.231489+0000","last_became_peered":"2024-04-02T05:04:49.231489+0000","last_unstale":"2024-04-02T05:08:04.982403+0000","last_undegraded":"2024-04-02T05:08:04.982403+0000","last_fullsized":"2024-04-02T05:08:04.982403+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:03.014106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:54.894248+0000","last_clean_scrub_stamp":"2024-04-02T05:08:03.014106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:05.384593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"961'20","reported_seq":245,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.434362+0000","last_change":"2024-04-02T05:07:02.262832+0000","last_active":"2024-04-02T05:08:10.434362+0000","last_peered":"2024-04-02T05:08:10.434362+0000","last_clean":"2024-04-02T05:08:10.434362+0000","last_became_active":"2024-04-02T05:06:58.350725+0000","last_became_peered":"2024-04-02T05:06:58.350725+0000","last_unstale":"2024-04-02T05:08:10.434362+0000","last_undegraded":"2024-04-02T05:08:10.434362+0000","last_fullsized":"2024-04-02T05:08:10.434362+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":0,"last_scrub":"961'20","last_scrub_stamp":"2024-04-02T05:07:02.262604+0000","last_deep_scrub":"937'19","last_deep_scrub_stamp":"2024-04-02T05:05:46.936756+0000","last_clean_scrub_stamp":"2024-04-02T05:07:02.262604+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:31.096682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1877,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399144+0000","last_change":"2024-04-02T05:07:22.480713+0000","last_active":"2024-04-02T05:08:04.399144+0000","last_peered":"2024-04-02T05:08:04.399144+0000","last_clean":"2024-04-02T05:08:04.399144+0000","last_became_active":"2024-04-02T05:04:11.238138+0000","last_became_peered":"2024-04-02T05:04:11.238138+0000","last_unstale":"2024-04-02T05:08:04.399144+0000","last_undegraded":"2024-04-02T05:08:04.399144+0000","last_fullsized":"2024-04-02T05:08:04.399144+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:22.480435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:14.397839+0000","last_clean_scrub_stamp":"2024-04-02T05:07:22.480435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:28.511395+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.666877+0000","last_change":"2024-04-02T05:06:57.710439+0000","last_active":"2024-04-02T05:08:04.666877+0000","last_peered":"2024-04-02T05:08:04.666877+0000","last_clean":"2024-04-02T05:08:04.666877+0000","last_became_active":"2024-04-02T04:49:26.264445+0000","last_became_peered":"2024-04-02T04:49:26.264445+0000","last_unstale":"2024-04-02T05:08:04.666877+0000","last_undegraded":"2024-04-02T05:08:04.666877+0000","last_fullsized":"2024-04-02T05:08:04.666877+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:57.710144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:31.233061+0000","last_clean_scrub_stamp":"2024-04-02T05:06:57.710144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:22.891060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"930'32","reported_seq":1440,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.007283+0000","last_change":"2024-04-02T05:07:58.530087+0000","last_active":"2024-04-02T05:08:05.007283+0000","last_peered":"2024-04-02T05:08:05.007283+0000","last_clean":"2024-04-02T05:08:05.007283+0000","last_became_active":"2024-04-02T04:59:00.137738+0000","last_became_peered":"2024-04-02T04:59:00.137738+0000","last_unstale":"2024-04-02T05:08:05.007283+0000","last_undegraded":"2024-04-02T05:08:05.007283+0000","last_fullsized":"2024-04-02T05:08:05.007283+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'32","last_scrub_stamp":"2024-04-02T05:07:58.529819+0000","last_deep_scrub":"930'32","last_deep_scrub_stamp":"2024-04-02T05:05:29.330313+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.529819+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:25.425552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1880,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667803+0000","last_change":"2024-04-02T05:07:03.694867+0000","last_active":"2024-04-02T05:08:04.667803+0000","last_peered":"2024-04-02T05:08:04.667803+0000","last_clean":"2024-04-02T05:08:04.667803+0000","last_became_active":"2024-04-02T05:04:11.257466+0000","last_became_peered":"2024-04-02T05:04:11.257466+0000","last_unstale":"2024-04-02T05:08:04.667803+0000","last_undegraded":"2024-04-02T05:08:04.667803+0000","last_fullsized":"2024-04-02T05:08:04.667803+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:03.694645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:03.694645+0000","last_clean_scrub_stamp":"2024-04-02T05:07:03.694645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:31.460633+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":132,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399239+0000","last_change":"2024-04-02T05:07:27.757587+0000","last_active":"2024-04-02T05:08:04.399239+0000","last_peered":"2024-04-02T05:08:04.399239+0000","last_clean":"2024-04-02T05:08:04.399239+0000","last_became_active":"2024-04-02T05:04:58.335973+0000","last_became_peered":"2024-04-02T05:04:58.335973+0000","last_unstale":"2024-04-02T05:08:04.399239+0000","last_undegraded":"2024-04-02T05:08:04.399239+0000","last_fullsized":"2024-04-02T05:08:04.399239+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:27.757340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:27.757340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:44.793115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:06.980882+0000","last_change":"2024-04-02T05:08:06.980944+0000","last_active":"2024-04-02T05:08:06.980882+0000","last_peered":"2024-04-02T05:08:06.980882+0000","last_clean":"2024-04-02T05:08:06.980882+0000","last_became_active":"2024-04-02T05:04:11.228488+0000","last_became_peered":"2024-04-02T05:04:11.228488+0000","last_unstale":"2024-04-02T05:08:06.980882+0000","last_undegraded":"2024-04-02T05:08:06.980882+0000","last_fullsized":"2024-04-02T05:08:06.980882+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:06.980719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:43.063652+0000","last_clean_scrub_stamp":"2024-04-02T05:08:06.980719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:18.002283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1733,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399364+0000","last_change":"2024-04-02T05:06:54.437684+0000","last_active":"2024-04-02T05:08:04.399364+0000","last_peered":"2024-04-02T05:08:04.399364+0000","last_clean":"2024-04-02T05:08:04.399364+0000","last_became_active":"2024-04-02T04:49:26.262492+0000","last_became_peered":"2024-04-02T04:49:26.262492+0000","last_unstale":"2024-04-02T05:08:04.399364+0000","last_undegraded":"2024-04-02T05:08:04.399364+0000","last_fullsized":"2024-04-02T05:08:04.399364+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:54.437476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:06.838854+0000","last_clean_scrub_stamp":"2024-04-02T05:06:54.437476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:09.575894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"899'42","reported_seq":98,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396906+0000","last_change":"2024-04-02T05:07:32.601500+0000","last_active":"2024-04-02T05:08:04.396906+0000","last_peered":"2024-04-02T05:08:04.396906+0000","last_clean":"2024-04-02T05:08:04.396906+0000","last_became_active":"2024-04-02T05:05:54.282114+0000","last_became_peered":"2024-04-02T05:05:54.282114+0000","last_unstale":"2024-04-02T05:08:04.396906+0000","last_undegraded":"2024-04-02T05:08:04.396906+0000","last_fullsized":"2024-04-02T05:08:04.396906+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'42","last_scrub_stamp":"2024-04-02T05:07:32.601274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:32.601274+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:49.240468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1889,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400913+0000","last_change":"2024-04-02T05:07:48.661178+0000","last_active":"2024-04-02T05:08:04.400913+0000","last_peered":"2024-04-02T05:08:04.400913+0000","last_clean":"2024-04-02T05:08:04.400913+0000","last_became_active":"2024-04-02T05:04:11.236464+0000","last_became_peered":"2024-04-02T05:04:11.236464+0000","last_unstale":"2024-04-02T05:08:04.400913+0000","last_undegraded":"2024-04-02T05:08:04.400913+0000","last_fullsized":"2024-04-02T05:08:04.400913+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:48.661017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:03:51.374778+0000","last_clean_scrub_stamp":"2024-04-02T05:07:48.661017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:03.428976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1877,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396959+0000","last_change":"2024-04-02T05:07:24.462014+0000","last_active":"2024-04-02T05:08:04.396959+0000","last_peered":"2024-04-02T05:08:04.396959+0000","last_clean":"2024-04-02T05:08:04.396959+0000","last_became_active":"2024-04-02T04:43:58.296937+0000","last_became_peered":"2024-04-02T04:43:58.296937+0000","last_unstale":"2024-04-02T05:08:04.396959+0000","last_undegraded":"2024-04-02T05:08:04.396959+0000","last_fullsized":"2024-04-02T05:08:04.396959+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:24.461805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:56:40.845504+0000","last_clean_scrub_stamp":"2024-04-02T05:07:24.461805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:50.752216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":718,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399977+0000","last_change":"2024-04-02T05:07:49.591776+0000","last_active":"2024-04-02T05:08:04.399977+0000","last_peered":"2024-04-02T05:08:04.399977+0000","last_clean":"2024-04-02T05:08:04.399977+0000","last_became_active":"2024-04-02T05:04:11.244750+0000","last_became_peered":"2024-04-02T05:04:11.244750+0000","last_unstale":"2024-04-02T05:08:04.399977+0000","last_undegraded":"2024-04-02T05:08:04.399977+0000","last_fullsized":"2024-04-02T05:08:04.399977+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:49.591591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:32.383421+0000","last_clean_scrub_stamp":"2024-04-02T05:07:49.591591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:00.965159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1738,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073633+0000","last_change":"2024-04-02T05:07:02.972201+0000","last_active":"2024-04-02T05:08:05.073633+0000","last_peered":"2024-04-02T05:08:05.073633+0000","last_clean":"2024-04-02T05:08:05.073633+0000","last_became_active":"2024-04-02T05:04:11.228518+0000","last_became_peered":"2024-04-02T05:04:11.228518+0000","last_unstale":"2024-04-02T05:08:05.073633+0000","last_undegraded":"2024-04-02T05:08:05.073633+0000","last_fullsized":"2024-04-02T05:08:05.073633+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:02.971975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:30.067845+0000","last_clean_scrub_stamp":"2024-04-02T05:07:02.971975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:22.016275+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"959'28","reported_seq":1545,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074256+0000","last_change":"2024-04-02T05:08:04.028230+0000","last_active":"2024-04-02T05:08:05.074256+0000","last_peered":"2024-04-02T05:08:05.074256+0000","last_clean":"2024-04-02T05:08:05.074256+0000","last_became_active":"2024-04-02T05:04:49.241105+0000","last_became_peered":"2024-04-02T05:04:49.241105+0000","last_unstale":"2024-04-02T05:08:05.074256+0000","last_undegraded":"2024-04-02T05:08:05.074256+0000","last_fullsized":"2024-04-02T05:08:05.074256+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'28","last_scrub_stamp":"2024-04-02T05:08:04.028138+0000","last_deep_scrub":"959'28","last_deep_scrub_stamp":"2024-04-02T05:06:55.902636+0000","last_clean_scrub_stamp":"2024-04-02T05:08:04.028138+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:27.781946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":174,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003875+0000","last_change":"2024-04-02T05:07:16.208642+0000","last_active":"2024-04-02T05:08:05.003875+0000","last_peered":"2024-04-02T05:08:05.003875+0000","last_clean":"2024-04-02T05:08:05.003875+0000","last_became_active":"2024-04-02T05:05:16.546222+0000","last_became_peered":"2024-04-02T05:05:16.546222+0000","last_unstale":"2024-04-02T05:08:05.003875+0000","last_undegraded":"2024-04-02T05:08:05.003875+0000","last_fullsized":"2024-04-02T05:08:05.003875+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.208414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:27.895598+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.208414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.128614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":706,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:17.534277+0000","last_change":"2024-04-02T05:08:17.534332+0000","last_active":"2024-04-02T05:08:17.534277+0000","last_peered":"2024-04-02T05:08:17.534277+0000","last_clean":"2024-04-02T05:08:17.534277+0000","last_became_active":"2024-04-02T04:59:00.102092+0000","last_became_peered":"2024-04-02T04:59:00.102092+0000","last_unstale":"2024-04-02T05:08:17.534277+0000","last_undegraded":"2024-04-02T05:08:17.534277+0000","last_fullsized":"2024-04-02T05:08:17.534277+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:17.534117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:00:12.801839+0000","last_clean_scrub_stamp":"2024-04-02T05:08:17.534117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:36.300472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1332,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:08.534867+0000","last_change":"2024-04-02T05:08:08.534924+0000","last_active":"2024-04-02T05:08:08.534867+0000","last_peered":"2024-04-02T05:08:08.534867+0000","last_clean":"2024-04-02T05:08:08.534867+0000","last_became_active":"2024-04-02T04:53:54.908510+0000","last_became_peered":"2024-04-02T04:53:54.908510+0000","last_unstale":"2024-04-02T05:08:08.534867+0000","last_undegraded":"2024-04-02T05:08:08.534867+0000","last_fullsized":"2024-04-02T05:08:08.534867+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:08.534714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:53.375671+0000","last_clean_scrub_stamp":"2024-04-02T05:08:08.534714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:24.671108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"101'128","reported_seq":1333,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.872970+0000","last_change":"2024-04-02T05:07:31.985648+0000","last_active":"2024-04-02T05:08:04.872970+0000","last_peered":"2024-04-02T05:08:04.872970+0000","last_clean":"2024-04-02T05:08:04.872970+0000","last_became_active":"2024-04-02T05:05:57.337502+0000","last_became_peered":"2024-04-02T05:05:57.337502+0000","last_unstale":"2024-04-02T05:08:04.872970+0000","last_undegraded":"2024-04-02T05:08:04.872970+0000","last_fullsized":"2024-04-02T05:08:04.872970+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'128","last_scrub_stamp":"2024-04-02T05:07:31.985497+0000","last_deep_scrub":"101'128","last_deep_scrub_stamp":"2024-04-02T05:07:31.985497+0000","last_clean_scrub_stamp":"2024-04-02T05:07:31.985497+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:38.027767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1223,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:15.743094+0000","last_change":"2024-04-02T05:08:15.743154+0000","last_active":"2024-04-02T05:08:15.743094+0000","last_peered":"2024-04-02T05:08:15.743094+0000","last_clean":"2024-04-02T05:08:15.743094+0000","last_became_active":"2024-04-02T05:05:16.544892+0000","last_became_peered":"2024-04-02T05:05:16.544892+0000","last_unstale":"2024-04-02T05:08:15.743094+0000","last_undegraded":"2024-04-02T05:08:15.743094+0000","last_fullsized":"2024-04-02T05:08:15.743094+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:15.742936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:15.742936+0000","last_clean_scrub_stamp":"2024-04-02T05:08:15.742936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:21.715446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"905'36","reported_seq":185,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.003549+0000","last_change":"2024-04-02T05:07:21.201775+0000","last_active":"2024-04-02T05:08:05.003549+0000","last_peered":"2024-04-02T05:08:05.003549+0000","last_clean":"2024-04-02T05:08:05.003549+0000","last_became_active":"2024-04-02T05:04:11.251941+0000","last_became_peered":"2024-04-02T05:04:11.251941+0000","last_unstale":"2024-04-02T05:08:05.003549+0000","last_undegraded":"2024-04-02T05:08:05.003549+0000","last_fullsized":"2024-04-02T05:08:05.003549+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'36","last_scrub_stamp":"2024-04-02T05:07:21.201423+0000","last_deep_scrub":"905'36","last_deep_scrub_stamp":"2024-04-02T05:04:15.805570+0000","last_clean_scrub_stamp":"2024-04-02T05:07:21.201423+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:24.087735+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1896,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.668437+0000","last_change":"2024-04-02T05:07:42.689166+0000","last_active":"2024-04-02T05:08:04.668437+0000","last_peered":"2024-04-02T05:08:04.668437+0000","last_clean":"2024-04-02T05:08:04.668437+0000","last_became_active":"2024-04-02T05:05:16.546077+0000","last_became_peered":"2024-04-02T05:05:16.546077+0000","last_unstale":"2024-04-02T05:08:04.668437+0000","last_undegraded":"2024-04-02T05:08:04.668437+0000","last_fullsized":"2024-04-02T05:08:04.668437+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:42.688963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:46.514376+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.688963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:51.992363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":142,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:17.025769+0000","last_change":"2024-04-02T05:08:17.025826+0000","last_active":"2024-04-02T05:08:17.025769+0000","last_peered":"2024-04-02T05:08:17.025769+0000","last_clean":"2024-04-02T05:08:17.025769+0000","last_became_active":"2024-04-02T05:04:57.347965+0000","last_became_peered":"2024-04-02T05:04:57.347965+0000","last_unstale":"2024-04-02T05:08:17.025769+0000","last_undegraded":"2024-04-02T05:08:17.025769+0000","last_fullsized":"2024-04-02T05:08:17.025769+0000","mapping_epoch":914,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":915,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:17.025605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:06:58.250588+0000","last_clean_scrub_stamp":"2024-04-02T05:08:17.025605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:28.632646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":722,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398250+0000","last_change":"2024-04-02T05:07:00.366446+0000","last_active":"2024-04-02T05:08:04.398250+0000","last_peered":"2024-04-02T05:08:04.398250+0000","last_clean":"2024-04-02T05:08:04.398250+0000","last_became_active":"2024-04-02T05:06:31.702170+0000","last_became_peered":"2024-04-02T05:06:31.702170+0000","last_unstale":"2024-04-02T05:08:04.398250+0000","last_undegraded":"2024-04-02T05:08:04.398250+0000","last_fullsized":"2024-04-02T05:08:04.398250+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:00.366335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:30.933150+0000","last_clean_scrub_stamp":"2024-04-02T05:07:00.366335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:27.483325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":82,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395686+0000","last_change":"2024-04-02T05:07:36.602643+0000","last_active":"2024-04-02T05:08:04.395686+0000","last_peered":"2024-04-02T05:08:04.395686+0000","last_clean":"2024-04-02T05:08:04.395686+0000","last_became_active":"2024-04-02T05:05:54.288794+0000","last_became_peered":"2024-04-02T05:05:54.288794+0000","last_unstale":"2024-04-02T05:08:04.395686+0000","last_undegraded":"2024-04-02T05:08:04.395686+0000","last_fullsized":"2024-04-02T05:08:04.395686+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:36.602412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:36.602412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:02.228203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"899'74","reported_seq":2105,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395230+0000","last_change":"2024-04-02T05:07:19.776641+0000","last_active":"2024-04-02T05:08:04.395230+0000","last_peered":"2024-04-02T05:08:04.395230+0000","last_clean":"2024-04-02T05:08:04.395230+0000","last_became_active":"2024-04-02T05:05:54.287852+0000","last_became_peered":"2024-04-02T05:05:54.287852+0000","last_unstale":"2024-04-02T05:08:04.395230+0000","last_undegraded":"2024-04-02T05:08:04.395230+0000","last_fullsized":"2024-04-02T05:08:04.395230+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'74","last_scrub_stamp":"2024-04-02T05:07:19.776365+0000","last_deep_scrub":"133'50","last_deep_scrub_stamp":"2024-04-02T04:49:02.986701+0000","last_clean_scrub_stamp":"2024-04-02T05:07:19.776365+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:39.952209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.002401+0000","last_change":"2024-04-02T05:07:28.183493+0000","last_active":"2024-04-02T05:08:05.002401+0000","last_peered":"2024-04-02T05:08:05.002401+0000","last_clean":"2024-04-02T05:08:05.002401+0000","last_became_active":"2024-04-02T05:04:11.252720+0000","last_became_peered":"2024-04-02T05:04:11.252720+0000","last_unstale":"2024-04-02T05:08:05.002401+0000","last_undegraded":"2024-04-02T05:08:05.002401+0000","last_fullsized":"2024-04-02T05:08:05.002401+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:28.183264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:28.183264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.041853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1883,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.073883+0000","last_change":"2024-04-02T05:07:44.142715+0000","last_active":"2024-04-02T05:08:05.073883+0000","last_peered":"2024-04-02T05:08:05.073883+0000","last_clean":"2024-04-02T05:08:05.073883+0000","last_became_active":"2024-04-02T04:59:00.138125+0000","last_became_peered":"2024-04-02T04:59:00.138125+0000","last_unstale":"2024-04-02T05:08:05.073883+0000","last_undegraded":"2024-04-02T05:08:05.073883+0000","last_fullsized":"2024-04-02T05:08:05.073883+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:44.142492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:44.142492+0000","last_clean_scrub_stamp":"2024-04-02T05:07:44.142492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:05.673220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"958'2","reported_seq":1373,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:09.530285+0000","last_change":"2024-04-02T05:08:09.530356+0000","last_active":"2024-04-02T05:08:09.530285+0000","last_peered":"2024-04-02T05:08:09.530285+0000","last_clean":"2024-04-02T05:08:09.530285+0000","last_became_active":"2024-04-02T05:04:11.237558+0000","last_became_peered":"2024-04-02T05:04:11.237558+0000","last_unstale":"2024-04-02T05:08:09.530285+0000","last_undegraded":"2024-04-02T05:08:09.530285+0000","last_fullsized":"2024-04-02T05:08:09.530285+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'2","last_scrub_stamp":"2024-04-02T05:08:09.530129+0000","last_deep_scrub":"958'2","last_deep_scrub_stamp":"2024-04-02T05:06:48.497980+0000","last_clean_scrub_stamp":"2024-04-02T05:08:09.530129+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:14.123788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004379+0000","last_change":"2024-04-02T05:07:23.226101+0000","last_active":"2024-04-02T05:08:05.004379+0000","last_peered":"2024-04-02T05:08:05.004379+0000","last_clean":"2024-04-02T05:08:05.004379+0000","last_became_active":"2024-04-02T05:04:11.244814+0000","last_became_peered":"2024-04-02T05:04:11.244814+0000","last_unstale":"2024-04-02T05:08:05.004379+0000","last_undegraded":"2024-04-02T05:08:05.004379+0000","last_fullsized":"2024-04-02T05:08:05.004379+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:23.225832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:04:39.087140+0000","last_clean_scrub_stamp":"2024-04-02T05:07:23.225832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:48.274991+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"957'32","reported_seq":732,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:08.374405+0000","last_change":"2024-04-02T05:08:08.374537+0000","last_active":"2024-04-02T05:08:08.374405+0000","last_peered":"2024-04-02T05:08:08.374405+0000","last_clean":"2024-04-02T05:08:08.374405+0000","last_became_active":"2024-04-02T04:59:00.133276+0000","last_became_peered":"2024-04-02T04:59:00.133276+0000","last_unstale":"2024-04-02T05:08:08.374405+0000","last_undegraded":"2024-04-02T05:08:08.374405+0000","last_fullsized":"2024-04-02T05:08:08.374405+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'32","last_scrub_stamp":"2024-04-02T05:08:08.374201+0000","last_deep_scrub":"957'32","last_deep_scrub_stamp":"2024-04-02T05:06:40.469957+0000","last_clean_scrub_stamp":"2024-04-02T05:08:08.374201+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:20.376374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"105'2","reported_seq":710,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400234+0000","last_change":"2024-04-02T05:07:34.628774+0000","last_active":"2024-04-02T05:08:04.400234+0000","last_peered":"2024-04-02T05:08:04.400234+0000","last_clean":"2024-04-02T05:08:04.400234+0000","last_became_active":"2024-04-02T04:59:00.135857+0000","last_became_peered":"2024-04-02T04:59:00.135857+0000","last_unstale":"2024-04-02T05:08:04.400234+0000","last_undegraded":"2024-04-02T05:08:04.400234+0000","last_fullsized":"2024-04-02T05:08:04.400234+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-04-02T05:07:34.628682+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-04-02T04:45:22.711771+0000","last_clean_scrub_stamp":"2024-04-02T05:07:34.628682+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:54.427606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"902'13","reported_seq":2183,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.434362+0000","last_change":"2024-04-02T05:07:31.758360+0000","last_active":"2024-04-02T05:08:10.434362+0000","last_peered":"2024-04-02T05:08:10.434362+0000","last_clean":"2024-04-02T05:08:10.434362+0000","last_became_active":"2024-04-02T05:04:11.260020+0000","last_became_peered":"2024-04-02T05:04:11.260020+0000","last_unstale":"2024-04-02T05:08:10.434362+0000","last_undegraded":"2024-04-02T05:08:10.434362+0000","last_fullsized":"2024-04-02T05:08:10.434362+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'13","last_scrub_stamp":"2024-04-02T05:07:31.758131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:31.758131+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.441886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1880,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.401871+0000","last_change":"2024-04-02T05:07:16.856612+0000","last_active":"2024-04-02T05:08:04.401871+0000","last_peered":"2024-04-02T05:08:04.401871+0000","last_clean":"2024-04-02T05:08:04.401871+0000","last_became_active":"2024-04-02T05:05:54.275152+0000","last_became_peered":"2024-04-02T05:05:54.275152+0000","last_unstale":"2024-04-02T05:08:04.401871+0000","last_undegraded":"2024-04-02T05:08:04.401871+0000","last_fullsized":"2024-04-02T05:08:04.401871+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.856401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:20.038583+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.856401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:43.774861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1744,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667117+0000","last_change":"2024-04-02T05:07:58.937172+0000","last_active":"2024-04-02T05:08:04.667117+0000","last_peered":"2024-04-02T05:08:04.667117+0000","last_clean":"2024-04-02T05:08:04.667117+0000","last_became_active":"2024-04-02T05:05:16.544183+0000","last_became_peered":"2024-04-02T05:05:16.544183+0000","last_unstale":"2024-04-02T05:08:04.667117+0000","last_undegraded":"2024-04-02T05:08:04.667117+0000","last_fullsized":"2024-04-02T05:08:04.667117+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:58.936944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:20.453979+0000","last_clean_scrub_stamp":"2024-04-02T05:07:58.936944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:01.018261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"899'18","reported_seq":1895,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.072966+0000","last_change":"2024-04-02T05:07:06.997018+0000","last_active":"2024-04-02T05:08:05.072966+0000","last_peered":"2024-04-02T05:08:05.072966+0000","last_clean":"2024-04-02T05:08:05.072966+0000","last_became_active":"2024-04-02T04:53:54.940527+0000","last_became_peered":"2024-04-02T04:53:54.940527+0000","last_unstale":"2024-04-02T05:08:05.072966+0000","last_undegraded":"2024-04-02T05:08:05.072966+0000","last_fullsized":"2024-04-02T05:08:05.072966+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'18","last_scrub_stamp":"2024-04-02T05:07:06.996892+0000","last_deep_scrub":"899'18","last_deep_scrub_stamp":"2024-04-02T05:05:22.354664+0000","last_clean_scrub_stamp":"2024-04-02T05:07:06.996892+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:13.189718+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667426+0000","last_change":"2024-04-02T05:07:53.940505+0000","last_active":"2024-04-02T05:08:04.667426+0000","last_peered":"2024-04-02T05:08:04.667426+0000","last_clean":"2024-04-02T05:08:04.667426+0000","last_became_active":"2024-04-02T04:53:54.903279+0000","last_became_peered":"2024-04-02T04:53:54.903279+0000","last_unstale":"2024-04-02T05:08:04.667426+0000","last_undegraded":"2024-04-02T05:08:04.667426+0000","last_fullsized":"2024-04-02T05:08:04.667426+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:53.940291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:45:05.863867+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.940291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:11.453458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":118,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.871122+0000","last_change":"2024-04-02T05:06:54.680012+0000","last_active":"2024-04-02T05:08:04.871122+0000","last_peered":"2024-04-02T05:08:04.871122+0000","last_clean":"2024-04-02T05:08:04.871122+0000","last_became_active":"2024-04-02T05:05:16.542585+0000","last_became_peered":"2024-04-02T05:05:16.542585+0000","last_unstale":"2024-04-02T05:08:04.871122+0000","last_undegraded":"2024-04-02T05:08:04.871122+0000","last_fullsized":"2024-04-02T05:08:04.871122+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:54.679771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:06:54.679771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:20.134874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":74,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.398737+0000","last_change":"2024-04-02T05:07:20.508239+0000","last_active":"2024-04-02T05:08:04.398737+0000","last_peered":"2024-04-02T05:08:04.398737+0000","last_clean":"2024-04-02T05:08:04.398737+0000","last_became_active":"2024-04-02T05:05:54.282177+0000","last_became_peered":"2024-04-02T05:05:54.282177+0000","last_unstale":"2024-04-02T05:08:04.398737+0000","last_undegraded":"2024-04-02T05:08:04.398737+0000","last_fullsized":"2024-04-02T05:08:04.398737+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:20.507993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:20.507993+0000","last_clean_scrub_stamp":"2024-04-02T05:07:20.507993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:21.875519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1747,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:16.587217+0000","last_change":"2024-04-02T05:08:16.587276+0000","last_active":"2024-04-02T05:08:16.587217+0000","last_peered":"2024-04-02T05:08:16.587217+0000","last_clean":"2024-04-02T05:08:16.587217+0000","last_became_active":"2024-04-02T05:05:16.540588+0000","last_became_peered":"2024-04-02T05:05:16.540588+0000","last_unstale":"2024-04-02T05:08:16.587217+0000","last_undegraded":"2024-04-02T05:08:16.587217+0000","last_fullsized":"2024-04-02T05:08:16.587217+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:16.587035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:08:16.587035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:34.209505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"899'41","reported_seq":1986,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.397169+0000","last_change":"2024-04-02T05:07:42.612118+0000","last_active":"2024-04-02T05:08:04.397169+0000","last_peered":"2024-04-02T05:08:04.397169+0000","last_clean":"2024-04-02T05:08:04.397169+0000","last_became_active":"2024-04-02T05:05:16.536332+0000","last_became_peered":"2024-04-02T05:05:16.536332+0000","last_unstale":"2024-04-02T05:08:04.397169+0000","last_undegraded":"2024-04-02T05:08:04.397169+0000","last_fullsized":"2024-04-02T05:08:04.397169+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'41","last_scrub_stamp":"2024-04-02T05:07:42.611837+0000","last_deep_scrub":"899'41","last_deep_scrub_stamp":"2024-04-02T05:06:18.573690+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.611837+0000","objects_scrubbed":6,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:11.875159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396939+0000","last_change":"2024-04-02T05:07:17.678571+0000","last_active":"2024-04-02T05:08:04.396939+0000","last_peered":"2024-04-02T05:08:04.396939+0000","last_clean":"2024-04-02T05:08:04.396939+0000","last_became_active":"2024-04-02T04:59:00.122388+0000","last_became_peered":"2024-04-02T04:59:00.122388+0000","last_unstale":"2024-04-02T05:08:04.396939+0000","last_undegraded":"2024-04-02T05:08:04.396939+0000","last_fullsized":"2024-04-02T05:08:04.396939+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:17.678355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:17.678355+0000","last_clean_scrub_stamp":"2024-04-02T05:07:17.678355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:26.045987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399150+0000","last_change":"2024-04-02T05:07:23.532355+0000","last_active":"2024-04-02T05:08:04.399150+0000","last_peered":"2024-04-02T05:08:04.399150+0000","last_clean":"2024-04-02T05:08:04.399150+0000","last_became_active":"2024-04-02T04:59:00.131269+0000","last_became_peered":"2024-04-02T04:59:00.131269+0000","last_unstale":"2024-04-02T05:08:04.399150+0000","last_undegraded":"2024-04-02T05:08:04.399150+0000","last_fullsized":"2024-04-02T05:08:04.399150+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:23.532250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:07:23.532250+0000","last_clean_scrub_stamp":"2024-04-02T05:07:23.532250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:30.212439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":719,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.403477+0000","last_change":"2024-04-02T05:07:44.609515+0000","last_active":"2024-04-02T05:08:04.403477+0000","last_peered":"2024-04-02T05:08:04.403477+0000","last_clean":"2024-04-02T05:08:04.403477+0000","last_became_active":"2024-04-02T05:04:11.244470+0000","last_became_peered":"2024-04-02T05:04:11.244470+0000","last_unstale":"2024-04-02T05:08:04.403477+0000","last_undegraded":"2024-04-02T05:08:04.403477+0000","last_fullsized":"2024-04-02T05:08:04.403477+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:44.609296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:00.807998+0000","last_clean_scrub_stamp":"2024-04-02T05:07:44.609296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:58.947311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1746,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.707518+0000","last_change":"2024-04-02T05:08:05.707578+0000","last_active":"2024-04-02T05:08:05.707518+0000","last_peered":"2024-04-02T05:08:05.707518+0000","last_clean":"2024-04-02T05:08:05.707518+0000","last_became_active":"2024-04-02T05:05:54.280088+0000","last_became_peered":"2024-04-02T05:05:54.280088+0000","last_unstale":"2024-04-02T05:08:05.707518+0000","last_undegraded":"2024-04-02T05:08:05.707518+0000","last_fullsized":"2024-04-02T05:08:05.707518+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:05.707349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:50:39.321605+0000","last_clean_scrub_stamp":"2024-04-02T05:08:05.707349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:31.428024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"899'30","reported_seq":182,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005137+0000","last_change":"2024-04-02T05:07:35.119158+0000","last_active":"2024-04-02T05:08:05.005137+0000","last_peered":"2024-04-02T05:08:05.005137+0000","last_clean":"2024-04-02T05:08:05.005137+0000","last_became_active":"2024-04-02T05:04:11.242466+0000","last_became_peered":"2024-04-02T05:04:11.242466+0000","last_unstale":"2024-04-02T05:08:05.005137+0000","last_undegraded":"2024-04-02T05:08:05.005137+0000","last_fullsized":"2024-04-02T05:08:05.005137+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'30","last_scrub_stamp":"2024-04-02T05:07:35.118899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:35.118899+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:03.285725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"920'2","reported_seq":1375,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006434+0000","last_change":"2024-04-02T05:08:00.541852+0000","last_active":"2024-04-02T05:08:05.006434+0000","last_peered":"2024-04-02T05:08:05.006434+0000","last_clean":"2024-04-02T05:08:05.006434+0000","last_became_active":"2024-04-02T05:04:11.238352+0000","last_became_peered":"2024-04-02T05:04:11.238352+0000","last_unstale":"2024-04-02T05:08:05.006434+0000","last_undegraded":"2024-04-02T05:08:05.006434+0000","last_fullsized":"2024-04-02T05:08:05.006434+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'2","last_scrub_stamp":"2024-04-02T05:08:00.541348+0000","last_deep_scrub":"920'2","last_deep_scrub_stamp":"2024-04-02T05:05:09.467910+0000","last_clean_scrub_stamp":"2024-04-02T05:08:00.541348+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:29.978854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":81,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:07.668827+0000","last_change":"2024-04-02T05:08:07.668886+0000","last_active":"2024-04-02T05:08:07.668827+0000","last_peered":"2024-04-02T05:08:07.668827+0000","last_clean":"2024-04-02T05:08:07.668827+0000","last_became_active":"2024-04-02T05:05:54.277955+0000","last_became_peered":"2024-04-02T05:05:54.277955+0000","last_unstale":"2024-04-02T05:08:07.668827+0000","last_undegraded":"2024-04-02T05:08:07.668827+0000","last_fullsized":"2024-04-02T05:08:07.668827+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:07.668668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:08:07.668668+0000","last_clean_scrub_stamp":"2024-04-02T05:08:07.668668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:21.874085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1857,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.981553+0000","last_change":"2024-04-02T05:07:30.024340+0000","last_active":"2024-04-02T05:08:04.981553+0000","last_peered":"2024-04-02T05:08:04.981553+0000","last_clean":"2024-04-02T05:08:04.981553+0000","last_became_active":"2024-04-02T04:53:54.957186+0000","last_became_peered":"2024-04-02T04:53:54.957186+0000","last_unstale":"2024-04-02T05:08:04.981553+0000","last_undegraded":"2024-04-02T05:08:04.981553+0000","last_fullsized":"2024-04-02T05:08:04.981553+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:30.024251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:02:25.384498+0000","last_clean_scrub_stamp":"2024-04-02T05:07:30.024251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:53.695979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":712,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:10.453207+0000","last_change":"2024-04-02T05:08:10.453264+0000","last_active":"2024-04-02T05:08:10.453207+0000","last_peered":"2024-04-02T05:08:10.453207+0000","last_clean":"2024-04-02T05:08:10.453207+0000","last_became_active":"2024-04-02T05:04:11.244083+0000","last_became_peered":"2024-04-02T05:04:11.244083+0000","last_unstale":"2024-04-02T05:08:10.453207+0000","last_undegraded":"2024-04-02T05:08:10.453207+0000","last_fullsized":"2024-04-02T05:08:10.453207+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:08:10.453016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:34.065387+0000","last_clean_scrub_stamp":"2024-04-02T05:08:10.453016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:19.411666+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"899'39","reported_seq":184,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.004314+0000","last_change":"2024-04-02T05:07:56.246390+0000","last_active":"2024-04-02T05:08:05.004314+0000","last_peered":"2024-04-02T05:08:05.004314+0000","last_clean":"2024-04-02T05:08:05.004314+0000","last_became_active":"2024-04-02T05:04:11.243362+0000","last_became_peered":"2024-04-02T05:04:11.243362+0000","last_unstale":"2024-04-02T05:08:05.004314+0000","last_undegraded":"2024-04-02T05:08:05.004314+0000","last_fullsized":"2024-04-02T05:08:05.004314+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'39","last_scrub_stamp":"2024-04-02T05:07:56.246150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:07:56.246150+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:07.331098+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1349,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.006439+0000","last_change":"2024-04-02T05:07:18.632363+0000","last_active":"2024-04-02T05:08:05.006439+0000","last_peered":"2024-04-02T05:08:05.006439+0000","last_clean":"2024-04-02T05:08:05.006439+0000","last_became_active":"2024-04-02T04:53:54.943365+0000","last_became_peered":"2024-04-02T04:53:54.943365+0000","last_unstale":"2024-04-02T05:08:05.006439+0000","last_undegraded":"2024-04-02T05:08:05.006439+0000","last_fullsized":"2024-04-02T05:08:05.006439+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:18.632111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:18.632111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:40.990446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":81,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396628+0000","last_change":"2024-04-02T05:07:53.740460+0000","last_active":"2024-04-02T05:08:04.396628+0000","last_peered":"2024-04-02T05:08:04.396628+0000","last_clean":"2024-04-02T05:08:04.396628+0000","last_became_active":"2024-04-02T05:05:54.275404+0000","last_became_peered":"2024-04-02T05:05:54.275404+0000","last_unstale":"2024-04-02T05:08:04.396628+0000","last_undegraded":"2024-04-02T05:08:04.396628+0000","last_fullsized":"2024-04-02T05:08:04.396628+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:53.740246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:57.254461+0000","last_clean_scrub_stamp":"2024-04-02T05:07:53.740246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:10.903946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":168,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.005078+0000","last_change":"2024-04-02T05:07:42.178758+0000","last_active":"2024-04-02T05:08:05.005078+0000","last_peered":"2024-04-02T05:08:05.005078+0000","last_clean":"2024-04-02T05:08:05.005078+0000","last_became_active":"2024-04-02T05:04:11.225907+0000","last_became_peered":"2024-04-02T05:04:11.225907+0000","last_unstale":"2024-04-02T05:08:05.005078+0000","last_undegraded":"2024-04-02T05:08:05.005078+0000","last_fullsized":"2024-04-02T05:08:05.005078+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:42.178550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:59.289274+0000","last_clean_scrub_stamp":"2024-04-02T05:07:42.178550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:57.864068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1729,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.667699+0000","last_change":"2024-04-02T05:07:16.758675+0000","last_active":"2024-04-02T05:08:04.667699+0000","last_peered":"2024-04-02T05:08:04.667699+0000","last_clean":"2024-04-02T05:08:04.667699+0000","last_became_active":"2024-04-02T05:04:11.247660+0000","last_became_peered":"2024-04-02T05:04:11.247660+0000","last_unstale":"2024-04-02T05:08:04.667699+0000","last_undegraded":"2024-04-02T05:08:04.667699+0000","last_fullsized":"2024-04-02T05:08:04.667699+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:16.758458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:16.758458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:27.103181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"899'33","reported_seq":2028,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:14.035775+0000","last_change":"2024-04-02T05:08:14.035828+0000","last_active":"2024-04-02T05:08:14.035775+0000","last_peered":"2024-04-02T05:08:14.035775+0000","last_clean":"2024-04-02T05:08:14.035775+0000","last_became_active":"2024-04-02T05:05:16.543343+0000","last_became_peered":"2024-04-02T05:05:16.543343+0000","last_unstale":"2024-04-02T05:08:14.035775+0000","last_undegraded":"2024-04-02T05:08:14.035775+0000","last_fullsized":"2024-04-02T05:08:14.035775+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'33","last_scrub_stamp":"2024-04-02T05:08:14.035688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:55.199126+0000","last_clean_scrub_stamp":"2024-04-02T05:08:14.035688+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:18.691849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":711,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400951+0000","last_change":"2024-04-02T05:07:51.512534+0000","last_active":"2024-04-02T05:08:04.400951+0000","last_peered":"2024-04-02T05:08:04.400951+0000","last_clean":"2024-04-02T05:08:04.400951+0000","last_became_active":"2024-04-02T05:04:11.244435+0000","last_became_peered":"2024-04-02T05:04:11.244435+0000","last_unstale":"2024-04-02T05:08:04.400951+0000","last_undegraded":"2024-04-02T05:08:04.400951+0000","last_fullsized":"2024-04-02T05:08:04.400951+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:51.512226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:50.043598+0000","last_clean_scrub_stamp":"2024-04-02T05:07:51.512226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.909126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1869,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.395501+0000","last_change":"2024-04-02T05:06:56.489013+0000","last_active":"2024-04-02T05:08:04.395501+0000","last_peered":"2024-04-02T05:08:04.395501+0000","last_clean":"2024-04-02T05:08:04.395501+0000","last_became_active":"2024-04-02T04:43:58.296829+0000","last_became_peered":"2024-04-02T04:43:58.296829+0000","last_unstale":"2024-04-02T05:08:04.395501+0000","last_undegraded":"2024-04-02T05:08:04.395501+0000","last_fullsized":"2024-04-02T05:08:04.395501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:56.488795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:05.649928+0000","last_clean_scrub_stamp":"2024-04-02T05:06:56.488795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396917+0000","last_change":"2024-04-02T05:07:05.543736+0000","last_active":"2024-04-02T05:08:04.396917+0000","last_peered":"2024-04-02T05:08:04.396917+0000","last_clean":"2024-04-02T05:08:04.396917+0000","last_became_active":"2024-04-02T05:04:11.255981+0000","last_became_peered":"2024-04-02T05:04:11.255981+0000","last_unstale":"2024-04-02T05:08:04.396917+0000","last_undegraded":"2024-04-02T05:08:04.396917+0000","last_fullsized":"2024-04-02T05:08:04.396917+0000","mapping_epoch":901,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":902,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:05.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:57:41.058751+0000","last_clean_scrub_stamp":"2024-04-02T05:07:05.543525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:31.265009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1753,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.399001+0000","last_change":"2024-04-02T05:07:50.697827+0000","last_active":"2024-04-02T05:08:04.399001+0000","last_peered":"2024-04-02T05:08:04.399001+0000","last_clean":"2024-04-02T05:08:04.399001+0000","last_became_active":"2024-04-02T04:59:00.135538+0000","last_became_peered":"2024-04-02T04:59:00.135538+0000","last_unstale":"2024-04-02T05:08:04.399001+0000","last_undegraded":"2024-04-02T05:08:04.399001+0000","last_fullsized":"2024-04-02T05:08:04.399001+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:50.697605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:49:15.972156+0000","last_clean_scrub_stamp":"2024-04-02T05:07:50.697605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:09:06.144527+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"899'42","reported_seq":776,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400678+0000","last_change":"2024-04-02T05:07:33.588845+0000","last_active":"2024-04-02T05:08:04.400678+0000","last_peered":"2024-04-02T05:08:04.400678+0000","last_clean":"2024-04-02T05:08:04.400678+0000","last_became_active":"2024-04-02T05:05:16.544863+0000","last_became_peered":"2024-04-02T05:05:16.544863+0000","last_unstale":"2024-04-02T05:08:04.400678+0000","last_undegraded":"2024-04-02T05:08:04.400678+0000","last_fullsized":"2024-04-02T05:08:04.400678+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'42","last_scrub_stamp":"2024-04-02T05:07:33.588619+0000","last_deep_scrub":"402'32","last_deep_scrub_stamp":"2024-04-02T04:59:02.951751+0000","last_clean_scrub_stamp":"2024-04-02T05:07:33.588619+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:37.716255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1743,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:05.074583+0000","last_change":"2024-04-02T05:07:43.113383+0000","last_active":"2024-04-02T05:08:05.074583+0000","last_peered":"2024-04-02T05:08:05.074583+0000","last_clean":"2024-04-02T05:08:05.074583+0000","last_became_active":"2024-04-02T05:05:54.272364+0000","last_became_peered":"2024-04-02T05:05:54.272364+0000","last_unstale":"2024-04-02T05:08:05.074583+0000","last_undegraded":"2024-04-02T05:08:05.074583+0000","last_fullsized":"2024-04-02T05:08:05.074583+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:43.113176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:43:53.179277+0000","last_clean_scrub_stamp":"2024-04-02T05:07:43.113176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:45.599605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":716,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.400502+0000","last_change":"2024-04-02T05:06:55.339683+0000","last_active":"2024-04-02T05:08:04.400502+0000","last_peered":"2024-04-02T05:08:04.400502+0000","last_clean":"2024-04-02T05:08:04.400502+0000","last_became_active":"2024-04-02T05:05:16.544759+0000","last_became_peered":"2024-04-02T05:05:16.544759+0000","last_unstale":"2024-04-02T05:08:04.400502+0000","last_undegraded":"2024-04-02T05:08:04.400502+0000","last_fullsized":"2024-04-02T05:08:04.400502+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:06:55.339468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:01:14.663348+0000","last_clean_scrub_stamp":"2024-04-02T05:06:55.339468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:13.584923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":698,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-02T05:08:04.396681+0000","last_change":"2024-04-02T05:07:24.505337+0000","last_active":"2024-04-02T05:08:04.396681+0000","last_peered":"2024-04-02T05:08:04.396681+0000","last_clean":"2024-04-02T05:08:04.396681+0000","last_became_active":"2024-04-02T04:59:00.122322+0000","last_became_peered":"2024-04-02T04:59:00.122322+0000","last_unstale":"2024-04-02T05:08:04.396681+0000","last_undegraded":"2024-04-02T05:08:04.396681+0000","last_fullsized":"2024-04-02T05:08:04.396681+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T05:07:24.505123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T05:05:52.064373+0000","last_clean_scrub_stamp":"2024-04-02T05:07:24.505123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:08:39.647086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1818,"num_read_kb":1543,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":7284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1293,"ondisk_log_size":1293,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":563,"num_read_kb":558,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":394,"seq":1692217114800,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56976,"kb_used_data":2440,"kb_used_omap":57,"kb_used_meta":54470,"kb_avail":93688176,"statfs":{"total":95995035648,"available":95936692224,"internally_reserved":0,"allocated":2498560,"data_stored":723609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58482,"internal_metadata":55778190},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":58,"seq":249108103483,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101932,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93643220,"statfs":{"total":95995035648,"available":95890657280,"internally_reserved":0,"allocated":2789376,"data_stored":1049097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575214,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114108,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93631044,"statfs":{"total":95995035648,"available":95878189056,"internally_reserved":0,"allocated":2478080,"data_stored":722497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738744,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114432,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630720,"statfs":{"total":95995035648,"available":95877857280,"internally_reserved":0,"allocated":2809856,"data_stored":1049766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378980,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114360,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630792,"statfs":{"total":95995035648,"available":95877931008,"internally_reserved":0,"allocated":2736128,"data_stored":985531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051930,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114672,"kb_used_data":2984,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630480,"statfs":{"total":95995035648,"available":95877611520,"internally_reserved":0,"allocated":3055616,"data_stored":1312166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822992,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96084,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649068,"statfs":{"total":95995035648,"available":95896645632,"internally_reserved":0,"allocated":2502656,"data_stored":722621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463236,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51892,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93693260,"statfs":{"total":95995035648,"available":95941898240,"internally_reserved":0,"allocated":2732032,"data_stored":985330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T05:08:24.022 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-02T05:08:24.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca shell --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-02T05:08:24.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:08:24 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:08:24.783 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:24 smithi069 bash[21528]: audit 2024-04-02T05:08:23.220318+0000 mgr.y (mgr.14538) 941 : audit 0 from='client.44141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:08:24.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:24 smithi069 bash[28904]: audit 2024-04-02T05:08:23.220318+0000 mgr.y (mgr.14538) 941 : audit 0 from='client.44141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:08:24.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:24 smithi098 bash[23790]: audit 2024-04-02T05:08:23.220318+0000 mgr.y (mgr.14538) 941 : audit 0 from='client.44141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T05:08:25.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:25 smithi098 bash[23790]: cluster 2024-04-02T05:08:23.806643+0000 osd.4 (osd.4) 582 : cluster 0 10.1d scrub starts 2024-04-02T05:08:25.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:25 smithi098 bash[23790]: cluster 2024-04-02T05:08:23.810958+0000 osd.4 (osd.4) 583 : cluster 0 10.1d scrub ok 2024-04-02T05:08:25.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:25 smithi098 bash[23790]: audit 2024-04-02T05:08:24.033126+0000 mgr.y (mgr.14538) 942 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:25 smithi098 bash[23790]: cluster 2024-04-02T05:08:24.530142+0000 mgr.y (mgr.14538) 943 : cluster 0 pgmap v1577: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:25 smithi098 bash[23790]: cluster 2024-04-02T05:08:24.530645+0000 osd.3 (osd.3) 641 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:08:25.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:25 smithi098 bash[23790]: cluster 2024-04-02T05:08:24.537301+0000 osd.3 (osd.3) 642 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[21528]: cluster 2024-04-02T05:08:23.806643+0000 osd.4 (osd.4) 582 : cluster 0 10.1d scrub starts 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[21528]: cluster 2024-04-02T05:08:23.810958+0000 osd.4 (osd.4) 583 : cluster 0 10.1d scrub ok 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[21528]: audit 2024-04-02T05:08:24.033126+0000 mgr.y (mgr.14538) 942 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[21528]: cluster 2024-04-02T05:08:24.530142+0000 mgr.y (mgr.14538) 943 : cluster 0 pgmap v1577: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[21528]: cluster 2024-04-02T05:08:24.530645+0000 osd.3 (osd.3) 641 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[21528]: cluster 2024-04-02T05:08:24.537301+0000 osd.3 (osd.3) 642 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:08:26.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[28904]: cluster 2024-04-02T05:08:23.806643+0000 osd.4 (osd.4) 582 : cluster 0 10.1d scrub starts 2024-04-02T05:08:26.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[28904]: cluster 2024-04-02T05:08:23.810958+0000 osd.4 (osd.4) 583 : cluster 0 10.1d scrub ok 2024-04-02T05:08:26.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[28904]: audit 2024-04-02T05:08:24.033126+0000 mgr.y (mgr.14538) 942 : audit 0 from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-02T05:08:26.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[28904]: cluster 2024-04-02T05:08:24.530142+0000 mgr.y (mgr.14538) 943 : cluster 0 pgmap v1577: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:26.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[28904]: cluster 2024-04-02T05:08:24.530645+0000 osd.3 (osd.3) 641 : cluster 0 3.8 deep-scrub starts 2024-04-02T05:08:26.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:25 smithi069 bash[28904]: cluster 2024-04-02T05:08:24.537301+0000 osd.3 (osd.3) 642 : cluster 0 3.8 deep-scrub ok 2024-04-02T05:08:26.982 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:24.770782+0000 osd.4 (osd.4) 584 : cluster 0 1.5 scrub starts 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:24.773445+0000 osd.4 (osd.4) 585 : cluster 0 1.5 scrub ok 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.204073+0000 osd.0 (osd.0) 575 : cluster 0 10.4 deep-scrub starts 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.206655+0000 osd.0 (osd.0) 576 : cluster 0 10.4 deep-scrub ok 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.724206+0000 osd.4 (osd.4) 586 : cluster 0 4.18 scrub starts 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.726676+0000 osd.4 (osd.4) 587 : cluster 0 4.18 scrub ok 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.751922+0000 osd.7 (osd.7) 415 : cluster 0 3.13 deep-scrub starts 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.763341+0000 osd.7 (osd.7) 416 : cluster 0 3.13 deep-scrub ok 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.929082+0000 osd.2 (osd.2) 450 : cluster 0 4.1c scrub starts 2024-04-02T05:08:26.983 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:26 smithi098 bash[23790]: cluster 2024-04-02T05:08:25.931559+0000 osd.2 (osd.2) 451 : cluster 0 4.1c scrub ok 2024-04-02T05:08:27.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:24.770782+0000 osd.4 (osd.4) 584 : cluster 0 1.5 scrub starts 2024-04-02T05:08:27.035 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:24.773445+0000 osd.4 (osd.4) 585 : cluster 0 1.5 scrub ok 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.204073+0000 osd.0 (osd.0) 575 : cluster 0 10.4 deep-scrub starts 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.206655+0000 osd.0 (osd.0) 576 : cluster 0 10.4 deep-scrub ok 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.724206+0000 osd.4 (osd.4) 586 : cluster 0 4.18 scrub starts 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.726676+0000 osd.4 (osd.4) 587 : cluster 0 4.18 scrub ok 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.751922+0000 osd.7 (osd.7) 415 : cluster 0 3.13 deep-scrub starts 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.763341+0000 osd.7 (osd.7) 416 : cluster 0 3.13 deep-scrub ok 2024-04-02T05:08:27.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.929082+0000 osd.2 (osd.2) 450 : cluster 0 4.1c scrub starts 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[21528]: cluster 2024-04-02T05:08:25.931559+0000 osd.2 (osd.2) 451 : cluster 0 4.1c scrub ok 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:24.770782+0000 osd.4 (osd.4) 584 : cluster 0 1.5 scrub starts 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:24.773445+0000 osd.4 (osd.4) 585 : cluster 0 1.5 scrub ok 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.204073+0000 osd.0 (osd.0) 575 : cluster 0 10.4 deep-scrub starts 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.206655+0000 osd.0 (osd.0) 576 : cluster 0 10.4 deep-scrub ok 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.724206+0000 osd.4 (osd.4) 586 : cluster 0 4.18 scrub starts 2024-04-02T05:08:27.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.726676+0000 osd.4 (osd.4) 587 : cluster 0 4.18 scrub ok 2024-04-02T05:08:27.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.751922+0000 osd.7 (osd.7) 415 : cluster 0 3.13 deep-scrub starts 2024-04-02T05:08:27.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.763341+0000 osd.7 (osd.7) 416 : cluster 0 3.13 deep-scrub ok 2024-04-02T05:08:27.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.929082+0000 osd.2 (osd.2) 450 : cluster 0 4.1c scrub starts 2024-04-02T05:08:27.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:26 smithi069 bash[28904]: cluster 2024-04-02T05:08:25.931559+0000 osd.2 (osd.2) 451 : cluster 0 4.1c scrub ok 2024-04-02T05:08:27.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:27 smithi098 bash[23790]: cluster 2024-04-02T05:08:26.523966+0000 osd.3 (osd.3) 643 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:08:27.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:27 smithi098 bash[23790]: cluster 2024-04-02T05:08:26.526541+0000 osd.3 (osd.3) 644 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:08:27.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:27 smithi098 bash[23790]: cluster 2024-04-02T05:08:26.530968+0000 mgr.y (mgr.14538) 944 : cluster 0 pgmap v1578: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:27 smithi069 bash[21528]: cluster 2024-04-02T05:08:26.523966+0000 osd.3 (osd.3) 643 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:08:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:27 smithi069 bash[21528]: cluster 2024-04-02T05:08:26.526541+0000 osd.3 (osd.3) 644 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:08:28.033 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:27 smithi069 bash[21528]: cluster 2024-04-02T05:08:26.530968+0000 mgr.y (mgr.14538) 944 : cluster 0 pgmap v1578: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:27 smithi069 bash[28904]: cluster 2024-04-02T05:08:26.523966+0000 osd.3 (osd.3) 643 : cluster 0 2.1b deep-scrub starts 2024-04-02T05:08:28.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:27 smithi069 bash[28904]: cluster 2024-04-02T05:08:26.526541+0000 osd.3 (osd.3) 644 : cluster 0 2.1b deep-scrub ok 2024-04-02T05:08:28.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:27 smithi069 bash[28904]: cluster 2024-04-02T05:08:26.530968+0000 mgr.y (mgr.14538) 944 : cluster 0 pgmap v1578: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:28.804 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T05:08:28.824 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21782]: ::ffff:172.21.15.98 - - [02/Apr/2024:05:08:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-02T05:08:28.824 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.210794+0000 osd.0 (osd.0) 577 : cluster 0 10.1e deep-scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.215066+0000 osd.0 (osd.0) 578 : cluster 0 10.1e deep-scrub ok 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.486868+0000 osd.3 (osd.3) 645 : cluster 0 6.13 scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.489334+0000 osd.3 (osd.3) 646 : cluster 0 6.13 scrub ok 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.690100+0000 osd.1 (osd.1) 617 : cluster 0 3.0 scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.695440+0000 osd.1 (osd.1) 618 : cluster 0 3.0 scrub ok 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.737937+0000 osd.4 (osd.4) 588 : cluster 0 6.1e scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.740007+0000 osd.4 (osd.4) 589 : cluster 0 6.1e scrub ok 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.769968+0000 osd.7 (osd.7) 417 : cluster 0 10.c deep-scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[21528]: cluster 2024-04-02T05:08:27.772430+0000 osd.7 (osd.7) 418 : cluster 0 10.c deep-scrub ok 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.210794+0000 osd.0 (osd.0) 577 : cluster 0 10.1e deep-scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.215066+0000 osd.0 (osd.0) 578 : cluster 0 10.1e deep-scrub ok 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.486868+0000 osd.3 (osd.3) 645 : cluster 0 6.13 scrub starts 2024-04-02T05:08:28.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.489334+0000 osd.3 (osd.3) 646 : cluster 0 6.13 scrub ok 2024-04-02T05:08:28.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.690100+0000 osd.1 (osd.1) 617 : cluster 0 3.0 scrub starts 2024-04-02T05:08:28.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.695440+0000 osd.1 (osd.1) 618 : cluster 0 3.0 scrub ok 2024-04-02T05:08:28.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.737937+0000 osd.4 (osd.4) 588 : cluster 0 6.1e scrub starts 2024-04-02T05:08:28.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.740007+0000 osd.4 (osd.4) 589 : cluster 0 6.1e scrub ok 2024-04-02T05:08:28.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.769968+0000 osd.7 (osd.7) 417 : cluster 0 10.c deep-scrub starts 2024-04-02T05:08:28.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:28 smithi069 bash[28904]: cluster 2024-04-02T05:08:27.772430+0000 osd.7 (osd.7) 418 : cluster 0 10.c deep-scrub ok 2024-04-02T05:08:28.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.210794+0000 osd.0 (osd.0) 577 : cluster 0 10.1e deep-scrub starts 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.215066+0000 osd.0 (osd.0) 578 : cluster 0 10.1e deep-scrub ok 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.486868+0000 osd.3 (osd.3) 645 : cluster 0 6.13 scrub starts 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.489334+0000 osd.3 (osd.3) 646 : cluster 0 6.13 scrub ok 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.690100+0000 osd.1 (osd.1) 617 : cluster 0 3.0 scrub starts 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.695440+0000 osd.1 (osd.1) 618 : cluster 0 3.0 scrub ok 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.737937+0000 osd.4 (osd.4) 588 : cluster 0 6.1e scrub starts 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.740007+0000 osd.4 (osd.4) 589 : cluster 0 6.1e scrub ok 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.769968+0000 osd.7 (osd.7) 417 : cluster 0 10.c deep-scrub starts 2024-04-02T05:08:28.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:28 smithi098 bash[23790]: cluster 2024-04-02T05:08:27.772430+0000 osd.7 (osd.7) 418 : cluster 0 10.c deep-scrub ok 2024-04-02T05:08:29.823 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:29 smithi069 bash[21528]: cluster 2024-04-02T05:08:28.532829+0000 mgr.y (mgr.14538) 945 : cluster 0 pgmap v1579: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:29.823 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:29 smithi069 bash[21528]: cluster 2024-04-02T05:08:28.812597+0000 osd.7 (osd.7) 419 : cluster 0 2.5 scrub starts 2024-04-02T05:08:29.823 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:29 smithi069 bash[21528]: cluster 2024-04-02T05:08:28.814865+0000 osd.7 (osd.7) 420 : cluster 0 2.5 scrub ok 2024-04-02T05:08:29.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:29 smithi069 bash[28904]: cluster 2024-04-02T05:08:28.532829+0000 mgr.y (mgr.14538) 945 : cluster 0 pgmap v1579: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:29.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:29 smithi069 bash[28904]: cluster 2024-04-02T05:08:28.812597+0000 osd.7 (osd.7) 419 : cluster 0 2.5 scrub starts 2024-04-02T05:08:29.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:29 smithi069 bash[28904]: cluster 2024-04-02T05:08:28.814865+0000 osd.7 (osd.7) 420 : cluster 0 2.5 scrub ok 2024-04-02T05:08:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:29 smithi098 bash[23790]: cluster 2024-04-02T05:08:28.532829+0000 mgr.y (mgr.14538) 945 : cluster 0 pgmap v1579: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-02T05:08:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:29 smithi098 bash[23790]: cluster 2024-04-02T05:08:28.812597+0000 osd.7 (osd.7) 419 : cluster 0 2.5 scrub starts 2024-04-02T05:08:29.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:29 smithi098 bash[23790]: cluster 2024-04-02T05:08:28.814865+0000 osd.7 (osd.7) 420 : cluster 0 2.5 scrub ok 2024-04-02T05:08:30.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:30 smithi098 bash[23790]: cluster 2024-04-02T05:08:28.913891+0000 osd.6 (osd.6) 551 : cluster 0 10.1c scrub starts 2024-04-02T05:08:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:30 smithi098 bash[23790]: cluster 2024-04-02T05:08:28.916282+0000 osd.6 (osd.6) 552 : cluster 0 10.1c scrub ok 2024-04-02T05:08:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:30 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.651870+0000 osd.1 (osd.1) 619 : cluster 0 2.4 scrub starts 2024-04-02T05:08:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:30 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.654019+0000 osd.1 (osd.1) 620 : cluster 0 2.4 scrub ok 2024-04-02T05:08:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:30 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.815215+0000 osd.7 (osd.7) 421 : cluster 0 6.8 scrub starts 2024-04-02T05:08:30.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:30 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.817007+0000 osd.7 (osd.7) 422 : cluster 0 6.8 scrub ok 2024-04-02T05:08:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[28904]: cluster 2024-04-02T05:08:28.913891+0000 osd.6 (osd.6) 551 : cluster 0 10.1c scrub starts 2024-04-02T05:08:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[28904]: cluster 2024-04-02T05:08:28.916282+0000 osd.6 (osd.6) 552 : cluster 0 10.1c scrub ok 2024-04-02T05:08:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.651870+0000 osd.1 (osd.1) 619 : cluster 0 2.4 scrub starts 2024-04-02T05:08:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.654019+0000 osd.1 (osd.1) 620 : cluster 0 2.4 scrub ok 2024-04-02T05:08:31.033 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.815215+0000 osd.7 (osd.7) 421 : cluster 0 6.8 scrub starts 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.817007+0000 osd.7 (osd.7) 422 : cluster 0 6.8 scrub ok 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[21528]: cluster 2024-04-02T05:08:28.913891+0000 osd.6 (osd.6) 551 : cluster 0 10.1c scrub starts 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[21528]: cluster 2024-04-02T05:08:28.916282+0000 osd.6 (osd.6) 552 : cluster 0 10.1c scrub ok 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.651870+0000 osd.1 (osd.1) 619 : cluster 0 2.4 scrub starts 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.654019+0000 osd.1 (osd.1) 620 : cluster 0 2.4 scrub ok 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.815215+0000 osd.7 (osd.7) 421 : cluster 0 6.8 scrub starts 2024-04-02T05:08:31.034 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:30 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.817007+0000 osd.7 (osd.7) 422 : cluster 0 6.8 scrub ok 2024-04-02T05:08:31.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2923180000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc107430 msgr2=0x7f43bc107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:31.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2923180000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc107430 0x7f43bc107810 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0098f0 tx=0x7f43ac009c00 comp rx=0 tx=0).stop 2024-04-02T05:08:31.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2923180000 shutdown_connections 2024-04-02T05:08:31.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2923180000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc1026d0 0x7f43bc10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:31.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2923180000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc107430 0x7f43bc107810 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:31.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2923180000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43bc101d10 0x7f43bc102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2923180000 >> 172.21.15.69:0/2923180000 conn(0x7f43bc076460 msgr2=0x7f43bc076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2923180000 shutdown_connections 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2923180000 wait complete. 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 librados: starting msgr at 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 librados: starting objecter 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 Processor -- start 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- start start 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 librados: setting wanted keys 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 librados: calling monclient init 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 0x7f43bc1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc1026d0 0x7f43bc198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:31.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43bc107430 0x7f43bc19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc110a70 con 0x7f43bc101d10 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1108f0 con 0x7f43bc107430 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 -- 0x7f43bc110bf0 con 0x7f43bc1026d0 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 0x7f43bc1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43b9d9b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc1026d0 0x7f43bc198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 0x7f43bc1982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40570/0 (socket says 172.21.15.69:40570) 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 -- 172.21.15.69:0/2481637946 learned_addr learned my addr 172.21.15.69:0/2481637946 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 -- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc1026d0 msgr2=0x7f43bc198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc1026d0 0x7f43bc198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 -- 172.21.15.69:0/2481637946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43bc107430 msgr2=0x7f43bc19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43bc107430 0x7f43bc19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 -- 172.21.15.69:0/2481637946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43bc19d2f0 con 0x7f43bc101d10 2024-04-02T05:08:31.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43b9d9b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc1026d0 0x7f43bc198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T05:08:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ba59c700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 0x7f43bc1982e0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00fb40 tx=0x7f43ac00fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ab7fe700 10 client.?.objecter ms_handle_connect 0x7f43bc101d10 2024-04-02T05:08:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ab7fe700 10 client.?.objecter resend_mon_ops 2024-04-02T05:08:31.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac027070 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43bc19d580 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) ==== 1055+0+0 (secure 0 0 0) 0x7f43ac005040 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac004910 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 10 client.34080.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43bc1a4e60 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.762+0000 7f43c0b0b700 1 librados: init done 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43bc04ff00 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 22) ==== 91831+0+0 (secure 0 0 0) 0x7f43ac02a070 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f43a406f5e0 0x7f43a4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(969..969 src has 918..969) ==== 9015+0+0 (secure 0 0 0) 0x7f43ac08f220 con 0x7f43bc101d10 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 10 client.34080.objecter ms_dispatch 0x7f43bc001550 osd_map(969..969 src has 918..969) 2024-04-02T05:08:31.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 3 client.34080.objecter handle_osd_map got epochs [969,969] > 0 2024-04-02T05:08:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 3 client.34080.objecter handle_osd_map decoding full epoch 969 2024-04-02T05:08:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43b9d9b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f43a406f5e0 0x7f43a4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T05:08:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 20 client.34080.objecter dump_active .. 0 homeless 2024-04-02T05:08:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43ab7fe700 10 client.34080.objecter ms_handle_connect 0x7f43a406f5e0 2024-04-02T05:08:31.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.766+0000 7f43b9d9b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f43a406f5e0 0x7f43a4071aa0 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7f43bc076880 tx=0x7f43b000b3c0 comp rx=0 tx=0).ready entity=mgr.14538 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T05:08:31.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.770+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f43ac013030 con 0x7f43bc101d10 2024-04-02T05:08:31.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.758663+0000 osd.4 (osd.4) 590 : cluster 0 6.17 scrub starts 2024-04-02T05:08:32.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.761163+0000 osd.4 (osd.4) 591 : cluster 0 6.17 scrub ok 2024-04-02T05:08:32.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.808442+0000 osd.4 (osd.4) 592 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:32.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.808742+0000 osd.4 (osd.4) 593 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:32.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.944864+0000 osd.6 (osd.6) 553 : cluster 0 10.16 scrub starts 2024-04-02T05:08:32.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:29.948409+0000 osd.6 (osd.6) 554 : cluster 0 10.16 scrub ok 2024-04-02T05:08:32.078 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:30.533925+0000 mgr.y (mgr.14538) 946 : cluster 0 pgmap v1580: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:32.079 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:30.580910+0000 mon.a (mon.0) 4208 : cluster 0 osdmap e969: 8 total, 8 up, 8 in 2024-04-02T05:08:32.079 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:30.610502+0000 osd.1 (osd.1) 621 : cluster 0 4.7 scrub starts 2024-04-02T05:08:32.079 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:31 smithi098 bash[23790]: cluster 2024-04-02T05:08:30.613034+0000 osd.1 (osd.1) 622 : cluster 0 4.7 scrub ok 2024-04-02T05:08:32.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.982+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) -- 0x7f43bc102dc0 con 0x7f43bc101d10 2024-04-02T05:08:32.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.986+0000 7f43ab7fe700 1 -- 172.21.15.69:0/2481637946 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v31) ==== 118+0+0 (secure 0 0 0) 0x7f43ac05c310 con 0x7f43bc101d10 2024-04-02T05:08:32.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 10 librados: watch_flush enter 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 10 librados: watch_flush exit 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 20 client.34080.objecter shutdown clearing up homeless session... 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 10 client.34080.objecter successfully canceled tick 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f43a406f5e0 msgr2=0x7f43a4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f43a406f5e0 0x7f43a4071aa0 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7f43bc076880 tx=0x7f43b000b3c0 comp rx=0 tx=0).stop 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 msgr2=0x7f43bc1982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 0x7f43bc1982e0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00fb40 tx=0x7f43ac00fb70 comp rx=0 tx=0).stop 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 shutdown_connections 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:6800/1004417273,v1:172.21.15.69:6801/1004417273] conn(0x7f43a406f5e0 0x7f43a4071aa0 unknown :-1 s=CLOSED pgs=4070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43bc107430 0x7f43bc19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43bc101d10 0x7f43bc1982e0 secure :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00fb40 tx=0x7f43ac00fb70 comp rx=0 tx=0).stop 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 --2- 172.21.15.69:0/2481637946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43bc1026d0 0x7f43bc198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 >> 172.21.15.69:0/2481637946 conn(0x7f43bc076460 msgr2=0x7f43bc100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T05:08:32.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 shutdown_connections 2024-04-02T05:08:32.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 -- 172.21.15.69:0/2481637946 wait complete. 2024-04-02T05:08:32.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-02T05:08:31.990+0000 7f43c0b0b700 1 librados: shutdown 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.758663+0000 osd.4 (osd.4) 590 : cluster 0 6.17 scrub starts 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.761163+0000 osd.4 (osd.4) 591 : cluster 0 6.17 scrub ok 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.808442+0000 osd.4 (osd.4) 592 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.808742+0000 osd.4 (osd.4) 593 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.944864+0000 osd.6 (osd.6) 553 : cluster 0 10.16 scrub starts 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:29.948409+0000 osd.6 (osd.6) 554 : cluster 0 10.16 scrub ok 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:30.533925+0000 mgr.y (mgr.14538) 946 : cluster 0 pgmap v1580: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:32.082 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:30.580910+0000 mon.a (mon.0) 4208 : cluster 0 osdmap e969: 8 total, 8 up, 8 in 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:30.610502+0000 osd.1 (osd.1) 621 : cluster 0 4.7 scrub starts 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[21528]: cluster 2024-04-02T05:08:30.613034+0000 osd.1 (osd.1) 622 : cluster 0 4.7 scrub ok 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.758663+0000 osd.4 (osd.4) 590 : cluster 0 6.17 scrub starts 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.761163+0000 osd.4 (osd.4) 591 : cluster 0 6.17 scrub ok 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.808442+0000 osd.4 (osd.4) 592 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.808742+0000 osd.4 (osd.4) 593 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.944864+0000 osd.6 (osd.6) 553 : cluster 0 10.16 scrub starts 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:29.948409+0000 osd.6 (osd.6) 554 : cluster 0 10.16 scrub ok 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:30.533925+0000 mgr.y (mgr.14538) 946 : cluster 0 pgmap v1580: 184 pgs: 184 active+clean; 582 KiB data, 747 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:30.580910+0000 mon.a (mon.0) 4208 : cluster 0 osdmap e969: 8 total, 8 up, 8 in 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:30.610502+0000 osd.1 (osd.1) 621 : cluster 0 4.7 scrub starts 2024-04-02T05:08:32.083 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:31 smithi069 bash[28904]: cluster 2024-04-02T05:08:30.613034+0000 osd.1 (osd.1) 622 : cluster 0 4.7 scrub ok 2024-04-02T05:08:32.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21782]: [02/Apr/2024:05:08:32] ENGINE Bus STOPPING 2024-04-02T05:08:32.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21782]: [02/Apr/2024:05:08:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-02T05:08:32.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21782]: [02/Apr/2024:05:08:32] ENGINE Bus STOPPED 2024-04-02T05:08:32.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21782]: [02/Apr/2024:05:08:32] ENGINE Bus STARTING 2024-04-02T05:08:32.283 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21782]: [02/Apr/2024:05:08:32] ENGINE Serving on http://:::9283 2024-04-02T05:08:32.284 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21782]: [02/Apr/2024:05:08:32] ENGINE Bus STARTED 2024-04-02T05:08:32.759 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-02T05:08:32.769 INFO:tasks.cephadm:Teardown begin 2024-04-02T05:08:32.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T05:08:32.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T05:08:32.797 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-02T05:08:32.797 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T05:08:32.828 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T05:08:32.844 INFO:tasks.cephadm:Stopping all daemons... 2024-04-02T05:08:32.845 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-02T05:08:32.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: cluster 2024-04-02T05:08:31.589749+0000 osd.1 (osd.1) 623 : cluster 0 2.9 scrub starts 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: cluster 2024-04-02T05:08:31.592963+0000 osd.1 (osd.1) 624 : cluster 0 2.9 scrub ok 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: cluster 2024-04-02T05:08:31.849945+0000 osd.7 (osd.7) 423 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: cluster 2024-04-02T05:08:31.850370+0000 osd.7 (osd.7) 424 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: audit 2024-04-02T05:08:31.986574+0000 mon.a (mon.0) 4209 : audit 1 from='client.? 172.21.15.69:0/2481637946' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: audit 2024-04-02T05:08:31.990431+0000 mon.a (mon.0) 4210 : audit 1 from='client.? 172.21.15.69:0/2481637946' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-02T05:08:32.882 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: audit 2024-04-02T05:08:32.037648+0000 mon.b (mon.1) 365 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: audit 2024-04-02T05:08:32.040325+0000 mon.b (mon.1) 366 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: audit 2024-04-02T05:08:32.041574+0000 mon.b (mon.1) 367 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[21528]: audit 2024-04-02T05:08:32.047330+0000 mon.a (mon.0) 4211 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: cluster 2024-04-02T05:08:31.589749+0000 osd.1 (osd.1) 623 : cluster 0 2.9 scrub starts 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: cluster 2024-04-02T05:08:31.592963+0000 osd.1 (osd.1) 624 : cluster 0 2.9 scrub ok 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: cluster 2024-04-02T05:08:31.849945+0000 osd.7 (osd.7) 423 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: cluster 2024-04-02T05:08:31.850370+0000 osd.7 (osd.7) 424 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: audit 2024-04-02T05:08:31.986574+0000 mon.a (mon.0) 4209 : audit 1 from='client.? 172.21.15.69:0/2481637946' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: audit 2024-04-02T05:08:31.990431+0000 mon.a (mon.0) 4210 : audit 1 from='client.? 172.21.15.69:0/2481637946' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: audit 2024-04-02T05:08:32.037648+0000 mon.b (mon.1) 365 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:08:32.883 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: audit 2024-04-02T05:08:32.040325+0000 mon.b (mon.1) 366 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:08:32.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: audit 2024-04-02T05:08:32.041574+0000 mon.b (mon.1) 367 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:08:32.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:32 smithi069 bash[28904]: audit 2024-04-02T05:08:32.047330+0000 mon.a (mon.0) 4211 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:08:32.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: cluster 2024-04-02T05:08:31.589749+0000 osd.1 (osd.1) 623 : cluster 0 2.9 scrub starts 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: cluster 2024-04-02T05:08:31.592963+0000 osd.1 (osd.1) 624 : cluster 0 2.9 scrub ok 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: cluster 2024-04-02T05:08:31.849945+0000 osd.7 (osd.7) 423 : cluster 0 purged_snaps scrub starts 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: cluster 2024-04-02T05:08:31.850370+0000 osd.7 (osd.7) 424 : cluster 0 purged_snaps scrub ok 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: audit 2024-04-02T05:08:31.986574+0000 mon.a (mon.0) 4209 : audit 1 from='client.? 172.21.15.69:0/2481637946' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: audit 2024-04-02T05:08:31.990431+0000 mon.a (mon.0) 4210 : audit 1 from='client.? 172.21.15.69:0/2481637946' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: audit 2024-04-02T05:08:32.037648+0000 mon.b (mon.1) 365 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: audit 2024-04-02T05:08:32.040325+0000 mon.b (mon.1) 366 : audit 0 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: audit 2024-04-02T05:08:32.041574+0000 mon.b (mon.1) 367 : audit 1 from='mgr.14538 172.21.15.69:0/379550600' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T05:08:32.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:32 smithi098 bash[23790]: audit 2024-04-02T05:08:32.047330+0000 mon.a (mon.0) 4211 : audit 1 from='mgr.14538 ' entity='mgr.y' 2024-04-02T05:08:33.018 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:32 smithi069 systemd[1]: Stopping Ceph mon.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:33.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:33 smithi069 bash[21528]: debug 2024-04-02T05:08:33.014+0000 7f0239be2700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-02T05:08:33.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:33 smithi069 bash[21528]: debug 2024-04-02T05:08:33.014+0000 7f0239be2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-02T05:08:34.128 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service' 2024-04-02T05:08:34.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:33 smithi069 bash[221676]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-mon-a 2024-04-02T05:08:34.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:34 smithi069 bash[221735]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-mon-a 2024-04-02T05:08:34.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:34 smithi069 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-02T05:08:34.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 02 05:08:34 smithi069 systemd[1]: Stopped Ceph mon.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T05:08:34.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:34.195 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-02T05:08:34.195 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-02T05:08:34.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.c 2024-04-02T05:08:34.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:08:34 smithi098 bash[48594]: debug there is no tcmu-runner data available 2024-04-02T05:08:34.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:34 smithi069 systemd[1]: Stopping Ceph mon.c for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:34.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:34 smithi069 bash[28904]: debug 2024-04-02T05:08:34.366+0000 7fdc303ac700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-02T05:08:34.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 02 05:08:34 smithi069 bash[28904]: debug 2024-04-02T05:08:34.366+0000 7fdc303ac700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-02T05:08:35.482 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.c.service' 2024-04-02T05:08:35.507 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:35.508 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-02T05:08:35.508 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-02T05:08:35.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.b 2024-04-02T05:08:35.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:35 smithi098 systemd[1]: Stopping Ceph mon.b for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:35 smithi098 bash[23790]: debug 2024-04-02T05:08:35.736+0000 7ffa30313700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-02T05:08:35.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Apr 02 05:08:35 smithi098 bash[23790]: debug 2024-04-02T05:08:35.736+0000 7ffa30313700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-02T05:08:37.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mon.b.service' 2024-04-02T05:08:37.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:37.035 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-02T05:08:37.035 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-02T05:08:37.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y 2024-04-02T05:08:37.404 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 02 05:08:37 smithi069 systemd[1]: Stopping Ceph mgr.y for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:38.133 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-04-02T05:08:38.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:38.157 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-02T05:08:38.157 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-02T05:08:38.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.x 2024-04-02T05:08:38.473 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 02 05:08:38 smithi098 systemd[1]: Stopping Ceph mgr.x for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:39.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-04-02T05:08:39.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:39.221 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-02T05:08:39.221 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-02T05:08:39.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.0 2024-04-02T05:08:39.533 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 05:08:39 smithi069 systemd[1]: Stopping Ceph osd.0 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:39.533 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 05:08:39 smithi069 bash[31935]: debug 2024-04-02T05:08:39.342+0000 7f470beb1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:08:39.533 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 05:08:39 smithi069 bash[31935]: debug 2024-04-02T05:08:39.342+0000 7f470beb1700 -1 osd.0 970 *** Got signal Terminated *** 2024-04-02T05:08:39.533 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 05:08:39 smithi069 bash[31935]: debug 2024-04-02T05:08:39.342+0000 7f470beb1700 -1 osd.0 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:08:46.033 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 05:08:45 smithi069 bash[221969]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-0 2024-04-02T05:08:46.034 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 02 05:08:45 smithi069 bash[222035]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-0 2024-04-02T05:08:47.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.0.service' 2024-04-02T05:08:47.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:47.912 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-02T05:08:47.912 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-02T05:08:47.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.1 2024-04-02T05:08:48.283 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 05:08:47 smithi069 systemd[1]: Stopping Ceph osd.1 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:48.283 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 05:08:48 smithi069 bash[37262]: debug 2024-04-02T05:08:48.102+0000 7fa4d8d88700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:08:48.283 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 05:08:48 smithi069 bash[37262]: debug 2024-04-02T05:08:48.102+0000 7fa4d8d88700 -1 osd.1 970 *** Got signal Terminated *** 2024-04-02T05:08:48.283 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 05:08:48 smithi069 bash[37262]: debug 2024-04-02T05:08:48.102+0000 7fa4d8d88700 -1 osd.1 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:08:54.446 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 05:08:54 smithi069 bash[222172]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-1 2024-04-02T05:08:54.446 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 02 05:08:54 smithi069 bash[222237]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-1 2024-04-02T05:08:56.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.1.service' 2024-04-02T05:08:56.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:08:56.346 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-02T05:08:56.346 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-02T05:08:56.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.2 2024-04-02T05:08:56.783 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 05:08:56 smithi069 systemd[1]: Stopping Ceph osd.2 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:08:56.783 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 05:08:56 smithi069 bash[42900]: debug 2024-04-02T05:08:56.514+0000 7f8c52abb700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:08:56.783 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 05:08:56 smithi069 bash[42900]: debug 2024-04-02T05:08:56.514+0000 7f8c52abb700 -1 osd.2 970 *** Got signal Terminated *** 2024-04-02T05:08:56.783 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 05:08:56 smithi069 bash[42900]: debug 2024-04-02T05:08:56.514+0000 7f8c52abb700 -1 osd.2 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:08:59.473 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:08:59 smithi098 bash[52341]: ts=2024-04-02T05:08:59.201Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-02T05:08:59.474 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:08:59 smithi098 bash[52341]: ts=2024-04-02T05:08:59.203Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-02T05:08:59.474 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:08:59 smithi098 bash[52341]: ts=2024-04-02T05:08:59.206Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-02T05:08:59.474 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:08:59 smithi098 bash[52341]: ts=2024-04-02T05:08:59.206Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-02T05:09:02.779 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 05:09:02 smithi069 bash[222372]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-2 2024-04-02T05:09:02.789 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 02 05:09:02 smithi069 bash[222434]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-2 2024-04-02T05:09:04.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.2.service' 2024-04-02T05:09:04.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:04.583 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-02T05:09:04.583 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-02T05:09:04.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.3 2024-04-02T05:09:05.033 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:04 smithi069 systemd[1]: Stopping Ceph osd.3 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:05.033 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:04 smithi069 bash[48521]: debug 2024-04-02T05:09:04.757+0000 7fb2e48c0700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:09:05.033 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:04 smithi069 bash[48521]: debug 2024-04-02T05:09:04.757+0000 7fb2e48c0700 -1 osd.3 970 *** Got signal Terminated *** 2024-04-02T05:09:05.033 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:04 smithi069 bash[48521]: debug 2024-04-02T05:09:04.757+0000 7fb2e48c0700 -1 osd.3 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:09:05.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:05 smithi098 bash[38371]: debug 2024-04-02T05:09:05.352+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:06.149 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:05 smithi098 bash[27343]: debug 2024-04-02T05:09:05.800+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:06.473 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:06 smithi098 bash[38371]: debug 2024-04-02T05:09:06.368+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:06.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:06 smithi098 bash[32651]: debug 2024-04-02T05:09:06.144+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:07.111 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:06 smithi098 bash[27343]: debug 2024-04-02T05:09:06.848+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:07.406 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:07 smithi098 bash[32651]: debug 2024-04-02T05:09:07.108+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:07.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:07 smithi098 bash[38371]: debug 2024-04-02T05:09:07.404+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:07.783 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:07 smithi069 bash[48521]: debug 2024-04-02T05:09:07.405+0000 7fb2e0eb4700 -1 osd.3 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.321013+0000 front 2024-04-02T05:08:41.320965+0000 (oldest deadline 2024-04-02T05:09:07.221170+0000) 2024-04-02T05:09:08.154 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:07 smithi098 bash[27343]: debug 2024-04-02T05:09:07.880+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:08.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:08 smithi098 bash[32651]: debug 2024-04-02T05:09:08.148+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:08.474 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:08 smithi098 bash[38371]: debug 2024-04-02T05:09:08.352+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:08.783 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:08 smithi069 bash[48521]: debug 2024-04-02T05:09:08.445+0000 7fb2e0eb4700 -1 osd.3 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.321013+0000 front 2024-04-02T05:08:41.320965+0000 (oldest deadline 2024-04-02T05:09:07.221170+0000) 2024-04-02T05:09:09.136 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:08 smithi098 bash[27343]: debug 2024-04-02T05:09:08.844+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:09.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:09 smithi098 bash[32651]: debug 2024-04-02T05:09:09.132+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:09.473 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:09 smithi098 bash[38371]: debug 2024-04-02T05:09:09.368+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:09.783 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:09 smithi069 bash[48521]: debug 2024-04-02T05:09:09.477+0000 7fb2e0eb4700 -1 osd.3 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.321013+0000 front 2024-04-02T05:08:41.320965+0000 (oldest deadline 2024-04-02T05:09:07.221170+0000) 2024-04-02T05:09:10.129 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:09 smithi098 bash[27343]: debug 2024-04-02T05:09:09.824+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:10.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:10 smithi098 bash[32651]: debug 2024-04-02T05:09:10.124+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:10.473 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:10 smithi098 bash[38371]: debug 2024-04-02T05:09:10.340+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:10.973 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:10 smithi098 bash[27343]: debug 2024-04-02T05:09:10.812+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:10.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:10 smithi098 bash[69630]: debug 2024-04-02T05:09:10.716+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:11.283 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:10 smithi069 bash[222569]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-3 2024-04-02T05:09:11.283 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 02 05:09:11 smithi069 bash[222634]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-3 2024-04-02T05:09:11.388 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:11 smithi098 bash[32651]: debug 2024-04-02T05:09:11.120+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:11.691 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:11 smithi098 bash[38371]: debug 2024-04-02T05:09:11.384+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:11.973 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:11 smithi098 bash[27343]: debug 2024-04-02T05:09:11.856+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:11.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:11 smithi098 bash[69630]: debug 2024-04-02T05:09:11.684+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:12.435 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:12 smithi098 bash[32651]: debug 2024-04-02T05:09:12.112+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:12.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.3.service' 2024-04-02T05:09:12.719 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:12 smithi098 bash[38371]: debug 2024-04-02T05:09:12.428+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:12.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:12.736 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-02T05:09:12.736 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-02T05:09:12.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.4 2024-04-02T05:09:12.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:12 smithi098 bash[69630]: debug 2024-04-02T05:09:12.716+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:12.973 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:12 smithi098 systemd[1]: Stopping Ceph osd.4 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:12.973 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:12 smithi098 bash[27343]: debug 2024-04-02T05:09:12.900+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:13.468 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[32651]: debug 2024-04-02T05:09:13.068+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:13.468 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[27343]: debug 2024-04-02T05:09:13.184+0000 7fde94dd9700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:09:13.468 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[27343]: debug 2024-04-02T05:09:13.184+0000 7fde94dd9700 -1 osd.4 970 *** Got signal Terminated *** 2024-04-02T05:09:13.468 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[27343]: debug 2024-04-02T05:09:13.184+0000 7fde94dd9700 -1 osd.4 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:09:13.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[38371]: debug 2024-04-02T05:09:13.460+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:14.062 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[27343]: debug 2024-04-02T05:09:13.924+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:14.062 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:13 smithi098 bash[69630]: debug 2024-04-02T05:09:13.760+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:14.468 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:14 smithi098 bash[32651]: debug 2024-04-02T05:09:14.056+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:14.721 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:14 smithi098 bash[38371]: debug 2024-04-02T05:09:14.464+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:14.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:14 smithi098 bash[69630]: debug 2024-04-02T05:09:14.716+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:14.973 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:14 smithi098 bash[27343]: debug 2024-04-02T05:09:14.944+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:15.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:15 smithi098 bash[32651]: debug 2024-04-02T05:09:15.100+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:15.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:15 smithi098 bash[38371]: debug 2024-04-02T05:09:15.476+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:15.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:15 smithi098 bash[38371]: debug 2024-04-02T05:09:15.476+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:16.128 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:15 smithi098 bash[69630]: debug 2024-04-02T05:09:15.748+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:16.128 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:15 smithi098 bash[27343]: debug 2024-04-02T05:09:15.960+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:16.128 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:15 smithi098 bash[27343]: debug 2024-04-02T05:09:15.960+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.061348+0000 front 2024-04-02T05:08:52.061317+0000 (oldest deadline 2024-04-02T05:09:14.961025+0000) 2024-04-02T05:09:16.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[32651]: debug 2024-04-02T05:09:16.124+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:16.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[32651]: debug 2024-04-02T05:09:16.124+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:16.780 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[38371]: debug 2024-04-02T05:09:16.520+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:16.780 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[38371]: debug 2024-04-02T05:09:16.520+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:17.113 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[27343]: debug 2024-04-02T05:09:16.916+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:17.113 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[27343]: debug 2024-04-02T05:09:16.916+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.061348+0000 front 2024-04-02T05:08:52.061317+0000 (oldest deadline 2024-04-02T05:09:14.961025+0000) 2024-04-02T05:09:17.113 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:16 smithi098 bash[69630]: debug 2024-04-02T05:09:16.772+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:17.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[32651]: debug 2024-04-02T05:09:17.112+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:17.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[32651]: debug 2024-04-02T05:09:17.112+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:17.797 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[38371]: debug 2024-04-02T05:09:17.496+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:17.797 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[38371]: debug 2024-04-02T05:09:17.496+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:18.119 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[27343]: debug 2024-04-02T05:09:17.956+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.058691+0000 front 2024-04-02T05:08:41.058651+0000 (oldest deadline 2024-04-02T05:09:05.758280+0000) 2024-04-02T05:09:18.120 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[27343]: debug 2024-04-02T05:09:17.956+0000 7fde913cd700 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.061348+0000 front 2024-04-02T05:08:52.061317+0000 (oldest deadline 2024-04-02T05:09:14.961025+0000) 2024-04-02T05:09:18.120 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[69630]: debug 2024-04-02T05:09:17.792+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:18.120 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:17 smithi098 bash[69630]: debug 2024-04-02T05:09:17.792+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:18.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:18 smithi098 bash[32651]: debug 2024-04-02T05:09:18.112+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:18.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:18 smithi098 bash[32651]: debug 2024-04-02T05:09:18.112+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:18.843 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:18 smithi098 bash[38371]: debug 2024-04-02T05:09:18.540+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:18.843 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:18 smithi098 bash[38371]: debug 2024-04-02T05:09:18.540+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:18.843 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:18 smithi098 bash[69630]: debug 2024-04-02T05:09:18.744+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:18.844 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:18 smithi098 bash[69630]: debug 2024-04-02T05:09:18.744+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:19.473 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[101753]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-4 2024-04-02T05:09:19.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[32651]: debug 2024-04-02T05:09:19.120+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:19.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[32651]: debug 2024-04-02T05:09:19.120+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:19.973 INFO:journalctl@ceph.osd.4.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[101817]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-4 2024-04-02T05:09:19.973 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[38371]: debug 2024-04-02T05:09:19.496+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:19.973 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[38371]: debug 2024-04-02T05:09:19.496+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:19.974 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[69630]: debug 2024-04-02T05:09:19.744+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:19.974 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:19 smithi098 bash[69630]: debug 2024-04-02T05:09:19.744+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:20.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:20 smithi098 bash[32651]: debug 2024-04-02T05:09:20.116+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:20.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:20 smithi098 bash[32651]: debug 2024-04-02T05:09:20.116+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:20.973 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:20 smithi098 bash[38371]: debug 2024-04-02T05:09:20.548+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:20.973 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:20 smithi098 bash[38371]: debug 2024-04-02T05:09:20.548+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:20.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:20 smithi098 bash[69630]: debug 2024-04-02T05:09:20.716+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:20.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:20 smithi098 bash[69630]: debug 2024-04-02T05:09:20.716+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:21.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[32651]: debug 2024-04-02T05:09:21.132+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:21.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[32651]: debug 2024-04-02T05:09:21.132+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:21.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.4.service' 2024-04-02T05:09:21.812 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[38371]: debug 2024-04-02T05:09:21.560+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:21.813 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[38371]: debug 2024-04-02T05:09:21.560+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:21.813 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[69630]: debug 2024-04-02T05:09:21.736+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:21.813 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[69630]: debug 2024-04-02T05:09:21.736+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:21.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:21.815 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-02T05:09:21.815 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-02T05:09:21.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.5 2024-04-02T05:09:22.188 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:21 smithi098 systemd[1]: Stopping Ceph osd.5 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:22.188 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[32651]: debug 2024-04-02T05:09:21.992+0000 7ff941665700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:09:22.188 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[32651]: debug 2024-04-02T05:09:21.992+0000 7ff941665700 -1 osd.5 970 *** Got signal Terminated *** 2024-04-02T05:09:22.188 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:21 smithi098 bash[32651]: debug 2024-04-02T05:09:21.992+0000 7ff941665700 -1 osd.5 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:09:22.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:22 smithi098 bash[32651]: debug 2024-04-02T05:09:22.180+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:22.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:22 smithi098 bash[32651]: debug 2024-04-02T05:09:22.180+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:22.792 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:22 smithi098 bash[38371]: debug 2024-04-02T05:09:22.536+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:22.792 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:22 smithi098 bash[38371]: debug 2024-04-02T05:09:22.536+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:23.179 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:22 smithi098 bash[69630]: debug 2024-04-02T05:09:22.784+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:23.179 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:22 smithi098 bash[69630]: debug 2024-04-02T05:09:22.784+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:23.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[32651]: debug 2024-04-02T05:09:23.176+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:23.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[32651]: debug 2024-04-02T05:09:23.176+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:23.973 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[38371]: debug 2024-04-02T05:09:23.544+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:23.974 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[38371]: debug 2024-04-02T05:09:23.544+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:23.974 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[38371]: debug 2024-04-02T05:09:23.544+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:23.974 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[69630]: debug 2024-04-02T05:09:23.788+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:23.974 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:23 smithi098 bash[69630]: debug 2024-04-02T05:09:23.788+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:24.477 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[32651]: debug 2024-04-02T05:09:24.160+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:24.477 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[32651]: debug 2024-04-02T05:09:24.160+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:24.840 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[38371]: debug 2024-04-02T05:09:24.508+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:24.841 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[38371]: debug 2024-04-02T05:09:24.508+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:24.841 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[38371]: debug 2024-04-02T05:09:24.508+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:25.169 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[69630]: debug 2024-04-02T05:09:24.836+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:25.169 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:24 smithi098 bash[69630]: debug 2024-04-02T05:09:24.836+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:25.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[32651]: debug 2024-04-02T05:09:25.168+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:25.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[32651]: debug 2024-04-02T05:09:25.168+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:25.879 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[38371]: debug 2024-04-02T05:09:25.480+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:25.880 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[38371]: debug 2024-04-02T05:09:25.480+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:25.880 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[38371]: debug 2024-04-02T05:09:25.480+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:26.150 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[69630]: debug 2024-04-02T05:09:25.876+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:26.151 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:25 smithi098 bash[69630]: debug 2024-04-02T05:09:25.876+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:26.472 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[32651]: debug 2024-04-02T05:09:26.148+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:41.957586+0000 front 2024-04-02T05:08:41.957094+0000 (oldest deadline 2024-04-02T05:09:05.456856+0000) 2024-04-02T05:09:26.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[32651]: debug 2024-04-02T05:09:26.148+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.360102+0000 front 2024-04-02T05:08:52.360002+0000 (oldest deadline 2024-04-02T05:09:15.859855+0000) 2024-04-02T05:09:26.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[32651]: debug 2024-04-02T05:09:26.148+0000 7ff93dc59700 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.561185+0000 front 2024-04-02T05:09:00.561252+0000 (oldest deadline 2024-04-02T05:09:25.261295+0000) 2024-04-02T05:09:26.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[38371]: debug 2024-04-02T05:09:26.468+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:26.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[38371]: debug 2024-04-02T05:09:26.468+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:26.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[38371]: debug 2024-04-02T05:09:26.468+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:27.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[69630]: debug 2024-04-02T05:09:26.852+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:27.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:26 smithi098 bash[69630]: debug 2024-04-02T05:09:26.852+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:27.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:27 smithi098 bash[38371]: debug 2024-04-02T05:09:27.432+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:27.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:27 smithi098 bash[38371]: debug 2024-04-02T05:09:27.432+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:27.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:27 smithi098 bash[38371]: debug 2024-04-02T05:09:27.432+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:28.124 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:27 smithi098 bash[69630]: debug 2024-04-02T05:09:27.860+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:28.124 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:27 smithi098 bash[69630]: debug 2024-04-02T05:09:27.860+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:28.124 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:27 smithi098 bash[69630]: debug 2024-04-02T05:09:27.860+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:28.423 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[101951]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-5 2024-04-02T05:09:28.423 INFO:journalctl@ceph.osd.5.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[102017]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-5 2024-04-02T05:09:28.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[38371]: debug 2024-04-02T05:09:28.416+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:28.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[38371]: debug 2024-04-02T05:09:28.416+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:28.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[38371]: debug 2024-04-02T05:09:28.416+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:29.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[69630]: debug 2024-04-02T05:09:28.828+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:29.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[69630]: debug 2024-04-02T05:09:28.828+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:29.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:28 smithi098 bash[69630]: debug 2024-04-02T05:09:28.828+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:29.655 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:29 smithi098 bash[38371]: debug 2024-04-02T05:09:29.376+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:29.655 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:29 smithi098 bash[38371]: debug 2024-04-02T05:09:29.376+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:29.655 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:29 smithi098 bash[38371]: debug 2024-04-02T05:09:29.376+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:29.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:29 smithi098 bash[69630]: debug 2024-04-02T05:09:29.844+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:29.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:29 smithi098 bash[69630]: debug 2024-04-02T05:09:29.844+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:29.973 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:29 smithi098 bash[69630]: debug 2024-04-02T05:09:29.844+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:30.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.5.service' 2024-04-02T05:09:30.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:30.374 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-02T05:09:30.374 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-02T05:09:30.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.6 2024-04-02T05:09:30.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[38371]: debug 2024-04-02T05:09:30.420+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:30.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[38371]: debug 2024-04-02T05:09:30.420+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:30.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[38371]: debug 2024-04-02T05:09:30.420+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:30.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 systemd[1]: Stopping Ceph osd.6 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:30.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[38371]: debug 2024-04-02T05:09:30.560+0000 7ff2e9c14700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:09:30.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[38371]: debug 2024-04-02T05:09:30.560+0000 7ff2e9c14700 -1 osd.6 970 *** Got signal Terminated *** 2024-04-02T05:09:30.724 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[38371]: debug 2024-04-02T05:09:30.560+0000 7ff2e9c14700 -1 osd.6 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:09:31.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[69630]: debug 2024-04-02T05:09:30.828+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:31.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[69630]: debug 2024-04-02T05:09:30.828+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:31.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:30 smithi098 bash[69630]: debug 2024-04-02T05:09:30.828+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:31.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:31 smithi098 bash[38371]: debug 2024-04-02T05:09:31.464+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:31.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:31 smithi098 bash[38371]: debug 2024-04-02T05:09:31.464+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:31.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:31 smithi098 bash[38371]: debug 2024-04-02T05:09:31.464+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:32.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:31 smithi098 bash[69630]: debug 2024-04-02T05:09:31.856+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:32.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:31 smithi098 bash[69630]: debug 2024-04-02T05:09:31.856+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:32.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:31 smithi098 bash[69630]: debug 2024-04-02T05:09:31.856+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:32.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:32 smithi098 bash[38371]: debug 2024-04-02T05:09:32.440+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:32.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:32 smithi098 bash[38371]: debug 2024-04-02T05:09:32.440+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:32.723 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:32 smithi098 bash[38371]: debug 2024-04-02T05:09:32.440+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:33.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:32 smithi098 bash[69630]: debug 2024-04-02T05:09:32.848+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:33.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:32 smithi098 bash[69630]: debug 2024-04-02T05:09:32.848+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:33.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:32 smithi098 bash[69630]: debug 2024-04-02T05:09:32.848+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:33.880 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[38371]: debug 2024-04-02T05:09:33.472+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:34.095 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[38371]: debug 2024-04-02T05:09:33.472+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:34.095 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[38371]: debug 2024-04-02T05:09:33.472+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:34.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[69630]: debug 2024-04-02T05:09:33.872+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:34.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[69630]: debug 2024-04-02T05:09:33.872+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:34.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[69630]: debug 2024-04-02T05:09:33.872+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:34.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:33 smithi098 bash[69630]: debug 2024-04-02T05:09:33.872+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:34.924 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[38371]: debug 2024-04-02T05:09:34.524+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:39.918241+0000 front 2024-04-02T05:08:39.918331+0000 (oldest deadline 2024-04-02T05:09:05.217553+0000) 2024-04-02T05:09:34.925 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[38371]: debug 2024-04-02T05:09:34.524+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:49.919295+0000 front 2024-04-02T05:08:49.918962+0000 (oldest deadline 2024-04-02T05:09:14.618685+0000) 2024-04-02T05:09:34.925 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[38371]: debug 2024-04-02T05:09:34.524+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:09:00.922155+0000 front 2024-04-02T05:09:00.922180+0000 (oldest deadline 2024-04-02T05:09:23.221888+0000) 2024-04-02T05:09:34.925 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[38371]: debug 2024-04-02T05:09:34.524+0000 7ff2e6208700 -1 osd.6 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:06.723588+0000 front 2024-04-02T05:09:06.723508+0000 (oldest deadline 2024-04-02T05:09:33.723806+0000) 2024-04-02T05:09:35.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[69630]: debug 2024-04-02T05:09:34.920+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:35.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[69630]: debug 2024-04-02T05:09:34.920+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:35.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[69630]: debug 2024-04-02T05:09:34.920+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:35.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:34 smithi098 bash[69630]: debug 2024-04-02T05:09:34.920+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:36.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:35 smithi098 bash[69630]: debug 2024-04-02T05:09:35.956+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:36.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:35 smithi098 bash[69630]: debug 2024-04-02T05:09:35.956+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:36.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:35 smithi098 bash[69630]: debug 2024-04-02T05:09:35.956+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:36.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:35 smithi098 bash[69630]: debug 2024-04-02T05:09:35.956+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:37.223 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:36 smithi098 bash[102149]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-6 2024-04-02T05:09:37.223 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 02 05:09:36 smithi098 bash[102214]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-6 2024-04-02T05:09:37.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:36 smithi098 bash[69630]: debug 2024-04-02T05:09:36.948+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:37.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:36 smithi098 bash[69630]: debug 2024-04-02T05:09:36.948+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:37.224 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:36 smithi098 bash[69630]: debug 2024-04-02T05:09:36.948+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:37.224 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:36 smithi098 bash[69630]: debug 2024-04-02T05:09:36.948+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:38.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:37 smithi098 bash[69630]: debug 2024-04-02T05:09:37.944+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:38.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:37 smithi098 bash[69630]: debug 2024-04-02T05:09:37.944+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:38.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:37 smithi098 bash[69630]: debug 2024-04-02T05:09:37.944+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:38.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:37 smithi098 bash[69630]: debug 2024-04-02T05:09:37.944+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:38.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.6.service' 2024-04-02T05:09:38.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:38.981 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-02T05:09:38.982 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-02T05:09:38.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:38 smithi098 bash[69630]: debug 2024-04-02T05:09:38.936+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:38 smithi098 bash[69630]: debug 2024-04-02T05:09:38.936+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:38 smithi098 bash[69630]: debug 2024-04-02T05:09:38.936+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:38 smithi098 bash[69630]: debug 2024-04-02T05:09:38.936+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 systemd[1]: Stopping Ceph osd.7 for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.152+0000 7f5afa91f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:09:39.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.152+0000 7f5afa91f700 -1 osd.7 970 *** Got signal Terminated *** 2024-04-02T05:09:39.224 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.152+0000 7f5afa91f700 -1 osd.7 970 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T05:09:40.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.912+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:40.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.912+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:40.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.912+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:40.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.912+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:40.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:39 smithi098 bash[69630]: debug 2024-04-02T05:09:39.912+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.98:6804 osd.4 since back 2024-04-02T05:09:18.152087+0000 front 2024-04-02T05:09:18.152177+0000 (oldest deadline 2024-04-02T05:09:39.252005+0000) 2024-04-02T05:09:41.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:40 smithi098 bash[69630]: debug 2024-04-02T05:09:40.960+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:41.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:40 smithi098 bash[69630]: debug 2024-04-02T05:09:40.960+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:41.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:40 smithi098 bash[69630]: debug 2024-04-02T05:09:40.960+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:41.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:40 smithi098 bash[69630]: debug 2024-04-02T05:09:40.960+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:41.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:40 smithi098 bash[69630]: debug 2024-04-02T05:09:40.960+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.98:6804 osd.4 since back 2024-04-02T05:09:18.152087+0000 front 2024-04-02T05:09:18.152177+0000 (oldest deadline 2024-04-02T05:09:39.252005+0000) 2024-04-02T05:09:42.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:41 smithi098 bash[69630]: debug 2024-04-02T05:09:41.928+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:42.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:41 smithi098 bash[69630]: debug 2024-04-02T05:09:41.928+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:42.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:41 smithi098 bash[69630]: debug 2024-04-02T05:09:41.928+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:42.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:41 smithi098 bash[69630]: debug 2024-04-02T05:09:41.928+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:42.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:41 smithi098 bash[69630]: debug 2024-04-02T05:09:41.928+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.98:6804 osd.4 since back 2024-04-02T05:09:18.152087+0000 front 2024-04-02T05:09:18.152177+0000 (oldest deadline 2024-04-02T05:09:39.252005+0000) 2024-04-02T05:09:43.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:42 smithi098 bash[69630]: debug 2024-04-02T05:09:42.940+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:43.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:42 smithi098 bash[69630]: debug 2024-04-02T05:09:42.940+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:43.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:42 smithi098 bash[69630]: debug 2024-04-02T05:09:42.940+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:43.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:42 smithi098 bash[69630]: debug 2024-04-02T05:09:42.940+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:43.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:42 smithi098 bash[69630]: debug 2024-04-02T05:09:42.940+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.98:6804 osd.4 since back 2024-04-02T05:09:18.152087+0000 front 2024-04-02T05:09:18.152177+0000 (oldest deadline 2024-04-02T05:09:39.252005+0000) 2024-04-02T05:09:44.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:43 smithi098 bash[69630]: debug 2024-04-02T05:09:43.955+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-02T05:08:44.147002+0000 front 2024-04-02T05:08:44.146843+0000 (oldest deadline 2024-04-02T05:09:10.046843+0000) 2024-04-02T05:09:44.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:43 smithi098 bash[69630]: debug 2024-04-02T05:09:43.955+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-02T05:08:52.948589+0000 front 2024-04-02T05:08:52.948316+0000 (oldest deadline 2024-04-02T05:09:17.048130+0000) 2024-04-02T05:09:44.223 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:43 smithi098 bash[69630]: debug 2024-04-02T05:09:43.955+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-02T05:08:57.049212+0000 front 2024-04-02T05:08:57.049181+0000 (oldest deadline 2024-04-02T05:09:27.049470+0000) 2024-04-02T05:09:44.224 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:43 smithi098 bash[69630]: debug 2024-04-02T05:09:43.955+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-02T05:09:07.050486+0000 front 2024-04-02T05:09:07.050250+0000 (oldest deadline 2024-04-02T05:09:32.950159+0000) 2024-04-02T05:09:44.224 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:43 smithi098 bash[69630]: debug 2024-04-02T05:09:43.955+0000 7f5af6f13700 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.98:6804 osd.4 since back 2024-04-02T05:09:18.152087+0000 front 2024-04-02T05:09:18.152177+0000 (oldest deadline 2024-04-02T05:09:39.252005+0000) 2024-04-02T05:09:45.473 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:45 smithi098 bash[102347]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T05:09:45.473 INFO:journalctl@ceph.osd.7.smithi098.stdout:Apr 02 05:09:45 smithi098 bash[102411]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T05:09:47.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@osd.7.service' 2024-04-02T05:09:47.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:47.404 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-02T05:09:47.404 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-04-02T05:09:47.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@rgw.foo.a 2024-04-02T05:09:47.783 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Apr 02 05:09:47 smithi069 systemd[1]: Stopping Ceph rgw.foo.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:47.783 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Apr 02 05:09:47 smithi069 bash[52915]: debug 2024-04-02T05:09:47.545+0000 7f18f6480700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-02T05:09:47.783 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Apr 02 05:09:47 smithi069 bash[52915]: debug 2024-04-02T05:09:47.545+0000 7f1904cffac0 -1 shutting down 2024-04-02T05:09:58.233 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Apr 02 05:09:58 smithi069 bash[222771]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-rgw-foo-a 2024-04-02T05:09:58.247 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@rgw.foo.a.service' 2024-04-02T05:09:58.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:58.270 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-04-02T05:09:58.271 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-02T05:09:58.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@prometheus.a 2024-04-02T05:09:58.662 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 systemd[1]: Stopping Ceph prometheus.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:09:58.662 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-02T05:09:58.662 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-02T05:09:58.662 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.414Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.415Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.415Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-02T05:09:58.663 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Apr 02 05:09:58 smithi098 bash[52341]: ts=2024-04-02T05:09:58.415Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-02T05:09:59.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@prometheus.a.service' 2024-04-02T05:09:59.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T05:09:59.279 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-02T05:09:59.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T05:09:59.444 INFO:teuthology.orchestra.run.smithi069.stdout:Deleting cluster with fsid: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T05:10:04.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:04.283 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:04.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:04.853 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:04.898 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:04.899 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.150 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.150 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:04 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.436 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.436 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.437 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.437 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.699 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.699 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.961 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:05.961 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:05 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:06.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:06 smithi069 systemd[1]: Stopping Ceph node-exporter.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:10:07.033 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:06 smithi069 bash[223222]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-node-exporter-a 2024-04-02T05:10:07.033 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:06 smithi069 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-02T05:10:07.033 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:06 smithi069 bash[223284]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-node-exporter-a 2024-04-02T05:10:07.033 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:06 smithi069 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-02T05:10:07.033 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:06 smithi069 systemd[1]: Stopped Ceph node-exporter.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T05:10:07.368 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 02 05:10:07 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:07.369 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:07 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:07.369 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:07 smithi069 systemd[1]: Stopping Ceph alertmanager.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:10:07.369 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:07 smithi069 bash[59696]: ts=2024-04-02T05:10:07.301Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-02T05:10:08.258 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:07 smithi069 bash[223353]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-alertmanager-a 2024-04-02T05:10:08.258 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:08 smithi069 bash[223411]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-alertmanager-a 2024-04-02T05:10:08.259 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:08 smithi069 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-02T05:10:08.259 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:08 smithi069 systemd[1]: Stopped Ceph alertmanager.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T05:10:08.532 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 02 05:10:08 smithi069 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:08.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T05:10:08.991 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T05:10:13.840 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:13 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:13.841 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:13 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:13.841 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:13 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.102 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.103 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.402 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.402 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.703 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.704 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.955 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.955 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:14.955 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:14 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:15.223 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:15 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:15.223 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:15 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:15.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:15 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:15.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:15 smithi098 systemd[1]: Stopping Ceph iscsi.iscsi.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:10:15.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:15 smithi098 bash[48594]: debug Shutdown received 2024-04-02T05:10:26.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:25 smithi098 bash[102884]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-iscsi-iscsi-a 2024-04-02T05:10:26.468 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:26 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:26.468 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:26 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:26.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:26 smithi098 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-04-02T05:10:26.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:26 smithi098 bash[102949]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-iscsi-iscsi-a 2024-04-02T05:10:26.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:26 smithi098 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-04-02T05:10:26.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:26 smithi098 systemd[1]: Stopped Ceph iscsi.iscsi.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T05:10:26.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi098.stdout:Apr 02 05:10:26 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:27.792 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:27.792 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:27.792 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:27.793 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:27 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:28.223 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:27 smithi098 systemd[1]: Stopping Ceph node-exporter.b for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:10:28.996 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:28 smithi098 bash[103203]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-node-exporter-b 2024-04-02T05:10:28.996 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:28 smithi098 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-02T05:10:28.996 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:28 smithi098 bash[103263]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-node-exporter-b 2024-04-02T05:10:28.996 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:28 smithi098 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-02T05:10:28.996 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:28 smithi098 systemd[1]: Stopped Ceph node-exporter.b for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T05:10:29.390 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:29 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:29.391 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:29 smithi098 systemd[1]: Stopping Ceph grafana.a for f978400e-f0a9-11ee-b647-cb9ed24678a4... 2024-04-02T05:10:29.391 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Apr 02 05:10:29 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:29.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:29 smithi098 bash[51598]: logger=server t=2024-04-02T05:10:29.39059022Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-02T05:10:29.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:29 smithi098 bash[51598]: logger=ticker t=2024-04-02T05:10:29.390672007Z level=info msg=stopped last_tick=2024-04-02T05:10:20Z 2024-04-02T05:10:29.658 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:29 smithi098 bash[51598]: logger=tracing t=2024-04-02T05:10:29.390695822Z level=info msg="Closing tracing" 2024-04-02T05:10:30.670 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:30 smithi098 bash[103330]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-grafana-a 2024-04-02T05:10:30.670 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:30 smithi098 bash[103392]: Error response from daemon: No such container: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4-grafana-a 2024-04-02T05:10:30.670 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:30 smithi098 systemd[1]: ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-04-02T05:10:30.671 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:30 smithi098 systemd[1]: Stopped Ceph grafana.a for f978400e-f0a9-11ee-b647-cb9ed24678a4. 2024-04-02T05:10:30.671 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Apr 02 05:10:30 smithi098 systemd[1]: /etc/systemd/system/ceph-f978400e-f0a9-11ee-b647-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-02T05:10:31.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T05:10:31.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T05:10:31.245 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-02T05:10:31.247 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/remote/smithi069/crash 2024-04-02T05:10:31.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T05:10:31.288 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T05:10:31.288 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2024-04-02T05:10:31.290 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/remote/smithi098/crash 2024-04-02T05:10:31.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T05:10:31.302 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T05:10:31.302 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-04-02T05:10:31.304 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-02T05:10:31.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-02T05:10:31.351 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-02T05:10:31.351 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-02T05:10:31.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[SEC\]' /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-02T05:10:31.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]' /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-02T05:10:31.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[WRN\]' /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-02T05:10:31.516 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-02T05:10:31.517 INFO:tasks.cephadm:Compressing logs... 2024-04-02T05:10:31.517 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T05:10:31.561 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T05:10:31.569 INFO:teuthology.orchestra.run.smithi069.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-02T05:10:31.575 INFO:teuthology.orchestra.run.smithi098.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-02T05:10:31.580 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T05:10:31.582 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-04-02T05:10:31.582 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-04-02T05:10:31.582 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-04-02T05:10:31.582 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-04-02T05:10:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T05:10:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-04-02T05:10:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-04-02T05:10:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.3.log: 89.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T05:10:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T05:10:31.585 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/objectstore_tool.7.log 2024-04-02T05:10:31.587 INFO:teuthology.orchestra.run.smithi098.stderr: 81.6% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T05:10:31.590 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/tcmu-runner.log 2024-04-02T05:10:31.590 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-04-02T05:10:31.590 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/tcmu-runner.log: -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T05:10:31.590 INFO:teuthology.orchestra.run.smithi098.stderr: 67.3% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/tcmu-runner.log.gz 2024-04-02T05:10:31.590 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T05:10:31.590 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T05:10:31.597 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.1.log: /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log: 93.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T05:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-04-02T05:10:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-volume.log: 92.5% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-04-02T05:10:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-04-02T05:10:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.audit.log: 95.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/objectstore_tool.7.log.gz 2024-04-02T05:10:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-04-02T05:10:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T05:10:31.605 INFO:teuthology.orchestra.run.smithi098.stderr: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T05:10:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.4.log: 85.5% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T05:10:31.609 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-volume.log: 91.8% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T05:10:31.609 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T05:10:31.612 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T05:10:31.613 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log 2024-04-02T05:10:31.617 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T05:10:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.audit.log: /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log: 95.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T05:10:31.908 INFO:teuthology.orchestra.run.smithi069.stderr: 93.6% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log.gz 2024-04-02T05:10:32.138 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-04-02T05:10:32.837 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-04-02T05:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr: 92.5% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-04-02T05:10:34.382 INFO:teuthology.orchestra.run.smithi069.stderr: 91.8% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-04-02T05:10:38.406 INFO:teuthology.orchestra.run.smithi098.stderr: 93.9% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-02T05:10:38.541 INFO:teuthology.orchestra.run.smithi098.stderr: 94.2% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-02T05:10:41.410 INFO:teuthology.orchestra.run.smithi098.stderr: 94.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-02T05:10:41.701 INFO:teuthology.orchestra.run.smithi069.stderr: 94.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-02T05:10:41.713 INFO:teuthology.orchestra.run.smithi069.stderr: 94.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-02T05:10:41.882 INFO:teuthology.orchestra.run.smithi098.stderr: 94.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-02T05:10:41.883 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-04-02T05:10:41.883 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m10.317s 2024-04-02T05:10:41.883 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m33.808s 2024-04-02T05:10:41.883 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m1.424s 2024-04-02T05:10:41.929 INFO:teuthology.orchestra.run.smithi069.stderr: 94.0% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-02T05:10:42.247 INFO:teuthology.orchestra.run.smithi069.stderr: 94.1% -- replaced with /var/log/ceph/f978400e-f0a9-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-02T05:10:42.248 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-02T05:10:42.248 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m10.685s 2024-04-02T05:10:42.248 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m44.560s 2024-04-02T05:10:42.248 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m1.683s 2024-04-02T05:10:42.249 INFO:tasks.cephadm:Archiving logs... 2024-04-02T05:10:42.249 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/remote/smithi069/log 2024-04-02T05:10:42.250 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T05:10:47.544 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/remote/smithi098/log 2024-04-02T05:10:47.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T05:10:51.726 INFO:tasks.cephadm:Removing cluster... 2024-04-02T05:10:51.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 --force 2024-04-02T05:10:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:Deleting cluster with fsid: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T05:10:53.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f978400e-f0a9-11ee-b647-cb9ed24678a4 --force 2024-04-02T05:10:53.490 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: f978400e-f0a9-11ee-b647-cb9ed24678a4 2024-04-02T05:10:54.859 INFO:tasks.cephadm:Removing cephadm ... 2024-04-02T05:10:54.859 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T05:10:54.865 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T05:10:54.871 INFO:tasks.cephadm:Teardown complete 2024-04-02T05:10:54.871 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T05:10:54.881 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T05:10:54.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T05:10:54.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T05:10:54.979 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T05:10:54.988 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T05:10:54.989 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T05:10:54.991 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T05:10:55.002 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-04-02T05:10:55.002 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-04-02T05:10:55.002 INFO:teuthology.orchestra.run.smithi098.stdout:*hv01.front.sepi 44.4.53.2 2 u 128 128 377 0.085 -0.789 0.537 2024-04-02T05:10:55.002 INFO:teuthology.orchestra.run.smithi098.stdout:+hv02.front.sepi 63.231.80.2 3 u 1 128 377 0.089 +0.231 0.651 2024-04-02T05:10:55.002 INFO:teuthology.orchestra.run.smithi098.stdout:+hv03.front.sepi 74.6.168.72 3 u 5 128 377 0.100 -6.641 0.311 2024-04-02T05:10:55.002 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-02T05:10:55.004 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2024-04-02T05:10:55.004 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2024-04-02T05:10:55.004 INFO:teuthology.orchestra.run.smithi069.stdout:*hv01.front.sepi 44.4.53.2 2 u 127 128 377 0.082 -2.425 2.078 2024-04-02T05:10:55.004 INFO:teuthology.orchestra.run.smithi069.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 128 377 0.106 -0.317 1.913 2024-04-02T05:10:55.004 INFO:teuthology.orchestra.run.smithi069.stdout:+hv03.front.sepi 74.6.168.72 3 u 12 128 377 0.076 -9.687 2.760 2024-04-02T05:10:55.004 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-02T05:10:55.004 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T05:10:55.013 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T05:10:55.013 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T05:10:55.045 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T05:10:55.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T05:10:55.114 INFO:teuthology.task.internal:Duration was 2861.662879 seconds 2024-04-02T05:10:55.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T05:10:55.146 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T05:10:55.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T05:10:55.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T05:10:55.186 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T05:10:55.186 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-04-02T05:10:55.186 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T05:10:55.237 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-04-02T05:10:55.237 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T05:10:55.247 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T05:10:55.248 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T05:10:55.281 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T05:10:55.406 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T05:10:55.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T05:10:55.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T05:10:55.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T05:10:55.873 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T05:10:55.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T05:10:55.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T05:10:55.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T05:10:55.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:10:55.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:10:55.937 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-04-02T05:10:55.954 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-04-02T05:10:55.977 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:10:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:10:55.995 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:10:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:10:56.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T05:10:56.039 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T05:10:56.040 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/remote/smithi069 2024-04-02T05:10:56.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T05:10:56.109 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/remote/smithi098 2024-04-02T05:10:56.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T05:10:56.143 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T05:10:56.143 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T05:10:56.146 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T05:10:56.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T05:10:56.202 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T05:10:56.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T05:10:56.249 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T05:10:56.249 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T05:10:56.252 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T05:10:56.256 INFO:teuthology.orchestra.run.smithi069.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 05:10 /home/ubuntu/cephtest 2024-04-02T05:10:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 05:10 /home/ubuntu/cephtest 2024-04-02T05:10:56.260 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T05:10:56.281 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T05:10:56.356 INFO:teuthology.nuke:Checking targets against current locks 2024-04-02T05:10:56.381 DEBUG:teuthology.nuke:shortname: smithi069 2024-04-02T05:10:56.382 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T05:10:56.405 DEBUG:teuthology.nuke:shortname: smithi098 2024-04-02T05:10:56.406 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T05:10:56.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:14:49.115539', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP+U30wtui0wY2oeBYbZqBievav6XXSANBn6lZOKYlIggCY/eZy+GPZhGjCfwjfGOgPq4IAllYWlG36iUPUivRk='} 2024-04-02T05:10:56.431 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:14:49.113955', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6C5vGUfmkJ7b9kYheJBNcdxX2++k3gI92WASO5w8vmpR7OEjJWL8hIsZMFJ9j6XHK4Fsp0bPaVnO5AZUPcOXU='} 2024-04-02T05:10:56.463 INFO:teuthology.orchestra.console.smithi069:Power off 2024-04-02T05:10:56.464 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T05:10:56.487 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-04-02T05:10:56.487 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:10:56.505 INFO:teuthology.orchestra.console.smithi098:Power off 2024-04-02T05:10:56.505 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T05:10:56.515 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-02T05:10:56.528 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-04-02T05:10:56.528 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:10:56.551 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-02T05:11:00.516 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:11:00.552 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:11:00.628 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-02T05:11:00.662 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-02T05:11:04.630 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:11:04.664 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:11:04.744 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-02T05:11:04.774 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-02T05:11:08.746 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:11:08.777 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:11:08.857 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-04-02T05:11:08.888 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-04-02T05:11:08.959 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-04-02T05:11:08.990 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-04-02T05:11:09.094 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} duration: 2861.662878513336 failure_reason: '"2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-04-02T05:11:09.094 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T05:11:09.174 INFO:teuthology.run:FAIL